This sounded more and more crazy to me as I kept reading.

That's nowhere near enough to retire. It's not even close.

I would only consider retiring if I had a fully paid off house (to lock down my expenses as much as possible) and a portfolio big enough where a 3% annual withdrawal is equivalent to an "upper middle class" salary - adjusted annually for inflation after the first year.
I know the "rule" is 4% but that has a small chance to fail. I'm not taking that chance.