I've known several couples that started small like that, but haven't met any that downsized like that while their kids were at home. I've known people that lived in old-style travel trailers that were smaller than that, and they were quite nice. One couple added a nice screen porch to theirs and it was one of the nicest places to visit.
The ones that started small never had a mortgage. They'd buy small (usually a run-down mobile home), then keep saving for the next one, and eventually end up in a nice house that was paid off and new when they hit middle age.
I don't know that it's necessarily better, but it is different and the bank doesn't get as much money from you in the long run.