Mike, The bigger the better in my opinion. I have a large work area so I made a 4X8' reloading table that doubles for cleaning guns and even some "light" gunsmithing. In your space, a 4X4' would be nice. It's great to be able to work on all sides, kinda like a pool table. I have a bench vise mounted on one corner and my presses mounted on one side. That leaves plenty of room to spread out stuff when I load. You'll end up with a case trimmer, lube pad, scale, tricklier, primer tray, loading blocks, and other "stuff" spread over the surface.
I built my table out of 2X4s and particle board. The top is 3/4" and the shelves are 1/2". The standard height for a work bench is 30" but I'm tall and like to stand straight when I work. So, I made the top 36" from the floor. Works great for me. I built two full sized shelves so, counting the floor, I have 3 levels of storage, about 1 foot high. This comes in very handy for storing empty brass, bullets, powder, tools, cleaning supplies, and accessories. The table is free standing and plenty heavy for the press. With the shelves, it eliminates the need for external storage.
In my last house, I had a bench connected to a wall. It worked OK but I found a more accessible surface really helps. Good luck on your project.