You could keep the weight down by using a low wall action. I don't have one to compare to a Handi, but IIRC they are a little smaller in the action area. With a thin barrel you'd have a pretty light little gun.
Have you heard of the 'Chicoppee' action? It was created by Frank DeHaas and written up in his "Mr Single Shot's Book of Rifle Plans" It is a fairly straight forward action buildable in a small shop. You could do it with limited machines if you have the skill. It's also a fairly streamlined action and can be built in either rimfire or small centerfire (two builds are shown in the book, a 22lr and a centerfire that was something like a 218 or a 22hornet). A few guys on the web have put up pictures of their pistol caliber builds. There is one on weapons guild being built as a single shot pistol in 45LC. 357mag should be fine. 357max might be a bit over reaching, but the mag should be OK with proper material selection.
Check out DeHaas' book if you are thinking of a single shot rifle build. He has a few books, all are worth getting if you can find them.
