Have patience - it will come out eventually. The important thing is to not get frustrated and get the sledge hammer out! Years ago, I had a '51 Navy that just about drove me nuts the first time I tried to get the wedge to slide out on it. I finally applied penetrating oil to it, let it sit for a day or so and then warmed the area up with a propane torch. I then used a brass hammer on it - there was enough sticking out - and eventually it let go. When I examined the wedge and slot in the cylinder pin, I discovered some small "burrs" left over from the machining. A couple of swipes on an oily stone honed them off of the wedge and a fine file cleaned up the slot in the cylinder pin. After that, it was easier to get the wedge to "let go" and with time, it worked like it was intended to. In this day of NC machining, etc., it seems like the fit should be almost perfect but you still have humans doing the work - some with heavier hands than others!
Some like the Colts, some like the Remingtons, some like the Rugers - all for different reasons. The truth be known, anybody that likes shooting BP revolvers generally likes them all - just some more than others! My g-g-grandfather carried a Remington in the 11th Mich. Cav. - for that reason, I wanted one and greatly enjoy it. But, I still love the '51 Colt Navy as well - good luck with the wedge - it will come out eventually. Just be careful to not get frustrated and end up damaging or marring the surrounding area. I've read a number of things in regards to the originals and they had their problems as well. As in the days of the originals - when they lived with the styles and designs and their shortcomings - we do the same with the replicas - that's one of the things that gives us the appreciation of what our ancestors put up with - and what makes the hobby fun!