In my experience, the primer probably has the least to do with overall accuracy of a hand load. That's not to say that a primer isn't an important part of the finished product but bullet and powder load will be where the biggest gains are made.
I'm sure others are likely do disagree.......as each cartridge combination and gun is different........my statement is from observations working with my own rifles........so it is general, not specific in nature.
Testing is really the only way to tell........
I have found primer and weight of powder to be extremely correlated. Several cases in point:
A buddy and I have DW 357 Supermags. I purchased them direct from DW and they have consecutive serial numbers. I found a sweet load for mine using Remington 7 1/2 primers. My buddy, using same cast bullets (he casts them), same powder (I bought two 8lb jugs same lot) but Winchester primers, had to boost his powder 2.0 grains to get same velocity as my load. When the velocity came, so did the accuracy.
I have a Remington XP-100 with Hart 7TCU barrel on it. I knew a specific bullet and powder would be accurate. I had some Federal Match primers that I thought would give me minimum deviations in velocity. I tried my load in four weight variations in the area I knew it should give excellent accuracy. My best group was about 2" at 100 yards. Good std deviations, but rotten accuracy. I took that best load and changed only the primer. Remington 7 1/2s shrunk the group to about 1". Winchester primers gave me just under 1/2". This is two 5 shots groups at 100 under 1/2".
I try working up my loads around a specific primer that has given me good std deviations with specific powders. If you get a group that looks promising, don't just give up. Try switching primers and see if you get what you are looking for. $1.50 for a different primer is cheaper than $20 for a different powder.
Steve