Asparagus grows here like a weed. Unfortunately, I couldn't tell you the secret to making it thrive. I've never actually planted any - All I do is cut some when I want it. The patch of asparagus we have here on the family farm was planted sometime back in the early 70's by my great grandfather, and its still here today.
Now, as far as veggies that cause me grief - that would have to be carrots primarily. My garden is a fairly heavy sandy clay, and it gets too tight as the summer progresses for the root crops to develp well. The result is a lot of small, oddly shaped carrots. Potatoes and sweet potatoes fall into the same category. If they do grow well, it seems like rodents chew half of them up, so I dont put a lot of effort into them.