In my experience it's more the weather.
If there are lots of groundhogs around, you'll find enough to have fun practically all day long. The absolute best time depends on the weather. They are similar to people in that respect. If the weather is so-so, some will come out. The nicer the weather, the more will be out. If mid afternoon is blistering hot, there will be a lull in the action and it will pick up in the evening. If it's been raining for three days, the first nice day will be good all day long.
In the fall after the farmers cut most of the "stuff" out is a good time too. The food that was nearby is gone and the groundhogs have to travel farther, and they are trying to stuff themselves for winter.