If he was talking to you, but wouldn't come in, he probably had some hens with him. Hard to get them away from the real thing.
Last year, in a big hayfield, my buddy, his dad, and myself all watched a huge old tom put on show for 3 hours. He was HOT!! He called back to us like crazy, but he wasn't leaving his girls. I stalked to within 40 yard, waiting for him to top the knoll, when BLAM!! Chad's dad had another bird walk out 10 yards from where he was sitting. It never uttered a sound. We had no idea it was there. Mike kept telling me sorry for messing up my stalk, and I told him he as a goober, if he hadn't shot that bird, I'd have beaten him silly. 5 minutes later, we were walking over the hill, and another bird was 25 yards from where we were standing. Chad got that one. Father and son busted two birds 5 minutes apart, both with 8" beards. It was too cool.
I just started myself about 2 years ago, being serious about it, I mean. What kind of calls are you using? I stink at diaphrams, I can't get them to do anything. Jeez, I can't even whistle though, so I must have a defective mouth?
I use slates and box calls. Stick with it, turkeys aren't perfect, and they don't care if you sound like you are. My 6 year old called in a tom last year with my box call, and I thought he sounded horrible.
He got that bird fired up though. Too bad it was on the wrong side of the creek!
I wish I could help with the hunting, but the farms I us will only let family hunt them.