Someone is "mistaken" about the property line and land ownership back there. That much is certain. Maybe a meeting is in order with the adjacent property owner - if you are on speaking terms.
Contact the Game Warden and local Sheriff. This isn't the sort of thing that you do by yourself. Eventually, the Law is going to have to get involved and you want them on your side with your "hands clean" in this.
Ask the neighbor if they have granted access to their land for hunting this season as you are trying to reduce "misunderstandings". Ask if it is OK for you to cross the property line to retrieve down deer, should one wander their way. Grant them the same privilege provided they call you first (and you escort them if desired). Exchange phone numbers. That would be a good start. Be friendly and non-accusatory at this stage as there is no evidence to the contrary that leads you to them - unless they admit something or act strange (to you) in any way.
Maybe put the tree stand back in place along with multiple game cameras to watch the site and several cameras to watch the game cameras watching the site (just in case the first one or two are found and destroyed or stolen). Game cameras have gotten real cheap (half a dozen for less than $500.00) and do a fine job of taking fine grained pictures from which blow-ups and prosecution can be achieved. Hunting men with cameras isn't so much different than photographing animals, and in this instance maybe even easier as you know exactly where they will be, just not when.
Seems to me that you are not an absentee land owner. Correct me if I am wrong. Circle the property daily. Eventually, that will bring you face to face with the trespasser(s). Look for parked trucks/vehicles that do not belong there. Snap pics of the license plate(s).
Find them, contact Law Enforcement immediately by cell phone for backup, take the trespasser's picture if you can with telephoto lens for close-up ID, and wait for the LE to support your approach.
Deer hunting is not something to die for. It is important, but not like that. A confrontation won't be good in the woods between a trespasser and an angry land owner, both of whom will be armed.