Banzai attack and Kamikaze tactics used by the Japanese were in the minds of U.S. planners along with the knowledge that Japan was building its own atomic bomb.
there's some truth to that - the Japanese defense of Iwo Jima was very tough, and that contributed toward the overly-high casualty estimates for Coronet/Olympic, and it hardened resolve. Note that on Iwo there weren't much in the way of banzai attacks, being that they weren't very effective against jarheads with semi-auto rifles; instead they used to silent night attacks, to deadlier effect.
Japanese submarine attacks and fire balloon attacks along the west coast of the U.S. and Canada was clear evidence they were ready to use whatever means needed to defeat us.
Japanese sub warfare was surprisingly impotent; very little success for the effort expended, most of it wasted on operations against warships instead of commerce (and they were up against the 2nd best ASW force in the world, after the RN). 'fire balloon' attacks were meaningless gestures.
Seeing part of a freighter that had been torpedo by a Japanese submarine in the Crescent City harbor made clear to me just how close the enemy was.
http://www.militarymuseum.org/Emedio.html
Deceptive; very little effective IJN efforts near US shores.
Nuke might have been the right answer - we don't know, it
was the road taken. There is an argument to be made that moving off the unconditional surrender demand, combined with continued sub/carrier ops/starvation of Japan, might have done the trick. We'll never know. And of course, the Russians might have a
lot more of Japan than they do now, had we dallied.