I was just sitting at the mall taking a break from mall walking. A man walked over to me when I wasn’t even looking up. He said "I haven’t seen you here before. " I said I wasn’t really here to talk to people and got up and walked away. He said he would leave, he may even have said sorry. I said I was ready to leave anyway. In retrospect I think he may have just been slightly mentally challenged so I wish I had been a little softer, he did not turn out to be aggressive. Looking for suggestions on something still clear and firm but more polite? This scared me for some reason, I think how fast he walked up and sat down near me.
Editing to add a key detail, he sat down close to me which is probably what made me more nervous than I would have otherwise been. I appreciate all the thoughtful feedback I’ve gotten and interesting discussion.


Oh well, it’s a marginally different situation. Might as well just discount my entire comment. It’s what I’m going to do with your forest-for-the-trees comment rather than engage any further.
Their comment directly responds to your point by claiming that the mall, where the OP’s incident took place, is a “place with an implied invitation to socialize”.