Can my landlord and neighbors ban a friend of mine they don't like due to sexual orientation from visiting me?

My landlord a couple months ago starting spreading lies about a friend of mine. My friend is a trans gender, however he is being accused of being a sex offender due to his sexual orientation (he cross dresses). He doesn't bother the neighbors and he keeps to himself when he and his wife visits. They visit often to help my son and I out, due to no transportation. Because I don't allow smoking in my house, he goes out to his truck to smoke and surf on the internet on his phone. Due to the lies spread from the landlord, the parents in my trailer park told their children more extreme lies about him and therefore the children started to harass my son and tell more lies about my friend. About a month ago the landlord got all the neighbors to gang up on me and my friends accusing my friend of taking pictures of the little girls in the park, just because my friend was on his phone surfing while he smokes. There were no pictures on his phone and even showed the officer that was called by the landlord to try to forcibly remove my friends. One of neighbors was trying provoke my friend by yelling defamatory comments towards my friend concerning his sexual orientation. His wife held him back and got him back indoors until the officer came to diffuse the situation. The officer ran my friend's name and said that my friend is NOT a sex offender and he laughed about the pictures on the phone, as they were all pictures of our dogs. There were no children pictures at all on the phone! The officer recommended that he file an anti-harassment against the neighbors, but I don't know their names. He was having trouble finding an attorney and the money to do it all. Now I'm hearing the neighbors are putting together a petition to keep them out. I can't use the bus system due to both mine and my son's disabilities and they are my only option to get rides to get groceries and such. We are at a lost as what to do to stop this slander, lies, and harassment. I am also worried about retaliation from my landlord. I am a single parent and can't afford to move and I am on housing authority.