Don't mind about those who matter because those who matter don't mind. No matter how kind or good you are, people will always talk. They tend to blind themselves with the bad in you and don't get to see the good in you. 

Live your life the way you want to, cause the last time I checked the road to heaven doesn't pass through any one's house. If someone can't love you for who you are that's their problem. There are thousands of people in this world and the right one will come at the right time. Don't live your life to impress someone rather than yourself. 

Life is short so enjoy it to the fullest.