In my eyes, wrong accusations hurt that much, because
No matter how hard you try
No matter how much you work on yourself to better yourself
No matter how many thoughts you have spent on that topic
No matter how hard you try to break your habits
No matter what, people will always see your old mistakes.
They tend to stick to your past mistakes and totally ignore your attempts to better yourself.
It's like a NO MATTER HOW HARD YOU TRY - IT DOES NOT MATTER.
People tend to put their focus on the bad, not the good.