The pain of being neglected and hated was not something he could either forget or forgive. To live in a house where he was forgotten until his parents decided he needed a punishment cause Harry Potter to grow up unloved while his young brother was worshiped. He simply could not forgive them. Instead, he would make them pay for everything the did to him.