He had to have known about the Dursleys’ opinions on magic and how they treated Harry. And in the same vein he had to have known what it was like for Tom Riddle in the orphanage. If he’d tried harder to get through to him, Tom might not have become the Dark Lord.
But mostly because when I read the sixth book he reminded me of a boy that used to study with me. The perfect golden boy that got all the attention, the best grades, popularity when really he was the most narcissistic and mean person. He looked down on everyone and always blamed others for things he did and always got away with that.