The Doctor never forgot the Ponds. They never really forgot him, either. They held a mainly normal life, got jobs, had children. But they never really forgot. And that was their fatal mistake.