"We could have been happy - I know that. And it is perhaps the hardest thing to know."