Link, a fresh-faced Highschool graduate, knew almost nothing of the Imprisoned Servents. Zelda told him they were a gang of thieves, but Impa said otherwise.
"Link, the Imprisoned Servents are a mob of sorts. They are drug pushers. Crossing paths with them is very, very dangerous. IF you ever meet one, be cautious."
He didn't WANT to cross paths with anyone from the Imprisoned Servents, but the way Impa spoke was very matter of fact. She was a strange woman. She often knew the future and seemed wiser than her years. Well, it's not hard to sound wise to a nineteen-year-old when you're in your late twenties.
and why, pray do tell, was Link recollecting his thoughts of this conversation?
Because he somehow got caught up in the Imprisoned Servents.