When Brimsley gets the job to be the queen's man, he is overjoyed at the excitement of working at living at the Palace. It is everything he dreamed of: the palace itself, his job, and someone he meets. The writers of Queen Charlotte/Bridgerton probably will not come up with a back story for Brimsley and Reynolds so this is my interpretation of how they came to be.