It's not an Ushijima and Oikawa interaction without "You should have come to Shirotorizawa" being said.
Bruh imagine them in a normal workforce. Some years in the force goes by and Oikawa comes back from the big bosses office being denied a promotion and as he's walking back to his cubicle Ushijima is just shaking his head at the water station sipping at his paper cup, looks him straight in the eyes and says "you should have come to Shirotorizawa"