I think that Xaden and Garrick are using Imogen's love for Garrick to make her do things for them. Most of the time she's described as being in the room or area she doesn't talk, she's just there. A possible reason for her being quiet is maybe when they aren't with Violet, Xaden and Garrick could possibly be punishing Imogen for saying something they didn't want her to say. If Imogen gets these punishments when she says something Xaden disagrees with, of course she would want to talk less.
Garrick and Imogen's relationship is pretty one sided so I don't think he would have much trouble yelling or pointing out Imogen's flaws with Xaden. This could also be the reason to why Imogen and Quinn are best friends; Imogen doesn't feel comfortable or safe near the other marked ones so she hangs out with Quinn (an unmarked one).
