Obviously attention is a problem but it should all be okay if you make sure to set time for each member with themselves and your self, and the whole of the harem.
Special needs may be a problem; ie: panic attacks, mental breakdowns, nightmares, or needing to be clingy for a bit.
With these you want to make sure they're dealt with swiftly but carefully. Do not rush a mental or panic attack. Those are fragile and may hurt the harem party suffering the attack.
