Ivan has a really hard time looking women (other than his sisters) in the eye because his sisters made him afraid of most other women. He naturally looks a little below their faces so that it looks like he is looking at them eye-to-eye. However, women often think he is staring at their boobs, so most of the time he gets slapped and called a pervert (or sometimes hit on, which spooks him even more), which further solidifies his fear of women. To him, they're hurting him ad being mean for no reason.
Alfred feels super bad for him because he always looks a bit downcast whenever it happens, so he started trying to coach Ivan on looking above their face if not eye-to-eye. Whenever it happens and he is around, he'll hold Ivan's hand and squeeze it every time he catches him looking downwards.
It has been happening less and less, but sometimes it happens anyway and Alfred has to try and explain why they react like that. It normally doesn't end well though. Ivan just doesn't really understand it because "I am not trying to look there, they just assume that I am!"
There are rare times when Alfred gets frustrated with trying to explain and starts yelling, which always causes Ivan to tear up and yell things back, like, "It's not my fault!! I am not even looking at their boobs!!!"
- Aella