Loki had committed treason on a level Asgard had never seen. He would not be welcome back to his home realm. He would remain at the mercy of those he tried to control: the humans, SHIELD, and, of course, HYDRA. By the time the Avengers come knocking HYDRA's door, Loki's long gone in every sense but the physical. Fortunately, even in his silent, fearful state, he's managed to make some friends. Wanda and Pietro won't let anyone hurt him. The Avengers are nothing but a threat, and Ultron makes them a promise that they'll be free of those so-called heroes soon enough. (Basically, it's an Age of Ultron au where Wanda and Pietro have adopted a very broken Loki and they're determined to keep him safe.)