"The more time I spend with them, the less time they spend with my community," Vargas told Newsweek.