Ethnic male + white female couples tend to be lower class. Lower class people don't care if their children grow up poor because it's what they did themselves
White men with non white women are primarily college educated and middle class. Middle class families tend to wait to have children untill they can afford to porovide for them and are deeper into their careers. Hence they wait later and have less kids
