De Bruyne was abandoned, but when the money came, his family wanted to take him back