I think he has hated them his whole life too but I don't think he didn't express those feelings because he values his relationship with Robin. I think he has expressed his true feelings about them and he values Robin to hide behind while expressing those feelings.