Gary and Ash are half brother. Gary knows about the affair, that’s why he hates Ash. In his teenage brain, his mother left because of Ash not because of his horny dad.