>>281717548
Because he feels like he's being compared unfavorably to her admitted trash husband. She basically challenged his ability to please her by way of saying her husband was better than him. He abandoned his paper-thin morals at the slightest provocation because he didn't want to be seen as lesser than him.
Remember, this isn't the first time Gomi caved after putting up a weak resistance. He wants to act like he's got a spine but he blows over super easily because, in the end, it's all empty justification as he pursues pleasure and validation. However, the problem that lies therein is that he does actually have some vague notion of morality and what's right, but he's such a slave to his need for approval and affection that he can't help but cave. He's got some idea that what he's doing is wrong and his gut tells him frequently that he should walk away or be more careful, but he's so weak-willed that he gets swept up in the vortex of Yumezora's depravity.