>>534837249
Philip literally protests "But he has a wife and children!" in the cutscene where Mr.clown is fucking with him, so he just assumed his master was a cheater on top of it.
The thing we'll never know is if these were just passing thoughts that he had randomly, or to make himself feel better about the inadequacy he felt, he was actively imagining this all the time, as Oswald refuses to give Pluto a clear answer when he asks about the nature of the vision.