My headcanon is that the reason he scratched out the eyes of Caleb, is because when he found Caleb so young compared to him, instead of it being from good nutrition and healthy lifestyles with his witch wife, he had learned about the Grimwalkers and thought, in the fireglow, his brother's eyes were crimson red like the Grimwalkers, and he attacked them to try and save him. Then instead it just paralyzed Caleb, and he was dying from his injuries, Phillip tried to solve him with his limited puritan medicine, but after giving up and going to Evelyn and getting blasted at (but not fully cursed that was by his own hand with the Palisman) he returned to find his brother dead and a decision to not leave until every witch on the Isles was dead. The complication of a human from the future made him wait to clear as much as he could, before deciding to expand his scams into a full on cult empire.