>>105588207
I think so. It was around the same time that he increased the lr for an epoch and it broke the model. I know he rolled the lr change back, but I think he kept that quasi-distillation method.