OP here: I'm giving up. I tried changing ref/bframe count and fucking with the psy/trellis stuff but none of that even made a dent. The veryslow preset also seems eek out the practical limits of H264 compression efficiency as using placebo improved my SSIM score very little and was like 10X slower.

SSIM 0.955608

There must be some hidden spooky esoteric -x264-params that YIFY might have used that we all don't know about but given what >>106240591 posted it does sound like they cheated by making 1080p video easier to encode with something like HQDN3D at the cost of destroying grain and fine detail.