Today I learned that godot doesn't extract textures automatically from a GLTF file and compress them if you don't have your import settings configured. This texture was 12MB while embedded in the GLTF file. Also it doesn't generate mimaps which was causing this nauseating shimmering for things at a distance.