Finally got the new text rendering working, poor kerning is a thing of the past. Some layout stuff obviously needs fixing but I'm happy that the text actually looks right now.
Still using freetype, but with harfbuzz actually rustybuzz for shaping and positioning which seems to have improved things.
Also had an issue with gamma correction that was affecting the previous text rendering too, so the subpixel stuff in general just looks way better now.