While you guys wait for my new upscale setup, I've tested and recommend giving this a try: https://github.com/the-database/VideoJaNai
It does have some flaws like not handling DAR, fucked up framerate or letting you set model scale.
But testing it with ESRGAN's AnimeVideoV3 model at 4x, upscaling from 720p to 1440p went as fast as 1.03x speed, around something like 25 fps.
It is insanely fast, even when I was forced to use 4x instead of 2x. It comes included with AnimeJaNai model that is 2x which ended up being about 1.84x speed at around 44 fps, but I'm not really fond of how AnimeJaNai looks. It doesn't clean up as good.
The really handy thing is that ontop of using TensorRT, it's also upscaling directly into a video, saving you time as it doesn't need to extract frames, upscale then encode back into a video.
You do however need to convert the ESRGAN pth model with ChaiNNer. Which I've done so you can just use mine: https://files.catbox.moe/82oiik.onnx
Just rename to realesr-animevideov3.onnx if you care about the filename.
In the GUI, I recommend setting the ffmpeg parameters to this:
libx264 -crf 15 -preset slow -pix_fmt yuv420p10le -refs 1 -bsf:v h264_metadata=matrix_coefficients=6 -max_interleave_delta 0
VideoJaNai needs to use .mkv rather than .mp4 and it will mux the subtitles and such into the upscaled video, though one could just remove that manually if one wanted.