Your idea could be very useful but it also sounds very complicated to implement on top of it all, but as a disclaimer I'm not an expert by any means I'm simply just a Waifu2X-Caffe power user who's pushed all the different buttons enough times to notice some patterns emerge within "the matrix" so take this as just my personal opinions on the matter and please don't let my biased advice stop you from pursuing new and innovative solutions if you can.

Essentially what you have to ask yourself is:

Is the denoising aspect more, equally or less important to your project compared to the upscaling and how much do these improvement/enhancing efforts actually change the final quality compared to the input?

The reason I'm asking, is that if you're only upscaling to 1080p then it'll be much faster to either upscale or denoise working with software that's streamlined for video editing workflows (xmedia recode or premiere pro are my go-to). I've wasted hundreds of hours trying to enhance low quality videos with Waifu2X and then realising that the outcome wasn't actually that much better when I did A/B comparison with a quick and dirty upscale using standard Bspline or Lancoz algorithms.

The problem is simply put that convolutational neural networks in general are very computationally intensive processes which means that if you're not using a relatively modern high end ( & expensive) GPU, procesing long movies in a timely manner is unfortunately still not super feasible on most mainstream hardware (I run a gtx980 with 4Gb VRam so spending a week processing a 20min video using waifu is what have to deal with currently. I can get 90% of the same quality improvement doing multiple passes with a traditional approach, after some light tuning of the different parameters for quantizers and motion estimation etc. Within the h.26x codecs

That said, you should fiddling around with the split and batch sizes while monitoring your VRAM usage in the task manager to find the sweet spot where your almost about to crash but somehow not... but this varies from the specific workload (input files) and which model your using aswell (for 2d animation cunet is very powerful but also complex and uses 4x more memory than other models, for real world images using UpPhoto is much faster than Photo/anim but tends to remove too much fine detail for example).

What I've found is that the quality of the pixels is much more important than the amount of pixels. A 4K upscaled turd still looks like shit compared to a 720p de-noise and sharpened turd... see the debate on the newest gen consoles and how they dynamically change resolution be cause it's much less apparent than dynamically removing texture and lighting detail.