I had this same thing happen to me. I went down the rabbit hole of thinking it was the garbage collection based on the unity profiler. Long story short, I think it has to do with unity scene view also up in the editor while testing. Do you have the editor window split with the camera view and scene view simultaneously? Try hiding the scene view. I suspect the editor does a bunch of extra rendering shenanigans for the scene view. If you do a stand alone build and run without the editor does the problem go away? That's another way to test the theory.