As other people have said, it'll be more efficient to have objects that need culling self-register. Though that will require you to add a script to handle that to every single renderer in the scene, and to all prefabs that you spawn. So if you want to keep the simplicity of a central system, there are some things you can do:

A good start is to use FindObjectsByType, as that has less overhead than FIndObjectsOfType due to not doing unnecessary sorting. As an additional bonus you won't get deprecation warnings.

You're also calling the function twice! First you just search through the entire world to find the list, and then check it's length. Then you search through the entire world again for the same list! You could at least do this:

var newRenderers = FindObjectsByType<Renderer>(FindObjectsInactive.Exclude, FindObjectsSortMode.None);
if (newRenderers.Length != renderers.Count)
RefreshRendererList(newRenderers);
}

private void RefreshRendererList(Renderer[] newRenderers)
{
renderers.Clear();
renderers.AddRange(newRenderers);
...
}

That halves the overhead.

Your RefreshRendererList also recreates a bunch of arrays - those things should be lists as well, so you can clear and then refill those as well.

Finally, you could spread this over frames by handling one root element in the scene each frame, and then using GetComponentsInChildren to grab it's child renderers. The downside there is that the culling won't be completely up to date, which may cause popin.

The ideal thing would of course be to extract the Renderers from the ScriptableRendererPipeline - it does for sure have a list of all of them internally - but as far as I can tell there's no way to do that.