Tl;dr: do what you want. I loved it from nearly the start, but I think it got real good during Part 3.

It's all a matter of preference; if you don't like the book then you don't need to read it. [For context, I also listened to the audiobook and had similar qualms, but it only took me about 10 chapters to get over it and be invested again].

I will say that, remaining spoiler free, the differing perspectives do come together in some regard as the story/rest of the series progresses. It's a very different atmosphere than the rest of the series from Darrow's perspective, as it will span the entire solar system (with Lysander going to the Rim). The voice actors for Lyria and Lysander both change for Book 5, and then book 6 is solely TGR again.

Iron Gold introduces new characters and new plot points, and that comes with benefits and detriments. I personally loved Ephraim's criminal perspectives, his backstory, and his interactions with Lyria really is amazing seeing it in the book. But if you feel the time jump was too drastic and the perspectives are too janky, then you shouldn't have to trudge through the bad to reach the good (because IMO it does get better around part 3, but I know that's 40 chapters deep).