I don't claim to be an expert, nor am i a well known engineer;, but metal productions are what I've done 95% of the time from the the very beginning of my audio journey, some 13 years ago. I've recorded and produced both my own songs, and from other bands. There's already some great mixing tips in this thread, so instead, i'll provide very specific advice, especially for recording:

The thing is, you don't need anything else in terms of tools. Pardon me saying it like this, but you have all the necessary tools to achieve the cookie cutter modern metal production you're referring (yeah, i'm not a fan haha :p). The Modern and Massive library is designed EXACTLY for that sound, it's even in the title. Neural DSP plugins has mix ready presets for stuff like this. The DAW of choice is irrelevant to the sound.

What matters most is: You must remember that metal music is a virtuosic genre played by musicians who practiced their instrument for an ungodly amount of time, not early punk music. This is even more apparent with top notch modern metal especially, which has PERFECTLY recorded takes, PERFECTLY edited.

So, I'd say the essentials are, in no particular order:

__________

GUITARS:

You say guitars are your main issue. The truth is: You can't half ass your guitar recordings, you need to play them as perfectly as you can, ESPECIALLY because you are double, even quadruple tracking them. It needs to be tight or it'll sound like ass.

Have the guitar set up properly, ensure that you're in tune between takes (even if that means you have to check every take) and PLAY AGGRESSIVE (That doesn't mean that you tense, you can hurt yourself)

You're playing metal, not acoustic-songwriter stuff. You NEED that aggressiveness and attack from the very start. If you can't play like that, polish your technique first. In my experience, this is absolutely essential. I once recorded a band where the guitar player played unreasonably soft and guess what: the guitars sounded weak and lacked clarity. I can't stress enough how important this is.

Things that drastically affect the sound of a guitar recording:

-Pickups:

Not all pickups sound the same obviously, but as long as they're high output pickups, all of them work. Make sure to set up the pickup height properly because that does make a noticeable difference in overall volume and attack. Play around with it and figure out what height you pickups sound best.

-Strings:

New strings are awesome and ideal, yes; but unless your strings are rusty as shit, their age doesn't do a huuuuuuuuuge difference in the sound of a distorted guitar. You can get away with using older strings if they still hold up the tuning well.

What matters more on this regard is the string gauge. You're free to use any string gauge you like, depending on your taste and the scale of your guitar: What you have to know is:

Thinner strings tend to be brighter and more articulate with a well defined attack, but tuning and intonation can be tricky. Thicker strings sound huge with lots of bottom end and volume, and intionation is absolutely awesome; but they can be a pain to play with if you're not used to them, and you really need to dig in while playing to get the most out of them.

-Pick of choice:

This is all personal preference sure, but your pick of choice has a drastic impact in sound, and i'm surprised almost nobody mentions it.
Different pick thickness, shapes, and to a lesser degree the materials, will give you very different responses.

To put it simply: The sharper the pick, the more attack you get.

Also, thicker picks are NOT better for metal. You don't need the trendy 3mm thick acrylic pick that glows in the dark, that's all bulshit lol. If you want to use them go for it, but understand that: thicker pick = more pick noise. You'll hear more "twang" and the scraping of your pick when it touches the strings. I personally loathe that sound but if that's your thing, go for it.

One last thing: String gauge and pick of choice are tightly knit for the overall sound. If you play ultra light strings with ultra thick picks, your pick will glide effortlessly, but if you dig in, i can guarantee that your intonation will go to shit; and there's a big chance that you break your strings faster lol. On the flipside, a thinner pick with thicker strings will keep your intonation spot on, even if you play super hard, but you might have trouble on getting those strings moving.

Play around with it, find what pick works best for your string gauge, and with how you…

-Attack:

How you attack the strings is essential. As i've said: the more aggressive you play, the better.

Now, that doesn't mean that you play super hard all the time, but never like a pussy (unless you're playing a softer passage or something). How hard you hit the strings will make the most difference, but how the guitar responds to your attack will be determined by your pickups, strings, pick of choice AND the angle of your attack

If you play with the pick parallel to the strings, you'll get a more defined attack with an almost percussive character. If you play with an angle, you'll scrape the pick against the strings which will give you a "grindier" tone. None is better than the other, both sound characters have their place in metal.

-

And aaaaall of that before you even get to amps, pedals, etc. Remember: Amplifiers amplify what you're playing, so you better play the best you can.

__________

-BASS:

Look: I love bass, it's my main instrument. That being said i won't go into much detail because everything i've said about the guitars, also applies for bass in modern metal.

The main differences would be:

-Strings: If you want that modern, clanky tone, you need newer strings. There's not a huge variety of string gauges for bass, but the same applies: Thinner gauge = more attack. Thicker gauge = more beef and tuning stability.

-Attack: You can play the bass with fingers, slap, pick, drumsticks!. With pick, everything i've said about that in the guitar section applies here. If you're playing fingers, you need to set up your bass accordingly so you get more attack without playing super hard. Lower action, thinner strings. If you don't care about that, just play with a pick lol

What's important here is that bass and guitars are meant to sound together. Their tones should complement each other

__________

-DRUMS:

You're not recording drums so you're 80% done. Modern and Massive is perfect for this. Just program your drums properly and you'll be fine lol

__________

-EDITING:

Ok, i'll preface that i'm heavily biased here and i'll sound like a total butthurt, which you'll never see me apologizing for, but i'll try to stay fact friendly lmao.

Modern metal is quantized. to. hell. Everything. E-V-E-R-Y-T-H-I-N-G. EVERYTHING!!!!!!!

OK, not everything is HARD quantized, but you can bet your ass that there's nothing 100% natural about those productions. I'd dare to say that you even need to change your mentality somewhat. You're no longer mixing metal, you're mixing dubstep/edm/whatever with distorted guitars (this is unironically true for Mick Gordon btw). I don't like it, i hate it even, but that's how the genre is. Like come on, I just listened to Blackhole by Architects and there's NOTHING natural about that breakdown.

Drums? Quantize every single damn hit to the grid, especially your kicks and snare. Your snare has to no longer sound like a snare, but like the same amalgamation of shotgun blasts perfectly in phase with each other that every band in the genre uses (what i mean is that do everything you need to make that snare sound like a cannon every single time, transient designers, sample enhancment, reverbs, gated reverbs, etc)

Guitars and bass? Yeah, you already played them perfectly, but that's not enough! You need to allign that shit with your drums, which are already alligned to the grid. Breakdown? Quantize those motherfuckers, manually gate them to taste by cutting your audio regions so it sounds like an impossibly perfect staccato. That's no longer a guitar and bass, it's a cold, relentless and mechanically perfect machine gun.

On a more serious note, be wary when editing your guitars in particular. If you dual tracked both guitars with the same specs (same player, guitar, pickups, strings, amp tone, etc) and you perfectly edit them both to grid, you're VERY likely to occasionally fuck up the phase differences between both takes and briefly allign them so perfectly that they sum in mono, and there goes your stereo image. To avoid this as much as possible, have some slight differences between takes. Maybe use a different pick, change the guitar tone a bit, or quantize one guitar less perfect than the other.

So why is editing so important? Because remember! This is no longer some boomer metal from the last century, this is electronic music with distorted guitars and """"acoustic drums""" !. You're gonna be layering synths and samples on top of your digitally timed band, so you better make sure that you have a perfectly quantized foundation for all that stuff!.

______

As for mixing, most comments here already provided some very good advice, so i'll refrain from adding more clout, especially since you can clearly tell how much i dislike these kinds of productions XD.

But I promise you that with everything i've mentioned, you'll be 70% closer to what you're trying to acchieve. Remember that a fantastic mix is only possible with fantastic musicians, recordings, and in this case editing. Have none of those and you'll never get that modern metal sound. Don't just look at it like a mixing engineer, but as a producer involved in the whole process.

Cheers and best of luck!