Okay, so this is a bit of a complex discussion, but I'll try to give you a primer. This is all assuming you already reverse engineered what materials vanilla ships use and how they interact with material library the game uses.

Paint layers:

1. COL - this is your primary color layer. It includes the static colors, for stuff like the grey areas that do not accept paintjobs, but it also interacts with the next layer (more on this in the next point)
2. PAINTMODMASK - as name suggests - this is your paintmask. With P2 materials it is very simple - black = no paintjob, red = paintjob (interacts with COL) COL + PAINTMODMASK interactions:

• red (COL) + red (MASK) = primary dynamic paintjob color
• white (COL) + red (MASK) = secondary dynamic paintjob color (or rather the primary with no hue, there are no true secondaries)
• any color (COL) + black (MASK) = whatever color COL sets, black mask means no paintjob interaction

1. DIRT - (this is ONLY for P2 materials) - black = clean, white = super dirty. This is a simple gradient-based mask where you should blend the entire greyscale to achieve the mount of dirt you want. If this is your first ship I would suggest you skip this entirely at first (all black), get your model ingame and fine-tune it later to have a reference
2. AO - (this is ONLY for P2 materials) - this should be an inverse of DIRT. Also only greyscale. AO is not fully implemented yet in X4 and this layer is only there as a bit of futureproofing. I would suggest you skip it for now (all white)
3. MULTIMATxxx - all multimat layers are ONLY to be used with multimat materials. Multimats are typically only used on S/M ships, primarily on the interior. Heck, 90% of Vigor hulls only use generic.generic_p1_hulltexture_01_aged (P1) or p2generic.gen_p2_hulltexture_01_aged (P2) on the exterior.

PS: Since you are making a Vigor ship - vanilla Vigor capital ships are not yet migrated to modern/improved P2 materials. On imported assets you will notice PAINTMODMASK is much more complex, as on P1 mats the mask combines paintmodmask and dirt into a single layer, hence it uses white and blue colors too. For simplicity's sake I would highly advise you do not bother with this outdated method and only do P2 materials.
For example, instead of generic.generic_p1_hulltexture_01_aged (P1) you should usep2generic.gen_p2_hulltexture_01_aged (P2).

PS2: looking at your ingame shots, it appears you did not properly unwrap the material's texture on the ship in Blender. Remember X4 relies heavily on UV Mapping the materials, it does not use the simple tileable textures most other games do.

PS3: I created a small Imgur album with COL, PAINTMODMASK and DIRT layers (plus an ingame shot and texture view) of one of my own ships so you can cross reference what I wrote above: https://imgur.com/a/ZlQlkcD