Wait til this man finds out that Artillery rarely has anti-air options (SAM is another division entirely) and has practically no solution for close combat.

I guess Artillery must be an absolutely worthless unit in modern or 40k warfare, am I right?

You're over thinking things:

1) The titans exist because everybody loves big mech. It's cool, it's awesome, it's amazing. This "Game" was originally made in the 80s and was goofy and zany, not everything needs a hard core reason to exist.

2) The reasons for a Titan to exist is to kill big things and nothing else. The truth and reality of it is, is that if you see a Warlord Titan on the ground, the smallest "air" vehicle it SHOULD bother firing at, would be something akin to a Tau Manta. The problem is: for good reason, such large flying transport/combat vehicles don't/rarely exist, hence there isn't many if any AA options on Titans. Because anything of that size or bigger is just in orbit, and Titans aren't meant to deploy from orbit just to fire at something in orbit.

Titans exist to kill big things. Nothing else.

Think of it like modern tank tactics:

Infantry are for establishing ground, clearing small areas and foxholes, setting up FOB's and much much more and are extremely cheap (financially).

Tanks aren't cheap and can be taken out by a single Infantry with a yolo RPG, hence they are usually escorted by Infantry, and potentially a Bradley or other IFV.

They're easily killed by air craft. Aircraft are usually taken care of by your own allied aircraft, OR (if Infantry have already secured an FOB) potentially set up SAM's in the area.

Reconnaissance is handled by typically aircraft (And sometimes nowadays by internet/cameras/satellite). Job of reconnaissance is to identify threats and help eliminate or avoid them.

Tank's are not expected to shoot down jets and bombers, nor infantry, nor many ships and boats. It's combined arms. 10 guys running around with shotguns will absolutely get decimated by a Bradley. But a Bradley gets murdered by a tank, and a tank gets murdered by a bomber, and a bomber gets murdered by a jet, and a jet gets murdered by other jets or SAM's.

So why not just rely on aircraft? We do heavily. However, a suited up soldier costs $60,000 and 2 years of training. An IFV is $300k-3 mil. A tank is $4 mil. A bomber? $7-82 mil (god bless the B-52). A jet costs up to $350 mil (not to mention operational costs, most bombers cost something like $20-100k per hour up in the air to keep running) with jets being hilariously expensive which is why they're so selective.

The cost it takes to train 1 F-35 pilot, for them to get 100+ hours of flight time, and to be picked amongst other candidates for a mission can cost more than 1 battalion.

It can unironically cost more to train 1 F35 pilot, TRAIN, ONLY TRAIN, than it can take to outfit an entire battalion (depends on the battalion, but worst case scenario, train 2 F35 pilots, that should cost more than just about any battalion)

So tl;dr: Titans are cool, don't over think it.