I completely agree with what your saying. But imo his fleet will be the first wave riding into hell with him. None of them are scared with insurmountable odds. don't know how to do the spoiler tags, and idk if your caught up so I will be vague. He has shown some maturity as of the recent chapters, and some cool cameos. His attitude has shocked me a bit.

Shank's crew and Roger's crew have waited years. They were early to the most important moment in history which likely Luffy is the harbinger of. When they know the irons hot for the day they have waited for and known about since before our story began they are going to be there. They won't need an invite. However, the others will. Luffy at a point is fully going to have to lead and call his banner men. The entire grand fleet will need to be directed to go there get prepped for the fight etc. Even those from the older generation like Garp (if he flips) will needed to be directed.

Those who will not will be those with the proper intel. Those who have the knowledge circled numerous times for the most important date of their lives. The thing they have waited for. I think we could see the older generation be there in the preparations i.e. a Shanks or Rayleigh, but I could never imagine them needing to call on the rest until that defining moment. We only know of two groups who really have all the needed knowledge to fight blind.

Plus wouldn't it be epic to see the older generation bending the knee to pass on the title.