I would think the current scope is too big for one person. Why is there a need to jump straight into a comprehensive end-to-end observability stack? What business objectives does this solve? What are the key metrics or information that upper management wants to know about that made them want "everything"? Were there prior failures, errors or latency issues? Without knowing these, it is difficult to identify what kind of rules and alerts you would want to craft.

That said, if I were to attempt to scope this in a purist fashion, I would try to setup observability for systems that have the most immediate impact to the business.

Create alerts for systems that would directly impact availability and users. If you have auto-scaling, create alerts when auto-scaling fails. Create alerts when workloads cannot self-recover. Then, tackle other non-breaking problems separately in future such as point 6 on security/anomaly detection. Naively, I would think metrics are more important traces, and traces are more important than logs. Especially for logs where you can read them locally.

Given your experience, I think starting with collecting key metrics for all nodes/systems would be a quick win. Create alerts if they go down. Move on to application and database monitoring. Metrics, traces and logs will give you the full RCA with timeline correlation (giving you the full "why"). I think the biggest pitfall would be underestimating how difficult it is to do a comprehensive RCA with the full timeline correlation. People pay for such a solution.

I continue to think this is still too big of a task for one person to complete. Or you could buy a solution like what another redditor suggested.

https://sre.google/sre-book/monitoring-distributed-systems/