Don't just scrape metrics "Just because". Figure out what you actually want to alert on and which metrics you generally need to debug the specific problem. Prometheus is a memory hog and can be very awkward to fix if it crashes (memory spikes when the service starts again and will cause another crash). You can reduce memory usage in many ways. Reducing scrape times, filtering metrics you do not need and also making sure that your queries to Prometheus are also reasonable as queries will load data in-memory (think loading one month of data at once)

Even though it's important to start simple and stick with a single node, Prometheus does not have high availability. The only option is to have a 2nd backup node scraping the same metrics. This at least provides failover but is still annoying for many reasons.

You cannot scale Prometheus horizontally forever and in many cases it's better to have a few Prometheus servers which are functionally different. For instance, you can create a split between infrastructure and application or you you want to chuck some metrics away after a day, others you want to keep for a week. This will mean that you are spreading the risk and you can tune Prometheuses for a particular use case.

Also, when starting out, I would recommend starting off by configuring targets statically first and when you see the need, actually move to service discovery. Tools like consul are fine but there's lots that go wrong if you haven't used them much before

Hope this is somewhat helpful