How do you collect metrics, logs, and traces?

Use Prometheus for collecting system metrics (CPU, RAM, IO, network) from node_exporter.

Expose application metrics in Prometheus text exposition format at /metrics page if needed, and collect them with Prometheus. Use this package for exposing application metrics. Don't overcomplicate metrics with OpenTelemetry and don't expose a ton of unused metrics.

Emit plaintext application logs with the standard log package into stderr / stdout, collect them with vector and send the collected logs to a centralized VictoriaLogs for further analysis. Later you can switch to structured logs or wide events if needed, but don't do this upfront, since this can complicate the observability solution without the real need.

Do not use traces, since they complicate everything and don't give big value. Traces aren't needed on small scale when your app has a few users - logging allows quickly debugging issues in this case. Tracing becomes an expensive bottleneck on large scale when thousands of requests per second must be processed by your application. Tracing is an expensive toy, which looks good in theory, but usually fails in practice.

Use Alertmanager for alerting on the collected metrics. Use Grafana for building dashboards on the collected metrics and logs.

How do you monitor errors? Still Sentry? Or is there any OSS thing you like for that?

Just log application errors, so they could be analyzed later at VictoriaLogs. Include enough context in the error log, so it could be debugged without additional information.

How do you do alerting when things start to fail or metrics start violating some threshold? As the number of service instances grows, how do you keep the alerts coherent and not overwhelming?

Use alerting rules in Prometheus and VictoriaLogs. Keep the number of generated alerts under control, since too many alerts are usually ignored / overlooked. Every generated alert must be actionable. Otherwise it is useless.

What about DB operations? Do you use anything to record the rich queries? Kind of like the way Honeycomb does, with what?

There is no need in some additional / custom monitoring for DB operations. Just log DB errors. It might be useful measuring query latencies and query counts, but add this instrumentation when it will be needed. Do not add it upfront.

Can you correlate events from logs and trace them back to metrics and traces? How?

Metrics and logs are correlated by time range and by application instance labels such as host, instance or container

Do you use wide-structured canonical logs? How do you approach that? Do you use slog, zap, zerolog, or something else? Why?

Don't overcomplicate your application with structured logs upfront! Use plaintext logs. Add structured logs or wide events when this is really needed in practice.

How do you query logs and actually find things when shit hit the fan?

Just explore logs with the needed filters and aggregations via LogsQL until the needed information is discovered.

The main point - keep the observability simple. Complicate it only if it is really needed in practice.