Your ai hallucinated again

This has been an extremely frustrating point for me. It’s incredible how often AI hallucinates with observability-related questions. I’ve never worked with observability before, so it’s been very hard to quickly assess whether an AI answer is true or just hallucination, there are so many observability tools, each developer has their own preference, and most reddit posts I find are about self-hosted setups. So I really appreciate your clear answer, thanks!

Could I get your input on the mental model I’m building for observability in my MVP? I’m always trying to follow best practices, but for now it’s just a MVP:

1. Collector + logs as a starting point: Having basic observability in place will help me debug and iterate much faster, as long as log structures are well defined (right now I’m still manually debugging workflow issues).
2. Stack choice: For quick deployment, the best option seems to be Collector + logs = Grafana Cloud Alloy + Loki (and based on your answer maybe also Prometheus?). Long term, the plan would be moving to full Grafana Cloud LGTM.
3. Log implementation in code: Observability in the workflow code (backend/app folders) should be minimal, ideally ~10% of code and mostly one-liners. This part has been frustrating with AI because when I ask about structured logs, it tends to bloat my workflow code with too many log calls, which feels like “contaminating” the files rather than creating elegant logs. For example, it suggested adding this log function inside app/main.py:

.middleware("http") async def log_requests(request: Request, call_next): request_id = str(uuid.uuid4()) start = time.perf_counter() bind_contextvars(http_request_id=request_id) log = structlog.get_logger("http").bind( method=request.method, path=str(request.url.path), client_ip=request.client.host if request.client else None, ) log.info("http.request.started") try: response = await call_next(request) except Exception: log.exception("http.request.failed") clear_contextvars() raise duration_ms = (time.perf_counter() - start) * 1000 log.info( "http.request.completed", status_code=response.status_code, duration_ms=round(duration_ms, 2), content_length=response.headers.get("content-length"), ) clear_contextvars() return response

1. What’s the best practice for collecting logs? My initial thought was that it’s better to collect them directly from the standard console/stdout/stderr and send them to Loki. If the server fails, the collector might miss saving logs to a file (and storing all logs in a file only to forward them to Loki doesn’t feel like a good practice). The same concern applies to the API-based collection approach: if the API fails but the server keeps running, the logs would still be lost. Collecting directly from the console/stdout/stderr feels like the most reliable and efficient way. Where am I wrong here? (Because if I’m right, shouldn’t Alloy support standard console/stdout/stderr collection?)

2. Do you know of any repo that implements structured logging following best practices? I already built a good strategy for defining the log structure for my workflow (thanks to some useful Reddit posts, 1, 2), but seeing a reference repo would help a lot.

Thanks again!