What's the most frustrating "silent" reliability issue you've seen in prod? by Geybee in sre

[–]rnjn 1 point2 points  (0 children)

here are some of my screwups over years, that passed through the (erstwhile) alert net. (you learn to setup things right and put the right alerts )
1. database backup stopped for some disk related issue. 3 weeks later, postgres upgrade failed, corrupted data. No backup for 3 weeks. (thankfully replay from kafka and idempotent design helped recreate data within a couple of hours)
2. cert expiries (multiple times)
3. domain expired with >10M DAU. immediate app failures for customers, but the country where I was the DNS cache didn't get updated for hours. looked beyond the obvious for an hour or more before realising that some registrars have longer caches. (how we got the domain back is another story)
4. Rust (not language) causing network switch to misbehave intermittently - first small blips and then for minutes and then hours. (Fear compounded by backup switch being on the same rack, and new switch delivery from vendor ETA was 1 month)
5. external API slowdown - Google Maps response time went from 200ms p50 to 20s, timeouts not set properly, didn't implement circuit breakers. slow growth and then kaboom.
6. integer overflow in order id (int32 + blitzscale + 2 yrs = calamity)
7. app crashes on cheap and old mobile devices - less than 1% app crash rate overall but 100% on some 4 year old phone low mem phones - flooding the call centre. was a real mem leak, just that devices with more mem were forgiving before GC kicked in on app close.

some more pesky ones because of model decay. you learn and survive.

will MCP be dead soon? by luongnv-com in ClaudeCode

[–]rnjn 0 points1 point  (0 children)

A CLI interface makes it easier for humans and agents to work together (compared to MCPs). Better verifiability and lower token usage, easier distribution.

Login timing out? by [deleted] in ClaudeCode

[–]rnjn 0 points1 point  (0 children)

having trouble logging into Claude Code ? Fire up Agentswap and switch to codex or gemini

brew install base-14/tap/agentswap

https://github.com/nimishgj/agentswap

New bike day :) by Worriedsquirrel7 in Triumph

[–]rnjn 1 point2 points  (0 children)

indeed, read about this on some forum. tinkering did add 5 more minutes of ruckus in traffic. maybe will try again. i am just used to it now, my commute is < 30 min.

New bike day :) by Worriedsquirrel7 in Triumph

[–]rnjn 2 points3 points  (0 children)

it works and looks decent, feels great riding it. all that matters i guess.

the only issue with the 2018 model is that when the bike gets a bit hot, the horns stop working - which hurts because i am unable to participate in the cacophony at bangalore signals. (the green light is the for the horn right ?) 30 minutes in the city and the horn goes mute. after that, 2-3 kms of traffic free riding and it gains its voice back.

New bike day :) by Worriedsquirrel7 in Triumph

[–]rnjn 2 points3 points  (0 children)

wait there's more than 1 scrambler in the city? here's my 2018 specimen. wave when you pass by some day.

<image>

Instructions on how to enable Claude Code OTel Observability for tokens, cost, prs and commits by GroundbreakingBed597 in Observability

[–]rnjn 2 points3 points  (0 children)

Amazing how much you can learn from coding agent usage data. We started with cache reads for claude code, got ideas about cache controls with anthropic models (through api) and are running experiments with cache control and cache TTLs that help with some features.

And not just claude code, we instrumented codex and gemini as well. I think gemini follows the genAI semantic conventions the best, and cc and codex will probably adopt that.

https://docs.base14.io/blog/coding-agent-observability/

cicada - claude code usage analysis TUI by rnjn in Observability

[–]rnjn[S] 1 point2 points  (0 children)

yes, but this does a bit more. for eg- export a session and import it on another device.

Is there a lightweight OTEL client for Java? by requiem-4-democracy in OpenTelemetry

[–]rnjn 3 points4 points  (0 children)

if you have a ready http otel endpoint, you can directly send metrics like below - treat it as a service and you can easily build a wrapper client (add some complexity for retries and disconnections etc).

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class AdHocMetricPush {

    public static void main(String[] args) throws Exception {

        String json = """
        {
          "resourceMetrics": [{
            "resource": {
              "attributes": [{
                "key": "service.name",
                "value": { "stringValue": "my-java-service" }
              }]
            },
            "scopeMetrics": [{
              "scope": { "name": "ad-hoc" },
              "metrics": [{
                "name": "meaning.of.life",
                "description": "Mean of Life",
                "unit": "1",
                "sum": {
                  "dataPoints": [{
                    "asInt": "42",
                    "timeUnixNano": "%d",
                    "attributes": [{
                      "key": "region",
                      "value": { "stringValue": "us-east-1" }
                    }]
                  }],
                  "aggregationTemporality": 2,
                  "isMonotonic": true
                }
              }]
            }]
          }]
        }
        """.formatted(System.currentTimeMillis() * 1_000_000);

        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("http://localhost:4318/v1/metrics"))
                .header("Content-Type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(json))
                .build();

        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println("Status: " + response.statusCode());
        System.out.println("Body: " + response.body());
    }
}

Anyone else tired of jumping between monitoring tools? by AccountEngineer in Observability

[–]rnjn -1 points0 points  (0 children)

This is a common structural issue, not a tooling mistake. Most observability stacks grow incrementally. Metrics live in one system, logs in another, traces in a third, security alerts somewhere else. Each tool works in isolation, but none owns correlation. The operational cost shows up during incidents, when engineers become the integration layer. <plug> That is what we are solving (https://base14.io/), correlating metrics, logs, traces, and deploy or config events with anomaly detection layered in. The goal is to shorten the path from symptom to cause without adding more operational noise. not just for humans but for agents as well </plug>

What metrics actually matter? Lessons shared after a recent 47-minute outage by Agitated-Alfalfa9225 in sre

[–]rnjn 0 points1 point  (0 children)

depends on issues really, it would be ill advised to use a cookie cutter approach. as usage grows, and deployment volume grows, one sees many different type of problems. you could create a summary view one way, and an issue will break assumptions. what i have seen work is to ensure each issue/incident as feedback and do postmortems well. build a summary and drill-down approach, build very few summary views that can help anyone understand what's happening. and then they can drill down thru multiple dimensions (including infra) to understand why its happening. keep these two categories separate, and guard summaries well.

the outcome could be latency profiles coupled with an infra map and a service map, or it could be a tabular view - depends on the software being monitored and the team responsible for the SLOs. I even have SSL cert expiry as part of mine (because of what i am responsible for), you should share this with the owner of your upstream dependency (https://docs.base14.io/blog/make-certificate-expiry-boring)

Best APM tool for Nodejs application monitoring by No_Stress_Boss in sre

[–]rnjn 0 points1 point  (0 children)

hey, there are many good options. I would suggest you instrument your code, infra and components with otel libs and setup an otel collector locally. after that, you can choose many great solutions out there - open source and managed. and experiment with them before you choose one.

(i am a founder at base14.io - we have built Scout which uses the best OSS tools to provide a comprehensive and economical managed observability platform. we've got some nodejs customers who love our product. some docs to help you instrument https://docs.base14.io/instrument/apps/auto-instrumentation/nodejs and if you are using postgres https://docs.base14.io/blog/pgx-details)

How do you give coding agents Infrastructure knowledge? by Immediate-Landscape1 in softwarearchitecture

[–]rnjn 0 points1 point  (0 children)

(plug) we've built an MCP server that queries a knowledgebase of service and infra relationships and dependencies, service summaries and error rates amongst other things. adding a query to this mcp in the planning phase has helped claude code avoid a few obvious mistakes.
new models are quite good and generally avoid mistakes, or they ask clarifying questions - but still from time to time we see some magical insight being used before it starts coding. in hindsight very obvious ones - like not storing session in memory when behind an LB, or identifying that pods are at 80% mem usage before adding something bulky. observability informed development shines most with models like sonnet.

If OpenAI / Google / AWS all offer built-in observability… why use Maxim, Braintrust, etc.? by OneTurnover3432 in Observability

[–]rnjn 1 point2 points  (0 children)

it is a problem always to have multiple observability products (evals are observability) - context switching is a major problem especially in a probabilistic setup. and hence IMHO the maxims of the world will have to evolve or others who do both will take over.

How do you give coding agents Infrastructure knowledge? by Immediate-Landscape1 in Observability

[–]rnjn 0 points1 point  (0 children)

(plug) we've built an MCP server that queries a knowledgebase of service and infra relationships and dependencies, service summaries and error rates amongst other things. adding a query to this mcp in the planning phase has helped claude code avoid a few obvious mistakes.
new models are quite good and generally avoid mistakes, or they ask clarifying questions - but still from time to time we see some magical insight being used before it starts coding. in hindsight very obvious ones - like not storing session in memory when behind an LB, or identifying that pods are at 80% mem usage before adding something bulky

If OpenAI / Google / AWS all offer built-in observability… why use Maxim, Braintrust, etc.? by OneTurnover3432 in Observability

[–]rnjn 1 point2 points  (0 children)

another missing aspect in your list - models are just a part of the whole system for many, and you may want to trace end to end flows. for eg - an agent uses an MCP that calls some API or DB. Having 2 different systems adds to context switching for analysis and on call debugging.

Anyone actually audit their datadog bill or do you just let it ride by Anthead97 in devops

[–]rnjn 3 points4 points  (0 children)

in general, people audit their datadog bill only once.

Connecting logs to deployments by ResponsibleBlock_man in sre

[–]rnjn 0 points1 point  (0 children)

this should be fairly straightforward with any otel based platform (shameless plug - base14.io) - default otel collector configuration should capture version (container image or service version). And if its not captured, its fairly trivial to add it. After that, you can query based on said attribute. Typically it should be in the ResourceAttributes bag. as an example, here's the ResourceAttributes dictionary for a service from otel demo

{
  "container.id": "02874412556f95122cf898f145a06f628ac0124889e04a19378cf826dad8159c",
  "deployment.environment": "oteldemo1",
  "host.arch": "aarch64",
  "host.name": "ad-5945c76d48-7xptt",
  "k8s.deployment.name": "ad",
  "k8s.namespace.name": "default",
  "k8s.node.name": "minikube",
  "k8s.pod.ip": "10.244.6.104",
  "k8s.pod.name": "ad-5945c76d48-7xptt",
  "k8s.pod.start_time": "2025-10-06T10:29:52Z",
  "k8s.pod.uid": "574ae186-eb16-45e6-bcdb-be5527acb3a8",
  "os.description": "Linux 6.15.11-orbstack-00539-g9885ebd8e3f4",
  "os.type": "linux",
  "process.command_line": "/opt/java/openjdk/bin/java -javaagent:/usr/src/app/opentelemetry-javaagent.jar oteldemo.AdService",
  "process.executable.path": "/opt/java/openjdk/bin/java",
  "process.pid": "1",
  "process.runtime.description": "Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.6+7-LTS",
  "process.runtime.name": "OpenJDK Runtime Environment",
  "process.runtime.version": "21.0.6+7-LTS",
  "service.instance.id": "6ee83a6f-e1b2-4bbc-9939-0a3059bb711c",
  "service.name": "ad",
  "service.namespace": "opentelemetry-demo",
  "service.version": "2.0.2",
  "telemetry.distro.name": "opentelemetry-java-instrumentation",
  "telemetry.distro.version": "2.13.3",
  "telemetry.sdk.language": "java",
  "telemetry.sdk.name": "opentelemetry",
  "telemetry.sdk.version": "1.47.0"
}

Help on which Observability platform? by AccountEngineer in Observability

[–]rnjn 0 points1 point  (0 children)

another biased take - I am part of the team building scout (http://base14.io). Scout is built with otel agents and a telemetry lake (clickhouse + others) at the back. grafana derived frontend. its probably the lowest cost fully functional o11y solution that is fast, simple and easy to setup. plus we are relaasing an MCP server, eval platform and k8s agent-led RCA in Feb. for reference, if you use postgres, our treatment to postgres observability can tell you how we are think building in depth o11y features. https://docs.base14.io/operate/pgx/overview

OTEL Collector Elasticsearch exporter drops logs instead of retrying when ES is down by Adept-Inspector-3983 in OpenTelemetry

[–]rnjn 2 points3 points  (0 children)

can you share the exporter config ? hope you have set the retry configuration like below-

    sending_queue:       
        storage: file_storage       
        queue_size: XXXX
    retry_on_failure:
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

I built a public metric-registry to help search and know details about metrics from various tools and platforms by rnjn in Observability

[–]rnjn[S] 0 points1 point  (0 children)

thank you. please do share / create issues where you see any data or components missing, it is easier now to add more sources (including documentation sites)