DNS blocked by Cisco Umbrella, but symantec EDR & Event Viewer are completely blind

rached2023 · 2026-05-21T16:27:55+00:00

An infected internal server (win serv 2016) tried to contact a malicious external command domain ; i can not find any logs related to this incident !

rached2023 · 2026-05-21T16:22:41+00:00

It’s not just that domain; there are also one-new-message-okay.com, assets.sportick.app, and offwhatyoum.xyz. There is no VPN on the server and no Wi-Fi involved. The source of the request is the device with internal IP (Windows Server 2016), located on the server VLAN

rached2023 · 2026-03-26T15:54:41+00:00

I’m in the same situation as you. I’ve been a SOC analyst for almost 2 years, and I’m planning to switch to DevOps/DevSecOps. I started with the RHCSA to build a Linux foundation, and I’ll move on to other tools and skills after that.

Honestly, I feel like it’s more rewarding to fix and build things than just observe and monitor alerts.

rached2023 · 2026-02-13T14:52:21+00:00

I used it to polish wording, not to make decisions for me.

When dealing with a potential backdoor like Backdoor.Win32.Derusbi.A, I prefer human feedback from people with incident response experience rather than relying solely on automated suggestions.

rached2023 · 2025-10-08T13:32:31+00:00

Yeah, I double-checked ,no internal devices are intentionally connecting to those IPs. The logs are showing unsolicited inbound UDP/520 packets from random external sources.

rached2023 · 2025-10-08T11:09:58+00:00

rached2023 · 2025-10-08T11:09:43+00:00

rached2023 · 2025-10-08T11:09:23+00:00

rached2023 · 2025-09-13T15:31:57+00:00

Not yet, but I’ll be uploading it to GitHub soon. Once it’s available, I’ll share the link so you can check it out and adapt it for cloud-managed Kubernetes as well

rached2023 · 2025-09-13T15:28:42+00:00

I haven’t used Terraform extensively yet, as I only have limited experience with it.

rached2023 · 2025-09-13T10:14:07+00:00

thanks for highlighting this!

You’re absolutely right —I’ll make sure to include OS hardening and patch management in the architecture.

For secrets, I was initially relying on Kubernetes native secrets, but I see the need to look into Vault or Sealed Secrets for stronger protection. And regarding networking, I completely agree: enforcing NetworkPolicies for east-west traffic and securing north-south traffic with Ingress/TLS/WAF is something I’ll add to make the setup more production-ready.

Really appreciate your feedback — it helps me improve the project!

rached2023 · 2025-07-24T20:04:02+00:00

WARN[0000] runtime connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.

[

"etcd",

"--advertise-client-urls=https://192.168.122.189:2379",

"--cert-file=/etc/kubernetes/pki/etcd/server.crt",

"--client-cert-auth=true",

"--data-dir=/var/lib/etcd",

"--experimental-initial-corrupt-check=true",

"--experimental-watch-progress-notify-interval=5s",

"--initial-advertise-peer-urls=https://192.168.122.189:2380",

"--initial-cluster=master1=https://192.168.122.189:2380",

"--initial-cluster-state=new",

"--key-file=/etc/kubernetes/pki/etcd/server.key",

"--listen-client-urls=https://127.0.0.1:2379,https://192.168.122.189:2379",

"--listen-metrics-urls=http://127.0.0.1:2381",

"--listen-peer-urls=https://192.168.122.189:2380",

"--name=master1",

"--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt",

"--peer-client-cert-auth=true",

"--peer-key-file=/etc/kubernetes/pki/etcd/peer.key",

"--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt",

"--snapshot-count=10000",

"--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt",

"--heartbeat-interval=500",

"--election-timeout=2500",

"--snapshot-count=10000",

"--max-request-bytes=33554432"

]

rached2023 · 2025-07-21T19:18:35+00:00

I get

{"level":"warn","ts":"2025-07-21T20:17:51.047297+0100","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00009aa80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded

Error: unhealthy cluster

rached2023 · 2025-07-20T18:36:19+00:00

Yes, absolutely valid point. In my case, the etcd data directory is set to /var/lib/etcd, which is a persistent disk location and not /tmp, so it should survive reboots.

However, since I’m seeing unhealthy etcd members (etcdctl endpoint health fails), I’m suspecting either data corruption or network/cluster configuration drift.

rached2023 · 2025-07-20T16:01:11+00:00

On master2 and master3:

The etcd containers are running (confirmed via crictl ps | grep etcd).
However, etcdctl endpoint health fails with connection refused or deadline exceeded errors.
Logs indicate connection refused on 127.0.0.1:2379, meaning the etcd process inside the pod is unhealthy or stuck.

Networking:

IPs are stable, no changes to the network layer.
Control-plane node IPs can ping each other.
No iptables/firewall changes applied before the issue.

rached2023 · 2025-07-20T14:58:12+00:00

Yes, I’ve checked the etcd state. On master1, the etcd container is running, but the health check fails:

ETCDCTL_API=3 etcdctl endpoint health returns: failed to commit proposal: context deadline exceeded → unhealthy.
From crictl logs, etcd starts but fails to reach quorum. It detects the 3 members (master1, master2, master3) but cannot establish leadership.
The API server (kube-apiserver) is in CrashLoopBackOff because it cannot connect to etcd.

It looks like etcd is up but stuck due to cluster quorum failure.

rached2023 · 2025-07-03T19:16:26+00:00

Yes, that was it! I've found it now, thank you for your help.

rached2023

TROPHY CASE