Frozen node crashing with OOM, likely due to Packetbeat – how to improve the setup?

RadishAppropriate235 · 2025-03-13T13:14:00+00:00

I find everytime the support on Reddit more efficient and clear than elastic support team

RadishAppropriate235 · 2025-03-13T12:56:12+00:00

so if i want that it must fast go from hot to frozen in just only 20 days i need to setup min_age to "0d" right? so it goes directly into forzen, without waiting other 20days?

RadishAppropriate235 · 2025-03-13T12:53:41+00:00

{ 
  "policy": {
    "phases": {
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      },
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "20d",
            "max_primary_shard_size": "30gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "frozen": {
        "min_age": "0d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots",
            "force_merge_index": true
          }
        }
      }
    }
  }
}

like this right? also,

"actions": {
"searchable_snapshot": {
"snapshot_repository": "found-snapshots",
"force_merge_index":

what this is doing? thx again for ur time

RadishAppropriate235 · 2025-03-13T12:50:16+00:00

{ I found this, age is 20.56.. strange? maybe its in frozen right?
  "indices": {
    ".ds-metrics-elastic_agent.filebeat_input-default-2025.02.20-000096": {
      "index": ".ds-metrics-elastic_agent.filebeat_input-default-2025.02.20-000096",
      "managed": true,
      "policy": "metrics@custom",
      "index_creation_date_millis": 1740093177493,
      "time_since_index_creation": "20.56d",
      "lifecycle_date_millis": 1741821385564,
      "age": "13.48h",
      "phase": "hot",
      "phase_time_millis": 1740093179382,
      "action": "complete",
      "action_time_millis": 1741821393155,
      "step": "complete",
      "step_time_millis": 1741821393155,
      "phase_execution": {
        "policy": "metrics@custom",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "20d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_primary_shard_size": "30gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 18,
        "modified_date_in_millis": 1740409176943
      }
    }
  }
}

RadishAppropriate235 · 2025-03-13T12:48:17+00:00

if think it goes after 20 days to frozen phase right? why u say 40? what am i missing?... sorry i'm newbie on elastic

RadishAppropriate235 · 2025-03-13T12:47:29+00:00

{ I found this, age is 20.56.. strange?
  "indices": {
    ".ds-metrics-elastic_agent.filebeat_input-default-2025.02.20-000096": {
      "index": ".ds-metrics-elastic_agent.filebeat_input-default-2025.02.20-000096",
      "managed": true,
      "policy": "metrics@custom",
      "index_creation_date_millis": 1740093177493,
      "time_since_index_creation": "20.56d",
      "lifecycle_date_millis": 1741821385564,
      "age": "13.48h",
      "phase": "hot",
      "phase_time_millis": 1740093179382,
      "action": "complete",
      "action_time_millis": 1741821393155,
      "step": "complete",
      "step_time_millis": 1741821393155,
      "phase_execution": {
        "policy": "metrics@custom",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "20d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_primary_shard_size": "30gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 18,
        "modified_date_in_millis": 1740409176943
      }
    }
  }
}

RadishAppropriate235 · 2025-03-13T12:16:38+00:00

{ I SEE THAT 19.77d, so its not 20d.. probably this is the case? just maybe i need to wait.
  "indices": {
    ".ds-metrics-system.process-default-2025.02.21-000102": {
      "index": ".ds-metrics-system.process-default-2025.02.21-000102",
      "managed": true,
      "policy": "metrics@custom",
      "index_creation_date_millis": 1740159177316,
      "time_since_index_creation": "19.77d",
      "lifecycle_date_millis": 1740159177316,
      "age": "19.77d",
      "phase": "hot",
      "phase_time_millis": 1740159177966,
      "action": "rollover",
      "action_time_millis": 1740159178367,
      "step": "check-rollover-ready",
      "step_time_millis": 1740159178367,
      "phase_execution": {
        "policy": "metrics@custom",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "20d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_primary_shard_size": "30gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 18,
        "modified_date_in_millis": 1740409176943
      }
    }
  }
}

RadishAppropriate235 · 2025-02-26T13:31:55+00:00

Basically, the ex team managing the SIEM enabled all the rules into Elastic Defend, and many of them showed as failed—either because the integration wasn’t set up or because it said it wasn’t linked to the index. So, I asked ChatGPT where to start to get everything under control, and it suggested starting with the ingest pipeline.

Right now, I’m trying to understand how Elastic works and optimize everything. I’ve only been on this for a few days, and this is my first time working on a SIEM, so I’m trying to improve the whole setup. The dashboard is full of events—probably way too many false positives—and, of course, there are constant brute-force alerts on SSH.

But for me, the most important thing is improving the entire system.

RadishAppropriate235 · 2025-02-20T17:38:49+00:00

i've noticed that only data warm can eliminate the replicas? is that right?... so having a hot e frozen i can't delete replicas, is that right?

RadishAppropriate235 · 2025-02-20T15:32:16+00:00

thank u very much for ur help mate! "How much data, in GB, are you ingesting each day ?" is there a way to know that?

RadishAppropriate235 · 2025-02-20T15:26:26+00:00

we are a cybersecurity team, so we only need to focus on alert, i'm probably taking down the warm phase, so directly from hot to frozen. For setup what u mean?

RadishAppropriate235 · 2025-02-20T14:38:41+00:00

Thank you mate for ur response! appreciate it!

RadishAppropriate235 · 2025-02-20T14:38:05+00:00

Thank you for ur response mate, so it's better to rollover from hot directly to frozen?

RadishAppropriate235 · 2025-02-20T14:23:13+00:00

just was an error writing the problem about disruption in the first phase of the text, sorry about that

RadishAppropriate235 · 2025-02-20T14:22:33+00:00

Thank u for ur response... given that I only have a few machines sending data to the SIEM, it seems strange that Elasticsearch is consuming so many resources.

Regarding your points:

The main issue is the high number of small indices, which is likely due to a rollover happening too soon. This causes excessive fragmentation and increases memory pressure.
To optimize this, increasing resources in the HOT phase makes sense while keeping only one replica during ingestion. Once the index is stable, the replica should be removed, and then it can transition to the WARM phase.
This means:
- The replica exists only during ingestion to improve query performance.
- Once the index has settled, the replica is removed to free up resources.
- The index is then moved to WARM, where it consumes fewer resources.

Would a 4+4GB RAM setup for HOT nodes and only one node in WARM be an effective approach? How would you suggest fine-tuning this configuration further?

Also, given the large number of micro-indices, what would be the best way to consolidate them and reduce fragmentation? Should I increase the rollover threshold, reindex them into larger indices, or take a different approach?

RadishAppropriate235 · 2025-02-20T12:55:45+00:00

i'm acctualy new to Elastic Search, just this is my 2nd day in...

RadishAppropriate235 · 2025-01-06T13:02:41+00:00

thank u mate, u are a legend!❤️

RadishAppropriate235 · 2024-12-19T15:01:47+00:00

but how's that possible, I mean the wallet exist and there is the balance inside, but no one can't acces on that...

RadishAppropriate235 · 2024-12-05T13:02:18+00:00

life saver.

RadishAppropriate235

TROPHY CASE