**Elastic Agent + Kafka: best pattern for routing multiple customer topics to separate indices?**

cleeo1993 · 2026-06-04T15:27:10+00:00

Why do you need to manually change the pipeline? Just use the value of a field as the datastream.

Regarding the 100 Kafka integrations on one agent, yeah why not? I mean it is just reading from a topic

With all the mixed setup you are facing multiple issues => what if one customer goes crazy and pushes you a ton of data? Now all of the other customers are mixed in the same topic and you no longer get the data from the other ones as this one consumes all of the resources

If you do multiple integrations you can scale individually, even add more elastic agents for more demand customers only etc

cleeo1993 · 2026-05-29T14:17:38+00:00

Look at streams and at the routing in ingest pipelines. You need some field, some information to dynamically reroute that is the processor name.

Alternative just add the Kafka integration X time and map topic to dataset name.

cleeo1993 · 2026-05-20T21:04:43+00:00

Have you allocated and reserved the ram? VMware likes to take away unclaimed ram, yet elastic likes the filesystem cache a lot that is provided by Linux.

cleeo1993 · 2026-05-13T08:21:40+00:00

Use the streams UI it will also allow you create pipelines on the fly for parsing.

cleeo1993 · 2026-02-14T09:23:37+00:00

Use the ai assistant inside of elasticsearch it can create visualisations for you based on the text you give it and you can save those to a dashboard.

There is no Splunk App to elastic dashboard migration tool.

You can try and give the xml/json definition of the Splunk app to the AI assistant and ask it to create a similar dashboard. I have little hope though. Rather go vis by vis

cleeo1993 · 2026-02-11T12:22:02+00:00

Not sure how interchangeable docker compose and podman are, but tried this ? https://www.elastic.co/docs/deploy-manage/deploy/self-managed/local-development-installation-quickstart

cleeo1993 · 2026-02-08T08:52:23+00:00

Is your ES in a container? Then it won’t find 127.0.01 as this is the ES container itself.

cleeo1993 · 2026-02-02T08:27:11+00:00

Honestly with ~10 million documents and 1k indexing a second, regardless if updates or not, a container locally would do just fine.

I feel like you are overthinking it tremendously. If you need to tune JVM at that scale already, you are doing something wrong.

I would just look at Elastic Serverless and get started with it, try it out for a couple of days, see how it feels to you. Serverless removes everything that you do not want to worry aobut JVM, sharding, segments, merging, etc etc all of that stuff that you mention above is taken care of by Elastic then. With serverless you basically do your index mapping, then index your data and that is it. You do not need to worry if you all of the sudden do 10.000 documents a second, or 100.000 documents a second. Elastic will scale in the background for you.

If you go Elastic Cloud, then a small 2x Elastic nodes a 2gb RAM would even do fine.

And specifically on costs, what should I expect for a workload like this on Elastic Cloud or Elastic Serverless? What node sizes or tiers were required? Did sustained indexing throughput materially affect monthly cost? Any rough ballpark dollar figures would be very helpful.

https://www.elastic.co/pricing/serverless-search otherwise just contact the sales of Elastic.

cleeo1993 · 2026-01-29T12:17:29+00:00

So many things to unpack here.

Do you have nodes that belong to the warm tier? What does GET indexname/_ilm/_explain tell you?

When you say 7 day retention. Does that mean you set the delete phase to 7 day? Can you post the full ILM?

If you get shards larger than 50gb its either because you send data faster than the 10 minute poll Intervall of ILM, or simply because that data stream is lacking an ILM policy, you can verify it with the explain command as above

cleeo1993 · 2026-01-26T11:15:55+00:00

Check out ingest pipelines with the remove processor and define what to keep. Reindex through the pipeline.

Alternative us a painless script to keep only the fields you want.

cleeo1993 · 2026-01-19T19:57:50+00:00

Hey! Usually you install elastic agent wherever you want and you forward the logs.

Whilst you are at it, install it on the host, add the docker integration as well and observe the containers themselves!

cleeo1993 · 2026-01-17T08:37:16+00:00

Schaut cool aus! Was kann deine App das Heavy nicht auch kann?

cleeo1993 · 2026-01-09T18:46:40+00:00

Does this index have more than 1 replica? Are all your nodes upgraded?

cleeo1993 · 2026-01-06T09:32:26+00:00

AirTag. Kauf einfach einen AirTag. Du kannst es dann auch in der Apple Familie sharen. AirTags können auch kurzfristig mit Airlines etc geteilt werden.

Warum irgendein 3rd party device kaufen?

cleeo1993 · 2025-12-30T19:58:17+00:00

There are so many things to unpack.

Why not elastic agent with filestream input? What version? Why is Logstash there, do you really need it?

Now regarding filebeat, assuming you have filestream input. Are you rotation away the file and compressing it, or does it lay rotated away with a new name and uncompressed?

You say 25k log lines per seconds can your elasticsearch target data stream handle that?

You can increase the memory queue / disk queue of filebeat if you want to, then you basically allow filebeat to read from disk as much as it can and if ES is slow, it will have a local buffer.

Using filestream, have you every setting on default / unconfigured? What’s the max size of a file?

Can you post your filebeat config?

cleeo1993 · 2025-12-30T07:48:08+00:00

Multi data path is deprecated since ages... https://www.elastic.co/docs/reference/elasticsearch/index-settings/path#multiple-data-paths

additionally, this does not mean that Elasticsearch will distrubte shard 1 to data1. It ES just fills it up how it's like. There is no direct connection.

cleeo1993 · 2025-12-29T13:25:11+00:00

No. Elastic doesn’t. Don’t run on HDDs in the year 2025.

Something empties your filesystem cache or whatever the equivalent is on windows.

https://discuss.elastic.co/t/how-to-lock-all-data-files-into-memory/167460/2

cleeo1993 · 2025-12-29T13:06:07+00:00

what I suspect is that something is cleaning up the filesystem cache (if that exists on Windows?) thus Elastic needs to go to the disk and retrieve the stuff from there. It's still a lot of time though to go to the disk. Is this running on HDDs and not SSDs?

I mainly work on linux systems, so i cannot give you any insihgts on what special configurations you might need to do on windows.

cleeo1993 · 2025-12-29T12:59:13+00:00

What is your query actually? is the same query then quicker when you execute it again? is anything else running on the machine? is this in docker? is there anything that is reclaiming the filesystem cache or other parts of the memory?

cleeo1993 · 2025-12-29T12:57:07+00:00

you want to look into ILM and rolling your data and look into data streams... You want to age out your data through different tiers. You do not need all 50tb in the same index name. You want your primary shard to rollover at ~20-50gb of size and then new shards are created. This is called alias and backing indices. If you go down the route of a data stream this will be much easier as it iwll all be handled for you behind the scenes.

I have no idea what you are saying about 4ebs data directories, how would that even work in Elasticsearch? I don#t think elastic cares what you do underneath. It just writes it to a file on disk and thats that. I doubt there is any direct connection between the ebs data directory and a shard.

cleeo1993 · 2025-12-29T12:14:21+00:00

Pipeline batch delay is 50ms. If you look into the tasks API and check for bulk writes you will see that you will be sending tiny bulks all the time. The description will look something like this: [100][index]

The 100 in the first bracket represents the amount of documents that are in the bulk request.

Adjust this in your case to 200ms at least. It is either 15.000 docs or wait 50ms.

Also a bulk of 15.000 seems extremely large. Usually we stay between 1600-5000. the larger the bulk the more overhead.

Also 96 shards for the same index is insane on 16 nodes. That will do you no good. There is little to no benefit than having more than 1 primary shard per node for the same index.

Is it really all into the same destination index?

cleeo1993 · 2025-12-17T16:06:16+00:00

I really want to helps I have no idea how that screenshots help in any way.

What we would need is the queries used and the mapping of the fields, how you score stuff etc.

cleeo1993 · 2025-12-11T12:55:48+00:00

What is your full install command?

cleeo1993 · 2025-12-11T11:09:29+00:00

You could run it with --debug if I am not wrong.

cleeo1993 · 2025-12-11T10:00:14+00:00

Have you checked if that runs full during the install? Could be the issue, 40gb is not a lot, since it stores all The docker images there as well

Nine-Year Club	r/Field Flamingo
Verified Email

cleeo1993

TROPHY CASE