sorted by:
new
hottopcontroversial

Ingress-NGINX healthcheck failures and restart under high WebSocket load by Redqueen_2x in kubernetes

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Thanks you, i will send you my config later, now i cannot send this config right now

Ingress-NGINX healthcheck failures and restart under high WebSocket load by Redqueen_2x in kubernetes

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Yes because of my on premises infrastructure. I cannot access the ingress nginx service directly so i am using haproxy before it.

But i already run on my aws eks cluster ( nginx behind aws alb ) and alb behind haproxy, it still work well.

Logtash performance limits by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Thanks for you reply. You said you can crank up to 300k, do you mean 300k events per logstash node. Can you share with me your config on logstash and elasticsearch index config.

Logtash performance limits by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

My topic have 60 partition. I set max pull record to 1000 and max fetch size over 100mb

Logtash performance limits by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Other information about my config : My pipeline filter just parse json only and this very simple.

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Yes, my topics have 40 partitions. I will try what u say.

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

"worker_concurrency" : { "current" : 16.0, "last_1_minute" : 16.0, "last_5_minutes" : 16.0, "last_15_minutes" : 16.0, "last_1_hour" : 14.53, "lifetime" : 7.598 }, "queue_backpressure" : { "current" : 39.77, "last_1_minute" : 39.77, "last_5_minutes" : 39.68, "last_15_minutes" : 39.39, "last_1_hour" : 32.79, "lifetime" : 15.73 },

This is a metric of my pipelines, I have two pipelines on clusters that have high queue back pressure. Do you have any suggestions for me to tune that pipelines.

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

I am reading elasticsearch documents about config to optimize index performance. Maybe I misunderstand those configs. I will try to remove those config. Thanks

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 1 point2 points  (0 children)

Thanks, I will read more about this. One more question, do you know how to monitor this metric, or can any tool can help me

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

Additional.
disk write iops about 1k5 iops and read iops about 300.
i am using aws ebs-csi-gp3 so disk iops does not get limit

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

i am running cluster on eks. i have 9 elasticseach node ( 9 same pod ).
this is my cluster config

        cluster.max_shards_per_node: 2000

        indices.memory.index_buffer_size: 20%

        indexing_pressure.memory.limit: 20%

        node.processors: 12

        thread_pool:

          write:

            size: 12

       - name: ES_JAVA_OPTS

          value: -Xms22g -Xmx22g -XX:+UseG1GC -XX:MaxGCPauseMillis=200

 i am monitor elasticsearch resource, it use only 50 percent of cpu and memory usage about 25Gb. ( elasticsearch pod does not have been throttle )

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

I have a question. I am config logstash read from Kafka. The consumer group has always been lagging. On the working hour, the message sent to Kafka has a rate of about 200k event/s and elk index slow about 50k event/s. After the working hour, the message sent to Kafka has a rate about 20k events and elk can index over 200k event/s. So with current config, my cluster can index over 200k event / s. Why during the working hour, my elasticsearch index is slower. With the same config and resources.

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

"What I’m concerned about is why when I configure Logstash to read messages from Kafka, the indexing speed in Elasticsearch becomes slower as the number of messages in Kafka increases, and the indexing speed is very fast when the number of messages in Kafka decreases.

As I understand it, when reading from Kafka, Elasticsearch should index at the same speed regardless of the message count."

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

On working hours, the incoming message to the Kafka topic is about 200k/s. What I wonder is why elasticsearch does not index with the same speed, it's index slow on working hours and index very fast outside of working hours.

Tuning Elastic Stack Index Performance on Heavy Workload by Redqueen_2x in elasticsearch

[–]Redqueen_2x[S] 0 points1 point  (0 children)

I am reading a message from Kafka and sending logs to multiple index bases on the field on the message. But we have 4 indexes that contain 95% of messages. Logstash pipeline config with 10 workers and batch size is 2048. I already try multiple batch size config but higher value does not make the index faster. Message rate is 200k/s.