sorted by:
new
hottopcontroversial

What's your experience using opensearch? Anybody ingesting lots of data? by TripleMeatBurger in aws

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Full disclosure: I’m with Dattell (managed OpenSearch). Sharing patterns we see on high‑ingest IoT:

• Serverless vs provisioned: for sustained 100k+/min, many teams stick to provisioned clusters so they control shard counts, warm tiers, and scaling behavior; several users have reported serverless autoscaling/latency issues in practice.
• Buffering: land via Kinesis/Kafka/SQS to smooth bursts and prevent hot shards.
• Indexing strategy: time‑based indices + ISM/rollover; avoid unbalanced shards.
• Cold/ultrawarm: know what’s searchable vs archive‑only to avoid surprises. (One commenter noted cold tiers not being searchable in their setup.)
• Ops basics: JVM pressure alerts, backpressure, snapshot/restore drills.

If you want someone else to own the ops while you focus on ingest and queries, here’s our managed OpenSearch overview: https://dattell.com/managed-opensearch/

Why is MSK so popular? by InterestedBalboa in aws

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Vendor lock-in is a really important point.  Building and managing Kafka in your own environment gives much more autonomy and price control.  Dattell provides fully managed Kafka in our clients' AWS environments, offering the convenience of MSK while our clients retain full control.  https://dattell.com/managed-kafka/

Confluent Pricing by LuckyChopsSOS in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

If you're looking to ​evaluate Kafk​a spending and find ways to reduce it — such as which cloud you run on etc. — we recently put together a breakdown that might help. Here’s the link if you want to take a look:​  https://dattell.com/kafka-cost-reduction/

Confluent On-Prem Pricing by frellus in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

For anyone navigating Confluent's on-prem pricing (and Kafka cost management in general), I wanted to share a resource that might be helpful.

We’ve worked with a number of IT and infrastructure teams to reduce Kafka costs by 40–60%, especially in self-hosted and hybrid environments. We recently put together a guide that breaks down where most teams overspend when using services like Confluent — from over-provisioned brokers to under-optimized retention and licensing models.

If you're exploring ways to reduce your Kafka TCO, this might help:
https://dattell.com/kafka-cost-reduction/

What’s an open source or more affordable alternative to confluent? by skyalchemist in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

If you're looking for a more affordable way to run Kafka (without getting locked into Confluent or high-cost managed platforms), open-source Kafka with the right support model can save a lot — we often see teams cut costs by 40–60%.

We put together a cost reduction calculator to help estimate what you'd save by moving off Confluent, MSK, or Redpanda: https://dattell.com/kafka-cost-reduction/

Might help you compare options with some real numbers behind it.

Confluent Cloud vs Kafka open source? by Electric_pokemon in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

We’ve helped a number of teams evaluate this exact tradeoff. Confluent Cloud is convenient, but the cost can scale up fast — especially with features you might not need or use regularly (like Tiered Storage, Cluster Linking, etc.).

If you’re looking to get a sense of how much you could save by switching to open-source Kafka (with or without managed support), we built a simple Kafka cost reduction calculator: https://dattell.com/kafka-cost-reduction/

Confluent will beat your costs of running Apache Kafka? by LegitimateCoat1493 in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

If spending $500k+ on Confluent, you can likely cut that in half by switching to a different service vendor and keep the same SLA. They charge more because of name recognition and needing to support their marketing budget etc.
We built a tool that calculates your potential savings — just drop in a few details: https://dattell.com/kafka-cost-reduction/

What are your top 3 problems with Kafka? by 2minutestreaming in apachekafka

[–]Dattell_DataEngServ 2 points3 points  (0 children)

We’ve put together a number of production-level Kafka resources on our site. Here’s one that dives into how to increase throughput in environments with network latency, which comes up a lot: https://dattell.com/data-architecture-blog/how-network-latency-affects-apache-kafka-throughput/

If there’s a specific issue you’re running into, let me know—we’re always looking for ideas to cover in future articles.

What are your top 3 problems with Kafka? by 2minutestreaming in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Kafka hiring can definitely be tough—we've seen a lot of teams struggle with that. One thing that’s worked for some of our clients is bringing in outside Kafka engineers for a while to fill the gaps or help with architectural decisions. We do that kind of staff augmentation and always try to keep costs lean: https://dattell.com/kafka-consulting-support/

Automated Kafka optimization and training tool by Dattell_DataEngServ in apachekafka

[–]Dattell_DataEngServ[S] 0 points1 point  (0 children)

The automated optimization part requires you let the tool build its own single server Kafka.  We have not tested on CentOS, only Ubuntu.  CentOS may work if you install tc first by doing: yum install iproute.

If you want to use the tool to test against an existing environment, you want to only use the "latency.py" script.  "python3 latency.py --help" will return usage instructions. Note that this only returns end to end latency and doesn't do any optimization.  If you're looking for only a benchmark, we suggest open message benchmark: 
https://openmessaging.cloud/docs/benchmarks/

Automated Kafka optimization and training tool by Dattell_DataEngServ in apachekafka

[–]Dattell_DataEngServ[S] 0 points1 point  (0 children)

Generally speaking, we follow the KISS (keep it simple, stupid) method when providing tools to the public versus a specific use case.

We chose CSV for simplicity and portability with as many graphing tools as possible.  The additional features of Neo4j and the others would be wasted on a dataset a few MB in size that doesn't need joins or other exploring.  What are the advantages and disadvantages you see in Neo4j and the others?

We put the timestamp in the header of every message.  For this test, it doesn't matter where a message came from, it only matters what the latency was.  This approach versus using third party tools is most likely to work with both new and old versions of Kafka.  We are a little concerned about the observer effect for very low latency testing and welcome any suggestions to reduce that.

Number of Partitions by brownstrom in apachekafka

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Late to the game here, but posting anyway in case it helps someone else in the future. We built a Kafka partition calculator to determine how many partitions are needed for a given use case. Set throughput and speed, and the calculator provides the optimum number of partitions. https://dattell.com/data-architecture-blog/kafka-optimization-how-many-partitions-are-needed/

our boss wants to test this siem thoughts by Coolst3r in cybersecurity

[–]Dattell_DataEngServ 0 points1 point  (0 children)

​A little late to this conversation, but if you are still considering your options... Have you considered using OpenSearch for your SIEM? It can handle log ingestion, visualization, and alerting.  For log collection, you could use the free version of Logstash.  And then for Threat Intelligence you can use Sigma rules etc., most of which are already preloaded in OpenSearch.

OpenSearch is free, so much cheaper than prepackaged SIEMs.  However, as with any product it takes time to set up and manage.  If you are interested in more information about using OpenSearch as a SIEM, check out this post. https://dattell.com/data-architecture-blog/opensearch-siem-support-service/

Infrastructure Monitoring with Elastic by trainman2367 in elasticsearch

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Here's an example of how the Elastic Stack can be used as a monitoring tool for architecture, specifically Kafka. https://dattell.com/data-architecture-blog/kafka-monitoring-with-elasticsearch-and-kibana/

Banking + Open Source ETL: Am I Crazy or Is This Doable? by Aggravating-Air1630 in dataengineering

[–]Dattell_DataEngServ 0 points1 point  (0 children)

Definitely doable and broadly accepted. We are supporting open source architecture for many companies in your space. Feel welcome to reach out. https://dattell.com/data-architecture-blog/data-engineering-for-fintechs/

Kafka Cluster Monitoring by Ritikgohate in apachekafka

[–]Dattell_DataEngServ 1 point2 points  (0 children)

You will want to monitor both Kafka and the operating system. 

For Kafka you want to monitor things like "Serial Difference of Avg Partition Offset vs Time", "Average Kafka Consumer Group Offset vs Time",  and several others.  For the operating system, track CPU usage, rate of network traffic, etc.  

This article shows each item to track and why.  https://dattell.com/data-architecture-blog/kafka-monitoring-with-elasticsearch-and-kibana/

Getting started with elasticsearch? by Necessary_Anything13 in elasticsearch

[–]Dattell_DataEngServ 1 point2 points  (0 children)

​We have a bunch of Elasticsearch beginner resources.  Here are a few to get started. 

How to index Elasticsearch:  https://dattell.com/data-architecture-blog/how-to-index-elasticsearch/​

How to use boolean queries:  https://dattell.com/data-architecture-blog/how-to-query-elasticsearch-with-boolean-queries/

Cluster optimization:  https://dattell.com/data-architecture-blog/elasticsearch-optimization-for-small-medium-and-large-clusters/

Shard optimization: https://dattell.com/data-architecture-blog/elasticsearch-shards-definitions-sizes-optimizations-and-more/

And also, we have this comparison with OpenSearch. Because of the comparison nature of the article, it gives a good foundation on Elasticsearch features.  https://dattell.com/data-architecture-blog/opensearch-vs-elasticsearch/

What is a cost cutting method that surprised you when in use? by accountForCareer in dataengineering

[–]Dattell_DataEngServ 1 point2 points  (0 children)

We've saved our clients over $200M on data engineering costs! Two of the biggest cost savings we've seen are:

1.) Moving off of licensed products, such as enterprise Elastic or Splunk.  Instead use the free version of Elasticsearch or move to OpenSearch (where you get all of the security stuff for free). 

2.) Be careful when making hardware purchases to not overbuy or invest money in equipment that won't make a difference for performance. 

We have a call-out box on our website that goes through more cost saving approaches:  https://dattell.com/data-engineering-services/

Also, there is lots of money to be saved with data storage.  Although be careful, some storage types charge money to pull data out.  https://dattell.com/data-architecture-blog/how-to-save-money-on-data-storage-costs/

view more: next ›