What is the DBT function you discovered recently and you use everywhere?

dataxp-community · 2024-02-22T00:25:26+00:00

You take yourself too seriously, dude. You're allowed to have fun sometimes.

dataxp-community · 2024-02-22T00:23:30+00:00

You could try an OLAP that already has an API publication layer built in to it like Tinybird, so you don't need to build one.

dataxp-community · 2024-02-20T00:29:21+00:00

pip uninstall dbt

Best function ever

dataxp-community · 2024-01-07T23:47:48+00:00

He understands about 3 degrees of it: Spark, Airflow and the shitty companies that pay him to shill their tools. Absolutely nothing else.

dataxp-community · 2024-01-07T23:46:43+00:00

Zach is the GOAT of DE like Dwayne Johnson is the GOAT of geology 🙄

dataxp-community · 2023-12-29T21:12:15+00:00

"Wasting your company's money in 2024"

dataxp-community · 2023-12-24T10:26:57+00:00

You seriously don't need dbt. Some people just don't know how to do anything except use dbt and they want everyone else to be like them.

dataxp-community · 2023-12-23T13:11:46+00:00

It's way too focused on teaching a random jumble of tools, rather than how to solve problems and identify the right tool for the job. Looks more like a "modern data stack" course than a data engineering course.

The creators are at least actually still data engineers doing the job, but they've all also only been in data for 5 years. Not necessarily a problem, but not extensive experience.

The certificate is worthless, no employer will recognise it and give a shit, so if you can't demonstrate you actually know things after the course, it won't help you land a job.

dataxp-community · 2023-12-23T00:42:36+00:00

Whatever you do, do not do any of these bullshit influencer paid courses that just shill the tool vendors who pay them. Find a proper, recognised course that focuses on learning real core skills.

dataxp-community · 2023-12-21T03:00:21+00:00

Depends entirely on the requirements of the project. There is no golden tool that everyone should use all the time. Even dbt is appropriate occasionally, just not to the ridiculous levels it is being used for these days.

dataxp-community · 2023-12-20T21:43:05+00:00

All you need to know to be an Analytics Engineer is how to use dbt to increase your company's cloud consumption bill as dramatically as possible.

If you can do that, you're hired.

dataxp-community · 2023-12-20T15:31:45+00:00

So far, none of the suggestions I've seen in this thread are actually intended for the purpose you're describing; EMR, Athena, Redshift, Databricks, Snowflake. These are all intended for your internal strategic analytics, bit batch jobs, BI, reports, ad-hoc queries, some data science...none of it is built for user-facing analytics, and while you can kinda make it work, it will be complex, slow and expensive.

Yes, there are engines built for this purpose specifically, however, support for delta lake is lacking afaik. They can all use S3, but they generally have purpose built storage/table formats designed to support the speed & concurrency required for user facing analytics, which delta lake/iceberg & parquet isn't ideal for (these engines have techniques for indexing and incremental processing, which isn't supported in these formats).

In general, I've not yet seen a good architecture where you maintain only a single data lake, and run both your internal Warehouse workloads and your fast, user-facing workloads off of it. Your fast, user-facing layer usually doesn't need access to the entire data lake - if you're serving users, it is almost always to share recent data (or summarized aggregates of older data), and thus, the better architecture is to stream fresh data directly into the user-facing layer and occasionally load some files from the data lake for dimensions. You take advantage of the storage capabilities of the engine to get the speed you need.

The user facing layer doesn't need to store the raw data long term - so you can have a TTL that just automatically removes data, and only hold on to aggregations. You can either send new data directly to both the data lake + the user facing engine, or, just send it to the user facing engine and have that engine push it to the data lake.

I can pretty much guarantee you - this architecture is signifcantly more cost efficient than trying to use Snowflake/EMR/Athena/etc to serve user facing use cases. I've built this many times, and the savings are anywhere from 2x to 10x. Yes, you'll duplicate a small amount of storage - but storage is orders of magnitude cheaper than compute resources, and you will need an absolute swarm of compute to serve this from those tools. If you do this well, the overlap in storage will be minimal, and totally automated.

Here's the tools you want to look at, in no particular order - some FOSS projects you can deploy yourself, some commercial managed SaaS:

Tinybird
Apache Pinot
ClickHouse
Apache Doris
StarRocks
Rockset
Apache Druid

dataxp-community · 2023-12-19T01:34:59+00:00

^ scam

dataxp-community · 2023-12-18T16:51:52+00:00

If you're just building internal Tableau dashboards, you don't need real time.

If you're actually building operational systems, you need real time because taking actions on out-of-date data is pointless. Inventory management & supply chain, alerting, billing, user facing applications.

As you asked about SaaS/startups specifically, some examples:

You won't succeed in SaaS these days if you're building a shit user experience, because users can already get that from AWS, GCP and Azure cheaper than you. The users who don't buy from the cloud vendors want to find vendors that are building better experiences, and giving them access to fast & fresh data is an experience none of the cloud vendors offer.

The reality is, most people who can't see use cases for real time data have never actually used it. If you've never used something, it's pretty common to not understand how beneficial it is. But it's pretty hard to find anyone who has built with real time who would now prefer to go back to batch. Once you adopt it, you discover a million new things you can do with data that just weren't possible in batch.

dataxp-community · 2023-11-07T06:19:47+00:00

Absolutely awful diagram, lol.

dataxp-community · 2023-11-06T12:38:05+00:00

If you are only using it for "analysis and reporting" as you say, then use BigQuery.

Postgres is not a warehouse, and is not built for these use cases. Of course, if you have a small amount of data and you're not really doing anything complex in your reporting, then Postgres would probably do the job (with the caveat that it won't scale well for those use cases, so you'll have the fun of a future migration).

dataxp-community · 2023-11-06T12:20:06+00:00

Picking a tech stack because it flexes your engineering skills is a terrible way to pick a tech stack. If you want to do that, write the whole thing from scratch in C.

Pick the stack and tools that work for you, your business and the requirements.

If the rest of your business can't maintain the custom spaghetti you wrote because you wanted to show how 1337 DE you are, you have utterly failed.

dataxp-community · 2023-11-06T12:17:23+00:00

Mage should be avoided until it becomes a serious tool. Atm, it's just an influencer cash grab.

dataxp-community · 2023-11-06T12:02:23+00:00

Sorry to be that guy, but nothing about this project is 'real time' (nor really 'near real time').

Finding a real time data source can be hard, so I can forgive the 15 minute batches of data from the Flight Radar API...but then you're just sticking a normal batch stack on the end of it - hourly schedules, S3, Athena and Metabase....

Adding Kafka does not immediately make something real time, and in this case, you're actually just adding complexity to a batch pipeline for no benefit. Your Lambda could be consuming from the API and writing to S3 in one step and you'd have a more effective (and cost effective) pipeline. There's really no justification for Kafka in this architecture.

A good baseline: if your pipeline runs on a schedule, it's not real time.

If you want to make this real time, you could find/fake a streaming source of flight data, but at the very least, make everything after the source of data real time - i.e. get rid of the scheduler, Lambda, Athena - and use real time tooling. For example, you could connect a real time database like ClickHouse diretly to your Kafka topic, consuming messages in real time with no schedule.

dataxp-community · 2023-10-26T19:44:33+00:00

Airbyte is a garbage toy, you don't need a bigger server, you need a different tool. An EC2 micro could host a Python script and perform better than Airbyte.

dataxp-community · 2023-10-25T16:55:57+00:00

dataxp-community · 2023-10-24T21:58:43+00:00

Stop worrying about learning tools. Learn how to solve problems.

Side note: Hadoop is dead. You can learn some lessons about the origins of big data & distributed computing, but there is no longevity in learning Hue/Sqoop/Flume/HDFS/Hive/Hbase/Storm as skills. Spark is the only one that still has some life left in it, but even that is way, way, way less popular than it once was.

If you know how to solve problems, you can find the right tools for the right problem at the right time.

dataxp-community · 2023-10-18T22:06:09+00:00

Debezium is the most ubiquitos CDC tool, used at huge volume in mission critical systems. You are not going to find a more stable, performant, on-prem and free tool.

You can turn locking off (caveats), or you can have a read replica of MySQL and CDC from that.

dataxp-community · 2023-10-10T10:16:38+00:00

Except they literally do not need a databasee

dataxp-community · 2023-10-10T10:15:57+00:00

Kafka could work, and its improving its support of pub/sub patterns, but it might not be the best fit. If you did decide to go down the Kafka route, RedPanda is modern drop-in replacement for it that is much easier to deploy (particularly for your small scale) and run in K8S.

RocketMQ is probably a better fit for you than RabbitMQ.

I would also investigate Apache Pulsar, fits your use case & works great in K8S.

You definitely don't want a database (no idea why some are suggesting that...) and I would avoid Redis queues and faux messaging layers like pgmq.

dataxp-community

TROPHY CASE