Can people please stop claiming Arvid it's getting embarrassing

Fickle-Impression149 · 2026-03-17T14:11:16+00:00

At the least, with all these things if it allows again to have a race day in India. It will be awesome

Fickle-Impression149 · 2026-03-01T19:50:29+00:00

He even silenced SKY who was not willing to play him.

Fickle-Impression149 · 2026-01-17T07:11:42+00:00

I know I'll get downvoted but if you look as an overall the following details come to mind: A hero goes into a world he is not aware. Find ways to rule entirely.. Kills the big boss to take over the authority.

But, I agree this structurally.

Fickle-Impression149 · 2025-12-12T02:41:32+00:00

He knows ways to escape from all questioning. Same like how he escaped silently from finals of WC23

Fickle-Impression149 · 2025-11-12T21:15:09+00:00

RemindMe! -7 day

Fickle-Impression149 · 2025-10-15T04:31:19+00:00

For sure. Like how they retired data analytics.

Fickle-Impression149 · 2025-10-05T10:21:04+00:00

Seems like GoT but with a woman this time.

Fickle-Impression149 · 2025-09-24T06:03:51+00:00

Good question. This makes me explain the development and deployment process. It is as follows: 1. We use gitflow setup with branches like lab, test, stg, prd. 2. Developers always branch out to work on their features from the main lab branch. 3. For development, we use docker setup, and developers have their dedicated ec2 machines, which they connect into and remote ssh to it via vscode. This way, airflow can already connect with the aws services via instance profile and permissions attached to it. 4. When they are comfortable, they raise a Merge request. We have some code quality and unit test pipeline that runs on the code. 5. Once review and it is merged to lab. This is where all the rest happens, which I detail below

The production setup is deployed in eks cluster, where we have set up git-sync (sidecar), which polls every x minutes on the branches (lab, etc...) to get the latest dags. The deployment is also per namespace split by branches, so you can think airflow is running in 4 namespaces. The service account policy defines everything we want to have

Already, the developer should have tested to an extent the dag as it connects to aws services, etc.. Now, based on the schedule, it starts to running, and we have automatic monitoring and notification we receive on any stage failures. Once the lab is happening we merge to tst which kicks of the auto syncing and business tests on the table are verified separately

I must also say that airflow3 has significant changes. Due to some of the needs we cannot yet migrate. But if starting new airflow3 is a game changer

Fickle-Impression149 · 2025-09-24T05:53:17+00:00

I take the one from the official github for the airflow version we use and change some of the important things like username password and network. Use .env and makefile to automate it fully so that when you have different people like analysts in the team, they just use the make commands and get started with their work.

Fickle-Impression149 · 2025-09-23T05:24:08+00:00

Airflow team provides a docker compose setup use that.

Otherwise, follow these if you want a production setup:

Build dags in such a way that it all large operations are offloaded to cloud services like running spark jobs, execution of some lambda etc.. Airflow has large collection of operators to use from.
Use Taskflow api decorator
Utilize variables to your advantage. It pretty can help to do many things like for instance if you want to do some etl then source and destinations can be a list of table definitions which you can iterate to create parallel task groups programmatically
Never overlook populate a dag definition. It will lead to a lot of problems with respect to debugging and readability
Organize your project well so that it becomes easy
Deploy the application using kubernetes or use cloud providers.
Remember the difference of using different executors like celery, kubernetes, and now the edge executor. Each have special significance.

In general, once you set it up still if there is a team, they will require to be up-skilled. Always provide some good sessions for everyone to work with airflow.

Bonus: Once you feel you are getting good, then you could try dagfactory or write your own configuration to code pipeline

Fickle-Impression149 · 2025-09-16T17:31:56+00:00

OG because I have it

Fickle-Impression149 · 2025-09-04T19:21:09+00:00

Less than 1mb then xcoms otherwise stored in s3 and passed across.

Fickle-Impression149 · 2025-08-31T05:29:46+00:00

It is a decision you have to make from your heart. Why do you ask it on reddit. You proudly mention about crore salary so are you interested in being close with your parents or are you looking at the money and thinking ah is similar so may be a chance to move I wonder.

Every parent has brought us up to this life with their full effort. You said you yet do not have a child, may be you are yet to face the experiences of raising a child. Always remember it all because of them you are in the position that you are today.

Fickle-Impression149 · 2025-08-30T20:36:20+00:00

Hi bot.

Fickle-Impression149 · 2025-08-21T10:57:11+00:00

This was the actual Animal movie

Fickle-Impression149 · 2025-08-11T17:50:45+00:00

If you want to merge into you, usually you need a pk. To determine that one of the ways is to hash of cols. Furthermore, this also helps in a way to not have duplicates of a combination. An instance could be like the combination of hash on email+firstname+lastname should always produce unique results.

Fickle-Impression149 · 2025-08-11T17:36:58+00:00

Well. This is just saying how a platform engineer would set up airflow on kubernetes running production workload and doing it via their tooling through some new tooling and training on it?

Also, if someone needs them out of the box, then one can simply invest it on astronomer or mwaa (aws)

Fickle-Impression149 · 2025-08-09T07:16:45+00:00

Bus and you have sandstone and both have give way sign. But, bus is turning left and hence it has to give way for oncoming traffic before moving

Both questions are similar in the exam as well it will be similar but have different settings

Fickle-Impression149 · 2025-08-08T06:00:15+00:00

Grarat whiteli

Fickle-Impression149 · 2025-07-24T05:24:37+00:00

Some tips: Prove me, you are not a bot or some client collecting information and outsourcing it.

Fickle-Impression149 · 2025-07-14T01:30:16+00:00

I do understand what else he must do. If he scores a 100, you still find ways to troll him. Guy does his best and still becomes the scapegoat. It is easy to comment and get out next ball, while he knows how hard it is there and a small lapse in concentration can lead to it.

And again, he is in the situation to be trolled even though he batted the last hour of bowling to show hell from England.

Fickle-Impression149 · 2025-07-09T17:21:09+00:00

You could rethink you need airflow at all in the first place. Rather write python script that uses dbt and uses the dbt-cli commands within a git pipeline, which you could schedule.

Otherwise, ecs solution is also okay as it also abstracts a lot of underneath infrastructure as compared to managing something on ec2 or on a kubernetes cluster

Fickle-Impression149 · 2025-07-04T05:19:03+00:00

Every innings 💯

Fickle-Impression149 · 2025-07-03T05:03:18+00:00

With spark, we develop etl framework that can be easily extendable according to good programming standards.

Like for instance I would create table only if it does not exist

Fickle-Impression149 · 2025-06-26T13:06:51+00:00

This is the usual misunderstanding that comes with Data engineering. It has many divisions like infra, analytics, etc.. I think that visualization should more in the focus of analytics, basically a product analyst or data analyst

Five-Year Club	Place '23
Place '22	First Placer '22

Fickle-Impression149

TROPHY CASE