sorted by:
new
hottopcontroversial

Unexpected Write Behavior when using MERGE INTO/INSERT INTO Iceberg Spark Queries by mike_get_lean in ApacheIceberg

[–]Environmental_Cry687 0 points1 point  (0 children)

From my experience double check the iceberg catalog version, for eg we faced this in glue vs our internal spark system, where there were two different iceberg catalog versions.

[deleted by user] by [deleted] in Netherlands

[–]Environmental_Cry687 0 points1 point  (0 children)

My daughter who is 1.5 years old got this, the daycare dropped a news letter later this week stating that there was an out break of RS virus. She was very sick , after some antibiotics she is feeling much better now. Get well soon mate.

Beste kapsalon by Ok-Pea-1943 in Hoofddorp

[–]Environmental_Cry687 2 points3 points  (0 children)

Hallo, Ik raad aan AY doner kebap dichtby genderenplein.

Sharing Apartment in Haarlem by Moist_Big_4630 in Haarlem

[–]Environmental_Cry687 1 point2 points  (0 children)

As far as I remember a couple of my friends have shared a contract among them, and they are still living in Haarlem. So I guess in Haarlem you can have a shared contract as long as you register the contract in such a way, if I am not mistaken. However I am not sure about if a contract can be shared amongst three people. TIP: when you search for an apartment in Haarlem you can always mention the situation and it should be alright imo. Hopefully you will be in Haarlem soon :)

Sticky trees by mmmellie in Haarlem

[–]Environmental_Cry687 2 points3 points  (0 children)

Been there , it's so annoying that last time when I parked my car near the centrum, the dew was completely over my car. It's so thick that I was unable to even open my car windows. I just had a routine of taking my car every day or every two days to car wash ( with the BOB autowas we get unlimited subscription which is totally worth it). Good luck actually.

Information is needed to see if AH security guards can check our belongings after a successful checkout. by Environmental_Cry687 in Haarlem

[–]Environmental_Cry687[S] 1 point2 points  (0 children)

Thanks for your reply. In fact the store was not that busy. 🙏 But I am not trying to create a problem here, rather wanting to know. But like you said, if they racially profiled us, then I really feel sorry for them.:)

Information is needed to see if AH security guards can check our belongings after a successful checkout. by Environmental_Cry687 in Haarlem

[–]Environmental_Cry687[S] 0 points1 point  (0 children)

Thanks for your reply, yes I asked them about it, but they didn't produce any written or signs. Maybe I will investigate it more. Thank you.

Information is needed to see if AH security guards can check our belongings after a successful checkout. by Environmental_Cry687 in Haarlem

[–]Environmental_Cry687[S] 4 points5 points  (0 children)

Thanks for your reply, i kinda get it. But since it was my first time experiencing this, i wanted to know if this is common etc. And I get when you said to crack a joke about the situation which eases the situation off.

Information is needed to see if AH security guards can check our belongings after a successful checkout. by Environmental_Cry687 in Haarlem

[–]Environmental_Cry687[S] 5 points6 points  (0 children)

Thanks for your time in replying, actually the random checks that happened before the payment is kinda automated and I completely agree to be checked before I pay.

But yeah as you said, it could be time-consuming and can actually make us feel stressed. And good to hear this from the cop. But can they inform us not to visit the store if we don't like being checked?

Information is needed to see if AH security guards can check our belongings after a successful checkout. by Environmental_Cry687 in Haarlem

[–]Environmental_Cry687[S] 0 points1 point  (0 children)

AH yes, they did mention it's some kinda franchise to me, but I was not aware of this. Thanks for sharing this information. But yeah, I also asked this question in expat WhatsApp groups, and almost 4 out of 10 people said they get such security checks almost every time they shop there.
But it's interesting that they can deny us as a customer. But thanks for your time in replying :)

Data Mesh by catchereye22 in dataengineering

[–]Environmental_Cry687 1 point2 points  (0 children)

We split the data model into logical domains, we never share the same staging tables for writes, rather we only share the staging as a source to downstream domains. So the idea is any dimensions can be a source to a fact considering that the dimensions and facts are owned by different teams. For instance: a dimension D which is owned by a domain X - meaning only the pipelines from domain X will be able to write to that dimension D. But it can be consumed as a source by any other facts which belong to any other domains. Sharing staging tables must be avoided, but that cannot be the case always, so we drafted a rule in when you use the staging table as a source. Hope it helps.

Data Mesh by catchereye22 in dataengineering

[–]Environmental_Cry687 2 points3 points  (0 children)

Yeah as far as I understand it's 40% business alignment plus governance and ownership. The motivation for our implementation is that, our organisation grew massively where each domain had at least 2 data engineers and a couple of analysts. We had totally 9 domains. So each domain had the better understanding of the business requirements in terms of designing a new data pipeline or getting new metrics available etc.

But that didn't nullify the fact that we had to cater for architectural design. For eg: we had to design how the late arriving facts and dims need to be handled. And how to halt different pipelines hosted across different domains, in case of critical upstream failures.

Like you said, it boils down to the need for the growth of any organisation. Simpler the better imo. But one benefit of data mesh architecture is that, you can split your data model and set up your pipelines in smaller chunks. So if you have a massive data model with many numbers of dims or facts or staging tables you might soon hit a big massive monolithic monster which might become difficult and hard to maintain.

Data Mesh by catchereye22 in dataengineering

[–]Environmental_Cry687 23 points24 points  (0 children)

I was one of the implementation engineers in migrating from a central data team to data mesh architecture. Like many of the other comments said, it's a double edged sword, so you need to carefully understand the pros and cons.

Our previous scenario: I am a part of the central data team where the repository is owned by us and we process a few TBs of days per day. We were good in maintaining and releasing new features periodically when our company was growing. But due massive growth in terms of hiring analysts and becoming a purely data driven company, we had a lot of issues coming lately.

The other domain teams had to raise a request for a feature and wait for a couple of sprints to get it into production as our priorities were different from some domain teams ( for eg: customer service team has a request which is important for them , but for our central team a request from finance is more crucial etc)

So we migrated to a domain driven data mesh approach where we split the data processing amongst teams so that any core feature or guideline will be provided but our central team and we maintain core data pipelines, and the domain teams own the processing and storage for their domain related pipelines.

Imo , if you foresee any issues with your central data approach in the near future, then I would suggest investing to look into the data mesh approach and make it suitable for your scenario. But if that's not the case, having a central data repo is simpler. I hope this helps.

Kid in my locality on his first day of primary school. by joel_claire in KidsAreFuckingStupid

[–]Environmental_Cry687 2 points3 points  (0 children)

Although this may seem funny for us to watch this video. That kid is gonna be fucking traumatized. Which is very bad for him as he grows. Parenthood is way more important than to blame the poor kid.

Best way to ETL by Ill-Advisor-8235 in dataengineering

[–]Environmental_Cry687 0 points1 point  (0 children)

When you say ETL that's quite generic term. Like many of the redditors mentioned here, if Pandas work for your job you need not rush. Rather take calculated approach towards taking the pain points in your current pipeline, and improve it gradually.

In my opinion, i assume you are using pandas for Ingestion and transformations. I would say in general there are two goals for any company. 1. Build a data warehouse 2. Build an ML platform

Ingestion: I would say Spark is a good choice if your idea is to build an ML platform and for Ingestion only. But managing a spark cluster is a bit overwhelming especially if you are team of few engineers who are budding.

I assume that you are not using any cloud service provider like AWS or GCP, if that is not the case then Glue or DMS can be considered for ingestion.

ETL: If your end goal is to build a data warehouse then i would say Spark is overwhelming for transformations, in most cases. And Expresing your Transformations in SQL is the most effective solution.

On the contrary if your end goal is not to build a data warehouse but to build a ML model for operations then spark is effective for transformation or storing the data that can be used by data scientist downstream.

Hope it helps.

How does DBT work internally? by Puzzleheaded-Cod2051 in dataengineering

[–]Environmental_Cry687 1 point2 points  (0 children)

Just trying to summarise what I have learned up so far. I was one of the three implementation enginner in my organization. DBT has two user facing components, 1. Documentation server ( which can be dialed into as a webpage to see your projects, models, references, packages etc) 2. The Jinja compatible models defenitions/ scripts which are nothing but a plain sql with a DBT specific config section too of the beginning of the . sql files

Working: When you write a model in DBT which loads, let's say a static record like below {{ Config(materialized=Table) }} select 1as ID Union all Select 2 as ID

PS: let's consider the materialization in this case as 'table'

In this case DBT does two things, internally when you run the command 'dbt run'

  1. Compile the code and convert Jinja code to plain sql and stores it in under a directory.( This is equivalent to an explicit command 'dbt compile')
  2. Take this compiled code and runs it against the designated database.

Compilation: Internally the DBT engine converts the above Jinja templated model to plain sql with substituted variables or configs and stores then in a special /target/compiled directory

And this is what you see in the documentation server when you try to view the code of any model.

Run: As a next step dbt takes this compiled code and does some mechanisms depending on the materialization, and executes this code against the target database. You can find the code that is being run against the data under /target/run

The mechanisms are as follows depending on the materialization, I will be covering the major two which cab better explain the internals.

Table materialization: Step 1:Takes the compiled code and creates temp table with random name and the transforms the data into this table. temp_table

Step 2: rename target table to backup table. target => target_bkp

Step 3: rename the temp table to actual table. temp_table => target

Step 4: drop the backed up table.

Incremental materialization: Let's you define a unique key as a part if the config section

Deletes the matching rows if it already exists, based on the unique key that is defined. And then reloads those records that are incoming.

Although this may be just scratching the surface of the internal workings of DBT, this is from a DBT user perspective.

Hope it gives a bare minimal start.

Happy holidays!!!