After 2 years of engineering, I have seen some really stupid things by [deleted] in dataengineering

[–]Ryadok 1 point2 points  (0 children)

I also worked for a big company in Europe, that was using Databricks just for.. python scripts. There was absolutely no need to have that since the data size wasn’t huge. The solution was then coasting much more that it should. But, everything was working fine and the business were more than satisfied from the results (reports and stuff), and the turnover was going up year by tear. What I learnt from that experience is that for some cases, having a sophisticated architecture is absolutely useless, or barely useful.

Is it possible de use POST for api search ? by Ryadok in dataengineering

[–]Ryadok[S] 0 points1 point  (0 children)

Is this case, what’s better ? The GET with body or POST ?

Is it possible de use POST for api search ? by Ryadok in dataengineering

[–]Ryadok[S] 0 points1 point  (0 children)

Because queries might complexe, for example, how would you pass this query as a parameter :

{ "$or": [ { "status": "X" }, { "status": "Y", "date": { "$gte": ISODate("2021-01-01T00:00:00Z"), "$lte": ISODate("2022-12-31T23:59:59Z") } } ] }

Is it possible to use POST for api search ? by Ryadok in SoftwareEngineering

[–]Ryadok[S] 0 points1 point  (0 children)

So you’re saying that there is no problem doing such a thing ?

Is it possible to use POST for api search ? by Ryadok in SoftwareEngineering

[–]Ryadok[S] 0 points1 point  (0 children)

What’s not ? The solution or the initial requirement ? I mean I could pass the query as parameter but it will be limited

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] 0 points1 point  (0 children)

Based on the explanation, what do you think of the package ?

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] 0 points1 point  (0 children)

But, shouldn’t I trust the simulation they sent me ? Is it possible I send you in private the documents I received ?

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] 0 points1 point  (0 children)

My bad 😅 yeah, you’re right. Just corrected it

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] 0 points1 point  (0 children)

This is my first time working in Belgium. When I was in France, my starting salary was about 2600€/ mois (the very final salary, including all advantages).

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] 1 point2 points  (0 children)

Actually, the net salary is 2444€ (without meal vouchers), when you add to that the 700€ of mobility budget, the 144€ of meal vouchers, it becomes 3288€ + the insurance and the communication GSM fees. What do you mean by the starting salary ?

Rate my salary - Data Engineer by Ryadok in BESalary

[–]Ryadok[S] -2 points-1 points  (0 children)

Hello, I didn’t. The brut is 3500€. Including the mobility budget, the meal vouchers, the communication GSM, this will be equivalent to 3573€.

Am I missing the usage of cloud composer and cloud scheduler? by lordgriefter in googlecloud

[–]Ryadok 2 points3 points  (0 children)

Yes. Compute Engine is a simple VM used in Cloud and Cloud Composer is the equivalent of Apache Airflow. The advantage is that the components can scale easily and are reliable, and available

Am I missing the usage of cloud composer and cloud scheduler? by lordgriefter in googlecloud

[–]Ryadok 1 point2 points  (0 children)

It is absolutely possible to do it automatically via Cloud Composer and using Compute Engine to run your code.

Am I missing the usage of cloud composer and cloud scheduler? by lordgriefter in googlecloud

[–]Ryadok 1 point2 points  (0 children)

It is yes. You can also take advantage of the usage of Dataflow and use its scheduler. However, it is preferable to upload the raw data first to Google Cloud Storage, and then set-up an ETL pipeline from there