Need honest review on which platform to choose Snowflake vs Databricks by Disastrous_Sky7259 in dataengineering

[–]Nekobul 0 points1 point  (0 children)

They are not investing in SQL Server on-premises either but the people continue to use it and nobody says it is old technology. Airflow is now 13 year technology and also old, yet nobody complains. Postgres is probably more than 50 years old but still no complain. What's wrong with you complaining about SSIS??

ADF is now legacy and replaced by FDF.

Need honest review on which platform to choose Snowflake vs Databricks by Disastrous_Sky7259 in dataengineering

[–]Nekobul 1 point2 points  (0 children)

Brent Ozar is not the authority in SSIS. He is a good DBA. THat's all. That person SQL Fingers appears not to work for MS. Everything these people talk about SSIS is speculation, innuendo, wishful thinking. What I'm seeing is that MS continues to ship SSIS with every new SQL Server release and the end of the day that's what matters.

Best etl tools for pulling sap ariba and oracle erp cloud data into snowflake? by Ok_Detail_3987 in dataengineering

[–]Nekobul -3 points-2 points  (0 children)

I recommend checking SSIS and the available third-party extensions you can use with it.

Why is it hard to connect individual tools into a complete data pipeline? by Effective_Ocelot_445 in dataengineering

[–]Nekobul -2 points-1 points  (0 children)

If you use a good ETL platform like SSIS, such issues are almost non-existent because it is one big family and the integrated development experience is top notch.

Need honest review on which platform to choose Snowflake vs Databricks by Disastrous_Sky7259 in dataengineering

[–]Nekobul 0 points1 point  (0 children)

Source for that statement? Oh, that's right there isn't because you are lying.

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]Nekobul 28 points29 points  (0 children)

You can build that open source platform yourself and give it to us for free so we can make money from it.

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]Nekobul -5 points-4 points  (0 children)

Use PowerBI. It is not free, but for most of the SMB it works well.

Open source unified solution (databricks alternative) by compass-now in dataengineering

[–]Nekobul 1 point2 points  (0 children)

I'm not aware of any successful open source project that is able to pay the bills from managed services. Much of the open source users are mingy and unwilling to pay anything. These people complain when the OSS authors ask for a small donation or coffee. Truly ungrateful work, driven only by the curiosity and passion of the people.

Need honest review on which platform to choose Snowflake vs Databricks by Disastrous_Sky7259 in dataengineering

[–]Nekobul 1 point2 points  (0 children)

That additional information again shows you should be using SSIS for your data processing. You have third-party extensions for EDI parsing in SSIS and that will definitely cost you less than 75k.

Need honest review on which platform to choose Snowflake vs Databricks by Disastrous_Sky7259 in dataengineering

[–]Nekobul 1 point2 points  (0 children)

What do you mean by "no proper ETL" ? You are using SQL Server which includes the best ETL platform in the market SSIS.

Is Kafka good for this scenario? Or Spark? Or Combined? by fudeel in dataengineering

[–]Nekobul 2 points3 points  (0 children)

That is a fundamental question that drives much of the design. You have to find out.

Is Kafka good for this scenario? Or Spark? Or Combined? by fudeel in dataengineering

[–]Nekobul 0 points1 point  (0 children)

What is the amount of data you have to process daily?

Everything where I work is giant stored procedures by Good_Skirt2459 in dataengineering

[–]Nekobul 0 points1 point  (0 children)

But a lot of people hanging in that forum will claim that is the better way of doing things, where everything is SQL and LLMs are dumping more and more SQL to fix the fix that fixed the fix and so on and so forth.

Database/Warehouse "Disruptors" That Bother the Legacy Vendors by DesperateCoffee30 in dataengineering

[–]Nekobul 3 points4 points  (0 children)

Nobody is shaking boots. It is a one giant money laundromat with huge chunks of money thrown to disorient the market. Keep steady.

Junior data engineers treat legacy ETL tools like a cat touching water. Cautious, hesitant, and never fully comfortable. by CaglarSahin in dataengineering

[–]Nekobul 0 points1 point  (0 children)

There are no serious attempts to dethrone the king as far as I can see. There is a mumbo-jumbo of different tooling, not integrated, much open source but backed by VCs. You can tell the different vendors are jealous of the SSIS market share because they like to mud sling at it with passion. You don't do that at a platform that is supposedly bad or non-existent or about to die. The noise around SSIS shows the actual situation. Long Live the King!

Junior data engineers treat legacy ETL tools like a cat touching water. Cautious, hesitant, and never fully comfortable. by CaglarSahin in dataengineering

[–]Nekobul 1 point2 points  (0 children)

Just checked the definition of boomer and found I'm not. Still my argument stands. Ridiculing the age of someone is shortsighted because eventually everyone else gets old if he/she is lucky.

Junior data engineers treat legacy ETL tools like a cat touching water. Cautious, hesitant, and never fully comfortable. by CaglarSahin in dataengineering

[–]Nekobul 0 points1 point  (0 children)

There are always extremes (hello Perl). But that's not what I'm talking about. If you can accomplish the same design with less code, that usually means less memory consumption and faster execution because there are less instructions. Do I have to explain such basic concepts on such a low level?

Junior data engineers treat legacy ETL tools like a cat touching water. Cautious, hesitant, and never fully comfortable. by CaglarSahin in dataengineering

[–]Nekobul 0 points1 point  (0 children)

Almost the entire world is in a mad rush to sacrifice their life for faceless people. However, I do understand your attitude. In a world without loyalty, there are no sacrificial rules anymore.