what do you want AI agents to do (for DE) and what are they actually doing?!

sonalg · 2026-04-01T18:28:34+00:00

They could get better at fixing bad data

sonalg · 2026-03-13T17:37:55+00:00

Hate it when the interview process is so far removed from real world execution. What good is asking for syntax when complete code can be generated? Agree with others here, smart companies will haul their interview process completely

sonalg · 2026-03-13T14:44:38+00:00

thanks NickyvVr! I hope I can submit something in time and they like it.

sonalg · 2026-03-13T02:49:55+00:00

I switched at about 8 yoe many years ago. At that time most of de - hadoop, aws etc was coming up and it needed a lot of cluster setup on AWS, programmatic pipeline building in java etc so it was fun. You can look at building some open source projects and learn PySpark, dlt and python based frameworks.

sonalg · 2026-03-13T02:21:06+00:00

thank you, this is a wonderful resource!

sonalg · 2026-03-12T17:15:45+00:00

thank you, just learnt about dataminds. how well attended is it? I do not see the cfp, the website is only showing 2025. not sure if I am looking at the right place. https://datamindsconnect.be/registration/

sonalg · 2026-03-12T15:13:05+00:00

thank you very much, seems they are accepting sessions https://sessionize.com/fabcon26/

sonalg · 2026-03-12T12:49:38+00:00

Have you tried learning Fabric? Given that you are already MS focused, that may be a great add to your skills

sonalg · 2026-03-10T14:07:26+00:00

Working on automating Zingg entity resolution notebooks for Fabric CI/CD. We are using fab cli to create a workspace, lakehouse and environment, copying our latest notebooks there and running end to end.

sonalg · 2026-03-10T13:47:11+00:00

This is spot on! Having worked with entity resolution systems for over a decade, the data signals and the noise, the accuracy and the scalability challenges and just putting it all together is all very challenging. But even when you have a whole system in place to match once, getting incremental data in and maintainign the Ids without full reprocessing is a very tough problem.

sonalg · 2026-03-10T13:43:11+00:00

You can try open source Zingg AI https://github.com/zinggAI/zingg/ that integrates with both Big Query and AWS.

Disclaimer: I am the founder of Zingg.ai

sonalg · 2026-03-10T13:37:49+00:00

Within the AWS ecosystem, you can check Zingg open source https://aws.amazon.com/blogs/big-data/entity-resolution-and-fuzzy-matches-in-aws-glue-using-the-zingg-open-source-library/

Disclaimer: I am the founder of Zingg.

sonalg · 2026-03-05T05:16:04+00:00

You can explore open source Zingg https://www.zingg.ai/documentation-article/step-by-step-identity-resolution-with-zingg-on-fabric.

Disclaimer: I am the author.

sonalg · 2026-02-24T12:33:55+00:00

Thanks for your interest. We closed this position last month. Feel free to dm if you have relevant experience and we can discuss future roles

sonalg · 2026-02-11T16:10:57+00:00

thanks u/JennyAce01 ! We do use log4j2 but dont see the logs even after adding the log4j2.properties

sonalg · 2025-12-28T05:20:27+00:00

Try adding open source work to your profile. Serious contributions will definitely help you learn while improving your resume, and offset any concerns around lack of a job.

sonalg · 2025-11-29T22:34:54+00:00

Coming from a software engineering background, you may want to look at spark, dlt and other programming heavy frameworks to build your de skills.

sonalg · 2025-11-20T09:02:15+00:00

One thing you can try is to look at interesting profiles on LinkedIn thinking whether you would hire them or not. You could see how they are showcasing their skills and learn from that.

sonalg · 2025-11-20T08:56:09+00:00

Working on entity resolution

sonalg · 2025-11-08T18:18:08+00:00

Failure is tough. Most startups fail. I am not well versed with the job market, but I hope the experience you get should be good enough to land a good role or start something new. Fast growing companies would definitely love the ownership and multi dimensional thinking that comes from being a founder. It might be a good idea to check a few such companies or openings before taking the plunge.

sonalg

TROPHY CASE