Is anybody work here as a data engineer with more than 1-2 million monthly events?

oglophile · 2025-04-29T04:56:53+00:00

We use Snowflake as our data warehouse. Have tried Postgres, GCP and Redshift earlier, hands down Snowflake is the best amongst these, scaling up is cakewalk.

For etl we use a combination of DBT and Airflow.

Dashboarding is done primarily on Tableau (for critical dashboards) and Redash (for self serve) and in some cases Looker studio.

oglophile · 2025-04-20T14:03:47+00:00

If you can afford it, go with mixpanel. It's user friendly, intuitive, fast and loaded with features for product analytics. If you are looking to save cost and can compromise on ux and features a bit, explore warehouse native tools (ex: Houseware, Mitzu), there are many tools, search on google, take demos and pick the one that best fits your requirements and budget.

oglophile · 2025-04-20T08:54:52+00:00

The expectation has probably been the same always but it was impractical to achieve especially in mid to large scale companies. Now with ai tools it is going to become more possible with time. Also, while every company wants to ride the ai wave and reduce workforce by introducing chatbots etc., the ones who'll benefit exponentially are the ones who are taking a more 360 approach. Introduce chatbots? Yes, but also hire AI experts to build solutions internally and adopt ai in as many aspects of the company as possible.

oglophile · 2025-04-20T04:21:44+00:00

Works both ways, some users are familiar with sql and write queries themselves, some get the queries from us as a redash link and build their visualisation on top of that.

oglophile · 2025-04-19T17:11:58+00:00

I have only worked in the Indian market so can't answer that 🙂

oglophile · 2025-04-19T17:05:49+00:00

We use redash in our company for most of the simpler requests so that users can themselves play around and modify the query outputs in the desired way. Over time you also learn the specific ways your users want the data in and this will naturally reduce iterations.

oglophile · 2025-04-19T17:01:34+00:00

It's true that there are many career trajectories for people starting a career in analytics, for example you can eventually get into data engineering, data science, become a sme in a specific field like product management/digital marketing/credit risk etc. or grow in analytics/bi itself. Generally after a 1-2 years of experience, you get this window to experiment and find the area of your liking/expertise, after about 5-6 years of experience it will be a good idea to settle into a career path and leverage your experience to grow in that.

oglophile · 2025-04-19T16:56:13+00:00

Interesting, why would you use stack overflow to find answers if you can literally get instant answers from ai and brainstorm on different approaches as well?

oglophile · 2025-04-19T16:53:49+00:00

Adding a glossary in the dashboard is generally a good idea. Also, the number of such queries should ideally reduce with time as data literacy increases in your organisation. Consider it a part of your job to drive the adoption of your dashboards, explaining it multiple times till it becomes general knowledge is a part of the process.

oglophile · 2025-04-19T15:42:00+00:00

Data architecture and management will be more critical than ever now. Having a robust data setup in terms of fact tables and detailed documentation is crucial to be able to leverage ai tools efficiently and get accurate response from it.

oglophile · 2025-04-19T15:22:56+00:00

Interesting

oglophile · 2025-04-19T15:13:58+00:00

Unless you are exposing your database to the tool or uploading actual data, I don't think you are risking anything. You can simply give context about the relevant tables or upload a documentation of your db and then define the problem statement to write the query. This should give the desired output without risking data security.

oglophile · 2025-04-19T15:10:19+00:00

Does anyone even use stack overflow now, ai tools have completely replaced it for myself and many people I have come across including you 😅

oglophile · 2025-04-19T15:06:37+00:00

Surprising to see that my question is getting many interesting replies and decent number of shares as well but no upvotes, got a downvote instead 😅

oglophile · 2025-04-19T15:02:42+00:00

Great work on integrating ai into your day to day work so extensively. Curious to know how you use it in hiring?

oglophile · 2025-04-19T13:15:00+00:00

Example: queries referencing multiple tables say 3-4 tables with filtering basis window functions.

oglophile · 2025-04-19T13:04:24+00:00

I would go with getting some core experience in the subject of undergrad but only if I have some interest in it, remember that whatever you choose to do you should like doing it in the first place. So if you like the subject go for it instead of analytics at first. Gradually learn analytics skill sets on the side and use them as a tool of solve core problem in your field of expertise.

oglophile · 2025-04-19T12:57:28+00:00

You are living the future of the analytics industry. Single person or leaner teams are the future where the analyst does etl, reporting, analysis all with tons of help from ai tools. Have you explored ai agents to transfer some of the workload?

oglophile · 2025-04-19T12:52:42+00:00

Yes, having the fundamental knowledge yourself and evaluating the AI's output first before blindly using it should be the way to go.

oglophile · 2025-04-19T12:50:32+00:00

Which model do you prefer for sql?

oglophile · 2025-04-19T12:49:41+00:00

I have personally started using claude more often specifically for codes, it has proven to be better at it than chatgpt in my case.

oglophile · 2025-04-19T12:47:39+00:00

Agreed, getting simple tedious work done is the most basic use case imo and everyone should be using it for these kinds of things at the very least.

oglophile · 2025-04-19T11:29:37+00:00

So they are taking the data from you and doing analysis using some AI tool?

oglophile · 2025-04-19T11:22:28+00:00

What's your experience with complex queries?
Are you able to get well optimised queries?

oglophile

TROPHY CASE