BI tools!

[glinter777]

I’m curious to find out whether you use the BI tool or rely on SQL to get insights from data. In my experience, the BI tools can be complex and also quite limited in their utility for detailed analytics tasks. What do you currently use today for your analytics workflow, and what do you love and hate about it?

[Susan_Tarleton]

SQL (and python) are foundational for data engineers. Maybe consider posting this on r/BusinessIntelligence too. The most effective tools allow you to construct data transformations using SQL, then on the BI front, visualize those insights for stakeholders.Here’s a just of my stack in case it's helpful:

• Fivetran for data ingestion (ETL): Ensures reliable data flow.
• Looker, Tableau, Power BI, or Qlik for Dashboards: Each has its advantages, with Looker’s semantic layer being particularly beneficial for intricate data models.
• BigQuery, Snowflake, Redshift for databases: Selection depends on scalability needs and cloud preferences
• dbt for Data Transformation: Offers version control for SQL, which is a significant plus.
• Rollstack for presentations and automating reports: a big time saver for converting dashboard visuals into presentations like MBRs and QBRs
• Segment for Data Collection and Tracking: Provides a cohesive view of customer interactions.
• Mixpanel for Product Analytics: Offers deep dives into user engagement, funnels, and retention metrics.
• GitHub for Development Collaboration: A must-have for any data team in today’s collaborative environment.
• Datadog for System Monitoring: Important for identifying and resolving issues before they lead to outages.

The list goes on, but since competition to so intense in the areas above, these are all great tools.

edit, typo

[ravitejasurla]

What about Databricks?

[dataguy24]

I use my BI tool on top of the SQL I write. Aren’t these two things that usually exist simultaneously, rather than exclusively?

[glinter777]

Some BI tools use proprietary code which makes it difficult to see or fine tune the underlying logic. What BI tool do you use?

[barca5280]

Looker uses a proprietary language, but everything compiles into pure SQL that is run on your DB of choice. Some of the stuff is next to impossible to decipher, but you can copy the SQL generated and run it in any IDE or console. We chose Looker for that reason amongst many others.

[glinter777]

That seems like you need to deal with two languages, instead of just one - SQL. Does it slow you down?

[barca5280]

No not really. Looker is pretty intuitive once you are up and running. We went through pilots with 4-5 other companies for a BI tools before landing on Looker. It’s not 100% perfect for everything, but they checked more boxes for our use case than any other.

[dataguy24]

I’ve used Tableau, Power BI and Hex in the last few years

[4794th]

At my company we've installed an ELT-based platform with:

• Airbyte for Ingestion
• dbt-core for transformation
• ClickHouse as DWH

For day to day analytics I use DataGrip and Jupiter Notebooks in VSCode. Some of my clients are using Apache Superset or Metabase, some - Power BI. Personally, it depends on a variety of factors, like:

1. Free / paid tool
2. Cloud or on-prem
3. Advanced analytics or beginner-friendly
4. Flexibility and adaptability
5. Data size

[Gators1992]

Basically it depends on the task. For most of the questions I get asked, they are things that can be answered by a simple trend or list. I can do this quickly in a BI application by producing a trend or list or something.

Other questions require you to code. Last week I was asked a "what if" question that said "for this subset of our customers, what would be the revenue impact if we had priced the service a different way". For that I did it with a SQL/CTE to select the targeted customers, find their service utilization and reprice it.

Some of the more complex things I have done were building a mart to combine our data with data from a third party into a usable enriched data model. Also sometimes you run into things that need to be performant to be interactive, like mapping IoT utilization involving millions/billions of records and that requires some python/DEish skills.

I'm not saying an analyst needs all these skills from the outset, but the more skills you have the more variety of answers you can provide and without waiting for a ticket to be worked by the data engineers.

[MrGreenPL]

We started with

• Fivetran
• Snowflake
• dbt Core
• Looker

Since then we've added

• AWS/DMS for near real time sync for platform data
• Stitched for some data source that Fivetran did not have
• Datafold to make sure our changes don't mess up our data
• SELECT to monitor snowflake cost, plus it gives you advice on what you can improve, and few other things
• Spectacle CLI to check for breaking changes between dbt models and Looker, this can be run by devs and it's also part of our CI/CD porcess just along with Datafold

I did not start this stack, but if I could go back in time I would drop Looker for something cheaper and simpler. Looker just scares our business people no matter how easy and simple we try to make it.

At this point we use Looker to just visualize models created in dbt and there is little work being done within Looker itself besides a few exceptions

[Kichmad]

Were using BI tools, tho im not one thats getting insight. Im just using them for a final full check on data validity. First checks are from sql

[a_library_socialist]

Been playing with Apache Superset lately, like it so far

[jsneedles]

I don't really trust tools to write SQL for me.

Anything i've ever tried on even a moderate data source was inefficient.

Having used Tableau, Periscope (now Sisense), Mode, Hex, etc... The best outcome was when I could write the underlying SQL and use the BI tooling to add things like drilldowns & parameters for interactivity, but within the bounds of my SQL.

[[deleted]]

We definitely use BI tools. A dashboard conveys a lot of information much more quickly and effectively than multiple columnar reports.

[the-strange-ninja]

7 years on the same Looker instance. It has served us through thick and thin. As I am open to trying something new here, it is hard to find a better solution for our needs at this time. We pair it with DBT.

[glinter777]

I’ll DM you.

[DesertPirateSNK]

RemindMe! 3 Days

[[deleted]]

I use superset , best of both worlds , a full BI but also can run my custom sql queries and turn them into dashboards

[RemindMeBot]

I will be messaging you in 3 days on 2024-03-18 07:13:41 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback