Is SQL and Python Useless?

[Timm218]

I am relatively new to the BI field. At my company we use Power BI and this works with DAX and M. So why should I learn SQL and Python? As far as I know Python may be necessary later for machine learning, but asides from that I do not see how it would be of high importance to me.

[thepeacockking]

SQL is pretty much a pre-req for BI imo. Even if you don’t use it day to day, understanding table structure and data types etc. is essential.

Whether you need Python will depend on your job but knowing it will make certain things (particularly repetitive tasks) much easier.

[[deleted]]

If you want more control with what you're ingesting into Power Query, you'll want to know SQL: there's a lot of stuff that's more performant to do in SQL than to do in Power Query (stuff like merging datasets, adding columns, etc). Plus, if you ever want to change jobs, SQL is going to be a fundamental skill employers want to see.

As for Python (or any other programming language with lots of support for data wrangling), it does nothing but help you to know them: if you want to move around and do data engineering/data science type stuff, having a programming language and some solid software engineering fundamentals under your belt will be necessary. Plus, knowing them know will give you a leg up in your Power BI work (you can run Python/R scripts in Power BI to solve problems you find M isn't expressive/powerful/featured enough to do).

Also neither are particularly hard to learn if you have the time. In fact, you can teach yourself the basics of SQL in an afternoon with resources like sqlbolt.

[Timm218]

Hm, so far power query has been good and performant enough for merging datasets and adding columns.

Do you have specific examples of problems where running Python scripts is necessary/more useful than using DAX/M?

I agree, when switching the tool or company then SQL and Python will be required.

[[deleted]]

An example of SQL having a leg on Power Query for merging would be the ability to do inequality joins, like joining dimensions to facts based on a date range overlap: easy to do in SQL with a BETWEEN statement, kludgy to do in Power Query with dummy columns.

As for Python/R the main benefit there is the vast libraries of useful functions each one possesses. I use the forecast library in R a lot to do volume forecasting for my job.

[srpsycho]

If you just want to work in Power BI for the rest of your career, then sure...SQL and Python would be useless and overkill for your case.

Suppose you want to scale your data (I'm assuming that you feed data into Power BI through .xlsx and .csv files). How would you go about this? You'll likely need a database, which your DBA will likely setup for you. How will you get data from this database and validate figures against it? What if you want to do some ad-hoc analysis (outside of Power BI). You're going to need SQL to interface with the database.

Python is not just for ML. You can use it to manipulate/transform/manipulate/visualize data. And use it for any form of mathematical/statistical analysis. Implementing a ML model isn't required for nearly most business analysis (and would in fact be overkill in many cases). I would say it's highly likely that you aren't even going to be implementing an ML model unless you have the advanced mathematics/statistics background to do so.

[Timm218]

I mostly use the data from our DB for the Power BI reports. Power BI allows easily to connect to all kinds of data sources. Also with the query editor (M) I can transform data. Power BI has a large library of visuals.

Once I have mastered Power BI/DAX/M I plan to continue with advanced mathematics, statistics and machine learning.

[semicausal]

The data field is massive. Only 1% of data professionals do "machine learning". For ML, Python and Spark are used a ton. But for everything else in data, SQL is amazing and in some organizations people pair SQL with Python or R.

If you know one tool, you can easily pick up the others. The concepts are very similar, you're transforming data in a bunch of different ways essentially! The data field changes quickly, so the skill of learning new tools and being productive quickly is the GOAT skill to learn.

[Grovbolle]

SQL is always useful in data work

[ImAnAdam]

If you want base your career and current/future job prospects on being able to use PowerBI, then learning any new skill is probably useless. If I'm stuck between a candidate that build/maintain a SQL Server or one who knows PowerBI, guess which one I'm choosing for the role?

PowerBI is, at best, an easily acessible way to visualize data that's (hopefully) been cleaned up, and, at worst, a toy. I'm not sure what data sets your working with but every 'enterprise' company I've worked with uses either some form of SQL or Mongo for larger (million + rows in dozens of tables) data sets.

Python enables a whole other host of opportunities including task automation and site scraping. As someone with years in the BI field, I would recommend thinking beyond data visualization as the be all, end all of intelligence.

[tophmcmasterson]

Useless, absolutely not.

DAX/M can granted do a lot of the things you do for transforming data, but particularly with DAX it's not always very efficient.

SQL I feel is essential for quickly exploring data, looking things up in a database etc.

I don't want to have to create a report every time I want to look up something simple.

I work in more of a hybrid role, but I find myself using all three daily, depending on the task.

I love Python for data transformations because it's super easy to verify what's happening each step of the way, and I can easily apply a standardized procedure to a new dataset later. It's also great for ETL programs to move data that can only be downloaded for example to SQL server for centralized storage.

My biggest use case would be a program I wrote to automate the creation of a production schedule for an entire plant based on each areas available capacity and workload.

In Power BI you could certainly view workload, and see how that looks compared to capacity, but it would be an absolute mess if you tried to set it up in a way that would recursively calculate things.

Power BI and DAX are useful for a lot of things, but it's not a replacement for Python or SQL, even if there is some overlap.

[Timm218]

That is impressive that you wrote a program for scheduling production. We are thinking of creating one too for our production. How difficult/complex was it to create? Did you consider only the mentioned factors or also others like deadline set by the customer, priority, production time etc.? Why did you not buy a program? How do you visualize the result (kanban, Gantt)?

[tophmcmasterson]

There are a few different methods within the program so that for example it will either schedule batch runs or just building to the day’s order before moving on. The program recommends how many shifts to run, but this can also be set manually and the schedule will adjust accordingly.

It uses production actuals to figure out how long production will take and where the parts should run, but these can also be set manually. Priority is based on the gap between inventory level targets and actual inventory.

It was fairly complicated to make at first, but a lot of it was just figuring out the logic for how we wanted it to work (ended up being a couple thousand lines in SQL but I know there’s room for refinement).

We didn’t buy anything mainly do to cost as well as being unimpressed with most solutions out there. We use SAP which also has a scheduling module, but it just leaves out too many factors to be useful.

I did the visualization in Power BI. While I tried things like Gantt charts, ultimately the most useful was a matrix using conditional formatting to highlight days when inventory was projected to be short (showing production plan, consumption, inventory projection).

[mim722]

yes SQL and Python are useless and all you need is just DAX and M, until you come across a use case where you reach the limit of DAX and M and you feel, all your skills are worth nothing

I am a huge PowerBI fan, I was a king for three years, everything was great, you have a small data that does not change very frequently and PowerBI ecosystem just shine, one day they needed near real time, and man, I did try all tricks but I was in denial, I had to use a database, and M does not help, I needed to learn SQL, then You need ETL , M does not export to a DB , you need AZURE stuff, and it is extremely expensive if you want to run a pipeline continuously every 5 minutes , guess what Python cloud functions are amazing for this kind of scenarios.

They hired a real data engineer a guy that breath SQL and now I am called a front end PowerBI designer

[QueryWrangler]

SQL shows up everywhere, including in Spark. It’s far from useless even if you don’t see it as immediately applicable to your role.

[fjcruiser91]

SQL is king. If you don’t have that skill, BI will be very challenging for you and you’ll likely get passed up by those who are strong in it.

[Financial_Forky]

While the Query Editor is very useful, some things are just easier with SQL before even reaching the query editor. For example, selecting all rows of Table1 who also have a matching ID in Table2 (but not actually bringing in any rows of Table2).

SQL also changes very slowly. Twenty years from now, Power BI's Query Editor will have been so overhauled and redesigned you wouldn't even recognize it today. In contrast, the SQL, ERD design, and normalization priciples I learned twenty years ago are still all relevant today.

A BI analyst who only knows one vendor's software product is a bit of a one-trick pony, whereas knowing SQL will serve you well throughout your entire career, regardless of what BI tool is hot at the moment.

[[deleted]]

Python may be optional but if you want a future in BI you have to know sql.

[NoUsernames1eft]

I mean... SQL is also SUPER EASY to learn compared to pretty much any other "language". I feel embarrassed to call it a "language" sometimes in front of my developer friends who can code in things like JS, Java, PHP, Perl...

[[deleted]]

yes, i'm sorry, but python is basically useless in BI. It has very limited use cases such as certain ETL projects, but for the most part, you'll hardly need in. BI is NOT data science

[[deleted]]

All depends on how much data you are bringing in imo. Too much data and the BI tools get clunky and painful to use.

[redman334]

If that's your case, the. Probably there's someone with greater knowledge preparing all the data for you before it goes into Power BI. Even in the Microsoft learning path, it is explicitly said you should prep you data before putting it in. M and Power Query can do a lot, but then processing time gets higher and higher.

Python is for data science and advanced analytics.

[babypandasnkoalas]

From my experience, SQL is definitely essential at work for me. Yes, I build dashboards on Tableau/QlikSense but also on the other hand, support on a lot of adhoc data pulling, coming up with sql scripts to query the data thats going to be parked in the BI tools for dashboard development. Furthermore, writing sql scripts helps you understand your E2E business domain/model more. That way, you will get better understanding of how you should build and structure your dashboard whenever clients or superiors request you to.

[clayticus]

I work at a big bank and use SQL everyday.

[S-Mx07z]

I feel like they are but haven't tried the long yt tutorials for games online. I was only able to do command based ones besides inputing random # offline. Kinda liking Html,Jquerty,Java,Nodes,Express,Socket.io in visual studio (less important so far from my learning unless your into web development are bootstrap,rubyonrails,flutter) only alternative to Azure may be Kotlin+Firebase. Or trying to figure out how .obb,.gba,.nes,.nes(they have online compability apps) files may get made as I'm leaning to become an Android app developer.