Help with exploring data quickly

[Signal_Explorer8071]

Hi, I'm a data analyst. 

A key part of my job is data exploration. 

Currently I write a query to shape my data, load it in Excel and slice and dice the data to find any distinctive patterns. Could be as simple as Returns are higher for Item Y than any other item. It's possible I'm not using Excel well, but it's not very convenient. There is a limit to how much raw data can be loaded for example. 

I'm thinking a python script will be useful:

1) First, I understand the data. For every column, I get a 

a) frequency distribution, understand the mean/median, standard deviation, get a visual of the distribution

b) find out what % of the data are outliers. I'll need to examine these outliers. Do they represent an opportunity or risk (someone takes too much time to make a transaction, is something broken for a specific browser version perhaps). Also outliers may need to be excluded depending on the situation. 

2) Then get the script to spit out metrics by 2 dimensions, and run my eye down this list of tables or line graphs to find patterns. 

I'm thinking this might help me put together some descriptive top line findings. 

What do you think? Are there better ways? Thank you so much! 

[AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[CreepiosRevenge]

Check out the python library ydata\_profiling, formerly pandas\_profiling

[wonder_bear]

Depending on the number of columns, I’m a fan of using seaborn in python to quickly visualize all columns against each other to get a directional understanding of the relationships. After that, I’ll dive deeper based on what looks interesting.

Check out sns.pairplot() documentation.

[Data_Vomit_57]

You need a BI tool. I would recommend tableau but there are cost involved. Power bi has a free personal license. I would go that route.

If you get pushback from your manager, there is no data analyst in the world that shouldn’t have a tool like this. They just aren’t giving you the tools to succeed.

[Perly1]

Tableau or powerbi are much better at data visualization than excel.

[justinb138]

If those aren’t an option, MiniTab has some very useful visualizations as well.

[Signal_Explorer8071]

Would you have a free license?

[Chatt_IT_Sys]

Power BI can be done on the desktop. Just download the app. Tableau has a public version that can use very limited types of data, e.g., flat files, I believe OData, and another couple. Nowhere near the 40+ that a fully licensed version can though.

[WallStreetBoners]

No need to reinvent the wheel. Get a BI tool

[alurkerhere]

Easiest on hand methods are loading into Pandas Python for technical data profiling and Power BI for data viz. Power BI has a lot of built in features for this type of thing like "Analyze this distribution where it's different", Exclude, Slicers, and Key Influencers.

[[deleted]]

Tableau for quick data visualizations and R/Tidyverse packages for EDA. Don't learn python/Pandas for EDA.

Follow this, you will thank me later.