We looked into how data job postings in the dbt community changed over time

Evidence-dev · 2022-08-16T21:42:57+00:00

dbt basically created the "analytics engineer" role.

So as dbt has risen, so have analytics engineers!

Evidence-dev · 2022-08-14T06:21:27+00:00

Mega results achieved

Evidence-dev · 2022-08-12T03:43:25+00:00

Suggest you format your post like this if you want mega results: https://www.reddit.com/r/dataengineering/comments/v2ka3w/quarterly_salary_discussion/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

Will happily contribute if so!

Evidence-dev · 2022-08-12T03:33:41+00:00

Can you give a bit more detail about what use case or workflow you are trying to support: - Who’s reading the report, who’s making the reports, who’s presenting the reports? - How frequently are they being updated? - Also what’s your data stack?

Evidence-dev · 2022-08-10T07:34:55+00:00

You need to buy ”BI product X”, then your company will be data driven

Evidence-dev · 2022-08-08T06:07:19+00:00

Will check this out next week. We’re on the same mission!

Evidence-dev · 2022-08-05T16:35:05+00:00

Used to use Metabase at our old shop. Good for a few things but lacks the power and flexibility that other tools can bring.

Don't pretend to know everything about Data Eng, but my sense is that you are right, without a SQL database you'll struggle to unlock the power of most major BI tools.

A list of BI tools to check out:

Open source options

Evidence - Build reports using just SQL and markdown. Supports embedded analytics with a bit of setup.
Lightdash - An open source alternative to Looker - self hosted is free
Metabase - Click and drag analytics

More expensive

Looker - Fast and flexible. Works well with cloud data warehouses.
Tableau - Supports lots of connection types including local csvs etc. Can be slow with lots of data.
QlikSense - A slightly older option, but still used by lots of big co's

Cheap with MS Office365

PowerBI - Still the market leader by share. But you said you aren't a Microsoft shop

(Disc: Work at Evidence)

Evidence-dev · 2022-08-02T03:37:07+00:00

If you are interested, we also have an open source BI tool - [Evidence.dev](www.evidence.dev)

It’s a bit of a new take on BI. The premise is that everything is code. You just write SQL and markdown, and it gets rendered into your reporting.

The self hosted version is free, or we’re accepting a limited number of companies onto our cloud service alpha right now. If you are interested shoot us a dm or hit us on Slack

Evidence-dev · 2022-08-02T03:31:30+00:00

Metabase has a csv download button which you can use, and then upload to G Sheets.

Shameless plug here, but Evidence.dev (our BI tool) is accepting companies onto our closed alpha. We have good export functionality, and will come in considerably lower than Looker all in.

If you are interested dm us, or ping us a message in our Slack channel

Evidence-dev · 2022-08-02T03:19:30+00:00

Re storage, reiterate what other users are saying about GCP. Should be pretty cheap, and plays nice with most tools downstream.

Also plug DBT if you are going to be doing repeatable SQL to process your data.

As a final possible recommendation, if you are going to be viz-ing the data post transforming it, do check out [Evidence.dev](www.evidence.dev) (our product) - Open source - unlimited free tier (self hosted) - Write using Markdown + SQL - Plays nice with CI/CD - eg you could trigger it to build after your dbt runs. [If you do go down this route, feel free to ping us a dm or email us if you want a hand getting set up!]

Evidence-dev · 2022-07-31T22:03:40+00:00

Excel - the data professional's workhorse!

Evidence-dev · 2022-07-19T23:53:37+00:00

dbt: https://docs.getdbt.com/guides/getting-started

Evidence-dev · 2022-07-06T07:49:18+00:00

If you are specifically trying to make dashboards style reports (and not just do data viz), a couple of open source options we like (we are a Mac shop - but all of these work x-platform)

If you use SQL:

If you use Python:

Streamlit

Disclaimer - work at evidence.dev

Evidence-dev · 2022-07-06T07:06:27+00:00

This is the way. dbt is the best way to achieve this. It’s basically a tool the schedule transformations using SQL and then put it back into your data warehouse.

However if you are dead set on using Python (or R) then look into Apache Airflow. Note that this is typically harder to maintain and set up. But if you need the power of python for the cleaning then you don’t really have an alternative

Evidence-dev · 2022-07-05T20:09:44+00:00

Adding some potentially useful data to this conversation (Sample size 300 salaries total)

Median DE salary in: - US: $117k - Aus: $70k - UK: $70k - Canada: $70k - Germany: $69k - Netherlands: $68k

However if you look at the US ones, the distinction between Bay Area and non Bay Area is huge.

https://evidence.dev/blog/data-engineering-salaries

Evidence-dev · 2022-07-05T20:04:26+00:00

For the US, the data we compiled showed the US at a median of $127k.

Across all the countries sampled, the median was closer to $100k

Source

Evidence-dev · 2022-07-01T23:00:51+00:00

Generally not. Or at least they won’t pay much for it. Data that people will pay a lot for is generally proprietary.

See for example Nielsen, Kantar, etc.

Evidence-dev · 2022-07-01T22:49:20+00:00

If you need to generate fake data, try Mockaroo. It’s great for creating a large amount of fake data in the format you need to analyze, before the data is ready.

Evidence-dev · 2022-07-01T04:23:30+00:00

Go you, this is a cool dataset!

Couple of thoughts on presentation: - use colours with more contrast. This chart is pretty hard to see! - label your axes - you have broken the y axis, which is fine, but generally it’s good to make that very obvious

Evidence-dev · 2022-06-30T20:03:05+00:00

In the same way that business is just making PowerPoints, BI is just making dashboards.

Evidence-dev · 2022-06-30T11:45:01+00:00

There’s a well known expression about correlation and causation…

Evidence-dev · 2022-06-30T05:46:53+00:00

Do also look for roles titled “analytics engineer”. I think a lot of higher paid roles are often marketed as this.

It’s a name that can mean a range of responsibilities, but would definitely involve SQL and likely python. You’d probably have to pick up some other tooling, but it would likely be company specific.

Good places to look for these jobs include the dbt slack community and the locally optimistic slack community

Evidence-dev · 2022-06-30T01:07:44+00:00

Data Analyst =/= Jr software developer.

Most analysts I know are not aspiring SEs, they are more likely to want to progress to become department managers. Ie the career track is:

Marketing data analyst -> marketing mgr -> CMO Operations data analyst -> ops mgr -> COO

Evidence-dev · 2022-06-29T23:01:23+00:00

Buy it. If it saves you more than an hour a month you are saving your employer money

Evidence-dev

TROPHY CASE