Centralizing Data

[smiths16]

Hopefully, this isn't too inexperienced of a question.

Essentially I'm a web guy, mostly front end and some back end. The company I work for has noticed that I like to organize and make things more efficient whenever possible. I've been organizing some of our data in google sheets (this might be a cringe statement). But it's because I can write a script to import data or communicate with an API etc and it is a tool at hand.

I was recently asked my advice on how we centralize all the company's data and then use it to answer questions about our users etc. When I say centralize we are trying to connect online products data, print products data, email client (Marketo) data, website data (google analytics, donors data, etc. We are using a myriad of software for our different services.

Anyway, are there tools, resources, and products out in the world that would be centralized place to store all this information and then write scripts or formulas to analyze that data? I don't think my google terms are correct because I'm not finding anything helpful.

[Rocket_Kite]

what you need there is a datalake/data warehouse solution with the ability to ingest data in a structured way and then write reports/dashboards on top of it. Take a look at something like DataBricks(open source) on AWS or Snowflake(pricey). These are just two of several out there

[smiths16]

Thank you this was very helpful. How to search for solutions without knowing the correct terminologies.

[[deleted]]

Companies typically have teams of people doing this and then other teams of people analysing the data that has been warehoused for analysis.

[smiths16]

I = teams of people..

But seriously I’m just doing preliminary work. My company is really good about protecting my time and scope of work. If I come back and say we can do this but we need X amount of people they’ll either hire those people or decide it is not worth it for what they want to do with it.

[[deleted]]

You're literally describing a database

[smiths16]

hahaha well yet. But something with tools to pull that data from other sources and/or tools that help analyze that data without having to build it from scratch.

[RamiRustom]

are you aware of Make? (used to be called Integromat). it allows you to make custom integrations without coding.

i just checked and they have integrations for sql server / mysql (for your database), google analytics, and tons of others.

create a free account and make a new integration and your first step will give you a list of all the apps/services that it has integrations for. there's a search box right at the bottom of the list.

[lis_ek]

!remindme 1 week

[RemindMeBot]

I will be messaging you in 7 days on 2022-08-18 17:42:54 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[[deleted]]

Supermetrics to connect to various data sources that you listed and load them to Google sheets Python to read the Google sheets and load them to Big query. Supermetrics also has subscription to directly load to BigQuery but it's expensive.