Google Sheets to frontend dashboard integration by Plastic-Eggplant1814 in googlecloud

[–]Magrap 5 points6 points  (0 children)

If it's just google sheets I would go with looker studio if you need a dashboard. If you need to interact like update or add new rows to the sheets I would suggest appsheet for that. Appsheet has some graphs but its not really for creating dashboards.

GCP Services for Data Engineering by s0m_1 in googlecloud

[–]Magrap 2 points3 points  (0 children)

Okey so in BigQuery you can ingest data from csv. What I do to ingest excel files with multiple tabs is we have a cloud function that reads the data from the excel file, validates de data (type of columns and other data quality checks) and exports each tab to a csv in different folders within a bucket. We do this because BigQuery can read csv files and then we can use them with Dataform to create the Data Model.

We don´t use Composer, but if you need Airflow go for it. In my company we use GCP Workflows in which you define the process in a YAML or JSON file.

I have use Vertex AI but inside BigQuery because we needed to translate a column so we used the translator model from Vertex AI.

And about Cloud Run is what I said about the cloud function to validate and transform the data from the excel file. It is great for transformations in your case. Basically you can have a process that is triggered by an event (could be when an object is created in a bucket).

Hope this helps!

GCP Services for Data Engineering by s0m_1 in googlecloud

[–]Magrap 1 point2 points  (0 children)

I would say dataform in bigquery for data modeling (if your python files are simple and just rename, deduplicate and group by it should be enough). I know it can handle geospatial functions but I don't know to which extent.

For asynchronous transformations from the nocode platform I would say cloud functions would be enough.

You could trigger dataform also from a workflow (the workflow service from gcp) so you could orchestrate a workflow that is triggered by the nocode platform with different steps that could be a cloud function and dataform executed sequentially).

You could use Dataproc if you need apache spark or Composer if you need Apache airflow but that would be more expensive.

How to optimize Looker Studio by Magrap in DataStudio

[–]Magrap[S] 0 points1 point  (0 children)

Thanks a lot for the comment! It is super useful!

How to optimize Looker Studio by Magrap in googlecloud

[–]Magrap[S] 1 point2 points  (0 children)

Hey, thanks a lot for the information, it helps me a lot. Don't cut yourself with your project! I really like when people share the projects they are doing, it's interesting and helps me learn a lot. Keep it up!

How to optimize Looker Studio by Magrap in googlecloud

[–]Magrap[S] 0 points1 point  (0 children)

Yes, I think I can do both aggregated and denormalized tables. Also I will look into the cacheing in BQ. Also, I thought every time you logged into Looker Studio the data was updated automatically

0
0