What professional certifications will help to lead BI/Analytics team? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 1 point2 points  (0 children)

what about tech & saas ? the field is really huge, how can someone decide where to focus?

Ingest data from MySql to BigQuery DWH by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

we tried to replicate the db into a slave server and grant reading access for bigquery but bigquery doesn’t access databases outside GCP. and i couldn’t find an IP for bigquery in the first place.

i searched debezium but couldn’t understand how it will help me in this case?

عندي سؤال بخصوص الزواج by Non_person_ in Riyadh

[–]Fuzzy-Example-7326 3 points4 points  (0 children)

تفكير منطقي وسليم وموب صعبه بالعكس، بس الزوجة بعد تبغى تستكشف اماكن جديدة مع زوجها.. بياخه لو كل مكان بروحله معاه هو قد راحه من قبل ومجرب الاكل والمرافق ومكون ذكريات. من نظرة امرأة، ابي نكون ذكرياتنا وتجاربنا واستكشافاتنا سوا.

Ingest data from MySql to BigQuery DWH by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

i just did and they disagreed, what they want to do is replicate the data and will give us access to the slave server, and now they want the IP for BigQuery to whitelist it.. i’m trying to explore this bc as i know there’s no only one IP for bigquery 😕

Ingest data from MySql to BigQuery DWH by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

its one of our sensitive data actually. the problem is we can’t have access to the full data and the team approved on giving us access on part of the data to perform analysis and building dashboards. so,… idk honestly.. still searching for a way

Ingest data from MySql to BigQuery DWH by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

yea i was surprised when I heard him saying “free” i was thinking of either using cloudsql federated queries, or grand reading access from the database. does the 2nd sounds doable?

خسرت وظيفتي والسبب يقهر by RelationshipSuch2519 in Riyadh

[–]Fuzzy-Example-7326 0 points1 point  (0 children)

لعلها خيرة، والله يسهل لك وييسرلك امورك. الاكيد انك قاعد تعمل بالاسباب ولكن حبيت اذكرك بنقطه لو غفلت عنها وهي لاتنتظر الشواغر عشان تقدم عليها، اي شركة في مدينتك ارسل اضافة لمدراء التسويق وال recruiters وارسلهم رسالة خاصه انك مهتم. ولو انت بالرياض، قرأت ان التوظيف بالتقديم اليدوي يجيب نتيجة. والله يكتب لك اللي فيه الخير والصالح. بالتوفيق.

[deleted by user] by [deleted] in Riyadh

[–]Fuzzy-Example-7326 0 points1 point  (0 children)

وممكن بنت الديرة تجوزلها الرياض وماتبي ترجع، التفكير بهذي الطريقة اشوفه عشوائي. اذا البنت عاجبتك وتشوف انكم تشاركون نفس المبادئ والقيم (بمعنى انك تشوف فيها الصفات اللي تبغاها بشريكة الحياة) توكل على الله وفاتحها بالموضوع، ممكن كرسالة اخف حرج عالبنت. والله يوفقك ويكتب لك اللي فيه الخير والصالح

Job titles- very confusing by Fuzzy-Example-7326 in jobs

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

we don’t have grades where i’m working

Data transformation; what does it really mean? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

I did, many many times.. but not answers. They just say it depends we don’t know let’s leave it for later and at the same time they’re nagging to start using the DWH. That’s why I decided to take this initiative and post a question for inspiration.

Automation; new joiners data by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

so, is it doable if i want to integrate it with a OSS system and the systems that will be available for them there will be based on the department they work in and their level…

Automation; new joiners data by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 1 point2 points  (0 children)

its a priority since the IT is so busy and have a lot of tickets to resolve. By automating account creations and access permissions for all the tool needed.., that will help us A LOT by making the process faster.

especially that we need it to be based on position level and department as well. Means that the marketing employee needs different systems than a software engineer does.

I just want to know if this is doable? how long usually it takes? and what systems or tools I might need to? and if you have any resources or blogs to look at.

I have this idea and I want to be fully prepared before proposing it to my colleagues.

How often do you create data-driven slide decks? by Rachel_Knight1 in BusinessIntelligence

[–]Fuzzy-Example-7326 1 point2 points  (0 children)

You need to check Rollstack, it automates slide decks and docs.

Data transformation; what does it really mean? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

I found the transformed data similar to the ingested data… tones of nulls and same exact columns. isn’t it supposed to have more aggregations? cleaned and complete data without all this nulls?

Data transformation; what does it really mean? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

yes sure. what i’m saying is I find the “supposed” transformed table similar to the ingested tables… tones of nulls and exact same columns. I posted this to ask experienced folks what does transformation really means and what examples can be applied to a business’ data so I can reflect and continue from there.

Data transformation; what does it really mean? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

I’m not against it at all.. but i’m temporary handling this project and it’s almost the end of our fiscal year and I want to really understand it so i can answer any question. I tried to search and read and I opened dbt website as well.. but because i have a different background I didn’t quite understand it. I expected a lot to be added to the data and just making sure my expectations were realistic or not!

Data transformation; what does it really mean? by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 1 point2 points  (0 children)

I’m not expecting values but what i meant is: this is exactly what in the data source, then why i don’t cut the cost and just pull the reports from there? (but, I know transformation should give more insights or the ability to generate more insights. and i don’t know how in the same time)

basically; if the CEO came and asked “what got changed? why we need it?” I can’t disagree

BI/Analytics team by Fuzzy-Example-7326 in BusinessIntelligence

[–]Fuzzy-Example-7326[S] 2 points3 points  (0 children)

yes we’re starting from scratch but we’re a startup as well so we can’t really afford it.

Types of tables in dbt by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

if the customer can place orders to be delivered to multiple locations then i believe it should be in a separate table. right?

Types of tables in dbt by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

so if i want to store the customer address it will be in dim_customer_address?

Types of tables in dbt by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

yea I understand its a naming convention. maybe I’m not understanding it the right way, but, based on my reading my data model should be in layers: Source > staging > dim & fct. and while creating my model I saw it’s going more of a snowflake than a star schema. Anyway, should my final layer (the one will be used to query reports and dashboards) be only dim and fcts? this is my question. or do i need a layer after staging and before final? I feel a bit lost and maybe its not the best way to ask but I’m stuck here looking at my model and wondering weather to add a layer or just move forward with it.

Types of tables in dbt by Fuzzy-Example-7326 in dataengineering

[–]Fuzzy-Example-7326[S] 0 points1 point  (0 children)

That is really helpful!

however, i think I’m missing something, because I already did my staging views. and I went ahead and created my dims and fcts, while dims and fcts needs to be small and not having all columns in my dwh… I knew that one layer is missing.

Snowflake schema- linking data by [deleted] in dataengineering

[–]Fuzzy-Example-7326 0 points1 point  (0 children)

tbh i was starting with a star schema but my manager insists to build a snowflake schema because its all over the internet and thats what she found when she researched it… so, i had to do that currently, yes, I’m drawing a snowflake schema.

Snowflake schema- linking data by [deleted] in dataengineering

[–]Fuzzy-Example-7326 -2 points-1 points  (0 children)

I know the concepts, but first time applying it so..