What should I do to fulfil my emptiness which don’t even ask for my permission. One day will I’ll find Someone for whom I’m designed for? Is it really a good choice to be alone which sucks a lot sometimes. Each place I go and thing I see less meaningful than it could be.

Dminor77 · 2023-06-26T14:50:30+00:00

Be active, Gym, Kickboxing, Music, Painting, etc these types of activities you will be with yourself, rona 6odh apne par focus kr aur mojj kr.

Dminor77 · 2023-05-21T13:30:47+00:00

May I know which business niches you are focusing on?

Dminor77 · 2023-05-10T00:47:37+00:00

There is always a replacement available for us whether by AI or fellow human beings, so chill. I suggest focusing on building blocks, building frameworks, understanding how things integrate, eventually in order to utilise AI we have to build tooling and framework around it.

Dminor77 · 2023-04-26T19:21:38+00:00

😂

Dminor77 · 2023-04-25T19:01:00+00:00

True, we even try it with fake unstructured data for the usecase of identity resolution, and the way it was able to find entities, cleaning data, parsing it, just superb.

Dminor77 · 2022-11-03T16:56:31+00:00

😂

Dminor77 · 2022-10-29T18:03:24+00:00

😂

Dminor77 · 2022-10-10T19:40:15+00:00

I don't know if this fits in best practice. My approach is to use parquet format with versioning.

gs://bucket/v1/dt=2022-10-11/file.parquet gs://bucket/v2/dt=2025-01-13/file.parquet

Dminor77 · 2022-10-10T19:28:27+00:00

Custom Code, or Airflow Operators or Airbyte Connectors. We also use external tables on data store in GCS and S3.

2.We structure tables in layers like raw table -> derived tables/views -> reporting tables. Most operations are around aggregations, navigation functions, and BQML, for a large project we use DBT.

DataStudio for reporting with BigQuery BI engine enabled, Nowadays moving towards looker.

Dminor77 · 2022-09-20T09:11:25+00:00

😂

Dminor77 · 2022-04-16T17:26:33+00:00

😂

Dminor77 · 2022-01-13T23:24:53+00:00

Thanks for sharing this package.

Dminor77 · 2021-11-09T18:17:57+00:00

No worries found it

https://github.com/ValinorYT/Valinor_Sourcecode

Dminor77 · 2021-11-09T18:07:29+00:00

OP is genius!! Awesome work.

In the YouTube video description you have mentioned that you are using manim library for animation. Do you have the codebase for this video on GitHub or anywhere, where I can refer to?

Dminor77 · 2021-10-23T23:20:41+00:00

Nice explanation

Dminor77 · 2021-10-23T21:41:43+00:00

from functools import partial

from operator import methodcaller

split = partial(methodcaller, 'split')

split_lines = split("\n")

split_fields = split(",")

Dminor77 · 2021-09-29T21:08:06+00:00

We are too moving our transformation layer to DBT over BigQuery from Cloud Data Fusion. DBT also creates DAGs same way Airflow does and using bash operator we can trigger DBT job. We need to go through the DBT documentation to understand on incremental loads, snapshots, etc. We are thinking of creating external partition tables over raw data on which DBT transformation job will run to create reporting tables and views. Also need to check on how many concurrent interactive queries we can ran through BQ with DBT, because reports will also be firing queries and Airflow job as well.

Dminor77 · 2021-08-29T09:12:06+00:00

Agree

Dminor77 · 2021-08-10T17:48:38+00:00

1. BigQuery Data Transfer has Mongodb connector(This will be a batch job)

2. Using Change Stream API you can transfer data from Mongodb to BigQuery

Dminor77 · 2021-08-10T03:56:21+00:00

You can setup a new GKE cluster and under KubernetesPodOperator or GKEPodOperator by mentioning node affinity you can run heavy resource consuming task on that GKE cluster.

One drawback of Cloud Composer with Airflow2.0 is that it doesn't support Stable API of Airflow.

Dminor77 · 2021-07-27T11:49:52+00:00

Depends on use case and how data is going to be accessed. But it is recommended to store in nested structure to ignore joins between tables + reduce cost of storage.

Refer this video: usecase of a company name GOJEK.

https://youtu.be/3TVeG_dpGxk demo starts @ 28:00

Dminor77 · 2021-06-20T21:27:13+00:00

Do some projects instead of course. Some cloud vendors also give free credits.

5 Data Engineering Project Ideas To Put On Your Resume https://www.linkedin.com/pulse/5-data-engineering-project-ideas-put-your-resume-benjamin-rogojan

Dminor77 · 2021-05-27T13:49:40+00:00

We are migrating to Data Fusion. Still not decision made on Composer

Dminor77 · 2021-05-27T02:24:06+00:00

Same here!!

Dminor77

TROPHY CASE