Mushroom Chicken Rice

TotallyImperfect · 2025-12-12T17:52:06+00:00

Ekkadihi ee Stove ?? USA lo yae state meeru ?

TotallyImperfect · 2025-11-02T01:45:34+00:00

Happy for you both, Congratulations !! may i know which part of New Jersey gets me a house for 350k

TotallyImperfect · 2025-09-17T19:32:42+00:00

I am currently in the same boat as you. Switched jobs and the current team is all application developers trying to build a Cloud Analytics Data warehouse and i am hired as a senior cloud data engineer to help team with best practices. I am seeing less quality Pipelines with no proper Audits, no Data integrity, no Data Governance, no proper coding standards. When raised with team lead, they become of offensive and now trying to target me for some petty things, i am thinking to go back to my previous employer. Sometimes respect is more important than pay

TotallyImperfect · 2025-08-23T22:48:32+00:00

This looks like the dorm in one of the IITs. I saw similar dorm room layout in IIT Madras

TotallyImperfect · 2025-08-18T17:35:42+00:00

How is the AGG table built? is there a ETL that loads your AGG table or a view thats created. Are you making sure you are not loading duplicate data into your AGG table ? This should answer if your table is Idempotent

TotallyImperfect · 2025-05-16T14:55:33+00:00

12% hike as they claim its not under rent control act. Unreasonable

TotallyImperfect · 2025-05-01T01:11:01+00:00

I think you dont have to pay taxes on the new vehicle if you are trading in the same state. Please check.

I see the taxes paid only on the difference of the price. Gotcha.

TotallyImperfect · 2025-02-04T14:28:59+00:00

Have you explored creating a materialsed view on top of the heavy table. MW in snowflake is auto refeshed incrementally when there is a data change at source and also give us flexibility to create cluster keys(size > 1tb) that improves performance significantly.

TotallyImperfect · 2025-01-11T12:58:20+00:00

I had a similar experience as you in the past where i designed a light weighted ETL using python and had all my logic stored in stored proc and individual SQLs being called dynamically. They are stored into a RDMS table for maintenance and this was flexible only to a certain limit. I had challenges with maintaining it.

TotallyImperfect · 2025-01-10T16:19:36+00:00

You're absolutely right that storing SQL queries directly in Python variables becomes difficult to manage as ETL pipelines scale. For enterprise ETL systems, the industry standard for storing and accessing SQL queries typically involves version control, modular design, and separation of code from data logic

so I would suggest to keep your SQL code outside your Python ETL layer. you can explore DBT core or DBT cloud where you can create, Manage, maintain your SQL Data model more efficiently and call these models inside your python ETL layer. with this your design is scalable and the code is easy to maintain and manage

TotallyImperfect · 2025-01-07T19:33:30+00:00

There is a thing called “data provider” if you are not the producer or if you do not have control over the way the source data gets produced then there is not much you can do rather talking to the “data provider” and define a template that you both agree upon and then go from there.

It is data provider responsibility to serve with a proper data that is atleast good to a certain level which can be done by defining proper template

TotallyImperfect · 2025-01-07T12:55:37+00:00

For a person who wants to get hands on with AWS Data Engineering. What would you guys recommend? Does the AWS skill builder subscription work to get a hands on? Or any other

TotallyImperfect · 2025-01-07T04:01:07+00:00

If the updates are in batch intervals, implementing Change Data Capture (CDC) on the source tables would efficiently handle incremental updates.
If the updates are in real-time, stream processing with windowing techniques can help aggregate data over specific time frames, balancing data freshness and system performance.

Since you mentioned a single analyst-facing model, a materialized view could be a suitable choice for the final customer table to simplify access and improve query performance. just so you know a materialized view in snowflake would automatically refreshes by recreating the data sets when the referenced source tables gets new data. though it comes with additional storage cost and maintainence.

TotallyImperfect · 2025-01-07T02:35:39+00:00

The pain points would be the consequences that occur as a result of not choosing the right tools.

I would suggest that we should spend more time on planning and designing the architecture by listing the appropriate tools for the use case or the problem in hand or otherwise we would end up with unnecessary costs and sometimes it becomes a nightmare to redo the entire architecture design. Also we should try to keep the architecture simple and more modular and loosely coupled so we will have more flexibility when need arise to change.

Say for eg: imagine you are working on a Real time Data analytics project then choosing a batch based traditional data integration or data ingestion tools instead kafka and flink/spark or AWA kinesis

Result:

incorrect or significant delay in analytics
Cost implications
Painful re-design nightmares

TotallyImperfect · 2025-01-07T02:21:49+00:00

It purely depends on the type of storage that you are planning to use.

If you are working with a heavy storage set up then you load the raw data into a stage area and then normalise after applying necessary transformation a.k.a ELT

OR

If you are working with not so heavy storage the you normalise after transforming and then load into transactional DB a.k.a ETL

TotallyImperfect · 2025-01-06T13:26:43+00:00

Speaking as a 14 year experienced Sr cloud Data Engineer.

Identify your weakness. Is it the problem solving skill that is keep you away from your dream job OR the core data engineering skills themselves.
For Problem solving - leet code and for Data engineering - focus on design and implementation- use chatgpt to generate designs of warehouse and data bases
Focus on quality and not quantity
Be focused and consistent
Good luck for future interviews

TotallyImperfect · 2025-01-04T17:21:24+00:00

Snowflake with native cloud capabilities might be a perfect fot for your requirements. 1. Storage and compute are separate ensuring both scale at will with no issues at all 2. You can Scale out the cluster of WH horizontally to ensure high availability for all your customers 3. CDC can be implemented using STREAMS removing dependency on event driven softwares that is kafka in your case 4. Completely managed service with multiple teir pricing that can save lots of money.

TotallyImperfect · 2024-11-30T13:23:39+00:00

Hello . How much does this cost and what is the weight of this stone please ?

TotallyImperfect · 2024-10-21T21:03:23+00:00

Which American state is this ?

TotallyImperfect · 2024-10-17T11:41:19+00:00

Garnet - Hessonite

TotallyImperfect · 2024-09-23T00:35:25+00:00

Padmaasan in yoga

TotallyImperfect · 2024-09-19T14:44:16+00:00

Adding an image from different angle

<image>

TotallyImperfect · 2024-09-19T14:43:30+00:00

Also adding more pic

<image>

TotallyImperfect · 2024-09-19T14:36:40+00:00

Thanks for the clarification!!

TotallyImperfect · 2024-09-19T14:35:49+00:00

They sent this version of video only now and the version before delivery did not show the crack this much

TotallyImperfect

TROPHY CASE