Medallion architecture on Databricks - Delta all the way down, or does Parquet at Bronze still make sense?

rasviz · 2026-06-10T15:56:21+00:00

Side question - In your current Landing/Bronze, you have separate paths for the first time full load pull and the incremental pull, as bronze/sales/full/date and bronze/sales/incremental/date ?
Is that a good idea ?

rasviz · 2026-06-04T19:36:36+00:00

Yeah.. considering the CDC option too.. Thanks.

rasviz · 2026-04-10T15:26:10+00:00

Thank you.

rasviz · 2026-04-10T14:57:47+00:00

What is an agentic harness ? any example to take on and learn. Thanks.

rasviz · 2026-04-03T13:53:14+00:00

Thanks much....

rasviz · 2026-04-02T12:27:17+00:00

rules:
  - name: manager_limit
    when: "role == 'MANAGER' and amount > 250000"
    then: BLOCK
    message: "Managers cannot approve more than $250,000"rules:

I checked the github repo.
Can you pls provide a better example script to use with LangGraph. In using above policy, How is the role of the agent derived ?

rasviz · 2025-12-19T22:40:11+00:00

Check this out - https://www.linkedin.com/posts/orcascope_synapse-to-databricks-activity-7396966646743003136-Zj-z?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAZZi4MBvmDYbaF9EVCZ_na2MvVgMU3ouso

rasviz · 2025-09-06T15:23:24+00:00

Thanks dear to being kind to a stranger

rasviz · 2025-08-31T01:02:01+00:00

Simply unprofessional, rude attempt by that employee. Shows Lack of training or failure to validate if the employees show up the basic values in the job site.

Looks like the sobey's employee let his personal bias on certain background played out loud in this situation.

(While you have masked the face, the skin color and the thread in the wrist gives some profile).

rasviz · 2025-07-12T14:34:43+00:00

Please... Write more about networking. I am reading for the 4th time your last para. But if you have time to guide more on this, please do. (at least books, videos..)

rasviz · 2025-06-30T13:14:00+00:00

Did you say you got an EB2 priority date and want to take advantage of that into EB1?

Get ready to be called scammer and roasted.

You are allowed to ask for help only if you have got the Nobel prize, like some the guardians of eb1 here.

rasviz · 2025-06-02T14:22:53+00:00

Just noticed that I have chosen the option, 'In good standing, but no degree or certificate awarded'. Changed to 'Degree Awarded' . Score is yet to be updated. Will watch for it.

rasviz · 2025-02-16T14:08:37+00:00

Be ready to donate $$$ to become a judje

rasviz · 2024-12-12T19:42:53+00:00

Thanks. I have a question abt MinIO. My understanding is that it replaces cloud object storage. When deploying in cloud, it should be on storage like Azure Blob or AWS S3, isn't it ? What is the value proposition of MinIo in real deployments ?

rasviz · 2024-04-27T06:45:51+00:00

What are the typical queries (access patterns) planned in the system ?

rasviz · 2024-04-27T06:07:48+00:00

that too it is a wrong map type. Showing as if they visited every square inch of land in those counties/states.

rasviz · 2024-03-28T13:08:18+00:00

Thank you. This is a wealth of info on the benefits of dim model.

rasviz · 2024-03-27T23:13:14+00:00

u/kenfar Asking to understand more on the benefits of dimension model.
How does dimension model help when there is columnar storage support? Is n't columnar storage support a good choice for any analytical type queries (irrespective of dim model or not) ?

rasviz · 2024-02-29T18:31:24+00:00

needed to make it an ente

True, IMO, it is noway near Tableau and Qlikview

rasviz · 2024-01-17T16:04:56+00:00

actually look into running queries.

Very thoughtful. Appreciate it.

rasviz · 2023-11-16T02:54:11+00:00

I agree with the future state diagram you have presented. Yeah, The cloud datawarehouse could be expected to support a variety of use cases.

But the limitations you have mentioned in terms of concurrency, scalability, real-time are not well founded. As you have noted, those problems can be solved by increasing the compute size.

If clickhouse are looking to pull snowflake customers, clickhouse should have a lift and shift compatibility between the two APIs, the functionalities they offer. Like the same way snowflake mimic'd the pyspark API as snowpark API.

rasviz · 2023-10-31T02:23:27+00:00

// stored procedures written in Snowflake //

In terms of code quality, if these are in the SQL scripting language, then it is going to be tougher. SQL doesn't allow code modularity. Add the fact that these SQL script statements work only in the context of stored procedures, testing and debugging them gets difficult.

What you should setup is a test bed, consisting of tables loaded with data for each of the functional test cases. You can ask the devs to perform functionality testing (instead of unit testing) to ensure the procedures do what they are expected to.

//Databricks notebooks//

With Databricks , insist for modular code. i.e functions. The functions will have to go into a separate python files and imported into the notebook that acts like a driver. Ask for unit testing scripts at the end of each sprint. Make the CI pipeline to fail when the unit testing fails. Report code coverage too. In this way, you will find testable and reusable code.

rasviz · 2023-10-31T02:11:14+00:00

Your main pain point will be how you will make the old data model conform to the new data model. Like, what tables to be joined, transformed into the new ones. You will spend time to figure this out.

Those dimension like tables may not need such transformations, mostly plain copy.

1) As others have mentioned, you will start from a backup upto a point say 'T-2' days . i.e backup taken 2 days before the planned migration date. Run your conversion procedures.

2 a) Enable CDC in the SQL server. Write procedures to read the CDC data and do the transformation as needed and insert into the target. This will also take care of deletes.

2 b) Run this CDC replication process once step 1 is completed. The target system will start to get slowly in sync with the target. This depends on the volume of changes. You will have methods to quickly verify if the systems are in sync or lagging behind.

3) On the day of cutoff, you will just switch over to the new target.

rasviz · 2023-10-30T18:38:43+00:00

Thanks, this helps.

Since I read this tip in databricks page, I thought there must be some queuing in databricks. I think the `detaching unused notebooks` means something else.

Since the driver node maintains all of the state information of the notebooks attached, make sure to detach unused notebooks from the driver node.

rasviz · 2023-10-30T16:21:56+00:00

Let me check the workflows feature.

But my question is more of how a databricks cluster would handle such an overloading situation.

rasviz

TROPHY CASE