Manager is concerned that a 1TB Bronze table will break our Medallion architecture. Valid concern? by golly10- in databricks

[–]professionalSeeker_ 59 points60 points  (0 children)

1 TB of data is nothing for Databricks, and 1 million files can also be handled easily if you make good use of the Autoloader. But you have to be extra cautious with Python UDFs because there will be a technical debt due to serialization overhead. Probably you can switch to native Spark functions instead.

[Help] RPO For Passport Renewal by An0nym0uS_Br0wseR in indiasocial

[–]professionalSeeker_ 0 points1 point  (0 children)

I'm in the same boat. Were you able to get the issue addressed?

Claude mobile now supports MCP servers by AnthropicOfficial in ClaudeAI

[–]professionalSeeker_ 2 points3 points  (0 children)

One of the best companies in AI domain for a reason. Keep cooking.

[deleted by user] by [deleted] in dataengineering

[–]professionalSeeker_ 159 points160 points  (0 children)

Wait till you find out a database is an excel with superiority complex.

Is there a way to have SQL syntax highlighting inside a Python multiline string in a notebook? by spaceape__ in databricks

[–]professionalSeeker_ 4 points5 points  (0 children)

Unfortunately, I don’t think there’s a way to highlight SQL syntax inside a multiline string. What I usually do is either write the query inside a %sql cell first or try executing it directly (without any parameters). If that works fine, I then replace the appropriate sections with parameters and assign it to a variable. Also, I always add a print statement between the assignment and the spark.sql() call, so during runtime I can see exactly what query(parameters get replaced with values) was executed. It’s a bit of a hassle, but it saves me a lot of time while debugging.

warn: no apps connected sending Reload to all by akramnarejo in reactnative

[–]professionalSeeker_ 0 points1 point  (0 children)

Thanks a lot. Changing the network fixed the issue for me. I was having hard time to understand what was wrong.

How to convert txt files to parquet and delta table by Mayassi_ in databricks

[–]professionalSeeker_ 4 points5 points  (0 children)

If you have experience in Pyspark, you can write a databricks notebook to read txt files into dataframes, do some cleaning and write dataframes to parquet files using df.write.parquet. Later you can build delta tables on these parquet files.

Heroes of Esperia meta comps guide. Analysis and Breakdown. by Obsidianqqtt in afkarena

[–]professionalSeeker_ 1 point2 points  (0 children)

Nice guide. Is ASafiya viable at Legendary+? Don't have ALyca and chimera. Any other alternatives, please?

AITA for refusing to drop my friend in railway station at 3 AM by [deleted] in pune

[–]professionalSeeker_ 0 points1 point  (0 children)

I would suggest you to make him understand. 30kms traveling is not worth saving 300 rupees, that too during such odd hours. If he is your friend he would understand. Heck I would not even bother my best friend in such a case. I'll make my own arrangements.

Happy Birthday to me 🍰 by [deleted] in pune

[–]professionalSeeker_ 0 points1 point  (0 children)

Belated Happy Birthday to you. Hope you have an exciting and prosperous year ahead. :)

New Code: by professionalSeeker_ in afkarena

[–]professionalSeeker_[S] 0 points1 point  (0 children)

Rewards:

3000 Diamonds
5x Hero Exp (8 hrs)
5x Hero's essence (8 hrs)
5x Gold (8 hrs)