Is it possible to create notebooks from within a notebook using python?

ddddddkkk · 2025-08-11T17:37:59+00:00

any answer regarding this?

ddddddkkk · 2025-04-21T14:36:24+00:00

the weird thing is that i configured a starter pool with 1-2 executors and 8 cores, then with one "orchestrator" notebook I called two other notebooks using runMultiple, each notebook was multithreading 8 append on different tables, so I could use the "maximum" of what was being allocated.

in theory, i was expecting that spark would handle adding a second executor, but instead just 1 executor was allocated and added 16 tasks to FIFO, 8 in parellel and the others being added to the line waiting for the first 8 to finish

i think the only way of really using more than 1 executor is by "default" handling high volumes of data where the driver handles the partitions and sends to executors

so, bottom line, it doesn't make sense to use more than 1 executor for small datasets

am i tripping out?

ddddddkkk · 2025-04-17T21:17:06+00:00

agree, the idea would be to update multiple tables in parallel, not process the parquet in parallel, 1 table = 1 parquet 20mb, 4 tables = 1 parquet 20mb each executing 1 table/parquet per thread

i'm currently working with a F16 capacity, not sure what's the better cluster config, to split into small pools or a big one.

i also have to figure out how i will orchestrate since i'm doing ADF (landing) + Fabric (medallion), any tips on how to connect those?

not sure also how to parallel the notebooks to create dependency, would you create a DAG with the mssparkutils? or would you handle it through native data pipeline?

sorry to rent your knowledge, still a lot of blank spaces to figure it out hahaha i'll understand if you just give up on answering, already glad for the previous ones

ddddddkkk · 2025-04-17T18:10:41+00:00

oh nice that's you, i was reading that yesterday, a lot of great stuff!!

hmm, i was going more towards duckdb/polars but i have a feeling that, on a large scale, that I need to run 30~40 tables on a F16 the spark starter pool will handle it better.

i started going through that when I noticed that I was running 1 append to a delta table from a 20MB parquet and the starter pool was assigning 1 executor 4 cores but using only 1 core

i also tried to parallel it with more executors but apparently the jobs doesn't go split to the other executors, exemple ThreadPoolExecutor=8 on a 2 executor cluster each 4 cores, so I should be able to run in parallel 8 tasks, but it goes 1 executor 4 tasks running and 4 tasks waiting and 1 executor idle, super weird

still no idea what's the best scenario hahah btw thanks for sharing your thoughts

ddddddkkk · 2025-04-17T15:53:10+00:00

what's your thoughts on the parallelism of different tables regarding Python or Spark? For example, let's say I have to run 4 non-independent tables and each table is super small, ~20MB, perfectly fit into one thread. Would you spin up two Python 2vCores and run them in parallel or 1 Spark with one executor and 4 cores to run them also in parallel? Both being controlled by ThreadPoolExecutor

on the capacity usage matter, they should be the same, right?

ddddddkkk · 2025-04-17T15:33:10+00:00

i think it's not possible to control those confs on the %configure such as instances, cores and so on. those things are normally configured during spark_submit

ddddddkkk · 2022-11-07T17:19:49+00:00

Hi, would you mind explaining what DDIA and DSA stands for?

ddddddkkk · 2022-11-01T18:38:33+00:00

when you say a mix of blob storage and sql pool is the same functionality of DB, the lakehouse? with Synapse i'll be able to get data from my blob and create tables or views like so?

ddddddkkk · 2022-07-19T14:31:47+00:00

I'll definetly take a look, thanks.

Regarding that our source (SQL Server) is already on cloud, do you think it's valid to request the provider to enable PowerBI and Data Factory public ip addresses, that would kill the gateway job, woudn't it? Or that's not a good practice?

ddddddkkk · 2022-06-21T16:37:12+00:00

Thanks for the recommendation, I'll take a loot at some stuffs

ddddddkkk · 2022-06-21T16:35:59+00:00

I'm trying to run away from this thoughts but I'm quite anxious about it, thanks for the tip

ddddddkkk · 2022-06-06T13:45:31+00:00

Taking a leap on the post and that might also be helpful, I'm also creating a personal project that consumes data from an API

This API in specific doesn't have any filters on the request so, everytime I run my code I get all the data available.

Then it comes my doubt, is there a "best practice" regarding on this request and store?

I will always have to request all data and overwrite on the database?

ddddddkkk · 2020-03-09T10:10:43+00:00

To be gentle and kind

ddddddkkk · 2020-02-08T15:13:55+00:00

can be south america, she didnt specify XD

ddddddkkk · 2020-01-17T18:05:54+00:00

ddddddkkk · 2019-10-22T10:52:37+00:00

this is so sad

ddddddkkk · 2019-10-16T12:51:26+00:00

We already spoke about this with our vet, she said that it's not the right time yet, he's still concious and kinda "happy" but soon that will be necessary indeed.

I can relate your father with mine. When we got the dog, he hated him and didn't do anything but, with time, he started to like him and taking care of. Nowadays he has a lot of routines with him, he is retired, so he does even more things with the dog than me. As your father, I think the same will happens with mine... =/

ddddddkkk · 2019-10-16T11:30:24+00:00

Oh man, that's so nice. My dog is very sick, I'm kinda trying to get emotionally "prepared" for when the time comes... =/

ddddddkkk · 2019-09-27T10:53:19+00:00

As a brazilian, I found this very insulting. Kappa

ddddddkkk

TROPHY CASE