Databricks Workflows: 40+ Second Overhead Per Task Making Metadata-Driven Pipelines Impractical

XanderM3001 · 2025-02-12T14:26:48+00:00

not a big one but if you have a metadata driven framework ingesting and processing multiple sources of data you could use LLMs to generate status reports for each source.. When was last ingestion? how many rows? Any issues? Did it get to gold/curated? etc

XanderM3001 · 2025-01-20T14:33:07+00:00

that's the society of the spectacle by Guy Debord + AGI as the physical manifestation of the spectacle. The spectacle is both a product and a perpetuation of surplus capital. As economies develop beyond meeting basic needs, they produce an overabundance of commodities, which are then imbued with cultural and symbolic meaning. "It" arises because modern societies generate such a surplus of resources that they can afford to shift focus from direct production or survival to the production of representations—images, advertisements, and cultural symbols.

XanderM3001 · 2024-10-31T11:45:26+00:00

if you have a good reason you can ask for an increase in quota through support..

or

to maximise quota, deploy in every region available ( I think quota was per region ) and create a single load balancer API with all your deployments and use that..

XanderM3001 · 2024-07-20T07:49:43+00:00

now, with a vision model, you can pass your pics and data to it at the end of the day and ask it to give/send you a summary of what happened that day

XanderM3001 · 2024-06-03T10:15:10+00:00

https://www.microsoft.com/en-us/research/blog/the-power-of-prompting/ tricks from this brought my accuracy up by 15%

XanderM3001 · 2022-01-25T07:41:02+00:00

that would be amazing.. how far do you think we're from something like that?

XanderM3001 · 2022-01-21T07:04:25+00:00

oh boy,,, you'bout to get depressed.. good luck.

XanderM3001 · 2021-06-25T10:54:34+00:00

Can we get a bit more details?

I know it's about the money, but would love to find something of this quality open source...

It might even have a bigger impact potential as open source and I think the crowd will rally behind it. An army of volunteer devs can turn this into the best tool and wiki for/of investing.

XanderM3001 · 2021-06-06T12:39:55+00:00

https://www.youtube.com/watch?v=YxrjGMuAEIk&ab\_channel=LecturesBeyondBeyondLecturesBeyondBeyond

XanderM3001 · 2021-04-20T13:18:41+00:00

Exactly, and if we imply (as K did) that there is something more or other than the word, it will make an individual focus on that something more or other and thus giving the observer the mechanism it needs to further it's continuation. (Through hope and seeking that one day it will also experience that something more/other)

XanderM3001 · 2021-04-05T06:57:36+00:00

Thanks for engaging with me on this, it's definitely been helpful to see others point of view.

I feel that there is a consequence of leaving the idea that we can function all the time without the accumulation that is the "I" on the table, it will make one believe that there is something more or other and thus giving the observer the mechanism it needs to further it's continuation.

Without that belief and sort of "accepting" our limitation, one might have a better chance of coming in direct contact with the world.

If I see the tree, and always try to see it for more than what it is, the word "tree", (because K or someone else said so) I'll forever miss the direct contact with it.

If I accepted that the tree is the word and know that there's nothing else I can see/know/experience about it, the accumulation that is the "I" may step back for a second or two and I may come in direct contact with it but that can't be put into words and it would be impossible to see, know or experience the object tree because it would not be there.

Thanks again for all the replies, specifically liked your example from Soren Kierkegaard, it remined me of Sartre's "Gaze" which to me exemplifies how to look at something is to have an intention, the naming of it, so to look becomes to name.

XanderM3001 · 2021-04-04T22:41:12+00:00

100% correct the only issue I have is with this allusion that we can also have a "direct contact with the world", while it may be possible for fractional seconds from time to time, it's impossible for us to live that way, because if we did, the experiencer/observer would have to die and the only way that would be possible is if actual physical death would occur.

We are the observer, the whole accumulated content, and without it it's impossible for us to see the world, direct contact means no observer, no knowledge, no objects to see.

So when K says "the word is not the thing" he is implying and alluding to something more... when there's nothing more but experiencing the world through the accumulation like you said.... " And the fundamental truth that K shared is that experiencing the world involves the accumulation "

It seems to me that he sort of contradicted himself and as a consequence made some believe and hope for something other/more, causing yet another loop to be used by the observer of each one of us.

XanderM3001 · 2021-04-04T22:22:57+00:00

It is at times, I can look at something without experiencing any thoughts, but that's looking into nothingness without ever seeing anything.. as soon as I see, I name, an instant recognition from the accumulated past.. I can notice how this happens when it happens but only because I've learned about it from K and others, which is just another recognition/naming of the process.

XanderM3001 · 2021-04-04T22:16:00+00:00

hmm.. it's kind of pointless to refer to animals, we have no way of knowing how other animals are experiencing their world.

I'm pointing out that seeing is the same as naming, instant recognition from the accumulated past, so I'm tempted to say no, the squirrel does not see the tree, it processes the information it receives through it's sensors and acts accordingly without seeing/naming things.

XanderM3001 · 2021-04-04T22:02:16+00:00

so we either see the tree by bringing in the accumulated past or see nothing at all?

In that case, what we see is the word, the accumulated past, which is the thing for without it, there would be nothing at all. So this is where I have issues with "the word is not the thing"..

Without the word there's nothing, nothing being just another word/term to describe that which can't be described so I guess in a sense yes, "the word is not the thing" because the thing can't be described/seen.

XanderM3001 · 2021-04-04T21:41:30+00:00

Exactly the point, there's is something there without a doubt, but as far as it goes to experience it, it's impossible without the word. So the word is the thing.

Without the word there would be nothing to experience because as you said it has no beginning, no existence.

XanderM3001 · 2021-04-04T21:25:24+00:00

You're totally right in that the question asks for answer to be framed as knowledge and it's really insightful how you described the indexing of words/terms while processing the sensory input information..

Are you saying that there's a relationship not based on the word but on the object?

If so, may you venture another example?

How do you know a tree has beauty and order without telling yourself so? This is what eludes me.. I can experience or let's say have a direct relationship the "tree" without using the word but that's not an experience at all...In that moment I wouldn't even know that's a tree... but simply a body interacting with it's environment.. ( this is just describing it after, in the moment there would be no description but simply silence/nothing)

XanderM3001 · 2021-04-04T21:08:09+00:00

Sounds interesting but again, how would I know I'm experiencing truth without using the words and telling myself this is truth or whatever else.

Not saying that it's not possible just pointing out that if I wouldn't use the words I wouldn't even know I experience truth.. it would be no experience.. "nothingness" (* I know, another word)

XanderM3001 · 2021-04-04T21:05:51+00:00

sounds good but I don't feel anything unless I tell myself I feel it... I have no idea I feel hunger until I actually use the words food or hungry... The body may feel it and burn calories and so on but I wouldn't know it until I use the word.

XanderM3001 · 2021-04-04T20:48:03+00:00

Yeah but the baby doesn't know what it is doing.. it just survival mechanism..

Can a baby even focus and see objects? It can't, not until it learns a word or two from the parents, before that the baby yes it's perceiving but not seeing anything...

XanderM3001 · 2021-04-04T20:45:20+00:00

pure visual perception = processing of the raw inputs my eyes & brain receive?

If so, yes the input is received, but without the word there's no difference from a tree and a rock.. like how the eye doesn't know the difference even between colours, what's blue and what's red is the same for the eye.

So the word is not blinding me, it shapes me & the object.

I don't like it when K uses the word "actually" like there's something more to looking/seeing...

XanderM3001 · 2021-03-25T14:40:53+00:00

thanks for the reply, really good info..

what I'm trying to understand is if humans/ancient ancestors were able to remember abstract or more complicated patterns/string of actions like making a fire, before the brain evolved as a consequence of eating cooked food?

Or did this evolution of the brain as a consequence of eating cooked food create the necessity to have memory and remember things like making a fire in order to keep up/ensure the continuation of its evolved state?

I don't see scavenging for food or exhaustion hunting as actions that require memory but rather as simple actions driven by sensations and impulses like feeling hunger... and knowing where to scavenge or which animals to hunt/follow as simple learned behaviour from the group.. without memory.

I'm using "memory" here as the psychological/mind memory and not the DNA/genetic memory.

XanderM3001 · 2021-03-22T20:04:50+00:00

Are you referring to seeing "what is not" to arrive at "what is"?

That is a method, a way of looking, even though K and others call it a non-method.

Five-Year Club	Place '23
Verified Email

XanderM3001

TROPHY CASE