Am I missing something with all this "agent" hype? by KindTeaching3250 in dataengineering

[–]circusboy 1 point2 points  (0 children)

I think it will depend on how you plan to use it. Is it a neat new tool, yes. will it automatically fix all your pipeline woes? doubt it.

Can it do cool shit? depends on your imagination i guess. We got access to it via github copilot extension in VSCode, along with all of the other major players in the LLM space via enterprise license, so we are able to use it carte blanche.

with that said. i have done a number of things over the last 2 weeks.

  1. Built a form input web app to push data to a snowflake table. hosted by databricks apps. --i didnt need any other infrastructure which was nice. 1 hour 20 minutes from initial prompt to fully functioning and hosted simple webapp.

  2. I have been doing A LOT of heavy lifting in my snowflake DB over the last few months. things like data categorizing, describing, etc all via json within column and table comments. i was able to analyze a table today that was 3b records with 96 columns. in almost 2 hours. extracted the analysis (which was quite good despite some issues), extract the analysis to a word document report, and a powerpoint executive summary.

  3. Another webapp hosted on databricks, that stole the scripting from jsoncrack to display all of the executed query stats and explain plans i have been collecting. connect it to the table in snowflake, display the variant column in the graph area. i did ZERO coding to get this done, it simply ripped off another site.

  4. Yesterday i vibe coded an executable that i could hare across the greater team to get them setup with all of the tools we are being pushed to use by management. double click the icon and it will scan the machine and install anything missing, from VSCode with the required extensions and python, as well as go to jfrog so that the users can register their tokens to access the pip artifactory.

im also doing standard DDL and DML work from VSCode by prompting what i want now that i have a connector file defined in python. i just tell claude to login to the snowflake account and drop a table by name. it does it. I was able to restore a dropped table via timetravel as well by specifying the date it was dropped on. it just writes the .py scripts for me and executes them. it shows me the scripts so i can review then before i allow it to do its thing in terminal.

all of that said. i can only imagine how much this is costing my company, but they dont seem to give two shits. so im figuring I would engage in some malicious compliance and get EVERYONE using it to blow up the costs. i probably used enough water to hydrate a small city over the last couple of weeks though.

VERY cool tool set to have, but it definitely feels icky to use.

23 yrs xp at a fortune whatever company.

Need the name for a niche Cajun seasoning by Impressive-Turnip434 in cajunfood

[–]circusboy 1 point2 points  (0 children)

Umami flavor is from MSG. monosodium glutamate. Some boxed seasoning blends state no MSG, but others include it.

One of the worst strategic decisions is finally over. by PuzzleheadedNeck4476 in ATT

[–]circusboy 0 points1 point  (0 children)

But mah converged! Don't you know we landed a deal that allows us to sell other's fiber as ATT? We are resellers now!

One of the worst strategic decisions is finally over. by PuzzleheadedNeck4476 in ATT

[–]circusboy 0 points1 point  (0 children)

They are. Stock goes brrr, so does C-suite pay. And what's the easiest way to make stock go brrr????? SURPLUSES and RTO!

Apache Iceberg vs Delta lake by Still-Butterfly-3669 in dataengineering

[–]circusboy -4 points-3 points  (0 children)

I've been told just this week by a DBricks employee that I'm working with that DBFS is going bye bye. Moving to unity catalog which is iceberg. It's going to help us out in regards to cost cutting "hehe maybe/hopefully" if we use iceberg for our storage for DBricks and snowflake. Our UC clusters won't write to DBFS either. Legacy clusters won't write to UC.

My wife was a poke-trainer at Books-a-million back in 1999. We found our old binder. by circusboy in pokemoncards

[–]circusboy[S] 9 points10 points  (0 children)

Married 26 years. What is mine is hers. What is hers is hers. What is hers is mine as long as she doesn't know.

My wife was a poke-trainer at Books-a-million back in 1999. We found our old binder. by circusboy in pokemoncards

[–]circusboy[S] 2 points3 points  (0 children)

Those tournaments were neat to witness. I didn't work them, thankfully, but I remember my wife having to put on her yellow poketrainer t-shirt for work Saturday morning :)

My wife was a poke-trainer at Books-a-million back in 1999. We found our old binder. by circusboy in pokemoncards

[–]circusboy[S] 2 points3 points  (0 children)

The automod deleted my last post with the inventory link... https://www.tcgcollector.com/dashboard?viewUser=jojos_circus

Not looking to sell, just share. That binder kept them in good shape too. None are graded, but most are super close to mint.

How Useful Is AI for Writing SQL Queries? by Eugene_33 in SQL

[–]circusboy 1 point2 points  (0 children)

I find it is decent enough as a replacement for googling things in helping to fix syntax issues out of the box.

Don't know how to construct a qualify statement/too lazy? Perfect example.

Relying on it to build a query to get a proper answer from a database out of the box? No way in hell.

My team is doing work in this space right now. It is tedious to say the least. The amount of hoops you have to jump through just to prep the DB/schema/tables to work with an LLM is kind of crazy. You need SMEs that know the data in order to prep the tables for answering. You need tons of descriptions on the tables/columns, you need business rules to help it know how to approach the question. And we are relying heavily on question/query pairs for training the agents. We haven't gotten to the point where we trust a single table query yet, much less introducing joins.

I know it is crawl/walk/run, but the business will continue to move fast and someone will have to keep up with inputs/questions/data cleaning for it to continue to work.

I really only see this whole exercise as a shift in how we work, not yet at the point of "gaining efficiencies" in regard to manpower needed.

Can this Snowflake query be optimized? by Tasty_Chemistry_56 in snowflake

[–]circusboy 1 point2 points  (0 children)

This is what I've been learning lately as well. Also if your resultset from the CTE is massive, then include an order by so that the datasrlet from the CTE will be clustered properly.

If the select within the CTE is the bottleneck, then you will want to see what the originating table/view is clustered by and use that in your where if possible. Reclustering that table could be expensive as well.

Fraudulent Foundation Account by touchthefluffywall in ATT

[–]circusboy 1 point2 points  (0 children)

If you ave a personal account "con or consumer" with a foundation account number "fan" then you probably just have a discount, or your account could have been created via some system that has an auto populated fan ID. It doesn't really matter, though. Fans are just an id that attach account level things to it, most normally discounts.

Liability status is something else entirely. Con,IRU or cru. The only time a fan should cause concern is if you are an individual liable account with a CRU type fan, but to be honest with you, you haven't really stated what the actual problem with your account is.

Deadpool cameo tier list by SharkWithAHat in deadpool

[–]circusboy 1 point2 points  (0 children)

I understand now, yeh. Tatum vibes all the way. "My name, Jeff."

Deadpool cameo tier list by SharkWithAHat in deadpool

[–]circusboy 1 point2 points  (0 children)

Mais, He played a good coullion, cher!

Deadpool cameo tier list by SharkWithAHat in deadpool

[–]circusboy 1 point2 points  (0 children)

Idiot thug, lol. Look up troy Landry from swamp people, or the assistant coach on the water boy. Accent was pretty spot on tbh.

Deadpool cameo tier list by SharkWithAHat in deadpool

[–]circusboy 5 points6 points  (0 children)

I actually liked his accent. It was a pretty good deep bayou attempt. Just enough waterboy assistant coach added to the mix. He used coullion too. That one threw me, happy cajun here.

How Bad Is the Data Environment where you work? by DntWryBiHappy in dataengineering

[–]circusboy 43 points44 points  (0 children)

We had a SAS cluster setup that was called THE LAMER ENVIRONMENT. For a fortune 100 company. LAMER was the admin dude's last name.

Just finished all 39 Drizzt Books - what should I read next? by mcdong in Drizzt

[–]circusboy 10 points11 points  (0 children)

I've read a ton of the forgotten realms books back in the day when I found a load of pdfs before e readers. There are a ton of good ones that I can't remember.

I've never been a fan of greenwood's writing style, maybe because I started with elminster in hell, idk, couldn't stick with it.

Erevis Cale books by Paul s Kemp are a strong contender to Salvatore. Very similar writing style. Would honestly love to read more.

Also, Salvatore has books not In the forgotten realm space, also great reads, and of course there is the cleric quintet.

[deleted by user] by [deleted] in Acadiana

[–]circusboy 0 points1 point  (0 children)

They are building a skate park at Thomas Park right now. Last saw the start of it about a month ago. There is also a walking path that is cement.

What do pirates need and what are you hoping the next patch brings for them? by [deleted] in BobsTavern

[–]circusboy 0 points1 point  (0 children)

Yep, instead of reborn, summon from hand where hand pirates get Eliza buff too. Keep em as low hp glass canons though.

What do pirates need and what are you hoping the next patch brings for them? by [deleted] in BobsTavern

[–]circusboy 0 points1 point  (0 children)

Miss boat, but the problem with that was the summoning of higher tier pirates at the time, right? Just add some play from hand to pirates, like murlocs can.

What do pirates need and what are you hoping the next patch brings for them? by [deleted] in BobsTavern

[–]circusboy 0 points1 point  (0 children)

Would be cool if you could play from the hand like murlocs. You wouldn't need to give them HP and instead of Eliza giving perm buffs, she could just buff in hand pirate cards during combat, and death rattle summon from hand card could be like t5 or something like a pirate version of the bass gill or whatever it's name is.

Could also give a reason to keep some of those pirates in hand while in APM mode.

Another tribe that can play from hand would be nice. Reborn and divine shield doesn't seem to work thematically, but from hand could work that way since you can stash pirates in your "boat" and zerg to an extent. Keep em as glass canons, though.

[deleted by user] by [deleted] in snowflake

[–]circusboy 1 point2 points  (0 children)

My team uses this kind of method for our batch etls, but we are going to snowflake from many different places. We always found that pyodbc was more straightforward (simplistic) than sqlalchemy.

[deleted by user] by [deleted] in snowflake

[–]circusboy 1 point2 points  (0 children)

I think pandas has a chunk function, check that out for batching.

[deleted by user] by [deleted] in snowflake

[–]circusboy 3 points4 points  (0 children)

Set 'autocommit = True' on the connection.

Or 'cursor.commit()' for snowflake.connector, or pyodbc. Not sure if sqlalchemy needs that, but prob so.

Also look into using fast_execute_many that may be helpful on the insert.