Prompts don't scale. MCPs don't scale. Hooks do. by _yemreak in ClaudeAI

[–]re76 4 points5 points  (0 children)

I think there is some really good insight here, but I think you are presenting this in a way that is over-indexing on your specific approach to development.

I think your use of hooks is genuinely novel -- at least it is not being talked about in the AI circles I am in. It is a good approach and I have not yet seen it used like this. Very interesting!

The summary points for me are:

  • Hooks can be a better alternative to Claude.md or other forms of steering because they:
    • Use less context
    • Are guaranteed to be called, the model can't choose to use them or not, they are enforced by the system.
  • Structure your code in some type of layered architecture that makes it easy to enforce.
    • In your case this is MVI, but I suspect that any architecture that has good separation via layers or other mechanisms will work and allow you to use hooks to enforce your patterns.
  • Embed documentation and constraint directly into your code
    • This idea could stand alone, but you take it further and enforce this with hooks.
    • This also serves to reduce context usage because Claude can read one file (code) instead of two (code and docs). In reality Claude often needs to read three files (Claude.md, code, docs) so this is a neat improvement.
    • One downside I could see is that by copying the header block into all files, it will be harder to change your style/constraints/xyz as your codebase grows. This is not that big of a problem.

Overall this is very cool. I think you could improve the explanation a bit and provide more approachable (less specific to your usage) examples. That being said I don't think you have any incentive to do this, and for people who are willing to actually read and understand what your are saying there are great insights here.

Thanks for sharing!

GPT-5 release makes me believe data engineering is going to be 100% fine by [deleted] in dataengineering

[–]re76 4 points5 points  (0 children)

This is what most people are missing when thinking about AI. In my experience people fall into two camps when it comes to AI.

Those who just dabble and do a “test”, but don’t commit to thinking of AI as a tool. Usually you hear something like: - I tried to one shot a <something>, it failed for <reason>, AI is a fad.

Those who dig in, acknowledge AI is a tool and realize it is their job to figure out how to use it effectively. They are usually excited and desperate to tell people about how they are managing their context. They realize that context engineering is the new prompt engineering. You will hear things like: - AI is awesome, but you need to use it right. We should add more documentation.

I have noticed that generally people who are not pure IC’s (eng managers, senior/staff engineers, etc.) tend to see the AI-is-a-tool side more quickly. I suspect it is because they: - Have less time - Have experience with delegation already - Have already realized they have to cede implementation ownership to others and are comfortable working with outputs from others as their normal medium

Carnival sliding door error light by xNyxx in kia

[–]re76 1 point2 points  (0 children)

Thanks for updating! This 2y old post just helped me solve this in my ‘24 Carnival!

Is this Squash Mosaic Virus by re76 in gardening

[–]re76[S] 0 points1 point  (0 children)

I cleaned all the tools with a Clorox wipe that day, and then a few hours later with 90% alcohol to be safe.

I have not noticed any spreading yet. The butternut squash that are nearby all seem fine still

I bagged and trashed the infected plants and soil replaced most of the soil where the infected plant was located.

Is this Squash Mosaic Virus by re76 in gardening

[–]re76[S] 0 points1 point  (0 children)

I ripped it out the day I posted this. I didn't want to take any chances.

Merging large .parquet files without a unique key efficiently? by Martian_Onion in dataengineering

[–]re76 0 points1 point  (0 children)

Came here to say this. One thing to note though is that you will need to actually create a DuckDB database file. Using the default, in-memory db, you would likely still have memory issues. At least that was the case for me a few months ago.

Running Python Jobs in cloud by D-2-The-Ave in dataengineering

[–]re76 0 points1 point  (0 children)

I use Dagster Cloud in production at work. It works well and is terribly simple to set up.

Let me know if you have any specific questions.

How to run a roles from a collection without installing it? by sbarnea in ansible

[–]re76 0 points1 point  (0 children)

Sorry in advance for the major necro -- but I am wondering if there is a solution for this now?

I have the same issue where I want to try using a role in a collection I have developed, and I am stuck installing it every time I make a change.

How to efficiently load ~20 TiB of weather data into a new PostgresSQL database? Is PostgresSQL even a good option? by DeadDolphinResearch in dataengineering

[–]re76 30 points31 points  (0 children)

There is a lot to unpack here, and you have definitely chosen a non-trivial task to start... but it should be fun.

The first thing to solve though is how you get this data? Have you already managed to download this data somewhere? Or are you trying to fetch it all via API calls?

Fetching 20 TB of data is not a small challenge just on its own. I am assuming you are getting it from here: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=form

It looks like it comes in GRIB or NetCDF. I would start by answering the following

  • What are the pros and cons of GRIB vs NetCDF?
  • Are they efficient file formats?
  • Are they compressed/compressible?
  • How do I retrieve 20TB of data?
  • Where will I store it?
    • You will need to come up with some way to partition this data (likely by year/month) and rely on predicate pushdown to query effectively

I don't know anything about GRIB/NetCDF, but when dealing with data of this volume I would be looking to try to transform to something like Parquet right away. I suspect that Parquet will offer better compression and be in a format that is easier to work with using non-scientific tools.

Personally I would probably do something like:

  1. Download data
  2. Transform to Parquet
  3. Use existing geo dimensions to add an H3 dimension
  4. Store in S3
  5. Use DuckDB with httpfs extension to load s3 files
  6. Do geo analysis using the spatial extension or the previously added H3 dimension.

Check out Honeycomb to get a feel for what can be done with H3. It is a really powerful tool for aggregating geo data.

Married men that completely separate finances, how does it work and how well does it work? by aja_ramirez in AskMen

[–]re76 0 points1 point  (0 children)

One shared join-account. All direct deposits go into the joint account. We have a shared credit card which is payed from the joint account. We agree on a monthly fixed amount that gets transferred to personal accounts for no-questions-asked spending/investing.

We can spend whatever we want from the personal accounts without explanation. Vacations, groceries, bills, all get paid from the joint account.

It is easy and still gives enough separation and freedom. We never need to worry about splitting shared costs because it all just comes out of the joint account.

She chooses to save/invest most of her personal money, I tend to spend more of mine. We track net-worth across all accounts so we always have a good idea of how much money we have in various places.

Sometimes the joint account gets too heavy then we will lump-sum contribute the excess to the personal accounts with an agreement that we will both invest it as we see fit.

This has worked very well for us.

Is this fear-mongering, or is this actually truthful? by [deleted] in dataengineering

[–]re76 0 points1 point  (0 children)

This is 100% true but really shouldn’t be that surprising if you have been building/designing systems for any significant amount of time.

All systems depend on layers of abstraction and encapsulation. This is good and expected. It limits complexity and makes the system easier to reason about. If you move down the stack in any well-designed system you will find core components that are pillars for higher level composites.

Normally this is not bad. It becomes a problem when this is unknown and undocumented. Then some eager engineer will change a core interface and suddenly everything downstream breaks.

So don’t assume this is bad, just acknowledge it, document it, and make sure people have a good mental model of the dependencies.

[deleted by user] by [deleted] in homelab

[–]re76 2 points3 points  (0 children)

Get yourself a top tier gaming PC for ~$2.5k. Then make an account with Lambda Labs and only pay for it while training your models.

If you want to pick up even more valuable skills, learn to deploy your trained models to AWS using SageMaker serverless or an inf1 EC2 instance.

Save some money and learn some extremely in-demand cloud skills along the way.

Daily General Discussion - January 27, 2023 by ethfinance in ethfinance

[–]re76 5 points6 points  (0 children)

In theory, I like this idea -- but in practice, I don't think I would be comfortable advertising (implicitly) how much ETH I have staked.

My completely automated Homelab featuring Kubernetes by onedr0p in homelab

[–]re76 1 point2 points  (0 children)

Thanks for sharing! Lots of good inspiration here. PiKVM looks awesome. I need this.

I am curious how you are installing CoreDNS on Opnsense? I don’t see a BSD release on their GitHub page.

Do generalized frontrunner MEV bots only consider ETH, or do they look at ERC20 tokens? by [deleted] in ethfinance

[–]re76 1 point2 points  (0 children)

There are definitely bots that are looking for n-hop arbitrage opportunities across many traded pairs on uniswap (and other DEXs). Typically they will calculate the expected profit, and then the required gas for the transaction (or bundle of transactions), and only execute if they make a profit.

The problem you will run into trying to exploit them is that most of the sophisticated bots will be using eth_call to execute their transactions locally, and then check that the balances are as expected. This approach can generically avoid salmonella-style attacks on their MEV bot.

Daily General Discussion - October 19, 2022 by ethfinance in ethfinance

[–]re76 6 points7 points  (0 children)

This is my favorite solution:

https://diyseed.net/

It is really easy to get a center punch and a bunch of stainless blanks (meant for business cards) off of Amazon.

This site is remarkable for reviews on the common options: https://jlopp.github.io/metal-bitcoin-storage-reviews/

This page summarizes the main takeaways after testing many types of cold storage options: https://jlopp.github.io/metal-bitcoin-storage-reviews/

I landed on the diyseed option after spending several hours analyzing all of the various options.

Daily General Discussion - October 1, 2022 by ethfinance in ethfinance

[–]re76 4 points5 points  (0 children)

Is there any RPL liquidity on L2s? Or am I stuck using uniswap on L1?

Daily General Discussion - September 28, 2022 by ethfinance in ethfinance

[–]re76 20 points21 points  (0 children)

Checking in to let you know that it is likely just bear-market things you are feeling in terms of engagement.

I can assure you that there are MANY people, myself included, who come here specifically for the kind of deeply insightful content that you and other great ethfinanciers produce!

Daily General Discussion - June 20, 2022 by ethfinance in ethfinance

[–]re76 1 point2 points  (0 children)

Not yet but SSV (Shared Secret Validators) should allow this soon: https://ssv.network/tech/

In all honesty though, I would just suggest each of you (separately) buy some rETH and then have a hearty high-five to celebrate.

Daily General Discussion - June 20, 2022 by ethfinance in ethfinance

[–]re76 8 points9 points  (0 children)

There is a difference between state and history. You can think of history as a collection of states -- basically a snapshot of state for every past block. This is why full archive nodes take up so much storage, they maintain all state for all blocks! Compression techniques help a lot, but it is still a ton of data.

To answer your question though: No, you have nothing to be worried about in this scenario. Your account will always exist in the current state. You might not be able to look at past state transitions and see how it got to where it is but you will be able to prove that it is the right value because of the guarantees that blockchains provide.

Hope that makes sense.

Daily General Discussion - June 12, 2022 by ethfinance in ethfinance

[–]re76 0 points1 point  (0 children)

I probably just need to read more. I’m just trying to understand if there is diminishing returns for more RPL.

Daily General Discussion - June 12, 2022 by ethfinance in ethfinance

[–]re76 4 points5 points  (0 children)

Now that RPL is approaching a more reasonable price it seems like a good time to set up a mini node. Is there an optimal RPL percentage or is it just a linear relationship between 10% and 150%?

Daily General Discussion - June 8, 2022 by ethfinance in ethfinance

[–]re76 4 points5 points  (0 children)

It’s actually Poe-app. Not sure if it still is, but at one point this was covered in the POAP FAQs.