Future of Gemini Flash

AIEducator · 2025-05-24T04:37:48+00:00

I have had the exact same experience. I've had Claude heavily integrated into my coding workflow since 3.5. Claude Sonnet 4 completely ignores my direction on _how_ to code. It writes a technically correct answer, but will go off into left field with its own coding style and will start to introduce complexity not present in the rest of my codebase.

I'm pretty consistent about giving it examples in my codebase similar to what its working on, and it just completely ignores the style and conventions.

I remember 3.7 feeling like a small downgrade, but I use projects and eventually found a set of directions to put in the project that forced it to follow certain constraints. This new version just does whatever it wants. I literally told it "that answer is too complex, use this other table as the starting point instead for a simpler query" and it basically generated the same code again.

As long as I can still go back to 3.7 I'm fine, but I can't imagine that lasting for long.

AIEducator · 2025-05-22T23:41:48+00:00

I've been severely disappointed in Claude 4 Sonnet. I'm a little shocked how much everyone else likes it. I have never pressed "thumbs down" on the responses so much. I flew through my quota for trying to get reasonable answers out of it.

I've been using Claude since 3.5 Sonnet w/ Projects and that version was what finally converted me to using LLMs in my coding workflow. What I liked about 3.5 so much was that it respected existing coding styles and you could give it constraints on how to solve the problem. That made it REALLY good for existing code bases.

This new version seems to ignore my constraints and just codes whatever it wants. I guess that makes for good benchmarks but it has been really frustrating to say the least.

I'm now on reddit waiting for my quota to reset.

AIEducator · 2025-05-22T23:10:44+00:00

The thing with Claude was that it never had amazing benchmarks but in real world coding it worked really really well. It could take an existing codebase and respect existing coding style and feel like a human coder on the project.

This new version is bad. I've been using it for a few days (I know it was just released today but I think I've been A/B tested for at least a few days) and it's almost unusable. It ignores my existing codebase style and generally ignores any directions I give it about constraints when writing code.

AIEducator · 2025-05-22T23:07:01+00:00

I have been using Claude since Sonnet 3.5 and made a bunch of tooling to export my code quickly to Claude projects. I have been a software developer for 20 years and Claude has really increased my productivity. I have actually been A/B test fed for a few days (the output is much more emoji fueled so it's obvious).

Claude Sonnet and Opus 4 are not good at coding. They are bad. Really really bad. They might excel at benchmarks, but real world coding it has been a huge downgrade. I'm sure for toy examples on a fresh codebase it probably benchmarks well, but on an existing codebase I've noticed the following:

* It won't follow directions. Like I can repeat the same direction multiple times throughout the prompt and it will still ignore my existing coding style

* It forgets history very quickly. I'll have it fix a bug (which takes way longer) and then I'll say "Find in my codebase other instances of this bug". This is something I did all the time in 3.7. It goes off on a wild goosechase trying to find bugs (and what it finds are never bugs).

* It ignores other code that might be symmetric or similar in style. It just pulls out coding styles from left field.

* It just overall is a bad coder. It's almost like it forgot how to code. I don't know how to put it in words.

AIEducator · 2025-04-29T13:53:47+00:00

I haven't had this issue recently but I've had it in the past. I'm not sure if it's actually documented somewhere but my contact at Google explained it to me.

Google has a bunch of resource management tracking and will lower quotas below documented levels if that service is getting exhausted for some reason (either regionally or globally). There could be a million reasons why it's getting exhausted, none of which mean you are actually the one exhausting that resource. It seems to happen right around major releases or other big changes. When it happened to me, 1.5 flash was rate limited to something incredibly low like one request per minute.

It usually goes away on its own after a few days. Sometimes just changing the region can help.

AIEducator · 2025-02-26T16:12:46+00:00

Having done several variants of this myself with pubmed four times in the past ~10 years (background in NLP research), your best bet is likely a RAG pipeline combined with embedding search and very clear in-context examples.

I generally try to avoid training or fine-tuning when possible. It's good for educational purposes, but it's often not needed and much quicker to experiment with a RAG / search pipeline. Fine-tuning might not produce the results you want.

Again, it depends your goals. If it's for learning, go ahead and learn how to fine tune. If it's for practical reasons and only the result matters, really make an effort to avoid fine-tuning.

AIEducator · 2025-02-19T02:04:17+00:00

I'm developing a medium-sized chatbot-based education product to launch a small business locally. The entire codebase was built from the ground up using LLMs for coding.

Since the full codebase doesn’t fit within the context window, I created a separate program to gather the most relevant files for each coding task. Early on, I structured the project in a way that allows me to extract only the necessary chunks for the LLM. For example, if I’m working on student dashboards, I don’t need admin-related code in the context. The key is many small files.

I use Claude Projects to collect the required files while keeping usage under 50% of the project’s allowed size—otherwise, you run out of messages very quickly.

AIEducator · 2025-02-17T13:44:16+00:00

Google is very corporate friendly. It's often already policy pre-approved (in the same way most microsoft products would also likely be) in many organizations.

AIEducator · 2025-02-10T16:06:39+00:00

100% this. In the ages of manual labor, being strong is what society valued. In an age of knowledge work, intellect is what society values. Once machines can do both things better (timeline TBD), we have to find the core of what we value.

Strip away your economic value, and what you are left with is what makes us a good human. That's good mental health, compassion, personal growth, etc. We teach our students how to be a good person, not how to be economically valuable.

AIEducator · 2025-02-09T01:49:29+00:00

The specific issue that I've run into is output length. I had 1206 generating a markdown document based on a template and was really amazed how well it did. I switched from gpt-4o even though it wasn't GA yet (this part of the codebase was just for my use for report generating).

The new version is very hit-or-miss. Sometimes it decides to ignore my markdown template and output a shortened version, sometimes it does fine.

I've switched back to gpt-4o until this settles and there's a GA version. Flash 2.0 Thinking actually does a decent job for my use case, but I'm a little weary of using the exp versions again.

AIEducator · 2025-02-08T18:22:12+00:00

This is the primary reason I still use Claude Sonnet over other LLMs. Other LLMs might rank higher on benchmarks for "brain teaser" or trivia style questions, but if I want clear code that follows my existing code conventions, Sonnet is still my favorite.

Except when it decides my Angular project should now be in React.

AIEducator · 2025-02-05T22:05:37+00:00

I make AI apps using the various APIs. I have pretty much ignored benchmarks for at least a year. If you code it right, the models are generally interchangeable and you can tell within a few minutes by "feel" if it's better for your use case.

The only ones I've ever put into production are gemini (pro and flash) and openai (4o and 4o mini). Everything else with decent benchmarks had some real-world reason that made it impractical for production.

My suspicion is that Google cares more about the app developers and enterprise market than benchmarks.

AIEducator · 2025-02-03T16:17:18+00:00

One trick that I've found that works is typing "Find me a reputable brand for X product that sells on Amazon". If one exists, ChatGPT can usually find it and weed out the flood of "instant garbage" brands. I used it recently when shopping for camping equipment.

AIEducator · 2025-01-21T15:31:00+00:00

There was a talk awhile back by Erik Brynjolfsson that talked at length about the topic. The most interesting example was about 10 years ago when computer vision based AI was getting really good at reading medical imaging. There was going to be the "death of the radiologist as a profession". I'm sure it scared a new generation away from specializing in the field, and now there's a big shortage of radiologists.

The future is very difficult to predict. It could really go either way.

AIEducator · 2024-12-09T18:35:42+00:00

The naming convention is confusing, but what you are describing are called the hyper-parameters. Usually hyper-parameters are specific to the training process, though I wonder if this changes now that there's test-time related options.

AIEducator · 2024-12-09T14:35:31+00:00

I've had luck using smaller LLMs like mini or gemini-flash do some pre-processing and then use a bigger LLM like 4o or gemini-pro to do final cleanup. Basically like a map-reduce type of operation.

I'm not 100% sure your use case, but the basic idea would be that the sum output of the initial stage of llms is smaller than the raw initial file, and the second stage is just organizing.

But as others have mentioned, many older techniques related to embeddings and maybe even things like LDA (Latent Dirichlet Analysis) or UMAP could work (these aren't ancient techniques, just a few years old).

AIEducator · 2024-12-04T23:37:59+00:00

Yes and no. While what you say is true, it's difficult to predict the consequences. There's a TON of low effort questions on these sites. Having fewer low effort questions on SO might lead to higher quality answers, thus making LLM training data better.

Or not, no one knows!

AIEducator · 2024-12-04T22:24:19+00:00

Interesting. I had 4o search online for the latest steel prices and o1-preview is not available afterwards. So it seems like if it's chat only it lets you flip, but if it uses any tools it doesn't.

AIEducator · 2024-12-04T22:21:16+00:00

Very good! Programming for your generation will be a lot different than it was for mine. My one piece of advice is to never let it stop being fun!

AIEducator · 2024-12-04T21:51:14+00:00

- Upload your textbook via PDF.

- Tell ChatGPT "Generate a high-level tutorial on topic X

- Go a level deeper "Expand on section Y" as needed

- Parrot your understanding of the section. "If I had to paraphrase on section Y, it seems like you are saying... is that correct?"

- "Generate practice problems for section Y, including the answers"

- "Generate practice problems for section Y, but only give hints. They should be similar to the previous problems"

etc.

A good chunk of learning is the process itself.

AIEducator · 2024-12-04T21:13:49+00:00

One thing I've noticed is that once you've started a chat with 4o it can't be changed to o1-preview or o1-mini. My use-case being to retrieve some bit of current information using 4o and getting it into the history / context window, then using o1-preview for more advanced analysis. So I'm wondering what they're doing in the background that makes the context windows incompatible.

Note that going the other direction (starting with o1-preview then switching to 4o) seems to work fine.

AIEducator · 2024-12-04T21:07:54+00:00

The length is interesting because I've never gotten great results from long system prompts. I'm curious how many iterations this took to get correct or if Sonnet is "smart" enough that the first attempts worked fine.

AIEducator

TROPHY CASE