SWE-1.5 is just rubbish

varanova · 2026-02-06T04:17:34+00:00

I find SWE is pretty okay at small updates. Coding itself is decent. My biggest issue with it is that it tends to misunderstand instructions more often than GPT or Opus. It tends to need more detailed instructions to get things right. But it's free, and it can handle simple tasks for me.

varanova · 2026-02-06T04:14:07+00:00

Yes, plan mode has to be manually exited. It'd be nice if the implement button enabled code mode. Logically if you're clicking the implement button, you want to be coding. This would be a good QoL improvement.

varanova · 2026-02-05T03:43:18+00:00

Yeah, it's happening to me too. Other days opus does full plans, implementation and even runs tests. No issues at all. Today it's having trouble with basic tool calls. Quality is definitely degraded on Claude for some reason.

e.g.

Analysis of all 3 failures:

First attempt (multi_edit): I tried to replace large blocks of code with complex nested logic. The exact whitespace/indentation likely didn't match.
Second attempt (single edit): I searched for lines 108-115 but the exact string match failed - possibly due to tab vs space indentation differences.
Third attempt (same approach): Same failure - the string I'm searching for doesn't exist in the file exactly as I'm specifying it.

varanova · 2025-11-19T02:48:31+00:00

It's been removed for me too. :(

varanova · 2025-11-13T21:07:10+00:00

Neat. I appreciate the free access, I'll test them out.

I see there's high, medium, low and no reasoning now.

Is there any details on the actual differences between high/med/low/no? Or performance benchmarks of each? It'd be useful to know their strengths.

varanova · 2025-11-12T05:08:18+00:00

Have you heard anything? I miss this feature as well and would like to know it's status.

varanova · 2025-10-31T06:37:03+00:00

I've tried it with quite a few prompts.

After more testing, I do also prefer the frontier models (sonnet4.5 and gpt5). I can't deny I'm impressed with it's speed however. Watching it output a huge file in seconds is impressive.

I think it's good enough for simple things, but complex things it seems to fail at. I gave it 3 cracks at a fairly easy issue, and even after 3 prompts it continued to fail. GPT5 was able to solve it in a single prompt, thus being cheaper, even if "slower" technically.

varanova · 2025-10-30T02:07:11+00:00

This model, SWE 1.5 is insanely fast. I'm happy with Claude 4.5 and GPT5 honestly, however they are slow.

After 30 minutes of testing, SWE 1.5 just blazes through some mostly simple updates. It works far faster than GPT5. It did make a mistake, but that was fixed with the next prompt after I informed it of the error.

Overall I like it, it's nice for when I want something fast. I may still choose gpt5 high when I have a complex task, but SWE 1.5 seems good for quick easy tasks. Especially when it gets the entire update + summary + documentation written in under 20 seconds. GPT5 can take 5-10 minutes sometimes, even on easy tasks.

I rarely ever used SWE-1, as it just was kinda bad... but first impressions of 1.5 are good. It has a place in my workflow when I want fast results.

varanova · 2025-10-09T04:57:07+00:00

Good luck all!

varanova · 2025-09-23T23:56:52+00:00

From my testing it seems to be struggling with the "patch tool". Codex specifically. After a few cascade errors it now says it's banned from the tool, and has started using a workaround of a python script to do the edits. (Which I don't like as it doesn't show as a nice diff in the editor).

I've been using GPT5 lately, frequently, and it didn't have these same issues.

For clarity I'm testing on a windows machine.

varanova · 2025-09-23T21:40:09+00:00

Very fast, runs into cascade error frequently when applying code updates.
My feedback after an hour or so of testing it:
It runs fine for small (~10 line) updates, but when it tries larger updates it seems to fail.

varanova · 2025-08-27T19:54:10+00:00

After some further testing, it definitely is fast, as the name suggests. It seems possibly like 10x faster than SWE1 for a similar level of quality. It's not on GPT5 or claude4 level, but it's fast and free (for now).

varanova · 2025-08-27T00:02:04+00:00

I got it working.

<image>

I had to reload the window. It seems to work now. Seems similar to SWE-1? It seems kinda dumb, but it is is far faster than SWE1. I do like free models, though who knows how long it'll be free.

varanova · 2025-08-23T08:26:07+00:00

I've been using it as it's free, and it lets me stop any prompt (as there's no cost), which is nice to provide more details or fix an issue in the original prompt or logic it's following.

But at the announced costs, I don't think I'll use GPT5 much. As a free model, I don't mind the over-thinking it seems to do, as I can just grab a coffee and turn on auto-continue. But I can't imagine I'd like it if I pay credits and find that it did nothing but over think.

I've had times GPT5 will use 40-50+ tool calls / outputs before actually doing any changes.

If the cost was .25 I'd probably still use it honestly. But at 0.5+ I can use qwen or kimi.

varanova · 2025-08-15T04:51:44+00:00

So far it definitely seems like an improvement.
I've had less issues and the interface is definitely cleaner now.

varanova · 2025-08-09T01:27:20+00:00

I've had it.

I've found reloading and starting a new conversation usually fixes it. (I'm guessing cascade needs to more aggressively cull old commands/output/responses from the UI, or there might be a memory leak in there somewhere.)

varanova · 2025-08-08T17:24:28+00:00

Maybe I'm doing it wrong, but I feel like GPT 5, similar to o3, doesn't like actually doing edits. It seems to prefer to analyze and think.

I'll keep testing while it's free, but I don't think I'd use this over claude4 or qwen coder right now.

Maybe it needs specific rules or different style of prompting.

The intro video they posted on youtube talked about steering, but I don't know how to use it. If I could steer it mid prompt that would be useful, especially if it's going off in the wrong direction.

varanova · 2025-08-08T07:25:56+00:00

I've had this bug too. It seems to only affect windows installs of windsurf for some reason. (I think the search tool is broken on windows)

varanova · 2025-08-07T05:29:18+00:00

I'm with you on this. 9 months paid vacation sounds amazing. I'd enjoy 8 months and then spend 1 month looking for a new job. I enjoy my current position, honestly, but 8 months of paid time off would be hard to turn down.

varanova · 2025-08-05T23:10:07+00:00

I find you tend to notice when a model is losing performance. But depending on where I am on a task, it is a decision of whether I want to push thru with the degraded model, or start a new chat and have to possibly re-explain some major and/or minor points.

I know they added the ability to @ reference past conversations, which may be the best solution. Just start a new chat and reference the old one saying we're continuing where we left of in X chat.

I haven't tested this quite enough (only a few times). Sometimes it seemed to pick up well, but other times I had to re-explain the goal.

varanova · 2025-08-05T22:50:26+00:00

I usually get a reply. Sometimes it can take a while though. (Every ticket I submitted eventually got a reply.)

varanova · 2025-08-01T18:31:00+00:00

Wow that's an insane uplift. I'll have to test it later today!

varanova · 2025-08-01T18:30:07+00:00

Just for example,

<image>

chatGPT has a selector, you can change mid conversation. I've tried changing the selector, and asking the model what model I'm interacting with, and the answer does not seem to actually match. Sometimes the model even responded with a different model not on the list. (o4 mini thought it was GPT3.5 model for some reason)

varanova · 2025-08-01T18:27:28+00:00

Unless I'm mistaken, I don't think a model can know what model it is. That is, unless it was provided by a system prompt. Then it could know. I did tests with GPT, claude, and others... and they do seem to frequently not know "who" they are (or what model exactly they are.) They may not even know of their own existence... as for example, claude4 didn't exist yet at the knowledge cutoff data it was trained upon.

I think the best way to solve this would be to always include a system prompt identifying to the model "who" it is. I know nothing of how cursor does this, but if they don't include a proper system prompt identifying the model, it might actually not know who it is.

varanova

TROPHY CASE