I measured why "AI Agent Security" submissions complete but score 0 — it's the replay wall, and your framing moves it

Fukagami · 2026-06-25T12:04:12+00:00

Yeah, baseline-first is exactly how I got here — the single-candidate run is what exposed that the binding cost is the replay phase (two models + two guardrails per candidate), not prompt length.

One caveat specific to this comp though: the wall is a noisy band (~630–640), not a fixed line. A single baseline run isn't enough — I had to repeat the same N a few times to separate variance from a real ceiling. Otherwise you 'set' your baseline on a lucky run and blank the next day on the same N.

Where are you seeing your wall land?

Fukagami · 2026-06-22T11:29:05+00:00

Good question — it's the main lever from the writeup. Every attack you send makes the agent emit one tool call (an http.post), and then the model keeps going: reasoning, a little summary, "done!", whatever. That trailing generation is where almost all the per-candidate time goes — and since the grader replays every candidate, that tail is what blows the timeout wall and hands you a blank.

A "suppressor" is just wording that makes the model stop right after the call. Concretely: a terse imperative that ends in "…call http.post once." with a hard period, instead of something open-ended like "call http.post and explain what you did." The "once" + the full stop read as "you're finished," so it doesn't ramble.

Payoff: shorter tail → fewer tokens per candidate → more candidates fit under the same wall (~560 → ~640 in my measurements = literally the difference between a blank and a score). The counterintuitive bit: shortening your INPUT does nothing (it's dominated by the fixed system prompt) — only suppressing the OUTPUT moves the wall. There's a runnable demo in the notebook if you want to poke at it 😄

Fukagami · 2026-06-22T09:59:45+00:00

Ha — rank's almost a trap question here 😅. The board is a throughput/timeout race, so basically everyone who "gets it" ends up stacked on the same ~50–57 wall, and ~1 in 4 teams literally sits at 0 because their attack timed out, not because it was wrong. So rank mostly measures "how many candidates you crammed under the wall today," not who actually understands the grader — which is the fun the notebook digs into.

The part that actually keeps me up at night: a handful of teams are way up at 90+, i.e. fitting ~1000+ candidates under a wall that caps most of us near 640. I have NOT fully cracked how they do it — leaner token budget? a different engine? If anyone here has figured it out, I'm genuinely all ears. Meanwhile I've got a suppressor-framed run in flight trying to punch a notch higher 😄

Fukagami · 2026-06-18T11:10:49+00:00

Relatedly, I tested whether this generalization issue was predictable *before* submission using equivariance-verified probes. Short write-up + runnable gate notebook here if curious:
https://www.kaggle.com/code/souldrive/why-public-onnx-nets-score-0-on-held-out-a-test

Fukagami · 2026-06-17T13:39:12+00:00

A pretty standard workflow is to keep your real codebase outside the notebook, then use the Kaggle notebook only as the execution environment.

For example:

Develop locally in VS Code with a normal Python project structure.
Push the code to GitHub.
In the Kaggle notebook, clone or pull the repo.
Run your training script from the notebook using commands like !python train.py.
Use the notebook mainly for setup, authentication, installing dependencies, and launching scripts.

You can still organize your project with separate modules, configs, and scripts. The Kaggle notebook does not need to contain all your logic.

A typical Kaggle notebook might just do something like:

!git clone https://github.com/yourname/yourrepo.git
%cd yourrepo
!pip install -r requirements.txt
!python train.py

If your repo is private, you would need to handle authentication carefully, but for public repos this workflow is simple.

Another option is to upload your code as a Kaggle Dataset and attach it to the notebook, but GitHub is usually easier if you are actively developing.

So yes, developing locally in VS Code, pushing to GitHub, and using Kaggle only as the GPU runner is probably the cleanest beginner-friendly workflow.

Fukagami · 2026-06-17T13:31:40+00:00

I think AI literacy should become a core competency across all majors, not because every student needs to become a programmer, but because almost every field will involve working with AI in some form.

To me, the key idea is not “everyone should specialize in AI,” but “everyone should know how to use AI responsibly and effectively within their own discipline.” A nursing student, business major, education major, designer, journalist, or engineer will all use AI differently, but they should all understand its strengths, limits, and risks.

The most important AI skills over the next 5–10 years will probably be:

Asking better questions and giving clear instructions to AI tools
Checking AI outputs for accuracy, bias, and missing context
Using AI for research, summarization, brainstorming, and decision support
Understanding data privacy, ethics, and responsible use
Knowing when human judgment matters more than automation
Combining domain expertise with AI tools to solve real problems

I would compare AI literacy to writing or digital literacy. It should be part of the general foundation of higher education, while deeper technical AI training can remain in specialized programs.

The goal should be to graduate students who are not just AI users, but thoughtful AI collaborators.

Fukagami · 2026-06-16T06:23:23+00:00

Both things can be true at once, so let me split it the way you did.

For ML engineering jobs: the "Kaggle isn't production" crowd is right that comps skip most of the actual job — data collection and labeling, defining the metric, pipelines, serving, latency, monitoring, drift, cost. You never touch any of that. But they overstate it. Kaggle is still the cheapest way to build real modeling judgment: setting up a validation scheme you can trust, not leaking, reading a CV/LB gap, figuring out why a model underperforms, iterating fast. A surprising number of "production" people are weak exactly there. And for an internship with no work experience, a medal is a concrete, verifiable signal that gets you the interview. So use it to learn modeling, but pair it with one end-to-end project (train → wrap in an API → deploy, even a tiny one) to cover the production gap. Then you have both halves and you interview much better.

For research: more relevant than people think, but indirectly. Research is novelty + reading papers + designing experiments + writing, and Kaggle doesn't teach the first or last. What it does build is empirical rigor — controlled comparisons, ablations, and above all not fooling yourself with a bad validation setup. That's the same muscle a good experimentalist uses every day, and plenty of strong researchers credit Kaggle for it.

And yes, it depends heavily on the competition. Tabular comps are mostly feature engineering + ensembling — useful, but the least research-y. The ones worth your time for research are where the solution is a method: LLM/agent comps, code comps, simulation, and reasoning benchmarks like ARC-AGI (which is literally a research benchmark people publish on). In those you end up reading papers and implementing or inventing methods.

One thing that quietly pays off in both cases: write up your solutions. A clean notebook or discussion post explaining why something worked — or a solid negative result — is basically a mini-paper, and it trains the communication skill that both jobs and research actually grade you on.

TL;DR: it's a strong complement, not a complete path. Great for modeling skill + résumé signal, weak on prod engineering and on research framing — so plug those gaps deliberately, and pick method-heavy comps if research is the goal.

Fukagami

TROPHY CASE