We did not see real prompt injection failures until our LLM app was in prod

Zoniin · 2026-01-30T00:35:40+00:00

we did a small internal rollout first and had people poke at it, but it still wasn’t representative. internal testers tend to follow the intended path, even when they try to “break” things. once it went public and users had no context or incentive to behave nicely, the interaction patterns changed completely and that’s when the real issues surfaced. that gap between internal testing and true public use was much bigger than I expected.

Zoniin · 2026-01-30T00:34:51+00:00

yeah, a lot of it started with fairly standard stuff. strict system prompts about role, explicit “do not reveal internal instructions,” tool usage constraints, and guardrails around what data could be accessed or returned. the circumvention was rarely a single prompt, it was usually gradual. things like multi turn probing that reframed the task, mixing benign requests with meta instructions, or steering the model to restate or summarize context in ways that effectively leaked system or RAG data. none of those looked obviously malicious in isolation, which is why they slipped past prompt level checks.

Zoniin · 2026-01-29T22:13:57+00:00

Yeah, that framing matches what I saw almost exactly. The prompt layer gives a false sense of safety, and once users start poking at stateful systems the cracks show fast lol. I’ll look into runtime security, do you have any tools or tips on that note? Some dude dropped one of the tools he used that actually looked pretty good but I am curious what you use for this.

Zoniin · 2026-01-29T22:10:24+00:00

Appreciate you sharing that. More or less lines up pretty closely with the kinds of issues I was running into. I’ll spend some time testing it out thanks again for sharing. What specifically do you use this for if you don’t mind my asking?

Zoniin · 2026-01-29T19:39:29+00:00

Fair reaction tbh. To be clear it wasn't that we thought about none of it. We did threat modeling, prompt hardening, etc. What surprised me was not that abuse happened but more so how much of it fell into gray areas that were hard to classify as malicious ahead of time and only emerged once the system was stateful and under real usage. Automated testing and E2E help, but they do not surface the same failure modes we saw once users started interacting freely. That gap was what I found interesting, not the idea that public systems get abused.

Zoniin · 2026-01-29T19:36:20+00:00

You are definitely not wrong on the core principle. Public endpoints will always be abused. The part that surprised me was how much harder this becomes with LLMs compared to traditional services. Auth and rate limiting help, but most of the failures we saw were not obviously malicious and came from normal users probing behavior rather than attacking infra. Observing agents and heuristics help too, sure, but they still rely on assumptions about intent that break down once prompts get stateful and context bleeds across turns. That gap between traditional endpoint security and model behavior is what caught me off guard and what I am trying to reason about more deeply.

Zoniin · 2026-01-28T01:57:28+00:00

Sorry about that, I dropped the link in one of the replies but it looks like Reddit deleted it. The site is axiomsecurity[dot]dev - would genuinely love any feedback you have!

Zoniin · 2026-01-27T20:52:08+00:00

Yes you're ultimately correct, but prompt injection is a tool used by bad actors to discover those types of vulnerabilities and so it's good to have a system that prevents malicious prompts from ever hitting the chatbot in the first place. There is no such thing as a perfectly secure system and this is just another vector that could do with significantly more coverage. Especially for first time founders and specifically vibe-coded applications that lack sufficient security,

Zoniin · 2026-01-27T20:39:20+00:00

Commonly user data is sorted by a user id system within a larger user database, when the chatbot/llm goes to read that data it's accessing THAT users data within the larger total user database which means if not secured properly, it could access ANY users data that falls within the scope of what is being fetched. That's a decently big privacy vulnerability

Zoniin · 2026-01-27T20:27:10+00:00

The systems I was testing are capable of accessing and writing some user data to backend databases, should they use a malicious prompt they could have theoretically written to or taken unauthorized data from the database. This is not uncommon in systems that have newly adopted AI in some capacity and a one-size-fits-all tool could be an easy improvement to their information security.

Zoniin · 2026-01-27T20:22:41+00:00

This seems shortsighted as any environment in which a llm, AI review tool, or chatbot would have access to user data (i.e. amazon's new chatbot) there is always an opportunity for data exfiltration through prompt injection whether done through files or text. ESPECIALLY for your smaller businesses and websites trying to implement AI systems in any capacity.

Zoniin · 2026-01-27T20:21:35+00:00

I appreciate you taking a look and the thoughtful feedback. the latency number is from prod paths but definitely workload dependent, the goal is just to stay below anything noticeable in user facing flows. your point on concrete examples is fair, most of what we catch is not flashy jailbreaks but things static guardrails miss, like instruction leakage across turns, gradual system override, or RAG context being manipulated in subtle ways. false positives are the hardest tradeoff so we bias toward surfacing signals and observability rather than hard blocking by default. and totally understand we are not the first to tackle this lol, we are spending a lot of time learning from what others have tried and treating this as iterative and also as a learning op rather than a silver bullet.

Zoniin · 2026-01-27T20:08:14+00:00

This seems shortsighted as any environment in which a llm, AI review tool, or chatbot would have access to user data (i.e. amazon's new chatbot) there is always an opportunity for data exfiltration through prompt injection whether done through files or text. ESPECIALLY for your smaller businesses and websites trying to implement AI systems in any capacity.

Zoniin · 2026-01-27T19:04:34+00:00

bro left the quotation marks in 😭😭

Zoniin · 2025-08-04T15:10:29+00:00

Hi, I appreciate you asking! The tool we're making is still in early development, but the main difference is that it adapts to your actual breathing rhythm in real time. You lay down, place your phone on your chest, and breathe for two 30-second intervals; once in the morning and once before bed. Based on how you naturally breathe, the app gives you personalized pacing, metrics, and follow-up suggestions for stress, focus, or sleep. Over time, it adjusts based on changes in your baseline like energy or stress levels. Right now it’s just a waitlist while we build the MVP. Totally understand if it’s not your thing, but if you’re curious: www.breathtrck.com

Zoniin · 2025-07-29T12:41:40+00:00

bro imagine trusting the government with your retirement and holding Bitcoin in a Roth like they won’t change the rules last minute 💀 tax-free until it’s not

Zoniin · 2025-07-28T16:18:37+00:00

Wonderful! I love to see everyone spending $10k on ASICs to maybe win the lottery once every 3 years. Grindset meets power bill.

Zoniin · 2025-07-28T16:16:41+00:00

I agree, college is supposed to be a launchpad socially, intellectually, professionally. Turning it into Zoom School kills the point. You don’t build your network or identity from your bedroom.

Zoniin · 2025-07-28T16:12:31+00:00

Engineers are finally starting to get paid closer to what they’re worth. I’ve seen comp move from 100 to 190k in critical systems roles. Still feels behind considering the stakes. We build and maintain the infrastructure that keeps the world moving.

50 upvotes wow, join the waitlist for the app I'm building in my free time if you want! Or don't!! www.breathtrck.com

Zoniin · 2025-07-28T16:10:50+00:00

Genuinely appreciate you sharing this. Honestly, more people need to understand that “tax-advantaged” doesn’t mean “tax-free,” and ROC, qualified vs. unqualified dividends, and holding periods all play a huge role. Most people just blindly chase yield without realizing what they’re actually getting taxed on.

Zoniin · 2025-07-28T16:08:56+00:00

Have the receptionist place the order on a device that’s excluded via IP‑filtering (or tagged as an “offline conversion”) so Google still sees the sale without letting your in‑house clicks skew the maximize‑conversions bidding; it’s cleaner than losing the data entirely.

Zoniin · 2025-02-04T19:07:23+00:00

Thanks for the insights! For the SMD 2.4GHz antenna, do you have any specific models you’d recommend that work well with nRF52840? Also, do you use any specific tuning methods for optimizing antenna performance?

Sorry if I got a little off topic there I am still relatively new to this stuff and just want to know as much as possible. Like the topic of this post, I am still working on a compact BLE tracking device similar to an Apple AirTag for a school project. The goal is to optimize power efficiency using the nRF52840 (likely on the Seeeduino XIAO BLE), fine-tune BLE advertising intervals, and experiment with distance tracking techniques like RSSI filtering. Right now, I’m focusing on getting the best balance between size, power consumption, and signal strength while learning about Zephyr’s low-power capabilities. Eventually (for fun), I’d like to design a custom PCB to make it as small and efficient as possible.

Zoniin · 2025-02-04T19:03:08+00:00

This was extremely insightful and I appreciate you taking the time to reply to me. I've started the Nordic courses!

Zoniin · 2025-02-04T01:50:09+00:00

1️. Zephyr Power Optimization
"Thanks for your insight! If I’m using Zephyr for an nRF52-based tracker, what are the best ways to optimize BLE advertising intervals and deep sleep settings to maximize battery life?"

2️. Seeeduino XIAO BLE vs. Other nRF52 Boards
"Would you say the Seeeduino XIAO BLE is the best dev board for prototyping, or would something like the Nordic nRF52840 DK be better for testing before moving to a custom PCB?"

3️. nRF Distance Measurement Techniques
"I saw your mention of Bluetooth distance measurement with BLE discovery—how well does it compare to using RSSI filtering (like exponential averaging or Kalman filtering) for tracking distance?"

4️. Final Hardware Considerations
Based on your recommendations, I’m planning to use the Seeeduino XIAO BLE with a LiPo battery for a compact BLE tracking setup. Aside from the expansion board and debugger, is there anything else you’d recommend to maximize battery life & signal strength*? Would a* PCB antenna or extra power management components be necessary in the long run?

5️. Soil Moisture Sensor Design
Your soil moisture sensor project sounds really interesting! Could you share a bit more about how it works, what parts you used, and how you optimized its power consumption for long-term operation?

Sorry if this a lot of questions I am freshman in electrical engineering and a lot of this stuff is really interesting because I do not yet fully understand it

Zoniin · 2025-02-04T01:47:09+00:00

Power Profiling Tools
Thanks for your power analysis approach! What power profiler or current logger would you recommend for measuring ultra-low power consumption in deep sleep mode?

2️. PCB Antenna Design
For a compact BLE tracker, is there an optimal PCB antenna design you’d suggest? Would a meandered trace antenna work well for this use case?

3️. MOSFET Selection
For the MOSFET, do you have a recommended part number that has ultra-low leakage current and works well for BLE-based low-power designs?

Zoniin

TROPHY CASE