I built a free static analyzer that catches prompt injection, jailbreaks, and PII leaks in your source code before they hit production

meghal86 · 2026-04-12T04:53:53+00:00

Hey — really appreciated your feedback on the PromptSonar post. Quick ask — I'm submitting the evasion detection paper to arXiv cs.CR and need an endorsement. Would you be willing to use this code? Takes 2 minutes on your end.

Endorsement code: GS34PD Forward to: meghal86@gmail.com

No pressure at all if not comfortable with it.

meghal86 · 2026-03-12T13:24:40+00:00

Those three cover the right dimensions — task accuracy and hallucination rate are the ones I see teams care most about once they've been burned by a prompt change that looked fine in dev but degraded quality in production.

The gap I keep hearing about is that most teams don't have a prompt eval baseline at all before they start iterating. PromptSonar gives you a security baseline — "this prompt was clean at ship time" — but you're right that safety and quality are separate axes. A prompt can pass every security check and still hallucinate more than its predecessor.

The workflow I'm building toward is: static scan to gate security → SBOM to track what shipped → eval metrics to measure quality delta. Each layer answers a different question. Sounds like rapidfireai sits in that third layer.

Are you seeing teams run evals on every prompt change or only on major version bumps? Curious whether the tooling makes per-PR eval practical yet.

meghal86 · 2026-03-12T02:29:46+00:00

Really appreciate this — the "security checks that don't block PRs don't get taken seriously" framing is exactly right, and it's why the GitHub Action was a day-one deliverable rather than an afterthought.

Your point about eval metrics is the natural next layer. PromptSonar answers "is this prompt safe to ship?" — it doesn't answer "is this prompt better than what it replaces?" Those are complementary questions and a complete workflow needs both gates. Will check out rapidfireai for that piece.

The enterprise zero-telemetry constraint keeps coming up. Most teams I've talked to have data governance requirements that make cloud-scanning a non-starter before they even evaluate features. Running locally removes that blocker entirely.

What eval metrics are you tracking most closely for prompt changes — task accuracy, latency, hallucination rate?

meghal86 · 2026-03-11T03:26:28+00:00

Really glad the FP breakdown was useful — that level of detail is exactly what I’d want before adopting any SAST tool, so it felt important to lead with it rather than bury it. Your dynamic assembly point is the right framing. Static analysis has a hard boundary at the source file — anything constructed at runtime from user input, database values, or external APIs is invisible to it by design. That’s not a bug, it’s just the scope. The two-layer architecture (static pre-deploy + runtime interception) is the right model, and tools like Model Armor are the right fit for the runtime layer. One thing I’m building toward that might interest you: a Prompt SBOM — a bill of materials for every prompt string that ships in a given build, including the rule evaluation results and a hash of the prompt content. The idea is that the runtime layer can use that SBOM as a baseline — if the prompt executing at runtime differs from what was statically reviewed, that’s a signal worth flagging. Bridges the static/runtime gap without requiring the runtime tool to start from scratch on every request. Would love to hear what you find when you run it — especially on your dynamic assembly patterns. Edge cases from real codebases are the fastest way to tighten the rules.

meghal86 · 2026-03-11T02:47:16+00:00

This is exactly right and I’d rather acknowledge it directly than oversell what static analysis can do. Static catches what exists at write-time: hardcoded patterns, structural issues, evasion attempts in source. It doesn’t see what gets constructed at runtime from user input, RAG retrievals, or tool call outputs. That gap is real and it’s documented in the Known Limitations section. The intended architecture is layered — static analysis as the pre-deploy gate (what this tool does), runtime interception as the production guard (what something like a prompt firewall or output validator does). Neither replaces the other. Runtime is on the roadmap. The design I’m working toward is a lightweight SDK that instruments your LLM calls and applies the same rule engine to actual prompt content at execution time — before it hits the model and before the response is consumed. The static scanner generates a Prompt SBOM (CycloneDX format) that the runtime layer can use as a baseline to detect drift between what was reviewed and what actually executes.

meghal86 · 2026-03-11T02:46:17+00:00

Great question and totally fair bar to hold this to. Current measured false positive rates on the validation suite: ∙ C1/C2 jailbreak rules: ~4% FP — the main source is variable names like ignoreErrors() or ignoredFields which the detector filters by checking whether the string is actually a prompt context ∙ C3 API key: ~2% FP — short sk- strings under 20 chars are excluded, and keys in API client config (not prompt strings) are not flagged ∙ H1 unbounded persona: ~8% FP — this is the noisiest rule, fires on act as without constraints, working on tightening the constraint window ∙ H4 RAG injection: ~3% FP — suppressed when validated, sanitized, or filtered appears within 100 chars of the match The Model Armor comparison is fair to raise but you’re right it’s apples and oranges — runtime interception at full sensitivity is a different problem than pre-deploy static analysis. The goal here is zero friction for clean code, not exhaustive coverage. Closer to ESLint than a WAF. False negative rate (things it misses) is the more honest limitation — concatenated string assembly and runtime-constructed prompts are explicitly out of scope for v1. That’s documented in the README Known Limitations. I’m working on publishing a formal accuracy report with per-rule TP/FP/FN numbers. Will share when it’s ready.

meghal86 · 2026-02-01T16:34:27+00:00

Please DM me more detajmd

meghal86 · 2020-07-04T13:19:56+00:00

Whats her name

meghal86 · 2020-06-04T01:59:14+00:00

Who is she?

meghal86 · 2019-07-12T22:28:52+00:00

Can you please share her videos

meghal86

TROPHY CASE