Nebius × NVIDIA Global AI Hackathon / Agentic Engineering

A security agent that can prove it fixed your code.

Most AI code agents ask you to trust the diff. This one hands you a receipt. NVIDIA Nemotron writes the patch — then a deterministic verifier decides whether the patch is real: the finding's instance count must drop, no new finding may appear, and the target's tests must still pass. Anything else is rejected, and the failure goes to a stronger model.

See the receipt Run it yourself Source

Nothing here is simulated: every figure is from a run on 28 Oct 2026 at commit 30313a1, reproducible with one command. A browser cannot hold an API key, so this page shows recorded runs rather than a live button.

VEXAR-AUTOFIX

RECEIPT
NO. 0001

2026-10-28T13:05Z · commit 30313a1
Nebius Token Factory · NVIDIA Nemotron

vulnerable_server.py4 → 46

helper_delegation_server.py73 → 100

clean_server.py100 · untouched

VERIFIED FIXES7

REGRESSIONS0

MODEL CALLS18

TOTAL COST$0.0226

VERIFIED BY RESCAN · 0 NEW FINDINGS

PAID · DETERMINISTIC
2026-10-28 · $0.0226

30313A1 · 20261028 · NEBIUS/NVIDIA

The artefact this project exists to produce.

Rule instance count dropped

No new finding appeared

Target test suite still passes

Verified fixes

7/ 7 queued

Every queued patch cleared its rule and introduced nothing new.

Regressions

0tolerated

The eval exits non-zero if a patch ever trades one finding for another.

Model calls

183 tiers

Nano triages, Super patches, Ultra is consulted only after a failed check.

Cost of the run

$0.0226total

About a third of a cent per verified fix, at list price.

01The receipt

Three fixtures. A receipt for every file.

Hand-written fixtures with a declared rule set, so the denominator is known rather than estimated. Each run is produced by python eval/run_eval.py against NVIDIA Nemotron on Nebius Token Factory. Press Replay on any file to watch the pipeline walk the recorded figures stage by stage.

vulnerable_server.py 4 security rules

Sinks written directly in the tool body. Nine rules, twelve instances — four rules have a canonical mechanical remedy, five do not.

4 fixed 4 queued 7 refused 0 regressions

Four security rules, each with a canonical remedy: every confirmed finding was fixed and no new finding appeared. The score still reads F on purpose — the refusals are findings too, and the report never hides them to make a number look better.

Scan detail · 12 instances, 9 rules, from Vexar-MCP

Rule	Sev	Line	Tool	Scanner message	Has remedy
MCP001	error	23	run	Tool 'run' has an empty or overly brief description (0 chars; minimum 12 recommended).	refused
MCP008	error	23	run	Tool name 'run' is overly generic or violates `[a-z][a-z0-9_-]{2,63}`.	refused
MCP003	warning	23	run	Parameter 'cmd' in tool 'run' has no type annotation.	refused
MCP004	info	23	run	Parameter 'limit' in tool 'run' has no upper bound (`maximum` / `maxItems` / `Field(le=...)` / `min(...)`).	refused
MCP015	info	23	run	2 of 2 parameters in tool 'run' have no description: 'cmd', 'limit'.	refused
MCP005	error	25	run	Tool 'run' invokes dangerous execution sink `subprocess.run(shell=True)` with parameter 'cmd'.	canonical
MCP005	error	26	run	Tool 'run' invokes dangerous execution sink `os.system` with parameter 'cmd'.	canonical
MCP003	warning	31	fetch_and_save_remote_blob	Parameter 'payload' in tool 'fetch_and_save_remote_blob' uses unconstrained type 'Any'.	refused
MCP015	info	31	fetch_and_save_remote_blob	3 of 3 parameters in tool 'fetch_and_save_remote_blob' have no description: 'target_url', 'output_path', 'payload'.	refused
MCP006	warning	33	fetch_and_save_remote_blob	Tool 'fetch_and_save_remote_blob' passes parameter 'target_url' into outbound request `urllib.request.urlopen` without visible URL/host validation (potential SSRF).	canonical
MCP010	warning	33	fetch_and_save_remote_blob	Tool 'fetch_and_save_remote_blob' performs HTTP sink `urllib.request.urlopen` with parameter 'target_url' without an explicit `timeout=` argument.	canonical
MCP007	warning	35	fetch_and_save_remote_blob	Tool 'fetch_and_save_remote_blob' passes parameter 'output_path' to `open(write)` without `.resolve()` / base-directory confinement (path traversal risk).	canonical

"Has remedy" is a property of the rule, not a promise about this run: triage queued four of the five autofixable instances, and every patch it attempted verified.

Recorded run replays recorded figures · no model is called

Replay walks the recorded figures stage by stage — scan, triage, patch, verify, receipt.

helper_delegation_server.py sinks one hop away

The tool body looks clean; the dangerous call lives inside a module-local helper. This is the shape most production MCP servers actually have.

100

3 fixed 3 queued 0 refused 0 regressions

A scanner that only reads the tool body misses these sinks entirely. Following the delegation one hop into the helper is what turns a 73 into a 100 — with the helper's own confinement logic left intact, and read_note never flagged because its helper already confines the path.

Scan detail · 3 instances, 3 rules, from Vexar-MCP

Rule	Sev	Line	Tool	Scanner message	Has remedy
MCP005	error	41	run_backup	Tool 'run_backup' invokes dangerous execution sink `subprocess.run(shell=True)` with parameter 'command'.	canonical
MCP006	warning	45	download_report	Tool 'download_report' passes parameter 'url' into outbound request `urllib.request.urlopen` without visible URL/host validation (potential SSRF).	canonical
MCP010	warning	45	download_report	Tool 'download_report' performs HTTP sink `urllib.request.urlopen` with parameter 'url' without an explicit `timeout=` argument.	canonical

Recorded run replays recorded figures · no model is called

Replay walks the recorded figures stage by stage — scan, triage, patch, verify, receipt.

clean_server.py control

The control case. If the agent can be made to "fix" something that is already correct, every other receipt is worthless.

100

0 findings 0 edits

Clean already — 100/100. Nothing to fix, and the agent declines to touch the file. A tool that always finds something is not measuring anything.

Recorded run replays recorded figures · no model is called

Replay walks the recorded figures stage by stage — scan, triage, patch, verify, receipt.

02Why the verifier exists

The scanner says “cleared”. That is not the same as “safe”.

The first live run accepted four patches. Three were wrong — and the scanner accepted all of them, because it matches patterns, not intent. A verifier that reports “finding cleared, tests green” while handing back a broken tool is worse than no verifier at all: it manufactures confidence.

You are the verifier Four patches the model really produced on its first try. Accept or reject each one — then see what the deterministic audit said. Called correctly: 0 / 0

MCP005 shell execution — the model flipped the boolean SPECIMEN 1 / 4

          @@ def run(cmd, limit: int = 1000) -> str:
          -     subprocess.run(cmd, shell=True, check=False)
          -     os.system(cmd)
          +     subprocess.run(cmd, shell=False, check=False)
        

The rule is gone from the scanner's report. Do you accept the patch?

Rejected by the patch audit

shell=False on a single string. subprocess now looks for a file literally named after the whole command, so every command with arguments raises:

>>> subprocess.run('git status --short', shell=False)
FileNotFoundError: [Errno 2] No such file or directory: 'git status --short'

The scanner alone would have accepted this: shell=True is gone.

MCP006 SSRF — the model validated the URL's scheme SPECIMEN 2 / 4

          @@ def fetch_and_save_remote_blob(target_url, output_path, payload):
          +     if urllib.parse.urlsplit(target_url).scheme not in ("http", "https"):
          +         raise ValueError("unsupported URL scheme")
                with urllib.request.urlopen(target_url) as resp:
        

A scheme check is a real validation. Do you accept it as an SSRF fix?

Rejected by the patch audit

No host allowlist. http://169.254.169.254/latest/meta-data/ has a valid scheme and a valid host — the check returns True for the cloud metadata endpoint that SSRF attacks are actually aimed at. A fix here needs an allowlist of hosts, not a test of syntax.

MCP007 Path traversal — the model confined by string prefix SPECIMEN 3 / 4

          @@ def fetch_and_save_remote_blob(target_url, output_path, payload):
          +     resolved = Path(output_path).resolve()
          +     if not str(resolved).startswith(str(BASE_DIR)):
          +         raise ValueError("path escapes the base directory")
                with open(output_path, "w", encoding="utf-8") as fh:
        

It resolves the path and checks the base directory. Accept?

Rejected by the patch audit

String prefix is not a boundary. '/app-evil/secret'.startswith('/app') is True, so a sibling directory that shares the prefix walks straight through. The remedy is os.path.commonpath or Path(resolved).is_relative_to(base) — a path-component test, not a text test.

MCP005 the same rule, fixed the canonical way SPECIMEN 4 / 4

          @@ def run(cmd, limit: int = 1000) -> str:
          -     subprocess.run(cmd, shell=True, check=False)
          -     os.system(cmd)
          +     subprocess.run(shlex.split(cmd), check=False)
        

Smaller diff than specimen 1, and it adds a function call. Accept?

Accepted — all three gates pass

Patch audit: no anti-pattern, and the remedy marker (shlex.split) is present. Rescan: MCP005 instances for run drop 2 → 0, and no new finding appears. Tests: unchanged. One patch, two sinks, both provably gone.

The fix: a third deterministic stage that reads only the lines a patch added

Never the existing file — so pre-existing code cannot trigger a rejection — and it rejects remedies that cannot work. It cannot hallucinate, it costs nothing, and it encodes the review a security engineer would do anyway. Rejection is fail-closed: a retry costs a fraction of a cent, a wrong patch costs trust in every other patch the tool produces.

src/vexar_autofix/verifier.py

# Patterns look only at lines the patch ADDED, never the existing file.
SMELLS = {
    "MCP005": ((
        r"shell\s*=\s*False",
        "passing the command as one string; pass an argv list",
    ),),
    "MCP007": ((
        r"startswith\s*\(",
        "'<base>-evil' passes a check against '<base>'",
    ),),
}

# Absence is not proof of a bad patch, so markers are enforced only
# where the canonical remedy has a uniquely identifiable shape.
REMEDY_MARKERS = {
    "MCP006": (
        r"(ALLOWED_HOSTS|allowlist|hostname\s*(not\s+)?in\s)",
        "validating the scheme does not prevent SSRF",
    ),
}

Why not just ask a stronger model whether the patch is good?

Because that is the same failure one tier up: a model opinion, priced higher. This is a rule that returns the same verdict twice, runs in microseconds, and can be read and argued with by the person reviewing the diff.

Deleting the dangerous call is reported, not rejected

Sometimes deletion is right; sometimes it silently drops functionality. No mechanical check can tell which, so the report raises a review note and lets a human decide.

03How it works

A scanner on both sides of the model.

Five stages, three model tiers, one oracle. The scanner finds the sink, the models work on it, and the scanner is the thing that decides whether the work counted. It is the only participant the model cannot argue with.

deterministic

Scan

Vexar-MCP

Finds the sinks, follows one hop into local helpers, and scores the file.

no model · free

every finding

Triage

Nemotron-3 Nano 30B A3B

Exploitable, or already guarded? A classification, not a reasoning task.

$0.06 / $0.24 per 1M

per queue entry

Patch

Nemotron-3 Super 120B A12B

Writes the smallest remedy as an exact anchor edit — never a whole-file rewrite.

$0.30 / $0.90 per 1M

always

Verify

Rescan + patch audit + tests

Instance count dropped? Nothing new appeared? Suite still green? Fail-closed.

no model · free

only on failure

Escalate

Nemotron-3 Ultra 550B A55B

Receives the rejected patch and the exact reason verification failed.

$1.00 / $3.00 per 1M

The routing signal is the unusual part

Tiered demos usually route on request difficulty — a quantity nobody can measure before answering. This routes on verified failure: a signal that only exists because a deterministic checker is in the loop. The expensive model is never consulted on a guess, and it never has to work out what went wrong — the verifier tells it, in the words it will be graded against.

What gets auto-fixed — and what gets refused

Four rules have remedies mechanical enough to automate without judgement. Everything else is reported and left to a human.

MCP005shell execution on tool input

subprocess.run(cmd, shell=True) becomes subprocess.run(shlex.split(cmd), check=False)

The audit rejects shell=False on a single string: it silences the rule and breaks every call with arguments.

MCP006SSRF sink with no allowlist

urlopen(user_url) becomes host = urlsplit(url).hostname
if host not in ALLOWED_HOSTS: raise

The audit rejects scheme-only validation: the link-local metadata endpoint has a valid scheme and host.

MCP007path from tool input, unconfined

open(output_path, "w") becomes resolved = Path(output_path).resolve()
if not resolved.is_relative_to(BASE_DIR): raise

The audit rejects startswith and any base directory derived from the current working directory.

MCP010outbound HTTP without a timeout

urlopen(url) becomes urlopen(url, timeout=10)

An unbounded request is a denial-of-service waiting for a slow server; the fix is one keyword argument.

Property 1

Exact anchor edits

A patch either matches the source byte-for-byte or fails loudly. The model cannot silently reformat, reorder, or delete code it was not asked to touch.

Property 2

Working tree untouched

Every attempt runs against a throwaway copy in a temp directory. --apply is the only thing that writes, and only after verification.

Property 3

One finding per attempt

Each patch is small enough to review, a failure is attributable to one change, and partial progress survives an unsolvable rule.

04What it refuses

A confidently wrong fix is worse than a known finding.

Seven of the twelve findings in the vulnerable fixture are deliberately left alone. A tool description that is too short has no mechanical fix, and a model guessing at the author's intent produces a diff that looks plausible and is wrong. So the agent says so and moves on — and the report has a refused bucket so a skipped finding is never invisible.

Rule	Instances	Fate	Why it was left alone
MCP001	1	refused	Brief tool description — the remedy is wording, i.e. the author's intent.
MCP003	2	refused	Missing type annotations and an unconstrained `Any` parameter.
MCP004	1	refused	No upper bound on a parameter — a schema decision, not a bug fix.
MCP008	1	refused	Generic tool name — renaming changes the published tool contract.
MCP015	2	refused	Undocumented parameters; describing them is authorship, not repair.
7 refused instances · 4 security rules queued · 0 silently dropped

Every finding lands in exactly one bucket

Fixed, unfixed, refused, or skipped — the report accounts for all of them. An unaccounted finding is indistinguishable from a clean file, which would make the whole receipt meaningless.

The refusals are the honest part of the receipt

“Fixed 4 of 12” is a worse headline than “fixed 4, refused 7, hid nothing”. It is also the only version a security reviewer can act on.

05Evidence

Measured, not asserted.

Claims about an AI agent are cheap. Each fixture declares the exact rule set it contains — hand-written, so the denominator is known — and the eval reports findings, queued, fixed, refused and regressions per fixture. Any regression fails the run, so CI goes red instead of the demo looking good.

Fixture	Found	Queued	Fixed	Refused	Regressions	Score
vulnerable_server.py	9	4	4	7	0	4 → 46
helper_delegation_server.py	3	3	3	0	0	73 → 100
clean_server.py	0	0	0	0	0	100
Evaluated with NVIDIA Nemotron on Nebius Token Factory · 18 model calls · $0.0226 · exit non-zero on any regression

“Found” counts distinct rules; the vulnerable fixture contains twelve rule instances across those nine rules.

Run against real code, not only fixtures

A scanner that only fires on the deliberately-vulnerable fixture proves nothing about real code.

Nebius's own MCP server real-world

Python, MIT. Four tools across eight analysed files.

4 tools 8 files 1 finding 20/100 risk index no security findings

The single observation is MCP002 — context budget: one tool description consumes ~659 tokens against a 250-token budget. A well-built server, scored A. Included because a tool that never says “this code is fine” is not a scanner, it is a sales pitch.

Two bugs the receipts caught

Both were found by measuring, are pinned by a test, and both would have made a demo look great while being wrong.

Line-shifta perfect fix that read as six new findings

Findings were matched on (rule, tool, line). A patch that deleted two lines and added one renumbered everything below it, so a correct fix reported “1 cleared, 6 new findings appeared” and was rejected. Verification now compares rule-instance counts per tool, which survives reordering — and a targeted rule must actually drop.

Pinned by tests/test_verifier.py::test_line_shift_is_not_a_new_finding.

Self-comparereview notes that never surfaced

The review step compared the patched text with itself, because the variable had already been reassigned before the comparison. It silently reported nothing, forever. Fixed by capturing before_this_patch first; a deleted sink now produces a note a human sees next to the diff.

Both are the same lesson as the verifier itself: an output that is quiet is not evidence of correctness.

06Run it yourself

Everything on this page is four commands away.

The demo is static because a browser cannot hold an API key without exposing it — so instead of a button that lies about being live, every figure carries its timestamp and commit and can be regenerated locally.

bash

$ git clone https://github.com/RANJITHROSAN17/vexar-autofix
$ cd vexar-autofix && pip install -e .
$ export NEBIUS_API_KEY=...        # tokenfactory.nebius.com
$ vexar-autofix --preflight          # key, endpoint, routed models
$ vexar-autofix --target examples/vulnerable_server.py

Python 3.10+ No key? --provider mock --rehearse-accept Dry run by default · --apply to write Eval python eval/run_eval.py

What you will see, unedited

Triage verdicts with reasons, every attempt with its verdict and the exact failure text, the fixed / refused / skipped buckets, and a diff. If verification never passed, the tool prints not fixed — target file untouched instead of a patch.

Provenance

captured: 2026-10-28T13:05:00Z
commit: 30313a1
provider: Nebius Token Factory
triage: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B
patch: nvidia/nemotron-3-super-120b-a12b
escalate: nvidia/Nemotron-3-Ultra-550b-a55b
scanner: Vexar-MCP @ dd4c07b
figures: copied from terminal output, not estimated

The scanner detail tables are the deterministic half of the receipt: they are produced by the pinned scanner at that commit, so they reproduce identically on any machine. The model half is recorded — regenerate it with python eval/run_eval.py, or let .github/workflows/demo-capture.yml refresh this page when NEBIUS_API_KEY is a repository secret.