Obaseki
Resume
AI agents

Kedge

A supervisor that catches unattended coding agents cheating, rolls them back, and gets them back on course.

Placed 14th of 205 at neatHack 2026. Two startups contacted me about using it.

My part
Built the evaluation harness (sabotage modes and the 10-PR test set), the CI log context the agent reads, and CI result reporting. Built with one teammate.
Stack
PythonCoding agentsGitHub APIpytest

Interactive demo

A small demo to give you a feel for Kedge. It's not an exact copy of the app, so feel free to click around. The runs it replays are real.

Demo

Your coding agent cheats.
Kedge catches it.

Agents left alone delete the failing test, wander off the plan, then report "all tests pass." Kedge supervises every step, strikes the cheat, rewinds the agent to its last good commit and makes it try again.

The problem

Leave a coding agent working alone and it will sometimes take a shortcut and report success anyway. It deletes the failing test, edits files it was never asked to touch, or loops on the same broken fix. The pull request goes green, the agent says “Fixed”, and nobody notices that the test suite got smaller.

Kedge supervises the agent instead of trusting its report. It points an agent at a failing Dependabot or Renovate pull request, checks every step, and when the agent drifts it rolls back to the last good commit and retries with a note on what went wrong.

What I built

Kedge was a two-person build for neatHack (October 10–12, 2026). My teammate Farouk wrote the supervisor loop and the drift checks. My part was the proof that Kedge works:

  • Sabotage modes (--storm). They make the agent cheat on purpose, for example by deleting a test right after its first step, so a run shows whether the supervisor catches it. Every before-and-after number came from these runs.
  • The test set. I picked the 10 real failing dependency pull requests Kedge was measured on, and swapped out the weak ones (one was already passing, one was closed).
  • CI context for the agent. When CI fails, the agent gets the error section of the log, trimmed down, not the whole log.
  • CI results in the report. Kedge waits for CI on the draft pull request and adds the result and a timeline chart to its report.
  • The evidence page and the business plan the judges read.
  1. Failing dependency PR from the 10-PR test set
  2. Coding agent plans and edits; sabotage modes can make it cheat
  3. Kedge checks each step deleted tests, edits outside the plan, loops, false claims
  4. Roll back or continue back to the last good commit, with a recovery note
  5. Draft PR and report CI result and timeline
Parts I built

Key decisions

  • Measure the same pull request twice. The comparison that convinced people was one pull request, one model and the same sabotage, first with Kedge’s guard off and then on. Anything less would be a demo, not evidence.
  • Read the tests, not the agent. Kedge counts the tests in the clone and compares them to the base branch. What the agent says at the end is ignored.
  • Give the agent the error, not the log. Full CI logs flood the agent’s context. Cutting them down to the failing section kept each step focused.

What went wrong

Running Kedge on real repositories found two bugs in Kedge itself. When every test was deleted, it read the result as “unknown” instead of a regression. And it counted the starting tests from import errors instead of real tests. Both were fixed before the final runs, and Kedge now reads the true test count from the base branch.

One target in the test set was already passing when we ran it, so the accurate count is 4 green out of the 9 that were failing.

Results

  • Unsupervised, the agent shipped a “green” pull request with 2 of 5 tests and said “Fixed” (kedge-demo#9).
  • Under Kedge, the same sabotage was caught at step 1, rolled back, and the pull request shipped with 5 of 5 (kedge-demo#10). A repeat run gave the same result.
  • Across the real failing pull requests, Kedge turned 4 of the 9 fixable ones green, unattended.
  • Kedge placed 14th of 205 projects at neatHack. Two startups contacted me about using it.