v0.1 pilot archive
A repair you can inspect.
Preserved single-module pilot. Inspect recorded decisions, tests and patches; current research is available in ForgeBench .
Only curated tasks run in this workspace. No code upload or repository access.
Loading regression tasks
The task description and starting implementation will appear here.
Starting implementation
PythonWaiting for task data…Visible test cases —
- Test descriptions load with the selected task.
Ready when you are.
Start a repair or inspect a recorded run below.
- Public tests
- —
- Held-out tests
- —
- Repair attempts
- —
- Tokens
- —
- Est. cost
- —
- Elapsed
- —
Waiting for the hosted model response; recorded runs remain available.
The diff is the evidence.
Generated edits appear here after a model call. Test outcomes remain visible, including failures.
No test evaluation yet. Public tests guide the repair; held-out checks assess the final candidate.
- The execution trace will show real events as they arrive.
No candidate has been generated.
Inspect the evidence
Recent runs
Loading recorded runs…
Measured comparisons
The policy, under test.
Compare outcomes on the same authored tasks. A smaller cost only matters alongside the solve rate.
Policy evaluation
Results load from the recorded experiment artifacts.
| Policy | Runs / planned | Solved | Solve rate | Mean calls | Mean tokens | Mean est. cost | Mean time |
|---|---|---|---|---|---|---|---|
| Loading recorded measurements… | |||||||
Like for like
Paired task runs
Paired runs will appear when an evaluation is available.
Design & boundaries
Traceable by design.
A bounded experiment in choosing how much reasoning a repair needs.
The controller chooses an action.
The model proposes an edit.
The tests provide the evidence.
ForgeRL studies a finite decision policy around model calls. Its actions are a short-budget call, a deliberate call, replan or stop. It does not claim to fine-tune the language model.
Training and evaluation status is loaded from the running service. Results are reported only when recorded artifacts exist.
- 01
Observe
Read an authored task, its initial Python implementation and visible regression tests.
- 02
Choose
A fixed, heuristic or adaptive policy selects the next action from observable state and budget.
- 03
Repair
A model proposes a candidate. Execution is isolated and bounded by resource and call limits.
- 04
Evaluate
Report public and held-out checks separately. Keep the patch, failures, latency and recorded cost.
What this demonstrates
- Inspectable model-driven repair on a curated Python task suite.
- Explicit decision policies with a bounded inference budget.
- Trace exports linking an outcome to its task, candidate and execution events.
- Prospective policy comparison when evaluation artifacts are available.
What it does not establish
- Performance on arbitrary repositories or a standard external benchmark.
- That an adaptive policy is better before its held-out evaluation shows it.
- Security or correctness of generated code beyond the documented boundaries.
- Language-model fine-tuning, autonomous deployment or access to private code.
Training, validation and held-out task families are separated. The adaptive controller must be labeled as a fallback until a trained artifact is loaded. Negative results remain part of the experiment.
Cost estimates use $10 per million tokens for each endpoint; provider invoices may differ. Live runs contain curated tasks only. An essential session cookie and hashed network address enforce limits; quota identifiers expire after 48 hours.