ReconForge: lost on accuracy. Caught every HIGH.
Collection
Loses accuracy to DeepSeek v4-flash, wins severity-weighted recall 0.901 vs 0.872 at 1.000 on HIGH. Code: https://github.com/caiotheodoro/reconforge • 2 items • Updated
ReconForge Recon: financial reconciliation verdict model
LoRA adapter on mlx-community/Qwen3-1.7B-4bit. It takes one ledger entry and
one bank statement entry and returns a JSON verdict: MATCH, EXCEPTION (type,
severity), or ESCALATE. Trained in about 100 minutes on an Apple M5. Parse rate
is 1.000 on 800 ReconEval v0.1.0 tasks.
The eval is why this adapter is published. Severity-weighted recall (R_w) scores
the exception subset only. Flag precision and F1 score any non-MATCH prediction
over all 800 tasks, including the 419 clean pairs. Report both. Gold tasks:
caiotheodoro/recon-eval.
This checkpoint: accuracy 0.805, R_w 0.901, HIGH-severity recall 1.000, flag F1 0.824. Four training seeds, same data and hyperparameters, scored R_w 0.7457 / 0.8198 / 0.8353 / 0.9451. This file is the 0.9451. Across-seed SD is 0.0823; a retrain should land nearer 0.80. The other three runs miss HIGH-severity exceptions (operational cost 0.150 / 0.267 / 0.167; this run 0.000).
The training prompt is part of the weights. A prompt that defines exception classes instead of only naming them moves accuracy from 0.805 to 0.434 and flag precision from 0.824 to 0.508, while R_w rises to 0.940 because the model starts flagging clean pairs. Use the training prompt. Pair R_w with flag precision.
Threats to validity and per-seed results are in the source repo.
This adapter and the gold tasks live in the
ReconForge collection.
Examples (not live data): examples/match.json,
examples/amount_mismatch.json.
Quick start
# Block 1: load the adapter
# Requires: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11+ (pinned to this repo's
# training/eval environment; see reconforge/model/uv.lock)
from huggingface_hub import snapshot_download
from mlx_lm.lora import load
adapter_dir = snapshot_download("caiotheodoro/reconforge-recon-lora")
model, tokenizer = load(
"mlx-community/Qwen3-1.7B-4bit",
adapter_path=adapter_dir,
tokenizer_config={"trust_remote_code": True},
)
# Block 2: build the prompt
import json
SYSTEM_PROMPT = """You are ReconForge, a financial back-office reconciliation operations engine. \
You reconcile a single ledger entry against a single bank statement entry and return a structured verdict.
Your output MUST be exactly one JSON object with these keys:
- "verdict": "MATCH" | "EXCEPTION" | "ESCALATE"
- "exception_type": null or one of AMOUNT_MISMATCH, FX_CONVERSION_ERROR, \
BENEFICIARY_MISMATCH, COUNTERPARTY_MISMATCH, VALUE_DATE_MISMATCH, MISSING_MESSAGE, \
DUPLICATE, FIELD_CORRUPTION, PARTIAL_MATCH
- "severity": "LOW" | "MEDIUM" | "HIGH"
- "confidence": float in [0, 1]
- "reason": short reason, under 10 words
- "resolution": one of "auto-adjust", "escalate", "reject", "rebook", "flag-review"
"""
user_prompt = """Reconcile the following ledger entry against the bank statement.
LEDGER ENTRY:
{
"amount": "10000.00", "ccy": "USD", "counterparty": "Acme Corp",
"beneficiary": "Acme Corp", "value_date": "2024-01-15",
"message_type": "MT300", "ref": "OUR-REF-000001"
}
BANK STATEMENT:
{
"amount": "9999.50", "ccy": "USD", "counterparty": "Acme Corp",
"beneficiary": "Acme Corp", "value_date": "2024-01-15",
"message_type": "MT940", "ref": "CP-PAY-000001"
}
Return the verdict JSON object only."""
# Block 3: generate
from mlx_lm.generate import generate
for attr in ("has_thinking", "enable_thinking"):
if hasattr(tokenizer, attr):
setattr(tokenizer, attr, False)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_prompt},
]
prompt = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, enable_thinking=False, return_dict=False
)
response = generate(
model, tokenizer, prompt, max_tokens=128, # single greedy sample --
# a minimal usage example,
# NOT the eval config
)
verdict = json.loads(response)
print(f"{verdict['verdict']} / {verdict['exception_type']} / {verdict['severity']}")
This Quick start uses one greedy sample. It is a minimal usage example, not the eval configuration. The Results below were produced with self-consistency sampling (5 samples, temp=0.6, majority vote); see the Run configuration table. A single greedy sample will not reproduce the reported numbers.
The model is trained in non-thinking mode (
enable_thinking=False) and emits ~38 tokens per verdict on average. Leaving thinking enabled changes the output format and breaks the parse.
Results
Eval set: ReconEval v0.1.0, 800 held-out tasks, seed 777, exact-overlap
contamination 0/800 (near-duplicate rate 3.6% at Jaccard ≥ 0.8; see
recon-eval).
Selection caveat. Every checkpoint and configuration choice behind these
results (run-1 vs the B2 rebalanced-mix run, ×3 vs ×5 self-consistency
sampling) was made by comparing on this same seed-777 800-task set that the
headline metrics are then reported on. It was never held out from selection. A
newly designated seed (999) is frozen as the test set: its task signatures are
committed in docs/validation/frozen-test-seed-999-signatures.json and nothing
has been scored against it. Selection discipline going forward is in
docs/TRAINING.md (Selection policy).
All CIs are 95% bootstrap intervals, 10,000 resamples, seed 11, over the same
800-task set per model (reconforge/model/scripts/intervals.py for accuracy /
R_w / HIGH recall; reconforge/model/scripts/rescore_flag_metrics.py for flag
precision / F1 / severity-weighted cost).
R_w is severity-weighted recall over the exception subset only (381 of 800
tasks). A "flag" is any predicted verdict ≠MATCH, scored over all 800 tasks.
R_w cannot see false positives on clean pairs: a degenerate always-ESCALATE
model scores R_w 0.6673 on this benchmark at accuracy 0.0
(docs/BENCHMARK.md). Parse failures count as misses and stay in the
denominator (reconforge_model/metrics.py).
| Model | Params | Accuracy [95% CI] | R_w [95% CI] | Flag precision [95% CI] | Flag F1 [95% CI] | HIGH Recall [95% CI] | Norm. cost [95% CI] | Parse | Cost |
|---|---|---|---|---|---|---|---|---|---|
| ReconForge Recon | 1.7B | 0.805 [0.778, 0.833] | 0.901 [0.876, 0.924] | 0.824 [0.784, 0.862] | 0.824 [0.793, 0.853] | 1.000 [1.000, 1.000] | 0.000 [0.000, 0.000] | 1.000 | $0 |
| DeepSeek v4-flash | n/a | 0.876 [0.853, 0.899] | 0.872 [0.843, 0.899] | 1.000 [1.000, 1.000] | 0.904 [0.880, 0.926] | 0.995 [0.983, 1.000] | 0.013 [0.001, 0.031] | 0.996 | API (per-token; not independently verified here) |
| Base Qwen3-1.7B | 1.7B | 0.678 [0.645, 0.710] | 0.600 [0.548, 0.650] | 0.852 [0.809, 0.894] | 0.717 [0.677, 0.755] | 0.875 [0.825, 0.921] | 0.145 [0.089, 0.206] | 0.999 | $0 |
DeepSeek has zero false positives on the 419 clean pairs (flag F1 0.904
[0.880, 0.926]). This run over-flags 67 clean pairs (flag F1 0.824 [0.793,
0.853]). Severity-weighted cost (cost_esc = 1.0 per review, cost_missed_high
= 5.0 per missed HIGH; metrics.severity_weighted_cost defaults) is 0.000 on
this run (no escalations, no missed HIGH) and 0.013 for DeepSeek.
Paired bootstrap (same 800 tasks, 10,000 resamples):
| Comparison | Difference | 95% CI | Significant (α=0.05) |
|---|---|---|---|
| Accuracy (Recon − DeepSeek) | −0.071 | [−0.096, −0.046] | Yes (DeepSeek higher) |
| R_w (Recon − DeepSeek) | +0.029 | [0.008, 0.052] | Yes (ReconForge Recon higher) |
Run configuration:
| ReconForge Recon | DeepSeek v4-flash | Base Qwen3-1.7B | |
|---|---|---|---|
| Temperature | 0.6 | 0.0 | 0.0 |
| top_p | 1.0 | (provider default) | 1.0 |
| max tokens | 256 | 1024 | 256 |
| Samples per task | 5 (self-consistency, majority vote) | 1 | 1 (greedy) |
| Prompt version | v1 |
v1 |
v1 |
| Thinking mode | off | n/a | off |
| Eval date | 2026-08-08 | 2026-08-08 | 2026-08-08 |
| Model revision | adapters/champion @ this repo |
API snapshot 2026-08-08 (provider exposes no pinned revision id) | mlx-community/Qwen3-1.7B-4bit, no adapter |
Full machine-readable run configs: docs/validation/runconfig.json in the
source repo.
ECE for ReconForge Recon is 0.0875 (self-consistency ECE, from the champion eval run). DeepSeek and base Qwen do not use self-consistency sampling in this eval, so a comparable ECE was not computed for them here. Reported as not available rather than estimated.
Matched prompt (exploratory)
The table above uses the training prompt (v1) and unmatched sample budgets
(this adapter ×5, DeepSeek ×1). Giving both models a prompt that defines
exception classes, at ×5, is a different experiment: it measures prompt
lock-in, not a published head-to-head.
On seed 777, PARTIAL_MATCH surface pairs overlap training completely (fraction
1.0), which favours this adapter. The comparison is exploratory; the
pre-registered confirmatory pool is seed 999 (docs/BENCHMARK.md,
docs/validation/matched-comparison.json).
| Condition | Accuracy | R_w | Flag precision | Flag F1 |
|---|---|---|---|---|
| Champion, training prompt, ×5 (this card) | 0.805 | 0.901 | 0.824 | 0.824 |
| Champion, rubric prompt, ×5 | 0.434 | 0.940 | 0.508 | 0.665 |
| DeepSeek, rubric prompt, ×5 | 0.949 | 0.969 | 1.000 | 0.960 |
The rubric prompt is the same change described in the lede: this adapter's accuracy and flag precision drop; R_w rises because it over-flags clean pairs. That is why R_w is paired with flag F1.
Per-exception recall (ReconForge Recon)
Exact verdict+type match recall, with 95% bootstrap CI, grouped by the expected exception type:
| Exception Type | Recall [95% CI] | n | R_w weight |
|---|---|---|---|
| AMOUNT_MISMATCH | 1.000 [1.000, 1.000] | 73 | 1.0 |
| FX_CONVERSION_ERROR | 1.000 [1.000, 1.000] | 32 | 1.0 |
| BENEFICIARY_MISMATCH | 1.000 [1.000, 1.000] | 42 | 0.9 |
| MISSING_MESSAGE | 1.000 [1.000, 1.000] | 45 | 0.6 |
| COUNTERPARTY_MISMATCH | 0.865 [0.744, 0.970] | 37 | 0.9 |
| VALUE_DATE_MISMATCH | 0.692 [0.564, 0.814] | 52 | 0.6 |
| PARTIAL_MATCH | 0.688 [0.517, 0.846] | 32 | 0.5 |
| FIELD_CORRUPTION | 0.270 [0.133, 0.421] | 37 | 0.2 |
| DUPLICATE | 0.000 [0.000, 0.000] | 31 | 0.2 |
R_w = 0.901 with two recall figures below 0.3 (FIELD_CORRUPTION, DUPLICATE) because those types carry the lowest weights. Without the weight column the headline looks inflated. At n = 31–52 the CIs are wide: COUNTERPARTY_MISMATCH 0.865 sits in [0.744, 0.970]. Treat single decimal-place differences between runs at this sample size as noise.
Training
| Parameter | Value |
|---|---|
| Base model | mlx-community/Qwen3-1.7B-4bit |
| Method | MLX-LoRA |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | All linear layers (16 layers) |
| Optimizer | AdamW |
| Learning rate | 1e-5 |
| Batch size | 2 |
| Max sequence length | 2048 |
| Grad checkpointing | Enabled |
| Training-run seed | 7 (MLX-LoRA run; distinct from generation seed 101 and split seed 7; see note) |
| Iterations | ran to step 740 (early stop at plateau); champion = last persisted checkpoint iter 700 (adapter saves every 50 steps) |
| Champion adapter SHA-256 | 4754fe569b703f075725f0415a9ae70664fda6d7d66a865e956be2ff69bacdfa (== adapters/lora-full/0000700_final.safetensors) |
| Train loss | 2.4 → 0.088 |
Compute: Apple M5, 16 GB unified memory. Wall time ~100 min (740 iterations). Peak memory 3.346 GB. No GPU cluster, no distributed training. macOS version was not recorded.
Framework versions: mlx==0.32.0, mlx-lm==0.31.3, Python 3.11.15
(from reconforge/model/uv.lock).
CO2: CO2 was not measured. About 100 minutes of M5 package power on a
laptop-class chip; co2_eq_emissions was not computed.
Training data: 3,198 synthetic reconciliation pairs (train) + 802 (val),
from generation seed 101 (regenerating the pool at seed 101 reproduces
data/train.jsonl and data/val.jsonl byte-for-byte; seeds 7 and 777 do
not). Stratified split by (difficulty decile, exception type) under split
seed 7. Three seeds are in play and were previously conflated: generation
seed 101, split seed 7, training-run seed 7; see ../docs/TRAINING.md.
Contamination guard: SHA-256 field-level pair signatures, zero exact overlap
between train and eval (re-confirmed under generation seed 101: overlap 0.0);
see recon-eval
for the full audit including the near-duplicate rate this exact-hash check
does not cover.
Ablation: DUPLICATE recall across two checkpoints
| Run | Train pairs | Self-consistency | DUPLICATE recall | R_w | Accuracy |
|---|---|---|---|---|---|
| champion (iter 700) | 3,198 | x5 | 0.000 (0/31) | 0.901 | 0.805 |
| b2 (590 iters) | 3,201 | x3 | 0.032 (1/31) | 0.723 | 0.769 |
(Flag precision / F1 for the b2 run was not re-scored; outside issue #8's scope. The champion ×5 run's flag F1 is 0.824 [0.793, 0.853]; see Results.)
Both runs used almost identical training-pair counts (3,198 vs 3,201), so this
pair does not isolate a data-scale effect. b2 also differs in checkpoint (590
vs 700 persisted iterations) and eval self-consistency (x3 vs x5), both of
which move R_w and accuracy on their own. At comparable data volume, DUPLICATE
recall stayed at or near zero across both checkpoints (0/31 and 1/31), while
every other metric moved between the two runs. That is consistent with the
discriminating signal for DUPLICATE (statement.reference == ledger.reference
while all other fields match) being a single exact-equality predicate the
model does not reliably attend to, rather than a data-volume problem. It does
not prove that. A clean data-scaling ablation (same checkpoint, different
train-set sizes) has not been run; treat "architecture/scale limit, not data
limit" as a hypothesis this ablation is consistent with. DUPLICATE detection
is free and exact with a deterministic reference-equality pre-check; see
Out-of-scope use.
Uses
Direct use: classifying ledger↔statement reconciliation pairs into MATCH / EXCEPTION(type, severity) / ESCALATE with a structured JSON verdict.
Downstream use: as one scorer inside a larger reconciliation pipeline, behind a rule pre-check and in front of a human review queue.
Out-of-scope use (read this before deploying):
- Duplicate detection: recall is 0.000 on the eval set. Use a deterministic reference-equality check. Routing duplicates to this model means missing (nearly) all of them.
- Uncertainty signalling: the model emits ESCALATE zero times in this eval. It has no demonstrated way to say "unsure." Every HIGH-severity verdict must route to human review unconditionally. That is a deployment requirement.
- Live financial data: trained on synthetic pairs only. Performance on real ledgers is unmeasured.
- Autonomous judging: the production recalibration judge is DeepSeek +
JUDGE_SYSTEM_PROMPT(the rubric-extended prompt), which reaches Cohen's kappa 0.9037 against the oracle on the 100-task golden set. Its 95% bootstrap CI over that set is [0.828, 0.968], whose lower bound is below the 0.85 threshold, so the point estimate clears the bar but the data at n=100 cannot distinguish this judge from one sitting exactly at it. This adapter is not that judge. Asked to self-judge under the same rubric prompt it was never trained on, its kappa is 0.367 (a regression from 0.736 on the bare prompt; extended rubric text is off-distribution for a 1.7B fine-tuned on one fixed prompt; seedocs/DECISIONS.mdC2). Do not use this adapter for unsupervised judging. - Any non-reconciliation domain: single-domain by construction; does not transfer to payment repair or settlement.
Limitations and bias
- Synthetic data only. Generated from a parameterized oracle, not sampled from production systems. Real reconciliation traffic has correlations, seasonality, and counterparty long tails this data does not model. Treat the absolute numbers as an upper bound.
- DUPLICATE recall ~0. See Ablation above; not fixable by the data-volume change tested so far.
- Zero escalations observed in this eval run; see Out-of-scope use.
- Judge calibration gap. The production judge (DeepSeek +
JUDGE_SYSTEM_PROMPT) is at kappa 0.9037 [0.828, 0.968] vs oracle, above the 0.85 bar on the point estimate, with an interval that includes values below it, so "calibrated" is not yet demonstrable at n=100. This adapter self-judging under the same rubric is at kappa 0.3672 [0.270, 0.467] (see Autonomous judging above anddocs/DECISIONS.mdC2). ECE 0.0875 self-consistency; treat any single-sample deployment as uncalibrated relative to this number, which was measured under 5-sample self-consistency. - Single-domain. No transfer evidence to payment repair or settlement.
- Generator bias. The eval set inherits every bias of the generator, including its exception-type mix and difficulty distribution. A model tuned to that generator will look better than it is on any other distribution. The near-duplicate audit (3.6% at Jaccard ≥ 0.8) bounds train/eval leakage; it says nothing about generalization beyond the generator's distribution.
Card metadata
- Authors: Caio Theodoro
- Contact: via the HF repo discussions tab
- Repository: https://github.com/caiotheodoro/reconforge
Synthetic data. All tasks are generated, not drawn from live financial systems. No real counterparties, account identifiers, or personal data are present.
Not production-validated. Nothing here is financial advice or a validated control. Any deployment touching real money requires independent validation and a human-in-the-loop review path for high-severity cases.
Citation
@misc{theodoro2026reconforge,
title = {ReconForge: Severity-Weighted Evaluation for Financial Reconciliation Agents},
author = {Caio Theodoro},
year = {2026},
url = {https://github.com/caiotheodoro/reconforge},
note = {LoRA adapter for Qwen3-1.7B, Apache-2.0}
}
Hardware compatibility
Log In to add your hardware
Quantized