Evaluation & source
AI HACKATHON 2026 TOKYO · Finalist · 56 teams
Give the model a structure the app can check
The photograph becomes a description of fields and records. The application checks that structure, then builds the screen.
Select a stage to follow the diagram. This is an illustration, not a live extraction.
How it works & what was checked
The model produces AppSpec, a JSON description of the application. AppSpec defines field types, table columns, the first record and its line items. React components interpret that description through a fixed renderer. This boundary was present in the initial implementation; it was an architectural choice from the beginning.
The application checks the completed output against its schema, including permitted keys and value types. It also rejects an empty application name or a spec with no fields. These checks happen after generation finishes, because a partially received object is naturally incomplete. A valid object still needs factual review: a schema can recognize that a field contains a number, but it cannot establish that the number was read correctly from the paper.
Changes stay inside the same boundary
After generation, a text instruction can request a change through SpecDiff. Its operation set includes adding a field, renaming one, setting a numeric limit, and adding an approval-step definition. The model selects among the supported operations; a deterministic reducer applies them to the existing spec. Rejected changes return the original spec. The application retains control of how an operation changes the interface.
Reading uncertainty has its own path. The live code finds the first field whose reported confidence is below its threshold and asks whether to include it. That question is an opportunity for human review, not evidence that the confidence number has been calibrated. Source boxes can show the region associated with a field, but their accuracy on live photographs was not quantitatively evaluated.
Correct values do not guarantee a complete document
Read the value counts and document verdicts together. Correct extracted values do not establish that a document is complete.
| Recorded metric | v1 | v2 |
|---|---|---|
| Documents passing the rubric | 38 / 50 | 48 / 50 |
| Partial / failed | 11 / 1 | 2 / 0 |
| Values scored correct | 704 / 707 | 1,516 / 1,518 |
| Values marked as fabricated | 0 | 0 |
How it works & what was checked
The first recorded evaluation scored 704 of 707 extracted values as correct. Yet only 38 of 50 documents received a successful verdict. Eleven were partial, and one live analysis fell back to demo data and was counted as a failure. The high value score concealed an important limitation: information that never reaches the extracted record cannot be recovered by checking only the values that did.
Repeated line items exposed that gap. The original representation was largely flat; the evaluation notes report that later rows were often omitted. The next version added a table definition and an array of rows. Output allowance and instructions for reading decimal separators also changed, so this was an engineering revision involving several changes. Its results cannot isolate the effect of the data-model change alone.
Read both passes together
Compare the two retained evaluations of the same 50 documents. These are saved grades, not a new extraction test.
Recorded verdict and value agreement for each case
| Case | v1 | v2 | v1 values | v2 values |
|---|---|---|---|---|
| K01 | Partial | Pass | 17 / 17 | 36 / 36 |
| K02 | Pass | Pass | 9 / 9 | 33 / 33 |
| K03 | Partial | Pass | 11 / 11 | 48 / 48 |
| K04 | Pass | Pass | 10 / 10 | 13 / 13 |
| K05 | Partial | Pass | 9 / 9 | 20 / 20 |
| K06 | Partial | Pass | 14 / 14 | 35 / 35 |
| K07 | Pass | Pass | 13 / 13 | 15 / 15 |
| K08 | Partial | Pass | 19 / 19 | 112 / 112 |
| K09 | Pass | Pass | 4 / 4 | 4 / 4 |
| K10 | Partial | Pass | 16 / 16 | 124 / 124 |
| K11 | Pass | Pass | 24 / 24 | 51 / 51 |
| K12 | Pass | Pass | 26 / 26 | 68 / 69 |
| K13 | Pass | Pass | 21 / 21 | 59 / 59 |
| K14 | Pass | Pass | 23 / 23 | 35 / 35 |
| K15 | Pass | Pass | 22 / 22 | 41 / 41 |
| K16 | Partial | Pass | 23 / 24 | 65 / 65 |
| K17 | Pass | Pass | 21 / 21 | 56 / 56 |
| K18 | Pass | Pass | 26 / 26 | 35 / 35 |
| K19 | Pass | Partial | 23 / 23 | 41 / 41 |
| K20 | Pass | Pass | 22 / 22 | 26 / 26 |
| K21 | Pass | Pass | 14 / 14 | 16 / 16 |
| K22 | Pass | Pass | 13 / 13 | 15 / 15 |
| K23 | Pass | Pass | 14 / 14 | 18 / 18 |
| K24 | Pass | Pass | 14 / 14 | 16 / 16 |
| K25 | Pass | Pass | 14 / 14 | 16 / 16 |
| K26 | Pass | Pass | 10 / 10 | 33 / 33 |
| K27 | Pass | Pass | 14 / 14 | 20 / 20 |
| K28 | Pass | Pass | 13 / 13 | 15 / 15 |
| K29 | Pass | Pass | 15 / 15 | 20 / 20 |
| K30 | Pass | Pass | 15 / 15 | 20 / 20 |
| K31 | Pass | Pass | 9 / 9 | 13 / 13 |
| K32 | Pass | Pass | 9 / 9 | 10 / 10 |
| K33 | Pass | Pass | 9 / 9 | 13 / 13 |
| K34 | Pass | Pass | 9 / 9 | 13 / 13 |
| K35 | Pass | Pass | 9 / 9 | 14 / 14 |
| K36 | Pass | Pass | 9 / 9 | 11 / 11 |
| K37 | Partial | Partial | 8 / 9 | 10 / 11 |
| K38 | Pass | Pass | 9 / 9 | 12 / 12 |
| K39 | Pass | Pass | 9 / 9 | 10 / 10 |
| K40 | Partial | Pass | 8 / 9 | 12 / 12 |
| K41 | Partial | Pass | 5 / 5 | 60 / 60 |
| K42 | Pass | Pass | 14 / 14 | 44 / 44 |
| K43 | Fail | Pass | Not evaluated | 17 / 17 |
| K44 | Pass | Pass | 15 / 15 | 22 / 22 |
| K45 | Pass | Pass | 18 / 18 | 35 / 35 |
| K46 | Pass | Pass | Not evaluated | Not evaluated |
| K47 | Pass | Pass | 20 / 20 | 22 / 22 |
| K48 | Partial | Pass | 16 / 16 | 36 / 36 |
| K49 | Pass | Pass | 21 / 21 | 22 / 22 |
| K50 | Pass | Pass | 18 / 18 | 34 / 34 |
How it works & what was checked
The retained v2 grades record 48 rubric passes, two partial results and no failed documents. They score 1,516 of 1,518 values as correct and record 196 expected and 196 extracted line items. Pairing the cases shows ten partial results and the failed run becoming successful. It also shows one regression from success to partial. That regression belongs in the comparison.
“Pass” means the stored rubric verdict. It does not mean every detail was correct: K12 passed with 68 of 69 values scored correct. Conversely, K19 was graded partial despite agreement on all 41 evaluated values, because the document verdict also considered omitted information. The case table preserves both columns so readers can see this distinction directly.
What these records establish
No fabricated values were recorded by the LLM graders in either pass. The combined denominator is 2,225 evaluated values across two passes over the same 50 documents. It is not a count of independent tests, and zero recorded detections is not a guarantee that the system cannot invent a value. Agreement between the graders and human reviewers was not measured.
The evaluation is recorded as 6 July 2026 on the direct provider path. The retained verdict files were recounted to produce this table. The original evaluation images are unavailable, and the grader model identifier was not preserved. A reader can recalculate this numerical projection, but cannot use it to regrade the documents or rerun the original set. Its performance figures also cannot be carried over to a different model or routed deployment without a new evaluation.
KAMIWAZA remains a prototype with a defined boundary. Approval steps are displayed as a design, without executing approvals or rejections. Generated data does not persist across reloads. Those gaps separate a working demonstration of the paper-to-application mechanism from a system ready for continuing operational use.
Evidence & method
Verification notes
The model marked no values as fabricated among 707 values in v1 and 1,518 in v2. This is the result for those documents; a rubric pass does not mean every field is correct.
Finalist among 56 teams.