Test evidence
Criterio CISO · Information securityStatistical analysis of the tests
How Alexis, Guillermo and Amparo perform across their first round of testing, with the margin of every result and each failure with its origin. It is regenerated from the repository’s evidence with every round.
Most recent round: 1
At a glance
The most recent complete round, round 1, on the material's 6 entities: 100 of 100 seeded signals found (95% margin: 96.3–100%) and 0 extra; 48 of 48 data items extracted right (100%, 95% margin: 92.5–100%); and 2 of 2 control entities with no signal at all.
Across rounds 1–1, 100 of 100 signals found, 0 extra and 48 of 48 data items right. The rounds repeat the same material: they are not independent samples, and the margin is computed over each round's cases.
What was tested and how
On each entity, Alexis, Guillermo and Amparo run their setup and processing —the commands that process the documents and write the records—, with a token cap set before starting, on material whose key is kept apart and never seen. The other commands calculate on those records with the engine, which is tested on every change to the repository. Signals are compared by name and key; extraction, field by field, against what the written documents say. Control entities carry no seeded defect: any signal in them is a failure.
Coverage: of the seeded signals, how many were found. Precision: of those raised, how many were right. 95% margin: Wilson confidence interval over the cases.
- Rounds
- 1; the most recent complete one is round 1
- Material
- Banking · Colombia, Retail · Mexico, Technology · Chile, salud · Peru
- Control entities
- Control · Colombia, Control · Mexico
- Entity runs
- 6
- Seeded signals
- 100
- Data items graded
- 48
Cumulative results · rounds 1–1
| Measure | Cases | Result | 95% margin |
|---|---|---|---|
| Signals · coverage | 100 of 100 | 100% | 96.3–100% |
| Signals · precision | 100 of 100 | 100% | 96.3–100% |
| Extraction · data right | 48 of 48 | 100% | 92.5–100% |
| Control · with no signal | 2 of 2 | 100% | 34.2–100% |
Round by round
| Round | Entities | Signals · coverage | Signals · precision | Extraction | Control with no signal | Tokens | Defects |
|---|---|---|---|---|---|---|---|
| 1 | 6 | 100% | 100% | 100% | 2 of 2 | 31.9 M | 4 |
The most recent complete round · round 1
| Measure | Cases | Result | 95% margin |
|---|---|---|---|
| Signals · coverage | 100 of 100 | 100% | 96.3–100% |
| Signals · precision | 100 of 100 | 100% | 96.3–100% |
| Extraction · data right | 48 of 48 | 100% | 92.5–100% |
| Control · with no signal | 2 of 2 | 100% | 34.2–100% |
By industry and country · round 1
| Material | Agents | Signals | Extra | Extraction | Tokens |
|---|---|---|---|---|---|
| Banking · Colombia | Alexis, Amparo, Guillermo | 25 of 25 | 0 | 8 of 8 | 5.3 M |
| Retail · Mexico | Alexis, Amparo, Guillermo | 25 of 25 | 0 | 8 of 8 | 5.1 M |
| Technology · Chile | Alexis, Amparo, Guillermo | 25 of 25 | 0 | 8 of 8 | 5 M |
| salud · Peru | Alexis, Amparo, Guillermo | 25 of 25 | 0 | 8 of 8 | 6 M |
| Control · Colombia | Alexis, Amparo, Guillermo | 0 of 0 | 0 | 8 of 8 | 5.3 M |
| Control · Mexico | Alexis, Amparo, Guillermo | 0 of 0 | 0 | 8 of 8 | 5.2 M |
What failed, one by one · rounds 1–1
No failures.
| Round | Material | What | Where | Defect | Origin |
|---|
Defects
4 defects logged, each with its origin, its fix and the run that verified it: 4 verified and 0 to verify. 1 came from the material, the key or the test harness, not from the agents.
| Origin | Defects |
|---|---|
| Code | 2 |
| Test harness | 1 |
| Agent instructions | 1 |
| Agent | Defects |
|---|---|
| Amparo | 2 |
| Alexis | 2 |
| Guillermo | 2 |
Cost and next step
The 1 rounds used 31.9 million tokens: 31.9 million in the graded runs and 0 in the ones repeated after fixing a defect. On average, 5.3 million per entity.
The material is synthetic. Whether a version is stable or laboratory is said by this evidence, not its number: stable, with the floor of the 95% margin at 95% or more for signals and data over each entity's most recent run, and no signal in the controls (decision 0020).
Where every figure comes from
The analysis is generated by scripts/analisis_familias.py from tests/evidencias/familias/ciso/, and the repository’s verification checks on every change that it says what the evidence says. This page publishes it without recomputing anything.