# NER OSS Reference Benchmark

Generated at: `2026-06-05T14:33:28.688573+00:00`

## Headline Findings

- `English`: strongest average flat F1 is `Flair NER EN` at `0.720`; fastest average p50 is `spaCy EntityRecognizer XX` at `1.54 ms`.
- `Polish`: strongest average flat F1 is `Stanza NER PL` at `0.684`; fastest average p50 is `spaCy EntityRecognizer XX` at `6.31 ms`.
- Caveat: 21 long documents were truncated to 384 tokens (input length range 1217-2645).

## Machine

- Hostname: `MacBookPro`
- Platform: `macOS-26.1-arm64-arm-64bit`
- Machine: `arm64`
- Processor: `Apple M1 Pro`
- Logical CPUs: `8`
- Python: `3.12.10`

## Metric Glossary

- `matched`: number of provider entities matched against gold on the current track.
- `predicted`: number of entities emitted by the provider.
- `gold`: number of reference entities in the gold set for the current track.
- `precision`: `matched / predicted`.
- `recall`: `matched / gold`.
- `F1`: harmonic mean of `precision` and `recall`.
- `p50 ms`: median per-document latency measured on this machine in this run.
- `p95 ms`: 95th percentile per-document latency measured on this machine in this run.
- `status=completed`: provider ran on this dataset.
- `status=skipped`: provider was not run for this dataset, for example because of language support.
- `status=failed`: provider initialization or inference failed for this dataset.

## Benchmark Notes

- Dataset-level benchmark tables below use the benchmark-core labels: `PER`, `ORG`, `LOC`, `MISC`.
- Some official model scores come from native label spaces such as OntoNotes or from package-internal validation sets.
- The default shipped benchmark does not include a redistributable OntoNotes slice. Official OntoNotes numbers remain visible as external alignment metadata only.
- `WikiNER EN/PL` is sourced from the official Figshare release and uses a deterministic 10% hash-held-out test slice from the `wp3` corpora.
- `CoNLL-2003`, `WikiANN`, and `PolEval-2018` remain part of the benchmark, but their prepared artifacts stay in `data/local/` under the public-safe-only policy.
- `NKJP` appears only as provenance context for some Polish model package metrics. It is not an active benchmark dataset in the shipped configuration.

## Dataset Provenance & License Status

This section records the canonical source, license basis, public commit decision, and storage path for each active benchmark dataset.

| Dataset | Upstream | License | Commit Policy | Storage Path | Evidence | Notes |
| --- | --- | --- | --- | --- | --- | --- |
| CoNLL-2003 EN Test | https://trec.nist.gov/data/reuters/reuters.html | Reuters Organizational and Individual Agreements via NIST for Reuters Corpus Volume 1 source text | local-only | data/local/conll2003/<lang>/<split>.jsonl | https://trec.nist.gov/data/reuters/reuters.html ; https://aclanthology.org/W03-0419/ | The English CoNLL-2003 benchmark uses Reuters Corpus Volume 1 source text. A local community loader may be used only as a preparation convenience after lawful access. |
| WikiNER EN Test | https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 | CC BY 4.0 | tracked | data/prepared/wikiner/<lang>/<split>.jsonl | https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 ; https://creativecommons.org/licenses/by/4.0/ | The benchmark uses a deterministic 10% hash-held-out test slice from the official wp3 corpora. |
| WikiANN EN Test | https://elisa-ie.github.io/wikiann/ | ODC-By with additional research-use-only restriction on the official ELISA page | local-only | data/local/wikiann/<lang>/<split>.jsonl | https://elisa-ie.github.io/wikiann/ ; https://opendatacommons.org/licenses/by/ | Publicly downloadable, but not commit-safe under the repo's public-safe-only policy because the official source restricts use to research. |
| PolEval-2018 PL Test | https://gonito.net/challenge/poleval-2018-ner | No explicit public redistribution license found in the reviewed official materials as of 2026-03-15 | local-only | data/local/poleval2018/<lang>/<split>.jsonl | https://gonito.net/challenge/poleval-2018-ner ; https://mozart.ipipan.waw.pl/~axw/poleval2018/POLEVAL-NER_GOLD.json | Use the official public URLs for local fetch only. |
| WikiNER PL Test | https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 | CC BY 4.0 | tracked | data/prepared/wikiner/<lang>/<split>.jsonl | https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 ; https://creativecommons.org/licenses/by/4.0/ | The benchmark uses a deterministic 10% hash-held-out test slice from the official wp3 corpora. |
| WikiANN PL Test | https://elisa-ie.github.io/wikiann/ | ODC-By with additional research-use-only restriction on the official ELISA page | local-only | data/local/wikiann/<lang>/<split>.jsonl | https://elisa-ie.github.io/wikiann/ ; https://opendatacommons.org/licenses/by/ | Publicly downloadable, but not commit-safe under the repo's public-safe-only policy because the official source restricts use to research. |

### English Quality And Speed Snapshot

| Dataset | Best quality | F1 | p50 ms | Fastest | p50 ms | Fastest F1 |
| --- | --- | ---: | ---: | --- | ---: | ---: |
| CoNLL-2003 EN Test | Flair NER EN | 0.912 | 42.44 | spaCy EntityRecognizer XX | 1.38 | 0.604 |
| WikiNER EN Test | spaCy EntityRecognizer XX | 0.811 | 2.05 | spaCy EntityRecognizer XX | 2.05 | 0.811 |
| WikiANN EN Test | Flair NER EN | 0.491 | 34.81 | spaCy EntityRecognizer XX | 1.21 | 0.447 |

## English Benchmark Summary

Dataset cells report `flat F1`. Full `matched / predicted / gold / P / R / p50 / p95` details stay in the per-dataset sections below.

| Provider | Official Benchmark | Official F1 | CoNLL-2003 EN Test | WikiNER EN Test | WikiANN EN Test | Notes |
| --- | --- | --- | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer EN | OntoNotes 5 | 0.843 | 0.438 | 0.410 | 0.311 | Official spaCy package metric is on OntoNotes 5. The default shipped benchmark does not include a redistributable OntoNotes slice, so harness runs here focus on CoNLL, WikiNER and WikiANN. |
| spaCy EntityRecognizer XX | WikiNER | 0.831 | 0.604 | 0.811 | 0.447 | Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks. |
| Flair NER EN | CoNLL-03 | 0.929 | 0.912 | 0.757 | 0.491 | Official Flair fast English NER metric is on CoNLL-03. Harness tests the same model on CoNLL, WikiNER and WikiANN. |
| Flair NER EN OntoNotes | OntoNotes | 0.893 | 0.546 | 0.541 | 0.366 | Official Flair OntoNotes model metric is on OntoNotes. The default shipped benchmark does not include a redistributable OntoNotes slice, so this model is shown through CoNLL, WikiNER and WikiANN plus its published OntoNotes metric. |
| Stanza NER EN | OntoNotes | 0.888 | 0.495 | 0.537 | 0.379 | Default Stanza English package resolves to the OntoNotes NER model in this harness. The default shipped benchmark keeps only CoNLL, WikiNER and WikiANN as runnable public datasets. |
| GLiNER Multi | N/A | N/A | 0.357 | 0.351 | 0.269 | GLiNER is treated here as a multilingual zero-shot span baseline. The model card reports several benchmarks, but this harness keeps the comparison on the public EN and PL datasets above instead of collapsing to one official scalar. |

## English Official Alignment

This table compares a model's published native / official metric with the closest runnable dataset inside this harness. Missing harness values mean we do not have a like-for-like public slice wired yet.

| Provider | Official Benchmark | Official F1 | Harness Dataset | Harness F1 | Delta | Notes |
| --- | --- | ---: | --- | ---: | ---: | --- |
| spaCy EntityRecognizer EN | OntoNotes 5 | 0.843 | N/A | N/A | N/A | Official spaCy package metric is on OntoNotes 5. The default shipped benchmark does not include a redistributable OntoNotes slice, so harness runs here focus on CoNLL, WikiNER and WikiANN. |
| spaCy EntityRecognizer XX | WikiNER | 0.831 | WikiNER EN Test | 0.811 | -0.021 | Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks. |
| Flair NER EN | CoNLL-03 | 0.929 | CoNLL-2003 EN Test | 0.912 | -0.017 | Official Flair fast English NER metric is on CoNLL-03. Harness tests the same model on CoNLL, WikiNER and WikiANN. |
| Flair NER EN OntoNotes | OntoNotes | 0.893 | N/A | N/A | N/A | Official Flair OntoNotes model metric is on OntoNotes. The default shipped benchmark does not include a redistributable OntoNotes slice, so this model is shown through CoNLL, WikiNER and WikiANN plus its published OntoNotes metric. |
| Stanza NER EN | OntoNotes | 0.888 | N/A | N/A | N/A | Default Stanza English package resolves to the OntoNotes NER model in this harness. The default shipped benchmark keeps only CoNLL, WikiNER and WikiANN as runnable public datasets. |

### Polish Quality And Speed Snapshot

| Dataset | Best quality | F1 | p50 ms | Fastest | p50 ms | Fastest F1 |
| --- | --- | ---: | ---: | --- | ---: | ---: |
| PolEval-2018 PL Test | Stanza NER PL | 0.732 | 581.55 | spaCy EntityRecognizer XX | 16.01 | 0.372 |
| WikiNER PL Test | spaCy EntityRecognizer XX | 0.857 | 1.68 | spaCy EntityRecognizer XX | 1.68 | 0.857 |
| WikiANN PL Test | Stanza NER PL | 0.656 | 43.89 | spaCy EntityRecognizer XX | 1.25 | 0.610 |

## Polish Benchmark Summary

Dataset cells report `flat F1`. Full `matched / predicted / gold / P / R / p50 / p95` details stay in the per-dataset sections below.

| Provider | Official Benchmark | Official F1 | PolEval-2018 PL Test | WikiNER PL Test | WikiANN PL Test | Notes |
| --- | --- | --- | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | NKJP / package validation | 0.804 | 0.632 | 0.527 | 0.463 | Official spaCy package metric is on NKJP-backed package validation. No public span-level NKJP split is wired here yet, so Polish harness relies on PolEval, WikiNER and WikiANN. |
| spaCy EntityRecognizer XX | WikiNER | 0.831 | 0.372 | 0.857 | 0.610 | Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks. |
| Stanza NER PL | NKJP | 0.887 | 0.732 | 0.666 | 0.656 | Official Stanza Polish metric is on NKJP. Public span-level NKJP is not wired, so harness uses PolEval, WikiNER and WikiANN. |
| GLiNER Multi | N/A | N/A | 0.166 | 0.524 | 0.426 | GLiNER is treated here as a multilingual zero-shot span baseline. The model card reports several benchmarks, but this harness keeps the comparison on the public EN and PL datasets above instead of collapsing to one official scalar. ; 21 long documents were truncated to 384 tokens (input length range 1217-2645). |

## Polish Official Alignment

This table compares a model's published native / official metric with the closest runnable dataset inside this harness. Missing harness values mean we do not have a like-for-like public slice wired yet.

| Provider | Official Benchmark | Official F1 | Harness Dataset | Harness F1 | Delta | Notes |
| --- | --- | ---: | --- | ---: | ---: | --- |
| spaCy EntityRecognizer PL | NKJP / package validation | 0.804 | N/A | N/A | N/A | Official spaCy package metric is on NKJP-backed package validation. No public span-level NKJP split is wired here yet, so Polish harness relies on PolEval, WikiNER and WikiANN. |
| spaCy EntityRecognizer XX | WikiNER | 0.831 | WikiNER PL Test | 0.857 | +0.026 | Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks. |
| Stanza NER PL | NKJP | 0.887 | N/A | N/A | N/A | Official Stanza Polish metric is on NKJP. Public span-level NKJP is not wired, so harness uses PolEval, WikiNER and WikiANN. |

## CoNLL-2003 EN Test

Dataset ID: `conll2003-en-test`
Samples: `3453`
Supported dataset tracks: `flat, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer EN | completed | 3130 | 8657 | 5648 | 0.362 | 0.554 | 0.438 | 3.38 | 7.94 | - |
| spaCy EntityRecognizer XX | completed | 3400 | 5606 | 5648 | 0.606 | 0.602 | 0.604 | 1.38 | 2.89 | - |
| Flair NER EN | completed | 5142 | 5632 | 5648 | 0.913 | 0.910 | 0.912 | 42.44 | 183.01 | - |
| Flair NER EN OntoNotes | completed | 3756 | 8117 | 5648 | 0.463 | 0.665 | 0.546 | 42.74 | 177.34 | - |
| Stanza NER EN | completed | 3710 | 9357 | 5648 | 0.396 | 0.657 | 0.495 | 46.86 | 190.91 | - |
| GLiNER Multi | completed | 1562 | 3091 | 5648 | 0.505 | 0.277 | 0.357 | 59.96 | 83.47 | - |

### Overlap Track

`overlap` is not reported for this dataset because the prepared slice does not declare support for this track.

### Nested Track

`nested` is not reported for this dataset because the prepared slice does not declare support for this track.

## WikiNER EN Test

Dataset ID: `wikiner-en-test`
Samples: `14423`
Supported dataset tracks: `flat, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer EN | completed | 14560 | 39868 | 31216 | 0.365 | 0.466 | 0.410 | 5.30 | 9.09 | - |
| spaCy EntityRecognizer XX | completed | 25323 | 31244 | 31216 | 0.810 | 0.811 | 0.811 | 2.05 | 3.38 | - |
| Flair NER EN | completed | 23515 | 30881 | 31216 | 0.761 | 0.753 | 0.757 | 108.38 | 219.68 | - |
| Flair NER EN OntoNotes | completed | 19162 | 39563 | 31216 | 0.484 | 0.614 | 0.541 | 107.97 | 219.31 | - |
| Stanza NER EN | completed | 19576 | 41748 | 31216 | 0.469 | 0.627 | 0.537 | 117.37 | 234.43 | - |
| GLiNER Multi | completed | 7913 | 13811 | 31216 | 0.573 | 0.253 | 0.351 | 68.63 | 90.83 | - |

### Overlap Track

`overlap` is not reported for this dataset because the prepared slice does not declare support for this track.

### Nested Track

`nested` is not reported for this dataset because the prepared slice does not declare support for this track.

## WikiANN EN Test

Dataset ID: `wikiann-en-test`
Samples: `10000`
Supported dataset tracks: `flat, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer EN | completed | 4835 | 17137 | 13958 | 0.282 | 0.346 | 0.311 | 2.87 | 4.71 | - |
| spaCy EntityRecognizer XX | completed | 6573 | 15481 | 13958 | 0.425 | 0.471 | 0.447 | 1.21 | 1.92 | - |
| Flair NER EN | completed | 7094 | 14921 | 13958 | 0.475 | 0.508 | 0.491 | 34.81 | 86.46 | - |
| Flair NER EN OntoNotes | completed | 5973 | 18671 | 13958 | 0.320 | 0.428 | 0.366 | 35.19 | 88.95 | - |
| Stanza NER EN | completed | 6291 | 19208 | 13958 | 0.328 | 0.451 | 0.379 | 39.84 | 97.03 | - |
| GLiNER Multi | completed | 3000 | 8382 | 13958 | 0.358 | 0.215 | 0.269 | 59.96 | 72.44 | - |

### Overlap Track

`overlap` is not reported for this dataset because the prepared slice does not declare support for this track.

### Nested Track

`nested` is not reported for this dataset because the prepared slice does not declare support for this track.

## PolEval-2018 PL Test

Dataset ID: `poleval2018-pl-test`
Samples: `1828`
Supported dataset tracks: `flat, overlap, nested, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | completed | 18216 | 23707 | 33984 | 0.768 | 0.536 | 0.632 | 61.40 | 74.51 | - |
| spaCy EntityRecognizer XX | completed | 11752 | 29179 | 33984 | 0.403 | 0.346 | 0.372 | 16.01 | 19.34 | - |
| Stanza NER PL | completed | 21177 | 23896 | 33984 | 0.886 | 0.623 | 0.732 | 581.55 | 974.22 | - |
| GLiNER Multi | completed | 3187 | 4389 | 33984 | 0.726 | 0.094 | 0.166 | 307.28 | 449.45 | 21 long documents were truncated to 384 tokens (input length range 1217-2645). |

### Overlap Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | completed | N/A | N/A | N/A | N/A | N/A | N/A | 61.40 | 74.51 | Provider does not support this track. |
| spaCy EntityRecognizer XX | completed | N/A | N/A | N/A | N/A | N/A | N/A | 16.01 | 19.34 | Provider does not support this track. |
| Stanza NER PL | completed | N/A | N/A | N/A | N/A | N/A | N/A | 581.55 | 974.22 | Provider does not support this track. |
| GLiNER Multi | completed | 3368 | 4389 | 33984 | 0.767 | 0.099 | 0.176 | 307.28 | 449.45 | 21 long documents were truncated to 384 tokens (input length range 1217-2645). |

### Nested Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | completed | N/A | N/A | N/A | N/A | N/A | N/A | 61.40 | 74.51 | Provider does not support this track. |
| spaCy EntityRecognizer XX | completed | N/A | N/A | N/A | N/A | N/A | N/A | 16.01 | 19.34 | Provider does not support this track. |
| Stanza NER PL | completed | N/A | N/A | N/A | N/A | N/A | N/A | 581.55 | 974.22 | Provider does not support this track. |
| GLiNER Multi | completed | 0 | 0 | 14897 | 0.000 | 0.000 | 0.000 | 307.28 | 449.45 | 21 long documents were truncated to 384 tokens (input length range 1217-2645). |

## WikiNER PL Test

Dataset ID: `wikiner-pl-test`
Samples: `20456`
Supported dataset tracks: `flat, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | completed | 21229 | 43450 | 37119 | 0.489 | 0.572 | 0.527 | 6.38 | 10.64 | - |
| spaCy EntityRecognizer XX | completed | 31696 | 36855 | 37119 | 0.860 | 0.854 | 0.857 | 1.68 | 2.81 | - |
| Stanza NER PL | completed | 27144 | 44434 | 37119 | 0.611 | 0.731 | 0.666 | 104.34 | 234.27 | - |
| GLiNER Multi | completed | 15568 | 22267 | 37119 | 0.699 | 0.419 | 0.524 | 68.18 | 92.05 | - |

### Overlap Track

`overlap` is not reported for this dataset because the prepared slice does not declare support for this track.

### Nested Track

`nested` is not reported for this dataset because the prepared slice does not declare support for this track.

## WikiANN PL Test

Dataset ID: `wikiann-pl-test`
Samples: `10000`
Supported dataset tracks: `flat, surface`

### Flat Track

| Provider | Status | Matched | Predicted | Gold | Precision | Recall | F1 | p50 ms | p95 ms | Notes |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| spaCy EntityRecognizer PL | completed | 6672 | 13854 | 14960 | 0.482 | 0.446 | 0.463 | 4.66 | 7.16 | - |
| spaCy EntityRecognizer XX | completed | 9233 | 15295 | 14960 | 0.604 | 0.617 | 0.610 | 1.25 | 2.00 | - |
| Stanza NER PL | completed | 9897 | 15233 | 14960 | 0.650 | 0.662 | 0.656 | 43.89 | 107.18 | - |
| GLiNER Multi | completed | 4968 | 8391 | 14960 | 0.592 | 0.332 | 0.426 | 60.58 | 74.47 | - |

### Overlap Track

`overlap` is not reported for this dataset because the prepared slice does not declare support for this track.

### Nested Track

`nested` is not reported for this dataset because the prepared slice does not declare support for this track.