NER OSS Reference Benchmark

Generated at: 2026-06-05T14:33:28.688573+00:00

Headline Findings

English: strongest average flat F1 is Flair NER EN at 0.720; fastest average p50 is spaCy EntityRecognizer XX at 1.54 ms.
Polish: strongest average flat F1 is Stanza NER PL at 0.684; fastest average p50 is spaCy EntityRecognizer XX at 6.31 ms.
Caveat: 21 long documents were truncated to 384 tokens (input length range 1217-2645).

Machine

Hostname: MacBookPro
Platform: macOS-26.1-arm64-arm-64bit
Machine: arm64
Processor: Apple M1 Pro
Logical CPUs: 8
Python: 3.12.10

Metric Glossary

matched: number of provider entities matched against gold on the current track.
predicted: number of entities emitted by the provider.
gold: number of reference entities in the gold set for the current track.
precision: matched / predicted.
recall: matched / gold.
F1: harmonic mean of precision and recall.
p50 ms: median per-document latency measured on this machine in this run.
p95 ms: 95th percentile per-document latency measured on this machine in this run.
status=completed: provider ran on this dataset.
status=skipped: provider was not run for this dataset, for example because of language support.
status=failed: provider initialization or inference failed for this dataset.

Benchmark Notes

Dataset-level benchmark tables below use the benchmark-core labels: PER, ORG, LOC, MISC.
Some official model scores come from native label spaces such as OntoNotes or from package-internal validation sets.
The default shipped benchmark does not include a redistributable OntoNotes slice. Official OntoNotes numbers remain visible as external alignment metadata only.
WikiNER EN/PL is sourced from the official Figshare release and uses a deterministic 10% hash-held-out test slice from the wp3 corpora.
CoNLL-2003, WikiANN, and PolEval-2018 remain part of the benchmark, but their prepared artifacts stay in data/local/ under the public-safe-only policy.
NKJP appears only as provenance context for some Polish model package metrics. It is not an active benchmark dataset in the shipped configuration.

Dataset Provenance & License Status

This section records the canonical source, license basis, public commit decision, and storage path for each active benchmark dataset.

Dataset	Upstream	License	Commit Policy	Storage Path	Evidence	Notes
CoNLL-2003 EN Test	https://trec.nist.gov/data/reuters/reuters.html	Reuters Organizational and Individual Agreements via NIST for Reuters Corpus Volume 1 source text	local-only	data/local/conll2003/<lang>/<split>.jsonl	https://trec.nist.gov/data/reuters/reuters.html ; https://aclanthology.org/W03-0419/	The English CoNLL-2003 benchmark uses Reuters Corpus Volume 1 source text. A local community loader may be used only as a preparation convenience after lawful access.
WikiNER EN Test	https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500	CC BY 4.0	tracked	data/prepared/wikiner/<lang>/<split>.jsonl	https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 ; https://creativecommons.org/licenses/by/4.0/	The benchmark uses a deterministic 10% hash-held-out test slice from the official wp3 corpora.
WikiANN EN Test	https://elisa-ie.github.io/wikiann/	ODC-By with additional research-use-only restriction on the official ELISA page	local-only	data/local/wikiann/<lang>/<split>.jsonl	https://elisa-ie.github.io/wikiann/ ; https://opendatacommons.org/licenses/by/	Publicly downloadable, but not commit-safe under the repo's public-safe-only policy because the official source restricts use to research.
PolEval-2018 PL Test	https://gonito.net/challenge/poleval-2018-ner	No explicit public redistribution license found in the reviewed official materials as of 2026-03-15	local-only	data/local/poleval2018/<lang>/<split>.jsonl	https://gonito.net/challenge/poleval-2018-ner ; https://mozart.ipipan.waw.pl/~axw/poleval2018/POLEVAL-NER_GOLD.json	Use the official public URLs for local fetch only.
WikiNER PL Test	https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500	CC BY 4.0	tracked	data/prepared/wikiner/<lang>/<split>.jsonl	https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500 ; https://creativecommons.org/licenses/by/4.0/	The benchmark uses a deterministic 10% hash-held-out test slice from the official wp3 corpora.
WikiANN PL Test	https://elisa-ie.github.io/wikiann/	ODC-By with additional research-use-only restriction on the official ELISA page	local-only	data/local/wikiann/<lang>/<split>.jsonl	https://elisa-ie.github.io/wikiann/ ; https://opendatacommons.org/licenses/by/	Publicly downloadable, but not commit-safe under the repo's public-safe-only policy because the official source restricts use to research.

English Quality And Speed Snapshot

Dataset	Best quality	F1	p50 ms	Fastest	p50 ms	Fastest F1
CoNLL-2003 EN Test	Flair NER EN	0.912	42.44	spaCy EntityRecognizer XX	1.38	0.604
WikiNER EN Test	spaCy EntityRecognizer XX	0.811	2.05	spaCy EntityRecognizer XX	2.05	0.811
WikiANN EN Test	Flair NER EN	0.491	34.81	spaCy EntityRecognizer XX	1.21	0.447

English Benchmark Summary

Dataset cells report flat F1. Full matched / predicted / gold / P / R / p50 / p95 details stay in the per-dataset sections below.

Provider	Official Benchmark	Official F1	CoNLL-2003 EN Test	WikiNER EN Test	WikiANN EN Test	Notes
spaCy EntityRecognizer EN	OntoNotes 5	0.843	0.438	0.410	0.311	Official spaCy package metric is on OntoNotes 5. The default shipped benchmark does not include a redistributable OntoNotes slice, so harness runs here focus on CoNLL, WikiNER and WikiANN.
spaCy EntityRecognizer XX	WikiNER	0.831	0.604	0.811	0.447	Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks.
Flair NER EN	CoNLL-03	0.929	0.912	0.757	0.491	Official Flair fast English NER metric is on CoNLL-03. Harness tests the same model on CoNLL, WikiNER and WikiANN.
Flair NER EN OntoNotes	OntoNotes	0.893	0.546	0.541	0.366	Official Flair OntoNotes model metric is on OntoNotes. The default shipped benchmark does not include a redistributable OntoNotes slice, so this model is shown through CoNLL, WikiNER and WikiANN plus its published OntoNotes metric.
Stanza NER EN	OntoNotes	0.888	0.495	0.537	0.379	Default Stanza English package resolves to the OntoNotes NER model in this harness. The default shipped benchmark keeps only CoNLL, WikiNER and WikiANN as runnable public datasets.
GLiNER Multi	N/A	N/A	0.357	0.351	0.269	GLiNER is treated here as a multilingual zero-shot span baseline. The model card reports several benchmarks, but this harness keeps the comparison on the public EN and PL datasets above instead of collapsing to one official scalar.

English Official Alignment

This table compares a model's published native / official metric with the closest runnable dataset inside this harness. Missing harness values mean we do not have a like-for-like public slice wired yet.

Provider	Official Benchmark	Official F1	Harness Dataset	Harness F1	Delta	Notes
spaCy EntityRecognizer EN	OntoNotes 5	0.843	N/A	N/A	N/A	Official spaCy package metric is on OntoNotes 5. The default shipped benchmark does not include a redistributable OntoNotes slice, so harness runs here focus on CoNLL, WikiNER and WikiANN.
spaCy EntityRecognizer XX	WikiNER	0.831	WikiNER EN Test	0.811	-0.021	Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks.
Flair NER EN	CoNLL-03	0.929	CoNLL-2003 EN Test	0.912	-0.017	Official Flair fast English NER metric is on CoNLL-03. Harness tests the same model on CoNLL, WikiNER and WikiANN.
Flair NER EN OntoNotes	OntoNotes	0.893	N/A	N/A	N/A	Official Flair OntoNotes model metric is on OntoNotes. The default shipped benchmark does not include a redistributable OntoNotes slice, so this model is shown through CoNLL, WikiNER and WikiANN plus its published OntoNotes metric.
Stanza NER EN	OntoNotes	0.888	N/A	N/A	N/A	Default Stanza English package resolves to the OntoNotes NER model in this harness. The default shipped benchmark keeps only CoNLL, WikiNER and WikiANN as runnable public datasets.

Polish Quality And Speed Snapshot

Dataset	Best quality	F1	p50 ms	Fastest	p50 ms	Fastest F1
PolEval-2018 PL Test	Stanza NER PL	0.732	581.55	spaCy EntityRecognizer XX	16.01	0.372
WikiNER PL Test	spaCy EntityRecognizer XX	0.857	1.68	spaCy EntityRecognizer XX	1.68	0.857
WikiANN PL Test	Stanza NER PL	0.656	43.89	spaCy EntityRecognizer XX	1.25	0.610

Polish Benchmark Summary

Dataset cells report flat F1. Full matched / predicted / gold / P / R / p50 / p95 details stay in the per-dataset sections below.

Provider	Official Benchmark	Official F1	PolEval-2018 PL Test	WikiNER PL Test	WikiANN PL Test	Notes
spaCy EntityRecognizer PL	NKJP / package validation	0.804	0.632	0.527	0.463	Official spaCy package metric is on NKJP-backed package validation. No public span-level NKJP split is wired here yet, so Polish harness relies on PolEval, WikiNER and WikiANN.
spaCy EntityRecognizer XX	WikiNER	0.831	0.372	0.857	0.610	Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks.
Stanza NER PL	NKJP	0.887	0.732	0.666	0.656	Official Stanza Polish metric is on NKJP. Public span-level NKJP is not wired, so harness uses PolEval, WikiNER and WikiANN.
GLiNER Multi	N/A	N/A	0.166	0.524	0.426	GLiNER is treated here as a multilingual zero-shot span baseline. The model card reports several benchmarks, but this harness keeps the comparison on the public EN and PL datasets above instead of collapsing to one official scalar. ; 21 long documents were truncated to 384 tokens (input length range 1217-2645).

Polish Official Alignment

Provider	Official Benchmark	Official F1	Harness Dataset	Harness F1	Delta	Notes
spaCy EntityRecognizer PL	NKJP / package validation	0.804	N/A	N/A	N/A	Official spaCy package metric is on NKJP-backed package validation. No public span-level NKJP split is wired here yet, so Polish harness relies on PolEval, WikiNER and WikiANN.
spaCy EntityRecognizer XX	WikiNER	0.831	WikiNER PL Test	0.857	+0.026	Official spaCy multilingual package metric is on WikiNER. Harness runs the same family through public WikiNER EN/PL and broader WikiANN / CoNLL / PolEval checks.
Stanza NER PL	NKJP	0.887	N/A	N/A	N/A	Official Stanza Polish metric is on NKJP. Public span-level NKJP is not wired, so harness uses PolEval, WikiNER and WikiANN.

CoNLL-2003 EN Test

Dataset ID: conll2003-en-test Samples: 3453 Supported dataset tracks: flat, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer EN	completed	3130	8657	5648	0.362	0.554	0.438	3.38	7.94	-
spaCy EntityRecognizer XX	completed	3400	5606	5648	0.606	0.602	0.604	1.38	2.89	-
Flair NER EN	completed	5142	5632	5648	0.913	0.910	0.912	42.44	183.01	-
Flair NER EN OntoNotes	completed	3756	8117	5648	0.463	0.665	0.546	42.74	177.34	-
Stanza NER EN	completed	3710	9357	5648	0.396	0.657	0.495	46.86	190.91	-
GLiNER Multi	completed	1562	3091	5648	0.505	0.277	0.357	59.96	83.47	-

Overlap Track

overlap is not reported for this dataset because the prepared slice does not declare support for this track.

Nested Track

nested is not reported for this dataset because the prepared slice does not declare support for this track.

WikiNER EN Test

Dataset ID: wikiner-en-test Samples: 14423 Supported dataset tracks: flat, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer EN	completed	14560	39868	31216	0.365	0.466	0.410	5.30	9.09	-
spaCy EntityRecognizer XX	completed	25323	31244	31216	0.810	0.811	0.811	2.05	3.38	-
Flair NER EN	completed	23515	30881	31216	0.761	0.753	0.757	108.38	219.68	-
Flair NER EN OntoNotes	completed	19162	39563	31216	0.484	0.614	0.541	107.97	219.31	-
Stanza NER EN	completed	19576	41748	31216	0.469	0.627	0.537	117.37	234.43	-
GLiNER Multi	completed	7913	13811	31216	0.573	0.253	0.351	68.63	90.83	-

Overlap Track

overlap is not reported for this dataset because the prepared slice does not declare support for this track.

Nested Track

nested is not reported for this dataset because the prepared slice does not declare support for this track.

WikiANN EN Test

Dataset ID: wikiann-en-test Samples: 10000 Supported dataset tracks: flat, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer EN	completed	4835	17137	13958	0.282	0.346	0.311	2.87	4.71	-
spaCy EntityRecognizer XX	completed	6573	15481	13958	0.425	0.471	0.447	1.21	1.92	-
Flair NER EN	completed	7094	14921	13958	0.475	0.508	0.491	34.81	86.46	-
Flair NER EN OntoNotes	completed	5973	18671	13958	0.320	0.428	0.366	35.19	88.95	-
Stanza NER EN	completed	6291	19208	13958	0.328	0.451	0.379	39.84	97.03	-
GLiNER Multi	completed	3000	8382	13958	0.358	0.215	0.269	59.96	72.44	-

Overlap Track

overlap is not reported for this dataset because the prepared slice does not declare support for this track.

Nested Track

nested is not reported for this dataset because the prepared slice does not declare support for this track.

PolEval-2018 PL Test

Dataset ID: poleval2018-pl-test Samples: 1828 Supported dataset tracks: flat, overlap, nested, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer PL	completed	18216	23707	33984	0.768	0.536	0.632	61.40	74.51	-
spaCy EntityRecognizer XX	completed	11752	29179	33984	0.403	0.346	0.372	16.01	19.34	-
Stanza NER PL	completed	21177	23896	33984	0.886	0.623	0.732	581.55	974.22	-
GLiNER Multi	completed	3187	4389	33984	0.726	0.094	0.166	307.28	449.45	21 long documents were truncated to 384 tokens (input length range 1217-2645).

Overlap Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer PL	completed	N/A	N/A	N/A	N/A	N/A	N/A	61.40	74.51	Provider does not support this track.
spaCy EntityRecognizer XX	completed	N/A	N/A	N/A	N/A	N/A	N/A	16.01	19.34	Provider does not support this track.
Stanza NER PL	completed	N/A	N/A	N/A	N/A	N/A	N/A	581.55	974.22	Provider does not support this track.
GLiNER Multi	completed	3368	4389	33984	0.767	0.099	0.176	307.28	449.45	21 long documents were truncated to 384 tokens (input length range 1217-2645).

Nested Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer PL	completed	N/A	N/A	N/A	N/A	N/A	N/A	61.40	74.51	Provider does not support this track.
spaCy EntityRecognizer XX	completed	N/A	N/A	N/A	N/A	N/A	N/A	16.01	19.34	Provider does not support this track.
Stanza NER PL	completed	N/A	N/A	N/A	N/A	N/A	N/A	581.55	974.22	Provider does not support this track.
GLiNER Multi	completed	0	0	14897	0.000	0.000	0.000	307.28	449.45	21 long documents were truncated to 384 tokens (input length range 1217-2645).

WikiNER PL Test

Dataset ID: wikiner-pl-test Samples: 20456 Supported dataset tracks: flat, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer PL	completed	21229	43450	37119	0.489	0.572	0.527	6.38	10.64	-
spaCy EntityRecognizer XX	completed	31696	36855	37119	0.860	0.854	0.857	1.68	2.81	-
Stanza NER PL	completed	27144	44434	37119	0.611	0.731	0.666	104.34	234.27	-
GLiNER Multi	completed	15568	22267	37119	0.699	0.419	0.524	68.18	92.05	-

Overlap Track

overlap is not reported for this dataset because the prepared slice does not declare support for this track.

Nested Track

nested is not reported for this dataset because the prepared slice does not declare support for this track.

WikiANN PL Test

Dataset ID: wikiann-pl-test Samples: 10000 Supported dataset tracks: flat, surface

Flat Track

Provider	Status	Matched	Predicted	Gold	Precision	Recall	F1	p50 ms	p95 ms	Notes
spaCy EntityRecognizer PL	completed	6672	13854	14960	0.482	0.446	0.463	4.66	7.16	-
spaCy EntityRecognizer XX	completed	9233	15295	14960	0.604	0.617	0.610	1.25	2.00	-
Stanza NER PL	completed	9897	15233	14960	0.650	0.662	0.656	43.89	107.18	-
GLiNER Multi	completed	4968	8391	14960	0.592	0.332	0.426	60.58	74.47	-

Overlap Track

overlap is not reported for this dataset because the prepared slice does not declare support for this track.

Nested Track

nested is not reported for this dataset because the prepared slice does not declare support for this track.