README.md

osm-stats — Hugging Face bucket

This bucket is the canonical, citable copy of every artifact produced by the osm-stats project. It is updated together with the GitHub repository, and it is the place to look for a reproducible snapshot of the data referenced in the companion blog post OSM data analysis for environment and agriculture.

The bucket lives at hf://buckets/NoeFlandre/osm_stats. The web view is at https://huggingface.co/buckets/NoeFlandre/osm_stats.

What is in here

Folder	Purpose	Approx. size
caches/	The two thresholded sqlite caches the pipelines run on	51 MB
outputs/filter_first/	All artifacts from the filter-first path (TF-IDF + embeddings)	~28 MB
outputs/standardize_first/	All artifacts from the standardize-first path (TF-IDF + embeddings)	~28 MB
outputs/stats/	Global stats written by the CLI summary command	~2 KB

Element-type analysis

outputs/standardize_first/tfidf/element_type_stats.csv and outputs/standardize_first/embeddings/element_type_stats.csv carry the element-type breakdown of the manually-kept superclusters (157 for TF-IDF, 169 for embeddings) in two views, side by side:

• sc_* columns: per-supercluster view (noise excluded). One row per supercluster = one row per base key in the XLSX. Computed by src.core.db.supercluster_element_type_stats from the cluster_memberships CSV, joined to the source DB for element-type split. This is the primary view for the env/agri study: the kept base keys are superclusters, so the right unit of analysis is the cluster-membership rollup, not the global base-key rollup.
• src_* columns: per-base-key view (source-DB rollup). One row per base key, computed by src.core.db.element_type_stats from the source taginfo.sqlite. This is a flat pre-clustering view that includes source rows that ended up in noise.

The XLSX files (outputs/standardize_first/{tfidf,embeddings}/base_key_families.xlsx) have the same sc_* columns appended after the user's manual keep column. The keep column is preserved as-is.

TF-IDF pipeline — 157 kept out of 427 superclusters

The TF-IDF pipeline produced 8,832 real clusters + 78,270 noise points from the 225,684 standardized tags in the cache. The non-noise 147,414 tags cover 2,246,255,835 occurrences; the noise 78,270 tags cover another 1,122,085,693. Together the 225,684 tags in the cluster memberships file cover the 3,368,341,528 occurrences reported in the blog post.

	Tags	Occurrences	Real clusters
All cluster memberships (incl. noise)	225,684	3,368,341,528	8,832
Real clusters (noise excluded)	147,414	2,246,255,835	8,832
Noise (cluster_id = -1)	78,270	1,122,085,693	—
Real, by base-key label (157 yes kept)	14,858	1,167,476,227	859
Real, by base-key label (270 not-kept: 54 uncertain + 216 no)	132,556	1,078,779,608	7,973

The 157 "yes" labels cover 10.1 % of all real-cluster tags (14,858 / 147,414) and 52.0 % of all real-cluster occurrences (1,167,476,227 / 2,246,255,835), but only 9.7 % of the clusters (859 / 8,832). The kept superclusters are the high-volume ones (building, highway, landuse, natural, tiger, area, surface, water, wetland, etc.) — the long tail of small, specialized base keys was filtered out by the manual labeling.

Embeddings pipeline — 169 kept out of 433 superclusters

The embeddings pipeline (potion-base-8M) produced 4,954 real clusters + 106,498 noise points from the same 225,684 standardized tags. The non-noise 119,186 tags cover 2,565,137,600 occurrences; the noise 106,498 tags cover another 803,203,928.

	Tags	Occurrences	Real clusters
All cluster memberships (incl. noise)	225,684	3,368,341,528	4,954
Real clusters (noise excluded)	119,186	2,565,137,600	4,954
Noise (cluster_id = -1)	106,498	803,203,928	—
Real, by base-key label (169 yes kept)	17,612	978,614,046	511
Real, by base-key label (264 not-kept: 57 uncertain + 207 no)	101,574	1,586,523,554	4,443

The 169 "yes" labels cover 14.8 % of all real-cluster tags (17,612 / 119,186) and 38.2 % of all real-cluster occurrences (978,614,046 / 2,565,137,600), but only 10.3 % of the clusters (511 / 4,954).

Per-supercluster numbers (noise excluded, cluster-member rollup)

TF-IDF (157 kept)

	Polygon-friendly	Point-heavy	All 157
Base keys (superclusters)	110 / 157 (70.1 %)	47 / 157 (29.9 %)	157
Occurrences	1,061,684,644 (90.9 %)	105,791,583 (9.1 %)	1,167,476,227
Tags (cluster members)	12,504 (84.2 %)	2,354 (15.8 %)	14,858
Real clusters	670 (78.0 %)	189 (22.0 %)	859

Embeddings (169 kept)

	Polygon-friendly	Point-heavy	All 169
Base keys (superclusters)	118 / 169 (69.8 %)	51 / 169 (30.2 %)	169
Occurrences	798,138,011 (81.6 %)	180,476,035 (18.4 %)	978,614,046
Tags (cluster members)	15,683 (89.0 %)	1,929 (11.0 %)	17,612
Real clusters	359 (70.3 %)	152 (29.7 %)	511

is_polygon_friendly is (count_ways + count_relations) / count_all >= 0.5, exposed as POLYGON_FRIENDLY_THRESHOLD in src/core/db/element_type_stats.py. The CSVs do not address polygon size; that requires a PBF extract and a separate step.

These numbers are independently verified by:

• tests/core/db/test_supercluster_stats_audit.py (TF-IDF pipeline, 11 tests)
• tests/core/db/test_embeddings_supercluster_stats_audit.py (embeddings pipeline, 12 tests)

Each suite re-computes every metric two ways from the raw input files and asserts they match the function's output. | scripts/ | The 12 pipeline scripts, copied verbatim from scripts/ in the GitHub repo | ~30 KB | | reproducibility/ | A single shell script that rebuilds the bucket from a fresh checkout | ~1 KB | | MANIFEST.md | Per-file inventory: every file in the bucket, its size and a sha256 prefix | — | | SOURCE.md | Where the source data comes from, how to (re-)download it, what the caches are | — |

Total: 43 files, ~106 MB.

What is NOT in here (and why)

• The source taginfo.sqlite (14 GB). This is the raw input to the pipeline. It is a public dataset, re-downloadable from taginfo.openstreetmap.org, and would dominate the bucket size. See SOURCE.md for the download and extraction recipe.
• The .venv/, .git/, and __pycache__/ directories from the working tree.
• The 192 M-row tags source table. It is a verbatim copy of what is already in the public source DB.

How to use this bucket

Inspect a specific output

HF storage buckets are accessed via hf:// paths (not via the plain HTTP huggingface.co/... URLs of regular repos). Any fsspec-compatible library can read them directly:

import pandas as pd
df = pd.read_csv("hf://buckets/NoeFlandre/osm_stats/outputs/standardize_first/tfidf/base_key_families.csv")
print(df.head(20))

For binary files (XLSX, sqlite) the same path works through huggingface_hub.HfFileSystem:

from huggingface_hub import HfFileSystem
fs = HfFileSystem()
with fs.open("hf://buckets/NoeFlandre/osm_stats/outputs/standardize_first/tfidf/base_key_families.xlsx", "rb") as f:
    ...

DuckDB, pyarrow, polars and any other fsspec-aware tool are also supported; see the HF access patterns docs for the full list.

Re-run the pipeline from a fresh checkout

git clone https://github.com/NoeFlandre/osm-stats
cd osm-stats
# (follow SOURCE.md to get taginfo.sqlite and build the cache)
.venv/bin/python -m scripts.reproducibility.reproduce

The reproducibility/reproduce.sh script is a single command that:

1. Downloads taginfo.sqlite if not already on disk.
2. Builds the two caches.
3. Runs the four pipeline variants (TF-IDF and embeddings on each of the two caches).
4. Produces the base-key-family CSVs and XLSX files.
5. Verifies that the resulting artifact hashes match the bucket's MANIFEST.md.

Reproduce just the standardize-first outputs

.venv/bin/python -m scripts.build_cache_standardize_first
.venv/bin/python -m scripts.profile_clusters_standardize_first
.venv/bin/python -m scripts.profile_clusters_embeddings_standardize_first
.venv/bin/python -m scripts.save_base_key_families_standardize_first
.venv/bin/python -m scripts.compare_pipelines_standardize_first

The total wall time on an M-class laptop is roughly 10 minutes for the cache build + 4 minutes for TF-IDF + 40 minutes for the embeddings pipeline.

Provenance and timestamps

Step	Source of truth	Reproducible?
Source taginfo.sqlite	taginfo DB, data_until = 2026-06-09 00:59:09	yes, re-download
tag_features.sqlite (filter-first)	python -m src.cli --build-cache --threshold 500	yes
tag_features_standardize_first.sqlite	python -m scripts.build_cache_standardize_first	yes, but HDBSCAN is non-deterministic
All outputs/* CSVs / XLSX / MD	The scripts/*.py files in this bucket, in the order listed above	yes (with the same caveat)

The two caches were built on a 2026-06-16 snapshot of the source DB. Re-running the pipeline on a newer snapshot will produce a slightly different cache and slightly different cluster assignments; this is expected.

Companion resources

• Project GitHub: github.com/NoeFlandre/osm-stats
• Blog post: noeflandre.com/posts/osm-data-analysis
• Taginfo source: taginfo.openstreetmap.org/download
• Embedding model used: potion-base-8M by Minish Lab (via model2vec)

License

The source data is from OpenStreetMap, distributed under the ODbL. The code in this bucket is MIT (see the GitHub repo's LICENSE file).