# osm-stats — Hugging Face bucket

This bucket is the canonical, citable copy of every artifact produced
by the [osm-stats](https://github.com/NoeFlandre/osm-stats) project.
It is updated together with the GitHub repository, and it is the place
to look for a reproducible snapshot of the data referenced in the
companion blog post
[OSM data analysis for environment and agriculture](https://noeflandre.com/posts/osm-data-analysis).

The bucket lives at
`hf://buckets/NoeFlandre/osm_stats`. The web view is at
<https://huggingface.co/buckets/NoeFlandre/osm_stats>.

## What is in here

| Folder | Purpose | Approx. size |
|---|---|---|
| `caches/` | The two thresholded sqlite caches the pipelines run on | 51 MB |
| `outputs/filter_first/` | All artifacts from the **filter-first** path (TF-IDF + embeddings) | ~28 MB |
| `outputs/standardize_first/` | All artifacts from the **standardize-first** path (TF-IDF + embeddings) | ~28 MB |
| `outputs/stats/` | Global stats written by the CLI summary command | ~2 KB |

### Element-type analysis

`outputs/standardize_first/tfidf/element_type_stats.csv` and
`outputs/standardize_first/embeddings/element_type_stats.csv`
carry the element-type breakdown of the manually-kept
superclusters (157 for TF-IDF, 169 for embeddings) in two views,
side by side:

- `sc_*` columns: **per-supercluster** view (noise excluded). One
  row per supercluster = one row per base key in the XLSX.
  Computed by `src.core.db.supercluster_element_type_stats` from
  the cluster_memberships CSV, joined to the source DB for
  element-type split. This is the primary view for the env/agri
  study: the kept base keys are superclusters, so the right
  unit of analysis is the cluster-membership rollup, not the
  global base-key rollup.
- `src_*` columns: **per-base-key** view (source-DB rollup). One
  row per base key, computed by `src.core.db.element_type_stats`
  from the source `taginfo.sqlite`. This is a flat pre-clustering
  view that includes source rows that ended up in noise.

The XLSX files (`outputs/standardize_first/{tfidf,embeddings}/base_key_families.xlsx`)
have the same `sc_*` columns appended after the user's manual
`keep` column. The `keep` column is preserved as-is.

#### TF-IDF pipeline — 157 kept out of 427 superclusters

The TF-IDF pipeline produced 8,832 real clusters + 78,270 noise
points from the 225,684 standardized tags in the cache. The
non-noise 147,414 tags cover 2,246,255,835 occurrences; the noise
78,270 tags cover another 1,122,085,693. Together the 225,684
tags in the cluster memberships file cover the 3,368,341,528
occurrences reported in the blog post.

| | Tags | Occurrences | Real clusters |
|---|---:|---:|---:|
| All cluster memberships (incl. noise) | **225,684** | 3,368,341,528 | 8,832 |
| &nbsp;&nbsp;Real clusters (noise excluded) | 147,414 | 2,246,255,835 | 8,832 |
| &nbsp;&nbsp;Noise (cluster_id = -1) | 78,270 | 1,122,085,693 | — |
| Real, by base-key label (157 yes kept) | **14,858** | **1,167,476,227** | **859** |
| Real, by base-key label (270 not-kept: 54 uncertain + 216 no) | 132,556 | 1,078,779,608 | 7,973 |

The 157 "yes" labels cover 10.1 % of all real-cluster tags
(14,858 / 147,414) and 52.0 % of all real-cluster occurrences
(1,167,476,227 / 2,246,255,835), but only 9.7 % of the clusters
(859 / 8,832). The kept superclusters are the high-volume ones
(`building`, `highway`, `landuse`, `natural`, `tiger`, `area`,
`surface`, `water`, `wetland`, etc.) — the long tail of small,
specialized base keys was filtered out by the manual labeling.

#### Embeddings pipeline — 169 kept out of 433 superclusters

The embeddings pipeline (potion-base-8M) produced 4,954 real
clusters + 106,498 noise points from the same 225,684 standardized
tags. The non-noise 119,186 tags cover 2,565,137,600 occurrences;
the noise 106,498 tags cover another 803,203,928.

| | Tags | Occurrences | Real clusters |
|---|---:|---:|---:|
| All cluster memberships (incl. noise) | **225,684** | 3,368,341,528 | 4,954 |
| &nbsp;&nbsp;Real clusters (noise excluded) | 119,186 | 2,565,137,600 | 4,954 |
| &nbsp;&nbsp;Noise (cluster_id = -1) | 106,498 | 803,203,928 | — |
| Real, by base-key label (169 yes kept) | **17,612** | **978,614,046** | **511** |
| Real, by base-key label (264 not-kept: 57 uncertain + 207 no) | 101,574 | 1,586,523,554 | 4,443 |

The 169 "yes" labels cover 14.8 % of all real-cluster tags
(17,612 / 119,186) and 38.2 % of all real-cluster occurrences
(978,614,046 / 2,565,137,600), but only 10.3 % of the clusters
(511 / 4,954).

#### Per-supercluster numbers (noise excluded, cluster-member rollup)

##### TF-IDF (157 kept)

| | Polygon-friendly | Point-heavy | All 157 |
|---|---:|---:|---:|
| Base keys (superclusters) | **110 / 157 (70.1 %)** | 47 / 157 (29.9 %) | 157 |
| Occurrences | **1,061,684,644 (90.9 %)** | 105,791,583 (9.1 %) | 1,167,476,227 |
| Tags (cluster members) | **12,504 (84.2 %)** | 2,354 (15.8 %) | 14,858 |
| Real clusters | **670 (78.0 %)** | 189 (22.0 %) | 859 |

##### Embeddings (169 kept)

| | Polygon-friendly | Point-heavy | All 169 |
|---|---:|---:|---:|
| Base keys (superclusters) | **118 / 169 (69.8 %)** | 51 / 169 (30.2 %) | 169 |
| Occurrences | **798,138,011 (81.6 %)** | 180,476,035 (18.4 %) | 978,614,046 |
| Tags (cluster members) | **15,683 (89.0 %)** | 1,929 (11.0 %) | 17,612 |
| Real clusters | **359 (70.3 %)** | 152 (29.7 %) | 511 |

`is_polygon_friendly` is `(count_ways + count_relations) / count_all >= 0.5`,
exposed as `POLYGON_FRIENDLY_THRESHOLD` in
`src/core/db/element_type_stats.py`. The CSVs do **not** address
polygon size; that requires a PBF extract and a separate step.

These numbers are independently verified by:

- `tests/core/db/test_supercluster_stats_audit.py` (TF-IDF pipeline, 11 tests)
- `tests/core/db/test_embeddings_supercluster_stats_audit.py` (embeddings pipeline, 12 tests)

Each suite re-computes every metric two ways from the raw input
files and asserts they match the function's output.
| `scripts/` | The 12 pipeline scripts, copied verbatim from `scripts/` in the GitHub repo | ~30 KB |
| `reproducibility/` | A single shell script that rebuilds the bucket from a fresh checkout | ~1 KB |
| `MANIFEST.md` | Per-file inventory: every file in the bucket, its size and a sha256 prefix | — |
| `SOURCE.md` | Where the source data comes from, how to (re-)download it, what the caches are | — |

Total: **43 files, ~106 MB**.

## What is NOT in here (and why)

- **The source `taginfo.sqlite` (14 GB)**. This is the raw input to the
  pipeline. It is a public dataset, re-downloadable from
  [taginfo.openstreetmap.org](https://taginfo.openstreetmap.org/download/taginfo-db.db.bz2),
  and would dominate the bucket size. See `SOURCE.md` for the
  download and extraction recipe.
- **The `.venv/`, `.git/`, and `__pycache__/` directories** from the
  working tree.
- **The 192 M-row `tags` source table**. It is a verbatim copy of
  what is already in the public source DB.

## How to use this bucket

### Inspect a specific output

HF storage buckets are accessed via `hf://` paths (not via the
plain HTTP `huggingface.co/...` URLs of regular repos). Any
fsspec-compatible library can read them directly:

```python
import pandas as pd
df = pd.read_csv("hf://buckets/NoeFlandre/osm_stats/outputs/standardize_first/tfidf/base_key_families.csv")
print(df.head(20))
```

For binary files (XLSX, sqlite) the same path works through
`huggingface_hub.HfFileSystem`:

```python
from huggingface_hub import HfFileSystem
fs = HfFileSystem()
with fs.open("hf://buckets/NoeFlandre/osm_stats/outputs/standardize_first/tfidf/base_key_families.xlsx", "rb") as f:
    ...
```

DuckDB, pyarrow, polars and any other fsspec-aware tool are also
supported; see the
[HF access patterns docs](https://huggingface.co/docs/hub/storage-buckets-access)
for the full list.

### Re-run the pipeline from a fresh checkout

```bash
git clone https://github.com/NoeFlandre/osm-stats
cd osm-stats
# (follow SOURCE.md to get taginfo.sqlite and build the cache)
.venv/bin/python -m scripts.reproducibility.reproduce
```

The `reproducibility/reproduce.sh` script is a single command that:

1. Downloads `taginfo.sqlite` if not already on disk.
2. Builds the two caches.
3. Runs the four pipeline variants (TF-IDF and embeddings on each
   of the two caches).
4. Produces the base-key-family CSVs and XLSX files.
5. Verifies that the resulting artifact hashes match the bucket's
   `MANIFEST.md`.

### Reproduce just the standardize-first outputs

```bash
.venv/bin/python -m scripts.build_cache_standardize_first
.venv/bin/python -m scripts.profile_clusters_standardize_first
.venv/bin/python -m scripts.profile_clusters_embeddings_standardize_first
.venv/bin/python -m scripts.save_base_key_families_standardize_first
.venv/bin/python -m scripts.compare_pipelines_standardize_first
```

The total wall time on an M-class laptop is roughly **10 minutes for
the cache build + 4 minutes for TF-IDF + 40 minutes for the
embeddings pipeline**.

## Provenance and timestamps

| Step | Source of truth | Reproducible? |
|---|---|---|
| Source `taginfo.sqlite` | [taginfo DB](https://taginfo.openstreetmap.org/download/taginfo-db.db.bz2), `data_until = 2026-06-09 00:59:09` | yes, re-download |
| `tag_features.sqlite` (filter-first) | `python -m src.cli --build-cache --threshold 500` | yes |
| `tag_features_standardize_first.sqlite` | `python -m scripts.build_cache_standardize_first` | yes, but HDBSCAN is non-deterministic |
| All `outputs/*` CSVs / XLSX / MD | The `scripts/*.py` files in this bucket, in the order listed above | yes (with the same caveat) |

The two caches were built on a 2026-06-16 snapshot of the source
DB. Re-running the pipeline on a newer snapshot will produce a
slightly different cache and slightly different cluster assignments;
this is expected.

## Companion resources

- Project GitHub: [github.com/NoeFlandre/osm-stats](https://github.com/NoeFlandre/osm-stats)
- Blog post: [noeflandre.com/posts/osm-data-analysis](https://noeflandre.com/posts/osm-data-analysis)
- Taginfo source: [taginfo.openstreetmap.org/download](https://taginfo.openstreetmap.org/download/taginfo-db.db.bz2)
- Embedding model used: [`potion-base-8M`](https://huggingface.co/minishlab/potion-base-8M) by Minish Lab (via `model2vec`)

## License

The source data is from OpenStreetMap, distributed under the
[ODbL](https://www.openstreetmap.org/copyright). The code in this
bucket is MIT (see the GitHub repo's `LICENSE` file).