Most published vegetation-forecasting studies report accuracy figures that don't survive a leakage audit. The Leakage Audit Protocol names four specific channels (V/O/C/P), provides a test for each, and ships as an open leakage_features.yaml reference list that any framework can adopt.
Leakage in NDVI forecasting has been discussed informally in the literature, but no single source previously enumerated the four channels by name with a test for each. V/O/C/P gives the community a shared vocabulary.
Each channel has a concrete check that any pipeline can run. The protocol ships as an open leakage_features.yaml reference list — a versioned column-name registry that any framework can adopt without rewriting their pipeline.
Already running as the training-time validation gate inside Bundle Framework. Auto-detects matched columns, sets them to "(ignore)" with a FAIL-2030 marker, and logs the exclusion in the bundle metadata for reviewers.
Inflated accuracy figures don't stay inside the paper they were published in. They propagate into policy reports, climate-funding allocations, and conservation prioritization — at scale.
Say you want to predict vegetation health in a region from climate, soil, and topography. You train a model, test it on held-out data, and report the accuracy.
If one of your input columns was computed using vegetation data (even indirectly — say, a "leaf area index" derived from the same satellite bands as your target), your model is partly predicting X from X. The reported accuracy is inflated, not measured.
Keep a versioned list of columns known to be derived from the target, and check every input against that list before training. If a match is found, drop the column and log the exclusion. That's the protocol.
TL;DR: Leakage = using the answer (or something derived from it) as one of your inputs without realizing it. The audit catches the most common cases automatically.
Each channel describes a distinct way a feature can secretly carry information from the target. They overlap somewhat — a single feature can fail more than one check — but having four named buckets makes the audit explicit and contributable.
What's the failure mode?
Using features computed from the NDVI time series — like phenology start-of-season day, peak-NDVI day, growing-season length, GPP, fPAR, LAI — as inputs when predicting NDVI itself. The model effectively learns the identity function dressed up as climate inference.
Concrete examples that get flagged
lai_mean (Leaf Area Index — derived from NDVI bands)fpar_mean (fraction of PAR — derived from NDVI)gpp_mean (Gross Primary Production — modeled from NDVI)sos_doy (start-of-season day-of-year — peak of NDVI curve)pos_doy (peak-of-season day — argmax of NDVI)The smell test
Ask: "Could you compute this feature without ever having seen NDVI?" If the answer is no, it fails the V check.
What's the failure mode?
Including the raw MODIS / Landsat / Sentinel optical bands (red, NIR, blue) as inputs — even though NDVI itself is just (NIR − red) / (NIR + red). The model can trivially reconstruct NDVI from those two bands, so the reported accuracy reflects band-arithmetic, not climate inference.
Concrete examples that get flagged
modis_red, modis_nir (raw bands)evi_mean (Enhanced Vegetation Index — uses the same bands)savi_mean (Soil-Adjusted Vegetation Index — same bands)nbr_mean (Normalized Burn Ratio — overlapping band)The smell test
Ask: "Does this feature use the red or NIR band that NDVI is computed from?" If yes, it fails the O check.
What's the failure mode?
Using features produced by another supervised model that was itself trained against vegetation observations. MODIS land cover (MCD12Q1), tree cover percentage, biome maps — all of these are model outputs that encode vegetation information learned from past NDVI / red / NIR observations.
Concrete examples that get flagged
mcd12q1_landcover (MODIS supervised land-cover class)treecover_pct (Hansen et al. — trained on Landsat)cropland_pct (GFSAD30 — trained on optical bands)biome_class (WWF biome — derived from vegetation patterns)plant_functional_type (PFT classifications)The smell test
Ask: "Was this feature produced by a model trained on vegetation observations?" If yes, it fails the C check — even if the supervision happened years ago at a different institution.
What's the failure mode?
Including features that depend on knowing the target year's phenological behavior — for example, "growing-season-mean NDVI" computed over the year being predicted, or "peak-NDVI date" for that year. The feature label says it's a "prior," but its value was computed from the future.
Concrete examples that get flagged
growing_season_mean_y when y is the prediction yearpeak_ndvi_value_y for the prediction yearannual_max_ndvi_y (the answer, basically)year_of_max_greening when computed including the test yearThe smell test
Ask: "Could this feature value be computed in advance, before the prediction year began?" If no, it fails the P check.
The protocol is intentionally simple: a single YAML file, a single audit step at the start of training, and a logged exclusion list in the output metadata. The full check runs in five concrete steps:
leakage_features.yaml — a versioned registry of column names known to fail at least one of V/O/C/P, organized by channel for traceability.
"(ignore)" in the schema with a FAIL-V, FAIL-O, FAIL-C, or FAIL-P marker. A banner alerts the operator that columns were excluded.
training.audit block listing every excluded column, the matched rule, and the YAML version used. Reviewers can verify exactly what was skipped without re-running the pipeline.
The audit is data, not code. The entire reference list is a single human-readable YAML file with this shape:
The same YAML is read by the Bundle Framework's SchemaWidget and by any external pipeline that adopts the protocol. New columns get added by pull request as new leakage patterns are discovered.
You don't need to use Bundle Framework to use the protocol. Three integration levels, in order of effort:
Open leakage_features.yaml, scan your input feature list against it, drop the matches. Takes 10 minutes per project. Fine for one-off papers.
Add a 30-line script that reads the YAML, matches your schema against the rules, and raises an error if any leaky columns are present. Stops accidents during refactoring.
The audit becomes a hard gate at training start: matched columns are auto-set to ignore, the bundle metadata records what was excluded, reviewers can verify the audit without re-running training.
The protocol is intentionally open. New leakage patterns get discovered as new datasets and feature pipelines appear — the YAML should grow to track them.
leakage_features.yaml with the column name and a one-line note explaining why it fails.exceptions: block scoped to specific contexts (e.g., when a derived feature is used outside its leakage context).leakage-audit package (no Bundle dependency)leakage-audit check schema.yaml --rules v0.2.yamlLeakage in machine-learning evaluation is a long-discussed topic; the contribution here is specifically the named-channel taxonomy for NDVI forecasting and the operational gate around it.
The companion paper (planned for 2027) will provide a more thorough literature review, with worked examples of each channel applied to a small public dataset so the protocol can be cited independently of Desertflow.
Four was the smallest number that covers the failure modes we kept hitting in practice during Desertflow's development, without merging distinct mechanisms. Vegetation circularity (V) and Optical bands (O) look related but require different tests — one is about derived products, the other is about raw bands. Classifier-derived (C) is its own thing because the leakage is laundered through an intermediate model. Pheno-prior (P) is temporal-only, which makes it different from V/O/C in how the test runs. If a 5th channel proves necessary later, the protocol allows adding one — it's not closed.
The V/O/C/P naming is specific to NDVI forecasting. But the shapes of the four failures generalize: "circularity" (target-derived features), "raw inputs" (the underlying bands the target was computed from), "classifier leak" (features from supervised models trained on the target), and "temporal contamination" (features whose value depends on the prediction-window data). For soil-moisture forecasting, fire-risk forecasting, or surface-temperature forecasting, the same four shapes apply with different column names — that's a follow-up domain-specific YAML.
Production models suffer more from leakage, not less — because a leaky training set produces a model that quietly underperforms in deployment, and the gap doesn't show up until weeks later when the test-set numbers diverge from real-world performance. Adopting the audit as a pre-deployment gate is cheaper than debugging the deployment gap.
Most papers do have at least one — usually O (including raw optical bands) or P (computing a "prior" over a window that includes the target year). The audit isn't accusatory; it's a single-page YAML you can run your column list against in 10 minutes. If the audit comes back clean, you have something concrete to cite in your methodology section.
The naming is intentionally boring (single letters) so credit stays with the protocol, not any individual contributor. If your group identifies a fifth channel, the PR adding it credits you in the YAML metadata; the channel itself gets a letter.
Those papers document the existence of the problem — they show that published cross-region accuracy collapses when spatial validation is applied properly. This protocol is one level downstream: it provides an operational gate with a versioned reference list, so the failure modes get caught before training instead of being diagnosed after publication. The two complement each other.
Pull requests against leakage_features.yaml, channel proposals, or replication offers — (click to copy)
Code: github.com/mohtheghost
← Back to all projects