AI Agents · Guide
Prevent Data Leakage: Splits, SQL and Data Checks
Choose time and group boundaries that match intended use, check data quality and prevent future labels or learned preprocessing from leaking into evaluation.
Metarix Editorial · Nov 8, 2026 · 4 min read
Documentation-based research
On this page
A model can look accurate because its evaluation accidentally provides information that will not exist when it is used. This is data leakage. It can arise from future measurements, repeated customers in incompatible partitions, preprocessing fit on the whole dataset, or target information hidden in a feature.
This lesson in the Machine Learning Engineer learning path establishes the data boundaries before feature engineering. We use a synthetic sensor fixture so you can inspect every rule without downloading a real dataset.
Split according to the intended use
A random train/test split can be suitable when observations are sufficiently independent and the evaluation matches deployment. It is not automatically suitable for time-dependent data. If the system predicts future events, train on earlier information and evaluate later periods. If it must generalize to unseen people, organizations or devices, consider entity-group separation as well.
Chronological order alone does not guarantee safety. A label may arrive days after its prediction time. A rolling feature may cover an overlapping window. Two records may describe the same event. Design a gap and grouping policy around those mechanisms rather than selecting an arbitrary number of rows.
The capstone declares an eight-hour label delay. Its training candidates whose labels are not available before validation starts are removed. The same check separates validation from test. The synthetic rows have no real asset histories, so this is an example of timing control, not proof that its split would suit a real fleet.
Give each partition one job
Training data fits model parameters and learned preprocessing. Validation data supports development decisions such as choosing a threshold. A test set is reserved for evaluating the selected system after those decisions are fixed.
Repeatedly checking the test score while changing the system turns that set into development feedback. Renaming it “test” does not restore independence. If further development follows a test result, plan fresh evaluation data or a properly designed nested procedure.
Cross-validation can make better use of training data, but its folds must respect the same temporal and entity constraints. A pipeline helps keep learned transformations inside each fit. The scikit-learn cross-validation guide distinguishes the available split strategies and their assumptions.
Check the data before fitting
Validate schema, units, ranges, missingness, duplicates and label values. Inspect row counts and class counts by time and relevant groups. Check whether features actually arrive when the contract says they do. A correct data type does not establish correct meaning.
Exploratory analysis can reveal problems, but do not use the reserved test set to choose features or tune the model. Keep a record of which partitions informed a decision. In our example, missing load readings are handled by a median learned from training rows only.
SQL is useful here because joins and aggregations often create the training table. A join with several matching rows can multiply examples; a join to a current-state table can silently replace historical values. Understand table grain and timestamp logic before adding more features.
For a SQL-backed project, a duplicate check might look like this; it assumes you have already created and populated a table named sensor_events:
SELECT sample_id, COUNT(*) AS row_count
FROM sensor_events
GROUP BY sample_id
HAVING COUNT(*) > 1;
The downloadable project uses CSV directly and does not create a PostgreSQL database. The PostgreSQL tutorial is supplemental practice, not another prerequisite installation for this exercise.
Exercise: verify the label-availability boundaries
From the extracted ML capstone directory:
python - <<'PY'
from ml_capstone import HERE, load_data, split_rows, timestamp
train, validation, test = split_rows(load_data(HERE / "sensor_events.csv"))
print(len(train), len(validation), len(test))
assert timestamp(train[-1]["label_available_at"]) < timestamp(validation[0]["measured_at"])
assert timestamp(validation[-1]["label_available_at"]) < timestamp(test[0]["measured_at"])
print("label boundaries checked")
PY
Expect 352, 112 and 120 rows, followed by the success message. Sixteen of the original 600 rows are excluded at the two boundaries. The assertions explain the reason for the gap; do not replace them with a gap chosen only to improve a score.
Write down how the split would change if several rows belonged to the same device and the intended product served entirely new devices. You may need grouping as well as time separation. You are ready to continue when you can explain what information each partition is allowed to contribute.
Sources and method
Commands were checked with the local synthetic project on Python 3.12.14 and scikit-learn 1.8.0. The optional SQL illustration was not run against a database.