Evaluate and Tune Models Without Fooling Yourself

A score answers a question only when the metric, population and evaluation procedure match the decision. High accuracy can be unhelpful when the important class is rare. A good ranking score does not tell you which action threshold to use. A threshold chosen on the test set invalidates the claim that the test was held out.

This Machine Learning Engineer learning path lesson separates these decisions. It focuses on trained-model evaluation. Evaluating a document assistant’s retrieval and generated answers is a different application-level task, covered in our Applied AI Engineer path.

Start with support and a confusion matrix

Record how many examples were evaluated, how many belong to each class, and how the sample relates to intended use. A score based on a small synthetic set carries a very different claim from a prospective evaluation on representative live data.

A binary confusion matrix counts true negatives, false positives, false negatives and true positives. Precision asks how many predicted positives were actually positive. Recall asks how many actual positives were found. F1 combines precision and recall into one summary; it does not encode every cost or capacity constraint.

The capstone uses F1 as an educational threshold objective. A real fault-inspection policy could need a minimum recall, a maximum daily inspection load, or explicit decision costs. Those should be decided with the domain owner before selecting a threshold.

Separate ranking, probabilities and decisions

ROC AUC summarizes ranking across thresholds using true-positive and false-positive rates. Average precision summarizes a precision–recall relationship and depends on the evaluation population, including class prevalence. Neither is a substitute for checking the actual operating point. For imbalanced settings, inspect precision–recall behavior and support rather than relying on ROC AUC alone.

Probability calibration is another question: do cases assigned a probability near a value occur at roughly that rate in comparable data? A classifier can rank well and still give poorly calibrated probabilities. Check calibration with appropriate held-out data if decisions depend on probability values; calibrating on the same data used to judge calibration gives optimistic evidence.

Finally, a decision threshold turns a score into an action. Changing that threshold can alter precision and recall without retraining the underlying model. The scikit-learn threshold guide distinguishes model fitting from decision-threshold tuning.

Tune on development data, then lock the system

Hyperparameters, preprocessing choices, features and thresholds are development decisions. Training-only cross-validation or a dedicated validation scheme can support them, using splits appropriate to time and entity structure. When trying many alternatives, account for selection effects; nested evaluation may be needed for an honest estimate of the selection procedure.

Our small example deliberately fixes the logistic model and its C setting. It chooses a threshold from 0.1–0.9 on validation F1, breaking ties toward the higher threshold. Only after freezing that choice does it calculate the test report. It does not implement a general hyperparameter search.

Reading the test report is the end of this version’s selection process. If you use its mistakes to redesign the model, treat that as a new development cycle and plan fresh evaluation data. A repeatable command does not make repeated test-guided selection unbiased.

Exercise: explain the locked operating point

After running the capstone training command:

python - <<'PY'
import json
from pathlib import Path
report = json.loads(Path("artifacts/report.json").read_text())
print(report["selection"]["threshold"])
print(report["selection"]["test_used_for_selection"])
print(report["test_model"]["rows"], report["test_model"]["positive_labels"])
print(report["test_model"]["confusion_matrix_true_rows_predicted_columns"])
PY

With the shipped fixture and tested environment, expect threshold 0.3, false for test selection, 120 test rows with 38 positive labels, and matrix [[55, 27], [12, 26]]. Rows are true classes and columns predicted classes, both ordered 0, 1. These counts describe synthetic data, not real equipment reliability.

Write a short explanation of the 27 false positives and 12 false negatives. Propose an alternative development objective without selecting a new threshold from this test matrix. You are ready to continue when you can distinguish a ranking metric, a probability estimate, an action threshold and an untouched evaluation set.

Sources and method

Reported fixture outputs were reproduced locally with Python 3.12.14 and scikit-learn 1.8.0. No business-impact study or real-world validation was performed.

• scikit-learn: model evaluation and scoring.
• scikit-learn: tuning decision thresholds.
• scikit-learn: cross-validation.