Evaluation & Metrics
foreblocks.evaluation.ModelEvaluator wraps a trained Trainer and provides batched inference, rolling cross-validation, loss curve plotting, and a human-readable training summary.
Setup
ModelEvaluator takes a Trainer instance directly — it reuses the model, device, and training history stored on it.
from foreblocks.training import Trainer
from foreblocks.evaluation import ModelEvaluator
trainer = Trainer(model, ...)
trainer.fit(train_loader, val_loader, epochs=50)
evaluator = ModelEvaluator(trainer)use_amp=True enables automatic mixed precision on CUDA; it is silently ignored on CPU.
Point metrics
y_test = torch.randn(200, 24, 7) # (samples, horizon, channels)
metrics = evaluator.compute_metrics(X_test, y_test)
# {'mse': ..., 'rmse': ..., 'mae': ..., 'mape': ...}
print(metrics)Return dict keys:
| Key | Type | Description |
|---|---|---|
overall | dict | Aggregate MAE / RMSE / MAPE / MSE over all windows |
window_metrics | list[dict] | Per-window metrics including start_idx / end_idx |
predictions | Tensor | Concatenated predictions across all windows |
targets | Tensor | Concatenated targets across all windows |
n_windows | int | Number of windows actually evaluated |
total_points | int | Total sample count |
Model is not retrained per fold
This is a walk-forward evaluation of a fixed model, not k-fold retraining. Use it to assess generalisation across temporal shifts, not for model selection.
Plots
All plotting methods require matplotlib:
pip install foreblocks[plotting]
Cross-validation results
fig = evaluator.plot_cv_results(cv, figsize=(15, 8))
fig.savefig("cv_results.png")Three subplots: train/val loss, learning rate schedule, and (if using distillation) task vs. distillation loss components.
Training summary
evaluator.print_summary()