Skip to content

Changes in 2.0.0

stemflow 2.0.0 combines the community-project features with new execution, calibration and monitoring components. Existing model classes and native Pandas/DuckDB/Parquet interfaces remain available. The immediate code baseline is main at 564e729 (1.1.9); the previous conda-forge recipe is 1.1.6. Changes already present in 1.1.9 are distinguished below.

Compared with 1.1.9

Area 1.1.9 2.0.0
Training Native DataFrame/database execution Native remains default; optional read-only numeric memmaps, bounded dispatch and concurrency controls
Prediction Native ensemble execution and result collection Streaming mean/std, smaller model payloads, optional memmap input/result handling and file-backed ensemble matrices
Input preparation Users prepare database tables and IDs DataFrame-friendly database/context helpers validate IDs and align labels
Calibration User-provided base-model wrappers Composable out-of-fold calibration, blocked CV, local beta maps and optional tuning
Diagnostics Existing model evaluation and plotting Overall, spatial, temporal, space-time and actual-stixel calibration tables/plots
Resource information Progress output Optional live stage logs and persisted sampled memory/timing reports
Community features Separate project branch Supplied partition geometry, additional quadtree criteria and per-stixel training filters
Correctness Previously affected bootstrap/logit/jitter paths Corrected multiplicity, aggregation, alignment and planar jitter with compatibility checks
DuckDB APIs Earlier relation calls Compatibility adapters for tested 1.x and 2.x releases, declared as >=1.1.3,<3

Pandas 3/NumPy 2 compatibility work and several empty-result fixes already reached main during the 1.1.x series. They belong in the cumulative upgrade story from 1.1.6, but are not attributed to this rewrite. DuckDB/Parquet inputs and lazy model loading also existed in 1.1.x; 2.0.0 extends their execution choices.

If upgrading from the original v1.1 release, also account for the intervening 1.1.x changes: ensemble bootstrapping and custom prediction methods in 1.1.3, and DuckDB/Parquet inputs plus finer-grained lazy estimator loading in 1.1.5. These remain available in 2.0.0. Spherical indexing and parallel modeling were already part of v1.1.

Numerical changes

  • Bootstrap draws preserve their actual multiplicity. The old repeated-ID selection could apply duplication twice.
  • Logit aggregation retains positive probabilities; arithmetic/logit means, sample standard deviations and missing masks have explicit checks. Logit standard deviations remain on the logit scale.
  • Named target labels retain their alignment. Temporal prequeries derive coordinates from coherent feature extracts; source databases are read-only and worker scratch is isolated.
  • Planar jitter changes grid boundaries relative to observations instead of canceling at the origin. Newly generated geometry and fitted predictions can therefore change with the same seed. Exported ensemble_df columns and transform meanings remain compatible with tested geometry reuse.
  • AdaSTEM and STEM default to fully random rotation. Set completely_random_rotation=False for the evenly spaced angular schedule.

Intentional constant zero/one classification stixels remain supported. Missing coverage is not silently converted to an absence. API compatibility does not imply bitwise reproduction of previously affected computations or unrestricted pickle compatibility across dependency versions.

Performance evidence

These measurements are local Apple Silicon examples, with workload-specific baselines. They are not timings directly against untouched 1.1.9.

Workload and baseline Time before → after Memory before → after
Resident forests, 50k rows / 8 ensembles; merged main/community baseline 43.47 → 25.72 s 2,476 → 1,465 MiB macOS footprint
Acadian, 100k rows / 8 ensembles; same merged baseline 18.11 → 17.81 s 1,363 → 1,156 MiB macOS footprint
Prediction: 3M rows / 50 separate columns; identical fitted model, native → memmap 23.71 → 5.88 s 7.64 → 5.64 GiB summed RSS
Prediction: 1M rows / 50 ensembles, tuned XGB/local beta; native → memmap 28.27 → 19.87 s 7.93 → 5.66 GiB summed RSS

Some already-lazy small models showed little improvement. Memmap does not bound arbitrary estimator workspace or remove output storage. Summed RSS can count shared pages repeatedly and differs from macOS footprint or Slurm accounting. The retained whole-job report and prediction stress report identify seeds, baselines and limitations.

Installation and migration

After the distribution is published, install the core package or optional features:

python -m pip install --upgrade 'stemflow==2.0.0'
python -m pip install --upgrade 'stemflow[monitoring,tuning]==2.0.0'

monitoring adds psutil; tuning adds XGBoost and Optuna. Neither is required for a fixed generic base estimator. Conda-forge updates separately; its core package does not imply that optional estimators are installed.

Follow the migration guide, execution example and calibration example. The two new notebooks retain executed 2.0.0 outputs. Older examples retain historical outputs and require external data, as explained in the examples overview.

The release install matrix checks wheel/source installation on Python 3.9–3.15. The full pytest matrix covers 3.11–3.15; this does not imply every Python/dependency/platform combination has been tested. Final revision-specific CI evidence is recorded in PR #110.