Skip to content

stemflow.lazyloading.data_preparation

Helpers for preparing model-ready DuckDB input from numeric Pandas DataFrames. See parallel execution and memory for workflow examples and cleanup behavior.

Create model-ready DuckDB inputs from familiar Pandas objects.

make_duckdb_database(frame, path, *, memory_limit='500MB', overwrite=False)

Save a numeric DataFrame as one model-ready DuckDB table.

Observation IDs come from frame.index and row order is retained. No SQL knowledge, manual ID column, or SQL index creation is needed. Returns an absolute filename suitable for split/predict. Existing files are protected unless overwrite=True. The caller's DataFrame is not modified or freed.

Source code in stemflow/lazyloading/data_preparation.py 
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
def make_duckdb_database(frame, path, *, memory_limit="500MB", overwrite=False):
    """Save a numeric DataFrame as one model-ready DuckDB table.

    Observation IDs come from ``frame.index`` and row order is retained.
    No SQL knowledge, manual ID column, or SQL index creation is needed.
    Returns an absolute filename suitable for ``split``/``predict``.
    Existing files are protected unless ``overwrite=True``.
    The caller's DataFrame is not modified or freed.
    """
    _validate_frame(frame)
    path = Path(path).expanduser().resolve()
    if path.suffix != ".duckdb":
        raise ValueError("The destination must end in .duckdb.")
    if path.exists() and not overwrite:
        raise FileExistsError(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    with tempfile.TemporaryDirectory(prefix="stemflow_write_", dir=path.parent) as temporary:
        staged = Path(temporary) / path.name
        _write_database(frame, staged, memory_limit)
        staged.replace(path)
    return str(path)

make_training_databases(X, y, directory, *, memory_limit='500MB', overwrite=False)

Create paired X/y DuckDB files and return (X_path, y_path).

Pandas labels align by observation ID, including shuffled labels. NumPy/list labels align by position. Both database files contain one tmp table and the ID column required by AdaSTEM. They are closed before returning, so parallel workers can open them read-only.

Use training_databases for automatic temporary-file cleanup.

Source code in stemflow/lazyloading/data_preparation.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def make_training_databases(X, y, directory, *, memory_limit="500MB", overwrite=False):
    """Create paired X/y DuckDB files and return ``(X_path, y_path)``.

    Pandas labels align by observation ID, including shuffled labels.
    NumPy/list labels align by position. Both database files contain one
    ``tmp`` table and the ID column required by AdaSTEM. They are closed
    before returning, so parallel workers can open them read-only.

    Use ``training_databases`` for automatic temporary-file cleanup.
    """
    _validate_frame(X)
    target = _aligned_target(X, y)
    directory = Path(directory).expanduser().resolve()
    paths = [directory / "X.duckdb", directory / "y.duckdb"]
    for path in paths:
        if path.exists() and not overwrite:
            raise FileExistsError(path)
    directory.mkdir(parents=True, exist_ok=True)
    with tempfile.TemporaryDirectory(prefix="stemflow_write_", dir=directory) as temporary:
        staged = [Path(temporary) / path.name for path in paths]
        for frame, path in zip([X, target], staged):
            _write_database(frame, path, memory_limit)
        for source, destination in zip(staged, paths):
            source.replace(destination)
    return tuple(str(path) for path in paths)

training_databases(X, y, *, directory=None, memory_limit='500MB')

Yield temporary (X_path, y_path) and clean up on success/failure.

directory optionally selects node-local scratch, such as os.environ.get('SLURM_TMPDIR'). Each context gets its own folder. Keep the context open until fit and other input reads complete.

Source code in stemflow/lazyloading/data_preparation.py 
115
116
117
118
119
120
121
122
123
124
125
126
@contextmanager
def training_databases(X, y, *, directory=None, memory_limit="500MB"):
    """Yield temporary ``(X_path, y_path)`` and clean up on success/failure.

    ``directory`` optionally selects node-local scratch, such as
    ``os.environ.get('SLURM_TMPDIR')``. Each context gets its own folder.
    Keep the context open until ``fit`` and other input reads complete.
    """
    with tempfile.TemporaryDirectory(prefix="stemflow_training_", dir=directory) as temporary:
        paths = make_training_databases(X, y, temporary, memory_limit=memory_limit)
        del X, y
        yield paths