maxframe.liteframe.LiteFrame.apply_chunk#

LiteFrame.apply_chunk(func, *, dtypes=None, skip_infer=False, input_frame_type='pandas', args=(), **kwargs)#

Apply func per execution chunk. Row count may change.

Unlike transform / transform_chunk, the row count of the output is allowed to differ from the input — the function is applied once per execution chunk and the results are concatenated. Hidden columns are dropped from the input before invocation; select them first if needed.

Parameters:

  • func (callable or callable class) –

    Function taking a DataFrame chunk and returning a DataFrame (or a tuple of DataFrames for multiple outputs).

    If func is a class, it is instantiated once per chunk, __call__(batch, end=True) is invoked once with end=True (each call is a complete invocation that must flush its output), and close() (if defined) is always called at the end of the chunk.

  • dtypes (dict or pd.Series, optional) – Expected output dtypes.

  • skip_infer (bool, default False) – If True, skip dtype inference.

  • input_frame_type ({"pandas", "polars"}, default "pandas") – Frame type passed to func.

  • args (tuple) – Extra positional arguments forwarded to func.

  • **kwargs – Extra keyword arguments forwarded to func.

Returns:

Lazy LiteFrame whose materialized output is the per-chunk concatenation of func results. The row count may differ from the input. Call .execute() to materialize.

Return type:

LiteFrame

See also

apply

Apply a function per column or per row.

transform_chunk

Transform per execution chunk.

flat_apply

Apply a function per row, expanding to 0-N rows.

Examples

>>> import maxframe.liteframe as mlf
>>> lf = mlf.LiteFrame({'a': [1, 2, 3, 4], 'b': [10, 20, 30, 40]})

Filter rows within each chunk (row count may change):

>>> result = lf.apply_chunk(lambda df: df[df['a'] > 2],
...                         dtypes={'a': int, 'b': int})
>>> result.execute()
   a   b
0  3  30
1  4  40

Produce multiple outputs by returning a tuple of DataFrames:

>>> results = lf.apply_chunk(lambda df: (df, df * 2),
...                          dtypes=[{'a': int, 'b': int},
...                                  {'a': int, 'b': int}])
>>> len(results)
2
>>> results[0].execute()
   a   b
0  1  10
1  2  20
2  3  30
3  4  40