maxframe.liteframe.LiteFrame.rechunk#

LiteFrame.rechunk(chunk_size: int | Tuple[int] | Tuple[Tuple[int], ...], reassign_worker=False, axis=None)#

Rechunk LiteFrame data.

This function is used to reorganize the chunk structure of data, which can change how data is distributed in distributed computing. By adjusting chunk sizes, memory usage and computational performance can be optimized. LiteFrame is only ever chunked along the row axis (axis 0); the column axis always stays a single chunk. Note that this function is only effective in DPE for now.

Parameters:

  • chunk_size (chunk_size_type) –

    New chunk size configuration. Can be one of the following forms:

    • Integer: row chunk size, e.g. 100 rechunks rows into chunks of 100.

    • Tuple: per-dimension sizes; only the first element (rows) takes effect. For instance, (100, 2) rechunks rows into chunks of 100.

    • Dictionary: chunk sizes for specific dimensions, for instance, {0: 100} rechunks dimension 0 into 100 rows per chunk, and {0: (100, 100, 100)} rechunks dimension 0 into 3 chunks where each chunk has 100 rows.

  • reassign_worker (bool, default False) – Reserved for future use.

  • axis (int, optional) – If specified, chunk_size applies only to that axis. Internally normalized to the dict form {axis: chunk_size}. Cannot be used together with a dict chunk_size.

Returns:

The rechunked LiteFrame object.

Return type:

TileableType

Examples

>>> import maxframe.liteframe as mlf
>>> lf = mlf.read_odps_table("test_table")
>>> # Rechunk to 100 rows per chunk
>>> lf_rechunked = lf.rechunk(100)
>>> # Rechunk only axis 0 (rows)
>>> lf_rechunked = lf.rechunk(100, axis=0)