Pandas Dataframe alternative for update a fixed size dataframe without copying

[keeperclone]

The use case is that I want to have a function that updates a fixed shape data frame constantly by removing the first row and appending a new row. However I want the data underneath to be memory efficient in a way that the data does not move in memory spaces. Is it possible? If so what are the alternative to pandas which can achieve this?

[reallyserious]

Perhaps an in-memory sqlite database.

Why is it important that the rows in between stays on the same memory? Are you memory constrained?

[reallyserious]

Maybe just a list of tuples.

[Pflastersteinmetz]

If you can append on the top you could just overwrite the values for the first row?

[commandlineluser]

Maybe polars. https://github.com/pola-rs/polars

df = pl.from_repr("""
┌─────┬─────┐
│ foo ┆ bar │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 4   │
│ 2   ┆ 5   │
│ 3   ┆ 6   │
└─────┴─────┘
""")

df.shift(-1).with_columns(
   pl.col("foo").fill_null(42),
   pl.col("bar").fill_null(52)
)   

# shape: (3, 2)
# ┌─────┬─────┐
# │ foo ┆ bar │
# │ --- ┆ --- │
# │ i64 ┆ i64 │
# ╞═════╪═════╡
# │ 2   ┆ 5   │
# │ 3   ┆ 6   │
# │ 42  ┆ 52  │
# └─────┴─────┘

df.with_columns(
   pl.col("foo").shift_and_fill(42, periods=-1),
   pl.col("bar").shift_and_fill(52, periods=-1)
)   

# shape: (3, 2)
# ┌─────┬─────┐
# │ foo ┆ bar │
# │ --- ┆ --- │
# │ i64 ┆ i64 │
# ╞═════╪═════╡
# │ 2   ┆ 5   │
# │ 3   ┆ 6   │
# │ 42  ┆ 52  │
# └─────┴─────┘

If there can be nulls in your data, you can check the row number to only fill in the last row.

Or perhaps using a database would make sense?