Dask item assignment. Cannot use loc for item assignment
You can use map_partitions
in this case where you can use raw pandas functionality. I.e.
ddf.map_partitions(item_assignment)
this operates on the individual pandas constituent dataframes of the dask dataframe
df = pd.DataFrame({"OtherCol":[0b010, 0b110, 0b100, 0b110, 0b100, 0b010]})
ddf = dd.from_pandas(df, npartitions=2)
ddf.map_partitions(item_assignment).compute()
And we see the result as expected:
OtherCol NewCol
0 2 1
1 6 0
2 4 -1
3 6 0
4 4 -1
5 2 1
You can replace your loc
assignments with dask.dataframe.Series.mask
:
df['NewCol'] = 0
df['NewCol'] = df['NewCol'].mask(new_col == 0b010, 1)
df['NewCol'] = df['NewCol'].mask(new_col == 0b100, -1)