nick_halden :
I have the following df:
testcol_45 testcol_76 testcol_99
dates
2020021918 -1.33 -1.29 -1.38
2020022000 -1.24 -1.33 -1.29
2020022006 -999.00 -1.23 -1.33
2020022012 -0.88 -999.00 -1.21
2020022018 -0.70 -0.88 -999.00
2020022100 -0.46 -0.70 -0.89
2020022106 -0.37 -0.45 -0.70
I would like to shift the columns up, and not include the cells containing the -999 values once a -999 is reached for every column.
My desired result would look like the following:
testcol_45 testcol_76 testcol_99
dates
2020021918 -0.88 -0.88 -0.89
2020022000 -0.70 -1.70 -0.70
2020022006 -0.46 -0.45
2020022012 -0.37
2020022018
2020022100
2020022106
(The length of column once this operation is achieved does not matter in this case; additionally the column names can be named anything so I cannot reference them by name).
I have tried various iterations of df.ffill and df.bfill, however to no avail.
Thank you for your time in advance.
Quang Hoang :
You can mask the top with nan and use the ideas in this question to shift the data. For example, you can mask with:
df.where(df.eq(-999)[::-1].cumsum().eq(0))
Output:
testcol_45 testcol_76 testcol_99
dates
2020021918 NaN NaN NaN
2020022000 NaN NaN NaN
2020022006 NaN NaN NaN
2020022012 -0.88 NaN NaN
2020022018 -0.70 -0.88 NaN
2020022100 -0.46 -0.70 -0.89
2020022106 -0.37 -0.45 -0.70