H_A :
I am looking to do similar task as in this thread: Merge multiple column values into one column in python pandas
But the difference is that I would like to create a new column that merges all the non-null values in a column after grouping by another column(s). Here is a toy example:
df= pd.DataFrame({'ID1' : [1,1,2,2,3,3,3],'ID2' : ['a','a','b','b','c','c','c'],
'Status' : pd.Series([np.nan,'1', np.nan,'1','2',np.nan,'1'],
dtype="category")})
df
Out[74]:
ID1 ID2 Status
0 1 a NaN
1 1 a 1
2 2 b NaN
3 2 b 1
4 3 c 2
5 3 c NaN
6 3 c 1
I then want to groupby ID1 and ID2:
gr = df.groupby(['ID1','ID2'])
Then, I would like my result look like this:
Out:
NewCol
0 1
1 1
2 2,1
So it is a new DataFrame that contains the non-null values of Status column grouped by ID1 and ID2.
Thanks in advance.
jezrael :
Use GroupBy.agg with lambda function is most general solution:
df1 = df.groupby(['ID1','ID2'])['Status'].agg(lambda x: ','.join(x.dropna())).reset_index()
print (df1)
ID1 ID2 Status
0 1 a 1
1 2 b 1
2 3 c 2,1
Another idea is remove duplicated in first step, but if some group with only misisng values it is removed from output, so is necessary next processing like merge:
#first group with only NaNs
df= pd.DataFrame({'ID1' : [1,1,2,2,3,3,3],'ID2' : ['a','a','b','b','c','c','c'],
'Status' : pd.Series([np.nan,np.nan, np.nan,'1','2',np.nan,'1'],
dtype="category")})
#first group is dropped
df11 = (df.dropna(subset=['Status'])
.groupby(['ID1','ID2'])['Status']
.agg(','.join)
.reset_index())
print (df11)
ID1 ID2 Status
0 2 b 1
1 3 c 2,1
#added missing pairs and `NaN`s converted to empty strings:
df2 = df.drop_duplicates(['ID1','ID2'])[['ID1','ID2']].merge(df11, how='left').fillna('')
print (df2)
ID1 ID2 Status
0 1 a
1 2 b 1
2 3 c 2,1
First solution:
df1 = df.groupby(['ID1','ID2'])['Status'].agg(lambda x: ','.join(x.dropna())).reset_index()
print (df1)
ID1 ID2 Status
0 1 a
1 2 b 1
2 3 c 2,1
Guess you like
Origin http://43.154.161.224:23101/article/api/json?id=351062&siteId=1