A phenomenon that occurs in the use of row_number() over partition by

 

 

At present, there is such a demand in the warehouse. According to books, subjects, and knowledge points, the grouping is sorted according to the chapters corresponding to the knowledge points. After sorting, only the first item of data is obtained, because only the first item is needed, in order to prevent the occurrence of tie First, the windowing function supported by hive is used here:

row_number() over partition by

 

The data that ran out in the following screenshot:

 

The screenshot data phenomenon is like this,

The field values ​​of partition by subject_id, book_id, kp_id

It is the same as the field value unit_sort of order by, but the week is different.

So my question is, the data that comes out is that after hive processes the week in descending order by default, the values ​​of the other fields are the same, whether the book_kp_unitsort_rank is assigned according to the week method or the book_kp_unitsort_rank assignment is random, 

 

I re-run the environment hue three or four times. I thought that the hue was displayed by default for sorting, and then re-run in the hive command line is still the result of the above picture.

 

There is no conclusion in writing this post. I still don’t know if the data is random and coincidental. If week=20 is the first ranking or the function of hive will follow this way,

 

In order to prevent hidden dangers (the rankings are given randomly by week), I have updated one, and added the week desc method after order by;

 

as follows:

select
subject_id,
book_id,
unit_id,
unit_sort,
kp_id,
week ,
start_date,
row_number() over(partition by subject_id,book_id, kp_id order by unit_sort asc , week desc )  as book_kp_unitsort_rank   
from
ods_common_book_unit_kp_schedule where day='2018-01-08'  and  subject_id in ('010', '030')  
and  kp_id is not null
and  book_id = '010003001104100'   and kp_id in ('1015800','1015720')

 

 

I hope that friends who see the post can leave a message for me to express their opinions, thank you