[hive] Use of time-related functions (time stamp function unix_timestamp()/from_unixtime(), date processing function datediff()/date_sub()/date_add(), etc.)

Tip: After the article is written, the table of contents can be automatically generated. How to generate it can refer to the help document on the right

---

1. Timestamp function

1、unix_timestamp()

• unix_timestamp() is called without parameters to obtain the current system timestamp with a 10-digit bigint value, which is only accurate to seconds.

select unix_timestamp();
> 1680227307 

• unix_timestamp(string timestamp) The default input format is "yyyy-MM-dd HH:mm:ss", if it does not match, it will return null.

select unix_timestamp('2023-03-31 01:48:27') ; 
> 1680227307

select unix_timestamp('2023-03-31');
> NULL

• unix_timestamp(string date, string pattern) Convert the string in the specified time format into a timestamp, and return null if it does not match.

--固定日期转换成时间戳
select unix_timestamp('2023-03-31','yyyy-MM-dd');
> 1680220800 

select unix_timestamp('2023-03-31 01:48:27','yyyy-MM-dd HH:mm:ss');  
> 1680227307

select unix_timestamp('2023-03-31T10:02:41Z', "yyyy-MM-dd'T'HH:mm:ss'Z'");
> 1680256961

select unix_timestamp('2023-03-31','yyyy-MM-dd HH:mm:ss')   
> NULL

• unix_timestamp() - unix_timestamp() is the subtraction of two times converted to timestamp. The unit of timestamp is seconds, and the difference between the two times is the number of seconds after the subtraction.

cast((unix_timestamp() - unix_timestamp()) % 60 as int) -- is the difference in seconds.
cast((unix_timestamp() - unix_timestamp()) / 60 as int) % 60 -- is the difference in minutes.
cast((unix_timestamp() - unix_timestamp()) / (60 * 60) as int) % 24 -- is the difference in hours.
concat(cast((unix_timestamp() - unix_timestamp()) / (60 * 60 * 24) as int) -- is the difference in days.

--是相差的秒数
select unix_timestamp('20230329185800','yyyyMMddHHmmss')- unix_timestamp('20230329185604','yyyyMMddHHmmss');
> 56
select cast((unix_timestamp('20230329185800','yyyyMMddHHmmss')- unix_timestamp('20230329185604','yyyyMMddHHmmss'))%60 AS int);
> 56

--是相差的分钟数
select cast((unix_timestamp('20230329185800','yyyyMMddHHmmss')- unix_timestamp('20230329185604','yyyyMMddHHmmss'))/60 AS int);
> 1

--是相差的小时数
select cast((unix_timestamp('20230329215800','yyyyMMddHHmmss')- unix_timestamp('20230329185604','yyyyMMddHHmmss'))/(60 * 60) AS int) % 24;
> 3

--是相差的天数
select cast((unix_timestamp('20230331185800','yyyyMMddHHmmss')- unix_timestamp('20230329185604','yyyyMMddHHmmss'))/(60 * 60 * 24) AS int)  ;
> 2

Note : the first parameter of unix_timestamp class only accepts string type

2、from_unixtime()

• from_unixtime(bigint unixtime, string format) Convert the timestamp seconds into UTC time and represent it with a string. You can specify the time format through format and specify the output time format. Among them, unixtime is a 10-bit timestamp value, but the 13-bit so-called millisecond is not allowed.

-- 时间戳转换成固定日期
select from_unixtime(1680257881,'yyyy-MM-dd HH:mm:ss');
> 2023-03-31 10:18:01

select from_unixtime(1680257881,'yyyy-MM-dd');
> 2023-03-31

select from_unixtime(unix_timestamp('31/Mar/2023:10:18:01 +0800', 'dd/MMM/yyy:HH:mm:ss Z'))
> 2023-03-31 02:18:01

select from_unixtime(1680257881);
> 2023-03-31 10:18:01 

-- 取当前系统时间
select from_unixtime(unix_timestamp(),'yyyy-MM-dd HH:mm:ss');
> 2023-03-31 02:37:17

Note : The first parameter of the from_unixtime class only accepts bigint type.

3. unix_timestamp() is used in combination with from_unixtime()

Example: Given that the call start time is "20230329185604" and the call duration is 56 seconds, calculate the end time of the call.

select from_unixtime((unix_timestamp(concat(substr('20230329185604',0,4),'-',substr('20230329185604',5,2),'-',substr('20230329185604',7,2),' ',substr('20230329185604',9,2),':',substr('20230329185604',11,2),':',substr('20230329185604',13,2)))+56),'yyyy-MM-dd HH:mm:ss');
> 2023-03-29 18:57:00 

select from_unixtime((unix_timestamp('20230329185604','yyyyMMddHHmmss')+56),'yyyy-MM-dd HH:mm:ss');
> 2023-03-29 18:57:00 

Summarize

1. The way to obtain the timestamp in Hive is the unix_timestamp() function. This function can only be accurate to the time of the second level. This function is not suitable for applications that require high time accuracy.

2. When Hive obtains the millisecond-level timestamp of the current time, it needs to use the cast function to convert current_timestamp() to a double type and multiply it by 1000 to obtain a millisecond-level timestamp.

3. For the timestamp with millisecond precision stored in the Hive library, in order to ensure that the time precision is not lost, you need to use the to_utc_timestamp() function. This function supports time errors at the millisecond level, but you need to specify the current time zone.

-- 1.Hive中使用current_timestamp()函数获取当前时间,精确到毫秒。
select current_timestamp();
> 2023-03-31 11:18:48.184 


--2.Hive中获取当前时间戳为10位的bigint类型数值，该数值只精确到秒级别。
select unix_timestamp(current_timestamp());   
> 1680261703

--3.Hive中将时间戳转换为日期类型，结果可以看到时间的毫秒是无法正常获取到，因为时间戳只是精确到秒级别的，from_unixtime()函数也只支持秒级别的时间戳转换。
select from_unixtime(1680261703,'yyyy-MM-dd HH:mm:ss:SSS');
> 2023-03-31 11:21:43:000

-4.Hive中获取毫秒级别的时间戳，获取到了一个13位的数值，该数值精确到毫秒即为当前的时间的时间戳。
select current_timestamp(), cast(current_timestamp() as double) ;

+--------------------------+-------------------+
|           _c0            |        _c1        |
+--------------------------+-------------------+
| 2023-03-31 11:25:26.802  | 1.680261926802E9  |
+--------------------------+-------------------+

--5.Hive中处理毫秒级别的时间戳，使用Hive提供的to_utc_timestamp()函数将毫秒级别的时间戳转换为相应的时间并且精确到了毫秒，与上一步获取时间戳的时间一致。
select to_utc_timestamp(1.680261926802E9, 'GMT');
> 2023-03-31 11:25:26.802  

4. To convert the thirteen-digit timestamp into a date in Hive, you need to divide the "thirteen-digit timestamp" by 1000 and convert it to a bigint type, and use from_unixtime() to convert it into the desired date format.

--十三位时间戳转换成日期,将十位时间戳/1000,转为bigint类型
select from_unixtime(cast(1679206709158/1000 as bigint),'yyyy-MM-dd HH:mm:ss');
> 2023-03-19 06:18:29

2. Date processing function

1、date_format()

• date_format(expr, fmt) converts a timestamp to a string in fmt format .
  expr: DATE, TIMESTAMP("YYYY-MM-DD:HH-MM-SS"), or STRING in valid date/time format.
  fmt: A STRING expression describing the desired format.

select date_format('2023-03-31','yyyy-MM');
> 2023-03

select date_format('2023-03-31','yyyyMMdd');
> 20230331

select date_format('2023/03/31', 'y'); --识别不了/ 这种分隔符
> NULL

2、date_sub()

• date_sub(startDate,numDays) Subtracts the number of days from the current date, returning the date.
  startDate: date expression.
  numDays: An integer expression.

date_sub('yyyy-MM-dd',n/-m) , returns the date n days before and m days after the initial date.

select date_sub('2023-03-31',5); --2023-03-31 5天前
> 2023-03-26

select date_sub('2023-03-31',-2); --2023-03-31 2天后
> 2023-04-02

3、date_add()

• date_add(startDate,numDays) Adds the number of days to the current date, returning the date.
  startDate: date expression.
  numDays: An integer expression.

date_sub('yyyy-MM-dd',n/-m) , returns the date that is n days after the initial date and m days before.

select date_add('2023-03-31',5); --2023-03-31 5天后
> 2023-04-05

select date_add('2023-03-31',-2); --2023-03-31 2天前
> 2023-03-29

4、datediff()

• datediff('endDate','startDate') returns the difference in days between the preceding and following dates, int type.
  endDate: date expression.
  startDate: date expression.

select datediff('2023-03-31','2023-03-21');
> 10

select datediff('2023-03-31','2023-04-05');
> -5

--"yyyy/MM/dd"转换成"yyyy-MM-dd"
select datediff(regexp_replace('2023/03/31', "/", "-"), regexp_replace('2023/03/12', "/", "-"));
> 19

-- 求时间差
select datediff(from_unixtime(unix_timestamp('20221001','yyyyMMdd'),'yyyy-MM-dd'),from_unixtime(unix_timestamp(substr('2022-09-09',1,10),'yyyy-MM-dd'),'yyyy-MM-dd'));
> 22

5、last_day()

• last_day(expr) Find the last day of the month
  expr: a DATE expression.

select last_day('2023-03-21');
> 2023-03-31
select last_day('2023-03-21 10:18:01');
> 2023-03-31

6、next_day()

• next_day(expr, dayOfWeek), returns a DATE.
  expr: A DATE expression.
  dayOfWeek: A STRING expression identifying the day of the week.

dayOfWeek must be one of the following (case insensitive):
'SU', 'SUN', 'SUNDAY'
'MO', 'MON', 'MONDAY'
'TU', 'TUE', 'TUESDAY'
'WE', 'WED', 'WEDNESDAY'
'TH', 'THU', 'THURSDAY'
'FR', 'FRI', 'FRIDAY'
'SA', 'SAT', 'SATURDAY'

--取当前天的下一个周一
select next_day('2023-03-31','MO');
> 2023-04-03

--取当前周的周一
select date_sub(next_day('2023-03-31','MO'),7);
> 2023-03-27

7、add_months()

• add_months(startDate, numMonths)
  startDate: A DATE expression.
  numMonths: integer.

add_months('yyyy-MM-dd',n/-m) , returns the date that is n months after the initial date and m months before.

select add_months('2023-03-01',1); --1个月后
> 2023-04-01

select add_months('2023-03-01',-6); --6个月前
> 2022-09-01

8. Other related functions

#1、返回当天的三种方式
select current_date;
> 2023-03-31
select current_timestamp;
> 2023-03-31 15:20:11.566  
select from_unixtime(unix_timestamp());
> 2023-03-31 07:20:30

#2、返回日期中的年
select year('2023-03-31 01:48:27');
> 2023 

#3、返回日期中的月份
select month('2023-03-31 01:48:27');
> 3

#4、返回日期中的日
select day('2023-03-31 01:48:27');
> 31

#5、返回日期中的时
select hour('2023-03-31 01:48:27');
> 1

#6、返回日期中的分
select minute('2023-03-31 01:48:27');
> 48

#7、返回日期中的秒
select second('2023-03-31 01:48:27');
> 27

#8、返回日期中的当前月份的第几天
select dayofmonth('2023-03-31 01:48:27');
> 31

#9、返回日期中的当前的周数(今年第几周)
select weekofyear('2023-03-31 01:48:27');
> 13

#10、返回当月第一天
select date_sub(current_date,dayofmonth(current_date)-1);
> 2023-03-01 
select trunc('2023-03-31','MM');
> 2023-03-01 
select from_unixtime(unix_timestamp(date_format(from_unixtime(unix_timestamp('20230331','yyyymmdd'),'yyyy-mm-dd'),'yyyy-MM-01'),'yyyy-MM-dd'),'yyyyMMdd')
> 20230301 

#11、返回当年的第一天
select trunc('2023-03-31','YEAR');
> 2023-01-01

Summarize

1. date_format()/date_sub()/date_add()/datediff()/last_day()/next_day()/add_months function date can only be 'yyyy-MM-dd' format & 'yyyy-MM-dd HH:mm :ss' format