Spark-SQL commonly used built-in date and time functions

Spark-SQL常用内置日期时间函数

One, get the current time

1、current_date

• Get the current date
• Time format:yyyy-MM-dd

	spark.sql("select current_date").show(false)

The results show that:

+--------------+
|current_date()|
+--------------+
|2020-11-25    |
+--------------+

2、now()

• Get the current date and time
• Time format:yyyy-MM-dd HH:mm:ss.毫秒

	spark.sql("select now()").show(false)

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 11:02:50.484|
+-----------------------+

3、current_timestamp

• Get the current date and time
• The result is exactly the samenow()

	spark.sql("select current_timestamp").show(false)

Two, extract the field in the time

1、year

• Year in the extraction time

    spark.sql("select now()").show(false)
    spark.sql("select year(now()) as year").show()

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 12:04:28.859|
+-----------------------+

+----+
|year|
+----+
|2020|
+----+

2、month

• Month in extraction time

    spark.sql("select now()").show(false)
    spark.sql("select month(now()) as month").show()

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 12:07:44.976|
+-----------------------+

+-----+
|month|
+-----+
|   11|
+-----+

3、day、dayofmonth

• Extract the day of the date
• day and dayofmonth have exactly the same effect

    spark.sql("select now()").show(false)
    spark.sql("select day(now()) as day").show()
    spark.sql("select dayofmonth(now()) as dayofmonth").show()

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 12:09:45.178|
+-----------------------+

+---+
|day|
+---+
| 25|
+---+

+---+
|day|
+---+
| 25|
+---+

4、hour

• Get time hours

    spark.sql("select now()").show(false)
    spark.sql("select hour(now()) as hour").show()

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 12:12:58.917|
+-----------------------+

+----+
|hour|
+----+
|  12|
+----+

5、minute

• Get time minutes

    spark.sql("select now()").show(false)
    spark.sql("select minute(now()) as minute").show()

The results show that:

+-----------------------+
|current_timestamp()    |
+-----------------------+
|2020-11-25 12:14:47.916|
+-----------------------+

+------+
|minute|
+------+
|    14|
+------+

6、second

• Get the time in seconds

    spark.sql("select second(2020-11-25 12:16:25.172) as second").show()

The results show that:

+------+
|second|
+------+
|    25|
+------+

3. Day of the year\week\quarter

1、dayofyear

• Get the specified date is the first day of the year

	spark.sql("select dayofyear(now()) as dayNum").show()

The results show that:

+------+
|dayNum|
+------+
|   330|
+------+

2、weekofyear

• Get the first few weeks of the current year for the specified date

	spark.sql("select weekofyear(now()) as weekNum").show()

The results show that:

+-------+
|weekNum|
+-------+
|     48|
+-------+

3、dayofweek、date_format

• Get the day of the week of the specified date, you can use the Get Current Date function
• dayofweek was introduced in hive 2.2.0
• When the version is lower, you can use the following methods to get the day of the week
• Note that the second parameter in the function is lowercase u

	spark.sql("select date_format(now(),'u') as weekMany").show()

The results show that:

+--------+
|weekMany|
+--------+
|       3|
+--------+

4、quarter

• Returns the quarter of the specified date
• January to March is the first quarter, three months a quarter

	spark.sql("select quarter(now())").show()

The results show that:

+------------------------------------------+
|quarter(CAST(current_timestamp() AS DATE))|
+------------------------------------------+
|                                         4|
+------------------------------------------+

5、trunc

• Select year or month, the date after the selection is represented by the default value of 01
• The first parameter is the date without hours, minutes and seconds, you need to write the specified date, you cannot use the time function to get the current date
• The second parameter is optional:, "year"、"yy"、"month"、"mon"、"mm"otherwise it returns null
• Only the date is processed here, that is, only the date is displayed in the result

	spark.sql("select trunc('2020-11-25','YEAR') as year").show(false)

The results show that:

+----------+
|year      |
+----------+
|2020-01-01|
+----------+

Four, date and time conversion

1、unix_timestamp

• Returns the unix timestamp of the current time
• The date can be specified, and the date format needs to be specified when specifying the date

	spark.sql("select unix_timestamp() as unix").show()
	spark.sql("select unix_timestamp('2020-11-25','yyyy-MM-dd') as unix").show()

The results show that:

+----------+
|      unix|
+----------+
|1606302965|
+----------+

+----------+
|      unix|
+----------+
|1606233600|
+----------+

2、to_unix_timestamp

• Convert time to timestamp
• The first parameter is the time, the second parameter is the time format
• Can be passed in to get the current date function, at this time do not need to pass in the date format
• The effect is the unix_timestampsame as

spark.sql("select to_unix_timestamp(now()) as unix").show()
spark.sql("select to_unix_timestamp('2020-11-25','yyyy-MM-dd') as unix").show()

3、from_unixtime

• Convert timestamp to current time
• The output date format can be customized, if not defined, it will be the defaultyyyy-MM-dd HH:mm:ss

spark.sql("select from_unixtime('1606303452') as time").show()

The results show that:

+-------------------+
|               time|
+-------------------+
|2020-11-25 19:24:12|
+-------------------+

4、to_date、date

• Convert string to date format
• to_date and date have the same effect

	spark.sql("select to_date('2020-11-25') as time").show()

	spark.sql("select to_date('2020-11-25','yyyy-MM-dd') as time").show()

The results show that:

+----------+
|      time|
+----------+
|2020-11-25|
+----------+

5. Date and time calculation

1、months_between

• Returns the number of months between two dates

	spark.sql("select months_between(now(),'2020-10-29') as monNum").show()

The results show that:

+----------+
|    monNum|
+----------+
|0.89760581|
+----------+

2、add_months

• Return the date n months after the date

	spark.sql("select add_months('2020-11-11',1) as afterMonthTime").show()

The results show that:

+--------------+
|afterMonthTime|
+--------------+
|    2020-12-11|
+--------------+

3. date_add (plus), date_sub (minus)

• Returns the date after adding (subtracting) n days

	spark.sql("select date_add(now(),3) as addDay").show()
	spark.sql("select date_sub(now(),3) as addDay").show()

The results show that:

+----------+
|    addDay|
+----------+
|2020-11-28|
+----------+

+----------+
|    addDay|
+----------+
|2020-11-22|
+----------+

4、datediff

• The number of days between two dates, that is, subtract the two dates

	spark.sql("select datediff(now(),'2020-11-20') as diff").show()

The results show that:

+----+
|diff|
+----+
|   5|
+----+

5、last_day(date)

• Returns the last day of the current month at the specified time

	spark.sql("select last_day('2020-10-20') as last_day").show()

The results show that:

+----------+
|  last_day|
+----------+
|2020-10-31|
+----------+

6、next_day(start_date, day_of_week)

• Find the first day of the week after the specified date (the second parameter)
• That is, the second parameter is the abbreviation of the day of the week

	spark.sql("select next_day('2020-11-20','we') as next_day").show()

The results show that:

+----------+
|  next_day|
+----------+
|2020-11-25|
+----------+