1、Hive
1. The table field after not in cannot be empty, otherwise it is invalid.
create table test0201_1 as select null as id union all select 1 as id;
create table test0201_2 as select 2 as id union all select 1 as id;
select count(1) from test0201_2 where id not in(select id from test0201_1); --0
select count(1) from test0201_2 t1
left join test0201_1 t2
on t1.id=t2.id
where t2.id is null; --1
2. If the table is a partitioned table and the table structure has changed, the table structure is different from the table structure of the specified partition.
3. Hive: I encountered a scenario where columns were added to the partition table one after another, and the previous column was replaced by the latter one. By searching the metadata, column 1 can be found in the partition table structure of date no.1, and column 1 is not listed in the partition table structure of date no.2, but column 2. Avoid adding new columns to impala in the future, which may be a problem in this regard.
4. Hive: expressions are fields, and the sorting of window functions can be done like this if there are different scenarios corresponding to different sorting strategies:
row_number() over(partition by part_col order by col1 asc,case when type=1 then colx end asc,case when type=2 then colx end desc)
2、Shell、Mysql
1. Shell connection to Mysql database, MySQL command line
reference: MySQL command line parameters full version :
HOSTNAME="xxx.xxx.xxx.xxx"
PORT="xxxx"
USERNAME="xxxx"
PASSWORD="xxxx"
DBNAME="xxx"
select_sql="select group_concat(schedule_time) from dag_list where id in(select dag_list_id from task_list)"
#执行查询*
result=`mysql -h${
HOSTNAME} -P${
PORT} -u${
USERNAME} -p${
PASSWORD} ${
DBNAME} -Bse "${select_sql}"`
#B参数：打印结果，使用tab作为列间隔符，每个行占用新的一行；e参数：statement 执行语句并退出；s参数：沉默模式。产生少的输出。
schedule_time_str=`echo ${
result} | awk '{print $1}'` #获取到的第1个值
schedule_time_array=(${
schedule_time_str//,/ }) #以逗号分割字符串
for schedule_time in ${
schedule_time_array[@]}
do
echo ${
schedule_time}
done
2. Shell implements split string according to comma
#!/bin/bash
string="hello,shell,haha"
#与/之间与分割的字符 ，另外/后有一个空格不可省略
array=(${
string//,/ })
for var in ${
array[@]}
do
echo $var
done
3. MySQL = is not necessarily an exact match. It is related to the DBA setting. If there are special characters, you may not be able to see it. You can check the length by length.
4. Can mysql unique keys have multiple null values? can
3、Python
1. Python time calculation
#1、添加半小时
import datetime
d1 = datetime.datetime.now()
d3 = d1 + datetime.timedelta(minutes=30)
print(d1)
print(d3)
#2、字符串转时间对象
execution_datetime = dateutil_parser.parse("2021-02-02 15:44" + " +08:00") #转换时区
#3、指定时间格式转换：
latest_complete_time = execution_datetime.strftime("%H:%M") #15:44
2. Python3 JSON data analysis
Python3 can use the json module to encode and decode JSON data. It contains two functions:
json.dumps(): encode data.
json.loads(): Decode the data.
Explore json:
>>> dict_tmp = {
"name":"anla"}
>>> print(dict_tmp)
{
'name': 'anla'}
>>> import json
>>> json_obj = json.dumps(dict_tmp)
>>> print(json_obj)
{
"name": "anla"}
>>> type(json_obj)
<class 'str'>
>>> dict_tmp = json.loads(json_obj)
>>> print(dict_tmp)
{
'name': 'anla'}
>>> type(dict_tmp)
<class 'dict'>
3. Python assert is used to judge an expression and trigger an exception when the expression condition is false.
Assertions can directly return an error when the conditions are not met when the program is running, without waiting for the program to crash after running.
4. isinstance(object, classinfo)
object-instance object.
classinfo-can be direct or indirect class names, basic types, or tuples composed of them.
If the type of the object is the same as the type of parameter two (classinfo), it returns True, otherwise it returns False.