Table of contents
1. How to set a cool background image for IDEA?
2. Use Python to execute sql scripts
3. PL/SQL SQL query for sub-database or sub-table
4. How to correctly use non-static member variables in the spring controller?
6. Gracefully stop a running thread in Java
7. What is the difference between 0.0.0.0, 127.0.0.1, and localhost?
8. Use xxl-job to restart the first machine regularly at 0:00 am every day
9. Win+r common query in Windows system
10. Use Redis to count website user visits
1. How to set a cool background image for IDEA?
It's easy to set up in just a few steps:
- file---settings---keymap: Search for Set Background Image.
- Add a shortcut key: such as ctrl+alt+3, save and exit.
- Next time you use this shortcut key, the background setting pop-up box will be called out, as shown in the figure:
- Select a native image, adjust transparency, etc.
- Effect:
2. Use Python to execute sql scripts
Background: xxl-job is often used in work recently, which is a distributed task scheduling framework that can execute complex timing scheduling scenarios in parallel. However, before starting xxl-job officially, some sql scripts need to be run on the MySQL database. Here, I didn't just copy the sql script statement to the database and execute it. Instead, I thought of writing a script in Python to execute the sql script.
Just imagine, if the amount of data to execute the sql script is small, it is definitely most convenient to directly copy to the database client for execution, but when there are millions of lines of statements to be executed, it is undoubtedly a more appropriate choice to process through Python. There are three steps to implement:
- Read the .sql file;
- Preprocess the .sql file;
- Execute the script;
The demo code is as follows:
import pymysql
'''database一定要存在，否则会报错 1049'''
def get_connection():
print('开始连接数据库.....')
connection = pymysql.connect(host="localhost",
port=3306,
user="root",
--pwd自行填写,
database="xxl_job")
print('连接数据库ok')
return connection
def process_sql(filename, connection):
with connection.cursor() as cursor:
with open(filename, "r", encoding='utf-8') as f:
sql_list = f.read().split(";")[0:-1]
for sql in sql_list:
if "\n" in sql:
sql.replace("\n", " ")
if " " in sql:
sql.replace(" ", " ")
sql_item = sql + ";"
# print(sql_item)
cursor.execute(sql_item)
connection.commit()
if __name__ == '__main__':
con = get_connection()
process_sql(r"D:\items\xxl-job\doc\db\tables_xxl_job.sql", con)
print("sql执行成功！")Execution effect:
3. PL/SQL SQL query for sub-database or sub-table
PL/SQL is Oracle's procedural extension of the SQL language, which means adding procedural processing statements (such as branches, loops, etc.) to the SQL command language on the basis of the original addition, deletion, query and modification, so that the SQL language has process processing capabilities.
In the design of database tables, there may be sub-databases or sub-tables. If there is a table T_CHG_RECORD, which has 16 sub-databases chg01~chg16, how to count the total data volume of this table? The easiest way is to write 16 SQL query statements, but there must be a more advanced way. At this time, PL/SQL can come in handy. The writing method is as follows:
------ 先查CHG01~CHG09
declare
v_count number;
v_total number := 0;
begin
for idx in 1 .. 9 loop
execute immediate 'select count(*) from CHG0'|| idx ||'.T_CHG_RECORD ' into v_count;
v_total := v_total + v_count;
--dbms_output.put_line(idx ||':' || v_count);
end loop;
dbms_output.put_line('total:' || v_total);
end;
------再查CHG10~CHG16
declare
v_count number;
v_total number := 0;
begin
for idx in 10 .. 16 loop
execute immediate 'select count(*) from CHG'|| idx ||'.T_CHG_RECORD ' into v_count;
v_total := v_total + v_count;
--dbms_output.put_line(idx ||':' || v_count);
end loop;
dbms_output.put_line('total:' || v_total);
end;If there is a table T_ORDER_RECORD for sub-table design, which is divided into 64 tables, then it is easier to use PL/SQL to query the total data volume of 64 tables, written as follows:
---统计2023年之前的订单总数量
declare
v_count number;
v_total number := 0;
begin
for idx in 1 .. 64 loop
execute immediate 'select count(*) from ORDDB.T_ORDER_RECORD'|| idx ||' t where t.updatetime< to_timestamp(''2023-01-01 00:00:00.0'',''yyyy-MM-dd hh24:mi:ss.ff'')' into v_count;
v_total := v_total + v_count;
--dbms_output.put_line(idx ||':' || v_count);
end loop;
dbms_output.put_line('total:' || v_total);
end;Since the PL/SQL syntax supports logic such as loops, the statements for querying sub-databases/sub-tables become elegant and concise. execute immediate ...... into ...... is the core statement of the query, according to your own needs It can be transformed.
4. How to correctly use non-static member variables in the spring controller?
Spring's Controller is a singleton by default. If a static non-member variable is used and the variable is not synchronized, it will cause thread safety issues.
for example:
@Controller
@Slf4j
public class ScopeTestController {
private int num = 0;
@RequestMapping("/testScope1")
public void testScope1() {
log.info("testScope1：{}", ++num);
}
@RequestMapping("/testScope2")
public void testScope2() {
log.info("testScope2：{}", ++num);
}
}First visit http://localhost:8080/testScope1, the answer is 1; then visit http://localhost:8080/testScope2, the answer is 2.
As a result, thread safety issues arise. To avoid this kind of problem, it is best not to define non-static member variables in the class. If it is really necessary to define the non-static member variable, how to use it correctly? There are two ways:
The first one is to use the multi-case pattern and add @Scope("prototype") annotation
@Controller
@Scope("prototype")
@Slf4j
public class ScopeTestController {
private int num = 0;
@RequestMapping("/testScope1")
public void testScope1() {
log.info("testScope1：{}", ++num);
}
@RequestMapping("/testScope2")
public void testScope2() {
log.info("testScope2：{}", ++num);
}
}Second, use ThreadLocal to ensure that variables are not modified
@Controller
@Slf4j
public class ThreadLocalController {
// private int num = 0;
private static final ThreadLocal<Integer> num = ThreadLocal.withInitial(() -> 0);
@RequestMapping("/test1")
public void test1() {
int value1 = num.get();
if (value1 == 0){
log.info("test1：{}", ++value1);
}
num.remove();
}
@RequestMapping("/test2")
public void test2() {
int value2 = num.get();
if (value2 == 0){
log.info("test2：{}", ++value2);
}
num.remove();
}
}5. Solve the problem that the name of the entity class in mybatis is inconsistent with the field name of the table
This is a basic and common problem, and there are two ways to solve it:
The first is to modify the name of the entity class, or use an alias for the fields of the database table, and keep the names of the two consistent (this method is not recommended).
The second is to use the resultMap tag, for example as follows:
6. Gracefully stop a running thread in Java
<1>. Since it is elegant, the super-violent method of killing the main process and terminating the thread must not be used (regardless).
<2>. Call the stop() method in the Thread class, which is a thread termination method marked as obsolete, because the stop method will release the lock and force the thread to terminate, which may cause half of the thread to be terminated, resulting in data inconsistency problem (not recommended).
<3>. The best practice to stop the thread gracefully is to call the interrupt method + interrupt judgment method of the Thread class (highly recommended).
The code demonstration is as follows:
@Test
public void testInterrupted() throws InterruptedException {
Thread thread = new Thread() {
@SneakyThrows
public void run() {
log.info("线程启动了...");
//一直循环
while (true) {
log.info(String.valueOf(isInterrupted()));
Thread.sleep(1000);
//如果线程被中断，就退出死循环
if (isInterrupted()) {
log.info("线程结束了...");
return;
}
}
}
};
thread.start();
Thread.sleep(2000); //等待2秒
thread.interrupt(); //中断线程
}7. What is the difference between 0.0.0.0, 127.0.0.1, and localhost?
The main differences are as follows:
The 0.0.0.0 address, used in IPv4 to indicate an invalid, unknown or unavailable destination.
The 127.0.0.1 address is a type of loopback address (the loopback address means that all data packets sent to this type of address should be looped back). Therefore, you can test the network on a certain machine by using ping 127.0.0.1 Whether the device, operating system, or TCP/IP implementation is working properly.
localhost, to be precise, is a domain name, not an IP address, used to refer to this computer or this host, which can be used to obtain network services running on this machine. In fact, most machine addresses will map localhost to the 127.0.0.1 address for IPv4, or the ::1 address for IPv6.
Extras:
0.0.0.0 address: When a host has not been assigned an IP address by DHCP, it means the host itself. When used on the server side, it means any IPV4 address on the machine. When used as a route, it represents the default route, that is, the corresponding route when no exact matching route is found in the routing table.
127.0.0.1 address: It should be noted that all addresses with a network number of 127 are loopback addresses, and 127.0.0.1 is just one of the loopback addresses. In practical applications, it is used as the IP address bound to the domain name localhost in most web container tests. In DDos attack defense, point the domain name to 127.0.0.1, allowing attackers to attack themselves.
localhost: As a domain name, it is necessary to distinguish whether to use ipv4 or ipv6. On most systems, the binding is the 127.0.0.1 address of IPv4, or the ::0 address of IPv6. In the local web test, the localhost domain name is generally used.
8. Use xxl-job to restart the first machine regularly at 0:00 am every day
Background: When doing a data audit project, a machine always reports an error "java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED" every few days.
After investigation, it was found that the basic reason: the call chain inside the ES client is IOReactor->performRequestAsync's Listener -> onFailure. When a short-term jitter triggers an exception thrown in onFailure, the entire IO Reactor will eventually become unavailable, and subsequent requests will be affected. Need Restart to recover.
Therefore, configure two scheduled tasks through xxl-job, and restart the error-reporting machine at 0:00 am every day (make sure that there is no scheduled task executing on the machine around 0:00 am). Write script code in xxl-job's GLUE IDEA, as follows:
Restart the first machine regularly every day --kill
#!/bin/bash
echo "xxl-job: hello shell"
ssh 用户名@机器IP < . /home/xxw/tmp/audit-kill.sh
echo "Good bye!"
exit 0Restart the first machine regularly every day --restart
#!/bin/bash
echo "xxl-job: hello shell"
ssh 用户名@机器IP < . /home/xxw/tmp/audit-restart.sh
echo "Good bye!"
exit 0A shell script is involved:
audit-kill.sh
#!/bin/bash
kill -9 `jps |grep 'xxxxx.jar' |awk '{print $1}'`audit-restart.sh
#!/bin/bash
echo "restart begin....."
sh xxxxx/cfg/start.sh start
echo "restart end....."9. Win+r common query in Windows system
Use the shortcut key win+r to call out the run pop-up box, and you can open the corresponding content panel through the commands that come with the Windows system.
According to function, it can be divided into:
computer management
compmgmt.msc
-------------【Computer Management】sub-item comes with commands ------------------
# Device Manager: devmgmt.msc
# Disk Management: diskmgmt.msc
# Local services: services.msc
# Performance Monitor: perfmon.msc
# Local users and groups: lusrmgr.msc
# Event viewer: eventvwr
computer management
# Task Manager: taskmgr generally uses the shortcut key ctrl+shift+esc
# File Explorer: explorer generally uses the shortcut key Win+E
# Registry Editor: regedit (used when deleting some programs)
# Certificate Management Device: certmgr.msc (view all certificates on the computer, not commonly used)
# Local Security Policy secpol.msc (computer security settings, not commonly used, no win10 home version)
# Group policy editor gpedit.msc (win10 home version none)
control Panel
control
------------- [Control Panel] sub-items have their own commands ------------------
desk.cpl (desktop settings)
main .cpl (Mouse Settings)
inetcpl.cpl (Internet Properties)
ncpa.cpl (Network Connections)
mmsys.cpl (Sounds and Audio Devices)
powercfg.cpl (Power Options)
sysdm.cpl (System Properties)
firewall.cpl (Firewall)
System comes with tools
calc (calculator)
osk (virtual keyboard)
write (WordPad)
notepad (notepad)
mspaint (drawing)
magnify (magnifying glass)
snippingtool (screenshot tool, supports irregular screenshots)
mplayer2 (simple widnows media player)
Sndvol (volume control program)
shutdown command
shutdown -s -t 0 (shutdown after x seconds, x ranges from 0-300, 0 means shutdown immediately)
shutdown -r -t 0 (restarts after x seconds, x ranges from 0-300, 0 means restart immediately)
tsshutdn (60-second countdown to shutdown, no win10 home version)
rononce -p (15-second countdown to shutdown, no win10 home version)
Most of the above commands are personally tested and practical~
10. Use Redis to count the number of user visits to the website
There are the following solutions, each with its own advantages and disadvantages, which can be selected according to the actual scene requirements.
The first one uses the hash data type (hset+hlen command) .
Use the hset command, key=website homepage address+date of the day, field, if the user has logged in to store the user ID, if not logged in, a unique identifier will be randomly generated to represent the user ID, and the value is set to 1 by default. When finally counting the website user visits on a certain day, use the hlen command.
# Simulate the process of users visiting the homepage of the website. The fourth command belongs to repeated users and will not be counted in the number of visitors. > hset http://123/index.html20230307 user_id1 1 1 >
hset
http://123/index .html20230307 user_id2 1
1
> hset http://123/index.html20230307 radom_id1 1
1
> hset http://123/index.html20230307 user_id2 1
0
> hlen http://123/index.html20230307
3
The advantages of this method are: simple and easy to implement, convenient to query, and high statistical accuracy. However, the disadvantages are also obvious: as more and more keys occupy more memory, performance bottlenecks may occur.
Please note that this method can be used for websites with low traffic volume, but not for e-commerce websites!
The second is to use the bitmap structure.
Bitmap uses a bit to represent the value or state corresponding to an element, and the key is the corresponding element itself. Commonly used commands are:
- SETBIT key offset value: For the string value stored in the key, set or clear the bit (bit) at the specified offset.
- BITCOUNT key [start] [end]: Count the number of bits set to 1 in a given string.
Just like this, bitmap is often used to do: user sign-in, active users, online users and other functions. Here, the setbit command, key=website home page address+date of the day, offset=bit identification (mark 1 indicates a user), value is 1 by default. Use the bitcount command to count the number of daily visits to the web page.
# Simulate the process of using bitset to count users visiting the homepage of the website
> setbit http://123/index.html20230307 1 1
0
> hset http://123/index.html20230307 11 1
0
> hset http://123/index.html20230307 100 1
0
> hset http://123/index.html20230307 100 1
1
> bitcount http://123/index.html20230307
3
The advantage of using bitmap is that it occupies less memory, is convenient to query, and has high statistical accuracy. However, non-login users may be mapped to the same user ID, and additional overhead will be added if they need to be distinguished.
In addition, bitmap is suitable for non-sparse user distribution, and users who are too sparse may occupy more memory than the first method!
The third is to use the HyperLogLog probability algorithm.
Redis HyperLogLog is an algorithm for cardinality statistics. The advantage is that when the number or volume of input elements is very, very large, the space required to calculate the cardinality is always fixed and small.
In Redis, each HyperLogLog key only needs 12 KB of memory to calculate the cardinality of nearly 2^64 different elements. This is in stark contrast to collections where more elements consume more memory when computing cardinality. However, because HyperLogLog only calculates the cardinality based on the input elements and does not store the input elements themselves, it cannot return the individual elements of the input like a set.
When the user visits the website, we can use the PFADD command (the Pfadd command adds all element parameters to the HyperLogLog data structure, format: PFADD key element [element ...]), set the corresponding command, and finally we only need to pass PFCOUNT The final result can be calculated smoothly, because this is only a probability algorithm, so there may be an error of 0.81%.
> pfadd http://123/index.html20230310 a b c d e f g h i j k l m n o p q
(integer) 1
>pfcount http://123/index.html20230310
(integer) 17
Advantages: It takes up very little memory. For a key, it only needs 12kb. It is especially suitable for e-commerce websites with a large number of users.
Disadvantages: When querying a specified user, errors may occur, after all, the storage is not specific data. There is also some error in the total. It can only count the number of people online, and cannot realize any other functions. This solution can only count the total number of online users in a certain period of time, but it can't do anything to the list of online users, but it saves memory. We can consider this solution when there are not many requirements for statistical data.
Summary of the second week of March, here it is~