Table of contents
(2) Database addition, deletion, modification and query
3. Java reads python scripts and implements dynamic parameter passing
Foreword:
This time I am doing a project in the university data synthesis practice, I will not talk about the specific content of the project. Here I mainly want to write a part of my work on CSDN and record it. At the same time, it would be better if I can provide reference for others.
1. Project overview
Let me briefly talk about my project first. I made a project for predicting the click rate of advertisements. This project comes from the 2021 iFLYTEK AI Developer Contest-Advertising Click-through Rate Prediction Challenge . The data set is "Advertising Click-through Rate Estimation.csv", which contains 13 feature fields, 6-day data, and a total of more than 390,000 entries. The following is a description of its characteristic fields:
Then our requirements are:
1. Preprocessing the data
2. Data visualization operation
3. Using machine learning or deep learning methods to predict
4. Use Mysql to store data sets to realize data addition, deletion, modification and query
5. Design the judgment button, use the trained model to predict the data queried in the database and output the results to the designed interface. This function also needs to meet the following requirements: In addition to displaying the prediction results on the interface, it also needs to provide a save function to save it in the form of a file on the local disk
Here I will focus on telling you about the related work I have done and the problems I have encountered.
2. Mysql database
Before that, I didn't have much knowledge about databases, or rather superficial. So you have to learn from scratch. For downloading and installing software, I won’t go into details here. I am using Mysql benchwork and the console that comes with it.
(1) Import data
First of all, we put our 390,000 pieces of data, what should we do? right! Let's create a table first, probably as follows: I created a new database named hym , and created a new table named gg in the database
It is worth noting that when I build the table, I need to determine whether each column variable is an integer (int) or a character (varchar (255)) based on the original csv file. Be careful not to make these mistakes when building the table, otherwise An error will be reported later. If the string is very long, remember to note that the length allocated to it must be greater than its own length, otherwise an error will be reported, such as varchar( 255 )
For the various data types of Mysql, you can read the following articles!
Then how to import the data? There are two methods here. If you have very little data, you can directly use the software's built-in data import function, as shown below:
We right-click our table, there is a Table Date Import Wizard, my junior high school English level can understand directly, it is data import! We choose our file and import it (there are also many tutorial articles on the Internet). But this one has a very serious drawback! ! ! It’s just that the import is too slow when there is too much data , because it is imported one by one. If you have millions of data, the result can be imagined (⌯꒪꒫꒪)੭
So we directly choose the second method and use the console command to import. We enter the following console and enter our root password:
Then we only need to enter a few simple lines of instructions to get it done~, but remember to select the database first. (Veterans know it, write it to help some noobs who don’t know anything)
Load data infile 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/delete_new.csv' into table gg fields terminated by ',';Load data infile is an instruction to import data, which can be followed by many parameters. delete_new.csv is the data set I have processed, 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/' is the address where we put the data set csv file, so that the database can recognize the data (this is the case for my computer ), into table gg means that on the gg table we created, terminated by ',' means that their separators are separated by ','.
However, you still need to pay attention to some details when importing. For example, for the null values in the data set, you need to process them in advance or use relevant codes to standardize them in advance. Otherwise, an error will appear during the data entry process, or it will be troublesome to re-import. I am lazy. , chirp chirp (*∩ω∩).
After the import is complete, let's check to see if it was successful:
We can directly view the top 1000 entries directly in the workbench software!
What? ? You said you want to see all the data? ! ◔ ‸◔? There should be many, many methods on the Internet, but it is better to go to the csv file directly, or use the database command to search!
(2) Database addition, deletion, modification and query
Then I used python to simply implement the addition, deletion, modification and query of the database. Here, the python library pymysql is directly called. The following code is the connection operation of the database. Remember to open your database before connecting.
import pymysql
def Connection():
try:
db = pymysql.connect(host="localhost", user="root"
, password="自己的密码", database="自己的数据库")
print('数据库连接成功!')
except pymysql.Error as e:
print('数据库连接失败'+str(e))
finally:
db.close()
Connection()For the addition, deletion, modification and query, the basic commands of the database are used. Let’s briefly talk about it here. If you need it, you can search it online! For example, this: Python connects to the mysql database to add, delete, modify and check_Lazy_Goat's blog-CSDN blog .
1. Add command (increase): use the insert statement, for example, if I want to add new data for my project, I need to enter the value of each column variable (I omitted it here, only the first three variables), but the general format is this!
2. Delete command (delete): use the delete statement, the following figure is to delete the data according to the id number.
3. Modify/update command (modify): Use the update statement, which is also to modify the other three variables for the id number. I only wrote some variables here, and of course you can add them yourself as needed.
4. Search command (search): use the find statement, the following figure searches for data based on the id number, but if you have fuzzy query needs, you can find out by yourself, I didn't do it here.
3. Java reads python scripts and implements dynamic parameter passing
(1) Initial exploration
I use python and C++ a lot, and I use very little java ૮₍ ˃ ⤙ ˂ ₎ა, so this is the place where I spend the longest time, because basically it starts from 0, I borrow from the Internet A lot of information. I found that there are several ways to read python scripts. I will focus on the two I explored here. The following articles are examples of some methods
(1 message) How to use python library in java - CSDN library 
https://wenku.csdn.net/answer/20587f5dae09441a8acec27633ab7bb7#:~:text=java%E8%B0%83%E7%94%A8python%201%20% E4%BD%BF%E7%94%A8%20Java%20%E8%87%AA%E5%B8%A6%E7%9A%84%20Java%20Runtime%EF%BC%88java.lang.Runtime%EF% BC%89%E7%B1%BB%E7%9A%84%20exec, Java%20%E5%92%8C%20Python%20%E4%B9%8B%E9%97%B4%E5%88%9B% E5%BB%BA%E4%BA%92%E7%9B%B8%E8%B0%83%E7%94%A8%E7%9A%84%E9%80%9A%E9%81%93%EF% BC%8C%E5%8F%AF%E4%BB%A5%E5%9C%A8%20Java%20%E7%A8%8B%E5%BA%8F%E4%B8%AD%E8%B0%83% E7%94%A8%20Python%20%E7%A8%8B%E5%BA%8F%E3%80%82 I tried Jython first , but I found out that Jython has not been updated for a long time, and it does not support Python's third-party library , well, go straight to G. I have to do machine learning for this project! And data analysis! There are many third-party libraries.
So we simply turned our attention to the most basic method, using the exec() that comes with java , this one is even more heavyweight, I worked on it for two days before it was done! Maybe this is pediatrics for many big guys, but it's a bit cruel for me, a cute newbie. I frantically searched for information. Of course, there are many ideas on the Internet, which gave me a lot of reference, but moderate did not solve my needs. So I thought hard and did all kinds of debugging, and finally succeeded! I hope the following content can help those novices and friends who have needs in this regard! ✧*｡ (ˊᗜˋ*) ✧*｡ The following are some articles I borrowed from, and you can also read them if you need them:
(1 message) Java Runtime.getRuntime().exec() has no return value when calling python with a third - party library spm=1001.2101.3001.6650.1&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-1-106955890-blog-104648277.235%5Ev28%5Epc_relevant_t0_down load&depth_1-utm_source=distribute.pc_relevant.none-task-blog- 2%7Edefault%7ECTRLIST%7ERate-1-106955890-blog-104648277.235%5Ev28%5Epc_relevant_t0_download&utm_relevant_index=2 ( 1 message) Dynamically pass parameters in Java to call Python script___Thorny's blog-CSDN blog https://blog.csdn. net/thorny_v/article/details/61417386
good! Now back to the topic:
In the case of a packaged py script, if you do not need to pass in parameters but only need to execute the py file, you can get a lot of written code directly online, such as the following article, which I think is very well written good. It also contains the most basic dynamic parameter passing, but it can't solve my problem. (1 message) Dynamically pass parameters in Java to call Python script___Thorny's Blog-CSDN Blog https://blog.csdn.net/thorny_v/article/details/61417386
Dynamic parameter passing problem:
I searched a lot of information and found that the sys module must be introduced in the python script file, and sys.argv[i] is the script name. i starts from 1, if we need to pass in 9 parameters, it will be from 1 to 9. The following article is an article that gave me ideas
How Python Gets Command Line Parameters
Let's first look at the more basic dynamic parameter passing code (java)
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
public class python {
public static void main(String[] args) {
// Python脚本文件路径
String pythonScriptPath = "你的python文件的路径";
// 参数传递给Python脚本的字符串
String arg1 = " ";
try {
// 构建命令行执行Python脚本的命令
String[] cmd = {"编译器的环境，如果你有虚拟环境的编译器，可以直接输入到python.exe", pythonScriptPath, arg1};
Process process = Runtime.getRuntime().exec(cmd);
// 获取Python脚本的输出流
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
// 读取Python脚本的输出
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
// 等待Python脚本执行完毕
process.waitFor();
// 获取Python脚本的返回值
int exitCode = process.exitValue();
System.out.println("Python脚本执行完毕，返回值：" + exitCode);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The above code is a relatively basic code for reading python scripts. I have marked some places with comments. It is worth emphasizing that the parameters followed in string cmd[ ]: the first one is the compiler environment, like my This kind of virtual environment is built, and third-party libraries are downloaded in it. I will enter my compiler address into the virtual environment I created and find python.exe. The second is the address of the python script file, and the third is the parameters that need to be passed in. Of course, there can be unlimited parameters in the back, according to your own needs.
Based on the above code, combined with my own exploration and various search information, I finally figured out how to combine it with the database! Next, I will take it out and tell you in detail! (◍˃̶ᗜ˂̶◍)✩
(2) Specific examples
a. Database delete operation
Take it out first because this is relatively simple compared to others. If we delete data based on the id number, let's first look at the java code:
private void deleteData(String id) {
String sql = "DELETE FROM gg WHERE id=" + id; // SQL删除语句
String[] cmd = { "C:\\Users\\Lenovo\\AppData\\Local\\Continuum\\anaconda3\\envs\\tf2\\python.exe"
, "D:\\myeclipse workspace\\java418\\src\\student\\delete.py", sql }; // Python脚本文件和参数
try {
ProcessBuilder pb = new ProcessBuilder(cmd);
Process p = pb.start();
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
// 显示删除结果
textArea.append(line + "\n");
}
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}This sql is the id parameter we want to pass in . The previous address is my path, you can change it to your own, and gg is my data table, you have to change it to your own. Next, let's look at the corresponding py script file:
import pymysql
import sys
# 从Java传递的第一个参数是SQL语句
sql = sys.argv[1] #！！！！！！！！！！！！！！！
# 数据库连接信息
host = "localhost"
user = "root"
password = "password"
database = "hym"
# 连接数据库
conn = pymysql.connect(host='localhost', user='root', password='自己的密码', db='自己的数据库')
cursor = conn.cursor()
# 执行SQL语句
try:
cursor.execute(sql)
conn.commit()
print("删除成功!")
except Exception as e:
conn.rollback()
print("删除失败 错误信息：", e)
# 关闭数据库连接
cursor.close()
conn.close()Notice! See that sys.argv[1], that is very important! Must have, so that the id will be passed in
There is another important question, do not use the input() function to input in python, the input() function is useless in dynamic parameter passing in java, we just give it directly.
b. Add operation of database
java code:
// 调用Python脚本添加数据
String[] cmd = {"C:\\Users\\Lenovo\\AppData\\Local\\Continuum\\anaconda3\\envs\\tf2\\python.exe",
"D:\\myeclipse workspace\\java418\\src\\student\\add.py",
id, date, user_id, product, campaign_id, webpage_id, product_category_id,
user_group_id, gender, age_level, user_depth, var_1, isClick};
try {
Process process = Runtime.getRuntime().exec(cmd);
process.waitFor();
} catch (Exception ex) {
ex.printStackTrace();
}Python script code:
import pymysql
import sys
# 从Java传递的参数是13个值：id,date,user_id,product,campaign_id,webpage_id,product_category_id,
# user_group_id,gender,age_level,user_depth,var_1,isClick
id = sys.argv[1]
date = sys.argv[2]
user_id = sys.argv[3]
product = sys.argv[4]
campaign_id = sys.argv[5]
webpage_id = sys.argv[6]
product_category_id = sys.argv[7]
user_group_id = sys.argv[8]
gender = sys.argv[9]
age_level = sys.argv[10]
user_depth = sys.argv[11]
var_1 = sys.argv[12]
isClick = sys.argv[13]
# 连接数据库
conn = pymysql.connect(host='localhost', user='root', password='自己的密码', db='自己的数据库')
cursor = conn.cursor()
# 执行插入操作
try:
sql = "INSERT INTO gg (id,date,user_id,product,campaign_id,webpage_id,product_category_id,user_group_id,gender,age_level,user_depth,var_1,isClick) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"
val = (id,date,user_id,product,campaign_id,webpage_id,product_category_id,user_group_id,gender,age_level,user_depth,var_1,isClick)
cursor.execute(sql, val)
conn.commit()
print("数据插入成功!")
except Exception as e:
conn.rollback()
print("数据插入失败! 错误信息：", e)
cursor.close()
conn.close()As for the search operation and update operation, they are similar to the above, except that the database statement is different and the number of parameters passed is different. You can try it yourself. If you have any needs, you can private message me or give me a comment!
4. Other details
Regarding other aspects of the project, such as data analysis, machine learning methods, and the construction of the final interface , I have participated in and modified them, but the content is a bit too much, so I won’t go into details here. If you have a need, you can also chat with me privately to discuss with me. Below are screenshots of some projects~
Summarize
This time is a little bit of my learning experience. Welcome everyone to communicate with me. I am also constantly learning! If you have any problems, you can point them out, let's make progress together and work hard together. If this article is helpful to you or you think it is well written, I still hope that everyone will give me a thumbs up to encourage me hahahahahaha, thank you everyone~
