How does a security novice have his own arsenal from scratch

I. Introduction

A few days ago, I was engaged in offense and defense with a big guy. I saw that the big guy was super god and I was super ghost. I had to admire the big guy. I saw that the big guy used many tools that I had never seen before. I asked what these tools were. Show me his big baby, hey, good guy, the whole folder is a tool developed by myself and some tools for magic modification, I am really envious, looking at those tools of the big guy, I fell into deep thought, am I the same? You can have your own arsenal, so you can't be a hand-out party, and you can be familiar with the language and strengthen yourself in code auditing. I just worked on development for a period of time before, and I still have my own ideas about developing some tools, so I just do it. This series is an article about how a security novice can have his own tool library from scratch. In terms of development I am also a rookie, so the development ideas of some tools may be different from those of popular tools. If there is any bad writing, I hope the big guys can guide me.

2. Preliminary preparation

2.1 Language selection

For development tools, you must know one or two languages. I believe that you guys must have mastered more than three or four languages. The younger brother has no ability. He only masters C, java, python, php, and go for development. The more common tools are java, python, and go. I am more familiar with java and python, and some common tools are also developed with java and python. I won’t introduce the difference between these two languages. Everyone understands Yes, which one to choose mainly depends on personal needs. If you want to save some trouble, you can use python. The library in it will only know how good it is, but the python virtual machine is not as strong as java. After all, the java virtual machine is the core of java, and python is Fully dynamic, you can modify your own code at runtime, java can only be realized through workarounds. Python variables are dynamic, while Java variables are static and need to be declared in advance, so the code hint function of Java IDE is better than that of Python IDE.

If you use java to develop tools, you are mainly used to using java to develop tools with interfaces, and if you use python, it is mainly because the library is powerful, you can save a lot of writing, and save a lot of time. Of course, it does not mean that python cannot create interfaces. Python has launched some Frameworks are used to develop interfaces, and the most commonly used one is Django. Django is a relatively advanced Python web framework known for its rapid development and practical and concise design. The choice of language depends on personal habits. No one is better. Bad statement.

2.2 Compiler

The compilers used for java are IDEA and eclipse, and the compilers used for python are PyCharm. These are very good compilers, but some of them can only be tried or found to be cracked. After downloading the compiler, build a project first, and take eclipse here As a demonstration, come in and build a project first, choose 1.8.0 here, then name it, create a com.AX.Database_listening, and then finish directly.

After finishing the construction, first look at the directory structure. SRC is where we write code later, and then JRE is the package we imported. We imported other packages in advance, so it will exist directly after creation.

Generally, we use things written by others when developing, and some of them need to be imported by you. After we download other people’s jars, we create a lib under our own project, pull in the jars we downloaded, and then right click project, click Properties, and select Java Build Path

Find the jar you just pulled in, select to add, and then save, the whole process is over, so it will be much more convenient for us to write other functions.

Then you can start writing your own code.

2.3 Language ability

After the language and compiler are selected, someone should ask, what level do I have to master the language to develop my own tools, hey, I don’t know how to answer this question, after all, I don’t learn it very deeply , as far as JAVA is concerned, if you have received systematic learning, you will basically enter the knowledge of threads, exceptions, file access, interfaces, inheritance, etc. after learning the basic grammar, and you may learn such as network communication protocols later. It allows you to write some small demos for communication. Of course, how far you need to be able to develop your own tools depends on your personal needs. I think basically as long as you know basic grammar and know functions, classes, subclasses, and parent classes, Rewrite, inherit and so on, you can develop some tools yourself.

If your tools are only for local things, you only need to understand the principle you want to achieve, and then design your development architecture based on this principle, then you don’t need to use any network communication protocols at all, but if If you need to connect to some resources on the network, then you still have to know the communication protocol. Of course, these are exceptions for Baidu engineers. If you search on Baidu for what you want to achieve, basically there will be answers, and Baidu can achieve 100%. of 99 difficulties.

2.4 Development ideas

No matter what you are developing, you must have a general framework for the results. It is not necessary to think about the interface first, but the functions, what functions to achieve, what effects these functions can achieve, and how to achieve these functions. Most tool development processes are consistent, basically three parts:

1. Data acquisition: how to obtain the data you want, such as how to communicate with the website, how to connect to the database, how to obtain the data of the database, etc.

2. Data processing: After getting the data, how to process the data and get the part of the data you want. Generally, when we get the data, there is no way to directly get the part of the data you want. Basically It is all brought back in one pot, and then screened and filtered according to the displayed data. Another situation is that the data you get needs further processing. For example, the data you get back needs another round of data analysis, or with you Compared with existing features, etc., the two aspects of data acquisition and data processing are the most difficult places when developing tools.

3. Data display: If the previous data acquisition and data processing are completed, then you have come to the simplest place. How to present your results, whether it is a command line or an interface, depends on personal preference.

Therefore, based on these three parts, you can basically develop most of the tools, and when you develop these tools, your thinking will be very clear, and you will always know which part you are in, and which part you are doing. There are many tutorials on the Internet for the solution of blocks. In fact, tools are similar to puzzles. You find out how to realize them here, then find out how to realize them, and then splicing them together, it becomes a tool. In a word, CCV, a stud, punch That's it. Attached here is my regular mind map when I was working on gadgets.

3. Development process

It would be a bit vague to talk about it here. Let me use a small tool I made before as an example. It would be better to directly talk about a development idea when I developed this tool. This is a detection tool for subdomains. The main function is to detect sub-domain names according to the domain name you input, and then eliminate those that cannot be accessed according to the status of the page, and save the accessible ones to save data. This is what I will use later to collect page samples and then classify the pages. So there are other functions. Here I mainly talk about the design ideas and development process of how to detect the status value of subdomain names.

3.1 Development Architecture

For the architecture, prepare according to the three processes, and then make a mind map (although it is not necessary, but it can make your own thinking clear later and will not go astray)

1. Data acquisition: how to connect to the website through the domain name, and then obtain the status value of the website

2. Data processing: how to process the data after obtaining the data, organize the data into the format I want, for example, domain name, IP, open port, website status, etc., and take out the useful data separately

3. Data display: After the data is processed, the collected results are displayed through the interface, and then the data is exported and saved in a fixed format to facilitate subsequent operations.

Paste the mind map, here is only the part of detecting the status value of the website, other functions are similar, everyone's thinking will be different, so there is no big problem here.

3.2 Achieving goals

3.2.1 Get website status value

How to obtain the status value of the website has been written in the mind map above, and the status code can be obtained through java's uConnection.getResponseCode(). Then judge whether the website exists, how to use the specific code and the returned value, the type can be Baidu by yourself, I will post the key code here

URL u = new URL("http://www.baidu.com");
try {  
HttpURLConnection uConnection = (HttpURLConnection)u.openConnection();  
try {  
uConnection.connect();
System.out.println(uConnection.getResponseCode());  
} catch (Exception e) {  
e.printStackTrace();  
}                    
} catch (IOException e) {  
e.printStackTrace();  
}

The above is a simple way to connect to the website through the url, get the status value returned by it, and then output the status value, because this is the connected www.baidu.com , so the return is 200, you can run it directly to see the content , this is a simple implementation process, and then you can import your own dictionary in, get the data of your dictionary, splice it in front of .baidu.com, and then make a multi-thread to let him keep detecting and get the returned value. .

3.2.2 Filter the data you want

Above we have obtained the status value of the website, then we can judge whether the data should be saved according to the returned status value, and an if can solve the problem. If you want to classify according to the status value, then use a switch That's it, for example, if we want to filter some data, use if to filter the data

if (!logres.equals(logsql) && !logres.contains("/* mysql") && !logres.contains("SHOW WARNINGS") && !logres.contains("select event_time,argument from%%"))
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------"+logres+"\n");
​

This is convenient, but when you want to filter too much data, you need to list very long conditions, which is too troublesome. At this time, you need to use other skills, such as saving the blacklist to the file and then According to the contents of the document to intercept and other methods.

3.2.3 Interface Implementation

Here are the next few panels. This is a relatively common panel in JAVAFX. In fact, the interface is nothing, just like HTML, one set of one, and a set of dolls. You just need to think about the position you want to display your own data in advance. Then go to arrange the position of the button, and the position of the data display will do.

• StackPane: Nodes are displayed from left to right and top to bottom in the order they were added.

• FlowPane: Nodes are displayed in 5 areas - top, bottom, left, right, center

• GridPane (grid panel): Nodes are arranged in a flexible row and column grid, which can arrange the positions of those elements according to their own arrangement (0,0), (0,1), (1,0) (1,1 )so

• BorderPane (boundary panel): Arrange the nodes in a horizontal row in an orderly manner. When a node reaches the border of the panel, it will not be displayed on a new line.

• Hbox, Vbox (box panel): Hbox arranges nodes in a vertical column in an orderly manner. When the node reaches the border of the panel, it will not be displayed in a new line, and the Vbox will arrange the nodes in an orderly manner. When a node reaches the border of the panel, it will not be displayed on a new line.

In fact, this is a very simple panel introduction, which is basically stacked according to these panels. I will post a sample code of a simple panel plus buttons here. It may be more understandable to run this.

label = new Label("accumulated count"+count);
Button btnPush = new Button ("+1"); 
//Set a label and button
btnPush.setOnAction(this::btnPushHandler);
//Set the event of the button, what will happen after the button is clicked
FlowPane pane = new FlowPane (btnPush,label);
//Then put the buttons and labels in the panel, this one is streamlined and arranged horizontally
pane.setHgap(20);
pane.setAlignment(Pos.CENTER);
Scene scene = new Scene(pane, 300,100);
//Set the panel size
stage.setScene(scene);
stage.setTitle("panel example");
//panel title
stage.show();
//display

This panel is also completed. After you design the panel, put your completed code into each button event. The above sentence btnPush.setOnAction(this::btnPbushHandler) is to set the btnPbushHandler event to Go inside the btnPush button, when the btnPush button is clicked, execute the event, for example

Connect_SQL = new Button("Connect to database");
//Set a button called Connect_SQL
Connect_SQL.setOnAction(this::ConnectSQL);
//Execute the ConnectSQL event after clicking the button
------------------------------------------------------------------------------------------------------------------------
//Write ConnectSQL, when the button is clicked, the ConnectSQL event is executed to connect to the database
public void ConnectSQL(ActionEvent event){
try {
ConnectSQL.Connect(sql_addr.getText(), sql_port.getText(), sql_user.getText(), sql_password.getText());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

In this way, the button and the event response are connected, and the corresponding buttons and events are designed to achieve the goal you want to achieve. Basically, the tool is completed. After the tool is written, it is impossible to enter the compiler every time. Go run it, export it into a .jar file, and then execute the file directly.

Right-click the project, click Export, and select jar file

Then check your own project, directly finish, and you are done.

At this point, the writing of the entire tool is completed

Four, actual combat

4.1 Tool description

The idea and implementation method of a Mysql monitoring tool realizes the specified database name to monitor the sql statement related to it, and displays the specific sql statement separately. The online tools I used at the beginning were not very ideal, so I planned to write one myself. At present, only these two functions have been implemented, and it will be changed into a proxy audit tool later to realize jumping from sql statements to codes and so on.

4.2 Development ideas

The idea of ​​the tool is actually very simple, mainly these four pieces

1. Connect to the database

2. Monitor database logs

3. Filter data in database logs

4. Display Data

The ultimate goal of the tool is: after the user connects to the database, when testing code or website functions, the tool can be filtered to locate the executed SQL statement, which is convenient for code auditing

4.3 Preliminary preparation

• Connect to the database to achieve

• Compatible with different Mysql versions

• How to monitor the database

• filter dirty data

• Accurately filter out the sentences you want

4.4 Goal Achievement

4.4.1 Connecting to the database

Implementation ideas

To connect to the database, I use JDBC to connect to the MySQL database. If you want to connect to MySQL, you need a driver package. The latest version of the download link is: MySQL :: Download Connector/J . After decompression, you will get the jar library file, and then import the library into the corresponding project. document. If you want a compatible version, you need two jars, one compatible with 8 or more, and one compatible with the following.

The database connection part of MySQL 8.0 and above is different:

MySQL 8.0 or above driver package version mysql-connector-java-8.0.16.jar .

com.mysql.jdbc.Driver is replaced by com.mysql.cj.jdbc.Driver.

Versions above MySQL 8.0 do not need to establish an SSL connection and need to be explicitly disabled.

allowPublicKeyRetrieval=true allows the client to obtain the key from the server, and finally needs to set the CST

The way to load the driver and connect to the database is as follows:

Class.forName("com.mysql.cj.jdbc.Driver");
conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/test_demo?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC","root","password");
MySQL version below 8.0 - JDBC driver name and database URL   
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver"; 
static final String DB_URL = "jdbc:mysql://[localhost:3306/RUNOOB](http://localhost:3306/RUNOOB)";
// MySQL 8.0 and above - JDBC driver name and database URL    
static final String JDBC_DRIVER = "com.mysql.cj.jdbc.Driver"; 
static final String DB_URL = "jdbc:mysql://[localhost:3306/RUNOOB?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC](http://localhost:3306/RUNOOB?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC)";

key code

public static EventHandler<ActionEvent> Connect(String sql_addr, String sql_port, String sql_user,String sql_password)throws ClassNotFoundException, SQLException  {
// TODO Auto-generated method stub
USER = sql_user;
PASS = sql_password;
Connection conn = null;
Statement stmt = null;
try{
JDBC_DRIVER = "com.mysql.jdbc.Driver";
DB_URL = "jdbc:mysql://" + sql_addr + ":" + sql_port + "/mysql?useUnicode=true&characterEncoding=utf8&serverTimezone=GMT%2B8&useSSL=false";
// Register JDBC driver
Class.forName(JDBC_DRIVER);
// open the link
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------Connecting to the database..."+"\n");
// System.out.println("Connecting to the database...");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
conn.prepareStatement("SET global general_log=on").executeUpdate();
conn.prepareStatement("SET GLOBAL log_output='table'").executeUpdate();
DatabaseMetaData metaData = (DatabaseMetaData) conn.getMetaData();
String version = metaData.getDatabaseProductVersion();//Get database version information
// System.out.println(time.format(Long.valueOf(System.currentTimeMillis()))+"--The current database version is: "+version);
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------The current database version is: "+version+"\n");

}catch(SQLException se){

JDBC_DRIVER = "com.mysql.cj.jdbc.Driver";
DB_URL = "jdbc:mysql://"+sql_addr+":"+sql_port+"/mysql?useUnicode=true&characterEncoding=utf8&serverTimezone=GMT%2B8&useSSL=false";
// Register JDBC driver
Class.forName(JDBC_DRIVER);
// open the link
// System.out.println("Connecting to the database...");
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------Connecting to the database..."+"\n");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
DatabaseMetaData metaData = (DatabaseMetaData) conn.getMetaData();
conn.prepareStatement("SET global general_log=on").executeUpdate();
conn.prepareStatement("SET GLOBAL log_output='table'").executeUpdate();
String version = metaData.getDatabaseProductVersion();//Get database version information
// System.out.println(time.format(Long.valueOf(System.currentTimeMillis()))+"--The current database version is: "+version);
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------The current database version is: "+version+"\n");

}catch(Exception e){
// Handle Class.forName errors
e.printStackTrace();
}
return null;
}

4.4.2 Monitor database logs

Monitoring ideas

At the beginning, I thought about directly monitoring the logs of a certain database, but later found that the configuration file of the database needs to be modified, which is not very convenient. After all, I don’t want to change the configuration file of the database every time I monitor one (I want to be lazy), and It is to monitor the cache file and monitor a table in the mysql database. I think it may be changed into a code audit tool in the future, so I directly monitor the entire mysql log, and then filter the dirty data later.

According to the above idea, I directly monitor the general_log table in Mysql, but later I found that there are a lot of dirty data, and because this is the entire Mysql, the data of those database operations will also be displayed, which made it embarrassing for me to filter dirty data later. , in order to verify the feasibility, directly use if to filter those useless data, and later change it to a blacklist, directly add those dirty data and then filter it

key code

String logsql = "select * from mysql.general_log where command_type =\"Query\" OR command_type =\"Execute\" order by event_time desc limit 1,15";
Connection conn = DriverManager.getConnection(DB_URL, USER, PASS);
ResultSet rs = conn.prepareStatement(logsql).executeQuery();
while (rs.next()) {
String logres = rs.getString("argument");
//System.out.println(time.format(Long.valueOf(System.currentTimeMillis()))+"："+logres);
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------"+logres+"\n");
}

Connect to the database, then query the data in the table, and then output continuously, but too much useless data is displayed, basically every time it will display SET NAMES utf8, SELECT @@session.autocommit, SHOW WARNINGS these data come out, too disturbing to see those It is a normal statement, so I just used a layer of if to filter these data at first, but later found that it is not very flexible, so I made a blacklist, and I can dynamically add and delete rules to intercept those data.

4.4.3 Filtering data

filter ideas

When I first started to acquire, I saw a lot of other data interference, such as SELECT QUERY_ID, SHOW STATUS, SHOW FULL TABLES WHERE Table_type != 'VIEW', etc., as mentioned above, I used an if to filter those data with this feature, In fact, it is very easy to use, but I want to dynamically adjust these blacklists later, so I use a drop-down list to store data, and then process the data in the drop-down list after sending the data, and then process the obtained The database information is filtered out one by one to achieve the effect of filtering.

key code

if (!logres.equals(logsql) && !logres.contains("/* mysql") && !logres.contains("SHOW WARNINGS") && !logres.contains("select event_time,argument from%%") && !logres.contains("SELECT STATE") && !logres.contains("SHOW STATUS") && !logres.contains("SELECT QUERY_ID")){
Database_listening.sql_date.appendText(time.format(Long.valueOf(System.currentTimeMillis()))+"------"+logres+"\n");
}

At this point, the tool is basically finished. This is a relatively simple tool, and the development idea is relatively simple, and there are no strange operations. It is basically basic grammar, which is very suitable for beginners to develop tools.

After typesetting the panel, add the function to each button event to export the jar file, and it can be used after running. Of course, it can also be used directly in the compiler. It depends on personal choice.

V. Summary

In general, developing tools is actually not difficult, just split each piece into multiple small points to complete, and then splicing them together. This is the first time I write this kind of article about tool development, some may not express it well, I try my best Write down the points that need to be paid attention to. This is for Xiaobai, so some simple things should be carefully explained. Big brothers can skip selectively. This is the first article, and the following words will be based on I usually write some development ideas or tutorials for some plug-ins and tools that I usually develop. I may not talk about the panel later. If there is something wrong, I hope the big guys will guide me.