Actual Combat: Building a Java Crawler by Hand - Based on JDK11 Native HttpClient (2)

Table of contents

Frontier

Basic tool package

Date processing tool package (DateUtil)

File operation tool package (FileUtil)

---

        In the previous article, we introduced the basic environment. In this chapter, we started to package tools.

Frontier

To achieve a relatively perfect visual crawler effect, we need to encapsulate three things:

1) Basic tool package

2) Http request encapsulation

3) Netty message service encapsulation

Therefore, let's start by packaging some common tools.

Basic tool package

Date processing tool package (DateUtil)

DateUtil.java

package com.vtarj.pythagoras.tools.date;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

/**
 * @Author Vtarj
 * @Description 日期操作工具箱
 * @Time 2022/4/7 10:33
 **/
public class DateUtil {
    /**
     * 日期转字符串
     * @param date  待转换的日期
     * @param model 待转换的格式
     * @return  转换后的日期字符串
     */
    public static String dateToString(Date date,String model) {
        SimpleDateFormat sdf = new SimpleDateFormat(model);
        return sdf.format(date);
    }

    /**
     * 字符串转日期
     * @param dateStr   待转换的日期字符串
     * @param model 待转换的格式
     * @return  转换后的日期
     */
    public static Date stringToDate(String dateStr,String model) {
        SimpleDateFormat sdf = new SimpleDateFormat(model);
        try {
            return sdf.parse(dateStr);
        } catch (ParseException e) {
            throw new RuntimeException(e);
        }
    }
}

File operation tool package (FileUtil)

FileUtil.java 

package com.vtarj.pythagoras.tools.file;

import java.io.*;

/**
 * @Author Vtarj
 * @Description 文件操作工具箱
 * @Time 2022/4/7 9:45
 **/
public class FileUtil {

    /**
     * 创建文件
     * @param path  文件路径
     * @return  返回空文件
     */
    public static File build(String path) {
        File file = new File(path);
        if (!file.getParentFile().exists()) {
            file.getParentFile().mkdirs();
        }
        if (!file.exists()) {
            try {
                file.createNewFile();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return file;
    }

    /**
     * 删除文件
     * @param path  文件路径
     */
    public static void delete(String path) {
        File file = new File(path);
        if (file.exists()) {
            file.delete();
        }
    }

    /**
     * 写入文件操作
     * @param file  待写入的文件
     * @param context   待写入内容
     * @param isAppend  是否追加内容，true 追加，false 覆盖
     */
    public static void write(File file,String context,boolean isAppend) {
        FileWriter writer = null;
        PrintWriter printer = null;
        try {
            writer = new FileWriter(file,isAppend);
            printer = new PrintWriter(writer);
            printer.print(context);
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (writer != null) {
                try {
                    writer.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            if (printer != null) printer.close();
        }
    }

    /**
     * 读取文件内容，以字符串形式输出
     * @param file  待读取的文件
     * @return  文件内容
     */
    public static String read(File file) {
        StringBuilder result = null;
        FileReader reader = null;
        try {
            reader = new FileReader(file);
            char[] chars = new char[24];
            int len;
            while ((len = reader.read(chars)) != -1) {
                result = (result == null ? new StringBuilder("null") : result).append(new String(chars, 0, len));
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }
        return result == null ? null : result.toString();
    }

    /**
     * 根据文件路径返回文件
     * @param path  文件路径
     * @return  文件
     */
    public static File get(String path) {
        File file = new File(path);
        return file.exists() ? file : null;
    }

    /**
     * 重命名文件
     * @param file  待重命名的文件
     * @param newName   新文件名称
     */
    public static void rename(File file,String newName) {
        //获取原文件名称
        String fileName = file.getName();
        //获取新文件名称，并拼接后缀
        newName = newName + fileName.substring(fileName.lastIndexOf("."));
        //获取目标文件路径
        String destPath = file.getParentFile().getPath() + "/" + newName;
        //判断目标文件是否已存在
        File destFile = new File(destPath);
        if (destFile.exists()) {
            throw new RuntimeException("文件已存在，请重新命名！");
        }
        //重命名文件
        file.renameTo(destFile);
    }
}

Of course, you can also refer to other third-party toolkits for the basic tools, but we said that it is purely handcrafted, so write it yourself~~

To be continued~~~

Previous: Actual Combat: Building a Java Crawler by Hand - Based on JDK11 Native HttpClient (1)

Next: Actual Combat: Building a Java Crawler by Hand - Based on JDK11 Native HttpClient (3)