The front end obtains the image stream through http and dumps it

1. Scene

        Yesterday, the company raised a requirement to import competitors' SKUs into our own database through some means. Common data is easier to handle, but we can only get one url for static resources such as pictures, so we need to perform a dump.

2. Realize

        There are two ideas here, one is to leave it to the back end to do this, but if the back end does it, it will expose our own back end and leave traces for the opponent. The second is to do it on the front end. Every time you use a different computer to access the website to do this data transfer operation, it will make it more difficult to track, because the other party does not know whether we are accessing resources normally.

        In the end, I chose the front end to do it. The first step is to obtain the data of the website through http, simulate login to obtain token, and intercept its product list data. This will not post the code, and it can also be achieved without using code.

        second step. Read and upload the pictures in the obtained product data

/** 
* @descrip 将blob对象转换成file对象
* @param {Blob} blob  需要转换成文件的blob对象
* @param {string} fileName 文件名
* @param {string} fileType 文件类型
* @return {Promise} 返回一个promise对象，即处理之后的文件对象
**/
const transToFile = async(blob, fileName, fileType) => {
    return new window.File([blob], fileName, { type: fileType })
}

/** 
* @descrip 将互联网上面的图片读取成buffer并且转存至私有的服务器
* @param {string} url  需要转换成的图片的公网地址 eg:https://image.xxx.com/uploads/image/product/62b2babba1c9c_thumb.jpg
* @return {Promise} 返回一个promise对象，即上传成功之后的图片url
**/
const getFileStreamAndUpload = async (url) => {
    return new Promise((resolve, reject) => {
        let suffix = "jpg"
        let fileNameArr = url.split("/")
        let fileName = fileNameArr[fileNameArr.length - 1]
        if (url.split(".").length > 1) {
            suffix = url.split(".")[1]
        }
        // 通过axios读取图片时一定要加上这个responseType，不然是乱码
        axios.get(url, {responseType: 'arraybuffer'}).then(async res => {
            // 构建blob对象
            const blob = new Blob([res.data], { type: `image/${suffix};charset=utf-8` })
            let getFile = transToFile(blob, fileName, `image/${suffix}`)
            getFile.then(async res1 => {
                // 通过FormData将文件提交给后端
                let formData = new FormData()
                formData.append('file', res1)
                formData.append('folderName', "fsImage")
                let res2 = await bar.uploadFile(formData)
                if (res2.code == 200) {
                    resolve(res2.data)
                } else {
                    reject(new Error("图片上传失败"))
                }
            })
        })
    })
}

             // 获取数据成功之后
            const loadingInstance = ElLoading.service({ fullscreen: true })
            // 这里的res.data.data 就是竞争对手的商品列表数据
            let fsData = res.data.data
            for (let i = 0; i < fsData.length; i++) {
                const item = fsData[i];
            // 这里可能会出现跨域请求资源的情况所以使用代理将真实的域名代理到本地的开发服务器上面
                item.productUrl = await getFileStreamAndUpload(item.thumb_path.replace("https://image.xxx.com", "/fsimgapi"))
            }
            console.log(fsData);
            loadingInstance.close()
            ElMessage.success("数据转移成功")

 In the effect diagram, after the conversion is completed, a converted array is directly output, and then the array can be submitted to the backend according to business requirements.

 

 

3. Summary 

In fact, this kind of function is best done by the backend or nodejs, and the pure frontend will inevitably encounter cross-domain problems. The main technical point is to read the static resources on the network into the browser cache in the form of streams, and then use the mutual conversion between Blob objects and File objects to obtain file objects one by one. Finally, through FormData uploads the data.