Chrom extension development cooperates with Baidu image and text recognition to realize automatic login (backend.net core web api)

I haven’t done browser plug-in development for a long time, because the company’s bastion machine requires you to enter your account password and verification code every time you log in. It's such a waste of time, so I thought of making an extension in the form of a right-click menu.

The implementation idea is also very simple. I will record it here for reference in the next development.

First, let’s first understand the necessary files for chrome extension development.

manifest.jsonAlso called a manifest file.
Let’s take a brief look at the configuration:

 //需要做C#兼职  或者浏览器插件兼职的请加我QQ:3388486286
{
    
    
    "manifest_version": 2,
    "name": "堡垒机自动化",
    "version": "1.0",
    "description": "右键菜单跳转到堡垒机页面,并自动登录",
    "permissions": 
    [
        "tabs", 
        "contextMenus",
        "webRequest",
        "webRequestBlocking",
        "https://127.0.0.4/*",
        "http://127.0.0.4/*"
    ],
    "background": {
    
    
      "scripts": ["background.js"]
    },
    "content_scripts": [
        {
    
    
          "matches": [
           "https://127.0.0.4/*",
           "http://127.0.0.4/*"
        ],
          "js": ["JS/log.js"]
        }
      ],
    "web_accessible_resources": [
        "JS/config.json"
      ],
    "icons": {
    
    
      "16": "Images/icon.png",
      "48": "Images/icon.png",
      "128": "Images/icon.png"
    }
  }
  

The above configuration basically includes common configurations for plug-in development. Now let’s briefly introduce the role of each configuration:

1.: manifest_versionIt means version, the version of the plug-in you need to use, but it is already the third version. Because I have used 2 for development before, the syntax of 2 and 3 is different. It is estimated that this version still uses the syntax of the old version. . I don’t have time to study the new version, so it can be used anyway.

2.: namePlug-in name (customized)

3.: versionThe version of the current plug-in (customized)

4 description.: Plug-in description

5.: permissionsPermissions (which APIs you need to use require corresponding activation permissions, also called registration services)

 "tabs", (该权限可以打开新的页面)
"contextMenus",(该权限可以右键创建新的菜单)
"webRequest",(该权限可以监听网络请求)
"webRequestBlocking",(该权限可以监听网络请求并且修改请求)
"https://127.0.0.4/*",(表示插件需要访问以 HTTPS 协议开头、IP 地址为 127.0.0.4 的所有页面或资源。这意味着插件将被允许在浏览器中与这些特定的 IP 地址和相关页面进行通信,无论是通过 HTTP 还是 HTTPS 访问。)
"http://127.0.0.4/*"(表示插件需要访问以 HTTP 协议开头、IP 地址为 127.0.0.4 的所有页面或资源。)
添加对应的地址作用:这意味着插件将被允许在浏览器中与这些特定的 IP 地址和相关页面进行通信,无论是通过 HTTP 还是 HTTPS 访问。
  1. "background":
    { "scripts": ["background.js"] }该脚本会一直运行监听。

The specific meaning is as follows:

"background":表示指定插件的后台脚本。
"scripts":表示指定后台脚本文件的路径。
"background.js":表示插件的后台页面脚本文件名为 "background.js"。
通过将脚本文件名添加到 "scripts" 数组中,你可以告诉浏览器插件在加载时要运行哪个脚本文件作为后台页面。这个后台页面脚本通常用于处理插件的核心功能、与浏览器 API 进行交互以及响应来自其他插件部分(如内容脚本或浏览器操作栏)的事件。

7.content_scripts

 //content_scripts  为业务JS注入,就是你要操作前端页面元素的JS  其中matches表示你JS要注入到哪些页面地址。
 "content_scripts": [
        {
    
    
          "matches": [
           "https://127.0.0.4/*",
           "http://127.0.0.4/*"
        ],
          "js": ["JS/log.js"]   //需要注入的Js文件
        }
      ]

8. web_accessible_resources: There are often some static global configuration files placed here.
9.: iconsIcon file.

2. Let’s talk about background.jsthe idea first.

1. Register the right-click menu first, so that when you right-click on the browser, you can see your extension.

chrome.contextMenus.create({
    
    
    title:"堡垒机自动化",
    onclick:function(){
    
    
        console.log('准备跳转..')
        //跳转指定页面
        chrome.tabs.create({
    
    url:'https://127.0.0.4/index.php/login'},(tab)=>{
    
    
            console.log('跳转成功..')
            console.log(tab)
            //执行业务代码 注入JS
            chrome.tabs.executeScript(tab.id,{
    
     file: 'JS/content_script.js'})
        })
  
    }

})

It executeScriptis also a way to inject JS, which is more flexible and can also interact with page elements.

2. After jumping to the specified page, we need to enter the account number and password. This business logic is written content_script.jsin


var credentials = {
    
    };

// 读取配置文件并存储到全局对象
async function getConfig() {
    
    
  try {
    
    
    const response = await fetch(chrome.runtime.getURL('JS/config.json'));
    credentials = await response.json();
    console.log(credentials); // 打印全局对象
    // 调用填充函数
    fillCredentials();
  } catch (error) {
    
    
    console.error('Error reading config file:', error);
  }
} 

// 在页面上填充账号和密码
function fillCredentials() {
    
    
    document.querySelector("#pwd_username").value = credentials.username;
    document.querySelector("#pwd_pwd").value = credentials.password;
    //GetAccessToken();
  }

Here we call getConfig()the method to read the configuration file, and then assign it to the global variable storage. , and then call to fillCredentials()assign the account password.

3. Recognize the verification code and assign a value to log in.
Our verification code is a 4-digit letter picture. I implemented this logic directly by calling Baidu API. I originally wanted to implement it on my own backend, but when I used a third-party library, either the response time was too long or the recognition was inaccurate. . Baidu API directly helps you solve these two troubles. But there is a fee, and I only used the 100 free trials.
Now let’s talk about the specific implementation ideas. I originally wanted to send the request for recognition directly to the front end, but the browser has a same-origin policy and cross-domain... This will eventually require the use of back-end services, so I used the back-end API to do image recognition. Functional.

3.1 First obtain the URL address of the verification code image.
Here we background.jsadd network monitoring in because the image URL has a fixed format and is easy to match. The specific code is as follows:

// 声明一个变量用于保存监听器
let requestListener;
// 启动网络请求监听器

    requestListener = function(details) {
    
    

      // 检查请求地址是否以指定的 URL 开头
      if (details.url.startsWith('https://127.0.0.4/captcha/')) {
    
    

        // 提取图片地址
        const imageUrl = details.url;
  
        // 输出图片地址
        console.log('图片地址:', imageUrl);
  
        // 在这里可以进行进一步的处理
        sendImageURLToContentScript(imageUrl);
  
  
      }
    }
  
    chrome.webRequest.onBeforeRequest.addListener(
      requestListener,
      {
    
     urls: ['https://127.0.0.4/captcha/*'] }, // 监听以指定 URL 开头的请求
      ['blocking']
    );

By chrome.webRequest.onBeforeRequest.addListenerturning on network monitoring, when the information at the beginning of the specified URL is matched, it means that the URL is an image URL, and then passed to it. Here is a imageUrlstatement content_script.js
:
background.jsit cannot communicate directly with the page being operated, and background.jsmainly operates the API provided by the browser. The actual operation of Dom elements requires business JS. The business JS here is content_script.js
so background.jshow content_script.jsto communicate with it, look at the code:

 // 在background.js中获取到图片地址后,延迟1秒发送消息给业务 JS
  function sendImageURLToContentScript(imageUrl) {
    
    
    chrome.tabs.query({
    
     active: true, currentWindow: true }, function(tabs) {
    
    
      if (tabs.length > 0) {
    
    
        const tabId = tabs[0].id;
  
        setTimeout(function() {
    
    
          chrome.tabs.sendMessage(tabId, {
    
     imageUrl: imageUrl }, function(response) {
    
    
            // 在收到业务 JS 的响应后进行处理(可选)
            console.log('收到业务 JS 的响应:', response);
          });
        }, 500); // 延迟1秒后发送消息
      }
    });
  }

Send a message via chrome.tabs.sendMessage.

// 业务 JS 接收消息,并进行处理
chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    
    
    // 接收到来自 background.js 的消息
    if (request.imageUrl) {
    
    
      const imageUrl = request.imageUrl;
      // 在这里进行业务逻辑处理,使用获取到的图片地址
      console.log('收到来自 background.js 的图片地址:', imageUrl);
      GetCode(imageUrl);
      //GetBase64(imageUrl);
      // 可以通过 sendResponse 向 background.js 发送响应消息(可选)
      sendResponse({
    
     message: '已收到图片地址' });
    }
  });

By chrome.runtime.onMessage.addListenerreceiving messages
, communication background.jswith and content_script.js.
As for why it is delayed by 500 milliseconds before sending, it is because it is injected into the page after we right-click, content_script.jsand at this time, background.jsit is already running again. All delays will resend the message to avoid content_script.jsnot receiving the message there.

3.2 Business JS sends imageUrl to the back-end api for image verification code recognition

 function GetCode(imageUrl)
  {
    
    
    fetch(credentials.getcode_url, {
    
    
        method: "POST",
        headers: {
    
    
          "Content-Type": "application/json"
        },
        body: JSON.stringify(imageUrl)
      })
      .then(response => response.text())
      .then(result => {
    
    
        let data = JSON.parse(result);

        // 获取 words_result 字段的值  
        let wordsResult = data.words_result;
        
        // 获取数组中第一个元素的 words 字段的值
        let words = wordsResult[0].words;
        console.log(words);  // 输出: "code"

        //赋值code
        document.querySelector("#pwd_captcha").value = words;

        //点击登录
       document.querySelector("#sign-box > form.form-vertical.login-content.active > div.submit-row > button").click();

      })
      .catch(error => {
    
    
        console.error("请求出错:", error);
      });

  }

Okay, this way, the verification code recognition is realized, and then the login operation is assigned.

3. Give all relevant codes

config.json:

{
    
    
    "username": "username",
    "password": "password",
    "client_id":"FrktGAjFVjGv9SSA6S3",
    "client_secret":"IFL6FbU6tuFrPoCjaYnvvRrCGd",
    "token_url":"https://aip.baidubce.com/oauth/2.0/token",
    "getcode_url":"http://localhost:5270/api/CodeIdentity"
}

background.js


// 声明一个变量用于保存监听器
let requestListener;
// 启动网络请求监听器

    requestListener = function(details) {
    
    

      // 检查请求地址是否以指定的 URL 开头
      if (details.url.startsWith('https://127.0.0.4/captcha/')) {
    
    

        // 提取图片地址
        const imageUrl = details.url;
  
        // 输出图片地址
        console.log('图片地址:', imageUrl);
  
        // 在这里可以进行进一步的处理
        sendImageURLToContentScript(imageUrl);
  
  
      }
    }
  
    chrome.webRequest.onBeforeRequest.addListener(
      requestListener,
      {
    
     urls: ['https://127.0.0.4/captcha/*'] }, // 监听以指定 URL 开头的请求
      ['blocking']
    );

  
  // 在background.js中获取到图片地址后,延迟1秒发送消息给业务 JS
  function sendImageURLToContentScript(imageUrl) {
    
    
    chrome.tabs.query({
    
     active: true, currentWindow: true }, function(tabs) {
    
    
      if (tabs.length > 0) {
    
    
        const tabId = tabs[0].id;
  
        setTimeout(function() {
    
    
          chrome.tabs.sendMessage(tabId, {
    
     imageUrl: imageUrl }, function(response) {
    
    
            // 在收到业务 JS 的响应后进行处理(可选)
            console.log('收到业务 JS 的响应:', response);
          });
        }, 500); // 延迟1秒后发送消息
      }
    });
  }
  
  

chrome.contextMenus.create({
    
    
    title:"堡垒机自动化",
    onclick:function(){
    
    
        console.log('准备跳转..')
        //跳转指定页面
        chrome.tabs.create({
    
    url:'https://127.0.0.4/index.php/login'},(tab)=>{
    
    
            console.log('跳转成功..')
            console.log(tab)
            //执行业务代码 注入JS
            chrome.tabs.executeScript(tab.id,{
    
     file: 'JS/content_script.js'})
        })
  
    }

})


// // 启动网络请求监听器
// chrome.webRequest.onBeforeRequest.addListener(
//     function(details) {
    
    
//       // 检查请求地址是否以指定的 URL 开头
//       if (details.url.startsWith('https://127.0.0.4/captcha/')) {
    
    
//         // 提取图片地址
//         const imageUrl = details.url;
  
//         // 输出图片地址
//         console.log('图片地址:', imageUrl);
  
//         // 在这里可以进行进一步的处理
//         sendImageURLToContentScript(imageUrl);
//       }
//     },
//     { urls: ['https://127.0.0.4/captcha/*'] }, // 监听以指定 URL 开头的请求
//     ['blocking']
//   );



// 在background.js中获取到图片地址后,延迟1秒发送消息给业务 JS
// function sendImageURLToContentScript(imageUrl) {
    
    
//     chrome.tabs.query({ active: true, currentWindow: true }, function(tabs) {
    
    
//       if (tabs.length > 0) {
    
    
//         const tabId = tabs[0].id;
  
//         setTimeout(function() {
    
    
//           chrome.tabs.sendMessage(tabId, { imageUrl: imageUrl }, function(response) {
    
    
//             // 在收到业务 JS 的响应后进行处理(可选)
//             console.log('收到业务 JS 的响应:', response);
//                  // 移除监听器
//         chrome.webRequest.onBeforeRequest.removeListener(requestListener);
//           });
//         }, 100); // 延迟1秒后发送消息
//       }
//     });
//   }
  
  // 开始监听网络请求
//startRequestListener();
  

content_script.js

// 业务 JS 接收消息,并进行处理
chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
    
    
    // 接收到来自 background.js 的消息
    if (request.imageUrl) {
    
    
      const imageUrl = request.imageUrl;
      // 在这里进行业务逻辑处理,使用获取到的图片地址
      console.log('收到来自 background.js 的图片地址:', imageUrl);
      GetCode(imageUrl);
      //GetBase64(imageUrl);
      // 可以通过 sendResponse 向 background.js 发送响应消息(可选)
      sendResponse({
    
     message: '已收到图片地址' });
    }
  });



var credentials = {
    
    };

// 读取配置文件并存储到全局对象
async function getConfig() {
    
    
  try {
    
    
    const response = await fetch(chrome.runtime.getURL('JS/config.json'));
    credentials = await response.json();
    console.log(credentials); // 打印全局对象
    // 调用填充函数
    fillCredentials();
  } catch (error) {
    
    
    console.error('Error reading config file:', error);
  }
} 

// 在页面上填充账号和密码
function fillCredentials() {
    
    
    document.querySelector("#pwd_username").value = credentials.username;
    document.querySelector("#pwd_pwd").value = credentials.password;
    //GetAccessToken();
  }

  /**
 * 使用 AK,SK 生成鉴权签名(Access Token)
 * @returns 鉴权签名信息(Access Token)
 */
//  function GetAccessToken() {
    
    
//     const url = credentials.token_url;
//     const data = {
    
    
//       grant_type: 'client_credentials',
//       client_id: credentials.client_id,
//       client_secret: credentials.client_secret
//     };
  
//     const requestOptions = {
    
    
//       method: 'POST',
//       mode: 'cors',
//       headers: { 'Content-Type': 'application/json' },
//       body: JSON.stringify(data)
//     };
  
//     return fetch(url, requestOptions)
//       .then(response => {
    
    
//         if (!response.ok) {
    
    
//           throw new Error('Network response was not ok');
//         }
//         return response.json();
//       })
//       .then(data => {
    
    
//         console.log(data);
//         console.log(data.access_token);
//         return data.access_token;
//       })
//       .catch(error => {
    
    
//         console.error(error);
//       });
//   }


  function GetCode(imageUrl)
  {
    
    
    fetch(credentials.getcode_url, {
    
    
        method: "POST",
        headers: {
    
    
          "Content-Type": "application/json"
        },
        body: JSON.stringify(imageUrl)
      })
      .then(response => response.text())
      .then(result => {
    
    
        let data = JSON.parse(result);

        // 获取 words_result 字段的值  
        let wordsResult = data.words_result;
        
        // 获取数组中第一个元素的 words 字段的值
        let words = wordsResult[0].words;
        console.log(words);  // 输出: "code"

        //赋值code
        document.querySelector("#pwd_captcha").value = words;

        //点击登录
       // document.querySelector("#sign-box > form.form-vertical.login-content.active > div.submit-row > button").click();

      })
      .catch(error => {
    
    
        console.error("请求出错:", error);
      });

  }




  //图片转base64
//   function getFileContentAsBase64(path) {
    
    
//     return new Promise((resolve, reject) => {
    
    
//       fetch(path)
//         .then(response => response.arrayBuffer())
//         .then(buffer => {
    
    
//           const bytes = new Uint8Array(buffer);
//           let binary = '';
//           for (let i = 0; i < bytes.byteLength; i++) {
    
    
//             binary += String.fromCharCode(bytes[i]);
//           }
//           const base64 = btoa(binary);
//           resolve(base64);
//         })
//         .catch(error => reject(error));
//     });
//   }
  

// function GetBase64(captchaUrl)
// {
    
    
// // 使用fetch函数获取验证码图片的二进制数据
// fetch(captchaUrl)
//   .then(response => response.blob())
//   .then(blob => {
    
    
//     // 创建一个FileReader对象来读取blob数据
//     const reader = new FileReader();
//     reader.onloadend = function() {
    
    
//       // 读取完成后,将二进制数据转换为Base64编码
//       const base64 = reader.result.split(',')[1];
      
//       // 调用getFileContentAsBase64方法进行后续处理
//       getFileContentAsBase64(base64)
//         .then(result => {
    
    
//           console.log("base64:",result);
//         })
//         .catch(error => {
    
    
//           console.error(error);
//         });
//     };
//     reader.readAsDataURL(blob);
//   })
//   .catch(error => {
    
    
//     console.error(error);
//   });

// }



//识别验证码

// 获取配置并存储到全局对象
getConfig();






Backend code:

using CodeIdentify.Servers;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.AddScoped<IBaiDuCodeIdentity, BaiDuCodeIdentityRepostoty>();
builder.Services.AddCors(options => {
    
    
    options.AddPolicy("AllowAll", builder =>
    {
    
    
        builder.AllowAnyOrigin()
               .AllowAnyMethod()
               .AllowAnyHeader();
    });
});

var app = builder.Build();

app.UseCors("AllowAll");
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    
    
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseAuthorization();

app.MapControllers();

app.Run();

namespace CodeIdentify.Servers
{
    
    
    public interface IBaiDuCodeIdentity
    {
    
    
        Task<string> GetAccessTokenAsync();

        Task<string> GetCodeAsync(string base64);

        Task DownloadImageAsync(string url , string saveDirectory);

        string GetFileContentAsBase64(string path);

        void DeleteImage(string path);
    }
}

using Microsoft.OpenApi.Models;
using Newtonsoft.Json;
using RestSharp;
using System.Net;

namespace CodeIdentify.Servers
{
    
    
    public class BaiDuCodeIdentityRepostoty : IBaiDuCodeIdentity
    {
    
    
        public void DeleteImage(string path)
        {
    
    
          
                try
                {
    
    
                    // 获取指定文件夹中的所有图片文件
                    string[] imageFiles = Directory.GetFiles(path, "*.jpg");

                    foreach (string file in imageFiles)
                    {
    
    
                        // 删除文件
                        File.Delete(file);

                        Console.WriteLine($"已删除文件: {
      
      file}");
                    }

                    Console.WriteLine("所有图片文件已删除。");
                }
                catch (Exception ex)
                {
    
    
                    throw new Exception(ex.Message);
                    Console.WriteLine($"删除图片文件时出错:{
      
      ex.Message}");
                }
            

        }

        public async Task DownloadImageAsync(string imageUrl, string saveDirectory)
        {
    
    
            // 创建自定义的 HttpClientHandler,并禁用证书验证
            var handler = new HttpClientHandler()
            {
    
    
                ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => true
            };

            // 创建 HttpClient 实例,并使用自定义的 HttpClientHandler
            using (HttpClient httpClient = new HttpClient(handler))
            {
    
    
                try
                {
    
    
                    // 发送 GET 请求并获取响应消息
                    HttpResponseMessage response = await httpClient.GetAsync(imageUrl);
                    response.EnsureSuccessStatusCode();

                    // 从响应消息中获取图片内容
                    byte[] imageBytes = await response.Content.ReadAsByteArrayAsync();

                    // 创建文件保存路径
                    Directory.CreateDirectory(saveDirectory);
                    string savePath = Path.Combine(saveDirectory, "image.jpg"); // 要保存的文件名

                    // 将图片内容保存到本地文件
                    File.WriteAllBytes(savePath, imageBytes);

                    Console.WriteLine("图片下载完成。");
                }
                catch (Exception ex)
                {
    
    
                    throw new  Exception($"图片下载失败:{
      
      ex.Message}");
                    Console.WriteLine($"图片下载失败:{
      
      ex.Message}");
                }
            }
        }


        public async Task<string> GetAccessTokenAsync()
        {
    
    
            try
            {
    
    
                var client = new RestClient($"https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=FrktGAjFVjSSA6S3Tcs0f&client_secret=IFL6FbU6rPoCjaSiKjMLYnvvRrCGd");
                var request = new RestRequest(String.Empty, Method.Post);
                request.AddHeader("Content-Type", "application/json");
                request.AddHeader("Accept", "application/json");
                var body = @"";
                request.AddParameter("application/json", body, ParameterType.RequestBody);
                var response = await client.ExecuteAsync(request);
                var result = JsonConvert.DeserializeObject<dynamic>(response.Content??"");
                Console.WriteLine(result?.access_token.ToString());
                return result?.access_token.ToString()??"";
            }
            catch (Exception e)
            {
    
    

                throw new Exception($"可能次数用完了:{
      
      e.Message}");
            }

        }

        public async Task<string> GetCodeAsync(string base64)
        {
    
    
            var token = await GetAccessTokenAsync();
            var client = new RestClient($"https://aip.baidubce.com/rest/2.0/ocr/v1/accurate_basic?access_token={
      
      token}");
            var request = new RestRequest(String.Empty, Method.Post);
            request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
            request.AddHeader("Accept", "application/json");
            // image 可以通过 GetFileBase64Content('C:\fakepath\captcha.png') 方法获取
            request.AddParameter("image", base64);
            var  response = await client.ExecuteAsync(request);
            Console.WriteLine(response.Content);
            return response.Content??"";
        }

        public string GetFileContentAsBase64(string path)
        {
    
    
            using (FileStream filestream = new FileStream(path, FileMode.Open))
            {
    
    
                byte[] arr = new byte[filestream.Length];
                filestream.Read(arr, 0, (int)filestream.Length);
                string base64 = Convert.ToBase64String(arr);
                return base64;
            }
        }
    }
}

using CodeIdentify.Servers;
using Microsoft.AspNetCore.Mvc;

namespace CodeIdentify.Controllers
{
    
    
    [Route("api/[controller]")]
    [ApiController]
    public class CodeIdentityController : ControllerBase
    {
    
    
        private readonly IBaiDuCodeIdentity _baiDuCodeIdentity;
        public CodeIdentityController(IBaiDuCodeIdentity baiDuCodeIdentity) 
        {
    
    
            _baiDuCodeIdentity = baiDuCodeIdentity;
        }
        [HttpPost]
        public async Task<IActionResult> GetCode([FromBody] string url) 
        {
    
    
            string path = "Images\\image.jpg";
            string deletepath = "Images";
            await _baiDuCodeIdentity.DownloadImageAsync(url, "Images");
            string code = await _baiDuCodeIdentity.GetCodeAsync(_baiDuCodeIdentity.GetFileContentAsBase64(path));
            if(string.IsNullOrWhiteSpace(code)) return BadRequest("空字符串,识别有误");
            _baiDuCodeIdentity.DeleteImage(deletepath);
            return Ok(code);
         

        } 
    }
}

Guess you like

Origin blog.csdn.net/csdn2990/article/details/132476200