How to realize all kinds of API debugging on the pure Web side?

In the process of software development, it is very important to debug various APIs. API debugging is a key step in verifying and testing the validity and correctness of application programming interfaces. Traditional API debugging methods usually rely on stand-alone tools or desktop applications, limiting the flexibility and efficiency of the debugging process.

In order to promote the development of API debugging in a more convenient and efficient direction, more and more developers are beginning to seek solutions to complete various API debugging on the pure Web side. Pure Web-side API debugging has many advantages, including no need to install additional software, cross-platform support, and easy team collaboration. This article will take the open source project AREX as an example to introduce how to realize the debugging function of various APIs on the Web side.

About AREX

AREX ( http://arextest.com/) is an open source automated regression testing platform based on real requests and data. It uses Java Agent technology and comparison technology to achieve fast and effective regression testing through traffic recording and playback capabilities. At the same time, it provides a wealth of automated testing functions such as interface testing and interface comparison testing.

Difficulty 1: Cross-domain restrictions

In order to realize the debugging of various APIs on the pure Web side, the first problem to be solved is to deal with the browser's cross-domain restrictions.

what is cross domain

Browser cross-domain issues refer to the limitations encountered when using JavaScript code to access resources of another domain name from a web page of one domain name in web development. The browser implements a security policy called Same-Origin Policy (Same-Origin Policy) to protect the security of user information. The same-origin policy requires that JavaScript in a web page can only access resources with the same origin (protocol, domain name, and port number), while access to resources with different domain names will be restricted.

Due to the browser's cross-domain restrictions, we cannot send HTTP requests freely on the browser side, which is determined by the browser's security policy.

solution

After investigation, there are two ways to break through this limitation: **Chrome** plug-in proxy and server-side proxy . The following is a comparison of the two methods.

	Chrome Plugin Proxy	server proxy
visit local	Can	Can't
speed	No request time loss	The speed of the whole process is affected by the proxy interface
actual request	Origin sources are known to be modified to source for Chrome plugins	exactly the same

On balance, AREX chose the Chrome plug-in proxy method. The principle is to use the ability of the background in the Chrome plug-in to send cross-domain requests. We communicate the requests intercepted by the browser window.postmassagewith Chrome plugin content-scriptas a data bridge).

The specific implementation is as follows:

in page script

Generate a random string, convert it to string form, and store it in the `tid` variable.
Use window.postMessage()the method to send a message to other extensions, the message includes an AREX_EXTENSION_REQUESTidentifier of type , tid, and params parameters.
Adds an messageevent listener receiveMessagefor messages sent by other extensions.

4. In receiveMessagethe function , check that the received message is of type AREX_EXTENSION_RESand that the tid matches the tid of the previously sent message. If the match is successful, the event listener is removed.

in the content script

Adds an messageevent listener for receiving messages from page scripts or other extensions.
In the event listener, check if the received message is of type AREX_EXTENSION_REQUEST, and if so, use chrome.runtime.sendMessage()the method to send the message to the background script.
After receiving a response from a background script, use window.postMessage()the method to send a response message back to the page script or other extension.

in background script

Use chrome.runtime.onMessage.addListener()the method to add a listener for messages sent from content scripts or other extensions.
In the listener you can process the received message and respond as needed.

// arex
const tid = String(Math.random());
window.postMessage(
  {
    type: '__AREX_EXTENSION_REQUEST__',
    tid: tid,
    payload: params,
  },
  '*',
);
window.addEventListener('message', receiveMessage);
function receiveMessage(ev: any) {
  if (ev.data.type === '__AREX_EXTENSION_RES__' && ev.data.tid == tid) {
    window.removeEventListener('message', receiveMessage, false);
  }
}
// content-script.js
window.addEventListener("message", (ev) => {
  if (ev.data.type === "__AREX_EXTENSION_REQUEST__"){
    chrome.runtime.sendMessage(ev.data, res => {
      //   与background通信
      window.postMessage(
        {
          type: "__AREX_EXTENSION_RES__",
          res,
          tid:ev.data.tid
        },
        "*"
      )
    })
  }
})
// background.js
chrome.runtime.onMessage.addListener((req, sender, sendResponse) => {

})

Difficulty 2: API debugging

The cross-domain problem has been solved above, and the next step is how to implement the function of API debugging.

solution

Postman is a mature API debugging tool in the industry. Standing on the shoulders of the giant Postman, we introduced Postman's JavaScript sandbox in AREX, and used its sandbox to run pre-scripts, post-scripts, and assertions to debug APIs. .

The following is a flowchart of the AREX request:

When clicking to send a request, the data in the form will be gathered together, and the data structure is:

export interface Request {
  id: string;
  name: string;
  method: string;
  endpoint: string;
  params: {key:string,value:string}[];
  headers: {key:string,value:string}[];
  preRequestScript: string;
  testScript: string;
  body: {contentType:string,body:string};
}

This is the data structure of AREX, which we will convert to the data structure of Postman. Then call PostmanRuntime.Runner() the method , pass in the converted Postman data structure and the currently selected environment variable, and the Runner will execute preRequestScriptthe and testScriptscripts . preRequestScriptBefore the request, you can intersperse the request and operate on the request parameters and environment variables. testScriptAfter the request, you can assert the data returned by the response, and you can also console.logoutput to debug on the console.

var runner = new runtime.Runner(); // runtime = require('postman-runtime');

// 一个标准的postman集合对象
var collection = new sdk.Collection();

runner.run(collection, {}, function (err, run) {
    run.start({
      assertion:function (){}, //断言
      prerequest:function (){}, // 预请求勾子
      test:function (){}, //测试勾子
      response:function (){} //返回勾子
    });
});

There are also cross-domain issues in the Postman sandbox. Since the integration PostmanRuntimeof , we have adopted Ajax interception technology. By intercepting the Ajax request on the browser side, we can modify the request, add custom logic or perform other processing operations. This enables global control and customization of requests and responses.

When the Postman sandbox sends a request, it will carry a request header named "postman-token". After we intercept the Ajax request, we assemble the request parameters and send them to the browser plug-in through window.postMessage. The browser plug-in constructs the fetch request again, returns the data to the Postman sandbox, and makes it output the final result, including the response (response), test result (testResult) and console log (console.log). It should be noted that the responseType must be specified as arraybuffer.

The specific process is as follows:

Register a request handler using xspy.onRequest()the method . This handler accepts two parameters: request and sendResponse. The request parameter contains information about the request, such as method, URL, header, request body, etc. sendResponse is a callback function used to send a response to the requester.

2. In the handler, check whether the request is from Postman by checking whether postman-tokenthere .

If this header is present, it means that the request was sent through Postman. Then use AgentAxios to initiate a new request, using the method, URL, header and request body of the original request. AgentAxios returns an agentData object, which contains information such as the status code, header and data of the response. Creates a response object dummyResponsenamed containing information related to the original request. The status field of dummyResponse is the status code of agentData, the headers field is the result of converting the header array of agentData into object format, the ajaxType field is the string xhr, the responseType field is the string arraybuffer, and the response field is the data conversion of agentData into JSON String and wrap the result in Buffer. Finally, send the response to the requester using sendResponse(dummyResponse).
If the request is not from Postman, call sendResponse() directly, indicating that no response is returned.

xspy.onRequest(async (request: any, sendResponse: any) => {
  // 判断是否是pm发的
  if (request.headers['postman-token']) {
    const agentData: any = await AgentAxios({
      method: request.method,
      url: request.url,
      headers: request.headers,
      data: request.body,
    });
    const dummyResponse = {
      status: agentData.status,
      headers: agentData.headers.reduce((p: any, c: { key: any; value: any }) => {
        return {
          ...p,
          [c.key]: c.value,
        };
      }, {}),
      ajaxType: 'xhr',
      responseType: 'arraybuffer',
      response: new Buffer(JSON.stringify(agentData.data)),
    };
    sendResponse(dummyResponse);
  } else {
    sendResponse();
  }
});

Difficulty 3: Binary object serialization transfer

It is also worth mentioning that x-www-form-urlencodedfor Rawrequests of type and , since they are common JSON objects, it is easier to handle them. But form-datafor binaryrequests of type and , support for transferring binary file payloads is required. However, postMessagethe communication does not support the direct passing of binary objects, which makes it impossible to directly handle these two types of requests.

solution

To solve this problem, AREX uses base64 encoding technology. When the user selects a file, AREX converts the binary file to a base64 string and then transmits it. On the Chrome plug-in side, AREX will decode the base64 data and use it to construct the actual fetchrequest . This bypasses the limitation of passing binary objects directly.

This flowchart describes the process of converting a binary file in FormData to a Base64 string, and converting it back to a file through the Chrome plugin agent for further processing.

form-data binary (A): Indicates a FormData form data containing a binary file.
FileReader (B): Use the FileReader object to read binary files.
readAsDataURL base64 string: FileReader uses the readAsDataURL method to read a binary file as a Base64 string.
Chrome extension agent (C): After the Base64 string is read, it is passed to the Chrome extension agent for further processing.
base64 string: Indicates the Base64 string obtained after the binary file is read by FileReader.
Uint8Array (D): Converts a Base64 string to a Uint8Array in the Chrome plugin agent.
File(E): Create a new File object using the data of Uint8Array.
fetch (F): Use the fetch method or other methods to further process the newly created File object, such as uploading to the server or performing other operations.

code analysis

The following is the analysis at the code level:

The toBase64 function takes a File object as a parameter and returns a Promise object that resolves to a Base64 string representing the file.

Inside the function, a FileReader object is created. Read the file as a Data URL by calling reader.readAsDataURL(file). When the read operation is complete, the read result is resolved to a string by the reader.onload event handler and passed to the Promise using resolve. If an error occurs, the error will be passed to the Promise using reject. The base64ToFile function takes two parameters: dataurl (Base64 string) and filename (filename), and returns a File object.

First, split the dataurl into an array arr using commas, and if the split result is empty, set it to an array containing an empty string. Match the content in arr[0] through the regular expression to extract the MIME type, that is, the type of data. Use atob to decode a Base64 string to a binary string bstr. Creates a Uint8Array array u8arr of length n. Use a loop to traverse bstr, putting the Unicode encoding of each character into u8arr. Finally, use the File constructor to create and return a new File object that contains the file data, filename, and MIME type read from u8arr. Export the base64ToFile function for use elsewhere.

// 文件转Base64
const toBase64 = (file: File): Promise<string> =>
  new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.readAsDataURL(file);
    reader.onload = () => resolve(reader.result as string);
    reader.onerror = reject;
  });
// base64转文件
function base64ToFile(dataurl: string, filename: string) {
  const arr = dataurl.split(',') || [''],
    mime = arr[0].match(/:(.*?);/)?.[1],
    bstr = atob(arr[1]);
  let n = bstr.length;
  const u8arr = new Uint8Array(n);
  while (n--) {
    u8arr[n] = bstr.charCodeAt(n);
  }
  return new File([u8arr], filename, { type: mime });
}

export default base64ToFile;

AREX documentation: arextest.com/en-US/doc…
AREX official website: arextest.com/
AREX GitHub：github.com/arextest
AREX official QQ communication group: 656108079