Android Charles captures packets, how to capture packets

Charles captures packets on the Android side

Author: Yang Chong
Source: https://juejin.cn/post/6874903020677791758

Catalog introduction

• 01. Download and install
• 02. Capture proxy settings
• 03. Packet capture Https operation
• 04. Introduction to the principle of packet capture
• 05. Introduction to packet capture data
• 06. Summary of common problems
• 07. Android interception and packet capture

01. Download and install

• Download address (just download the corresponding platform software)
  • www.charlesproxy.com/download/
• download crack file
  • assets.examplecode.cn/file/charle…
  • Open Finder, select Charles in Applications and right-click to select Show Package Contents
  • After displaying the contents of the package, copy the cracked file in the Content/Java directory and replace the original file.
  • If you are prompted when you open Charles: The program is damaged and cannot be opened. You should move it to the trash. At this point, you need to execute the following command in the terminal: sudo spctl --master-disable

02. Capture proxy settings

• charles proxy settings
  • You can set the packet capture data type, including http and socket data. You can check it under the proxies column as needed. Here is a simple operation to set, the default port of Proxy —> Proxy Settings is 8888, which can be modified according to the actual situation.
    • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-UIPV5P0V-1684069809248)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/89a88baa0a3747dea6bbc4b89eec16fc ~tplv-k3u1fbpfcp-zoom-1.image)]
• Android phone proxy settings
  • First get the ip address of the computer
    • The first way: View the local IP address: Help —> Local IP Addresses
      • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-EnHVXhge-1684069809250)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/f0b6dc7968464cf597268286aecf4d83 ~tplv-k3u1fbpfcp-zoom-1.image)]
    • The second way: command line mode, just enter ifconfig
      • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-jnT8K2ci-1684069809251)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/4c92a5b7d40f4bf59921a4f824ed0394 ~tplv-k3u1fbpfcp-zoom-1.image)]
  • Then open the phone to set the proxy
    • Note: The mobile phone needs to use the same Wi-Fi network as the computer, this is the premise! ! !
    • Operation steps: Open the WiFi list —> long press the connected WiFi to modify the network settings proxy —> set the proxy information
      • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-YYyJhGix-1684069809251)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/0380983c56404b8d87f4b746d33aa39c ~tplv-k3u1fbpfcp-zoom-1.image)]
• The final capture is as follows
  • The packet capture data is as follows
    • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-xrQG4luS-1684069809252)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/10cfb65297484292a5e6d281f29b14c9 ~tplv-k3u1fbpfcp-zoom-1.image)]

03. Packet capture Https operation

• what needs to be done

  • 1. The certificate needs to be installed on the computer
  • 2. The certificate needs to be installed on the mobile phone
  • 3. The Android project code setting is compatible

• 1. The certificate needs to be installed on the computer

  • The first step is to install the certificate: help —> SSl Proxying —> install charles root certificate —> install the certificate
  • The second step is to set the SSL properties: Proxy —> SSL Proxy Settings —> then add operation (set port to 443). As follows
    • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-7S06V777-1684069809252)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/ec7c67168b914e18b645129d8d60aa28 ~tplv-k3u1fbpfcp-zoom-1.image)]
  • Then try capturing packets, you will find that Android 7.0 phones can capture packets before, but after Android 7.0, they cannot capture packets
    • Error message: Client SSL handshake failed: An unknown problem occurred while processing the certificate (certificate_unknown)
    • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-e81uLKwS-1684069809253)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/8fd031a7898a4e7a8fa5aa08d7beb242 ~tplv-k3u1fbpfcp-zoom-1.image)]
  • How to solve the problem of capturing https information after Android 7.0, and then look down.

• 2. The certificate needs to be installed on the mobile phone

  • The first step is to download the certificate
    • Open the browser, enter: chls.pro/ssl, it will be downloaded to the mobile phone by itself, here you need to remember the path where the download is completed and saved to the local.
  • The second step is to install the certificate
    • Settings—>More Settings—>System Security—>Encryption and Credentials—>Install from SD card, select the path where the certificate was saved before.
    • Note that some mobile phones can be installed directly by clicking the downloaded file...
  • The installation operation is shown in the figure below
    • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-5ii1Pfs1-1684069809253)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/14635e45662243ca9bf84c06e9342944 ~tplv-k3u1fbpfcp-zoom-1.image)]

• 3. The Android project code setting is compatible

  • Add a security profile. As follows:

  • java.security.cert.CertPathValidatorException: Trust anchor for certification path not found This exception, the solution is as follows:

    <?xml version="1.0" encoding="utf-8"?>

  //Manifest file configuration
  

  • Why can't Android 7.0 and above easily capture the plaintext data requested by Https?
    • In Android 7.0(API 24), there is a new security feature called "Network Security Configuration". The goal of this new feature is to allow developers to customize their web security settings without modifying application code. If the application is running with a system version higher than or equal to 24, and targetSdkVersion>=24, only the system (system) certificate will be trusted. Therefore, the Charles root certificate imported by the user (user) is not trusted.

• The final result of grabbing https is as follows

  • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-PFXLPiWa-1684069809254)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/877a24aeeb17435585d340e8beaa33bf ~tplv-k3u1fbpfcp-zoom-1.image)]
  • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-vFFpeBjE-1684069809254)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/87765c0fd0f148bcb867f69229cb3f1c ~tplv-k3u1fbpfcp-zoom-1.image)]

04. Introduction to the principle of packet capture

• 1. The principle of packet capture:
  • acting. Client request -> Proxy -> Arrive at server Server return -> Proxy -> Arrive at client
• 2. Can any Https app capture it?
  • Below 7.0 is possible, as long as the corresponding CA certificate is installed in the mobile phone, such as capturing packets with charles, the mobile phone needs to install the certificate provided by charles.
  • After Android 7.0, Google introduced a more stringent security mechanism. Apps do not trust user certificates by default (install certificates in mobile phones), and their own apps can be resolved through configuration, but HTTPS requests from other apps will not work.
• 3. How to avoid packet capture
  • 1. Based on the principle of packet capture, you can directly use okhtttp to prohibit the proxy, builder.proxy(Proxy.NO_PROXY); after testing, you can avoid packet capture
  • 2. Directly use the encryption protocol, all the fields are garbled, and replace the domain name with IP. This is basically difficult for others to catch, like confusion
• 4. Schematic diagram of charles packet capture
  • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-afsT3FuH-1684069809255)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/610be0fb64d74013ab8d3cc9cd783207 ~tplv-k3u1fbpfcp-zoom-1.image)]
• 5. Approximate steps process
  • In the first step, the client initiates an HTTPS request to the server, Charles intercepts the HTTPS request sent by the client to the server, and Charles pretends to be the client to send a request to the server for a handshake.
  • In the second step, the server sends back the response. Charles obtains the server's CA certificate, uses the root certificate (the root certificate here is the certificate issued by the CA certification center to itself) to decrypt the public key, verifies the server data signature, and obtains the server CA certificate public key. Then charles forges its own CA certificate (the CA certificate here is also the root certificate, but it is the root certificate forged by charles), pretending to be the server certificate and passing it to the client browser.
  • The third step is the same as the operation of the client in the normal process. The client performs certificate verification based on the returned data, generates the password Pre_master, encrypts with the certificate public key forged by Charles, and generates the symmetric key enc_key for HTTPS communication.
  • In the fourth step, the client sends important information to the server, which is intercepted by Charles. Charles decrypts the intercepted ciphertext with the private key of his forged certificate, obtains and calculates the symmetric key enc_key for HTTPS communication. Charles encrypts the symmetric key with the public key of the server certificate and transmits it to the server.
  • The fifth step is the same as the server-side operation in the normal process. The server uses the private key to unlock and establish trust, and then sends an encrypted handshake message to the client.
  • In the sixth step, Charles intercepts the ciphertext sent by the server, decrypts it with a symmetric key, and then encrypts it with the private key of his forged certificate and sends it to the client.
  • Step 7: After the client gets the encrypted information, it uses the public key to unlock it and verify the HASH. The handshake process is officially completed, and the "trust" is established between the client and the server.
  • In the subsequent normal encrypted communication process, how does Charles act as a third party between the server and the client?
  • Server—>Client: Charles receives the ciphertext sent by the server, uses the symmetric key to decrypt it, and obtains the plaintext sent by the server. Encrypt it again and send it to the client.
  • Client -> Server: The client encrypts with a symmetric key, and after being intercepted by Charles, decrypts to obtain the plaintext. Encrypt it again and send it to the server. Since charles always has the symmetric key enc_key for communication, the information is transparent to it during the entire HTTPS communication process.
• 6. Summarize
  • The principle of HTTPS packet capture is quite simple. To put it simply, Charles acts as a "middleman agent" and obtains the public key of the server certificate and the symmetric key of the HTTPS connection. The premise is that the client chooses to trust and install the CA certificate of Charles, otherwise The client will "alarm" and abort the connection. In this way, HTTPS is still very secure .

05. Introduction to packet capture data

• The structure of the HTTP request packet

  • request message

    • Request message structure format:

      请求行： <method> <request-URL> <version>
      

      head:
      body:

    • Schematic diagram of the request message structure:

      • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-RsnsWCnS-1684069809256)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/2ddaa81110db4c619bc1094fa23ee7a4 ~tplv-k3u1fbpfcp-zoom-1.image)]

    • example:

      • After the request, you will receive a response packet (if there is an HTTP server on the opposite side)

        POST /meme.php/home/user/login HTTP/1.1
        

        Host: 114.215.86.90
        Cache-Control: no-cache
        Postman-Token: bd243d6b-da03-902f-0a2c-8e9377f6f6ed
        Content-Type: application/x-www-form-urlencoded

        tel=13637829200&password=123456

  • common are those

    • User-Agent: The type of browser that made the request.
    • Accept: list of response content types recognized by the client;
    • Accept-Language: the natural language acceptable to the client;
    • Accept-Encoding: The encoding and compression format acceptable to the client;
    • Host: The requested host name, which allows multiple domain names to be in the same IP address, that is, a virtual host;
    • Connection: connection mode (close or keep-alive);
    • Cookie: stored in the client extension field, and send the cookie belonging to the domain to the server of the same domain name;

• HTTP response packet structure

  • response message

    • Response message structure format:

      状态行：  <version> <status> <reason-phrase>
      

      Response header:
      Response body:

    • Schematic diagram of the response message structure:

      • [External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-Ib6BetZn-1684069809257)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b6ed92e417c74163959f372a8f391335 ~tplv-k3u1fbpfcp-zoom-1.image)]

    • example:

      HTTP/1.1 200 OK
      

      Date: Sat, 02 Jan 2016 13:20:55 GMT
      Server: Apache/2.4.6 (CentOS) PHP/5.6.14
      X-Powered-By: PHP/5.6.14
      Content-Length: 78
      Keep-Alive: timeout=5, max=100 Connection: Keep-Alive
      Content-Type: application/json; charset=utf-8

      {“status”:202,“info”:“\u6b64\u7528\u6237\u4e0d\u5b58\u5728\uff01”,“data”:null}

  • Common response header parameters

    • AllowWhich request methods (such as GET, POST, etc.) are supported by the server.
    • Content-EncodingThe encoding (Encode) method of the document.
    • Content-LengthIndicates the content length. This data is only required if the browser uses persistent HTTP connections.
    • Content-TypeIndicates what MIME type the following document belongs to.
    • Serverserver name.
    • Set-CookieSets the cookie associated with the page.
    • ETag: The entity value of the requested variable. ETag is a token (MD5 value) that can be associated with a web resource.
    • Cache-Control: This field is used to specify the instructions that all caching mechanisms must obey throughout the request/response chain.

• Response message status code

  • Contains the status code and reason phrase to inform the client of the result of the request.

  • About the status code, you can read this article, http status code . status code	category	reason phrase
    1XX	Informational (informational status code)	The received request is being processed
    2XX	Success (success status code)	The request is processed normally
    3XX	Redirection (redirection status code)	Additional action is required to complete the request
    4XX	Client Error (client error status code)	The server was unable to process the request
    5XX	Server Error (server error status code)	An error occurred while the server was processing the request

06. Summary of common problems

• 1. Unable to open APP after configuration
  • When we crawled, we encountered some APPs that could not be opened after configuring the proxy. This is mainly because the APP has been processed to prevent crawling, such as verifying whether the certificate of https is legal, etc. This solution can be decompiled APP, view Source code solution is more difficult.
• 2. The captured content is garbled
  • In order to prevent crawling, some apps encrypt the returned content with a layer of encryption, so the content seen from Charles is garbled. In this case, you can only decompile the APP and study its encryption and decryption algorithm for decryption.

07. Android interception and packet capture

[External link picture transfer failed, the source site may have an anti-theft link mechanism, it is recommended to save the picture and upload it directly (img-HdaA9gIw-1684069809257)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/c5a3ca45be574551b38740e4adfb6422 ~tplv-k3u1fbpfcp-zoom-1.image)]
[External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img-VxYbemBW-1684069809258)(//p3-juejin.byteimg .com/tos-cn-i-k3u1fbpfcp/e40a220a27c84b63881a9311df83f60e~tplv-k3u1fbpfcp-zoom-1.image)]
[External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture and upload it directly (img- dzTPKpmz-1684069809259)(//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d97aff6debdd479f9970d056b1787cc5~tplv-k3u1fbpfcp-zoom-1.image)]
[External link image transfer failed, the source site may be anti-theft the chain mechanism, it is recommended to save the picture and upload it directly (img-9Oo2PtH6-1684069809259) (//p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/98048847fe9b446db66eda995cfa5f2e~tplv-k3u1fbpfcp-zoom-1.image )]

• Network interception analysis mainly analyzes network traffic loss, as well as request and respond process time. Build web analytics tools...
• Project code address: github.com/yangchong21…
• If you think this intercepting network assistant is convenient for testing and viewing network data during development, you can star it...

Network interception library: github.com/yangchong21…