A .Net stress measurement tool comparable to JMeter - Crank Introduction

1 Introduction

Crank is the benchmarking infrastructure used by the .NET team to run benchmarks, including (but not limited to) scenarios from the TechEmpower Web Framework Benchmark, a new project introduced at .NET Conf 2021 and formerly known as Benchmarks .

One of Crank's goals is to provide developers with a tool that allows them to handle performance and measure potential improvements very easily. Some of these features are:

Deploy and benchmark multi-tier applications based on .NET or Docker containers

By specifying the .Net project (local path or git remote repository address), it supports direct deployment or deployment of applications through Docker for benchmarking)

Through Yml configuration, it not only supports the results to be stored in JSON, SQL Server but also to csv files for charts

At present, some small partners have proposed to store support in es

Supports changing the Franework environment of custom applications and testing performance in different environments
Collect diagnostic trace information

2. Core Composition

Crank consists of two parts: Agent and Controller

Controller is the scheduler of tasks, which can schedule load tasks and output results

Agent is the benchmark agent, the actual executor of the task, receives the task from the Controller and executes it.

3. Installation

If you want to do good work, you must first sharpen your tools. Let's first learn how to install crank and how to verify whether the installation is successful.

3.1. Preparations

Install .NET 5.0 .
Open a shell: Install Crank Controller

Installation command:

dotnet tool update Microsoft.Crank.Controller --version "0.2.0-alpha.21567.1" --global

Verify command:

crank

Open shell: Install Crank Agent

Installation command:

dotnet tool update Microsoft.Crank.Agent --version "0.2.0-alpha.21567.1" --global

Verify command:

crank-agent

3.2. Summary

For the convenience of reading, Crank Controller is referred to as Crank in the article, and Crank Agent is referred to as Agent for short.

Agent and Crank need to be installed according to the actual situation, which can be divided into the following situations:

Just to learn Crank, without a separate test environment, you need to install Agent and Controller separately

The Agent provides a separate test environment, so the Agent does not need to be installed locally, only the Controller can be installed

Agent provides a separate test environment, and the pressure test task is triggered and executed by ci, so no configuration needs to be installed locally, and the pressure plan can be completed by building the ci task

Open the shell: check the Agent and Controller versions

dotnet tool list -g

4. Basic knowledge

4.1. variables: parameters

Variables are divided into two types: local parameters and global parameters. The root node is a global parameter, and the other nodes are local parameters.

example:

hello.benchmarks.yml > scenarios > hello-load > serverPort and path under the variables node and serverAddress under the profiles>local>variables node are local parameters

scenarios:
  hello:
    application:
      job: server
    load:
      job: bombardier
      variables:
        serverPort: 5000
        path: /

profiles:
  local:
    variables:
      serverAddress: localhost

bombardier.yml > variables > headers are global parameters

variables:
  headers:
    none: ''
    plaintext: '--header "Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7" --header "Connection: keep-alive"'
    html: '--header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" --header "Connection: keep-alive"'
    json: '--header "Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7" --header "Connection: keep-alive"'
    connectionclose: '--header "Connection: close"'
    ---------------------------------------------------------------------

4.2. profiles: configuration

Profiles are actually configuration file information. Profiles are allowed to be used multiple times, which can be found in the documentation .

Usage: crank [options]

Options:
  -?|-h|--help                       Show help information

  These options are not specific to a Job

  ----------------------------------------------------------------------
  --profile <profile>                Profiles to apply. Can be used multiple times.

Naming convention: *.profiles.yml is recommended

4.3. jobs: tasks

Define what we want to do as a job. Easy to reuse later. Things here refer to a class of things, not to a specific thing.

For example, the built-in definition of Microsoft's bombardier is a job. This job is to benchmark it through the bombardier , and record and output the results, but it doesn't really care which interface is benchmarked against.

Jobs are divided into remote and local according to the application source.

Local source:

jobs:
  server: #任务名称，可根据任务作用自行命名
    source: #任务源
      localFolder: ../hello 
      project: hello.csproj #要构建的 .NET 项目的文件名
    readyStateText: Application started. #控制台中通知服务器它已启动的文本

The local source localFolder can be compared to the relative path where the current crank --config execution command is located. After the task starts, the local project will be sent to the agent and then the task will be executed.

remote source

jobs:
  server:
    source:
      repository: https://github.com/dotnet/crank
      branchOrCommit: main #远程源执行任务的分支
      project: samples/hello/hello.csproj #要构建的 .NET 项目的文件名，格式：相对根的相对路径+项目名.csproj
    readyStateText: Application started.

The remote source will send the warehouse information to the Agent, and the Agent will first download the warehouse and switch to the specified branch before executing the build task to start the project

4.4. scenarios: scenarios

A job cares about a class of things, but not a specific thing. Who cares more about the specific thing? Yes, that is the scenario, that is, scenarios. Scenarios use multiple jobs to complete the benchmark test of the specified scenario. What is done is the arrangement of specific tasks

4.5. imports: imports

Imports provide us with the possibility of yml reuse. Because of the support of imports, we can extract the public yml into a separate yml, and import the used yml through imports, which is similar to the import of js and css. The magic

4.6. Summary

In crank, variables and profiles are not necessary, but because of their existence, we can develop with object-oriented thinking, and we can complete benchmark tests by adding variables or specifying configurations, which will be detailed later in the actual combat. explain

5. Getting Started

After the previous study, we also have a certain understanding of the basic configuration of crank. In the next time, we will first try to learn the Sample that the official has prepared for us. The following tutorial will also explain the role of each configuration in detail. , I hope to understand the basic principles of Crank through the following learning

5.1. Start the Agent

crank-agent --dotnethome "/home/{your-account}/dotnet"

5.1.0.1. Start the Agent and specify the dotnet environment

Format: crank-agent –dotnethome "dotnet installation address"
crank-agent --dotnethome "C:\Program Files\dotnet" (windows)
crank-agent --dotnethome "/usr/share/dotnet" (Linux)

When starting the agent, it is strongly recommended that you increase the dotnethome configuration and specify the environment for the agent to run, so as to avoid getting stuck in the install sdk due to environmental problems when running the task .

Because the local dotnet of the demo machine uses the installation path of /home/gushoudao/dotnet, the commands run in the video are different, and you need to adjust the path according to the actual local situation (for video recording reasons, in After recording, the agent will be stopped. We do not need to exit after starting it in real use. Once the agent is exited, the task cannot be performed.)

5.1.0.2. Start the Agent and specify not to clean up temporary files

crank-agen --no-cleanup (specify not to clean up temporary files)

By default, the temporary files generated during the execution of the current task will be deleted after the execution of the task by the agent is completed.

5.1.0.3. Start the Agent and specify the maximum duration of the build task

crank-agent --build-timeout

The default build task has a maximum duration of 10 minutes

For more configuration click to view

5.2. New hello.benchmarks.yml configuration

The configuration file source code comes from hello.benchmarks.yml

imports:
  - https://raw.githubusercontent.com/doddgu/crank/sample/src/Microsoft.Crank.Jobs.Bombardier/bombardier.yml

jobs:
  server:
    source:
      repository: https://github.com/doddgu/crank
      branchOrCommit: sample
      project: samples/hello/hello.csproj
    readyStateText: Application started.

scenarios:
  hello:
    application:
      job: server
    load:
      job: bombardier
      variables:
        serverPort: 5000
        path: /

profiles:
  local:
    variables:
      serverAddress: localhost
    jobs: 
      application:
        endpoints: 
          - http://localhost:5010
      load:
        endpoints: 
          - http://localhost:5010

5.3. Starting a task

Start the agent (open and set aside):

crank-agent --dotnethome "/usr/share/dotnet"

Start the task (start a new shell):

git clone https://github.com/doddgu/crank.git
cd crank
git checkout sample
crank --config ./samples/hello/hello.original.benchmarks.yml --scenario hello --load.framework net5.0 --application.framework net5.0

Then we wait for a while and the following result will be output

crank-agent：

crank：

| load                  |                |
| --------------------- | -------------- |
| CPU Usage (%)         | 39             |  CPU使用率
| Cores usage (%)       | 631            |  多核CPU使用率
| Working Set (MB)      | 35             |  内存使用率
| Private Memory (MB)   | 35             |  进程使用的私有内存量
| Build Time (ms)       | 4,853          |  构建应用程序需要多长时间（毫秒）
| Start Time (ms)       | 386            |  启动应用程序需要多长时间（毫秒）
| Published Size (KB)   | 66,731         |  已发布应用程序的大小 (KB)
| .NET Core SDK Version | 5.0.403        |  .Net Core SDK 版本
| ASP.NET Core Version  | 5.0.12+0bc3c37 |  .Net Core版本
| .NET Runtime Version  | 5.0.12+7211aa0 |  .Net运行时版本
| First Request (ms)    | 172            |  第一个请求耗时（这里请求是Get）
| Requests              | 2,086,594      |  总发送请求数
| Bad responses         | 0              |  糟糕请求数（响应状态码不是2**也不是3**）
| Mean latency (us)     | 1,833          |  平均延迟时间
| Max latency (us)      | 89,001         |  最大延迟时间
| Requests/sec          | 138,067        |  每秒支持请求数
| Requests/sec (max)    | 255,442        |  每秒最大支持请求数

When you can output the above information, it proves that you have successfully run through the whole process

In the above, we can clearly see the test results under the scene hello, including CPU usage, multi-core CPU usage, memory usage, and the number of requests executed per second, etc.

At this moment, do you suddenly feel that this crank is very powerful? Although it is not clear how to do it, it is really amazing! ! Are you interested in it at this moment and want to know what it can do and why it can output the above results?

6. The end

In order to ensure that the operation according to the document will not be unavailable due to subsequent updates, the source code is forked from the official source, most of which are from the officially provided Sample, and some files will be adjusted to a certain extent in order to better meet personal habits. .

Source address: https://github.com/doddgu/crank/tree/sample

Reference link: