1 官方文档地址
Ethloop：Ethloop - local simulator for testing Linux QoS disciplines
HTB home：HTB home
2 Ethloop安装
a)官网下载
b) 防丢连接：
https://download.csdn.net/download/duansirui/243659483 执行命令
./ ethloop <testbench.txt> output
<> 为测试脚本文件
output 为任意测试结果输出文件名
如果不产生输出文件：
ethloop <testbench.txt
4 脚本编写
4.1Controling program
第一列为时间，单位毫秒，精确度500
第二列为命令
第三列为flow
第四列为参数
The controling program is line based text file. Each line has fixed format and comment lines behins with #. The format of a line is:
5000 R 2 4500000
Number of spaces is optional and tabs can be used. Input is parsed by sscanf "%u %c %u %n". The first number is time in miliseconds from program start. The second character determines operation to do. Third number is flow id (see below) and remainder of line is operation dependent. However the remainder can be text or number and numbers are allowed in decimal or hex (if starting with 0x). Number can end with k to multiply it by 1024.
The line above means: at time 5sec set rate of flow 2 to 4.5 MBps.
Lines are first read into struct progdata in memory (performance reasons) and should be in increasing time order. Ethloop performs no sorting by time so be carefull about it. Each line is executed at its time and can change ongoing traffic generators.
4.2 Flows
Flow is elementary object in Ethloop. Flow has many properties as sending rate, rate jitter, incoming and outgoing interface .. see struct flowtab for complete list if you are interested.
Each flow is really "flow" of packets with some properties. Ethloop generates the flow and measures some properties of recieved packets which belongs to the flow. Each packet carries information about flowid inside. You can exploit it with u32 QoS filter but there are better ways.
See command reference later in this document to understand all flow related parameters.
flow可以用两种方式匹配队列class。
1 使用P命令进行匹配
例如，flow 5 送入flowid 1:10的队列：
0 P 5 0x10010
2 .使用filter进行匹配
经过wireshark抓包，ethloop发送的数据包非IP协议，因此filter的protocol项填写all，flow的标志在data域的11个字节。
例如，将flow5 送入 flowid 1:10队列的过滤命令
tc filter add dev lo protocol all parent 1:0 pref 1 u32 match u8 0x08 0xff at 10 flowid 1:10
4.3 Command reference
For each command I describe format of its parameter first then function.
X Both flowid and argument are ignored, exit Ethloop and output measured data.
i String. Set interface as incoming and outgoing for the flow.
t String. Override outgoing interface only.
S Number. Byte size of each packet in this flow.
J Number. Jitter of packet size in bytes.
R Number. Transmitter rate in bytes per second.
P Number. Priocode set by setsockopt(sock,SOL_SOCKET,SO_PRIORITY,...). It is very convenient way to test qdisc because majority of classfull qdisc will use classid stored in priority field. To say that this flow should go to class 3:4f use P 0x3004f. This way you need no filters.
G Number. Priority jitter. The priority will be randomly changed in range defined by P code and P+G. It allows you easy testing of qdisc with thousand of active classes by naming them 1:7,1:8,1:9,1:a,1:b....1:fff for example and set P 0x10007, G 0xff8 in one flow.
s String. System command. It allows you to execute arbitrary command at some time. Flowid is irelevant here and should be 0. system (2) call is used. It is nice if you want to change setup of qdisc in the same measurement (5000 s 0 tc class change root classid 1:1 prio 2).
4.4 Example
Let's see simple example. We want to simulate two companies sharing single link with HTB qdisc. We will use interface lo for test (everyone has one). We will add TBF to the lo interface with 100kbps rate.
tc qdisc add dev lo root tbf rate 100kbps burst 10k limit 10k
Now let's create simple simulation program. Both companies will start to transmit packets at 100kbps and second one will change to 50kbps and then to 10kbps later.
# Two flow simulation
# lo will be used to TX and RX
0 i 0 lo
0 i 1 lo
# set simulation packet size to 1k
0 S 0 1k
0 S 1 1k
# start both flows
0 R 0 100k
0 R 1 100k
# flow 1 to 50k and 10k later
2000 R 1 50k
4000 R 1 10k
# finish at 6sec
6000 X 0 0
4.5 注意事项：
- 脚本输入的速度单位为byte，
- 为方便测试，使用lo设备进行验证
- 不管有没有声明flow 0，测试结果都从0开始输出，例如只是用了flow 5，但结果里面包括了0~4的数据。因此匹配flow和队列时，flow 从0开始依次递增。
5 测试结果
测试结果格式：
0.5 53506 42798 49 0 53506 43610 63 0
1.0 67953 62392 84 0 67953 36488 97 0
1.5 70691 68129 84 0 70691 33971 99 0
2.0 71781 68019 87 0 71781 34555 92 0
第一列为时间，单位s
第2列开始，每4列为一个flow的数据
依次为发送速度，接收速度，延迟，任意数据
注意！在显示带宽时，一定是看接收速度！
5.1matlab 绘图简单代码：
clc
clear all
output = importdata('output');
data=output(:,3:4:end);
data(:,size(data,2)+1)=sum(data,2);
plot(data)
legend('0','1','2','3','4','5','6','7','8','9','total')
6 HTB队列使用注意事项
- burst 和cburst取值，应至少为实际带宽的100分之一（intel平台）
- HTB叶子类接的队列可以是pfifo或者sfq，但必须限制等待令牌数量，需添加limit 10 命令，否则无法达到设计性能。该问题在HTB FAQ 中第二个问题解答。
- 当队列带宽到500mbit以上后，HTB无法准确按照比例分配带宽（特别是队列较多时），分析估计已经达到了系统时钟分辨率的上限。 9个队列时，180mbit 测试准确。
解决办法：将队列的limit根据rate的比例分配。这样当入栈速率过快后，由于limit的增大，导致系统内存不够丢包（2中的问题）。此时需要增加系统转发内存
方法如下：
修改 /etc/sysctl.conf
修改默认的转发缓存
net.core.wmem_default = 2048000
net.ipv4.tcp_wmem = 10240 2048000 12582912
（初始是87380）
保持后立刻执行：
/sbin/sysctl -p
注意：wmem是发送缓存，ipv4.tcp只是tcp协议的缓存，因此 net.core.wmem_default应当大于 net.ipv4.tcp_wmem
2048000数值只验证了一个端口 入栈总速度达到3.6Gbit的时候不丢包，多端口要考虑总发送带宽。
- 解决方案3只测试了让HTB在800mbit出口带宽时，保持相近的传输性能。如果需要更准确的带宽分配方案，建议使用hfsc队列。
7 HFSC队列介绍
7.1 官方文档
tc-hfsc(7) - Linux manual page
http://linux-tc-notes.sourceforge.net/tc/doc/sch_hfsc.txt
7.2 使用方法
简单概括HFSC用了服务曲线的方式描述带宽速率，包括rt、ls、sc、ul
Classes at the lowest level of the hierarchy can be assigned a real-time curve (rt) as well as a link-sharing curve (ls), where inner classes can only have a link-sharing curve. By using the ul service curve, an upper limit on service actually rendered to each class can be defined. Instead of specifying two identical rt and ls curves, a single sc curve can be specified. A service curve is described by its transmission rate, which correlates with the slope of the curve. If the curve consists of two parts, it can be specified with dmax the maximum delay at a certain transmission rate umax.
通过两段式的服务曲线，可以保证瞬时的延迟，不过在高速带宽时，两段式的提升不明显，只使用sc rate 和 ul rate 即可表示与HTB相似功能。
7.3 举例
tc qdisc add dev eth0 root handle 1: hfsc
tc class add dev eth0 parent 1: classid 1:1 hfsc sc rate 1000kbit ul rate 1000kbit
tc class add dev eth0 parent 1:1 classid 1:10 hfsc sc rate 500kbit ul rate 1000kbit （工作速度500k，上限1000k）
tc class add dev eth0 parent 1:1 classid 1:20 hfsc sc rate 500kbit ul rate 1000kbit
tc class add dev eth0 parent 1:10 classid 1:11 hfsc sc umax 1500b dmax 53ms rate 400kbit ul rate 1000kbit
tc class add dev eth0 parent 1:10 classid 1:12 hfsc sc umax 1500b dmax 30ms rate 100kbit ul rate 1000kbit
（两段式服务曲线表示，第一段斜率为umax/dmax，维持时间dmax）
7.4 注意事项
- HFSC在1Gbit以上，仍然保持准确的比例分配带宽
- HFSC不具备prio功能，因此所有空闲带宽都是根据sc规定的速率按比例分配
- 叶子队列配上sfq，limit选项仍然会对性能产生影响，如果现在limit为10，带宽波动较大，如采用默认limit（127p），同样会导致内存不足问题，需要修改配置，同6章3）的方法。
- HFSC对带宽限制的精确性没有HTB准确，（仿真测试时，恒定流量会有波动），因此在低速率时，建议仍然使用HTB队列更加准确
- HFSC的burst设置方法，ul 需要用m1 d m2 方式配置代替rate直接配置。m1 允许短时间内超速的上限，d 超速持续时间 m2 平均限速