BestHttp time-consuming data analysis
Article directory
The BestHttp plug-in has its own module with time-consuming statistics, but each time-consuming information may not be understood very clearly.
request process
When we use BestHttp to make HTTP requests, the typical method is as follows:
HTTHPRequest request = new HTTPRequest(new Uri(url), OnRequestFinished);
request.Send();
When we generate HTTPRequest through new, the time statistics module is initialized:
public sealed class HTTPRequest : IEnumerator, IEnumerator<HTTPRequest> {
public HTTPRequest(Uri uri, HTTPMethods methodType, bool isKeepAlive, bool disableCache, OnRequestFinishedDelegate callback) {
// ...
this.Timing = new TimingCollector();
}
}
TimingCollector is the place used to store the corresponding time in BestHttp. The simplified code is as follows:
public sealed class TimingCollector {
/// <summary>
/// When the TimingCollector instance created.
/// </summary>
public DateTime Start {
get; private set; }
/// <summary>
/// List of added events.
/// </summary>
public List<TimingEvent> Events {
get; private set; }
public TimingCollector() {
this.Start = DateTime.Now;
}
/// <summary>
/// Add an event. Duration is calculated from the previous event or start of the collector.
/// </summary>
public void Add(string name) {
DateTime prevEventAt = this.Start;
if (this.Events.Count > 0)
prevEventAt = this.Events[this.Events.Count - 1].When;
this.Events.Add(new TimingEvent(name, DateTime.Now - prevEventAt));
}
/// <summary>
/// Add an event with a known duration.
/// </summary>
public void Add(string name, TimeSpan duration) {
this.Events.Add(new TimingEvent(name, duration));
}
}
We can see that the current time is recorded as the start time of the network request when the constructor is executed. Subsequent Addoperations are used to add corresponding events and set the time-consuming. After observing the code, we can see that the strings corresponding to these event names are stored in TimingEventNames.csthe file. At the same time, the definitions of these events can also be given in the BestHttp official document. For details, please refer to the Timing API . The code is as follows:
public static class TimingEventNames
{
public const string Queued = "Queued";
public const string Queued_For_Redirection = "Queued for redirection";
public const string DNS_Lookup = "DNS Lookup";
public const string TCP_Connection = "TCP Connection";
public const string Proxy_Negotiation = "Proxy Negotiation";
public const string TLS_Negotiation = "TLS Negotiation";
public const string Request_Sent = "Request Sent";
public const string Waiting_TTFB = "Waiting (TTFB)";
public const string Headers = "Headers";
public const string Loading_From_Cache = "Loading from Cache";
public const string Writing_To_Cache = "Writing to Cache";
public const string Response_Received = "Response Received";
public const string Queued_For_Disptach = "Queued for Dispatch";
public const string Finished = "Finished in";
public const string Callback = "Callback";
}
timing of each event
1. Full Cache Load
When the HTTPRequest object calls the Send method, it first goes back to determine whether the request is stored in the cache. If it exists in the cache, it will add a "Full Cache Load" event, and the time-consuming is the current time minus the request start time.
// HTTPManager.cs
public static HTTPRequest SendRequest(HTTPRequest request) {
// ...
#if !BESTHTTP_DISABLE_CACHING
// If possible load the full response from cache.
if (Caching.HTTPCacheService.IsCachedEntityExpiresInTheFuture(request))
{
DateTime started = DateTime.Now;
PlatformSupport.Threading.ThreadedRunner.RunShortLiving<HTTPRequest>((req) => {
if (Connections.ConnectionHelper.TryLoadAllFromCache("HTTPManager", req, req.Context)) {
req.Timing.Add("Full Cache Load", DateTime.Now - started);
req.State = HTTPRequestStates.Finished;
}
else {
// If for some reason it couldn't load we place back the request to the queue.
request.State = HTTPRequestStates.Queued;
RequestEventHelper.EnqueueRequestEvent(new RequestEventInfo(request, RequestEvents.Resend));
}
}, request);
}
else
#endif
{
request.State = HTTPRequestStates.Queued;
RequestEventHelper.EnqueueRequestEvent(new RequestEventInfo(request, RequestEvents.Resend));
}
return request;
}
2. Queued 和 Queued for redirection
If the corresponding data is not fetched from the cache, the request will be stored in the request queue. At this time, the event of this request will be marked as RequestEvents.Resend. The enqueued request will be called in the OnUpdate event in HTTPManager. Find the connection (HTTPConnection) of the corresponding host (Host) according to the setting event. In addition, let's talk more about the control of connection recovery and the maximum number of connections.
After getting the corresponding connection, use the connection (ConnectionBase, which may actually be FileConnection or HTTPConnection, we only discuss HTTPConnection here) for Processactual processing. The Process will eventually be processed by multi-threading.
At this time, if it is a redirected request or a normal request, the time of the event will be recorded. code show as below:
// HTTPConnection.cs
public sealed class HTTPConnection : ConnectionBase {
protected override void ThreadFunc() {
if (this.CurrentRequest.IsRedirected)
this.CurrentRequest.Timing.Add(TimingEventNames.Queued_For_Redirection);
else
this.CurrentRequest.Timing.Add(TimingEventNames.Queued);
}
}
3. DNS Lookup
Through the client TcpClient in the TCPConnector in the connection (HTTPConnection). The connection is assumed to be established from scratch. Then first obtain the IP address through the domain name, and complete the recording time after obtaining the IP address DNS Lookup:
try {
if (Client.ConnectTimeout > TimeSpan.Zero) {
// https://forum.unity3d.com/threads/best-http-released.200006/page-37#post-3150972
using (System.Threading.ManualResetEvent mre = new System.Threading.ManualResetEvent(false)) {
IAsyncResult result = System.Net.Dns.BeginGetHostAddresses(uri.Host, (res) => {
try {
mre.Set(); } catch {
} }, null);
bool success = mre.WaitOne(Client.ConnectTimeout); // 这里在等待
if (success) {
addresses = System.Net.Dns.EndGetHostAddresses(result);
} else {
throw new TimeoutException("DNS resolve timed out!");
}
}
} else {
addresses = System.Net.Dns.GetHostAddresses(uri.Host);
}
}
finally {
request.Timing.Add(TimingEventNames.DNS_Lookup);
}
4. TCP Connection
After obtaining the address through the domain name, you need to connect to the corresponding server. TcpClient actually uses C# Socket to connect:
// TCPClient.cs
client = new Socket(family, SocketType.Stream, ProtocolType.Tcp);
// ...
var mre = new System.Threading.ManualResetEvent(false);
IAsyncResult result = client.BeginConnect(remoteEP, (res) => mre.Set(), null);
active = mre.WaitOne(ConnectTimeout);
if (active)
client.EndConnect(result);
else
{
try {
client.Disconnect(true); }
catch {
}
throw new TimeoutException("Connection timed out!");
}
Log the event after the connection is complete TCP Connection:
// TCPConnector.cs
try {
Client.Connect(addresses, uri.Port, request);
}
finally {
request.Timing.Add(TimingEventNames.TCP_Connection);
}
5. Proxy Negotiation
After the connection is completed, if there is a Proxy set, Proxy Negotiationthe event will be processed:
if (request.HasProxy) {
try {
request.Proxy.Connect(this.Stream, request);
}
finally {
request.Timing.Add(TimingEventNames.Proxy_Negotiation);
}
}
6. TLS Negotiation
The time of TLS negotiation. If TLS is enabled for TLS negotiation, the event will be recorded after the negotiation is completed:
if (isSecure)
{
DateTime tlsNegotiationStartedAt = DateTime.Now;
// ... 进行对应的TLS协商操作
request.Timing.Add(TimingEventNames.TLS_Negotiation, DateTime.Now - tlsNegotiationStartedAt);
}
7. Request Sent
After all connection operations are completed, HTTPConnection will set a processing function (requestHandler), here is HTTP1Handleran example . When the execution function is called, the data will be written Streamto it, and Request Sentthe event will be recorded after the data is written:
// HTTP1Handler.cs
// Write the request to the stream
this.conn.CurrentRequest.QueuedAt = DateTime.MinValue;
this.conn.CurrentRequest.ProcessingStarted = DateTime.UtcNow;
this.conn.CurrentRequest.SendOutTo(this.conn.connector.Stream);
this.conn.CurrentRequest.Timing.Add(TimingEventNames.Request_Sent);
8. Waiting (TTFB) 和 Headers
After the data is sent, the HTTP message structure needs to be analyzed. The event will be logged when it is
received . The event will be logged when the is received .status lineWaiting (TTFB)HTTP headersHeaders
public virtual bool Receive(int forceReadRawContentLength = -1, bool readPayloadData = true, bool sendUpgradedEvent = true) {
try {
// Read out 'HTTP/1.1' from the "HTTP/1.1 {StatusCode} {Message}"
statusLine = ReadTo(Stream, (byte)' ');
} catch {
// ...
}
if (!this.IsProxyResponse)
baseRequest.Timing.Add(TimingEventNames.Waiting_TTFB);
// ...
//Read Headers
ReadHeaders(Stream);
if (!this.IsProxyResponse)
baseRequest.Timing.Add(TimingEventNames.Headers);
// ...
}
10. Response Received
After receiving the data, record Response Receivedthe event. Since the bottom layer is connected by Socket, it is used to NetworkStreamread and write data (transmitting, synchronizing), and the simplified code is as follows;
// HTTP1Handler.cs
// Receive response from the server
bool received = Receive(this.conn.CurrentRequest);
this.conn.CurrentRequest.Timing.Add(TimingEventNames.Response_Received);
11. Queued for Dispatch and Finished in and Callback
When the network request is completed, HTTPRequestthe state will be changed to HTTPRequestStates.Finished, and RequestEventafter the state is sent and changed, it will be processed and recorded in the next upadte Finished in. If there is a callback for this request, the callback will be invoked and the event will be logged when the callback completes Callback.
// RequestEvent.cs
internal static void HandleRequestStateChange(RequestEventInfo @event) {
HTTPRequest source = @event.SourceRequest;
switch (@event.State) {
case HTTPRequestStates.Queued:
source.QueuedAt = DateTime.UtcNow;
if ((!source.UseStreaming && source.UploadStream == null) || source.EnableTimoutForStreaming)
BestHTTP.Extensions.Timer.Add(new TimerData(TimeSpan.FromSeconds(1), @event.SourceRequest, AbortRequestWhenTimedOut));
break;
case HTTPRequestStates.ConnectionTimedOut:
case HTTPRequestStates.TimedOut:
case HTTPRequestStates.Error:
case HTTPRequestStates.Aborted:
source.Response = null;
goto case HTTPRequestStates.Finished;
case HTTPRequestStates.Finished:
// ... 执行一些处理
source.Timing.Add(TimingEventNames.Queued_For_Disptach);
source.Timing.Add(TimingEventNames.Finished, DateTime.Now - source.Timing.Start); // 注意： 结束事件为开始时间 到 当前时间
if (source.Callback != null) {
try {
source.Callback(source, source.Response);
source.Timing.Add(TimingEventNames.Callback);
}
catch (Exception ex) {
}
}
break;
}
}
Special case
When sending a request, two events will be added, one is to change the status of the request to Queued and the other is the Resend event. code show as below:
request.State = HTTPRequestStates.Queued;
RequestEventHelper.EnqueueRequestEvent(new RequestEventInfo(request, RequestEvents.Resend));
When currently setting the state of a request, a state changed event is added. In the case of a state event change, the state Queuedwill be recorded in the timer. The request will be terminated after the timer finishes.
// RequestEvent.cs
internal static void HandleRequestStateChange(RequestEventInfo @event) {
switch (@event.State) {
// ...
case HTTPRequestStates.Queued:
source.QueuedAt = DateTime.UtcNow;
if ((!source.UseStreaming && source.UploadStream == null) || source.EnableTimoutForStreaming)
BestHTTP.Extensions.Timer.Add(new TimerData(TimeSpan.FromSeconds(1), @event.SourceRequest, AbortRequestWhenTimedOut));
break;
}
}
private static bool AbortRequestWhenTimedOut(DateTime now, object context)
{
HTTPRequest request = context as HTTPRequest;
if (request.State >= HTTPRequestStates.Finished)
return false; // don't repeat
// Protocols will shut down themselves
if (request.Response is IProtocol)
return false;
if (request.IsTimedOut)
{
HTTPManager.Logger.Information("RequestEventHelper", "AbortRequestWhenTimedOut - Request timed out. CurrentUri: " + request.CurrentUri.ToString(), request.Context);
request.Abort();
return false; // don't repeat
}
return true; // repeat
}