Let me talk about demand.
There is a service program scheduled scanning a designated folder for all files, file contains a plurality of users (clients) and HTTP transmission address information and the corresponding transmission data. Now the service program needs to extract the user information, and then sending bulk; http sent after the need to return to the previous results and the current user; the last batch save. Of course, in order to reduce bulk to save database pressure.
Here's bulk data stored in the use of sqlbulkcopy this not be introduced.
Here to talk about their own ideas and methods and the beginning of the last implementation.
In response to this demand is the first time I really experienced so little experience.
See sent asynchronously, I first thought of using .Net 4.0 Task inside, because the task is very simple to use, more intuitive and the code is based on the thread pool.
Talk about the difference task and thread of what I see:
I performed the same task under comparison Task, Thread, ThreadPool.
Test code:
the Main void static (String [] args)
{
Console.WriteLine ( "the ThreadPool");
for (int I = 0; I <10; I ++)
{
// http with thread pool to execute tasks
ThreadPool.QueueUserWorkItem (GeneralHttpTest, i) ;
}
// wait delay is set to finish the above thread does not affect our test data subsequent viewing
the Thread.Sleep (5000);
Console.WriteLine ( "the thread");
for (int I = 0; I <10; I ++)
{
// http with thread to perform the task
the thread the thread new new TH = (new new ParameterizedThreadStart (GeneralHttpTest));
th.Start (I);
}
the Thread.Sleep (5000);
Console.WriteLine ( "the Task");
for (int I = 0; I <10; I ++)
{
// http task to perform the task with
new new the Task (() =>
{
GeneralHttpTest (I);
}) the Start (). ;
}
the Console.ReadKey ();
}
static void GeneralHttpTest (Object I)
{
// http request and returns the result printed http
int Status = 0;
String respStr = SimpleHttpHelper.HttpGetRequest (URL, Status OUT);
Console.WriteLine (I + "" + respStr);
}
result:
Looking at the results from this task which I feel have a task buffer; it until the buffer is full or the time interval starts to perform all tasks together. We call the task to 10 cycles per cycle of the current value of the loop once it passed. Each time the task cycle does not seem to immediately execute our method, but wait until the cycle was finished together to execute all methods. So, when the value of each cycle is 10.
Of course, where you can use some kind of thread-safe i do to replace this value is passed such ConcurrentBag, ConCurrentDictionay and so on. This time with the task to ensure the consistency of the data transfer.
The above said buffer This is a description of my own; maybe I to understand this is not quite correct, because I do not have a deep study of the task.
At first I was sent with the bulk of the task, carried out with WaitAll task of waiting for the completion of all tasks and then save the data. Such drawbacks finally found is that we have to wait for the current to be transmitted after the next batch of data this group (ie, data within a file) data is complete. This will result in loss of performance, when I have tens of thousands of batches of data to be transmitted so speed is very slow significantly.
Later, for some reason I sent asynchronously useless task, and later direct use of asynchronous execution method BeginGetResponse HttpWebRequest.
Send HttpWebRequest asynchronous I can see "Send the HttpWebRequest asynchronous Http of BeginGetResponse" and there are code I package.
When to save data batch
I started saving idea is to have each of the data added to a list, because you want to get results http return, so when you add is added after the asynchronous return results. Then, every time this data is always on the incidental asynchronous transmission number, the callback after each request, the total number of contrast over time with the current total number of transfer list and compare for equality. Equal, direct save the data. This has the disadvantages if the middle of a thread is lost, so the total number of the total number of list and every time we came with a request to inconsistent; this will result in no data on never saved.
The final program is implemented Queue or queue thread safe ConcurrentQueue .Net inside; each time data are added to the queue, then the timing of fetching data from the queue and then stored. Benefits queues is that adding and removing data independently of each other. Recommended for temporary storage of data in a multi-threaded inside with less queue list ..