Basics Lua learning nine <Lua coroutines (Coroutine)>

introduction

Cheng talked about the Association, the first to introduce the difference between threads and coroutines

lua coroutine and multithreading

In common: they have their own independent top of stack, local variables and PC counter, while sharing global variables, and most other things with other coroutine

The difference: a multi-threaded program can run simultaneously several threads (concurrent execution, seize), and the process it takes to run the Association in cooperation with each other, not a true multi-threaded, that is, more than one coroutine program can only be run at the same time a coroutine, and it is performed only when it coroutine explicitly required suspended (suspend), its execution will be suspended (no preemption, uncomplicated).

note:

Coroutine in Lua can not stop it on the outside, it may lead to blocking program
Running is to call coroutine.yield () when the main thread being given LuaException: attempt to yield from outside a coroutine

Correlation function coroutine function

method	description	return value
coroutine.create (f)	Create a principal function f of new coroutine. f must be a Lua function of. Returns this new coroutine, it is a type of "thread" object, and resume when you wake up with the use of function calls	Returns its control (an object for the thread) object
coroutine.resume (co [, val1, ···])	From the co-operation of the pending changes to (1) activating coroutine, that is, let coroutine function starts running;. (2) wake yield, the co-pending then the last place to continue to run. This function can pass parameters	A cyiled the incoming call parameters before running successfully and returns true; otherwise false + error message
coroutine.yield (···)	Suspend execution of the calling coroutine. Passed to `yield`the parameter will be converted to `resume`an additional return value.	Return to resume the argument passed
coroutine.status (co)	Return coroutine co, as a string state: When coroutine is running (it is the status of the call), returns "running"; if the coroutine suspends or call the yield has not started running, return to "suspended"; if coroutine is active, are not running (that is, it is a continuation of another coroutine), return to "normal"; if the coroutine runs out of body function or stop because of an error, return to "dead".	running,suspended,normal,dead
coroutine.wrap (f)	Create a principal function f of new coroutine. f must be a Lua function of. Returns a function, each call to the function will be a continuation of the coroutine. The parameters are passed to this function as a resume of additional parameters. Resume and return the same value, but not the first Boolean. If any errors occur, this error is thrown.	.
coroutine.running()	Returns the currently running coroutine add a Boolean. If you are running coroutine is the main thread, it is true.
coroutine.isyieldable ()	If you are running coroutine can make out, it returns true. Not in the main thread or not can not be let out of a C function, the current co-routines is to get out of.

The following examples will be described simply created with coroutine is first create, look application example to understand the return parameters

print("coroutine start!");
 
--没有yield的协程
local newCor1 = coroutine.create(function()
    return 1,"a"
end)
 
local ret1, num1, str1 = coroutine.resume(newCor1)
 
print("1-----", ret1, num1, str1)
 
--包含一个yield的协程，主要看参数相关
local newCor2 = coroutine.create(function(x)
    x = x+10;
    --str和y的值为resume传入的值
    local str, y = coroutine.yield(x);
    return str, y + x
end)
 
local ret2, x = coroutine.resume(newCor2, 50)
print("2-----", x)
local ret3, str2, y = coroutine.resume(newCor2, "sss", 100);
print("2-----", str2, y)

--输出如下：
coroutine start!
1-----  true    1   a
2-----  60
2-----  sss 160

Then wrap can see the way, except that the specific, the return value is a wrap function, it need not arouse when invoked coroutine resume method, direct call function returned to wrap. There is a return and resume the difference is that the return value in less Boolean values coroutine successful operation.

local newCor3 = coroutine.wrap(function(x)
    x = x - 10;
    local y = coroutine.yield(x);
    return y;
end)
 
--不需要resume函数来唤起，直接调用wrap返回的值
local ret4 = newCor3(100);
print("3-----", ret4)
local ret5 = newCor3(10);
print("3-----", ret5)

3-----  90
3-----  10

Continue taste usage.


co = coroutine.create(
    function(i)
        print(i);
    end
)
 
print(coroutine.status(co)) --suspended
coroutine.resume(co, 1)   -- 1
print(coroutine.status(co))  -- dead
coroutine.resume(co, 1) 
 
print("----------")
print(coroutine.status(co)) --dead
print(coroutine.resume(co)) --false cannot resume dead coroutine
 
co = coroutine.wrap(
    function(i)
        print(i);
    end
)
 
co(1)
 
print("----------")
 
co2 = coroutine.create(
    function()
        for i=1,10 do
            print(i)
            if i == 3 then
                print(coroutine.status(co2))  --running
                print(coroutine.running()) --thread:XXXXXX
            end
            coroutine.yield() --这里被挂起，也就是说for循环只会进行一次
        end
    end
)
 
coroutine.resume(co2) --1
coroutine.resume(co2) --2
coroutine.resume(co2) --3
 
print(coroutine.status(co2))   -- suspended
print(coroutine.running())
 
print("----------")

suspended
1
dead
----------
1
----------
1
2
3
running
thread: 0x7f9c12c07c78  false
suspended
thread: 0x7f9c13001008  true
----------

coroutine.running can be seen, coroutine implementation is a thread at the bottom.

When create a coroutine it is to register an event in the new thread.

When using resume triggering event, create a coroutine function is executed, and when faced with the yield on behalf suspends the current thread, waiting again resume triggering event.

function foo (a)
    print("foo 函数输出", a)
    print("执行次数", 1,os.date())
    return coroutine.yield(2 * a) -- 返回  2*a 的值
end
 
co = coroutine.create(function (a , b)
    print("第一次协同程序执行输出", a, b) -- co-body 1 10
    local r = foo(a + 1)
     
    print("第二次协同程序执行输出", r)
    local r, s = coroutine.yield(a + b, a - b)  -- a，b的值为第一次调用协同程序时传入
     
    print("第三次协同程序执行输出", r, s)
    return b, "结束协同程序"                   -- b的值为第二次调用协同程序时传入
end)
        
print("main", coroutine.resume(co, 1, 10)) -- true, 4
print("--分割线----")
print("main", coroutine.resume(co, "r")) -- true 11 -9
print("---分割线---")
print("main", coroutine.resume(co, "x", "y")) -- true 10 end
print("---分割线---")
print("main", coroutine.resume(co, "x", "y")) -- cannot resume dead coroutine
print("---分割线---")

第一次协同程序执行输出 1   10
foo 函数输出    2
执行次数    1   Tue Nov  5 13:23:41 2019
main    true    4
--分割线----
第二次协同程序执行输出 r
main    true    11  -9
---分割线---
第三次协同程序执行输出 x   y
main    true    10  结束协同程序
---分割线---
main    false   cannot resume dead coroutine
---分割线---

to sum up:

Call resume, will coordinate the program wakes up, returns true resume the operation was successful, otherwise it returns false;
Collaborative program run, run to the yield statement;
coroutine yield suspend, resume the first return; (Note: This return yield, resume the parameter is a parameter)
Second resume, again awakened cooperative programs; (Note: The parameters here resume, in addition to the first argument, the remaining parameters as yield parameters)
yield returns;
Collaborative program continues to run;
If you continue to run collaborative program uses methods continue to call the resume after the completion of the output: can not resume dead coroutine

And resume power of the yield is complex in that, in the main resume process will external state (data) is passed to the internal coordination program; will yield the interior of the state (data) is returned to the main process.

Overall, it looks a bit like a breakpoint and then executed. yield function may be running coroutine suspends, and can be re-awakened at the right time, and then continue to run, which is the essence of coroutines .

Coroutine execution Uniqueness

When a coroutine A resume when another coroutine B, A state does not become suspended, we can not go to resume it; but it is not running the state, because it is currently running a B. A state at this time is actually the normal state.

As already mentioned the return value case, then listed separately below deepen understanding

the yield is not the main function, when calling resume, extra parameters are passed as parameters to the main function, the following example, 123 is the value of each of the abc:

co = coroutine.create(function (a,b,c)
print(a,b,c)
end
)
coroutine.resume(co,1,2,3)

1   2   3

yield has the main function, all the parameters passed to the yield, are returned. Thus resume the return value, in addition to the correct operation flag true , there is passed the parameter values to yield:

co = coroutine.create(function(a,b)
coroutine.yield(a+b,a-b)end)
a,b,c = coroutine.resume(co,10,20)
print(a,b,c)

true    30  -10

the extra parameters will yield to the corresponding return resume, as follows:

co = coroutine.create(function()
    print("co",coroutine.yield())
    end)
    print(coroutine.status(co))
    coroutine.resume(co)
    print(coroutine.status(co))
    print(coroutine.resume(co,4,5))

suspended
suspended
co  4   5
true

But there is a phenomenon, resume first return turned out to be empty? Not to implement, is not it strange? Look at the example:

coroutineFunc = function (a, b) 
    for i = 1, 10 do
        print(i, a, b)
        coroutine.yield()
    end
end

co2 = coroutine.create(coroutineFunc)        --创建协同程序co2
coroutine.resume(co2, 100, 200)                -- 1 100 200 开启协同，传入参数用于初始化
coroutine.resume(co2)                        -- 2 100 200 
coroutine.resume(co2, 500, 600)                -- 3 100 200 继续协同，传入参数无效

co3 = coroutine.create(coroutineFunc)        --创建协同程序co3
coroutine.resume(co3, 300, 400)                -- 1 300 400 开启协同，传入参数用于初始化
coroutine.resume(co3)                        -- 2 300 400 
coroutine.resume(co3)                        -- 3 300 400

The co-parameter passing situation is very flexible, must pay attention to distinguish, at the start coroutine time, resume parameters are passed to the main program; wake yield when the parameter is passed to the yield. Look at the following example:

co = coroutine.create(function (a, b) print("co", a, b, coroutine.yield()) end)
coroutine.resume(co, 1, 2)        --没输出结果，注意两个数字参数是传递给函数的
coroutine.resume(co, 3, 4, 5)        --co 1 2 3 4 5，这里的两个数字参数由resume传递给yield

resume-yield data exchange:

(1) resume the parameters passed to the program (corresponding to the arguments to the call);

　　(2) data is passed to yield the Resume;

　　Parameter (3) resume transmitted to the yield;

　　(4) at the end of the return value of the code synergistic also passed to resume

Nonsense step further, for example to explain the role of the above, let you pass a hundred percent perfect logic to understand,

coroutineFunc = function (a, b) 
    key  = a 
    print ("co",a)
    coroutine.yield(b)       
end

co2 = coroutine.create(coroutineFunc)      
print(coroutine.resume(co2, 100, 200))   
--co    100
--true  200

We all know the value returned resume; part is the result of collaboration itself return, the other part of the implementation of successful return true + value (this value is passed in yield), this is not straightened.

When a coroutine end, all the main function return values are returned to the resume:

co = coroutine.create(function()
return 6,7 end)
print (coroutine.resume(co))

true 6   7

to sum up

We are in the same coroutine, few of these features described above will all spend, but all of these functions are very useful.

So far, we have learned some knowledge of the coroutine in Lua. Now we need to clear a few concepts. Lua provides is asymmetric coroutine, meaning that it takes a function (yield) to suspend a coroutine, but need another function (resume) to awaken the suspended coroutine. Corresponding to the number of languages provided Symmetric coroutine, for switching the function of the current only a coroutine.

Some people want the Lua coroutine called semi-coroutine, but the word has been used as another sense, it is used to represent a restriction of some functions to achieve out of the coroutine, such coroutine, only the call stack in a coroutine in when no residual any pending calls, will be suspended, in other words, only the main be suspended. The generator like Python is such a similar semi-coroutine.

With the difference between asymmetric coroutine and symmetric coroutine different, different (Python in) that the coroutine and generator, generator and it has a powerful coroutine, with a number of interesting features coroutine can be achieved with a generator can not be achieved. Lua provides a fully functional coroutine, if someone likes symmetric coroutine, you can own conduct some simple package.

pipes和filters

The following example is a good resume and yield practical applications.

function receive (prod)
    local status, value = coroutine.resume(prod)
    return value
end

function send (x)
    coroutine.yield(x)
end

function producer()
    return coroutine.create(function ()
        while true do
            local x = io.read() -- produce new value
                print("我进来了吗？1")
            send(x)
               print("我进来了吗？2")
        end
    end)
end
    
function consumer (prod)
    while true do
        local x = receive(prod) -- receive from producer
        io.write(x, "\n") -- consume new value
    end
end

p = producer()
consumer(p)

In short order of execution of this program first calls the consumer, then recv function to resume wake producer, a value produce, send to the consumer, and then continue to wait for the next wake-up time resume.

Also following the above example is split, to help understand the process.

function send(x) 
    coroutine.yield(x)
end

co = coroutine.create(
function()
    while true do
    local x = io.read()
    send(x) end
    end)
print(coroutine.status(co))
print(coroutine.resume(co))

We can continue to expand at above example, add a filter, do some data conversion between the producer and the consumer about her. So what do we do filter in it? We look at the logic did not increase before the filter, is the basic producer to send, send to consumer, consumer to recv, recv from producer, so to understand it. After adding a filter, because the filter needs to do some data conversion operation, so this time the logic, producer to send, send tofilter, filter to recv, recv from producer, filter to send, send to consumer, consumer to recv, recv from filter.


function send(x)
    coroutine.yield(x)
end

function receive (prod)
    local status, value = coroutine.resume(prod)
    print("echo 1")
    print("value is",value)
    return value
end

 
function producer()
    return coroutine.create(function ()
        print("echo 3")
        while true do
            local x = io.read()
            send(x)
        end 
    end)
end
 
function consumer(prod)
    
    while true do
        print("why ?")
        local x = receive(prod)
        if x then
            print("echo 2")
            io.write(x, '\n')
        else
            break
        end 
    end 
end
 
function filter(prod)                                                                                                              
    return coroutine.create(function ()
        for line = 1, math.huge do
            print("echo ")
            local x = receive(prod)
            x = string.format('%5d %s', line, x)
            send(x)
        end 
    end)
end
 
p = producer()
f = filter(p)
consumer(f)

Read the entire string auxiliary printed Sequence, see, Consumer executed, alarm call through the filter coroutine receive resume, while the filter comes in through the producer cried out again after resume receive the coroutine.

In talking about this, do you think the unix pipe? coroutine that is how multithreading kind. Use pipe, each task to be executed in the respective process, but with coroutine, each task is performed in the respective coroutine. pipe provides a buffer between the writer (producer) and the reader (consumer), thus opposing the operating speed is still quite possible. This is a very important characteristic of a pipe, because inter process communication, the cost is still a little big. Use coroutine, cost of switching between different task smaller, mostly a function call, therefore, writer and reader can almost be said to go hand in hand, ah.

Implement iterators with coroutine

We can put the iterator loop as is a special producer-consumer example: iterator produce, loop consume. Here we take a look at the powerful coroutine provides us with a coroutine to implement iterators.

We have to traverse the full array of arrays. Look at the common loop implementation code as follows:

function printResult(a)
    for i = 1, #a do
        io.write(a[i], ' ')
    end 
    io.write('\n')
end
 
function permgen(a, n)                                                                                                             
    n = n or #a
    if n <= 1 then
        printResult(a)
    else
        for i = 1, n do
            a[n], a[i] = a[i], a[n]
            permgen(a, n-1)
            a[n], a[i] = a[i], a[n]
        end 
    end 
end
 
permgen({1,2,3})

Coroutine employed to achieve the following:


function printResult(a)
    for i = 1, #a do
        io.write(a[i], ' ')
    end 
    io.write('\n')
end  
        
function permgen(a, n)
    n = n or #a
    if n <= 1 then
       coroutine.yield(a) 
    else
        for i = 1, n do
            a[n], a[i] = a[i], a[n]
            permgen(a, n-1)
            a[n], a[i] = a[i], a[n]
        end 
    end 
end  
        
function permutations(a)
    local co = coroutine.create(function () permgen(a) end)                                                                        
    return function ()
        local code, res = coroutine.resume(co)
        return res 
    end 
end  
        
for p in permutations({"a", "b", "c"}) do
    printResult(p)
end

b c a 
c b a 
c a b 
a c b 
b a c 
a b c

permutations in Lua function uses a conventional mode, to resume a coroutine encapsulated in the corresponding function. Lua provides a function coroutine.wap this pattern. Like with create, wrap creates a new coroutine, but does not return to the coroutine, but returns a function, this function is called, the corresponding coroutine to be awakened to run. With the original resume difference it is that the function does not return errcode as the first return value, if there is error occurred, quit (C-like language assert). Use wrap, permutations can be implemented as follows:

function permutations (a)
    return coroutine.wrap(function () permgen(a) end)
end

wrap with than create simple, it is really the return of the thing we need most: a coroutine can wake up the corresponding function. But not flexible enough. There is no way to check the status coroutine created the wrap, you can not check runtime-error (no return errcode, but directly assert)

Non-preemptive multithreading

We know, coroutine to run a series of collaborative multithreading. Each coroutine is equivalent to a thread. Of a yield-resume control may switch between different thread. However, with the conventional multithr different, coroutine is non-preemptive. A coroutine at run time, can not be other coroutine suspend it from the outside only by explicitly calling itself yield will hang, and relinquish control. For some programs, this is no problem, on the contrary, because the non-preemptive reason, the program easier. We do not need to worry about synchronization bug problem, because the synchronization between threads are the explicit. We just need to make sure to call the yield at the time on it.

However, the use of non-preemptive multithreading, regardless of which thread calls a blocking operation, the entire program will be blocked, this can not be tolerated. For this reason, many programmers do not think coroutine can replace the traditional multithread, but here we can see an interesting solution.

A typical multithreading scenarios: remote files by downloading more than http. Let's look at how to download a file, which requires the use LuaSocket library,

local socket = require("socket")

require("socket")

host = "www.w3.org"
file = "/standards/xml/schema"

c = assert(socket.connect(host, 80))
c:send("GET " .. file .. " HTTP/1.0\r\n\r\n") -- 注意GET后和HTTP前面的空格

while true do
    local s, status, partial = c:receive(2^10)
    io.write(s or partial)
    if status == "closed" then
        break
    end
end

c:close()

Now we know how to download a file of. Now back to the issue said earlier downloads of multiple remote files. When we receives a remote file, the program spent most of its time waiting for the arrival of data, that is, the call receive function is blocked. Therefore, if we can simultaneously download all the files, the program runs much faster speed. Here we look at how to use coroutine to simulate this implementation. We create a download task for each thread, there is no time data is available in a thread, it calls the program will yield control to a simple dispatcher, the dispatcher to wake up another thread. Let the code before the first written as a function, but there are a few changes, the output file will no longer content to stdout, while only between the output filesize.

function download(host, file)
    local c = assert(socket.connect(host, 80))
    local count = 0  --  counts number of bytes read
    c:send("GET " .. file .. " HTTP/1.0\r\n\r\n")
    while true do
        local s, status, partial = receive(c)
        count = count + #(s or partial)
        if status == "closed" then
            break
        end 
    end 
    c:close()
    print(file, count)
end
function receive (connection)
    return connection:receive(2^10)
end

However, if you want to download multiple files, then this function must be non-blocking receive data. In the absence of data received, call hang yield, relinquish control. Implementation should be as follows:

function receive(connection)   
    connection:settimeout(0)  -- do not block          
    local s, status, partial = connection:receive(2^10)
    if status == "timeout" then
        coroutine.yield(connection)
    end                        
    return s or partial, status
end

settimeout (0) is connected to this non-blocking mode. When the status changes to "timeout", meaning that the operation is not yet complete returns, and in this case, the thread will yield. Non-false parameter passed to the yield, telling the dispatcher thread is still running. Note that even if a timeout, the connection will still return what it has received, there is a partial variable.

The following code shows a simple dispatcher. Table threads hold a series of running thread. Function get sure that each task individually download a thread. dispatcher itself is a cycle, continuous through all thread, one by one to resume. If a download task has been completed, be sure to delete the thread from the thread table. When there is no thread running when the cycle stopped.

Finally, the program creates threads it needs, and calls the dispatcher. For example, four documents downloaded from w3c site, the program is as follows:

local socket = require("socket")

function receive(connection)
    connection:settimeout(0)  -- do not block
    local s, status, partial = connection:receive(2^10)
    if status == "timeout" then
        coroutine.yield(connection)
    end
    return s or partial, status
end

function download(host, file)
    local c = assert(socket.connect(host, 80))
    local count = 0  --  counts number of bytes read
    c:send("GET " .. file .. " HTTP/1.0\r\n\r\n")
    while true do
        local s, status, partial = receive(c)
        count = count + #(s or partial)
        if status == "closed" then
            break
        end
    end
    c:close()
    print(file, count)
end

threads = {}  -- list of all live threads

function get(host, file)
    -- create coroutine
    local co = coroutine.create(function ()
        download(host, file)
    end)
    -- intert it in the list
    table.insert(threads, co)
end

function dispatch()
    local i = 1
    while true do
        if threads[i] == nil then  -- no more threads?
            if threads[1] == nil then -- list is empty?
                break
            end
            i = 1  -- restart the loop
        end
        local status, res = coroutine.resume(threads[i])
        if not res then   -- thread finished its task?
            table.remove(threads, i)
        else
            i = i + 1
        end
    end
end

host = "www.w3.org"
get(host, "/TR/html401/html40.txt")
get(host, "/TR/2002/REC-xhtml1-20020801/xhtml1.pdf")
get(host, "/TR/REC-html32.html")
get(host, "/TR/2000/REC-DOM-Level-2-Core-20001113/DOM2-Core.txt")
dispatch() -- main loop

/TR/html401/html40.txt  629
/TR/REC-html32.html 606
/TR/2000/REC-DOM-Level-2-Core-20001113/DOM2-Core.txt    229699
/TR/2002/REC-xhtml1-20020801/xhtml1.pdf 115777

Running around 10s, 4 files have been downloaded completed

Again by blocking the download retry order a bit, take time becomes longer when a local test, do not know the network problem, blocking a multi-file download code is as follows, in fact, a code in the above paragraphs

local socket = require("socket")

function receive (connection)
    return connection:receive(2^10)
end

function download(host, file)
    local c = assert(socket.connect(host, 80))
    local count = 0  --  counts number of bytes read
    c:send("GET " .. file .. " HTTP/1.0\r\n\r\n")
    while true do
        local s, status, partial = receive(c)
        count = count + #(s or partial)
        if status == "closed" then
            break
        end 
    end 
    c:close()
    print(file, count)
end



host = "www.w3.org"

download(host, "/TR/html401/html40.txt")
download(host, "/TR/2002/REC-xhtml1-20020801/xhtml1.pdf")
download(host, "/TR/REC-html32.html")
download(host, "/TR/2000/REC-DOM-Level-2-Core-20001113/DOM2-Core.txt")

In addition to speed, there is no other room for optimization, the answer is there. When there is no data reception thread, each thread Dispatcher traverses it has no data to look up, blocking this process could be the result of multiple versions of a 30-fold cost cpu.

To avoid this situation, we use the select function LuaSocket provided. It runs a program block while waiting for a set of sockets state changes. Code changes is relatively small, in the cycle, the table is connected to the collector connections timeout when all connections have a timeout, to wait for the call to select Dispatcher connection state changes. The version of the program, the bloggers development environment test, just less than 7s, download the complete four files, in addition to the consumption of a lot of cpu is small, only a little more than clog version only. The new dispatch code is as follows:

function dispatch()
    local i = 1 
    local connections = {}
    while true do
        if threads[i] == nil then  -- no more threads?
            if threads[1] == nil then -- list is empty?
                break
            end 
            i = 1  -- restart the loop
            connections = {}
        end       
        local status, res = coroutine.resume(threads[i])
        if not res then   -- thread finished its task?
            table.remove(threads, i)
        else   
            i = i + 1 
            connections[#connections + 1] = res 
            if #connections == #threads then   -- all threads blocked?
                socket.select(connections)
            end                                                                                                                    
        end       
    end           
end

Coroutine about completed, the above example of its implementation are well documented, for your reference.