publish and subscribe model is still a lot of usage scenarios, I remember the interview when asked if you want to have you do a version of the automatic upgrade, how would you do? At that time we are trying to introduce etcd in the project, and then consider the next idea that you can also use publish and subscribe to solve this problem.
publish the code generated gitlab merge message to the master, the configuration can be considered to achieve webhook, jenkins course with no problem, after the build is complete, the final construct results jenkins curl transmission.
subscribe each node is a need to install a server agent program, the agent program watch their own projects concern the update, once an update is found, pull the latest version of the program to be updated.
publish by long links can also be achieved by http interfaces, no problem. If you subscribe to ensure real-time, need to be achieved through long link, if less demanding real-time, set the time interval in rotation call http interface is also no problem.
Entire functions divided into three blocks:
- broker: a pub and sub transit of
- publisher: Post Owner
- subscriber: Messaging Subscriber
Draw a simple sketch:
Roughly on such a thought. Then consider the next, as if the whole model code to write is not very complicated, and would like to write hands up.
The first is the broker, broker to serve the publisher and subscriber, publisher standard htt interfaces can be used, subscriber can use websocket or own handwriting a long connection, you can also use some of the long connection keep alive the library open source, like smux I often used in a tcp library.
websocket usually do not deal with codec problems, but also need to add heartbeat keep-alive, smux need to deal heartbeat keep alive (the library itself provides a heartbeat mechanism), but the codec to their own separate treatment.
publisher can not achieve, providing an interface specification on the line.
subscriber clients achieve a long connection, read the message on the message content can be printed (not consider an updated version of the process design)
First start designing broker and subscribe communication protocol proto.go
package proto
const (
CMD_S2B_HEARTBEAT = iota
CMD_B2S_HEARTBEAT
CMD_B2S_MSG
)
type B2SBody S2BBody
type S2BBody struct {
Cmd int `json:"cmd"`
Data interface{} `json:"data"`
}
type S2BSubscribe struct {
Topics []string `json:"topics"`
}
type S2BHeartbeat struct{}
type B2SHeartbeat struct{}
复制代码ok, then you can start writing code for a broker, broker and subscriber websocket be finally adopted, broker and publisher uses http protocol.
import (
"encoding/json"
"io/ioutil"
"log"
"net/http"
"sync"
"github.com/ICKelin/pubsub/proto"
"github.com/gorilla/websocket"
)
type Broker struct {
addr string
muEntry sync.RWMutex
entry map[string][]*subscriber
done chan struct{}
}
func NewBroker(pubaddr, subaddr string) *Broker {
return &Broker{
addr: pubaddr,
entry: make(map[string][]*subscriber),
done: make(chan struct{}),
}
}
func (b *Broker) Run() {
http.HandleFunc("/pub", b.onPublish)
http.HandleFunc("/sub", b.onSubscriber)
http.ListenAndServe(b.addr, nil)
}
type pubBody struct {
Topic string `json:"topic"`
Msg interface{} `json:"msg"`
}
func (b *Broker) onPublish(w http.ResponseWriter, r *http.Request) {
bytes, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Println(err)
return
}
body := &pubBody{}
err = json.Unmarshal(bytes, &body)
if err != nil {
log.Println(err)
return
}
topic := body.Topic
msg := body.Msg
log.Println("publish topic ", topic, msg)
b.muEntry.RLock()
subscribers := b.entry[topic]
b.muEntry.RUnlock()
if subscribers == nil {
// drop message once no subscriber
// TODO: store msg
return
}
for _, s := range subscribers {
s.push(msg)
}
}
type subscribeMsg struct {
Topics []string `json:"topics"`
}
func (b *Broker) onSubscriber(w http.ResponseWriter, r *http.Request) {
upgrade, err := websocket.Upgrade(w, r, nil, 1024, 1024)
if err != nil {
log.Println(err)
return
}
subMsg := proto.S2BSubscribe{}
upgrade.ReadJSON(&subMsg)
s := newSubscriber(subMsg.Topics, upgrade.RemoteAddr().String())
b.muEntry.Lock()
for _, topic := range s.topics {
b.entry[topic] = append(b.entry[topic], s)
}
b.muEntry.Unlock()
s.serveSubscriber(upgrade)
upgrade.Close()
b.muEntry.Lock()
for _, topic := range s.topics {
for i, s := range b.entry[topic] {
if s.raddr == upgrade.RemoteAddr().String() {
log.Println("remove subscriber: ", s.raddr, " from topic ", topic)
if i == len(b.entry[topic])-1 {
b.entry[topic] = b.entry[topic][:i]
} else {
b.entry[topic] = append(b.entry[topic][:i], b.entry[topic][i+1:]...)
}
break
}
}
}
b.muEntry.Unlock()
}
复制代码broker handling of subscriber
import (
"log"
"time"
"github.com/ICKelin/pubsub/proto"
"github.com/gorilla/websocket"
)
type subscriber struct {
topics []string
raddr string
done chan struct{}
writebuf chan *proto.B2SBody
// parent string
// children []*subscriber
}
func newSubscriber(topics []string, raddr string) *subscriber {
return &subscriber{
topics: topics,
raddr: raddr,
done: make(chan struct{}),
writebuf: make(chan *proto.B2SBody),
}
}
func (s *subscriber) serveSubscriber(conn *websocket.Conn) {
go s.reader(conn)
s.writer(conn)
}
func (s *subscriber) reader(conn *websocket.Conn) {
defer close(s.done)
for {
var obj proto.S2BBody
err := conn.ReadJSON(&obj)
if err != nil {
log.Println(err)
break
}
switch obj.Cmd {
case proto.CMD_S2B_HEARTBEAT:
conn.SetWriteDeadline(time.Now().Add(time.Second * 5))
conn.WriteJSON(&proto.S2BBody{Cmd: proto.CMD_B2S_HEARTBEAT})
conn.SetWriteDeadline(time.Time{})
}
}
}
func (s *subscriber) writer(conn *websocket.Conn) {
for {
select {
case buf := <-s.writebuf:
conn.SetWriteDeadline(time.Now().Add(time.Second * 5))
conn.WriteJSON(buf)
conn.SetWriteDeadline(time.Time{})
// drop msg once writejson fail
case <-s.done:
return
}
}
}
func (s *subscriber) push(data interface{}) {
s.writebuf <- &proto.B2SBody{Cmd: proto.CMD_B2S_MSG, Data: data}
}
复制代码This two pieces of code there are several very obvious problem
- If there is no subscriber, then publish this message will be discarded
- If the message sent from the broker to the subscriber timeout, the message will be discarded as
- Send success, does not mean the end of successfully received, and how to ensure that the message is really successful release
In order to address three questions above, you may want to consider data persistence and message ack mechanisms, this is a major problem.
Look at the realization of subscriber, subscriber can be implemented simply send a heartbeat packet goroutine, goroutine receiving a packet, the corresponding callback function is called after processing the received message.
package main
import (
"log"
"time"
"github.com/ICKelin/pubsub/proto"
"github.com/gorilla/websocket"
)
type subscriber struct {
topics []string
broker string
cb func(msg interface{})
}
func newSubscriber(topics []string, broker string, cb func(msg interface{})) *subscriber {
return &subscriber{
topics: topics,
broker: broker,
cb: cb,
}
}
func (s *subscriber) Run() error {
conn, _, err := websocket.DefaultDialer.Dial(s.broker, nil)
if err != nil {
return err
}
defer conn.Close()
subMsg := proto.S2BSubscribe{Topics: s.topics}
err = conn.WriteJSON(&subMsg)
if err != nil {
return err
}
go func() {
for {
body := &proto.S2BBody{
Cmd: proto.CMD_S2B_HEARTBEAT,
Data: &proto.S2BHeartbeat{},
}
conn.WriteJSON(body)
time.Sleep(time.Second * 3)
}
}()
for {
var body = proto.S2BBody{}
conn.ReadJSON(&body)
switch body.Cmd {
case proto.CMD_B2S_HEARTBEAT:
log.Println("hb from broker")
case proto.CMD_B2S_MSG:
s.cb(body.Data)
}
}
}
func main() {
s := newSubscriber([]string{"gtun", "https://www.notr.tech"}, "ws://127.0.0.1:10002/sub", func(msg interface{}) {
log.Println(msg)
})
s.Run()
}
复制代码