Kestrel is an open source message middleware written by the twitter development team in the scala language.
Kestrel is a simple, distributed message queue written on the JVM, based on Blaine Cook's "starling".
Each server handles a set of reliable, ordered message queues, with no cross communication, resulting in a cluster of k-ordered ("loosely ordered") queues. Kestrel is fast, small, and reliable.
Kestrel is:
1)fast
It runs on the JVM so it can take advantage of the hard work people have put into java performance.
2)small
Currently about 2500 lines of scala, because it relies on Netty (a rough equivalent of Danger's ziggurat or Ruby's EventMachine) -- and because Scala is extremely expressive.
3)durable
Queues are stored in memory for speed, but logged into a journal on disk so that servers can be shutdown or moved without losing any data.
4)reliable
A client can ask to "tentatively" fetch an item from a queue, and if that client disconnects from kestrel before confirming ownership of the item, the item is handed to another client. In this way, crashing clients don't cause lost messages.
Has the following characteristics:
Fast: Kestrel runs on the JVM, and users can use advanced optimizations as a java project.
Small: Kestrel is about 2500 lines of scala code
Persistence: For good performance, queues are stored in memory. But at the same time the logs are kept on the hard disk, so the server shutdown can guarantee no data loss.
Reliability: The client can try to get the data item in the queue. If the client disconnects the link before confirming the acquisition of the data item, the data item can also be acquired by other clients, that is to say, the crash of the client will not lead to the loss of the data item .
Support multi-request protocol: currently supports three protocols,
memcache protocol (there is no complete implementation of the memcache protocol, only part of the operation protocol is supported),
text protocol,
thrift protocol
Kestrel is not:
strongly ordered
While each queue is strongly ordered on each machine, a cluster will appear "loosely ordered" because clients pick a machine at random for each operation. The end result should be "mostly fair".
transactional
This is not a database. Item ownership is transferred with acknowledgement, but kestrel does not support grouping multiple operations into an atomic unit.