Back-end components after component sharing - high-performance and flexible stream processor benthos based on Golang

background

Recently, we are exploring various common components and tools on the front-end, back-end, and system side, and reorganize some of the common components to form a standardized component topic, which will later include some commonly used components in various languages. Welcome everyone to continue to pay attention.

Component Basic Information

Components: benthos
Open source agreement: MIT license
Official website: www.benthos.dev

content

In this section, we share the high-performance and elastic stream processor benthos based on Golang , which can connect various sources and sinks in various proxy modes , and perform hydration, enrichment, transformation and filtering on the payload .

It comes with a powerful mapping language , is easy to deploy and monitor, and can be dropped into your pipeline as a static binary, docker image, or serverless function , making it cloud-native.

image.png

Benthos is fully declarative, stream pipelines are defined in a single configuration file, allowing you to specify a list of connectors and processing stages:

input:
  gcp_pubsub:
    project: foo
    subscription: bar

pipeline:
  processors:
    - bloblang: |
        root.message = this
        root.meta.link_count = this.links.length()
        root.user.age = this.user.age.number()

output:
  redis_streams:
    url: tcp://TODO:6379
    stream: baz
    max_in_flight: 20

Supported Sources and Sinks

Apache Pulsar, AWS (DynamoDB, Kinesis, S3, SQS, SNS), Azure (Blob storage, Queue storage, Table storage), Cassandra, Elasticsearch, File, GCP (Pub/Sub, Cloud storage), HDFS, HTTP (server and client, including websockets), Kafka, Memcached, MQTT, Nanomsg, NATS, NATS JetStream, NATS Streaming, NSQ, AMQP 0.91 (RabbitMQ), AMQP 1, Redis (streams, list, pubsub, hashes), MongoDB, SQL (MySQL, PostgreSQL, Clickhouse, MSSQL), Stdin/Stdout, TCP & UDP, sockets and ZMQ4.

The specific usage is as follows:

1. docker installation

# Using a config file
docker run --rm -v /path/to/your/config.yaml:/benthos.yaml jeffail/benthos

# Using a series of -s flags
docker run --rm -p 4195:4195 jeffail/benthos \
  -s "input.type=http_server" \
  -s "output.type=kafka" \
  -s "output.kafka.addresses=kafka-server:9092" \
  -s "output.kafka.topic=benthos_topic"

For specific usage, please refer to this document
. For guidance on how to configure more advanced stream processing concepts (such as stream connections, augmented workflows, etc.), please view the manual section.

For guidance on building your own custom plugins in Go, check out the public API.

This article declares:

88x31.png

Creative Commons License
This work is licensed by CN Hua Shao under the Creative Commons Attribution-Noncommercial 4.0 International License.