Back-end components after component sharing - ETL component package transporter

background

Recently, we are exploring various common components and tools on the front-end, back-end, and system side, and reorganize some of the common components to form a standardized component topic, which will later include some commonly used components in various languages. Welcome everyone to continue to pay attention.

Component Basic Information

Component: transporter
Open source agreement: BSD-3-Clause license

content

In this section, we share an ETL component package transporter , which officially only supports mongodb and postgresql adapters. Support for other databases will be added later. Other adapters may or may not work. You are encouraged to still use v0.5.2 for non-mongo/postgres migrations.

Transporter allows users to configure many data adapters as sources or sinks. These can be databases, files or other resources. Data is read from a source, converted to a message format, and sent down to a sink, where the message is converted to a writable format for its destination. Users can also create data transformations in JavaScript that can sit between sources and sinks and manipulate or filter message flows.

Adapters may be able to track changes that occur in source data. This "tail" function allows the transport aircraft to keep running and keep the receivers in sync.

The latest binary release is available from the Github repository

Use the following format:

transporter init [source adaptor name] [sink adaptor name]

The specific use cases are as follows:

$ transporter init mongodb elasticsearch
$ cat pipeline.js
var source = mongodb({
  "uri": "${MONGODB_URI}"
  // "timeout": "30s",
  // "tail": false,
  // "ssl": false,
  // "cacerts": ["/path/to/cert.pem"],
  // "wc": 1,
  // "fsync": false,
  // "bulk": false,
  // "collection_filters": "{}",
  // "read_preference": "Primary"
})

var sink = elasticsearch({
  "uri": "${ELASTICSEARCH_URI}"
  // "timeout": "10s", // defaults to 30s
  // "aws_access_key": "ABCDEF", // used for signing requests to AWS Elasticsearch service
  // "aws_access_secret": "ABCDEF" // used for signing requests to AWS Elasticsearch service
})

t.Source(source).Save(sink)
// t.Source("source", source).Save("sink", sink)
// t.Source("source", source, "namespace").Save("sink", sink, "namespace")
$

Each adapter has its own README page with details about configuration and functionality

For more usage methods, please refer to the README

This article declares:

88x31.png

Creative Commons License
This work is licensed by CN Hua Shao under the Creative Commons Attribution-Noncommercial 4.0 International License.