demand
- Start a stand-alone version of Druid
- Write some data directly through a client, not through other kafka or batch import
Researched plan
- Start Druid + tranquility
- Druid+Tranquility quick start: http://druid.io/docs/latest/tutorials/tutorial-tranquility.html
- Tranquility warehouse: https://github.com/druid-io/tranquility
- Write data into Druid through tranquility
- Sample code: https://github.com/druid-io/tranquility/blob/master/core/src/test/java/com/metamx/tranquility/example/JavaExample.java
- data schema configuration file
- druid： http://druid.io/docs/0.13.0-incubating/ingestion/ingestion-spec.html
- tranquility： https://github.com/druid-io/tranquility/blob/master/docs/configuration.md
- The configuration of Tranquility encapsulates some configuration items for druid outside of druid
- Partially solve the tranquility dependency problem: https://groups.google.com/forum/#!topic/druid-user/5afMWd17sbU
Druid stand-alone configuration
-
Official website Quickstart
-
Download Druid
-
Download Zookeeper and rename it to zk and put it in the root directory of Druid
curl https://archive.apache.org/dist/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz -o zookeeper-3.4.11.tar.gz tar -xzf zookeeper-3.4.11.tar.gz mv zookeeper-3.4.11 zk -
Start Druid
bin/supervise -c quickstart/tutorial/conf/tutorial-cluster.conf
Tranquility
-
Ctrl+C kill Druid
-
Download Tranquility and do it in the root directory of Druid
curl http://static.druid.io/tranquility/releases/tranquility-distribution-0.8.2.tgz -o tranquility-distribution-0.8.2.tgz
tar -xzf tranquility-distribution-0.8.2.tgz
mv tranquility-distribution-0.8.2 tranquility
- Modify the quickstart/tutorial/conf/tutorial-cluster.conf file and remove the # before the tranquility-server line
# Uncomment to use Tranquility Server
!p95 tranquility-server tranquility/bin/tranquility server -configFile quickstart/tutorial/conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
- Start Druid + Tranquility
bin/supervise -c quickstart/tutorial/conf/tutorial-cluster.conf
Sample maven project
-
Project code: https://github.com/qiaojialin/TranquilityExample
-
Create a new resources folder
-
Create a new file in it
example.json::{ "dataSources": [ { "spec": { "dataSchema": { "dataSource": "wikipedia", "parser": { "type": "string", "parseSpec": { "format": "json", "timestampSpec": { "column": "timestamp", "format": "auto" }, "dimensionsSpec": { "dimensions": [ "page", "language", "user", "unpatrolled", "newPage", "robot", "anonymous", "namespace", "continent", "country", "region", "city" ], "dimensionExclusions": [], "spatialDimensions": [] } } }, "metricsSpec": [ { "type": "count", "name": "count" }, { "type": "doubleSum", "name": "added", "fieldName": "added" }, { "type": "doubleSum", "name": "deleted", "fieldName": "deleted" }, { "type": "doubleSum", "name": "delta", "fieldName": "delta" } ], "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "queryGranularity": "NONE" } }, "tuningConfig": { "type": "realtime", "maxRowsInMemory": 100000, "intermediatePersistPeriod": "PT10m", "windowPeriod": "PT10M" } } } ], "properties": { "zookeeper.connect": "localhost:2181", "druid.selectors.indexing.serviceName": "druid/overlord", "druid.discovery.curator.path": "/druid/discovery", "druidBeam.taskLocator": "overlord", "druidBeam.overlordPollPeriod": "PT5S" } } -
Sample code
import com.google.common.collect.ImmutableMap; import com.metamx.common.logger.Logger; import com.metamx.tranquility.config.DataSourceConfig; import com.metamx.tranquility.config.PropertiesBasedConfig; import com.metamx.tranquility.config.TranquilityConfig; import com.metamx.tranquility.druid.DruidBeams; import com.metamx.tranquility.tranquilizer.MessageDroppedException; import com.metamx.tranquility.tranquilizer.Tranquilizer; import com.twitter.util.FutureEventListener; import org.joda.time.DateTime; import scala.runtime.BoxedUnit; import java.io.InputStream; import java.util.Map; public class JavaExample { private static final Logger log = new Logger(JavaExample.class); public static void main(String[] args) { // Read config from "example.json" on the classpath. final InputStream configStream = JavaExample.class.getClassLoader().getResourceAsStream("example.json"); final TranquilityConfig<PropertiesBasedConfig> config = TranquilityConfig.read(configStream); final DataSourceConfig<PropertiesBasedConfig> wikipediaConfig = config.getDataSource("wikipedia"); final Tranquilizer<Map<String, Object>> sender = DruidBeams.fromConfig(wikipediaConfig) .buildTranquilizer(wikipediaConfig.tranquilizerBuilder()); sender.start(); try { // Send 10000 objects for (int i = 0; i < 10000; i++) { // Build a sample event to send; make sure we use a current date final Map<String, Object> obj = ImmutableMap.<String, Object>of( "timestamp", new DateTime().toString(), "page", "foo", "added", i ); // Asynchronously send event to Druid: sender.send(obj).addEventListener( new FutureEventListener<BoxedUnit>() { @Override public void onSuccess(BoxedUnit value) { log.info("Sent message: %s", obj); } @Override public void onFailure(Throwable e) { if (e instanceof MessageDroppedException) { log.warn(e, "Dropped message: %s", obj); } else { log.error(e, "Failed to send message: %s", obj); } } } ); } } finally { sender.flush(); sender.stop(); } } } -
Add dependency (this step is the worst)
- Ensure that the scala of jackson and tranquility are both 2.11
- jackson-2.9.2 is ok, 2.9.8 is not, other versions have not been tried
<dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-scala_2.11</artifactId> <version>2.9.2</version> </dependency> <dependency> <groupId>com.metamx</groupId> <artifactId>java-util</artifactId> <version>0.27.9</version> </dependency> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.9.4</version> </dependency> <dependency> <groupId>io.druid</groupId> <artifactId>tranquility-core_2.11</artifactId> <version>0.8.2</version> </dependency> -
The above was tested in a clean project. In another project of mine, I don't know which conflict with it. I can't find a way again, so I added three more dependencies.
<dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-scala_2.11</artifactId> <version>2.9.2</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-jaxb-annotations</artifactId> <version>2.9.2</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.2</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-annotations</artifactId> <version>2.9.2</version> </dependency>
Other pits
Dependence is not the worst. I was able to write data yesterday, but I can’t write it today. Various MessageDroppedExceptions are said to be a time window problem on the Internet, and it is useless to adjust the date.
The biggest pitfall is that Tranquility only supports data writing for a period of time, which is very uncomfortable, so I gave up and prepared to switch to Kafka Indexing service.