The replay and display of cryptocurrency handicap and transaction data can help quantitative researchers test quantitative strategies, and also help traders resume trading and deepen their insight into the market. DolphinDB can realize high-speed playback of handicap and transaction data, and query the playback results point by point.
DolphinDB supports synchronous playback and publishing of multiple distributed tables to the streaming data table, such as the synchronous playback of the two tables of Handicap and Transaction. The front-end JavaScript uses DolphinDB Web API to poll the streaming data table of the playback output to realize the visual playback of the market and transaction data. DolphinDB comes with a web server, and the entire process can be completed in DolphinDB without external dependencies.
Encrypted currency handicap and transaction data playback can be achieved through the following 4 steps. Users can also use docker to quickly experience the playback function, please refer to the introduction at the end of the article for details.
1. Deploy DolphinDB nodes
Download the latest version of DolphinDB and deploy the cluster. For the deployment tutorial, please refer to the single-server cluster deployment tutorial .
2. Download handicap and transaction data
This article uses the cryptocurrency transaction data provided by Huobi Research Institute, which can be obtained through the Huobi Data API . The sample code for obtaining data can refer to the python sample code or the java sample code .
3. Import data to DolphinDB
This article saves the acquired tick-level data of orderBook as a csv file, and quickly imports the file into the database through the loadTextEx function. Users can also import data into DolphinDB through Python API or Java API . The following code is executed in DolphinDB GUI .
(1) Data preprocessing
If the first line of the saved csv file is irrelevant information, you can use the following script to preprocess the data, and save the processed file to a certain directory. In this case, save the two files to /hdd/data/orderBook-processed and /hdd/data/tick-processes directory. If there is no irrelevant information in the first line of the csv file, you can ignore this step.
//删除数据文件第一行无关信息
def dataPreProcess(DIR){
if(!exists(DIR+ "-processed/"))
mkdir(DIR+ "-processed/")
fileList = exec filename from files(DIR) where isDir = false, filename like "%.csv"
for(filename in fileList){
f = file(DIR + "/" + filename)
y = f.readLines(1000000).removeHead!(1)
saveText(y, DIR+ "-processed/" + filename)
}
}
dataPreProcess("/hdd/data/orderBook")
dataPreProcess("/hdd/data/tick")(2) Create a DolphinDB database
According to the data volume and query fields, the database can be combined and partitioned according to the transaction code and business time. In this case, the name of the database is dfs://huobiDB. If you need to modify, you must also modify the name of the database in replay.html.
def createDB(){
if(existsDatabase("dfs://huobiDB"))
dropDatabase("dfs://huobiDB")
//按照数据集的时间跨度，请自行调整VALUE分区日期范围
db1 = database(, VALUE, 2018.09.01..2018.09.30)
db2 = database(, HASH, [SYMBOL,20])
db = database("dfs://huobiDB", COMPO, [db1,db2])
}
def createTick(){
tick = table(100:0, `aggregate_ID`server_time`price`amount`buy_or_sell`first_trade_ID`last_trade_ID`product , [INT,TIMESTAMP,DOUBLE,DOUBLE,CHAR,INT,INT,SYMBOL])
db = database("dfs://huobiDB")
return db.createPartitionedTable(tick, `tick, `server_time`product)
}
def createOrderBook(){
orderData = table(100:0, `lastUpdateId`server_time`buy_1_price`buy_2_price`buy_3_price`buy_4_price`buy_5_price`buy_6_price`buy_7_price`buy_8_price`buy_9_price`buy_10_price`buy_11_price`buy_12_price`buy_13_price`buy_14_price`buy_15_price`buy_16_price`buy_17_price`buy_18_price`buy_19_price`buy_20_price`sell_1_price`sell_2_price`sell_3_price`sell_4_price`sell_5_price`sell_6_price`sell_7_price`sell_8_price`sell_9_price`sell_10_price`sell_11_price`sell_12_price`sell_13_price`sell_14_price`sell_15_price`sell_16_price`sell_17_price`sell_18_price`sell_19_price`sell_20_price`buy_1_amount`buy_2_amount`buy_3_amount`buy_4_amount`buy_5_amount`buy_6_amount`buy_7_amount`buy_8_amount`buy_9_amount`buy_10_amount`buy_11_amount`buy_12_amount`buy_13_amount`buy_14_amount`buy_15_amount`buy_16_amount`buy_17_amount`buy_18_amount`buy_19_amount`buy_20_amount`sell_1_amount`sell_2_amount`sell_3_amount`sell_4_amount`sell_5_amount`sell_6_amount`sell_7_amount`sell_8_amount`sell_9_amount`sell_10_amount`sell_11_amount`sell_12_amount`sell_13_amount`sell_14_amount`sell_15_amount`sell_16_amount`sell_17_amount`sell_18_amount`sell_19_amount`sell_20_amount`product,[INT,TIMESTAMP,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,DOUBLE,SYMBOL])
db = database("dfs://huobiDB")
return db.createPartitionedTable(orderData, `orderBook, `server_time`product)
}
(3) Import text data into the database
def loadTick(path, filename, mutable tb){
tmp = filename.split("_")
product = tmp[1]
file = path + "/" + filename
t = loadText(file)
t[`product]=product
tb.append!(t)
}
def loopLoadTick(mutable tb, path){
fileList = exec filename from files(path,"%.csv")
for(filename in fileList){
print filename
loadTick(path, filename, tb)
}
}
def loadOrderBook(path, filename, mutable tb){
tmp = filename.split("_")
product = tmp[1]
file = path + "/" + filename
t = loadText(file)
t[`product] = product
tb.append!(t)
}
def loopLoadOrderBook(mutable tb, path){
fileList = exec filename from files(path, "%.csv")
for(filename in fileList){
print filename
loadOrderBook(path, filename, tb)
}
}
login("admin","123456")
tb = createOrderBook()
loopLoadOrderBook(tb, "/hdd/data/orderBook-processed")
tb = createTick()
loopLoadTick(tb, "/hdd/data/tick-processed")(4) Define the data playback function
def replayData(productCode, startTime, length, rate){
login('admin', '123456');
tick = loadTable('dfs://huobiDB', 'tick');
orderbook = loadTable('dfs://huobiDB', 'orderBook');
schTick = select name,typeString as type from tick.schema().colDefs;
schOrderBook = select name,typeString as type from orderbook.schema().colDefs;
share(streamTable(100:0, schOrderBook.name, schOrderBook.type), `outOrder);
share(streamTable(100:0, schTick.name, schTick.type), `outTick);
enableTablePersistence(objByName(`outOrder), true,true, 100000);
enableTablePersistence(objByName(`outTick), true,true, 100000);
clearTablePersistence(objByName(`outOrder));
clearTablePersistence(objByName(`outTick));
share(streamTable(100:0, schOrderBook.name, schOrderBook.type), `outOrder);
share(streamTable(100:0, schTick.name, schTick.type), `outTick);
enableTablePersistence(objByName(`outOrder), true,true, 100000);
enableTablePersistence(objByName(`outTick), true,true, 100000);
endTime = temporalAdd(startTime, length, "m")
sqlTick = sql(sqlCol("*"), tick, [<product=productCode>, <server_time between timestamp(pair(startTime, endTime))>]);
sqlOrder = sql(sqlCol("*"), orderbook, [<product=productCode>, <server_time between timestamp(pair(startTime, endTime))>]);
cutCount = length * 60 / 20
trs = cutPoints(timestamp(startTime..endTime), cutCount);
rds = replayDS(sqlTick, `server_time , , trs);
rds2 = replayDS(sqlOrder, `server_time , , trs);
return submitJob('replay_huobi','replay_huobi', replay, [rds,rds2], [`outTick,`outOrder],`server_time ,, rate);
}
addFunctionView(replayData);4. Data playback
Download the html compressed package of the data playback interface, download address: https://github.com/dolphindb/applications/raw/master/cryptocurr_replay/replay.zip . Unzip replay.zip to the web directory of the DolphinDB package.
Enter http://[host]:[port]/replay.html in the browser address bar to open the data playback interface. Here host and port refer to the IP address and port number of the data node, such as http://192.168.1.135:8902/replay.html .
Before data playback, we can set the following parameters:
- Product: cryptocurrency code
- Replay Rate: The playback speed, that is, the number of records played back per second. If the market generates 100 transactions per second, the Replay Rate is set to 1000 to replay at 10 times the speed.
- Start Time: the start time of the data
- Length: The time span of the data, in minutes. If the Start Time is set to 2018.09.17 00:00:00 and the Length is set to 60, it means that the playback data was generated between 2018.09.17 00:00:00 and 2018.09.17 00:59:59.
After the playback is over, click the square icon button ("End" button) in the upper left corner. Click on a point in the price trend chart, and the table will display 10 data before that point in time. For specific operations, please check the picture https://raw.githubusercontent.com/dolphindb/Tutorials_CN/master/images/replay/v.gif .
Use docker to quickly experience the playback function
We provide a docker container containing DolphinDB Server and demo data, and package it into a tar file for download. Users only need to install the docker environment , download the packaged files, and run the following commands to quickly complete the demo environment deployment.
Download the tar file: https://www.dolphindb.cn/downloads/cryptocurr_replay.tar.gz
gunzip cryptocurr_replay.tar.gz
##docker若没有赋予非管理访问权限，可以使用 sudo docker
docker import cryptocurr_replay.tar ddb/replay:v1
##生成并启动容器
docker run -dt -p 8888:8848 --name replay1 ddb/replay:v1 /bin/bash /dolphindb/start.shAfter starting the container, the access port of DolphinDB database in docker is mapped to port 8888 of the host, open the browser to access http://[宿主机ip]:8888/replay.html, and enter the playback demo interface.
In order to control the size of the docker container and facilitate downloading, the demo data only contains ETHUSDT,ETHBTC,BTCUSDT transaction data with a cryptocurrency number of 2018.09.17 .
Note: Because the built-in license file will expire, you need to download the latest community version license from the official website to replace it. Download the community version and unzip it and enter the server directory, copy the dolphindb.lic file to overwrite the file with the same name in the /dolphindb/ directory in docker.
sudo docker cp ./dolphindb.lic replay1:/dolphindb/dolphindb.lic
Restart docker:
sudo docker restart replay1Precautions
- This case is currently limited to single user use and does not support simultaneous playback of multiple users.
- In order to simplify the operation, the database name and database user information are solidified in the web page, if necessary, please modify the replay.html file by yourself