在实时流数据处理方面，supermap iobjects for spark产品的streaming模块已实现了对socket，http，kafka，jms等
来源的数据处理，但依然无法满足每一种业务的需求，本篇介绍如何对接udp服务来源的数据流，以实时接收北大导航返回
的gps点数据为示例进行处理及输出。
1. 数据介绍
每个gps点坐标是以二进制数据包返回，数据体包含了经纬度，速度，方向，高程等信息。gps点如下：
aaaacccc0002000000353138383535343036383432000000000000000000a7e7dd58502c5d40575a46ea3d5d40404a0073000000ef97e207021c12010e0700
2. 解析数据
根据数据包的结构进行解析，得到字符串格式的点坐标信息，其主要代码如下：
public String doParse()
{
if (buffer == null) {
return "";
}
byte[] code = new byte[20];
buffer.position(10);
buffer.get(code);
String codeStr = new String(code).trim();
byte[] lou = new byte[8];
buffer.get(lou);
double longitude = bytes2Double(lou);
byte[] lat = new byte[8];
buffer.get(lat);
double latitude = bytes2Double(lat);
byte[] speed = new byte[2];
buffer.get(speed);
double catSpeed = byteToShort(speed);
byte[] direction = new byte[2];
buffer.get(direction);
short carDirec = byteToShort(direction);
byte[] altitude = new byte[2];
buffer.get(altitude);
short carAltitude = byteToShort(altitude);
byte[] precision = new byte[2];
buffer.get(precision);
short dataPrecision = byteToShort(precision);
byte[] year = new byte[2];
buffer.get(year);
Calendar calendar = Calendar.getInstance();
calendar.set(byteToShort(year), buffer.get(), buffer.get(), buffer.get(), buffer.get(), buffer.get());
Date date = calendar.getTime();
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String dateStr = df.format(date);
return codeStr + "," + dateStr + "," + longitude + "," + latitude + "," + carAltitude + "," + carDirec + "," + catSpeed + "," + dataPrecision;
}
3. 数据发送端
数据发送端采用udp协议进行数据的实时发送，udp服务无需连接，其主要代码，如下:
InetAddress address = InetAddress.getByName("localhost");
String ip = address.getHostAddress();
System.out.println(ip);
DatagramSocket ds = new DatagramSocket();
while (true)
{
DatagramPacket dp = new DatagramPacket(str.getBytes(), 0, str.length(), address, 5467);
ds.send(dp);
Thread.sleep(1000);
System.out.println("发送完成");
}
4. 数据接收器
基于超图streaming模块里的ReceiverBase接口实现UDP服务数据流的接收器，代码清单如下:
class CustomUDPReceiver(val port: Int) extends ReceiverBase with Serializable {
override def receive(ssc: StreamingContext): DStream[String] = {
ssc.receiverStream(new SocketUDPStreamingReceiver(port))
}
}
class SocketUDPStreamingReceiver(port: Int) extends Receiver[String](StorageLevel.MEMORY_ONLY) {
def onStart(): Unit = {
val start = new Runnable() {
def run(): Unit = {
val executorService = Executors.newCachedThreadPool()
val catchAndSend = new Runnable() {
def run(): Unit = {
val ds = new DatagramSocket(port)
val length = 10000
val buf = new Array[Byte](length)
val dp = new DatagramPacket(buf, length)
ds.receive(dp)
val message = new String(buf, 0, dp.getLength);
message.split("\n").foreach(line => store(line))
ds.close()
restart("")
}
}
executorService.submit(catchAndSend)
}
}
new Thread(start).start()
}
def onStop(): Unit =
{
}
}
5. 构建流模型文件
构建流模型文件的接收器节点，数据处理节点，过滤节点，输出节点等信息，以下为接收器节点代码：
"stream": {
"nodeDic": {
"CustomUDPReceiver": {
"port": 5467,
"reader": {
"separator": ",",
"className": "com.supermap.bdt.streaming.formatter.CSVFormatter"
},
"metadata": {
"title": "",
"epsg": 4326,
"fieldInfos": [
{
"name": "id",
"source": "0",
"nType": "TEXT"
},
{
"name": "datatime",
"source": "1",
"nType": "DATETIME"
},
{
"name": "X",
"source": "2",
"nType": "DOUBLE"
},
{
"name": "Y",
"source": "3",
"nType": "DOUBLE"
},
{
"name": "carAltitude",
"source": "4",
"nType": "SINGLE"
},
{
"name": "carDirection",
"source": "5",
"nType": "SINGLE"
},
{
"name": "carSpeed",
"source": "6",
"nType": "SINGLE"
},
{
"name": "carPrecision",
"source": "7",
"nType": "SINGLE"
}
],
"featureType": "POINT",
"idFieldName": "id",
"dateTimeFormat": "yyyy-MM-dd HH:mm:ss"
},
"name": "CustomUDPReceiver",
"caption": "",
"description": "",
"prevNodes": [],
"nextNodes": [
"esAppendSender"
],
"className": "CustomUDPReceiver"
}
6. 启动
启动实时流程序前，先启动udp服务的发送端，其主要代码如下：
val startupRun = Startup.fromJson(json)
StreamingRunner.run(startupRun.asInstanceOf[StartupDefault])
7. 结果如图
完整代码地址