FlinkSQL消费kafka设置WaterMark

消费kafka:

val createTableSql =
    """
      |CREATE TABLE aggs_test(
      | data VARCHAR,
      | account_id VARCHAR,
      | android_id VARCHAR,
      | iccid VARCHAR,
      | imei VARCHAR,
      | imsi VARCHAR,
      | operator VARCHAR,
      | oa_id VARCHAR,
      | pre_imei VARCHAR,
      | serial VARCHAR,
      | app_version VARCHAR,
      | channel VARCHAR,
      | install_server_time VARCHAR,
      | uid VARCHAR,
      | trusted_id VARCHAR,
      | uid VARCHAR,
      | utd_id VARCHAR,
      | source_versioncode VARCHAR,
      | wmac VARCHAR,
      | idfa VARCHAR,
      | key_chain_idfa VARCHAR,
      | idfv VARCHAR,
      | ip VARCHAR,
      | message_type VARCHAR,
      | ts AS TO_TIMESTAMP(FROM_UNIXTIME(server_timestamp, 'yyyy-MM-dd HH:mm:ss')),
      | WATERMARK FOR ts AS ts - INTERVAL '1' SECOND  --1s的watermark
      |
      |)
      |WITH(
      | 'connector.type' ='kafka',
      | 'connector.version' = 'universal',
      | 'connector.topic' = 'xxx',
      | 'connector.startup-mode' = 'earliest-offset',
      | 'connector.properties.0.key' = 'bootstrap.servers',
      | 'connector.properties.0.value' = 'xxx',
      | 'update-mode' = 'append',
      |  'format.type' = 'json',
      |  'format.derive-schema' = 'true'
      |)
    """.stripMargin```

SQL:

```sql
SELECT
ad_id,
data_date,
ad_location,
ad_platform,
TUMBLE_START(ts,INTERVAL '5' second) as window_start,
TUMBLE_END(ts,INTERVAL '5' second) as window_end,
COUNT(1) AS click_count
FROM  aggs_temp_table
where event_type='adclick'
GROUP BY
ad_id,
data_date,
ad_location,
ad_platform,
TUMBLE(ts ,INTERVAL '5' second)  -- 5s一个窗口

猜你喜欢

转载自blog.csdn.net/xiaozhaoshigedasb/article/details/107249647