firebase bigquery使用

一、开启bigquery

firebase.google.com->settings->ACCOUNT LINKING->开启（需要填写信用卡号、选中“广告标识”，否则统计到的数据里没有用户唯一编号ad_id）

二、bigquery数据控制台

https://bigquery.cloud.google.com/welcome?pli=1

数据表：app_events

三、统计每日的新增和卸载用户数据，并同步到本地，可以使用ad_id与本地数据汇总

以下是本地服务器定时执行的shell脚本

i 创建用户唯一表：

#!/bin/bash
mydate=`date -d -1day +%Y%m%d`


bq query --destination_table=XXX:com_XXXXXX.app_users_info_$mydate "SELECT
  user_dim.device_info.resettable_device_id,
  user_dim.traffic_source.user_acquired_medium,
  user_dim.traffic_source.user_acquired_source,
  user_dim.traffic_source.user_acquired_campaign,
  user_dim.geo_info.country
FROM
  [XXX:com_XXXX.app_events_$mydate]
GROUP BY
  1,
  2,
  3,
  4,
  5"
#remove event


bq query --destination_table=XXX:com_XXX.app_remove_users_$mydate "SELECT
  user_dim.device_info.resettable_device_id, event_dim.timestamp_micros
FROM
[XXX:com_XXX.app_events_$mydate]
WHERE
 event_dim.name='app_remove'
GROUP BY
  1,2"

ii 上传至google云存储

#!/bin/bash
mydate=`date -d -1day +%Y%m%d`
filename="XXX:com_XXX.app_users_info_$mydate"
bq extract --destination_format=NEWLINE_DELIMITED_JSON $filename gs://xxx-data/users_info_$mydate.json
filename1="XXX:com_XXX.app_remove_users_$mydate"
bq extract --destination_format=NEWLINE_DELIMITED_JSON $filename1 gs://xxx-data/remove_users_$mydate.json

iii 下载至本地

#!/bin/bash
mydate=`date -d -1day +%Y%m%d`
/opt/google-cloud-sdk/bin/gsutil cp gs://xxx-data/users_info_$mydate.json /mydata/bigdata-event
/opt/google-cloud-sdk/bin/gsutil cp gs://xxx-data/remove_users_$mydate.json /mydata/bigdata-event

iiii 删除

#!/bin/bash
mydate=`date -d -4day +%Y%m%d`
#echo $mydate
/opt/google-cloud-sdk/bin/gsutil rm gs://xxx-data/users_info_$mydate.json
bq rm -f -t XXX:com_XXX.app_events_$mydate
bq rm -f -t XXX:com_XXX.app_users_info_$mydate
/opt/google-cloud-sdk/bin/gsutil rm gs://xxx-data/remove_users_$mydate.json
bq rm -f -t XXX:com_XXX.app_remove_users_$mydate

四、授权用户（不同账号可以访问不同google账号下的数据，需要授权）

https://console.cloud.google.com/iam-admin/iam/project?project=XXXX

添加其他google账号为项目管理员

五、本地服务器配置

下载并配置google cloud sdk（https://blog.csdn.net/yangyun_1999/article/details/79880549）

https://cloud.google.com/sdk/docs/quickstart-linux

安装bq

五、其他

表字段

https://support.google.com/firebase/answer/7029846?hl=zh-Hans

查询生成临时表cache或永久表

https://cloud.google.com/bigquery/docs/running-queries#bigquery-query-web