Spark user access session analysis

basic data structure

• user_visit_action click stream data (hive table)

  date  //日期：代表用户点击行为是在哪一天发生 采用时间戳好比较
  user_id  //代表这个点击行为是哪一个用户执行的
  session_id //唯一标识了某个用户的一个访问session
  page_id //页面的id，点击品类，进入某个页面
  action_time//这个点击行为发生的时间点
  search_keyword //搜索的关键词
  click_category_id //网站首页点击了某个品类
  click_product_id //网站列表页点击了某个商品
  order_category_ids //代表了将某些商品加入了购物车，然后一次性的对商品下了单，某次下单行为中，有哪些商品品类
  order_product_ids //某次下单行为中，有哪些商品
  pay_category_ids //一次支付行为中对应了哪些品类
  pay_product_ids  //支付行为中，对应了哪些具体的商品

• user_info user information table (hive table)

  user_id //用户的唯一标识
  username //用户的登录名
  name //用户名
  age //年龄
  professinal //职业
  city //城市

• task table (mysql table)

  task_id   //主键
  task_name //任务名称
  create_time //创建时间
  start_time //开始运行的时间
  finish_time //结束运行的时间
  task_type //任务类型
  task_status //任务状态,对应spark作业运行的状态
  task_param //用来使用json的格式来封装用户提交的任务的特殊筛选参数

• Interactive process     

               1. Users of the j2ee platform submit tasks (including task parameters) to the platform, and insert the tasks into the task table in the mysql table

               2. Runtime, Process and other APIs to execute a linux shell script that encapsulates the spark-submit command

               3. Then write the jar and submit it to run in the spark cluster

demand analysis

• Filter sessions by condition

         1. Users who have searched for certain keywords

        2. Users whose access time is within a certain period of time

        3. Users within a certain age range

        4. Users whose occupation is within a certain range

        5. A session initiated by a city

        Function: Conduct business analysis on interested user groups

• The eligible sessions are counted, and the access duration is 1s~3s, 4s~6s, 7s~9s, 10s~30s, 30s~60s, 1m~3m, 3m~10m, 10m~30m  

        , the proportion of sessions in each range above 30m, and the access step (visit time divided by the clicked page) is in the range of 1~3, 4~6, 7~9, 10~30, 30~60, 60 and above The proportion of sessions

         Function: From a global perspective, user groups that meet certain conditions, and some habits of using our products

• In the eligible sessions, randomly select 1000 sessions according to the time ratio

        Function: For eligible sessions, sample according to the time ratio, and observe the click flow behavior of each session in detail.

• In eligible sessions, get the top 10 categories in terms of clicks, orders and payments
• 对于排名前10的品类，分别获取其点击次数排名前10的session

 

技术架构设计

      数据表结构设计（MySQL面向结果）

•       session_aggr_stat 存储session聚合统计的结果

  CREATE TABLE `session_aggr_stat`(
  `task_id` int(11) NOT NULL,
  `session_count` int(11) DEFAULT NULL,
  `1s_3s` double DEFAULT NULL,
  `4s_6s` double DEFAULT NULL,
  `7s_9s` double DEFAULT NULL,
  `10s_30s` double DEFAULT NULL,
  `30s_60s` double DEFAULT NULL,
  `1m_3m` double DEFAULT NULL,
  `3m_10m` double DEFAULT NULL,
  `10m_30m` double DEFAULT NULL,
  `30m` double DEFAULT NULL,
  `1_3` double DEFAULT NULL,
  `4_6` double DEFAULT NULL,
  `7_9` double DEFAULT NULL,
  `10_30` double DEFAULT NULL,
  `30_60` double DEFAULT NULL,
  `60` double DEFAULT NULL,
  PRIMARY KEY (`task_id`)
  )ENGINE=InnoDB

   

• session_random_extract 存储按时间比例随机抽取出来的1000的session

  CREATE TABLE `session_random_extract`(
  `task_id` int(11) NOT NULL,
  `session_id` varchar(255) DEFAULT NULL,
  `start_time` varchar(50) DEFAULT NULL,
  `end_time` varchar(50) DEFAULT NULL,
  `search_keywords` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`task_id`)
  )

   

• top10_category  存储按点击，下单，支付的品类数据

  CREATE TABLE `top10_category`(
  `task_id` int(11) NOT NULL,
  `category_id` int(11) DEFAULT NULL,
  `click_count` int(11) DEFAULT NULL,
  `order_count` int(11) DEFAULT NULL,
  `pay_count` int(11) DEFAULT NULL,
  PRIMARY KEY (`task_id`)
  )

   

• top10_category_session 存储top10品类的点击top10session

  CREATE TABLE `top10_category_session`(
  `task_id` int(11) NOT NULL,
  `category_id` int(11) DEFAULT NULL,
  `click_count` int(11) DEFAULT NULL,
  `session_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`task_id`)
  )

   

• session_detail 存储随机抽取出来的session的明细数据，top10品类的session明细数据

  CREATE TABLE `session_detail`(
  `task_id` int(11) NOT NULL,
  `user_id` int(11) DEFAULT NULL,
  `session_id` varchar(255) DEFAULT NULL,
  `page_id`  int(11) DEFAULT NULL,
  `page_name` varchar(255) DEFAULT NULL,
  `action_name` varchar(255) DEFAULT NULL,
  `search_keywords` varchar(255) DEFAULT NULL,
  `click_category_id` int(11) DEFAULT NULL,
  `click_product_id` int(11) DEFAULT NULL,
  `order_category_ids` varchar(255) DEFAULT NULL,
  `order_product_ids` varchar(255) DEFAULT NULL,
  `pay_category_ids` varchar(255) DEFAULT NULL,
  `pay_product_ids` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`task_id`)
  )