flume加载问题整理

• failed to start agent because dependencies were not found in classpath
  

  flume-env.sh中需要同时设置JAVA_HOME和HCAT_HOME

• failed to connecting to Endpoint
  
  hive服务器没有启动

       bin/hive --service metastore &
       bin/hive --service hiveserver2 &

• transaction property is not set on table

  
  表transactions未设置，创建表时需要添加

  CREATE TABLE students5 (name VARCHAR(64), age INT, gpa DECIMAL(3, 2)) CLUSTERED BY (age) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES("transactional"="true")；
  

• transactionBatch txnIds=XXX on endPoint=XXX has been closed

  
  分隔符设置问题：前一个双引号，后一个单引号

  agent1.sinks.alarmk2.serializer.delimiter=", "
  agent1.sinks.alarmk2.serializer.serdeSeparator =","

  改成：

  agent1.sinks.alarmk2.serializer.delimiter="\t"
  agent1.sinks.alarmk2.serializer.serdeSeparator ='\t'

• This command is not allowed on an ACID table XXX with a non-ACID
  transaction manager

  
  问题出在:当设置成TBLPROPERTIES(“transactional”=”true”)，hive中需要在配置文件中设置：

  Client Side

  • hive.support.concurrency – true
  • hive.enforce.bucketing – true (Not required as of Hive 2.0)
  • hive.exec.dynamic.partition.mode – nonstrict
  • hive.txn.manager – org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

  Server Side (Metastore)

  • hive.compactor.initiator.on – true (See table below for more details)
  • hive.compactor.worker.threads – a positive number on at least one instance of the Thrift metastore service

  官网中有介绍：https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions

• 查看hive中数据出现”NULL“，行数据均作为一个属性填入数据库，其它属性默认为空
  设置数据属性间用”\t“分割
  

• The channel is full, and cannot write data now

  
  内存满
  将channel的参数capacity、transactionCapacity调大