linux客户端中文乱码

hive中文问题解决办法：

关键是终端console-->linux OS环境变量-->hdfs上的hive table对应文件三个层次，都设为使用UTF-8。

1. 到sourceforge.net上，下载putty英文版0.63（不要下中文版，曾有带木马），在如下位置设为UTF-8：

2. 将linux OS层/etc/sysconfig/i18n中LANG修改为en_US.UTF-8

[hadoop@CNSZ141196 ~]$ cat /etc/sysconfig/i18n
LANG="en_US.UTF-8"
SYSFONT="lat0-sun16"

3. hive table文件默认为UTF-8字符集编码，无需改动

4. 测试如下，可见在putty终端console上可正常输入中文，写入hive表；select出来也可正常显示；load into、 hive -f 方式也测试通过：
[hadoop@CNSZ141196 ~]$ hive -e "insert into table test01.test01 select '中文测试' from dual;"
……
[hadoop@CNSZ141196 ~]$ hive -e "select * from test01.test01;"
Logging initialized using configuration in file:/appcom/hive-config/hive-log4j.properties
Hive history file=/tmp/hadoop/[email protected]_201309111524_494470832.txt
OK
中文测试
中国
中文
Time taken: 3.199 seconds, Fetched: 3 row(s)
[hadoop@CNSZ141196 ~]$

OS语言环境变量配置
for i in cnsz0317{74..87};do /home/hadoop/scripts/autoLoginRoot.sh hadoop $i paic1234 anyword ""'sed -i "/LANG=/{s%"en_US"%"en_US.UTF-8"%g}" /etc/sysconfig/i18n'"";done
检查点：
for i in cnsz0317{74..87};do ssh $i ""'echo "#############";hostname;cat /etc/sysconfig/i18n;sleep 0.5'""; done

问题：
hive comment无法支持中文
处理：
修改mysql中hive_meta数据库使hive comment支持中文：

alter table DBS modify column `DESC` varchar(4000) character set utf8;
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;

查看修改结果：
mysql> SHOW CREATE TABLE TABLE_PARAMS \G
*************************** 1. row ***************************
Table: TABLE_PARAMS
Create Table: CREATE TABLE `TABLE_PARAMS` (
`TBL_ID` bigint(20) NOT NULL,
`PARAM_KEY` varchar(256) CHARACTER SET latin2 COLLATE latin2_bin NOT NULL,
`PARAM_VALUE` varchar(4000) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`TBL_ID`,`PARAM_KEY`),
KEY `TABLE_PARAMS_N49` (`TBL_ID`),
CONSTRAINT `TABLE_PARAMS_FK1` FOREIGN KEY (`TBL_ID`) REFERENCES `TBLS` (`TBL_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin2
1 row in set (0.00 sec)
修改hive-site.xml
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://CNSZ141196.app.paic.com.cn:3306/hive_meta?useUnicode=true&characterEncoding=UTF-8&createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
<name>hive.stats.dbconnectionstring</name>
<value>jdbc:mysql://CNSZ141196.app.paic.com.cn:3306/hive_stats?useUnicode=true&characterEncoding=UTF-8&user=hive&password=hv123.&createDatabaseIfNotExist=true</value>
<description>The default connection string for the database that stores temporary hive statistics.</description>
</property>

linux客户端中文乱码

猜你喜欢