cassandra之copy迁移数据及错误排查

一、copy迁移数据：

copy导出数据

#!/bin/bash
for kspname in {xn_dolphin_1,xn_im,xn_imstatus,magpie_persistent,eagle_persistent,callserver}
do 
	createKeyspacePath="/root/cassandra/create/$kspname"
	if [ ! -x "$createKeyspacePath" ]; then  
		mkdir -p $createKeyspacePath
	fi 
	cqlsh 192.23.24.13 -ucassandra -pcassandra -e  "desc  $kspname" >$createKeyspacePath/$kspname.cql
	
	dstKeyspacePath="/root/cassandra/$kspname"
	if [ ! -x "$dstKeyspacePath" ]; then  
		mkdir -p $dstKeyspacePath
	fi 
	
	for table in $(cqlsh 192.23.24.13  -ucassandra -pcassandra  -k $kspname  -e  'DESC TABLES')
	do
		cqlsh 192.23.24.13 -ucassandra -pcassandra -k $kspname  -e  "COPY $table TO '$dstKeyspacePath/${table}.CSV'"
	done
done

建库脚本更改操作：
sed -i ‘s/datacenter1/DC1-K8/g’ xn_dolphin_1.cql

copy导入数据

#!/bin/bash
currentPath=$PWD
for kspname in `ls cassandra/`;
do
    for table in `ls cassandra/$kspname/`
        do
            backTableFile=`basename $table`
            currentTableName=`basename $table|awk -F'.' '{print $1}'`
            tablePath="$currentPath/cassandra/$kspname/$backTableFile"
            echo $currentTableName
            cqlsh  192.23.24.113 30222 -ucassandra -pcassandra -k $kspname  -e  "COPY $currentTableName from '$tablePath'"
        done
done

二、错误排查

1、错误：

--Starting copy of xn_dolphin_1.dolphin_conversation_message with columns [siteid, converid, messageid, arrivetime, createat, duration, errortype, expiredtime, extension, fromuser, message, msgtype, size, sourceurl, sys, tousers, type, url].
<stdin>:1:Failed to import 92 rows: ParseError - Failed to parse {''} : Empty values are not allowed,  given up without retries
<stdin>:1:Failed to import 24 rows: ParseError - Failed to parse {''} : Empty values are not allowed,  given up without retries

解决：

在导入的csv文件里存在"{''}" 无法解析
 sed -i "s#{''}##g" dolphin_conversation_message.csv

2、错误：

--Starting copy of xn_dolphin_1.dolphin_visitorinfo_snapshot with columns [siteid, customerid, customerinfo, lasttime].
later, attempt 3 of 5
<stdin>:1:Failed to import 20 rows: InvalidRequest - Error from server: code=2200 [Invalid query] message="Batch too large",  will retry later, attempt 4 of 5
<stdin>:1:Failed to import 20 rows: InvalidRequest - Error from server: code=2200 [Invalid query] message="Batch too large",  given up after 5 attempts

解决：batch_size_fail_threshold_in_kb的默认值只有50,一条DML语句就超过了这个阈值.调整大小。

3、错误

<stdin>:1:Failed to import 5000 rows: Error - field larger than field limit (131072),  given up after 1 attempts
<stdin>:1:Exceeded maximum number of insert errors 1000
<stdin>:1:Failed to process 5336 rows; failed rows written to import_xn_dolphin_1_dolphin_visitorinfo_snapshot.err

解决：

root@k8s-master ~]#vim /etc/cassandra/conf/cqlshrc.sample 
; field_size_limit = 13107200000

注：
网上提供的解决办法：
https://stackoverflow.com/questions/24168235/cassandra-cqlsh-text-field-limit-on-copy-from-csv-field-larger-than-field-limit
尝试操作失败