Can‘t get JDBC type for struct<date:int,day:int,hours:int,minutes:int,month:int,seconds:int,time:big

An error was reported during the process of using Spark to save DataFrame type data to Oracle:

Exception in thread "main" java.lang.IllegalArgumentException: Can't get JDBC type for struct <data:int, day:int, hours:int, minutes:int, month:int, seconds:int, time:bigint, timeZoneOffset:int, year:int>

The Oracle table data structure is like this:

pd_id-->varchar
ywxtmc-->varchar
sqljb-->varchar
rzlx-->varchar
cjbbh-->varchar
rzsj-->date
rksj-->date

The pojo class I created at the beginning is like this:
Insert picture description here
rzsj and rksj are both string types, which causes the save to fail and an error is reported.

Then the string type was changed to the date type and the title error was reported:

java.lang.IllegalArgumentException: Can't get JDBC type for struct<date:int,day:int,hours:int,minutes:int,month:int,seconds:int,time:bigint,timezoneOffset:int,year:int>

Solution:
Idea: Time field is saved in timestamp format

1. First, still use the string type
Insert picture description here

2. After obtaining the dataFrame, use sparkSQL built-in functions to convert to timestamp format

RDD<String> stringRDD = sparkContext.textFile("C:\\Users\\yanni\\Documents\\WeChat Files\\yanning1994-4\\FileStorage\\File\\2020-07\\catalina.out", 5);
JavaRDD<String> stringJavaRDD = stringRDD.toJavaRDD();

JavaRDD<MGOutput> rowRDD = stringJavaRDD .map(line -> {
    
    
            String parts[] = line.split("&##&");

            MGOutput stu = new MGOutput();
            stu.setPd_id(parts[0]);
            stu.setSqljb(parts[1]);
            stu.setRzsj(parts[2]);
            stu.setYwxtmc("业务系统");
            stu.setRzlx("采集类型");
            stu.setCjbbh("版本号");

            SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
            String format = df.format(new Date());
            stu.setRksj(format);
            return stu;
});

//完整数据的DataFrame
Dataset<Row> dataFrame = sparkSession.createDataFrame(rowRDD, MGOutput.class);
dataFrame.createTempView("rowDataset");
Dataset<Row> result = sparkSession.sql("select pd_id ,to_timestamp(rzsj) as rzsj,to_timestamp(rzsj) as rzsj from rowDataset");

//保存数据
result.write().mode("overwrite").format("jdbc")
                .option("url","jdbc:oracle:thin:@//192.168.1.xxx:11521/xxx")
                .option("user","xxx")
                .option("password","xxx")
                .option("dbtable","xxx")
                .option("batchsize","10").save();

3. Get a new dataFrame to save it successfully.

Guess you like

Origin blog.csdn.net/weixin_44455388/article/details/107462263