spark load properties configuration file method

First of all, let me introduce the wrong way of writing. This method of loading the configuration file is wrong:

 

[java] view plain copy

1. val props = new Properties();  
2. val loader = getClass.getClassLoader;  
3. props.load(new FileInputStream(loader.getResource("config.properties").getFile()))  

This is to put the configuration file directly in the resource directory to obtain the configuration file information. This writing method will report that the configuration file cannot be found in the spark program.

Correct spelling:

 

[java] view plain copy

1. val props = new Properties();  
2.   
3. props.load(new FileInputStream("config.properties"));  
4. val hdfspath = props.getProperty("hdfspath");  
5. val mysqlpath = props.getProperty("mysql");  

You can load configuration files in these places

1.

[java] view plain copy

1. kafkaStream.foreachRDD {rdd =>  
2.            rdd.foreachPartition {partition =>  
3.   
4.                val filePath = "config.properties"  
5.                LogUtil.info(filePath)  
6.                val props = new Properties()  
7.                props.load(new FileInputStream(filePath))  
8.   
9.                LogUtil.info("一")  
10.                props.keySet().toArray().foreach { x =>  
11.                    LogUtil.info(x + "\t一" + props.getProperty(x.toString()))  
12.                }  
13.   
14.              

[java] view plain copy

1. 2.    partition.foreach { x =>  
2.                     LogUtil.info(x)  
3.   
4.                     val filePath1 = "config.properties"  
5.                     LogUtil.info(filePath1)  
6.                     val props1 = new Properties()  
7.                     props1.load(new FileInputStream(filePath1))  
8.   
9.                     LogUtil.info("二")  
10.                     props1.keySet().toArray().foreach { x =>  
11.                         LogUtil.info(x + "\t二" + props1.getProperty(x.toString()))  
12.                     }  
13.   
14.                 }  

3.

 

[java] view plain copy

1. def main(args: Array[String]): Unit = {  
2.   
3.        var kafkaZkQuorum = ""  
4.        var group = "EventETL_test_group"  
5.        var topics = ""  
6.        var numThreads = 1  
7.        var timeDuration = 3  
8.   
9.        var checkpointDir = "/Users/test/sparktemp"  
10.   
11.        println("Usage: configuration file")  
12.   
13.        val filePath = "config.properties"  
14.        LogUtil.info(filePath)  
15.        val props = new Properties()  
16.        props.load(new FileInputStream(filePath))