Parameters can be passed using the constructor or the withParameters(Configuration) method, the parameters will be serialized as part of the function object and passed to the task instance! Official website address batch
1 Use the constructor method
DataSet<Integer> toFilter = env.fromElements(1, 2, 3);
toFilter.filter(new MyFilter(2));
private static class MyFilter implements FilterFunction<Integer> {
private final int limit;
public MyFilter(int limit) {
this.limit = limit;
}
@Override
public boolean filter(Integer value) throws Exception {
return value > limit;
}
}
2 withParameters(Configuration) method
This method will take a Configuration object as a parameter, which will be passed to the Rich Function's open method (for the Rich Function, see: rich function ). The Configuration object is a Map that stores Key/Value key-value pairs.
DataSet<Integer> toFilter = env.fromElements(1, 2, 3);
Configuration config = new Configuration();
config.setInteger("limit", 2);
toFilter.filter(new RichFilterFunction<Integer>() {
private int limit;
@Override
public void open(Configuration parameters) throws Exception {
limit = parameters.getInteger("limit", 0);
}
@Override
public boolean filter(Integer value) throws Exception {
return value > limit;
}
}).withParameters(config);
3 Use the global the ExecutionConfig method
Parameters can be obtained by all rich functions
Configuration conf = new Configuration();
conf.setString("mykey","myvalue");
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
env.getConfig().setGlobalJobParameters(conf);
public static final class Tokenizer extends RichFlatMapFunction<String, Tuple2<String, Integer>> {
private String mykey;
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
ExecutionConfig.GlobalJobParameters globalParams = getRuntimeContext().getExecutionConfig().getGlobalJobParameters();
Configuration globConf = (Configuration) globalParams;
mykey = globConf.getString("mykey", null);
}
// ... more here ...