MapReduce development environment construction

MapReduce runs on the Linux system, but for development, the system platform may be the Windows platform, because the program needs to run on the Hadoop cluster on Linux during the view test. The first way is: to establish a connection between windows and linux, the development platform eclipse is on windows, and the running interface and file storage interface of hadoop are called. (hadoop is not configured on windows, so it cannot be debugged directly on windows, only after development, packaged and sent to linux for execution). The second way: configure hadoop on windows, connect linux in eclipse for running test (cannot debug). (Real business operation). The third: can be debugged. Configure the local hadoop file. The first two are also called server environments, and the third is a local test environment

• local test environment

  1. Unzip and copy hadoop to the windows system, and add winutils.exe to the bin directory
  2. Configure hadoop environment variables
  3. Modify the hadoop source code, that is, overwrite the jar package that comes with hadoop
     
  4. 
     Add in the main function of configuring hadoop connection on linux
     

     Configuration config = new Configuration();
     config.set("fs.defaultFS", "hdfs://node1:8020");
     config.set("yarn.resourcemanager.hostname", "node1");

  

• Server environment
  1. Package the jar directly, upload it to the linux server, and run it hadoop jar filename.jar com.package.MainRun
  2. Call it directly in eclipse, and the execution process is still on linux

  1. Packaged as a jar file and placed directly locally
  2. Modify the hadoop source code as above
  3. add an attributeconfig.set("mapred.jar", "C:\\Users\\Administrator\\Desktop\\wc.jar");
     
  4. Copy the hadoop configuration file on the server to the src of the local project
     

result: