Hadoop序列化的实现

简介序列化和反序列化

序列化和反序列化的定义：

序列化： 将对象化转化为字节流，以便在网上传输或者写在磁盘上进行永久存储；
反序列化： 将字节流回转成对象。

序列化在分布式数据处理中的使用场景：
序列化在分布式数据处理的两个领域经常出现： 进程间通信和永久存储。

Hadoop序列化操作

序列化Person对象，并将序列化的对象反序列化出来：
1.序列化实体类：

package MapReduce;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;

public class Person implements WritableComparable<Person> {
    
    
	
	private Text name=new Text();
	private IntWritable age=new IntWritable();
	private Text sex=new Text();
	
	public Person(String name,int age,String sex) {
    
    
		this.name.set(name);
		this.age.set(age);
		this.sex.set(sex);
	}
	
	public Person(Text name,IntWritable age,Text sex) {
    
    
		this.name = name;
		this.age = age;
		this.sex = sex;
	}
	public Person()
	{
    
    
		
	}
	
	@Override
	public void readFields(DataInput in) throws IOException {
    
    
		// TODO Auto-generated method stub
		name.readFields(in);
		age.readFields(in);
		sex.readFields(in);
		
	}

	@Override
	public void write(DataOutput out) throws IOException {
    
    
		// TODO Auto-generated method stub
		name.write(out);
		age.write(out);
		sex.write(out);
	}
	

	@Override
	public int compareTo(Person o) {
    
    
		// TODO Auto-generated method stub
		int result = 0;
		
		int comp1 = name.compareTo(o.name);
		if(comp1!=0)
		{
    
    
			return comp1;
		}
		
		int comp2=age.compareTo(o.age);
		if(comp2!=0)
		{
    
    
			return comp2;
		}
		
		int comp3 = sex.compareTo(o.sex);
		if(comp3!=0)
		{
    
    
			return comp3; 
		}
		
		return result;
	}
	
	public int hashCode() {
    
    
		final int prime = 31;
		int result = 1;
		result = prime*result+((age==null)?0:age.hashCode());
		result = prime*result+((name==null)?0:name.hashCode());
		result = prime*result+((sex==null)?0:sex.hashCode());
		return result;
	}
	
	public boolean equals(Object obj) {
    
    
		if(this==obj)
			return true;
		if(this==null)
			return false;
		if(getClass()!=obj.getClass())
			return false;
		Person other =(Person)obj;
		if(age==null)
		{
    
    
			if(other.age!=null)
			
				return false;
		}else if(!age.equals(other.age))
				return false;
			if(name==null)
			{
    
    
				if(other.name!=null)
					return false;
			}else if(!name.equals(other.name))
				return false;
			if(sex==null)
			{
    
    
				if(other.sex!=null)
					return false;
			}else if(!sex.equals(other.sex))
				return false;
			return true;
		}
		
		public String toString() {
    
    
			return "Person[name="+name+",age="+age+",sex="+sex+"]";
		}		
}

2.序列化工具类：

package MapReduce;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;

import org.apache.hadoop.io.Writable;

public class HadoopSerializationUtil {
    
    
	
	public static byte[]serialize(Writable writable) throws IOException{
    
    
		ByteArrayOutputStream out= new ByteArrayOutputStream();
		DataOutputStream dataout = new DataOutputStream(out);
		writable.write(dataout);
		dataout.close();
		return out.toByteArray();
	
	}
	
	public static void deserialize(Writable writable,byte[] bytes) throws Exception
	{
    
    
		ByteArrayInputStream in=new ByteArrayInputStream(bytes);
		DataInputStream datain=new DataInputStream(in);
		writable.readFields(datain);
		datain.close();
	}
}

3.测试类：

package MapReduce;

public class Test {
    
    
	
	public static void main(String[] args) throws Exception {
    
    
		Person person = new Person("zhangsan",27,"man");
		byte[] values=HadoopSerializationUtil.serialize(person);

		Person p=new Person();
		HadoopSerializationUtil.deserialize(p, values);
		System.out.println(p);//反序列化结果
		System.out.println(person);//序列化结果
		}
}

序列化操作的结果展示

程序执行结果如下图所示：

上面为反序列化结果，下面为序列化结果。