Hadoop序列化的实现
简介序列化和反序列化
序列化和反序列化的定义:
序列化: 将对象化转化为字节流,以便在网上传输或者写在磁盘上进行永久存储;
反序列化: 将字节流回转成对象。
序列化在分布式数据处理中的使用场景:
序列化在分布式数据处理的两个领域经常出现: 进程间通信和永久存储。
Hadoop序列化操作
序列化Person对象,并将序列化的对象反序列化出来:
1.序列化实体类:
package MapReduce;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
public class Person implements WritableComparable<Person> {
private Text name=new Text();
private IntWritable age=new IntWritable();
private Text sex=new Text();
public Person(String name,int age,String sex) {
this.name.set(name);
this.age.set(age);
this.sex.set(sex);
}
public Person(Text name,IntWritable age,Text sex) {
this.name = name;
this.age = age;
this.sex = sex;
}
public Person()
{
}
@Override
public void readFields(DataInput in) throws IOException {
// TODO Auto-generated method stub
name.readFields(in);
age.readFields(in);
sex.readFields(in);
}
@Override
public void write(DataOutput out) throws IOException {
// TODO Auto-generated method stub
name.write(out);
age.write(out);
sex.write(out);
}
@Override
public int compareTo(Person o) {
// TODO Auto-generated method stub
int result = 0;
int comp1 = name.compareTo(o.name);
if(comp1!=0)
{
return comp1;
}
int comp2=age.compareTo(o.age);
if(comp2!=0)
{
return comp2;
}
int comp3 = sex.compareTo(o.sex);
if(comp3!=0)
{
return comp3;
}
return result;
}
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime*result+((age==null)?0:age.hashCode());
result = prime*result+((name==null)?0:name.hashCode());
result = prime*result+((sex==null)?0:sex.hashCode());
return result;
}
public boolean equals(Object obj) {
if(this==obj)
return true;
if(this==null)
return false;
if(getClass()!=obj.getClass())
return false;
Person other =(Person)obj;
if(age==null)
{
if(other.age!=null)
return false;
}else if(!age.equals(other.age))
return false;
if(name==null)
{
if(other.name!=null)
return false;
}else if(!name.equals(other.name))
return false;
if(sex==null)
{
if(other.sex!=null)
return false;
}else if(!sex.equals(other.sex))
return false;
return true;
}
public String toString() {
return "Person[name="+name+",age="+age+",sex="+sex+"]";
}
}
2.序列化工具类:
package MapReduce;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import org.apache.hadoop.io.Writable;
public class HadoopSerializationUtil {
public static byte[]serialize(Writable writable) throws IOException{
ByteArrayOutputStream out= new ByteArrayOutputStream();
DataOutputStream dataout = new DataOutputStream(out);
writable.write(dataout);
dataout.close();
return out.toByteArray();
}
public static void deserialize(Writable writable,byte[] bytes) throws Exception
{
ByteArrayInputStream in=new ByteArrayInputStream(bytes);
DataInputStream datain=new DataInputStream(in);
writable.readFields(datain);
datain.close();
}
}
3.测试类:
package MapReduce;
public class Test {
public static void main(String[] args) throws Exception {
Person person = new Person("zhangsan",27,"man");
byte[] values=HadoopSerializationUtil.serialize(person);
Person p=new Person();
HadoopSerializationUtil.deserialize(p, values);
System.out.println(p);//反序列化结果
System.out.println(person);//序列化结果
}
}
序列化操作的结果展示
程序执行结果如下图所示:
上面为反序列化结果,下面为序列化结果。