Two implementation methods of Avro serialization: static implementation and dynamic implementation

 Avro Features:

1. Rich data structures

2. Compact, fast, binary data format

3. Container files, persistent data

4. Support RPC

5. Code generation is optional for easy integration with dynamic languages

 

Avro vs Thrift, Protobuf:

1. Dynamic typing: Avro does not require that code be generated. Data is always accompanied by a schema that permits full processing of that data without code generation, static datatypes, etc. This facilitates construction of generic data-processing systems and languages.

2. Untagged data: Since the schema is present when data is read, considerably less type information need be encoded with data, resulting in smaller serialization size.

3. No manually-assigned field IDs: When a schema changes, both the old and new schema are always present when processing data, so differences may be resolved symbolically, using field names.

 

The following are the download and installation steps:

 

1. Download avro-tools-1.8.1.jar and import the development package

http://avro.apache.org/releases.html

https://mirrors.tuna.tsinghua.edu.cn/apache/avro/avro-1.8.1/java/ 

 

Import the development package:

<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.8.1</version>
</dependency>

 

2. Define the schema file user.avsc

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

 

 3. Generate java code, static implementation

 语法：java -jar /path/to/avro-tools-1.8.1.jar compile schema <schema file> <destination>

java -jar /path/to/avro-tools-1.8.1.jar compile schema user.avsc .

public class SerializationByAvroSchema {
	
	
	public static void main(String[] args) throws Exception{
		readAvro ();
	}

	static void writeAvro() throws Exception{
		Schema schema = new Schema.Parser().parse(new File("D:/tmp/user.avsc"));
		GenericRecord user1 = new GenericData.Record(schema);
		user1.put("name", "Alyssa");
		user1.put("favorite_number", 256);
		user1.put("favorite_color", "red");
		
		GenericRecord user2 = new GenericData.Record(schema);
		user2.put("name", "Alyssa2");
		user2.put("favorite_number", 258);
		user2.put("favorite_color", "green");

		DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema);
		DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter);
		
		dataFileWriter.create(schema, new File("D:/tmp/users.avro"));
		dataFileWriter.append(user1);
		dataFileWriter.append(user2);
		dataFileWriter.close();
	}
	
	static void readAvro() throws Exception{
		Schema schema = new Schema.Parser().parse(new File("D:/tmp/user.avsc"));
		
		File file = new File("D:/tmp/users.avro");
		DatumReader<GenericRecord> userDatumReader = new GenericDatumReader<GenericRecord>(schema);
		DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, userDatumReader);
		GenericRecord user = null;
		while (dataFileReader.hasNext()) {
			user = dataFileReader.next(user);
			System.out.println(user);
		}
		dataFileReader.close();
	}
}

 

4. Do not generate dynamic implementation of java code

public class SerializationByAvroClass {

	public static void main(String[] args) throws Exception{
		// readAvro ();
		//writeJavaObject();
	}
	
	static void writeAvro() throws Exception{
		User user1 = new User();
		user1.setName("Alyssa");
		user1.setFavoriteNumber(256);

		User user2 = new User("Ben", 7, "red");

		User user3 = User.newBuilder()
		             .setName("Charlie")
		             .setFavoriteColor("blue")
		             .setFavoriteNumber (null)
		             .build();

		DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class);
		DataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter);
		dataFileWriter.create(user1.getSchema(), new File("D:/tmp/users.avro"));
		dataFileWriter.append(user1);
		dataFileWriter.append(user2);
		dataFileWriter.append(user3);
		dataFileWriter.close();
	}
	
	static void readAvro() throws Exception{
		File file = new File("D:/tmp/users.avro");
		DatumReader<User> userDatumReader = new SpecificDatumReader<User>(User.class);
		DataFileReader<User> dataFileReader = new DataFileReader<User>(file, userDatumReader);
		User user = null;
		while (dataFileReader.hasNext()) {
			user = dataFileReader.next(user);
			System.out.println(user);
		}
		dataFileReader.close();
	}

}