Apache ORC the smallest, fastest columnar storage for Hadoop workloads.
The Apache ORC file format is a columnar storage format in the Hadoop ecosystem. It was produced as early as early 2013 and was originally produced by Apache Hive to reduce Hadoop data storage space and accelerate Hive query speed.
ACID Support
Includes support for ACID transactions and snapshot isolation
Built-in Indexes
Jump to the right row with indexes including minimum, maximum, and bloom filters for each column.
Complex Types
Supports all of Hive's types including the compound types: structs, lists, maps, and unions
ORC (OptimizedRC File) storage is derived from the storage format of RC (RecordColumnar File). RC is a columnar storage engine and has poor support for schema evolution (modifying the schema requires regenerating data), while ORC is an improvement to RC. However, it still has poor support for schema evolution, mainly in terms of compression encoding and query performance. RC/ORC was originally used in Hive, and finally developed well and became a separate project. The Hive 1.x version supports transaction and update operations based on ORC (other storage formats are not currently supported). With the development of ORC today, it has some very advanced features, such as support for update operations, support for ACID, support for struct, and array complex types. You can use complex types to build a nested data architecture similar to parquet, but when the number of layers is very large, it is very troublesome and complicated to write, and the schema expression provided by parquet is easier to express multi-level nested data types.
ACID support
Historically, the only way to atomically add data to a table in Hive was to add a new partition. Updating or deleting data in partition required removing the old partition and adding it back with the new data and it wasn’t possible to do atomically.
However, user’s data is continually changing and as Hive matured, users required reliability guarantees despite the churning data lake. Thus, we needed to implement ACID transactions that guarantee atomicity, consistency, isolation, and durability. Although we support ACID transactions, they are not designed to support OLTP requirements. It can support millions of rows updated per a transaction, but it can not support millions of transactions an hour.
Additionally, we wanted to support streaming ingest in to Hive tables where streaming applications like Flume or Storm could write data into Hive and have transactions commit once a minute and queries would either see all of a transaction or none of it.
HDFS is a write once file system and ORC is a write-once file format, so edits were implemented using base files and delta files where insert, update, and delete operations are recorded.
Indexes
ORC provides three level of indexes within each file:
file level - statistics about the values in each column across the entire file
stripe level - statistics about the values in each column for each stripe
row level - statistics about the values in each column for each set of 10,000 rows within a stripe
The file and stripe level column statistics are in the file footer so that they are easy to access to determine if the rest of the file needs to be read at all. Row level indexes include both the column statistics for each row group and the position for seeking to the start of the row group.
Column statistics always contain the count of values and whether there are null values present. Most other primitive types include the minimum and maximum values and for numeric types the sum. As of Hive 1.2, the indexes can include bloom filters, which provide a much more selective filter.
The indexes at all levels are used by the reader using Search ARGuments or SARGs, which are simplified expressions that restrict the rows that are of interest. For example, if a query was looking for people older than 100 years old, the SARG would be “age > 100” and only files, stripes, or row groups that had people over 100 years old would be read.