This article is the reading notes for Chapter 5 Managing and Maintaining Time-Based Information of Oracle 19c VLDB and Partitioning Guide.
Oracle Database provides policies for managing and maintaining data based on time.
This chapter discusses the components in the Oracle Database that enable building time-based strategies for managing and maintaining data.
Although most organizations have long considered their data storage to be one of their most valuable corporate assets, how this data is managed and maintained varies from company to company. Initially, data was used to help achieve operational goals, run the business, and help determine the company's future direction and success.
However, new government regulations and guidelines are key drivers of how and why data is retained. Today's regulations require organizations to retain and control information for extended periods of time. As a result, information technology (IT) managers today are striving to meet other goals:
- Store large amounts of data at the lowest possible cost
- Meet new regulatory requirements for data retention and protection
- Increase business opportunities through better analytics based on more data
5.1 Managing Data in Oracle Database With ILM
With Information Lifecycle Management (ILM), you can manage data in an Oracle Database using the rules and regulations that apply to that data.
Today's information comes in many types, such as e-mail messages, photos, or orders in an online transaction processing (OLTP) system. Once you understand the types of data and how it is used, you can understand how it evolves and how it is ultimately disposed of.
A challenge for every organization is to understand how its data evolves and grows, monitor how its use changes over time, and decide how long it should be retained, while complying with all the rules and regulations that now apply to that data . Information lifecycle management (ILM) aims to address these issues through a combination of processes, policies, software, and hardware so that the appropriate technology can be used at each stage of the data lifecycle .
5.1.1 About Oracle Database for ILM
Oracle Database provides an ideal platform for implementing ILM solutions.
The Oracle Database platform provides the following capabilities:
-
Application Transparency
Application transparency is very important in ILM because it means that there is no need to customize the application, and that various changes can be made to the data without any impact on the applications that use the data. Data can be easily moved through different stages of its lifecycle, and access to the data can be optimized using the database. Another important benefit is that application transparency provides the flexibility needed to quickly adapt to any new regulatory requirements, again without any impact on existing applications. -
Fine-Grained Data
Oracle can view data at a very fine-grained level and group related data, whereas storage devices only see bytes and blocks. -
Low-Cost Storage
With so much data being retained, using low-cost storage is a key factor in implementing ILM. Because Oracle can take advantage of many types of storage devices, it can hold the largest amount of data at the lowest possible cost. -
Enforceable compliance policies
When information is retained for compliance reasons, it must be shown to regulators that the data is retained and managed in accordance with the regulations. In Oracle Database, you can define security and audit policies that enforce and log all access to data.
5.1.1.1 Oracle Database Manages All Types of Data
Information lifecycle management touches all data in an organization.
This data includes not only structured data, such as orders in an OLTP system or sales history in a data warehouse, but also unstructured data, such as emails, documents, and images. Oracle Database supports BLOBs and Oracle SecureFiles for storing unstructured data, and a sophisticated document management system is available in Oracle Text.
If all the information in your organization is contained within an Oracle Database, you can take advantage of the features and capabilities the database provides to manage and move the evolution of data through its lifecycle without having to manage multiple types of data stores .
5.1.1.2 Regulatory Requirements
Many organizations must retain specific data for a specific period of time. Failure to comply with these regulations can result in organizations having to pay substantial fines.
Various regulatory requirements around the world, such as Sarbanes-Oxley, HIPAA, DOD5015.2-STD in the US, and the European Data Privacy Directive in the EU, are changing the way organizations manage their data. These regulations dictate what data must be retained, whether it can be changed, and for how long it must be retained , which can be 30 years or more.
These regulations often require that electronic data be secured from unauthorized access and alteration, and that an audit trail be maintained of all changes to the data and by whom . Oracle Database can retain large amounts of data without impacting application performance . It also includes the features needed to restrict access and prevent unauthorized changes to data, and can be further enhanced with Oracle Audit Vault and Database Firewall. Oracle Database also provides cryptographic functions that can prove that a privileged user did not intentionally modify data. Using Flashback Data technology, you can store all versions of a row during its lifetime in a tamper-proof historical archive.
5.1.1.3 The Benefits of an Online Archive
There are many benefits to archiving online .
During the lifecycle of data, there usually comes a point at which it is no longer regularly accessed and is considered eligible for archiving. Traditionally, data is removed from the database and stored on tape, where you can store large amounts of information at very low cost. Today, instead of archiving this data to tape, it can be kept in the database or transferred to a central online archive database. All of this information can be stored using low-cost storage devices that cost very close to tape per gigabyte.
There are many benefits to keeping all data in an Oracle database for archiving. The most important benefit is that data is always immediately available . Therefore, time is not wasted finding tapes for archived data and determining whether the tapes are readable and still in a format that can be loaded into the database.
If the data has been archived for many years, development time may also be required to write the program to reload the data from the tape archive into the database. This can prove to be expensive and time-consuming, especially if the data is very old. If the data is kept in the database then this is not a problem as it is online and in the latest database format.
Keeping historical data in the database no longer affects the time required to back up the database and the size of the backup. When RMAN is used to back up a database, it only includes changed data in the backup. Because historical data is unlikely to change, backed up data will not be backed up again .
Another important factor to consider is how data is physically removed from the database , especially if the data is being transferred from a production system to a central database archive. Oracle provides the ability to move this data quickly between databases by using transportable tablespaces or partitions , which move the data as a complete unit.
When you need to delete data from the database, the fastest way is to delete a set of data. This is achieved by keeping the data in its own partition. Partitions can be deleted, which is a very quick operation. However, if this method cannot be used because data relationships must be maintained, a regular SQL delete statement must be issued. You should not underestimate the time it takes to issue a delete statement.
If there is a need to remove data from the database, and there may be a need to return data to the database in the future, then consider removing data in a database format such as transportable tablespace, or using XML Oracle Database's ability to extract information in an open format.
Consider archiving your data online to an Oracle database for the following reasons:
- The cost of disk is close to the cost of tape, so you save yourself the time of finding the tape that contains the data and the cost of recovering that data
- Data stays online when needed, giving you faster access to meet business needs
- Online data means instant access , so regulators are less likely to be fined for failing to provide data
- Data can be accessed using current applications , so you don't need to waste resources building new applications
5.1.2 Implementing ILM Using Oracle Database
Building an Information Lifecycle Management solution with Oracle Database is simple.
An ILM solution can be accomplished in four simple steps, although step 4 is optional if ILM is not implemented for compliance:
- Step 1: Define the data class
- Step 2: Create a storage layer for the data class
- Step 3: Create Data Access and Migration Policies
- Step 4: Define and enforce a compliance policy
5.1.2.1 Step 1: Define the Data Classes
To effectively utilize information lifecycle management, start by examining all the data in your organization before implementing an information lifecycle management solution.
After reviewing the data, determine the following:
- What data is important, where is it stored and what must be retained
- How this data flows within the organization
- How will this data change over time and whether it is still needed
- Required Data Availability and Degree of Protection
- Data retention for legal and business requirements
Once you understand the purpose of the data, you can classify the data on this basis. The most common types of classification are by year or date, but other types are possible, such as by product or privacy. Mixed classifications are also available, such as by privacy and year.
To treat data classes differently, the data must be physically separated . When information is initially created, it is usually accessed frequently, but over time it may be less referenced. For example, when a customer places an order, they periodically check the order to see its status and whether the order has been shipped. Once an order arrives, they may never refer to it again. This order will also be included in the periodic reports that are run to see what items are being ordered, but will not appear in any reports over time and will only be referenced in the future if someone does a detailed analysis involving this . For example, orders can be categorized by fiscal quarters Q1, Q2, Q3, and Q4, or historical orders.
The advantage of using this approach is that when data is grouped at the row level by category, in this case order date, all orders for Q1 can be managed as a single unit , while orders for Q2 will reside in a different in class. This can be achieved by using partitions. Because partitioning is transparent to the application , the data is physically separated, but the application can still find all orders.
5.1.2.1.1 Partitioning for ILM
Partitioning involves physically placing data based on its value, and a frequently used technique is to partition information by date.
Figure 5-1 illustrates a scenario where orders for Q1, Q2, Q3, and Q4 are stored in separate partitions, while orders for previous years are stored in other partitions.
Figure 5-1 Assigning data classes to partitions
Oracle provides several different partitioning methods. Range partitioning is a common partitioning method in ILM. Interval and reference partitions are also particularly well suited for use in ILM environments.
Partitioning data has many benefits. Partitioning provides an easy way to distribute data across the appropriate storage devices based on its usage, while still keeping data online and stored on the most cost-effective device . Because partitioning is transparent to anyone accessing the data, no application changes are required , so partitioning can be implemented at any time. When new partitions are needed, they are simply added using the ADD PARTITION clause, or if interval partitioning is used, they are created automatically.
Among other benefits, each partition can have its own local index. When the optimizer uses partition pruning, queries access only relevant partitions instead of all partitions, improving query response time.
5.1.2.1.2 The Lifecycle of Data
An analysis of your data may reveal that initially it is accessed and updated very frequently. As the data ages, its access frequency decreases to be negligible (if at all).
Most organizations find themselves in a situation where many users access current data and very few users access legacy data, as shown in Figure 5-2. Data is considered: active, less active, historical or ready for archiving .
Since so much data is kept, it should be moved to different physical locations during its lifetime . Depending on where the data is in its lifecycle, it must be on the most appropriate storage device.
Figure 5-2 Data usage over time
5.1.2.2 Step 2: Create Storage Tiers for the Data Classes
Because Oracle Database can take advantage of many different storage options, the second step in implementing an information lifecycle management solution is to establish the required storage tiers.
Although you can create as many storage tiers as you want, it is recommended to start with the following tiers:
-
High performance
The high performance storage tier stores all important and frequently accessed data, such as partitions holding our Q1 orders. This layer uses smaller, faster disks on high-performance storage devices. -
Low cost
The low cost storage tier stores infrequently accessed data, such as partitions holding Q2, Q3, and Q4 orders. This tier is built using high-capacity disks, such as those in Modular Storage Arrays or low-cost ATA disks, which provide the maximum amount of inexpensive storage. -
Online Archive
The Online Archive storage tier is where all data that is rarely accessed or modified is stored. This storage layer can be very large and stores the largest amount of data. You can use various techniques to compress data. Data stored on low-cost storage devices such as ATA drives remains online and available at a cost only slightly higher than storing this information on tape, without the disadvantages that come with archiving data to tape. If the Online Archive storage tier is marked as read-only, it is impossible to change the data and subsequent backups are not required after the initial database backup. -
Offline Archive (Optional)
The Offline Archive storage tier is an optional tier because it is only used when data needs to be removed from the database and stored in another format such as XML on tape.
Figure 5-2 illustrates how the data is used within a time interval. Using this information, it can be determined that to retain all of this information, multiple storage tiers are required to hold all the data, which also has the benefit of significantly reducing overall storage costs.
After the storage tier is created, the data classes identified in Step 1: Define Data Classes will be physically implemented inside the database using partitions. This approach provides an easy way to distribute data across appropriate storage devices based on its usage, while still keeping data online and readily available, and stored on the most cost-effective device.
You can also use Oracle Automatic Storage Management (Oracle ASM) to manage data across storage tiers. Oracle ASM is a high-performance, easy-to-manage file storage solution for Oracle Database. Oracle ASM is a volume manager that provides a file system designed for database use. To use Oracle ASM, you need to allocate partitioned disks for the Oracle database, with priority given to striping and mirroring. Oracle ASM manages disk space and distributes I/O load across all available resources to optimize performance while eliminating the need for manual I/O tuning. For example, you can increase the disk size of a database or move part of a database to a new device without shutting down the database.
5.1.2.2.1 Assigning Classes to Storage Tiers
Once the storage tiers are defined, the data classes (partitions) identified in Step 1 can be assigned to the appropriate storage tiers.
This allocation provides an easy way to distribute data across appropriate storage devices based on its usage, keeping data online and available, and stored on the most cost-effective device. In Figure 5-3, data identified as active, inactive, historical, or ready for archiving are assigned to the high-performance tier, low-cost storage tier, online archive storage tier, and offline archive, respectively. Using this approach, no application changes are required, as the data is still visible.
Figure 5-3 Data life cycle
5.1.2.2.2 The Costs Savings of Using Tiered Storage
One of the benefits of implementing an ILM strategy is the cost savings that can be achieved by using multiple tiers of storage.
Suppose you want to store 3 TB of data, with 200 GB for high performance, 800 GB for low cost, and 2 TB for online archiving. Assume a cost per GB of $72 for the high-performance tier, $14 for the low-cost tier, and $7 for the online archive tier.
Table 5-1 illustrates the potential cost savings of using tiered storage instead of storing all data on one type of storage. As you can see, the cost savings are substantial, and further savings can be achieved if the data is suitable for OLTP and HCC database compression.
Table 5-1 Cost Savings Using Tiered Storage
| storage layer | All use high-performance disks | multiple storage tiers | Multiple storage tiers + compression |
|---|---|---|---|
| High performance (200 GB) | $14,400 | $14,400 | $14,400 |
| Low cost (800 GB) | $57,600 | $11,200 | $11,200 |
| Online Archive (2 TB) | $144,000 | $14,000 | $5,600 |
| total per column | $216,000 | $39,600 | $31,200 |
5.1.2.3 Step 3: Create Data Access and Migration Policies
The third step in implementing an information lifecycle management solution is to ensure that only authorized users can access data and to specify how data is moved through its lifecycle.
There are various techniques for migrating data between storage tiers as it ages.
5.1.2.3.1 Controlling Access to Data
Security of data is another very important part of information lifecycle management as access rights to data may change during its lifetime.
Additionally, there may be regulatory requirements that impose strict requirements on how data is accessed.
Data in an Oracle database can be protected using database features such as:
- database security
- view
- Virtual Private Database (VPD)
A Virtual Private Database (VPD) defines a very fine-grained level of access to the database. Security policies determine which rows can be viewed and which columns are visible. Multiple policies can be defined so that different users and applications see different views of the same data. For example, most users can see information for Q1, Q2, Q3, and Q4, while only authorized users can view historical data.
Security policies are defined at the database level and applied transparently to all database users. The benefit of this approach is that it provides a secure and controlled environment for accessing data that cannot be overridden and can be enforced without requiring any application changes. Additionally, read-only tablespaces can be defined to ensure that data does not change.
5.1.2.3.2 Moving Data using Partitioning
During its lifetime, data must move, and partitioning is one technique that can be used.
Mobile data can occur due to the following reasons:
- For performance, only a limited number of orders are kept on high-performance disks
- Data is no longer being accessed frequently and is using valuable high-performance storage and must be moved to low-cost storage
- The law requires that the information is always available for a given time interval and must be kept securely at the lowest possible cost
There are several ways to physically move data within an Oracle Database to take advantage of different storage tiers. For example, if the data is partitioned, the partition containing the Q2 orders can be moved online from the high-performance storage tier to the low-cost storage tier. Because data is moved within the database, it can be physically moved without affecting the applications that need it or causing disruption to regular users.
Sometimes individual data items must be moved, rather than a group of data. For example, suppose data is classified according to privacy level, and a previously confidential report is now made available to the public. If the classification changes from secret to public and the data is partitioned according to its privacy classification, the row will automatically be moved to the partition containing the public data.
Whenever data is moved from its original source, it is important to ensure that the chosen process complies with any regulatory requirements, such as that the data cannot be altered, protected from unauthorized access, easily read, and stored in an approved location.
5.1.2.4 Step 4: Define and Enforce Compliance Policies
The fourth step in an information lifecycle management solution is to create a compliance policy.
When data is fragmented and distributed, compliance policies must be defined and enforced at each data location, which can easily lead to compliance policies being overlooked. However, using an Oracle database to provide a central location to store data means that compliance policies can be enforced very easily because they are all managed and enforced from one central location .
When defining a compliance policy, consider the following:
- data retention
- invariance
- privacy
- audit
- Expired
5.1.2.4.1 Data Retention
A retention policy describes how data is retained, how long it must be retained, and what happens to the data after its lifetime.
An example of a retention policy is that records must be stored in their original form, no modification allowed, must be retained for seven years, and then may be deleted. Using Oracle Database Security, you can ensure that data remains intact and that only authorized processes can delete data at the appropriate time. Retention policies can also be defined through lifecycle definitions in the ILM Assistant.
5.1.2.4.2 Immutability
Immutability involves proving to external parties that data is complete and has not been modified.
Oracle Database can generate a password or digital signature and keep it inside or outside the database to show that the data has not been changed.
5.1.2.4.3 Privacy
Oracle Database provides several methods to ensure data privacy.
Access to data can be tightly controlled using security policies defined with a virtual private database (VPD). In addition, individual columns can be encrypted so that anyone viewing the raw data cannot see their contents.
5.1.2.4.4 Auditing
Oracle Database keeps track of all access and changes to data.
These auditing capabilities can be defined at the table level or through fine-grained auditing, which specifies criteria for when audit records are generated. Auditing can be further enhanced using Oracle Audit Vault and Database Firewall.
5.1.2.4.5 Expiration
Eventually, data may expire for business or regulatory reasons and must be deleted from the database.
Oracle Database can delete data very quickly and efficiently by simply dropping the partition that contains the information to be deleted.
5.2 Implementing an ILM Strategy With Heat Map and ADO
To implement an information lifecycle management (ILM) strategy for data movement in the database, you can use the heat map and automatic data optimization (ADO) features.
Note: Heatmaps and ADO are supported in Oracle Database 12c Release 2 multi-tenant environments. These two functions are only supported after 12c.
5.2.1 Using Heat Map
To implement your ILM strategy, you can use heat maps in Oracle Database to track data access and modification.
Heatmaps provide data access tracking at the segment level as well as data modification tracking at the segment and row levels . You can enable this feature with the HEAT_MAP initialization parameter.
Heatmap data can assist Automatic Data Optimization (ADO) to manage the content of In-Memory column store (IM column store) using ADO strategy. Using heat map data that includes column statistics and other relevant statistics, the IM column store can determine when it is nearly full (under memory pressure). If it is determined that it is almost full, inactive segments can be evicted if there are more frequently accessed segments that would benefit from population in the IM column store.
5.2.1.1 Enabling and Disabling Heat Map
You can enable and disable heat map tracking at the system or session level with the ALTER SYSTEM or ALTER SESSION statement using the HEAT_MAP clause.
For example, the following SQL statement enables heatmap tracking for a DB instance.
ALTER SYSTEM SET HEAT_MAP = ON;
When heatmap is enabled, all visits are tracked by the in-memory activity tracking module. Objects in the SYSTEM and SYSAUX tablespaces are not tracked .
The following SQL statement disables heatmap tracking.
ALTER SYSTEM SET HEAT_MAP = OFF;
When the heatmap is disabled, the in-memory activity tracking module does not track visits. The default value of the HEAT_MAP initialization parameter is OFF .
The HEAT_MAP initialization parameter also enables and disables Automatic Data Optimization (ADO). For ADO, heatmaps must be enabled at the system level .
5.2.1.2 Displaying Heat Map Tracking Data With Views
View heat map trace data using the V$ , ALL , DBA*, and USER* heat map views.
Example 5-1 shows an example of the information provided by the heat map view. The V$HEAT_MAP_SEGMENT view displays real-time segment access information. The ALL_, DBA_, and USER_HEAT_MAP_SEGMENT views show the latest segment access time for all segments visible to the user. The ALL_, DBA_, and USER_HEAT_MAP_SEG_HISTOGRAM views display segment access information for all segments visible to the user. The DBA_HEATMAP_TOP_OBJECTS view displays heat map information for the most active objects. The DBA_HEATMAP_TOP_TABLESPACES view displays heat map information for the most active tablespaces.
Example 5-1 Heatmap view
/* enable heat map tracking if necessary*/
SELECT SUBSTR(OBJECT_NAME,1,20), SUBSTR(SUBOBJECT_NAME,1,20), TRACK_TIME, SEGMENT_WRITE,
FULL_SCAN, LOOKUP_SCAN FROM V$HEAT_MAP_SEGMENT;
SUBSTR(OBJECT_NAME,1 SUBSTR(SUBOBJECT_NAM TRACK_TIM SEG FUL LOO
-------------------- -------------------- --------- --- --- ---
SALES SALES_Q1_1998 01-NOV-12 NO NO NO
SALES SALES_Q3_1998 01-NOV-12 NO NO NO
SALES SALES_Q2_2000 01-NOV-12 NO NO NO
SALES SALES_Q3_1999 01-NOV-12 NO NO NO
SALES SALES_Q2_1998 01-NOV-12 NO NO NO
SALES SALES_Q2_1999 01-NOV-12 NO NO NO
SALES SALES_Q4_2001 01-NOV-12 NO NO NO
SALES SALES_Q1_1999 01-NOV-12 NO NO NO
SALES SALES_Q4_1998 01-NOV-12 NO NO NO
SALES SALES_Q1_2000 01-NOV-12 NO NO NO
SALES SALES_Q1_2001 01-NOV-12 NO NO NO
SALES SALES_Q2_2001 01-NOV-12 NO NO NO
SALES SALES_Q3_2000 01-NOV-12 NO NO NO
SALES SALES_Q4_2000 01-NOV-12 NO NO NO
EMPLOYEES 01-NOV-12 NO NO NO
...
SELECT SUBSTR(OBJECT_NAME,1,20), SUBSTR(SUBOBJECT_NAME,1,20), SEGMENT_WRITE_TIME,
SEGMENT_READ_TIME, FULL_SCAN, LOOKUP_SCAN FROM USER_HEAT_MAP_SEGMENT;
SUBSTR(OBJECT_NAME,1 SUBSTR(SUBOBJECT_NAM SEGMENT_W SEGMENT_R FULL_SCAN LOOKUP_SC
-------------------- -------------------- --------- --------- --------- ---------
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
SALES SALES_Q1_1998 30-OCT-12 01-NOV-12
...
SELECT SUBSTR(OBJECT_NAME,1,20), SUBSTR(SUBOBJECT_NAME,1,20), TRACK_TIME, SEGMENT_WRITE, FULL_SCAN,
LOOKUP_SCAN FROM USER_HEAT_MAP_SEG_HISTOGRAM;
SUBSTR(OBJECT_NAME,1 SUBSTR(SUBOBJECT_NAM TRACK_TIM SEG FUL LOO
-------------------- -------------------- --------- --- --- ---
SALES SALES_Q1_1998 31-OCT-12 NO NO YES
SALES SALES_Q1_1998 01-NOV-12 NO NO YES
SALES SALES_Q1_1998 30-OCT-12 NO YES YES
SALES SALES_Q2_1998 01-NOV-12 NO NO YES
SALES SALES_Q2_1998 31-OCT-12 NO NO YES
SALES SALES_Q2_1998 30-OCT-12 NO YES YES
SALES SALES_Q3_1998 01-NOV-12 NO NO YES
SALES SALES_Q3_1998 30-OCT-12 NO YES YES
SALES SALES_Q3_1998 31-OCT-12 NO NO YES
SALES SALES_Q4_1998 01-NOV-12 NO NO YES
SALES SALES_Q4_1998 31-OCT-12 NO NO YES
SALES SALES_Q4_1998 30-OCT-12 NO YES YES
SALES SALES_Q1_1999 01-NOV-12 NO NO YES
SALES SALES_Q1_1999 31-OCT-12 NO NO YES
...
SELECT SUBSTR(OWNER,1,20), SUBSTR(OBJECT_NAME,1,20), OBJECT_TYPE, SUBSTR(TABLESPACE_NAME,1,20),
SEGMENT_COUNT FROM DBA_HEATMAP_TOP_OBJECTS ORDER BY SEGMENT_COUNT DESC;
SUBSTR(OWNER,1,20) SUBSTR(OBJECT_NAME,1 OBJECT_TYPE SUBSTR(TABLESPACE_NA SEGMENT_COUNT
-------------------- -------------------- ------------------ -------------------- -------------
SH SALES TABLE EXAMPLE 96
SH COSTS TABLE EXAMPLE 48
PM ONLINE_MEDIA TABLE EXAMPLE 22
OE PURCHASEORDER TABLE EXAMPLE 18
PM PRINT_MEDIA TABLE EXAMPLE 15
OE CUSTOMERS TABLE EXAMPLE 10
OE WAREHOUSES TABLE EXAMPLE 9
HR EMPLOYEES TABLE EXAMPLE 7
OE LINEITEM_TABLE TABLE EXAMPLE 6
IX STREAMS_QUEUE_TABLE TABLE EXAMPLE 6
SH FWEEK_PSCAT_SALES_MV TABLE EXAMPLE 5
SH CUSTOMERS TABLE EXAMPLE 5
HR LOCATIONS TABLE EXAMPLE 5
HR JOB_HISTORY TABLE EXAMPLE 5
SH PRODUCTS TABLE EXAMPLE 5
...
SELECT SUBSTR(TABLESPACE_NAME,1,20), SEGMENT_COUNT
FROM DBA_HEATMAP_TOP_TABLESPACES ORDER BY SEGMENT_COUNT DESC;
SUBSTR(TABLESPACE_NA SEGMENT_COUNT
-------------------- -------------
EXAMPLE 351
USERS 11
SELECT COUNT(*) FROM DBA_HEATMAP_TOP_OBJECTS;
COUNT(*)
----------
64
SELECT COUNT(*) FROM DBA_HEATMAP_TOP_TABLESPACES;
COUNT(*)
----------
2
5.2.1.3 Managing Heat Map Data With DBMS_HEAT_MAP Subprograms
The DBMS_HEAT_MAP package provides additional flexibility for displaying heat map data using the DBMS_HEAT_MAP subroutine.
DBMS_HEAT_MAP includes a set of APIs that externalize heatmaps at different storage levels such as block, extent, segment, object, and tablespace; a second set of APIs externalizes heatmaps materialized by background processes for top-level tablespaces.
Example 5-2 shows an example of using the DBMS_HEAT_MAP package subroutine.
Example 5-2 Using the DBMS_HEAT_MAP package subroutine
SELECT SUBSTR(segment_name,1,10) Segment, min_writetime, min_ftstime
FROM TABLE(DBMS_HEAT_MAP.OBJECT_HEAT_MAP('SH','SALES'));
SELECT SUBSTR(tablespace_name,1,16) Tblspace, min_writetime, min_ftstime
FROM TABLE(DBMS_HEAT_MAP.TABLESPACE_HEAT_MAP('EXAMPLE'));
SELECT relative_fno, block_id, blocks, TO_CHAR(min_writetime, 'mm-dd-yy hh-mi-ss') Mintime,
TO_CHAR(max_writetime, 'mm-dd-yy hh-mi-ss') Maxtime,
TO_CHAR(avg_writetime, 'mm-dd-yy hh-mi-ss') Avgtime
FROM TABLE(DBMS_HEAT_MAP.EXTENT_HEAT_MAP('SH','SALES')) WHERE ROWNUM < 10;
SELECT SUBSTR(owner,1,10) Owner, SUBSTR(segment_name,1,10) Segment,
SUBSTR(partition_name,1,16) Partition, SUBSTR(tablespace_name,1,16) Tblspace,
segment_type, segment_size FROM TABLE(DBMS_HEAT_MAP.OBJECT_HEAT_MAP('SH','SALES'));
OWNER SEGMENT PARTITION TBLSPACE SEGMENT_TYPE SEGMENT_SIZE
---------- ---------- ---------------- ---------------- -------------------- ------------
SH SALES SALES_Q1_1998 EXAMPLE TABLE PARTITION 8388608
SH SALES SALES_Q2_1998 EXAMPLE TABLE PARTITION 8388608
SH SALES SALES_Q3_1998 EXAMPLE TABLE PARTITION 8388608
SH SALES SALES_Q4_1998 EXAMPLE TABLE PARTITION 8388608
SH SALES SALES_Q1_1999 EXAMPLE TABLE PARTITION 8388608
...
5.2.2 Using Automatic Data Optimization
To implement your ILM strategy, you can use Automatic Data Optimization (ADO) to automatically compress and move data between different storage tiers in your database.
This capability includes the ability to create policies that specify different levels of compression for each layer, as well as control when data movement occurs.
To use automatic data optimization, you must enable heatmaps at the system level. You can enable this feature with the HEAT_MAP initialization parameter.
5.2.2.1 Managing Policies for Automatic Data Optimization
You can specify policies for ADO at the row, segment, and tablespace granularity levels when using SQL statements to create and alter tables. In addition, ADO strategies can perform operations on indexes.
By specifying policies for ADO, you can automate data movement between different storage tiers in the database . These policies also enable you to specify different compression levels for each tier and control when data movement occurs .
ADO policies for tables
The ILM clause of the SQL CREATE and ALTER TABLE statements enables you to create, drop, enable, or disable ADO policies. ILM policy clauses determine the compression or storage tiering policy and contain other clauses, such as AFTER and ON clauses, to specify the conditions under which policy actions should occur. New policies can be added for ADO when a table is created. You can alter this table to add more policies or to enable, disable, or delete existing policies. You can add policies to an entire table or to partitions of a table. When adding an ADO policy to a table or a partition of a table, you can specify only one condition type for the AFTER clause. ILM ADO policies are given a system-generated name such as P1, P2, etc. through Pn.
Segment-level policies are executed only once. After a policy has successfully executed, it is disabled and will not be evaluated again. However, you can explicitly enable the policy again. Row-level policies continue to execute and are not disabled after successful execution.
The scope of an ADO policy can be specified for a group of related objects or at the segment or row level using the keywords GROUP, ROW, or SEGMENT.
The default compression maps that can be applied to Group Policy are:
- COMPRESS ADVANCED on heap tables maps to standard compression for indexes and LOW for LOB segments.
- COMPRESS FOR QUERY LOW/QUERY HIGH on heap tables maps to standard compression for indexes and MEDIUM for LOB segments.
- COMPRESS FOR ARCHIVE LOW/ARCHIVE HIGH on heap tables maps to standard compression for indexes and HIGH for LOB segments.
In short, standard compression is used for indexes and different compression levels for LOB segments.
Cannot change compression map. GROUP can only be applied to segment level policies. Storage tiering policies are only available at the segment level and cannot be specified at the row level.
ADO Strategies for In-Memory Column Store
Automatic Data Optimization (ADO) supports in-memory column stores (IM column stores) with INMEMORY, INMEMORY MECOMPRESS, and NO INMEMORY policy types.
- To enable object population in an in-memory column store , include INMEMORY in the ADD POLICY clause.
- To increase the compression level of objects in the IM column store , include INMEMORY MEMCOMPRESS in the ADD POLICY clause.
- To explicitly evict objects that would benefit least from IM column storage, include NO INMEMORY in the ADD POLICY clause. For example:
Following is an example of using the NO INMEMORY clause to evict an object from the IM column store.
ALTER TABLE sales_2015 ILM ADD POLICY NO INMEMORY
AFTER 7 DAYS OF NO ACCESS;
ADO policies with the In-Memory Column Store clause can only be section-level policies. The USER/DBA_ILMDATAMOVEMENTPOLICIES and V$HEAT_MAP_SEGMENT views include information about ADO policies for in-memory column stores.
Custom ADO Strategies
You can customize strategies using the ON PL/SQL_function option, which provides the ability to determine when to execute the strategy. The ON PL/SQL_function option applies only to section-level policies. For example:
CREATE OR REPLACE FUNCTION my_custom_ado_rules (objn IN NUMBER) RETURN BOOLEAN;
ALTER TABLE sales_custom ILM ADD POLICY COMPRESS ADVANCED SEGMENT
ON my_custom_ado_rules;
5.2.2.2 Creating a Table With an ILM ADO Policy
Use the ILM ADD POLICY clause with the CREATE TABLE statement to create a table with an ILM ADO policy.
The SQL statement in Example 5-3 creates a table and adds an ILM policy.
Example 5-3 Creating a table using the ILM ADO strategy
/* Create an example table with an ILM ADO policy */
CREATE TABLE sales_ado
(PROD_ID NUMBER NOT NULL,
CUST_ID NUMBER NOT NULL,
TIME_ID DATE NOT NULL,
CHANNEL_ID NUMBER NOT NULL,
PROMO_ID NUMBER NOT NULL,
QUANTITY_SOLD NUMBER(10,2) NOT NULL,
AMOUNT_SOLD NUMBER(10,2) NOT NULL )
PARTITION BY RANGE (time_id)
( PARTITION sales_q1_2012 VALUES LESS THAN (TO_DATE('01-APR-2012','dd-MON-yyyy')),
PARTITION sales_q2_2012 VALUES LESS THAN (TO_DATE('01-JUL-2012','dd-MON-yyyy')),
PARTITION sales_q3_2012 VALUES LESS THAN (TO_DATE('01-OCT-2012','dd-MON-yyyy')),
PARTITION sales_q4_2012 VALUES LESS THAN (TO_DATE('01-JAN-2013','dd-MON-yyyy')) )
ILM ADD POLICY COMPRESS FOR ARCHIVE HIGH SEGMENT
AFTER 12 MONTHS OF NO ACCESS;
/* View the existing ILM ADO polices */
SELECT SUBSTR(policy_name,1,24) POLICY_NAME, policy_type, enabled
FROM USER_ILMPOLICIES;
POLICY_NAME POLICY_TYPE ENABLE
------------------------ ------------- ------
P1 DATA MOVEMENT YES
What are the other Policy Types?
5.2.2.3 Adding ILM ADO Policies
Add the ILM ADO policy to the table using the ILM ADD POLICY clause with the ALTER TABLE statement.
The SQL statement in Example 5-4 provides an example of adding an ILM policy to a partition of the sales table.
Example 5-4 Adding an ILM ADO policy
/* Add a row-level compression policy after 30 days of no modifications */
ALTER TABLE sales MODIFY PARTITION sales_q1_2002
ILM ADD POLICY ROW STORE COMPRESS ADVANCED ROW
AFTER 30 DAYS OF NO MODIFICATION;
/* Add a segment level compression policy for data after 6 months of no modifications */
ALTER TABLE sales MODIFY PARTITION sales_q1_2001
ILM ADD POLICY COMPRESS FOR ARCHIVE HIGH SEGMENT
AFTER 6 MONTHS OF NO MODIFICATION;
/* Add a segment level compression policy for data after 12 months of no access */
ALTER TABLE sales MODIFY PARTITION sales_q1_2000
ILM ADD POLICY COMPRESS FOR ARCHIVE HIGH SEGMENT
AFTER 12 MONTHS OF NO ACCESS;
/* Add storage tier policy to move old data to a different tablespace */
/* that is on low cost storage media */
ALTER TABLE sales MODIFY PARTITION sales_q1_1999
ILM ADD POLICY
TIER TO my_low_cost_sales_tablespace;
/* View the existing polices */
SELECT SUBSTR(policy_name,1,24) POLICY_NAME, policy_type, enabled
FROM USER_ILMPOLICIES;
POLICY_NAME POLICY_TYPE ENABLE
------------------------ ------------- ------
P1 DATA MOVEMENT YES
P2 DATA MOVEMENT YES
P3 DATA MOVEMENT YES
P4 DATA MOVEMENT YES
P5 DATA MOVEMENT YES
5.2.2.4 Disabling and Deleting ILM ADO Policies
Use the ILM DISABLE POLICY or ILM DELETE POLICY clause with the ALTER TABLE statement to disable or delete the ILM ADO policy.
You can disable or delete ADO's ILM strategy, as shown in the SQL statement in Example 5-5. Occasionally, you may need to delete existing ILM policies if they conflict with new policies that you are adding.
Example 5-5 Disabling and deleting an ILM ADO policy
/* You can disable or delete an ADO policy in a table with the following */
ALTER TABLE sales_ado ILM DISABLE POLICY P1;
ALTER TABLE sales_ado ILM DELETE POLICY P1;
/* You can disable or delete all ADO policies in a table with the following */
ALTER TABLE sales_ado ILM DISABLE_ALL;
ALTER TABLE sales_ado ILM DELETE_ALL;
/* You can disable or delete an ADO policy in a partition with the following */
ALTER TABLE sales MODIFY PARTITION sales_q1_2002 ILM DISABLE POLICY P2;
ALTER TABLE sales MODIFY PARTITION sales_q1_2002 ILM DELETE POLICY P2;
/* You can disable or delete all ADO policies in a partition with the following */
ALTER TABLE sales MODIFY PARTITION sales_q1_2000 ILM DISABLE_ALL;
ALTER TABLE sales MODIFY PARTITION sales_q1_2000 ILM DELETE_ALL;
5.2.2.5 Specifying Segment-Level Compression and Storage Tiering With ADO
You can specify compression at the segment level within a table using segment-level compression tiering policies.
Combined with row-level compression tiering strategies, you have fine-grained control over how data is stored and managed in the database.
Example 5-6 illustrates how to create a policy for ADO to enforce compression and storage tiering on the sales_ado table, reflecting the following business requirements:
- bulk load data
- Run OLTP workloads
- Compressed archive high after six months of no updates
- Move to low-cost storage
Example 5-6 Using segment-level compression and storage tiering
/* Add a segment level compression policy after 6 months of no changes */
ALTER TABLE sales_ado ILM ADD POLICY
COMPRESS FOR ARCHIVE HIGH SEGMENT
AFTER 6 MONTHS OF NO MODIFICATION;
Table altered.
/* Add storage tier policy */
ALTER TABLE sales_ado ILM ADD POLICY
TIER TO my_low_cost_tablespace;
SELECT SUBSTR(policy_name,1,24) POLICY_NAME, policy_type, enabled
FROM USER_ILMPOLICIES;
POLICY_NAME POLICY_TYPE ENABLED
------------------------ ------------- -------
...
P6 DATA MOVEMENT YES
P7 DATA MOVEMENT YES
5.2.2.6 Specifying Row-Level Compression Tiering With ADO
In addition to Basic and Advanced compression, Automatic Data Optimization (ADO) policies also support Hybrid Columnar Compression (HCC).
HCC row-level policies can be defined on any table, regardless of the table's compression type . HCC can be used to compress rows from cold blocks when there is DML activity in other parts of the segment.
For HCC policies on non-HCC tables, row movement may occur during an update if the row resides in an HCC compression unit (CU). Also, similar to other use cases for row movement, index maintenance is necessary to update index entries referencing moved rows.
Oracle Database 12c Release 1 (12.1) supports row-level policies: however, the database must be compatible with 12.2 or later to use the HCC row-level compression policy.
Example 5-7 Creating an ADO Policy Using Row-Level Hybrid Columnar Compression
The SQL statement in Example 5-7 uses HCC to create a policy on the rows of the table employees_ilm.
ALTER TABLE employees_ilm
ILM ADD POLICY COLUMN STORE COMPRESS FOR QUERY ROW
AFTER 30 DAYS OF NO MODIFICATION;
Example 5-8 Creating an ADO Policy Using Row-Level Advanced Compression
The SQL statement in Example 5-8 creates a policy that uses advanced compression for the rows of the table sales_ado.
ALTER TABLE sales_ado
ILM ADD POLICY ROW STORE COMPRESS ADVANCED ROW
AFTER 60 DAYS OF NO MODIFICATION;
SELECT policy_name, policy_type, enabled
FROM USER_ILMPOLICIES;
POLICY_NAME POLICY_TYPE ENABLE
------------------------ ------------- -------
...
P8 DATA MOVEMENT YES
5.2.2.7 Managing ILM ADO Parameters
You can customize the ADO environment using the ILM ADO parameters set using the CUSTOMIZE_ILM procedure in the DBMS_ILM_ADMIN PL/SQL package.
The various ILM ADO parameters are described in Table 5-2.
Table 5-2 ILM ADO parameters
| name | describe |
|---|---|
| ABSOLUTEJOB LIMIT | Limits the absolute number of concurrent ADO jobs. |
| DEGREEOF PARALLELISM | Determines the degree of parallelism with which to run ADO strategy jobs. |
| ENABLED | Controls ADO background evaluation and execution. Enabled by default (TRUE or 1). The setting of ENABLED and the HEAT_MAP initialization parameter interact as follows: - If the HEAT_MAP initialization parameter is set to ON and the ENABLED parameter is set to FALSE (0), heat map statistics are collected, but ADO does not automatically operate on the statistics. - If the HEAT_MAP initialization parameter is set to OFF and the ENABLED parameter is set to TRUE (1), no heatmap statistics are collected and ADO does nothing because ADO cannot rely on heatmap statistics. ADO behaves as if ENABLED is set to FALSE. |
| EXECUTION MODE | Controls whether ADO executes in online or offline mode. The default is online (2). |
| EXECUTION INTERVAL | Determines how often ADO starts background evaluation. The default is 15 minutes . |
| JOB LIMIT | Controls the maximum number of ADO jobs at any one time. The maximum number of concurrent ADO jobs is calculated as (JOB LIMIT) x (number of instances) x (number of CPUs per instance). The default value is 2. |
| POLICY TIME | Determines whether the ADO policy is specified in seconds or days. Values are 1 second or 0 days (default). |
| RETENTION TIME | Specifies the length of time that data from completed ADO tasks is retained before being purged. The default is 30 days. |
| TBS PERCENT USED | Specifies the percentage of the tablespace quota when the tablespace is considered full. The default value is 85% . |
| TBS PERCENT FREE | The value of the TBS_PERCENT_FREE parameter specifies the target free percentage for the table space. The default value is 25% . |
For the values of the TBS_PERCENT* parameters, ADO will do its best, but not guaranteed . When the percentage of the table space quota reaches the value of TBS_PERCENT_USED, ADO starts moving data so that the free percentage of the table space quota approaches the value of TBS_PERCENT_FREE. For example, suppose TBS_PERCENT_USED is set to 85, TBS_PERCENT_FREE is set to 25, and the tablespace is 90% full. ADO then initiates the operation to move the data so that the tablespace quota is at least 25% free, which can also be interpreted as the tablespace quota usage being below 75%.
You can display parameters using the DBA_ILMPARAMETERS view. For example, the following query displays the values of ADO related parameters.
SQL> SELECT NAME, VALUE FROM DBA_ILMPARAMETERS;
---------------------------------------------------------------- ----------
ENABLED 1
RETENTION TIME 30
JOB LIMIT 2
EXECUTION MODE 2
EXECUTION INTERVAL 15
TBS PERCENT USED 85
TBS PERCENT FREE 25
POLICY TIME 0
ABSOLUTE JOB LIMIT 10
DEGREE OF PARALLELISM 4
...
5.2.2.8 Using PL/SQL Functions for Policy Management
You can use the PL/SQL DBMS_ILM and DBMS_ILM_ADMIN packages for advanced policy management and customization to implement more complex ADO scenarios and control when policies actively move and compress data.
Using the PL/SQL DBMS_ILM and DBMS_ILM_ADMIN packages, you can manage ADO's ILM activities so that they do not negatively impact critical production workloads. Database Compatibility must be set to at least 12.0 to use these packages.
The EXECUTE_ILM procedure of the DBMS_ILM package creates and schedules jobs to execute ADO strategies. The EXECUTE_ILM() procedure provides this functionality regardless of any previously scheduled ILM jobs. All jobs are created and scheduled to run immediately ; however, whether they run immediately depends on how many jobs are queued by the scheduler.
The EXECUTE_ILM procedure can be used if you want more control over the execution of your ILM jobs and do not want to wait until the next maintenance window .
The STOP_ILM procedure of the DBMS_ILM package stops all jobs, all running jobs, jobs based on task ID, or specific jobs.
The CUSTOMIZE_ILM procedure in the DBMS_ILM_ADMIN PL/SQL package enables you to customize the settings of ADO, as shown in Example 5-9.
For example, you can set the values of the TBS_PERCENT_USED and TBS_PERCENT_FREE ILM parameters or set the ABS_JOBLIMIT ILM parameter. TBS_PERCENT_USED and TBS_PERCENT_FREE determine when data is moved according to tablespace quotas, and ABS_JOBLIMIT sets the absolute number of concurrent ADO jobs.
You can also use the DBMS_METADATA PL/SQL package to recreate objects containing policies.
Example 5-9 Customizing ADO Settings Using CUSTOMIZE_ILM
SQL> BEGIN
2 DBMS_ILM_ADMIN.CUSTOMIZE_ILM(DBMS_ILM_ADMIN.TBS_PERCENT_USED, 85);
3 DBMS_ILM_ADMIN.CUSTOMIZE_ILM(DBMS_ILM_ADMIN.TBS_PERCENT_FREE, 25);
4 END;
5 /
SQL> BEGIN
2 DBMS_ILM_ADMIN.CUSTOMIZE_ILM(DBMS_ILM_ADMIN.ABS_JOBLIMIT, 10);
3 END;
4 /
5.2.2.9 Using Views to Monitor Policies for ADO
You can use the DBA_ILM* and USER_ILM* views to view and monitor the ADO policies associated with your database objects, making it easier to change policies as needed.
- The DBA/USER_ILMDATAMOVEMENTPOLICIES view displays information about data movement related properties for ADO-specific ILM policies.
- The DBA/USER_ILMTASKS view shows the task IDs for the procedure EXECUTE_ILM. Each time the user calls the procedure EXECUTE_ILM, a task ID is returned to keep track of this particular call. A task ID is also generated to track recurring internal ILM tasks for the database. This view contains information about all ILM tasks for ADO.
- The DBA/USER_ILMEVALUATIONDETAILS view displays detailed information about the strategies considered for a particular task. If a policy is selected for evaluation, it also shows the name of the job that executed the policy. If the policy is not enforced, this view also provides the reason.
- The DBA/USER_ILMOBJECTS view shows all ADO objects and strategies in the database. Many objects inherit policy through their parent object, or because they were created in a specific tablespace. This view provides a mapping between policies and objects. In the case of inherited policies, this view also indicates the level of inherited policies.
- The DBA/USER_ILMPOLICIES view displays detailed information about all ADO policies in the database.
- The DBA/USER_ILMRESULTS view displays information about ADO's data movement related jobs in the database.
- The DBA_ILMPARAMETERS view displays information about ADO-related parameters.
5.2.3 Limitations and Restrictions With ADO and Heat Map
This topic discusses limitations and limitations related to ADO and heatmaps.
Limitations and restrictions related to ADO and heatmaps include:
- Time validity supports partition-level ADO and compression, with the exception of row-level ADO policies, which compress rows past their valid time (accessed or modified).
- Partition-level ADO and compression support in-database archiving if partitioned on an ORA_ARCHIVE_STATE column.
- ADO's custom policies (user-defined functions) are not supported if the policy is defaulted at the tablespace level.
- When using storage tiering, ADO does not check for storage space in the target tablespace .
- ADO is not supported for tables with object types or materialized views.
- ADO is not supported for index-organized tables or clusters.
- ADO concurrency (the number of concurrent policy jobs for ADO) depends on the concurrency of the Oracle scheduler. If a policy job for ADO fails more than twice, the job will be marked as disabled and must be manually enabled later.
- ADO has limitations related to moving tables and table partitions.
5.3 Controlling the Validity and Visibility of Data in Oracle Database
You can use In-Database Archiving and Time Validation to control the validity and visibility of data in Oracle Database.
5.3.1 Using In-Database Archiving
In-Database Archiving enables you to archive rows in a table by marking them as inactive.
These inactive rows are in the database and can be optimized using compression, but are invisible to the application . The data in these rows can be used for compliance purposes by setting session parameters if desired.
With in-database archiving, you can store more data in a single database for longer periods of time without impacting application performance . Archive data can be compressed to help improve backup performance, and updates to archive data can be deferred during application upgrades to improve upgrade performance.
To manage in-database archiving for a table, you must enable ROW ARCHIVAL for the table and manipulate the table's ORA_ARCHIVE_STATE hidden column. Alternatively, you can specify ACTIVE or . ALLROW ARCHIVAL VISIBILITY
For example, you can use SQL statements similar to those in Example 5-10 to hide or show rows in a table. The intent is to display only active data in most cases, but in specific cases all data needs to be maintained.
Live SQL: View and run related examples on Oracle Live SQL in Oracle Live SQL: Working with In-Database Archive Examples.
Example 5-10 Using in-database archiving
/* Set visibility to ACTIVE to display only active rows of a table.*/
ALTER SESSION SET ROW ARCHIVAL VISIBILITY = ACTIVE;
CREATE TABLE employees_indbarch
(employee_id NUMBER(6) NOT NULL,
first_name VARCHAR2(20), last_name VARCHAR2(25) NOT NULL,
email VARCHAR2(25) NOT NULL, phone_number VARCHAR2(20),
hire_date DATE NOT NULL, job_id VARCHAR2(10) NOT NULL, salary NUMBER(8,2),
commission_pct NUMBER(2,2), manager_id NUMBER(6), department_id NUMBER(4)) ROW ARCHIVAL;
/* Show all the columns in the table, including hidden columns */
SELECT SUBSTR(COLUMN_NAME,1,22) NAME, SUBSTR(DATA_TYPE,1,20) DATA_TYPE, COLUMN_ID AS COL_ID,
SEGMENT_COLUMN_ID AS SEG_COL_ID, INTERNAL_COLUMN_ID AS INT_COL_ID, HIDDEN_COLUMN, CHAR_LENGTH
FROM USER_TAB_COLS WHERE TABLE_NAME='EMPLOYEES_INDBARCH';
NAME DATA_TYPE COL_ID SEG_COL_ID INT_COL_ID HID CHAR_LENGTH
---------------------- -------------------- ---------- ---------- ---------- --- -----------
ORA_ARCHIVE_STATE VARCHAR2 1 1 YES 4000
EMPLOYEE_ID NUMBER 1 2 2 NO 0
FIRST_NAME VARCHAR2 2 3 3 NO 20
LAST_NAME VARCHAR2 3 4 4 NO 25
EMAIL VARCHAR2 4 5 5 NO 25
PHONE_NUMBER VARCHAR2 5 6 6 NO 20
HIRE_DATE DATE 6 7 7 NO 0
JOB_ID VARCHAR2 7 8 8 NO 10
SALARY NUMBER 8 9 9 NO 0
COMMISSION_PCT NUMBER 9 10 10 NO 0
MANAGER_ID NUMBER 10 11 11 NO 0
DEPARTMENT_ID NUMBER 11 12 12 NO 0
/* Insert some data into the table */
INSERT INTO employees_indbarch(employee_id, first_name, last_name, email,
hire_date, job_id, salary, manager_id, department_id)
VALUES (251, 'Scott', 'Tiger', '[email protected]', '21-MAY-2009',
'IT_PROG', 50000, 103, 60);
INSERT INTO employees_indbarch(employee_id, first_name, last_name, email,
hire_date, job_id, salary, manager_id, department_id)
VALUES (252, 'Jane', 'Lion', '[email protected]', '11-JUN-2009',
'IT_PROG', 50000, 103, 60);
/* Decrease the ORA_ARCHIVE_STATE column size to improve formatting in queries */
COLUMN ORA_ARCHIVE_STATE FORMAT a18;
/* The default value for ORA_ARCHIVE_STATE is '0', which means active */
SELECT employee_id, ORA_ARCHIVE_STATE FROM employees_indbarch;
EMPLOYEE_ID ORA_ARCHIVE_STATE
----------- ------------------
251 0
252 0
/* Insert a value into ORA_ARCHIVE_STATE to set the record to inactive status*/
UPDATE employees_indbarch SET ORA_ARCHIVE_STATE = '1' WHERE employee_id = 252;
/* Only active records are in the following query */
SELECT employee_id, ORA_ARCHIVE_STATE FROM employees_indbarch;
EMPLOYEE_ID ORA_ARCHIVE_STATE
----------- ------------------
251 0
/* Set visibility to ALL to display all records */
ALTER SESSION SET ROW ARCHIVAL VISIBILITY = ALL;
SELECT employee_id, ORA_ARCHIVE_STATE FROM employees_indbarch;
EMPLOYEE_ID ORA_ARCHIVE_STATE
----------- ------------------
251 0
252 1
Both the above example and the example in LiveSQL just demonstrate the visibility of data, but they don't say how not to affect performance.
5.3.2 Using Temporal Validity
Time validity enables you to track time periods of real-world validity. The effective time of the data can be set by the user and the application program, and the data can be selected according to the specified effective time or effective time range.
Applications typically record the validity (or validity) of facts recorded in a database, along with date or time stamps relevant to business management. For example, an employee's hire date in a human resources (HR) application determines the effective date of insurance coverage, which is an effective date. This date is contrasted with the date or time the employee record was entered into the database. The temporal attribute of the former (hire date) is called effective time (VT), while the latter (date entered into the database) is called transaction time (TT). Valid times are usually controlled by the user, while transaction times are managed by the system. Pay attention to the concept and difference between VT and TT
Time finiteness uses the concept of VT.
For ILM, the time-of-validity attribute can represent when a fact is valid and when it is not valid in the business world. Using the Time in Valid attribute, queries can display only currently valid rows, and not rows containing currently invalid facts, such as closed orders or future hires.
Concepts integral to efficient-time temporal modeling include:
-
Valid Time
This is a user-defined representation of time. Examples of effective times include project start and end dates, and employee hire and termination dates. -
Tables with efficient time semantics
These tables have one or more user-defined time dimensions, each with a start and an end. -
Effective Time Flashback Query
This is the ability to perform as of date and version queries using the effective time dimension.
A valid time period consists of the two datetime columns specified in the table definition. You can add valid time periods by adding the column explicitly, or create the column automatically. A valid time period can be added during the create table or alter table process.
To support session-level visibility control for temporal table queries, the DBMS_FLASHBACK_ARCHIVEPL/SQL package provides the ENABLE_AT_VALID_TIME procedure. To execute the procedure, you need the required system and object privileges.
The following PL/SQL procedure sets the effective time visibility for a given time.
SQL> EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time
('ASOF', '31-DEC-12 12.00.01 PM');
The following PL/SQL procedure sets the visibility of temporal data at the session level to the currently valid data for the valid time period.
SQL> EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT');
The following procedure sets the visibility of temporal data to full table, which is the default temporal table visibility.
SQL> EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');
5.3.3 Creating a Table With Temporal Validity
The examples in this topic show how to create time-effective tables.
Example 5-11 shows the use of time validity.
Live SQL: View and run related examples on Oracle Live SQL in Oracle Live SQL: Creating Time-Efficient Tables .
Example 5-11 Create a table with time validity
-- PERIOD FOR即时间有效性的关键字
/* Create a time with an employee tracking timestamp */
/* using the specified columns*/
CREATE TABLE employees_temp (
employee_id NUMBER(6) NOT NULL, first_name VARCHAR2(20), last_name VARCHAR2(25) NOT NULL,
email VARCHAR2(25) NOT NULL, phone_number VARCHAR2(20), hire_date DATE NOT NULL,
job_id VARCHAR2(10) NOT NULL, salary NUMBER(8,2), commission_pct NUMBER(2,2),
manager_id NUMBER(6), department_id NUMBER(4),
PERIOD FOR emp_track_time);
DESCRIBE employees_temp
Name Null? Type
------------------------------------------------------- -------- ---------------
EMPLOYEE_ID NOT NULL NUMBER(6)
FIRST_NAME VARCHAR2(20)
LAST_NAME NOT NULL VARCHAR2(25)
EMAIL NOT NULL VARCHAR2(25)
PHONE_NUMBER VARCHAR2(20)
HIRE_DATE NOT NULL DATE
JOB_ID NOT NULL VARCHAR2(10)
SALARY NUMBER(8,2)
COMMISSION_PCT NUMBER(2,2)
MANAGER_ID NUMBER(6)
DEPARTMENT_ID NUMBER(4)
SQL> SELECT SUBSTR(COLUMN_NAME,1,22) NAME, SUBSTR(DATA_TYPE,1,28) DATA_TYPE, COLUMN_ID AS COL_ID,
SEGMENT_COLUMN_ID AS SEG_COL_ID, INTERNAL_COLUMN_ID AS INT_COL_ID, HIDDEN_COLUMN
FROM USER_TAB_COLS WHERE TABLE_NAME='EMPLOYEES_TEMP';
NAME DATA_TYPE COL_ID SEG_COL_ID INT_COL_ID HID
---------------------- ---------------------------- ------ ---------- ---------- ---
EMP_TRACK_TIME_START TIMESTAMP(6) WITH TIME ZONE 1 1 YES
EMP_TRACK_TIME_END TIMESTAMP(6) WITH TIME ZONE 2 2 YES
EMP_TRACK_TIME NUMBER 3 YES
EMPLOYEE_ID NUMBER 1 3 4 NO
FIRST_NAME VARCHAR2 2 4 5 NO
LAST_NAME VARCHAR2 3 5 6 NO
EMAIL VARCHAR2 4 6 7 NO
PHONE_NUMBER VARCHAR2 5 7 8 NO
HIRE_DATE DATE 6 8 9 NO
JOB_ID VARCHAR2 7 9 10 NO
SALARY NUMBER 8 10 11 NO
COMMISSION_PCT NUMBER 9 11 12 NO
MANAGER_ID NUMBER 10 12 13 NO
DEPARTMENT_ID NUMBER 11 13 14 NO
/* Insert/update/delete with specified values for time columns */
INSERT INTO employees_temp(emp_track_time_start, emp_track_time_end, employee_id, first_name,
last_name, email, hire_date, job_id, salary, manager_id, department_id)
VALUES (TIMESTAMP '2009-06-01 12:00:01 Europe/Paris',
TIMESTAMP '2012-11-30 12:00:01 Europe/Paris', 251, 'Scott', 'Tiger',
'[email protected]', DATE '2009-05-21', 'IT_PROG', 50000, 103, 60);
INSERT INTO employees_temp(emp_track_time_start, emp_track_time_end, employee_id, first_name,
last_name, email, hire_date, job_id, salary, manager_id, department_id)
VALUES (TIMESTAMP '2009-06-01 12:00:01 Europe/Paris',
TIMESTAMP '2012-12-31 12:00:01 Europe/Paris', 252, 'Jane', 'Lion',
'[email protected]', DATE '2009-06-11', 'IT_PROG', 50000, 103, 60);
UPDATE employees_temp set salary = salary + salary * .05
WHERE emp_track_time_start <= TIMESTAMP '2009-06-01 12:00:01 Europe/Paris';
SELECT employee_id, SALARY FROM employees_temp;
EMPLOYEE_ID SALARY
----------- ----------
251 52500
252 52500
/* No rows are deleted for the following statement because no records */
/* are in the specified track time. */
DELETE employees_temp WHERE emp_track_time_end < TIMESTAMP '2001-12-31 12:00:01 Europe/Paris';
0 rows deleted.
/* Show rows that are in a specified time period */
SELECT employee_id FROM employees_temp
WHERE emp_track_time_start > TIMESTAMP '2009-05-31 12:00:01 Europe/Paris' AND
emp_track_time_end < TIMESTAMP '2012-12-01 12:00:01 Europe/Paris';
EMPLOYEE_ID
-----------
251
/* Show rows that are in a specified time period */
SELECT employee_id FROM employees_temp AS OF PERIOD FOR
emp_track_time TIMESTAMP '2012-12-01 12:00:01 Europe/Paris';
EMPLOYEE_ID
-----------
252
5.3.4 Limitations and Restrictions With In-Database Archiving and Temporal Validity
This topic lists limitations and limitations related to in-database archiving and time validity.
Limitations and constraints include:
- ILM does not support time-efficient OLTP table compression. Segment-level ILM and compression are supported if partitioning is done on the end time column.
- OLTP table compression for in-database archiving does not support ILM. ORA_ARCHIVE_STATE supports segment-level ILM and compression if partitioned on the column.
reference:
- https://oracle-base.com/articles/12c/temporal-validity-12cr1
- https://connor-mcdonald.com/2021/09/17/is-my-table-temporal/
- https://www.researchgate.net/publication/282835771_Checking_and_Verifying_Temporal_Data_Validity_Using_Valid_Time_Temporal_Dimension_and_Queries_in_Oracle_12C
5.4 Implementing an ILM System Manually Using Partitioning
You can manually implement an information lifecycle management (ILM) system using partitions.
Example 5-12 illustrates how to manually create storage tiers and partition tables across those storage tiers, then set a virtual private database (VPD) policy on that database to restrict access to the online archive tier data.
Example 5-12 Implementing an ILM system manually
REM Setup the tablespaces for the data
REM These tablespaces would be placed on a High Performance Tier
CREATE SMALLFILE TABLESPACE q1_orders DATAFILE 'q1_orders'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE q2_orders DATAFILE 'q2_orders'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE q3_orders DATAFILE 'q3_orders'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE q4_orders DATAFILE 'q4_orders'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM These tablespaces would be placed on a Low Cost Tier
CREATE SMALLFILE TABLESPACE "2006_ORDERS" DATAFILE '2006_orders'
SIZE 5M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE "2005_ORDERS" DATAFILE '2005_orders'
SIZE 5M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM These tablespaces would be placed on the Online Archive Tier
CREATE SMALLFILE TABLESPACE "2004_ORDERS" DATAFILE '2004_orders'
SIZE 5M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE old_orders DATAFILE 'old_orders'
SIZE 15M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM Now create the Partitioned Table
CREATE TABLE allorders (
prod_id NUMBER NOT NULL,
cust_id NUMBER NOT NULL,
time_id DATE NOT NULL,
channel_id NUMBER NOT NULL,
promo_id NUMBER NOT NULL,
quantity_sold NUMBER(10,2) NOT NULL,
amount_sold NUMBER(10,2) NOT NULL)
--
-- table wide physical specs
--
PCTFREE 5 NOLOGGING
--
-- partitions
--
PARTITION BY RANGE (time_id)
( partition allorders_pre_2004 VALUES LESS THAN
(TO_DATE('2004-01-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE old_orders,
partition allorders_2004 VALUES LESS THAN
(TO_DATE('2005-01-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE "2004_ORDERS",
partition allorders_2005 VALUES LESS THAN
(TO_DATE('2006-01-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE "2005_ORDERS",
partition allorders_2006 VALUES LESS THAN
(TO_DATE('2007-01-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE "2006_ORDERS",
partition allorders_q1_2007 VALUES LESS THAN
(TO_DATE('2007-04-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE q1_orders,
partition allorders_q2_2007 VALUES LESS THAN
(TO_DATE('2007-07-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE q2_orders,
partition allorders_q3_2007 VALUES LESS THAN
(TO_DATE('2007-10-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE q3_orders,
partition allorders_q4_2007 VALUES LESS THAN
(TO_DATE('2008-01-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE q4_orders);
ALTER TABLE allorders ENABLE ROW MOVEMENT;
REM Here is another example using INTERVAL partitioning
REM These tablespaces would be placed on a High Performance Tier
CREATE SMALLFILE TABLESPACE cc_this_month DATAFILE 'cc_this_month'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
CREATE SMALLFILE TABLESPACE cc_prev_month DATAFILE 'cc_prev_month'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM These tablespaces would be placed on a Low Cost Tier
CREATE SMALLFILE TABLESPACE cc_prev_12mth DATAFILE 'cc_prev_12'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM These tablespaces would be placed on the Online Archive Tier
CREATE SMALLFILE TABLESPACE cc_old_tran DATAFILE 'cc_old_tran'
SIZE 2M AUTOEXTEND ON NEXT 1M MAXSIZE UNLIMITED LOGGING
EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO ;
REM Credit Card Transactions where new partitions
REM are automatically placed on the high performance tier
CREATE TABLE cc_tran (
cc_no VARCHAR2(16) NOT NULL,
tran_dt DATE NOT NULL,
entry_dt DATE NOT NULL,
ref_no NUMBER NOT NULL,
description VARCHAR2(30) NOT NULL,
tran_amt NUMBER(10,2) NOT NULL)
--
-- table wide physical specs
--
PCTFREE 5 NOLOGGING
--
-- partitions
--
PARTITION BY RANGE (tran_dt)
INTERVAL (NUMTOYMINTERVAL(1,'month') ) STORE IN (cc_this_month )
( partition very_old_cc_trans VALUES LESS THAN
(TO_DATE('1999-07-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE cc_old_tran ,
partition old_cc_trans VALUES LESS THAN
(TO_DATE('2006-07-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE cc_old_tran ,
partition last_12_mths VALUES LESS THAN
(TO_DATE('2007-06-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE cc_prev_12mth,
partition recent_cc_trans VALUES LESS THAN
(TO_DATE('2007-07-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE cc_prev_month,
partition new_cc_tran VALUES LESS THAN
(TO_DATE('2007-08-01 00:00:00'
,'SYYYY-MM-DD HH24:MI:SS'
,'NLS_CALENDAR=GREGORIAN'
)) TABLESPACE cc_this_month);
REM Create a Security Policy to allow user SH to see all credit card data,
REM PM only sees this years data,
REM and all other uses cannot see the credit card data
CREATE OR REPLACE FUNCTION ilm_seehist
(oowner IN VARCHAR2, ojname IN VARCHAR2)
RETURN VARCHAR2 AS con VARCHAR2 (200);
BEGIN
IF SYS_CONTEXT('USERENV','CLIENT_INFO') = 'SH'
THEN -- sees all data
con:= '1=1';
ELSIF SYS_CONTEXT('USERENV','CLIENT_INFO') = 'PM'
THEN -- sees only data for 2007
con := 'time_id > ''31-Dec-2006''';
ELSE
-- others nothing
con:= '1=2';
END IF;
RETURN (con);
END ilm_seehist;
/
The above example includes three parts. The first part is an example of Range Partition, the second part is an example of Interval Partition, and the third part is creating a row-level security policy, where con means condition, that is, where condition. CLIENT_INFO is the session information stored by the application.
5.5 Managing ILM Heat Map and ADO with Oracle Enterprise Manager
You can use Oracle Enterprise Manager Cloud Control to manage heat maps and automatic data optimization.