Overview Oracle SecureFiles
Contemporary organizations deal with data that includes traditional data stored in relational tables, semi-structured data such as XML and word processing documents, and unstructured data such as media and imaging data. Oracle has used LOBs since the Oracle 8i release to take care of semi-structured and unstructured data, but the implementation of LOBs suffered from several drawbacks, as follows:
- The LOBs were created for mostly “write once, read many times” operations and couldn’t handle requent updates.
- LOBs assumed low concurrent usage data.
- You had to control the amount of undo retained by setting the retention and pctversion parameters, which led to additional management burden.
- LOBs weren’t expected to be very large in size.
- It was assumed that the LOB sizes were uniform. The chunk size could be only 32 KB at its maximum.
- LOBs were not planned with the concurrent usage requirements in Oracle RAC in mind.
The reality is that unstructured and semi-structured data today are very different from the way they were just a few years ago. Today’s LOBs can be quite large, and they could come in all sizes. High concurrency in a LOB environment is quite common, and the LOBs in the previous release couldn’t efficiently deal with highly concurrent RAC environments.
In Oracle Database 11g, Oracle offers a completely new way of handling unstructured data to address the concerns resulting from the way LOBs are currently implemented. The newly reengineered LOB data type is called Oracle SecureFiles.
You can refer to the older LOB implementation as BasicFiles. Oracle SecureFiles offer intelligent compression and transparent encryption capabilities and improve performance while being easy to manage and implement. Oracle SecureFiles use variable chunk sizes, which can be as large as 64 MB.By storing these chunks next to one another, Oracle also minimizes fragmentation.SecureFiles relieve the user from version control tasks by determining whether to generate full redo records or to generate them only for the changed data. Read and write performance is also higher with SecureFiles because they offer a new client/server network layer that allows fast data transfer. SecureFiles also maintain internal memory and space usage statistics that enable the database to maintain the SecureFiles with minimal specification of parameters by you.
Initialization Parameters
The SecureFile functionality is available once the COMPATIBLE initialization parameter is set to 11.0.0.0.0 or higher.
The DB_SECUREFILE initialization parameter controls the default action of the database with regards to LOB storage. The allowable values are:
- ALWAYS - All LOBs in ASSM tablespaces are created as SecureFile LOBs. LOBs in non-ASSM tablespaces are created as BasicFile LOBs unless explicitly specified as SecureFiles. BasicFile storage options are ignored, and SecureFile default storage options are used for any unspecified options.
- FORCE - All LOBs are created as SecureFile LOBs. If the LOB is being created in a non-ASSM tablespace, an error is thrown. BasicFile storage options are ignored, and SecureFile default storage options are used for any unspecified options.
- PERMITTED - The default setting, which allows SecureFile LOB storage when the SECUREFILE keyword is used. The default storage method is BASICFILE.
- NEVER - Creation of SecureFile LOBs is not permitted.
- IGNORE - Prevent creation of SecureFile LOBs, and ignore any errors associated with SecureFile storage options.
The parameter is dynamic, so it can be set using the ALTER SYSTEM command.
SQL> alter system set db_securefile=ignore;
Note:If you use any SecureFile options such as encryption, compression, and deduplication for a BasicFile, you’ll get an error.When you set the db_securefile as NEVER or IGNORE,you wouldn't get any error while you create a Securefile data.Actually database would automatically create a BASICFILE lob for you.
LOB Deduplication
The DEDUPLICATE option of SecureFiles allows decuplication of entries within a LOB at the table or partition level. As you would expect, the processing associated with preventing duplication incurs an overhead. The KEEP_DUPLICATES options explicitly prevents deduplication. The example below compares the space usage of a regular SecureFile and a deduplicated SecureFile.
CREATE TABLE keep_duplicates_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE keepdup_lob(
KEEP_DUPLICATES
);
CREATE TABLE deduplicate_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE dedup_lob (
DEDUPLICATE
);
DECLARE
l_clob CLOB := RPAD('X', 10000, 'X');
BEGIN
FOR i IN 1 .. 1000 LOOP
INSERT INTO keep_duplicates_tab VALUES (i, l_clob);
END LOOP;
COMMIT;
FOR i IN 1 .. 1000 LOOP
INSERT INTO deduplicate_tab VALUES (i, l_clob);
END LOOP;
COMMIT;
END;
/
EXEC DBMS_STATS.gather_table_stats(USER, 'keep_duplicates_tab');
EXEC DBMS_STATS.gather_table_stats(USER, 'deduplicate_tab');
COLUMN segment_name FORMAT A30
SELECT segment_name, bytes
FROM user_segments
WHERE segment_name IN ('KEEPDUP_LOB', 'DEDUP_LOB');
SEGMENT_NAME BYTES
------------------------------ ----------
DEDUP_LOB 262144
KEEPDUP_LOB 19267584
2 rows selected.
Notice how much smaller the deduplicated segment is. The space savings depend on the extent of the duplication within the LOB segment.
The duplication mode can be reset using the ALTER TABLE command.
ALTER TABLE deduplicate_tab MODIFY LOB(clob_data) (
KEEP_DUPLICATES
);
EXEC DBMS_STATS.gather_table_stats(USER, 'deduplicate_tab');
COLUMN segment_name FORMAT A30
SELECT segment_name, bytes
FROM user_segments
WHERE segment_name IN ('KEEPDUP_LOB', 'DEDUP_LOB');
SEGMENT_NAME BYTES
------------------------------ ----------
DEDUP_LOB 76808192
KEEPDUP_LOB 68091904
2 rows selected.
LOB Encryption
Ecryption of SecureFile LOBs relies on a wallet, or Hardware Security Model (HSM), to hold the encryption key. You can refer to following blog for details,
http://czmmiao.iteye.com/blog/1954082
http://www.oracle-base.com/articles/10g/transparent-data-encryption-10gr2.php
so complete that before trying the following examples.
The ENCRYPT option is available in SecureFiles to perform block-level encryption of LOB contents. An optional USING clause defines which encryption algorithm is used (3DES168, AES128, AES192, or AES256), the default being 'AES192'. The NO SALT option is not available for SecureFile encryption. The encryption is applied on a per-column basis, so it affects all partitions within a LOB. The DECRYPT option is used to explicitly prevent encryption. The example below shows creation of a table with an encrypted SecureFile LOB.
CREATE TABLE encrypt_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE encrypt_lob(
ENCRYPT USING 'AES256');
The ALTER TABLE command is used to encrypt or decrypt an existing column. To switch between encryption algorithms you must use the REKEY option.
ALTER TABLE encrypt_tab MODIFY (
clob_data CLOB DECRYPT);
ALTER TABLE encrypt_tab MODIFY (
clob_data CLOB ENCRYPT USING '3DES168');
ALTER TABLE encrypt_tab REKEY USING 'AES192';
Encryption is not supported by the original import and export utilities or by transportable tablespaces, so data must be transfered using the Data Pump import and export utilities.
LOB Compression
The COMPRESS option of SecureFiles enables compression of LOB contents at table or partition level. The level of compression is indicated using the optional MEDIUM and HIGH keywords. If no compression level is specified, MEDIUM is used. There is an overhead associated with compressing the LOB contents, so using a high level of compression may be counterproductive to system performance. The compression in SecureFiles does not affect table compression, and vice versa. The example below compares the space usage of a regular and a compressed SecureFile LOB.
CREATE TABLE nocompress_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE nocompress_lob(
NOCOMPRESS);
CREATE TABLE compress_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE compress_lob (
COMPRESS HIGH);
DECLARE
l_clob CLOB := RPAD('X', 10000, 'X');
BEGIN
FOR i IN 1 .. 1000 LOOP
INSERT INTO nocompress_tab VALUES (i, l_clob);
END LOOP;
COMMIT;
FOR i IN 1 .. 1000 LOOP
INSERT INTO compress_tab VALUES (i, l_clob);
END LOOP;
COMMIT;
END;
/
EXEC DBMS_STATS.gather_table_stats(USER, 'nocompress_tab');
EXEC DBMS_STATS.gather_table_stats(USER, 'compress_tab');
COLUMN segment_name FORMAT A30
SELECT segment_name, bytes
FROM user_segments
WHERE segment_name IN ('COMPRESS_LOB', 'NOCOMPRESS_LOB');
SEGMENT_NAME BYTES
------------------------------ ----------
COMPRESS_LOB 131072
NOCOMPRESS_LOB 193134592
2 rows selected.
We can see the compressed LOB segment is significantly smaller than the uncompressed segment. The level of space saving depends on the type of data stored in the LOB segement.
The compression mode can be reset using the ALTER TABLE command.
ALTER TABLE compress_tab MODIFY LOB(clob_data) (
NOCOMPRESS );
EXEC DBMS_STATS.gather_table_stats(USER, 'compress_tab');
COLUMN segment_name FORMAT A30
SELECT segment_name, bytes
FROM user_segments
WHERE segment_name IN ('COMPRESS_LOB', 'NOCOMPRESS_LOB');
SEGMENT_NAME BYTES
------------------------------ ----------
COMPRESS_LOB 38928384
NOCOMPRESS_LOB 193134592
2 rows selected.
Note:the compress clob can't decompress to the no compress clob directly,it'll decompress data gradually,the more decompress command you execute,the bigger size will be.ALTER TABLE compress_tab MODIFY LOB(clob_data) (
NOCOMPRESS );
EXEC DBMS_STATS.gather_table_stats(USER, 'compress_tab');
COLUMN segment_name FORMAT A30
SELECT segment_name, bytes
FROM user_segments
WHERE segment_name IN ('COMPRESS_LOB', 'NOCOMPRESS_LOB');
SEGMENT_NAME BYTES
------------------------------ ----------
COMPRESS_LOB 39976960
NOCOMPRESS_LOB 193134592
2 rows selected.
Note:features—deduplication, encryption, and compression—either independently or together. If you adopt all three features, Oracle will first perform deduplication of the data and then compress it before encrypting the data.
Storage Options for SecureFiles
Specifying the old storage clauses chunk, pctversion, freepools, freelists and freelistgroups isn’t necessary when you use SecureFiles. If you do specify any of these clauses, the database will parse but not interpret these clauses. Instead of these clauses, you now have the following new storage- related clauses.
- maxsize Specifies the maximum LOB segment size.
- retention Specifies the version control policy by telling the database which versions it must retain. Here are the options you can specify for the retention parameter:
- max Specifies that the database start reclaiming the old version once a segment reaches its maxsize value.
- min Specifies that the database retain old versions for at least the minimum time specified (in seconds).
- auto Lets the database automatically determine the retention time. This is the default setting.
- none Specifies that the database use old versions as much as possible.
LOB Cache and Logging
Both Basicfile and SecureFile LOBs share some basic caching and logging options. The common caching options are:
- CACHE - LOB data is placed in the buffer cache.
- CACHE READS - LOB data is only placed in the buffer cache during read operations, not write operations.
- NOCACHE - LOB data is not placed in the buffer cache. This is the default option for BasicFile and SecureFile LOBs.
The basic logging options are:
- LOGGING - LOB creation and changes generate full redo. This is the default setting.
- NOLOGGING - The operations are not logged in the redo logs and are therefore not recoverable. This is useful during initial creation and during large loads that can be replayed in the event of failure.
- In addition, SecureFile LOBs can have a logging option of FILESYSTEM_LIKE_LOGGING, where only the metadata is logged, which still allows recovery of the segment in the event of failure.
The CACHE option implies LOGGING, so you cannot use CACHE in combination with NOLOGGING or FILESYSTEM_LIKE_LOGGING.
The following code shows an example of explicitly setting the caching and logging options during and after table creation.
CREATE TABLE caching_and_logging_tab (
id NUMBER,
clob_data CLOB)
LOB(clob_data) STORE AS SECUREFILE(
NOCACHE
FILESYSTEM_LIKE_LOGGING);
ALTER TABLE caching_and_logging_tab MODIFY LOB(clob_data) (
CACHE);
Guidelines for ENABLE or DISABLE STORAGE IN ROW
The maximum amount of LOB data stored in the row is the maximum VARCHAR2 size (4000). This includes the control information and the LOB value. If you indicate that the LOB should be stored in the row, once the LOB value and control information is larger than approximately 4000, then the LOB value is automatically moved out of the row.
This suggests the following guidelines:
The default, ENABLE STORAGE IN ROW, is usually the best choice for the following reasons:
- Small LOBs: If the LOB is small (less than approximately 4000 bytes), then the whole LOB can be read while reading the row without extra disk I/O.
- Large LOBs: If the LOB is big (greater than approximately 4000 bytes), then the control information is still stored in the row if ENABLE STORAGE IN ROW is set, even after moving the LOB data out of the row. This control information could enable us to read the out-of-line LOB data faster.
However, in some cases DISABLE STORAGE IN ROW is a better choice. This is because storing the LOB in the row increases the size of the row. This impacts performance if you are doing a lot of base table processing, such as full table scans, multi-row accesses (range scans), or many UPDATE/SELECT to columns other than the LOB columns.
PL/SQL APIs
The DBMS_LOB package is used to access both BasicFile and SecureFile LOBs. General LOB interaction is not affected by the move to SecureFiles. The SETOPTIONS procedure and GETOPTIONS function allow compression, encryption and deduplication options to be set and retrieved on a per-LOB basis.
SETOPTIONS Procedures
This procedure enables/disables CSCE features on a per-LOB basis, overriding the default LOB column settings.
DBMS_LOB.SETOPTIONS (
lob_loc IN BLOB,
option_types IN PLS_INTEGER,
options IN PLS_INTEGER);
DBMS_LOB.SETOPTIONS (
lob_loc IN CLOB CHARACTER SET ANY_CS,
option_types IN PLS_INTEGER,
options IN PLS_INTEGER);
EXCEPTION
SECUREFILE_BADLOB: Unsupported object type for the operation
INVALID_ARGVAL: A parameter value was invalid
QUERY_WRITE: Cannot perform operation during a query
BUFFERING_ENABLED: Cannot perform operation with LOB buffering enabled
Usage Notes
- DBMS_LOB.SETOPTIONS cannot be used to enable or disable encryption on individual LOBs.
- You cannot turn compression or deduplication on or off for a SecureFile column that does not have those features on. The GETOPTIONS Functions and SetOptions Procedures work on individual SecureFiles. You can turn off a feature on a particular SecureFile and turn on a feature that has already been turned off by SetOptions, but you cannot turn on an option that has not been given to the SecureFile when the table was created.
CREATE TABLE securefile_tab (
id NUMBER,
clob_data CLOB
)
LOB(clob_data) STORE AS SECUREFILE securefile_lob(
encrypt
compress
);
INSERT INTO securefile_tab VALUES (1, 'ONE');
INSERT INTO securefile_tab VALUES (2, 'TWO');
COMMIT;
SET SERVEROUTPUT ON
DECLARE
l_clob CLOB;
BEGIN
SELECT clob_data
INTO l_clob
FROM securefile_tab
WHERE id = 1
FOR UPDATE;
DBMS_OUTPUT.put_line('Compression : ' || DBMS_LOB.getoptions(l_clob, DBMS_LOB.opt_compress));
DBMS_OUTPUT.put_line('Encryption : ' || DBMS_LOB.getoptions(l_clob, DBMS_LOB.opt_encrypt));
DBMS_OUTPUT.put_line('Deduplication: ' || DBMS_LOB.getoptions(l_clob, DBMS_LOB.opt_deduplicate));
ROLLBACK;
END;
/
DBMS_SPACE.SPACE_USAGE Procedures
This form of the procedure shows the space usage of data blocks under the segment High Water Mark. You can calculate usage for LOBs, LOB PARTITIONS and LOB SUBPARTITIONS. This procedure can only be used on tablespaces that are created with auto segment space management. The bitmap blocks, segment header, and extent map blocks are not accounted for by this procedure. Note that this overload cannot be used on SECUREFILE LOBs.
DBMS_SPACE.SPACE_USAGE(
segment_owner IN VARCHAR2,
segment_name IN VARCHAR2,
segment_type IN VARCHAR2,
unformatted_blocks OUT NUMBER,
unformatted_bytes OUT NUMBER,
fs1_blocks OUT NUMBER,
fs1_bytes OUT NUMBER,
fs2_blocks OUT NUMBER,
fs2_bytes OUT NUMBER,
fs3_blocks OUT NUMBER,
fs3_bytes OUT NUMBER,
fs4_blocks OUT NUMBER,
fs4_bytes OUT NUMBER,
full_blocks OUT NUMBER,
full_bytes OUT NUMBER,
partition_name IN VARCHAR2 DEFAULT NULL);
The second form of the procedure returns information about SECUREFILE LOB space usage. It will return the amount of space in blocks being used by all the SECUREFILE LOBs in the LOB segment. The procedure displays the space actively used by the LOB column, freed space that has retention expired, and freed space that has retention unexpired. Note that this overload can be used only on SECUREFILE LOBs.
DBMS_SPACE.SPACE_USAGE(
segment_owner IN VARCHAR2,
segment_name IN VARCHAR2,
segment_type IN VARCHAR2,
segment_size_blocks OUT NUMBER,
segment_size_bytes OUT NUMBER,
used_blocks OUT NUMBER,
used_bytes OUT NUMBER,
expired_blocks OUT NUMBER,
expired_bytes OUT NUMBER,
unexpired_blocks OUT NUMBER,
unexpired_bytes OUT NUMBER,
partition_name IN VARCHAR2 DEFAULT NULL);
SET SERVEROUTPUT ON
DECLARE
l_segment_size_blocks NUMBER;
l_segment_size_bytes NUMBER;
l_used_blocks NUMBER;
l_used_bytes NUMBER;
l_expired_blocks NUMBER;
l_expired_bytes NUMBER;
l_unexpired_blocks NUMBER;
l_unexpired_bytes NUMBER;
BEGIN
DBMS_SPACE.SPACE_USAGE(
segment_owner => 'TEST',
segment_name => 'SECUREFILE_LOB',
segment_type => 'LOB',
segment_size_blocks => l_segment_size_blocks,
segment_size_bytes => l_segment_size_bytes,
used_blocks => l_used_blocks,
used_bytes => l_used_bytes,
expired_blocks => l_expired_blocks,
expired_bytes => l_expired_bytes,
unexpired_blocks => l_unexpired_blocks,
unexpired_bytes => l_unexpired_bytes);
DBMS_OUTPUT.put_line('segment_size_blocks:' || l_segment_size_blocks);
DBMS_OUTPUT.put_line('segment_size_bytes :' || l_segment_size_bytes);
DBMS_OUTPUT.put_line('used_blocks :' || l_used_blocks);
DBMS_OUTPUT.put_line('used_bytes :' || l_used_bytes);
DBMS_OUTPUT.put_line('expired_blocks :' || l_expired_blocks);
DBMS_OUTPUT.put_line('expired_bytes :' || l_expired_bytes);
DBMS_OUTPUT.put_line('unexpired_blocks :' || l_unexpired_blocks);
DBMS_OUTPUT.put_line('unexpired_bytes :' || l_unexpired_bytes);
END;
/
Migrating to SecureFiles
You can use two basic methods to migrate to SecureFiles: partition exchange and online redefinition. Let’s see how you migrate to SecureFiles using both of these approaches. When you use partition exchange to migrate to SecureFiles, you have to make sure that a number of factors are taken care of such as ensuring you have enough space to accommodate the largest of the partitions of the table. The migration job takes a long time and therefore, you must have a long maintenance window to perform the migration. You must also maintain the indexes during the partition exchange. In addition, the table or partition must be offline in order for you to perform the partition exchange. Oracle recommends that you use the online redefinition method to migrate to SecureFiles. Online redefinition means you don’t take the table offline. You can perform the migration in parallel. You must rebuild global indexes and must have additional storage equal to the entire table. You can, however, cut back on the additional space requirements by performing the online redefinition on a partition basis.Here a simple example to demonstrate how to perform online redefinition to migrate to SecureFiles from a traditional BasicFiles LOB implementation:
Create a table using BasicFiles:
CREATE TABLE tab1(ID NUMBER NOT NULL, c CLOB)
lob(c) store as basicfile;
The store as lob clause creates a BasicFile LOB.
Insert some test data into the BasicFile-based table.
insert into tab1 values(1,'BasicFile Lob');
Create a new, interim table using SecureFiles, but with the same columns as the first table:
create table tab2(id number not null, c clob)
LOB(c) STORE AS securefile;
The store as securefile clause creates a SecureFile LOB.
Execute the DBMS_REDFINITION procedure to convert the BasicFile- based table into a SecureFile-based table:
DECLARE
L_ERROR_COUNT BINARY_INTEGER;
BEGIN
DBMS_REDEFINITION.START_REDEF_TABLE('SCOTT','TAB1','TAB2');
DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS('SCOTT','TAB1','TAB2',1,
TRUE,TRUE,TRUE,TRUE,L_ERROR_COUNT);
DBMS_REDEFINITION.FINISH_REDEF_TABLE('SCOTT','TAB1','TAB2');
END;
You can perform the redefinition in parallel to make it faster.
There is no automatic method for migrating a column from a BasicFile to a SecureFile LOB. Besides, you have following option to manually convert the data:
CREATE TABLE ... AS SELECT ...
INSERT INTO ... SELECT ...
Online table redefintion.
Export/Import
Create a new column, update the new column with the values in the original column, then drop the old column.
Create a new column, update the new column with the values in the original column, rename the table and create a view with the original name that only references the new column.
With the exception of the export/import method, all the options will require considerable amounts of disk space when converting LOBs containing large amounts of data.
Oracle Streams does not currently support SecureFiles, so avoid migrating LOBs that are using in conjunction with streams.
参考至: 《McGraw.Hill.OCP.Oracle.Database.11g.New.Features.for.Administrators.Exam.Guide.Apr.2008》
http://www.oracle-base.com/articles/11g/secure-files-11gr1.php
http://docs.oracle.com/cd/E11882_01/appdev.112/e40758/d_lob.htm#ARPLS66746
本文原创，转载请注明出处、作者
如有错误，欢迎指正