ambiente de teste
centos7
Para o guia de construção do mysql, consulte: https://lrting.top/backend/2111/
Instale dependências:
yum install gcc make flex bison byacc git
Baixe ferramentas TPC-DS
Endereço de download: https://www.tpc.org/tpc_documents_current_versions/download_programs/tools-download-request5.asp?bm_type=TPC-DS&bm_vers=3.2.0&mode=CURRENT-ONLY
Após o download, você receberá um pacote compactado com um nome de arquivo semelhante ao seguinte: 1d87cca3-7268-4f0a-8678-36564d0b45f9-tpc-ds-tool.zip, descompacte e compile
unzip 1d87cca3-7268-4f0a-8678-36564d0b45f9-tpc-ds-tool.zip
cd DSGen-software-code-3.2.0rc1/tools/
make
Geração e importação de dados
Crie um novo banco de dados tpcds no mysql
create database tpcds DEFAULT CHARSET utf8 COLLATE utf8_general_ci
Importe a estrutura da tabela para o banco de dados
mysql -u root -p -D tpcds < DSGen-software-code-3.2.0rc1/tools/tpcds.sql
Gere dados de teste, onde -SCALE 10 significa gerar dados 10G
mkdir /home/hadoop/data
cd DSGen-software-code-3.2.0rc1/tools/
./dsdgen -DIR /home/hadoop/data -SCALE 10 -TERMINATE N, tpcds.sql
importação de dados
LOAD DATA LOCAL INFILE '/home/hadoop/data/call_center.dat' INTO TABLE call_center FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
Erro relatado:
ERROR 1290 (HY000): The MySQL server is running with the --secure-file-priv option so it cannot execute this statement
Solução:
Em relação à questão do secure-file-priv, primeiro verifique o valor atual do secure-file-priv,
mostre variáveis como '%secure%';
se for NULL, significa que a restrição não pode importar e exportar dados e precisa ser modificado em mysql-5.7/my.cnf
Adicione secure_file_priv='' a e
reinicie o mysql
Reimportar, importar SQL
LOAD DATA LOCAL INFILE '/home/hadoop/data/call_center.dat' INTO TABLE call_center FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/customer.dat' INTO TABLE customer FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/income_band.dat' INTO TABLE income_band FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/ship_mode.dat' INTO TABLE ship_mode FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/warehouse.dat' INTO TABLE warehouse FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/catalog_page.dat' INTO TABLE catalog_page FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/customer_demographics.dat' INTO TABLE customer_demographics FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/inventory.dat' INTO TABLE inventory FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/store.dat' INTO TABLE store FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/web_page.dat' INTO TABLE web_page FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/catalog_returns.dat' INTO TABLE catalog_returns FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/date_dim.dat' INTO TABLE date_dim FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/item.dat' INTO TABLE item FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/store_returns.dat' INTO TABLE store_returns FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/web_returns.dat' INTO TABLE web_returns FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/catalog_sales.dat' INTO TABLE catalog_sales FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/dbgen_version.dat' INTO TABLE dbgen_version FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/promotion.dat' INTO TABLE promotion FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/store_sales.dat' INTO TABLE store_sales FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/web_sales.dat' INTO TABLE web_sales FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/customer_address.dat' INTO TABLE customer_address FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/household_demographics.dat' INTO TABLE household_demographics FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/reason.dat' INTO TABLE reason FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/time_dim.dat' INTO TABLE time_dim FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';
LOAD DATA LOCAL INFILE '/home/hadoop/data/web_site.dat' INTO TABLE web_site FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n';