[ Google - GCP ] Solution: BigQuery error, Invalid uri specification. Option uri value must be a wild card URI

Problem Description

During the process of exporting data to the GS bucket using BigQuery, the database warehouse of Google Cloud, an error occurred:

Invalid uri specification. Option ‘uri’ value must be a wild card URI.

The BigQuery statements executed are as follows:

EXPORT DATA OPTIONS(
  uri='gs://test_bucket/GENERATEDB_DATA.csv',
  format='CSV',
  overwrite=true,
  header=true,
  field_delimiter=',') AS
SELECT * FROM `test.test_table`
limit 50;

After execution, an error message will appear in the command window, as shown in the figure below:

Insert image description here

Cause Analysis:

As can be seen from the error message, the cause of the error is that the URI is incorrect. The final solution is to modify the path of the URI. So why is it incorrect? Here is an explanation.

First, the normal official export data statement in GCP is as follows:

EXPORT DATA OPTIONS(
  uri='gs://bucket/folder/*.csv',
  format='CSV',
  overwrite=true,
  header=true,
  field_delimiter=';') AS
SELECT field1, field2 FROM mydataset.table1 ORDER BY field1 LIMIT 10

Note that the value of the uri above is a fixed writing method gs:// + bucket的名字 + 文件夹（可选） + 通配的csv文件名. The EXPORT DATA syntax does not support direct writing of a single file for export. For official instructions click here .

solution:

Modify the EXPORT DATA syntax and change uri to a wildcard expression. The code is as follows:

EXPORT DATA OPTIONS(
  uri='gs://test_bucket/GENERATEDB_DATA_*.csv',
  format='CSV',
  overwrite=true,
  header=true,
  field_delimiter=',') AS
SELECT * FROM `test.test_table`
limit 50;

After the final execution is successful again, the exported file can be seen in the GS bucket:

Insert image description here

Added : BigQuery supports a single wildcard operator (*) in each URI. Wildcards can appear anywhere in the URI but cannot be included in the bucket name. Use the wildcard operator to instruct BigQuery to create multiple shard files based on the provided pattern. The wildcard operator replaces a number (starting at 0) with 12 bits padded to the left. For example, if a URI has a wildcard character at the end of its file name, then when the files are created, the first file will have 000000000000 appended to the end of its name, the second file will have 000000000001 appended to its name, and so on.

[ 本文作者 ]   bluetata
[ 原文链接 ]   https://bluetata.blog.csdn.net/article/details/135508108
[ 最后更新 ]   01/10/2024 18:16
[ 版权声明 ]   如果您在非 CSDN 网站内看到这一行，
说明网络爬虫可能在本人还没有完整发布的时候就抓走了我的文章，
可能导致内容不完整，请去上述的原文链接查看原文。