- BIGQUERY LOAD JSON UPDATE
- BIGQUERY LOAD JSON CODE
Replace destination dataset location accordingly.Replace GCS object URL to your own one.Replace BigQuery destination table name.
BIGQUERY LOAD JSON UPDATE
Please remember to update the following parts:
BIGQUERY LOAD JSON CODE
The code snippet is from the official tutorial of Google Cloud BigQuery documentation with a few changes. Print("Loaded rows.".format(destination_table.num_rows)) Load_job.result() # Waits for the job to complete.ĭestination_table = client.get_table(table_id) Location=" australia-southeast1", # Must match the destination dataset location. Uri = "gs:// YOUR_BUCKET_NAME/json_lines.jsonl" Create a Python script to load the fileĬreate a Python script file (gcs-to-bq.py) with the following content in Cloud Shell: from google.cloud import bigquery InfoEnsure the region are the same for the bucket and BigQuery.
Run the following bq command to test access to BigQuery:. The command should display all the bucket list. Run the following command to test access to Cloud Storage:. Remember to update the service account and key file to your own ones. Gcloud auth activate-service-account -key-file PATH_TO_THE_UPLOADED_CREDENTIAL_FILE.json Activate the service account in Cloud Shell so that we can verify the access:. This is important as the credential file will be used directly by the Python client functions though we can also explicitly specify the credential file path. Remember to update the credential file path to your own. Run the following command to edit bash profile:Įxport GOOGLE_APPLICATION_CREDENTIALS="PATH_TO_THE_UPLOADED_CREDENTIAL_FILE.json". 
Upload the client credential file to Cloud Shell. 
For simplicity ( not best practice), I am adding BigQuery Admin and Storage Admin role to my service account. For this tutorial, you only need to assign read access to GCS and read and write access to BigQuery (,, ). You don't necessarily assign project owner to the service account.
Follow article Quickstart: Using client libraries | BigQuery | Google Cloud to configure a service account and download client JSON credential file. Initialize client credentialįollow these steps to setup client credential. InfoFor simplicity, the Python script used in this article is run in Cloud Shell (Google Cloud Terminal) environment.
0 kommentar(er)
