Tag a BigQuery table by using Data Catalog

This quickstart helps you complete the following tasks:

  1. Create a BigQuery dataset and table.

  2. Create a tag template with a schema that defines five tag fields of distinct types. These are string, double, boolean, enumerated, and richtext.

  3. Lookup the Data Catalog entry for your table.

  4. In the Google Cloud console, create business metadata for your entry that includes an overview, data steward, and a tag.

Data Catalog lets you search and tag entries such as BigQuery tables with metadata. Some examples of metadata that you can use for tagging include public and private tags, data stewards, and rich text overview.

Before you begin

  1. Set up your project.
    1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

    2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

      Go to project selector

    3. Enable the Data Catalog and BigQuery APIs.

      Roles required to enable APIs

      To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

      Enable the APIs

    4. Install the Google Cloud CLI.

    5. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    6. To initialize the gcloud CLI, run the following command: 

      gcloud init

    7. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

      Go to project selector

    8. Enable the Data Catalog and BigQuery APIs.

      Roles required to enable APIs

      To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

      Enable the APIs

    9. Install the Google Cloud CLI.

    10. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    11. To initialize the gcloud CLI, run the following command: 

      gcloud init

Add a public data entry to your project

Data Catalog entries include data resources such as a BigQuery dataset or a Pub/Sub topic.

  • Add a public dataset to your project.

    1. In Google Cloud console, go to the BigQuery page.

      Go to BigQuery

    2. In the Explorer section, click Add data and select Public datasets from the list.

    3. In the Marketplace panel, search for New York taxi trips and click the relevant search result.

    4. Click View Dataset.

Create a dataset and a table

  1. Create a dataset.

    1. In the Google Cloud console, open the BigQuery page.

      Go to BigQuery

    2. In the Explorer panel, select the project where you want to create the dataset.

    3. Click the Actions icon and click Create dataset.

    4. In the Create dataset page, fill in the following details:

      • For Dataset ID, enter demo_dataset.
      • For Data location, select us (multiple regions in United States).
      • Enable table expiration and specify the number of days.
      • For Encryption, leave the Google-managed encryption key option selected.

      • Click Create dataset.

  2. Copy a publicly accessible table to demo_dataset.

    1. In the Google Cloud console, open the BigQuery page.

      Go to BigQuery

    2. In the Explorer pane, search for tlc_yellow_trips tables (click Broaden search to all projects if required) and select one of them, such as tlc_yellow_trips_2017. Then click Copy.

    3. In the Copy table pane, fill in the following information:

      1. In the Project name drop-down list, select your project.
      2. In the Dataset name drop-down list, select demo_dataset.
      3. For the Table name, enter trips, then click Copy.

    4. In the Explorer pane, confirm that the trips table is listed in demo_dataset.

You add Data Catalog tags to the table in the next section.

Create a public tag template and attach a tag for your entry

You must be the dataset owner to attach a tag to a table in the dataset. For more information about public and private tags, see Public and private tags.

In a tag template, tag fields are optional. You don't have to provide a value for a field when attaching a tag to a Data Catalog entry. However, if a template defines a field as required, you must provide a value for the field. If the value is not provided, an error is generated.

You can use lower case letters and underscores to define field names. The tag template fields created in this example are demo fields and aren't auto-updated or synced with BigQuery.

Console

  1. Go to the Knowledge Catalog Tag Templates page.

    Go to Tag templates

  2. Click Create tag template and enter the following details:

    1. Enter the Template name as Demo Tag Template.
    2. Retain the default location.
    3. Retain the tag template visibility as Public.

    4. Click Add field to add 5 fields. Use the following table and keep Field description empty.