Troubleshoot

You can use this document to resolve common issues when working with Cloud Trace, such as missing span data, query failures in Observability Analytics, and validation errors when creating alerting policies.

Known issues

This section lists known issues:

Spans written to your Google Cloud project by using the Telemetry API aren't accessible to the Cloud Trace API. For example, if you try to list these traces, then the command fails with a 404 Not Found error.

Troubleshoot Observability Analytics

This section describes how to resolve failures you might see when using Observability Analytics to query your trace data.

You can't save your alerting policy because of a validation error

You try to save an alerting policy that monitors your trace data and receive an error similar to the following:

The following error occurred when validating your SQL Alert: Error authenticating service account `service-12345@gcp-sa-monitoring-notification.iam.gserviceaccount.com`. BigQuery returned an error.

This error message indicates that the Monitoring Service Account hasn't been granted the required permissions or that it doesn't exist. This account is automatically created by the system when certain user-initiated actions occur. However, if the Cloud Monitoring API is disabled, then the system can't create the service account.

To resolve the failure, do the following:

In the Google Cloud console, go to the APIs & Services page and enable the Cloud Monitoring API:
Go to APIs & Services
In the Google Cloud console, go to the IAM page:
Go to IAM

If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
On the IAM page, do the following:
1. Select Include Google-provided role grants.
2. If the Monitoring Service Account isn't listed, then create a SQL-based alerting policy and try to save the policy.
  
  When you save the policy, the system creates the Monitoring Service Account. The save action fails because this service account doesn't have the required IAM roles.
3. Grant the Monitoring Service Account the following roles:
  - Monitoring Service Agent (roles/monitoring.notificationServiceAgent) on your project.
  - BigQuery Data Viewer (roles/bigquery.dataViewer) on your linked BigQuery dataset.
  Note: We recommend that you restrict the BigQuery Data Viewer role grant to the linked BigQuery dataset for your trace data. Don't grant this role at the project level.

Error message stating a view does not exist

You enter a SQL query in the query pane of the Observability Analytics page, but the SQL parser displays the following error:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/views/OBS_VIEW_ID does not exist

The previous error is reported when the view specified in the FROM statement can't be found.

To resolve this error, verify that your view has the proper syntax:

Verify that the fully-qualified name of the view follows the syntax required by the Observability Analytics naming scheme. You can find the required syntax for a view by displaying its default query.
If the Google Cloud project ID, location, bucket ID, dataset ID, or view ID contains period characters, (.), then verify that the field is wrapped by single backquotes, (`).

For example, if the ID of your Google Cloud project is example.com:bluebird, then the FROM statement is as follows:
```
FROM `example.com:bluebird`.`us`.`_Trace`.`Spans`.`_AllSpans`
```

Get started with Observability Analytics message is shown

You open the Observability Analytics page and it displays a window with a message similar to the following:

Get started with Observability Analytics

To use Observability Analytics, on the window, click Close.

The previous message is shown when you don't have any log buckets that are upgraded to use Observability Analytics. However, your trace data isn't stored in a log bucket.

Join of multiple views fails

You write a query that joins multiple views, but the query is marked as invalid.

Not all views can be joined.

To join views, the following restrictions apply:

The locations of the views satisfy one of the following:
- All views have the same location.
- All views are in either the global or us location.
When storage resources use customer managed encryption keys (CMEK), one of the following is be true:
- Storage resources that use CMEK use the same Cloud KMS key.
- Storage resources that use CMEK have a common ancestor, and that ancestor specifies a default Cloud KMS key that is in the same location as the storage resources.
When one or more storage resources uses CMEK, the system encrypts temporary data generated by the join with either the common Cloud KMS key or the ancestor's default Cloud KMS key.

For example, suppose that you have two views that reside in the same location. Then you can join these views when any of the following are true:

The storage resources don't use CMEK.
One storage resource uses CMEK and the other doesn't.
Both storage resources use CMEK and both use the same Cloud KMS key.
Both storage resources use CMEK but they use different keys. However, the resources share an ancestor that specifies a default Cloud KMS key that is in the same location as the storage resources.

For example, suppose that the resource hierarchy for a log bucket and observability bucket includes the same organization. You can join views on those buckets when, for that organization, you've configured the default resource settings for Cloud Logging and for observability buckets with the same default Cloud KMS key for the storage location.

Creating a linked BigQuery dataset fails with a permission error

You attempt to create a linked BigQuery dataset, but the operation fails with an error similar to the following:

ERROR: (gcloud.beta.observability.buckets.datasets.links.create) {
  "code": 7,
  "message": "The caller does not have permission"
}

To resolve this issue, do the following:

Make sure that you have been granted the required IAM roles. For a list of these roles, see Create a link on a dataset.
Review your organization's policies to determine whether there are constraints that apply to BigQuery datasets. Suppose that you create a custom constraint that requires BigQuery datasets to be in a specific location. For this case, you can only create a linked BigQuery dataset on an observability dataset that is in that specific location.

Querying a linked BigQuery dataset fails with dataset not found

You issue a query to a linked BigQuery dataset and the command fails with an error message similar to "Dataset not found".

To resolve this problem, try the following:

Make sure that the path to the dataset name is correct.
Run the following gcloud beta observability buckets datasets links list command and verify that the response lists your linked BigQuery dataset.

If your linked BigQuery dataset isn't listed, then the dataset is orphaned. To resolve this scenario, delete the linked BigQuery dataset.

A project can have an orphaned linked BigQuery dataset if you revoke IAM role grants from a service account that Google Cloud Observability uses to manage data and then delete a link on an observability dataset. In that scenario, Google Cloud Observability can't delete the linked BigQuery dataset.

No data in the Trace Explorer page

You have an application that is sending trace data to your Google Cloud project. However, when you open the Trace Explorer page, no data is shown.

There are several possible reasons why you can't view trace data:

You aren't granted the permissions required to view the data.
Trace spans weren't sent to your project.
Your application doesn't have the permissions required to write trace data.
Your trace spans aren't being stored.

The following subsections provide information about how to troubleshoot the listed failure scenarios.

Verify that you have permission to view trace data

To view trace data, make sure that you have been granted the Cloud Trace User role (roles/cloudtrace.user).

Verify that trace spans are sent to your project

To verify that spans are being sent to your project, do the following:

Enable the Cloud Trace and Telemetry APIs.