Bigtable HBase Beam connector

To help you use Bigtable in a Dataflow pipeline, two open source Bigtable Beam I/O connectors are available.

If you are migrating from HBase to Bigtable or your application calls the HBase API, use the Bigtable HBase Beam connector (CloudBigtableIO) discussed on this page.

In all other cases, you should use the Bigtable Beam connector (BigtableIO) in conjunction with the Cloud Bigtable client for Java, which works with the Cloud Bigtable APIs. To get started using that connector, see Bigtable Beam connector.

For more information on the Apache Beam programming model, see the Beam documentation.

Get started with HBase

The Bigtable HBase Beam connector is written in Java and is built on the Bigtable HBase client for Java. It's compatible with the Dataflow SDK 2.x for Java, which is based on Apache Beam. The connector's source code is on GitHub in the repository googleapis/java-bigtable-hbase.

This page provides an overview of how to use Read and Write transforms.

Set up authentication

To use the Java samples on this page in a local development environment, install and initialize the gcloud CLI, and then set up Application Default Credentials with your user credentials.

Install the Google Cloud CLI.
If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.
If you're using a local shell, then create local authentication credentials for your user account:
```
gcloud auth application-default login
```
You don't need to do this if you're using Cloud Shell.

If an authentication error is returned, and you are using an external identity provider (IdP), confirm that you have signed in to the gcloud CLI with your federated identity.

For more information, see Set up authentication for a local development environment.

For information about setting up authentication for a production environment, see Set up Application Default Credentials for code running on Google Cloud .

Add the connector to a Maven project

To add the Bigtable HBase Beam connector to a Maven project, add the Maven artifact to your pom.xml file as a dependency:

<dependency>
  <groupId>com.google.cloud.bigtable</groupId>
  <artifactId>bigtable-hbase-beam</artifactId>
  <version>2.12.0</version>
</dependency>

Specify the Bigtable configuration

Create an options interface to allow inputs for running your pipeline:

public interface BigtableOptions extends DataflowPipelineOptions {

  @Description("The Bigtable project ID, this can be different than your Dataflow project")
  @Default.String("bigtable-project")
  String getBigtableProjectId();

  void setBigtableProjectId(String bigtableProjectId);

  @Description("The Bigtable instance ID")
  @Default.String("bigtable-instance")
  String getBigtableInstanceId();

  void setBigtableInstanceId(String bigtableInstanceId);

  @Description("The Bigtable table ID in the instance.")
  @Default.String("mobile-time-series")
  String getBigtableTableId();

  void setBigtableTableId(String bigtableTableId);
}

When you read from or write to Bigtable, you must provide a CloudBigtableConfiguration configuration object. This object specifies the project ID and instance ID for your table, as well as the name of the table itself:

CloudBigtableTableConfiguration bigtableTableConfig =
    new CloudBigtableTableConfiguration.Builder()
        .withProjectId(options.getBigtableProjectId())
        .withInstanceId(options.getBigtableInstanceId())
        .withTableId(options.getBigtableTableId())
        .build();

For reading, provide a CloudBigtableScanConfiguration configuration object, which lets you specify an Apache HBase Scan object that limits and filters the results of a read. See Reading from Bigtable for details.

Read from Bigtable

To read from a Bigtable table, you apply a Read transform to the result of a