bigtable_rs

KO | EN

gitlite — search

#python #ai #typescript #go #golang #android #machine-learning #rust #macos #nodejs #kubernetes #pytorch #security #mcp

(★ 26)

A simple Rust library for working with Google Bigtable Data and Admin API V2

bigtable_rs Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# bigtable_rs

A simple Rust library for working
with [Google Bigtable](https://cloud.google.com/bigtable/docs/) [Data API V2](https://cloud.google.com/bigtable/docs/reference/data/rpc/google.bigtable.v2)
.

[![ci_badge](https://github.com/liufuyang/bigtable_rs/actions/workflows/CI.yaml/badge.svg)](https://github.com/liufuyang/bigtable_rs/actions/workflows/CI.yaml)
[![Crates.io](https://img.shields.io/crates/v/bigtable_rs)](https://crates.io/crates/bigtable_rs)
[![Documentation](https://docs.rs/bigtable_rs/badge.svg)](https://docs.rs/bigtable_rs)
[![codecov](https://codecov.io/github/liufuyang/bigtable_rs/graph/badge.svg?token=U9LTK0V10S)](https://codecov.io/github/liufuyang/bigtable_rs)
[![Crates.io](https://img.shields.io/crates/l/bigtable_rs)](https://raw.githubusercontent.com/liufuyang/bigtable_rs/main/LICENSE)

## Disclaimer - might be ready for production use

This library might be production ready. Any contribution or help is highly appreciated.

I had the idea to build a client like this and got some input from others
see [here](https://github.com/durch/rust-bigtable/issues/4)
and porting code from [here](
https://github.com/solana-labs/solana/tree/master/storage-bigtable)
(thanks to [@mvines](https://github.com/mvines)).

Code
for [read_rows parsing](https://github.com/liufuyang/bigtable_rs/blob/main/bigtable_rs/src/bigtable/read_rows.rs#L36-L212)
logic
is [tested](https://github.com/liufuyang/bigtable_rs/blob/main/bigtable_rs/tests/read_rows/read_rows_test.rs)
by [porting](https://github.com/liufuyang/bigtable_rs/blob/main/bigtable_rs/tests/read_rows/read_rows_test.json)
Google's Java client
[tests code](https://github.com/googleapis/java-bigtable/blob/main/google-cloud-bigtable/src/test/java/com/google/cloud/bigtable/data/v2/stub/readrows/ReadRowsMergingAcceptanceTest.java)
with [json test cases](https://github.com/googleapis/conformance-tests/blob/main/bigtable/v2/readrows.json)
as raw
input.

## Introduction

Current idea is to make this library very light weighted, and you assemble requests based
on [Google Bigtable V2 protobuf schema](https://github.com/googleapis/googleapis/blob/master/google/bigtable/v2/bigtable.proto)
and send the requests via [tonic gRPC over HTTP/2](https://github.com/hyperium/tonic). So the user
have the flexibility
of creating any type of Bigtable request and use this client to talk to Bigtable service.

Compiled Bigtable API proto as Rust code is also included in the repo here so users don't need to
compile from proto
again.

The returned row values from Bigtable is parsed by this library.

See more examples on how to use:

* [ReadRows](examples/src/simple_read.rs)
* [MutateRows](examples/src/simple_write.rs)
* [SampleRowKeys](examples/src/sample_row_keys.rs)
* [Read with a prefix](examples/src/prefix.rs)

For other gRPC APIs/methods, one should be able to use the gRCP client directly and assemble any
customized request you
need to
interact with Bigtable service via building the Protobuf messages (already complied as rs files and
included here).
See [this example](./examples/src/custom_query.rs).

[gcp_auth](https://github.com/hrvolapeter/gcp_auth) is used, which
supports:

* Application Default Credentials
* Connection authenticated via Google service account key `json` file
  (by setting `GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json` environment parameter)
* Default service account by retrieving a token from the gcloud metadata server

Depends on [googleapis_tonic_google_bigtable_v2](https://crates.io/crates/googleapis_tonic_google_bigtable_v2) - which is a crate
contains all the Bigtable Rust types generated from
googleapis Bigtable protos by using tonic build.

You can use the library as follows:

```toml
[dependencies]
bigtable_rs = "0.4.0"
tokio = { version = "1.0", features = ["rt-multi-thread"] }
googleapis_tonic_google_bigtable_v2 = "0.36.0"
env_logger = "0.11.1"
```

Documentation is on [crate.io](https://docs.rs/bigtable_rs/0.1.3/bigtable_rs/).
See [examples](./examples) folders for more examples. The following example showing how to do a key
range scan

```rust
use bigtable_rs::bigtable;
use googleapis_tonic_google_bigtable_v2::google::bigtable::v2::row_filter::{Chain, Filter};
use googleapis_tonic_google_bigtable_v2::google::bigtable::v2::row_range::{EndKey, StartKey};
use googleapis_tonic_google_bigtable_v2::google::bigtable::v2::{ReadRowsRequest, RowFilter, RowRange, RowSet};
use env_logger;
use std::error::Error;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    env_logger::init();

let project_id = "project-id";
    let instance_name = "instance-1";
    let table_name = "table-1";
    let channel_size = 4;
    let timeout = Duration::from_secs(10);

let key_start: String = "key1".to_owned();
    let key_end: String = "key4".to_owned();

// make a bigtable client
    let connection = bigtable::BigTableConnection::new(
        project_id,
        instance_name,
        true,
        channel_size,
        Some(timeout),
    )
        .await?;
    let mut bigtable = connection.client();

// prepare a ReadRowsRequest
    let request = ReadRowsRequest {
        table_name: bigtable.get_full_table_name(table_name),
        rows_limit: 10,
        rows: Some(RowSet {
            row_keys: vec![], // use this field to put keys for reading specific rows
            row_ranges: vec![RowRange {
                start_key: Some(StartKey::StartKeyClosed(key_start.into_bytes())),
                end_key: Some(EndKey::EndKeyOpen(key_end.into_bytes())),
            }],
        }),
        filter: Some(RowFilter {
            filter: Some(Filter::Chain(Chain {
                filters: vec![
                    RowFilter {
                        filter: Some(Filter::FamilyNameRegexFilter("cf1".to_owned())),
                    },
                    RowFilter {
                        filter: Some(Filter::ColumnQualifierRegexFilter("c1".as_bytes().to_vec())),
                    },
                    RowFilter {
                        filter: Some(Filter::CellsPerColumnLimitFilter(2)),
                    },
                ],
            })),
        }),
        ..ReadRowsRequest::default()
    };

// calling bigtable API to get results
    let response = bigtable.read_rows(request).await?;

// simply print results for example usage
    response.into_iter().for_each(|(key, data)| {
        println!("------------\n{}", String::from_utf8(key.clone()).unwrap());
        data.into_iter().for_each(|row_cell| {
            println!(
                "    [{}:{}] \"{}\" @ {}",
                row_cell.family_name,
                String::from_utf8(row_cell.qualifier).unwrap(),
                String::from_utf8(row_cell.value).unwrap(),
                row_cell.timestamp_micros
            )
        })
    });

Ok(())
}

```

---

## Play with example code

To start develop or test the example code above,
install [cbt](https://cloud.google.com/bigtable/docs/cbt-overview) tool if you haven't,
then start a test Bigtable instance locally and insert some test data like this:

```
# at one terminal, start a bigtable insatnce locally
. start_bigtable_local.sh

# at another terminal, load some data into it
. start_load_table_local.sh
```

Then run an example with this command:

```
BIGTABLE_EMULATOR_HOST=localhost:8086 RUST_LOG=bigtable_rs=trace cargo run --bin simple_read
```

If you see error related to google authentication or gcp project not found,
then remember to set environment parameter `BIGTABLE_EMULATOR_HOST=localhost:8086`
so to let the example's client connect onto the local emulator.

To run it against real Bigtable instance:

```
GOOGLE_APPLICATION_CREDENTIALS=<path_to_key>/service_account_key.json cargo run --bin simple_read
```

Or if you want to use your `gcloud auth login` auth then simply run the client without any special
env settings:

```
cargo run --bin simple_read
```

See `examples` folders for more examples.

## Development

Clone this repo (and checkout the submodules if rebuilding of the
Googleapis proto is needed), and run `cargo build`.

```
cargo build

# Or set BUILD_BIGTABLE_RS_GOOGLE_PROTO=true to enable rebuild of google protos
# BUILD_BIGTABLE_RS_GOOGLE_PROTO=true cargo build
```

See `bigtable_rs/src/build.rs` for more details.

Running tests:

```
cargo test -- --nocapture

# Or run with integration tests (requires a bigtable emulator running)

BIGTABLE_EMULATOR_HOST=localhost:8086 cargo test --features integration_tests
```

See code coverage:

```
rustup component add llvm-tools-preview
cargo install cargo-llvm-cov

cargo llvm-cov --no-cfg-coverage  --ignore-filename-regex "google/" --html --open

# Or run with integration test (requires a bigtable emulator running)

BIGTABLE_EMULATOR_HOST=localhost:8086 cargo llvm-cov --no-cfg-coverage  --ignore-filename-regex "google/" --features integration_tests --html --open
```

bigtable_rs

bigtable_rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

bigtable_rs

bigtable_rs Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README