We are currently refactoring our documentation. Please excuse any problems you may find and report them here.

In this guide, you’ll learn how to quickly get a tiny TiKV cluster running locally, then you’ll use our Rust client to get, set, and scan data in TiKV. Then you’ll learn how to quickly start and stop a TiKV cluster to accompany your development environment.

This guide won’t worry about details such as security, resiliency, or production-readiness. (Those topics are covered more in detail in the deploy guides.) Nor will this guide cover how to develop TiKV itself (See CONTRIBUTING.md.) Instead, this guide focuses on an easy development experience and low resource consumption.

Overview

In order to get a functioning TiKV service you will need to start a TiKV service and a PD service. PD works alongside TiKV to act as a coordinator and timestamp oracle.

Communication between TiKV, PD, and any services which use TiKV is done via gRPC. We provide clients for several languages, and this guide will briefly show you how to use the Rust client.

Using Docker, you’ll create pair of persistent services tikv and pd and learn to manage them easily. Then you’ll write a simple Rust client application and run it from your local host. Finally, you’ll learn how to quicky teardown and bring up the services, and review some basic limitations of this configuration.

Architecture

In a production deployment there would be at least three TiKV services and three PD services spread among 6 machines. Most deployments also include kernel tuning, sysctl tuning, robust systemd services, firewalls, monitoring with prometheus, grafana dashboards, log collection, and more. Even still, to be sure of your resilience and security, consider consulting our maintainers.

If you are interested in deploying for production we suggest investigating the deploy guides.

While it’s possible to use TiKV through a query layer, like TiDB or Titan, you should refer to the user guides of those projects in order to set up test clusters. This guide only deals with TiKV, PD, and TiKV clients.

Prerequisites

This guide assumes you have the following knowledge and tools at your disposal:

  • Working knowledge about Docker (e.g. how to run or stop a container),
  • A modern Docker daemon which can support docker stack and the Compose File 3.7 version, running on a machine with:
    • A modern (circa >2012) x86 64-bit processor (supporting SSE4.2)
    • At least 10 GB of free storage space
    • A modest amount of memory (4+ GB) available
    • A ulimit.nofile value higher than 82920 for the docker service

While this guide was written with Linux in mind, you can use any operating system as long as the Docker service is able to run Linux containers. You may need to make small adaptations to commands to suite your operating system, let us know if you get stuck so we can fix it!

Starting the stack

The maintainers from PingCAP publish battle-tested release images of both pd and tikv on Docker Hub. These are used in their TiDB Cloud kubernetes clusters as well as opt-in via their tidb-ansible project.

For a TiKV client to interact with a TiKV cluster, it needs to be able to reach each PD and TiKV node. Since TiKV balances and replicates data across all nodes, and any node may be in charge of any particular Region of data, your client needs to be able to reach every node involved. (Replicas of your clients do not need to be able to reach each other.)

In the interest of making sure this guide can work for all platforms, it uses docker stack to deploy an ultra-minimal cluster that you can quickly tear down and bring back up again. This cluster won’t feature security, persistence, or have static hostnames.

Unless you’ve tried using docker stack before, you may need to run docker swarm init. If you’re unsure, it’s best just to run it and ignore the error if you see one.

To begin, create a stack.yml:

  1. version: "3.7"
  3. x-defaults: &defaults
  4.     init: true
  5.     volumes:
  6.         - ./entrypoints:/entrypoints
  7.     environment:
  8.         SLOT: "{{.Task.Slot}}"
  9.         NAME: "{{.Task.Name}}"
  10.     entrypoint: /bin/sh
  11.     deploy:
  12.         replicas: 1
  13.         restart_policy:
  14.             condition: on-failure
  15.             delay: 5s
  17. services:
  18.     pd:
  19.         <<: *defaults
  20.         image: pingcap/pd
  21.         hostname: "{{.Task.Name}}.tikv"
  22.         init: true
  23.         networks:
  24.             tikv:
  25.                 aliases:
  26.                     - pd.tikv
  27.         ports:
  28.             - "2379:2379"
  29.             - "2380:2380"
  30.         command: /entrypoints/pd.sh
  31.     tikv:
  32.         <<: *defaults
  33.         image: pingcap/tikv
  34.         hostname: "{{.Task.Name}}.tikv"
  36.         networks:
  37.             tikv:
  38.                 aliases:
  39.                     - tikv.tikv
  40.         ports:
  41.             - "20160:20160"
  42.         command: /entrypoints/tikv.sh
  44. networks:
  45.     tikv:
  46.         name: "tikv"
  47.         driver: "overlay"
  48.         attachable: true

Then create entrypoints/pd.sh with the following:

  1. #! /bin/sh
  2. set -e
  4. if [ $SLOT = 1 ]; then 
  5.     exec ./pd-server \
  6.         --name $NAME \
  7.         --client-urls http://0.0.0.0:2379 \
  8.         --peer-urls http://0.0.0.0:2380 \
  9.         --advertise-client-urls http://`cat /etc/hostname`:2379 \
  10.         --advertise-peer-urls http://`cat /etc/hostname`:2380
  11. else
  12.     exec ./pd-server \
  13.         --name $NAME \
  14.         --client-urls http://0.0.0.0:2379 \
  15.         --peer-urls http://0.0.0.0:2380 \
  16.         --advertise-client-urls http://`cat /etc/hostname`:2379 \
  17.         --advertise-peer-urls http://`cat /etc/hostname`:2380 \
  18.         --join http://pd.tikv:2379
  19. fi

Last, an entrypoints/tikv.sh with the following:

  1. #!/bin/sh
  2. set -e
  4. exec ./tikv-server \
  5.     --addr 0.0.0.0:20160 \
  6.     --status-addr 0.0.0.0:20180 \
  7.     --advertise-addr `cat /etc/hostname`:20160 \
  8.     --pd-endpoints pd.tikv:2379

Next, you can deploy the stack to Docker:

  1. docker stack deploy --compose-file stack.yml tikv

The output should look like this:

  1. Creating network tikv
  2. Creating service tikv_pd
  3. Creating service tikv_tikv

Managing services

Check the state of running services:

  1. $ docker service ls
  2. ID                  NAME                MODE                REPLICAS            IMAGE                 PORTS
  3. 6ia0pefrd811        tikv_pd             replicated          1/1                 pingcap/pd:latest     *:2379-2380->2379-2380/tcp
  4. 26u77puqmw4d        tikv_tikv           replicated          1/1                 pingcap/tikv:latest   *:20160->20160/tcp

Turn off the services:

This will delete data!

  1. $ docker service scale tikv_pd=0 tikv_tikv=0
  2. tikv_pd scaled to 0
  3. tikv_tikv scaled to 0
  4. overall progress: 0 out of 0 tasks 
  5. verify: Service converged 
  6. overall progress: 0 out of 0 tasks 
  7. verify: Service converged

Turn services back on:

This creates brand new containers!

  1. $ docker service scale tikv_pd=1 tikv_tikv=1
  2. tikv_pd scaled to 1
  3. tikv_tikv scaled to 1
  4. overall progress: 1 out of 1 tasks 
  5. 1/1: running   [==================================================>] 
  6. verify: Service converged 
  7. overall progress: 1 out of 1 tasks 
  8. 1/1: running   [==================================================>] 
  9. verify: Service converged

Inquire into the metrics:

Normally, these would be pulled by Prometheus, but it is human readable and functions as a basic liveliness test.

  1. $ docker run --rm -ti --network tikv alpine sh -c "apk add curl; curl http://pd.tikv:2379/metrics"
  2. # A lot of output...
  3. $ docker run --rm -ti --network tikv alpine sh -c "apk add curl; curl http://tikv.tikv:20180/metrics"
  4. # A lot of output...

Inquire into the resource consuption of the containers:

  1. $ docker stats
  2. CONTAINER ID        NAME                                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
  3. c4360f65ded3        tikv_tikv.1.a8sfm113yotkkv5klqtz5cvrn   0.36%               689MiB / 30.29GiB     2.22%               8.44kB / 7.42kB     0B / 0B             66
  4. 3f18cc8f415b        tikv_pd.1.r58jn3kolaxgqdbyb8w2mcx8r     1.56%               22.21MiB / 30.29GiB   0.07%               8.11kB / 7.75kB     0B / 0B             21

Remove the stack entirely:

This will delete data!

  1. $ docker stack rm tikv

Creating a project

Below, you’ll use the Rust client, but you are welcome to use any TiKV client.

Because you will eventually need to deploy the binary into the same network as the PD and TiKV nodes,

You can create a new example project then change into the directory:

  1. cargo new tikv-example
  2. cd tikv-example

You will need to use a nightly toolchain that supports the async/await feature in order to use the TiKV client in the guide below. This is expected to become stable in Rust 1.38.0.

For now, please echo 'nightly-2019-08-25' > ./rust-toolchain in the project directory.

Next, you’ll need to add the TiKV client as a dependency in the Cargo.toml file:

  1. [dependencies]
  2. tikv-client = { git = "https://github.com/tikv/client-rust.git" }
  3. tokio = "0.2.0-alpha.4"

Then you can edit the src/main.rs file with the following:

  1. use tikv_client::{Config, RawClient, Error};
  3. #[tokio::main]
  4. async fn main() -> Result<(), Error> {
  5.     let config = Config::new(vec!["http://pd.tikv:2379"]);
  6.     let client = RawClient::new(config)?;
  7.     let key = "TiKV".as_bytes().to_owned();
  8.     let value = "Works!".as_bytes().to_owned();
  10.     client.put(key.clone(), value.clone()).await?;
  11.     println!(
  12.         "Put: {} => {}",
  13.         std::str::from_utf8(&key).unwrap(),
  14.         std::str::from_utf8(&value).unwrap()
  15.     );
  17.     let returned: Vec<u8> = client.get(key.clone()).await?
  18.         .expect("Value should be present.").into();
  19.     assert_eq!(returned, value);
  20.     println!(
  21.         "Get: {} => {}",
  22.         std::str::from_utf8(&key).unwrap(),
  23.         std::str::from_utf8(&value).unwrap()
  24.     );
  25.     Ok(())
  26. }

TiKV works with binary data to enable your project to store arbitrary data such as binaries or non-UTF-8 encoded data if necessary. While the Rust client accepts String values as well as Vec<u8>, it will only output Vec<u8>.

Now, because the client needs to be part of the same network (tikv) as the PD and TiKV nodes, you must to build this binary into a Docker container. Create a Dockerfile in the root of your project directory with the following content:

  1. FROM ubuntu:latest
  2. # Systemwide setup
  3. RUN apt update
  4. RUN apt install --yes build-essential protobuf-compiler curl cmake golang
  6. # Create the non-root user.
  7. RUN useradd builder -m -b /
  8. USER builder
  9. RUN mkdir -p ~/build/src
  11. # Install Rust
  12. COPY rust-toolchain /builder/build/
  13. RUN curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain `cat /builder/build/rust-toolchain` -y
  14. ENV PATH="/builder/.cargo/bin:${PATH}"
  16. # Fetch, then prebuild all deps
  17. COPY Cargo.toml rust-toolchain /builder/build/
  18. RUN echo "fn main() {}" > /builder/build/src/main.rs
  19. WORKDIR /builder/build
  20. RUN cargo fetch
  21. RUN cargo build --release
  22. COPY src /builder/build/src
  23. RUN rm -rf ./target/release/.fingerprint/tikv-example*
  25. # Actually build the binary
  26. RUN cargo build --release
  27. ENTRYPOINT /builder/build/target/release/tikv-example

Next, build the image:

  1. docker build -t tikv-example .

Then start the produced image:

  1. docker run -ti --rm --network tikv tikv-example

At this point, you’re ready to start developing against TiKV!

Want to keep reading? You can explore Deep Dive TiKV to learn more about how TiKV works at a technical level.

Want to improve your Rust abilities? Some of our contributors work on creating Practical Network Applications, an open self guided study to master Rust while making fun distributed systems.