Installation and Use

Compile

Environmental dependence

  1. CentOS, Ubuntu and Mac OS are all OK (recommend CentOS >= 7.2).

  2. go >= 1.19 required.

  3. gcc >= 5 required.(recommend gcc >= 9 if want to use scann)

  4. cmake >= 3.17 required.

  5. OpenBLAS.

  6. tbb，In CentOS it can be installed by yum. Such as: yum install tbb-devel.x86_64.

  7. RocksDB == 6.2.2 (optional). You don’t need to install it manually, the script installs it automatically. But you need to manually install the dependencies of rocksdb. Please refer to the installation method: https://github.com/facebook/rocksdb/blob/master/INSTALL.md

  8. CUDA >= 9.0, if you want GPU support.

Compile

  • Enter the GOPATH directory, cd $GOPATH/src mkdir -p github.com/vearch cd github.com/vearch

  • Download the source code: git clone https://github.com/vearch/vearch.git ($vearch denotes the absolute path of vearch code)

  • To add GPU Index support: change BUILD_WITH_GPU from “off” to “on” in $vearch/engine/CMakeLists.txt

  • To add Scann Index support: change BUILD_WITH_SCANN from “off” to “on” in $vearch/engine/CMakeLists.txt

  • Compile vearch and gamma

    1. cd build

    2. sh build.sh

    generate vearchfile compile success

Deploy

Before run vearch, you shuld set LD_LIBRARY_PATH, Ensure that system can find gamma dynamic libraries. The gamma dynamic library that has been compiled is in the $vearch/build/gamma_build folder.

Local Model:

  • generate configuration file conf.toml
  1. [global]
  2.     # the name will validate join cluster by same name
  3.     name = "vearch"
  4.     # you data save to disk path ,If you are in a production environment, You'd better set absolute paths
  5.     data = ["datas/"]
  6.     # log path , If you are in a production environment, You'd better set absolute paths
  7.     log = "logs/"
  8.     # default log type for any model
  9.     level = "debug"
  10.     # master <-> ps <-> router will use this key to send or receive data
  11.     signkey = "vearch"
  12.     skip_auth = true
  14. # if you are master you'd better set all config for router and ps and router and ps use default config it so cool
  15. [[masters]]
  16.     # name machine name for cluster
  17.     name = "m1"
  18.     # ip or domain
  19.     address = "127.0.0.1"
  20.     # api port for http server
  21.     api_port = 8817
  22.     # port for etcd server
  23.     etcd_port = 2378
  24.     # listen_peer_urls List of comma separated URLs to listen on for peer traffic.
  25.     # advertise_peer_urls List of this member's peer URLs to advertise to the rest of the cluster. The URLs needed to be a comma-separated list.
  26.     etcd_peer_port = 2390
  27.     # List of this member's client URLs to advertise to the public.
  28.     # The URLs needed to be a comma-separated list.
  29.     # advertise_client_urls AND listen_client_urls
  30.     etcd_client_port = 2370
  32. [router]
  33.     # port for server
  34.     port = 9001
  36. [ps]
  37.     # port for server
  38.     rpc_port = 8081
  39.     # raft config begin
  40.     raft_heartbeat_port = 8898
  41.     raft_replicate_port = 8899
  42.     heartbeat-interval = 200 #ms
  43.     raft_retain_logs = 10000
  44.     raft_replica_concurrency = 1
  45.     raft_snap_concurrency = 1
  • start
  1. ./vearch -conf conf.toml all

Cluster Model:

  • vearch has three module: ps(PartitionServer) , master, router, run ./vearch -f conf.toml ps/router/master start ps/router/master module

Now we have five machine, two master, two ps and one router

  • master

    • 192.168.1.1

    • 192.168.1.2

  • ps

    • 192.168.1.3

    • 192.168.1.4

  • router

    • 192.168.1.5
  1. [global]
  2.     name = "vearch"
  3.     data = ["datas/"]
  4.     log = "logs/"
  5.     level = "info"
  6.     signkey = "vearch"
  7.     skip_auth = true
  9. # if you are master, you'd better set all config for router、ps and router, ps use default config it so cool
  10. [[masters]]
  11.     name = "m1"
  12.     address = "192.168.1.1"
  13.     api_port = 8817
  14.     etcd_port = 2378
  15.     etcd_peer_port = 2390
  16.     etcd_client_port = 2370
  17. [[masters]]
  18.     name = "m2"
  19.     address = "192.168.1.2"
  20.     api_port = 8817
  21.     etcd_port = 2378
  22.     etcd_peer_port = 2390
  23.     etcd_client_port = 2370
  24. [router]
  25.     port = 9001
  26.     skip_auth = true
  27. [ps]
  28.     rpc_port = 8081
  29.     raft_heartbeat_port = 8898
  30.     raft_replicate_port = 8899
  31.     heartbeat-interval = 200 #ms
  32.     raft_retain_logs = 10000
  33.     raft_replica_concurrency = 1
  34.     raft_snap_concurrency = 1
  • on 192.168.1.1 , 192.168.1.2 run master
  1. ./vearch -conf config.toml master
  • on 192.168.1.3 , 192.168.1.4 run ps
  1. ./vearch -conf config.toml ps
  • on 192.168.1.5 run router
  1. ./vearch -conf config.toml router