Cloud

cloud cloud

Scalable cloud-native applications can be built with txtai. The following cloud runtimes are supported.

  • Container Orchestration Systems (i.e. Kubernetes)
  • Docker Engine
  • Serverless Compute
  • txtai.cloud (coming in 2024)

Images for txtai are available on Docker Hub for CPU and GPU installs. The CPU install is recommended when GPUs aren’t available given the image is significantly smaller.

The base txtai images have no models installed and models will be downloaded each time the container starts. Caching the models is recommended as that will significantly reduce container start times. This can be done a couple different ways.

  • Create a container with the models cached

  • Set the transformers cache environment variable and mount that volume when starting the image

    1. docker run -v <local dir>:/models -e TRANSFORMERS_CACHE=/models --rm -it <docker image>

Build txtai images

The txtai images found on Docker Hub are configured to support most situations. This image can be locally built with different options as desired.

Examples build commands below.

  1. # Get Dockerfile
  2. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/base/Dockerfile
  4. # Build Ubuntu 22.04 image running Python 3.10
  5. docker build -t txtai --build-arg BASE_IMAGE=ubuntu:22.04 --build-arg PYTHON_VERSION=3.10 .
  7. # Build image with GPU support
  8. docker build -t txtai --build-arg GPU=1 .
  10. # Build minimal image with the base txtai components
  11. docker build -t txtai --build-arg COMPONENTS= .

Container image model caching

As mentioned previously, model caching is recommended to reduce container start times. The following commands demonstrate this. In all cases, it is assumed a config.yml file is present in the local directory with the desired configuration set.

API

This section builds an image that caches models and starts an API service. The config.yml file should be configured with the desired components to expose via the API.

The following is a sample config.yml file that creates an Embeddings API service.

  1. # config.yml
  2. writable: true
  4. embeddings:
  5.   path: sentence-transformers/nli-mpnet-base-v2
  6.   content: true

The next section builds the image and starts an instance.

  1. # Get Dockerfile
  2. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/api/Dockerfile
  4. # CPU build
  5. docker build -t txtai-api .
  7. # GPU build
  8. docker build -t txtai-api --build-arg BASE_IMAGE=neuml/txtai-gpu .
  10. # Run
  11. docker run -p 8000:8000 --rm -it txtai-api

Service

This section builds a scheduled workflow service. More on scheduled workflows can be found here.

  1. # Get Dockerfile
  2. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/service/Dockerfile
  4. # CPU build
  5. docker build -t txtai-service .
  7. # GPU build
  8. docker build -t txtai-service --build-arg BASE_IMAGE=neuml/txtai-gpu .
  10. # Run
  11. docker run --rm -it txtai-service

Workflow

This section builds a single run workflow. Example workflows can be found here.

  1. # Get Dockerfile
  2. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/workflow/Dockerfile
  4. # CPU build
  5. docker build -t txtai-workflow . 
  7. # GPU build
  8. docker build -t txtai-workflow --build-arg BASE_IMAGE=neuml/txtai-gpu .
  10. # Run
  11. docker run --rm -it txtai-workflow <workflow name> <workflow parameters>

Serverless Compute

One of the most powerful features of txtai is building YAML-configured applications with the “build once, run anywhere” approach. API instances and workflows can run locally, on a server, on a cluster or serverless.

Serverless instances of txtai are supported on frameworks such as AWS Lambda, Google Cloud Functions, Azure Cloud Functions and Kubernetes with Knative.

AWS Lambda

The following steps show a basic example of how to build a serverless API instance with AWS SAM.

  • Create config.yml and template.yml
  1. # config.yml
  2. writable: true
  4. embeddings:
  5.   path: sentence-transformers/nli-mpnet-base-v2
  6.   content: true
  1. # template.yml
  2. Resources:
  3.   txtai:
  4.     Type: AWS::Serverless::Function
  5.     Properties:
  6.       PackageType: Image
  7.       MemorySize: 3000
  8.       Timeout: 20
  9.       Events:
  10.         Api:
  11.           Type: Api
  12.           Properties:
  13.             Path: "/{proxy+}"
  14.             Method: ANY
  15.     Metadata:
  16.       Dockerfile: Dockerfile
  17.       DockerContext: ./
  18.       DockerTag: api
  1. # Get Dockerfile and application
  2. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/aws/api.py
  3. wget https://raw.githubusercontent.com/neuml/txtai/master/docker/aws/Dockerfile
  5. # Build the docker image
  6. sam build
  8. # Start API gateway and Lambda instance locally
  9. sam local start-api -p 8000 --warm-containers LAZY
  11. # Verify instance running (should return 0)
  12. curl http://localhost:8080/count

If successful, a local API instance is now running in a “serverless” fashion. This configuration can be deployed to AWS using SAM. See this link for more information.

Kubernetes with Knative

txtai scales with container orchestration systems. This can be self-hosted or with a cloud provider such as Amazon Elastic Kubernetes Service, Google Kubernetes Engine and Azure Kubernetes Service. There are also other smaller providers with a managed Kubernetes offering.

A full example covering how to build a serverless txtai application on Kubernetes with Knative can be found here.

txtai.cloud

txtai.cloud is a planned effort that will offer an easy and secure way to run hosted txtai applications.