Pulsar isolation

In an organization, a Pulsar instance provides services to multiple teams. When organizing the resources across multiple teams, you want to make a suitable isolation plan to avoid the resource competition between different teams and applications and provide high-quality messaging service. In this case, you need to take resource isolation into consideration and weigh your intended actions against expected and unexpected consequences.

To enforce resource isolation, you can use the Pulsar isolation policy, which allows you to allocate resources (broker and bookie) for the namespace.

Broker isolation

In Pulsar, when namespaces (more specifically, namespace bundles) are assigned dynamically to brokers, the namespace isolation policy limits the set of brokers that can be used for assignment. Before topics are assigned to brokers, you can set the namespace isolation policy with a primary or a secondary regex to select desired brokers.

You can set a namespace isolation policy for a cluster using one of the following methods.

  • Admin CLI
  • REST API
  • Java admin API
  2. pulsar-admin ns-isolation-policy set options

For more information about the command pulsar-admin ns-isolation-policy set options, see here.

Example

  2. bin/pulsar-admin ns-isolation-policy set \
  3. --auto-failover-policy-type min_available \
  4. --auto-failover-policy-params min_limit=1,usage_threshold=80 \
  5. --namespaces my-tenant/my-namespace \
  6. --primary 10.193.216.*  my-cluster policy-name

PUT /admin/v2/namespaces/{tenant}/{namespace}

For how to set namespace isolation policy using Java admin API, see here.

Bookie isolation

A namespace can be isolated into user-defined groups of bookies, which guarantees all the data that belongs to the namespace is stored in desired bookies. The bookie affinity group uses the BookKeeper rack-aware placement policy and it is a way to feed rack information which is stored as JSON format in znode.

You can set a bookie affinity group using one of the following methods.

  • Admin CLI
  • REST API
  • Java admin API
  2. pulsar-admin namespaces set-bookie-affinity-group options

For more information about the command pulsar-admin namespaces set-bookie-affinity-group options, see here.

Example

  2. bin/pulsar-admin bookies set-bookie-rack \
  3. --bookie 127.0.0.1:3181 \
  4. --hostname 127.0.0.1:3181 \
  5. --group group-bookie1 \
  6. --rack rack1
  8. bin/pulsar-admin namespaces set-bookie-affinity-group public/default \
  9. --primary-group group-bookie1
note
  • Do not set a bookie rack name to slash (/) or an empty string ("") if you use Pulsar earlier than 2.7.5, 2.8.3, and 2.9.2. If you use Pulsar 2.7.5, 2.8.3, 2.9.2 or later versions, it falls back to /default-rack or /default-region/default-rack.
  • When RackawareEnsemblePlacementPolicy is enabled, the rack name is not allowed to contain slash (/) except for the beginning and end of the rack name string. For example, rack name like /rack0 is okay, but /rack/0 is not allowed.
  • When RegionawareEnsemblePlacementPolicy is enabled, the rack name can only contain one slash (/) except for the beginning and end of the rack name string. For example, rack name like /region0/rack0 is okay, but /region0rack0 and /region0/rack/0 are not allowed. For the bookie rack name restrictions, see pulsar-admin bookies set-bookie-rack.

POST /admin/v2/namespaces/{tenant}/{namespace}/persistence/bookieAffinity

For how to set bookie affinity group for a namespace using Java admin API, see here.