Configuring the Network Observability Operator

Configuring the Network Observability Operator

You can update the Flow Collector API resource to configure the Network Observability Operator and its managed components. The Flow Collector is explicitly created during installation. Since this resource operates cluster-wide, only a single FlowCollector is allowed, and it has to be named cluster.

View the FlowCollector resource

You can view and edit YAML directly in the OKD web console.

Procedure

In the web console, navigate to Operators → Installed Operators.
Under the Provided APIs heading for the NetObserv Operator, select Flow Collector.
Select cluster then select the YAML tab. There, you can modify the FlowCollector resource to configure the Network Observability operator.

The following example shows a sample FlowCollector resource for OKD Network Observability operator:

Sample FlowCollector resource

apiVersion: flows.netobserv.io/v1beta1
kind: FlowCollector
metadata:
  name: cluster
spec:
  namespace: netobserv
  deploymentModel: DIRECT
  agent:
    type: EBPF                                (1)
    ebpf:
      sampling: 50                            (2)
      logLevel: info
      privileged: false
      resources:
        requests:
          memory: 50Mi
          cpu: 100m
        limits:
          memory: 800Mi
  processor:
    logLevel: info
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 800Mi
    conversationEndTimeout: 10s
    logTypes: FLOW                            (3)
    conversationHeartbeatInterval: 30s
  loki:                                       (4)
    url: 'http://loki-gateway-http.netobserv.svc:8080/api/logs/v1/network'
    statusUrl: 'https://loki-query-frontend-http.netobserv.svc:3100/'
    authToken: FORWARD
    tls:
      enable: true
      caCert:
        type: configmap
        name: loki-gateway-ca-bundle
        certFile: service-ca.crt
  consolePlugin:
    register: true
    logLevel: info
    portNaming:
      enable: true
      portNames:
        "3100": loki
    quickFilters:                             (5)
    - name: Applications
      filter:
        src_namespace!: 'openshift-,netobserv'
        dst_namespace!: 'openshift-,netobserv'
      default: true
    - name: Infrastructure
      filter:
        src_namespace: 'openshift-,netobserv'
        dst_namespace: 'openshift-,netobserv'
    - name: Pods network
      filter:
        src_kind: 'Pod'
        dst_kind: 'Pod'
      default: true
    - name: Services network
      filter:
        dst_kind: 'Service'

1	The Agent specification, `spec.agent.type`, must be `EBPF`. eBPF is the only OKD supported option.
2	You can set the Sampling specification, `spec.agent.ebpf.sampling`, to manage resources. Lower sampling values might consume a large amount of computational, memory and storage resources. You can mitigate this by specifying a sampling ratio value. A value of 100 means 1 flow every 100 is sampled. A value of 0 or 1 means all flows are captured. The lower the value, the increase in returned flows and the accuracy of derived metrics. By default, eBPF sampling is set to a value of 50, so 1 flow every 50 is sampled. Note that more sampled flows also means more storage needed. It is recommend to start with default values and refine empirically, to determine which setting your cluster can manage.
3	The optional specifications `spec.processor.logTypes`, `spec.processor.conversationHeartbeatInterval`, and `spec.processor.conversationEndTimeout` can be set to enable conversation tracking. When enabled, conversation events are queryable in the web console. The values for `spec.processor.logTypes` are as follows: `FLOWS` `CONVERSATIONS`, `ENDED_CONVERSATIONS`, or `ALL`. Storage requirements are highest for `ALL` and lowest for `ENDED_CONVERSATIONS`.
4	The Loki specification, `spec.loki`, specifies the Loki client. The default values match the Loki install paths mentioned in the Installing the Loki Operator section. If you used another installation method for Loki, specify the appropriate client information for your install.
5	The `spec.quickFilters` specification defines filters that show up in the web console. The `Application` filter keys,`src_namespace` and `dst_namespace`, are negated (`!`), so the `Application` filter shows all traffic that does not originate from, or have a destination to, any `openshift-` or `netobserv` namespaces. For more information, see Configuring quick filters below.

Additional resources

For more information about conversation tracking, see Working with conversations.

Configuring the Flow Collector resource with Kafka

You can configure the FlowCollector resource to use Kafka. A Kafka instance needs to be running, and a Kafka topic dedicated to OKD Network Observability must be created in that instance. For more information, refer to your Kafka documentation, such as Kafka documentation with AMQ Streams.

The following example shows how to modify the FlowCollector resource for OKD Network Observability operator to use Kafka:

Sample Kafka configuration in FlowCollector resource

  deploymentModel: KAFKA                                    (1)
  kafka:
    address: "kafka-cluster-kafka-bootstrap.netobserv"      (2)
    topic: network-flows                                    (3)
    tls:
      enable: false                                         (4)

1	Set `spec.deploymentModel` to `KAFKA` instead of `DIRECT` to enable the Kafka deployment model.
2	`spec.kafka.address` refers to the Kafka bootstrap server address. You can specify a port if needed, for instance `kafka-cluster-kafka-bootstrap.netobserv:9093` for using TLS on port 9093.
3	`spec.kafka.topic` should match the name of a topic created in Kafka.
4	`spec.kafka.tls` can be used to encrypt all communications to and from Kafka with TLS or mTLS. When enabled, the Kafka CA certificate must be available as a ConfigMap or a Secret, both in the namespace where the `flowlogs-pipeline` processor component is deployed (default: `netobserv`) and where the eBPF agents are deployed (default: `netobserv-privileged`). It must be referenced with `spec.kafka.tls.caCert`. When using mTLS, client secrets must be available in these namespaces as well (they can be generated for instance using the AMQ Streams User Operator) and referenced with `spec.kafka.tls.userCert`.

Export enriched network flow data

You can send network flows to Kafka, so that they can be consumed by any processor or storage that supports Kafka input, such as Splunk, Elasticsearch, or Fluentd.

Prerequisites

Installed Kafka

Procedure

In the web console, navigate to Operators → Installed Operators.
Under the Provided APIs heading for the NetObserv Operator, select Flow Collector.
Select cluster and then select the YAML tab.

Edit the FlowCollector to configure spec.exporters as follows:

apiVersion: flows.netobserv.io/v1alpha1
kind: FlowCollector
metadata:
  name: cluster
spec:
  exporters:
  - type: KAFKA
      kafka:
        address: "kafka-cluster-kafka-bootstrap.netobserv"
        topic: netobserv-flows-export   (1)
        tls:
          enable: false                 (2)

1 The Network Observability Operator exports all flows to the configured Kafka topic.

2 You can encrypt all communications to and from Kafka with SSL/TLS or mTLS. When enabled, the Kafka CA certificate must be available as a ConfigMap or a Secret, both in the namespace where the flowlogs-pipeline processor component is deployed (default: netobserv). It must be referenced with spec.exporters.tls.caCert. When using mTLS, client secrets must be available in these namespaces as well (they can be generated for instance using the AMQ Streams User Operator) and referenced with spec.exporters.tls.userCert.

After configuration, network flows data can be sent to an available output in a JSON format. For more information, see Network flows format reference

Additional resources

For more information about specifying flow format, see Network flows format reference.

Updating the Flow Collector resource

As an alternative to editing YAML in the OKD web console, you can configure specifications, such as eBPF sampling, by patching the flowcollector custom resource (CR):

Procedure

Run the following command to patch the flowcollector CR and update the spec.agent.ebpf.sampling value:

$ oc patch flowcollector cluster --type=json -p "[{"op": "replace", "path": "/spec/agent/ebpf/sampling", "value": <new value>}] -n netobserv"

Configuring quick filters

You can modify the filters in the FlowCollector resource. Exact matches are possible using double-quotes around values. Otherwise, partial matches are used for textual values. The bang (!) character, placed at the end of a key, means negation. See the sample FlowCollector resource for more context about modifying the YAML.

The filter matching types “all of” or “any of” is a UI setting that the users can modify from the query options. It is not part of this resource configuration.

Here is a list of all available filter keys:

Table 1. Filter keys
Universal*	Source	Destination	Description
namespace	`src_namespace`	`dst_namespace`	Filter traffic related to a specific namespace.
name	`src_name`	`dst_name`	Filter traffic related to a given leaf resource name, such as a specific pod, service, or node (for host-network traffic).
kind	`src_kind`	`dst_kind`	Filter traffic related to a given resource kind. The resource kinds include the leaf resource (Pod, Service or Node), or the owner resource (Deployment and StatefulSet).
owner_name	`src_owner_name`	`dst_owner_name`	Filter traffic related to a given resource owner; that is, a workload or a set of pods. For example, it can be a Deployment name, a StatefulSet name, etc.
resource	`src_resource`	`dst_resource`	Filter traffic related to a specific resource that is denoted by its canonical name, that identifies it uniquely. The canonical notation is `kind.namespace.name` for namespaced kinds, or `node.name` for nodes. For example, `Deployment.my-namespace.my-web-server`.
address	`src_address`	`dst_address`	Filter traffic related to an IP address. IPv4 and IPv6 are supported. CIDR ranges are also supported.
mac	`src_mac`	`dst_mac`	Filter traffic related to a MAC address.
port	`src_port`	`dst_port`	Filter traffic related to a specific port.
host_address	`src_host_address`	`dst_host_address`	Filter traffic related to the host IP address where the pods are running.
protocol	N/A	N/A	Filter traffic related to a protocol, such as TCP or UDP.

Universal keys filter for any of source or destination. For example, filtering name: 'my-pod' means all traffic from my-pod and all traffic to my-pod, regardless of the matching type used, whether Match all or Match any.