Network Observability Operator in OKD

Network Observability is an OpenShift operator that deploys a monitoring pipeline to collect and enrich network traffic flows that are produced by the Network Observability eBPF agent.

Viewing statuses

The Network Observability Operator provides the Flow Collector API. When a Flow Collector resource is created, it deploys pods and services to create and store network flows in the Loki log store, as well as to display dashboards, metrics, and flows in the OKD web console.

Procedure

  1. Run the following command to view the state of FlowCollector:

    1. $ oc get flowcollector/cluster

    Example output

    1. NAME      AGENT   SAMPLING (EBPF)   DEPLOYMENT MODEL   STATUS
    2. cluster   EBPF    50                DIRECT             Ready

  2. Check the status of pods running in the netobserv namespace by entering the following command:

    1. $ oc get pods -n netobserv

    Example output

    1. NAME                              READY   STATUS    RESTARTS   AGE
    2. flowlogs-pipeline-56hbp           1/1     Running   0          147m
    3. flowlogs-pipeline-9plvv           1/1     Running   0          147m
    4. flowlogs-pipeline-h5gkb           1/1     Running   0          147m
    5. flowlogs-pipeline-hh6kf           1/1     Running   0          147m
    6. flowlogs-pipeline-w7vv5           1/1     Running   0          147m
    7. netobserv-plugin-cdd7dc6c-j8ggp   1/1     Running   0          147m

flowlogs-pipeline pods collect flows, enriches the collected flows, then send flows to the Loki storage. netobserv-plugin pods create a visualization plugin for the OKD Console.

  1. Check the status of pods running in the namespace netobserv-privileged by entering the following command:

    1. $ oc get pods -n netobserv-privileged

    Example output

    1. NAME                         READY   STATUS    RESTARTS   AGE
    2. netobserv-ebpf-agent-4lpp6   1/1     Running   0          151m
    3. netobserv-ebpf-agent-6gbrk   1/1     Running   0          151m
    4. netobserv-ebpf-agent-klpl9   1/1     Running   0          151m
    5. netobserv-ebpf-agent-vrcnf   1/1     Running   0          151m
    6. netobserv-ebpf-agent-xf5jh   1/1     Running   0          151m

netobserv-ebpf-agent pods monitor network interfaces of the nodes to get flows and send them to flowlogs-pipeline pods.

  1. If you are using a Loki Operator, check the status of pods running in the openshift-operators-redhat namespace by entering the following command:

    1. $ oc get pods -n openshift-operators-redhat

    Example output

    1. NAME                                                READY   STATUS    RESTARTS   AGE
    2. loki-operator-controller-manager-5f6cff4f9d-jq25h   2/2     Running   0          18h
    3. lokistack-compactor-0                               1/1     Running   0          18h
    4. lokistack-distributor-654f87c5bc-qhkhv              1/1     Running   0          18h
    5. lokistack-distributor-654f87c5bc-skxgm              1/1     Running   0          18h
    6. lokistack-gateway-796dc6ff7-c54gz                   2/2     Running   0          18h
    7. lokistack-index-gateway-0                           1/1     Running   0          18h
    8. lokistack-index-gateway-1                           1/1     Running   0          18h
    9. lokistack-ingester-0                                1/1     Running   0          18h
    10. lokistack-ingester-1                                1/1     Running   0          18h
    11. lokistack-ingester-2                                1/1     Running   0          18h
    12. lokistack-querier-66747dc666-6vh5x                  1/1     Running   0          18h
    13. lokistack-querier-66747dc666-cjr45                  1/1     Running   0          18h
    14. lokistack-querier-66747dc666-xh8rq                  1/1     Running   0          18h
    15. lokistack-query-frontend-85c6db4fbd-b2xfb           1/1     Running   0          18h
    16. lokistack-query-frontend-85c6db4fbd-jm94f           1/1     Running   0          18h

Network Observablity Operator architecture

The Network Observability Operator provides the FlowCollector API, which is instantiated at installation and configured to reconcile the eBPF agent, the flowlogs-pipeline, and the netobserv-plugin components. Only a single FlowCollector per cluster is supported.

The eBPF agent runs on each cluster node with some privileges to collect network flows. The flowlogs-pipeline receives the network flows data and enriches the data with Kubernetes identifiers. If you are using Loki, the flowlogs-pipeline sends flow logs data to Loki for storing and indexing. The netobserv-plugin, which is a dynamic OKD web console plugin, queries Loki to fetch network flows data. Cluster-admins can view the data in the web console.

Network Observability eBPF export architecture

If you are using the Kafka option, the eBPF agent sends the network flow data to Kafka, and the flowlogs-pipeline reads from the Kafka topic before sending to Loki, as shown in the following diagram.

Network Observability using Kafka

Viewing Network Observability Operator status and configuration

You can inspect the status and view the details of the FlowCollector using the oc describe command.

Procedure

  1. Run the following command to view the status and configuration of the Network Observability Operator:

    1. $ oc describe flowcollector/cluster