Troubleshooting Network Observability

To assist in troubleshooting Network Observability issues, you can perform some troubleshooting actions.

Configuring network traffic menu entry in the OKD console

Manually configure the network traffic menu entry in the OKD console when the network traffic menu entry is not listed in Observe menu in the OKD console.

Prerequisites

  • You have installed OKD version 4.10 or newer.

Procedure

  1. Check if the spec.consolePlugin.register field is set to true by running the following command:

    1. $ oc -n netobserv get flowcollector cluster -o yaml

    Example output

    1. apiVersion: flows.netobserv.io/v1alpha1
    2. kind: FlowCollector
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   consolePlugin:
    7.     register: false

  2. Optional: Add the netobserv-plugin plugin by manually editing the Console Operator config:

    1. $ oc edit console.operator.openshift.io cluster

    Example output

    1. ...
    2. spec:
    3.   plugins:
    4.   - netobserv-plugin
    5. ...

  3. Optional: Set the spec.consolePlugin.register field to true by running the following command:

    1. $ oc -n netobserv edit flowcollector cluster -o yaml

    Example output

    1. apiVersion: flows.netobserv.io/v1alpha1
    2. kind: FlowCollector
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   consolePlugin:
    7.     register: true

  4. Ensure the status of console pods is running by running the following command:

    1. $ oc get pods -n openshift-console -l app=console

  5. Restart the console pods by running the following command:

    1. $ oc delete pods -n openshift-console -l app=console

  6. Clear your browser cache and history.

  7. Check the status of Network Observability plugin pods by running the following command:

    1. $ oc get pods -n netobserv -l app=netobserv-plugin

    Example output

    1. NAME                                READY   STATUS    RESTARTS   AGE
    2. netobserv-plugin-68c7bbb9bb-b69q6   1/1     Running   0          21s

  8. Check the logs of the Network Observability plugin pods by running the following command:

    1. $ oc logs -n netobserv -l app=netobserv-plugin

    Example output

    1. time="2022-12-13T12:06:49Z" level=info msg="Starting netobserv-console-plugin [build version: , build date: 2022-10-21 15:15] at log level info" module=main
    2. time="2022-12-13T12:06:49Z" level=info msg="listening on https://:9001" module=server

Flowlogs-Pipeline does not consume network flows after installing Kafka

If you deployed the flow collector first with deploymentModel: KAFKA and then deployed Kafka, the flow collector might not connect correctly to Kafka. Manually restart the flow-pipeline pods where Flowlogs-pipeline does not consume network flows from Kafka.

Procedure

  1. Delete the flow-pipeline pods to restart them by running the following command:

    1. $ oc delete pods -n netobserv -l app=flowlogs-pipeline-transformer

Failing to see network flows from both br-int and br-ex interfaces

br-ex` and br-int are virtual bridge devices operated at OSI layer 2. The eBPF agent works at the IP and TCP levels, layers 3 and 4 respectively. You can expect that the eBPF agent captures the network traffic passing through br-ex and br-int, when the network traffic is processed by other interfaces such as physical host or virtual pod interfaces. If you restrict the eBPF agent network interfaces to attach only to br-ex and br-int, you do not see any network flow.

Manually remove the part in the interfaces or excludeInterfaces that restricts the network interfaces to br-int and br-ex.

Procedure

  1. Remove the interfaces: [ 'br-int', 'br-ex' ] field. This allows the agent to fetch information from all the interfaces. Alternatively, you can specify the Layer-3 interface for example, eth0. Run the following command:

    1. $ oc edit -n netobserv flowcollector.yaml -o yaml

    Example output

    1. apiVersion: flows.netobserv.io/v1alpha1
    2. kind: FlowCollector
    3. metadata:
    4.   name: cluster
    5. spec:
    6.   agent:
    7.     type: EBPF
    8.     ebpf:
    9.       interfaces: [ 'br-int', 'br-ex' ] (1)
    1Specifies the network interfaces.

Network Observability controller manager pod runs out of memory

You can increase memory limits for the Network Observability operator by patching the Cluster Service Version (CSV), where Network Observability controller manager pod runs out of memory.

Procedure

  1. Run the following command to patch the CSV:

    1. $ oc -n netobserv patch csv network-observability-operator.v1.0.0 --type='json' -p='[{"op": "replace", "path":"/spec/install/spec/deployments/0/spec/template/spec/containers/0/resources/limits/memory", value: "1Gi"}]'

    Example output

    1. clusterserviceversion.operators.coreos.com/network-observability-operator.v1.0.0 patched

  2. Run the following command to view the updated CSV:

    1. $ oc -n netobserv get csv network-observability-operator.v1.0.0 -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].resources.limits.memory}'
    2. 1Gi