Transparent Proxying

What is Transparent Proxying?

A transparent proxy is a type of server that can intercept network traffic to and from a service without changes to the client application code. In the case of Kuma it is used to capture traffic and redirect it to kuma-dp so Mesh policies can be applied.

To accomplish this, Kuma utilizes iptables and offers additional, experimental support for eBPF. The examples provided in this section will concentrate on iptables to clearly illustrate the point.

Below is a high level visualization of how Transparent Proxying works

  2.  sequenceDiagram
  3.  autonumber
  4.      participant Browser as Client(e.g. mobile app)
  5.      participant Kernel as Kernel
  6.      participant ServiceMeshIn as kuma sidecar(15006)
  7.      participant Node as example.com:5000(Front-end App)
  8.      participant ServiceMeshOut as kuma sidecar(15001)
  9.      Browser->>+Kernel: GET / HTTP1.1Host: example.com:5000
  11.      rect rgb(233,233,233)
  12.      Note over Kernel,ServiceMeshOut: EXAMPLE.COM
  13.      Note over Node: (Optional) Apply inbound policies
  14.      Note over ServiceMeshOut: (Optional) Apply inbound policies
  15.      Kernel->>+ServiceMeshIn: Capture inbound TCP trafficand Redirect to the sidecar (listener port 15006)
  16.      ServiceMeshIn->>+Node: Redirect to theoriginal destination (example.com:5000)
  17.          Node->>+Kernel: Send the Front-end Response
  18.      Kernel->>+ServiceMeshOut: Capture outbound TCP trafficand Redirect to the sidecar (listener port 15001)
  19.      end
  20.      ServiceMeshOut->>+Browser: Response to Client
  21.      %% Note over Browser,ServiceMeshOut: Traffic Flow Sequence

A life without Transparent Proxying

If you choose to not use transparent proxying, or you are running on a platform where transparent proxying is not available, there are some additional considerations.

  • You will need to specify inbound and outbound ports to capture traffic on
  • .mesh addresses are unavailable
  • You may need to update your application code to use the new capture ports
  • No support for a VirtualOutbound

Without manipulating IPTables to redirect traffic you will need to explicitly tell kuma-dp where to listen to capture it. As noted, this can require changes to your application code as seen below:

Here we specify that we will listen on the address 10.119.249.39:15000 (line 7). This in turn creates an envoy listener for the port. When consuming a service over this 15000 it will cause traffic to redirect to 127.0.0.1:5000 (line 8) where our app is running.

  1.   type: Dataplane
  2.   mesh: default
  3.   name: demo-app
  4.   networking: 
  5.     address: 10.119.249.39 
  6.     inbound: 
  7.       - port: 15000
  8.         servicePort: 5000
  9.         serviceAddress: 127.0.0.1
  10.         tags: 
  11.           kuma.io/service: app
  12.           kuma.io/protocol: http

How it works

Inbound TCP Traffic

The inbound port, 15006, is the default for capturing requests to the system. This rule allows us to capture and redirect ALL TCP traffic to port 15006.

  1. --append KUMA_MESH_INBOUND_REDIRECT --protocol tcp --jump REDIRECT --to-ports 15006

An envoy listener is also created for this port which we can see in the admin interface (:9901/config_dump). In the below example you can see the listener created on all interfaces (line 8) and port 15006 (line 9).

  1.      "name": "inbound:passthrough:ipv4",
  2.      "active_state": {
  3.       "listener": {
  4.        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  5.        "name": "inbound:passthrough:ipv4",
  6.        "address": {
  7.         "socket_address": {
  8.          "address": "0.0.0.0",
  9.          "port_value": 15006
  10.         }
  11.        },
  12.       ...
  13.        "use_original_dst": true,
  14.        "traffic_direction": "INBOUND",

Notice the setting use_original_dst (line 13). This listener will send traffic to a special type of cluster, ORIGINAL_DST. This is important since we are redirecting traffic here based on the IPtables rules, which means when this service was requested it was not likely it was requested over this port, 15006, but rather whatever the target application is listening on (i.e. demo-app port 5000)

  1.      "name": "inbound:10.244.0.6:5000",
  2.      "active_state": {
  3.       "version_info": "9dac7d53-3560-4ad4-ba42-c7e563db958e",
  4.       "listener": {
  5.        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  6.        "name": "inbound:10.244.0.6:5000",
  7.        "address": {
  8.         "socket_address": {
  9.          "address": "10.244.0.6",
  10.          "port_value": 5000
  11.         }
  12.        }
  13.       }
  14.      }

Using the Kuma counter demo app as an example, when the client needs to talk to the node app, it does not do so over 15006, but rather the actual application port, 5000. This is the “transparent” part of the proxying as it is not expected that apps will need to be redesigned or changed in any way to utilize mesh.

So, when the request comes into the system, the IPTables rule grabs the traffic and sends it to envoy port 15006. Once here, we check where the request was originally intended to go, in this case 5000 and forward it.

A further review of the envoy config will show our Node app listener where the IP address, 10.244.0.6, is that of the demo-app pod. Now that envoy is in control of the traffic we can now (optionally) apply filters/Mesh policies.

  1.      "name": "inbound:10.244.0.6:5000",
  2.      "active_state": {
  3.       "version_info": "9dac7d53-3560-4ad4-ba42-c7e563db958e",
  4.       "listener": {
  5.        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  6.        "name": "inbound:10.244.0.6:5000",
  7.        "address": {
  8.         "socket_address": {
  9.          "address": "10.244.0.6",
  10.          "port_value": 5000
  11.         }
  12.        },
  13.           "filter_chains": [
  14.             {
  15.               "filters": [
  16.                 {
  17.                   "name": "envoy.filters.network.http_connection_manager",
  18.                   "typed_config": {
  19.                     "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
  20.                     "stat_prefix": "localhost_5000",
  21.                     "route_config": {
  22.                     "http_filters": [
  23.                       {
  24.                         "name": "envoy.filters.http.fault",
  25.                         "typed_config": {
  26.                           "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault",
  27.                           "delay": {
  28.                             "fixed_delay": "5s",
  29.                             "percentage": {
  30.                               "numerator": 50,
  31.                               "denominator": "TEN_THOUSAND"
  32. ...

Outbound TCP Traffic

The outbound port, 15001, is the default for capturing outbound traffic from the system. That is, traffic leaving the mesh. This rule allow us to capture and redirect all TCP traffic to 15001.

  1. --append KUMA_MESH_OUTBOUND_REDIRECT --protocol tcp --jump REDIRECT --to-ports 15001

An envoy listener is also created for this port which we can see in the admin interface (:9901/config_dump). In the below example you can see the listener created on all interfaces (line 8) and port 15001 (line 9). This will allow us to capture and outbound traffic policies.

  1.      "name": "outbound:passthrough:ipv6",
  2.      "active_state": {
  3.       "listener": {
  4.        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  5.        "name": "outbound:passthrough:ipv6",
  6.        "address": {
  7.         "socket_address": {
  8.          "address": "::",
  9.          "port_value": 15001
  10.         }
  11.        },
  12.       ...
  13.        "use_original_dst": true,
  14.        "traffic_direction": "OUTBOUND"