Stream Proxy

A stream proxy operates at the transport layer, handling stream-oriented traffic based on TCP and UDP protocols. TCP is used for many applications and services, such as LDAP, MySQL, and RTMP. UDP is used for many popular non-transactional applications, such as DNS, syslog, and RADIUS.

APISIX can serve as a stream proxy, in addition to being an application layer proxy.

How to enable stream proxy?

By default, stream proxy is disabled.

To enable this option, set apisix.proxy_mode to stream or http&stream, depending on whether you want stream proxy only or both http and stream. Then add the apisix.stream_proxy option in conf/config.yaml and specify the list of addresses where APISIX should act as a stream proxy and listen for incoming requests.

Stream Proxy - 图1note

This “apisix.stream_proxy” option has only been added in versions after 3.2.1.

  1. apisix:
  2.   stream_proxy:
  3.     tcp:
  4.       - 9100 # listen on 9100 ports of all network interfaces for TCP requests
  5.       - "127.0.0.1:9101"
  6.     udp:
  7.       - 9200 # listen on 9200 ports of all network interfaces for UDP requests
  8.       - "127.0.0.1:9211"

If apisix.stream_proxy is undefined in conf/config.yaml, you will encounter an error similar to the following and not be able to add a stream route:

  1. {"error_msg":"stream mode is disabled, can not add stream routes"}

How to set a route?

You can create a stream route using the Admin API /stream_routes endpoint. For example:

Stream Proxy - 图2note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

  1. admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
  1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "remote_addr": "192.168.5.3",
  4.     "upstream": {
  5.         "nodes": {
  6.             "192.168.4.10:1995": 1
  7.         },
  8.         "type": "roundrobin"
  9.     }
  10. }'

With this configuration, APISIX would only forward the request to the upstream service at 192.168.4.10:1995 if and only if the request is sent from 192.168.5.3. See the next section to learn more about filtering options.

More examples can be found in test cases.

More stream route filtering options

Currently there are three attributes in stream routes that can be used for filtering requests:

  • server_addr: The address of the APISIX server that accepts the L4 stream connection.
  • server_port: The port of the APISIX server that accepts the L4 stream connection.
  • remote_addr: The address of client from which the request has been made.

Here is an example:

  1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "server_addr": "127.0.0.1",
  4.     "server_port": 2000,
  5.     "upstream": {
  6.         "nodes": {
  7.             "127.0.0.1:1995": 1
  8.         },
  9.         "type": "roundrobin"
  10.     }
  11. }'

It means APISIX will proxy the request to 127.0.0.1:1995 when the server address is 127.0.0.1 and the server port is equal to 2000.

Here is an example with MySQL:

  1. Put this config inside config.yaml

    1. apisix:
    2.   stream_proxy: # TCP/UDP proxy
    3.     tcp: # TCP proxy address list
    4.       - 9100 # by default uses 0.0.0.0
    5.       - "127.0.0.10:9101"

  2. Now run a mysql docker container and expose port 3306 to the host

    1. $ docker run --name mysql -e MYSQL_ROOT_PASSWORD=toor -p 3306:3306 -d mysql mysqld --default-authentication-plugin=mysql_native_password
    2. # check it using a mysql client that it works
    3. $ mysql --host=127.0.0.1 --port=3306 -u root -p
    4. Enter password:
    5. Welcome to the MySQL monitor.  Commands end with ; or \g.
    6. Your MySQL connection id is 25
    7. ...
    8. mysql>

  3. Now we are going to create a stream route with server filtering:

    1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
    2. {
    3.     "server_addr": "127.0.0.10",
    4.     "server_port": 9101,
    5.     "upstream": {
    6.         "nodes": {
    7.             "127.0.0.1:3306": 1
    8.         },
    9.         "type": "roundrobin"
    10.     }
    11. }'

    It only forwards the request to the mysql upstream whenever a connection is received at APISIX server 127.0.0.10 and port 9101. Let’s test that behaviour:

  4. Making a request to 9100 (stream proxy port enabled inside config.yaml), filter matching fails.

    1. $ mysql --host=127.0.0.1 --port=9100 -u root -p
    2. Enter password:
    3. ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

    Instead making a request to the APISIX host and port where the filter matching succeeds:

    1. mysql --host=127.0.0.10 --port=9101 -u root -p
    2. Enter password:
    3. Welcome to the MySQL monitor.  Commands end with ; or \g.
    4. Your MySQL connection id is 26
    5. ...
    6. mysql>

Read Admin API’s Stream Route section for the complete options list.

Accept TLS over TCP connection

APISIX can accept TLS over TCP connection.

First of all, we need to enable TLS for the TCP address:

  1. apisix:
  2.   stream_proxy: # TCP/UDP proxy
  3.     tcp: # TCP proxy address list
  4.       - addr: 9100
  5.         tls: true

Second, we need to configure certificate for the given SNI. See Admin API’s SSL section for how to do. mTLS is also supported, see Protect Route for how to do.

Third, we need to configure a stream route to match and proxy it to the upstream:

  1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "upstream": {
  4.         "nodes": {
  5.             "127.0.0.1:1995": 1
  6.         },
  7.         "type": "roundrobin"
  8.     }
  9. }'

When the connection is TLS over TCP, we can use the SNI to match a route, like:

  1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "sni": "a.test.com",
  4.     "upstream": {
  5.         "nodes": {
  6.             "127.0.0.1:5991": 1
  7.         },
  8.         "type": "roundrobin"
  9.     }
  10. }'

In this case, a connection handshaked with SNI a.test.com will be proxied to 127.0.0.1:5991.

Proxy to TLS over TCP upstream

APISIX also supports proxying to TLS over TCP upstream.

  1. curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
  2. {
  3.     "upstream": {
  4.         "scheme": "tls",
  5.         "nodes": {
  6.             "127.0.0.1:1995": 1
  7.         },
  8.         "type": "roundrobin"
  9.     }
  10. }'

By setting the scheme to “tls”, APISIX will do TLS handshake with the upstream.

When the client is also speaking TLS over TCP, the SNI from the client will pass through to the upstream. Otherwise, a dummy SNI “apisix_backend” will be used.