ngx_http_tunnel_module

KO | EN

gitlite — search

#python #nodejs #typescript #deep-learning #golang #machine-learning #markdown #macos #matlab #react #vue #selenium

ngx_http_tunnel_module

★ 10 Open GitHub ↗

A tunnel connect module for nginx. Supports classic connect over HTTP/1.1, H2 and H3. Extended connect support for connect-udp

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Nginx HTTP Tunnel Connect Module

> [!NOTE]
> Throughput can be limited due to configurations. Try tune TCP
> kernel buffers, use BBR congestion control, set larger
> `http3_stream_buffer_size`, `client_body_buffer_size` and
> `client_max_body_size` values can boost performance greatly.

**This provides a subset features of HTTP CONNECT
over HTTP/1.1, HTTP2, and HTTP/3.**

## Table of contents

- [Tested nginx versions](#tested-nginx-versions)
- [Why implement this?](#implementation-tricks)
- [How to build?](#how-to-build)
- [Example conf file](#example-configuration-file)
- [Detailed config directives](#config-directives)
- [Performance Comparison](#performance-comparison)

## Intro

```txt
Naive Proxy used padding scheme is implemented in here,
works on h2 and h3 only. HTTP/1.1 is not targeted and it is
recommended to use nginx/1.31.* version.

- [x] HTTP/1.1 CONNECT 
- [x] HTTP/2 CONNECT
- [x] HTTP/3, QUIC CONNECT
- [x] Non CONNECT method can co-exist with other HTTP methods
- [x] Proxy authentication
- [x] Map based ACL
- [x] Naive Style Padding Scheme
- [x] Connect-udp with capsule protocol, without QUIC DATAGRAM

Module build relies on some nginx patches.
See patches, apply them to nginx source code,

The patches do following:

1. Let nginx upstream module only open TCP/UDP connection
without any input, using `ignore_input` flag.

2. Let h2/h3 pseudo header parsing pass nginx core.

Read scripts/ to find out more information about
map based acl.

Features that are in WIP:

Refactors that are in WIP:

Any PR or Issues are welcomed. If you are using Agent coding, please review
before submitting it.
```

## Tested Nginx Versions

- Nginx 1.29/28 and some privious version should work, but not
  recommended to use due to some CVE in nginx.

- Nginx 1.30.* is tested to work.

- Nginx 1.31.0 - 1.31.2 is tested to work, apply `header_parsing.patch` and `upstream-1.31.patch`

## How to build

First fetch source code

```bash
mkdir -p build
cd build

git clone https://github.com/nginx/nginx.git
git switch --detach {version} # Checkout on specific version

git clone https://github.com/ZihaoFU245/ngx_http_tunnel_module.git
git switch --detach {version} # Checkout on specific version

cd nginx
```

Apply patches in `patches/` folder, for nginx versions below
1.30.* and including 1.30.*. Apply `header_parsing.patch` and `upstream.patch`
For 1.31.* and above inclusion, apply `upstream-1.31.patch` instead.

On nginx 1.31.0 and newer, nginx core handles proxy authentication for
CONNECT requests. This module therefore does not build its `auth.c` auth path
on those versions.

```bash
git apply ../ngx_http_tunnel_module/patches/header_parsing.patch

git apply ../ngx_http_tunnel_module/patches/upstream.patch
# OR
git apply ../ngx_http_tunnel_module/patches/upstream-1.31.patch
# BASED ON YOUR NGINX VERSION
```

Check nginx build dependency before continue.
Below is an example build configuration.

```bash
./auto/configure \
	--prefix=/usr/local/share/nginx \
	--sbin-path=/usr/local/sbin/nginx \
	--conf-path=/etc/nginx/nginx.conf \
	--pid-path=/run/nginx.pid \
	--lock-path=/var/lock/nginx.lock \
	--error-log-path=/var/log/nginx/error.log \
	--http-log-path=/var/log/nginx/access.log \
	--http-client-body-temp-path=/var/lib/nginx/body \
	--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
	--http-proxy-temp-path=/var/lib/nginx/proxy \
	--http-scgi-temp-path=/var/lib/nginx/scgi \
	--http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
	--with-compat \
	--with-threads \
	--with-file-aio \
	--with-pcre-jit \
	--with-http_ssl_module \
	--with-http_v2_module \
	--with-http_v3_module \
	--with-http_realip_module \
	--with-http_auth_request_module \
	--with-http_mp4_module \
	--with-http_gunzip_module \
	--with-http_gzip_static_module \
	--with-cc-opt='-g -O2 -fPIC -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=3' \
	--with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' \	
	--add-dynamic-module=../ngx_http_tunnel_module
	# use --add-module to compile it in nginx binary
```

Finally, run `make -j$(nproc)`.

## Implementation tricks

1. It is chosen to build as a nginx module as it can reuse nginx
core functionalities, including but not limited to, mature h2/h3
implementation, multiplexing, limit conn, upstream module.

2. This module relies on nginx upstream and http_v2, see header file.

## Example Configuration file

> [!TIP]
> Check `examples/` for minimum configuration files.

> [!IMPORTANT]
> Strongly recommended to use [headers-more module](https://github.com/openresty/headers-more-nginx-module) in nginx to clear
> `Proxy-Authenticate` header in 405 response, specifically for nginx
> v1.31.0 and above. Otherwise, any MITM with active probing capability
> may discover the header, and classify nginx as a forward proxy.

Below is a detailed config flags illustration.

```nginx
load_module ngx_http_tunnel_module.so;  # If build as an dynamic module

user www-data;
worker_processes auto;
worker_cpu_affinity auto;

events {
	worker_connections 1024;
}

http {
	tcp_nopush on;
	tcp_nodelay on;
	server_tokens off;

# If you are using with a file server
    sendfile on;
	include mime.types;

ssl_protocols TLSv1.2 TLSv1.3;
	ssl_prefer_server_ciphers off;

limit_conn_zone $binary_remote_addr zone=addr:1m;

# ---------------------------------------------
	# A map is used in ACL for O(1) lookup,
	# $connect_target_host variable provides the raw
	# authority for classic CONNECT and the parsed
	# target authority for CONNECT-UDP. It is your job
	# to regex match these authorities. Test the ACL
	# before production, some clients put authority
	# as raw ip, raw host, or even host:port.
	# This can be tricky, be careful!
	#
	# Allowed mapping values:
	# 0/1: deny/allow
	# 2/3: deny/allow + logging
	# 
	# Example:
	# Blocking a single hostname:
	# ~^example\.com(:[0-9]+)?$ 
	# --------------------------------------------
	map $connect_target_host $is_granted {
		default 1;								# default allow

~(^|:)fr\.a2dfp\.net(:|$)       0;		# deny
		~(^|:)static\.a-ads\.com(:|$)   2;		# deny + log
	}

# Use Proxy Auth but on CONNECT only
	map $request_method $connect_auth_realm {
        default  off;
        CONNECT  "proxy";
    }

server {
		listen 0.0.0.0:443 ssl;
		listen 0.0.0.0:443 quic reuseport;

server_name example.com;

http2 on;
        http3 on;

http3_stream_buffer_size 256k;
        quic_gso on;

add_header Alt-Svc 'h3=":443"; ma=86400';

limit_conn addr 100;
		limit_conn_status 429;

client_body_buffer_size 256k;
		client_max_body_size 16M;

# Resolver must be set, otherwise will result in 502
		resolver 1.1.1.1 8.8.8.8;

ssl_certificate  fullchain.pem;
		ssl_certificate_key privkey.pem;

tunnel_connect;                        	# Enable tunnel module
		tunnel_connect_buffer_size 64k;         # Buffer size for tunnel relay 
		# tunnel_connect_tcp_nodelay on;        # TCP_NODELAY for CONNECT only

# NGINX 1.30 AND OLDER ONLY
		# tunnel_connect_proxy_auth_user_file /path/to/.htaccess;
		# tunnel_connect_probe_resistance off;
		# tunnel_connect_probe_resistance_allow_methods "";

# NGINX 1.31.0 AND NEWER
		auth_basic $connect_auth_realm;
		auth_basic_user_file /path/to/.htaccess;
		error_page 407 =405 @probe_resistance;

location @probe_resistance {
            # On Auth Failure, this header is still present, use
            # headers more module to remove this header entirely.
            more_clear_headers 'Proxy-Authenticate';
		    return 405;
		}

tunnel_connect_padding off;             # Opt in padding scheme for h2/h3
        tunnel_connect_upstream_timeout 60s;
		tunnel_connect_idle_timeout 120s;

# 0: deny, 1: allow, 2: deny + log, 3: allow + log.
		# $connect_target_host is the effective target authority.
		tunnel_connect_acl_eval_on $is_granted;

tunnel_connect_udp off; 						# Enable connect udp with capsule protocol
		tunnel_connect_udp_path $request_uri; 			# Path variable for parsing masque path

location / {
		    proxy_pass http://localhost:8090;

# To avoid The Discriminative Power of Cross-layer 
			# RTTs in Fingerprinting Proxy Traffic
			# It is recommended to either proxy_pass to a server
			# running on the same nginx instance or
			# run a file server here directly.
		}
    }

server {
		listen 127.0.0.1:8090;

location / {
			return 200 "Hello there!\n";
		}
	}
}
```

## Config Directives

1. `tunnel_connect_pass`: Enable tunnel module, allow HTTP connect.

2. `tunnel_connect_buffer_size`: Size, a symmetric buffer size for tunnel module,
for upstream buffer and client buffer. Default to 64k, with minimum 1k.

3. `tunnel_connect_proxy_auth_user_file`: path to .htaccess file. This will only support
nginx version 1.30 and below.

4. `tunnel_connect_probe_resistance`: Default to off. When auth failed, it will return 405.
Only nginx 1.30 and below are supported.

5. `tunnel_connect_probe_resistance_allow_methods`: Default to an empty string. And will
not send "Allow" header when it is empty, used with `tunnel_connect_probe_resistance`.

6. `tunnel_connect_padding`: Default to off. Enable padding scheme defined by naiveproxy.

7. `tunnel_connect_upstream_timeout`: Time, default to 60s. On timeout, internal server error
will be send. This happens when upstream server has no response.

8. `tunnel_connect_idle_timeout`: Time, default to 120s. On timeout, tunnel will be closed.

9. `tunnel_connect_tcp_nodelay`: Default to off. Enable TCP_NODELAY for CONNECT tunnels
without enabling nginx core `tcp_nodelay` for normal HTTP requests. If nginx core
`tcp_nodelay` is on, CONNECT tunnels already use TCP_NODELAY.

10. `tunnel_connect_acl_eval_on`: Complex value. Accept, integers  from 0-3. 0/1 means access
deny/allow. 2/3 means access deny/allow with logging.

11. `tunnel_connect_udp`: Default to off. With connect udp over MASQUE capsule protocol. Client
send header must present `capsule-protocol = ?1`, otherwise 400 rejected.

12. `tunnel_connect_udp_path`: Complex value, MASQUE encode target host and port in path. Default
to `$request_uri`.

## Performance Comparison

![compare](https://raw.githubusercontent.com/ZihaoFU245/tunnel-benchmark/refs/heads/master/comparison.png)

This compares memory usage (RSS) between Caddy forward proxy and
this nginx tunnel module.

The test is under 5 rounds of 100 tunnels, each transfer data at 1Mbps,
total 100 Mbps for 30 seconds with 5 seconds rest interval.

Nginx is more memory efficient and memory growth is more deterministic.
buffer size is used at 64k, the default value. *Valgrind* shows no
definitely leak, all memory pools are properly deleted on CONNECT finalize.

[**benchmark tools and more details**](https://github.com/ZihaoFU245/tunnel-benchmark)

ngx_http_tunnel_module

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README