maxhttp

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 12 Open GitHub ↗

Blazing-fast Python HTTP client offering unified sync/async APIs with smart pooling.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# maxhttp

A small, fast, and extensible HTTP/1.1 client library with both synchronous and asynchronous APIs, connection pooling, streaming support, retry/backoff, and automatic decoding of compressed responses.

[![PyPI version](https://badge.fury.io/py/maxhttp.svg)](https://badge.fury.io/py/maxhttp)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![Python Version](https://img.shields.io/pypi/pyversions/maxhttp.svg)](https://pypi.org/project/maxhttp/)

## Table of Contents
- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
  - [Client](#client)
  - [Request](#request)
  - [Response](#response)
  - [Timeouts](#timeouts)
  - [Retry Policy](#retry-policy)
  - [Cookies](#cookies)
- [WebSocket Client](#websocket-client)
- [Advanced Usage](#advanced-usage)
- [File Uploads](#file-uploads)
- [Synchronous Wrapper](#synchronous-wrapper)
- [Error Handling](#error-handling)
- [Benchmarks](#benchmarks)
- [Development](#development)
- [Contributing](#contributing)
- [License](#license)

## Features

- **Sync & Async Support**: Single API that can work synchronously or asynchronously
- **Connection Pooling**: Efficient connection reuse to improve throughput
- **Automatic Compression**: Built-in support for gzip and optional brotli (`br`) compression
- **Multipart Uploads**: Stream-friendly file upload helper with automatic boundary/content-type handling
- **Streaming Support**: Stream large responses without loading everything into memory
- **JSON Handling**: Automatic JSON decoding with charset and compression support
- **Timeout Management**: Flexible timeout configuration for connections and reads
- **Retry & Backoff**: Configurable retry policies with exponential backoff
- **Circuit Breaker**: Optional circuit breaker pattern to prevent cascading failures
- **Cookie Management**: Full cookie jar support
- **HTTP/1.1 Compliant**: Properly implements the HTTP/1.1 specification
- **Optional HTTP/2**: Seamlessly upgrade HTTPS requests to HTTP/2 when the server supports ALPN, with automatic fallback to HTTP/1.1
- **Lightweight**: Minimal dependencies, built on the `h11pro` library

## Installation

Install using pip:

```bash
pip install maxhttp
```

For development with all optional dependencies:

```bash
pip install -e .[dev]
```

Note: Brotli (`br`) support is optional and available via the `brotli` package (included in the `dev` extras). Install with `pip install brotli` for brotli compression support.

## Quick Start

### Synchronous Usage

```python
from maxhttp import Client
from maxhttp.timeouts import Timeout

# Basic GET request
with Client() as client:
    resp = client.get("https://httpbin.org/get")
    print(f"Status: {resp.status_code}")
    print(f"Response: {resp.json()}")

# With custom timeout
with Client(timeout=Timeout(connect=5, read=10)) as client:
    resp = client.get("https://httpbin.org/get")
    print(resp.json())
```

### Asynchronous Usage

```python
import asyncio
from maxhttp import Client
from maxhttp.timeouts import Timeout

async def main():
    # Basic async request
    async with Client() as client:
        resp = await client.get("https://httpbin.org/get")
        print(f"Status: {resp.status_code}")
        print(f"Response: {resp.json()}")
    
    # With custom timeout
    async with Client(timeout=Timeout(connect=5, read=10)) as client:
        resp = await client.get("https://httpbin.org/get")
        print(resp.json())

asyncio.run(main())
```

## API Reference

### Client

The `Client` class provides an asynchronous HTTP client with connection pooling that can be used either synchronously or asynchronously. When used directly, it requires async/await syntax.

```python
from maxhttp import Client

# Basic client
with Client() as client:
    response = client.get("https://api.example.com/users")

# Client with base URL
with Client(base_url="https://api.example.com") as client:
    response = client.get("/users")  # Will make request to https://api.example.com/users

# Client with custom timeout
from maxhttp.timeouts import Timeout
with Client(timeout=Timeout(connect=5, read=10, total=30)) as client:
    response = client.get("https://api.example.com/users")
```

#### Client Methods:
- `get(url, **kwargs)` - Send a GET request (async by default)
- `post(url, **kwargs)` - Send a POST request (async by default)
- `put(url, **kwargs)` - Send a PUT request (async by default)
- `patch(url, **kwargs)` - Send a PATCH request (async by default)
- `delete(url, **kwargs)` - Send a DELETE request (async by default)
- `head(url, **kwargs)` - Send a HEAD request (async by default)
- `options(url, **kwargs)` - Send an OPTIONS request (async by default)
- `request(method, url, **kwargs)` - Send a custom method request (async by default)

### Request

The `Request` class represents an HTTP request.

```python
from maxhttp import Request

# Create a request object
request = Request(
    method="GET",
    url="https://api.example.com/users",
    headers={"Authorization": "Bearer token"},
    params={"page": 1, "limit": 10},
    data={"key": "value"}
)
```

### Response

The `Response` class represents an HTTP response.

```python
from maxhttp import Client

with Client() as client:
    resp = client.get("https://httpbin.org/json")
    
    # Access response properties
    print(f"Status: {resp.status_code}")
    print(f"Headers: {resp.headers}")
    print(f"Text: {resp.text()}")
    print(f"JSON: {resp.json()}")
    
    # Check status
    resp.raise_for_status()  # Raises an exception for 4xx/5xx responses
```

#### Response Properties:
- `status_code` - HTTP status code
- `headers` - Response headers
- `content` - Raw response content as bytes
- `encoding` - Response encoding

#### Response Methods:
- `text()` - Get response as text string
- `json()` - Parse response as JSON
- `raise_for_status()` - Raise an exception for error status codes
- `iter_bytes()` - Iterate over response content in chunks (sync)
- `aiter_bytes()` - Asynchronously iterate over response content in chunks
- `iter_text()` - Iterate over response content as text chunks (sync)
- `aiter_text()` - Asynchronously iterate over response content as text chunks

### Timeouts

Configure connection, read, and total timeouts.

```python
from maxhttp.timeouts import Timeout

# Different timeout configurations
no_timeout = Timeout()  # No timeouts
connect_timeout = Timeout(connect=5)  # 5 seconds to connect
read_timeout = Timeout(read=10)  # 10 seconds to read
full_timeout = Timeout(connect=5, read=10, total=30)  # Full configuration
```

### Retry Policy

Configure retry behavior with optional circuit breaker.

```python
from maxhttp import Client
from maxhttp.retry import RetryPolicy

# Basic retry policy
retry_policy = RetryPolicy(max_attempts=3)

# Advanced retry policy with circuit breaker
retry_policy = RetryPolicy(
    max_attempts=5,
    backoff_factor=0.5,
    status_codes=[500, 502, 503, 504],
    circuit_breaker=True,
    cb_failure_threshold=3,
    cb_recovery_seconds=60
)

with Client(retry=retry_policy) as client:
    resp = client.get("https://api.example.com/resource")
```

#### Retry Policy Parameters:
- `max_attempts` - Maximum number of attempts (including the initial request)
- `backoff_factor` - Factor for exponential backoff between retries
- `status_codes` - List of HTTP status codes that should trigger a retry
- `circuit_breaker` - Whether to enable the circuit breaker
- `cb_failure_threshold` - Number of failures before opening the circuit
- `cb_recovery_seconds` - Seconds to wait before attempting to close the circuit

### Cookies

Manage cookies with the `CookieJar`.

```python
from maxhttp import Client
from maxhttp.cookies import CookieJar

# Create a cookie jar
jar = CookieJar()
jar.set("session_id", "abc123", domain="example.com", path="/", secure=True)

with Client(cookies=jar) as client:
    resp = client.get("https://example.com/protected")
```

## WebSocket Client

maxhttp ships with a lightweight WebSocket client powered by [wsproto](https://github.com/python-hyper/wsproto). It integrates with the HTTP client's timeout settings, supports the async context-manager pattern, and offers helpers for text, binary, and JSON payloads.

### Quick example

```python
import asyncio
from maxhttp import WebSocket

async def main():
    async with WebSocket.connect("wss://echo.websocket.org") as ws:
        await ws.send_text("hello")
        print("Text echo:", await ws.recv_text())

await ws.send_bytes(b"\x00\x01")
        print("Binary echo:", await ws.recv_bytes())

await ws.send_json({"ping": True})
        try:
            print("JSON echo:", await ws.recv_json())
        except WebSocketDecodeError:
            print("Peer did not return JSON.")

asyncio.run(main())
```

### API overview

| Method | Description |
| ------ | ----------- |
| `WebSocket.connect(url, *, headers=None, subprotocols=None, extensions=None, timeout=None, verify=True, ssl_context=None)` | Returns an awaitable/context-manager connector that establishes the WebSocket handshake. |
| `send_text(str, *, final=True)` / `send_bytes(bytes, *, final=True)` / `send_json(obj, *, dumps=None)` | Send text, binary, or JSON frames. |
| `recv()` | Receive the next complete message (returns `str` for text, `bytes` for binary). |
| `recv_text()` / `recv_bytes()` / `recv_json(*, loads=None)` | Typed helpers that validate the payload and raise `WebSocketMessageTypeError` or `WebSocketDecodeError` when the data does not match expectations. |
| `ping(payload=b"")` / `close(code=1000, reason=None)` | Control frames and graceful shutdown. |
| `async for message in ws:` | Iterate over messages until the connection closes. |

### Using through `Client`

The HTTP client exposes a convenience helper that reuses shared timeout defaults and base URLs:

```python
async with Client(base_url="wss://example.org") as client:
    async with await client.websocket("/chat") as ws:
        await ws.send_text("hi!")
        async for message in ws:
            print("Incoming:", message)
```

### Error handling

WebSocket-specific exceptions live in `maxhttp.errors`:

- `WebSocketHandshakeError`: Upgrade/handshake failures.
- `WebSocketClosed`: Connection is already closed (contains `code` and `reason`).
- `WebSocketProtocolError`: Invalid URLs or protocol violations.
- `WebSocketMessageTypeError`: Payload type mismatch (e.g., expected text but received binary).
- `WebSocketDecodeError`: JSON or UTF-8 decoding failures for binary/text frames.
- `WebSocketError`: Base class for all WebSocket-related issues.

Wrap interactions with `try/except` to surface meaningful errors to your users:

```python
try:
    async with WebSocket.connect("wss://example.org/socket") as ws:
        ...
except WebSocketHandshakeError as exc:
    print("Server rejected upgrade:", exc)
except WebSocketDecodeError as exc:
    print("Bad payload:", exc)
```

## Advanced Usage

### Authentication

maxhttp ships with pluggable authentication handlers that mirror the HTTP exchanges performed by web servers.

```python
from maxhttp import Client, BasicAuth, DigestAuth, AuthBase

# 1) Basic authentication (tuple shorthand)
with Client(base_url="https://api.example.com", auth=("user", "pass")) as client:
    resp = client.get("/basic-auth")
    resp.raise_for_status()

# 2) Digest authentication
digest = DigestAuth("digest-user", "digest-pass")
with Client(base_url="https://api.example.com") as client:
    resp = client.get("/digest-endpoint", auth=digest)
    resp.raise_for_status()

# 3) Custom authentication by subclassing AuthBase
class APIKeyAuth(AuthBase):
    def __init__(self, key: str) -> None:
        self.key = key

def _on_request(self, request):
        request.headers["X-API-Key"] = self.key

with Client(base_url="https://api.example.com") as client:
    resp = client.get("/custom-auth", auth=APIKeyAuth("my-secret"))
    resp.raise_for_status()
```

Authentication handlers receive every outgoing request (`on_request`) and can optionally inspect responses (`on_response`) to perform challenge/response flows (Digest auth does this automatically). You can pass the handler globally via `Client(..., auth=...)` or per-call via `client.get(..., auth=...)`.

#### HTTP/2 Preference & Per-Request Overrides

maxhttp can opportunistically negotiate HTTP/2 on HTTPS connections via [hyper-h2](https://python-hyper.org/projects/h2/en/stable/). Install `h2` (e.g. `pip install h2`) and set `http2=True` on the client to opt-in globally:

```python
async with Client(http2=True) as client:
    resp = await client.get("https://http2.golang.org/reqinfo")
```

You can also override the preference per request:

```python
resp = await client.get("https://example.com/data", http2=False)   # force HTTP/1.1
resp = await client.get("https://example.com/data", http2=True)    # force HTTP/2 attempt
```

If the server declines HTTP/2 during ALPN, maxhttp transparently falls back to HTTP/1.1 and retries the request.

### Response Hooks

maxhttp exposes a `hooks` argument mirroring the Requests API so you can register callbacks that observe (or replace) responses. Hooks can be provided globally when instantiating `Client` or passed for individual requests:

```python
from maxhttp import Client

def log_status(resp):
    print("Got status:", resp.status_code)

async def uppercase_hook(resp):
    # Hooks may be async and may return a replacement Response
    from maxhttp import Response
    return Response(
        status_code=resp.status_code,
        headers=resp.headers,
        content=resp.text().upper().encode("utf-8"),
        reason=resp.reason,
        request=resp.request,
    )

with Client(base_url="https://api.example.com", hooks={"response": log_status}) as client:
    resp = client.get("/resource", hooks={"response": [log_status, uppercase_hook]})
    resp.raise_for_status()
```

Hook callbacks receive the `Response` object, may be sync or async, and can return either `None` (no change) or a new `Response` instance which replaces the current one. Multiple hooks can be registered per event; `response` is currently the supported event.

### Streaming Large Responses

```python
from maxhttp import Client

with Client() as client:
    resp = client.get("https://httpbin.org/stream/20", stream=True)
    
    # Process response in chunks
    for chunk in resp.iter_bytes():
        print(f"Received chunk: {len(chunk)} bytes")
```

### Async Streaming

```python
import asyncio
from maxhttp import Client

async def stream_example():
    async with Client() as client:
        resp = await client.get("https://httpbin.org/stream/20", stream=True)
        
        # Process response in chunks
        async for chunk in resp.aiter_bytes():
            print(f"Received chunk: {len(chunk)} bytes")

asyncio.run(stream_example())
```

### Custom Headers and Parameters

```python
from maxhttp import Client

with Client() as client:
    # With custom headers and query parameters
    resp = client.get(
        "https://httpbin.org/headers",
        headers={"User-Agent": "MyApp/1.0"},
        params={"key": "value"}
    )
    print(resp.json())
```

### POST Request with Data

```python
from maxhttp import Client

with Client() as client:
    # Send JSON data
    resp = client.post(
        "https://httpbin.org/post",
        json={"key": "value"}
    )
    print(resp.json())
    
    # Send form data
    resp = client.post(
        "https://httpbin.org/post",
        data={"field": "value"}
    )
    print(resp.json())
```

### Circuit Breaker

Enable the circuit breaker to prevent cascading failures:

```python
from maxhttp import Client
from maxhttp.retry import RetryPolicy

# Configure circuit breaker
retry_policy = RetryPolicy(
    max_attempts=1,  # Only try once before circuit breaker takes over
    circuit_breaker=True,
    cb_failure_threshold=3,      # Open circuit after 3 failures
    cb_recovery_seconds=60       # Wait 60 seconds before trying again
)

with Client(base_url="https://api.example.com", retry=retry_policy) as client:
    resp = client.get("/resource")
```

## File Uploads

maxhttp natively handles multipart file uploads via the `files` argument or the standalone `MultipartEncoder`.

```python
from maxhttp import Client

async with Client(base_url="https://api.example.com") as client:
    files = {
        "avatar": ("photo.jpg", open("photo.jpg", "rb")),
        "metadata": ("meta.json", b'{"public": true}', "application/json"),
    }
    data = {"user_id": "123"}
    resp = await client.post("/upload", data=data, files=files)
    resp.raise_for_status()
```

Key details:

1. `files` accepts dictionaries or lists of `(filename, content[, content_type])` tuples. Content can be bytes, strings, paths, sync/async streams, or async iterators.
2. When `files` is provided, bodies are streamed and `Content-Length` is automatically managed (chunked transfer).
3. Use `maxhttp.formdata.MultipartEncoder` directly for custom pipelines:

```python
from maxhttp.formdata import MultipartEncoder

encoder = MultipartEncoder(
    fields={"description": "Sample upload"},
    files={"file": ("large.bin", open("large.bin", "rb"), "application/octet-stream")},
)

resp = await client.post("/upload", content=encoder.iter_bytes(), headers={
    "Content-Type": encoder.content_type,
})
```

This integration ensures efficient uploads for multi-gigabyte files without loading them entirely into memory.

## Synchronous Wrapper

The library provides a synchronous wrapper for async classes:

```python
from maxhttp import Client

# Now you can use async classes synchronously
with Client(base_url="https://api.example.com") as client:
    resp = client.get("/users")  # No await needed
    data = resp.json()
```

## Error Handling

The library provides specific exception types for different error conditions:

```python
from maxhttp import Client
from maxhttp.errors import (
    MaxHTTPError,
    RequestError,
    ResponseError,
    HTTPStatusError,
    PoolError
)

with Client() as client:
    try:
        resp = client.get("https://httpbin.org/status/500")
        resp.raise_for_status()  # Raises HTTPStatusError for 5xx responses
    except HTTPStatusError as e:
        print(f"HTTP error occurred: {e}")
    except RequestError as e:
        print(f"Request error occurred: {e}")
    except MaxHTTPError as e:
        print(f"General MaxHTTP error occurred: {e}")
```

### Available Exceptions:
- `MaxHTTPError` - Base exception class
- `RequestError` - Errors during request processing
- `ResponseError` - Errors during response processing
- `HTTPStatusError` - HTTP error status codes (4xx, 5xx)
- `PoolError` - Connection pool errors

## Benchmarks

The repository ships with a heavy-load benchmarking harness that compares maxhttp against popular Python HTTP clients under identical settings.

### How to run

```bash
python benchmark.py \
  --url https://httpbin.org/get \
  --duration 15 \
  --concurrency 64 \
  --body-size 1024 \
  --warmup 10
```

Use `--help` to discover additional flags (custom headers, TLS toggle, payload sizes, latency sampling controls, etc.). Ensure `maxhttp`, `aiohttp`, `httpx`, and `requests` are installed in the active environment.

### Latest results

_Hardware/Network_: User laptop, residential network. Results may vary with different environments, servers, or payloads.

| Library   | Requests | Success | Req/s | Avg Lat (ms) | P95 Lat (ms) | P99 Lat (ms) | Mbps |
|-----------|----------|---------|------:|-------------:|-------------:|-------------:|-----:|
| maxhttp  | 1.5K     | 1.5K    |    92 |        649.7 |       1635.6 |       2380.5 | 0.22 |
| requests  | 1.2K     | 1.2K    |    64 |        812.7 |       2653.1 |       4668.8 | 0.17 |
| aiohttp   | 1.1K     | 1.1K    |    63 |        937.2 |       2123.0 |       3039.7 | 0.16 |
| httpx     | 0.5K     | 0.5K    |    23 |       1930.9 |       3438.8 |       5803.9 | 0.06 |

Configuration: GET https://httpbin.org/get, 15s per client, concurrency=64, 1KB payload, TLS verification enabled, 10 warmup hits per client.

> **Note:** maxhttp is under active development; rerun the benchmark after significant changes or on infrastructure that matches your production constraints for more representative numbers.

## Development

### Setup

```bash
# Clone the repository
git clone https://github.com/shayanheidari01/maxhttp.git
cd maxhttp

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e .[dev]
```

### Running Tests

```bash
# Run all tests
python -m pytest

# Run tests with verbose output
python -m pytest -v

# Run specific test file
python -m pytest test.py
```

### Code Quality

The project uses `ruff` for linting and `mypy` for type checking:

```bash
# Run linter
ruff check .

# Run type checker
mypy maxhttp/
```

## Contributing

Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for detailed contribution guidelines.

## License

This project is licensed under the GNU General Public License v3 (GPLv3). See the [LICENSE](LICENSE) file for details.

maxhttp

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README