KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #codex #claude-code #mcp #rust #deepseek-harness #typescript #ai-agent #cli

mlx-swift-structured

★ 76 Open GitHub ↗

Structured output generation in Swift

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MLX Structured

[MLX](https://github.com/ml-explore/mlx-swift) Structured is a Swift library for structured output generation using constrained decoding. It's built on top of the [XGrammar](https://github.com/mlc-ai/xgrammar) library, which provides efficient, flexible, and portable structured generation. You can learn more about the XGrammar algorithm in their [technical report](https://arxiv.org/abs/2411.15100).

## Installation

To use MLX Structured in your project, add the following to your `Package.swift` file:

```swift
dependencies: [
    .package(url: "https://github.com/petrukha-ivan/mlx-swift-structured", from: "0.2.0")
]
```

Don't forget to add the library as a dependency for your targets:

```swift
dependencies: [
    .product(name: "MLXStructured", package: "mlx-swift-structured")
]               
```

## Usage

### Grammar

Start by defining a `Grammar`. You can use JSON Schema to describe the desired output:

```swift
let schema = JSONSchema.object(
    description: "Person info",
    properties: [
        "name": .string(),
        "age": .integer()
    ], required: [
        "name",
        "age"
    ]
)

let grammar = try Grammar.schema(schema)
```

Starting with macOS 26 and iOS 26, you can use a `@Generable` type as a grammar source:

```swift
@Generable
struct PersonInfo: Codable {
    
    @Guide(description: "Person name")
    let name: String
    
    @Guide(description: "Person age")
    let age: Int
}

let grammar = try Grammar.generable(PersonInfo.self)
```

You can also use a regex:

```swift
let regex = #"^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$"# // Simple email regex
let grammar = Grammar.regex(regex)
```

Or define your own grammar rules with [EBNF](https://en.wikipedia.org/wiki/Extended_Backus–Naur_form) syntax:

```swift
let ebnf = #"root ::= ("YES" | "NO")"# // Answer only "YES" or "NO"
let grammar = Grammar.ebnf(ebnf)
```

### Complex Grammar

You can define rich, composable grammar rules via a grammar builder. This enables you to describe structured output formats precisely:

```swift
let grammar = try Grammar {
    SequenceFormat {
        ConstTextFormat(text: "Hello!")
        OrFormat {
            JSONSchemaFormat(...)
            RegexFormat(...)
        }
    }
}
```

This can be used in different ways. Here is an example of a constrained Qwen3 tool-calling format:

```swift
let grammar = try Grammar {
    SequenceFormat {
        if forceThinking {
            TagFormat(begin: "<think>", end: "</think>") {
                AnyTextFormat()
            }
        }
        TriggeredTagsFormat(triggers: ["<tool_call>"], options: [.atLeastOne, .stopAfterFirst]) {
            for tool in tools {
                TagFormat(begin: "<tool_call>\n{\"name\": \"\(tool.name)\", \"arguments\": ", end: "}\n</tool_call>") {
                    JSONSchemaFormat(schema: tool.parameters)
                }
            }
        }
    }
}
```

### Generation

To use a defined grammar during text generation, use the convenient `generate` method. These overloads are fully compatible with MLXLM generation APIs, with the grammar passed as an additional argument:

```swift
let stream = try await generate(
    input: input, 
    context: context, 
    grammar: grammar
)

for await generation in stream {
    switch generation {
    case .chunk(let chunk):
        print(chunk, terminator: "")
    case .toolCall(let toolCall):
        // Handle tool call
    case .info(let info):
        // Handle completion info
    }
}
```

You can also decode constrained JSON output into a `Decodable` type:

```swift
let model = try await generate(
    input: input, 
    context: context, 
    schema: schema,
    generating: PersonInfo.self
)
```

With a `Generable` type, you can generate a validated value directly:

```swift
let model = try await generate(
    input: input, 
    context: context,
    generating: PersonInfo.self
)
```

You can also stream partial `Generable` updates, which return `PartiallyGenerated` content for your type:

```swift
let stream = try await generate(
    input: input, 
    context: context, 
    partially: PersonInfo.self
)

for try await content in stream {
    print("Partially generated:", content)
}
```

To enable jump forwarding, pass the `.jumpForwarding` option to any of `generate` function:

```swift
let stream = try await generate(
    input: input,
    options: .jumpForwarding,
    context: context,
    grammar: grammar
)
```

You can find more usage examples in the `MLXStructuredCLI` target and in the unit tests.

## Experiments

### Performance

Constrained decoding has effectively **zero-overhead**, so you can produce valid output without sacrificing generation speed. Moreover, in some cases, jump forwarding can make generation even faster by skipping model calls when the output is already determined by the current grammar state. For example, the model can skip predictable JSON syntax such as keys, brackets, and whitespace while generating tokens only for field values.

| Model | Plain (tokens/s) | Constrained (tokens/s) | Jump forwarding (tokens/s) |
| - | - | - | - |
| Qwen3-0.6B-4bit | 394.1 | 394.3 | 604.4 |
| Qwen3-4B-4bit | 98.9 | 98.3 | 173.1 |
| Qwen3-8B-4bit | 58.1 | 58.1 | 100.9 |
| Qwen3-14B-4bit | 32.8 | 32.4 | 51.0 |
| Qwen3-32B-4bit | 14.4 | 14.5 | 23.6 |

These benchmark results were collected on an Apple M3 Max using the `MLXStructuredCLI` benchmark command and the movie record extraction example from the next section. The speedup from jump forwarding depends on the grammar and the ratio of static to dynamic output. In the benchmark example, nearly half of the output is predefined syntax, so jump forwarding can approach almost a 2x speedup.

### Accuracy

For example, given a task to extract a movie record from text and output it in JSON format, the prompt is:

```plain
Instruction: Extract movie record from the text, output in JSON format according to schema: \(schema)
Text: The Dark Knight (2008) is a superhero crime film directed by Christopher Nolan. Starring Christian Bale, Heath Ledger, and Michael Caine.
```

And the grammar definition looks like this:

```swift
let grammar = try Grammar.schema(.object(
    description: "Movie record",
    properties: [
        "title": .string(),
        "year": .integer(minimum: 1900, maximum: 2026),
        "genres": .array(items: .string(), maxItems: 3),
        "director": .string(),
        "actors": .array(items: .string(), maxItems: 5)
    ], required: [
        "title",
        "year",
        "genres",
        "director",
        "actors"
    ]
))
```

For large proprietary models like ChatGPT, this is not a problem. With the right prompt, they can successfully generate valid JSON even without constrained decoding. However, with smaller models like Gemma3 270M (especially when quantized to 4-bit), the output almost always contains invalid JSON, even if the schema is provided in the prompt.

```plain
[
  "title": "The Dark Knight",
  "actors": [
    "Christian Bale",
    "Heath Ledger",
    "Michael Caine"
  ],
  "genre": "crime",
  "director": "Christopher Nolan",
  "actors": [
    "Christian Bale",
    "Heath Ledger",
    "Michael Caine"
  ],
  "description": "The Dark Knight is a superhero crime film directed by Christopher Nolan. Starring Christian Bale, Heath Ledger, Michael Caine."
]
```

This output has several issues:

- Root starts with `[` instead of `{`
- Incorrect key and type for `genres` field
- Missing required `year` field
- Duplicated `actors` field
- Extra `description` field

Here is the output using constrained decoding:

```plain
{
  "title": "The Dark Knight",
  "year": 2008,
  "genres": [
    "superhero",
    "crime"
  ],
  "director": "Christopher Nolan",
  "actors": [
    "Christian Bale",
    "Heath Ledger",
    "Michael Caine"
  ]
}
```

The output is fully valid JSON that exactly matches the provided schema. This shows that, with the right approach, even small models like Gemma3 270M 4-bit (which is just 150 MB) can produce correct structured output.

## Troubleshooting

This library is still in an early stage of development. While it is already functional, it may have unexpected issues or even crash your program. If you encounter a problem, please create an issue or open a pull request. Contributions are welcome!

mlx-swift-structured

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README