KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

fonnx

★ 296 Open GitHub ↗

ONNX runtime for Flutter.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="header.png"
     alt="FONNX image header, bird like Flutter mascot DJing. Text reads: FONNX. Any model
on any edge. Run ONNX model & runtime, with platform-specific acceleration,  inside Flutter, a modern, beautiful, cross-platform development
framework."
     style="float: left; margin-right: 0px;" />
| Platform     | Status |
|--------------|--------|
| __Android__  | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/android-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/android-build/latest_build) |
| __iOS__  | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/ios-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/ios-build/latest_build) |
| __Linux__    | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/linux-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/linux-build/latest_build) |
| __macOS__ | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/macos-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/macos-build/latest_build) |
| __Web__  | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/web-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/web-build/latest_build) |
| __Windows__  | [![Codemagic build status](https://api.codemagic.io/apps/652897766ee3f7af8490a79f/windows-build/status_badge.svg)](https://codemagic.io/app/652897766ee3f7af8490a79f/windows-build/latest_build) |

# Changelog

## 2026 Jul 20
- Upgraded ONNX Runtime from 1.16.1 to 1.27.0 on every native platform.
- Replaced vendored binaries, CocoaPods, Maven runtime dependencies, and
  Kotlin/Swift platform channels with SHA-256-verified native code assets and
  one Dart FFI inference implementation.
- Added Linux aarch64 support for Raspberry Pi/flutter-pi.
- Pinned a selected-op ONNX Runtime Extensions build for Whisper's
  `ai.onnx.contrib:BpeDecoder`; Magika and Pyannote were verified as core-ORT
  models and no longer register Extensions.
- Native support is Android armv7/arm64/x64, iOS arm64 device/simulator, Linux
  x64/aarch64, macOS arm64, and Windows x64/arm64. Intel macOS is dropped.

## 2026 Jul 19
- Upgraded voice activity detection to the official Silero VAD v6.2.1 model.
- Added the v6 streaming state/context contract on native, mobile, and web.

## 2024 Apr 22
- CI builds for all platforms are now running on Codemagic..

## 2024 Feb 26
- Google's [Magika](https://google.github.io/magika/) for file identification supported on all platforms.
- Example app including full voice assistant flow, with Whisper, Silero voice activity detection. Available at [telosnex.github.io/fonnx/](https://telosnex.github.io/fonnx/)

## 2024 Feb 19
- Whisper supported on all platforms, including web.

## 2024 Feb 13

- Whisper now supported on all platforms besides web.
- Whisper models support timestamps. (not exposed via API, yet)
- Silero VAD added to all platforms besides web.
- Silero VAD enables detecting when the user is done speaking with a much higher success rate than relying on volume levels.
- Example contains `SttService`, an example of how to integrate the VAD and Whisper together with an easy to use interface. (Stream<String>)

# FONNX

## Any model on any edge

Run ML models natively on any platform. ONNX models can be run on iOS, Android, Web, Linux, Windows, and macOS.

## What is FONNX?

FONNX is a Flutter library for running ONNX models.
Flutter, and FONNX, run natively on iOS, Android, Web, Linux, Windows, and macOS.
FONNX leverages [ONNX](https://onnx.ai/) to provide native acceleration capabilities, from CoreML on iOS, to Android Neural Networks API on Android, to WASM SIMD on Web.
Most models can be easily converted to ONNX format, including models from Pytorch, Tensorflow, and more.

## Getting ONNX Models

### Hugging Face

[🤗 Hugging Face](https://huggingface.co/models) has a large collection of models, including many that are ONNX format. 90% of the models are Pytorch, which can be converted to ONNX.

Here is a search for [ONNX models](https://huggingface.co/models?sort=trending&search=onnx).

### Export ONNX from Pytorch, Tensorflow, & more

A command-line tool called `optimum-cli` from HuggingFace converts Pytorch and Tensorflow models. This covers the vast majority of models. `optimum-cli` can also quantize models, significantly reduce model size, usually with negligible impact on accuracy.

See [official documentation](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model) or the
quick start [snippet on GitHub](https://github.com/huggingface/optimum#run-the-exported-model-using-onnx-runtime).  
Another tool that automates conversion to ONNX is [HFOnnx](https://neuml.github.io/txtai/pipeline/train/hfonnx/). It was used to export the text embeddings models in this repo. Its advantages included a significantly smaller model size, and incorporating post-processing (pooling) into the model itself.

- Brief intro to how ONNX model format & runtime work [huggingface.com](https://huggingface.co/docs/optimum/onnxruntime/concept_guides/onnx)
- [Netron](https://netron.app/) allows you to view ONNX models, inspect their runtime graph, and export them to other formats

### Text Embeddings

These models generate embeddings for text.
An embedding is a vector of floating point numbers that represents the meaning of the text.  
Embeddings are the foundation of a vector database, as well as retrieval augmented generation - deciding which text snippets to provide in the limited context window of an LLM like GPT.

Running locally using FONNX provides significant privacy benefits, as well as latency benefits.
For example, rather than having to store the embedding and text of each chunk of a document on a server, they can be stored on-device.
Both MiniLM L6 V2 and MSMARCO MiniLM L6 V3 are both the product of the Sentence Transformers project. Their website has excellent documentation explaining, for instance, [semantic search](https://www.sbert.net/examples/applications/semantic-search/README.html)

#### MiniLM L6 V2

Trained on a billion sentence pairs from diverse sources, from Reddit to WikiAnswers to StackExchange.
MiniLM L6 V2 is well-suited for numerous tasks, from text classification to semantic search.
It is optimized for [symmetric search](https://www.sbert.net/examples/applications/semantic-search/README.html#symmetric-vs-asymmetric-semantic-search), where text is roughly of the same length and meaning.
Input text is divided into approximately 200 words, and an embedding is generated for each.  
[🤗 Hugging Face](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)

#### MSMARCO MiniLM L6 V3

Trained on pairs of Bing search queries to web pages that contained answers for the query.
It is optimized for [asymmetric semantic search](https://www.sbert.net/examples/applications/semantic-search/README.html#symmetric-vs-asymmetric-semantic-search), matching a search query to an answer.
Additionally, it has 2x the input size of MiniLM L6 V2: it can accept up to 400 words as input for one embedding.  
[🤗 Hugging Face](https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3/tree/main)

#### Benchmarks

**iPhone 14**: 67 ms  
**Pixel Fold**: 33 ms  
**macOS**: 13 ms  
**WASM SIMD**: 41 ms

Avg. ms for 1 Mini LM L6 V2 embedding / 200 words.

- Run on Thurs Oct 12th 2023.
- macOS and WASM-SIMD runs on MacBook Pro M2 Max.
- Average of 100 embeddings, after a warmup of 10.
- Input is mix of lorem ipsum text from 8 languages.

# Integrating FONNX

## Native platforms via Dart FFI and code assets

Android, iOS, Linux, macOS, and Windows use the same Dart FFI implementation.
`hook/build.dart` selects the target artifact, downloads it into a
content-addressed cache, verifies its pinned SHA-256, extracts the library, and
emits a bundled Flutter code asset. There are no FONNX CocoaPods/Gradle native
dependencies and no platform channels.

Microsoft's published iOS artifact is static and cannot be loaded as a Dart
code asset. FONNX release CI runs Microsoft's official Apple framework script
in its supported dynamic mode and publishes arm64 device/simulator artifacts
for the hook. iOS 15.1 or newer is required. macOS 14 or newer and Apple
Silicon are required; Intel support was intentionally dropped.

### Required iOS project setup

Set both the Runner deployment target and `platform :ios` in the Podfile to
15.1 or newer. Flutter 3.44.2's iOS native-assets pipeline generates framework
`Info.plist` files with Flutter's iOS 13 baseline. It does not currently derive
that value from a consuming Runner project's higher deployment target. The
related upstream history is [flutter/flutter#148044](https://github.com/flutter/flutter/issues/148044),
and Flutter's source TODO points to the still-open
[flutter/flutter#145104](https://github.com/flutter/flutter/issues/145104).

ORT 1.27's Apple Mach-O binaries require iOS 15.1. App Store Connect rejects a
framework that advertises a lower minimum than its binary with `ITMS-90208`.
Until Flutter propagates the app target into native-asset packaging, add a Run
Script phase immediately **after** Flutter's `Thin Binary` phase. The phase
must set `MinimumOSVersion` on `onnxruntime.framework` and
`ortextensions.framework` to `IPHONEOS_DEPLOYMENT_TARGET`, verify that this
minimum is not lower than any embedded Mach-O minimum, and re-sign the modified
frameworks. The complete fail-closed
implementation is in
[`example/ios/fix_fonnx_framework_minimum_os.sh`](example/ios/fix_fonnx_framework_minimum_os.sh),
and the example Xcode project shows the required phase ordering.

ONNX Runtime Extensions is also a separately bundled code asset. The build is
selected to the one custom operator in the current model inventory:
Whisper's `ai.onnx.contrib:BpeDecoder`.

## Web

Sending these headers with the request for the ONNX JS package gives a 10x speedup:

```
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin
```

See [this GitHub issue](https://github.com/nagadomi/nunif/issues/34) for details. TL;DR: It allows use of multiple threads by ONNX's WASM implementation by using a SharedArrayBuffer.

### Developing with Web

While developing, two issues prevent it work working on the web.
Both have workarounds

#### WASM Mime Type

You may see errors in console logs about the MIME type of the
.wasm being incorrect and starting with the wrong bytes.

That is due to local Flutter serving of the web app.

To fix, download the WASM files from the same CDN folder that hosts ort.min.js
(see _\_worker.js) and also in _\_minilm_worker.js, remove the // in front of ort.env.wasm.wasmPaths = "".

Then, place the WASM files downloaded from the CDN next to index.html.

In release mode and deployed, this is not an issue, you do not need to host the WASM files.

#### Cross-Origin-Embedder-Policy

To safely use SharedArrayBuffer, the server must send the Cross-Origin-Embedder-Policy header with the value require-corp.

See here for how to workaround it: https://github.com/nagadomi/nunif/issues/34

Note that the extension became adware, you should have Chrome set up its
permissions such that it isn't run until you click it. Also, note that you have to do
that each time the Flutter web app in debug mode's port changes.

# License

FONNX is licensed under a dual-license model.

The code as-is on GitHub is licensed under GPL v2. That requires distribution of the integrating app's source code, and this is unlikely to be desirable for commercial entities. See LICENSE.md.

Commercial licenses are also available. Contact info@telosnex.com. Expect very fair terms: our intent is to charge only entities, with a launched app, making a lot of money, with FONNX as a core dependency. The base agreement is here: https://github.com/lawndoc/dual-license-templates/blob/main/pdf/Basic-Yearly.pdf

fonnx

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README