toki

KO | EN

gitlite — search

#hacktoberfest #react #llm #golang #go #machine-learning #nodejs #java #windows #cli #awesome-list #security #openai #open-source

(★ 12)

Go internationalization framework

toki Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Toki

Toki is an i18n framework for Go and a Textual Internationalization Key
([**TIK**](https://github.com/romshark/tik))
processor implementation.

`toki generate` parses the source code, lints it reporting any misuse or errors,
extracts localized texts and generates a localization bundle.

[app-resource-bundle (.arb)](https://github.com/google/app-resource-bundle)
files are used as intermediate translation storage for
[ICU messages](https://unicode-org.github.io/icu/userguide/format_parse/messages/)
generated from [TIKs](https://github.com/romshark/tik) extracted from the source code.

`toki webedit` provides a web-based UI for more comfortable manual translation editing.

- [Quick Start Guide](#quick-start-guide)
  - [1. Create a new Go project.](#1-create-a-new-go-project)
  - [2. Prepare your Go project for internationalization.](#2-prepare-your-go-project-for-internationalization)
  - [3. Localize your application for other languages and regions.](#3-localize-your-application-for-other-languages-and-regions)
  - [4. Integrate Toki into your CI/CD pipeline.](#4-integrate-toki-into-your-cicd-pipeline)
- [Domains](#domains)
- [Bundle File Structure](#bundle-file-structure)

## Quick Start Guide

### 1. Create a new Go project.

```sh
mkdir tokiexample && cd tokiexample;
go mod init tokiexample;
```

**tokiexample/main.go**
```go
package main

import (
	"fmt"
	"time"
)

func main() {
	now, numMsgs := time.Now(), 42
	fmt.Printf(
		"It's %s and you have %d message(s)\n",
		now.Format("Monday, Jan 2, 2006"), numMsgs,
	)
}
```

### 2. Prepare your Go project for internationalization.

1. First, we need to generate a new Toki bundle package
with default language set to `en` (English):

```sh
go run github.com/romshark/toki@latest generate -l en && go mod tidy
```

2. Second, you'll need to adjust your messages in the source code to TIKs.
Use the [TIK cheatsheet](https://romshark.github.io/tik-cheatsheet/)
for help and guidance.

```go
package main

import (
	"fmt"

"tokiexample/tokibundle"

"golang.org/x/text/language"
)

func main() {
	now, numMsgs := time.Now(), 42

// Get a localized reader for British English.
	// Toki will automatically select the most appropriate translation catalog available.
	reader, _ := tokibundle.Match(language.BritishEnglish)

// This comment describes the text below and is included in the translator context.
	fmt.Println(reader.String(
		`It's {date-full} and you have {# messages}`,
		now, numMsgs,
	))
}
```

3. Now regenerate your Toki bundle to include the new localized message:

```sh
go run github.com/romshark/toki@latest generate
```

This will update your `catalog_en.arb` file and add the new message to it.
However, Toki can't translate your messages fully so you will see in the generator report
that your translations for `en` are incomplete yet.

To manually complete the translation, add the missing `one {# message}`
[plural option](https://cldr.unicode.org/index/cldr-spec/plural-rules) to the
[ICU message](https://unicode-org.github.io/icu/userguide/format_parse/messages/)
such that it becomes:

```
"msga5a0f2138b9d6598": "It''s {var0, date, full} and you have {var1, plural, one{# message} other {# messages}}",
```

You can also make it a bit better by adding a special case for `0`:

```
"msga5a0f2138b9d6598": "It''s {var0, date, full} and you have {var1, plural, =0{no messages} one{# message} other {# messages}}",
```

**TIP:** The double apostrophe `''` is not a typo, it's
[ICU's escape for a single quote](https://unicode-org.github.io/icu/userguide/format_parse/messages/#quotingescaping).

**TIP:** Use command `toki webedit` to open a browser-based UI for more comfortable translation editing.

4. After tweaking the catalog files, rerun the generator to update your bundle once again:

```sh
go run github.com/romshark/toki@latest generate
```

### 3. Localize your application for other languages and regions.

1. Run the generator, but this time, add a new parameter `-t en-US -t de -t fr`

```sh
go run github.com/romshark/toki@latest generate -t en-US -t de -t fr
```

This will create three new catalogs for
American English (`en-US`), German (`de`) and French (`fr`).

2. You can now provide your translators with the new generated `.arb` files
   or repeat the same procedure as we did for `en` yourself.

3. Once you get your hands on translated `.arb` files, simply replace the ones in the
   Toki bundle package and rerun the generator to update it:

```sh
go run github.com/romshark/toki@latest generate
```

**Coming Soon: LLM-based auto-translator.**

**TIP:** If you're writing your TIKs in a different language than English you may find the
[CLDR plural rules table](https://www.unicode.org/cldr/charts/47/supplemental/language_plural_rules.html)
helpful. Don't worry, Toki will make sure you're not using illegal options for your
default locale and will report any missing options.

### 4. Integrate Toki into your CI/CD pipeline.

You've now (mostly) mastered the entire i18n workflow for Toki.
However, to make sure you never deploy broken or unfinished localizations,
add Toki to your CI/CD setup:

```sh
go run github.com/romshark/toki@latest lint -require-complete
```

You may also use `toki generate -require-complete` and additionally git diff
to ensure your generated Toki bundle package is up to date.

## Domains

Toki supports [TIK domains](https://github.com/romshark/tik/blob/main/SPECIFICATION.md#domains)
via `.tokidomain.yml` files. A domain provides translators and LLMs with additional context
and instructions on how to translate within a directory tree, such as the expected tone,
style, target audience, or terminology conventions. Domains also scope TIK identity,
so identical TIK strings in separate domains are treated as distinct messages.

When you run `toki generate` for the first time, a `.tokidomain.yml` file is automatically
created at your module root using the Go module name.

A `.tokidomain.yml` file is a simple YAML file:

```yaml
# The TIK domain for this directory tree.
# The description provides context for translators working on this domain.
# Also see: https://github.com/romshark/tik/blob/main/SPECIFICATION.md#domains

# This defines the name of the domain.
name: myapp

# The description provides translators with additional context and instructions on
# how to translate within a directory tree, such as the expected tone, style,
# target audience, or terminology conventions.
description: ""
```

Domains can be nested. Place additional `.tokidomain.yml` files in subdirectories
to create sub-domains:

```yaml
myapp/
├─ .tokidomain.yml # myapp
└─ internal/
   ├─ storefront/
   │  ├─ .tokidomain.yml # myapp.storefront
   │  ├─ checkout/
   │  │   └─ .tokidomain.yml # myapp.storefront.checkout
   │  └── catalog/ # (inherits myapp.storefront)
   └─ admin/
      └─ .tokidomain.yml # myapp.admin
```

Directories without a `.tokidomain.yml` file inherit the nearest ancestor's domain.
A sub-domain is a distinct domain from its parent, so the same TIK string
in `myapp` and `myapp.storefront` will produce separate translations.

## Bundle File Structure

- `bundle_gen.go` contains Toki's core source code and package API.
  - **Not editable** 🤖 Any manual change is always overwritten. DO NOT EDIT.
- `catalog_<locale>_gen.go` contains the localized catalog and message writers.
  - **Not editable** 🤖 Any manual change is always overwritten. DO NOT EDIT.
- `catalog_<locale>.arb` is an app resource bundle file containing translations
  in the ICU message format.
  - **Editable 📝**
  - Changed translations are preserved. You may edit translations.
  - If a new message isn't found in the translation file it's automatically added.
  - If a message is no longer used in the source code it's removed from this file.
- `head.txt` is a text file defining the head comment to use in generated files.
  - **Editable 📝**
  - If this file isn't found a new blank file is always automatically created.
    Use this file for any copyright notices and similar purposes.

All other files in the bundle package are ignored and preserved.
`.go` files that don't have the following head comment are always preserved:

```go
// Generated by github.com/romshark/toki. DO NOT EDIT.
```

toki

toki Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

toki

toki Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README