KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agent #developer-tools #codex #typescript #cli #deepseek-harness #claude-code #ai-agents #mcp #rust

QuanSyn

★ 40 Open GitHub ↗

QuanSyn: A Python Package for Quantitative Syntax Analysis.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

---

# QuanSyn Documentation (English)

`QuanSyn` is a Python package designed for quantitative syntax analysis. Currently, it includes three modules: `depval`, `lingnet`, and `lawfitter`.

- `Depval` provides various methods to calculate dependency-based metrics, including Mean Dependency Distance (MDD), Mean Hierarchy Distance (MHD), Dependency Direction, Dependency Distance Distribution, etc. It also offers a convertor for treebank formats such as `conllu`, `conll`, `pmt`, and `mcdt`.
  
- `Lingnet` can quickly convert a treebank into syntactic dependency networks and co-occurrence networks for language system modeling and language network analysis.
  
- `Lawfitter` can quickly fit data to a target mathematical model and detect potential structural patterns in languages.

It also be packed as applications with UIs(QuanSyn Mini/QuanSyn Studio,https://github.com/YuhuYang/QuanSyn_app/releases/tag/v0.0.1).
---

## Table of Contents

- [Terms and Concepts](#terms-and-concepts)
- [Installation](#installation)
- [Quick Start](#quick-start) (for beginners)
- [Usage](#usage) (for experts)
- [License](#license)
- [Contact](#contact)
- [Citing](#citing)

---

## Terms and Concepts

1. **Metric Hierarchy in Depval**

!['Dependency Directed Graph'](https://raw.githubusercontent.com/YuhuYang/QuanSyn/main/figure/example.jpg)
(This image is sourced from: Chen, R., Deng, S., & Liu, H. (2021). Syntactic Complexity of Different Text Types: From the Perspective of Dependency Distance Both Linearly and Hierarchically. *Journal of Quantitative Linguistics, 29*(4), 510-540. https://doi.org/10.1080/09296174.2021.2005960)

Take Dependency Distance (dd) as an example, which can be used at the word, sentence, and discourse levels.  
In `QuanSyn`, the word-level dd refers to the linear distance between a target word and its head word when the target word is a dependent.  
The sentence-level mdd refers to the mean dependency distance within a sentence.  
The text-level mdd refers to the mean dependency distance in a treebank.  
The "level" refers to the subject of the statistical metric, which changes the calculation at each level. The specifics are as follows:

- **Word-level**: dd, hd, ddir, v  
    - dd: Dependency Distance, the linear distance between a target word and its governor. For example, the dd between *Bush* and *will* is 1.  
    - hd: Hierarchical Distance, the Euclidean distance between a target word and its governor in a tree. For example, the hd of *Bush* is 1.  
    - ddir: Dependency Direction, the direction of the dependency relation. 
    - v: Valency, the dynamic generalized valency of a target word, referring to the number of words that link to the target word. For example, *Bush* only has a dependency with *will*, so v = 1.

- **Sentence-level**: mdd, ndd, mhd, mhdd, tdl, sl, mv, vk, tw, th, hi, hf, rd  
    - mdd: Mean Dependency Distance, the average of all dependency distances in a sentence. The MDD of the sentence in the figure is (8+1+1+3+1+1+1+1+1+2+1+1+1+2)/14 = 1.79.  
        - $mdd = \frac{1}{sl-1}\sum_{i=1}^{sl-1}dd_i$
        - Liu, H. (2008). Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2), 159-191.
    - mhd: Mean Hierarchical Distance, the average of all hierarchical distances in a sentence. The MHD of the sentence in the figure is (2+5+4+3+4+5+2+2+3+4+4+5+6+6)/14 = 3.93.  
        - $mhd = \frac{1}{sl-1}\sum_{i=1}^{sl-1}hd_i$
        - Jing, Y., & Liu, H. (2015, August). Mean hierarchical distance augmenting mean dependency distance. In Proceedings of the third international conference on dependency linguistics (Depling 2015) (pp. 161-170).
    - mhdd: Mean Hierarchy Dependency Distance, the number of dependency relationships divided by the number of hierarchical levels in the sentence. The mhdd for the sentence in the figure is 14/6 = 2.33.  
        - $mhdd = \frac{sl-1}{MAXHL}$
        - Chen, R., Deng, S., & Liu, H. (2021). Syntactic Complexity of Different Text Types: From the Perspective of Dependency Distance Both Linearly and Hierarchically. Journal of Quantitative Linguistics, 29(4), 510-540. 
    - tdl: Total Dependency Length, the sum of all dependency distances in a sentence. The TDL for the sentence in the figure is 25.  
        - $tdl = \sum_{i=1}^{sl-1}dd_i$
        - Futrell, R., Mahowald, K., & Gibson, E. (2015). Large-scale evidence of dependency length minimization in 37 languages. Proceedings of the National Academy of Sciences, 112(33), 10336-10341.    
    - sl: Sentence Length, the length of the sentence. In this example, the sentence length is 15.  
    - mv: Mean Valency, the average valency for all words in a sentence.  
        - $mv = \frac{1}{sl}\sum_{i=1}^{sl}v_i$

1. Yan, J., & Liu, H. (2021, May). Quantitative analysis of Chinese and English verb valencies based on probabilistic valency pattern theory. In Workshop on Chinese Lexical Semantics (pp. 152-162). Cham: Springer International Publishing. 
        2. Cech, R., Pajas, P., & Macutek, J. (2010). Full valency. Verb valency without distinguishing complements and adjuncts. Journal of quantitative linguistics, 17(4), 291-302.
    - vk: Variance of Dynamic Valencies, the variance of dynamic generalized valencies for the sentence.  
        - $vk = \frac{1}{sl}\sum_{i=1}^{sl}(v_i-\bar{v})^2$
        - Lu, Q., Lin, Y., & Liu, H. (2018). Dynamic valency and dependency distance. Quantitative analysis of dependency structures, 145-166.
    - tw: Tree Width, the width of a tree.  
    - th: Tree Height, the height of a tree. 
        - tw & th: Zhang, H., & Liu, H. (2018). Interrelations among dependency tree widths, heights and sentence lengths. Quantitative Analysis of Dependency Structures, 72, 31-52.  
    - hi: Number of head-initial dependencies.  
    - hf: Number of head-final dependencies.  
    - rd: Root Distance, the linear position of the root in the sentence (1-based index).  
        - Lei, L., & Jockers, M. L. (2020). Normalized Dependency Distance: Proposing a New Measure. Journal of Quantitative Linguistics, 27(1), 62–79. 
    - ndd: Normalized Dependency Distance, defined as $|\log(\frac{mdd}{\sqrt{rd\cdot sl}})|$.  
        - Lei, L., & Jockers, M. L. (2020). Normalized Dependency Distance: Proposing a New Measure. Journal of Quantitative Linguistics, 27(1), 62–79. 
    - ddir & hi & hf: Liu, H. (2010). Dependency direction as a means of word-order typology: A method based on dependency treebanks. Lingua, 120(6), 1567-1578.

- **Text-level**: mdd, ndd, mhd, mhdd, mtdl, msl, mv, vk, mtw, mth, hi, hf, mrd  
    - mdd: Mean Dependency Distance, the average of all dependency distances across all sentences in the text.  
    - mhd: Mean Hierarchical Distance, the average of all hierarchy distances across all sentences in the text.  
    - mhdd: Mean Hierarchical Dependency Distance, the average of all mhdd values across all sentences in the text.  
    - mtdl: Mean Total Dependency Length, the average of all tdl values across all sentences in the text.  
    - msl: Mean Sentence Length, the average of all sentence lengths across the text.  
    - mv: Mean Valency, the average of all valency values across all sentences in the text.  
    - vk: Variance of Dynamic Valencies, the average of all VK values across all sentences in the text.  
    - mtw: Mean Tree Width, the average of all tree widths across all sentences in the text.  
    - mth: Mean Tree Height, the average of all tree heights across all sentences in the text.  
    - mrd: Mean Root Distance, the average root position across all sentences in the text.  
    - ndd: Mean Normalized Dependency Distance across all sentences in the text.  
    - hi: Proportion of head-initial dependencies in the text.  
    - hf: Proportion of head-final dependencies in the text.

- **Distributions**: dd, hd, v, sl, tw, th, rd, pos, deprel  
    - dd: Dependency Distance Distribution.  
    - hd: Hierarchical Distance Distribution.  
    - v: Valency Distribution.  
    - sl: Sentence Length Distribution.  
    - tw: Tree Width Distribution.  
    - th: Tree Height Distribution.  
    - rd: Root Distance Distribution.  
    - pos: Part-of-Speech Distribution.  
    - deprel: Dependency Relation Distribution.

- **Probabilistic Valency Patterns**: pos, deprel  
    - pvp: Probabilistic Valency Pattern, the probability distribution of dependency relations of a given word or word class.  
    - References:  
        1. Liu, H., & Feng, Z. (2007). Probabilistic Valency Pattern Theory for Natural Language Processing. *Linguistic Science, 6*(3), 32-41.  
        2. Yan, J., & Liu, H. (2021, May). Quantitative Analysis of Chinese and English Verb Valencies Based on Probabilistic Valency Pattern Theory. In *Workshop on Chinese Lexical Semantics* (pp. 152-162). Cham: Springer International Publishing.  
        3. Cech, R., Pajas, P., & Macutek, J. (2010). Full Valency: Verb Valency Without Distinguishing Complements and Adjuncts. *Journal of Quantitative Linguistics, 17*(4), 291-302.

*Note: All metric calculations ignore punctuation marks.*  
*Please strictly adhere to the abbreviations and coding specified above.*

2. **Treebank Formats**

Currently, only dependency treebanks are supported. The most common format for dependency treebanks is `conllu`. For further details on the `conllu` format, please refer to [conllu format](https://universaldependencies.org/format.html). However, other treebank formats are also commonly used, so it is necessary to support additional formats. The currently supported formats, in addition to `conllu`, include `conll`, `pmt`, and `mcdt`. If you want to use formats other than `conllu`, please first convert them to the `conllu` format using `depval.convert()` or `depval.Converter()`.

- **conll**: The `conllu` format with the `deps` and `misc` columns removed, with no other differences.

- **pmt**: This refers to the treebank format used by [PKUMultiViewTreebank](https://github.com/qiulikun/PKUMultiviewTreebank) format. In this format, each sentence spans four lines: the word form, part-of-speech tag, dependency relation type, and the linear index of the governing word.

|this|is|an|example|.|
    |  ----  | ----  |  ----  | ---- | ---- |
    |pron|verb|num|noun|punct|
    |subj|root|det|obj|punct|    
    |2|0|4|2|2|

- **mcdt**: This refers to the treebank format used by [MCDT / Modern Chinese Dependency Treebank](https://www.researchgate.net/publication/33017655_A_Chinese_Dependency_Syntax_for_Treebanking?_sg%5B0%5D=lsuUD85kTI5AWaecdmMgXv57mfrfiqz9742jcIKrQmKBRhix_-fCLMz-IcPHbBZkIZYfTMEn-5iMaycvEr3l-pSDv-y2MhjgWuel4Wd7.oTJp5BOdI_dKovhiuwNKWZHNf71LozC_j0Az1REHfauR6UtG91S0DEQbiCe6MsKWaeDkp0RDS5A1KWQdioUIqA&_tp=eyJjb250ZXh0Ijp7ImZpcnN0UGFnZSI6InB1YmxpY2F0aW9uIiwicGFnZSI6InByb2ZpbGUiLCJwcmV2aW91c1BhZ2UiOiJwcm9maWxlIiwicG9zaXRpb24iOiJwYWdlQ29udGVudCJ9fQ). 
    |sn|id|form|pos|head|gform|gpos|deprel|
    |  ----  |  ----  | ----  |  ----  | ---- | ---- |  ----  | ---- |
    |1|1|this|pron|2|is|verb|subj|
    |1|2|is|verb|0|.|punct|root|
    |1|3|an|num|4|example|noun|det|
    |1|4|example|noun|2|is|verb|obj|
    |1|6|.|punct|

---

## Installation

You can install the `QuanSyn` package via `pip`:

```bash
pip install quansyn
```

This package depends on [`conllu`](https://github.com/EmilStenstrom/conllu).

---

## Quick Start

Not everyone in the world knows how to program, but when the door is closed, a window must open. We provide some convenient APIs that allow you to save all analysis results locally with just a few lines of code, and you only need to change a few arguments.

### Calculating Syntactic Metrics
```python
from quansyn.depval import analyze

# The content in quotes needs to be changed: the *folder* path where the treebank files are stored and the *folder* path where the results should be saved.
# Note: These must be folders!!!
treebanks_path = r'path/to/treebank'
output_path = r'path/to/output'

# Perform the calculation (default: serial)
analyze(treebanks_path, output_path)

# Check the results saved locally, then you can use software like Excel, SPSS, jamovi for statistical analysis.
```

### Converting Treebank Formats
```python
from quansyn.depval import convert

# The content in quotes needs to be changed: the *folder* path where the source files are stored, and the *folder* path where the output files should be saved.
# Note: These must be folders!!!
input_path = r'path/to/treebank'
output_path = r'path/to/output'

# input_format and output_format refer to the source file format and output file format, change them as needed.
# The supported formats are: 'conllu', 'conll', 'pmt', 'mcdt'. Please refer to the previous introduction for details on these four formats.
input_format = 'conll'
output_format = 'conllu'

convert(input_path, output_path, input_format, output_format)
```

### Building a Linguistic Network
```python
from quansyn.lingnet import load_edges

# The content in quotes needs to be changed: the *folder* path where the source files are stored and the *folder* path where the output files should be saved.
# Note: These must be folders!!!
input_path = r'path/to/treebank'
output_path = r'path/to/output'
# Select 'dependency' or 'adjacency' mode, corresponding to the dependency relations or co-occurrence relations in the treebank.
mode = 'dependency'
# Load the treebank and generate the network
load_edges(
    input_path,
    output_path,
    mode=mode,
    lowercase=True,
    ignore_punct=True,
    weighted=False,
    directed=False
)
```

---

## Usage

Here, we will explain in detail the structure and usage of the `QuanSyn` package.

### Example 1: Calculating Dependency-Level / Word-Level Metrics

You can calculate dependency-level or word-level metrics:

```python
from quansyn.depval import DepValAnalyzer

# Load the dependency treebank
treebank_path = 'path/to/treebank.conllu'
analyzer = DepValAnalyzer(open(treebank_path, encoding='utf-8'))
# DepValAnalyzer also accepts a parsed sentence list or a raw conllu string.
# Use cleaned=True if your input is already preprocessed.

# Set the set of metrics (please follow the hierarchy and encoding strictly)
# If no metrics set is provided, all metrics are calculated by default.
metrics = ['dd', 'hd', 'ddir', 'v']
dep_metrics = analyzer.calculate_dep_metrics(metrics=metrics)

# Print the results
dep_metrics

>>> {'dd': [[1,2,3,...],[2,3,5,6,...]], 'hd': [[4,1,1,...],[2,2,5,4,...]], ...}
```
The return format is a dictionary `dict`, where the `key` is the metric name, and the `value` is a list, with each element corresponding to a sentence and containing **the respective metric value for the dependency or word**.

### Example 2: Calculating Sentence-Level Metrics

Similarly, you can calculate sentence-level metrics with a similar code structure:

```python
from quansyn.depval import DepValAnalyzer

treebank_path = 'path/to/treebank.conllu'
analyzer = DepValAnalyzer(open(treebank_path, encoding='utf-8'))

metrics = ['mdd', 'ndd', 'mhd', 'mhdd', 'tdl', 'sl', 'mv', 'vk', 'tw', 'th', 'hi', 'hf', 'rd']
sent_metrics = analyzer.calculate_sent_metrics(metrics=metrics)

sent_metrics    
>>> {'mdd': [1,2,3,2,3,5,6,...], 'mhd': [4,1,1,2,2,5,4,...], ...}
```
The return format is a dictionary `dict`, where the `key` is the metric name, and the `value` is a list, with each element corresponding to **the respective metric value for each sentence**.

### Example 3: Calculating Text-Level Metrics

You can also calculate text-level metrics in a similar manner:

```python
from quansyn.depval import DepValAnalyzer

treebank_path = 'path/to/treebank.conllu'
analyzer = DepValAnalyzer(open(treebank_path, encoding='utf-8'))

metrics = ['mdd', 'ndd', 'mhd', 'mhdd', 'mtdl', 'msl', 'mv', 'vk', 'mtw', 'mth', 'hi', 'hf', 'mrd']
text_metrics = analyzer.calculate_text_metrics(metrics=metrics)

text_metrics
>>> {'mdd': 2.32, 'mhd': 2.21, ...}
```
The return format is a dictionary `dict`, where the `key` is the metric name, and the `value` is **the respective metric value for the entire text**.

### Example 4: Calculating Metric Distributions

```python
from quansyn.depval import DepValAnalyzer

treebank_path = 'path/to/treebank.conllu'
analyzer = DepValAnalyzer(open(treebank_path, encoding='utf-8'))

metrics = ['dd', 'hd', 'sl', 'v', 'tw', 'th', 'rd', 'deprel', 'pos']

# normalize=True means to normalize the frequency, the default is False
distributions = analyzer.calculate_distributions(metrics=metrics, normalize=True)

# Print the results
distributions
>>> {'dd': ([1,2,3,4],[0.1,0.2,0.3,0.4]), 'hd', ...}
```
The return format is a dictionary `dict`, where the `key` is the distribution name, and the `value` is a tuple containing two elements: the first being `x` and the second being `y` (frequency or frequency ratio). Both `x` and `y` are lists.

### Example 5: Calculating Probabilistic Valency Patterns

```python
from quansyn.depval import DepValAnalyzer

# Load the dependency treebank      
treebank_path = 'path/to/treebank.conllu'
analyzer = DepValAnalyzer(open(treebank_path, encoding='utf-8'))

# Calculate the pvp (probabilistic valency patterns) for a specific word class, e.g., nouns.
# The treebank uses the 'sud' annotation system.
# 'input' is the word class; if not specified, pvp is calculated for all word classes.
# 'target' specifies whether to calculate dependency relation probabilities ('deprel') or part-of-speech probabilities ('pos'). Default is 'deprel'.
pvp = analyzer.calculate_pvp(input='NOUN', target='deprel', normalize=True)

pvp
>>> {'act as a gov': [('subj', 0.32), ('mod', 0.21), ...],
'act as a dep': [('mod', 0.28), ('comp:obj', 0.11), ...]}
```
The return format is a dictionary `dict`, where the `key` indicates governor/dependent role and the `value` is a list of `(label, probability)` pairs.

### Example 6: Converting Treebank Formats

```python
from quansyn.depval import Converter

# Load the treebank
treebank_path = 'path/to/treebank.conllu'
converter = Converter(open(treebank_path, encoding='utf-8'))

# Convert the format
# The first argument is the source format, and the second argument is the target format
converted_treebank = converter.style2style('conllu', 'pmt')

# Save the result
# converter.save(treebank, treebank_format, output_path)
converter.save(converted_treebank, 'pmt', 'output_path/to/treebank.txt')
```

### Example 7: Building a Linguistic Network

```python
from quansyn.lingnet import conllu2edge
import networkx as nx

# Load the treebank
treebank_path = 'path/to/treebank.conllu'
treebank = open(treebank_path, encoding='utf-8')

# Extract dependency relations as edges, or co-occurrence relations by setting mode to 'adjacency'
# edges is a list of tuples, where each tuple is (head, dependent)
edges = conllu2edge(
    treebank,
    mode='dependency',
    lowercase=True,
    ignore_punct=True,
    weighted=False,
    directed=False
)

# Build the network using NetworkX
G = nx.Graph(edges)
```

### Example 8: Fitting Parameters

```python
from quansyn.lawfitter import fit

# Load the data
data = ([1,2,3,4,5],[0.1,0.2,0.3,0.4,0.5])

# Fit built-in distributions
fit(data, 'zipf')

# Fit a custom distribution
def custom_dist(x, a):
    return a / (x ** 2)

fit(data, customized_law=custom_dist)

>>> {'params': [0.123456789], 'r^2': -16.229684950920042}
```

---

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Contact

- GitHub: [@YuhuYang](https://github.com/YuhuYang)
- Email: yangmufy@163.com

## Citing

If our project has been helpful to you, please give it a star and cite our articles. We would be very grateful.

```
@article{Yang03042025,
author = {Mu Yang and Haitao Liu},
title = {QuanSyn: A Package for Quantitative Syntax Analysis},
journal = {Journal of Quantitative Linguistics},
volume = {32},
number = {2},
pages = {181--198},
year = {2025},
publisher = {Routledge},
doi = {10.1080/09296174.2025.2471157},
URL = {https://doi.org/10.1080/09296174.2025.2471157},
eprint = { https://doi.org/10.1080/09296174.2025.2471157}}

```

QuanSyn

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README