LMPNN

KO | EN

gitlite — search

#python #hacktoberfest #ai #typescript #llm #android #deep-learning #linux #docker #kubernetes #awesome-list #security #pytorch #mcp

(★ 16)

Logical Message Passing Networks with One-hop Inference in Atomic Formulas (ICLR 2023)

LMPNN 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Logical Message Passing Networks with One-hop Inference on Atomic Formula

Implementation for ICLR 2023 paper:
> Logical Message Passing Networks with One-hop Inference on Atomic Formula

see the [arXiv version](https://arxiv.org/abs/2301.08859) and the [OpenReview version](https://openreview.net/forum?id=SoyOsp7i_l).

In this documentation, we detail how to reproduce the results in the paper based on existing checkpoints released by other researchers.

- [Logical Message Passing Networks with One-hop Inference on Atomic Formula](#logical-message-passing-networks-with-one-hop-inference-on-atomic-formula)
  - [Requirement of this repository](#requirement-of-this-repository)
  - [Version](#version)
  - [Preparation](#preparation)
    - [(A) Prepare the dataset](#a-prepare-the-dataset)
    - [(B1) Pretrain KGE checkpoints with external submodules](#b1-pretrain-kge-checkpoints-with-external-submodules)
      - [Choice 1: Pretrain with uma-pi1/kge](#choice-1-pretrain-with-uma-pi1kge)
        - [Step 0: Prepare the environment and config](#step-0-prepare-the-environment-and-config)
        - [Step 1: Prepare the dataset](#step-1-prepare-the-dataset)
        - [Step 2: Train the checkpoint](#step-2-train-the-checkpoint)
      - [Choice 2: Pretrain with facebookresearch/ssl-relation-prediction](#choice-2-pretrain-with-facebookresearchssl-relation-prediction)
        - [Step 0: Prepare the environment](#step-0-prepare-the-environment)
        - [Step 1: Prepare the dataset](#step-1-prepare-the-dataset-1)
        - [Step 2: Train the checkpoint](#step-2-train-the-checkpoint-1)
    - [(B2) Convert pretrained KGE checkpoints into the acceptable format](#b2-convert-pretrained-kge-checkpoints-into-the-acceptable-format)
      - [Sources 1. uclnlp/cqd](#sources-1-uclnlpcqd)
  - [Train LMPNN](#train-lmpnn)
  - [Answering Existential First Order (EFO) queries](#answering-existential-first-order-efo-queries)
  - [Summarize the results from log files](#summarize-the-results-from-log-files)
    - [Sample training trajectories of LMPNN with command provided above.](#sample-training-trajectories-of-lmpnn-with-command-provided-above)
    - [(depreciated) LMPNN trajectory without freezed KGE (weak implementation)](#depreciated-lmpnn-trajectory-without-freezed-kge-weak-implementation)
  - [Citing this paper](#citing-this-paper)

## Requirement of this repository
- pytorch
- jupyter
- tqdm

Requirement of other submodules will be discussed accordingly.

My conda environment could be found at `ENV1.yaml`

## Version

1. Update 2023/07/20. We fix the non-freezed KG embedding issue before the commit id e83a95a. New version fixes the KG embedding and is now fully consistent as the paper.

** This repo is under construction for usability. The key results for the paper can already be reproduced. **

Todo features:
- [x] Implement CQD CO Extended, CQD(E) in the paper
- [ ] Introduce several ways to train KGE checkpoints with released repositories, which can be backbones for LMPNN.

## Preparation

### (A) Prepare the dataset
Please download the dataset from [snap-stanford/KGReasoning](https://github.com/snap-stanford/KGReasoning).

Specifically, one can run:
```bash
mkdir data
cd data
wget http://snap.stanford.edu/betae/KG_data.zip # a zip file of 1.3G
unzip KG_data.zip
```

Then the `data` folder will contain the following folders and files:
```
FB15k-237-betae
FB15k-237-q2b
FB15k-betae
FB15k-q2b
KG_data.zip
NELL-betae
NELL-q2b
```

We rearange them into different subfolders:
```
mkdir betae-dataset
mv *betae betae-dataset
mkdir q2b-dataset
mv *q2b q2b-dataset
```

Finally, run `convert_beta_dataset.py` to convert the data into the graph forms for LMPNN. One can find the new dataset folders in `./data`.

An example converted dataset format is
```
data/FB15k-237-betae
  - kgindex.json
  - train_kg.tsv
  - valid_kg.tsv
  - test_kg.tsv
  - train-qaa.json
  - valid-qaa.json
  - test-qaa.json
```
where
- `kgindex.json` file stores the relation/entity names and their ids,
- `{train/valid/test}_kg.tsv` stores the triples in three knowledge graphs (triples in `train_kg.tsv` is the subset of those in `valid_kg.tsv`, and triples `valid_kg.tsv` is also the subset of those in `test_kg.tsv`)
- `{train/valid/test}-qaa.json` stores the Query, easy Answers and hard Answers for train, valid, and test set.

### (B1) Pretrain KGE checkpoints with external submodules

We consider two different repositories to pretrain the KGE checkpoints.
Including
1. [uma-pi1/kge](https://github.com/uma-pi1/kge)
2. [facebookresearch/ssl-relation-prediction](https://github.com/facebookresearch/ssl-relation-prediction)

To initialize these modules, please run
```bash
git submodule update
```

How to train the checkpoints with these submodules is discussed in this section.

Generally, there are two steps, once the KG is prepared:
1. Convert the KG triples into the format that can be used in each submodule.
2. Train the checkpoints.

#### Choice 1: Pretrain with [uma-pi1/kge](https://github.com/uma-pi1/kge)

##### Step 0: Prepare the environment and config

To run `libkge` submodule, one need editable installation.

```bash
cd kge
pip install -e .
```

##### Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into `./kge/data`.

```sh
python convert_kg_data_for_kge.py
```

##### Step 2: Train the checkpoint

We provide a config in `config/kge/fb15k-237-complex.yaml`. Tailor the config to train checkpoints with `libkge`.
```sh
kge start config/kge/fb15k-237-complex.yaml --job.device cuda:0
```

The obtained checkpoints can be found at `kge/local`.

#### Choice 2: Pretrain with [facebookresearch/ssl-relation-prediction](https://github.com/facebookresearch/ssl-relation-prediction)

##### Step 0: Prepare the environment
```
pip install ogb networkx wandb
```

##### Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into `./ssl-relation-prediction/data`.

```sh
python convert_kg_data_for_ssl.py
```

Notably, in the `ssl-relation-prediction/src/main.py` file, the command arg parser prohibits external sources of datasets in line 47-50.
```python
parser.add_argument(
    '--dataset', choices=datasets,
    help="Dataset in {}".format(datasets)
)
```

Let's change it into
```python
parser.add_argument(
    '--dataset'
)
```

##### Step 2: Train the checkpoint
The training process can be initialized by running
```sh
cd ssl-relation-prediction
python src/main.py --dataset FB15k-237-betae \
               --score_rel True \
               --model ComplEx \
               --rank 1000 \
               --learning_rate 0.1 \
               --batch_size 1000 \
               --lmbda 0.05 \
               --w_rel 4 \
               --max_epochs 100 \
               --model_cache_path ./ckpts/FB15k-237-complex/
```

The obtained checkpoints can be found at `ssl-relation-prediction/ckpts/FB15k-237-complex`.

### (B2) Convert pretrained KGE checkpoints into the acceptable format

We convert external KGE checkpoints into the format that can be loaded by LMPNN. We consider three sources of external checkpoints

1. Checkpoints released by [uclnlp/cqd](https://github.com/uclnlp/cqd).
2. Checkpoints released / pretrained by [uma-pi1/kge](https://github.com/uma-pi1/kge)
2. Checkpoints released by [facebookresearch/ssl-relation-prediction](https://github.com/facebookresearch/ssl-relation-prediction)

The pretrained checkpoints are managed in the folder `pretrain`.
```sh
mkdir pretrain
```

#### Sources 1. [uclnlp/cqd](https://github.com/uclnlp/cqd)

This source of checkpoints is used to repreduced the results shown in the paper.

```bash
cd pretrain
wget http://data.neuralnoise.com/cqd-models.tgz # a .tgz file of 4.8G
tar xvf cqd-models.tgz
mv models raw_cqd_pretrain_models
```

Then we can convert the checkpoints into the format used in this repo.
```bash
python convert_cqd_pretrain_ckpts.py
```

## Train LMPNN

Sample usage at FB15k-237
```bash
python3 train_lmpnn.py \
  --task_folder data/FB15k-237-betae \
  --output_dir log/FB15k-237/lmpnn-complex1k-default \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt \
  --embedding_dim 1000 \
  --device cuda:0
```

Sample usage at FB15k
```bash
python3 train_lmpnn.py \
  --task_folder data/FB15k-betae \
  --checkpoint_path pretrain/cqd/FB15k-model-rank-1000-epoch-100-1602520745.pt \
  --device cuda:1 \
  --hidden_dim 8192 \
  --output_dir log/FB15k/lmpnn-complex1k-default
```

Sample usage at NELL
```bash
python3 train_lmpnn.py \
  --task_folder data/NELL-betae \
  --checkpoint_path pretrain/cqd/NELL-model-rank-1000-epoch-100-1602499096.pt \
  --device cuda:2 \
  --hidden_dim 8192 \
  --temp 0.1 \
  --batch_size 512 \
  --batch_size_eval_dataloader 8 \
  --batch_size_eval_truth_value 1 \
  --output_dir log/NELL/lmpnn-complex1k-default
```

## Answering Existential First Order (EFO) queries

In this repository, the capability of answering EFO-1 queries is implemented by the `reasoner`s.

- CQD-CO is implemented as `GradientEFOReasoner`, which is refered as CQD(E) in the paper.
- LMPNN is implemented as `GNNEFOReasoner` with `LogicalGNNLayer`

```bash
python train_lmpnn.py \
  --reasoner gradient \
  --device cuda:1 \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt
```

## Summarize the results from log files

We use script `read_eval_from_log.py` to summarize the results from the log file.

For example, the results on FB15k-237 in file `log/FB15k-237/pretrain_complex1000-default/output.log` can be summarized by the following piece of code.
```bash
python3 read_eval_from_log.py --log_file log/FB15k-237/pretrain_complex1000-default/output.log
```

Then the code will output the markdown table of the trajectory over valid and test set.

### Sample training trajectories of LMPNN with command provided above.

For FB15k-237, a possible output trajectory could be like

| Validation Set  |      1p |       2p |       3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |      3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|---------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 43.1522 |  8.96845 | 7.30686 | 28.5481 | 42.3203 | 12.6179 | 10.7356 | 10.8988 | 7.9376  | 5.15343 |  9.14102 | 6.64968 | 3.48541 | 3.33012 |     19.1651 |    5.55193 |
| (10, 'mrr')  | 43.1469 |  9.72417 | 7.70923 | 29.5263 | 44.0089 | 14.4818 | 12.1041 | 11.0914 | 8.02962 | 5.83145 |  9.66503 | 7.01406 | 3.67668 | 3.64304 |     19.9803 |    5.96605 |
| (15, 'mrr')  | 43.8283 |  9.78142 | 7.81307 | 29.8871 | 44.3047 | 15.0133 | 12.4008 | 11.139  | 8.22536 | 5.83979 |  9.86778 | 7.0962  | 3.70381 | 3.79685 |     20.2659 |    6.06089 |
| (20, 'mrr')  | 43.8607 | 10.5252  | 8.27492 | 30.7076 | 45.3577 | 16.3521 | 13.0123 | 11.2875 | 8.52308 | 6.13657 | 10.185   | 7.29428 | 3.74871 | 3.91729 |     20.8779 |    6.25637 |
| (25, 'mrr')  | 44.0598 | 10.411   | 8.37461 | 30.2869 | 45.1289 | 16.2044 | 12.8736 | 11.2198 | 8.30653 | 6.02084 |  9.82021 | 7.21201 | 3.9356  | 4.02344 |     20.7628 |    6.20242 |
| (30, 'mrr')  | 43.8415 | 10.4777  | 8.47792 | 30.7736 | 45.5773 | 16.8041 | 13.0142 | 11.0647 | 8.35672 | 6.05398 | 10.335   | 7.39058 | 3.73573 | 3.95478 |     20.932  |    6.29401 |
| (35, 'mrr')  | 44.0388 | 10.4904  | 8.66033 | 30.8991 | 45.9508 | 17.6397 | 13.2279 | 11.2514 | 8.44008 | 6.16027 | 10.3354  | 7.46953 | 3.80763 | 3.92426 |     21.1776 |    6.33942 |
| (40, 'mrr')  | 43.7431 | 10.7154  | 8.9294  | 30.9381 | 46.091  | 17.4797 | 13.3665 | 11.1077 | 8.54201 | 6.10274 | 10.445   | 7.4784  | 3.8803  | 4.01805 |     21.2125 |    6.38489 |
| (45, 'mrr')  | 44.0037 | 10.8814  | 8.71511 | 31.0536 | 46.4162 | 17.7432 | 13.4061 | 11.2816 | 8.40346 | 6.2916  | 10.5165  | 7.46814 | 3.88711 | 3.94484 |     21.3227 |    6.42164 |
| (50, 'mrr')  | 44.1079 | 10.8688  | 8.62299 | 31.3121 | 46.5769 | 17.9518 | 13.4356 | 11.3143 | 8.59079 | 6.23409 | 10.398   | 7.76916 | 3.77844 | 3.99303 |     21.4201 |    6.43454 |
| (55, 'mrr')  | 44.2142 | 11.2075  | 9.15606 | 31.7812 | 47.2926 | 19.3201 | 13.9342 | 11.3463 | 8.5981  | 6.29677 | 10.6873  | 7.93258 | 4.02479 | 4.05711 |     21.8722 |    6.59971 |
| (60, 'mrr')  | 44.1963 | 11.2091  | 9.26216 | 31.7813 | 47.5042 | 19.5144 | 13.9952 | 11.271  | 8.60634 | 6.35608 | 10.7975  | 7.85765 | 4.06059 | 4.00284 |     21.9267 |    6.61492 |
| (65, 'mrr')  | 44.1878 | 11.2833  | 9.23951 | 31.784  | 47.5272 | 19.8171 | 14.0538 | 11.2238 | 8.56164 | 6.31577 | 10.7261  | 7.89386 | 4.12592 | 3.97752 |     21.9642 |    6.60783 |
| (70, 'mrr')  | 44.1557 | 11.3497  | 9.26543 | 31.8273 | 47.6473 | 19.8378 | 14.0356 | 11.1839 | 8.57884 | 6.28838 | 10.7356  | 7.88885 | 4.12011 | 4.00915 |     21.9868 |    6.60841 |
| (75, 'mrr')  | 44.1301 | 11.3565  | 9.37637 | 31.8551 | 47.6066 | 19.965  | 14.0906 | 11.1939 | 8.55513 | 6.30744 | 10.7699  | 7.76835 | 4.11392 | 3.93539 |     22.0144 |    6.57901 |
| (80, 'mrr')  | 44.1289 | 11.3645  | 9.34517 | 31.865  | 47.8433 | 20.0669 | 14.1551 | 11.2427 | 8.65353 | 6.30529 | 10.7471  | 7.92971 | 4.12401 | 3.95507 |     22.0739 |    6.61224 |
| (85, 'mrr')  | 44.1127 | 11.4596  | 9.25204 | 31.7441 | 47.8254 | 19.9052 | 14.126  | 11.1764 | 8.62984 | 6.32005 | 10.7557  | 7.89449 | 4.16561 | 3.90074 |     22.0257 |    6.60732 |
| (90, 'mrr')  | 44.1258 | 11.3737  | 9.32294 | 31.9144 | 47.8743 | 20.0821 | 14.1917 | 11.2108 | 8.62529 | 6.34314 | 10.7944  | 7.92542 | 4.1587  | 3.94927 |     22.0801 |    6.63419 |
| (95, 'mrr')  | 44.1069 | 11.4469  | 9.35151 | 32.0017 | 47.8526 | 20.354  | 14.2962 | 11.1435 | 8.56965 | 6.39189 | 10.7905  | 7.83577 | 4.21076 | 3.96381 |     22.1248 |    6.63854 |
| (100, 'mrr') | 44.1638 | 11.41    | 9.49996 | 31.8933 | 47.9468 | 20.2255 | 14.3075 | 11.1681 | 8.64452 | 6.34371 | 10.73    | 7.96422 | 4.20087 | 3.91509 |     22.1399 |    6.63077 |

| Test set |      1p |      2p |       3p |      2i |      3i |      pi |      ip |      2u |       up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|---------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 44.8991 | 10.3212 |  8.21212 | 31.3772 | 43.8674 | 14.1751 | 13.3347 | 13.6166 |  9.15165 | 7.003   | 10.8464 | 6.3307  | 3.73892 | 4.149   |     20.995  |    6.41361 |
| (10, 'mrr')  | 44.7485 | 11.0422 |  9.01639 | 32.5366 | 45.5268 | 16.815  | 14.8415 | 13.68   |  9.72893 | 7.78859 | 11.6098 | 6.77476 | 4.03303 | 4.59245 |     21.9929 |    6.95973 |
| (15, 'mrr')  | 45.4805 | 11.4756 |  9.00117 | 32.716  | 46.0811 | 17.2183 | 15.2297 | 13.7089 |  9.82214 | 8.14234 | 11.7426 | 6.82549 | 3.95403 | 4.8102  |     22.3037 |    7.09494 |
| (20, 'mrr')  | 45.5168 | 11.8394 |  9.10475 | 33.2281 | 46.7741 | 18.0997 | 15.6644 | 13.8107 | 10.1112  | 8.58388 | 12.3831 | 6.96503 | 4.17977 | 4.99503 |     22.6832 |    7.42136 |
| (25, 'mrr')  | 45.6835 | 11.8071 |  9.46537 | 33.4615 | 46.7437 | 18.3839 | 15.7297 | 13.6642 | 10.0942  | 8.47826 | 12.0139 | 7.21188 | 4.31563 | 4.96695 |     22.7815 |    7.39732 |
| (30, 'mrr')  | 45.5894 | 11.8349 |  9.42184 | 33.2727 | 47.1798 | 19.1685 | 16.0285 | 13.5988 | 10.0361  | 8.26477 | 12.2796 | 7.02654 | 4.05282 | 5.24071 |     22.9034 |    7.37289 |
| (35, 'mrr')  | 45.6687 | 12.2878 |  9.57961 | 33.8039 | 47.5376 | 19.7294 | 16.4946 | 13.7893 |  9.99798 | 8.55637 | 12.4467 | 7.06137 | 4.13864 | 5.08787 |     23.2099 |    7.45819 |
| (40, 'mrr')  | 45.401  | 12.1917 |  9.29751 | 33.5219 | 47.5335 | 19.7207 | 16.4359 | 13.6639 | 10.03    | 8.39218 | 12.417  | 7.24734 | 4.20833 | 5.00166 |     23.0885 |    7.45331 |
| (45, 'mrr')  | 45.8186 | 12.365  |  9.92397 | 34.0337 | 47.9003 | 20.0247 | 16.6008 | 13.6968 |  9.96648 | 8.62486 | 12.8463 | 7.27353 | 4.38398 | 5.05679 |     23.37   |    7.63709 |
| (50, 'mrr')  | 45.633  | 12.3816 | 10.1212  | 33.9538 | 48.0138 | 20.1938 | 16.5077 | 13.6692 | 10.2213  | 8.70553 | 12.7164 | 7.41705 | 4.48108 | 4.98651 |     23.4106 |    7.6613  |
| (55, 'mrr')  | 45.8942 | 12.7871 | 10.2142  | 34.3762 | 48.621  | 21.5511 | 17.0483 | 13.6615 | 10.2328  | 8.79375 | 12.9544 | 7.61405 | 4.55091 | 5.12093 |     23.8207 |    7.80681 |
| (60, 'mrr')  | 45.8769 | 13.0076 | 10.3082  | 34.548  | 48.4693 | 21.8219 | 17.2902 | 13.6215 | 10.2962  | 8.82371 | 13.0351 | 7.62129 | 4.6169  | 5.07511 |     23.9156 |    7.83441 |
| (65, 'mrr')  | 45.8871 | 12.8002 | 10.2563  | 34.5614 | 48.7442 | 22.0247 | 17.1587 | 13.7247 | 10.1597  | 8.81356 | 13.0102 | 7.63786 | 4.55925 | 5.04153 |     23.9241 |    7.81248 |
| (70, 'mrr')  | 45.915  | 12.9762 | 10.3503  | 34.6932 | 48.8386 | 21.9847 | 17.2467 | 13.6548 | 10.1373  | 8.75037 | 13.1279 | 7.60491 | 4.55571 | 5.04706 |     23.9774 |    7.8172  |
| (75, 'mrr')  | 45.9074 | 12.9137 | 10.2342  | 34.7737 | 48.7164 | 22.0299 | 17.3165 | 13.663  | 10.1987  | 8.74679 | 13.062  | 7.66346 | 4.63074 | 5.10834 |     23.9726 |    7.84227 |
| (80, 'mrr')  | 45.9094 | 12.9309 | 10.376   | 34.7534 | 48.8564 | 22.3067 | 17.2492 | 13.6232 | 10.1518  | 8.752   | 13.0546 | 7.64423 | 4.68432 | 5.12836 |     24.0174 |    7.85271 |
| (85, 'mrr')  | 45.8954 | 12.9381 | 10.273   | 34.7352 | 48.873  | 22.3331 | 17.2259 | 13.5965 | 10.1396  | 8.73849 | 13.0252 | 7.67358 | 4.59914 | 5.08056 |     24.0011 |    7.8234  |
| (90, 'mrr')  | 45.84   | 12.8973 | 10.3091  | 34.8627 | 49.0908 | 22.4607 | 17.3106 | 13.5644 | 10.1515  | 8.77689 | 13.1038 | 7.59491 | 4.66412 | 5.04967 |     24.0541 |    7.83788 |
| (95, 'mrr')  | 45.8448 | 12.8836 | 10.3528  | 34.8965 | 49.0815 | 22.5072 | 17.3211 | 13.4828 | 10.119   | 8.78041 | 13.0549 | 7.62073 | 4.70466 | 5.098   |     24.0544 |    7.85175 |
| (100, 'mrr') | 45.8614 | 13.0062 | 10.4779  | 34.9732 | 49.0683 | 22.5373 | 17.3939 | 13.578  | 10.2351  | 8.69003 | 13.0802 | 7.68441 | 4.70692 | 5.01153 |     24.1257 |    7.83462 |

For FB15k dataset, a possible training trajectory could be

|Validation set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |      pin |      pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|---------:|------------:|-----------:|
| (5, 'mrr')   | 72.764  | 18.1599 | 14.3685 | 50.1483 | 63.4033 | 20.3927 | 21.7411 | 24.0656 | 17.3729 | 16.0573 | 14.3387 |  8.96579 | 5.63852 |  8.01369 |     33.6018 |    10.6028 |
| (10, 'mrr')  | 73.2114 | 20.7276 | 16.2229 | 50.9103 | 64.7822 | 24.5783 | 24.0674 | 24.2767 | 18.4504 | 16.0651 | 14.6001 |  9.98227 | 6.18293 |  9.6944  |     35.2475 |    11.305  |
| (15, 'mrr')  | 74.041  | 21.7555 | 16.8663 | 51.4346 | 65.7271 | 26.7591 | 25.3408 | 24.7065 | 18.9669 | 16.5432 | 14.7984 | 10.4507  | 6.67602 | 10.1124  |     36.1775 |    11.7161 |
| (20, 'mrr')  | 74.4192 | 22.4445 | 17.3096 | 51.8621 | 66.3508 | 28.1335 | 26.2893 | 25.1039 | 19.3605 | 17.0578 | 15.3382 | 10.7319  | 6.7229  | 10.4774  |     36.8081 |    12.0657 |
| (25, 'mrr')  | 74.6472 | 23.0718 | 17.8927 | 52.3799 | 66.7333 | 29.4432 | 27.0306 | 25.102  | 19.7324 | 17.0759 | 15.3593 | 11.0009  | 7.12063 | 10.7602  |     37.337  |    12.2634 |
| (30, 'mrr')  | 74.6737 | 23.4607 | 17.8685 | 52.2627 | 66.8866 | 30.3678 | 27.282  | 25.3029 | 19.5946 | 17.0104 | 15.299  | 11.007   | 7.04523 | 10.8055  |     37.5222 |    12.2334 |
| (35, 'mrr')  | 74.9351 | 23.8835 | 18.3777 | 52.3996 | 67.0009 | 30.8495 | 27.6875 | 25.4789 | 20.0324 | 16.8463 | 15.297  | 11.1668  | 7.17088 | 10.9909  |     37.8495 |    12.2944 |
| (40, 'mrr')  | 75.1099 | 24.4404 | 18.5153 | 52.6634 | 67.2669 | 31.201  | 28.1146 | 25.4405 | 20.0116 | 17.1899 | 15.7448 | 11.5347  | 7.43448 | 11.1738  |     38.0848 |    12.6155 |
| (45, 'mrr')  | 75.175  | 24.4533 | 18.7198 | 52.7764 | 67.4387 | 32.2673 | 28.5103 | 25.4906 | 20.0293 | 17.2964 | 15.8747 | 11.5726  | 7.28784 | 11.2956  |     38.3179 |    12.6654 |
| (50, 'mrr')  | 75.2242 | 24.5531 | 18.645  | 53.0605 | 67.4554 | 32.5306 | 28.4636 | 25.503  | 20.0741 | 17.2653 | 15.9408 | 11.5878  | 7.41323 | 11.3201  |     38.3899 |    12.7055 |
| (55, 'mrr')  | 75.7348 | 25.3315 | 19.4048 | 53.5306 | 68.1116 | 34.0854 | 29.3051 | 25.6027 | 20.3288 | 17.5561 | 16.1879 | 11.9783  | 7.51217 | 11.4066  |     39.0484 |    12.9282 |
| (60, 'mrr')  | 75.861  | 25.5841 | 19.5346 | 53.7438 | 68.2576 | 34.5876 | 29.5732 | 25.6164 | 20.4787 | 17.6198 | 16.3073 | 12.0556  | 7.65011 | 11.4113  |     39.2486 |    13.0088 |
| (65, 'mrr')  | 75.9349 | 25.7382 | 19.6538 | 53.6697 | 68.3454 | 34.8048 | 29.6301 | 25.6537 | 20.4627 | 17.6294 | 16.3302 | 12.139   | 7.6883  | 11.4152  |     39.3215 |    13.0404 |
| (70, 'mrr')  | 75.9818 | 25.7803 | 19.7334 | 53.8455 | 68.6032 | 35.0239 | 29.7125 | 25.6243 | 20.5497 | 17.6473 | 16.4369 | 12.1637  | 7.76578 | 11.4374  |     39.4283 |    13.0902 |
| (75, 'mrr')  | 76.0647 | 25.8406 | 19.8401 | 53.9193 | 68.5568 | 35.2081 | 29.7754 | 25.6366 | 20.5592 | 17.6685 | 16.3975 | 12.1659  | 7.8314  | 11.3745  |     39.489  |    13.0875 |
| (80, 'mrr')  | 76.0764 | 25.815  | 19.8477 | 53.8205 | 68.5533 | 35.353  | 29.7676 | 25.6507 | 20.5044 | 17.6077 | 16.4779 | 12.226   | 7.8704  | 11.3885  |     39.4876 |    13.1141 |
| (85, 'mrr')  | 76.1621 | 25.8629 | 19.9085 | 53.8178 | 68.594  | 35.352  | 29.8925 | 25.6998 | 20.6121 | 17.5867 | 16.5415 | 12.2125  | 7.92277 | 11.3823  |     39.5446 |    13.1292 |
| (90, 'mrr')  | 76.1505 | 25.8591 | 19.941  | 53.8449 | 68.7232 | 35.3287 | 30.0221 | 25.7584 | 20.5822 | 17.6385 | 16.4569 | 12.2411  | 7.92856 | 11.4002  |     39.5789 |    13.1331 |
| (95, 'mrr')  | 76.1381 | 25.9548 | 19.9634 | 53.8797 | 68.7455 | 35.6256 | 29.9943 | 25.7073 | 20.6724 | 17.622  | 16.6394 | 12.2738  | 7.97295 | 11.4501  |     39.6312 |    13.1916 |
| (100, 'mrr') | 76.2425 | 25.9843 | 20.0225 | 53.9112 | 68.7771 | 35.6986 | 30.0373 | 25.8397 | 20.8087 | 17.6471 | 16.7057 | 12.2231  | 7.98628 | 11.414   |     39.7024 |    13.1952 |

|Test set|      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |      pin |     pni |   epfo mean |   Neg mean |
|              |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |      pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 80.0214 | 26.1631 | 20.6792 | 63.1397 | 70.4824 | 26.7026 | 31.8146 | 35.0759 | 25.2013 | 26.7359 | 26.8873 | 10.0949 |  6.86781 | 11.2488 |     42.1422 |    16.3669 |
| (10, 'mrr')  | 80.6118 | 29.9317 | 23.0558 | 64.1709 | 71.9207 | 32.2211 | 35.1749 | 34.2256 | 27.085  | 26.0799 | 27.0576 | 11.7576 |  7.76882 | 13.5857 |     44.2664 |    17.2499 |
| (15, 'mrr')  | 81.6128 | 32.087  | 24.2246 | 64.7785 | 72.8758 | 35.1365 | 36.9434 | 35.6952 | 28.4065 | 27.7164 | 27.5466 | 12.3355 |  8.14011 | 14.713  |     45.7512 |    18.0903 |
| (20, 'mrr')  | 82.1223 | 32.9022 | 24.7883 | 65.2956 | 73.2986 | 36.3822 | 37.6886 | 36.0638 | 28.7675 | 29.0385 | 27.9598 | 12.9415 |  8.56375 | 15.3342 |     46.3677 |    18.7676 |
| (25, 'mrr')  | 82.4252 | 34.2613 | 25.7724 | 65.5153 | 73.8097 | 38.1548 | 38.6816 | 36.4394 | 29.2493 | 28.9039 | 28.0133 | 13.1606 |  8.90367 | 15.7501 |     47.1454 |    18.9463 |
| (30, 'mrr')  | 82.6365 | 34.6101 | 25.5823 | 65.5032 | 74.0651 | 38.9272 | 39.0842 | 36.2058 | 29.4809 | 28.8655 | 28.0349 | 13.4159 |  8.92127 | 15.4652 |     47.3439 |    18.9406 |
| (35, 'mrr')  | 82.8815 | 35.4994 | 26.4092 | 65.9638 | 74.3764 | 40.0052 | 39.5615 | 36.8258 | 29.8588 | 28.8328 | 27.9218 | 13.6509 |  9.0599  | 15.8999 |     47.9313 |    19.0731 |
| (40, 'mrr')  | 83.0528 | 35.6612 | 26.4634 | 66.229  | 74.5047 | 40.617  | 39.6554 | 36.4271 | 29.9298 | 28.8021 | 28.2275 | 13.9195 |  9.4794  | 16.4576 |     48.06   |    19.3772 |
| (45, 'mrr')  | 83.136  | 36.7407 | 26.5522 | 66.3764 | 74.7943 | 42.0012 | 41.0161 | 36.5704 | 30.1945 | 29.4151 | 28.6116 | 13.789  |  9.48379 | 16.541  |     48.598  |    19.5681 |
| (50, 'mrr')  | 83.3121 | 36.643  | 26.7993 | 66.705  | 74.9823 | 42.057  | 40.0862 | 36.6947 | 30.2306 | 28.8638 | 28.5118 | 13.9804 |  9.23777 | 16.38   |     48.6122 |    19.3948 |
| (55, 'mrr')  | 84.0474 | 38.0631 | 28.1092 | 67.6499 | 76.0039 | 44.2674 | 41.6678 | 36.5543 | 30.9187 | 29.0836 | 29.0125 | 14.499  |  9.72798 | 16.6574 |     49.698  |    19.7961 |
| (60, 'mrr')  | 84.2658 | 38.3675 | 28.4186 | 67.8149 | 76.2868 | 44.6406 | 41.99   | 36.5638 | 31.2279 | 29.1775 | 29.1025 | 14.6062 |  9.85346 | 16.714  |     49.9529 |    19.8907 |
| (65, 'mrr')  | 84.3822 | 38.5743 | 28.5948 | 67.9548 | 76.3456 | 44.9868 | 42.1976 | 36.7303 | 31.1506 | 29.1546 | 29.2271 | 14.7821 |  9.87208 | 16.7348 |     50.1019 |    19.9541 |
| (70, 'mrr')  | 84.4594 | 38.726  | 28.5233 | 67.9601 | 76.3959 | 45.4252 | 42.4645 | 36.632  | 31.2578 | 29.2664 | 29.2771 | 14.7912 |  9.95378 | 16.6008 |     50.2049 |    19.9779 |
| (75, 'mrr')  | 84.5264 | 38.7699 | 28.6037 | 68.0511 | 76.5039 | 45.4822 | 42.545  | 36.7839 | 31.2311 | 29.1764 | 29.2714 | 14.7585 |  9.94432 | 16.6274 |     50.2775 |    19.9556 |
| (80, 'mrr')  | 84.6096 | 38.6981 | 28.5022 | 68.0766 | 76.4818 | 45.5297 | 42.3989 | 36.7637 | 31.2135 | 29.102  | 29.3295 | 14.8571 |  9.99067 | 16.6122 |     50.2527 |    19.9783 |
| (85, 'mrr')  | 84.6776 | 38.6621 | 28.4035 | 68.0234 | 76.4476 | 45.7052 | 42.4011 | 36.6235 | 31.1659 | 29.134  | 29.289  | 14.8435 |  9.99    | 16.5764 |     50.2344 |    19.9666 |
| (90, 'mrr')  | 84.7225 | 38.9814 | 28.4973 | 68.0495 | 76.5384 | 45.9682 | 42.7049 | 36.6288 | 31.2611 | 29.2248 | 29.3559 | 14.8518 |  9.99594 | 16.5391 |     50.3725 |    19.9935 |
| (95, 'mrr')  | 84.7482 | 39.0329 | 28.4649 | 68.1035 | 76.3899 | 46.021  | 42.7329 | 36.7371 | 31.1983 | 29.1817 | 29.2837 | 14.8702 | 10.0342  | 16.5788 |     50.381  |    19.9897 |
| (100, 'mrr') | 84.8463 | 39.3027 | 28.729  | 68.1155 | 76.6316 | 46.1163 | 42.9275 | 36.8473 | 31.4271 | 29.1328 | 29.2369 | 14.852  | 10.0518  | 16.6147 |     50.5493 |    19.9776 |

For NELL dataset, a possible training trajectory could be

| Validation set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 51.9866 | 16.4089 | 13.3687 | 31.5865 | 43.7179 | 18.1878 | 19.2869 | 14.4951 | 11.9546 | 3.18955 |  9.70108 | 10.7595 | 3.79393 | 4.22667 |     24.5548 |    6.33414 |
| (10, 'mrr')  | 58.0504 | 18.9486 | 16.2135 | 35.7542 | 49.4412 | 22.1829 | 22.2309 | 15.9189 | 13.0717 | 8.04612 | 10.5261  | 12.1496 | 4.10794 | 4.4874  |     27.9791 |    7.86342 |
| (15, 'mrr')  | 58.1901 | 19.5215 | 16.9156 | 36.2875 | 50.5697 | 23.5127 | 23.0488 | 15.886  | 13.1331 | 8.38497 | 10.8321  | 12.4829 | 3.99677 | 4.58691 |     28.5628 |    8.05675 |
| (20, 'mrr')  | 58.1312 | 19.7626 | 17.1061 | 36.6888 | 51.3984 | 24.2156 | 23.8144 | 15.744  | 13.2207 | 8.42841 | 10.7424  | 12.3675 | 4.01913 | 4.49603 |     28.898  |    8.0107  |
| (25, 'mrr')  | 58.2135 | 19.3948 | 16.7125 | 37.0836 | 51.5466 | 23.8573 | 23.4707 | 15.7727 | 13.1974 | 8.13385 | 10.7643  | 12.3361 | 4.0185  | 4.51245 |     28.8055 |    7.95303 |
| (30, 'mrr')  | 58.3422 | 19.9796 | 16.9005 | 37.3361 | 52.2049 | 25.0439 | 24.258  | 15.8504 | 13.529  | 8.25035 | 11.0144  | 12.8912 | 3.94366 | 4.48501 |     29.2716 |    8.11691 |
| (35, 'mrr')  | 58.3317 | 19.8246 | 17.0437 | 37.0317 | 52.0876 | 24.9918 | 24.1882 | 15.7776 | 13.2863 | 8.11579 | 11.0137  | 12.5773 | 3.95141 | 4.43461 |     29.1737 |    8.01855 |
| (40, 'mrr')  | 58.4374 | 20.0042 | 17.2376 | 37.5069 | 52.4436 | 25.5753 | 24.3363 | 15.8403 | 13.2857 | 8.04121 | 11.0236  | 12.65   | 4.00914 | 4.48272 |     29.4075 |    8.04135 |
| (45, 'mrr')  | 58.3851 | 20.0087 | 17.1159 | 37.2356 | 52.586  | 25.8343 | 24.3349 | 15.7666 | 13.269  | 7.99957 | 10.8519  | 12.4951 | 3.96799 | 4.50756 |     29.3929 |    7.96441 |
| (50, 'mrr')  | 58.4344 | 20.066  | 17.2659 | 37.5026 | 52.583  | 25.8256 | 24.5214 | 15.8159 | 13.3767 | 8.00606 | 10.9744  | 12.7093 | 3.98905 | 4.44717 |     29.4879 |    8.02518 |
| (55, 'mrr')  | 58.5214 | 20.3003 | 17.4971 | 37.7731 | 53.2195 | 26.6214 | 24.9088 | 15.8859 | 13.3979 | 7.99773 | 11.0119  | 12.8666 | 4.00092 | 4.609   |     29.7917 |    8.09724 |
| (60, 'mrr')  | 58.5588 | 20.2791 | 17.4278 | 37.794  | 53.4732 | 26.7336 | 24.8551 | 15.9778 | 13.3923 | 8.05524 | 10.9982  | 12.7967 | 4.03883 | 4.58863 |     29.8324 |    8.09553 |
| (65, 'mrr')  | 58.5393 | 20.3012 | 17.3731 | 37.7334 | 53.5801 | 26.7743 | 25.0042 | 15.9129 | 13.4806 | 7.98234 | 10.9909  | 12.871  | 4.0504  | 4.57675 |     29.8555 |    8.09429 |
| (70, 'mrr')  | 58.5464 | 20.3899 | 17.636  | 37.8725 | 53.5272 | 26.8348 | 25.017  | 15.8722 | 13.3682 | 8.00063 | 10.9787  | 12.8349 | 4.09206 | 4.54549 |     29.896  |    8.09037 |
| (75, 'mrr')  | 58.5244 | 20.3452 | 17.5419 | 37.9096 | 53.6987 | 26.9608 | 24.9611 | 15.8747 | 13.3377 | 8.0151  | 10.9958  | 12.8542 | 4.07362 | 4.58094 |     29.906  |    8.10395 |
| (80, 'mrr')  | 58.4666 | 20.3634 | 17.4013 | 37.864  | 53.7288 | 26.872  | 25.0944 | 15.8941 | 13.4476 | 8.00644 | 11.0202  | 12.889  | 4.03659 | 4.56708 |     29.9036 |    8.10385 |
| (85, 'mrr')  | 58.5246 | 20.3625 | 17.4951 | 37.8034 | 53.7097 | 26.8793 | 25.0511 | 15.7963 | 13.332  | 8.00327 | 10.9794  | 12.8392 | 4.04728 | 4.55166 |     29.8838 |    8.08415 |
| (90, 'mrr')  | 58.5385 | 20.2787 | 17.4881 | 37.8201 | 53.708  | 26.9801 | 25.1586 | 15.8513 | 13.3464 | 8.01791 | 10.9962  | 12.8424 | 4.04243 | 4.56984 |     29.9078 |    8.09376 |
| (95, 'mrr')  | 58.5262 | 20.3174 | 17.407  | 37.9197 | 53.704  | 27.007  | 24.9786 | 15.8147 | 13.3705 | 8.03061 | 11.0391  | 12.7607 | 4.0563  | 4.51442 |     29.8939 |    8.08023 |
| (100, 'mrr') | 58.4876 | 20.3739 | 17.5426 | 37.8685 | 53.7853 | 27.0437 | 25.0912 | 15.8391 | 13.3677 | 8.03276 | 11.0813  | 12.8224 | 4.02579 | 4.43816 |     29.9333 |    8.08009 |

| Test set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 53.3178 | 17.4509 | 13.921  | 34.3557 | 42.5244 | 18.9655 | 19.9409 | 15.5624 | 14.1614 | 3.09049 |  9.33451 | 10.4117 | 3.55259 | 3.61193 |     25.5778 |    6.00025 |
| (10, 'mrr')  | 60.0538 | 20.7879 | 16.6129 | 38.033  | 47.0486 | 24.1357 | 23.079  | 17.3772 | 15.8912 | 8.24614 | 10.7019  | 11.5071 | 3.87626 | 4.72569 |     29.2244 |    7.81142 |
| (15, 'mrr')  | 60.0531 | 21.0174 | 17.3599 | 38.6041 | 48.159  | 25.4717 | 23.4817 | 17.3495 | 15.6388 | 8.76846 | 10.9286  | 11.972  | 3.92587 | 4.70564 |     29.6817 |    8.06013 |
| (20, 'mrr')  | 60.0223 | 21.3595 | 17.3279 | 38.7731 | 48.7933 | 25.9714 | 24.0966 | 17.2853 | 15.8172 | 8.98501 | 10.8567  | 11.8679 | 4.01083 | 4.63997 |     29.9385 |    8.07209 |
| (25, 'mrr')  | 60.1999 | 21.4644 | 17.2901 | 38.9674 | 49.2416 | 26.0032 | 24.1828 | 17.2671 | 15.7017 | 8.6811  | 11.0215  | 11.7445 | 3.92222 | 4.81752 |     30.0353 |    8.03736 |
| (30, 'mrr')  | 60.39   | 21.9168 | 17.0032 | 39.2801 | 49.9484 | 27.3184 | 24.6839 | 17.1173 | 15.9443 | 8.57809 | 10.8312  | 12.1397 | 3.83815 | 4.77605 |     30.4003 |    8.03264 |
| (35, 'mrr')  | 60.3503 | 21.7325 | 17.2967 | 39.1128 | 49.7495 | 27.0008 | 24.6514 | 17.0884 | 15.861  | 8.59063 | 10.7803  | 12.1338 | 3.84769 | 4.80637 |     30.3159 |    8.03174 |
| (40, 'mrr')  | 60.4626 | 22.0204 | 17.3283 | 39.5281 | 50.6019 | 27.4767 | 24.6566 | 17.0303 | 15.7524 | 8.46947 | 10.864   | 12.333  | 3.92515 | 4.89782 |     30.5397 |    8.09789 |
| (45, 'mrr')  | 60.3293 | 22.1564 | 17.5824 | 39.4073 | 50.1337 | 27.8943 | 25.0868 | 16.9976 | 15.9754 | 8.70057 | 10.8894  | 12.1503 | 3.98509 | 4.82957 |     30.6181 |    8.111   |
| (50, 'mrr')  | 60.5006 | 21.9947 | 17.398  | 39.4997 | 50.1538 | 27.7933 | 24.8773 | 16.97   | 15.7931 | 8.42343 | 10.8244  | 12.2081 | 3.92818 | 4.73758 |     30.5534 |    8.02432 |
| (55, 'mrr')  | 60.5633 | 22.3015 | 17.636  | 39.9431 | 50.9257 | 28.5783 | 25.3599 | 17.0022 | 15.833  | 8.50931 | 10.8985  | 12.411  | 4.0115  | 4.83301 |     30.9048 |    8.13267 |
| (60, 'mrr')  | 60.5973 | 22.2343 | 17.6546 | 39.9555 | 50.9812 | 28.7155 | 25.2521 | 17.0157 | 15.7824 | 8.48907 | 10.9854  | 12.2893 | 3.96769 | 4.85918 |     30.9098 |    8.11811 |
| (65, 'mrr')  | 60.5589 | 22.3278 | 17.6482 | 40.0772 | 51.1738 | 28.8743 | 25.255  | 16.9732 | 15.8177 | 8.4574  | 10.9471  | 12.3755 | 3.99949 | 4.80515 |     30.9673 |    8.11693 |
| (70, 'mrr')  | 60.5047 | 22.2669 | 17.734  | 40.0701 | 51.2012 | 28.9174 | 25.2583 | 16.8859 | 15.7568 | 8.42383 | 11.0485  | 12.3892 | 4.00967 | 4.80891 |     30.955  |    8.13604 |
| (75, 'mrr')  | 60.5458 | 22.2862 | 17.6367 | 40.2007 | 51.2726 | 28.9536 | 25.2717 | 16.8917 | 15.7468 | 8.43721 | 10.9711  | 12.4433 | 3.97777 | 4.82057 |     30.9784 |    8.13    |
| (80, 'mrr')  | 60.5429 | 22.2779 | 17.5793 | 40.0852 | 51.1626 | 28.9895 | 25.3941 | 16.8825 | 15.7659 | 8.48395 | 10.9586  | 12.4836 | 3.98999 | 4.7884  |     30.9644 |    8.14091 |
| (85, 'mrr')  | 60.5592 | 22.352  | 17.6074 | 40.2472 | 51.2478 | 28.9383 | 25.3371 | 16.908  | 15.7983 | 8.41977 | 10.9511  | 12.4619 | 4.01124 | 4.80731 |     30.9995 |    8.13026 |
| (90, 'mrr')  | 60.5871 | 22.3782 | 17.4865 | 40.2134 | 51.3309 | 29.0312 | 25.318  | 16.9407 | 15.744  | 8.39682 | 10.9253  | 12.4119 | 4.03235 | 4.80307 |     31.0033 |    8.11389 |
| (95, 'mrr')  | 60.5521 | 22.3003 | 17.5146 | 40.4396 | 51.5141 | 28.99   | 25.3643 | 16.8287 | 15.7321 | 8.39906 | 11.0132  | 12.4137 | 4.01778 | 4.77633 |     31.0262 |    8.12402 |
| (100, 'mrr') | 60.5551 | 22.4637 | 17.6964 | 40.2327 | 51.3726 | 29.0957 | 25.449  | 16.8115 | 15.7569 | 8.42328 | 10.9956  | 12.3207 | 4.04476 | 4.85645 |     31.0482 |    8.12816 |

<details>
<summary> Results from versions before e83a95a are obtained from the training process where both the KG embedding and the MLP is trainable, which is an accidental mistake and NOT what proposed in the paper. The results with trainable KGE is be weaker than those with freezed KGE which is proposed in the paper. Click to see the sample output trajectory of trainable KGE (results of weak implementation of LMPNN) </summary>

### (depreciated) LMPNN trajectory without freezed KGE (weak implementation)

For FB15k-237, a possible output trajectory could be like

| Validation Set  |      1p |       2p |       3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |      3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|---------:|---------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 43.3374 |  9.3963  |  7.36107 | 28.8171 | 43.2848 | 12.7592 | 11.338  | 10.9825 | 8.06806 | 4.89454 |  9.03825 | 6.84014 | 3.57948 | 3.43009 |     19.4827 |    5.5565  |
| (10, 'mrr')  | 43.5847 | 10.2061  |  8.38958 | 30.1158 | 44.8668 | 13.7348 | 12.4519 | 11.1331 | 8.40302 | 5.50171 |  9.92851 | 7.19045 | 3.68003 | 3.68664 |     20.3207 |    5.99747 |
| (15, 'mrr')  | 43.1136 |  9.93098 |  8.41377 | 30.7293 | 45.7028 | 14.1373 | 12.4096 | 11.17   | 8.48811 | 5.90808 |  9.97395 | 7.14056 | 3.78556 | 3.88563 |     20.455  |    6.13876 |
| (20, 'mrr')  | 43.6212 | 10.5731  |  8.92229 | 30.8057 | 45.9869 | 15.2658 | 13.1256 | 11.0792 | 8.84129 | 5.74387 | 10.4465  | 7.32727 | 3.87902 | 3.76653 |     20.9135 |    6.23263 |
| (25, 'mrr')  | 43.5993 | 10.6286  |  9.14751 | 30.7887 | 46.2739 | 16.2113 | 13.8994 | 11.2555 | 8.81353 | 5.98244 | 10.6118  | 7.51721 | 3.7699  | 3.55735 |     21.1798 |    6.28774 |
| (30, 'mrr')  | 43.6251 | 10.8747  |  9.03938 | 30.9388 | 46.5563 | 16.4211 | 13.543  | 11.2324 | 9.15598 | 5.83169 | 10.5604  | 7.63332 | 3.9878  | 3.64097 |     21.2652 |    6.33083 |
| (35, 'mrr')  | 43.4778 | 11.0762  |  9.3528  | 31.3629 | 47.202  | 17.1742 | 13.701  | 11.2211 | 9.16062 | 5.97082 | 10.7614  | 7.62482 | 3.90409 | 3.7512  |     21.5254 |    6.40247 |
| (40, 'mrr')  | 43.4053 | 11.0612  |  9.56793 | 31.3726 | 47.2989 | 16.9117 | 13.6576 | 10.8811 | 9.21064 | 5.86364 | 10.6366  | 7.91589 | 3.90675 | 3.52537 |     21.4852 |    6.36965 |
| (45, 'mrr')  | 43.2611 | 11.0465  |  9.57849 | 31.2661 | 47.44   | 17.705  | 13.9036 | 11.0584 | 9.11347 | 5.98817 | 10.6902  | 7.86165 | 3.93147 | 3.75511 |     21.5969 |    6.44531 |
| (50, 'mrr')  | 42.9117 | 11.4134  |  9.45842 | 31.4618 | 47.5948 | 17.8229 | 13.9    | 11.0091 | 9.40493 | 5.89662 | 10.722   | 7.65385 | 4.07289 | 3.60841 |     21.6641 |    6.39074 |
| (55, 'mrr')  | 43.2503 | 11.6637  |  9.83592 | 31.7507 | 48.095  | 19.3408 | 14.4881 | 11.0594 | 9.40362 | 6.04338 | 10.8817  | 8.20831 | 4.18475 | 3.58321 |     22.0986 |    6.58026 |
| (60, 'mrr')  | 43.3788 | 11.7293  | 10.0087  | 31.8454 | 48.1824 | 19.6966 | 14.5867 | 11.0317 | 9.52452 | 6.06797 | 10.8322  | 8.18048 | 4.23077 | 3.65369 |     22.2205 |    6.59303 |
| (65, 'mrr')  | 43.265  | 11.6121  | 10.0383  | 32.0212 | 48.3501 | 19.8917 | 14.6462 | 11.0356 | 9.44041 | 6.03938 | 10.9391  | 8.08408 | 4.3106  | 3.65148 |     22.2556 |    6.60492 |
| (70, 'mrr')  | 43.2776 | 11.6904  |  9.98802 | 32.0637 | 48.5154 | 20.0188 | 14.7324 | 11.0485 | 9.54078 | 5.98941 | 11.0414  | 8.21996 | 4.22696 | 3.7204  |     22.3195 |    6.63963 |
| (75, 'mrr')  | 43.3091 | 11.7581  | 10.0377  | 32.1079 | 48.6767 | 20.1973 | 14.8317 | 11.0657 | 9.52549 | 6.03904 | 11.0798  | 8.28582 | 4.34577 | 3.63607 |     22.39   |    6.6773  |
| (80, 'mrr')  | 43.3248 | 11.7603  |  9.98525 | 32.2027 | 48.5901 | 20.2498 | 14.8366 | 11.0489 | 9.5386  | 5.98985 | 11.0667  | 8.26597 | 4.35415 | 3.66858 |     22.393  |    6.66906 |
| (85, 'mrr')  | 43.2465 | 11.7877  | 10.0583  | 32.1016 | 48.7363 | 20.4814 | 14.9262 | 10.9962 | 9.51707 | 6.02979 | 11.2297  | 8.3184  | 4.39278 | 3.72155 |     22.4279 |    6.73846 |
| (90, 'mrr')  | 43.376  | 11.6823  | 10.0461  | 32.241  | 48.6458 | 20.4281 | 14.8565 | 11.0369 | 9.5371  | 6.03664 | 11.1348  | 8.34831 | 4.32066 | 3.62985 |     22.4278 |    6.69404 |
| (95, 'mrr')  | 43.291  | 11.7701  |  9.97534 | 32.3046 | 48.8179 | 20.5445 | 14.8694 | 11.1149 | 9.53749 | 6.02634 | 11.2056  | 8.25653 | 4.34473 | 3.6933  |     22.4695 |    6.70531 |
| (100, 'mrr') | 43.2896 | 11.8108  |  9.95793 | 32.3719 | 48.9575 | 20.4634 | 14.8929 | 11.0351 | 9.56184 | 6.06656 | 11.1147  | 8.34275 | 4.35363 | 3.69552 |     22.4823 |    6.71463 |

| Test set |      1p |      2p |       3p |      2i |      3i |      pi |      ip |      2u |       up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|---------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|--------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 44.9465 | 10.4843 |  8.6229  | 31.7221 | 44.4341 | 14.2661 | 14.1401 | 13.6675 |  9.80541 | 6.86755 | 11.2436 | 6.54754 | 3.84052 | 4.55932 |     21.3432 |    6.61171 |
| (10, 'mrr')  | 45.1145 | 11.2116 |  9.19404 | 32.7899 | 46.5547 | 15.7508 | 15.1701 | 13.5092 |  9.95767 | 7.41947 | 11.6497 | 6.58225 | 3.93812 | 4.52801 |     22.1392 |    6.8235  |
| (15, 'mrr')  | 44.7565 | 11.6183 |  9.65737 | 33.0242 | 46.9324 | 16.0322 | 15.6126 | 13.9755 | 10.2088  | 7.83579 | 12.0131 | 6.88489 | 4.00784 | 4.6131  |     22.4242 |    7.07094 |
| (20, 'mrr')  | 45.3177 | 12.0233 |  9.57721 | 33.2067 | 47.472  | 17.4125 | 16.2467 | 13.6488 | 10.4001  | 8.05987 | 12.2843 | 7.14386 | 4.14163 | 4.45509 |     22.8117 |    7.21695 |
| (25, 'mrr')  | 45.1459 | 12.1309 |  9.8724  | 33.4177 | 47.7942 | 17.7643 | 16.4459 | 13.4371 | 10.4981  | 7.80966 | 12.3091 | 7.22651 | 4.23409 | 4.35564 |     22.9452 |    7.187   |
| (30, 'mrr')  | 45.2453 | 12.516  | 10.2946  | 33.8225 | 47.6455 | 18.1683 | 16.3805 | 13.5795 | 10.5048  | 7.82047 | 12.0024 | 7.65289 | 4.37242 | 4.55625 |     23.1285 |    7.28089 |
| (35, 'mrr')  | 44.9832 | 12.6714 | 10.3581  | 33.9382 | 48.1022 | 19.0562 | 16.7391 | 13.4728 | 10.3644  | 7.80266 | 12.4172 | 7.50164 | 4.27899 | 4.18954 |     23.2984 |    7.23801 |
| (40, 'mrr')  | 45.0497 | 12.9713 | 10.5534  | 33.9162 | 48.2494 | 19.4182 | 17.0646 | 13.3876 | 10.7647  | 7.76955 | 12.0863 | 7.76417 | 4.58433 | 4.18861 |     23.4861 |    7.27859 |
| (45, 'mrr')  | 44.8709 | 12.7598 | 10.6207  | 34.0123 | 48.4076 | 20.1168 | 16.7293 | 13.4826 | 10.7217  | 7.7207  | 12.2151 | 7.79834 | 4.38069 | 4.2472  |     23.5246 |    7.27242 |
| (50, 'mrr')  | 44.4331 | 12.9428 | 10.3695  | 33.9687 | 48.3719 | 20.2204 | 17.002  | 13.2265 | 10.7709  | 7.51472 | 12.3501 | 7.86527 | 4.52629 | 4.30918 |     23.4784 |    7.31311 |
| (55, 'mrr')  | 44.8656 | 13.2094 | 10.7748  | 34.5006 | 48.9649 | 21.6445 | 17.5028 | 13.3423 | 10.9163  | 7.63899 | 12.6289 | 8.30786 | 4.61518 | 4.30892 |     23.969  |    7.49997 |
| (60, 'mrr')  | 44.9011 | 13.3243 | 10.8495  | 34.6471 | 49.0874 | 22.1415 | 17.4582 | 13.3771 | 10.8835  | 7.62007 | 12.5587 | 8.2811  | 4.68471 | 4.34016 |     24.0744 |    7.49695 |
| (65, 'mrr')  | 44.882  | 13.4375 | 10.8931  | 34.7594 | 49.126  | 22.3102 | 17.6696 | 13.408  | 10.9255  | 7.63332 | 12.6301 | 8.24105 | 4.70683 | 4.2976  |     24.1568 |    7.50179 |
| (70, 'mrr')  | 44.8828 | 13.651  | 11.0718  | 34.7735 | 49.2482 | 22.703  | 17.7788 | 13.4791 | 11.0766  | 7.68911 | 12.7907 | 8.35583 | 4.76243 | 4.40152 |     24.2961 |    7.59992 |
| (75, 'mrr')  | 44.9174 | 13.555  | 11.012   | 34.906  | 49.3564 | 22.7873 | 17.6738 | 13.396  | 11.0057  | 7.63671 | 12.7132 | 8.41619 | 4.61401 | 4.29353 |     24.2899 |    7.53473 |
| (80, 'mrr')  | 44.9689 | 13.694  | 11.0733  | 34.9882 | 49.5035 | 22.9782 | 17.7274 | 13.3799 | 11.0732  | 7.6816  | 12.7935 | 8.37103 | 4.75101 | 4.35969 |     24.3763 |    7.59136 |
| (85, 'mrr')  | 44.8746 | 13.5736 | 11.0516  | 35.0654 | 49.612  | 23.1731 | 17.7409 | 13.4019 | 11.0125  | 7.63303 | 12.8699 | 8.32122 | 4.76494 | 4.31249 |     24.3895 |    7.58032 |
| (90, 'mrr')  | 44.963  | 13.6078 | 11.0434  | 35.0212 | 49.539  | 22.9025 | 17.6933 | 13.3906 | 10.9815  | 7.68771 | 12.8578 | 8.27871 | 4.70053 | 4.26779 |     24.3491 |    7.5585  |
| (95, 'mrr')  | 44.8846 | 13.665  | 11.0756  | 35.1892 | 49.8396 | 23.0492 | 17.7944 | 13.3872 | 11.0303  | 7.66467 | 12.8382 | 8.39552 | 4.71225 | 4.3251  |     24.435  |    7.58714 |
| (100, 'mrr') | 44.9048 | 13.6668 | 11.1112  | 35.2322 | 49.6774 | 22.859  | 17.8386 | 13.3444 | 11.0184  | 7.70513 | 12.857  | 8.45513 | 4.74044 | 4.26359 |     24.4059 |    7.60425 |

For FB15k dataset, a possible training trajectory could be

|Validation set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |      pin |      pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|---------:|------------:|-----------:|
| (5, 'mrr')   | 72.5264 | 18.9707 | 15.167  | 50.393  | 64.2903 | 21.3955 | 22.4212 | 23.7883 | 17.7087 | 15.89   | 14.4604 |  9.5854 |  6.1095  |  8.80865 |     34.0735 |    10.9708 |
| (10, 'mrr')  | 73.4743 | 21.3927 | 16.8809 | 51.2315 | 65.4991 | 25.2254 | 25.0429 | 24.3405 | 18.9479 | 16.2819 | 14.5931 | 10.1545 |  7.01239 | 10.0011  |     35.7817 |    11.6086 |
| (15, 'mrr')  | 74.0354 | 22.2156 | 17.726  | 51.3907 | 65.5969 | 27.1767 | 25.7572 | 24.2847 | 19.393  | 16.4947 | 14.6412 | 10.7316 |  7.59753 | 10.4718  |     36.3974 |    11.9873 |
| (20, 'mrr')  | 74.1509 | 23.2951 | 18.3861 | 52.2457 | 66.8295 | 28.6767 | 26.7481 | 24.5087 | 20.1395 | 16.8086 | 15.0185 | 10.912  |  8.17509 | 10.3257  |     37.22   |    12.248  |
| (25, 'mrr')  | 74.3886 | 23.6987 | 18.7169 | 52.2513 | 66.6926 | 29.6802 | 27.3806 | 24.6967 | 20.545  | 16.4154 | 15.0412 | 11.3016 |  8.45187 | 10.6056  |     37.5612 |    12.3631 |
| (30, 'mrr')  | 74.5874 | 24.1382 | 18.9701 | 52.1197 | 67.0708 | 31.0522 | 27.5473 | 24.7312 | 20.6051 | 16.5712 | 15.397  | 11.4983 |  8.42595 | 10.7639  |     37.8691 |    12.5313 |
| (35, 'mrr')  | 74.6497 | 24.6292 | 19.136  | 52.0571 | 66.9698 | 32.1509 | 28.2427 | 24.8105 | 20.8558 | 16.3211 | 15.2092 | 11.6967 |  8.96677 | 10.7479  |     38.1669 |    12.5883 |
| (40, 'mrr')  | 74.8573 | 24.9687 | 19.3029 | 52.2527 | 67.0853 | 33.2529 | 28.3276 | 24.8803 | 21.0188 | 16.4776 | 15.1542 | 11.8346 |  9.1182  | 11.0053  |     38.4385 |    12.718  |
| (45, 'mrr')  | 74.8077 | 25.2605 | 19.5295 | 52.5364 | 67.5456 | 33.4174 | 28.8868 | 24.7764 | 20.9093 | 16.5528 | 15.2427 | 11.7762 |  9.26389 | 11.0442  |     38.63   |    12.776  |
| (50, 'mrr')  | 74.7147 | 25.3948 | 19.6865 | 52.1841 | 66.9479 | 33.9548 | 28.9421 | 24.589  | 21.1452 | 16.2819 | 15.3491 | 11.9337 |  9.17793 | 10.933   |     38.6177 |    12.7351 |
| (55, 'mrr')  | 75.2392 | 26.0655 | 20.4374 | 52.7575 | 68.0126 | 35.7402 | 29.763  | 24.9457 | 21.5412 | 16.7172 | 15.8523 | 12.292  |  9.61107 | 11.1913  |     39.3892 |    13.1328 |
| (60, 'mrr')  | 75.338  | 26.2711 | 20.4958 | 53.005  | 68.2882 | 36.3051 | 29.9832 | 24.9604 | 21.5683 | 16.7698 | 16.0298 | 12.4912 |  9.97094 | 11.293   |     39.5795 |    13.311  |
| (65, 'mrr')  | 75.4915 | 26.3433 | 20.7189 | 53.1031 | 68.4577 | 36.5135 | 30.125  | 24.9532 | 21.6608 | 16.7159 | 16.1205 | 12.445  | 10.1391  | 11.28    |     39.7075 |    13.3401 |
| (70, 'mrr')  | 75.6082 | 26.5294 | 20.858  | 53.2512 | 68.4007 | 36.8932 | 30.2989 | 25.08   | 21.7732 | 16.8102 | 16.2661 | 12.5809 | 10.1644  | 11.3312  |     39.8548 |    13.4306 |
| (75, 'mrr')  | 75.6264 | 26.779  | 20.9627 | 53.4878 | 68.5542 | 36.972  | 30.3147 | 25.0093 | 21.8702 | 16.8719 | 16.4148 | 12.5987 | 10.1621  | 11.4264  |     39.9529 |    13.4948 |
| (80, 'mrr')  | 75.6596 | 26.7891 | 20.9883 | 53.5604 | 68.6648 | 37.0346 | 30.4991 | 25.0008 | 21.9547 | 16.9228 | 16.4881 | 12.6153 | 10.5     | 11.3379  |     40.0168 |    13.5728 |
| (85, 'mrr')  | 75.7871 | 26.9072 | 21.0725 | 53.4992 | 68.6661 | 37.3166 | 30.5706 | 25.0393 | 21.9667 | 16.9707 | 16.5363 | 12.6744 | 10.5678  | 11.3997  |     40.0917 |    13.6298 |
| (90, 'mrr')  | 75.7685 | 26.9004 | 21.1033 | 53.5543 | 68.6671 | 37.4762 | 30.6839 | 25.0264 | 21.9165 | 16.9988 | 16.5336 | 12.6405 | 10.6902  | 11.2698  |     40.1219 |    13.6266 |
| (95, 'mrr')  | 75.8367 | 27.112  | 21.1882 | 53.6209 | 68.8099 | 37.4012 | 30.7478 | 25.1237 | 22.027  | 17.024  | 16.577  | 12.7249 | 10.588   | 11.4293  |     40.2075 |    13.6687 |
| (100, 'mrr') | 75.821  | 26.9972 | 21.1739 | 53.5756 | 68.8016 | 37.3668 | 30.7325 | 25.0958 | 21.967  | 16.9533 | 16.6115 | 12.6655 | 10.5621  | 11.4197  |     40.1702 |    13.6424 |

|Test set|      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |      pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|---------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 79.6252 | 27.0495 | 22.0475 | 63.0068 | 71.0271 | 26.8145 | 31.631  | 33.8454 | 26.428  | 25.348  | 26.4778 | 11.1268 |  7.37001 | 12.2931 |     42.3861 |    16.5231 |
| (10, 'mrr')  | 80.8636 | 30.8524 | 24.7873 | 64.5688 | 72.6059 | 32.4962 | 34.8987 | 34.8972 | 28.6328 | 26.4557 | 26.8073 | 12.2922 |  8.45513 | 14.3868 |     44.9559 |    17.6794 |
| (15, 'mrr')  | 81.5185 | 32.2857 | 25.5074 | 64.8339 | 72.9823 | 34.3122 | 36.1444 | 34.3031 | 29.9918 | 26.6691 | 27.0838 | 13.1753 |  9.03323 | 14.2974 |     45.7644 |    18.0518 |
| (20, 'mrr')  | 81.9271 | 34.1768 | 26.5718 | 65.453  | 73.8918 | 36.7933 | 37.8788 | 34.451  | 31.1975 | 26.7253 | 26.6824 | 13.0502 |  9.25748 | 14.8176 |     46.9268 |    18.1066 |
| (25, 'mrr')  | 82.1761 | 35.3178 | 27.2472 | 65.4745 | 74.0307 | 38.0896 | 38.8817 | 34.9898 | 31.5301 | 26.2991 | 26.4839 | 13.6902 |  9.53723 | 14.8901 |     47.5264 |    18.1801 |
| (30, 'mrr')  | 82.3555 | 35.6731 | 27.6514 | 65.0369 | 73.8001 | 40.1415 | 38.939  | 34.6648 | 31.4103 | 25.8967 | 26.2618 | 14.1097 |  9.66685 | 14.7401 |     47.7414 |    18.135  |
| (35, 'mrr')  | 82.5593 | 36.5284 | 27.9251 | 65.3617 | 74.0039 | 40.9399 | 39.6183 | 34.3864 | 31.5693 | 25.4552 | 26.0457 | 14.1313 |  9.76101 | 14.6804 |     48.0992 |    18.0147 |
| (40, 'mrr')  | 82.9059 | 37.9752 | 28.8003 | 65.5398 | 74.1289 | 42.1209 | 40.5559 | 34.4631 | 32.6308 | 25.4917 | 25.8993 | 14.6716 | 10.0891  | 14.8624 |     48.7912 |    18.2028 |
| (45, 'mrr')  | 82.8705 | 37.8872 | 29.0582 | 65.5965 | 74.1744 | 42.4366 | 40.5069 | 34.3821 | 32.6076 | 25.2364 | 25.7732 | 14.508  | 10.2368  | 14.9662 |     48.8356 |    18.1441 |
| (50, 'mrr')  | 82.9409 | 38.4537 | 29.3314 | 65.409  | 74.1168 | 43.6435 | 40.9284 | 34.5042 | 32.8697 | 24.8741 | 25.5653 | 14.8343 | 10.3195  | 15.0117 |     49.1331 |    18.1209 |
| (55, 'mrr')  | 83.6995 | 40.1008 | 30.5288 | 66.6587 | 75.1619 | 45.9406 | 42.3054 | 34.6511 | 33.7162 | 25.3503 | 26.1966 | 15.3033 | 10.573   | 15.2319 |     50.307  |    18.531  |
| (60, 'mrr')  | 83.8872 | 40.7408 | 30.9499 | 67.0044 | 75.4608 | 46.6269 | 42.8369 | 34.5836 | 34.0223 | 25.475  | 26.432  | 15.5052 | 10.7476  | 15.2947 |     50.6792 |    18.6909 |
| (65, 'mrr')  | 84.0518 | 40.849  | 30.9894 | 67.2783 | 75.6766 | 46.6758 | 42.9262 | 34.6531 | 34.0006 | 25.4424 | 26.5234 | 15.5519 | 10.7439  | 15.2883 |     50.789  |    18.71   |
| (70, 'mrr')  | 84.2106 | 41.3092 | 31.257  | 67.3251 | 75.8018 | 47.0875 | 43.2535 | 34.654  | 34.1175 | 25.3795 | 26.5757 | 15.6222 | 10.8237  | 15.2704 |     51.0018 |    18.7343 |
| (75, 'mrr')  | 84.2636 | 41.5525 | 31.5103 | 67.5114 | 75.7793 | 47.4208 | 43.6347 | 34.6501 | 34.2698 | 25.5468 | 26.6772 | 15.6955 | 10.8787  | 15.3086 |     51.1769 |    18.8214 |
| (80, 'mrr')  | 84.3512 | 41.571  | 31.4847 | 67.5161 | 75.975  | 47.2654 | 43.582  | 34.6848 | 34.3082 | 25.5773 | 26.7497 | 15.7046 | 10.9515  | 15.3087 |     51.1932 |    18.8584 |
| (85, 'mrr')  | 84.4401 | 41.7911 | 31.6551 | 67.6595 | 76.0263 | 47.5891 | 43.8328 | 34.6937 | 34.4209 | 25.6277 | 26.8003 | 15.8049 | 10.9324  | 15.4512 |     51.3454 |    18.9233 |
| (90, 'mrr')  | 84.4638 | 41.8142 | 31.7417 | 67.8218 | 76.1267 | 47.763  | 43.5757 | 34.6339 | 34.3527 | 25.4679 | 26.857  | 15.781  | 11.0279  | 15.3128 |     51.3659 |    18.8893 |
| (95, 'mrr')  | 84.6069 | 41.7902 | 31.7677 | 67.8683 | 76.2519 | 47.8811 | 43.8575 | 34.5829 | 34.5235 | 25.4988 | 26.8536 | 15.815  | 11.0273  | 15.3696 |     51.4589 |    18.9129 |
| (100, 'mrr') | 84.5962 | 41.8732 | 31.7129 | 67.8121 | 76.2344 | 47.7903 | 43.8667 | 34.6276 | 34.5437 | 25.6159 | 26.8515 | 15.8538 | 11.0693  | 15.3367 |     51.4508 |    18.9455 |

For NELL dataset, a possible training trajectory could be

| Validation set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 58.1204 | 17.1394 | 14.8171 | 35.1191 | 48.1434 | 18.216  | 19.8213 | 15.6274 | 12.3501 | 7.47669 | 10.6017 | 11.8062 | 4.0412  | 4.38074 |     26.5949 |    7.66129 |
| (10, 'mrr')  | 57.9101 | 17.9723 | 15.7368 | 35.6915 | 49.0143 | 19.5558 | 20.8033 | 15.7196 | 12.6459 | 8.01705 | 10.6385 | 12.3644 | 4.12027 | 4.34362 |     27.2277 |    7.89678 |
| (15, 'mrr')  | 57.9716 | 19.0338 | 16.2968 | 35.9256 | 49.8854 | 22.1367 | 21.8386 | 15.4901 | 12.8566 | 7.80956 | 10.5861 | 12.4902 | 4.04185 | 4.31369 |     27.9373 |    7.84828 |
| (20, 'mrr')  | 57.4168 | 18.7561 | 16.7115 | 35.977  | 49.8357 | 20.4362 | 21.6066 | 15.3487 | 12.7226 | 7.66178 | 10.4122 | 12.2952 | 4.02626 | 4.42086 |     27.6457 |    7.76325 |
| (25, 'mrr')  | 57.0351 | 19.1695 | 16.6744 | 36.701  | 51.1331 | 23.5681 | 22.4403 | 15.3076 | 12.8107 | 7.91373 | 10.5872 | 12.7712 | 4.07904 | 4.35931 |     28.3155 |    7.94209 |
| (30, 'mrr')  | 57.1963 | 19.2891 | 16.9377 | 36.7586 | 51.0399 | 23.756  | 22.522  | 15.2679 | 12.7877 | 7.5976  | 10.5313 | 12.6761 | 4.02105 | 4.47263 |     28.395  |    7.85975 |
| (35, 'mrr')  | 56.9236 | 19.3807 | 17.0099 | 36.9825 | 51.3761 | 23.6143 | 22.3366 | 15.2227 | 13.0573 | 7.32792 | 10.6499 | 12.4046 | 4.1504  | 4.44791 |     28.4337 |    7.79616 |
| (40, 'mrr')  | 56.8403 | 19.4833 | 16.8577 | 36.5935 | 51.3677 | 23.9328 | 22.81   | 15.1275 | 13.0253 | 7.45651 | 10.7676 | 12.6684 | 4.27722 | 4.56407 |     28.4487 |    7.94677 |
| (45, 'mrr')  | 56.5277 | 19.4434 | 16.9731 | 36.6514 | 50.8131 | 23.8585 | 22.523  | 14.9389 | 13.0073 | 7.48076 | 10.4497 | 12.7325 | 4.09825 | 4.52448 |     28.304  |    7.85714 |
| (50, 'mrr')  | 56.5823 | 19.3082 | 17.0722 | 36.9401 | 51.1826 | 24.1982 | 22.5559 | 14.8032 | 12.8865 | 7.30333 | 10.58   | 12.8101 | 4.07694 | 4.2912  |     28.3921 |    7.81231 |
| (55, 'mrr')  | 56.7491 | 19.6617 | 17.3232 | 37.4124 | 51.9701 | 25.0041 | 23.0953 | 14.9818 | 12.9978 | 7.29917 | 10.6776 | 13.0981 | 4.1533  | 4.38353 |     28.7995 |    7.92233 |
| (60, 'mrr')  | 56.747  | 19.7013 | 17.3851 | 37.4219 | 51.9863 | 25.0253 | 23.2136 | 15.0225 | 13.0185 | 7.31288 | 10.7236 | 13.1495 | 4.09421 | 4.39861 |     28.8357 |    7.93576 |
| (65, 'mrr')  | 56.7386 | 19.7831 | 17.3879 | 37.5423 | 51.9313 | 25.2353 | 23.2883 | 14.9899 | 13.0879 | 7.33864 | 10.685  | 13.1793 | 4.1111  | 4.405   |     28.8872 |    7.9438  |
| (70, 'mrr')  | 56.7031 | 19.809  | 17.4155 | 37.5206 | 52.1556 | 25.4837 | 23.2963 | 15.0402 | 13.174  | 7.32878 | 10.7495 | 13.2333 | 4.09398 | 4.40422 |     28.9553 |    7.96194 |
| (75, 'mrr')  | 56.6914 | 19.7878 | 17.407  | 37.4702 | 52.2481 | 25.3543 | 23.3908 | 15.0409 | 13.1152 | 7.31722 | 10.7442 | 13.1681 | 4.13749 | 4.43393 |     28.9451 |    7.96018 |
| (80, 'mrr')  | 56.7012 | 19.8272 | 17.4449 | 37.5907 | 52.2383 | 25.4022 | 23.4482 | 14.989  | 13.0254 | 7.35492 | 10.777  | 13.1616 | 4.14352 | 4.39464 |     28.963  |    7.96634 |
| (85, 'mrr')  | 56.7686 | 19.8192 | 17.5098 | 37.4502 | 52.1744 | 25.5558 | 23.3911 | 15.0243 | 13.064  | 7.28512 | 10.8985 | 13.197  | 4.1342  | 4.45095 |     28.973  |    7.99316 |
| (90, 'mrr')  | 56.7139 | 19.8626 | 17.4418 | 37.6441 | 52.174  | 25.4519 | 23.5212 | 14.9292 | 13.0997 | 7.30827 | 10.9621 | 13.1915 | 4.17801 | 4.43536 |     28.982  |    8.01505 |
| (95, 'mrr')  | 56.6114 | 19.8485 | 17.6015 | 37.5606 | 52.3552 | 25.4303 | 23.4222 | 14.9639 | 13.0974 | 7.3714  | 10.9159 | 13.2139 | 4.17094 | 4.42943 |     28.9879 |    8.02032 |
| (100, 'mrr') | 56.5304 | 19.837  | 17.5056 | 37.557  | 52.61   | 25.68   | 23.4784 | 14.9258 | 13.0932 | 7.39956 | 10.8602 | 13.1604 | 4.17135 | 4.40587 |     29.0242 |    7.99949 |

| Test set |      1p |      2p |      3p |      2i |      3i |      pi |      ip |      2u |      up |     2in |     3in |     inp |     pin |     pni |   epfo mean |   Neg mean |
|:-------------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|------------:|-----------:|
| (5, 'mrr')   | 59.9925 | 18.7431 | 15.3488 | 37.131  | 45.7525 | 19.6135 | 20.6219 | 17.2678 | 14.6512 | 7.65617 | 10.6113 | 11.183  | 3.77651 | 4.2455  |     27.6802 |    7.4945  |
| (10, 'mrr')  | 59.3839 | 19.619  | 15.7485 | 37.7121 | 46.9139 | 21.1344 | 21.3127 | 16.9185 | 14.8788 | 8.27667 | 10.6757 | 11.5308 | 3.65816 | 4.32677 |     28.1802 |    7.69362 |
| (15, 'mrr')  | 59.515  | 20.5792 | 16.5123 | 38.2234 | 47.3892 | 23.91   | 22.5951 | 16.8399 | 15.4843 | 8.11972 | 10.6834 | 12.132  | 3.79222 | 4.60354 |     29.0054 |    7.86619 |
| (20, 'mrr')  | 59.0143 | 20.6289 | 17.0332 | 38.2814 | 47.7713 | 21.9448 | 22.3699 | 16.8168 | 15.3931 | 7.85623 | 10.8544 | 11.9253 | 3.80776 | 3.9679  |     28.806  |    7.68231 |
| (25, 'mrr')  | 58.7151 | 21.418  | 17.1238 | 38.6686 | 48.7802 | 24.9183 | 22.9127 | 16.5135 | 15.4847 | 8.29464 | 10.7124 | 12.1862 | 3.90752 | 4.75549 |     29.3927 |    7.97125 |
| (30, 'mrr')  | 58.7759 | 21.42   | 17.1072 | 38.729  | 48.6958 | 24.9884 | 22.9775 | 16.5603 | 15.5697 | 7.92933 | 10.7024 | 12.1284 | 4.05308 | 4.57014 |     29.4249 |    7.87666 |
| (35, 'mrr')  | 58.5786 | 21.377  | 17.0758 | 38.9681 | 48.9368 | 25.2203 | 22.4404 | 16.4391 | 15.3977 | 7.92863 | 10.5248 | 12.2395 | 3.94388 | 4.69192 |     29.3815 |    7.86574 |
| (40, 'mrr')  | 58.3774 | 21.4076 | 17.0121 | 39.0265 | 49.08   | 25.3503 | 23.0431 | 16.1658 | 15.5395 | 7.9038  | 10.4779 | 12.3383 | 3.98636 | 4.66744 |     29.4447 |    7.87476 |
| (45, 'mrr')  | 58.0577 | 21.2919 | 17.2616 | 38.8111 | 48.8938 | 25.5967 | 23.1196 | 16.1437 | 15.38   | 7.93512 | 10.43   | 12.1186 | 3.95103 | 4.61853 |     29.3951 |    7.81065 |
| (50, 'mrr')  | 58.2683 | 21.2357 | 17.0527 | 39.0259 | 48.6985 | 25.8154 | 22.7295 | 16.2029 | 15.108  | 7.81959 | 10.5743 | 11.8353 | 3.9347  | 4.52063 |     29.3486 |    7.7369  |
| (55, 'mrr')  | 58.4405 | 21.8109 | 17.518  | 39.4844 | 49.5178 | 26.5499 | 23.3775 | 16.2771 | 15.5987 | 7.91742 | 10.5566 | 12.1564 | 3.93595 | 4.59566 |     29.8417 |    7.8324  |
| (60, 'mrr')  | 58.4016 | 21.886  | 17.6116 | 39.6262 | 49.5139 | 26.4611 | 23.2744 | 16.2556 | 15.4958 | 7.90128 | 10.5776 | 12.3606 | 3.93631 | 4.58478 |     29.8362 |    7.87211 |
| (65, 'mrr')  | 58.3851 | 21.8665 | 17.5549 | 39.5868 | 49.6965 | 26.753  | 23.4442 | 16.2629 | 15.5942 | 7.90126 | 10.5395 | 12.3257 | 4.00136 | 4.59561 |     29.9049 |    7.87268 |
| (70, 'mrr')  | 58.3645 | 21.8921 | 17.6507 | 39.5146 | 49.9237 | 26.8391 | 23.553  | 16.2191 | 15.5815 | 7.85621 | 10.5537 | 12.3672 | 3.96091 | 4.63597 |     29.9487 |    7.87479 |
| (75, 'mrr')  | 58.3917 | 21.9669 | 17.7499 | 39.7617 | 49.9059 | 26.797  | 23.6986 | 16.1471 | 15.7068 | 7.90412 | 10.5253 | 12.3506 | 3.97828 | 4.63258 |     30.014  |    7.87817 |
| (80, 'mrr')  | 58.3979 | 22.1004 | 17.8992 | 39.8158 | 49.8562 | 27.1052 | 23.7746 | 16.2492 | 15.7949 | 7.94933 | 10.5536 | 12.3678 | 3.97498 | 4.61852 |     30.1104 |    7.89284 |
| (85, 'mrr')  | 58.3532 | 22.0636 | 17.7837 | 39.9643 | 50.0419 | 27.1929 | 23.7004 | 16.2874 | 15.6986 | 7.92375 | 10.549  | 12.439  | 3.98941 | 4.65393 |     30.1207 |    7.911   |
| (90, 'mrr')  | 58.3807 | 22.1138 | 17.7913 | 39.807  | 50.0646 | 27.2867 | 23.715  | 16.2238 | 15.7525 | 7.8916  | 10.5542 | 12.402  | 4.04165 | 4.60234 |     30.1262 |    7.89835 |
| (95, 'mrr')  | 58.3548 | 21.9695 | 17.8834 | 39.8838 | 50.0381 | 27.0707 | 23.5618 | 16.1987 | 15.5789 | 7.87723 | 10.5658 | 12.3075 | 4.06027 | 4.62175 |     30.06   |    7.88651 |
| (100, 'mrr') | 58.2411 | 22.1544 | 17.8277 | 39.9782 | 50.2438 | 27.1149 | 23.6624 | 16.2382 | 15.6717 | 7.91782 | 10.6511 | 12.4242 | 3.99122 | 4.63306 |     30.1258 |    7.92348 |

</details>

## Citing this paper

```bibtex
@inproceedings{LMPNN,
  author    = {Zihao Wang and
               Yangqiu Song and
               Ginny Y. Wong and
               Simon See},
  title     = {Logical Message Passing Networks with One-hop Inference on Atomic Formulas},
  booktitle = {The Eleventh International Conference on Learning Representations, {ICLR} 2023, Kigali Rwanda, May 1-5, 2023},
  publisher = {OpenReview.net},
  year      = {2023},
  url       = {https://openreview.net/forum?id=SoyOsp7i_l},
}
```

LMPNN

LMPNN 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

LMPNN

LMPNN 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README