Skip to content
/ LMPNN Public

Logical Message Passing Networks with One-hop Inference in Atomic Formulas (ICLR 2023)

15 stars 3 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

HKUST-KnowComp/LMPNN

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

41 Commits
config/kge
config/kge
 
 
cqd @ bcb5d14
cqd @ bcb5d14
 
 
kge @ f4fa52d
kge @ f4fa52d
 
 
src
src
 
 
ssl-relation-prediction @ a050286
ssl-relation-prediction @ a050286
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
ENV1.yaml
ENV1.yaml
 
 
README.md
README.md
 
 
convert_beta_dataset.py
convert_beta_dataset.py
 
 
convert_cqd_pretrain_ckpts.py
convert_cqd_pretrain_ckpts.py
 
 
convert_kg_data_for_kge.py
convert_kg_data_for_kge.py
 
 
convert_kg_data_for_ssl.py
convert_kg_data_for_ssl.py
 
 
read_eval_from_log.py
read_eval_from_log.py
 
 
requirements.txt
requirements.txt
 
 
script_prepare.sh
script_prepare.sh
 
 
train_lmpnn.py
train_lmpnn.py
 
 

Repository files navigation

Logical Message Passing Networks with One-hop Inference on Atomic Formula

Implementation for ICLR 2023 paper:

Logical Message Passing Networks with One-hop Inference on Atomic Formula

see the arXiv version and the OpenReview version.

In this documentation, we detail how to reproduce the results in the paper based on existing checkpoints released by other researchers.

Requirement of this repository

  • pytorch
  • jupyter
  • tqdm

Requirement of other submodules will be discussed accordingly.

My conda environment could be found at ENV1.yaml

Version

  1. Update 2023/07/20. We fix the non-freezed KG embedding issue before the commit id e83a95a. New version fixes the KG embedding and is now fully consistent as the paper.

** This repo is under construction for usability. The key results for the paper can already be reproduced. **

Todo features:

  • Implement CQD CO Extended, CQD(E) in the paper
  • Introduce several ways to train KGE checkpoints with released repositories, which can be backbones for LMPNN.

Preparation

(A) Prepare the dataset

Please download the dataset from snap-stanford/KGReasoning.

Specifically, one can run:

mkdir data
cd data
wget http://snap.stanford.edu/betae/KG_data.zip # a zip file of 1.3G
unzip KG_data.zip

Then the data folder will contain the following folders and files:

FB15k-237-betae
FB15k-237-q2b
FB15k-betae
FB15k-q2b
KG_data.zip
NELL-betae
NELL-q2b

We rearange them into different subfolders:

mkdir betae-dataset
mv *betae betae-dataset
mkdir q2b-dataset
mv *q2b q2b-dataset

Finally, run convert_beta_dataset.py to convert the data into the graph forms for LMPNN. One can find the new dataset folders in ./data.

An example converted dataset format is

data/FB15k-237-betae
  - kgindex.json
  - train_kg.tsv
  - valid_kg.tsv
  - test_kg.tsv
  - train-qaa.json
  - valid-qaa.json
  - test-qaa.json

where

  • kgindex.json file stores the relation/entity names and their ids,
  • {train/valid/test}_kg.tsv stores the triples in three knowledge graphs (triples in train_kg.tsv is the subset of those in valid_kg.tsv, and triples valid_kg.tsv is also the subset of those in test_kg.tsv)
  • {train/valid/test}-qaa.json stores the Query, easy Answers and hard Answers for train, valid, and test set.

(B1) Pretrain KGE checkpoints with external submodules

We consider two different repositories to pretrain the KGE checkpoints. Including

  1. uma-pi1/kge
  2. facebookresearch/ssl-relation-prediction

To initialize these modules, please run

git submodule update

How to train the checkpoints with these submodules is discussed in this section.

Generally, there are two steps, once the KG is prepared:

  1. Convert the KG triples into the format that can be used in each submodule.
  2. Train the checkpoints.

Choice 1: Pretrain with uma-pi1/kge

Step 0: Prepare the environment and config

To run libkge submodule, one need editable installation.

cd kge
pip install -e .
Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into ./kge/data.

python convert_kg_data_for_kge.py
Step 2: Train the checkpoint

We provide a config in config/kge/fb15k-237-complex.yaml. Tailor the config to train checkpoints with libkge.

kge start config/kge/fb15k-237-complex.yaml --job.device cuda:0

The obtained checkpoints can be found at kge/local.

Choice 2: Pretrain with facebookresearch/ssl-relation-prediction

Step 0: Prepare the environment
pip install ogb networkx wandb
Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into ./ssl-relation-prediction/data.

python convert_kg_data_for_ssl.py

Notably, in the ssl-relation-prediction/src/main.py file, the command arg parser prohibits external sources of datasets in line 47-50.

parser.add_argument(
    '--dataset', choices=datasets,
    help="Dataset in {}".format(datasets)
)

Let's change it into

parser.add_argument(
    '--dataset'
)
Step 2: Train the checkpoint

The training process can be initialized by running

cd ssl-relation-prediction
python src/main.py --dataset FB15k-237-betae \
               --score_rel True \
               --model ComplEx \
               --rank 1000 \
               --learning_rate 0.1 \
               --batch_size 1000 \
               --lmbda 0.05 \
               --w_rel 4 \
               --max_epochs 100 \
               --model_cache_path ./ckpts/FB15k-237-complex/

The obtained checkpoints can be found at ssl-relation-prediction/ckpts/FB15k-237-complex.

(B2) Convert pretrained KGE checkpoints into the acceptable format

We convert external KGE checkpoints into the format that can be loaded by LMPNN. We consider three sources of external checkpoints

  1. Checkpoints released by uclnlp/cqd.
  2. Checkpoints released / pretrained by uma-pi1/kge
  3. Checkpoints released by facebookresearch/ssl-relation-prediction

The pretrained checkpoints are managed in the folder pretrain.

mkdir pretrain

Sources 1. uclnlp/cqd

This source of checkpoints is used to repreduced the results shown in the paper.

cd pretrain
wget http://data.neuralnoise.com/cqd-models.tgz # a .tgz file of 4.8G
tar xvf cqd-models.tgz
mv models raw_cqd_pretrain_models

Then we can convert the checkpoints into the format used in this repo.

python convert_cqd_pretrain_ckpts.py

Train LMPNN

Sample usage at FB15k-237

python3 train_lmpnn.py \
  --task_folder data/FB15k-237-betae \
  --output_dir log/FB15k-237/lmpnn-complex1k-default \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt \
  --embedding_dim 1000 \
  --device cuda:0

Sample usage at FB15k

python3 train_lmpnn.py \
  --task_folder data/FB15k-betae \
  --checkpoint_path pretrain/cqd/FB15k-model-rank-1000-epoch-100-1602520745.pt \
  --device cuda:1 \
  --hidden_dim 8192 \
  --output_dir log/FB15k/lmpnn-complex1k-default

Sample usage at NELL

python3 train_lmpnn.py \
  --task_folder data/NELL-betae \
  --checkpoint_path pretrain/cqd/NELL-model-rank-1000-epoch-100-1602499096.pt \
  --device cuda:2 \
  --hidden_dim 8192 \
  --temp 0.1 \
  --batch_size 512 \
  --batch_size_eval_dataloader 8 \
  --batch_size_eval_truth_value 1 \
  --output_dir log/NELL/lmpnn-complex1k-default

Answering Existential First Order (EFO) queries

In this repository, the capability of answering EFO-1 queries is implemented by the reasoners.

  • CQD-CO is implemented as GradientEFOReasoner, which is refered as CQD(E) in the paper.
  • LMPNN is implemented as GNNEFOReasoner with LogicalGNNLayer
python train_lmpnn.py \
  --reasoner gradient \
  --device cuda:1 \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt

Summarize the results from log files

We use script read_eval_from_log.py to summarize the results from the log file.

For example, the results on FB15k-237 in file log/FB15k-237/pretrain_complex1000-default/output.log can be summarized by the following piece of code.

python3 read_eval_from_log.py --log_file log/FB15k-237/pretrain_complex1000-default/output.log

Then the code will output the markdown table of the trajectory over valid and test set.

Sample training trajectories of LMPNN with command provided above.

For FB15k-237, a possible output trajectory could be like

Validation Set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 43.1522 8.96845 7.30686 28.5481 42.3203 12.6179 10.7356 10.8988 7.9376 5.15343 9.14102 6.64968 3.48541 3.33012 19.1651 5.55193
(10, 'mrr') 43.1469 9.72417 7.70923 29.5263 44.0089 14.4818 12.1041 11.0914 8.02962 5.83145 9.66503 7.01406 3.67668 3.64304 19.9803 5.96605
(15, 'mrr') 43.8283 9.78142 7.81307 29.8871 44.3047 15.0133 12.4008 11.139 8.22536 5.83979 9.86778 7.0962 3.70381 3.79685 20.2659 6.06089
(20, 'mrr') 43.8607 10.5252 8.27492 30.7076 45.3577 16.3521 13.0123 11.2875 8.52308 6.13657 10.185 7.29428 3.74871 3.91729 20.8779 6.25637
(25, 'mrr') 44.0598 10.411 8.37461 30.2869 45.1289 16.2044 12.8736 11.2198 8.30653 6.02084 9.82021 7.21201 3.9356 4.02344 20.7628 6.20242
(30, 'mrr') 43.8415 10.4777 8.47792 30.7736 45.5773 16.8041 13.0142 11.0647 8.35672 6.05398 10.335 7.39058 3.73573 3.95478 20.932 6.29401
(35, 'mrr') 44.0388 10.4904 8.66033 30.8991 45.9508 17.6397 13.2279 11.2514 8.44008 6.16027 10.3354 7.46953 3.80763 3.92426 21.1776 6.33942
(40, 'mrr') 43.7431 10.7154 8.9294 30.9381 46.091 17.4797 13.3665 11.1077 8.54201 6.10274 10.445 7.4784 3.8803 4.01805 21.2125 6.38489
(45, 'mrr') 44.0037 10.8814 8.71511 31.0536 46.4162 17.7432 13.4061 11.2816 8.40346 6.2916 10.5165 7.46814 3.88711 3.94484 21.3227 6.42164
(50, 'mrr') 44.1079 10.8688 8.62299 31.3121 46.5769 17.9518 13.4356 11.3143 8.59079 6.23409 10.398 7.76916 3.77844 3.99303 21.4201 6.43454
(55, 'mrr') 44.2142 11.2075 9.15606 31.7812 47.2926 19.3201 13.9342 11.3463 8.5981 6.29677 10.6873 7.93258 4.02479 4.05711 21.8722 6.59971
(60, 'mrr') 44.1963 11.2091 9.26216 31.7813 47.5042 19.5144 13.9952 11.271 8.60634 6.35608 10.7975 7.85765 4.06059 4.00284 21.9267 6.61492
(65, 'mrr') 44.1878 11.2833 9.23951 31.784 47.5272 19.8171 14.0538 11.2238 8.56164 6.31577 10.7261 7.89386 4.12592 3.97752 21.9642 6.60783
(70, 'mrr') 44.1557 11.3497 9.26543 31.8273 47.6473 19.8378 14.0356 11.1839 8.57884 6.28838 10.7356 7.88885 4.12011 4.00915 21.9868 6.60841
(75, 'mrr') 44.1301 11.3565 9.37637 31.8551 47.6066 19.965 14.0906 11.1939 8.55513 6.30744 10.7699 7.76835 4.11392 3.93539 22.0144 6.57901
(80, 'mrr') 44.1289 11.3645 9.34517 31.865 47.8433 20.0669 14.1551 11.2427 8.65353 6.30529 10.7471 7.92971 4.12401 3.95507 22.0739 6.61224
(85, 'mrr') 44.1127 11.4596 9.25204 31.7441 47.8254 19.9052 14.126 11.1764 8.62984 6.32005 10.7557 7.89449 4.16561 3.90074 22.0257 6.60732
(90, 'mrr') 44.1258 11.3737 9.32294 31.9144 47.8743 20.0821 14.1917 11.2108 8.62529 6.34314 10.7944 7.92542 4.1587 3.94927 22.0801 6.63419
(95, 'mrr') 44.1069 11.4469 9.35151 32.0017 47.8526 20.354 14.2962 11.1435 8.56965 6.39189 10.7905 7.83577 4.21076 3.96381 22.1248 6.63854
(100, 'mrr') 44.1638 11.41 9.49996 31.8933 47.9468 20.2255 14.3075 11.1681 8.64452 6.34371 10.73 7.96422 4.20087 3.91509 22.1399 6.63077
Test set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 44.8991 10.3212 8.21212 31.3772 43.8674 14.1751 13.3347 13.6166 9.15165 7.003 10.8464 6.3307 3.73892 4.149 20.995 6.41361
(10, 'mrr') 44.7485 11.0422 9.01639 32.5366 45.5268 16.815 14.8415 13.68 9.72893 7.78859 11.6098 6.77476 4.03303 4.59245 21.9929 6.95973
(15, 'mrr') 45.4805 11.4756 9.00117 32.716 46.0811 17.2183 15.2297 13.7089 9.82214 8.14234 11.7426 6.82549 3.95403 4.8102 22.3037 7.09494
(20, 'mrr') 45.5168 11.8394 9.10475 33.2281 46.7741 18.0997 15.6644 13.8107 10.1112 8.58388 12.3831 6.96503 4.17977 4.99503 22.6832 7.42136
(25, 'mrr') 45.6835 11.8071 9.46537 33.4615 46.7437 18.3839 15.7297 13.6642 10.0942 8.47826 12.0139 7.21188 4.31563 4.96695 22.7815 7.39732
(30, 'mrr') 45.5894 11.8349 9.42184 33.2727 47.1798 19.1685 16.0285 13.5988 10.0361 8.26477 12.2796 7.02654 4.05282 5.24071 22.9034 7.37289
(35, 'mrr') 45.6687 12.2878 9.57961 33.8039 47.5376 19.7294 16.4946 13.7893 9.99798 8.55637 12.4467 7.06137 4.13864 5.08787 23.2099 7.45819
(40, 'mrr') 45.401 12.1917 9.29751 33.5219 47.5335 19.7207 16.4359 13.6639 10.03 8.39218 12.417 7.24734 4.20833 5.00166 23.0885 7.45331
(45, 'mrr') 45.8186 12.365 9.92397 34.0337 47.9003 20.0247 16.6008 13.6968 9.96648 8.62486 12.8463 7.27353 4.38398 5.05679 23.37 7.63709
(50, 'mrr') 45.633 12.3816 10.1212 33.9538 48.0138 20.1938 16.5077 13.6692 10.2213 8.70553 12.7164 7.41705 4.48108 4.98651 23.4106 7.6613
(55, 'mrr') 45.8942 12.7871 10.2142 34.3762 48.621 21.5511 17.0483 13.6615 10.2328 8.79375 12.9544 7.61405 4.55091 5.12093 23.8207 7.80681
(60, 'mrr') 45.8769 13.0076 10.3082 34.548 48.4693 21.8219 17.2902 13.6215 10.2962 8.82371 13.0351 7.62129 4.6169 5.07511 23.9156 7.83441
(65, 'mrr') 45.8871 12.8002 10.2563 34.5614 48.7442 22.0247 17.1587 13.7247 10.1597 8.81356 13.0102 7.63786 4.55925 5.04153 23.9241 7.81248
(70, 'mrr') 45.915 12.9762 10.3503 34.6932 48.8386 21.9847 17.2467 13.6548 10.1373 8.75037 13.1279 7.60491 4.55571 5.04706 23.9774 7.8172
(75, 'mrr') 45.9074 12.9137 10.2342 34.7737 48.7164 22.0299 17.3165 13.663 10.1987 8.74679 13.062 7.66346 4.63074 5.10834 23.9726 7.84227
(80, 'mrr') 45.9094 12.9309 10.376 34.7534 48.8564 22.3067 17.2492 13.6232 10.1518 8.752 13.0546 7.64423 4.68432 5.12836 24.0174 7.85271
(85, 'mrr') 45.8954 12.9381 10.273 34.7352 48.873 22.3331 17.2259 13.5965 10.1396 8.73849 13.0252 7.67358 4.59914 5.08056 24.0011 7.8234
(90, 'mrr') 45.84 12.8973 10.3091 34.8627 49.0908 22.4607 17.3106 13.5644 10.1515 8.77689 13.1038 7.59491 4.66412 5.04967 24.0541 7.83788
(95, 'mrr') 45.8448 12.8836 10.3528 34.8965 49.0815 22.5072 17.3211 13.4828 10.119 8.78041 13.0549 7.62073 4.70466 5.098 24.0544 7.85175
(100, 'mrr') 45.8614 13.0062 10.4779 34.9732 49.0683 22.5373 17.3939 13.578 10.2351 8.69003 13.0802 7.68441 4.70692 5.01153 24.1257 7.83462

For FB15k dataset, a possible training trajectory could be

Validation set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 72.764 18.1599 14.3685 50.1483 63.4033 20.3927 21.7411 24.0656 17.3729 16.0573 14.3387 8.96579 5.63852 8.01369 33.6018 10.6028
(10, 'mrr') 73.2114 20.7276 16.2229 50.9103 64.7822 24.5783 24.0674 24.2767 18.4504 16.0651 14.6001 9.98227 6.18293 9.6944 35.2475 11.305
(15, 'mrr') 74.041 21.7555 16.8663 51.4346 65.7271 26.7591 25.3408 24.7065 18.9669 16.5432 14.7984 10.4507 6.67602 10.1124 36.1775 11.7161
(20, 'mrr') 74.4192 22.4445 17.3096 51.8621 66.3508 28.1335 26.2893 25.1039 19.3605 17.0578 15.3382 10.7319 6.7229 10.4774 36.8081 12.0657
(25, 'mrr') 74.6472 23.0718 17.8927 52.3799 66.7333 29.4432 27.0306 25.102 19.7324 17.0759 15.3593 11.0009 7.12063 10.7602 37.337 12.2634
(30, 'mrr') 74.6737 23.4607 17.8685 52.2627 66.8866 30.3678 27.282 25.3029 19.5946 17.0104 15.299 11.007 7.04523 10.8055 37.5222 12.2334
(35, 'mrr') 74.9351 23.8835 18.3777 52.3996 67.0009 30.8495 27.6875 25.4789 20.0324 16.8463 15.297 11.1668 7.17088 10.9909 37.8495 12.2944
(40, 'mrr') 75.1099 24.4404 18.5153 52.6634 67.2669 31.201 28.1146 25.4405 20.0116 17.1899 15.7448 11.5347 7.43448 11.1738 38.0848 12.6155
(45, 'mrr') 75.175 24.4533 18.7198 52.7764 67.4387 32.2673 28.5103 25.4906 20.0293 17.2964 15.8747 11.5726 7.28784 11.2956 38.3179 12.6654
(50, 'mrr') 75.2242 24.5531 18.645 53.0605 67.4554 32.5306 28.4636 25.503 20.0741 17.2653 15.9408 11.5878 7.41323 11.3201 38.3899 12.7055
(55, 'mrr') 75.7348 25.3315 19.4048 53.5306 68.1116 34.0854 29.3051 25.6027 20.3288 17.5561 16.1879 11.9783 7.51217 11.4066 39.0484 12.9282
(60, 'mrr') 75.861 25.5841 19.5346 53.7438 68.2576 34.5876 29.5732 25.6164 20.4787 17.6198 16.3073 12.0556 7.65011 11.4113 39.2486 13.0088
(65, 'mrr') 75.9349 25.7382 19.6538 53.6697 68.3454 34.8048 29.6301 25.6537 20.4627 17.6294 16.3302 12.139 7.6883 11.4152 39.3215 13.0404
(70, 'mrr') 75.9818 25.7803 19.7334 53.8455 68.6032 35.0239 29.7125 25.6243 20.5497 17.6473 16.4369 12.1637 7.76578 11.4374 39.4283 13.0902
(75, 'mrr') 76.0647 25.8406 19.8401 53.9193 68.5568 35.2081 29.7754 25.6366 20.5592 17.6685 16.3975 12.1659 7.8314 11.3745 39.489 13.0875
(80, 'mrr') 76.0764 25.815 19.8477 53.8205 68.5533 35.353 29.7676 25.6507 20.5044 17.6077 16.4779 12.226 7.8704 11.3885 39.4876 13.1141
(85, 'mrr') 76.1621 25.8629 19.9085 53.8178 68.594 35.352 29.8925 25.6998 20.6121 17.5867 16.5415 12.2125 7.92277 11.3823 39.5446 13.1292
(90, 'mrr') 76.1505 25.8591 19.941 53.8449 68.7232 35.3287 30.0221 25.7584 20.5822 17.6385 16.4569 12.2411 7.92856 11.4002 39.5789 13.1331
(95, 'mrr') 76.1381 25.9548 19.9634 53.8797 68.7455 35.6256 29.9943 25.7073 20.6724 17.622 16.6394 12.2738 7.97295 11.4501 39.6312 13.1916
(100, 'mrr') 76.2425 25.9843 20.0225 53.9112 68.7771 35.6986 30.0373 25.8397 20.8087 17.6471 16.7057 12.2231 7.98628 11.414 39.7024 13.1952

|Test set| 1p | 2p | 3p | 2i | 3i | pi | ip | 2u | up | 2in | 3in | inp | pin | pni | epfo mean | Neg mean |

1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 80.0214 26.1631 20.6792 63.1397 70.4824 26.7026 31.8146 35.0759 25.2013 26.7359 26.8873 10.0949 6.86781 11.2488 42.1422 16.3669
(10, 'mrr') 80.6118 29.9317 23.0558 64.1709 71.9207 32.2211 35.1749 34.2256 27.085 26.0799 27.0576 11.7576 7.76882 13.5857 44.2664 17.2499
(15, 'mrr') 81.6128 32.087 24.2246 64.7785 72.8758 35.1365 36.9434 35.6952 28.4065 27.7164 27.5466 12.3355 8.14011 14.713 45.7512 18.0903
(20, 'mrr') 82.1223 32.9022 24.7883 65.2956 73.2986 36.3822 37.6886 36.0638 28.7675 29.0385 27.9598 12.9415 8.56375 15.3342 46.3677 18.7676
(25, 'mrr') 82.4252 34.2613 25.7724 65.5153 73.8097 38.1548 38.6816 36.4394 29.2493 28.9039 28.0133 13.1606 8.90367 15.7501 47.1454 18.9463
(30, 'mrr') 82.6365 34.6101 25.5823 65.5032 74.0651 38.9272 39.0842 36.2058 29.4809 28.8655 28.0349 13.4159 8.92127 15.4652 47.3439 18.9406
(35, 'mrr') 82.8815 35.4994 26.4092 65.9638 74.3764 40.0052 39.5615 36.8258 29.8588 28.8328 27.9218 13.6509 9.0599 15.8999 47.9313 19.0731
(40, 'mrr') 83.0528 35.6612 26.4634 66.229 74.5047 40.617 39.6554 36.4271 29.9298 28.8021 28.2275 13.9195 9.4794 16.4576 48.06 19.3772
(45, 'mrr') 83.136 36.7407 26.5522 66.3764 74.7943 42.0012 41.0161 36.5704 30.1945 29.4151 28.6116 13.789 9.48379 16.541 48.598 19.5681
(50, 'mrr') 83.3121 36.643 26.7993 66.705 74.9823 42.057 40.0862 36.6947 30.2306 28.8638 28.5118 13.9804 9.23777 16.38 48.6122 19.3948
(55, 'mrr') 84.0474 38.0631 28.1092 67.6499 76.0039 44.2674 41.6678 36.5543 30.9187 29.0836 29.0125 14.499 9.72798 16.6574 49.698 19.7961
(60, 'mrr') 84.2658 38.3675 28.4186 67.8149 76.2868 44.6406 41.99 36.5638 31.2279 29.1775 29.1025 14.6062 9.85346 16.714 49.9529 19.8907
(65, 'mrr') 84.3822 38.5743 28.5948 67.9548 76.3456 44.9868 42.1976 36.7303 31.1506 29.1546 29.2271 14.7821 9.87208 16.7348 50.1019 19.9541
(70, 'mrr') 84.4594 38.726 28.5233 67.9601 76.3959 45.4252 42.4645 36.632 31.2578 29.2664 29.2771 14.7912 9.95378 16.6008 50.2049 19.9779
(75, 'mrr') 84.5264 38.7699 28.6037 68.0511 76.5039 45.4822 42.545 36.7839 31.2311 29.1764 29.2714 14.7585 9.94432 16.6274 50.2775 19.9556
(80, 'mrr') 84.6096 38.6981 28.5022 68.0766 76.4818 45.5297 42.3989 36.7637 31.2135 29.102 29.3295 14.8571 9.99067 16.6122 50.2527 19.9783
(85, 'mrr') 84.6776 38.6621 28.4035 68.0234 76.4476 45.7052 42.4011 36.6235 31.1659 29.134 29.289 14.8435 9.99 16.5764 50.2344 19.9666
(90, 'mrr') 84.7225 38.9814 28.4973 68.0495 76.5384 45.9682 42.7049 36.6288 31.2611 29.2248 29.3559 14.8518 9.99594 16.5391 50.3725 19.9935
(95, 'mrr') 84.7482 39.0329 28.4649 68.1035 76.3899 46.021 42.7329 36.7371 31.1983 29.1817 29.2837 14.8702 10.0342 16.5788 50.381 19.9897
(100, 'mrr') 84.8463 39.3027 28.729 68.1155 76.6316 46.1163 42.9275 36.8473 31.4271 29.1328 29.2369 14.852 10.0518 16.6147 50.5493 19.9776

For NELL dataset, a possible training trajectory could be

Validation set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 51.9866 16.4089 13.3687 31.5865 43.7179 18.1878 19.2869 14.4951 11.9546 3.18955 9.70108 10.7595 3.79393 4.22667 24.5548 6.33414
(10, 'mrr') 58.0504 18.9486 16.2135 35.7542 49.4412 22.1829 22.2309 15.9189 13.0717 8.04612 10.5261 12.1496 4.10794 4.4874 27.9791 7.86342
(15, 'mrr') 58.1901 19.5215 16.9156 36.2875 50.5697 23.5127 23.0488 15.886 13.1331 8.38497 10.8321 12.4829 3.99677 4.58691 28.5628 8.05675
(20, 'mrr') 58.1312 19.7626 17.1061 36.6888 51.3984 24.2156 23.8144 15.744 13.2207 8.42841 10.7424 12.3675 4.01913 4.49603 28.898 8.0107
(25, 'mrr') 58.2135 19.3948 16.7125 37.0836 51.5466 23.8573 23.4707 15.7727 13.1974 8.13385 10.7643 12.3361 4.0185 4.51245 28.8055 7.95303
(30, 'mrr') 58.3422 19.9796 16.9005 37.3361 52.2049 25.0439 24.258 15.8504 13.529 8.25035 11.0144 12.8912 3.94366 4.48501 29.2716 8.11691
(35, 'mrr') 58.3317 19.8246 17.0437 37.0317 52.0876 24.9918 24.1882 15.7776 13.2863 8.11579 11.0137 12.5773 3.95141 4.43461 29.1737 8.01855
(40, 'mrr') 58.4374 20.0042 17.2376 37.5069 52.4436 25.5753 24.3363 15.8403 13.2857 8.04121 11.0236 12.65 4.00914 4.48272 29.4075 8.04135
(45, 'mrr') 58.3851 20.0087 17.1159 37.2356 52.586 25.8343 24.3349 15.7666 13.269 7.99957 10.8519 12.4951 3.96799 4.50756 29.3929 7.96441
(50, 'mrr') 58.4344 20.066 17.2659 37.5026 52.583 25.8256 24.5214 15.8159 13.3767 8.00606 10.9744 12.7093 3.98905 4.44717 29.4879 8.02518
(55, 'mrr') 58.5214 20.3003 17.4971 37.7731 53.2195 26.6214 24.9088 15.8859 13.3979 7.99773 11.0119 12.8666 4.00092 4.609 29.7917 8.09724
(60, 'mrr') 58.5588 20.2791 17.4278 37.794 53.4732 26.7336 24.8551 15.9778 13.3923 8.05524 10.9982 12.7967 4.03883 4.58863 29.8324 8.09553
(65, 'mrr') 58.5393 20.3012 17.3731 37.7334 53.5801 26.7743 25.0042 15.9129 13.4806 7.98234 10.9909 12.871 4.0504 4.57675 29.8555 8.09429
(70, 'mrr') 58.5464 20.3899 17.636 37.8725 53.5272 26.8348 25.017 15.8722 13.3682 8.00063 10.9787 12.8349 4.09206 4.54549 29.896 8.09037
(75, 'mrr') 58.5244 20.3452 17.5419 37.9096 53.6987 26.9608 24.9611 15.8747 13.3377 8.0151 10.9958 12.8542 4.07362 4.58094 29.906 8.10395
(80, 'mrr') 58.4666 20.3634 17.4013 37.864 53.7288 26.872 25.0944 15.8941 13.4476 8.00644 11.0202 12.889 4.03659 4.56708 29.9036 8.10385
(85, 'mrr') 58.5246 20.3625 17.4951 37.8034 53.7097 26.8793 25.0511 15.7963 13.332 8.00327 10.9794 12.8392 4.04728 4.55166 29.8838 8.08415
(90, 'mrr') 58.5385 20.2787 17.4881 37.8201 53.708 26.9801 25.1586 15.8513 13.3464 8.01791 10.9962 12.8424 4.04243 4.56984 29.9078 8.09376
(95, 'mrr') 58.5262 20.3174 17.407 37.9197 53.704 27.007 24.9786 15.8147 13.3705 8.03061 11.0391 12.7607 4.0563 4.51442 29.8939 8.08023
(100, 'mrr') 58.4876 20.3739 17.5426 37.8685 53.7853 27.0437 25.0912 15.8391 13.3677 8.03276 11.0813 12.8224 4.02579 4.43816 29.9333 8.08009
Test set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 53.3178 17.4509 13.921 34.3557 42.5244 18.9655 19.9409 15.5624 14.1614 3.09049 9.33451 10.4117 3.55259 3.61193 25.5778 6.00025
(10, 'mrr') 60.0538 20.7879 16.6129 38.033 47.0486 24.1357 23.079 17.3772 15.8912 8.24614 10.7019 11.5071 3.87626 4.72569 29.2244 7.81142
(15, 'mrr') 60.0531 21.0174 17.3599 38.6041 48.159 25.4717 23.4817 17.3495 15.6388 8.76846 10.9286 11.972 3.92587 4.70564 29.6817 8.06013
(20, 'mrr') 60.0223 21.3595 17.3279 38.7731 48.7933 25.9714 24.0966 17.2853 15.8172 8.98501 10.8567 11.8679 4.01083 4.63997 29.9385 8.07209
(25, 'mrr') 60.1999 21.4644 17.2901 38.9674 49.2416 26.0032 24.1828 17.2671 15.7017 8.6811 11.0215 11.7445 3.92222 4.81752 30.0353 8.03736
(30, 'mrr') 60.39 21.9168 17.0032 39.2801 49.9484 27.3184 24.6839 17.1173 15.9443 8.57809 10.8312 12.1397 3.83815 4.77605 30.4003 8.03264
(35, 'mrr') 60.3503 21.7325 17.2967 39.1128 49.7495 27.0008 24.6514 17.0884 15.861 8.59063 10.7803 12.1338 3.84769 4.80637 30.3159 8.03174
(40, 'mrr') 60.4626 22.0204 17.3283 39.5281 50.6019 27.4767 24.6566 17.0303 15.7524 8.46947 10.864 12.333 3.92515 4.89782 30.5397 8.09789
(45, 'mrr') 60.3293 22.1564 17.5824 39.4073 50.1337 27.8943 25.0868 16.9976 15.9754 8.70057 10.8894 12.1503 3.98509 4.82957 30.6181 8.111
(50, 'mrr') 60.5006 21.9947 17.398 39.4997 50.1538 27.7933 24.8773 16.97 15.7931 8.42343 10.8244 12.2081 3.92818 4.73758 30.5534 8.02432
(55, 'mrr') 60.5633 22.3015 17.636 39.9431 50.9257 28.5783 25.3599 17.0022 15.833 8.50931 10.8985 12.411 4.0115 4.83301 30.9048 8.13267
(60, 'mrr') 60.5973 22.2343 17.6546 39.9555 50.9812 28.7155 25.2521 17.0157 15.7824 8.48907 10.9854 12.2893 3.96769 4.85918 30.9098 8.11811
(65, 'mrr') 60.5589 22.3278 17.6482 40.0772 51.1738 28.8743 25.255 16.9732 15.8177 8.4574 10.9471 12.3755 3.99949 4.80515 30.9673 8.11693
(70, 'mrr') 60.5047 22.2669 17.734 40.0701 51.2012 28.9174 25.2583 16.8859 15.7568 8.42383 11.0485 12.3892 4.00967 4.80891 30.955 8.13604
(75, 'mrr') 60.5458 22.2862 17.6367 40.2007 51.2726 28.9536 25.2717 16.8917 15.7468 8.43721 10.9711 12.4433 3.97777 4.82057 30.9784 8.13
(80, 'mrr') 60.5429 22.2779 17.5793 40.0852 51.1626 28.9895 25.3941 16.8825 15.7659 8.48395 10.9586 12.4836 3.98999 4.7884 30.9644 8.14091
(85, 'mrr') 60.5592 22.352 17.6074 40.2472 51.2478 28.9383 25.3371 16.908 15.7983 8.41977 10.9511 12.4619 4.01124 4.80731 30.9995 8.13026
(90, 'mrr') 60.5871 22.3782 17.4865 40.2134 51.3309 29.0312 25.318 16.9407 15.744 8.39682 10.9253 12.4119 4.03235 4.80307 31.0033 8.11389
(95, 'mrr') 60.5521 22.3003 17.5146 40.4396 51.5141 28.99 25.3643 16.8287 15.7321 8.39906 11.0132 12.4137 4.01778 4.77633 31.0262 8.12402
(100, 'mrr') 60.5551 22.4637 17.6964 40.2327 51.3726 29.0957 25.449 16.8115 15.7569 8.42328 10.9956 12.3207 4.04476 4.85645 31.0482 8.12816
Results from versions before e83a95a are obtained from the training process where both the KG embedding and the MLP is trainable, which is an accidental mistake and NOT what proposed in the paper. The results with trainable KGE is be weaker than those with freezed KGE which is proposed in the paper. Click to see the sample output trajectory of trainable KGE (results of weak implementation of LMPNN)

(depreciated) LMPNN trajectory without freezed KGE (weak implementation)

For FB15k-237, a possible output trajectory could be like

Validation Set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 43.3374 9.3963 7.36107 28.8171 43.2848 12.7592 11.338 10.9825 8.06806 4.89454 9.03825 6.84014 3.57948 3.43009 19.4827 5.5565
(10, 'mrr') 43.5847 10.2061 8.38958 30.1158 44.8668 13.7348 12.4519 11.1331 8.40302 5.50171 9.92851 7.19045 3.68003 3.68664 20.3207 5.99747
(15, 'mrr') 43.1136 9.93098 8.41377 30.7293 45.7028 14.1373 12.4096 11.17 8.48811 5.90808 9.97395 7.14056 3.78556 3.88563 20.455 6.13876
(20, 'mrr') 43.6212 10.5731 8.92229 30.8057 45.9869 15.2658 13.1256 11.0792 8.84129 5.74387 10.4465 7.32727 3.87902 3.76653 20.9135 6.23263
(25, 'mrr') 43.5993 10.6286 9.14751 30.7887 46.2739 16.2113 13.8994 11.2555 8.81353 5.98244 10.6118 7.51721 3.7699 3.55735 21.1798 6.28774
(30, 'mrr') 43.6251 10.8747 9.03938 30.9388 46.5563 16.4211 13.543 11.2324 9.15598 5.83169 10.5604 7.63332 3.9878 3.64097 21.2652 6.33083
(35, 'mrr') 43.4778 11.0762 9.3528 31.3629 47.202 17.1742 13.701 11.2211 9.16062 5.97082 10.7614 7.62482 3.90409 3.7512 21.5254 6.40247
(40, 'mrr') 43.4053 11.0612 9.56793 31.3726 47.2989 16.9117 13.6576 10.8811 9.21064 5.86364 10.6366 7.91589 3.90675 3.52537 21.4852 6.36965
(45, 'mrr') 43.2611 11.0465 9.57849 31.2661 47.44 17.705 13.9036 11.0584 9.11347 5.98817 10.6902 7.86165 3.93147 3.75511 21.5969 6.44531
(50, 'mrr') 42.9117 11.4134 9.45842 31.4618 47.5948 17.8229 13.9 11.0091 9.40493 5.89662 10.722 7.65385 4.07289 3.60841 21.6641 6.39074
(55, 'mrr') 43.2503 11.6637 9.83592 31.7507 48.095 19.3408 14.4881 11.0594 9.40362 6.04338 10.8817 8.20831 4.18475 3.58321 22.0986 6.58026
(60, 'mrr') 43.3788 11.7293 10.0087 31.8454 48.1824 19.6966 14.5867 11.0317 9.52452 6.06797 10.8322 8.18048 4.23077 3.65369 22.2205 6.59303
(65, 'mrr') 43.265 11.6121 10.0383 32.0212 48.3501 19.8917 14.6462 11.0356 9.44041 6.03938 10.9391 8.08408 4.3106 3.65148 22.2556 6.60492
(70, 'mrr') 43.2776 11.6904 9.98802 32.0637 48.5154 20.0188 14.7324 11.0485 9.54078 5.98941 11.0414 8.21996 4.22696 3.7204 22.3195 6.63963
(75, 'mrr') 43.3091 11.7581 10.0377 32.1079 48.6767 20.1973 14.8317 11.0657 9.52549 6.03904 11.0798 8.28582 4.34577 3.63607 22.39 6.6773
(80, 'mrr') 43.3248 11.7603 9.98525 32.2027 48.5901 20.2498 14.8366 11.0489 9.5386 5.98985 11.0667 8.26597 4.35415 3.66858 22.393 6.66906
(85, 'mrr') 43.2465 11.7877 10.0583 32.1016 48.7363 20.4814 14.9262 10.9962 9.51707 6.02979 11.2297 8.3184 4.39278 3.72155 22.4279 6.73846
(90, 'mrr') 43.376 11.6823 10.0461 32.241 48.6458 20.4281 14.8565 11.0369 9.5371 6.03664 11.1348 8.34831 4.32066 3.62985 22.4278 6.69404
(95, 'mrr') 43.291 11.7701 9.97534 32.3046 48.8179 20.5445 14.8694 11.1149 9.53749 6.02634 11.2056 8.25653 4.34473 3.6933 22.4695 6.70531
(100, 'mrr') 43.2896 11.8108 9.95793 32.3719 48.9575 20.4634 14.8929 11.0351 9.56184 6.06656 11.1147 8.34275 4.35363 3.69552 22.4823 6.71463
Test set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 44.9465 10.4843 8.6229 31.7221 44.4341 14.2661 14.1401 13.6675 9.80541 6.86755 11.2436 6.54754 3.84052 4.55932 21.3432 6.61171
(10, 'mrr') 45.1145 11.2116 9.19404 32.7899 46.5547 15.7508 15.1701 13.5092 9.95767 7.41947 11.6497 6.58225 3.93812 4.52801 22.1392 6.8235
(15, 'mrr') 44.7565 11.6183 9.65737 33.0242 46.9324 16.0322 15.6126 13.9755 10.2088 7.83579 12.0131 6.88489 4.00784 4.6131 22.4242 7.07094
(20, 'mrr') 45.3177 12.0233 9.57721 33.2067 47.472 17.4125 16.2467 13.6488 10.4001 8.05987 12.2843 7.14386 4.14163 4.45509 22.8117 7.21695
(25, 'mrr') 45.1459 12.1309 9.8724 33.4177 47.7942 17.7643 16.4459 13.4371 10.4981 7.80966 12.3091 7.22651 4.23409 4.35564 22.9452 7.187
(30, 'mrr') 45.2453 12.516 10.2946 33.8225 47.6455 18.1683 16.3805 13.5795 10.5048 7.82047 12.0024 7.65289 4.37242 4.55625 23.1285 7.28089
(35, 'mrr') 44.9832 12.6714 10.3581 33.9382 48.1022 19.0562 16.7391 13.4728 10.3644 7.80266 12.4172 7.50164 4.27899 4.18954 23.2984 7.23801
(40, 'mrr') 45.0497 12.9713 10.5534 33.9162 48.2494 19.4182 17.0646 13.3876 10.7647 7.76955 12.0863 7.76417 4.58433 4.18861 23.4861 7.27859
(45, 'mrr') 44.8709 12.7598 10.6207 34.0123 48.4076 20.1168 16.7293 13.4826 10.7217 7.7207 12.2151 7.79834 4.38069 4.2472 23.5246 7.27242
(50, 'mrr') 44.4331 12.9428 10.3695 33.9687 48.3719 20.2204 17.002 13.2265 10.7709 7.51472 12.3501 7.86527 4.52629 4.30918 23.4784 7.31311
(55, 'mrr') 44.8656 13.2094 10.7748 34.5006 48.9649 21.6445 17.5028 13.3423 10.9163 7.63899 12.6289 8.30786 4.61518 4.30892 23.969 7.49997
(60, 'mrr') 44.9011 13.3243 10.8495 34.6471 49.0874 22.1415 17.4582 13.3771 10.8835 7.62007 12.5587 8.2811 4.68471 4.34016 24.0744 7.49695
(65, 'mrr') 44.882 13.4375 10.8931 34.7594 49.126 22.3102 17.6696 13.408 10.9255 7.63332 12.6301 8.24105 4.70683 4.2976 24.1568 7.50179
(70, 'mrr') 44.8828 13.651 11.0718 34.7735 49.2482 22.703 17.7788 13.4791 11.0766 7.68911 12.7907 8.35583 4.76243 4.40152 24.2961 7.59992
(75, 'mrr') 44.9174 13.555 11.012 34.906 49.3564 22.7873 17.6738 13.396 11.0057 7.63671 12.7132 8.41619 4.61401 4.29353 24.2899 7.53473
(80, 'mrr') 44.9689 13.694 11.0733 34.9882 49.5035 22.9782 17.7274 13.3799 11.0732 7.6816 12.7935 8.37103 4.75101 4.35969 24.3763 7.59136
(85, 'mrr') 44.8746 13.5736 11.0516 35.0654 49.612 23.1731 17.7409 13.4019 11.0125 7.63303 12.8699 8.32122 4.76494 4.31249 24.3895 7.58032
(90, 'mrr') 44.963 13.6078 11.0434 35.0212 49.539 22.9025 17.6933 13.3906 10.9815 7.68771 12.8578 8.27871 4.70053 4.26779 24.3491 7.5585
(95, 'mrr') 44.8846 13.665 11.0756 35.1892 49.8396 23.0492 17.7944 13.3872 11.0303 7.66467 12.8382 8.39552 4.71225 4.3251 24.435 7.58714
(100, 'mrr') 44.9048 13.6668 11.1112 35.2322 49.6774 22.859 17.8386 13.3444 11.0184 7.70513 12.857 8.45513 4.74044 4.26359 24.4059 7.60425

For FB15k dataset, a possible training trajectory could be

Validation set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 72.5264 18.9707 15.167 50.393 64.2903 21.3955 22.4212 23.7883 17.7087 15.89 14.4604 9.5854 6.1095 8.80865 34.0735 10.9708
(10, 'mrr') 73.4743 21.3927 16.8809 51.2315 65.4991 25.2254 25.0429 24.3405 18.9479 16.2819 14.5931 10.1545 7.01239 10.0011 35.7817 11.6086
(15, 'mrr') 74.0354 22.2156 17.726 51.3907 65.5969 27.1767 25.7572 24.2847 19.393 16.4947 14.6412 10.7316 7.59753 10.4718 36.3974 11.9873
(20, 'mrr') 74.1509 23.2951 18.3861 52.2457 66.8295 28.6767 26.7481 24.5087 20.1395 16.8086 15.0185 10.912 8.17509 10.3257 37.22 12.248
(25, 'mrr') 74.3886 23.6987 18.7169 52.2513 66.6926 29.6802 27.3806 24.6967 20.545 16.4154 15.0412 11.3016 8.45187 10.6056 37.5612 12.3631
(30, 'mrr') 74.5874 24.1382 18.9701 52.1197 67.0708 31.0522 27.5473 24.7312 20.6051 16.5712 15.397 11.4983 8.42595 10.7639 37.8691 12.5313
(35, 'mrr') 74.6497 24.6292 19.136 52.0571 66.9698 32.1509 28.2427 24.8105 20.8558 16.3211 15.2092 11.6967 8.96677 10.7479 38.1669 12.5883
(40, 'mrr') 74.8573 24.9687 19.3029 52.2527 67.0853 33.2529 28.3276 24.8803 21.0188 16.4776 15.1542 11.8346 9.1182 11.0053 38.4385 12.718
(45, 'mrr') 74.8077 25.2605 19.5295 52.5364 67.5456 33.4174 28.8868 24.7764 20.9093 16.5528 15.2427 11.7762 9.26389 11.0442 38.63 12.776
(50, 'mrr') 74.7147 25.3948 19.6865 52.1841 66.9479 33.9548 28.9421 24.589 21.1452 16.2819 15.3491 11.9337 9.17793 10.933 38.6177 12.7351
(55, 'mrr') 75.2392 26.0655 20.4374 52.7575 68.0126 35.7402 29.763 24.9457 21.5412 16.7172 15.8523 12.292 9.61107 11.1913 39.3892 13.1328
(60, 'mrr') 75.338 26.2711 20.4958 53.005 68.2882 36.3051 29.9832 24.9604 21.5683 16.7698 16.0298 12.4912 9.97094 11.293 39.5795 13.311
(65, 'mrr') 75.4915 26.3433 20.7189 53.1031 68.4577 36.5135 30.125 24.9532 21.6608 16.7159 16.1205 12.445 10.1391 11.28 39.7075 13.3401
(70, 'mrr') 75.6082 26.5294 20.858 53.2512 68.4007 36.8932 30.2989 25.08 21.7732 16.8102 16.2661 12.5809 10.1644 11.3312 39.8548 13.4306
(75, 'mrr') 75.6264 26.779 20.9627 53.4878 68.5542 36.972 30.3147 25.0093 21.8702 16.8719 16.4148 12.5987 10.1621 11.4264 39.9529 13.4948
(80, 'mrr') 75.6596 26.7891 20.9883 53.5604 68.6648 37.0346 30.4991 25.0008 21.9547 16.9228 16.4881 12.6153 10.5 11.3379 40.0168 13.5728
(85, 'mrr') 75.7871 26.9072 21.0725 53.4992 68.6661 37.3166 30.5706 25.0393 21.9667 16.9707 16.5363 12.6744 10.5678 11.3997 40.0917 13.6298
(90, 'mrr') 75.7685 26.9004 21.1033 53.5543 68.6671 37.4762 30.6839 25.0264 21.9165 16.9988 16.5336 12.6405 10.6902 11.2698 40.1219 13.6266
(95, 'mrr') 75.8367 27.112 21.1882 53.6209 68.8099 37.4012 30.7478 25.1237 22.027 17.024 16.577 12.7249 10.588 11.4293 40.2075 13.6687
(100, 'mrr') 75.821 26.9972 21.1739 53.5756 68.8016 37.3668 30.7325 25.0958 21.967 16.9533 16.6115 12.6655 10.5621 11.4197 40.1702 13.6424
Test set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 79.6252 27.0495 22.0475 63.0068 71.0271 26.8145 31.631 33.8454 26.428 25.348 26.4778 11.1268 7.37001 12.2931 42.3861 16.5231
(10, 'mrr') 80.8636 30.8524 24.7873 64.5688 72.6059 32.4962 34.8987 34.8972 28.6328 26.4557 26.8073 12.2922 8.45513 14.3868 44.9559 17.6794
(15, 'mrr') 81.5185 32.2857 25.5074 64.8339 72.9823 34.3122 36.1444 34.3031 29.9918 26.6691 27.0838 13.1753 9.03323 14.2974 45.7644 18.0518
(20, 'mrr') 81.9271 34.1768 26.5718 65.453 73.8918 36.7933 37.8788 34.451 31.1975 26.7253 26.6824 13.0502 9.25748 14.8176 46.9268 18.1066
(25, 'mrr') 82.1761 35.3178 27.2472 65.4745 74.0307 38.0896 38.8817 34.9898 31.5301 26.2991 26.4839 13.6902 9.53723 14.8901 47.5264 18.1801
(30, 'mrr') 82.3555 35.6731 27.6514 65.0369 73.8001 40.1415 38.939 34.6648 31.4103 25.8967 26.2618 14.1097 9.66685 14.7401 47.7414 18.135
(35, 'mrr') 82.5593 36.5284 27.9251 65.3617 74.0039 40.9399 39.6183 34.3864 31.5693 25.4552 26.0457 14.1313 9.76101 14.6804 48.0992 18.0147
(40, 'mrr') 82.9059 37.9752 28.8003 65.5398 74.1289 42.1209 40.5559 34.4631 32.6308 25.4917 25.8993 14.6716 10.0891 14.8624 48.7912 18.2028
(45, 'mrr') 82.8705 37.8872 29.0582 65.5965 74.1744 42.4366 40.5069 34.3821 32.6076 25.2364 25.7732 14.508 10.2368 14.9662 48.8356 18.1441
(50, 'mrr') 82.9409 38.4537 29.3314 65.409 74.1168 43.6435 40.9284 34.5042 32.8697 24.8741 25.5653 14.8343 10.3195 15.0117 49.1331 18.1209
(55, 'mrr') 83.6995 40.1008 30.5288 66.6587 75.1619 45.9406 42.3054 34.6511 33.7162 25.3503 26.1966 15.3033 10.573 15.2319 50.307 18.531
(60, 'mrr') 83.8872 40.7408 30.9499 67.0044 75.4608 46.6269 42.8369 34.5836 34.0223 25.475 26.432 15.5052 10.7476 15.2947 50.6792 18.6909
(65, 'mrr') 84.0518 40.849 30.9894 67.2783 75.6766 46.6758 42.9262 34.6531 34.0006 25.4424 26.5234 15.5519 10.7439 15.2883 50.789 18.71
(70, 'mrr') 84.2106 41.3092 31.257 67.3251 75.8018 47.0875 43.2535 34.654 34.1175 25.3795 26.5757 15.6222 10.8237 15.2704 51.0018 18.7343
(75, 'mrr') 84.2636 41.5525 31.5103 67.5114 75.7793 47.4208 43.6347 34.6501 34.2698 25.5468 26.6772 15.6955 10.8787 15.3086 51.1769 18.8214
(80, 'mrr') 84.3512 41.571 31.4847 67.5161 75.975 47.2654 43.582 34.6848 34.3082 25.5773 26.7497 15.7046 10.9515 15.3087 51.1932 18.8584
(85, 'mrr') 84.4401 41.7911 31.6551 67.6595 76.0263 47.5891 43.8328 34.6937 34.4209 25.6277 26.8003 15.8049 10.9324 15.4512 51.3454 18.9233
(90, 'mrr') 84.4638 41.8142 31.7417 67.8218 76.1267 47.763 43.5757 34.6339 34.3527 25.4679 26.857 15.781 11.0279 15.3128 51.3659 18.8893
(95, 'mrr') 84.6069 41.7902 31.7677 67.8683 76.2519 47.8811 43.8575 34.5829 34.5235 25.4988 26.8536 15.815 11.0273 15.3696 51.4589 18.9129
(100, 'mrr') 84.5962 41.8732 31.7129 67.8121 76.2344 47.7903 43.8667 34.6276 34.5437 25.6159 26.8515 15.8538 11.0693 15.3367 51.4508 18.9455

For NELL dataset, a possible training trajectory could be

Validation set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 58.1204 17.1394 14.8171 35.1191 48.1434 18.216 19.8213 15.6274 12.3501 7.47669 10.6017 11.8062 4.0412 4.38074 26.5949 7.66129
(10, 'mrr') 57.9101 17.9723 15.7368 35.6915 49.0143 19.5558 20.8033 15.7196 12.6459 8.01705 10.6385 12.3644 4.12027 4.34362 27.2277 7.89678
(15, 'mrr') 57.9716 19.0338 16.2968 35.9256 49.8854 22.1367 21.8386 15.4901 12.8566 7.80956 10.5861 12.4902 4.04185 4.31369 27.9373 7.84828
(20, 'mrr') 57.4168 18.7561 16.7115 35.977 49.8357 20.4362 21.6066 15.3487 12.7226 7.66178 10.4122 12.2952 4.02626 4.42086 27.6457 7.76325
(25, 'mrr') 57.0351 19.1695 16.6744 36.701 51.1331 23.5681 22.4403 15.3076 12.8107 7.91373 10.5872 12.7712 4.07904 4.35931 28.3155 7.94209
(30, 'mrr') 57.1963 19.2891 16.9377 36.7586 51.0399 23.756 22.522 15.2679 12.7877 7.5976 10.5313 12.6761 4.02105 4.47263 28.395 7.85975
(35, 'mrr') 56.9236 19.3807 17.0099 36.9825 51.3761 23.6143 22.3366 15.2227 13.0573 7.32792 10.6499 12.4046 4.1504 4.44791 28.4337 7.79616
(40, 'mrr') 56.8403 19.4833 16.8577 36.5935 51.3677 23.9328 22.81 15.1275 13.0253 7.45651 10.7676 12.6684 4.27722 4.56407 28.4487 7.94677
(45, 'mrr') 56.5277 19.4434 16.9731 36.6514 50.8131 23.8585 22.523 14.9389 13.0073 7.48076 10.4497 12.7325 4.09825 4.52448 28.304 7.85714
(50, 'mrr') 56.5823 19.3082 17.0722 36.9401 51.1826 24.1982 22.5559 14.8032 12.8865 7.30333 10.58 12.8101 4.07694 4.2912 28.3921 7.81231
(55, 'mrr') 56.7491 19.6617 17.3232 37.4124 51.9701 25.0041 23.0953 14.9818 12.9978 7.29917 10.6776 13.0981 4.1533 4.38353 28.7995 7.92233
(60, 'mrr') 56.747 19.7013 17.3851 37.4219 51.9863 25.0253 23.2136 15.0225 13.0185 7.31288 10.7236 13.1495 4.09421 4.39861 28.8357 7.93576
(65, 'mrr') 56.7386 19.7831 17.3879 37.5423 51.9313 25.2353 23.2883 14.9899 13.0879 7.33864 10.685 13.1793 4.1111 4.405 28.8872 7.9438
(70, 'mrr') 56.7031 19.809 17.4155 37.5206 52.1556 25.4837 23.2963 15.0402 13.174 7.32878 10.7495 13.2333 4.09398 4.40422 28.9553 7.96194
(75, 'mrr') 56.6914 19.7878 17.407 37.4702 52.2481 25.3543 23.3908 15.0409 13.1152 7.31722 10.7442 13.1681 4.13749 4.43393 28.9451 7.96018
(80, 'mrr') 56.7012 19.8272 17.4449 37.5907 52.2383 25.4022 23.4482 14.989 13.0254 7.35492 10.777 13.1616 4.14352 4.39464 28.963 7.96634
(85, 'mrr') 56.7686 19.8192 17.5098 37.4502 52.1744 25.5558 23.3911 15.0243 13.064 7.28512 10.8985 13.197 4.1342 4.45095 28.973 7.99316
(90, 'mrr') 56.7139 19.8626 17.4418 37.6441 52.174 25.4519 23.5212 14.9292 13.0997 7.30827 10.9621 13.1915 4.17801 4.43536 28.982 8.01505
(95, 'mrr') 56.6114 19.8485 17.6015 37.5606 52.3552 25.4303 23.4222 14.9639 13.0974 7.3714 10.9159 13.2139 4.17094 4.42943 28.9879 8.02032
(100, 'mrr') 56.5304 19.837 17.5056 37.557 52.61 25.68 23.4784 14.9258 13.0932 7.39956 10.8602 13.1604 4.17135 4.40587 29.0242 7.99949
Test set 1p 2p 3p 2i 3i pi ip 2u up 2in 3in inp pin pni epfo mean Neg mean
(5, 'mrr') 59.9925 18.7431 15.3488 37.131 45.7525 19.6135 20.6219 17.2678 14.6512 7.65617 10.6113 11.183 3.77651 4.2455 27.6802 7.4945
(10, 'mrr') 59.3839 19.619 15.7485 37.7121 46.9139 21.1344 21.3127 16.9185 14.8788 8.27667 10.6757 11.5308 3.65816 4.32677 28.1802 7.69362
(15, 'mrr') 59.515 20.5792 16.5123 38.2234 47.3892 23.91 22.5951 16.8399 15.4843 8.11972 10.6834 12.132 3.79222 4.60354 29.0054 7.86619
(20, 'mrr') 59.0143 20.6289 17.0332 38.2814 47.7713 21.9448 22.3699 16.8168 15.3931 7.85623 10.8544 11.9253 3.80776 3.9679 28.806 7.68231
(25, 'mrr') 58.7151 21.418 17.1238 38.6686 48.7802 24.9183 22.9127 16.5135 15.4847 8.29464 10.7124 12.1862 3.90752 4.75549 29.3927 7.97125
(30, 'mrr') 58.7759 21.42 17.1072 38.729 48.6958 24.9884 22.9775 16.5603 15.5697 7.92933 10.7024 12.1284 4.05308 4.57014 29.4249 7.87666
(35, 'mrr') 58.5786 21.377 17.0758 38.9681 48.9368 25.2203 22.4404 16.4391 15.3977 7.92863 10.5248 12.2395 3.94388 4.69192 29.3815 7.86574
(40, 'mrr') 58.3774 21.4076 17.0121 39.0265 49.08 25.3503 23.0431 16.1658 15.5395 7.9038 10.4779 12.3383 3.98636 4.66744 29.4447 7.87476
(45, 'mrr') 58.0577 21.2919 17.2616 38.8111 48.8938 25.5967 23.1196 16.1437 15.38 7.93512 10.43 12.1186 3.95103 4.61853 29.3951 7.81065
(50, 'mrr') 58.2683 21.2357 17.0527 39.0259 48.6985 25.8154 22.7295 16.2029 15.108 7.81959 10.5743 11.8353 3.9347 4.52063 29.3486 7.7369
(55, 'mrr') 58.4405 21.8109 17.518 39.4844 49.5178 26.5499 23.3775 16.2771 15.5987 7.91742 10.5566 12.1564 3.93595 4.59566 29.8417 7.8324
(60, 'mrr') 58.4016 21.886 17.6116 39.6262 49.5139 26.4611 23.2744 16.2556 15.4958 7.90128 10.5776 12.3606 3.93631 4.58478 29.8362 7.87211
(65, 'mrr') 58.3851 21.8665 17.5549 39.5868 49.6965 26.753 23.4442 16.2629 15.5942 7.90126 10.5395 12.3257 4.00136 4.59561 29.9049 7.87268
(70, 'mrr') 58.3645 21.8921 17.6507 39.5146 49.9237 26.8391 23.553 16.2191 15.5815 7.85621 10.5537 12.3672 3.96091 4.63597 29.9487 7.87479
(75, 'mrr') 58.3917 21.9669 17.7499 39.7617 49.9059 26.797 23.6986 16.1471 15.7068 7.90412 10.5253 12.3506 3.97828 4.63258 30.014 7.87817
(80, 'mrr') 58.3979 22.1004 17.8992 39.8158 49.8562 27.1052 23.7746 16.2492 15.7949 7.94933 10.5536 12.3678 3.97498 4.61852 30.1104 7.89284
(85, 'mrr') 58.3532 22.0636 17.7837 39.9643 50.0419 27.1929 23.7004 16.2874 15.6986 7.92375 10.549 12.439 3.98941 4.65393 30.1207 7.911
(90, 'mrr') 58.3807 22.1138 17.7913 39.807 50.0646 27.2867 23.715 16.2238 15.7525 7.8916 10.5542 12.402 4.04165 4.60234 30.1262 7.89835
(95, 'mrr') 58.3548 21.9695 17.8834 39.8838 50.0381 27.0707 23.5618 16.1987 15.5789 7.87723 10.5658 12.3075 4.06027 4.62175 30.06 7.88651
(100, 'mrr') 58.2411 22.1544 17.8277 39.9782 50.2438 27.1149 23.6624 16.2382 15.6717 7.91782 10.6511 12.4242 3.99122 4.63306 30.1258 7.92348

Citing this paper

@inproceedings{LMPNN,
  author    = {Zihao Wang and
               Yangqiu Song and
               Ginny Y. Wong and
               Simon See},
  title     = {Logical Message Passing Networks with One-hop Inference on Atomic Formulas},
  booktitle = {The Eleventh International Conference on Learning Representations, {ICLR} 2023, Kigali Rwanda, May 1-5, 2023},
  publisher = {OpenReview.net},
  year      = {2023},
  url       = {https://openreview.net/forum?id=SoyOsp7i_l},
}

About

Logical Message Passing Networks with One-hop Inference in Atomic Formulas (ICLR 2023)

Topics

pytorch knowledge-graph question-answering query-answering logic-query knowledge-graph-reasoning

Resources

Readme
Activity
Custom properties

Stars

15 stars

Watchers

3 watching

Forks

3 forks
Report repository

Contributors 2

Languages