Thread by @aarzchan

🚨 Our #ICLR2021 paper shows that KG-augmented models are surprisingly robust to KG perturbation! 🧐

arXiv: arxiv.org/abs/2010.12872
Code: github.com/INK-USC/deceiv…

To learn more, come find us at Poster Session 9 (May 5, 5-7PM PDT): iclr.cc/virtual/2021/p….

🧵[1/n]

KGs have helped neural models perform better on knowledge-intensive tasks and even “explain” their predictions, but are KG-augmented models really using KGs in a way that makes sense to humans?

[2/n]

We primarily investigate this question by measuring how the performance of KG-augmented models changes when the KG’s semantics and/or structure are perturbed, such that the KG becomes less human-comprehensible.

[3/n]

If the KG has been greatly perturbed in this manner, then a “human-like” KG-augmented model should achieve much worse performance with the perturbed KG than with the original KG.

[4/n]

We propose four heuristics (RS, RR, ER, ED) and one RL algorithm (RL-RR) for perturbing the semantics and/or structure of the KG. Unlike the heuristics, RL-RR aims to maximize the downstream performance of models using the perturbed KG.

[5/n]

Interestingly, for both commonsense QA and item recommendation, the KG can be extensively perturbed with little to no effect on KG-augmented models’ performance! Here, we show results for KGs perturbed using RL-RR.

[6/n]

Plus, we find that original KG and perturbed KG (RL-RR) paths successfully utilized by KG-augmented models are hard for humans to read or use. This suggests that models and humans process KG info differently.

[7/n]

These findings raise doubts about the role of KGs in KG-augmented models and the plausibility of KG-based explanations. We hope that our paper can help guide future work in designing KG-augmented models that perform and explain better.

[8/n]

This work was led by @MrigankRaman, an undergrad intern at INK Lab (inklab.usc.edu). Many thanks to all of our co-authors: Siddhant Agarwal, Peifeng Wang, Hansen Wang, Sungchul Kim, Ryan Rossi, Handong Zhao, Nedim Lipka, and @xiangrenNLP.

[9/n]

Oops, adding Siddhant's Twitter handle here too: @agsidd10.

[10/n]