Language and graph foundational models: Distillation and Pretraining - Vasileios Ioannidis

IMA Industrial Problems Seminar
Vasileios Ioannidis (Amazon Search AI)
"Language and graph foundational models: Distillation and Pretraining"
Abstract: Graph neural networks (GNNs) learn from complex graph data and have been remarkably successful in various applications and across industries. This presentation first introduces GNNs via the message passing framework and dives into popular GNN variants. Next, it explores the fusion of textual data with heterogeneous graph structures to improve semantic and behavioral representations. It introduces the Language Model GNN (LM-GNN), a framework that efficiently combines large language models and Graph Neural Networks (GNNs) through fine-tuning. LM-GNN supports various tasks like node classification and link prediction and demonstrates its effectiveness. Another aspect addressed is the challenge of effective node representation learning in textual graphs. The Graph-Aware Distillation (Grad) framework is proposed, which encodes graph structures into a Language Model (LM) to enable fast and scalable inference. Grad optimizes GNN and a graphless student model, resulting in superior performance in node classification tasks. Finally, the presentation discusses pre-training text and graph models on large, heterogeneous graphs with textual data using the Graph-Aware Language Model Pre-Training (GALM) framework. It highlights the framework's effectiveness through experiments on real datasets.
Presented at the University of Minnesota on 11/10/2023.
You can learn more about the IMA Industrial Problems seminar here: cse.umn.edu/im...