Workshop on Knowledge Graph and Large Language Models (KaLLM)

News Flash: We welcome papers accepted at the main conference (both Findings and regular papers) to be presented at our workshop.

Large Language Models (LLMs) have revolutionised the field of Natural Language Processing (NLP) by improving contextual understanding, zero-shot and few-shot learning, text generation, and question answering. However, LLMs have limitations such as accuracy, bias, interpretability, and context. They often produce factually inaccurate information, struggle to understand complex contexts, and may unintentionally produce biased or stereotyped information. KGs, on the other hand, have high-quality explicit knowledge, symbolic reasoning potential, and the ability to evolve with new knowledge, making them essential for various applications. The intersection of LLMs and KGs has sparked significant interest and innovation in NLP. LLM-enhanced KGs can be introduced into pre-training and inference stages to provide external knowledge and assess interpretability. LLM-augmented KGs are designed to improve coverage and ease the use of KGs in various tasks, including embedding learning, completion, construction, KG-to-text generation, and KGQA. Combining the benefits of LLMs and KGs can also improve performance in knowledge representation.

The workshop intends to provide a platform for researchers, practitioners, and industry professionals to explore the synergies between LLMs and KGs. We aim to provide a space for the LLM community and the community of KG researchers to interact and explore how these two communities could collaborate and support one another. The goal of the workshop is to seize on the tremendous opportunities arising from investigating cutting-edge approaches, addressing challenges and limitations, and applications in different domains.

Call for Papers

The submissions should focus on the interaction between LLMs and KGs in the context of NLP. The workshop will cover a diverse range of topics related to the integration of LLMs and KGs, including but not limited to:

Knowledge-enhanced language generation
KG-based question answering using LLMs
Fact validation and bias mitigation
KG creation and completion using LLMs
Privacy considerations in LLM-KG integration
Interpretability and explainability
Cross-domain applications
KG-based text summarisation with LLMs
Ethical implications of LLM-KG technologies
Multimodality of KGs and LLMs
Multilingual LLMs for KGs and vice-versa

Submission Guidelines

We invite the following types of submissions:

Long papers must be original unpublished work, with concrete evaluation and analysis. They can be up to 8 (eight) pages, plus unlimited pages of references and an appendix.
Short papers should be original unpublished work, including a small focused contribution, a negative result, an opinion piece, or an interesting application. They can be of up to 4 (four) pages of content with unlimited pages of references and appendix.
Resource papers must include new dataset(s) or other resource(s) directly relevant to KGs and LLMs, including the publication of that resource. The resource papers can be up to 8 (eight) pages, plus unlimited pages of references and appendix.

All submissions should be anonymous and must follow ACL formatting guidelines. The ACL 2024 templates must be used; these are provided in LaTeX and also Microsoft Word format. Submissions will only be accepted in PDF format. The peer review will be double-blind. The workshop will not accept work that is under review or has already been published in or accepted for publication in any other peer-reviewed venue. However, papers published in arXiv are allowed to be submitted to the workshop. We consider both archival and non-archival submissions. But only accepted archival/regular papers will be included in the proceedings.

All submissions have to be made through the following Openreview link.

Authors should be aware of OpenReview's moderation policy for newly created profiles:

New profiles created without an institutional email will go through a moderation process that can take up to two weeks.
New profiles created with an institutional email will be activated automatically.

Camera Ready Version

Final versions of accepted papers will be given one additional page of content (up to 9 pages for long papers, upto 5 pages for short papers) to address reviewers’ comments.

Best Paper Award

We will recognise the best paper with an award. Reviewers will be asked to flag papers they deem worthy of a prize. The general chairs will set up a small panel that will determine the winner. The award will be sponsored by Bloomberg.

Important Dates

Submission Deadline	~~May 22, 2024~~ May 30, 2024
Submission Starts	Feb 05, 2024
Author Notifications	June 25, 2024
Camera-Ready Deadline	July 03, 2024
Workshop Date	August 15, 2024

All deadlines are at 23:59, UTC-12h in the Anywhere on Earth (AOE) time zone.

Program

09:00 - 9:15	Opening Remarks
9:15 - 10:00	Keynote Talk: Heng Ji
10:00 - 10:30	Elevator Pitch Marathon
10:30 - 11:00	Coffee Break
11:00 - 11:45	Keynote Talk: Marko Grobelnik
11:45 - 13:00	Poster Session I
13:00 - 14:00	Lunch
14:00 - 14:45	Keynote Talk: Xin Luna Dong
14:45 - 15:30	Keynote Talk: Ivan Titov
15:30 - 16:00	Coffee Break
16:00 - 16:45	Poster Session II
16:45 - 17:15	Panel Discussion: Xin Luna Dong, Marko Grobelnik, Heng Ji, Ivan Titov, Isabelle Augenstein Topics: The Future of LLMs and KGs: Convergence and Synergy, Adaptability and Resilience of LLMs and KGs
17:15 - 17:30	Closing Remarks

Speakers and Panelists

Xin Luna Dong is a Principal Scientist at Meta Reality Labs, leading the ML efforts in building an intelligent personal assistant. She has spent more than a decade building knowledge graphs, such as the Amazon Product Graph and the Google Knowledge Graph. She has co-authored books "Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases" and “Big Data Integration”. She was named an ACM Fellow and an IEEE Fellow for "significant contributions to knowledge graph construction and data integration", and awarded the VLDB Women in Database Research Award and VLDB Early Career Research Contribution Award. She serves on the PVLDB advisory committee, was a member of the VLDB endowment, and a PC co-chair for the KDD’2022 ADS track, WSDM’2022, VLDB’2021, and Sigmod’2018.

Talk: The Journey to A Knowledgeable Assistant with Retrieval-Augmented Generation (RAG)
For decades, multiple communities (Database, Information Retrieval, Natural Language Processing, Data Mining, AI) have pursued the mission of providing the right information at the right time. Efforts span web search, data integration, knowledge graphs, question answering. Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities in comprehending and generating human language, revolutionizing techniques in every front. However, their inherent limitations such as factual inaccuracies and hallucinations make LLMs less suitable for creating knowledgeable and trustworthy assistants.
This talk describes our journey in building a knowledgeable AI assistant by harnessing LLM techniques. We start with our findings from a comprehensive set of experiments to assess LLM reliability in answering factual questions and analyze performance variations across different knowledge types. Next, we describe our federated Retrieval-Augmented Generation (RAG) system that integrates external information from both the web and knowledge graphs for trustworthy text generation on real-time topics like stocks and sports, as well as on torso-to-tail entities like local restaurants. Additionally, we brief our explorations on extending our techniques towards multi-modal, contextualized, and personalized Q\&A. We will share our techniques, our findings, and the path forward, highlighting how we are leveraging and advancing the decades of work in this area.

Marko Grobelnik is a researcher in the field of Artificial Intelligence (AI). Focused areas of expertise are Machine Learning, Data/Text/Web Mining, Network Analysis, Semantic Technologies, Deep Text Understanding, and Data Visualization. Marko co-leads the Artificial Intelligence Lab at Jozef Stefan Institute, co-founded the UNESCO International Research Center on AI (IRCAI), and is the CEO of Quintelligence.com specialises in solving complex AI tasks for the commercial world. He collaborates with major European academic institutions and major industries such as Bloomberg, British Telecom, European Commission, Microsoft Research, New York Times, and OECD. Marko is co-author of several books, co-founder of several start-ups and has been involved in over 100 EU-funded research projects in various fields of Artificial Intelligence. Significant organisational activities include Marko being general chair of LREC2016 and TheWebConf2021 conferences. Marko represents Slovenia in the OECD AI Committee (AIGO/ONEAI), the Council of Europe Committee on AI (CAHAI/CAI), NATO (DARB), and the Global Partnership on AI (GPAI). In 2016 Marko became the Digital Champion of Slovenia at the European Commission.

Talk: Extracting Common Sense World Models from LLMs
LLMs are often criticized for not operating with a notion of world models, which could provide robustness, explainability, and multi-hop reasoning. In this keynote, we will show the methodology and concrete examples of how to extract non-trivial symbolic world models from LLMs for an arbitrary domain. The extracted world models will be represented in an operational first-order logic; concretely in the Prolog programming language, in its basic and probabilistic versions. In the second step, the extracted world models will be used to ground textual data into the semantics of world models to enable reasoning, explanation, and possibly efficient agent communication operating with explicit representations. The approach aims to integrate black-box LLM representations with transparent symbolic representations close to humans without significant loss of information for practical applications.

Heng Ji is a professor at the Computer Science Department and an affiliated faculty member at the Electrical and Computer Engineering Department and Coordinated Science Laboratory of the University of Illinois Urbana-Champaign. She is an Amazon Scholar. She is the Founding Director of the Amazon-Illinois Center on AI for Interactive Conversational Experiences (AICE). She received her B.A. and M.A. in Computational Linguistics from Tsinghua University, and her M.S. and Ph.D. in Computer Science from New York University. Her research interests focus on Natural Language Processing, especially on Multimedia Multilingual Information Extraction, Knowledge-enhanced Large Language Models and Vision-Language Models. She was selected as a "Young Scientist" by the World Laureates Association in 2023 and 2024. She was selected as a "Young Scientist" and a member of the Global Future Council on the Future of Computing by the World Economic Forum in 2016 and 2017. She was named as part of Women Leaders of Conversational AI (Class of 2023) by Project Voice. The other awards she received include two Outstanding Paper Awards at NAACL2024, "AI's 10 to Watch" Award by IEEE Intelligent Systems in 2013, NSF CAREER award in 2009, PACLIC2012 Best Paper runner-up, "Best of ICDM2013" paper award, "Best of SDM2013" paper award, ACL2018 Best Demo paper nomination, ACL2020 Best Demo Paper Award, NAACL2021 Best Demo Paper Award, Google Research Award in 2009 and 2014, IBM Watson Faculty Award in 2012 and 2014 and Bosch Research Award in 2014-2018. She was invited to testify to the U.S. House Cybersecurity, Data Analytics, & IT Committee as an AI expert in 2023. She was selected to participate in DARPA AI Forward in 2023. She was invited by the Secretary of the U.S. Air Force and AFRL to join the Air Force Data Analytics Expert Panel to inform the Air Force Strategy 2030, and invited to speak at the Federal Information Integrity R&D Interagency Working Group (IIRD IWG) briefing in 2023. She is the lead of many multi-institution projects and tasks, including the U.S. ARL projects on information fusion and knowledge networks construction, DARPA ECOLE MIRACLE team, DARPA KAIROS RESIN team and DARPA DEFT Tinker Bell team. She has coordinated the NIST TAC Knowledge Base Population Task 2010-2020. She is the Chief Editor of Data Intelligence Journal. She served as the associate editor for IEEE/ACM Transaction on Audio, Speech, and Language Processing, and the Program Committee Co-Chair of many conferences including NAACL-HLT2018 and AACL-IJCNLP2022. She was elected as the secretary of the North American Chapter of the Association for Computational Linguistics (NAACL) 2020-2023. Her research has been widely supported by the U.S. government agencies (DARPA, NSF, DoE, ARL, IARPA, AFRL, DHS) and industry (Amazon, Google, Bosch, IBM, Disney).

Talk: Making Large Language Model’s Knowledge More Accurate, Organized, Up-to-date and Fair
Large language models (LLMs) have demonstrated remarkable performance on knowledge reasoning tasks, owing to their implicit knowledge derived from extensive pretraining data. However, their inherent knowledge bases often suffer from disorganization and illusion, bias towards common entities, and rapid obsolescence. Consequently, LLMs frequently make up untruthful information, exhibit resistance to updating outdated knowledge, or struggle with generalizing across multiple languages. In this talk, I will discuss our recent research efforts to tackle these challenges. I will begin by presenting theoretical and empirical analyses that illuminate when and why LLMs frequently produce factual errors and struggle to determine knowledge-updating boundaries in order to reach the “ripple effect”. Our investigations reveal several underlying causes. First, LLMs acquire implicit knowledge primarily through attention-weighted associations between words, rather than explicit understanding of concepts, entities, attributes, relations, events, semantic roles, and logics. Second, frequent word associations overshadow uncommon ones due to training data imbalance and wide context, particularly in contexts involving dynamic events. Third, counter-intuitive updating behaviours are elucidated through a novel gradient similarity metric. Fourth, LLMs are often unaware of real-world events occurring after their pretraining phase, complicating the anchoring of related knowledge updates. While existing methods focus largely on updating entity attributes, our research underscores the necessity of updating factual knowledge—such as participants, semantic roles, time, and location—based on real-world events. We propose a novel framework for knowledge updating in LLMs that leverages event-driven signals to identify factual errors preemptively and introduce a training-free self-contrastive decoding approach to mitigate inference errors.

Ivan Titov is a Full Professor at the University of Edinburgh, UK, and also a faculty member at the University of Amsterdam, Netherlands. Ivan's current interests lie in making deep learning models interpretable, robust, and controllable, or more generally in machine learning for NLP. He has received awards at leading NLP conferences. Ivan has been a program co-chair of ICLR 2021 and CoNLL 2018 and has served on the editorial boards of the Transactions of the ACL, Journal of Artificial Intelligence Research, and Journal of Machine Learning Research, and the advisory board of the European chapter of ACL. Ivan is an ELLIS fellow and co-directs the ELLIS NLP program and Edinburgh ELLIS unit. Ivan's research group has been supported by personal fellowships (e.g., ERC, Dutch Vici, and Vidi grants) as well as industrial funding (e.g., Google, SAP, Booking.com and Amazon).

Talk: Understanding and Navigating Human Control and Transparency in LLMs
Language models are an exciting technology that has transformed our field and is used by millions of people daily. However, both users and researchers often find themselves puzzled by LLM’s responses and struggle to understand the underlying decision processes or attribute their responses to specific data sources. I will talk about our work which tries to enhance the transparency of these models for human users, ensure their behaviour is systematic, and uncover the sources of their decisions. This transparency should enable finer control of these models, including model editing, the unlearning of undesirable behaviours or data sources, and the integration of extra information (e.g., in the form of knowledge bases).
In this talk, I will discuss the approaches my group (as well as colleagues) have been developing, highlighting not only methods but also some cautious lessons learned along the way. This includes pitfalls in data attribution and the challenges of guiding model responses with human rationale. Although progress in these areas may seem slow and sometimes illusory, it is a crucial direction, given the growing reliance on collaboration between humans and large language models. I also hope to convince you that this area holds a diverse range of intriguing open problems for us, researchers, to explore.

Isabelle Augenstein is a Professor at the University of Copenhagen, Department of Computer Science, where she heads the Copenhagen Natural Language Understanding research group as well as the Natural Language Processing section. Her main research interests are fair and accountable NLP, including challenges such as explainability, factuality and bias detection. Prior to starting a faculty position, she was a postdoctoral researcher at University College London, and before that a PhD student at the University of Sheffield. In October 2022, Isabelle Augenstein became Denmark’s youngest-ever female full professor. She currently holds a prestigious ERC Starting Grant on 'Explainable and Robust Automatic Fact Checking', as well as the Danish equivalent of that, a DFF Sapere Aude Research Leader fellowship on 'Learning to Explain Attitudes on Social Media’. She is a member of the Royal Danish Academy of Sciences and Letters, and President of SIGDAT, which organises the EMNLP conference series.