Large Language Models for Causal Relations Extraction in Social Media: A Validation Framework for Disaster Intelligence

Authors: Ujun Jeong, Saketh Vishnubhatla, Bohan Jiang et al.

arXiv · PDF

Summary

The authors test whether LLMs can extract causal relations (e.g., "X caused casualties") from disaster-related social media posts to improve situational awareness. They propose an evaluation framework that compares LLM-generated causal graphs to expert-derived reference graphs from disaster reports, and check whether extracted relations are supported by real post-event evidence or just reflect model priors (baked-in assumptions from training). The work highlights both promise and risks for using LLMs in disaster decision-support.

Main takeaways:

• Disaster social media posts are informal, fragmented, and often describe personal experiences rather than explicit causal chains.
• The authors build an expert-grounded framework to validate LLM causal extraction against real disaster reports.
• They test whether extracted causal relations come from actual post content or from model priors (learned patterns from pretraining).
• The goal is to identify what causes casualties, damage, or cascading impacts during disasters.
• Findings show both potential and substantial risks when relying on LLM extraction for high-stakes decision-making.

Relevance

Not directly related to my persona installation or conditional behavior work. Tangential connection to pretraining priors vs. context-driven behavior—my Qwen3 backdoor result similarly shows models fire on exact triggers rather than semantic paraphrases, which relates to their question of whether models extract real causal links or just surface priors.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.11348v1 Announce Type: new Abstract: During disasters, extracting causal relations from social media can strengthen situational awareness by identifying factors linked to casualties, physical damage, infrastructure disruption, and cascading impacts. However, disaster-related posts are often informal, fragmented, and context-dependent, and they may describe personal experiences rather than explicit causal relations. In this work, we examine whether Large Language Models (LLMs) can effectively extract causal relations from disaster-related social media posts. To this end, we (1) propose an expert-grounded evaluation framework that compares LLM-generated causal graphs with reference graphs derived from disaster-specific reports and (2) assess whether the extracted relations are supported by post-event evidence or instead reflect model priors. Our findings highlight both the potential and risks of using LLMs for causal relation extraction in disaster decision-support systems.