Abstract:Prior highly-tuned image parsing models are usually studied in a certain domain with a specific set of semantic labels and can hardly be adapted into other scenarios (e.g.sharing discrepant label granularity) without extensive re-training. Learning a single universal parsing model by unifying label annotations from different domains or at various levels of granularity is a crucial but rarely addressed topic. This poses many fundamental learning challenges, e.g.discovering underlying semantic structures among different label granularity or mining label correlation across relevant tasks. To address these challenges, we propose a graph reasoning and transfer learning framework, named "Graphonomy," which incorporates human knowledge and label taxonomy into the intermediate graph representation learning beyond local convolutions. In particular, Graphonomy learns the global and structured semantic coherency in multiple domains via semantic-aware graph reasoning and transfer, enforcing the mutual benefits of the parsing across domains (e.g.different datasets or co-related tasks). The Graphonomy includes two iterated modules: Intra-Graph Reasoning and Inter-Graph Transfer modules. The former extracts the semantic graph in each domain to improve the feature representation learning by propagating information with the graph; the latter exploits the dependencies among the graphs from different domains for bidirectional knowledge transfer. We apply Graphonomy to two relevant but different image understanding research topics: human parsing and panoptic segmentation, and show Graphonomy can handle both of them well via a standard pipeline against current state-of-the-art approaches. Moreover, some extra benefit of our framework is demonstrated, e.g., generating the human parsing at various levels of granularity by unifying annotations across different datasets.

Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining

Probing Structural Knowledge from Pre-trained Language Model for Argumentation Relation Classification

Multi-view Hierarchical Graph Neural Network for Argumentation Mining

Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models

Enhancing Argumentative Relation Classification by Multi-Granularity Retrieval and Heterogeneous Graph Reasoning.

ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Pretraining Language Models with Text-Attributed Heterogeneous Graphs

A Hierarchical Language Model For Interpretable Graph Reasoning

Augmenting Reasoning Capabilities of LLMs with Graph Structures in Knowledge Base Question Answering

Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs

Graphonomy: Universal Image Parsing Via Graph Reasoning and Transfer

Reason more like human: Incorporating meta information into hierarchical reinforcement learning for knowledge graph reasoning

A New Pipeline for Knowledge Graph Reasoning Enhanced by Large Language Models Without Fine-Tuning

Distilling Large Language Models for Text-Attributed Graph Learning

Graph Pre-training for AMR Parsing and Generation

Syntax-informed Question Answering with Heterogeneous Graph Transformer

Graphonomy: Universal Human Parsing Via Graph Transfer Learning

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering