Advacheck at GenAI Detection Task 1: AI Detection Powered by Domain-Aware Multi-Tasking

German Gritsai,Anastasia Voznyuk,Ildar Khabutdinov,Andrey Grabovoy

2024-11-19

Abstract:The paper describes a system designed by Advacheck team to recognise machine-generated and human-written texts in the monolingual subtask of GenAI Detection Task 1 competition. Our developed system is a multi-task architecture with shared Transformer Encoder between several classification heads. One head is responsible for binary classification between human-written and machine-generated texts, while the other heads are auxiliary multiclass classifiers for texts of different domains from particular datasets. As multiclass heads were trained to distinguish the domains presented in the data, they provide a better understanding of the samples. This approach led us to achieve the first place in the official ranking with 83.07% macro F1-score on the test set and bypass the baseline by 10%. We further study obtained system through ablation, error and representation analyses, finding that multi-task learning outperforms single-task mode and simultaneous tasks form a cluster structure in embeddings space.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to accurately distinguish between texts written by humans and texts generated by language models in monolingual subtasks. Specifically, the research team Advacheck aims to improve the accuracy of machine - generated text detection through the Multi - Task Learning (MTL) architecture, especially in cross - domain and cross - generator settings.

### Problem Background

With the continuous development of large - language models (LLMs), the quality and coherence of machine - generated texts keep improving. This makes traditional binary - classification methods (such as Transformer - based fine - tuning models or zero - shot methods) perform poorly when there are changes in different domains, generation models or languages. In addition, the poor data quality in detection tasks also increases the difficulty of detection.

### Paper Goals

1. **Improve Detection Accuracy**: By introducing the multi - task learning architecture, enhance the model's adaptability to different domains and generation models, thereby improving the accuracy of detecting machine - generated texts.
2. **Form Better Text Representations**: Through multi - task learning, enable the shared encoder to form more fine - grained text representations and ignore data - dependent noise, improving the model's generalization ability.
3. **Reduce the Risk of Overfitting**: Multi - task learning can help the model focus better on key features and reduce the risk of overfitting.

### Solutions

The research team proposed a multi - task learning architecture that includes a shared Transformer encoder and multiple Custom Classification Heads (CCH). Specifically:

- **Binary - Classification CCH**: Used to solve the initial monolingual subtask, distinguishing between texts written by humans and machine - generated texts.
- **Multi - Classification CCH**: Used to identify sub - sources in the HC3 and M4GT datasets respectively, helping the model better understand text features in different domains.

Through this multi - task learning architecture, the research team not only improved the detection performance of the model but also formed a meaningful cluster structure in the embedding space, further enhancing the model's robustness and generalization ability.

### Experimental Results

The experimental results show that this multi - task learning architecture achieved a macro - F1 score of 83.07% on the official test set, which is 10% higher than the baseline model, and won the first place in the final ranking.

In summary, by introducing the multi - task learning architecture, this paper successfully solved the shortcomings of existing methods in detecting machine - generated texts in cross - domain and cross - generator settings, and significantly improved the accuracy and robustness of detection.