Abstract:This paper describes the submissions of Alibaba DAMO Academy to the TREC Precision Medicine (PM) Track in 2020, which achieve state-of-the-art performance in the evidence quality assessment. The focus of the TREC PM Track is to retrieve academic papers that report critical clinical evidence for or against a given treatment in a population specified by its disease and gene mutation. We use a two-step approach that includes: 1) a baseline retriever using query expansion with Elasticsearch (ES) and 2) an automatic or expert-in-the-loop reranker: the automatic re-ranker uses features of the ES scores, pre-trained BioBERT scores, publication type scores and citation count scores; the expert-in-the-loop re-ranker uses expert annotations, fine-tuned BioBERT as well as features used in the automatic re-ranker. For the expert-in-the-loop re-ranker, we use a novel active learning annotation strategy that is sample-efficient: at each iteration of the annotation, 1) we fine-tune the BioBERT using all expert annotations of query-document relevance; 2) we let human experts annotate the actual relevance of the most relevant unannotated query-document pairs predicted by the fine-tuned BioBERT. Our submissions outperform the median topic-wise scores in the phase 1 assessment for general relevance and achieve state-of-the-art performance in the phase 2 assessment for evidence quality. Our analyses show that evidence quality is a distinct aspect than the general relevance, and thus additional modeling of it is necessary to assist IR for Evidence-based Precision Medicine

QCRI at TREC 2013 Microblog Track.

QCRI at TREC 2014: Applying the KISS Principle for the TTG Task in the Microblog Track.

ICTNET at Microblog Track in TREC 2014.

Learning to Rank Microblog Posts for Real-Time Ad-Hoc Search.

Microblog Search and Filtering with Real-Time Dynamics Based on BM25

Microblog Track 2011 of FDU.

Microblog Search and Filtering with Time Sensitive Feedback and Thresholding Bsed on BM25.

Exploring Tweets Normalization and Query Time Sensitivity for Twitter Search

Compact Indexing and Judicious Searching for Billion-Scale Microblog Retrieval.

Ranking model selection and fusion for effective microblog search

THUIR at TREC 2008: Relevance Feedback Track

ICTNET at Microblog Track in TREC 2013.

FDU at TREC 2007: Opinion Retrieval of Blog Track.

Real-Time Search over a Microblogging System

Aliababa DAMO Academy at TREC Precision Medicine 2020: State-of-the-art Evidence Retriever for Precision Medicine with Expert-in-the-loop Active Learning

THUIR at TREC2008: Blog Track 1

THUIR at TREC 2009 Web Track: Finding Relevant and Diverse Results for Large Scale Web Search

Peking University at the TREC 2005 Question and Answering Track.

At the web track of TREC 2003

Team IELAB at TREC Clinical Trial Track 2023: Enhancing Clinical Trial Retrieval with Neural Rankers and Large Language Models

TREC-10 Web Track Experiments at MSRA.