The Task First, Please
5 pages; PDF.
by Valentin Jijkoun and Maarten de Rijke (University of Amsterdam)

From the abstract:

We examine the current state of evaluation exercises for automatic Question Answering (QA) systems, specifically targeting the QA task (QA@CLEF) as it is being evaluating with the setting of the Cross-Language Evaluation Forum (CLEF). We describe several key issues for the evaluation of QA systems and show how they are problematic in the current setup of the tasks at QA@CLEF. We argue that many of the problems are caused by the lack of a clear understanding of the QA task that should include potential users, types of information needs, types of available information resources. Finally, we propose several scenarios for QA and focused retrieval tasks that address these problematic issues. Our main conclusion is simple but important: a clear task definition is paramount for a meaningful evaluation of automatic systems, as evidenced by the overview of the QA evaluation setups.

This paper will be presented at SIGIR 2007, Workshop on Focused Retrieval July 27, 2007, Amsterdam, The Netherlands

This entry was posted on Thursday, July 5th, 2007 and is filed under Info Management and Retrieval, Information Science, Information Seeking, Search News. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.