More about use case, data and evaluation process

Chiraz Latiri, Julio Gonzalo, Malek Hajjem — 2018-02-09T09:39:44Z

Detailed description

use case

Given, a selected of festivals name from popular festivals on FlickR English and French language, participants have to search for the most argumentative tweets in a collection covering 18 months of news about festivals in different languages. The identified tweets have to be a summary of ranked tweets according to their probability of being argumentative tweets. This use case was proposed to help festival organiser treating such set of tweets on priority. That is why the more the summary of ranked tweets is variant the in term of argumentation the more the run is useful.
For each language English and French (English and French), a monolingual scenario is expected : Given a festival name from topics file, participants have to search, from the microblog collection, the set of the most argumentative tweets in the same query language.
Samples of argumentative Tweets are provided here: English_Sample, French_Sample

Topics

English and French contain respectively 12 and 4 festival name. They represent a set of some popular festivals on FlickR for which we have pictures. Topics were carefully selected by organizer to ensure that selected topics have enough related argumentative tweets in our corpus. Such manual selection was conduct to to ensure a possible evaluation.

The choice of FlickR as source of topic was motivated by the fact that such social media platform had a high quality amateur pictures. This personal involvement serves our goal as we are interested mainly to personal tweets.

Microblog Corpus

A login is required to access the data, once registered to CLEF

The complete stream of 70 000 000 microblogs is available here for registered participants. This document collection is provided by GAFES. Microblogs are provided with their meta-information and expanded URLs on a MySQL server.
Due to legal terms the access to this database is restricted to registered participants under privacy agreement.
An [indri Index with a web interface-https://mc2.talne.eu/data/clef/api] are available to query the whole set of microblogs

Evaluation

The official evaluation measure is NDCG.

This ranking measures will give a score for each retrieved tweet with a discount function over the rank. As we are mostly interested in top ranked arguments, this ranking measures meet our expectation.
This measure was also used in TREC Microblog Track [1]. A tweet is considered as highly relevant when it is a personal and contains an argument that directly refers to the festival (topic).

[1] Overview of the TREC-2015 Microblog Track
Jimmy Lin,Miles Efron, Yulu Wang, Garrick sherman, Ellen Voorhees

Result Submission
The runs must respect the classical trec top files format as describe above. Only the top 100 results for each query run must be given. Each run in each language, must contain 3 fields:
– Id : a long integer representation of the unique identifier of this Tweet
– Scores : The probability of being an argument tweet accorded by participant system
– Rank : The accorded position of the tweet in the grading list of argument tweets
– Content: Microblog textual content

Diversity criteria:
The more a run detects different arguments about a cultural event, the more it is interesting.

Exemples about "Cannes festival name:
– I ve seen some people saying they're boycotting Cannes because of the high heels rule. I'm not sure they'll notice.
– Not going to lie, one of my favorite things about the Cannes festival is all of these handsome men in tuxedos.
– Cannes is relevant because movies get timed standing ovations.

How to get the data?

To get an access to the Microblog corpus, email malek.hajjem@univ-avignon.fr or registered to CLEF
The English topics can be downloaded here
The French topics can be downloaded here.

Contact Information

If you have any question, email us through this address mail : malek.hajjem@univ-avignon.fr

Towards Argumentative Ranking

Chiraz Latiri, Julio Gonzalo, Malek Hajjem — 2018-02-08T18:17:15Z

Organizers:

Chiraz Latiri, Julio Gonzalo, Malek Hajjem

Task 2 participation deadline April 30, 2018

Argumentative Ranking of Microblogs

Argumentation mining is a new problem in corpus-based text analysis that addresses the challenging task of automatically identifying the justifications provided by opinion holders for their judgment. Several approaches of argumentation mining have been proposed so far in areas such as legal documents, on-line debates, product reviews, newspaper articles and court cases, as well as in dialogical domains.
With the popularization of social networks, argumentation mining is considered as an extension of the opinion mining issue from social network content. The aim is to automatically identify reason-conclusion structures that can lead to model social web user's positions about a service or an event expressed through social networks platforms like Twitter. Indeed, when we need to form an opinion on a new topic or make a decision, arguments will be all what we are looking for.
To make argumentation structures available, in case of Twitter, a robust automatic recognition of it is required, based on resources that should be created in a reproducible fashion to be reliable. However, the ambiguity of natural language text produced in social media, with different writing styles, implicit context and heterogeneous content make argumentation, on Twitter, very challenging.

Another possible way to pick up the argumentation structures, from a generic tweet corpus, is to use approaches based on information extraction. The idea is to perform a search process that focus on claims about a given topic out in a massive collection. This approach relates to the field of focused retrieval, that aims to provide users with direct access to relevant information in retrieved documents. In this task, relevant information is expressed in the form of arguments. [1]

Success of such argumentation ranking will require interdisciplinary approaches based on the combination of different research issues. In fact, to better understand a short text and be able to detect the argumentative structures within a microblog, we could restore a « text contextualization » as a way to provide more information on the corresponding text [2]. Providing such information in order to detect argumentative tweets, would highlight relevant ones, in other words, tweets expressed in the form of arguments. Thus, argumentation mining in this situation will tend to act in the same way of an Information Retrieval (IR) system where potential argumentative tweets had to come first. Similar approach that addresses such purpose is presented in [3], where the output of the priority task will be a ranking of tweets according to their probability of being a potential threat to the reputation of some entity.

[1] Argumentative Ranking
Marco Lippi and Paolo Sarti and Paolo Torroni DISI - Universita degli Studi di Bologna Proceedings of Natural Language Processing meets Journalism - IJCAI-16 Workshop (NLPMJ 2016), New York, (July 2016)
[2] INEX Tweet Contextualization task : Evaluation, results and lesson learned
Patrice Bellot, Véronique Moriceau, Josiane Mothe, Eric SanJuan, Xavier Tannier:- Inf. Process. Manage. 52(5): 801-819 (2016)
[3] Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems
Enrique Amigo, Jorge Carrillo de Albornoz, Irina Chugur, Adolfo Corujo Julio Gonzalo, Tamara Martin, Edgar Meij Maarten de Rijke and Damiano Spina

MC2 2018 Lab

More about use case, data and evaluation process

Detailed description

Towards Argumentative Ranking