Tweets represent a great wealth of information useful to understand recent trends and user behaviors in real-time. Usually, natural language processing techniques would be applied to such pieces of information in order to make them machine-understandable. Named Entity rEcongition and Linking (NEEL) is a particularly useful technique aiming at assigning semantics to a text. However, due to the noisy nature and shortness of tweets, this technique is more challenging in this context than elsewhere. International initiatives provide evaluation frameworks for this task, e.g. the Making Sense of Microposts workshop, currently re-organizing the NEEL challenge for the fourth time, or the W-NUT workshop at ACL 2015, but the focus is always and strictly on the English language. We foresee an opportunity to (i) encourage the development of language independent tools for for Named Entity Recognition (NER) and Linking (NEL) systems and (ii) establish an evaluation framework for the Italian community. NEEL-IT at EVALITA has the vision to establish itself as a reference evaluation framework in the context of Italian tweets.
NEEL-IT will follow a setting similar to NEEL challenge for English Micropost on Twitter. The task consists of annotating each named entity mention (like people, locations, organizations, and products) in a text by linking it to a knowledge base (DBpedia). In the annotation process, a named entity is a string in the tweet representing a proper noun that: 1) belongs to one of the categories specified in a taxonomy and/or 2) can be linked to a DBpedia concept. This means that some concepts have a NIL DBpedia reference: these concepts belong to one of the categories but they have no corresponding concept in DBpedia. The taxonomy is defined by the following categories: Thing, Event, Character, Location, Organization, Person and Product.
From the annotation are excluded the preceding article (like il, lo, la, etc.) and any other prefix (e.g. Dott., Prof.) or post-posed modifier.
Each participant is asked to produce an annotation file with multiple lines, one for each annotation.
A line is a tab separated sequence of
linked concept in DBpedia, and
For example, given the tweet:
Chameleon Launcher in arrivo anche per smartphone: video beta privata su Galaxy Note 2 e Nexus 4: Chameleon Laun...
the annotation should produce the following output:
288976367238934528 0 18 NIL Product 288976367238934528 73 86 http://it.dbpedia.org/resource/Samsung_Galaxy_Note_II/ Product 288976367238934528 89 96 http://it.dbpedia.org/resource/Nexus_4 Product
The annotation should also link Twitter mentions (
@) and tags (
#) that refer to a named entities, like:
otation is expected:
288380466413854721 1 16 http://it.dbpedia.org/resource/Piazzapulita Product 288380466413854721 41 50 http://it.dbpedia.org/resource/Renata_Polverini Person
Participants are allowed to submit up to 3 runs of their system as TSV files. An example of the submission format will be released with the development set. We encourage participants to make available their system to the community to facilitate reuse.
We will use the TAC KBP scorer (https://github.com/wikilinks/neleval/wiki/Evaluation) to evaluate the results and in particular we will focus on:
- tagging strong_typed_mention_match (check entity name boundary and type)
- linking strong_link_match
- clustering mention_ceaf (NIL detection)
The strongtypedmention_match evaluates the micro average F-1 score for all annotations considering the mention boundaries and their types. The stronglinkmatch is the micro average F-1 score for annotations considering the correct link for each mention. The mention_ceaf (Constrained Entity-Alignment F-measure)  is a clustering metric developed to evaluate clusters of annotations. It evaluates the F-1 score for both NIL and non-NIL annotations in a set of mentions. The final score will be computed as follows:
score = 0.4 ∗ mention_ceaf + 0.3 ∗ strong_typed_mention_match + 0.3 ∗ strong_link_match
We will use the official scorer proposed for the TAC KBP task.
Giuseppe Rizzo, Amparo Elizabeth Cano Basave, Bianca Pereira, and Andrea Varga. Making sense of microposts (#microposts2015) named entity recognition and linking (NEEL) challenge. In Making Sense of Microposts, pages 44–53, 2015.
Amparo Elizabeth Cano Basave, Giuseppe Rizzo, Andrea Varga, Matthew Rowe, Milan Stankovic, and Aba-Sah Dadzie. Making sense of microposts (#microposts2014) named entity extraction & linking challenge. In Making Sense of Microposts, pages 54–60, 2014.
Xiaoqiang Luo. On coreference resolution performance metrics. HLT/EMNLP’05, 2005