Instructions for Participants

 📕 | How to Participate

Participants will be allowed to submit a maximum total of 15 runs per model/system as follows.

• LLM-based models:

1. Runtype Zero-shot (mandatory, up to 5 runs): no exploitation of the GSI:detect development data

2. Runtype Few-shot (up to 5 runs): exploitation of the GSI:detect development data only as examples

3. Runtype Fine-tuned (up to 5 runs): exploitation of the GSI:detect data for fine-tuning

• Encoder-only solutions (e.g., Bert-based models):

1. Runtype Fine-tuned (up to 5 runs): exploitation of the GSI:detect data for fine-tuning

Use of  additional external resources
The development set is released under CC-BY-NC-SA 4.0 license.
There are no restrictions in the use of additional resources. Important: all data used must be specified in your system report.

Use of data for in-context learning
Participants may also use the guidelines_for_annotation.pdf file included in the zipped dataset, that can also be downloaded from the Dataset Section.

Required output

The output must be provided in .jsonl format, where each line represents a single JSON object containing the following fields:

{

 "id": "unique_identifier",

 "text": "Italian_short_text",

 "gs_value": "number_between_0_and_1_with_two_decimals-predicted",

 "gs_category": "predicted_gender_stereotype_category_[role, personality, competence, physical, sexual, relational]"

}

📨 | How to Submit

After running your system(s) on the raw test data, submit your results (i.e., .jsonl output files) according to the recommendations below and those provided on the EVALITA website, in line with the Important Dates.

1. File Naming: 

   • choose a team name and name your files following this format: gsidetect_TeamName_SystemName_Runtype_RunNumber

   • specify the number of examples used in few-shot runtype (e.g.,  gsidetect_TeamName_SystemName_few-shot10_RunNumber ).
     
     

2. Submission:

   • Compress all relevant files into a single .zip archive:

     1. 1. .jsonl output files in the format of the example snippet above (Required output paragraph);

        2. a brief .txt file with information about all data resources used, including any additional data used.

   • Send it via email to: gsievalita@gmail.com

   • Subject line: “gsidetect_outputs – TeamName”.

3. Technical report

After the evaluation, submit a report using the template and guidelines that will be published soon on the EVALITA website.

Your report must include:

a. Detailed description of the methodology;

b. All data resources used, including any additional data used;

c. Experimental details — prompts, preprocessing, hyper-parameter tuning, etc.

d. Analysis of results.