About

Data Example

The training data will consist of text snippets from sustainability reports, along with their corresponding content classification labels. Each snippet is 3–5 sentences long and represents different reporting criteria sections such as "Ressourcenmanagement" or "Wesentlichkeit". Furthermore the last sentence of the snippet is annotated regarding it's verifiability. You can find some trial data here . A sample snippet and its classification might look like this:

Content Class:
Resource Management

Verifiability Rating:
0.8

The training data and development data are constructed with permission from publicly available German-language company reports indexed in the German Sustainability Code (Deutscher Nachhaltigkeitskodex, DNK). Text snippets are sampled semi-automatically and then processed to ensure well-formedness and to exclude personally identifiable information.

Important Dates

Subtasks

Task A: Content Classification

Participants are tasked with assigning a content class to text snippets from sustainability reports. The text snippets are sampled from different sections of German-language company reports indexed in the German Sustainability Code (DNK). Each snippet corresponds to one of the predefined reporting criteria in DNK, and the goal is to classify the snippet according to its corresponding criterion section.

Evaluation: Accuracy
Classes:

Strategy	Process Management	Environment	Society
1. Strategic Analysis and Action 2. Materiality 3. Objectives 4. Depth of the Value Chain	5. Responsibility 6. Rules and Processes 7. Control 8. Incentive Systems 9. Stakeholder Engagement 10. Innovation and Product Management	11. Usage of Natural Resources 12. Resource Management 13. Climate-Relevant Emissions	14. Employment Rights 15. Equal Opportunities 16. Qualifications 17. Human Rights 18. Corporate Citizenship 19. Political Influence 20. Conduct that Complies with the Law and Policy

Task B: Verifiability Rating

This task focuses on rating the verifiability of the last sentence in each text snippet, with the prior sentences serving as context. The verifiability score is assigned on a scale from 0.0 (not verifiable) to 1.0 (clearly verifiable). The task is evaluated by the Kendall τ rank correlation with human ratings.

Evaluation: Kendall τ rank correlation (version B)

Questions & Answers

Which models can I use for the competition?

In order to promote diversity of modeling approaches in a fair manner, we offer several tracks. Everyone automatically competes in the Open Track, where any data may be used as training data, except additional DNK reports, as these may include parts of the evaluation data, and any open-weights model may be used, including pre-trained LLMs. If the used external resources (models, data) are all compliant with a list of reproducible re-sources we will compile and publish by March 2025, participants also compete in the Reproducible Track. This list will likely contain established open-source models like BERT, RoBERTa, DeBERTa, T5, GPT-2, as well as their German-specific variants, and may impose an upper limit on model size of 7 billion parameters (details to be confirmed). Lastly, we highly encourage participants to build small explainable models from scratch using only our training data, which will compete in the Explainable Track.
How can I participate?

You can register on CodaBench as soon as we publish the link.
Will special prizes be awarded?

Prizes will be awarded for the best overall performance and for special achievements such as sustainability-focused models, insightful analysis, and interdisciplinary approaches.

Contact

Shared Task Email (contact for all questions): sustaineval@gmail.com

Jakob Prange, Universität Augsburg (contact for task specific questions): jakob.prange@uni-a.de

Charlott Jakob, TU Berlin (contact for organisational questions): c.jakob@tu-berlin.de

Annemarie Friedrich, Universität Augsburg