About

Data Example

The training data will consist of text snippets from sustainability reports, along with their corresponding content classification labels. Each snippet is 3–5 sentences long and represents different reporting criteria sections such as "Ressourcenmanagement" or "Wesentlichkeit". Furthermore the last sentence of the snippet is annotated regarding it's verifiability. A sample snippet and its classification might look like this:

Content Classification:
Resource Management

Verifiability Rating:
0.8

Important Dates

Subtasks

Task A: Content Classification

Participants are tasked with assigning a content class to text snippets from sustainability reports. The text snippets are sampled from different sections of German-language company reports indexed in the German Sustainability Code (DNK). Each snippet corresponds to one of the predefined reporting criteria in DNK, and the goal is to classify the snippet according to its corresponding criterion section.

Evaluation: Accuracy
Classes:

Strategy	Process Management	Environment	Society
1. Strategic Analysis and Action 2. Materiality 3. Objectives 4. Depth of the Value Chain	5. Responsibility 6. Rules and Processes 7. Control 8. Incentive Systems 9. Stakeholder Engagement 10. Innovation and Product Management	11. Usage of Natural Resources 12. Resource Management 13. Climate-Relevant Emissions	14. Employment Rights 15. Equal Opportunities 16. Qualifications 17. Human Rights 18. Corporate Citizenship 19. Political Influence 20. Conduct that Complies with the Law and Policy

Task B: Verifiability Rating

This task focuses on rating the verifiability of the last sentence in each text snippet, with the prior sentences serving as context. The verifiability score is assigned on a scale from 0.0 (not verifiable) to 1.0 (clearly verifiable). The task is evaluated by the Kendall τ rank correlation with human ratings.

Evaluation: Kendall τ rank correlation (version B)

Questions & Answers

Which models can I use for the competition?

In order to promote diversity of modeling approaches in a fair manner, we offer several tracks, which enforce increasing limitations. Everyone automatically competes in the Open Track, where any data may be used as training data, except additional DNK reports, as these may include parts of the evaluation data, and any open-weights model may be used, including pre-trained LLMs. If the used external resources (models, data) are all compliant with a list of reproducible re-sources we will compile and publish by March 2025, participants also compete in the Reproducible Track.
How can I participate?

You can register on CodaBench as soon as we publish the link.
Will special prizes be awarded?

Prizes will be awarded for the best overall performance and for special achievements such as sustainability-focused models, insightful analysis, and interdisciplinary approaches.

Contact

Shared Task Email (contact for all questions): sustaineval@gmail.com

Jakob Prange, Universität Augsburg (contact for task specific questions): jakob.prange@uni-a.de

Charlott Jakob, TU Berlin (contact for organisational questions): c.jakob@tu-berlin.de

Annemarie Friedrich, Universität Augsburg

References

Bingler, J. A., Kraus, M., Leippold, M., & Webersinke, N. (2024). How cheap talk in climate disclosures relates to climate initiatives, corporate emissions, and reputation risk. Journal of Banking & Finance, 164, 107191.

Diggelmann, T., et al. (2020). Climate-FEVER: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614.

Schimanski, T., et al. (2023). ClimateBERT-netzero: Detecting and assessing net-zero and reduction targets. arXiv preprint arXiv:2310.08096.