Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts

Jenny Chim; Adam Tsakalidis; Dimitris Gkoumas; Dana Atzil-Slonim; Yaakov Ophir; Ayah Zirikly; Philip Resnik; Maria Liakata

Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts

Jenny Chim, Adam Tsakalidis, Dimitris Gkoumas, Dana Atzil-Slonim, Yaakov Ophir, Ayah Zirikly, Philip Resnik, Maria Liakata

Department of Education

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

17 Scopus citations

Abstract

We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their predetermined suicide risk level (‘Low’, ‘Moderate’ or ‘High’) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.

Original language	English
Title of host publication	CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop
Editors	Andrew Yates, Bart Desmet, Emily Prud�hommeaux, Ayah Zirikly, Steven Bedrick, Sean MacAvaney, Kfir Bar, Molly Ireland, Yaakov Ophir, Yaakov Ophir
Publisher	Association for Computational Linguistics (ACL)
Pages	177-190
Number of pages	14
ISBN (Electronic)	9798891760806
State	Published - 2024
Event	9th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2024 - St. Julian's, Malta Duration: 21 Mar 2024 → …

Publication series

Name	CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop

Conference

Conference	9th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2024
Country/Territory	Malta
City	St. Julian's
Period	21/03/24 → …

Cite this

Chim, J., Tsakalidis, A., Gkoumas, D., Atzil-Slonim, D., Ophir, Y., Zirikly, A., Resnik, P., & Liakata, M. (2024). Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. In A. Yates, B. Desmet, E. Prud�hommeaux, A. Zirikly, S. Bedrick, S. MacAvaney, K. Bar, M. Ireland, Y. Ophir, & Y. Ophir (Eds.), CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop (pp. 177-190). (CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

Chim, Jenny ; Tsakalidis, Adam ; Gkoumas, Dimitris et al. / Overview of the CLPsych 2024 Shared Task : Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop. editor / Andrew Yates ; Bart Desmet ; Emily Prud�hommeaux ; Ayah Zirikly ; Steven Bedrick ; Sean MacAvaney ; Kfir Bar ; Molly Ireland ; Yaakov Ophir ; Yaakov Ophir. Association for Computational Linguistics (ACL), 2024. pp. 177-190 (CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop).

@inproceedings{2d4e6132540e41fbac1094843319d85f,

title = "Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts",

abstract = "We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their predetermined suicide risk level ({\textquoteleft}Low{\textquoteright}, {\textquoteleft}Moderate{\textquoteright} or {\textquoteleft}High{\textquoteright}) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.",

author = "Jenny Chim and Adam Tsakalidis and Dimitris Gkoumas and Dana Atzil-Slonim and Yaakov Ophir and Ayah Zirikly and Philip Resnik and Maria Liakata",

note = "Publisher Copyright: {\textcopyright}2024 Association for Computational Linguistics.; 9th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2024 ; Conference date: 21-03-2024",

year = "2024",

language = "אנגלית",

series = "CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "177--190",

editor = "Andrew Yates and Bart Desmet and Emily Prud�hommeaux and Ayah Zirikly and Steven Bedrick and Sean MacAvaney and Kfir Bar and Molly Ireland and Yaakov Ophir and Yaakov Ophir",

booktitle = "CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop",

address = "ארצות הברית",

}

Chim, J, Tsakalidis, A, Gkoumas, D, Atzil-Slonim, D, Ophir, Y, Zirikly, A, Resnik, P & Liakata, M 2024, Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. in A Yates, B Desmet, E Prud�hommeaux, A Zirikly, S Bedrick, S MacAvaney, K Bar, M Ireland, Y Ophir & Y Ophir (eds), CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop. CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 177-190, 9th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2024, St. Julian's, Malta, 21/03/24.

Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. / Chim, Jenny; Tsakalidis, Adam; Gkoumas, Dimitris et al.
CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop. ed. / Andrew Yates; Bart Desmet; Emily Prud�hommeaux; Ayah Zirikly; Steven Bedrick; Sean MacAvaney; Kfir Bar; Molly Ireland; Yaakov Ophir; Yaakov Ophir. Association for Computational Linguistics (ACL), 2024. p. 177-190 (CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Overview of the CLPsych 2024 Shared Task

T2 - 9th Workshop on Computational Linguistics and Clinical Psychology, CLPsych 2024

AU - Chim, Jenny

AU - Tsakalidis, Adam

AU - Gkoumas, Dimitris

AU - Atzil-Slonim, Dana

AU - Ophir, Yaakov

AU - Zirikly, Ayah

AU - Resnik, Philip

AU - Liakata, Maria

PY - 2024

Y1 - 2024

N2 - We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their predetermined suicide risk level (‘Low’, ‘Moderate’ or ‘High’) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.

AB - We present the overview of the CLPsych 2024 Shared Task, focusing on leveraging open source Large Language Models (LLMs) for identifying textual evidence that supports the suicidal risk level of individuals on Reddit. In particular, given a Reddit user, their predetermined suicide risk level (‘Low’, ‘Moderate’ or ‘High’) and all of their posts in the r/SuicideWatch subreddit, we frame the task of identifying relevant pieces of text in their posts supporting their suicidal classification in two ways: (a) on the basis of evidence highlighting (extracting sub-phrases of the posts) and (b) on the basis of generating a summary of such evidence. We annotate a sample of 125 users and introduce evaluation metrics based on (a) BERTScore and (b) natural language inference for the two sub-tasks, respectively. Finally, we provide an overview of the system submissions and summarise the key findings.

UR - http://www.scopus.com/inward/record.url?scp=85185569942&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85185569942

T3 - CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop

SP - 177

EP - 190

BT - CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop

A2 - Yates, Andrew

A2 - Desmet, Bart

A2 - Prud�hommeaux, Emily

A2 - Zirikly, Ayah

A2 - Bedrick, Steven

A2 - MacAvaney, Sean

A2 - Bar, Kfir

A2 - Ireland, Molly

A2 - Ophir, Yaakov

PB - Association for Computational Linguistics (ACL)

Y2 - 21 March 2024

ER -

Chim J, Tsakalidis A, Gkoumas D, Atzil-Slonim D, Ophir Y, Zirikly A et al. Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts. In Yates A, Desmet B, Prud�hommeaux E, Zirikly A, Bedrick S, MacAvaney S, Bar K, Ireland M, Ophir Y, Ophir Y, editors, CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2024. p. 177-190. (CLPsych 2024 - 9th Workshop on Computational Linguistics and Clinical Psychology, Proceedings of the Workshop).

Overview of the CLPsych 2024 Shared Task: Leveraging Large Language Models to Identify Evidence of Suicidality Risk in Online Posts

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this