Enhancing Natural Language Processing Capabilities in Geriatric Patient Care: An Annotation Scheme and Guidelines

Manual data annotation of information related to geriatric syndrome (GS), which includes various conditions affecting older adults, is required to train machine learning (ML) models to classify patients’ health data for information that is otherwise poorly coded in structured electric health records. Such classification can be highly beneficial to support patients’ healthcare in several ways, including early detection and diagnosis of geriatric syndrome, improving healthcare practices and patient empowerment and education (leading to improved adherence to treatment plans and better overall health outcomes). This paper presents an annotation scheme used for labelling information related to GS (i.e. falls, frailty, dementia, etc.) from electronic health records and its application to a regional, Scottish National Health Service (NHS) dataset. The annotation scheme also captures contextual information on GS. An initial pilot involving the annotation of 163 documents manually annotated by two annotators using this scheme yielded very encouraging inter-annotator-agreement (IAA) results (macro average f1-score = 0.750). This pilot was used to proceed to some fine-tuning of the annotation guidelines with input from clinicians. The resulting annotation scheme is now being used for annotating patient discharge summaries, radiology reports and referral letters provided by a Scottish regional trusted research environment. So far 719 documents have been annotated, resulting in 1,684 GS annotations. We have also begun the annotation of MIMIC IV data using the same annotation scheme. Our final aim is a cross-country comparison of GS-related concept detection using different algorithms and ML models as well as variations in language formulation by clinicians in the USA and Scotland/England.