Automatic Extraction of Mental Health Disorders From Domestic Violence Police Narratives: Text Mining Study

Introduction

Domestic violence (DV) can be defined as “any incident of threatening behavior, violence, or (psychological, physical, sexual, financial, emotional) abuse between adults who are or have been an intimate partner or family member, regardless of gender or sexuality” [1]. DV can also occur in other relationships, such as between a caregiver and a dependent person or those living together in a household in a nonintimate relationship [2]. It is recognized as one of the most common forms of interpersonal violence and is an international social and public health problem with important health care consequences affecting the lives of thousands, mostly women, each year [3-5]. According to the World Health Organization’s (WHO) multicountry study of violence, the prevalence of physical and sexual partner violence toward women ranges from 15% to 71% globally [3,5]. In 2014, almost 50,000 people in Australia were recorded by the police as the victims of DV [6]. The cost of DV is significant with estimates suggesting that, in Australia, the cost of violence against women was approximately Aus $22.2 billion in 2015-2016, and in the United Kingdom and the United States, £17 billion and US $4.1 billion, respectively [4,5,7].

Domestic violence shares a complicated relationship with the onset, duration, and recurrence of mental health disorders, including substance abuse, eating disorders, posttraumatic stress, and suicidal tendencies, as well as exacerbation of psychotic symptoms [3-5,8]. Previous reports have suggested an increased risk of DV in populations with mental health disorders in comparison to those with no mental illness [3,9]. Over the past 20 years, a consensus has emerged that there is a modest (yet statistically significant) relationship between severe mental illness and violence, with severe mental illness increasing the risk of an individual to be violent toward others [10].

In 2017, the New South Wales Police Force (NSWPF) recorded 123,330 DV-related events in WebCOPS, a Web-based interface for the Computerised Operational Policing System (COPS) that enables the police to capture and analyze crime information on an organization-wide basis, with approximately 37% resulting in an offense being recorded (NSW Police Force, personal communication, June 2018). Information about DV events contained in WebCOPS is available as both structured form (eg, fields documenting information such as date of birth, Aboriginal status, whether weapons were used) and free unstructured text (“event narratives”). Each event contains one or more text narratives that describe in detail the alleged incident(s) that occurred between the person of interest (POI) and the victim, the circumstances of the event, and any action(s) taken by the police. The narratives are often written without a specific structure, populated with frequent misspellings and typographical errors, often with (sometimes informal) acronyms and abbreviations that can bear ambiguous meaning depending on the context.

The large number of DV events and the associated text narratives, however, prevent the extraction of potentially useful information using traditional ethnographic/qualitative approaches. One recent research paper commented that “...there is no systematic way to extract information from these [police] narratives other than by manual review” [11].

Still, automated methods for large-scale processing of free text known as text mining have been used for over 30 years to harvest information from unstructured text in many domains, particularly in biomedicine [12-15]. Recent attempts have aimed to utilize text mining to identify crime-related information from online media publications [16,17]. However, few efforts have been conducted in processing police reports [18-20]. Limited work included identification of offenders’ names, narcotic drugs, and weapons with various degrees of success (F-score [a measure of a method’s accuracy] ranging from 46% to 81%) through named entity extractors [18,19] and classification of police reports as DV or non-DV related by applying an unsupervised clustering technique that classified 44% of the reports set aside for manual inspection correctly [20].

Several attempts have been also made to extract mental health-related information from various free-text resources [21-27]. For example, drug side effects were extracted from psychiatric narratives by applying either hybrid methodologies of machine learning and dictionaries with rules, or rule-based approaches only that returned F-scores between 75% and 85% [21,24,25]. Treatment outcomes for major depressive disorders were identified from electronic medical records using a supervised approach with logistic regression, with precision ranging from 78% to 86% [22]. Mini Mental State Examination results were recognized from clinical notes and health record correspondence between clinicians with 85% and 91% F-scores, respectively, through a rule-based method [23]. Jackson et al [26] and Karystianis et al [27] both identified symptoms of mental illness from clinical discharge summaries and psychiatric records using either regular expression pattern matching or a rule-based approach with 88% and 81% F-scores, respectively [26,27].

In this paper, we examine whether automatic text mining of DV police event narratives is feasible in identifying mentions of mental health disorders at the narrative level among those involved in DV events by employing a knowledge-driven approach. This approach is based on lexicalized rules combined with manually constructed dictionaries that characterize mental health disorders in both POIs and victims involved in domestic disputes recorded by the NSWPF. We further perform a large-scale analysis of 492,393 DV events and report the results. To our knowledge, there has not been any application of text mining in the area of DV using real-world events and this is the first attempt of its kind to capture important mental health information in a large-scale analysis of DV events as recorded by the police.

Methods

Overview

Mentions of mental health disorders (including traumatic brain injury) were identified among POIs and victims in DV disputes based on the full list of the disorders (Textbox 1) according to the WHO’s International Classification of Diseases, Tenth Revision (ICD-10) for mental and behavioral disorders [28]. We also recognized mentions of unspecified mental disorders reported in the narratives (eg, “the defendant has mental health issues,” “victim is suffering from a severe mental disorder”), mentions of psychotropic medications by name or drug class (eg, “the victim takes Valium,” “accused takes a number of antidepressants”), and mentions of traumatic brain injury, drug prescription abuse, substance abuse, and drug-induced disorders.

Data

We obtained records of 492,393 DV events from WebCOPS from January 2005 to December 2016 flagged either as “domestic violence related” or the description of violence in WebCOPS was coded as “domestic” or the relationship between the victim and the POI included any of the following: spouse/partner (including ex-spouse/ex-partner), boyfriend/girlfriend (including ex-boyfriend/ex-girlfriend), parent/guardian (including step/foster), child (including step/foster), sibling, other member of family (including kin), or carer. These events covered the following categories: various types of assaults, breaches of Apprehended Violence Orders, homicides, malicious damage to property, and offense against another person such as intimidation, kidnapping, abduction, and harassment. The records also contained incidents where no crime was committed but the police attended the DV event nonetheless. All event narratives contained personal information (eg, first name, surname, address) and therefore are not available to the general public. Permission to access the narratives was granted by the NSWPF following ethics approval from the University of New South Wales Human Research Ethics Committee (reference: HC16558) with access limited only to some authors of this study (GK, AA, TB). Strict security protocol ensured that text mining of the narratives could only be undertaken on site at the NSWPF headquarters and only deidentified extracted outputs could be taken off-site. A hypothetical example of a deidentified narrative is shown in Multimedia Appendix 1.

We randomly selected 100 events containing mental health disorder mentions for our training set, and an extra set of 100 other randomly chosen ones as a development set to optimize the performance of the text-mining system.

Knowledge-Based System Development

Our approach involved the design and implementation of rule-based language expression patterns combined with term dictionaries to identify mentions of mental health disorders in both POIs and victims involved in DV events at the narrative level (see Figure 1 for an overview).

Our text-mining methodology consisted of the following steps (Figure 1):

1. Creation of specific dictionaries relevant to mental health disorders;
2. Design and implementation of rules to capture mental health disorder mentions in text;
3. Standardization and mapping of the extracted mental health disorder mentions into ICD-10; and
4. Elimination of duplicate mentions in each narrative to reach narrative level unification.

Dictionaries

Mentions of several task-specific semantic groups were identified through a set of custom-made dictionaries. The dictionaries were manually tailored by examining the training and development sets for the use of terms describing the associated mental health disorder mentions as well as expressions related to these conditions. For the identification of mental health disorders, we made use of terms and synonyms from the ICD-10, as well as common misspellings (eg, “schitzophrenia,” “aspergus syndrome”) or other indicative descriptive sentences (eg, “abuses alcohol,” “anger issues”) that were present in the event reports. A total of 13 dictionaries were crafted by the first author, GK (Table 1).

Rules

After inspecting the training set, we based our rules on lexical patterns in the text that indicated the presence of a mental health disorder for the POI, the victim, or both in a DV event. In the following example of a lexical pattern observed in a DV event (“accused is suffering from schizophrenia”) to identify a mental health disorder mention (“schizophrenia”), the word “accused” (the POI) is matched via a dictionary that contains variations of terms representing a POI (see Table 1) in which “is suffering from” is a semifrozen expression for the identification of the mental health disorder mention and “schizophrenia” gets a match through a dictionary containing various terms of mental health disorders (official and unofficial ones). The lexical patterns make use of (1) frozen lexical expressions as anchors for certain elements that are built through specific verbs, noun phrases, and prepositions (eg, “defendant suffers from”); and (2) semantic place holders (identifiable through the application of the manually crafted dictionaries (eg, all potential synonyms characterizing an individual as a victim such as “victim,” “vic,” “pn”) suggesting the presence of a mental health disorder.

‎View this figure

Concept enumeration was also implemented because it appears frequently in the training data (eg, “POI has a history of depression, self-harm, and suicidal tendencies [mental health disorder mentions for POI]”). More than one lexical pattern may be matched in an event report and may refer to one or more disorder mentions (that can be duplicates) for the victim, the POI, or both.

For the generation and implementation of the rules, we used General Architecture for Text Engineering (GATE) [23], a text-mining framework for annotating and categorizing text that enables the identification of targeted information. GATE was chosen due to its support for rule-based text-mining approaches. The observed patterns in the text were converted into rules using the Java Annotations Pattern Engine (JAPE), a pattern matching language for GATE. A total of 264 rules were created with 137 for the POI and 127 for the victim, respectively. Figure 2 displays rule examples for the identification of mental health disorders.

The rules use lenient token matching (lowercase or uppercase), such as {Token.string==~”(?i)to”} matches “to”; various dictionaries contain variants, abbreviations, and synonyms of terms of interest, such as (victim), (POI), and (verbs) contain terms for victims, POIs, and verbs in various forms and tenses that describe victims or POIs suffering from a mental health disorder, respectively (see Table 2); ({Token!Lookup.majorType==”negated”})[0,1] will match any token that is not a part of the dictionary “Negated” (which contains negated indicators such as “not”); and the presence of “?” at the end of a rule component suggests its nonconditional nature (ie, it can appear or not in the text).

Mapping of Extracted Mental Health Disorder Mentions to the International Classification of Diseases, Tenth Revision

Since the extracted mental health disorder mentions are highly variable (synonyms, misspellings), any further analysis requires them to be mapped into standard mental health concepts such as the ICD-10 mental and behavioral disorder categories. This was done automatically through a heuristic algorithm that relies on groups of terms that are representative of various ICD-10 categories. If a given mention matched one term from a specific ICD-10 category, then it was mapped to that category.

The mapping was done at four levels (see Multimedia Appendix 2). The first level was the most generic (26 categories), representing the overall type of mental health disorders as specified by ICD-10 (see Textbox 1). The original ICD-10 was expanded using eight customized categories to map mentions for which no direct mapping was obvious. Four of these eight categories involved mentions of psychotropic medications (“medications-antidepressants,” “medications-antianxiety,” “medications-antipsychotics,” “medications-neuroleptics”). For example, in event narratives where a medication class (eg, antidepressant medication) or a brand name (eg, “Zoloft”) was specified, we mapped them to a category called “medications-antidepressants.” The other four categories included “drug prescription abuse,” “substance abuse (unspecified),” “traumatic brain injury,” and “unspecified drug-induced disorder.” Cases in which we recognized that either the victim or the POI had an unknown mental health disorder or an unknown drug-induced mental disorder, were assigned into the categories of “unspecified mental disorder” or “unspecified drug-induced disorder,” respectively.

‎View this figure

Cases in which mental health disorder mentions were more specific, were mapped to lower level ICD-10 categories. The second and third levels of mapping had 62 and 98 categories, respectively. For example, “paranoid schizophrenia” was classified as “paranoid schizophrenia” at the third level according to the ICD-10 schema. Since that mention has a third level mapping in the ICD-10, this indicated that it can also be mapped backward in the second level (“schizophrenia”) and in the first level (“schizophrenia, schizotypal, delusional, and other non-mood psychotic disorders”). The mapping between levels was done manually by an expert in the field of psychiatry (PS).

A fourth level of ICD-10 classification (27 categories) was recorded in some narratives. However, for the purpose of reporting the results in our paper, we merged this level with the third classification level. For example, instead of reporting “other impulse disorders” (third level), we included “intermittent explosive disorder” (fourth level) in the third classification level for the representation of results only. Thus, although there were no explicit mentions of “other impulse disorders” (for example), this mapping did not result in any loss of information regarding mentions of mental health disorders. Table 2 shows some examples of extracted mental health disorder mentions mapped into the ICD-10 schema.

After the mapping of the extracted mental health mentions into the ICD-10 categories, we eliminated any duplicates at the narrative level. The elimination of duplicates led to narrative level unification since unique mentions of mental health disorders for either victims or POIs were present in each event.

Results

Principal Findings

The system was evaluated on a set of 100 unseen, randomly chosen DV events with mentions of mental health disorders. The set was manually inspected and annotated by two domain experts—in DV (CG) and psychiatry (PS)—who identified mentions of mental health disorders for POIs and victims. The interannotator agreement was 90%, calculated as the absolute agreement rate [29], suggesting consistent and reliable annotations by the experts.

Performance of our methodology was evaluated at the narrative level (after the mapping and elimination of any duplicate mental health disorder mentions). We calculated the precision, recall, and F-score for the mental health disorder mentions related to POIs and victims using standard definitions [30] (Multimedia Appendix 3). Table 3 displays the summarized results on the evaluation set, and the performance on the training and development sets.

The F-scores were greater than 80% suggesting reliable results with 87% for mentions related to POIs and 81% for mentions related to victims. Precision ranged from 87% to 97% indicating only a small drop in performance from our development set (1.2%-1.8%). Recall was relatively stable at 79% for the POI (0.3% drop), although for the victim it had a significant drop of 11%, which was expected because our goal was to capture precise mentions of mental health disorders at the narrative level while avoiding noise. It should be noted that victims had fewer mental health disorder mentions at the narrative level when compared to the POIs (36 vs 154, respectively). The false extraction or the nonidentification of a mental health disorder related to a victim affects more the overall extraction performance of the victims than that one of the POIs. Therefore, the values of precision, recall, and F-score for the victims should be taken with caution.

Large-Scale Corpus Application

Given the relatively accurate results of the methodology to reliably identify mental health disorders, we applied it to all 492,393 DV events. The results revealed 77,995 (15.83%, 77,995/492,393) DV events that involved a mental health disorder mention for either the POI, victim, or both. More than three-quarters (76.96%, 60,032/77,995) of DV events included identified mental health disorders related to POIs versus 16.47% (12,852/77,995) for victims. A total of 5111 (6.55%) DV events had mental health disorders for both the victim and POI (Table 4).

Standardized mental health disorder mentions were grouped into the respective ICD-10 categories (including our own customized ones) at three levels: first, second, and third. For example, if an event narrative mentioned “antisocial personality disorder,” it was mapped to three levels (third level: antisocial personality disorder; second: specific personality disorders; first: disorders of adult personality and behavior).

All mental health disorders were mapped to the first level but not all contained sufficient detail to enable them to be allocated to the second and to the third levels (eg, “unspecified mental disorder,” “intellectual disability, unspecified”). The total number of classified mental health disorder mentions at the first level was 103,232, whereas 62,526 mental health disorder mentions contained sufficient information allowing them to be mapped to the second level, with one-third of mentions (32,479, 31.46%) mapped to the third level (Table 4).

At the first level (Table 5), almost one-third of the 81,942 mentions of mental health disorders (32.46%, 26,598) for the POI and one-fifth (22.79%, 4851) for victims had “unspecified mental health disorders” not explicitly recorded in the narratives by the attending police officer(s). “Mood (affective) disorders” (eg, bipolar disorder, depression) had the highest number of mentions among POIs (15,330, 18.71%) and victims (4946, 23.23%) with “mental and behavioral disorders due to psychoactive substance use” (including alcohol abuse) ranking fourth and fifth for both POIs (6790, 8.29%) and victims (1259, 5.91%), respectively. In all, 12.02% of POIs (9848) and 10.45% of victims (2224) had mentions of “behavioral and emotional disorders with their onset usually occurring in childhood and adolescence” (eg, “attention deficit hyperactivity disorders,” “conduct disorders”) being the third and fourth biggest group of disorders in both POIs and victims. Although mentions of “intellectual disabilities” among POIs (1517, 1.85%) were higher in number than in the victims (939, 4.41%), the rates were higher among victims than POIs. Mentions of traumatic brain injury (eg, “the victim has suffered a brain injury due to a car accident”) were reported for 0.84% of POIs and 1.17% victims (688 and 250 mentions, respectively).

In the second level categories (Table 6), “alcohol abuse” was the second highest mental health disorder among POIs (5829, 12.19%) and the fifth highest reported among victims (1180, 8.03%) reinforcing the established link between DV and alcohol use [31-33]. Additionally, there were 644 victims with “dementia, unspecified” (4.38%, 644/14,609) and 546 POI ones (1.14%, 546/47,600).

In the third level categories (Table 7), “bipolar disorder, unspecified” ranked first in mentions for both POIs (5445, 21.59%) and victims (1553, 21.36%) with similar rates. However, it was observed that in POIs that “unspecified behavioral and emotional disorders with onset usually occurring in childhood and adolescence” were second in mentions (4888, 19.38%) unlike with victims that had “anxiety disorder, unspecified” (1459, 20.07%).

Discussion

Overview

Text mining the police event narratives yielded a rich vein of data on the mental health status of victims and POIs involved in DV events that could be useful in policy formulation and prevention that to date has been unavailable. By mining a large cohort of DV police events, we identified many mental health disorder mentions for both the POIs and the victims highlighting the possible role of mental health disorders in DV. Studies have shown that mental illness can increase the likelihood of being in an abusive relationship [3,9], which is consistent with the higher prevalence of mental health disorder mentions among victims (16%).

We aimed to recognize and assign mental health disorders to the POIs and the victims involved in a DV event at the narrative level. Therefore, our rules were focused on precision in order to enable the assignment of the respective disorders to either the POIs or the victims. Many mental health mentions in a single narrative were (varied) mentions of the same disorder for the same individual. This explains the high precision (87.1%-97.5%) when compared to recall (78.5%-79.0%).

Error Analysis

We inspected the evaluation set for sources of false positive and false negative errors in the extraction of mental health disorder mentions. There was a limited number of false positives for either the POI’s or victim’s mental health disorder mentions. In some cases, the lexical patterns used in the rules were ambiguous and assigned a mental health disorder to the wrong person. For example, in the following sentence, “POI has the potential to become violent with the victims due to her alcoholism,” “alcoholism” was extracted incorrectly as a mental health disorder for the victim instead of the POI. In other instances, the specific mention did not refer to an actual mental health problem and the rules incorrectly identified a mental health disorder mention due to the ambiguous nature of a specific situation that mapped to a term in the mental health disorder dictionary (eg, “As a result of the glass on the floor the defendant had cut herself [false positive for POI]”).

In one-third of the false negative cases (33%), the lexical patterns had not been incorporated as they were previously unseen in the training and development sets (eg, “There has been a history of alcohol abuse [false negative: mental health disorder mention for POI] and malicious damage perpetrated by the accused,” “The victim also stated to police that during her time with the POI she was intoxicated as she has an alcohol addiction [false negative: mental health disorder mention for victim]”). Additionally, in almost 40% of false negatives, the rules ignored the correct mental health disorder mention related to either the POI or the victim due to the lack of a semantic anchor specifying the role of the individual (eg, “XXX [name of victim] was admitted to YYY house for depression and anorexia [false negative: mental health disorder mention for victim],” “Her child’s behavior is because of a condition ADHD [false negative: mental health disorder mention for POI]”). In such cases, we chose not to engineer any rules in order to protect the system’s precision and avoid the generation of false positives for potentially other individuals (eg, witnesses, children at risk, friend, neighbor) that could be involved in a DV event and suffering from a mental disorder.

Limitations and Future Work

We designed the rules after inspecting and exploring a relatively small training and development set. However, these sets contained significant numbers of mental health disorder mentions (Table 3). Still, the total number of victim mentions in the evaluation set was significantly lower (almost three times lower) than for POI, which may explain the relatively lower performance for the victim mentions. It is possible that a set focusing only on victim mentions (as opposed to a set that has mental health mentions for either POIs or victims) might have helped to cast a wider net of rules for the identification of the mental health disorders for the victims. Since we based our rules on common lexical patterns, they potentially could be used to process similar types of police-recorded narratives (eg, sexual assaults and other recorded crimes). Although the rules might work on other data, they could require further adjustments both in lexical and dictionary coverage (eg, identification of non-mental health diseases).

We were unaware if the extracted mentions of mental health disorders are valid as they were recorded by police officers who are not expert in mental health and therefore caution is warranted when interpreting the findings. Information on mental health status can be provided to the police by victims, POIs, and witnesses. We plan to examine the veracity of these “informal” mentions of mental health disorders by using formal diagnoses contained in administrative data collections.

We will also expand our set of targeted information from the police narratives in order to assess the characteristics of the POIs and victims for risk groups such as the elderly, those in same sex relationships, and those in carer relationships. The extracted information can be used in designing predictive models to investigate whether we can predict DV recurrent events for groups at risk and inform prevention strategies.

Conclusions

We have designed, implemented, and evaluated a rule-based approach for the extraction of mental health disorders for both POIs and victims involved in DV events as recorded by the NSWPF in event narratives that could not be examined manually on a large scale. Performance was promising, with precision of 87.1% for the victims and 97.5% for the POIs. The results are encouraging and indicate that automated text-mining methods can be used to extract important information from police narratives with reasonable performance. The information extracted from a large-scale set of DV reports allowed us to identify and confirm patterns and links between DV events and mental health disorders. The identified information can be used for further research that aims to assess the characteristics and features of victims and POIs involved in DV events.