Beyond Predictive Algorithms in Child Welfare

Erina Seh-Young Moon,Devansh Saxena,Tegan Maharaj,Shion Guha

2024-02-26

Abstract:Caseworkers in the child welfare (CW) sector use predictive decision-making algorithms built on risk assessment (RA) data to guide and support CW decisions. Researchers have highlighted that RAs can contain biased signals which flatten CW case complexities and that the algorithms may benefit from incorporating contextually rich case narratives, i.e. - casenotes written by caseworkers. To investigate this hypothesized improvement, we quantitatively deconstructed two commonly used RAs from a United States CW agency. We trained classifier models to compare the predictive validity of RAs with and without casenote narratives and applied computational text analysis on casenotes to highlight topics uncovered in the casenotes. Our study finds that common risk metrics used to assess families and build CWS predictive risk models (PRMs) are unable to predict discharge outcomes for children who are not reunified with their birth parent(s). We also find that although casenotes cannot predict discharge outcomes, they contain contextual case signals. Given the lack of predictive validity of RA scores and casenotes, we propose moving beyond quantitative risk assessments for public sector algorithms and towards using contextual sources of information such as narratives to study public sociotechnical systems.

Machine Learning,Computers and Society,Human-Computer Interaction

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are the effectiveness and fairness issues of Predictive Risk Models (PRMs) in the current child welfare system. Specifically, the researchers are concerned with:

1. **The impact of Predictive Risk Assessments (RAs) on case outcomes**: Researchers analyzed the effectiveness of common risk assessment tools (such as AAPI and NCFAS) in predicting the outcome of whether children can be reunited with their biological parents. The study found that these traditional risk assessment tools perform poorly in predicting non - reunification (i.e., children cannot be reunited with their biological parents), indicating that the existing risk assessment tools may not fully capture the key factors affecting case outcomes.

2. **The impact of case narratives on decision - making and outcomes in child welfare cases**: The researchers also explored how case narratives (i.e., case notes recorded by social workers) can provide information different from risk assessment scores and how this information affects decision - making and outcomes in child welfare cases. Through text analysis, the researchers found that case narratives can reveal situational details among individuals, and these details may affect the development trajectory and final outcome of the case.

3. **How to better support social workers in child welfare cases**: Based on the above research findings, the researchers proposed suggestions for improving the application of algorithms in the child welfare system, emphasizing that we should go beyond a single quantitative risk assessment and make more use of situational information (such as case narratives) to understand and handle complex problems in public social - technical systems.

Overall, through a comparative analysis of existing risk assessment tools and case narratives, this paper aims to reveal the limitations of predictive models in the current child welfare system and propose improvement directions, in the hope of better supporting social workers to make more just and effective decisions.