Implement human-in-the-loop systems to build trust

Many variables useful for clinical research — such as patient disease state and treatment regimens — are trapped in free-text clinical notes. Structuring them typically involves a tedious manual search through long clinical timelines. Natural language processing systems present an opportunity for automating this workflow, but algorithms still struggle with the most complex patient cases, which are best deferred to experts. This paper presents a framework that automatically structures simple patient cases, but when needed, iteratively requests human input in the form of a single note label that would most reduce model uncertainty. The method was tested on two tasks from a cohort of oncology patients: identifying the date of metastasis onset and oral therapy start. Compared to standard search heuristics, it reduces 80% of model errors with less than 15% of the manual annotation effort.