Advertisement
OA08 MOVING BEYOND NEW DRUGS: WHAT ELSE MATTERS? THURSDAY, SEPTEMBER 09, 2021 - 09:30-10:30| Volume 16, ISSUE 10, SUPPLEMENT , S861-S862, October 2021

OA08.04 Validation of Scalable, Automated Data Extraction in an Advanced Lung Cancer Patient Population

      Introduction

      Manual extraction from electronic health records (EHRs) is currently the standard approach for accessing real-world healthcare data but can be time consuming and challenging to maintain over time. Automated data extraction using natural language processing (NLP) is emerging as a viable method of data extraction from structured and unstructured fields of EHRs. While speed of NLP-based data extraction is established, some question the validity of the extracted data. This study compares the accuracy of, and concordance between, manual and NLP-extracted data from EHRs of patients with advanced lung cancer (aLC).

      Methods

      EHRs of 1209 patients with aLC were screened using the AI engine, DARWEN™, to identify a subset of 333 patients diagnosed and treated with systemic therapy at Princess Margaret Cancer Centre in Toronto between January 2015 and December 2017. Full feature models were run on all 333 patients to extract data from EHRs, from which 100 patients were randomly selected for manual data extraction by two trained abstractors to validate against NLP-extracted data. An expert adjudicator reviewed inconsistencies between manual and NLP-extracted results and was referenced as the gold standard when calculating accuracy and concordance.

      Results

      NLP-extracted data from EHRs proved to be accurate and concordant with manual extraction methods (Table 1). Features with lower syntactic and semantic variation such as patient demographics (i.e., age and sex), characteristics (i.e., histologic subtype and comorbid conditions), and treatment details were reported with high accuracy and concordance. These tend to be the cases where manual reviewers would agree. Conversely, features with richer syntactic and semantic variation requiring deeper clinical interpretation had slightly lower accuracy by NLP extraction and, typically, manual review. By nature of the varying ways that biomarker testing and reporting is documented, extracting this data can be challenging. While NLP detection of biomarker testing was highly accurate and concordant, detection of results was more variable. NLP out-performed manual extraction in identifying metastatic sites with the exception of lung and lymph node metastases, which was due to analogous terms used in radiology reports that were not applied to variable definitions used to train DARWEN™.
      TableAccuracy and concordance between manual and NLP data extraction.
      Accuracy (%)Concordance (%)
      NLPManual
      Date of birth1009999
      Sex100100100
      Date of Stage IV diagnosis (+/- 30 days)94.083.077.0
      ECOG PS at Stage IV diagnosis93.078.071.0
      Smoking status88.094.082.0
      Histologic subtype98.098.096.0
      First line treatment type95.0-99.096.0-10092.0-99.0
      Treatment type (Any line)94.0-99.084.0-98.083.0-96.0
      Biomarker Testing Performed98.0-99.097.0-10096.0-98.0
      Biomarker Status (Positive or Negative)86.2-10094.7-10086.2-100
      Metastatic Sites of Disease66.0-99.071.0-10058.0-99.0
      Immunosuppressive medications80.0-10086.0-10076.0-100
      Comorbidities96.0-10096.0-10093.0-100

      Conclusion

      The use of NLP technology in oncology provides opportunity for real-world evidence studies at a larger scale than ever before. NLP was not only faster than manual extraction but, for many features, was also more accurate than a traditional manual approach, demonstrating the advances of modern NLP techniques as a scalable alternative to manual extraction.