Research and Applications

Feasibility of capturing real-world data from health infor-

mation technology systems at multiple centers to assess

cardiac ablation device outcomes: A fit-for-purpose infor-

matics analysis report

Guoqian Jiang,1

Sanket S. Dhruva ,2

Jiajing Chen,3

Wade L. Schulz,4,5

Amit A. Doshi,6

Peter A. Noseworthy,7

Shumin Zhang,8

Yue Yu,9

H. Patrick Young ,10

Eric Brandt,3 Keondae R. Ervin,11 Nilay D. Shah,12 Joseph S. Ross,5,10 Paul Coplan,13,14

and Joseph P. Drozda Jr 3

1Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA, 2School of Medicine, University

of California, San Francisco, and San Francisco Veterans Affairs Medical Center, San Francisco, California, USA, 3Mercy Re-

search, Mercy, Chesterfield, Missouri, USA, 4Department of Laboratory Medicine, Yale University School of Medicine, New Ha-

ven, Connecticut, USA, 5Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut,

USA, 6Mercy Clinic, Mercy, St. Louis, Missouri, USA, 7Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minne-

sota, USA, 8Medical Device Epidemiology and Real-World Data Science, Office of the Chief Medical Officer, Johnson & Johnson,

New Brunswick, New Jersey, USA, 9Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota, USA, 10De-

partment of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA, 11National Evaluation System for Health

Technology Coordinating Center, Medical Device Innovation Consortium, Arlington, Virginia, USA, 12Robert D. and Patricia E. Kern

Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA, 13Medical Device Epidemiology and

RWD Science, Office of the Chief Medical Officer, Johnson & Johnson, New Brunswick, New Jersey, USA, and 14Perelman

School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA

Corresponding Author: Guoqian Jiang, Department of Artificial Intelligence and Informatics, Mayo Clinic, 200 First Street,

SW, Rochester, MN 55905, USA ([email protected])

Received 25 January 2021; Revised 22 April 2021; Editorial Decision 25 May 2021; Accepted 28 May 2021

ABSTRACT

Objective: The study sought to conduct an informatics analysis on the National Evaluation System for Health

Technology Coordinating Center test case of cardiac ablation catheters and to demonstrate the role of informat-

ics approaches in the feasibility assessment of capturing real-world data using unique device identifiers (UDIs)

that are fit for purpose for label extensions for 2 cardiac ablation catheters from the electronic health records

and other health information technology systems in a multicenter evaluation.

Materials and Methods: We focused on data capture and transformation and data quality maturity model speci-

fied in the National Evaluation System for Health Technology Coordinating Center data quality framework. The

informatics analysis included 4 elements: the use of UDIs for identifying device exposure data, the use of stan-

dardized codes for defining computable phenotypes, the use of natural language processing for capturing un-

structured data elements from clinical data systems, and the use of common data models for standardizing

data collection and analyses.

Results: We found that, with the UDI implementation at 3 health systems, the target device exposure data could

be effectively identified, particularly for brand-specific devices. Computable phenotypes for study outcomes

VC The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association.

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/),

which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact

[email protected] 2241

Journal of the American Medical Informatics Association, 28(10), 2021, 2241–2250

doi: 10.1093/jamia/ocab117

Advance Access Publication Date: 27 July 2021

Research and Applications

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

could be defined using codes; however, ablation registries, natural language processing tools, and chart

reviews were required for validating data quality of the phenotypes. The common data model implementation

status varied across sites. The maturity level of the key informatics technologies was highly aligned with the

data quality maturity model.

Conclusions: We demonstrated that the informatics approaches can be feasibly used to capture safety and ef-

fectiveness outcomes in real-world data for use in medical device studies supporting label extensions.

Key words: informatics analysis, medical device evaluation, cardiac ablation catheters, real-world evidence, RWE, unique device

identifier, UDI

INTRODUCTION

With the increasing availability of digital health data and wide

adoption of electronic health records (EHRs), there is an opportu-

nity to capture and analyze real-world data (RWD) to generate real-

world evidence (RWE) from health information technology (IT) sys-

tems for evaluations of medical product safety and effectiveness.1

Under the 21st Century Cures Act, signed into law in 2016,2 the

Food and Drug Administration (FDA) has been tasked with develop-

ing a program to evaluate the use of RWE to support approval of ex-

panded indications for approved medical products or to meet

postmarket surveillance requirements. In the FDA guidance docu-

ment focused on medical devices, RWE is defined as the clinical evi-

dence regarding the usage and potential benefits and risks of a

medical product derived from the analysis of RWD.3 In particular,

the FDA and the medical device evaluation community have envi-

sioned a system that can not only promote patient safety through

earlier detection of safety signals,4 but also generate, synthesize, and

analyze evidence on real-world performance and patient outcomes

in situations in which clinical trials are not feasible.5

In this context, the FDA created the National Evaluation System

for health Technology Coordinating Center (NESTcc),6 which seeks

to support the sustainable generation and use of timely, reliable, and

cost-effective RWE throughout the medical device life cycle, using

RWD that meet robust methodological standards. As part of the

funding commitment to NESTcc from the FDA and the Medical De-

vice User Fee Amendment, some of the pilot projects (or test cases)

need to focus on medical devices that are either in the premarket ap-

proval (PMA) or 510k phase of the total product life cycle. Test

cases that can generate regulatory grade data that are fit for purpose

and can support a regulatory submission will prove the strength and

reliability of RWD as an effective and alternative method to tradi-

tional clinical trials.7 The intent of NESTcc is to provide accurate

and detailed information regarding medical devices including the

identification of devices that may result in adverse events and act as

a neutral conduit reporting on device performance in clinical prac-

tice. Notably, the NESTcc has released a Data Quality Framework8

developed by its Data Quality Working Committee to be used by all

stakeholders across the NESTcc medical device ecosystem, laying

out the foundation for the capture and use of high-quality data for

evaluation of medical devices. The framework focuses on the use of

RWD generated in routine clinical care, instead of data collected

specifically for research or evaluation purposes.

The goal of this article is to conduct an analysis to demonstrate

the role of informatics approaches in data capture and transforma-

tion in a NESTcc test case, helping to determine if these data are of

sufficient relevance, reliability, and quality to generate evidence

evaluating the safety and effectiveness of target devices. The NESTcc

test case study aimed to explore the feasibility of capturing RWD

from the EHRs and other health IT systems of 3 NESTcc Network

Collaborators (Mercy Health, Mayo Clinic, and Yale New Haven

Hospital [YNHH]), including data from hospital EHRs, and deter-

mining whether RWD are fit for purpose for postmarket evaluation

of outcomes when 2 ablation catheters were used in new popula-

tions and to support submissions to the FDA for indication expan-

sion. The study was proposed to the NESTcc by Johnson & Johnson

(New Brunswick, NJ), with the objective of evaluating the safety

and effectiveness of 2 cardiac ablation catheters when used in rou-

tine clinical practice. The specific catheters of interest are the Ther-

moCool Smarttouch catheters, initially approved by the FDA in

February 2014, and the ThermoCool Smarttouch Surround Flow

catheters, initially approved by the FDA in August 2016. The hy-

potheses of the NESTcc test case are whether the safety and effec-

tiveness of versions of ThermoCool catheters that do not have a

labeled indication for ventricular tachycardia (VT) are noninferior

to ThermoCool catheters that already have such an FDA approved

indication, and similarly versions of ThermoCool catheters that do

not have labeled indications for persistent atrial fibrillation (AF) are

noninferior to those that do.

Background

Unique device identifiers for device exposure data capture

Data standardization is key for documentation of and linking medi-

cal device identification information to diverse data sources.9,10 The

FDA has recognized the need to improve the tracking of medical de-

vice safety and performance, with implementation of unique device

identifiers (UDIs) in electronic health information as a key strat-

egy.11 Notably, the FDA initiated the regulation of the UDI imple-

mentation and established a Global Unique Device Identification

Database 12 for making unique medical device identification possi-

ble. By September 24, 2018, all Class III and Class II devices were

required to bear a permanent UDI. Meanwhile, a number of demon-

stration projects have demonstrated the feasibility of using informat-

ics technology to build a medical device evaluation system and to

identify keys to success and challenges of achieving targeted

goals.10,11,13,14 These projects served as the proof of concept that

UDIs can be used as the index key to combine device and clinical

data in a database useful for device evaluation.

Common data models for standardized data capture

and analyticsA variety of data models have been developed to provide a standard-

ized approach to store and organize clinical research data.15 These

approaches often support query federation, which is the ability to

run a standardized query within separate remote data repositories

and facilitate the conduct of distributed data analyses where each

healthcare system keeps its information, yet a standardized analysis

can be conducted across multiple healthcare systems. Examples of

2242 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

these models include the FDA Sentinel Common Data Model

(CDM),16 the Observational Medical Outcomes Partnership

(OMOP) CDM,17 the National Patient-Centered Research Net-

works (PCORnet) CDM,18 the Informatics for Integrating Biology

and the Bedside (i2b2) Star Schema,19 and the Accrual to Clinical

Trials ACT model.20 However, the applicability of CDMs to medi-

cal device studies, particularly whether there is sufficient granularity

of device identifiers and aggregate codes for procedures, is an unan-

swered question. We described and analyzed the CDM implementa-

tion status at each site and assessed their potential contributions to

the data quality maturity model.

NESTcc data quality frameworkThe NESTcc data quality framework6 focuses primarily on the use

of EHR data in the clinical care setting, and is composed of 5 sec-

tions which cover data governance, characteristics, capture and

transformation, curation, and the NESTcc data quality maturity

model. The NESTcc data quality maturity model addresses the vary-

ing stages of an organization’s capacity to support these domains,

which allows collaborators to indicate progress toward achieving

optimal data quality. Supplementary Table S1 shows the description

and core principles of the 5 sections in the framework.

MATERIALS AND METHODS

In this study, we analyzed the successes and challenges of acquiring

RWD that are fit for purpose for evaluation of outcomes from 2 ab-

lation catheters, focusing on data capture and transformation and

the data quality maturity model (as defined in the NESTcc data

quality framework) from an informatics analysis perspective, while

also highlighting differences between data quality and fit for pur-

pose in RWD studies of medical devices. The informatics analysis in-

cluded the use of UDIs for identifying device exposure data, the use

of standardized codes (eg, International Classification of Diseases

[ICD], Current Procedural Terminology [CPT], RxNorm) to define

computable phenotypes that could identify study cohorts, covariates

and outcome endpoints accurately, the use of natural language proc-

essing (NLP) for capturing unstructured data elements from clinical

data systems, and the use of CDMs for standardizing data collection

and analyses (Supplementary Table S2).

Use of UDIs for identifying device exposure dataWe identified a typical process (see Figure 1) for the use of UDIs for

collecting device exposure data and described it as follows.

1. Identifying UDIs for target devices. In this study, the UDIs and de-

vice catalogue numbers of the target devices were identified and

provided by Johnson & Johnson. The FDA Global Unique Device

Identification Database was used for the UDI identification. The

rationale for relying on UDIs is that target devices are Thermo-

Cool devices, which are brand specific, and the hypotheses to be

tested involved comparing 2 different versions of the ThermoCool

catheters (ie, those with vs those without the target label). A col-

lection of UDIs for each of brand-specific devices was used to cap-

ture related device data.

2. Locating UDIs documented in the health IT systems in each site.

At the Mayo Clinic, UDIs are documented in different health IT

systems. As Epic EHR system (Epic Systems, Verona, WI) was in-

troduced as of May 2018, the UDI-linked device data after May

2018 are documented in the Supplyþ (Cardinal Health, Dublin,OH) and Plummer (Epic), which have worked together to stan-

dardize multiple clinical and business processes to improve effi-

ciency and optimize inventory. Supplyþ (Cardinal Health) is anenterprise-wide, integrated inventory management system to im-

plement standardized surgical and procedure inventory manage-

ment. Historical device data dating back to January 2014 are

documented in the Mayo Clinic supply chain management system

known as SIMS. SIMS was a Mayo-designed and supported sys-

tem to improve surgical case management and Mayo Group Prac-

tices across Mayo enterprise. At Mercy, manufacturer numbers

and UDIs were used to extract the devices of interest from

Mercy’s OptiFlex (Omnicell, Mountain View, CA) point of

care barcode scanning system for devices used after 2016—the

year this system was installed. To pull device-related data prior

to 2016 (before OptiFlex was installed), Mercy identified proce-

Figure 1. A data flow diagram illustrating a typical process for the use of unique device identifiers (UDIs) for collecting device exposure data and clinical data.

EHR: electronic health record; GUDID: Global Unique Device Identification Database; IT: information technology; J&J: Johnson & Johnson; PCORnet: Patient-Cen-

tered Research Network; YNHH: Yale New Haven Hospital.

Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10 2243

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

dures linked to patient information using HCPCS (Healthcare

Common Procedure Coding System) codes and Mercy-specific

charge codes for device billing. At YNHH, device data elements

were captured within the QSightSM (Owens and Minor, Mechan-

icsville, VA) inventory management system, in use since October

2017, during the normal course of clinical care and administrative

activities.

3. Identifying patient cohorts with device exposure using the UDIs.

At the Mayo Clinic, the UDI-linked device data are documented

in patient level, and a unique patient clinic number (that is used

across enterprise health IT systems, including patient medical

records) can be retrieved with this linkage for patient cohort iden-

tification. At Mercy, the device data were joined with transaction

data to obtain patient and encounter information for each in-

stance of device use. At YNHH, device-related records extracted

from QSightSM were linked via transaction data to procedure en-

counter records within the Epic EHR to verify the specific use of

the device and to link with the clinical record.

4. Linking UDIs with clinical data (eg, procedures of interest) in

EHR systems. At the Mayo Clinic, the Unified Data Platform

(UDP) has been implemented to provide practical data solutions

and creates a combined view of multiple heterogeneous EHR data

sources (including Epic) through effective data orchestration,

along with a number of data marts based on CDMs. The UDP

serves as a data warehouse that contains millions of patient data

points for the support of both clinical practice and research. The

UDP is updated in real time, data are cleaned, and many of the

medical elements are matched with standard medical terminolo-

gies such as ICD and SNOMED CT (Systematized Nomenclature

of Medicine Clinical Terms) codes. The UDP was used to collect

device-related EHR data. We used all of the patient clinic num-

bers of the device users as the identifiers to extract the data from

UDP. At Mercy, device data were joined to ablation procedure

data based on patient ID and the dates of procedure in order to

examine the device usage during procedures. Mercy utilized Epic

Clarity to research the presence of the various diagnosis and pro-

cedural codes relevant to the study. At YNHH, clinical data from

the EHR are populated by a vendor-provided extract, transform,

and load (ETL) process from a nonrelational model into a rela-

tional model (Clarity), followed by a second vendor-provided

ETL to create a clinical data warehouse (Caboodle). Data were

transformed from the Caboodle data warehouse into the PCOR-

net common data model and analyzed within the YNHH data an-

alytics platform.21 The Caboodle-PCORnet ETL process removes

test patients and also standardizes the representation of certain

elements such as dates and encounter types.

Use of standard codes for defining computable

phenotypesFor the NESTcc test case study, standardized codes were used to de-

fine the algorithms to compute phenotypes that could identify target

indications (ie, either VT or persistent AF using ICD codes), proce-

dures of interest (ie, cardiac ablation for either VT or persistent AF

using CPT codes), outcome endpoints (eg, ischemic stroke, acute

heart failure, and rehospitalization with arrhythmia using ICD

codes), and covariates of interest (eg, therapeutic drugs using

RxNorm codes). Supplementary Table S3 provides a list of pheno-

type definitions using standardized codes. These standardized codes

serve as a common language that would reduce ambiguous interpre-

tation of the algorithm definitions across sites.

Data quality validationData quality validation using registry data and chart review is an im-

portant component in the study design. In particular, it is well recog-

nized in the research community that the accuracy of phenotype

definitions based on simple ICD codes is not optimal, except for

markers of healthcare utilization,22 such that these codes cannot be

used as a “gold standard.” We found that clinical registry data (if

available) constitute a very valuable resource to enable efficient data

quality check, if the variables of interest are similar between the

real-world study and registry. The Mayo Clinic utilized this internal

registry as a data validation source, and the AF cases were classified

as paroxysmal, persistent, or longstanding persistent by physicians

through a manual review process (note that the physician-based con-

firmation was done as part of registry-building process, not as a sep-

arate effort for this research study).

Use of NLP for unstructured clinical dataIn the NESTcc test case, the NLP technology is used in the following

aspects.

First, Mercy used a previously validated NLP algorithm to vali-

date AF patient phenotypes. As Mercy does not participate in an AF

registry, an NLP tool and validated dataset were used as the gold

standard for validation of the extracted data. Specifically, Lingua-

matics (IQVIA, Danbury, CT) software was utilized within Mercy’s

Hadoop warehouse for NLP. This tool was built and validated on a

group of patients who were diagnosed with arrhythmia and stroke

for a previous Johnson & Johnson project. All EHR notes of those

patients were queried and validated for their AF diagnoses. We used

this group of patients as our test case to validate ICD codes for the

following 3 AF types: paroxysmal, persistent, and chronic. The diag-

noses defined by the previously developed NLP tool served as the

gold standard for AF subtypes for this project.

Second, left ventricular ejection fraction (LVEF) is one of the

covariates of interest to identify. NLP-based methods were used to

extract LVEF from echocardiogram reports when it is not available

in a structured format.

Use of CDMs for standardizing data collection and

analysesIn the NESTcc test case study, we realized that there would be of

great value if we could standardize the data collection process across

sites, and the infrastructure of CDM-based health IT systems makes

this possible. We investigated the CDM implementation status (ie,

whether a prevailing CDM such as i2b2, PCORnet, OMOP, Senti-

nel, and Fast Healthcare Interoperability Resources [FHIR] has been

implemented) in the 3 health systems.

Conducting a maturity level analysisWe also conducted a maturity level analysis on the key informatics

technologies used in data capture and transformation, highlighting

current maturity level (ie, conceptual, reactive, structured, complete,

and advanced) of the key technologies and their correlations with

the NESTcc data quality domains (ie, consistency, completeness,

CDM, accuracy, and automation) as defined in the NESTcc data

quality framework. Two representatives from each site assessed the

maturity level of the 4 key technologies for their respective system

and assigned the maturity level scores.

2244 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

RESULTS

Initial population and device exposure dataUsing standard codes (Supplementary Table S3), we were able to re-

trieve initial populations of AF and VT patients and their procedure

of interest. A total of 337 181 AF patients were identified, including

27 865 patients with persistent AF, and a total of 59 425 VT

patients were identified, including 39 092 patients with ischemic VT

from 3 sites (Table 1). In addition, a total of 8676 cardiac catheter

ablation procedures were identified for AF population and 1865 ab-

lation procedures for VT population (data not shown). Using UDIs,

we were able to break down device counts for target population by

brand-specific device subtypes (Table 2). Notably, no analyses of

safety and effectiveness outcomes by catheter type were conducted

in this feasibility study to avoid influencing the second stage study

that will test the hypotheses.

Data quality validationTable 3 shows cross-validation results for the AF subtype cases iden-

tified using ICD codes and registry data at Mayo Clinic. Positive

predictive values (PPVs) were calculated as results. For AF cases

identified by ICD–Ninth Revision code 427.31, we identified 304

cases of paroxysmal AF and 427 cases of persistent AF from the

Mayo Clinic registry, indicating that registry data provide specific

subtypes. For 496 cases of paroxysmal AF identified by ICD–Tenth

Revision (ICD-10) code I48.0, a total of 260 (PPV ¼ 52.4%) wereconfirmed as true cases from the registry. For 176 cases of persistent

AF identified by ICD-10 code I48.1, 124 cases (PPV¼70.5%) wereconfirmed as true persistent AF cases from the registry. The results

indicated that the case identification algorithms based on ICD-10

codes at Mayo Clinic are not optimal and that the clinical registry

had great value in validating the case identification algorithms,

though the accuracy of the registry itself has not been validated (and

uses retrospective diagnosis based on chart review by a nurse clini-

cian to determine AF type). Note that Mercy used a previously vali-

dated NLP algorithm to validate AF patient phenotypes (see details

in the following section), and YNHH participates in the National

Cardiovascular Data Registry AF Ablation Registry, which is an-

other registry resource used for AF data quality validation in the

NESTcc test case study.

Table 1. AF and VT patient counts by disease subtype (Note that 1 patient may have more than 1 diagnosis)

AF

Paroxysmal

AF

Persistent

AF

Permanent

AF

Unspecified

and other AF VT

Ischemic

VT

Nonischemic

VT

Mercy (01/01/2014-02/20/2020) 169 062 88 387 11 898 31 753 145 903 24 401 16 379 8022

Mayo Clinic (01/01/2014-12/31/2019) 133 298 60 999 12 372 21 800 98 839 20 920 13 114 7806

YNHH (02/01/2013-08/13/2019) 54 821 15 007 3594 14 961 21 259 14 104 9599 4505

Total 357 181 164 393 27 864 68 514 266 001 59 425 39 092 20 333

AF: atrial fibrillation; VT: ventricular tachycardia; YNHH: Yale New Haven Hospital.

Table 2. Device counts for AF patients by brand-specific subtypes of interest

Paroxysmal AF Persistent AF

ThermoCool ST ThermoCool STSF

ThermoCool ST (treatment

catheter)

ThermoCool STSF (control

catheter)

Mercy (01/01/2014-02/20/

2020)

377 408 251 492

Mayo Clinic (01/01/2014-

12/31/2019)

625 248 233 100

YNHH (02/01/2013-08/13/

2019)

96 135 65 115

Total 1098 791 549 707

AF: atrial fibrillation; ST: Smarttouch; STSF: Smarttouch Surround Flow; YNHH: Yale New Haven Hospital.

Table 3. Validation of the AF subtype cases identified using ICD codes against the prospective nurse-abstracted registry data at Mayo Clinic

Code Vocabulary Term Paroxysmal AF in registry Persistent AF in registry Total

427.31 ICD-9 AF 304 (41.6) 427 (58.4) 731 (100)

I48.0 ICD-10 Paroxysmal AF 260 (52.4) 236 (47.6) 496 (100)

I48.1 ICD-10 Persistent AF 52 (29.5) 124 (70.5) 176 (100)

I48.2 ICD-10 Chronic AF 4 (19.0) 17 (81.0) 21 (100)

I48.91 ICD-10 Unspecified AF 251 (41.8) 349 (58.2) 600 (100)

Values are n (%). ICD-9 codes were used prior to October 2015 and ICD-10 codes thereafter.

AF: atrial fibrillation; ICD: International Classification of Diseases; ICD-9: International Classification of Diseases–Ninth Revision; ICD-10: International Clas-

sification of Diseases–Tenth Revision;

Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10 2245

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

For outcome endpoint validation, a manual chart review process

was used to confirm target cases. Owing to time and funding restric-

tions, the consensus was to focus on 3 primary outcome endpoints:

ischemic stroke, acute heart failure, and rehospitalization with ar-

rhythmia. We started the algorithms based on codes obtained from

a published literature review and refined this further using consensus

clinician review from several practicing electrophysiology physi-

cians, data scientists, epidemiologists, and other team members.

Once the code algorithms were finalized, we identified the patient

counts for each of the 3 outcome endpoints. We also used the full

algorithms that restrict patients within 30 days of postablation (ie, a

time window used to identify outcomes) and identified a subset

of patients. We then randomly selected 25 cases from the results

of the full algorithm for each of the 3 outcome endpoints. Clinicians

at each site performed manual chart review to evaluate the clinical

outcomes’ algorithms. PPVs were calculated as results (data not

shown).

NLP for unstructured dataMercy does not participate in an AF registry; therefore, a NLP tool

was used on a group of patients who were diagnosed with arrythmia

and stroke using a collection of ICD codes for a previous Johnson &

Johnson project. Table 4 shows the summary of predictive values

for ICD codes by AF type.

In addition, we found that LVEF is not readily available in a

structured format. At Mercy, LVEF was extracted using an NLP

method and at the Mayo Clinic, it was extracted from echocardio-

gram reports using an open-source NLP program.23 Yale was able

to capture ejection fractions available in structured fields.

CDM implementation statusTable 5 shows the CDM implementation status of 3 health systems:

the Mayo Clinic, Mercy, and YNNH. Both the Mayo Clinic and

YNNH have majority of CDMs (i2b2, PCORnet, OMOP, and

FHIR) implemented, whereas Mercy have Sentinel CDM and FHIR

implemented. This indicates that the CDM implementation varies

across 3 sites.

Maturity level analysisFigure 2 shows the maturity level analysis results for the key tech-

nologies used in the data capture and transformation.

By design, the maturity model can help researchers identify

weaknesses, in terms of the ability to capture data consistently and

completely, to represent data via CDMs, to validate the accuracy of

data, and to then use the data through automated queries. These are

examples of key processes that drive data quality.

A summary of the informatics analysisThe successes and challenges of the informatics analysis are de-

scribed in detail in Table 6.

DISCUSSION

Use of UDIsWe found that, with the UDIs implemented in the health IT systems,

the target device exposure data can be effectively identified, particu-

larly for brand-specific devices as targeted in the NESTcc case study.

For example, when another device was identified as a potential com-

parator for a VT ablation study, we needed to assess initial counts

of its usage to inform availability of comparator or control data for

a potential label extension study for the catheter of interest. The

project team at the Mayo Clinic, Mercy, and Johnson & Johnson

was able to identify device UDIs and use them to get initial counts of

its usage in a short turnaround.

One of key challenges is that the UDI implementation is uneven

across sites. For example, Mercy implemented UDIs in its health IT

systems in 2016. As mentioned previously, to pull device-related

data prior to 2016 (before OptiFlex was installed), Mercy identified

procedures linked to patient information using HCPCS codes and

Mercy-specific charge codes for device billing. These codes were

reviewed and confirmed by Johnson & Johnson before data extrac-

tion. Device data were joined together with those UDI-linked data

to create a final dataset after duplicates were removed. Supplemen-

tary Table S4 shows the UDI implementation status of 3 health sys-

tems (Mayo Clinic, Mercy, and YNHH).

Use of standardized codesWe found that coming to an agreement on standard computable co-

variate and outcome definitions took more time than we foresaw. In

particular, this consensus process involved input from clinicians to

ensure algorithm definitions were clinically meaningful and precise.

For example, to define cardiac ablation as a procedure of interest,

we used CPT procedure codes. The initial list of the CPT codes in-

cluded 93650 (atrioventricular node ablation), and through discus-

sion with the clinical group, the CPT code was questioned as not

Table 4. Summary of predictive values for ICD codes by AF type at Mercy as compared with an natural language processing tool

Paroxysmal AF (%) Persistent AF (%) Chronic AF (%)

Sensitivity (recall) 82.80 62.70 74.80

Specificity 86.50 95.90 90.80

Positive predictive value (precision) 94.20 80.40 87.60

Negative predictive value 65.70 90.60 80.70

AF: atrial fibrillation; ICD: International Classification of Diseases.

Table 5. The CDM implementation status of 3 health systems:

Mayo Clinic, Mercy, and YNNH

CDM Implementation Status Mayo Clinic Mercy YNHH

i2b2 Star Schema X X

PCORnet CDM X X

OMOP CDM X (in progress) X

Sentinel CDM X

FHIR X X X

CDM: common data model; CPT: Current Procedural Terminology; FHIR:

Fast Healthcare Interoperability Resources; i2b2: Informatics for Integrating

Biology and the Bedside; OMOP: Observational Medical Outcomes Partner-

ship; PCORnet: Patient-Centered Research Network; YNHH: Yale New Ha-

ven Hospital.

2246 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

representing AF ablation and consensus was achieved to remove the

CPT code 93650 from the definition list.

Use of NLP technologyThe use of NLP in this study was limited to a number of specific

tasks. The main challenges for using NLP technology include (1) re-

quiring advanced expertise in using existing NLP tools or developing

fit-for-purpose NLP probes to search clinical text and notes, (2) lack

of NLP solutions that are portable across sites, and (3) challenges in

validating NLP probes. In addition, we also noticed that NLP, in

general, has its own challenges including accuracy and maintenance

issues, and potential for accidental privacy breaches.24,25

Use of CDMsThe advantages of using CDM-based research repositories are de-

scribed in Table 6. Note that if there is no CDM, the researcher still

must understand the source data and convert it to a usable form that

is consistent across the multiple healthcare systems participating in

the study, so they have similar work with or without the data model.

But a CDM can provide significant benefit when provided by a coor-

dinating center for use by individual researchers, helping make lan-

guage consistent related to queries developed, thus saving the

investigator significant work. One of the main challenges is that the

implementation of CDMs often requires significant time and effort

to extract and convert data from clinical data information systems,

such as EHRs and laboratory information systems, to the format re-

quired to load into each CDM. Fortunately, this challenge can be al-

leviated with the advancement of mature ETL technology involved

in the CDM implementation. Moreover, the processing and trans-

formation of data into CDMs provides a logical pathway for en-

abling standardized analyses that are portable and consistent across

sites, the benefit of which can help make a decision for the invest-

ment on the CDM implementation.

Data quality vs fit for purposeFit for purpose is defined as a conclusion that the level of validation

associated with a medical product tool is sufficient to support its

context of use.26 The NESTcc Data Quality Framework6 has made

clear that useful data must be both reliable (high quality) and rele-

vant (fit for purpose) across a broad and representative population

based on the experimental, approved, or real-world use of a medical

device.

The nature of capturing RWD from health IT systems for device

evaluation is the secondary use of the data for a research purpose. The

underlying data can have quality issues (eg, typed in wrong value,

only captured a portion of UDI when a standard operating procedure

calls for identifier capture in its entirety, manual data entry instead of

barcode scanning). However, it is important to separate those issues

from data that may not be present because the data weren’t needed

(or needed in structured formats) for direct clinical care.

In addition, lacking a gold standard, the reports of detected data

quality rely heavily on the quality of the evaluation plan. We found

that different modalities such as ablation registries, NLP tools and

chart reviews were required for validating data quality of the pheno-

types.

Clinical aspects of the NESTcc test caseThe focus of this article is on the informatics approaches used in the

NESTcc test case. A separate clinical article reports on the feasibility

of using the informatics approaches to capture RWD from the

EHRs and other health IT systems at 3 health systems that are fit for

purpose for postmarket evaluation of outcomes for label extensions

of 2 cardiac ablation catheters. In brief, such evaluation was prelimi-

narily determined feasible based on (1) the finding of adequate sam-

ple size of device of interest and control device use; (2) the presence

of sufficient in-person encounter follow-up data (up to 1 year); (3)

the availability of adequate data quality validation modalities, in-

cluding clinician chart reviews; and (4) the potential use of CDMs

for distributed data analytics. Reporting the detailed findings of the

project’s clinical aspects and feasibility assessments is beyond the

scope of the article.

CONCLUSION

We demonstrated that the informatics approaches can be feasibly

used to capture RWD that are fit for purpose for postmarket evalua-

Use of UDIsUse of

Standardized CodeUse of NLP Use of CDMs

Mayo Clinic 5 5 4 4

Mercy 5 4 4 4

YNHH 4 4 4 4

0

1

2

3

4

5

Mat

urity

Lev

el

Maturity Level Analysis by Three Sites

Figure 2. The maturity level analysis results by 3 sites for the key technologies used in the data capture and transformation. Maturity level consists of 5 levels (ie,

1 ¼ conceptual, 2 ¼ reactive, 3 ¼structured, 4 ¼ complete, and 5 ¼ advanced). CDM: common data model; NLP: natural language processing; UDI: unique deviceidentifier; YNHH: Yale New Haven Hospital.

Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10 2247

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

Table 6. Successes and challenges from informatics analysis

Successes vs Challenges

Use of UDIs• Data capture• Data transformation• Maturity

Successes:• The use of UDIs had been planned in the proposal stage, which had been envisioned as a key method

to identify device exposure data.• The use of UDIs is particularly effective in identifying brand-specific devices and relevant device ex-

posure data as targeted in the NESTcc test case (see details in text).

Challenges:• The source of UDI information varies by healthcare system, requiring tailored approaches to extract-

ing it and linking it to EHRs.• UDIs are documented in different health IT systems and efforts are needed to identify and link them

with clinical data in EHR systems.• UDI implementation in health IT systems is uneven across sites. For example, data on medical devi-

ces used in YNHH prior to October 2017 are currently not readily available and were not routinely

captured within the EHR.

Use of Standardized Codes• Data capture• Data transformation• Maturity

Successes:• The algorithms for identifying conditions and outcome endpoints mainly rely on ICD codes. The ad-

vantage of the approach is that data can be readily collected across NCs.• Data quality validation using registry data and chart review as a “gold standard” is an important

component in the study design.

Challenges:• Coming to an agreement on standard computable covariate and outcome definitions took more time

than we foresaw and requires domain-specific (cardiac electrophysiologist) clinical expertise.• Data validation is a complex task for which we had not planned. We were able only to assess posi-

tive predictive values for study outcomes in small samples due to time and funding constraints.• The accuracy of the algorithms using only ICD-10 codes is not optimal for some outcomes (largely

owing to carry over of past diagnoses into subsequent healthcare visits) and more complex algo-

rithms will need to be explored in the future study, eg, only using primary diagnosis vs also include

secondary diagnosis codes, applying the requirement of no reported diagnosis prior to the index pro-

cedure, only including inpatient events for some outcomes such as stroke, adding additional data

types (eg, procedures, medications) and using unstructured clinical notes searched by NLP. Refine-

ment of the algorithms to define rehospitalization and reason for rehospitalization with consensus

across NCs is required for future work.• One key issue is that important diagnoses (eg, arrhythmias, stroke) are carried forward for a signifi-

cant period of time (eg once a patient is diagnosed with AF, they may continue to carry this diagnosis

into the future even though the arrhythmia may not necessarily have recurred, especially in ambula-

tory care visits); this makes ascertaining arrhythmia recurrence using diagnosis codes a challenge as

an effectiveness study outcome and will require algorithm development (eg, restricting stroke events

to inpatient diagnoses), refinement, and validation for use in a regulatory grade study. Simply exam-

ining all diagnoses from ICD-10 codes during follow-up will lead to misclassification and thereby

low positive predictive value.• Some of the “gold standard” measures used in the validation of AF diagnoses had not been validated

themselves so their diagnostic accuracy is unknown, ie, the ablation registry at Mayo Clinic and the

NLP probe at Mercy• YNHH uses an internal coding for procedures, which are not all mapped to the standard CPT codes,

and often less specific, multiple procedure records can exist for the same procedure with some lag in

entry time. Some of these records can persist even when the procedure did not take place, and in

some instances, more than 1 ablation procedure may have taken place. These issues may require

manual chart review to resolve, which can be time-consuming.

Use of NLP technology• Data capture• Data transformation• Maturity

Successes:• We have successfully leveraged NLP to identify covariates like left ventricular ejection fraction from

echocardiogram reports, and to validate atrial fibrillation patient phenotypes (see details in text).• The value of the NLP technology in adding additional data points for improving accuracy of pheno-

typing algorithms has been realized (see details in text).

Challenges:• Requiring advanced expertise in using existing NLP tools or developing fit-for-purpose NLP algo-

rithms.• Lacking NLP solutions that are portable across sites.

Use of CDMs• Data capture• Data transformation• Maturity

Advantages:• The OMOP CDM has specified a device exposure table, with a field to capture UDI information.• i2b2 star schema is a generic model that can handle device data by leveraging device vocabularies in

its ontology cell.• PCORnet CDM is working on expanding the model to capture UDI and device exposure data.• Sentinel CDM is designed primarily for insurance claims data and contains no device data.

(continued)

2248 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

tion of outcomes for label extensions of 2 ablation catheters from

the EHRs and other health IT systems in a multicenter evaluation.

While variations of such systems in each institution caused some

data quality issues on data capture and transformation, we argue

that development of coordination across otherwise perfectly fit-for-

use (for other purposes) systems would be required for the device

data integration needs of the postmarket surveillance study. How-

ever, we also identified a number of challenging areas for future

improvements, including integrating UDI-linked device data with

clinical data into a research repository; improving the accuracy of

phenotyping algorithms with additional data points such as timing,

medication use, and data elements extracted from unstructured clin-

ical notes using NLP; specifying a chart review guideline to stan-

dardize the chart review process; and using CDM-based research

repositories to standardize the data collection and analysis process.

FUNDING

This project was supported by a research grant from the Medical Device Inno-

vation Consortium as part of the National Evaluation System for Health

Technology (NEST), an initiative funded by the U.S. Food and Drug Adminis-

tration (FDA). Its contents are solely the responsibility of the authors and do

not necessarily represent the official views nor the endorsements of the De-

partment of Health and Human Services or the FDA. While the Medical De-

vice Innovation Consortium provided feedback on project conception and

design, the organization played no role in collection, management, analysis,

and interpretation of the data, nor in the preparation, review, and approval of

the manuscript. The research team, not the funder, made the decision to sub-

mit the manuscript for publication. Funding for this publication was made

possible, in part, by the FDA through grant 1U01FD006292. Views expressed

in written materials or publications and by speakers and moderators do not

necessarily reflect the official policies of the Department of Health and Hu-

man Services; nor does any mention of trade names, commercial practices, or

organization imply endorsement by the U.S. government. In the past 36

months, JSR received research support through Yale University from the

Laura and John Arnold Foundation for the Collaboration for Research Integ-

rity and Transparency at Yale, from Medtronic and the FDA to develop meth-

ods for postmarket surveillance of medical devices (U01FD004585) and from

the Centers of Medicare and Medicaid Services (CMS) to develop and main-

tain performance measures that are used for public reporting (HHSM-500-

2013-13018I); JSR currently receives research support through Yale Univer-

sity from Johnson & Johnson to develop methods of clinical trial data shar-

ing, from the FDA for the Yale-Mayo Clinic Center for Excellence in

Regulatory Science and Innovation program (U01FD005938); from the

Agency for Healthcare Research and Quality (R01HS022882); from the Na-

tional Heart, Lung, and Blood Institute of the National Institutes of Health

(R01HS025164, R01HL144644); and from the Laura and John Arnold Foun-

dation to establish the Good Pharma Scorecard at Bioethics International.

SSD reports receiving research support from the National Heart, Lung, and

Blood Institute of the National Institutes of Health (K12HL138046), the

Greenwall Foundation, Arnold Ventures, and the NEST Coordinating Center.

PAN reports receiving research support from the National Institute of Aging

(R01AG 062436-1) and the Heart, Lung, and Blood Institute (R21HL

140205-2, R01HL 143070-2, R01HL 131535-4) of the National Institutes of

Health and the NEST Coordinating Center.

AUTHOR CONTRIBUTIONS

GJ, JPD, SSD and WLS developed the initial drafts of the manuscript. JC, YY,

PT, EB, SZ, WLS, GJ contributed to the data collection and analysis. AAD,

PAN, SSD and JPD contributed their clinical expertise required for the study.

JPD, NSS, JSR, PC and KRE led the conception, and provided oversight and

interpretation, of the project and the manuscript. All authors reviewed and

approved of the submitted manuscript and have agreed to be accountable for

its contents.

ETHICS STATEMENT

This study was approved by the institutional review boards (IRBs) of Mercy

Health (IRB Submission No. 1349229-1), Mayo Clinic (IRB Application No.

19-001493), and Yale New Haven Hospital (IRB Submission No.

2000024523).

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Infor-

matics Association online.

ACKNOWLEDGMENTS

We thank Kim Collison-Farr for her work as the project manager, Ginger

Gamble for her work as Yale project manager, Lindsay Emmanuel for her

work as Mayo Clinic project manager, and Robbert Zusterzeel, MD, at the

NEST Coordinating Center for his support throughout the project.

DATA AVAILABILITY STATEMENT

The data underlying this article cannot be shared publicly due to ethical/pri-

vacy reasons (ie, they are patient-level device data).

Table 6.. continued

Successes vs Challenges

• CDMs can be used for standardizing data collection and analysis process across sites, facilitatingmeaningful collaborations.

Challenges:• The implementation of CDMs often requires significant time and effort to extract and convert data

from clinical data information systems, such as EHRs and laboratory information systems, to the

format required to load into each CDM.• Multiple CDMs may be difficult to maintain for each health system and the health systems may im-

plement different CDMs, thus decreasing the value of use of CDMs.• The CDMs lack definitive rules for storing the UDI, and therefore, more generic identifiers such as a

device identifier without a product identifier may be present in these fields.

CDM: common data model; CPT: Current Procedural Terminology; EHR: electronic health record; i2b2: Informatics for Integrating Biology and the Bedside;

ICD-10: International Classification of Diseases–Tenth Revision; IT: information technology; NC: network collaborator; NESTcc: National Evaluation System

for Health Technology Coordinating Center; NLP: natural language processing; OMOP: Observational Medical Outcomes Partnership; PCORnet: Patient-Cen-

tered Research Network; UDI: unique device identifier; YNHH: Yale New Haven Hospital.

Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10 2249

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022

CONFLICT OF INTEREST STATEMENT

WLS was an investigator for a research agreement, through Yale University,

from the Shenzhen Center for Health Information for work to advance intelli-

gent disease prevention and health promotion; collaborates with the National

Center for Cardiovascular Diseases in Beijing; is a technical consultant to

HugoHealth, a personal health information platform, and cofounder of

Refactor Health, an AI-augmented data management platform for healthcare;

and is a consultant for Interpace Diagnostics Group, a molecular diagnostics

company. PC and SZ are employees of Johnson & Johnson and own stock in

the company; Johnson & Johnson’s cardiac ablation catheters were the re-

search topic of the NESTcc test case, although this study was a feasibility

study to evaluate the quality of the data to support regulatory decisions.

REFERENCES

1. Corrigan-Curay J, Sacks L, Woodcock J. Real-world evidence and real-

world data for evaluating drug safety and effectiveness. JAMA 2018; 320

(9): 867–8.

2. 21st Century Cures Act. 2020. https://www.fda.gov/regulatory-informa-

tion/selected-amendments-fdc-act/21st-century-cures-act. Accessed

March 29, 2020.

3. FDA Guidance on RWE for medical devices. 2020. https://www.fda.gov/

media/99447/download. Accessed October 19, 2020.

4. Gottlieb S, Shuren J. Director of the Center for Devices and Radiological

Health, on FDA’s updates to Medical Device Safety Action Plan to en-

hance post-market safety. 2020. https://www.fda.gov/news-events/press-

announcements/statement-fda-commissioner-scott-gottlieb-md-and-jeff-

shuren-md-director-center-devices-and-2. Accessed April 10, 2020.

5. Krucoff MW, Sedrakyan A, Normand SL. Bridging unmet medical device

ecosystem needs with strategically coordinated registries networks. JAMA

2015; 314 (16): 1691–2.

6. National Evaluation System for Health Technology Coordinating Center

(NESTcc). 2020. https://nestcc.org/. Accessed March 29, 2020.

7. MDUFA Performance Goals and Procedures Fiscal Years 2018 Through

2022. 2020. https://www.fda.gov/media/100848/download. Accessed

January 13, 2021.

8. NESTcc Data Quality Framework. 2020. https://nestcc.org/data-quality-

and-methods/. Accessed March 29, 2020.

9. Jiang G, Yu Y, Kingsbury PR, Shah N. Augmenting medical device evalua-

tion using a reusable unique device identifier interoperability solution

based on the OHDSI common data model. Stud Health Technol Inform

2019; 264: 1502–3.

10. Zerhouni YA, Krupka DC, Graham J, et al. UDI2Claims: planning a pilot

project to transmit identifiers for implanted devices to the insurance claim.

J Patient Saf 2018 Nov 21 [E-pub ahead of print].

11. Drozda JP Jr, Roach J, Forsyth T, Helmering P, Dummitt B, Tcheng JE.

Constructing the informatics and information technology foundations of

a medical device evaluation system: a report from the FDA unique device

identifier demonstration. J Am Med Inform Assoc 2018; 25 (2): 111–20.

12. Global Unique Device Identification Database (GUDID). 2020. https://www.

fda.gov/medical-devices/unique-device-identification-system-udi-system/global-

unique-device-identification-database-gudid. Accessed March 30, 2020.

13. Drozda J, Zeringue A, Dummitt B, Yount B, Resnic F. How real-world ev-

idence can really deliver: a case study of data source development and use.

BMJ Surg Interv Health Technologies 2020; 2 (1): e000024.

14. Tcheng JE, Crowley J, Tomes M, et al.; MDEpiNet UDI Demonstration

Expert Workgroup. Unique device identifiers for coronary stent postmar-

ket surveillance and research: a report from the Food and Drug Adminis-

tration Medical Device Epidemiology Network Unique Device Identifier

demonstration. Am Heart J 2014; 168 (4): 405–13.e2.

15. Weeks J, Pardee R. Learning to share health care data: a brief timeline of

influential common data models and distributed health data networks in

U.S. health care research. EGEMS (Wash DC) 2019; 7 (1): 4.

16. FDA Sentinel Common Data Model. 2020. https://www.sentinelinitiative.

org/sentinel/data/distributed-database-common-data-model. Accessed

April 4, 2020.

17. OHDSI OMOP Common Data Model. 2020. https://github.com/OHDSI/

CommonDataModel/wiki. Accessed April 4, 2020.

18. PCORnet Common Data Model Specification 2020. https://pcornet.org/

wp-content/uploads/2019/09/PCORnet-Common-Data-Model-v51-

2019_09_12.pdf. Accessed April 4, 2020.

19. I2B2 Star Schema. 2020. https://i2b2.cchmc.org/faq. Accessed April 4,

2020.

20. Visweswaran S, Becich MJ, D’Itri VS, et al. Accrual to Clinical Trials

(ACT): a clinical and translational science award consortium network.

JAMIA Open 2018; 1 (2): 147–52.

21. McPadden J, Durant TJS, Bunch DR, et al. Health care and precision med-

icine research: analysis of a scalable data science platform. J Med Internet

Res 2019; 21 (4): e13043.

22. Guimaraes PO, Krishnamoorthy A, Kaltenbach LA, et al. Accuracy of

medical claims for identifying cardiovascular and bleeding events after

myocardial infarction: a secondary analysis of the TRANSLATE-ACS

study. JAMA Cardiol 2017; 2 (7): 750–7.

23. Adekkanattu P, Jiang G, Luo Y, et al. Evaluating the portability of an

NLP system for processing echocardiograms: a retrospective, multi-site

observational study. AMIA Annu Symp Proc 2019; 2019: 190–9.

24. Liu C, Ta CN, Rogers JR, et al. Ensembles of natural language processing

systems for portable phenotyping solutions. J Biomed Inform 2019; 100:

103318.

25. Li M, Carrell D, Aberdeen J, et al. Optimizing annotation resources for

natural language de-identification via a game theoretic framework. J

Biomed Inform 2016; 61: 97–109.

26. FDA-NIH Biomarker Working Group. BEST (Biomarkers, EndpointS, &

other Tools) Resource. Silver Spring, MD: U.S. Food and Drug Adminis-

tration; 2016.

2250 Journal of the American Medical Informatics Association, 2021, Vol. 28, No. 10

Dow

nloaded from https://academ

ic.oup.com/jam

ia/article/28/10/2241/6328966 by guest on 28 March 2022