The Same Algorithm, Half the Patients

TL;DR — Reusing someone else's finding means reconstructing their data model: their tables, their fields, their definitions of what counts as what. Measured evidence says that reconstruction routinely fails — and that almost every group builds its model from scratch anyway.

** How 1% becomes 12% **

In 2015, Voss, Ma and Ryan compared two large US claims databases: Truven MarketScan CCAE and Optum Clinformatics. Both held records from millions of people. Both were used for the same kind of observational research. Yet looking at diagnoses recorded during inpatient visits, each condition appeared on average 12% more often in Optum than in CCAE.

The cause was mundane. The two databases did not define an inpatient visit the same way; each provider had its own method of rolling raw claims up into encounters.

When the researchers applied one consistent definition while loading both databases into the OMOP common data model, the average difference fell from 12% to 1%. Across the 7,039 conditions examined, prevalence estimates moved closer together for two-thirds of them [1].

No trial was run. No patients were recruited. The structure and classification of the data were made consistent, and the result changed.

We tend to treat the data model as infrastructure sitting underneath the science: tables, fields, relationships, naming rules. But those choices determine what can be recorded, how it is interpreted, and what an analysis is able to find. In this case the schema was not describing the result; It was producing it.

** The missing half of the method **

When a group publishes a finding, the paper describes the method. A group trying to test that finding has to turn the description back into a working system: what counted as a visit, a patient, an exposure, an outcome; which fields existed; how they related; how values were encoded.

That reconstruction is rarely exact.

Richesson and colleagues applied seven published definitions of diabetes to the same population of 173,503 patients. Each definition produced a different diabetic population; the resulting cohorts overlapped with one another by between 50% and 86% [2].

Pacheco and colleagues compared a standardised, executable patient-selection algorithm against the versions five institutions had already implemented from the written specification. Three produced closely matching cohorts. At the other two, only about half the selected cases overlapped — traced to local readings of details such as age criteria and which clinical codes to include [3].

The REPEAT Initiative measured the same problem at scale. Of 250 real-world-evidence studies, three described all the key parameters without leaving the reproducing team to make assumptions. When 150 were re-executed against the same databases, roughly one in five produced a study population differing from the published one by more than half [4].

This is not a documentation failure. A methods section is a description of a working data model, not the model itself. However carefully it is written, the next team still has to rebuild what the first team already built.

** Rebuilding is the norm **

The same pattern holds outside reproduction work. Across 11.4 million biological sample records, 97.6% of attribute names had been invented by the submitter rather than drawn from the dictionary the repository already provided — 18,198 distinct custom field names for what is largely the same handful of things [5]. Across biomedical ontologies, a quarter to a third of terms overlap, while fewer than 9% are reused [6].

The cause is structural. A data model can be described, cited and admired, but it cannot be installed. Reading someone's schema and retyping it is the entire available workflow. Given that, rebuilding is the rational choice.

** What a solution would have to do**

If the data model is part of the method, it should travel like one. That implies a short list of requirements.

• • It has to be an artefact, not a description. Something with an identity that can be published, versioned and cited, rather than a paragraph to be re-implemented.
• • It has to install rather than be retyped. It should arrive as working structure — tables, types, constraints, relationships — connected up in the receiving workspace.
• • It has to be publishable without the data. For most groups the records cannot leave: patient privacy, intellectual property, competitive exposure. A model that can only be shared together with its contents will not be shared.
• • It has to keep its lineage. The receiving team should be able to see where it came from, take updates, and eventually take ownership and diverge.
• • It has to travel with the method that uses it. The protocol and the structure it produces should arrive together, or they drift apart again.

** Data Environments**

In Dalea, that artefact is a Data Environment: tables and result schemas, columns and types, constraints and relationships, naming schemes, encodings, display patterns.

It publishes as a package containing the schema and its definitions and none of the records — the export walks environments, tables and columns, and never touches a row. Another group installs it into their own workspace instead of recreating it from a paper. References are rewired to the corresponding tables and columns on their side. The installed copy keeps its provenance, stays linked to its source until they detach it and make it their own, and can travel inside a bundle alongside the protocol, templates and documents that use it.

A paper tells us what a group did. The data model tells us what they meant.

If research is meant to travel, we should publish both.

References

1. Voss EA, Ma Q, Ryan PB. The impact of standardizing the definition of visits on the consistency of multi-database observational health research. BMC Medical Research Methodology, 2015. doi.org/10.1186/s12874-015-0001-6

2. Richesson RL, Rusincovitch SA, Wixted D, et al. A comparison of phenotype definitions for diabetes mellitus. Journal of the American Medical Informatics Association, 2013. doi.org/10.1136/amiajnl-2013-001952

3. Pacheco JA, Rasmussen LV, Kiefer RC, et al. A case study evaluating the portability of an executable computable phenotype algorithm across multiple institutions and EHR environments. Journal of the American Medical Informatics Association, 2018. doi.org/10.1093/jamia/ocy101

4. Wang SV, Sreedhara SK, Schneeweiss S, et al. Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions. Nature Communications, 2022. doi.org/10.1038/s41467-022-32310-3

5. Gonçalves RS, Musen MA. The variable quality of metadata about biological samples used in biomedical experiments. Scientific Data, 2019. doi.org/10.1038/sdata.2019.21

6. Kamdar MR, Tudorache T, Musen MA. A systematic analysis of term reuse and term overlap across biomedical ontologies. Semantic Web, 2017. doi.org/10.3233/SW-160238