AI models trained on electronic health record (EHR) data have already demonstrated significant value, helping researchers uncover patterns in clinical data that can improve disease prediction and personalize care. Researchers are now exploring what is possible when AI models can learn from additional health data modalities to create a more complete understanding of human health.
One example comes from health technology and platform company Verily Health. Collaborating with NVIDIA and the National Institutes of Health (NIH) All of Us Research Program, the team developed the first multimodal foundation model integrating EHR and genomic data. The work offers an early example of how multimodal AI can bring together diverse sources of health data to advance precision medicine.
Connecting multiple dimensions of human health
While models trained on clinical data alone remain extremely valuable, integrating additional modalities, such as imaging and genomics, represents an important step forward.
"Electronic health records capture an important piece of a patient's story, but they're only one chapter," explained Jonathan Amar, senior manager of Verily Data Science. "To truly understand disease risk, AI needs to learn from multiple dimensions of human health, including both our biology at birth and our lived clinical experience."
According to Amar, combining EHR data with genomic information is particularly promising because it can reveal inherited disease risk long before symptoms appear. Connecting the "nature" and "nurture" dimensions of health data and contextualizing how they influence one another has long been one of the biggest challenges for healthcare AI.
Putting the multimodal foundation model into action
To test the potential of an integrative approach, Verily's research set out to combine EHR and genomic data in a single foundation model.
Foundation models are particularly well-suited for integrating diverse health data modalities to identify complex disease risk patterns because they can learn broad patterns from large, diverse datasets before being adapted for specific prediction tasks. As larger multimodal datasets become available, foundation models will create new opportunities for AI to uncover relationships that would be difficult to detect from a single source of health data alone.
But the approach was not without its challenges. Amar noted, "There were several factors in play — starting with the complexity of integrating static genetic data with dynamic clinical histories. They're fundamentally different types of data with very different structures. You also have the computational demands of processing both together that most CPU-based workflows can't handle."
As another layer of complexity, access to genomic data is typically unavailable for many patients. That's where nation-level initiatives like the All of Us Research Program — which collects multimodal data, including genomic data — open new doors for developing models that capture a more nuanced, precise understanding of health.
Amar explained their approach, noting, "We were able to integrate genetic risk scores (measurements that predict an individual's likelihood of developing a disease based on their DNA) with EHR data in a single foundation model, powered by NVIDIA GPUs — with exciting results."
When genetic risk information was added to the model, its ability to identify people at high risk for Type 2 diabetes improved substantially. By accurately spotting more true cases while reducing incorrect alerts, the model demonstrated a significant boost in performance. The model also showed it could project risk over clinically meaningful timeframes of five to ten years, rather than focusing only on near-term predictions.
On the computational side, the model achieved a threefold increase in pre-training efficiency by using the NVIDIA NeMo AutoModel compared to Hugging Face Accelerate on identical H100 GPUs. This translated to faster time to insight and faster iteration in a complex research environment.
This pioneering model is now available to all researchers via GitHub as Forecast™ 1.0. \Forecast 1.0 gives global research teams hands-on access to accelerate their own disease risk discovery pipelines within a secure Trusted Research Environment (TRE).
Laying the foundation for what's next
In this case, combining EHR and genomic data improved the precision of Type 2 diabetes prediction. But the implications extend far beyond a single disease or even genomics.
This work provides an early blueprint for how multimodal foundation models could integrate many different dimensions of human health — including wearables, clinical notes, medical imaging and patient-reported outcomes — to advance precision medicine.
To realize this future, Verily continues to research and develop improvements in our predictive and generative models. Driven by the continuous expansion of relevant, multidimensional datasets — including rich, unstructured clinician notes — with a vision to evolve Forecast™ models over time. By continuously enriching the datasets the model learns from, Forecast will capture an increasingly high-resolution, dynamic picture of patient health.
“By bringing together two types of typically mismatched data, we demonstrated how multimodal AI can improve disease prediction while laying the groundwork for a future that incorporates many more health data modalities,” emphasized Amar. “This provides a preview of what's possible in areas such as pharmaceutical biomarker discovery, diagnostic risk stratification and health system precision medicine programs."
For a deeper dive into the Verily-NVIDIA project, including the technology, methodology and results behind the multimodal foundation model, download the whitepaper.
