Virtual cell models can identify promising genetic targets without resource-intensive wet lab experiments
Biotechnology company Shift Bioscience, which studies cell rejuvenation, has released research detailing an improved framework for evaluating benchmark metric calibration in virtual cell models.
The study demonstrates that virtual cell models consistently outperform key baselines.
We believe that this work opens the door to more widespread use of virtual cells and reinforces our confidence in the virtual cell models that are helping to drive our target identification programme for cell rejuvenation,” said Shift’s head of machine learning Dr Henry Miller.
Genetic perturbation response models are a subset of AI virtual cells used to predict how cells will respond to various genetic alterations. They can identify promising genetic targets without the time and resources required for wet lab experiments.
Shift’s finding contradicts recent published papers questioning the ability of these models to correctly identify gene targets. The study shows that poor model performance is largely associated with miscalibration.
“This latest research from our talented team provides clear evidence that the reports of poor performance in AI virtual cells is largely due to limitations of metrics, not due to issues with the models,” said Miller. “We showed that when models are evaluated on well-calibrated metrics, they perform quite well and consistently outperform key baselines.”
The team then developed an improved framework for metric calibration. Using 14 perturb-seq datasets, the team identified several rank-based and Differentially Expressed Gene (DEG)-aware metrics that are well-calibrated across datasets.
Virtual cell models evaluated using these well-calibrated metrics were able to consistently outperform uninformative mean, control and linear baselines. This shows that when appropriate calibration is applied, virtual cell models can distinguish biologically significant signals.
