In/consistency in drug sensitivity profiling
A provocative study some time ago reported poor consistency in high-throughput drug response measurements, leading to broad concerns about the reproducibility of phenotypic drug screening, and even questioning the feasibility of precision medicine. This write-up gives my personal view of the ongoing debate, and how it links to the FIMM drug testing projects.
Disclaimer: This story is inspired by the epic film series, Stars Wars, however, any resemblance to actual persons, living or dead, or actual events is purely coincidental, likely products of the author’s imagination. Similar to the Star Wars episodes, the events may not take place in a strictly chronological order, and many times it is difficult to say who is among the good guys and who has “fallen” to the dark side, even though initially being one of the Jedis, the guardians of scientific truth, the “Force” in this story.
First episode - Spark of Rebellion
Not so long ago, a group of scientists published a controversial study in Nature, in which they cross-compared published data sets from the Broad and Sanger institutes in overlapping cancer cell lines, and concluded that drug sensitivity measurements showed highly discordant results, whereas gene expression profiles were reasonably consistent. This evoked some screaming headlines in many science magazines about the feasibility of personalized cancer treatments.
Even though this rebellion against the Broad and Sanger “empires” was kind of heroic and brave, given it was made to defend scientific truth, the result was quite surprising to many of us that have witnessed the high reproducibility of our in-house drug sensitivity profiling studies. These studies have demonstrated, on the contrary, that it is possible to achieve good consistency both within and between laboratories for drug response measurements.
Second episode - The Empire Strikes Back
Meanwhile, the Broad and Sanger researchers joined forces and made a comprehensive re-analysis of their datasets, in which they showed that reasonable consistency can be achieved between the two sites, provided that the evaluation metrics are aligned with the objectives of the functional profiling.
Further support for the empire came in the study from Genentech, which also reported similar consistency of drug sensitivity profiling, provided that the screening assays and experimental protocols are similar enough. It seemed for a while that these counterstrikes set a balance to the Force.
Third episode – Return of the Jedi
However, the “rebels” were not at all satisfied, and they made still another re-analysis of the Sanger and Broad datasets. This re-re-analysis was not published in Nature, instead they chose another publication forum, F1000Research, where they criticized the unfair comparison setup and different drug response metrics used by the Broad and Sanger investigators.
Like in the Star Wars tri-trilogy, it is impossible to say how many episodes there still remain to be seen in this series of strikes and counterstrikes. For many of us who are working on drug profiling projects, the lessons learned so far are not too surprising. Many key conclusions were actually pointed out already in the News & Views commentary to the first comparative study.
What happened to our comparative study? After a long period of silence in the galaxy, there seems to be A New Hope, while the ending of the story is still wide open. So, stay tuned for sequels…
Image source: Wikipedia Commons (en.wikipedia.org/wiki/Star_Wars)