Queensland: You feel unwell and go to your doctor. The doctor asks you some questions and takes some blood for testing; After a few days they call and say that you have been diagnosed with a disease. How likely is it that you will actually have this disease? For some common diagnostic tests, this probability is surprisingly low. Few medical tests are 100 percent accurate.
One reason for this is that people naturally vary, but many trials are also conducted on limited or biased samples of patients – and our own work has shown that researchers deliberately overstate the effectiveness of new tests. Can tell. This does not at all mean that we should stop trusting clinical tests, but a better understanding of their strengths and weaknesses is essential if we are to use them wisely.
people are different
An example of a widely used incomplete test is prostate-specific antigen (PSA) screening, which measures the level of a particular protein in the blood as an indicator of prostate cancer. The test catches an estimated 93 percent of cancers — but its false positive rate is very high, because about 80 percent of men with a positive result don’t actually have cancer. For 80 percent of people, the results create unnecessary stress and the possibility of further testing, including painful biopsies. Rapid antigen testing for COVID-19 is another widely used imperfect test. A review of these tests found that only 52 percent of people who were asymptomatic but had a positive test result actually had Covid. Among people who had symptoms of corona and the results were positive, the accuracy of the tests increased to 89 percent. This shows how the performance of a test cannot be summarized by a single number and depends on the individual context.
Why aren’t diagnostic tests accurate?
One major reason is that every person’s body is different. For example, a high temperature for you may be completely normal for someone else. For blood tests, many external factors can affect the results, such as the time of day or how much you have recently eaten. Even the ubiquitous blood pressure testing can be inaccurate. Results may vary depending on whether the cuff is properly placed on your arm, if you have your legs crossed, and if you are talking when the test is completed.
Small Samples and Statistical Games
Heavy research is going on on new diagnostic models. New models often make headlines as “medical breakthroughs,” such as how your handwriting can detect Parkinson’s disease, how your pharmacy loyalty card can detect ovarian cancer earlier, or how eye movements can detect schizophrenia. Can find out. But staying in the headlines is often a different story. Many clinical models are developed based on small sample sizes. One review found that half of the clinical studies used only more than 100 patients. It is difficult to get a true picture of the accuracy of a diagnostic test from such small samples. For accurate results, the patients using the test should be the same as those used to develop the test.
For example, the Framingham Risk Score, widely used to identify people at high risk of heart disease, was developed in the US and is known to perform poorly in Aboriginal and Torres Strait Islander peoples. Similar disparities in accuracy have been found for “polygenic risk scores.” These combine information from thousands of genes to predict disease risk, but were developed in European populations and perform poorly in non-European populations. Recently, we identified another significant problem: researchers have overestimated the accuracy of some models in order to achieve journal publication.
There are many ways to overestimate the performance of a test, such as removing difficult-to-predict patients from the sample. Some tests are also not truly predictive, as they include future information, such as a predictive model of infection that includes whether the patient was advised to take antibiotics. Perhaps the most extreme example of overstating the power of a diagnostic test was the Theranos scandal, in which a finger prick blood test that diagnoses a number of health conditions attracted millions of dollars from investors. This was not true – and the mastermind has now been convicted of fraud.
Big data can’t make tests perfect In the age of precision medicine and big data, it seems tempting to combine tens or hundreds of pieces of information about a patient to provide highly accurate predictions – perhaps using machine learning or artificial intelligence. Is. However, this promise has far outstripped reality. One study estimated that 80,000 new prediction models were published between 1995 and 2020. That means around 250 new models every month.
Are these models transforming healthcare?
We don’t see any sign of that – and if they really were having a big impact, we certainly wouldn’t need such large quantities of new models. For many diseases there are data problems that no amount of sophisticated modeling can fix, such as measurement errors or missing data that make accurate predictions impossible. Some diseases are inherently random, and involve complex chains of events that no patient can describe and no model can predict.
Examples might include injuries or past illnesses a patient suffered decades ago that they cannot remember and that are not in their medical notes. Clinical trials are never perfect. Acknowledging one’s own shortcomings allows doctors and their patients to have informed discussions about what the test results mean – and most importantly, what to do next. (agency)