Finding and appraising qualitative evidence
The commercial, academic or other vested interests of researchers and organizations tend to be reflected in reports of treatment research in which they are involved.
In 1764, a Dr R James published the 6th edition of his book ‘A dissertation on fevers and inflammatory distempers’. In it, he claimed that his secret ‘Fever Powder’ was successful in treating “smallpox, yellow fever, slow fever and rheumatism”. In support of his claims, he cited the testimonies of satisfied patients and a decline in the national mortality rate which had followed his introduction of his miraculous ‘cure-all’. ‘Snake oil salesmen’ like Dr James have probably been a feature of medical practice for as long as patients have looked to doctors and others to help them deal with health problems.
A few years after Dr James made his claims, Elisha Perkins patented and marketed ‘tractors’ – small metal rods which he claimed cured a variety of ailments through ‘electrophysical force’. The cartoon accompanying this paragraph shows a doctor treating a wealthy client with Perkins’ tractors. John Haygarth (1800) conducted an experiment to test the claims made by another salesman. In a pamphlet entitled ‘Of the imagination as a cause and as a cure of disorders of the body: exemplified by fictitious tractors’, Haygarth reported how he put Perkins’ claims to the test. In a series of patients who were unaware of the details of his evaluation, he used a cross-over study to compare the patented, metal tractors (which were meant to work through ‘electrophysical force’) with wooden ‘tractors’ that looked identical (‘placebo tractors’). He was unable to detect any benefit of the metal tractors (Haygarth 1800).
During the 19th century, the ground rules for testing treatment claims began to become clearer. Alternation began to be used to ensure that like would be compared with like (Chalmers et al. 2011); and blinding became recognised as a way of reducing observer biases (Kaptchuk 2011). For example, comparisons of homeopathic with orthodox medical treatments (Löhner 1835; Tessier 1852; Storke et al. 1880; Cook County Board of Commissioners 1882) demonstrated not that homeopathy was effective, but that it was safer than the bleeding and purging being offered by mainstream doctors.
By the early years of the 20th century, a pharmaceutical industry had begun to emerge which was profit-driven, and thus tempted to take liberties in its claims for its products and the use of data to support these. In 1917, Torald Sollmann, an American pharmacologist, set out the principles to be observed in testing treatments, and noted that “Those who collaborate with [commercial firms] should realize frankly that under present conditions they are collaborating, not so much in determining scientific value, but rather in establishing commercial value” (Sollmann 1917). Concerns about these sponsor and researcher biases – and sometimes frank fraud – grew throughout the 20th century, fuelled increasingly by evidence going beyond anecdotes (Davidson 1986; Gøtzsche 1987; Djulbegovic et al. 2000; Lexchin et al. 2003; Melander et al. 2003; Whittington et al 2004; Yank et al. 2007; Turner et al. 2008; Rising et al. 2008). Sponsor and researcher biases make active use of other biases in pursuit of their vested interests, particularly design bias, analysis bias, and reporting bias.
Recognising and reducing research biases and frank fraud remains a substantial challenge.
The text in these essays may be copied and used for non-commercial purposes on condition that explicit acknowledgement is made to The James Lind Library (www.jameslindlibrary.org).
Chalmers I, Dukan E, Podolsky SH, Davey Smith G (2011). The advent of fair treatment allocation schedules in clinical trials during the 19th and early 20th centuries. JLL Bulletin: Commentaries on the history of treatment evaluation (www.jameslindlibrary.org).
Cook County Board of Commissioners (1882). Proceedings, 1881-1882. Chicago, Illinois, 222-223.
Davidson RA (1986). Source of funding and outcome of clinical trials. Journal of General Internal Medicine 1:156-158.
Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, Adams JR, Kuderer NM, Lyman GH (2000). The uncertainty principle and industry-sponsored research. Lancet 356:635-638.
Gøtzsche PC (1987). Reference bias in reports of drug trials. BMJ 295:654-656.
James R (1764). A dissertation on fevers and inflammatory distempers. 6th edn. London: Newbery.
Haygarth J (1800). Of the imagination, as a cause and as a cure of disorders of the body: exemplified by fictitious tractors, and epidemical convulsions. Bath: R. Crutwell.
Kaptchuk TJ (2011). A brief history of the evolution of methods to control of observer biases in tests of treatments. JLL Bulletin: Commentaries on the history of treatment evaluation (www.jameslindlibrary.org).
Lexchin J, Bero LA, Djulbegovic B, Clark O (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326:1167-70.
Löhner G (1835). Die Homoöopathischen Kochsalzversuche zu Nürnberg [The homeopathic salt trials in Nurnberg]. Nürnberg in März.
Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B (2003). Evidence b(i)ased medicine – selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 326:1171-3.
Rising K, Bacchetti P, Bero L (2008). Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 5(11): e217. doi:10.1371/journal.pmed.0050217
Sollmann T (1917). The crucial test of therapeutic evidence. JAMA 69:198-199.
Storke EF, Martin R, Rosenkrans EM, Ford J, Schloemilch A, McDermott GC, Carlson OW (1880). Final report of the Milwaukee test of the thirtieth dilution. Homoeopathic Times 7:12/280-1.
Tessier JP (1852). De la médication homoeopathique [On homoeopathic medication]. Paris: JB Ballière.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 358:252-60.
Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, Boddington E (2004). Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet; 363:1341-1345.
Yank V, Rennie D, Bero LA (2007). Financial ties and concordance between results: retrospective cohort study.
Read more about the evolution of fair comparisons in the James Lind Library.
Finding and appraising qualitative evidence
For lecture on 3 June 2021
Resources for teaching LR etc