I am extraordinarily grateful that Dr. Morey, a well-known researcher with expertise in personality disorders, has had the interest, time, and energy to comment on a couple of statistical issues in my presentation.  I could simply say that I do not disagree with anything he says and stop there, but it would not be fair considering his effort, so I am going to provide a lengthy answer.

The first point I want to stress is that Dr. Morey has missed the context of my lecture.  This is not a presentation for research psychologists or even psychologists, in general.  Psychologists have a much more sophisticated level of statistical training than physicians (and other prescribers of psychiatric drugs such as nurse practitioners). I am certain that none of the psychiatry residents that I teach would understand the statistical comments that Dr. Morey has made and that I am going to make to answer some specific comments made by Dr. Morey. This is a presentation originally developed to teach “basic” statistics to psychiatry residents so they can read psychopharmacology articles. That was clearly explained in the “warnings” slides of my presentation. Slide 4 says, “To be an expert in psychiatry, a psychiatrist should have a basic understanding of statistics for interpreting published psychopharmacological studies.” 

As Dr. Morey is a psychologist working in a department of psychology, I am not sure that he is aware of the difficulties of teaching statistics to physicians.  All physicians take a statistics course in medical school, but they forget statistics as soon as they pass the exam for that course.  I have been supervising psychiatry residents and/or psychiatrists for >20 years, including supervision in writing articles and/or discussing published articles during clinical rotations. During that time, I have supervised hundreds of psychiatrists and have written articles with at least 50 of them, from medical students to professors of psychiatry.  But only two of them had enough statistical knowledge to get an article published in a good psychiatric journal without the help of a statistician (or a psychologist trained in statistics). One is a current professor of psychiatry in Spain who knew statistics before rotating with me just before entering a residency program in psychiatry.  The other is a current medical student from my university who is finishing the MD/PhD degree in May 2016.  In recent years, the National Institute of Mental Health (NIMH) has proposed that psychiatry residents should get better scientific training (Yager et al 2004). In my opinion that is highly unrealistic; it is not possible to teach psychiatrists about science without a basic understanding of statistics, which is missing in medical school education unless you have gone through an MD/PhD or other additional training (master’s in public health), extremely rare among psychiatrists.

As Slide 2 on Educational Objectives described, the specific goal of this presentation is to help psychiatry residents and psychiatrists understand some basic concepts, including meta-analysis and randomized controlled trial (RCT). It is written in the context of what is called evidence-based medicine (EBM). According to EBM experts, physicians should be able to understand the meta-analyses that summarize the combined results of multiple RCTs. In my experience, most residents or psychiatrists that I know, cannot understand a meta-analysis article published in a psychiatric or medical journal that focuses on the efficacy and/or safety of psychiatric drugs. I have the fantasy that my presentation will help meet that goal.

Regarding “number needed to treat (NNT)”, my intention was not to say that NNT is an excellent indicator of treatment effects.  I am sure that I did not say that, but if it appears that I implied it in any way, I apologize. Unfortunately, neither Dr. Morey’s nor my opinions on the strengths or weaknesses of NNT is relevant.  As the presentation describes, NNT is used to summarize treatment effects in meta-analyses in psychiatric and medical journals. I am not surprised that it is not frequently used in psychological journals since, as I already indicated, psychologists are much more sophisticated in their statistical approach than physicians. I do not disagree with any of the critiques of NNT that Dr. Morey made.  As a matter of fact, I have never particularly liked NNT.  I have published > 270 articles in the last 30 years and until 2005 I did not include NNT in any of my articles.  In 2005, I started collaborating on meta-analyses of psychopharmacology drugs.  If you want to publish a meta-analysis in a psychiatric or medical journal, you need to provide NNTs and number-needed-to-harm (NNH), so one of my 2015 meta-analysis articles uses these concepts.

My major personal problem with NNT is that it represents the result of an RCT or group of RCTs (meta-analysis) as an average response. I have serious doubts that the EBM approach of summarizing treatments through the use of mean effects is wise. I should apologize to Dr. Morey, since my ideas about NNTs and other approaches focused on mean effects are described in another PowerPoint lecture called “Evidence-Based Medicine vs. Personalized Medicine”, which is not finished yet.  I got completely “saturated” from trying to explain statistics to physicians and testing the presentation with my residents, and I decided to leave that presentation for the end of the course, after I have had more rest. The introductory lecture called “Training Psychiatrists to Think Like Pharmacologists – Introduction” is available at INHN but, due to a minor mistake, it is currently linked to “1. Introduction to Clinical Pharmacology”. I hope that this mistake will be corrected soon. In Slide 28 of “Training Psychiatrists to Think Like Pharmacologists – Introduction”, I list “Evidence-Based Medicine vs. Personalized Medicine” as one of the 14 theoretical lectures in this course. If Dr. Morey does not want to wait 2 months for that lecture, he can read my opinion in an editorial in a psychopharmacology journal (de Leon 2002). That editorial was written for pharmacologists and psychiatrists, not psychologists, and criticizes the EBM approach, which may not be very relevant for psychologists but is the latest fad in medicine. If Dr. Morey is kind enough to read the lecture called “Training Psychiatrists to Think Like Pharmacologists – Introduction”, I suggest that he pay attention to Slide 35 that states, “Similarly, as Dr. de Leon has no formal training in statistics, it is possible that on rare occasions when he jumps into the troubled waters of statistics and tries explaining statistical concepts to make them understandable to clinicians, he has oversimplified too much. To combat this problem, he provides references that can be checked by readers.”

Regarding the critique on the use of CIs to compare the significance of different NNTs, I do not disagree with his critiques. I only complain that Dr. Morey did not pay attention to my comment on several slides that clearly states they are “approximations”. On the other hand, Dr. Morey’s recommendation is not realistic. He proposes “to use the typical statistic test suitable for the analysis in question (e.g., a chi-square test) to determine statistical significance”. If Dr. Morey reads any of the published meta-analysis articles used as examples, it will be seen that they do not provide tests of significance to compare the various NNTs from different drugs. If he wants the psychiatrist or the psychiatry resident to get the data described by calculating the NNT and entering the data in a statistical program and calculating several dozen chi-square tests, he has never discussed statistics with psychiatry residents. If he finds one who can do that without help, I urge him to tell this psychiatry resident to contact me. I am willing to further train him/her in statistics and consider him/her a collaborator for future articles. For years, I have been looking for a psychiatrist willing to learn statistics. As described, many psychiatrists want to publish with me but almost none want to learn statistics, meaning that I have to do the statistics or beg my statistical collaborator (F. J. Diaz) to do the statistical calculations for our articles.     

Regarding Cohen’s d, I am grateful that Dr. Morey gives me the opportunity to discuss Cohen’s contributions to statistics. As mentioned above, I never used NNTs in my articles until 2015, but many of my published articles quote as a reference Cohen’s textbook (Cohen 1988) and/or Cohen’s article (Cohen 1992) listed by Dr. Morey. I am aware that Dr. Cohen was a psychologist who changed statistical testing in psychology and, after many years of delay, his ideas finally reached medicine. My statistician (Dr. Diaz) and I have always emphasized effect sizes more than statistical significance in our articles, since effect sizes are much more important in the clinical environment than statistical significance and we are in debt to Dr. Cohen’s writings for that.  On the other hand, I would be highly surprised if >10% of US academic psychiatrists know who Dr. Cohen is.  If I am right and my colleagues who are professors of psychiatry do not know who Cohen is, I do not think it makes sense to mention his name in a presentation designed for my residents.  

Regarding presenting the “d” formula, I avoided that on purpose.  I am completely incompetent in explaining that formula or similar mathematical formulas to any of my psychiatry residents.  If Dr. Morey is willing (and able) to teach that formula to any of my psychiatry residents, I would be happy to send him/her to him.

There is one of Dr. Morey’s ideas that I am seriously considering.  I really think that a slide on Cohen’s convention for classifying effect sizes would be an excellent addition to a new version of the course. I will need to test a slide like that with my residents before inserting it. My fantasy is that in 2016 I will develop an improved version of this course.  I say fantasy because first I need to finish the 2015 version, which has 34 lectures (14 theoretical and 20 cases). Then, assuming that there is enough interest from INHN visitors in an updated 2016 version, I will need to update the 34 lectures and hopefully add another 16 cases. My experience is that current psychiatry residents much prefer learning psychopharmacology from cases than from theoretical lectures. They particularly do not like having a statistical lecture although they understand it is needed. They understand that some of these statistical concepts are important since they are included in the annual exam that they take in preparation for psychiatry boards, but they are aware that practicing psychiatrists do not use statistics in their practice.

Finally, I have to give thanks to Dr. Morey for being so gentle and kind with his critiques.  This week, I am working on another “numerical issue” for a presentation focused on the use of drug half-life in clinical practice.         I am afraid that pharmacologists are not going to like my numerical simplifications.  I would sign up right now to get a pharmacologist reviewer who is as kind with my pharmacological simplifications as Dr. Morey has been with my statistical simplifications.  As the introductory lecture describes, I have the same problem in pharmacology as in statistics. “As Dr. de Leon has no formal training in pharmacology, it is possible that in the process of explaining pharmacological concepts to make them understandable to clinicians, he has oversimplified too much. To combat this problem, he provides references that can be checked by readers.” 

References

Cohen, J.  Statistical Power Analysis for the Behavioral Sciences (2nd Ed.). New York: Academic Press; 1988.

Cohen, J. A power primer. Psychological Bulletin 1992; 112, 155-9.

de Leon, J. Evidence-based medicine versus personalized medicine: are they enemies? Journal of Clinical Psychopharmacology 2002; 32:153-64.

Yager J, Greden J, Abrams M, Riba M.  The Institute of Medicine's report on research training in psychiatry residency: strategies for reform – background, results, and follow up. Academic Psychiatry, 2004; 28: 267¬74.

Jose de Leon
March 3, 2016