## Saturday, January 9, 2016

### Asymmetric funnel plots without publication bias

In my last post about standardized effect sizes, I showed how averaging across trials before computing standardized effect sizes such as partial $\eta^2$ and Cohen's d can produce arbitrary estimates of those quantities. This has drastic implications for meta-analysis, but also for the interpretations of these effect sizes.  In this post, I use the same facts to show how one can obtain asymmetric funnel plots — commonly taken to indicate publication bias — without any publication bias at all. You should read the previous post if you haven't already.

A funnel plot is a commonly-used meta-analytic technique for the detection of bias in a subset of the scientific literature. The basic thinking is that if a literature is unbiased, the average estimates of an effect should not depend on the sample size (or some other measure of the "precision" of a study). For a given sample size, estimates of the effect size should be spread around the true effect size, with this spread decreasing as sample size gets larger.

Publication bias, which is often assumed to manifest itself as 1) a tendency for statistically significant results to be published, and 2) a tendency for researchers to publish effects consistent with their theoretical outlook, will result in asymmetric funnel plots. Read this Neuroskeptic post about a paper by Shanks and colleagues for an example how asymmetric funnel plots are used to argue for publication bias. Notice that the plots use a standardized effect size on the x axis.

#### A (not so) hypothetical paradigm

Since many priming effects have been called into question of recently, I will use a priming example. Suppose we are interested in an emotional face priming: we ask participants to perform a lexical decision task, but prior to every trial we "subliminally" (ie, very quickly) present either an excited face or a sad face, thinking that the excited face will speed performance on the task. Participants perform a number of trials in both priming conditions, which are averaged to obtain two "observations" per participant: an average RT in each condition. This is very common in the psychological literature. A paired t test is used to assess the effect of the prime.

Now suppose this same paradigm is used across many labs, with only variation in sample sizes. Each lab reports the standard statistics: the mean difference in RTs across participants, its standard error, and the t statistic. A skeptic comes along, collects all the statistics across all the papers, and computes Hedge's g standardized effect size (a variation on the standardized difference score) from the t statistic. They produce the funnel plot shown below by plotting the sample size1 (number of participants) against the standardized effect size:
This is a massively asymmetric funnel plot, and would likely be taken as strong evidence of publication bias. However, because I simulate the data, I know that there is no publication bias at all. This is merely an artifact of averaging and standardized effect sizes. You can obtain my simulation code here: github gist

Why is the funnel plot asymmetric? In all studies, the total number of trials performed was approximately the same: 2000 trials. The way these broke down across participants was different. Some studies had 100 trials per condition and 10 participants; others, 10 trials per condition and 100 participants. The standard deviation of the difference scores around their mean is a function of the number of trials performed per participant. When the number of trials is high, the standardized effect size is high, just as discussed in the previous blog post. But here, because the total amount of "effort" per study is conserved (that is, all studies have the same number of total trials), the studies with larger numbers of trials per participant have a smaller number of participants. The funnel plot therefore looks problematic, but it is an artifact.

One wonders if this Cross Validated query was related to this artifact.

Creating a funnel plot from the raw effect sizes removes the asymmetry; a funnel plot with the standard error on the y axis also does so.

This does not mean that using the standard error on the y axis fixes the problem. Consider another way number of trials and number of participants can be divided: positively correlated, rather than negatively as before. That is, studies that run more participants also run more trials per condition. The funnel plots end up looking very strange, with an asymmetry that is reverse of the one we expect. Larger effect sizes are obtained with larger numbers of participants.
Without reflection, this pattern might be offered as evidence that there was something very strange happening in a literature. But there's nothing strange here, except with the analysis. If there were publication bias, though, this artifact might actually mask it.

#### Wrap up

I suspect there are other artifacts one could generate using standardized effect sizes in a meta-analysis2. How can we keep from getting fooled? In some cases, perhaps the correction I mentioned in the previous post might be of use. Since a funnel plot is often used for detecting problematic bias in a literature rather than estimating the effect size, the fact that there is no "true" effect size is not problematic.

For future research, data sharing and reporting of different effect size measures will help. Modifications of Cohen's d and Hedge's g exist which will reduce this problem (see "Computing d and g from studies that use pre-post scores or matched groups", for instance), but these modified statistics cannot be computed from typically-reported statistics. The fact that we need statistics that are not typically reported in order to perform reasonable meta-analyses raises the question of whether current reporting practices really allow a cumulative science.

#### Footnotes

1Funnel plots can be created with a variety of statistics on the y axis. Different researchers make different recommendations for both axes (see, for instance, Peters et al 2006), and as we will see, this can have a dramatic effect on the conclusions.

2Sterne et al (2011) note minor asymmetries caused by a correlation between an effect and a standard error, as can be caused in estimation of extreme proportions or similar parameters, but nothing as dramatic or fundamental as shown here. Their asymmetries are mostly problematic for asymmetry tests, which can pick up minor asymmetries with larger samples.

1. A variation on the theme of small studies are more rigorous causing larger effects in smaller studies. In your case, if one suspects that some rigor variable is causing an artifact (in your case, trails per subject), one could regress it out first.

1. Am jeremiah, I am testifying about a great hebal man that cured my wife of hepatitis B and liver cirhosis. his name is Dr oniha. My wife was diagnose of hepatities two years ago, i almost spent all i had then, until i saw dr oniha recommendation online, and i call him, then he told me how to get the herb. You can call him on +2347089275769 or email him at dronihaspell@yahoo.com

2. My symptoms started at the age of 47. My fingers on my left hand were stiff and were difficult to move. People noticed that my walk was not normal. I was often asked did I hurt. I noticed nothing different about my walk. It was difficult getting up from` a chair and getting out of a car. I was diagnosed a year later ,it was the onset of tremors starting in my right hand that caused my other symptoms to be recognized as Parkinson's.. I am now 59. With the new herbal medicine i purchase from http://totalcureherbsfoundation.com  was my only way to get rid of my PD,the herbal formula effectively reverse my condition and alleviate all symptoms.

2. I can't see why, in the first example, one would call any of the trials more "rigorous" than any others. They all have the same number of trials, and as one can see from Figure 2 (right) they all have essentially the same standard error. This artifact would be solely attributable to funnel-analyst carelessness.

There's no reason to regress it out, though; one knows how it should affect the effect size (by the square root of the number of trials). The correction I suggested in the previous post --- in this case, dividing the standardized effect size by the square root of the number of trials in a study --- makes the asymmetry go away.

3. Hi Richards,
both in the fixed, random and mixed models m-a, each study effect size is weighted by the inverse of variance which take in account the study numerosity. This does not solve the problem of the number of trials, but I wonder if it mitigates the limitations you raised.

4. Writing your personal statement... The personal statement is your opportunity to sell yourself in the application process, and it generally falls into one of two categories - A comprehensive personal statement - This allows you maximum freedom in terms of what you write and is the type of statement often prepared for application forms See more statement of purpose sample for mba

5. Noteworthy utilization of tenses and additionally creative method for composing made this blog appealing. I read this blog deliberately and discovered nothing unfortunate identified with any reality. First rate work.
LENOVO WinServ 2012 CAL

6. This blog presented itself in a very easy and clear way. Owner of this blog found a very simple way to express its view but when you read this blog completely, you would get to know about how hard it could be to express such in a easy way.LENOVO WinServ 2012 R2 Standard

7. COEPD LLC- Center of Excellence for Professional Development is the most trusted online training platform to global participants. We are primarily a community of Business Analysts who have taken the initiative to facilitate professionals of IT or Non IT background with the finest quality training. Our trainings are delivered through interactive mode with illustrative scenarios, activities and case studies to help learners start a successful career. We impart knowledge keeping in view of the challenging situations individuals will face in the real time, so that they can handle their job deliverables with at most confidence.

http://coepd.us/

8. Irrespective of receiving daily oral or future injectable depot therapies, these require health care visits for medication and monitoring of safety and response. If patients are treated early enough, before a lot of immune system damage has occurred, life expectancy is close to normal, as long as they remain on successful treatment. However, when patients stop therapy, virus rebounds to high levels in most patients, sometimes associated with severe illness because i have gone through this and even an increased risk of death. The aim of “cure”is ongoing but i still do believe my government made millions of ARV drugs instead of finding a cure. for ongoing therapy and monitoring. ARV alone cannot cure HIV as among the cells that are infected are very long-living CD4 memory cells and possibly other cells that act as long-term reservoirs. HIV can hide in these cells without being detected by the body’s immune system. Therefore even when ART completely blocks subsequent rounds of infection of cells, reservoirs that have been infected before therapy initiation persist and from these reservoirs HIV rebounds if therapy is stopped. “Cure” could either mean an eradication cure, which means to completely rid the body of reservoir virus or a functional HIV cure, where HIV may remain in reservoir cells but rebound to high levels is prevented after therapy interruption.Dr Itua Herbal Medicine makes me believes there is a hope for people suffering from,Parkinson's disease,Schizophrenia,Cancer,Scoliosis,Fibromyalgia,Fluoroquinolone Toxicity
Syndrome Fibrodysplasia Ossificans Progressiva.Fatal Familial Insomnia Factor V Leiden Mutation ,Epilepsy Dupuytren's disease,Desmoplastic small-round-cell tumor Diabetes ,Coeliac disease,Creutzfeldt–Jakob disease,Cerebral Amyloid Angiopathy, Ataxia,Arthritis,Amyotrophic Lateral Sclerosis,Alzheimer's disease,Adrenocortical carcinoma.Asthma,Allergic diseases.Hiv_ Aids,Herpes,Inflammatory bowel disease ,Copd,Diabetes,Hepatitis,I read about him online how he cure Tasha and Tara so i contacted him on drituaherbalcenter@gmail.com even talked on whatsapps +2348149277967 believe me it was easy i drank his herbal medicine for two weeks and i was cured just like that isn't Dr Itua a wonder man? Yes he is! I thank him so much so i will advise if you are suffering from one of those diseases Pls do contact him he's a nice man.

10. I am bold enough among many others to state that there is now a potent cure to this sickness but many are unaware of it. I discovered that I was infected with the virus 3 months ago, after a medical check-up. My doctor told me and I was shocked, confused and felt like my world has crumbled. I was dying slowly due to the announcement of my medical practitioner but he assured me that I could leave a normal life if I took my medications (as there was no medically known cure to Herpes). I went from churches to churches but soon found that my case needed urgent attention as I was growing lean due to fear of dying anytime soon. In a bid to look for a lasting solution to my predicament, I sought for solutions from the herbal world. I went online and searched for every powerful trado-medical practitioner that I could severe, cos I heard that the African Herbs had a cure to the Herpes syndrome. It was after a little time searching the web that I came across one Dr Itua(A powerful African Herbal Doctor), who offered to help me at a monetary fee. I had to comply as this was my final bus-stop to receiving a perfect healing. My last resolve was to take my life by myself, should this plan fail. At last it worked out well. He gave me some steps to follow and I meticulously carried out all his instructions. Last month, to be precise, I went back to the hospital to conduct another test and to my amazement, the results showed that negative,Dr Itua Can As Well Cure The Following Desease…Cancer,Hiv,Herpes, Hepatitis B,Liver Inflammatory,Diabetis,Fribroid,,Non Hodgkin Lymphoma,Skin Cancer,Uterine Cancer,Prostate Cancer Dercum,Infertility,fibromyalgia,Get Your Ex Back,Als,SYPHILLIS,Genetic disease,Epilepsy, Parkinson's disease..You can free yourself of this Herpes virus by consulting this great African Herbal Doctor via this e-mail: drituaherbalcenter@gmail.com or call and whatsapp him on +2348149277967 He will help you and his herb medication is sure. he has the cure on all disease .You can talk to me on INSTAGRAM..tashamoore219....

11. My name is Gloria from Senegal, I came in here to confirm Hepatitis B. testimony that I red online some moths ago. I red about how Dr. Iyabiye cured one Mr. Henry and I decided to give it a try to see if my sister could be healed or cured too, behold my sister got cured after she applied his medication for a month. I am here to thank Dr. Iyabiye and to encourage people and reach him for help through: iyabiyehealinghome@gmail.com Call/Whatsapp: +2348072229413

12. I have long felt a special connection with Herbal Medicine. For one thing is natural, Charlie we went to the same small college in Southern California — Claremont Men’s College — though he dropped out and ultimately enrolled at the Julliard School of the Performing Arts in New York. Had he stayed at Claremont, he would have been a senior the year I started there; I have often kidded that I was the reason he left when he finds out I had herpes. So my life have been lonely like hell all day I couldn't bear the outbreak pain, Tasha then introduce me to Dr Itua who use his herbal medicine to cure her two weeks of drinking it. I place an order form him and he delivers to my post office then I picked it up and used for two weeks all my sore was healed completely no more outbreak I'm honestly telling you this man is a great man, I trust him and his herbal medicine so much I'm sharing this to show my appreciation also to let sick people know there is hope with Dr Itua Herbal Made Medicine.Dr Itua Contact Email.drituaherbalcenter@gmail.com,Whatsapp.+2348149277967
He cures.
Herpes,
STROKE.
Hepatitis
H.P.V TYPE 1 TYPE 2 TYPE 3 AND TYPE 4. TYPE 5.
HIV
SYPHILIS.
Liver/Kidney Inflammatory
Epilepsy

13. I have long felt a special connection with Herbal Medicine. For one thing is natural, Charlie we went to the same small college in Southern California — Claremont Men’s College — though he dropped out and ultimately enrolled at the Julliard School of the Performing Arts in New York. Had he stayed at Claremont, he would have been a senior the year I started there; I have often kidded that I was the reason he left when he finds out I had herpes. So my life have been lonely like hell all day I couldn't bear the outbreak pain, Tasha then introduce me to Dr Itua who use his herbal medicine to cure her two weeks of drinking it. I place an order form him and he delivers to my post office then I picked it up and used for two weeks all my sore was healed completely no more outbreak I'm honestly telling you this man is a great man, I trust him and his herbal medicine so much I'm sharing this to show my appreciation also to let sick people know there is hope with Dr Itua Herbal Made Medicine.Dr Itua Contact Email.drituaherbalcenter@gmail.com,Whatsapp.+2348149277967
He cures.
Herpes,
STROKE.
Hepatitis
H.P.V TYPE 1 TYPE 2 TYPE 3 AND TYPE 4. TYPE 5.
HIV
SYPHILIS.
Liver/Kidney Inflammatory
Epilepsy

14. Am jeremiah, I am testifying about a great hebal man that cured my wife of hepatitis B and liver cirhosis. his name is Dr oniha. My wife was diagnose of hepatities two years ago, i almost spent all i had then, until i saw dr oniha recommendation online, and i call him, then he told me how to get the herb. You can call him on +2347089275769 or email him at dronihaspell@yahoo.com

15. If I have some questions about writing task I use the help of essays service. I try to write assignments by myself but sometimes I need advice or reading some samples. To my mind, it's very gooing that nowadays we have a lot of opportunities.