Sunday, February 9, 2014

The BayesFactor package

This blog is a companion to the BayesFactor package in R (website), which supports inference by Bayes factors in common research designs. Bayes factors have been proposed as more principled replacements for common classical statistical procedures such as $p$ values; this blog will offer tutorials in using the package for data analysis.
In this first post, I describe the general logic of Bayes factors using a very simple research example. In the coming posts, I will show how to do a more complete Bayesian data analysis using the R package.

What is a Bayes factor?

Suppose that two researchers are interested in public opinion about public smoking bans. Paul believes that 70% of the public support such bans; Carole believes that the support is less, at 60%. Paul and Carole decide ask 100 randomly selected people whether they support public smoking bans.
Because Paul and Carole have very specific beliefs about the true support for the public smoking bans, they also have predictions about how the sample will turn out. Of course, Paul's best guess is that 70 out of 100 will support smoking bans, and likewise Carole's best guess is 60. However, Paul and Carole can be more specific because the Binomial distribution is appropriate for model the kind of random sample. The figure below shows the predictions of Carole (blue) and Paul (red).

Having made predictions, Paul and Carole collect their random sample. Of the 100 people in the sample, 62 are supportive. This seems to support Carole because the observation is closer to Carole's average prediction; however, Paul points out that he also predicted that 62 was possible, and so his hypothesis is not ruled out. It does seem obvious that the observation supports Carole, but by how much?
One way of answering this question is to ask about the relative weight of the evidence: that is, how convincing is the observation with respect to the two hypothesis in question? In order to answer this question, we need a way of understanding what it means for evidence to change what we believe. We start with the idea of prior odds, which describe the degree to which we favor one hypothesis over another before we see the data. These can be written as $\frac{P(\cal H_c)}{P(\cal H_p)}$ where $P$ here represents plausibility and $\cal H_c$ and $\cal H_p$ are hypotheses of Carole and Paul. We can also describe the posterior odds, which describe the degree to which we favor one hypothesis over another after observing the data. $\frac{P(\cal H_c\mid y)}{P(\cal H_p\mid y)}$ where $y$ is what we observed. The question now is how we move from the prior odds to the posterior odds in the right way. Obviously, we cannot simply do this in any way we like. There must be some ways that are better than others. Bayes' rule gives a way to answer this question. Bayes' rule says that the relative plausibility of two hypotheses must be changed by the data in a particular way. $\frac{P(\cal H_c\mid y)}{P(\cal H_p\mid y)} = \frac{P(y\mid \cal H_c)}{P(y\mid\cal H_p)}\times\frac{P(\cal H_c)}{P(\cal H_p)}$ The term in the middle is a factor that we multiply the prior odds by to become the posterior odds. It represents how our relative beliefs should change in light of the data, and is called the Bayes factor. Luckily, the logic is quite simple:

The Bayes factor is the relative evidence in the data. The evidence in the data favors one hypothesis, relative to another, exactly to the degree that the hypothesis predicts the observed data better than the other.
We can now easily compute the evidence in the data favoring Carole over Paul. Carole specified that the observation $y=62$ had a probability of 0.0754. Paul specified that the observation $y=62$ had a probability of 0.0191. Under Carole's hypothesis, the observed data is much more likely than under Paul's, as shown in the figure below:

The Bayes factor — the evidence in the data — is precisely factor by which Carole's line is taller than Paul's, which is:
dbinom(y, N, carole)/dbinom(y, N, paul)

## [1] 3.953

or about 4. The observed data favors Carole by a factor of 4.
The figure below shows how much the data favors Carole for many possible observations. The data favor Carole when the Bayes factor is greater than 1, which happens for all observations 65 or less; the data favor Paul when the Bayes factor is less than 1, which happens when the observation greater than 65. For very large or small observations, the evidence is strong; for observations around 65, the evidence is slight, making it difficult to distinguish between the two hypotheses.

Bayes factors when the parameter is uncertain

The above example was adequate for a simple demonstration, but lacks an important feature common to research: uncertainty about the true parameter value. Carole and Paul both had very specific hypotheses. Typically, this is not the case. Unless a hypothesis is very specific about a particular parameter (for instance, that an experimental manipulation has no effect), hypotheses are more diffuse.
Carole and Paul might have hypotheses that look something like the figure below:

Note that on the $x$-axis is the true proportion of the population who support the smoking bans. Suppose we call the true proportion of supporters $\theta$. Instead of believing that the true proportion is exactly 60%, Carole now believes that it is “around” 60%. The blue distribution shows how her belief drops off as the proportion is further away from 0.6. Likewise, Paul has adopted a distribution around 0.7. We refer to these curves as $p_c(\theta)$ for Carole and $p_p(\theta)$ for Paul.
Since the Bayes factor is determined by the probability that each hypothesis assigns to the observed data, we must determine what Paul and Carole would now predict. Carole's hypothesis $\cal H_c$ is a distribution of values of $\theta$, each weighted differently. Some values, like 0.6, are quite plausible; other values, like 0.7, are implausible. In order to determine the plausibility of a particular observation, we can determine the probability that it would occur given a particular value for $\theta$ (that is, $P(y\mid\theta)$). We then weight this probability by the plausibility given to that particular $\theta$ value, $p(\theta)$. For each hypothesis, this can be represented by an integral: $P(y\mid{\cal H}_c) = \int P(y\mid \theta)p_c(\theta)\,d\theta$ where $p_c(\theta)$ is the function representing Carole's hypothesis. The probability of an observation is thus a weighted average over all the different possible values for $\theta$. The figure below shows how the more diffuse hypothesis affects the predictions for data. In gray are the predictions from the old, single point hypothesis. In color are the new predictions from the diffuse hypothesis. The diffuse hypotheses have made the predictions for the data more diffuse. The bottom row shows the new predictions made by Carole and Paul side-by-side.

We can now compare the two hypotheses in light of the observation of 62 supporters out of 100, as the figure below shows. The Bayes factor has been reduced to 2.8267 in favor of Carole. This makes sense; both Carole and Paul have hedged their predictions for the data, making them more spread out and in the process less distinct. The reduction in the Bayes factor reflects this by becoming attenuated.

Uncertain hypotheses are not a problem for the Bayes factor to handle, as long as they can be specified as probability distributions, but hedging one's bets with more uncertain predictions will limit reduce the amount of evidence that can be accumulated in favor of one's hypothesis. This means Bayes factors penalizes flexible hypotheses, yielding a natural Occam's razor.

Strength of the evidence

One might ask how much evidence is provided by the Bayes factors of 3.9534 and 2.8267 found above. The simple answer is that the number itself is directly interpretable, since it arises as the shift of relative odds. In this, the Bayes factor is different from other proposed (but ultimately flawed) evidential statistics, such as the $p$ value, because such statistics have no direct evidential interpretation.
Nevertheless, various labels for the “strength” of a Bayes factor have been proposed. Kass and Raftery (1995), for instance, propose that Bayes factors between 1/3.2 and 3.2 are “not worth more than a bare mention”. See the Wikipedia entry for Bayes factors for more details.

Summary

• Bayes factors represent the weight of evidence in the data for competing hypotheses. Bayes factors are the degree to which the data shift the relative odds between two hypotheses. There are principled reasons why we should interpret the Bayes factor as a measure of the strength of the relative evidence.
• The Bayes factor is intimately linked to the predictions of a hypothesis. Because the evidence for a hypothesis given by the degree to which it has predicted the observed data, hypotheses that do not have predictions for data cannot accumulate evidence. Hypotheses with no predictions, such as $\mu\neq0$ are not allowed.
• The Bayes factor can be directly interpreted, without recourse to labels. The strength of the Bayes factor is reflected by the fact that it is a multiplicative change in odds. However, some authors provide labels to help interpret evidence.
In the next post, we will discuss Bayes factors for one-sample designs with the BayesFactor package.

1. Thanks so much for these great posts.

Is there an example of calculating a Bayes Factor for the difference of proportions as described in the example above? Can this be done in the BayesFactor package?

Thanks!

1. Hi Max, I don't have a blog post about that yet, but Bayes factors of the sort you want can be calculated using the contingencyTestBF() function. See the manual here for an example: http://bayesfactorpcl.r-forge.r-project.org/#ctables

2. Hi Richard, this post is genial. The Bayes factor is common tool in phylogenetic research to decide between two evolutionary hypotheses (models).
This is calculated as BF= 2(harmonic mean 1(complex model) - harmonic mean2 (simple model).
Do you know this procedure?
Best regards

1. Oops, missed the reply. See response below.

3. Hi Violeta, yes, I am aware of the harmonic mean estimator. BayesFactor does not use that estimator; instead, it uses importance sampling, simple Monte Carlo sampling, Gaussian quadrature, and a Laplace approximation (depending on the situation). They all estimate the same quantity, but the harmonic mean estimate is notoriously unstable (see Radford Neal's comment here).

4. Hi,
Thank you for this post. Do you have the R code for this example? I downloaded the package and I didn't see an example like this one.

5. Thank you so much for posting this article. It helped me a lot.

In the last paragraph, you mentioned that "The Bayes factor is intimately linked to the predictions of a hypothesis."

Then how can I get the predictive density of x given a specific hypothesis?
I guess p(x|H) is not a predictive probability density, but just a probability that gives us information about the plausibility of observed data x under H...

6. Thank-you for your nice post Richard!

Could you give further detail about how you obtained the 2.8267 result for the more diffuse hypotheses? In particular, what type of distributions are those shown in the first figure of the section "Bayes factors when the parameter is uncertain"? If I am not mistaken those are the priors, right?

Thanks

7. So far the best explanation (to me) that I came across, thanks very much Richard!

8. This comment has been removed by the author.

9. Thank you so much for posting this article. It helped me a lot.

ibm-message-broker training in chennai

11. • Very good effort in collecting information.........
websphere training in chennai

oracle training in chennai

13. Hello Mr Richard,
Thank you for your posts. I don't really understand much since i'm not good at maths.
Can you help me simplify the calculation of bayes factor from the following problem?

I run Beast using HIV dataset testing two clock model Relax Clock and Strict Clock. So from result of log file from the Beast which can views in tracer, how can i calculate bayes factor to compare these two model?
Sophearot (from Cambodia)

14. Also there should be high amount of transparency as to where the software stands in terms of quality, field service software

and rectify it. Selenium is the best tool ever to test web based applications. Thank you for your information.

Regards:

Selenium Training in chennai |
Selenium Training

16. Can I just say what a relief to discover somebody that really knows what they're discussing online. You definitely understand how to bring a problem to light and make it important. You can review my latest post about ms office More people should check this out and understand this side of your story. I can't believe you are not more popular given that you definitely possess the gift.

18. Get intopc for best pc software you can get here

19. Gta 5 Mobile Dwgamez for best pc software you can get here

20. This comment has been removed by the author.

21. This comment has been removed by the author.

22. Hey, Thanks for such a nice share. You can download windows 7 lite

23. Thanks for sharing this details.

Online ERP Software

24. The very next time I read a blog, Hopefully it does not fail me just as much as this one. I mean, Yes, it was my choice to read, nonetheless I really believed you would probably have something helpful to talk about. All I hear is a bunch of moaning about something you could possibly fix if you weren't too busy seeking attention.
Get the Windows 10 Gamer Edition

25. What are the features of the kindle? This article talks about top six features of the wireless reading device, Kindle Fire. Kindle adbfire download enables user to surf the internet, listen to music, watch movie, read books and etc. Here are the top six features of the Kindle Fire that I am going to share with you.

26. This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..

rpa online training |
rpa course in bangalore |
rpa training in bangalore |
rpa training institute in bangalore

27. Read all the information that i've given in above article. It'll give u the whole idea about it.
python course in pune | python course in chennai | python course in Bangalore

28. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing.. Believe me I did wrote an post about tutorials for beginners with reference of your blog.
excel advanced excel training in bangalore | Devops Training in Chennai

29. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
excel advanced excel training in bangalore | Devops Training in Chennai

30. I found this informative and interesting blog so i think so its very useful and knowledge able.I would like to thank you for the efforts you have made in writing this article.
angularjs Training in marathahalli

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in chennai

automation anywhere online Training

31. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
python course in pune
python course in chennai
python course in Bangalore

32. I am sure this post has helped me save many hours of browsing other related posts just to find what I was looking for. Many thanks!

Core Java interview questions and answers

Java training in Chennai | Java training in Tambaram

Java training in Chennai | Java training in Velachery

Java training in Chennai | Java training in Omr

33. Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information.

rpa training in chennai
rpa training in bangalore
rpa course in bangalore
best rpa training in bangalore
rpa online training

34. Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…
python training institute in marathahalli
python training institute in btm
Python training course in Chennai

35. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...

Best Devops training in sholinganallur
Devops training in velachery
Devops training in annanagar
Devops training in tambaram

36. I likable the posts and offbeat format you've got here! I’d wish many thanks for sharing your expertise and also the time it took to post!!
angularjs Training in bangalore

angularjs Training in bangalore

angularjs online Training

angularjs Training in marathahalli

37. Greetings. I know this is somewhat off-topic, but I was wondering if you knew where I could get a captcha plugin for my comment form? I’m using the same blog platform like yours, and I’m having difficulty finding one? Thanks a lot.
AWS Training in Pune | Best Amazon Web Services Training in Pune

Amazon Web Services Training in OMR , Chennai | Best AWS Training in OMR,Chennai

AWS Training in Chennai |Best Amazon Web Services Training in Chennai

38. It's wonderful commands like mind relaxes with understand words of knowledge by information's. check out https://www.ayevainstitute.com/

It looks like you spend a lot of effort and time on your blog.Keep Doing.
Digital Marketing Training in Bangalore
German Language Training in Bangalore
Best German Classes in Bangalore
German Coaching Classes in Bangalore

40. Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.
Advanced AWS Training in Bangalore | Best Amazon Web Services Training in Bangalore

41. Superior post, keep up with this exceptional work. It's nice to know that this topic is being also covered on this web site so cheers for taking the time to discuss this! Thanks again and again! Adobe Flash CS3 Professional Free Download

42. Thank you for your efforts. This blog is very impressed to me and I got a lot of information about your post. Please keep it.
PHP Training in Bangalore
PHP Course in Bangalore
PHP Training in Anna Nagar
PHP Course in Anna Nagar
PHP Training in Tnagar
PHP Course in Tnagar
PHP Training in Omr
PHP Course in Omr

43. Download your favorite Sinhala songs with Sindu. So, you can get Lyrics, Chords and Sinhala sindu, Sindu Potha, Sindu Sinhala MP3 for free, hundreds of new songs published everyday!

44. Nice and interesting information and informative too.
Can you please let me know the good attraction places we can visit: Bangalore to Chennai

45. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging
samsung mobile service center in chennai
samsung mobile service center
samsung mobile service chennai
samsung mobile repair
samsung mobile service center near me
samsung service centres in chennai

46. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
Thanks & Regards,
VRIT Professionals,
No.1 Leading Web Designing Training Institute In Chennai.

And also those who are looking for
Web Designing Training Institute in Chennai
SEO Training Institute in Chennai
Photoshop Training Institute in Chennai
PHP & Mysql Training Institute in Chennai
Android Training Institute in Chennai