Saturday, January 17, 2015

Multiple Comparisons with BayesFactor, Part 1


One of the most frequently-asked questions about the BayesFactor package is how to do multiple comparisons; that is, given that some effect exists across factor levels or means, how can we test whether two specific effects are unequal. In the next two posts, I'll explain how this can be done in two cases: in Part 1, I'll cover tests for equality, and in Part 2 I'll cover tests for specific order-restrictions.

Before we start, I will note that these methods are only meant to be used for pre-planned comparisons. They should not be used for post hoc comparisons.

An Example

Suppose we are interested in the basis for feelings of moral disgust. One prominent theory, from the embodied cognition point of view, holds that feelings of moral disgust are extensions of more basic feelings of disgust: disgust for physical things, such as rotting meat, excrement, etc (Schnall et al, 2008; but see also Johnson et al., 2014 and Landy & Goodwin, in press). Under this theory, moral disgust is not only metaphorically related to physical disgust, but may share physiological responses with physical disgust.

Suppose we wish to experimentally test this theory, which predicts that feelings of physical disgust can be “transferred” to possible objects of moral disgust. We ask 150 participants to fill out a questionnaire that measures the harshness of their judgments of undocumented migrants. Participants are randomly assigned to one of three conditions, differing by the odor present in the room: a pleasant scent associated with cleanliness (lemon), a disgusting scent (sulfur), and a control condition in which no unusual odor is present. The dependent variable is the score on the questionnaire, which ranges from 0 to 50 with higher scores representing harsher moral judgment.

Hypothetical data, simulated for the sake of example, can be read into R using the url() function:
# Read in the data from the learnbayes.org
disgust_data = read.table(url('http://www.learnbayes.org/disgust_example.txt'),header=TRUE)

A boxplot and means/standard errors reveal that the effects appear to be in the predicted direction:



(note that the axes are different in the two plots, so that the standard errors can be seen)

And we can perform a classical ANOVA on these data:
# ANOVA
summary(aov(score ~ condition, data = disgust_data))
##              Df Sum Sq Mean Sq F value Pr(>F)  
## condition     2    263   131.4    2.91  0.058 .
## Residuals   147   6635    45.1                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The classical test of the null hypothesis that all means are equal just fails to reach significance at \(\alpha=0.05\).

A Bayes factor analysis

We can easily perform a Bayes factor test of the null hypothesis using the BayesFactor package. This assumes that the prior settings are acceptable; because this post is about multiple comparisons, we will not explore prior settings here. See ?anovaBF for more information.

The anovaBF is a convenience function to perform Bayes factor ANOVA-like analyses. The code for the Bayes factor analysis is almost identical to the code for the classical test:
library(BayesFactor)
bf1 = anovaBF(score ~ condition, data = disgust_data)
bf1
## Bayes factor analysis
## --------------
## [1] condition : 0.7738 ±0.01%
## 
## Against denominator:
##   Intercept only 
## ---
## Bayes factor type: BFlinearModel, JZS
The Bayes factor in favor of a condition effect is about 0.774, or 1/0.774 = 1.3 in favor of the null (the “Intercept only” model). This is not strong evidence for either the null or the alternative, which given the moderate p value is perhaps not surprising. It should be noted here that even if the p value had just crept in under 0.05, the Bayes factor would not be appreciably different, which shows the inherent arbitrariness of significance testing.

Many possible hypotheses?

This analysis is not the end of the story, however. The hypothesis tested above — that all means are different, but with no further specificity — was not really the hypothesis of interest. The hypothesis of interest was more specific. We might consider an entire spectrum of hypotheses, listed in increasing order of constraint:
  • (most constrained) The null hypothesis (control = lemon = sulfur)
  • (somewhat constrained) Unexpected scents cause the same effect, regardless of type (lemon = sulfur ≠ control; this might occur, for instance, both “clean” and “disgusting” scents prime the same underlying concepts)
  • (somewhat constrained) Only disgusting scents have an effect (control = lemon ≠ sulfur)
  • (somewhat constrained) Only pleasant scents have an effect (control = sulfur ≠ lemon)
  • (unconstrained) All scents have unique effects (control ≠ sulfur ≠ lemon)
The above are all equality constraints. We can also specify order constraints, such as lemon < control < sulfur. The unconstrained model tested above (control ≠ sulfur ≠ lemon) does not give full credit to this ordering prediction. In the next section, I will show how to test equality constraints. In Part 2 of this post, I will show how to test order constraints.

Testing equality constraints

To test equality constraints, we must first consider what an equality constraint means. Claiming that an equality constraint holds is the same as saying that your predictions for data would not change if the two conditions are supposed to be the same had exactly the same label. If want to to impose the constraint that lemon = sulfur ≠ control, we merely have to give lemon and sulfur the same label.

In practice, this means making a new column in the data frame with the required change:
# Copy the condition column that we will change
# We use 'as.character' to avoid using the same factor levels
disgust_data$lemon.eq.sulfur = as.character(disgust_data$condition)
# Change all 'lemon' to 'lemon/sulfur'
disgust_data$lemon.eq.sulfur[ disgust_data$condition == "lemon" ] = 'lemon/sulfur'
# Change all 'sulfur' to 'lemon/sulfur'
disgust_data$lemon.eq.sulfur[ disgust_data$condition == "sulfur" ] = 'lemon/sulfur'
# finally, make the column a factor
disgust_data$lemon.eq.sulfur = factor(disgust_data$lemon.eq.sulfur)
We now have a data column, called lemon.eq.sulfur, that labels the data so that lemon and sulfur have the same labels. We can use this in Bayes factor test:
bf2 = anovaBF(score ~ lemon.eq.sulfur, data = disgust_data)
bf2
## Bayes factor analysis
## --------------
## [1] lemon.eq.sulfur : 0.1921 ±0%
## 
## Against denominator:
##   Intercept only 
## ---
## Bayes factor type: BFlinearModel, JZS
The null hypothesis is now preferred by a factor of 1/0.192 = 5.2, which is expected given that lemon and sulfur were the least similar pair of three means. The null hypothesis accounts for the data better than this constraint.

One of the conveniences of using Bayes factors is if we have two hypotheses that are both tested against the same third hypothesis, we can test the two hypotheses against one another. The BayesFactor package makes this easy; any two BayesFactor objects compared against the same denominator — in this case, the intercept-only null hypothesis — can be combined together:
bf_both_tests = c(bf1, bf2)
bf_both_tests
## Bayes factor analysis
## --------------
## [1] condition       : 0.7738 ±0.01%
## [2] lemon.eq.sulfur : 0.1921 ±0%
## 
## Against denominator:
##   Intercept only 
## ---
## Bayes factor type: BFlinearModel, JZS

We could, for instance, put all equality-constraint tests into the same object, and then compare them like so:
bf_both_tests[1] / bf_both_tests[2]
## Bayes factor analysis
## --------------
## [1] condition : 4.029 ±0.01%
## 
## Against denominator:
##   score ~ lemon.eq.sulfur 
## ---
## Bayes factor type: BFlinearModel, JZS
The fully unconstrained hypothesis, represented by condition, is preferred to the lemon = sulfur ≠ control hypothesis by a factor of about 4.

In the next post, we will use the posterior() function to draw from the posterior of the unconstrained model, which will allow us to test ordering constraints.

84 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. Thanks for sharing this informative content that guided me to know the details about the training offered in different technology.
    digital marketing course in chennai | digital marketing training

    ReplyDelete
  3. I blog often and I genuinely thank you for your content. This article has really peaked my interest. I'm going to take a note of your ms 10 link site and keep checking for new information about once a week. I opted in for your RSS feed as well.

    ReplyDelete
  4. Very good blog you have to shared. It is very informative one. Thanks for your sharing.
    SEO services in chennai
    SEO company in chennai

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Thanks for sharing this useful information with us. Keep updating us with your awesome ideas.

    best seo company in chennai

    top seo company in chennai

    digital agency in chennai

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Thank Your for sharing your post.
    Digital Marketing Agency Chennai

    ReplyDelete
  11. It is very informative one. Thank you for Share this blog..

    Digital Marketing Agency Chennai

    ReplyDelete
  12. Nice blog thanks for sharing Instagram and Facebook have provided an amazing place for new brands to grow and flourish. We can find the perfect niche for your brand on the best social media marketing in chennai.

    ReplyDelete
  13. Effective blog thanks for sharing Set up a aesthetic work environment that employees love to spend time in and relieve their stress. Your company needs the best corporate gardening service in Chennai and Karuna Nursery Gardens in happy to oblige you in the endeavour to make your infrastructure something worth flaunting about.

    ReplyDelete
  14. Very effective blog thanks for sharing Cloudzsolution is the Best CRM Software in Australia. Cloudz organizes all the information in a centralized system. This Software enables data related to customers, contact details, etc. to be recovered easily.

    ReplyDelete
  15. The сlarity in уour pοst iѕ simplу excellent and і could easily find that yоu are аn expert οn this subject, keep up your act.home decorators in chennai , wallpapers shop in chennai

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. Thanks for sharing.
    http://thespicepeople.com
    https://allcottonandlinen.com
    https://allcottonandlinen.store

    ReplyDelete
  18. Thanks for sharing the best list of every quality site. I respect the valuable information you nitty gritty here.Digital Masala is a Digital Marketing agency in Bangalore I have been perusing the vast majority of them and a portion of my top picks , Digital Marketer, Moz, Buffer and Unbounce. Continue sharing some progressively informative stuff this way. Thanks!!

    ReplyDelete
  19. We provide support for all Roku models whether you have the oldest or the most recent one. One of the most common problems that people come across in their Roku devices is finding the correct fix Roku Device Overheating. If you are dealing with the same problem, then you can get all the help you need at our support facility

    ReplyDelete
  20. Nowadays the users of Roku are facing an error that can cause an obstruction in their live stream. That error is known as Roku Error Code 018. And get instant solutions from our technicians to fix this issue. Call us anytime on our toll-free number USA/Canada: +1-888-480-0288 & UK: +44-800-041-8324.

    ReplyDelete
  21. Call the Roku technical expert at Toll Free Number USA/Canada: +1 888-480-0288, UK: +44-800-041-8324, if you’re unable to fix Roku Errors anymore. We are one of the leading Roku technical support service provider and we offer our services round the clock. So whenever you need any help with Roku device, just call us on our toll-free number.

    ReplyDelete
  22. Thanks for sharing this useful blog. why is it trending and how does it work? Digital marketing is nothing but marketing a product and services or business using digital platforms to get favorable or good results. This is the best platform for large, medium, and small businesses to get an online identification and reach out to a large customer base thereby positively increasing the chances of driving new business. digital marketing in Chennai is one that provides all kinds of services to its clients so that the visibility of their client's website improves phenomenally on top search engines.
    digital marketing in Chennai | digital marketing in velachey | seo company in Chennai | seo in Chennai | best digital marketing company in Chennai | digital marketing services in Chennai | top seo companies in chennai | top digital marketing companies in Chennai | google authorised agency in Chennai

    ReplyDelete
  23. Thanks for sharing with us, this blog was usefull to me.SEM also known as Search Engine Marketing, is focused on increasing visibility of one's newly setup or a pre-existing website. You are able to improve your site traffic SEO and SMO service. If you want digital marketing agency then go with sem services in Chennai is the best place for it.
    sem services in chennai | social media marketing companies in chennai | top 10 seo companies in chennai | top digital marketing companies in chennai | digital marketing services in chennai | best digital marketing company in chennai

    ReplyDelete
  24. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    best digital marketing company
    digital marketing agency
    digital marketing services
    website design company

    ReplyDelete
  25. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates. data logger

    ReplyDelete
  26. Best Software Development Companies, Robotic Process Automation & Artificial Intelligence Software Services Zintzotek Provides Enterprise Resource Planning(ERP), Robotic Process Automation(RPA), Artificial Intelligence(AI), Customer Relationship Managemen(CRM), Computerized Maintanance Management System (CMMS) & Game software development Services in Hyattsville, MD, USA.
    visit: https://zintzotek.com/
    Contact: +1 201-781-6587

    ReplyDelete

  27. Graphic,Website designer,logo designs,freelance in Chennaiwe provide high-quality affordable criteria.Dedicated web services for UI design, website Development, Photography, Video Creation by producing exclusive designs that make them unique in the industry.
    Visit: http://graphicdesignerchennai.in/
    Contact: 9884341363

    ReplyDelete
  28. I agree, information is valid Wedding, Corporate, Birthday Party Catering Services in Kovilambakkam, Thoraipakkam, Pallavaram, Chennai Nala makes this exciting pre wedding ceremony a pleasant memory. Be it the food or Mehandi artist or music troop for Sangeet. Nala does it all.
    visit: http://nala.co.in/
    comtact: 9884623322 / 9841078122

    ReplyDelete
  29. Astrum InfoTech Agency is the Best Digital Marketing Company in Delhi, They help to generate the profit and visitors traffic on website. If you are looking for grow the online visibility of your business then please contact us. We Offer Best Digital Marketing Service like SEO Service, SMO Service, PPC Service, Facebook Marketing Services, Email marketing service, graphic design services, website designing service and website development service etc. https://www.astruminfotech.com/

    ReplyDelete
  30. These are common problems which can easily hp laserjet p1102w wireless setup. we should focus on problems that are connected with the ink cartridges because these are the ones that most people have no idea how to fix.

    ReplyDelete
  31. We are provide best digital marketing service and search engine optimization, social media optimization, adwords, email marketing, web design etc,..

    digitalmarkethubs

    ReplyDelete
  32. We deal with all brands and our Whirlpool Refrigerator technicians are well experienced in all type of Whirlpool Refrigerator in the market. We provide door to door Whirlpool Refrigerator service centre in Ahmedabad. We have our own transport facility for the customers and Ahmedabad Refrigerator trained all the technicians. Complaint your booking now to resolve your Ahmedabad Refrigerator issues.

    ReplyDelete
  33. This was website contain many things to study, Thank you for sdharing this.eCommerce photography in chennai

    ReplyDelete
  34. Amazing to peruse this sorts of blog, continue giving the new updates. Continue sharing Thanks... Regards Activated carbon manufacturing companies in India

    ReplyDelete
  35. Seo company in Varanasi, India : Best SEO Companies in Varanasi, India: Hire Kashi Digital Agency, best SEO Agency in varanasi, india, who Can Boost Your SEO Ranking, guaranteed SEO Services; Free SEO Analysis.

    Best Website Designing company in Varanasi, India : Web Design Companies in varanasi We design amazing website designing, development and maintenance services running from start-ups to the huge players


    Wordpress Development Company Varanasi, India : Wordpress development Company In varanasi, india: Kashi Digital Agency is one of the Best wordpress developer companies in varanasi, india. Ranked among the Top website designing agencies in varanasi, india. wordpress website designing Company.

    E-commerce Website designing company varanasi, India : Ecommerce website designing company in Varanasi, India: Kashi Digital Agency is one of the Best Shopping Ecommerce website designing agency in Varanasi, India, which provides you the right services.

    ReplyDelete
  36. Seo company in Varanasi, India : Best SEO Companies in Varanasi, India: Hire Kashi Digital Agency, best SEO Agency in varanasi, india, who Can Boost Your SEO Ranking, guaranteed SEO Services; Free SEO Analysis.

    Best Website Designing company in Varanasi, India : Web Design Companies in varanasi We design amazing website designing, development and maintenance services running from start-ups to the huge players


    Wordpress Development Company Varanasi, India : Wordpress development Company In varanasi, india: Kashi Digital Agency is one of the Best wordpress developer companies in varanasi, india. Ranked among the Top website designing agencies in varanasi, india. wordpress website designing Company.

    E-commerce Website designing company varanasi, India : Ecommerce website designing company in Varanasi, India: Kashi Digital Agency is one of the Best Shopping Ecommerce website designing agency in Varanasi, India, which provides you the right services.

    ReplyDelete
  37. The article unambiguously indicated every one of the positive and negative sides of the issue I'm exceptionally bewildered with the nature of the substance top 10 branding companies in chennai which you have written down. This is a marvelous article! Your article has all the essential data on the individual theme branding companies in chennai

    ReplyDelete
  38. Are you trying to cancel membership of your Amazon prime? Don’t know the process? If yes, then don’t look further than Alexa Helpline. Alexa Helpline is a group of expert technicians who can help you to cancel your Amazon prime free membership account. How to connect Roomba to Alexa

    ReplyDelete
  39. Website Development in Kolkata
    Thank you from gowebs.in
    https://www.gowebs.in/web-development

    ReplyDelete
  40. Want to know how to fix Kindle won’t connect to wifi error? Then get connected with us. Our team is very dedicated and skilled and will offer you the best solution. You will easily resolve kindle wifi connectivity errors. To know more visit Ebook Helpline.

    ReplyDelete
  41. If you are looking for a solution to fix canon printer won't connect to wifi error then you are at the right place. We are a team of experts who provide you the best solution to fix this error. We are available round the clock to help you. To know more visit Printer Offline Error.

    ReplyDelete
  42. Are you looking for someone who can fix your Echo device related issues? If yes, then you have reached the right place. Echo Helpline is a group of expert technicians who can fix your Echo device related issues in a short span of time. So hurry up and dial Echo customer service number UK: +44-800-041-8324 & US/Canada +1 844-601-7233.

    ReplyDelete
  43. nice blog. Thanks for the post. To reach as the same for https://i3diligence.com/

    ReplyDelete
  44. Nice Blog !
    QuickBooks is one such accounting application that is developed to serve all the accounting needs of the business. In case you need immediate help with QuickBooks errors, call us on QuickBooks Customer Service Number to get the best technical assistance for QuickBooks.

    ReplyDelete
  45. It is amazing and wonderful to visit your Blog.Thanks for sharing this information,this is useful to us. Keep posting!, If you face any trouble using QuickBooks, Contact:QuickBooks Customer Servicefor you solution.

    ReplyDelete

  46. Thanks for sharing this useful information with us. Keep updating us with your awesome ideas.In case if you need QuickBooks technical assistance, you may contact at:QuickBooks Support phone numberfor quick solution.

    ReplyDelete
  47. Your Blog is very nice, and it's very helping us this post is unique and interesting, thank you for sharing this awesome information.If you face any problem in QuickBooks, Contact:QuickBooks Support Phone NumberFor Quick resolution.

    ReplyDelete
  48. Thanks for sharing the informative post about.
    I have learned many new information from your post.If You want to play online mobile game.
    Visit now Game Development Kolkata

    Game Development Company in Kolkata,
    Kolkata Game Developers,
    Game development Kolkata,
    List of Game Design Courses,
    mobile game development company in india,
    Game development company in India
    Game Development Kolkata
    https://www.gowebs.in/game-development-kolkata

    Other Services :
    web design in Kolkata
    https://www.gowebs.in/web-design-kolkata

    web design Howrah
    https://www.gowebs.in/website-design-in-howrah

    web design Baripada
    https://www.gowebs.in/web-design-Baripada



    ReplyDelete
  49. 4. Love to read this amazing and helpful post.
    I specially congratulate you for this kind of post.
    Satta Matka Game App Development
    Satta Matka Game App Development
    https://www.gowebs.in/satta-matka-software-development

    ReplyDelete
  50. Are you facing issues regarding Orbi device setup and login issues? looking for a solution to fix this with Orbilogin.net / www.orbilogin.net / www.orbilogin.com/. Then visit our website now and get the solution to fix this error or you can get help form the experts, our experts are available 24*7 to fix your issue. Orbilogin.net

    ReplyDelete
  51. Get Your Website to Top-10 on Google With Backlinks from High PA-DA Sites and Guaranteed Results.

    Moz
    SEO For Blog
    SEO For Website
    SEO For Business
    Selfgrowth

    ReplyDelete
  52. Kent Pecoy, Owner of Pecoy Homes - The Premier Residential Home Builder and Remodeler in Western Massachusetts, Cape Cod, and Northern and Central Connecticut.

    ReplyDelete
  53. Thank you for your informative post!!!
    Village Talkies a top-quality professional corporate video production company in Bangalore and also best explainer video company in Bangalore & animation video makers in Bangalore, Chennai, India & Maryland, Baltimore, USA provides Corporate & Brand films, Promotional, Marketing videos & Training videos, Product demo videos, Employee videos, Product video explainers, eLearning videos, 2d Animation, 3d Animation, Motion Graphics, Whiteboard Explainer videos Client Testimonial Videos, Video Presentation and more for all start-ups, industries, and corporate companies. From scripting to corporate video production services, explainer & 3d, 2d animation video production , our solutions are customized to your budget, timeline, and to meet the company goals and objectives.
    As a best video production company in Bangalore, we produce quality and creative videos to our clients.

    ReplyDelete
  54. Nice Blog !
    Get quick and effective assistance for QuickBooks POS Error 100060.We provide quick and effective service to every QuickBooks client.

    ReplyDelete
  55. Thank you for giving important information. This information is very much important for me. If you are looking for ecommerce agency Ireland then contact the Smarketing Cloud.

    ReplyDelete
  56. digital agency in ahmedabad


    As a leading best digital marketing company in ahmedabad, we will help you outperform the competition. Right from search engine optimization.


    to get more - https://growyourbusinesstoday.in/

    ReplyDelete

  57. We provide the Alexa Echo customer services and resolve the Alexa problem such as installing Echo Setup, Alexa Echo offline, Amazon Echo won’t connect to wifi, Alexa connectivity to Wifi and more. Contact anytime our Echo helpline.

    ReplyDelete
  58. Thanks for sharing beauty diy providing a best Skin and hair care products in chennai. Run your saloons and spa tying up with the best cosmetics shop in chennai we also have Pedicure/Manicure kits,Serums,Fairness Night Crème,Spf,Hair serums, 100% Pure Argan Oil,Natural Body and Face washes,Hair Wash etc…Quality is the only Motto we have!

    ReplyDelete
  59. Nice Blog on Bayesfactor... Keep Sharing

    Niraj Panchal

    Nice Blog on Bayesfactor... Keep Sharing


    First DigiAdd

    ReplyDelete
  60. best accounting software for small business in india
    thank you for this excellent information . it is very helping and am learning many things . you are doing good. thank you so much

    ReplyDelete
  61. Find the best-fixed mortgage rate in London that works perfectly for you. We make it easy to compare rates in London big banks and top brokers for free. Best mortgage rate in London

    ReplyDelete
  62. Nice blog, thanks for sharing it.
    Here I am sharing our link for people who are looking for Top Advertising agencies in Chennai.

    ReplyDelete
  63. This blog post is a goldmine of information! The tips and strategies shared here are incredibly helpful.

    And when it comes to web development, Maven Technology is named as the Best Web Development Company in Noida, your go-to solution for all digital needs.

    Our innovative solutions, top-notch designs, and flawless execution set them apart.

    Thanks..!!

    ReplyDelete
  64. Hey It was very nice content on BayesFactor.

    Visit Chennai Best Digital Marketing Agency Chennai

    ReplyDelete