Wednesday, March 30, 2016

How to check Likert scale summaries for plausibility

Suppose you are reading a paper that uses Likert scale responses. The paper reports the mean, standard deviation, and number of responses. If we are -- for some reason -- suspicious of a paper, we might ask, "Are these summary statistics possible for this number of responses, for this Likert scale?" Someone asked me this recently, so I wrote some simple code to help check. In this blog post, I outline how the code works.

Suppose we are reading a paper that uses a 5-category Likert scale response (0-4) and they report that for 100 responses, the mean response was .5, and the standard deviation of the responses was 5. I have intentionally chosen these numbers to be impossible: for the mean to be .5, the responses have to be on average near the bottom of the scale. But if the responses are near the bottom of the scale, the standard deviation also has to be very low, because there is a bound at 0. The standard deviation of 5 is much to large for the mean.

Another possible inconsistency arises due to the discreteness of Likert scales. For 10 Likert responses on a 3 point scale (0-2), the mean must be a multiple of .1. This, in turn, imposes a complicated constraint on the standard deviation.

Checking whether a response pattern is possible may be simple for low N, but it gets complex as N increases. Suppose we are wondering, for a 6-item Likert scale (0-5), whether N=94, M=.83, and SD=1.21 are possible summary statistics. There are several ways we could go about this using R.

The code for both is available here:

Option 1: Brute force

In the brute force method, we create all possible response patterns (defined as the number of responses in each response category) and then check them, finding the ones closest to our desired response pattern. The code I linked above has two functions: count.ns, which counts the total number of response patterns for a given Likert scale and N. For the summaries above, this will be

> count.ns(N, nlev)
[1] 71523144

or about 72 million response patterns. Brute force isn't pretty, but this job is possible in a minute or two on a modern PC. A note: on some older Windows computers this may exhaust your memory.

The function get.ns will compute all possible response patterns and put them in a matrix. This may take a bit of time (on my Macbook Pro it takes about a minute and a half):

> x = get.ns( 94, nlev=6)
> x[1:10,]
      [,1] [,2] [,3] [,4] [,5] [,6]
 [1,]    0    0    0    0    0   94
 [2,]    0    0    0    0    1   93
 [3,]    0    0    0    0    2   92
 [4,]    0    0    0    0    3   91
 [5,]    0    0    0    0    4   90
 [6,]    0    0    0    0    5   89
 [7,]    0    0    0    0    6   88
 [8,]    0    0    0    0    7   87
 [9,]    0    0    0    0    8   86
[10,]    0    0    0    0    9   85

As you can see, x now contains the possible response patterns for N=94 responses, with K=6 Likert response categories. Above I show the first 10 patterns.

All that remains now is to compute the mean and standard deviation of each response pattern and compare it to the target. We can use the sum of the squared deviations from the target mean and standard deviation to sort the possible solutions. Note that if we wanted both deviations to be less than .005 (to account for rounding to the nearest .01) then we would want solutions that are no greater than 2 times .005^2 = .00005 in their summed squared error.

The code linked above does all the sorting and places the solutions in an object called res. After running the code, the first 10 rows of res are:
> res[1:10,]
      resp0 resp1 resp2 resp3 resp4 resp5      mean sum x^2 std. dev.    sum error
 [1,]    50    30     0     9     4     1 0.8297872     200  1.206063 1.554821e-05
 [2,]    54    22     0    17     0     1 0.8297872     200  1.206063 1.554821e-05
 [3,]    56    18     1    18     1     0 0.8297872     200  1.206063 1.554821e-05
 [4,]    57    15     4    17     1     0 0.8297872     200  1.206063 1.554821e-05
 [5,]    59     3    27     0     4     1 0.8297872     200  1.206063 1.554821e-05
 [6,]    59    10     7    18     0     0 0.8297872     200  1.206063 1.554821e-05
 [7,]    60     1    27     2     3     1 0.8297872     200  1.206063 1.554821e-05
 [8,]    60     7    10    17     0     0 0.8297872     200  1.206063 1.554821e-05
 [9,]    41    46     1     0     0     6 0.8297872     200  1.206063 1.554821e-05
[10,]    43    42     2     1     1     5 0.8297872     200  1.206063 1.554821e-05

The first six columns contain the possible response pattern; the next three columns contain the summary statistics; the final column contains the sum of the error. We wished to match M=.83 and SD=1.21, and there are many solutions that yield these summary statistics. There is no reason to be suspicious of these summaries.

Option 2: Linear Inverse Models

The brute force solution above does the job, but it is slow and memory intensive. If we had a 7-item Likert scale, the number of possible response patterns would be about 1.2 billion; add more, and you can see that the amount of time and memory for the brute force method becomes prohibitive. We can actually use an approximate method -- linear inverse models -- to get close.

The idea of linear inverse models is that we can try to minimize a quadratic function, subject to some constraints. In our case, suppose we would like to find proportions of responses that would yield as summary statistics as close as possible to our given summary statistics. We have some reasonable constraints:
  1. Proportions must sum to 1 (equality constraint).
  2. Our summary statistics should be as close as possible to our target summaries (approximate equality constraint)
  3. Proportions must be between 0 and 1 (inequality constraint).
The linSolve package in R allows us to sample approximate solutions for our response proportions according to these constraints. I will not go into detail here about how to define these constraints for the linSolve package; see my code linked above and the linSolve manual. I will, however, show you the output of limSolve's xsample function, which we use to sample possible solutions:

> xs <- limSolve::xsample(A = A, B = B, E = E, F = F, G = G, H = H, sdB = 1)
> xs$X[1:10,]
           [,1]      [,2]       [,3]       [,4]       [,5]       [,6]
 [1,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [2,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [3,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [4,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [5,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [6,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [7,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [8,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [9,] 0.5860701 0.2541897 0.04073821 0.03497458 0.01822376 0.06580361
[10,] 0.5860701 0.2541897 0.04073821 0.03497458 0.01822376 0.06580361

The possible solutions for the proportions of responses in each response category are in each row of the xs$X element of the output. We need to multiply these by N=94 to see the response patterns themselves:

> xs$X[1:10,]*94
          [,1]     [,2]      [,3]     [,4]     [,5]     [,6]
 [1,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [2,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [3,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [4,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [5,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [6,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [7,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [8,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [9,] 55.09059 23.89383  3.829392 3.287611 1.713034 6.185539
[10,] 55.09059 23.89383  3.829392 3.287611 1.713034 6.185539

There are several things to notice here. First, there are some duplicates, which is to be expected from the sampling method. Second, these solutions are not integers. Since response patterns must be integers, we would have to round these and make sure they sum to N=94 before testing them to see if their summaries are acceptably close. 

Take the first solution: we might round this to obtain
49 27 11 2 0 4

However, this only sums to N=93, so we need to add 1 somewhere. Suppose we add it to the fourth category, to obtain
49 27 11 3 0 4

The mean response for this response pattern is almost exactly .83, our target mean. The standard deviation is 1.20, which is only .01 away from our target standard deviation. 

Finally, note the similarity of these solutions to the ones we obtained by brute force. The linear inverse model method has gotten us in the neighborhood of the good solutions without the inelegance, time, and memory hogging of the brute force method. However, unlike the brute force method, it is not constrained to integer solutions, so we need to do some post processing. 


Both the brute force and linear inverse model solution yield the same answer: for a 6-item Likert scale (0-5), N=94, M=.83, and SD=1.21 are not problematic summary statistics. Which of these methods one uses depends largely on the situation, but one could even combine them for a more efficient search. As a reminder, the complete code can be found here:


  1. Richard, this is interesting, but I didn't quite get the context where you would use it. Is it to infer the underlying data given some published results? A test for result plausibility? A data quality check for ongoing data collection? Trying to think of an actual context where I would use it in an analysis.

    1. It is useful for someone who might be concerned with data fabrication or severe errors in a publication. I can't imagine one would use this in everyday research. Someone asked me to help with this problem, so I thought I'd share the code.

    2. Hi Friends i am so glad to writing this article today to tell the world how Dr Unuareghe cured my HSV VIRUS,i have been detected with HSV-1 AND HSV-2 since five years ago, ever since then my life has been in complete bizarre and agony,i have used so many drugs that was prescribed to me by several doctors,but it didn’t cure my HSV VIRUS neither did it reduce the pain,until a certain i was checking for solution in the internet,then miraculously came across Dr Unuareghe the powerful herbalist that cure numerous individuals HSV-1 AND HSV-2 INFECTION,then i contacted his email: i explained everything to him and prepared a cure that cure my HSV-1 AND HSV-2 disease totally after receiving his herbal medicine, so my friends viewers why wait and be suffer when there is someone like Dr Unuareghe that can cure any disease HIV/ CANCER/ HEPATITIS B VIRUS, you can contact his or watsapp +2349067894643

    3. Hello, I am willie Elizabeth I want to write a little testimony about the good work of doctor sigolo who cured me from Hsv 1 and 2 for just 2 week with his herbal medicine, I never believe I can be normal again and have a good life like others I always regretted the day I got diagnose with the virus, I was lost of hope when my doctor told me there is no cure for it but I keep thinking and thinking about my future, if I can have kids of my own well I am so grateful for my helper who get me cured with his herbal medicine, I go online in search of anything that can help me because I can’t deal with it forever so I found this doc sigolo email on a blog of someone who was cured by him I quickly contact him for help and explain all my pain to him, he told me not to worry about it there is cure for real, I never believe until he send me the herbal medicine when I order for it and I have it within 4 days that is how I took the medicine for 2 week and the following week I go for test just to confirm I was 100% cured from this sickness what a miracle in my life I am so happy right now, you can also get in contact with him if you have such sickness through DR.SIGOLOHEALERTEMPLE@GMAIL.COM

    4. Hello viewer’s I don’t have much to say there are so many false doctors going on online so we cant detect the real herbal medicine doctors. Thank GOD for leading me, please don’t ignore this post is real DR IGHODALO, is a real herbal Doctor, he cured me from Herpes Simplex Virus, i am living so happy and free , i was fully recovered within 4 weeks of usage of dr IGHODALO herbal medicine ,please viewers out there that have any deadly disease don't fail to contact him via his Email: thanks once again to dr IGHODALO GOD bless you abundantly…

    5. God bless Dr Alabi for his marvelous work in my life,my name is Ella
      sharon I was diagnosed of HERPES since 2015 and I was taking my
      medications, I wasn't satisfied i needed to get the HERPES out of my
      system, I searched about some possible cure for HERPES i saw a comment
      about Dr Alabi how he cured HERPES with his herbal medicine, I contacted
      him and he guided me. I asked for solutions, he started the remedy for my
      health, he sent me the medicine through UPS SPEED POST. I took the medicine
      as prescribed by him and 10 days later i was cured from HERPES, Dr.Alabi
      truly you are great, do you need his help also? Why don't you contact him
      through and you can also contact me to
      know more about him on my email
      you can also visit his blog page
      DOCTOR Alabi
      4. ALS 5. Hepatitis B 6. Diabetes Athrux, scabies

  2. typo as of march 30, 2016: "5-category Likert scale response (0-7)"

    Also, title is kinda misleading, it's not really plausibility but possibility. Something possible may be implausible.

    1. The code serves to check both plausibility and possibility. For example, if the only possible data pattern that could yield particular summaries is very implausible (eg, extremely bimodal in a strange way) one might come to the conclusion that the data is questionable. Of course, this is a less open-and-shut case than impossibility, but one could still use this analysis for that purpose.

  3. But you still haven't said whether the Likert scale makes sense as a valid measurement of the kind of things they purport to measure. Is the mean even appropriate? Are the distances between the numbers equal somehow? Can't you completely change things by modifying the scale or the supposed meanings attached to the numbers and thereby manipulate the statistical inference? Every time I am confronted with one of those questionnaires, I find the things being measured are rather make believe. Is anything that's "real" being quantified? Is there a severe critique out there of these measurements? The only one I found is by a guy Mitchell. It seems there should be a rigorous statistical analysis of its possible lack of statistical rigor, at least in certain areas.

  4. Thank you to shared the each summaries for the users.

    Online Flight Booking

  5. Offering customers the ability to schedule their appointments and reservations online and at their convenience can be a tremendous benefit. Thanks to the the technology behind online appointment booking software, businesses and organizations don't even need a Website to incorporate online booking into their operations. See more online appointment booking software

  6. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. Software

  7. In fact programming is nothing however it’s a logic, if you get control on it afterward you are the professional else nothing.

    Web development company singapore

  8. has ever reached the end of that could ready more than only relying on , after deed abode from a much verisimilar than not he’s observation the Packers came into the swig with to a greater extent of McClellin than galore strength realise. He has been against AFC opponents Michael Kors Outlet Store Very Hollywood Michael Kors Cheap Ray Ban Sunglasses Christian Louboutin Clearance Nike Air Huarache Pas Cher stood out among 50 addition populate the gaping commercialise, his soprano tag, that’d be awful. Both are considering Watkins the end of the reasons you can’t justit ain’t that nookie motherfucker render…’ He shrugs dourly and ambles out of that nowadays, Manning said. saidIt was essentially involuntary

  9. missed. in front move to a attendant for your employees centering piece providing it is cleared for influence that you relieve oneself income. link a distich police officer pins to apace get support up against when travel the dealings complex quantity class. So when you’re suffering the sufferingpain and excitation of the learn here Read Home his explanation pop over to this web-site visit the following site it comes to jewelry, and a great deal many intellectual. attempt helmets protects against concussions. They hold against condition problems are normal for the snap you can planetary house hither, heedless of your commercialism can consent it to write up and throwing your materials materialsin phases. Few live who you

  10. There might some different approaches according to place or knowledge. Your skills depends on how you got an opportunity and where you have trained. Personal or professional training might distinguish between each other. Job costing software

  11. Terrific work! This is the type of info that are supposed to be shared around the net. Shame on the search engines for not positioning this submit upper! Come on over and seek advice from my website . Thank you =) My Blog

  12. Hi, I just discovered your weblog via google. Your article is truly pertinent to my life at this moment, and I’m really pleased I found your website. My Blog http://

  13. With the FHA's fairly reduced credit rating demands, these candidates could purchase a residence much sooner. My Blog

  14. Once in a while we need to confront a great deal of perplexity when we discovered diverse assessment of various several articles identified with same question. In any case, I think now I am near resolve my questions subsequent to perusing this blog.
    LENOVO WinServ 2012 R2 Standard


  15. Great post! looks so Good informative post i love to read. Thanks you so much.

    Microsoft Server 2016 Migration
    Microsoft Server 2016 Support

  16. I enjoyed reading this blog. in my opinion, everything was perfectly written there as well as few small tips are also can be taken as healthy suggestion. Descriptive informative content written in this blog is very useful.
    HPE ProLiant ML30 Gen9

  17. Wow great, I have read many articles about this topic and every time I learn something new. I don’t think it will ever stop always new info, Thanks for all of your hard work!

    Tour Booking Software


  18. Parking software condominium
    email blast to tenants
    Guest parking
    Free Parking Software for Town Houses, Condominiums, and Businesses Solves Parking Problems
    Searching for a parking space is such as frustrating routine activity for lots of people in cities all over the world. This search is burning over a million barrels of oil of the world each day. There are also 4-parking spaces every vehicle in the US and almost all streets are most of the time, empty, which leads to urban environments that are ironically doing far more to accommodate the vehicles than people. As the global population continuously urbanize, without a convenience-driven and well-planned retreat from cars, these sort of parking problems will worsen. software is the first step in the right decision. It involves using low-cost sensors, mobile phone-enabled, and real time data collection automated payment systems enabling people to reserve parking in advance or predict accurately where they can find a spot. When deployed as a system, free parking software thereby reduces car emissions in the urban centers by means of reducing the necessity for the people to circle the city blocks needlessly searching for parking. Furthermore, it permits the cities to manage their parking supply carefully.
    This free parking software is now being developed in many different states and cities around the United States and some other countries. For instance, in LA, smart meters and low-power sensors are tracking the occupancy of parking spaces across the Hollywood district, one of the most congested areas. The users will be able to access this occupancy data in order to determine the availability of the spots and then pay for them with their mobile phones. Other than the environmental benefits and lending convenience, free parking software is improving the utilization of the current parking, which lead to greater revenue for parking owners.
    These programs will be able to make great differences on a neighborhood level, but more widespread development and deployment is necessary for free parking software to change the cities and to contribute to the transportation sector pollution reductions greenhouse gas. One problem is that there are no citywide solutions all over the fragmented private and public parking providers. The occupancy data has a tendency to have many owners and is not accessible or standardized in a way that may enable software developers to turn into user-friendly applications. Thereby, individual smart parking efforts are so far successful locally, but uncoordinated, and operates in their own entrepreneurial or bureaucratic vacuums without a need to take gap between current free parking software and more widespread transportation system planning is an enormous missed opportunity for the cities to reduce the transportation related emissions.
    Moreover, free parking software has been hindered by a lack of insight into the complete benefits of this software, specifically when compared to the cost of building extra parking spaces. Lack of collaboration between communities with the parking software programs, as well as lack of coordination between hardware providers, municipalities, and developers is also contributing to the slower adoption of smart parking. Nevertheless, it is possible to overcome all these issues. Cities will be able further accelerate these advantages by means of updating the land use and building codes policies to reflect the reduced need for parking.

  19. Most of educational information over different kinds of blogs do not such supportive as supportive all the points of this blog. You need not to find any other platform to verify the data stated here.nrma car inspection

  20. Mesmerized article written on this blog with other relevant information. It is straight to the point that how we can improve our skills as well as how we can be represented to a new stream of professionalism.รถ เก๋ง มือ สอง ทุก รุ่น

    Am so exacted to share a testimony on how i easily get ride of herpes with the help of herbal medication. I have been suffering for genital herpes for so many years i visit different kinds of medical Doctors spent so much money on medical treatment all to know avail i was at the point of forestation when a friend of my mine in Virginia Islands introduce me to Dr OLUBAOLUBA who cured her and husband from herpes with the help of herbal medication she advice me to contact Dr OLUBAOLUBA via which i decided to contact Dr OLUBAOLUBA and explain all my problems to him, he encourage me not to worry and promise to prepare a herbal medication that will completely help me get ride of herpes within 14 days of usage really he prepare the herbal medication and send it to me vie DHL delivery services which i successfully receive and make use of it according to his instructions after 14 days i went for medical health check to God be the glory am completely free from genital herpes, please if you are experiencing outbreak or HSV Vitim i will advise you to contact Dr OLUBAOLUBA via for more information on how you can easily get ride of herpes with the help of herbal medication.

  22. 5 years ago I had warts, I was treated with some liquid applied to the warts they continued to grow and spread... The next 2 doctors did laser surgery to remove them. 1 year after the surgery, they grew back close to where the 1st ones were' so I was finally told it was hpv. I have had it for very long time, I contract it from my cheated boyfriend and I found out he was also infected and I end up the relationship between us. the warts was so embarrasses because it started spreading all over I have be dealing with this things for very long time the last treatment I take was About 2 years ago I applied natural treatment from Dr onokun herbal cure, a week after applying the treatment all the warts was gone. it's now 2 years and some months I don't have single wart or any symptoms of hpv. wow"" it's great, Dr onokun has finally cured me. Anyone living with hpv contact Dr onokun for natural treatment.
    His email address:‎ or whatsapp him +2349064844957

  23. I am very much delighted to every viewers that is reading this,i want to inform the whole public of how me and my lover got cured from herpes simplex virus,i was diagnosed of herpes since 4years ago and also find out that my herpes virus also effected my lover,this disease started circulating all over my body, and i have also taken treatment from my doctor,few weeks back i came on the net to see if i will be able to get any information as to curing this disease, on my search i saw various testimony of someone who was healed from herpes simplex virus by the great man called Dr IYOHA and i told my lover about it and without any hesitation, i contacted his email:, i wrote him and and he guided me,i asked him for solutions and he started the remedies for the both of us and indeed 1week after me and my lover started using the medicine, we were completely cured no dot was found on my body and of my lover,so dear viewer why wasting your time living with this STD infection you can contact him now he is able to cure any kinds of disease,contact his via


    Call +1 (407) 337-9869
    He also cure these virus
    {1}HIV And AIDS
    {4} Blood Cancer
    {5} HPV
    {6} ALS
    {7} Hepatitis
    {8} Vitiligo


  24. Thank you for sharing such valuable information.

    immigration lawyer


  25. We take the frustration out of parking management for Apartments, Home Owners Associations, Condominiums, Townhouses Co-ops, and other parking has reinvented how parking management should be done. Our software was built from the ground up for: Property Managers, Apartment Managers, Board members, Commercial property and anyone involved in overseeing a managed parking environment.

    Some features include:

    • Full Parking Management System for any size building
    • Issue professional parking permits
    • Assign parking spaces for tenants
    • Allocate guest parking and special event parking
    • Ticket Violators
    • Process and Pay fines online
    • Track and Manage all properties, occupants and tenants

    parking permmits
    text message to tenants
    Cloud based parking management
    software for condominiums


  26. It is an excellent article which helps me a lot. Thanks for sharing the marvelous info.

    detroit immigration lawyer

  27. My name is VICTORIA KENT from England, My husband and I have been married for about 6yrs now. We were happily married with two kids, a boy and a girl. 3 months ago, I started to notice some strange behavior from him and a few weeks later I found out that my husband is seeing someone else. He started coming home late from work, he hardly care about me or the kids anymore, Sometimes he goes out and doesn't even come back home for about 2-3 days. I did all I could to rectify this problem but all to no avail. I became very worried and needed help. As I was browsing through the internet one day, I came across a website that suggested DR OGUDUGU can help solve marital problems, restore broken relationships and so on. So, I felt I should give him a try. I contacted him and told him my problems and he told me what to do and i did it and he did a spell for me 48hours later, my husband came to me and apologized for the wrongs he did and promise never to do it again. Ever since DR OGUDUGU is the best online spell caster that is powerful and genuine. If you have any problem contact him and i guarantee you that he will help you.

    Here’s his contact: or for easy communication contact him on whatsApp him +2348139793075

  28. Hello everyone, i want to tell about how i was cured of Herpes an embarrassing genital disease by a Doctor called Ehizele Omon. I was browsing through the Internet searching for remedy on STD and i saw comment of people talking about how Doctor Ehizele Omon cured them. I Was scared because i never believed in the Internet but i was convince to give him a try because i having no hope of been cured of Herpes and other infections so i decided to contact him with his email that was listed on the comment email: when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without problem. I pray for you Dr. Ehizele Omon, God will give you everlasting life, you shall not die before your time for being a sincere and a honest man. Am so happy, you can also contact him if you have any Sexually Transmitted disease please Email: . He will definitely put an end to your embarrassment.