Why Surveys Suck

Well, not all surveys suck (necessarily). But the more surveys I see, the more I doubt the value of any of them. Why? Because there seems to be an exponentially increasing use of survey technology without any understanding of how statistics work – and very often no brains behind it. These surveys are less than worthless; they are merely pretending to be something they are not. In the wrong hands with the wrong interpretation, these surveys are dangerous as they pretend to represent some kind of truth – while in reality they are lies.

Click Here To Download A Free Ebook: “23 Stats About Social Media You Should Know In 2016”

Let me give you some examples:Why Surveys Suck: The more surveys I see the more I doubt the value of most of them. Without any understanding of how statistics work most surveys are less then worthless.

I recently came across a German News article about a study that claimed that people who trained long distance running regularly would not be healthier than people who didn’t. (I am not a doctor, and I am not a sports scientist, I do not claim to know if running is good for everyone – I feel better if I go running regularly). But I am a mathematician. And I seriously doubt the value of a study that examines the data of 413 non-jogging people and a few sentences further down states that over the course of the study of twelve years 128 of the not running people died. Sorry guys, but there seems to something seriously wrong with your sample if over a quarter of them died.

When I went to university, a magazine did a ranking of German universities. The university I went to totally sucked at computer science. I wondered why they were rated so bad and took a closer look. One of the factors the ranking measured was the number of students that graduated each year. The computer science program at my university was new – barely two years old, naturally, there were no students graduating – yet. – Seriously?!

Or what about a company asking their online (!) community what their favorite source of information was. Does it surprise you that the answer was online (!) media.

Also, in Germany, there seems to be an over-fondness for using surveys in student theses especially concerning startups and entrepreneurship. There used to be times when we received, at least, one request to fill out a survey per day. 99% of the ones I even looked at told me simply by looking at the questions that they were not worth the effort, as they would never answer any questions in any realistic way. That’s why I decided not to bother with answering them anymore. These students can go and waste someone else’s time.

Do you want to learn how to grow traffic, leads, and customers from social media? Check out our ebook “The Social Traffic Code!”


Surveys don’t need to suck

Do not get me wrong: We do use surveys to get answers from our audience, and there is nothing wrong with doing this as long as you understand the basics of statistics and invest some brains in what kind of answers these surveys give you and what they can never tell you.

Between mathematicians, there used to be a saying: Statistics are always as good as the person faking them. Meaning: If done well, you can fool many people with statistics and assume things they do not imply by hiding some facts like who you asked and how you twisted your questions in the direction you want to.

With the easy to use new online possibilities for throwing out surveys and getting people to answer them, no one seems to care anymore about a basic understanding of statistics before putting up a survey.

If you want to use surveys for your business, be careful the surveys do not fool you instead. They can easily deceive you into following a path that is totally wrong for you, by disguising facts and hiding the truth behind seemingly plain statements while hiding the real answer behind a wrong audience, or badly posed questions. You need to be sure if what and who you are asking is answering what you are trying to find. Getting valuable information out of a survey means more than getting a handful of people to answer a handful of simple questions. There is much more to it.

You need to be clear about whom it is you want to ask, and if you even know a way to get exactly these people to answer your survey. You need to be very careful to phrase your questions without any implication of the answer you want to get and if not a totally different interpretation of the answers would also be realistic. And you need to ask yourself if the number of answers you got is enough to cover possible exceptions from the rule.

And if you just want to use your survey as part of your marketing efforts and don’t care about how correct the results are – please fake it in a way that isn’t that obvious to me. Customers want to be taken seriously!

  • http://makingmoneyfastandslow.com/ Bobby

    As long as you have a completely random sample and remove as many biases as possible, surveys can be a powerful way to get a pulse on your customer base.

    • Candice D-Blackman

      That’s just it though. Getting a completely random sample requires getting the sampling frame correct which can be pretty expensive and time consuming exercise. I believe in doing surveys the right way but it can be out of reach for start-ups and small businesses. How does one get around that?

  • Pete Austin

    Great examples, mostly about “Sampling Bias”. I share your pain.

    Here’s a blog post I wrote about the top ways in which online statistics can mislead…

    And @Bobby: It’s more complicated than that. As a minimum, if you care about the results from any science experiment (including surveys) you need to repeat them. It’s amazing how many results go away when you try to get confirmation, because they were really only random effects.

  • http://thinksem.com thinksem

    Surveys can definitely be deceiving! Nice article

  • Echo it

    Surveys are closed off, like email, and do you know who (if anyone) is reading your surveys?
    How often do you hear back?
    Do you know if your survey ever changed anything?
    Do you even have any of these expectations or are we all just wasting our breath?

    We created a platform called Echo it, where consumers everywhere can simply echo (as easy as tweet) their suggestions, concerns or ideas to brands so they can improve. The feedback becomes social and open, so others can support it.

    Now you can see if other people already suggested what you had in mind, or have experienced the same problem you have, so you can easily unite by Re-Echoing. The company now can see just how many people want to see a certain improvement/change/addition, etc…

    We want to empower and enable consumers to influence brands and truly get their voices heard, and we hope to see your echoes too! 🙂

  • http://robynwilliamswriter.blogspot.com.au/ Robyn Williams

    Great article guys.

    There does seem to be a rapid rise in the use of online surveys as a tool for lead generation.

    They give me the shits.

    There is a time and a place for surveys; for data generation, especially when dealing with such things as population demography.

    But the shallow tabloid style surveys are pointless, as you point out.

    I am enjoying reading your articles. Keep them coming.

    Robyn Williams

  • http://blog.responster.com Brandon Landis

    Yikes. I write for a company whose primary product is survey software and I cringe regularly at how poorly many surveys are conceived. Questions that lead, samples groups that were doomed to deliver useless results, and question progression/structure that’s so annoying respondents just give up halfway through.

    I think a big part of the problem is that people love to reverse engineer their own assumptions by putting out a survey after they’ve already formed a hypothesis, article idea, etc., and then (whether consciously or not) creating a survey that is likely to give them the results/outcomes they were looking for in the first place, regardless of how objectively accurate they are or are not.

    We tried to design something that bypassed the ease-of-response barriers (shameless plug: http://responster.com), but the other stuff is learned and earned and no amount of helpful survey software is going to save you from yourself (i.e. you’ve just got to put in the time and know-how to avoid biases and get anything actionable and useful back from a survey).