Falsifiability or Falsehood in Dog Training? (Part 1)

What if we had to know our animal training theory and practice so well that we could easily tell someone what would disprove the hypotheses that inform our methods? That’s what scientists do. If we are going to claim to base our training methods on science, I think we should get with the program. 

There’s a concept in science that is not much discussed in the world of dog training. The concept is falsifiability. Learning about it can save us a world of hurt in assessing statements about training methods. Focusing on how we would disprove our own methods may seem counterintuitive at first, but bear with me.

All dog training methods are based on science because the processes involved can be explained by science. But only some trainers actually study the science and base their decisions on what they learn from it. For the purposes of this article, I’ll call them science-based trainers.

Whatever they call themselves, in my opinion, anyone who says they base their training on science should be able to explain how they do it. They should also be able to tell you what would falsify, i.e., disprove, the hypotheses they use.

Falsifiability and The Scientific Method

The Scientific Method: photo courtesy of Wikimedia Commons

Falsifiability, also known as testability, is the ability of a hypothesis or statement to be scientifically disproven, and is intrinsic to the scientific method. Any researcher who puts forth a hypothesis should be able to state ways it could be falsified, and should expect those methods to be attempted in the future. Many researchers perform repeated experiments to attempt to falsify their own findings. I admire these people who are willing to put their ego investment on a back burner to seek ever more accurate information.

Jerry Coyne, author of Why Evolution Is True, has a post on his website that lists seven items that, if observed and confirmed, would shoot holes in parts of the theory of evolution. Taking a look at his article will give you a good grasp of what falsifiability is. It will also demonstrate the high level of expertise in a subject needed to understand how to falsify it.

(Falsifiability is not used as a metric in some sciences, such as physics and astronomy, because some hypotheses can’t be tested. For instance, much of string theory is famously unfalsifiable. That doesn’t mean that the hypotheses involved are accepted without rigorous examination though. In the main we don’t have those kind of problems in behavior science)

The scientific method gives us a good guide for questioning dog trainers, their methods, and statements about them. Every training method involves one or more hypotheses about cognition or ethology. You can use this concept of falsifiability to ask someone about their overall philosophy or just a particular method. Here’s how you can go about that.

  1. Identify the training challenge or part of their philosophy you want to discuss.
  2. Ask the trainer what the general research is behind their approach. They don’t need to name specific studies or textbooks. But they should know the concepts and be able to explain their relevance.
  3. If discussing a method within a training system, ask the trainer if that specific method has any specific research behind it, i.e., has it been scientifically tested?
  4. If the method has been tested, ask the results and assess the evidence.
    1. If the method has been tested and found to be successful, ask whether the results have been replicated by other research.  Replication is essential. Hanging on the results of one study is not good practice when other studies are possible and ethical to perform.
    2. If the method has not been tested, ask how they would design an experiment and what would falsify the method. Again, there doesn’t have to be a lot of detail. But this will show whether they understand the claim they are making.
  5. Whatever the answers to the above, ask them what it would take to disprove their method or philosophy. 

The last thing is perhaps the most important. You are not only putting the trainer’s knowledge to the test, but also potentially running up against their ego. We all get attached to our methods and an ego response is natural. But pursuing falsifiability forces one to override the ego.

Remember, we can’t “prove” a hypothesis. What we hope to do when we research it is to amass evidence for it. But we can disprove it by finding examples within the scope of the hypothesis in which the outcome is not as predicted. In that case the experimental methods should be checked. After further experiments, the hypothesis might be modified or scrapped.

Falsifying My Own Hypothesis: A Specific Situation

In a previous post I wrote about teaching one of my dogs two different cues for the same behavior. Both were cues to come in the house, but in one case I reinforced the behavior with high value treats, and my dog came running virtually all the time when I used that cue.  I reinforced the other cue with one piece of kibble. I set this latter cue up to offer my dog the option of coming in the house. I kept the reinforcer low value so if she was having a better time in the yard she might choose to stay instead. She responded to that cue intermittently.

My hypothesis is that the reason for the lower probability of response with the “kibble” recall is the lower value of the food. So let’s go through it with a view to falsifiability.

  1. My hypothesis: when using the same reinforcement schedule (in this case a continuous schedule, where the behavior is reinforced every time it is performed on cue) for the same behavior with two different cues, the response to the cue with the greater magnitude reinforcer will be more frequent.
  2. What research supports this? I’m not going to present a literature review, but here is the information one would need to investigate the issue. Schedules of reinforcement and reinforcement magnitude have been much studied. Some of the big names in the field are J. A. Nevin, H. L. Miller, B. J. Herrnstein, W. W. Fisher, J. E. Mazur, and P. De Villiers. Keywords for an article search are “reinforcement magnitude,” “contrast effect,” “concurrent schedule,” “reinforcement variety,” and “matching law.” You don’t even have to read scholarly articles if you have access to a learning theory textbook. These topics will be included.
  3. Has the hypothesis been formally tested? Yes, in a lab setting.
  4. What were the results? The hypothesis was confirmed and replicated.
  5. What would falsify the hypothesis? The hypothesis could be falsified if this body of research was overturned with the results of new, replicated studies that showed no correlation between the animal’s response and the quality of the reinforcer, or a negative correlation. I could possibly falsify the application of the hypothesis to my own situation by finding that there was an interfering detail in my setup.
  6. Could problems turn up if I attempted my own experiment? Sure. Since my hypothesis has to do with learning and behavior in the real world, my application of this hypothesis could have problems. If I got substantially different results when testing under the most controlled conditions I could create in the real world, that would not necessarily falsify the hypothesis. It might show that I was not applying the science successfully. (I will discuss homegrown research and experimental design in the next post.) But hopefully I would be able to analyze the problems and try again. If I exerted excellent controls on my experiment and still got results contrary to my hypothesis I might contact an expert in the field. If interested, this person could advise me on how to perform the experiment with more skill. Or if they were convinced that I had discovered something new in the world, they might choose to pursue the line of research. (In this case a new discovery would be extremely unlikely.)

Ask the Question

The point of all this is to give us a framework to determine if a trainer is knowingly basing their practices on science. So if a trainer is touting a new or branded method, consider the questions above, but especially Question #5 about falsifiability.  You don’t always need to ask the details about research and outcomes. It’s a good litmus test if you will just ask the following:

What would convince you this method is ineffective or doesn’t work in the way you claim it does? 

You might be very surprised at the responses when you ask that question.

What If Something Can’t Be Falsified?

The interesting thing about falsifiability is that the absence of it is not a good thing. We don’t say, “This hypothesis is so strong that it can’t be falsified!” On the contrary, a hypothesis or method that can’t be falsified, i.e., its practitioner can’t identify a test and outcome that would disprove it, is not scientifically based.

So there’s a paradox here.  The people who are the most attached to their methods, the most forceful in describing them, the most certain of themselves, are often the ones you should run away from. Instead, seek out the science-based trainer who can tell you what would falsify their methods, who is able to share with you the limitations of any procedure they use, and who is willing to admit when they are wrong.

I don’t recommend that pet owners ask this series of questions when interviewing dog trainers.  Jean Donaldson has a much more practical set of questions for dog owners to ask potential trainers.  And I understand that from the trainers’ point of view, most clients are more interested in the fact that something works than the details of how. But my falsifiability question is in the spirit of Ms. Donaldson’s push for generalized transparency in the training industry.

The graphic below shows some typical answers you might get when asking dog trainers, even within the positive reinforcement-based training community, how they would falsify their methods. Only one of the answers is acceptable.

Click on the graphic for a larger, readable version.

 

 

Eminent philosopher of science Karl Popper stated that the difference between pseudo-science and science is that pseudo-science seeks confirmations and science seeks falsifications. If this is true, then there is a lot of pseudo-science floating around in the dog training community. It is time to raise the bar??

What’s Next?

I have one or two more posts brewing on this subject. I want to discuss experimental design. Although our back yards are not the ideal places for controlled scientific experiments, we need to be able to assess whether the ways we train our dogs are working. We need to learn to be better observers. There are some criteria we can follow that will help make our assessments more accurate.

I’d also like to discuss the kinds of claims made by trainers that are unfalsifiable. Whoo boy!

Are you with me so far? Have I explained sufficiently the importance of being able to prove a theory, a hypothesis, or belief to be wrong?

Thank you to Sorrel Robinson, Skye Anderson, “V,” and Hayl Bergeland for advice on the scientific method and falsifiability. Readers will see more of their input in Part 2, but they helped get me off to a good start here as well. All mistakes are my own and they aren’t responsible for anything I wrote here!

Addendum 4/29/17

Thank you to commenter “A” who pointed out a flaw in the diagram. In the green box, I didn’t actually give specific examples that would falsify tenets of the science of learning theory. I didn’t answer the question in the blue box (which was the whole point, not only of the diagram, but the whole post!). I have replaced the text in the green box and hope I did a better job. (Just when I start to “get” this falsification stuff, it slips away!) In case you are curious, here is a link to the previous version of the graphic, where I just gave general instructions on how to find evidence, rather than suggesting specifics that would falsify aspects of the science of learning.

Copyright 2017 Eileen Anderson

Share Button

Easy tweets! (They include a link to this post)

  • Can your trainer explain what would disprove her methods? Is she willing to try?
This entry was posted in Critical Thinking, Research, Terminology and tagged , . Bookmark the permalink.

15 Responses to Falsifiability or Falsehood in Dog Training? (Part 1)

  1. Christy says:

    A comment not about the content (which I haven’t read yet) but about the email distribution of the articles. The reason I haven’t read it yet is because the formatting as displayed in Microsoft Outlook is so wide that I have to scroll back and forth to reach each line.

    In the web display above, the first sentence of the article wraps and takes up 2 lines. In the email version the first line contains the whole first paragraph in both preview mode and when opening the email to read it. Most of the paragraphs are on one line. A few of the longer ones do wrap, so there is some sort of line length defined. Changing the window sizing doesn’t adjust the wrapping. Zooming out so that the whole line is visible without scrolling makes the text too small to read.

    While I can always jump to the web posting, I would prefer to read the emails. Anything you can do to improve their readability would be appreciated.

    • Eileen Anderson says:

      Thanks SO MUCH for letting me know. That is automatic on WordPress’ part, but I might be able to fix it with a plugin. I want my stuff to be readable in every format! Thanks again.

  2. fearfuldogs says:

    Thanks Eileen! I can’t wait for the next installment. Dog training as an industry has been a petri dish for alternative facts for a long time. It’s great to have a system for inoculating ourselves against them.

  3. jay says:

    I’ve been reading your informative blog for a little while now. Thanks to a like from my breeder. For me though this post takes it to another level..i wanted to shout yes! And yes!
    I would like to suggest inserted between the existing points, 4.2 “Read the evidence and access it.” It’s a mistake to accept a piece of research as evidence if it’s poorly done. Even if you agree with the outcome.

    • Eileen Anderson says:

      Thanks so much, Jay. This was venturing into some new ground for me, too. Good idea on your part–I made an edit. –Eileen

  4. Giselle Scull-Monroe says:

    Great article! You make me think, even when I don’t want to. lol 😉

    Just a mention, I don’t see a button/link to your Facebook page here on your blog.

    • Eileen Anderson says:

      Ha! Sorry/not sorry! Seriously, thanks! I’ll consider the FB button. Mostly I just announce the posts on the FB page so once people are here I don’t worry about it. Bad marketing attitude, I know! Thanks for the comment.

  5. rheather says:

    This is making me think about something I’ve never even knew to think about! Thank you!

  6. A says:

    Great article! I have one objection, which is to the colourful illustration with speech bubble: the correct answer in the green speech bubble doesn’t seem to directly answer the question in the blue speech bubble, it sounds more like a general statement re. what to look for when considering the strength of a paradigm or scientific “fact”

    • Eileen Anderson says:

      Great comment! It took me a while to get what you were saying. It’s hard to falsify such a general statement (being a “science based trainer”), but I think if I add a part that the studies in question contradicted what we thought to be true before it would help? I’ll word it better. We both have to have the studies, and they would need to falsify some basic tenets of learning theory. I will work on it. Thanks! Just the kind of feedback I need.

  7. Pingback: Articles I’ve enjoyed recently | Little Brown Dog Blog

  8. Laika says:

    Hi Eileen,

    I’m a longtime admirer of your blog and the curiosity, rigour, and compassion displayed within it, though I’ve never commented. But, as someone who works in philosophy of science and who is a part-time professional dog trainer, you’ve hit on a subject that is very near and dear to my heart! I’m so happy to see someone discussing the concept of falsifiability in the +R community- we who often tout the “scientific basis” of our methods need a reminder as much as–if not more than–the rest of the dog training world. So, thank you!

    You’ve also hit on another very interesting subject, here and in your previous posts about choice, that I wanted to ask you about.
    Choice is indeed being bandied about in many contexts in the the humane hierarchy +R side of the dog training world (in most other parts of dog training, it never left, and was used as justification for all sorts of punishment and aversives as just consequences for poor choices and ways to teach “responsibility” for choices.) The more I hear it, the more I wonder what we could mean by the concept, especially as behaviourism has historically had no use for it. We could even call choice the concept that Skinner most sought to undermine (i.e. Skinner famously declared that that all behavior can and should be understood without any reference to choice, building his theory on the express rejection of choice as an essential or even useful part of the picture of behaviour).

    Skinner would not, of course, agree that you have given your dog any more of an option by using a lower-value reinforcer on a CR schedule, but I suspect you have good reasons to use and hold on to the term. I guess the succinct way of asking this is– Why do you think your dog is “choosing” to come in or stay out, rather than just exhibiting a weaker but no less determined behavior? What sort of evidence would you need to falsify this part of the hypothesis? And, if you do think she is choosing in the low value recall case, why would you say your dog is NOT choosing, or has less choice in, responding to her RRR cue, rather than just saying she chooses the high value reinforcer more often, but just as freely?

    Now that the +R community has rediscovered choice (not that it ever left, but you know what I mean). I find myself wondering how and whether and when we should be using the concept and how that should affect our understanding of the kind of training we are doing. I for one am VERY happy to see it back in the lingo, but think at least the science nerds among us should think carefully about the coherence of our theory, aims, and methods as we bring together these two rather different explanations of behaviours.

    Would love to hear your thoughts, thank you!

    • Eileen Anderson says:

      Wow, Laika, fantastic questions. By the way, my next blog is about the idea of determining dogs’ preference by giving them a “choice” and how that is fraught with problems. There is a whole sub-topic in ABA of preference assessment and it’s a lot more complicated to do well than it first appears.

      Very interesting idea to do away with the concept of choice altogether…that makes me want to go check the Catania studies because they were studying “choice” in an ABA context or at least a psych one. Herrnstein too–lots of language about choice when there are concurrent schedules. But the Matching Law shows how predictable we all can be about concurrent reinforcers. That does leave “choice” rather by the wayside.

      I am using the term choice because I do believe it can represent something beneficial for our dogs, and also the concept of deliberately training a cue with an intended low response rate is rarely talked about. But I’m thinking hard about the problem you mention.

      For me, thinking about it, choice is really not about the dog’s behavior, but about mine. I have loosened my control on the situation. I can’t predict what my dog is likely to do, and in one sense I don’t care (i.e., most possibilities are fine).

      So about falsifiability. Let’s say my hypothesis is that my behavior of teaching an informational cue of “you can come in now and get a low value reinforcer” gives my dog more choice than my giving the cue that has a very high rate of response. But if we look at the numbers, choice doesn’t have much meaning. The dog’s behavior will roughly follow the matching law. Perhaps the dog will respond 20% of the time to the low value cue and 95% to the high value cue. If she’s making a choice in the one case, she’s making a choice in the other. (Or in neither–however you want to look at it.) So in that way we could say my statement is false. There is no more choice intrinsic to one cue than to another. (As I have pointed out in another way in the post, but I didn’t put two and two together.) So thank you for helping me see that!

      Another way to go about falsification would be to delve into the scientific meaning of the term choice, perhaps including the neuroscience. (Did you know that the neuroscientists have made huge inroads in showing that we don’t have free will?) Anyway.

      I could also claim that my highly reinforced cue gives less choice because if the dog doesn’t come, I’m going to go get her, hopefully in an indirect and gentle enough way that I don’t poison the cue, i.e., show the hidden force behind my intentions. But that, also, is not intrinsic to the cue. It’s about what happens afterwards.

      So, I think you have helped me falsify my own statement. Ironic that my intent was to falsify other claims about choice! You’ve helped me take my thoughts to a new level. Thank you. IF you’ve got more thoughts about this please reply!

      Eileen

Comments are closed.