What if we had to know our animal training theory and practice so well that we could easily tell someone what would disprove the hypotheses that inform our methods? That’s what scientists do. If we are going to claim to base our training methods on science, I think we should get with the program.
There’s a concept in science that is not much discussed in the world of dog training. The concept is falsifiability. Learning about it can save us a world of hurt in assessing statements about training methods. Focusing on how we would disprove our own methods may seem counterintuitive at first, but bear with me.
All dog training methods are based on science because the processes involved can be explained by science. But only some trainers actually study the science and base their decisions on what they learn from it. For the purposes of this article, I’ll call them science-based trainers.
Whatever they call themselves, in my opinion, anyone who says they base their training on science should be able to explain how they do it. They should also be able to tell you what would falsify, i.e., disprove, the hypotheses they use.
Falsifiability and The Scientific Method
Falsifiability, also known as testability, is the ability of a hypothesis or statement to be scientifically disproven, and is intrinsic to the scientific method. Any researcher who puts forth a hypothesis should be able to state ways it could be falsified, and should expect those methods to be attempted in the future. Many researchers perform repeated experiments to attempt to falsify their own findings. I admire these people who are willing to put their ego investment on a back burner to seek ever more accurate information.
Jerry Coyne, author of Why Evolution Is True, has a post on his website that lists seven items that, if observed and confirmed, would shoot holes in parts of the theory of evolution. Taking a look at his article will give you a good grasp of what falsifiability is. It will also demonstrate the high level of expertise in a subject needed to understand how to falsify it.
(Falsifiability is not used as a metric in some sciences, such as physics and astronomy, because some hypotheses can’t be tested. For instance, much of string theory is famously unfalsifiable. That doesn’t mean that the hypotheses involved are accepted without rigorous examination though. In the main we don’t have those kind of problems in behavior science)
The scientific method gives us a good guide for questioning dog trainers, their methods, and statements about them. Every training method involves one or more hypotheses about cognition or ethology. You can use this concept of falsifiability to ask someone about their overall philosophy or just a particular method. Here’s how you can go about that.
- Identify the training challenge or part of their philosophy you want to discuss.
- Ask the trainer what the general research is behind their approach. They don’t need to name specific studies or textbooks. But they should know the concepts and be able to explain their relevance.
- If discussing a method within a training system, ask the trainer if that specific method has any specific research behind it, i.e., has it been scientifically tested?
- If the method has been tested, ask the results and assess the evidence.
- If the method has been tested and found to be successful, ask whether the results have been replicated by other research. Replication is essential. Hanging on the results of one study is not good practice when other studies are possible and ethical to perform.
- If the method has not been tested, ask how they would design an experiment and what would falsify the method. Again, there doesn’t have to be a lot of detail. But this will show whether they understand the claim they are making.
- Whatever the answers to the above, ask them what it would take to disprove their method or philosophy.
The last thing is perhaps the most important. You are not only putting the trainer’s knowledge to the test, but also potentially running up against their ego. We all get attached to our methods and an ego response is natural. But pursuing falsifiability forces one to override the ego.
Remember, we can’t “prove” a hypothesis. What we hope to do when we research it is to amass evidence for it. But we can disprove it by finding examples within the scope of the hypothesis in which the outcome is not as predicted. In that case the experimental methods should be checked. After further experiments, the hypothesis might be modified or scrapped.
Falsifying My Own Hypothesis: A Specific Situation
In a previous post I wrote about teaching one of my dogs two different cues for the same behavior. Both were cues to come in the house, but in one case I reinforced the behavior with high value treats, and my dog came running virtually all the time when I used that cue. I reinforced the other cue with one piece of kibble. I set this latter cue up to offer my dog the option of coming in the house. I kept the reinforcer low value so if she was having a better time in the yard she might choose to stay instead. She responded to that cue intermittently.
My hypothesis is that the reason for the lower probability of response with the “kibble” recall is the lower value of the food. So let’s go through it with a view to falsifiability.
- My hypothesis: when using the same reinforcement schedule (in this case a continuous schedule, where the behavior is reinforced every time it is performed on cue) for the same behavior with two different cues, the response to the cue with the greater magnitude reinforcer will be more frequent.
- What research supports this? I’m not going to present a literature review, but here is the information one would need to investigate the issue. Schedules of reinforcement and reinforcement magnitude have been much studied. Some of the big names in the field are J. A. Nevin, H. L. Miller, B. J. Herrnstein, W. W. Fisher, J. E. Mazur, and P. De Villiers. Keywords for an article search are “reinforcement magnitude,” “contrast effect,” “concurrent schedule,” “reinforcement variety,” and “matching law.” You don’t even have to read scholarly articles if you have access to a learning theory textbook. These topics will be included.
- Has the hypothesis been formally tested? Yes, in a lab setting.
- What were the results? The hypothesis was confirmed and replicated.
- What would falsify the hypothesis? The hypothesis could be falsified if this body of research was overturned with the results of new, replicated studies that showed no correlation between the animal’s response and the quality of the reinforcer, or a negative correlation. I could possibly falsify the application of the hypothesis to my own situation by finding that there was an interfering detail in my setup.
- Could problems turn up if I attempted my own experiment? Sure. Since my hypothesis has to do with learning and behavior in the real world, my application of this hypothesis could have problems. If I got substantially different results when testing under the most controlled conditions I could create in the real world, that would not necessarily falsify the hypothesis. It might show that I was not applying the science successfully. (I will discuss homegrown research and experimental design in the next post.) But hopefully I would be able to analyze the problems and try again. If I exerted excellent controls on my experiment and still got results contrary to my hypothesis I might contact an expert in the field. If interested, this person could advise me on how to perform the experiment with more skill. Or if they were convinced that I had discovered something new in the world, they might choose to pursue the line of research. (In this case a new discovery would be extremely unlikely.)
Ask the Question
The point of all this is to give us a framework to determine if a trainer is knowingly basing their practices on science. So if a trainer is touting a new or branded method, consider the questions above, but especially Question #5 about falsifiability. You don’t always need to ask the details about research and outcomes. It’s a good litmus test if you will just ask the following:
What would convince you this method is ineffective or doesn’t work in the way you claim it does?
You might be very surprised at the responses when you ask that question.
What If Something Can’t Be Falsified?
The interesting thing about falsifiability is that the absence of it is not a good thing. We don’t say, “This hypothesis is so strong that it can’t be falsified!” On the contrary, a hypothesis or method that can’t be falsified, i.e., its practitioner can’t identify a test and outcome that would disprove it, is not scientifically based.
So there’s a paradox here. The people who are the most attached to their methods, the most forceful in describing them, the most certain of themselves, are often the ones you should run away from. Instead, seek out the science-based trainer who can tell you what would falsify their methods, who is able to share with you the limitations of any procedure they use, and who is willing to admit when they are wrong.
I don’t recommend that pet owners ask this series of questions when interviewing dog trainers. Jean Donaldson has a much more practical set of questions for dog owners to ask potential trainers. And I understand that from the trainers’ point of view, most clients are more interested in the fact that something works than the details of how. But my falsifiability question is in the spirit of Ms. Donaldson’s push for generalized transparency in the training industry.
The graphic below shows some typical answers you might get when asking dog trainers, even within the positive reinforcement-based training community, how they would falsify their methods. Only one of the answers is acceptable.
Click on the graphic for a larger, readable version.
Eminent philosopher of science Karl Popper stated that the difference between pseudo-science and science is that pseudo-science seeks confirmations and science seeks falsifications. If this is true, then there is a lot of pseudo-science floating around in the dog training community. It is time to raise the bar??
I have one or two more posts brewing on this subject. I want to discuss experimental design. Although our back yards are not the ideal places for controlled scientific experiments, we need to be able to assess whether the ways we train our dogs are working. We need to learn to be better observers. There are some criteria we can follow that will help make our assessments more accurate.
I’d also like to discuss the kinds of claims made by trainers that are unfalsifiable. Whoo boy!
Are you with me so far? Have I explained sufficiently the importance of being able to prove a theory, a hypothesis, or belief to be wrong?
Thank you to Sorrel Robinson, Skye Anderson, “V,” and Hayl Bergeland for advice on the scientific method and falsifiability. Readers will see more of their input in Part 2, but they helped get me off to a good start here as well. All mistakes are my own and they aren’t responsible for anything I wrote here!
Thank you to commenter “A” who pointed out a flaw in the diagram. In the green box, I didn’t actually give specific examples that would falsify tenets of the science of learning theory. I didn’t answer the question in the blue box (which was the whole point, not only of the diagram, but the whole post!). I have replaced the text in the green box and hope I did a better job. (Just when I start to “get” this falsification stuff, it slips away!) In case you are curious, here is a link to the previous version of the graphic, where I just gave general instructions on how to find evidence, rather than suggesting specifics that would falsify aspects of the science of learning.
Copyright 2017 Eileen Anderson
Easy tweets! (They include a link to this post)
- Can your trainer explain what would disprove her methods? Is she willing to try? Tweet