Module 7 Appraise evidence from the scientific literature

This episode accompanies Module 7 of the course, which is about judging the quality of the evidence from the scientific literature that we’ve identified during our earlier searching. This module aligns to chapter 7 of the Evidence-Based Management book.

Modules 5, 6 and 7 all focus on the scientific literature, so when you listen to the corresponding podcast episodes, the picture will hopefully be complete. 

In this episode we look at the process of learning how to appraise the studies, and what their findings mean in the context of the real world that we are faced with. If something is statistically significant and there is a large effect size measured by the study, does that mean that the finding is practically relevant for our organisation? 

Evidence from the scientific literature is undoubtedly important, but shouldn’t be emphasised more than other sources of evidence and it’s good to be reminded that it’s only when these come together than we can really make an evidence-based decision.

Host: Karen Plum

Guests:

• Eric Barends, Managing Director, Center for Evidence-Based Management 
• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University
• Rob Briner, Professor of Organizational Psychology, Queen Mary University of London
• Lisa J Griffiths, CEO, OzChild National Support Office

Find out more about the course here.

Transcript

00:00:00 Karen Plum

Hello and welcome to the evidence-based management podcast. This episode accompanies Module 7 of the course, which is about appraising the evidence that we've identified during our search for appropriate academic studies. There's more on the scientific evidence in modules 5 and 6 and their corresponding podcast episodes. 

Module 7 helps us practice critically appraising the scientific literature. The module is very practical with lots of studies that we comb through to try to establish whether the evidence is trustworthy and robust for our purposes. 

I'm Karen Plum, fellow student of evidence-based management and in this episode I'm joined by four experts that you’ll  have heard in earlier episodes - Eric Barends, Managing Director of the Center for Evidence-Based Management, Professor Denise Rousseau from Carnegie Mellon University, Rob Briner, Professor of Organizational Psychology at Queen Mary University in London and Lisa Griffiths, CEO of Australian childcare organization, OzChild. 

Let's get going. 

00:01:22 Karen Plum

In episode 5 we looked at the challenges of finding the right search terms and in understanding enough about each study to judge whether it contains trustworthy conclusions for our purposes. Perhaps like many people, I had some assumptions about the scientific literature and the world of academia, that weren't accurate. 

Rob explains that it isn't as orderly and accessible as people think. 

00:01:47 Rob Briner

I think before you go to it, that body of scientific evidence, I think there's an assumption - certainly I made that many years ago - that it's quite orderly, it's quite tidy, it's well sorted out, everything’s in its place, things refer to other things correctly, it's all ordered. 

But actually it's a mess. And I think it's important to note that partly to explain to students that when they're searching, when they're using words, the reason they're doing it, is because this body of work is not - you can't just go in and put your hand in and get what you want. It won't happen. 

It's not well ordered and I think the search terms issue comes from lots of reasons. Some researchers use different words for the same thing, they use the same word for different things. And in research like other areas of practice, people are often rediscovering the wheel so they say it's a new thing and then you look at that new thing and it turns out it's not really such a new thing, it's the same as the old thing. And it's quite confusing. 

00:02:47 Karen Plum

So how might we start on our journey to identify the right search terms? Although perhaps counter intuitive, Rob suggests approaching the literature first with more of an open mind and just to learn about what's being researched. 

00:03:02 Rob Briner

Although it sounds a little bit counter intuitive to some people, you actually need in my view to get quite a good feel for the literature first. 

So what I typically say to people is if you're trying to look for search terms, might be a term like, you know, flexible working. There's lots and lots of different terms for flexible working and different researchers will use completely different terms. even when they're talking about exactly the same thing. So if you read a review of the field, say around flexible working, you may discover in that review that it uses a number of terms itself. You may also want to look through the references, look at how other authors have used that term, look at a few abstracts and basically make sure that you are capturing all the words, the terms that different researchers used for the same thing. 

So I think getting a feel for the literature first is very, very important. What sometimes people do is when we talk about finding synonyms for words they go look in a dictionary. And again, what I always say to people is, well you're not after kind of linguistic synonyms, you're after scientific synonyms. So, that's the key thing to me, is understanding a bit about the literature first before you search. 

00:04:11 Karen Plum

I guess it's about casting a wide net to start with. Of course it depends on the topic, but I like the idea of educating myself a bit more before I dive into the searching process and appreciating that it isn't all orderly and aligned, helps me understand that this isn't going to be a straightforward process. 

Rob's reference to synonyms is echoed by Denise, who explains why it's important to recognize that academics use language very precisely, in a professional context, not in the loose approximate way that we use words in everyday life. 

00:04:46 Denise Rousseau

One of the things that's very important that practitioners often don't realize, is how precisely scholars use terminology and how flexibly and inconsistently normal people use terminology. A scholar is like a surgeon who's asking you for a particular scalpel and you really don't want another kind of scalpel or another kind of tool for doing surgery. You want precisely that particular instrument. 

And that level of precision exists in scientific research so that the scientists can communicate with each other clearly and not get the wrong scalpel, mixing my metaphors - that is not the way we talk in normal life, we use all sorts of synonyms and approximations. So understanding the reason why vocabulary, terminology matters in a professional context, which is what a peer reviewed article is - it's a professional context - might help us appreciate that this is precision, not a political process or not an artificial construct for nefarious means. It's actually for efficiency that the terminology exists. 

00:06:02 Karen Plum

I think part of the difficulty of my journey through the course is trying to navigate this new world of science and academia. Appreciating the different ways that language is used is just one aspect. I was reminded of something Denise told me about the cognitive load associated with the learning process. 

Not only am I learning new stuff, I'm frequently needing to review and question how I think and react to new knowledge. 

00:06:30 Denise Rousseau

We know from years of research on feedback and in other domains in learning. that when people’s sense of self surfaces in the process, that that's kind of a cognitive load, that's a cognitive burden. Because I'm I got a lot to protect, I got a lot of interest in myself. And that I'm learning that in fact, geez, I've been making a lot of assumptions about which authority is trustworthy. Or heck, if this stuff is true, that means that next time my senior leader claims such as the case. I should be a little more skeptical? Maybe I should even ask a question? That's the self-implicating part. 

What should I do in future? That is, that's a little bit emotionally churning. It's like going into a new territory, a new country whose culture and ways of being are so different from what you know from your home country and even though travel is broadening, it's also stressful. So preparing people for that is a part of this process of saying, the focus is on learning and reflection, and you will learn things about yourself as well as things about the process. 

00:07:44 Karen Plum

And perhaps that explains why I found this module quite draining at times. I also saw my desire to be right surfacing again and it was getting in the way of seeing the bigger picture. More about the bigger picture later. 

For now, I'm sure I'm not the only student that was surprised by how many exercises there are in module 7, and how many studies we had to review. I found some quite challenging and frankly rather tiring. I also thought that I was getting the hang of things only to then get tripped up on some detail that I'd missed. I put these points to Eric - firstly, why so many exercises? 

00:08:22 Eric Barends

So the reason why there are so many learn by doing exercises or did I get these exercises, or you're presented with so many studies, is that it takes time before you get it and you build up a kind of routine. And the first time is like oh my God this is a paper with 32 pages and oh my God, it's boring and it's hard to read and I don't even understand the abstract to begin with. 

And that's why we take you by the hand and ask specific questions and say go here, go there so we take it apart. And actually, the way you read a study is by taking it apart by first of all, what type of study is it, is it a cross sectional study, was there a control group or whatsoever? It's literally learn by doing in this case, that's absolutely true. 

You need to go through two, three, four, six, eight, twelve papers before you pick up a kind of routine and you'll learn to quickly scan a paper and to figure out where the relevant information is. It's hard and you will become better at it. It's like riding a bike the first time you want to make sure that the bike is stable and that the surface is stable and then there's no people around you, and then you get on a bike. That's how you learn. But when you have mastered and you know how to ride a bike, you just grab your bike, jump on it and start pedaling and you're off. 

It's the same here. When you start with the module, it's hard you have to - where is it? And you make mistakes and you get better and better and better at it. 

00:10:01 Karen Plum

I do think it's a good example of how you think you get something and then you realize that you don't. The critical thing is whether the design was an appropriate way to address the research question. 

To illustrate the point, Denise explains that students tend to be good at figuring out whether the research methodology is sound, because the module is so good at teaching this to us. 

00:10:24 Denise Rousseau

My students, I think, are relatively comfortable critiquing the research literature and the design because we train them in that. That's one of the things the modules are very, very good for, is teaching you what kind of research design for the question I have. 

Many of the questions that students and practitioners are interested in are causal questions - does X cause Y? Or would X be a success factor in dealing with the problem of Y? Where you're really trying to see does the needle change, does the needle move when we look at this phenomena in this way? And in those cases, one of the critical issues we find is that students search on, for their question, on whether or not there are field experiments or whether or not there are meta-analyses (summaries of studies). Because for causal questions, that's an extraordinary claim - that X causes Y and you need stronger evidence for extraordinary claims than for claims that say X is related to Y, you know where correlation or survey is enough. 

So screening on methodology is important, and so like using the term field experiment or meta-analysis are ways of filtering through large bodies of research. Another way to do this in terms of trying to strike gold in doing a literature search, is to look and see if there have been practice oriented studies on your question. And there are several journals and outlets that focus on practice oriented research. 

There are journals of implementation science and there are journals of applied behavioral science and JABS, the Journal of Applied Behavioral Science, does look at practical interventions and their effects. Knowing that such a journal might be available to you, could be important, and that way you don't have to search through abstracts, you can search through the recent publications in a journal that's interested in application. 

00:12:33 Karen Plum

So another pointer there on how to find more sources of evidence on how things have been applied. 

I'm sure you'll have noticed that the examples and exercises cover a wide variety of topics and situations, so each time it's important to consider whether what the researchers did was appropriate to the question they were addressing. We know the peer review process looks at this, and if we find meta-analysis as we discussed in episode 5, we also get more help in figuring out the appropriateness of studies. 

The exercises push us to look for any weaknesses in the studies because we may be going to rely on this information in the decision we have ahead of us, and it may be an important decision with a critical outcome. 

One of the things that interested me was that in one study we examined, the authors hadn't reported an effect size, but that wasn't called out as a serious weakness. I wondered why this was and it turns out that although not reporting effect sizes is, in Eric's view, a cardinal sin, it doesn't necessarily reflect the trustworthiness of the findings. 

00:13:40 Eric Barends

In our field, social studies, management studies, organizational behavior studies, IO psychology, more and more authors and researchers report effect sizes. It is a requirement, as I recall by the APA the (American) Psychological Association. But still a lot of researchers fail to do so. Which is a bad thing, but that doesn't say anything about the trustworthiness of the findings. It says something about the precision or the impact of the findings. 

00:14:16 Karen Plum

And Rob points out that even if you measure something statistically significant and establish an effect size which might look meaningful, it still doesn't speak to its practical relevance in the real world. 

00:14:29 Rob Briner

So for example, you might have a five-point scale measuring, say wellbeing, and the average on that five-point scale going from one to five might be 4.5, and then you do piece of research and you look at kind of the effects of something, on wellbeing. And yes, it's true, this thing has an effect on wellbeing, but actually what you're talking about is maybe something like moving from 4.3 to 4.4. 

Now it might be very statistically significant, it might be quite a big effect size and other sorts of ways, but then the question is you might be able to consider, does that really – I wonder if that really matters, in the real world. 

So again, it's that statistically significant to effect size to does it matter? And to whom and in what way? And that bit is very hard to get from a paper. 

00:15:15 Karen Plum

So you really have to take all these things together, in context. During my journey through the academic literature modules, I started to feel a little despondent that so many were flawed in a variety of different ways. Discussing this with Eric, he encouraged me not to become an evidence-based fundamentalist. 

00:15:35 Eric Barends

Yes, so our modules are very active. That means that students come in and they have a high respect for research and they say oh, wow, it's scientific research, this is very important stuff and must be true. And then they do our modules and they come to the conclusion that most of the research is absolute crap and is flawed and doesn't mean anything. 

And that's why we always warn - don't become an evidence based fundamentalist. They need to go through the journey and learn that indeed a lot of research is flawed and weak, etc. But they should move up a little bit and understand that still there's sufficient good enough research out there to make an evidence-based decision or more reliable decision. 

And there are students that are very good learners, but go a little bit too far and dismiss all studies that are not controlled and say, yeah, this is actually worthless. No, that's not what you should do. You should understand that there are serious limitations, but you know there are limitations in life everywhere and we still know how to deal with that. Don't search for the perfect evidence, it's about and this is, I think, crucial to understand - the best available evidence.

And sometimes the best evidence is not available. We need to deal with the best available evidence and that may be two or three cross-sectional studies, a qualitative paper and a whole bunch of expert opinions, and that's all we have, but that's OK. That's an evidence-based approach too, it's better than just your gut feeling or a crystal ball. 

00:17:26 Karen Plum

And that resonated because one of the things that struck me once I'd completed modules 5, 6 and 7, is that delving into all the details of the academic studies, I'd lost perspective? I got lost in the weeds if you like - losing sight of why I wanted to know more about what was in the studies. I think there's a certain inevitability about this. You have to dive into the detail so you can pull yourself back out and climb into the helicopter to gain that bigger perspective. 

00:17:55 Rob Briner

I think the thing that’s important when looking at evidence from multiple sources to try and make better informed decisions, both about problems or opportunities in the organization and about potential solutions or interventions, that's what we're really trying to do. And for me, yes, we must understand the techniques and the technologies that help us collect that evidence, to collate it, to critically appraise it, to aggregate it, that's really, really important, but it's also important we don't lose sight of what we're trying to do.

We're trying to make better informed decisions. And I think that's sometimes lost when we delve into some of the techniques that we need to do those other things. So I think they're really important. We should never lose sight of what this is about. We're trying to make better informed decisions. 

00:18:45 Karen Plum

As we discussed in Episode 5, just because you know how to do a deep dive on the academic literature doesn't mean that you always do. Taking a pragmatic approach might be fine for many occasions, but if you need to explore something in more depth, you need to know how to do that as well. 

00:19:02 Eric Barends

As managers, being under time pressure and we need to deliver results and that means that we often take a more pragmatic approach to the evidence at hand and also when it comes to critical appraisal. Sometimes, or often, you don't need to go in such detail to get your question answered. 

You also need to take into account how important is this decision, so the level of detail is also determined by the seriousness or the impact of the decision you're going to make. 

The bigger picture is not only in terms of how important is the decision and how detailed should we go and look into the quality of the research and how thoroughly should we search and rigorously should we search, in all these databases? 

It's also on the wider scale that the evidence based approach, even when you do just a teeny weeny little bit, will have a positive impact when you apply this again and again and again to multiple decisions and your colleagues also take an evidence based approach. Then the effect will be way way larger in magnitude. 

00:20:19 Karen Plum

And I think therein lies the challenge for managers in organisations if they are but one voice promoting an evidence-based approach, it will likely take much, much longer to reach that cumulative impact. Doesn't mean it's not worth doing, far from it. If the decision you're faced with has far reaching implications then even an isolated episode of rigorous examination of multiple sources of evidence is worthwhile. 

And let's not lose sight of the wealth of other skills that are part of evidence-based practice, such as asking good questions and not leaping straight to solutions. These also bring a lot of value to our day-to-day management practices. When I talked to Lisa Griffiths from Australian childcare organization OzChild, she reinforced the value of having colleagues that were following the same broad evidence-based approach. 

00:21:11 Lisa Griffiths

By doing the evidence-based management it gives managers then the skills to be able to go, ah I get now what this means, instead of just hearing the term evidence-based practice or evidence-based program or an evidence-based model which are all terms attached to interventions. They can actually say well if I'm a leader, what are the things I need to be sure I'm covering off so that if I'm going to make a decision about one of these things, this evidence-based intervention or this evidence-based practice, or this evidence-based model - that I've paid attention to, you know the four areas that are really meaningful in terms of the scientific data, the organizational data, or the sector data at large, my own practice wisdom, my tacit knowledge as well of course, what our key stakeholder concerns are, and they all packaged together. Really makes sense to managers. 

00:22:10 Karen Plum

I think it's really important to remember the different sources of evidence as we make our way through the course. Even though we're immersed in each one, they will eventually come together. Rob feels there's often a tendency to overemphasize the scientific literature. Maybe that's to do with the reverence many people feel about science. 

00:22:30 Rob Briner

There is a danger, I think, in evidence-based practice that we tend to overemphasize both the scientific literature, scientific evidence above other forms of evidence which to me isn't correct. And also, we tend to imply even within the scientific evidence, that it's necessarily we can identify it as being very trustworthy and valid and reliable. Which we can to some extent of course, and that actually if it's published in a peer reviewed journal it’s probably fairly decent. 

And as we know from looking at the stuff in detail as I know you have as well, it doesn't turn out to be the case. Some of it is, of course, but some is not. Yeah, I think there’s a danger that we over inflate sometimes the quality of science, so we don't see it as human, we see it as something else. 

00:23:15 Karen Plum

Perhaps there's a very human tendency to look for silver bullets. The answers to life, the universe, and everything. If we do find a meaningful relationship through research, there can be a tendency to want to extrapolate it, to fix everything. 

00:23:29 Rob Briner

It's almost like a bias I think, that we want this to be a thing that is more important than anything in predicting everything, and if only we could understand that thing then, you know, we’ve kind of solved all the problems in the world. 

So I think it is a kind of bias or a temptation. The other thing is it's very contextual, so I think it may be the case that yes, this particular behavior is predicted quite a lot by this particular thing in this circumstance, in this setting, in this particular time period, there is a kind of sense in which that could happen - but then what we tend to want do then is generalize to other settings, other time periods, other contexts which maybe doesn't make so much sense. 

00:24:09 Karen Plum

So keep grounded when you're looking at the evidence. As we've discussed in other episodes - question everything. Don't assume that because academics are clever people, they must have all the answers to our problems. As with so many things in life, you have to do the work - to make your own informed decisions and judgments. 

One of the guiding questions that Eric tells me I should always ask is “how did they measure that?”. 

00:24:36 Eric Barends

How did they measure that is, in daily life actually always the first question you should ask. Regardless of research methodology and whatsoever, the way they measure outcomes or variables is actually so impactful and determines the quality of the study, maybe even way more than the research design or possible weaknesses. 

So we train and teach critical appraisal by what's the resource design? Was there pre measure, post measure was it cross sectional etc., and then we go into weaknesses. But to be honest, the first question I always ask and you should ask is, how did they measure that? And just think about it. Think about it. 

If they say well, organizations in which the Board of Directors are more proactive will have more successful long-term strategies than companies with Boards that are more ad hoc or whatsoever. Yeah, probably makes sense. Sounds plausible. How on earth would you measure that? How on earth, really? 

I mean, long-term strategy. That means that at least I need to have a longitudinal study or measure over a long period of time and what is actually successful - when we have a cookie factory compared to a, I don't know automotive industry or? How would you measure that? And that question is just common sense. 

00:26:13 Karen Plum

I've started to ask that question more and more. It's a great one for so many situations. Whenever I hear a headline on the news or someone tells me about a new research finding, a little alarm bell rings in my head. OK, sounds good, but how on Earth did they measure that? 

So many of us are getting tuned to letting these things wash over us with our short attention spans and attraction to fads and fashions. But really, does it sound plausible? Sometimes only a small amount of digging into the source is enough to put doubt into your mind. 

00:26:47 Eric Barends

I think we use the example of research has shown that the average American flosses his teeth three times a day. Really, how did they measure that? 

Did they put a camera at every American household at the bathroom? And no! So they probably just asked a sample of people in United States and say how often do you floss? Well, three times a day, really? I mean, is that a reliable way to measure stuff?

So how did they measure this goes beyond questionnaire that they used, but it actually indirectly already what kind of research design did they use - did they ask two people? 600 people? Did they measure before and after? So it's actually the question where everything comes together. 

00:27:39 Karen Plum

But now to wrap up our discussion on the scientific literature, Rob reinforces the case for multiple sources of evidence, and we'll move on to evidence from the organization in episode 8. 

00:27:52 Rob Briner

It's very hard to apply scientific findings without a lot of knowledge of the organization, of the stakeholder perspectives and the professional expertise. So even if you have a big body of research that looks quite rigorous scientifically, that seems very clear in terms of the findings, in and of itself, you cannot apply that. You just can't. Unless you have all those other things. 

And that to me, is one of the most important things so yes, going to the scientific literature is important. Understanding what it might be telling you is really important, of course, but you need those other sources of data and evidence to make sense of it and to see if it's usable and relevant, and whether incorporating into your decision making is going to get the outcomes you want or not.