Module 5 Acquire evidence from the scientific literature

This episode accompanies Module 5 of the course, which focuses on gathering evidence from the scientific literature (academic studies) that will help us address the question we are trying to answer. This module aligns to chapter 6 of the Evidence-Based Management book.

Modules 5, 6 and 7 all focus on the scientific literature, so when you listen to their corresponding podcast episodes, the picture will hopefully become more complete. 

In this episode we discuss the process of searching for academic studies; the difference between everyday / managerial terms and academic constructs; how to find the right terms to search for; the importance of obtaining studies from the right sources, most especially peer reviewed academic journals; the difference between journals and “magazines”; and the importance of documenting the process you followed to identify your evidence base.

We also explore how the peer review process and the meta-analysis help us get to the best available evidence.

 Host: Karen Plum

Guests:

• Eric Barends, Managing Director, Center for Evidence-Based Management 
• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University
• Barbara Janssen, Board Member and Fellow of the Center for Evidence-Based Management

Find out more about the course here:   https://cebma.org/resources-and-tools/course-modules/ 

Transcript

00:00:00 Karen Plum

Hello! Welcome to the Evidence-Based Management podcast. This episode accompanies Module 5 of the course, which is the start of our exploration of the evidence from scientific literature, the studies carried out by academics in our field of interest. 

The remainder of the journey is covered in modules 6 and 7 and their corresponding podcast episodes. 

Module 5 looks at how we acquire the evidence we're going to use to explore and answer the question that we've identified earlier in the process. The course teaches a thorough method of searching for scientific studies and documenting the process used to find them. This so called ‘nuclear option’ isn't always necessary. Sometimes a quick search in Google Scholar will be sufficient, but the course equips students to use either approach. 

I'm Karen Plum, a fellow student of evidence-based management, and in this episode, I'm joined by two podcast regulars - Eric Barends, Managing Director of the Center for Evidence-Based Management and Professor Denise Rousseau from Carnegie Mellon University. My 3rd guest is Barbara Janssen, a Board Member and Fellow of the Center for Evidence-Based Management. Let's get started. 

00:01:27 Karen Plum

Having had some exposure to the search for academic studies, I thought this module might be fairly straightforward. In my day job, I've been responsible for commissioning searches of the academic literature on several subjects of interest to my organization. 

Eric, Barbara, and their colleagues conducted the searches and critically appraised the studies they found - presenting us with the best available evidence. 

As with anything you don't fully understand, this seemed a straightforward process which delivered me interesting, relevant studies upon which I could take action and help others understand what we'd found. I now fully admit that I didn't pay huge attention to the process that lay behind the results, and having now done this module, I realize there's a lot more to it. 

So let's go back to basics and ask why we're searching for academic literature in the first place. Essentially, we have a problem that we're exploring - looking for the best sources of evidence to guide our decision-making. We could just go to Google and search for the subject and see what comes up. Then we could look at what's relevant or interesting, or what aligns with our assumptions about what's going on in our organization. 

At this point, alarm bells should be ringing in your head. What's going to happen if we do this? When we search for evidence, wherever we search, we don't know whether it's good, reliable or trustworthy. But if we search first and then judge the goodness of the findings, then that may be a good start. But it's not enough, as Eric explains. 

00:03:05 Eric Barends

We want to have a number of studies that are on the topic that we're interested in or about the issue we're trying to solve. We want to do that in a way that we don't miss important, relevant studies. That is actually what we’re doing 

So that’s why it's a bit tedious. You need to take steps in a specific order, and that is because it needs to be reproducible and transparent. So when someone would ask me - where did you find these studies, where do they come from? You can say well they are published in the past ten years or twenty years and you can find them in these databases, if you search in this specific way. And you also want to make sure that the way you search - the search terms you use - are indeed appropriate terms, relevant terms, and it's important that you don't miss important studies because you used the incorrect or inappropriate terms. 

00:04:08 Karen Plum

So if we just go to Google and put in some terms that we think are right, we'll get tens, hundreds or thousands of results. It would be difficult to reproduce the number of studies, or even the studies themselves, in amongst the noise that is Google. 

Being systematic and methodical and documenting how you identified your sources is a critical part of being evidence-based, transparent and reliable. It doesn't avoid all errors, but at least it makes them more obvious, so that someone else might spot a flaw in your process, helping you to amend it and refine the evidence base. 

The purpose here isn't to be secretive about our methods, but to show that the best available evidence was used, rather than relying on opinions, personal experiences, or gut feelings. Module 5 explains in detail how to step through the process Eric talked about. 

One of the things we spend some time on is identifying the right search terms to find the appropriate studies. As I make my way through the course, the importance of language keeps being reinforced. One of the dangers is that we all think we know what a particular word or term means. Chances are that the term might be one used commonly in management circles, but Eric explains that academics will use different terms or ‘constructs’. 

00:05:30 Eric Barends

The biggest challenge is finding the right search terms. The problem here is that academics and practitioners are living in separate worlds - or they live in different countries. And therefore they speak a different language. That means that we managers, we practitioners, use terms or as we refer to it, constructs (construct is the more scientific term for an idea or a variable or an insight or something you do). 

For instance, employee engagement. That's an idea, that's a word we use, a term we use, but we're not completely sure what it means. Same counts for instance, organizational culture. We managers use the term all the time. But from an academic point, you will notice that it's not really clear what it means. 

00:06:38 Karen Plum

So we have to find the right academic search terms, not managerial search terms, so that we can search for the things that academics have researched and written about. We need to know what the construct is. So firstly, Denise explains what a construct is and how it differs from a simple term or phrase. 

00:06:59 Denise Rousseau

Construct in psychology is used to refer to an abstract idea or entity that can be observed. What the construct does is to specify the features that define the construct, so that no one we're observing phenomena, whether what we're looking at conforms to the definition of the construct. 

So a construct - in my own research is the construct of a psychological contract. OK, so it's defined as a system of beliefs regarding an exchange relationship a person has with another - could be an employer, it could be a team, etc. 

00:07:42 Karen Plum

Clearly there's more to this than a simple definition, and maybe explains why some of the academic terms really aren't that guessable by those outside the field. So if we don't know what the academic term is that corresponds to the problem or topic that we're researching. What do we do? 

Here's Barbara Janssen, who's done a lot of this type of work. 

00:08:03 Barbara Janssen

First of all, Wikipedia is your best friend to look for search terms. It's not a reliable source content wise, but it really helps with coming up with some terms that are widely used. Then I always go to Google or Google Scholar, especially Google Scholar is very helpful because you will find some relatable terms to the ones that you have entered, this always leads to some different search terms. 

You can check databases, research databases, just enter those certain search terms and you will find things - and in these research databases there's always a thesaurus option. 

There are also thesaurus websites where you can check terms, so synonyms or related terms might come in handy. And then if you check back again in the databases, you will find terms that will generate the most hits, or when you quickly look through some of the titles and abstracts of those hits you will see which are the most relevant to your questions. So I think that will be my main steps if I'm not sure. 

00:09:07 Karen Plum

I asked Barbara for an example of a topic that she was working on, where the academic term wasn't obvious. She told me about a company that was interested in the impact on workplace performance, when people have financial worries. 

00:09:21 Barbara Janssen

We looked into Google Scholar and then ‘financial distress’ popped up, but equally other terms like financial wellbeing, financial satisfaction, financial wellness. And the difficulty sometimes also is that they aren't always defined very clearly. Terms are yours used interchangeably. 

This makes it kind of difficult. So in the end we noticed that there was a way to measure financial distress and mostly used measurement was called financial distress / financial wellbeing scale. So that led to some important terms and then we checked them in the databases to see the number of hits that would come up. 

And in the end, we were including more terms, so we included financial distress, financial wellbeing just because it was a difficult topic. There was a lot of research, but the search terms weren’t always very distinctive. 

00:10:19 Karen Plum

Certainly I would have struggled to come up with financial distress, and clearly it wasn't the only term used, but I think it serves to illustrate the point. 

All the guests on this episode also suggested just asking an academic - an expert in the field - when tracking down the right search terms. According to Denise, even Nobel laureates like Herbert Simon when asked how he would find out about a new topic, says “I always ask someone, it's easier”. 

There's no dishonor in asking for this information, it's just efficient, and you may be able to find a particular academic’s email address on their university website, if you have a question for them.

Now we've talked about the terms, let's turn to how we use those terms to identify studies in our area of interest. During the module, we used Business Source Elite from EBSCO, which let me say I found quite intimidating to start with. So many options. I appreciate that there are probably just a few that you use regularly and after a bit of searching I did start to get more comfortable with it.

I expect like everyone I made the usual mistakes of mixing up AND and OR. Both Eric and Barbara admitted this still regularly trips them up too, so I found that reassuring. 

00:11:37 Eric Barends

You will make that mistake again and again and again and you will all need to be alert that you make that mistake. You need to check and when something weird happens like you have suddenly zillion number of papers or the other way around, you end up with only three papers, but you know, the first search you had 10,000. What happened? Well, you probably used AND where you should have used OR. Or the other way around.

00:12:06 Karen Plum

So that we get the best available evidence, we are recommended to search for peer reviewed studies which are found in peer reviewed journals. So it's firstly important to distinguish between these and non peer reviewed sources, as Denise explains. 

00:12:21 Denise Rousseau

Most academic journals are peer reviewed. Non peer reviewed journals we call magazines. The Harvard Business Review is a magazine, it’s not a journal and that's an important distinction, but magazines are very important! They illuminate information and invite often very talented authors, but whether or not they provide peer reviewed evidence that has been critiqued for trustworthiness, magazines don't do that. That's not their job. 

00:12:50 Karen Plum

So magazines play an important role for sharing ideas, opinions and experience, providing inspiration and discussion. I've read plenty of great HBR articles - they get me thinking and I share them with colleagues. But if we have a business critical decision to make, then we need good quality evidence from peer reviewed journals rather than a single study written up in the Harvard Business Review.

I've also seen academic research written up in an article in HBR. And it's not always clear how it was conducted. Even when I tracked down one particular study, there were many questions and gaps. Many of our clients were asking my colleagues about this article and so I drafted a response, simply highlighting the gaps and the things that they didn't talk about, which caused me to be cautious about the claims that they made. 

To this day, people still quote the research to me as the last word on the topic. So it goes to show that these articles really have staying power. 

Anyway, I was curious to know more about the peer review process. I think it's important to be able to explain why you did or didn't include studies in your evidence base. And being able to clearly explain this to others is part of being credible and knowing your subject. 

Academics obviously want to get their studies published, so other academics can see their work and to contribute to the body of work on their topic. The authors submit their study to an appropriate peer reviewed academic journal (oh, and more on journals later). 

The journal editor will then invite 3 experts in the field to critically review the study and make comments and suggestions. All of this is done anonymously and I wondered why. 

00:14:29 Denise Rousseau

Oh, I think the anonymity of the peer review process is really important. Number one, the reviewer is a stand in for the field as a whole, not for himself or herself, and so it takes issues of our own identity out of what we're writing. So when I give somebody feedback on a paper I'm not giving them feedback as Denise Rousseau, researcher on organizational change or psychological contract; I'm coming at it from the perspective of - I'm a psychologist with some years of experience and I'm trying to help this person do better work. 

One of the issues I think with that developmental orientation that has become increasingly important in the peer review process in the last couple of decades, has been that you're trying to help the author find the trustworthy part of his or her work and strengthen and show that off.

That can often mean you ask them to peel away things that are more gratuitous, or aren't really strong enough to stand up as a trustworthy claim, to help them craft what is effective. 

00:15:38 Karen Plum

I guess if you knew who was commenting, you might be open to authority bias, or you might take what you know about that person and contextualize their feedback. Maybe you start to think oh, that person doesn't like this sort of research or whatever, whereas this way you have less opportunity to find ways to dismiss what they've said. 

I can imagine that sometimes you may get conflicting feedback from different reviewers, and Denise explained that it's then the editor's role to help the author to navigate that journey. 

Given that Denise has been a journal editor and is also an active peer reviewer of other people's work, I wondered what sort of feedback reviewers typically provide. Denise identified three areas. 

00:16:21 Denise Rousseau

First is that how well framed is the question or the problem and whether or not it comports with what's already known about the problem - are they missing existing literature, you know, if you kept using the word morale over and over again and morale is not a scientific construct, but job satisfaction is. You know, they're gonna say, well, wait a minute, what do you really mean by that and what literatures are you trying to tap in the framing of your question. So the first is the setup and the logic of how the question’s being framed.

Then there's that broad swath of - how did you study your question? What research design, what samples, what surveys, what interventions, what background information have you presented? You know it's often much easier to get consensus on the methodological quality of the study than it is on the logic and the grounding in the literature. So that's the second area that you'll get a lot of critique on. 

And I'd say the third area - I call this basically it's the marketing of your ideas. At the conclusion of the paper it would be the implications you know, are you really telling us much about how the theory should change, how the research practice should be different? Reviewers are often very good at developing the connections between the big ideas in the field and what you found here, and what the research implications are and the practice implications. 

And that has the nice knock on effect of then threading back into the beginning framing of the article when you talk about your contributions and in my experience that's one of the areas where reviewers are particularly helpful, is having the scholar think in terms of how does my research contribute to the field and framing it up front, so that when you get to the red thread at the end, it's clear what impact you've had. 

00:18:09 Karen Plum

As a researcher, I guess it must be good to know that there are other academics out there that are prepared to provide you with support for your work. If you're working alone or in a small team, and you're fully immersed in the process, it must be good to get fresh eyes on what you're doing. 

A bit like expanding the expertise in your team, if only remotely. As a support for learning, I think that must be great and by the same token, if you've documented your search process and you're sharing the results with others, there's always a likelihood that someone will mention the study you didn't find or didn't include. 

I asked Barbara how she reacts when this happens. 

00:18:48 Barbara Janssen

What you want to do with evidence-based practice in general, is to try to get everyone to make use of different sources of data and to get people to think about it. So the good thing is, when someone comes up with an article, it's someone that's interested, had a look at it, thinks this might be interesting - why didn't you include it?

And I think there's always a discussion that’s worthwhile to have with someone, because it makes you better, it makes someone else better, it makes the process clear about why we are looking in a certain way. In that sense, it's a gift to the process, I think. 

00:19:23 Karen Plum

And that's the thing with evidence-based practice. There are always opportunities to learn, to improve and to share. 

Moving on and returning to the peer reviewed journals, all of which have electronic or printed pages to fill, there must be a sense of needing to feed the beast, as well as to maintain their standing as respected and prized sources of research. But there are also others that we may not have heard of. 

That said, the standing of the journal isn't a guarantee for the quality of the research that they publish. Sometimes studies published in highly ranked journals are later found to have spurious findings that can't be replicated. And in any case, what do we mean by highly ranked? 

Eric explains that this is about impact factor and relates to the number of times studies are cited or referenced by other researchers. 

00:20:15 Eric Barends

If your study is published in a journal with a high impact factor, you get a lot of points and if you have a lot of points, that means you're doing very well, that you make an impact. However, however, that is not an indication for the quality of the study. A lot of people think it is, but it's not.

And again, it's often the other way around. Journals that are frequently cited are cited frequently because the studies they publish have very special or surprising findings. However, surprising findings, or new findings, or completely new insights often turn out to be wrong after a while when we try to replicate them. 

So again, that is not an indication for the quality. We need to determine the quality of the study by critically appraising the study, not by looking at where it was published. 

00:21:25 Karen Plum

Even the fact that the studies are peer reviewed doesn't guarantee quality. And when we talk about quality, this relates to whether the study design was appropriate for answering the research question. The methodology may have been sound and a correlation identified, but if we're looking for evidence of a causal relationship, the ultimate prize, then this doesn't advance our cause, as Denise explains. 

00:21:49 Denise Rousseau

So is a study published in a peer reviewed journal automatically trustworthy? No! It’s made a cut - there's some odds that it can be trustworthy. 

But critical issue in evidence-based practice is the definition of trustworthiness of a scholarly article or publication, is a function of the research question you're asking and the design the study used in order to answer the question. So if you as a practitioner are searching on an issue where you have a causal question - what are the success factors that lead to psychological safety - that's your question, it’s a causal question - how do we get from here to there. 

You need studies that actually can test whether there is a causal relationship. They test whether or not there's been a change in psychological safety, as a function of the success factors that you're interested in or looking at. You have to have measured that change or controlled for the change, so the study can't just be a survey. It can't be a case study, you know, because neither of those looks directly at change and rules out alternative explanations for why a variable moved - the needle on a concept moved. 

You need a certain design, and that means therefore that us as the evidence-based practitioner needs to make a decision for himself or herself, whether the kind of study that the study that is reported in the journal, provides trustworthy evidence on the question you ask. 

So trustworthiness is a function of the question that you ask, it's not a function of the study of where it appears. Shoe on the other foot, however, when that researcher did the study that is now published in a peer reviewed journal, they asked a certain kind of question and they asked it in a certain way. The job of the peer review process is to identify whether the question that the scholar asked has been responded to in a trustworthy way in the design that's used in the published study. 

And one of the things I think that is likely to be the case is that the peer reviewers have done a decent job of identifying the trustworthiness of the design and the methodology used for the question the author asked. 

00:24:20 Karen Plum

As with other aspects of the evidence-based process, we take nothing for granted. It's all about being vigilant in terms of what's appropriate and relevant for what we're researching. Another way of getting a good body of evidence about your topic is the meta-analysis. 

If your search identifies one or more of these, then that's a great result. Apart from anything, it'll confirm that the search terms you're using are well established and will provide good explanations about what's being researched and identified so far. So what is a meta-analysis? 

00:24:55 Eric Barends

It's a study where the researchers have done a lot of the work that we intend to do. They searched for studies on a specific topic in a very thorough way, combined all these studies, had to look at the effect sizes and calculated the pooled as we say, or mean average effect size based on all these studies - the aggregated effect size based on all these studies. 

And that's great because instead of having just one study and a second study and a third study -meta-analysis are studies of studies. So they are often based on at least ten or twenty or sometimes even sixty or even more studies. So if you find a meta-analysis, you should be happy because then the researchers have already done the work for you. 

00:26:00 Karen Plum

Naturally, the date of the meta-analysis is important. If it was done within the last few years, then it's likely it has strong and good evidence. Anything conducted longer ago may also be good, but as we're taught, we need to ensure we find out what has been researched more recently, that may impact the results of that meta-analysis. All of that said, Eric explains that there is something else we need to bear in mind about meta-analyses. 

00:26:27 Eric Barends

A meta-analysis often only provides you a correlation. We will explain in module 6 and 7 exactly what that is. But as you probably already heard, a correlation is not the same as a causation. So when there is a relationship between two phenomena, between an intervention and an outcome, it doesn't mean that the outcome is caused by this phenomena or this intervention. 

So that is something that is very important to take into account that most of these meta-analysis are based on cross sectional studies, correlational studies, so what they provide is a relationship. Is there a relationship between these two variables or between this construct or this intervention and the outcome. 

And only a handful of meta-analyses are based on controlled studies or even randomized controlled studies. Obviously, when you find meta-analysis based on randomized controlled studies, that would be absolutely great. However as you will see in modules 6 and 7 where we explain what is a randomized controlled study and what is a controlled study or longitudinal study, there are not many out there.

Again, most of these meta-analysis give you information about whether there's a relationship between two variables, or a construct and an outcome. Still, that can give you a lot of information. 

00:28:03 Karen Plum

So more to come in the next two episodes. 

Having learned and discussed the process of searching for studies, it's important to recognize the difference between what Eric calls the ‘nuclear option’ and what we would typically do in everyday life, when we need evidence on a specific topic. 

The process we've learned is there when that's what you need, but if you need to do something in the moment, to see if there's anything available on a particular topic, what do you do? 

00:28:31 Eric Barends

The way the module teaches you how to search is kind of the nuclear option, rather than you would go about this in daily practice. You should take into account that if you want to see whether there's any relevant research on the specific topics on a daily basis because someone makes a claim during a meeting or whatsoever, you probably will just go to Google Scholar. Just use one search term and see whether there are meta-analyses and that's it. 

That'll be your search and that'll give you a good idea of whether there are any studies out there, and if there are meta-analysis that's already an indication that there are a lot of studies, otherwise there would not be a meta-analysis. Start with a simple search with one term, see what's out there in Google Scholar and go from there. 

00:29:32 Karen Plum

And finally, I've spent a lot of time in this episode discussing peer review and why it's important to always look for studies that have been through this process. It's easily done by checking the box on the research databases, so that only these studies appear in your search. 

If you'd like a succinct summary of why it's important, I think this explanation from Denise pretty well wraps it up. 

00:29:56 Denise Rousseau

Peer review is important because it is a way of de-biasing the assumptions that the author has made about his or her work. It is a methodological check on the appropriateness of the research design and it is a logical check on the arguments and assumptions and framing that the author has engaged in, in positioning the study and interpreting the results.