Taking the course

This episode gives a background to the course - how it was created, its founding principles, the science behind the “learning by doing” approach used by the course (which shows that it is evidence-based!), and how to get the best out of it.

The course design recognizes that students come from a variety of backgrounds and have different experience and perspectives. Some study as a preparation for a career in management, while others are already managers who want to make better decisions and critically appraise the evidence before them. Some people study with others – either as part of a college or university course, or within their organisation - while others study by themselves. 

Whether you are a lone learner, or a part of a group, there are tips and recommendations from our learning and subject experts which will hopefully help you understand how to get the best from the course. You will also learn what other students enjoy and find challenging about this method of learning.

The podcast is a companion to the course – another way to engage with the material, to deepen your learning as you progress through the modules. 

Author mentioned by Erin: Ken Koedinger

Host: Karen Plum

Guests

• Eric Barends, Managing Director, Center for Evidence-Based Management 
• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University 
• Rob Briner, Professor of Organizational Psychology, Queen Mary University of London
• Erin Czerwinski, Manager, Learning Engineering, The Simon Initiative, Carnegie Mellon University

Find out more about the course here:   https://cebma.org/resources-and-tools/course-modules/

Transcript

00:00:01 Karen Plum

Hello and welcome to this episode of the Evidence-Based Management podcast. This time we're talking about the course itself, how it was created, and how to get the best out of it. 

I'm Karen Plum, a fellow student of evidence-based management, and I'm joined this time by three of the course authors Eric Barends, Denise Rousseau and Rob Briner, and in addition I'm joined by Erin Czerwinski, a learning expert at Carnegie Mellon University and all of them are answering my questions about the course and how it helps us learn the skills we need. Let's get started. 

00:00:39 Karen Plum

As a new student of evidence-based management, I was keen to know what advice experts like Denise and Eric gives students embarking on the course. 

00:00:57 Denise Rousseau

Essentially, the course is about learning to be a reflective practitioner. So learning is more important than performance, than the score you get. You only have to pass about 70% of the questions to qualify, but the point is that you get feedback and you learn as you go. And the reflection part is to begin thinking about why is that a good answer? Why is that something that makes the evidence more or less trustworthy, and in that reflection is where learning happens. 

00:01:38 Eric Barends

Take a learning approach. It's really important that you do this course with an open perspective, an open mind and learn from the feedback and learn from the exercises, rather than rushing through it and you know, get your grades and get it done.

The modules are really practical. They teach you skills, you can apply this. It's not knowledge that's in your head and slowly will disappear. No, it's a hands on approach. You know how to do things. You know how to solve issues. 

And you know how to go about issues you come across in your organization rather than being powerless. So it empowers you, I think, and therefore, that would also be my recommendation. Look at it in that way. How can you use this in your daily practice and try to learn from it rather than just ticking the boxes and go through it very, very fast. 

00:02:38 Karen Plum

The course is taken by different types of students. Some who have been working as managers and others who have yet to enter that world - undergraduates or postgraduates studying evidence-based management as a standalone course, or perhaps as an elective topic or as part of an MBA. 

Clearly everyone experience will be different. Experienced managers bring their own context and knowledge to bear when considering the exercises and the quiz questions and they may react differently to the answers. That's why the feedback is so important. It explains how the authors were thinking about the problems that they present us with. Those with no practical management experience have different challenges, as they haven't yet experienced the scenarios that we're presented with, and so it probably all feels quite theoretical. 

I asked Rob Briner how he addresses these situations with students. 

00:03:30 Rob Briner

You can try and talk them through some of the other sources of evidence such as experience, organizational data, stakeholder perspectives, you can talk them through that, sure. But they don't have access to that and probably never used it in their experience, so what you can focus on a bit more is the scientific evidence part of it.

So typically again, with these relatively inexperienced students, I always try and give them very practical, hands on experiences in terms of, for example, in this case, trying to do mini systematic reviews of the scientific evidence around particular management questions, but also providing them with the context in which the question is being asked. 

So it's quite hands on thing, and particularly I think it's focusing on the quality of data and evidence and information. 

In the stuff I teach at that level, it's very much around, just in general thinking about the quality and trustworthiness of all kinds of data and information. Claims are reasonable, claims are not. And honestly, focusing in particular on scientific evidence.

When it comes to students who are part time and working in organizations, it's sort of obviously much easier to incorporate those other four sources of evidence too. 

But I think the same principles apply whatever source of information is, what we might also call critical appraisal, that’s always a very helpful term, but it means, yeah, how trustworthy or reliable is this information. 

So applying it to that as well, and I think probably with managers is trying to get over what is different about evidence-based practice or evidence-based management from the way we usually do things. Also I spend quite a lot of time saying look, this is some of what we know about how decisions are typically made. This is what a more evidence-based or evidence informed decision would look like. 

So really trying to get over the differences. Otherwise, I think students and possibly managers as well are left a little bit the impression that it's something like only that you use a bit more information and it kind of is that, but it doesn't really capture the differences very well I think. 

00:05:32 Karen Plum

The founding principle of the course is learning by doing. I wondered if this is more challenging for those studying the course before embarking on a career in management. Part of the learning will be drawing on the experience of lecturers or other experts contributing to the course, and that will be valuable, of course. But what about the doing part? 

Some courses provide opportunities to practice through internships, but Eric explained that there is another opportunity, not only to practice, but to share what you're learning and to make a contribution to other people’s thinking. 

00:06:05 Eric Barends

So if you are doing this course with limited experience in an organization, but you're doing this because you want to become a manager and you think this may help, yeah, it certainly will help, but of course you will learn more if you have an opportunity to practice. In that case, think it would help if you would try to find people that are already in a more or less managerial situation. So maybe your sister is a Head Nurse, your neighbor is working as a supervisor somewhere in an organization or you have an uncle or an aunt that is a director or a manager. 

Get in touch with them, explain what you're doing, and have a discussion with them. So how does your organization typically make decisions? A lot of the assignments - you can try them out with this person and ask them so how our decisions typically made. So all the assignments also apply to them, so that's how you can learn and I think they will be interested if you would say, listen, I want to do a small assignment, what kind of questionnaires are being used in your organization? 

Can I have one because I want to have a look at the wording and maybe make recommendations for improvements. Hey, that's helpful. You're already making a very good contribution and you add value as a manager even though you're not a manager and not hired by these people yet. 

00:07:38 Karen Plum

Rob reinforces the use of practice and of course, the more you practice, the better you become and the more confidence you gain. 

00:07:46 Rob Briner

So for example, the idea of looking at multiple sources, why does that make sense? You can describe in a classroom, we can give some examples, but actually really doing it on the ground, as it were, multiple sources makes more sense. Following a structured approach makes more sense if you've got a real problem or issue to deal with. Simply, I think taking a structured approach also does. I guess like any skill, the more chance you have to practice it, the better and it's getting that practice time I guess. 

00:08:15 Karen Plum

People studying the course as part of a group at a college, university or within their own organization, have the benefit of being able to discuss the problems and challenges presented in the course with each other. 

Learning from others is another valuable part of the experience. We aren't looking for black and white answers a lot of the time, life isn't like that. We need to be able to navigate to find the best available evidence for our context, and at that moment in time. 

So let's turn now to the design of the course. I was curious to know the thinking behind the structure and approach of this online learning course. Is it evidence-based? I asked Erin Czerwinsky, learning engineering manager, at the Simon initiative at Carnegie Mellon University. 

00:09:01 Erin Czerwinski

We wanted to figure out how can we make or prove that online learning, if designed a certain way, could really be just as effective as face to face or even tutoring, one on one tutoring. And we did. 

We did actually bear it out, and it's because we used what we knew about the science of learning and essentially the principles that are in a book called How Learning Works that was written by a bunch of folks, but many of them were at Carnegie Mellon University at the time. And really taking what we learned in the research at Carnegie Mellon and distill that out into actual design principles and ideas that any teacher or designer could use when building any kind of learning, actually, but it turns out these principles bear out just as well in face to face as in online learning and what those are essentially you learn by doing, making sure that practice is tied to goals. 

We also paid careful consideration to the learning objectives, making sure that they're articulated in a way that are student centered and measurable, so that you know learning when you see it, because you're measuring it that accurately. 

00:10:23 Karen Plum

In terms of measurement, Erin explained that as students progress through the modules, exercises, and quizzes, the system tracks their progress not just to see how well they're doing, but to understand whether the concepts are explained well enough. If we all trip over something, maybe it's not well presented or explained and needs to be reviewed by the learning scientists and subject experts. 

Other data can be shared with instructors, teachers, or faculty so that difficult things can be used in class for further discussion and exploration. 

So if Big Brother is watching you, then he seems to have a good heart and wants you to succeed! But is this course in evidence-based management, indeed evidence-based? I asked Erin about the evidence that learning by doing is the best approach. 

00:11:12 Erin Czerwinski

The research is clear on this really. More active learning opportunities are better than passive, and that's proven out in a framework called ICAP. It's a framework for understanding what activities are most useful for learning and when. 

And then the research that Ken Koedinger did on learn by doing really proves this out, and it's been replicated even to the same results. We know that just the more of this that they can do, that the students are exposed to, the better off they'll be, even in the real world. 

Again, what we're trying to get to is that transfer of learning, meaning that when students are given a new scenario, where it has information that they need from their foundational learning, that they can actually not only remember that and use it, but extend it in new situations, and we've learned that you know this approach definitely leads to that transfer of learning. 

00:12:15 Karen Plum

Erin also shared with me a long list of sources that lie behind the approach that has been followed in this course, both in terms of general learning, online learning, and learning by doing. I find that reassuring and love that the course in evidence-based management is indeed evidence-based. 

I also looked up the ICAP framework, which describes four types of engagement, interactive, constructive, active and passive. This framework, as Erin mentioned, helps teachers and learning designers to determine whether their activities and assignments lead students towards the intended outcomes and whether they are engaging in active learning along the way. 

From my point of view, I'm enjoying learning about the depth and value of taking an evidence-based approach. I've worked as a management consultant for 20 odd years and I'm frankly amazed by how much I have to learn. I value the prompts to take time for reflection. 

As an introvert, I know that this is where the magic happens, but even then I've been surprised by the journey I've been on, even through the first few modules. So I asked Eric about what other students enjoy about the course. 

00:13:26 Eric Barends

The examples! They enjoy them because we of course, in designing the course, had to take into account the background of the students. So some students do an MBA, so that's a lot of stuff about big corporations and strategic decisions. Other students do a program or a course that is, on the level of a supervisor. That deals way more with people - there's large respect on focus on people management.

Or they do a program on change management. Or they’re in the realm of an insurance company because they do an internal course. So we try to come up with examples that vary all over the spectrum. So not only the domain - insurance, healthcare, consumer goods, IT, etc. So the examples come from different things, but also the type of problems differ. 

Sometimes it's people management, sometimes it's change management, sometimes it's logistics, sometimes it's sales, sometimes it's finance - to signal that all these domains, all these disciplines benefit from an evidence-based approach. It's applicable to all these domains. But it also makes it more fun to do all these learn by doing exercises with all the different examples, so that's what students really enjoy. 

The other thing that students seem to enjoy is to be challenged on their assumptions. In particular, students that have already substantial experience in management. They feel tricked in a pleasant way and they are challenged by what is brought forward.

They have assumption of their strong ideas, but then they realize that the way they make decisions so far is actually pretty flawed or could be improved. But sometimes it's just that they recognize all the examples of companies and they can relate to that because they've seen it in their own organization. 

00:15:37 Karen Plum

When I first started the course, I got very frustrated by some of the learning by doing and quiz questions. Given a pass mark of 70%, I was gutted to score lower than that on each of the first two quizzes that I did. I'm not used to failing. I started to get a bit despondent, thinking I wasn't up to understanding what the course was trying to teach me. How had I become so dumb? 

Next, I got a bit angry when looking at the feedback to understand the correct answers. Often it seemed we were splitting hairs. How could I possibly tell the difference between “somewhat” and “very little”? I felt it was impossible to calibrate my answers to get more questions right. I asked Denise if this is a common reaction. 

00:16:21 Denise Rousseau

Oh, that's always one of the challenges of older professionals who have notions of their capabilities! I, for example, teach physicians, medical doctors in evidence-based management - because they're moving into managerial roles. And if they don't get 95% of the answers right on the evidence-based management modules, they go ballistic!

So my job is to calm them down and say what really matters here is that you're learning. Are you learning new things? OK, yeah, can you reflect on what it is you're learning? Not so much what it takes to get a better score, but what's the basis of the answer and what makes it right? And why is that information you can use?

That's sufficient. Our goals are lifelong learning, as an evidence-based practitioner and having the tools that come from the evidence-based management class positions you on that trajectory. 

00:17:25 Karen Plum

And yes, I do fall into that older professional category, so that was another gotcha moment. I asked Erin what the science tells us about this. 

00:17:34 Erin Czerwinski

If you're going to put a credential to something, if you're going to know that folks have learned some foundational knowledge, can speak the language, can understand the concepts so that when they're in a situation where those concepts are being exercised more deeply, they can join in. They might still be learning, but at least they can speak the language they can hold their own in terms of that. 

And we need some way of knowing that, and so the score is a way of just seeing have folks met at least some threshold of retaining some of that knowledge, hoping that that will help with transfer down the road, right? When they encounter a situation they can not only recognize something at play because they learned it and they remembered it, but they might then at that point, be able to take it just even a little further. It's that experience, that kind of adds layers to that initial foundational learning. 

00:18:35 Karen Plum

In a good sense, this is preparing the new manager for the world that they're going to join. Building genuinely useful skills that many people they will be working with, simply don't have. And that's not just about a certificate or a qualification that their colleagues might not have. It's about a way of questioning, thinking and evaluating, not taking things as read because a senior person has stated it, or because that's the way we do things around here. Denise expands on this.

00:19:05 Denise Rousseau

What we've learned recently in learning science is that there's a big difference between learning and performance. That you have to learn to perform better. But many of our participants are used to trying to perform to get the answer and that really isn't the point. The modules are designed to help people understand the fundamentals and practice under different conditions, different problem sets or different cases, to use the concepts so that they get a chance to expand the connections in their understanding to actually make them work in the day to day. 

In order to do that you have to have an open mind to what works where and why. And that is inherent in learning research. Is the idea of exploration, that mistakes are a way to get feedback and that feedback isn't about the right answer, it's about the why. 

00:20:11 Karen Plum

To me, this is all evidence that I'm not learning facts or processes. I am being trained to think and to think differently. This is challenging, but it's also liberating and what makes it so valuable. The joy of the exercises and quiz questions is that they teach you so much about what you need to know. Through making you think and providing you with feedback, if there's a better answer. 

It takes a bit of getting used to though. Moving away from that performance approach and not getting hung up about being right all the time. 

I asked Denise and Eric what else do students find challenging about the course? 

00:20:49 Denise Rousseau

All of us have habits of mind about how we think about information, truth, what sources we respect, the opinion leaders who we grant credence to, the assumptions we make about what senior leaders know or don't. What is an expert? We have a lot of pre-existing beliefs. 

If we were blank slates, it would be a different world, but we're we are not. We come with many embedded concepts and then you're exposed to evidence-based approaches. 

I put great attention on figuring out two things. One is the trustworthiness of the information you possess. Trustworthy means well based on certain conditions, about features in the science, about controls, features in opinions and expertise about prior experience and the validity of the experience, qualities of reliability and validity, in your own organizational data. You're learning across all of these different types of evidence. What makes it more or less reliable, more or less valid to test the claim you might make? 

And many people aren't used to doing that amount of work, thinking about what they think about you know, is this really reliable? Is this something that really I should act on? And that's self-implicating. Second one that is challenging for people is becoming aware that very often when we're trying to make a decision or trying to make sense of information, we start too far down the path. But one of the critical issues in evidence-based practice is the concept of ask. What questions are important to the problem or the situation that we face? What's the real problem we're trying to solve? 

For many of us, well for all of us at some point in time, we think we know what the issue is. I want to fix this. I want to get this thing removed. But is what I'm focused on really the problem that needs to be solved?

00:23:02 Eric Barends

It is often again the exercises and the learn by doing stuff because the answer is different than they expected or they bring in their own context. So it's really hard first of all, for experienced students with the background, executive students, to have an open mind and not bring your own experience in context and starting to argue, yeah, I don't agree with this answer because you know, in a situation like A or B. These are clever people. 

You will always be able to come up with a situation where actually this does not apply. Or the answer’s probably slightly different or whatsoever, but that's not the point we're making. Take an open, learning perspective and try to say OK, in this situation I've come across several situations where this doesn't apply, but I get the central message here. This is actually a very important insight. 

00:24:03 Karen Plum

The way the course is designed, we are given information and then there are exercises where we're encouraged to learn by doing. This really is where real learning happens and where we start to consolidate our understanding. It continues through the quiz process, which again provides feedback to help us understand the nuances of the right, or the better answer. There are plans to add assignments to the online course and this is what university and college courses tend to do as well. 

In this way, students get the opportunity to think things through and apply their emerging skills in real world situations. We've probably all been on training courses where we get to the office and don't have an opportunity to practice what we learned and then we forget stuff really quickly. Eric’s advice is to practice as you go along. When you learn something new in the modules, practice it and this will help you and it will also add value to your organization. 

00:25:00 Eric Barends

If you ask these critical questions and ask about how are decisions being made, do you have evidence on this or that or whatsoever? Do you critically appraise? Do you look at the trustworthiness of the evidence. You’re already having an impact, I'm sure so it would even be better if you would have a CEO or an executive manager in your network somewhere, because there you can have the biggest impact.

And the great thing is, if you don't have any experience you can ask stupid questions. I mean, we just say - I have some very ignorant, stupid questions. And then you actually ask very provoking and challenging questions and you just act being stupid. So, but how do you make decisions? The way you tell me it seems to me, da di da. So you can be a little bit naughty there and yeah, that's great fun to do and you learn a lot from this. 

00:25:59 Karen Plum

Finally, as you go through the course, you may spot the odd typo or error in the material, and some of the exercise and quiz answers are still open to debate, as Eric explains. 

00:26:11 Eric Barends

The course modules are not perfect. We have developed and we brought in our own perspective and context etc. And we noticed that there are still learn by doing, did I get this or quiz questions out there that there is room for debate regarding the answer.

So sometimes your answer may be marked as incorrect, but there may be good reasons to discuss and to debate this. And of course there are errors, there are typos, there are things that don't work as it should be, and it takes a long time to figure that out. I discussed this with one of the learning engineers. I said wow there's still things you know there's small errors still there. 

And we have this module now for two years and they say Eric, don't worry, we have a stats course that's running for 10 years and hundreds of thousands of students have done this course. There are still errors and typos there, and they still crop up. So of course they will still be there and we would welcome all feedback you have for us. 

00:27:24 Karen Plum

I guess that shows that the course is a work in progress. There is continual learning and development to give students the best opportunity to learn about and acquire the skills they will need as evidence-based management practitioners. 

To finish this episode, here is a final piece of advice from Erin. 

00:27:42 Erin Czerwinski

Be patient and kind to yourself, right? You're learning and you're growing and you're changing. You're trying to enact a change in yourself in some way and that takes practice, it takes time and the materials are here for you wherever you are in that learning journey and just again - do the practice, do the suggestions in the course and you'll get there.