{"id":831,"date":"2021-11-05T21:55:00","date_gmt":"2021-11-05T21:55:00","guid":{"rendered":"https:\/\/cogitateandpercolate.com\/?p=831"},"modified":"2026-06-29T14:03:54","modified_gmt":"2026-06-29T14:03:54","slug":"survey-does-not-show-that-instructional-designers-drive-better-student-outcomes","status":"publish","type":"post","link":"https:\/\/cogitateandpercolate.com\/?p=831","title":{"rendered":"Survey does NOT show that Instructional Designers Drive Better Student Outcomes"},"content":{"rendered":"

 <\/p>\n

<\/a><\/div>\n<\/p>\n
\n

This article, Instructional Designers Drive Better Student Outcomes<\/a>,
\n made the rounds in 2019 and now that it is time for end-of-year
\nreviews, it is popping up again. I need to make my objections to the
\nconclusions of this article known because if I was a CEO and handed this
\n article as justification for an Instructional Design (ID) department,
\nI’d toss the article back across the table.<\/span><\/p>\n

Poor research is worse than no research.<\/span><\/p><\/blockquote>\n

Before
\n I articulate my objections and show you some tips on how to analyze
\nresearch papers, I would like to state my philosophical bias clearly: I
\n am, by degree & interest, an Instructional Designer. I believe that
\n Instructional Designers can create and improve instruction. <\/b>There
\n is a plethora of bad instruction out there and I’m part of the group
\nthat supports better instruction.
 <\/span><\/p>\n

Please be clear on this point because
\nnow that I’ve said that, I think the 2019 CHLOE 3<\/a> Report (hereafter referred to as the report, APA reference at bottom of this page) together with the Campus Technology article<\/a> (hereafter referred to as the article) headline are garbage.<\/span><\/p>\n

Objection #1: Analyze the population and sample.<\/span><\/h2>\n

From the very first paragraph of the article, the claims seem compelling:<\/span><\/p>\n

“When
\n faculty are compelled to work with instructional designers on
\ndevelopment of their online courses, students have better outcomes,
\naccording to a recent survey from Quality Matters<\/a> and Eduventures Research<\/a>,
\n the research division of ACT\/NRCCUA. In schools where instructional
\ndesign for online course development was absent or optional, 58 percent <\/b>of
\n “chief online officers” (COOs) believed that students taking those
\ncourses would perform at least as well if not better than those in
\nface-to-face classes; that jumped to 70 percent <\/b>where instructional design was mandated.”<\/span><\/p>\n

Mind
\n you, depicted directly above this paragraph is a graph which does not
\nshow a 70% Y axis interval.<\/span><\/p>\n

Hmm…70% in the text and no 70% on the graph? That’s your first sign that something might be wrong. <\/b>These
\n numbers might be aggregated…as in…more than one category of
\nresponses has been added together to get a bigger number. At this point,
\n that is not a research sin<\/i>. But which numbers were added?
\nBetter and same? Same and worse? What? I get the feeling that the graph
\nand this paragraph, even as they are snuggled together, might not go
\ntogether. It might mean that the writer is reaching for conclusions
\n(pushing the data) that the data doesn’t show. Hackles raised. I’ve got
\nto get my hands on the actual data.<\/span><\/p>\n

I surfed off to find the 2019 CHLOE 3 Report.<\/span><\/p>\n

In order to obtain access, I had to fill out a form.<\/span><\/p>\n

Interestingly, on the form I was given the option to disclose that I, myself, am a COO<\/u>. I gave that a moment of thought. Hmm…okay, so I am technically the COO of my own home office<\/i>. But…does that mean I will be surveyed for the next CHLOE Report?? We’ll see<\/i>.
\n (I *will* update this article.) Immediately, the problem here has me
\nasking, “Who exactly did they survey?” Just people that disclosed they
\nwere COOs? And by what standard? Because they said so? Because they
\nclicked “I am the Chief Online Officer or equivalent?” Is it possible
\nthat only progressive, forward-looking campus representatives even read
\nor know of the CHLOE survey? Maybe only the technologically jazziest
\nCOOs reply to surveys? Or maybe COOs that have time to read their email
\nand are, perhaps, not the least bit jazzy?<\/span><\/p>\n

Point of order: whether you are a COO or not, why do you need this<\/i> information to give me access to your report? Casting my very best Spock raised eyebrow at you.<\/span><\/p>\n

My
\n notes say I received this report on June 26, 2019 at 7:23 a.m. Eastern.
\n CHLOE does a breakdown of the respondents on pages 6-8. I can’t filter
\nout the “jazzy” factor but this report goes to great lengths to tell me
\nthe demographics of these respondents. Wow. The stats person on the
\nother side of this was on full caffeine the day they ran this data.<\/span><\/p>\n

Objection #2: The data. What was actually collected?<\/span><\/h2>\n

I’m
\n searching for the data on the use of instructional designers. ID is on
\npages 22-24. I’m looking for 70% (a good anchor number to scan for since
\n it was at the front of the article).<\/span><\/p>\n

A search found 70% in two places in this PDF between pages 22 and 24.<\/span><\/p>\n

What’s
\n this? This time the report snuggles the 70% directly up to the same
\nimage from the article and yet, there is no 70% on this diagram either.<\/span><\/p>\n

This
\n 70% does not seem to be related to that headline. It’s just reporting
\nthat among enterprise institutions, 70% of them grant faculty
\nautonomy\/academic freedom as to why instructional design is absent or
\noptional. Not relevant for my writing.<\/span><\/p>\n

But I want to get to the
\nheart of the data question: What did they measure? I scoured this
\nsection (and the whole report really) and found this was the most
\ndescriptive response:<\/span><\/p>\n

“A higher proportion of COOs at institutions that require ID use in online course development judge <\/b>their fully online students as performing comparably to or better than on-ground students than claimed<\/b> by COOs from institutions that do not require ID support”<\/span><\/p><\/blockquote>\n

Bold<\/b>
\n emphasis added by me. Thus, the survey respondents judged and claimed.
\nNo problem with that as surveys can collect opinion results. But let’s
\nlook at the headline again:<\/span><\/p>\n

Instructional Designers Drive Better Student Outcomes<\/span><\/p>\n

That’s not what the data shows is it? The survey did not measure student outcomes. It measured the perception (judge, claim) of student outcomes. <\/i><\/b>
\n It collected subjective data. Is it possible that COOs love their ID
\ndepartments and don’t want anything to make them look bad so they
\nresponded, “Of course my ID department helps, that’s why I have one!”<\/span><\/p>\n

The
\n survey did not appear to ask for objective student outcome data at all.
\n *CHLOE 3 authors if you did ask that question, please clarify. Hint: If
\n I was a CEO, *that’s* the data I’d like handed to me.<\/span><\/p>\n

At this point, my interest is really peaked.<\/p>\n

<\/span><\/p>\n\n\n\n\n
<\/a><\/td>\n<\/tr>\n
CHLOE 3 Figure 15<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

<\/p>\n

I’m struggling to make the diagram match anything<\/i>
\n in the written accompanying text. <\/span><\/p>\n

Objection #3: What does the diagram actually show?<\/span><\/h2>\n

Let’s use our graph reading skills and dive in.<\/span><\/p>\n

First, the data appears to be split into 2 groups:<\/span><\/p>\n

Option A:<\/span><\/p>\n

    \n
  1. Left side: Online student performance (perceived)<\/span><\/li>\n
  2. Right side: On-ground student performance (perceived)<\/span><\/li>\n<\/ol>\n

    Or<\/span><\/p>\n

    Option B: No sure which columns refer to online versus on-ground, but X axis seems to claim with or without ID support.<\/span><\/p>\n

      \n
    1. Left
      \n side: With ID support (assuming that knowing whether your institution
      \nhas ID support is a determinable fact, not an opinion)<\/span><\/li>\n
    2. Right side: Without ID support.<\/span><\/li>\n<\/ol>\n

      Since
      \n the title takes up a great deal of real estate in a graph, it’s fair to
      \n say that it should be the more dominant piece of data, so I’ll assume
      \nOption A is the correct interpretation of the two groups. Bear in mind
      \nat this point, that this graph is showing the (perceived) difference ID
      \nmakes in two different populations (online versus on-ground) and not
      \nwhat difference ID makes versus non-ID. Remember the headline said<\/i> Instructional Designers Drive Better Student Outcomes, no mention of online versus on-ground.<\/i>
      \n However, I’m willing to analyze this graph based on three separate
      \nassumptions (ID versus non-ID, online versus on-ground, and mix those
      \ngroups up too) so let’s continue.<\/span><\/p>\n

      The Y axis is showing us
      \npercents so really quickly, we should decide if this is a number that
      \n*should* add up to 100% maximum or could the total percent go higher?
      \nSome aggregates can go higher than 100% if a respondent can pick more
      \nthan one answer. Scanning these columns, it looks like if you added all
      \nof the columns of a group together, we’d get 100%. OK. So these
      \nrespondents could not be part of more than one column. Each respondent
      \nhad to be part of a discrete, exclusive set.<\/span><\/p>\n

      Reading the colors, the turquoise columns are those COOs that felt that their student performance (interestingly, not student outcomes?<\/i>
      \n Watch that vocabulary crossfire boys!) was better with ID. So referring
      \n to my philosophical bias, we like this group. Yay! But they seem to hit
      \n the 15% and 19% mark. Ugh, those are not high numbers.<\/span><\/p>\n

      The dark
      \nblue column is the COOs that felt that their student performance was the
      \n same. Now I will make a leap here. I can be argued against this but I’m
      \n going to take the stance that any set of data that indicates a result
      \nof ‘same’ probably shouldn’t make headlines. Notice however, that these are the largest columns.<\/u>
      \n So in an alternative universe, the article could have been titled
      \nInstructional Designers Seem To Make No Difference In Student Outcomes.
      \nNo column crossed the 60% threshold either.<\/span><\/p>\n

      The orange column is the COOs that felt that their student performance was worse with IDs.<\/span><\/p>\n

      OK, I gotta stop here for a moment.<\/span><\/p>\n

      Bwhahahahahahahaha!<\/span><\/p><\/blockquote>\n

      I love it when a plan…<\/span><\/p><\/blockquote>\n

      totally explodes.<\/span><\/p><\/blockquote>\n

      As you can tell, I have disdain this group and wish it did not exist. However, the scientist in me loves the fact that I have to acknowledge that it is possible<\/i>
      \n that the presence of ID could totally muck up the works. Statistically,
      \n it is possible that there are institutions filled with awesome
      \nteachers, doing the best instruction, utilizing the best learning
      \nscience, and creating wonderful learning outcomes. Insert a mediocre (at
      \n best) ID with a job task to, for example, input boilerplate language
      \ninto every syllabus, and it is possible that (perceived) student
      \noutcomes take a hit. This data just makes me laugh. But I’m not laughing
      \n that much because those columns are pulling 30% and 42%, both numbers
      \nhigher than my beloved turquoise. Owch. Pipe down.<\/span><\/p>\n

      OK.<\/span><\/p>\n

      Still, look at that headline again: Instructional Designers Drive Better Student Outcomes<\/span><\/p>\n

      Remember that I’m not nit-picking an insignificant detail here. This diagram is the prominent one showing in the article<\/i>.
      \n But does the data supported that headline? Even if I tried to pretend
      \nit did…how does 15% and 19% surpass 59%, 45%, 30%, and 42%– all of
      \nthe columns that said that ID presence had no effect or a worse effect.
      \nThere are times when a low number can and should make a headline.
      \nThis…is not one of those times.<\/u><\/span><\/p>\n

      In conclusion, pulling
      \nthe article headline from this report was a pretty gross exaggeration of
      \n the data. But CHLOE 3, you are not exempt either because I’m still
      \nspinning on your 70%. This was the paragraph immediately preceding<\/i> the graph:<\/span><\/p>\n

      “In
      \n Figure 15, CHLOE 3 supplements these findings with evidence that a
      \nhigher proportion of COOs at institutions that require ID use in online
      \ncourse development judge their fully online students as performing
      \ncomparably to or better than on-ground students than claimed by COOs
      \nfrom institutions that do not require ID support (70% vs. 58%).
      \nConversely, 12% or more COOs from institutions that do not require the
      \nuse of ID expertise reported worse performance by online students than
      \nCOOs from schools mandating ID involvement.”<\/span><\/p>\n

      I take issue with: “a
      \n higher proportion..that require ID use…in online…judge…as
      \nperforming comparably or better than on-ground that do not require ID
      \n(70% vs. 58%).” This one sentence almost needs a flow diagram to unpack.<\/span><\/p>\n

      So they want to compare:<\/span><\/p>\n

      Online AND “better or same”, that’s left side turquoise plus blue, 15% plus 59% = 74%<\/span><\/p>\n

      against<\/span><\/p>\n

      Ground
      \n AND “not require ID” (no other different specification in the sentence,
      \n so we’ll go with a continuation of the first specification of “better
      \nor same”), that’s right side turquoise plus blue, 19% plus 45% = 64%.<\/span><\/p>\n

      The
      \n difference between those two groups (74% minus 64%) is 10%. They say
      \nit’s 12% (the difference between 70 and 58, not the 12% in the following
      \n sentence of their quote). I spent over an hour playing with the numbers
      \n from the graph but I cannot easily show you the calculations on this
      \nLinkedIn article. In summary, I’m more comfortable finding 58% from the
      \ngraph than 70%.<\/span><\/p>\n

      To be fair, it is very possible that these sets of
      \n numbers, 70% and 58%, reside inside a set of data that the report
      \nauthors is not providing, despite starting the sentence with “In Figure
      \n15.” Said another way, the data is real and valid, just not available to
      \n my eyes. That is a completely fair possibility. Occam’s Razor actually
      \npredicts that that <\/i>has the highest probability of being true
      \nbecause despite me playing with those percentages and trying to add them
      \n up in different ways; it takes contortion to get 70 and 58. Thus, I am willing to go with their 70% and 58% <\/i>because I have no other stronger evidence to work from.<\/span><\/p>\n

      Don’t
      \n mistake what I am getting at here. I’m not saying that their 70 and 58
      \nare false, fake, or poor. I’m saying if you’ve raised my interest, I’m
      \ngoing to use all of my skills to second-guess your work. As my former
      \nstudents know, I will fight hard if I align myself with your interests. But first you have to win me<\/u> over<\/b>.<\/span><\/p>\n

      I need to step back and look at these numbers again.<\/span><\/p>\n

      Reading
      \n that sentence over again a few times, they are comparing apples to
      \noranges: Online courses with ID support perform better or the same than
      \non-ground courses with no ID. Um…that’s not a fair comparison, is it?
      \n(I know, I’ll carry the Clark versus Kozma debate with me for a VERY
      \nlong time. Only IDs get that reference<\/i>.) To be clear, the confounding variables in this comparison are that online courses are:<\/span><\/p>\n