from
While today’s negative-attention-seeking elected officials are as exotic and dangerous as a pit of saw-scaled vipers, no political figure of the last century is as interesting as Jerry Brown. Governor of California for 16 years – two stints separated by nearly 30 years, making him both California’s youngest and oldest governor – Brown also ran for president multiple times and served Mayor of Oakland, a talk show host, an assistant to Mother Teresa, a student of Zen Buddhism, and – naturally – a professor at Berkeley. During his first term as governor in the 70s, Brown refused to live in the governor’s mansion, saying “it kind of looks like a Safeway store,” opting instead for his tiny Sacramento apartment with a mattress on the floor. All the more remarkable because he was dating Linda Ronstadt at the time.
Perhaps the strangest thing about Brown was his relationship with Jacques Barzaghi. Bald, dressed only in black, never seen without his trademark beret, and never heard without a thick French accent, Barzaghi was (in alphabetical order) Brown’s barber, bodyguard, chauffeur, clerk, decorator, and fashion consultant. Heavily tattooed, Barzaghi once insisted on appointing his personal tattooist to the Oakland Cultural Arts Commission. Despite also being Zen like Jerry, Jacques was married six times and notoriously hard to deal with. When asked a question, he had a tendency to stare a while, then say something cryptic like “don’t sell the skin of the bear before you shoot the bear.” During Brown’s run for the Democratic nomination in 1992, the New York Times profiled Barzaghi and, in response to a question about the state of the troubled campaign, extracted a humdinger we'd all be well-served to repurpose whenever possible: “We are not disorganized. Our campaign transcends understanding.”
Here's another campaign that transcends understanding: the rush to buy online tutoring for K-12 students. Since the start of the pandemic, school districts have spent tens of billions of dollars on optional tutoring. Combatting Covid learning loss by allowing students to “drop in” to see online tutors feels right – as Pittsburgh superintendent Wayne Walters proffered, it feels right to be able to give all students access to something. But like Linda Ronstadt sings, what feels right can turn out to be so wrong. As Hechinger Report noted last month, students who need the most help are least likely to seek it with opt-in rates as low as 2%. Brown University researchers warn that rather than reducing inequality, optional online tutoring could very well increase it. What does work? Mandatory intensive tutoring. But that’s eye-wateringly expensive. So like the drunk looking for his lost car keys under the streetlight (because that’s where he can see), school leaders have been spending like sailors on optional tutoring.
This begs the question of whether schools pay any attention to what works. Do schools care about research? When it comes to making purchasing decisions, schools blame providers, claiming they’re not getting the information they need: 75% of administrators and teachers say edtech companies don’t provide much evidence of effectiveness. But the simplest explanation is schools don’t ask. According to edtech companies, schools don’t ask for evidence-based outcomes like correlational or quasi-experimental studies let alone (the research gold standard) randomized controlled trials (RCTs). What do schools ask for? Case studies and customer references.
A report released earlier this year by LearnPlatform found that of the 100 most popular edtech products in K-12 schools, only a quarter had produced any research on efficacy. Not coincidentally, experts estimate that up to 85% of edtech products purchased by schools don’t or won’t work. Not caring about research has hampered edtech’s immense promise to improve student outcomes. Equally frustrating, it’s established a culture of cynicism when it comes to learning tools.
***
Even if schools are at sea, surely colleges do better. After all, many are research universities. But the evidence suggests otherwise. A Columbia Teachers College study from 2019 found fewer than 20% of higher education buyers mention research as a factor in purchasing edtech and concluded that “educational technology decision-makers in higher education rarely use externally-produced, scientifically rigorous research to inform their decisions.” Four years and a pandemic later, little has changed. When a recent survey asked over 300 universities about the most important attributes for making edtech purchasing decisions (I was shown the results on condition of only revealing this much), here’s what mattered, in order of priority:
So it’s not surprising that only 7% of edtech companies report investing in RCTs. Meanwhile companies that do have some existential questions for Jacques Barzaghi.
Mainstay is the leading learning success platform for the higher education lifecycle: enrollment, retention, and academics. Its AI-powered coach instantly answers 90% of student questions about everything from financial aid to parking no matter the day or hour. So when Mainstay launched, according to founder and CEO Drew Magliozzi, “the first thing we did was collaborate with Georgia State University on an RCT. From the beginning, we felt like having research to back up our work was absolutely critical — not just to be taken seriously, but also to prove that this idea we had would actually work.” Mainstay has now invested in nine research studies. Results demonstrating Mainstay’s impact on metrics like FAFSA completion, applications from first-generation students, yield rate, retention, and grades feature prominently in Mainstay’s marketing.
When I asked Drew how he felt about Mainstay’s stronger commitment to research than some of the research universities he sells to, he was charitable, referencing “pressures edtech decision-makers face that don’t always allow them to focus on research.” Mainstay plans to continue investing in research and encouraging other companies to follow suit.
But instead of investing in RCTs, what seems to be happening across the industry is short cuts – passing off case studies and surveys as research – and free-riding. According to Bart Epstein, CEO of the Edtech Evidence Exchange and research associate professor at the University of Virginia School of Education, it’s tempting for low-quality edtech companies to “hire a Ph.D for $10K to write a ‘white paper’ that looks just like an efficacy study, but is really just a case study of a single cherry-picked program.” And one edtech company that did not want to be named told me they were quite aware competitors were drafting off their research. The idea is simple: if an RCT demonstrates a product can increase retention, surely the product category has a similar impact. So a devious competitor with a similar product but no research whatsoever wins by flashing the RCT and pricing at a discount. This is particularly effective when media coverage of RCTs focuses on universities and the remarkable outcomes they achieve without naming the product, facilitating free-riding.
***
Colleges allow this to happen by failing to ask the right questions. According to Karl Rectanus, co-founder of LearnPlatform, higher education decision-makers are even less likely to demand proof of efficacy than K-12 buyers. The reasons:
***
K-12 seems to be getting its act together. The 2016 Every Student Succeeds Act (ESSA) provided an accountability framework for edtech by introducing four levels of evidence. Covid ESSER funding provided additional incentives for school districts to allocate more RFP points to evidence of efficacy. There are now a plethora of buyers guides advising K-12 decision-makers to ask for more than case studies and customer references. And LearnPlatform equips both districts and providers to run ESSA-aligned evaluations at a fraction of the cost of traditional research models.
Unfortunately, we’re not seeing equivalent progress in higher education. Higher education is all about freedom: freedom of speech, freedom of inquiry, and freedom from accountability. Because accreditors and boards of trustees don’t hold colleges accountable for student outcomes, the most important ones – employment, learning – don’t get measured. So while it’s possible to measure edtech inputs, there’s no systematic way to correlate them to outputs. As a result, few schools bother.
Of course, none of this would be possible without the $500 billion in taxpayer dollars flowing annually to colleges and universities. So where are the regulators on edtech efficacy? What is the Department of Education’s plan for higher education on while ESSA’s evidentiary standards only apply to K-12? The Department’s big edtech initiative is expanding the scope of third-party servicers. As I noted back in the spring, the push is to label all edtech companies as third-party servicers (TPS) – subjecting them to annual compliance audits and liability for compliance violations – in order “to protect the interests of institutions, taxpayers, and students.” While the current status of TPS is as enigmatic as Jacques Barzaghi, because federal financial aid is not conditional on any outcomes (let alone evidentiary standards), compliance has no connection whatsoever to efficacy. I don’t know what edtech tree the Department of Education is barking up, but if the Department is serious about protecting students, TPS isn’t the right one.
***
K-12 spends money on optional tutoring because it’s easy. Similarly, one reason colleges and universities privilege research over teaching is that it’s easier to measure research productivity (grants, publications, citations) than learning. This despite questionable efficacy – depending on the discipline, anywhere from 45% (sciences) to 98% (arts and humanities) of published research is never cited in any other publication. Much of this taxpayer-funded work feels good, but turns out to be a research bridge to nowhere.
K-12 school districts measure learning but aren’t organized around research. Universities are organized around research, but don’t measure learning. Ironically, in determining the efficacy of learning tools, higher education’s research-first agenda is getting in the way of research.
Jacques Barzaghi was once given a tour of a state prison and famously summed up his impression to the press: “We are all prisoners.” His boss’s girlfriend probably agreed; as she sang so beautifully in Desperado “your prison is walking through this world all alone.” You know who else agrees? High-integrity edtech companies. Because when it comes to edtech, colleges are prisoners of their own research-first agendas.