Certifying EdTech for Educational Efficacy: An FDA for EdTech
Today I want to share some thoughts we've had recently about measuring educational efficacy. The goal of the EdTech industry is to harness technological advances that have sped many other industries forward to improve educational outcomes. This requires collaboration between education practitioners and technology practitioners; a sharing of their separate expertise, in a complex, iterative back and forth.
Proof Needed!As you may know, Ponder is a critical reading tool designed to scaffold the individual critical reading process as well as support a collaborative, social group work experience. Our guiding hypothesis is that we can help students improve these skills with a service that enhances the reading they are doing for class. Ponder scaffolds their critical thinking and facilitates collaboration around those readings. The fact that the service performs those functions doesn't necessarily mean it achieves the intended goals; only a carefully designed scientific study will prove that.
Proof is needed for any educational tool.
We all must prove that the student's learning experience is improved by the addition of tool X into their classroom. Now, agreeing on what "improving the student's learning experience" means exactly or how to measure it is outside the scope of this blog post. However, I think we can all agree that when a school is considering purchasing a new tool, they need to know that it does what it says it does (e.g. makes a positive academic impact). Currently there is too much marketing mumbo-jumbo flying around, and the districts don’t have the expertise or the resources to certify companies' claims. That certification needs to be derived from a test. Considering how many unproven tools there are, we need a good testing and certification process urgently. (On a side note, because it's easier to measure, it seems that logistical concerns around tech integration often trump educational impact when it comes to comparing technologies. Will this tool work with this other tool my school uses? This is not an unreasonable question, but it should not be confused with educational effectiveness.) So, how do we create a good testing and certification process?Testing and Certification There are two big challenges with running studies in K12:
- Minimizing learning disruptions to the current students who are part of the test.
- Maintaining the objectivity of the organizations and individuals involved in administering the study.
First, testing a tool inevitably brings with it the risk of disrupting learning. While to a certain extent that is unavoidable, schools need help managing that risk so that the testing process doesn't do more damage than good. Without testing, no one can find out what works; without a testing facilitator to take responsibility for controlling chaos, schools will rightfully remain reluctant to try new technologies. As a school, I would be skeptical of any company's claims at minimizing disruption in my classrooms. After all, they are likely more concerned about running their study than possibly causing trouble for the students in the study.Second, administering a controlled study requires disinterested, skilled professionals to design the study and then evaluate the results. It requires time from school personnel and may require new infrastructure to support the study. Those requirements cost money. Schools should also be skeptical of a study that was paid for by the company behind the product being tested. Even well-intentioned people running a test evaluating a product they have a financial interest in will have difficulty being objective. Not so well-intentioned people may use money to influence the outcome of the study. To address these two challenges, I think there needs to be an objective third-party whose ultimate success is measured in student performance improvements.We Need an Objective Third-partyLet's call that entity an EdTech "broker". It's a bit like having an FDA for EdTech. The US Food and Drug Administration (FDA) provides an essential objectivity to the success of many food and drug industries and a similar provider is essential to the success of EdTech.
This broker will create a level, merit-driven playing field that focuses corporate education investments on impact rather than marketing.
So the next question is, who covers the costs of the broker? At first blush one might think it should be the companies. After all, the study is on their product, they are potentially wasting the school's time, and they stand to make money if it is proven effective. I don't think that will work. As a society if we have decided to apply technological innovation to improve the free education that is provided to all children in the US, we should invest money in creating the political, technical and managerial infrastructure to facilitate the evaluation of these educational technologies.Some core tenets of such an entity:
- Standard evaluation processes
- Transparency of funding sources
- Shared governance by educational and technological stakeholders
- Politically non-partisan
Some process ideas:
- Low processing fee to filter out spurious test applications but not so high as to make venture backing a pre-requisite
- Guidelines around maximum participation for a given student to minimize disruptions
- Streamlined re-certification process for technology updates
- Non-financial benefits (access to additional infrastructure?) to participating schools to encourage participation
- Publicly available study data for public learnings
- Low-friction certification verification process (and/or policing of certification claims)
- Marketing/distribution opportunities for certified
Some ideas for who should be involved:
- Pedagogical Efficacy Oversight:
- Information, Curation and Community:
- Funding:
- School Districts and Charter networks:
Who or what are we missing? We'd love to hear your thoughts!