A vendor gives a compelling demonstration. The interface is clean, the analytics dashboard looks authoritative, three schools are named as existing clients, and a slide reports that students using the platform improved by a striking margin. The head of department is enthusiastic. A budget line gets committed for three years.
Almost nothing in that sequence constitutes evidence that the product improves learning. This is not a criticism of the people making these decisions, who are typically doing it between timetabling and staffing crises with no training in research appraisal. It is a description of how most institutional technology purchasing across the region actually works.
UNESCO’s 2023 Global Education Monitoring Report put a number on it. In a survey of teachers and administrators across 17 US states, only 11 percent requested peer-reviewed evidence before adopting education technology. The supply side matches the demand side: a review of the UK market found that just 7 percent of education technology companies had conducted a randomised controlled trial, 12 percent had used third-party certification, and 18 percent had engaged in academic studies at all.
If almost nobody is asking, and most vendors have nothing rigorous to show, the market clears on persuasion rather than proof.
What the Evidence Actually Says
It is worth being precise here, because the argument is frequently caricatured in both directions. The research does not say education technology fails. Systematic reviews conducted over roughly two decades generally find small to medium positive effects on learning outcomes compared with traditional approaches.
Small to medium is a real result. It is also very far from the transformation claimed in most sales material, and it is an average across wildly different products, contexts and implementations. An average of that kind tells you almost nothing about whether a specific tool will help your students, which is the only question your budget actually turns on.
There is a further complication that institutional buyers rarely account for. The same report notes that companies supplying education technology often also advise the decision-makers who purchase it — citing, among other examples, a single provider estimated to hold around 70 percent of the primary education technology market in the Netherlands, and national artificial intelligence curricula in more than one country built on a vendor’s own programme. The equivalent at institutional level is the vendor who provides your teacher training, writes your digital strategy and then quotes for the solution. That arrangement is not necessarily improper, but it means the advice and the product come from the same place.
An Evidence Hierarchy for Institutional Buyers
Not all evidence is the same kind of thing. The table below ranks what a vendor might put in front of you by how much weight it can bear.
| Level | What you are shown | What it actually proves | Who produced it | Main weakness |
|---|---|---|---|---|
| 1 | Testimonials and named client logos | Someone bought it | The vendor, with client permission | Purchase is not outcome; selection is favourable |
| 2 | Usage and engagement data | Students opened it | The vendor’s own platform | Time on task is not learning gain |
| 3 | Vendor case study with before-and-after scores | Scores rose at that school in that period | The vendor | No control group; teachers were probably trying harder too |
| 4 | Independent evaluation with a comparison group | The gain likely came from the product | A university or research body | May not have been conducted in a context like yours |
| 5 | Randomised trial or recognised third-party certification | The effect is attributable and measured against a standard | Independent researchers or a certifying body | Rare; expensive; often on an older product version |
| 6 | Replication in a comparable setting | The effect holds in conditions resembling yours | Multiple independent parties | Rarest of all in this sector |
Two observations follow. First, most of what circulates in EdTech sales sits at levels one to three, and almost all of it is produced by the party with a commercial interest in the result. Second, the jump from level three to level four is where the claim changes character entirely, because that is where somebody has bothered to ask what would have happened anyway.
The absence of level four evidence is not automatically disqualifying. Many worthwhile tools are too new to have it. But a vendor’s response to being asked for it is informative. Providers who acknowledge the gap and explain how they intend to close it are behaving differently from providers who redirect to another testimonial.
The Question Procurement Usually Skips
“Does this work?” is the wrong question. The Global Education Evidence Advisory Panel — convened by the World Bank, UNICEF, USAID and the UK’s FCDO — searched more than 13,000 documents and drew on over 400 rigorous studies to sort education interventions into tiers running from Great Buys down to Bad Buys. What separates the tiers is not whether something works in isolation. It is cost-effectiveness at scale relative to alternatives.
Structured support for teachers and teaching students at their actual learning level rank among the most cost-effective interventions available. Distributing hardware without accompanying pedagogical support has repeatedly landed near the bottom.
Translated to an institution, the question becomes sharper and more uncomfortable: is this the best available use of this money, compared with what else this budget could buy? A platform that produces a modest gain may still be the wrong purchase if the same sum spent on teacher development would produce a larger one.
Why Regional Averages May Not Describe Your Students
Southeast Asia’s EdTech sector has expanded quickly, but research conducted through the ASEAN-UK SAGE programme — delivered by the British Council and the SEAMEO Secretariat with EdTech Hub and the Australian Council for Educational Research, drawing on interviews with providers and funders across eight countries — found that solutions frequently remain inaccessible or limited for marginalised learners even where infrastructure and adoption have grown.
This matters directly for evaluation. A product evaluated on urban students with reliable connectivity, personal devices and strong instruction in the language of the platform may perform very differently in a rural school where several of those conditions do not hold. When you review an evaluation, the sample description is not preamble to skip. It is the part that tells you whether the finding applies to you.
Adoption Scale Is a Different Claim From Learning Impact
There is a category of vendor claim that is genuinely verifiable and genuinely useful, and which buyers often conflate with effectiveness evidence: scale. How many institutions use this. How many learners have been reached. How long the product has operated. Whether a deployment was a first of its kind in a market.
These are countable, dated facts, and unlike learning-outcome claims they can be documented and independently checked. Institutions and education organisations across the region already appear among Asia Record’s listed record holders, where measurable achievements are assessed against submitted evidence before being confirmed and published openly. For a claim about deployment reach or participation scale, that kind of independent documentation is considerably stronger than a vendor’s own slide.
What it does not do — and this is the distinction to hold onto — is establish that the product improved learning. Scale evidence answers whether an organisation can deliver and has delivered. Impact evidence answers whether the delivery achieved anything. A serious provider should be able to speak to both and should not offer the first when asked for the second. A serious buyer should ask for them separately.
Common Mistakes
Treating the pilot as a test. A pilot run by enthusiastic volunteers with vendor support present tells you how the product performs under conditions that will never recur. Pilot with an average teacher, not your best one.
Accepting engagement metrics as outcomes. Login counts and minutes on platform measure compliance. If a vendor leads with engagement data when asked about learning, note what was not answered.
Ignoring the total cost. Licence fees are the visible portion. Teacher time, training days, device refresh, technical support and the cost of the thing you did not buy instead are the rest, and they usually exceed the licence.
Letting the supplier write the requirements. If a specification is drafted around one product’s feature set, the evaluation is decided before it begins.
Not defining success before purchase. Institutions that do not state what improvement they expect, in what measure, by when, cannot later determine whether the investment worked — and almost always renew by default.
A Practical Checklist
- Write down the specific learning problem before looking at any product. If you cannot state it in one sentence, you are shopping rather than solving.
- Ask which level of the hierarchy above the vendor’s strongest evidence occupies, and ask directly whether any independent evaluation exists.
- Read the sample description in any evaluation offered. Check whether those students resemble yours in connectivity, device access and language of instruction.
- Establish who funded and conducted the research, and whether the same organisation supplies your advice, your training and your product.
- Cost the full three-year commitment including teacher time, then name the alternative use of that money you are declining.
- Separate scale claims from impact claims and verify each on its own terms.
- Define your success measure, baseline it before rollout, and set a review date with the authority to discontinue.
Asking Is the Intervention
The most useful finding in the UNESCO research is not about products at all. It is that only around one in ten educators surveyed asked for peer-reviewed evidence before adopting. A market where nobody asks produces vendors who do not need answers.
Schools and universities across Asia are collectively among the largest EdTech buyers in the world, and buyers set standards through what they require. An institution that routinely asks for independent evidence, states what it expects to improve, and reviews honestly against that expectation will make better decisions than one that does not. It will also, over time, help produce a market with better evidence in it.
OFFICIAL ASIA RECORD https://asiarecord.com/
KNOW OFFICIAL ASIA RECORD https://asiarecord.com/about/
OFFICIAL ASIA RECORD Nomination https://asiarecord.com/application/
WHY OFFICIAL ASIA RECORD https://asiarecord.com/faq/
FIND OFFICIAL ASIA RECORD https://asiarecord.com/contact-page/