The Untestable Abstractions of School Improvement

There are many claims about what schools should be and how they can be improved. Most are well intentioned, if not always well informed. As a school leader, I find myself swimming in a sea of ‘oughts’, and drowning in intuitively appealing claims and counter-claims. Where is the life jacket?

In Seymour B. Sarason’s seminal text ‘The Culture of the School and the Problem of Change’, he argues for the need, and suggests the method, for determining which of the virtuous claims are worthy of taking seriously, and which are merely untestable abstractions. He describes the untestable abstractions as the ‘should be’ and ‘ought to be’ types of statements that are virtuous, but do not specify any way of determining whether actions based on these statements are consistent with their claims.

Sarason gives a range of examples of such untestable statements:

The classroom should not be a dull and uninteresting place but one which brings out the creativity in children.

The potential of children is not being realized. The classroom should be a place where self-actualization is constantly occurring.

School systems in general, and classrooms in particular, are authoritarian settings. The democratic spirit must become more pervasive.

Schools have become encapsulated settings within our society and unresponsive to it. They must become more open.

Anyone working within the school system will recognise such claims and could list off an almost endless amount of others. They often have an intuitive appeal and sound attractively virtuous – ‘common sense’, even. However, without elaboration these claims are untestable as they do not state clearly the observable consequences – the behaviour, practices, or relationships – which link the intent of the assertion to action and claimed effect. Whilst there are testable elements to the statements above, the language (in italics) is insufficiently precise to provide clarity about the advocated actions or what will result from these actions. The untestable abstraction is often hidden within or tucked behind a perfectly contestable thesis.

It is worth noting at this stage that these claims may indeed be true, but that does not make them helpful. We are unable to prove them to be either true or false and therefore they are not a wise basis on which to take action. Such untestable abstractions can dominate discourse about school improvement (see pretty much any argument on Twitter among teachers, school leaders and other interested parties). They also lay false trails which distract the school system from pursuing actions which we have some hope of showing to be beneficial or otherwise. We might find ourselves slavishly following a virtuous mission, full of self-satisfaction and evangelical belief, but never knowing whether our efforts have ever been worthwhile. What a waste of a career.

How might we distinguish between analysable and unanalysable abstractions? Sarason draws on the work of Sidney Hook (1966) who wrote about this problem in relation to social inquiry in general. Hook observed that all such claims must be expressed as a sentence (or proposition or statement), and for this to be meaningful we must ‘know how to go about testing it, and what would constitute evidence to confirm or refute it’. Although the sentence itself may not contain all the information we need, we must be able to derive from it other statements which themselves are testable.

Every statement, then, which purports to be a true account of what is or isn’t, enables us by the use of certain rules or inference to derive other statements that direct us to do certain things and to make certain observations.

In other words, to be useful, claims about what schools should be, or do, must be sufficiently provable or falsifiable to inform action. If those making the proposition cannot state ‘the conditions or situations in which certain observations can be carried out to test it’ then we are right to ignore their claims and search for a more informed foundation for our labour.

Consider some further examples of ‘oughts’ which may be more familiar to contemporary ears:

Schools should be data-rich.

The school system should address social disadvantage. Every teacher has a responsibility to eliminate gaps between those from disadvantaged backgrounds and their peers.

We know that the best teachers make a significant difference to educational outcomes for the children they teach.  Therefore, teachers should continually strive to improve.

To create a better society, schools must focus relentlessly on improving.

Our gut feeling may be towards agreeing with these statements, and who is to say they are untrue? The point is are they helpful? Sarason’s case is to argue that they are only helpful to us to the extent that they are testable. Perhaps if the proposer were able to clearly define terms like ‘social disadvantage’, ‘disadvantaged background’, ‘best teachers’, ‘better society’, or what they mean by improvement when it comes to a school, we may be closer to specifying the observable behaviours, practices and relationships which tie intent to action.

This problem bedevils the whole discourse around school improvement. There is an ambiguity of language which means we often avoid saying exactly what it is we intend to influence, and what the mechanism is for doing so. Instead of saying ‘we should increase the proportion of our students progressing to study level 3 qualifications at age 16’, we talk about ‘raising aspirations’ and ‘helping students reach their potential’. Indeed, the very notion of school improvement is arguably one huge untestable abstraction.

Those most often charged with the task of delivering school improvement (who we like to call ‘school leaders’ nowadays – a title which itself encapsulates all manner of untestable abstractions) are not helped by the well intentioned oughts. And yet we encourage them to bathe in these waters. Education policy is often predicated on such un-falsifiable claims and dressed up in appealing language and imagery. Demands on schools are designed to appeal to our self-identity as saviours of the needy, disadvantaged and downtrodden. Of course, there is nothing wrong with possessing a strong moral purpose to help those who need it most, but we are made vulnerable to the siren song of policy-makers if we are not careful. But we also do it to ourselves. Flick through the pages of educational literature and you will find a plethora of untestable oughts. Or examine the curriculum and resources for professional qualifications to reveal the virtuous, but un-provable, propositions about the role, duty and attributes of school leaders.

Headteachers must possess a compelling vision for their school.

Educational must transform the lives of every young person.

School leaders should create a culture of success which enables every pupil to succeed.

Leadership is the key to turning around failing schools.

That last one is barely without italic! What exactly do you mean when you say leadership? Are you claiming that this quality alone will be powerful enough to transform the school? What are you turning around from? In what sense was the school ‘failing’? The proposition is at once almost magnetic in its appeal and verging on being entirely meaningless.

Imploring school leaders in this way is a severe distraction from purposeful endeavour. There is such a long distance between the ‘active ingredient’ being advocated and the intended effect, and the meaning is all but lost in ambiguous language and ill-defined mechanisms for delivery. Such statements contain no instruction. No theory of action. They are mere wishes.

Again, I would note that this does not mean these claims are untrue. However, they are most unhelpful in the discourse around school improvement, particularly for those charged with its delivery.

None of this is to say that abstractions and generalisations are unhelpful per se. Indeed, they are the basis for knowledge creation. Expanding our collective human intellectual capital depends on us making bold, general claims about the way things are. This is as true for our understanding of school improvement as it is for any other domain. However, we move forward by testing the validity of these claims. We must continue to assert that X leads to Y, and argue why this is so. Without such a process we cannot hope to take our learning from one time and place and use this knowledge to quicken progress elsewhere. If our claims for how we have succeeded in improving a school cannot be verified, abstracted and tested beyond our context, then nothing has been gained.

What does a testable abstraction look like in relation to school improvement?

(The relevant) school leaders should have a good understanding of employment law to ensure employees are fairly treated.

Taking time to understand complex problems from different perspectives will result in a more nuanced problem definition. The consequent solutions will be more likely to achieve their objectives.

Improving attendance will result in improved performance in formal tests.

Schools should reduce exclusions to create more opportunities to address the needs of pupils who behave anti-socially. This will reduce further disruption.

As with untestable propositions, the above claims are not necessarily true or untrue, but we at least have a chance to prove them so (or find evidence of their correctness, even if proof is not possible). Seeking to confirm or refute their truthfulness will add to our understanding of school improvement. The propositions are abstract and general in that they claim to work across context, but this opens up an opportunity to gain insights into when the statement is true, and when it is false.

In the discourse around school improvement, determining which contentions are testable or untestable is important if we are to make informed decisions and take wise action. By which I mean, school improvement which is predicated on testable claims will be more likely to achieve its aims than that which relies on untestable abstractions.

Of course, I don’t know for certain that the above claim is true. I invite you to test it.