The School Leader as Plumber

Problems are gritty, grounded, local. We only understand them close up, case by case. We’re tempted by high level solutions when we should be getting down and dirty. The solution may lay in the idiosyncratic and pragmatic, not the intellectual.
Michael Blastland, The Hidden Half (2019)

I’m going to go out on a limb here and say that cognitive science has, and will continue to have, a fairly limited impact on educational standards.

It is not that I reject the findings of this branch of psychology (although many of the studies, we are beginning to realise, may fail to replicate), or that I think we should turn our backs on the discipline. On the contrary: we should glean all we can from this, and other disciplines, which offer something to our understanding of education. My concern is that the implementation of cognitive science – or indeed any research in social psychology – is problematic.

There are many reasons why the application of knowledge from research, or theory, to the classroom is difficult. I will explore just one of these reasons here, which is the problem of the particular.

Access to water

The economist Esther Deflo tells the story of the effort to bring clean water to homes in Tangier in 2007. The firm responsible for this effort spent considerable amounts of money building a large network of pipes and installing taps and toilets in homes across the city. To access this infrastructure, households were offered interest-free loans to fund the cost of connecting their home to the supply. The incentive to take this offer was high (the benefits and convenience of clean, fresh water every day without carrying it from pumps). To many in poorer households in Tangier, the cost was high, but not unaffordable. The rational choice was to take the loan, and yet less than 10% did.

The designers of the programme no doubt felt that they had done everything they could to deliver clean water to thousands of households. The behaviour of those given this opportunity was mystifying; that is until they moved away from their drawing boards and observed what was happening on the ground.

What they found was that, to access the loan, numerous pieces of documentation were required. This meant that people needed to copy original documents and take these to the municipal offices. However, in most communities there was no access to photocopying facilities. When the team looking into this problem visited houses and offered to copy documentation and submit on their behalf, take up increased to 69%. The grit in the mechanism was enough to almost grind it to a halt.

The economist as plumber

Deflo points out the importance of detail in policy delivery, but suggests this detail is often overlooked by the designers of solutions. Her field of expertise is economic development, but the message has wider appeal. She admits that the role of engineer is an important one for economists faced with solving real-world problems. However, there is also a need for economists to be plumbers. Why plumber?

Plumbers try to predict as well as possible what might work in the real world, mindful that tinkering and adjusting will be necessary since our models give us very little theoretical guidance on what (and how) details matter.
Esther Deflo, The Economist as a Plumber (2017)

The engineer will design the pipes but it is the plumber who must fit them. It is the process of fitting that reveals the messiness and diversity of reality. The plumber’s craft is in making elegant designs fit around the existing installation, to solve a particular problem, in a way that meets the expectations of the customer. The plumber’s skill is rooted in compromise and the particular, not ideals and generalisations.

For those charged with solving problems at scale, there is a tendency (a necessity) to simplify both the reason and the solution. It is expedient for there to be a single (or limited number of) root causes, an understandable causal mechanism, and a trigger – an intervention- which will set off a chain reaction to address the problem.

Duflo’s contemporary and collaborator, Abhijit Banerjee, labels the simple solutions that we long for ‘buttons’. These buttons fire up the machine. The machine metaphor is convenient to policy makers because it casts the social system as being reducible to components whose interaction can be understood, therefore whose faults can be fixed. This simplification – this denial of the complex reality – is problematic enough. But more than this, Banerjee argues, policy makers might even assume that the machine runs on its own or does not run at all (it is either functioning or stalled) and the task is therefore to leave it alone or to fire the cylinders. ‘The reason we like buttons so much,’ he says, ‘is that they save us the trouble of stepping into the machine… we avoid having to go looking for where the wheels are getting caught and figuring out what small adjustments it would take to get the machine to run properly.’

An interest in the particular

Details that we… might consider relatively uninteresting are in fact extraordinarily important in determining the final impact of a policy or a regulation.
Michael Blastland, The Hidden Half (2019)

It is not that we don’t need (or benefit from) big ideas, research evidence, policy, or new technologies and methods, just that the way we implement them must pay regard to the detail. The general must be made particular. But this is problematic.

Michael Blastland, quoted above, in his book ‘The Hidden Half’ (2019) goes beyond development economics to explore how a blindness to detail can thwart progress, particularly when tackling problems in social contexts. This is a problem across the social sciences as principles described at a level of abstraction brush up against the nuance of local context. There is a richness and complexity to social systems which is not captured by generalised theories and models. Rather than reject engineered solutions based on assumptions and generalisations, he concurs with Duflo that we should pay more attention to what happens when intention meets reality, and be prepared to tinker!

The solution Blastland offers – to pay more attention to the detail – is attractive. However, it is itself a simplification of the challenge inherent in doing this. Unlike the example of providing water supply to homes in Tangier, we may find that there is not just one overlooked detail which derails our carefully engineered policy, but many. We may also find it impractical, or even impossible, to ascertain what exactly is going wrong.

The devil in the detail

To illustrate my somewhat pessimistic point, let us turn to cognitive psychology, which is the subject of my opening critique.

I use cognitive science as an example not because it is alone in being subject to the detail dilemma but because it is currently ‘quite the thing’. I keep hearing about the various ways research findings are being rolled out in schools. In particular, retrieval practice – the idea that long-term memory will benefit from an intentional effort to recall information, bringing it to the forefront of your mind, thus strengthening neural pathways in the brain – has been seized upon by many. I should state at this point that the ‘evidence’ behind retrieval practice appears to me to be about as robust as you can get in the field of cognitive science. Furthermore, rather than being an inaccessible, abstract concept, one can easily imagine practical applications in the classroom. It is these two features – evidential support and ease of application – that has made retrieval practice a popular policy subject in schools, but also a useful example for illustrating the ‘problem of the particular’. Retrieval practice should work as a method by which to improve teaching and learning – it could be one of our best shots.

I’ve heard of various approaches for rolling out retrieval practice. However, I’m not interested in critiquing the method. My own view is that policies whereby, for example, all teachers are told they must do five retrieval questions at the start of every lesson is at the crass end of the spectrum, whilst coaching which promotes a more nuanced and appropriate adoption of retrieval practice are more palatable and likely to change practice in useful ways. Rather than be distracted by arguments about how best to improve teaching practice, I will instead focus on the peculiarities, idiosyncrasies, and details of the classroom that may act as grit in the wheels. I will leave you to imagine how this grit may disrupt the various policy approaches you have experienced.

So, what are the details which may disrupt? You may wish to skim the following list to reach the conclusion, but I offer it to make the point that classrooms are very gritty places.

Disrupting a regularity. The regularities of behaviour in classrooms happen for a reason and we disrupt them at our peril. Take, for example, the act of taking a register. For some teachers this is a habitualised and effective mechanism for establishing order and signalling the start of the lesson. The replacement of this routine with another – say a retrieval starter – will have an opportunity cost that must be accounted for. New pedagogical approaches should not only be judged by their efficacy, but in comparison to what is foregone.
Practical constraints. If we ask all teachers to line up pupils outside before they enter the lesson we disadvantaged those for whom this is not physically possible. Similarly, classroom layout, the availability of equipment, the peripatetic room changes for some teachers, the number of text books available, and numerous other practical constraints will inhibit delivery and effectiveness.
Competence. It sounds obvious, but it is a factor often overlooked: some teachers are more competent than others and all have aspects of their work which they are better in than others. The support, preparation and rehearsal required will vary across the cohort of teachers. Those who go in ill-prepared to try something new will be most likely to fail and be least likely to embed the practice – therefore those who most need to improve will be those least likely to.
Local priorities. All teachers have pressing problems which, in their minds, are worthy of their attention. For some it is getting class 11A to behave, for others it is meeting their performance management target, impressing the observer, or ensuring a student gets the grade they need to get into their chosen university course. Our problem may not be their problem. By making it their priority, what attention do we displace from their pressing problem?
Relationships. The teacher’s relationship with their class will be a key determining factor in the success of any new practice. Does the teacher feel psychologically safe to take risks?
Timing. Is this the right time to adopt the ‘proven practice’?
Interdependence and group norms. The relationship between colleagues will affect a teacher’s response to pedagogical intervention. Are they competitive or collaborative? What are the group norms around teaching adaptation?
Pupil experience. How does the pupil perceive the change? Have they noticed the sudden flood of retrieval practice? Do they perceive that this change in the regularities of their lessons is imposed upon them and the teacher, or a small adjustment of the like that their teacher makes all the time?
Administrative barriers. What is the turnaround time for ordering mini whiteboards? How long is the queue at the photocopier this morning?
Feedback loops. Teachers receive signals about the effect of their actions continuously. These signals are rarely, if ever, in relation to whether a particular action on their behalf has improved learning. And yet, any change in teaching practice encouraged by a school will (we assume) be with the intent of improving learning. Unfortunately, teachers adapt according to more visible and immediate information: how did the pupils react?; did they get the answers right?; did I run out of time to cover what I wanted to cover? Reinforcing signals will determine future behaviour such as whether new approaches become embedded in a teacher’s practice.
Habits. The adoption of a new habit also often involves the abandonment of another.
Instrumentalising behaviour. Tools may not be used for the purpose we intended them to be used for. Teachers will likely adapt a tool to meet their emerging priorities. If they want to satisfy themselves that their teaching is working, they will instrumentalise assessment to this affect by posing questions that can be answered. If they want to quieten a chatty class, they will insist on silence for retrieval practice exercises. These are not always conscious choices. If you hand someone with food stuck between their teeth a paper clip, they may use it as a toothpick.
The diversity of children. We conveniently call them a ‘class’, but that is about all a group of pupils have in common.
Workload barriers. For the teacher who has all their lessons filed on PowerPoint, ready to roll out, the insistence of the inclusion of retrieval exercises will mean changing every resource. For newer teachers who rely on borrowing resources from others to get through the week, our intervention could mean they work even later into the evening.
Random events. Broken projectors. Wasps. A burp. At early stages of adoption, it does not take much to sour enthusiasm for a new approach.
Sustainability. Doing something a few times is quite different from doing it continuously. Novelty fades. Variety suffers. Other initiatives are rolled out.
Performance anxiety. It doesn’t take much to undermine confidence. Some ill-judged feedback from an observer may be enough. Equally, misjudged praise or misunderstood expectations can lead to a superficial adoption of a method without an understanding of the conditions which may lead to success.

We could look at the above as implementation issues to be accounted for in policy design, but that is a false way to view them. Firstly, it assumes that we know ahead of time what might happen to derail implementation of a pedagogical approach, but we usually don’t. Secondly, we cannot proactively mitigate the variety of obstacles in the path of our teaching intervention. Even if we could anticipate what might happen, we often have no means by which to prevent it. the classroom environment is mischievous and spontaneous.

it is a very long road from ‘it worked somewhere’ to the conclusion you need – ‘it will work here,’ and it is not an easy one to traverse.
Nancy Cartwright (2012)

What might work ‘on average’ may not work here… or here… or here. Dylan Wiliam said that everything works somewhere, but we may equally say that nothing works in most places – at least not in like ways. The evidence we are usually presented with (the effect sizes, the results of randomised controlled trials, the mean effect) tell us little about the conditions under which positive outcomes were achieved. Context and happenstance matter; more than we might like to admit.

Where does this leave us? Well, not without hope or purpose. We can roll our sleeves up and get down and dirty in the machine. If school improvement isn’t messy, you ain’t doing it right. I don’t believe it is possible, or desirable, for those in too senior positions to do too much tinkering. There is a role for engineers and a role for plumbers. However, the engineers need to not be too precious about their creations and give license to the plumbers to make things work on the ground. Practice needs to be improved up close, case by case. If we get too distracted by high level solutions we will miss the important detail that keeps things moving forward on the ground.

Theoretical models are of secondary importance to the particulars of practice. Our emphasis should be on knowledge we can use which means a great deal of translation is required between the research and the classroom. It is possible to marry the general with the particular, but we should pay more attention to the how than the what.

References

Blastland, M. (2019) The Hidden Half, Atlantic Books.

Duflo, E. (2017) The Economist as Plumber, American Economic Review, vol 107(5), pages 1-26.

Banerjee, A. V., and Duflo, E. (2007) The Economic Lives of the Poor, Journal of Economic Perspectives, 21 (1): 141-168.

Cartwright, N. and Hardie, J. (2012) Evidence Based Policy: A Practical Guide to Doing it Better, Oxford, Oxford University Press.