Failing to learn

I’ve just finished reading Matthew Syed’s ‘Black Box Thinking’, which references Tim Harford’s book ‘Adapt’, both great reads on the same theme; failure. If you haven’t read either then Tim Harford’s Ted talk on the subject (https://www.ted.com/talks/tim_harford#t-1067823) gives you a sense of the territory.

Syed’s approach is to examine the culture which has developed in aviation whereby failure (in the form of accidents and near misses) are systematically and openly picked apart and the lessons learnt used to inform improvements to safety. Over time this has resulted in an incredible safety record for the commercial airline industry. Syed calls this approach an ‘open loop system’, meaning that failure leads to progress as feedback is rationally acted upon. This, he argues, is radically different to the ‘closed loop system’ found in healthcare whereby mistakes are hidden, go unrecorded and subsequently do not lead to improvement.

I was so taken with the ideas in the book that I jotted down some key points which I felt were applicable to an educational context and added a few aide memoirs for later reference.

1. Failure is an opportunity to learn. What are our failures in education? My first thoughts were lessons that go badly and exam results that weren’t what they should be. And then what about when a student’s behaviour means they must be removed from a lesson? Then I started to think about policy failures (I’ve seen quite a few if these over the years!). How do we respond to these failures? In my experience we do one or more of the following…blame someone, moan, brush it under the carpet or quickly move on to another ‘good’ idea. How could we learn to embrace these failures as opportunities to learn?

2. People need to feel ‘safe’ in admitting their mistakes. I fear we are some way from this being the case in schools. If we have a disastrous lesson, do we tell others? If we’re lucky enough to work in a department in which there is this level if trust, what about your senior management team – would you tell them? And what is the response when things do go wrong? If your class achieves poor results what is the reaction? Will it be ‘okay, let’s see what we can learn from this’? Furthermore, how do senior management respond when a policy is clearly not having the desired effect? Do they hold their hands up and admit their mistake, and do we contribute to a culture where they wouldn’t dare show this ‘weakness’?

Syed illustrates the desired culture through the example of a system adopted by Toyota in their manufacturing. Production workers would pull a chord when something went wrong. The result was that a cluster of senior managers would almost immediately appear to help resolve the problem. This system only worked once the workers trusted that the mistake could be openly admitted and the senior managers were focused on finding a solution, not looking to blame. What would it look like in schools if teachers could ‘pull a chord’? Could we envisage a culture whereby teachers could immediately call in support to help get a lesson back on track without fear of blame or a label of weakness?

3. Feedback is essential. Syed uses a golfing analogy whereby a golfer looks at the accuracy of each ball played and uses this feedback to subtly adjust his swing, gradually becoming a better player over time. Without the benefit of feedback, Syed argues, we are like golfers playing in the dark. Experience only leads to improvement when feedback is immediate, clear and acted upon. In schools, we are happy to apply this belief to students but are we as prepared to gain feedback on our own error? Syed talks about ‘error signals’ which are indicators that things are going wrong. What are the error signals in schools and which sources of feedback are most instructive? This section got me thinking a great deal about low-stakes testing as a source of valuable feedback to the teacher on their teaching – designed well, tests are choc full of error signals! Without regular data feedback on our impact as teachers are we just ‘playing golf in the dark’?

4. Interpreting data is not easy. Syed cites the work of Abraham Wald during the Second World War. Wald became involved in trying to improve survival rates for fighter pilots. To do this, the returning planes were examined to establish where the bullets hit and therefore where additional armour was needed. This data showed that the tail and cockpit were rarely hit and therefore no additional armour was required in these areas. Wald pointed out that the data was flawed. Rather than the planes which had returned being examined, the useful data was on the planes which had not returned. Wald inferred that the absence of bullet holes on the tail and cockpit were the reason for these planes returning, and that those which had been brought down had been hit in these areas. The conclusion was that the additional armour should in fact be placed exactly where the bullets holes did not appear. This counter intuitive reasoning drastically improved the survival rates of British pilots. Syed warns us to take account of all the available data but also the data that is not immediately available.

5. Beware theories that cannot be refuted. Syed gives the example from the field of Psychology of Adler’s ‘inferiority complex’. This theory suggests that human behaviour is driven by the desire to overcome one’s own feelings of inferiority. However, opposite behaviours might equally be explained by the same theory. For example, failing to defend a friend against criticism may be cited as evidence that you have overcome the social pressure to put loyalty above reason, whereas leaping to the defence of your friend may be cited as evidence that you have overcome your fear of confrontation. This idea is linked to the problem of confirmation bias which is the tendency to seek out information, and interpret it in such a way as to confirm your pre-existing beliefs. To avoid these traps Syed advocates seeking to falsify your beliefs, not to confirm them. We should ask ‘when does this NOT hold true?’. We hold true many theories about education and rarely seek out evidence that they do not hold true, or do not always apply. How healthy it would be to question when some of our cherished approaches might not actually be effective.

6. Trial and error as an approach can harness the power of failure. It is at this point in the book that Syed references Tim Harford’s book, ‘Adapt’, which in my view gives a more authoritative and compelling account of the power of trial and error. Harford, as he explains in the above Ted talk, makes a powerful case against top-down design in complex systems, arguing instead for a more evolutionary trial and error approach. Both Syed and Harford give the example of the development of a particular nozzle needed in the production of washing powder. Unilever had employed their best mathematicians to design a nozzle that would create the fine powder required. Their designs failed. In desperation they turned to a team of Biologists. They had no idea how to design a solution to the problem, but they did have an approach to finding a solution. The team took ten random designs and tested them. The most effective nozzle was selected and ten slightly varied versions produced. The process was repeated and after 45 generations and 449 failures the final highly effective nozzle was produced. Harford uses the term ‘God complex’ to refer to our tendency to believe we know how the world works when often we don’t. These ideas resonate strongly with me. I believe that school leaders are frequently guilty of possessing a God complex and attempting to design and impose systems and policies in a complex context with no evidence that these will be effective. Senior leaders are full of ‘good ideas’ but their whimsical policy making can be damaging and undermine confidence. It is possible to run schools in a different way. Let’s take the example of a school’s behaviour management approach. These are, in my experience, designed and redesigned with little evidence of what works. Why? It is surely possible to evolve a more effective system through careful trial and error. This would need to be a disciplined approach with clear criteria set for evaluating success. To be completely sure a Randomised Controlled Trial (RCT) could be set up with a control group as a point of comparison. Alternatively, Unilever’s approach of trying out various methods and evolving variants on the most effective iteration could be employed. Most of the methods tried would fail, or at least not be as effective as they could be, but the end result would be an approach which has evolved through adaptation and selection.

There is a lot to think about here. I think the main message is that improvement will only happen when failure becomes an opportunity to learn. To achieve this you need:

– to detect mistakes and feel safe to expose them

– to avoid confirmation bias

– to have testable/falsifiable theories

– to look to falsify, not just confirm

– to have immediate, clear and analysable feedback data

– to analyse with care – it is difficult

– to avoid the arrogance of top-down design when faced with complexity and instead develop trial and error approaches that build on evidence

Applied to a school context it could look like this:

1. Get stuck in when mistakes happen – don’t blame unless there is clear evidence of negligence – treat a mistake as a mistake. Apply this thinking to poor exam results, bad lessons, students not making progress, failing systems.
2. Bring expertise to bear on the problem. Don’t leave people alone to solve it. Allow teachers to ‘pull a chord’ when things go wrong and ensure they don’t feel ashamed to do so.
3. Share the lessons learnt. Celebrate failure as an opportunity to learn. Make CPD about failure as well as success. Share worst practice.
4. Generate opportunities for feedback – turn the lights on! Frequent low-stakes testing to get feedback in your teaching. Peer observation. Coaching. Structured reflection. Student and parent feedback. SLT open to and seeking criticism. Be sensitive to error signals.
5. Openly admit when things aren’t going well. Share evidence of impact openly. School leaders to show that change is evidence based. Look for a solution, not a culprit.
6. Adopt a systematic approach to trial and error. Accept that failure must come before success. Adapt what works. Avoid God complex by resisting the urge to design the perfect system. Apply this approach to teaching, behaviour management, improving attendance, increasing participation in sport, reporting to parents… Any aspect of the school which needs improvement.
7. Identify where RCTs might be must effective and practical e.g. Reward systems, specific teaching techniques, marking approaches.

Okay, time to start failing.