There have always been doubts about the success of most attempts at major change. Over time, these misgivings have become more vocal, and the reasons for the frequent lack of success have become better understood. Complexity science offers reasons enough to explain failure. But there is a further reason that I have never seen discussed in quite the manner below.
Attempts to move from an old organisation design or culture to a new and better one are mostly rational and formal in concept and execution. During this process the organisation’s informal shadow system remains constant – if recognised at all – and this presents an obstacle to change. Instances where the intervention includes the shadow system, and where the change agents have an active presence there, are rare.
The problem is that although the organisation’s surface appearance is affected (probably structural changes, new reward systems, new performance management systems, new competency frameworks, new training programmes, etc.) the dark underbelly of the organisation hasn’t changed. Most of the people are still there, with all their comfortable habits, ambitions, rivalries, power struggles, jealousies, etc.) and the organisation still has its in-groups and out-groups, turf disputes, bullying, politics, etc. Even if their existence is acknowledged by the planners and consultants, these shadow elements are much harder to shift.
A parallel exists with adopting children from a troubled background, maybe where they have been in care in the meantime. The contrast between the old family and the new is dramatic and a wonderful improvement. The new parents may assume (like planned culture change) that the new family will ‘cure’ the child of dysfunctional behaviour. The ‘old’ will become history. But they are often proven wrong. The child can be thought of rather like the organisation’s shadow system. The child has a very different worldview of what it means to be in a family, and he/she will carry this worldview (and hence the behaviour that stems from it) into the new family.
A few writers have an understanding of this problem, most notably Patricia Shaw. But messing with the organisation’s inherent messiness isn’t easy, and may not be permissible. Those who hold the purse strings in the organisation can get rather anxious about paying OD consultants to roam in the shadow system and to listen to what people have to say about the official system.