An interesting piece in today’s Telegraph from Tom Chivers highlights the disconnect between popular perception and harsh statistical reality. UK crime rates are demonstrably going down, but the majority of people in the UK believe crime to be on the rise. The UK’s foreign aid budget is a small fraction of overall expenditure on pensions and education, yet many believe it to be larger than both. And yet, even when confronted with the evidence, many find it hard to accept.

As our ability to create, collect and harvest data has grown over the past decade, so we are increasingly confronted with patterns, trends and statistical models that radically undermine our working assumptions. In spite of the increased investment that companies – and governments – are making to understand the data they have at their disposal, disappointingly few concrete changes are being made in practice. In my experience, the biggest challenge for those of us working in the field of business intelligence is human intuition.

Senior executives of large corporations are prone to claiming that they “know” their business is making a profit in a particular sector. When the data backs them up, they feel reassured – and are often keen to continue investing in data analysis. But when the facts conflict with a particular claim to knowledge, it can be very hard for them to accept. When data miners start to be seen as data underminers, it is not uncommon for entire projects to be prematurely cancelled.

At the root of all this is a fundamental problem of philosophy. Read any Socratic dialogue, and you will come across characters who, despite having followed and accepted every step of Socrates’ reasoning will still at the end of it disagree with his conclusion. Why? Because it just doesn’t “feel right”. This tactic continues to be employed in the field of ethics to this day. Our intuitions are powerful governors of the way we think. They help us to function in a world in which we may not have all the data available. And as working assumptions, they are not an entirely bad thing: sometimes they’re all we have, and gut instinct has helped many an entrepreneur establish a successful business. But their utility must always be weighed up against what is actually known. When the data is available, and the patterns are sufficiently clear, we must be brave enough to discard our intuition.

This is why the work of organisations like Full Fact and the RSS is so important. It is only by checking the facts and statistics that underpin the assumptions of leaders and policy makers – in business or in government – that we can effectively challenge intuition and put it to the test. Sometimes our intuitions will be proven correct; but when they’re shown to be as insubstantial as the Emperor’s new clothes, we must not be afraid to pick a different outfit.

After all, assumption may sometimes give birth to good ideas, but it isn’t always the best of mothers.