It’s been over ten years since I coded my first ever ETL routine. Since then I’ve collected together a number of important lessons and best practices, presented here as my very own ‘Ten Commandments’ of ETL. 1. Know your data Whether you’re defining an ETL strategy, designing a set of data flows or writing the code, the single […]
Those clever folks over at Nate Silver’s FiveThirtyEight have been analysing the polling figures for the forthcoming UK General Election. More than any in the UK’s electoral history, this is set to be the most unpredictable ever, with a first-past-the-post electoral system unfit for a multi-party democracy likely to throw up all kinds of unexpected outcomes in […]
After you’ve worked with data for a while, there is a temptation to think about it purely in terms of the conventional way in which it is stored: rows and columns. That is, after all, how it appears on screen whenever you run a simple query; it’s very often how data is displayed in a report, […]
To start with, this book was nothing like what I expected – but that proved to be no bad thing, far from it. Dr. Devlin cuts through all the hype about big data, the cloud, data science and the rest to present his thesis on what really matters when it comes to information management – […]
“We need the new system to produce 300 reports.” “That’s a lot of reports. Why so many?” “Because that’s the number of reports we have now.” Almost every BI professional will, at some point in their career, have had a conversation with a client that goes something like that. In most business transformation projects, a common expectation is […]
This evening I attended a seminar hosted by the University of Toronto entitled “Material Evidence: is a picture really worth a thousand words?” The panel comprised the Director of the Global City Indicators Facility (GCIF) Prof. Patricia McCarney; media artist Ben Rubin; and the visual op-ed columnist for the New York Times Charles M. Blow. […]
For those who never have, the idea of hiring consultants may seem like an easy way to blitz a departmental budget with relatively little to show for it in return. This is particularly acute when you may already have employees who could do the work without any of the additional overhead. As a seasoned consultant, […]
A new colleague joined my project this week. He has never worked with data architects before and was curious about what, exactly, we do. He knew about the end result: typically an analytical report of some kind, measuring key performance indicators or providing answers to common business questions. “Don’t you just need to write a […]
An interesting piece in today’s Telegraph from Tom Chivers highlights the disconnect between popular perception and harsh statistical reality. UK crime rates are demonstrably going down, but the majority of people in the UK believe crime to be on the rise. The UK’s foreign aid budget is a small fraction of overall expenditure on pensions and education, […]
I’ve just come across this article, which claims that the combination of social media and big data will be responsible for who wins the US Presidential election in 2016. It’s a bold assertion, and it comes on the back of many other opinion pieces that vaunt the major role of big data in the 2012 election. […]
Busy Intelligence 