I lead on digital evaluation at BEIS (from the former BIS digital side).
We were always ‘alright’ at doing evaluation; we would report back on web analytics or the volume of mentions, number of impressions, number of RTs and include a nice image of the most successful tweet.
But what did this really tell us? Were we doing a good job or having an impact? What actionable insight could we gather? How could we showcase our outcomes in a more meaningful way?
BEIS journey
Over the last 2 years we have changed our approach to evaluation within the Communications Directorate. Evaluation became a developmental priority and everyone in Communications had an evaluation objective. We created an evaluation steering group – pulling together representatives from all the disciplines across Communications to drive up our skills in evaluation.
Through that steering group we carried out an audit of evaluation skills in our different comms specialisms; we all benefited from agreeing at senior levels what ‘good’ evaluation actually involved. The steering group also focused on:
- creating a standardised evaluation template everyone could use
- sharing example evaluation objectives, accompanied by a directorate-wide ambition for everyone to be at Government Communication Service (GCS) practitioner level by the end of 16/17
- regularly reviewing colleagues evaluation work and feeding back about their work
- holding an ‘evaluation month’, a series of sessions with internal/external speakers on all aspects of communications evaluation
There is no doubt that the quality of our evaluation across the directorate as a whole has improved; we are more integrated and colleagues know what’s expected of them. It’s been great to see teams setting their success measures at the beginning of individual projects, and reporting on these consistently.
Digital team journey
But what about the digital team, where I sit? As I said, we were always comfortable at reporting on data, but we needed to do more and turn data analysis into actionable insight. A team restructure meant that we now have an SIO and IO in post looking specifically at evaluation and insight. This, and the push and drive from the steering group actions has really helped improve our evaluation from ‘alright’ to excellent.
I wanted to move us from just the measurement of metrics, to developing a bank of actionable insights that we could use more formatively when developing new projects. We now have benchmark data on all channels and effective online influencer lists (that we use to help shape engagement plans).
I particularly wanted to see if we could track and report on the elusive ‘advocacy’ element of the GCS evaluation framework, and track where and when our advocates were helping us spread the word about our announcements and policies.
The joys of Ripjar
This is where Ripjar came in. We procured Ripjar last year via the Digital Marketplace and it has really helped us. It is a data analysis and visualisation tool that pulls in loads of online data. We can now find our online influencers more quickly and track how they are responding to our communications. We no longer have to manually double check information via Tweetdeck, searching through multiple Twitter feeds or blogs for relevant comments. We can also easily interrogate the data by different date, phrase, sentiment (and more) ranges.
The element of Ripjar I really love is being able to set up multiple ‘actors’ lists to see with ease how different groups of key influencers are participating in online conversations.
The tool is by no means perfect – we sometimes struggle with the sheer volume of data coming in and filtering out non-relevant conversations, but it has really helped our evaluation.
And in the spirit of digital embedding (my other hat!), we’ve been building the capacity of teams across the Communications Directorate to use this tool to analyse their own policy and campaign areas.
I secretly jump for joy when colleagues talk about using actionable insight and can track the sentiment of our influencers – but that’s the evaluation geek in me!
Stay up-to-date by signing up for email alerts from this blog.
Leave a comment