During meetings with RD&E clients, I am often amazed at the level of resistance to the use of performance measuring, particularly in these times of increased accountability.  Generally this resistance occurs at the coalface – however in some cases it remains at senior levels in organisations.

So it was refreshing to work with a client group strongly aligned to a culture of reporting progress against organisational KPIs.  However, there still remained pockets of resistance to measuring progress in meeting project and program objectives.

How did we tackle the discussion – and what is the plan to bring everyone on board in implementing a culture of MER?

Creating MER champions

Firstly, senior management are strongly aligned to introducing a culture of change and accountability.  So the strategy for introducing a MER process in the organisation was determined in a roundtable setting allowing us to establish a common process aligning with the organisational change process.

This created a group of people who were prepared to champion the introduction of MER into the wider organisation and who personally see real value in its operation.

Determining the benefit and value of MER

Secondly, we determined that MER is a key element of measuring the success of their change process.  We also determined that MER was central to annually measuring progress over the 5-years of their strategic plan and integral to how they will be able to report on the effectiveness of internal projects and programs.  This will allow them to report to key stakeholders at a ‘whole of organisation level’.

Developing organisational case studies

Thirdly, we decided to work on three case study projects or programs that already have demonstrated need for MER (that have a budget) and whose key personnel have a natural inclination and desire to implement MER.

It was determined that a formal MER project in the organisation ought to be created and that a willing person can act as an identified ‘project manager’. Not only does this give MER ‘status’ in the organisation, it also provides an identity for a real person to take carriage of this process.

Using external help

Fourthly, the cultural change will be managed through an external process of internal push from management and external push, using ourselves as outside facilitators and supporters of this cultural change.

What else can be useful?

In addition to the above several other important issues:

• Training – rather than simply training people in ‘how to DO MER’, it was decided to embed any training into support of the ‘cultural change’ process that is already occurring
• Simplicity and ease – I re-iterated our philosophy that all MER must be made simple and easy, and that user personnel have to take part in determining the type and processes of data gathering that they will personally be using.
• Ownership – we always recommend that a strong sense of personal ownership is created in the personnel who will be undertaking MER, as well as a realisation of their own personal ‘value proposition’ in using MER that creates personal benefit.
• Using a form of project logic – as a way of ensuring that those using MER create the correct context in which the MER will be used.  This includes developing clear project / program objectives as well as identifying the needs of key stakeholders, so we can measure change in KASA and engagement against the objectives that were set for the onground work.
• Regular checking in – so that those using MER have access to continued mentoring so that when times get busy, or complex, the use of MER remains ‘on the radar’.  This process includes internal or external reporting back to field personnel so they are briefed on their results of MER so they see the ‘end product’ for themselves and can access their results to support their reporting back to their own stakeholders.

I am finding that the jump from understanding M&E Information needs to allocating data collection methods is a bigger chasm than I thought.  After training workshops, participants who become reasonably proficient at developing a log frame with M&E indicators and identifying information for collection – have baulked at the process of allocating practical data collection methods.

This is the jump from theory to practice – from paperwork to practicality.  If you think that you need a survey to benchmark capacity and practice levels pre and post within a project or activity, then it starts to raise a whole lot of issues in your mind:

• What questions would we need to ask?  Who would design them?  When we actually carry it out?
• How would we select the sample? How many do we need to make it “significant”? How would we get the contact details?  Aren’t they already surveyed out?
• Do we do a phone survey, mail survey, web survey, face to face survey?
• What would the cost be?  Do we have the budget?  Is it worth spending the money on?
• Who would actually do it?  Project staff or contractors?
• What about the analysis and reporting?  Does that mean another job?  Is it really worth the effort?
• How will we use this information?  What difference would it make if we didn’t do it?

So a simple logical activity of allocating a particular data collection method to an information need in an M&E log frame is sometimes a very loaded step to take!  This could be the same for other data collection tools.

It looks like we need a simple process to demystify the selection of these tools so the implications can be explicitly considered as the planning is taking place with the log frame.

Let’s consider the survey and how to quickly address likely questions in relation to a capacity and practice change benchmark survey:

More information on sampling: *http://www.ruralsurveyspecialists.com.au/costs/how-many-should-i-survey

More information on costs: **http://www.ruralsurveyspecialists.com.au/costs/survey-cost-calculator

This same type of table to help with choosing other data collection methods can be developed and used in the log frame planning.  Keep and eye out for this type of information in future newsletters.

When you’re developing a social media program, make sure you know what you want to get out of each tool. For example, if you are going to get into Twitter – make sure you understand why this tool is useful and how you can use it effectively to engage your audiences.

So there are some questions you need to answer (I’m using Twitter as an example here but you could easily insert blog/facebook/flickr)

• How are you using Twitter – purely as a listening post to see what the conversations are around subjects you are interested in, or as a way to start your own conversations?
• Why are you using Twitter to engage your audiences?
• Are you trying to increase awareness?
• Are you trying to increase knowledge?
• Are you trying to change behaviour?
• Do you want people to be advocates for you?

When you answer these – you’ll be well on the way to understanding what insights you should be looking for to inform your decisions. You can then develop a dashboard of metrics looking at your specific measures of success.

Thinking back to Twitter then, apart from the basic quantitative measures you collect anyway (e.g. number of followers, number of clicks on your links, number of retweets, number of mentions) it’s the qualitative data where the exciting information is:

• What people are talking about in relation to your program/project/industry – have you influenced the conversation?
• What influential Twitter users are saying about you? (Look at www.traackr.com for a bit more of an overview of Twitter influentials)
• Are people saying that they have changed what they are doing as a result of what you have been doing?
• Are the conversations becoming more informed?
• Are people recommending you to others?

Once you start monitoring your Twitter traffic and start to answer these types of questions you then have a benchmark from which you can start to look at measuring your social media return on investment.

Today’s food and fibre industries continue to change markedly. You will notice in the trends outlined below that the changes tend to be structural, less about food production and more about consumers.

• Changes in types of food preferred by a range of consumers – many are value conscious, while others are quality conscious, including a focus on organic and low chemical inputs
• Changes in how we eat and where we eat – for many people fast foods, convenience foods and social dining are important parts of our lives.  This includes for many of us that ‘special eating experience’
• Access to our foods and shopping – for many the supermarkets remain a mainstay of shopping; for others delis and farmers markets are featuring more strongly.  Some are even seeking ‘quality foods delivered to the door’
• Niche markets – to accommodate these changes, many farmers are seeking to supply into higher value niche markets where they can attract a premium price.  However they recognise that quality is king so they realise that quality management through tracking of their product is crucial to their success
• Collaboration amongst farmers – many of us are familiar with grower groups doing regional R&D trials. We are less familiar with the desire of many to integrate and collaborate with fellow like-minded farmers.

Another growing trend is major supermarkets reducing their interest in ‘branded products’ and focusing more on ‘generic products’.  This means that producers supplying the branded products need to look elsewhere to niche markets if they wish to maintain those brands. An opportunity is perhaps meeting the needs of the hospitality industry where the biggest issues are accessibility of quality food and reliability of supply.

I have interviewed a number of innovative and focused farmers recently who are very in tune with all these trends and seeking to use them to their advantage.

Switched on farmers want to see ROI

Switched on farmers are increasingly looking at return on investment (ROI) and looking to alternative ways of producing their product.  They are focused on the mantra of quality management which is ‘you can only manage what you can measure’.

The concepts and terminology of KPIs (Key Performance Indicators), SMART outcomes and ROI are no longer the province of researchers or administrative managers, they are increasingly the language of mainstream innovative farmers and their value chain. This is particularly the case if they are quality or value conscious and producing to defined markets with clear specifications.

Everyone is speaking the same language

While monitoring and evaluation was once ‘the province of the few’, today we are all being asked to justify our ROI and demonstrate being able to meet agreed KPIs with objective data.

Just like the switched on farmers, program/project/organisational leaders are being increasingly proactive ensuring there are answers to questions such as ‘to what extent have you achieved what you committed to achieve?’ or ‘show me some evidence of your level of success over time’.

As monitoring and evaluation professionals we are really starting to see the added benefits of different stakeholders (from farmers to program leaders) talking the same language and the common ground between us all just continues to increase.

Amy Samson

If you don’t think that rural Australia is interested in Twitter, you may need to think again. One Sunday, last August (2011), ABC’s Landline (@abclandline) became the center of a Twitter storm. Viewers of a story on social media (#SM) and the growing agvocacy (#agvocacy) movement in the United States were invited to tweet with American cattleman Troy Hadrick, (@TroyHadrick). The number of people who joined the conversation meant that #abclandline became one of the most talked about topics in Australia that afternoon.

“Social media is an awesome thing for country people it empowers and connects.” (South Australian lamb producer Richard Gunner @Richard_Gunner – 2,884 followers)

While still a relatively small section of the rural community, Twitter use is growing and providing a communication space connecting all different types. For example #AgChatOz (@AgChatOz – every Tuesday between 8-10pm) is one of the most well attended agricultural conversations online including journalists, farming foundations, winery owners and politicians (on occasions).

Co founder Tom Whitty said that social media is allowing for myths to be dispersed about the reality of farming, as well as giving farmers and rural businesses a voice and providing invaluable networking opportunities for continuous two-way communication.

The clear message is that social media and the Twittersphere has the potential to and is already connecting rural people beyond their community like never before.

At the project/program level Twitter can be a useful way to distribute information very quickly, create a sense of community and allow stories to be shared.

But you have to be careful because an ill-timed or ill-conceived tweet can quickly become a #fail and Tweeps can be very unforgiving.

Think the Qantas Twitter fiasco late last year where the PR team ran a Twitter competition inviting people to describe their “dream luxury holiday” close on the heels of the airline being grounded after breakdowns in union negotiations. Epic fail Qantas. The lesson is to make sure you have a clear social media policy – ‘to tweet or not to tweet – that is the question!’

Almost every evaluation I have been involved with finds that people like to talk and learn from other people in similar situations. It’s why farm walks and field days are so popular as extension tools. But what also comes up is that it is hard for farmers to travel at certain times and time away from the farm is often hard to manage. Perhaps getting on Twitter at a particular time in the week/fortnight/month could be one way to keep the group dynamic alive.

I’ve put some ideas together that you could consider when looking at using Twitter for a project / program

• Create a hashtag (e.g. #projectname) where everyone can contribute to and follow a conversation.
• Make a regular time when group members can hop online and have a conversation about a set topic (shared in advance) e.g. pest control, best practice
• Live tweet from a workshop/field day including photos and information about what is happening, who is presenting etc (don’t forget to let people know ahead of time what hashtag you will use so it can be followed)
• Encourage project stakeholders/members to tweet photos of new practices being implemented and what has changed on farm
• Join other online conversations (e.g. #agchatoz)
• Follow people / companies who are of interest and engage with people online – tweet and retweet and respond if something grabs your attention
• Be a source of information, be personable, be transparent and share your story (project’s story) – people do want to hear!

Well known Australian dairy blogger The Milk Maid Marion says that the rise of a new, grass-roots media (Twitter, blogs and Facebook) means we can tell our own stories. And what has amazed me is just how many ordinary Australians want to hear them.

The basis of all extension programs is engaging with people.  However the term ‘engagement’ means different things to different people – and there are different types of engagement for different purposes.  So if one of your Key Evaluation Questions is “how well did the project engage with their target groups” how do you answer this?

Engagement has a number of elements that should be considered.  There are different modes of engagement and each mode has different characteristics to consider.  The table below considers a sample of extension approaches, their engagement characteristics and potential evaluation indicators (click table image to see a larger version).

There are, of course, many other modes of engagement such as: discussion/learning groups; webinars; steering committees; working groups; forums; and conferences.  These could also be included in a table with their characteristics described as shown in the sample.

Given these different characteristics, the better evaluation question might be “how appropriately and effectively did the project engage with its target group – given the project objectives?”

So, an evaluation of engagement within a project or a program could be summarised something like this (click table image to see a larger version):

And of course, a follow up question could be “is the engagement process on track and working?”.  If not (because of evident gaps in the above table), then the question would be “Why not?” and then “What could be done differently to appropriately and effectively engage the target group/stakeholders as needed?”

One of the crucial responsibilities of a CEO or manager is to be accountable for money spent and work undertaken. It stands to reason that if clear organisational, program or project objectives aren’t in place – progress can’t be measured over time.

In the same way, there needs to be a middle point where progress can be checked on and stakeholders can see progress against objectives.  This is important so that key stakeholders, in particular funders, have confidence that everything is on track to achieve agreed outcomes.

User friendly Key Performance Indicators (KPIs) help. KPIs are simply a very clear reflection of the burning issues that people have committed to resolve through their work activities. Analysing captured project/program data against KPIs as you go, means that a picture of progress emerges that can be useful at all levels of the organisation.

This means that it is important that everyone in the organisation/program gets an evaluation report.

Last week I was reporting the results of a longitudinal evaluation project to 20 or so team members of an organisation.  Every six months an impact report is put on the table and the results are workshopped by the team.

The team has the opportunity to see the results for themselves and read what others will read.  For some it is still a shock to see negative comments – however combined with explanatory notes of ‘why things happened’ (the context level of our QualDATA logframe) understanding the good news and bad can be very helpful.  It is important also to recognise good work – and team members see it for themselves in the results.

For CEO’s and team leaders, a report such as this means that there are ‘no surprises’.  Team members are fully engaged in understanding the implications of the report and senior personnel have the opportunity to resolve outstanding issues of concern and any irregularities from a continuous improvement perspective.  Together senior management and the team can make these corrections in light of full knowledge of what is going on – rather than based on sometimes erroneous assumptions.

By far the most beneficial element of such a process is when one team member asks another ‘how come your ratings are so high?’ In several cases this has lead to very useful roundtable discussions on how to resolve both strategic and operational issues.

Equally when key stakeholders are engaged in similar processes, they too come to understand what is happening and why.  At the very least, some form of KPI focussed impact report contributes significantly to stakeholder confidence in the team. Particularly in terms of outcomes being delivered on time and according to agreements.

Effective leaders therefore must have – and can have – the right information about the right issues at the right time – delivered in a simple understandable impact based reporting format. An effective MER process makes this possible.

How do you value the pride a family may feel in having improved their land condition over time? Or the incredible relief they feel when an enterprise change means that they can relax knowing they meet their bills and put the children through school?  Or the improvements to family relationships when they finally work through succession arrangements?  Or the increased confidence one may have with better communication skills?

We talk about the triple bottom line in Evaluation – Economic, Environmental Social.  Benefit Cost Analysis helps us with the economic; Environmental Assessments help us with environmental – but, at least in rural development, we are not terribly good in social impact assessment – especially the more “intangibles”.

In a project for the Cooperative Venture in Human Capacity Building I was involved in a study on Evaluating Empowerment: the human element of capacity building (Kate Roberts and Jeff Coutts).  This publication is still available for a small cost on the RIRDC website. It considered indicators and tools to capture the level and gains in such “social intangible” elements as:

• Critical thinking
• Conceptual knowledge
• Planning
• Communication
• Effectiveness/ “sense of efficacy”
• Social intelligence
• Facilitation
• Communication skills
• Conflict resolution/ negotiation
• Problem solving
• Community cooperation/ networks
• Leadership
• Access to institutions (political, legal, economic, social)

Our study – and subsequent report and workshops – showed that comparative improvements in these elements could be measured, captured and reported as part of the evaluation process.

A story in the UK newspaper, The Guardian, called SROI: How can you measure the unmeasurable reported on a study undertaken using Social Return on Investment (SROI) methodology to determine the impact and social value of specialist women’s organisations.

SROI is described as a form of cost-benefit analysis that seeks to measure and value the key changes, or outcomes, created by a programme or activity. The study used a combination of “Impact mapping” (using diagrams to show the cause and effect relationships and resulting benefits), qualitative description and direct quotes; and economic assessments.

The economic assessments used a standard Benefit/Cost analysis approach (for example, looking at the financial savings on providing women refuge places) where calculations were made using a range of financial proxies for the different outcomes which were then divided across specific outcomes for clients and multiplied by attribution levels over a five year period.

A similar approach could be used for social benefits in rural development programs.  The table below shows some of the thinking around evaluating these intangibles using both qualitative and quantitative measures.

The main message is that “if we can name it we can measure it”.  There are tools and approaches that can be used to capture the rich benefits that often result from a well-designed program targeting these social intangibles.

It doesn’t matter what program or project you may be running, if you are dealing with stakeholders (internal/external) there will be a communication element to your strategy.

More and more, social media is something that you need to think about – not only in terms of getting your message out there, but also understanding what impact you are having on your audience.  It can also be a pretty useful research tool to monitor chatter that may already be happening around issues/activities of interest.

It’s a jungle out there, and new programs seem to be popping up every other day! To give you an idea of how much each site is being used, each month Socialmedianews.com.au ranks the top 15 most used social media sites in Australia. In January 2012 the top of the pops was…..drumroll……

1. Facebook – 10,703,340 users
2. YouTube – 11,000,000 UAVs (Unique Australian Views)
3. Blogspot – 3,500,000
4. LinkedIn – 1,800,000
5. Twitter – 1,800,000
6. WordPress – 1,500,000
7. Google Plus – 1,200,000 users (Estimation* based on 90 million users globally)
8. Tumblr – 1,000,000
9. Flickr – 830,000
10. MySpace – 470,000
11. Instagram – 350,000
12. Reddit – 170,000
13. StumbleUpon – 140,000
14. Digg – 70,000
15. Delicious – 39,000

Over the next few months I’m going to pick a few of these sites (and maybe some others) and take a look at how they can be used for communication as well as evaluation and research purposes. I’ll also go through some ideas on how to measure the impacts of using these sites, particularly in terms of feeding back into program/project outcomes.

In March – get ready to Tweet Tweeps!

Golden Rule 1 – Know what you are evaluating!

Projects and programs are full of high level outcomes (“improve the social sustainability of rural Queensland”) or very activity-based outputs (“run five workshops”).  You need to be very clear about what the project/program is meant (and is able) to achieve.  A good way to do this is by developing an impact statement – as if the program has finished and you are succinctly reporting the successful outcome. These articles are helpful in defining these achievable outcomes:

Focus your evaluation with KRAs

QualDATA Factsheet – KRA table

Golden Rule 2 – Do a Log Frame

Log Frames are short for Logical Frameworks – and they show how you are going to use your resources to arrive at your achievable outcomes (or Key Result Areas).  Log Frames can be done even if your project has already started (it’s a great team building exercise) and helps ensure that all are on the same page.  Doing a Log Frame at any stage helps you check to see if you are on track and whether changes are needed – either in the planned outcomes themselves or in the activities you are undertaking to achieve them.

More information on log frames – and in particular the QualDATA log frame is in the article: Launching a new QualDATA Log Frame

Golden Rule 3 – Build your evaluation activities into your overall project activities

It’s good to have a Log Frame and determine what M&E data to capture, but if it just stays in a document as a good idea – it will never happen – or at least not happen systematically or effectively.  Taking the time to transfer the data collection activities from the Log Frame and allocating timing and responsibilities to them in an action table means that they are built into people’s work programs and have a lot more chance of happening.

This article has a free download of what a MER action activity table could look like: Evaluation action activity table – taking the next step

Golden Rule 4 – Plan how you will capture, collate and present the M&E data collected

M&E data gets lost on people’s computers or in piles of paper feedback sheets. It often lacks consistency in terms of the type of demographic and impact data collected and so is difficult to collate.  It also sometimes lacks a direct link to the achievable outcomes that you have promised to deliver on!  So taking time to plan and having systems in place to ensure that M&E data is collected, collated and in a format that helps reporting – is time well spent!

This article provides more information on this step and takes you to a simple on-line M&E data system for collating and reporting on individual activities.  We are working with a number of projects to tailor such systems to meet their total M&E data collection needs – and ensure that it meets reporting needs (contact us for more information).

Golden Rule 5 – Report clearly against your promised achievements

The Log Frame provides a great framework for reporting.  It clearly articulates what it is that you can deliver on during the life of the program and how you went about doing it.  The data collected systematically above provides the basis for reporting against the Log Frame – in terms of activities, outputs, process, progress, achievements and outcomes. We have found that working with project teams to provide a “mock up” of what the final M&E report might look like at the start – certainly helps to focus data collection and analysis.

This article could be useful in terms of reporting on activities: Six steps to dismissing the myth that reporting on activities is too hard

If you would like to see some examples of how you can report effectively for impact against your log frame, please contact us.

Jeff Coutts 07 4630 1297