Golden Rule 1 – Know what you are evaluating!

Projects and programs are full of high level outcomes (“improve the social sustainability of rural Queensland”) or very activity-based outputs (“run five workshops”).  You need to be very clear about what the project/program is meant (and is able) to achieve.  A good way to do this is by developing an impact statement – as if the program has finished and you are succinctly reporting the successful outcome. These articles are helpful in defining these achievable outcomes:

Focus your evaluation with KRAs

QualDATA Factsheet – KRA table

Golden Rule 2 – Do a Log Frame

Log Frames are short for Logical Frameworks – and they show how you are going to use your resources to arrive at your achievable outcomes (or Key Result Areas).  Log Frames can be done even if your project has already started (it’s a great team building exercise) and helps ensure that all are on the same page.  Doing a Log Frame at any stage helps you check to see if you are on track and whether changes are needed – either in the planned outcomes themselves or in the activities you are undertaking to achieve them.

More information on log frames – and in particular the QualDATA log frame is in the article: Launching a new QualDATA Log Frame

Golden Rule 3 – Build your evaluation activities into your overall project activities

It’s good to have a Log Frame and determine what M&E data to capture, but if it just stays in a document as a good idea – it will never happen – or at least not happen systematically or effectively.  Taking the time to transfer the data collection activities from the Log Frame and allocating timing and responsibilities to them in an action table means that they are built into people’s work programs and have a lot more chance of happening.

This article has a free download of what a MER action activity table could look like: Evaluation action activity table – taking the next step

Golden Rule 4 – Plan how you will capture, collate and present the M&E data collected

M&E data gets lost on people’s computers or in piles of paper feedback sheets. It often lacks consistency in terms of the type of demographic and impact data collected and so is difficult to collate.  It also sometimes lacks a direct link to the achievable outcomes that you have promised to deliver on!  So taking time to plan and having systems in place to ensure that M&E data is collected, collated and in a format that helps reporting – is time well spent!

This article provides more information on this step and takes you to a simple on-line M&E data system for collating and reporting on individual activities.  We are working with a number of projects to tailor such systems to meet their total M&E data collection needs – and ensure that it meets reporting needs (contact us for more information).

Golden Rule 5 – Report clearly against your promised achievements

The Log Frame provides a great framework for reporting.  It clearly articulates what it is that you can deliver on during the life of the program and how you went about doing it.  The data collected systematically above provides the basis for reporting against the Log Frame – in terms of activities, outputs, process, progress, achievements and outcomes. We have found that working with project teams to provide a “mock up” of what the final M&E report might look like at the start – certainly helps to focus data collection and analysis.

This article could be useful in terms of reporting on activities: Six steps to dismissing the myth that reporting on activities is too hard

If you would like to see some examples of how you can report effectively for impact against your log frame, please contact us.

Jeff Coutts 07 4630 1297

In the QualDATA Log Frame there is a level that is called “Uptake Strategies”. This is the link between research and development and achieving change on the ground.

We don’t call it “extension” (as in the old technology transfer model which sees a linear transfer from research to farmer through extension) because there are many different ways of achieving change of which extension is just one.

The State Extension Leader Network (a national group across Australia) looked at extension in the context of the total strategies to bring about change and they also showed the dimensions of extension itself that go beyond “show and tell”.

A lot of programs aimed at bringing about change in natural resource management in Australia, for example, rely a lot on providing incentives to land managers to change practices in favour of better environmental outcomes. Governments at local, state and national level also use policy to bring about regulation and/or tax incentives to make changes.

Extension programs use a range of approaches including working with groups, organising field demonstrations and workshops – all with varying degrees of participation and capacity building. Increasingly electronic information and interaction is being used to support change.

Our evaluation in monitoring these approaches is to see how well they are working and whether changes are needed. The following table shows some of the uptake strategies used in agricultural RD&E activities, associated evaluation questions – and how this information could be captured.

Download fact sheet.

It’s the same with monitoring. It provides the signs along the course of the project/program letting you know if you are on the right track. For example, you may be aiming to change a particular farming practice of at least 20% of producers in a particular region over 2 years.

At the one-year mark, how do you know if you are on track to reach this target if you aren’t undertaking program/project monitoring activities? What if you are only on a trajectory to achieve 5% of change over the two years? What are the barriers to change? Has the context changed?What can be done to overcome these barriers? Do new targets need to be put in place?

If you are strategically monitoring your activities and impacts from the beginning, then you will be able to see when and if changes need to be made to meet your targets – or even change your targets. And then by monitoring those changes you will know if they are working or not.

In addition, when it does come to the final project/program evaluation, your monitoring activities will have produced a strong body of evidence from which to draw conclusions about its impacts in the short as well as projected long term.

I recently worked with two project teams (representing several grain projects and a horticulture project) supporting them in gaining skills for developing a Monitoring and Evaluation (M&E) Activity Plan.  The aim was to give team members the personal skills and a common understanding at team level to manage this within their projects.

The highly interactive training program helped participants realise the importance of implementing structured monitoring, evaluation and reporting processes in their projects.  Several team members also had the opportunity to practice an evaluation technique – in this case an ORID.

So what were some of the lessons learned?

Chris Schelfhout, Project Manager, Gascoyne Horticulture Development and Chris Zaicou, Research Officer, Grain Crop Agronomy, Geraldton, Carnarvon demonstrating the use of an ORID

• Data gathering is far more effective when soundly based in a program logic – A number of group members acknowledged that, while appearing ‘the long way round’, gaining an understanding of key principles and processes about Monitoring, Evaluation and Reporting (MER), makes it easier and more cost efficient to identify the most effective data gathering techniques.
• Understanding the logic of practice change and adoption is important for those conducting primarily R&D projects – An understanding of Bennett’s Hierarchy and the concept of KASA (Knowledge, Attitudes, Skills, Aspirations) is crucial for those undertaking R&D work, as well as those responsible for extension and communication.  One key reason is to understand the valuable feedback that can come from routine stakeholder engagement.  This can be used to demonstrate that the R&D work is on-track to deliver its aims and objectives.
• Starting with the ‘end in mind’ (practice change and adoption of R&D outcomes) is an important element of understanding MER – The MER connection between those involved in R&D work and those involved in extension appears at first glance tenuous.  However, after the training it became apparent that MER commonly links the outcomes of R&D to be delivered through extension.  This is because the R&D and extension/communication/adoption people start to talk ‘a common language’ – in this case ‘measuring success over time’
• Stakeholders are not just stakeholders – The group members reviewed the various stakeholders involved in both R&D and extension and adoption work in their projects. It became apparent that there are various segments of the end user/farmer and next user (consultants and reseller agribusinesses) target audiences based on geography, type of enterprise, attitude to change, openness to adopting new ideas, etc. So the feeling was that greater segmentation based on improved understanding of the key drivers of target audiences was a crucial part of the process going forward.
• Practice change is far different from ‘raising awareness’ – While discussing how to measure change over time, it became clear that it is more difficult to report the success of awareness raising activities than for many practice change activities. This is where the importance of considering changes in KASA amongst target audiences became apparent. This is a defined way of demonstrating change over time and can show that some key audiences are on track to take on board the key new ideas or technologies that project teams are responsible for.
• Clear measurable project or program objective are crucial – Most members of the group initially struggled to refine their project objectives to make them measureable, so they could readily demonstrate progress towards achieving their committed outcomes.  However during the training, and as they came to understand key principles of MER, these objectives were reviewed into SMART (Specific, Measurable, Attainable, Realistic and Timely) terminology and methods of measuring success against much more measurable objectives were able to be set up.

The one and a half day program featured group and individual work as well as the theory of MER, which assisted team members to work through the necessary changes in thinking about how to implement MER effectively at project level.  It also helped thinking about how their projects ‘fit’ with wider organisational and program objectives.

Not surprisingly the focus on food (pizza and pasta) and wine (to delight any palate) and a strong focus on faith (with Churches everywhere) were obvious.  I wondered why so few overweight Italians were evident and was told that ‘portion control’ is the answer.  And a devotion to flavors (rather than mixing everything together as we tend to do) with each food item and flavor to be savored.

I have to admit that I had little evaluation on my mind so bear with me if I give you a few observations on Italy and its relationship to Australian agriculture!

We spent a few days in a Tuscany farm apartment and explored areas away from the normal tourist route.  Here I met a sommelier who explained that bio-dynamic and organic wines are an becoming an important part of the Italian wine industry and sought after by top Rome restaurants.

He also commented that Italians are increasingly interested in organic foods and are concerned about declining food quality and the food production imbalance (as Italy is a net food importer).

The question on his mind was – are food producing nations like Australia aware that others in the world are increasingly seeking higher and higher quality foods?

Ploughed fields in Italy

One thing I noticed, and discussed with an agronomist on our trip, was how they ploughed the country.  Most ploughing was done up the hill and down into the valley – the absolute opposite of our contour ploughing.  We couldn’t come up an explanation although we noticed that most of their tractor work is done by crawlers rather than wheeled tractors.

We ended up in a world-renowned city starting with V… and the site of many movies! However the point of this story is that I wandered around the backblocks of that city to see how the locals lived and was impressed by the local greengrocer supplying the local population with high quality fruit and veg.

Fresh fruit and veg in Venice

More non evaluation anecdotes next newsletter!

However, sometimes outcomes are influenced not only by the quality of our project activities but by the broader context in which we are working.  For this reason, the QualDATA M&E Log Frame, includes a level (Row 6 – underpinning the rest of the log frame) called External Influences on Outcomes – Political, economic, climatic and other factors that can impact on the success or otherwise of the program.

We do this to ensure that program management and funders think about those things that can unduly influence the outcomes – so that programs and projects are assessed within this context.

For example, if you are working on a project with an aim to increase productivity in agriculture – and the project falls in a withering drought – than the resultant productivity decline is probably due to the drought rather than failures in the project.

In this case, monitoring should be about looking at what changes may be needed in KPIs and associated activities to make best use of project resources to deal with this challenge – and evaluation should be about how successful the project was in adapting to these circumstances and what productivity loss was reduced in the circumstances.

Factors such as: availability of/access to capital; market prices; input costs; interest rates; changing regulations or policies – can all have impacts on project outcomes.

In the Chilean dairy industry (which I have an opportunity to explore) the culture is not to borrow large amounts of capital and there is a ready and relatively cheap labour force.  The combination means that innovation requiring heavy capital investment is very slow – regardless of the best extension efforts within programs.

Recognising and monitoring such context issues assists with realistic expectations as well as providing an opportunity to modify and renegotiate program outcomes and activities.

Also note that sometimes you need to acknowledge context factors that helped you achieve – or even exceed – your project objectives.  Excellent seasons, prices, new innovations, low interest rates and other parallel programs can all positively impact on your project outcomes.

By recognising these and describing their contributions you provide a more realistic evaluation and better learning opportunities for future projects.

Jeff Coutts

At the top of the list is the article about measuring your achievements with key performance indicators (KPIs) with the next most read being about focusing your evaluation with key result areas.

These articles are consistently being accessed on our site from month to month with a small increasing trend over 2011.

It is becoming more and more important to ensure that projects and programs are evaluated in a purposeful and targeted way to provide evidence of successes as well as monitor where improvements can be made.

The evidence is clear, evaluation can not be left to the end of a project/program as an afterthought, it needs to be included from the ‘get go’, integrated with the KRAs and KPIs from the planning stage, and used to monitor its direction along the way.

And my association with sustainable wildlife management: As we drove to the north of the province (Caraquet), what I did notice were the miles and miles of (regenerated) forest as we drove and the lack of farms and fields. (I am told that there are other parts of the province where there are large farms) – and coming back, we saw at least half a dozen “trucks” (read “utes”) with a recently shot moose ceremoniously laid out in the back.  Apparently it was day 1 of the 3 day moose hunting season – where hunters enter a lottery for the right to part of the national cull.

Hopewell Rocks

There are fascinating elements to this part of the world including: the towns where French is the main language and, while sitting in a cafe enjoying coffee and chocolate croissants, you could swear you were in the centre of France; the Hopewell  Rocks which has the biggest and fastest tides in the world; and  Prince Edward Island with its 12.9km Confederation Bridge joining Prince Edward Island and New Brunswick (longest in the world).

I did search on the Net to find something about evaluation and extension (which I could have done from Toowoomba) and found the Canadian Evaluation Society with its associated Canadian Journal of Program Evaluation which looks like it has great material – and I did find an interesting approach to benchmarking and comparative program evaluation which used a series of evaluation questions around: Appropriateness, Cost Effectiveness; Management Effectiveness; and Program Best Practices.

I will look further for country insights into extension and evaluation and hope to report more in a later newsletter!  Still trying to get accustomed to Tim Horton’s coffee!

We have found that people have had difficulty in thinking through the appropriate KASA for given practice changes. This Fact Sheet takes a logical  look at how to identify relevant KASA changes for practice change. Download Fact Sheet.

Sometimes when I’m working on or planning an evaluation I go hunting for online resources that could provide me with useful insights. It is interesting to see how different frameworks and approaches work and what they might value-add to what we do in QualDATA. I thought I would share some that I found with you.

1. Australasian Evaluation Society (AES) website – A source of good evaluation information, particularly if you are a member. This page provides lots of useful links including discussion lists, websites and evaluation organisations.
2. Charity Evaluation Services – A UK based registered charity established in 1990 to strengthen the voluntary sector through offering free and below-cost support and services to charities and community organisations. Their monitoring and evaluation section provides some useful information.
3. The American Evaluation Association list of evaluator and evaluation blogs – A really interesting list of blogs you might like to read and tweeters you may like to follow. Is there a list of Australian blogs and tweeters out there? Let me know.
4. Governance and Social Development Resource centre (GSDRC) – This site has been designed to be an online guide pointing to up to date and credible resources in a development context. The M&E page is definitely worth looking through.
5. W.K Kellogg Foundation evaluation handbook – Provides a framework for thinking about evaluation and outlines a blueprint for designing and conducting evaluations. The W.K. Kellogg Foundation was established in 1930 to help people help themselves through the practical application of knowledge and resources to improve their quality of life and that of future generations.