Testing the Processes: Communities at Work

Communities at Work are the largest community organisation in the ACT and were one of Amplify Online’s first users. They engaged CSI back in 2019 for some guidance around evaluation and it’s great to now see them testing the processes to sustainably conduct their own outcomes measurement.

We chatted to Michael Malone, who worked at Communities at Work for 2 years in a Community Development role.

Catch up on our chat below to hear about how they’re finding Amplify Online:

What does Communities at Work do?

We’re a broad-based organisation in Canberra, offering a range of supports and programs across children’s services, services for seniors, community transport and crisis support services, among lots of other things.

My time at Communities at Work is sadly coming to an end as I’m moving interstate but I’m keen to take everything I’ve learnt with me. Community development is pretty close to my heart!

What kind of impact measurement work have you done?

We prioritise evaluation as an organisation broadly speaking, but social impact evaluation is something we really want to pick up steam on.

We’ve used surveys as part of our reporting in some of our programs in the past. An example that comes to mind is our community transport service, which is essentially a service we provide to people who may not have the means to get around – for example, people who may have specific health needs or are elderly. We were interested in reviewing the kinds of goals people hoped to achieve through accessing that service and whether it helped them to achieve those goals.

We engaged with CSI in 2019, who led our Theory of Change workshops, and worked collaboratively to develop our Theory of Change model and guide our indicator selection, which we ended up applying to other program areas.

Since then, I’ve been reviewing our internal processes and coordinating the development of a social impact evaluation framework to use both for our community development work and across the organisation.

What kind of challenges has Communities at Work experienced around measuring impact?

Resourcing is a big one. Children’s services are one of our biggest offerings and there's currently a nationwide workforce shortage. Programs obviously need to deliver a service and have key priorities to focus on - it’s not that easy to find the resourcing to conduct evaluation.

We're at the stage where a coordinator such as myself can do things like analysis and compiling reports, but actually collecting program data takes a decent chunk of staff time that we don't necessarily have.

Another challenge is that being such a broad organisation, each service has different funding models and industry standards - and often their own evaluation processes - so there’s a lot of segmentation. That’s why integrating social impact evaluation across the organisation is so exciting for us. We can become more consistent in understanding our impact and success.

How have you used Amplify Online so far?

We’ve been getting a feel for the platform but haven’t put anything out yet. I drafted a survey using indicators I found on Indicator Engine for a program called ‘Wellness through Watercolours’ – which involved a series of workshops facilitated by an art therapist. The program was in response to the effects of lockdown on people’s wellbeing, and its aim was to improve wellbeing holistically and in a way that’s consistent with the goals of community development.

We wanted to understand changes experienced by workshop attendees over time. Indicator Engine was a useful tool for finding indicators and helping program staff to see what kind of questions they’d be asking. So, I used it to draft a paper survey, which we plan to transfer to the platform and analyse the data through Yardstick.

We’ve also played around a bit with custom questions but have been waiting for the platform enhancements.

How did you find the indicator databank?

Community developers are interested in that intersection between connection with community and wellbeing. There were some key indicators that really suited what we were trying to find for ‘Wellness through Watercolours’ around positive emotional wellbeing, personal resilience and social interaction.

We were especially happy to find the positive emotional wellbeing indicator as it doesn’t use heavy language. ‘Wellness through Watercolours’ offers indirect support for people with specific mental health conditions, so we wanted to keep the survey fairly light to reflect that.

From your perspective, what is the value of Amplify Online?

The value for us is that we can find easily communicable details about what our actions are achieving – internally, through to our clients and to the broader sector and community.

Tracking our impact internally is important for implementing social impact evaluation across the organisation. If staff members can see hard figures on what they're achieving and say, for example, 60% of people who have been engaged by the service said they feel better about themselves now or that their physical health has improved because of us - a lot of the time you have suspicions about that - but being able show it is really motivating.

We're trying to implement a whole system of social impact evaluation where we're understanding the outcomes of what we do using standardised indicators. Being able to find those indicators on Indicator Engine and integrate them into what we're already doing will make it so much easier.

What’s next for Communities at Work?

A big focus is getting people used to doing evaluation and testing the processes. Once we're used to the broader, quantitative research processes, we can introduce some of the more demanding components of measurement, like interviews.

Most people have already got their processes, but we’re looking towards systematising that in a more comprehensive and consistent framework. Quantitative measurement a good place to start for people who are just getting their toes wet in social impact evaluation, in terms of organisational change and bringing about a culture of evaluation.

Amplify Online is an incentive, or at least an encouragement for us to say, “What are we waiting for?” We've been working towards implementing social impact evaluation for three years across the organisation. Obviously, we've had COVID interrupt this, so having Amplify ready to go will be such an enabling factor.