image
image
image
image
image

Evaluation

imageimageimage


March 2013
Hey friends!

A lot of spectres continue to join us at morning coffee, here at troublemakers central, after the ‘Spectres of Evaluation’ forum. It was a challenging two day event, run by the VCA’s Centre for Cultural Partnerships, with about 40 participants from across CACD, including artists, arts funders, scholars and researchers. Thanks to everyone to making this happen!
As usual, we have more questions than answers [see also our paper from Dec 2012 below]:
What are we, as gatekeepers and trend makers not hearing, not seeing?
Who was not in the room?

What is the value of evaluation – especially for us as artists?
Where do we place ourselves in the spectacle of evaluation?
Be wary of any activity that demands objective truth, and detracts from subjective sharing – even gifts come within a context of power…
Who is evaluating the evaluators?
Do we have or need a CACD Ombudsperson, in light of funders increasingly wanting to act as producers?

Who defines the terms of engagement for evaluation across the arts sectors? And why?
And how can we make this process of defining ‘evaluation’ an equitable negotiation, [the act of definition is a dance of language, the linguistics of power].
How will the terms of engagement have provable benefits for all parties, in a way that we all can share, if the participants and artists are the subjects?

With all the tools we discussed at the forum, from negative value, politics of aesthetics, to networked thinking and dialogical approaches, how do we reach a re-connection between artists, participants, audience, funders and researchers, that doesn’t feed into oppressive tactics?
And how to speak to (our) privilege and the power relationship within the structures of cultural production?
Making art is about taking risks, making commodities is about the economy. Why do we assume we can evaluate the risks others take and control the purse so they can or cannot do it again? They will do it anyway, so what’s in it for us, what are we scared of?

If we evaluate community art in a society promoting social band-aids, not social revolution, then our lens most likely will value compliance not liberation.
Lots of talk about network analysis, but where is the debate on social exclusion, oppressive tolerance and the potential for social control through mapping and trapping?
What happens between the nodes of the connected social networks? Historically the majority of the world’s population doesn’t trend well.
Will more data and better algorithms provide us with better means to make ‘better art’ and be more effective – at what?
We struggled to hear a shared horizon being discussed. And beyond the horizons, what are our dreams, our visions?

How is your liberation bound up with mine? How can we generate hope AND income? How do we disconnect from ‘story theft’ and colonization, built into the very fabric of our social network interfaces and data harvest machinery?
How is evaluation different from unilateral data harvesting?
What can evaluators do for the evaluated, formerly know as the artist?
How can we determine practical solidarity, in an agile, flexible way, so that evaluation can feed into our work and build trust, rather then fear-driven compliance?
Amongst the sensation that we are a fractured sub-group of creatives within the Virtual Class, was the desire for shared language.
Business language keeps seeping into arts and culture debates. We squirm at our use of ‘equity’, but find it useful as a litmus test for power relationships, as money and symbolic exchange flows through all of our interactions.
What language can we create that can support us as cognitive workers to make sense of the new paradigm shift prompted by digital technologies, globalization and the re-emergence of nationalism in the wake of brutalised and fractalized societies?

And are we being too precious, too fragile, too invested in self-preservation? May be we should not care who steals stories and who owns the data and reads its patterns?
But what about communities who are vulnerable? Do we care? Where is this care coming from?
In a fully realized global network of Big Data, are narratives still important or do we enter the age of participation over story?
How do we speak to power struggles in our work, when we are muzzled by risk adversity?

Where do we build critical literacy into the submission, work and evaluation cycles?
We all evaluate constantly and critically reflect on our projects, but where do we share these mostly internal processes, so that the sector could gain strength and artists take part in the power of influence?
Or do we need to?
Would it be more competitive not to share? Or even increase the risks of funding getting slashed?
Or does a deeper, radical sharing of our process in terms of negative value take the ‘magic’ away?
Or could it benefit the parties involved and even the wider sector?
How can artists be transparent about process AND be able to divert and recombine resources when they see it necessary, when funding agreements demand control and compliance through pseudo-objective evaluation according to nebulous guidelines?

Our two key responses to the underlying fear and frustration we felt amongst our peers are practical solidarity and self-determination.
How do we link our work to self-determination, without succumbing to identity politics driven by fear and belonging within a culture of lack?

Perhaps we should send all the funds directly to the communities and let them go through the same trial and error we do, as ‘experts, artists, evaluators, facilitators, curators, arts managers, etc’ – that’s a lot of play money, a bag of autonomy and as many successes as the current model provides.
How do we proceed towards ‘agile trans-formative evaluation’ that could build trust through mutual risk taking and equitable process of negotiation?
What about ‘glorious failures’ as a category?
Below are a few interesting links on some of the (contentious) issues around Networked Thinking / social network theories, that struck us as one of the most vibrant and difficult debates at the forum.
We are entering the next stage of our negative value project and will have an update soon.
Please send us ideas, reading material and other stuff that makes for interesting discussion on ‘harm’ and negative indicators. And chocolate, please.

With love, Jen and Carl

The tyranny of nodes: Towards a critique of social network theories
The End of Theory: The Data Deluge Makes the Scientific Method Obsolete
Big data, language and the death of the theorist (Wired UK)
PLOS ONE: Global Civil Unrest: Contagion, Self-Organization, and Prediction
The media utopia of the avant-garde. Franco Berardi
The Warrior, the Merchant and the Sage
Futurism and the reversal of the future

December 2012
More questions than answers: Change Media’s thoughts on CACD evaluation in the age of Big Data and social media



We practice innovative disruption, the art of creatively disrupting experiences to evoke change, our collaborative artworks record and reflect these creative iterations – so evaluation is a playful fabric of our work.

And then there is project evaluation. We don’t put much faith into the old evaluation models that are mostly used for community arts and cultural development in Australia. Our concern is that the information gained from data is treated as ‘objective’, yet it is always interpreted through someone’s cultural values. We see the information gained as a constructed narrative, which we believe can be better told by all the stakeholders, specific to each project, a dynamic, iterative, subjective, in-situ, responsive representation, which may have relevance for other projects. We call it ‘agile trans-formative evaluation!’

As part of our Change Media methodology, we use a variety of performative indicators, positive and negative as well as recording delivery and milestone notes, with the aim that they all will support innovation. The most poignant evaluation appears to be in rich media content. We incorporate this into the creative process, and advise project participants that they can actively participate in the collection / production of this feedback, so they are directly contributing to and informing the process and outcome with us. This includes text, photos, video, images, concept prototyping storyboards/ artworks, website feedback, recording participant feedback rounds, filming themselves to create training tips and in action ‘behind the scenes’. We also document our management process to monitor project patterns to see if we can improve our delivery to surpass the aims and outcomes of each project, using standard issue ‘positive indicators’ such as participant numbers, workshop duration, aims, challenges, feedback recorded against outcomes/ milestones.

Agile trans-formative evaluation means for us, that we constantly review our process and methodology to check our assumptions. However, even if we use a 3-point model combining quantitative and qualitative evaluation methods with rich media outcomes, we still struggle with the problem of pseudo-objective data collection and interpretations – in short: we can make any statistic look and say what we or our clients want to hear, unless we look at evaluation in the narrative and socio-political context of the work: This is where our evaluation can disrupt our experience to evoke change.
Whose story is it, who benefits, what are the needs, assets, and how can we determine equitable process and outcomes? What is the aim for – and who needs the evaluation – our funders, partners, participants – and for what purpose?
Where is the power in the room, what is the risk and for whom? How can we reach excellence while working towards an equitable process and outcome?
Trying to answer these questions over the last years led us to view our creative process as a series of critical yet glorious failures, the cracks and gaps that arise from social innovation, opening our practise to new possibilities and renewed thoughtfulness.

How can we apply our findings to the co-creative process as soon and directly as possible?
Change Media conducts feedback sessions each day during workshops and we record behind-the-scenes materials (video and photos), which is often done by our workshop participants as a part of the co-creative process and skills transfer.
The review of feedback videos and behind-the-scenes documentation feeds into the next days process and invariably into the final art work. The inclusion of the collaborative process into the creative narrative invites the audience into a sophisticated meta-level appreciation of the creation and is an evaluation highlight for community partners, funders and clients.

We find ‘negative indicators’ are a much more challenging yet effective tool set to build and apply. We try to ask questions that reveal our own ignorance, challenge our assumptions, to question the validity of our work and our clients desires:
Who is benefiting from the workshop? How do we determine creative control, power of influence, equitably shared benefits across process and outcomes? Do skills really get transferred? What are the obstacles? How sustainable is it for the participants, ie what can they use by themselves once we are gone? What are we not seeing/hearing? Where/ what is ‘the elephant in the room’? What would happen if we or the participants don’t share their stories? How is our liberation bound up with the participants? How relevant is the outcome for an audience? Have we, as co-creators, grown through this process, have we been challenged, embarrassed, appreciated – and if not, why not?

These questions guide our innovative disruptive processes and remind us to constantly reconsider our co-creative approach, through all stages of the art production:
Who is the target audience – is it relevant for them? Why are we doing this? Where is the drama, the humour and the conflict in our narrative – this applies across all art forms, but especially to digital media…
Does the project build capacity beyond anecdotal evidence – do we see resilience growing in the community? Have we asked participants what they need, what they have already and how they see this fit in with their future outlook? What are the delivery items they need to proceed and excel?

We then aim to build DIY training tool kit productions into the creative workflow of most of our projects – if our budgets allow – so that participants train each other once they learn a new skill and ideally record their newly acquired skills in simple training videos that they produce themselves. Ideally the final artwork links all these aspects together by showcasing some of the process involved in creating the work – or offers clear links to access training content and separate documentations, available on our website or delivered to clients and participants directly on DVD or flash drives.

Depending on budget and timeframes of the projects, we then follow up before each scheduled workshop to get an idea what we need for the next stage of the project, to develop the next narrative together with the participants. For this we usually use email correspondence and when people don’t have web access, by phone. This process usually requires strong partnership development with our on-ground partners, to do follow up work with the participants, ideally it is an email to all participants.
The democratisation of media revealed a digital fantasia, connecting community arts to the world. And with it all the gremlins of hyper-connectivity, digital noise, a million flies doesn’t mean it’s not shit ‘trends’ and the evaluators nightmare, what to do with all that data? We have concerns about the commercial aspect of social media and the commodification / use of data that is accumulating. Just because some content trends well on social media sites, which are run by privately owned companies, doesn’t mean that we as creative instigators or as members of a community get a more authentic view on what people really think or feel about our work.
But we do use hit rates on Youtube and Vimeo, data on website traffic, e-newsletter click-through rates and other statistics to monitor our range of social media tools.
We believe the most effective digital media tool is narrative literacy: the reflective co-creative production process and critical media literacy through de-construction of media narratives plus documentation of process as part of the final art work. Showing how it is done and why, reflects directly on social exclusion, ability and access of our participants – and on our ability to deliver on our funding promises…

Another important aspect for us is the proliferation of excellence through digital media. Social media networks are great for spreading stories, raising awareness and participants can now see what is possible and will be less susceptible to the practise of lowering expectations. We have often heard comments like: ‘they are marginalised, they won’t know any better, it is too hard for them, lets not disappoint them any further’. As arts practitioners we are now better equipped to raise expectations by showcasing what community collaborations can achieve and share our process and outcomes from our projects, globally, which we have seen can increase self confidence and awareness for our participants. Especially important is the notion of local relevance – seeing your local community members achieve makes it so much more accessible than to watch highly produced content made abroad or in a completely different context. We have seen participants ’switch off’ and consume when we screened amazing films made by internationally acclaimed artists, but then seen their engagement explode when the content was clearly made by people they could relate to or knew.

In short: we believe there is no cookie cutter solution for authentic community arts and cultural development work, every project requires specific evaluation to effectively support social change.

And finally, we would be superhuman if we didn’t feel uncomfortable and challenged by our own methodology, we are constantly unsettled as we hit walls of our ignorance… For us the value of reframing our evaluation is to explore how we can co-create equitably, so that everyone will feel they have a seat at the table where they can experience the malleability of power lines and the value of their own power.

07 April 2010
What risks do we actually take when we claim to ‘collaborate’ with disengaged communities?



Change Media is now working with several groups to create real economic opportunities, so they can set up their own media micro-businesses. Our aim is to remove ourselves from each step once the community can do it alone.
Recently, however, we’ve had some interesting encounters as a growing number of documentary makers engage with marginalized communities… and I began to rethink our strategy and the use of one-liners such as:

When you witness your story making a difference, you see you can change the world.

Sure, I love to hear these feel good success stories. It is a privilege to work with communities as they develop the skills to make their own media,… but at present many marginalized communities still only experience the power of media when an external, privileged documentary team parachutes in, to document local issues or re-tell their stories, from a “media expert’s” view point. I believe this disparity of creative control and ownership is a crucial element of the colonial language, which keeps marginalized people dis-empowered and illiterate, (no matter how good the story makes us feel in the short run).
As long as these visiting artists don’t train, share and leave sustainable capacities behind, this process prolongs existing dependencies, which means marginalized communities must rely on outsiders to be heard within the ‘mainstream’.

Over the last few years we have been developing a code of conduct for privileged artists working with marginalized communities. Last week it was exciting to share experiences and develop a global project, with Mervin from the iStreet Lab, Jamaica. Working with artists and academics in Australia, Jamaica, Canada, England and the USA, we are preparing an ambitious project to put into practice a global manifesto – to kick-start a discussion about practical ethics and measurable sustainability for authentic media cross-cultural collaborations and provide a benchmark for transparency in CCD work. I believe this is essential as we witness a push from the Creative Industries to engage with communities.

If you want to contribute to the development of the new manifesto, post a comment below or contact me via email here.

Find a link to our current manifest here.

Cheers,

Jen

 

Toolkit

News

arrow

Recent Project: Ngarrindjeri Culture Hub

 

The new Ngarrindjeri Culture Hub website is now live!

And the exhibition date is set for June 4th, with invites to go out mid May:

read more

Projects

Recording My Elders
Pinnaroo Heist
Don't Be That Guy - 90 Second version
Showing Ur True Colourz
Square of Light - Trailer
Christie Walk - A Closer Look
Mission Possible
Everything Is Connected - Ngarrindjeri Weaving
Meningie Foreshore Restoration
Vandalism
Surviva Nunga Style
Pinnaroo Surfer Series - Heavyweight Champion
Ngarrindjeri Yarluwar Ruwe
Marlpa Holiday
Putting Out Fires
Legends of Quorn
River Redgum Restoration
Everything Is Connected - Ngarrindjeri Dance
Tanks
Don't Be That Guy - TVC
Renmark - Hard to Swallow
10x14 Bricks - Hope
RAA - Remote Directing
Pinnaroo Surfer Series - My Watering Hole
Mai Palya. Anangu Kunpu. Kata Palya - Behind the scenes
10x14 Bricks - Anger
Pinnaroo Surfer Series - The Fish Whisperer
Wake Up
Pinnaroo Surfer Series - Painting a Masterpiece
Steve Edwina Goes Wild
Everything Is Connected - Ngarrindjeri Carving - 30 sec Trailer
The 56 Floods

Watch All Films