By Mick McCarthy (This article was first published in the October 2012 issue of Decision Point, The Monthly Magazine of the Environmental Decisions Group)
Billions of dollars have been spent over the last couple of decades on programs to improve the condition of Australia’s environment. These programs include Landcare, the Natural Heritage Trust, and Caring for Our Country. So, how do we know that these investments are working? Well, actually, we don’t. The Australian National Audit Office has repeatedly criticized the Australian government for its inability to measure the benefits of these large investments (Hajkowicz 2009).
One key aspect of measuring benefits of conservation investments is knowing what would have happened in the absence of that management. Where the status quo is highly predictable, that might be easy. We can manage an area, monitor what happens, and compare that to what essentially is a guess about what would have happened in the absence of management.
It is hard to imagine that the Australian National Audit Office will be satisfied with a guess. Auditors like to see documented evidence, perhaps especially when assessing the benefits of billion dollar investments. That is their job.
Determining what would have happened in the absence of management will often be difficult. The best way to do this is to not only monitor the areas that are managed, but also to monitor some analogous areas that are not managed.
The importance of monitoring outcomes across a range of management intervention is well illustrated by stream monitoring at Hubbard Brook, a forested catchment in New Hampshire, USA. It’s one of the world’s most famous and influential ecosystem studies (see box 1).
Box 1 The Hubbarb Brook experience
The Hubbard Brook Ecosystem Study is a long-term ecological research project within the Hubbard Brook Experimental Forest, a 3,160 hectare reserve in the White Mountain National Forest, New Hampshire. It is managed by the USDA Forest Service Northern Research Station. On-site research has produced some of the most extensive and longest continuous data bases on the hydrology, biology, geology and chemistry of a forest and its associated aquatic ecosystems.
More info: http://www.hubbardbrook.org/
The study was designed to examine the influence of tree cover on streamflow. The trees in one catchment area, named ‘Watershed 2’, were harvested in 1965. Crucially, the monitoring of streamflow had commenced several years prior to this. In addition to this, monitoring of a nearby catchment, which was not harvested, was also undertaken.
Using data provided from the Hubbard Brook website on annual stream flow, it is clear that annual streamflow is highly variable. Because of this, an effect of harvesting in the watershed is hard to see when only examining the annual streamflow of this watershed (figure 1).
However, a comparison of the streamflow in an unharvested watershed makes the effect of harvesting transparent, especially if we plot the ratio of streamflow in the two watersheds (figure 2).
What does this mean for measuring the benefits of large conservation investments by governments? First, when monitoring effects of management, these effects need to be measured against what would have happened in the absence of management. This means that outcomes in the absence of management need to be very predictable, or that outcomes in unmanaged areas also need to be monitored. That might be logistically difficult, because comparable sites might be hard to find. However, measuring or predicting what would happen in the absence of management is vital.
From Hajkowicz (2009):
“The most pressing and challenging dilemma facing natural resource program managers of the future will be demonstrating returns on investment. We have struggled to answer the question “what did we get, or will we get, for our money?”
“Reports from auditors have, for over 10 years, remained critical of the evaluation, targeting and monitoring of investments. There is insufficient evidence that programs are achieving intended outcomes and the investment models used to target expenditure are often unclear, ineffective or absent. This is similar to the story told by auditors in the United States, Europe and other OECD countries.”
A second important point is that monitoring outcomes in both the managed and unmanaged areas prior to applying the management intervention can be valuable. Look at the graph of relative streamflow (figure 2). The impacts of harvesting are obvious partly because data prior to the harvesting were available.
The above points (we should monitor before the management intervention, and we should also monitor areas that are not managed) are standard aspects of before-after-control-impact (BACI) monitoring designs. There is nothing new here.
However, my impression is that organisations that fund conservation works are sometimes reluctant to fund monitoring in areas where conservation works have not been conducted or to fund monitoring prior to management. As a result, capacity to monitor the benefits of those works will be reduced. The basics of BACI designs should figure prominently in the minds of those wishing to measure the benefits of their conservation actions. Monitoring should be planned more frequently as an integral part of the management actions.