Some of the most controversial writing surrounding the science of climate change results when the work of the Intergovernmental Panel on Climate Change scientist-authors is translated into summaries suitable for policy-making audiences.
In recent years, some scientists have criticized the summaries as being susceptible to political manipulation, or, on the other hand, as hard to read by the general public.
But a new examination of the process found that the editorial tug-of-war among scientists and government delegates often produces summaries that are a bit longer, but also clearer than the first drafts. The report, by four researchers from the Carnegie Institute for Science, also concluded the summaries were generally not oversimplified, yet were still likely comprehensible by a lay person or a lawmaker.
“The summary approval process is a little jewel,” said study author Chris Field, who added it “should be cherished by both the scientific community and governments as the only place where scientists get to road test their approach to explaining things and governments get to play a role in fine-tuning and crystallizing messaging in a way that works for them.” Field is a director at Carnegie Institution and was the co-chair of the IPCC’s Working Group 2 team, which produced the policymaker summary on impacts, adaptation and vulnerability for the most recent assessment.
In this new study, published Friday in the journal Science Advances, researchers used formal methods of “text analysis” to comb through eight policy summaries produced during the last two IPCC assessments in 2007 and 2014. For example, scientists compared at least an initial draft and final draft of each summary to see how much the text changed during revisions tied to feedback from policymakers.
Despite a handful of glaring cases where highly political material got deleted in large chunks from specific sections, most summaries emerged from government feedback longer, clearer and filled with more examples, the report found.
The study also found the accessibility of the summaries mostly improved during the revision process and the final summaries were about as readable as, say, a report by the World Bank. But these summaries “will never be as easy to read as” as newspaper articles, the authors wrote.
“This paper … opens up what could be seen as a black box,” said Kirstin Dow, a geography professor at the University of South Carolina who was not involved in the study. She added that the analysis shows exactly “what happens through different stages of this complex review process.” Dow was a lead author on the fifth IPCC assessment Working Group 2 report and contributed to its summary for policymakers.
IPCC assessments involve hundreds of scientists from across the world collaborating on a series of reports to explain what is known, and to what degree of confidence, about climate change’s causes, impacts and solutions. After the reports are completed, often running thousands of pages long, the authors begin writing shorter summaries for policymakers. Scientists write an initial summary that’s distributed to most country governments. Thousands of comments flow back from around the world, and scientists incorporate these responses into a new version. Then something unique happens: The scientists and government delegates meet in person to review the draft, line-by-line, with approval by consensus.
“Discussions famously go through not only a week of approval, but they often extend through the nights in the final days,” said lead study author Katharine Mach. She said the recent analysis set out to determine whether this final part of the approval process added value.
Mach and her colleagues found that in every case, the text of these final summaries were between 17 percent to 53 percent longer than the original draft. The researchers also checked individual passages that were “politically sensitive.” For example, passages on observed temperature increases expanded 67 percent and text discussing a global warming “hiatus” grew 137 percent.
But a notorious section from the IPCC’s Fifth Assessment Report in 2014, on international cooperation relating to mitigation from the Working Group 3 (WG3) summary, famously shrunk by 66 percent.
On the final night of the Working Group 3 in-person review in April 2014, it became clear that many country delegates were unhappy with the existing language on international cooperation. Scientists and delegates worked through the night, unable to find a compromise on what to specifically say about mitigation solutions to climate change As a result, most of the text on this issue were deleted. (Another section in this summary on country-specific emissions also was heavily edited down.) In the weeks that followed, several frustrated participants, including Harvard economist Robert Stavins, publicly questioned and criticized the process.
Stavins, though, said the problems with the 2014 draft happened in part because the discussions included people involved in both the global climate negotiations and the summary review process. People were being asked to quickly validate text on material on which they were in the midst of negotiating and had strong opinions. “That may just have been an impossible task,” Stavins said.
Another WG3 member Gabriel Chan, an assistant professor at the University of Minnesota, said the study accurately reflected his experiences with the review process. But, he said the study lacked an examination of how representative the final summaries are of the content of the full reports. He described this as, “one really important piece of the analysis that was missing here.”
Chan, Stavins and some other WG3 colleagues published a paper last year with recommendations for how to improve the summary review process, such as expanding the amount of time policymakers and scientists have access to each other prior to the in-person review.
A separate part of the analysis looked at the readability of the final summaries. Earlier this year, a previous text analysis study by KEDGE Business School’s Ralf Barkemeyer and colleagues concluded the papers were not approachable. (According to Barkemeyer’s team, Albert Einstein’s physics papers were easier to read than these summaries).
Mach and her team replicated some of their analysis and concluded the readability of the summaries matches the complexity of the material. They did, however, offer suggestions for improvements in the future and also recommended the process of communicating this material shouldn’t stop with the summaries.
“What’s most important, in my opinion, is getting to consensus,” said Linda Mearns, a senior scientist at the Institute for the Study of Society and Environment (ISSE). If getting to consensus and maintaining scientific rigor is your main goal, she said, “you might expect that readability could suffer and certainly in my experience it has suffered.”
Mearns, who was not involved in the study and who has participated in both the IPCC working groups 1 and 2, told InsideClimate News she was heartened to see that so many reports ended up more readable after the intensive line-by-line review process. She said she expects an increased focus on readability in future assessments.
With multiple rounds of IPCC assessments under her belt, Mearns added: “The IPCC is difficult to change. It’s kind of like a very large ocean liner. It’s not easy…to steer and change course quickly.”
