Playing the Game of Diplomacy

23 Mar 2012

The US State Department reaches out to audiences abroad with people-to-people contacts and social media. Now it wants to measure the effectiveness of its soft-power ‘games’.

 The United States government conducts robust public diplomacy. These efforts received new life under the Bush administration after 9/11 and have continued in the Obama presidency. These exercises in the use of non-coercive soft power seek to win the hearts and minds of target foreign populations.

With that goal in mind, the US State Department conducts a variety of programs, including people-to-people exchanges (which involved some 50,000 individuals last year) as well as maintaining a social media presence on Twitter and Facebook, and other Web 2.0 platforms such as YouTube.

In one 2011 project, the State Department's Bureau of International Information sought to dramatically increase the number of its fans on Facebook. It succeeded: within the space of three months the Department's followers grew from 800,000 to four million.

These numbers are impressive, but they aren't useful for scoring success in the soft-power game. They don't measure the qualitative outcomes sought by public diplomacy.

The State Department was faced with the issue of measuring outcomes in yet another program. The department's Bureau of Educational and Cultural Affairs recently piloted an English language skills program in Tunisia accessible over mobile phones. The program is used by 500,000 unique visitors a month – again, an impressive number considering Tunisia's total population of ten million.

But this raw data is hardly a measure a public diplomacy success. "The traditional measure of the success of such a program is the extent to which it improves English language skills," Suzanne Hall, a senior advisor in the Bureau of Educational and Cultural Affairs, told ISN Insights. "Now the metric is more about reach."

In other words, the State Department wants to be able to measure the effectiveness of the Tunisia program from the perspective of whether it is achieving public diplomacy goals.

Goals, scores and indices

It is not an unreasonable task; after all, public diplomacy is analogous to political and marketing campaigns. Politicians and marketers are constantly taking the pulse of their target publics to make sure their messages are getting through. That is how those games are played; public diplomacy efforts should be subject to the same scrutiny.

"We live by other indexes like the consumer price index and the Dow Jones industrial average," Amy Zalman, a researcher at Science Applications International Corporation (SAIC) told ISN Insights. "So why not a soft power index?"

Zalman is calling for a measure of how US public diplomacy efforts are faring overall. "We need a comparative soft power index and tools to provide policy makers with an intelligent way to view the US position vis-a-vis other actors in a competitive environment over time," she said.

Zalman cited several reasons. "One is for planning purposes," she said. "Government needs a way to figure out where to put its resources. It's also a good idea to score something where the US is in a particularly competitive environment in order to explain to the American people why the government is making certain expenditures. Finally, other countries, both competitors and collaborators, have soft power strategies that we can ignore only at our peril."

Zalman's ideas are not entirely new. An attempt to set forth a series of metrics for the measurement of effectiveness of public diplomacy was proposed in a external pagereport to the US Advisory Commission on Public Diplomacy by The LBJ School of Public Affairs of the University of Texas in 2010.

That report posited a construct it called the ‘Public Diplomacy Model for the Assessment of Performance’ (PD-MAP), which it described as "a flexible framework that allows an evaluator to quantify the results of public diplomacy programs and evaluate their success" in meeting three strategic goals or outcomes: "increasing understanding of US policy and culture; increasing favorable opinion towards the US; and increasing the US's influence in the world."

Public diplomacy ‘evaluators’ would first create goals and measurements for programs. Field officers would collect data on the impact of the program and enter it into an interface. Through PD-MAP, an evaluator could run calculations and produce scores for programs based on their success in meeting strategic goals. The State Department could then use those scores to increase the scope of successful programs and improve or close those that are underperforming.

Zalman's proposed metric is global in scope. It does not focus on individual public diplomacy programs but on the effectiveness of US public diplomacy as a whole, compared to other international actors. Developing that sort of index, Zalman asserted, requires an update to the rules of the game.

"Messages don't exist in a vacuum," she said. "Treating them as such doesn't help [us] understand how attitudes, emotions, and ideas attach. Many factors contribute to influencing foreign publics and we need to use as many as we can. We need a way to assess the way power expresses itself in the twenty-first century. It's not the same as it used to be."

Several key factors have shifted in recent years. Organizations have shifted from the hierarchical to networked. "How connected an actor is may be a form of power," said Zalman.

Power was once equated with size. Now, smaller actors have many advantages: Agility and speed, rather than size, better inform the measure of effectiveness, according to Zalman.

The notion of the rational actor has given way to a better understanding of the role of emotion and unconscious motivations. The idea that actors were autonomous and individuated has yielded to the understanding that boundaries are porous and that environmental factors shape actions and identities.

"Different people perceive and react to the same information in different ways," Zalman said.

Through the lens of these various factors, Zalman proposes three measures: resources, discourses, and interactions. "Resources show commitment and what the US can bring to the dissemination of ideas and values," said Zalman. "Discourse is the place where ideas gather and change." Interactions refer to the networked component of how people organize and how they receive and exchange information.

Zalman suggested that text mining can be used to analyze how perceptions and attitudes change in reaction to public diplomacy efforts. Another tool is to conduct polling that is narrative in nature.

"Instead of asking directed questions," Zalman explained, "you let them go on and talk. If they do, they reveal the structure of their thoughts, how they think about certain issues, and how they frame them. All of this information can be digitized and analyzed with tools that can analyze semantic tones."

All of this involves a qualitative analysis of how target audiences are thinking, and, Zalman conceded, "you can't measure everything. Some things will elude us." But her proposed scoring "can sharpen the analysis in such a way as to make public diplomacy do what it is supposed to do."

The State Department has already started text-mining social media sites in order to sharpen the focus of its programs. "For too long foreign service officers have been shooting in the dark," Nick Namba, an acting deputy coordinator at the department's Bureau of International Information told ISN Insights. "We want to give them an idea of what publics are interested in."

Namba's unit uses programs to scrape and analyze chatter generated on social media sites. "We suck it all in and try to use the data to provide useful information for the secretary and the field," he said. "Everyone is trying to do it but no one quite has a handle on it yet."

JavaScript has been disabled in your browser