Blog Feed

The mission of Upwell is to condition the climate for change in marine conservation, and to ready people to take action. In order to do this, our team sifts through the vast amount of real-time online content about the ocean, and amplifies the best of it. Upwell’s campaigning model capitalizes on the insights we glean from Big Listening and other curation efforts, and responds to the currents of online conversation. Through an iterative process of lots and lots of campaign testing, we find ways to create spikes of attention in conversations. Ultimately, we hope to raise the day-to-day, week-to-week, month-to-month, year-to-year baseline of those conversations.

What is an Upwell campaign?

Upwell’s campaigning model combines a few key elements. Our campaigns are attention campaigns, focused on raising attention to ocean issues. They are minimum viable campaigns, operating on short time-frames and focused on rapid delivery of content, continuous learning and iteration. They are run and amplified across a distributed network, rather than being housed on, and amplified by way of our own platforms. 

The Attention Campaign

The nonprofit community has deeply-held ideas of what constitutes a campaign. Often, organizations build campaigns with institutional goals (e.g.: awareness, list-building, advocacy and fundraising campaigns), and compete with other entities in the same sector/issue space. Upwell’s attention campaigns operate on a different plane, one in which success (greater attention) elevates the work of everyone in Team Ocean, and is tied to no particular institutional outcome other than generating conversation. 

What we do with attention campaigns is try to drive more attention to existing content and actions that are not on our properties. They’re not associated with our brand. We use this loosely held connection, tying into the momentum of the news cycle, and being strategically opportunistic in the pursuit of creating spikes in attention. 

We focus on shareability, and measure our success by the same, simple attention metric we use to measure online conversations: social mentions. Social mentions are the currency of attention, and represent small bits of action. In contrast, awareness is a less meaningful measurement, representing what someone thinks they might do, not what they have done. 

Over time, we believe that increased attention to ocean issues will raise the daily baseline of conversation about ocean issues. We have been experimenting with trying to understand what makes baselines go above the expected, or historical level (i.e., what causes spikes in conversation), with an eye toward making these increases in attention sustainable.

The Minimum Viable Campaign

“You can't just ask customers what they want and then try to give that to them. By the time you get it built, they'll want something new.” - Steve Jobs

On the advice of Sean Power, Data Scientist at Cheezburger, Upwell has adapted an agile development principle from the  lean startup movement - the minimum viable product. Our campaign lifecycle embodies the Build-Measure-Learn cycle that software developers have used in order to quickly release products with the minimum amount of functional features, in order to gather immediate insight that can inform later iterations. 

The cycle of agile software development

Through our minimum viable campaigns, we employ ongoing, iterative, continuous delivery of content, resisting our urges toward perfection and providing irreverent, timely, contextual content to audiences immediately instead of strategizing for six months, or a year. We focus on the quickest, dirtiest thing we can get out the door that we think will have a measurable effect on a conversation. 

Our campaigns have short lifecycles - anywhere from a couple hours to a few days, and they are inspired and informed by hot news that feels really immediate to those campaigns. We move very rapidly through a process of hatching an idea, finding or creating the campaign product(s), putting it out into the world and getting back data. We are constantly learning how to be more effective. In our first year running over 160 minimum viable attention campaigns, we have learned that even a tiny bit of effort can make a huge difference in how campaigns get picked up.

Combining These Models

The minimum viable campaign model could be applied to not just attention campaigns, but also fundraising, advocacy, or other types of campaigns. Likewise, an attention campaign could certainly be run at different time scales. For instance, there is no reason why Red Cross couldn’t start doing minimum viable campaigns. Keeping everything else the same, they could tighten up their campaign time cycles and run experimental campaigns to engage their base in different ways. Red Cross could also start running attention campaigns. If they believed that Ushahidi was doing really good work, they could run an attention campaign, pointing at Ushahidi’s work and amplifying attention to it. This could turn out to be a faster path to achieving their own mission.

By applying both these models, Upwell has crafted a new way of campaigning that is easily delivered, measured, and adapts to the ever-changing sea of conversation. In summary, through our campaigns, Upwell:

• Surfs existing conversations in order to increase and expand attention.
• Measures social mentions (rather than policy outcomes, petition signatures, or public opinion) to evaluate the success of our campaign efforts.
• Delivers, measures, and learns from campaigns on a short time cycle, embedding lessons and insights immediately. We sacrifice perfection.
• Collaborates with a network of ocean stakeholders and curates a diverse set of existing ocean content, rather than building on our own brand, and creating our own content. Our campaigns are not aligned with Upwell program priorities or policy goals, but instead amplify attention to the priorities and goals of those in our network.
• Runs our campaigns across a distributed network of ocean communicators, rather than relying on our own platforms as information hubs.

The Upwell Network

The key to our campaigns’ success is in our network. Our attention campaigns are amplified not by us, or by a dedicated base of supporters we’ve built over the years, but rather by the network of ocean communicators that we regularly contact through the Tide Report, our social media channels, and our blog. We call this “running a campaign across a distributed network.” It’s more of a syndication model than a direct-to-consumer model.

We built our network proactively to respond to several trends. With the rising cacophony of the internet, the rapidly increasing pace at which news spreads, and the shift toward people finding news through their friends on social media channels rather than getting it directly from “official” channels, we decided to approach network campaigns in a new light. It would have been cost prohibitive to buy the attention (through ads or purchasing email lists) or to build a world-class, unbranded media hub. Rather than collect a large set of official MOU’s and partner logos to put up on our website, we built a loosely held, distributed network. We’ve reached out to nodes of people who control the communications channels that reach lots of people who are interested in ocean issues. We’ve been scrappy and ruthless about who we put into that distributed network, trying to make it diverse and ensure the reach is big.

Campaigning across a distributed network means that we have that golden ticket of communications - message redundancy - but those redundant messages are all tailored by the individual nodes in our network for their audiences. It’s the job of the individual people in our network to know their audience really well. They take our messages and content and they translate them out to their audiences through the communications channels they maintain. 

As a point of comparison, Upworthy, a similar effort that launched just after Upwell and that shares our goal of making social change content more shareable and “viral,” approached the problem of distribution from a different angle. Rather than build their own network through which they could distribute the content they curate, they built their own media hub, repackaging content under the Upworthy banner, and rapidly scaling up an audience and brand of their own. This model certainly brings eyes to worthy content, but doesn’t (yet) effectively pass on engagement to the organizations and individuals it supports - it retains that engagement for its own channels.

We wanted to build an issue-specific network, and through our networked campaigns, strengthen our network’s members’ and supporters’ potential for future action. 

Below are the values that guide Upwell in building and strengthening our distributed network:

• Trust: we share only science-based content, ensuring that other science-based institutions know that the content we share is trustworthy.
• Transparency: we share our campaign and big listening data with our network, so that they can apply our lessons in their own work.
• Brand-agnostic: we work as willingly with Greenpeace as we do with Deep Sea News, as we do with the Facebook page “I Fucking Love Science.” We will share an organization or individual’s content or campaign, as long as it promotes ocean conservation goals and fits our curation criteria. Often, promoting content from an array of brands meant releasing control of the message.
• Issue-agnostic: We aren’t working just on overfishing, or GMO salmon, or catch shares to cultivate the network. We amplify any ocean campaign or content as long as it fits our curation criteria.
• Personal: We build relationships with humans, not organizations. The liveliest online conversations happen between people, not institutions. We model the authentic behavior of the internet.
• Generous: We provide small bits of advice and feedback to help our network do better. If their work will get more people talking about the ocean online, it fits with our mission.

***

We've spent the last six weeks reflecting on our pilot project, and want to share our results with you. This post is one in a series of pieces about what we've learned over the last 10 months.

If you like this post check out:

• Upwell's Ocean Conversation Baseline Methodology
• Upwell's Spike Quantification of the Ocean Conversation
• Conversation Metrics for Overfishing and Sustainable Seafood

What is a Baseline?

When Upwell says that we listen to the ocean conversation, what that means in practice is that we're conducting Big Listening on English-language conversations in the following eight "ocean conversation" topic areas: Overfishing, Sustainable Seafood, Marine Protected Areas (MPAs), Oceans (broadly defined), Cetaceans (whales and dolphins), Sharks, Tuna, Gulf of Mexico and Ocean Acidification. For each topic, both real-time and historical data provide essential context for understanding the volume, evolution and characteristics of the overall conversation.

Each topic we monitor is characterized and defined by a set of search terms (including exclusions) that we refine on an ongoing basis. While we recognize the limitations of “keyword groups,” such as their reliance on text-based results and the absence of  contextual awareness, they do provide a powerful tool for analyzing online attention. The development and active refinement of keyword groups is at the heart of Big Listening methodology.

We use Big Listening in order to:

• identify and target high-value items for campaign purposes,
• compare the relative size of different ocean sub issues (e.g. sharks vs. whales), and
• measure the impact of our campaigns.

Since Upwell is a campaign agency (among other things), we needed a way to characterize these conversations as they exist, absent our interventions.

Enter the Baseline.

Baselines help us to anchor campaign performance targets in measures of past conversational volume. We set goals informed by the Baseline (as well as by spikes), and then campaign to meet and exceed those targets.

Upwell informally defines a conversation’s Baseline at the point below which the daily volume doesn’t drop. It can be thought of as a floor (although it is often quite high - in the tens of thousands for a conversation like Cetaceans), or as the number of social mentions performed each day by the topic’s diehard conversationalists. If everyone else left the party, the Baseline would still be there, dancing by itself.

Robyn - euro popstar, solitary dancer, human Baseline metaphor

The Baseline: Up Close and Personal

Upwell’s Baseline methodology has evolved to capture the highly dynamic conversations we watch, especially cyclical variations by day of week. These cyclical variations often result from usage and posting patterns. For example, people tend to talk on the internet when they’re at work. Over the course of our pilot grant, Upwell has used three different version of baseline methodology to better measure the dynamic online conversation space:

• Baseline v1.0: The lowest level of daily social mentions for a given conversation, for a given period (implemented using Upwell topical keyword groups) [in use through late August 2012]
• Baseline v2.0: The median daily social mentions for a given conversation/keyword group for a given period [in use through mid-November 2012]
• Baseline v3.0: The average of the lowest 10% of social mention values for a topic on a given day of the week [in use through early January 2013]
• Baseline v3.1: The average of the lowest 20% of social mention values for a topic on a given day of the week [currently in use]

Our Baseline quantification methodology was created with input from leaders in the field including: K. D. Payne, Chairman & founder of Salience/ KDPaine & Partners, and co-author of the recently released Measuring The Networked Nonprofit; leading nonprofit technologist and Packard Fellow, Beth Kanter; and a senior Educational Policy Analyst for a leading national measurement/ social statistics firm contracted by the Gates Foundation, the Broad Foundation and the U.S. Department of Education.

To calculate the Baseline for a particular topic we begin by compiling all available social mention data for the period since we started monitoring it (mid-October 2011 or later, depending on the topic). We then disaggregate the data by the day of the week in order to deal with cyclical variations in post volume and compare Mondays to Mondays, and Sundays to Sundays. Once that’s done we calculate the average (mean) of the lowest 20% of values for each of the seven days. Taken together, those day-of-the-week values are what we refer to as the Baseline.

We selected the average (mean) to establish a specific value for each day of the week for three reasons. First, the mean is the starting point for calculating standard deviations used in our spike quantification methodology. Second, given the small size of most ocean conversations, the mean is the most typically consistent and available measurement when analyzing the conversation on a by day-of-the-week basis.

Baseline Social Mentions by Day-of-Week for Upwell’s Sustainable Seafood keyword group (10/17/11 - 1/29/13)

These daily Baseline values are then graphed against social mention data over time. The graph below shows the result.

Social mentions for Upwell’s Sustainable Seafood keyword group vs. Upwell’s Sustainable Seafood Baseline, June 1, 2012 - August 1, 2012

Context and Challenges for Baseline Quantification

As seen in the overfishing and sustainable seafood conversations, day-of-week periodicity is highly evident in social mention volume, largely driven by mention increases during U.S. working hours across Eastern to Pacific time zones, and with substantial drop offs on weekends. Using the current Baseline as a reference for setting campaign goals removes the disincentives present in previous Baseline quantifications to campaign on lower volume days, and gives a more accurate picture of success on all campaign days.

Our campaign efforts, along with unexpected conversational developments (think: the release of a new report or the arrival of a natural disaster) require us to add new terms to the relevant keyword group so that our work, or online mentions of some relevant, unanticipated development, is captured in our search terms. In the inevitable cases where we find noise in the results returned by particular terms (new and old), we respond by tightening or removing those terms from the keyword group.

Since keyword groups have keywords added and subtracted on an ongoing basis, there are some inherent challenges. How should Baseline calculations account for these changes? How and when should Baselines be refreshed? Should campaign targets be refreshed retrospectively? How should we treat a spike that loses or gains volume with a refreshed keyword group? Moving forward, Upwell will continue to drive innovations in our baseline methodology and in the integration of that methodology into our campaign process.

Finding the Baseline

Developing a Baseline for a topic or conversation requires an iterative process of definition, testing and measurement. And once you’ve got it, you have to refresh it to account for conversational evolutions. It’s like a marriage in that way, or so we’re told.

Measuring something requires you to define it. The trick in measuring a conversation is that conversations change over time as participants engage in dialogue. As conversations dynamically evolve over time, so too do the methods of expression (i.e. terminology, imagery, metaphors, etc.), the composition of participants, and the accompanying platforms. Much like species evolution, these changes are not always in a direction that we perceive to be fruitful. Evolution can lead to progress just as it can lead to dead-ends or fragmentation. Because of this, no conversational “listening” can ever be exhaustive (as some elements of the conversation will always be overlooked), nor can it be perfectly accurate (as noise will always creep in). This is why we continue to add and subtract terms to our Baseline keyword groups. Ongoing monitoring and modification of a Baseline keyword group is the most effective way to keep that keyword group refreshed and accurate. Since designing a keyword is as much about selective addition as selective omission, our ability to effectively evolve keywords and keyword groups is informed just as much by our listening practice with Radian6 as it is by our personal listening, professional network and subject matter expertise. No robot can do this, although some are trying, and interns probably can’t do it either.

In “baselining” a conversation, Upwell begins by developing a conceptual framework for the topic in question. For the purposes of explanation, let’s imagine our topic is Marine Debris. We would begin our Baseline development process by outlining the conceptual and temporal boundaries of analysis for marine debris. The temporal aspect is important because a keyword group developed for one time period may lose significant accuracy (and utility) when applied to another period. For the conceptual outlines we often make use of a mind map such as the one for Oceans shown partially below.

Upwell mind map for Oceans keywords, January 2012

For marine debris, the concept map would include items such as marine trash, the pacific gyre, marine plastics, great pacific garbage patch, seaplex (minus exclusions for the botanical shampoo of the same name), albatross AND plastic. The concept map would also include people, campaigns, expeditions and organizations such as Miriam Goldstein (a marine debris expert), The Trash Free Seas Alliance, the Plastiki, and Seaplex.

The concept map becomes a design artifact for further conversations about the conversation. Although we sometimes shortcut this process in the interests of time, we refine the map through a series of discussions and email exchanges with subject matter experts and knowledgeable people in the industry or industries at play.

Once we have a solid map of the conversation, we turn the map into a series of keywords. Keywords are textual search terms, much like something you might google. A keyword for Upwell, for example, would be “Upwell.” But it might also be a distinctive phrase (or fragment thereof) such as our tagline, “the ocean is our client.” A keyword, in this way, can actually be a number of elements (such as multiple words in a phrase) despite its singular form. This fact will become more important as we discuss more of those elements. A collection of keywords is called a keyword group, and search results for keyword groups are the foundational output of Big Listening.

A couple wrinkles make the construction of keywords and keyword groups significantly more challenging that one might expect. The first is noise. When you type something into a Google search bar and click on a result, you are deploying a potent combination of Google’s massive computational power, billions of dollars in said company’s algorithmic investments, and the concentrated smarts of your own interpretive brainpower. The last element is particularly important. Whereas Google displays dozens, if not thousands of search results for you to choose from -- and then asks you to filter those results-- the keyword queries we construct for Big Listening must be built so as to filter out as much noise as possible. To return to friend-of-Upwell Miriam Goldstein, mentioned above for her marine debris expertise, a well-constructed keyword group for that subject would probably not include her name as a standalone keyword -- the reason being that she talks about other, non-marine-debris subjects as well. Entering “Miriam Goldstein” as a keyword nets you any mention of her full name, whether that takes the shape of a blog post about ocean trash or a friend’s tweet referencing her attendance at a particularly yummy brunch meeting. What you leave out of a keyword or keyword group is as important as what you put in.

Pruning out extraneous results through proper keyword construction brings us to the second wrinkle: exclusions. Exclusions are also textual search terms, but their purpose is to filter out results that match their terms. Exclusions can be tied to specific keywords or to entire keyword groups. A well-constructed (or scoped) exclusion can be the difference between finding online mentions of sharks, the creatures, or finding online mentions of the San Jose Sharks, the hockey team, creatures of a different sort. Exclusions can filter out things beyond keywords (such as entire categories of website domains, particular geographies of origin, or what a computer determines to be a particular language) using a variety of tools used in Big Listening-- source filters, to give one example.

Another wrinkle in keyword construction is proximity. Proximity is not available in every tool that might be applied in a Big Listening process, but it is present in Radian6, our tool of choice at the moment. Proximity is a modifier that can be applied to two or more words in a keyword, say “marine” and “debris.” Proximity, denoted by “~”, tells the tool/service how close a set of words must be in order to return a match. Closeness basically means: how many other words come in between? If we were to set proximity to zero for “marine debris,” Radian6 would return only items that include that exact phrase. If we set proximity to three, for example, we might get results such as “marine layer clotted with debris.” That distinction becomes increasingly important when Radian6 scrapes long forum discussions, news articles, or blog posts in which a topic might be mentioned in an extremely peripheral manner. Proximity provides another tool to scope a given keyword and focus the results in a particular way.

Keyword development feeds into an ongoing measurement process of Scope / Test / Adapt / Share. The cycle is presented below.

SCOPE

Initial investigation

1. Outline the conceptual and temporal boundaries of analysis for the topic

2. In consultation with subject-matter experts and other stakeholders, create a seed list of topics, subtopics, potential online influencers, and known online sources, events, and campaigns

TEST

Generating preliminary keywords

3. Use the seed list to develop initial keyword inputs for online search and social media monitoring services

4. Develop a more detailed set of keywords

5. Verify keyword accuracy and relevance using Radian6 to graph and spot-check search results, adding exclusion terms to filter extraneous results/noise, or various degrees of proximity to widen the net

ADAPT

Refining keyword groups

6. Share keyword lists with key informants (subject experts, foundation staff, campaigners etc.) and incorporate feedback (e.g., additional terms, scope adjustments)

7. Repeat steps 3, 4 and 5 with updated keywords

Cultivating and maintaining keywords

8. Campaign or otherwise monitor keyword group results on an ongoing basis

9. Update Baseline keyword groups with new inclusions and exclusions based on current events, campaigns and other developments (while always testing for the introduction of noise, per step 5)

10. On an as-needed basis, generally after at least three months of listening, share Baseline keyword groups with subject matter experts and other groups to gather feedback and potential improvements

SHARE

Exporting and preparing data

11. On a monthly, quarterly or to-order basis, export Big Listening data based on the most current keyword groups

12. Recalculate Baseline values

13. Graph and annotate charts with spike identifications

Packaging and distributing insights

14. Create reports, blog posts and other types of synthesis for external audiences

Improving the methodology

15. Gather feedback and process what we’ve discovered

16. Iterate our overall set of procedures

A crucial detail of the final stage of this process is the fact that exporting the data freezes it in time. To offer a contemporary example, all of the data in this report was current as of the end of January 2013, and then it was frozen in a spreadsheet. It’s important to remember that the conversations we monitor continue to change, even as we’re measuring and reporting on them. 

Because the exported data is a snapshot of results for a particular conversation’s keyword group, as it existed at a particular time (of export), from a particular tool (or combination of tools), the resulting values cannot and should not be separated from the keyword group that produced them. Furthermore, due to the item volume returned by some of some of the larger keyword groups, exporting data will sometimes produce variations in measurements for the same hour, day or time period. This variation is due to the tools we use and is generally extremely small given the scale of the topics we’re monitoring. These two factors combine to reinforce our belief that Big Listening data can only be fully interpreted if the underlying keywords are available -- anything less is a black box

***

We've spent the last six weeks reflecting on our pilot project, and want to share our results with you. This post is one in a series of pieces about what we've learned over the last 10 months.

If you like this post check out:

Upwell's Spike Quantification of the Ocean Conversation

Upwell's Distributed Network Campaigning Method

Conversation Metrics for Overfishing and Sustainable Seafood