Sofus Midtgaard

Measuring the success of Open Innovation & Idea Platforms

November 2, 2009

User-driven Innovation, Open Innovation, Crowdsourcing, Co-creation, Idea Platforms… the buzz-words are many from the world of Innovation- & Social Media Consultants.

Much debate has spun around how to measure the effect of initiatives and whether these are good value for money. I the following I will zoom in, and try to get closer to a practical approach to setting goals and measuring the success of Open Innovation & Idea-platforms. For a more on Idea-platforms see a list of some of the most prominent implementations of idea-platforms.

What to measure?

It’s hard to set up a universal metrics to evaluate the success of an Open Innovation or Open Idea Platform initiative. Of course the number and quality of ideas, debate as well as the number of votes gives a first indication of success. More critical though is the organizations ability to interact with customers, tap into ideas and launch new profitable products and services based on the feedback.

As in all projects, it is important to make a distinction between Project goals and Effect goals. The Project Manager is typically responsible for meeting the Project goals – whereas the Project owner or steering board is responsible for meeting the Effect goals as they require the active participation of more business units/departments to reach.

Project goals

Below is a list of possible Project goals for and open innovation/idea platform initiatives sub grouped after Quantity, Quality and Diversity of participation.

Quantity of Participation

• Number of ideas submitted
• Number of votes
• Number of comments

Quality of Participation

• Quality of the ideas
• Quality of comments
• Quality of dialogue

Diversity of Participation

• Number people who submitted ideas / Number of ideas submitted
• Number of people who voted / Number of votes
• Number of people who commented / Number of comments

Of course the project goals will differ depending on the type of project and organization, but based on the above metrics a scorecard could look like this:

Evaluating the quality of ideas

The number of incoming ideas, debates and votes are quite easy to quantify and measure. The number of people who submitted ideas, commented and voted should also be easy measurable. But when it comes to defining the quality of ideas, comments and dialogue it becomes a harder.

The principal of Crowdsourcing and Idea platforms is of course to tap into the thoughts and feelings of customers, employees, external experts or citizens. The first evaluation of the quality of ideas is done by users voting and commenting on published ideas. It will quite quickly be evident whether you have few or many great ideas.

Expanding voting and feedback posibilities

A way to further qualify the ideas posted is to expand the possibillities for user feedback. Beyond voting up/down you can let users make a five star evaluation of how:

• Innovative: (1-5)
• Cost reducing: (1:5)
• Easy to implement: (1-5)
• Affordable: (1-5)

…the ideas are. Tempting as it seems this also raises the bar and complexity for user participation.

Internal crowdsourcing strategic scoring of ideas
Another simple way to evaluate the quality of ideas, comments and dialogue is to gather a small but diverse group consisting of representatives from business development, sales/marketing, communication, product development, technical department and make a quick grading (1-10) of the ideas, comments and dialogue in general.

If you want a more thorough internal evaluation of the best ideas, you can expand the internal scoring process and make the participant score ideas on criteria’s like:

• Strategic Match
• Market Potential
• Instant customer value
• Long term customer value
• Potential for cost reduction
• Level of competition
• Technical complexity
• Estimated cost of development
• Etc.

Effect goals

Below is a list of possible Effect goals for an commercial open innovation/idea platform initiative:

• Perception of company as more innovative (brand)
• An increase in general customers satisfaction
• Increased customers satisfaction with relevant products
• The financial performance of new products inspired by the customer feedback
• The financial performance of improved products inspired by the customer feedback
• Increased customer loyalty (reduced churn)
• Reduced cost due to more efficient processes inspired by the customer feedback

Effect goals are harder to quantify than project goals and messuring will take longer, as data typically derive from different company systems and surveys. Moreover it is often difficult to track the effect directly back to the project/initiative. An ideal effect goal scorecard based on the above could look like this:

In reality though you should consider how much resources you spend measuring an initiative and to what extend you will be able to track the effect back to the project/initiative. But being clear on what you want to achieve from the start – even if you cant measure it accurately – will help you focus your initiative/project.

Government or Non-profit

If you are working in Government, Public administration or a Non-profit the relevant effect goals will of course differ from commercial companies. A scorecard for an open innovation /idea platform initiative within a public organization could contain one or all of the following effect goals:

Incremental vs. radical innovation

Critics has pointed out that customer driven/inspired innovation will seldom generate unique or revolutionary ideas – and that ideas for more radical innovation has to come from other sources. This might be true in some cases, but does in my opinion depend very much on the framing of the initiative.

Despite its claimed tendency to favor ‘product improvement’ ideas, a major strengths of open innovation platforms is that ideas – even obvious ones – gets out in public. This creates a pressure on management to respond and fix problems that might otherwise be forgotten among more prestigious management initiatives.

Before launching an open idea platform

Be clear on what do you want to achieve before launching an open idea platform! With your goals defined – consider if  there are other activities or tools that could help you reach the same goals spending the same or even fewer ressources?

To often it seems that companies are persuaded by consultants to launch open idea-platforms to communicate openness and increase brand value. Both are very valuable goals, but an initiatives will surely backfire if the framing is not clear and the initiative is not an integrated part of further products development priorities.

Some important preparations before you launch an open idea platform:

• Set a clear framing for ideas and suggestions
• Allocate a good moderator
• Secure that moderator or key employees/managers welcome and comment on ideas
• Make incoming ideas and feedback visible within company
• Integrate ideas and feedback in strategic priorities and product development priorities
• Launch new products and improvement and remember to thank users

Here is a list of some of the most prominent commercial and governmental idea-platforms…

Feel free to comment

I would like to hear your comments or experiences working with open innovation/idea platforms. What are the primary goals companies formulate before starting a project – if at all? And how is the evaluation being done – if at all?