To Test or Not to Test – part 1

We’ve received several requests lately to write our point of view and/or provide tips on testing concepts or creative executions. It’s an important topic, and one worthy of a thoughtful approach.

Here are a few things we’ve learned in our travels. This is divided into two posts, for ease of use. (And because, like me, you may prefer things in bite sized chunks.)

1. Be concrete, not abstract. (what to test)

There are things that lend themselves well to testing, and those that do not. The more abstract or ‘conceptual’ the stimuli are, the less likely your research respondents will be able to give you useful information.

For example, people do not understand, nor should they be expected to understand, the difference between a strategy and a tactic. If you’re hell-bent on testing positioning statements or other strategic elements, which I do not advise, then illustrate in some way how that strategy is going to feel and behave in reality. Do not ever make the mistake of having a researcher print your positioning statement or brand idea on a sheet of paper, and ask respondents “what do you think of this?” and “do you like it?”. You will get entirely misleading information back.

And even when you do provide some creative expression or illustration of the strategy, don’t be surprised if the comments focus primarily on the colour you used, or the placeholder photo, or the font. Fortunately, a good moderator will be able to guide and probe beyond those tendencies and glean some useful insights for you. (See below for section on moderators.)

2. Get personal. (how to test)

I strongly recommend against online panels and other quantitative ways of testing creative – the experience is too superficial and the opportunity to observe behaviours and see nuances is nearly non-existent. Gravitate toward qualitative research, where you can ask further questions and start to notice discomfort or interest that’s non-verbal.

And further, consider 1-on-1 interviews in settings that are familiar, such as the person’s home, office, or a quiet cafe. You will avoid the ‘telling the group of strangers what I think they want to hear’ phenomenon, and get more candid answers. And book plenty of time so the moderator or interviewer can circle back and be a bit more ‘investigative’ about the questions. Note: With a good moderator conducting an interview in the home of a member of your ‘target audience’, they will make useful observations about lifestyle and environment that can be very valuable.

3. The moderator is everything (who should test)

When choosing a qualitative research company, it’s not how slick their website is or even how impressive their client list is that matters. It is the quality of the moderator(s) who will be working on your assignment. Ask around to see whom other companies have used time and again. Ask who the most insightful and agile researchers are. Nothing beats a great moderator, who can sense what people are not saying or sniff out an insight that’s just under the surface.

In next week’s post, we’ll focus more on your role as agency or client in the testing process.