Most teams pick between two headlines, two designs or two positioning options the same way: a meeting, a debate and whoever has the biggest title wins. A preference test replaces the meeting with the people you're trying to sell to.

Here's what preference testing is, how it compares to A/B testing, how many people you need to call a real winner, and how to run one in B2B.

What is preference testing?

Preference testing is showing two or three versions of something to people from your target audience, asking which one they prefer for a specific purpose, and asking why. The "something" can be a headline, a value proposition, a homepage design, an ad, a product name or a pricing page layout.

In UX research, preference tests are mostly used to compare visual designs. In B2B marketing, the bigger use is comparing messages and offers: which value proposition makes a buyer want to learn more, which positioning sounds different from competitors, which offer is more compelling.

The "why" is the point. A vote tells you which option won. The explanations tell you what made it win, what made the loser lose, and what to fix in both.

Preference testing vs A/B testing

They aren't rivals. A preference test tells you which direction to take and why. An A/B test confirms what happens when real visitors meet it. In B2B, where most pages never get enough traffic for a clean A/B test, the preference test often has to carry the decision. (Here's the traffic math for landing pages.)

When to use a preference test

• Choosing between value propositions or headlines before a homepage or campaign launch.
• Picking a positioning direction. Show two or three ways of describing what you do and see which one buyers find most relevant and most different.
• Naming. Product names, feature names, category names.
• Ad creative. Which direction to run before you spend on media. More in our guide to B2B ad testing.
• Page designs and layouts, such as two pricing page structures.
• You vs. competitors. Put your value proposition next to two competitors' with the brand names removed and ask which one they'd want to learn more about.

Use something else when you need to know whether a single message is clear and believable line by line. That's a message test. And if you're testing an idea that doesn't exist yet, start with concept testing.

How many people do you need?

Fewer than for an A/B test, but more than most teams think. The problem is close results. With two options, the share picking the winner has to be big enough that it isn't just chance:

These are rough thresholds at 95% confidence, using an exact two-sided binomial test. We recommend at least 30 respondents for a preference test and 50 for a clearer winner.

A close split isn't a failure. It tells you neither option is clearly better. Read the reasons: often each version has one strong element, and the best answer is a combination.

How to run a B2B preference test

1. Decide what the test will decide. "We'll launch whichever value proposition wins" is a decision. "Let's see what people think" isn't.
2. Make the options genuinely different. Different angles, not different adjectives. If your team wouldn't be happy shipping an option, cut it.
3. Keep everything else equal. Same length, same format, same level of design polish. Otherwise you're testing production quality, not the idea.
4. Recruit your actual buyers. Same job titles, seniority and company size as your target market. Colleagues, friends and customers all know too much.
5. Ask a specific question. Not "which do you like?" but "which of these would make you most likely to find out more?" or "which one best describes a tool you'd want for [job]?"
6. Always ask why. An open follow-up after the choice. This is where the insight is.
7. Randomize the order. People pick the first option more often just because it's first.
8. Call it with the threshold you set in advance, then use the reasons to improve the winner.

Preference testing mistakes

• Testing tiny differences. Two headlines that differ by one word will split 50/50 and teach you nothing.
• Asking which they "like." Liking isn't buying. Ask about the job the message has to do.
• Testing on the wrong people. A preference test is only as good as its panel. Fake "directors" will happily pick an option.
• Declaring a winner on 55/45. That's a tie. Read the reasons instead.
• Skipping the losers. Why an option lost is often more useful than why the other one won.
• Treating preference as proof of behavior. It's strong evidence for direction. Confirm in market when you can.

Running a preference test on Wynter

On Wynter, a preference test puts up to three options in front of verified B2B professionals filtered by job title, seniority, company size and industry. Each respondent picks one and explains why. Results come back in 12 to 48 hours.

Common uses: homepage value propositions, ad creative, positioning statements, product and feature names, and you-vs-competitor comparisons with the brand names hidden. Pay-as-you-go tests start at $899.

Wynter's panel is B2B only and strongest in tech roles. If you're testing designs with consumers, a UX tool with a consumer panel is a better fit.

FAQ

What is a preference test? A preference test shows two or three versions of a design, message or offer to people from your target audience, asks which one they prefer for a specific purpose, and asks why. It's used to choose a direction before launch.

What is the difference between preference testing and A/B testing? A preference test asks target buyers which option they prefer and why, before launch, with 30 to 50 people. An A/B test splits live traffic between versions and measures which converts better, which usually needs thousands of visitors per version.

How many participants do you need for a preference test? At least 30, and 50 for a clearer result. With 30 respondents and two options, you need roughly a 70/30 split to call a winner; with 50, about 66/34.

What can you preference test? Headlines, value propositions, homepage and landing page designs, ad creative, product and feature names, pricing page layouts, and your positioning compared with competitors.

How many options should a preference test have? Two or three. More options split the vote so thinly that you need a much bigger sample to find a winner.

Related guides

Landing page testing · B2B ad testing · Concept testing · Copy testing · Market research survey questions