You've probably seen a lot of great post-campaign output organisations produce themselves, or via agencies on campaign performance. Indeed, the selection of campaign 'participants' is also somewhat of a science and is essential to producing not only good campaigns, but campaigns upon which actions for the future can be built. In this article, however, we are going to focus upon campaign analysis and in particular testing. So your campaign has finished, the response data has been collected, and now you need to assess your campaign's performance. There are a multitude of campaign testing techniques that can be employed but where do you start with your analysis?

Our approach

At Talking Numbers we like to think of two different types of campaign analysis:

• PCAR (Post Campaign Analysis: Reporting)
• PCAF (Post Campaign Analysis: Factors)

PCAR concentrates more closely upon actual results; for example Creative A outperforming Creative B, and building reports to present all interesting findings from the campaign's performance. These reports will typically be built in Microsoft Excel with various metrics such as response rates, CPT (cost per thousand), CPR (cost per response), CPS (cost per sale) and ROI etc. See example excel file A report like this might also include response curves for instance showing cumulative response by week. In the example below, we have used our own Campaign analysis module to find out that 95% of the response had occurred by week 4 for this specific campaign. PCAF looks closely at the factors explaining why Proposition/Creative A outperformed B. It goes a step further in explaining which aspects of the campaign have worked well from proposition to selection criteria. For example, a direct mail campaign based on a model will have all aspects of that model assessed in terms of how they contributed towards the campaign performance, enabling marketers to identify the components that were really driving the model. In this article we cannot go through all the various techniques we would use in producing one of these reports but some of the things they might include are:

• Significance testing
• Comparing response rates
• Calculating roll-out
• Campaign costing and income
• Analysing tabular data using chi square test (covered in earlier e-shot)
• Comparing means using T-test (covered in earlier e-shot)
• CHAID (covered in earlier e-shot)
• Sampling methods for selections

All of the above are highly useful tools/techniques when undertaking campaign testing but it is the first three on this list that we will focus upon in this article.

Hang on a sec - check the data first!!

Just taking a step back from these techniques for a minute, the first port of call (as with most analysis projects) is ensuring that the response data you have is reliable: Do you have records of ALL responses received? Can the responses be attributed to the correct source? More often than not you cannot answer "Yes" to both or even one of these questions. In these instances you will just have to make do with what you have at your disposal and caveat your findings. It is often the case that, due to incomplete data, campaign analysis cannot be exact! Moreover, even if you can answer "Yes" to these questions it often occurs that where a response is attributed by a call centre or customer code in the situation of a multi-channel campaign that the response might not be allocated correctly. For example a caller stating they saw an advert on a leaflet and being processed as a press insert response whereas in reality it was a door-drop. There are ways to 'back-process' these responses but for the time being we just need to be aware of their existence.

So I have different responses within my campaign. Great, but is this significant?

Speaking generally, once your response data has been collected and sense checked an excellent place to start will be examining the response rates to your campaigns! However, a key factor in looking at your response rates is checking that they are significant. In other words if one cell has an apparently better response rate than another does this mean that you should alter your strategy or resources. So before releasing information on response rates that look fantastic (or in some cases dire) it is important to understand the figures you are looking at, and determine if they actually mean anything? This is probably the most mathematical part of campaign testing, but it is important to examine significance testing before considering using other analytical methods. There are a number of terms that are useful to understand first.

Test versus Control

Control: The current situation or the base to which a value is to be compared. Test: The situation that you want to compare with the base. For example: The standard pack has a response rate of 1% and a new whizzy pack gets a response rate of 1.5%. In this case the standard pack is the control and the new pack is the test. As you're all aware as marketers, a test could sometime be an offer, a new selection e.g. SMEs rather than bigger businesses, a proposition, creative, contact method e.g. a cell that got a mail pack and phone call versus just a call or pack. So we want to know did our test work or did it flop?

So first a hypothesis!

When working with statistics and comparing values an analyst tries to prove that there is a difference rather than there is no difference. The "null" hypothesis as it is known, is always that there is no difference between the control and test samples. The alternative hypothesis could be that the response rate for the new mailing pack is greater than for the control. For example: if an analyst was looking at investigating whether there is an association between industry type and propensity for the organisation to require a franking machine. So the null hypothesis is that there is no difference. The alternative hypothesis would be that there is an association. The statistical method would then need to prove that there is an association by rejecting the null hypothesis. The amount of evidence needed will depend on the level of confidence required.

Next - Are we confident in the result?

The level of confidence is a measure of how strong the evidence is required before the test and control can be considered to be different. A standard level of 95% confidence or a 1 in 20 chance of making the wrong decision is used most of the time. Where it is very important to make sure that there is a difference between the control and test samples then higher levels of confidence, 99% or 99.9% are used.

Confidence intervals

These are measures of how extreme a value should be before a difference can be determined. In a Normal distribution 5% of values are outside 2 standard deviations from the mean. In statistical testing terms a p-value of 0.05 is any value outside this range. Confidence intervals can be used to show how far a value needs to be from a set point before a significant result can be determined.

Another concept - Significance and p values

This describes the level of probability that you will make a mistake by rejecting the idea that there is no difference between the control and test groups. A p value of 0.01 means that there is a 1 in a 100 chance that you will reject the idea that there is a difference between the control and test groups when there is in fact no difference. i.e. we're likely to have got it wrong 1% of the time. However a word to the wise: Just because a result is significant does not mean that it is interesting in marketing terms! Testing the significance of response rates is rarely the only analysis you need to do on the campaign. More often than not it is just the tip of the iceberg!0 Evidence Graph

So now let's compare response rates

One of the most frequent pieces of analysis carried out is to compare response rates. There are three principal ways of doing this:

1. Use a set of tables
2. Calculate the figures from first principles
3. Use the TN Calculator

The problem with the first two is that it can be time consuming and difficult to use tables and a certain amount of statistical and software knowledge is required for the second. So I really recommend you get a programme to aid you with comparisons, or buy our TN Calculator. Let me explain why. Hold onto your hats for some maths! The formula for comparing two response rates can be found in most statistical textbooks, but can be daunting for some. this is often referred to as the p formula Where:

• p1 is the control response rate
• p2 is the test response rate
• n1 is the control population
• n2 is the test population

If Z is greater than 1.96 then the two response rates can be regarded as being different. An exercise working through the example using the formula: Two mailing packs have been sent out, a control and a test. The size of the control mailing was 2,000 and elicited a 4% response. The test mailing was 2,500 and elicited a response rate of 5.3%. Was the test mailing significantly better than the control at 95% confidence level? You could work through the formula to find (or take our word for it!) that the test mailing performed significantly better than the control at the 95% confidence level.

Great a successful test, now I want to calculate a roll-out

So now you have tested your mailing, as per the example above, found it to be a success but what sort of response rate should you expect when you roll the campaign out? A perfectly reasonable question, if you are planning to go back to the Business Universe and buy more data. If we continue with our example we assume our response rate will be about 5.3% (based on the test that we have carried out) and we plan to mail 20,000 organisations; these are the only two pieces of data we need to make this calculation. Let's also continue working to the 95% confidence level. p1 is the anticipated response rate n1 is the mail cell size Putting our figures into the formula would look like this: Working through this formula shows us that we can expect our response rate to fall between 5 and 5.6% for roll-out. Well, now we've covered three techniques which should go some way to help you plan and analyse campaigns. Of course you would want to link the rollout depth to cost and likely margin to calculate a breakeven point. OK - another time. Just a quick and unsubtle plug - no doubt you would find these calculations quicker and more convenient to use a bespoke calculator to do the hard work for you. The Talking Numbers DM Calculator is available at a cost of £150 per copy and will help you to:

• Calculate sample size required
• Compare response rates
• Calculate roll-out

It really does avoid the pains of constructing the formulae required for many campaign testing techniques there are a number of products available which will make all the calculations for you. It comes with full documentation and examples. And just for fun, and if you've got this far, here's our free prize draw to win one and check you've been paying attention. Please answer the question below and send your answer by the end of April 2007 to It really does avoid the pains of constructing the formulae required for many campaign testing techniques there are a number of products available which will make all the calculations for you. It comes with full documentation and examples. And just for fun, and if you've got this far, here's our free prize draw to win one and check you've been paying attention. Please answer the question below and send your answer by the end of April 2007 to It really does avoid the pains of constructing the formulae required for many campaign testing techniques there are a number of products available which will make all the calculations for you. It comes with full documentation and examples. And just for fun, and if you've got this far, here's our free prize draw to win one and check you've been paying attention. Please answer the question below and send your answer by the end of April 2007 to

The BIG Prize Draw

n.b. this is not Riemann's hypothesis (OK - so that has a $1m reward for getting that one right), it's a response rate comparison question: A test mailing of 5000 elicited a 1% response rate. What is the expected roll-out response at 95% confidence? The first incorrect answer drawn out from our responses will win the calculator. N.b. If you get the answer right you obviously don't need it. So if you would like to know more about campaign testing by all means contact me and I can send you further details of our training course. Best and happy campaign planning & testing! Luke