Dear Statisticians,

I am a researcher with the AAA Foundation for Traffic Safety, a non-profit organization that seeks to improve automobile, pedestrian, and cycling safety.

I've run into a real-life business application for a statistics problem that was probably covered in a college course I took many years ago.

My co-workers and I have come up with different estimates of the z-statistic, and are in particular disagreement over the calculation of Variance for this problem. I would be extremely grateful to anyone who could offer assistance, or if you dare, a solution for the following problem:

A survey was conducted to see if a new membership renewal form yielded significantly more renewals than the previous form. 2,026 of the new forms and 2,040 of the old forms were sent out to a randomly selected sample of members (with, obviously, no replacement). Of those members who were sent the new form, 1,855 renewed their memberships (yielding a renewal rate of 91.5%). Of those members who were sent the old form, 1,849 renewed their memberships (yielding a renewal rate of 90.6%).

I am trying to find out, with a five percent level of significance, whether the new form yields a higher rate of return than the old form. Any thoughts, comments, or solutions?

Any help is truly appreciated.

Sincerely,
Chris
Research Analyst
AAA Foundation for Traffic Safety

Your question is a good one. The authors of statistics textbooks disagree on how to do this problem, and I have been in many debates as to the correct method. The way I see it you have two proportions, p1 the proportion of people who will return the old form and p2 the proportion who will return the new form. You want to see if p2 > p1. As I expect you know, the hypothesis test has null hypothesis that p2 = p1 and the alternate hypothesis that p2 > p1.

You have sample sizes n2 = 2,026 and n1 = 2,040 and the number of returns X2 = 1,855 and X1 = 1,849. Let F1 = X1/n1, F2 = X2/n2 then the statistic you want is Z = (F2 - F1)/sigma, where sigma is the square root of the variance of p2-p1. The conclusion is then to reject the null hypothesis in favour of the null hypothesis if Z > 1.645. The question, as you have said, is what is the variance, sigma^2?

Some authors suggest that sigma^2 be estimated by F1(1-F1)/n1 + F2(1-F2)/n2 and others by F(1-F)*(1/n1 + 1/n2) where F = (X1+X2)/(n1 + n2). In your case the difference is moot since the first expression gives an estimated sigma of 0.0089299 and the second 0.0089325 and the resulting Z-values of 1.03 in either case. Since this Z-value does not exceed 1.645 you cannot conclude that the new form results in significantly more renewals.

I hope this helps.
Harley