Designing research with a statistical issues in mind

We have discovered that the goal when analyzing a between-subjects research design is to find (in a t test) that the differences between the groups (X1 - X2) is about twice as large as the difference that is expected by chance when sampling the same population twice (SQRT [(SEM1)2 +(SEM 2)2]).  Such a finding is deemed statistically significant, or put differently, that the independent variable (the groups) is related (perhaps causally) to the dependent variable (what was measured).

The purpose of this handout is to drive home the point that there is an important connection between the decisions made when designing research and the statistical analysis of the resulting data.  Specifically, the goal of this handout is for you to understand how a study can be designed to maximize the likelihood of finding statistical significance. The five data sets here are created to help you see how 1) limiting error variance (by using reliable instruments, minimizing extraneous variables, sampling people with less variability, etc.)  and 2) maximizing differences between groups (by creating extreme groups or conditions, etc.) matter in the testing for statistical significance. 

The first data set and the one to which the other data sets will be compared.  It is all worked out for you. Ob. is observation, Diff = (X - M), and Diff2 = (X -M)2

GROUP 1						          GROUP 2 
Ob.	Mean		Diff		Diff2			Ob.	Mean		Diff		Diff2

7	6		1		1			1	2		-1		1
6	6		0		0			2	2		 0		0
5	6	           -1		1			3	2		 1		1

Mean = 6							Mean = 2
Standard Deviation = 1 					Standard Deviation = 1		
Standard Error: .58						Standard Error: .58
value of t =  4.90

Notice that data set 1 had a t value which is very high. The difference between groups was significant! However, the t statistic is very sensitive to difference between group means, such that the same variability within each group but less of a difference between the two groups will produce a smaller t value.  To see this, compute the results for data set 2.
 
GROUP 1			                                              GROUP 2 
Ob.	Mean		Diff		Diff2    		Ob.	Mean		Diff              Diff2 

6	5		1		1			2	3	           -1		1
5	5		0		0			3	3		0		0
4	5	          -1		1			4	3		1		1

Mean = 5							Mean = 3
Standard Deviation = 1 					Standard Deviation = 1		
Standard Error: _____					Standard Error: _____
value of t = _____

Now consider how sensitive the t value is to increasing the variability within each group (i.e., increasing the standard deviation), even when the means of the groups remain the same.  To see this, compute the results for data set 3.

GROUP 1						        GROUP 2
Ob.	Mean		Diff		Diff2 			Ob.	Mean		Diff		Diff2

4	5		-1		1			4	3		1		1
4	5		-1		1			4	3		1		1
7	5		 2		4			1	3	          -2		4

Mean = 5							Mean = 3
Standard Deviation = 1.73 					Standard Deviation = 1.73		
Standard Error: _____					Standard Error: _____
value of t = _____

This time, see what happens to the value of t when taking a similar data set and just ever so slightly deceasing both the variability within each group and the difference between the groups.

GROUP 1						         GROUP 2
Ob.	Mean		Diff		Diff2 			Ob.	Mean		Diff		Diff2

4	4.7		 -.7		  .49			4	3.3		   .7		 .49
4	4.7		 -.7		  .49			4	3.3		   .7		 .49
6	4.7		1.3	    	1.69			2	3.3		-1.3		1.69

Mean = 4.67							Mean = 3.33
Standard Deviation = 1.15 					Standard Deviation = 1.15		
Standard Error: _____					Standard Error: _____
value of t = _____

Data set 4 showed that there are no changes to the t value when equally reducing the difference between groups and the variability within groups.  But, see what happens to the t value when slightly lowering the variability within each group and but not changing the differences between the groups,.

GROUP 1						          GROUP 2
Ob.	Mean		Diff		Diff2			Ob.	Mean		Diff		Diff2

4	4.7		-.7		.49			4	3.3		 .7		 .49
5	4.7		 .3		.09			3	3.3		-.3		 .09
5	4.7		 .3	            .09			3	3.3		-.3		 .09

Mean = 4.67	 						Mean = 3.33
Standard Deviation = .58 					Standard Deviation = .58	
Standard Error: _____					Standard Error: _____
value of t = _____