Designing research with a statistical issues in mind

We have discovered that the goal when analyzing a between-subjects research design is to find (in a t-test) that the differences between the groups (M1 - M2) is about twice as large as the difference that is expected by chance when sampling the same population twice (SQRT [(SEM1)2 +(SEM 2)2]).  Such a finding is deemed statistically significant, or put differently, that the independent variable (the groups) is related (perhaps causally) to the dependent variable (what was measured).

The purpose of this handout is to drive home the point that there is an important connection between the decisions made when designing research and the statistical analysis of the resulting data.  Specifically, the goal of this handout is for you to understand the methodological issues in designing a fair test of hypotheses using the techniques of null hypothesis testing.  The five data sets here are created to help you see how 1) limiting error variance (by using reliable instruments, minimizing extraneous variables, sampling people with less variability, etc.) and 2) maximizing differences between groups (by creating extreme groups or conditions, etc.) are important considerations in testing for statistical significance. 

The first data set is all worked out for you. You can run this and subsequent analyses on SPSS or by hand or caculator using the formulae SEM = Sd / SQRT (N) and t = (M1 - M2) / (SQRT [(SEM1)2 +(SEM 2)2]).  Abbreviation are define das follows: Con = Condition, Ob = observation, Diff = (X - M), and Diff2 = (X -M)2

HUMANS (Condition 1)				SQUIRRELS (Condition 2)
	Con	Ob.	Mean	Diff	Diff2			Con	Ob.	Mean	Diff	Diff2
	1	7	6	1	1			2	1	2	-1	1
	1	6	6	0	0			2	2	2	 0	0
	1	5	6	-1	1			2	3	2	 1	1

	Mean = 6						Mean = 2
	Standard Deviation = 1 				Standard Deviation = 1		
	Standard Error of Measurement: .58			Standard Error of Measurement: .58
value of t =  4.90

Notice that data set 1 had a high t value suggesting that the difference between groups was significant! However, the t statistic is very sensitive to difference between group means, such that the same variability within each group but less of a difference between the two groups will produce a smaller t value.  To see this, compute the results for data set 2.
 
Humans (Condition 1)				Squirrels (Condition 2) 
	Con	Ob.	Mean	Diff	Diff2    		Con	Ob.	Mean	Diff	Diff2 

	1	6	5	1	1			2	2	3	-1	1
	1	5	5	0	0			2	3	3	0	0
	1	4	5	-1	1			2	4	3	1	1

	Mean = 5						Mean = 3
	Standard Deviation = 1 				Standard Deviation = 1		
	Standard Error of Measurement: _____		Standard Error of Measurement: _____
value of t = _____

Now consider how sensitive the t value is to increasing the variability within each group (i.e., increasing the standard deviation), even when the means of the groups remain the same.  To see this, compute the results for data set 3.

Humans (Condition 1)				Squirrels (Condition 2)
	Con.	Ob.	Mean	Diff	Diff2 			Con.	Ob.	Mean	Diff	Diff2
	1	4	5	-1	1			2	4	3	1	1
	1	4	5	-1	1			2	4	3	1	1
	1	7	5	 2	4			2	1	3	-2	4

	Mean = 5						Mean = 3
	Standard Deviation = 1.73 				Standard Deviation = 1.73		
	Standard Error of Measurement: _____		Standard Error of Measurement: _____
value of t = _____

This time, see what happens to the value of t when taking a similar data set and just ever so slightly deceasing both the variability within each group and the difference between the groups.

Humans (Condition 1)				Squirrels (Condition 2)
	Con	Ob.	Mean	Diff	Diff2 			Con	Ob.	Mean	Diff	Diff2
	1	4	4.7	-.7	.49			2	4	3.3	.7	.49
	1	4	4.7	-.7	.49			2	4	3.3	.7	.49
	1	6	4.7	1.3	1.69			2	2	3.3	-1.3	1.69

	Mean = 4.67						Mean = 3.33
	Standard Deviation = 1.15 				Standard Deviation = 1.15		
	Standard Error of Measurement: _____		Standard Error of Measurement: _____
value of t = _____

Data set 4 showed that there are no changes to the t value when equally reducing the difference between groups and the variability within groups.  But, see what happens to the t value when slightly lowering the variability within each group and but not changing the differences between the groups,.

Humans (Condition 1)				Squirrels (Condition 2)
	Con	Ob.	Mean	Diff	Diff2 			Con	Ob.	Mean	Diff	Diff2
	1	4	4.7	-.7	.49			2	4	3.3	.7	.49
	1	5	4.7	.3	.09			2	3	3.3	-.3	.09
	1	5	4.7	.3	.09			2	3	3.3	-.3	.09

Mean = 4.67	 						Mean = 3.33
Standard Deviation = .58 					Standard Deviation = .58	
Standard Error of Measurement: _____			Standard Error of Measurement: _____
value of t = _____