In many social projects, our purpose is to raise the well-being of certain abilities (e.g. reading or communication skills) of the beneficiaries. For the frontline workers, their interest is chiefly on whether the beneficiaries showed substantial changes in their life. Some beneficiaries would show a bigger change while the others showed little.
On the other hand, from a program perspective, the evaluation will usually be done to show if the program effect (i.e. the changes in peoples’ life) is statistically significant.
Sadly, these two perspectives are not entirely consistent with each other, and both speak in different perspectives and focus.
For a program to be statistically significant, there are usually three things to satisfy:-
The magnitude of changes is big enough, for example, a program aims at raising the speaking/reading skills of Cantonese of non-Chinese communities, the score of improvement before & after the intervention are measured and compared. The magnitude of change may be depicted as from an average of 30 to 65 out of full marks of 100.
The standard deviation of the data is small.
The sample size, i.e. the number of data, is big in order to prove that the changes are the result of direct intervention and is not of randomness.
All sounds good except for one. For instance, if you raise the speaking/reading skills of the beneficiaries from a score of 30 to 31, and provided that the sample size is very very big, one can still get a statistically significant result to reach a conclusion to the program effectiveness.
Nevertheless, a one-point improvement (from 30 to 31) is unable to give the beneficiaries a meaningful change regarding their inclusion in the Chinese community, i.e. practically there is no change in beneficiaries’ daily life. As such, it would be wise for the frontline worker to show the program effect in a different way. He/she would focus on the magnitude of change is significant and propose that a meaningful change in score shall be at least 20 or more in order to create practical change in peoples’ life. Effect Size is designed to show this perspective, which does not require a large sample size but needed a big magnitude of change and a small standard deviation.
Alternatively, the frontline worker can show the percentage of beneficiaries that had shown an improvement of 20 marks, in order to demonstrate the success of the program. Such presentation will be more relevant to funders or other stakeholders and is easier for laymen to digest, especially with the help of a graph.
The academic world has its own logic and the use of statistical significance, especially for proving the policy effectiveness or for programs that run on a large scale population, carries its importance. However, this does not preclude the use of other indicators or methods to show the practical changes in people’s life.
Chief Instructor and Financial Analyst
Ted Kwan