Research Study on English Islands' Automated Literacy Tutor

Executive Research Summary
Study Period: May 1-31, 2024
Purpose
This study examined student performance before and after a one-month implementation of English Islands across four student groups. The primary goals were to determine:
• Whether student assessment scores improved following the intervention
• Whether the number of terms practiced on English Islands was associated with greater improvement
• Whether students could demonstrate learning of material practiced independently through the platform
The study included 56 students across Grade 1, Grade 2, Grade 3, and SpEd. Participants were instructed to use English Islands a minimum of three times per week, for 10 minutes per session, during the May 1-31, 2024 study period. The analysis of student engagement is based on the number of terms practiced, which was directly recorded by the platform.
Assessment Method
Students completed the same grade-level assessment immediately before and after the intervention period. Both assessments were administered through English Islands, and student responses were scored using automated speech recognition under the same scoring process.
The post-intervention assessment was identical to the pre-intervention assessment. However, students could not access the assessment during the intervention period, preventing them from repeatedly practicing or reviewing the assessment itself between administrations.
Test 1 represents the pre-intervention assessment, and Test 2 represents the post-intervention assessment.
Executive Summary of Findings
The results showed a clear and statistically significant improvement in student performance following the intervention. Across all 56 students:
Average Test 1 score: 81.66
Average Test 2 score: 87.18
Average improvement: +5.52 points
Percentage improvement: +6.8% increase
Of the 56 students:
• 43 students improved, representing 77% of the sample
• 2 students remained unchanged
• 11 students recorded slightly lower post-intervention scores
The overall improvement was highly statistically significant (p < .001). The estimated 95% confidence interval for the average gain was approximately +3.60 to +7.43 points. The standardized effect size was Cohen's dz = 0.77, indicating a substantial pre- to post-intervention difference.
.png)
Results by Grade
Grade 1
Students: 17
Average Test 1 score: 74.65
Average Test 2 score: 81.88
Average gain: +7.24 points
Statistical result: p = .0119
Grade 1 demonstrated a statistically significant improvement.
This group is particularly important for interpreting the study because the students advanced into curriculum content that they had not previously learned and were not being taught in their regular classroom during the study period. Their practice of this new material occurred through English Islands. As a result, the Grade 1 findings provide promising evidence of students' ability to learn new material independently through the platform.
This does not replace the need for a formal control group and does not, by itself, establish causation. However, because the material was not concurrently being taught in class and students practiced it through English Islands, concurrent classroom instruction is a less plausible explanation for improvement on that material.
Grade 2
Students: 17
Average Test 1 score: 83.24
Average Test 2 score: 88.76
Average gain: +5.53 points
Statistical result: p = .0011
Grade 2 demonstrated a strong and statistically significant improvement.
Grade 3
Students: 19
Average Test 1 score: 87.00
Average Test 2 score: 91.00
Average gain: +4.00 points
Statistical result: p = .0007
Grade 3 also demonstrated a strong and statistically significant improvement.
This result is noteworthy because students began with an average score of 87.0, leaving comparatively less room for additional improvement.
SpEd
Students: 3
Average Test 1 score: 78.67
Average Test 2 score: 84.00
Average gain: +5.33 points
Statistical result: p = .323
The SpEd group showed a positive average improvement similar in magnitude to the other groups.
However, because only three students participated, the sample was too small to determine whether the observed improvement was statistically significant.

Student Engagement and Terms Practiced
The study also examined whether students who practiced a greater number of terms on English Islands tended to demonstrate greater improvement. Across the full sample, there was a statistically significant positive relationship between terms practiced and percentage improvement.
Pearson correlation: r = .297
Statistical significance: p = .026
Students who practiced more terms therefore tended, on average, to demonstrate larger percentage gains. A simple regression analysis estimated that an additional 100 terms practiced was associated with approximately 5.9 additional percentage points of improvement.
The relationship accounted for approximately 8.8% of the variation in percentage improvement.
This should be viewed as evidence of an association rather than proof that practicing a specific additional number of terms will produce a particular increase in scores. Other differences among students may also contribute to outcomes.
Home-Based Use and Engagement
The Grade 1 implementation also differed from the other classes in an important way.
Because students did not have convenient access to classroom devices, Grade 1 students used English Islands at home rather than during regular classroom use. Despite this, Grade 1 students practiced substantially more terms than students in the other groups:
Grade 1 average: 238.1 terms practiced per student
Other three groups combined: 136.3 terms practiced per student
This means that Grade 1 students practiced approximately 74.8% more terms per student than students in the other three groups combined. This finding is noteworthy because it suggests that students were able to engage extensively with English Islands outside the classroom environment.
It should not be interpreted as evidence that home use itself causes greater engagement. Grade 1 differed from the other groups in several respects, so the study cannot isolate the effect of home versus classroom use. However, the results demonstrate that substantial independent use of the platform was achievable in this implementation.
Results Among the Most Highly Engaged Students
The relationship between engagement and improvement was particularly visible among the students who practiced the greatest number of terms. The highest-engagement students, representing approximately the top 10% of the sample, achieved an average individual percentage improvement of: 19.3%
This figure represents the average of each student's individual percentage improvement.
All of the highest-engagement students were in Grade 1. As a result, the high-engagement finding cannot be fully separated from other characteristics of that class and should be regarded as descriptive evidence rather than a controlled comparison.
Evidence of Independent Learning
The Grade 1 results provide an additional form of evidence that is not captured by the overall pre- and post-test statistics alone. Students in this group moved ahead into material that:
• They had not previously learned
• Was not being taught in their regular classroom during the research period
• Was practiced solely through English Islands
The students also used English Islands at home rather than relying on classroom device access.
Their statistically significant improvement, combined with substantial practice of new material through the platform, provides promising evidence that students were able to acquire new knowledge through independent use of English Islands.
This finding does not have the evidentiary strength of a randomized controlled study. Nevertheless, the absence of concurrent classroom instruction on the new material makes the Grade 1 result particularly relevant when evaluating English Islands' potential to support independent learning.
Methodology and Statistical Interpretation
The primary analysis used paired student data, meaning each student's post-intervention performance was compared directly with that same student's pre-intervention performance.
For the combined sample:
t(55) = 5.77
p < .001
95% confidence interval for improvement: +3.60 to +7.43 points
Because the distribution of individual score changes was not perfectly normal, a nonparametric Wilcoxon signed-rank test was also conducted. That test produced the same overall conclusion:
p < .001
This increases confidence that the overall improvement was not dependent on a single statistical assumption. The relationship between terms practiced and percentage improvement was assessed using correlation and regression analysis.
The Pearson correlation was statistically significant:
r = .297, p = .026
Considerations for School Board Decision-Making
The study provides encouraging preliminary evidence that English Islands is associated with improved student performance. Several findings may be particularly relevant to schools considering implementation.
1. Consistent Pre- to Post-Intervention Improvement
Students improved by an average of 5.52 points, and more than three-quarters of students improved individually. Statistically significant gains were observed independently in Grade 1, Grade 2, and Grade 3.
2. Evidence of Independent Learning
Grade 1 students demonstrated statistically significant gains while working with curriculum content that was not being taught in their classroom during the intervention. This provides promising evidence that English Islands may support students in learning new material independently.
3. Engagement Associated With Improvement
Students who practiced a greater number of terms tended to demonstrate greater percentage gains. The top 10% of highest-engagement students averaged 19.3% improvement.
4. High Engagement Outside the Classroom
Grade 1 students used English Islands at home because of limited access to classroom devices.
Despite this limitation, they practiced approximately 74.8% more terms per student than students in the other three groups combined. This suggests that the platform may be capable of supporting substantial student engagement beyond scheduled classroom use.
Important Limitations
These findings should be considered in the context of the study design.
No Control Group
The study did not include an untreated or randomly assigned comparison group. Therefore, the data establish that student performance improved following the intervention but cannot prove that English Islands was responsible for the entirety of the observed improvement.
The Grade 1 independent-learning context strengthens the plausibility that English Islands contributed to those students' gains, particularly because the relevant material was not being taught concurrently in class. However, it does not provide the same level of causal evidence as a controlled trial.
Selection Bias and Confounding
The study did not statistically control for selection bias. Students who practiced more terms may differ from students who practiced fewer terms in motivation, family support, attendance, baseline characteristics, or other factors. Similarly, the Grade 1 group differed from the other classes both in usage setting and in the number of terms practiced. The study cannot determine which of these differences contributed to the observed outcomes.
Same Pre- and Post-Assessment
Students completed the same assessment before and after the intervention. Students were unable to access the assessment during the intervention period, which limits direct practice of the test itself. Nevertheless, some familiarity from the initial administration cannot be completely ruled out.
Sample Size
The study included 56 students. Additional research with larger and more diverse samples would provide greater confidence in the generalizability of the findings.
Long-Term Retention
The study measured outcomes immediately following the May intervention period. It did not assess whether gains persisted over a longer period.
Conclusion
The May 1-31, 2024 study provides promising preliminary evidence of improved student outcomes following the use of English Islands. Across 56 students, average scores increased from 81.66 to 87.18, an average improvement of 5.52 points, with 76.8% of students improving individually.
The overall improvement was highly statistically significant, and statistically significant gains were independently observed in Grade 1, Grade 2, and Grade 3. The study also found a positive association between terms practiced and percentage improvement, suggesting that greater platform engagement may be associated with stronger student outcomes.
The Grade 1 results offer an additional finding of particular relevance. These students practiced new curriculum material through English Islands that was not being taught to them in class during the study period, used the platform at home, practiced approximately 74.8% more terms per student than the other classes combined, and demonstrated statistically significant improvement.
Taken together, these results provide promising evidence that English Islands can support both improved student performance and independent learning, including learning that occurs outside regular classroom instruction.
For school boards considering English Islands, the findings support further evaluation through a larger pilot or phased implementation, with continued measurement of pre- and post-intervention achievement, terms practiced, outcomes across student populations, and longer-term retention.
A future study incorporating a comparison or control group would provide stronger evidence about the extent to which observed gains can be causally attributed to English Islands.
.png)
