Developing A2i Spanish Adaptive Progress Monitoring Assessments for PK-3rd Grade

Developing A2i Spanish Adaptive Progress Monitoring Assessments for PK-3rd Grade is sponsored by IES - Institute of Education Sciences. Develops and validates adaptive progress monitoring assessments in Spanish for PreK-3rd grade students.

## [](https://pmc. ncbi. nlm.

nih. gov/articles/PMC10249657/)Abstract Bringing effective, research-based literacy interventions into the classroom is challenging, especially given the cultural and linguistic diversity of today’s classrooms.

We examined the promise of Assessment-to-Instruction (A2i) technology redesigned to be used at scale to support teachers’ implementation of the individualized student instruction (ISI) intervention from kindergarten through third grade. In seven randomized controlled trials, A2i and ISI have demonstrated efficacy. However, the research version of A2i was not scalable.

In order to bring A2i to scale in schools serving linguistically diverse students, we carried out the current study across two phases. This study represents both an exploration of what it takes to bring an educational intervention to scale (Phase 1) and a quasi-experiment on the literacy outcomes of learners whose teachers used the technology (Phase 2).

We integrated assessments of vocabulary, word decoding, and reading comprehension; revised the A2i algorithms to account for the constellation of skills English learners (ELs) bring to the classroom; updated the user interfaces and added new graphic features; and improved bandwidth and stability of the technology.

Findings were mixed, including several non-significant results, a marginally significant intent-to-treat effect on word reading in kindergarten and first grade for English monolingual students and ELs, and one significant interaction effect, which suggested ELs and students with less developed reading skills in second and third grade benefitted most from the intervention.

With some caution, we conclude that A2i demonstrates potential to be used at scale and promise of effectiveness for improving code-focused skills for diverse learners. ## [](https://pmc. ncbi.

nlm. nih. gov/articles/PMC10249657/)The Present Study – Purpose Statement The purpose of this effort was to describe the transition from the research version of A2i to a more generalizable platform that contained the needed components vital for improving student literacy outcomes.

We had to ensure that the A2i technology had the flexibility and stability for effective implementation in schools nationwide. In the present study, we report aspects of both an exploration of what it takes to bring an educational intervention to scale and a quasi-experiment on the literacy outcomes of linguistically diverse learners whose teachers used A2i.

Through this interactive process, we begin to establish evidence of consequential validity of the A2i technology. We present both aspects of scalability within Phase 1 and student level outcomes from the quasi-experiment within Phase 2 together because technology best improves education when it is considered in tandem with student learning rather than on its own (Hantula, 2019; McKnight, 2016).

Moreover, implementing at scale includes considering the populations that will be affected by the intervention as it reaches more students within classrooms. For example, ELs are more likely to be reached by an intervention as it spreads to more classrooms. Hence, this paper intends to serve as a description of the scalability process while also providing initial evidence of or promise for the effectiveness of A2i at scale.

We begin with presenting the theoretical frameworks that underlie the A2i research technology and briefly outline the features of the tool to provide a foundation for the current project. We then present a model drawn from the implementation science field that we used to guide our process for “scaling up. ” The project is organized across two phases.

Phase 1 is the Exploration Phase (2014–2015). Here, we outline the process and procedures of the exploratory work that provided the foundation for executing Phase 2. We also reflect on lessons learned during the implementation process that allowed us to identify barriers and enact responsive solutions to bringing a revised A2i to scale in kindergarten through third grade classrooms.

Phase 2 (2015–2016) is the Quasi-Experimental Phase. Here, we describe our process for developing valid, reliable, and adaptive literacy assessments integrated into the revised A2i technology using a linguistically diverse sample of students. We also present the procedures of and findings from the quasi-experiment.

We outline the Methods and Results of Phases 1 and 2 separately; however, we interpret our findings from both phases in light of the potential for national scalability. ## [](https://pmc. ncbi.

nlm. nih. gov/articles/PMC10249657/)Theoretical Frameworks Underlying A2i Technology The theoretical basis for the development of A2i was heavily influenced by the Simple View of Reading (Hoover & Gough, 1990), which outlines the importance of both decoding (code-focused) and language comprehension (meaning-focused) skills for successful reading comprehension.

This theoretical model posits that strong code-focused and meaning-focused skills are necessary for reading and comprehending text—without the development of both skills, reading comprehension is jeopardized. There has been extensive empirical evidence supporting the Simple View of Reading not only for monolingual English speakers but also for ELs (e.g., Florit & Cain, 2011; Kim 2017; Mancilla-Martinez & Lesaux, 2017; Proctor et al.

, 2006). This justified the recommendations of both code- and meaning-focused instruction provided by A2i for both monolingual English speakers and ELs. A2i has more recently been informed by the Lattice Model (Connor, 2016; Connor et al.

, 2016), which places instruction as a central force for change in students’ literacy learning. Aligned with Cronbach’s (1975) idea of aptitude by treatment interaction effects, the Lattice Model emphasizes that the effect of instruction depends on each student’s linguistic, text-specific, cognitive, and social-emotional skills (i.e., child characteristic by instruction interaction effects; Connor et al. , 2007).

In other words, the effects of instruction may differ based on students’ baseline skills across various developmental domains. Moreover, according to the Lattice Model, there are reciprocal or bi-directional effects such that, as instruction improves literacy skills, it also improves linguistic, cognitive, and social-emotional skills. At the same time, these developmental areas help to improve students’ literacy skills (Connor et al.

, 2016). This idea of students’ characteristics (skills) by instruction interaction effects on literacy, as supported by the Lattice Model, are the premise for individualizing student instruction. We next provide a brief overview of A2i.

We refer the reader to Connor (2019) for a full description of the A2i features. ## [](https://pmc. ncbi.

nlm. nih. gov/articles/PMC10249657/)Components of the A2i Technology – Overview of the Research Version ### DFI Algorithms and the Classroom View As supported by the Lattice Model, A2i provides the means for teachers to individualize instruction based on the characteristics that their students bring with them into the classroom, in this case, their literacy skills.

At the heart of A2i, and the premise for individualizing student instruction, there are dynamic forecasting intervention (DFI) algorithms. These DFI algorithms are patented (Connor, 2013) and developed from empirical studies (e.g., Connor et al. , 2004).

DFI algorithms compute recommended amounts (in minutes) of four types of literacy instruction that will optimize literacy gains based on individual student’s language and literacy skills.

The four types of literacy instruction include code-focused instruction with the teacher (e.g., phonological awareness, phonics, spelling, word fluency), meaning-focused instruction with the teacher (e.g., language, vocabulary comprehension, metacognition), code-focused instruction with peers or alone (e.g., phonics worksheets) and meaning-focused instruction with peers or alone (e.g., independent sustained silent reading, buddy reading).

With the right information about individual students, teachers can predict students’ potential trajectories as they learn to read, taking into account documented sources of influence (e.g., amount of literacy instruction, support from home) and constraints (e.g., previous achievement, home resources). The recommended amounts of instruction are displayed for each student in the _Classroom View_ of the A2i technology.

As students are assessed throughout the year, the calculated recommendations are automatically updated so that more recent information about students’ literacy skills is taken into consideration. The DFI algorithms used in the A2i technology have been tested for efficacy in multiple research studies (Al Otaiba et al. , 2011; Connor et al.

, 2011a; Connor et al. , 2013; Connor et al. , 2007; Connor et al.

, 2011b; Connor et al. , 2009). ### A2i Assessments and Graphs In the research version of A2i, we used standardized reading and vocabulary assessments, administered to students within their schools and entered into the technology by research assistants.

Once entered, A2i uses the scores in the DFI algorithms to compute the recommended amounts and types of literacy instruction needed for optimal growth. Each student’s assessment results and targeted growth over a one-year period as well as their instructional recommendations are then displayed for teachers within graphs. ## [](https://pmc.

ncbi. nlm. nih.

gov/articles/PMC10249657/)Implementation of A2i within Kindergarten–Third Grade Classrooms Although the research version of A2i provided a means for teachers to individualize student instruction, the tool was not feasible nor scalable for classroom use without support from the research team.

Previous studies examining the development and effectiveness of A2i have been grounded in design-based implementation research (DBIR)—to develop a tool in collaboration with practitioners that is by design, feasible and implementable (Connor et al. , 2015; Fishman et al. , 2004).

Our aim for this study, however, was that individualizing student instruction, using A2i along with a professional development (PD) protocol, be scalable. In the current paper, we draw from the Exploration, Preparation, Implementation, Sustainment Model (EPIS; Aarons, Hurlburt, & Horwitz, 2011; Moullin et al.

, 2020) to outline a set of practices and procedures for supporting the implementation of A2i within kindergarten through third grade classrooms with high percentages of ELs. We describe each area in the EPIS model below and contextualize our stages of implementation by drawing from experiences with our school partners across two academic years (2014–2016). Within the EPIS model, the stage of exploration (Odom et al.

, 2019) takes place at the level of an outer contextual factor (e.g., school districts) and an inner contextual factor (e.g., school administrators; Aarons et al. , 2011). In educational settings, these are the district leaders and school principals who make decisions about changes to instruction with which teachers will be tasked.

In relation to our project, we met with school principals prior to the start of the study in order to develop a common research objective. The leaders were tasked with implementing district-mandated Response to Intervention (RTI) within their schools, which included universal literacy screening and multi-tiered, targeted instruction.

Demonstrating how individualizing student instruction with the use of A2i aligned with RTI was the beginning of our mutual partnership, with the shared objective of supporting literacy gains in all learners, including ELs. Schools and teachers possess individual characteristics that vary.

During the preparation stage, initial training is provided to site-specific teachers in order to prepare the climate for implementation, ensuring that schools and teachers have what is needed to create change (Odom et al. , 2019). Researchers who work with teachers act as bridging factors or interconnections between research and implementation (Aarons et al.

, 2011). They must foster trust and “buy-in” of teachers. These teachers, in turn, work with their students to support classroom learning—they act as bridging factors between researchers and students.

While this shifting of roles may seem complex, it is in part due to the dynamic and reciprocal nature of implementation of change illustrated by the EPIS model (Aarons et al. , 2011). To understand the varying needs and experiences of our school partners, we interviewed school leaders and led workshops with teachers.

Our goal was to gather information about the school environment (access to computers and headphones, internet availability and bandwidth, class size and student characteristics, etc.) as well as individual experiences using technology and running flexible small groups. We used the information learned during this time to prepare the climate for implementation. We then created a roadmap of changes needed for successful scale up.

We designed an online professional development (PD) protocol that aligned with the needs of our school partners while also addressing critical components for using A2i to individualize literacy instruction within kindergarten through third grade classrooms. Implementation of an educational intervention positions teachers as learners (Odom et al. , 2019).

Teachers both provide information and receive feedback on implementation of an intervention, and in turn, use their new learning to change their practice. Fidelity of implementation is critical at this stage as teachers communicate feasibility concerns. In addition, the research team maneuvers or adjusts approaches for different teachers at different stages of “uptake.

” This might include teachers with different types of experience, degree of openness, and levels of trust that influence intervention implementation. We supported teachers’ implementation of A2i through personalized and continuous PD across the school year. We monitored and adjusted our approaches as needed to respond to individual needs, ensure uptake of new practices with fidelity, and facilitate change.

Sustainment can be understood in the context of bringing an educational intervention to scale as the continued implementation of an intervention that has been fully taken up by school sites in classrooms (Odom et al. , 2019). Sustainment occurs after researchers have fostered relationships, supported teachers in changing practices, and communicated findings (Aarons et al.

, 2011). Fostering relationships often begins at the exploration stage and continues throughout the stages. These linkages, as described by the EPIS model, often operate through human and institutional relationships (Aarons et al.

, 2011). In the case of educational interventions at scale, this would include relationships between teachers and principals, teachers and their students and families, researchers and teachers, districts and researchers, and various combinations of the aforementioned.

At the stage of sustainment, our goal was to give our school partners the tools they needed to continue implementing A2i school-wide without extensive support from the research team, while also maintaining a positive school-researcher partnership.

We therefore discussed their progress, shared findings from across the school year, and ensured that everyone (principals and teachers) continued to have access to A2i and the online PD protocol. We also offered continued technical support as needed and an open door for future communication and collaboration.

### Phase 1 (2015–2016): Research Objective and Methods To ensure effective, school-wide implementation of A2i, the primary research objective of Phase 1 was to explore thoroughly the process of scaling up. That is, we examined the transition between implementing the research version of A2i to a more generalizable tool.

In Phase 1, we recruited 24 kindergarten through third grade teachers and four principals (one per school site) from two large schools in Phoenix, Arizona (AZ) with substantial EL student populations and two schools in Pittsburg, Pennsylvania (PA).

At the start of the academic year, we carried out in-person structured interviews with the school principals from each site to gather information on the individual needs of their schools and establish a reciprocal school-researcher partnership. We inquired about district-level and school-level concerns and noted areas for potential collaboration.

Although the schools were tasked with different district-level charges, they shared the common goal of improving literacy outcomes in their early elementary students. We developed a year-long plan for partnership centered on implementing A2i in kindergarten through third grade classrooms to support individualized literacy instruction, while studying the process and gathering feedback from teachers.

The schools shared their beginning and end of year progress monitoring data (i.e., DIBELS), and the research team uploaded the scores to A2i per classroom. The school year started with a “kick-off” in-person training for teachers at each school site.

The training consisted of two half-day workshops in which we gathered information about the school implementation climate and the needs and experiences of individual teachers and grade-level teams. We also provided information regarding A2i as an evidence-based literacy tool, discussed the features of the research version, and assisted teachers in using A2i in their classrooms to individualize student instruction.

#### Monthly Communities of Practice Meetings. In addition, two classroom educators from our research team facilitated monthly grade-level communities of practice meetings (e.g., Bos et al. , 1999) at the AZ school sites only, as these schools were local to the research team.

We developed a working handbook, which included guiding questions and monthly topics (setting up your classroom, using A2i recommendations to drive instruction) to structure the meetings and facilitate discussion.

The monthly meetings followed a similar sequence across the schools and grade-levels, including a “check-in” period to inquire about strengths and concerns with individualizing instruction using A2i, delivery of content, and discussion with reflection. #### Classroom Observations.

In addition to these monthly communities of practice, the classroom educators from our research team observed each of the AZ teachers in their classrooms three times during the year (fall, winter, spring).

Specifically, we were interested in understanding whether and how teachers effectively used A2i to plan and deliver literacy instruction within individualized, small groups and differentiated learning centers for their diverse student body.

We assisted teachers as needed in understanding the A2i recommendations, creating individualized small groups and learning centers based on the A2i recommendations, and preparing the A2i recommended curricula materials and evidence-based activities. Finally, we carried out focus groups with teachers from each site to gather information on their experiences using A2i in their classrooms.

For the AZ schools, the teachers, research team, and program developers participated in focus groups (one focus group per site). In the PA schools, the research team met with teachers, gathering notes to share with the program developers at a later time. The focus group questions centered on teachers’ experiences with specific features of A2i.

We inquired, for example, about the A2i features teachers found most helpful and how easily they were able to navigate the tool as well as readability of tables and figures and usefulness of the A2i recommended materials and activities. This information was critical, as it helped to inform the updates we made to the A2i technology prior to Phase 2.

We collected detailed notes from the initial planning meeting with the school principals, the “kick-off” training, and the monthly communities of practice meetings with our AZ schools. We compared notes from the monthly communities of practice meetings across groups to outline similarities and differences between the different grade levels and schools.

In addition, we gathered field notes during the classroom observations and monitored teachers’ usage of A2i to support their students’ learning as a means for gauging fidelity. Finally, we iteratively reviewed the records taken from the focus groups, in which we elicited teachers’ feedback about their experiences using A2i.

Taken together, we identified four themes that we addressed prior to the quasi-experiment carried out during the 2015–2016 school year. We next outline barriers and solutions derived from the four themes. See Table 1 for a summary of this process.

## [](https://pmc. ncbi. nlm.

nih. gov/articles/PMC10249657/)Barriers and Solutions to Implementation – Redesigning A2i Technology ### Barrier and Solution 1, Effort from Research Team and Integrated Assessments Perhaps the most daunting barrier identified was the high level of effort required from the research team to administer, score, and enter the assessments that allow the A2i algorithms to make instructional recommendations for individual students.

As a result, we determined that A2i would need integrated assessments that students could take with relatively little teacher intervention. We realized that the assessments would need to be short enough for students to take multiple times in a school year, and they would need to provide reliable, valid estimates of students’ language and literacy skills.

The assessments would also need to be scored automatically, without researcher support. With this in mind, we developed three adaptive assessments validated for students in kindergarten through third grade that could be integrated into A2i: an online vocabulary assessment (Word Match Game [WMG]) and two reading assessments (Letters to Meaning [L2M] and Reading to Comprehension [R2C]).

Details on item development and psychometric properties are reported in Table 1 and in the Method section. ### Barrier and Solution 2, User Interface and Improved Lesson Plans The second barrier was related to the user’s experience of the user interface (i.e., how easy A2i was to navigate and use).

Teachers and administrators reported wanting additional information about the lesson plans, specifically how they related to the Common Core State Standards (CCSS; Common Core State Standards Initiative, 2010) and better tools to visualize teacher usage of A2i and student progress across the school year.

To be responsive to these requests, we improved and expanded the lesson planning feature, which was used to facilitate automatic lesson planning for the implementation of individualized instruction in the classroom. Specifically, we included search and navigation menus, a wider curriculum selection, indexed curriculum activities linked to the CCSS, and recommended open-source materials linked directly to the lesson plans.

We also included enhanced reports for student progress and teacher usage, improved reporting features as well as added more web-based PD resources. See Table 1 for details and Appendix A for screenshots. ### Barrier and Solution 3, Recommendations for ELs and Updating the A2i Algorithm A third theme that emerged from the data was teachers’ desire to understand how to interpret the A2i recommendations for ELs.

The initial studies that demonstrated the efficacy of A2i were conducted in areas that had a diverse cultural and racial makeup, but they were not diverse linguistically. Considering the growing number of ELs attending elementary school in the United States, and the fact that the teachers involved in Phase 1 of the study were in AZ and PA, it is not unsurprising that this issue arose.

Having an intervention that scales up means having an intervention that works for all students, including students from culturally and linguistically diverse backgrounds.

Although scholars of effective instruction for ELs call for more research on modifications to classroom instruction for ELs, they have identified several strategies that are advantageous to literacy development including, individualizing (or differentiating) instruction (Gunn et al. , 2000; Kamps et al.

, 2007), providing ongoing teacher support and student monitoring (Haager & Windmueller, 2001), identifying similarities and differences between students’ first and second languages (Giambo & McKinney 2004; Kramer et al. , 1983), and capitalizing on first language strengths (August et al. , 2014; August & Shanahan, 2010).

A number of classroom-level intervention studies that have focused on ELs have also shown positive effects in enhancing students’ language and literacy skills (e.g., Cheung & Slavin, 2012; Collins, 2014; Dianda et al. 1995; Calderón et al. , 1998; Vaughn et al.

, 2005). Drawing from this evidence and from the Simple View of Reading framework, we concluded that individualizing instruction using both code- and meaning-focused instructional recommendations from A2i would be appropriate for ELs, but we considered the need to revise the A2i algorithms to accommodate ELs’ unique constellations of skills.

Given that the integrated A2i assessments were developed to measure literacy skills in English, we re-evaluated the appropriateness of the algorithms to make instructional recommendations for ELs (who were receiving English-only instruction) based on their current literacy skills in English.

The information that feeds the algorithm for recommendations related to time spent in meaning-focused instruction is pulled from student performance on the vocabulary assessment (for kindergarten and first grade) and from the reading comprehension assessment (for second and third grade).

ELs with limited oral language proficiency in English would be expected to score lower than children with higher levels of English oral language proficiency on these assessments, which would lead the algorithms to recommend more time in teacher-managed, meaning-focused instruction.

Increased time in small-group instruction that supports oral language development aligns with recommendations within the existing literature related to how best to support ELs in the classroom (e.g., August et al. , 2016; August et al. , 2018; Baker et al.

, 2014; Crevecoeur et al. , 2013; Gersten & Baker, 2000; Gunn et al. , 2000; Shanahan & Beck, 2006).

We recognize, however, that more precise recommendations could likely be made by incorporating both English and native language skill—this is a direction of future work.

When considering the A2i algorithm’s recommendations for teacher-managed, code-focused instruction, we explored whether to base this recommendation solely on word reading skills (as had been the case with previous A2i studies among English-only students) or to include vocabulary scores so that students with lower levels of vocabulary would receive recommendations for larger amounts of teacher-managed, code-focused instruction.

Our rationale for ultimately altering this algorithm to include both word reading and vocabulary skills was that students with less developed English vocabularies would benefit from spending relatively more instructional time with the teacher where they would be most likely to receive explicit, code-focused instruction tailored to their individual needs.

Again, we based this conclusion on theory as well as the literature related to best instructional practices for ELs (e.g., Baker et al. , 2014; Cunningham & Stanovich, 1997; Ouelette, 2006; Perfetti & Hart, 2002; Scarborough, 2001; Thomas & Sénéchal, 2004). See Table 1 for additional information and further rationale.

### Barrier and Solution 4, Bandwidth The final barrier was identified as a result of teacher reports of occasional slower-than-normal response times from the website, which the research team identified as being related to times when website traffic was high.

To address the increase in traffic inherent in scale up, the infrastructure of the servers, codebase, and internal data tables were enhanced to account for additional users without reducing performance. To reduce traffic on the main website, a protocol was also developed to enable students to access the online assessments directly, without having to navigate A2i. See Table 1 for further detail.

### Phase 2 (2015 – 2016): The Quasi-Experimental Phase – Research Objectives Phase 2 aimed to test whether our revised, scalable version of A2i demonstrated promise of effectiveness when implemented by elementary school teachers serving both English monolingual students and ELs. There were three research questions in this quasi-experiment. 1.

What is the validity of the newly developed, integrated A2i assessments that are embedded within the A2i technology? 2. What effect does teachers’ use of the revised A2i technology, with on-going professional development (PD), have on students’ literacy outcomes (intent-to-treat)?

Does the effect of A2i depend on students’ initial language and literacy skills? Does the effect of A2i depend on whether students are monolingual or EL? 3.

Controlling for pre-intervention reading scores, are post-intervention reading scores higher for those students whose teachers spent more time using the A2i technology? 1 To what extent does teachers’ use of the revised A2i technology, calculated from user logs (treatment teachers only), predict students’ reading outcomes? Does this vary by students’ monolingual or EL status?

Is teacher use of A2i related to PD uptake? ## [](https://pmc. ncbi.

nlm. nih. gov/articles/PMC10249657/)Method ### Transparency and Openness Statement This research was conducted following a grant proposal funded by the Institute of Education Sciences (IES; Grant # R305A160404), which pre-specified the research questions, theoretical framework, implementation strategy, data collection, and analysis plan.

As an IES Development Grant, there was no requirement for public release of data, and the IRB protocol and consent forms for this study do not allow for sharing data with third parties. Data analyses were conducted using HLM7 and SAS 9. 4; data analysis code is available from the authors upon request.

Selected materials from the study (e.g., the implementation fidelity rubric) are also available from the authors upon request. A2i is now a commercial product, and the authors include a conflict of interest statement printed elsewhere in this article.

During the 2015–2016 academic year, we conducted a quasi-experiment to assess the promise of the effectiveness of using A2i to support teachers as they individualized their students’ literacy instruction. Two large schools in AZ were randomly assigned to either use A2i at the beginning of the school year (immediate treatment) or to wait until April of the school year (delayed treatment). The school year for both schools ended in June.

Both schools used the same curriculum: Wonders, published by McGraw Hill (12/program/microsites/MKTSP-BGA07M0/wonders. html). The Wonders curriculum was indexed (embedded within A2i) so that teachers could access recommended lessons from the A2i _Lesson Plan_ based on their students’ grade level and reading ability.

Thirty-three kindergarten through third grade teachers and their students (_N_ = 763) participated in the quasi-experiment. There were four or five classrooms per grade level at each school. Sixty-eight percent (68%) of the participants qualified for the US National School Lunch Program (NSLP), which is frequently used as a proxy for socio-economic status.

Eighty percent (80%) of the students were Hispanic/Latinx, with 25% designated as ELs. In this district, students identified as non-proficient English Learners (ELs) were assigned to an English immersion classroom (EL classroom), with one EL classroom per grade level per school. EL classrooms had a dedicated four-hour English language block to support English language development.

This four-hour block was at the academic expense of other content areas, with mathematics as the exception. ### Professional Development All participating teachers across both treatment conditions received professional development (PD) delivered by educators (certified teachers or classroom specialists) on the research team. However, the PD protocol varied by treatment condition.

The teachers in both conditions participated in two half-day workshops prior to the beginning of the school year, but only the immediate treatment condition was given access to A2i at this time. With access to A2i, they were able to access the online PD materials and utilize all of the A2i features (_Lesson Plan, Classroom View_, etc.).

In addition, the teachers in the immediate treatment condition received personalized coaching in the classroom three times per year and monthly grade-level communities of practice meetings. In the delayed treatment condition, teachers were given access to A2i starting in April. Students were administered a battery of well-established, valid, and reliable standardized literacy measures as well as the A2i online literacy assessments.

For both conditions, all assessments, excluding the A2i online assessments, were administered in the fall (between August and September depending on classroom schedules) and again in the spring (April).

Students in the immediate treatment condition completed the A2i online assessments in the fall and spring; Students in the delayed treatment condition completed the A2i assessments only in spring, just before their teachers began using A2i since accessing the assessments required access to A2i. The spring assessment scores represent the outcome measures for the quasi-experiment.

In addition, as a measure of implementation fidelity, we monitored teachers’ A2i usage through user-logs and gauged teachers’ PD uptake using a researcher-developed rubric. #### Standardized Literacy Measures ##### Woodcock-Johnson III Test of Achievements (WJ-III). The WJ-III (Woodcock et al.

, 2001) is a standardized assessment, normed on a nationally representative sample that measures a wide range of students’ cognitive and academic abilities. The Letter-Word Identification subtest (LW) was used to assess kindergarten and first graders’ ability to name and decode words out of context. Research personnel administered the LW subtest individually to students in a quiet area outside the classroom.

Reliability on the subtest in the students’ age range varied from . 93 to . 98.