Technology in the classroom is becoming a ubiquitous force in the industry, and educators are all too aware of pros and cons. But the latest trend, online testing, has presented some new logistical and pedagogical challenges, especially as states have begun mandating the swift movement away from pencil and paper.

For some schools the technology has helped them advance towards School 2.0; but for others, the difficulties in navigating ed tech vendors and their assessment products has set them back. As K-12 leaders become obligated to administer tests online, what challenges should they be expecting, and how can they make the transition smoother? 

1. Navigating the flood of testing ed tech vendors can be tricky — invest in good vendor relationships

A handful of major testing companies, such as McGraw Hill and ACT, have a handle over the market in-terms of wide scale state testing. And already, many states have experienced situations where they had to get out of contracts with one of these companies to go for another. 

For instance, last year Tennessee had to cancel its statewide contract with a testing vendor because it consistently failed to meet deadlines. In an email to Education Dive, Commissioner McQueen explained that finding the "right fit" for the state has been a challenging journey. 

"Building a technology platform that is able to deliver the very high performance, scalability, and security profiles required for a state-wide, online administration is difficult and requires extensive software engineering skill.  It has been our experience that relatively few vendors in the industry have attained the level of engineering ability needed to do this successfully at the scale of a state. A solution that might work adequately in a district of 100,000 students will not necessarily work in a state with 1,000,000 students," wrote McQueen. 

At the state level, she writes that TN has worked to fix this issue by making sure the vendor meets its requirements. "TN currently has a vendor that has made the necessary investments in software engineering to ensure their platform meets our requirements.  Successful online administration this past school has produced confidence for significant online scale up this school year," she wrote. 

But beyond the state level, schools are also seeing a marketplace open up for more formative tests. Bob Schaeffer, public education director at FairTest, a research group that advocates for fair testing practices, says that navigating this market place can be a challenge, as finding a vendor that fits the needs of the district is critical to whether it will be successfully implemented. And, relying on the vendor to figure that out is not a viable strategy.

"There are a bunch of newer players in the field who have been able to gain marketing advantage, in part because of the old standardized companies poor performance [...] and from Fair test's perspective, there is no accountability for many companies...," said Schaeffer.

"I've talked a lot with superintendents, and their career paths is not one that gives them much of a technology background, so they often rely on on an assistant superintendent who has more understanding of that world and have to navigate the wild west of computerized assessment products," he said. "[Administrators can] consult with peers who have gone through the process so that [they] aren't trying to navigate the world of vendors from scratch." 

Fiona Hollands, senior researcher at the Center for Benefit-Cost Studies of Education at Teachers College, Columbia University, who has researched the world of ed tech vendors says that another method of staying ahead of this challenge is to focus on building partnerships. 

"It's a two way relationship [...] vendors shouldn't just be looking to make a sale, but on the other side of this, the educators need to understand its very costly for vendors to customize their products for them," said Hollands, "but if they have a really big client or want a long term relationship with someone it may be worthwhile." 

"They are more in the mind of creating partnerships — educators give ideas on how the product could be more useful in the long term, and the vendors doing the development," she said. "And, from the decision maker side they shouldn't get stars in their eyes ... they need to sit down and think about how is this going to be help me achieve our goals." 

2. Ed tech can come with technical complications — consider infrastructure and give feedback

Testing by its nature can be pretty stressful, and proponents of computerized testing often argue that digital platforms can streamline the process through benefits like fast student outcome data, interactive components, and personalized learning options. Doug Levin, president of EdTech Strategies, LLC, says that these benefits, among others, is part of the reason why the shift to online testing is happening and isn't likely to go away. But he also says that such advantages can be sidelined by technical and operational issues, which many states have already faced. 

"Not every school district has the technology and infrastructure to deliver online testing, so when they are falling short where or feel like they don't have enough, there's always a question of 'whose budget does that come out of?'" said Levin. "In some states have said this is a local responsibility you have to find some way to figure it out — in other cases you see states themselves make big investments in school district technology to help districts come up to speed." 

And due to the lack of proper infrastructure to handle testing technology, added complications in the form of technical glitches, data security, and crashing computer screens can add an unexpected layer of anxiety during an assessment. In fact, Fair Test tracked the number of states from 2013-2016, and found that at least 35 states had experienced either computer glitches during online assessments, or even widespread disruption of test administration because of technology failure — a statistic which means that in the overall shift to online assessments, school districts ought to be preparing themselves for the likelihood of a technological issue. So how can districts stay ahead of this? 

Levin mentions a number of considerations district administrators should have at the top of mind: whether there is actual space to fit large numbers of kids to test on computers or devices on the same day, servers that handle enough bandwidth for large amounts of data exchange, and whether the hardware is modern enough to manage the testing software. Before buying into a technology for various assessments, ed tech decision makers should have these factors at the top of mind. 

Of course, in instances where states are mandating tests, schools must also work with legislators to communicate what is actually possible for them. Schaeffer explained that often times lack of proper equipment becomes an issue for schools with few resources, which are typically those already serving low income kids in rural and urban areas of poverty. Already, a number of schools have already petitioned states to use pencil and paper and phase out compliance, so when it comes to avoiding serious impacts of technical difficulties — it's clear that school feedback is important. 

"The process of testing is a year long endeavor, and what we've seen particularly from more leading states is that there's almost an annual cycle of conversations that have to happen with school districts," said Levin. " There needs to be conversations largely with school districts technology staff on what school districts are going to need to deploy, on what their broadband does, what the server is going to do..."

"In the best circumstances there's a dialogue between school districts, technology leaders, state staff and the vendor about what the needs of district are." 

McQueen, for example, explained that district administrators were open to online testing, when they were given options and time to experience the technology. 

"During this past school year, we made online assessment optional for high schools, and while many districts opted for paper at the time, the most noteworthy quote coming from a “paper” district reflecting on the experience of districts who opted for online, was “if we had known online was going to be this simple and trouble free, we would have opted in!”

3. Technology may test technical literacy skills, not outcomes — train students before assessment day

"Some students do better on paper than they do online, and it's probably because they aren't familiar with the technology... which is why its important to do a trial run," said Hollands. "If you are testing online you have to have decent computers to use on a regular basis." 

Partnership for Assessment of Readiness for College and Careers examined performance on eight of their assessments in mathematics and English language arts/literacy categories for students taking them on non-tablets and tablets. The data collected showed that there were no "large or consistent differences in assessment results from tablet and non-tablet administrations," meaning that if student's aren't faring well, there are external factors educators should be considering, like the one Hollands mentions. 

For example, Research from the Council of Chief State School Officers found that device differences, and student difficulties to adapt to them, made it difficult to fully assess testing results, because many had problems operating the actual device — screens that needed scrolling made it challenging to read large passages, and choosing answers through touching the screen was challenging to write precise answers. 

For instance, Indiana also dealt with significant technological setbacks in testing last year. Adam Baker, press secretary for Indiana Department of Education, said that in terms of finding the right technology to meet student needs and also meet testing goals often depends on how students are interacting with technology generally. 

"Students engage well in the assessment environment as they engage more with technology in day to day life.  However, there are continued challenges with aligning instruction and assessment," he wrote. 

To deal with this issue, Levin suggests that school leaders take steps to train students and prepare them as the shift to online testing happens inevitably, and concerns about students' technology literacy skills grow.  

"There has been some research that suggests students have less familiarity with using those computers, perform less well on their tests. It's not because they don't know the content, its because they aren't familiar with teh technology," said Levin. "So, there has been a big push to make sure that students have the opportunity to use that technology outside of testing so they are familiar with how the mouse looks, the keyboard layout, they have some ability to type a certain number of words per minute that are relatively correct or not."

"Students that don't have those experiences, we find, don't do as well at measuring their academic skills, instead of measuring their technology skills."