How should districts setting out to find a new ed tech device or product go about assessing its feasibility? Should more attention be paid to a product fitting into a budget, or is it the other way around? The solution may lie in cost-benefit analysis.

According to Fiona Hollands, associate director of the Center for Benefit-Cost Studies of Education at Teachers College, Columbia University, the first steps should be practical and pragmatic.

First, the district should be absolutely clear about what it is trying to achieve with the new device or product, she says. It’s important to establish what would be considered to be a successful outcome resulting from its use. Second, district officials should ask colleagues in other districts, as well as potential vendors, a series of questions.

Holland recommends asking the following:

• Is there any evidence that the product/device can help achieve the district's stated goal(s), and if so, what is the necessary "dosage" and mode of use?
• What are the costs of the product/device for the quantity needed by the district? 

Then, more practical questions directly related to the implementation are appropriate, Holland says.

• What are the technical requirements for hardware, software, wireless vs. wired Internet access, bandwidth, etc.? 
• What are the on-site technical or pedagogical support requirements and what support is provided by the vendor? Is the latter at an additional cost or part of the purchase price?
• What training is needed for district personnel, including technical support staff, administrators, and instructional staff? 
• Are the current district facilities (classrooms/offices/library etc.) capable of supporting use of a new device? For example, are there enough electrical outlets to allow students to recharge devices throughout the day without tripping over a mass of cables, or will the devices last throughout the school day without charging? 
• How often do the devices break/need upgrading? What are the common obstacles encountered by existing users?

“Answers to these practical questions should inform the district as to how easy or difficult it will be for the product to be assimilated into existing practice at its schools,” Holland says.

Mark Ray, the current chief digital officer for Vancouver Public Schools (VPS) in Washington state, is a former teacher librarian and the 2012 Washington state Teacher of the Year. In 2015, he was named one of the National School Boards Association's "20 to Watch" educational technology leaders.

At VPS, Ray oversees the district’s 1:1 technology initiative in addition to its digital transformation. In 2013, a $24 million technology levy was approved by voters, and all VPS students in grades 3-12 were mandated to receive a device by the year 2019. The funding also supports professional development, infrastructure support, computer lab upgrades, standard classroom equipment, and digital content and resources.

Ray says that all of the district's digital content and device purchases are driven primarily by instructional needs.

“While we evaluate if and how a tool will function within our environment, we do our best to structure decision-making around what’s best for kids and will support educators,” he noted. “Our goal is not simply to put devices in students’ hands, it is to enable innovative instructional practices and prepare students for college, careers and life.”

When choosing a device, the district looks at cost, durability, support, management and usability.

“Utilizing pilots and scaled deployments, we have been able to “go slow to go fast,” learning from both students and teachers,” Ray explained. “This has helped ensure that we have sufficient support, management and training in place. A significant percentage of our technology investment has been allocated to professional development and support costs to ensure success.“

Tools to help evaluate ed tech options

A tool developed by the company Digital Promise also aims to help facilitate decision-making around ed tech purchasing. The tool was designed to help assess effectiveness by evaluating various studies related to the device, product, or platform.

Administrators can use the tool by entering their “best guess” response related to the product, filling out four sections consisting of just three questions each: product information, study relevancy, study source, and study design. Depending on the answers, the tool will advise as to whether a pilot period is recommended for the school.

If a pilot is recommended, Digital Promise also has a sample plan available free of charge.

When it comes to whether a new ed tech tool or platform should fit into a budget, or if a budget should be tailored to the implementation of a new ed tech device, Hollands says that it largely depends on whether the district's current priorities skew toward improving student outcomes or are focused on cost reduction. 

“In either event,” she explained, “the district needs to decide what would be good value for money. If, for example, the district cares most about improving reading skills for elementary students, it should consider a range of products that address reading skills and pick the one it has reason to believe will yield the greatest improvement in reading.”

Still, she said, costs need to be compared.

That’s because two similar products that are expected to have similar results need to be evaluated in terms of economics.  

One of the projects Hollands is involved with is CostOut, an online tool that helps decision-making by clarifying and comparing the costs, cost-effectiveness, and physical resource demands of implementing different programs or products.

“The idea of developing a free online tool that can be used widely for cost and cost-effectiveness analysis came from the Institute of Education Sciences (IES) at the U.S Department of Education,” Hollands explained. “IES has been trying to build capacity for both researchers and education decision-makers to improve resource allocation decisions by considering the costs of educational programs and products, in addition to whether they improve outcomes for students.” 

Since its October 2015 launch, the tool has already amassed 450 registered users who include education researchers, evaluators, consultants, program analysts, administrators from school districts, state education departments, and executives from companies offering educational services.

Districts must also weigh is whether a new product will require a reshuffling of resources.

“It's important to consider from where resources are being diverted to make sure that some other program area or student outcome will not suffer unduly as resources are channeled towards the new product/device,” Hollands said.

And of course, teacher and community buy-in is a must.

“While buy-in is not something that is easily quantifiable,” Holland said, “if it's missing, results will surely suffer. Specifically, if teachers and parents don't buy into an ed tech product or device, its unlikely that they’ll use it as intended or encourage the students to do so, so the chances of achieving the expected outcomes for students will be reduced.”

To that end, it’s critically important for administrators to convey the district's vision and goals to parents, community members, and school staff, and explain how the new product or device will help achieve goals. 

“Community and teacher buy-in are critical to the success of ed tech decisions,” Ray opined. “We’ve engaged staff members, students, parents and community members in our long-term strategic planning.”

At VPS, community engagement happens through various ways, including technology showcases, articles, a website, TV shows, social media and a specific informational campaign aimed at boosting digital citizenship.

“As we deploy devices throughout the district, teachers and principals receive their devices in advance of the rollout to students,” Ray said. “This helps teachers and principals build confidence and capacity in how the tools can be used in the classroom.”

And the tech plan at VPS isn’t cast in cement. Instead, the district’s IT team has adopted what Ray calls “an ethos of iteration.” That means they’re “continuously evaluating decisions, current practices and emerging trends,” Ray explained.

“While we have a multi-year plan for deployment and implementation of digital tools, we also are learning lessons and changing plans to respond better to student and teacher needs,” he said.

Would you like to see more education news like this in your inbox on a daily basis? Subscribe to our Education Dive email newsletter! You may also want to read Education Dive's look at what districts should keep in mind when weighing new federal testing guidelines.