EdTech Integration Troubleshooting: Software Updates

Ponder is excited to have the Washington Heights Expeditionary Learning School (or WHEELS) as one of our K12 pilot schools. InsideSchools.org was right when they described their faculty as "dedicated." WHEELS is an impressive school. Yesterday, as Ponder's Founder and CEO, I went to WHEELS to help troubleshoot some challenges their teachers faced when using Ponder on school technology resources. I think the challenges they face are common for academic technologists trying to deploy new tools. Kudos to our friends at WHEELS for working through this rather than deciding it would be simpler to give up! Hopefully our experience yesterday can be helpful to all of you as you confront similar situations. 

Pacing school hallways around the world there are solitary school IT managers asking themselves "To Update, or Not to Update?"

In my experience, schools rarely have the IT support they need, and teachers are often left to deal with updates, security warnings, unnecessary toolbars, draconian web filters, and more interfering with basic uses of technology in class. These experiences illustrates the problems caused by not updating, while hopefully not ignoring why updates are annoying, and then offers some recommendations for schools and EdTech software developers.The faculty at WHEELS had their classes setup, had planned integration with the curriculum and their students had successfully registered their Ponder accounts in the classes - everything went as planned. Then they discovered that on the shared student laptops, Ponder's add-on wouldn't install. Instead they got the following cryptic message: "An error has occurred: There was a problem adding the item to chrome. Please refresh the page and try again. " See screenshot. Needless to say, refreshing the page and trying again had no effect. Ponder has run on various hardware and OSes for over 1,700 students at more than a dozen institutions, so it was most likely a configuration issue specific to their setup rather than a general bug. Remotely it was difficult to tell what was going on. Aside from the error message itself, we knew:

• Ponder was working and installing on the faculty computers without issue
• The school had separate student email accounts with Google

We were not sure:

• What hardware they had
• What OS version or Chrome version they were using
• Whether the machines were managed with a Google admin account
• Whether we would be able to load an "unpacked" or debug version of our extension on their machines to test

Tony and I brainstormed together the following possible explanations:

• Extension installation at the machine level completely blocked
• Chrome version so old that we didn't support it (<17)
• Google account management in place with restrictions that prevented our installation
• A bug in our client

For some reason I latched on to the idea that the school had simply disallowed the installation of browser add-ons for student accounts, but I got on the A train to 181st street with my laptop and a USB stick with:

• a link to google's simple "high contrast" add-on which we surmised had fewer requirements specified in it's manifest, and therefore might be allowed to install.
• a link to a fairly complex add-on that would probably trip more restrictive management settings - - if this worked, it might suggest that there was in fact a Ponder bug.
• The uncompiled "unpacked" source of the latest version of our extension
• A slightly older version of the Ponder extension that was missing some recent functionality (around extending PDF support) that we thought could conceivably cause a problem.

When I got there, I quickly learned the following:

• the school tech guy confirmed that the student's Google accounts were not managed, and that they didn't depend on students logging into chrome. This ruled out the google account level admin configuration issue.
• They were running on recent Apple laptop hardware.

I attempted the install of our client with the Chrome console open (command-option j on a mac) and saw the following "[INVALID MANIFEST]".We had seen this before, and it's Chrome's way of saying that the extension has a configuration element that it doesn't recognize. This immediately suggested that they were on an old version of Chrome.I checked, and indeed they were running version 20, and in the past we had seen issues when people had versions below 17, but the current version is 29, so that still seemed to be a likely culprit. The tech guy explained that the laptops were imaged by the NYC Department of Education before they arrived, which explained why it was so old. So, no big deal - we can just update them.I was instant messaging with Tony, and he had a great idea which I put into motion while we were waiting for the machines to update. I loaded the debug version of our add-on locally and got a more detailed error:Could not load extension from '/Users/student/Desktop/0.10.0.13.prod'. Invalid value for 'content_security_policy'.According to Tony's plan, I removed the line in our manifest for content_security_policy and re-installed. This time it worked without a problem! The "invalid value" error message suggested that their version of Chrome interpreted "content_security_policy" differently than the current version. Ah-hah!Unfortunately, Chrome wouldn't update, and gave us "error 12". I investigated a little bit, but the tech guy was more pragmatic - he simply downloaded the latest Chrome .dmg and ran the installer successfully. With the latest version of Chrome, Ponder installed successfully and they were on their way.

Problem solved! WHEELS was now back on track. ;-)

While WHEELS' students were now ready to go, this was obviously an important lesson to learn so that we could avoid this issue with other schools. When I got back to the office, I had two follow-ups I was curious about.First, earlier this week I had read an article on EdSurge explaining to teachers how to minimize logistical hassles when implementing blended learning. One bullet point that caught my eye was a pet peeve that I had flagged on Ponder to other EdTechers almost as a joke. It said "Avoid interrupted class time by making sure you turn off automatic updates on each of your computers." and recommended using Deep Freeze.

I wanted to write this blog post to shed light on the challenges this mentality creates - it isn't as simple as turning off updates.

Minor tangent alert: Unpatched machines mean security holes, and security holes lead to infected machines and infected machines infect more machines. Infected machines become part of Zombie Computer Botnets, which are used to steal people's identities, send spam, execute denial of service attacks, etc. Having worked on the Microsoft service that manages patching all Windows computers over the Internet, this is a bit of a sore spot for me, but I digress.Perhaps more relevant to this scenario, it also means that developers making software that works with Chrome will primarily be testing against the most recent version of Chrome. The likelihood of them testing against the last dozen versions of your browser is small because its a LOT of work. That said, because updating school laptops (which are closed and asleep on laptop carts when they aren't actively being used in class) is a logistical problem that I don't see a good answer for, one can't really fault the schools for struggling here.For school tech folks, some starting points that may or may not be useful: For Windows, check out WSUS (Windows Software Update Services) and System Center, for iOS the Apple Configurator might help, for Macs, OS X Server has NetInstall, though I don't know much about it.

As far as I know, none of these tools will open each of the laptops on your cart, patch them and then put them back to sleep for you.

The second thing I wanted to investigate is how, for the umpteenth time, we could try to find out ahead of time when Google makes changes that are not backward compatible, and better manage our release process. This is especially relevant to the software update issue, because

Google assumes the exact opposite of the EdSurge recommendation: that everyone is updating constantly all the time.

It turns out, the addition of the "content_security_policy" manifest field is called out on the "what's new" site, but at version 14! So something had changed about that field between version 14 and version 20, where the WHEELS laptops were, without any indication on this developer status page. And, now that I looked more closely, the page hasn't been updated since version 26, and now we're at version 29.What especially got me about this issue is that all extensions are installed through Google's servers. That means that Google's servers happily passed our extension to an old version of Google Chrome, which couldn't read the manifest, with ZERO FEEDBACK.

No feedback to the schools trying to install it, and no feedback to us, the developer, that our extension was failing installation on a certain version of Chrome.

Clearly it would be too much to ask that when we submit our app to the store they could preemptively validate which versions of Chrome it will work for. Here the gate-keeping mechanism in the Apple App Store at least has a concept of per-platform version.My recommendation to schools is a twist on the EdSurge recommendation: Update your computers on a predictable schedule. If your IT folks can prevent updates from popping in class, and instead take a "Patch Tuesday" approach to getting machines updated, maybe on a rotation - for example one cart per week, you will hopefully minimize short-term disruption and also avoid these larger compatibility issues.And, fellow Chrome developers, remember,1. Google assumes everyone is updating Chrome all the time.2. People, especially schools and businesses, hate updating because it is constantly interrupting them, and they see little value in it; generally they try to prevent it from happening. In short, schools are NOT updating Chrome all the time.3. Because of the previous two lessons, this is a royal pain for developers. Google should figure out how to make updating less painful and more pervasive in schools, and in the meantime, should work harder to help developers manage compatibility with their ever-changing versions. In the meantime, as a developer, be very aware of what APIs you are using and what you require in your install manifest - the less surface area for change, the better!(The cynic in me has to imagine that Google's education folks are focused more on addressing Chromebook deployment issues.)Hopefully by drawing some attention to these issues we can avoid them in the future. Good luck!