Knowledge Center

By Michael Perretta, co-creator of Docket®

“It takes 20 years to build a reputation and five minutes to ruin it.” – Warren Buffett

Taking a “Shot” at Immunization Records

There wasn’t a noticeable demand for immunization records back in 2017, but there also wasn’t a modern digital solution. Everybody needs their immunization records at some point and they can be onerous to collect, especially for folks that have had multiple health care providers over the course of their care. My team and I figured this could be an opportunity for us to pivot our bootstrapped company and attract more users to our platform. I pitched Docket® at various public health conferences across the country for 18 months before ultimately identifying Utah as our first official state partner in July 2019.

The Utah Department of Health launched Docket® in May 2020, months before the COVID-19 vaccine would be publicly available. With our focus on childhood vaccines and back-to-school enrollment, my team and I could not have anticipated the extent to which Docket® would be in the public eye. I would never have believed that the Utah Jazz, for example, would promote Docket® to their fans during gameday. Docket® had less than five thousand active users at the end of 2020. Today, that number is over two million across multiple states.

Crunch Time

The Vaccine Credential Initiative was founded in 2020 as a “supergroup” of both public and private sector stakeholders tasked with standardizing COVID-19 vaccine credentialing. The result of their work was the SMART Health Card framework which could produce cryptographically-signed QR codes. By October 2021, some of our public health partners reached out with a request for Docket® to support the new standard. Meanwhile, an editor at a well-known tech blog had been working to reverse-engineer the platform. He emailed me at 7:41 AM EST on October 26 regarding a potential bug where an authenticated (but not authorized) user could guess user ID numbers via API to indiscriminately generate machine-readable COVID-19 records.

He posted an exposé the very next day, a perilously short window to both fix the issue and make sure the right actions were taken in lockstep with our public health partners. Thankfully my team found the bug, patched it, and had deployed the fix in under 90 minutes. I immediately informed all our state partners while Docket®’s CTO reviewed all our data to evaluate the impact. Five QR codes were inappropriately accessed all by one user matching the tech editor’s email. We directly informed each of the five impacted users and enlisted a state-recommended cyber security firm for additional black-box testing.

It was the toughest day in the company’s history. I was of course distressed that a tech blog would publish a negative story with security implications about Docket® without at least first giving us sufficient time to follow-up with our users. My team and I have always worked hard to be accurate and thorough. Unfortunately the article was live, and the damage in the public’s mind was done even if the blast radius had been contained. I want to be clear: my team and I always welcome security reports. Data security and privacy is our top priority. This bug was a deep regret for both me and the team, and we have taken additional mitigation steps to ensure that our users’ data remains protected.

Lessons Learned for Digital Health Companies

I believe that the three most important pillars for any digital health company are security, competency, and honesty. This event was challenging, but it strengthened our resolve to create positive outcomes in the consumer healthcare space. Docket® is designed to be secure. Just one faulty line of code is unacceptable. But policy also plays a key role: Docket® informs users in plain English (or Spanish) exactly what data is required to use the service. We do not share protected user data with third parties for any reason. It is important for us to recognize that health data belongs to the consumer; this message is worth reinforcing through intelligent human-centric design principles.

Competent digital health companies have eyes on their systems. Docket® had already implemented quality assurance procedures, end-to-end data encryption, and rate limiting. As a part of any design or new software rollout, you should consider ways to automatically monitor your logs for authorization violations. That would have given us an early indication without an email from the editor (i.e. in the case of a truly malicious actor). This was not a typical part of our threat modelling before this event. Consider systems that can also handle authorization policies built out-of-band from the code. This forces everyone to think more deeply about who is accessing what instead of relying on code checks.

Whenever faced with two or more data and privacy standards, it is always best for tech companies to assume they are governed by the one that is more restrictive: GDRP, COPPA, CCPA, NIST, etc. Docket® always operates under the assumption that it is bound by HIPAA, for example. These regulations are not just legal requirements and red tape; they are also a blueprint for creating trust. Mistakes will inevitably happen. Responsible tech companies take ownership and are made stronger by doing so. Bad news ages poorly – inform your stakeholders as soon as you know something has gone awry.

We have managed to learn, grow, and overcome this incident with our three pillars in mind.