Since 2010, the root of the DNS, the root zone, has been signed using DNSSEC.
Every three months, ICANN conducts a Root Signing Ceremony at one of two facilities located in the USA, where a select few get together to activate this process. It requires access to devices that are stored in safes, one holding electronic equipment, including a laptop and two Hardware Security Module (HSM) devices, and the other containing security deposit boxes, inside of which are the smart cards used to enable/activate the HSM devices.
Access to each element is controlled by a different person who all have to come together at the time of these ceremonies to bring all the elements together.
As with any system, maintenance work is regularly necessary and carried out by ICANN staff and contractors performing specialized functions. In the days leading up to the most recent ceremony at the West Coast facility located in El Segundo, California, replacement work on the safes’ locks was scheduled as the original locks were being EOL’ed (‘End of Lifed’) by the manufacturer (see a comparison of the old and new locks) This meant these locks would be opened one last time and then be replaced.
As luck would have it, while the replacement took place without incident on one of the safes, when attempting to perform the replacement on the second safe the installed lock failed and no longer opened the safe.
A meeting was quickly convened with the parties involved to examine possible next steps: each of these ceremonies involves getting together a significant number of people and while some are locally-based ICANN staff, others, like external auditors and, in particular, the Trusted Community representatives (TCRs), travel from all over the world and are usually only around for the two days (primary and backup) scheduled for the Ceremony. To be able to carry out the ceremony, at least three TCRs are required to be present and have their keys to open the safe-deposit boxes, as three is the minimum quantity of smart cards necessary to activate the HSM (out of a maximum of seven).
Thankfully, this time around, we could find three TCRs, of which I am one, who could stay an additional few days and wait for the process of gaining access to the safe in order to carry out the Signing Ceremony; without having to implement more elaborate plans such as reconvening at a later date or call for a modification of the upcoming ceremony in the East Coast facility (due in May 2020). This situation also illustrates the benefits of having a process that allows sufficient flexibility in the root zone publication pipeline to absorb potentially significant time delays in any of the steps.
We then proceeded to schedule the safe intervention, which basically means that a certified locksmith has to access the safe room and drill through the safe to release the lock. Initially meant to be a ‘matter of a few hours’; it turned out to be a two-day event. While it may sound strange that one would just start drilling into this kind of equipment, this is, in fact, the manufacturer-indicated method of dealing with this sort of mishap and the materials and instructions to carry it out are readily available.
Finally, on the evening of Saturday, 15 February, we were able to carry out the originally intended ceremony — skipping a few optional steps which had been included originally to take advantage of the meeting — and end just in time for me to catch my postponed flight to APRICOT 2020 in Melbourne, Australia.
I would like to end by thanking ICANN/PTI for being so transparent about all these unusual events and the path followed to address this unpredictable situation.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.