APNIC resolved an information security incident today and I wanted to share the details here with the community in the interest of transparency.
While staff were performing maintenance work on APNIC’s RDAP service, a ‘dump’ file of the whois SQL database was copied to a Google Cloud storage ‘bucket’ that was believed to be private. However, a configuration error meant this bucket was actually publicly visible for a period of three months.
The file contained hashed authentication details for APNIC whois maintainer and IRT objects, and also included some private whois objects that are not visible on APNIC’s regular public whois service.
APNIC was alerted to the problem on 4 June by an independent security researcher. Upon confirming the problem, APNIC rectified the configuration error and removed the copy of the database.
It is not known if the data was accessed, as complete log files are not available, however initial investigations reveal no sign of suspicious update activity.
As a precaution, APNIC worked with resource holders to reset all maintainer and IRT passwords from 15 June and completed the process today. No further action is required by APNIC resource holders.
APNIC apologises for any inconvenience and concern that this incident has caused. It is clear that on this occasion APNIC has not met the information security expectations of the Members and community, nor the high standards it sets for itself.
Why were there private objects in the whois data file?
Until October 2017, when new private objects were created in the APNIC Whois Database a copy of the object would be included in audit logs. New functionality implemented in October 2017 ended this practice and as such, the private objects were only current to October 2017.
The dump file of the whois database in this incident included these audit logs.
The data contained in the private objects varies, as there were comments added by resource holders in the ‘descr’ and ‘remarks’ attributes. The review of this data has found that it predominantly consists of corporate contact details.
What was the password issue?
A maintainer (mntner) is an object in the APNIC Whois Database. Every object in the APNIC Whois Database is protected by a maintainer via the ‘mnt-by’ attribute. This ensures that only authorized people that have access to this maintainer can make changes to other objects that are protected by this maintainer.
An Incident Response Team (IRT) object is an object in the APNIC Whois Database that contains contact information for an organization’s administrators responsible for receiving reports of network abuse activities.
The ‘auth’ attribute in a Maintainer or IRT object specifies the hashing format used and stores the password in its hashed format.
The ‘auth’ hashes were included in the database that was inadvertently publicly visible due to the configuration error.
Although password details are hashed, there is a possibility that passwords can be derived from the hash by a malicious actor.
If that occurred, whois data could potentially be corrupted or falsified for misuse. Our investigations to date have found no evidence of this occurring.
(It is important to note, however, that any public misrepresentation of registry contents on whois would not result in a permanent transfer of IP resources, as these functions are protected by MyAPNIC access mechanisms, and authoritative registry data is held internally by APNIC).
What action was taken?
Firstly, the misconfiguration was fixed on the cloud storage bucket to make it private, and the copy of the database was removed.
To eliminate any risk of the exposed password hashes from being used, APNIC decided to reset all Maintainer and IRT object passwords.
Most resource holders make updates using these objects very rarely (over 12 months between updates), and many use MyAPNIC to manage the process which means the passwords are invisible to the user when making updates. For these resource holders, APNIC reset all passwords automatically.
A smaller group of resource holders (around 50) had actively submitted updates to APNIC’s whois service via email in the past six months. APNIC’s risk assessment determined it would be better to not reset these passwords remotely, but instead guide the active resource holders through the password reset process to minimise disruption to their network operations.
This process was completed today, and we are sharing this information now that there is no further risk to resource holders by doing so.
The data included in the private objects in the database is still being assessed to determine if any further remedial action should be taken by APNIC.
Do resource holders need to take any action?
APNIC continues to analyse its logs to search for any signs of whois misuse as a result of this error. So far, we have found no evidence of irregularities.
All Maintainer and IRT passwords have now been reset, so there is no need to change them again if you are an APNIC resource holder. However, if you wish to change the new passwords to something more memorable, you should not choose the previous password (and if the old password was being used elsewhere on other systems, you should change those passwords).
Please note, this issue is completely unrelated to MyAPNIC login credentials. If you have a MyAPNIC account, there is no need to change your MyAPNIC password.
If you are making updates with your Maintainer via email, APNIC recommends using PGP.
Of course, if you have any questions or concerns, our Helpdesk is happy to assist.
What will APNIC do now?
Trust is one of APNIC’s core values. We recognise that on this occasion we have regrettably impacted our Members’ trust in us. We need to do better.
However, accountability is another of APNIC’s values, and this blog post disclosure is just the beginning. A thorough post-incident review is already underway to determine where process, technology and oversight improvements can be made to avoid similar incidents in future.
The actions from that review will be implemented as a priority in the coming weeks. A report on the results of that review, and actions taken, will be presented at APNIC 52.
Again, we have no evidence that any whois data, including hashed passwords or private objects, were accessed as a result of this incident. However we continue to monitor for any evidence of such access, indicated by suspicious activity and access logs.
APNIC thanks resource holders for their patience and support during the resolution of this incident.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.