APNIC has a dual role in Resource Public Key Infrastructure (RPKI). The first, and most important, is that APNIC operates as a Trust Anchor (TA) in the X.509 hierarchy of certification, tied to its role in the distribution of Internet number resources in the Asia Pacific region.
Secondly, APNIC operates its own networks, existing in the Border Gateway Protocol (BGP) default-free zone as at least three distinct Autonomous System Numbers (ASNs). APNIC believes strongly in being a visible participant in global BGP routing, both as a duty of care to understand resource delegates’ lived experience, and to ensure APNIC’s services are widely visible in the face of any ‘to the door’ transit issues of its networks. Therefore, APNIC peers widely at Internet Exchange Points (IXPs) within a reasonable distance of its infrastructure.
This post discusses that second role and how APNIC deployed RPKI for its own network assets. APNIC did this to ensure the integrity and stability of the first role — being a trustable source of authority to validate its delegated RPKI assertions. It focuses on the announcements made by APNIC in the Brisbane location, originated by AS4608.
Laying the groundwork
APNIC operations tested RPKI early in 2015 by signing Route Origin Authorizations (ROAs) for one of its core network assets. At the time, the amount of Relying Party (RP) BGP speakers running Route Origin Validation (ROV) was low, so there was little risk of causing a loss of reachability in the global default-free zone. APNIC was able to learn how the MyAPNIC route management system worked as a user (rather than as the developer and operator of the registry signing service). The route management system covered both RPKI and management of the Internet Routing Registry (IRR) records, which had been the sole routing policy declaration. APNIC saw no negative outcomes in terms of the visible BGP, or problems aligning the ROA and IRR route objects with BGP.
Perhaps the first lesson learned was — dip a toe in the water, make a move, and find out.
APNIC, the SME
APNIC operates what can best be described as a Small to Medium Enterprise (SME) network. There are multiple locations of service delivery in Japan and Singapore as well as in Brisbane, but for the purposes of the RPKI activity discussed here, we’re only discussing the APNIC location.
APNIC announces seven routes from six IPv4 ranges that have been delegated, and makes 13 announcements from a further four distinct IPv6 ranges. These resource holdings reflect APNIC’s operations in Brisbane, and some historical services delivered from Japan, Hong Kong, Ashburn (Washington DC) and Singapore, as well as anycast delivery of service with partners worldwide.
Aside from hosting some anycast services such as DNS root servers, a training network, and some research networks with our partners, APNIC does not provide transit for any other independent networks but peers widely at local IXPs to ensure direct reachability from as many ASes as possible, where it’s sensible to do so.
APNIC does, perhaps, have a slightly more complicated stub network than most SMEs, but there’s nothing particularly unusual about having more than one prefix to announce in BGP. Several SME networks have legacy resources, as well as more recent ones, and can be multi-homed in BGP for a range of reasons.
APNIC’s own-use addresses present from three distinct routers, which, at this time, are all Cisco IOS devices. APNIC has three primary relationships for transit with Telstra (AS1221), Vocus (AS4826) and AARNet (AS7575) but has many other peerings at Equinix and PIPE in Brisbane. So, from a BGP perspective, APNIC presents in Brisbane behind AS4608 as a slightly complex multi-homed stub network.
Making the ROA — keep it simple
In 2018, a decision was made to deploy ROAs for all the APNIC prefixes announced from AS4608, the core services network. As they have no complex multi-AS origination, the ROAs were deterministically simple — there should only be one origin-AS.
Because APNIC holds a small number of aggregatable IPv4 prefixes (aggregated to a /22 and a /21 above a /24) the MyAPNIC route management facility was used to create minimal ROAs that have a maximum length set to a /24 and encompasses the /21 and /22 above that level.
It would have been possible to produce explicit ROAs that did not use a maximum length because the holdings are quite small, but APNIC decided to keep things simple and use the MyAPNIC route management facility to keep the IRR route objects aligned with the ROAs. This may be an experience common to similar enterprises and it’s a reasonably straightforward decision to make.
Research added some complexity for one or two prefixes
Unlike most enterprises, APNIC also conducts research into BGP from a small number of the announcements for AS4608 and this presented some minor complexities in how to announce and propagate routes. To be able to validate RPKI, this complexity led to breaking out a network subset to be announced from another AS, using a virtually-hosted BGP speaker offsite. This subset runs a delegated RPKI system independently of the APNIC operationally signed RPKI products.
IPv6 added some very small complexities due to the large prefix extent
An enterprise with more than one location typically gets a /32 of resources and is therefore capable of making a large number of more specific announcements to the /48 (65,000) limit. IPv6 ROA production is slightly more complicated.
It’s not typically sensible to make a single ROA with a large maximum length over the whole space because it is logistically difficult to assert all possible specifics in BGP. It also makes it possible for a bad actor to announce (otherwise unannounced) more specific prefixes easily, and ‘steal’ the BGP route with a longer AS-PATH to your origin-AS. APNIC decided to sign over /32 – /35 maximum length prefixes, and sign the specific /48 that is announced at this time. These all lie under four IPv6 allocations.
APNIC chose not to self-host this RPKI, preferring to publish RPKI products in the main APNIC delegated repository like any other APNIC Member using MyAPNIC. As noted above, a small number of RPKI products used by APNIC Labs are self-published and operate distinctly from the core AS4608 declarations of intent.
Validation needs reliable data
Initially, APNIC tested RPKI validation using the RIPE NCC Java validator. This was deployed for internal use in APNIC, with a testbed visible to the public (APNIC doesn’t recommend that any pre-validated feed of RPKI be taken from an external AS but it can be helpful to see a web GUI over validation outcomes to help with debugging, so one was informally provided from before 2018).
This allowed APNIC to ‘dip another toe in the water’ by watching how a validator interpreted the flow of BGP coming to its door (the RIPE validator, as well as Routinator, regularly import a feed of BGP state and can show in a web GUI the invalid announcements being seen).
Run multiple redundant validators
Subsequently, this Java code from the RIPE NCC was withdrawn from support, and APNIC shifted to using the NLnet Routinator Rust code, deployed in two instances for redundancy. APNIC supports NLnet financially in the maintenance of this code, and their ‘Krill‘ Certificate Authority (CA) code. In return, APNIC gains operationally important software support. APNIC believes running a BGP service for operationally important networks isn’t sensible without software support.
This investment in Routinator was supplemented by an instance of the OpenBSD rpki-client, written in C (APNIC also supports the development of this product for community benefit). Therefore, APNIC has three points of RPKI validation that have code independence (different languages, different developers). This permitted APNIC to deploy RPKI-RTR, (the protocol that supplies ROV) to APNIC AS4608 routers from three sources.
Having three distinct feeds of ROV gives APNIC the added benefit of a confidence check inside the system — if they don’t agree to validity, APNIC gets a warning, which helps debug operational problems. There is a risk of being exposed to instability from the competing views of the state of RPKI such as ‘false positives’ but this can be managed by setting thresholds of the alerting system.
APNIC decided not to enable drop-invalid — here’s why
The final decision in deploying ROV is what you do when faced with an invalid BGP announcement under RPKI. Normally, after a period of ‘watch but report,’ you would immediately ‘drop’ so no invalid assertions can be used to fool your routing and forwarding decisions. However, APNIC has a high-level deliverable to be available to all networks and all regions, all the time.
During analysis, it was decided that enabling ‘drop-invalid’ risked denial-of-service to any APNIC resource delegates who accidentally misconfigure their routing intent. Rather than require them to use another provider path into APNIC for resource management, for now, APNIC logs invalids and continue to process them for BGP updates and withdrawals.
Note: If your network has RPKI invalid routes, you will not be able to reach the APNIC network and access APNIC services, including MyAPNIC, in the following scenarios:
- Your upstream provider(s) have implemented ROV and started rejecting RPKI invalid routes.
- A network, somewhere between APNIC and your network, starts rejecting RPKI invalid routes.
- All APNIC upstream providers start rejecting RPKI invalid routes.
You will need to use a network that has an RPKI valid or unknown announcement to access MyAPNIC and fix your incorrect or outdated ROAs.
Read: Cleaning up your RPKI invalid routes
It’s not really about stub networks in ROV
At one level, the decision for a stub network whose transits implement ROV is moot — stub networks don’t determine the BGP visibility of the routing intentions of other entities if the stub’s transit is applying ROV with drop. Because APNIC is a stub, all routing and forwarding affecting the network is defined by its transits.
At this time, two of the three have implemented ROV, and APNIC expects in due course to be completely covered for transit by BGP speakers with ROV. APNIC’s ROV regarding BGP is not a problem; it is an additional safeguard, but BGP filtering is now applied with one hop along the AS path to APNIC.
Due to APNIC’s reachability considerations for service delivery, the network team decided to flag invalids rather than drop them. However, it’s advised that stub networks doing a lot of direct peering should filter ROV invalid routes for better self-protection and not rely purely on their transit networks.
It’s a simple process with staged steps. Come on in!
There is no need to dive directly into live ROV with drop invalid, you can take a phased approach to deployment with these recommended steps:
- Sign your own assets and declare intent clearly. This has no direct risk to your own BGP as long as you keep these ROA signings in alignment with your BGP origin-AS intent.
- Keep your RPKI and IRR aligned. If you live inside the MyAPNIC route management system, it will do this for you automatically.
- Avoid having maximum length settings that you do not announce in BGP so that more specific announcements can’t be intruded on for your origin-AS with a false path.
- Deploy a validator that can integrate with BGP and show you a diagnostic of the state of validation worldwide before you configure it into your live BGP.
- Optionally, deploy more than one validator, to employ code diversity behind your source of ROV.
- Deploy monitoring and check scripts that inform you of problems in BGP (such as the BGPalerter code from Massimo Candela at NTT).
- Consider carefully if you want to move to drop invalid, before enabling RPKI-RTR feeds to your BGP speakers.
Check out the RPKI @ APNIC portal for more information on RPKI as well as useful deployment case studies, how-to posts, and links to hands-on APNIC Academy lessons and labs.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.