To mark APNIC’s 30-year anniversary, the APNIC Blog is running a series sharing stories, anecdotes, milestones and insights that capture some of the essence of the last three decades.
We’d love to hear your stories and memories of the APNIC community. Post them on social media using the tag #APNIC30th!
Since its inception, APNIC has been committed to supporting the growth of Network Operators Groups (NOGs), Internet Exchange Points (IXPs), and Computer Emergency Response Teams (CERTs). These three activities exemplify the best in the Internet’s ‘self-organizing’ principle — we’re stronger together, and cooperation makes the network bigger and better.
Unlike the registry function, none of these are bound by address policy, implemented by the Secretariat, or formally managed in a unitary structure. NOGs, IXPs, and CERTs live in the wider community, they organize on their own terms and are shaped by local conditions and contexts. They work because the community makes them work. They are the embodiment of self-organization among the APNIC community ‘in the wide’.
APNIC supports these activities in various ways including funding, infrastructure support, training and endorsement. APNIC staff also regularly present at, and provide support for meetings, and host APIX in APRICOT and APNIC conferences.
But where did it all start?
NOGs: The lifeblood of connectivity
NOGs are a response from the ISP community to the ongoing operational concerns that, at a day-to-day level, drive how the Internet works at scale. Based in email lists, NOGs develop into regular face-to-face meetings and form a repository of knowledge shared with competitors. Usually, the NOG list is where operators go to find out ‘what’s really going on’ in a spirit of mutual trust. Some NOGs have a ‘no journalists’ Chatham-house rules style of operation to ensure participants feel free to discuss their internal problems and concerns.
The original NOG was the North American Network Operators Group (NANOG), formed in 1994 as an outgrowth of a US regional initiative MERIT (operators of the NSFNet from 1987). While NANOG always had a US/Canada focus, it welcomes participation from all over the world. For many years, NANOG was the recognized place for global Internet operations conversations and continues to be a significant driver of global Internet operations. Perhaps most importantly for the APNIC community, NANOG inspired the birth and growth of NOGs in the Asia Pacific region. Now, many APNIC regional NOGs include active NANOG participants and cross-fertilize ideas and concerns.
There are more than 20 NOGs in our region, with significant contributions from JANOG, AusNOG, NZNOG, and SANOG.
JANOG
JANOG was established in 1997 for the Japanese Internet network operations community. Operating mostly in Japanese, it is a lively, well-connected community with over 7,000 participants and over 500 regular attendees at its bi-annual meetings.
“In Japan, the WIDE Project’s NSPIXP started in 1994 and with it came the interconnection of ISPs. Engineers at ISPs, including me at that time, definitely needed information about network operation. Exchanges happened in every NSPIXP meeting, which was held three or four times a year. We had Internet World Expo in 1996, which was another opportunity for ISPs to get together. In the US, NANOG was active in discussion and contained a lot of valuable technical information. Many ISP engineers in Japan were subscribed to the NANOG mailing list and up to a dozen people flew to the US and participated in the meetings in person,” Akinori Maemura recalled.
“From time to time, we discussed forming a Japanese NOG. Should it be JPNOG? Or JAPRICOT? I remember several months flew by like this. It was during the NANOG 9 meeting in San Francisco in February 1997 that ISP engineers from Japan got together at the lounge of the venue hotel and discussed this Japanese NOG again. Kuniaki Kondo, who later served as the second JANOG Chair, took action to configure a mailing list called JANOG on his personal server, which was under his office desk. This marked the very beginning of JANOG in February 1997.”
“I remember the early days of the JANOG mailing list where a bunch of frank and sometimes blunt discussions happened. The mailing list every day gained new subscribers and rapidly grew, quickly counting 100. Soon a volunteer group was formed to think about how JANOG should be organized further. They met as a ‘pre-pre-meeting’ in June 1997 to think about holding a meeting. The inaugural meeting JANOG0 (also known as pre-meeting) was held in July 1997 to gather 24 participants. I was a presenter there and it felt like a test run of a NOG-style conference, which was my very first technical presentation in my Internet career. JANOG1 in September 1997 was held at the NTT Office and finally got a normal format. We didn’t know that JANOG would grow up to gather more than 1,000 people at one meeting 25 years later,” Akinori continued.
“JANOG is proud of its transparency and has made the minutes of pre-pre- and pre-meetings, as well as materials from other meetings, available on the JANOG Web. Please enjoy it with your translator!”
AusNOG
AusNOG officially began in 2007 with its first conference. However, before that, the Australian Operations Mailing list (aussie-isp) had been operational since at least 1997, according to mail archives.
The inaugural meeting in 2007 saw over 175 attendees with technical presentations from the local and international, including Geoff Huston. Fast forward to AusNOG 2023, and the annual event has proven popular with 350 participants.
NZNOG
In 1998, Donald Neal founded NZNOG, a testament to the rich history of the New Zealand Internet and the community-driven initiatives that fostered IX developments in the region, such as WIX in Wellington, CHIX, in Christchurch and the APE in Auckland.
Currently functioning under the guidance of the NZIX/Citylink, NZNOG’s first email is preserved in the mail archives. It was a ‘check you got it’ posting!
OK, in reponse [sic] to Roger Hicks' posting below I've added the people
named in the headers of his message to a mailing list I've called nznog.
The address to send mail to is nznog(a)list.waikato.ac.nz. There's a
fair amount else the […] box can do, but there doesn't seem a lot of
point to explaining that just yet. Could people receiving this please
reply to me verifying that they have seen it?SANOG
Originally operating with the support of the Asia Pacific Internet Association (APIA, itself formed in 1997), SANOG started in 2003 with a focus on the South Asia Internet Operations community.
SANOG modelled itself on NANOG, the European Operators Forum (EOF) at the RIPE NCC meetings, and APRICOT held across the Asia Pacific region. Now, SANOG is endorsed by APRICOT, itself an outgrowth of APIA.
“SANOG started because I went to an APRICOT; that is pretty much the background to it. APRICOT was in Bangkok. They were going to do APRICOT that particular year somewhere in China, and then they cancelled it; it ended up being in Thailand. Back then I was doing some freelancing work with UN volunteers. The only reason I ended up going to APRICOT was that it was in Thailand. We could basically go find the cheapest ticket and actually go to Thailand!” Gaurab Raj Upadhaya recalled.
“I met a couple of other people from Sri Lanka and another gentleman from Bangladesh. We had a colleague from Cisco teaching workshops with Philip Smith, who came from India. It was a tiny South Asian community of five or six people. I went back home and said ‘We need to do something like this. We need to meet in the local region, and we need to get people to come and do tutorials’. Back then, I don’t think I even knew about JANOG and NZNOG, all I knew was APRICOT and NANOG. I thought these NOG things were supposed to be regional.”
SANOG grew quickly, as Gaurab recalls.
“We got around, I think, 25 to 30 people at first. Now, attendance is consistently around 200 to 300, and sometimes like in Bangladesh, we might get 450 to 500 people. Even back then it was multitrack. We just didn’t have workshops. We had tutorials, we had DNS training, we had security, we had the whole APNIC day.”
Creating a NOG can have much more impact than expected.
“We realized we were getting registrations and a whole bunch of people from the Middle East coming to Kathmandu. At SANOG 4, I had a conversation with Osama Al-Dosary and convinced him to come to APRICOT. That was the start of MENOG,” Gaurab said.
IXPs: Efficient local connectivity
With almost 40 IXPs across 19 economies in the Asia Pacific region, this community illustrates the cooperative nature of Internet interconnection. Logistically, almost all transit between operators depends on the Border Gateway Protocol (BGP), which operates between two parties cooperating as ‘peers’ in as much as they exchange information and modify their routing accordingly, without a hierarchy of dependency between them inside the BGP.
Crucially, the nature of this BGP connectivity is almost completely unspecified. It has to be a link capable of carrying IP, but otherwise can take any form they agree. Historically this took the form of point-to-point links between the BGP speakers. However, the escalating expenses associated with maintaining a growing network of links prompted the preference for a centralized meeting point, eliminating the need for a separate physical interface for each BGP peer or, in cases where such interfaces are necessary, avoiding the high cost of long-haul connections. When participants mutually agree, an open IXP is born.
“We set HKIX up with just thin coax cables to interconnect co-located routers installed in our data centre, which had spare space at the time. We told our management the cost was minimal to us (just the coax cables), but it could help us to save a lot of money (the overseas Internet transit) while we could have much faster connectivity to those ISPs connected. The most convincing justification was CUHK would have the best Internet connectivity in HK and, of course, among all universities in HK,” Che-Hoo Cheng, then director of HKIX said.
“We received full support from management, allowing us to proactively engage with all identifiable ISPs. We successfully convinced them to invest in local loops for connectivity and the co-located routers for placement at CUHK. Some were convinced right away and agreed to connect. Then, the other ISPs had no choice but to connect.”
“After HKIX was up, and running successfully, and with demonstrated benefits to CUHK, it was easy to increase the investment, including the upgrade to proper Ethernet switches,” Che-Hoo explained.
Gaurab was convinced from the start that Nepal needed a local IXP.
“At home, I had a connection from Worldlink, and I was frustrated that, you know, the messages were going across the world and coming back. Back then it was all satellite. It was going to Hawaii and then coming to Singapore. I basically ran some bundled dialups. I kind of created a script on Linux that dialled into the ISP all night and I was trying to get into those machines from home over telnet and it didn’t work,” he said.
“I came back from APRICOT, and then we kicked off the IXP part quite heavily. I went to Rupesh Shrestha’s (who was General Secretary of the ISP Association of Nepal) office and said ‘Hey, we need to do this. Can you organize something?’ He said, ‘Okay, let me call everybody into a meeting in two days or something’.”
“We brought people together and everybody was like, ‘Yeah, we want to but it is difficult for us to do it because we are all competitors’. So, I said ‘No, no, I will do the job, you guys just do what I tell you to do!’ and a few of us got together and emailed IXP pioneer Bill Woodcock. He flew in with a switch and a bunch of random gear. Then we emailed Internet pioneer Bill Manning, and he gave me a bunch of IP addresses.”
“Then one of the ISP guys said ‘Hey, guys, in this particular Nepal Telecom location, we all have equipment to connect to the phone lines. Why don’t you use the switch in my rack, and we’ll just wire up each other Ethernet sneakily’. The Nepal Telecom was not giving us point-to-point links to our data centre location. We basically snuck in and set up the IXP that way.
“When we finally got Nepal Telecom people convinced to join us, the engineers went and did some facilitation internally to get the real IXP site up and running. Rupesh and I did presentations to the Nepal Telecom senior management and again, my association with the Computer Association of Nepal helped. When we got everybody aligned, Nepal Telecom became the official founder when we incorporated.”
“Philip Smith helped us; we got a bunch of support from the predecessor of the APNIC Foundation, the Pan Asia R&D grants program. We got a small grant from them.”
“The IXP for me was 100% a local initiative. We were the first IXP anywhere in that region. Probably even broader, I think outside of maybe Australia, New Zealand, and Japan, maybe Hong Kong and Singapore. Sometimes people say, ‘Oh, you don’t have a lot of traffic’, but the point is almost all of our new page traffic is local. Now, you know 100% goes local — traffic is getting sent locally.”
CERTS respond to the rise in online threats
Computer Emergency Response Teams (CERTs) are another community-led response to the need for a safe, secure Internet. Often formed spontaneously between ISPs exposed to risk, they extend from the Internet routing and service-provider community out into the wider Internet. They have always included individuals, corporations, tertiary education, and government departments concerned about their security needs.
Related bodies called Computer Security Incident Response Teams (CSIRTs) are the approach used by individual entities to deal with threats and risk, coordinated through FIRST, the Forum of Incident Response and Security Teams.
Along with IXPs, establishing CERTs has been a significant goal for the island economies of the APNIC region. Adli Wahid, Internet Security Specialist and a board member of FIRST has been involved in training, development, and continuing community engagement of the CERT/CSIRT communities, helping Papua New Guinea, Tonga, and Vanuatu, among others, to improve their security posture in the face of risks worldwide.
“APNIC’s been involved since 2014 and now there are a few CERTs in the region. Our role is to support them getting the resources ready, helping them figure out what the mandate would be as a national entity and what is needed to operate this with very limited resources for years to come” Adli recalled.
“This year, I contacted one of the smaller island economies. I think they only have one ISP there and 100 or more of the home routers (a significant percentage of the island’s Internet) were infected. They didn’t know this because they don’t have any security monitoring in place. We contacted them, had excellent interaction, and were able to advise.”
“One of the most visible benefits of having a national CERT in the Pacific that I’ve seen is the increase of awareness because the CERTS will collect all this information unrelated to vulnerabilities, attacks, fraud, and use this to translate into something the public can understand.”
“There’s a spirit of ‘this is a long journey. Let’s do this together’ in the Pacific Islands. Volunteer trainers from established CERTs share their experience, policies, and practical advice with newer ones. I also have seen a lot of people from the CERT community actively reaching out to ISPs, telcos, and law enforcement to help with a broad understanding of issues. Looking to the future, the goal is to support some of the smaller islands like Nauru and Tuvalu,” Adli said.
The community is alive and well
Across the NOG, IXP, and CERT landscapes, we can see three distinguishing behaviours that reinforce a sense of viable communities. Firstly, they are strongly self-organizing, emerging from the Internet operations community organically.
Secondly, participation is open and mutual. Initially, informal methods may sometimes be necessary for bootstrapping. However, formal engagement with entities such as national telcos and regulators ensues when the function proves its effectiveness.
Lastly, the investment in time, effort, and materials is determined by necessity. Initially often volunteer and best effort, some activity can reach sufficient levels to justify paid staff and continuing funding, from the community or interested parties. We now have over two decades of locally organized, highly interactive, engaged community-led processes and discussions at a regional and national level inside the APNIC community at large, many of which feed into each other.
These dynamics are found at all levels within the APNIC community and create a mutually beneficial exchange of ideas, initiatives, and knowledge, reinforcing the vitality of these interconnected communities. As the Internet continues to scale, these community links will become even more important. It’s reassuring to know they have such a strong base.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.