The National University of Singapore (NUS) has built an extensive network, called NUSNET, covering four campuses: Kent Ridge, Bukit Timah, Outram, and NUS High School. The edge network has a topology comprising a single Cisco Software-Defined Access (SDA) campus fabric network for both wired and wireless, with a non-Cisco SDA wired and Wireless@SGx access network at hostels. Wireless@SGx is Info-communications Media Development Authority’s (IMDA’s) connection app.
The network consists of 1,700 units of SDA edge switches with approximately 81,600 POE+ switch ports (and another 400 units of non-SDA edge switches at hostels). There are also 8,500 units of SDA Wireless Access Points (WAPs) (and another 6,500 non-SDA WAPs at hostels). Daily, the network sees 50,000 unique wired clients and 200,000 unique wireless clients, downloading 40TB of wireless data and 20Gbps of uplink throughput per edge switch stack.
The core and distribution network topology, made up of 6,700 ports with a mixture of 10Gbps and 40Gbps interfaces with an 80Gbps uplink throughput, is fully distributed across four locations connected with dark fibre and forms the underlay backbone for the campus fabric network.
NUS also hosts the Singapore Open Exchange (SOX), a neutral Internet Exchange Point (IXP). SOX is a Layer 2 connectivity platform with Layer 3 peering for direct connection access. SOX differs from other IXPs in Singapore as it operates at OSI Layer 2 and does not provide any transit traffic. SOX’s goal is to improve routing efficiency at a lower cost via a collective effort, providing a landing for foreign IP networks. It has 25 participating members, some of which are big industry names like Amazon (40Gbps), Google (20Gbps), and Microsoft (20Gbps).
Yet, it hasn’t all been smooth sailing. There have been some technical hurdles to overcome.
Performance and coverage of the NUS indoor wireless network
It has been a recurring challenge to provide consistently high network performance for NUS staff and students due to the size of the campuses and the high density of users in certain locations. Moreover, our users connect using all types of devices that our wireless network needs to seamlessly support.
To overcome performance and coverage issues, NUS wireless was re-engineered starting from backend infrastructures such as wireless controllers, gateways to frontend wireless radio frequency deployments, and WAP installation. This solution was highly customized, and settings were calibrated to support high-density scenarios, general-purpose usage, and seamless roaming.
The WAP physical deployment in terms of positioning, mounting, height, and so on, was planned in detail to maximize the wireless coverage. Now, it can support the concurrent connectivity of a large number of devices, and at the same time minimize wireless radio frequency interference. It has enabled our users to enjoy smooth connectivity and good performance, for example, bandwidth-demanding video conferencing and AR/VR applications for teaching and learning.
The NUS wireless network has coped with huge surges in users in high-density regions at events with crowds and major online events, with few technical difficulties — events such as the annual NUS Open House, career fairs, countless e-lectures, and campus-wide e-examinations.
Coping with COVID-19
COVID-19 also presented issues. When the pandemic began, implementing crowd-controlling policies (Green/Red pass), crowd segregation by zone, crowd insights, and contact tracing became key priorities for NUS. All of these rely on wireless data to achieve their objectives and there was no technology platform or tool available then to provide such wireless data that was close to real-time with 100% accuracy.
With the following initiatives, carefully devised over the years, network infrastructure had become the foundation and key enabler in helping NUS to achieve its strategic priorities during the COVID years:
- NUS started to invest in expanding its wireless footprint in 2017 and achieved close to 100% indoor wireless coverage in 2020.
- Robust wireless controller infrastructure enabled us to come up with an unconventional method to retrieve wireless client data in two-minute intervals, which was close to real-time in tracking wireless user movement.
- Structured and organized WAP deployment and naming that enabled the location tracking of wireless users.
- Solid wireless infrastructure including authentication service provided 100% uptime and allowed users to stay connected anywhere, anytime.
802.1x
Previously, NUS had been on a heterogeneous platform for the access layer connectivity, which made it impossible to enforce 802.1x for wired authentication while the wireless network was already enforced with 802.1x. Due to this inconsistency, NUS was unable to implement segmentation uniformly across wired and wireless networks. This resulted in the wired network becoming the weakest link with open access.
As part of the campus network upgrade project, NUS adopted a homogenous platform for access layer connectivity and transformed the campus network into a Cisco SDA network.
With the new campus fabric network, NUS was able to achieve endpoint mobility throughout the campus via wired or wireless; and apply consistent authentication and segmentation policy to endpoints. Hence, wired access to all IT resources, including the Internet, is denied by default — except for authorized users. Now, segmentation policy can also be applied uniformly across wired and wireless endpoints to control lateral movement (East-West traffic).
Securing NUSNET
Currently, NUSNET uses authentication with segmentation to secure the network. All wired and wireless endpoints connecting to the NUS network are authenticated based on identity or a MAC address for non-supplicant devices. Since the upgrade, there is no more open access to the NUS network, which had significantly reduced the spreading of malware from compromised endpoints.
Segmentation plays a vital role in securing NUSNET. Endpoint segmentation is implemented to minimize the attack surface and control the lateral movement between endpoints. The campus fabric network has macro-segmentation where its North-South endpoint traffic is controlled by a firewall and micro-segmentation where its East-West endpoint is enforced using Security Group Access Control Lists (SGACL). NUSNET’s micro-segmentation policy is enforced based on user roles such as admin/academic/research staff, students, and so on. One special case is for guests and Kent Vale apartment users who are segmented into a dedicated virtual network that tunnels directly to the Internet without access to the NUS internal network.
The Internet of Things (IoT) network also presents a security sensitivity. Being a research-intensive university, NUS researchers deploy many IoT devices to collect big data and use wireless connectivity to transport that data to the cloud for processing and storage. To facilitate this, we have a dedicated isolated network segment for IoT with a stringent firewall policy. By doing this, we’re using the network architecture to segment services as a means to run a large campus-wide IoT system without some of the risks of IoT in the public sphere.
For security detection, there are network detection and response (NDR) solutions in place to detect abnormalities in the network, and capabilities to segment the network when necessary.
Priorities and future projects
Currently, our top priorities are to continue increasing network bandwidth to support university research goals and next-generation teaching and learning that use technologies such as augmented reality and virtual reality. We are also currently architecting and building an extranet and self-service managed network segment to facilitate the growth of cloud services (LaaS such as AWS and Azure, PaaS such as Dell and Boomi, and SaaS such as SAP) and secure network access from external parties. We also plan to rearchitect and upgrade the data centre network to achieve zero downtime maintenance, active-active data centre, telemetry data for analytics and troubleshooting, service automation and orchestration. Perhaps our biggest priority currently is to extend the indoor wireless network to the outdoor, with outdoor wireless hotspot portability, in order to achieve our borderless university vision.
5G for cable-less backhaul, supplying outdoor high-speed Wi-Fi
This outdoor extension of the NUS indoor wireless network is currently underway. The project’s goals are to bring it to market with the best-cost optimization possible. To achieve these goals, 5G is being used to provide backhaul connectivity for outdoor wireless hotspots.
Traditionally, for outdoor wireless deployment, excavation is required to lay fibre and power cable to connect the WAP and power it up. The time to market is typically very long as it involves authority approval for road excavation, logistic arrangement, safety assessment, weather conditions, reinstatement, and more. The cost can also be substantial where heavy machinery and civil works are involved.
By using a 5G connection as backhaul for the WAPs to the wireless controller in the data centre, NUS has eliminated the need to lay fibre to individual WAP locations. Solar panels will be used to power and activate the WAPs, so laying the power cable is also eliminated. We’ve found that by using this approach, the cost is decreased drastically by about 50% and the time to market can be reduced significantly by 90%.
Another advantage of NUS outdoor wireless using 5G backhaul is portability. In traditional outdoor wireless deployments, hotspots are permanently fixed at the location once installed. Whereas in NUS deployment, the outdoor wireless hotspot can be moved easily from one location to another location without going through the painful excavation process to lay fibre and power cable, allowing us to optimize placement easily after installation.
As proof of concept, NUS deployed a pilot setup successfully and won the SBR Technology Excellence award earlier this year. The next phase of campus-wide deployment is underway with the target to expand wireless coverage to major outdoor facilities such as car parks and sports venues.
As with all networks, improving network performance and security is a never-ending task but we see NUSNET’s improvement goals as ambitious. Not only are we seeking to improve educational outcomes via such improvements, we also aim to make campuses more sustainable. With these ambitious goals, we look forward to harnessing technology and becoming a truly borderless University.
Tiong Beng Ng is the Director of Infrastructure at NUS Information Technology, National University of Singapore.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.