How to: Set up a testbed for validating RPKI/ROA using PMACCT and correlating traffic with BGP

By on 27 May 2020

Category: Tech matters

Tags: , , , ,

Blog home

Resource Public Key Infrastructure (RPKI) is designed to secure traffic over the Border Gateway Protocol (BGP) and protect Internet infrastructure from BGP hijacking. 

In this post, I will show how RPKI can be set up on a testbed with PMACCT to collect and validate RPKI/Route Origin Authorization (ROA) information.

Network preparation

Simulators are useful tools to set up a testbed and troubleshoot potential problems when building a new network, or migrating current infrastructure. 

Figure 1 — A testbed lab setup with the virtual machine as a data collector for PMACCT.
Figure 1 — A testbed lab setup with the virtual machine as a data collector for PMACCT.

Both routers (RTR-01 and RTR-02) should be configured with NetFlow and the traffic exporting port address (port 9999 in the example) should be the same as that of data collector (pmbgpd/nfacctd).

 RTR-01#flow record NetRecord
       RTR-01#match ipv4 tos
       RTR-01#match ipv4 protocol
       RTR-01#match ipv4 source address
       RTR-01#match ipv4 destination address
       RTR-01#match transport source-port
       RTR-01#match transport destination-port
       RTR-01#match interface input 
       RTR-01#collect routing source as
       RTR-01#collect routing destination as
       RTR-01#collect interface output
       RTR-01#collect counter bytes
       RTR-01#collect counter packets
       RTR-01#collect timestamp sys-uptime first
       RTR-01#collect timestamp sys-uptime last
       RTR-01#collect application name
 RTR-01#flow exporter NetExport
       RTR-01#destination X.X.X.X
       RTR-01#source FastEthernet 0/0
       RTR-01#transport udp 9999 // the same port address should be set for pmbgpd configuration port
       RTR-01#template data timeout 60
       RTR-01#option application-table timeout 60
       RTR-01#option application-attributes timeout 300
 RTR-01#flow monitor NetMonitor
       RTR-01#exporter NetExport
       RTR-01#cache timeout active 60
       RTR-01#record NetRecord         

Once you’ve completed the initial installation and integration of the testbed, you can start installing the PMACCT tool.

How to set up PMACCT on Centos 7

PMACCT is an IP traffic accounting tool licensed under the GNU General Public License (GPL). Before getting started with the PMACCT, we need to set up the underlying environment on which PMACCT will run. 

Relying on MySQL for the underlying data collection is only recommended for test environments. For large sets of data collections, it doesn’t scale well and in that case, we need to make use of real-time data stores such as Apache Druid.

1. Install MySQL

 yum install mysql-community-server
 yum install libtool
 yum install mysql-community-devel

2. Run MySQL

$ systemctl start mysqld

3. Create a new database

 $ mysql –u root –p
 $ CREATE DATABASE pmacct;

Create a table for BGP data collection as follows:

 $ mysql -u root -p < pmacct-create-db_v1.mysql
 $ mysql -u root -p < pmacct-grant-db.mysql
 
 create table acct_bgp (
    id INT(4) UNSIGNED NOT NULL AUTO_INCREMENT,
         agent_id INT(4) UNSIGNED NOT NULL,
         as_src INT(4) UNSIGNED NOT NULL,
         as_dst INT(4) UNSIGNED NOT NULL,
         peer_as_src INT(4) UNSIGNED NOT NULL,
         peer_as_dst INT(4) UNSIGNED NOT NULL,
         peer_ip_src CHAR(15) NOT NULL,
         peer_ip_dst CHAR(15) NOT NULL,
         as_path CHAR(21) NOT NULL,
         local_pref INT(4) UNSIGNED NOT NULL,
         packets INT UNSIGNED NOT NULL,
         bytes BIGINT UNSIGNED NOT NULL,
         stamp_inserted DATETIME NOT NULL,
         stamp_updated DATETIME,
         PRIMARY KEY (id); 

Create a table for netflow data collection as follows:

 $ mysql –u root –p
 create table acct (
 mac_src CHAR(17) NOT NULL,
 mac_dst CHAR(17) NOT NULL,
 ip_src CHAR(15) NOT NULL,
 ip_dst CHAR(15) NOT NULL,
 src_port INT(2) UNSIGNED NOT NULL,
 dst_port INT(2) UNSIGNED NOT NULL,
 ip_proto CHAR(6) NOT NULL,
 packets INT UNSIGNED NOT NULL,
 bytes BIGINT UNSIGNED NOT NULL,
 stamp_inserted DATETIME NOT NULL,
 stamp_updated DATETIME,
 PRIMARY KEY (mac_src, mac_dst, ip_src, ip_dst, src_port, dst_port, ip_proto, stamp_inserted)
 ); 

Then exit from the MySQL prompt.

4. Install PMACCT

The simplest way to get and configure the PMACCT package is to download it from the official webpage or clone it from GitHub and then follow the steps below to configure it:

 $ git clone https://github.com/pmacct/pmacct
 $ cd pmacct
 $ ./autogen.sh
 $ ./configure - -enable-mysql
 $ make
 $ sudo make install

If you are using PostgreSQL or SQLite, follow the step below to configure it:

 $ ./configure - -enable-pgsql
 $ ./configure - -enable-sqlite3 

5. Create plugins and configuration files

Configuration files are used to define how the PMACCT tool can work for a different type of network traffic. There are various plugins available for the PMACCT tool to correlate network traffic with RPKI/ROA. Those that are useful for the networking testbed are:

pmacctd — This is a libpcap-based accounting daemon that captures network data packets from one or multiple interfaces. It can be used to collect data from the interfaces it configured and export this traffic data to the server system via NetFlow or sFlow protocols.

nfacctd  This is a NetFlow/IPFIX daemon that listens for NetFlow versions v5 or v9 and the IPFIX packet on all interfaces it configured for both IPv4 and IPv6. Like pmacctd daemon, it collects packets from the interfaces and can send or replicate this to third-party collectors.

pmbgpd  This is a standalone BGP collector daemon and can work as the passive iBGP and eBGP neighbor and can log real-time BGP traffic on regular intervals to the back-end systems (MySQL, Druid).

Configuration file for pmbgpd (BGP data collector)

The pmbgpd service supports 4-byte ASNs, IPv4 and IPv6. When you store the data in a customized MySQL table you can mix and match BGP primitives with other primitives.

pmbgpd is a standalone BGP collector daemon but it can be integrated with one of the above plugins. This thread solution is suitable for correlating BGP data with other data sources. The configuration file can be saved as a bgp.conf file and kept under the directory pmacct.

   $ mkdir -p /etc/pmacct
     $ vim /etc/pmacct/bgp.conf
 daemonize: true
 nfacctd_port: 9999
 nfacctd_time_new: true
 
plugins: mysql
 
 sql_optimize_clauses: true
 sql_dont_try_update: true
 sql_multi_values: 1024000
 
 sql_history_roundoff: m
 sql_history: 5m
 sql_refresh_time: 300
 sql_table: acct_bgp
 
 bgp_daemon: true
 bgp_daemon_ip: X.X.X.X
 bgp_daemon_max_peers: 100
 bgp_aspath_radius: 3
 bgp_follow_default: 1 
 nfacctd_as: bgp
 bgp_peer_src_as_type: map
 bgp_peer_src_as_map: /etc/pmacct/peers.map
 
 plugin_buffer_size: 10240
 plugin_pipe_size: 1024000
 aggregate: tag, src_as, dst_as, peer_src_as, peer_dst_as,
 peer_src_ip, peer_dst_ip, local_pref, as_path
 
 pre_tag_map: /path/to/pretag.map
 maps_refresh: true
 maps_entries: 3840 

Once the collector (pmbgpd) has been configured, the next step is to configure the router to export the traffic samples to the BGP collector (pmbgpd) on the Linux virtual machine. Cisco routers running iOS can be used in the following configuration: 

 router bgp xxx
  neighbor X.X.X.X remote-as xxx
  neighbor X.X.X.X update-source Loopbackxxx
  neighbor X.X.X.X version 4
  neighbor X.X.X.X send-community
  neighbor X.X.X.X route-reflector-client
  neighbor X.X.X.X description nfacctd 

6. Configuration file for nfacctd (NetFlow data collector)

      $ mkdir -p /etc/pmacct
      $ vim /etc/pmacct/netflow.conf
          interface: eth0                        
 sql_max_writers: 25                        
 sql_optimize_clauses: true
 pmacctd_force_frag_handling: true
 debug: true
 pmacctd_stitching: true
 plugins: mysql[c]                                
 aggregate[c]: src_host, dst_host, proto
 sql_db[c]: pmacct                               
 sql_table[c]: acct             
 sql_passwd[c]: xxxxx                  
 sql_user[c]: root                                 
 sql_refresh_time[c]: 6                       
 timestamps_secs: true            
 plugin_buffer_size: 10240
 plugin_pipe_size: 1024000
 snaplen: 1500 

7. Running PMACCT

You need to run the following command as a root user under the directory where you install pmacct:

nfacctd –f  /etc/pmacct/bgp.conf   // 

If you configure either pmbgpd or nfacctd, you can use the same command nfacctd followed by –f and the file name. The configuration file defines the type of data to be collected from the router.

To stop running the service, use the following command:

kill –INT nfacctd

You can verify your results on MySQL table’s acct for Netflow and acct_bgp for BGP data.

8. Configuring RPKI/ROA information

This configuration can be done either using downloaded RPKI data (ROAs) or via rpki_rtr_cache. The first method requires all ROAs to be downloaded and the second method can be used for live ROA updates.

ROAs can be imported from the rpki_roas_file config directive from a file into the RIPE NCC Validator format, as below:

 {
 "roas": [
     {"asn": "AS0", "maxLength": 11, "prefix": "80.128.0.0/11", "ta": "ripe"},
     {"asn": "AS0", "maxLength": 128, "prefix": "2001:12f8::/48", "ta": "lacnic"},
     {"asn": "AS0", "maxLength": 128, "prefix": "2001:43f8::/48", "ta": "afrinic"},
 ); 

The data in the above format can be downloaded from:  https://rpki.gin.ntt.net/api/export.json 

Once the rpki_roas_file or rpki_rtr_cache are defined, the configured traffic daemons above (pmbgpd and nfacctd) can be used to correlate the traffic with BGP and RPKI. The configuration file is as follows:

 nfacctd_as: longest
 nfacctd_net: longest
 ! < .. >
 !
 ! rpki_roas_file: /path/to/roas.json  // file can be downloaded and saved to file.
 !
 rpki_rtr_cache: 127.0.0.1:9999
 ! rpki_rtr_cache_version: 1 
 !
 bgp_daemon: true
 bgp_src_roa_type: bgp
 ! < .. > 
 !
 plugins: print[foo]  // Here the data will be printed to a file.
 !
 aggregate[foo]: peer_src_ip, peer_dst_ip, src_host, dst_host, src_roa, dst_roa  // user can add more attributes like src_as, dst_as, etc.
 ! < .. > 

The ROA field will be populated with one of the following values:

u — Unknown

v — Valid

i — Invalid

V — Invalid with a covering ‘Valid’ prefix

U —Invalid with a covering ‘Unknown’ prefix

Best current practice recommends having multiple RPKI/ROA validators such as ROUTINATOR, and RIPE NCC VALIDATOR installed and configured in the testbed.

Shinoj Pittandavida is a Network and Systems Engineer at IE Domain Registry CLG.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Leave a Reply

Your email address will not be published. Required fields are marked *

Please answer the math question * Time limit is exhausted. Please click the refresh button next to the equation below to reload the CAPTCHA (Note: your comment will not be deleted).

Top