Use Monitoring Interfaces and Packet Forwarding

Introduction

This guide explains how to configure the ExtraHop system to analyze network traffic using both monitoring interfaces and packet forwarding (RPCAP). This guide is intended for ExtraHop users with firmware version 3.10 and later. If you have firmware version 3.9, refer to ExtraHop Guide: Simultaneously Using Monitoring Interfaces and Packet Forwarding (Version 3.9).

Licensing the ExtraHop System

  1. Once the ExtraHop appliance has booted, browse to the Admin UI (https://<extrahop_management_ip>/admin). The following screen appears.

  2. Review the license agreement, select I Agree, and click Submit.

  3. On the Login screen, enter setup for the username.

    • The password for virtual appliances is default.

    • The password for physical appliances is the service tag number on the pullout tab on the front of the appliance.

  4. Go to the System Settings section and click License.

  5. Click Register to enter the product key.

  6. Enter the product key and then click Register. The ExtraHop system now contacts the license server and validates the product key. After the product key is validated, the license is downloaded.

    The following example shows a properly licensed ExtraHop on the License Administration page:

    The ExtraHop is now able to receive traffic from packet forwarders.

Enabling Packet Forwarding

You can use packet forwarding on 1GbE interfaces only. This reduces the packet processing resources available on other interfaces, which affects the total throughput. Refer to the following table for the maximum throughput of each ExtraHop appliance with monitoring interfaces and packet forwarding enabled.

ExtraHop Appliance Throughput Throughput with Packet Forwarding
EH2000v/EH3000 3GbE 3GbE
EH6000 10GbE 5GbE + 3GbE
EH8000 20GbE 10GbE + 3GbE

You can use interfaces 1 through 4 as management interfaces. You can use interfaces 2 through 4 for monitoring only, or you can disable them.

The following examples assume your network is DHCP-enabled. If your network does not support DHCP, refer to Appendix B to set a static IP address.

Example 1: 10GbE + 1GbE RPCAP (EH5000/6000/8000)

The following configuration captures traffic on the 10GbE interfaces normally, and also shows how to configure interface 2 to use RPCAP. ExtraHop does not recommend sending RPCAP traffic to the management interface because it may overload the management plane.

  1. Go to the Network Settings section and click Connectivity.

  2. Go to the Interface 2 section and click Change.

  3. Click the Interface Mode drop-down list, select Management Port + RPCAP/ERSPAN Target, click the DHCP checkbox, complete the IP Address and Netmask fields, and click Save. If your network does not support DHCP, refer to Appendix B to configure a static IP address.

  4. Go to the 10GbE Monitoring Interfaces section, click the Enabled checkbox, and click Save.

Example 2: Two 1GbE + 1GbE RPCAP (EH2000v/3000)

DHCP is enabled for interface 2 (the 1GbE interface closest to the management interface) and RPCAP traffic is sent to that address. The other 1GbE interfaces, interfaces 3 and 4, capture traffic through the physical feed.

The default configuration on the EH2000v/3000 uses interfaces 2 through 4 for monitoring. To use interface 2 for RPCAP, complete the following steps.

  1. Go to the Network Settings section and click Connectivity.

  2. Go to the Interface 1 section and verify the Interface mode uses the management port.

  3. Go to the Interface 2 section and click Change.

  4. Click the Interface Mode drop-down list, select Management Port + RPCAP/ERSPAN Target, click the DHCP checkbox, complete the IP Address and Netmask fields, and click Save. If your network does not support DHCP, refer to Appendix B to configure a static IP address.

  5. Go to the Interface 3 section and click Change.

  6. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

  7. Go to the Interface 4 section and click Change.

  8. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

Example 3: 10GbE + One 1GbE Monitoring + 2GbE RPCAP (EH5000/6000/8000)

By default, the ExtraHop appliance has 10GbE interfaces enabled. Interface 1 is used for management and interface 2 through 4 are disabled. In order to configure two interfaces for RPCAP, follow the steps below.

In this example, DHCP is enabled for interface 2 and interface 3, and RPCAP traffic is sent to those addresses. The 1GbE interface, interface 4, and the 10GbE interfaces capture traffic through the physical feed.

  1. Go to the Network Settings section and click Connectivity.

  2. Go to the Interface 2 section and click Change.

  3. Click the Interface Mode drop-down list, select Management Port + RPCAP/ERSPAN Target, click the DHCP checkbox, complete the IP Address and Netmask fields, and click Save.

  4. Go to the Interface 3 section and click Change.

  5. Click the Interface Mode drop-down list, select Management Port + RPCAP/ERSPAN Target, click the DHCP checkbox, complete the IP Address and Netmask fields, and click Save.

  6. Go to the Interface 4 section and click Change.

  7. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

  8. Go to the 10GbE Monitoring Interfaces section and ensure the Enabled checkbox is selected.

Example 4: 10GbE + Three 1GbE Monitoring (EH5000/6000/8000)

By default, the ExtraHop appliance has 10GbE interfaces enabled. Interface 1 is used for management and interface 2 through 4 are disabled. In order to configure three interfaces for monitoring, follow the steps below.

In this example, DHCP is enabled for the management interface, interface 1, and all interfaces capture traffic through the physical feed.

  1. Go to the Network Settings section and click Connectivity.

  2. Go to the Interface 1 section and verify the Interface mode uses the management port.

  3. Go to the Interface 2 section and click Change.

  4. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

  5. Go to the Interface 3 section and click Change.

  6. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

  7. Go to the Interface 4 section and click Change.

  8. Click the Interface Mode drop-down list, select Monitoring Port (receive only), and click Save.

  9. Go to the 10GbE Monitoring Interfaces section and ensure the Enabled checkbox is selected.

Opening Ports on the Firewall

The following ports must be open for packet forwarder traffic to reach the ExtraHop:

  • TCP ports 80 and 443 inbound to ExtraHop: These ports are used to download the installer. If opening these ports is difficult, you can copy the installer to each rpcapd machine manually. Refer to Installing the High-Speed Packet Forwarder on the Server Sending Traffic.

  • TCP/UDP port 2003 inbound to ExtraHop: By default, RPCAP will function correctly on port 2003 alone, but you may configure other ports as needed.

    By default, the ExtraHop system accepts RPCAP forwarded packets on port 2003. If you configure a port other than 2003 for the packet forwarder, you must modify the default ExtraHop configuration to listen on that port.

Monitoring Servers Using RPCAP

To monitor servers using RPCAP, you must do the following:

  • Ensure the high-speed packet forwarder is enabled on the ExtraHop appliance. Refer to Appendix C for optional settings.

  • Install the high-speed packet forwarder on the servers sending traffic.

  • Analyze packet-forwarding traffic in the ExtraHop Web UI.

Installing the High-Speed Packet Forwarder on the Server Sending Traffic

(Linux) You must run the packet forwarder command on each server to be monitored in order to forward packets to the ExtraHop system.

  1. Run the following command to download the packet forwarder on the server:

    export RPCAP_HOST_IP=<extrahop_management_ip>

    curl --connect-timeout 10 --fail \

    -k "http://$RPCAP_HOST_IP/tools/install-rpcapd.sh" > \

    install-rpcapd.sh

    Replace <extrahop_management_ip> with the ExtraHop system’s interface 1 (management) IP.

    Some PDF viewers may add extra newlines when copying and pasting commands. Ensure the text has copied correctly before running the command. To retrieve the command from the ExtraHop Admin UI, go to https://<extrahop_ip>/admin/capture/rpcapd/linux/.
  2. Run the following command to install and run the packet forwarder on the server:

    sudo sh ./install-rpcapd.sh <extrahop_rpcap_target_ip> <extrahop_rpcapd_port>

    Replace <extrahop_rpcap_target_ip> with the IP addresses on the ExtraHop system’s interface that is listening for the remote packet capture. This may be the same as the management IP if the instance has only one interface. You can look up IP addresses in two ways:

    • Run the CLI command show ip interface on the ExtraHop system.

    • In the Admin UI, go to Network Settings and click Connectivity.

    Replace <extrahop_rpcapd_port> with the port used for the packet forwarder, which is port 2003 by default.

Notes:

To start, stop, restart, reload, or check the status of the packet forwarder, run the command

sudo /etc/init.d/rpcapd {start|stop|status|restart|force-reload}

To view packet forwarder messages, run the command

tail /var/log/messages or tail /var/log/syslog

To run the packet forwarder manually for debugging purposes only, run the command

sudo /opt/extrahop/sbin/rpcapd -a <extrahop_rpcap_target_ip>,<extrahop_rpcapd_port>-n –v

To run the packet forwarder on servers with multiple interfaces, refer to Appendix E.

(Windows) You must run the packet forwarder command on each server to be monitored in order to forward packets to the ExtraHop system.

  1. Open a PowerShell shell with Administrator privileges on the Windows server.

  2. Run the following installation command to change the PowerShell execution policy:

    set-executionpolicy unrestricted

  3. Run the following command to download the packet forwarder on the server:

    (new-object system.net.webclient).downloadfile("http://<extrahop_management_ip>/tools/install-rpcapd.ps1", "install-rpcapd.ps1");

    Some PDF viewers may add extra newlines when copying and pasting commands. Ensure the text has copied correctly before running the command. To retrieve the command from the ExtraHop Admin UI, go to https://<extrahop_ip>/admin/capture/rpcapd/windows/.
  4. Run the following command to install the packet forwarder on the server:

    .\install-rpcapd.ps1 -MgmtIp <extrahop_management_ip> -RpcapIp <extrahop_rpcap_target_ip> -RpcapPort <extrahop_rpcapd_port>

    Replace <extrahop_management_ip> with the ExtraHop system’s interface 1 IP.

    Replace <extrahop_rpcap_target_ip> with the IP addresses on the ExtraHop system’s interface that is listening for the remote packet capture. This may be the same as the management IP if the instance has only one interface. You can look up IP addresses in two ways:

    • Run the CLI command show ip interface on the ExtraHop system.

    • In the Admin UI, go to Network Settings and click Connectivity.

    Replace <extrahop_rpcapd_port> with the port used for the packet forwarder, most commonly port 2003.

Notes:

To set the PowerShell execution policy back to the default, run the command

set-executionpolicy restricted

To start, stop, restart, or check the status of the packet forwarder, run the command

{start-service|stop-service|restart-service|get-service} rpcapd

To view packet forwarder messages, open the Event Viewer, click Windows Logs, and select Application. In the Application panel, sort by source and scroll down to rpcapd.

When reinstalling rpcapd, if a message appears that rpcapd is being used by another process, make sure the Event Viewer is closed.

To run the packet forwarder manually for debugging purposes only, run the command

“C:\Program Files\rpcapd\rpcapd” -a <extrahop_rpcap_target_ip>,<extrahop_rpcapd_port> -n –v

To run the packet forwarder on servers with multiple interfaces, refer to Appendix E.

Analyzing Packet Forwarding Traffic in the ExtraHop Web UI

To find out how much forwarded traffic the ExtraHop system is receiving, complete the following steps.

  1. Log in to the ExtraHop Web UI (https://<extrahop_management_ip>/extrahop) and click the Settings button in the top right corner.

  2. Click System Health to get more information about the packet forwarding traffic. This page displays a Packets and Throughput graph for each packet forwarder connected to the ExtraHop system.

    The RPCAP Packet and Throughput graphs contain four metrics:

    • Encapsulation: The total number of RPCAP encapsulation packets received by the ExtraHop system.

    • Tunnel Eligible: Total number of packets eligible to be forwarded to the ExtraHop system.

    • Tunnel Sent: Total number of RPCAP-tunneled packets forwarded to the ExtraHop system.

    • Tunnel Received: Total number of RPCAP-tunneled packets received by the ExtraHop system.

      The Tunnel Eligible, Tunnel Sent, and Tunnel Received values are equal if the ExtraHop system is receiving and processing all the packets sent by the server. If they are not, use the following reference for troubleshooting:

    • If Tunnel Sent is less than Tunnel Eligible, the server is not able to forward out all the traffic. This may indicate that packet forwarding requires more processing or outbound bandwidth resources on the server. Consider separating the forwarding process onto a separate CPU or allocating a dedicated interface for forwarding traffic.

    • If Tunnel Received is less than Tunnel Sent, the ExtraHop system is not receiving all the traffic forwarded by the server. This may be due to network congestion or insufficient resources on the ExtraHop system. If you suspect it is the latter, contact ExtraHop Support.

    Appendix A: Advanced Troubleshooting

    In the example below, Tunnel Eligible is 11.77M, peer sent is 11.11M, and processed is 7.8M. This means the ExtraHop system is seeing 7.8M out of 11.77M packets, or 66% of the traffic from this server running rpcapd.

    In the example above, eth0 of 10.0.2.65 had 11.77M (Tunnel Eligible) packets to forward. Ideally, the ExtraHop system would have processed all 11.77M packets. However, the ExtraHop system processed only 11.11M (Tunnel Sent) packets. This should match the number above, Tunnel Eligible, of 11.77M.

    Possible reasons for the drops:

    • Rpcapd is not pulling packets from libpcap fast enough.

    • Symptom: krnldrops in the rpcapd stats.

    • Solution: Increase the libpcap buffer size in the rpcapd parameters, -z, as explained later.

      • The network is saturated, so rpcapd is blocked and waiting to send packets.

    • Symptom: High RPCAP protocol throughput. Check the RPCAP protocol throughput graph. (Refer to the picture with both graphs above. The max is 473.11Mbps.)

    • Symptom: Displays as eagain or enobufs in the rpcapd stats.

    • Solution: Make sure RPCAP traffic is using a different interface than the monitored interface. For example, when monitoring eth0, make sure rpcapd is connecting to the ExtraHop system over eth1.

      In the example above, the number of packets the ExtraHop system processed is lower than the number of Tunnel Sent packets. The number of processed packets is 7.80M. This should match the number of Tunnel Sent packets, which is 11.11M.

      Possible reasons for the drops:

    • The RPCAP packets were dropped before reaching the ExtraHop system. The packet forwarder sends packets over UDP.

    • Symptom: High RPCAP protocol throughput.

    • Solution: The ExtraHop system supports 1G of throughput on each interface. Refer to the throughput table earlier in this document to ensure traffic being sent to the ExtraHop system does not exceed the maximum throughput.

    • The ExtraHop is receiving too much traffic.

    • Symptom: Go to the System Health page and view the ExtraHop Rpcap Thread X Packets graphs. If a thread is processing near 100K packets per second, the thread could be saturated and is receiving too many packets.

    • Solution: Spin up another ExtraHop appliance and point half of the rpcapd forwarders at the new appliance.

    Appendix B: Configuring a Static IP Address

    The ExtraHop system is delivered with DHCP enabled, but you can instead configure a static IP address. If your network does not support DHCP, you can set a route manually to determine where the traffic goes. To manually set a route:

    1. On the Network Settings for Interface X page, ensure the IP Address and Netmask fields are complete and saved, and click Edit Routes.

    2. In the Add Route section, complete the Network and Via IP fields, and click Add.

    3. Repeat the previous step for each route you want to add.

    4. Click Save, and the Admin UI redirects to the Network Settings for Interface X page.

      The default time server setting is pool.ntp.org. To configure the time servers manually, refer to the System Settings section of the ExtraHop Admin UI Users Guide.

    Appendix C: Configuring Additional RPCAP Settings

    By default, the ExtraHop system accepts RPCAP forwarded packets on port 2003. The servers using the packet forwarder are directed to forward all traffic as denoted by the wildcard in the Interface Address column.

    (Optional) To specify another port, subnet, or filter, complete the following steps.

    1. Go to the RPCAP Settings section and click Change.

    2. Change and modify the settings on the Add RPCAP Port Definition page.

      • Port: Specifies the listening port on the ExtraHop system. Each port must be unique for each interface subnet on the same device. Different subnets across servers can use the same port. This is both a TCP and UDP port. If you are configuring multiple software taps and multiple software tap listeners, the payload may traverse a range of UDP ports. The range consists 16 ports starting with the port defined.

      • Interface Address: Specifies a subnet to choose the interface from which to forward packets. If the server has multiple interfaces that match the interface address, the first interface on the server sends traffic to the ExtraHop system.

      • Interface Name: Indicates the interface on the packet-forwarding server from which to forward packets.

        You must specify an interface address or an interface name. If you specify both, then both criteria will apply.
      • Filter: Specifies the traffic to forward using Berkeley Packet Filter syntax. For example, TCP port 80 forwards only TCP traffic on port 80, and not TCP port 80 forwards only non-TCP traffic on port 80.

      Appendix D: Using Additional RPCAP Installation Commands

      Linux

      To download the packet forwarder manually, complete the following steps.

      1. Go to https://<extrahop_management_ip>/tools.

      2. Download the rpcapd file for Linux.

      3. Install it on the server by running the command

        sudo sh ./install-rpcapd.sh <extrahop_rpcap_target_ip> <extrahop_rpcapd_port>

      Windows

      To download the packet forwarder manually, complete the following steps.

      1. Go to https://<extrahop_management_ip>/tools.

      2. Download and unzip the rpcapd file for Windows.

      3. Open PowerShell and navigate to the directory containing the unzipped files.

      4. Run the command ./install-rpcapd.ps1 -InputDir . -RpcapIp <extrahop_rpcap_target_ip> -RpcapPort <extrahop_rpcapd_port>

      Appendix E: Monitoring Multiple Interfaces

      For servers with multiple interfaces, the packet forwarder can be configured to forward packets from a particular interface or from multiple interfaces by editing its configuration file on the server.

      Linux

      To edit the configuration file, complete the following steps.

      1. After installing the packet forwarder, open the configuration file on the server: /opt/extrahop/etc/rpcapd.ini

        After installation, the configuration file contains this text or similar:

        ActiveClient = 10.0.0.100,2003

        NullAuthPermit = YES

      2. Modify the existing ActiveClient line and create an ActiveClient line for each additional interface to be monitored. Specify each interface by its interface name or IP address.

        ActiveClient = <extrahop_management_ip>, <extrahop_rpcapd_port>, ifname=<interface_name>

        or

        ActiveClient = <extrahop_management_ip>, <extrahop_rpcapd_port>, ifaddr=<interface_address>

        <interface_name> is the name of the interface from which you want to forward packets.

        <interface_address> specifies the IP address of the interface from which the packets are forwarded. <interface_address> may be either the IP address itself, such as 10.10.1.100, or a CIDR specification (network IP address/subnet prefix length) that contains the IP address, such as 10.10.1.0/24.

        For every ActiveClient line, the packet forwarder independently forwards packets from the interface specified in the line.

      The following is an example of the configuration file specifying two interfaces using the interface name:

      ActiveClient = 10.10.6.45, 2003, ifname=eth0

      ActiveClient = 10.10.6.45, 2003, ifname=eth1

      NullAuthPermit = YES

      The following is an example of the configuration file specifying two interfaces using the interface IP address:

      ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.1.100

      ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.2.100

      NullAuthPermit = YES

      The following is an example of the configuration file specifying two interfaces using CIDR specifications that contain the interface IP address:

      ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.1.0/24

      ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.2.0/24

      NullAuthPermit = YES

    3. Save the configuration file and restart the packet forwarder. Run the command

      sudo /etc/init.d/rpcapd restart

      To reinstall the packet forwarder after changing the configuration file, run the installation command and replace <extrahop_management_ip> and <extrahop_rpcapd_port> with the –k flag in order to preserve the modified configuration file. For example:

      sudo sh ./install-rpcapd.sh –k

    Windows

    To edit the configuration file, complete the following steps.

    1. After installing the packet forwarder, open the configuration file on the server: C:\Program Files\rpcapd\rpcapd.ini

      After installation, the file contains this text or similar:

      ActiveClient = 10.0.0.100,2003

      NullAuthPermit = YES

    2. Modify the existing ActiveClient line and create an ActiveClient line for each additional interface to be monitored. Specify each interface by its interface name or IP address. For every ActiveClient line, the packet forwarder will independently forward packets from the interface specified in the line:

      ActiveClient = <extrahop_management_ip>, <extrahop_rpcapd_port>, ifname=<interface_address>

      or

      ActiveClient = <extrahop_management)ip>, <extrahop_rpcapd_port>, ifaddr=<interface_name>

      <interface_address> specifies the IP address of the interface from which the packets are forwarded. <interface_address> may be either the IP address itself, such as 10.10.1.100, or a CIDR specification (network IP address/subnet prefix length) that contains the IP address, such as 10.10.1.0/24.

      <interface_name> is the name of the interface from which the packets are forwarded. The name is formatted as \Device\NPF_{<GUID>}, where <GUID> is the globally unique identifier (GUID) of the interface. For example, if the interface GUID is 2C2FC212-701D-42E6-9EAE-BEE969FEFB3F, the interface name is \Device\NPF_{2C2FC212-701D-42E6-9EAE-BEE969FEFB3F}.

      The following is an example of the configuration file specifying two interfaces using the interface IP address:

    ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.1.100

    ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.2.100

    NullAuthPermit = YES

    The following is an example of the configuration file specifying two interfaces using CIDR specifications that contain the interface IP address:

    ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.1.0/24

    ActiveClient = 10.10.6.45, 2003, ifaddr=10.10.2.0/24

    NullAuthPermit = YES

    The following is an example of the configuration file specifying two interfaces using the interface name:

    ActiveClient = 10.10.6.45, 2003, ifname=\Device\NPF_{2C2FC212-701D-42E6-9EAE-BEE969FEFB3F}

    ActiveClient = 10.10.6.45, 2003, ifname=\Device\NPF_{3C2FC212-701D-42E6-9EAE-BEE969FEFB3F}

    NullAuthPermit = YES

  3. Save the configuration file and restart the packet forwarder by running the command restart-service rpcapd

    To reinstall the packet forwarder after changing the configuration file, run the installation command and replace –RpcapIp and –RpcapPort with the -KeepConfig flag in order to preserve the modified configuration file. For example:

    .\install-rpcapd.ps1 -MgmtIp <extrahop_management_ip> -KeepConfig

    or

    .\install-rpcapd.ps1 –InputDir . -KeepConfig

Appendix F: Tuning the Packet-Forwarding Configuration

To use the installer to change the packet-forwarding configuration, connect to the server running rpcapd and download the installer:

  1. Run the command:

    curl --connect-timeout 10 --fail –k

    'https://<extrahop_management_ip>/tools/install-rpcapd.sh' > install-rpcapd.sh

    Replace <extrahop_management_ip> with the ExtraHop system’s management IP address.

  2. Each of the following tweaks adjusts the DAEMON_ARGS in /etc/init.d/rpcapd. You can edit this file directly instead of using the installer. Afterward, remember to restart rpcapd by running the command sudo /etc/init.d/rpcapd restart

If Tunnel Eligible does not match Tunnel Sent, and the network is not saturated, try increasing the libpcap buffer size. The default size is 16MiB (16777216B). Try increasing the size to 32MiB (33554432B) or even 64MiB (67108864B) if the server has enough memory.

  • rpcapd parameter: -z 33554432

  • (Linux) sudo sh ./install-rpcapd.sh <extrahop_rpcap_target_ip> <port_in_Running_Config> -z 33554432

  • (Windows) .\install-rpcapd.ps1 -MgmtIp <extrahop_management_ip> -RpcapIp <extrahop_rpcap_target_ip> -RpcapPort <extrahop_rpcapd_port> -DaemonAddlArgs “-z 33554432”

To make the packet forwarder print stats every 30 seconds, run the following command:

  • rpcapd parameter: -S

  • (Linux) sudo sh ./install-rpcapd.sh <extrahop_rpcap_target_ip> <port_in_Running_Config> -S

  • (Windows) .\install-rpcapd.ps1 -MgmtIp <extrahop_management_ip> -RpcapIp <extrahop_rpcap_target_ip> -RpcapPort <extrahop_rpcapd_port> -DaemonAddlArgs “-S”

Make sure to stop this process when you are finished or it will fill the syslog.
Published 2017-12-11 19:17