ExtraHop Metrics

Get started with metrics

The ExtraHop system provides you with 4,000 built-in metrics for over a dozen protocols. A metric is a measurement of observed network behavior. Because the ExtraHop system provides so many L2 through L7 protocol metrics to view, it can be challenging to know where to find the metrics that are most important to you.

Metrics

Metrics are measurements of network behavior. Metrics help you to gain visibility into what is happening in your network in real-time. In the ExtraHop system, metrics are calculated from wire data, and then associated with devices and protocols. The ExtraHop system provides a large number of metrics, which you can explore from protocol pages in the Metrics section of the ExtraHop Web UI. You can also search for metrics in the Metric Catalog, in the Metric Explorer, and by searching for metrics by source and then protocol.

Protocol pages for sources and groups
To investigate metrics for individual devices or groups of devices, visit the Metrics section of the Discover appliance Web UI. Select an application, network, device, or device group, and then explore metrics by protocol for your selected source. Drill down or pivot on interesting metrics and devices.
Metric Catalog
To look up information about a custom or built-in metric, visit the Metric Catalog. Enter the name of a metric that you are looking for in the search field. The Metric Catalog will display an entry for each metric that provides information about metric parameters, such as the source type, metric type, and detail relationships. This information can be useful for writing API queries and adding metric variables in a text box widget. You also can delete and edit custom metrics through the Metric Catalog.
Metric Explorer
To visualize how metric data changes over time, visit the Metric Explorer. The Metric Explorer is a tool for creating and editing charts. Access the Metric Explorer when you click Create Chart from the command menu in a protocol page, or when you edit dashboard charts.

With the Metric Explorer, you can add metrics to a chart and immediately visualize how metric data behaves for the selected time interval. The preview pane dynamically updates as you make metric and chart type selections. You have the option to then save your chart to a dashboard.

Top-level metrics and detail metrics

Top-level metrics and detail metrics provide different views about network activity. Top-level metrics provide you with a big-picture value to help identify what is happening on your network. You can then drill down on a top-level metric to view detail metrics. Detail metrics provide you with a value for a specific key (such as a client or server IP address), which gives you insight into how a specific device, method, or resource is affecting the network.

On the Dashboard page, you can configure charts to display either top-level or detail metrics. On protocol pages, you can view top-level metrics and then drill down to view detail metrics.

A top-level, or base, metric gives you a sum of data for a specified time period. The ExtraHop system provides you with real-time updates about top-level metrics. For example, you can view the total number of HTTP requests sent by a device for the last 30 minutes.

In the following figure, a bar chart displays the top-level metric for the total number of HTTP requests that were sent to a web server during a specific time period.

Detail metrics provide you with a metric value for a specific key, such as a client IP address, server IP address, URI, hostname, referrer, certificate, or method. For example, you can drill down on the total number of HTTP requests to break out the number of requests sent per client. When you drill down, the ExtraHop system provides you with a topnset of detail metrics. A topnset is the top 1,000 key-value pairs calculated for the time interval you specify in the Time Selector. A topnset is not a complete data set because a topnset only represents the key-values that are recorded for a specific aggregation roll up (based on a specified time interval), and is limited to up to 1,000 keys per topnset.

In the following figure, a Bar chart displays detail metric values by client (which is a key) after drilling down on the top-level metric for HTTP requests. Specifically, the chart displays eight clients that sent the most requests to the web server during a specific time period. You can configure charts to show you either a specific key or a specific number of keys from a topnset.

Note:When drilling down to detail metrics from protocol pages, you might encounter a chart that includes more than 1,000 keys. Some charts in the ExtraHop system combine topnsets for multiple detail metrics into one table. You can then sort keys by detail metrics. For example, when you drill down on the responses metric by URI from the Metrics > Applications > All Activity > Web page, the chart displays both a topnset of URIs for HTTP Responses and a topnset of URIs for Server Processing Time.

Types of top-level metrics

Each top-level metric in the ExtraHop system is classified into a metric type. Understanding the distinctions between metric types can help you configure charts or write triggers to capture custom metrics. For example, a heatmap chart can only display dataset metrics.

Count
The number of events that occurred over a specific time period. You can view count metrics as a rate or a total count. For example, a byte is recorded as a count, and can either represent a throughput rate (as seen in a time series chart) or total traffic volume (as seen in a table). Rates are helpful for comparing counts over different time periods. A count metric can be calculated as a per-second average over time. When viewing high-precision, or 1-second, bytes and packet metrics, you can also view a maximum rate and minimum rate. Count metrics include errors, packets, and responses.
Distinct count
The number of unique events that occurred during a selected time interval. The distinct count metric provides an estimate of the number of unique items placed into a HyperLogLog set during the selected time interval.
Dataset
A distribution of data that can be calculated into percentiles values. Dataset metrics include processing time and round trip time.
Maximum
A single data point that represents the maximum value from a specified time period.
Sampleset
A summary of data about a detail metric. Selecting a sampleset metric in a chart enables you to display a mean (average) and standard deviation over a specified time period.
Snapshot
A data point that represents a single point in time.
Tip:Visit the Tip of the Week: Metric Types post on the ExtraHop community forum.

Wire data

With wire data, the ExtraHop system passively collects a copy of unstructured packets through a port mirror or tap and stores the data in the appliance datastore. The copied data goes through real-time stream processing, which transforms the packets into structured wire data through the following stages:
  1. TCP state machines are recreated to perform full-stream reassembly.
  2. Packets are constructed into flows.
  3. The structured data is analyzed and processed in the following ways:
    1. Transactions are identified
    2. Devices are automatically discovered by MAC and IP address and then classified by their activity.
    3. Metrics are generated and associated with protocols and sources, and the metric data is then aggregated into metric cycles. For more information, see the Sources and groups section.
      Note:Aggregation roll ups, also referred to as metric cycles, help determine the granularity of metric data in time series analyses. For more information, see the Time interval and data roll up section.
  4. As new metrics are generated and stored, and the datastore becomes full, the oldest existing metrics are overwritten according to the first-in first-out (FIFO) principle.

Time interval and data roll up

The time interval you specify in the Time Selector determines how metric data is aggregated, or rolled up into buckets. The aggregation roll up, also known as a metric cycle, provides information about the level of granularity for count metric data shown in time-series charts.

Note:The aggregation roll up is not displayed in list and value charts.

The following table provides information about how data will be rolled up for a specific time interval.

Time Interval Aggregation Roll Up (if available) Notes
Less than six minutes 1-second A 1-second roll up is only available for custom metrics and for the following built-in throughput and packet metrics:
  • Network source > Network Bytes (total throughput)
  • Network source > Network Packets (total packets)
  • Device source > Network Bytes (combined inbound and outbound throughput by device)
  • Device source > Network Bytes In (inbound throughput by device)
  • Device source > Network Bytes Out (outbound throughput by device)
  • Device source > Network Packets (combined inbound and outbound packets by device)
  • Device source > Network Packets In (inbound packets by device)
  • Device source > Network Packets Out (outbound packets by device)
120 minutes or less 30-second If a 30-second roll up is not available, a 5-minute or 60-minute roll up will be displayed.
Between 121 minutes and 24 hours 5-minute If 5-minute roll up is not available, a 60-minute roll up will be displayed.
Greater than 24 hours 60-minute  
Note:If you have an extended datastore that is configured for 24-hour metrics, a specified time interval of 30 days or longer will display a 24-hour aggregation roll up.

Explore how metric data changes over different time intervals with the Time Selector.

Drill-down functionality

The ExtraHop system enables you to easily drill down from a top-level metric into specific details about the devices, methods, or resources associated with that metric. Drilling down helps you to investigate root causes of interesting network activity. Click on the metric value in a table or chart and then select the detail, such as IP address, method, error, resource, or status code, that you want to know more about.

For example, if you see a large number of DNS request timeouts in a dashboard chart, you can drill down to see the top DNS servers in your environment that are associated with that metric, as well as the number of times each server did not respond to repeated DNS requests.

Note:When you drill down on a top-level metric by a key (such as a client IP address or resource), the ExtraHop system calculates a topnset of up to 1,000 metric value-key pairs. These metric value-key pairs are known as detail metrics. Learn more in the Top-level metrics and detail metrics and Explore drill-down metrics by key sections.
Note:If your Discover appliance is connected to an ExtraHop Explore appliance, you can drill down on a protocol or metric to view stored records. Learn more in the Records section.

Explore drill-down metrics by key

Drilling down on a metric lets you explore metric values broken down by key, such as client IP address, server IP address, methods, or resources. On the drill-down metric page, there are several ways to explore detail metrics (which are metric value-key pairs), which help you to learn how a specific device, method, or resource is linked to network activity.

The following figure shows all the available options for exploring detail metrics:

Filter results
You can filter drill-down results in the following ways:
  • Type in the filter field to dynamically filter results
  • Click the Any Field drop-down list and make a selection
  • Choose an operator to define parameters for your filter:
    • Select = to perform an exact string match.
    • Select to perform an approximate string match. The ≈ operator supports regular expression.
      Note:To exclude a result, enter a regular expression. Learn more in the Regular expression filter examples section.
    • Select > or to perform a match for values greater than (or equal to) a specified value.
    • Select < or to perform a match for values less than (or equal to) a specified value.

Click Add filter to save the filter settings. You can save multiple filters for one query. Saved filters are cleared if you select another key from the Details section in the left pane.

Observe changes over time in the chart
You can observe how a metric value changed over the selected time interval in the chart above the table. Select an individual row or multiple rows to change chart data. Hover over data points in the chart to view more information about each data point.
Pivot to more data
You can view metric values for different keys by clicking key names in the Details section in the left pane. If available, click a device name in the table to navigate to a Device page, which displays traffic and protocol activity associated with that device.
Adjust time interval and compare data from two time intervals
You can change the time interval in the Global Time Selector to view metric values from different time intervals. You can also perform a metric delta comparison from two different time intervals in the same table. Learn more in the Compare metric deltas section.
Note:The global time interval in the upper left corner of the page includes a blue refresh icon and gray text that indicates when the drill-down metrics were last polled. To reload the metrics for the specified time interval, click the refresh icon in the Global Time Selector display. Learn more in the Time interval and data roll up section.
Sort data in columns
You can sort by metrics to learn which keys are associated with the largest or smallest metric values. For example, when you drill down on HTTP responses by client for an HTTP server, you can sort on processing time to see which clients experienced the longest website load times. You can then click the host name to navigate to the Device page to learn more about the client.
Note:When you drill down on a response, request, or network byte metric, related metrics such as processing time are included in the table. For example, when you drill down on CIFS responses by files, related metrics such as goodput bytes and access time appear in the far right columns in the table.
Change data calculation for metrics
You can change the following calculations for metric values displayed in the table:
  • If you have a count metric in the table, click Count in the Options section in the left pane and then select Average Rate. Learn more in the Display rates or counts in a chart section.
  • If you have a dataset metric in the table, click Mean in the Options section in the left pane and then select Summary. When you select Summary, you can view the mean and the standard deviation.
Export data
You can download a PDF, CSV, or Excel file with all the drill-down results by right-clicking on the table.

In the Discover and Command appliances, you can search for metrics by source, such as an application, device, flow network, or a group of devices, and then by protocol.

To view all the metrics collected by the ExtraHop system for a source, click Metrics from the top menu.

The following fields and controls are available in the left pane:

Sources
Enables you to view metrics for applications, devices, and networks.
Groups
Enables you to view metrics for groups of devices based on their activity. You can also create a custom device group and view metrics associated with those groups. You can also view trouble groups, which are automatically generated based on network traffic. Trouble groups represent a collection of devices that meet specific criteria indicating potential problems.
Alerts
Enables you to view the alert history for saved alerts.

As you specify the sources or group of devices that you want to explore in the left pane, the center pane displays the available protocol pages associated with your selection. Select a protocol page to display all available metrics in tables, lists, and charts.

Tip:When looking for metrics that are relevant to you, start with a device you are already familiar with. Search for a device in the global search field, and then click on the device name. On the device protocol page, you can pivot across protocols in the left pane and view top-level metrics and charts in the center pane to see all of the network activity associated with your device.
Note:If there are no results for a metric, or if a protocol appears to be missing, the ExtraHop system did not detect any related activity or traffic for that source. To learn more about how the ExtraHop system collects metrics, see the Data sources in the ExtraHop system section.

Dashboards are another way to explore the metrics that are most relevant to you. For example, you can plan and build a custom dashboard with charts that highlight your top devices and most critical network traffic. For more information, see the Get started with dashboards section.

Search metrics by protocol

After selecting a source, such as an application, device, or group, or network object from the Metrics section, you can access several protocol metrics for your selected source. Click on a protocol in the left pane, such as AAA, CIFS, DNS, or Web (HTTP) to display metrics and built-in charts.

Metric Catalog

The Metric Catalog enables you to view information about built-in and custom metrics in the ExtraHop system. This information can be useful for writing API queries and adding metric variables in a text box widget. You also can delete and edit custom metrics through the Metric Catalog.

Note:For information on modifying custom metrics in the Metric Catalog, see the Custom metrics section.

Search for metrics by typing a keyword or metric name into the filter field. Click the command menu next to the Type to filter... field for sorting options.

When you select a metric from the search results, information about that metric displays in the right pane in the following sections.

The Parameter section provides the following information about the selected metric:

Source
Specifies how the metric was created. If the source is builtin, the selected metric is one of 4,000 metrics already built into the ExtraHop system. If the source is trigger, the selected metric is a custom metric created by a user.
Metric
Specifies the API parameter name of the selected metric.
Source Type
Specifies the source type (application, device, or network) that is associated with the metric.
Metric Type
Specifies the type of base (top-level) metric (such as a count or dataset) that is associated with the selected metric.
Type
Specifies whether the metric is a base (top-level) metric or a detail metric.

The Display section provides information about how the selected metric will be displayed in the Web UI:

Name
Specifies the display name of the metric.
Units
Specifies the unit for the metric, if available.
Description
Specifies a description for the metric.

The Detail Relationships section provides the name of the base (top-level) metric or detail metric that is related to the selected metric.

The REST API Parameter section provides an example of a JSON query structure for the selected metric with API parameters.

Metric Explorer

The Metric Explorer is a tool for creating and editing dashboard charts. In the Metric Explorer, you can add metrics to a chart and immediately view how metric data will appear in a preview pane. The preview pane dynamically updates as you make metric and chart type selections, which enables you to explore and change how your data is visualized in a dashboard.

The Metric Explorer provides the following components for configuring a chart.

Metrics tab
Add metric sets to your chart. A metric set consists of a single type of source and one or more metrics.
Note:You can add multiple metric sets to display in a single chart. For example, one metric set can contain a mix of device sources (such as servers) and another metric set can contain application sources.
Source
In the Source section, add a metric source, such as an application, device, group, or network capture.
Metric
In the Metric section, search for and select compatible metrics for the source. Depending on the type of metric you select, data calculation options are listed underneath the metric name. For example, when you select whether you select a count metric type (such as HTTP Requests or Network Bytes), you can select to display a rate or count. When you select a dataset metric type (such as Server Processing Time), you can choose to display a summary of percentile values or a specific percentile value.
Detail
Optionally, in the Detail section, drill down to display detail metrics for the entire metric set in your chart.
Time interval
Specify a time interval to view presentation of network data in your chart. You can change the time interval, but your changes will not be saved with other chart configurations. You must change the time interval with the Time Selector in your dashboard.
Analysis tab
Add a static threshold line and a dynamic baseline to your chart.
Options tab
Select configuration options, such as changing a chart title, units, and labels.
Preview section
Preview how metric data will display in your chart. The chart dynamically updates as you add and remove metrics from the Metrics tab.
Chart section
Select a chart type to display data. Toggle between different time-series and non time-series chart types to determine which chart is the best choice for visualizing the data you are interested in.
Note:Some charts have specific metric requirements.

The following figure displays a configured line chart. The chart is displaying data for one metric set, Application Metrics, the average rate data calculation for the HTTP Responses metric, and detail metric keys for client IP addresses.

Sources and groups

In the ExtraHop system, a metric is a measurement of observed network behavior. Metrics are generated from network traffic, and then each metric is associated with a source, such as an application, device, or network. When you select a source from the Metrics section of the Web UI, or in the Metric Explorer when building a chart, you can view metrics associated with that source. Each source provides access to a different collection of metrics.

Select from the following sources and groups as you configure dashboard widgets or navigate across protocol pages.

Applications

Applications are user-defined containers for metrics that are associated with multiple devices and protocols. These containers can represent distributed applications on your network environment. In the ExtraHop system, applications are created through triggers, which are custom scripts. Triggers can collect metrics across multiple types of network traffic to capture information with cross-tier impact. For example, if you want a unified view of all the network traffic associated with a website—from web transactions to DNS requests and responses to database transactions—you can write a trigger to create a custom application that contains all of these related metrics. The Applications page displays the default All Activity application, which contains metrics for every device on your network, and all custom applications created through triggers.

Note:For information about creating applications with triggers, see the Triggers section.

The table on the Applications page provides the following information:

Name
The name assigned to the application through the trigger. Click on an application name to navigate to the metric page associated with the application.
Capture
On a Discover appliance, this column identifies the wire data feed that is the source for the application. On a Command appliance, this column identifies the Discover appliance where the application was created.
Description
A user-defined description that is assigned to the application through a trigger.
You can filter applications by application name, network capture, or application description. To adjust filter criteria, click Any Column or .
Note:By default, the search feature performs a substring search on the value entered in the filter text box. For example, if you submit the letter z for a name search, the search results return all devices with a letter z in the name, regardless of position.
Note:Learn more by taking the Creating and Using App Containers training.

Drill down on metrics from application protocol pages

When you see an interesting high-level metric about protocol activity on an Application page, you can drill down to investigate which factors are linked to that activity. Drilling down on a metric lets you explore metric values broken down by key, such as client IP address, server IP address, methods, or resources.

Note:You can drill down on metrics from Device, Device Group, and Flow Network protocol pages. Learn more in the Drill down on metrics from device protocol pages and Drill down on flow network metrics sections.

Drill down from an application protocol page

  1. Click Metrics.
  2. Click Applications in the left pane.
  3. Click an application name.
  4. Click a protocol in the left pane.
  5. If you find an interesting metric, click the metric value. A drill-down menu appears with keys. The diagram below shows a drill-down menu for HTTP status codes with the client IP address key.
For an example of a drill-down workflow that explores DNS protocol data, see the Metrics walkthrough: finding DNS failures.

Devices

Devices are objects on your network with a MAC address and IP address that have been automatically discovered and classified by the ExtraHop system. Metrics are available for every discovered device on your network. An L2 device has a MAC address only; an L3 device has an IP address and MAC address.

Note:For more information about how devices are automatically discovered and classified by the ExtraHop system, see the Device discovery section.

You can filter devices on the Devices page by name, MAC address, VLAN, IP address, or node, and then click Search. To adjust filter criteria, click Any Column or Any Device.

Note:By default, the search feature performs a substring search on the value entered in the filter text box. For example, if you submit the letter z for a name search, then the list of devices returned by the search includes all devices that have a letter z in the name, regardless of position.

The table on the Devices page contains the following columns:

Name
The primary name for the device on the network. Click a device name to navigate to a metrics page where you can view protocol metrics.
MAC Address
The MAC address is a unique identifier for the device network interface.
VLAN
The virtual Local Area Network (VLAN) for the device. VLAN information is extracted from VLAN tags, if the traffic mirroring process preserves the tag on the mirror port.
Note:By default, this column does not appear on Discover appliances and values are set to null on Command appliances. VLAN information is displayed for devices only if the devices_accross_vlans setting is set to true through the running configuration file.
IP Address
The last IP address the device communicated from on the network.
Node
(Command appliance only) The name of the Discover appliance associated with the device.
Discovery Time
The date and time when the device was first discovered.
Description
An optional, user-defined description.

The counter at the bottom of the table identifies the number of devices currently displayed in the table. The table can display up to 1,000 devices per page.

Find a device

You can search for a specific client, server, router, or other device in the ExtraHop system by any device attribute, such as IP address, hostname, or custom name.

There are several ways to find a device:

  • From global search, where you can search by any device attribute.
  • From the device list page in the Metrics section of the Web UI, where you can filter search results.
  • From an activity group page, where you can find a device based on the type of protocol activity. Learn more in the Find a device within an activity group section.

This procedure shows you how to find a device from the device list page. You can search for a device by entering a value in the search field, or filter search results by device attribute. You can also sort on any column title.

  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then click Devices in the left pane.
    Note:By default, the search feature performs a substring search typed into the search field. For example, if you submit the letter z for a name search, then the list of devices returned by the search includes all devices that have a letter z in the name, regardless of position.
  3. Optional: Filter devices in the search results by device attributes. Click Any Column to select one of the following device attribute categories:
    Any Column
    Matches a substring in any device element.
    Name
    Matches a substring in the device name. Learn more about device names and how to change them.
    MAC address
    Matches a substring in the device MAC address.
    VLAN
    Matches a substring in the device Virtual Local Area Network (VLAN) tag.
    IP address
    Matches a substring in the device IP address. The IP address criteria can include CIDR notation in IP address or subnet prefix length format. For example, 10.10.0.0/16 for IPv4 networks or 2001:db8::/32 for IPv6 networks.
    Node
    (Command appliance only) Matches a substring in a connected Discover appliance name.
    Tag
    Matches a substring in the user-defined device tag.
    Type
    Matches a substring to a specified device attribute type that you select from the drop-down list. Select from the following options:
    Activity
    Specify active metrics. For example, selecting Activity: HTTP Server returns devices with HTTP server metrics, and any other device with the custom type set to HTTP server.
    Device Type
    Specify a device type, such as gateway, firewall, load balancer, file server, and custom device types.
    Class
    Specify a device class, such as node, remote, and custom devices.
    Vendor
    Matches a substring in the device vendor name as determined by the Organizationally Unique Identifier (OUI) lookup.
  4. Optional: Filter by L2 or L3 devices. Click All Devices and select one of the following categories:
    L2 device
    An L2 device in the ExtraHop system has a MAC address only. ExtraHop automatically creates a device based on a MAC address, and all activity is tracked against that device.
    L3 device
    An L3 device in the ExtraHop system has an observed IP address that comes from local traffic or from traffic coming from a router.
    Note:To learn more about these types of devices, see the Device discovery section.

Find peer devices talking to a specific device

From a device or device group protocol page, you can drill down by Peer IPs to find information about devices that are actively communicating with other devices or device groups.

Find performance metrics for peer devices
  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then select Device, Activity Group, or Device Group in the left pane.
  3. Click Overview or Network in the left pane.
  4. In the Details section near the upper right corner of the page, click Peer IPs.
    A list of peer devices appears, which are broken down by IP address, hostname (if available), the average number of bytes per second, and the average number of packets per second. The ExtraHop system determines hostname for peer devices by passively monitoring naming protocol activity, such as DNS, DHCP, or NetBIOS. In the Details section in the left pane, you can select different network protocols to view different performance metrics.
    Tip:You can convert the average rate of network byte and packet metrics to a total count for the selected time interval. In the Options section in the left pane, click Average Rate for an individual metric and then select Count.
Find network latency metrics for peer devices
  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then select Device, Activity Group, or Device Group in the left pane.
  3. Click TCP in the left pane.
  4. In the Details section near the upper right corner of the page, click Peer IPs.
    A list of peer devices appears, which is broken down by IP address, hostname (if available), round trip time, accepted connections, and initiated connections. Round trip time is a measurement of total network latency as data is transferred between devices. Accepted connections are the number of TCP connections sent by the peer device. Connected connections are the number of TCP connections initiated by the device with the peer device.

Change the name of a device

The ExtraHop system automatically names devices by passively monitoring naming protocol traffic (DNS, DHCP, NETBIOS, CDP). If naming protocol traffic is not observed for a device, the device name instead displays the IP address for L3 devices or the MAC address for L2 devices. In either condition, you can replace the automatic device name with a custom name. The custom name will appear throughout the ExtraHop system.

Note:The ExtraHop system does not perform DNS lookups for device names. The ExtraHop system derives the DNS name for a device by observing DNS traffic over wire data. Learn more in the Device discovery section.
  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and select Device in the left pane.
  3. Find a device and click the device name. A Device page appears, which displays traffic and protocol metrics associated with the device.
  4. In the Overview section in the upper left corner, click Properties.
  5. Click Display custom name.
  6. Type a custom name in the field.
  7. Click Save.

Change a device role

The ExtraHop system monitors all of the protocol activity associated with a device from wire data. Based on certain types of traffic and protocol activity that was observed, the ExtraHop system then assigns a role to a device. You can replace the automatic or default device role with a manually assigned role.

For example, the ExtraHop system assigns the WWW server role to devices associated with sending HTTP responses.

The following device roles can be automatically assigned to a device:

Gateway: Assigned to an L2 device when a large amount of unique IP addresses (past a certain threshold) are associated with the device. The device name will include the router name (for example, “Cisco B1B500”).

Database (DB) server: Assigned to a device when there are database responses sent from this device.

File server: Assigned to a device when there are NFS and CIFS responses sent from this device.

WWW server: Assigned to a device when there are HTTP responses sent from this device.

Auto: Assigned to a device by default.

The following device roles can be manually assigned to a device:

Load balancer: Assign to your device if you want to classify the device as a reverse proxy that helps distribute traffic across multiple servers.

Firewall: Assign to your device if you want to classify the device as a network security device that monitors incoming and outgoing network and blocks traffic according to security rules.

To change or manually assign a device role:

  1. Log into the Web UI on the Discover appliance.
  2. Find a device either through global search or the Device page in the Metrics section of the Web UI. Click the device name. A Device page appears, which displays traffic and protocol activity for the selected device.
  3. In the upper right corner, click Properties.
  4. In the Device Role section, click the drop-down list and select a role.
  5. Click Save.

Drill down on metrics from device protocol pages

When you see an interesting top-level metric about protocol activity on a Device, a Device Group, or an Activity Group page, you can drill down to investigate which factors are linked to that activity. Drilling down on a metric lets you explore metric values broken down by key, such as client IP address, server IP address, methods, or resources.

  1. Click Metrics and then click Device, Device Group, or Activity Group in the left pane.
  2. Click a device or group name.
  3. Click on a metric value or a metric label in the chart legend. A menu appears.
    Note:You can also click a drill-down shortcut button in the Details section in the upper right corner of the page.
  4. In the Drill down by… section, select a key. A drill-down metrics page with a topnset of metric values by key appears. You can view up to 1,000 key values in a topnset.
    Note:If a View More link appears at the bottom of a chart, click View More to drill down on the metric displayed in the chart.

Networks

The Networks page provides information about network capture and flow network sources discovered by the ExtraHop system. A network capture is the entry point into network devices and virtual LANs (VLANs) that are detected from wire data by the ExtraHop system. A flow network is a network device, such as a router or switch, that sends information about flows seen across the device. A flow network can have multiple interfaces. Click a network capture or flow interface to view protocol metrics about the data from those traffic sources.

This section describes the source attributes available on the Networks page.

Note:When starting from the Network page, keep in mind that information collected through network captures and flow networks are determined by port mirror configuration or flow configuration in the Admin UI.

In addition, if your organization manages multiple capture points or remote flow networks through the Command appliance, the Networks page displays a table of all capture points and flow networks for your entire networking environment.

The Network table provides the following information:

Name
The name of a network capture or flow interface. Click the drop-down icon next to a network capture or flow network to display VLANs or flow interfaces, respectively.

Click a network capture or VLAN to navigate to a protocol page and view top-level and detail protocol metrics.

Click a flow network or flow interface to navigate to a summary page and view top-level metrics.

Select the checkbox next to a network capture or flow network to access the following configuration options:

Devices
The number of devices in the network capture. This attribute does not apply to flow networks.
IP Address
The IP address of the Discover appliance responsible for the network capture or flow network.
Description
An optional description of the network obtained from the network capture or VLAN protocol page.
You can filter network captures and flow networks by name, devices, IP address, description, or application description. To adjust filter criteria, click Any Column or .
Note:By default, the search feature performs a substring search on the value entered in the filter text box. For example, if you submit the letter z for a name search, then the list of devices returned by the search includes all devices that have a letter z in the name, regardless of position.

View configured network captures

View a list of configured network captures and their associated VLANs. To configure network capture settings, see the Capture section in the Admin UI Guide.

  1. Click Metrics and then click Networks.
  2. In the content pane, the table provides the following information about the capture:
    Name
    Displays the name of the capture.
    Type
    Displays the whether the network is a Wire Network or a Flow Network.
    Devices
    The number of devices in the network capture.
    IP Address
    Displays the IP address of the Discover appliance providing wire data for the capture.
    Description
    An optional detailed description of the network. Click on the capture name to open an overview page where you can modify this description.
    Interface Speed
    This field is not applicable to network captures.
    Note:On a Command appliance, the table displays the capture for each Discover appliance.
  3. Click the drop-down arrow next to the capture name to see a list of associated VLANs.
  4. Click the capture name or VLAN name to view more information and built-in charts.

Change the name of a network capture or VLAN

  1. Click Metrics and then click Networks.
  2. In the content pane, select the checkbox for a single capture or VLAN.
  3. Click Rename in the upper right corner.
  4. Type a new name and then click OK.

View configured flow networks

View a list of configured flow networks and their attributes. To configure a flow network, see the Flow Networks section in the Admin UI Guide.

  1. Click Metrics and then click Networks.
  2. In the content pane, for each configured flow network, the table provides the following information:
    Name
    Displays the name of the flow network.
    Type
    Displays the whether the network is a Wire Network or a Flow Network.
    Devices
    This field is not applicable to flow networks.
    IP Address
    Displays the IP address of the flow network device.
    Description
    This field is not applicable to flow networks.
    Interface Speed
    Displays the speed of the network interface on the remote device. This setting requires SNMP to be enabled through the Admin UI of the Discover appliance.
  3. Click the drop-down arrow next to the flow network name to see a list of flow interfaces and their attributes.
  4. Click the flow network name or flow interface name to view built-in charts on summary pages.

Change the name of a flow network

  1. Click Metrics and then click Networks.
  2. In the content pane, select the checkbox for a single flow network or flow interface in the table.
  3. Click Rename.
  4. Type a new name and then click OK.

Assign triggers to a flow network or flow interface

  1. Click Metrics and then click Networks.
  2. In the content pane, select the checkbox for a flow network or flow interface.
  3. Click Assign Trigger.
  4. Select the checkboxes for the triggers that you want to assign to the flow network, and then click Assign Triggers.

Set a custom speed for a flow interface

Bandwidth utilization metrics about flow interfaces are calculated through the interface speed. If you have configured SNMP for your flow network, by default, the interface speed is set through SNMP. However, you can also set a custom speed for your flow interfaces on the ExtraHop Discover appliance.

For information on how to configure SNMP for your flow network, see the Flow networks section of the ExtraHop Admin UI Guide.

  1. Click Metrics and then click Networks.
  2. In the content pane, click on a flow network to expand the list of available flow interfaces.
  3. Select the checkbox for the flow interface you want to customize.
  4. Click Automatically set interface speed through SNMP.
  5. Select Manually set interface speed.
  6. Type a custom speed for the interface, and then click Set Interface Speed.

Flow network summary pages

Summary pages provide built-in charts for the IP traffic that exits and enters through remote network devices, such as NetFlow traffic, for configured flow networks and flow interfaces.

Summary pages contain three regions with charts for top-level, summary data:

Overview
View the total amount of network throughput (average bits per second) traveling in and out of either the flow network or flow interface. For flow interfaces only, you can also view the bandwidth utilization of throughput traveling in and out of the flow interface.
Protocols
IP flow packets are typically transferred across the flow network or flow interface by UDP and TCP ports. View the total amount of traffic for each protocol and port that is transferring data in the bar chart. In the line chart, compare protocol and port throughput changes over time. You can also hover over the protocol and port name in the legend of the line chart to isolate protocol data in the chart.
Endpoints
View the amount data that devices (endpoints) are sending and receiving across the flow network or flow interface in the following ways.
  • Top talker charts display individual devices with the highest volume of throughput.
  • Top sender charts display the throughput for devices sending data.
  • Top receiver charts display the throughput for devices receiving data.
  • Conversation charts display the highest volume of throughput by flow between two devices (endpoints).
  • Compare the top talkers, senders, and conversations in the bar chart.
  • In the line chart, compare changes in throughput activity for individual devices over time.
  • Hover over a device IP address in the line chart to isolate throughput data in the chart.

To create your own dashboard charts from the summary page or preview data in different chart types, see the following sections:

To configure the time interval for a specific region, see the Time Selector section.

Modify flow network chart display

Summary pages for flow networks and flow interfaces have built-in charts, and you can modify a limited set of options on the page to see how your data might display differently. If you like a different view, you can then create a custom dashboard chart with those settings.

  1. Click Metrics, and then click Networks.
  2. Click on a flow network or flow interface.
  3. On the Summary page, click a chart title.
  4. Select from the chart type options at the bottom of the page or select from the available Metrics options in the left pane.
    Note:Not all chart types are compatible for all metrics options. Warning icons appear when the chart type is incompatible with the selected option.
  5. Optional: If you want to save the modified chart to a new custom dashboard, click the command menu in the upper right corner of the chart, and select Create a chart from….
    1. Edit the chart as needed.
    2. When you finish configuring the chart, click Add to Dashboard.
    3. Either select New Dashboard to create a new dashboard with your chart, or select an existing dashboard name listed beneath New Dashboard.

Create a chart from flow network data

If you find interesting NetFlow traffic on your flow network or flow interface summary pages, you can modify the built-in charts and save the modified charts to an existing or new dashboard.

  1. Click Metrics, and then click Networks.
  2. Click on a flow network or flow interface.
  3. On the Summary page, click the command menu in the upper right corner of a chart.
  4. Select Create a chart from….
  5. Edit the chart as needed.
  6. When you finish configuring the chart, click Add to Dashboard:
    • Select New Dashboard to create a new dashboard.
    • Select an existing dashboard name from the list below New Dashboard.

Drill down on flow network metrics

When you see an interesting top-level metric about network activity on a Flow Network or Flow Interface page, you can drill down to investigate which factors are linked to the activity. Drilling down on a metric lets you explore metric values broken down by peer IP addresses, protocols and ports, conversations, and sender and receiver IP addresses.

For example, on a Flow Network page in the Endpoints region, click the chart legend label to drill down by peer IP addresses.

  1. Click Metrics and then click Networks in the left pane.
  2. Click a flow network or flow interface name.
  3. Click a metric value or a metric label in the chart legend. A menu appears.
  4. In the Drill down by… section, select a key. You will navigate to a page that contains a table of metric values by key from a topnset.
    Note:You can view up to 1,000 key values in a topnset.
    Note:For drill-down metric values, which are not polled automatically, you will see the snapshot of the global time interval, which includes a blue refresh icon and gray text that indicates when the metric or record query was last polled. To reload the metrics for the specified time interval, click the refresh icon in the Global Time Selector display.

Drill down on network capture and VLAN metrics

When you see an interesting top-level metric about network activity on a Network capture or VLAN page, you can identify which devices are linked to that activity.

Note:For information about how to drill down on metrics from a flow network or flow network interface page, see the Drill down on flow network metrics section.
  1. Click Metrics.
  2. Click Networks in the left pane.
  3. Click a network capture or VLAN interface name.
  4. Click a network layer in the left pane, such as L3 or L7 Protocols. Charts that display metric values for the selected time interval appear. For most protocols and metrics, a Device table also appears at the bottom of the page.
  5. Click the chart data, which updates the list to display only the devices that are associated with the data.
  6. Click a device name. A Device page appears, which displays traffic and protocol activity associated with the selected device.

Activity groups

Activity groups contain devices that are automatically grouped together based on their network traffic. A device with multiple types of traffic might appear in more than one activity group. Activity groups make it easy to identify all the devices associated with a protocol, or determine which devices were associated with protocol activity during a specific time interval.

You can filter activity groups by name or count. To adjust filter criteria, click Any Column or the includes () operator.
Note:By default, the search feature performs a substring search on the value entered in the filter text box. For example, if you submit the letter z for a name search, then the list of devices returned by the search includes all devices that have a letter z in the name, regardless of position.
The table on the Activity group page provides the following information:
Name
Specifies the name of the activity group, which is based on the type of protocol activity of the devices in the group. The name also indicates whether the group contains client or server devices. Click on the name of an activity group to navigate to a page where you can view protocol metrics for that group. For example, click the TCP Devices activity group to see the L4 TCP protocol metrics page, which lists all of the devices with TCP traffic.
Count
Specifies the number of devices that belong to the activity group.

Find a device within an activity group

Activity groups contain devices that are automatically grouped together based on their protocol traffic. Activity groups can be a quick way to find a device associated with a protocol, or discover decommissioned devices that are still active. For example, you can look at the HTTP Servers activity group and locate all devices that have sent HTTP responses over the wire.

  1. Log into the Web UI on the Discover appliance.
  2. From the top menu, click Metrics and then select Activity Group in the left pane.
  3. Select an activity group, such as HTTP Servers. The Device Group page for the activity group appears.
  4. In the top right corner of the page, click Group Members. A table with all of the devices within the activity group appears.
  5. Click on a device name in the table. The Device page appears, which displays traffic and protocol metrics associated with the selected device. All protocol activity associated with the device is listed in the left pane.
  6. Click HTTP Server in the left pane to see the number of HTTP responses sent by this device.

Device groups

Device groups help you track metrics across designated devices, typically grouped by their activity. Device groups can be static or dynamic. While you must manually add devices to a static device group, a dynamic device group automatically adds devices to the group that matches criteria that you define. The criteria can be a hostname, IP address, MAC address, or any of the filter criteria listed for the device on the Devices page. For example, you can create a dynamic group and then configure a rule to add all devices within a certain IP address range to that group automatically.

From the Metrics section in the Web UI, the Device Groups table includes the following information about all the device groups created in the ExtraHop system:

Name
Displays the name of the device group. The icon next to the name indicates whether the device group is a static or dynamic group. Click on the name to view the Assignments page for the device group, which has criteria for the group among other settings.
Count
Displays the number of devices that belong to the device group.
Description
Displays an optional, user-defined description for the device group.
Note:Learn more by taking the Working with Device Groups online training.

Next steps

Create a static device group

  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then click Device Groups.
  3. Click Add.
  4. In the Name field, type a name for the new group.
  5. For the Group Type option, select Static (add and remove devices manually).
  6. In the Description field, add information about this device group.
  7. Click OK.
    Your device group is now created. Next, assign devices to your group.
  8. Click Devices in the left pane.
  9. Find a device and then select the checkbox next to the devices you want to add to your group.
  10. At the top of the device table, hover over the icons and click the Add to Group icon.
  11. Select your new device group from the Select a group... drop-down list.
  12. Click Add to Group.
Add devices to a static device group

After creating a static device group, you must manually add devices to the group. If you know which device or devices you want to add, you can quickly add them from a list. Or you can first view information about a specific device and then add that device to the group

Add one or more devices from a list
  1. Click Metrics and then click Devices in the left pane.
  2. Find a device and then select the checkbox next to the devices.
  3. At the top of the device table, hover over the icons and click the Add to Group icon.
  4. Select your device group from the Select a group... drop-down list.
  5. Click Add to Group.
View device information and then add a device
  1. Click Metrics and select Device in the left pane.
  2. Find a device and then click on the device name. The traffic and protocol metrics associated with the device appear.
  3. Click Properties in the upper right corner of the page.
  4. Click the Group tab.
  5. In the Include in These Static Groups section, click the search field. A drop-down list of static device group appears. Type the name of the device group into the search field to filter results.
  6. Select the device group name.
    Note:You can also create a new static device group at this step. Type a name for the new static device group and press Enter. The device is automatically added to the new device group.
    Note:If the device also belongs to a dynamic device group, the device group name is displayed in the Included in Dynamic Groups section. For more information, see the Device groups section.
Remove devices from a static device group

You can only remove devices from a static device group individually from the Device page.

  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then click Device Groups in the left pane.
  3. Click on the device group name.
  4. In the upper right corner, click Group Members.
  5. Click the device name that you want to remove. The Device page appears.
  6. In the upper right corner, click Properties.
  7. Click the Groups tab.
  8. Locate the device group name in the Include in these Static Groups section, and then click the red x icon.

Create a dynamic device group

  1. Log into the Web UI on the Discover appliance.
  2. From the top menu, click Metrics and then click Device Groups in the left pane.
  3. Click Add.
  4. In the Name field, type a name for the new group.
  5. In the Group Type field, select the option for Dynamic with criteria.
  6. Click the drop-down list and select from one of the following criteria:
    IP address
    Adds devices that match a substring in the device IP address in IPv4, IPv6, or CIDR block.
    Name
    Adds devices that match a substring in the device name.
    Node
    (Command appliance only) Matches a substring in the node name.
    MAC address
    Adds devices that match a substring in the device MAC address.
    Tag
    Adds devices that match a substring in the user-defined device tag.
    Type
    Select the following options from the drop-down lists:
    Activity
    Adds devices that are associated with active metrics. For example, selecting Activity: HTTP Server adds devices with HTTP server metrics, and any other device with the custom type set to HTTP Server.
    Device type
    Adds devices that are classified as a gateway, firewall, load balancer, file server, or custom device.
    Note:When the Include custom devices checkbox is selected, custom devices will be added to your group.
    Class
    Adds devices that are classified as node, remote, custom, or pseudo.
    Vendor
    Adds devices that match a substring in the device vendor name as determined by the Organizationally Unique Identifier (OUI) lookup.
    VLAN
    Adds devices that match a substring in the device VLAN tag. VLAN information is extracted from VLAN tags, if the traffic mirroring process preserves them on the mirror port.
  7. In the Description field, add a brief description for the new group.
  8. Click OK.
Modify dynamic device group criteria

A dynamic device group automatically adds devices to the group that match criteria that you define. The criteria can be a hostname, IP address, MAC address, device tag, or any of the device attributes listed for the device on the Devices page.

  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then click Device Groups in the left pane.
  3. Click the device group name.
  4. In the upper right corner of the page, click Properties.
  5. In the Group Type section, complete one of the following steps:
    • Click the top drop-down field and select one of the following device attributes:
      Name
      Adds devices that match a substring in the device name.
      MAC address
      Adds devices that match a substring in the device MAC address.
      VLAN
      Adds devices that match a substring in the device Virtual Local Area Network (VLAN) tag. VLAN information is extracted from VLAN tags, if the traffic mirroring process preserves them on the mirror port.
      IP address
      Adds devices that match a substring in the device IP address in IPv4, IPv6, or CIDR block.
      Node
      (Command appliance only) Matches a substring in a connected Discover appliance name.
      Tag
      Adds devices that match a substring in the user-defined device tag.
      Type
      Select from the following options:
      Activity
      Adds devices that are associated with activity groups. For example, selecting Activity: HTTP Server adds devices with HTTP server metrics, and any other device with the custom type set to HTTP server.
      Device Type
      Adds devices that are classified as a gateway, firewall, load balancer, file server, or custom device.
      Note:When the Include custom devices checkbox is selected, custom devices are added to your group.
      Class
      Adds devices that are classified as node, remote, custom, or pseudo.
      Vendor
      Adds devices that match a substring in the device vendor name as determined by the Organizationally Unique Identifier (OUI) lookup.
    • In the second drop-down field, type the text you want to match for the dynamic group. For example, if you selected IP address from the list, type the IP address that you want to set as a criteria for this group in the field.
  6. Click Save.

Modify a device group name

You can only modify a device group name from a Device Group protocol page.

  1. Log into the Web UI on the Discover appliance.
  2. From the top menu, click Metrics and then click Device Groups in the left pane.
  3. Click the name of the device group you want to modify.
    A Device Group page appears.
  4. In the upper right corner, click Properties.
  5. In the Group Name field, type a new name for the device group.
  6. Click Save.

Modify a device group description

  1. Click Metrics and then click Device Groups.
  2. Click the name of the device group you want to modify.
    A Device Group page appears.
  3. In the upper right corner, click Properties.
  4. In the Group Description field, type a new description for the device group.
  5. Click Save.

View device group metrics

You can view all the protocol activity associated with a device group. You can also drill down by group member and navigate to a Device page to view metrics for an individual device.

  1. Log into the Web UI on the Discover appliance.
  2. Click Metrics and then click Device Groups in the left pane.
  3. The Device Groups list page displays all device groups on the appliance.
  4. Optional: To filter the contents of the table, select a field and an operator, and then enter a search term.
  5. Click the device group name. A Device Group page appears, which displays traffic and protocol metrics associated with the device group.

Trouble groups

The Discover appliance automatically generates trouble groups based on network traffic that meet specific criteria indicating potential problems.

The Trouble Groups table has the following information:

Name
Specifies the name of the trouble group.
Count
Identifies the number of devices that belong to this group.

Refer to the specific trouble group sections for the criteria that defines that group.

View trouble groups

To view details about the devices in a trouble group:

  1. Click Metrics and then click Trouble Groups.
  2. Click the trouble group name to view the list of devices in the group.
  3. On the device list page, click the device name.
When you click a device name from the device list page, you are redirected to the Devices page where device statics are displayed.

Aborted HTTP/DB transactions

Aborted HTTP/DB transactions indicate a high level of aborts during active HTTP or database transactions. Aborts are generally initiated by clients, so this might indicate that the server hangs on the response or does not complete the response in a timely manner.

Criteria Check for high levels of Requests Aborted or Responses Aborted
Devices Devices that show HTTP or DB server activity and are not gateways or load balancers
Update Hourly
Remedial Actions For HTTP transactions, check for URLs that take along time to process. For database transactions, check for long-running stored procedures

ADC SNAT pool too small

ADC SNAT pool too small indicates that a connection failed to initiate because the current device interpreted the SYN as belonging to a previous connection.

Criteria Check for any PAWS-Dropped-SYNs (In)
Devices Known ADCs only (based on MAC address OID lookup)
Update Hourly
Remedial Actions On the BIG-IP Application Delivery Controller (ADC), the SNAT pool size should be increased

ADC TCP connection throttling

ADC TCP connection throttling indicates that the connections are stalling in the Application Delivery Controller (ADC) and it is unable to keep up with the rate of data sent.

Criteria Check for Zero Windows (Out) as a factor of the number of established connections
Devices Known ADCs only (based on MAC address OID lookup)
Update Hourly
Remedial Actions On the BIG-IP Application Delivery Controller (ADC), the proxy_buffer_high setting in the TCP profile should be increased

Database server backups

Database server backups are caused by backups taking place over CIFS, NFS, or Veritas on active database servers.

Criteria Detect large amount of storage traffic exchanged from the server
Devices Devices that show CIFS, NFS, or TCP port 13724 activity (Veritas) and are not gateways or load balancers
Update Every 30 minutes
Remedial Actions Throttle down backups and schedule them during times with lower traffic

DNS missing entries

DNS missing entries might indicate a service availability problem.

Criteria Compare DNS NXDOMAINS responses with the total number of responses
Devices Devices that show DNS server activity and are not gateways or load balancers
Update Hourly
Remedial Actions If these queries are intended, add an entry to DNS. If not, find the clients making erroneous DNS requests and configure them to stop making these requests

Excessive CIFS metadata queries

Excessive CIFS metadata queries indicate a high level of file metadata queries compared to read/write activity (or "goodput") on a CIFS server.

Criteria Compare FSInfo to the number of Read and Write bytes
Devices Devices that show CIFS server activity and are not gateways or load balancers
Update Hourly
Remedial Actions Check clients that generate large numbers of CIFS for configuration issues that would cause them to perform an overly high level of directory scans

Excessive HTTP authorizations

Excessive HTTP authorizations should be checked for large numbers of HTTP authorization errors, which might indicate break-in attempts.

Criteria Check for 401 errors and compare them with the number of valid responses
Devices Devices that show HTTP server activity and are not gateways or load balancers
Update Hourly
Remedial Actions Log these HTTP authorization errors, as these errors might indicate break-in attempts

HTTP broken links indicate that a resource has been moved or deleted but the document might still points to the old location.

Criteria Check for 404s and compare it with the number of valid responses
Devices Devices that show HTTP server activity and are not gateways or load balancers
Update Hourly
Remedial Actions Track down the source of 404s

Path MTU mismatch

Path MTU mismatch displays the list of devices for which path MTU mismatch was detected. These devices are not respecting the Fragmentation Needed ICMP announcements.

Criteria Check for ICMP type 3 code 4
Devices All devices
Update Hourly
Remedial Actions Check documentation for devices that are not respecting path MTU announcements for configuration options

Problematic TCP offloading engine

Problematic TCP offloading engine. Indicates that the current device is sending too much data resulting in network congestion and dropped packets. This behavior has been seen with a number of TCP offloading engines.

Criteria Check for Bad Congestion Control (Out)
Devices NICs known to have problems (based on MAC address OID lookup)
Update Hourly
Remedial Actions Turn off TCP offloading

Server TCP connection throttling

Server TCP connection throttling is caused by server running out of buffer or CPU resources and throttling network connections as a result.

Criteria Check for the Zero Windows (Out) as a factor of the number of established connections
Devices Devices that are servers and are not gateways or load balancers
Update Every 30 minutes
Remedial Actions Check buffer sizes and CPU, and increase those resources, if necessary

SPAN oversubscription

SPAN oversubscription indicates that data coming over the SPAN port is incomplete. This can happen to data being dropped at the SPAN port due to oversubscription or microbursts.

Criteria Compare the desyncs to the number established connections
Devices All devices
Update Daily
Remedial Actions Filter down data coming over the SPAN port or use a larger capacity SPAN port

SSL Key Size < 2048

SSL key size < 2048 indicates a 1024-bit SSL key. In 2010, 1024-bit public keys have been declared insecure by NIST. As a result, certificate authorities are moving to 2048-bit keys.

Criteria Check for SSL public key size less than 2048 bits
Devices Devices that show SSL server activity and are not gateways
Update Hourly
Remedial Actions Deploy 2048-bit keys in place of potentially insecure ones

Virtual packet loss

Virtual packet loss indicates that a virtual instance is overwhelmed and cannot send packets out in a timely fashion. TCP interprets delayed ACKs as packet loss and sends less data.

Criteria Check for large numbers of RTOs coming from devices within virtualized environments
Devices Virtualized devices (based on MAC address OID lookup)
Update Hourly
Remedial Actions Provide more hardware resources to stressed VMs

Learn about the new 6.2 layout

We updated the Device and Device Group protocol pages in the ExtraHop Web UI to improve how you navigate and explore key system metrics. The Network and Application protocol pages will be updated in a future release.

Overview and Network Pages

Take a look at the sections below to see where your familiar icons are now located and to learn about changed and deprecated workflows.

Each device and device group has an Overview and Network page in the left pane. These pages now contain the following metrics and menu items that were previously available from the left pane.

New layout

  • L2 metrics, such as Packets, Throughputs, and Frame Counts, are now on the Network page.
  • L3 metrics, such as IP fragments, ICMP, and DSCP, are now on the Network page. Note that the Packet Count by Protocol chart is now the IP Protocols chart. Click Peer IPs from the upper right corner of the page to see the previous Peer Devices table.
  • L7 metrics are now on the Overview page, broken out by traffic in and traffic out. Click Peer IPs from the upper right corner of the page to see the previous Peer Devices table.
  • Alert History is now on the Overview page.
  • Multicast metrics are now on the Network page under Packet Distribution and are called Packet Types.
  • Properties and Assignments are now available from the upper right corner of the Overview page or through the command menu.

Client and Server Metrics

Instead of toggling between Client and Server in the Metric Type drop-down, you can now access client and server metrics pages from the left pane.

Old layout

New layout

Protocol Page Actions

Most of the action icons that appeared on the top of each protocol page have moved to the command menu in the top right corner of the page.

Important:You cannot add protocol pages to reports in the new layout. To add a protocol page to a report, click Switch to old layout in the lower left corner of the page and then add the page to your report through the previous workflow.

Old layout

New layout

Note:On the Command appliance, the menu option is Export to PDF.

Members in a Device Group

The list of members in a device group has moved from the bottom of the protocol pages to a DETAILS section on each protocol page. Click Group Members in the upper right corner of the page to display information about each member. Or, you can click on any metric and select Drill down by… > Group Member.

Old layout

New layout

Drill Down Options

The context-sensitive icons that appeared on the top of each protocol page have moved to the DETAILS section in the top right corner of the page. Note that Errors are now available in a value chart on the page and are no longer available through the link.

Old layout

New layout

Instead of clicking a value to drill down on a metric, click the value to see a menu with drill down options.

Old layout

New layout

Search Records and Packets

The Records and Packet search are still available from protocol pages, but are now in the top right corner of the page in the SEARCH section. These links only appear where records and packets are available.

Records are available in the Network, Server Activity, and Client Activity pages; packets are only available in the Network page. Both records and packets are also available by clicking a metric and selecting from the menu.

New layout

Changed and Deprecated Workflows

Some workflows were deprecated and can only be accessed by switching to the old layout or through a new workflow.

Device Names
Device names were previously changed through the top-level page for the device. Instead, click on the Overview page, and then click Properties in the top right corner to change the name.

New layout

Geomaps
You can no longer assign geomaps to a device and the geomaps assignments tab page was removed. Instead, click a count metric from a chart and then select an option from the menu for a drill down that is associated with an IP address (such as Server). A View on Map button is available below the chart and to the right.

New layout

Custom Pages
You cannot view custom pages for devices or device groups from the new layout. Instead, click Switch to old layout in the lower left corner of the page, and then view custom pages through the previous workflow.

Switch to old layout

We updated the Device and Device Group protocol pages in the ExtraHop Web UI to improve how you navigate and explore key system metrics. However, you might still need to access the old layout for certain workflows.

Revert to the old layout by completing the following steps.
  1. Click Metrics.
  2. Click Devices or Device Groups.
  3. Click the name of a device or device group from the list.
  4. In the lower left corner, click Switch to old layout.

Manage protocol data

If you discover interesting metric data from protocol pages, there are several ways to export and format data to share with others. For example, you can export data to an Excel or PDF file, add metrics to a dashboard, or generate activity maps and reports.

Export data to CSV

  1. Navigate to a protocol page by clicking Metrics and then select a source, such as an application, device, network, activity group, or device group.
  2. Right-click any table, chart, or metric on the page and select Export to CSV.

Export data to Excel

  1. Navigate to a protocol page by clicking Metrics and then select a source, such as an application, device, network, activity group, or device group.
  2. Right-click any table, chart, or metric on the page and select Export to Excel.
    Note:You cannot export data from heatmap or table charts to Excel.

Create a PDF of a protocol page

In the Command appliance, you can export a PDF file directly from a protocol page. In the Discover appliance, you can print a PDF file of a protocol page from your browser.

  1. Navigate to a protocol page by clicking Metrics and then select a source, such as an application, device, network, activity group, or device group.
  2. In the Command appliance, click the command menu in the upper right corner and then select Export to PDF.
  3. In the Discover appliance, click the command menu in the upper right corner and then select Print.
    Tip:To print through a keyboard shortcut, type pp.
    The print preview appears in a new window.
  4. Click Print Page and select PDF as a print option from your browser.

Create a chart from a protocol page

Protocol pages contain a large amount of metrics and data. While you cannot modify the charts on protocol pages, you can create a copy of an interesting chart on a protocol page and then add the copied chart to a dashboard. Your dashboard can be then modified and shared with other team members.

  1. Click Metrics and then select a source in the left pane.
  2. Find the chart that you want to copy. Click the chart title and select Create Chart. The Metric Explorer opens with the source and metric selected.
    Note:If you find a chart on an Application or Network Capture page, click Create Chart in the upper right corner of the page.
  3. Edit the chart as needed.
  4. Click Add to Dashboard:
    • Select New Dashboard to create a dashboard, and then click Create.
    • Select an existing dashboard from the list, and then click Close.

Pin a protocol page to a dashboard

  1. Navigate to a protocol page by clicking Metrics and then select a source, such as an application, device, network, activity group, or device group.
  2. Click Pin to Dashboards.
    The confirmation dialog box displays the name of the dashboard that the page was added to.
You can view the pinned protocol page in the My Dashboards folder in the left pane.

Create an activity map

An activity map is a directed graph that shows interconnections between objects on a network.

On an activity map, devices labeled in red indicate user-selected devices. Devices labeled in black indicate devices that were not selected, but have connections to the selected devices. A darker colored line between devices represents a connection with a high volume of traffic. A lighter colored line represents a connection with a low volume of traffic. Well-connected devices appear slightly larger and more centrally on the map.

  1. Navigate to a protocol page by clicking Metrics and then select a device or device group.
  2. In the upper right corner, click the command menu and select Generate Activity Map.
  3. Complete the following steps:
    1. Optional: Modify the default name of the activity map.
    2. Specify an output format, such as a PDF file.
    3. Select which activities to display.
    4. Optional: Write a description about the activity map.
  4. Click OK.

Sort metrics

On an application protocol page, if a metrics section on a protocol page contains a gear icon in the upper right corner, the metrics in that section can be sorted by key or value.

  1. Navigate to a protocol page by clicking Metrics and then select an application.
  2. Click the gear icon.
  3. Select Sort by Key or Sort by Value.

AAA

ExtraHop appliances collect metrics about Authentication, Authorization, and Accounting (AAA) activity.

AAA applications page

Application toolbar
The AAA application toolbar includes the following controls:
Errors
The chart displays the number of AAA errors. Mouse over the points to view a summary of a specific time or date. The table lists the AAA error messages and number of occurrences.
Clients
The chart displays the processing time for all clients. Mouse over the points to view a five-number summary of processing time (minimum, lower quartile, median, upper quartile, and maximum values). The orange bars represent a confidence interval around the median value bounded by the 25th and 75th percentile values.

The table lists client IP addresses, the host and device associated with each client as well as total time and processing time for each client. Mouse over the orange bars to view the mean time, standard deviation, and count for each metric.

Servers
The chart displays the processing time for all servers. Mouse over the points to view a five-number summary of processing time (minimum, lower quartile, median, upper quartile, and maximum values). The orange bars represent a confidence interval around the median value bounded by the 25th and 75th percentile values.
Application Details
Specifies the type of additional application information displayed. IP detail views display directly monitored IP addresses and IP addresses that appear through routed traffic. IP addresses that appear through routed traffic are preceded by the word via. Mousing over the counter next to each top-level metric opens a context menu with the following options in the drop-down list:
By Client IP
Displays application metrics by the client IP addresses.
By Server IP
Displays application metrics by the server IP addresses.

For example, Request Bytes is a top-level metric showing how many request bytes were transmitted in and out of the application within the selected time interval. Select By Client IP in the drop-down list while mousing over the Request Bytes counter to view which client IP addresses originated these requests.

L2-L4 Metrics
Contains the following metrics:
Request L2 Bytes
The number of L2 bytes associated with AAA requests.
Response L2 Bytes
The number of L2 bytes associated with AAA responses.
Request Packets
The number of packets associated with AAA requests.
Response Packets
The number of packets associated with AAA responses.
Request RTOs
The number of retransmission timeouts caused by congestion when clients were sending AAA requests. A retransmission timeout is a one-second stall in the TCP connection flow due to excessive retransmissions.
Response RTOs
The number of retransmission timeouts caused by congestion when servers were sending AAA responses. A retransmission timeout is a one-second stall in the TCP connection flow due to excessive retransmissions.
Request Zero Window
The number of zero window advertisements sent by AAA clients. A device advertises a zero window when it cannot process incoming data as quickly as it is arriving.
Response Zero Window
The number of zero window advertisements sent by servers while receiving AAA requests. A device advertises a zero window when it cannot process incoming data as quickly as it is arriving.
AAA Metrics
Contains the following metrics:
Requests
The number of AAA requests.
Responses
The number of AAA responses.
Errors
The number of AAA errors for the selected time interval.
Aborts
The number of aborted AAA sessions.
RADIUS Requests
The number of RADIUS requests.
Diameter Requests
The number of Diameter requests.
Methods
Displays the selected method types for the AAA client or server.
Transactions Per Second
Displays the number of protocol transactions per second as a function of time over the selected time interval. The chart is annotated with red data points to indicate errors. The volume of errors is denoted by the height of red bars under the chart. Click the red dot to see the number of errors that occurred at that time. Click and drag across the chart to select a particular region.
Response Time Breakdown
Displays the area chart containing median round-trip time, request transfer time, server processing time, and response transfer time over time in milliseconds. Click and drag across the chart to select a particular region.
Round-Trip Time (ms)
Displays the median round-trip time (RTT) in milliseconds (ms) from the current objects to clients as a function of time over the selected time interval. Vertical dotted lines indicate the upper and lower quartiles (75th and 25th percentiles) of the round-trip time metrics. Click and drag across the chart to select a particular region.
Congestion Requests: Goodput (bps) and RTOs
Displays goodput and RTOs into the object as a function of time over the selected time interval.
Congestion Responses: Goodput (bps) and RTOs
Displays goodput and RTOs out of the object as a function of time over the selected time interval.

Goodput is application-level throughput (the number of useful information bits) and RTOs are retransmission timeouts. The Congestion In and Out graphs show the relationship over time between the rate of good application throughput and RTOs. An increase in RTOs theoretically leads to a decrease in goodput due to TCP back-off and packet retransmissions. It is best to view these charts in a smaller window of time so the metrics taken over time are not rolled up or smoothed out. In a small timeframe (30 minutes or less), one could see a decrease in goodput associated with a large number of RTOs, assuming that most flows on the server during this time frame experience this behavior. If only one or two flows are affected by RTOs, then the decreased goodput correlation might be masked by superficially healthy flows.

AAA client page

AAA Summary

Total Transactions
This chart shows you when AAA errors occurred and how many responses the AAA client received. This information can help you see how active the client was at the time it received the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Metric Description
Responses The number of responses that the device received when acting as an AAA client.
Errors The number of response errors that the device received when acting as an AAA client.
Transactions
This chart displays the total number of AAA responses the client received and how many of those responses contained errors.
Metric Description
Responses The number of responses that the device received when acting as an AAA client.
Errors The number of response errors that the device received when acting as an AAA client.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The server processing time shows how long servers took to process requests from clients. Processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at the processing time, because this metric alone provides an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT and processing times are both high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

RTT only measures how long an immediate acknowledgement takes to be sent; it does not wait until all packets are delivered. Therefore, RTT is a good indicator of how your network is performing. If you see high processing times, but the TCP RTT is low, the issue is probably at the device-level. Check the network for latency issues if the TCP RTT and processing times are all both.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.



The processing time might be high because the server took a long time to transmit the response (possibly because the response was very large); however, the processing time could also be high because the response took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Metric Description
Server Processing Time When the device is acting as an AAA client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Time The time between when an AAA client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a client is acting slow, performance summary metrics can help you figure out whether the network or servers are causing the issue. These metrics show the median amount of time that servers took to process requests from the client versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the client is contacting slow servers. High TCP round trip times indicate that the client is communicating over slow networks.
Metric Description
Server Processing Time When the device is acting as an AAA client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Time The time between when an AAA client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

AAA Details

Top Methods
This chart shows which AAA methods the client called the most by breaking out the total number of requests the client sent by method.
Top Error Types
This chart shows which AAA error types the client received the most by breaking out the number of responses returned to the client by error type.

AAA Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Metric Description
AAA Client Server Processing Time When the device is acting as an AAA client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Server Processing Time
This chart shows the median processing time for the client.
Metric Description
AAA Client Server Processing Time When the device is acting as an AAA client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Time Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Metric Description
Round Trip Time The time between when an AAA client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the client.
Metric Description
Round Trip Time The time between when an AAA client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

AAA Metric Totals

Requests and Responses

Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the client might be sending more requests than the servers can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.

Note:It is unlikely that the total number of AAA requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of total requests that the device sent when acting as an AAA client.
Responses The number of responses that the device received when acting as an AAA client.
Errors The number of response errors that the device received when acting as an AAA client.
Diameter Request The number of Diameter requests that the device sent when acting as an AAA client.
RADIUS Request The number of RADIUS requests that the device sent when acting as an AAA client.
Aborts The number of aborted sessions that occurred when the device is acting as an AAA client.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AAA server page

AAA Summary

Total Transactions
This chart shows you when AAA errors occurred and how many AAA responses the server sent. This information can help you see how active the server was at the time it returned the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Metric Description
Responses The number of responses that the device sent when acting as an AAA server.
Errors The number of response errors that the device sent when acting as an AAA server.
Transactions

This chart displays the total number of AAA responses the server sent and how many of those responses contained errors.

Metric Description
Responses The number of responses that the device sent when acting as an AAA server.
Errors The number of response errors that the device sent when acting as an AAA server.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The server processing time shows how long servers took to process requests from clients. Processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at the processing time, because this metric alone provides an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT and processing times are both high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

RTT only measures how long an immediate acknowledgement takes to be sent; it does not wait until all packets are delivered. Therefore, RTT is a good indicator of how your network is performing. If you see high processing times, but the TCP RTT is low, the issue is probably at the device-level. Check the network for latency issues if the TCP RTT and processing times are all both.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.



The processing time might be high because the server took a long time to transmit the response (possibly because the response was very large); however, the processing time could also be high because the response took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Metric Description
AAA Server Server Processing Time When the device is acting as an AAA server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Round Trip Time The time between when an AAA server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a server is acting slow, performance summary metrics can help you figure out whether the network or the server is causing the issue. The performance summary metrics show the median amount of time the server took to process requests from clients versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the server is slow. High RTTs indicate that the server is communicating over slow networks.
Metric Description
AAA Client Server Processing Time When the device is acting as an AAA server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Round Trip Time The time between when an AAA server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

AAA Details

Top Methods
This chart shows which AAA methods were called on the server the most by breaking out the total number of requests the server received by method.
Top Error Types
This chart shows which AAA error types the server returned the most by breaking out the total number of responses the server sent by error type.

AAA Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Metric Description
AAA Server Server Processing Time When the device is acting as an AAA server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Server Processing Time
This chart shows the median processing time for the server.
Metric Description
AAA Server Server Processing Time When the device is acting as an AAA server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Round Trip Time Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Metric Description
Round Trip Time The time between when an AAA server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the server.
Metric Description
Round Trip Time The time between when an AAA server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

AAA Metric Totals

Requests and Responses

Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the server can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.

Note:It is unlikely that the total number of AAA requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of total requests that the device received when acting as an AAA server.
Responses The number of responses that the device sent when acting as an AAA server.
Errors The number of response errors that the device sent when acting as an AAA server.
Diameter Request The number of Diameter requests that the device received when acting as an AAA server.
RADIUS Request The number of RADIUS requests that the device received when acting as an AAA server.
Aborts The number of aborted sessions that occurred when the device is acting as an AAA server.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AAA client group page

Learn about charts on this page

AAA Summary for Group

Total Transactions
This chart shows you when AAA errors occurred and how many responses the AAA clients received. This information can help you see how active the clients were at the time they received the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses that the device received when acting as an AAA client.
Errors The number of response errors that the device received when acting as an AAA client.
Server Processing Time
If a client group is acting slow, the server processing time can help you figure out whether the issue is with the servers. The Server Processing Time chart shows the median amount of time servers took to process requests from the clients. High server processing times indicate that the clients are contacting slow servers.
Metric Description
Server Processing Time When the device is acting as an AAA client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
AAA Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the clients might be sending more requests than servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Requests The number of total requests that the device sent when acting as an AAA client.
Responses The number of responses that the device received when acting as an AAA client.
Errors The number of response errors that the device received when acting as an AAA client.
Diameter Request The number of Diameter requests that the device sent when acting as an AAA client.
RADIUS Request The number of RADIUS requests that the device sent when acting as an AAA client.
Aborts The number of aborted sessions that occurred when the device is acting as an AAA client.

AAA Details for Group

Top AAA Clients in Group
This chart shows which AAA clients in the group were most active by breaking out the total number of AAA requests the group sent by client.
Top Methods
This chart shows which AAA methods the group called the most by breaking out the total number of requests the group sent by method.
Top Error Types
This chart shows which AAA error types the group received the most by breaking out the number of responses returned to the group by error type.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AAA server group page

Learn about charts on this page

AAA Summary for Group

Total Transactions
This chart shows you when AAA errors occurred and how many AAA responses the servers sent. This information can help you see how active the servers were at the time they returned the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses that the device sent when acting as an AAA server.
Errors The number of response errors that the device sent when acting as an AAA server.
Server Processing Time
The Server Processing Time chart shows the median amount of time the servers took to process requests from clients. High server processing times indicate that the servers in a group are slow.
Metric Description
AAA Client Server Processing Time When the device is acting as an AAA server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
AAA Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of total requests that the device received when acting as an AAA server.
Responses The number of responses that the device sent when acting as an AAA server.
Errors The number of response errors that the device sent when acting as an AAA server.
Diameter Request The number of Diameter requests that the device received when acting as an AAA server.
RADIUS Request The number of RADIUS requests that the device received when acting as an AAA server.
Aborts The number of aborted sessions that occurred when the device is acting as an AAA server.

AAA Details for Group

Top AAA Servers in Group
This chart shows which AAA servers in the group were most active by breaking out the total number of AAA responses the group sent by server.
Top Methods
This chart shows which AAA methods were called on servers in the group the most by breaking out the total number of requests the group received by method.
Top Error Types
This chart shows which AAA error types the groups returned the most by breaking out the total number of responses the group sent by error type.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AAA devices page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
AAA Device Toolbar
The AAA device toolbar includes the following controls:
AAA Metric Type
Display metrics for devices acting as an AAA client or AAA server.
Errors
Click the Errors button to display the list of error messages sent to or received by the current device over the time interval. Errors are formatted as follows: Results-Code-Description:Session-Id:Error-Reporting-Host:Subscription-ID-Data.
  • Session-Id frequently contains multiple semicolon-separated records.
  • Error-Reporting-Host is not always present.
Records
Displays results for records that match the selected metric source and protocol.
AAA Client
If you select Client for the AAA Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of total requests that the device sent when acting as an AAA client.
Responses
Number of responses that the device received when acting as an AAA client.
Errors
Number of AAA errors for the selected time interval.
Aborts
Number of aborted sessions that occurred when the device is acting as an AAA client.
AAA Server
If you select Server for the AAA Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of total requests that the device received when acting as an AAA server.
Responses
Number of responses that the device sent when acting as an AAA server.
Errors
Number of AAA errors for the selected time interval.
Aborts
Number of aborted sessions that occurred when the device is acting as an AAA server.
Messages
Selected message types for the AAA server.
Status Codes
The AAA status codes for the selected time interval.
Processing Time Distribution
Displays a boxplot of times it took the server to process requests. Move the mouse pointer over each bar to display the time range it represents and the number of requests in this bin.
Transactions Per Second
Displays the number of protocol transactions per second as a function of time over the selected time interval. The chart is annotated with red data points to indicate errors. The volume of errors is denoted by the height of red bars under the chart. Click the red dot to see the number of errors that occurred at that time. Click and drag across the chart to select a particular region.
Response Time Breakdown
Displays the area chart containing median request transfer time, server processing time, and response transfer time over time in milliseconds. Click and drag across the chart to select a particular region.

AAA groups page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
AAA Groups Toolbar
The AAA groups toolbar includes the following controls:
Metric Type
Click the Metric Type drop-down list and select either Client or Server to display metrics for devices in the current group acting as an AAA client or AAA server, respectively.
Errors
Click the Errors button to display the list of error messages sent to or received by the current member over the time interval. Errors are formatted as follows: Results-Code-Description:Session-Id:Error-Reporting-Host:Subscription-ID-Data.

Session-Id frequently contains multiple semicolon-separated records. Error-Reporting-Host is not always present.

Records
Displays results for records that match the selected metric source and protocol.
AAA Client
If you select Client for the AAA Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of AAA requests for the selected time interval.
Responses
Number of AAA responses for the selected time interval.
Errors
Number of AAA errors for the selected time interval.
Aborts
Number of AAA aborted requests for the selected time interval.
Diameter Requests
Number of Diameter requests for the selected time interval.
Radius Requests
Number of RADIUS requests for the selected time interval.
AAA Server
If you select Server for the AAA Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of AAA requests for the selected time interval.
Responses
Number of AAA responses for the selected time interval.
Errors
Number of AAA errors for the selected time interval.
Aborts
Number of AAA aborted requests for the selected time interval.
Diameter Requests
Number of Diameter requests for the selected time interval.
Radius Requests
Number of RADIUS requests for the selected time interval.
Messages
Selected message types for the AAA server.

AMF

ExtraHop appliances collect metrics about Hypertext Transfer Protocol (HTTP) Action Message Format (AMF) activity.

AMF client page

AMF Summary

Transactions
This chart shows you when AMF errors occurred and how many responses the AMF client received. This information can help you see how active the client was at the time it received the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Metric Description
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Transaction Summary
This chart displays the total number of AMF responses the client received and how many of those responses contained errors.
Metric Description
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The transfer and processing time metrics show parts of a complete transaction. The request transfer time shows how long the client took to transmit requests onto the network; the server processing time shows how long servers took to process the requests; and the response transfer time shows how long servers took to transmit responses onto the network.

Transfer and processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at transfer and processing times, because these metrics alone provide an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high transfer or processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT, processing, and transfer times are all high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The ExtraHop system calculates the RTT value by measuring the time between the first packet of a request and the acknowledgement from the server, as shown in the following figure:

The request transfer time might be high because the client took a long time to transmit the request (possibly because the request was very large); however, the transfer time could also be high because the request took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

AMF Client Request Transfer Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the first packet and last packet of sent requests. A high value may indicate a large request or network delay.
AMF Client Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
AMF Client Response Transfer Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the first packet and last packet of received responses. High values may indicate a large response or network delay.
Round Trip Time The time between when a AMF client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a client is acting slow, performance summary metrics can help you figure out whether the network or servers are causing the issue. These metrics show the median amount of time that servers took to process requests from the client versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the client is contacting slow servers. High TCP round trip times indicate that the client is communicating over slow networks.
AMF Client Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Time The time between when a AMF client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

AMF Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Metric Description
AMF Client Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Server Processing Time
This chart shows the median processing time for the client.
Metric Description
AMF Client Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Metric Description
Round Trip Time The time between when a AMF client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the client.
Metric Description
Round Trip Time The time between when a AMF client sent a packet that required an immediate acknowledgment and when the client received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

AMF Metric Totals

Requests and Responses
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the client might be sending more requests than the servers can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.
Note:It is unlikely that the total number of AMF requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests that the device sent when acting as an HTTP-AMF client.
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Responses Without Length The number of responses that had no length, that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Requests Without Length The number of requests that had no length, that the device sent when acting as an HTTP-AMF client.
Request and Response Size
This chart shows the average size of requests and responses.
Metric Description
Request Size The distribution of sizes (in bytes) of requests that the device sent when acting as an HTTP-AMF client.
Response Size The distribution of sizes (in bytes) of responses that the device received when acting as an HTTP-AMF client.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AMF server page

AMF Summary

Transactions
This chart shows you when AMF errors occurred and how many AMF responses the server sent. This information can help you see how active the server was at the time it returned the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Metric Description
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Transaction Summary

This chart displays the total number of AMF responses the server sent and how many of those responses contained errors.

Metric Description
Responses The number of responses that the device sent when acting as an HTTP-AMF server.
Errors The number of response errors that the device sent when acting as an HTTP-AMF server.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The transfer and processing time metrics show parts of a complete transaction. The request transfer time shows how long clients took to transmit requests onto the network; the server processing time shows how long the server took to process requests; and the response transfer time shows how long the server took to transmit responses onto the network.

Transfer and processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at transfer and processing times, because these metrics alone provide an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high transfer or processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT, processing, and transfer times are all high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The ExtraHop system calculates the RTT value by measuring the time between the first packet of a request and the acknowledgement from the server, as shown in the following figure:

The request transfer time might be high because the client took a long time to transmit the request (possibly because the request was very large); however, the transfer time could also be high because the request took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

AMF Server Request Transfer Time When the device is acting as an HTTP-AMF server, the time between the ExtraHop system detecting the first packet and last packet of received requests. A high value may indicate a large request or network delay.
AMF Server Server Processing Time When the device is acting as an HTTP-AMF server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
AMF Server Response Transfer Time When the device is acting as an HTTP-AMF server, the time between the ExtraHop system detecting the first packet and last packet of sent responses. High values may indicate a large response or network delay.
Round Trip Time The time between when an AMF server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a server is acting slow, performance summary metrics can help you figure out whether the network or the server is causing the issue. The performance summary metrics show the median amount of time the server took to process requests from clients versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the server is slow. High RTTs indicate that the server is communicating over slow networks.
Metric Description
AMF Server Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
Round Trip Time The time between when an AMF server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

AMF Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Metric Description
AMF Server Server Processing Time When the device is acting as an HTTP-AMF server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Server Processing Time
This chart shows the median processing time for the server.
Metric Description
Server Processing Time When the device is acting as an HTTP-AMF server, the time between the ExtraHop system detecting the last packet of the received request and first packet of the sent response.
Round Trip Time Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Metric Description
Round Trip Time The time between when an AMF server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the server.
Metric Description
Round Trip Time The time between when an AMF server sent a packet that required an immediate acknowledgment and when the server received the acknowledgment. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

AMF Metric Totals

Requests and Responses

Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the server can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.

Note:It is unlikely that the total number of AMF requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests that the device received when acting as an HTTP-AMF server.
Responses The number of responses that the device sent when acting as an HTTP-AMF server.
Responses Without Length The number of responses that had no length, that the device sent when acting as an HTTP-AMF server.
Errors The number of response errors that the device sent when acting as an HTTP-AMF server.
Requests Without Length The number of requests that had no length, that the device received when acting as an HTTP-AMF server.
Request and Response Size
This chart shows the average size of requests and responses.
Metric Description
Request Size The distribution of sizes (in bytes) of requests that the device received when acting as an HTTP-AMF server.
Response Size The distribution of sizes (in bytes) of responses that the device sent when acting as an HTTP-AMF server.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AMF client group page

Learn about charts on this page

AMF Summary for Group

Total Transactions
This chart shows you when AMF errors occurred and how many responses the AMF clients received. This information can help you see how active the clients were at the time they received the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Server Processing Time
If a client group is acting slow, the server processing time can help you figure out whether the issue is with the servers. The Server Processing Time chart shows the median amount of time servers took to process requests from the clients. High server processing times indicate that the clients are contacting slow servers.
AMF Client Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
AMF Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the clients might be sending more requests than servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests that the device sent when acting as an HTTP-AMF client.
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Responses Without Length The number of responses that had no length, that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Requests Without Length The number of requests that had no length, that the device sent when acting as an HTTP-AMF client.

AMF Details for Group

Top AMF Clients in Group
This chart shows which AMF clients in the group were most active by breaking out the total number of AMF requests the group sent by client.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

AMF server group page

Learn about charts on this page

AMF Summary for Group

Total Transactions
This chart shows you when AMF errors occurred and how many AMF responses the servers sent. This information can help you see how active the servers were at the time they returned the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses that the device received when acting as an HTTP-AMF client.
Errors The number of response errors that the device received when acting as an HTTP-AMF client.
Server Processing Time
The Server Processing Time chart shows the median amount of time the servers took to process requests from clients. High server processing times indicate that the servers in a group are slow.
Metric Description
AMF Server Server Processing Time When the device is acting as an HTTP-AMF client, the time between the ExtraHop system detecting the last packet of the sent request and the first packet of the received response.
AMF Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests that the device received when acting as an HTTP-AMF server.
Responses The number of responses that the device sent when acting as an HTTP-AMF server.
Responses Without Length The number of responses that had no length, that the device sent when acting as an HTTP-AMF server.
Errors The number of response errors that the device sent when acting as an HTTP-AMF server.
Requests Without Length The number of requests that had no length, that the device received when acting as an HTTP-AMF server.

AMF Details for Group

Top AMF Servers in Group
This chart shows which AMF servers in the group were most active by breaking out the total number of AMF responses the group sent by server.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

HTTP-AMF devices page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
HTTP-AMF Devices Page
The HTTP-AMF device toolbar includes the following controls:
HTTP-AMF Metric Type
Displays metrics for the current device acting as an HTTP-AMF client or HTTP-AMF server.
Clients or Servers
Displays the associated client IP addresses when the device is acting as a server, and the associated server IP addresses when acting as a client.

HTTP-AMF Details specifies the type of additional HTTP-AMF information displayed. Moving the mouse pointer over the counter next to each top-level metric opens a context menu that includes the following options in the drop-down list:

By IP
Displays HTTP-AMF metrics by IP addresses.
By Target URI
Displays HTTP-AMF metrics by Target URI.

For example, HTTP-AMF Requests is a top-level metric showing how many requests were received by the HTTP server during the selected time frame. Selecting By IP in the drop-down list while moving the mouse pointer over the Requests counter displays which IP addresses originated these requests. Selecting By Target URI from the drop-down list while moving the mouse pointer over the HTTP-AMF Requests counter displays which URIs were accessed by the requesters.

IP Address HTTP-AMF Metrics
Click By IP in the drop-down list to display the following information in the details table.
IP Address
Represents the HTTP-AMF server's IP address.
Host
Represents the DNS host name of the HTTP-AMF server determined by passive analysis of the DNS traffic.
Device
Provides a link to the corresponding HTTP-AMF server device. For local HTTP-AMF servers, the link leads to the HTTP server device. For remote HTTP-AMF servers, the link leads to the gateway device through which the requests were routed.
<Metric value>
Displays the value for the selected metric.
Processing Time
Represents the time in milliseconds it took for HTTP servers to process requests for the currently selected HTTP client. Timing information is expressed as a confidence interval around the mean value bounded by one standard deviation. This metric is available for successful HTTP Responses only.
HTTP-AMF Metrics by Target URI
Click By Target URI in the drop-down list to display the following information in the details table.
Target URI
Represents the full HTTP target URI.
<Metric value>
Displays the value for the selected metric.
Processing Time
Represents the time in milliseconds it took to process URIs requested by the currently selected HTTP client. Timing information is expressed as a confidence interval around the mean value bounded by one standard deviation. This metric is available for successful HTTP Responses only.
HTTP-AMF Client
If you select Client for the HTTP-AMF Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of requests that the device sent when acting as an HTTP-AMF client.
Responses
Number of responses that the device received when acting as an HTTP-AMF client.
Errors
Number of HTTP-AMF errors for the selected time interval.
Requests w/o Length
Number of requests that had no length, that the device received when acting as an HTTP-AMF client.
Responses w/o Length
Number of responses that had no length, that the device sent when acting as an HTTP-AMF client.
HTTP-AMF Server
If you select Server for the HTTP-AMF Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of requests that the device received when acting as an HTTP-AMF server.
Responses
Number of responses that the device sent when acting as an HTTP-AMF server.
Errors
Number of HTTP-AMF errors for the selected time interval.
Requests w/o Length
Number of requests that had no length, that the device received when acting as an HTTP-AMFs server.
Responses w/o Length
Number of responses that had no length, that the device sent when acting as an HTTP-AMF server.

HTTP-AMF groups page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
HTTP-AMF Client
If you select Client for the HTTP-AMF Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of HTTP-AMF requests for the selected time interval.
Responses
Number of HTTP-AMF responses for the selected time interval.
Errors
Number of HTTP-AMF errors for the selected time interval.
Requests w/o Length
Number of HTTP-AMF requests without length.
Responses w/o Length
Number of HTTP-AMF responses without length.
HTTP-AMF Server
If you select Server for the HTTP-AMF Metric Type, the Discover appliance displays the following metrics. Click the counter next to each metric to break it down by group members in the table at the bottom of the page.
Requests
Number of HTTP-AMF requests for the selected time interval.
Responses
Number of HTTP-AMF responses for the selected time interval.
Errors
Number of HTTP-AMF errors for the selected time interval.
Requests w/o Length
Number of HTTP-AMF requests without length.
Responses w/o Length
Number of HTTP-AMF responses without length.

CIFS

ExtraHop appliances collect metrics about Common Internet File System (CIFS)/Server Message Block (SMB) activity. ExtraHop appliances support SMB, SMB2 and SMB3.

Important:Access time is the time it takes for a CIFS server to receive a requested block. There is no access time for operations that do not access actual block data within a file. Processing time is the time it takes for a CIFS server to respond to the operation requested by the client, such as a metadata retrieval request.

There are no access times for SMB2_CREATE. SMB2_CREATE creates a file that is referenced in the response by an SMB2_FILEID. The referenced file blocks are then read from or written to the NAS-storage device. These file read and write operations are calculated as access times.

CIFS client page

CIFS Summary

Total Transactions
This chart shows you when CIFS errors occurred and how many responses the CIFS client received. This information can help you see how active the client was at the time it received the errors.

If you see a large number of errors, you can view details about each error, including the error code. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of requests to responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Tip:To drill down by error code, click Errors and select Error from the menu.
Metric Description
Responses The number of responses that the device received when acting as a CIFS client.
Errors The number of responses received by this CIFS client with an SMB code other than SUCCESS or a warning. A high number of CIFS errors indicate corrupt profiles.
Transactions
This chart displays the total number of CIFS responses the client received and how many of those responses contained errors.
Metric Description
Responses The number of responses that the device received when acting as a CIFS client.
Errors The number of responses received by this CIFS client with an SMB code other than SUCCESS or a warning. A high number of CIFS errors indicate corrupt profiles.
Operations
This chart shows you when the CIFS client performed read, write, and file system information request operations.
Metric Description
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
File System Information Requests The number of file system metadata queries sent by this CIFS client.
Operation Summary
This chart shows you how many read and write operations the CIFS client performed.
Metric Description
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The access time shows how long servers took to process read or write operations that accessed block data within a file. Access times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at the access time, because this metric alone provides an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high access times, but the RTT is low, the issue is probably at the device-level. However, if the RTT and access times are both high, network latency might be affecting the transfer and access times, and the issue might be with the network.

RTT only measures how long an immediate acknowledgement takes to be sent; it does not wait until all packets are delivered. Therefore, RTT is a good indicator of how your network is performing. If you see high access times, but the TCP RTT is low, the issue is probably at the device-level. Check the network for latency issues if the TCP RTT and access times are all both.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The access time might be high because the server took a long time to transmit the response (possibly because the response was very large); however, the access time could also be high because the response took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Metric Description
CIFS Client Request Transfer Time When the device is acting as a CIFS client, the time between the ExtraHop system detecting the first packet and last packet of sent requests. High values may indicate a large request or network delay.
CIFS Client Access Time The time between the ExtraHop system detecting the last packet of the request sent by this CIFS client and first packet of the received response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
CIFS Client Response Transfer Time When the device is acting as a CIFS client, the time between the ExtraHop system detecting the first packet and last packet of received responses. High values may indicate a large response or network delay.
Round Trip Time Round trip time (RTT) is a measurement of total network latency. The ExtraHop system calculates RTT by measuring the time it took between sending a packet and receiving an immediate acknowledgement (ACK).

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a client is acting slow, performance summary metrics can help you figure out whether the network or servers are causing the issue. The performance summary metrics show the median amount of time servers took to process requests from the client versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server access times indicate that the client is contacting slow servers. High TCP round trip times indicate that the client is communicating over slow networks.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the request sent by this CIFS client and first packet of the received response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
Round Trip Time Round trip time (RTT) is a measurement of total network latency. The ExtraHop system calculates RTT by measuring the time it took between sending a packet and receiving an immediate acknowledgement (ACK).

CIFS Details

Top Methods
This chart shows which CIFS methods the client called the most by breaking out the total number of requests the client sent by method.
Top Users
This chart shows which users were most active on the client by breaking out the total number of CIFS requests sent by the client by user.
Top Files
This chart shows which files the client accessed the most by breaking out the total number of responses the client received by file path.

CIFS Performance

Access Time Distribution
This chart breaks out access times in a histogram to show the most common access times.
Metric Description
CIFS Client Access Time The time between the ExtraHop system detecting the last packet of the request sent by this CIFS client and first packet of the received response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
Access Time
This chart shows the median access time for the client.
Metric Description
CIFS Client Access Time The time between the ExtraHop system detecting the last packet of the request sent by this CIFS client and first packet of the received response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

CIFS Metric Totals

Operations
This chart shows you how many operations the CIFS client performed.
Metric Description
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
File System Info Requests The number of file system metadata queries sent by this CIFS client.
Locks The number of lock operation requests produced by this CIFS client.
Warnings The number of responses received by this CIFS client with an SMB code that indicates a warning.
Request and Response Size
This chart shows the average size of requests and responses.
Metric Description
CIFS Server Request Size The distribution of sizes (in bytes) of requests sent by this CIFS client.
CIFS Server Response Size The distribution of sizes (in bytes) of responses received when the device is acting as a CIFS client.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

CIFS server page

CIFS Summary

Total Transactions
This chart shows you when CIFS errors occurred and how many CIFS responses the server sent. This information can help you see how active the server was at the time it returned the errors.

If you see a large number of errors, you can view details about each error, including the error code. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of requests to responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Tip:To drill down by error code, click Errors and select Error from the menu.
Metric Description
Responses The number of responses sent by this CIFS server.
Errors The number of responses sent by this CIFS server with a CIFS (SMB) code other than SUCCESS or a warning. A high volume of errors should be investigated.
Transactions

This chart displays the total number of CIFS responses the server sent and how many of those responses contained errors.

Metric Description
Responses The number of responses sent by this CIFS server.
Errors The number of responses sent by this CIFS server with a CIFS (SMB) code other than SUCCESS or a warning. A high volume of errors should be investigated.
Operations
This chart shows you when the read, write, and file system information request operations were performed on the server.
Metric Description
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
File System Information Requests The number of file system metadata queries sent by this CIFS client.
Operation Summary
This chart shows you how many read and write operations were performed on the server.
Metric Description
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The access time shows how long servers took to process read or write operations that accessed block data within a file. Access times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at the access time, because this metric alone provides an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high access times, but the RTT is low, the issue is probably at the device-level. However, if the RTT and access times are both high, network latency might be affecting the transfer and access times, and the issue might be with the network.

RTT only measures how long an immediate acknowledgement takes to be sent; it does not wait until all packets are delivered. Therefore, RTT is a good indicator of how your network is performing. If you see high access times, but the TCP RTT is low, the issue is probably at the device-level. Check the network for latency issues if the TCP RTT and access times are all both.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The access time might be high because the server took a long time to transmit the response (possibly because the response was very large); however, the access time could also be high because the response took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Metric Description
CIFS Server Request Transfer Time When the device is acting as a CIFS server, the time between the ExtraHop system detecting the first packet and last packet of received requests. High values may indicate a large request or network delay.
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the received request by this CIFS server and first packet of the response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
CIFS Server Response Transfer Time When the device is acting as a CIFS server, the time between the ExtraHop system detecting the first packet and last packet of sent responses. High values may indicate a large response or network delay.
Round Trip Time Round trip time (RTT) is a measurement of total network latency. The ExtraHop system calculates RTT by measuring the time it took between sending a packet and receiving an immediate acknowledgement (ACK).

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a server is acting slow, performance summary metrics can help you figure out whether the network or the server is causing the issue. The performance summary metrics show the median amount of time the server took to process requests from clients versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server access times indicate that the server is slow. High RTTs indicate that the server is communicating over slow networks.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the received request by this CIFS server and first packet of the response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
Round Trip Time Round trip time (RTT) is a measurement of total network latency. The ExtraHop system calculates RTT by measuring the time it took between sending a packet and receiving an immediate acknowledgement (ACK).

CIFS Details

Top Methods
This chart shows which CIFS methods were called on the server the most by breaking out the total number of requests the server received by method.
Top Users
This chart shows which users were most active on the server by breaking out the total number of CIFS requests sent to the server by user.
Top Files
This chart shows which files on the server were accessed the most by breaking out the total number of responses the server sent by file path.

CIFS Performance

Access Time Distribution
This chart breaks out access times in a histogram to show the most common access times.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the received request by this CIFS server and first packet of the response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
Access Time
This chart shows the median access time for the client.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the received request by this CIFS server and first packet of the response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

CIFS Metric Totals

Operations
This chart shows you how many operations were performed on the CIFS server.
Metric Description
Reads The number of read operation requests received by this CIFS server.
Writes The number of write operation requests received by this CIFS server.
File System Info Requests The number of file system metadata queries device received by this CIFS server.
Locks The number of lock operation requests received by this CIFS server.
Warnings When the device is acting as a CIFS server, the number of responses sent with an SMB code that indicates a warning.
Request and Response Size
This chart shows the average size of requests and responses.
Metric Description
CIFS Server Request Size The distribution of sizes (in bytes) of requests received by this CIFS server.
CIFS Server Response Size The distribution of sizes (in bytes) of responses sent by this CIFS server.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

CIFS client group page

Learn about charts on this page

CIFS Summary for Group

Total Transactions
This chart shows you when CIFS errors occurred and how many responses the CIFS clients received. This information can help you see how active the clients were at the time they received the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses that the device received when acting as a CIFS client.
Errors The number of responses received by this CIFS client with an SMB code other than SUCCESS or a warning. A high number of CIFS errors indicate corrupt profiles.
Access Time
If a client group is acting slow, the access time can help you figure out whether the issue is with the servers. The Server Processing Time chart shows the median amount of time servers took to process requests from the clients. High access times indicate that the clients are contacting slow servers.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the request sent by this CIFS client and first packet of the received response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
CIFS Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the clients might be sending more requests than servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests sent by this CIFS client.
Responses The number of responses that the device received when acting as a CIFS client.
Reads The number of read operation requests sent by this CIFS client.
Writes The number of write operation requests sent by this CIFS client.
File System Info Requests The number of file system metadata queries sent by this CIFS client.
Locks The number of lock operation requests produced by this CIFS client.
Warnings The number of responses received by this CIFS client with an SMB code that indicates a warning.

CIFS Details for Group

Top CIFS Clients in Group
This chart shows which CIFS clients in the group were most active by breaking out the total number of CIFS requests the group sent by client.
Top Methods
This chart shows which CIFS methods the group called the most by breaking out the total number of requests the group sent by method.
Top Users
This chart shows which CIFS users were most active in the group by breaking out the total number of CIFS responses the group received by user.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

CIFS server group page

Learn about charts on this page

CIFS Summary for Group

Total Transactions
This chart shows you when CIFS errors occurred and how many CIFS responses the servers sent. This information can help you see how active the servers were at the time they returned the errors.

In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Metric Description
Responses The number of responses sent by this CIFS server.
Errors The number of responses sent by this CIFS server with a CIFS (SMB) code other than SUCCESS or a warning. A high volume of errors should be investigated.
Access Time
If a server group is acting slow, the Access Time chart can help you figure out whether the issue is with the servers. The Access Time chart shows the median amount of time the servers took to process requests from clients. High server access times indicate that the servers are slow.
Metric Description
CIFS Server Access Time The time between the ExtraHop system detecting the last packet of the received request by this CIFS server and first packet of the response. Access time is measured only for the first READ or WRITE operation on every flow in order to minimize the influence of pre-fetching and caching on timing metrics.
CIFS Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests received by this CIFS server.
Responses The number of responses sent by this CIFS server.
Errors The number of responses sent by this CIFS server with a CIFS (SMB) code other than SUCCESS or a warning. A high volume of errors should be investigated.
Reads The number of read operation requests received by this CIFS server.
Writes The number of write operation requests received by this CIFS server.
File System Info Requests The number of file system metadata queries device received by this CIFS server.
Locks The number of lock operation requests received by this CIFS server.
Warnings When the device is acting as a CIFS server, the number of responses sent with an SMB code that indicates a warning.

CIFS Details for Group

Top CIFS Servers in Group
This chart shows which CIFS servers in the group were most active by breaking out the total number of CIFS responses the group sent by server.
Top Methods
This chart shows which CIFS methods were called on servers in the group the most by breaking out the total number of requests the group received by method.
Top Users
This chart shows which CIFS users were most active in the group by breaking out the total number of CIFS responses the group sent by user.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

CIFS devices page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
CIFS Devices Toolbar
The CIFS device page toolbar includes the following controls:
CIFS Metric Type
Displays metrics for the current device acting as a CIFS client or CIFS server.
Errors
Displays the list of error messages sent to or received by the current device over the selected time interval.
Warnings
Displays the list of warning messages sent to or received by the current device over the selected time interval.
Methods
Displays the list of methods and associated bytes sent and received by the current device for the selected time interval. Methods are broken out by key parameters, such as the accessed file name and file access time.
Users
Displays the list of users accessing the file server and associated bytes sent and received for the selected time interval.
Files
Displays the list of files accessed and associated bytes sent and received for the selected time interval. The access time indicates the time to access a file on a CIFS partition and is measured by timing the first READ or WRITE on every flow.
Records
Displays results for records that match the selected metric source and protocol.

Where file name detail is presented, the Discover appliance displays both the file path and mount point, if available. The prefix '...' indicates that either the mount point or part of the path is not available. This might occur in instances when the capture process was restarted after the mount or a cd command was issued, or when the commands were lost due to desyncs.

Click the counters next to individual CIFS metrics to show the IP Address CIFS Metrics details for CIFS peer devices. For CIFS servers, the peer devices are CIFS clients. For CIFS clients, the peer devices are CIFS servers.

IP Address
Represents the IP address of the peer device.
Host
Represents the DNS host name of the peer device determined by passive analysis of the DNS traffic.
Device
Provides a link to the corresponding peer device. For local peer devices, the link leads to that device. For remote peer devices, the link leads to the gateway device through which the requests were routed.
CIFS Server
Displays additional IP address details.
Responses
Specifies the number of responses that the device sent when acting as a CIFS server.
Errors
Specifies the number of errors sent by the CIFS server.
Warnings
Displays the list of warning messages sent to or received by the CIFS server over the selected time interval.
Reads
Specifies the number of read operation requests that the device received when acting as a CIFS server.
Writes
Specifies the number of write operation requests that the device received when acting as a CIFS server.
Locks
Specifies the number of lock operation requests that the device received when acting as a CIFS server.
FSInfo
Specifies the number of file system metadata queries that the device received when acting as a CIFS server.
CIFS Client
Displays additional IP address details.
Responses
Specifies the number of responses that the device received when acting as a CIFS client.
Errors
Specifies the number of errors sent by the CIFS client.

CIFS groups page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
CIFS Groups Toolbar
The CIFS groups toolbar includes the following controls:
CIFS Metric Type
Displays metrics for devices in the current group acting as a CIFS client or server, respectively.
Errors
Displays the list of error messages sent to or received by devices in the current group over the selected time interval.
Warnings
Displays the list of warning messages sent to or received by devices in the current group over the selected time interval.
Methods
Displays the list of methods and associated bytes sent and received by devices in the current group during the selected time interval. Methods are broken out by key parameters, such as the accessed file name.
Users
Displays the list of users accessing the file server and associated bytes sent and received for the selected time interval.
Files
Displays the list of files accessed and associated bytes sent and received for the selected time interval. Access Time indicates the time it took for the server to access a file on disk.
Records
Displays results for records that match the selected metric source and protocol.

Where file name detail is presented, the Discover appliance displays both the file path and mount point, if available. The prefix '...' indicates that either the mount point or part of the path is not available. This might occur in instances when the capture process was restarted after the "mount" or a "cd" command was issued, or when the commands were lost due to desyncs.

CIFS Server
Click the counter next to the metric to break it down by group members in the table at the bottom of the page.
Responses
Specifies the number of responses sent by the CIFS server.
Errors
Specifies the number of errors sent by the CIFS server.
Warnings
Displays the list of warning messages sent to or received by devices in the CIFS server over the selected time interval.
Reads
Specifies the number of read operations requested from the CIFS server.
Writes
Specifies the number of write operations requested from the CIFS server.
Locks
Specifies the number of lock operations requested from the CIFS server.
CIFS Client
Click the counter next to the metric to break it down by group members in the table at the bottom of the page.
Responses
Specifies the number of responses received by the CIFS client.
Errors
Specifies the number of errors sent by the CIFS client.
Warnings
Displays the list of warning messages sent to or received by the CIFS client over the selected time interval.
Reads
Specifies the number of read operations requested by the CIFS client.
Writes
Specifies the number of write operations requested by the CIFS client.
Locks
Specifies the number of lock operations requested by the CIFS client.
Methods
Displays the CIFS methods for the selected time interval.

Click the counter next to the method to break it down by group members in the table.

Database

ExtraHop appliances collect metrics about database activity. Activity for the following databases is aggregated and displayed under Database metrics in the ExtraHop Web UI:

  • IBM DB2
  • IBM Informix
  • Microsoft SQL Server
  • MySQL
  • Oracle
  • PostgreSQL
  • Sybase ASE
  • Sybase IQ
Note:ExtraHop Discover appliances also monitor MongoDB database activity, which is displayed through a separate set of metrics specific to MongoDB.

Learn more by taking the Database Quick Peek training.

The following sections describe the top metrics that you should investigate for problems related to databases.

Errors

Database errors occur when a database request cannot be completed by the server. Errors can indicate a minor issue, such as a single log-in failure, or a more severe issue, such as an overloaded database server.

When investigating database errors, you can start by reviewing the total number of errors in your environment on the Metrics > Applications > All Activity > Database page. You can view details about each error, including the raw error message reported by the database, by clicking the Errors icon.

On the Applications > All Activity > Database page, you can break out metrics by database server by hovering over the Response Errors value and clicking By Server IP. You can then sort by the number of errors. If one database server is returning a large number of errors, you can click the server name, and then click the Errors icon to view the total number of errors for that server. However, if no server is contributing a large number of errors, the issue might be more complex, and you should investigate which methods were called on each database.

Methods

You can view which methods were called on databases in your environment. Poorly-formed database calls can cause performance issues, even if no errors exist. To see all methods that were called in your environment over a specified time interval, go to the Metrics > Applications > All Activity > Database page and click Methods.

If a method is called on a table, the table name is displayed after an @ symbol. For example, CREATE @ Configuration displays metrics about how many times the CREATE method was called on a table named Configuration. Methods can be sorted by processing time, which is the amount of time between when a server receives a request and when the server sends a response. Long processing times can indicate that the database is poorly optimized or that statements are poorly formatted.

Custom metrics and records

If the processing time for a database method is continuously long, you might want to investigate further by collecting the raw SQL statements that contain the method. You can record and view raw SQL statements by creating a custom metric or by generating records through a trigger. A custom metric enables you to view a graphical representation of the information; for example, you could create a chart of how many slow database requests occurred over time and break out each response by the SQL statement. Records enable you to view individual records of each event; for example, you could view exactly how much time it took the server to respond to each SQL statement.

The following trigger runs when a database response event occurs. If a database server takes more than 100 milliseconds to respond to a SELECT request on the Configuration table, the trigger records the SQL statement of the request in a custom metric. The trigger also records the total number of database requests that took the server more than 100 milliseconds to respond to.

// Event: DB_RESPONSE
if (DB.processingTime > 100 && DB.method == "SELECT" && DB.table == "Configuration") {

  // Record a custom metric.
  Device.metricAddCount('slow_performers', 1);
  Device.metricAddDetailCount('slow_performers_by_statement', DB.statement, 1);
}

The next trigger generates similar information, but in the form of a record for all database responses. The records contain the processing time, method, table name, and SQL statement for each response. After the records are collected, you can view the SQL statements for all SELECT requests on the Configuration table that took the server more than 100 milliseconds to respond to.

// Event: DB_RESPONSE
DB.commitRecord()
Note:Your Discover appliance must be connected to an Explore appliance to generate records.

After you create a trigger, you must assign the trigger to the devices you want to monitor. If you create a custom metric, you must create a dashboard to view the custom metric.

For more information about triggers, see Get started with triggers.

For more information about dashboards, see Get started with dashboards.

For more information about records, see Records.

Database applications page

Database Application Toolbar
The Database application toolbar includes the following controls:
Errors
The chart displays the total count for DB errors. Mouse over the points to view a summary of a specific time or date. The table lists DB error messages and the number of occurrences.
Methods

The chart displays responses compared to processing time. Mouse over the points to view a five-number summary of processing time (minimum, lower quartile, median, upper quartile, and maximum values). The orange bars represent a confidence interval around the median value bounded by the 25th and 75th percentile values.

The table lists methods, number of responses, total time, and processing time (ms) associated with each method. Mouse over the orange bars to view the mean time, standard deviation, and count for each metric.

Users
The chart displays the number of responses and errors for all users. Mouse over the chart to view a summary of a specific time or date. The table displays the list of users, and the number of responses and errors associated with each user.
Clients

The chart displays the total number of responses compared to processing time. Mouse over the points to view a five-number summary of processing time (minimum, lower quartile, median, upper quartile, and maximum values). The orange bars represent a confidence interval around the median value bounded by the 25th and 75th percentile values.

The table lists client IP addresses, the host and device associated with each client, the number of responses from each client, and the total time and processing time for each client. Mouse over the orange bars to view the mean time, standard deviation, and count for each metric.

Servers

The chart displays the total number of responses compared to processing time. Mouse over the points to view a five-number summary of processing time (minimum, lower quartile, median, upper quartile, and maximum values). The orange bars represent a confidence interval around the median value bounded by the 25th and 75th percentile values.

The table lists server IP addresses, the host and device associated with each server, the number of responses from each server, and the total time and processing time for each server. Mouse over the orange bars to view the mean time, standard deviation, and count for each metric.

Application Details
Specifies the type of additional application information displayed. IP detail views display directly monitored IP addresses and IP addresses that appear via routed traffic. IP addresses that appear via routed traffic are preceded by the word via. Mousing over the counter next to each top-level metric opens a context menu that includes the following options in the drop-down list:
By Client IP
Displays application metrics by the client IP addresses.
By Server IP
Displays application metrics by the server IP addresses.

For example, Request Bytes is a top-level metric showing how many request bytes were transmitted in and out of the application within the selected time interval. Select By Client IP in the drop-down list while mousing over the Request Bytes counter to view which client IP addresses originated these requests.

L2-L4 Metrics
Contains the following metrics:
Request L2 Bytes
The number of L2 bytes associated with database requests.
Response L2 Bytes
The number of L2 bytes associated with database responses.
Request Packets
The number of packets associated with database requests.
Response Packets
The number of packets associated with database responses.
Request RTOs
The number of retransmission timeouts caused by congestion when clients were sending database requests. A retransmission timeout is a one-second stall in the TCP connection flow due to excessive retransmissions.
Response RTOs
The number of retransmission timeouts caused by congestion when servers were sending database responses. A retransmission timeout is a one-second stall in the TCP connection flow due to excessive retransmissions.
Request Zero Window
The number of zero window advertisements sent by database clients. A device advertises a zero window when it cannot process incoming data as quickly as it is arriving.
Response Zero Window
The number of zero window advertisements sent by servers while receiving database requests. A device advertises a zero window when it cannot process incoming data as quickly as it is arriving.
DB Metrics
Contains the following metrics:
Requests
The number of database requests.
Responses
The number of database responses.
Response Errors
The number of database response errors.
Transaction Metrics
Transaction metrics display the timing components for all transactions associated with the current device. Timing components are expressed as a confidence interval around the median value bounded by the 25th and 75th percentile values. Mouse over each component to display a five-number statistical summary.
ReqXfer
Request transfer time. The time in milliseconds before the request was received by the server. A large ReqXfer value relative to the total transaction time indicates network delay. If the request size is large, some network delay due to transfer time is expected.
Process
Server processing time. The time in milliseconds between the time the request was received by the server and the time the response was sent. A large server processing time indicates application delay.
RspXfer
Response transfer time. The time in milliseconds before the server finished sending the response. A large RspXfer relative to the total transaction time indicates network delay. If the response size is large, some network delay due to transfer time is expected.
RTT
TCP round-trip time in milliseconds. Large round-trip time indicates that network latency is high.

Click the Transaction Metrics graph to display a chart showing responses compared to mean processing time during the selected time interval. The table below contains the total and mean time for each response.

Transactions Per Second
Displays the number of protocol transactions per second as a function of time over the selected time interval. The chart is annotated with red data points to indicate errors. The volume of errors is denoted by the height of red bars under the chart. Click the red dot to see the number of errors that occurred at that time. Click and drag across the chart to select a particular region.
Response Time Breakdown
Displays the area chart containing median round-trip time, request transfer time, server processing time, and response transfer time over time in milliseconds. Click and drag across the chart to select a particular region.
Round-Trip Time (ms)
Displays the median round-trip time (RTT) in milliseconds (ms) from the current objects to clients as a function of time over the selected time interval. Vertical dotted lines indicate the upper and lower quartiles (75th and 25th percentiles) of the round-trip time metrics. Click and drag across the chart to select a particular region.
Congestion Requests: Goodput (bps) and RTOs
Displays goodput and RTOs into the object as a function of time over the selected time interval.
Congestion Responses: Goodput (bps) and RTOs
Displays goodput and RTOs out of the object as a function of time over the selected time interval.

Goodput is application-level throughput (the number of useful information bits) and RTOs are retransmission timeouts. The Congestion In and Out graphs show the relationship over time between the rate of good application throughput and RTOs. An increase in RTOs theoretically leads to a decrease in goodput due to TCP back-off and packet retransmissions. It is best to view these charts in a smaller window of time so the metrics taken over time are not rolled up or smoothed out. In a small timeframe (30 minutes or less), one could see a decrease in goodput associated with a large number of RTOs, assuming that most flows on the server during this time frame experience this behavior. If only one or two flows are affected by RTOs, then the decreased goodput correlation might be masked by superficially healthy flows.

Database client page

Database Summary

Total Transactions
This chart shows you when database errors occurred and how many responses the database client received. This information can help you see how active the client was at the time it received the errors.

If you see a large number of errors, you can view details about each error, including the raw error message reported by the database. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of requests to responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Tip:To see more information about errors, click the Errors link at the top of the page.
Metric Description
Responses The number of responses received by this database client. Responses vary by the requested operation.
Errors The number of database request operations sent from this database client that failed. All database errors should be investigated.
Transactions
This chart displays the total number of database responses the client received and how many of those responses contained errors.
Metric Description
Responses The number of responses received by this database client. Responses vary by the requested operation.
Errors The number of database request operations sent from this database client that failed. All database errors should be investigated.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The transfer and processing time metrics show parts of a complete transaction. The request transfer time shows how long the client took to transmit requests onto the network; the server processing time shows how long servers took to process the requests; and the response transfer time shows how long servers took to transmit responses onto the network.

Transfer and processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at transfer and processing times, because these metrics alone provide an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high transfer or processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT, processing, and transfer times are all high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The ExtraHop system calculates the RTT value by measuring the time between the first packet of a request and the acknowledgement from the server, as shown in the following figure:

The request transfer time might be high because the client took a long time to transmit the request (possibly because the request was very large); however, the transfer time could also be high because the request took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Metric Description
Database Client Request Transfer Time When the device is acting as a database client, the time between the ExtraHop system detecting the first packet and last packet of sent requests. High values may indicate a large request or network delay.
Database Client Server Processing Time The time it took for this database client to receive the first packet of a response after sending the last packet of the query.
Database Client Response Transfer Time When the device is acting as a database client, the time between the ExtraHop system detecting the first packet and last packet of received responses. High values may indicate a large response or network delay.
Round Trip Time The time between when a database client sent a packet that required an immediate acknowledgment and when the client received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a client is acting slow, performance summary metrics can help you figure out whether the network or servers are causing the issue. These metrics show the median amount of time that servers took to process requests from the client versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the client is contacting slow servers. High TCP round trip times indicate that the client is communicating over slow networks.
Metric Description
Database Client Server Processing Time The time it took for this database client to receive the first packet of a response after sending the last packet of the query.
Round Trip Time The time between when a database client sent a packet that required an immediate acknowledgment and when the client received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

Database Details

Top Methods
This chart shows which methods the client called the most by breaking out the total number of database requests the client sent by method.
Top Status Codes
This chart shows which status codes the client received the most by breaking out the number of responses returned to the client by status code.
Top Users
This chart shows which users were most active on the client by breaking out the total number of database requests sent by the client by user.

Database Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Metric Description
Database Client Server Processing Time The time it took for this database client to receive the first packet of a response after sending the last packet of the query.
Server Processing Time
This chart shows the median processing time for the client.
Metric Description
Database Client Server Processing Time The time it took for this database client to receive the first packet of a response after sending the last packet of the query.
Round Trip Time Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Metric Description
Round Trip Time The time between when a database client sent a packet that required an immediate acknowledgment and when the client received the acknowledgement. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the client.
Metric Description
Round Trip Time The time between when a database client sent a packet that required an immediate acknowledgment and when the client received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

Database Metric Totals

Requests and Responses
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the client might be sending more requests than the servers can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.
Note:It is unlikely that the total number of database requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests sent by this database client. Requests cover a range of operations: connection negotiations, session configuration, data definition language (DDL), data modification language (DML), or data reads (select).
Responses The number of responses received by this database client. Responses vary by the requested operation.
Aborted Requests The number of requests that this database client began to send, but did not send completely.
Aborted Responses The number of responses that this database client began to receive, but did not receive completely when acting as a database client.
Request Size Mean The distribution of sizes (in bytes) of requests that the device sent when acting as a database client.
Response Size Mean The distribution of sizes (in bytes) of responses that the device received when acting as a database client.
Request and Response Size
This chart shows the average size of requests and responses.
Metric Description
Request Size The distribution of sizes (in bytes) of requests that the device sent when acting as a database client.
Response Size The distribution of sizes (in bytes) of responses that the device received when acting as a database client.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

Database server page

Database Summary

Total Transactions
This chart shows you when database errors occurred and how many database responses the server sent. This information can help you see how active the server was at the time it returned the errors.

If you see a large number of errors, you can view details about each error, including the raw error message reported by the database. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of requests to responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see Requests and Responses.

Tip:To see more information about errors, click the Errors link at the top of the page.
Responses The number of responses by all database instances on this server. Responses vary by the requested operation. For example, a response can include connection and session configurations, success or failure notifications, or a tabular data set.
Errors The number of database request operations that failed on all database instances running on this server. All database errors should be investigated.
Transactions
This chart displays the total number of database responses the server sent and how many of those responses contained errors.
Responses The number of responses by all database instances on this server. Responses vary by the requested operation. For example, a response can include connection and session configurations, success or failure notifications, or a tabular data set.
Errors The number of database request operations that failed on all database instances running on this server. All database errors should be investigated.
Performance (95th Percentile)
This chart shows the 95th percentile of timing metrics. The transfer and processing time metrics show parts of a complete transaction. The request transfer time shows how long clients took to transmit requests onto the network; the server processing time shows how long the server took to process requests; and the response transfer time shows how long the server took to transmit responses onto the network.

Transfer and processing times are calculated by measuring the time between when the first and last packets of requests and responses are seen by the ExtraHop system, as shown in the following figure:

It can be difficult to tell whether an issue is caused by a network or a device from looking only at transfer and processing times, because these metrics alone provide an incomplete picture. Therefore the round trip time (RTT) metric is also included in this chart. RTT metrics are a good indicator of how your network is performing. If you see high transfer or processing times, but the RTT is low, the issue is probably at the device-level. However, if the RTT, processing, and transfer times are all high, network latency might be affecting the transfer and processing times, and the issue might be with the network.

The RTT metric can help identify the source of the problem because it only measures how long an immediate acknowledgement takes to be sent from the client or server; it does not wait until all packets are delivered.

The ExtraHop system calculates the RTT value by measuring the time between the first packet of a request and the acknowledgement from the server, as shown in the following figure:

The request transfer time might be high because the client took a long time to transmit the request (possibly because the request was very large); however, the transfer time could also be high because the request took a long time to travel on the network (possibly because of network congestion).

Learn more about how the ExtraHop system calculates round trip time on the ExtraHop forum.

Request Transfer Time When the device is acting as a database server, the time between the ExtraHop system detecting the first packet and last packet of received requests. High values may indicate a large request or network delay.
Server Processing Time The time it took for this database server to send the first packet of a response after receiving the last packet of the query.
Response Transfer Time When the device is acting as a database server, the time between the ExtraHop system detecting the first packet and last packet of sent responses. High values may indicate a large response or network delay.
Round Trip Time The time between when a database server sent a packet that required an immediate acknowledgment and when the server received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

The Performance (95th percentile) chart shows the highest value for a time period while filtering outliers; the 95th percentile is the highest value that falls below 95% of the values for a sample period. By displaying the 95th value, rather than the true maximum, the chart gives you a more accurate view of the data:

Performance Summary (95th Percentile)
If a server is acting slow, performance summary metrics can help you figure out whether the network or the server is causing the issue. The performance summary metrics show the median amount of time the server took to process requests from clients versus the median time that packets from those requests (and their respective responses) took to be transmitted across the network. High server processing times indicate that the server is slow. High RTTs indicate that the server is communicating over slow networks.
Server Processing Time The time it took for this database server to send the first packet of a response after receiving the last packet of the query.
Round Trip Time The time between when a database server sent a packet that required an immediate acknowledgment and when the server received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

Database Details

Top Methods
This chart shows which database methods were called on the server the most by breaking out the total number of requests the server received by method.
Top Status Codes
This chart shows which database status codes the server returned the most by breaking out the total number of responses the server sent by status code.
Top Users
This chart shows which users were most active on the server by breaking out the total number of database requests sent to the server by user.

Database Performance

Server Processing Time Distribution
This chart breaks out server processing times in a histogram to show the most common processing times.
Server Processing Time The time it took for this database server to send the first packet of a response after receiving the last packet of the query.
Server Processing Time
This chart shows the median processing time for the server.
Server Processing Time The time it took for this database server to send the first packet of a response after receiving the last packet of the query.
Round Trip Time Distribution
This chart breaks out round trip times in a histogram to show the most common round trip times.
Round Trip Time The time between when a database server sent a packet that required an immediate acknowledgment and when the server received the acknowledgement. Round trip time (RTT) is a measurement of network latency.
Round Trip Time
This chart shows the median round trip time for the server.
Round Trip Time The time between when a database server sent a packet that required an immediate acknowledgment and when the server received the acknowledgement. Round trip time (RTT) is a measurement of network latency.

Network Data

This section shows you TCP information that is related to the current protocol. In general, host stalls indicate that there is an issue with either the server or the client, and network stalls indicate that there is an issue with the network.

Host Stalls
This chart shows the number of zero windows that were advertised or received by the device. Devices control the amount of data they receive by specifying the number of packets that can be sent to them over a given time period. When a device is sent more data than it can process, the device advertises a zero window to ask its peer device to stop sending packets completely until the device catches up. If you see a large number of zero windows, a server or client might not be not fast enough to support the amount of data being received.
Metric Definition
Zero Windows In The number of Zero Windows that were sent to the device to stop the flow of data over the connection. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows in indicates that a peer device was too slow to process the amount of data received.

Zero Windows Out The number of Zero Windows that were sent from the device to stop the flow of data. A device advertises a Zero Window when it cannot process incoming data as quickly as it is arriving.

A large number of zero windows out indicates that the client was too slow to process the amount of data received.

Network Stalls

This chart shows the number of retransmission timeouts that occurred. Retransmission timeouts (RTOs) occur when a network drops too many packets, usually due to packet collisions or buffer exhaustion. If a device sends a request or response and does not receive confirmation within a specified amount of time, the device retransmits the request. If too many retransmissions are unacknowledged, an RTO occurs. If you see a large number of RTOs, the network might be too slow to support the current level of activity.

Metric Definition
RTOs In The number of retransmission timeouts (RTOs) caused by network congestion as peers were sending data to the current device. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs in, the device did not send an acknowledgement to the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

RTOs Out The number of retransmission timeouts (RTOs) caused by network congestion as the device was sending data to its peers. An RTO is a one-second stall in the TCP connection flow due to excessive retransmissions.

If you see a large number of RTOs out, the device did not receive an acknowledgement from the server quickly enough, or the network might be too slow to support the current level of activity. Depending on the timeout value configured in the operating system, this delay can be anywhere from 1 to 8 seconds.

Database Metric Totals

Requests and Responses
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the server can handle or the network might be too slow. To identify whether the issue is with the network or the server, check RTOs and zero windows in the Network Data section.
Note:It is unlikely that the total number of database requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Requests The number of requests received by all database instances on this server. Requests cover a range of operations: connection negotiations, session configuration, data definition language (DDL), data modification language (DML), or data reads (select).
Responses The number of responses by all database instances on this server. Responses vary by the requested operation. For example, a response can include connection and session configurations, success or failure notifications, or a tabular data set.
Errors The number of database request operations that failed on all database instances running on this server. All database errors should be investigated.
Aborted Requests The number of requests that this database server began to receive, but did not receive completely.
Aborted Responses The number of responses that this database server began to send, but did not send completely.
Request and Response Size
This chart shows the average size of requests and responses.
Request Size The distribution of sizes (in bytes) of requests that the device received when acting as a database server.
Response Size The distribution of sizes (in bytes) of responses that the device sent when acting as a database server.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

Database client group page

Database Summary for Group

Total Transactions
This chart shows you when database errors occurred and how many database responses the servers sent. This information can help you see how active the servers were at the time they returned the errors.

If you see a large number of errors, you can drill down to find the specific status codes returned in the requests and learn why servers were unable to fulfill the requests. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of database requests to database responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Tip:To see more information about errors, click the Errors link at the top of the page.
Metric Description
Responses The number of responses received by this database client. Responses vary by the requested operation.
Errors The number of database request operations sent from this database client that failed. All database errors should be investigated.
Server Processing Time
If a client group is acting slow, the server processing time can help you figure out whether the issue is with the servers. The Server Processing Time chart shows the median amount of time servers took to process requests from the clients. High server processing times indicate that the clients are contacting slow servers.
Metric Description
Database Client Server Processing Time The time it took for this database client to receive the first packet of a response after sending the last packet of the query.
Database Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, the clients might be sending more requests than servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Metric Description
Requests The number of requests sent by this database client. Requests cover a range of operations: connection negotiations, session configuration, data definition language (DDL), data modification language (DML), or data reads (select).
Responses The number of responses received by this database client. Responses vary by the requested operation.
Errors The number of database request operations sent from this database client that failed. All database errors should be investigated.
Aborted Requests The number of requests that this database client began to send, but did not send completely.
Aborted Responses The number of responses that this database client began to receive, but did not receive completely when acting as a database client.

Database Details for Group

Top Database Clients in Group
This chart shows which database clients in the group were most active by breaking out the total number of database requests the group sent by client.
Top Methods
This chart shows which database methods the group called the most by breaking out the total number of requests the group sent by method.
Top Status Codes
This chart shows which database status codes the group received the most by breaking out the number of responses returned to the group by status code.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

Database server group page

Database Summary for Group

Total Transactions
This chart shows you when database errors occurred and how many database responses the servers sent. This information can help you see how active the servers were at the time they returned the errors.

If you see a large number of errors, you can drill down to find the specific status code returned in the request and learn why the servers were unable to fulfill the requests. However, if the number of errors is low, the issue might be more complex, and you should examine the ratio of database requests to database responses. In a healthy environment, the number of requests and responses should be roughly equal. For more information, see the Metric Totals section below.

Tip:To see more information about errors, click the Errors link at the top of the page.
Responses The number of responses by all database instances on this server. Responses vary by the requested operation. For example, a response can include connection and session configurations, success or failure notifications, or a tabular data set.
Errors The number of database request operations that failed on all database instances running on this server. All database errors should be investigated.
Server Processing Time
The Server Processing Time chart shows the median amount of time the servers took to process requests from clients. High server processing times indicate that the servers in a group are slow.
Server Processing Time The time it took for this database server to send the first packet of a response after receiving the last packet of the query.
Database Metric Totals
Requests and responses represent the conversation taking place between clients and servers. If there are more requests than responses, clients might be sending more requests than the servers can handle or the network might be too slow.
Note:It is unlikely that the total number of requests and responses will be exactly equal, even in a healthy environment. For example, you might be viewing a time period that captures a response to a request that was sent before the start of the time period. In general, the greater the difference between responses and errors, the greater the chance that there is an issue with those transactions.
Requests The number of requests received by all database instances on this server. Requests cover a range of operations: connection negotiations, session configuration, data definition language (DDL), data modification language (DML), or data reads (select).
Responses The number of responses by all database instances on this server. Responses vary by the requested operation. For example, a response can include connection and session configurations, success or failure notifications, or a tabular data set.
Errors The number of database request operations that failed on all database instances running on this server. All database errors should be investigated.
Aborted Requests The number of requests that this database server began to receive, but did not receive completely.
Aborted Responses The number of responses that this database server began to send, but did not send completely.

Database Details for Group

Top Database Servers in Group
This chart shows which database servers in the group were most active by breaking out the total number of database responses the group sent by server.
Top Methods
This chart shows which database methods were called on servers in the group the most by breaking out the total number of requests the group received by method.
Top Status Code
This chart shows which database status codes the groups returned the most by breaking out the total number of responses the group sent by status code.

Where to look next

Drill down on a metric: You can get more information about a metric by clicking the metric value or name and selecting an option from the Drill down by menu. For example, if you are looking at the total number of errors, click the number and select Servers to see which servers returned the errors.

Search the Metric Explorer: Built-in protocol pages include the most commonly referenced metrics for a protocol, but you can see additional metrics in the Metric Explorer. Click any chart title on a protocol page and select Create chart from.... When the Metric Explorer opens, click Add Metric in the left pane to display a drop-down list of comprehensive metrics for the device. If you find an interesting metric, click Add to Dashboard to add the metric to a new or existing dashboard.

Create a custom metric: If you want to view a metric that is not included in the Metric Explorer, you can create a custom metric through a trigger. For more information, see the following resources:

Check the Solution Bundles Gallery: Many bundles on the Solution Bundles Gallery contain custom metrics. Before you create your own custom metric, check to see if someone has already created a similar metric.

Database devices page

Note:This help topic describes a page in the ExtraHop Web UI that is available only when switched to the deprecated page layout.
Database Devices Toolbar
The Database device toolbar includes the following controls:
Database Metric Type
Displays statistics for the current device acting as a database client or database server.
Errors
Displays the list of error messages sent to or received by the current device over the time interval.
Methods
Displays the list of names and the associated number of responses and errors.
Users
Displays the list of users accessing the database server and associated bytes sent and received for the selected time interval.
Clients or Servers
Displays the associated client IP addresses when the device is acting as a server, and the associated server IP addresses when acting as a client.

Click the counters next to individual database metrics to show the IP Address Database Metrics for database peer devices. For database servers, the peer devices are database clients. For database clients, the peer devices are database servers.

Device Details
Click the counters next to individual database metrics to show the IP Address Database Metrics for database peer devices. For database servers, the peer devices are database clients. For database clients, the peer devices are database servers.
By IP
Displays database metrics by IP address.
By Database
Displays database metrics by database. For local peer devices, the link leads to that device. For remote peer devices, the link leads to the gateway device through which the requests were routed.
Database Client
If you select Client for the Database Metric Type, the Discover appliance displays the following metrics:
Responses
Specifies the number of responses that the device received when acting as a database client. Click to display the list of servers from which responses were sent.
Errors
Specifies the number of database protocol errors for the selected time interval. Click to display the list of servers for which there were errors.
Requests Aborted
Specifies the number of requests that the device began to send but did not send completely when acting as a database client.
Responses Aborted
Specifies the number of responses that the device began to receive but did not receive completely when acting as a database client.
Database Server
If you select Server for the Database Metric Type, the Discover appliance displays the following metrics:
Response