Observer GigaFlow

Documentation

Table of Contents

Observer GigaFlow Documentation

Documentation > How-To Guide for GigaFlow

FAQs and Troubleshooting

Which Web Browers Are Supported by GigaFlow?

We test against the latest Mozilla Firefox, MicroSoft Edge and Google Chrome browsers.

What Server Specification Do We Need?

See here for a useful server size calculator.

Flows and Server Sizing

GigaFlow has been tested and certified to support up to 1,000 concurrent devices or up to 40,000 flows per second (flow/s) from less than 20 devices.

The flow rate changes with the number of connected devices as follows:

Flow/sNumber of Devices
50,00010
40,00020
20,00040
10,00080
5,000160
2,500300
1,250600
1,0001,000
Disk Throughput for Flow Writing to Database

Allow for at least 600 bytes per flow record per second for I/O throughput, i.e.

Flow/sBytes/s (Sustained Write Performance)MB/s (Sustained Write Performance)
10060,0000.06
2,0001,200,0001.2
10,0006,000,0006.0
40,00024,000,00024.0
Disk I/O for Flow Writing To Database

With
f = flow/s
d = number of devices
I = Input/Output performance measurement (IOP), nominally sustained sequential writing.

I = 20 + (f / 500) + (d / 5)

i.e. allow for a base of 20 IOPs, add an additional 1 IOP/s for every 500 flow/s and another 1 IOP/s for every 5 devices.

Flow/sNumber of DevicesIOPs
1,0001,000222
5,0001,000230
10,00010060
40,00010102
Disk I/O for Flow Reading from Database

Allow for at least 100 IOPs read.

Disk Sizing

The server must support at least 300 MB/s sustained read and write to handle the peak device or flow count. Anything less than this will result in dropped flows. For Linux, we recommend EXT4 or XFS file systems as well a dedicated RAID partition for the database. Adding a hardware RAID controller that supports RAID 10, or at least RAID 5, will improve performance and provide hardware redundancy. The amount of storage required is directly related to the flow rate and features enabled.

Data TypeMinimum Space Per Record (Bytes)
Forensics Flow250
Event Record900

500 flow/s of forensics == 450 MB per hour == 11 GB disk space per day.

RAM Sizing

A basic installation should have 4 GB RAM available for the OS and additional 50 MB per device to monitor. More RAM will always improve performance:

Number of DevicesMinimum RAM (GB)
104.5
1009
50029
1,00054
CPU Sizing

CPU sizing in GigaFlow is based on the Postgre SQL database. Overall performance is also dependent on CPU performance.

While there is little to gain by going beyond 8 cores, more powerful CPUs will provide a better experience. Intel's Xeon X5680 3GHz or Core i7-3770S 3GHz are recommended as a minimum required specification.

Experimental Results

As a demonstration, GigaFlow was installed on a typical server with the following specifications:

  • HP DL360 Gen 6.
  • 72 GB RAM.
  • 2x 8 core (Intel Xeon, E5530 @ 2.40GHZ).
  • 4x 146 GB SAS 10K RPM drives (432320-001 3GB/s) in a RAID 10 configuration.
  • P410i RAID controller with 1 GB RAM.
  • Linux CentOS 7, single partition.

The results of a performance test were as follows:

DevicesFlowsTotal FlowsCPU IdleDisk WriteIO WritesDisk UtilisationNotes
-per device s-1-%MB s-1s-1%-
1015,00015,000080882509
503,000150,000789126011
1001,500150,000789126111At limit of flow cache before flows are dropped.
250400100,00085621425Must double RAM used by GigaFlow to 1,536 MB.
500200100,000865819010Must double RAM used by GigaFlow to 1,536 MB.
1,000100100,000855923210Must double RAM used by GigaFlow to 1,536 MB.
2,00050100,000825922010Must double RAM used by GigaFlow to 1,536 MB.

These results show that this relatively mid-specification machine can cope with 50 devices at 150K flows per second. The same system can handle 2,000 devices with a cumulative count of 100K flows/s.

We recommend a maximum of 1,000 devices per GigaFlow server. Above this, database query performance will degrade.

Does GigaFlow have an API?

Yes, there is a REST endpoint for all report data with a portal user definitions to control access. You can open your GigaFlow system for integration with third party applications.

For more information, see API articles at the official GigaFlow Wiki.

Does GigaFlow Require a Client?

No, your GigaFlow system is accessed via a HTML/Javascript front-end using your preferred browser. Output is rendered as HTML, .csv or .pdf.