Forensic Data Using the API
GigaFlows forensics API can be queried using a rest/json based HTTP/HTTPS request. A list of available request parameters is available at Forensic Data Using the API Request Fields.
API Endpoint
Requests should be sent to the api endpoint using the following URL convention:
http(s)://SERVER_IP:SERVER_PORT/rest/
If you aren’t making requests from a logged in web session, you can either supply your user credentials in the URL or use the "portalUser" functionality This allows requests from a predefined IP address (or regexp) and userid.
Failure in supplying suitable user credentials will result in the following message being returned
{"errors":"notLoggedIn","returnCode":"error"}
Request using user credentials
http://172.21.21.1:7902/rest/?userName=reportUser&userPass=reportuser254&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
Request using user Portal
http://172.21.21.1:7902/rest/?portalUser=admin&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
API Request
Gigaflow expects a post containing a single JSON object under the jsonRequest input form data field, e.g.
jsonRequest={"action":"get",
"type":"Graph",
"sourcetype":"forensics",
"device":["172.21.40.199","172.21.21.254"],
"starttime":1525086647000,
"endtime":1525093847000,
"reportName":"Application Flows"
}
or via URL params
jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows"}
The various (case sensitive) fields in this example are:
| Param | Meaning |
|---|---|
| type | graph or table depending on whether you want report results grouped by time or as a single table output. |
| sourcetype | Should always be forensics for accessing the forensics results |
| reportName | The name of the report you want to query, this will determine the fields and grouping of data returned. In this example, the “Application Flows” report is requested. A complete list of available reports and their configuration are available from within GigaFlow via Configuration->Reporting->Forensics Reports. From there you can select the report and see which fields are included and what grouping occurs. You can also clone existing reports to create new reports. |
| starttime | This is the start time for the data to be included in the report. This can either be in exact UTC milliseconds or a period relative to now e.g. "4 hours" |
| endtime | This is the end time for the data to be included in the report. This can either be in exact UTC milliseconds or a period relative to now e.g. "2 hours" |
| device | This should be an array [] containing the IP address of the infrastructure devices you want to have data processed from and included in this report. In this example data from both “172.21.40.199” and “172.21.21.254” will be included in this report. Where possible you should always specify the minimum number of devices required for the report. Excluding the device param will run the report over all devices in the database. |
When requesting a report, you can also supply a myid parameter, the will be attached to the query and the results so that they can be retrieved at a later date. It is highly recommended that if you specify this field, make sure that the ids are always unique
jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
If you don't specify a myid field, one will automatically be populated in the format of "report_TIMESTAMP_UUID".
{"action":"get","device":["172.21.40.199","172.21.21.254"],"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a","reportName":"Application Flows","sourcetype":"forensics","type":"Graph"}
API Response
When you make a request for forensics, normally the request session is held open until the data is available and returned in the response.
If you want to fire of a request but not wait for the results, you can add the "async" parameter to the request.
This will run the query but rather than wait and return the data, it will immediately just return the myid value for that request which can be used later to retrieve the data.
Request
jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","device":["172.21.40.199"],"reportName":"Application Flows","starttime":"4 hours","async":true}
Response
{"myid":"report_1578922060666_ccf5ac9e-692f-4bb2-83d0-6565640a2378"}
If you have the myid value, you can search to see if its still currently running or you can pull back the data from a completed query with that myid
Searching for completed query data
jsonRequest={"type":"queriescomplete","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}
Will return an empty results set (if query hasn't completed )
{"results":[]}
Or will return similar to
{"results":[{"duration":12999,"request":{"action":"get","device":["88.151.80.121"],
"myid":"report_1578926758090_70ce19c7-e66c-4c68-8767-04d1f22c249b",
"queryid":"1578926758087_2116768245_499073504","reportName":"Application Flows",
"sourcetype":"forensics","type":"Graph"},"returned":{"data":[
{"data":[[1578919440000,2570322],[1578919560000,264949293],[1578919680000,251347271],[1578919800000,244204257],[1578919920000,246595508],
[1578920040000,299455178],[1578920160000,294232363],[1578920280000,260187642],
[1578920400000,271777174],[1578920520000,252957499],[1578920640000,300129638],
[1578920760000,268875509],[1578920880000,251809442],
........
"keysAsParams":["null","null","null"],"keysDisplay":["{\"raw\":\"Others.\"}","",""],"others":true}],
"totalCount":16330863160,"totalEntries":61,"totalSeconds":7200},"tableInformation": {"maxx":1578926758089,"minx":1578919558089,
"moderator":120000,"reportName":"Application Flows","tableShowZebra":true},"timezone":{"id":"Europe/London","offset":"Z"}},"runtime":1578926758090,"userid":"admin"}]}
Searching for queries still running This can also be found in Reports->All Current Forensics
jsonRequest={"type":"queriesoutstanding","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}
Will return an empty results set (if query hasn't completed )
{"results":[]}
Or will return similar to
{"results":[{"request":{"action":"get","device":["88.151.80.121"],
"directfilter":"","endtime":1578926758000,"minwindow":"-1",
"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a",
"queryid":"1578927374204_2116768245_1664658726","reportName":"Application Flows","sourcetype":"forensics",
"starttime":1578919558000,"type":"Graph"},"runtime":1578927374207,"user":"admin"}]}
Simple reports available for forensics reporting:
"Address Pairs" "Addresses By Dest" "Addresses By Source" "All Fields" "Application Flows" "Applications" "ASs By Dest" "ASs By Source" "ASs Pairs" "Class of Service" "FW Event" "FW Ext Code" "Interface Pairs" "Interfaces By Dest" "Interfaces By Source" "MAC Address Pairs" "MAC Addresses By Dest" "MAC Addresses By Source" "Ports By Dest" "Ports By Source" "Posture" "Protocols" "Sessions Flows" "Sessions With Ints" "Sessions" "Subnet Class A By Dest" "Subnet Class A By Source" "Subnet Class B By Dest" "Subnet Class B By Source" "Subnet Class C By Dest" "Subnet Class C By Source" "TCP Flags" "URLs" "Users"
Enhanced reports available:
"Interfaces By Source Pct" "Interfaces By Destination Pct" "Applications With Flow Count" "Application Flows With User" "All Fields Max FPR" "All Fields Avg FPR" "Duration Max" "Duration Avg" "Ports As Dest By Dest Add Popularity" "Addresses As Sources And Dest Port By Dest Count" "Ports As Dest By Src Add Popularity" "Ports As Src By Dest Add Popularity" "Ports As Src By Src Add Popularity" "Servers As Src Address" "Servers As Dst Address" "Servers As Src With Ports" "Servers As Dest With Ports"
Key fields for each report:
"Application Flows" srcadd, dstadd, appid
"Address Pairs" srcadd, dstadd
"MAC Address Pairs" srcmac, dstmac
"Interfaces By Source" device, device||'_'||inif as difin
"Interfaces By Dest" device, device||'_'||outif as difout
"Interface Pairs" device, device||'_'||inif as difin, device||'_'||outif as difout
"Sessions" srcadd, srcport, dstadd, dstport, appid
"Sessions Flows" firstseen, srcadd, srcport, dstadd, dstport, appid, proto
"Sessions With Ints" srcadd, srcport, inif, dstadd, dstport, outif, appid
"Protocols" proto
"Posture" postureid
"Applications" appid
"URLs" url
"Ports By Source" srcport
"Ports By Dest" dstport
"TCP Flags" flags
"FW Event" fwevent
"FW Ext Code" fwextcode
"Class of Service" tos
"ASs Pairs" srcas, dstas
"ASs By Source" srcas
"ASs By Dest" dstas
"Users" userid
"MAC Addresses By Source" srcmac
"MAC Addresses By Dest" dstmac
"Addresses By Source" srcadd
"Subnet Class A By Dest" dstadd-modulus(dstadd,16777216) as dstsubneta
"Subnet Class B By Dest" dstadd-modulus(dstadd,65536) as dstsubnetb
"Subnet Class C By Dest" dstadd-modulus(dstadd,256) as dstsubnetc
"Subnet Class A By Source" srcadd-modulus(srcadd,16777216) as srcsubneta
"Subnet Class B By Source" srcadd-modulus(srcadd,65536) as srcsubnetb
"Subnet Class C By Source" srcadd-modulus(srcadd,256) as srcsubnetc
"Addresses By Dest" dstadd
"Interfaces By Source Pct" {"difin"}, "pct_avg_in", "afirstseen"
"Interfaces By Destination {"difout"}, "pct_avg_out", "afirstseen"
"Applications With Flow Count" {"appid"}, "flowcount", "afirstseen"
"Application Flows With User" {"srcadd", "dstadd", "appid", "userid"}, "bits_avgsec", "afirstseen"
"All Fields" {}, "bits_avgsec", "afirstseen"
"All ID Fields" new int[]{reportDomainForensics}, ROS.forensicsMinDuration, reportTypeForensics
"All Fields Max FPR" {}, "maxfpr", "afirstseen"
"All Fields Avg FPR" {}, "maxfpr", "afirstseen"
"Duration Max" {}, "maxduration", "afirstseen"
"Duration Avg" {}, "avgduration", "afirstseen"
"Ports As Dest By Dest Add Popularity" {"dstport"}, "dstcount", "afirstseen"
"Addresses As Sources And Dest Port By Dest Count" {"srcadd", "dstport"}, "dstcount", "afirstseen"
"Ports As Dest By Src Add Popularity" {"dstport"}, "srccount", "afirstseen"
"Ports As Src By Dest Add Popularity" {"srcport"}, "dstcount", "afirstseen"
"Ports As Src By Src Add Popularity" {"srcport"}, "srccount", "afirstseen"
"Servers As Src Address" {"srcadd"}, "bits_avgsec", "afirstseen"
"Servers As Dst Address" {"dstadd"}, "bits_avgsec", "afirstseen"
"Servers As Src With Ports" {"srcadd", "srcport"}, "dstsavgsec", "afirstseen"
"Servers As Dest With Ports" {"dstadd", "dstport"}, "srcsavgsec", "afirstseen"
Fields available:
Application ID as selected after profiling:appid Device ID:device Device And Input Interface ID:device||'_'||inif as difin Device And Output Interface ID:device||'_'||outif as difout Destination IP Address:dstadd Destination Class A Subnet:dstadd-modulus(dstadd,16777216) as dstsubneta Destination Class C Subnet:dstadd-modulus(dstadd,256) as dstsubnetc Destination Class B Subnet:dstadd-modulus(dstadd,65536) as dstsubnetb Destination AS Number:dstas Destination MAC Address:dstmac Destination IP Port Number:dstport MS Timestamp Of When Flow Record Was First Seen:firstseen TCP Flags:flags Forwarding Event Code:fwevent First Packet Response:fpr Extended Event Code:fwextcode Input Interface ID:inif Output Interface ID:outif Posutre ID Of Flow As Marked After Processing:postureid IP Protocol Nunmber:proto Source IP Address: srcadd Source Class A Subnet:srcadd-modulus(srcadd,16777216) as srcsubneta Source Class C Subnet:srcadd-modulus(srcadd,256) as srcsubnetc Source Class B Subnet:srcadd-modulus(srcadd,65536) as srcsubnetb Source AS number:srcas Source MAC Address:srcmac Source IP Port:srcport TOS/QOS Marking:tos Text Field:url User ID:userid