Difference between revisions of "Forensic Data Using the API"

From Observer GigaFlow Support | VIAVI Solutions Inc.
Jump to: navigation, search
(API Response)
(API Endpoint)
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
[[Category:API]]
 
[[Category:API]]
 
[[Category:NetFlow]]
 
[[Category:NetFlow]]
GigaFlows forensics API can be queried using a rest/json based HTTP/HTTPS request.
+
GigaFlows forensics API can be queried using a rest/json based HTTP/HTTPS request. A list of available request parameters is available at [[Forensic Data Using the API Request Fields]].
  
 
== API Endpoint ==
 
== API Endpoint ==
 
Requests should be sent to the api endpoint using the following URL convention:
 
Requests should be sent to the api endpoint using the following URL convention:
 
  http(s)://SERVER_IP:SERVER_PORT/rest/
 
  http(s)://SERVER_IP:SERVER_PORT/rest/
 
  
 
If you aren’t making requests from a logged in web session, you can either supply your user credentials in the URL or use the "portalUser" functionality
 
If you aren’t making requests from a logged in web session, you can either supply your user credentials in the URL or use the "portalUser" functionality
Line 15: Line 14:
  
 
Request using user credentials
 
Request using user credentials
  http://172.21.21.1:7902/rest/?userName=reportUser&userPass=reportuser254
+
  http://172.21.21.1:7902/rest/?userName=reportUser&userPass=reportuser254&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
 +
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
  
 
Request using user Portal
 
Request using user Portal
  http://172.21.21.1:7902/rest/?portalUser=admin
+
  http://172.21.21.1:7902/rest/?portalUser=admin&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
 +
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
  
 
== API Request ==
 
== API Request ==
Line 34: Line 35:
 
or via URL params
 
or via URL params
  
  http://172.21.21.1:7902/rest/?jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
+
  jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
 
  "device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows"}
 
  "device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows"}
  
Line 64: Line 65:
 
When requesting a report, you can also supply a '''myid''' parameter, the will be attached to the query and the results so that they can be retrieved at a later date.
 
When requesting a report, you can also supply a '''myid''' parameter, the will be attached to the query and the results so that they can be retrieved at a later date.
 
It is highly recommended that if you specify this field, make sure that the ids are always unique
 
It is highly recommended that if you specify this field, make sure that the ids are always unique
  http://172.21.21.1:7902/rest/?jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
+
  jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
 
  "device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
 
  "device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}
 
If you don't specify a '''myid''' field, one will automatically be populated in the format of "report_TIMESTAMP_UUID".
 
If you don't specify a '''myid''' field, one will automatically be populated in the format of "report_TIMESTAMP_UUID".
  {"action":"get","device":["172.21.40.199","172.21.21.254"],"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a","reportName":"Application Flows","sourcetype":"forensics","type":"Graph"}
+
  {"action":"get","device":["172.21.40.199","172.21.21.254"],'''"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"''',"reportName":"Application Flows","sourcetype":"forensics","type":"Graph"}
  
 
== API Response ==
 
== API Response ==
When you make a request for forensics, normally the request session is held open until the data is available and returned in the response to the request.  
+
When you make a request for forensics, normally the request session is held open until the data is available and returned in the response.  
  
 
If you want to fire of a request but not wait for the results, you can add the "async" parameter to the request.
 
If you want to fire of a request but not wait for the results, you can add the "async" parameter to the request.
  
 
This will run the query but rather than wait and return the data, it will immediately just return the '''myid''' value for that request which can be used later to retrieve the data.
 
This will run the query but rather than wait and return the data, it will immediately just return the '''myid''' value for that request which can be used later to retrieve the data.
  jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","device":["172.21.40.199"],"reportName":"Application Flows","starttime":"4 hours","async":true}
+
 
 +
'''Request'''
 +
  jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","device":["172.21.40.199"],"reportName":"Application Flows","starttime":"4 hours",'''"async":true'''}
 +
'''Response'''
 +
{"myid":"report_1578922060666_ccf5ac9e-692f-4bb2-83d0-6565640a2378"}
 +
 
 +
If you have the '''myid''' value, you can search to see if its still currently running or you can pull back the data from a completed query with that myid
 +
 
 +
'''Searching for completed query data'''
 +
jsonRequest={"type":"queriescomplete","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}
 +
Will return an empty results set (if query hasn't completed )
 +
{"results":[]}
 +
Or will return similar to
 +
{"results":[{"duration":12999,"request":{"action":"get","device":["88.151.80.121"],
 +
"myid":"report_1578926758090_70ce19c7-e66c-4c68-8767-04d1f22c249b",
 +
"queryid":"1578926758087_2116768245_499073504","reportName":"Application Flows",
 +
"sourcetype":"forensics","type":"Graph"},"returned":{"data":[
 +
{"data":[[1578919440000,2570322],[1578919560000,264949293],[1578919680000,251347271],[1578919800000,244204257],[1578919920000,246595508],
 +
[1578920040000,299455178],[1578920160000,294232363],[1578920280000,260187642],
 +
[1578920400000,271777174],[1578920520000,252957499],[1578920640000,300129638],
 +
[1578920760000,268875509],[1578920880000,251809442],
 +
........
 +
"keysAsParams":["null","null","null"],"keysDisplay":["{\"raw\":\"Others.\"}","",""],"others":true}],
 +
"totalCount":16330863160,"totalEntries":61,"totalSeconds":7200},"tableInformation": {"maxx":1578926758089,"minx":1578919558089,
 +
"moderator":120000,"reportName":"Application Flows","tableShowZebra":true},"timezone":{"id":"Europe/London","offset":"Z"}},"runtime":1578926758090,"userid":"admin"}]}
 +
 
 +
'''Searching for queries still running''' This can also be found in Reports->All Current Forensics
 +
jsonRequest={"type":"queriesoutstanding","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}
 +
Will return an empty results set (if query hasn't completed )
 +
{"results":[]}
 +
Or will return similar to
 +
{"results":[{"request":{"action":"get","device":["88.151.80.121"],
 +
"directfilter":"","endtime":1578926758000,"minwindow":"-1",
 +
"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a",
 +
"queryid":"1578927374204_2116768245_1664658726","reportName":"Application Flows","sourcetype":"forensics",
 +
"starttime":1578919558000,"type":"Graph"},"runtime":1578927374207,"user":"admin"}]}
 +
 
  
  

Latest revision as of 13:08, 18 August 2020

GigaFlows forensics API can be queried using a rest/json based HTTP/HTTPS request. A list of available request parameters is available at Forensic Data Using the API Request Fields.

API Endpoint

Requests should be sent to the api endpoint using the following URL convention:

http(s)://SERVER_IP:SERVER_PORT/rest/

If you aren’t making requests from a logged in web session, you can either supply your user credentials in the URL or use the "portalUser" functionality This allows requests from a predefined IP address (or regexp) and userid.

Failure in supplying suitable user credentials will result in the following message being returned

{"errors":"notLoggedIn","returnCode":"error"} 

Request using user credentials

http://172.21.21.1:7902/rest/?userName=reportUser&userPass=reportuser254&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,

"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}

Request using user Portal

http://172.21.21.1:7902/rest/?portalUser=admin&jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,

"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}

API Request

Gigaflow expects a post containing a single JSON object under the jsonRequest input form data field, e.g.

jsonRequest={"action":"get",
"type":"Graph",
"sourcetype":"forensics",
"device":["172.21.40.199","172.21.21.254"],
"starttime":1525086647000,
"endtime":1525093847000,
"reportName":"Application Flows"
}

or via URL params

jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows"}

The various (case sensitive) fields in this example are:

Param Meaning
type graph or table depending on whether you want report results grouped by time or as a single table output.
sourcetype Should always be forensics for accessing the forensics results
reportName The name of the report you want to query, this will determine the fields and grouping of data returned. In this example, the “Application Flows” report is requested. A complete list of available reports and their configuration are available from within GigaFlow via Configuration->Reporting->Forensics Reports. From there you can select the report and see which fields are included and what grouping occurs. You can also clone existing reports to create new reports.
starttime This is the start time for the data to be included in the report. This can either be in exact UTC milliseconds or a period relative to now e.g. "4 hours"
endtime This is the end time for the data to be included in the report. This can either be in exact UTC milliseconds or a period relative to now e.g. "2 hours"
device This should be an array [] containing the IP address of the infrastructure devices you want to have data processed from and included in this report. In this example data from both “172.21.40.199” and “172.21.21.254” will be included in this report. Where possible you should always specify the minimum number of devices required for the report. Excluding the device param will run the report over all devices in the database.

When requesting a report, you can also supply a myid parameter, the will be attached to the query and the results so that they can be retrieved at a later date. It is highly recommended that if you specify this field, make sure that the ids are always unique

jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","starttime":1525086647000,"endtime":1525093847000,
"device":["172.21.40.199","172.21.21.254"],"reportName":"Application Flows","myid":"TestMYID 123456"}

If you don't specify a myid field, one will automatically be populated in the format of "report_TIMESTAMP_UUID".

{"action":"get","device":["172.21.40.199","172.21.21.254"],"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a","reportName":"Application Flows","sourcetype":"forensics","type":"Graph"}

API Response

When you make a request for forensics, normally the request session is held open until the data is available and returned in the response.

If you want to fire of a request but not wait for the results, you can add the "async" parameter to the request.

This will run the query but rather than wait and return the data, it will immediately just return the myid value for that request which can be used later to retrieve the data.

Request

jsonRequest={"action":"get","type":"Graph","sourcetype":"forensics","device":["172.21.40.199"],"reportName":"Application Flows","starttime":"4 hours","async":true}

Response

{"myid":"report_1578922060666_ccf5ac9e-692f-4bb2-83d0-6565640a2378"}

If you have the myid value, you can search to see if its still currently running or you can pull back the data from a completed query with that myid

Searching for completed query data

jsonRequest={"type":"queriescomplete","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}

Will return an empty results set (if query hasn't completed )

{"results":[]}

Or will return similar to

{"results":[{"duration":12999,"request":{"action":"get","device":["88.151.80.121"],
"myid":"report_1578926758090_70ce19c7-e66c-4c68-8767-04d1f22c249b",
"queryid":"1578926758087_2116768245_499073504","reportName":"Application Flows",
"sourcetype":"forensics","type":"Graph"},"returned":{"data":[
{"data":[[1578919440000,2570322],[1578919560000,264949293],[1578919680000,251347271],[1578919800000,244204257],[1578919920000,246595508],
[1578920040000,299455178],[1578920160000,294232363],[1578920280000,260187642], 
[1578920400000,271777174],[1578920520000,252957499],[1578920640000,300129638],
[1578920760000,268875509],[1578920880000,251809442],
........
"keysAsParams":["null","null","null"],"keysDisplay":["{\"raw\":\"Others.\"}","",""],"others":true}],
"totalCount":16330863160,"totalEntries":61,"totalSeconds":7200},"tableInformation": {"maxx":1578926758089,"minx":1578919558089,
"moderator":120000,"reportName":"Application Flows","tableShowZebra":true},"timezone":{"id":"Europe/London","offset":"Z"}},"runtime":1578926758090,"userid":"admin"}]}

Searching for queries still running This can also be found in Reports->All Current Forensics

jsonRequest={"type":"queriesoutstanding","myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a"}

Will return an empty results set (if query hasn't completed )

{"results":[]}

Or will return similar to

{"results":[{"request":{"action":"get","device":["88.151.80.121"],
"directfilter":"","endtime":1578926758000,"minwindow":"-1",
"myid":"report_1578915304695_4be2423e-d5cf-4c85-8da0-5e39c1393a8a",
"queryid":"1578927374204_2116768245_1664658726","reportName":"Application Flows","sourcetype":"forensics",
"starttime":1578919558000,"type":"Graph"},"runtime":1578927374207,"user":"admin"}]}


Simple reports available for forensics reporting:

"Address Pairs"
"Addresses By Dest"
"Addresses By Source"
"All Fields"
"Application Flows"
"Applications"
"ASs By Dest"
"ASs By Source"
"ASs Pairs"
"Class of Service"
"FW Event"
"FW Ext Code"
"Interface Pairs"
"Interfaces By Dest"
"Interfaces By Source"
"MAC Address Pairs"
"MAC Addresses By Dest"
"MAC Addresses By Source"
"Ports By Dest"
"Ports By Source"
"Posture"
"Protocols"
"Sessions Flows"
"Sessions With Ints"
"Sessions"
"Subnet Class A By Dest"
"Subnet Class A By Source"
"Subnet Class B By Dest"
"Subnet Class B By Source"
"Subnet Class C By Dest"
"Subnet Class C By Source"
"TCP Flags"
"URLs"
"Users"

Enhanced reports available:

"Interfaces By Source Pct" 
"Interfaces By Destination Pct"
"Applications With Flow Count" 
"Application Flows With User"
"All Fields Max FPR" 
"All Fields Avg FPR"
"Duration Max"
"Duration Avg" 
"Ports As Dest By Dest Add Popularity"
"Addresses As Sources And Dest Port By Dest Count"
"Ports As Dest By Src Add Popularity"
"Ports As Src By Dest Add Popularity"
"Ports As Src By Src Add Popularity"
"Servers As Src Address"
"Servers As Dst Address"
"Servers As Src With Ports"
"Servers As Dest With Ports" 

Key fields for each report:

"Application Flows"                                      srcadd, dstadd, appid
"Address Pairs"                                          srcadd, dstadd
"MAC Address Pairs"                                      srcmac, dstmac
"Interfaces By Source"                                   device, device||'_'||inif as difin
"Interfaces By Dest"                                     device, device||'_'||outif as difout
"Interface Pairs"                                        device, device||'_'||inif as difin, device||'_'||outif as difout
"Sessions"	       	                                  srcadd, srcport, dstadd, dstport, appid
"Sessions Flows"                                         firstseen, srcadd, srcport, dstadd, dstport, appid, proto
"Sessions With Ints"	                                  srcadd, srcport, inif, dstadd, dstport, outif, appid
"Protocols"	                                          proto
"Posture"	      	                                  postureid
"Applications"	   	                                  appid
"URLs"	             	                                  url
"Ports By Source"	                            	  srcport
"Ports By Dest"                                          dstport
"TCP Flags"	      	                                  flags
"FW Event"	                                          fwevent
"FW Ext Code"	                                          fwextcode
"Class of Service"	                                  tos
"ASs Pairs"	                                          srcas, dstas
"ASs By Source"                                          srcas
"ASs By Dest"	                                          dstas
"Users"                                                  userid
"MAC Addresses By Source"                                srcmac
"MAC Addresses By Dest"                      	          dstmac
"Addresses By Source"	                                  srcadd
"Subnet Class A By Dest"                      	          dstadd-modulus(dstadd,16777216) as dstsubneta
"Subnet Class B By Dest"                      	          dstadd-modulus(dstadd,65536) as dstsubnetb
"Subnet Class C By Dest"                      	          dstadd-modulus(dstadd,256) as dstsubnetc
"Subnet Class A By Source"                      	  srcadd-modulus(srcadd,16777216) as srcsubneta
"Subnet Class B By Source"                      	  srcadd-modulus(srcadd,65536) as srcsubnetb
"Subnet Class C By Source"	                          srcadd-modulus(srcadd,256) as srcsubnetc
"Addresses By Dest"	                                  dstadd
"Interfaces By Source Pct"                               {"difin"}, "pct_avg_in", "afirstseen"
"Interfaces By Destination                               {"difout"}, "pct_avg_out", "afirstseen"
"Applications With Flow Count"                           {"appid"}, "flowcount", "afirstseen"
"Application Flows With User"                            {"srcadd", "dstadd", "appid", "userid"}, "bits_avgsec", "afirstseen"
"All Fields"                                             {}, "bits_avgsec", "afirstseen"
"All ID Fields"                                          new int[]{reportDomainForensics}, ROS.forensicsMinDuration, reportTypeForensics
"All Fields Max FPR"                                     {}, "maxfpr", "afirstseen"
"All Fields Avg FPR"                                     {}, "maxfpr", "afirstseen"
"Duration Max"                                           {}, "maxduration", "afirstseen"
"Duration Avg"                                           {}, "avgduration", "afirstseen"
"Ports As Dest By Dest Add Popularity"                   {"dstport"}, "dstcount", "afirstseen"
"Addresses As Sources And Dest Port By Dest Count"       {"srcadd", "dstport"}, "dstcount", "afirstseen"
"Ports As Dest By Src Add Popularity"                    {"dstport"}, "srccount", "afirstseen"
"Ports As Src By Dest Add Popularity"                    {"srcport"}, "dstcount", "afirstseen"
"Ports As Src By Src Add Popularity"                     {"srcport"}, "srccount", "afirstseen"
"Servers As Src Address"                                 {"srcadd"}, "bits_avgsec", "afirstseen"
"Servers As Dst Address"                                 {"dstadd"}, "bits_avgsec", "afirstseen"
"Servers As Src With Ports"                              {"srcadd", "srcport"}, "dstsavgsec", "afirstseen"
"Servers As Dest With Ports"                             {"dstadd", "dstport"}, "srcsavgsec", "afirstseen"

Fields available:

Application ID as selected after profiling:appid
Device ID:device
Device And Input Interface ID:device||'_'||inif as difin
Device And Output Interface ID:device||'_'||outif as difout
Destination IP Address:dstadd
Destination Class A Subnet:dstadd-modulus(dstadd,16777216) as dstsubneta
Destination Class C Subnet:dstadd-modulus(dstadd,256) as dstsubnetc
Destination Class B Subnet:dstadd-modulus(dstadd,65536) as dstsubnetb
Destination AS Number:dstas
Destination MAC Address:dstmac
Destination IP Port Number:dstport
MS Timestamp Of When Flow Record Was First Seen:firstseen
TCP Flags:flags
Forwarding Event Code:fwevent
First Packet Response:fpr
Extended Event Code:fwextcode
Input Interface ID:inif
Output Interface ID:outif
Posutre ID Of Flow As Marked After Processing:postureid
IP Protocol Nunmber:proto
Source IP Address: srcadd
Source Class A Subnet:srcadd-modulus(srcadd,16777216) as srcsubneta
Source Class C Subnet:srcadd-modulus(srcadd,256) as srcsubnetc
Source Class B Subnet:srcadd-modulus(srcadd,65536) as srcsubnetb
Source AS number:srcas
Source MAC Address:srcmac
Source IP Port:srcport
TOS/QOS Marking:tos
Text Field:url
User ID:userid