Difference between revisions of "Event Records"

From Observer GigaFlow Support | VIAVI Solutions Inc.
Jump to: navigation, search
(What is Recorded?)
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Database]]
 +
[[Category:Scripts]]
 
An Observer GigaFlow Event is a record that is created when a monitored flow pattern matches certain criteria.  
 
An Observer GigaFlow Event is a record that is created when a monitored flow pattern matches certain criteria.  
  
Line 11: Line 13:
 
* A connected device that stops sending flows.
 
* A connected device that stops sending flows.
  
[http://wiki.anuview.net/manual/glossary.html See the Event definition in the Glossary]
+
[http://gigaflowsupport.viavisolutions.com/manual/glossary.html See the Event definition in the Glossary]
[http://wiki.anuview.net/manual/events.html and the main entry for Events in the Reference Manual]
+
[http://gigaflowsupport.viavisolutions.com/manual/events.html and the main entry for Events in the Reference Manual]
  
 
== What is Recorded? ==
 
== What is Recorded? ==
  
All GigaFlow flow records contain 29 fields or table columns. In the '''myipfix''' database associated with the GigaFlow installation, the '''netflow''' tables contain all flow records. See [http://wiki.anuviewsoftware.com/index.php/Flow_Record_Fields Flow Record Fields] for the complete list.   
+
All GigaFlow flow records contain 29 fields or table columns. In the '''myipfix''' database associated with the GigaFlow installation, the '''netflow''' tables contain all flow records. See [http://gigaflowsupport.viavisolutions.com/index.php/Flow_Record_Fields Flow Record Fields] for the complete list.   
  
Similarly, the Events tables contain all the Event records. See [http://wiki.anuviewsoftware.com/index.php/Event_Record_Fields Event Record Fields] for a complete list with descriptions. The Event record fields are:  
+
Similarly, the Events tables contain all the Event records. See [http://gigaflowsupport.viavisolutions.com/index.php/Event_Record_Fields Event Record Fields] for a complete list with descriptions. The Event record fields are:  
  
 
<code>
 
<code>
Line 35: Line 37:
 
| int
 
| int
 
| -11
 
| -11
| Attempted access by blacklisted resources.
+
| Attempted access by blacklisted resources. See also [http://gigaflowsupport.viavisolutions.com/manual/watchlists.html Watchlists in the Reference Manual].
 
|-
 
|-
 
| postureBlackListDst
 
| postureBlackListDst
 
| int
 
| int
 
| -12
 
| -12
| Attempted access to blacklisted resources.
+
| Attempted access to blacklisted resources. See also [http://gigaflowsupport.viavisolutions.com/manual/watchlists.html Watchlists in the Reference Manual].
 
|-
 
|-
 
| profilerException
 
| profilerException
 
| int
 
| int
 
| -20
 
| -20
| Profile exceptions, i.e. GigaFlow administrator-defined behaviours deviating from norms. See also [http://wiki.anuview.net/manual/configurationprofiling.html Profiling in the Reference Manual].
+
| Profile exceptions, i.e. GigaFlow administrator-defined behaviours deviating from norms. See also [http://gigaflowsupport.viavisolutions.com/manual/configurationprofiling.html Profiling in the Reference Manual].
 
|}
 
|}
  
Line 59: Line 61:
 
| int
 
| int
 
| -100
 
| -100
| SYN Source Network Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Source Network Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synSrcPortException
 
| synSrcPortException
 
| int
 
| int
 
| -101
 
| -101
| SYN Source Port Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Source Port Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synSrcIPException
 
| synSrcIPException
 
| int
 
| int
 
| -102
 
| -102
| SYN Source Unreachable IP. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Source Unreachable IP. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synSrcIPAndPortException
 
| synSrcIPAndPortException
 
| int
 
| int
 
| -103
 
| -103
| SYN Source Network And Port Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Source Network And Port Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synDst
 
| synDst
 
| int
 
| int
 
| -120
 
| -120
| SYN Destination Network Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Destination Network Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synDstPortException
 
| synDstPortException
 
| int
 
| int
 
| -121
 
| -121
| SYN Destination Port Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Destination Port Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synDstIPException
 
| synDstIPException
 
| int
 
| int
 
| -122
 
| -122
| SYN Destination Unreachable Server. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Destination Unreachable Server. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| synDstIPAndPortException
 
| synDstIPAndPortException
 
| int
 
| int
 
| -123
 
| -123
| SYN Destination Network And Port Sweep. See also [http://wiki.anuview.net/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
+
| SYN Destination Network And Port Sweep. See also [http://gigaflowsupport.viavisolutions.com/manual/systemwidereports.html#synforensicsmonitoring SYN Forensics Monitoring in the Reference Manual].
 
|-
 
|-
 
| lldpNewNeighbour
 
| lldpNewNeighbour
Line 121: Line 123:
 
Data can be accessed:
 
Data can be accessed:
  
# As a normal user, via the GigaFlow interface. See [http://wiki.anuview.net/manual/events.html Events in the Reference Manual].
+
# As a normal user, via the GigaFlow interface. See [http://gigaflowsupport.viavisolutions.com/manual/events.html Events in the Reference Manual].
# As an automated portal user, via the GigaFlow API. See [http://wiki.anuviewsoftware.com/index.php/Forensics/APIs/forensics Forensic Data Using the API].
+
# As an automated portal user, via the GigaFlow API. See [http://gigaflowsupport.viavisolutions.com/index.php/Forensics/APIs/forensics Forensic Data Using the API].
 
# From the database directly using a viewer such as [https://www.pgadmin.org/ pgAdmin] or in code using a [https://wiki.postgresql.org/ postgreSQL] connector such as [https://wiki.postgresql.org/wiki/Psycopg2_Tutorial Psycopg2] for Python.
 
# From the database directly using a viewer such as [https://www.pgadmin.org/ pgAdmin] or in code using a [https://wiki.postgresql.org/ postgreSQL] connector such as [https://wiki.postgresql.org/wiki/Psycopg2_Tutorial Psycopg2] for Python.
  
 
== Sample Records ==
 
== Sample Records ==
  
Typical records from an Event table with obfuscated IP addresses and locations:
+
See [http://gigaflowsupport.viavisolutions.com/index.php/Event_Record_Fields#Sample_Records Event Record Fields for some typical records from an Event table with obfuscated IP addresses and locations]. As an example:
  
 
'''postureBlackListSrc (-11)'''
 
'''postureBlackListSrc (-11)'''
 +
 +
Cut and pasted directly from an Events table:
  
 
<code>
 
<code>
Line 135: Line 139:
 
</code>
 
</code>
  
'''synSrc (-100)'''
+
Arranged more neatly in a table:
  
<code>
+
{| class="wikitable" style="text-align: left;"
805752  0              "-1407899398"  "1541583572146"            -100      8              "xxx.xxx.86.55"    "Syn Src Port Sweep xxx.xxx.86.55->xxx.xxx.85.74"          1              "{"Application":"TCP/3392","Eventer":"xxx.xxx.86.55","Syn Type":"Source","appid":396608,"bytes":40,"device":"xxx.xxx.40.250","domain":"","dstadd":"xxx.xxx.85.74","dstport":3392,"duration":0,"eventname":"Syn Src Port Sweep","flags":2,"fwevent":0,"fwextcode":0,"inif":12,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":30,"packets":1,"proto":6,"srcadd":"xxx.xxx.86.55","srcport":43017,"time":1541583603834,"timeH":"7-Nov-2018 09:40:03.834","tos":40,"user":""}"        "Syn Source"      100        100        "Syn Src Port Sweep"    "Many" "RU"                "St.-Petersburg"              "0.00000000000"            "0.00000000000"            "IE"        "County Sligo"                "0.00000000000"            "0.00000000000"
+
|'''Record Field'''
</code>
+
|'''Value'''
 
+
|-
'''synDst (-120)'''
+
|id
 
+
|805729  
<code>
+
|-
805738 0             "29885185"        "1541583568627"            -120      8              "Many" "Syn Dst Unreachable Server xxx.xxx.80.248->xxx.xxx..45.208"         1              "{"Application":"HTTP TCP/80","Eventer":"xxx.xxx.45.208","Syn Type":"Destination","appid":393296,"bytes":44,"device":"xxx.xxx.3.1","domain":"","dstadd":"xxx.xxx.45.208","dstport":80,"duration":0,"eventname":"Syn Dst Unreachable Server","flags":2,"fwevent":0,"fwextcode":0,"inif":8,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":3,"packets":1,"proto":6,"srcadd":"xx.xxx..80.248","srcport":54028,"time":1541583584143,"timeH":"7-Nov-2018 09:39:44.143","tos":32,"user":""}"        "Syn Destination"            100        100        "Syn Dst Unreachable Server"                "xxx.xxx.45.208"                "CN"      "Guangdong"    "0.00000000000"            "0.00000000000"          "IE"        "null"                "0.00000000000"            "-0.00000000000"
+
|customerid
</code>
+
|0
 
+
|-
'''postureBlackListDst (-12)'''
+
|device
 
+
|"-1407899398"
<code>
+
|-
805801  0              "-1407899398"  "1541583616150"            -12          8              "xxx.xxx.195.7"   "Black List Dst Hit(Apache(WWW) Scan/Brute) xxx.xxx.86.213->xxx.xxx.195.7"     1             "{"Application":"HTTPS TCP/443","Black List":"https://lists.blocklist.de/lists/apache.txt","Black List Type":"Destination","Eventer":"xxx.xxx.195.7","appid":393659,"bytes":104,"device":"xxx.xxx.40.250","domain":"","dstadd":"xxx.xxx.195.7","dstport":443,"duration":0,"eventname":"Black List Dst","flags":16,"fwevent":0,"fwextcode":0,"inif":30,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":12,"packets":2,"proto":6,"srcadd":"xxx.xxx.86.213","srcport":51330,"time":1541583646846,"timeH":"7-Nov-2018 09:40:46.846","tos":0,"user":""}"         "https://lists.blocklist.de/lists/apache.txt"           75           25                "Apache(WWW) Scan/Brute"     "xxx.xxx.86.213"                 "IE"       "County Sligo"   "0.00000000000"           "0.00000000000"                 "US"     "null"     "0.00000000000"           "0.00000000000"
+
|firstseen
</code>
+
|"1541583542242"
 +
|-
 +
|eventtype
 +
| -11
 +
|-
 +
|eventsrctype
 +
|8
 +
|-
 +
|eventsrc
 +
|"xxx.xxx.208.28"
 +
|-
 +
|message
 +
|"Black List Src Hit(Botnet Strong) xxx.xxx.208.28->xxx.xxx.166.159"
 +
|-
 +
|datatype
 +
|1
 +
|-
 +
|data
 +
|{"Application":"SSH TCP/22","Black List":"https://lists.blocklist.de/lists/strongips.txt","Black List Type":"Source","Eventer":"xxx.xxx.208.28","appid":393238,"bytes":44,"device":"xxx.xxx.40.250","domain":"","dstadd":"xx.xxx.166.159","dstport":22,"duration":0,"eventname":"Black List Src","flags":2,"fwevent":0,"fwextcode":0,"inif":14,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":13,"packets":1,"proto":6,"srcadd":"xxx.xxx.208.28","srcport":28148,"time":1541583574022,"timeH":"7-Nov-2018 09:39:34.22","tos":0,"user":""}"  
 +
|-
 +
|datasource
 +
|"https://lists.blocklist.de/lists/strongips.txt"
 +
|-
 +
|confidence
 +
|75
 +
|-
 +
|severity
 +
|80
 +
|-
 +
|category
 +
|"Botnet Strong"
 +
|-
 +
|target
 +
|"xxx.xxx.166.159"
 +
|-
 +
|country_src
 +
|"CN"
 +
|-
 +
|division_src
 +
|"Gansu"
 +
|-
 +
|latit_src
 +
|"0.00000000000"
 +
|-
 +
|longd_src
 +
|"0.00000000000"
 +
|-
 +
|country_dst
 +
|"IE"
 +
|-
 +
|division_dst
 +
|"County Sligo"
 +
|-
 +
|latit_dst
 +
|"0.00000000000"
 +
|-
 +
|longd_dst
 +
|"0.00000000000"
 +
|}

Latest revision as of 11:31, 29 November 2018

An Observer GigaFlow Event is a record that is created when a monitored flow pattern matches certain criteria.

Contents

What Generates an Event?

Some things that will trigger an event record include:

  • Attempts to access blacklisted resources.
  • Profile exceptions, i.e. behaviours deviating from norms.
  • A SYN flood event.
  • A lost neighbour.
  • A new connected device sending flows.
  • A connected device that stops sending flows.

See the Event definition in the Glossary and the main entry for Events in the Reference Manual

What is Recorded?

All GigaFlow flow records contain 29 fields or table columns. In the myipfix database associated with the GigaFlow installation, the netflow tables contain all flow records. See Flow Record Fields for the complete list.

Similarly, the Events tables contain all the Event records. See Event Record Fields for a complete list with descriptions. The Event record fields are:

id, customerid, device, firstseen, eventtype, eventsrctype, eventsrc, message, datatype, data, datasource, confidence, severity, category, target, country_src, division_src, latit_src, longd_src, country_dst, division_dst, latit_dst, longd_dst

Three events have both eventtype and posture set in the netflow and event tables in the database, i.e. these three event types can be determined immediately from Netflow records:

Record Field Type Code Description
postureBlackListSrc int -11 Attempted access by blacklisted resources. See also Watchlists in the Reference Manual.
postureBlackListDst int -12 Attempted access to blacklisted resources. See also Watchlists in the Reference Manual.
profilerException int -20 Profile exceptions, i.e. GigaFlow administrator-defined behaviours deviating from norms. See also Profiling in the Reference Manual.

In addition, all of these event types have only eventtype set in the event tables, i.e. these event types are classified with additional information or information from many records and have no postureid set in the netflow tables:

Record Field Type Code Description
synSrc int -100 SYN Source Network Sweep. See also SYN Forensics Monitoring in the Reference Manual.
synSrcPortException int -101 SYN Source Port Sweep. See also SYN Forensics Monitoring in the Reference Manual.
synSrcIPException int -102 SYN Source Unreachable IP. See also SYN Forensics Monitoring in the Reference Manual.
synSrcIPAndPortException int -103 SYN Source Network And Port Sweep. See also SYN Forensics Monitoring in the Reference Manual.
synDst int -120 SYN Destination Network Sweep. See also SYN Forensics Monitoring in the Reference Manual.
synDstPortException int -121 SYN Destination Port Sweep. See also SYN Forensics Monitoring in the Reference Manual.
synDstIPException int -122 SYN Destination Unreachable Server. See also SYN Forensics Monitoring in the Reference Manual.
synDstIPAndPortException int -123 SYN Destination Network And Port Sweep. See also SYN Forensics Monitoring in the Reference Manual.
lldpNewNeighbour int -130 A new connected device sending flows.
lldpLostNeighbour int -131 A lost neighbour.
deviceRestartsSendingFlows int -150 A connected device that stopped sending flows restarts sending flows.
deviceStopsSendingFlows int -151 A connected device that stops sending flows.

How are the Records Accessed?

Data can be accessed:

  1. As a normal user, via the GigaFlow interface. See Events in the Reference Manual.
  2. As an automated portal user, via the GigaFlow API. See Forensic Data Using the API.
  3. From the database directly using a viewer such as pgAdmin or in code using a postgreSQL connector such as Psycopg2 for Python.

Sample Records

See Event Record Fields for some typical records from an Event table with obfuscated IP addresses and locations. As an example:

postureBlackListSrc (-11)

Cut and pasted directly from an Events table:

805729 0 "-1407899398" "1541583542242" -11 8 "xxx.xxx.208.28" "Black List Src Hit(Botnet Strong) xxx.xxx.208.28->xxx.xxx.166.159" 1 {"Application":"SSH TCP/22","Black List":"https://lists.blocklist.de/lists/strongips.txt","Black List Type":"Source","Eventer":"xxx.xxx.208.28","appid":393238,"bytes":44,"device":"xxx.xxx.40.250","domain":"","dstadd":"xx.xxx.166.159","dstport":22,"duration":0,"eventname":"Black List Src","flags":2,"fwevent":0,"fwextcode":0,"inif":14,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":13,"packets":1,"proto":6,"srcadd":"xxx.xxx.208.28","srcport":28148,"time":1541583574022,"timeH":"7-Nov-2018 09:39:34.22","tos":0,"user":""}" "https://lists.blocklist.de/lists/strongips.txt" 75 80 "Botnet Strong" "xxx.xxx.166.159" "CN" "Gansu" "0.00000000000" "0.00000000000" "IE" "County Sligo" "0.00000000000" "0.00000000000"

Arranged more neatly in a table:

Record Field Value
id 805729
customerid 0
device "-1407899398"
firstseen "1541583542242"
eventtype -11
eventsrctype 8
eventsrc "xxx.xxx.208.28"
message "Black List Src Hit(Botnet Strong) xxx.xxx.208.28->xxx.xxx.166.159"
datatype 1
data {"Application":"SSH TCP/22","Black List":"https://lists.blocklist.de/lists/strongips.txt","Black List Type":"Source","Eventer":"xxx.xxx.208.28","appid":393238,"bytes":44,"device":"xxx.xxx.40.250","domain":"","dstadd":"xx.xxx.166.159","dstport":22,"duration":0,"eventname":"Black List Src","flags":2,"fwevent":0,"fwextcode":0,"inif":14,"macdst":"00:00:00:00:00:00","macsrc":"00:00:00:00:00:00","outif":13,"packets":1,"proto":6,"srcadd":"xxx.xxx.208.28","srcport":28148,"time":1541583574022,"timeH":"7-Nov-2018 09:39:34.22","tos":0,"user":""}"
datasource "https://lists.blocklist.de/lists/strongips.txt"
confidence 75
severity 80
category "Botnet Strong"
target "xxx.xxx.166.159"
country_src "CN"
division_src "Gansu"
latit_src "0.00000000000"
longd_src "0.00000000000"
country_dst "IE"
division_dst "County Sligo"
latit_dst "0.00000000000"
longd_dst "0.00000000000"