Probe crashes daily

Keenenick

Good day,

I running a probe in TAP node which seems to keep crash randomly, usually after running for anywhere from 3hrs to 18hrs.

I am not sure what the cause is and would like some assistance in troubleshooting.

Thank you for any assistance.

Regards,

Nick

admin

Hi Nick,

Can you the version of the Probe and the Platform ? trisul --version and uname -a

Can you also try the following

dmesg -T Shows if there was any segfault in trisul. If the output is too much , try dmesg -T | grep trisul
Check Dashboards > System Performance > Memory Usage - check if there is a memory issue
Can you check the Probe log files around the time it crashes ? TO do it

# load the helper aliases
source /usr/local/share/trisul-probe/trisbashrc

# go to the log directory
cd.l

# get the latest log file 
ls -lrt

Thanks,

Vivek

Keenenick

Hi Vivek,

Thank you for your help and response. Here is the output you requested. I hope I have provided enough info to assist you if finding an issue. I appreciate your help and love the product.

Last login: Sun Aug 18 15:50:17 2019 from 192.168.1.7
[root@netmon ~]# trisul --version
6.5.2960
[root@netmon ~]# uname -a
Linux netmon.internal.keene.co.za 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@netmon ~]#

I couldn't find anything helpful arround that time from dmesg.

Here is the output from the log at the time of the crash:

[32mMon Aug 19 11:19:00 2019.869694 DEBUG FlushAndWait [1] Event signalled, will globflush prune now TS = 08-19-2019 11:19:00-000000[m
[34mMon Aug 19 11:19:00 2019.871254 INFO XTRFDB:1 GLOBALFLUSH [1] at 08-19-2019 11:19:00-000000[m
[34mMon Aug 19 11:19:00 2019.871438 INFO XTRFDB:1 GLOBALFLUSH POOL Run [1] at 08-19-2019 11:19:00-000000[m
[32mMon Aug 19 11:19:00 2019.871555 DEBUG [1] FlushBuckets [ Country] in 00h 00m 00s 000043u (live=5,evictee=0) [m
[32mMon Aug 19 11:19:00 2019.871666 DEBUG [1] BktPruneLive [ Country] in 00h 00m 00s 000008u (master=8,prune=0,pflow=0,pflowtmout=0,chain=8) [m
[32mMon Aug 19 11:19:00 2019.871734 DEBUG [EDGETABLE] Country Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:00 2019.880460 DEBUG FlushAndWait [0] Event signalled, will globflush prune now TS = 08-19-2019 11:19:00-000000[m
[34mMon Aug 19 11:19:00 2019.880553 INFO XTRFDB:0 GLOBALFLUSH [0] at 08-19-2019 11:19:00-000000[m
[34mMon Aug 19 11:19:00 2019.880643 INFO XTRFDB:0 GLOBALFLUSH POOL Run [0] at 08-19-2019 11:19:00-000000[m
[32mMon Aug 19 11:19:00 2019.880718 DEBUG [0] FlushBuckets [ Country] in 00h 00m 00s 000019u (live=4,evictee=0) [m
[32mMon Aug 19 11:19:00 2019.880770 DEBUG [0] BktPruneLive [ Country] in 00h 00m 00s 000006u (master=8,prune=0,pflow=0,pflowtmout=0,chain=8) [m
[32mMon Aug 19 11:19:00 2019.880796 DEBUG [EDGETABLE] Country Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.457885 DEBUG [1] CG-Glob-Flush [ Country][86] in 00h 00m 00s 421841u (u=2942,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.457897 DEBUG [0] CG-Glob-Flush [ Country][86] in 00h 00m 00s 412680u (u=15,caches %=100/0/0)[m
[31mMon Aug 19 11:19:01 2019.483922 ERROR lua_pcall [firehol.lua] Error c=onbeginflush f=/usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/firehol/firehol.lua[m
[31mMon Aug 19 11:19:01 2019.484001 ERROR lua_pcall [firehol.lua] lua Error:...onfig/lua//github.com_trisulnsm_apps/firehol/firehol.lua:142: attempt to index local 'h' (a nil value)[m
[31mMon Aug 19 11:19:01 2019.484838 ERROR lua_pcall [firehol.lua] Error c=onbeginflush f=/usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/firehol/firehol.lua[m
[31mMon Aug 19 11:19:01 2019.484878 ERROR lua_pcall [firehol.lua] lua Error:...onfig/lua//github.com_trisulnsm_apps/firehol/firehol.lua:142: attempt to index local 'h' (a nil value)[m
[32mMon Aug 19 11:19:01 2019.486726 DEBUG [1] FlushBuckets [ External Hosts] in 00h 00m 00s 001802u (live=59,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.488850 DEBUG [1] BktPruneLive [ External Hosts] in 00h 00m 00s 000111u (master=111,prune=0,pflow=0,pflowtmout=0,chain=111) [m
[32mMon Aug 19 11:19:01 2019.489919 DEBUG [1] NewKeyFlush [ External Hosts] in 00h 00m 00s 001005u (1)[m
[32mMon Aug 19 11:19:01 2019.492908 DEBUG [EDGETABLE] External Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.494890 DEBUG [1] CG-Glob-Flush [ External Hosts][86] in 00h 00m 00s 026346u (u=3773,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.497060 DEBUG [0] FlushBuckets [ External Hosts] in 00h 00m 00s 012986u (live=58,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.497300 DEBUG [0] BktPruneLive [ External Hosts] in 00h 00m 00s 000100u (master=94,prune=0,pflow=0,pflowtmout=0,chain=94) [m
[32mMon Aug 19 11:19:01 2019.497344 DEBUG [0] NewKeyFlush [ External Hosts] in 00h 00m 00s 000012u (3)[m
[32mMon Aug 19 11:19:01 2019.497362 DEBUG [EDGETABLE] External Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.511196 DEBUG [0] CG-Glob-Flush [ External Hosts][86] in 00h 00m 00s 041931u (u=9734,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.512996 DEBUG [1] FlushBuckets [ ASNumber] in 00h 00m 00s 010850u (live=5,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.513067 DEBUG [1] BktPruneLive [ ASNumber] in 00h 00m 00s 000008u (master=10,prune=0,pflow=0,pflowtmout=0,chain=10) [m
[32mMon Aug 19 11:19:01 2019.513099 DEBUG [EDGETABLE] ASNumber Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.514019 DEBUG [0] FlushBuckets [ ASNumber] in 00h 00m 00s 000018u (live=5,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.514156 DEBUG [0] BktPruneLive [ ASNumber] in 00h 00m 00s 000008u (master=12,prune=0,pflow=0,pflowtmout=0,chain=12) [m
[32mMon Aug 19 11:19:01 2019.514184 DEBUG [EDGETABLE] ASNumber Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.515347 DEBUG [1] CG-Glob-Flush [ ASNumber][86] in 00h 00m 00s 013237u (u=1701,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.515439 DEBUG [1] FlushBuckets [ Perf-Stats] in 00h 00m 00s 000015u (live=2,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.515475 DEBUG [1] BktPruneLive [ Perf-Stats] in 00h 00m 00s 000003u (master=2,prune=0,pflow=0,pflowtmout=0,chain=2) [m
[32mMon Aug 19 11:19:01 2019.515493 DEBUG [EDGETABLE] Perf-Stats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.516959 DEBUG [0] CG-Glob-Flush [ ASNumber][86] in 00h 00m 00s 002985u (u=1263,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.517122 DEBUG [0] FlushBuckets [ Perf-Stats] in 00h 00m 00s 000037u (live=2,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.517180 DEBUG [0] BktPruneLive [ Perf-Stats] in 00h 00m 00s 000003u (master=2,prune=0,pflow=0,pflowtmout=0,chain=2) [m
[32mMon Aug 19 11:19:01 2019.517213 DEBUG [EDGETABLE] Perf-Stats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.518313 DEBUG [1] CG-Glob-Flush [ Perf-Stats][86] in 00h 00m 00s 002906u (u=2,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.520070 DEBUG [0] CG-Glob-Flush [ Perf-Stats][86] in 00h 00m 00s 003021u (u=183777,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.520601 DEBUG [1] CG-Glob-Flush [ VLANStats][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.520752 DEBUG [1] FlushBuckets [ HTTP URL Category] in 00h 00m 00s 000021u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.520794 DEBUG [1] BktPruneLive [ HTTP URL Category] in 00h 00m 00s 000002u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.520826 DEBUG [EDGETABLE] HTTP URL Category Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.522770 DEBUG [0] FlushBuckets [ VLANStats] in 00h 00m 00s 000010u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.522834 DEBUG [0] BktPruneLive [ VLANStats] in 00h 00m 00s 000003u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.522859 DEBUG [EDGETABLE] VLANStats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.524159 DEBUG [1] CG-Glob-Flush [ HTTP URL Category][86] in 00h 00m 00s 003446u (u=4,caches %=75/0/25)[m
[32mMon Aug 19 11:19:01 2019.526011 DEBUG [0] CG-Glob-Flush [ VLANStats][86] in 00h 00m 00s 003298u (u=34977,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.526109 DEBUG [0] FlushBuckets [ HTTP URL Category] in 00h 00m 00s 000002u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.526173 DEBUG [0] BktPruneLive [ HTTP URL Category] in 00h 00m 00s 000001u (master=0,prune=0,pflow=0,pflowtmout=0,chain=0) [m
[32mMon Aug 19 11:19:01 2019.526195 DEBUG [EDGETABLE] HTTP URL Category Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.527025 DEBUG [1] CG-Glob-Flush [ Flow-ASN][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.527278 DEBUG [1] FlushBuckets [ TLS CAs] in 00h 00m 00s 000016u (live=4,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.527323 DEBUG [1] BktPruneLive [ TLS CAs] in 00h 00m 00s 000005u (master=8,prune=0,pflow=0,pflowtmout=0,chain=8) [m
[32mMon Aug 19 11:19:01 2019.527341 DEBUG [EDGETABLE] TLS CAs Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.528041 DEBUG [0] CG-Glob-Flush [ HTTP URL Category][86] in 00h 00m 00s 001962u[m
[32mMon Aug 19 11:19:01 2019.528088 DEBUG [0] CG-Glob-Flush [ Flow-ASN][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.528146 DEBUG [0] FlushBuckets [ TLS CAs] in 00h 00m 00s 000012u (live=2,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.528183 DEBUG [0] BktPruneLive [ TLS CAs] in 00h 00m 00s 000003u (master=4,prune=0,pflow=0,pflowtmout=0,chain=4) [m
[32mMon Aug 19 11:19:01 2019.528200 DEBUG [EDGETABLE] TLS CAs Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.529080 DEBUG [1] CG-Glob-Flush [ TLS CAs][86] in 00h 00m 00s 001832u (u=18,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.529127 DEBUG [1] CG-Glob-Flush [ Alert Classes][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.529156 DEBUG [1] CG-Glob-Flush [ FlowGens][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.529206 DEBUG [1] FlushBuckets [ HTTP Status Codes] in 00h 00m 00s 000007u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.529260 DEBUG [1] BktPruneLive [ HTTP Status Codes] in 00h 00m 00s 000002u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.529296 DEBUG [EDGETABLE] HTTP Status Codes Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.530254 DEBUG [0] CG-Glob-Flush [ TLS CAs][86] in 00h 00m 00s 002138u (u=5,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.530306 DEBUG [0] CG-Glob-Flush [ Alert Classes][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.530333 DEBUG [0] CG-Glob-Flush [ FlowGens][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.530367 DEBUG [0] CG-Glob-Flush [ HTTP Status Codes][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.530397 DEBUG [0] CG-Glob-Flush [ Long Fat Tail Hosts][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.530480 DEBUG [0] FlushBuckets [ Prefix] in 00h 00m 00s 000036u (live=13,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.530557 DEBUG [0] BktPruneLive [ Prefix] in 00h 00m 00s 000021u (master=35,prune=0,pflow=0,pflowtmout=0,chain=35) [m
[32mMon Aug 19 11:19:01 2019.530589 DEBUG [EDGETABLE] Prefix Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.533022 DEBUG [1] CG-Glob-Flush [ HTTP Status Codes][86] in 00h 00m 00s 003825u (u=1,caches %=0/0/100)[m
[32mMon Aug 19 11:19:01 2019.533222 DEBUG [1] CG-Glob-Flush [ Long Fat Tail Hosts][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.533344 DEBUG [1] FlushBuckets [ Prefix] in 00h 00m 00s 000027u (live=9,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.533402 DEBUG [1] BktPruneLive [ Prefix] in 00h 00m 00s 000010u (master=23,prune=0,pflow=0,pflowtmout=0,chain=23) [m
[32mMon Aug 19 11:19:01 2019.533424 DEBUG [EDGETABLE] Prefix Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.534385 DEBUG [0] CG-Glob-Flush [ Prefix][86] in 00h 00m 00s 003951u (u=2939,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.534482 DEBUG [0] FlushBuckets [ HTTP Methods] in 00h 00m 00s 000014u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.534521 DEBUG [0] BktPruneLive [ HTTP Methods] in 00h 00m 00s 000001u (master=0,prune=0,pflow=0,pflowtmout=0,chain=0) [m
[32mMon Aug 19 11:19:01 2019.534544 DEBUG [EDGETABLE] HTTP Methods Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.535034 DEBUG [1] CG-Glob-Flush [ Prefix][86] in 00h 00m 00s 001761u (u=25,caches %=88/0/12)[m
[32mMon Aug 19 11:19:01 2019.535166 DEBUG [1] FlushBuckets [ HTTP Methods] in 00h 00m 00s 000021u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.535214 DEBUG [1] BktPruneLive [ HTTP Methods] in 00h 00m 00s 000008u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.535235 DEBUG [EDGETABLE] HTTP Methods Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.538106 DEBUG [0] CG-Glob-Flush [ HTTP Methods][86] in 00h 00m 00s 003635u[m
[32mMon Aug 19 11:19:01 2019.538262 DEBUG [0] CG-Glob-Flush [Flow-Prefix-External][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.538325 DEBUG [0] CG-Glob-Flush [ Flow-APPID-NBAR][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.538521 DEBUG [0] FlushBuckets [ Aggregates] in 00h 00m 00s 000120u (live=3,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.538636 DEBUG [0] BktPruneLive [ Aggregates] in 00h 00m 00s 000005u (master=3,prune=0,pflow=0,pflowtmout=0,chain=3) [m
[32mMon Aug 19 11:19:01 2019.538666 DEBUG [EDGETABLE] Aggregates Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.539285 DEBUG [1] CG-Glob-Flush [ HTTP Methods][86] in 00h 00m 00s 004154u (u=1,caches %=0/0/100)[m
[32mMon Aug 19 11:19:01 2019.539577 DEBUG [1] CG-Glob-Flush [Flow-Prefix-External][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.539892 DEBUG [1] CG-Glob-Flush [ Flow-APPID-NBAR][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.540412 DEBUG [0] CG-Glob-Flush [ Aggregates][86] in 00h 00m 00s 002040u (u=40769,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.540460 DEBUG [1] FlushBuckets [ Aggregates] in 00h 00m 00s 000020u (live=3,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.540549 DEBUG [1] BktPruneLive [ Aggregates] in 00h 00m 00s 000008u (master=3,prune=0,pflow=0,pflowtmout=0,chain=3) [m
[32mMon Aug 19 11:19:01 2019.540576 DEBUG [EDGETABLE] Aggregates Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.540471 DEBUG [0] CG-Glob-Flush [ BGP-Origin AS][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.543663 DEBUG [1] CG-Glob-Flush [ Aggregates][86] in 00h 00m 00s 003276u (u=42676,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.544117 DEBUG [0] CG-Glob-Flush [ Organization][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.544171 DEBUG [0] CG-Glob-Flush [ DNSPorts][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.544244 DEBUG [0] FlushBuckets [ TLS Orgs] in 00h 00m 00s 000023u (live=6,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.544287 DEBUG [0] BktPruneLive [ TLS Orgs] in 00h 00m 00s 000007u (master=12,prune=0,pflow=0,pflowtmout=0,chain=12) [m
[32mMon Aug 19 11:19:01 2019.544307 DEBUG [EDGETABLE] TLS Orgs Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.545603 DEBUG [0] CG-Glob-Flush [ TLS Orgs][86] in 00h 00m 00s 001398u (u=2735,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.545657 DEBUG [0] CG-Glob-Flush [ BGP-ASPATH][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.545784 DEBUG [0] FlushBuckets [ Mac] in 00h 00m 00s 000077u (live=40,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.545862 DEBUG [0] BktPruneLive [ Mac] in 00h 00m 00s 000024u (master=44,prune=0,pflow=0,pflowtmout=0,chain=44) [m
[32mMon Aug 19 11:19:01 2019.545886 DEBUG [EDGETABLE] Mac Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.547008 DEBUG [1] CG-Glob-Flush [ BGP-Origin AS][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.547065 DEBUG [1] CG-Glob-Flush [ Organization][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.548257 DEBUG [1] CG-Glob-Flush [ DNSPorts][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.548719 DEBUG [0] CG-Glob-Flush [ Mac][86] in 00h 00m 00s 003028u (u=88254,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.551170 DEBUG [1] FlushBuckets [ TLS Orgs] in 00h 00m 00s 000043u (live=13,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.551254 DEBUG [1] BktPruneLive [ TLS Orgs] in 00h 00m 00s 000013u (master=17,prune=0,pflow=0,pflowtmout=0,chain=17) [m
[32mMon Aug 19 11:19:01 2019.551281 DEBUG [EDGETABLE] TLS Orgs Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.597489 DEBUG [1] CG-Glob-Flush [ TLS Orgs][86] in 00h 00m 00s 034143u (u=627,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.613076 DEBUG [1] CG-Glob-Flush [ BGP-ASPATH][86] in 00h 00m 00s 000003u[m
[32mMon Aug 19 11:19:01 2019.614364 DEBUG [1] FlushBuckets [ Mac] in 00h 00m 00s 000107u (live=35,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.627127 DEBUG [1] BktPruneLive [ Mac] in 00h 00m 00s 000050u (master=38,prune=0,pflow=0,pflowtmout=0,chain=38) [m
[32mMon Aug 19 11:19:01 2019.627218 DEBUG [EDGETABLE] Mac Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.641845 DEBUG [0] FlushBuckets [ Hosts] in 00h 00m 00s 053043u (live=89,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.647633 DEBUG XSTDomains [0]Total=1254 cidr28=989 dups = 8[m
[32mMon Aug 19 11:19:01 2019.647712 DEBUG XSTDomains [0] attempts=48 cidr32=28 cidr28=0 miss=20[m
[32mMon Aug 19 11:19:01 2019.647732 DEBUG Feedback PostFlushBuckets : From Engine Instance 0[m
[32mMon Aug 19 11:19:01 2019.653393 DEBUG [EDGETABLE] BaseDomains Flushed 3 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.654065 DEBUG XSTDomains [0] PostGlobalFlush Tunnel Maint =1254 top level domains [m
[32mMon Aug 19 11:19:01 2019.655775 DEBUG [0] BktPruneLive [ Hosts] in 00h 00m 00s 000189u (master=126,prune=0,pflow=0,pflowtmout=0,chain=126) [m
[32mMon Aug 19 11:19:01 2019.656234 DEBUG [0] KeyUpdates [ Hosts] in 00h 00m 00s 000268u (27)[m
[32mMon Aug 19 11:19:01 2019.660356 DEBUG [0] NewKeyFlush [ Hosts] in 00h 00m 00s 004055u (3)[m
[32mMon Aug 19 11:19:01 2019.660594 DEBUG [EDGETABLE] Hosts Flushed 3 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.661215 DEBUG [0] CG-Glob-Flush [ Hosts][86] in 00h 00m 00s 112418u (u=71723,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.665019 DEBUG [1] CG-Glob-Flush [ Mac][86] in 00h 00m 00s 050808u (u=13562,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.669815 DEBUG [0] FlushBuckets [ Meta Counter Group] in 00h 00m 00s 000106u (live=17,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.670775 DEBUG [0] BktPruneLive [ Meta Counter Group] in 00h 00m 00s 000014u (master=17,prune=0,pflow=0,pflowtmout=0,chain=17) [m
[32mMon Aug 19 11:19:01 2019.670846 DEBUG [EDGETABLE] Meta Counter Group Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.671572 DEBUG [0] CG-Glob-Flush [ Meta Counter Group][86] in 00h 00m 00s 004866u (u=32,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.691841 DEBUG [0] CG-Glob-Flush [ BGP-Prefix Home][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.696544 DEBUG [0] CG-Glob-Flush [ Meta Session Group][86] in 00h 00m 00s 000003u[m
[32mMon Aug 19 11:19:01 2019.699793 DEBUG [0] FlushBuckets [ TLS Ciphers] in 00h 00m 00s 000023u (live=3,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.713039 DEBUG [0] BktPruneLive [ TLS Ciphers] in 00h 00m 00s 000012u (master=6,prune=0,pflow=0,pflowtmout=0,chain=6) [m
[32mMon Aug 19 11:19:01 2019.715772 DEBUG [EDGETABLE] TLS Ciphers Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.725108 DEBUG [1] FlushBuckets [ Hosts] in 00h 00m 00s 037499u (live=94,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.729073 DEBUG XSTDomains [1]Total=1254 cidr28=989 dups = 8[m
[32mMon Aug 19 11:19:01 2019.729328 DEBUG XSTDomains [1] attempts=52 cidr32=23 cidr28=0 miss=29[m
[32mMon Aug 19 11:19:01 2019.732504 DEBUG Feedback PostFlushBuckets : From Engine Instance 1[m
[32mMon Aug 19 11:19:01 2019.733275 DEBUG [EDGETABLE] BaseDomains Flushed 3 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.734115 DEBUG XSTDomains [1] PostGlobalFlush Tunnel Maint =1254 top level domains [m
[32mMon Aug 19 11:19:01 2019.736641 DEBUG [0] CG-Glob-Flush [ TLS Ciphers][86] in 00h 00m 00s 036900u (u=996,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.738402 DEBUG [1] BktPruneLive [ Hosts] in 00h 00m 00s 000142u (master=146,prune=0,pflow=0,pflowtmout=0,chain=146) [m
[32mMon Aug 19 11:19:01 2019.740438 DEBUG [0] CG-Glob-Flush [ Blacklist][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.740563 DEBUG [0] CG-Glob-Flush [ Remote Office][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.740628 DEBUG [0] CG-Glob-Flush [ MPLSStats][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.740666 DEBUG [0] CG-Glob-Flush [ City][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.740880 DEBUG [0] FlushBuckets [ Base Domains] in 00h 00m 00s 000136u (live=13,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.740968 DEBUG [0] BktPruneLive [ Base Domains] in 00h 00m 00s 000013u (master=27,prune=0,pflow=0,pflowtmout=0,chain=27) [m
[32mMon Aug 19 11:19:01 2019.743020 DEBUG [1] NewKeyFlush [ Hosts] in 00h 00m 00s 004546u (1)[m
[32mMon Aug 19 11:19:01 2019.743085 DEBUG [EDGETABLE] Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.743670 DEBUG [1] CG-Glob-Flush [ Hosts][86] in 00h 00m 00s 074077u (u=55224,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.745296 DEBUG [0] NewKeyFlush [ Base Domains] in 00h 00m 00s 004290u (1)[m
[32mMon Aug 19 11:19:01 2019.745356 DEBUG [EDGETABLE] Base Domains Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.745837 DEBUG [0] CG-Glob-Flush [ Base Domains][86] in 00h 00m 00s 005136u (u=19,caches %=94/0/5)[m
[32mMon Aug 19 11:19:01 2019.746988 DEBUG [1] FlushBuckets [ Meta Counter Group] in 00h 00m 00s 000038u (live=10,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.747045 DEBUG [1] BktPruneLive [ Meta Counter Group] in 00h 00m 00s 000005u (master=10,prune=0,pflow=0,pflowtmout=0,chain=10) [m
[32mMon Aug 19 11:19:01 2019.747084 DEBUG [EDGETABLE] Meta Counter Group Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.761450 DEBUG [1] CG-Glob-Flush [ Meta Counter Group][86] in 00h 00m 00s 012798u (u=20,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.764628 DEBUG [0] CG-Glob-Flush [ Flow-IP-NextHop][86] in 00h 00m 00s 000004u[m
[32mMon Aug 19 11:19:01 2019.764629 DEBUG [1] CG-Glob-Flush [ BGP-Prefix Home][86] in 00h 00m 00s 000003u[m
[32mMon Aug 19 11:19:01 2019.790110 DEBUG [1] FlushBuckets [ Meta Session Group] in 00h 00m 00s 007004u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.790310 DEBUG [1] BktPruneLive [ Meta Session Group] in 00h 00m 00s 000004u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.792426 DEBUG [0] FlushBuckets [ Dir Mac] in 00h 00m 00s 009390u (live=72,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.793459 DEBUG [EDGETABLE] Meta Session Group Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.795307 DEBUG [0] BktPruneLive [ Dir Mac] in 00h 00m 00s 000162u (master=95,prune=0,pflow=0,pflowtmout=0,chain=95) [m
[32mMon Aug 19 11:19:01 2019.795400 DEBUG [EDGETABLE] Dir Mac Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.806343 DEBUG [0] CG-Glob-Flush [ Dir Mac][86] in 00h 00m 00s 031494u (u=50957,caches %=99/0/0)[m
[32mMon Aug 19 11:19:01 2019.809338 DEBUG [1] CG-Glob-Flush [ Meta Session Group][86] in 00h 00m 00s 044555u (u=2,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.813726 DEBUG [0] CG-Glob-Flush [ Flow-VRF][86] in 00h 00m 00s 000004u[m
[32mMon Aug 19 11:19:01 2019.816652 DEBUG [0] FlushBuckets [ Internal Hosts] in 00h 00m 00s 000207u (live=32,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.816859 DEBUG [0] BktPruneLive [ Internal Hosts] in 00h 00m 00s 000055u (master=32,prune=0,pflow=0,pflowtmout=0,chain=32) [m
[32mMon Aug 19 11:19:01 2019.816921 DEBUG [EDGETABLE] Internal Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.821994 DEBUG [1] FlushBuckets [ TLS Ciphers] in 00h 00m 00s 001480u (live=4,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.822151 DEBUG [1] BktPruneLive [ TLS Ciphers] in 00h 00m 00s 000011u (master=6,prune=0,pflow=0,pflowtmout=0,chain=6) [m
[32mMon Aug 19 11:19:01 2019.822195 DEBUG [EDGETABLE] TLS Ciphers Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.823211 DEBUG [0] CG-Glob-Flush [ Internal Hosts][86] in 00h 00m 00s 006862u (u=61951,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.876307 DEBUG [1] CG-Glob-Flush [ TLS Ciphers][86] in 00h 00m 00s 056053u (u=2780,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.885583 DEBUG [0] FlushBuckets [ HostsIPv6] in 00h 00m 00s 005150u (live=3,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.899448 DEBUG [1] CG-Glob-Flush [ Blacklist][86] in 00h 00m 00s 000006u[m
[32mMon Aug 19 11:19:01 2019.899616 DEBUG [0] BktPruneLive [ HostsIPv6] in 00h 00m 00s 000013u (master=3,prune=0,pflow=0,pflowtmout=0,chain=3) [m
[32mMon Aug 19 11:19:01 2019.902881 DEBUG [EDGETABLE] HostsIPv6 Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.916750 DEBUG [1] CG-Glob-Flush [ Remote Office][86] in 00h 00m 00s 000006u[m
[32mMon Aug 19 11:19:01 2019.922924 DEBUG [1] CG-Glob-Flush [ MPLSStats][86] in 00h 00m 00s 000004u[m
[32mMon Aug 19 11:19:01 2019.923014 DEBUG [1] CG-Glob-Flush [ City][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.924801 DEBUG [0] CG-Glob-Flush [ HostsIPv6][86] in 00h 00m 00s 091684u (u=28,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.929740 DEBUG [1] FlushBuckets [ Base Domains] in 00h 00m 00s 002748u (live=15,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.931415 DEBUG [0] CG-Glob-Flush [ Alert Priorities][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.931549 DEBUG [0] FlushBuckets [ LinkLayerStats] in 00h 00m 00s 000026u (live=5,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.932562 DEBUG [0] BktPruneLive [ LinkLayerStats] in 00h 00m 00s 000010u (master=6,prune=0,pflow=0,pflowtmout=0,chain=6) [m
[32mMon Aug 19 11:19:01 2019.932626 DEBUG [EDGETABLE] LinkLayerStats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.932655 DEBUG [1] BktPruneLive [ Base Domains] in 00h 00m 00s 000024u (master=28,prune=0,pflow=0,pflowtmout=0,chain=28) [m
[32mMon Aug 19 11:19:01 2019.932715 DEBUG [EDGETABLE] Base Domains Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.959364 DEBUG [1] CG-Glob-Flush [ Base Domains][86] in 00h 00m 00s 036295u (u=90,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.960298 DEBUG [0] CG-Glob-Flush [ LinkLayerStats][86] in 00h 00m 00s 028800u (u=1480,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.964821 DEBUG [0] CG-Glob-Flush [ Alert Signatures][86] in 00h 00m 00s 000004u[m
[32mMon Aug 19 11:19:01 2019.964886 DEBUG [0] CG-Glob-Flush [ BGP-Peer AS][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.965013 DEBUG [0] FlushBuckets [ DNSServers] in 00h 00m 00s 000035u (live=5,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.965088 DEBUG [0] BktPruneLive [ DNSServers] in 00h 00m 00s 000010u (master=11,prune=0,pflow=0,pflowtmout=0,chain=11) [m
[32mMon Aug 19 11:19:01 2019.965112 DEBUG [EDGETABLE] DNSServers Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.965553 DEBUG [0] CG-Glob-Flush [ DNSServers][86] in 00h 00m 00s 000639u (u=89,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.965623 DEBUG [0] FlushBuckets [Long Thin Tail Hosts] in 00h 00m 00s 000007u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.965664 DEBUG [0] BktPruneLive [Long Thin Tail Hosts] in 00h 00m 00s 000002u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.967673 DEBUG [1] CG-Glob-Flush [ Flow-IP-NextHop][86] in 00h 00m 00s 000003u[m
[32mMon Aug 19 11:19:01 2019.969206 DEBUG [1] FlushBuckets [ Dir Mac] in 00h 00m 00s 001395u (live=71,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.969413 DEBUG [1] BktPruneLive [ Dir Mac] in 00h 00m 00s 000083u (master=92,prune=0,pflow=0,pflowtmout=0,chain=92) [m
[32mMon Aug 19 11:19:01 2019.969447 DEBUG [EDGETABLE] Dir Mac Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.969945 DEBUG [1] CG-Glob-Flush [ Dir Mac][86] in 00h 00m 00s 002197u (u=38731,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.970004 DEBUG [1] CG-Glob-Flush [ Flow-VRF][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.970170 DEBUG [1] FlushBuckets [ Internal Hosts] in 00h 00m 00s 000121u (live=35,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.970233 DEBUG [1] BktPruneLive [ Internal Hosts] in 00h 00m 00s 000019u (master=35,prune=0,pflow=0,pflowtmout=0,chain=35) [m
[32mMon Aug 19 11:19:01 2019.970252 DEBUG [EDGETABLE] Internal Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.970640 DEBUG [1] CG-Glob-Flush [ Internal Hosts][86] in 00h 00m 00s 000607u (u=51379,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.970732 DEBUG [1] FlushBuckets [ HostsIPv6] in 00h 00m 00s 000022u (live=6,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.970770 DEBUG [1] BktPruneLive [ HostsIPv6] in 00h 00m 00s 000004u (master=7,prune=0,pflow=0,pflowtmout=0,chain=7) [m
[32mMon Aug 19 11:19:01 2019.970786 DEBUG [EDGETABLE] HostsIPv6 Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.975145 DEBUG [1] CG-Glob-Flush [ HostsIPv6][86] in 00h 00m 00s 004443u (u=56,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.975236 DEBUG [1] CG-Glob-Flush [ Alert Priorities][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.975344 DEBUG [1] FlushBuckets [ LinkLayerStats] in 00h 00m 00s 000039u (live=7,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.975410 DEBUG [1] BktPruneLive [ LinkLayerStats] in 00h 00m 00s 000006u (master=8,prune=0,pflow=0,pflowtmout=0,chain=8) [m
[32mMon Aug 19 11:19:01 2019.975442 DEBUG [EDGETABLE] LinkLayerStats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.978649 DEBUG [1] CG-Glob-Flush [ LinkLayerStats][86] in 00h 00m 00s 003370u (u=34954,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.978722 DEBUG [1] CG-Glob-Flush [ Alert Signatures][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.978743 DEBUG [1] CG-Glob-Flush [ BGP-Peer AS][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:01 2019.978823 DEBUG [1] FlushBuckets [ DNSServers] in 00h 00m 00s 000042u (live=6,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.978873 DEBUG [1] BktPruneLive [ DNSServers] in 00h 00m 00s 000010u (master=17,prune=0,pflow=0,pflowtmout=0,chain=17) [m
[32mMon Aug 19 11:19:01 2019.978892 DEBUG [EDGETABLE] DNSServers Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.981563 DEBUG [1] CG-Glob-Flush [ DNSServers][86] in 00h 00m 00s 002789u (u=35,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.988036 DEBUG [0] NewKeyFlush [Long Thin Tail Hosts] in 00h 00m 00s 022237u (1)[m
[32mMon Aug 19 11:19:01 2019.988207 DEBUG [EDGETABLE] Long Thin Tail Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.989039 DEBUG [0] CG-Glob-Flush [Long Thin Tail Hosts][86] in 00h 00m 00s 023424u (u=4,caches %=75/0/25)[m
[32mMon Aug 19 11:19:01 2019.990738 DEBUG [0] CG-Glob-Flush [ FlowIntfs][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:01 2019.990920 DEBUG [0] FlushBuckets [ HTTP Content Types] in 00h 00m 00s 000031u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.991001 DEBUG [0] BktPruneLive [ HTTP Content Types] in 00h 00m 00s 000003u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.991033 DEBUG [EDGETABLE] HTTP Content Types Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.991589 DEBUG [0] CG-Glob-Flush [ HTTP Content Types][86] in 00h 00m 00s 000744u (u=2,caches %=50/0/50)[m
[32mMon Aug 19 11:19:01 2019.991647 DEBUG [0] CG-Glob-Flush [ BGP-Prefix External][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.991844 DEBUG [0] FlushBuckets [ Apps] in 00h 00m 00s 000085u (live=29,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.991964 DEBUG [0] BktPruneLive [ Apps] in 00h 00m 00s 000058u (master=38,prune=0,pflow=0,pflowtmout=0,chain=38) [m
[32mMon Aug 19 11:19:01 2019.991999 DEBUG [EDGETABLE] Apps Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.992387 DEBUG [0] CG-Glob-Flush [ Apps][86] in 00h 00m 00s 000639u (u=1062,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.992433 DEBUG [0] CG-Glob-Flush [ Flow-Prefix-Home][86] in 00h 00m 00s 000000u[m
[32mMon Aug 19 11:19:01 2019.992485 DEBUG [0] FlushBuckets [ HTTP Hosts] in 00h 00m 00s 000013u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.992585 DEBUG [0] BktPruneLive [ HTTP Hosts] in 00h 00m 00s 000002u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.992617 DEBUG [0] NewKeyFlush [ HTTP Hosts] in 00h 00m 00s 000010u (1)[m
[32mMon Aug 19 11:19:01 2019.992633 DEBUG [EDGETABLE] HTTP Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.993052 DEBUG [0] CG-Glob-Flush [ HTTP Hosts][86] in 00h 00m 00s 000595u (u=4,caches %=75/0/25)[m
[32mMon Aug 19 11:19:01 2019.993114 DEBUG [0] FlushBuckets [ NetworkLayerStats] in 00h 00m 00s 000008u (live=1,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.993149 DEBUG [0] BktPruneLive [ NetworkLayerStats] in 00h 00m 00s 000002u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.993167 DEBUG [EDGETABLE] NetworkLayerStats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.993532 DEBUG [0] CG-Glob-Flush [ NetworkLayerStats][86] in 00h 00m 00s 000432u (u=34336,caches %=100/0/0)[m
[32mMon Aug 19 11:19:01 2019.993585 DEBUG [0] FlushBuckets [ ICMP Types] in 00h 00m 00s 000002u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.994084 DEBUG [1] FlushBuckets [Long Thin Tail Hosts] in 00h 00m 00s 000002u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:01 2019.995797 DEBUG [1] BktPruneLive [Long Thin Tail Hosts] in 00h 00m 00s 000004u (master=0,prune=0,pflow=0,pflowtmout=0,chain=0) [m
[32mMon Aug 19 11:19:01 2019.995881 DEBUG [EDGETABLE] Long Thin Tail Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.996839 DEBUG [0] BktPruneLive [ ICMP Types] in 00h 00m 00s 000006u (master=1,prune=0,pflow=0,pflowtmout=0,chain=1) [m
[32mMon Aug 19 11:19:01 2019.996912 DEBUG [EDGETABLE] ICMP Types Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:01 2019.997756 DEBUG [0] CG-Glob-Flush [ ICMP Types][86] in 00h 00m 00s 004171u[m
[32mMon Aug 19 11:19:01 2019.998180 DEBUG [1] CG-Glob-Flush [Long Thin Tail Hosts][86] in 00h 00m 00s 004123u[m
[32mMon Aug 19 11:19:02 2019.005300 DEBUG [1] CG-Glob-Flush [ FlowIntfs][86] in 00h 00m 00s 000003u[m
[32mMon Aug 19 11:19:02 2019.005428 DEBUG [1] FlushBuckets [ HTTP Content Types] in 00h 00m 00s 000002u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:02 2019.005470 DEBUG [1] BktPruneLive [ HTTP Content Types] in 00h 00m 00s 000002u (master=0,prune=0,pflow=0,pflowtmout=0,chain=0) [m
[32mMon Aug 19 11:19:02 2019.005494 DEBUG [EDGETABLE] HTTP Content Types Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.008538 DEBUG [1] CG-Glob-Flush [ HTTP Content Types][86] in 00h 00m 00s 003138u[m
[32mMon Aug 19 11:19:02 2019.008617 DEBUG [1] CG-Glob-Flush [ BGP-Prefix External][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:02 2019.009968 DEBUG [1] FlushBuckets [ Apps] in 00h 00m 00s 000120u (live=23,evictee=0) [m
[32mMon Aug 19 11:19:02 2019.010189 DEBUG [1] BktPruneLive [ Apps] in 00h 00m 00s 000030u (master=30,prune=0,pflow=0,pflowtmout=0,chain=30) [m
[32mMon Aug 19 11:19:02 2019.010227 DEBUG [EDGETABLE] Apps Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.010871 DEBUG [0] CG-Glob-Flush [ Flow-BGP-NextHop][86] in 00h 00m 00s 000002u[m
[32mMon Aug 19 11:19:02 2019.011449 DEBUG [1] CG-Glob-Flush [ Apps][86] in 00h 00m 00s 001656u (u=48428,caches %=100/0/0)[m
[32mMon Aug 19 11:19:02 2019.011500 DEBUG [1] CG-Glob-Flush [ Flow-Prefix-Home][86] in 00h 00m 00s 000001u[m
[32mMon Aug 19 11:19:02 2019.011544 DEBUG [1] FlushBuckets [ HTTP Hosts] in 00h 00m 00s 000002u (live=0,evictee=0) [m
[32mMon Aug 19 11:19:02 2019.011650 DEBUG [1] BktPruneLive [ HTTP Hosts] in 00h 00m 00s 000002u (master=0,prune=0,pflow=0,pflowtmout=0,chain=0) [m
[32mMon Aug 19 11:19:02 2019.011682 DEBUG [EDGETABLE] HTTP Hosts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.014724 DEBUG [1] CG-Glob-Flush [ HTTP Hosts][86] in 00h 00m 00s 003186u[m
[32mMon Aug 19 11:19:02 2019.014951 DEBUG [1] FlushBuckets [ NetworkLayerStats] in 00h 00m 00s 000027u (live=4,evictee=0) [m
[32mMon Aug 19 11:19:02 2019.015036 DEBUG [1] BktPruneLive [ NetworkLayerStats] in 00h 00m 00s 000006u (master=5,prune=0,pflow=0,pflowtmout=0,chain=5) [m
[32mMon Aug 19 11:19:02 2019.015070 DEBUG [EDGETABLE] NetworkLayerStats Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.016022 DEBUG [1] CG-Glob-Flush [ NetworkLayerStats][86] in 00h 00m 00s 001127u (u=9829,caches %=100/0/0)[m
[32mMon Aug 19 11:19:02 2019.016159 DEBUG [1] FlushBuckets [ ICMP Types] in 00h 00m 00s 000040u (live=4,evictee=0) [m
[32mMon Aug 19 11:19:02 2019.016225 DEBUG [1] BktPruneLive [ ICMP Types] in 00h 00m 00s 000004u (master=4,prune=0,pflow=0,pflowtmout=0,chain=4) [m
[32mMon Aug 19 11:19:02 2019.016252 DEBUG [EDGETABLE] ICMP Types Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.017045 DEBUG [1] CG-Glob-Flush [ ICMP Types][86] in 00h 00m 00s 000936u (u=245,caches %=100/0/0)[m
[34mMon Aug 19 11:19:02 2019.017796 INFO Establishing ZMQ channel for Flow Flushes [0] at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.022567 DEBUG Sent FLUSH_FLOW_BLOCK to flushd [0] [m
[34mMon Aug 19 11:19:02 2019.023277 INFO Flushing Tracker 1 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.023591 DEBUG Flush Toppers: Session Tracker Flushed : Traffic flows = 100[m
[32mMon Aug 19 11:19:02 2019.023637 DEBUG SessTracker FZMQ[Traffic] in 00h 00m 00s 000353u[m
[34mMon Aug 19 11:19:02 2019.023658 INFO Flushing Tracker 2 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.023895 DEBUG Flush Toppers: Session Tracker Flushed : Long lived flows = 100[m
[32mMon Aug 19 11:19:02 2019.023940 DEBUG SessTracker FZMQ[Long lived] in 00h 00m 00s 000275u[m
[34mMon Aug 19 11:19:02 2019.023960 INFO Flushing Tracker 3 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.024174 DEBUG Flush Toppers: Session Tracker Flushed : Transferred out flows = 100[m
[32mMon Aug 19 11:19:02 2019.024207 DEBUG SessTracker FZMQ[Transferred out] in 00h 00m 00s 000241u[m
[34mMon Aug 19 11:19:02 2019.024223 INFO Flushing Tracker 4 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.024519 DEBUG Flush Toppers: Session Tracker Flushed : Transferred in flows = 100[m
[32mMon Aug 19 11:19:02 2019.024564 DEBUG SessTracker FZMQ[Transferred in] in 00h 00m 00s 000330u[m
[34mMon Aug 19 11:19:02 2019.024589 INFO Flushing Tracker 9 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.024901 DEBUG Flush Toppers: Session Tracker Flushed : Payload Xfer out flows = 100[m
[32mMon Aug 19 11:19:02 2019.024987 DEBUG SessTracker FZMQ[Payload Xfer out] in 00h 00m 00s 000383u[m
[34mMon Aug 19 11:19:02 2019.025024 INFO Flushing Tracker 10 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.025385 DEBUG Flush Toppers: Session Tracker Flushed : Payload Xfer in flows = 100[m
[32mMon Aug 19 11:19:02 2019.025452 DEBUG SessTracker FZMQ[Payload Xfer in] in 00h 00m 00s 000421u[m
[32mMon Aug 19 11:19:02 2019.025497 DEBUG Session Group : Flushed 103 flows (term=31,timeout=33,active=39,pass=162)[m
[32mMon Aug 19 11:19:02 2019.030895 DEBUG [1] CG-Glob-Flush [ Flow-BGP-NextHop][86] in 00h 00m 00s 000024u[m
[34mMon Aug 19 11:19:02 2019.031035 INFO Establishing ZMQ channel for Flow Flushes [1] at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.032315 DEBUG Sent FLUSH_FLOW_BLOCK to flushd [1] [m
[34mMon Aug 19 11:19:02 2019.032897 INFO Flushing Tracker 1 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.033067 DEBUG Flush Toppers: Session Tracker Flushed : Traffic flows = 100[m
[32mMon Aug 19 11:19:02 2019.033086 DEBUG SessTracker FZMQ[Traffic] in 00h 00m 00s 000187u[m
[34mMon Aug 19 11:19:02 2019.033095 INFO Flushing Tracker 2 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.033330 DEBUG Flush Toppers: Session Tracker Flushed : Long lived flows = 100[m
[32mMon Aug 19 11:19:02 2019.033355 DEBUG SessTracker FZMQ[Long lived] in 00h 00m 00s 000256u[m
[34mMon Aug 19 11:19:02 2019.033364 INFO Flushing Tracker 3 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.033517 DEBUG Flush Toppers: Session Tracker Flushed : Transferred out flows = 100[m
[32mMon Aug 19 11:19:02 2019.033531 DEBUG SessTracker FZMQ[Transferred out] in 00h 00m 00s 000164u[m
[34mMon Aug 19 11:19:02 2019.033554 INFO Flushing Tracker 4 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.033684 DEBUG Flush Toppers: Session Tracker Flushed : Transferred in flows = 100[m
[32mMon Aug 19 11:19:02 2019.033700 DEBUG SessTracker FZMQ[Transferred in] in 00h 00m 00s 000143u[m
[34mMon Aug 19 11:19:02 2019.033712 INFO Flushing Tracker 9 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.033877 DEBUG Flush Toppers: Session Tracker Flushed : Payload Xfer out flows = 100[m
[32mMon Aug 19 11:19:02 2019.033898 DEBUG SessTracker FZMQ[Payload Xfer out] in 00h 00m 00s 000183u[m
[34mMon Aug 19 11:19:02 2019.033907 INFO Flushing Tracker 10 to ZMQ Flusher [m
[32mMon Aug 19 11:19:02 2019.034026 DEBUG Flush Toppers: Session Tracker Flushed : Payload Xfer in flows = 100[m
[32mMon Aug 19 11:19:02 2019.034049 DEBUG SessTracker FZMQ[Payload Xfer in] in 00h 00m 00s 000139u[m
[32mMon Aug 19 11:19:02 2019.034058 DEBUG Session Group : Flushed 126 flows (term=32,timeout=45,active=49,pass=177)[m
[32mMon Aug 19 11:19:02 2019.039105 DEBUG [EDGETABLE] XSessionGroup Flushed 6 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.039672 DEBUG [1] SG-Thr-Prune-Flush TCP Sessions in 00h 00m 00s 008644u (all=303,mru=303) [m
[32mMon Aug 19 11:19:02 2019.039760 DEBUG [EDGETABLE] XSessionGroup Flushed 6 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.040877 DEBUG [0] SG-Thr-Prune-Flush TCP Sessions in 00h 00m 00s 023126u (all=265,mru=265) [m
[32mMon Aug 19 11:19:02 2019.041878 DEBUG [EDGETABLE] Threshold Crossing Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.049639 DEBUG [EDGETABLE] Threshold Crossing Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.059077 DEBUG AG-Glob-Flush [ Threshold Crossing] in 00h 00m 00s 017257u (0) [m
[32mMon Aug 19 11:19:02 2019.059205 DEBUG [EDGETABLE] Threshold Band Alerts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.070863 DEBUG AG-Glob-Flush [ Threshold Crossing] in 00h 00m 00s 017471u (0) [m
[32mMon Aug 19 11:19:02 2019.070867 DEBUG AG-Glob-Flush [Threshold Band Alerts] in 00h 00m 00s 011492u (0) [m
[32mMon Aug 19 11:19:02 2019.071063 DEBUG [EDGETABLE] Threshold Band Alerts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.071257 DEBUG [EDGETABLE] Blacklist activity Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.071701 DEBUG AG-Glob-Flush [Threshold Band Alerts] in 00h 00m 00s 000701u (0) [m
[32mMon Aug 19 11:19:02 2019.071774 DEBUG [EDGETABLE] Blacklist activity Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.072081 DEBUG AG-Glob-Flush [ Blacklist activity] in 00h 00m 00s 001057u (0) [m
[32mMon Aug 19 11:19:02 2019.072167 DEBUG [EDGETABLE] External IDS Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.072479 DEBUG AG-Glob-Flush [ Blacklist activity] in 00h 00m 00s 000713u (0) [m
[32mMon Aug 19 11:19:02 2019.072598 DEBUG [EDGETABLE] External IDS Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.072629 DEBUG AG-Glob-Flush [ External IDS] in 00h 00m 00s 000497u (0) [m
[32mMon Aug 19 11:19:02 2019.075564 DEBUG AG-Glob-Flush [ External IDS] in 00h 00m 00s 002975u (0) [m
[32mMon Aug 19 11:19:02 2019.075677 DEBUG [EDGETABLE] User Alerts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.076237 DEBUG AG-Glob-Flush [ User Alerts] in 00h 00m 00s 000596u (0) [m
[32mMon Aug 19 11:19:02 2019.076305 DEBUG [EDGETABLE] Flow Tracker Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.076784 DEBUG AG-Glob-Flush [ Flow Tracker] in 00h 00m 00s 000495u (0) [m
[32mMon Aug 19 11:19:02 2019.076863 DEBUG [EDGETABLE] System Alerts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.077478 DEBUG AG-Glob-Flush [ System Alerts] in 00h 00m 00s 000619u (0) [m
[32mMon Aug 19 11:19:02 2019.080288 DEBUG RG-Glob-Flush [ HTTP URIs] in 00h 00m 00s 002705u (1) [m
[32mMon Aug 19 11:19:02 2019.089265 DEBUG RG-Glob-Flush [ SSL Certs] in 00h 00m 00s 008825u (10) [m
[32mMon Aug 19 11:19:02 2019.089430 DEBUG RG-Glob-Flush [ File Hashes] in 00h 00m 00s 000012u (0) [m
[32mMon Aug 19 11:19:02 2019.089944 DEBUG [EDGETABLE] User Alerts Flushed 4 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.091029 DEBUG AG-Glob-Flush [ User Alerts] in 00h 00m 00s 018339u (9) [m
[32mMon Aug 19 11:19:02 2019.091261 DEBUG [EDGETABLE] Flow Tracker Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.091953 DEBUG AG-Glob-Flush [ Flow Tracker] in 00h 00m 00s 000766u (0) [m
[32mMon Aug 19 11:19:02 2019.092078 DEBUG [EDGETABLE] System Alerts Flushed 0 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.092939 DEBUG AG-Glob-Flush [ System Alerts] in 00h 00m 00s 000892u (0) [m
[32mMon Aug 19 11:19:02 2019.093132 DEBUG RG-Glob-Flush [ HTTP URIs] in 00h 00m 00s 000048u (0) [m
[32mMon Aug 19 11:19:02 2019.094213 DEBUG RG-Glob-Flush [ SSL Certs] in 00h 00m 00s 001014u (12) [m
[32mMon Aug 19 11:19:02 2019.094370 DEBUG RG-Glob-Flush [ File Hashes] in 00h 00m 00s 000026u (0) [m
[32mMon Aug 19 11:19:02 2019.095506 DEBUG RG-Glob-Flush [ DNS Resources] in 00h 00m 00s 005993u (125) [m
[32mMon Aug 19 11:19:02 2019.103334 DEBUG RG-Glob-Flush [ DNS Resources] in 00h 00m 00s 008839u (118) [m
[32mMon Aug 19 11:19:02 2019.212790 DEBUG RG-Glob-Flush [ Intel harvested] in 00h 00m 00s 117169u (414) [m
[32mMon Aug 19 11:19:02 2019.212880 DEBUG FG-Glob-Flush [ DNS Records] in 00h 00m 00s 000016u (0) [m
[32mMon Aug 19 11:19:02 2019.231374 DEBUG RG-Glob-Flush [ Intel harvested] in 00h 00m 00s 127862u (314) [m
[32mMon Aug 19 11:19:02 2019.231485 DEBUG FG-Glob-Flush [ DNS Records] in 00h 00m 00s 000038u (0) [m
[32mMon Aug 19 11:19:02 2019.231529 DEBUG FG-Glob-Flush [ HTTP Headers] in 00h 00m 00s 000009u (0) [m
[32mMon Aug 19 11:19:02 2019.237165 DEBUG FG-Glob-Flush [ SSL Certs] in 00h 00m 00s 005606u (12) [m
[32mMon Aug 19 11:19:02 2019.237273 DEBUG Thread 0 processed 69 flush operations [m
[32mMon Aug 19 11:19:02 2019.239796 DEBUG [EDGETABLE] XTRFDB:0 Flushed 13 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.255538 DEBUG FG-Glob-Flush [ HTTP Headers] in 00h 00m 00s 042624u (2) [m
[32mMon Aug 19 11:19:02 2019.260627 DEBUG FG-Glob-Flush [ SSL Certs] in 00h 00m 00s 004998u (10) [m
[32mMon Aug 19 11:19:02 2019.260800 DEBUG Thread 0 processed 69 flush operations [m
[32mMon Aug 19 11:19:02 2019.263133 DEBUG [EDGETABLE] XTRFDB:1 Flushed 12 vertices at Mon Aug 19 11:19:00 2019-000000[m
[32mMon Aug 19 11:19:02 2019.294632 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/suricata-eve-unixsocket/suricata_eve_unixsocket.lua[m
[32mMon Aug 19 11:19:02 2019.294863 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/suricata-eve-unixsocket/suricata_eve_unixsocket.lua[m
[32mMon Aug 19 11:19:02 2019.296160 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/binaries.lua[m
[32mMon Aug 19 11:19:02 2019.296640 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/binaries.lua[m
[32mMon Aug 19 11:19:02 2019.297172 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/dns.lua[m
[32mMon Aug 19 11:19:02 2019.297276 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/dns.lua[m
[32mMon Aug 19 11:19:02 2019.300700 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/hosts.lua[m
[32mMon Aug 19 11:19:02 2019.300957 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/hosts.lua[m
[32mMon Aug 19 11:19:02 2019.301820 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/hostsv6.lua[m
[32mMon Aug 19 11:19:02 2019.301887 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/hostsv6.lua[m
[32mMon Aug 19 11:19:02 2019.303901 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/http.lua[m
[32mMon Aug 19 11:19:02 2019.304303 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/http.lua[m
[32mMon Aug 19 11:19:02 2019.304668 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/intel_resource.lua[m
[32mMon Aug 19 11:19:02 2019.304865 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/intel_resource.lua[m
[32mMon Aug 19 11:19:02 2019.305279 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/intel_tester.lua[m
[32mMon Aug 19 11:19:02 2019.305392 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/intel_tester.lua[m
[32mMon Aug 19 11:19:02 2019.306521 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/ja3.lua[m
[32mMon Aug 19 11:19:02 2019.306610 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/ja3.lua[m
[32mMon Aug 19 11:19:02 2019.309336 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/sni.lua[m
[32mMon Aug 19 11:19:02 2019.309486 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/sni.lua[m
[32mMon Aug 19 11:19:02 2019.310694 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/ssl.lua[m
[32mMon Aug 19 11:19:02 2019.315479 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/ssl.lua[m
[32mMon Aug 19 11:19:02 2019.315766 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/sslsha1.lua[m
[32mMon Aug 19 11:19:02 2019.319763 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/ioc-harvestor/sslsha1.lua[m
[32mMon Aug 19 11:19:02 2019.320216 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/firehol/firehol.lua[m
[32mMon Aug 19 11:19:02 2019.320416 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/firehol/firehol.lua[m
[32mMon Aug 19 11:19:02 2019.322836 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/alienvault-otx/check-leveldb.lua[m
[32mMon Aug 19 11:19:02 2019.322909 DEBUG LUAHOST: Unchanged no action /usr/local/var/lib/trisul-probe/domain0/probe0/context0/config/lua//github.com_trisulnsm_apps/alienvault-otx/check-leveldb.lua[m
[32mMon Aug 19 11:19:02 2019.327171 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//base-domain.lua[m
[32mMon Aug 19 11:19:02 2019.327255 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//base-domain.lua[m
[32mMon Aug 19 11:19:02 2019.328520 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//helpers/cgbounds_base.lua[m
[32mMon Aug 19 11:19:02 2019.328580 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//helpers/cgbounds_base.lua[m
[32mMon Aug 19 11:19:02 2019.330104 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-asn.lua[m
[32mMon Aug 19 11:19:02 2019.330201 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-asn.lua[m
[32mMon Aug 19 11:19:02 2019.330344 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-bgp-next-hop.lua[m
[32mMon Aug 19 11:19:02 2019.330425 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-bgp-next-hop.lua[m
[32mMon Aug 19 11:19:02 2019.330611 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-ip-next-hop.lua[m
[32mMon Aug 19 11:19:02 2019.330717 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-ip-next-hop.lua[m
[32mMon Aug 19 11:19:02 2019.331002 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-nbar.lua[m
[32mMon Aug 19 11:19:02 2019.331085 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-nbar.lua[m
[32mMon Aug 19 11:19:02 2019.331220 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-prefix-home.lua[m
[32mMon Aug 19 11:19:02 2019.331266 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-prefix-home.lua[m
[32mMon Aug 19 11:19:02 2019.331729 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-prefix.lua[m
[32mMon Aug 19 11:19:02 2019.331839 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-prefix.lua[m
[32mMon Aug 19 11:19:02 2019.331965 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-vrf.lua[m
[32mMon Aug 19 11:19:02 2019.332997 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//perf-stats.lua[m
[32mMon Aug 19 11:19:02 2019.333464 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-aspath.lua[m
[32mMon Aug 19 11:19:02 2019.333620 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//nf-vrf.lua[m
[32mMon Aug 19 11:19:02 2019.333687 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//perf-stats.lua[m
[32mMon Aug 19 11:19:02 2019.333701 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-originas.lua[m
[32mMon Aug 19 11:19:02 2019.333738 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-aspath.lua[m
[32mMon Aug 19 11:19:02 2019.333775 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-originas.lua[m
[32mMon Aug 19 11:19:02 2019.334222 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-peeras.lua[m
[32mMon Aug 19 11:19:02 2019.334545 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-prefix-external.lua[m
[32mMon Aug 19 11:19:02 2019.334900 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-prefix-internal.lua[m
[34mMon Aug 19 11:19:02 2019.334949 INFO [TSE:1]Engine [1] Loaded LUA Plugins for DB Engine [m
[32mMon Aug 19 11:19:02 2019.335557 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-peeras.lua[m
[32mMon Aug 19 11:19:02 2019.335642 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-prefix-external.lua[m
[32mMon Aug 19 11:19:02 2019.335710 DEBUG LUAHOST: Unchanged no action /usr/local/lib/trisul-probe/plugins/lua//geo-prefix-internal.lua[m
[34mMon Aug 19 11:19:02 2019.335776 INFO [TSE:0]Engine [0] Loaded LUA Plugins for DB Engine [m
[32mMon Aug 19 11:19:02 2019.338753 DEBUG NBQ Post Flush FBQ: blocks=0,ok=1351 Drop=0%[m
[32mMon Aug 19 11:19:02 2019.338824 DEBUG NBQ Post Flush FBQ: blocks=0,ok=1469 Drop=0%[m
[34mMon Aug 19 11:19:02 2019.372480 INFO [0] GLOBALFLUSH in 00h 00m 01s 480200u[m
[34mMon Aug 19 11:19:02 2019.371830 INFO [1] GLOBALFLUSH in 00h 00m 01s 489516u[m
[32mMon Aug 19 11:19:14 2019.439678 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:14 2019.439764 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:19 2019.548536 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:19 2019.599584 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:22 2019.348336 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:23 2019.356709 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:31 2019.255635 DEBUG Input Queue idle - dropping into long wait tmsg sz=165 waitsecs=1[m
[32mMon Aug 19 11:19:31 2019.554578 DEBUG Input Queue idle - dropping into long wait tmsg sz=1 waitsecs=1[m
[32mMon Aug 19 11:19:32 2019.189290 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:32 2019.434973 DEBUG Input Queue idle - dropping into long wait tmsg sz=91 waitsecs=1[m
[32mMon Aug 19 11:19:34 2019.198675 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:34 2019.198783 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:35 2019.578202 DEBUG Input Queue idle - dropping into long wait tmsg sz=46 waitsecs=1[m
[32mMon Aug 19 11:19:35 2019.728602 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:36 2019.064582 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:36 2019.972999 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:37 2019.488384 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:38 2019.658278 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:39 2019.179193 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:39 2019.760347 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m
[32mMon Aug 19 11:19:41 2019.684775 DEBUG Input Queue idle - dropping into long wait tmsg sz=0 waitsecs=1[m

admin

The quickest way to debug this is to just run a special build of Trisul inside of a debugger.

Here is how you can do it

Download this build of trisul probe with symbols : https://www.trisul.org/download/dbg.el7/trisul
open a terminal, use tmux or screen so you can detach later

stop the currently running probe

/usr/local/bin/trisulctl_probe
stop context default@probe0

Start the new trisul with symbols that you download from step 1 under gdb. Say you've kept it in your home directory /home/myusername
```
sudo gdb --args  /home/myusername/trisul -nodemon /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml -mode online_rxring

(gdb) run
```
5. Then wait for it to crash .. it should drop into a gdb prompt. Type bt and paste the output here.

Thanks,

Vivek

Keenenick

Thank you Vivek,

I have re-installed Trisul on Ubuntu 18.04 and am testing further. I will attempt the debugging method should I face another crash/halt.

admin

Keenenick

Great ! Since you moved to Ubuntu 18, if you want to use the debug build use the following instead of the el7 build.

trisul.org/download/dbg.bionic/trisul