Pfthb

I am new to GROK and I have an ELK stack would like to ingest some CWS/Scansafe logs into this baby. I have not been able to get a good parsing output. I am using http://grokdebug.herokuapp.com/ and grokconstructor.appspot.com/do/match and have had a hell of a time trying to get the right parsing. The log is W3C but separated by tabs t. I built this pattern but I cannot get the multi-word fields (which have spaces) to parse correctly as well as the null fields (not every record has every output). Any help breaking this down or pointing me in the right direction would be greatly appreciated.

Here is the pattern:

%TIMESTAMP_ISO8601:cws_datatime %WORD:timezonet%IPV4:c_ipt%NUMBER:x_ss_company_idt(%IPV4:cs_X_Forwarded_For)?(t%NOTSPACE:cs_username)?(t%WORD:cs_method)?(t%WORD:cs_uri_scheme)?(t%NOTSPACE:cs_host)?(t%NUMBER:cs_uri_port)?(t%NOTSPACE:cs_uri_path)?(t%DATA:cs_uri_query)?(t%DATA:user_agent)?(t%DATA:cs_Content_Type)?(t%NUMBER:cs_bytes)?(t%NUMBER:sc_bytes)?(t%NUMBER:sc_status)?(t%DATA:sc_Content_Type)?(t%IPV4:s_ip)?(t%NOTSPACE:x_ss_category)?(t%WORD:x_ss_last_rule_name)?(t%WORD:x_ss_last_rule_action)?(t%WORD:x_ss_block_type)?(t%WORD:x_ss_block_value)?t%IPV4:x_ss_external_ip(t%NOTSPACE:x_ss_referer_host)?

Here are a few sample logs:

#Fields: datatime c-ip x-ss-company-id cs(X-Forwarded-For) cs-username cs-method cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs(User-Agent) cs(Content-Type) cs-bytes sc-bytes sc-status sc(Content-Type) s-ip x-ss-category x-ss-last-rule-name x-ss-last-rule-action x-ss-block-type x-ss-block-value x-ss-external-ip x-ss-referer-host
2018-11-07 15:40:28 GMT 10.11.77.96 2100000000 10.10.77.96 CONNECT https d33t3vvu2t2yu5.cloudfront.net 443 / Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; MS-RTC LM 8; InfoPath.3; ) - 952 4982 0 13.32.243.161 c:infr default allow 9.99.239.26 
2018-11-07 15:40:28 GMT 10.11.167.50 2100000000 WinNT://localbobby CONNECT https pbs.twimg.com 443 / Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko - 2034 873898 0 12.21.91.70 c:snet default allow 9.99.239.26 
2018-11-07 15:40:30 GMT 10.79.239.26 2100000000 9.99.239.26 CONNECT https assets.customer.com 443 / Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36 - 1006 8508 0 9.99.187.71 c:busi default allow 9.99.239.26 
2018-11-07 15:40:35 GMT 10.79.239.26 2100000000 9.99.239.26 CONNECT https cloud-ec-asn.amp.cisco.com 443 / - 2754 7316 0 52.70.59.121 c:comp default allow 9.99.239.26 
2018-11-07 15:40:36 GMT 10.79.239.26 2100000000 9.99.239.26 CONNECT https inetupload.indsci.com 443 / - 1589 7038 0 198.187.140.58 c:busi default allow 9.99.239.26 
2018-11-07 15:40:44 GMT 10.79.239.26 2100000000 9.99.239.26 CONNECT https api.bing.com 443 / Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko - 0 0 0 13.107.5.80 c:srch default allow 9.99.239.26 
2018-11-07 15:41:03 GMT 10.14.144.19 2100000000 WinNT://localjane CONNECT https www.bing.com 443 / Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko - 22552 35245 0 13.107.21.200 c:srch default allow 9.99.239.26 
2018-11-07 15:41:04 GMT 10.79.239.26 2100000000 9.99.239.26 CONNECT https cloud-ec-asn.amp.cisco.com 443 / - 13742 9326 0 52.70.47.45 c:comp default allow 9.99.239.26 
2018-11-07 15:41:04 GMT 10.79.239.26 2100000000 9.99.239.26 GET http ocsp.verisign.com 80 /STAJBgUrDgMCGgUABBTSqZMG5M8TA9rdzkbCnNwuMAd5VgQUz5mp6nsm9EvJjo%2FX8AUm7%2BPSp50CEF7OSr0FTbDuGs71fE0%2FRCw%3D Microsoft-CryptoAPI/6.1 - 0 1660 200 application/ocsp-response 23.4.187.27 c:csec default allow 9.99.239.26 
2018-11-07 15:40:57 GMT 10.79.239.26 2100000000 9.99.239.26 GET http ourtripguide.pro 80 /35/wp-content/uploads/2018/10/10-Things-You-Need-To-Do-After-Checking-Into-A-Hotel-Room.jpg Outlook-iOS/696.1208477.prod.iphone (2.103.0) - 0 0 0 185.133.39.50 unclassified webrep block webrep Reputation-Viruses 9.99.239.26