extract text between two strings containing third string

Question

I'm trying to parse a mod security log file. I want to take an IP address and print out all the events related to that IP. I want to extract everything between A-- and Z-- containing the IP.

Example of the log:

--9eca103e-A--
[03/Nov/2020:05:34:31 +0000] X6Dr5woAAIMAALNHuSUAAAAD 13.92.27.130 2702 10.0.0.131 80
--9eca103e-B--
GET /.env HTTP/1.1
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.24.0

--9eca103e-F--
HTTP/1.1 403 Forbidden
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Last-Modified: Mon, 24 Sep 2018 20:04:57 GMT
Accept-Ranges: bytes
Content-Length: 496
Cache-Control: s-maxage=10
Content-Type: text/html

--9eca103e-H--
Message: Matched phrase "/.env" at REQUEST_FILENAME. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "140"] [id "930130"] [rev "1"] [msg "Restricted File Access Attempt"] [data "Matched Data: /.env found within REQUEST_FILENAME: /.env"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "7"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "57"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "73"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=5,RCE=0,PHPI=0,HTTP=0,SESS=0): Restricted File Access Attempt"] [tag "event-correlation"]
Action: Intercepted (phase 2)
Stopwatch: 1604381671400965 3720 (- - -)
Stopwatch2: 1604381671400965 3720; combined=1145, p1=348, p2=703, p3=0, p4=0, p5=93, sr=103, sw=1, l=0, gc=0
Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/3.0.2.
Server: Apache/2.4.18 (Ubuntu)
Engine-Mode: "ENABLED"

--9eca103e-Z--

--3b96fe48-A--
[03/Nov/2020:06:39:53 +0000] X6D7OQoAAIMAAM4EDycAAAAY 52.90.151.137 39046 10.0.0.131 80
--3b96fe48-B--
HEAD /masked_uri/?gsr=6brOO-NFkiMaAMa1 HTTP/1.1
X-Forwarded-Proto: https
X-Forwarded-Port: 443
User-Agent: Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber)
Cookie: =; path=
Accept: */*

--3b96fe48-F--
HTTP/1.1 403 Forbidden
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Last-Modified: Mon, 24 Sep 2018 20:04:57 GMT
Accept-Ranges: bytes
Content-Length: 496
Content-Type: text/html

--3b96fe48-H--
Message: Matched phrase "grabber" at REQUEST_HEADERS:User-Agent. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf"] [line "60"] [id "913100"] [rev "2"] [msg "Found User-Agent associated with security scanner"] [data "Matched Data: grabber found within REQUEST_HEADERS:User-Agent: mozilla/5.0 ecairn-grabber/1.0 (+http://ecairn.com/grabber)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-reputation-scanner"] [tag "OWASP_CRS/AUTOMATION/SECURITY_SCANNER"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "57"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "73"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): Found User-Agent associated with security scanner"] [tag "event-correlation"]
Action: Intercepted (phase 2)
Stopwatch: 1604385593815579 4549 (- - -)
Stopwatch2: 1604385593815579 4549; combined=2007, p1=375, p2=1419, p3=0, p4=0, p5=156, sr=129, sw=57, l=0, gc=0
Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/3.0.2.
Server: Apache/2.4.18 (Ubuntu)
Engine-Mode: "ENABLED"

--3b96fe48-Z--

So for this log, if I were to use sed or grep, I'd like to get the events for this IP 13.92.27.130 with an output like this:

--9eca103e-A--
[03/Nov/2020:05:34:31 +0000] X6Dr5woAAIMAALNHuSUAAAAD 13.92.27.130 2702 10.0.0.131 80
--9eca103e-B--
GET /.env HTTP/1.1
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.24.0

--9eca103e-F--
HTTP/1.1 403 Forbidden
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Last-Modified: Mon, 24 Sep 2018 20:04:57 GMT
Accept-Ranges: bytes
Content-Length: 496
Cache-Control: s-maxage=10
Content-Type: text/html

--9eca103e-H--
Message: Matched phrase "/.env" at REQUEST_FILENAME. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf"] [line "140"] [id "930130"] [rev "1"] [msg "Restricted File Access Attempt"] [data "Matched Data: /.env found within REQUEST_FILENAME: /.env"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "7"] [accuracy "8"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-lfi"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"]
Message: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "57"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf"] [line "73"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=5,RCE=0,PHPI=0,HTTP=0,SESS=0): Restricted File Access Attempt"] [tag "event-correlation"]
Action: Intercepted (phase 2)
Stopwatch: 1604381671400965 3720 (- - -)
Stopwatch2: 1604381671400965 3720; combined=1145, p1=348, p2=703, p3=0, p4=0, p5=93, sr=103, sw=1, l=0, gc=0
Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/3.0.2.
Server: Apache/2.4.18 (Ubuntu)
Engine-Mode: "ENABLED"

--9eca103e-Z--

I've started with sed -n -e '/A--/,/Z--/ p' /var/log/modsecurity/modsec_audit.log to get what's between A-- and Z--, but struggling to filter by the IP.

Please edit your question to add your attempts to solve the problem yourself. See How to Ask. — Ed Morton
– Ed Morton, Commented Nov 3, 2020 at 16:47

RavinderSingh13 · Accepted Answer · 2020-11-03 18:35:53Z

3

Could you please try following, written and tested with shown samples only.

awk '
/-Z--$/{
  if(start && value && found){
    print value ORS $0
  }
  start=found=value=""
}
/-A--$/{
  start=1
}
start && /^\[[0-9]{2}\/[a-zA-Z]{3}\/[0-9]{4}:([0-9]{2}:){2}[0-9]{2} /{
  if($0~/13\.92\.27\.130/){
     found=1
  }
}
{
  value=(value?value ORS:"")$0
}
' Input_file

Explanation: Adding detailed explanation for above.

awk '                                ##Starting awk program from here.
/-Z--$/{                             ##Checking condition if line ends with -Z-- then do following.
  if(start && value && found){       ##Checking if start, value and found is NOT NULL then do following.
    print value ORS $0               ##Printing value ORS and current line here.
  }
  start=found=value=""               ##Nullifying values here.
}
/-A--$/{                             ##Checking condition if line ends with -A-- then do following.
  start=1                            ##Setting start to 1 here.
}
start && /^\[[0-9]{2}\/[a-zA-Z]{3}\/[0-9]{4}:([0-9]{2}:){2}[0-9]{2} /{  ##Checking condition if start is NOT NULL and checking regex for month and date format here.
  if($0~/13\.92\.27\.130/){          ##Checking condition if line has specific ip then do following.
     found=1                         ##Setting found to 1 here.
  }
}
{
  value=(value?value ORS:"")$0       ##Creating value and keep on adding currne line value to it.
}
' Input_file                         ##Mentioning Input_file name here.

edited Nov 3, 2020 at 18:35

answered Nov 3, 2020 at 16:42

RavinderSingh13

135k14 gold badges61 silver badges100 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

user2966697 Over a year ago

Nice, thanks for the code. When I run it, I get everything starting from the timestamp, to and including the next A-- line. Works for my purposes, but if others are using this as an example, it does include text beyond the end string.

RavinderSingh13 Over a year ago

@user2966697, Could you please do let me know till where I need to print, I believe it worked fine as per shown samples, please do let me know on same.

user2966697 Over a year ago

So if I run your example, I get output like this: [03/Nov/2020:05:34:31 +0000] X6Dr5woAAIMAALNHuSUAAAAD 13.92.27.130 2702 10.0.0.131 80 --9eca103e-B-- GET /.env HTTP/1.1 X-Forwarded-Proto: https X-Forwarded-Port: 443 ... shortened for brevity ... --9eca103e-Z-- --3b96fe48-A--

RavinderSingh13 Over a year ago

@user2966697, sorry strings are too long in samples, is there any unique string from where to till where I could print this? Please do let me know.

user2966697 Over a year ago

-A-- newline to -Z-- newline

|

potong · Accepted Answer · 2020-11-03 19:52:37Z

2

This might work for you (GNU sed):

sed -n '/A--$/{:a;N;/Z--$/!ba;/52\.90\.151\.137/p}' file

Turn off implicit printing by using the -n option.

Gather up lines starting with a line ending A-- until another line ending Z--.

If the those lines contain the string 52.90.151.137 print them.

answered Nov 3, 2020 at 19:52

potong

59.3k6 gold badges55 silver badges92 bronze badges

Collectives™ on Stack Overflow

extract text between two strings containing third string

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related