I'm using Apache NiFi 2.4.0 and want to extend the default HTTP request logging (nifi-request.log) to include a custom attribute called logType with values like DATA_ACCESS, DATA_EXPORT, or INVALID_OBJECT based on the API endpoint or request parameters. The goal is to create a custom log file similar to nifi-request.log that captures these additional details for auditing and monitoring API interactions.
For Example,
When the endpoint GET /nifi-api/processors/{id} is called:
If the processor ID is invalid (does not exist or is incorrect), the log entry for this event should have: logType set to "invalid_object" Additional attributes relevant to this error scenario.
If the processor information is successfully retrieved, the event should be logged with: logType set to "DATA_ACCESS" Other relevant attributes for data access auditing.
I’ve explored logback.xml to add a custom appender but couldn’t figure out how to extract and categorize API requests dynamically.
I considered using a custom Jetty filter to inspect requests and log them with a logType, but I’m unsure how to integrate it with NiFi’s Jetty server and Logback.
I looked into the SiteToSiteBulletinReportingTask for bulletins, but it doesn’t cover HTTP request details or allow custom attributes like logType.
Questions:
How can I implement a custom Jetty filter in NiFi 2.4.0 to log API requests with a logType attribute based on the endpoint or parameters?
What’s the best way to configure logback.xml to capture these logs in a separate file similar to nifi-request.log?
Are there any NiFi OOB features (e.g., processors or reporting tasks) that can achieve this without custom code?
Please suggest if any better way we can do this.
