Re: [PATCH] Add log-format variable %HQ, to log HTTP query strings

Andrew-

This patch works well for my use case. Leaving the question mark in
mirrors the behavior of %q in httpd, fwiw.

Should it also print a hyphen in case the field is empty and the quotation
mode is not on? %HQ instead of %{+Q}HQ?

Phillip


On Fri, Jul 31, 2015 at 12:21 PM, Andrew Hayworth <
andrew.hayworth@getbraintree.com> wrote:

> Since this came up in another thread, it seems reasonable to add a
> patch that implements %HQ as a log-format variable to record the HTTP
> query string. Leaving the initial '?' is intentional, but I don't feel
> strongly one way or another.
>
> --
> - Andrew Hayworth
>
>
> From b87770d5e513fc923d0d94d2b1d0de00d88acb98 Mon Sep 17 00:00:00 2001
> From: Andrew Hayworth <andrew.hayworth@getbraintree.com>
> Date: Fri, 31 Jul 2015 16:14:16 +0000
> Subject: [PATCH 1/1] Add log-format variable %HQ, to log HTTP query strings
>
> Since sample fetches are not always available in the response phase,
> this patch implements %HQ such that:
>
> GET /foo?bar=baz HTTP/1.0
>
> ...would be logged as:
>
> ?bar=baz
> ---
> doc/configuration.txt | 1 +
> include/types/log.h | 1 +
> src/log.c | 38 ++++++++++++++++++++++++++++++++++++++
> 3 files changed, 40 insertions(+)
>
> diff --git a/doc/configuration.txt b/doc/configuration.txt
> index db97cc7..b3ba8a0 100644
> --- a/doc/configuration.txt
> +++ b/doc/configuration.txt
> @@ -13987,6 +13987,7 @@ Please refer to the table below for currently
> defined variables :
> | | %H | hostname | string
> |
> | H | %HM | HTTP method (ex: POST) | string
> |
> | H | %HP | HTTP request URI without query string (path) | string
> |
> + | H | %HQ | HTTP request URI query string (ex: ?bar=baz) | string
> |
> | H | %HU | HTTP request URI (ex: /foo?bar=baz) | string
> |
> | H | %HV | HTTP version (ex: HTTP/1.0) | string
> |
> | | %ID | unique-id | string
> |
> diff --git a/include/types/log.h b/include/types/log.h
> index bbfe020..d0fb966 100644
> --- a/include/types/log.h
> +++ b/include/types/log.h
> @@ -96,6 +96,7 @@ enum {
> LOG_FMT_HTTP_METHOD,
> LOG_FMT_HTTP_URI,
> LOG_FMT_HTTP_PATH,
> + LOG_FMT_HTTP_QUERY,
> LOG_FMT_HTTP_VERSION,
> LOG_FMT_HOSTNAME,
> LOG_FMT_UNIQUEID,
> diff --git a/src/log.c b/src/log.c
> index ffd8f10..1112f8a 100644
> --- a/src/log.c
> +++ b/src/log.c
> @@ -111,6 +111,7 @@ static const struct logformat_type
> logformat_keywords[] = {
> { "hsl", LOG_FMT_HDRRESPONSLIST, PR_MODE_TCP, LW_RSPHDR, NULL }, /*
> header response list */
> { "HM", LOG_FMT_HTTP_METHOD, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP
> method */
> { "HP", LOG_FMT_HTTP_PATH, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP path */
> + { "HQ", LOG_FMT_HTTP_QUERY, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP query
> */
> { "HU", LOG_FMT_HTTP_URI, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP full
> URI */
> { "HV", LOG_FMT_HTTP_VERSION, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP
> version */
> { "lc", LOG_FMT_LOGCNT, PR_MODE_TCP, LW_INIT, NULL }, /* log counter */
> @@ -937,6 +938,7 @@ int build_logline(struct stream *s, char *dst,
> size_t maxsize, struct list *list
> struct chunk chunk;
> char *uri;
> char *spc;
> + char *qmark;
> char *end;
> struct tm tm;
> int t_request;
> @@ -1578,6 +1580,42 @@ int build_logline(struct stream *s, char *dst,
> size_t maxsize, struct list *list
> last_isspace = 0;
> break;
>
> + case LOG_FMT_HTTP_QUERY: // %HQ
> + uri = txn->uri ? txn->uri : "<BADREQ>";
> +
> + if (tmp->options & LOG_OPT_QUOTE)
> + LOGCHAR('"');
> +
> + end = uri + strlen(uri);
> + // look for the first question mark
> + while (uri < end && *uri != '?')
> + uri++;
> +
> + qmark = uri;
> +
> + // look for first space or question mark after url
> + while (uri < end && !HTTP_IS_SPHT(*uri))
> + uri++;
> +
> + if (!txn->uri) {
> + chunk.str = "<BADREQ>";
> + chunk.len = strlen("<BADREQ>");
> + } else {
> + chunk.str = qmark;
> + chunk.len = uri - qmark;
> + }
> +
> + ret = encode_chunk(tmplog, dst + maxsize, '#', url_encode_map,
> &chunk);
> + if (ret == NULL || *ret != '\0')
> + goto out;
> +
> + tmplog = ret;
> + if (tmp->options & LOG_OPT_QUOTE)
> + LOGCHAR('"');
> +
> + last_isspace = 0;
> + break;
> +
> case LOG_FMT_HTTP_URI: // %HU
> uri = txn->uri ? txn->uri : "<BADREQ>";
>
> --
> 2.1.3
>