Benjamin Smith <lists@...> writes:
>
> Igor,
>
> Thanks for the response; I didn't see this email until just now as it
didn't
> go through the mailing list and so wasn't filtered as expected.
>
> I spent my morning trying everything I could think of to get haproxy's
agent-
> check to work consistently. The main symptom is that haproxy would
mark hosts
> with the status of "DRAIN" and provide no clues as to why, even with
log-
> health-checks on. After a *lot* of trial and error, I've found the
following
> that seem to be bugs, on the latest 1.5.11 release, running on CentOS
6.
>
> 1) agent-check output words sometimes handled inconsistently, ignored,
or
> misunderstood if " " was used instead of "," as a separator.
>
> This is understood:
> echo "ready,78%\r\n"
>
> This line often causes a DRAIN state. A restart of haproxy was
insufficient to
> clear the DRAIN state: (see #3)
> echo "ready 78%\r\n"
>
> 2) Inconsistent logging of DRAIN status change when health logging was
on.
> (the server would turn blue in the stats page without any logging as
to why.
> Logging status would sometimes say " Server $service/$name is UP
(leaving
> forced drain)" even as the stats page continues to report DRAIN state!
>
> 3) Even when the agent output was amended as above, for hosts that
were set to
> the DRAIN state pursuant to #1 issue were not brought back to ready/up
state
> until "enable health $service/$host" and/or "enable agent
$service/$host" was
> sent to the stats port.
>
> 4) Setting the server weight to 10 seems to help a significant amount.
If, in
> fact, haproxy can't handle 35% of 1 it should throw an error on
startup IMHO.
>
> See also my comments interspersed below:
>
> Thanks,
>
> Benjamin Smith
>
> On Tuesday, April 14, 2015 10:50:31 AM you wrote:
> > On Tue, Apr 14, 2015 at 10:11 AM, Igor Cicimov <
> >
> > igorc@...> wrote:
> > > On Tue, Apr 14, 2015 at 5:00 AM, Benjamin Smith <lists@...>
> > >
> > > wrote:
> > >> We have 5 Apache servers behind haproxy and we're trying to
enable use
> > >> the
> > >> "httpchk" option along with some performance monitoring. For some
reason,
> > >> haproxy keeps thinking that 3/5 apache servers are "down" even
though
> > >> it's
> > >> obvious that haproxy is both asking the questions and the servers
are
> > >> answering.
> > >>
> > >> Is there a way to log httpchk failures? How can I ask haproxy why
it
> > >> seems to
> > >> think that several apache servers are down?
> > >>
> > >> Our config:
> > >> CentOS 6.x recently updated, 64 bit.
> > >>
> > >> Performing an agent-check manually seems to give good results.
The below
> > >> result is immediate:
> > >> [root <at> xr1 ~]# telnet 10.1.1.12 9333
> > >> Trying 10.1.1.12...
> > >> Connected to 10.1.1.12.
> > >> Escape character is '^]'.
> > >> up 78%
> > >> Connection closed by foreign host.
> > >>
> > >>
> > >> I can see that xinetd on the logic server got the response:
> > >> Apr 13 18:45:02 curie xinetd[21890]: EXIT: calcload333 status=0
pid=25693
> > >> duration=0(sec)
> > >> Apr 13 18:45:06 curie xinetd[21890]: START: calcload333 pid=26590
> > >> from=::ffff:10.1.1.1
> > >>
> > >>
> > >> I can see that apache is serving happy replies to the load
balancer:
> > >> [root <at> curie ~]# tail -f /var/log/httpd/access_log | grep -i
"10.1.1.1 "
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:15 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:17 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:19 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> ^C
> > >
> > > I have a feeling you might have been little bit confused here. Per
my
> > > understanding, and your configuration:
> > >
> > > server server10 10.1.1.10:20333 maxconn 256 *check agent-check
agent-port
> > > 9333 agent-inter 4000*
> > >
> > > the HAP is doing a health check on the agent you are using and not
on the
> > > Apache so the apache response in this case looks irrelevant to me.
I don't
> > > know how did you setup the agent since you haven't posted that
part but
> > > this is an excellent article by Malcolm Turnbull, the inventor of
> > > agent-check, that might help:
> > >
> > >
> > > http://blog.loadbalancer.org/open-source-windows-service-for-
reporting-ser
> > > ver-load-back-to-haproxy-load-balancer-feedback-agent/
>
> We used this exact blog entry as our starting point. In our case, the
xinetd
> script compares load average, apache process count, cpu info and a
little salt
> to come up with a number ranging from 0% to 500%.
>
> > and press enter twice and check the output. Other option is using
curl:
> >
> > $ curl -s -S -i --http1.0 -X OPTIONS http://10.1.1.12:20333/
>
> [root <at> xr1 ~]# curl -s -S -i --http1.0 -X OPTIONS
http://10.1.1.12:20333
> HTTP/1.1 302 Found
> Date: Tue, 14 Apr 2015 23:39:40 GMT
> Server: Apache/2.2.15 (CentOS)
> X-Powered-By: PHP/5.3.3
> Set-Cookie: PHPSESSID=3ph0dvg4quebl1b2e711d8i5p1; path=/; secure
> Cache-Control: public, must-revalidate, max-age=0
> X-Served-By: curie.-SNIP-
> Location: /mod.php/index.php
> Vary: Accept-Encoding
> Content-Length: 0
> Connection: close
> Content-Type: text/html; charset=UTF-8
>
> > and some variations of the above that I often use to check the
headers only:
> >
> > $ curl -s -S -I --http1.0 -X OPTIONS http://10.1.1.12:20333/
> > $ curl -s -S -D - --http1.0 -X OPTIONS http://10.1.1.12:20333/
> >
> > You can also try the health check with HTTP/1.1 version which
provides
> > keepalive but you need to specify the Host header in that case.
> >
> > By the way, any errors in the haproxy logs? Maybe set the log mode
to debug?
>
> Originally there was very little useful data in the log files at all.
Adding
> the log-health-checks helped but it's still frustratingly incomplete.
>
>
I ran into a similar scenario yesterday on 'HA-Proxy version 1.5.4
2014/09/02'. We had a situation where a radosgw on one of the backend
servers was not releasing memory and apache was occasionally showing 500
error messages in the apache log, rados had stopped logging so we wanted
to stop new connections going to the server and then restart radosgw.
Initially we issued the 'server disable' command to the haproxy socket,
and saw in the health status that the server was in MAINT mode, but
didnt see new connections stop arriving at the server, so we waited and
watched it and about 30 minutes later we issued the 'status drain'
command to the socket, and immediately saw connections stop arriving.
We restarted the radosgw service at this point. We then issued the
'status ready' command to the socket and saw connections start arriving
again, but the server was flip-flopping between MAINT and DRAIN states
in the health status output. It was only after I re-issued the 'server
enable' command twice to the socket that the server appeared in the
health status output as being consistently colored green in its row, and
showing UP status. However I still saw the following in the logs after
issuing the last 'server enable' command.
from: haproxy-status.log:
+++++
Server swift_cluster/css-host1-036 is UP/READY (leaving forced
maintenance).
Server swift_nonssl_cluster/css-host1-036 is UP (leaving forced drain).
+++++
The hosts still appear to be receiving new connections, even before i
re-issued the 'server enable' command again to the socket and the health
status output was showing the host flip-flopping between the MAINT and
DRAIN states. Everything seems ok/UP now, and has been stable for the
last 12 hours, with no apparent change in the health status output.
....hope this helps...
Lucky
>
> Igor,
>
> Thanks for the response; I didn't see this email until just now as it
didn't
> go through the mailing list and so wasn't filtered as expected.
>
> I spent my morning trying everything I could think of to get haproxy's
agent-
> check to work consistently. The main symptom is that haproxy would
mark hosts
> with the status of "DRAIN" and provide no clues as to why, even with
log-
> health-checks on. After a *lot* of trial and error, I've found the
following
> that seem to be bugs, on the latest 1.5.11 release, running on CentOS
6.
>
> 1) agent-check output words sometimes handled inconsistently, ignored,
or
> misunderstood if " " was used instead of "," as a separator.
>
> This is understood:
> echo "ready,78%\r\n"
>
> This line often causes a DRAIN state. A restart of haproxy was
insufficient to
> clear the DRAIN state: (see #3)
> echo "ready 78%\r\n"
>
> 2) Inconsistent logging of DRAIN status change when health logging was
on.
> (the server would turn blue in the stats page without any logging as
to why.
> Logging status would sometimes say " Server $service/$name is UP
(leaving
> forced drain)" even as the stats page continues to report DRAIN state!
>
> 3) Even when the agent output was amended as above, for hosts that
were set to
> the DRAIN state pursuant to #1 issue were not brought back to ready/up
state
> until "enable health $service/$host" and/or "enable agent
$service/$host" was
> sent to the stats port.
>
> 4) Setting the server weight to 10 seems to help a significant amount.
If, in
> fact, haproxy can't handle 35% of 1 it should throw an error on
startup IMHO.
>
> See also my comments interspersed below:
>
> Thanks,
>
> Benjamin Smith
>
> On Tuesday, April 14, 2015 10:50:31 AM you wrote:
> > On Tue, Apr 14, 2015 at 10:11 AM, Igor Cicimov <
> >
> > igorc@...> wrote:
> > > On Tue, Apr 14, 2015 at 5:00 AM, Benjamin Smith <lists@...>
> > >
> > > wrote:
> > >> We have 5 Apache servers behind haproxy and we're trying to
enable use
> > >> the
> > >> "httpchk" option along with some performance monitoring. For some
reason,
> > >> haproxy keeps thinking that 3/5 apache servers are "down" even
though
> > >> it's
> > >> obvious that haproxy is both asking the questions and the servers
are
> > >> answering.
> > >>
> > >> Is there a way to log httpchk failures? How can I ask haproxy why
it
> > >> seems to
> > >> think that several apache servers are down?
> > >>
> > >> Our config:
> > >> CentOS 6.x recently updated, 64 bit.
> > >>
> > >> Performing an agent-check manually seems to give good results.
The below
> > >> result is immediate:
> > >> [root <at> xr1 ~]# telnet 10.1.1.12 9333
> > >> Trying 10.1.1.12...
> > >> Connected to 10.1.1.12.
> > >> Escape character is '^]'.
> > >> up 78%
> > >> Connection closed by foreign host.
> > >>
> > >>
> > >> I can see that xinetd on the logic server got the response:
> > >> Apr 13 18:45:02 curie xinetd[21890]: EXIT: calcload333 status=0
pid=25693
> > >> duration=0(sec)
> > >> Apr 13 18:45:06 curie xinetd[21890]: START: calcload333 pid=26590
> > >> from=::ffff:10.1.1.1
> > >>
> > >>
> > >> I can see that apache is serving happy replies to the load
balancer:
> > >> [root <at> curie ~]# tail -f /var/log/httpd/access_log | grep -i
"10.1.1.1 "
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:15 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:17 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> 10.1.1.1 - - [13/Apr/2015:18:47:19 +0000] "OPTIONS / HTTP/1.0"
302 - "-"
> > >> "-"
> > >> ^C
> > >
> > > I have a feeling you might have been little bit confused here. Per
my
> > > understanding, and your configuration:
> > >
> > > server server10 10.1.1.10:20333 maxconn 256 *check agent-check
agent-port
> > > 9333 agent-inter 4000*
> > >
> > > the HAP is doing a health check on the agent you are using and not
on the
> > > Apache so the apache response in this case looks irrelevant to me.
I don't
> > > know how did you setup the agent since you haven't posted that
part but
> > > this is an excellent article by Malcolm Turnbull, the inventor of
> > > agent-check, that might help:
> > >
> > >
> > > http://blog.loadbalancer.org/open-source-windows-service-for-
reporting-ser
> > > ver-load-back-to-haproxy-load-balancer-feedback-agent/
>
> We used this exact blog entry as our starting point. In our case, the
xinetd
> script compares load average, apache process count, cpu info and a
little salt
> to come up with a number ranging from 0% to 500%.
>
> > and press enter twice and check the output. Other option is using
curl:
> >
> > $ curl -s -S -i --http1.0 -X OPTIONS http://10.1.1.12:20333/
>
> [root <at> xr1 ~]# curl -s -S -i --http1.0 -X OPTIONS
http://10.1.1.12:20333
> HTTP/1.1 302 Found
> Date: Tue, 14 Apr 2015 23:39:40 GMT
> Server: Apache/2.2.15 (CentOS)
> X-Powered-By: PHP/5.3.3
> Set-Cookie: PHPSESSID=3ph0dvg4quebl1b2e711d8i5p1; path=/; secure
> Cache-Control: public, must-revalidate, max-age=0
> X-Served-By: curie.-SNIP-
> Location: /mod.php/index.php
> Vary: Accept-Encoding
> Content-Length: 0
> Connection: close
> Content-Type: text/html; charset=UTF-8
>
> > and some variations of the above that I often use to check the
headers only:
> >
> > $ curl -s -S -I --http1.0 -X OPTIONS http://10.1.1.12:20333/
> > $ curl -s -S -D - --http1.0 -X OPTIONS http://10.1.1.12:20333/
> >
> > You can also try the health check with HTTP/1.1 version which
provides
> > keepalive but you need to specify the Host header in that case.
> >
> > By the way, any errors in the haproxy logs? Maybe set the log mode
to debug?
>
> Originally there was very little useful data in the log files at all.
Adding
> the log-health-checks helped but it's still frustratingly incomplete.
>
>
I ran into a similar scenario yesterday on 'HA-Proxy version 1.5.4
2014/09/02'. We had a situation where a radosgw on one of the backend
servers was not releasing memory and apache was occasionally showing 500
error messages in the apache log, rados had stopped logging so we wanted
to stop new connections going to the server and then restart radosgw.
Initially we issued the 'server disable' command to the haproxy socket,
and saw in the health status that the server was in MAINT mode, but
didnt see new connections stop arriving at the server, so we waited and
watched it and about 30 minutes later we issued the 'status drain'
command to the socket, and immediately saw connections stop arriving.
We restarted the radosgw service at this point. We then issued the
'status ready' command to the socket and saw connections start arriving
again, but the server was flip-flopping between MAINT and DRAIN states
in the health status output. It was only after I re-issued the 'server
enable' command twice to the socket that the server appeared in the
health status output as being consistently colored green in its row, and
showing UP status. However I still saw the following in the logs after
issuing the last 'server enable' command.
from: haproxy-status.log:
+++++
Server swift_cluster/css-host1-036 is UP/READY (leaving forced
maintenance).
Server swift_nonssl_cluster/css-host1-036 is UP (leaving forced drain).
+++++
The hosts still appear to be receiving new connections, even before i
re-issued the 'server enable' command again to the socket and the health
status output was showing the host flip-flopping between the MAINT and
DRAIN states. Everything seems ok/UP now, and has been stable for the
last 12 hours, with no apparent change in the health status output.
....hope this helps...
Lucky