09:04:29 <aspiers> #startmeeting ha
09:04:30 <openstack> Meeting started Mon Oct  3 09:04:29 2016 UTC and is due to finish in 60 minutes.  The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:04:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:04:33 <openstack> The meeting name has been set to 'ha'
09:05:01 <aspiers> #topic status updates
09:05:17 <aspiers> I'm not sure if you saw the discussions on the clusterlab lists
09:05:28 <ddeja> unfortunately no
09:05:43 * ddeja should subscribe to clusterlabs list
09:06:12 <aspiers> #info discussions on clusterlab lists about the future of OpenStack RAs
09:06:18 <aspiers> http://clusterlabs.org/pipermail/developers/2016-September/000339.html
09:07:13 <aspiers> although Red Hat are mostly bypassing this discussion, since they are ditching Pacemaker for (almost?) all active/active services
09:07:35 <ddeja> I've also heard they want to do so
09:07:38 <aspiers> apparently they will rely on HAproxy and the services themselves for correctness, and some other monitoring system for reporting failures
09:08:06 <aspiers> so every service should be able to function correctly even when any of its dependencies are dead or misbehaving
09:08:22 <aspiers> that sounds like a bit of a leap of faith to me, at least at the current time
09:08:25 <aspiers> but ICBW
09:09:16 <aspiers> Barcelona is rapidly approaching! did you get travel approved yet?
09:09:26 <ddeja> nope...
09:09:29 <aspiers> :-/
09:09:50 <ddeja> yup, I think the same
09:10:05 <aspiers> mine is confirmed
09:10:14 <ddeja> that's cool
09:10:22 <aspiers> I really want to get 3-4 of the specs completed before we arrive
09:10:52 <ddeja> Me too, it is finally after the release week, so I have time for specs preparation
09:11:22 <aspiers> great
09:11:27 <aspiers> maybe we can form a quick plan now
09:11:31 <ddeja> ok
09:11:37 <radeks> are there any HA meetings in Barcelona calendar?
09:11:58 <aspiers> radeks: sadly not, I pushed quite hard for it but no luck
09:12:30 <radeks> yeahhh this explain why I couldn't find anything :)
09:12:43 <aspiers> radeks: http://lists.openstack.org/pipermail/openstack-dev/2016-August/100570.html
09:12:49 <ddeja> aspiers, radeks: but there would be some unofficial meetings, like in Austin
09:12:54 <aspiers> right
09:12:59 <ddeja> we would send email to OpenstackDev
09:13:08 <radeks> great
09:13:21 <aspiers> I think we could maybe use cross-project space, but only if we have specs which clearly explain the scope
09:13:36 <samP> sorry im late
09:13:49 <aspiers> hey samP! welcome back :)
09:14:05 <samP> aspiers: thanks
09:14:16 <aspiers> #topic specs
09:14:24 <aspiers> so I merged the VM monitoring spec
09:14:44 <ddeja> \o/
09:14:45 <aspiers> I think it's basically done, although a few more details on the event payload wouldn't hurt, and also something about SSL support
09:15:30 <ddeja> we could discuss it in Barcelona
09:15:39 <aspiers> for the benefit of lurkers, we are talking about the specs in https://etherpad.openstack.org/p/newton-instance-ha
09:15:46 <aspiers> lines 115 onwards
09:15:59 <aspiers> I will do host monitoring - that should be easy
09:16:09 <aspiers> obviously host recovery is a very important one
09:16:17 <aspiers> ddeja: you still OK to start that one?
09:16:27 <ddeja> aspiers: yes
09:16:27 <samP> I have to do remaining sepc..(TT)
09:16:40 <ddeja> aspiers: I'll send first draft this week
09:16:47 <aspiers> great :)
09:16:51 <ddeja> (you can put an action on me ;)
09:17:00 <aspiers> #action ddeja to draft host recovery spec
09:17:06 <aspiers> #action aspiers to draft host monitoring spec
09:18:40 <aspiers> I am wondering if we still need a libvirt OCF RA
09:19:00 <ddeja> aspiers: why not?
09:19:19 <aspiers> oh yeah, we do
09:19:19 <samP> isnt that basically coverd in VM monitoring?
09:19:36 <aspiers> let me find the clusterlabs list discussion on this
09:23:05 <aspiers> http://clusterlabs.org/pipermail/users/2016-May/003040.html was a long thread about informing RAs about recovery
09:23:30 <ddeja> aspiers: oh, thism thread
09:23:45 <ddeja> I remember reading it...
09:24:23 <aspiers> let me find a link to the consensus
09:24:34 <ddeja> but I don't remember how this thread is connected with monitoring or not the VMs in libvirt?
09:24:52 <ddeja> it was about services AFAIK
09:25:24 <aspiers> http://clusterlabs.org/pipermail/users/2016-June/003344.html
09:25:52 <aspiers> so the original discussion we had in Austin was: if nova-compute fails, we want to try to restart a few times before fencing - right?
09:26:03 <ddeja> there was such idea
09:26:10 <ddeja> indeed
09:26:21 <aspiers> but when we give up on the retries, we want to do nova service-disable
09:26:30 <ddeja> yup
09:26:36 <aspiers> and similarly for if libvirt fails
09:26:50 <ddeja> yup
09:26:50 <samP> aspiers: that discussion start with how to recover process failures
09:27:03 <aspiers> but I think the consensus of this long thread on the list was, *every* time nova-compute stops, do nova service-disable
09:27:11 <aspiers> not just after the final failed retry
09:27:20 <aspiers> and when it starts, do nova service-enable
09:27:32 <aspiers> so we don't handle the final retry as a special case
09:27:49 <ddeja> ok, agreee
09:27:55 <aspiers> if we take that approach, we don't even need an OCF RA for libvirt
09:28:11 <aspiers> since there is already an ordering (or group) constraint linking libvirtd with nova-compute
09:28:30 <aspiers> so if libvirtd fails, nova-compute is automatically stopped, and then nova  service-disable is called
09:28:38 <ddeja> aspiers: OCF RA for libvirt was meant for monitoring the vms inside libvirt?
09:29:00 <aspiers> no, just for monitoring libvirtd
09:29:18 <aspiers> the spec I just merged is for monitoring the VMs
09:29:18 <ddeja> oh, OK
09:29:30 <ddeja> now we are on the same page
09:29:36 <aspiers> cool :)
09:29:49 <aspiers> the only reason I can think of for writing a libvirtd OCF RA, is to improve the monitoring
09:30:03 <ddeja> so yes, if we want to disable nova every time it stops, we can skip the RA for libvirt
09:30:04 <aspiers> e.g. if libvirtd is running but hung, then better monitoring could catch this
09:30:05 <samP> aspiers: what about other processes such as iscsid or mutipathd
09:30:50 <aspiers> it could do virsh sysinfo or similar every 10 seconds
09:31:13 <aspiers> samP: interesting point
09:31:16 <samP> aspiers: I think we start that discussion for how monitor the processes.
09:31:41 <samP> aspiers: then, dicussion was limited to nova-compute and libvirtd
09:32:01 <aspiers> samP: I think it's the same situation for iscsid / multipathd / libvirtd
09:32:10 <aspiers> if all are dependencies of nova-compute
09:32:30 <aspiers> then Pacemaker needs to monitor all of them, and have order/group constraints
09:32:41 <aspiers> but I think that's not OpenStack-specific
09:33:24 <aspiers> e.g. there is already https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/iscsi
09:34:15 <aspiers> samP: but if you want to write a spec about the storage side, I'm definitely in favour :)
09:34:39 <aspiers> however if backend storage fails, I guess the only recovery action is fencing
09:34:42 <aspiers> which makes it simpler
09:34:48 <samP> aspiers: yep, I will write some details. so we can take a close look onit
09:35:04 <aspiers> OTOH, if libvirtd dies, the operator might choose a policy which keeps some VMs still running
09:35:20 <aspiers> they might prefer to do manual recovery of libvirtd, instead of killing important pets
09:35:40 <samP> aspiers: ype, that is one way to recover
09:36:17 <aspiers> ah, we already have that in the etherpad
09:36:24 <aspiers> lines 136-138
09:36:35 <aspiers> cool :)
09:37:09 <samP> aspiers: yep. missed it. thanks
09:37:55 <aspiers> #action aspiers to draft specs for libvirt and NovaCompute OCF RAs
09:38:09 <aspiers> samP: maybe you could start on the VM recovery spec?
09:38:31 <samP> yes, I will.
09:39:10 <aspiers> #action samP to draft VM recovery spec
09:39:16 <samP> aspiers: thx
09:39:28 <aspiers> OK great, I think that's a good enough plan for this week - plenty for everyone to do :)
09:39:37 <ddeja> aspiers: yup
09:39:42 <aspiers> BTW samP how was NYC Ops meetup?
09:39:58 <aspiers> or did we already discuss that? :)
09:40:00 <aspiers> I can't remember
09:40:19 <ddeja> I think we did
09:40:21 <samP> aspiers: I think we already did
09:40:32 <aspiers> OK never mind ;-)
09:40:33 <ddeja> samP sent us links to etherpad
09:40:45 <aspiers> oh yeah, I remember now
09:41:00 <aspiers> #topic local area meetups
09:41:06 <aspiers> I attended the OpenStack London meetup last week
09:41:12 <aspiers> there were ~70 people
09:41:19 <aspiers> maybe 50, not sure
09:41:30 <aspiers> anyway, I asked everyone how many pets they are seeing in OpenStack
09:41:39 <aspiers> and apparently it's still very, very common
09:41:52 <ddeja> that's somehow good for us
09:41:55 <aspiers> and lack of HA seems to be a significant problem, especially in forklift migrations from VMware
09:42:01 <aspiers> so yes, we are not wasting our time here :)
09:42:01 <ddeja> it means our work is not useless ;)
09:42:04 <aspiers> exactly
09:42:26 <aspiers> I think that will be true for a long time yet
09:42:45 <aspiers> thought you might appreciate hearing some extra motivation for our work ;)
09:42:53 <ddeja> :)
09:43:22 <aspiers> #topic next generation HA architecture
09:43:22 <ddeja> I was on a Wroclaw OpenStack meetup last month
09:43:30 <aspiers> oh sorry, changed topic too soon
09:43:39 <ddeja> but no discusson about Pets vs Cows
09:43:50 <samP> aspiers: ddeja : you may find the NY ops details in following meeting, http://eavesdrop.openstack.org/meetings/ha/2016/ha.2016-08-29-09.33.log.txt
09:43:53 <ddeja> Poland OpenStack community is not very mature unfortunately
09:44:00 <aspiers> samP: thanks :)
09:44:07 <aspiers> OK
09:44:19 <aspiers> I think my colleague Michal Jura goes to those
09:44:46 <ddeja> aspiers: I may saw him on attendes list
09:45:22 <ddeja> but not sure. Nevertheless, it is 350km from my city, so I'm not attending it every time
09:45:23 <aspiers> I'm pretty sure he has presented there a few times
09:45:27 <aspiers> OK
09:45:32 <aspiers> I guess it's not local for him either
09:45:50 <ddeja> yes, SUSE Poland is in different city also :)
09:46:03 <aspiers> right :)
09:47:03 <ddeja> ok, we can move to the topic ;)
09:47:33 <aspiers> ok
09:47:36 <aspiers> http://docs.openstack.org/ha-guide/intro-ha-controller.html#common-deployment-architectures
09:47:56 <aspiers> https://review.openstack.org/#/c/372843/
09:48:01 <aspiers> has been merged
09:48:07 <aspiers> to document Red Hat's new approach
09:48:40 <aspiers> which reduces use of Pacemaker to managing active/passive resources, like the VIPs
09:48:48 <aspiers> and compute HA, of course
09:49:26 <aspiers> I'm slowly coming around to this idea, but I think there are unresolved questions
09:49:40 <aspiers> e.g. it depends on all OpenStack services behaving correctly when their dependencies misbehave
09:49:58 <aspiers> like I said earlier, it sounds like a leap of faith - I'll believe it works when I see it :)
09:50:18 <aspiers> and also it requires some monitoring dashboard outside Pacemaker to see the health of all active/active services
09:50:18 <ddeja> aspiers: I belive Fuel works that way
09:51:05 <radeks> well it may move away some of the pain which comes from Pacemaker + Rabbit
09:51:28 <radeks> this is the biggest issue AFAIK and see in the field
09:51:36 <ddeja> My collegue wanted my help to configure something on a Fuel OpenStack cloud and I was suprised that there was only 5 or 6 services added to pacemaker
09:51:36 <aspiers> IIUC, here are the Fuel OCF RAs in use https://review.openstack.org/gitweb?p=openstack/fuel-library.git;a=tree;f=files/fuel-ha-utils/ocf;hb=HEAD
09:52:14 <aspiers> radeks: the biggest pain you see is Pacemaker+Rabbit?
09:52:20 <aspiers> radeks: you mean active/passive?
09:52:23 <ddeja> aspiers: yes, there was only network-related services
09:52:35 <aspiers> ddeja: were they using keepalived?
09:52:47 <ddeja> I don't know
09:53:37 <radeks> aspiers: what I mean by this is a problems rabbit has with full HA mirroring of queues, there is a performance penalty and many other like nasty behavior when network gets partitioned
09:54:25 <radeks> aaspires: and when you have to for whatever reason restart rabbitmq, all of them or one of them this is becoming a problem
09:54:50 <aspiers> radeks: ah, but with mirrored queues are you still using Pacemaker?
09:54:55 <aspiers> for Rabbit I mean
09:55:21 <radeks> yes, this is the default for TripleO for example
09:56:11 <radeks> aspires: if you are lucky with 3 controllers and you want to restart rabbitmq your OpenStack services goes down for at least 20 minutes
09:56:12 <aspiers> interesting, I didn't realise that active/active Rabbit with mirrored queues would still be controlled by Pacemaker
09:56:43 <aspiers> well anyway, we're running out of time
09:56:53 <ddeja> 5 minutes left ;)
09:56:54 <radeks> all the OpenStack services has dependency on rabbit, at least in TripleO
09:57:10 <aspiers> I don't expect any upstream actions on this right now
09:57:23 <aspiers> but I wanted to make sure everyone is aware of it, and the fact that it is now documented in the HA guide
09:57:25 <radeks> it's probably good subject for discussion in Barcelona
09:57:42 <aspiers> radeks: for sure :) although I think Red Hat is already set on this path 100% :)
09:58:54 <aspiers> I started this thread http://clusterlabs.org/pipermail/developers/2016-September/000339.html to try and minimise the difference between the two approaches
09:59:23 <aspiers> if the "older" approach of still using Pacemaker to manage active/active services could reuse systemd units, then there would not be a big difference
09:59:39 <aspiers> but currently all the OCF RAs reinvent the systemd wheel (or vice-versa)
09:59:52 <aspiers> but yes, let's discuss in Barcelona
10:00:06 <aspiers> radeks: if you are coming, please make sure to say hello :)
10:00:17 <aspiers> alright, we're out of time
10:00:25 <aspiers> if there is anything else, let's continue on #openstack-ha
10:00:30 <aspiers> thanks all, and bye for now!
10:00:35 <aspiers> #endmeeting