15:58:58 <odyssey4me> #startmeeting OpenStack-Ansible
15:58:59 <openstack> Meeting started Thu Jun 16 15:58:58 2016 UTC and is due to finish in 60 minutes.  The chair is odyssey4me. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:59:00 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:59:03 <openstack> The meeting name has been set to 'openstack_ansible'
16:00:00 <odyssey4me> #topic Agenda & rollcall
16:00:03 <prometheanfire> o/
16:00:04 <palendae> Morning
16:00:07 <d34dh0r53> o/
16:01:14 <odyssey4me> FYI mhayden and evrardjp are out today, and as no-one else volunteered to chair I'll go ahead with doing so
16:01:16 <mrhillsman> i'm here
16:01:22 <mrhillsman> o/
16:02:11 <odyssey4me> mrhillsman thanks for joining us - please invite any others in the OSIC team you're a part of to join
16:02:14 <antonym> o/
16:02:19 <adreznec> o/
16:02:35 <odyssey4me> #link https://wiki.openstack.org/wiki/Meetings/openstack-ansible
16:02:36 <michaelgugino> ahoy
16:02:51 <jmccrory_> o/
16:02:54 <bgmccollum> o/
16:03:00 <odyssey4me> mrhillsman ^ I recommend you add yourself (and your team members do too) to the regular attendees list so that you get pinged when the meeting starts
16:03:25 <odyssey4me> #topic Review action items from last week
16:03:35 <odyssey4me> #link http://eavesdrop.openstack.org/meetings/openstack_ansible/2016/openstack_ansible.2016-06-09-16.03.html
16:04:32 <odyssey4me> #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/096573.html
16:05:05 <odyssey4me> In the last two meetings we've discussed the testing split-out and there was reference to andymccr's email, as noted above.
16:05:24 <odyssey4me> Has anyone put any time into testing it and to provide feedback? I see no feedback on the ML.
16:05:59 <izaakk> o/
16:06:08 <inc0> o/
16:06:30 <raddaoui> o/
16:08:01 <odyssey4me> OK, I'll take that as a no so we'll have to carry it forward.
16:08:27 <odyssey4me> #action Everyone look through http://lists.openstack.org/pipermail/openstack-dev/2016-June/096573.html and respond asap.
16:08:55 <odyssey4me> It appears that phil_h isn't around and I haven't seen the Astara spec yet, so I guess that's still a WIP.
16:09:18 <odyssey4me> antonym You were working on Xen support. Do you have a quick progress update?
16:09:59 <antonym> yeah, i have bunch of it put together so far for 14.04, mainly just need to test it out and then split it out for the new multi os layout
16:10:14 <antonym> i can push up what i have so far and -1 if people want to start poking on it
16:10:57 <odyssey4me> antonym yeah, I'd encourage you to try and do a daily push - both so that you don't lose any work you've done, and so that anyone interested can review and provide guidance :)
16:11:26 <antonym> i had started testing it out on 14.04 with the UCA stuff but was getting some oddities, no not sure if it's just limping along on 14.04 or if i'll really need 16.04 to get it working
16:11:47 <antonym> sounds good, just didn't want to cause too much noise since it's still WIP
16:12:26 <antonym> there was some overlap with some of the libvirt stuff for kvm, so i kept it split for now, but it might be more optimal to merge some of the libvirt specific stuff together
16:12:47 <odyssey4me> antonym You had originally wanted to only target 16.04, so perhaps you should just run with that as the primary target first - then work out 14.04 later?
16:12:51 <spotz> o?
16:13:20 <antonym> odyssey4me: yeah, i had a cluster all set up on 14.04 already so i was poking that first, but yeah, i'll probably give 16.04 a try
16:13:59 <antonym> i think most of it should look pretty similiar for both versions, it's really the libvirt and xen versions that matter
16:14:12 <odyssey4me> antonym I started poking at the nova role yesterday and pushed https://review.openstack.org/330231
16:14:35 <antonym> yeah, i was just looking at that, i'll try and line up to what you're doing
16:14:45 <odyssey4me> the nova role is a bit of a mess thanks to organic growth... don't be afraid to propose patches that change the structure up to make more sense
16:15:05 <michaelgugino> nova plays for 16.04 were merged earlier this week, so it should be a good starting point
16:15:29 <antonym> sounds good
16:15:46 <odyssey4me> michaelgugino yeah, thanks for that - it seems that tempest testing caused some grief there... it's part of the pain of following trunk I guess
16:16:15 <michaelgugino> no, it was the fact that my patchet was rebased by gerrit without me telling it to do so or indicating that it had done so
16:16:35 <michaelgugino> the issue was with the patch that merged and enabled UCA by default, which is not available for 16.04 at this time.
16:16:53 <odyssey4me> michaelgugino heh, yeah - I haven't yet figured out exactly when it does and doesn't rebase - I know that it definitely does in the gate queue and I think that it doesn't in the check queue
16:17:15 <michaelgugino> it was failing the check queue
16:17:21 <odyssey4me> michaelgugino FYI there is now UCA for Xenial, the patch I mentioned above changes that default
16:17:46 <odyssey4me> UCA was published for Newton-1
16:18:06 <odyssey4me> anyway, thanks for persevering
16:18:29 <michaelgugino> no worries, I learned how to spin up tempest so it wasn't a complete waste of time
16:18:37 <odyssey4me> antonym thanks for the update - looking forward to seeing some patches :)
16:18:48 <odyssey4me> #topic Mid Cycle Planning
16:18:52 <antonym> yeah, no prob
16:19:05 <odyssey4me> #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/097218.html
16:19:13 <odyssey4me> #link https://etherpad.openstack.org/p/osa-midcycle-newton
16:19:36 <odyssey4me> It looks like we have a venue booked for August 10-12 in San Antonio.
16:20:01 <odyssey4me> Please note if you will be attending and also note your hotel needs.
16:20:02 <spotz> Yeah!
16:20:31 <odyssey4me> Rumour has it that mhayden may host us for a cookout too - we'll know more closer to the time. :)
16:20:59 <odyssey4me> Any questions/comments before we move on?
16:21:54 <spotz> Not sure where that rumour started:)
16:22:18 <mrhillsman> no objections from mhayden...may not be a rumour :)
16:23:27 <odyssey4me> Alright, let's move on.
16:23:37 <odyssey4me> #topic Power support update
16:23:51 <odyssey4me> adreznec do you have an update on progress
16:24:47 <adreznec> I'm working on revising the spec for multi-arch support right now, and we have 2 environments internally we're bringing up for development there.
16:25:28 <odyssey4me> sounds good - are there any blockers we can help with?
16:25:29 <adreznec> We're pursing finding replacements for the packages that aren't built for ppc64el today that aren't from the Rackspace repo
16:25:59 <odyssey4me> adreznec Which packages do you mean? Apt packages?
16:26:10 <adreznec> I think we're okay for now, we'll likely have more questions on details of the repo build process as we start peeling back that a bit
16:26:25 <adreznec> Yeah, things like percona-xtrabackup, qpress, etc
16:26:43 <odyssey4me> Ah OK - the non-Ubuntu things.
16:26:55 <adreznec> Some thing we can pull from upstream Xenial for now, other things like qpress we're still working
16:28:09 <odyssey4me> adreznec It may be a good idea to have an option to not use those external repositories as much as possible. Using those external repositories will have to remain an opt-in capability, but if we get MariaDB10 from the Ubuntu repository then that's a win.
16:28:50 <odyssey4me> But yeah, there will also be some other packages which are important for production (like xtrabackup) which may be harder to find.
16:28:50 <adreznec> odyssey4me: Yeah, I've been working a patch for that internally to see what I can solve there.
16:29:17 <odyssey4me> What may have to happen is that those extras just don't deploy on Power because they aren't available.
16:29:36 <adreznec> Yep, at least until we can get those extras built where needed
16:29:42 <odyssey4me> We can then perhaps have those be an opt-in thing for deployers who want them.
16:30:17 <adreznec> Yeah, that would be a documentation impact for the most part
16:31:02 <odyssey4me> Also, we aren't married to the RAX mirror. If there're other mirrors which carry packages and don't have rate limits then I'm all ears. The RAX mirror usage is partially historical, but also partially because we know there are no rate limits and we got killed int he gate by that in the past.
16:31:22 <michaelgugino> if you can run an x86 controller in a VM on power, or just use x86 controllers, that would work for the galera bits.
16:31:46 <adreznec> odyssey4me: Good to know. I know antonym was pursuing supporting ppc64el in the Rackspace mirrors
16:31:57 <antonym> yeah, still working on that
16:31:59 <odyssey4me> As an FYI I've requested two new mirrors from OpenStack-Infra - https://review.openstack.org/307831 (MariaDB) & https://review.openstack.org/330152 (UCA) - in order to hopefully gate more reliably using those packages.
16:32:06 <antonym> think the mirror person was out last week
16:32:30 <antonym> the power support is relevent to my interests as well :)
16:32:31 <adreznec> michaelgugino: Yeah, running the controller on x86 is the base case we're working, but we'd like both if possible
16:32:43 <adreznec> antonym: Great, thanks again for checking in to that
16:33:08 <antonym> np
16:33:55 <odyssey4me> great, thanks for the update - good to see the work progressing and it's always good to have a fresh set of eyes looking at what we're doing and suggesting different ways
16:34:06 <odyssey4me> moving along
16:34:11 <odyssey4me> #topic Release Planning and Decisions
16:34:45 <odyssey4me> Is anyone aware of any issues for the Mitaka/Liberty branches that should block the current branch from releasing?
16:35:31 <odyssey4me> The branches appear stable to me - I haven't seen any failures.
16:36:10 <odyssey4me> I'm still waiting for kilo-eol tags for Swift & Requirements before we do our kilo-eol... so that tag's on hold.
16:36:55 <odyssey4me> Alright, no objections so I'll request the releases after the meeting and push up the SHA bump reviews.
16:37:16 <odyssey4me> #topic Ubuntu 16.04 LTS Support
16:37:23 <odyssey4me> #link https://etherpad.openstack.org/p/openstack-ansible-newton-ubuntu16-04
16:37:43 <odyssey4me> It looks to me like we're making good progress. It seems like we're actually ahead of schedule, which is great.
16:37:57 <automagically> Don’t jinx it odyssey4me
16:38:00 <automagically> ;)
16:38:10 <odyssey4me> It would be nice to have the code churn bed down and be finalised by Newton-3 so that we can beat on it for bugs.
16:38:15 <odyssey4me> heh automagically :)
16:39:13 <odyssey4me> I'd appreciate some help keeping the etherpad up to date. Please take a look through and cross out any of the roles that are done.
16:39:19 <michaelgugino> I think we should start moving the checks into the normal process as nv, instead of doing check experimental for 16.04 on the projects where the patch is complete
16:39:29 <odyssey4me> Of course if you see something not yet done, please feel free to pick it up and go ahead with a patch.
16:40:17 <michaelgugino> it would be great to not introduce regressions if at all possible
16:40:18 <odyssey4me> michaelgugino yes, agreed - I'd like to move the current -nv checks to be voting for all the infrastructure roles
16:40:35 <odyssey4me> then move all the experimental checks on the os roles to be -nv
16:40:48 <odyssey4me> happy with that?
16:40:56 <spotz> +1
16:41:00 <michaelgugino> sounds good to me
16:41:02 <prometheanfire> +1
16:41:22 <odyssey4me> cool, I'll get that patch submitted tomorrow most likely
16:41:41 <odyssey4me> Meanwhile I've also submitted https://review.openstack.org/329892 to add experimental checks for CentOS and Xenial to the integrated gate.
16:42:45 <odyssey4me> It needs a revision, which will depend on https://review.openstack.org/330500 being merged into master...
16:42:56 <odyssey4me> so that may only happen next week
16:43:31 <odyssey4me> #action odyssey4me to convert passing -nv CentOS/Xenial checks to voting
16:43:52 <odyssey4me> #action odyssey4me to convert passing experimental CentOS/Xenial checks to -nv
16:44:17 <michaelgugino> with some of the 16.04 work done, the CentOS work should follow easily now that we have the systemd parts in place
16:44:17 <odyssey4me> #action odyssey4me to convert non-passing experimental CentOS/Xenial checks to -nv for greater visibility
16:44:45 <odyssey4me> michaelgugino agreed, although our primary target for Newton is Xenial it'd be nice to get CentOS done if we have the time
16:45:15 <odyssey4me> #action odyssey4me to implement experimental CentOS/Xenial checks for the integrated repo
16:45:39 <odyssey4me> ok, we need to move along
16:45:48 <odyssey4me> #topic Install Documentation Update
16:46:35 <odyssey4me> Both darrenc and asettle have been doing a great job of revising the existing documentation. Thanks to mhayden and automagically for the reviews to move things along too.
16:47:10 <odyssey4me> Any thoughts/questions. I know darrenc is an Aus so he can't attend, and asettle is on holiday so she can't update.
16:48:21 <odyssey4me> OK, moving on then.
16:48:27 <odyssey4me> #topic Ansible 2.1
16:49:08 <odyssey4me> The Ansible 2.1 execution appears to be very much slower than 1.9. I haven't definitively brought it down to that as the culprit personally.
16:49:25 <odyssey4me> Has anyone had a moment to figure out why we're suddenly hitting so many timeouts?
16:50:06 <odyssey4me> We've obviously added quite a bit to OSA in this and the last cycle, but the Mitaka build seems to pass just fine.
16:50:16 <odyssey4me> automagically any thoughts?
16:50:20 <automagically> I know palendae dug in to see if dynamic inventory changes might be the culprit, but he reported that timing there was pretty consistent
16:50:20 <odyssey4me> jmccrory_ ?
16:50:44 <palendae> automagically, That and if it's affecting more than just https://review.openstack.org/#/c/325380/, it can't be that change
16:50:47 <automagically> I’ve not personally spent any time analysing it
16:50:51 <odyssey4me> I saw that stevelle picked up that rabbitmq user/vhost additions were taking around a minute each, which is super slow.
16:51:28 <jmccrory> haven't dug into it much, can spend some time looking today
16:51:38 <odyssey4me> He's put in a PoC patch to try a few things there. https://review.openstack.org/330173
16:52:36 <stevelle> odyssey4me: I'm afraid that PoC will likely be abandoned but I will spend a little more time on it first
16:52:37 <odyssey4me> I'm really starting to lean towards not maintaining 1.9 compatibility for Newton if we're able to optimise more. I *really* don't want build times to get *longer* for Newton.
16:53:01 <palendae> I had thought we had some significant progress on build times dropping
16:53:08 <palendae> Evidently something's undone that
16:53:19 <odyssey4me> That said, after looking through the nova role, we're doing some very scary include nesting all over the place and I'm pretty sure that we can optimise the execution work flow a lot more while maintaining 1.9 compatibility.
16:53:55 <odyssey4me> palendae yeah, the build timeouts in the gate increased massive from 2 June
16:55:07 <odyssey4me> Anyway, we're almost out of time so let's move on. I just wanted to raise a concern there and ask for eyes to dig into it where possible.
16:55:13 <odyssey4me> #topic Open Discussion
16:55:44 <odyssey4me> A quick intro - inc0 mrhillsman izaakk raddaoui are an OSIC team working on some stuff related to OSA.
16:56:15 <odyssey4me> I'd like to ask you guys to please add an agenda item for next week to introduce yourselves, what you're doing and where you'd like help.
16:56:38 <automagically> Really appreciate the work you guys have been doing on upgrades…
16:56:39 <michaelgugino> I will be taking over the lxd integration directly.  My associate here is too busy to dive in at the moment, so I'm hoping to get that out the door in the next few weeks.  Also, I'll be writing the openstack-watcher plays along side that project, and will be sort of a cross project liason
16:57:15 <odyssey4me> We have a lot of work on the go and some of what you're doing probably relates to what pockets of other OSA members are doing, so it's important for there to be visibility and for you to use #openstack-ansible as a medium to connect with the rest of us.
16:57:18 <inc0> odyssey4me, will do, first thing would obviously be upgrades L->M
16:57:22 <inc0> manual even, just working
16:57:23 <michaelgugino> I'll be out all next week fyi.
16:57:25 <inc0> thanks
16:57:35 <odyssey4me> We're unfortunately out of time - please move discussion to #openstack-ansihble
16:57:41 <odyssey4me> #endmeeting