16:01:10 <evrardjp> #startmeeting openstack_ansible_meeting
16:01:11 <openstack> Meeting started Tue Jan 16 16:01:10 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:14 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:01:16 <evrardjp> Here is our bug list for today https://etherpad.openstack.org/p/osa-bugtriage
16:01:17 <cloudnull> strgout: ping the kolla folks however if you need anything let us know
16:01:25 <evrardjp> #topic rollcall
16:01:28 <cloudnull> o/
16:01:30 <prometheanfire> o/
16:01:30 <d34dh0r53> o/
16:01:40 <evrardjp> quite a sync there!
16:02:15 <evrardjp> #topic focus of the week
16:02:46 <prometheanfire> freeze next week (not sure if osa cares)
16:02:49 <evrardjp> I am sorry, I didn't got the chance to finish up stability fixes of last week. I expected this week to be focused on nspawn and bare metal
16:02:54 <hwoarang> o/
16:03:06 <evrardjp> prometheanfire: we do have 2 extra weeks on top of the deadline
16:03:26 <evrardjp> but on top of that we are not respecting 100%, we've been quite active in the past, for backports.
16:03:48 <evrardjp> I still suggest we fix up upgrades, we have too many things pending: inventory, ansible 2.4 bugs, ...
16:04:10 <evrardjp> If we finish the stability for this week, next week we can introduce large changes.
16:04:34 <evrardjp> any objections?
16:04:53 <odyssey4me> well, typically we'd want all the features locked in by m3 and then turn our attention to bug squashing from m3 onwards
16:05:30 <evrardjp> I agree. so m3 is Jan 25th for cycle intermediary
16:05:35 <odyssey4me> are we in a position to get the inventory work merged at all? or are we pushing that to the next release?
16:05:36 <evrardjp> we do have 2w extra.
16:05:40 <odyssey4me> *cycle
16:06:59 <evrardjp> for the inventory part, I didn't mean any change in the inventory
16:07:07 <evrardjp> I mean an inventory bug from the upgrade.
16:07:16 <andymccr> o/
16:07:27 <evrardjp> so not a new thing, but just keeping up with what we have in master.
16:07:59 <evrardjp> on that note, let's move on, it doesn't look like there is any big objection on the focus of the week
16:08:07 <hwoarang> fine by me
16:08:52 <evrardjp> #topic pending bugs
16:08:54 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1743032
16:08:54 <openstack> Launchpad bug 1743032 in openstack-ansible "Galera cluster maintenance in OpenStack-Ansible" [Undecided,New]
16:10:24 <evrardjp> Do we have an expert in galera that touched on systemd recently? SamYaple maybe (although I think he has his own scripts)
16:11:09 <evrardjp> if nobody can have a look at this, I will go to the next bug
16:12:35 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742749
16:12:35 <openstack> Launchpad bug 1742749 in openstack-ansible "Tests repo has too many repetitions of the branch name" [Undecided,New]
16:12:36 <cloudnull> I can look into this
16:13:01 <evrardjp> cloudnull: I can assign that to you, but I'd rather we work together on a big one :D
16:13:07 <evrardjp> up to you if you have cycles
16:13:18 <evrardjp> in any case, thank you for you showing up:)
16:13:22 <cloudnull> assigned to me :)
16:13:25 <evrardjp> thanks!
16:13:27 <quotaq> Hi, question about quotas. I have project where "cinder absolute-limits" shows totalGigabytesUsed=5198, but in cinder backend(ceph) I see just 744GB used. Lots of space available, but frequently hitting quota maxTotalVolumeGigabytes=6000. Do I need to tune some overcommitment? Using openstack-ansible Pike.
16:13:43 <SamYaple> evrardjp: cloudnull there is a conversation about this bug from acouple days back
16:13:48 <evrardjp> quotaq: we are currently in our triage process, could you come back with your question in 47 minutes please?
16:13:56 <evrardjp> SamYaple: oh.
16:14:01 <evrardjp> and?
16:14:07 <cloudnull> SamYaple: I'll look back over the logs
16:14:16 <evrardjp> thanks for the feedback SamYaple !
16:14:21 <evrardjp> ok let's move on
16:14:23 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742749
16:14:24 <openstack> Launchpad bug 1742749 in openstack-ansible "Tests repo has too many repetitions of the branch name" [Undecided,New]
16:14:27 <SamYaple> basically its just improve the docs a bit by showing howto get the seqno from nodes that were improperly shutdown
16:14:51 <evrardjp> SamYaple: it looks like it, probably worth testing and ensuring it's the case
16:15:15 <evrardjp> because here it's just a different doc (--wsrep new cluster)
16:15:17 <evrardjp> anyway
16:15:19 <evrardjp> let's move on
16:15:19 <SamYaple> sorry to derail :)
16:15:34 <evrardjp> for the bug just above, I'd mark it as confirm and wishlist
16:15:45 <evrardjp> anyone wanting to help us out there is very welcomed.
16:16:27 <evrardjp> everyone ok with confirmed wishlist?
16:16:38 <cloudnull> 4++
16:16:44 <evrardjp> if there is a new contributor willing to help there, we can discuss about the implementation later.
16:16:58 <evrardjp> ok next
16:17:00 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742732
16:17:01 <openstack> Launchpad bug 1742732 in openstack-ansible "SELinnux AVC: can't use nova's ssh keys" [Low,New] - Assigned to Major Hayden (rackerhacker)
16:17:23 <openstackgerrit> Periyasamy Palanisamy proposed openstack/openstack-ansible master: Make Opendaylight as the BGP speaker using Quagga  https://review.openstack.org/523907
16:17:42 <evrardjp> mhayden: what's the full impact of this bug?
16:18:03 <mhayden> evrardjp: it breaks live migration on centos
16:18:04 <evrardjp> I think if people can't do live migration it sounds like a "more than low" bug.
16:18:25 <mhayden> true, i probably set that in a hurry
16:18:47 <evrardjp> I'll tag it as confirmed and medium, rename it to "Live migration failure under Centos - SELinux AVC ..."
16:18:51 <evrardjp> thanks for the bug!
16:19:28 <evrardjp> next
16:19:30 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742717
16:19:32 <openstack> Launchpad bug 1742717 in openstack-ansible "haproxy still has a glance-registry backend, but glance-registry is deprecated" [Undecided,New]
16:19:46 <evrardjp> looks confirmed and low to me
16:19:55 <evrardjp> ok everyone?
16:20:45 <evrardjp> ok I will move on
16:20:49 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742538
16:20:51 <openstack> Launchpad bug 1742538 in openstack-ansible "OSA-deployed uWSGI used by Keystone warns: "!!! no internal routing support, rebuild with pcre support !!!"" [Undecided,New]
16:21:56 <odyssey4me> I'll pick up that haproxy one, given I did the work for glance.
16:22:04 <evrardjp> odyssey4me: thanks!
16:22:26 <SamYaple> the uwsgi one is just adding a dev libto thebuild repo
16:22:59 <odyssey4me> ja, I'll pick that one up too - low hanging fruit
16:22:59 <SamYaple> its super low priority in my opinion, since to my understanding we never exercisethe pcre code the way we use it
16:23:02 <evrardjp> yeah it looks like it again, which should be super easy
16:23:06 <odyssey4me> low/confirmed though
16:23:14 <evrardjp> yeah I agree with both of you.
16:23:17 <evrardjp> let's move on then
16:23:29 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742489
16:23:29 <openstack> Launchpad bug 1742489 in openstack-ansible "Keepalived will deploy and fail if auth_pass is not set" [Low,New]
16:23:31 <odyssey4me> really good bug reports from this guy
16:23:36 <evrardjp> yeah
16:23:44 <evrardjp> Christian Sarasin
16:23:58 <evrardjp> Sarrasin, sorry!
16:24:03 <evrardjp> anyway
16:24:20 <odyssey4me> sxc731 I believe
16:25:20 <evrardjp> I will take the next one. I can confirm it would be the case, but I am not sure where we should put the safeguard
16:25:44 <evrardjp> will have a look, and in the meantime, I'd say it's an user configuration issue, so I'd think it should be a confirmed low bug.
16:26:12 <openstackgerrit> Major Hayden proposed openstack/openstack-ansible-os_tempest master: Run tempest executables directly in the venv  https://review.openstack.org/534372
16:27:18 <evrardjp> agreed everyone?
16:27:37 <odyssey4me> ja, we've spoken about implementing a pattern to confirm pre-reqs before doing anything in the role
16:27:45 <odyssey4me> I think one or two roles actually have that done.
16:27:58 <odyssey4me> But we never really got traction for it.
16:28:49 <evrardjp> odyssey4me: maybe we should add this as healthcheck to not have too many things in the role. But yeah, it would mean running these healthcheck playbooks
16:29:17 <odyssey4me> it would also spread things all over the place, whereas the role can be self contained
16:29:58 <evrardjp> let's move on
16:30:20 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1742206
16:30:21 <openstack> Launchpad bug 1742206 in openstack-ansible "os-cinder-install.yml fails with conflicting mariadb packages on centos infra hosts" [Undecided,New]
16:32:35 <odyssey4me> we could probably add a task to remove known conflict packages
16:32:53 <odyssey4me> we do that in os_neutron and somewhere else I can't recall
16:33:17 <evrardjp> it doesn't sound bad to me
16:33:44 <odyssey4me> https://github.com/openstack/openstack-ansible-galera_server/blob/master/vars/ubuntu-16.04.yml#L68
16:34:10 <hwoarang> i can take care of that
16:34:25 <odyssey4me> http://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/tree/tasks/lxc_install_apt.yml#n16
16:34:35 <evrardjp> I'd say this is an important bug, but doesn't break gates or anything, so probably worth classifying as confirmed medium or confirmed high
16:34:50 <evrardjp> confirmed high?
16:35:06 <odyssey4me> not sure it's high given that this has not been picked up before
16:35:22 <odyssey4me> if hwoarang picks it up today, then high/medium makes no difference ;)
16:35:28 <evrardjp> fair point!
16:35:28 <openstackgerrit> Major Hayden proposed openstack/openstack-ansible-os_tempest master: Run tempest executables directly in the venv  https://review.openstack.org/534372
16:35:38 <evrardjp> let's mark it as medium confirmed and assign to hwoarang :)
16:35:42 <evrardjp> hwoarang: thank you very much!
16:35:48 <odyssey4me> yeah, thanks hwoarang
16:35:59 <odyssey4me> always cleaning up after mhayden :)
16:36:10 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1741221
16:36:11 <openstack> Launchpad bug 1741221 in openstack-ansible "P-Q upgrade fails due to removal of vars plugin" [Undecided,New] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard)
16:36:31 <mhayden> odyssey4me: what did i break now?!
16:36:51 <evrardjp> this last one needs a bump in stable/pike
16:37:04 <odyssey4me> mhayden to be honest, I just thought you were too quiet in the triage :p
16:37:06 <evrardjp> on top of that we're gonna have other issues, but I think we can mark it as in progress.
16:37:23 <cloudnull> evrardjp: on that issue I think we just need to disable the use of our plugins when bootstraping
16:37:30 <cloudnull> 1741221
16:37:33 <evrardjp> yes
16:37:36 <evrardjp> I agree
16:37:41 <evrardjp> that was my first approach
16:38:03 <cloudnull> the plugin system resolves the plugin system however the plugin system going from ansible 2.3 to 2.4 is broken.
16:38:05 <evrardjp> here the problem itself is solved though
16:38:09 <evrardjp> yeah
16:38:19 <evrardjp> so
16:38:33 <evrardjp> what the next thing that will appear is the strategy plugin failure
16:38:35 <evrardjp> so
16:38:41 <evrardjp> we have two choices
16:39:00 <evrardjp> fix the upgrade by preventing to use the git-clone method in the bootstrap ansible of the upgrade to Q
16:39:22 <evrardjp> or remove the old plugins repo folder from /etc/ansible/roles/plugins.
16:39:43 <evrardjp> both are fine, and will be addressed in a new bug I haven't filed yet :)
16:40:00 <evrardjp> cloudnull: let's work on it together.
16:40:18 <evrardjp> in the meantime, let's continue the bug triage
16:40:20 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1741085
16:40:21 <openstack> Launchpad bug 1741085 in openstack-ansible "Playbook ceph-install.yml won't boostrap cluster" [Undecided,New]
16:41:07 <cloudnull> evrardjp: I think we can set https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L24-L31 to /dev/null and it'll work in bootstrap
16:41:15 <cloudnull> sorry , was looking around .
16:41:27 <odyssey4me> perhaps it'd be best to just do a backup of /etc/ansible in the upgrade scripts, like we do for a bunch of other stuff 0 or perhaps just renamed the folder entirely
16:42:13 <evrardjp> odyssey4me: that's what i did in my previous patch, but I suggest we discuss this approach in my patch and we continue the triage. We have so many bugs still pending
16:42:16 <odyssey4me> the bootstrap, IIRC, uses the ansible binary directly so I don't think the wrapper or any of those settings are involved?
16:42:32 <evrardjp> odyssey4me: you are right
16:43:02 <evrardjp> for the ceph-ansible part, logan- are you there by any chance?
16:44:11 <evrardjp> for after the bug triage, we should discuss this https://review.openstack.org/#/c/531123/2/scripts/run-upgrade.sh (because of the remaining plugins not yet handled -- the strategy )
16:44:20 <evrardjp> anyway
16:44:24 <evrardjp> ceph-ansible part!
16:44:33 <evrardjp> or more ceph-install part...
16:45:16 <evrardjp> andymccr: maybe? Do you have the issue with that playbook if you tried standalone, or you are with a completely new set of playbooks for your use cases?
16:46:28 <evrardjp> the bug is https://bugs.launchpad.net/openstack-ansible/+bug/1741085
16:46:29 <openstack> Launchpad bug 1741085 in openstack-ansible "Playbook ceph-install.yml won't boostrap cluster" [Undecided,New]
16:47:41 <logan-> id like to know which task is failing specifically
16:47:46 <logan-> will comment the bug
16:47:53 <evrardjp> ok.
16:48:01 <odyssey4me> given that all the services have a var to override the serial behaviour, that's not an unreasonable request
16:48:02 <evrardjp> it sounds plausible, and reasonable to do
16:48:13 <evrardjp> but I wonder why no one got that before.
16:48:22 <evrardjp> thanks for your comment logan- !
16:48:39 <evrardjp> next
16:48:41 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1740190
16:48:42 <openstack> Launchpad bug 1740190 in openstack-ansible "[DOCS] maintenance-tasks/network-maintain.rst has header with no content" [Undecided,New] - Assigned to Amy Marrich (amy-marrich)
16:49:01 <openstackgerrit> Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Force ansible to ignore plugins when bootstrapping  https://review.openstack.org/534377
16:49:03 <evrardjp> looks like confirmed and low
16:50:38 <evrardjp> cloudnull: smart.
16:50:39 <evrardjp> :D
16:50:40 <andymccr> evrardjp: looking now
16:51:14 <evrardjp> andymccr: no urgent need, logan- will have a look, but two heads are always better than one I guess. (Not sure if that translates)
16:51:25 <andymccr> yeah it does ;P
16:51:32 <evrardjp> for the docs bug, I'd say confirmed and low
16:51:39 <evrardjp> it doesn't impact a deployment or expectations
16:52:03 <evrardjp> let's move on
16:52:07 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1740103
16:52:08 <openstack> Launchpad bug 1740103 in openstack-ansible "setup-openstack.yaml fails and throws 403 forbidden error in infra1 keystone container" [Undecided,New]
16:52:09 <andymccr> we shouldnt need serial on restarts after ceph-ansible v3 anyway
16:52:15 <andymccr> since ceph-ansible does that automatically
16:52:30 <evrardjp> ok that is nice to know
16:52:41 <evrardjp> could you comment it on the bug then?
16:53:19 <andymccr> yeah sure
16:53:52 <evrardjp> thank you :)
16:54:33 <evrardjp> for that bug, I don't think it's confirmed.
16:57:29 <evrardjp> ok I have commented on that bug
16:57:35 <evrardjp> marked it as incomplete
16:57:48 <evrardjp> next
16:57:53 <evrardjp> and probably last for today
16:57:55 <evrardjp> #link https://bugs.launchpad.net/openstack-ansible/+bug/1739475
16:57:56 <openstack> Launchpad bug 1739475 in openstack-ansible "haproxy config error during ocata deploy on centos7" [Undecided,New]
16:59:46 <evrardjp> marking that last one as incomplete too
16:59:50 <evrardjp> ok for everyone?
17:00:01 <evrardjp> ok I think we should wrap up.
17:00:19 <evrardjp> thank you everyone for attending the bug triage!
17:00:27 <cloudnull> cheers!
17:00:31 <hwoarang> thank you
17:00:33 <evrardjp> #endmeeting