16:02:37 #startmeeting openstack_ansible_meeting 16:02:38 Meeting started Tue Jun 5 16:02:37 2018 UTC and is due to finish in 60 minutes. The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:02:40 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:02:40 bad evrardjp!!!! 16:02:42 The meeting name has been set to 'openstack_ansible_meeting' 16:02:54 o/ 16:02:58 o/ 16:03:10 #topic Rollcall and last week highlights 16:03:22 o/ 16:03:22 welcome everyone 16:03:29 o/ 16:03:33 in the meantime ppl join, please have a look at what happened last week 16:03:36 evrardjp has pointed that master is broken, and help is welcomed to fix things. Zuul status should be monitored by more than one person. 16:03:39 evrardjp has posted on the ML a change about the way we do releases 16:03:49 cloudnull and ansmith asked for the inclusion of roles os_zun and qdrouterd into the OSA umbrella. 16:04:03 hwoarang is still working on distro packages, but is also working on opensuse15 support: https://review.openstack.org/#/q/topic:osa-add-leap-150+(status:open+OR+status:merged) 16:04:08 evrardjp reminded that this week is M2, and we are lagging behind the schedule: https://docs.openstack.org/openstack-ansible/latest/contributor/periodic-work.html#development-cycle-checklist . 16:04:13 andymccr is working on the bump of the ceph-ansible 16:04:17 evrardjp mentioned the rabbitmq version we are using is now EOL 16:04:27 odyssey4me has been working on https://review.openstack.org/#/q/topic:move-db-setup-into-roles+(status:open+OR+status:merged) 16:04:29 it should work fine but i think the gate is broken on master atm? 16:04:38 yup, see first point. 16:04:52 do we want to make some action items out of those? 16:05:08 mnaser: good question 16:05:18 so, there is also something new that wasn't listed there 16:05:29 mnaser: suggested to have a list of OSA etherpads 16:05:35 which I did on the wiki 16:05:47 https://wiki.openstack.org/wiki/OpenStackAnsible 16:06:05 there is nothing interesting on the wiki except that list. :p 16:06:08 sweet. i like that 16:06:12 * mnaser bookmarks 16:06:19 as you can see, https://etherpad.openstack.org/p/osa-rocky-m2-deprecations 16:06:33 if you want to help, just pick a thing you want to do 16:07:15 o/ 16:07:17 evrardjp: i think more critically we need to unbreak master for any of this stuff to go through, is anyone currently working on that? 16:07:23 for the rest of today's items, it's all about reviews. Help odyssey4me 's on the topic above, help hwoarang , and fix gates :) 16:07:24 o/ 16:07:33 if not, i can volunteer to speed up the unbreakage 16:07:41 I think the gate breakage came from ansmith :) 16:07:49 A few patches went in 16:08:06 I didn't get the chance to look at current state. I will do so today. 16:08:25 #action mnaser investigate gate and push up fixes 16:08:33 oh i'm not a chair i guess 16:08:44 #action mnaser investigate gate :p 16:08:58 Not sure it changes anything, it's recorded whether you are chair or not. 16:09:01 anyway 16:09:12 cloudnull: did you generate the bug list for today? 16:09:28 I did not. 16:09:37 I was just going to go off of the open issues 16:09:38 would it be okay if i added a few action items to be reviewed next week? 16:10:05 hahah 16:10:05 evrardjp can you make mnaser a chair ? 16:10:07 ok 16:10:17 #action andymccr bump ceph role variable defaults 16:10:20 how does that work? 16:10:31 i should be okay without chair according to the meetbot docs :p 16:10:40 yeah 16:10:45 #action evrardjp bump galera versions 16:10:51 #action cloudnull bump rabbitmq versions 16:10:59 (that way we can follow up on action items from last meeting) 16:11:18 and action anyone to do the role/upstream deprecation? 16:11:21 that's not gonna work :p 16:11:25 do we have anyone that can volunteer for the other 3 items? 16:12:18 upstream deprecations.. i think we're doing okay with that, i saw a few patches pushed up but i'm not sure there is much to do there 16:12:18 what other 3 times? i lost track 16:12:22 same :) 16:12:24 see https://etherpad.openstack.org/p/osa-rocky-m2-deprecations 16:12:27 ^ 16:12:36 tasks #1, #2 and #3 16:12:54 impossible for me 16:13:07 yeah it's massive work. 16:13:11 I'll take one 16:13:14 cool 16:13:42 d34dh0r53: which one will you be picking up, so we can have an action item? :) 16:13:56 I'll take #2 16:13:59 d34dh0r53: want to tag team on something? i don't think i can commit to something on my own 16:14:07 mattt: sure 16:14:12 cool 16:14:18 mattt: if you want, upstream deprecation isn't that difficult, you can get away with deploying an aio 16:14:19 look at us, teaming and all. 16:14:24 and grep the logs for 'deprecation' 16:14:35 mnaser: yeah ok, i can give it a shot 16:14:35 yup 16:14:55 then read the release notes, just in case :) 16:15:10 which is a pile of work :( 16:15:11 i guess for OSA deprecation that's a bunch of grep TODO across the source code, but yeah, it's a good exercise 16:15:12 hehe 16:15:31 #action d34dh0r53 finish up OSA deprecations 16:15:41 #action mattt work on cleaning up upstream deprecations 16:15:58 (don't feel like you will be held accountable, this is just to keep track of things, if things come up, let us know and we can work together) 16:16:13 can someone help me understand what is 'refresh static elements' :X 16:16:16 awesome 16:16:57 mnaser: yeah, so in the roles we have static versions 16:17:04 outside all of those things 16:17:10 just bump the defaults 16:17:18 to latest standards 16:17:19 yeah i don't think i can read release notes for dozens of projects, but i can certainly go through the reported deprecations 16:17:39 mattt: That was a joke, I did that in the past it's insanely long. 16:17:57 evrardjp: i'm still not following.. what roles for example that have static versions, maybe things like etcd and so? 16:18:03 evrardjp: well tbh it needs to be done! 16:18:55 Merged openstack/openstack-ansible-pip_install stable/queens: Ensure that pip.conf templates variables properly https://review.openstack.org/563597 16:19:11 mnaser: just taking rally as an example: https://github.com/openstack/openstack-ansible-os_rally/blob/master/defaults/main.yml#L61-L67 16:19:25 this is staticly set, it's not a dynamically generated list of versions 16:19:33 we do not bump it on a regular basis 16:19:37 it needs to be manually bumped. 16:19:55 oh i see 16:20:05 all the roles have things like that. 16:20:15 we just need to go through all the roles, see what's going on 16:20:17 Merged openstack/openstack-ansible-os_nova stable/ocata: Updated spice-html5 git repository URL https://review.openstack.org/572414 16:20:34 let me rephrase 16:20:38 all the role COULD have that 16:20:43 roles* 16:20:50 so we need to make sure we are up to date. 16:20:52 okay well, if there's no one volunteering, i can pick it up, i can use codesearch and search for *_pip_packages 16:21:07 it's not only that, but yeah it's a good start. 16:21:58 I read the keystone role, everything seems fine. I did the first! woot 16:22:00 :p 16:22:00 #action mnaser bump static versions in roles 16:22:02 anyway 16:22:07 let's move to bug triage 16:22:21 one more topic before if that's okay? 16:22:28 unless we'll have open discussion after 16:22:35 we have open discussion after 16:22:38 ++ ack 16:22:51 the first section is just to remember what's going usually :p 16:23:10 Kevin Carter (cloudnull) proposed openstack/openstack-ansible-rabbitmq_server master: Update the version of erlang to support rabbitmq https://review.openstack.org/572196 16:23:11 but I won't prevent the enthusiasm on fixing things, that's great! 16:23:13 anyway 16:23:18 #topic bugtriage 16:23:21 #link https://bugs.launchpad.net/openstack-ansible/+bug/1771489 16:23:22 Launchpad bug 1771489 in openstack-ansible "lxd.service starts before lxc-dnsmasq DHCP process is available" [Undecided,New] 16:24:12 I am not sure lxd.service is the right title name 16:24:32 it looks like the user already pushed up a patch for the fix 16:24:46 yeah 16:24:53 and it does make sense that we start up lxc-dnsmasq before lxc 16:24:54 it sounds like a low-hanging-fruit and valid 16:25:09 that said, taht seems like an ubuntu bug then 16:25:22 andymccr: we override the lxc-dnsmasq service afaik 16:25:29 https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/templates/lxc-dnsmasq-systemd-init.j2 16:25:39 hmm 16:26:00 andymccr: we are generating ours 16:26:01 yeah 16:26:04 thanks mnaser :) 16:26:27 maybe we should stop doing so 16:26:51 it'd be nice to move towards overrides 16:27:29 anyways, valid and confirmed and i'll take it and push up a patch 16:27:50 what has changed though - i mean that would be a pretty critical bug so between some release and now this has caused an issue 16:28:01 anyone new should step in there, IMO 16:28:02 majors comment in the linked task seems to suggest there is a reason for it 16:28:08 it's an easy to fix issue 16:28:10 andymccr: i'm going to guess non deterministic behaviour 16:28:54 so out of the loads of deploys that have happened on 16.04 only this bug has run into this? 16:29:13 i guess im saying it seems like somebody went out of their way to disable the service, that isnt something you would do randomly 16:29:19 im not saying its necessarily correct or incorrect 16:29:28 andymccr: maybe on a reboot? 16:29:46 We are disabling lxc-net and implementing things. 16:29:57 mnaser: yes I believe so, when I read the bug 16:30:06 we probably don't reboot our deploys much 16:30:10 (s)he tries to stop and restart containers, see how it goes 16:30:36 i see no harm in us adding 'Before=lxc.service' .. because it makes sense that it should be the case 16:30:42 let me see what dependencies lxc-net has 16:30:48 (or if someone can look into the unit file) 16:31:06 * mnaser doesnt have access to an ubuntu deploy, anyone? 16:31:32 https://github.com/lxc/lxc/blob/master/config/init/systemd/lxc-net.service.in 16:31:41 After=network-online.target\nBefore=lxc.service 16:31:54 so we're doing our own thing and not necessarily doing it The Right Way(tm) 16:32:45 Agreed. 16:33:18 next? 16:33:22 sure 16:33:36 marking confirmed high 16:33:43 and low-hanging-fruit 16:34:06 #link https://bugs.launchpad.net/openstack-ansible/+bug/1771760 16:34:06 Launchpad bug 1771760 in openstack-ansible "Ensure log rotation with rsyslog is properly done in the integrated repo automatically" [Undecided,New] 16:34:47 ok this is to add tests. 16:35:28 confirmed low? it cannot be wishlist, as we are basically not proving anything 16:35:51 yes 16:35:53 i agree 16:35:57 sure 16:36:32 ok next 16:36:35 #link https://bugs.launchpad.net/openstack-ansible/+bug/1771987 16:36:36 Launchpad bug 1771987 in Pike "lvm is reporting problems, not sending heartbeat. Service will appear "down"" [Undecided,New] 16:37:26 Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: Switch lxc-dnsmasq to start before lxc https://review.openstack.org/572459 16:37:43 hmm 16:37:46 needs more info 16:37:49 Logan V proposed openstack/openstack-ansible-rabbitmq_server master: Variable node address https://review.openstack.org/572397 16:37:52 we need cinder-volume logs 16:37:52 that looks like a configuration issue 16:38:02 but yeah 16:38:11 could you ask for more details in the bug mnaser? 16:38:15 I mark it as incomplete 16:38:18 ++ 16:38:56 done 16:39:06 #link https://bugs.launchpad.net/openstack-ansible/+bug/1772690 16:39:07 Launchpad bug 1772690 in openstack-ansible "Failed to find required executable rabbitmqctl" [Undecided,New] 16:39:34 that's a long name for a host 16:40:06 hmm 16:40:22 ok so 16:40:25 mysql failed 16:40:38 Merged openstack/openstack-ansible-repo_build master: Updated spice-html5 git repository URL https://review.openstack.org/572430 16:40:43 and then (s)he ran the next play, without fixing the first one 16:41:08 I will post a message asking for more logs 16:41:10 next 16:41:12 also um 16:41:19 #link https://bugs.launchpad.net/openstack-ansible/+bug/1772690 16:41:20 Launchpad bug 1772690 in openstack-ansible "Failed to find required executable rabbitmqctl" [Undecided,New] 16:41:20 i suggest setenforce 0 (hide from mhayden) 16:41:25 haha 16:41:34 we just need more details 16:41:47 evrardjp: if you'll post a message, ask for 'getenforce' output 16:41:53 because the selinux stuff in centos isn't so well done (another reason for packaging, yay) 16:44:16 I see 16:44:18 I asked 16:44:26 I didn't suggest setenforce 0. 16:44:34 I am respectful of mhayden legacy :p 16:44:38 :p 16:44:48 OH BURN 16:44:49 ok next 16:44:56 mhayden: haha. 16:45:04 had to drop out of a meeting to defend SELinux ;) 16:45:04 mhayden: go back to your hiding! 16:45:06 oops 16:45:08 DOUBLE BURN. 16:45:08 :) 16:45:10 o/ folks 16:45:15 oh crap i messed up an OSA meeting unintentionally 16:45:17 * mhayden is a winner :/ 16:45:22 hahah 16:45:27 * mhayden departs 16:45:29 ok next 16:45:32 #link https://bugs.launchpad.net/openstack-ansible/+bug/1772772 16:45:33 Launchpad bug 1772772 in openstack-ansible "When using federation sso_callback_template.html is missing" [Undecided,New] 16:46:24 * mnaser has no idea about keystone federated auth 16:46:26 odyssey4me: you arrive right on time 16:46:36 ah yes, that looks valid 16:46:40 I'll fix that 16:46:52 yup I thought so too 16:46:58 looks like confirmed low 16:47:03 ok for everyone 16:47:05 ? 16:47:43 I'd say high actually 16:47:48 * mnaser doesn't know enough about this to make a call so will defer 16:47:51 it's broken, and there's no workaround 16:49:05 https://docs.openstack.org/openstack-ansible/latest/user/source-overrides/index.html#overriding-other-upstream-projects-source-code 16:49:09 I've also assigned the related bug mentioned to myself 16:49:11 it's not broken 16:49:18 it's not properly overriden 16:49:29 expectations are broken, that's true 16:49:35 but the code itself works, 16:49:40 erm, are we talking about the same thing? 16:49:48 oh wait 16:49:51 I moved to the next bug 16:49:55 :D 16:49:59 the conditional is using the idp settings as a conditional, instead of the sp config - it's broken 16:50:02 my bad! 16:50:06 yeah that's broken 16:50:11 I thought it was a done deal 16:50:13 next 16:50:16 #link https://bugs.launchpad.net/openstack-ansible/+bug/1773124 16:50:17 Launchpad bug 1773124 in openstack-ansible "plugin py_pkgs.py does not adhere to normal ansible variable precedence rules" [Undecided,New] 16:50:22 lol, ok 16:51:07 erm, haha 16:51:17 the order precedence comes from a var, lemme get that 16:51:33 it's not a problem of var precedence 16:51:43 https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/repo_all.yml#L71-L74 16:51:46 it's a problem that the var was overriden with just the git branch 16:52:03 yep 16:52:05 so precedence was completely ignored 16:52:11 I can answer it if you like 16:52:27 you can answer, I already posted something 16:52:43 a more english and less spartian answer would be great. 16:52:50 anyway 16:52:51 ok, can do 16:52:52 next 16:53:11 #link https://bugs.launchpad.net/openstack-ansible/+bug/1774658 16:53:12 Launchpad bug 1774658 in openstack-ansible "Hash sum mismatch when fetching percona-xtrabackup_2.3.10-1.xenial_amd64.deb" [Undecided,New] 16:53:20 that's mine! heheh 16:53:37 could be transient, are you still running into it guilhermesp ? 16:53:37 woot 16:53:42 yep 16:53:51 ..yay 16:53:51 this morning the same thing 16:53:52 I can try deploying a 17.0.4 16:54:03 guilhermesp: you don't use a proxy, right? 16:54:37 looks like 2.3.10 was released a few days ago on the day of the breakage 16:54:41 well, i assume it's by deafult in an all in one, right? 16:54:59 maybe we should pin xtrabackup version? 16:55:07 I opened a bug in percona community, not sure if was the right place https://www.percona.com/forums/questions-discussions/percona-xtrabackup/51614-hash-sum-mismatch-percona-xtrabackup-percona-xtrabackup_2-3-10-1-xenial_amd64-deb 16:55:15 guilhermesp: nice! 16:55:18 cool 16:55:23 and if the description was that clear, but any answers till now 16:55:39 typo openstacak-ansible :D 16:55:50 hahahaha sorry for that 16:56:01 haha no worries, I am just toying with you :D 16:56:11 guilhermesp: hmm, can you try apt-get clean 16:56:11 I have a nervous typing sometimes 16:56:11 that's cool you did that thing 16:56:17 apt-get update and then try running it again? 16:56:19 mnaser: I did 16:56:24 apparently these messed up things could be cached 16:56:25 in the meantime I am deploying 17.0.4 16:56:25 oh excellent 16:56:28 we can yell at them then :D 16:56:39 so I can tell I have the same issue 16:56:45 same thing. I can clean my vm and try to deploy again, but will the the 3ยบ time 16:56:46 in the meantime, we should bump the source maybe 16:56:59 to a new version 16:57:02 evrardjp: cool, maybe you can set to confirmed? also, why wouldn't this be breaking in our gate then? 16:57:24 2.3.10 is the latest release published by percona 16:57:25 I will assign that to me, I will mark it as confirmed when I see the bug appearing, fine for everyone? 16:57:39 fine by me, but i'm just curious why our gate is not hitting the same bug 16:57:59 ok! So, going to my office, I was just wainting the bug to be discussed and to give you a heads up 16:58:02 that's why I am confirming/infirming manually :) 16:58:06 maybe guilhermesp you have a transparent proxy mucking things up? 16:58:07 guilhermesp: that's very cool 16:58:15 please do so anytime :) 16:58:25 we love reporters that show up :) 16:58:36 * mnaser sees edu.br so it could be a possibility, we'll let you know and work to fix it for ya (hopefully) 16:59:05 yeah, I was thinking it might be best for us to pin those packages too 16:59:06 mnaser: thank you! I will right back in 5 or 10 minutes 16:59:11 ok we are running out of time 16:59:24 odyssey4me: I think it's a good idea to pin them indeed. 16:59:26 we pin mariadb, so we should also pin everything relating to it 16:59:31 yup. 16:59:33 ++ 16:59:43 now that mariadb is better, I thought of unpinning it though. 16:59:55 but that's a longer story. 17:00:01 I think the pins are best to leave in place, and update from time to time 17:00:06 yeah 17:00:09 * mnaser is in favour of keeping things pinned so tagged releases don't ever break for users 17:00:10 at the end I think it's the most reliable 17:00:18 mnaser: that's not true 17:00:26 mnaser: about the proxy, my instance runs a apt-cacher proxy, could be that? 17:00:28 because the upstream source can disappear. 17:00:45 it's more likely with an old file, than a complete source 17:00:46 right, but that's less likely to happen than a new release that breaks stuff 17:00:49 but it's still better. 17:00:52 yeah 17:00:54 agreed. 17:01:07 guilhermesp: it could be possible that your apt cache is stale or has problems, maybe good to try and reset it! 17:01:09 anyways, next bug? 17:01:16 I think we are out of time 17:01:32 #topic next meeting chair 17:01:43 anyone wants to run the meeting? 17:02:06 i can do that 17:02:11 * hwoarang checks calendar 17:02:37 ok 17:02:42 yep i can do that 17:02:49 else it was mgariepy that was picked by my script. 17:02:54 cool, thanks hwoarang ! 17:03:02 thank you everyone 17:03:04 #endmeeting