16:00:29 <mnaser> #startmeeting openstack_ansible_meeting
16:00:30 <openstack> Meeting started Tue Jan 29 16:00:29 2019 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:33 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:00:34 <mnaser> #topic rollcall
16:00:35 <mnaser> o/
16:00:48 <hwoarang> o/
16:00:56 <evrardjp> o/
16:01:13 <prometheanfire> o/
16:01:35 <mnaser> (sorry for the past 2, i meant to send an email asking if someone could run it, only 2 weeks off :)
16:02:15 <mnaser> not much attendance
16:02:18 <mnaser> #topic last week highlights
16:02:28 <mnaser> section seems empty, is anyone around to share anything in specific?
16:02:52 <evrardjp> not really
16:03:01 <evrardjp> maybe jrosser or odyssey4me
16:03:18 <prometheanfire> sure :D
16:03:24 <odyssey4me> apologies - I'm stuck in another meeting
16:03:26 <mnaser> i see some gentoo patches finally ;)
16:03:43 <prometheanfire> https://review.openstack.org/#/q/topic:add-gentoo-support+status:open gentoo stuff is working, though we need systemd-241 to finally be released
16:03:45 <evrardjp> don't say it's a highlight of last week: p
16:03:55 <prometheanfire> of course it is, for me :P
16:04:03 <evrardjp> :)
16:04:33 <mnaser> hehe, gentoo is an interesting deployment target
16:04:35 <evrardjp> prometheanfire: if this isn;t a done deal yet, should we speak about that during open discussion?
16:04:39 <prometheanfire> once the dib change merges and dib release is made (I don't think os-infra builds from master) and then the gentoo image is rebiult osa-tests should pass
16:04:46 <evrardjp> mnaser: so is tumbleweed ? :D
16:04:55 <hwoarang> gentoo is always a highlight
16:04:56 <prometheanfire> evrardjp: sure
16:04:59 <jrosser> we need to get the tempest and nova stuff unstuck, but there seems to be progress on that today
16:05:06 <mnaser> evrardjp: you sold out! :P
16:05:16 <evrardjp> mnaser: :)
16:05:26 <mnaser> i was hoping to try and dive through, i know centos-7 is been bad
16:05:33 <mnaser> and as part of that was just trying to rip out the container bits
16:05:57 <evrardjp> I thought of comparing how much it would take me to build an arch linux OSA thing. Probably faster than doing it on gentoo :p but I will stop the flamebait there
16:06:05 <mnaser> but anyways, our triage list has grown big but i feel like everyone gets bored and disappears in triage :)
16:06:18 <evrardjp> that's kinda true
16:06:24 <mnaser> so i'm proposing a short open discussion portion where we can talk about this stuff now, then we can do bug triage with whoever survives hah
16:06:25 <evrardjp> it's sad
16:06:33 <evrardjp> yeah that sounds fair
16:06:45 <evrardjp> what about organising a bug killing day?
16:07:07 <mnaser> evrardjp: sounds like a good idea, i'll try to gather up and see what everyone's availabilties seem like over the ML
16:07:08 <evrardjp> I haven't done one in the last cycles, but I used to run one.
16:07:15 <evrardjp> thanks
16:07:33 <mnaser> jrosser: i see you had policy-in-code stuff in open discusison, was that from last week or meant for todays?
16:07:47 <jrosser> it was for last week but we were time out
16:08:06 <mnaser> i assume it's removing all the hard coded stuff we ship in our roles
16:08:07 <jrosser> just really for someone who knows the deal there to update on what still needs to be done
16:08:46 <chandankumar> odyssey4me: mnaser https://review.openstack.org/#/c/633732/ needs merges unblocks centos gates
16:08:46 <mnaser> there is a list of all projects that have moved to policy in code
16:09:15 <evrardjp> oh yeah I have another topic for open discussion: releasing. Bumping is now automatic, and I have a few patches in to have automatic versioning with setuptools, which should be good enough to not change code anymore. Releases would still require manual intervention to say what/when to tag, until releases CLI is working for us at 100% (stein and above)
16:09:21 <chandankumar> jrosser: it worked
16:09:30 <chandankumar> jrosser: we are good to go now :-)
16:09:47 <chandankumar> thanks to jrosser and slaweq for the gates fixes :-)
16:10:01 <odyssey4me> yeah, I was thinking that perhaps we need to organise a hack day around each milestone and get agreement from our employers to do it
16:10:17 <jrosser> mnaser: there seem to have been a few bugs crop up which felt related to policy stuff
16:10:26 <mnaser> odyssey4me: just sent an email to the ML about that, so that'd be cool :)
16:10:31 <odyssey4me> it's been quite tough to get focused attention, and loads of bugs are just sitting there with no attention
16:10:36 <mnaser> ++
16:11:21 <mnaser> chandankumar: good work on catching that, thank you.
16:11:56 <chandankumar> mnaser: it's a team work, we jrosser odyssey4me and slaweq did it :-)
16:11:59 <mnaser> evrardjp: i like that, simplifying our life is always a good thing.  we're all quite busy
16:13:03 <guilhermesp> now with more time to take a look at the osa bugs we found during some deployments, this week Im focusing on a bunch of PR to submit. One of the focus is related to upgrade jobs https://review.openstack.org/#/c/627782/
16:13:14 <evrardjp> should that series of patches merge, stein will be able to be released fully automatically. The patches can still be backported for simpler releasing in older branches, but not 100% perfect solution there.
16:13:27 <guilhermesp> I'm going to take a look at the failures but me and mnaser agreed that the workspace fix is still not complete https://review.openstack.org/#/c/633549/
16:14:05 <mnaser> right, upgrades has been rough for us, and i'm pretty sure there's a bug with the way we deploy rabbitmq too where a cluster failure results in the cluster not routing anything anymore unless you delete all queues
16:14:16 <mnaser> i've seen this repeatedly over multiple rocky envs, so there's still some clean up and work to do
16:14:19 <odyssey4me> ouch
16:14:36 <mnaser> deleting a vhost isnt enough, you have to delete every single queue, and it just magically starts working again
16:14:58 <mnaser> it's affected us and a few customers. i'm confident it's a confirmed issue by now as it's always been fixed this way. i haven't had time to dig deeper, but yeah.
16:15:22 <mnaser> anyhow, subjects so far: releases, upgrades and hackday.
16:15:54 <mnaser> releases => we will try to use the new tooling that evrardjp worked on and then *IF* someone has time, we could backport i guess
16:16:14 <mnaser> upgrades => guilhermesp is working on it and will continue to iterate, we're so so so close because it's failing in tempest after a full upgrade, so that's great news overall
16:16:27 <mnaser> hackday => i sent an email to ML, so if you can respond to it, that'd be awesome :)
16:16:32 <odyssey4me> yeah, let's see how it goes with stein - then work it back if it all goes well
16:17:00 <odyssey4me> for upgrades, I'm happy to help - although I need to focus back on figuring out the final bits for the python builds
16:17:08 <chandankumar> mnaser: on centos Jobs, we find errors in neutron logs, is there any plan to get rid of that
16:17:28 <chandankumar> in the morning, jrosser and I were discussing about that
16:17:52 <mnaser> odyssey4me: i think your time is well invested in the python build to wrap it up, in the meantime i'll work with guilhermesp to get upgrades done, it should be minor things afaik
16:17:54 <mnaser> chandankumar: do you mind explaining more about that?
16:18:39 <chandankumar> mnaser: grabbing the logs
16:20:34 <chandankumar> mnaser: http://logs.openstack.org/32/633732/2/check/openstack-ansible-functional-centos-7/a8cb2f1/logs/openstack/openstack1/neutron/neutron-dhcp-agent.log.txt.gz#_2019-01-29_15_09_31_526
16:20:52 <mnaser> chandankumar: thats probably because the service goes up before we setup the mq's
16:21:02 <odyssey4me> yeah, it'd be nice to sort that out
16:21:18 <odyssey4me> it should be a straightforward fix - just re-ordering some tasks
16:21:31 <chandankumar> mnaser: http://logs.openstack.org/32/633732/2/check/openstack-ansible-functional-centos-7/a8cb2f1/logs/openstack/openstack1/neutron/neutron-l3-agent.log.txt.gz#_2019-01-29_15_09_30_544
16:21:54 <chandankumar> we fixed libvirt import error issues
16:22:17 <chandankumar> mnaser: on tripleo side, we have a role named collect-logs to dump all errors in a single file
16:22:31 <chandankumar> mnaser: I will check with wes tomorrow how we can use it here
16:23:43 <mnaser> oh that's super awesome.  yes, let's share tooling. chandankumar
16:24:05 <mnaser> i have a subject -- evrardjp brought this up before but i think we should move to office hours instead of an actual meeting
16:24:13 <chandankumar> mnaser: odyssey4me something like this http://logs.openstack.org/85/633185/8/check/tripleo-ci-centos-7-standalone/9c2e95c/logs/undercloud/var/log/extra/errors.txt.gz
16:24:49 <mnaser> if you use a role to collect the logs, we can probably reuse it in the gate together
16:25:03 <chandankumar> https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/collect-logs
16:25:27 <chandankumar> there was a plan to move it to a seperate project but stalled due to other priorities
16:25:35 <chandankumar> I will check with team tomorrow and let you know
16:26:06 <mnaser> ok cool, it might be pretty beneficial in our gates too
16:26:10 <mnaser> i mean like, in all of openstack
16:28:02 <mnaser> so, thoughts about office hours instead of meetings?
16:28:23 <jrosser> i would be concerned that the bug triage gets even more out of hand - how would we handle that?
16:28:58 <jrosser> imho it's quite a good way of socialising whats broken and how folks are using our stuff
16:29:22 <odyssey4me> what's the difference between the two?
16:29:38 <odyssey4me> (office hours vs meetings)
16:29:56 <mnaser> jrosser: office hours is just a time where we try to all be available to discuss things (rather than async reaching each other), without a specific agenda, just a time where we're all there
16:30:03 <chandankumar> office hours ~= meeting without predefined agenda
16:30:10 <mnaser> the bug triage, i'm hoping that we can do some sort of bug smash thing every here and there.
16:30:22 <mnaser> the difficult part is that it ends up being 1 or 2 people doing most of the triage
16:30:27 <odyssey4me> well, we kinda have office hours daily during the crossover time between UK and US
16:31:02 <ThiagoCMC> I finally have OSA/Rock up and running with Ceph! At least Glance and Cinder are working!  Wheee!
16:31:07 <openstackgerrit> Merged openstack/openstack-ansible stable/pike: Bump SHAs for stable/pike  https://review.openstack.org/633348
16:31:08 <mnaser> ThiagoCMC: w00t
16:31:11 <ThiagoCMC> Trying to boot a VM now
16:31:15 <ThiagoCMC> I'm so happy!
16:31:17 <ThiagoCMC> :-D
16:31:17 <odyssey4me> I would rather try and do a bug triage/fix team rotation than let it slip to happening once every so often.
16:31:23 <jrosser> are we struggling for people to attend the meeting due to $dayjob pressure?
16:31:53 * redrobot sneaks in through the back
16:32:01 <mnaser> jrosser: i'm not sure.  i don't have much of an explanation. but i think it's largely a time constraint
16:32:14 <mnaser> i think its late in EU timezone, and conflicts with a lot of other meeting timeslots
16:32:32 <mnaser> i often see people mention they're inbetween meetings (and that's fine, i understand people need to get their jobs done), but yeah
16:32:37 <odyssey4me> I unfortunately have two meetings at the same time today - this one and my internal team meeting.
16:34:40 <mnaser> right, i'm all for keeping doing bug triage, but it ends up being a subset of folks that do it.  we can either look into a rotation, or maybe we can come up with another time where we have more resources/people to help do it
16:36:28 <jrosser> perhaps we should look at some bugs?
16:36:42 <mnaser> anyhow, we can defer this to next week and see how this weeks bug triage goes :)
16:36:50 <mnaser> #topic bug triage
16:36:56 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1813660
16:36:57 <openstack> Launchpad bug 1813660 in openstack-ansible "Upgrade from Pike to Queens skips setup-hosts when running neutron on bare metal" [Undecided,New] - Assigned to Bjoern Teipel (bjoern-teipel)
16:37:16 <mnaser> looks like that's already assigned
16:37:33 <jrosser> there may even be a patch for that
16:37:50 <mnaser> yeah, i'm trying to search under that name :p
16:37:57 <mnaser> https://review.openstack.org/#/q/owner:%22Bjoern+Teipel+%253Cbjoern.teipel%2540rackspace.com%253E%22 i don't think so
16:38:24 <guilhermesp> worth to ask updates for that guy?
16:38:42 <prometheanfire> guilhermesp: he's a coworker, should I bug him about something specific?
16:38:51 <prometheanfire> #1813660 ?
16:39:15 <mnaser> yep
16:39:16 <mnaser> i mean
16:39:18 <mnaser> reported 19 hours ago
16:39:20 <jrosser> odyssey4me: didnt you have a patch for this?
16:39:57 <prometheanfire> ya, kinda recent
16:40:26 <mnaser> ok so i think we can mark this down as confirmed medium
16:40:29 <mnaser> and we'll have a patch soon :)
16:40:38 <prometheanfire> ya, pinged him
16:41:03 <odyssey4me> jrosser sort-f, I made it work better - then for master I fixed it properly
16:41:17 <mnaser> oh, so fixed?
16:41:40 <odyssey4me> hang a sec
16:41:41 <odyssey4me> the issue there is pike->queens, right?
16:41:47 <guilhermesp> yep odyssey4me
16:42:15 <odyssey4me> ok, I think that bug is relating to the thing I fixed - yes, lemme provide a review
16:42:59 <odyssey4me> hmm: https://review.openstack.org/625898
16:43:17 <odyssey4me> that was rocky - there was a reason I didn't port that back to queens... but I can't remember what that reason is
16:44:17 <odyssey4me> in master I did a bunch more: https://review.openstack.org/624773
16:44:56 <mnaser> so we can safely triage this and figure out fix later? :)
16:45:49 <odyssey4me> yeah, it's valid and already set to medium
16:46:12 <odyssey4me> I'll comment what's already in place for queens & master. Bjoern can then decide what to do about Pike.
16:46:18 <evrardjp> odyssey4me: for once you don't remember? :p
16:46:29 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1813300
16:46:29 <openstack> Launchpad bug 1813300 in openstack-ansible "NFS mount point for Glance is created with wrong permissions" [Undecided,New]
16:47:00 <evrardjp> that rings me a bell ... haven't we changed that already in the past?
16:47:09 <evrardjp> but there is a patch included!
16:47:45 <odyssey4me> Yeah - I feel that this one keeps coming up, and a new patch goes in, and then another one later... and so on.
16:50:53 <chkumar|out> mnaser: if we have time I want to discuss about using https://trunk.rdoproject.org/centos7-master/delorean-deps.repo in OSa for installing dependencies not maintained around openstack ecosystem
16:51:21 <chkumar|out> mnaser: I was checking the openstack-ansible-tests code on nodepool test file but no clue how to use it
16:51:47 <chkumar|out> mnaser: http://codesearch.openstack.org/?q=delorean-deps.repo&i=nope&files=&repos=
16:51:55 <chkumar|out> mnaser: it is used in POI and tripleo
16:52:06 <chkumar|out> mnaser: can we use it here also?
16:52:19 <odyssey4me> chkumar|out I think we already do?
16:52:39 <chkumar|out> odyssey4me: we only use delorean.repo only
16:53:02 <odyssey4me> oh, I see
16:53:29 <odyssey4me> would this repo be used in production at all?
16:53:43 <chkumar|out> odyssey4me: https://github.com/openstack/openstack-ansible-tests/blob/401fc3d5cdef09f99470f20256c2ecd7e36925fa/common-tasks/test-set-nodepool-vars.yml#L49
16:53:53 <chkumar|out> odyssey4me: in downstream, we import packages from same
16:54:27 <mnaser> confirmed/medium for the nfs bug, i asked Juri if it's possible to work with them to get them to push it to gerrit
16:54:28 <chkumar|out> odyssey4me: it is maintained here https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml
16:55:05 <mnaser> chkumar|out: i'd be in favour, using delorean deps was very helpful and made our gate usually quite stable in poi times (it also helped crossgate with rdo)
16:55:34 <chkumar|out> mnaser: I need some pointers and I can make the changes in openstack-ansible-tests
16:55:50 <mnaser> chkumar|out: we can discuss post meeting if you're not "out" :)
16:56:22 <chkumar|out> mnaser: may be tomorrow, I can ping you in evening from my time zone
16:56:31 <mnaser> chkumar|out: great!
16:57:37 <mnaser> we're running close to time, maybe we can get one more triage in
16:57:44 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1813187
16:57:45 <openstack> Launchpad bug 1813187 in openstack-ansible "CentOS tempest test_server_basic_ops failure" [Undecided,New]
16:57:59 <mnaser> oh, that was resolved by the patch listed above
16:58:39 <mnaser> done
16:58:48 <mnaser> #link https://bugs.launchpad.net/openstack-ansible/+bug/1813149
16:58:49 <openstack> Launchpad bug 1813149 in openstack-ansible "Missing git respo: https://github.com/ceph/ansible-ceph-defaults" [Undecided,New]
16:59:09 <prometheanfire> cjloader: ^?
16:59:24 <odyssey4me> ja, that's all fixed
16:59:33 <mnaser> did we release since
16:59:39 <mnaser> looks like 16.0.24 is the tag the user used
16:59:54 <cjloader> yes was fixed
17:00:06 <odyssey4me> ocata: https://review.openstack.org/632182 & pike: https://review.openstack.org/632142
17:00:24 <mnaser> nice work cjloader
17:00:26 <odyssey4me> no release based on that yet, I think evrardjp did the release requests earlier today
17:00:36 <mnaser> cool, ill update the bug
17:01:46 <mnaser> ok, we're over time, but it looks like we don't need any bug triage cause everything just works ;) haha.
17:02:03 <mnaser> thanks everyone, and please please take time to respond to the hackday ML post on openstack-discuss
17:02:06 <mnaser> <3
17:02:07 <mnaser> #endmeeting