19:01:35 <clarkb> #startmeeting infra
19:01:36 <openstack> Meeting started Tue Apr 24 19:01:35 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:40 <openstack> The meeting name has been set to 'infra'
19:01:56 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:03 <clarkb> ianw has an apologies entry at the top of the agenda, I don't think ianw is going to be attending the meeting due to holiday in australia but I think that is related to the dib changes realted to initramfs generation?
19:03:12 <cmurphy> o/
19:03:20 <clarkb> I'll assume that is what it is and no apology necessary, we got it sorted out if not fully debugged and are functional again
19:03:27 <clarkb> #topic Announcements
19:03:46 <corvus> i read that as 'apologies for absence'
19:03:59 <clarkb> oh that may be what it is
19:04:05 <clarkb> in which case no apology necessary either :P
19:05:05 * corvus apologizes for absinthe
19:05:16 <clarkb> As for announcements, a TC election has just started. If youa are eligible to vote please don't forget to do so
19:05:42 <clarkb> you should have an email in your inbox with a unique voting url. If you don't have one and expected one you can reach out to the election officials to help debug that
19:06:25 <clarkb> #topic Actions from last meeting
19:06:32 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-04-17-19.01.txt minutes from last meeting
19:06:34 <pabelanger> hello
19:07:09 <clarkb> no actions recorded
19:07:42 <clarkb> #topics Specs approval
19:07:59 <clarkb> #topic Specs approval
19:08:20 <clarkb> #link https://review.openstack.org/563849 Direction for third party CI tooling
19:08:51 <clarkb> this is a spec that ianw pushed up recently that is the output of a previous meeting. I don't think its ready for approval yet but wanted to point it out since we were talking about thsi in previous zuul and infra meetings
19:09:00 <clarkb> if you get a moment review of that would be helpful
19:09:21 <clarkb> #topic Priority Efforts
19:09:34 <clarkb> fungi: Anything new from the land of storyboard?
19:10:00 <fungi> aside from prepping for barbican migration scheduled for friday, nothing new on my end yet
19:10:15 <fungi> #link https://review.openstack.org/564047 Add Barbican to StoryBoard
19:11:05 <clarkb> #topic Modernizing Control Plane Management
19:11:29 <clarkb> cmurphy has done some great work to revitalize the puppet stuff necessary to update. There is a new spec too
19:11:41 <clarkb> or rather an updated spec
19:11:49 <clarkb> #link https://review.openstack.org/#/c/449933/ Upgrade to puppet 4
19:12:09 <clarkb> I haven't seen an updated ansiblification spec nor mordred's container spec yet
19:12:29 <clarkb> pabelanger: mordred ^ it would be helpful to get those up soon as I think at laest on the puppet side we are ready to give it a proper consideration
19:12:56 <cmurphy> also proposed moving mordred's old puppet 4 spec out of the help wanted section https://review.openstack.org/#/c/563592/
19:13:12 <pabelanger> clarkb: ack, will make time tomorrow
19:14:00 <clarkb> if we get those specs all up this week I'll send email to the infra list and point them all out so that we can work on reviewing and reading them prior to next weeks meeting
19:14:14 <clarkb> (we can also use that thread for communicating about this as well)
19:14:35 <clarkb> I'll have to see where mordred is at too to see if that is still planned
19:14:41 <clarkb> (I think he is traveling in italy or something for a pycon)
19:15:06 <clarkb> cmurphy: considering you've been pushing on this anything else worth adding?
19:15:25 <cmurphy> i don't think so
19:15:46 <cmurphy> will probably be able to close out https://review.openstack.org/#/c/563592/ soonish
19:15:52 <clarkb> cool, lets aim for next week having concrete comparison then. Thank you for helping on this
19:15:57 <cmurphy> np
19:16:11 <clarkb> #topic General Topics
19:16:44 <clarkb> Quick status update on pip10 situation. It happened once, then sort of happened again once our images updated late last week
19:17:05 <clarkb> people seem to be aware of the general problem though and have been addressing it in projects as necessary
19:17:38 <fungi> fallout was less serious than i anticipated
19:17:39 <clarkb> This reminds me, that we should probably do a glance over of infra's deployment logs to ensure we aren't hitting it anywhere in our deployments (I can work on doing that)
19:17:47 <corvus> so, other than devstack, is everything using pip10 now?
19:17:54 <corvus> in jobs
19:18:03 <fungi> afaik, yes
19:18:11 <clarkb> its definitely what you get by default on our images
19:18:27 <clarkb> I'm not sure if other deployment testing tooling has had similar problems and if they have pinned
19:18:31 <fungi> if some teams are overriding it in their in-repo job configs, we likely won't find out
19:18:45 <pabelanger> yah, a few hiccups with centos-7, but think they've been all fixed by ianw
19:18:51 <clarkb> #link https://review.openstack.org/#/c/561426 Devstack psutil fix
19:19:03 <clarkb> #link https://review.openstack.org/#/c/562884 devstack workarounds
19:19:13 <clarkb> #link https://review.openstack.org/#/c/562937 Devstack don't install numpy
19:19:22 <clarkb> #link https://review.openstack.org/#/c/561597/9 Devstack use pip10
19:19:38 <corvus> and it seems like ianw is advocating for fixing the 3 things pip10 breaks in devstack and leaving it as-is, rather than clarkb's put-it-all-in-a-venv idea?
19:19:51 <clarkb> corvus: I think for the short term that is the suggestion.
19:20:10 <clarkb> corvus: on my put it all in a venv idea ianw has asked that we get broader consensus since it will affect all the things
19:20:21 <corvus> clarkb: like plugins, etc?
19:20:27 <clarkb> also brought up the idea of running it by the TC so I've added it to the TC's office hours discussion topics list
19:20:28 <clarkb> corvus: exactly
19:20:30 <fungi> longer-term, the venv solution seems like nice future proofind
19:20:32 <fungi> proofing
19:20:52 <fungi> since we don't know when the next dep will get added that conflicts with something distro-installed
19:21:17 <fungi> nor whether we can make the same compromises on other platforms the devstack team might want to extend support for
19:21:25 <clarkb> fungi: or if pip11 will be more aggressive in this upstream position
19:21:30 <fungi> yep
19:21:51 <corvus> clarkb: didn't you have a ml post?
19:21:59 <fungi> there was indeed
19:22:06 <fungi> with basically nobody replynig
19:22:17 <clarkb> ya that came up when I talked to doug briefly about it
19:22:38 <corvus> well, clarkb knows i support it; i didn't think it needed a +1 from me, but i'd be happy to reply with that if it would help
19:23:02 <clarkb> corvus: ya it might be good to have someone other than me explain why they think it would be a good idea too :)
19:23:34 <corvus> what i can't really weigh in on is the impact to plugins; i understand that less.
19:23:34 <clarkb> I think the fear is that we know it will make people's lives more difficult near term and no one wants to have the crwods wrath sent their way
19:23:43 <clarkb> so better to at least make sure we make the right decsions when we do that and point to the reasoning
19:24:23 <pabelanger> +1
19:24:44 <fungi> maybe making the venv codepath optional via a toggle which we warn teams we'll be throwing at the start of the stein cycle?
19:25:00 <clarkb> fungi: ya I have to make it option for grenade support anyways
19:25:03 <fungi> fewer torches and pitchforks maybe
19:25:08 <clarkb> fungi: so maybe thats a good first step, get the support in but leave it off by default
19:25:20 <clarkb> turn it on in some jobs, warn people to test it and flip going forward
19:25:21 <corvus> clarkb: is there a path to support grenade eventually?
19:25:36 <corvus> clarkb: (like, after the toggle is in both sides of a grenade test, grenade can use it?)
19:25:38 <pabelanger> clarkb: we should atleast have some jobs running it, since we did have USE_VENV but was completely broken
19:26:06 <clarkb> corvus: yes, one option is to backport it to all the old devstack and support grenade the same way in all envs. We would merge the feature to all devstack branches first, but not enabled it. Merge usage and enabling of feature to all grenade branches. Then enable it in devstack by default
19:26:12 <clarkb> (this is necessary to avoid force merging)
19:26:18 <clarkb> corvus: exactly
19:26:26 <corvus> clarkb: cool
19:26:42 <clarkb> the other is we roll forward and have feature in master (rocky) today then only use it in rocky -> stein upgrades and so on
19:27:09 <clarkb> I think ^^ leaves us open to greande jobs and older devstack jobs running on stable branches breaking due to reasons fungi pointed out
19:27:20 <clarkb> and its probably best to just get it addressed across the board
19:27:38 <corvus> yeah, though less likely on stable branches
19:27:53 <corvus> either way, the plans are compatible -- it's mostly a matter of how fast we do it :)
19:27:54 <fungi> yeah, pip is one of the few deps we shouldn't really be trying to pin, even on stable branches
19:27:58 <clarkb> corvus: yup
19:28:19 <fungi> but we should pick our battles
19:28:35 <fungi> getting things working safely for master can be first priority
19:28:47 <clarkb> I also think that now that dust is settling and people have had a few days to understand the underlying problem rather than just make jobs work as fast as possible we can better discuss long term direction
19:29:14 <clarkb> (I think the pre release discussions suffered from this problem, we said it was an issue and would break us but until it did it was low priority)
19:30:08 <clarkb> The other two items on the agenda are the Third party CI spec which I broght up earlier. Ianw asking for early feedback on it.
19:30:16 <clarkb> And DIB, Xenial, and rackspace problems
19:30:21 <fungi> i think the warnings we provided ahead of time, even if partly ignored, helped
19:30:43 <clarkb> One of the problems with the pip10 rollout was we had paused xenial image builds because they stopped booting on rax
19:31:11 <clarkb> we traced this back to a change made in dib to rebuild the initramfs at the end of the job which broke udev and glean and system cooperating to produce working networking at least on rax
19:31:18 <pabelanger> yah, still haven't really figured out why glean stopped working
19:31:23 <fungi> that was mostly unfortunate coincidence, but it did end up masking/delaying some issues
19:31:39 <clarkb> That change was reverted in dib but we should likely work to figure that out because it was made to fixing an lvm based image problem
19:31:55 <clarkb> if you have knowledge of the initramfs and udev and glean and systemd your help would be much appreciated
19:33:15 <clarkb> I think I've had to declare bankruptcy on getting normally scheduled work operations done ~twice in the last week or so
19:33:18 <clarkb> its been fun :)
19:33:27 <clarkb> unfortunately the wrong kind of fun
19:34:17 <clarkb> I think that is about it.
19:34:25 <clarkb> #topic open discussion
19:34:34 <clarkb> What did I miss?
19:35:01 <pabelanger> I could use a few more reviews on https://review.openstack.org/#/q/topic:debian-stable+status:open to help finalize the removal of debian-jessie from nodepool. I'll then start the process to clean up AFS mirrors
19:36:09 <corvus> well, you didn't miss this -- but clarkb, jlvillal, and i had a chat earlier and jlvillal is interested in helping on the gerritbot portion of the limnoria spec -- so i've created a feature/limnoria branch of gerritbot, jlvillal will start making a gerritbot limnoria plugin there, and later this week, i'll start on the puppet to install/run limnoria on eavesdrop.
19:36:28 <pabelanger> and mostly a FYI: been pushing on the removal of old jenkins based scripts from bindep: https://review.openstack.org/#/q/topic:bindep.txt+status:open all in the effort to help move bindep.txt in tree for projects. That allows us to then remove bindep-fallback.txt in 2 cycles
19:36:59 <corvus> there's *plenty* of opportunity for parallel work on that spec -- each of our bots should be converted at a minimum, and there's some new plugins to write.
19:37:24 <clarkb> right since the bots work as plugins to the central framework we don't need to produce them serially
19:37:24 <pabelanger> lastly, ubuntu-bionic release is this week. So we should keep an eye out for projects wanting to gate on it.
19:37:37 <clarkb> pabelanger: thursday is official day right?
19:37:48 <pabelanger> clarkb: yah
19:37:52 <clarkb> thankfully bionic is a lot less dramatic of an update for server users I expect
19:37:58 <clarkb> the big chagne is all on the desktop side of the world
19:38:06 <pabelanger> yup, we are in really good shape this time around
19:38:21 <corvus> pabelanger: we're carrying around 'legacy' stuff a bit longer than i'd like, but since devstack/tempest are still evolving a little, i suppose it's good to maintain it for a bit longer. ++
19:38:29 <fungi> #link https://review.openstack.org/557979 work is also progressing on the limesurvey implementation
19:38:45 <fungi> i'm hoping to have time to take the latest patchset for a spin tomorrow
19:38:47 <corvus> maybe we can set a cycle goal for stein or something to remove all legacy jobs
19:39:21 <pabelanger> corvus: yah, I think stein might work.
19:39:35 <fungi> yes, elimination of legacy imported job definitions might make for a nice cross-project goal
19:40:16 <pabelanger> the bindep changes didn't get much push back, and our bindep role was a nice noop, which made it nice and easy!
19:41:40 <clarkb> do we want to have 19 minutes back to eat lunch or have an afternoon snack or find breakfast or eat dinner?
19:41:59 <corvus> or all of the above?
19:42:07 <fungi> i need to heat up the wok for some shrimp fried rice, so would welcome a few spare minutes ;)
19:42:14 <clarkb> mmm breakfast for dinner during afternoon snack time
19:42:33 <clarkb> alright, if there is anything else feel free to find us on the infra mailing list or in #openstack-infra
19:42:34 <fungi> not sure whether it's lunch or dinner, but it'll be tasty either way
19:42:36 <clarkb> thank you everyone
19:42:41 <fungi> thanks clarkb!
19:42:43 <clarkb> #endmeeting