19:01:16 <clarkb> #startmeeting infra
19:01:17 <openstack> Meeting started Tue Oct 15 19:01:16 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:20 <openstack> The meeting name has been set to 'infra'
19:01:26 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-October/006493.html Our Agenda
19:01:33 <clarkb> #topic Announcements
19:01:48 <clarkb> The OpenStack release is happening this week (~tomorrow)
19:02:09 <clarkb> I believe the only potential need for infra help is requeuing any of the openstack release teams' tagging jobs
19:02:20 <clarkb> if they happen to timeout (there was a change to increase the timeout on them to avoid that in the first place)
19:02:46 <clarkb> but I guess keep an eye out for any issues they run into that we can help with
19:02:51 <corvus> tagging job > 30 min?
19:03:06 <clarkb> corvus: ya it serially ran through every project created a tag for it and pushed it to gerrit
19:03:21 <clarkb> and apparently when you do that over several hundred projects you take longer than 30 minutes (this happened with stein)
19:03:31 <corvus> i see
19:03:39 <clarkb> the job won't retag and repush things so it can be requeued and it basically skips ahead to where it left off safely
19:04:04 <fungi> i don't know that it happened with stein
19:04:15 <clarkb> oh it happend at some point
19:04:16 <fungi> it happened at stein and train rc1
19:04:19 <clarkb> ah
19:04:26 <fungi> because those also involve branch creation on every project
19:04:31 <fungi> and pushing new changes to each of them
19:04:52 <mordred> branch creation is slightly more expensive I guess?
19:05:07 <fungi> with all the associated change proposals it is
19:05:20 <mordred> ah - yeah - that makes sense
19:05:43 <fungi> we're just being extra cautious. i plan to be at the computer by 11:30z and the button mashing commences at 12:00z i believe
19:06:16 <clarkb> I fully endorse everything being done by the time I wake up :)
19:06:19 <fungi> i expect it will be mostly done, including the follow-on jobs triggered by the tag pushes, by 13:00z
19:06:45 <clarkb> other things to consider: we probably want to avoid zuul restarts tomorrow (at least until fungi gives us the all clear)
19:06:56 <fungi> unless something goes sideways, and even then that leaves us hours until the osf has the usual press release planned and the associated press embargoes lift
19:07:12 <fungi> which i think is usually around 15:00z or thereabouts
19:07:56 * diablo_rojo sneaks in late
19:08:16 <clarkb> but ya I don't anticipate problems. We've fixed or are fixing a bunch of our known infra issues recently and of those I don't think they would really affect the openstack release
19:08:24 <fungi> in release-related announcements, i also created the ussuri signing key a few weeks ago:
19:08:28 <fungi> #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xbba3b1e67a7303dd1769d34595bf2e4d09004514&fingerprint=on OpenStack Infra (Ussuri Cycle) artifact signing key
19:08:43 <clarkb> (things like NM on red hat distros and the debian buster mirror)
19:09:15 <fungi> #link https://docs.openstack.org/infra/system-config/signing.html#attestation infra signing key attestation procedure
19:09:32 <fungi> i'll push up changes to switch the train key out for that in a week or so
19:10:19 <clarkb> Anything else to announce?
19:10:32 * fungi knows of nothing else
19:10:57 <clarkb> #topic Actions from last meeting
19:11:03 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-08-19.01.txt minutes from last meeting
19:11:20 <clarkb> I started drafting up a thing for opendev governance
19:11:28 <clarkb> #link https://etherpad.openstack.org/p/rCF58JvzbF OpenDev governance email draft
19:11:45 <clarkb> I think it is still a bit rough and I've tried to bring up a few of the things I expect peopel to question/bring up
19:12:22 <clarkb> in particular how an OpenDev PTL is chosen, whether we need tie breaker rules on the proposed council and whether we want the "else" group to be covered there too
19:12:37 <clarkb> I think all of that is open for discussion at this point so please feel free to leave comments on the etherpad
19:13:31 <corvus> clarkb: thanks!  after collecting etherpad comments, what do you think is next?
19:14:31 <clarkb> corvus: I think draft a final version that we can all be happy with as a group, then take that to the TC as a "look this is what we'd like to do and we'd like your input and go ahead to do this formally"
19:14:45 <corvus> kk
19:14:47 <clarkb> I don't expect that to present any problems but would be good to make it all official like
19:16:34 <clarkb> we should probably also ask the other groups that will be part of the council for input
19:16:44 <clarkb> since we will be asking them to participate
19:16:59 <clarkb> fungi: ^ that may help answer the question re kata
19:17:05 <clarkb> "does this even interest you?"
19:17:12 <fungi> makes sense
19:18:00 <clarkb> Probably best to dive into this on the etherpad once we've collected more thoughts and opinions
19:18:10 <clarkb> we can come back to this at the ned of the meeting if we have time too
19:18:18 <clarkb> #topic Priority Efforts
19:18:22 <clarkb> #topic OpenDev
19:18:30 <clarkb> Any other opendev topics to bring up?
19:19:25 <clarkb> Sounds like no
19:19:27 <fungi> what's our current opendev focus?
19:19:43 <clarkb> for me it has become the governance topic
19:19:54 <fungi> okay, cool. just making sure i know where we are in the transition
19:20:06 <clarkb> Based on complaints I think our git hosting is fairly stable at this point other than the performance issue on large repos
19:20:07 <fungi> and i guess the gerrit upgrade
19:20:14 <clarkb> which is a known upstream issue with people working on it
19:20:14 <fungi> and the zuul log storage switch
19:20:31 <corvus> mordred's is gerrit upgrade
19:20:43 <fungi> right, i meant more collectively
19:21:04 <corvus> and ianw is driving static retirement
19:21:10 <fungi> like what do we want to complete before we start moving more services into opendev
19:21:24 <fungi> those all sound like good things to get behind us first
19:21:31 <corvus> more services like?
19:21:35 <corvus> like adding services, or renaming?
19:21:38 <fungi> etherpad et cetera
19:21:58 <fungi> well, it's never been clear to me if something is officially opendev while it's still on an openstack.org domain
19:22:02 <corvus> i wouldn't put up any blockers to moving anything over at this point
19:22:06 <ianw> (on static; yes very distracted with glean + centos8 this week, but will get back to updating that spec as discussed last week)
19:22:13 <mordred> yah- other than available human time
19:22:18 <clarkb> ya I think we can start shifting services like etherpad as soon as people have time to do them
19:22:26 <mordred> clarkb: ++
19:22:43 <fungi> got it. i know we were pushing back on that when the focus was still on stabilizing the gerrit/gitea migration
19:22:46 <corvus> i know i said "let's not move etherpad" in the past, because i think we needed all hands on deck for gerrit migration, but i don't feel like any current effort is like that
19:22:48 <clarkb> it might make moving services slightly easier if we prioritize the dns proposal implementation
19:22:55 <corvus> clarkb: ++
19:22:56 <fungi> awesome. thanks!
19:23:16 <fungi> and yes, i agree the dns/ssl plan is a high priority for that
19:23:18 <clarkb> speaking of the dns proposal I did bring it up with osf staff and heard no objections
19:23:28 <clarkb> jbryce did say it would be this week or next before he could really read and digest it though
19:23:29 <corvus> groovy
19:23:30 <fungi> not that there was really anything to object to
19:24:00 <fungi> and there's some possible interest in moving the openstack.org domain into our dns hosting, just not sure how much and on what timeline yet
19:24:22 <fungi> but i think the etherpadded plan is a simple interim solution even so
19:24:48 <corvus> yeah, it won't produce any roadblocks to future work like that
19:26:36 <fungi> thanks, that covers my questions. we just hadn't done a "where are we now?" on this in a few months
19:26:57 <clarkb> To summarize, mordred is working on improvements to gerrit config management using containers to help enable a big gerrit upgrade, ianw is working on static.openstack.org cleanup, clarkb is working on the governance side of things. Others should feel free to pick up other service "moves" which might be made easier if we focus on the dns proposal first
19:27:07 <clarkb> https://etherpad.openstack.org/p/openstack-org-dns is the dns proposal
19:27:57 <clarkb> #topic Update Config Management
19:28:21 <clarkb> mordred: anything new on gerrit container images?
19:28:38 <mordred> there's some patches up to update to bazel 1.0
19:28:46 <mordred> which are required to land the latest changes
19:28:59 <mordred> that opened up a can of worms with puppet - which I think is finally unwound
19:29:30 <mordred> https://review.opendev.org/#/c/688523/ and friends
19:30:02 <clarkb> ya that prompted some cleanup of the centos-7 puppet jobs we were running
19:30:20 <mordred> since I missed the last few meetings - the current status is that review-dev is running on the container image - so we know that bit works - the next bit is finishing the patch to actually be able to do that via config management
19:32:04 <mordred> at some point it might be good to rethink our gerrit server-side hook scripts that use jeepyb to be zuul jobs - as they currently require us to add jeepyb into the gerrit image itself - but it's not urgent
19:32:25 <corvus> ++
19:32:43 <mordred> also - we've got some patches up to add master-based images with checks plugin enabled that we can use for zuul testing - and eventually potentially be more ready for rollouts as we go forward
19:34:55 <clarkb> That sounds good. Anyone else have updates related to config management?
19:36:01 <clarkb> Lets move on
19:36:06 <clarkb> #topic Storyboard
19:36:17 <clarkb> fungi: diablo_rojo anything new to report here?
19:36:51 <fungi> we moved the storyboard "weekly" irc meeting to 14:00z thursdays
19:37:23 <fungi> in hopes we have critical mass for it to be closer to weekly in frequency again
19:38:13 <fungi> nothing new from me on the team management automation front yet. i've started fleshing out the module last week but haven't gotten to pushing anything into gerrit
19:38:23 <fungi> focus has mostly been elsewhere
19:38:51 <fungi> though in the sb meeting last week we got consensus on the direction i'm taking it
19:40:58 <clarkb> and that direction is using the api to manage group membership in storyboard instead of the database
19:41:13 <fungi> yep
19:41:25 <fungi> via ansible
19:41:26 <clarkb> great. Anything else?
19:41:42 <fungi> nothing i'm aware of
19:42:06 <clarkb> #topic General Topics
19:42:21 <clarkb> fungi: I noticed there were some wiki changes today, I assume we should all go and review those?
19:42:30 <fungi> i've imported fresh mysqldump and image tree copy from the production wiki onto wiki-dev, and then discovered mediawiki has a number of unpackaged dependencies which need installing via a php tool called "composer"
19:42:46 <fungi> #link https://review.opendev.org/#/q/project:opendev/puppet-mediawiki+is:open+topic:xenial-upgrades
19:42:49 <mordred> "neat"
19:43:18 <fungi> yeah, i see one has a beaker failure, need to dig into it
19:43:52 <fungi> also i've basically done that stuff manually on wiki-dev just to see what breaks next
19:44:19 <fungi> it's now giving me no output instead of an error in the browser, so time to go hunting in logs
19:44:38 <clarkb> progress at least
19:44:41 <fungi> yep
19:44:41 <mordred> fungi: your beaker failed is the thig I just fixed for gerrit
19:44:47 <fungi> oh, neat
19:44:53 <fungi> thanks mordred!
19:45:01 <fungi> will look after the meeting
19:45:06 <clarkb> ianw for static.o.o I think we can go ahead and update the spec with the dns proposal changes involved
19:45:07 <mordred> so once https://review.opendev.org/#/c/688555/ lands (has +A) - we can recheck that patch and it shoiuld be fine
19:45:27 <clarkb> ianw: other than waiting for new patchset to review on that is there anything we can do to help re static.o.o?
19:46:06 <ianw> clarkb: yes, will do for sure
19:46:31 <ianw> sorry, at this point i think it's stalled on us getting ducks in a row on spec
19:46:45 <ianw> /s/on/in/
19:46:55 <fungi> just needs some duck sauce then
19:47:16 <clarkb> no worries, just checking
19:47:24 <clarkb> Next up I wanted to bring up CentOS 8.
19:47:39 <clarkb> I think ianw managed to track down the network problems we've had on red hat distros (including 8) and those should now be fixed
19:47:54 <clarkb> please keep an eye out for odd behavior on centos and fedora on limestone and fortnebula in particular but we should be good now
19:48:09 <clarkb> zbr and ianw have updates to fix mirror management on centos 8 in zuul-jobs
19:48:19 <ianw> yep, and last night the configure-mirror changes went in, so devstack can start
19:48:19 <mordred> well done ianw!
19:48:27 <clarkb> I want to say enough of that has now merged that we should be able to run jobs on centos 8 now and have their actual workloads run
19:48:27 <ianw> ergo all the setup roles are working
19:48:32 <fungi> yay!
19:48:35 <clarkb> excellent thank you for confirming
19:48:39 <mordred> that was quite the digging for that one
19:48:55 <ianw> yep, i plan to announce the images to the list, along with some of the caveats about it being python3 only (by default) today
19:49:22 <zbr> tomorrow I am going to start enabling new jobs
19:49:22 <clarkb> ++ thank you for getting that done
19:49:39 <clarkb> also apparently we've managed to build centos 8 cloud images before upstream so some individuals have come to us looking for them :)
19:50:01 <fungi> hilarious
19:50:04 <fungi> but also sad
19:50:12 <ianw> yes, i need to sync with maybe karinbar there ... there are kickstarts, and their infra is building them but not publishing for some reason
19:50:24 <zbr> ianw: not sure if advertising is really needed,... i prefer to slowly add jobs, so we have time to fix the broken bits.
19:50:51 <clarkb> zbr: I think in the past we've not been aggressive enough in advertising updated distros which leads to peopel dragging their feet and we end up not updating as quickly as possible
19:50:52 <ianw> zbr: i'd like people to go in with a heads up that they're different wrt to pip/virtualenv setup to all our other images
19:51:17 <clarkb> I think we should let everyone know that they can (and should) update now that we are happy with it and that will help shake out any additional problems
19:52:11 <clarkb> Last up on the agenda is the PTG
19:52:20 <clarkb> #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 Add your PTG topic ideas here
19:52:45 <clarkb> #link https://www.openstack.org/ptg/#tab_schedule PTG schedule
19:53:01 <clarkb> I'm told I should hear back on my visa tomorrow. I am cautiously optomistic that there will be good news
19:54:22 <clarkb> #topic Open Discussion
19:54:26 <mordred> clarkb: I hope to see you in Shanghai
19:54:30 <clarkb> mordred: me too :)
19:54:41 <clarkb> I'll open it up to further discussion on past topics or new items
19:54:51 <mordred> fwiw - I'm now in Asia until the Summit (in Japan timezone)
19:55:27 <fungi> the debian mirror seems to be updated now, though seeing different build failures around the debian-security mirror now
19:55:32 <ianw> hangin' in the +9
19:55:32 <clarkb> mordred: is there a secret goal of visiting every timezone?
19:55:39 <fungi> i may need to perform some similar reprepro surgery on it
19:55:44 <mordred> clarkb: not very secret :)
19:56:15 <mordred> my intent is to work hours that are non-normal here so I have more overlap with folks - but I'm jetlagged at the moment and have't 100% sorted out what that looks like yet
19:57:15 <corvus> mordred: cool, thanks for keeping us updated
19:57:50 <clarkb> Oh we are removing opensuse 42.3 images from nodepool
19:57:56 <clarkb> thank you to ajaeger and corvus for pushing on that
19:58:12 <clarkb> there is one devstack change we are waiting on before rechecking the nodeset removal to see if that will go in now
19:58:17 <ianw> ++
19:58:22 <ianw> f28 should probably be next
19:58:26 <clarkb> and opensuse 150
19:58:35 <clarkb> but ya agreed
19:58:59 <corvus> i'm guessing we should hold off on the git repo cache move until friday?
19:59:04 <corvus> or at least post-release?
19:59:18 <clarkb> ++
19:59:27 <clarkb> that has the potential to have widespread impact probably best to play it safe
19:59:33 <corvus> i'm basically waiting for clarkb and fungi to tell me it's okay to break the debian-whatever and old suse images...
19:59:44 <corvus> the ones that haven't updated in ages
20:00:09 <clarkb> and we are at time
20:00:09 <corvus> https://review.opendev.org/682935
20:00:24 <clarkb> thank you everyone. Feel free to continue discussion in #openstack-infra
20:00:26 <clarkb> #endmeeting