19:00:29 <ianw> #startmeeting infra
19:00:30 <openstack> Meeting started Tue Jun 18 19:00:29 2019 UTC and is due to finish in 60 minutes.  The chair is ianw. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:33 <openstack> The meeting name has been set to 'infra'
19:01:02 <ianw> the agenda for today is @
19:01:03 <ianw> #link http://lists.openstack.org/pipermail/openstack-infra/2019-June/006403.html
19:01:15 <ianw> #topic Announcements
19:01:53 <ianw> clarkb obviously doing something more exciting than running this meeting :)
19:02:12 <ianw> i don't think we have anything else to announce
19:02:45 <fungi> oh, one thing
19:02:54 <fungi> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html Long overdue cleanups of Zuulv2 compatibility base configs
19:03:03 <mordred> \o/
19:03:19 <dmsimard> zuul-cloner will no longer be a thing ? :)
19:03:32 <fungi> #info zuul-cloner shim and bindep fallback are being removed from non-legacy abstract jobs on Monday, June 24
19:04:02 <fungi> gonna follow up to that announcement on the ml with a reminder
19:04:32 <fungi> and put on my flame retardant underpants when i get up monday
19:04:32 <ianw> ++
19:05:37 <ianw> #topic Actions from last meeting
19:05:49 <ianw> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.html
19:06:06 <ianw> corvus: work with jroll to get openstack github org self managed
19:06:17 <corvus> derp
19:06:22 <corvus> sorry
19:06:29 <corvus> did not happen :(
19:06:38 * mordred also supposed to make opendevadmin user - will do after meeting
19:06:47 <ianw> should we re-action or is it on todo list?
19:07:10 <ianw> mordred: set up github opendevadmin account for the opendevorg org and zuul app
19:07:30 <corvus> let's re-action it, not only as a safetly valve in case i forget again, but also, it seems like something i should report back on regardless
19:07:45 <ianw> #action corvus  work with jroll to get openstack github org self managed
19:07:53 <mordred> same with admin account
19:08:09 <ianw> #action mordred  set up github opendevadmin account for the opendevorg org and zuul app
19:08:40 <ianw> ok that was it for that
19:08:57 <ianw> #topic Priority Efforts
19:09:18 <ianw> #topic Update Config Management
19:09:48 <ianw> can talk about specific mirror stuff later
19:10:45 <fungi> did we have any efforts underway on the config management side?
19:10:58 <fungi> i guess it's down mostly to cleanup now
19:11:09 <ianw> i will have to check out the builders -- note i merged https://review.opendev.org/#/c/665014/ to stop building the control plane images
19:11:32 <fungi> and some of this is intertwined with the trusty upgrades as well as the mirror updates you noted
19:11:37 <ianw> last night i noticed that nb01 had about 250 threads which i don't think was right, errors trying to delete in-use images
19:11:57 <Shrews> that definitely does not sound right
19:12:07 <mordred> it doesn't sound ideal at the very least
19:12:39 <ianw> anyway will investigate more, i did py-bt dump all them but it wasn't pointing to a smoking gun inside nodepool-builder/openstacksdk
19:13:37 <ianw> anything else?
19:14:02 <ianw> #topic OpenDev
19:14:38 <dmsimard> For the thread about ara 1.0 on openstack-infra
19:15:31 <dmsimard> some pieces of the puzzle are falling in place -- we have an implementation of the sqlite middleware working for 1.0 that will land soon
19:15:38 <ianw> #undo
19:15:39 <mordred> woot
19:15:39 <openstack> Removing item from minutes: #topic OpenDev
19:15:56 <dmsimard> I somehow read that as open floor
19:16:02 <dmsimard> sorry for sidetracking
19:16:34 <ianw> oh, cool.  yeah, as noted by corvus i guess with swift logs we still have an issue, but at least getting status quo to 1.0 will be good
19:16:44 <dmsimard> there are some question marks around swift, yeah
19:17:17 <dmsimard> basically
19:17:39 <dmsimard> the equivalent middleware in 1.0 will give us arbitrary API endpoints to every job results
19:17:53 <corvus> there are question marks around middleware too.... our plan is to stop running it and rely entirely on statig generation
19:17:57 <dmsimard> and ara-web is able to connect to any of those arbitrary endpoints
19:18:37 <corvus> that plan is ready to go at a moments notice (for example, the next time our filesystem is corrupted)
19:19:08 <fungi> we're overdue for one of those events
19:19:34 <dmsimard> as far as full static generation goes, I have a half working proof of concept but we need some changes under the hood for it to work properly
19:19:58 <dmsimard> like, I have the actual html of the pages as if you would be browsing them but it still attempts to connect to the API
19:20:14 <dmsimard> this is tracked here: https://github.com/ansible-community/ara-web/issues/11
19:20:57 <ianw> #link  https://github.com/ansible-community/ara-web/issues/11
19:21:00 <corvus> cool -- i think i'd suggest that we not spend too much effort on doing ara 1.0 middleware in opendev given that we might switch to static generation at any time
19:21:10 <fungi> seconded
19:21:35 <fungi> we can always pin ara<1 until we switch to object storage
19:21:42 <dmsimard> works for me
19:23:00 <ianw> i suggested on list that maybe this is suitable for a single service that we sent the results to in post in the job, but i'll read through the static generation stuff to understand more too
19:23:26 <ianw> i didn't think react was amenable to that, but if it is, great!
19:24:06 <ianw> one other little thing is that i discussed having a vexxhost backup server with mnaser, and we agreed it was ok, so that's on my todo to bring up, and get to comments left in the review for ansible backups left last week
19:24:26 <ianw> ok we should move on
19:24:33 <ianw> #topic OpenDev
19:24:49 <ianw> i guess this was mostly the github mgmt bits which are in flight
19:25:26 <ianw> any other particular updates?
19:25:33 <fungi> on that note
19:25:53 <fungi> dmsimard brought up in #zuul that we still have stale mirrors of things in the openstack-infra org on gh
19:26:01 <fungi> things which didn't move to the openstack org
19:26:24 <fungi> worth deciding (not necessarily in-meeting) what we want to do with those
19:26:29 <mordred> I vote that we push up a retired commit with a pointer to opendev.org and then archive each of them
19:26:30 <corvus> i'd guess there's quite a lot of openstack-infra/ repos in that state
19:26:36 <pabelanger> ianw: where can one see the github mgmt bits? I haven't really been following along as much as I would like
19:26:39 <mordred> but also agree, we don't have to agree to a plan of action here
19:26:44 <fungi> whether we push readme files to all of them or delete them and update the org description or something else entirely
19:27:38 * mordred can probably handle doing the retire/archive before being AFK again, if people are comfortable with that approach
19:27:46 <ianw> pabelanger: sorry, yes very abstract, i was referring to the discussion started last week @ http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-06-11-19.01.log.html#l-105
19:28:08 <pabelanger> ianw: thanks
19:28:11 <corvus> mordred: +1
19:28:11 <fungi> i'm most ni favor of whatever someone else volunteers to do. i don't have strong feelings about our lingering repository presence on github
19:28:33 <mordred> cool. I'll take a stab at $something - and will either accomplish it or give up
19:28:40 <fungi> other than agreeing that stale git repositories there is probably the least desirable option
19:28:45 <mordred> fungi: ++
19:29:09 <ianw> ok, that sounds like
19:29:15 <fungi> thanks mordred! and thanks dmsimard for bringing it up!
19:29:34 <ianw> #action mordred look at stale github.com/openstack-infra repos and come up with retirement/removal plan/changes
19:29:38 <ianw> ok?
19:29:45 <mordred> where are we keeping the rename mapping file from the Great Rename? I imagine it'll make automating the retire commits easier
19:29:47 <fungi> (or jdi)
19:29:49 <mordred> ianw: ++
19:30:21 <fungi> there were multiple phases of renames
19:30:34 <fungi> may be easier to build a list by querying the gh api org object
19:30:55 <fungi> anything in the old openstack-infra org, retire
19:31:27 <corvus> well, we have a file from the most recent renames
19:31:29 <mordred> well - I was mostly thinking of getting the openstack-infra/zuul -> zuul/zuul but openstack-infra/puppet-zuul -> opendev/puppet-zuul mapping
19:31:39 <corvus> and i bet we have the file from the great rename too
19:31:44 <corvus> those 2 should be all that's needed
19:31:49 <mordred> I agree - it should be applied to all things in gh openstack-infra org
19:31:53 <fungi> oh, right, that file. i had already forgotten we put it all in there back to the migration batch
19:31:55 <mordred> corvus: yah
19:32:07 <pabelanger> mordred: I might have some code to share about adding 'archived' support to managed-projects, that is what we are using in zuul.a.c, and works well :)
19:32:22 <fungi> #link https://opendev.org/opendev/project-config/src/branch/master/renames
19:32:25 <mordred> pabelanger: sweet! I'll totally borrow that from you
19:32:37 <pabelanger> https://github.com/ansible/project-config/commit/bb2d4d18ed79ef68e6f8b22b08cecd0a14f6faf5
19:32:39 <mordred> fungi: AWESOME - thank you
19:32:49 <corvus> "unmanage-projects"
19:32:50 <pabelanger> happy to push that up
19:33:06 <mordred> pabelanger: that's nice and easy to copy
19:33:20 <mordred> pabelanger: I'll likely write a one-off script for this, so that's good enough for me
19:34:31 <pabelanger> ack
19:34:33 <mordred> (it will almost certainly be a very ugly script)
19:35:04 <ianw> #topic Storyboard
19:35:23 <fungi> storyboard-api releases on pypi are a thing now
19:35:38 <fungi> we released 1.0.0 last week and 1.0.1 over the weekend
19:35:56 <fungi> or actually yesterday morning my time
19:36:16 <mordred> neat
19:36:31 <fungi> over the weekend, logins were broken because of the distname change from storyboard to storyboard-api
19:36:53 <fungi> for whatever reason, the auth codepath includes asking pbr for some dist info
19:37:05 <mordred> o_O
19:37:06 <fungi> and we missed updatnig the string being passed into it
19:37:44 <fungi> that along with a very annoying but infrequent db lock contention bug were fixed in 1.0.1
19:38:11 <ianw> fungi: i think take a don't ask, don't tell policy with that :)
19:38:32 <ianw> (the pbr stuff)
19:38:37 <fungi> indeed
19:38:55 <fungi> i mean, we can pickaxe the history, but it likely will take us back to commit #1
19:39:11 <mordred> it's probably my fault
19:39:18 <fungi> also the karbor team changed their minds and decided they did want their lp bugs imported after all, so i did that yesterday
19:39:55 <ianw> ok, anything else?
19:39:58 <fungi> i don't know of any other pressing news, but maybe SotK or diablo_rojo are around?
19:40:10 <fungi> if not, we can surely move on
19:40:39 <ianw> #topic General Topics
19:40:48 <ianw> trusty upgrade
19:40:57 <ianw> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup
19:41:12 <fungi> still poking at the wiki-dev02 xenial build as tmie allows
19:41:21 <ianw> and then i think it's really just status.openstack.org ?
19:41:27 <mordred> I continue to not actually do the status upgrade
19:41:39 <mordred> but I'm totally enjoying continuing to not actually do it
19:41:47 <diablo_rojo> I got nothing pressing
19:42:15 <fungi> the current snag is that something like 2/3 of our ~40 extensions and our half a dozen skins can't be cloned by the vcsrepo module and their .git ends up being a file instead of a directory, with just a single line like "gitdir: ../../.git/modules/extensions/Renameuser"
19:42:25 <fungi> not sure if that means it's some submodule magic or what
19:42:59 <fungi> (does that pattern look familiar to anyone?)
19:43:38 <mordred> fungi: no - but I think your reasoning that leads you to suspect submodules seems likely
19:44:03 <fungi> we're cloning from https://gerrit.wikimedia.org/r/p/mediawiki/${type}s/${name}.git
19:44:09 <ianw> no but i seem to remember having some issue with vcsrepo that required a much later version ... trying to remember what that was ... i think with ask.o.o ...
19:44:13 <fungi> #link https://opendev.org/opendev/puppet-mediawiki/src/branch/master/manifests/extension.pp#L13
19:44:59 <fungi> https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git is also getting us a clone like that
19:45:21 <fungi> but if i got clone it manually i'm getting an actual repo
19:45:28 <fungi> s/got/git/
19:45:29 <mordred> oh my.
19:45:38 <mordred> got git?
19:45:44 <fungi> anyway, not going to solve it right now, bit anyone who has ideas, i'm all ears
19:46:00 <fungi> ianw's vcsrepo version concern is definitely worth digging into
19:46:20 <fungi> we can probably move on as there's more on the agenda
19:46:42 <ianw> ok
19:46:58 <ianw> yeah i wanted to update on the opendev.org mirror situation
19:47:11 <ianw> several things in flight there
19:47:21 <ianw> in short, if we could review
19:47:32 <ianw> #link https://review.opendev.org/#/q/status:open+branch:master+topic:kafs
19:48:00 <fungi> starred
19:48:40 <ianw> basically the openafs version shipping with bionic (1.8.0~pre5) is buggy; so we either use our own built later versions, or something else like kafs
19:49:20 <ianw> kafs, for those not involved, is the inbuilt kernel support for afs.  so we don't have to build any modules with dkms etc etc like with openafs.  the only issue is you need a very recent kernel
19:49:44 <ianw> like, top of tree master kernel (5.2-rcX)
19:49:49 <fungi> from what we can tell so far, one so new it's not released yet ;)
19:50:18 <ianw> so that is obviously not without issue either
19:50:45 <corvus> it is interesting to be on the bleeding edge of afs development.
19:50:54 <ianw> we have mirror.iad.opendev.org running this, currently setup by hand, but the reviews in the above topic make it all fairly nicely ansible controlled
19:51:10 <fungi> i'm just happy that there is still a bleednig edge for afs development
19:51:24 <ianw> we also have mirror.dfw.opendev.org running upstream 1.8.3 openafs taken from packages i put in our ppa
19:51:49 <ianw> afaict, that has been running fine for several days now
19:53:09 * mordred is supportive of both options
19:53:24 <ianw> i feel like at this point we would want to roll out the servers with openafs 1.8.3 -- devil you know --(changes to do that in the stack above too), but opt in a few to kafs?
19:53:47 <ianw> i would like to bring up a very remote to rax kafs node, to see what latency and network weirdness does to it
19:53:47 <fungi> it would be nice if ubuntu could at least replace their prerelease afs package with a more updated one that isn't thoroughly broken
19:53:55 <fungi> but my hopes are not high there
19:54:06 <ianw> fungi: i have filed an issue ... if we know a way to push on that i'm happy to
19:54:07 <fungi> (for bionic i mean)
19:54:15 <corvus> the shutdown of the kafs server is suspicious, but it still could be a random unrelated incident.
19:54:15 <ianw> there's a link in the reviews
19:54:37 <ianw> need to look into that ... i had a little background monitoring going on too after discussion in openafs last night
19:54:45 <fungi> right, i'm far from blaming kafs on an instance in rackspace spontaneously (from our perspective) going into shutdown state
19:54:46 <corvus> it's also probably easier to maintain our own openafs packages rather than a kernel
19:55:22 <fungi> this is true
19:55:35 <fungi> though the kafs implementation has an awful lot of cool factor to it
19:55:45 <fungi> and that sure counts for something
19:55:49 <ianw> yeah, with 1.8.3 seeming to be stable now, i think that should be our default.  but we've got great CI, and with those changes we can keep both paths in flight
19:56:10 <ianw> (secretly, i'm hoping upstream notice just how cool our CI is and want to get more involved than just occasional advice ...)
19:56:19 <corvus> ianw: i like both of your suggestions -- maybe start thinking about resolving the current situation with openafs and continue testing kafs in more situations, since it may be good for us long-term
19:56:42 <fungi> i too see kafs as worthwhile investment in the future
19:56:54 <fungi> even if we don't end up relying on it right now
19:57:28 <ianw> ++ seems like we're all on about the same page :)
19:57:44 <ianw> #topic Open discussion
19:57:52 <ianw> any last minute review requests, etc?
19:59:06 <fungi> #link https://review.opendev.org/655783 Don't hide Zuul CI comments
19:59:12 <fungi> might be nice to come to a decision around that
20:00:04 <fungi> jhesketh had some good suggestions
20:00:21 <ianw> hide all but last seems like a nice intermediate, if the hacking required isn't too crazy
20:01:41 <ianw> ok, thanks everyone, back to #openstack-infra for any other concerns!
20:01:45 <ianw> #endmeeting