19:01:16 <clarkb> #startmeeting infra
19:01:17 <openstack> Meeting started Tue Dec 10 19:01:16 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:20 <openstack> The meeting name has been set to 'infra'
19:01:22 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-December/006546.html Our Agenda
19:01:28 <clarkb> #topic Announcements
19:01:52 <clarkb> There will be no meeting December 24 or December 31. That means next week will be our last official meeting of the calendar year
19:02:23 <clarkb> #topic Actions from last meeting
19:02:28 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-12-03-19.01.txt minutes from last meeting
19:02:44 <clarkb> ianw: have we created AFS volumes for static.o.o replacement?
19:02:58 <ianw> yes, mostly ... the docs sites all done
19:03:19 <ianw> mnaser is working on jobs i see, and i'm working on bringing up the roles & server to publish them
19:03:40 <ianw> tarballs and releases are still something we need to think about
19:04:01 <clarkb> that was where we ran into the 50GB fs problem on rax test nodes right?
19:04:07 <clarkb> (the role and server)
19:04:31 <diablo_rojo> o/
19:04:35 <fungi> what's the complication with the releases site?
19:04:59 <fungi> i thought it was statically-generated content like the other sites
19:05:34 <ianw> fungi: no complication, as such, just need to look at the details of the jobs and how they are going to publish to afs
19:06:08 <fungi> ahh, okay
19:07:23 <clarkb> ok we've got an agenda item to go in depth for this topic in a bit too
19:07:30 <clarkb> #topic Priority Efforts
19:07:35 <clarkb> #topic Update Config Management
19:07:54 <clarkb> Mordred reports being unwell and hasn't made much progress here
19:08:02 <clarkb> remember to cook your turkeys to 165F
19:08:41 <clarkb> #topic OpenDev
19:08:49 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-December/006537.html Governance email
19:08:58 <clarkb> The governance email has gone out. Good feedback so far
19:09:40 <clarkb> I've made one revision to the proposal already and will prbably make at least one more to make things a bit clearer
19:09:45 <clarkb> if you've got feedback I'm happy to hear it
19:12:46 <clarkb> (sorry distracted by zuul upgrade fallout)
19:13:03 <clarkb> The other opendev item I wanted to call out is that the gitea upgrade does not seem to have fixed tonyb's nova repo fetching problem
19:13:29 <fungi> but at least we were able to rule that out
19:13:29 <clarkb> however the gitea upgrade doesn't seem to have had any other reported issues and looking at cacti graphs gitea 1.10 results in fewer tcp connections
19:13:43 <ianw> on the plus side, nobody else has reported anything?
19:14:37 <clarkb> ianw: not that I have seen
19:15:58 <fungi> yeah, seems to have gone very smoothly
19:16:06 <fungi> thanks to mordred in absentia!
19:16:10 <clarkb> ianw: at this point I think we are probably going to have to dig in with tracing and do our best to see where gitea fails to close those connections?
19:17:12 <ianw> clarkb: i think we know that git upload-pack is dying, at the bottom layer
19:17:22 <ianw> but then there's so much ontop of that ...
19:17:58 <clarkb> and we know that gitea doesn't close the tcp connection to the client (whcih is why the command hangs and doesn't do anything)
19:18:17 <clarkb> I guess if the connection was closed the fetch would still fail so we need to fix the dying upload-pack to properly solve the issue
19:20:59 <clarkb> Anything else on Opendev or should we continue?
19:22:10 <clarkb> #topic General Topics
19:22:24 <clarkb> fungi: anything new on the wiki upgrade?
19:22:37 <fungi> negatory
19:23:23 <clarkb> Next is the static.o.o replacement update
19:23:44 <clarkb> we got a short update earlier. ianw anything else causing problems for the replacement server or was that filesystem thing the only problem?
19:24:36 <ianw> i think it's going ok ... https://review.opendev.org/#/c/698128/ is the change in question that was not setting a small cache for all afs clients in the gate tests
19:24:54 <ianw> #link https://review.opendev.org/#/c/697587/
19:25:12 <ianw> is the static.opendev.org role ... i still want to iterate on that some more, better testinfra
19:25:28 <ianw> but initially, everything should be in place to switch governance.openstack.org to that
19:25:50 <ianw> we should be able to test it before we switch DNS, once the server is up
19:27:35 <clarkb> ++ and since we do dns acme we can even get ssl set up before updating the A/AAAA/CNAME records
19:27:57 <ianw> yep, i will need to setup the acme records before the server goes live
19:28:13 <ianw> with that process proven out, i think the other sites will be easy to switch too
19:28:26 <clarkb> sounds like good progress. Thank you
19:30:05 <clarkb> The nodepool dib item was on the agenda but yseterday ianw said we don't need to go over it in the meeting
19:30:09 <clarkb> ianw: ^ is that still the case?
19:30:54 <ianw> i don't think so ... we have progress merging the changes that build sibling projects with python-builder
19:31:22 <ianw> that is now all working, and the nodepool-builder container can build an image
19:31:53 <ianw> nodepool-launcher hasn't managed boot it in the functional test yet, but so i'd say we're half-way :)
19:33:19 <clarkb> ok
19:33:34 <clarkb> The last thing on the agenda is to point this list of services out again
19:33:38 <clarkb> #link https://etherpad.openstack.org/infra-service-list
19:33:52 <clarkb> as mentioned this isn't urgent but I'd like to pick up on some of that "spring cleaning" in the new year
19:35:01 <clarkb> and that is it for the scheduled agenda
19:35:08 <clarkb> I think others have been distracted by soe of the zuul fallout.
19:35:12 <clarkb> #topic Open Discussion
19:35:21 <clarkb> I'll open it up for anything we missed or went past too quickyl
19:35:41 <ianw> i wanted to bring up dib gate testing
19:35:48 <ianw> this only occurred yesterday afternoon
19:36:01 <ianw> #link https://review.opendev.org/#/c/698136/
19:36:21 <ianw> because devstack has dropped xenial, we're left without anywhere to run functional test for suse-minimal jobs
19:36:27 <ianw> bionic does not have zypper on the host
19:37:06 <ianw> two options here are to pin the suse jobs to devstack train -- this will require work in the install-devstack role the test uses to install branches
19:37:08 <fungi> can an older devstack be used?
19:37:12 <fungi> yeah, that
19:37:22 <fungi> got it
19:37:25 <clarkb> this is where mordred's container bootstrap would help too
19:37:36 <ianw> or, implement some of the other ideas about using containers as minimal images, etc
19:37:38 <clarkb> then we could use a suse container to get zypper and not care what platform we run dib on
19:37:57 <clarkb> I expect that the container bootstrap work will take time and not be a drop in replacement for a while
19:38:13 <clarkb> probably best to use train devstack while we work on the container bootstrapping?
19:38:37 <ianw> right, so yeah, for immediate "gate is broken" purposes, i'm going to have to put it into experimental
19:39:14 <corvus> i reckon a branch option for install-devstack could be generally useful.  esp since most public clouds don't run master.
19:39:27 <ianw> i'm more than happy to expedite review, but i'm not sure how much time i have to focus on fixing it
19:40:11 <corvus> if we have to triage dev/review time, i think that the container bootstrap is the best long-term investment
19:41:00 <corvus> if we have "extra" time, then adding branch support for install-devstack is also a good investment, but not as important as the first
19:41:18 <corvus> (but, obivously, something we can do relatively quickly and can do it first)
19:41:22 <ianw> that's something i'd definitely like to be testing under a nodepool+containers functional test, as we might be looked at nested containers in that case
19:41:39 <corvus> ianw: yep, there be dragons
19:43:30 <ianw> on a somewhat related note, we'll also have to think about when dib can go python3 only
19:43:50 <clarkb> it can probably go ahead and do that today?
19:44:05 <clarkb> at least for nodepool users it is already ypthon3 only
19:44:12 <ianw> the problem is the branchless nature, and so it will basically kill any centos7 users
19:44:49 <ianw> #link https://review.opendev.org/#/c/697211
19:45:04 <ianw> is an example of the work-around we'll have to do to keep python2
19:45:44 <clarkb> ianw: maybe put out a warning and then switch in a few months then?
19:45:52 <clarkb> give people time to go to centos 8 or similar
19:46:58 <ianw> yep, i think so, anyway, just a heads-up on that
19:49:12 <clarkb> I wanted to ask if there was any interest in a less formal activity with the team/group? One idea I had was everyone playing a few rounds of geoguessr (though that requires online registration with an email address and use of google maps)
19:50:36 <fungi> i feel like i wandered into a parallel universe now
19:51:16 <fungi> (or missed a transition in the discussion)
19:51:17 <clarkb> fungi: I also compiled mangband but I couldn't get the curses version to build and the sdl version doesn't allow me to move my character :)
19:51:24 <clarkb> sorry the last discussion seemed to have tapered off
19:51:39 <clarkb> and was wondering if anyone was interested in end of year "office party" except we are remote so what do
19:51:52 <fungi> got it
19:52:21 <fungi> i suppose geoguessr won't work with osm
19:52:37 <clarkb> geoguessr came to mind because its a browser game that tests geographical knowledge and a bunch of us travel a bit
19:52:42 <zbr> or maybe a pose on rdr2?
19:52:44 <clarkb> fungi: ya I think gmaps is built into their web app
19:52:55 <clarkb> zbr: I expect we might be the only two with rdr2 :)
19:53:06 <Shrews> I hated rdr2
19:53:16 <clarkb> I only just started playing it sunday night
19:53:32 <zbr> i loved rdr2 single player, online sucks.
19:54:11 <clarkb> fungi: basically you get a streetview that you can navigate around but have to determine where you are on a world map only from what you see in the streetview
19:54:23 <clarkb> fungi: and osm doesn't do streetview as far as I know
19:54:41 <zbr> as long nobody propose a candycrush competition we should be safe
19:54:46 <fungi> oh, i see. i'd never heard of geoguessr before
19:55:10 <zbr> me neither, i am now looking at geoguessr for the first time
19:55:53 <clarkb> well let me know. If a hour of geoguessr interests people I can pay for the pro version so we can share common maps together in challenges
19:56:06 <clarkb> or feel free to suggest alternatives.
19:56:15 <clarkb> I also tried supertuxracer but my gpu isn't good enoguh for it
19:56:38 <ianw> how is your gpu good enough for rdr2 but not tuxracer?
19:56:40 <corvus> i just did an anonymous trial round of geoguessr, and guessed new zealand for tokelau.  only 2500km off, but it drew a funny picture
19:56:46 <clarkb> ianw: ps4 for rdr2
19:57:17 <corvus> https://imgur.com/a/wr0LPP6
19:57:25 <ianw> ahh!  i have it on pc ... though I'd have to kick my kid off :)
19:57:29 <zbr> i am on ps4, posse /dev/null -- id "ssbarnea"
19:57:34 <clarkb> corvus: nice
19:58:10 <clarkb> We'll call that the meeting then
19:58:12 <clarkb> #endmeeting