19:01:04 <clarkb> #startmeeting infra
19:01:05 <openstack> Meeting started Tue Apr 23 19:01:04 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <openstack> The meeting name has been set to 'infra'
19:01:18 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-April/006333.html
19:01:32 <diablo_rojo> o/
19:03:17 <clarkb> I think people are fairly preoccupied with all the things today. Myself included so we'll let people trickle in
19:03:44 * mordred trickles
19:04:01 <clarkb> #topic Announcements
19:04:20 <clarkb> The Summit and PTG are next week. That means no infra meeting on the 30th
19:04:35 <clarkb> Looking forward to seeing people that are attending
19:05:22 <clarkb> I believe we have a project update that I need to start putting content together for but I've yet to have a moment to put a schedule together
19:06:04 <clarkb> #topic Actions from last meeting
19:06:14 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-04-16-19.01.txt minutes from last meeting
19:06:30 <clarkb> No real actions recorded other than the biggest gerrit migration we've ever done
19:06:44 <clarkb> so maybe we should just skip ahead a bit and start talking about how that went and what needs attention next
19:06:54 <clarkb> #topic Priority Efforts
19:07:00 <clarkb> #topic OpenDev
19:07:01 <mordred> clarkb: to the best of my knowledge, everythign is working perfectly right? )
19:07:13 <clarkb> mordred: all things considered things are working quite well
19:07:22 <mordred> clarkb: ++
19:07:36 <corvus> is openstack/openstack fixed yet?
19:07:37 <clarkb> fungi and corvus are currently dealing with some fallout due to restarting zuul as part of that migration
19:07:52 <clarkb> corvus: no the chagne to fix it is in check right now
19:07:54 <fungi> not sure it's migration related
19:08:01 <corvus> or, well, it was in check :|
19:08:01 <clarkb> fungi: the restart was migration related
19:08:27 <fungi> oh, the restart from the 16th
19:08:30 <corvus> the openstack/openstack issue is the only critical regression i know about, hopefully that'll be fixed otday
19:08:42 <clarkb> yup we have a fix up for it and I've been trying to shepherd it through
19:08:55 <fungi> yeah, the current restart is not necessarily migration related (seems unlikely but i wouldn't rule it out entirely just yet)
19:09:03 <corvus> we have gotten the feedback that replicating refs/changes and refs/notes would be nice
19:09:05 <clarkb> we've also found there are a non zero number of repos that need to be renamed (either we failed to rename them or renamed them inappropriately)
19:09:44 <corvus> that's a minor regression -- i think it's one we can live with but i think we've learned that's more important than we originally thought
19:10:02 <clarkb> airship/in-a-bottle should be airship/airship-in-a-bottle for example (I've asked them to fill out the rename request etc)
19:10:04 <corvus> and of course, searching isn't really adequate to retire hound yet; that will require gitea improvements.
19:10:36 <corvus> "airship/in-a-bottle" looks pretty cool to me :)
19:10:45 <mordred> speaking of refs/changes - have we put up a change to switch gerrit to use internal gitweb links?
19:10:53 <clarkb> mordred: I am not aware of that change
19:10:54 <fungi> airship/bring-your-own-bottle? ;)
19:10:59 <mordred> ok. I'll make one of those
19:11:19 <corvus> mordred: for what?
19:11:21 <clarkb> we also need someone to push the rename playbook fixes we had to write
19:11:33 <clarkb> (I volunteered to do that yseterday then got distracted by gitea git stack sizes)
19:11:36 <mordred> corvus: the gerrit links to browse a given change  don't wokr
19:11:42 <corvus> mordred: gotcha thx
19:12:03 <mordred> corvus: I think this makes replicating refs/changes slightly higher than "would be nice" - but also still there is a workaround so not *urgent*
19:12:15 <fungi> we also left behind the following repos: openstack-dev/ci-sandbox openstack-dev/os-http openstack-dev/sandbox openstack-infra/beaker-localhost openstack-infra/beaker-nodepool openstack-infra/netlify-sandbox openstack-infra/opendev-website openstack-infra/puppet-docker_registry openstack-infra/puppet-odsreg openstack-infra/vinz-webclient openstack-infra/zone-opendev.org
19:12:17 <fungi> openstack-infra/zone-zuul-ci.org
19:12:17 <corvus> mordred: yeah
19:12:30 <mordred> fungi: ++
19:12:45 <corvus> openstack-dev/ci-sandbox is a curious one
19:12:56 <corvus> along with openstack-dev/sandbox
19:12:59 <mordred> yeah
19:13:05 <corvus> maybe both should be opendev/ ?
19:13:06 <fungi> curious as in where do they belong?
19:13:20 <mordred> and should zone-zuul-ci.org be opendev/ or zuul/ ?
19:13:22 <fungi> i'm inclined to say opendev. they're places to test out interactions with our systems
19:13:25 <clarkb> ya I think they are generally applicable to the various users of our gerrit system
19:13:28 <corvus> mordred: zuul i think
19:13:36 <mordred> (I'm also inclined to say both in opendev/ for th sandboxes as well)
19:13:43 <mordred> corvus: that's where my brain was leaning
19:13:59 <clarkb> in particular sandbox is used for teaching people how to gerrit
19:14:15 <mordred> and ci-sandbox is for teachign ci's how to gerrit :)
19:14:33 <mordred> opendev-website can be retired - although we should probably rename it to opendev/ then retire it
19:14:41 <fungi> my only concern with putting the various zone repos somewhere other than opendev's namespace is the implication that they're safe for folks to update without risk of breaking our infrastructure
19:15:24 <fungi> pretty sure the right change to zone-zuul-ci.org could bring down the opendev nameservers
19:15:36 <corvus> yeah, if we feel like opendev should manage those, i'm game
19:15:41 <mordred> ++
19:15:59 <corvus> fungi: we can have tests!
19:16:01 <fungi> it's more like having consistent oversight to review and approve changes for them
19:16:03 <clarkb> anyway we don't have to figure out the correct mapping for everything right now. More wanted to go over known issues and things we need to fixup
19:16:10 <corvus> clarkb: ++
19:16:29 <fungi> corvus: that's a very good point. i guess with functional testing we have now changes which break our nameservers will likely also break those job builds
19:16:51 <clarkb> Sounds like at this point the big items are git stack size fix for openstack/openstack in gitea, gerrit gitweb enablement for change browsing, and updating our rename playbook. Then start collecting data on the next round of renames to clean up after ourselves
19:17:04 <fungi> so i'm maybe -0.5 on putting zone-zuul-ci.org in the zuul namespace
19:17:07 <clarkb> on the job front fungi's text replacement work did a really good job
19:17:22 <clarkb> we still had to fix jobs and scripts and stuff, but overall the transition was pretty easy
19:17:33 <corvus> huzzah!  a round of drinks in denver on fungi!
19:17:44 <fungi> hah, sure, why not
19:17:54 <Shrews> s/on fungi/in fungi/
19:18:03 <corvus> i'm always getting that confused
19:18:07 <fungi> there will definitely be plenty of the latter
19:18:27 <fungi> on the rename playbook, should have it also start updating/correcting issues links in gitea
19:18:39 <clarkb> fungi: +1
19:18:49 <fungi> and zuul keys?
19:19:03 <fungi> do we have a hitlist of things we know are currently missing from it?
19:19:22 <clarkb> fungi: I think half the hitlist is diff what is on bridge against what is in system-config and go from there
19:19:34 <mordred> yeah
19:19:37 <clarkb> (also our irc logs from friday are pretty good for this)
19:19:40 <fungi> ahh, i forgot we forked it there temporarily
19:20:22 <fungi> oh, at one point folks were also reporting stale search results from the gitea explore search which weren't taking the renames into account... did those eventually self-correct? anybody know?
19:21:08 <clarkb> searching for thingsl ike git-review and nodepool in the explore searcher shows up to date results
19:21:27 <clarkb> I don't know what the specific instances of this problem were off the top of my head though
19:21:57 <fungi> same. i'm willing to bet gitea reindexes in the background and takes time to catch up
19:22:04 <corvus> there's a list of tasks in the etherpad
19:22:24 <corvus> should have the things we need to do to update the rename playbook in it
19:22:29 <clarkb> oh right we can cross check against that for the playbook
19:23:24 <fungi> i've either closed or lost the tab for that pad in my endless sea of browser tabs... can we link it here for posterity?
19:23:31 <clarkb> worth saying again: thank you to everyone that helped with the migration and put time into making it happen and for putting up with the outage and transition necesary on the developer side too :)
19:23:39 <clarkb> #link https://etherpad.openstack.org/p/the-great-opendev-git-migration
19:23:42 <clarkb> fungi: ^
19:23:46 <fungi> thanks!
19:24:02 <fungi> how could i ever forget such a great pad title anyway?
19:24:02 <clarkb> anything else or should we move on?
19:24:06 <mordred> fungi: right?
19:24:22 <fungi> i got nothin'
19:27:15 <clarkb> ok I think corvus may be distracted by zuul things right now. I'll keep moving the meeting along since it helps get info recorded and out there but understand if others are a bit distracted
19:27:22 <clarkb> #topic Config Management Updates
19:28:32 <clarkb> The biggest chunk of news here is that we believe we've addressed the docker image build's problem with ipv6
19:28:44 <mordred> \o/
19:29:05 <clarkb> the "fix" ended up being having socat proxy ipv4 to ipv6 so that docker can pretend that ipv6 doesn't exist
19:29:14 <corvus> i'm bummed that skopeo is choosing to remain bug-compatible with docker on iths
19:29:18 <clarkb> (and uses /etc/hosts on the test nodes where we can edit /etc/hosts)
19:29:54 <mordred> corvus: and also bummed that docker isn't landing our patch
19:30:07 <corvus> mordred: there are 3000 open issues ahead of it
19:30:09 <clarkb> my patch is actually broken but I'm wondering if anyone will show up and tell me that
19:30:25 <clarkb> I haev to allow for : in the ipv6 addrs too
19:31:17 <clarkb> in any case we should now be able to reliably do image building and testing of those images regarless of cloud we run on
19:31:32 <mordred> that's very exciting
19:32:10 <clarkb> Oh related to the opdnev migration we don't need to run our cgit farm anymore. We've stopped running ansible and puppet on those nodes and we should probably delete them this week
19:32:17 <mordred> clarkb: ++
19:32:20 <clarkb> and result of that is we can remove centos7 testing of our stuff
19:32:27 <clarkb> which means config mgmt things don't need to run there anymore
19:33:17 <clarkb> #topic Storyboard
19:33:32 <clarkb> diablo_rojo: fungi: as with last week I've not had much time to follow storyboard recently.
19:33:36 <clarkb> Anything to bring up to the group?
19:34:32 <fungi> i don't know, i've been basically all opendev
19:34:37 <fungi> migration
19:34:56 <corvus> (and, for the record and those arriving late, we discussed some storyboard-related things earlier in the opendev meeting topic)
19:35:21 <fungi> i still have an outstanding task to reproduce the renames from the maintenance on storyboard-dev so that puppet gets un-broken on it
19:35:52 <clarkb> fungi: another thing to add to the rename script (rename on storyboard-dev too)
19:36:35 <fungi> yeah, either that or stop having manage-projects try to create projects and project-groups on stroyboard-dev
19:36:46 <fungi> i could go either way on it
19:37:25 <clarkb> ok something to think about when updating the playbook I guess
19:37:33 <clarkb> #topic General Topics
19:38:14 <clarkb> letsencrypt progress. I don't think ianw is here today. So probably not a ton to talk about
19:38:40 <clarkb> That said there are certs that need renewing. git.airshipit.org, git.starlingx.io and survey.openstack.org
19:38:54 <clarkb> Previously we had said get longest cert term possible and let those ride out into the sunset.
19:39:00 <fungi> and there will be some we want to renew for compatibility reasons for a while
19:39:13 <clarkb> that still seem like a reasonable plan given the difficulty to manage DNS on those omains?
19:39:30 <fungi> yeah, we could do 2-year validity on them and declare that their sunset date
19:39:33 <clarkb> I think that means a 2 year cert for those names (which I'm still happy to do just wanted to double check now that we've got letsencrypt working)
19:40:02 <clarkb> if there are no objections doing ^ I will likely do that after the summit
19:40:13 <fungi> it's less the difficulty of doing dns for them, and more the desire not to integrate our automation with proprietary dnsaas solutions
19:40:16 <clarkb> ianw_pto: ^ so when you are back from pto that might be something to weigh in on
19:40:45 <clarkb> Next up: Trusty upgrade progress.
19:40:51 <clarkb> Unsurprisingly basically no progress since the last meeting
19:40:57 <clarkb> This was expected
19:41:06 <clarkb> But I'd like us to finish this up real soon now if we can
19:41:13 <clarkb> fungi: we should follow up with the foundation on the groups servers
19:41:27 <fungi> yes
19:42:12 <clarkb> And now PTG Planning
19:42:18 <clarkb> #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning
19:42:39 <clarkb> as you can probably tell that has been fairly bare
19:42:57 <clarkb> now that the bulk of the opendev work is behind us I'd like to start filling in info there based on where we have ended up
19:43:33 <clarkb> But everyone elseshould feel free to add things too
19:43:59 <clarkb> this is likely to be a tomorrow task for me at the rate today has gone
19:44:03 <mordred> yeah
19:44:46 <clarkb> Also I thought it would be fun to go back to lowry beer garden if weather and scheduling permits
19:45:21 <fungi> that would indeed be fun and nostalgic
19:45:23 <clarkb> I can't keep track of when all the other events are happening
19:45:29 <clarkb> fungi: ^ do you happen to be more on top of that?
19:45:39 <fungi> nope, not even a whit
19:46:05 <clarkb> ok I'll try to propose a day/evening for that trip by tomorrow so that peopel can incorporate it into their planning
19:46:51 <mordred> clarkb: speaking of scheduling - the summit schedule doesn't seem to be udpating google calendars. I suppose I should probably report that to jimmy rather than you
19:47:01 <clarkb> mordred: or smarcet
19:47:23 <clarkb> things I do know about friday night is the ptg party
19:48:24 <clarkb> ok /me looked at stuff. Sunday night is a free night for official events (is TC or anyone meeting that night), Monday night is marketplace mixer, tuesday is the official part,y wednesday is recovery day, thursday is game night, friday is ptg party
19:48:35 <clarkb> so out of that list sunday, monday and wednesday are probably best
19:49:27 <clarkb> So ya I'll send out an email and we can pick a night :)
19:49:58 <clarkb> Anything else people want to know about prior to Denver or want to make sure we know about for denver?
19:50:02 <corvus> monday is gertty's birthday
19:50:19 <clarkb> corvus: I'd be up for celebrating gertty's birthday at lowry beer garden :)
19:50:31 <corvus> ++ it's the big "five"
19:51:08 <clarkb> long term forecast shows decent weather. On the cool side for outdoors depending on what your definition of warm is, but not looking like a lot of rain so this may just work out
19:52:15 <clarkb> #topic Open Discussion
19:52:48 <clarkb> I am likely to be afk for large portions of thursday and friday to ensure I get a bit of rest in before the marathon that starts saturday
19:53:39 <fungi> i think some members of the openstack tc said something about folks non-exclusively getting together for food/drinks informally after the joint leadership meeting, but it was more like "we will probably be hungry/thirsty"
19:54:20 <clarkb> I arrive Saturday afternoon too if anyone wants to meetup and find food
19:55:32 <corvus> i arrive some time on saturday.  i helpfully forget when.
19:56:04 <clarkb> oh the A train (the train the openstack release is named after) runs frequently from the airport to about a mile from the convention center
19:56:27 <clarkb> depending on how you feel after flying that might be more exciting than a taxi :)
19:57:12 <clarkb> And sounds like that may be it. Thank you everyone!
19:57:23 <clarkb> #endmeeting