19:01:17 <clarkb> #startmeeting infra
19:01:18 <openstack> Meeting started Tue Oct  8 19:01:17 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:21 <openstack> The meeting name has been set to 'infra'
19:01:31 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-October/006489.html Our Agenda
19:01:40 <clarkb> #topic Announcements
19:02:14 <clarkb> The OpenStack release happens in ~1 week. I don't think we have anything in flight we need to be concerned about impacting that, but figured I would mention it
19:02:34 <fungi> yeah, i think we're in a good spot
19:03:00 <fungi> job activity has died down a bit now that openstack's in its rc period too
19:03:54 <clarkb> I'm also going to be afk tomorrow. Trying to find the salmon while they are around this fall.
19:04:03 <clarkb> Anything else to announce?
19:04:28 <corvus> i think i will be around
19:04:35 <corvus> but i could disappear through the weekend
19:04:53 <clarkb> corvus: you need a laptop powered by hamster wheels
19:04:53 <fungi> i'm on hand all week
19:04:54 <corvus> depending on the whims of the electric utility provider
19:04:55 <fungi> no plans
19:05:15 <fungi> granted, electricity and internet access here are always a gamble
19:06:16 <clarkb> #topic Actions from last meeting
19:06:23 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-01-19.01.txt minutes from last meeting
19:06:36 <clarkb> There were none. Lets just keep moving to get into the bulk of the agenda
19:06:45 <clarkb> #topic Priority Efforts
19:06:53 <clarkb> #topic Update Config Management
19:07:06 <clarkb> mordred has reported that review-dev is actually running from the docker image
19:07:15 <clarkb> I don't think mordred is here today to talk about that more though
19:07:25 <corvus> ooh cool
19:07:33 <clarkb> It might be useful to start drafting up a plan to convert the production server though if anyone happens to see mordred
19:07:46 <corvus> so i guess next step is to switch out the folgers crystals on review.o.o
19:07:52 <clarkb> ya I think so
19:08:46 <clarkb> Are there any other config management updates from the last week?
19:09:48 <clarkb> #topic OpenDev
19:10:27 <clarkb> I have not yet written the email etherpad for opendev governance. Keep getting distracted by visa things and zuul and nodepool and glean
19:10:43 * clarkb scribbles a note to block the world out on thursday and write a draft
19:11:24 <clarkb> fungi: is this a good place to talk about the management of storyboard groups or should we do that under the storyboard topic? It sort of straddles these subjects
19:11:36 <fungi> maybe storyboard
19:12:14 <clarkb> Ok the last topic I know of re opendev was whether or not we should consider podman over docker.
19:12:15 <fungi> though this might be a good time to talk about dns/ssl for services hosted in opendev when their domains aren't in our dns management workflow?
19:12:25 <clarkb> fungi: that is udner the static.o.o topic
19:12:31 <clarkb> (its on the agenda we'll get to it)
19:12:35 <fungi> ahh, okay. sure it came out of that discussion anyway
19:13:21 <clarkb> I have no objection to using podman over docker though it would make me happier if podman was more willing to fix bugs in behaviors. Seems they prefer to try and mimic docker behavior of existing functionality as much as possible
19:13:38 <fungi> what is the recommended way to install podman on non-rh distros these days?
19:14:00 <corvus> the podman/docker thing isn't urgent -- i mostly brought it up for 2 reasons 1) to get feedback on whether the toolset is robust enough to be considered a complete replacement.  that soft of knowledge helps me know where to focus zuul efforts.   and 2) perhaps that ecosystem, being a bit more open, might be more aligned with our values.
19:14:02 <fungi> does its lack of current inclusion in debian/ubuntu pose any challenges?
19:14:13 <corvus> fungi: i reckon we'd have to use the ppa for now
19:14:19 <fungi> got it
19:14:20 <corvus> fungi: but istr you said it's on its way into debian?
19:14:22 <clarkb> fungi: depends on the distro. For ubuntu there is a PPA that tracks development. For suse its in the main repos aiui. Arch has it on the user package repo
19:14:42 <corvus> from what i can see, there is heavy red hat and suse involvement in development of the tools and libraries
19:14:52 <fungi> corvus: yeah, it seems to be getting worked on. still that's at best winding up in ubuntu 20.04 lts but maybe later still
19:15:19 <corvus> we already rely on the ppa for several other things, and i've been happy with that so far
19:15:43 <clarkb> skopeo in particular
19:15:46 <fungi> we rely on the ppa containing podman, or just ppas in general?
19:15:53 <fungi> aha, skopeo is in there then?
19:16:01 <corvus> er, weird typo like 10 lines up ^ s/that soft of knowledge/that sort of knowledge/
19:16:02 <clarkb> yes skopeo is part of the podman suite of tools
19:16:09 <clarkb> and comes from that particular ppa
19:16:27 <fungi> cool
19:17:14 <fungi> just to be clear, we're talking about use of podman instead of docker for setting up the services we run as a part of opendev, right?
19:17:19 <clarkb> fungi: yes
19:17:27 <clarkb> gitea (and soon gerrit) for example
19:17:36 <corvus> there's a podman-compose tool, so that's pretty straightforward
19:18:10 <corvus> biggest technical win is podman can run containers without a root-level daemon
19:18:42 <clarkb> it still needs root if you do certain things but ya no daemon and can run without root if you restrict your feature set
19:18:43 <corvus> that's also the biggest technical change; some more attention to uids may be required.  but that's pretty straightforward.
19:18:50 <fungi> yeah, i can't speak to the feature/bug parity between them, but from a which-one-is-more-openly-maintained perspective it sounds like a clear win
19:19:07 <clarkb> I guess the next step here for me is to install it locally and start using it over docker for a bit
19:19:27 <corvus> tristanC is experimenting with swapping it in to the zuul-quick-start job
19:19:46 <fungi> awesome
19:19:49 <corvus> which happens to have really good coverage of how we use docker/docker-compose, so we can learn some there
19:19:59 <corvus> and of course, we can do the same in our system-config testing
19:20:47 <corvus> i'm reading the consensus as 'cautious support for more experimentation'
19:20:52 <clarkb> ++
19:21:15 <fungi> sounds right to me
19:21:23 <clarkb> anything else on the opendev subject?
19:21:38 <fungi> just stuff we've got other topics for already
19:21:48 <clarkb> k lets move on to them then
19:21:50 <clarkb> #topic Storyboard
19:22:02 <clarkb> fungi: want to fill us in on storyboard group management?
19:22:47 <fungi> not a lot to cover since last week. so far it's mostly been investigation of the current crufty automation we have for project and project-group creation
19:22:58 <fungi> it's sort of a poster child for "what not to do" here
19:23:28 <fungi> puppet exec calling a utility script in the storyboard repo which pokes values directly through the orm into the db
19:23:58 <fungi> so best that we build a solution from scratch which can handle that and the group creation/project association bits via rest api interaction
19:24:24 <clarkb> that seems like a much safer interface to interact with :)
19:24:54 <fungi> plan for now is a module which we can call from ansible, using the python-storyboardclient sdk to interact with it to align resources between what's in sb and whats in project-config
19:25:21 <fungi> and deprecate the old overloading of the storyboard-db-manage utility
19:25:21 <corvus> is python-sbclient sufficiently developed?
19:25:36 <fungi> if it's not, then i'll be plumbing additional api calls through it
19:25:52 <fungi> though i may just look at what boardtty's doing
19:26:02 <fungi> i guess that's going direct to the rest api?
19:26:05 <corvus> tbh, i'm not convinced of the utility of it, since the rest api is so simple.  boartty just does raw rest
19:26:21 <fungi> that does sound like a great option in that case
19:26:48 <clarkb> the gitea module in ansible is probably a good example for direct http interaction with apis?
19:26:49 <corvus> (and at some point, the sb developers started to feel the same way so it lagged -- i honestly have no idea what the situation is now)
19:27:12 <clarkb> (specifically use a python module to avoid fork overhead, but then do simply python requests?)
19:27:23 <fungi> clarkb: thanks for the reminder, yes the gitea interaction we have in ansible is how i want to model it
19:27:48 <corvus> clarkb: yeah, and the approach i like with those kinds of modules is to make them callable from the command line as well as ansible, so it's easy to develop/test without ansible
19:28:17 <fungi> sounds like a plan
19:28:33 <clarkb> Anything else related to storyboard?
19:29:11 <diablo_rojo> I think fungi covered it
19:29:23 <fungi> yeah, nothing new since my update last week
19:29:41 <fungi> i hope to start punching at the keyboard to make the group management module in the coming days
19:29:48 <clarkb> sounds good thanks
19:29:52 <clarkb> #topic General Topics
19:29:56 <diablo_rojo> Thanks clarkb
19:30:04 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup
19:30:11 <clarkb> fungi: any progress with the wiki?
19:30:17 <fungi> nope
19:30:27 <clarkb> Let's talk static.o.o then
19:30:34 <clarkb> #link https://review.opendev.org/683852 Review spec
19:30:34 <fungi> always happy to help bring other volunteers up to speed on the wiki too ;)
19:30:51 <clarkb> we brought this spec up wtih the TC last week
19:30:58 <clarkb> #link https://etherpad.openstack.org/p/openstack-org-dns
19:31:07 <clarkb> this DNS management etherpad came out of those discussions
19:31:09 <mordred> clarkb: o/ I'm here for a half a second
19:31:52 <clarkb> Do we need or want to update the spec to take into account what we've decided we can do with dns?
19:32:05 <clarkb> in which case would getting agreement on dns chagnes from the foundation be step 0?
19:32:05 <mordred> clarkb: yes - review-dev is running the image - but it's been manualled - next step there is to finish the cfg-mgmt change to make sure we're rolling it out properly
19:32:07 <corvus> to be clear we should decide that now-ish :)
19:32:42 <corvus> this is basically a tc+fungi+corvus developed proposal for the infra/opendev team(s) to review
19:33:11 <ianw> i'll be happy to update the spec to cover that etherpad
19:33:53 <clarkb> After reading it yesterday and clarifying the intent behind it (the foundation didn't ask us to stop managing dns for example) I think I'm in favor of this change. In particular I like that it provides a bridge to show how well dns management can work from the existing system
19:33:58 <clarkb> to give us things like LE support
19:34:04 <corvus> if we like it; yeah, let's update the spec.  strictly speaking, i'm not sure foundation agreement/approval is required since we're actually doing *less* openstack dns than we are now, but certainly a heads up/consultation would be good.  we are friendly and talk.  jimmy was part of that discussion too.  he did request a writeup, so just sending that etherpad along would be good i think.
19:34:58 <clarkb> and ya I agree I think we don't awnt to go down the road of opdnev.org/docs/foo and a few months later grow the ability to support docs.foo.org
19:35:12 <clarkb> will likely make users unhappy unless they wanted the opendev.org domain in the first place
19:35:43 <fungi> yep, from my perspective it's mostly that we can precreate a bunch of cnames in openstack.org (and starlingx.io) for stuff and then probably never (or almost never) need to care about what the dns for those domains again
19:36:06 <clarkb> fungi and I can bring it up with the foundation (I can do that immediately after this meeting)
19:36:22 <clarkb> does anyone object from out end?
19:36:29 <clarkb> s/out/our/
19:36:43 <corvus> and yes, while this is not ideal (openstack-infra hat: i think the openstack project should have more direct involvement in its domain management), i think this helps us achieve our goals with opendev: opendev systems management doesn't rely on any resources which have restricted contribution channels.  it lets us make a sort of "hosting platform" for static content and let projects manage the
19:36:43 <corvus> public-facing dns for that.
19:37:38 <fungi> it also gives us a fairly easy answer for "so your project doesn't control its domain but you want to have a whitelabeled opendev service on a subdomain of it"
19:37:53 <corvus> yep
19:37:58 <fungi> i guess that's more or less what you said too ;)
19:38:25 <fungi> cname these records to us and we'll do the rest
19:38:37 <clarkb> I'm not hearing any objections then. ianw maybe fungi and/or I can leave a comment on the spec with a request for a new patchset assuming the foundation has no objections either (and as corvus points out it shouldn't affect the fqdns they manage in that zone)
19:38:48 <fungi> s/we'll do the rest/you can do the rest through opendev/
19:38:52 <corvus> fungi: or, you know, you can submit a patch to do the rest yourself too :)
19:38:54 <corvus> right that
19:39:31 <ianw> clarkb: sure; although if it would be easier to have more exact details to explain it in the spec to point people too, then i'd need a bit to update it first
19:39:41 <fungi> i'm perfectly happy to go through and precreate all the cnames we want
19:40:01 <clarkb> ianw: I think the etherpad should cover what we need to explain to our co managers of that dns zone
19:40:20 <clarkb> but I guess there isn't a strict dependency there if you want to go ahead and start updating it
19:40:23 <fungi> though we may want to couple that with an audit of dns cruft entries we can clean up, since there's something like a 500 rr limit per zone in rackspace's dns platform
19:40:50 <fungi> so precreating a bunch of cnames may push us over that limit
19:41:05 <clarkb> fungi: many of our hosts are already cnames
19:41:13 <clarkb> we'll just end up changing their targets
19:41:24 <clarkb> then I guess we add the acme cnames
19:41:26 <ianw> right, i was thinking the level of "this is the names we'll need", so if higher level is ok then etherpad is ok
19:41:27 <fungi> yeah, but any where we want to move ssl to letsencrypt will need an additional cname each
19:41:28 <clarkb> those could push us over I suppose
19:41:42 <fungi> right, that's what i was considering as a risk
19:41:59 <clarkb> fungi: ya in that case we should probably count the records we have now and estimate the total number of additions
19:42:19 <fungi> probably we should start with a list of which subdomain names we need to do this for
19:42:23 <clarkb> we'll also be deleting A and AAAA records
19:42:30 <clarkb> so it may end up balancing out
19:42:37 <fungi> that's a great point
19:42:52 <fungi> A+AAAA vs 2xCNAME
19:43:19 <clarkb> we can sort those details out after the meeting
19:43:24 <fungi> sounds good
19:43:30 <clarkb> ianw was there other items to discuss around this spec?
19:45:16 <clarkb> sounds like no. Lets move on then
19:45:17 <ianw> i think that will suffice for now ... maybe we want to consider the volume generation on AFS but i think let's get this portion squared first since there's some momentum there
19:45:24 <clarkb> ++
19:46:00 <clarkb> Next up is CentOS 8. I wanted to bring this up because we are continuing to struggle with Network Manager on ipv6 only clouds and we may want to point out to projects that the images exist once they are there to avoid late transitions like we've had with ubuntu in the past
19:46:06 <clarkb> #link https://review.opendev.org/#/c/686474/
19:46:10 <clarkb> #link https://review.opendev.org/#/c/686749/
19:46:42 <clarkb> are two glean changes that try to explicitly configure ipv6 for interfaces in NM to "trick" it into managing interfaces even if the kernel has gotten there first
19:47:13 <clarkb> I think they are actually ready at this point if people want to review them. We should also build a centos and fedora 29 image and boot them in FN and rax and ovh and ensure the various different setups work
19:47:43 <ianw> i've been out and haven't looked closely at that, sorry.  it seems weird we're the only ones who have this problem (although from the old debian bug report, maybe we weren't)
19:47:55 <clarkb> I can help with getting test images built and booting on the various clouds (maybe we introduce a fault in the nodepool integration job then hold them and copy those images out)
19:47:59 <clarkb> ianw: ya I don't think we are the only ones
19:48:15 <clarkb> that bug has had a lot of traction over many years
19:50:00 <clarkb> ianw: and once we have centos8 images ready I think we should advertise that broadly and encourage people to start uplifting 7 to 8 to avoid the problems we've had with ubnutu
19:50:00 <fungi> i think it's more that there aren't a lot of folks continuously booting vast numbers of servers in the right kinds of environments
19:50:05 <clarkb> or at least start hitting the problems earlier
19:50:20 <fungi> and also that some projects (like cloud-init) may have worked around it with configuration
19:50:53 <ianw> i am merging the centos 8 stuff with review from cgoncalves; it is booting in integration tests, i don't expect too many issues
19:51:04 <clarkb> excellent
19:51:20 <ianw> we don't currently have an upstream .qcow2 image for the image-based builds, which some things use
19:51:44 <ianw> cgoncalves has tested with his kickstart images, i believe, however
19:51:54 <clarkb> and we build it up from scratch
19:52:28 <ianw> right, so yeah the first release will only be -minimal support
19:53:19 <ianw> also we took the path discussed on the infra list of not doing pip/virtualenv magic and leaving that up to jobs to configure
19:53:52 <clarkb> that would be a good thing to note when we announce availability
19:53:56 <ianw> so it is a python-3 only image
19:54:24 <clarkb> We are almost out of time. I wanted to point out I've started populating the ptg etherpad
19:54:30 <clarkb> #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019
19:54:32 <ianw> i.e. "pip" and "virtualenv" don't exist ... that would be "pip3" and "python3 -m venv"
19:54:37 <clarkb> #link https://www.openstack.org/ptg/ We have a schedule
19:54:40 <clarkb> ianw: good to know
19:54:47 <clarkb> There is also a published schedule now
19:55:03 <clarkb> I've not heard anything recently on my visa status. Not sure if a good or bad thing. Will wait patiently
19:55:17 <clarkb> I expect to hear back this week or next
19:55:28 <clarkb> And with that I'll open it up to all the things
19:55:33 <clarkb> #topic Open Discussion
19:56:23 <fungi> yeah, the embassy still has my passport, no word yet on whether they'll return it with a visa or without or demand more proof i should be allowed to visit
19:59:41 <clarkb> And we are at time. Thank you everyone!
19:59:46 <corvus> clarkb: thanks!
19:59:46 <clarkb> #endmeeting