19:01:17 #startmeeting infra 19:01:18 Meeting started Tue Oct 8 19:01:17 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:21 The meeting name has been set to 'infra' 19:01:31 #link http://lists.openstack.org/pipermail/openstack-infra/2019-October/006489.html Our Agenda 19:01:40 #topic Announcements 19:02:14 The OpenStack release happens in ~1 week. I don't think we have anything in flight we need to be concerned about impacting that, but figured I would mention it 19:02:34 yeah, i think we're in a good spot 19:03:00 job activity has died down a bit now that openstack's in its rc period too 19:03:54 I'm also going to be afk tomorrow. Trying to find the salmon while they are around this fall. 19:04:03 Anything else to announce? 19:04:28 i think i will be around 19:04:35 but i could disappear through the weekend 19:04:53 corvus: you need a laptop powered by hamster wheels 19:04:53 i'm on hand all week 19:04:54 depending on the whims of the electric utility provider 19:04:55 no plans 19:05:15 granted, electricity and internet access here are always a gamble 19:06:16 #topic Actions from last meeting 19:06:23 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-01-19.01.txt minutes from last meeting 19:06:36 There were none. Lets just keep moving to get into the bulk of the agenda 19:06:45 #topic Priority Efforts 19:06:53 #topic Update Config Management 19:07:06 mordred has reported that review-dev is actually running from the docker image 19:07:15 I don't think mordred is here today to talk about that more though 19:07:25 ooh cool 19:07:33 It might be useful to start drafting up a plan to convert the production server though if anyone happens to see mordred 19:07:46 so i guess next step is to switch out the folgers crystals on review.o.o 19:07:52 ya I think so 19:08:46 Are there any other config management updates from the last week? 19:09:48 #topic OpenDev 19:10:27 I have not yet written the email etherpad for opendev governance. Keep getting distracted by visa things and zuul and nodepool and glean 19:10:43 * clarkb scribbles a note to block the world out on thursday and write a draft 19:11:24 fungi: is this a good place to talk about the management of storyboard groups or should we do that under the storyboard topic? It sort of straddles these subjects 19:11:36 maybe storyboard 19:12:14 Ok the last topic I know of re opendev was whether or not we should consider podman over docker. 19:12:15 though this might be a good time to talk about dns/ssl for services hosted in opendev when their domains aren't in our dns management workflow? 19:12:25 fungi: that is udner the static.o.o topic 19:12:31 (its on the agenda we'll get to it) 19:12:35 ahh, okay. sure it came out of that discussion anyway 19:13:21 I have no objection to using podman over docker though it would make me happier if podman was more willing to fix bugs in behaviors. Seems they prefer to try and mimic docker behavior of existing functionality as much as possible 19:13:38 what is the recommended way to install podman on non-rh distros these days? 19:14:00 the podman/docker thing isn't urgent -- i mostly brought it up for 2 reasons 1) to get feedback on whether the toolset is robust enough to be considered a complete replacement. that soft of knowledge helps me know where to focus zuul efforts. and 2) perhaps that ecosystem, being a bit more open, might be more aligned with our values. 19:14:02 does its lack of current inclusion in debian/ubuntu pose any challenges? 19:14:13 fungi: i reckon we'd have to use the ppa for now 19:14:19 got it 19:14:20 fungi: but istr you said it's on its way into debian? 19:14:22 fungi: depends on the distro. For ubuntu there is a PPA that tracks development. For suse its in the main repos aiui. Arch has it on the user package repo 19:14:42 from what i can see, there is heavy red hat and suse involvement in development of the tools and libraries 19:14:52 corvus: yeah, it seems to be getting worked on. still that's at best winding up in ubuntu 20.04 lts but maybe later still 19:15:19 we already rely on the ppa for several other things, and i've been happy with that so far 19:15:43 skopeo in particular 19:15:46 we rely on the ppa containing podman, or just ppas in general? 19:15:53 aha, skopeo is in there then? 19:16:01 er, weird typo like 10 lines up ^ s/that soft of knowledge/that sort of knowledge/ 19:16:02 yes skopeo is part of the podman suite of tools 19:16:09 and comes from that particular ppa 19:16:27 cool 19:17:14 just to be clear, we're talking about use of podman instead of docker for setting up the services we run as a part of opendev, right? 19:17:19 fungi: yes 19:17:27 gitea (and soon gerrit) for example 19:17:36 there's a podman-compose tool, so that's pretty straightforward 19:18:10 biggest technical win is podman can run containers without a root-level daemon 19:18:42 it still needs root if you do certain things but ya no daemon and can run without root if you restrict your feature set 19:18:43 that's also the biggest technical change; some more attention to uids may be required. but that's pretty straightforward. 19:18:50 yeah, i can't speak to the feature/bug parity between them, but from a which-one-is-more-openly-maintained perspective it sounds like a clear win 19:19:07 I guess the next step here for me is to install it locally and start using it over docker for a bit 19:19:27 tristanC is experimenting with swapping it in to the zuul-quick-start job 19:19:46 awesome 19:19:49 which happens to have really good coverage of how we use docker/docker-compose, so we can learn some there 19:19:59 and of course, we can do the same in our system-config testing 19:20:47 i'm reading the consensus as 'cautious support for more experimentation' 19:20:52 ++ 19:21:15 sounds right to me 19:21:23 anything else on the opendev subject? 19:21:38 just stuff we've got other topics for already 19:21:48 k lets move on to them then 19:21:50 #topic Storyboard 19:22:02 fungi: want to fill us in on storyboard group management? 19:22:47 not a lot to cover since last week. so far it's mostly been investigation of the current crufty automation we have for project and project-group creation 19:22:58 it's sort of a poster child for "what not to do" here 19:23:28 puppet exec calling a utility script in the storyboard repo which pokes values directly through the orm into the db 19:23:58 so best that we build a solution from scratch which can handle that and the group creation/project association bits via rest api interaction 19:24:24 that seems like a much safer interface to interact with :) 19:24:54 plan for now is a module which we can call from ansible, using the python-storyboardclient sdk to interact with it to align resources between what's in sb and whats in project-config 19:25:21 and deprecate the old overloading of the storyboard-db-manage utility 19:25:21 is python-sbclient sufficiently developed? 19:25:36 if it's not, then i'll be plumbing additional api calls through it 19:25:52 though i may just look at what boardtty's doing 19:26:02 i guess that's going direct to the rest api? 19:26:05 tbh, i'm not convinced of the utility of it, since the rest api is so simple. boartty just does raw rest 19:26:21 that does sound like a great option in that case 19:26:48 the gitea module in ansible is probably a good example for direct http interaction with apis? 19:26:49 (and at some point, the sb developers started to feel the same way so it lagged -- i honestly have no idea what the situation is now) 19:27:12 (specifically use a python module to avoid fork overhead, but then do simply python requests?) 19:27:23 clarkb: thanks for the reminder, yes the gitea interaction we have in ansible is how i want to model it 19:27:48 clarkb: yeah, and the approach i like with those kinds of modules is to make them callable from the command line as well as ansible, so it's easy to develop/test without ansible 19:28:17 sounds like a plan 19:28:33 Anything else related to storyboard? 19:29:11 I think fungi covered it 19:29:23 yeah, nothing new since my update last week 19:29:41 i hope to start punching at the keyboard to make the group management module in the coming days 19:29:48 sounds good thanks 19:29:52 #topic General Topics 19:29:56 Thanks clarkb 19:30:04 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup 19:30:11 fungi: any progress with the wiki? 19:30:17 nope 19:30:27 Let's talk static.o.o then 19:30:34 #link https://review.opendev.org/683852 Review spec 19:30:34 always happy to help bring other volunteers up to speed on the wiki too ;) 19:30:51 we brought this spec up wtih the TC last week 19:30:58 #link https://etherpad.openstack.org/p/openstack-org-dns 19:31:07 this DNS management etherpad came out of those discussions 19:31:09 clarkb: o/ I'm here for a half a second 19:31:52 Do we need or want to update the spec to take into account what we've decided we can do with dns? 19:32:05 in which case would getting agreement on dns chagnes from the foundation be step 0? 19:32:05 clarkb: yes - review-dev is running the image - but it's been manualled - next step there is to finish the cfg-mgmt change to make sure we're rolling it out properly 19:32:07 to be clear we should decide that now-ish :) 19:32:42 this is basically a tc+fungi+corvus developed proposal for the infra/opendev team(s) to review 19:33:11 i'll be happy to update the spec to cover that etherpad 19:33:53 After reading it yesterday and clarifying the intent behind it (the foundation didn't ask us to stop managing dns for example) I think I'm in favor of this change. In particular I like that it provides a bridge to show how well dns management can work from the existing system 19:33:58 to give us things like LE support 19:34:04 if we like it; yeah, let's update the spec. strictly speaking, i'm not sure foundation agreement/approval is required since we're actually doing *less* openstack dns than we are now, but certainly a heads up/consultation would be good. we are friendly and talk. jimmy was part of that discussion too. he did request a writeup, so just sending that etherpad along would be good i think. 19:34:58 and ya I agree I think we don't awnt to go down the road of opdnev.org/docs/foo and a few months later grow the ability to support docs.foo.org 19:35:12 will likely make users unhappy unless they wanted the opendev.org domain in the first place 19:35:43 yep, from my perspective it's mostly that we can precreate a bunch of cnames in openstack.org (and starlingx.io) for stuff and then probably never (or almost never) need to care about what the dns for those domains again 19:36:06 fungi and I can bring it up with the foundation (I can do that immediately after this meeting) 19:36:22 does anyone object from out end? 19:36:29 s/out/our/ 19:36:43 and yes, while this is not ideal (openstack-infra hat: i think the openstack project should have more direct involvement in its domain management), i think this helps us achieve our goals with opendev: opendev systems management doesn't rely on any resources which have restricted contribution channels. it lets us make a sort of "hosting platform" for static content and let projects manage the 19:36:43 public-facing dns for that. 19:37:38 it also gives us a fairly easy answer for "so your project doesn't control its domain but you want to have a whitelabeled opendev service on a subdomain of it" 19:37:53 yep 19:37:58 i guess that's more or less what you said too ;) 19:38:25 cname these records to us and we'll do the rest 19:38:37 I'm not hearing any objections then. ianw maybe fungi and/or I can leave a comment on the spec with a request for a new patchset assuming the foundation has no objections either (and as corvus points out it shouldn't affect the fqdns they manage in that zone) 19:38:48 s/we'll do the rest/you can do the rest through opendev/ 19:38:52 fungi: or, you know, you can submit a patch to do the rest yourself too :) 19:38:54 right that 19:39:31 clarkb: sure; although if it would be easier to have more exact details to explain it in the spec to point people too, then i'd need a bit to update it first 19:39:41 i'm perfectly happy to go through and precreate all the cnames we want 19:40:01 ianw: I think the etherpad should cover what we need to explain to our co managers of that dns zone 19:40:20 but I guess there isn't a strict dependency there if you want to go ahead and start updating it 19:40:23 though we may want to couple that with an audit of dns cruft entries we can clean up, since there's something like a 500 rr limit per zone in rackspace's dns platform 19:40:50 so precreating a bunch of cnames may push us over that limit 19:41:05 fungi: many of our hosts are already cnames 19:41:13 we'll just end up changing their targets 19:41:24 then I guess we add the acme cnames 19:41:26 right, i was thinking the level of "this is the names we'll need", so if higher level is ok then etherpad is ok 19:41:27 yeah, but any where we want to move ssl to letsencrypt will need an additional cname each 19:41:28 those could push us over I suppose 19:41:42 right, that's what i was considering as a risk 19:41:59 fungi: ya in that case we should probably count the records we have now and estimate the total number of additions 19:42:19 probably we should start with a list of which subdomain names we need to do this for 19:42:23 we'll also be deleting A and AAAA records 19:42:30 so it may end up balancing out 19:42:37 that's a great point 19:42:52 A+AAAA vs 2xCNAME 19:43:19 we can sort those details out after the meeting 19:43:24 sounds good 19:43:30 ianw was there other items to discuss around this spec? 19:45:16 sounds like no. Lets move on then 19:45:17 i think that will suffice for now ... maybe we want to consider the volume generation on AFS but i think let's get this portion squared first since there's some momentum there 19:45:24 ++ 19:46:00 Next up is CentOS 8. I wanted to bring this up because we are continuing to struggle with Network Manager on ipv6 only clouds and we may want to point out to projects that the images exist once they are there to avoid late transitions like we've had with ubuntu in the past 19:46:06 #link https://review.opendev.org/#/c/686474/ 19:46:10 #link https://review.opendev.org/#/c/686749/ 19:46:42 are two glean changes that try to explicitly configure ipv6 for interfaces in NM to "trick" it into managing interfaces even if the kernel has gotten there first 19:47:13 I think they are actually ready at this point if people want to review them. We should also build a centos and fedora 29 image and boot them in FN and rax and ovh and ensure the various different setups work 19:47:43 i've been out and haven't looked closely at that, sorry. it seems weird we're the only ones who have this problem (although from the old debian bug report, maybe we weren't) 19:47:55 I can help with getting test images built and booting on the various clouds (maybe we introduce a fault in the nodepool integration job then hold them and copy those images out) 19:47:59 ianw: ya I don't think we are the only ones 19:48:15 that bug has had a lot of traction over many years 19:50:00 ianw: and once we have centos8 images ready I think we should advertise that broadly and encourage people to start uplifting 7 to 8 to avoid the problems we've had with ubnutu 19:50:00 i think it's more that there aren't a lot of folks continuously booting vast numbers of servers in the right kinds of environments 19:50:05 or at least start hitting the problems earlier 19:50:20 and also that some projects (like cloud-init) may have worked around it with configuration 19:50:53 i am merging the centos 8 stuff with review from cgoncalves; it is booting in integration tests, i don't expect too many issues 19:51:04 excellent 19:51:20 we don't currently have an upstream .qcow2 image for the image-based builds, which some things use 19:51:44 cgoncalves has tested with his kickstart images, i believe, however 19:51:54 and we build it up from scratch 19:52:28 right, so yeah the first release will only be -minimal support 19:53:19 also we took the path discussed on the infra list of not doing pip/virtualenv magic and leaving that up to jobs to configure 19:53:52 that would be a good thing to note when we announce availability 19:53:56 so it is a python-3 only image 19:54:24 We are almost out of time. I wanted to point out I've started populating the ptg etherpad 19:54:30 #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 19:54:32 i.e. "pip" and "virtualenv" don't exist ... that would be "pip3" and "python3 -m venv" 19:54:37 #link https://www.openstack.org/ptg/ We have a schedule 19:54:40 ianw: good to know 19:54:47 There is also a published schedule now 19:55:03 I've not heard anything recently on my visa status. Not sure if a good or bad thing. Will wait patiently 19:55:17 I expect to hear back this week or next 19:55:28 And with that I'll open it up to all the things 19:55:33 #topic Open Discussion 19:56:23 yeah, the embassy still has my passport, no word yet on whether they'll return it with a visa or without or demand more proof i should be allowed to visit 19:59:41 And we are at time. Thank you everyone! 19:59:46 clarkb: thanks! 19:59:46 #endmeeting