19:01:20 <clarkb> #startmeeting infra
19:01:21 <openstack> Meeting started Tue Nov 12 19:01:20 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:24 <openstack> The meeting name has been set to 'infra'
19:01:30 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-November/006509.html Our Agenda
19:01:37 <clarkb> #topic Announcements
19:02:10 <clarkb> Last week was the summit. I am feeling extra zombie like after that trip. The jet lag has been harsh. I expect others are in a similar position
19:02:33 * fungi mutters, "braiiiiins..."
19:03:20 <fungi> some folks are still vacationing/travelling too
19:03:59 <clarkb> #topic Actions from last meeting
19:04:08 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-29-19.01.txt minutes from last meeting
19:04:27 <clarkb> The last meeting was a long time ago, no actions recorded
19:05:01 <clarkb> #topic Priority Efforts
19:05:09 <clarkb> #topic OpenDev
19:05:16 <clarkb> We need to renew the opendev.org cert
19:05:36 <clarkb> I think we can manage to do that via LE now
19:05:52 <clarkb> we'll just need a handler that restarts gitea backends when the cert updates (and maybe we can serialize those?)
19:06:11 <clarkb> figuring that out is likely to be my tomorrow task
19:06:31 <clarkb> and we have plenty of time to sort it out and if we can't for whatever reason can always buy another cert for a yer
19:06:38 * AJaeger waves late
19:07:06 <fungi> yeah, it's still something like 2.5 weeks out
19:07:11 <ianw> i guess each gitea host will have it's own cert?
19:07:23 <clarkb> ianw: yes I think so
19:08:45 <fungi> as long as le doesn't scoff at 8 hosts requesting ssl certs for the same name
19:09:07 <clarkb> that should be well within the rate limits iirc.
19:09:46 <ianw> fungi: i don't think it will matter; each will get a different txt value to put into _acme-challenge, and it should "just work"
19:09:50 <ianw> famous last words :)
19:10:02 <fungi> that'll presumably be easier than refactoring things to have ssl terminated on the lb
19:10:36 <clarkb> we can also have an altname that matches each backend specifically to help distinguish them?
19:10:44 <fungi> true
19:10:49 <clarkb> that may also make LE happier if the records are different each time
19:11:09 <fungi> if it works the way we've done it for other services, i think that happens by default
19:14:17 <clarkb> #topic Update Config Management
19:14:55 <clarkb> I'm not sure this has been called out elsewhere yet but was mentioned at the PTG. Apparently mordred thinks we shouldn't wholesale move to podman yet
19:15:24 <AJaeger> oh, why?
19:15:31 <clarkb> we'll be getting dev builds via the ubuntu ppa for podman and some commands don't work as before. Idea is to try it a bit first with review(-dev)
19:15:57 <AJaeger> so, "compatibility" ?
19:16:55 <clarkb> I think we are happy with compatibility since rhel8 docker command is podman so it must be fairly compatible. More just that we aren't sure if the dev packaging from the ppa poses any concerns yet and users will need to get back to using docker commands instead of docker-compose commands and so on
19:17:13 <clarkb> Then when we are happy with the results we can do broader changes
19:17:35 <AJaeger> I see...
19:17:48 <fungi> lack of an interleaved log viewer was noted as an unfortunate loss for switching away from docker-compose
19:19:00 <clarkb> Anything else on this topic or should we move on?
19:19:11 * fungi has nothing
19:19:21 <ianw> so for new work, like testing out the nodepool-builder images, should we use docker or podman?
19:19:28 <ianw> or in that case, does it not matter so much?
19:19:42 <clarkb> ianw: aiui mordred thinks we should use docker for now and keep podman to review(-dev)
19:20:04 <clarkb> but I may have misunderstood and maybe he meant just don't go changing existing docker usage yet
19:20:14 <clarkb> my hunch is that its probably ok for any new use of containers
19:21:12 <ianw> ok
19:22:47 <clarkb> #topic Storyboard
19:23:10 <clarkb> fungi: ^ anything new from storyboard? I expect not given the summit
19:23:45 <fungi> there was an sb onboarding session at the ptg, but nobody showed up so diablo_rojo_phon and tonyb turned it into an openstack technical election officials working session
19:23:55 <fungi> other than that, nothing new i'm aware of
19:24:18 <clarkb> #topic General topics
19:24:28 <clarkb> #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 PTG Notes
19:24:41 <clarkb> I/we tried to take notes at the summit/forum/ptg
19:25:09 <clarkb> The item on my todo list before opendev.org LE certification is to write up a summary and send it to the infra list
19:25:51 <clarkb> I did want to point out that the etherpad is there for people to browse too and that if you have questions about specific topics feel free to ping me nowish and I'll add it to the summary or you can followup on that thread when it exists
19:26:43 <clarkb> Our meeting with the gitea maintainers went well. We were able to show them some of our build and test tooling to test out the elasticsearch indexing
19:27:01 <clarkb> from that we discovered that the china golang and google golang proxies had different content for the CORS package :/
19:27:15 <fungi> that's scary
19:27:44 <clarkb> they also acknowledged that the performance issues we've hit are a major problem and they have other users struggling with that too. I hope that means we continue to see improvements with performance
19:28:59 <fungi> though we also upgraded
19:29:20 <fungi> and the new version should have at least some performance improvement over what we were running before last week
19:30:54 <clarkb> Also their community is largely on discord and is activ ethere if we want to communicate with them further
19:31:04 <clarkb> I've got a discord account somewhere, but I should prbably set that up for gitea now
19:32:22 <clarkb> and I'll try to write all this up properly between now and tomorrow sometime
19:32:47 <clarkb> fungi: seems like there were wiki changes prior to leaving for the summit. Were you able to check on those?
19:32:56 <fungi> nope
19:33:21 <fungi> i mean yep some changes merged, nope i did not find time yet to follow up further
19:34:20 <clarkb> For static.openstack.org mnaser volunteered to do the job changes on the openstack side of things
19:35:10 <clarkb> ianw: AJaeger ^ I think that means our next step is to create afs volumes, then mnaser can upate jobs to publish to afs and static.o.o, we verify content, then flip things over site by site?
19:35:27 <AJaeger> clarkb: I did the first jobs already, happy to hear that mnaser will do the rest!
19:35:43 <AJaeger> clarkb: https://review.opendev.org/#/q/status:open+project:openstack/project-config+branch:master+topic:static-services
19:35:47 <clarkb> thanks!
19:35:49 <ianw> ok, i can look at the volumes for ^
19:36:29 <AJaeger> getting those jobs done helped to answer a couple of questions and finish the spec ;)
19:36:43 <clarkb> I think we should be able to have both sites exist in parallel and then flip dns when we are happy with the new side of things
19:36:45 <AJaeger> please check also whether those jobs are really doing what we wanted
19:38:10 <clarkb> And finally we get to ask.openstack.org
19:38:21 <clarkb> #link http://paste.openstack.org/show/785843/ Some logs
19:38:37 <clarkb> these logs (that frickler collected?) seem to show there are segfaults
19:40:30 <ianw> "or similar nasty error"
19:40:35 <ianw> that's oddly unspecific
19:41:09 <clarkb> one thing we can try if it is the log rotate that causes this is to ensure we are using copy truncate
19:41:13 <clarkb> and not the rotate in place behavior
19:42:04 <ianw> systemd-sysv-generator[6394]: Overwriting existing symlink /run/systemd/generator.late/jetty8.service with real service.
19:42:08 <ianw> what does that mean?
19:42:22 <ianw> it must be coming from puppet, it's very frequent
19:43:11 <clarkb> ianw: if jetty is a sysv init script that may be a result of it setting up the compat shims
19:45:15 <ianw> there's a /etc/init.d/jetty8~ file
19:45:51 <ianw> i've removed that
19:45:55 <clarkb> k
19:46:06 <ianw> /etc/init.d/jetty is a symlink to /etc/init.d/jetty8
19:46:53 <ianw> istr doing that as a total hack on the xenial upgrade
19:47:20 <ianw> https://cgit.freedesktop.org/systemd/systemd/tree/src/sysv-generator/sysv-generator.c?id=4e5589836c9e143796c3f3d81e67ab7a9209e2b0#n166 suggests it's the backup file though
19:49:29 <clarkb> I've not yet had time to look myself other than reading that paste of logs quickly
19:49:42 <clarkb> I'll add it to the todo list to see if anything obvious jumps out at me though
19:50:24 <clarkb> #topic Open Discussion
19:50:28 <clarkb> Anything else?
19:50:33 <ianw> i'll try and keep an eye too ... but this server really does hang on by a thread :/
19:50:37 <clarkb> y
19:50:39 <clarkb> er yeah
19:51:22 <fungi> i've got nothing else
19:51:26 <clarkb> As mentioned before the jet lag has been harsh. I'm trying to get back to a normal schedule but haven't quite managed it yet so will be in and out while I nap :)
19:54:09 <clarkb> #endmeeting