19:01:13 <clarkb> #startmeeting infra
19:01:14 <openstack> Meeting started Tue Feb 25 19:01:13 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:18 <ianw> o/
19:01:18 <openstack> The meeting name has been set to 'infra'
19:01:22 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-February/006602.html Our Agenda
19:01:28 <AJaeger> o/
19:02:08 <clarkb> #topic Announcements
19:03:09 <clarkb> I'll be out tomorrow (going to try my hand at fishing) and then thursday morning I visit the tax man. Mostly just a heads up that like last week this week is a weird schedule for me. I'm trying to get stuff out of the way for next week (when i think fungi will not be around)
19:03:27 <clarkb> #topic Actions from last meeting
19:03:33 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-02-18-19.01.txt minutes from last meeting
19:04:00 <clarkb> This didn't get recorded as a formal action, but I volunteered to write a spec for the pip and virtualenv python situation with our test images. I did do that so lets dive right into specs
19:04:07 <clarkb> #topic Specs
19:04:08 <ttx> o/
19:04:24 <clarkb> #link https://review.opendev.org/#/c/709579/ Spec for cleaning up python dev tools on our CI images.
19:04:40 <clarkb> This comes with a variety of supporting documentation and changes. I'll post links for them in a sec
19:04:57 <corvus> clarkb: thanks, i haven't had a chance to review it yet, but that looks very helpful
19:05:02 <clarkb> It would be great if those involved in untangling the virtualenv upgrade mess could read that over
19:05:25 <clarkb> What I'd like to do is get it into shape then start passing it around with the constituent projects using the CI system
19:05:47 <clarkb> basically overcommunicate the chagne since it has potential to be a painful change. Though the plan is to make it as unnoticeable as possible
19:05:55 <clarkb> #link https://etherpad.openstack.org/p/pTFF4U9Klz : clarkb writeup of major issues
19:06:00 <clarkb> #link https://review.opendev.org/707499 : discussion in comments
19:06:04 <clarkb> #link https://review.opendev.org/707513 : use venv for glean
19:06:10 <clarkb> #link https://review.opendev.org/707750 : use venv in project-config elements; drop pip-and-virtualenv inclusion from element and move to individual configs; add node type with no pip-and-virtualenv.  this can be used for job testing.
19:06:25 * diablo_rojo sneaks in late
19:06:27 <clarkb> we should probably roughly agree to the plan in the spec before we start landing those early safer changes
19:06:51 <clarkb> I do think we can land the last two changes linked without buy in from users as the first is glean specific and the second does this in a testing capacity
19:07:25 <fungi> i hope it won't be as painful nor drawn-out as the migration off thick test images
19:07:42 <fungi> seems likely to have less overall impact at least
19:08:08 <clarkb> I think the end result will be fewer surprises for test jobs as our nodes will act a bit more like the ones you get from centos and ubuntu
19:08:13 <ianw> ++ will go through today
19:08:21 <fungi> even if it impacts more projects in total, it does so in more standardized ways we can work out solutions for
19:09:17 <clarkb> I look forward to your feedback. Thank you
19:10:23 <clarkb> Then we have coincidental overlap in the need for updating our simple 404 tracking (and website trackign in general). I had an afternoon with a couple foundation people where the subject came up and ianw had a change up to remove our script as it requires writing to afs volumes and complicates the files to static.o.o migration
19:10:54 <clarkb> This prompted me to look into options for tools around there. When I was debugging haproxy logs at one point I had hacked up a goaccess config to render graphs for me and it supports apache logs out of the bogx
19:11:05 <clarkb> This led to writing another different spec
19:11:10 <clarkb> #link https://review.opendev.org/#/c/709236/ Website activity stats
19:11:42 <fungi> yeah, goaccess seems really promising. when i was looking into/testing solutions for this i didn't find it
19:11:44 <clarkb> I think the key here is to agree on goals more so than specific technical details. For example are we all comfortable with publishing the stats planned there
19:12:17 <fungi> it also serves as a great first test for our new opendev governing body
19:12:41 <clarkb> ianw has had good feedback for implementation ideas and I think we can edit the spec as necessary to reflect that. What I don't want to do is surprise anyone with public stats reporting that might be too generous (though I think the plan is quite conservative and should avoid any issues)
19:12:51 <corvus> did ianw push up a change to partially implement this?
19:13:07 <ianw> in the mean time, if we'd like to restore the existing test and also do a little POC on zuul collecting these stats we have
19:13:10 <corvus> (i thought i saw something like a change that logged into static.o.o and did something via a zuul job)
19:13:11 <ianw> #link https://review.opendev.org/#/c/709382/
19:13:24 <clarkb> corvus: ya it basically takes our old 404 only reporter and adopts it to the newer hosting setup
19:13:44 <corvus> ah cool.  so that's that part of the process -- then if we like goaccess, we use that mechanism to run it?
19:13:49 <clarkb> I think that is a good in between spot to avoid regressing on the 404 reporting functionality while we spin up the richer reporting
19:14:03 <clarkb> corvus: yup exactly. One of the things goaccess can do is 404 reporting
19:15:05 <clarkb> corvus: if you have time to review the spec it would be great to get your perspective with a zuul hat on
19:15:11 <corvus> on it
19:15:31 <corvus> i'm also going to wear my "don't share pii hat"
19:15:34 <clarkb> ++
19:15:47 <fungi> i stapled that hat to my head already
19:16:13 <clarkb> Anything else on specs or should we continue?
19:16:23 <fungi> much of the plan there is a result of discussions we had in #openstack-infra about things we shouldn't report because of that
19:17:04 <clarkb> #topic Priority Efforts
19:17:07 <clarkb> #topic OpenDev
19:17:31 <clarkb> The openstack governance change has four TC Acks. I think we need 3 more votes in order to land that
19:18:06 <clarkb> don't be surprised if that lands in the near future :)
19:18:28 <clarkb> On the gitea side of things mordred has a change up to upgrade to 1.11.1
19:18:39 <clarkb> it is currently failing testing, I haven't had a chance to look at why yet though
19:18:40 <clarkb> #link https://review.opendev.org/#/c/705807/ Upgrade Gitea to 1.11.1
19:18:59 <clarkb> I did go through the release notes and called out a couple that I thought might impact us but on further investigation we should be fine (details in a comment on the change)
19:19:16 <mordred> clarkb: I set up an autohold
19:19:19 <mordred> because I can't tell from logs
19:19:34 <mordred> hopefully I'll both figure it out - and figure out how to make it possible to figure out from logs :)
19:19:50 <clarkb> GEtting to 1.11 will be our stepping stone to 1.12 which has the git commit cache support
19:20:13 <clarkb> we should probably consider deploying 1.12 pre release, but we can figure that out once on 1.11
19:20:27 <fungi> and that's what we hope will significantly speed up ui rendering for large repos?
19:20:32 <clarkb> fungi: yes
19:20:39 * fungi does a little dance
19:21:32 <clarkb> Anything else on OpenDev?
19:22:17 <clarkb> #topic Update Config Management
19:22:36 <clarkb> mordred: I believe there has been a slight change of direction in the way we build gerrit images. Want to fill us in?
19:23:05 <mordred> yeah - so - the previous way was based on a bazel builder image
19:23:20 <mordred> so that we could make an image that had things like bazel and java and whatnot pre-installed and just focus on building gerrit
19:24:02 <mordred> UNFORTUNATELY - bazel is still a quick moving target, so the gerrit folks have a .bazelversion file in repo that contols which verison of bazel should be used - and we were constantly chasing job breakage when the file would not match our buiklder image version
19:24:38 <mordred> so instead we're now using copies of roles corvus wrote for upstream gerrit which does the build on the build node using bazelisk - which installs the right version of bazel
19:24:46 <mordred> and then the dockerfile just copies in the war file built from that
19:24:59 <mordred> should be more future proof - as well as handling different bazel versions across different branches
19:25:02 <mordred> \o/
19:25:14 <clarkb> The biggest impact to us as individuals is we'll have to do the bazelisk pre step if building the images locally
19:25:24 <mordred> yeah
19:25:43 <mordred> but that honestly doesn't happen super often
19:25:52 <clarkb> mordred: did those chagnes land? put another way are there things that need review?
19:25:54 <corvus> yeah.  we could move that into a docker image build, but there'd be less direct upstream role reuse (but they're not that complicated so maybe it's not a big deal)
19:25:54 <fungi> but also we're better aligned with how gerrit builds gerrit?
19:25:59 <mordred> also - change up to add 3.1 to the list of gerrits we build
19:26:01 <mordred> https://review.opendev.org/#/c/709295/
19:26:16 <mordred> clarkb: the changes other than 3.1 landed
19:26:17 <corvus> in general, i'm in favor of going with the hybrid approach mordred has up now, and moving stuff into containers if we feel we need it.
19:26:25 <mordred> corvus: ++
19:26:30 <clarkb> #link https://review.opendev.org/#/c/709295/ Build a Gerrit 3.1 docker image
19:27:17 <corvus> fungi: it's a little hard for me to say how "gerrit builds gerrit"
19:27:43 <clarkb> any other config management updates to call out?
19:28:05 <mordred> oh - fwiw...
19:28:10 <corvus> for some values of that (ie, how a dev might expect to build it on their workstation) i'd say yes.  for other values (how they build release artifacts, container images, etc), i'd say this is different, but that we don't want to emulate that due to the high amount of gerritforce-ci-specific stuff involved.
19:28:27 <mordred> we found an issue with LE certs and review.opendev.org today that isn't super important but thought I'd mention the root cause ...
19:28:39 <fungi> corvus: ahh, thanks for the clarification
19:28:41 <mordred> we weren't making the LE certs for review.o.o that we thought we were (we're not using them yet, so no biggie)
19:28:46 <clarkb> mordred: note I think it may still be haivng problems (needs further investigating)
19:28:56 <fungi> oh, this was probably more for the opendev topic, but new gerrit talk reminded me, i pushed up a change to opendevify the welcome new contributor message:
19:28:58 <fungi> #link https://review.opendev.org/708975 Overhaul default welcome message for OpenDev
19:29:02 <mordred> and the reason is that we had host_vars in ansible for review01.opendev.org ... but the host is review01.openstack.org
19:29:13 <mordred> just to keep in mind as we work on things in the future
19:29:36 <mordred> clarkb: nod
19:29:47 <corvus> mordred: lesson learned: transitions are hard?
19:30:02 <clarkb> dotting all the Ts and crossing all the Is is hard
19:30:06 <mordred> clarkb: yah. as are names
19:30:35 <clarkb> #topic General Topics
19:30:46 <clarkb> ttx: did you want to start us off with the xwiki item?
19:30:52 <ttx> Sure. Hi!
19:30:56 <ttx> As you may know we've had demand for wiki space from a number of projects
19:31:03 <ttx> So far our response has been to ask them to squat on wiki.openstack.org
19:31:11 <ttx> But that is not a long-term solution. And we don't really want to set up more Mediawikis either.
19:31:18 <ttx> So I've been looking into open source wiki solutions that:
19:31:29 <ttx> - would have limited maintenance costs, like stronger spam prevention features
19:31:35 <ttx> - would have more of a "structured wiki" approach, to limit stale pages
19:31:46 <ttx> - would be pluggable into openstackID to avoid maintaining a new set of credentials (or increase our UbuntuOne tech debt).
19:31:55 <ttx> I've experimented with XWiki (see xwiki.org) and found it satisfactory.
19:32:00 <ttx> (or at least as satisfactory as a wiki can be)
19:32:14 <ttx> It's an open source solution, and the fine folks at XWiki.com offer free hosting on their "Xwiki cloud" for open source projects.
19:32:26 <ttx> It allows for "subwikis", which are basically wikis with a root document and search context which live under the same main wiki domain
19:32:44 <ttx> My current thinking is to take them up on their hosting offer, set up the main wiki under wiki.opendev.org, and experiment with a subwiki for the "Open Infra Labs" initiative
19:33:00 <ttx> If that experiment works well, we would extend the possibility of a subwiki for other projects.
19:33:23 <ttx> If it works *really* well, we could envision to transition wiki.openstack.org content and finally abandon Mediawiki maintenance.
19:33:36 <ttx> And if it works *too* well and we get past the open source offer hosting limits (10 subwikis) then we could always look at hosting a Xwiki instance ourselves.
19:33:51 <ttx> You can see a demo at: https://opendevwiki.demo.xwiki.com/
19:33:55 <ttx> You can log in at the topright corner
19:34:02 <mordred> ttx: the subwikis would be subdomains of opendev.org ?
19:34:05 <ttx> Opendev gets the wiki at the top level, which is used to list the subwikis, but could also be used to describe more of Opendev if you want.
19:34:12 <fungi> i wonder how this relates to discussions we've had about possibly leveraging the github-wiki-like feature in gitea
19:34:15 <ttx> no, more of a subdirectory
19:34:36 <ttx> But it looks like the root of a wiki. So if you search within a subwiki, it looks for local results
19:34:36 <mordred> ttx: nod. but still contained within the wiki.opendev.org domain
19:34:42 <mordred> gotcha
19:34:44 <ttx> Each subwiki can have a different theme and panels, which I tried to show in the demo.
19:35:08 <ttx> The demo has superlong URLs but I suspect they can collapse that a bit
19:35:15 <mordred> fungi: so far the github-wiki-like things in gitea seem to assume people need commit access to the git repo to make wiki changes - last I checked
19:35:23 <clarkb> operationally I think it would run similarly to gerrit (its a monolithic java app with a db connection)
19:35:31 <clarkb> if we end up needing to run it ourselves
19:35:32 <fungi> mordred: yeah, and is tied to specific repos i guess
19:35:35 <mordred> yeah
19:35:39 <ttx> https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/ContentOrganization/ explains subwikis vs. nested pages
19:35:49 <ttx> Should we pursue this, I volunteer to conduct the experiment
19:36:04 <mordred> I would be more than happy to not run a wiki
19:36:10 <ttx> They agreed to allow us to use a custom domain, which would make transitioning in the future a lot easier
19:36:27 <fungi> it also touches on uor still as-of-yet unresolved need for an opendev sso
19:36:32 <ttx> but yes, if the hosting works for us, I'd rather continue to have them run it
19:36:34 <fungi> s/uor/our/
19:36:44 <mordred> fungi: yeah. that's the biggest issue I can think of
19:36:48 <ttx> and encourage them to have a successfl wiki hosting business rather than continue developing "Pro" plugins
19:36:57 <corvus> fwiw, i didn't know about either the demand for wikis, or the openinfra labs project; hopefully with the new opendev governance model we can improve those communication channels
19:37:34 <ttx> corvus: we had Airship and StarlingX ask for wikis, we just redirected them to wiki.openstack.org
19:37:44 <fungi> most of the "demand for wikis" is that every time we say we're going to wind down wiki.openstack.org we get a lot of folks jumping up and down saying we can't take it away because they have no good alternative
19:38:02 <ttx> OpenInfraLabs is the new name for the Massachussets Open Cloud, extended beyond MA
19:38:07 <corvus> oh neat
19:38:18 <mordred> ttx: fungi's thing is the main thing I'd be worried about that I think we'd need to come up with an answer for
19:38:35 <corvus> mordred: auth?
19:38:38 <mordred> yeah
19:38:46 <ttx> It's new, but one of the first things they asked for wasa wiki (don't ask me why, I hate wikis)
19:38:51 <corvus> because in a hosted system, transitioning auth would be difficult/impossible?
19:39:07 <mordred> corvus: or someone else's problem we can just throw money at? :)
19:39:08 <clarkb> ttx: is it possible to allow arbitrary sso? so that users can choose what identity they log in with (github/openstackid/ubuntuone/etc)
19:39:54 <clarkb> that might be a reasonable intermediate step? That assumes people don't want to tie wiki edits back to CCLA or similar
19:40:06 <corvus> mordred: yeah, a few possibilites there, and knowing the answers before we start would be good
19:40:09 <ttx> clarkb: Not 100% sure.
19:40:19 <mordred> corvus: or - at least list that slving that is likely to be step 0
19:40:32 <ttx> By default when unconfigured it allows people to provide a OID URL in an ugly form
19:40:46 <ttx> But once configured it just forwards to openstackID very nicely
19:41:00 <corvus> depending on the answer, maybe we can proceed full steam ahead and clean up auth later, or maybe we have to implement the opendev multiplexer first
19:41:09 <mordred> I'd be happy to say "yeah, let's go for it, must solve auth"
19:41:11 <mordred> corvus: yeah
19:41:35 <ttx> I'm not super-concerned with migrating the few users we'll have in the PoC
19:42:04 <fungi> if there's sufficient admin-level access to dump the equivalent of the user auth ids, group memberships and acls, then we can in theory punt on that
19:42:04 <mordred> ttx: so we believe our xwiki friends can migrate a set of users for us should we need such a thing to be done?
19:42:13 <mordred> or fungi's thing
19:42:23 <corvus> good point. i don't think it has to block poc, but we should explore the question during the poc and have an answer before prod
19:42:29 <clarkb> corvus: ++
19:42:30 <mordred> ++
19:42:50 <ttx> mordred: I think we can also live with users being duplicated during transition
19:43:09 <corvus> yeah, that's a valid answer too :)
19:43:11 <mordred> yeah. I mean - also - it'sa . wiki, so it's not like it needs tight auth integration with anything else we do
19:43:18 <fungi> probably the biggest question for opendev is whether we're okay with the opendev name attached to that hosted service... i see this as the same question for the hosted zanata replacement we've been discussing as well
19:43:26 <ttx> I mean, they have been helpful so far, but the "open source offer" comes with minimal support
19:43:56 <fungi> i read that as "we'll host this for you and even offer some support"
19:44:05 <fungi> which sounds a lot better than no support
19:44:18 <ttx> but I'm trying to pay them back with logos and user studies, so maybe they will be happy with us. I know when the OSI calls them they are VERY helpful
19:44:28 <ttx> (OSI being another xwiki cloud user)
19:44:37 <mordred> yeah- we'd be a pretty high profile user for them
19:45:00 <ttx> It's more a "no guarantee" offer than a "no help"
19:45:05 <corvus> i also like that hosting it on a custom subdomain means we can be in control of a migration (which might even be seamless).  i think that's an important point.  also a question to explore during the poc is what a migration to self-hosting would look like (exploring data exports, etc)
19:45:22 <clarkb> as a time check we have several more topisc to get through (we should probably get to them when we have 10 minutes remaining)
19:45:28 <ttx> corvus: yes, I would not have considered it if they said custom domain was not a possibility
19:45:30 <fungi> and the xwiki project is under the ow2 consortium, who have been great friends to openstack in the past
19:45:42 <mordred> another question for POC - SSL
19:46:00 <corvus> mordred: ++
19:46:06 <fungi> yeah, i'm cool saying go forth and poc
19:46:09 <mordred> ++
19:46:09 <clarkb> I wonder if we can point an acme record at them and then they LE
19:46:14 <clarkb> but definitely somethign to check on
19:46:30 <ttx> Not sure what they will ask on that. Did not want to go any further without your blessing
19:46:42 <corvus> ttx: can you write up a spec with questions for the poc period?
19:46:49 <corvus> don't block on the spec
19:46:50 <ttx> If you say "OK" we'll tread carefully
19:46:53 <fungi> it can be a very short spec, probably
19:46:56 <ianw> they can probably do the file/webserver based auth for LE?
19:47:07 <corvus> more of a place where we can collect questions, answers, and develop plans for if we go to prod
19:47:11 <clarkb> ianw: ah yup
19:47:14 <ttx> ok, open questions on the poc
19:47:40 <ttx> not sure what LE is
19:47:46 <clarkb> ttx: letsencrypt
19:47:49 <ttx> hah
19:48:48 <ttx> I'll stop hijacking the meeting :)
19:49:26 <fungi> thanks ttx!
19:49:35 <corvus> can we agree on action items for 1) start the poc; 2) start a spec to collect feedback?
19:49:36 <ttx> action me on a spec, and get them to set up the instance, coordinate with clarkb on getting SSL done
19:49:49 <clarkb> #action ttx write xwiki spec to capture POC questions
19:49:55 <fungi> i agree on 1&2, yes
19:50:11 <clarkb> #agreed ok for ttx to proceed with xwiki POC while we review the spec
19:50:18 <corvus> ttx: yay thanks!
19:50:51 <clarkb> Next up is a note that I did request space at the vancouver PTG. It sounded like we'd have ~3.5 people there and that worked out in shanghai pretty well. I think my goal will be to have some official time and space as that allows others to find us easily then we can organize in a more ad hoc fashion for the other times which worked out in shanghai.
19:50:52 <ttx> thanks all!
19:51:15 <clarkb> If you'll be in vancouver feel free to find us!
19:51:31 <fungi> i'll wear something bright
19:51:49 <ttx> you always wear somethign bright
19:52:06 <clarkb> I expect the weather will be excellent when we are there so maybe we can do something fun as a group too
19:52:20 <clarkb> fungi: any progress with the existing wiki server management?
19:52:22 <clarkb> (speaking of wikis)
19:52:33 <fungi> clarkb: nope, nope, nopeski
19:52:56 <clarkb> And that takes us to the files.openstack.org and static.openstack.org migration to static.opendev.org
19:53:09 <clarkb> ianw: I think docs.opendev.org is the last remaining proper site then we have to add redirects?
19:53:14 <clarkb> link https://review.opendev.org/#/c/709402/1 Move docs.opendev.org to static.opendev.org
19:53:17 <clarkb> #link https://review.opendev.org/#/c/709402/1 Move docs.opendev.org to static.opendev.org
19:54:08 <ianw> yes, that docs one should be quick
19:54:20 <clarkb> ianw: AJaeger and everyone else that helped: Thank you for pushing this along
19:54:20 <ianw> there's a few things open
19:54:21 <ianw> https://review.opendev.org/#/q/status:open+topic:static-services
19:54:52 <ianw> files.openstack.org got caught up in this too, so we might as well push on moving that too.  the only thing remaining there is the git redirectors, which have a change to migrate up
19:55:13 <clarkb> #link https://review.opendev.org/#/q/status:open+topic:static-services Finish up static.openstack.org migration
19:55:25 <fungi> also we figured out yesterday that we're not providing backward-compatible redirects for projects outside the openstack namespace. i meant to try and find more of those besides osf/openstackid but haven't tried to match up the lists yet
19:55:52 <ianw> we've discussed it before, but i'm not sure the idea of the service load balancer is getting traction, the reviews have been there for several months
19:55:56 <fungi> for tarballs.o.o redirects i mean
19:56:08 <ianw> if we want to just punt on it and move it into apache config, i don't mind
19:56:48 <clarkb> ianw: I expect that is fungi's preference. I think I've largely been convinced that apache would be fine particularly since we don't have to juggle ssl
19:57:07 <fungi> my only real objection to the haproxy idea is that we're already running apache on the servers we'd be redirecting to, so could accomplish the same with some aliases in existing vhost configs
19:57:10 <ianw> it's certainly simpler, and we can save the haproxy setup for something when we need more of the load-balancing features
19:57:45 <fungi> and one less service to manage if apache is already required either way
19:58:33 <clarkb> And we are just about at time
19:58:36 <fungi> i mean, i think using haproxy for redirects is probably a fine solution and would work
19:58:37 <clarkb> #topic Open Discussion
19:58:47 <clarkb> I'll open the floor if there is anything else in the last minute
19:58:57 <ianw> ok, i'll work up a change to put them in apache, i'm not really fussed i just want it finished now :)
19:59:09 <clarkb> ++ to finishing
19:59:13 <fungi> thanks ianw!
20:00:06 <clarkb> And that is all we had time for. Thank you everyone.
20:00:08 <clarkb> #endmeeting