19:01:23 <clarkb> #startmeeting infra
19:01:23 <openstack> Meeting started Tue Mar 31 19:01:23 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:26 <openstack> The meeting name has been set to 'infra'
19:01:28 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006616.html Our Agenda
19:01:59 <AJaeger> o/
19:02:07 <ianw> o/
19:02:15 <clarkb> #topic Announcements
19:02:22 <clarkb> no new announcements this week
19:02:51 <clarkb> #topic Actions from last meeting
19:02:57 <fungi> osf "community meeting" conference call tomorrow
19:03:05 <fungi> er, thursday
19:03:13 <clarkb> #undo
19:03:14 <openstack> Removing item from minutes: #topic Actions from last meeting
19:03:27 * fungi can't remember what day of the week it is at the best of times
19:03:39 <clarkb> fungi: thats the high level overview one right? then the one where meetpad is particularly relevant is april 2, 4 6? /me looks for that email
19:03:51 <fungi> yeah
19:04:12 <clarkb> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013699.html
19:04:44 <clarkb> 2, 6, 7 for time zone coverage on PTG brainstorming
19:04:53 <clarkb> details in that email (which was sent broadly
19:05:08 <clarkb> #topic Actions from last meeting
19:05:16 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-24-19.01.txt minutes from last meeting
19:05:33 <clarkb> fungi took actions to create the new opendev meeting channel and a service-discuss mailing list
19:05:44 <clarkb> those changes are up for review at gerrit topic:opendev-comms
19:05:44 <AJaeger> thanks, fungi!
19:06:24 <fungi> the bikeshed decisions there were to call the new ml "service-discuss" and the new irc channel "opendev-meeting"
19:06:39 <fungi> and also whether to include statusbot in the new irc channel
19:06:44 <clarkb> and we've got a topic a bit later to discuss what that means for the future of this meeting (as well as other things)
19:07:04 <fungi> it might fight with meetbot over channel topics, but i expect our uses of the two won't overlap sufficiently in there for it to become an issue
19:07:49 <clarkb> ya I expect it will be fine and if the topic gets out of sync many channel lurkers will be able to update it
19:07:53 <AJaeger> so, opendev-meeting is only us - not a public place like #openstack-meeting?
19:08:31 <mordred> o/
19:08:34 <clarkb> AJaeger: its public in that anyone can join but the topic/theme of the channel will be focused on opendev meetings and incident management
19:08:45 <clarkb> AJaeger: it won't host arbitrary meetings
19:09:12 <corvus> do we think there might be a desire for a general meeting channel?
19:09:28 <clarkb> it seems that more and more projects are choosing to host meetings in their own channels
19:09:39 <corvus> maybe we just wait until someone asks if they can use it, and otherwise don't worry about it? :)
19:09:39 <fungi> it seems like even in openstack the desire for a central meeting channel has faded
19:09:41 <AJaeger> clarkb: ok, we're on the same page - then, another name might be better since it's a different policy than #openstack-meeting
19:09:45 <clarkb> corvus: ya thats what I am thinking
19:10:27 * fungi is happy to update those changes if someone has a better name, but maybe we can talk about it during the corresponding meeting topic
19:10:34 <clarkb> ya lets continue
19:10:41 <clarkb> #topic Specs approval
19:10:49 <clarkb> we landed the meetpad spec and work there has progressed!
19:11:14 <clarkb> I wanted to call out the xwiki spec had a merge conflict as a result and could use some trivial reacks (though I think I'm comfortable approving it without those if people don't want to double check the erbase)
19:11:19 <clarkb> #link https://review.opendev.org/#/c/710057/ xwiki for wikis
19:12:19 <clarkb> mostly just a heads up there that a new ps has arrived and votes weren't preserved, but I'd like to not hold that back for too long
19:12:25 <clarkb> #topic Priority Efforts
19:12:32 <clarkb> #topic Update Config Management
19:12:43 <clarkb> There are exciting updates to our gerrit with docker changes
19:12:51 <mordred> heck yes
19:12:53 <clarkb> mordred: ^ would you like to fill us in?
19:13:08 * diablo_rojo sneaks in late
19:13:12 <mordred> manage-projects is now mostly working (there's one recent glitch- but we think that was related to a weird manual run)
19:13:14 <mordred> but ...
19:13:16 <mordred> drumroll
19:13:27 <mordred> we're now triggering manage-projects with zuul and not from cron!
19:13:39 <corvus> \o/
19:13:45 <mordred> which means project-config changes for project creation will get applied immediately
19:14:06 <clarkb> ya we've seen ~2 repos be weird, but possibly they are just side effects from when things were broken
19:14:07 <mordred> and - it's showing that our structure for doing infra-root playbooks works - so we shoudl be able to peel off the rest of run_all in pretty short order
19:14:23 <clarkb> we should fix them (probably manually after reading jeepyb) then continue to monitor subsequent things
19:14:27 <mordred> yeah- we're also going to have web-accessible logs - so people should be able to see when something is being weird
19:14:38 <mordred> I agree - I think they are manual pushes
19:15:27 <mordred> but we should start actually seeing logs from project creation on the promote job on the p-c patch - so we should know when stuff goes poorly
19:15:35 <clarkb> ++
19:15:43 <fungi> once that patch also lands, right?
19:15:45 <mordred> in fact- once this is solid ...we might want to have jeepyb start exiting 1
19:15:51 <clarkb> another thing we should probably call out is our use of allowed-projects and semaphores?
19:15:59 <mordred> https://review.opendev.org/#/c/716376/ <-- collecting logs
19:16:19 <corvus> mordred: why have it exit 1?
19:16:31 <fungi> exiting 1 on a creation failure
19:16:32 <mordred> corvus: because that way we'll see red on the job to create the projects
19:16:37 <mordred> yeah - when something fails
19:16:52 <corvus> ack
19:16:54 <mordred> since it'll be running in response to file changes rather than just on a cron
19:17:10 <mordred> yeah - so this: https://review.opendev.org/#/c/716359/ is adding a semaphore since we're also triggering this from project-config
19:17:31 <fungi> though that raises the question, when a change creates multiple repos and one breaks do we want it to keep going or just short-circuit and fail
19:17:32 <mordred> we don't need to do it on all of our infra-prod jobs - but we do when it's a job that could get triggered by more than just system-config
19:17:38 <clarkb> with goaccess we run periodically and have a job timeout that is less than the day period for periodic jobs. that means we largely avoid any overlapping jobs. In general though I think these zuul applied ansible playbooks will want semaphores as production isn't throw away test isntances with overlap :)
19:17:43 <mordred> so - a thing for opendev folks to keep in mind
19:17:56 <mordred> fungi: I think keep going - but report that there was an error
19:18:01 <clarkb> mordred: ya thats the key is if it can be enqueued more than once (supercedent promote pipeline avoids the issue with single repo case)
19:18:11 <mordred> yup
19:18:12 <fungi> so stash error conditions and then summarize at the end
19:18:19 <mordred> yeah. well - it'll alreayd log them
19:18:37 <mordred> so the behavior is currently right - we just want to flag that there was an issue somewhere and the exit code should be non-zero
19:18:42 <mordred> everything else should work great
19:19:04 <fungi> right, that's what my poorly-constructed sentence was attempting to convey
19:19:14 <fungi> thanks for confirming
19:19:14 <mordred> fungi: sentences are english hard being
19:19:52 <clarkb> is there anything else to call out on this subject? are we running gerrit in a container yet?
19:20:55 <mordred> no - we still need to do a restart
19:21:01 <fungi> gerrit2   95054  186 38.0 65169020 23520072 ?   Sl   Mar20 30043:21 GerritCodeReview -Xmx48g -jar /home/gerrit2/review_site/bin/gerrit.war daemon -d /home/gerrit2/review_site --run-id=1584714516.94871
19:21:04 <fungi> nope
19:21:22 <mordred> and there's still a cut list on the etherpad to followup on
19:21:26 <mordred> gerritbot is next on my list
19:21:35 <mordred> then there's some utility cron jobs and stuff
19:21:47 <clarkb> do we have a link to that etherpad?
19:21:58 <fungi> the plan for gerritbot is to move it to eavesdrop as part of this, right?
19:22:08 <mordred> https://etherpad.openstack.org/p/gerrit-2020-03-20
19:22:10 <fungi> or was that going to be a later step?
19:22:16 <mordred> fungi: yeah- well - first step I'm containerizing it
19:22:21 <fungi> got it
19:22:30 <clarkb> #link https://etherpad.openstack.org/p/gerrit-2020-03-20 gerrit on docker todo list
19:22:31 <mordred> then yeah- since it's new ansible - might as well do it on eavesdrop
19:22:55 <corvus> i think i suggested that eavesdrop might be a better long-term home for it, but it's not critical, so whatever ends up being easier :)
19:23:11 <mordred> yeah - I thnik it'll be the exact same amount of work since it's completely net-new ansible
19:23:44 <fungi> eavesdrop or eavesdrop replacement if we want to start migrating the other things there to bionic, i don't suppose it matters
19:24:34 <clarkb> ya we can probably figure out what makes the most sense as it gets ansible dockered
19:25:00 <mordred> ++
19:25:23 <clarkb> ianw: you've also been pushing on the nodepool builder on docker and fedora-31 images. Does it make sense to bring that up here?
19:27:01 * mordred just +A'd the nodepool change to bump dib
19:27:41 <clarkb> probably the biggest thing to be aware of there is that fedora31 images will be joined fedora30 images on the new nb04.opendev.org builder running on docker
19:27:41 <ianw> umm, if that bump is in, then we can merge the configs and i can try things out today
19:27:43 <mordred> also - if people haven't seen the new python-stow element they should look at it - it's super cool and is, I think, a great pttern
19:28:18 <ianw> if that's all going well, we can think about migrating some of the other builds to the container builder, and eventaully replace the other builders
19:28:24 <mordred> ++ \o/
19:28:37 <ianw> then we can think about containerising the launchers
19:28:39 <clarkb> exciting
19:29:38 <clarkb> #topic OpenDev
19:30:26 <clarkb> I've had a think on what the formalization of the opendev leadership and advisory board next steps are and I think we should get the new comms channels up (particualrly the mailing list for this) then use that as a venue to solicit volunteers
19:30:56 <clarkb> that way the history of the thing will be preserved in a place where people are likely to go looking for it in the future
19:31:31 <clarkb> in addition to that I think it would be great if we can have our next meeting in the new channel
19:31:31 <fungi> i concur
19:32:00 <mordred> ++
19:32:24 <clarkb> so if you've got a moment it would be great to brainstorm the channel name (here for a few minutes or on the change(s))
19:32:47 <clarkb> however, I expect the mailing list addition is less contentious and maybe we can land that one more quickly
19:33:27 <fungi> what was the other suggestion, #opendev-annex?
19:33:45 <clarkb> fungi: or -auxilliary (except its hard to spell and I'm sure I got it wrong there)
19:34:02 <frickler> can you post the links to the reviews for reference?
19:34:07 <clarkb> annex is easy to spell and short I'm on baord with that too fwiw
19:34:10 <fungi> i'm sure we'd just shorten that to #opendev-aux anyway
19:34:31 <clarkb> #link https://review.opendev.org/#/q/topic:opendev-comms Opendev comms changes
19:34:42 <fungi> #link https://review.opendev.org/715972 Add a service discussion mailing list for OpenDev
19:34:43 <clarkb> fungi: ^
19:35:14 <fungi> ahh, thought he wanted them individually linked, but sure
19:35:23 <fungi> #undo
19:35:25 <clarkb> er that was for frickler but tab complete failed :)
19:35:31 <clarkb> individual links are fine too :)
19:37:02 <clarkb> then for the actual leadership position we can make a call for volunteers on the new mailing list and if we have more than 1 run an election among opendev contributors. I'm personally happy to continue in this role, but would also be happy to see others take it on as well :)
19:37:59 <fungi> if we're worried about folks mistaking the #opendev-meeting channel for a general meeting space, then we can likely just clarify its focus in theh /topic
19:38:18 <corvus> this all sounds good to me; i don't have a preference on the secondary channel name
19:39:01 <fungi> yeah, i'm not going to bikeshed over the name but if folks want to propose alternatives i'm fine with us entertaining our options
19:39:34 <fungi> i gave it some thought before i pushed the patch and couldn't come up with anything better either, fwiw
19:39:35 <clarkb> fungi: that sounds like a plan. AJaeger ^ perhaps you can suggest an alternative and we take it from there?
19:40:16 <frickler> you should -w the patches if you want further discussion on them, I almost approved them this morning already
19:40:19 * AJaeger is bad at naming
19:41:15 <clarkb> I think the mailing list can land as is, I've not heard any concerns about that one, then maybe -W the meeting channel change for now in order to continue discussion on name options in gerrit?
19:41:17 <clarkb> fungi: ^
19:41:36 <AJaeger> clarkb, fungi, give it two days, if no names come up, we merge...
19:41:52 <clarkb> AJaeger: wfm
19:42:29 <clarkb> That takes us to the last opendev item I had on the agenda. A heads up about improving documentation to give peopel ending up at https://opendev.org a bit more context on where to go next
19:42:31 <fungi> sure, i can wip the system-config change for the irc channel
19:42:57 <fungi> er, i guess the project-config change actually as that's the earlier onw
19:43:01 <clarkb> so far we've added a getting started link to the links on the nav bar in gitea. This takes you to the opendev infra manual getting started doc which is a simplified version of the dev docs
19:43:45 <clarkb> I expect there is more we can do in this space, whether that is improving themeing of the docs to feel more "cohesive" or updating content to better address the neds of people arriving from gitea
19:44:23 <corvus> that sounds great :)
19:44:43 <clarkb> oh also we removed the openstack theme from the infra manual
19:44:57 <clarkb> it uses the alabaster sphinx theme now with an opendev logo
19:45:33 <clarkb> definitely interested to hear feedback on this as I think one issue I suffer from is familiarity with the system means I gloss over the docs :)
19:45:48 <clarkb> ideas are good!
19:46:39 <clarkb> before we move on, I'm not sure if it was clear before but next week I'd like to have the opendev meeting in the new channel but will continue to keep the existing time slot
19:46:50 <clarkb> I'll be sure to send out specifics when I send an agenda out
19:46:58 <clarkb> #topic General Topics
19:47:19 <clarkb> fungi: anything to report on wiki? I know you indicated you would try to keep updating the puppet as a guide for potential replacement tooling
19:48:23 <fungi> nope
19:48:31 <fungi> not yet anyway
19:48:59 <clarkb> #topic Open Discussion
19:49:05 <clarkb> Anything else?
19:50:26 <clarkb> I'm excited about the progress we've made over the beginning of the yaer. Feels like a lot of the plans and ideas we've hard are coming together
19:50:32 <mordred> ++
19:50:45 <clarkb> also I can't type
19:50:52 <corvus> typing is hard
19:51:35 * fungi types harder
19:51:42 <frickler> what are the next steps on meetpad? any help needed there?
19:53:08 <corvus> yeah, it's up and running, but we found in testing that the etherpad function wasn't working in some (maybe even most) cases
19:53:30 <clarkb> corvus: mark tried ti and discovered there isn'
19:53:32 <corvus> it was returning errors, so it's pretty obvious when it doesn't work
19:53:32 <clarkb> er
19:53:39 <clarkb> isn't an http -> https redirect? we may want to add that too
19:53:43 <corvus> clarkb: correct
19:53:47 <corvus> and http doesn't work at all
19:53:51 <corvus> so we need to add that
19:53:57 <corvus> that should be easy
19:54:14 <corvus> the hard part is figuring out what's (sometimes) broken about etherpad
19:54:47 <fungi> this is presumably also newer etherpad than on our existing etherpad.o.o
19:54:51 <corvus> so if anyone wants to visit it (via https) and see if they get an error from etherpad, and see if they can figure out what the problem is, that would be great
19:54:59 <corvus> fungi: no, it is our existing etherpad.o.o
19:55:08 <corvus> that's the main attraction of the service
19:55:14 <fungi> oh, okay
19:55:21 <clarkb> one idea that came up on friday was maybe we need to update etherpad though
19:55:22 <fungi> so not a separate containerized etherpad deployment
19:55:26 <corvus> correct
19:55:37 <clarkb> jitsi is passing flags that our etherpad doesn't seem to honor so it may have other integration points that we need to accomodate
19:55:41 <fungi> we probably do need to update etherpad anyway
19:55:57 <corvus> clarkb: true, but this is only a sometimes failure
19:55:58 <fungi> however, worth keeping in mind that they've also changed the theme in etherpad recently
19:56:03 <fungi> so it's going to look a lot different
19:56:08 <clarkb> corvus: ya I doubt it will fix the underlying problems
19:56:58 <frickler> o.k., I'll try to take a look tomorrow
19:56:59 <fungi> etherpad-dev.o.o is offline too, i can take a look at that after the meeting
19:57:06 <corvus> it's possible, but since it sometimes works and sometimes fails, it means that if the problem were to be solved by an etherpad upgrade, it'd have to be some kind of stabilization or race thing, not a straightforward incompatibility.
19:57:42 <corvus> the good news is that our deployment mechanism worked flawlessly the first time.  including LE certs.  :)
19:58:03 <fungi> we're getting good at this
19:58:06 <corvus> though due to a bug i introduced in the config, it's currently in the emergency file until https://review.opendev.org/715572 lands
19:58:51 <clarkb> that change has now been approved
19:59:22 <clarkb> and we are just about at time.
19:59:33 <clarkb> Thanks everyone! feel free to continue conversation on irc or the mailing list
19:59:43 <clarkb> and we'll see you next week hopefully in the new location!
19:59:52 <clarkb> #endmeeting