19:01:06 <clarkb> #startmeeting infra
19:01:07 <openstack> Meeting started Tue Aug 25 19:01:06 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:10 <openstack> The meeting name has been set to 'infra'
19:01:19 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2020-August/000079.html Our Agenda
19:01:28 <ianw> o/
19:01:33 <clarkb> As mentioned lets see how many manage to make it today before digging in too deep
19:02:00 <clarkb> #topic Announcements
19:02:04 <clarkb> There are no announcements
19:02:12 <clarkb> #topic Actions from last meeting
19:02:20 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-08-18-19.01.txt minutes from last meeting
19:02:25 <clarkb> There were no actions from last meeting
19:02:52 <clarkb> Thats the bookkeeping done. I'll give it a few minutes to see if anyone else makes it then continue from there
19:05:26 <clarkb> ianw: zbr: may just be the three of us. Should we do a less formal agenda then?
19:05:36 <zbr> sure
19:06:04 <ianw> ++
19:06:17 <clarkb> cool I'll start with the item I want to bring up then will open it to other items
19:06:21 <clarkb> #topic Non master repo HEAD support for new projects
19:06:31 <clarkb> #link https://review.opendev.org/741277 Needed in Gerritlib first as well as a Gerritlib release with this change.
19:06:40 <clarkb> #link https://review.opendev.org/741279 Can land once Gerritlib release is made with above change.
19:06:57 <clarkb> Calling this out as it came up in the board meeting today (though I didn't manage to attend so can't provide direct details)
19:07:41 <clarkb> I also plan to attend the diversity and inclusion working group meeting on monday to talk about the technical aspects of any potential changes to git branch naming
19:07:46 <zbr> clarkb: maybe there is a chance to also get https://review.opendev.org/#/c/729966/ into the new release?
19:08:18 <clarkb> zbr: I can review that one too ya
19:08:55 <ianw> i don't think i've looked over gerritlib code before, but it lgtm, and mnaser has looked too, i think it's as gtg as ever
19:09:02 <clarkb> ianw: thanks
19:09:34 <clarkb> its been good to be able to point to steady progress on this as people have higher level conversations around the topic
19:09:40 <clarkb> anyway thats all I had on this subject
19:09:52 <clarkb> ianw: zbr anything you'd like to bring up?
19:10:15 <ianw> nope; lgtm.  i'm sure there's things lurking if it gets changed, but just have to deal with it as it comes up
19:10:42 <ianw> known unknowns, unknown unknowns and all that :)
19:10:57 <clarkb> indeed
19:11:35 <clarkb> #topic Github 3rd Party CI
19:11:50 <clarkb> ianw: anything new on this? I see commentary about arm64 focal openafs in the other channel :)
19:12:45 <ianw> umm, since last week we dug deeper on the wheel generation issues, and found it wasn't exactly the toolchain, but actually patchelf, as part of the wheel construction, messing the alignment
19:13:22 <ianw> #link https://github.com/pypa/manylinux/issues/735
19:13:45 <ianw> that has stalled a bit as the changes need to make it through a few projects
19:14:02 <ianw> and then the other angle has been pyca/cryptography pushing rust into the build
19:14:47 <ianw> so zuul jobs now has an ensure-rust, so we're ok with our extant arm64 testing ... the wheel story, something about rust and musl libc and the builder that i lost track of a bit
19:15:01 <ianw> i don't think it quite works even on x86 atm
19:15:38 <ianw> and upstream switched from travis-ci.org to travis-ci.com as well, which somehow apparently gives them access to arm64 hardware in aws
19:16:22 <ianw> so ... while a lot happened, in terms of "is there a .whl file for arm64" the answer is still no :)
19:17:32 <ianw> oh, and the focal mirror ... yeah the bionic arm64 mirror turned itself off again too
19:18:02 <ianw> getting the focal mirror up has been an exercise too, ipv4 broke in the cloud which kevinz had to fix
19:18:28 <clarkb> I know back in the early days debian was considered to be the most stable on arm64 (or at least I seem to recall that)
19:18:36 <clarkb> do we need to consider maybe running more debian?
19:19:00 <ianw> not sure, the oops appears to be from openafs, and at least to an initial search, appears to be unique
19:19:50 <ianw> one of these great things we seem to hit where the only occurrence of the error message on google is the error string in the actual code
19:19:59 <clarkb> nice
19:20:52 <ianw> https://www.google.com/search?nonzero+refcount+in+shutdown_osisleep
19:20:54 <ianw> :)
19:21:32 <clarkb> anything else on this subject?
19:21:54 <ianw> no, sorry progress has been slow.  it's breaking a lot of new ground in various places
19:22:00 <clarkb> no worries
19:22:08 <clarkb> #topic Gerrit Upgrade Thoughts/Ideas/Info
19:22:14 <clarkb> I continue to poke at Gerrit upgrade things
19:22:42 <clarkb> I looked at review-test and noticed that the git repos are currently on the root device for the review-test server which is different than production gerrit. In prod we hvae the git repos on a fast cinder volume
19:23:12 <clarkb> I think we may want to build a new review-test and better line up the git repo locations as that will likely be important for timing data as well as having enough head room to do the notedb migration
19:23:33 <clarkb> that got me thinking about how viable keeping in sync with prod is if we're doing snapshots and rolling back
19:23:56 <clarkb> Mostly wondering if we should pick a point in time and not worry about keeping in sync with prod to simplify (that should still give us a really good idea for disk usage and migration times)
19:24:14 <clarkb> Separately I started digging into what notedb means for replication configs
19:24:36 <clarkb> for the account and group data we'll already avoid replicating them with our exisitng config due to the replicatePermissions = false setting on our replication configs
19:24:44 <clarkb> I think that is a good thing at least to start
19:25:05 <clarkb> the change notedb data goes in refs/changes/XY/ABCDXY/meta though
19:25:38 <clarkb> which means we'll be replicating all of that and i odn't know of a way to construct a refspec that replicates res/changes/XY/ABCDXY/1 2 3 etc but not meta
19:26:15 <clarkb> I think it would be fine to replicate the metadata fwiw. Just worry about disk usage on the gitea mirrors if we do that (we can check repo sizes for that during testing with prod data I guess)
19:26:19 <ianw> are you supposed to replicate that?  like are there client tools that might show interesting things if you have it locally?
19:26:44 <clarkb> ianw: so far it seems that the only things that really consume it are other gerrit servers
19:26:49 <clarkb> for active standby setups
19:27:56 <clarkb> that got me diving into how the new notedb setup stores data
19:28:20 <ianw> i wonder if it ends up that much bigger?  it would really only be the change descriptions right?  the refs/changes we are already doing?
19:28:37 <clarkb> ianw: its all of the code review comments
19:28:46 <clarkb> which ya for most change swon't be that large
19:29:14 <clarkb> the other potential issue there is the increase in refs
19:29:22 <clarkb> gitea had trouble with that before but has gotten much better
19:29:51 <clarkb> One thing I noticed digging around in the notedb storage details is that All-Users:refs/meta/external-ids may need special permissions changes
19:30:15 <clarkb> I need to ask luca about that one. They do special perms for refs/users but not the external-ids and I expect they want that
19:30:32 <clarkb> basically a long winded way of saying I've learned a lot about notedb recently
19:31:19 <clarkb> Hope to send another email to luca in near future with the questions that have popped up
19:31:23 <clarkb> #topic Open Discussion
19:33:04 <clarkb> If there is anything else feel free to bring it up now. I'll leave the floor open for about 5 minutes and if we get nothing call that the meeting
19:37:55 <clarkb> Alright. Thanks everyone!
19:38:01 <clarkb> I'll call that a meeting
19:38:06 <clarkb> #endmeeting