| clarkb | Meeting time | 19:00 |
|---|---|---|
| fungi | ahoy! | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Nov 16 19:01:28 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| ianw | o/ | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2021-November/000296.html Our Agenda | 19:02 |
| frickler | o/ | 19:02 |
| clarkb | #topic Announcements | 19:03 |
| clarkb | Gerrit User Summit is happening virtually December 2&3 | 19:03 |
| fungi | also openinfra live keynotes tomorrow and thursday | 19:04 |
| clarkb | also virtual | 19:04 |
| fungi | 15:00-17:00 utc both days, though may run longer | 19:04 |
| fungi | basically this year's replacement for the openinfra summit | 19:05 |
| clarkb | Anyway thought I would throw that out there as I've always missed previous gerrit user summits as they don't seem toget as much advertisement | 19:05 |
| clarkb | and others may be interested in attending | 19:05 |
| clarkb | Also next week is a US holiday week. I wasn't planning to take the whole week off just the end of the week but then school last minute cancelled for the whole week | 19:06 |
| fungi | i expect to be around most of the week, in case anything comes up | 19:06 |
| clarkb | This means I'm going to try and avoid being around during the whole week to spend time with family. I'm inclined to cancel our team meeting next week as a result but happy for others to run a meeting if they like | 19:06 |
| fungi | though will likely be busy thursday, maybe friday | 19:06 |
| ianw | i'd do it but given low attendence doesn't seem worth it | 19:08 |
| ianw | seems better to force some time away :) | 19:08 |
| clarkb | wfm, we'll cancel next week then. See you all here in 2 weeks | 19:08 |
| clarkb | #topic Actions from last meeting | 19:08 |
| fungi | yeah, i'm fine cancelling | 19:08 |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-11-09-19.01.txt minutes from last meeting | 19:08 |
| clarkb | We don't seem to have recorded any specific actions | 19:09 |
| clarkb | #topic Topics | 19:09 |
| clarkb | #topic Improving OpenDev's CD Throughput | 19:09 |
| clarkb | I've got new motivation to pick this up again and realyl do hope to do so this week (I was sick last week otherwise I'd like to think I would've looekd at these) | 19:09 |
| clarkb | Yesterday we landed a change to our base inventory stuff which caused all the jobs to run and it took a couple of hours | 19:10 |
| clarkb | Right now we are in good shape when we run jobs in a targetted manner but when we do those base updates it is painful waiting for everything to complete. | 19:10 |
| clarkb | Anyway thats the motivation. I really do hope to look at the changes and figure out why CI isn't super happy with the later changes | 19:11 |
| clarkb | #topic Gerrit Account Cleanups | 19:11 |
| clarkb | I still haven't heard from the most recent user I updated so assume that was fine. I haven't done the mass emails yet but need to explicitly block off time for that. Maybe week after next and just work through those | 19:12 |
| clarkb | It is too easy to deprioritize this effort, but it is very close to being done so I should just get it done | 19:12 |
| clarkb | #topic Zuul Multi Scheduler | 19:14 |
| clarkb | Zuul is still running with two schedulers and seems significantly more stable | 19:14 |
| clarkb | zuul01 and zuul02 are the two hosts and zuul02 is the primary and runs zuul-web | 19:14 |
| fungi | been running that way since the weekend, and we even had a zero-downtime rolling restart of the schedulers | 19:15 |
| clarkb | zuul-web has been updated to talk to zk directly which means you should always get a consistent view from the status pages now | 19:15 |
| clarkb | This is feeling like it is stabilising now, and maybe we'll get another zuul release in the near future (though there are a few inflight fixes that we will ideally restart opendev on to sanity check first, good news is those can hopefully be done without downtime now) | 19:16 |
| clarkb | Anyway just an update on that as its been a thing lately. Hopefully less and less a thing as we stabilize it further | 19:17 |
| clarkb | #topic User management on our systems | 19:18 |
| clarkb | Thank you fungi and ianw and frickler and others for the reviews on this. We've landed the first changes in starting to clean this up | 19:18 |
| clarkb | unnecessary users should now be removed and we've adjusted the uid and gid range on our systems (though any uids/gids outside of that range haven't been moved) | 19:19 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/816769/ Give gerritbot and matrix-gerritbot a shared user | 19:19 |
| clarkb | is the next step in that process I think | 19:19 |
| clarkb | please review that one carefully as I suspect this will act as a sort of template as we go through and update other services to do similar | 19:19 |
| clarkb | separately I think that if we want to work on shifting mariadb uids over to match the services the db is running for I think we can probably start that work in parallel though I expect it to be fiddly | 19:21 |
| ianw | thanks, sorry reviewing of these got a bit distracted, but will do | 19:22 |
| clarkb | thanks. | 19:22 |
| clarkb | re mariadb I suspect that etherpad would be a good first candidate | 19:22 |
| clarkb | since it is fairly self contained and doesn't ahve a lot of moving parts like gitea or gerrit | 19:22 |
| clarkb | #topic Caching of openstack/openstack on our DIB image builds | 19:23 |
| fungi | with mariadb, the idea would be to have a specific mariadb user, or use the same user as the etherpad service runs under? | 19:23 |
| clarkb | #undo | 19:23 |
| opendevmeet | Removing item from minutes: #topic Caching of openstack/openstack on our DIB image builds | 19:23 |
| fungi | sorry, i'm typing slowly today | 19:23 |
| clarkb | fungi: I think use the same user as the etherpad service. Since all of the services running etherpad have rw access to the db there isn't much to gain from separateing them | 19:23 |
| clarkb | if we ran a central db and shared it with many services I would say they should be separate but in this case we run a mariadb separately for each service | 19:24 |
| clarkb | and this helps simplify things. THough we could split users if there was a good reason to | 19:24 |
| fungi | yeah, makes sense as long as multiple services don't share a common db server | 19:24 |
| fungi | thanks, that answers my question | 19:25 |
| ianw | mariadb already runs as 999 though right? | 19:25 |
| clarkb | ianw: yes | 19:26 |
| fungi | which unfortunately conflicts with the range of accounts that might get taken by distro packages of things | 19:26 |
| clarkb | right the concern is we don't want the overlap with system usage | 19:26 |
| clarkb | we should shift it to our uses and having it run as the same user as the service seems fine since the db is rw and separate for each service | 19:26 |
| fungi | hence our desire to carve up separate ranges | 19:26 |
| ianw | ++ | 19:27 |
| clarkb | #topic Caching of openstack/openstack on our DIB image builds | 19:29 |
| clarkb | I wasn't sure if this would be fixed or not by the time of the meeting so I had this on the agenda to discuss simply removing openstack/openstack from caching | 19:29 |
| clarkb | But ianw was able to track this down to a weird git interaction (I kinda think it might be a bug?) between using the git dir option and how submodules are checked | 19:30 |
| clarkb | thankfully there is the -C option as an alternative to the git dir option and that should fix things. The dib change to switch to -C has landed but I think we need a release and new nodepool iamges to take advantage of it | 19:30 |
| ianw | it might be a bug that git made it so confusing as to what was going on, but it does appear to be mostly our fault | 19:30 |
| ianw | at least git is written in C and builds with make, so it wasn't a multi-day setup to instrument it and figure out what it thought it was doing :) | 19:32 |
| clarkb | ya I mean it seems liek git should be resilient to people passing flags like that | 19:32 |
| clarkb | either by saying "I can't function in this situation" explicitly or by figuring out a way to function | 19:33 |
| ianw | i'll probably do a dib point release and update nodepool, because it's very racy getting images built atm | 19:33 |
| clarkb | sounds good, and thank you for looking into that. An upside to using containers this way I guess | 19:34 |
| ianw | i think openstack/openstack project is fine, but it's still not clear to me how it's actually setup | 19:34 |
| clarkb | ianw: gerrit has a magic flag where it will auto update submodules in a repo if it hosts the submodule repos too | 19:35 |
| clarkb | ianw: so basically if you add a submodule to that repo and it is on our gerrit gerrit will make commits to it automatically as we merge chagnes to hose repos | 19:35 |
| clarkb | (its an explicit config option on the repo that has to be set) The idea when it was set up was that it could be used to track a log of the order things land in | 19:35 |
| clarkb | but it never really got used and has become very bit rotten (it lacks repos iirc) | 19:35 |
| ianw | ahh, yeah that's what i was wondering, what maintains it | 19:36 |
| ianw | it looks like there's a "generate-gitmodules.py" | 19:38 |
| ianw | but, it doesn't look like this auto-runs or auto-proposes? | 19:38 |
| clarkb | I think it is in gerrit itself | 19:39 |
| clarkb | ianw: https://gerrit-review.googlesource.com/Documentation/user-submodules.html | 19:40 |
| fungi | in theory the openstack tc is supposed to be maintaining the list of included submodules in it | 19:41 |
| ianw | $ git review | 19:42 |
| ianw | ssh://iwienand@review.opendev.org:29418/openstack/openstack.git did not work. Description: fatal: Upload denied for project 'openstack/openstack' | 19:42 |
| fungi | there might be some automation in openstack/governance, like the script they run which creates github projects for mirroring into | 19:42 |
| ianw | interesting. it looks like AJaeger used to mostly run the script and update .gitmodules in there | 19:42 |
| fungi | anyway, worth checking with the opemstack tc to see if anyone's maintaining it | 19:43 |
| ianw | "2. configure the submodule to allow having a superproject subscribed" was the step i was unclear on | 19:43 |
| ianw | submodule.enableSuperProjectSubscriptions seems like it defaults to true, so that explains that bit | 19:44 |
| ianw | i guess we have done something in All-Projects for this repo? | 19:45 |
| clarkb | that could be | 19:45 |
| clarkb | I wasn't invovled in the setup so don't recall how it was done | 19:46 |
| clarkb | #topic Open Discussion | 19:47 |
| ianw | fungi: i think the answer empirically is "no" https://opendev.org/openstack/openstack/commits/branch/master/.gitmodules | 19:47 |
| clarkb | Figured I'd open it up to any other discussion | 19:47 |
| ianw | clarkb: https://review.opendev.org/c/opendev/system-config/+/817301 was a quick one from your comment on doing a mark/unmark/mark cycle on the gerrit testing | 19:49 |
| clarkb | ah yup +2 | 19:50 |
| ianw | i had been under the impression this repo was automatically keeping itself up to date | 19:50 |
| ianw | given that it isn't, i guess i'll post to the list and propose retiring it | 19:50 |
| fungi | one of the main reasons they've wanted it kept around is the "cncf landscape" | 19:51 |
| fungi | for a project to be included on the landscape, it must be represented by one (and only one) repo on github | 19:51 |
| fungi | so the github mirror of the openstack/openstack superrepo is how the lf/cncf measures the level of "activity" for the openstack project | 19:52 |
| ianw | interesting | 19:52 |
| ianw | since i couldn't push an update to the .gitmodules, i'm guessing it's locked down to some group | 19:53 |
| ianw | the first thing i'm thinking is that the proposal bot should run the project update | 19:53 |
| fungi | push = group Release Managers | 19:54 |
| fungi | exclusiveGroupPermissions = Push | 19:54 |
| fungi | that's in the [access "refs/for/refs/*"] section of gerrit/acls/openstack/openstack.config | 19:54 |
| ianw | or, could we do something more like have zuul run it in a periodic job? | 19:55 |
| ianw | but where would the zuul job get permissions to propose/push the change? | 19:55 |
| fungi | could use the same account which pushes tags for openstack | 19:56 |
| fungi | it's a member of release managers | 19:56 |
| ianw | seems reasonable | 19:57 |
| clarkb | you'd still need someone to review them? or are you saying just push directly? | 19:58 |
| ianw | if it ran and just +2 +W'd it's change it would seem to keep it in sync | 19:58 |
| ianw | yeah, i'm thinking that reality of people reviewing is low | 19:59 |
| fungi | we could adjust the acl to give create = group Release Managers in [access "refs/heads/*"] and bypass reviewing, i suppose | 19:59 |
| ianw | since it's based on the YAML file that is reviewed it seems low risk | 19:59 |
| ianw | is it running any testing from project-config? | 20:01 |
| fungi | i don't think so | 20:01 |
| ianw | https://review.opendev.org/c/openstack/openstack/+/741207 ... just noops | 20:01 |
| fungi | which is another reason direct pushing to the branch might be more sensible | 20:01 |
| ianw | ok, well i'll put it on the todo. could be an interesting exercise in zuul jobbing | 20:02 |
| fungi | seems like we've probably finished meeting. thanks clarkb | 20:10 |
| fungi | #endmeeting | 20:10 |
| opendevmeet | Meeting ended Tue Nov 16 20:10:36 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:10 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2021/infra.2021-11-16-19.01.html | 20:10 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2021/infra.2021-11-16-19.01.txt | 20:10 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2021/infra.2021-11-16-19.01.log.html | 20:10 |
| clarkb | oh sorry I spaced out on the logging thing | 20:17 |
| clarkb | For some reason I thought I had ended the meeting but nope. Thank you for doing that | 20:17 |
| fungi | no worries, i figured you had already switched context | 20:17 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!