| opendevreview | Goutham Pacha Ravi proposed openstack/project-config master: Add editHashtags permissions to some governance repositories https://review.opendev.org/c/openstack/project-config/+/919627 | 01:27 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Add devstack-gate back to central config for retirement prep https://review.opendev.org/c/openstack/project-config/+/919625 | 05:13 |
| opendevreview | Jens Harbott proposed openstack/project-config master: Retire Sahara: remove project from infra https://review.opendev.org/c/openstack/project-config/+/919391 | 08:34 |
| opendevreview | Jens Harbott proposed openstack/project-config master: Retire ec2-api: remove project from infra https://review.opendev.org/c/openstack/project-config/+/919397 | 08:34 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Remove networking-ovn completely https://review.opendev.org/c/openstack/project-config/+/919417 | 09:27 |
| fungi | clarkb: good catch, i guess i should add that first, wait for the change to deploy, then replace the a/aaaa records with a cname | 13:13 |
| fungi | i'll do that shortly | 13:13 |
| opendevreview | Brian Rosmaita proposed openstack/project-config master: Add more permissions for 'glance-ptl' group https://review.opendev.org/c/openstack/project-config/+/910641 | 13:46 |
| fungi | infra-root: our cloud-launcher deploy to update keypairs failed because of, looks like, finding multiple security groups in osuosl | 13:53 |
| fungi | the error returned from ansible is challenging to parse, unfortunately: | 13:54 |
| fungi | "Found more a single matching security group rule which match the given parameters." | 13:54 |
| fungi | that was for TASK [cloud-launcher : Processing security_group_rule for opendevci-osuosl RegionOne] | 13:55 |
| opendevreview | Takashi Kajinami proposed opendev/yaml2ical master: Remove unused extras https://review.opendev.org/c/opendev/yaml2ical/+/919736 | 14:07 |
| frickler | nice how that error has been copypasted across that module ;) https://codesearch.openstack.org/?q=Found%20more%20a%20single&i=nope&literal=nope&files=&excludeFiles=&repos= | 14:32 |
| frickler | I wonder if that's a regression in openstacksdk maybe. the rules look like this: https://paste.opendev.org/show/bLnyq8AofeMSUst5CXw9/ , rules #5+6 seem redundant with #4 in place, still that error shouldn't happen IMO | 14:39 |
| fungi | yeah, i think rules #1-4 are the only ones we care about, #5-6 might have been already present as a default? | 14:41 |
| frickler | also that failure seems to be happening for a longer time already | 14:44 |
| frickler | actually latest success was "2022-12-06 03:14:18" ... :-/ | 14:45 |
| frickler | makes it difficult to find logs | 14:45 |
| fungi | probably went away when we replaced the bridge server | 14:46 |
| fungi | the logs did, i mean | 14:46 |
| frickler | we seem to delete logs after 1 month, at least for this task | 14:47 |
| fungi | oh, so wouldn't have mattered anyway | 14:48 |
| opendevreview | Merged openstack/project-config master: Add more permissions for 'glance-ptl' group https://review.opendev.org/c/openstack/project-config/+/910641 | 14:52 |
| frickler | so on 2022-12-07 this was merged https://review.opendev.org/c/opendev/system-config/+/866633 and likely resulted in reqs updates on bridge | 14:53 |
| fungi | the timing is certainly suspicious | 14:54 |
| frickler | side note: I'm still seeing long response times (estimated around 30s) for searches like https://zuul.opendev.org/t/openstack/builds?project=opendev%2Fsystem-config&result=SUCCESS&skip=0 | 14:55 |
| fungi | so, yeah, maybe something changed in ansible a while before then and we weren't updating | 14:55 |
| frickler | also all the security group rules themselves are much older, updated_at 2021-04-something | 14:58 |
| frickler | so my idea now would be to manually delete #5+#6 and then wait to see what the next periodic run does | 14:58 |
| frickler | or maybe first crosscheck with another cloud | 14:59 |
| *** dmellado4 is now known as dmellado | 15:07 | |
| clarkb | ++ to crosscheck with another cloud. #5 and #6 in this case are on lines 9 and 10? if so I agree they are redundant and we can remove them to see if that addresses the problem | 15:09 |
| clarkb | frickler: I think we can manually edit the rules since we should only need to do it the one time | 15:17 |
| frickler | so vexxhost only has the first 4 rules, which matches what is defined at https://opendev.org/opendev/system-config/src/branch/master/inventory/service/group_vars/bastion.yaml#L16-L34 | 15:17 |
| frickler | (copying over from #-infra) | 15:18 |
| frickler | do we want to add the others with "state: absent" there for automated cleanup or should I just do it manually now? | 15:18 |
| clarkb | my message above was in response to the "automated or manually" qusetion and I think manually is fine | 15:18 |
| frickler | ah, o.k., so I'll do that now and then check logs again tomorrow | 15:19 |
| fungi | $ host _acme-challenge.api.openstack.org | 16:14 |
| fungi | _acme-challenge.api.openstack.org is an alias for acme.opendev.org. | 16:14 |
| fungi | clarkb: ^ that look right? | 16:14 |
| clarkb | fungi: ya that seems to match tarballs.openstack.org for example | 16:15 |
| fungi | cool, i'll approve the redirect change in that case | 16:22 |
| gthiemonge | hey Folks, how to report spam in launchpad? | 16:28 |
| clarkb | gthiemonge: other people have reported via asking questions against launchpad itself here: https://answers.launchpad.net/launchpad and upstream says that works for them | 16:30 |
| gthiemonge | clarkb: cool, thanks! i'll report there | 16:31 |
| fungi | clarkb: going back through the dns record deletions i proposed in the meeting yesterday, i see i included the a/aaaa records for refstack01.openstack.org which point to the current production server, i'm guessing i should *not* delete those after all | 16:43 |
| fungi | not sure why i included them | 16:43 |
| clarkb | fungi: oh right we didn't give that server .opendev.org records because its really an openstack specific service | 16:44 |
| fungi | cool, thanks, just making certain | 16:44 |
| fungi | #status log Deleted 41 obsolete/unused DNS records from the openstack.org domain: https://paste.opendev.org/show/bk9sSCPn5j4dZEbDGz8A/ | 16:48 |
| opendevstatus | fungi: finished logging | 16:48 |
| clarkb | infra-root I've just gone through and retested the upgrade from gerrit 3.8 to 3.9 then a downgrade back to 3.8 again. Took notes in https://etherpad.opendev.org/p/gerrit-upgrade-3.9 The process is straightforward. Then I redid the upgrade with a focus on testing the topic limit stuff and behavior is consistent with the old 3.9 image that I tested so no new concerns there | 16:48 |
| fungi | awesome! | 16:49 |
| clarkb | There are still a few open todos on that etherpad. Would be great if yall could look it over at this point and leave any notes about concerns or extra testing that you would like to see | 16:49 |
| fungi | thanks for testing that | 16:49 |
| clarkb | one of the todos has to do with handling of emails and I hate to say it but the easiest way to deal with that may be to upgrade and if we have problems with them then downgrade :/ | 16:49 |
| clarkb | because I don't think we can easily replicate that state in our test envirnment as current gerrit is much more strict about those entries getting in in the first place | 16:50 |
| clarkb | but otherwise we can probably start thinking about when we want to upgrade. And honestly having the ability to revert is a really nice escape hatch here | 16:50 |
| clarkb | maybe may 31? | 16:55 |
| fungi | wfm, i don't appear to have anything that day | 16:56 |
| fungi | and it's definitely not my 10-year wedding anniversary this time either | 16:56 |
| clarkb | do you need to double check :) | 17:07 |
| fungi | oh, i did ;) | 17:12 |
| fungi | i no longer trust my ability to remember such things | 17:12 |
| opendevreview | Merged opendev/system-config master: Host a redirect for api.openstack.org https://review.opendev.org/c/opendev/system-config/+/919616 | 17:29 |
| fungi | dns has been updated for ^ now that it's deployed to the server | 17:51 |
| fungi | dns queries confirm it's a cname to static now, and putting https://api.openstack.org/ in my browser still redirects to developer.o.o as intended | 17:52 |
| fungi | considering this done | 17:52 |
| clarkb | ya seems to work for me as well (I checked I was using new dns records too) | 17:53 |
| fungi | running some quick errands, bbiab | 18:22 |
| *** dmellado6 is now known as dmellado | 20:15 | |
| clarkb | anyone know where ubuntu stashes their pub keys for signing packages? I'm poking at noble mirroring change and suspect there may be a new key given the age of the existing keys we 've got | 20:39 |
| fungi | clarkb: best source i could find for key ids is https://wiki.ubuntu.com/SecurityTeam/FAQ#GPG_Keys_used_by_Ubuntu | 20:46 |
| clarkb | wow ubuntu's tutorial for verifying signatures basically invalidates any signature verification | 20:46 |
| clarkb | https://ubuntu.com/tutorials/how-to-verify-ubuntu#4-retrieve-the-correct-signature-key | 20:46 |
| clarkb | it says you try to verify and if that fails just blindly fetch the keys that would verify the sigs | 20:46 |
| fungi | they *should* be publishing those keys to the ubuntu keyserver too | 20:46 |
| fungi | so you could use gpg to fetch them by id/fingerprint from the keyserver, then export them from your local keyring | 20:47 |
| clarkb | yes but anyone can publish keys right? because they are part of the larger network?: | 20:47 |
| clarkb | you need a step that verifies the key is more likely than not to be one from ubuntu themselves | 20:47 |
| clarkb | anyway we have the keys listed in that wiki page trusted so maybe there are no new keys | 20:48 |
| fungi | i suppose it depends on whether you trust the key id listed on the wiki, yeah | 20:48 |
| fungi | "The Ubuntu Archive Master key" seems to not get rotated, like ever, they just issue new signing subkeys from it | 20:48 |
| clarkb | well and the key that gpg wants to use to verify signatures shouldn't simply be blindly trusted as that tutorial says to do | 20:49 |
| clarkb | ideally there would be a non wiki https hosted page on ubuntu.com that says "these are our keys" | 20:49 |
| clarkb | anyway I think we can proceed under the assumption the keys haven't changed based on the wiki page you found | 20:50 |
| fungi | in theory you could get it from the archive-keyring package assuming you already have a trusted install of ubuntu to do that on | 20:51 |
| clarkb | ya bouncing through a container image may be the easiest thing | 20:52 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add ubuntu noble to our package mirrors https://review.opendev.org/c/opendev/system-config/+/919777 | 20:52 |
| fungi | popping back out to grab dinner, back in an hour-ish | 20:52 |
| clarkb | I think the change to mirror noble is as simple as ^ but we'll need to increase quotas and babysit that change | 20:52 |
| clarkb | maybe plan to do that tomorrow so we have all day to fight afs fires should they occur? | 20:53 |
| *** dmellado9 is now known as dmellado | 22:04 | |
| fungi | clarkb: sounds good to me | 22:26 |
| tonyb | also sounds good to me. | 22:36 |
| tonyb | and may 31 for a 3.9 upgrade also works for me. | 22:37 |
| clarkb | tonyb: not sure what that looks like timeznoe wise for you | 22:38 |
| clarkb | but I dont' think you need to work on a weekend for that if you don't want to | 22:38 |
| fungi | when you track down an annoying bug only to discover that the last comment on it was made by you long enough ago that you've forgotten you already went through this at least once before | 22:47 |
| clarkb | fungi: I hate when I google a problem and my bug is the only thing that pops up | 22:48 |
| fungi | also when i perform a web search for an error message and the only results are one of our irc channels where we discussed finding this confusing error... | 22:50 |
| fungi | i'm starting to think that running the openafs kernel module on a bleeding-edge distro is just a lost cause, because the module being out-of-tree seems to result in new kernel support being very much reactive rather than proactive, so there are long spans of time where openafs-modules-dkms just fails to build on debian sid | 22:52 |
| fungi | to the point where it got autoremoved from testing months ago now | 22:53 |
| tonyb | clarkb: I'm still in MN until mid June so for me is earlier the fungi but later than you | 22:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!