| opendevreview | Andy Wu proposed openstack/project-config master: Add cinder-nfs charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/879469 | 00:14 |
|---|---|---|
| *** Trevor is now known as Guest10019 | 04:42 | |
| *** amoralej|off is now known as amoralej | 06:11 | |
| opendevreview | Michal Nasiadka proposed opendev/irc-meetings master: kolla: Move meeting one hour backwards (DST) https://review.opendev.org/c/opendev/irc-meetings/+/879536 | 08:35 |
| opendevreview | Merged opendev/irc-meetings master: kolla: Move meeting one hour backwards (DST) https://review.opendev.org/c/opendev/irc-meetings/+/879536 | 08:51 |
| opendevreview | waleed mousa proposed openstack/diskimage-builder master: fix ifupdown pkg map for dhcp-all-interfaces of redhat family https://review.opendev.org/c/openstack/diskimage-builder/+/879537 | 09:38 |
| opendevreview | waleed mousa proposed openstack/diskimage-builder master: fix ifupdown pkg map for dhcp-all-interfaces of redhat family https://review.opendev.org/c/openstack/diskimage-builder/+/879537 | 09:44 |
| *** amoralej is now known as amoralej|lunch | 11:21 | |
| *** blarnath is now known as d34dh0r53 | 12:14 | |
| *** amoralej|lunch is now known as amoralej | 12:26 | |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add github sqlalchemy/alembic to the project list https://review.opendev.org/c/openstack/project-config/+/879549 | 12:46 |
| opendevreview | Merged openstack/project-config master: Add github sqlalchemy/alembic to the project list https://review.opendev.org/c/openstack/project-config/+/879549 | 13:16 |
| TheJulia | good morning opendev! Looks like job 38b5b05b25894795b4997a7935c9ad89 would have a held node for the ironic-grenade job for us to investigate | 13:25 |
| fungi | checking | 13:26 |
| fungi | TheJulia: i have a very nice server in rax-ord you might like | 13:28 |
| fungi | what ssh keys should i authorize? | 13:28 |
| TheJulia | ooooo ahhhhhhhhh | 13:28 |
| fungi | TheJulia: ssh root@23.253.160.187 | 13:30 |
| fungi | let us know whenever you're done with it and we can clean up the hold | 13:30 |
| * TheJulia does the whole evil laugh thing | 13:30 | |
| fungi | no rush | 13:30 |
| TheJulia | thanks | 13:41 |
| opendevreview | Andy Wu proposed openstack/project-config master: Add cinder-nfs charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/879469 | 13:45 |
| fungi | infra-root: there's some outstanding project creation changes. should we hold those until friday in order to avoid creating merge conflicts for the renames? | 14:21 |
| fungi | renames and retirements both i guess | 14:21 |
| frickler | fungi: sounds reasonable to me | 14:22 |
| fungi | i was about to approve some, and then stopped myself | 14:22 |
| frickler | btw. fri+mon are bank holidays here, so I won't be around much then | 14:22 |
| fungi | i mentioned it during the meeting, but just as a reminder i'll be gone all next week | 14:24 |
| opendevreview | Michal Nasiadka proposed openstack/project-config master: Stop using Storyboard for Magnum projects https://review.opendev.org/c/openstack/project-config/+/879560 | 14:30 |
| gthiemonge | Hi Folks, we (the Octavia team) have a lot of issues with vexxhost's Ubuntu Jammy nested virt hosts (we are still on ubuntu focal controlers, the commit that updates the jobs fails in the CI) because of kernel crashes: https://bugs.launchpad.net/octavia/+bug/2015244 | 14:50 |
| gthiemonge | It seems that neutron had the same issue in the past: https://bugs.launchpad.net/neutron/+bug/1999249 | 14:50 |
| gthiemonge | should we consider removing those hosts from the jammy nested-virt pool? | 14:51 |
| fungi | gthiemonge: it looks like that would only leave ovh providing those labels, though we do have two ovh regions with them so there would be some redundancy | 15:06 |
| gthiemonge | fungi: some vexxhost nodes work properly (maybe 60% of the nodes based on the neutron commit), could we identify the broken nodes and keep only the good ones? | 15:08 |
| fungi | gthiemonge: zuul records the host-id hash which we can correlate and then provide to mnaser or guilhermesp and they can see if those correspond to specific underlying operating system versions, kernel versions, processor models, whatever | 15:11 |
| fungi | gthiemonge: example... https://zuul.opendev.org/t/openstack/build/4dcaa22599dc4aba8c36f6dbd20deb2b/log/zuul-info/inventory.yaml#13 | 15:12 |
| clarkb | gthiemonge: fungi: friendly reminder that those using the virt labels are expected to engage the cloud directly rather than play telephone with us. Part of adding clouds to those labels is double checking with the cloud that they are happy with that | 15:13 |
| gthiemonge | fungi: do we know how many nodes are in this pool? | 15:13 |
| clarkb | I think the best thing would be to engage the cloud directly and if the issue can't be resolved in a reasonable time period we can remove the cloud from the label | 15:14 |
| clarkb | also a friendly reminder that nested virt is highly untable and you should expect these problems to occur and require debugging | 15:14 |
| gthiemonge | clarkb: ack, first we will identify which nodes are problematic | 15:14 |
| fungi | actually, i don't see evidence we're providing any nested virt labels in vexxhost? we have them configured for the sjc1 region but it's set to max-servers of 0 | 15:16 |
| fungi | oh, never mind, we do have them in the ca-ymq-1 region | 15:16 |
| fungi | gthiemonge: roughly appears to be 50 in vexxhost ca-ymq-1 vs 199 between ovh bhs1 and gra1 regions | 15:18 |
| fungi | so of the quota (based on our max-servers settings, which isn't perfectly accurate) where we could boot those labels, vexxhost accounts for roughly 20% | 15:19 |
| fungi | i'm heading out to lunch, but should be back in an hour-ish | 15:19 |
| clarkb | genekuo: fungi: ya my focus for the next little bit within opendev will likely be updating old bionic servers. That involves changes like: https://review.opendev.org/q/topic:add-static02 these are probably less appraochable to a new contributor though. As far as things that are more approachable go I would point out our system-config-run-* zuul jobs that deploy our infrastructure | 15:21 |
| clarkb | like production and test things like gerrit, gitea, and so on. Getting familiar with that and maybe adding some tests to less well tested services could be helpful and a good way to get familiar with our tooling | 15:21 |
| clarkb | genekuo: fungi: then a possible good followup to that would be picking up the dedicated uid work for our docker containers. and/or updating mariadb versions in services that depend on a mariadb installation | 15:22 |
| clarkb | But also as fungi pointed out there is the keycloak openid integration work and general keycloak opendev id service bring up. | 15:22 |
| noonedeadpunk | folks, I'm not sure if you aware or not, but centos is completely broked atm, including container images, cloud images, etc, due to release of gnupg2-2.3.3-3 - https://bugzilla.redhat.com/show_bug.cgi?id=2184640 | 15:23 |
| noonedeadpunk | So might be good to know. I'm not sure how much this could affect us though, but it for sure will freshly-built zuul images in case they won't rollout a fix | 15:24 |
| clarkb | noonedeadpunk: thanks for the heads up | 15:25 |
| clarkb | that doesn't look like the sort of thing we could work around eve nif we wanted to (and generally we try to expose platform issues and force the software to workaround it as that is the value in testing on the platform) | 15:27 |
| noonedeadpunk | well, they said that explicit `rpm --import` should still work | 15:28 |
| noonedeadpunk | but things may get worse as miltiple SIGs also don't have supported GPG keys, like NFV where I spotted the issue originally | 15:29 |
| noonedeadpunk | (rocky is not affected as you might guess) | 15:30 |
| clarkb | ya this looks stream specific | 15:31 |
| clarkb | similar to the other stream issues we've hit previously where packages update and break but then it takes time to correct because the fixing happens in rhel future first or something | 15:31 |
| noonedeadpunk | yeah. which is super frustrating... So it's 3rd major breakage of Stream for OSA in last 2 weeks fwiw | 15:33 |
| clarkb | definitely seems like if centos stream is where people break first that it should also be fixed first | 15:33 |
| clarkb | https://review.opendev.org/c/openstack/diskimage-builder/+/879537 this dib change is failing on the issue | 15:33 |
| noonedeadpunk | yeah, exactly the issue | 15:35 |
| noonedeadpunk | I wonder if it's worth writing a ML for holding rechecks | 15:35 |
| noonedeadpunk | As I really have no idea how centos jobs are widespread | 15:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add static02 to inventory https://review.opendev.org/c/opendev/system-config/+/879383 | 15:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Make etherpad configuration more generic for multiple hosts https://review.opendev.org/c/opendev/system-config/+/879385 | 15:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add etherpad02 to inventory https://review.opendev.org/c/opendev/system-config/+/879386 | 15:36 |
| clarkb | infra-root ^ that should make for better testing of those services on jammy before we merge anything. Sorry I missed that on the first pass | 15:36 |
| clarkb | noonedeadpunk: ++ | 15:37 |
| noonedeadpunk | well, I've jsut spotted email from TripleO on the same topic with solution like that https://review.opendev.org/c/openstack/tripleo-quickstart/+/879535/3/config/release/tripleo-ci/CentOS-9/master.yml | 15:49 |
| noonedeadpunk | Likely this can be applied to dib as well? | 15:50 |
| clarkb | we try to avoid those changes if we can because it hides the fact that your software is actually broken on centos | 15:51 |
| clarkb | what tripleo is doing is fine because it is the software and it is correcting the issue for itself | 15:51 |
| clarkb | however when the issue is deep in the package management system things get weird for sure and there may need to be compromise | 15:52 |
| clarkb | I think maybe doing the rpm --import is better though then we aren't using old gpg and instead working around current gpg? | 15:52 |
| noonedeadpunk | The tricky thing about rpm --import is that nobody know full list of affected repos atm | 15:53 |
| clarkb | ah that is repo specific because each repo has a different signing key. got it | 15:54 |
| noonedeadpunk | so then everyone need to do that for each affected one which is quite annoying and plenty of work | 15:54 |
| noonedeadpunk | yeah, so if repo has gpg signed with sha-2 - it shouldn't be an issue | 15:54 |
| noonedeadpunk | or well !sha-1 at least | 15:55 |
| clarkb | the gpg keys themselves are independent of the hash as is the pubkey material right? The issue is on the signing side so theoretically they could resign everything and push and we'd be fine? | 15:56 |
| clarkb | genekuo: fungi: also I would be happy to set up some time to talk on jitsi meet or similar if that would be helpful. I'm currently in UTC-7 which might make selecting a time painful but I'm sure we can make something work. I don't mind an early morning or later evening call | 15:58 |
| noonedeadpunk | clarkb: oh, yes, exactly | 16:00 |
| noonedeadpunk | so it's not big technical issue, it's jsut super annoying and matter of not being able to do much in a proper way | 16:02 |
| clarkb | ya part of the issue here is endusers can't properly fix anything themselves | 16:02 |
| noonedeadpunk | And once they release it we need to also wait for mirrors to get these updated packages | 16:03 |
| clarkb | as it has to do with the trust relationship between the end user and the distro rather than the content of the distro itself | 16:03 |
| noonedeadpunk | yup... | 16:03 |
| noonedeadpunk | I kind of wonder how that could pass any CI, but it's completely different topic.... | 16:04 |
| *** amoralej is now known as amoralej|off | 16:14 | |
| frickler | at some point we said we'd only support LTS releases, not unstable things. if we continue with this, we could as well start supporting sid or buntu 23.04 | 16:23 |
| clarkb | frickler: yes, personally I think we should probably continue to push more towards rocky/openeuler for the bulk of the rhel like testing | 16:24 |
| clarkb | We didn't have that option initially (since openeuler is a different kernel I don't want to force people that way if they don't want to use it), but now we've had rocky 8 and 9 for a bit and it seems reasonably stable | 16:25 |
| frickler | I agree about rocky, openeuler keeps falling over, too | 16:25 |
| frickler | see devstack+kolla | 16:25 |
| clarkb | and then maybe also consider if fedora or centos stream provide more benefit from an upcoming updates perspective and focus on one. I think the transition from centos to centos stream has made this all a bit painful though and we're still slowly working through it | 16:26 |
| clarkb | personally I realy like the idea of a forward looking distro to catch issues as early as possible too, but that requires dedicated effort and we've never really been able to find that person or people | 16:31 |
| clarkb | linux 6.2 broke s3 suspend on my laptop and now I have to use the more battery hungry s10x or whatever its called | 16:32 |
| clarkb | catching problems like that early before they cause problems for stable releases that affect many more users would be great | 16:32 |
| frickler | sure, but that should likely be an optional, non-voting scenario, not one where everyone goes screaming about their whole CI being borked | 16:34 |
| clarkb | for sure | 16:35 |
| johnsom | Hi there. I don't seem to have channel op status on #openstack-lbaas any more. It won't let me update the channel topic any longer. Can I get added to the op list for the channel? | 16:36 |
| clarkb | johnsom: you will want ot edit https://opendev.org/openstack/project-config/src/branch/master/accessbot/channels.yaml#L193 | 16:37 |
| johnsom | ack, thanks | 16:37 |
| clarkb | It was very likely lost in the oftc migration since there was no garuntee that your old nick was the same on oftc we didn't port those from freenode | 16:37 |
| johnsom | right | 16:38 |
| opendevreview | Michael Johnson proposed openstack/project-config master: Add johnsom ops for #openstack-lbaas and -dns https://review.opendev.org/c/openstack/project-config/+/879672 | 16:42 |
| fungi | okay, lunch completed | 16:49 |
| frickler | johnsom: also you don't need to actually op yourself, better set the topic via chanserv. we still need to merge the above patch for that | 17:00 |
| johnsom | Yeah, chanserv was rejecting me when I did the set topic command | 17:00 |
| johnsom | "You do not have access to the TOPIC command on channel #openstack-lbaas." | 17:01 |
| frickler | ah, o.k., that's the right command, wasn't clear from your earlier description. that should work soon, then | 17:02 |
| johnsom | Thanks! | 17:02 |
| opendevreview | Merged openstack/project-config master: Add johnsom ops for #openstack-lbaas and -dns https://review.opendev.org/c/openstack/project-config/+/879672 | 17:05 |
| fungi | johnsom: the deploy job finished so you should have access now | 17:46 |
| johnsom | Thank you | 17:47 |
| fungi | yw. and yes, as clarkb noted we didn't copy access lists from freenode to oftc because we couldn't know if people were squatting some of the account names, but also extending the accessbot code to set those acls makes it easier for us to track permission requests now | 17:48 |
| clarkb | afs01.ord was migrated off of failing hardware and should be online again according to email | 18:35 |
| clarkb | I can look more closely after lunch | 18:35 |
| clarkb | also Element Matrix Services will be doing maintenance on our instances April 13 between 02:24 UTC and 6:24 UTC for a predicted maximum downtime of 60 minutes | 18:36 |
| clarkb | I suspect no one will really notice in that time block. I'll be sure to checkn on it when I wake April 14th | 18:37 |
| clarkb | maybe ianw will notice but its friday afternoon/evening for ianw in that timeframe anyway | 18:38 |
| *** travissoto7 is now known as travissoto | 19:48 | |
| clarkb | fungi: if you get a chance can you review https://review.opendev.org/q/topic:add-static02 particularly the static replacement changes? I think landing those should be pretty safe? If we want ot wait until after the gerrit stuff that is fine too, but hoping to have eyeballs on them before you go on vacation at least | 20:35 |
| fungi | yep, meant to look at those today | 20:46 |
| clarkb | the four changes related to tomorrow's gerrit outage all lgtm (3 renames and 1 to update the gerrit config to 3.7) | 20:47 |
| ianw | oh is that the second time for that? | 21:15 |
| ianw | ord | 21:17 |
| ianw | genekuo: in approximately 25 hours from now we'll be going through https://etherpad.opendev.org/p/gerrit-upgrade-3.7 for the gerrit upgrade, and discussing it here | 21:18 |
| ianw | you are of course welcome to follow along | 21:18 |
| ianw | 99% of our maintenance is not nearly so hands-on, as it were. most everything else is gitops driven | 21:20 |
| ianw | noonedeadpunk/clarkb: we can pause our builds, if we haven't already rebuilt with it | 21:21 |
| ianw | centos9 i mean | 21:21 |
| ianw | i think it's actually building right now ... https://nb02.opendev.org/centos-9-stream-0000010785.log | 21:22 |
| ianw | 2023-04-05 20:47:28.542 | gnupg2 x86_64 2.3.3-3.el9 baseos 2.5 M | 21:23 |
| clarkb | ianw: I don't think it will help will it? youdo a yum update in the job and it will break. But I guess maybe we don't do that everywhere? | 21:24 |
| clarkb | ianw: oh I'm just bad at noticing timestamps. I think ord is fine as you point out | 21:25 |
| ianw | clarkb: yeah .. it might save something but not a generic solution | 21:26 |
| clarkb | ianw: is there anything else you can think of that needs eyeballs prior to the downtime and work tomorrow? I think all of my concerns have been addressed at this point | 21:47 |
| ianw | clarkb: i don't think so, i'm happy with the checklist -- i assume you're happy with the approach in renames to merge two + wait for manage-proj to fail + unemergency + merge last? | 21:58 |
| clarkb | yup I think that plan sounds great | 22:05 |
| opendevreview | Merged opendev/system-config master: install-launch-node: upgrade launch env periodically https://review.opendev.org/c/opendev/system-config/+/879387 | 22:45 |
| ianw | Requirement already satisfied: openstacksdk>=0.103 in /usr/launcher-venv/lib/python3.10/site-packages (from opendev-launch==1.0.0) (0.103.0) | 23:14 |
| ianw | so i guess "pip install -U <path-to-launch>" doesn't upgrade *everything*, only the launch script | 23:14 |
| ianw | which is obvious now i think about it with hindsight | 23:15 |
| fungi | yeah, the default upgrade strategy is conservative, i should have thought of that | 23:15 |
| fungi | we need --upgrade-strategy=eager | 23:16 |
| fungi | the default strategy is "only-if-needed" | 23:16 |
| opendevreview | Merged opendev/zone-opendev.org master: Add static02 https://review.opendev.org/c/opendev/zone-opendev.org/+/879382 | 23:17 |
| opendevreview | Merged opendev/zone-opendev.org master: Add etherpad02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/879384 | 23:17 |
| Clark[m] | My parents are in town as of today. Going to skip out early today for dinner to catch up with them so I feel less bad when I ignore them tomorrow | 23:18 |
| fungi | have fun! | 23:19 |
| opendevreview | Ian Wienand proposed opendev/system-config master: install-launch-node: upgrade all packages https://review.opendev.org/c/opendev/system-config/+/879712 | 23:21 |
| fungi | that looks right, thanks! | 23:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!