| clarkb | corvus: do we have zuul launcher managed ubuntu images now? | 00:09 |
|---|---|---|
| clarkb | I just added 942018 to the meeting agenda though I think it can land sooner than that. last call for other agenda items I'll send it out shortly | 00:10 |
| clarkb | and sent | 00:18 |
| clarkb | looks like the zuul dogfooding of the new nodes is hitting node failures | 00:21 |
| corvus | clarkb: jammy and noble https://zuul.opendev.org/t/zuul/images | 00:23 |
| corvus | and yeah, the node failures are due to missing images due to the space issue | 00:23 |
| corvus | so -- dogfooding having the intended effect :) | 00:24 |
| corvus | it looks like it's recovered now though, so i'm going to put in another recheck | 00:24 |
| clarkb | fingers are crossed here | 00:24 |
| corvus | meh, now it's quota issues. that's a known problem. we'll try again later :) | 00:26 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add a role to set ulimits https://review.opendev.org/c/zuul/zuul-jobs/+/930493 | 00:27 |
| clarkb | looking at syslog on grafana to see if anything looks broken when corvus said it wasn't reachable I'm not seeing anything either. I do notice that we seem to be hitting https://github.com/net-snmp/net-snmp/pull/785 which implies that at least some of the snmpd stuff isn't working on noble? | 00:32 |
| clarkb | I think that will affect all of the content consumed from /proc/net/snmp for noble until patched | 00:33 |
| clarkb | hrm apparently we should expect things to work otherwise. Maybe it simply ignores the extra field | 00:39 |
| clarkb | https://bugs.launchpad.net/ubuntu/+source/net-snmp/+bug/2056257 is the preexisting ubuntu issue | 00:39 |
| clarkb | and net-snmp hasn't made a new release with the fix? | 00:40 |
| clarkb | I'll just live with the logspam I guess | 00:40 |
| clarkb | oh shoot I just remember a new agenda item: using the additional rax flex region | 00:45 |
| * clarkb makes a note locally to bring that up | 00:45 | |
| *** tkajinam is now known as Guest9449 | 13:10 | |
| fungi | heads up to keep openssh's VerifyHostKeyDNS option set to "no" (normally the default) until you have a client patched for CVE-2025-26465: https://www.openwall.com/lists/oss-security/2025/02/18/1 | 13:38 |
| fungi | looks like distros are patching quickly, the debian advisory for it was the next thing in my inbox after that oss-security ml post | 13:39 |
| opendevreview | Merged opendev/system-config master: Use a dedicated zuul launcher temp dir on /opt https://review.opendev.org/c/opendev/system-config/+/942018 | 14:15 |
| frickler | fancy | 14:20 |
| clarkb | this is how I discover my local machine ahs a complicated ssh config that is basically a noop (so uses defaults) | 15:46 |
| clarkb | I don't see any objections to deleting zuul-lb01 and codesearch01 this morning. At this point they should both be out of the inventory files and have their DNS records cleaned up. I'll proceed with taht in a couple hours after my first block of meetings | 15:48 |
| fungi | also didn't see any concerns raised about the bindep changes so i guess i'll self approve them now | 15:53 |
| -opendevstatus- NOTICE: nominations for the OpenStack PTL and TC positions are closing soon, for details see https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/7DKEV7IEHOTHED7RVEFG7WIDVUC4MY3Z/ | 15:57 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Use mirrored qemu-user-static image https://review.opendev.org/c/zuul/zuul-jobs/+/942127 | 16:58 |
| clarkb | I'm deleting zuul-lb01 and codesearch01 now | 18:19 |
| clarkb | #status log Deleted zuul-lb01 (35a9ef8b-1a23-4bf0-848b-d10cb8826bd4) as it has been replaced by zuul-lb02 | 18:20 |
| opendevstatus | clarkb: finished logging | 18:20 |
| clarkb | #status log Deleted codesearch01 (5e9f1186-b9dc-4738-9fa5-ed71938be0c1) as it has been replaced by codesearch02 | 18:21 |
| opendevstatus | clarkb: finished logging | 18:22 |
| clarkb | both services still respond to me as expected | 18:22 |
| clarkb | just sanity checking things | 18:22 |
| NeilHanlon | fyi https://lists.gnu.org/archive/html/grub-devel/2025-02/msg00024.html | 19:36 |
| fungi | yeah, i skimmed those, most of the risk seems to be regarding secure boot bypass | 19:36 |
| NeilHanlon | mostly, yeah | 19:36 |
| NeilHanlon | there will be a new shim soon probably to revoke older grubs via sbat | 19:37 |
| fungi | and pretty much all require someone having access to supply a malicious filesystem or file for grub to read | 19:37 |
| NeilHanlon | yeep | 19:37 |
| clarkb | cardoe: we're having login issues with raxflex sjc3. Is it possible there is an account mapping heuristic problem again? | 20:00 |
| clarkb | dan_with: ^ you may know too? | 20:01 |
| fungi | yeah, our credentials, which haven't changed, started resulting in errors like "The request you have made requires authentication. (HTTP 401)" | 20:04 |
| clarkb | I'm going to pop out for lunch now then maybe a bike ride. I should be back well before 00:00 UTC to make the service coordinator thing official | 20:04 |
| cardoe | I think they were doing something with fernet token syncing | 21:01 |
| cardoe | I pinged | 21:07 |
| fungi | thanks. it's apparently only impacting one of our two accounts/tenants/projects | 21:15 |
| opendevreview | Merged opendev/bindep master: Use PBR's pyproject.toml build-backend support https://review.opendev.org/c/opendev/bindep/+/816741 | 21:48 |
| opendevreview | Merged opendev/bindep master: Evacuate most metadata out of setup.cfg https://review.opendev.org/c/opendev/bindep/+/938520 | 21:48 |
| opendevreview | Merged opendev/bindep master: Drop support for Python 3.6 https://review.opendev.org/c/opendev/bindep/+/938568 | 21:54 |
| cloudnull | corvus clarkb - Doug mentioned that there were some auth issues with Flex, is this still something that is happening? | 22:12 |
| fungi | checking | 22:14 |
| fungi | "The request you have made requires authentication. (HTTP 401)" | 22:15 |
| fungi | cloudnull: it's happening for one of our tenant accounts but not the other | 22:15 |
| cloudnull | fungi does that tenant account have the project-id set? | 22:16 |
| cloudnull | if so, there was a bug that got fixed today that had us rehash projects in SJC to match the global scheme. So you may need to define a new value | 22:17 |
| fungi | cloudnull: both do, and both are using the local project ids rather than the federated ones | 22:17 |
| fungi | because earlier during setup we observed that the federated ids didn't seem to work unless someone logged into skyline with them first (to get them cached i guess?) | 22:18 |
| opendevreview | Merged zuul/zuul-jobs master: Role ensure-tox: Remove obsoleted comment about tox v3 pin https://review.opendev.org/c/zuul/zuul-jobs/+/941063 | 22:21 |
| cloudnull | fungi let me know if you continue to have issus. dm me the project name if needed, happy to help troubleshoot | 22:38 |
| clarkb | fungi: any chnce the info was passed along to cloudnull ^ | 22:45 |
| fungi | yes, we're talking over privmsg | 22:45 |
| fungi | looks like some local project ids changed | 22:45 |
| clarkb | thanks! | 22:46 |
| clarkb | infra-root I forget to mention in the meeting today but https://review.opendev.org/c/opendev/system-config/+/941997 is an attempt at optimizing our jobs that run when we edit LE things | 22:49 |
| clarkb | since adding new servers tends to edit LE things it is helpful to not run gitea and gerrit jobs there (they are long and can hit docker rate limits) | 22:49 |
| clarkb | cloudnull: fungi: one thing I awnted to ask is if we have to use a region specific keystone for dfw3 or if we can just add it as a new region to our existing clouds.yaml (though figuring out the auth problem seems like the priority there) | 22:53 |
| fungi | clarkb: i've updated the project ids in our private hostvars... i guess we need to manually adjust clouds.yaml on nodepool servers in the meantime until the daily deploy? | 22:55 |
| clarkb | fungi: nodepool gets deployed hourly | 22:56 |
| clarkb | the bridge side may be hourly too | 22:56 |
| fungi | ah, okay, top of the hour is almost upon us, so i'll just wait for it | 22:57 |
| cloudnull | right now you'd have to use region specific keystone | 22:57 |
| cloudnull | over the next couple weeks it'll just be a region from any keystones region | 22:58 |
| cloudnull | ** keystone endpoint | 22:58 |
| fungi | i don't suppose there are updates on ipv6 availability? | 22:58 |
| clarkb | ack. I wonder if we can use a single cloud profile with different auth urls per region or if we need a different profile per region | 22:58 |
| cloudnull | ipv6 - no updates, other than its something we want to do but haven't yet :'( | 22:59 |
| fungi | no worries, just making sure i hadn't missed it. thanks! | 23:00 |
| fungi | new project ids are working on bridge | 23:05 |
| fungi | clouds.yaml on nl01 has updated now too | 23:06 |
| fungi | hopefully we'll see the deleting count fall shortly on https://grafana.opendev.org/d/6d29645669/nodepool3a-rackspace-flex | 23:06 |
| fungi | though the nodepool-launcher process may need restarting? | 23:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add DFW3 to raxflex cloud profiles on bridge https://review.opendev.org/c/opendev/system-config/+/942155 | 23:10 |
| clarkb | fungi: I thought nodepool was supposed to detect updates to clouds.yaml but I'm not 100% certain of that | 23:10 |
| clarkb | I think 942155 has the hack that mordred added to do keystone per region in a single profile in it | 23:10 |
| clarkb | but I haven't tested that that works | 23:10 |
| fungi | looks like nodepool-launcher is still logging the 401 error exception | 23:19 |
| fungi | should i restart its container? | 23:19 |
| clarkb | that should be fine. We auto reload those containers when new images show up anyway so it is used ot it | 23:20 |
| fungi | or sighup? | 23:20 |
| clarkb | I don't think sighup helps but that might be a good feature to add to zuul launcher if it still needs to reread config/ | 23:20 |
| clarkb | https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/AMASOEXESXFINAX5KU4FFV7IMWVBELUS/ made my self nomination official with 40 minutes to spare | 23:20 |
| fungi | the nodepool-launcher container down/up worked, looking at the graph | 23:35 |
| clarkb | cool | 23:35 |
| fungi | so it definitely didn't notice the clouds.yaml change on its own | 23:35 |
| clarkb | re 942155 it just occured to me that {region_name} might be interpretted by the jinja template engine or does it ignore single {}s? | 23:35 |
| corvus | i think that's right. testing should presumably confirm or fail | 23:39 |
| corvus | i'm planning on seeing luca tomorrow for https://www.meetup.com/gerritmeets/events/305718795/ | 23:48 |
| fungi | cool! | 23:53 |
| clarkb | ya I was going to try and catc hthe live stream if thre is one | 23:58 |
| clarkb | anyone have time for https://review.opendev.org/c/opendev/system-config/+/941997 it should be a quick review that I'd liek to get in before launching more noble servers | 23:59 |
| clarkb | should speed up the process of landing config updates for new servers | 23:59 |
| clarkb | but I need a sanity check that there isn't a good reason to have gitea and gerrit run those tests when adding say codesearch | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!