| clarkb | I got it working via flags=(attach_disconnected) in the rsyslogd profile but I think I may have had to restart processes (both for rsyslogd and haproxy) for it to take effect. sarnold was very helpful in the ubuntu security channel | 00:14 |
|---|---|---|
| clarkb | not in a great spot to push a patch up for that at the moment as that meeting is happeing in ~40 minutes and I need to help get kids out the door to an activity. I'll try and followup in the morning | 00:21 |
| fungi | dan_with: cloudnull: just saw another "blip" reaching sjc3. lost contact with a vm there at precisely 00:08:44 utc and regained access at about 00:20 utc. during the disconnect i was getting this when trying to reach the api with openstackclient: "Request to https://keystone.api.sjc3.rackspacecloud.com/v3/auth/tokens timed out" | 00:22 |
| fungi | so again neither server instances nor the api were reachable | 00:23 |
| clarkb | I filed https://bugs.launchpad.net/apparmor/+bug/2098148 at sarnold's request | 00:44 |
| clarkb | I'll leave zuul-lb02 in a working manually configured state for now and look to addressing that with a new change tomorrow morning | 00:48 |
| *** dmellado075539372 is now known as dmellado07553937 | 06:50 | |
| opendevreview | Vladimir Kozhukalov proposed zuul/zuul-jobs master: [remove-registry-tag] Allow using in a loop https://review.opendev.org/c/zuul/zuul-jobs/+/941516 | 08:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install apparmor when installing podman https://review.opendev.org/c/opendev/system-config/+/941471 | 16:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix haproxy access to rsyslogd on Noble https://review.opendev.org/c/opendev/system-config/+/941576 | 16:31 |
| clarkb | I split that out into two changes. The first installs apparmor with podman on noble and disables the new test on zuul-lb since it is already noble running podman and broken with apparmor. Then the second change applies the fix that was suggested to me yesterday (and manually applied to zuul-lb02) as well as readding the test that would fail | 16:32 |
| clarkb | 941471 is still bouncing off rate limits but 941576 looks like it will pass | 18:01 |
| clarkb | https://c3380332c392aeb2c300-d82475c388238604ca15d044b3307e59.ssl.cf2.rackcdn.com/941576/1/check/system-config-run-zuul/e4307bb/bridge99.opendev.org/ara-report/results/767.html I think this and then the subsequent apparmor_parser -r command task as well as successful testinfra tests are a good indication this is happy now | 18:05 |
| clarkb | reviews welcome as well as thoughts on how we want ot land 941471 (we could temporarily trim the job list or force merge ore just recheck into oblivion) | 18:06 |
| fungi | i need to go run a couple of errands briefly before weather gets here, but will take a look in an hour-ish | 18:39 |
| TheJulia | Question regarding cloud providers, is rax all the newer rax infra? | 18:39 |
| fungi | TheJulia: no, rax-dfw, rax-iad and rax-ord are all the old rackspace classic; raxflex-sjc3 is the new rackspace flex | 19:30 |
| fungi | and we just got word this week there's a new flex region so we'll hopefully add raxflex-dfw3 in short order | 19:30 |
| fungi | but right now we have about 10x as much quota in classic as flex, so most jobs that run in rackspace are on the older hardware still | 19:31 |
| clarkb | I thinjk it would be great to expand our quota in sjc3 if rax is up to it but before we do that we should redeploy everything to get the new network mtus | 19:32 |
| clarkb | infra-root https://review.opendev.org/c/opendev/system-config/+/941304/ is a change that should be an easy review that isn't directly imapcted by docker rate limits | 19:34 |
| clarkb | and now that I've got a handle on haproxy and noble and podman and apparmor https://review.opendev.org/c/opendev/system-config/+/941130 would be a good one to continue on with to deploy the new codesaerch if we're comfortable with where things ended up. I suppose we could wait for the apparmor installation in testing to ensure we don't miss any bugs there | 19:35 |
| clarkb | thinking about 941471 more I think the risk it presents is very low as it only affects testing and noble nodes (of which we don't have many and they have the new packges preinstalled in prod) | 19:42 |
| clarkb | I'm willing to give that maybe one more recheck but if it consistently fails I think we should consider force merging given the low risk but also reltiavely high beneift to the noble rollout | 19:43 |
| fungi | we have seen all the jobs pass for 941471, just not all at the same time | 19:51 |
| fungi | and the failures have been docker rate limit issues, not failures in the changes, and getting all 30+ to avoid those rate limits in one go is not realistic, and this change is in service of trying to incrementally address the problem causing the failures in the first place... given all that, i'm fine bypassing the gate to merge it | 19:53 |
| clarkb | yup though I did just push a new patchset toady the only difference for it is it removed a test case that a different change pulled in | 19:53 |
| clarkb | that train of thought is basically where I ended up. its safe it should help eventually make the rate limits better so maybe we just go for it | 19:54 |
| clarkb | then the followup change to fix haproxy on noble can go through normal review and ci | 19:54 |
| clarkb | as that one is a bit more meaningful | 19:54 |
| fungi | i concur. if another infra-root is on hand to +2 941471 i'm willing to do the commands to bypass gating | 19:56 |
| opendevreview | Merged opendev/system-config master: Mirror docker tool images https://review.opendev.org/c/opendev/system-config/+/941304 | 19:56 |
| Clark[m] | fungi: thanks. I'm in search of lunch now and hesitate to +2 my own change for that purpose | 20:00 |
| fungi | sure, i don't think it's dire though if we have to wait a bit for someone else to confirm consensus | 20:01 |
| Clark[m] | ++ | 20:03 |
| fungi | dan_with: cloudnull: yet another "blip" reaching sjc3. lost contact at precisely 20:37:27 utc and regained access at 20:48:51 utc. during the disconnect, my traceroutes to api.sjc3.rackspacecloud.com were dying at my service provider's border, but once it was restored i started seeing responses from datapipe and your network... could it be routes dropping out of bgp? | 20:51 |
| clarkb | I guess corvus and/or tonyb would be the folks available to weigh in on force merging 941471 as a second +2 at this time of day (probably a bit early for tonyb still) | 20:53 |
| corvus | agree | 20:56 |
| clarkb | thanks! fungi you mentioend volunteering to do the honors I think we're ready whenever you are | 20:59 |
| fungi | yep, on it now | 21:01 |
| opendevreview | Merged opendev/system-config master: Install apparmor when installing podman https://review.opendev.org/c/opendev/system-config/+/941471 | 21:06 |
| fungi | opendev-promote-docs failed for ^ but i think it's because there was no gate build artifact for it to pull | 21:09 |
| fungi | i also removed and readded my approval on the child change since it seemed like the way i merged it didn't signal to zuul that it could act | 21:10 |
| fungi | but it's enqueued normally now | 21:10 |
| clarkb | the child landing should hopefully address the docs problem too | 21:14 |
| clarkb | I don't think there are important doc updates anyway | 21:14 |
| fungi | yep | 21:34 |
| JayF | clarkb: it's here https://usercontent.irccloud-cdn.com/file/A3q4CZFE/irccloudcapture4588956697937453146.jpg | 21:39 |
| clarkb | JayF: it seems to have stopped here and moved to your neighborhood | 21:41 |
| clarkb | we ended up with about 1-2" so not much but enough to be fun/dangerous. Supposedly more tonight and tomorrow morning | 21:41 |
| clarkb | the gitea-lb job completed and seems to have nooped as expected | 21:44 |
| clarkb | oh though this first chagne would only affect noble nodes since the only prod chagne was adding apparmor to the package list | 21:46 |
| clarkb | looking at grafana02 I think we've nooped there as well | 21:46 |
| clarkb | ya the install docker compose and friends task reports no change on grafana02 | 21:48 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Reapply "Switch zuul.o.o to zuul-lb02" https://review.opendev.org/c/opendev/zone-opendev.org/+/941605 | 22:15 |
| clarkb | the change to apply the rsyslogd fixups should merge soon | 22:15 |
| clarkb | if that deploys happily then I think we're ready to try 941605 again | 22:16 |
| opendevreview | Merged opendev/system-config master: Fix haproxy access to rsyslogd on Noble https://review.opendev.org/c/opendev/system-config/+/941576 | 22:41 |
| *** dmellado075539373 is now known as dmellado07553937 | 22:48 | |
| clarkb | that deployment appears to have been a noop for zuul-lb01 and zuul-lb02. In lb02's case that is beacuse I manually made those cahgnes yesterday and lb01 properly skipped as the platform is too old | 22:50 |
| clarkb | https://zuul-lb02.opendev.org seems to be working for me. Any objection to approving 941605? | 22:51 |
| fungi | i approved it just now | 22:52 |
| clarkb | thanks! | 22:52 |
| fungi | i should at least be around long enough to test once it deploys and dns changes propogate | 22:53 |
| clarkb | I'm around. I'll poke out at some point for snow round two but I'm not in a huge hurry | 22:55 |
| opendevreview | Merged opendev/zone-opendev.org master: Reapply "Switch zuul.o.o to zuul-lb02" https://review.opendev.org/c/opendev/zone-opendev.org/+/941605 | 22:59 |
| clarkb | zuul.opendev.org resolves to zuul-lb02 for me now | 23:03 |
| clarkb | performing a hard refresh against my open browser tab for zuul status gets me content | 23:03 |
| clarkb | I'm starting to see connections trickle into the /var/log/haproxy.log file on zuul-lb02 too | 23:03 |
| fungi | yeah, resolving to zuul-lb02 for me here too, and i'm browsing it fine | 23:05 |
| fungi | i didn't have an open tab | 23:05 |
| clarkb | tomorrow I can unwind zuul-lb01 and work on its cleanup and also start with codesarch02 again https://review.opendev.org/c/opendev/system-config/+/941130 is the change to get the initial deployment rolling (dns for it is already in place so should be safe to land that whenever) | 23:06 |
| opendevreview | Merged opendev/system-config master: Deploy codesearch02 https://review.opendev.org/c/opendev/system-config/+/941130 | 23:55 |
| fungi | it's already deploying, good timing! | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
