| @mnasiadka:matrix.org | Clark: the cron jobs are also processing orhpans, but IIRC in a delayed manner, so it removes orphans from previous run - so either manual or 2 days patience | 06:32 |
|---|---|---|
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org marked as active: [openstack/project-config] 1000918: Replace 2026.2/Hibiscus key with 2027.1/Indri https://review.opendev.org/c/openstack/project-config/+/1000918 | 13:50 | |
| @clarkb:matrix.org | I think we actually run it every 4 hours so maybe that will automatically clear itself up after removing the source packages within a reasonable timeframe | 14:34 |
| @mnasiadka:matrix.org | ah right | 14:54 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [openstack/project-config] 1000918: Replace 2026.2/Hibiscus key with 2027.1/Indri https://review.opendev.org/c/openstack/project-config/+/1000918 | 15:17 | |
| @clarkb:matrix.org | if we want to land the contents file removal then the debian source package removal changes I should be able to keep half an eye on them. I need to prepare for our meetings tomorrow and I said I would do some issue tracker testing before braindumping into a system but all of that should be interruptable if it comes to that | 15:23 |
| @clarkb:matrix.org | firefox 157 is very purple | 15:34 |
| @clarkb:matrix.org | https://review.opendev.org/c/openstack/project-config/+/1008533 is a simply cleanup of the wheel mirror volumes from afs' grafana dashboard. And https://review.opendev.org/c/openstack/project-config/+/1008493 drops systemd and ntp panels from the node exporter grafana dashboard. YOu can see they are gone at: https://6775e6836acabb2eb71a-a48d7e7ba394f54f00f691c127d9dbeb.ssl.cf2.rackcdn.com/openstack/453259ef653a49a2a3852af45d565905/screenshots/node-exporter-full.png | 15:42 |
| @clarkb:matrix.org | Looks like the disk metrics are still not reporting. I'm thinking maybe I need to manually restart node exporter on prometheus01? I'll take a look at that shortly | 15:42 |
| @clarkb:matrix.org | ok looks like node exporter was automatically restarted when the docker compose file updated which is good that works as expected. I checked the metrics with curl and still got the errors though. I went ahead and down up'd node-exporter just in case it was a race with the file updating or something like that but this hasn't changed anything. I suspect this means there are additional issues with collecting the data and they are not directly reproducable in CI :/ | 15:52 |
| @clarkb:matrix.org | I wonder if the issue is that the filesystem collector short circuits on first failure and its not able to find info about another filesystem | 15:56 |
| @clarkb:matrix.org | `node_filesystem_device_error{device="/dev/vda1",device_error="no such file or directory",fstype="ext4",mountpoint="/"} 1` is the error in the metrics. mount shows `/dev/vda1 on /host type ext4 (ro,relatime,discard,errors=remount-ro,commit=30)` but /dev has no vda or vda1 entries. I'm guessing that is the issue | 15:59 |
| @clarkb:matrix.org | but now I wonder why this apparently works in CI but not in the actual deployment. May need to hold a node and make comparisons I guess | 15:59 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 893571: DNM Forced fail on Gerrit to test 3.14 upgrades and downgrades https://review.opendev.org/c/opendev/system-config/+/893571 | 16:05 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008830: DNM force failure on node exporter test node https://review.opendev.org/c/opendev/system-config/+/1008830 | 16:08 | |
| -@gerrit:opendev.org- Clark Boylan proposed: | 17:35 | |
| - [opendev/system-config] 1008830: DNM force failure on node exporter test node https://review.opendev.org/c/opendev/system-config/+/1008830 | ||
| - [opendev/system-config] 1008848: Fix node exporter testinfra test cases host value https://review.opendev.org/c/opendev/system-config/+/1008848 | ||
| @clarkb:matrix.org | I think that at least partially explains the mismatch in behahaviors. The testing wasn't doing what we expected and skipped those test cases | 17:36 |
| @clarkb:matrix.org | Mondays are my do system updates day and today was no different. Ran into a weird behavior that is probably hardware specific? with fwupdmgr on the laptop. It wanted to update the bios and uefi dbx db. It failed to update the uefi dbx on the first pass I think because the reboot only handled the bios firmware update. Or maybe they ship different versions for different bios/uef versions? | 18:36 |
| @clarkb:matrix.org | in any case its worth checking fwupdmgr after you reboot for updates to make sure that all updates apply as expected because I've just run into a case where I had to do two passes to get updated | 18:36 |
| @fungicide:matrix.org | i saw the same thing last week, might have been the same update. after a reboot fwupd claimed the previously-applied update was not successfully applied, but then stopped complaining after that | 18:48 |
| @fungicide:matrix.org | and i saw it on several systems, so maybe not especially hardware-specific at all | 18:51 |
| @clarkb:matrix.org | huh weird | 18:51 |
| @clarkb:matrix.org | but ya reapplying it and rebooting again and now it is happy | 18:51 |
| @clarkb:matrix.org | I'm working on meeting agenda updates and just updated the agenda with a rundown of the openafs situation as I see it. Gives pointers to where things can be reviewed etc | 18:54 |
| @clarkb:matrix.org | if anyone has time to look at that and keep things moving forward that would be great | 18:54 |
| @clarkb:matrix.org | in particular there is a change to drop debian source packages. I think of all the proposals that is the only non impactful change that will have a measurable difference at this point. Everything else I can see is likely to have impacts | 19:01 |
| @clarkb:matrix.org | so getting that done (note it has a parent change to drop contents files) is probably a good idea so we can determine if we need to go further from there | 19:02 |
| @fungicide:matrix.org | yes, i think those two changes together will free up quite a bit of space and speed up reprepro run duration as well | 19:02 |
| @fungicide:matrix.org | we do need to make sure we don't include `deb-src` in the sources list for our images though | 19:03 |
| @fungicide:matrix.org | only 'deb' entries | 19:03 |
| @clarkb:matrix.org | I think that is already the case because we don't include source repos in ubuntu or older debian | 19:04 |
| @clarkb:matrix.org | and they all share a common heritage in the repo setup in the jobs iirc | 19:04 |
| @fungicide:matrix.org | and i think it will negatively impact jobs that try to do something like `apt build-dep` or similar to install package build dependencies, but also that won't work without `deb-src` sources list entries anyway | 19:05 |
| @clarkb:matrix.org | enable_deb_src_repos is the flag for that looks like | 19:05 |
| @fungicide:matrix.org | in dib? | 19:05 |
| @clarkb:matrix.org | in zuul-jobs | 19:05 |
| @clarkb:matrix.org | https://codesearch.opendev.org/?q=enable_deb_src_repos&i=nope&literal=nope&files=&excludeFiles=&repos= we don't set it anywhere and we default to false | 19:06 |
| @clarkb:matrix.org | basically we haven't had source packages anywhere else but we accidentally added them to newer debian mirrors. I think our practice of not having them in general means we are unlikely to be relying on them in new debian (not impossible though_) | 19:07 |
| @clarkb:matrix.org | fwiw I don't see deb-src showing up in dib either with | 19:18 |
| @clarkb:matrix.org | * fwiw I don't see deb-src showing up in dib either with git grep | 19:18 |
| @fungicide:matrix.org | yeah, seems like we should be fine | 19:19 |
| @clarkb:matrix.org | ok fixing the node name issue in the testinfra tests so that we actually run the node exporter test cases causes them to fail so we are reproducing this issue in CI | 20:14 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008848: Fix node exportor command args and test cases https://review.opendev.org/c/opendev/system-config/+/1008848 | 20:48 | |
| @clarkb:matrix.org | ok after debugging on the held node I think it was my docker compose yaml edits that broke it :/ ^ should work now though I may have bugs in the test case as I haven't run the testinfra stuff on the held node | 20:48 |
| @clarkb:matrix.org | I'm going to pop out for a bike ride but I'll followup on ^ when I get back and send out tomorrow's meeting agenda as well | 20:55 |
| @clarkb:matrix.org | Looks like it worked as is and the test case ran https://zuul.opendev.org/t/openstack/build/8e0df5c1a0774edaa4df476e6e222251/log/job-output.txt#26450 | 22:49 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
