| ianw | ok, done as much on the nodepool gate breakage as i can, going to have a quick crack at ^ now | 04:20 |
|---|---|---|
| ianw | i've got a root screen up on mirror-update | 04:22 |
| *** ysandeep|out is now known as ysandeep | 04:31 | |
| ianw | i'm doing the deleteunreferenced now | 04:53 |
| ianw | that took us from 395 - 256 = 140gb ... not bad | 06:19 |
| ianw | i've dropped the lock, but will leave the root screen just incase we need a scrollback on what i did | 06:19 |
| ianw | #status log To save mirror volume space, we have removed source packages from the ubuntu-ports repository | 06:20 |
| opendevstatus | ianw: finished logging | 06:20 |
| *** jpena|off is now known as jpena | 07:27 | |
| *** ysandeep is now known as ysandeep|lunch | 08:12 | |
| *** ysandeep|lunch is now known as ysandeep | 08:52 | |
| jrosser | i kicked the tyres on the ubuntu jammy image and I think the disk may not be expanded? https://zuul.opendev.org/t/openstack/build/23ce4c4d007a478da4067e48c0bd4638 | 09:58 |
| *** rlandy|out is now known as rlandy | 10:21 | |
| ianw | jrosser: hrm, interesting. the nodepool functional test does actually test our growroot grows. not to say that it's impossible, but it has slipped past testing at least :) | 10:26 |
| ianw | https://opendev.org/zuul/nodepool/src/branch/master/tools/functional-test-check.sh#L35 is where we do that | 10:28 |
| frickler | ianw: jrosser: I'm not sure if that's disk full on the node actually, or log size on the executor reaching the limit? also note that our jammy repo mirror may still be broken | 10:31 |
| frickler | actually devstack got along much further today than yesterday. but indeed I see no growroot being done there, either https://1ae34c77782edc957e3d-ea76e3e710a8219c6c723d407750283e.ssl.cf1.rackcdn.com/839389/2/check/devstack-platform-ubuntu-jammy/2dfd8e4/job-output.txt | 10:33 |
| frickler | ah, that's because devstack trims the log. the root partition size looks fine here https://1ae34c77782edc957e3d-ea76e3e710a8219c6c723d407750283e.ssl.cf1.rackcdn.com/839389/2/check/devstack-platform-ubuntu-jammy/2dfd8e4/controller/logs/df.txt | 10:36 |
| jrosser | aaaah ok i misunderstood the error message there | 10:36 |
| ianw | frickler: i think it's a service, so will happen on boot and not be captured in job-output.txt; syslog dumps on devstack may have it i guess | 10:37 |
| ianw | oh, but it seems it's probably growing. and yeah, i'd agree that if zuul is reporting DISK_FULL it's from an executor POV, as it won't know the disk is full on the executing node side | 10:38 |
| *** dviroel|rover|out is now known as dviroel|rover | 11:16 | |
| fungi | note that we also record things like df output at the start of most jobs with a common role and capture that in a log like zuul-info/zuul-info-ubuntu-jammy.txt | 11:34 |
| *** pojadhav is now known as pojadhav|afk | 11:48 | |
| gthiemonge | Hi Folks, we got a weird failure in the CI this morning: https://zuul.opendev.org/t/openstack/build/7443aa7e7fbd4948b199f79fd6b7faee | 11:52 |
| fungi | gthiemonge: we occasionally see that when the cloud has a rogue virtual machine squatting some ip address nova/neutron think is available and have allocated to a new instance | 12:00 |
| fungi | the gateway's arp table ends up with that entry repeatedly overwritten by the new and old mac, so ~half of connections to the ip address go to the rogue vm | 12:01 |
| fungi | if i can manage to ssh into the old node (by randomly trying that same ip address over and over), i'll shut it down | 12:01 |
| fungi | of course, that assumes the old vm was also one of our test nodes (it may belong to another tenant entirely) | 12:02 |
| gthiemonge | fungi: ack | 12:04 |
| fungi | i managed to catch it hitting the old vm i think, because i just saw the host key change on me, but attempts to ssh into it are hanging | 12:05 |
| fungi | and when i finally managed to get in, i wound up back at the new instance again | 12:05 |
| fungi | yeah, i think it wasn't one of our test nodes. i finally ended up getting a password prompt instead of my admin key logging me straight in | 12:07 |
| fungi | anyway, the provider (rackspace in this case) seems to run periodic cleanup sweeps for such situations so hopefully it'll disappear on its own soon enough, but if not we can open a support ticket | 12:08 |
| *** pojadhav|afk is now known as pojadhav | 12:26 | |
| *** ysandeep is now known as ysandeep|afk | 12:42 | |
| *** ysandeep|afk is now known as ysandeep | 13:00 | |
| *** ysandeep is now known as ysandeep|out | 14:21 | |
| *** marios is now known as marios|out | 15:28 | |
| *** dviroel|rover is now known as dviroel|rover|lunch | 15:29 | |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Revert "Fallback to persistent netifs names with systemd" https://review.opendev.org/c/openstack/diskimage-builder/+/838863 | 15:41 |
| mnasiadka | Hello - is someone working on ubuntu-jammy-arm64? Only x86_64 is available at the moment from what I see | 15:54 |
| clarkb | mnasiadka: we're taking it one step at a time right now. x86 jammy isn't fully functional yet I don't think | 15:54 |
| clarkb | mnasiadka: the next steps for arm64 are pruning our AFS disk usage a bit more so that we can add the arm64 jammy mirror. And then we can add the image | 15:55 |
| mnasiadka | great, thanks - just asking | 15:55 |
| clarkb | yseterday we removed ubuntu-ports source packages which pruned about 140GB of disk. Next we should prune ubuntu x86 mirror source packages | 15:55 |
| clarkb | fungi: ^ do you knwo if there is a change for that assuming we're happy with the ports results? | 15:55 |
| fungi | there isn't yet, but i did double-check things this morning and they look good after ianw manually pruned the mirror | 15:56 |
| fungi | i'll get one pushed up shortly i hope, today's just been full of other distractions | 15:56 |
| clarkb | thanks! | 15:57 |
| clarkb | and ya that was a good improvement on the prots side, would be good to get that applied to ubuntu and eventually debian once we're set with zuul-jobs' configure-mirrors | 15:57 |
| mtreinish | gmann: I think we'll have to update the tempest removal process. The data was missing from subunit2sql anyway the last time I looked because the prune cron job deleted everything after the newer oslo-db/sqlachemy version got pulled into the worker and broke the updates | 16:03 |
| mtreinish | so there shouldn't be anything blocking deleting the db | 16:03 |
| fungi | thanks for confirming, mtreinish! | 16:04 |
| gmann | mtreinish: thanks, yeah I will update the tempest process. | 16:07 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Enable deb-src on Debian with a flag in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/839592 | 16:17 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch enable_src_repos to False in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/839593 | 16:17 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add enable_deb_src_repos support for Ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/839594 | 16:17 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 16:18 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add enable_deb_src_repos support for Ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/839594 | 16:18 |
| opendevreview | Maksim Malchuk proposed openstack/diskimage-builder master: Revert "Fallback to persistent netifs names with systemd" https://review.opendev.org/c/openstack/diskimage-builder/+/838863 | 16:29 |
| corvus | clarkb: fungi any zuul weirdness overnight? | 16:34 |
| clarkb | corvus: the only thing I saw was the cloud arp fight between instances. Nothing on the zuul end | 16:34 |
| corvus | the buildsets page looks a lot more sane to me (which is the main thing i was brown-bag-fixing) | 16:36 |
| corvus | i'll go ahead and push the tag now | 16:36 |
| opendevreview | Jeremy Stanley proposed openstack/diskimage-builder master: Switch to release-notes-jobs-python3 https://review.opendev.org/c/openstack/diskimage-builder/+/839599 | 16:43 |
| fungi | corvus: i concur, nothing out of the "ordinary" (or what passes for ordinary in these parts) | 16:45 |
| clarkb | mtreinish: thank you for confirming | 16:50 |
| clarkb | infra-root I'm going to proceed with deleting the subunit2sql trove mysql db now | 16:50 |
| clarkb | 5d0dc5ac-7a3e-4568-9acb-e62406129666 subunit2sql-MySQL <- that is the instance that will be deleted | 16:53 |
| clarkb | it hosts a mysql db named subunit2sql with query and subunit2sql users. That all looks like the correct one to me so I'm going to click the button now | 16:54 |
| clarkb | and done | 16:55 |
| clarkb | it doesn't show up in the last anymore | 16:55 |
| clarkb | s/last/list/ | 16:55 |
| clarkb | infra-root any objection to me approving the puppet config management retirements for ELK and subunit2sql/health now? | 16:56 |
| clarkb | fungi reviewed them and abandoned changes | 16:56 |
| opendevreview | Merged zuul/zuul-jobs master: Enable deb-src on Debian with a flag in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/839592 | 16:58 |
| *** jpena is now known as jpena|off | 17:01 | |
| clarkb | #status log Deleted the old subunit2sql database now that the OpenStack CI Health dashboard and subunit2sql workers have been removed. | 17:02 |
| opendevstatus | clarkb: finished logging | 17:02 |
| fungi | clarkb: feel free to self-approve, or i'm happy to single-core approve them | 17:04 |
| TheJulia | What would it take to get two VMs part of a multinode job held to permit manual investigation of the state? | 17:07 |
| clarkb | TheJulia: you just need to let us know what job and change, then we mark it and you recheck if necessary. Then once failed we can add your ssh key to the hosts | 17:07 |
| clarkb | fungi: ya I'll go ahead and start approving them | 17:07 |
| TheJulia | change 839086,9, job ironic-grenade-multinode-multitenant. Presently running at the moment. | 17:09 |
| fungi | i can add a hold for that | 17:09 |
| TheJulia | Much appreciated! | 17:09 |
| opendevreview | Merged opendev/puppet-subunit2sql master: Retire this project https://review.opendev.org/c/opendev/puppet-subunit2sql/+/839242 | 17:09 |
| opendevreview | Merged opendev/puppet-logstash master: Retire this project https://review.opendev.org/c/opendev/puppet-logstash/+/839240 | 17:09 |
| opendevreview | Merged opendev/puppet-log_processor master: Retire this project https://review.opendev.org/c/opendev/puppet-log_processor/+/839239 | 17:09 |
| opendevreview | Merged opendev/puppet-elasticsearch master: Retire this project https://review.opendev.org/c/opendev/puppet-elasticsearch/+/839238 | 17:09 |
| opendevreview | Merged opendev/puppet-kibana master: Retire this project https://review.opendev.org/c/opendev/puppet-kibana/+/839237 | 17:10 |
| TheJulia | Unfortunately it looks like traffic just doesn't make it across the vxlan tunnel, and I've solved the most recent two errors afaik, and it still looks like from what has scrolled across the screen that this is the case... :\ | 17:10 |
| clarkb | vxlan itself should be fairly stable considering the number of jobs that make use of it. Wouldn't surprise me if it is a route or firewall issue (and holding nodes in that case can definitely help) | 17:11 |
| TheJulia | yeah, that is kind of what I'm thinking at this point | 17:12 |
| clarkb | fungi: https://review.opendev.org/c/openstack/project-config/+/839243 is the change to finalize those repo retirements | 17:12 |
| fungi | | 0000000040 | openstack | opendev.org/openstack/ironic | ironic-grenade-multinode-multitenant | refs/changes/86/839086/9 | 1 | TheJulia troubleshooting XVLAN connectivity problems | | 17:13 |
| fungi | TheJulia: when that build fails, let me know what ssh key(s) you want added to the nodes for it | 17:13 |
| TheJulia | fungi: awesome | 17:14 |
| TheJulia | fungi: ended https://paste.openstack.org/show/bHHmaYhfkP2YOm1qCy1q/ | 18:16 |
| fungi | TheJulia: ssh root@198.72.124.82 and 198.72.124.122 | 18:20 |
| TheJulia | fungi: much appreciated, I'll let you know when I'm done | 18:20 |
| TheJulia | hopefully this will be quick and easy | 18:20 |
| fungi | my pleasure. have "fun" | 18:20 |
| TheJulia | lol | 18:21 |
| clarkb | fungi: I'm thinking for cleanup of the debian source packages we can probably wait until we flip the default to false in zuul-jobs? Or do you think it is prudent to udpate our base job? | 18:33 |
| clarkb | I suspect that clearing out ubuntu and ubuntu-ports source packages will be sufficient to free up plenty of room for arm64 jammy packages | 18:33 |
| clarkb | justbased on the numbers we already got from cleaning up -ports | 18:33 |
| fungi | clarkb: i have it noted in the commit message, just a moment | 18:38 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for debian-docker https://review.opendev.org/c/opendev/system-config/+/839621 | 18:38 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for ubuntu https://review.opendev.org/c/opendev/system-config/+/839622 | 18:38 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for debian https://review.opendev.org/c/opendev/system-config/+/839623 | 18:38 |
| fungi | since i don't see a wip change to switch the default yet, i'll wip that last change until there's something it can depends-on | 18:39 |
| fungi | but to answer your question, i would wait for the zuul-jobs change unless we want to serve as a canary and add an override in our base job | 18:41 |
| fungi | i agree we're probably in no real hurry for the debian change, ubuntu should free plenty already | 18:42 |
| fungi | the only reason i can think of to override in base now is to make sure the non-default value works. but it's ultimately 3 changes instead of just 1 so i wouldn't unless we're worried it won't | 18:43 |
| fungi | (3 changes to override in base-jobs, switch the default in zuul-jobs, and then clean up the redundant override in base-jobs) | 18:44 |
| Clark[m] | fungi: https://review.opendev.org/c/zuul/zuul-jobs/+/839593 is the change you can depends on | 18:44 |
| fungi | more than 3 changes if we also want to be cautious and exercise one or both base-jobs changes in base-test too | 18:44 |
| fungi | aha, thanks, i missed 839593 | 18:44 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for debian https://review.opendev.org/c/opendev/system-config/+/839623 | 18:45 |
| *** artom__ is now known as artom | 18:53 | |
| clarkb | fungi: I think https://review.opendev.org/c/opendev/system-config/+/839621/1/playbooks/roles/reprepro/files/debian-security/config/distributions was meant to go in the last change of the stack not the first? | 19:09 |
| clarkb | fungi: if you have time for https://review.opendev.org/c/openstack/project-config/+/839243 that would be great too | 19:18 |
| fungi | clarkb: oops, thanks! | 19:27 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for debian-docker https://review.opendev.org/c/opendev/system-config/+/839621 | 19:31 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for ubuntu https://review.opendev.org/c/opendev/system-config/+/839622 | 19:31 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop mirroring source packages for debian https://review.opendev.org/c/opendev/system-config/+/839623 | 19:31 |
| clarkb | thats weird the diff between patchsets doesn't show a diff because the parent always had the change before | 19:34 |
| opendevreview | Gage Hugo proposed opendev/irc-meetings master: Update security sig chair details https://review.opendev.org/c/opendev/irc-meetings/+/839632 | 19:34 |
| clarkb | unfortaunte that it makes updates like this difficult to see but at least it makes snse | 19:34 |
| fungi | yeah | 19:39 |
| fungi | rebases are always a bit... wonky in gerrit | 19:39 |
| fungi | you should be able to see the file removed from the patchset diff in the first change, and nothing different in the third change | 19:40 |
| clarkb | ya the file isn't listed in the first two patchsets but then shows up in the third but no delta between ps1 and ps3 or ps2 and ps3 due to the parent carrying those updates previously | 19:42 |
| opendevreview | Merged openstack/project-config master: Finalize ELK puppetry retirement https://review.opendev.org/c/openstack/project-config/+/839243 | 19:44 |
| *** rlandy is now known as rlandy|mtg | 20:28 | |
| *** dviroel|rover is now known as dviroel|rover|biab | 21:12 | |
| *** rlandy|mtg is now known as rlandy | 21:27 | |
| *** rlandy is now known as rlandy|bbl | 22:10 | |
| TheJulia | fungi: you can take back those vms now, thanks! | 22:24 |
| fungi | thanks TheJulia! done | 22:25 |
| *** dviroel|rover|biab is now known as dviroel|rover | 22:31 | |
| *** dviroel|rover is now known as dviroel|rover|out | 22:54 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!