| ianw | it looks like $releasever isn't being expanded in the nodepool builders | 00:00 |
|---|---|---|
| clarkb | fungi: yes I expect this next run will vos release successfully | 00:00 |
| clarkb | but it takes time to get to that point so will be a few minutes before I Know for sure | 00:00 |
| clarkb | do y'all think I neeed to rerun this script a second time or just release the lock once I've got success and let the regular process handle it | 00:01 |
| ianw | clarkb: did you run with NO_TIMEOUT=1 ? | 00:02 |
| clarkb | oh I did not. But its fine it hasn't been that long | 00:03 |
| clarkb | like its getting to that error and fialing not getting to a timeout and failing | 00:04 |
| clarkb | its just like 10 minute sor something | 00:04 |
| clarkb | This is a complete release of volume 536870949 | 00:15 |
| clarkb | hrm maybe that is why I needed to set NO_TIMEOUT? | 00:15 |
| clarkb | ianw: ^ do you know what happens if we timeout the vos_release? | 00:15 |
| ianw | actually i think that bit is ok these days | 00:16 |
| clarkb | anyway reprepro completed successfully. We're just waiting on the vos release now. It is running in screen window 2 on mirror-update and it logs to the regular log file | 00:16 |
| ianw | that is actually an untimed ssh call to do the release on the afs01 directly | 00:16 |
| clarkb | I'll try to check on it after dinner and release the lock, but if it goes longer than that you may need to release the lock | 00:16 |
| ianw | but yeah, we don't want to kill that if we can avoid it, because then we have to clear the locks and do full releases | 00:17 |
| clarkb | ya its running in screen so it should be fine as long as the timeout doens't get it | 00:18 |
| clarkb | crazy idea: default to no timeout then set the timeout flag in the cron jobs sothat manual runs don't have to remember | 00:19 |
| *** dviroel|rover is now known as dviroel|out | 00:20 | |
| ianw | we may be at a point of stability we could avoid the timeouts all together. we haven't had anything hang that i know of in a long time | 00:21 |
| fungi | well, the timeouts might be hiding it though | 00:21 |
| ianw | true, i guess we don't vos release if something times out, so we don't see the partial updates | 00:22 |
| clarkb | Released volume mirror.ubuntu successfully its done already | 00:22 |
| clarkb | do you think we need to run it again or can I release the lock? | 00:22 |
| fungi | but i agree it seems like a worthwhile experiment. those timeouts seem more likely to break than fix stuff these days | 00:22 |
| ianw | yay! i think you can probably release the lock | 00:22 |
| clarkb | done. I'll leave the screen up for further cleanups | 00:23 |
| clarkb | 826GB to 581GB | 00:24 |
| clarkb | not bad | 00:24 |
| clarkb | and now dinner. frickler I think we can probably continue to add jammy stuff now as that frees up quite a bit of room | 00:25 |
| ianw | ok, so the rpm in the centos7 chroot thinks that nothing is installed | 01:00 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Make centos reset-bls-entries behave the same as rhel https://review.opendev.org/c/openstack/diskimage-builder/+/839830 | 01:41 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Parse block device lvm lvs size attributes https://review.opendev.org/c/openstack/diskimage-builder/+/839829 | 01:41 |
| fungi | #status log Replaced block storage volume backup01.ord.rax.opendev.org/main02 with main04 in order to avoid service disruption from upcoming provider maintenance activity | 02:00 |
| opendevstatus | fungi: finished logging | 02:00 |
| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/839837 | 02:28 |
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/839837 | 02:55 |
| ianw | ok, so we create the initial chroot from outside tools (dnf/rpm) on buster. it seems that something has changed, and centos 7 rpm can not read the rpmdb created by this | 03:09 |
| ianw | so, rpm -qa returns none. this flows through to yum, which somehow uses rpm to figure out how to populate $releasedir | 03:10 |
| ianw | $releasever even | 03:10 |
| ianw | and so we see the error where yum can not download | 03:10 |
| ianw | what is weird is that everything is installed. so if you tell yum manually "--releasever=7" it somehow works. it goes off and installs everything, presumably rewrites the rpmdb and the build seems to work | 03:11 |
| ianw | i think i see the problem. the rpm on the host side is creating a rpmdb in sqlite format. centos 7 expects a bdb format rpmdb. these don't share files in common; to centos 7 it looks like a blank directory | 03:59 |
| ianw | you can convert from bdb -> sqlite, but not the other way (the bdb backend is read-only, it seems). the workaround of setting releasever and letting the in-chroot rpm figure it out seems like the best idea now | 04:00 |
| ianw | especially given how much time i want to spend on centos7 issues, which is ~ none | 04:00 |
| *** ysandeep|out is now known as ysandeep | 04:11 | |
| ianw | note centos-7-0000264326 is an "accidentally" fixed build that has been uploaded now | 04:14 |
| ianw | i was working on nb01 because this doesn't replicate in the gate environment, and this build went to completion. i've left the changes in the currently running nb01 container, but if that restarts, it won't build centos-7 again | 04:15 |
| ianw | this is fine for now until we get fixes in, and at least we have a refreshed image | 04:16 |
| ianw | this has worked to build the openafs rpms -- see https://zuul.opendev.org/t/openstack/build/eee983e184de48a2a970de2e97f5dc46 | 04:16 |
| *** bhagyashris|ruck is now known as bhagyashris|sick | 04:37 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: workaround missing $releasedir variable https://review.opendev.org/c/openstack/diskimage-builder/+/839840 | 04:42 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Parse block device lvm lvs size attributes https://review.opendev.org/c/openstack/diskimage-builder/+/839829 | 04:55 |
| *** marios is now known as marios|ruck | 05:06 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Test openafs roles on CentOS 9-stream https://review.opendev.org/c/opendev/system-config/+/839841 | 05:14 |
| *** ysandeep is now known as ysandeep|afk | 05:39 | |
| *** ysandeep|afk is now known as ysandeep | 06:00 | |
| *** jpena|off is now known as jpena | 07:07 | |
| *** ysandeep is now known as ysandeep|lunch | 07:32 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Test openafs roles on CentOS 9-stream https://review.opendev.org/c/opendev/system-config/+/839841 | 07:36 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Remove puppet-kibana https://review.opendev.org/c/opendev/system-config/+/839867 | 07:36 |
| *** ysandeep|lunch is now known as ysandeep | 09:16 | |
| *** marios|ruck is now known as marios|ruck|lunch | 10:23 | |
| *** marios|ruck|lunch is now known as marios|ruck | 10:46 | |
| *** ysandeep is now known as ysandeep|afk | 11:03 | |
| *** iurygregory__ is now known as iurygregory | 11:14 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| frickler | anyone remember the fix for https://github.com/pypa/setuptools/issues/3197 ? this is now affecting keystone-specs cf. https://zuul.opendev.org/t/openstack/build/101ff54f13ba42d4acf8403948515767 | 11:52 |
| frickler | ah, https://review.opendev.org/q/topic:setuptools-issue-3197 seems to have some eamples | 11:54 |
| fungi | frickler: yeah, i still think there's probably something we could do in pbr itself to solve that, since pbr-using packages want to let pbr decide what files to package and not rely on setuptools' file finder feature at all | 12:02 |
| fungi | but it's easily worked around by just preempting the feature in setuptools through configuration | 12:03 |
| opendevreview | chandan kumar proposed zuul/zuul-jobs master: [DNM] ovs debug https://review.opendev.org/c/zuul/zuul-jobs/+/839938 | 12:20 |
| frickler | someone mentioned that https://wiki.openstack.org/wiki/UsingIRC is using an example pic with a big "freenode" on it. I don't think that it's worth updating the pic, maybe just drop it and add a link to our docs page instead? | 13:02 |
| frickler | mainly just mentioning it because this seems to be a relevant path for newcomers to find us | 13:02 |
| * frickler updated the wiki page now | 13:08 | |
| *** ysandeep|afk is now known as ysandeep | 13:36 | |
| *** pojadhav is now known as pojadhav|afk | 13:50 | |
| fungi | thanks! i agree with your suggestion there | 14:04 |
| fungi | i hadn't noticed it (and forgot we even had that page) | 14:05 |
| fungi | #status log Replaced block storage volume mirror01.ord.rax.opendev.org/main01 with main02 in order to avoid service disruption from upcoming provider maintenance activity | 14:25 |
| opendevstatus | fungi: finished logging | 14:25 |
| fungi | that pvmove took close to 12 hours even though the volume was only ~256gb. not sure whether it's because of a slow backend or heavy write activity on the fs | 14:26 |
| fungi | anyway, that's all three of the volumes they warned us about, so i'll close out that ticket | 14:27 |
| *** ysandeep is now known as ysandeep|out | 14:59 | |
| fungi | clarkb: have you been unenrolling servers from ua before deleting? | 15:27 |
| fungi | oh, maybe we didn't enroll things we knew we were planning to decommission | 15:28 |
| fungi | status.o.o: "UA Infra: Extended Security Maintenance (ESM) is not enabled." | 15:28 |
| fungi | so nothing to unenroll there anyway | 15:28 |
| Clark[m] | fungi: but of a slow start today. If I remember to unenroll then yes, but a couple have been missed. And ya ELK and friends weren't enrolled due to quantity and expectation they would go away | 15:37 |
| fungi | thanks for confirming, and yes it dawned on me after i asked that we had consciously omitted those | 15:39 |
| *** dviroel is now known as dviroel|lunch | 15:45 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Decommission status.openstack.org and services https://review.opendev.org/c/opendev/system-config/+/839963 | 15:50 |
| fungi | i expect zuul to tell me i missed something there | 15:50 |
| fungi | the server is offline and being imaged currently | 15:51 |
| fungi | it looks like at a minimum we'll be able to retire puppet-elastic_recheck and puppet-reviewday repos after that merges | 15:52 |
| fungi | probably also the reviewday repo itself | 15:52 |
| *** marios|ruck is now known as marios|out | 15:53 | |
| clarkb | fungi: yes reviewday itself too. But not e-r itself as it is used still | 15:54 |
| clarkb | just not in our systems | 15:54 |
| fungi | right | 15:55 |
| fungi | though i would at this point encourage the current e-r maintainers to merge the rdo branch back into master | 15:55 |
| clarkb | ya probably a good idea so that people don't get confused | 15:55 |
| fungi | status.o.o image has saved successfully. i'm deleting the server and associated dns records now | 16:05 |
| fungi | server instance and dns records all deleted now | 16:12 |
| fungi | #status log Decommissioned the status.openstack.org server as it was no longer hosting any working services: http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028279.html | 16:13 |
| opendevstatus | fungi: finished logging | 16:13 |
| clarkb | fungi: the error on your change above may be related to me missing removal of a repo from our system-config zuul jobs required projects | 16:14 |
| clarkb | fungi: do you want to just fix that in your change or should I push up a separate fix for it? | 16:14 |
| fungi | checking | 16:15 |
| clarkb | specifically I retired puppet-kibana but didnt' remove it from those jobs | 16:15 |
| fungi | oh, good catch. i can roll it into this | 16:16 |
| clarkb | thanks | 16:16 |
| fungi | it's also still in modules.env | 16:16 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Decommission status.openstack.org and services https://review.opendev.org/c/opendev/system-config/+/839963 | 16:18 |
| clarkb | fungi: looks like I also failed to remove health from groups. I'll take a look fixing that with a followup momentarily | 16:21 |
| fungi | i can add it | 16:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove health group from our ansible groups https://review.opendev.org/c/opendev/system-config/+/839966 | 16:26 |
| clarkb | fungi: ^ got it | 16:26 |
| fungi | oh, you got it | 16:26 |
| clarkb | fungi: looks like ianw found the puppet-kibana thing https://review.opendev.org/c/opendev/system-config/+/839867 | 16:30 |
| clarkb | I'm inclined to approve that and then rebase your change on top? Any objections to that? | 16:30 |
| fungi | wfm | 16:30 |
| fungi | approved it | 16:30 |
| *** jpena is now known as jpena|off | 16:30 | |
| clarkb | anyone know if we've got a jammy ubuntu-ports change yet? | 16:31 |
| fungi | i'll also reorder the health cleanup ahead of my change | 16:31 |
| fungi | i haven't seen a jammy ubuntu-ports addition yet, but could have missed it | 16:31 |
| clarkb | ya a quick search on gerrit shows no results. I'll push one up now that we have room | 16:32 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove health group from our ansible groups https://review.opendev.org/c/opendev/system-config/+/839966 | 16:35 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Decommission status.openstack.org and services https://review.opendev.org/c/opendev/system-config/+/839963 | 16:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Mirror Jammy arm64 ubuntu-ports https://review.opendev.org/c/opendev/system-config/+/839972 | 16:35 |
| clarkb | heh that stack has that same diff problem again | 16:35 |
| fungi | yep, because i reordered changes | 16:36 |
| clarkb | fungi: the puppet list is looking very small now :) | 16:39 |
| clarkb | slowly but surely we've made progress | 16:39 |
| opendevreview | Merged opendev/system-config master: Remove puppet-kibana https://review.opendev.org/c/opendev/system-config/+/839867 | 16:44 |
| *** dviroel|lunch is now known as dviroel | 16:47 | |
| * fungi cheers | 16:51 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Enable Gerrit httpd requestLog https://review.opendev.org/c/opendev/system-config/+/839976 | 17:08 |
| clarkb | that is another suggestion that has come out of discussion about Gerrit 3.5's increased memory consumption | 17:08 |
| clarkb | Our test jobs should collect that file and we can compare memory costs between the 3.4 test instance adn 3.5 test instance to start | 17:09 |
| opendevreview | Merged opendev/system-config master: Remove health group from our ansible groups https://review.opendev.org/c/opendev/system-config/+/839966 | 17:10 |
| fungi | yay! 839963 seems to be passing tests, so i guess i didn't miss anything after all | 17:12 |
| clarkb | fungi: we can probably single core approve https://review.opendev.org/c/opendev/system-config/+/839963/ since the server is gone now | 17:23 |
| fungi | yep | 17:23 |
| fungi | please do | 17:23 |
| fungi | it was announced a week ago and there were no concerns raised, so i think we're well covered | 17:24 |
| opendevreview | Merged opendev/system-config master: Decommission status.openstack.org and services https://review.opendev.org/c/opendev/system-config/+/839963 | 17:39 |
| clarkb | looking at the gerrit httpd log from the chagne above and there are some requests that consume more memory on 3.4 and some that use more on 3.5 | 18:08 |
| clarkb | I think we'll need to recheck it a few times and see if we have a stable baseline within a version of gerrit | 18:08 |
| clarkb | and then compare those averages (or discard the info beacuse it is too inconsistent) | 18:08 |
| fungi | makes sense | 18:10 |
| clarkb | that said even if it is too inconsistent in CI I think we should land this for production as I expect it will be far more stable there | 18:11 |
| clarkb | I guess I could try running it without the performance logging thing toggled off and see if we notice a difference in CI too | 18:23 |
| clarkb | ya after this recheck I'll reorder the changes so that we get httplogs with and without performance logging disabled | 18:23 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Clean up defunct OpenStack mailing lists https://review.opendev.org/c/opendev/system-config/+/839990 | 18:29 |
| fungi | that removes almost half of the remaining lists on lists.openstack.org | 18:33 |
| clarkb | its got my +2 :) | 18:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Enable Gerrit httpd requestLog https://review.opendev.org/c/opendev/system-config/+/839976 | 22:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Explicitly disable Gerrit tracing.performanceLogging https://review.opendev.org/c/opendev/system-config/+/839251 | 22:44 |
| mnaser | i don't assume zuul in opendev was restarted with the new unrestricted ansible stuff? | 22:44 |
| clarkb | mnaser: it has not been | 22:44 |
| clarkb | soon probably, but not yet | 22:44 |
| mnaser | aw okay, i'm looking super forward to it :) probably won't happen on a friday ;) | 22:44 |
| fungi | probably this weekend | 22:47 |
| fungi | or first thing in the week | 22:48 |
| clarkb | looking at the two 3.4 http logs and the two 3.5 http logs they do seem reasonably stable within the same version of gerrit | 22:51 |
| clarkb | and then the differences between the two versions are fairly minor | 22:51 |
| clarkb | git-upload-pack used more memory on 3.4 than 3.5 | 22:52 |
| clarkb | adding an ssh key used more memory on 3.5 than 3.4 but not significantly so its a few percent | 22:53 |
| clarkb | ssh key handling is the biggest diffefrence I've been able to find between the tw | 22:54 |
| clarkb | I guess we need to see the numbers without performance logging disabled | 22:54 |
| clarkb | because nothing here is making me concerned yet | 22:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!