| tkajinam | frickler, sorry I was out because of some personal matters | 03:17 |
|---|---|---|
| tkajinam | frickler, so far telemetry is blocked by it (we attempted to fix it in https://review.opendev.org/c/openstack/telemetry-tempest-plugin/+/910572 but are hitting different neutron problem it seems...) | 03:20 |
| tkajinam | there are a few more patches related to tripleo removal currently blocked. quickly checking these, some of thedse may be different problems but I'm unsure if we can fully complete these before the release timing | 03:21 |
| frickler | tkajinam: well I looked at that tempest failures and I really fail to see how they could be related to the dib-utils removal. so I'm pretty convinced the re-adding it will not help and the tempest team will fix their CI even after that | 06:50 |
| frickler | *will need to fix | 06:55 |
| tkajinam | yeah. I agree with that | 07:37 |
| opendevreview | Merged openstack/project-config master: Retire OpenStack-Chef: remove project from infra https://review.opendev.org/c/openstack/project-config/+/910100 | 07:53 |
| opendevreview | Ettore Simone proposed openstack/diskimage-builder master: Cleanup uses of DIB_IMAGE_ROOT_FS_UUID variable https://review.opendev.org/c/openstack/diskimage-builder/+/905556 | 11:10 |
| *** tosky_ is now known as tosky | 11:36 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add more info to afs fileserver recovery docs https://review.opendev.org/c/opendev/system-config/+/910662 | 16:22 |
| clarkb | fungi: ianw ^ that should address your comments | 16:22 |
| fungi | thanks! | 16:22 |
| clarkb | for mariadb upgrades we did the easy one :). Any opinions on which we should do next out of gerrit, mailman3, etherpad, refstack, and gitea? I'm leaning towards etherpad and refstack. I think gitea can be done as well, but given the recent db stuff there I think it may deserve a bit more thought. Gerrit would be easy if it didn't imply a service outage. | 16:26 |
| fungi | for gitea, we could try upgrading one backend and then if it exhibits problems just take it out of the haproxy pool while we work through that. or we could try holding a test node and importing a recent backup mysqldump, then upgrading if we want a preliminary sniff test | 16:35 |
| fungi | actually, for partial production upgrade, we could take a backend down in haproxy first, upgrade it, test it, then put it back into the pool if everything seems okay, and exercise it for a while looking for errors before we upgrade the others | 16:35 |
| clarkb | fungi: ya I think testing on a copy of the db is what I'd like to do. My main concern is that the utf8 stuff seems to get stricter in newer mariadb versiosn and I'd like to ensure we understand if that affects anything in the upgrade (it really shouldn't only system tables are touched by the upgrade not application tables, but still) | 16:36 |
| fungi | makes sense, yep | 16:41 |
| clarkb | for some reason I thought today is tomorrow and that I'd have a bunch of meetings. | 16:44 |
| clarkb | I'm going to send an email to openstack-discuss about centos 7 cleanup. The vast majority of cleanup needs to be done in openstack-ansible, swift, devstack, freezer, keystone etc | 17:30 |
| fungi | sounds reasonable | 17:31 |
| clarkb | I can handle the zuul-jobs cleanup and the stuff within opendev and haev already pushed a change for starlingx. But not sure I can wait around for all of those to cleanup | 17:31 |
| clarkb | hoping an email gets things moving | 17:31 |
| fungi | i think we can just announce a date and then rip it out | 17:32 |
| clarkb | ya do we think March 15 is too early? | 17:32 |
| clarkb | I know it may impact the release aprticularly with the concurrent tripleo cleanup. But at the same time I would really like to get this done before we add any new images | 17:33 |
| clarkb | like ubuntu 24.04 | 17:33 |
| fungi | that's rc1 deadline for the openstack release, but i suppose it should only impact stable branches anyway? | 17:33 |
| clarkb | fungi: no all of the things I listed above are master using centos 7 | 17:34 |
| fungi | oh wow | 17:34 |
| clarkb | this is why I keep saying openstack need to clean stuff up at release time | 17:34 |
| clarkb | rather than at eol time | 17:34 |
| clarkb | fungi: ya the flip side of letting this go to RC time is that we'll have another branch that needs cleanup | 17:37 |
| clarkb | if we deal with it now then its less work overall | 17:37 |
| clarkb | but I suspect most people would like us to not impact the release at all then just ignore the problem | 17:37 |
| fungi | thankfully the number of job references for https://codesearch.opendev.org/?q=centos-7 seems fairly limited in the openstack/ namespace | 17:38 |
| clarkb | so ya an objections to saying we'll drop things March 15? | 17:41 |
| clarkb | noonedeadpunk: mnasiadka fyi ^ a coupel of openstack ansible repos show centos 7 still | 17:44 |
| clarkb | I know you indicated this wasn't an issue, but would be good to clean that up to avoid problems in the first place /me will send email shortly too | 17:44 |
| fungi | skimming the search results, it looks like most of the affected job definitions aren't in any pipelines, they stopped running them but missed cleaning up where they were defined? | 17:45 |
| clarkb | seems likely | 17:45 |
| fungi | i'm trying to put together a picture of which affected jobs are actually still running on master vs just unused cruft configuration | 17:46 |
| jrosser | for OSA //review.opendev.org/c/openstack/openstack-ansible-ops/+/910972 | 17:47 |
| jrosser | https://review.opendev.org/c/openstack/openstack-ansible-ops/+/910972 | 17:47 |
| clarkb | jrosser: ansible-plugin-container-connection is the other repo with centos-7. Also thanky ou | 17:47 |
| jrosser | thats not one i recognise immediately | 17:48 |
| clarkb | jrosser: in 910972 you deleted a - job: line which made zuul unhappy | 17:48 |
| clarkb | but otherwise that lgtm. It also cleans up other old stuff whcih is great | 17:49 |
| noonedeadpunk | openstack/openstack-ansible-ops is our crap | 17:49 |
| fungi | looks like some jobs are referenced in project-templates which are no longer applied in any projects, e.g. https://codesearch.opendev.org/?q=ansible-role-functional-jobs-centos7 | 17:49 |
| jrosser | i suspect ansible-plugin-container-connection is very dead | 17:49 |
| jrosser | it would be nice if we could search with a wildcard in the zuul errors page | 17:50 |
| jrosser | monorepo == easy there | 17:50 |
| jrosser | dozens of repos not so easy | 17:50 |
| noonedeadpunk | yeah, unfortunatelly zuul doesn't have parentproject concept | 17:51 |
| clarkb | jrosser: note they aren't errors yet | 17:51 |
| jrosser | no, but i did make a bunch of patches for errors last week | 17:51 |
| clarkb | ah gotcha | 17:51 |
| jrosser | https://review.opendev.org/q/topic:%22osa/zuul-errors%22 | 17:51 |
| jrosser | and i suspect that some intervention will be needed to merge the bulk of those | 17:52 |
| clarkb | spot checking it looks like typical bitrot in jobs. Could mark them non voting if you think there is interest in fixing them or just remove them if not | 17:53 |
| clarkb | I've pushed a change to delete a job in pbr for similar reasons this mornign | 17:53 |
| clarkb | ok I'm going to send the march 15 date and then see if there is much screaming. I suspect the impact is small enough we can make that happen | 17:53 |
| jrosser | actually ansible-plugin-container-connection looks like it in the Ansible SIG https://opendev.org/openstack/governance/src/branch/master/reference/sigs-repos.yaml#L5 | 17:55 |
| fungi | i guess the concern is what happens if, say, 2024.1 rc1 is tagged for a number of affected projects that week, and then we remove the configuration and don't notice that jobs have broken for them until we go to tag the final release a couple weeks later | 17:55 |
| clarkb | fungi: yes though maybe that is the kick we need to actually clean this up before we make releases | 17:56 |
| fungi | fair | 17:56 |
| clarkb | fungi: the major issue here is that we don't have a post or pre release task to go and cleanup all the old cruft we don't need or want anymore | 17:56 |
| clarkb | fungi: so we've ended up with centos 7 in master somehow | 17:56 |
| clarkb | when it should've been removed most places years ago | 17:56 |
| fungi | also zuul should call out the stable/2024.1 branch as having errors in those projects and we can identify and correct them before release day | 17:56 |
| clarkb | I'll make a note that this can impact the release and I'd welcome feedback | 17:58 |
| clarkb | at the same time we've got people wanting ubuntu 24.04 though | 17:58 |
| clarkb | and I really feel like this is a give and take situation where we clear room and then add new stuff | 17:58 |
| jrosser | from a deployment project POV we quite often stretch support for an OS for some cycles after the main projects move forward | 17:59 |
| jrosser | we did that fairly spectacularly with centos-7 and bionic | 17:59 |
| clarkb | jrosser: yup and we've tried to be accomodating in the CI system too | 17:59 |
| clarkb | centos 7 is 10 years old | 17:59 |
| jrosser | oh sure yes this was ages ago | 17:59 |
| clarkb | and will EOL very soon | 18:00 |
| fungi | and about to be eol by rh | 18:00 |
| fungi | looking at the search results, i think swift will be the hardest hit. they have 5 jobs they're currently running on master which they'll need to drop | 18:02 |
| jrosser | slurp releases have made things stretch more too | 18:02 |
| timburke | fungi, not for long we won't ;-) https://review.opendev.org/c/openstack/swift/+/910829 is approved; i'll work on some backports to clean out stable branches | 18:02 |
| clarkb | fungi: yup and timburke is already on it (I reached out to them first because they did seem to have the biggest usage) | 18:02 |
| fungi | swift-probetests-centos-7, swift-tox-py27-centos-7, swift-tox-func-py27-centos-7, swift-tox-func-encryption-py27-centos-7, swift-tox-func-ec-py27-centos-7 | 18:03 |
| clarkb | email sent | 18:04 |
| clarkb | thinking out loud here I suspect that we can drop centos-7 platform testing in zuul-jobs and stop wheel cache/mirror builds prior to that date | 18:07 |
| clarkb | but then wait fro teh 15th to merge the base-jobs nodeset removal and actual image and node cleanups in nodepool | 18:07 |
| fungi | yep. i can try pushing some removals to, like, devstack and openstack-zuul-jobs to see what errors they report, if nobody's done that yet | 18:10 |
| clarkb | fungi: I don't think anyone has. I'm starting with ozj and project-cnfig and will try to excise what looks safe | 18:10 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Call grub2-mkconfig with --update-bls-cmdline https://review.opendev.org/c/openstack/diskimage-builder/+/910977 | 18:11 |
| fungi | clarkb: is there a common review topic string you're using? | 18:11 |
| clarkb | fungi: 'drop-foo' is what I'e been doing so 'drop-centos-7' should work | 18:12 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:18 |
| clarkb | I believe ^ and https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/910979 should be fairly safe as long as zuul doesn't get made about removing anything there | 18:20 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:24 |
| opendevreview | Clark Boylan proposed openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 18:26 |
| fungi | https://review.opendev.org/c/openstack/devstack/+/910980 Drop the devstack-single-node-centos-7 nodeset | 18:28 |
| fungi | curious to see what errors zuul reports on that | 18:28 |
| clarkb | the keystone job likely uses that nodeset | 18:31 |
| clarkb | and maybe octavias too | 18:31 |
| fungi | i'm expecting mostly problems from stable branches | 18:31 |
| fungi | but there are master branch references in zuul configs for keystone, neutron-vpnaas, solum, and x/networking-opencontrail | 18:33 |
| fungi | which leaves me wondering, if there's nobody to approve a cleanup change in x/networking-opencontrail do we drop it from the tenant config so that the devstack cleanup change can merge? | 18:33 |
| fungi | oh, you know what, the devstack master branch cleanup isn't going to fail because zuul will just use the nodeset definition from one of devstack's many stable branches | 18:36 |
| fungi | is there a specific order they need to be cleaned up in? | 18:38 |
| fungi | (oldest first, latest first, doesn't matter) | 18:38 |
| fungi | looks like devstack currently has 7 other branches this needs to be cleaned up in | 18:40 |
| clarkb | I don't think order matters because the definitions all have to match otherwise it is an error? | 18:41 |
| clarkb | and yes removing networking-opencontrail from zuul is what I would do 970978 does this to x/collectd-openstack-plugins | 18:42 |
| fungi | i also just realized this intersects with the stable to unmaintained transition for wallaby and xena branches | 18:58 |
| fungi | so far only stable/yoga has been replaced by unmaintained/yoga, so if i propose nodeset cleanup changes to stable/wallaby or stable/xena they're going to get auto-closed when the unmaintained/wallaby and unmaintained/xena branches are created | 19:00 |
| fungi | and if we don't actually merge the nodeset cleanup across every open branch, we'll never see the errors from other projects referencing that nodeset | 19:00 |
| clarkb | yes I didn't udnersatnd why the oldest branches weren't udpated first | 19:01 |
| clarkb | for unmaintained I mean | 19:01 |
| fungi | elodilles: ^ perhaps you know? | 19:03 |
| clarkb | fungi: https://review.opendev.org/c/openstack/project-config/+/910978 should be good to go regardless of all the other changes to rip out centos 7 | 21:25 |
| clarkb | and with that in we can recheck the ozj change that won't pass zuul config checks until after x/collectd-openstack-plugins is out of zuul | 21:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade Refstack's MariaDB to 10.11 https://review.opendev.org/c/opendev/system-config/+/910999 | 21:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade Etherpad's MariaDB to 10.11 https://review.opendev.org/c/opendev/system-config/+/911000 | 21:30 |
| clarkb | CI should give us an idea if the database version change presents a compatibility problem (it won't test the upgrade itself) | 21:31 |
| fungi | thanks | 21:32 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add more info to afs fileserver recovery docs https://review.opendev.org/c/opendev/system-config/+/910662 | 21:48 |
| clarkb | and that addresses frickler's comment | 21:48 |
| opendevreview | Merged openstack/project-config master: CentOS 7 removal prep changes https://review.opendev.org/c/openstack/project-config/+/910978 | 21:49 |
| clarkb | reminder to get your team meeting agenda items onto the agenda today. I've got to do a school run in a few but then I'll be putting an agenda together when I get back | 22:03 |
| opendevreview | Merged opendev/system-config master: Add more info to afs fileserver recovery docs https://review.opendev.org/c/opendev/system-config/+/910662 | 22:07 |
| corvus | I'm going to perform a rolling restart of the entire zuul cluster to pick up a recently merged change which should fix a regression in executor job distribution. | 22:20 |
| corvus | i'm going to do that by running the same command we use for the weekly restart cron in a screen on bridge | 22:21 |
| corvus | #status log began graceful restart of zuul cluster | 22:23 |
| opendevstatus | corvus: finished logging | 22:23 |
| fungi | thanks corvus! | 22:36 |
| clarkb | fungi: shoudl we go ahead and approve teh git-review updates for the commit msg hook? | 23:03 |
| clarkb | then I can drop it from the meeting agenda (i'll probably do that either way as we got reviews) | 23:03 |
| fungi | i need to double-check whether i got all the comments addresses | 23:03 |
| fungi | addressed | 23:03 |
| clarkb | pretty sure you addressed my concerns | 23:04 |
| fungi | mm, yeah timburke had some suggestions on the chmod change i was going to address | 23:04 |
| timburke | speaking of git review, is there anything else i should do for https://review.opendev.org/c/opendev/git-review/+/907097 and https://review.opendev.org/c/opendev/git-review/+/907101 ? | 23:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade Etherpad's MariaDB to 10.11 https://review.opendev.org/c/opendev/system-config/+/911000 | 23:12 |
| clarkb | found a super minor incompatibility afterall. Just a test thing though | 23:12 |
| clarkb | timburke: I went ahead and approved the classifier one. On the importlib vs pkg_resources change maybe fungi can double check it for comaptibility and the napprove | 23:14 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Don't make hook script read-only https://review.opendev.org/c/opendev/git-review/+/910268 | 23:14 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Vendor a copy of Gerrit's commit-msg Git hook https://review.opendev.org/c/opendev/git-review/+/910275 | 23:14 |
| fungi | looks like we don't have a test of git review --version but i can whip one up real quick | 23:20 |
| clarkb | my agenda edits are in. I'll send that out in about half an hour | 23:26 |
| ianw | clarkb: speaking of afs; https://review.opendev.org/c/opendev/system-config/+/906013 is some fixups to the client role that i think are all useful | 23:26 |
| clarkb | oh yes having a clear "the kernel is too old" message would be helpful for debugging | 23:30 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Test that the --version option returns something https://review.opendev.org/c/opendev/git-review/+/911010 | 23:30 |
| opendevreview | Merged opendev/git-review master: Add classifiers for Python 3.10 and 3.11 https://review.opendev.org/c/opendev/git-review/+/907097 | 23:37 |
| opendevreview | Jeremy Stanley proposed opendev/git-review master: Test that the --version option returns something https://review.opendev.org/c/opendev/git-review/+/911010 | 23:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!