| bauzas | sorry folks, we're late for nova RC1 | 07:20 |
|---|---|---|
| bauzas | but we have the sdk issue | 07:20 |
| bauzas | I see that https://review.opendev.org/c/openstack/python-openstackclient/+/929236 is now merged but I don't see any release for OSC https://review.opendev.org/q/project:openstack/releases+file:%5E.*python-openstackclient.* | 07:26 |
| *** bauzas_ is now known as bauzas | 07:34 | |
| elodilles | bauzas: note that the patch has to land on stable/2024.2 | 07:34 |
| bauzas | because OSC already branched RC1 ? | 07:35 |
| bauzas | I don't see that in the above gerrit link | 07:35 |
| elodilles | not, RC1 (cycle-with-intermediery) but branched, yepp | 07:35 |
| bauzas | oh, wait, nvm, found it https://review.opendev.org/c/openstack/releases/+/928838/1/deliverables/dalmatian/python-openstackclient.yaml | 07:35 |
| bauzas | that's sad but okay | 07:36 |
| elodilles | bauzas: i've cherry picked the patch now: https://review.opendev.org/c/openstack/python-openstackclient/+/929402 | 07:37 |
| gthiemonge | Hi Folks, in Octavia we updated the requirements with "cryptography>=42.0.0" on 2024.2 and master (we replaced a deprecated function by a new one - introduced in 42), but one of the tested runtime (Debian 12) only includes cryptography 37 | 08:17 |
| gthiemonge | is there a way to fix that? can we revert the change in requirements.txt (on master and stable)? | 08:17 |
| zigo | cryptography 38 | 08:20 |
| zigo | I don't think it's a concern in the CI, as it's using pip. But for packaging, backporting cryptography is a real pain: it needs so many newer packages. | 08:21 |
| zigo | So I wrote this patch: | 08:22 |
| zigo | https://salsa.debian.org/openstack-team/services/octavia/-/blob/debian/dalmatian/debian/patches/compat-python3-cryptography-bookworm.patch?ref_type=heads | 08:22 |
| zigo | I'm not sure when not_valid_after_utc() was introduced, but for me, checking on version 42 is enough. | 08:22 |
| gthiemonge | yeah we can do something similar in the octavia code, but we would still have this cryptography>=42 in the reqs | 08:29 |
| zigo | IMO, just don't bother ... :P | 08:45 |
| ttx | gthiemonge: we can probably still revert... when was that introduced? | 08:49 |
| gthiemonge | https://review.opendev.org/c/openstack/octavia/+/921752 | 08:51 |
| ttx | release-team: proceeding to approve release-announce change... please hold on any release approvals | 08:51 |
| gthiemonge | in June, so it's already in 2024.2 | 08:51 |
| frickler | ttx: ack | 08:51 |
| opendevreview | Merged openstack/releases master: Remove description to simplify release-announce https://review.opendev.org/c/openstack/releases/+/929126 | 09:04 |
| zigo | ttx: Oh, I think it's a shame to remove the description from the announce. Many times, I discovered new stuff reading the announce list... :/ | 09:05 |
| zigo | ttx: One very easy way to get the short desk is to wget the pypi .xml file. | 09:05 |
| zigo | s/.xml/.json/ | 09:06 |
| zigo | SHORT_DESC=$(cat ${PKG_NAME}.json | jq --raw-output '. | .info.summary') | 09:06 |
| ttx | yeah.. although that would create a dependency on having PyPI responding, which would create another SPOF | 09:09 |
| ttx | zigo: great to see someone is still reading those, though :) | 09:09 |
| zigo | :) | 09:10 |
| zigo | I guess I'll follow links when I see something new then... | 09:10 |
| ttx | The "proper" fix would be to include the description in the tag metadata | 09:13 |
| ttx | but that would be a big fix this late in the release cycle | 09:13 |
| zigo | No big deal ... | 09:14 |
| zigo | I'd prefer things to be released on time. Once more that's painful to see things like clients being re-released so late ... | 09:15 |
| ttx | agreed | 09:15 |
| opendevreview | Merged openstack/releases master: Releasing release-test https://review.opendev.org/c/openstack/releases/+/929284 | 09:16 |
| opendevreview | Thierry Carrez proposed openstack/releases master: stable/2024.2 branches for Ironic deliverables https://review.opendev.org/c/openstack/releases/+/929409 | 09:31 |
| ttx | OK the change seems to be working... slowly proceeding to approve the RC1s that were not explictly -1ed | 09:50 |
| opendevreview | Merged openstack/releases master: Release watcher RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928571 | 10:03 |
| opendevreview | Merged openstack/releases master: Release watcher-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928570 | 10:06 |
| opendevreview | Merged openstack/releases master: Release zun-ui RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928574 | 10:38 |
| opendevreview | Merged openstack/releases master: Release trove RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928565 | 10:45 |
| opendevreview | Merged openstack/releases master: Release mistral-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928540 | 10:45 |
| opendevreview | Merged openstack/releases master: Release masakari-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928537 | 10:45 |
| opendevreview | Merged openstack/releases master: Release masakari-monitors RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928538 | 10:46 |
| opendevreview | Merged openstack/releases master: Release masakari RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928539 | 10:46 |
| opendevreview | Merged openstack/releases master: Release trove-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928564 | 10:47 |
| opendevreview | Merged openstack/releases master: Release magnum RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928534 | 10:50 |
| opendevreview | Merged openstack/releases master: Release magnum-ui RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928533 | 10:50 |
| opendevreview | Stephen Finucane proposed openstack/releases master: python-openstackclient 7.1.1 https://review.opendev.org/c/openstack/releases/+/929454 | 11:32 |
| opendevreview | Stephen Finucane proposed openstack/releases master: python-openstackclient 7.1.1 https://review.opendev.org/c/openstack/releases/+/929454 | 11:56 |
| ttx | Need a second +2 (and w+1) on https://review.opendev.org/c/openstack/releases/+/928519 and https://review.opendev.org/c/openstack/releases/+/928541 | 12:00 |
| opendevreview | Merged openstack/releases master: Release Horizon 25.1.0 for Dalmatian https://review.opendev.org/c/openstack/releases/+/928847 | 12:04 |
| opendevreview | Merged openstack/releases master: Release cloudkitty RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928521 | 12:04 |
| opendevreview | Merged openstack/releases master: Release cloudkitty-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928520 | 12:04 |
| opendevreview | Merged openstack/releases master: Release ceilometer RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928518 | 12:05 |
| opendevreview | Merged openstack/releases master: Release adjutant-ui RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928508 | 12:05 |
| opendevreview | Merged openstack/releases master: Release aodh RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928513 | 12:05 |
| opendevreview | Merged openstack/releases master: Release tacker RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928562 | 12:05 |
| elodilles | ttx: +2+W'd (sorry, i'm a bit distracted with downstream meetings o:)) | 12:11 |
| ttx | thanks! | 12:14 |
| opendevreview | Merged openstack/releases master: Release manila-ui RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928535 | 12:16 |
| opendevreview | Merged openstack/releases master: Release designate-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928524 | 12:16 |
| opendevreview | Merged openstack/releases master: Release designate RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928525 | 12:16 |
| opendevreview | Merged openstack/releases master: Release vitrage-dashboard RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928568 | 12:18 |
| opendevreview | Merged openstack/releases master: Release vitrage RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928569 | 12:18 |
| opendevreview | Merged openstack/releases master: Release cinder RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928519 | 12:18 |
| opendevreview | Merged openstack/releases master: Release mistral RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928541 | 12:24 |
| frickler | hmm, placement switched back from storyboard to launchpad, but that change didn't make it into the deliverable file and thus the announcement is a bit wrong | 12:46 |
| frickler | seems the same holds for a couple of other repos, at least octavia* and osc/sdk | 12:48 |
| frickler | I don't have time this week to check all of them, but might be good to ensure those get fixed before stuff gets copied for 2025.1 | 12:49 |
| opendevreview | Merged openstack/releases master: python-openstackclient 7.1.1 https://review.opendev.org/c/openstack/releases/+/929454 | 13:16 |
| *** ykarel_ is now known as ykarel | 14:01 | |
| opendevreview | Elod Illes proposed openstack/releases master: Release barbican-tempest-plugin for 2024.1 Caracal https://review.opendev.org/c/openstack/releases/+/929511 | 14:35 |
| opendevreview | Elod Illes proposed openstack/releases master: Release blazar-tempest-plugin for 2024.1 Caracal https://review.opendev.org/c/openstack/releases/+/929512 | 14:36 |
| opendevreview | Elod Illes proposed openstack/releases master: Release barbican-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929511 | 14:37 |
| opendevreview | Elod Illes proposed openstack/releases master: Release blazar-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929512 | 14:37 |
| opendevreview | Elod Illes proposed openstack/releases master: Release cinder-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929515 | 14:40 |
| opendevreview | Elod Illes proposed openstack/releases master: Release cyborg-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929518 | 14:42 |
| opendevreview | Elod Illes proposed openstack/releases master: Release designate-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929520 | 14:45 |
| opendevreview | Elod Illes proposed openstack/releases master: Release glance-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929522 | 14:46 |
| opendevreview | Elod Illes proposed openstack/releases master: Release heat-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929523 | 14:47 |
| opendevreview | Elod Illes proposed openstack/releases master: Release ironic-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929526 | 14:52 |
| opendevreview | Elod Illes proposed openstack/releases master: Release keystone-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929528 | 14:55 |
| opendevreview | Elod Illes proposed openstack/releases master: Release kuryr-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929530 | 14:57 |
| opendevreview | Elod Illes proposed openstack/releases master: Release magnum-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929531 | 15:00 |
| opendevreview | Elod Illes proposed openstack/releases master: Release manila-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929532 | 15:01 |
| opendevreview | Elod Illes proposed openstack/releases master: Release mistral-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929534 | 15:02 |
| opendevreview | Elod Illes proposed openstack/releases master: Release neutron-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929536 | 15:05 |
| opendevreview | Elod Illes proposed openstack/releases master: Release octavia-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929537 | 15:06 |
| *** bauzas_ is now known as bauzas | 15:17 | |
| opendevreview | Elod Illes proposed openstack/releases master: Release telemetry-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929541 | 15:21 |
| sean-k-mooney | hi on an internal call but we need to discuss https://review.opendev.org/c/openstack/requirements/+/929504 | 15:26 |
| sean-k-mooney | we either need to proceed with that or revert to <7.0.0 opentack clinet | 15:27 |
| opendevreview | Elod Illes proposed openstack/releases master: Release trove-tempest-plugin for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/929544 | 15:28 |
| dansmith | How should I express my preference for going back to <7.x? | 15:28 |
| sean-k-mooney | :) ^ is effective | 15:29 |
| dansmith | I'm pretty concerned about bumping at this super late date given that we've seen little testing _and_ had a pretty significant regression | 15:29 |
| sean-k-mooney | or just -1 the gerrit reveiw i guess | 15:29 |
| dansmith | and it's blocking something we *have* to get into rc1, which we're already late for | 15:29 |
| bauzas | hah, just asked the release team in -nova | 15:30 |
| dansmith | I -1'd the patch as well for visibility | 15:31 |
| elodilles | thanks for the heads up. i thought that release would unblock nova RC1 :'( | 15:33 |
| bauzas | it could, but we don't know and that's the problem dansmith is raising | 15:34 |
| bauzas | ideally, I'd prefer 2024.2 to not depend on OSC 7.x while master could, if we want | 15:35 |
| dansmith | elodilles: it might but we are not positive, and it will make nova start using a bunch of new code that we haven't been testing with all cycle, and in which we already found one major regression | 15:35 |
| dansmith | elodilles: so I think given how very late it is, we should take the most conservative route at this point | 15:35 |
| elodilles | the problem is that we already have 7.0.0, 7.1.0 and 7.1.1 release in Dalmatian (and they are listed in our release page, announced, etc) | 15:37 |
| elodilles | but yeah, i have some deja vu | 15:37 |
| elodilles | :( | 15:37 |
| elodilles | at least not an oslo project this time. | 15:37 |
| elodilles | :( | 15:37 |
| elodilles | (7.0.0 was release in early August) | 15:38 |
| dansmith | were we testing with 7.0 then? | 15:39 |
| dansmith | I thought it was broken in all of 7.x | 15:40 |
| elodilles | my question would be that how dangerous is that we keep python-openstackclient 7.1.1 release and do bugfixes if any issue comes to light | 15:40 |
| elodilles | dansmith: well, let me check when was u-c bumped for 7.x | 15:40 |
| elodilles | 7.0 | 15:40 |
| dansmith | seems like it wasn't, from my quick github history browsing? | 15:42 |
| sean-k-mooney | 7.0.0 never got in but 7.1.0 was bumpped about a week ago | 15:42 |
| dansmith | right | 15:42 |
| elodilles | yepp, you are right | 15:43 |
| elodilles | sad :/ | 15:43 |
| elodilles | releaste-team: please read ^^^ | 15:43 |
| dansmith | elodilles: definitely sucks, but you see my point (hopefully) that this is a major bump very late into rc1 | 15:43 |
| bauzas | to clarify, we never tested OSC7.x until last week, right? | 15:45 |
| dansmith | right | 15:45 |
| bauzas | that's when the gate went AWOL | 15:45 |
| bauzas | then | 15:45 |
| bauzas | due to the password param | 15:45 |
| dansmith | right | 15:45 |
| elodilles | well, the crossjobs tested, and some were broken, some projects needed some fix until all jobs passed. so the question is how well the cross-jobs cover things | 15:47 |
| elodilles | have you pinged SDK folks? | 15:48 |
| dansmith | elodilles: not well enough for me | 15:48 |
| elodilles | i guess stephenfin would fight the opposite thing: to have it released o:) | 15:48 |
| bauzas | I'm not fighting OSC to be released | 15:50 |
| bauzas | I'm fighting late bumps that are after FF and prevent us merging | 15:50 |
| elodilles | note, that we did in the past, to revert all the breaking changes, do another MAJOR version bump, and re-release the given deliverable, so yes, that is an option, if the teams agree that that is the only way-forward :/ | 15:50 |
| dansmith | I don't understand why we can't allow it to be released but keep u-c back on <7 | 15:51 |
| elodilles | bauzas: you know that relmgt team also fight to *avoid* late lib releases o:) | 15:51 |
| melwitt | was there not a requirements freeze weeks ago? https://releases.openstack.org/dalmatian/schedule.html#d-rf that's what I don't understand about this | 15:51 |
| dansmith | melwitt: yeah, that's confusing for me as well.. seems like this should not be happening | 15:51 |
| elodilles | melwitt: yes, there was, but as we understood, python-openstackclient was needed for nova RC1, so it was a release blocker thing | 15:53 |
| dansmith | elodilles: only because it was bumped and broke us, AFAIK | 15:54 |
| elodilles | dansmith: i see. well, as i see, what happened is similar like we had at a couple of cycles with oslo libs: there was a release, but the cross-jobs caught some issue, so the version is bumped only after all the jobs passed (which means things were fixed), but then it was late in the cycle and other things got broken, that were not covered with the crossjobs | 16:02 |
| elodilles | ( https://review.opendev.org/c/openstack/requirements/+/925763 ) | 16:02 |
| dansmith | right, I don't understand why osc wouldn't be held to the same requirement as oslo libs, | 16:02 |
| dansmith | where if we're not testing against osc master all cycle, there needs to be an earlier-than-rc1 deadline for when we release and cut over | 16:03 |
| elodilles | hence we agreed that oslo libraries should have early FF. (but not too early, because that would mean the team would not have time for improvements). | 16:03 |
| dansmith | yep | 16:03 |
| elodilles | so it seems we have to extend that early FF to python-openstackclient as well. (and i don't think SDK team / stephenfin will be happy about that) | 16:05 |
| dansmith | I don't see how it can reasonably work any other way though | 16:06 |
| melwitt | what is the reason it is not in the same boat as oslo libs? | 16:06 |
| ttx | I think it would be fine to have it early frozen in future cycles... won;t save us for this time though | 16:07 |
| bauzas | another problem is that upper-constraints doesn't catch the regression | 16:07 |
| bauzas | it was merged and then broke the nova gate | 16:08 |
| bauzas | that's also why we're asking some earlier FF for OSC | 16:08 |
| ttx | I think the rationale was that we allow python-*client changes later as the main component may land a feature that requires a client bump... and python-openstackclient inherited that | 16:08 |
| dansmith | so again, I really don't understand why we can't allow this to be released (it already is), just keep the version we depend on in u-c pinned to <7.. people can use the newer version on their workstations for clients, but just keep it pinned for the projects at this point | 16:08 |
| bauzas | if somehow the gate is able to test OSC preemptively without breaking, then that's another problem | 16:08 |
| ttx | dansmith: that might work... | 16:09 |
| dansmith | that seems to be the easiest and most straightforward way out of the current box to me.. it doesn't require a wholesale dumping of the work done in 7.x, any user client stuff can still be leveraged | 16:10 |
| dansmith | and then going forward, early FF for it next cycle | 16:10 |
| dansmith | IMHO, we can merge the blacklist right now so we can get rc1 out, then immediately bump master to 7.x and roll from there | 16:10 |
| ttx | It's just "weird" to have a u-c <7 pin in dalmatian deliverables when dalmatian OSC is all >7 | 16:11 |
| ttx | but maybe not any weirder than alternate solutions | 16:11 |
| ttx | It might make distro work a bit difficult? | 16:12 |
| elodilles | dansmith: in the past we asked the revert+release because it's confusing that the releases are listed under openstack's releases page under '2024.2 Dalmatian', but not marked in any way that is not used (despite u-c is not bumped). | 16:12 |
| dansmith | probably, although they can just not package osc7.x pretty easy | 16:12 |
| ttx | like which version of OSC are they going to package in their 2024.2 dalmatian release | 16:12 |
| dansmith | container-based distros will be fine though | 16:12 |
| dansmith | elodilles: I understand, I'm just saying that we've *already* had a major regression from this lineage alone and we're in the week past rc1 and still have critical fixes to merge to nova | 16:13 |
| ttx | So... option 1 would be to release a 7.2 that is just the latest 6.x rereleased. Option 2 would be to just pin it to <7 in u-c... Option 3 would be to run forward and hope 7.1 just works for everyone? | 16:14 |
| dansmith | 7.1.1, but yeah I think those are the options | 16:16 |
| elodilles | (and 8.0 for option 1, but yes o:)) | 16:16 |
| bauzas | you lost me with option 1 | 16:16 |
| dansmith | #2 is the weirdest but the least work/churn, #1 seems like a ton of work, and #3 is the least responsible/most aggressive I guess | 16:17 |
| JayF | option 1 is a roll-back-by-rolling-forward: you just put the old code in a new release so the versions make sense if you don't look at the code | 16:17 |
| elodilles | (as we will be backward incompatible with 7.x) | 16:17 |
| ttx | bauzas: what jayF said | 16:17 |
| bauzas | I'd personnally consider that 7.1.1 would become then 8.x | 16:17 |
| dansmith | if #3 works then obviously it's the right choice, but you don't know until the plane crashes and we've already had one engine flameout :) | 16:18 |
| bauzas | if we go with 7.2 branding old 6.x | 16:18 |
| ttx | I think option 2 does not preclude us from trying option 3 | 16:18 |
| sean-k-mooney | this is one of the options https://review.opendev.org/c/openstack/requirements/+/929552 | 16:19 |
| ttx | We could release 7.1.1 and see, and be ready to pin to <7 if that still looks like a can of crabs | 16:19 |
| dansmith | ttx: you're assuming that one passing test run means it's good? | 16:19 |
| elodilles | dansmith: again, the question is, what coverage the cross-jobs gives us on the upper-constraints bumping patches. (we see that not 100% as nova gate broke after the u-c bump, but i guess we never thought it as 100% coverage) | 16:20 |
| sean-k-mooney | that blocks 7.0.0 and 7.1.1 which are know to be broken but allows a future bump to 7.1.1 | 16:20 |
| sean-k-mooney | *blocks 7.0.0 and 7.1.0 | 16:20 |
| dansmith | elodilles: AFAIK, no multi-node, no live migration, etc right? | 16:20 |
| ttx | dansmith: no, one passing test means it's already better than 7.1.0 :) | 16:21 |
| sean-k-mooney | elodilles: it broke because evacuate is not tested in the cross gating | 16:21 |
| dansmith | ttx: ack, well, you know my concern there I guess :) | 16:21 |
| dansmith | ttx: given it will be the first run ever to pass, instead of a couple weeks of trial-by-fire | 16:21 |
| dansmith | I have to step away for a bit, but I think I've made all my arguments here. if we decide to just damn the torpedoes then so be it, but it seems like not the best plan to me | 16:22 |
| ttx | what would give us trust in a 7.1.1 release? If nothing, I agree that revert or pinning are the only options. | 16:23 |
| sean-k-mooney | we already have a 7.1.1 release by the way | 16:24 |
| sean-k-mooney | we just didnt wnat spend time creatign a ci run to fully test hat with all our gate jobs | 16:24 |
| sean-k-mooney | i tried geting pre merge converage for the requession fix but the base devstack job does not pull in openstack client form souce | 16:25 |
| stephenfin | just to clarify, there was a late bump because the tooling did not pick up OSC to do an automatic release back when we froze. I discussed this with elodilles last week. https://meetings.opendev.org/irclogs/%23openstack-release/%23openstack-release.2024-09-10.log.html | 16:25 |
| sean-k-mooney | so we did not get the fix | 16:25 |
| stephenfin | We respected freeze and did not merge during code freeze period | 16:25 |
| ttx | yeah not sure an earlier freeze would have saved us | 16:26 |
| sean-k-mooney | technially the UC bump was merged during the feeze perod | 16:26 |
| sean-k-mooney | for 7.1.0 | 16:26 |
| sean-k-mooney | that what stopped nova doing or rc1 since we coudl not merge the final reuqired patches | 16:26 |
| stephenfin | sean-k-mooney: read the chat, please | 16:26 |
| stephenfin | we had to cut it late because the tooling missed OSC and did not cut a release earlier | 16:27 |
| elodilles | stephenfin: note that 7.0 u-c bump never happened because of failing cross-jobs. so the 'tooling not picked up' thing can be de-coupled from this problem. that is another issue. | 16:27 |
| sean-k-mooney | yes | 16:27 |
| sean-k-mooney | and im saying we should not have megered https://review.opendev.org/c/openstack/requirements/+/928948 | 16:27 |
| ttx | I feel like we can give 7.1.1 a couple of days of trial by fire before pushing a u-c pin to bypass it? | 16:27 |
| sean-k-mooney | untill all project had rc1 rleased | 16:28 |
| stephenfin | elodilles: Oh, I didn't know. Pity I hadn't seen that or I'd have fixed the issue weeks ago :( | 16:28 |
| sean-k-mooney | part of the issue here is master on the requiremets repo is in use for both stable/2024.2 and master | 16:29 |
| elodilles | ttx: is that an option to ask teams to test their gates with python-openstackclient 7.1.1 and see the gates state? | 16:30 |
| bauzas | the root problem is that (and I do understand why people care about it) OSC7.x is quite a major change | 16:31 |
| bauzas | IIUC, we now start fully using the SDK, right? | 16:31 |
| * bauzas needs to look at the relnotes | 16:31 | |
| sean-k-mooney | elodilles: im not personally agaisnt using 7.1.1 for stable/2024.2 but if we do that we need to get it in the gate today | 16:31 |
| sean-k-mooney | and we need to be ready to role back really quickly | 16:32 |
| elodilles | sean-k-mooney: i agree. | 16:32 |
| ttx | sounds reasonable | 16:32 |
| sean-k-mooney | https://review.opendev.org/c/openstack/requirements/+/929552 does not prevent use upgrading ti 7.1.1 | 16:32 |
| ttx | In all cases, a full-revert or a uc-pin are going to be ugly, so probably not the release-team decision to make... I'd rather make sure teh TC is in the loop | 16:33 |
| ttx | are all components affected or just nova and a few friends? | 16:34 |
| fungi | but also, the tc is in its ~last week of the term, so may be hesitant to make decisions with new tc members joining/old tc members leaving next week | 16:34 |
| bauzas | ttx: that's not that easy to know | 16:34 |
| bauzas | nova is impacted for sure, since its gate is broken | 16:34 |
| bauzas | but that doesn't prevent other projects to be impacted too | 16:35 |
| bauzas | if they just don't call test the client, they won't notice it | 16:35 |
| elodilles | sean-k-mooney: if we see that things are broken and python-openstackclient needs to be reset to <7.0 state, then we either should revert all critical changes in python-openstackclient and re-release ASAP ( stephenfin / SDK team can say how feasible this is; this is option #1) or we have to bump u-c back to 6.x, which is quite easy to do, but could result confusion for packagers, consumers | 16:35 |
| ttx | Do we still have translations merges between RC1 and RC2? could be a good way to test a bunch of gates | 16:35 |
| elodilles | (option #2) | 16:35 |
| sean-k-mooney | ttx: i think this is just nova currently but other project might have post-run playbooks that use osc | 16:35 |
| stephenfin | elodilles: Borderline impossible, I'm afraid. There were 88 changes between 6.6.0 and 7.0.0. Fortunately, nova is the only project likely to be affected since the bulk of the changes in 7.0.0 were novaclient -> SDK transition | 16:37 |
| elodilles | ttx: i guess those translation jobs don't trigger all jobs (due to 'irrelevant-files' settings in .zuul.yaml, but only guessing) | 16:38 |
| elodilles | s/translation jobs/translation patches/ | 16:38 |
| ttx | OK so how about we give 7.1.1 a chance, but be ready to step on the uc-pin nuclear button if after a few days we still have doubts that it's going to work... and in the meantime we inform the TC of our dire options | 16:39 |
| elodilles | stephenfin: i see, then it's a bigger number of changes than we had with oslo.* in the past :-/ | 16:39 |
| stephenfin | I don't understand the panic in this. We're not talking about CVEs or anything here. In addition, OSC is not a direct dependency of Nova and we're seeing these issues as a side-effect of OSC usage in jobs. Once the jobs are passing again, aren't we...done? | 16:39 |
| dansmith | what are the user-visible changes in OSC 7.x? Meaning, if it's all internal cleanup, that's high risk for regressions, low benefit for users, such that a uc freeze at <7 would not be the end of the world | 16:39 |
| elodilles | so option #1 is not feasible. we only have option #2 if gates are failing | 16:40 |
| bauzas | stephenfin: the problem is that nova RC1 is still on hold | 16:41 |
| stephenfin | elodilles: Agreed. And the alternative to reverting oslo.db was significantly more involved (basically projects had to drop all SQLAlchemy 2.x-incompatible calls immediately). Versus here, where the fixes are exclusively contained in OSC | 16:42 |
| stephenfin | bauzas: It wouldn't be if you'd merged the u-c bump 3 hours ago 🤷♂️ | 16:42 |
| bauzas | we could release Dalmatian with 7.1.1 but we're afraid of any potential regression we may have missed (particularly when saying that most of the 66 patches were about novaclient > SDK) | 16:42 |
| stephenfin | if the gate jobs are passing then as far as nova is concerned, the release is good enough. There have been bugs in OSC before. There will be bugs in the future. The issue here is that the gate is failing and that's what we need to fix | 16:43 |
| sean-k-mooney | so on the nova side we coudl propose dnms to use the two propsoed uc patches | 16:44 |
| sean-k-mooney | bump and revert | 16:44 |
| sean-k-mooney | but its nearing end of day and we wanted to do the rc release today | 16:44 |
| sean-k-mooney | which requries 4 patches to merged then the release patch | 16:45 |
| stephenfin | or just merge the u-c bump? it's a single new change that will at least address this issue | 16:45 |
| sean-k-mooney | at this point RC1 will not happen today anyway so i dont mind wich of the two option we take as long as we dont dely the desiosson any longer | 16:45 |
| stephenfin | dansmith: There are quite a few things there. Most of them should be in our release notes https://docs.openstack.org/releasenotes/python-openstackclient/2024.2.html | 16:50 |
| stephenfin | (And yes, looking at that, some of those "New features" should really be in the "upgrade notes" section. One to pay closer attention to in future reviews 🤓) | 16:51 |
| dansmith | yep, I went looking.. there are a few new user-visible things there, but seems like the vast majority are internal things a user would not notice | 16:52 |
| dansmith | I'm still fuzzy on how we got here. This was supposed to be released and u-c'd weeks ago such that we'd have the requisite burn time, but it just didn't because of some technicality and now we're bumping in release week anyway? | 16:53 |
| frickler | the 7.0.0 release was made 6 weeks ago and failed https://review.opendev.org/c/openstack/requirements/+/925763 all that happened afterwards was bug fixing | 17:01 |
| frickler | having more help for bug fixing in the sdk/osc team likely would be helpful. adding some tips testing for nova+neutron on osc might also help | 17:03 |
| *** bauzas_ is now known as bauzas | 19:31 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!