| *** bauzas- is now known as bauzas | 07:22 | |
| zigo | When installing Dalmatian, I have (yet) another regression with openstackclient: | 08:17 |
|---|---|---|
| zigo | https://paste.opendev.org/show/bSKLTvF36ChOGBfUgVhw/ | 08:17 |
| zigo | Somehow, it fails loading the metric module with stevedore. | 08:17 |
| zigo | Reverting to openstackclient 6.6.0 fixed everything. | 08:17 |
| ttx | stephenfin: see above ^ | 08:46 |
| elodilles | bauzas also said on nova channel: 10:14 < bauzas> so, I'd prefer to rollback the u-c by OSC6 now | 08:54 |
| zigo | That's what I did in my CI, though it'd be nicer to understand what's going on. | 08:59 |
| frickler | zigo: can you check whether the "metric delete" command actually is present for you in 6.6.0? I think what's new in 7.x is only the logging for the import failure. otherwise that's a bug in https://opendev.org/openstack/python-observabilityclient/src/branch/master/observabilityclient/v1/cli.py | 08:59 |
| zigo | What is observabilityclient btw? I packaged it, but I don't get it. | 09:00 |
| frickler | something invented by the telemetry project afaict, I don't remember the details | 09:01 |
| zigo | frickler: Sorry, can't really check easily, my CI seems broken somehow, I have to repair it. | 09:01 |
| zigo | Oh, just got what's happening ... :P | 09:02 |
| zigo | One VM staid up after clean-up, looks like. | 09:02 |
| zigo | I'll be able to tell soon. | 09:02 |
| zigo | oh, silly me, it's easy to check, | 09:03 |
| zigo | Carcal is what I have in production, hang on ! :) | 09:03 |
| zigo | Ah no, Caracal is openstackclient 5.4.0 ... | 09:04 |
| zigo | FAIL again ... :P | 09:04 |
| zigo | frickler: | 09:04 |
| zigo | # openstack metric delete | 09:04 |
| zigo | usage: openstack metric delete [-h] [--resource-id RESOURCE_ID] metric [metric ...] | 09:04 |
| zigo | openstack metric delete: error: the following arguments are required: metric | 09:04 |
| zigo | That's with python3-openstackclient 6.6.0 | 09:05 |
| zigo | python3-gnocchiclient being installed. | 09:05 |
| frickler | oh, wait, that's with gnocchiclient, not observabilityclient? the former isn't even openstack | 09:06 |
| frickler | I only checked with codesearch, but couldn't find the "HOME" reference | 09:06 |
| *** bauzas_ is now known as bauzas | 09:06 | |
| zigo | frickler: gnocchiclient is indeed what brings the "metric" sub-command. | 09:07 |
| zigo | That's indeed from telemetry, and not from OpenStack, thought there's no other ways, is there? | 09:07 |
| zigo | The fact that gnocchi and gnocchiclient are not maintained within OpenStack is IMO a **very bad** thing. | 09:08 |
| zigo | The only reason Red Hat doesn't care, is because Red Hat (wrongly) believe it's not needed and that everything should go to prometheus (which cannot handle the load of a moderately large public cloud...). | 09:09 |
| frickler | well iirc that was the decision from the gnocchi team, not from openstack | 09:11 |
| frickler | also while we're getting ever more offtopic: what backend do you use with gnocchi for scale? I remember using ceph a long time ago and that didn't scale well either | 09:13 |
| zigo | We use Ceph indeed. | 09:13 |
| zigo | We have about 20 / 25 metrics per VM, and like 6k VMs, and it's ok-ish ... | 09:13 |
| zigo | We have bottlenecks elsewhere. | 09:14 |
| zigo | Like so many gnocchi-api calls ... | 09:14 |
| zigo | Influxdb is nice and fast (500k timeseries point per second), but it's license isn't allowing clustering unless you pay, so we don't use it. | 09:15 |
| *** bauzas_ is now known as bauzas | 09:16 | |
| frickler | ok, interesting, thx for the data point | 09:22 |
| gibi | are we going to merge https://review.opendev.org/c/openstack/requirements/+/929552 now as osc 7.1.1. is still broken, or does the release team still prefers to roll forward to 7.1.2. with https://review.opendev.org/c/openstack/python-openstackclient/+/929726 ? | 09:39 |
| gibi | to be fair I preferred rolling to 7.1.1. but now that it turned out to be broken too. I have to say that dansmith was right and we should not rush these things | 09:41 |
| sean-k-mooney | we can do both | 09:47 |
| sean-k-mooney | if we merge my patch first it will unblock nova and we can try an test 7.1.2 with a dnm form nova to the requirements patch i think | 09:48 |
| sean-k-mooney | i need to confrim the rquiremetns repo is in requred porjects | 09:48 |
| ttx | I'd like to make the u-c pin decision by end of week, for sure. In the meantime we should definitely roll forward | 09:52 |
| ttx | but agree things are trending in the pin direction right now | 09:52 |
| sean-k-mooney | we really do not have that kind of time | 09:52 |
| sean-k-mooney | i mean the final rc is ment to be in 8 days | 09:53 |
| bauzas | fwiw we also have another bug report with evacuate once we use my OSC change for providing the right parameter https://bugs.launchpad.net/nova/+bug/2081023 | 09:54 |
| gibi | yeah we have RC bugs waiting to land before we can cut RC | 09:54 |
| ttx | yeah | 09:54 |
| bauzas | that's why I'd prefer to pin up to OSC6 | 09:54 |
| bauzas | at least for Dalmatian | 09:54 |
| bauzas | and then once we have nova RC1, we could just accept OSC7.1.2 (once it's created) | 09:54 |
| ttx | I'd like to hear from stephenfin first, see if he agrees at this stage it looks like the best compromise... but yes ideally we'd make the call today | 09:57 |
| sean-k-mooney | i can try and create some patches to devstack and nova to try an pull in the different things in flight for 7.1.2 but i would perfer to proceeed with https://review.opendev.org/c/openstack/requirements/+/929552 and see if we can land the rc regresions in nova in the next hour or so | 09:57 |
| ttx | Ideally we'd also have the TC weigh on this because a dalmatian pin that excludes all dalmatian releases is going to look bad... but I recognize it might be hard to get | 09:58 |
| sean-k-mooney | we also neeed them to weigh in on possible delaying the integrated relese | 09:59 |
| sean-k-mooney | because we are getting close to needing to consider that | 09:59 |
| sean-k-mooney | once the osc issue is resolved im expecting it to take a day or 2 for our patches to actully land before nova can cut rc1 | 10:00 |
| zigo | I'm all for 6.6.0, though I'd prefer if openstackclient 7.x was released with reverted patches instead, so we can continue in a monotonic increase. | 10:00 |
| zigo | Just like you wrote ttx ... | 10:01 |
| sean-k-mooney | zigo: we dont really know what other bugs there are | 10:01 |
| sean-k-mooney | so we dont realy know what the revert would be | 10:01 |
| zigo | git diff -u -r 6.6.0 0 -r 7.1.1 >1.patch ; patch -p1 -R <1.patch | 10:01 |
| zigo | :) | 10:02 |
| sean-k-mooney | reverting everything and asking the sdk team to do all that owrk again is not really an option unless you just ment on stable not master | 10:02 |
| sean-k-mooney | at which point its better to just use 6.6.0 | 10:03 |
| zigo | Yeah, just on stable ... so we can have a 7.2 that works. | 10:03 |
| sean-k-mooney | well we are not really ment ot bump uc on stable once a release happens | 10:04 |
| zigo | I only uploaded 7.x to Experimental, so I don't really care, leaving it bitrot there until there's something better is ok for me. | 10:04 |
| sean-k-mooney | so it 7.2 woudl not be pulled into stable | 10:04 |
| sean-k-mooney | so im not seeing the point of a revert fo all the code just on stable | 10:04 |
| zigo | The point was only for having an always increasing version, if someone downstream already released 7.x. | 10:04 |
| zigo | (not my case, as I just wrote) | 10:05 |
| bauzas | that's why I'd prefer to pin to 6.x | 10:14 |
| bauzas | I know this is hard for the OSC team but they can document that people can use OSC7 if they wish, only if they don't need to evacuate | 10:14 |
| bauzas | that's what releasenotes are made for | 10:14 |
| bauzas | and distros can choose to ship OSC7 with Dalmatian once 7.1.2 is there | 10:15 |
| bauzas | we're just talking of the integrated release | 10:15 |
| zigo | Right, though as I wrote, that's not the only bug. I have this issue where "user show" didn't work. Hard to tell in which case, though I'm having this with my keystone_user provider resource ... | 10:18 |
| zigo | I'll try and reproduce, but I'm not sure how (yet). | 10:18 |
| sean-k-mooney | i have pushed 3 patches to test things | 10:31 |
| sean-k-mooney | i have a dnm to nova to pull in the pin to 6.6.0 | 10:31 |
| sean-k-mooney | and then a pair of patches to nova/devstack to use osc form master | 10:31 |
| gibi | just because we don't have an agreement yet, should we also prepare the 7.1.2 osc release so that if the agreement will be to roll forward then we have the release to roll to? | 11:02 |
| sean-k-mooney | i would say prepare but maybe dont merge until we get feedback form the job using master | 11:04 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/buildset/8f1b2e082a1a4a3e85334ca0dfa5a96f | 11:04 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/3a9bc76be9d6488cadf5b5a14c3e0fae | 11:06 |
| sean-k-mooney | nova live migration passed | 11:06 |
| sean-k-mooney | so the evacuate issue realy does seam to be fixed now | 11:06 |
| opendevreview | Riccardo Pittau proposed openstack/releases master: Release ironic-inspector 12.3.0 for dalmatian https://review.opendev.org/c/openstack/releases/+/929764 | 11:28 |
| opendevreview | Balazs Gibizer proposed openstack/releases master: python-openstackclient 7.1.2 https://review.opendev.org/c/openstack/releases/+/929765 | 11:28 |
| gibi | proposed 7.1.2 ^^ | 11:29 |
| opendevreview | Elod Illes proposed openstack/releases master: Release cyborg RC1 for 2024.2 Dalmatian https://review.opendev.org/c/openstack/releases/+/928523 | 11:48 |
| elodilles | thanks gibi o/ | 11:55 |
| *** mnasiadka1 is now known as mnasiadka | 12:28 | |
| opendevreview | Riccardo Pittau proposed openstack/releases master: Release ironic-inspector 12.3.0 for dalmatian https://review.opendev.org/c/openstack/releases/+/929764 | 13:04 |
| dansmith | my preference is to either pin 6.x or mega-revert 7.x. However, I'd say pinning 6 makes more sense to me.. | 13:29 |
| dansmith | pinning 6 but releasing 7 tells people that we have only confirmed 6 but they could install 7 on a workstation to get new features if they work and/or 7 after enough backports are made and we lift a pin or something | 13:29 |
| dansmith | but I understand that's complicated for distros | 13:29 |
| bauzas | yup, as I said earlier, I'm on the same direction | 13:30 |
| bauzas | we should pin dalmatian u-c to 6 so we could eventually release nova RC1 | 13:30 |
| bauzas | and once that done, we can bump u-c, I'm fine | 13:31 |
| bauzas | but for the moment, we're holding two regression fixes to be merged plus the procedural patches for paperwork | 13:31 |
| bauzas | sorry, but while I understand that OSC folks prefer to have the latest in Dalmatian due to their efforts, the nova folks are tho themselves impacted while they also worked hard for fixing those regressions quickly | 13:32 |
| elodilles | release-team: this is waiting for a 2nd core review: python-openstackclient 7.1.2 https://review.opendev.org/c/openstack/releases/+/929765 | 13:52 |
| elodilles | dansmith bauzas : i'm not happy about it, but if no other possibility exist to be able to release nova rc1 soon, then capping osc <7.0 is the option we should choose. (which we tried to avoid as much as possible in the past) | 13:55 |
| dansmith | elodilles: https://review.opendev.org/c/openstack/requirements/+/929552 | 13:55 |
| dansmith | elodilles: yep, it's unfortunate for sure, but I think we've shown (faster than normal, and luckily before the users did) that this was not a good idea | 13:56 |
| bauzas | elodilles: I'm also a bit concerned by the behavioural change in OSC that defaults to the latest microversion | 13:56 |
| dansmith | ("this" being a rushed roll forward) | 13:56 |
| dansmith | yeah, I haven't confirmed, but if that's the case, the risk is *way* higher than I even thought | 13:56 |
| bauzas | and I want us to buy time with only supporting 7.x in Epoxy, so we could fix bugs in the service projects | 13:56 |
| bauzas | if we allow 7.x to ship with Dalmatian, that's doable to fix bugs by backporting them, but that's an unnecessary hurry | 13:57 |
| bauzas | as I already said, this doesn't prevent users and distros to use 7.x with Dalmatian, provided OSC mentions in their relnotes the known bugs | 13:58 |
| bauzas | (which I never saw) | 13:58 |
| bauzas | btw. I don't see any bug report for the evacuate parameter issue, I just uploaded a fix without a tracker and I dislike that | 13:59 |
| dansmith | also reminder that nova is waiting to merge a pretty substantial bug fix that *has* to go in this release, and the longer we wait the the less soak-time that gets as well, | 13:59 |
| dansmith | so we're really compounding the bad decisions here :/ | 14:00 |
| ttx | alright, we are running out of time, and need to make a call | 15:04 |
| ttx | there is no good solution but the least worse at this stage seems to be to pin to <7 | 15:04 |
| dansmith | thanks ttx | 15:09 |
| dansmith | elodilles: can you +W or are you waiting for more comments? | 15:10 |
| ttx | fungi elodilles frickler hberaud last call for objections | 15:10 |
| elodilles | ttx: i don't object | 15:12 |
| elodilles | i don't like it, but probably this is the easiest way-forward | 15:12 |
| fungi | i suppose the main weirdness with that choice is that the version listed as part of the coordinated release isn't actually used in testing that release | 15:12 |
| elodilles | fungi: yepp | 15:12 |
| ttx | fungi: yes | 15:13 |
| ttx | note that the pin is on specific versions, so a 7.1.2 would bypass it, we need to be careful with that | 15:13 |
| ttx | especially with https://review.opendev.org/c/openstack/releases/+/929765 being proposed | 15:13 |
| dansmith | doesn't the u-c prevent that though? | 15:14 |
| ttx | we might want to turn it into a proper <7, but that can wait | 15:14 |
| dansmith | if not then yes let's just convert the g-r to <7 just to be safe and tweak later | 15:15 |
| fungi | so how does this factor into stable branch matching? there's a stable/2024.2 branch of python-openstackclient which was created from and includes the 7.1.1 tag | 15:16 |
| fungi | i guess any fix for the existing problems would need to be backported to stable/2024.2 then? | 15:16 |
| ttx | good question... will stable/2024.2 testing be able to find 6.6.1 while it's only on stable/2024.1 ? | 15:18 |
| fungi | er, i guess it was branched from 7.1.0 technically, and then 7.1.1 was tagged only on stable/2024.2 | 15:18 |
| ttx | oh well we can merge that and find out, i guess | 15:19 |
| ttx | it's not as if the requirements change could not be reverted | 15:19 |
| fungi | i suppose the state of stable/2024.2 could be bulk reverted back to what was in 6.6.1 and then tagged, but should probably only be tagged there as a patch version increase like 7.1.2 unless we can guarantee that any subsequent releases from master will still be higher | 15:20 |
| elodilles | (sorry, i need to commute home now (office day) but will look back later) | 15:20 |
| fungi | also release notes always get weird when trying to roll back versions like that, so has its own confusion to contend with | 15:21 |
| ttx | elodilles: would be good to approve it in the coming hours to unblock the nova RC1 | 15:21 |
| fungi | i wonder if would make sense to plan to backport fixes into stable/2024.2 asap after release and then try increasing the reqs pin for that branch to bring it into consistency with the coordinated release itself? | 15:23 |
| ttx | fungi: that would be tricky to reconcile with deliverable files... branches are cut from one of the release in the series | 15:23 |
| fungi | yeah, i don't mean changing the branch state, i mean merging a huge revert commit in stable/2024.2 as one of the options if we weren't going to the conflicting reqs pin | 15:24 |
| fungi | er, didn't mean recreating the branch from a different tag | 15:25 |
| fungi | though i'm talking about too many possible alternative approaches in parallel right now | 15:25 |
| ttx | eh yes | 15:25 |
| fungi | i think where my thoughts are headed at this point is: 1. go ahead with the proposed requirements pin for !7.1.0,!7.1.1 (or <7, whatever); 2. after the dust from the coordinated release settles, the maintainers merge regression fix backports into stable/2024.2 and request a 7.1.2 tag there, 3. remove the pin in requirements for stable/2024.2 and reconcile the constraints list to use | 15:29 |
| fungi | the newer point release so that what's tested is consistent with what the release page says is included | 15:29 |
| fungi | it's an exception to our processes, but no matter what we choose it will still be an exception and at least this one leads to a state of eventual consistency and won't be too much of a surprise if it's stated as a clear plan | 15:30 |
| fungi | as it stands right now, the distros will probably end up shipping the newer osc version because that's what we say is part of 2024.2, but at least the window of time where that's not what we're actually using upstream would be fairly short | 15:32 |
| fungi | the later part is also a plan we can check with the tc on, since it's not urgent to resolve | 15:36 |
| clarkb | seems like we should be able to communicate one way or another to distros that they should avoid those releases | 15:37 |
| dansmith | isn't that kinda the point of u-c? | 15:37 |
| dansmith | I mean, it's weird for us to exclude our own thing, but that's basically what we're trying to do here :) | 15:38 |
| dansmith | maybe just some global reno to explain the situation? | 15:38 |
| fungi | well, upper-constraints is for us to freeze the versions of our external dependencies we test with, not to serve as a recommendation for what versions of our dependencies should be packaged by distributions (because they often need to make compromises around shared dependencies with other software in their distribution) | 15:41 |
| sean-k-mooney | fungi: its kind of historically been both | 15:42 |
| fungi | that we also end up pinning our own internally-developed dependencies in constraints is more of a side effect, and one we work around by manually increasing them when we tag new versions | 15:42 |
| sean-k-mooney | its the know workign baseline for a distobution to pull form | 15:42 |
| fungi | it's "this is what we tested with, if you use something else then we can't make any promises it works" yes | 15:42 |
| fungi | on the other hand, the releases.openstack.org site lists what versions of the software our community has developed for each coordinated release | 15:43 |
| fungi | and for logistical reasons, it only rolls forward, we can't really rewind versions there | 15:43 |
| fungi | my point was, the versions of software developed by our community, if appearing in upper-constraints.txt, are an implementation detail and not a description of our coordinated release | 15:45 |
| fungi | i don't expect distribution package maintainers to take what version of python-openstackclient appears in the stable/2024.2 branch upper-constraints.txt as an indication of which version they should package for openstack 2024.2/dalmatian, i expect them to go by what's in our release announcements and listed on the releases.openstack.org site | 15:46 |
| sean-k-mooney | well i kind of would | 15:47 |
| fungi | a conflict between the two is likely to cause confusion, but it's confusion we could plan now to resolve shortly after the coordinated release | 15:47 |
| sean-k-mooney | it alwasy annoys me when our downstream doesnt :) | 15:47 |
| dansmith | it sounds like we've got at least a high level approval.. can we please approve *some* pin so nova can land and cut rc1? | 16:02 |
| * bauzas makes cat eyes | 16:06 | |
| bauzas | fwiw, I don't want to hold OSC 7.1.2 | 16:08 |
| bauzas | if packagers want to ship 7.1.2 with Dalmatian, that will be their decision, not ours | 16:09 |
| bauzas | but for the sake of the integrated gate, the pin is unfortunately the right way forward | 16:09 |
| dansmith | fungi: thanks | 16:11 |
| fungi | sean-k-mooney: it's all well and good until there's a security vulnerability found affecting python-openstackclient v6 and we have no choice but to figure out how to patch that in stable/2024.2 without bringing back the regressions from v7 | 16:19 |
| fungi | which is why i think we need to reconcile it in stable soon after the release rather than just blithely keep it pinned and hope everything will be okay | 16:19 |
| sean-k-mooney | i explcitly didn not pin it to <7 to allow going to a futre 7.x.y | 16:20 |
| fungi | yep, though if we did pin to <7 that can also be undone fairly trivially. it's more about making sure we're okay with rolling forward from 6.6.1 to something >7.1.1 in stable at some point | 16:21 |
| sean-k-mooney | ya so with in the limit of our curernt testing both could be ok because the sepcific issues we hit are adressed on master | 16:22 |
| fungi | we're not resolving the problem, we're just choosing to postpone the fix in stable a bit so we can get through the release push | 16:22 |
| sean-k-mooney | but we know there are some open bugs on masster not nit in the gate jobs | 16:22 |
| sean-k-mooney | we are fully green with 6.6.0 https://review.opendev.org/c/openstack/nova/+/929758?tab=change-view-tab-header-zuul-results-summary and have one unrelated test failure with master osc https://review.opendev.org/c/openstack/nova/+/929757 | 16:25 |
| sean-k-mooney | if we can get RC1 of nova out this week we could go to 7.2.0 or whatever for osc next weeks and see if its stabel enouch | 16:25 |
| sean-k-mooney | going forward we may want to consier actully merging https://review.opendev.org/c/openstack/devstack/+/929754 so that we test from master osc by default to avoid this or have a tips job that runs in the weekly job | 16:27 |
| sean-k-mooney | if we are going to treat osc like a service project we shoudl be using it form master not release anyway | 16:28 |
| clarkb | sean-k-mooney: fwiw I don't think testing with master by default is the answer either | 16:29 |
| clarkb | because then you get into the trap of things only working with unreleased code. Ideally we check both things that development tip works as well as the latest release | 16:30 |
| sean-k-mooney | ya so part of the issue is the devstack functional job in osc | 16:30 |
| sean-k-mooney | is not covering all the edgecases | 16:30 |
| clarkb | once upon a time everything was tested from git master together and then we found out that nothing worked with the various lib releases and switched to the current default but the expectation was that testing against master would also continue | 16:30 |
| sean-k-mooney | we could imporve that and then have a single tips job in consuming projects | 16:31 |
| sean-k-mooney | for example nova-next coudl use it form master whiel the reset use it form main | 16:31 |
| sean-k-mooney | *form release not main | 16:31 |
| sean-k-mooney | clarkb: the quetion is is the clint a lib or a standalone deliverable | 16:32 |
| sean-k-mooney | if its a lib then it was subject to the final release for client lib which was m3 | 16:33 |
| clarkb | sean-k-mooney: right that exact setup is what we expected people to do when we moved to pulling lib releases by default. And it happend for a while but I think overtime people forgot it it became less obvious and now we have the old problem in reverse | 16:33 |
| clarkb | sean-k-mooney: functioanlly nova appears to be using it as a library | 16:33 |
| sean-k-mooney | no we are just it as a clinet via a ansibel hook | 16:34 |
| clarkb | sean-k-mooney: wait | 16:34 |
| clarkb | its just a test thing with an osc install in an ansible playbook? | 16:34 |
| fungi | there are some blurred lines between openstackclient and openstacksdk at this point | 16:34 |
| clarkb | or is it something that nova installs as part of a nova installation to make stuff work? | 16:35 |
| sean-k-mooney | osc is installed by devstack and we have a post run playbook that test evacuate using the client | 16:35 |
| clarkb | because if it is ^ then it is functionally a lib | 16:35 |
| fungi | so in this case, a test dependency | 16:35 |
| sean-k-mooney | correct | 16:35 |
| sean-k-mooney | nova does not use osc at all | 16:35 |
| sean-k-mooney | in the nova code | 16:35 |
| clarkb | and we can't update the code in testing to do the right thing for new osc bceause new osc doesn't support the required functionality? Basically we can't fix the test because the require tooling ws removed? | 16:36 |
| sean-k-mooney | clarkb: the code in the test was correct | 16:37 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/files/test_evacuate.sh#L28 | 16:37 |
| fungi | osc switched its nova interaction backend from novaclient to the combined sdk | 16:37 |
| sean-k-mooney | openstack client was broken | 16:37 |
| fungi | and there were regressions as a result | 16:37 |
| clarkb | ya just double checking this wasn't an expected behavior chagne and thus the major version bump. But sounds like no it is a proper bug so osc needs fixing | 16:38 |
| fungi | which went undiscovered in master osc because it doesn't exercise every function it supports | 16:38 |
| fungi | its testing doesn't exercise every function i mean | 16:38 |
| sean-k-mooney | yep and it was but we fixed it incorrectly becasue of reasons | 16:38 |
| sean-k-mooney | so its fixed again correclty this time but we have not released it yet | 16:38 |
| sean-k-mooney | however in the last week there have been at least 4 diffent osc regression reported | 16:39 |
| sean-k-mooney | fungi: ya so one of the thing we could do is make the devstack-functional job multi node and add in this hook or simialr | 16:39 |
| sean-k-mooney | or have some other jobs that will trigger on some changes. | 16:40 |
| clarkb | seems like we should focus on osc testing to improve coverage of use cases to avoid regressions when changing implementations. Then also in the future try to make those changes early in cycles and release them early so that we can ensure things work well before the big integrated release | 16:40 |
| sean-k-mooney | i.e. you change compute/* run this extra job | 16:40 |
| clarkb | not sure we necessarily need to stop releaseing tools like osc after m3 but try to get big updates in before then | 16:40 |
| sean-k-mooney | that pretty hevery weight however becuase we dont really use the client much | 16:40 |
| sean-k-mooney | clarkb: well we are not ment to based on the release schdule | 16:40 |
| sean-k-mooney | well i gues that depend | 16:41 |
| sean-k-mooney | https://releases.openstack.org/dalmatian/schedule.html#final-release-for-client-libraries | 16:41 |
| clarkb | sean-k-mooney: sure. Neither is opendev but I haven't changed the default ansible versions to version 9 for openstack yet because I'm trying to avoid unexpected impacts to the release | 16:41 |
| sean-k-mooney | that why i asked is it a lib or a standalone deliverable | 16:41 |
| clarkb | sean-k-mooney: just because we aren't explicitly beholden to a schedule doesn't mean we can't make good decisions to try and make potentially disruptive updates as painless as possible | 16:42 |
| sean-k-mooney | if its a standalone deliverbale (which i think it is) then that freeze does not apply | 16:42 |
| clarkb | right, but I think that is the wrong way of looking at it | 16:42 |
| clarkb | you should still try to coordinate and work with the larger pciture | 16:42 |
| clarkb | infra/opendev have long been "slushy" around the openstack release | 16:43 |
| clarkb | we don't actually freeze, but we ask people to take more care | 16:43 |
| sean-k-mooney | right and we coudl also impove that by improvign testing in the requiremetn repo | 16:43 |
| sean-k-mooney | you realise that we are tryign to take the wider view, we tried to fix the issue proactively and get the release out to unblock things and role forward to 7.x | 16:45 |
| clarkb | I'm not talking about any specific release. I'm merely referring to your question on whether or not you must freeze at m3 | 16:45 |
| clarkb | I'm suggesting that no a complete freeze is probably not necessary. But if we're planning big updates like replacing the nova backend for osc ideally we would do those before m3 that is all | 16:46 |
| clarkb | similar to how opendev does not make big changes to zuul job runtimes or gerrit releases around the openstack release | 16:46 |
| clarkb | we make lots of other changes (we just upgraded etherpad for example) | 16:46 |
| clarkb | its all about risk and potential for fallout | 16:46 |
| stephenfin | clarkb: the issue is that we did. I'm writing a mail about this now, but OSC 7.0.0 was release at the beginning of August | 16:46 |
| stephenfin | but the upper-constraint bump failed CI and never merged, and no one told us it didn't merge so we couldn't even prioritise a 7.0.1 | 16:47 |
| clarkb | gottcha | 16:47 |
| sean-k-mooney | clarkb: it was relese in augst but not promoted until 11th of september 1 week ago | 16:47 |
| sean-k-mooney | https://releases.openstack.org/dalmatian/schedule.html#final-release-for-client-libraries | 16:47 |
| sean-k-mooney | sorry wrong link | 16:48 |
| sean-k-mooney | https://github.com/openstack/requirements/commit/59ddc5f862e46c065337467bad673f3d716dc77a | 16:48 |
| stephenfin | sean-k-mooney: worse: it never got promoted. 7.1.0 got promoted | 16:48 |
| sean-k-mooney | stephenfin: well yes but that was for other reasons | 16:48 |
| clarkb | but also I'm not part of the release team. I'm just trying to offer my perspective as someone who does try to assess risk and apply updatse carefuly around the openstack release | 16:48 |
| dansmith | yeah I initially thought we needed a process change from this, but I think it's more just that multiple cascading fails caused us to get into a bad situation | 16:49 |
| stephenfin | clarkb: not to ruin the premise of my email too much, but this is *exactly* what happened with oslo.db and castellan in previous releases | 16:49 |
| dansmith | I think the bigger potential process improvement we need is some gospel over how to handle a post-rc-week breaking change like this (i.e. what we're doing right now) | 16:49 |
| clarkb | and maybe an explicit step in the reelase process that ensures we're up to date with all of our own deliverables before rc time? | 16:50 |
| stephenfin | dansmith: Yes. But also, a failing u-c bump for one of our own dependencies should send alarm bells ringing | 16:50 |
| fungi | i concur | 16:50 |
| dansmith | stephenfin: yes, fair point of improving whatever happened there as well | 16:50 |
| sean-k-mooney | ya but im not sure why tht dint | 16:50 |
| sean-k-mooney | woudl that not have come up in the sdk/clint team meeting | 16:51 |
| stephenfin | We could have fixed this weeks ago and been sitting drinking (virgin) piña coladas right now, but alas 😅 | 16:51 |
| fungi | maybe we can find a better way to plumb notice of such failures to the maintainers of whatever the dependency was that failed | 16:51 |
| clarkb | stephenfin: soju | 16:51 |
| stephenfin | No emails, no IRC pings, nothing. I can't speak for gtema but I only became aware of the issue yesterday | 16:51 |
| fungi | i'll take a virgin piña coladas and a double rum, please | 16:52 |
| stephenfin | clarkb: touché :) | 16:52 |
| frickler | stephenfin: the current issues were only discovered very recently, when 7.1.0 was released and could be bumped in u-c. the issues with 7.0.0 were discovered early in august and I discussed them with you and you fixed those, like the "-c ID" vs. "-c id" issue. | 17:35 |
| stephenfin | frickler: Yes, I knew there were issues and we worked to fix those quickly (I'm pretty sure I fixed it the same day). What I did not know was that that issue was preventing a u-c bump | 17:36 |
| stephenfin | frickler: tbc, I'm not blaming you or anyone else. The failure lies with our processes, not our people | 17:37 |
| frickler | stephenfin: sure. I just checked logs and I linked to https://zuul.opendev.org/t/openstack/build/ce0ea76a7c394b35aaeec5ee02d9914d back then, but I wasn't explicit about being a reqs blocker. I'm also not sure what processes could get improved. what I do think is both the sdks and the reqs teams could use more help and this is what shows here | 17:40 |
| stephenfin | Agreed. I would really like it if the Nova team was as involved with SDK/OSC as many other services are. We frequently get both patches and reviews from Manila, Cinder, Neutron etc. However, in this instance, something as simple as e.g. an IRC bot alerting us to these failures would likely have avoided much of this | 17:45 |
| stephenfin | Also, the saying the reqs team could do with more help is a definite understatement. But hopefully at least I personally have pulled my weight (and continue to pull my weight) there | 17:47 |
| *** bauzas_ is now known as bauzas | 19:25 | |
| sean-k-mooney | stephenfin: im not going to revert your patch but i really think it was incorrect ot go with <7 and inconsitent with our previous stance on puting hard caps as a last resort | 23:14 |
| sean-k-mooney | im also sad that that patch was merge basically with no discssuion while the patch that actully unblocked the gate took a week to merge. im not casting blame here just expressing my furstration as im trying to manage my personal burnout and this really has not helped | 23:17 |
| *** bauzas_ is now known as bauzas | 23:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!