| stevebaker[m] | bleh, now sushy-tools-tempest-uefi-redfish-vmedia is failing | 01:00 |
|---|---|---|
| opendevreview | Merged openstack/metalsmith stable/xena: Do a precise node lookup for unprovision https://review.opendev.org/c/openstack/metalsmith/+/849585 | 01:45 |
| ajya | Good morning, sushy-tempest-uefi-redfish-vmedia and sushy-tools-tempest-uefi-redfish-vmedia started to fail yesterday. Should that be related to werkzeug change or is it smth else? | 06:33 |
| rpittau | good morning ironic! o/ | 06:35 |
| rpittau | ajya: I think it's the werkzeug change, the route redirect does not work anymore and it breaks virtualmedia | 06:36 |
| ajya | ok, I'll do a recheck on one of my patches | 06:36 |
| ajya | ..or does it need sushy-tools release first/is it building from source? | 06:38 |
| rpittau | someone already open an issue https://github.com/pallets/werkzeug/issues/2471 | 06:38 |
| opendevreview | Riccardo Pittau proposed openstack/sushy-tools master: [DNM] test werkzeug 2.2 routing change https://review.opendev.org/c/openstack/sushy-tools/+/851162 | 07:16 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent master: WIP: Enable skipping disks for cleaning https://review.opendev.org/c/openstack/ironic-python-agent/+/850861 | 07:17 |
| *** undefined is now known as Guest6176 | 08:19 | |
| opendevreview | Merged openstack/sushy stable/wallaby: Handle AccessError with Basic Auth instead of "reauth" https://review.opendev.org/c/openstack/sushy/+/851061 | 08:32 |
| opendevreview | Merged openstack/sushy stable/yoga: Handle AccessError with Basic Auth instead of "reauth" https://review.opendev.org/c/openstack/sushy/+/851059 | 08:32 |
| opendevreview | Merged openstack/sushy stable/xena: Handle AccessError with Basic Auth instead of "reauth" https://review.opendev.org/c/openstack/sushy/+/851060 | 08:32 |
| rpittau | dtantsur, TheJulia the redirect just seems to not work anymore, see the url change here https://review.opendev.org/c/openstack/sushy-tools/+/851162 | 08:40 |
| rpittau | there's an issue open against werkzeug 2.2.0 https://github.com/pallets/werkzeug/issues/2471 and getting more reports | 08:40 |
| *** undefined is now known as Guest6178 | 09:23 | |
| dtantsur | rpittau: lovely.. but the DNM patch seems to pass, maybe it's our immediate workaround? | 09:48 |
| rpittau | dtantsur: yeah, not sure you checked the issue but apparently the precedent behavior was not correct, I'm trying to move the discussion forward | 09:53 |
| rpittau | anyway, I'll complete the patch and remove the DNM | 09:53 |
| dtantsur | ++ | 10:00 |
| dtantsur | TheJulia, JayF, I remember some talks about no-heartbeat mode for IPA. Has it moved anywhere? | 10:01 |
| opendevreview | Riccardo Pittau proposed openstack/sushy-tools master: Fix for werkzeug 2.2 routing change https://review.opendev.org/c/openstack/sushy-tools/+/851162 | 10:01 |
| iurygregory | good morning Ironic | 11:30 |
| rpittau | hey iurygregory :) | 11:36 |
| TheJulia | dtantsur: I don't really remember | 13:02 |
| ajya | Hi, any 2nd core can spare some time on these backports - https://review.opendev.org/c/openstack/ironic/+/848358 https://review.opendev.org/c/openstack/ironic/+/850356 thanks | 13:46 |
| TheJulia | ajya: approved the first one, the second one needs a different core | 14:04 |
| yoctozepto | morning | 14:04 |
| TheJulia | good morning yoctozepto | 14:04 |
| yoctozepto | any idea what have changed recently in ironic that we are now seeing 404 from `openstack baremetal introspection rule list` | 14:04 |
| yoctozepto | good morning TheJulia | 14:05 |
| TheJulia | yoctozepto: so... what version of flask do you have? | 14:05 |
| yoctozepto | TheJulia: the one from upper-constraints, let me check... | 14:05 |
| yoctozepto | 2.1.3 | 14:05 |
| TheJulia | hmm | 14:06 |
| yoctozepto | I will verify the versions inside the image though | 14:06 |
| TheJulia | could you try 2.1.2 real quick? | 14:07 |
| yoctozepto | TheJulia: "real quick" no but can try in general | 14:09 |
| TheJulia | I suspect https://github.com/pallets/flask/pull/4610/files which dropped in 2.1.3 has broken us | 14:10 |
| TheJulia | in multiple locations :( | 14:10 |
| yoctozepto | ugh :-( | 14:11 |
| TheJulia | yup, inspector is very broken it looks like | 14:16 |
| yoctozepto | ok, the previously working images had 2.1.2 so I guess no need to split the hair | 14:17 |
| yoctozepto | thanks TheJulia | 14:17 |
| yoctozepto | btw, what's your plan now? blacklist the new version or fix the inspector? | 14:18 |
| yoctozepto | s/fix/adapt | 14:18 |
| yoctozepto | well, fix by adapting | 14:18 |
| yoctozepto | or whatever :D | 14:18 |
| TheJulia | dunno yet | 14:18 |
| TheJulia | the conundrum basically is that I *think* the patch might be in the right for flask | 14:19 |
| yoctozepto | ack | 14:19 |
| TheJulia | so blacklist and fix and then unblacklist is likely the only route | 14:20 |
| yoctozepto | ack, please keep me in the loop if possible | 14:22 |
| dtantsur | could we put a cap to global-requirements until we can fix it? | 14:24 |
| dtantsur | mm, I guess blacklist was about the same | 14:24 |
| TheJulia | yeah | 14:25 |
| TheJulia | basically the same, effectively | 14:25 |
| TheJulia | not what any of us needed right now | 14:25 |
| dtantsur | le sigh | 14:25 |
| TheJulia | I'm creating an item in storyboard | 14:25 |
| TheJulia | sushy-tools is also straight flask right? | 14:26 |
| dtantsur | yeah, although a different issue? | 14:27 |
| dtantsur | I think rpittau mentioned 2.2.0 changing the URL handling | 14:27 |
| TheJulia | well. 2.2.0 has not dropped yet | 14:27 |
| TheJulia | at least on pypi | 14:27 |
| dtantsur | ahhhh was werkzeug | 14:29 |
| dtantsur | so it may not be jinja itself | 14:29 |
| dtantsur | yoctozepto: check the werkzeug versions as well | 14:30 |
| TheJulia | ahh, yeah | 14:30 |
| TheJulia | just found it | 14:30 |
| yoctozepto | Werkzeug-2.2.0 | 14:30 |
| * TheJulia tries downgrading | 14:31 | |
| yoctozepto | working Werkzeug-2.1.2 | 14:31 |
| TheJulia | downgraded to 2.1.2 and unit tests pass | 14:31 |
| TheJulia | so it is, indeed, Werkzeug | 14:31 |
| TheJulia | https://github.com/pallets/werkzeug/pull/2433/files feels like it likely should have dropped as Werkzeug 3... | 14:34 |
| TheJulia | so, ironic-inspector, sushy-tools. What else? | 14:37 |
| TheJulia | IPA is fine, fwiw | 14:37 |
| yoctozepto | sweet mother of commits | 14:37 |
| TheJulia | yoctozepto: heh, started reading that one did you? | 14:40 |
| yoctozepto | TheJulia: looked at the +/- stat, it's a rewrite of a core part | 14:40 |
| TheJulia | and it appears to be in the default logic path | 14:42 |
| TheJulia | not that I know it's internal path very well | 14:42 |
| yoctozepto | https://github.com/pallets/werkzeug/issues/2471 | 14:45 |
| yoctozepto | precisely rpittau there | 14:45 |
| TheJulia | yup | 14:53 |
| * TheJulia suspects, based upon maintainer's reply, that we're going to have to chagne our code | 14:53 | |
| TheJulia | yeah, originating with the new request router | 14:55 |
| dtantsur | and we'll need to figure out the way to make it compatible with both versions.. sigh | 14:57 |
| rpittau | sushy-tools should be fixed by https://review.opendev.org/c/openstack/sushy-tools/+/851162 | 14:57 |
| TheJulia | yup.... | 14:57 |
| rpittau | and yeah we'll ahve to change our code | 14:57 |
| dtantsur | rpittau: do we need to bump the requirement or will it work with the old version as well? | 14:57 |
| TheJulia | That doesn't seem horrible, actually | 14:58 |
| TheJulia | If that requires the new request router, given the decorator, I'm not sure it is "easy"... short of a version aware meta-decorator | 14:58 |
| ajya | thanks, TheJulia , rloo | 14:59 |
| rloo | yw :) | 14:59 |
| rpittau | dtantsur: mmm it hosuld work with the older version, version 2.2.0 just enforces the strict_slashes rule if I understood correctly | 15:00 |
| * TheJulia gives it a spin | 15:01 | |
| rpittau | unit tests are ok | 15:01 |
| TheJulia | with 2.1.2? | 15:01 |
| rpittau | yep | 15:01 |
| rpittau | and 2.2.0 | 15:01 |
| TheJulia | okay, that was what I was about to do | 15:01 |
| rpittau | ah ok | 15:01 |
| TheJulia | Okay, then I can go just fix the routing in inspector | 15:01 |
| TheJulia | do we need to assert a minimum? | 15:01 |
| TheJulia | a.. reasonable... minimum? | 15:02 |
| rpittau | TheJulia: for Werkzeug ? | 15:03 |
| TheJulia | yeah | 15:06 |
| TheJulia | hmmmm | 15:06 |
| TheJulia | we have our own decorator and we're invoking flask | 15:07 |
| TheJulia | well... All but one test fixed :) | 15:08 |
| *** Guest6178 is now known as rcastillo | 15:11 | |
| rpittau | probably best stay at elast at version 2.0 | 15:14 |
| rpittau | but it's also Flask that drives that | 15:15 |
| rpittau | when we use it | 15:15 |
| TheJulia | looks like just stripping off the trailing slash fixes almost everything unit test wise | 15:19 |
| rpittau | yep | 15:20 |
| TheJulia | conundrum is the /<version> route | 15:20 |
| TheJulia | there is no /v2.. | 15:20 |
| JayF | dtantsur: I implemented it downstream for Yahoo. It was awful, and not of the quality to be upstreamed. I couldn't determine a way to do it reasonably upstream. | 15:22 |
| JayF | dtantsur: not at least in the timebox I had | 15:22 |
| JayF | dtantsur: literally just stashing huge json dicts in the node object and parsing them out; it only worked there because we didn't have extensive clean/deploy steps | 15:23 |
| rpittau | need to split, see ya tomorrow! o/ | 15:24 |
| dtantsur | JayF: oh, I see.. | 15:36 |
| JayF | dtantsur: iirc the flow was: agent heartbeats, the heartbeat endpoint in conductor would check to see if there was a stored pending response, else it would do normal-heartbeat-things async as usual, but instead of calling back, it stored the result to feed back to the client on next heartbeat | 15:37 |
| JayF | so each step took an extra heartbeat in most cases | 15:38 |
| dtantsur | hmm, that's actually the opposite direction from the one that interests me :) | 15:38 |
| JayF | Yeah, sync heartbeats would be the way to do it upstream | 15:38 |
| dtantsur | I'm interested in resilience to Ironic moving around without doing a proper HA with HAProxy and stuff | 15:39 |
| JayF | are you saying you want IPA-polling-only vs Ironic-api-calling-only? | 15:39 |
| JayF | I remember when TheJulia and I talked about this originally, hell this might even be in the spec; I wanted it to be a 4 way option: bidirectional, poll-only, no-poll, automatic | 15:40 |
| dtantsur | one of the ideas we're considering is to get rid of heartbeats | 15:40 |
| JayF | I don't connect the dots there as to how that gets you where you're going | 15:40 |
| dtantsur | (it's not our favourite idea, just not discarding anything) | 15:40 |
| JayF | I guess I'm trying to say | 15:40 |
| JayF | get rid of heartbeats isn't an idea; the idea is what we'd use in its place :D | 15:41 |
| dtantsur | Ironic calling the agent periodically | 15:41 |
| dtantsur | (the load on conductors, I know) | 15:41 |
| JayF | You're basically going back towards an iscsi driver style scaling model like that | 15:41 |
| JayF | because you take the load on that has been distributed across all the nodes | 15:42 |
| dtantsur | not fully, but somewhat, yeah | 15:42 |
| JayF | maybe not quite as extreme, but it'd have a big hurt on scalability | 15:42 |
| dtantsur | yup | 15:42 |
| TheJulia | Ideally, we support both ways | 15:42 |
| JayF | I also think we have to consider ... yes | 15:42 |
| dtantsur | JFYI our to-go idea right now is to install a small proxy on all masters | 15:42 |
| dtantsur | (we could also go full HA, but we're a bit under time pressure) | 15:43 |
| TheJulia | Well, I have a different unit test failing now :( | 15:43 |
| dtantsur | now it fails differently \o/ | 15:43 |
| TheJulia | okay, doubling up the routes seems to be the magic | 15:45 |
| opendevreview | Merged openstack/ironic stable/wallaby: Fix redfish RAID failed tasks https://review.opendev.org/c/openstack/ironic/+/848358 | 15:50 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: Fix Werkzeug 2.2.0 compatability https://review.opendev.org/c/openstack/ironic-inspector/+/851238 | 15:57 |
| TheJulia | ironic-cores ^ | 15:58 |
| opendevreview | Merged openstack/ironic stable/xena: Fix Redfish RAID to update raid_config https://review.opendev.org/c/openstack/ironic/+/850356 | 16:14 |
| opendevreview | Verification of a change to openstack/ironic bugfix/19.0 failed: Fix redfish RAID failed tasks https://review.opendev.org/c/openstack/ironic/+/848592 | 16:15 |
| opendevreview | Merged openstack/sushy-tools master: Fix for werkzeug 2.2 routing change https://review.opendev.org/c/openstack/sushy-tools/+/851162 | 17:36 |
| TheJulia | iurygregory: we can single core approve to fix gates, if you want on that inspector patch | 17:43 |
| iurygregory | TheJulia, I was planing to do that (was waiting to see if other cores would show up ^^) | 17:47 |
| TheJulia | ahh yeah | 17:47 |
| TheJulia | makes sense | 17:47 |
| iurygregory | now I'm trying to figure out how to fix this issue downstream :D | 17:49 |
| iurygregory | we already install a specific version of sushy-tools, so we probably need to pin the werkzeug version to be installed on it | 17:50 |
| TheJulia | likely yeah | 17:53 |
| iurygregory | going to request a new release for sushy-tools just in case since we already have a fix | 18:01 |
| iurygregory | oh we dropped the support for py36 :D | 18:07 |
| iurygregory | sushy-tools 0.20.0 \o/ | 18:08 |
| ashinclouds[m] | Cool cool | 18:23 |
| opendevreview | Merged openstack/ironic-inspector master: Fix Werkzeug 2.2.0 compatability https://review.opendev.org/c/openstack/ironic-inspector/+/851238 | 19:17 |
| stevebaker[m] | morning | 20:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!