| cardoe | https://review.opendev.org/c/openstack/ironic/+/928106 should be good for a workflow now that the CI is happy | 00:58 |
|---|---|---|
| opendevreview | Doug Goldstein proposed openstack/ironic master: Remove legacy Query support from _paginate_query https://review.opendev.org/c/openstack/ironic/+/891505 | 01:00 |
| cardoe | dtantsur: took ya up on saying anyone can help get your patches over the line. I addressed the review comment | 01:01 |
| opendevreview | Merged openstack/ironic master: Remove legacy Query support from _paginate_query https://review.opendev.org/c/openstack/ironic/+/891505 | 05:53 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Drop description for ZeroMQ https://review.opendev.org/c/openstack/ironic/+/929212 | 06:18 |
| opendevreview | cid proposed openstack/ironic master: Add microversion headers to root ('/') endpoint https://review.opendev.org/c/openstack/ironic/+/928491 | 10:31 |
| cid | dtantsur: I updated the patchset for #928491 ^^. | 10:35 |
| dtantsur | cardoe: thanks! | 11:07 |
| opendevreview | cid proposed openstack/ironic master: Add microversion headers to root ('/') endpoint https://review.opendev.org/c/openstack/ironic/+/928491 | 11:11 |
| opendevreview | Merged openstack/ironic stable/2024.1: Fix inspection if bmc_address or bmc_v6address is None https://review.opendev.org/c/openstack/ironic/+/929137 | 12:10 |
| dtantsur | Core folks, could you please review the other branches? https://review.opendev.org/q/Ibeef33ff9a0acdb7c605bc46ef9e5d203c7aaa6d | 12:34 |
| dtantsur | thank you TheJulia | 13:12 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Make sure qemu-img command is available in debian/suse https://review.opendev.org/c/openstack/ironic/+/929272 | 13:12 |
| TheJulia | I guess https://review.opendev.org/c/openstack/ironic/+/928491 is one way to do it | 13:16 |
| TheJulia | ... There really is no better way at this moment | 13:16 |
| dtantsur | the next CI challenge: wtf is going on with ironic-inspector-tempest | 13:18 |
| TheJulia | Makes sense to figure it out before we figure out what to do with that job in the short and long term | 13:20 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 13:24 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 13:27 |
| opendevreview | Merged openstack/ironic bugfix/26.0: Fix inspection if bmc_address or bmc_v6address is None https://review.opendev.org/c/openstack/ironic/+/929134 | 13:27 |
| opendevreview | Merged openstack/ironic bugfix/25.0: Fix inspection if bmc_address or bmc_v6address is None https://review.opendev.org/c/openstack/ironic/+/929135 | 13:27 |
| dtantsur | TheJulia: I have a vague guess that led me to https://review.opendev.org/c/openstack/ironic-inspector/+/928783 | 13:28 |
| dtantsur | but I cannot even put it in words properly.. | 13:28 |
| TheJulia | what about improperly? | 13:28 |
| dtantsur | I think our network packages have found a different path through the crazy networking setup we have there | 13:29 |
| * TheJulia raises an eyebrow | 13:32 | |
| TheJulia | would pairing up on it help? | 13:32 |
| dtantsur | I'm a bit tired today already but would appreciate help on Monday | 13:32 |
| dtantsur | if the patch passes CI, it can be a sign I'm on the right path at least | 13:33 |
| TheJulia | I should be able to help take a look on Monday | 13:33 |
| TheJulia | ++ | 13:33 |
| dtantsur | great! | 13:33 |
| opendevreview | cid proposed openstack/ironic master: Add microversion headers to root ('/') endpoint https://review.opendev.org/c/openstack/ironic/+/928491 | 13:36 |
| cardoe | I cannot win with https://review.opendev.org/c/openstack/ironic/+/928283 | 13:54 |
| TheJulia | heh | 13:58 |
| TheJulia | cardoe: what is going on? | 13:58 |
| cardoe | Just keeps failing on different tests. | 13:58 |
| cardoe | ironic-tempest-functional-python3 this time | 13:58 |
| cardoe | metal3-integration the other day. | 13:58 |
| dtantsur | mmm, btw, we should re-enable the metal3 job | 13:59 |
| cardoe | Node a051f6e4-81e1-4ae8-b1c1-89fabdd7984b is locked by host np0038485220, please retry after the current operation is completed | 14:00 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Bring back the metal3-integration job https://review.opendev.org/c/openstack/ironic/+/929286 | 14:00 |
| cardoe | Wondering if there's some issue that a wait isn't happening? | 14:00 |
| dtantsur | conductor logs could tell | 14:00 |
| TheJulia | hmmmmmmmmm | 14:01 |
| dtantsur | maybe it's just slow, and we don't wait enough.. | 14:01 |
| dtantsur | or it's a race in the tests, which would be worse | 14:01 |
| cardoe | I'm still learning the stack so struggling to find where it's at. | 14:01 |
| dtantsur | find the logs called "ir-cond" | 14:02 |
| TheJulia | I suspect what is happening based on the test output... that might not expect a lock to already be there | 14:03 |
| TheJulia | but locks can also be transient | 14:03 |
| TheJulia | https://www.irccloud.com/pastebin/kyIk8GgZ/ | 14:05 |
| TheJulia | from https://f0f942e3b9d6b19b2805-6ddd9a666081d58184e95a13f7a762f8.ssl.cf1.rackcdn.com/928283/1/check/ironic-tempest-functional-python3/a84280b/controller/logs/screen-ir-cond.txt | 14:05 |
| TheJulia | looks like the VM took a smoke break... | 14:06 |
| TheJulia | note the recorded times | 14:07 |
| TheJulia | So the database was in a poll as well, so I suspect we're dealing with cross-core throttling | 14:10 |
| TheJulia | a few other things logged in there | 14:10 |
| TheJulia | in that window | 14:10 |
| TheJulia | I *suspect* the test needs to check to see if the node is not locked before making a request | 14:11 |
| TheJulia | cardoe: if you look at the test output: https://f0f942e3b9d6b19b2805-6ddd9a666081d58184e95a13f7a762f8.ssl.cf1.rackcdn.com/928283/1/check/ironic-tempest-functional-python3/a84280b/testr_results.html you can see we did setup, and the very first action in the test is to "deploy" the node, and that itself failed due what feels like cross-core contention | 14:16 |
| TheJulia | I think the cleanest thing will be to amend the tests to explicitly check if the node is locked before issuing any commands | 14:19 |
| TheJulia | https://github.com/openstack/ironic-tempest-plugin/blob/b294d966d155d53f6e5785caee852a031f43fc5d/ironic_tempest_plugin/tests/api/base.py#L286-L287 | 14:19 |
| TheJulia | https://github.com/openstack/ironic-tempest-plugin/blob/b294d966d155d53f6e5785caee852a031f43fc5d/ironic_tempest_plugin/tests/api/base.py#L235-L274 | 14:20 |
| TheJulia | Something like this might explain some of the more spurious random weird failures on that job over the years | 14:21 |
| opendevreview | Merged openstack/ironic stable/2023.2: Fix inspection if bmc_address or bmc_v6address is None https://review.opendev.org/c/openstack/ironic/+/929138 | 14:28 |
| opendevreview | Merged openstack/ironic bugfix/24.0: Fix inspection if bmc_address or bmc_v6address is None https://review.opendev.org/c/openstack/ironic/+/929136 | 14:28 |
| opendevreview | Merged openstack/ironic master: Drop description for ZeroMQ https://review.opendev.org/c/openstack/ironic/+/929212 | 14:33 |
| dtantsur | TheJulia: https://review.opendev.org/c/openstack/ironic-inspector/+/928783 passed which is curious. Now waiting for a recheck on the DNM patch without this fix. | 14:42 |
| dtantsur | and without the fix it failed again | 14:55 |
| dtantsur | so I think my guess my have some ground | 14:55 |
| dtantsur | maybe we just merge the workaround to have the tempest plugin CI going? | 14:55 |
| TheJulia | dtantsur: I guess it is distinctly possible if br-ex (which should be the public bridge) if some logic changes have occured causing traffic to flow to or through it | 15:04 |
| TheJulia | *we* haven't afaik, but doesn't mean something with neutron or ovs/ovn might not have | 15:05 |
| cardoe | TheJulia: thanks for the details. Now I'm following along better. sorry it was a bit async | 16:50 |
| TheJulia | cardoe: no worries, you grok what is going on? | 16:50 |
| cardoe | Yep | 17:49 |
| cardoe | Just over here fighting my keystone. Why did I try to do nice SSO? | 17:50 |
| cardoe | JayF: look what just landed in python-openstackclient... switching to ruff. | 17:50 |
| JayF | yep, I commented on it in -sdks when it happened | 17:50 |
| cardoe | keystone just makes me wanna go stabby | 17:52 |
| TheJulia | I would *really* like keystone to become an option amongst a list | 18:35 |
| TheJulia | And for some that might sound harsh, but I think it is healthy to enable such things, just like Horizon is not everyone's answer | 18:36 |
| * TheJulia looks at her empty pile of spoons and thinks "do I really have enough spoons to even troubleshoot why my backup internet access is not working" | 18:37 | |
| TheJulia | I don't know what starlink has changed, but many many interruptions today | 18:37 |
| * TheJulia blushes at 800G in 46 days | 18:40 | |
| * TheJulia goes to reboot the router | 18:40 | |
| JayF | lol you're on the throttle mac list | 18:54 |
| JayF | :( | 18:54 |
| cardoe | keystone would be fine if it didn't try to do too much | 19:16 |
| cardoe | jack of all trades master of none | 19:16 |
| cardoe | If it was just a service provider and authorization for OpenStack, fine. | 19:17 |
| JayF | When thinking about stuff like this, I always look at why it was created, and the world it was created for | 19:17 |
| JayF | Keystone being what it is made a *lot* more sense in a 2010s world than it does now; I think it's easy to forget how much authentication and security has been in the forefront lately | 19:17 |
| cardoe | I mean it was created in a world before SAML, OAuth and such. | 19:18 |
| JayF | we just have to figure out the better way to adapt to the new world, which I think is mostly happening, just gaps are still being filled in | 19:18 |
| cardoe | I might have gotten a clue as to what's biting me | 19:18 |
| TheJulia | Eh, OAuth and SAML did exist then, but they needed something "openstack" in that mix to fill the void | 19:19 |
| JayF | I think widely adopted is probably more accurate | 19:19 |
| TheJulia | I guess, yeah, before the very early days | 19:20 |
| TheJulia | around that same time | 19:20 |
| JayF | and I've been telling this to cid and adamcarthur5 a bit lately: it's easy to look back and be like "gosh, why didn't we use X, it already existed" when in many cases, 18 X's existed and we just didn't know which one to pick | 19:20 |
| cardoe | Yep. But ya gotta be willing to adopt the winning horse. | 19:23 |
| JayF | I know why that didn't happen oftentimes, and it usually involved me making this argument to my boss (like 9 years ago) and getting pointed at features instead :) | 19:25 |
| JayF | Maintenance work is much more difficult to justify to people who aren't technical or invested in the big picture | 19:26 |
| cardoe | Well then it just ends up as a horrific pile of hacks. | 19:33 |
| TheJulia | Investment in the big picture is key | 19:33 |
| cardoe | You don't want me to start spamming you with keystone code and the docs along side. | 19:34 |
| JayF | No, because then keystone devs could do the same to us :) | 19:34 |
| JayF | we all have these kinda problems | 19:34 |
| TheJulia | reminiscent of a discussion I had with someone earlier today which was much more about big picture context and challenge sharing | 19:35 |
| * TheJulia feels super low spoon county and worries too many context switches have happened today | 19:36 | |
| JayF | I've had a productive friday. Booked next podcast guest, published my current ep, recordded a video on the recent Ironic CVE for next month, and did some bug work | 19:37 |
| JayF | all things that don't consume too much of my mental energy, which I'm still recharging from the ossa tbh | 19:38 |
| TheJulia | yeah | 19:38 |
| TheJulia | that was a slog | 19:38 |
| cardoe | well I finally got it. | 19:46 |
| TheJulia | And unfortunately, what I planned to do first thing got derailed several times including by a broken cloud | 19:47 |
| * TheJulia goes and takes a nap, in hopes of spoon reserve being slightly recharged in an hour | 19:48 | |
| cardoe | https://docs.openstack.org/keystone/latest/admin/federation/mapping_combinations.html#mappings-examples *rage* | 19:48 |
| * TheJulia tries to remember what some folks from japan were proposing for the keystoneauth library | 19:48 | |
| TheJulia | I think it would have given services enough smarts to handle oauth2 endpoints | 19:49 |
| TheJulia | and map everything through based upon standardized field names | 19:49 |
| TheJulia | just don't remember where that ended up | 19:50 |
| TheJulia | anyway | 19:50 |
| * TheJulia .sleep() | 19:50 | |
| cardoe | Yeah that would be great. | 19:56 |
| cardoe | Anyway my issue is the docs are 100% wrong. | 19:56 |
| JayF | yeah, like I said, don't look too closely at ironic docs | 19:57 |
| JayF | we just had a commit to remove a bit of a config description that became invalid /in stein/ | 19:57 |
| cardoe | Anything in list form is very squirrelly to convert | 19:57 |
| JayF | so I know I'm in a glass house and keep all the stones far away :D | 19:57 |
| cardoe | Now I gotta figure out how to escape this to get past the jsonschema validation. Cause the code requires an object and the schema is a string | 20:10 |
| JayF | TheJulia: I think knikolla was working on some of that; it'd be nice to know just to know | 20:22 |
| JayF | Do one of us need to write the prelude for the release with Riccardo on PTO? I don't know when he's going to be back | 20:23 |
| TheJulia | Yeah… seems like something we just need to do | 21:16 |
| JayF | I'm about to put the laptop up, I might take a swing at it over the weekend | 21:16 |
| TheJulia | ++ | 21:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!