| opendevreview | Julia Kreger proposed openstack/ironic master: Remove autocommit, again. https://review.opendev.org/c/openstack/ironic/+/862832 | 00:25 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 00:25 |
| rpittau | good morning ironic! o/ | 06:48 |
| rpittau | happy friday :) | 06:48 |
| rpittau | vanou: hi! o/ we're very much oriented toward the lextudio ecosystem since it keeps all the various libraries together in a consistent way, you can find more info here https://github.com/etingof/pysnmp/issues/429 | 06:50 |
| opendevreview | Riccardo Pittau proposed openstack/virtualpdu master: Migrate to pysnmp lextudio ecosystem https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 07:28 |
| vanou | rpittau: Thanks for information. I gave glance to code at lextudio/pysnmp and that repo looks like to support crypto protocols required by iRMC as in etingof/pysnmp(SHA, SH256, SHA384, SHA512 and AES). However lextudio/pysnmp still depends on pycrypdomex which latest commit of etingof/pysnmp no longer depends (at pysnmp/proto/secmod/rfc3826/priv/aes.py). This makes issue in | 08:53 |
| vanou | downstream. | 08:53 |
| vanou | ^ s/pycrypdomex/pycryptodomex/ | 08:54 |
| iurygregory | morning Ironic | 11:52 |
| rpittau | vanou: They're open to new patches, so if pycrypdomex dependency is a problem, a patch could be submitted to remove it, following the work of Ilya. The real problem is that there are no other alternatives that I know of for pysnmp that include the entire ecosystem, that project took the entire set, forked it, and it's maintaining it | 12:11 |
| rpittau | vanou: also consider that once PEP 541 work is done the packages names will be moved to lextudio, so at some point the migration will be unavaidable | 12:16 |
| vanou | rpittau: Thanks for comment. I understand lack of best candidate. I may need to make patch against lextudio. Thanks for info of PEP541. | 13:34 |
| rpittau | vanou: no problem! Let's keep the discussion open, glad to help if needed | 13:34 |
| TheJulia | good morning folks | 13:50 |
| TheJulia | vanou: it is also moderately important to highlight, we also cannot go back and add dependencies after the fact to prior releases | 13:52 |
| TheJulia | hjensas: you going to be around for an hour or so? | 14:41 |
| hjensas | TheJulia: yes | 14:42 |
| TheJulia | cool, I'm pondering removing a lock around networking to make metal3 ci happier, and will likely want to discuss once I've gotten through my current pass | 14:44 |
| rpittau | bye everyone, have a great weekend! See you next tuesday o/ | 15:05 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 15:05 |
| TheJulia | hjensas: so, w/r/t ^, i looked at the metal3 logs for removing autocommit... again. And found we were getting lots of locks from conflicting interactions, and I noticed we pull a hard lock (meaning, select, node update, port create, node update) which means we have two extra node updates which inter-twine | 15:07 |
| TheJulia | now, fwiw, the run *with* does have less OperationalErrors | 15:08 |
| TheJulia | https://cd11bacc403da694b347-14879265802d142fe2b972960013518d.ssl.cf2.rackcdn.com/881765/2/check/metal3-integration/1dc4ad4/controller/before_pivoting/ironic.log | 15:08 |
| TheJulia | 72 in that log, I think the prior one was in the hundreds | 15:08 |
| TheJulia | I've changed the logging level for sqlalchemy as an added data point | 15:11 |
| TheJulia | https://417038b3085987ee0140-3dbed99ac93ec835406ebfd1c96f804e.ssl.cf1.rackcdn.com/862832/4/check/metal3-integration/91f0495/controller/before_pivoting/ironic.log has 600 | 15:15 |
| TheJulia | nice | 15:15 |
| opendevreview | Merged openstack/ironic master: Configure docs: we no longer use storyboard https://review.opendev.org/c/openstack/ironic/+/881490 | 15:45 |
| hjensas | That's a nice improvement. | 15:58 |
| TheJulia | JayF: fyi, replied to the question on https://review.opendev.org/c/openstack/ironic/+/855705 | 15:59 |
| TheJulia | hjensas: seeing a similar trend across runs will be the real tell | 16:00 |
| TheJulia | but any open read is a lock for any write to be able to start | 16:00 |
| TheJulia | and vise versa | 16:00 |
| opendevreview | Julia Kreger proposed openstack/sushy master: Retry on ilo state error https://review.opendev.org/c/openstack/sushy/+/880542 | 16:06 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Fix Bandit errors https://review.opendev.org/c/openstack/ironic-python-agent/+/879912 | 16:17 |
| opendevreview | Merged openstack/ironic-specs master: Firmware Interface https://review.opendev.org/c/openstack/ironic-specs/+/878505 | 16:25 |
| dtantsur | iurygregory: ^^ \o/ | 16:32 |
| dtantsur | I think it's time to admit that today is not my day with a braincell and leave for the week :) Holiday on Monday, will talk to you on Tuesday. | 16:40 |
| TheJulia | Have a wonderful weekend! | 16:41 |
| TheJulia | well... 12 OperationalError | 16:44 |
| iurygregory | dtantsur, \o/ | 16:46 |
| iurygregory | one good news for me before the week is over yay | 16:46 |
| JayF | only one? | 16:48 |
| JayF | we need to do something more nice for you iurygregory | 16:48 |
| JayF | You're my favorite person whose name starts with two vowels, is that good news? | 16:48 |
| iurygregory | it totally counts JayF o/ :D | 16:49 |
| iurygregory | yesterday I discovered I have a problem on my right knee, so from time to time I will have to receive some hyaluronic acid injections | 16:51 |
| TheJulia | yeouch | 16:52 |
| JayF | I have two bad knees, you have maximum sympathy from me | 16:52 |
| JayF | hopefully the injections do some magic for you :) | 16:52 |
| iurygregory | they probably do, I need to hope to only get them once a year | 16:53 |
| JayF | I tore a meniscus + MCL in the left knee 20y ago, and tore MCL+PCL+ACL+calf tendon in the right like 5y ago | 17:01 |
| JayF | it's all supposedly healed, but your knee is never quite the same again | 17:01 |
| iurygregory | yeah I can imagine | 17:07 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 17:14 |
| * TheJulia crosses her fingers | 17:15 | |
| JasonF | (not sure if my last messages made it thru) | 18:27 |
| JasonF | I have another grenade failure caused by missing baremetal flavor :| | 18:27 |
| JasonF | I thought we had tracked that down? | 18:27 |
| JasonF | https://zuul.opendev.org/t/openstack/build/a3928ff8369e4fa6841fdf6f637e138c/log/controller/logs/screen-n-api.txt#1965 | 18:27 |
| *** JayF is now known as Guest12444 | 18:27 | |
| *** JasonF is now known as JayF | 18:27 | |
| TheJulia | it was the client I believe | 18:30 |
| JayF | oh, there was an earlier hidden failure when creating the flavor, right? | 18:31 |
| TheJulia | no... it was something with the client causing an error to get generated and then internally the client tried to figure out what to do | 18:32 |
| TheJulia | and I think a flavor name is also not supported, but we end up in that code path ebcause of the client | 18:33 |
| TheJulia | becasue | 18:33 |
| hjensas | TheJulia: made a comment on the WIP patch, I think you meant to assign res = query.one() and then return res ? | 18:58 |
| TheJulia | yeah | 18:59 |
| TheJulia | well, or ref for reference | 18:59 |
| TheJulia | I was thinking res originally, but ref was the pattern already used | 18:59 |
| TheJulia | hmm, 24 OperationalErrors | 19:01 |
| TheJulia | something about port creation in the timeline.... | 19:01 |
| TheJulia | and of course, nothing actually logged out of sqlalchemy's engine | 19:11 |
| JayF | it looks like you lowered the loglevels for sqla in that change | 19:24 |
| JayF | I thought that seemed weird; could that be impacting your debugging output? | 19:24 |
| TheJulia | well, explicitly setting overall to info got all of sqlalchemy's extra instance details :\ | 19:42 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 19:44 |
| TheJulia | an explict set to debug | 19:44 |
| TheJulia | lets see what we get | 19:45 |
| JayF | honestly, at some point I wonder if query logs + aggressive grepping might have to be the path if there are stragglers | 19:54 |
| JayF | or maybe that just reflects how I approach sqla problems with brute force :D lol | 19:54 |
| TheJulia | heh, unfortunately no db side query logs :) | 20:46 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DPU modeling - parent_node DB/Model/API https://review.opendev.org/c/openstack/ironic/+/880114 | 21:20 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: execute on child node support https://review.opendev.org/c/openstack/ironic/+/880545 | 21:20 |
| TheJulia | JayF: iurygregory: a look at that second patch would be appreciated. Please respond with happiness as cookies, sadness as... well.. you'll figure something out :) | 21:21 |
| iurygregory | TheJulia, ack o/ | 21:23 |
| opendevreview | Verification of a change to openstack/ironic master failed: Upgrade to latest hacking - v6 https://review.opendev.org/c/openstack/ironic/+/881256 | 21:41 |
| TheJulia | so I guess we have a conundrum with sqlite | 21:44 |
| TheJulia | we need no active readers (including api consumers) to actually lock for a write | 21:45 |
| TheJulia | ) | 21:45 |
| NobodyCam | a little late: but Good Afternoon Ironic Folks... | 21:45 |
| TheJulia | s/)// | 21:45 |
| NobodyCam | tgif | 21:45 |
| TheJulia | so, I'm wondering if we need to add retry logic to write ops to allow other threads to exit out.... | 21:45 |
| TheJulia | at least walking the methods of the few first db interactions, nothing seems to be held open, but another set of eyes might find something | 21:46 |
| TheJulia | the only variable I can see is metal3 hitting the api tons which would cause it if we're mid-flight on a read | 21:46 |
| TheJulia | o/ NobodyCam | 22:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 23:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!