| rpittau | good morning ironic! o/ | 05:38 |
|---|---|---|
| rpittau | JayF: the automation of bugfix branches like we do for other releases? that would be really nice | 05:44 |
| rpittau | JayF btw to answer your email about bugfix branches, I don't remember if we talked about this but the list in the Supported Branches section in the ironic whiteboard is the source of truth for the bugfix branches, if it's not mentioned there it can be nuked | 05:44 |
| rpittau | we know that a bugfix branch lasts 6 months, after that timeframe it can go away | 05:44 |
| arne_wiebalck | Good morning rpittau and Ironic! | 08:10 |
| rpittau | hey arne_wiebalck :) | 08:10 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-python-agent bugfix/9.6: Update .gitreview for bugfix/9.6 https://review.opendev.org/c/openstack/ironic-python-agent/+/892651 | 08:59 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic-inspector bugfix/11.6: Update .gitreview for bugfix/11.6 https://review.opendev.org/c/openstack/ironic-inspector/+/892653 | 09:00 |
| opendevreview | OpenStack Release Bot proposed openstack/ironic bugfix/22.1: Update .gitreview for bugfix/22.1 https://review.opendev.org/c/openstack/ironic/+/892654 | 09:01 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: Add a CentOS Stream 9 bios job https://review.opendev.org/c/openstack/metalsmith/+/892658 | 09:10 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/9.6: Update .gitreview for bugfix/9.6 https://review.opendev.org/c/openstack/ironic-python-agent/+/892651 | 09:40 |
| opendevreview | Merged openstack/ironic-inspector bugfix/11.6: Update .gitreview for bugfix/11.6 https://review.opendev.org/c/openstack/ironic-inspector/+/892653 | 09:42 |
| opendevreview | Merged openstack/ironic-inspector master: tox: Remove basepython https://review.opendev.org/c/openstack/ironic-inspector/+/890313 | 09:51 |
| opendevreview | Mahnoor Asghar proposed openstack/ironic master: [WIP] Add optional inspection hooks 5-8 https://review.opendev.org/c/openstack/ironic/+/892661 | 09:52 |
| opendevreview | Merged openstack/ironic-specs master: tox: Remove basepython https://review.opendev.org/c/openstack/ironic-specs/+/890316 | 10:02 |
| opendevreview | Merged openstack/ironic bugfix/22.1: Update .gitreview for bugfix/22.1 https://review.opendev.org/c/openstack/ironic/+/892654 | 10:38 |
| opendevreview | Merged openstack/ironic-python-agent master: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/891357 | 11:26 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: Transiton to Storage resource from SimpleStorage https://review.opendev.org/c/openstack/ironic/+/892252 | 11:41 |
| iurygregory | good morning Ironic | 11:58 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [DNM] Aggressive logging of SQL sessions https://review.opendev.org/c/openstack/ironic/+/892673 | 12:56 |
| TheJulia | good morning | 13:15 |
| rpittau | anyone with a spare minute please check CS9 legacy job for metalsmith https://review.opendev.org/c/openstack/metalsmith/+/892658 thanks! | 14:13 |
| TheJulia | why not drop the stream8 jobs? | 14:15 |
| TheJulia | I guess the question becomes, what are we really testing that is *different* and non-standard | 14:16 |
| rpittau | it's habit to not drop anything suddenly, just deprecate first :) | 14:24 |
| TheJulia | or are we testing the test code | 14:24 |
| rpittau | if we want to drop, let's drop | 14:25 |
| TheJulia | I'm in the drop camp :) | 14:25 |
| rpittau | ok, so need to merge that, then change every places where we still use the cs8 jobs, then drop them | 14:25 |
| TheJulia | button mashed | 14:26 |
| rpittau | thanks! | 14:26 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Use sparkingly new metalsmith cs9 job https://review.opendev.org/c/openstack/ironic/+/892680 | 14:28 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Use sparkingly new metalsmith cs9 job https://review.opendev.org/c/openstack/ironic-python-agent/+/892681 | 14:30 |
| rpittau | these ^ are to avoid excessive entropy before the drop | 14:31 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: preserve/handle config drives on 4k block devices https://review.opendev.org/c/openstack/ironic-python-agent/+/888794 | 15:10 |
| dtantsur | Even the summary sounds "fun" ^^^ | 15:10 |
| TheJulia | oh yes | 15:13 |
| TheJulia | so very fun | 15:13 |
| TheJulia | very very fun | 15:13 |
| dtantsur | well, I'm dealing with TLS (again..), I know what fun is | 15:15 |
| TheJulia | ugh | 15:16 |
| dtantsur | I should probably give up on NOT using insecure=true.. ever | 15:18 |
| dtantsur | TheJulia: btw, I've spent some time today scratching my head about whether https://opendev.org/openstack/ironic/src/branch/master/ironic/db/sqlalchemy/api.py#L390 is a safe thing to do | 15:18 |
| dtantsur | I got stuck trying to understand if the resulting Row objects may have any link to a session... | 15:19 |
| dtantsur | The docs swear it's just a transparent namedtuple-like object. Dunno whether to believe them. | 15:19 |
| TheJulia | commented on https://review.opendev.org/c/openstack/ironic/+/892414, specifically it was already breaking from my point of view, I have been wondering if we should extend the ipa timeout/give-up on that heartbeat anyway. | 15:20 |
| TheJulia | oh, ouch, yeah, we should likely shift that out | 15:21 |
| dtantsur | TheJulia: yeah, your IPA patch solves my concern fully | 15:21 |
| TheJulia | I'm still wondering if the IPA side timeout should be something like 600 seconds | 15:21 |
| TheJulia | dunno | 15:21 |
| dtantsur | ah, right, that patch hasn't modified the timeout, has it? | 15:22 |
| TheJulia | it *does* exit, the failure case we were seeing in metal3 should be fully eliminated by just the ironic side change to short circuit the failure from happening | 15:22 |
| dtantsur | I must have missed that | 15:22 |
| TheJulia | that is correct, it didn't | 15:22 |
| TheJulia | I was strugglign with justifying it | 15:22 |
| TheJulia | I 'll change it | 15:22 |
| dtantsur | We repeat heartbeats forever, so why not.. | 15:22 |
| TheJulia | yeah | 15:22 |
| TheJulia | and if it fails, it exits | 15:22 |
| TheJulia | and gets restarted | 15:22 |
| TheJulia | so *shrug* | 15:22 |
| dtantsur | also true | 15:23 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Extend the lookup timeout to 600 seconds https://review.opendev.org/c/openstack/ironic-python-agent/+/892686 | 15:31 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/2023.1: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892593 | 15:31 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/zed: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892594 | 15:31 |
| dtantsur | TheJulia: I'll probably keep rechecking https://github.com/metal3-io/ironic-image/pull/430 until it produces a failure | 15:33 |
| JayF | dtantsur: I'm gonna be honest, I'm really confused about your comment on Julia's change about it not being impactful | 15:33 |
| JayF | dtantsur: I thought it was an impactful chagne too, can you help me understand how it was a noop? | 15:33 |
| dtantsur | JayF: I'm confused why y'all are confused.. how is it NOT noop? | 15:33 |
| TheJulia | sure, do we know what the merge in opendev to -> it being in that job window is | 15:33 |
| * TheJulia is just confused | 15:34 | |
| dtantsur | you're shifting DECREF maybe 1 or 2 opcodes below or above. okay? | 15:34 |
| *** TheJulia is now known as confused | 15:34 | |
| *** confused is now known as TheJulia | 15:34 | |
| JayF | dtantsur: my understanding was that you aren't guaranteed when that list comp will get evaluated, especially in a world with monkey patched eventlet not giving you guaranteed exec order in all cases | 15:34 |
| * JayF admits this is one of his weakest parts in understanding of python | 15:35 | |
| JayF | it's likely there's some hair to split here that I'm not aware of | 15:35 |
| TheJulia | the challenge is doing a return when your in a with block | 15:35 |
| JayF | like (x for x in y) is different than [x for x in y] | 15:35 |
| TheJulia | the underlying session has to be unwound | 15:35 |
| dtantsur | JayF: if I understand anything about eventlet at all, it cannot switch context without I/O | 15:35 |
| dtantsur | unless it uses tracking to hack into opcodes, which would be HORRIBLY slow | 15:35 |
| TheJulia | the list comprehension stuff is an extension of sorts, i.e. (do we have results yet, or not) | 15:35 |
| dtantsur | JayF: which exactly line are we talking about, to be specific? | 15:36 |
| TheJulia | dtantsur: ++ to the clarification | 15:36 |
| TheJulia | or to the point seeking it | 15:36 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/892621/1#message-968ac90da82fe6ef1d9fa1534dab31697d4f548f | 15:36 |
| dtantsur | JayF: I mean, which line you're defending? | 15:36 |
| JayF | the changes in ironic/db/sqlalchemy/api.py | 15:37 |
| dtantsur | JayF: that's the most puzzling change. How is adding a new variable changing anything? | 15:37 |
| JayF | I want to get real understanding bceause if you're right, I wasted a crapton of time when we were troubleshooting the migrations stuff, just chasing my own tail | 15:38 |
| dtantsur | re "with". "with" is a fancy try..finally block. | 15:38 |
| JayF | reference my original comment about being unsure when the list comp is evaluated | 15:38 |
| JayF | I'm working off the assumption you're right fwiw I'm just trying to understand | 15:38 |
| dtantsur | List comprehensions are fancy loops. | 15:38 |
| dtantsur | (Python does not have special opcodes for them, so they're not JUST syntactic sure, but close to that) | 15:39 |
| JayF | would ( ) be different than [ ] in that case | 15:39 |
| dtantsur | () will be very different | 15:39 |
| JayF | because you'd return a generator and not pre-create the list? | 15:39 |
| TheJulia | yeah, but with py310, we've discovered the with block doesn't get unwound completely | 15:39 |
| JayF | > JayF | like (x for x in y) is different than [x for x in y] | 15:39 |
| dtantsur | () is a generator expression, it does not evaluate anything | 15:39 |
| * JayF had a hunch | 15:39 | |
| dtantsur | [... for ... ] and { ... for ... } are evaluated in place | 15:40 |
| dtantsur | JayF: returning from try..finally.. can be confusing at times (try to guess what https://paste.opendev.org/show/b8LCdwRhe4mBoRalRx1M/ does) | 15:41 |
| dtantsur | but the finally block is always evaluated | 15:41 |
| JayF | this sorta gets to my second question | 15:41 |
| JayF | is it harmful to make that extra var if it makes things more clear? | 15:41 |
| JayF | actual meaningful performance hit or just churn? | 15:41 |
| JayF | dtantsur: that seems wrong. I feel like returning from finally is a smell | 15:42 |
| JayF | dtantsur: either that or I avoid it because f-if-i-know what happens lol | 15:42 |
| dtantsur | JayF: sure, that's horrible code, but an interesting mental experiment | 15:42 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/yoga: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892687 | 15:42 |
| JayF | ugh I have no idea | 15:43 |
| JayF | probably the return doesn't happen | 15:43 |
| JayF | because you can't catch that raise later if you've also already returned | 15:43 |
| JayF | it's almost inherently racey so I assume it's just a rule that establishes ordering in that case | 15:43 |
| dtantsur | JayF: https://paste.opendev.org/show/bqSnt7vAA87HRKawmQdU/ | 15:43 |
| dtantsur | :) | 15:43 |
| JayF | I appreciate that you think I can read that ;) | 15:44 |
| dtantsur | well, I'm somewhat struggling with opcodes myself now | 15:44 |
| dtantsur | I knew that it would return 42 (already checked back in the days) | 15:44 |
| * TheJulia wonders if she should just sit back and let you guys argue it out | 15:45 | |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/xena: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892595 | 15:46 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/wallaby: Handle the node being locked https://review.opendev.org/c/openstack/ironic-python-agent/+/892596 | 15:46 |
| JayF | TheJulia: I am outta brain | 15:46 |
| JayF | TheJulia: and mainly just wanted the conversation to happen since that landed over an objection and talking is good | 15:46 |
| JayF | especially for technical stuff when there's a chance to learn | 15:46 |
| TheJulia | dtantsur: which version of python did you do that with locally? | 15:47 |
| dtantsur | TheJulia: 3.11.4 | 15:48 |
| dtantsur | I did some comparisons, and I'm sure this is because of 36 POP_EXCEPT | 15:48 |
| TheJulia | I think metal3 is ending up with 3.10 | 15:48 |
| dtantsur | yeah, ignore the exception stuff, this was for JayF's entertainment | 15:49 |
| TheJulia | yeah, i know :) | 15:49 |
| dtantsur | what I'm trying to say is: return in the main body of try..finally or with: does not prevent the "finally" part from running | 15:49 |
| TheJulia | That makes sense | 15:49 |
| TheJulia | my whole concern around db sessions as we've seen is not trying to return from with-in them since we've basically discovered that starting in py310, the sessions didn't seem to get entirely unwrapped | 15:50 |
| TheJulia | at least, immediately as we were sort of expecting, and from our point of view, seems better to unwrap sooner rather than later | 15:50 |
| TheJulia | specifically line 1511 in https://review.opendev.org/c/openstack/ironic/+/892621/1/ironic/db/sqlalchemy/api.py | 15:52 |
| dtantsur | OH, list comprehensions are now implemented via MAKE_FUNCTION? Fun times. Should not change much though. | 15:52 |
| opendevreview | Merged openstack/ironic-python-agent master: tox: Remove basepython https://review.opendev.org/c/openstack/ironic-python-agent/+/890315 | 15:52 |
| dtantsur | TheJulia: I'm curious which logic SQLA uses to close sessions. I.e. can it decide "You've called __exit__ but I think I'll keep this open for a few minutes more" :) | 15:54 |
| TheJulia | so, it depends on the objects you have because the session is held on to objects until you extract the contents or delete the object. Which is why we had to make the whole api.py logic be more careful with ORM objects in particular | 15:57 |
| * dtantsur wants to rewrite everything in pure SQL.... | 15:59 | |
| TheJulia | ... we've rewritten a lot to go away from pure orm | 15:59 |
| TheJulia | the thing that has kept us from completing that is things like pagination | 15:59 |
| TheJulia | replace the backend paginate query stuff, and I'll buy you a beer | 15:59 |
| dtantsur | I think it's easier for me to buy myself a beer than to rewrite the pagination stuff :-P | 16:00 |
| JayF | What Julia says tracks with my understanding: the goal is to try and ensure that any sqla-session-attached objects are detached/copied out/whatever so that the connection closes when the cxt mgr exits | 16:00 |
| TheJulia | seriously though, oslo.db's pagination handler *does* do the right thing as long as you just don't ask it to query the model and hydrate the entire thing | 16:01 |
| dtantsur | JayF: I sorta get it, I just don't quite get how you get there | 16:01 |
| JayF | dtantsur: me either, obviously ;) | 16:01 |
| TheJulia | lots of sqlalchemy pain | 16:01 |
| TheJulia | and whiskey | 16:01 |
| TheJulia | and and no migraines | 16:01 |
| dtantsur | JayF: take https://review.opendev.org/c/openstack/ironic/+/892621/1/ironic/db/sqlalchemy/api.py#1490. Adding rest does not make query garbage collected before the session exits. | 16:01 |
| dtantsur | s/rest/res/ | 16:02 |
| JayF | does not returning from inside the cxt mgr make a difference? | 16:02 |
| dtantsur | depending on what cpython decides to do, "query" may survive until the whole frame is collected | 16:02 |
| dtantsur | JayF: no. Context managers are not lexical scopes. | 16:02 |
| dtantsur | I.e. they don't build a frame. I.e. the variables in them only get collected once the function exits. | 16:03 |
| TheJulia | dtantsur: so, that sqlalchemy result from the comprehension should just be a clean list | 16:03 |
| opendevreview | Merged openstack/metalsmith master: Add a CentOS Stream 9 bios job https://review.opendev.org/c/openstack/metalsmith/+/892658 | 16:05 |
| JayF | Do we run in OpenStack or Metal3 gate with osprofiler enabled? | 16:05 |
| rpittau | good night! o/ | 16:06 |
| TheJulia | JayF: afaik, no | 16:07 |
| JayF | okay | 16:08 |
| dtantsur | JayF: do you think it would be helpful? I can make a DNM patch like in https://github.com/metal3-io/ironic-image/pull/430 | 16:13 |
| JayF | dtantsur: I should not be the arbiter of python correctness. :D I just wanted to make sure you and Julia had a chance to chat about that comment and that everyone was OK and had a shared understanding | 16:13 |
| JayF | I don't want two of the more prolific folks in Ironic to have differing mental models of what's going on | 16:14 |
| dtantsur | Yep, that's essentially the reason I blocked the patch instead of just waving it through. So that we talk and understand what we're trying to do. | 16:14 |
| JayF | I thought it landed though? | 16:14 |
| JayF | your block was too late | 16:15 |
| JayF | did I misread? | 16:15 |
| dtantsur | I think it did not, but I don't care THAT much (as much as I care about discussing it) | 16:15 |
| JayF | oh I did misread | 16:15 |
| TheJulia | it did not merge | 16:19 |
| dtantsur | TheJulia: (unrelated) pinged you on k8s slack because someone is struggling with Director Operator | 16:22 |
| dtantsur | (in case you don't follow its notifications) | 16:23 |
| TheJulia | I have no idea where it even sends those | 16:23 |
| dtantsur | good for you | 16:23 |
| TheJulia | I have slack open on the work computer in three spaces, none of them are the main k8s space | 16:23 |
| JayF | fwiw, if there are any slack channels/environments I should be in I'm not, lmk. I have to slack for work anyway :) | 16:24 |
| JayF | It's nice to see all the metal3 questions fly by and sometimes even know what they're talking about ;) | 16:24 |
| JayF | "I need to add a controller to my florp, but the glorper stopp heartbeating. How do I glorp my florp?" | 16:24 |
| dtantsur | ROFL | 16:24 |
| TheJulia | I've sort of enjoyed some of the side discussion in the scientific sig slack, but there is discussion of moving off slack because the matrix bridge support is going to go away or something along those lines | 16:24 |
| dtantsur | :sadpanda: | 16:25 |
| dtantsur | okay, https://github.com/metal3-io/ironic-image/pull/430 is producing a ton of information, I only need it to fail now | 16:30 |
| TheJulia | heh | 16:47 |
| TheJulia | and then dig through it | 16:47 |
| dtantsur | yep... | 16:49 |
| * TheJulia slides a pour of Angel's Envy Whiskey towards dtantsur | 16:50 | |
| dtantsur | cheers! | 16:50 |
| dtantsur | I was actually going to go climbing this evening, but I feel so bloody tired | 16:51 |
| TheJulia | It was difficult for me to get out of bed this morning, I woke up with a migraine and didn't take meds until about an hour ago | 16:52 |
| dtantsur | sigh, I hope it gets better soon | 16:54 |
| TheJulia | it is actually doing a lot better | 17:05 |
| TheJulia | hjensas: by chance, have you revised my ovn patch, or dropped the size by ?30? bytes? | 17:06 |
| TheJulia | hjensas: I'm guessing since PMTU discovery also doesn't seem like a thing because mtu size handling is lacking | 17:06 |
| TheJulia | looks like the answer is no | 17:08 |
| TheJulia | I think we could use the outbound path mtu limit method on routes | 17:08 |
| TheJulia | but I need to look that up and make sure it behaves the way I think | 17:09 |
| TheJulia | So once we have v4 OVN working in CI, do we have any feelings on making it the default? | 17:10 |
| JayF | I think there needs to be some baking time | 17:10 |
| JayF | and also like, I don't know where we are on this currently | 17:10 |
| JayF | I'm trying to figure out how to ask it | 17:10 |
| JayF | is OVN more or less "real" of a testing scenario than we have right now? | 17:11 |
| TheJulia | I feel mixed on it, because I know neutron folks more or less want to burn classic ovs usage down to the ground | 17:11 |
| TheJulia | but I *do* agree, ovn needs a bit more baking | 17:11 |
| JayF | I've spent the last 3 days in IRC hearing multiple parts of our community grumble about OVN and reading every "it doesn't do X" issue in the world | 17:11 |
| TheJulia | the fact it lacks fragmentation is a huge giant blinking red light flag | 17:11 |
| JayF | including lots of edges around fragmentation, dns, etc | 17:11 |
| TheJulia | of doom, and sadness | 17:11 |
| TheJulia | yeah, agree | 17:12 |
| JayF | So that's all to say; we need emperical evidence gathered over time it's better | 17:12 |
| TheJulia | it is never going to get there without some umph of usage I guess | 17:12 |
| TheJulia | I think the decision "its better" was long ago made by the neutron team | 17:12 |
| JayF | For purposes of what Ironic cares about testing, is it different? | 17:12 |
| TheJulia | and honestly, just playing around with it, it *does* nicely shift some of the configuration barrier/burden out of config files to OVN | 17:12 |
| TheJulia | but that totally needs books written about it | 17:13 |
| JayF | In terms of actually testing Ironic properly | 17:13 |
| JayF | not in "it's less crashy" or "it's faster" | 17:13 |
| TheJulia | well, given neutron hasn't merged v6 support yet, it is incomplete from my point of view | 17:13 |
| JayF | that to me is the primary question; if it tests Ironic as well or better than it was before, I'm game to promote whatever is most stable | 17:13 |
| JayF | ah there's the trick eh :D | 17:13 |
| TheJulia | yeah | 17:16 |
| TheJulia | it is definitely more invasive, and if this migraine clears I should go ahead and write prose on the subject | 17:16 |
| JayF | it's hard for me to tell in neutron code in our testing | 17:17 |
| JayF | where we're just plumbing things up to get Ironic tests to the next step | 17:17 |
| JayF | and when we're actually testing something reasonably unique | 17:18 |
| TheJulia | the uniqueness is more the environment | 17:24 |
| TheJulia | but yeah | 17:25 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892705 | 18:29 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 18:31 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 18:32 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 18:33 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 18:34 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 18:49 |
| opendevreview | Merged openstack/ironic-python-agent master: Use sparkingly new metalsmith cs9 job https://review.opendev.org/c/openstack/ironic-python-agent/+/892681 | 18:57 |
| TheJulia | JayF: I feel like you found an ovn doc on limitations, did you happen to stumble across one? | 20:38 |
| TheJulia | nvmd, found it | 20:44 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM Enable OVN https://review.opendev.org/c/openstack/ironic/+/885087 | 21:09 |
| TheJulia | so, lets see if that works | 21:09 |
| TheJulia | ^^^ hjensas | 21:09 |
| mnaser | has anyone ran into an issue in the past when using host aggregates and upgrading past xena (where it starts using placement for az) | 21:27 |
| mnaser | the map_az_to_placement_aggregate function seems to add the host aggregates that correspond to the az | 21:27 |
| mnaser | and that eliminates all options from the placement | 21:27 |
| mnaser | so `openstack --debug allocation candidate list --limit 1000 --member-of 'af1ba1cf-d045-4042-a4eb-480a126b0021,c2591cf3-4c60-453f-8065-b3944904eaaf' --resource CUSTOM_AMD7402_MEM_512G_DISK_2X240G_2X960G='1'` simulates what nova does, the member-of is what's breaking it all, and i cant seem to find a way to get ironic to make the baremetal nodes be part of the placement aggregate | 21:29 |
| TheJulia | hmm, never looked at anything along those lines but also don't know/understand the placement mechanics or impact/mechanism related to member-of | 21:41 |
| JayF | same, I don't have much understanding of placement mechanics | 21:42 |
| mnaser | TheJulia, JayF: well, right now i think what is happening is nova is trying to schedule the bm system in the 'nova' az, and since we use az, it picks 2 host/placement aggregates which do NOT include the resource provider for the ironic node | 21:45 |
| mnaser | https://bugs.launchpad.net/nova/+bug/2002400 and because of this, cant really add them via nova to the resource provider aggregate :\ | 21:45 |
| TheJulia | sigh | 21:47 |
| mnaser | im trying to manually run stuff like: ` openstack resource provider aggregate set --aggregate af1ba1cf-d045-4042-a4eb-480a126b0021 --generation 110 8513757c-a9f2-4ceb-9b6b-c3ed71328263` | 21:47 |
| TheJulia | I suspect the next re-execution, based upon pas-na's original bug would undo it anyway | 21:48 |
| mnaser | yeah that was the whole root cause of this | 21:51 |
| mnaser | so i think if you're using nova + az + ironic = bad time | 21:52 |
| JayF | I don't understand nova well enough to get the full grasp of the situation; but it's somewhat worrisome that the proposed patch there had -1s that appeared to be "we don't want this ever" on it | 21:53 |
| JayF | mainly | 21:53 |
| JayF | > we decided a while ago to restrict ironic support for placement aggregates to map the existing nova behaviour, which was to have one compute as as host... | 21:53 |
| TheJulia | I mean, if someone comes with a valid reason, that seems like an artificial barrier to erect which only hurts users | 21:56 |
| JayF | I don't know enough about the Nova model to know and/or make the case it's an artificial barrier. | 21:57 |
| JayF | It just appears, with no external information, that the bugfix might have been chased away :( | 21:57 |
| JayF | I'm fairly sure my downstream is gonna hit this too | 21:57 |
| TheJulia | it might not be artificial, but should explained why it can't be done. The effort is an investment | 21:58 |
| JayF | yeah seems like there's value in getting a workaround | 21:59 |
| JayF | mnaser: Do you have nova time or resources (either you personally or via vexxhost) which could help us address this upstream? | 21:59 |
| TheJulia | actually reading the change comments, at least they are verbose and do explain | 21:59 |
| JayF | Yeah, it's not all a bad comment, it's just a sign to me that I'm not going to be able to pick up that patch; someone with deeper nova understanding would need to grab it, run with it, and try to find a way around | 22:00 |
| JayF | bluntly if I got that comment I probably would've peaced-out from the patch too :D | 22:00 |
| mnaser | JayF: i can try and drive some discussion and fix if i figure out why, i'm just in the middle of a billion fires right now, but that's generating a long list of "broken shit" that i'll get through | 22:00 |
| JayF | mnaser: ack; thank you. I'll try and get some nova resources from my side too | 22:01 |
| TheJulia | JayF: oh yeah, it is very much a "we don't want to fix this" comment | 22:02 |
| TheJulia | well, comments, there are more then one | 22:02 |
| opendevreview | Illia Polliul proposed openstack/ironic-python-agent-builder master: Exclude .pyc encoding files and add unzip package. https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/892706 | 22:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!