| opendevreview | Julia Kreger proposed openstack/ironic master: Use centos grub artifacts with centos ramdisk for vmedia https://review.opendev.org/c/openstack/ironic/+/869103 | 00:42 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix grub config path default https://review.opendev.org/c/openstack/ironic/+/869134 | 00:42 |
| TheJulia | okay, lets see if *that* works | 00:42 |
| TheJulia | stevebaker[m]: you were right, it is environmental | 00:42 |
| * TheJulia wishes grub would printf "couldn't find my config" | 00:43 | |
| stevebaker[m] | oh yeah, I'm surprised ubuntu EFI assets ever worked in centos | 00:46 |
| TheJulia | we wrote a half baked config that did work | 00:47 |
| TheJulia | and the code always dropped in the ubuntu location | 00:47 |
| TheJulia | with ubuntu artifacts | 00:47 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Use centos grub artifacts with centos ramdisk for vmedia https://review.opendev.org/c/openstack/ironic/+/869103 | 01:05 |
| opendevreview | Verification of a change to openstack/ironic master failed: Temporarily mark redfish vmedia as non-voting https://review.opendev.org/c/openstack/ironic/+/869107 | 01:11 |
| opendevreview | Merged openstack/ironic-python-agent master: Remove old, unused proxy.sh file https://review.opendev.org/c/openstack/ironic-python-agent/+/869105 | 01:19 |
| TheJulia | ipa-centos9-master.tar.gz2023-01-04 01:28393M | 01:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Reset VM footprint to 2.6GB https://review.opendev.org/c/openstack/ironic/+/869136 | 01:44 |
| rpittau | good morning ironic! o/ | 08:52 |
| rpittau | TheJulia, JayF, iurygregory, stevebaker[m], can you please have a look at https://review.opendev.org/c/openstack/ironic/+/868521 when you have a moment? besides being a very subtle issue we need that to unlock bifrost CI | 08:56 |
| rpittau | also looking for final approval for https://review.opendev.org/c/openstack/ironic/+/869084 | 09:04 |
| zigo | Hi there! Can someone help me solve https://bugs.debian.org/1026524 ? | 09:10 |
| rpittau | zigo: hi! is that with inspector master ? | 09:10 |
| zigo | rpittau: That's with stable/zed | 09:11 |
| zigo | (which hopefully will be part of Debian Bookworm if we solve this bug) | 09:11 |
| rpittau | ok, I'll have a look in a minute | 09:12 |
| zigo | Thanks ! | 09:12 |
| zigo | IMO, that's probably an issue with the newer python3-cryptography package that has changed in Debian Unstable. | 09:13 |
| rpittau | I'm having issues with another lib at the moment | 09:25 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Document ngs_trunk_ports and ngs_physical_networks https://review.opendev.org/c/openstack/networking-generic-switch/+/868313 | 09:37 |
| rpittau | zigo: I was able to reproduce the issue, now let's see if I can confirm the root cause | 09:53 |
| zigo | gr8 ! | 09:54 |
| opendevreview | Riccardo Pittau proposed openstack/ironic stable/zed: Fix CI https://review.opendev.org/c/openstack/ironic/+/868789 | 10:06 |
| rpittau | zigo: doesn't look like something related to cryptography, I'm seeing the same issue on master | 10:28 |
| zigo | Oh, ok. | 10:29 |
| opendevreview | Merged openstack/ironic-python-agent-builder stable/yoga: Remove /var/tmp/* from images https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/869120 | 10:31 |
| rpittau | zigo: is it possible to get the exact python version used, I suspect a change in most recent python versions that broke (or fixed?!?) threading, it works fine in python 3.9 for example | 10:41 |
| zigo | rpittau: Python 3.11. | 10:41 |
| zigo | Currently, 3.11.1 in Debian Unstable. | 10:42 |
| rpittau | zigo: I confirm it does NOT work with python 3.11.1 and 3.10.9, it works with python 3.9.16 and 3.10.6, so there's been some change in threading recently that's causing the issue | 10:51 |
| zigo | Oh, good to know! | 10:57 |
| opendevreview | Merged openstack/networking-generic-switch master: Remove lib/neutron-legacy leftovers https://review.opendev.org/c/openstack/networking-generic-switch/+/868193 | 10:59 |
| rpittau | zigo: we're not directly using threading, we rely on oslo_concurrency lockutils module, maybe the oslo guys will be able to get a better understanding of what's going on | 11:01 |
| zigo | rpittau: Oh, so you believe it's something to be fixed in oslo.concurrency then? | 11:01 |
| rpittau | maybe | 11:02 |
| rpittau | another possibility, now that I see it, is that the hack we were using for the release_lock function does not work anymore and we'll have to finally deal with that :/ | 11:13 |
| iurygregory | good morning Ironic | 11:29 |
| iurygregory | rpittau, https://review.opendev.org/c/openstack/ironic/+/868521 lgtm, just wondering if we could add a unit test, but I'm ok with it so I've +2 | 11:31 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix selinux context of published image hardlink https://review.opendev.org/c/openstack/ironic/+/868521 | 11:51 |
| rpittau | iurygregory: added the check to current unit tests ^ | 11:52 |
| iurygregory | rpittau, tyvm! | 11:53 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: setup.py: Remove 'py_modules' https://review.opendev.org/c/openstack/ironic/+/869197 | 12:02 |
| opendevreview | Stephen Finucane proposed openstack/ironic-inspector master: setup.py: Remove 'py_modules' https://review.opendev.org/c/openstack/ironic-inspector/+/869198 | 12:05 |
| iurygregory | interesting... rpittau fyi ^ | 12:12 |
| iurygregory | I love the fact zuul already failed lol (*joke*) | 12:13 |
| rpittau | yep, saw that | 12:13 |
| rpittau | just curious to understand if pbr does not use auto discovery how come the py_modules workaround works | 12:19 |
| opendevreview | Oleksandr Kozachenko proposed openstack/networking-generic-switch master: Add ArubaOS-CX switch support https://review.opendev.org/c/openstack/networking-generic-switch/+/868598 | 13:18 |
| *** mraineri_ is now known as mraineri | 14:02 | |
| opendevreview | Oleksandr Kozachenko proposed openstack/networking-generic-switch master: Add ArubaOS-CX switch support https://review.opendev.org/c/openstack/networking-generic-switch/+/868598 | 14:02 |
| TheJulia | good morning | 14:09 |
| TheJulia | rpittau: question for you posted to https://review.opendev.org/c/openstack/ironic/+/868521 | 14:12 |
| zigo | rpittau: If I do: | 14:13 |
| zigo | if isinstance(node_info._lock, type(threading.Lock())): | 14:13 |
| zigo | if node_info._lock.is_locked(): | 14:13 |
| zigo | node_info.release_lock() | 14:13 |
| zigo | Then the only errors I get are: | 14:13 |
| zigo | AssertionError: Expected 'release_lock' to be called once. Called 0 times. | 14:13 |
| zigo | (patching ironic_inspector/node_cache.py in release_lock() ) | 14:14 |
| zigo | Does this help? | 14:14 |
| rpittau | zigo: I'm trying different things, also something similar to that, I'm not sure what changed | 14:15 |
| rpittau | TheJulia: ack | 14:15 |
| rpittau | TheJulia: not sure we want to fall back to the copy, the hardlink would've been created at that point, unless we unlink it? | 14:19 |
| TheJulia | I think we ought to | 14:20 |
| TheJulia | I mean... | 14:20 |
| TheJulia | I guess I feel like we've discovered a situation where we know it will fail | 14:20 |
| TheJulia | we're just going to log it presently, which seems... bad | 14:20 |
| TheJulia | we should really just be ensuring that the end goal is reached for the user | 14:20 |
| TheJulia | ... Since not everyone will have logging access | 14:20 |
| TheJulia | At least, that is the internal conflict dialog going on in my head | 14:21 |
| rpittau | it could fail just because restorecon does not exist, but between handling that case or just fall back to copy, maybe the latter is safer | 14:22 |
| TheJulia | well, and restorecon *should* raise OSError if not already caught | 14:23 |
| TheJulia | the challenge here is if it is after the link | 14:23 |
| TheJulia | because copying over a link... | 14:23 |
| * TheJulia wonders if that replaces the file... | 14:23 | |
| * TheJulia files the momentary thought under "unknown behavior" | 14:23 | |
| rpittau | safer to unlink and raise oserror | 14:23 |
| TheJulia | yeah, if it fails. | 14:24 |
| TheJulia | The other thing is maybe we should be doing that *anyway* afterwards | 14:24 |
| * zigo is preparing to file a CVE over a symlink attack... :) | 14:24 | |
| TheJulia | also... it occurs to me that elevated privs are needed to run restorecon | 14:25 |
| zigo | Note: I have no clue what you're discussing, just trying to be funny ... | 14:25 |
| TheJulia | zigo: I figured :) | 14:25 |
| rpittau | zigo: sounds like something coming out from an anime, "symlink attack!" | 14:25 |
| * TheJulia suddenly wonders if she is in a world where a giant robot is about to appear | 14:26 | |
| rpittau | TheJulia: I think in our case we can run it as the dir belongs to the ironic user | 14:26 |
| TheJulia | hmm, yeah that should be the right way, but some people try shared group access | 14:27 |
| TheJulia | dunno | 14:27 |
| TheJulia | there are too many ways to configure it! | 14:27 |
| rpittau | :D | 14:27 |
| rpittau | let me handle the fallback first | 14:28 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix selinux context of published image hardlink https://review.opendev.org/c/openstack/ironic/+/868521 | 14:33 |
| TheJulia | so https://review.opendev.org/c/openstack/ironic/+/869107 can now be abandoned if we can merge https://review.opendev.org/c/openstack/ironic/+/869103 | 14:35 |
| rpittau | yep | 14:36 |
| rpittau | oh forgot to vote there :D | 14:36 |
| rpittau | iurygregory: when you have a minute can you please check https://review.opendev.org/c/openstack/ironic/+/869084 ? | 14:46 |
| iurygregory | rpittau, sure | 14:47 |
| rpittau | thanks | 14:47 |
| * TheJulia pokes tox | 14:54 | |
| TheJulia | there it goes | 14:56 |
| * TheJulia wonders if a faster computer is needed now | 14:56 | |
| slaweq | Hi Ironic cores, can You review https://review.opendev.org/c/openstack/ironic/+/868191 ? Thx in advance | 14:57 |
| TheJulia | +2'ed, it likely won't pass until we get another fix merged first | 14:58 |
| TheJulia | so holding off on workflow | 14:58 |
| TheJulia | iurygregory: a quick review on https://review.opendev.org/c/openstack/ironic/+/869103 would also be appreicated which should allow us to un-wedge the ironic gate | 15:01 |
| TheJulia | also rpittau wrt ^^ that highlights a bug I'm working on fixing which is the possibility of vendor differences and the need to set it on at least per-node | 15:01 |
| rpittau | TheJulia: ack | 15:03 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix grub config path default https://review.opendev.org/c/openstack/ironic/+/869134 | 15:33 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Permit node level override of Grub ESP config path https://review.opendev.org/c/openstack/ironic/+/869230 | 15:33 |
| TheJulia | rpittau: you can single core approve https://review.opendev.org/c/openstack/ironic/+/867228 as it is doc only | 15:36 |
| JayF | There's an email on ML saying to not set py_modules=[] in the setup.py file | 15:51 |
| JayF | I'm pretty sure we already did that... | 15:51 |
| opendevreview | Julia Kreger proposed openstack/ironic master: [DOC] Add entry regarding cleaning+raid https://review.opendev.org/c/openstack/ironic/+/867674 | 15:53 |
| JayF | https://github.com/openstack/ironic/blob/master/setup.py#L21 + https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031658.html = :( | 15:58 |
| rpittau | JayF: we did it in some places as workaround for an issue on unit tests | 16:02 |
| JayF | According to that post; it means you wouldn't be able to import ironic code | 16:04 |
| JayF | I don't have another source for that, but we need to make sure we revert when the pbr fixes go. | 16:04 |
| rpittau | JayF: weird, we put that in place because we were not able to import any module from ironic :) | 16:06 |
| JayF | LOL | 16:06 |
| JayF | I don't know the actual answer; but if that post is wrong we should reply | 16:06 |
| JayF | if it's right, we should revert when PBR is fixed | 16:06 |
| JayF | if it's a grey area then just spin around three times :P | 16:07 |
| * rpittau spin around three times | 16:07 | |
| JayF | whoa; now we're both dizzy! lol | 16:07 |
| rpittau | I sure am | 16:07 |
| rpittau | need to leave now, let's wait for the pbr fix, I'll have a deeper look tomorrow if nothing has changed yet | 16:08 |
| rpittau | have a great rest of the day o/ | 16:08 |
| JayF | I might push up the revert change just to see how CI likes it | 16:08 |
| JayF | but I won't land or pursue landing it until we know what's actually up | 16:08 |
| * TheJulia worries we'll need whiskey or something before the week is out | 16:08 | |
| JayF | I'm lucky in that at least I have a short day. I have live TV to attend! | 16:09 |
| JayF | (wrestling event, and it's live on the east coast so it starts @ 4pm local) | 16:09 |
| TheJulia | enjoy | 16:09 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Temporarily mark redfish vmedia as non-voting https://review.opendev.org/c/openstack/ironic/+/869107 | 16:10 |
| opendevreview | Merged openstack/ironic-python-agent master: modify error word node https://review.opendev.org/c/openstack/ironic-python-agent/+/868669 | 16:12 |
| opendevreview | Merged openstack/ironic master: Create IRONIC_VM_MACS_CSV_FILE if it does not exist https://review.opendev.org/c/openstack/ironic/+/869084 | 16:16 |
| opendevreview | Merged openstack/ironic master: Add doc for lsblk: /dev/nvme0n1p2: not a block device https://review.opendev.org/c/openstack/ironic/+/867228 | 16:17 |
| TheJulia | JayF: we don't need to move it to Non-voting, fixing shim worked | 16:17 |
| JayF | that already landed? wow | 16:17 |
| TheJulia | not quiet yet, but 1x+2 and passed like a charm | 16:18 |
| JayF | mine or someone elses'? | 16:25 |
| JayF | do I need to +2a? | 16:25 |
| TheJulia | someone else's | 16:26 |
| TheJulia | you can +2+a if you like | 16:26 |
| TheJulia | There are a couple follow-up patches upon my realization that "this is broken" | 16:26 |
| TheJulia | one also fixes doc | 16:27 |
| sschmitt | This one is really out of left field, but has anyone ever tried integrating the ovn-bgpagent stuff with switches running EVPN-VXLAN for multi-tenant baremetal networking | 16:29 |
| TheJulia | sschmitt: out of the left star system... I think | 16:30 |
| JayF | I know some of those words ;) | 16:30 |
| sschmitt | someone's gotta ask the hard questions ;) | 16:30 |
| TheJulia | sshnaidm: afaik no, I'm assuming some glue or integration might be needd there | 16:30 |
| TheJulia | since our docs say "use vlans", you can use vxlans if the switch/ml2 driver supports it | 16:30 |
| JayF | We used vxlan managed by nicera for a time at one environment I worked in; but only in some special cases | 16:31 |
| TheJulia | the switch becoming a router there with bgp integration.... and it doing the forwarding, *mindblown* | 16:31 |
| JayF | and IIRC none of that glue code was ever open source (and tbh it wasn't that good of a system either...) | 16:31 |
| JayF | TheJulia: yep, exactly. terminate vxlan at switch; expose vlans to endpoint | 16:31 |
| TheJulia | sschmitt: of course, it might just work, dunno | 16:31 |
| JayF | I bet you need a lot of glue to make it work | 16:31 |
| JayF | but not anything significant | 16:32 |
| sschmitt | thats what I was thinking | 16:32 |
| TheJulia | I think the key is if the bgp agent talks to the switch directly in the vxlan native terms | 16:32 |
| TheJulia | then things should go fairly cleanly, but yeah | 16:32 |
| TheJulia | dunno | 16:32 |
| TheJulia | sschmitt: so it occurs to me that networking-baremetal will not trigger on vxlan ports... but that might be okay | 16:42 |
| sschmitt | We actually have tested with a small change to that and it works | 16:43 |
| TheJulia | oh good | 16:43 |
| TheJulia | feel free to post it :) | 16:43 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch stable/yoga: Fix CI https://review.opendev.org/c/openstack/networking-generic-switch/+/867874 | 16:46 |
| TheJulia | hmm, that wet stuff is falling from the sky again | 16:48 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix selinux context of published image hardlink https://review.opendev.org/c/openstack/ironic/+/868521 | 17:02 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use centos grub artifacts with centos ramdisk for vmedia https://review.opendev.org/c/openstack/ironic/+/869103 | 17:13 |
| TheJulia | ugh | 17:13 |
| TheJulia | UGH | 17:13 |
| JayF | No nasty opcode errors at least | 17:22 |
| JayF | :\ | 17:22 |
| opendevreview | Merged openstack/sushy master: Fix exceeding retries https://review.opendev.org/c/openstack/sushy/+/867675 | 17:24 |
| TheJulia | well, yeah | 17:38 |
| TheJulia | https://www.irccloud.com/pastebin/oPGlfVfF/ | 17:38 |
| TheJulia | that is a boot to ISO image file over http. EFI firmware said "nope!" | 17:40 |
| TheJulia | lets hope it is transient | 17:40 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use centos grub artifacts with centos ramdisk for vmedia https://review.opendev.org/c/openstack/ironic/+/869103 | 17:44 |
| JayF | TheJulia: from your lips to zuul's ears :| | 17:44 |
| JayF | oh, that's the bad notification again | 17:45 |
| JayF | where ARM64 runs, doesn't change V+ vote, and notifies us again | 17:45 |
| JayF | that dang bug fools me everytime :) | 17:45 |
| * TheJulia thinks it is time for another cup of coffee and then to jump back into metrics... | 17:48 | |
| JayF | yeah, I'm looking at hardware enablement track chair stuff (it's me, arne, and dmitry) then I need to pick up on my shards work | 17:49 |
| opendevreview | Merged openstack/ironic-inspector master: SQLAlchemy 2.0 Support https://review.opendev.org/c/openstack/ironic-inspector/+/860731 | 17:54 |
| * TheJulia dances | 17:59 | |
| opendevreview | Merged openstack/networking-generic-switch master: Introduce new option ngs_save_configuration https://review.opendev.org/c/openstack/networking-generic-switch/+/868314 | 18:48 |
| opendevreview | Verification of a change to openstack/networking-generic-switch master failed: Document ngs_trunk_ports and ngs_physical_networks https://review.opendev.org/c/openstack/networking-generic-switch/+/868313 | 18:48 |
| JayF | TheJulia: https://octodon.social/@nivex/109632466961915912 | 19:15 |
| JayF | TheJulia: I think that's another instance of our bug; I already replied to him | 19:15 |
| JayF | (he's an old acquaintance from back in NC; you may have run into them as well) | 19:16 |
| TheJulia | eh, we... do that all the time | 19:17 |
| TheJulia | but that does seem weird | 19:17 |
| JayF | I'm saying, I think that's an instance of invalid opcode | 19:18 |
| JayF | just that ipxe is swallowing it and returning a slightly different err | 19:18 |
| JayF | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1020594 | 19:18 |
| TheJulia | well, the opcode fail we've seen is a cpu halting fault | 19:19 |
| TheJulia | which I don't think ipxe would consume | 19:19 |
| JayF | Unless ipxe was doing some kind of validation | 19:19 |
| JayF | which is possible but much less likely | 19:19 |
| TheJulia | I bet that might be what they are hitting | 19:19 |
| TheJulia | hmm | 19:19 |
| JayF | https://ipxe.org/err/2e0080 | 19:20 |
| TheJulia | url is not accessible | 19:20 |
| JayF | TheJulia: use the debian link I put in here; he screwed up the link in the social post | 19:20 |
| TheJulia | I did from the bug report | 19:21 |
| TheJulia | it is the artifact he linked to | 19:21 |
| JayF | oh; that's almost 10000% private on his network | 19:21 |
| TheJulia | there is really no way to work that without being able to reproduce it and also see what the compiled option flags are in ipxe | 19:21 |
| JayF | int.home.nivex.net | 19:21 |
| TheJulia | yeah | 19:21 |
| JayF | I mean, I buy it's likely unrelated now | 19:21 |
| JayF | but I wanted to check and see | 19:21 |
| TheJulia | yeah | 19:22 |
| TheJulia | and you know... the opcode we got was from grub | 19:22 |
| TheJulia | not ipxe | 19:22 |
| TheJulia | but grub doesn't handle hard failures very well | 19:22 |
| TheJulia | ... partly by design where as ipxe has a bunch of fallback logic | 19:23 |
| opendevreview | Merged openstack/ironic master: Use centos grub artifacts with centos ramdisk for vmedia https://review.opendev.org/c/openstack/ironic/+/869103 | 20:31 |
| JayF | \o/ | 20:46 |
| JayF | Gate is unblocked! | 20:46 |
| TheJulia | woot | 20:55 |
| opendevreview | Merged openstack/sushy master: Fix volume deletion on newer iDRACs https://review.opendev.org/c/openstack/sushy/+/864845 | 21:41 |
| TheJulia | wheee time for a new devstack machine | 22:02 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Get conductor metric data https://review.opendev.org/c/openstack/ironic/+/865447 | 22:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!