| opendevreview | Anishere Mariam proposed openstack/ironic master: Add node history state timestamps https://review.opendev.org/c/openstack/ironic/+/982555 | 00:00 |
|---|---|---|
| opendevreview | Kaifeng Wang proposed openstack/ironic-tempest-plugin master: Allow removing volume connectors on some states https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/981859 | 01:38 |
| opendevreview | David Nwosu proposed openstack/ironic master: Add global node history endpoint https://review.opendev.org/c/openstack/ironic/+/983144 | 07:50 |
| opendevreview | David Nwosu proposed openstack/ironic master: Add global node history endpoint https://review.opendev.org/c/openstack/ironic/+/983144 | 08:06 |
| opendevreview | Abhishek Bongale proposed openstack/ironic master: Limit interfaces to fake-graphical for testing https://review.opendev.org/c/openstack/ironic/+/982270 | 08:07 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Use startswith for interface validation in tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/982960 | 08:28 |
| opendevreview | Abhishek Bongale proposed openstack/ironic-tempest-plugin master: Add conditional graphical console validation https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/982077 | 08:28 |
| VictorAkoh[m] | hroy: Hello! I trust you're good. Were you able to check it for me, please? | 08:29 |
| opendevreview | Pierre Riteau proposed openstack/networking-generic-switch stable/2026.1: CI: Remove setuptools pin https://review.opendev.org/c/openstack/networking-generic-switch/+/983521 | 09:52 |
| victor_obi_ | TheJulia, JayF: Hi, working on bug #2097102. Reviewers noted that conductor_affinity and _filter_by_conductor have different semantics. Is there a recommended approach to push conductor filtering to the DB layer while preserving hash ring semantics? | 11:42 |
| TheJulia | good morning everyone | 13:14 |
| cardoe | down in the weeds of devstack | 13:39 |
| cardoe | Gotta update our docs or find default config | 13:39 |
| TheJulia | Regarding victor_obi_'s question, I'm not sure there really is :( It feels like we likely need to quantify and have a bug that we just don't really understand completely becuase of perception bias and we also don't have solid data beyond kaifeng's bug report | 13:44 |
| TheJulia | And I think we might need to change the meaning, slightly in that case, dunno. | 13:44 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove IRONIC_RAMDISK_TYPE devstack setting https://review.opendev.org/c/openstack/ironic/+/983490 | 13:52 |
| TheJulia | cardoe: are you referring to https://review.opendev.org/c/openstack/ironic/+/983490/2/doc/source/contributor/devstack-guide.rst ? | 14:05 |
| TheJulia | or other issues? | 14:05 |
| cardoe | Other issues. I used the local.conf from there. I removed tinyipa | 14:05 |
| cardoe | So network segment range plugin isn't enabled by default. I've enabled it. But my vxlan doesn't have an allocation range that it can use. | 14:09 |
| TheJulia | OH | 14:18 |
| TheJulia | so you tried vlxan with a skelaton of the settings from the guide? | 14:18 |
| TheJulia | use the CI job local.conf, it will make you happier, so much happier | 14:18 |
| JayF | after you fix the documented one though, please :D | 14:18 |
| JayF | I use the local.conf from devstack guide, but usually carry forward a working local.conf for a while | 14:19 |
| TheJulia | well, its a mixed state, he is pulling in other services which require extra config outside of the base example | 14:19 |
| JayF | I don't have a working one r/n but that doc is how GR-OSS people use it | 14:19 |
| JayF | aha | 14:19 |
| TheJulia | There is already a change pending with the docs being fixed inside | 14:19 |
| JayF | yeah more services I can't make promises, only so much [stuff] can fit in a bucket ;) | 14:19 |
| TheJulia | yeah | 14:19 |
| TheJulia | There is now a bare metal bear doing disco in the slide deck | 14:48 |
| JayF | Do you think there's value in having like a 5-minute here's where ironic fits in the stack as an intro to that update before I start emceeing in all of the individual updates? | 14:56 |
| TheJulia | I think that brings value, yes | 14:56 |
| TheJulia | I've been on this theme of "over communicate; just don't assume people know what your talking about" kick as of recent | 14:57 |
| TheJulia | We likely ought to also point back to launchpad at the very end | 14:57 |
| opendevreview | Pavlo Shchelokovskyy proposed openstack/ironic master: Only label kernel-less ISO as config drive https://review.opendev.org/c/openstack/ironic/+/983576 | 14:59 |
| JayF | Yeah it makes sense. I will get some good slides wrapping the start and end today | 14:59 |
| TheJulia | I'm already on the end | 15:01 |
| TheJulia | and at least, a little bit of artwork | 15:01 |
| TheJulia | "artwork" | 15:01 |
| * TheJulia walks the canine overlord | 15:01 | |
| TheJulia | err, corgi. Its a corgi, corgis are corgis, there is no alternative to corgis | 15:02 |
| TheJulia | Just so we're all on the same page: JayF is being the host/introducer/closer. alegacy is doing standalone netowrking, clif is doing TBN, I'm doing vxlan, cardoe is doing other additions. Jay do you want to do deprecations as well? | 15:22 |
| JayF | yeah | 15:33 |
| cardoe | I guess I need to flesh out the other additions slides. | 15:38 |
| clif | sounds good to me | 15:38 |
| cardoe | TheJulia, JayF: I'm not pulling in any other services. I literally just tried to use the local.conf from the devstack guide. | 15:39 |
| cardoe | I replaced "ipmi" with "redfish" globally. | 15:39 |
| JayF | the ironic w/nova config there is the one I usually use | 15:39 |
| JayF | no don't do that | 15:39 |
| JayF | there's one place to change, with a comment on it, to change to redfish | 15:39 |
| JayF | only change that | 15:39 |
| TheJulia | use of redfish requires the sushy-tools repo afaik | 15:39 |
| cardoe | I said "globally" I mean I read the comment and did the change the comment said. | 15:40 |
| JayF | ack good | 15:40 |
| JayF | then IDK why it would boom, I am in meetings all morning but can help if there's a specific error/stuck point | 15:40 |
| cardoe | It came up just fine but then ate itself on tinyipa. | 15:40 |
| cardoe | Which Julia is fixing. | 15:41 |
| JayF | ah, so IRONIC_RAMDISK_TYPE=dib would fix it too | 15:41 |
| JayF | good | 15:41 |
| cardoe | I deleted the line and ran ./stack.sh again and it's BOOM | 15:41 |
| cardoe | Error while executing command: HttpException: 503, Unable to create the network. No project network is available for allocation. | 15:41 |
| cardoe | ++lib/neutron_plugins/services/l3:create_neutron_initial_network:202 oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id | 15:41 |
| JayF | you're going to need to backtrack from there to find why it failed | 15:42 |
| JayF | 503 there implies a bad config in a service perhaps | 15:42 |
| JayF | Is this 100% upstream or using some patched repos? | 15:42 |
| opendevreview | David Nwosu proposed openstack/ironic master: Add project field to node history and record deployment transitions https://review.opendev.org/c/openstack/ironic/+/983349 | 15:43 |
| cardoe | https://www.irccloud.com/pastebin/GqppFXAZ/ | 15:49 |
| cardoe | No patched repos. I literally git cloned devstack. Had this diff (without the tinyipa removal line). Ran ./stack.sh and it was up and all that. I tried to do something but tinyipa was problematic. I mentioned that here. TheJulia gave a patch to remove the tinyipa line. So I removed it locally and ran ./stack.sh again. And I'm getting the 503. | 15:50 |
| cardoe | I've tried to run unstack.sh and then stack.sh again and same issue. | 15:50 |
| JayF | That config should work, although I'll note unstack/stack with Ironic doesn't always work especially if the local.conf changes or the stack itself was busted the first time | 15:51 |
| TheJulia | it won't work without sushy-tools being pulled in afaik | 15:51 |
| JayF | If it's broken it's probably a regression; someone fixed something in CI but not in the devstack docs | 15:51 |
| JayF | TheJulia: The code pip-installs sushy-tools from pypi if it's not in required-projects IIRC. If that changed it would break things here. | 15:51 |
| TheJulia | I'm not sure it really does, but requires taking a look. i.e. its easy to bikeshed this issue and someone just needs to give it as spin/repro/fix | 15:52 |
| JayF | AFAICT there is no reason the config as shown by cardoe shouldn't work which implies to me he may be hitting a local issue or a new bug :( (new bug :CCCC) | 15:53 |
| TheJulia | cardoe: trying to remember but I think that is when the actual allocation fails out for segmenting which makes me think $something else is needed config wise | 15:53 |
| cardoe | That's what I think. There's a chunk of config missing. | 15:53 |
| cardoe | So now locally I've edited neutron.conf and added service_plugins += network-segment-range | 15:54 |
| TheJulia | oh, you need "segments" | 15:54 |
| TheJulia | maybe? | 15:54 |
| cardoe | Well no. You automagically get segments always. | 15:54 |
| JayF | My main question would revolve around why is this broken in devstack-local but not CI | 15:54 |
| JayF | it implies that we changed a thing to fix CI, but I don't think we did | 15:54 |
| TheJulia | uhhh, I haven't I don't think | 15:55 |
| cardoe | the segments plugin just enables the ability to query the segments API endpoints | 15:55 |
| cardoe | But when I enable network-segment-range, there's no available ranges to allocate from. | 15:55 |
| cardoe | Which is where my 503 is happening. | 15:55 |
| cardoe | I dunno zuul well enough to understand how it's local.conf in CI is being built up. | 15:58 |
| cardoe | I had a VM I built forever ago with devstack and have been using it. | 15:58 |
| JayF | look for the .local.conf.auto in the CI job | 15:58 |
| JayF | that's basically a rendered local.conf that has more detail than you should need locally but could still contain useful bits | 15:59 |
| cardoe | But I'm a derp and I nuked it. So just trying to build it up again using the docs. I previously used the docs which is why I submitted some fixes a long while back. | 15:59 |
| JayF | yeah I think it's extremely extremely important for that local.conf in docs to work | 16:00 |
| JayF | so anything we can do to make it go would be appreciated | 16:00 |
| JayF | that's the discovarble "one machine ironic" new developers go to | 16:00 |
| cardoe | The issue is that "initialize_network_segment_range_support()" is not being called within Neutron. | 16:02 |
| TheJulia | disjointed network config? | 16:03 |
| cardoe | I'm fairly certain the reason why this worked the first time around is cause the DB didn't exist and when it was initially populated it triggers another code path. | 16:03 |
| TheJulia | did you run clean to wipe it all out? | 16:03 |
| cardoe | Well they've got like 3 different ways to possibly initialize it. | 16:03 |
| TheJulia | That is one thing I've always done | 16:03 |
| cardoe | I can. I just thought that changing a setting and running ./stack.sh should work. | 16:04 |
| cardoe | Like it should just switch me to dib from tinyipa. | 16:04 |
| TheJulia | If the destination file names are the same, I think the plugin will skip the download again if the files are already there | 16:05 |
| cardoe | I'm not even getting there again though. | 16:06 |
| cardoe | Somewhere the neutron DB is getting wiped in the ./stack.sh call. | 16:09 |
| cardoe | Cause manually telling it to use network-segment-range in /etc/neutron/neutron.conf and systemctl restart devstack@neutron-api.service results in stuff working again. | 16:09 |
| cardoe | But running ./stack.sh again turns off network-segment-range and the segment allocations DB table is empty again resulting in the 503. | 16:10 |
| cardoe | Testing out 'Q_SERVICE_PLUGIN_CLASSES=network_segment_range' in my local.conf | 16:15 |
| cardoe | Well that got me farther. | 16:27 |
| cardoe | ++/opt/stack/ironic/devstack/lib/ironic:create_bridge_and_vms:2577 sudo ip route add 10.0.6.0/24 via 172.24.5.250 dev enp3s0 onlink | 16:27 |
| JayF | I think ./unstack ./stack is hurting you | 16:27 |
| JayF | a clean VM might help | 16:27 |
| cardoe | RTNETLINK answers: File exists | 16:27 |
| cardoe | JayF: it was literally a fresh VM yesterday. | 16:27 |
| cardoe | Ran ./stack.sh using the local.conf from the docs. But it used tinyipa so I just tried to switch to dib. So I deleted the line I was told and ran ./stack.sh again. | 16:28 |
| TheJulia | this is why I always run ./clean.sh | 16:28 |
| cardoe | I'm running ./clean.sh now. | 16:29 |
| JayF | yeah I think that's when things went sideways; it broke the first time and you are trying ont he same VM | 16:30 |
| JayF | any devstack stack failure for me === kill the VM with fire | 16:30 |
| cardoe | Stacking again. | 16:32 |
| cardoe | I can. It just shouldn't be that bad. | 16:33 |
| opendevreview | Victor proposed openstack/ironic master: Fix API pagination limit for conductor filtering https://review.opendev.org/c/openstack/ironic/+/983449 | 16:33 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Migrate package metadata from setup.cfg to pyproject.toml https://review.opendev.org/c/openstack/ironic-python-agent/+/982864 | 16:34 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Update hacking to 7.0.0 https://review.opendev.org/c/openstack/ironic-python-agent/+/982867 | 16:34 |
| cardoe | same failure in ip route add... I don't see us ever cleaning that up. | 16:38 |
| TheJulia | the virtual interface is likely not removed in that case. you can likely manually clean it but removal or recation is kind of dubious since it likely doesn't matter if it already exists | 16:39 |
| JayF | or make the add idempotent | 16:40 |
| cardoe | Well "replace" is the idempotent version of add. | 16:42 |
| cardoe | We use "replace" in almost all the other calls. | 16:42 |
| JayF | I'm trying to remember if `ip r r` errs if it doesn't exist | 16:42 |
| JayF | if not, you may have a winner | 16:42 |
| cardoe | TheJulia: well it errored on that route even after running ./clean.sh | 16:45 |
| TheJulia | so yeah, lets fix the call or remove it ? | 16:46 |
| opendevreview | David Nwosu proposed openstack/ironic master: Add global node history endpoint https://review.opendev.org/c/openstack/ironic/+/983144 | 16:46 |
| * cardoe table flips. | 16:47 | |
| cardoe | https://www.irccloud.com/pastebin/Dz1WdMgF/ | 16:47 |
| cardoe | I dunno why it's initializing sometimes and not others. | 16:48 |
| cardoe | That's after a clean.sh. It's almost like a race condition. | 16:48 |
| cardoe | Which is mentioned in the driver code... | 16:48 |
| TheJulia | if tenant_network_type is not set, its geneve by default so not really sure of the entire context of where your at | 16:48 |
| TheJulia | the plugin otherwise generally works on provider networks as well, the downside is the tenant_network_type takes over for default private network creation in the neutron plugin | 16:49 |
| cardoe | uhh yeah there's some race. | 16:51 |
| TheJulia | Yes, inherent race due to async plugin execution | 16:51 |
| cardoe | neutron-api is populating it... then there's DB errors with MySQL while it's running "init_neutron" and allocations are empty. | 16:52 |
| cardoe | after the DB errors | 16:52 |
| TheJulia | ugh | 16:52 |
| cardoe | restarting neurton-api makes the neutron API work again. | 16:53 |
| cardoe | So something is bad touching the DB in the middle. | 16:53 |
| TheJulia | I encountered this exact same issue at one point.... | 16:53 |
| TheJulia | uhhhhhh | 16:53 |
| TheJulia | It might have been intermittent on my machine | 16:53 |
| cardoe | I'm about to throw out my stupid idea of "this should be repairable" and just start with a fresh VM | 16:54 |
| JayF | I think your idea is solid | 16:54 |
| JayF | you're just running into the same wall others with the same mind have hit | 16:54 |
| JayF | I don't disagree that stack/unstack/clean should all work forever on a good VM, it's just never really worked more than a little while consistently | 16:54 |
| cardoe | So I'm just trying to test my migrations for the runbook traits. | 16:55 |
| cardoe | And then also try to work out the changes I've got for devstack. | 16:55 |
| cardoe | JayF: I changed the wording on https://review.opendev.org/c/openstack/python-ironicclient/+/981805 btw | 16:56 |
| cardoe | https://review.opendev.org/c/openstack/devstack/+/983361 is what I'm trying to change for devstack but it's broken on neutron... neutron is why I'm trying to change it. | 16:57 |
| -opendevstatus- NOTICE: Load on the opendev.org Gitea backends is under control again for now, if any Zuul jobs failed with SSL errors or disconnects reaching the service prior to 16:15 UTC they can be safely rechecked | 17:03 | |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Remove iLO hardware type and interfaces https://review.opendev.org/c/openstack/ironic/+/982706 | 17:12 |
| *** dmellado471 is now known as dmellado47 | 17:48 | |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Enforce TLS 1.2 minimum across all connections https://review.opendev.org/c/openstack/ironic-python-agent/+/982174 | 18:56 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Upgrade TLS certificates to P-384 curve with configurable support https://review.opendev.org/c/openstack/ironic-python-agent/+/982175 | 19:27 |
| cardoe | TheJulia: https://bugs.launchpad.net/neutron/+bug/2114451 I think we're able to do this with the BindLevel right? | 20:21 |
| TheJulia | I mean, At least with our part of the binding code I I'm unsure it really matters | 20:40 |
| TheJulia | because we're not treating segments as spine/leaf subnets | 20:40 |
| TheJulia | we're treating segments as intermediary networks which are attached at different binding levels | 20:40 |
| TheJulia | so, does it really matter when its the L2 domain. | 20:40 |
| opendevreview | Victor proposed openstack/ironic master: Fix API pagination limit for conductor filtering https://review.opendev.org/c/openstack/ironic/+/983449 | 20:59 |
| cardoe | self flogging with devstack complete | 21:25 |
| cardoe | change ip route add -> ip route replace. enable network-segment-range plugin. Are the two changes I needed from our documented local.conf but that's not all | 21:26 |
| cardoe | you must manually systemctl stop devstack@neutron-api.service and devstack@g-api.service | 21:26 |
| cardoe | We never teardown that route even with a ./clean.sh | 21:28 |
| cardoe | So the replace might be wrong-ish. | 21:29 |
| TheJulia | sounds like there is something which doesn't occur in devstack in ci, but occurs locally which may be an extra variable since we're generally not having to restart neutron today | 21:43 |
| JayF | TheJulia: clif: others; I've modified the slides in the ironic oi live -- I think we're good to go. I'm assuming I'm handling the "misc" feature portions and transitioning myself in between all speakers to help draw the through line | 21:50 |
| cardoe | TheJulia: it’s totally a race with neutron. If it’s running and it restarts it then it ends up with the VLAN and VXLAN allocations empty. If it’s stopped first then the wipe happens before it starts it up. | 22:58 |
| cardoe | It also only happens on my shiny box. My dusty box it doesn’t happen on. | 22:58 |
| cardoe | Dusty is still a Zen 5 box. | 23:00 |
| TheJulia | fun! | 23:05 |
| TheJulia | :( | 23:05 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!