| opendevreview | Julia Kreger proposed openstack/ironic master: Replace GreenThreadPoolExecutor in conductor https://review.opendev.org/c/openstack/ironic/+/952939 | 00:15 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: Set the backend to threading. https://review.opendev.org/c/openstack/ironic/+/953683 | 00:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove direct mapping from API -> DB https://review.opendev.org/c/openstack/ironic/+/956512 | 00:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Optional indirection API use https://review.opendev.org/c/openstack/ironic/+/956504 | 00:15 |
| TheJulia | well, this will be interesting! | 00:15 |
| opendevreview | Merged openstack/bifrost master: Move all CS9 jobs to non-voting https://review.opendev.org/c/openstack/bifrost/+/955660 | 00:16 |
| *** janders8 is now known as janders | 01:23 | |
| opendevreview | OpenStack Proposal Bot proposed openstack/ironic-ui master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-ui/+/956624 | 03:25 |
| opendevreview | Morten Stephansen proposed openstack/ironic-python-agent master: Fix for motherboards where efibootmgr returns UTF-8. https://review.opendev.org/c/openstack/ironic-python-agent/+/956068 | 06:58 |
| rpittau | good morning ironic! o/ | 07:08 |
| rpittau | python 3.9 is officially a thing of the past | 07:26 |
| rpittau | hopefully its ghost won't come back hunting for us :) | 07:26 |
| dtantsur | cardoe: I'm very much in favour of one base option for /var/lib/ironic (and one for /run/ironic if needed). I've been thinking about the same when working on the local RPC change. | 08:52 |
| dtantsur | guilhermesp: I really suspect our code does not account for holder partitions in the format of /dev/md/something. I wonder if it's your case. You can confirm by finding the lsblk output in the IPA logs. | 08:56 |
| rpittau | when a core has a moment, can you please review https://review.opendev.org/c/openstack/ironic/+/956701 and https://review.opendev.org/c/openstack/ironic/+/956685 ? thanks! | 10:41 |
| TheJulia | ugh metal3-integration didn't actually log anything aside of the console log | 12:06 |
| TheJulia | :( | 12:06 |
| *** darmach48 is now known as darmach4 | 12:15 | |
| guilhermesp | yeah i was trying to look for it but no ipa logs stores anywhere =( | 12:18 |
| guilhermesp | for now, no way around other then deploy the os in a single disk and try to convert it to software raid at some point | 12:19 |
| dtantsur | guilhermesp: if I create a patch, will you be able to test it? | 12:30 |
| guilhermesp | i think i can! this deployment has some spare nodes i could test this out, but since ppl are in a rush, i have to move forward with the prod nodes and asked them to just get rid of all vrocs lol | 12:35 |
| dtantsur | I see, let me cook something testable | 12:36 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent master: WIP allow holder devices with slashes https://review.opendev.org/c/openstack/ironic-python-agent/+/956782 | 12:37 |
| dtantsur | guilhermesp: try this ^^ | 12:37 |
| guilhermesp | cool cool -- i have some meetings booked this morning and need to wait ppl to come in to see if they can get vroc on one of the spare nodes to get this in. will keep you posted dtantsur ! thx | 12:40 |
| dtantsur | np. if you can enable collecting ramdisk logs, it will help | 12:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove direct mapping from API -> DB https://review.opendev.org/c/openstack/ironic/+/956512 | 13:05 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Optional indirection API use https://review.opendev.org/c/openstack/ironic/+/956504 | 13:05 |
| TheJulia | on a plus side, it looks like the metal3 got close to actually working with the last patch, which makes me wonder. Would we be okay with disabling the metal3 job temporarily to merge the "this will definitely break" patches and then merge in the fixing patches after, or do we feel the need to stack the order to avoid the CI job config change overall. (That is, at the cost of rebasing the order) | 13:08 |
| dtantsur | TheJulia: it's not just the job though, you're going to break the whole metal3 project, no? | 13:08 |
| TheJulia | For the time between the patches, if they are consuming master, yes. If we can keep that time narrow and land in rapid succession, they may never notice | 13:09 |
| TheJulia | adam-metal3: you may care about this conversation, fyi | 13:09 |
| TheJulia | (as an aside, unrelated to metal3, we need to try and fix grenade in the next week or so) | 13:10 |
| dtantsur | We don't rebuild images on every commit, so there is a chance it goes unnoticed | 13:11 |
| rpittau | I have a release planned in metal3 for ironic-image, I will have to skip the broken jobs I guess | 13:12 |
| TheJulia | I guess, at least trying to get things in a happy state, I sort of like the "oh, I broke it and i'm fixing it" approach just from the standpoint that right now the switch to threading change doesn't actually log anything except the console log :( | 13:12 |
| adam-metal3 | TheJulia, yeah we only build when we merge to ironic-image repo on our end | 13:13 |
| TheJulia | regardless, I'm under no illusions, this is sort of a both big and small changes at the same time. | 13:14 |
| adam-metal3 | there was an upstream breakage yesterday anyhow I heared the solution was discussed on our community meeting, so folks on our end anyhow very aware the main can break any time :D | 13:15 |
| adam-metal3 | I will let the community know if you ping me when it starts | 13:16 |
| adam-metal3 | I mean the period of potential instability | 13:16 |
| TheJulia | That could definitely work. I think the biggest issue we're going to create is the move from greenthreading to real threads is going to very much be a moment where folks may blink | 13:17 |
| TheJulia | (different runtime process model and related memory footprint (and very different VSS since threads) | 13:18 |
| TheJulia | err, vsz | 13:19 |
| TheJulia | I keep thinking RSS and thinking it should be VSS as a result | 13:19 |
| TheJulia | rpittau: so realistically, I'm sort of hoping to make progress in the next couple of weeks. We're in the sort of time where we're going to need to release the next version of ironic for the end of cycle sometime in the next 4-ish weeks it looks like | 13:22 |
| dtantsur | let's not break metal3 until rpittau does the release :) | 13:24 |
| TheJulia | I did talk to JayF and his feeling was that it might be that the ironic community might need to merge the changes around removing eventlet and then spend a weke or two kicking the tires | 13:24 |
| * dtantsur nods | 13:24 | |
| TheJulia | rpittau: when will that be? | 13:24 |
| rpittau | TheJulia: the release should happen a soon as the 2 patches I mentioned before merge :D | 13:25 |
| TheJulia | ENOCONTEXT | 13:25 |
| rpittau | https://review.opendev.org/c/openstack/ironic/+/956701 and https://review.opendev.org/c/openstack/ironic/+/956685 | 13:25 |
| rpittau | the release is based on the new bugfix branch | 13:25 |
| TheJulia | oh, the two you asked about 2.75 hours ago? | 13:26 |
| rpittau | yeah :D | 13:26 |
| TheJulia | ok | 13:26 |
| TheJulia | done and done | 13:26 |
| rpittau | thanks | 13:26 |
| opendevreview | Merged openstack/ironic-ui master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-ui/+/956624 | 13:28 |
| TheJulia | dtantsur: btw, if you didn't notice, I got object indirection to happily work over jsonrpc :) | 13:31 |
| TheJulia | (actually, worked out of the box) | 13:32 |
| dtantsur | wow, sweet! | 13:32 |
| dtantsur | no, I haven't noticed, still overwhelmed by conversations since I'm back | 13:32 |
| TheJulia | no worries, I had to excise the direct database calls out of the API, but I kept it lightweight | 13:32 |
| TheJulia | so no intermediate object stop, just get dicts and that is the way the api was written to use the responses anyhow. | 13:33 |
| dtantsur | interesting | 13:35 |
| TheJulia | the indirection stuffs only works for oslo versioned objects or dicts | 13:35 |
| TheJulia | not, collections, or sqlalchemy objects | 13:35 |
| TheJulia | and we did both while trying to keep that extra object conversion overhead out of the mix | 13:36 |
| TheJulia | I've not put a "no, you can't talk to the DB flag" in, and in single process mode it would be super hard to do like Nova did for their components which do indirection, but I do detach api.request.dbapi | 13:37 |
| TheJulia | so it doesn't become an appealing thing to try and use. | 13:38 |
| dtantsur | :D | 13:38 |
| opendevreview | Merged openstack/ironic bugfix/31.0: Comment out metal3 integration job https://review.opendev.org/c/openstack/ironic/+/956685 | 13:39 |
| TheJulia | we could just entirely excise the idea, I don't think any usage of api.request.dbapi even exists after my change | 13:42 |
| TheJulia | (but if folks have patches or mods downstream, that is likely super breaky( | 13:43 |
| opendevreview | Merged openstack/ironic bugfix/31.0: Fix local RPC IPv6 detection to use socket binding instead of file checks https://review.opendev.org/c/openstack/ironic/+/956701 | 13:51 |
| TheJulia | oooh ahh, it looks like metal3 works with indirection | 14:24 |
| TheJulia | err, failing, but more data I guess | 14:30 |
| opendevreview | Jakub Jelinek proposed openstack/ironic-python-agent master: Fix skip block devices for RAID arrays https://review.opendev.org/c/openstack/ironic-python-agent/+/937342 | 14:33 |
| kubajj | If anybody had a minute, we are working on fixing the skip_block_devices property for RAIDs. We managed to fix the cleaning for the scenario where the arrays span the whole disk, but still need to fix the scenario where the disks are shared between arrays. Should we add it to the same change or create a follow-up? | 14:38 |
| dtantsur | I think I need more context. Is it much harder to fix the 2nd part as well? | 14:48 |
| rpittau | TheJulia: I started the release procedure for ironic-image but I doubt it will be all done before tomorrow | 14:50 |
| rpittau | I guess we can stay broken as the only PR needed is for the release notes and we don't really care about CI runs, wdyt dtantsur adam-metal3 ? | 14:50 |
| dtantsur | rpittau: I'm less worried about the CI and more about the actual ironic-image working | 14:50 |
| kubajj | dtantsur: we are working on it, hopefully won't take almost a year like the first part, lol | 14:50 |
| rpittau | dtantsur: release branch has been cut from bugfix-31.0 so new changes on master won't impact it | 14:51 |
| dtantsur | cool! | 14:52 |
| TheJulia | Looks like were close on metal3, just need to resolve one more nested object which can't be serialized over rpc | 14:59 |
| TheJulia | at least, that is what is front and center issue wise right now | 15:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove direct mapping from API -> DB https://review.opendev.org/c/openstack/ironic/+/956512 | 15:01 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Optional indirection API use https://review.opendev.org/c/openstack/ironic/+/956504 | 15:01 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: grenade: restart neutron-rpc-server https://review.opendev.org/c/openstack/ironic/+/956801 | 15:36 |
| TheJulia | it passed, muahahhaha | 16:33 |
| TheJulia | does the metal3-integration job stack patches or only pull the latest patch? Asking because it appears that its trying to run with eventlet which is removed in a prior patch.... | 16:41 |
| TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a1a/openstack/a1a36f27076d4063b50d36c4cc5f61d1/controller/before_pivoting/ironic.log | 16:42 |
| TheJulia | yup, the build process only pulls the ref, not the stack of them :\ | 17:07 |
| TheJulia | https://github.com/metal3-io/ironic-image/blob/main/ironic-packages-list#L3-L9 | 17:08 |
| JayF | makes sense, conceptually in github the ref has the stack. easy thing to overlook :( | 17:08 |
| TheJulia | yup | 17:08 |
| TheJulia | that makes me super nervious, but also means I think the only way we're going to get a solid result is to go ahead and start moving down the path of merging | 17:09 |
| JayF | I have a mild suggestion, if you think it's worth it | 17:10 |
| JayF | push a DNM squashed version of the stack to get a CI run | 17:11 |
| TheJulia | would really be the only way to do it | 17:11 |
| TheJulia | likely also the easiest path to get clarity before we're "ready ready" | 17:12 |
| JayF | or fix ironic-image | 17:13 |
| * JayF touches his nose | 17:13 | |
| TheJulia | it should just be changing the conditional check, I think. But I'm also not really equipped with the spoons to drag this out when I really need to cycle/loop as close to the issue as possible to get a yay/nay level of confidence | 17:17 |
| JayF | ++ | 17:18 |
| TheJulia | but hey, on a plus side, tossing it all over to indirection worked so... \o/ | 17:19 |
| TheJulia | Looking at some other job logs, looks like there *might* be slight unhappienss in everything around allocations, but the tests still pass and the job which uses it still works just fine. Looks like it was a race between threads and one realized 'oh, I've got some stale data' | 17:39 |
| TheJulia | so, really more threading than anything else | 17:39 |
| TheJulia | JayF: w/r/t the question your, I think the idea will be to eventually just detach the dbapi out from the api, but that is also more something we need to telegraph: https://review.opendev.org/c/openstack/ironic/+/956504/8/ironic/api/hooks.py - Eventually just remove the dbhook might be the goal. | 17:46 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM - Test change for Metal3 for threading changes https://review.opendev.org/c/openstack/ironic/+/956820 | 17:54 |
| * TheJulia crosses fingers | 17:55 | |
| alegacy | Has anyone had success running bifrost on a CentOS VM with python3.12? I had been running with python3.9, but NGS now requires >=3.10 so I upgraded my install and now I'm having all sorts of grief with bifrost not finding packages that it needs (one such pkg is 'firewall'). | 18:03 |
| TheJulia | alegacy: I'd assume all ironic related repos have a similar issue, but I've not tried on bifrost with py3.12 specifically since I've been so focused on eventlet stuffs recently | 18:21 |
| alegacy | TheJulia: I'm rebuilding my VM now just in case something went sideways... | 18:23 |
| TheJulia | https://review.opendev.org/c/openstack/bifrost/+/955181 merged yesterday, so it might just be timing and the status of your local bifrost branch | 18:59 |
| JayF | TheJulia: [not urgent] I was looking with Adam at CI failures in gophercloud/github actions running Ironic. It's failing on https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L2583 which was added, I think, to enable vmedia with a "Nexthop has invalid gateway.". If you have any quick insights, it'd be nice to hear em. I suspect it's something environmental and am trying to "hold" a github runner. | 19:11 |
| JayF | My current hypotheses are that neutron/ovs/whoever didn't do the work properly to setup that subnet, or there's some weirdness (ip overlap? security limitations?) in the environment causing the issue. | 19:12 |
| JayF | although I wonder if it's racing with neutron somehow (I guess that'd be adjacent to the first hypothesis) | 19:13 |
| alegacy | TheJulia: Maybe I'm going about this wrong? I changed the symlinks for /bin/python3* to point to their python3.12 equivalents. That seems to move things along, but then firewalld won't start because it can't find the dbus library as there's no dnf package equivalent for 3.12... installing it manually via pip doesn't help since firewalld starts with a "python3 -s" which excludes /usr/local/... from the sys.path | 19:30 |
| alegacy | i think part of the difficulty is that bifrost is hardcoded to build the venv from the system path of python3 | 19:46 |
| alegacy | changing the system interpreter is not trivial and shouldn't necessarily be tied to what you want to use in the venv. | 19:46 |
| alegacy | i'll try changing bifrost to accept an override and see if that works better to decouple it from the system interpreter. If so I'll propose a patch. | 19:47 |
| TheJulia | is 3.10 available? I don't have a centos9 machine in front of me | 19:51 |
| alegacy | doesn't appear to be. 3.11 and 3.12 are there. | 20:00 |
| frickler | from what I heard, you can only use system python ie py39 for this kind of things. so centos 9 is out, need to use centos 10 | 20:30 |
| TheJulia | rpittau is likely the best person to answer/discuss/engage on this, but he is also likely asleep at the moment | 20:49 |
| guilhermesp | dtantsur: btw i still didnt get a chance to test your pr... it was a very busy day, i will try asap and keep you posted o/ | 21:08 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Clarify testenv and install usage order in testenv.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 21:19 |
| TheJulia | so good news is that it looks like ironic + metal3 might be fine.... except the bad news, it looks like the job will need more ram. There appears to be zero swap as well so the memory footprint caused it to hit the edge. I remember it being pretty close. Anyhow I'll look at changing the job to use a bigger node tomorrow. | 22:29 |
| TheJulia | https://d9d62a197302b635235c-e6ddc7380eb470b6a5615640c6b2397b.ssl.cf5.rackcdn.com/openstack/2289c36ec84e4686882f97aef45672b7/controller/index.html | 22:29 |
| JayF | maybe a good opportunity to see if we can find a knob to make ram usage profile more similar? | 22:34 |
| TheJulia | already went down that path | 22:35 |
| TheJulia | We functionally go from 1 -> 3 processes (1 master, 2 workers) and with threads. We already tune the thread sack way down, almost to as low as it can tolerate to still run. | 22:36 |
| * TheJulia goes to order an espresso martini | 22:36 | |
| JayF | ack | 22:38 |
| JayF | this is really going to be an annoying release for operators :( | 22:38 |
| JayF | especially container-deployers who limit their resources | 22:38 |
| TheJulia | yup | 22:39 |
| janders | TheJulia I created the bugs for the servicing issue we discussed: https://bugs.launchpad.net/ironic/+bug/2119989 https://bugs.launchpad.net/ironic/+bug/2119990 - let me know if this is good enough or if I need to refine these further | 22:40 |
| janders | thank you for looking into this | 22:41 |
| TheJulia | janders: thanks | 23:04 |
| TheJulia | Julia was not kidding… https://usercontent.irccloud-cdn.com/file/Ejq4LyHC/1754608383.JPG | 23:13 |
| janders | TheJulia thank YOU! | 23:23 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
