| zigo | Is ironic-inspector aware of iptables-nft ?!? | 09:02 |
|---|---|---|
| zigo | Or should I set iptables-legacy as default in my Ironic boxes? | 09:02 |
| zigo | Even with iptables-legacy, I get: | 09:28 |
| zigo | DEBUG ironic_inspector.pxe_filter.iptables [-] Ignoring failed iptables ('-D', 'INPUT', '-i', 'ens3', '-p', 'udp', '--dport', '67', '-j', 'ironic-inspector'): Unexpected error while running command. | 09:28 |
| zigo | Command: sudo /usr/bin/ironic-inspector-rootwrap /etc/ironic-inspector/rootwrap.conf iptables -w -D INPUT -i ens3 -p udp --dport 67 -j ironic-inspector | 09:28 |
| zigo | Exit code: 2 | 09:28 |
| zigo | Stdout: '' | 09:28 |
| zigo | Stderr: "iptables v1.8.9 (legacy): Couldn't load target `ironic-inspector':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n" _iptables /usr/lib/python3/dist-packages/ironic_inspector/pxe_filter/iptables.py:188 | 09:28 |
| TheJulia | good morning | 13:59 |
| TheJulia | zigo: aware, no | 14:00 |
| TheJulia | hmm.... I think hjensas might have an idea, unfortunately I don't remember the filter implementation details | 14:00 |
| hjensas | zigo: iptables-nft should be a fully compatible drop in replacement of iptables legacy. i.e there should be no reason to use legacy. | 14:24 |
| hjensas | zigo: do you see that "no such file or directory" only on start up? Or does it show up every time the filter does a change and re-sync? | 14:25 |
| TheJulia | The issue seems to be the target was never loaded | 14:47 |
| * TheJulia wonders if we're even going to have quorum for a meeting today | 14:48 | |
| zigo | hjensas: Every time. | 14:48 |
| JayF | o/ | 15:01 |
| JayF | #startmeeting ironic | 15:01 |
| opendevmeet | Meeting started Mon May 1 15:01:53 2023 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:01 |
| TheJulia | o/ | 15:01 |
| JayF | Who all is here today? | 15:02 |
| TheJulia | I don't think we have quorum | 15:02 |
| arne_wiebalck | o/ | 15:02 |
| TheJulia | well, maybe we do... | 15:02 |
| JayF | I'm going to do this meeting in ultra-speed mode | 15:02 |
| JayF | and not give a lot of pauses for input | 15:02 |
| JayF | just like o/ or similar if you wanna slow me down | 15:02 |
| JayF | no need in wasting our time if there's only three of us :) | 15:02 |
| JayF | #topic Announcements/Reminder | 15:02 |
| JayF | #note As always, please hashtag any patches ready for review with ironic-week-prio, and review using https://tinyurl.com/ironic-weekly-prio-dash | 15:03 |
| JayF | #note Please avoid running a `recheck` command bare, without any other comments. Obviously, best case is to troubleshoot and fix an issue in CI, but in case of some ephemeral failure, please just note that -- e.g. `recheck jobname failed`. This is tracked at an OpenStack level (https://etherpad.opendev.org/p/recheck-weekly-summary) and I've noticed more contributors doing | 15:03 |
| JayF | bare rechecks. | 15:03 |
| JayF | That's it for announcements/reminders | 15:03 |
| JayF | No action items from last night; I'm skipping that item. | 15:04 |
| JayF | #topic Review CI Status / Update Whiteboard if needed | 15:04 |
| JayF | I think our CI is more happy than it's been all cycle? | 15:05 |
| TheJulia | ... I think so | 15:05 |
| TheJulia | maybe | 15:05 |
| TheJulia | I'm digging deep into sqlalchemy issues with metal3 at the moment | 15:05 |
| JayF | sounds like a good thing to do, but a horrible thing while doing it | 15:06 |
| JayF | good luck o/ | 15:06 |
| JayF | #topic Review ongoing 2023.2 workstreams | 15:06 |
| JayF | #link https://etherpad.opendev.org/p/IronicWorkstreams2023.2 | 15:06 |
| JayF | This exists now, we can put progress on our workstreams in there. | 15:06 |
| JayF | (Since the spec landed) | 15:06 |
| JayF | I've got sharding in there. | 15:06 |
| JayF | #topic Open Discussion | 15:07 |
| JayF | That's all for the standing agenda, anything for open discussion? | 15:08 |
| TheJulia | I wouldn't mind pairing up with someone to dig at this 65MB ironic.log file | 15:09 |
| JayF | I'm going to end the meeting then lets chat about that | 15:09 |
| JayF | #endmeeting | 15:09 |
| opendevmeet | Meeting ended Mon May 1 15:09:52 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:09 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-01-15.01.html | 15:09 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-01-15.01.txt | 15:09 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-05-01-15.01.log.html | 15:09 |
| JayF | TheJulia: what are you looking for? | 15:09 |
| arne_wiebalck | JayF: when will sharding be available? (we will set up a 2nd Ironic soon :-) | 15:10 |
| JayF | We never ever promise when something will be done | 15:10 |
| TheJulia | metal3's sqlalchemy debug log | 15:10 |
| JayF | but the goal is for Bobcat to ship with it | 15:10 |
| arne_wiebalck | JayF: thanks | 15:10 |
| TheJulia | 2nd ironic... O.o | 15:11 |
| TheJulia | Speaking of! | 15:11 |
| arne_wiebalck | (CERN is building a new data centre) | 15:12 |
| TheJulia | I need to dig up the t-shirt order I did back in 2019 and see if it is still available to order | 15:12 |
| TheJulia | arne_wiebalck: nice | 15:12 |
| arne_wiebalck | and it will get an independent Ironic | 15:12 |
| arne_wiebalck | for our current prod to move from CGs to sharding may be not worth it, but for a new deployment ... | 15:13 |
| TheJulia | quite possibly | 15:14 |
| JayF | TheJulia: you got that ironic log pasted somewhere? You want a redundant set of eyes or shard the work? | 15:26 |
| TheJulia | I think on a call might be best, but let me get the link | 15:33 |
| TheJulia | https://52f229209c149e782c36-b5f24c1063ad83372f17d0890078a578.ssl.cf1.rackcdn.com/881765/6/check/metal3-integration/29be9ae/controller/before_pivoting/ironic.log <-- 66 MB | 15:35 |
| TheJulia | oh wow, osx doesn't let you share terminal windows | 15:38 |
| TheJulia | https://meet.google.com/hvq-uvsk-ouc | 15:39 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: database lock/session fixes https://review.opendev.org/c/openstack/ironic/+/881765 | 16:33 |
| TheJulia | JayF: ^ crossing fingers | 16:33 |
| JayF | did it make any difference locally, as far as you could tell? | 16:33 |
| TheJulia | well, I could determine it was triggering it | 16:33 |
| TheJulia | ... because I typo-ed pragma | 16:34 |
| JayF | that was a clever intentional thing to do to find certainty it works ;) | 16:34 |
| TheJulia | indeed | 16:34 |
| TheJulia | so... with debug logging on still... it will be ~2 hours before we have results | 16:35 |
| JayF | sounds good. I have guys in my crawlspace inspecting my sewer pipes so my house won't explode (something about gas pipes running thru sewer pipes?) and a meeting at 10a, so that should be about when I'm ready | 16:35 |
| * TheJulia blinks | 16:36 | |
| * TheJulia blinks once more | 16:36 | |
| * TheJulia blinks... again. | 16:36 | |
| JayF | I learned thru this process that apparently most residential gas explosions are caused by automatic digging and pipelaying devices which happily dig thru sewer pipes | 16:37 |
| JayF | then if some uncareful plumber goes through and uses a root cutter in that sewer pipe | 16:37 |
| JayF | so I think they do inspections pretty regularly now | 16:37 |
| TheJulia | horizontal bores? | 16:47 |
| JayF | yeah I think so | 16:47 |
| JayF | irony is I'm fairly certain they just determined the sewer pipe goes outta my house in the opposite direction the gas pipe dues lol | 16:48 |
| TheJulia | heh | 16:59 |
| * TheJulia takes a break while CI does its thing to do yoga | 17:13 | |
| TheJulia | JayF: metal3 worked without errors | 17:51 |
| JayF | \o/ | 17:51 |
| TheJulia | looking for other db related issues in the logs while I've got a clean log | 17:51 |
| JayF | I'm tempted to ask in the slack for metal3 | 17:51 |
| JayF | if running sqlite in WAL mode for Ironic would be impacting to them | 17:52 |
| JayF | maybe just pinging dtantsur here and saying "sqlite WAL journal mode" makes him weep ;) | 17:52 |
| JayF | **and seeing if saying | 17:52 |
| TheJulia | looks like we've got a bad join on get_active_hardware_type_dict | 17:52 |
| TheJulia | the fact it worked, without any errors... *while* doing debug logging is kind of impressive | 17:53 |
| JayF | sqlite+WAL is magic | 17:55 |
| JayF | as long as you're on a real (not shared) FS | 17:55 |
| JayF | and don't have multiple processes fighting over the file lol | 17:55 |
| TheJulia | yeah, we luckily don't in this case | 17:57 |
| JayF | TBH I just don't know k8s well enough to know, for instance, if pods are sometimes backed by network storage | 18:00 |
| JayF | that's really where any of my concern lies -- with any use cases in k8s+metal3 beyond my knowledge | 18:00 |
| JayF | TheJulia: one other thing we should do before considering landing this: getting a clean run on metal3 with existing code, and compare (benchmark) the speeds with the WAL enabled | 18:01 |
| JayF | to ensure we aren't crippling perf (in fact; I suspect we might've boosted it) | 18:01 |
| TheJulia | might be a bit difficult given CI variability | 18:02 |
| TheJulia | and we would need to disjoin it from sqlalchemy autocommit | 18:03 |
| TheJulia | Anyway! I'm going to leave a note on the join I suspect needs to be revisited, someone can take a look at that at some point | 18:03 |
| TheJulia | now to fix the base patch up | 18:06 |
| JayF | TheJulia: oh, that's true. I can just push a patch changing WAL only, and look at the timing (and hope I hit the same cloud) | 18:16 |
| JayF | if it's within CI variance that is (arguably) a good enough result | 18:16 |
| opendevreview | Merged openstack/sushy stable/zed: workaround: requests verify handling if env is set https://review.opendev.org/c/openstack/sushy/+/880832 | 18:18 |
| TheJulia | if you want, sure. I'm going to keep working on this and looking at the failed ci jobs of which there are many at the moment but I'm also 2 patches deep | 18:22 |
| JayF | ack | 18:24 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix DB/Lock session handling issues https://review.opendev.org/c/openstack/ironic/+/881765 | 19:09 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix DB/Lock session handling issues https://review.opendev.org/c/openstack/ironic/+/881765 | 19:14 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove autocommit, again. https://review.opendev.org/c/openstack/ironic/+/862832 | 19:14 |
| TheJulia | order swapped around | 19:14 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix DB/Lock session handling issues https://review.opendev.org/c/openstack/ironic/+/881765 | 20:00 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove autocommit, again. https://review.opendev.org/c/openstack/ironic/+/862832 | 20:19 |
| JayF | TheJulia: https://zuul.opendev.org/t/openstack/build/582d0a4d06b247c3bfdf33bd69cd6c50 \o/ | 20:44 |
| JayF | TheJulia: good job | 20:44 |
| TheJulia | looks to be on-par performance wise compared to the job history | 20:55 |
| JayF | yep, it looks good all the way around | 20:55 |
| JayF | I'll go re-read the code and put a +2 on it :D | 20:55 |
| JayF | TheJulia: +2 with an advisory comment | 21:05 |
| TheJulia | JayF: lmk if the answer makes sense | 22:23 |
| TheJulia | I do like the idea of re-using the variable to keep from driving people looking at it crazy and adding a comment :) | 22:24 |
| JayF | yeah I mean, my whole beef is just if you look at the method quickly, it looks like somehting is wrong so you re-read it lol | 22:27 |
| JayF | which is a nit at best | 22:28 |
| JayF | I didn't grok your answer, but I view my comment as a style/aesthetic one | 22:28 |
| TheJulia | there is a legit reason, but I get what your saying | 22:29 |
| TheJulia | and I do agree | 22:29 |
| JayF | I worked at a place for a while which absolutely forbode early returns in python, and that influcenced my style a little | 22:30 |
| JayF | sometimes I can't tell if I'm using that lens by accident :) | 22:30 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix DB/Lock session handling issues https://review.opendev.org/c/openstack/ironic/+/881765 | 22:35 |
| TheJulia | JayF: revising so quick +2 | 22:35 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove autocommit, again. https://review.opendev.org/c/openstack/ironic/+/862832 | 22:36 |
| TheJulia | and quick rebase | 22:36 |
| JayF | +2'd | 22:36 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!