| rpittau | good morning ironic! o/ | 07:15 |
|---|---|---|
| yde_ | hi, i'm trying to build an ipa image using the ipa-builder, i just dont get how to tell the ipa-builder how to use my local IPA repo instead of https://opendev.org/openstack/ironic-python-agent | 08:03 |
| yde_ | any clue ? thanks a lot | 08:03 |
| rpittau | yde_: you should be able to override ipa_source_path | 08:39 |
| yde_ | rpittau: ok but how to pass it to the builder ? through env vars ? through extra-args ? | 08:41 |
| yde_ | rpittau: i've seen this env var DIB_REPOLOCATION_ironic_python_agent | 08:43 |
| rpittau | yde_: tghat's defined by ipa_source_path in the ansible role | 08:44 |
| rpittau | yde_: there's also an example here https://docs.openstack.org/ironic-python-agent-builder/latest/contributor/index.html | 08:45 |
| rpittau | sorry, I meant override DIB_REPOLOCATION_ironic_python_agent :) | 08:45 |
| yde_ | rpittau: thanks a lot | 08:47 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Remove outdated install pyyaml with pip2 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/886379 | 08:48 |
| Nisha_Agarwal | rpittau, ironic weekly meeting is there today, right? | 13:09 |
| rpittau | mmm she quit | 13:28 |
| rpittau | anyway, should be | 13:28 |
| opendevreview | Baptiste Jonglez proposed openstack/networking-generic-switch master: Fix batching error due to outdated etcd3gw https://review.opendev.org/c/openstack/networking-generic-switch/+/886404 | 13:57 |
| opendevreview | Baptiste Jonglez proposed openstack/networking-generic-switch master: Honor ngs_save_configuration setting when using batch commands https://review.opendev.org/c/openstack/networking-generic-switch/+/886405 | 13:58 |
| rpittau | JayF: I'm starting to have doubts about the meeting today, is it happening? | 14:53 |
| JayF | in 7 minutes | 14:53 |
| rpittau | ack thanks | 14:53 |
| JayF | not sure how useful my tired self will be beyond that today :D | 14:55 |
| JayF | but going to do that for sure | 14:55 |
| JayF | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Jun 19 15:00:36 2023 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| JayF | Goooood morning Ironic, who all is here for our meeting today | 15:00 |
| rpittau | o/ | 15:00 |
| matfechner | o/ | 15:00 |
| masghar | o/ | 15:01 |
| JayF | I would expect lower attendance than usual as some contributors may still be on holiday/travelling home from the summit. | 15:01 |
| JayF | #topic Announcements/Reminder | 15:01 |
| JayF | #note Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash | 15:01 |
| JayF | #note Please ensure to fully investigate CI failures, and especially make note of TIMED_OUT failures in unit tests, and do not spam recheck as we are trying to work the issues out. | 15:02 |
| JayF | #topic Review Action items from previous meetings | 15:02 |
| JayF | last meeting was June 5, and there were no actions. Moving on. | 15:02 |
| JayF | #topic Review Ironic CI Status | 15:02 |
| JayF | does anyone have a meaningful update on the unit test pain? | 15:03 |
| JayF | I lost the thread on that at summit | 15:03 |
| rpittau | well my patches with the timeout setting are just waiting for a +W :) | 15:03 |
| rpittau | they could maybe give us some more hints | 15:03 |
| JayF | can you link that here? I'll land it | 15:03 |
| rpittau | yep, it's a chain that starts from here https://review.opendev.org/c/openstack/ironic/+/885372 | 15:05 |
| JayF | ack | 15:05 |
| JayF | I will ensure that lands after the meeting | 15:05 |
| rpittau | thanks! | 15:05 |
| JayF | #topic Review ongoing 2023.2 Workstreams | 15:05 |
| JayF | #link https://etherpad.opendev.org/p/IronicWorkstreams2023.2 | 15:05 |
| JayF | does not look like significant progress during summit week, as expected | 15:05 |
| JayF | Any comments on ongoing 2023.2 work before I move on | 15:06 |
| JayF | #topic Open Discussion | 15:07 |
| JayF | looks like bug 2021995 is on the agenda | 15:07 |
| JayF | Nisha_Agarwal: I presume that was you? What's up! | 15:07 |
| JayF | #link https://bugs.launchpad.net/ironic/+bug/2021995 | 15:08 |
| Nisha_Agarwal | JayF, Hi | 15:08 |
| Nisha_Agarwal | Yes | 15:08 |
| Nisha_Agarwal | we wanted to have clarity on the bug...how the bug is reproduced | 15:08 |
| Nisha_Agarwal | and which hardware it is seen | 15:08 |
| JayF | It's iLO Gen 10 as mentioned in the bug; I don't know the specific SKU but you can ask there and Scott or someone else shoudl be able to get it | 15:09 |
| Nisha_Agarwal | as of now if it is Gen10 or gen10+ then proliantutils bank on sushy for connecting to hardware for redfish calls | 15:09 |
| JayF | as far as reproduction, we haven't found a tight reproducer | 15:09 |
| JayF | I'd prefer we keep most of the commentary in the bug though; the impacted parties are not in the Ironic meeting, and we (me/you/the impacted bug filers) are all in different timezones | 15:10 |
| JayF | so it'd be easier to coordinate there | 15:10 |
| Nisha_Agarwal | as far as i understand, during power_state_sync ironic is trying to power on while for some reason node is getting rebooted/power off | 15:10 |
| JayF | so the hardware is power looping in this case | 15:10 |
| Nisha_Agarwal | and probably ilo is not able to understand which one to honor | 15:11 |
| JayF | it's set to pxe and is failing | 15:11 |
| Nisha_Agarwal | may be a race condition in ilo | 15:11 |
| JayF | that's exactly what I'm wondering | 15:11 |
| JayF | and like, I don't expect the power status loop to get a sane value out of a node with flapping power statuses | 15:11 |
| Nisha_Agarwal | but to connect to hardware team we need to reproduce the issue so that we could tell them the exact reproducer | 15:11 |
| JayF | but we should ensure it doens't get stuck or left in a bad state | 15:11 |
| Nisha_Agarwal | there was a bug in gen9 for blade enclosures | 15:12 |
| Nisha_Agarwal | around 6-7 years back | 15:12 |
| JayF | I believe you may be able to reproduce the circumstances by: 1) setting node to pxe 2) rebooting node into environment with no pxe boot 3) see power status flap | 15:12 |
| Nisha_Agarwal | where i guess it was fixed for something similar in ilo(and a workaround was provided in proliantutils for gen9) | 15:12 |
| JayF | but I am remote to the hardware as well, so that's as far as my insight goes | 15:12 |
| JayF | if you have specific advice for trying to repro the bug, if we get that into the bug text I can make sure Scott sees it. | 15:12 |
| Nisha_Agarwal | I tried to reach Aaron/Scott on the launchpad but didnt get any response so far | 15:13 |
| JayF | Scott was at Summit | 15:13 |
| Nisha_Agarwal | even on IRC i couldnt get him | 15:13 |
| Nisha_Agarwal | Ohk | 15:13 |
| JayF | he's not usually on IRC, the bug is the best way to communicate | 15:13 |
| Nisha_Agarwal | Ok | 15:13 |
| JayF | and we work directly together so if I see a comment has sat for a few days, I'll pass it on :) | 15:13 |
| Nisha_Agarwal | Thanks Jayf, | 15:14 |
| Nisha_Agarwal | That will help | 15:14 |
| JayF | absolutely :) Thanks for taking a look! | 15:14 |
| JayF | Is there anything else for the meeting? More on this bug? Another open discussion topic? | 15:14 |
| Nisha_Agarwal | i will try the reproducer you have given and see if we can hit the issue | 15:14 |
| Nisha_Agarwal | for now this topic is doe | 15:14 |
| Nisha_Agarwal | done* | 15:14 |
| Nisha_Agarwal | Thank you | 15:14 |
| JayF | Going to give a minute for anyone to come forward with addl. open discussion topics | 15:15 |
| JayF | One thing to note: I'll be looking in a day or two to email a short summart of the summit PTG sessions; there are some notes in | 15:15 |
| JayF | #link https://etherpad.opendev.org/ironic-openinfra-2023 | 15:15 |
| rpittau | nice, was going to ask about that :) | 15:15 |
| JayF | but mainly just time spent thinking about what Ironic might look like in 5/10 years | 15:15 |
| JayF | and some design discussion for NGS | 15:16 |
| JayF | I think is mostly all that happened | 15:16 |
| JayF | I'm going to call it, thank you for those who've been here o/ | 15:16 |
| JayF | #endmeeting | 15:16 |
| opendevmeet | Meeting ended Mon Jun 19 15:16:51 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:16 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-06-19-15.00.html | 15:16 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-06-19-15.00.txt | 15:16 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-06-19-15.00.log.html | 15:16 |
| JayF | rpittau: landed that chain, assuming zuul cooperates | 15:45 |
| JayF | rpittau: anytime you have something like that pending review, and I don't have a +2 on it, and it's univeresally helpful, feel free to link to me directly ... especially in cases like last week where I'm not closely tuned into IRC/gerrit | 15:45 |
| rpittau | JayF: thanks, will do :) | 15:51 |
| rpittau | the idea with that is that since we can control the timeout we should be able to see what fails in a reasonable time | 15:52 |
| JayF | ++ | 15:52 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add test timeout to tox config https://review.opendev.org/c/openstack/ironic/+/885372 | 16:02 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use tox env variables in coverage tests https://review.opendev.org/c/openstack/ironic/+/885507 | 16:02 |
| opendevreview | Verification of a change to openstack/ironic master failed: Allow setting migrations timeout value from tox https://review.opendev.org/c/openstack/ironic/+/885837 | 16:02 |
| rpittau | JayF: one more thing if you have a moment https://review.opendev.org/c/openstack/ironic/+/869052 | 16:02 |
| JayF | +2a | 16:02 |
| rpittau | mmm the timeout patch failed not for timeout this time | 16:03 |
| rpittau | weird | 16:03 |
| rpittau | I'll have a look tomorrow | 16:04 |
| rpittau | good night! o/ | 16:04 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove instance image after deployment https://review.opendev.org/c/openstack/ironic/+/878680 | 16:10 |
| zorun | JayF, TheJulia: following our NGS design discussions at the summit, I opened a RFE bug to discuss it further: https://bugs.launchpad.net/networking-generic-switch/+bug/2024385 | 17:07 |
| zorun | feel free to chime in on the HA & hash-ring topic, this is the part I'm least knowledgeable about | 17:08 |
| JayF | I'd suggest posting it to the list with some context | 17:08 |
| JayF | to see if you can catch more interested parties | 17:09 |
| JayF | I know many of the folks at the summit interested do not often frequent irc | 17:09 |
| zorun | right, good idea | 17:09 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 17:14 |
| iurygregory | good morning Ironic | 17:24 |
| * iurygregory is alive but still a bit tired of the travel... | 17:24 | |
| JayF | Glad you managed to find your way home; looked like on socials you had a bit of a headache | 17:25 |
| iurygregory | yeah =( | 17:35 |
| iurygregory | if anyone has time today https://review.opendev.org/c/openstack/ironic/+/883062 https://review.opendev.org/c/openstack/ironic/+/885276/ I would appreciate o/ | 19:16 |
| JayF | ah, firmware interface | 19:19 |
| JayF | I will review but would like someone who cares more about the api design, e.g. dmitry, to be one of the cores signing off in the end if possible | 19:19 |
| JayF | just stating I'm reviewing for code correctness, less so for api correctness and friendliness :D | 19:19 |
| iurygregory | ack | 19:19 |
| iurygregory | no worries | 19:19 |
| iurygregory | CI still in a bad shape right? | 19:20 |
| * iurygregory checks weekly meeting logs | 19:20 | |
| JayF | I think we're all basically a week outta date around that | 19:20 |
| JayF | rpittau's change to add timeouts was approved by me, but it failed CI | 19:21 |
| JayF | in what looked like a suspicious failure in both standalone jobs | 19:21 |
| JayF | (a -maybe- net new issue? oh boy! | 19:21 |
| iurygregory | I just saw this patch https://review.opendev.org/c/openstack/ironic/+/885276/ | 19:22 |
| JayF | yeah even our failures are kinda probabalistic | 19:22 |
| iurygregory | yup =( | 19:23 |
| iurygregory | just me or zuul is really slow today? | 19:23 |
| iurygregory | from the results in this patch I think TheJulia had some patch to change something related to ConnectionEvents.. | 19:24 |
| JayF | given the failures in our tests that we were tracking before were sqlite | 19:26 |
| JayF | er, I mean | 19:26 |
| JayF | were mysql/postgres migration tests | 19:26 |
| JayF | it's hard to imagine that sqlite change being impactful | 19:26 |
| JayF | but I'd believe a lot of stuff at this point lol | 19:26 |
| iurygregory | yup! | 19:27 |
| JayF | https://github.com/openstack/ironic/commit/75b881bd31952b8f858b4708fb83f7d2ca7c42cc might be what you were thinking of, yeah? | 19:27 |
| JayF | or at least one of the changes in there | 19:28 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/885797 | 19:29 |
| iurygregory | this one | 19:29 |
| JayF | that's just fixing up an error in sql2.0 unit tests | 19:29 |
| JayF | that don't impact <2.0 | 19:29 |
| JayF | just a relocated thing | 19:29 |
| iurygregory | yeah, this would fix what we see in https://review.opendev.org/c/openstack/ironic/+/885276/ I think | 19:30 |
| iurygregory | but doesn't solve the weird failures we see in our CI .-. | 19:30 |
| TheJulia | We could cast unit tests with different parameters most likely | 19:31 |
| JayF | the best path forward for CI, I think, is getting rpittau's timeout changes in | 19:31 |
| TheJulia | Well, launch. There is a special code path | 19:31 |
| JayF | so we change TIMED_OUT failures to actually troubleshootable errors | 19:31 |
| JayF | TheJulia: I don't understand | 19:31 |
| TheJulia | So if it is the sqlalchemy changes from like early may, we can change it to isolate the code path since unit tests get launched in a different way | 19:32 |
| * JayF wonders if https://github.com/openstack/ironic/commit/75b881bd31952b8f858b4708fb83f7d2ca7c42cc#diff-caf8ada388bea74c1068bdaa9b41e7243bd2aef6a37697db86790bc9e2fc8c10R30 is hooked up in a way to force sequential execution | 19:33 | |
| JayF | or if it's succeptable to an error where the connection goes away before it can execute | 19:33 |
| JayF | I can imagine bugs which would cause our behavior in something like this | 19:33 |
| TheJulia | So there is a whole separate db launch path in the db tests code because of the pre population of schema | 19:53 |
| JayF | we likely don't use that path for migration testing then, right? | 19:53 |
| JayF | since we don't want a schema there? | 19:54 |
| TheJulia | I just checked into my last hotel, flying back early tomorrow | 19:54 |
| TheJulia | Oh, that is a super good question | 19:54 |
| JayF | it would finally explain the "why is this different" problem, if that's the case | 19:54 |
| JayF | I *must* be missing something | 20:02 |
| JayF | https://github.com/openstack/ironic/blob/master/ironic/tests/unit/db/sqlalchemy/test_migrations.py#L1346 | 20:02 |
| JayF | that self.engine looks like it's a MagicMock() | 20:02 |
| JayF | but there are some that inherit from it | 20:04 |
| JayF | and must inject an engine | 20:04 |
| TheJulia | As well as migration_api | 20:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!