Monday, 2024-05-06

opendevreviewOpenStack Release Bot proposed openstack/bifrost master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/bifrost/+/91825406:21
opendevreviewOpenStack Release Bot proposed openstack/ironic-inspector master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-inspector/+/91825606:21
opendevreviewOpenStack Release Bot proposed openstack/ironic-prometheus-exporter master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/91825906:22
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent-builder master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/91826106:22
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-python-agent/+/91826306:23
opendevreviewOpenStack Release Bot proposed openstack/ironic-ui master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-ui/+/91826506:23
opendevreviewOpenStack Release Bot proposed openstack/ironic master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic/+/91826706:23
opendevreviewOpenStack Release Bot proposed openstack/metalsmith master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/metalsmith/+/91826906:24
opendevreviewOpenStack Release Bot proposed openstack/networking-baremetal master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/networking-baremetal/+/91827106:24
opendevreviewOpenStack Release Bot proposed openstack/networking-generic-switch master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/networking-generic-switch/+/91827306:24
opendevreviewOpenStack Release Bot proposed openstack/python-ironic-inspector-client master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/91827606:24
opendevreviewOpenStack Release Bot proposed openstack/python-ironicclient master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/python-ironicclient/+/91827806:25
opendevreviewOpenStack Release Bot proposed openstack/sushy master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/sushy/+/91828006:25
rpittaugood morning ironic! o/07:01
opendevreviewMerged openstack/ironic-specs master: Add work items for 2024.2 Dalmatian development cycle  https://review.opendev.org/c/openstack/ironic-specs/+/91629508:00
opendevreviewMerged openstack/ironic master: doc: Fix wrong example of [ipmi] cipher_suite_versions  https://review.opendev.org/c/openstack/ironic/+/91815808:34
opendevreviewVerification of a change to openstack/ironic-python-agent stable/2023.1 failed: Unmount config drives  https://review.opendev.org/c/openstack/ironic-python-agent/+/91785209:19
opendevreviewMerged openstack/python-ironicclient master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/python-ironicclient/+/91827809:23
opendevreviewMerged openstack/sushy master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/sushy/+/91828009:23
opendevreviewMerged openstack/networking-generic-switch master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/networking-generic-switch/+/91827309:25
opendevreviewMerged openstack/ironic-python-agent master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-python-agent/+/91826309:26
opendevreviewMerged openstack/python-ironic-inspector-client master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/91827609:28
opendevreviewMerged openstack/ironic-prometheus-exporter master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/91825909:28
opendevreviewMerged openstack/ironic-python-agent-builder master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/91826109:28
opendevreviewMerged openstack/ironic-ui master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-ui/+/91826509:33
opendevreviewMerged openstack/bifrost master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/bifrost/+/91825409:35
dtantsurrpittau, TheJulia, JayF, wdyt about instantly EOL'ing some of our deliverables instead of this ^^^?09:47
dtantsurI think bifrost is the obvious candidate, but also ironic-ui, and clients are also not super useful09:48
dtantsurIPA-builder can technically work across branches09:48
dtantsurmetalsmith09:48
opendevreviewMerged openstack/ironic-tempest-plugin master: Remove agent_ipmitool/pxe_ipmitool tests  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/91790209:53
opendevreviewMerged openstack/ironic-inspector master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic-inspector/+/91825609:53
opendevreviewMerged openstack/ironic master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/ironic/+/91826709:53
opendevreviewVerification of a change to openstack/ironic stable/2023.1 failed: ci: stable-only: explicitly pin centos build  https://review.opendev.org/c/openstack/ironic/+/91811810:29
opendevreviewMerged openstack/metalsmith master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/metalsmith/+/91826910:44
opendevreviewMerged openstack/ironic master: docs: document stance on partition image use  https://review.opendev.org/c/openstack/ironic/+/91538810:49
opendevreviewMahnoor Asghar proposed openstack/ironic master: Add inspection rules  https://review.opendev.org/c/openstack/ironic/+/91830310:52
opendevreviewMerged openstack/networking-baremetal master: reno: Update master for unmaintained/zed  https://review.opendev.org/c/openstack/networking-baremetal/+/91827111:06
rpittaudtantsur: I guess that would work too, but I'm not sure we can avoid the unmaintained status anyway as it's Openstack-wide11:29
dtantsurrpittau: there have been a discussion about skipping unmaintained status for some projects like openstacksdk11:31
rpittaudtantsur: having a precedent helps, I see if I can start a discussion in that direction for some of our projects12:19
JayFLet me check with my downstream too 12:42
JayFBut we don't use bifrost or metalsmith at least for sure12:42
dtantsurBifrost won't even work correctly without some modifications12:49
iurygregorygood morning Ironic12:50
TheJuliagood morning13:05
TheJuliaRegarding instantly EOL-ing some of our deliverables, specifically the non-service/component ones, I'm good with doing so13:07
opendevreviewJulia Kreger proposed openstack/ironic-specs master: Intermediate networking service  https://review.opendev.org/c/openstack/ironic-specs/+/91612614:48
TheJuliaUpdates from discovery and increased understanding over the weekend14:48
rpittau#startmeeting ironic15:00
opendevmeetMeeting started Mon May  6 15:00:31 2024 UTC and is due to finish in 60 minutes.  The chair is rpittau. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'ironic'15:00
iurygregoryo/15:00
masgharo/15:00
TheJuliagood morning15:00
rpittauHello everyone!15:00
rpittauWelcome to our weekly meeting!15:00
rpittauThe meeting agenda can be found here:15:00
rpittau#link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_May_6.2C_202415:00
dtantsuro/15:00
rpittau#topic Announcements/Reminders15:01
rpittau#info Standing reminder to review patches tagged ironic-week-prio and to hashtag any patches ready for review with ironic-week-prio: https://tinyurl.com/ironic-weekly-prio-dash15:01
rpittau#info Work items for 2024.2 have been merged https://review.opendev.org/c/openstack/ironic-specs/+/91629515:01
rpittau#info Ironic Meetup/BareMetal SIG June 5, OpenInfra Days June 6 @ CERN.15:02
rpittauSignup is now closed15:02
rpittau#info 2024.2 Dalmatian Release Schedule https://releases.openstack.org/dalmatian/schedule.html15:02
rpittauanything else to announce/remind ?15:02
iurygregory#info CFP for OIS Asia'24 closes on May 2915:02
rpittauthanks iurygregory 15:02
iurygregoryyw15:03
rpittauI do have an announcement, I will be out for a couple of weeks, I'll be back on Monday May 2615:04
rpittauI will skip two meetings, so need one/two volunteer(s) to run the next 2 meetings :)15:04
TheJuliaI've got nothing planned so I guess I can15:04
dtantsurI'll be out for 2 Mondays as well15:04
rpittauthanks TheJulia :)15:04
rpittau#info TheJulia will run the meetings on 13 and 20 of May15:05
iurygregoryI can run15:05
iurygregoryoh nvm =D15:05
rpittauthanks iurygregory, I guess you can alternate with TheJulia :)15:05
iurygregoryyeah =) for sure15:06
rpittauok, moving on!15:06
rpittau#topic Review Ironic CI status15:06
rpittauhaven't seen anything particular in CI last week15:07
JayFI believe Julia landed some fixes for stable branches in IPA15:07
JayFOther than that I've not seen anything new15:07
TheJuliaYeah, CI seemed mostly okay last week with the exception being some stable branches being a little cranky15:08
rpittautrue, thanks TheJulia!15:08
rpittaunow that I think about it, I've seen some failures on grenade jobs in stable releases, maybe due to zed going into unmaintained mode15:08
rpittauwe may need to remove the job in zed to be able to land the latest changes in the branch, just simple branch updates15:09
rpittauI don't see any discussion topics for this week, do we want to jump directly to Bug Deputy updates ?15:10
TheJuliaYeah, oldest grenade always needs to be disabled15:10
rpittau#action disable grenade in zed15:11
TheJuliaBug Deputy update sounds like it is next15:11
rpittau#topic Bug Deputy Updates15:12
rpittauTheJulia: anything to reprt? :)15:12
TheJuliaTwo items to note15:12
TheJuliaFirst, I cleaned up ironic-inspector bugs, mostly marked stuff that was super old to indicate we were not planning on fixing the items15:12
TheJuliaI also was able to close out a few bugs in ironic as already fixed or no longer applicable15:13
rpittaugreat, thanks!15:13
JayFI'll take it this week15:14
iurygregoryI can be the bug dep after next week (ie, the week from May 20 - 24)15:14
rpittauthanks JayF I was going to ask :D15:14
rpittauand thanks iurygregory :)15:14
rpittauI guess we can move to15:15
rpittau#topic RFE Review15:15
rpittauwe have 2 RFEs proposed by rozzi but I don't see him here today15:15
rpittauthey look both reasonable to me15:16
rpittauand I saw the development already started15:16
TheJuliaI concur, I think the block device list combo one makes sense15:16
rpittauyeah, absolutely15:16
TheJuliathat seems more like a bug to me, tbh15:16
TheJuliathe other, I have some mixed feelings on but I can see the reasoning15:17
rpittausure on the edge15:17
rpittauI guess the scenario proposed makes sense15:17
* iurygregory looks at the RFEs15:17
JayFCan we link the rfe's in here so that they are in the minutes? I also don't have the agenda up lol15:17
rpittau#link https://bugs.launchpad.net/ironic-python-agent/+bug/206143715:17
rpittau#https://bugs.launchpad.net/ironic-python-agent/+bug/206136215:17
rpittau#link https://bugs.launchpad.net/ironic-python-agent/+bug/206136215:18
opendevreviewVerification of a change to openstack/ironic stable/2023.1 failed: ci: stable-only: explicitly pin centos build  https://review.opendev.org/c/openstack/ironic/+/91811815:18
JayF2061362, I'm not sure I'm on board with approving that15:18
JayFWe have methods to disable the step entirely which seems to be the correct move if you have a disc that you do not want cleaned15:19
dtantsurMixed feelings here too. We can disable the step or we can skip a disk by using hints.15:19
iurygregorywe can also skip cleaning a given disk 15:19
iurygregoryyeah15:19
TheJuliaBased upon the described case, it sounds like they have shared luns across hosts15:20
rpittaummm I can see your point too, I'm more on the verge of accepting honestly15:20
JayFIs the actual feature a way to skip shared devices? Like a device hint that points to any shared devices that they can skip cleaning15:20
TheJuliaso, if we can better understand the exact case in which the device is locked, that might help us understand how this feature could be scoped and accepted as an "optional" thing15:20
JayFI'm okay with a declarative way of saying don't clean this. I'm less okay with saying just ignore certain types of errors during cleaning15:20
TheJuliacan't necessarilly know what the device is or if it will be in use in a shared lab15:20
dtantsurI'd even be okay with "skip any remote device" as an option15:21
JayFI do think it's not unreasonable to ask someone to use a custom hardware manager for a lab case if it is a unique use case15:21
JayFSame dtantsur 15:21
masgharI think their trouble is having many disks and a random one randomly failing15:22
TheJuliaskipping "remote" might make sense, not sure I'd ever let a customer use it though :)15:22
TheJuliabut that is always a documentation thing15:22
rpittauJayF, dtantsur, when you have a minute please add a comment in the RFE, I guess we need more details on that?15:22
dtantsurmasghar: it's not a normal situation there? we're curious why they fail.15:22
JayFmasghar: another way of looking at that is should  ironic say a machine is available if it has a random disk failure15:22
masghardtantsur, JayF, I see...but then their whole fleet can't be cleaned?15:23
rpittauI will approve the first one15:23
dtantsurmasghar: I'd be really curious what is going on in this case15:24
dtantsurLike, in my whole career in OpenShift, I don't remember a single such case15:24
masghardtantsur: true, I guess disk failures are too serious to ignore15:24
dtantsurA harsher way to put that would be "if you don't care if some of your disks don't get cleaned, why enable cleaning at all?"15:25
TheJuliato me, it reads as if there are quorum disks in the environment15:25
TheJuliaand we hit one while another node is up which shares the quorum disk, and we fail at that point15:26
masgharoh I see!15:26
dtantsurSure but... why have cleaning enabled?15:26
dtantsurMetadata cleaning (the only type supported in metal3) is designed to prevent software conflicts, such as Ceph not liking old Ceph partitions15:27
JayFOr why not as part of your process, skip cleaning on shared/remote discs. Even if the feature was oriented that way, we'd likely approve it as skipping cleaning on remote devices optionally15:27
TheJuliadtantsur: I think because they are re-cycling the environment15:27
rpittauthe error is not going to be ignored though, just not fail the entire cleanup15:27
rpittaubut I guess at that point excluding the disk (removing it physically or logically) would be better than leaving it there in any case, and then redoing the cleaning15:27
dtantsurYeah, but if you leave random data around on a shared disk, you still hit software problems15:27
TheJuliadtantsur: so if you have a host already cleaning, and it grabs the quorum disk first, it has the lock15:27
TheJuliaanother one comes along and will fail15:27
TheJuliadtantsur: if the other node is being cleaned, the data gets wiped out, but another node doesn't know about it15:28
dtantsurOr maybe it uses the disk, and you try to clean it :) I think this environment should have no cleaning or some very custom cleaning15:28
dtantsurexactly15:28
dtantsurhow is cleaning the way we do it safe there?15:28
TheJuliawe kind of need to take a step back from the problem15:28
TheJuliaand provide greater clarity15:28
TheJuliais there a way to identify this as an "acceptable" failure15:29
TheJuliafor that device, or not15:29
TheJuliafor example, I ran 8GB quorum disks in clusters years ago shared across multiple nodes, the lock holder was always the leader15:29
dtantsurRight, yeah. We're guessing already, we should get back to Adam15:29
TheJulia++15:29
rpittauyep, I agree we need more clarification at this point15:29
dtantsurI think the option to ignore errors should be our last resort15:31
JayFYeah basically we open ourselves up to a security vulnerability if someone can figure out how to break our agent using something crazy on disk15:31
dtantsurWell, this is metadata cleaning15:31
dtantsurFor full cleaning, I'd be hard -2 on even considering that15:32
TheJuliaJayF: if a disk is locked, you might not be able to read it depending on the lock15:32
JayFYeah, I guess for metadata cleaning the worst cases that will get very low quality bugs around people disabling error checking and then being shocked that their unclean disc can't be deployed to15:32
TheJuliayeah15:32
JayFI still think we should round trip some questions about details. I'm not a hard and fast opposed no matter what with that nuance in mind though15:33
TheJulia++15:33
dtantsuragreed15:33
rpittau++15:34
rpittauplease just add a comment in the RFE to get more info from Adam15:35
TheJulia+=15:35
TheJulia++15:35
JayFI'll toss that comment in here in the next 15 minutes15:36
rpittauthanks JayF :)15:36
JayFIf y'all want to just let me do it so that we don't crush Adam with five things that say the same thing LOL15:36
rpittausounds good :D15:36
* dtantsur nods15:37
rpittauwe can probably move on15:37
rpittauis there anything for open discussion ?15:37
JayFAh that Nova bug I linked earlier in the ironic driver, it's worth a look if anyone has some time15:38
JayFI'm mainly curious if we need to add tempest coverage on the nova driver side15:39
JayFfor things like reboot15:39
rpittauI think I lost it15:39
rpittaucan you link it here?15:39
JayF https://review.opendev.org/c/openstack/nova/+/918195 15:39
rpittauthanks15:39
JayF#link https://launchpad.net/bugs/206482615:39
JayFtl;dr we broke nova reboot last cycle, CI didn't catch it15:39
rpittauwhoops15:39
JayFit's being fixed but I wanted to see if it is an oversight or an explicit decision to not test this15:39
JayFif an oversight, I'll see if I can swing a test for it15:40
JayFbut always like asking since I know we are shaving our test surfaces, don't wanna readd something that was intentionally removed15:40
TheJuliaafaik, our basic ops plugin doesn't exercise reoot15:40
TheJuliareboot15:40
TheJuliadoing reboot testing as part of it would also add some time to plugin execution15:41
TheJulia... Mileage may vary, of course as with all things tempest plugin wise15:41
JayFIt sorta goes back to my statement a cycle ago ... I wish we could just have a fake ironic node for some of these interactions/tests15:41
JayFbut I think that's more than I'm willing  to take on right now, so I might just swing at a reboot test and see what the actual impact is in minutes15:41
TheJuliawell, we sort of can but the test can't ping something entirely fake15:42
TheJuliayour ether doing something entirely fake, or your doing stuff which is a full scenario15:42
TheJuliaintegrative is scenario inherently15:42
JayFwell, Nova<>Ironic driver, some of those things are just "did we tell Ironic to do the thing"15:43
TheJuliadoing stuff in the middle leads fixes like https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918001 having to come along15:43
JayFheh, fair15:43
JayFI'll keep wishing for utopia and living in the real world instead then :P15:43
TheJuliaheh15:43
TheJuliaadding to our basic ops likely isn't a huge issue, just add a knob for the time15:44
TheJuliaif a reasonable one doesn't already exist15:44
JayFack15:44
rpittauthanks for reporting that JayF 15:44
rpittauanything else we want to discuss today?15:45
dtantsurAt some point, we should probably delineate stuff we test in Nova and stuff we test in Ironic15:45
dtantsurotherwise, our testing matrix will keep being chaotic.15:45
JayFdtantsur: that's a line of chat Sean Mooney and I had last cycle; a tempest scenario set specifically designed for nova driver, and to run it more frequently15:46
TheJuliaYeah, I'm not a fan of doing an extra reboot when exercising rescue15:46
TheJuliaand then doing that across however many tests that ends up triggering on15:46
JayFbut realistically I'm already majorly overcommitted this cycle15:46
rpittauI guess this needs a longer discussion and involving the nova team15:47
rpittauwe need to see if we can find the resources during this cycle15:47
TheJuliawell15:47
TheJuliaThey are not going to really grasp the current scale15:47
JayFSean is sorta our ambassador to the nova team in some ways these days15:48
JayFbut I don't wanna engage a chat at that level if I don't have time to take action items outta it15:48
rpittauyep, exactly15:48
TheJuliabut we do need to likely take a look and sort out if we've got appropriate coverage with the plugin15:48
* JayF has had a problem with saying yes more times than he delivered a "yes" result15:48
dtantsurwe know this feeling15:50
rpittauwe can probably reevaluate the situation during the cycle15:50
rpittaulooks like we're closing15:51
rpittauthanks everyone15:51
rpittau#endmeeting15:51
opendevmeetMeeting ended Mon May  6 15:51:55 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:51
opendevmeetMinutes:        https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-05-06-15.00.html15:51
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-05-06-15.00.txt15:51
opendevmeetLog:            https://meetings.opendev.org/meetings/ironic/2024/ironic.2024-05-06-15.00.log.html15:51
rpittaubye everyone, see you in 3 weeks! o/16:07
opendevreviewMohammed Boukhalfa proposed openstack/sushy-tools master: Add fake ironic python agent (fake_ipa)  https://review.opendev.org/c/openstack/sushy-tools/+/87536616:23
ikoliveiraTheJulia and iurygregory  thanks for supporting me, your recommendations solved my issues with deploying baremetal nodes and now we are able to use it. Thank you so much!18:14
TheJulia\o/18:15
TheJuliaCongrats!18:15
ikoliveira\o/18:15
JayFwoohoo18:19
*** dking is now known as Guest428119:06
*** Guest4281 is now known as dking19:06
opendevreviewMerged openstack/ironic stable/2023.1: ci: stable-only: explicitly pin centos build  https://review.opendev.org/c/openstack/ironic/+/91811819:07
opendevreviewMaximilian Brandt proposed openstack/ironic-python-agent master: Add bmc_subnet information to introspection and Consolidated BMC information retrieval logic into a single function _get_bmc_info().  https://review.opendev.org/c/openstack/ironic-python-agent/+/91834019:13
opendevreviewMaximilian Brandt proposed openstack/ironic-python-agent master: Add bmc_subnet information to introspection.  https://review.opendev.org/c/openstack/ironic-python-agent/+/91834019:16
*** Guest2968 is now known as diablo_rojo20:38
iurygregoryikoliveira, glad it worked \o/ now you can add ironic to the production cloud =D20:39
NobodyCamGood evening Ironic folks,23:18
NobodyCamany updated doc on getting unit test going?23:19
TheJuliaNobodyCam: does jut running "tox" not sort things out for unit tests?23:26
TheJulias/jut/just/23:27
NobodyCam`failed with pass_env values cannot contain whitespace, use comma to have multiple values in a single line, invalid values found 'http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY'`23:27
NobodyCamtox.ini looks okay23:27
NobodyCamI was looking at: https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html#unit23:28
TheJuliaso you've got an older version with a newer tox which is incompatible23:29
TheJuliahttps://www.irccloud.com/pastebin/PneTjf2s/23:30
opendevreviewcid proposed openstack/ironic master: Flexible IPMI credential persistence method configuration  https://review.opendev.org/c/openstack/ironic/+/91722923:30
NobodyCamoh23:31
NobodyCamThank you!23:33

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!