15:00:13 <TheJulia> #startmeeting ironic
15:00:14 <openstack> Meeting started Mon Dec  9 15:00:13 2019 UTC and is due to finish in 60 minutes.  The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:15 <iurygregory> o/
15:00:15 <TheJulia> o/
15:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <openstack> The meeting name has been set to 'ironic'
15:00:24 <kaifeng_> o/
15:00:25 <rpittau> o/
15:00:26 * iurygregory was too fast
15:00:33 <TheJulia> heh
15:00:38 <arne_wiebalck> o/
15:00:41 <TheJulia> \o
15:00:46 <TheJulia> Good morning everyone!
15:01:11 <Guest23212> o/
15:01:11 <rpioso> o/
15:01:21 <mgoddard> \o
15:01:22 <Guest23212> jerrywang1
15:01:28 <cdearborn> o/
15:01:33 <khansa> o/
15:01:40 <TheJulia> Our meeting agenda can be found on the wiki!
15:01:42 <TheJulia> #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting
15:01:43 <jerrywang1> o/
15:02:04 <TheJulia> #topic Announcements/Reminders
15:02:31 <bfournie> o/
15:03:05 <hjensas> o/
15:03:06 <TheJulia> On the announcements front, It is fairly clear CI is very broken right now and the default IPv6 in the flat networks in devstack have sort of made things less reliable.
15:03:10 <TheJulia> dtantsur: do we have a bug filed for ^
15:03:20 <dtantsur> not from me, sorry
15:03:26 <TheJulia> dtantsur: c'est la vie
15:03:36 <dtantsur> but I have a patch
15:03:51 <dtantsur> #link https://review.opendev.org/698012 potential CI fix
15:03:52 <patchbot> patch 698012 - ironic - CI: disable IPv6 in neutron - 1 patch set
15:03:55 <TheJulia> On the reminder front, CERN has offered to host a midcycle. There is a sign-up page.
15:03:59 <dtantsur> let's recheck it once before approving though
15:04:01 <TheJulia> arne_wiebalck: if you hav the link handy
15:04:04 <TheJulia> dtantsur: thanks
15:04:08 <arne_wiebalck> https://indico.cern.ch/event/863986/
15:04:17 <rpioso> arne_wiebalck: Thank you!
15:04:17 <TheJulia> #link https://indico.cern.ch/event/863986/
15:04:24 <arne_wiebalck> please register if you plan to attend
15:04:33 <TheJulia> Does anyone have anything else to announce or remind us of?
15:04:50 * rpioso thinks about his holiday shopping
15:04:51 <iurygregory> Dell CI is not with accessible logs =(
15:05:03 <TheJulia> rajinir: ^^^
15:05:16 <dtantsur> I'm not sure any 3rd party CI is working atm :(
15:05:22 <iurygregory> HP CI I haven't saw reports in a while since they were with POST_FAILURE and no answer to my email =(
15:05:25 <rpioso> iurygregory: I've informed rajinir thru our internal IM system.
15:05:26 <iurygregory> dtantsur, yeah
15:05:28 <dtantsur> I've mostly see Dell and IBM recently
15:05:45 <dtantsur> would be great to get the status of the others
15:05:52 <TheJulia> I'll see if I can find some time to follow-up on 3rd party CIs this week.
15:06:01 <TheJulia> iurygregory: if you can forward that email to me, that would be appreciated.
15:06:08 <TheJulia> rpittau: thanks
15:06:09 <iurygregory> TheJulia, sure!
15:06:10 <TheJulia> err
15:06:12 <TheJulia> rpioso: thanks
15:06:21 <TheJulia> Okay! sounds like we can move on then!
15:06:22 <rpioso> TheJulia: :-)
15:06:34 <TheJulia> oh, wait
15:07:21 <TheJulia> rpioso raises a good point about holidays. Around the 15th people tend to start disappearing for two to three weeks as end of year deadlines, training, and time off begin.
15:08:06 <TheJulia> If there are any cores that will be available during for the next few weeks, even if it is "I have to wave a special flag to get attention"
15:08:25 <dtantsur> I'm out 16th-1st
15:08:36 <TheJulia> That would be helpful to at least get CI fixes merged without blocking the entire project until the end of the first week of Janurary
15:08:39 <TheJulia> dtantsur: k
15:08:56 <TheJulia> I've got no specific time off planned, but I'll surely become less responsive the 24th-1st
15:08:58 <dtantsur> Can be pinged mid-holidays after 22nd via personal means
15:09:00 <rpittau> I will be available for the next 2 weeks, except on fridays, and then available also 23rd 24th, then out until 1st
15:09:06 <TheJulia> dtantsur: thanks
15:09:18 <openstackgerrit> Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder master: Add efibootmgr  https://review.opendev.org/698026
15:09:43 <arne_wiebalck> I should be around most of the time.
15:09:56 <TheJulia> That at least helps. I suspect our next meeting will be the last for a couple weeks since it may not make sense to hold them but perhaps that is better off as an open discussion topic
15:10:06 <TheJulia> #topic Review Action Items from the prior week
15:10:15 <TheJulia> I didn't get to my one and only action item, and it seems I have two now. :)
15:10:24 <TheJulia> #action TheJulia follow-up on third party CI
15:10:41 <TheJulia> #action TheJulia email the foundation regarding case studies
15:11:34 <TheJulia> ^^^ arne_wiebalck I have more motivation to do that since redhat's submission fell in my lap this past week.
15:11:50 <TheJulia> #topic Review subteam status reports
15:12:03 <TheJulia> #link https://etherpad.openstack.org/p/IronicWhiteBoard
15:12:16 <TheJulia> Starting at line 273
15:13:24 <TheJulia> looks like we have some todo's from last week
15:13:29 * TheJulia suspects everyone was really busy last week
15:14:29 <arne_wiebalck> TheJulia: :)
15:14:49 <TheJulia> Node retirement spec merged \o/
15:15:03 <iurygregory> \o/
15:15:18 <arne_wiebalck> \o/
15:16:18 <TheJulia> tzumainn: I think I saw you were going to take on adding policy code around the node.owner field?
15:17:36 <dtantsur> this ^^ plus I suggest banning changing node.owner if an owned allocation exists
15:17:39 <dtantsur> * for this node
15:17:46 <TheJulia> I think that really covers it for things that are getting attention right now.
15:17:58 <tzumainn> TheJulia, yep!
15:18:04 <TheJulia> dtantsur: reasonable... I think
15:18:28 <TheJulia> o/ stendulker
15:18:34 <stendulker> o/
15:18:50 <stendulker> Sorry, I'm late.
15:18:58 <TheJulia> No worries
15:19:16 <TheJulia> Are we good to proceed on from the reviewing statuses and shift to priorities for this coming week?
15:19:43 <iurygregory> I think we are
15:19:46 <rpittau> let's
15:20:18 <rajinir> iurygregory,TheJulia> ,rpioso, dtansur Dell CI is down, I have updated the Ironic Whiteboard. There seems to be some networking issue with our labs. We are working on it.
15:20:27 <dtantsur> thanks rajinir
15:20:33 <iurygregory> rajinir, tks!
15:20:35 <rloo> o/ sheepish...
15:20:53 <dtantsur> stendulker: hi, do you know the HPE CI status? could you update the whitboard if so?
15:21:31 <TheJulia> Okay, onward to Priorites!
15:21:37 <TheJulia> #topic Priorities for the coming week
15:21:42 <stendulker> yes, its down. Will update. There seems some issue with the devstack image creation. Newly cerated images do not work. It does not take static image.
15:21:48 <TheJulia> #link https://etherpad.openstack.org/p/IronicWhiteBoard
15:22:07 <stendulker> Need thsi patch for iLO CI https://review.opendev.org/697953
15:22:07 <patchbot> patch 697953 - ironic - Fixes issue with checking whether ISO is passed - 1 patch set
15:22:29 <TheJulia> Starting at line 169
15:24:07 <dtantsur> I could use reviews on https://review.opendev.org/#/c/697451/
15:24:08 <patchbot> patch 697451 - ironic - Correct power state handling for managed in-band i... - 1 patch set
15:24:35 <TheJulia> Looks like the attributeerror fix is now into stable backports. link added
15:24:42 * dtantsur added under managed inspection
15:24:46 <TheJulia> dtantsur: thanks
15:25:27 <dtantsur> also https://review.opendev.org/#/c/698011/ will hopefully prevent us from breaking rescue again
15:25:28 <patchbot> patch 698011 - ironic-tempest-plugin - Actually test rescue in the standalone job - 1 patch set
15:25:44 <dtantsur> (also added)
15:25:53 * iurygregory adds links for ipa and ipa-builder patches =)
15:27:20 <TheJulia> dtantsur: thanks!
15:27:36 <TheJulia> I went ahead and listed a needing feedback item on a WIP I have up
15:28:10 <TheJulia> Adding IPA<-> Conductor authentication, the very first patch.. Once the gate is happier, it should actually pass CI without any issues.
15:28:38 <TheJulia> Does anyone have anything else that needs to go on this list?
15:29:22 <kaifeng_> i have a patch on the stable branch
15:29:24 <kaifeng_> https://review.opendev.org/697775
15:29:24 <patchbot> patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set
15:30:37 <rloo> kaifeng_: is there a bug/story associated with that? it isn't a backport?
15:31:01 <dtantsur> CI change, so a story is optional
15:31:20 <kaifeng_> it's not a backport, but as I understand it, is required for the grenade on master
15:31:29 <rloo> dtantsur: ah, i didn't look 'below the fold' (Just read the commit stuff.)
15:31:36 <TheJulia> Seems reasonable, but I'm wondering if master has been changed
15:31:49 <dtantsur> kaifeng_: it seems that master requires the same change
15:31:50 <TheJulia> oh, that actually makes sense then because of grenade
15:32:11 <kaifeng_> the change on master is https://review.opendev.org/697634
15:32:12 <patchbot> patch 697634 - ironic - Fix ipxe interface to perform ipxe boot without ip... - 2 patch sets
15:32:39 <dtantsur> kaifeng_: could you split them apart then?
15:32:44 <kaifeng_> which actually fix the issue in the commit message
15:32:48 <dtantsur> I mean, apply the CI change to master, backport it, then apply this?
15:33:02 <dtantsur> (and probably also backport)
15:33:07 <TheJulia> the ironic change was already made
15:33:11 <TheJulia> in the patch kaifeng_ just linked
15:33:28 <dtantsur> TheJulia: yep, but it needs to pass the CI
15:33:39 <dtantsur> and it won't until we patch train. which needs patching master, then backporting.
15:33:42 <dtantsur> am I missing something?
15:33:54 <rloo> i'm confused. why doesn't the change in master (https://review.opendev.org/#/c/697634/) have a story/bug associated with it?
15:33:54 <patchbot> patch 697634 - ironic - Fix ipxe interface to perform ipxe boot without ip... - 2 patch sets
15:34:19 <TheJulia> It is a chicken/egg situation with grenade, the fix to the older branch needs to land first
15:34:20 <rloo> it has a release note?
15:34:35 <dtantsur> TheJulia: not if you split out the CI part only, I think
15:34:43 <dtantsur> at least I wonder if it has been tried
15:34:46 <TheJulia> Oh yeah, possibly
15:34:53 <TheJulia> as the first of two changes
15:34:55 <rloo> i'm ok with a fix in older branch being needed, to support a new change. but i don't know why the new change doesn't have much info...
15:35:09 <rloo> if there is a story, the fix in older branch can reference it
15:35:34 <kaifeng_> the thing is ipxe without ipxe_enabled=True is not working
15:36:04 <rloo> kaifeng_: is there a story/bug to track that?
15:36:05 <kaifeng_> so my thought is to make decouple that dependency
15:36:06 <rpittau> kaifeng_: I think it would be easier if we could reference a Story/Task to it
15:36:34 <dtantsur> I think I understand what kaifeng_ is trying to do, but I'd still prefer https://review.opendev.org/#/c/697775/ applied to master first, then to train.
15:36:35 <patchbot> patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set
15:36:39 <TheJulia> Yeah, a story as a bug would be good because it shouldn't be a broken case, but we likely merged something along the way that broke it
15:36:41 <rloo> and i am more confused. if ipxe_enabled = False, why should ipxe work?
15:36:44 <kaifeng_> no problem, i will file a story for the patches
15:36:50 <dtantsur> rloo: they're orthogonal
15:37:11 <dtantsur> ipxe_enabled only affects the older pxe interface (because of backward compatibility)
15:37:12 <rpittau> kaifeng_: thanks
15:37:18 <rloo> dtantsur: oh. i need to refresh my memory wrt ipxe_enabled...
15:37:26 <TheJulia> Or at least, that is how it is supposed to work :)
15:37:31 <dtantsur> driver composition is my speciality :D
15:37:31 <rloo> dtantsur: ah, thx for explaining.
15:37:39 <TheJulia> Okay, I think we can move on
15:37:41 <rloo> kaifeng_: ^^ that info in a story/bug would have helped me :D
15:38:01 <kaifeng_> the patch on master won't pass unless the patch merged in train..
15:38:02 <TheJulia> Are we good to proceed
15:38:08 <TheJulia> arne_wiebalck: anything baremetal sig wise to bring up?
15:38:14 <arne_wiebalck> TheJulia: no
15:38:17 <kaifeng_> unless we add additional migration path
15:38:29 <rloo> kaifeng_: open a story, link the patches to the story so it is clear why the train patch is needed.
15:38:34 <dtantsur> kaifeng_: why won't it?
15:38:50 <TheJulia> Lets continue this in open discussion
15:38:53 <dtantsur> https://review.opendev.org/#/c/697775/ does not break any compatibility?
15:38:53 <patchbot> patch 697775 - ironic (stable/train) - Explicitly enable ipxe as boot interface when it's... - 1 patch set
15:38:55 <dtantsur> yeah, okay
15:39:14 <TheJulia> So jumping directly to RFE review if there are no objections
15:39:22 <kaifeng_> ok
15:39:29 <TheJulia> #topic RFE Review
15:39:46 <TheJulia> We had a new RFE proposed, to enable scoping of introspection rules
15:39:51 <TheJulia> #link https://storyboard.openstack.org/#!/story/2006995
15:40:14 <arne_wiebalck> We filed an RFE to scope inspection rules. This is meant to ease the handling od multiple deliveries and to avoid purging the rules all the time.
15:40:15 <dtantsur> I influences this proposal, so consider me +1
15:40:19 <dtantsur> * influenced
15:40:39 <TheJulia> This seems reasonable
15:40:43 <arne_wiebalck> s/od/of/
15:41:14 <arne_wiebalck> We put this up here to see if there are any major concerns.
15:41:14 <rpittau> seems like a needed change
15:41:37 <arne_wiebalck> The other reason is: does this require a spec?
15:41:52 <kaifeng_> does the new field in properties interfere with scheduling?
15:42:10 <arne_wiebalck> kaifeng_: it should not
15:42:10 <TheJulia> that is a really good question kaifeng_
15:42:40 <TheJulia> If someone is using the json matching scheduler, I suspect it would... but I think that is not advisable
15:43:19 <dtantsur> given that node.properties is user-updateable, we cannot guarantee anything about it
15:43:30 <TheJulia> I don't really see an issue with the RFE. I feel like I'm lacking a piece of the puzzle
15:43:34 <TheJulia> dtantsur: exactly
15:43:36 <dtantsur> I'm fine with using node.extra as well, but we tend not to interpret its values anyhow
15:44:15 * arne_wiebalck wasn't aware that users can update node.properties
15:44:19 <TheJulia> I'd prefer we leave node.extra to operators
15:44:29 <rloo> i don't think we should use node.extra. isn't that for the user? I don't think we should code assuming anything in .extra
15:44:33 <TheJulia> arne_wiebalck: users as in the running user or user as in admin api user
15:44:47 <arne_wiebalck> TheJulia: ah, that makes sense, thanks
15:45:31 <arne_wiebalck> Shouldn't scheduling be based entirely on the resource_class (and traits)?
15:45:40 <kaifeng_> one exception is inspector would put an autodiscoverd=True to the node.extra during discovery, looks like we are short of fields :)
15:45:52 <mgoddard> arne_wiebalck: can you do this without code changes?
15:46:09 <arne_wiebalck> mgoddard: ?
15:46:16 <mgoddard> just add an additional first condition, node.properties == 'constant'
15:46:45 <mgoddard> missing some bits there...
15:47:00 <mgoddard> node.properties['inspection_scope'] == 'constant'
15:47:21 <arne_wiebalck> we could do this by tweaking the rules, seems awkward, though
15:47:49 <mgoddard> ok
15:48:26 <arne_wiebalck> with inspection of active nodes, there might be additional use cases
15:48:35 <dtantsur> if you allow some bikeshedding, I'd just the property to inspection_rules_scope
15:48:41 <dtantsur> s/just/change/ (wut)
15:48:42 <arne_wiebalck> like regularly checking the nodes
15:49:00 * TheJulia thinks we've reached Open Discussion
15:49:05 <arne_wiebalck> dtantsur: sounds good
15:49:33 <TheJulia> Moving on then!
15:49:37 <TheJulia> #topic Open Discussion
15:49:54 <rpioso> I have an open discussion thing. Has anyone successfully used devstack to deploy on real hardware with local boot?
15:49:55 <TheJulia> So there was the previous discussion about grenade + ipxe
15:50:22 * rpioso has wrestled with that the past two weeks-ish.
15:50:23 <dtantsur> I'd also follow-up on previously discussed https://storyboard.openstack.org/#!/story/2006910 and https://storyboard.openstack.org/#!/story/2006936
15:50:37 <TheJulia> I also wonder if we should call next week's meeting the last for the year
15:50:45 <dtantsur> TheJulia++
15:50:59 <rpittau> TheJulia: we probably should
15:51:54 <rloo> ++ to last meeting next week
15:52:05 <TheJulia> dtantsur: that latter story, they would expect us to magically make work to account to handle someone else's behavior change. I'm okay with the change, but... ugh
15:52:14 <TheJulia> well, changes, it is going to be in multiple places
15:52:20 <dtantsur> yeah, it's not ideal
15:52:51 <TheJulia> rpioso: to your question, it has been... a while for me. I'm actually 3d printing a new case for a new lab router so I can rebuild the home lab.... #geekswithcircuitboardsandnocases....
15:52:53 <dtantsur> I'm also concerned about cost/profit balance of this work
15:53:22 <dtantsur> one benefit is that we can probably save some bandwidth when streaming images (with the direct deploy)
15:53:27 <TheJulia> dtantsur: internally, we have way too much on our plate to guarentee that in a short term anyway
15:53:43 <dtantsur> also true
15:53:58 <dtantsur> I think the deployment API brings much more to the project than that
15:54:08 <dtantsur> so I'd love some feedback on it as well
15:54:11 <TheJulia> absolutely agree
15:55:37 <kaifeng_> well i am not sure i can finish the ipxe in 5 minutes
15:57:23 <dtantsur> rpioso: I thinks you have two separate problems: devstack and local boot. They seem orthogonal to me.
15:57:52 <rpioso> dtantsur: I'm all eyes OO
15:58:18 <TheJulia> kaifeng_: I likely need a little more context, but I suspect your kind of heading in the right path because without the pre-existing config then the "upgraded" code needed post upgrade will fail.
15:58:18 <dtantsur> rpioso: well, without even knowing symptoms it's hard to say what you have. But you can at least try to bisect the problem.
15:58:37 <TheJulia> kaifeng_: but if you merge devstack changes in separately, it will liekly also be needed
15:58:38 <dtantsur> If you have problems with local boot, you can likely rule out the "devstack" side of your question
15:59:07 <rpioso> dtantsur: I believe I've gotten successful devstack ironic node deployments with network boot.
15:59:32 <kaifeng_> TheJulia: i am raising for some reviews so i can make sure taking the right path.
15:59:40 <TheJulia> well, "devstack" all depends on the settings and image content being used too...
15:59:52 <kaifeng_> the issue is ipxe didn't work without the ipxe configuration option enabled, change in master branch fixed that, but nodes enrolled in train was with the pxe interface
16:00:19 <kaifeng_> while we actually used pxe+pxe_enabled as ipxe by default
16:00:22 <dtantsur> rpioso: does the image you're using have grub2?
16:00:24 <TheJulia> kaifeng_: but you also have two grenade jobs to pass. One Train->Master+patch, the other master->master
16:00:33 <rpioso> dtantsur; The most recent symptom is that the tinycore IPA ramdisk can't get the config. TheJulia believes tinycore may not have the needed Intel NIC driver and that tinycore is used in VM environments, not real HW.
16:00:43 <dtantsur> correct
16:00:46 <TheJulia> err, master->master+patch
16:01:18 <dtantsur> I think we can wrap up the meeting and continue with discussions
16:01:19 * etingof just used tinycore on Dell R640
16:01:23 <kaifeng_> no, i didn't turn ipxe_enabled=False, so either ipxe, or pxe works
16:01:37 <kaifeng_> sounds a bit tricky
16:01:42 <rpioso> dtantsur: Which images should I use? And which devstack settings are needed.
16:01:43 <TheJulia> I'm going to end the meeting, we can carry on with discussions
16:01:44 <dtantsur> etingof: it depends on your luck. if no fancy drivers are needed - cool.
16:01:53 <TheJulia> Thanks everyone!
16:01:57 <TheJulia> #endmeeting