15:00:42 #startmeeting ironic 15:00:43 Meeting started Mon Feb 10 15:00:42 2020 UTC and is due to finish in 60 minutes. The chair is TheJulia. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:44 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:44 o/ 15:00:46 \o 15:00:47 The meeting name has been set to 'ironic' 15:00:50 Good morning everyone! 15:00:56 o/ 15:00:58 o/ 15:01:00 o/ 15:01:00 o/ 15:01:01 o/ 15:01:02 \o 15:01:11 Our agenda this morning can be found on the wiki - https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting 15:01:16 #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting 15:01:26 o/ 15:01:27 o/ 15:01:37 o/ 15:01:41 #topic Announcements / Reminders 15:02:13 It seems like the mid-cycle is definitely on! 15:02:18 #info OpenStack Ironic Mid-cycle Meetup, 25-26 February 2020, CERN, Geneva, Switzerland 15:02:25 #link https://etherpad.openstack.org/p/ironic-ussuri-midcycle 15:02:32 please tell arne_wiebalck if you're coming and provide the necessary information ASAP! 15:03:02 Arne messaged me to let me know he closed registrations yesterday. If anyone is still interested in attending arne_wiebalck needs to know ASAP 15:03:07 * dtantsur wonders how many people do come in the end 15:03:18 * rpittau will be there 15:03:23 rpittau: w00t! 15:03:33 me and my wife actually :D 15:03:37 ditto :) 15:03:41 :) 15:03:45 He also indicated the tour is full based upon registrations, so if your registered and will not be joining us, arne_wiebalck needs to know. 15:03:48 triple! 15:04:08 lovely :) 15:04:14 #info Opendev + PTG Vancouver, Dates: Mon, Jun 8, 2020 - Thu, Jun 11, 2020 15:04:37 TheJulia: have you told the foundation anything about our presence? 15:05:04 They are looking for track chairs and hardware automation is something they are interested in. If anyone is interested in helping out, please feel free to reach out to me. 15:05:33 I'd not mind helping, but I don't know if I'm coming (and won't know soon) 15:05:38 dtantsur: I have not, but I would assume that they wouldn't be unaware as some of the staffers do join the channel occassionally for our meetings 15:06:05 * rpittau wouldn't mind helping and definitely wouldn't mind some maple syrup 15:06:05 Also, if anyone is in the eastern US, there is a conference that is looking for topics 15:06:10 * TheJulia looks for the link 15:07:03 #info 2020 Open Cloud Workshop - March 2-3 15:07:04 ping me if you need an ironic talk in the Central Europe :) 15:07:08 #link https://docs.google.com/forms/d/e/1FAIpQLSen4GUE_9MkJIomYazU933k8lRnNpdJgU-cwbpC1DVZn2upsQ/viewform 15:07:33 That closes in just a few days 15:07:53 Anyway, Does anyone have anything else to announce or remind us of this morning? 15:08:02 dtantsur, pycon in Ostrava? ;) 15:08:13 Ostrava, oh my :D 15:08:21 dtantsur: I feel like maybe we should somehow demo kexec in Berlin :) 15:08:21 (very much Central Europe, as you wish!) 15:08:33 TheJulia: mmmmm :) 15:08:49 of course, patches merge conflicted long ago 15:08:55 and not yet tested 15:09:11 kexec + fast-forward, going to be hot! 15:09:13 We had no action items last week, so it seems like we can skip past that 15:09:22 mmmm, fast-track 15:09:38 Add in the ability for an instance to be redeployed from a running instance and the supercomputer folks will drool 15:09:52 that sounds much trickier though 15:10:05 Less than you'd think, but we can discuss it in a few weeks 15:10:13 sure 15:10:17 Anyway, moving on 15:10:24 damn, we should have requested 3 days in CERN 15:10:29 #topic Review subteam status reports 15:10:32 dtantsur: doh! 15:10:53 what about a follow-up hiking with technical discussions? :D 15:10:57 Oh, one annoucement, I'm taking the rest of the week off after the midcycle 15:11:03 ditto ^^^ 15:11:36 etingof, why is node starting cleaning if i have automated_clean=false on ironic conductor? 15:11:37 The following week I may also be busy, unsure yet 15:11:48 i see that on log: Entering new state 'cleaning' in response to event 'provide' on_enter /usr/lib/python3.6/site-packages/ironic/common/states.py:300ESC[00m 15:11:50 Anyway! 15:11:52 #link https://etherpad.openstack.org/p/IronicWhiteBoard 15:12:01 Starting at line 160 15:12:06 err 15:12:09 yolanda: it always walks the state machine, I think. even if cleaning is no-op. 15:12:12 no, starting at 281 15:14:19 I'm really worried about the deploy steps stuff landing super late int he cycle 15:14:47 I think it would be good if we could get some folks to at least perform a review of the PoC patches? 15:14:48 thoughts? 15:14:54 I'm sure mgoddard would be happy 15:14:57 I should have done it long ago 15:15:03 * dtantsur is sorry 15:15:25 c'est la vie 15:15:48 * TheJulia needs to work on her very minimal french 15:16:10 wsme/flask/etc, does anyone have an update? 15:16:36 no update at my side, no one review my code change yet 15:16:42 for wsme 15:16:59 675853? 15:17:03 yes 15:17:15 didn't we talk about wsme last week? I forgot how it ended... 15:17:20 TheJulia: for Flask, there was not a mutual understanding that we were going that direction btw 15:17:30 My change was for json schema 15:17:32 oh, okay, I was hopful there 15:17:33 wasn't there? I think it has been decided many times 15:17:34 or what the exact plan is 15:17:39 Last week was others 15:17:51 * TheJulia feels like she has opened a "can of worms" 15:17:58 worms, yummy! 15:18:01 lol 15:18:05 dtantsur: yeah, but I remember the last discussion where the execution was not clear for someone 15:18:10 gummy worms ? 15:18:25 well, one thing is clear: we need to get off wsme 15:18:49 pecan is painful to use, but we can probably stay on it 15:18:54 It feels like we explicitly need to have an explicit plan gotten together since we've let it be nebulous to try and encourage progress as opposed to try design it forever 15:18:58 * dtantsur assumes pecan is not dead 15:19:04 Umm... 15:19:08 * TheJulia gets a parrot out 15:19:23 Or would Doctor Mccoy work? 15:19:24 last commit in August: https://github.com/pecan/pecan 15:19:38 * dtantsur as usual misses the references :) 15:19:42 Well that is good 15:19:53 dtantsur: go watch the original star trek and look for "its dead jim" 15:20:00 oh gosh :D 15:20:01 anyway we're off topic 15:20:17 * dtantsur won't admit he hasn't watched star trek for $insert_silly_reasons 15:20:18 do we want to get that to the mid-cycle instead ? 15:20:36 rpittau: the schedule is packed and we won't have a quorum anyway 15:20:37 DHCP-less has had good discussions, I feel like we just need to focus it a little bit more and everyone will be happy 15:20:43 mmmm right 15:20:44 ++ 15:21:05 * etingof is focusing on that 15:21:12 Looks like uefi software raid is starting to get some review traction 15:21:16 and activity which is awesome 15:21:42 Node retirement is done, at least in ironic. I wonder if there is anything besides openstacksdk? 15:21:48 or do we consider that out of scope? 15:22:03 Well, I take that bcak 15:22:08 the osc commands have not merged https://review.opendev.org/#/c/703982 15:22:09 patch 703982 - python-ironicclient - Add support for retired{_reason} fields. (MERGED) - 5 patch sets 15:22:22 I've been slowly poking people towards starting to take openstacksdk in scope 15:22:32 note that, being an SDK core, I'm clearly biased 15:23:01 * TheJulia adds the OSC patch to the retirement support list under priorities since the other two have merged 15:23:26 i had some questions about the client part of the retired * stuff (but i commented in the PR, after it merged though.) 15:23:48 i thnk we should add that all the ironic API stuff ought to be avail in the openstacksdk. 15:24:04 rloo: I'll respond to your comments 15:24:07 That kind of seems I concur, it seems reasonable to me, as long as we're able to time/committment wise 15:24:19 Anyway, I think we're good to move on? 15:24:28 (wonder if we should update our spec template to mention that, if it hasn't been updated already) 15:24:32 thx dtantsur 15:24:39 rloo: good point 15:24:45 afaik, it has not been updated 15:24:54 TheJulia: i'll offer to do that :D 15:25:00 rloo: thanks! 15:25:24 Moving on! 15:25:43 #topic Deciding on priorities for the coming week! 15:25:53 #link https://etherpad.openstack.org/p/IronicWhiteBoard 15:26:03 Starting at 160 15:26:04 rloo: tl;dr what we did with maintenance was bad and I feel bad for not preventing it :( 15:26:20 * TheJulia starts by removing merged things which is a number of things 15:26:26 dtantsur: OH. (talk later) 15:26:48 I can offer anyone a good rant with my API SIG hat on :) 15:27:08 Any objection to removing the reconfiguration spec from the list of things to review? 15:27:14 not from me 15:27:36 Ilya Etingof proposed openstack/sushy master: Fix 'None' field value processing https://review.opendev.org/703652 15:28:45 +1 on removing reconfiguration spec from list 15:28:54 iurygregory: looks like you've got a few things you need reviewed :) 15:29:09 TheJulia, yeah XD 15:29:26 Ilya Etingof proposed openstack/sushy master: Relax required Redfish fields handling https://review.opendev.org/703825 15:29:27 * iurygregory finally got redfish CI job working =D 15:29:47 TheJulia: I've checked the agent_token stuff as promised. It seems to need another rebase :( 15:29:52 * iurygregory the i-p-e =) 15:29:57 tzumainn: any node ownership stuff needed? 15:30:05 are we preparing to do some releases? I see some PRs related to that. if so, should they go in this week's list? 15:30:07 dtantsur: I'll rebase it either during the next meeting or after 15:30:23 rloo: already on the list 15:30:30 I'll need review on https://review.opendev.org/#/c/706863/ once I'm done testing it. It's quite serious, the DIB-IPA jobs don't test IPA. 15:30:31 patch 706863 - ironic - Actually use ironic-python-agent from source in so... - 1 patch set 15:30:45 in the wrong place of course 15:30:47 * TheJulia shuffles the list 15:31:00 TheJulia: ah, that explains it, cuz I stared a bit more and didn't see ! 15:31:01 dtantsur: I'll add it to my list 15:31:05 thx 15:31:11 TheJulia, hi! just the lessee implementation patch 15:31:36 tzumainn: do you have the link handy 15:31:47 ah, yep! one sec 15:32:03 https://review.opendev.org/#/c/706864/ 15:32:03 patch 706864 - ironic - Add node lessee field - 1 patch set 15:32:09 I think the list is good 15:32:12 I also put it on the whiteboard somewhere, maybe in the wrong place 15:32:55 The ipa bugs I found this week seem to relate to our standalone test failure rate, I'll try and get that patch fixed and then some of the approved stuff may have more luck getting past the raid tests 15:33:44 Does anyone have anything to add to the list? 15:33:54 https://review.opendev.org/706445 can use some reviews 15:33:55 patch 706445 - ironic - Automatic port allocation for the serial console - 2 patch sets 15:34:04 kaifeng: already added :) 15:34:10 Thanks for that by the way! 15:34:17 TheJulia: thanks :D 15:34:45 * TheJulia takes silence as agreement with the list and to proceed 15:35:02 ++ 15:35:07 #topic Discussion 15:35:09 We have three topics 15:35:18 (Which is why I'm trying to move us through the meeting quickly) 15:35:32 Topic #1, do we cancel the IRC meeting the week of the midcycle? 15:35:52 at least the two of us will be travelling, right? 15:35:53 that's probably a good idea 15:35:55 yes 15:36:18 I think my train arrives in geneva around 4 PM local, so i may still be on the move to the hotel 15:36:34 we might be on the same train :) 15:36:54 Any objections to canceling? 15:37:12 * TheJulia gives a chance for objections before moving along 15:38:08 rpittau: Gare de Lyon departure? 15:38:11 yeah 15:38:17 very likely then! 15:38:19 * dtantsur flies in 15:38:22 ++ to cancel =) 15:38:23 Okay, moving on! 15:38:41 * iurygregory arrives at 5PM 15:38:45 #2, PTG in Vancouver? Is anyone interested in attending? 15:38:52 * rpittau very 15:38:58 (maple syrup) 15:39:00 They are polling for project space and I likely need to respond (soon) 15:39:12 heh 15:39:21 I'm interested, not sure about budget and visa stuff 15:39:30 That is fair 15:39:35 same for the budget :/ 15:40:01 I planned for ~12 people to appear in Shanghai and 15 joined us, so I'm thinking slightly more seats 15:40:15 Interested. It will likely depend on travel budget. 15:40:16 yeah, a much higher NA presence 15:40:45 Yeah 15:40:51 Okay, I'll respond to the foundation 15:41:28 Topic #3! That ye olde bug about users being able to orphan vifs in neutron that prevent the node from being deployed/reused/etc due to duplicate mac addresses 15:41:42 #link https://review.opendev.org/665835 15:41:42 patch 665835 - ironic - Block port deletions where vif is present - 8 patch sets 15:42:43 \o 15:42:47 sorry, been in an intervie 15:43:00 Does anyone actually care about this bug? Because there has been some back and forth on requirements and the intent was to guard against humans doing silly things and as a result perceptions have differed and it has been slightly frustrating 15:43:53 If at least one person cares, I'll try and clarify things further and see if I can make reviewers happy and operators have suffient guards/information 15:43:54 * rpioso wishes mgoddard good luck ;-) 15:44:16 TheJulia: do you aware the reasons why the vif is not removed? 15:44:17 rpioso: luck would be better directed at the candidate :) 15:44:24 I suspect that guarding deletions becomes more important with multitenant access 15:44:42 kaifeng: node put in maintenance state and ports and or the node itself removed. 15:44:54 it seems like a valid bug and i think we ought to address it. 15:45:41 upon removal, if there is any sort of vif, even an internal vif for ironic, that port is orphaned in neutron 15:46:11 s/that port/ the neutron vif/port/ 15:46:40 I think we found it downstream when someone decided that they were just going to delete everything due to being frustrated and then couldn't ever deploy again 15:47:04 Anyway, I'll go ahead and carry it forward 15:47:21 based on my last comment, I think I'm good with the idea. Just that, if I recall, this adds a new method to our NetworkInterface API, so I wanted that to be somewhat 'solid'. 15:48:02 solid as in we're not going to change its behavior after adding? 15:48:16 I suspect if I just rename the method people might become happy 15:48:19 truthfully 15:48:29 i wasn't thinking that but solid wrt description etc so if someone subclasses with their own network, they know what that method should do 15:49:05 Regarding moving on, It doesn't look like we have anything to visit for the Bare Metal SIG or RFEs, so I'll move us to Open Discussion. 15:49:11 ie, what does 'get_current_vif_with_tag()' mean/do? Not that clear to me. I'm fine if that is clear to others. 15:49:33 hmm 15:49:47 (it is possibly more than one vif) 15:49:49 TheJulia: i am thinking in cases ironic failed to remove vif from neutron, it seems we have no other ways to remove it from ironic governance 15:49:56 rloo: thank you, I think I understand your feedback now. 15:50:06 TheJulia: :) Sorry if it wasn't clear. 15:50:20 rloo: you likely were and I just didn't read it right 15:50:34 TheJulia: no worries, glad you asked! :) 15:51:27 kaifeng: Yeah, if someone short circuits processes.. .we can search by mac and possibly replace ports but unique macs may not always be... unique. 15:51:34 #topic Open Discussion 15:51:43 * TheJulia feels trying to auto-resolve things may be better eventually 15:51:55 with a giant "this may do undesirable things" warning 15:52:14 a periodic task to clean up orphaned VIFs? :) 15:52:15 Anyone have anything to discuss today? 15:52:27 hi! quick question - has anyone ever tried non-admin deployments upon a node using standalone Ironic? I tried it for a bit and ran into the issue of having to create Neutron ports, which required admin API access 15:52:48 ehmmm, creating ports shouldn't require admin 15:52:52 dtantsur: reconciling our ports versus neutron's ports seems like it could be... slightly complex. 15:53:04 TheJulia: since when are we afraid of complexity? :D 15:53:22 I think the part that required admin was associating a neutron port with a mac address 15:53:29 oh wow 15:53:32 dtantsur: I'm more worried about interaction latency, we've seen how data syncs with neutron can be painful 15:53:46 yes, adding a mac is an admin action 15:53:49 maybe it's not a shared network 15:53:51 ouch 15:54:10 Are folks open to adding the driver for the Intel I350 NIC to the Tiny Core IPA ramdisk? 15:54:11 It always has been... We likely need to revisit our neutron code then 15:54:37 rpioso: only if it doesn't change the resulting size substantially IMO 15:54:49 rpioso: how big is that ? 15:54:54 rpioso: I am not since it is for VM testing and and I looked for other drivers and it doesn't seem to really have much of a selection for network drivers 15:54:57 TheJulia, tzumainn, then we may need to use an admin token for this particular operation 15:55:20 dtantsur: indeed, I think we did originally but session code changes likely caused us to change it to re-using the nova request token 15:55:21 yeah, essentially what TheJulia said. we need to invest into DIB images as our hardware solution. 15:55:48 tzumainn: okay, so it's probably a bug. let's start with collecting the details and filing it. 15:55:57 dtantsur: ++ on both items 15:56:05 dtantsur, okay, can I ask you a few more questions post-meeting? 15:56:06 TheJulia: It has the driver for Intel X710 10/40 Gbps NIC. The I350 is a 1 Gbps port. 15:56:10 tzumainn: sure thing 15:56:47 TheJulia: Dell EMC's lab equipment has an I350 connected to the provisioning network. 15:57:06 rpioso: do you envision any problems with using DIB images? 15:57:10 rpioso: I'm afraid the drivers just don't exist for tinycore as far as I've seen 15:57:40 TheJulia, I'm wondering if the fixes in ironic-prometheus-exporter would be ok to backport to stable/train =) 15:57:42 3 minute warning 15:57:43 by the way, TinyCore 11 is out :) 15:57:51 dtantsur: ohhh 15:57:56 iurygregory: should be fine 15:57:59 iurygregory: "fixed" usually are 15:58:09 dtantsur: I have had probs using it and RDO's IPA ramdisk to Redfish vmedia boot with the UEFI / local / partition image workflow. 15:58:34 dtantsur: tinyipa appears to be close to working. 15:58:39 dtantsur, also the redfish job (not a fix but a real testing job) XD 15:59:00 dtantsur: For at least RDO's IPA ramdisk, the kernel crashes. 15:59:17 rpioso: well, it should be fixed. that's what the customers will be using. 15:59:30 iurygregory: if it fixes issues or makes operational use better/more reasonable on that branch, It should be fine when it comes to releasing it 15:59:34 iurygregory: any CI changes can be backported with a reasonable motivation. 15:59:43 the policies only affect the actual code we ship 15:59:48 awesome =) 16:00:02 dtantsur: We hope to explore that at Geneva. Meanwhile, it would be helpful to have something we can inform operators works. 16:00:30 dtantsur: So far, nothing has worked for UEFI / local / partition image. 16:00:36 we, as ironic upstream, don't really support using tinyIPA in production 16:00:38 The policies are also slightly flexible when operators are kept in mind and we can do appropriate versioning 16:01:02 rpioso: we have been doing this for a while in TripleO.. 16:01:02 dtantsur: Which do we support? 16:01:10 DIB images are now official for production 16:01:16 dtantsur: Who is "we"? 16:01:27 rpioso: "in TripleO" so ~ Red Hat 16:01:33 dtantsur: what rpioso indicated is that his grub, which coems from his ubuntu devstack host is spitting out an error about the ramdisk being too being, apparently before crashing. I suspect the ramdisk is getting truncated on his machine and I bet the kernel crashes. 16:01:39 We're at time, Thanks everyone! 16:01:46 dtantsur: With Redfish vmedia boot? 16:02:02 rpioso: that's the bit you didn't mention :) worth asking yolanda 16:02:05 thx TheJulia! 16:02:16 #endmeeting