| janders | good morning Ironic o/ | 00:03 |
|---|---|---|
| *** tosky has quit IRC | 00:09 | |
| iurygregory | morning janders o/ | 00:12 |
| *** sshnaidm is now known as sshnaidm|off | 00:12 | |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/victoria: Add fstab pointer to EFI partition https://review.opendev.org/c/openstack/ironic-python-agent/+/782343 | 00:26 |
| stevebaker | backporting ^^ to ussuri will be hard | 00:26 |
| stevebaker | let-alone train | 00:27 |
| iurygregory | yeah =( | 00:27 |
| stevebaker | also I'm tempted to backport token support to train so I can test stable/train IPA images with a recent ironic. But I might just be delirious due to this cold | 00:35 |
| janders | I'm addressing Dmitry's feeback regarding https://review.opendev.org/c/openstack/ironic/+/774391 (merged). Do I submit the follow up change 1) in any special way (how) or 2) is follow up a completely separate patch and no need to set anything special on it? | 01:05 |
| iurygregory | I know gerrit now has a "Follow-Up" button (I never used) | 01:13 |
| iurygregory | normally what I do is just to create a new change and say in the commit message that is a follow-up for I3debcd1f32a2627dafd8456ec73a71fc7c402ebb | 01:13 |
| iurygregory | I hope this helps janders ^ | 01:14 |
| janders | it does! thanks heaps iurygregory | 01:15 |
| iurygregory | np | 01:15 |
| openstackgerrit | Merged openstack/ironic master: Add Redfish RAID interface to idrac HW type https://review.opendev.org/c/openstack/ironic/+/774433 | 01:48 |
| openstackgerrit | liuyuanfeng proposed openstack/ironic master: when the port_date is dict,port date from neutron doesnot have is_smartnic property https://review.opendev.org/c/openstack/ironic/+/782344 | 01:53 |
| *** jamesdenton has quit IRC | 02:13 | |
| *** jamesden_ has joined #openstack-ironic | 02:13 | |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: Option to enable bootloader config failure bypass https://review.opendev.org/c/openstack/ironic-python-agent/+/781569 | 02:20 |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: Refactor part of image module https://review.opendev.org/c/openstack/ironic-python-agent/+/782348 | 02:20 |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: Mount all vfat partitions before calling grub2 https://review.opendev.org/c/openstack/ironic-python-agent/+/782349 | 02:20 |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: More refactoring of the image module https://review.opendev.org/c/openstack/ironic-python-agent/+/782350 | 02:20 |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: Prevent broken partition image UEFI deploys https://review.opendev.org/c/openstack/ironic-python-agent/+/782351 | 02:20 |
| openstackgerrit | Steve Baker proposed openstack/ironic-python-agent stable/ussuri: Add fstab pointer to EFI partition https://review.opendev.org/c/openstack/ironic-python-agent/+/782352 | 02:20 |
| *** rcernin has quit IRC | 02:28 | |
| *** rcernin has joined #openstack-ironic | 02:39 | |
| *** bdodd has quit IRC | 03:57 | |
| *** viks____ has quit IRC | 04:20 | |
| *** mkrai has joined #openstack-ironic | 04:23 | |
| *** gyee has quit IRC | 04:26 | |
| *** tzumainn has quit IRC | 05:06 | |
| *** jamesden_ has quit IRC | 05:25 | |
| *** jamesdenton has joined #openstack-ironic | 05:25 | |
| *** rcernin has quit IRC | 05:25 | |
| *** rcernin has joined #openstack-ironic | 05:26 | |
| *** ricolin_ has joined #openstack-ironic | 06:17 | |
| *** jamesdenton has quit IRC | 06:20 | |
| *** jamesden_ has joined #openstack-ironic | 06:21 | |
| *** ricolin_ has quit IRC | 06:23 | |
| openstackgerrit | ankit proposed openstack/ironic master: Add security dashboard clean steps to ilo drivers https://review.opendev.org/c/openstack/ironic/+/782364 | 06:33 |
| openstackgerrit | Alexander Evseev proposed openstack/bifrost master: Add network mask to the dnsmasq config https://review.opendev.org/c/openstack/bifrost/+/782365 | 06:36 |
| *** frigo has joined #openstack-ironic | 06:43 | |
| *** frigo has quit IRC | 07:03 | |
| *** mkrai has quit IRC | 07:04 | |
| *** mkrai_ has joined #openstack-ironic | 07:05 | |
| *** moshiur has joined #openstack-ironic | 07:11 | |
| *** uzumaki has quit IRC | 07:17 | |
| *** uzumaki has joined #openstack-ironic | 07:22 | |
| *** mkrai_ has quit IRC | 07:23 | |
| arne_wiebalck | Good morning, ironic! | 07:28 |
| *** abhishekyadav has joined #openstack-ironic | 07:40 | |
| *** mkrai_ has joined #openstack-ironic | 07:43 | |
| *** rcernin has quit IRC | 07:46 | |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| rpittau | good morning ironic! o/ | 08:13 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-inspector master: Increase memory of ironic vm in grenade https://review.opendev.org/c/openstack/ironic-inspector/+/782371 | 08:18 |
| *** rcernin has joined #openstack-ironic | 08:23 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-prometheus-exporter stable/wallaby: Increase RAM for ironic vm https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/782128 | 08:24 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-prometheus-exporter stable/victoria: Increase RAM for ironic vm https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/782129 | 08:25 |
| *** anuradha1904 has joined #openstack-ironic | 08:32 | |
| *** sshnaidm|off has quit IRC | 08:34 | |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/wallaby: Fix --fields network_data https://review.opendev.org/c/openstack/python-ironicclient/+/782130 | 08:36 |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/victoria: Fix --fields network_data https://review.opendev.org/c/openstack/python-ironicclient/+/782131 | 08:36 |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/ussuri: Fix --fields network_data https://review.opendev.org/c/openstack/python-ironicclient/+/782132 | 08:36 |
| *** ociuhandu has joined #openstack-ironic | 08:51 | |
| *** ociuhandu has quit IRC | 08:59 | |
| *** lucasagomes has joined #openstack-ironic | 09:02 | |
| *** derekh has joined #openstack-ironic | 09:02 | |
| *** tosky has joined #openstack-ironic | 09:04 | |
| *** ociuhandu has joined #openstack-ironic | 09:09 | |
| *** ociuhandu has quit IRC | 09:09 | |
| *** ociuhandu has joined #openstack-ironic | 09:20 | |
| openstackgerrit | Mark Goddard proposed openstack/tenks master: DNM: test libvirt-host patch https://review.opendev.org/c/openstack/tenks/+/782383 | 09:25 |
| janders | hey arne_wiebalck and rpittau o/ | 09:25 |
| rpittau | hey janders :) | 09:27 |
| dtantsur | morning ironic | 09:44 |
| janders | good morning dtantsur o/ | 09:45 |
| *** abhishekyadav has quit IRC | 09:47 | |
| *** abhishekyadav has joined #openstack-ironic | 09:47 | |
| openstackgerrit | Dhuldev Valekar proposed x/sushy-oem-idrac master: Add the high-level reset iDRAC method https://review.opendev.org/c/x/sushy-oem-idrac/+/782254 | 10:00 |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Update idrac-redfish RAID release note https://review.opendev.org/c/openstack/ironic/+/782391 | 10:03 |
| *** uzumaki has quit IRC | 10:12 | |
| *** uzumaki has joined #openstack-ironic | 10:13 | |
| *** sshnaidm has joined #openstack-ironic | 10:13 | |
| *** sshnaidm is now known as sshnaidm|off | 10:13 | |
| *** tosky has quit IRC | 10:13 | |
| *** tosky has joined #openstack-ironic | 10:14 | |
| *** ociuhandu has quit IRC | 10:16 | |
| *** ociuhandu has joined #openstack-ironic | 10:22 | |
| *** ociuhandu has quit IRC | 10:28 | |
| *** uzumaki has quit IRC | 10:29 | |
| *** uzumaki has joined #openstack-ironic | 10:30 | |
| openstackgerrit | liuzhuangzhuang proposed openstack/ironic master: update grub2 file name https://review.opendev.org/c/openstack/ironic/+/780481 | 10:30 |
| *** uzumaki has quit IRC | 10:31 | |
| *** uzumaki has joined #openstack-ironic | 10:32 | |
| *** ociuhandu has joined #openstack-ironic | 10:38 | |
| *** uzumaki has quit IRC | 10:44 | |
| *** rcernin has quit IRC | 10:46 | |
| *** mkrai_ has quit IRC | 10:47 | |
| *** anuradha1904 has quit IRC | 10:48 | |
| iurygregory | good morning arne_wiebalck janders rpittau dtantsur and Ironic o/ | 10:49 |
| rpittau | hey iurygregory :) | 10:50 |
| janders | hey iurygregory o/ | 10:50 |
| iurygregory | o/ | 10:50 |
| * arne_wiebalck upgraded to Ussuri | 10:57 | |
| arne_wiebalck | Here is a plot of the thundering herd (which I stretched already). https://usercontent.irccloud-cdn.com/file/BVKPfFQW/upgrade_thundering_herd.png | 10:58 |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: Follow up of Use OOB inspection to fetch MACs for IB inspection https://review.opendev.org/c/openstack/ironic/+/782406 | 10:58 |
| arne_wiebalck | And it seems, lazy loading is not as good as my first patch when it comes to DB activity. https://usercontent.irccloud-cdn.com/file/LRTPWSE6/upgrade_db_activity.png | 10:59 |
| arne_wiebalck | This is not a fair comparison as there is of course an upgrade in between, though. | 11:00 |
| arne_wiebalck | So maybe Ussuri does more on the DB ... ? | 11:00 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost stable/victoria: Increase memory of vm to adjust for new tinyipa https://review.opendev.org/c/openstack/bifrost/+/782407 | 11:00 |
| dtantsur | arne_wiebalck: not impossible, although I don't remember any increased activity | 11:01 |
| dtantsur | but then, I cannot imagine why lazy loading would be worse | 11:01 |
| dtantsur | has anyone looked at the bifrost CI? looks like insufficient RAM in upgrade jobs? | 11:01 |
| rpittau | dtantsur: jsut submitted a patch for that | 11:01 |
| arne_wiebalck | dtantsur: the real activity, as per the first plot, is about equal | 11:01 |
| dtantsur | rpittau: thanks! | 11:02 |
| arne_wiebalck | dtantsur: real == InnoDB activity | 11:02 |
| dtantsur | rpittau: we need to start with Victoria because of the upgrade, right? | 11:02 |
| *** yoctozepto has quit IRC | 11:02 | |
| rpittau | dtantsur: correct | 11:02 |
| rpittau | actually on master all's good, it's 1024 already | 11:03 |
| *** yoctozepto has joined #openstack-ironic | 11:03 | |
| dtantsur | nice | 11:03 |
| rpittau | I will rename tinyipa as moderatelysizedipa | 11:03 |
| dtantsur | :D | 11:05 |
| dtantsur | our cute little baby has grown | 11:05 |
| rpittau | oh yeah | 11:06 |
| *** rcernin has joined #openstack-ironic | 11:09 | |
| arne_wiebalck | dtantsur: the reason why lazy-loading is worse could be that the lazy-loading patch is not doing we think it does :) | 11:11 |
| arne_wiebalck | dtantsur: comparing with the plots I had before my local patch (which did not touch ports etc at all), the db activity is basically back to the old level | 11:13 |
| *** rcernin has quit IRC | 11:17 | |
| *** rcernin has joined #openstack-ironic | 11:18 | |
| arne_wiebalck | or, better, Arne branched before the lazy-load patch was back-ported and merged in stable/ussuri ... | 11:18 |
| *** ociuhandu has quit IRC | 11:22 | |
| dtantsur | arne_wiebalck: do you have a link to your lazy-loading patch? | 11:22 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/776930 | 11:23 |
| arne_wiebalck | dtantsur: the patch is not in the code I run | 11:24 |
| dtantsur | well :) | 11:24 |
| arne_wiebalck | dtantsur: I branched before it was merged in stable/ussuri | 11:24 |
| dtantsur | riddle solved! | 11:24 |
| arne_wiebalck | yeah | 11:24 |
| *** rcernin has quit IRC | 11:25 | |
| arne_wiebalck | I will create new packages, let's see ... | 11:25 |
| *** lourot has quit IRC | 11:25 | |
| *** ociuhandu has joined #openstack-ironic | 11:29 | |
| arne_wiebalck | at least we will get a direct effect of the patch :-D | 11:33 |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Add known iDRAC virtual media boot issue https://review.opendev.org/c/openstack/ironic/+/782438 | 11:35 |
| openstackgerrit | Merged openstack/ironic-inspector master: Increase memory of ironic vm in grenade https://review.opendev.org/c/openstack/ironic-inspector/+/782371 | 11:36 |
| openstackgerrit | Merged openstack/ironic-prometheus-exporter stable/wallaby: Increase RAM for ironic vm https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/782128 | 11:36 |
| openstackgerrit | Merged openstack/ironic-prometheus-exporter stable/victoria: Increase RAM for ironic vm https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/782129 | 11:36 |
| openstackgerrit | Merged openstack/ironic master: Revise release notes for 17.0 release https://review.opendev.org/c/openstack/ironic/+/780973 | 11:36 |
| iurygregory | moderatelysizedipa LOL | 11:39 |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: Follow up of Use OOB inspection to fetch MACs for IB inspection https://review.opendev.org/c/openstack/ironic/+/782406 | 11:50 |
| *** lourot has joined #openstack-ironic | 11:52 | |
| *** ociuhandu_ has joined #openstack-ironic | 12:12 | |
| *** ociuhandu has quit IRC | 12:13 | |
| *** ociuhandu has joined #openstack-ironic | 12:13 | |
| openstackgerrit | Merged openstack/ironic stable/victoria: [stable] Remove lower-constraints job https://review.opendev.org/c/openstack/ironic/+/767058 | 12:15 |
| *** ociuhandu_ has quit IRC | 12:17 | |
| openstackgerrit | Merged openstack/bifrost stable/victoria: Increase memory of vm to adjust for new tinyipa https://review.opendev.org/c/openstack/bifrost/+/782407 | 12:19 |
| *** frigo has joined #openstack-ironic | 12:19 | |
| frigo | the moderatelysized IPA ended up filling my /tmp because of https://storyboard.openstack.org/#!/story/2008742 | 12:21 |
| frigo | (although you can't really blame it on tinyipa:D ) | 12:22 |
| rpittau | lol | 12:22 |
| rpittau | frigo: patch welcome :) | 12:22 |
| iurygregory | hehehe smooltinyipa | 12:25 |
| frigo | on it, on it... | 12:25 |
| *** ociuhandu has quit IRC | 12:27 | |
| openstackgerrit | Merged openstack/ironic stable/ussuri: Fix broken configdrive_use_object_store https://review.opendev.org/c/openstack/ironic/+/777349 | 12:29 |
| *** ociuhandu has joined #openstack-ironic | 12:33 | |
| *** ociuhandu has quit IRC | 12:38 | |
| openstackgerrit | Francois Rigault proposed openstack/sushy-tools master: Delete temporary file after media ejection https://review.opendev.org/c/openstack/sushy-tools/+/782514 | 12:47 |
| *** rloo has joined #openstack-ironic | 12:53 | |
| *** mgoddard has quit IRC | 12:53 | |
| *** ociuhandu has joined #openstack-ironic | 12:57 | |
| *** ociuhandu has quit IRC | 13:02 | |
| *** ociuhandu has joined #openstack-ironic | 13:08 | |
| *** ociuhandu has quit IRC | 13:08 | |
| *** ociuhandu has joined #openstack-ironic | 13:10 | |
| TheJulia | good morning | 13:13 |
| *** ociuhandu has quit IRC | 13:15 | |
| dtantsur | TheJulia: morning | 13:15 |
| * TheJulia did not sleep well | 13:17 | |
| iurygregory | morning TheJulia | 13:20 |
| * iurygregory is also in the same boat | 13:21 | |
| TheJulia | bad wind storm here last night | 13:21 |
| iurygregory | for me it was because I received some bad news | 13:23 |
| TheJulia | I'm sorry :( | 13:25 |
| iurygregory | tks ^^ | 13:26 |
| *** ociuhandu has joined #openstack-ironic | 13:26 | |
| arne_wiebalck | aha! :) https://usercontent.irccloud-cdn.com/file/gBh3DQ04/lazy-loading.png | 13:27 |
| *** abhishekyadav has quit IRC | 13:28 | |
| arne_wiebalck | before 10:00 train with local DB patch, at 10:35 Ussuri vanilla with delayed startup (thundering herd), 14:00 Ussuri with lazy-loading and all conductors started at the same moment, and then the 5 min power sync interval | 13:29 |
| arne_wiebalck | so: lazy-loading reduces the DB load as intended, and it seems it also removes the thundering herd issue | 13:30 |
| TheJulia | well | 13:30 |
| TheJulia | "removes as in reduces load so things don't thrash" | 13:30 |
| arne_wiebalck | right | 13:31 |
| *** ociuhandu has quit IRC | 13:31 | |
| arne_wiebalck | but at least one can start the conductors all at once without having 10 times more load on the DB | 13:31 |
| arne_wiebalck | note that the initial start at 10:35 took more than 30mins | 13:32 |
| arne_wiebalck | while the one at 14:00 took 5mins | 13:32 |
| * arne_wiebalck is not saying there is no room for further improvement :) | 13:33 | |
| *** lmcgann has joined #openstack-ironic | 13:35 | |
| arne_wiebalck | jitter when starting the power sync would still be sensible IMO | 13:38 |
| TheJulia | Yeah, i'm still pondering | 13:38 |
| arne_wiebalck | that seems a low-hanging fruit to at least smear out requests | 13:39 |
| TheJulia | yeah | 13:39 |
| arne_wiebalck | rather than me having a bash loop with shuf :) | 13:40 |
| *** ociuhandu has joined #openstack-ironic | 13:42 | |
| *** tzumainn has joined #openstack-ironic | 13:43 | |
| mnaser | is there a simple doc somewhere on how i can get a devstack env up with something like virtualbmc | 13:43 |
| mnaser | or a shell script i can run and get a bunch of vms managed by ironic? | 13:43 |
| * mnaser has grown tired of rebooting physical systems non stop | 13:44 | |
| TheJulia | Devstack does it by default.... You supply it the number of VMs you want, you just need to ensure you don't exceed (remember that software!?) the amount of RAM | 13:44 |
| TheJulia | tenks may be an option, but I've never used it. | 13:45 |
| TheJulia | Bifrost has playbooks for this as well | 13:45 |
| *** bdodd has joined #openstack-ironic | 13:45 | |
| mnaser | oh devstack actually does it | 13:45 |
| mnaser | i'll just use that then | 13:45 |
| iurygregory | devstack has by default, you can get local.conf from our jobs and change some variables :D | 13:46 |
| *** ociuhandu has quit IRC | 13:47 | |
| *** ociuhandu has joined #openstack-ironic | 13:50 | |
| *** ociuhandu has quit IRC | 13:52 | |
| *** ociuhandu has joined #openstack-ironic | 13:52 | |
| *** mgoddard has joined #openstack-ironic | 13:57 | |
| *** ociuhandu has quit IRC | 14:04 | |
| *** ayoung has quit IRC | 14:05 | |
| dtantsur | arne_wiebalck, TheJulia, jitter is probably a one-liner with futurist | 14:06 |
| dtantsur | we just need to pick a different scheduling strategy IIRC | 14:06 |
| TheJulia | dtantsur: Yeah | 14:07 |
| * TheJulia adds to the ptg | 14:07 | |
| TheJulia | already there | 14:07 |
| dtantsur | maybe Just Do It (tm)? | 14:08 |
| * dtantsur wishes the strategies were better documented in https://docs.openstack.org/futurist/latest/reference/index.html#periodics | 14:09 | |
| * TheJulia adds more words to the ptg etherpad | 14:09 | |
| dtantsur | last_started_jitter is probably what we need | 14:09 |
| *** ociuhandu has joined #openstack-ironic | 14:10 | |
| TheJulia | possibly, if we can set the base jitter factor on the size of the deployment, we would likely be golden | 14:11 |
| dtantsur | not sure about the jitter factor, need to dive further | 14:11 |
| TheJulia | yeah, the docs render a little weird :( | 14:11 |
| dtantsur | I can make you a patch in like minutes | 14:12 |
| TheJulia | \o/ | 14:12 |
| dtantsur | arne_wiebalck: will you be able to test? | 14:14 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add prelude for Ironic 17.0. https://review.opendev.org/c/openstack/ironic/+/780974 | 14:15 |
| TheJulia | rpittau: ^^^ | 14:15 |
| *** ociuhandu has quit IRC | 14:15 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Version 17.0 mapping/doc updates https://review.opendev.org/c/openstack/ironic/+/782306 | 14:16 |
| TheJulia | just a rebase | 14:16 |
| *** ayoung has joined #openstack-ironic | 14:17 | |
| *** frigo has quit IRC | 14:19 | |
| TheJulia | iurygregory: question for you on https://review.opendev.org/c/openstack/ironic/+/770526 | 14:20 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Use jitter for periodic tasks https://review.opendev.org/c/openstack/ironic/+/782529 | 14:21 |
| dtantsur | arne_wiebalck, TheJulia ^^ | 14:21 |
| openstackgerrit | Merged openstack/ironic master: Add known iDRAC virtual media boot issue https://review.opendev.org/c/openstack/ironic/+/782438 | 14:22 |
| iurygregory | looking | 14:23 |
| TheJulia | iurygregory: just posted second thing | 14:23 |
| TheJulia | I likely need more coffee :) | 14:23 |
| * iurygregory press f5 | 14:23 | |
| *** frigo has joined #openstack-ironic | 14:24 | |
| * TheJulia seriously considers going back to bed | 14:25 | |
| dtantsur | as soon as you start seriously considering it, it's probably a good idea | 14:25 |
| openstackgerrit | Merged openstack/python-ironicclient stable/wallaby: Fix --fields network_data https://review.opendev.org/c/openstack/python-ironicclient/+/782130 | 14:25 |
| JayF | I was sick for over half of last week. Let me assure you that laying down all day is overrated. At least by the third day :P | 14:26 |
| rpittau | frigo: do you mind also adding the task number to the commit message in https://review.opendev.org/c/openstack/sushy-tools/+/782514 ? | 14:26 |
| *** ociuhandu has joined #openstack-ironic | 14:27 | |
| TheJulia | it gets super booring | 14:27 |
| openstackgerrit | Francois Rigault proposed openstack/sushy-tools master: Delete temporary file after media ejection https://review.opendev.org/c/openstack/sushy-tools/+/782514 | 14:28 |
| openstackgerrit | Francois Rigault proposed openstack/sushy-tools master: Delete temporary file after media ejection https://review.opendev.org/c/openstack/sushy-tools/+/782514 | 14:28 |
| frigo | like that rpittau ? | 14:30 |
| rpittau | frigo: yes, thanks :) | 14:30 |
| openstackgerrit | Merged openstack/bifrost master: Relax format of checksum file https://review.opendev.org/c/openstack/bifrost/+/778578 | 14:30 |
| frigo | \o/ | 14:31 |
| iurygregory | TheJulia, I've added some answers =) | 14:31 |
| openstackgerrit | Alexander Evseev proposed openstack/bifrost master: Drop external tftp service in favor to use dnsmasq's one https://review.opendev.org/c/openstack/bifrost/+/782531 | 14:31 |
| iurygregory | wow ^ | 14:32 |
| TheJulia | likely a good thing | 14:32 |
| TheJulia | more people use dnsmasq now then ever before | 14:32 |
| TheJulia | or ever before the older tftp services | 14:32 |
| TheJulia | so... | 14:32 |
| dtantsur | we do it in metal3 too | 14:32 |
| dtantsur | I suspect there may be upgrade concerns, let me see the patch | 14:32 |
| *** ociuhandu has quit IRC | 14:35 | |
| TheJulia | likely | 14:36 |
| dtantsur | but it's always great to see complete strangers contributing | 14:37 |
| TheJulia | ++ | 14:37 |
| * dtantsur wonders if Alexander is here | 14:37 | |
| iurygregory | ++ | 14:38 |
| dtantsur | fg | 14:44 |
| dtantsur | mmm, won't work here, will it? | 14:44 |
| TheJulia | no | 14:44 |
| TheJulia | sadly | 14:44 |
| dtantsur | :( | 14:44 |
| TheJulia | If it did, would it be like putting on a VR headset? | 14:44 |
| * dtantsur returns to fish | 14:44 | |
| dtantsur | heh | 14:45 |
| dtantsur | btw I've never tried one | 14:45 |
| TheJulia | iurygregory: replied, one point of clarification needed. Do you mean that you wrote your unit tests execting it to already exist? | 14:45 |
| iurygregory | TheJulia, yeah, we have a case where cipher_suite is not set for the node so driver_info['cipher_suite'] returns None | 14:46 |
| TheJulia | dtantsur: I've not tried the current generation, latest I've tried was first generation at home stuff. It was okay but since my vision is not perfect it looks blurry and i can see the pixel lines | 14:46 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Support pre-built deploy/rescue ISO in Redfish https://review.opendev.org/c/openstack/ironic/+/782266 | 14:46 |
| iurygregory | but I can change to get('cipher_suite') without problems =) | 14:46 |
| *** outbrito has quit IRC | 14:46 | |
| TheJulia | iurygregory: so when does driver_info['cipher_suite'] get set first? | 14:47 |
| iurygregory | TheJulia, https://github.com/openstack/ironic/blob/2773c5fb257f06226409e1f40c7820d98e3839ae/ironic/drivers/modules/ipmitool.py#L306 | 14:48 |
| TheJulia | ahh | 14:48 |
| TheJulia | that | 14:48 |
| TheJulia | \o/ for layers of indirection through naming | 14:48 |
| TheJulia | iurygregory: in that case, I'd just amend the docs I guess | 14:49 |
| TheJulia | w/r/t how to determine the supported cipher suites | 14:49 |
| iurygregory | TheJulia, ack =) I will also add a test case with only one value in the list | 14:49 |
| iurygregory | =) | 14:49 |
| TheJulia | ++ | 14:50 |
| *** frigo has quit IRC | 14:55 | |
| arne_wiebalck | dtantsur: yes, I can test ... not sure if today, though | 15:03 |
| mnaser | darn hmm | 15:06 |
| mnaser | my little raid patch seems to have not solved things (yet) | 15:06 |
| mnaser | driver_internal_info does not seem to include the raid create/delte config with the right priority | 15:07 |
| mnaser | https://www.irccloud.com/pastebin/rxgq1GLE/ | 15:08 |
| mnaser | even though it is reporting the following on startup | 15:08 |
| mnaser | Mar 23 15:03:27 ironic-devstack ironic-conductor[25562]: DEBUG oslo_service.service [-] deploy.delete_configuration_priority = 5 {{(pid=25562) log_opt_values /usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py:2589}} | 15:09 |
| TheJulia | hmm | 15:09 |
| TheJulia | could it be loading/replacing priority from the agent itself | 15:10 |
| mnaser | https://www.irccloud.com/pastebin/FJ5O2Gmm/ | 15:11 |
| mnaser | followed by | 15:12 |
| mnaser | [ 141.209906] ironic-python-agent[442]: 2021-03-23 15:07:04.883 442 DEBUG root [-] Clean steps before deduplication: {'GenericHardwareManager': [{'step': 'erase_devices', 'priority': 10, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'erase_devices_metadata', 'priority': 99, 'interface': 'deploy', 'reboot_requested': False, 'abortable': True}, {'step': 'delete_configuration', | 15:12 |
| mnaser | 'priority': 0, 'interface': 'raid', 'reboot_requested': False, 'abortable': True}, {'step': 'create_configuration', 'priority': 0, 'interface': 'raid', 'reboot_requested': False, 'abortable': True}]} get_clean_steps /opt/ironic-python-agent/lib64/python3.6/site-packages/ironic_python_agent/extensions/clean.py:42 | 15:12 |
| mnaser | i think the problem here is that clean_steps does not include anything except erase_devices_metadata and erase_devices | 15:14 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Support pre-built deploy/rescue ISO in Redfish https://review.opendev.org/c/openstack/ironic/+/782266 | 15:15 |
| dtantsur | TheJulia: would be cool to fit in wallaby ^^ | 15:15 |
| dtantsur | mnaser: I have a vague feeling that automated in-band cleaning filtered out interfaces other than 'deploy' | 15:15 |
| dtantsur | but don't quote me on that | 15:15 |
| mnaser | i'm trying to find the code that sets clean_steps for automated in-band cleaning | 15:16 |
| dtantsur | mnaser: look in ironic/drivers/modules/agent_base.py | 15:17 |
| *** mkrai has joined #openstack-ironic | 15:17 | |
| dtantsur | mnaser: https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/agent_base.py#L873 | 15:18 |
| dtantsur | interface='deploy' | 15:18 |
| dtantsur | I may be responsible for this code, and no, I don't remember why | 15:18 |
| * dtantsur wonders if mgoddard remembers | 15:18 | |
| mnaser | dtantsur: this is because raid is interface='raid' and so it wont include that step, right? | 15:20 |
| mgoddard | dtantsur: isn't it just that the deploy interface gathers all in-band steps? | 15:20 |
| mgoddard | maybe not | 15:21 |
| dtantsur | mnaser: correct | 15:22 |
| * mnaser is git blaming | 15:22 | |
| arne_wiebalck | TheJulia: software RAID and UEFI, something which we just ran into is that the ESPs get out of sync (which I considered to be a non-issue as the grub-efi.cfg is not in the ESP, but it seems it is ...) So, if the node is not booted in a while, but has updated kernels several times, the UEFI f/w may use an ESP which points to non-existing kernels ... or am I missing sth here? | 15:23 |
| arne_wiebalck | TheJulia: This results in nodes which do not boot. | 15:23 |
| * TheJulia sighs | 15:24 | |
| mnaser | dtantsur: https://opendev.org/openstack/ironic/commit/a19e9ce6759b018025790ca91b200796f63bb7eb this looks like the change that made it enforce deploy steps only | 15:24 |
| mnaser | it actually looks like it changed the behaviour then | 15:24 |
| arne_wiebalck | TheJulia: I did not understand initially why we never had an issue so far, but I think it only is a problem when a sufficient number of kernels is installed, so that the non-update ESP has only references to meanwhile deleted kernels. | 15:25 |
| mnaser | oh nevermind, it was already filtering for deploy before | 15:25 |
| * mnaser does more git blaming | 15:25 | |
| mgoddard | dtantsur: perhaps there was an intention to write a similar get_deploy_steps hook for other interfaces? | 15:26 |
| mgoddard | that would make in-band raid steps appear on the raid interface | 15:26 |
| TheJulia | arne_wiebalck: well... basically grub is doing stuff which will cause problems because its use model is predicated upon either the storage being reliable and redundant to begin with, or not at all and not caring. software raid is a unfortunate victim in this circumstance | 15:26 |
| arne_wiebalck | TheJulia: I think whatever updates grub-efi.cfg needs to search and update all ESPs. | 15:27 |
| mnaser | https://opendev.org/openstack/ironic/commit/534d9ee96ac738f24a23087a26c98804934ea72d -- looks like i went back 6 years ago | 15:27 |
| mnaser | and the first addition has "Filter steps to only return deploy steps" | 15:27 |
| TheJulia | arne_wiebalck: that would take require convincing grub maintainers and their last thing was to actually break grub2-install instead of keeping compatability | 15:28 |
| mnaser | https://review.opendev.org/c/openstack/ironic/+/161453/19/ironic/drivers/modules/deploy_utils.py | 15:28 |
| TheJulia | so.... I don't know how much success we will have, but the case can be made | 15:28 |
| arne_wiebalck | TheJulia: so, just to confirm we have an actual issue here, right? | 15:28 |
| arne_wiebalck | TheJulia: users will run into this sooner or later | 15:29 |
| TheJulia | ?possibly? | 15:29 |
| mnaser | dtantsur / mgoddard: think it would make sense to append get_steps(..., interface='raid') into https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/agent_base.py#L858-L860 ? | 15:29 |
| arne_wiebalck | TheJulia: :-D | 15:29 |
| arne_wiebalck | TheJulia: what we could do is make the ESP a RAID-1 device, rather than two independent devices | 15:29 |
| TheJulia | That may be the only way | 15:30 |
| arne_wiebalck | TheJulia: the kernel would assemble it, and updates go to both disks | 15:30 |
| dtantsur | or use interface='deploy' for RAID steps, like in https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware.py#L1877 ? | 15:30 |
| TheJulia | the question may be 1) does the firmware see through it, and 2) can the firmware still mount it | 15:30 |
| arne_wiebalck | TheJulia: the UEFI f/w would not assemble the RAID but still be able to read it | 15:30 |
| arne_wiebalck | TheJulia: yes, this works | 15:30 |
| arne_wiebalck | TheJulia: our Ceph team does have a RAID1 ESP | 15:31 |
| TheJulia | I'm not sure all platforms will be okay with the raid header | 15:31 |
| dtantsur | mnaser: an argument could be made that any deploy step coming from IPA should have interface='deploy' | 15:31 |
| arne_wiebalck | TheJulia: the header needs to be at the end | 15:31 |
| TheJulia | arne_wiebalck: that is a relatively good sign | 15:31 |
| TheJulia | ++ | 15:31 |
| dtantsur | (although the same argument does not apply to clean steps, soooo) | 15:31 |
| * dtantsur is confused | 15:31 | |
| mnaser | dtantsur: yeah, and i guess the 'raid' interface is a thing because hardware raid can be a thing too | 15:31 |
| *** mkrai has quit IRC | 15:32 | |
| mnaser | so technically, as part of automated cleaning, you may actually want hardware raid steps to happen automagically | 15:32 |
| dtantsur | mnaser: well, yes, exactly. if you set raid_interface='idrac', you probably don't want software RAID to happen | 15:32 |
| arne_wiebalck | TheJulia: the only other (clonky) option is to mount ESPs regularly and sync | 15:32 |
| *** mkrai has joined #openstack-ironic | 15:32 | |
| arne_wiebalck | TheJulia: the only other option I see atm | 15:32 |
| mnaser | dtantsur: so either move raid steps to deploy interface or add get_steps(...,interface=raid) | 15:33 |
| TheJulia | arne_wiebalck: I think we're going to need to backport a lot of grub/esp/uefi patches soon, so I think we may just bbackport this fix as well once we have it | 15:33 |
| dtantsur | mnaser: remind me, what exactly are you trying? | 15:33 |
| mnaser | dtantsur: i want to make automated cleaning actually delete_configuration for software raid, then run the disk wipes/etc, and then create_configuration again | 15:33 |
| mnaser | dtantsur: i did this https://opendev.org/openstack/ironic/commit/6ba1ecad17bea362f4f9ea8f1641e9d6d87bcfb8 but it looks like it's pretty much broken, because interface=raid | 15:34 |
| mnaser | so we're not really overriding anything by doing that | 15:34 |
| arne_wiebalck | TheJulia: ok ... I will first need to fix our ESP manually now to prevent more damage ... | 15:34 |
| dtantsur | okay, so automated cleaning and RAID.. | 15:34 |
| mgoddard | I think the simplest option is to set interface=deploy in the HW manager, as dtantsur suggested | 15:35 |
| arne_wiebalck | TheJulia: ESPs ... plural ... many | 15:35 |
| TheJulia | arne_wiebalck: good luck | 15:35 |
| dtantsur | mnaser: I think you need to update the 'agent' RAID interface with its own get_clean_steps. | 15:35 |
| mgoddard | dtantsur: +1, I think that's the 'correct' fix | 15:35 |
| dtantsur | mnaser: it could be that doing https://opendev.org/openstack/ironic/commit/6ba1ecad17bea362f4f9ea8f1641e9d6d87bcfb8 in the deploy interface is not right | 15:35 |
| mgoddard | because then the steps correctly show as being raid steps | 15:36 |
| mgoddard | and interface priorities will work | 15:36 |
| dtantsur | yes, and you don't get conflicts with hardware RAID | 15:36 |
| mnaser | mgoddard: will they work though? even if i have get_steps(..., interface='deploy') there? | 15:36 |
| dtantsur | mnaser: no, in the RAID interface you need to use interface='raid' | 15:36 |
| openstackgerrit | Merged openstack/python-ironicclient stable/victoria: Fix --fields network_data https://review.opendev.org/c/openstack/python-ironicclient/+/782131 | 15:36 |
| mnaser | so get_clean_steps inside the raid interface | 15:36 |
| dtantsur | pretty much a copy of get_deploy_steps with s/deploy/raid/ | 15:36 |
| dtantsur | yes | 15:36 |
| mnaser | and get rid of the ones i added inside agent_base.py | 15:37 |
| dtantsur | I think so | 15:37 |
| openstackgerrit | Merged openstack/ironic master: Add prelude for Ironic 17.0. https://review.opendev.org/c/openstack/ironic/+/780974 | 15:37 |
| *** irclogbot_3 has quit IRC | 15:37 | |
| *** irclogbot_1 has joined #openstack-ironic | 15:38 | |
| TheJulia | - Failed: 6048 \o/ | 15:41 |
| TheJulia | hmmm | 15:43 |
| TheJulia | anyone having issues on train if they rebuild their tox environment? | 15:43 |
| mnaser | https://www.irccloud.com/pastebin/5XG3Zj6m/ | 15:43 |
| mnaser | dtantsur, mgoddard ^ going to try the above | 15:44 |
| mnaser | Mar 23 15:44:45 ironic-devstack ironic-conductor[27273]: DEBUG ironic.drivers.base [None req-baa9c114-5eb8-4d77-8142-37d37687a643 None None] Found clean steps [{'step': 'create_configuration', 'priority': 0, 'abortable': False, 'argsinfo': None, 'interface': 'raid'}, {'step': 'delete_configuration', 'priority': 0, 'abortable': False, 'argsinfo': None, 'interface': 'raid'}] for interface raid {{(pid=27273) | 15:45 |
| mnaser | __new__ /opt/stack/ironic/ironic/drivers/base.py:261}} | 15:45 |
| mnaser | ok.. ok.. looking good | 15:45 |
| mnaser | if that works, i wonder if i should move the config option to [raid]/create_configuration_priority .. | 15:46 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/train: Fixes the issue that instance bond port can't get IP address https://review.opendev.org/c/openstack/ironic/+/782543 | 15:49 |
| mnaser | https://www.irccloud.com/pastebin/llHYnZKB/ | 15:49 |
| mnaser | yay thanks mgoddard / dtantsur ! :D | 15:49 |
| *** irclogbot_1 has quit IRC | 15:49 | |
| mnaser | ironic_python_agent.errors.CleaningError: Clean step failed: Error performing clean_step create_configuration: Software RAID caused unknown error: Could not validate Software RAID config for f09c61e0-42bf-422d-acf0-375a3974e913: Software RAID Configuration requires RAID-1 for the first logical disk | 15:50 |
| mnaser | that's just me being a bit slow this morning i guess | 15:50 |
| *** mkrai has quit IRC | 15:51 | |
| * TheJulia sighs at backports that all have to be done manually | 15:52 | |
| *** irclogbot_0 has joined #openstack-ironic | 15:52 | |
| *** ociuhandu has joined #openstack-ironic | 15:56 | |
| dtantsur | mnaser: np, yeah, the patch looks about right | 15:58 |
| mnaser | i'll test this here more and then propose a patch | 15:58 |
| mnaser | with passing tests, etc. | 15:58 |
| *** dking has quit IRC | 16:01 | |
| iurygregory | TheJulia, maybe because it's trying to run a base python3 train doesn't support? | 16:05 |
| TheJulia | iurygregory: that is what I'm thinking as well | 16:05 |
| iurygregory | If I try locally will use 3.9 and things will *boom* | 16:05 |
| TheJulia | but it errors in kombu... | 16:05 |
| TheJulia | and import_lib | 16:05 |
| iurygregory | do you have a link for the patch? I can run locally | 16:06 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/victoria: IPMI: Handle vendor set boot device differences https://review.opendev.org/c/openstack/ironic/+/782547 | 16:06 |
| dtantsur | TheJulia: I seem to recall these problems, yes | 16:08 |
| dtantsur | I tend to set basepython = python2 when dealing with train | 16:08 |
| iurygregory | maybe worth setting in the tox.ini of stable/train =D | 16:08 |
| rpittau | mmm that should be needed for pep8 only though | 16:08 |
| iurygregory | either python3.x or python2 | 16:09 |
| TheJulia | I only have 3.8 right now | 16:09 |
| TheJulia | but I could have sworn I ran tests last week | 16:09 |
| iurygregory | *magic* | 16:09 |
| rpittau | TheJulia: I think it would be worth to add ignore_basepython_conflict=true to tox.ini, otherwise it will ignore the python version you want to test and use the default one you have | 16:10 |
| rpittau | so if you run tox -epy27 but you have python 3.8, will run with python 3.8 | 16:11 |
| rpittau | the problem is that I think oyu need also to increase the tox minversion | 16:11 |
| TheJulia | I don't even think I have py2 | 16:12 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Support deploy vendor data in addition to network data https://review.opendev.org/c/openstack/ironic/+/782548 | 16:12 |
| dtantsur | dunno if it's a good idea or not, but it came to me ^^ | 16:12 |
| TheJulia | I've been going down the list of pending community identified backports and doing the needful. We keep forgetting to tag these things, so hopefully we'll get better with backport candidate | 16:13 |
| dtantsur | yeah, a tag is helpful when a backport is not trivial | 16:13 |
| rpittau | TheJulia: I just verified, the unit tests should run correctly locally, so I gues something weird with that patch | 16:15 |
| TheJulia | On a different patch I'm having issues on ussuri now, but it may just be transient | 16:16 |
| * TheJulia re-runs | 16:16 | |
| iurygregory | I'm writting some script to add a hashtag to patches with the backport-candidate label | 16:17 |
| iurygregory | WIP =) | 16:17 |
| dtantsur | iurygregory: why a hashtag? | 16:18 |
| TheJulia | I think it was the consensus from a meeting like 1.5 months ago | 16:19 |
| dtantsur | I mean, we can filter by the label itself, no? | 16:19 |
| dtantsur | why another hash tag? | 16:19 |
| iurygregory | dtantsur, it would be easier to track with the tag I would say =) | 16:19 |
| iurygregory | we can ofc look at just the label =) | 16:20 |
| dtantsur | iurygregory: well, if we create an ironic specific tag, yes | 16:20 |
| iurygregory | if it's set or not =) | 16:20 |
| iurygregory | ironic-backport | 16:20 |
| TheJulia | i though twe were going to use the same tag | 16:20 |
| iurygregory | ironic-week-prio? .-. | 16:20 |
| dtantsur | backports != weekly priorities | 16:20 |
| iurygregory | yeah ^ | 16:20 |
| TheJulia | true | 16:20 |
| dtantsur | different actions needed | 16:20 |
| dtantsur | I agree with a new tag, but JFYI https://review.opendev.org/q/project:openstack/ironic+label:Backport-Candidate%252B1 | 16:20 |
| rpittau | I thought the tag was decided just for the priorities | 16:21 |
| iurygregory | ironic-week-prio it is | 16:21 |
| rpittau | yeah, jsut that | 16:21 |
| iurygregory | humm the url is not that bad =D | 16:21 |
| TheJulia | dtantsur: I tagged it because that is something that we absolutley need to get backported as grub is making hard breaking changes that are starting to filter in... :( | 16:21 |
| TheJulia | Thus, is a candidate :( | 16:22 |
| iurygregory | but we would need the project set.. if my script works we would just search using a new hashtag and would cover all ironic projects | 16:23 |
| dtantsur | TheJulia: yeah, I'm showing how to use a label in queries | 16:23 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Refactor: use mounted from ironic-lib https://review.opendev.org/c/openstack/ironic-python-agent/+/782259 | 16:24 |
| TheJulia | oh | 16:24 |
| TheJulia | okay | 16:24 |
| TheJulia | yeah | 16:24 |
| TheJulia | hmm | 16:24 |
| iurygregory | and I was also thinking about working on the script to remove the ironic-week-prio after the patch is merged | 16:25 |
| *** moshiur has quit IRC | 16:25 | |
| iurygregory | I think it would make sense | 16:26 |
| TheJulia | gah | 16:26 |
| TheJulia | I see what happened and why things are so weird | 16:26 |
| iurygregory | this is a good thing ^ | 16:26 |
| TheJulia | we need to be really careful with backports making sure there were no prior patches that also need to be backported that the patch is based upon | 16:26 |
| TheJulia | example: detect_vendor | 16:26 |
| iurygregory | yeah | 16:26 |
| TheJulia | or if there are, to grab them too | 16:27 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Hardware manager call to load configuration from vmedia https://review.opendev.org/c/openstack/ironic-python-agent/+/782262 | 16:27 |
| dtantsur | looking at https://zuul.opendev.org/t/openstack/build/81cd79ceac694b2fbc400b42e0ef2e83/log/controller/logs/screen-ir-cond.txt (with jitter), periodic tasks do start to slowly drift away | 16:30 |
| dtantsur | but it probably doesn't solve the restart problem since the maximum jitter is 5% | 16:30 |
| rpittau | dtantsur: https://bugs.python.org/issue35022 | 16:30 |
| dtantsur | rpittau++ | 16:31 |
| TheJulia | dtantsur: I think we can do that as a separate patch | 16:32 |
| dtantsur | re jitter? | 16:32 |
| TheJulia | dtantsur: I've been thinking a startup "size the deployment and hold the initailization for random second window based upon the size" | 16:32 |
| dtantsur | TheJulia: I was more thinking about randomizing the starting point of each periodic | 16:33 |
| TheJulia | like 1 second per conductor if >5 conductors or something like that | 16:33 |
| dtantsur | so that they a fixed but random distance apart | 16:33 |
| TheJulia | and multiple by # of nodes or something like that | 16:33 |
| dtantsur | you don't know the number of nodes on start-up | 16:33 |
| TheJulia | yeah | 16:33 |
| TheJulia | I was thinking at least on conductor based data | 16:33 |
| TheJulia | and maybe doing an index based count from the db which is an easy query | 16:33 |
| TheJulia | select count(id) from nodes; | 16:34 |
| dtantsur | *shrug* just randomizing it is probably easier | 16:34 |
| TheJulia | maybe yeah | 16:34 |
| * dtantsur tries one crazy idea | 16:34 | |
| TheJulia | I want it to be semi-aware so we give enough of a space window for some conductors to restart to naturally allow things to spread out to begin with | 16:35 |
| dtantsur | oh, well, of course futurist doesn't allow a custom strategy.. | 16:35 |
| dtantsur | a random offset for each periodics solves more problems: it reduces load peaks because of periodic tasks | 16:36 |
| TheJulia | yeah, I'm thinking something baked really early on in startup | 16:37 |
| TheJulia | but yeah | 16:37 |
| TheJulia | only ideas, haven't tried to hack on it yet | 16:37 |
| dtantsur | I'll try my idea with a fixed offset | 16:37 |
| TheJulia | k | 16:37 |
| openstackgerrit | Mohammed Naser proposed openstack/ironic master: Allow running RAID cleaning steps with in-band cleaning https://review.opendev.org/c/openstack/ironic/+/782552 | 16:37 |
| mnaser | TheJulia / dtantsur / arne_wiebalck / mgoddard: ^ that works for me here locally | 16:40 |
| dtantsur | great | 16:40 |
| mnaser | https://www.irccloud.com/pastebin/TMGHkKtF/ | 16:41 |
| mnaser | looks awesome :) | 16:41 |
| dtantsur | :) | 16:46 |
| TheJulia | \o/ | 16:48 |
| *** uzumaki has joined #openstack-ironic | 16:57 | |
| *** lucasagomes has quit IRC | 17:00 | |
| *** iurygregory has quit IRC | 17:02 | |
| dtantsur | TheJulia: I'm starting to think that a proper jitter is hard to implement without futurist changes :( | 17:04 |
| *** ociuhandu_ has joined #openstack-ironic | 17:10 | |
| *** ociuhandu has quit IRC | 17:14 | |
| *** ociuhandu_ has quit IRC | 17:15 | |
| rpittau | good night! o/ | 17:26 |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| *** iurygregory has joined #openstack-ironic | 17:29 | |
| *** uzumaki has quit IRC | 17:31 | |
| *** frigo has joined #openstack-ironic | 17:35 | |
| TheJulia | dtantsur: start jitter, or proper jitter upfront? or jitter in general? | 17:41 |
| dtantsur | proper jitter upfront, I guess | 17:41 |
| dtantsur | we can add jitter to spacing, but it's not exactly what we need | 17:42 |
| TheJulia | I'm not sure we really can or *should* I guess | 17:42 |
| TheJulia | yay for moderately nebulous problems | 17:42 |
| dtantsur | if futurist allowed that, it would solve most of the problems | 17:42 |
| TheJulia | yeah | 17:42 |
| dtantsur | now.. short of adding a sleep before each periodic task, I have no other ideas :) | 17:42 |
| dtantsur | automating a sleep is something I can do, dunno how ugly it is | 17:43 |
| TheJulia | that seems ugly | 17:44 |
| dtantsur | I can look into changing futurist, but it won't be backportable | 17:45 |
| TheJulia | so hash ring population is part of the very start of the headache and the herd thundering | 17:45 |
| TheJulia | so maybe not change futurist, but do the jitter that can evolve on its own in with the existing periodics, and then separately handle the startup behavior since that select can vary from 30seconds to 5 minutes in arne's deployment if they all collide | 17:46 |
| TheJulia | i guess what I'm saying is disjoint he problems | 17:47 |
| TheJulia | s/he/the/ | 17:47 |
| TheJulia | they are definitely related, but they can be solved separately and the initial start would setup the later ones to be perfectly fine | 17:47 |
| TheJulia | and arne shoudl end up with a even line on his graphs in the end | 17:47 |
| TheJulia | 3 hour spans of meetings is just a bad idea | 17:47 |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Add configuration mold storage https://review.opendev.org/c/openstack/ironic/+/769928 | 17:57 |
| TheJulia | reviewjam? | 17:59 |
| *** derekh has quit IRC | 18:03 | |
| openstackgerrit | Aija Jauntēva proposed openstack/ironic master: Add import, export configuration to idrac-redfish https://review.opendev.org/c/openstack/ironic/+/759428 | 18:10 |
| openstackgerrit | Francois Rigault proposed openstack/sushy-tools master: Return an integer value for Cores and Threads https://review.opendev.org/c/openstack/sushy-tools/+/782577 | 18:14 |
| openstackgerrit | Francois Rigault proposed openstack/ironic-inspector master: [wip] Use port.id instead of port.uuid https://review.opendev.org/c/openstack/ironic-inspector/+/782580 | 18:20 |
| arne_wiebalck | Added jitter manually to cut off the spikes. Looks even better then pre-update now :) https://usercontent.irccloud-cdn.com/file/Id7ZlkBH/ussuri_upgrade_db.png | 18:45 |
| arne_wiebalck | s/then/than/ | 18:45 |
| *** dtantsur is now known as dtantsur|afk | 18:55 | |
| *** dtantsur|afk is now known as dtantsur|afk|afk | 18:55 | |
| *** dtantsur|afk|afk is now known as dtantsur|afk | 18:55 | |
| *** frigo has quit IRC | 19:04 | |
| openstackgerrit | Merged openstack/ironic-python-agent master: Refactor: use mounted from ironic-lib https://review.opendev.org/c/openstack/ironic-python-agent/+/782259 | 20:14 |
| *** frigo has joined #openstack-ironic | 20:30 | |
| openstackgerrit | Mohammed Naser proposed openstack/ironic master: Allow running RAID cleaning steps with in-band cleaning https://review.opendev.org/c/openstack/ironic/+/782552 | 20:35 |
| *** ociuhandu has joined #openstack-ironic | 20:39 | |
| *** frigo has quit IRC | 20:42 | |
| *** lmcgann has quit IRC | 20:53 | |
| stevebaker | TheJulia: oh, what do you think of backporting IPA token support to stable/train, to make local development with new ironic easier | 20:58 |
| stevebaker | (it might be too hard) | 20:58 |
| *** rcernin has joined #openstack-ironic | 21:01 | |
| JayF | That seems like the sort of thing that would be invalid to backport | 21:03 |
| JayF | I'd certainly not be a fan of that change going that far back... | 21:04 |
| *** ociuhandu has quit IRC | 21:05 | |
| TheJulia | Yeah, I wouldn't try to backport that | 21:06 |
| TheJulia | it was intended kind of to be a hard delimiter, even though it would make some development stuff harder it was also to enforce the support boundry established as well | 21:08 |
| * TheJulia suspects she will just pickup the backport list tomorrow | 21:10 | |
| *** ociuhandu has joined #openstack-ironic | 21:11 | |
| TheJulia | stevebaker: I mentioned this briefly on the call we were just on, but I've been going through "we should backport this" list on the whiteboard and trying to close those things out | 21:12 |
| TheJulia | Anywya, will resume with ipmi stuffs tomorrow | 21:15 |
| *** rcernin has quit IRC | 21:16 | |
| *** ociuhandu has quit IRC | 21:16 | |
| * TheJulia goes and exercises now that the migraine has actually subsided | 21:16 | |
| *** rcernin has joined #openstack-ironic | 21:46 | |
| *** rloo has quit IRC | 22:02 | |
| *** rcernin has quit IRC | 22:10 | |
| *** rcernin has joined #openstack-ironic | 22:10 | |
| *** hoonetorg has joined #openstack-ironic | 22:53 | |
| *** jamesden_ has quit IRC | 23:16 | |
| *** jamesdenton has joined #openstack-ironic | 23:17 | |
| openstackgerrit | Merged openstack/ironic master: Version 17.0 mapping/doc updates https://review.opendev.org/c/openstack/ironic/+/782306 | 23:36 |
| openstackgerrit | Francois Rigault proposed openstack/ironic-inspector master: Use port.id instead of port.uuid https://review.opendev.org/c/openstack/ironic-inspector/+/782580 | 23:43 |
| janders | good morning Ironic o/ | 23:48 |
| stevebaker | good afternoon | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
