| *** gyee has quit IRC | 00:21 | |
| *** rcernin has quit IRC | 01:11 | |
| *** rcernin has joined #openstack-ironic | 01:11 | |
| *** eandersson has quit IRC | 01:29 | |
| *** eandersson has joined #openstack-ironic | 01:29 | |
| *** rloo has quit IRC | 01:49 | |
| openstackgerrit | Verification of a change to openstack/ironic failed: DRAC: Drives conversion from JBOD to RAID https://review.opendev.org/738509 | 02:34 |
|---|---|---|
| openstackgerrit | Merged openstack/ironic master: Enforce autospec in irmc tests https://review.opendev.org/745642 | 02:53 |
| *** rcernin has quit IRC | 03:06 | |
| *** riuzen has joined #openstack-ironic | 03:09 | |
| *** riuzen has quit IRC | 03:19 | |
| *** riuzen has joined #openstack-ironic | 03:31 | |
| janders | stevebaker I have a quick gerrit question if you have time? | 03:32 |
| *** rcernin has joined #openstack-ironic | 03:52 | |
| *** ricolin has joined #openstack-ironic | 04:04 | |
| *** mkrai has joined #openstack-ironic | 04:23 | |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 04:23 |
| janders | hopefully this revision looks better than CI :) | 04:27 |
| janders | s/than/in | 04:27 |
| janders | *better than the previous one :) | 04:27 |
| *** JamesBenson has quit IRC | 05:25 | |
| *** riuzen has quit IRC | 05:26 | |
| *** JamesBenson has joined #openstack-ironic | 05:27 | |
| *** JamesBenson has quit IRC | 05:31 | |
| openstackgerrit | Merged openstack/ironic master: Follow up to I44336423194eed99f026c44b6390030a94ed0522 https://review.opendev.org/745474 | 06:10 |
| openstackgerrit | Merged openstack/ironic-python-agent master: Add possibility to pass global request ID https://review.opendev.org/728529 | 06:15 |
| *** tzumainn has quit IRC | 06:20 | |
| iurygregory | good morning Ironic | 06:41 |
| *** JamesBenson has joined #openstack-ironic | 06:50 | |
| *** JamesBenson has quit IRC | 06:55 | |
| rpittau | good morning ironic! o/ | 06:56 |
| iurygregory | good morning rpittau o/ | 07:03 |
| rpittau | hey iurygregory :) | 07:04 |
| janders | good morning iurygregory and rpittau | 07:06 |
| iurygregory | janders, o/ | 07:06 |
| janders | welcome to the last working day before the Recharge long weekend :) | 07:06 |
| iurygregory | yeah | 07:06 |
| rpittau | hey janders :) | 07:06 |
| iurygregory | long weekend yay! | 07:06 |
| janders | (I'm mostly done with mine, but will be back for a couple hours later :) | 07:06 |
| janders | if you have time, I will have some questions on how to read CI failures :) | 07:07 |
| janders | but let's wait for CI for my patch to properly finish first | 07:07 |
| rpittau | janders: sure thing | 07:07 |
| iurygregory | we can help with that =) | 07:07 |
| janders | thank you! :) | 07:08 |
| *** abdysn has joined #openstack-ironic | 07:18 | |
| rpittau | if anyone got a minute, this will fix functional test in ironicclient https://review.opendev.org/745834 | 07:19 |
| patchbot | patch 745834 - python-ironicclient - Fix expected exception message in one test - 1 patch set | 07:19 |
| *** mkrai has quit IRC | 07:21 | |
| iurygregory | rpittau, looking | 07:23 |
| *** mkrai has joined #openstack-ironic | 07:26 | |
| rpittau | iurygregory: thanks | 07:26 |
| iurygregory | np | 07:26 |
| *** dougsz has joined #openstack-ironic | 07:36 | |
| *** tosky has joined #openstack-ironic | 07:42 | |
| iurygregory | I just noticed that my patch set #3 had ironic-lib-partition-bios-ipmi-direct-src green with privsep but ironic-lib-wholedisk-bios-ipmi-iscsi-src failed because couldn't link the image converted .-. | 07:43 |
| openstackgerrit | Merged openstack/ironic master: [Trivial]Fix some typos in docs https://review.opendev.org/745773 | 07:54 |
| gudrutis0 | hi ironic | 07:56 |
| *** Lucas_Gray has joined #openstack-ironic | 08:01 | |
| *** lucasagomes has joined #openstack-ironic | 08:04 | |
| *** Wryhder has joined #openstack-ironic | 08:14 | |
| *** Lucas_Gray has quit IRC | 08:15 | |
| *** Wryhder is now known as Lucas_Gray | 08:15 | |
| *** mkrai has quit IRC | 08:17 | |
| *** brtknr has left #openstack-ironic | 08:21 | |
| rpittau | heu gudrutis0 :) | 08:27 |
| rpittau | oooook I think we're good to request a release for ironic | 08:27 |
| *** belmoreira has joined #openstack-ironic | 08:30 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: migrate testing to ubuntu focal https://review.opendev.org/739448 | 08:34 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: Use ubuntu focal for ipa src https://review.opendev.org/725794 | 08:35 |
| iurygregory | rpittau, release \o/ | 08:40 |
| iurygregory | and I think I've found the problem with my partition/wholedisk testing | 08:40 |
| rpittau | oh cool :) | 08:40 |
| iurygregory | I think | 08:41 |
| iurygregory | nova can't match the capabilities so we end up with no valid Host | 08:41 |
| iurygregory | so maybe (I'm not 100% sure) I need to add some bits in https://github.com/openstack/ironic/blob/a2a27d57fafe1073313a8e97dfcd844a7ceec02e/devstack/lib/ironic#L2294 | 08:42 |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/ussuri: Temporary switch ironicclient-tempest non-voting https://review.opendev.org/746026 | 08:45 |
| openstackgerrit | Merged openstack/python-ironicclient master: Fix expected exception message in one test https://review.opendev.org/745834 | 08:46 |
| openstackgerrit | Merged openstack/ironic-python-agent bugfix/6.2: Ignore devices with size 0 when collecting inventory https://review.opendev.org/745493 | 08:46 |
| *** mkrai has joined #openstack-ironic | 08:48 | |
| *** JamesBenson has joined #openstack-ironic | 08:51 | |
| *** JamesBenson has quit IRC | 08:56 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/6.2: Fix bootloader install issue with MDRAID https://review.opendev.org/746029 | 08:57 |
| rpittau | mmm still issues with vexxhost apparently | 08:59 |
| *** rcernin has quit IRC | 09:05 | |
| iurygregory | rpittau, if I want to update the properties field of a node would be http://paste.openstack.org/show/796809/ ? | 09:22 |
| iurygregory | this is the equivalent for penstack baremetal node set --property capabilities="boot_option:local" <node> | 09:22 |
| *** Lucas_Gray has quit IRC | 09:22 | |
| rpittau | iurygregory: yep, that should work | 09:24 |
| iurygregory | fingers crossed | 09:24 |
| *** JamesBenson has joined #openstack-ironic | 09:26 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops https://review.opendev.org/741445 | 09:27 |
| *** Lucas_Gray has joined #openstack-ironic | 09:30 | |
| *** JamesBenson has quit IRC | 09:30 | |
| *** Lucas_Gray has quit IRC | 09:48 | |
| *** Wryhder has joined #openstack-ironic | 09:48 | |
| *** Wryhder is now known as Lucas_Gray | 09:49 | |
| *** rcernin has joined #openstack-ironic | 09:57 | |
| rpittau | carbonara is calling me, brb | 10:11 |
| iurygregory | enjoy! | 10:12 |
| janders | carbonara - yum! :) | 10:28 |
| *** mkrai has quit IRC | 10:28 | |
| janders | iurygregory would you have two minutes to run me through debugging CI failures of my change? | 10:28 |
| iurygregory | janders, I'm about to start my lunch =X since I will have 2hrs of meetings in 30min... | 10:29 |
| iurygregory | send me the link and I will look after the meetings =) | 10:29 |
| janders | ok! :) | 10:31 |
| janders | thank you | 10:31 |
| *** mkrai has joined #openstack-ironic | 10:31 | |
| janders | and we can chat more in detail on Monday | 10:31 |
| iurygregory | totally! | 10:31 |
| janders | iurygregory https://review.opendev.org/#/c/744117/ | 10:34 |
| patchbot | patch 744117 - ironic - [WIP]Generic way to configure clean step priorites - 4 patch sets | 10:34 |
| janders | all the HP stuff is likely properly broken. I ported the code from the abandoned patch. It runs and passes linter tests but I had no way of testing it against real hardware. | 10:34 |
| janders | the rest - I was a bit surprised with, cause my tests succeeded... | 10:35 |
| janders | but again the trouble with testing in VMs I suppose... | 10:35 |
| *** rcernin has quit IRC | 10:40 | |
| *** yolanda has quit IRC | 10:42 | |
| *** yolanda has joined #openstack-ironic | 10:46 | |
| *** rcernin has joined #openstack-ironic | 10:47 | |
| iurygregory | janders, seems like only problems with unit testing | 10:57 |
| iurygregory | if you run locally after fixing the problems should be fine | 10:57 |
| iurygregory | tox -r -vv -e py36 | 10:57 |
| iurygregory | tox -r -vv -e py38 | 10:57 |
| iurygregory | tox -r -vv -e cover | 10:57 |
| iurygregory | tox -r -vv -e unit-with-driver-libs | 10:58 |
| iurygregory | and tox -r -vv -e lower-constraints | 10:58 |
| janders | thank you iurygregory | 10:59 |
| rpittau | iurygregory: are you in-tox-icating janders ? :) | 10:59 |
| iurygregory | rpittau, LOL | 11:00 |
| janders | had one beer with dinner, I suppose that doesnt count | 11:00 |
| iurygregory | janders, I will try to reproduce locally after the meetings | 11:00 |
| janders | :) | 11:00 |
| *** rcernin has quit IRC | 11:00 | |
| *** lucasagomes is now known as lucas-hungry | 11:03 | |
| *** lucas-hungry is now known as lucasagomes | 11:03 | |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/ussuri: Temporary switch jobs to non-voting https://review.opendev.org/746026 | 11:06 |
| *** yolanda has quit IRC | 11:13 | |
| *** yolanda has joined #openstack-ironic | 11:20 | |
| openstackgerrit | Riccardo Pittau proposed openstack/python-ironicclient stable/ussuri: Temporary switch jobs to non-voting https://review.opendev.org/746026 | 11:28 |
| *** mkrai has quit IRC | 11:43 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:55 | |
| *** rh-jelabarre has quit IRC | 11:55 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:56 | |
| janders | wrapping up for the night, have a great long weekend everyone o/ | 12:02 |
| rpittau | janders: thanks! you too! | 12:07 |
| *** riuzen has joined #openstack-ironic | 12:45 | |
| *** JamesBenson has joined #openstack-ironic | 12:46 | |
| *** JamesBenson has quit IRC | 12:51 | |
| *** rloo has joined #openstack-ironic | 12:56 | |
| *** JamesBenson has joined #openstack-ironic | 13:01 | |
| rpittau | TheJulia: we unfortunately need to do this -> https://review.opendev.org/746026 | 13:07 |
| patchbot | patch 746026 - python-ironicclient (stable/ussuri) - Temporary switch jobs to non-voting - 3 patch sets | 13:07 |
| iurygregory | enjoy the long weekend janders o/ | 13:07 |
| * iurygregory is alive after 3 consecutive meetings \o/ | 13:08 | |
| TheJulia | Joy | 13:08 |
| *** ociuhandu has quit IRC | 13:09 | |
| *** trandles has joined #openstack-ironic | 13:09 | |
| *** mkrai has joined #openstack-ironic | 13:15 | |
| TheJulia | on a plus side, the change should merge quickly | 13:19 |
| rpittau | yeah, sorry about that :/ | 13:22 |
| rpittau | we missed a couple of backports and zuul punished us | 13:22 |
| TheJulia | c'est la vie | 13:23 |
| *** ociuhandu has joined #openstack-ironic | 13:24 | |
| TheJulia | sigh, our ci is not happy | 13:27 |
| *** ociuhandu has quit IRC | 13:29 | |
| iurygregory | good morning TheJulia =) | 13:31 |
| *** tkajinam has quit IRC | 13:32 | |
| openstackgerrit | Merged openstack/python-ironicclient stable/ussuri: Temporary switch jobs to non-voting https://review.opendev.org/746026 | 13:32 |
| openstackgerrit | Julia Kreger proposed openstack/python-ironicclient stable/ussuri: Revert "Temporary switch jobs to non-voting" https://review.opendev.org/746136 | 13:34 |
| *** tzumainn has joined #openstack-ironic | 13:38 | |
| TheJulia | I'm glad it is my end of week | 13:53 |
| iurygregory | ++ | 13:54 |
| TheJulia | rpioso: You may want to add some dell folks to sushy-oem-idrac-core in gerrit since right now it is ilya and yourself. | 13:57 |
| *** ociuhandu has joined #openstack-ironic | 14:02 | |
| rpioso | TheJulia: ack | 14:03 |
| *** abdysn has quit IRC | 14:06 | |
| iurygregory | yay community voting for the Summit Sessions is open | 14:13 |
| *** ociuhandu_ has joined #openstack-ironic | 14:19 | |
| iurygregory | funny, I'm Foundation Member but I can't vote LOL | 14:19 |
| TheJulia | wtf | 14:20 |
| *** ociuhandu has quit IRC | 14:22 | |
| iurygregory | well I think the page has problems atm, since it only shows a field and Log in as someone else hehehe | 14:23 |
| TheJulia | rpittau: thoughts on merging https://review.opendev.org/#/c/745935/ for the release of ironic, so people outside UTC can run unit tests? | 14:35 |
| patchbot | patch 745935 - ironic - Fix time usage in unit tests for agent power inter... - 1 patch set | 14:35 |
| *** riuzen has quit IRC | 14:35 | |
| * iurygregory looks | 14:36 | |
| iurygregory | I would say it's worth merging and put in the release | 14:37 |
| rpittau | oh didn't see that | 14:37 |
| rpittau | I'll update the release request as soon as it merges | 14:38 |
| TheJulia | k | 14:38 |
| *** vmud213 has joined #openstack-ironic | 14:45 | |
| *** mkrai has quit IRC | 14:52 | |
| *** mkrai_ has joined #openstack-ironic | 14:52 | |
| *** k_mouza has joined #openstack-ironic | 14:54 | |
| TheJulia | rpioso: thanks! I asked because there are some backport requests for patches that have not merged yet. | 14:58 |
| openstackgerrit | Merged openstack/ironic master: Fix time usage in unit tests for agent power interface https://review.opendev.org/745935 | 14:58 |
| openstackgerrit | Merged openstack/python-ironicclient stable/ussuri: Fix missing tox in functional devstack-minimal based job https://review.opendev.org/737920 | 14:58 |
| openstackgerrit | Merged openstack/python-ironicclient stable/ussuri: Restore default netboot boot option https://review.opendev.org/744659 | 14:58 |
| *** rloo has quit IRC | 15:08 | |
| rpittau | TheJulia, iurygregory, I've been made aware that we ahven't released ironic-prometheus-exporter and ironic-ui in a while, we should probably get to that as well, especially the former | 15:15 |
| iurygregory | rpittau, we need bugfix branches for them? | 15:15 |
| iurygregory | I would say we can just do the stable release | 15:16 |
| *** JamesBen_ has joined #openstack-ironic | 15:16 | |
| TheJulia | we shouldn't need to stable branch them | 15:16 |
| rpittau | they're both cycle-with-intermediary so they should have bugfix branches as the others with the same release model | 15:16 |
| TheJulia | ++ | 15:16 |
| *** JamesBenson has quit IRC | 15:17 | |
| iurygregory | but we need to create the branch even if we haven't merge something? (just wondering) | 15:17 |
| TheJulia | although, mild changes may not justify a bugfix branch | 15:17 |
| TheJulia | because we shouldn't artificially change the minor number | 15:17 |
| iurygregory | the ipe would be worth, I'm going to update the patch that someone submitted and forgot | 15:17 |
| TheJulia | (this is where openstack's release model is broken and insane) | 15:18 |
| rpittau | we COULD release not as intermediate but as simple bugfix | 15:18 |
| TheJulia | That would meet the additional two releases a year requirement | 15:18 |
| TheJulia | err | 15:18 |
| TheJulia | two a cycle | 15:18 |
| TheJulia | blah | 15:18 |
| rpittau | that yeah :D | 15:18 |
| iurygregory | ++ | 15:18 |
| TheJulia | its artificially insane | 15:18 |
| TheJulia | anyway! | 15:19 |
| * TheJulia looks at the ui | 15:19 | |
| rpittau | Sean was suggesting that we could change both to cycle-with-rc | 15:20 |
| TheJulia | that too is insane | 15:21 |
| TheJulia | there is no need to do release candidates | 15:21 |
| rpittau | ok | 15:21 |
| * TheJulia sighs | 15:21 | |
| * rpittau really need some rhum ice-cream | 15:21 | |
| TheJulia | it looks like ironic-ui, we should just cut a release | 15:21 |
| TheJulia | no bugfix branch, it is more tied to openstack as a whole | 15:22 |
| rpittau | sounds good to me | 15:22 |
| TheJulia | it can be a minor or even pure bugfix release | 15:22 |
| TheJulia | one of the patches actually could use backporting... | 15:22 |
| TheJulia | but yeah, its mild | 15:22 |
| rpittau | ok, let me check the list | 15:22 |
| iurygregory | we can do a release of the ipe | 15:23 |
| TheJulia | ipe looks like a minor release would work because six was removed | 15:23 |
| rpittau | so maybe we can cut a bugfix branch for that ? | 15:24 |
| TheJulia | yeah, that seems reasonable | 15:24 |
| iurygregory | can be, we have a few commits since ussuri | 15:24 |
| TheJulia | not sure we'll use it but it doesn't really cost anything | 15:24 |
| iurygregory | py38 metadata, removal of requirements | 15:24 |
| TheJulia | oh, see the py38 metadata is a minor change | 15:25 |
| rpittau | mmmm I think we need to cut ironic-ui feature as well as it's the first for victoria | 15:25 |
| TheJulia | gah, yeah | 15:26 |
| TheJulia | 4.0.0 was the last | 15:26 |
| rpittau | yep | 15:26 |
| TheJulia | well | 15:26 |
| TheJulia | we only need to if we anticipate needing a branch | 15:26 |
| TheJulia | and needing to patch it | 15:26 |
| TheJulia | it could just as easily be 4.0.1 | 15:26 |
| rpittau | the script heavily objects | 15:27 |
| rpittau | new-release: error: The first release for a series must be at least a feature release to allow for stable releases from the previous series. | 15:27 |
| * TheJulia sighs | 15:27 | |
| rpittau | :/ | 15:27 |
| TheJulia | we've codified anti-semver | 15:27 |
| TheJulia | minor it I guess :( | 15:28 |
| rpittau | I'll go for 4.1.0 | 15:28 |
| TheJulia | \k, thanks | 15:28 |
| *** k_mouza has quit IRC | 15:30 | |
| rpittau | there it is https://review.opendev.org/746161 | 15:30 |
| patchbot | patch 746161 - releases - Release ironic-ui 4.1.0 for victoria - 1 patch set | 15:30 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops https://review.opendev.org/741445 | 15:39 |
| iurygregory | ok maybe now it won't hut the inspector jobs | 15:41 |
| iurygregory | fingers crossed | 15:41 |
| *** belmoreira has quit IRC | 15:42 | |
| rpittau | rmrr I think I forgot std-with-versions | 15:45 |
| *** priteau has joined #openstack-ironic | 15:46 | |
| rpittau | hehhhhh we didn't actually mention ironic-ui in the new release model, nor ipe | 15:49 |
| iurygregory | =X | 15:49 |
| iurygregory | no bugfix branches lol | 15:49 |
| rpittau | I'm wondering if it makes sense for ironic-ui, considering that is a plugin | 15:50 |
| rpittau | ipe does make sense to me | 15:50 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops https://review.opendev.org/741445 | 15:52 |
| iurygregory | ipe does | 15:52 |
| *** mgoddard has quit IRC | 15:54 | |
| * rpittau ice-cream is getting closer | 15:56 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-inspector bugfix/10.3: Update .gitreview for bugfix/10.3 https://review.opendev.org/746174 | 15:56 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-inspector bugfix/10.3: Update TOX_CONSTRAINTS_FILE for bugfix/10.3 https://review.opendev.org/746175 | 15:56 |
| iurygregory | hehe the tox is not needed | 15:58 |
| openstackgerrit | OpenStack Release Bot proposed openstack/bifrost bugfix/8.3: Update .gitreview for bugfix/8.3 https://review.opendev.org/746178 | 15:58 |
| openstackgerrit | OpenStack Release Bot proposed openstack/bifrost bugfix/8.3: Update TOX_CONSTRAINTS_FILE for bugfix/8.3 https://review.opendev.org/746179 | 15:58 |
| *** mgoddard has joined #openstack-ironic | 16:01 | |
| *** rloo has joined #openstack-ironic | 16:12 | |
| *** mkrai_ has quit IRC | 16:22 | |
| *** dougsz has quit IRC | 16:31 | |
| *** lucasagomes has quit IRC | 16:33 | |
| *** Lucas_Gray has quit IRC | 16:36 | |
| *** gyee has joined #openstack-ironic | 16:40 | |
| *** ociuhandu_ has quit IRC | 16:43 | |
| *** tosky has quit IRC | 16:43 | |
| *** ociuhandu has joined #openstack-ironic | 16:44 | |
| *** ociuhandu has quit IRC | 16:50 | |
| *** ociuhandu has joined #openstack-ironic | 16:57 | |
| *** vmud213 has quit IRC | 17:00 | |
| *** ociuhandu has quit IRC | 17:04 | |
| *** JayF has quit IRC | 17:15 | |
| *** JayF has joined #openstack-ironic | 17:16 | |
| openstackgerrit | Merged openstack/python-ironicclient stable/ussuri: Revert "Temporary switch jobs to non-voting" https://review.opendev.org/746136 | 17:35 |
| *** priteau has quit IRC | 17:36 | |
| *** priteau has joined #openstack-ironic | 17:45 | |
| *** priteau has quit IRC | 17:53 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-inspector master: Fix URL parsing to enable redfish_address matching https://review.opendev.org/745972 | 18:05 |
| dking | How would I go about finding out why a server is stuck in "wait call-back" after deploy? I'm logged into the server with a devuser. I saw the message previously that the deploy image was being written to disk, but after that, I've been seeing "heartbeat successful" messages. How could I go about finding out what it's waiting on before being rebooted? | 18:05 |
| TheJulia | Anyone know if any vendors have implemented the redfish host bus interface spec yet? | 18:13 |
| TheJulia | dking: is it in maintenance state? | 18:14 |
| TheJulia | dking: in maintenance the heartbeats can be succesfful but they won't trigger the next action | 18:14 |
| dking | TheJulia: Surprisingly, no. It's just been sitting like this for over an hour, on 3 separate nodes. | 18:15 |
| TheJulia | hmmm | 18:15 |
| TheJulia | what exactly si the last thing ipa has logged? | 18:15 |
| TheJulia | what does the process list look like intenrally to the ramdisk | 18:15 |
| TheJulia | ? | 18:15 |
| TheJulia | Also, any chance you can get the openstack baremetal node show <uuid> output for us? | 18:15 |
| TheJulia | tl;dr, it shouldn't be stuck | 18:16 |
| TheJulia | but depending on deploy_interface and some other settings, something might be going on | 18:16 |
| dking | I can get the later. What's the best way to get the others? | 18:16 |
| TheJulia | also what version of the code? | 18:16 |
| TheJulia | well, you mentioned you logged into the ramdisk, I'd just go back in, run "ps auxf" and save that, and then check either the journalctl output for ipa or /var/log/messages | 18:17 |
| dking | This should be train, but I can get the specifics. It was probably current train from a few months ago. | 18:17 |
| TheJulia | yeah, we basically need to figure out what exactly is going on because it shouldn't be doing what your observing | 18:18 |
| dking | Yeah, I haven't seen it before, either. I'm also interested in knowing the troubleshooting process. So, the last things in /var/log/messages is a message about "Error canonicalizing /boot/grub2/grubenv ..." and before "Failed to start Mark boot as successful. | 18:21 |
| TheJulia | dking: ++ yeah, I guess I'm trying to get you to collect the data to dig through things | 18:21 |
| dking | nevermind, that's systemd | 18:21 |
| TheJulia | :( | 18:21 |
| TheJulia | keep looking :) | 18:21 |
| TheJulia | what deploy_interface? | 18:22 |
| openstackgerrit | Merged openstack/ironic-inspector bugfix/10.3: Update .gitreview for bugfix/10.3 https://review.opendev.org/746174 | 18:22 |
| openstackgerrit | Merged openstack/ironic-inspector bugfix/10.3: Update TOX_CONSTRAINTS_FILE for bugfix/10.3 https://review.opendev.org/746175 | 18:22 |
| openstackgerrit | Merged openstack/ironic-python-agent bugfix/6.2: Fix bootloader install issue with MDRAID https://review.opendev.org/746029 | 18:22 |
| dking | "deploy_interface": "direct" | 18:23 |
| TheJulia | hmm | 18:23 |
| TheJulia | so everything should be happening on the ramdisk | 18:24 |
| dking | Let me get the whole node show for you, and some more info. | 18:24 |
| dking | Yep. | 18:24 |
| TheJulia | likely good check to see that nothing is still running as a process from the ironic-python-agent | 18:24 |
| TheJulia | which is why I asked about the ps | 18:24 |
| TheJulia | since while it is still running the agent will continue to heartbeat | 18:24 |
| dking | http://paste.openstack.org/show/796827/ | 18:25 |
| dking | Yeah. ps is large, and I'm having to use a KVM, which is annoying, but I'm glad that I have that. | 18:25 |
| TheJulia | it looks like it is just waiting for the ramdisk to be done | 18:28 |
| TheJulia | what is the virtual size of the qcow? | 18:28 |
| dking | There was just a bunch of "GET /v1/commands", but filtering that out, and the heartbeats, /var/log/messages shows last htat there was a "POST /v1/commands?wait=false" with a 200 response, and going backwards, RuntimeWarning, `static_root` is only used when `debug` is True, ignoring, and before that, Writing image with command... | 18:29 |
| dking | I think the virtual size is about 4G. The image itself is 2.6G. | 18:30 |
| dking | Oh, wow. Write_image.sh is still running. | 18:31 |
| TheJulia | so your getting there :) | 18:31 |
| TheJulia | any kernel errors in the ramdisk "dmesg" | 18:31 |
| dking | I'm installing strace | 18:31 |
| TheJulia | ++ | 18:32 |
| dking | Let me check that, too | 18:32 |
| dking | No such dmesg messages that I can see. The last things in there are the NICs coming up. | 18:33 |
| dking | Well, wouldn't you know, the script is just waiting, too. strace can only see "wait4(-1 " | 18:33 |
| dking | iotop shows an occasional "qemu-img convert -t directsync ..." | 18:36 |
| dking | Well, I suppose that mostly answers my Ironic portion of the issue. I suppose that IPA is trying to run write_image.sh, which is probably running qemu-img in a shell, and that's hung, but as far as Ironic is concerned, it's still writing the image, and as long as heartbeat is being answered, it won't be considered dead? | 18:41 |
| JayF | exactly | 18:42 |
| dking | Neat. Well, I suppose that it's time for me to start looking into what's up with qemu. It's odd, though, that I just launched another node a few minutes prior on a machine that should be similar to these. | 18:43 |
| TheJulia | It kind of sounds like maybe the hypervisor backend io is throttled | 18:59 |
| TheJulia | I've seen similar things when IO is not timing out, but is taking a super long time to complete | 18:59 |
| dking | hypervisor? | 18:59 |
| dking | This is baremetal. | 18:59 |
| TheJulia | oh right | 18:59 |
| TheJulia | yeah | 18:59 |
| TheJulia | weird | 18:59 |
| TheJulia | well.. hmm | 19:00 |
| JayF | disk failure? | 19:00 |
| dking | yeah. I do see that other people have had similar issues in the past, and some were patched. | 19:00 |
| dking | I'd like to try running the command directly myself. Is there a way to stop that command safely? I'm also not terribly familiar with how IPA works on the node. If I kill the process, would that fail and trigger Ironic to request a reboot? | 19:01 |
| *** openstackgerrit has quit IRC | 19:02 | |
| TheJulia | you can run it directly from with-in the ramdisk, but set the node to maintenance in the conductor before you kill the agent off | 19:39 |
| *** tosky has joined #openstack-ironic | 20:10 | |
| *** openstackgerrit has joined #openstack-ironic | 20:26 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-tempest-plugin master: Update basic_ops https://review.opendev.org/741445 | 20:26 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-prometheus-exporter bugfix/2.1: Update .gitreview for bugfix/2.1 https://review.opendev.org/746223 | 20:35 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-prometheus-exporter bugfix/2.1: Update TOX_CONSTRAINTS_FILE for bugfix/2.1 https://review.opendev.org/746224 | 20:35 |
| stevebaker | morning | 20:43 |
| JayF | Do we support Available->Cleaning->Available manual cleaning? e.g. manual cleaning for a node without taking the instance off? | 21:12 |
| dking | TheJulia: Thank you. So, as long as it's in maintenance mode, it won't alter the power state manually? | 21:16 |
| JayF | dking: correct | 21:18 |
| dking | That's handy. I appreciate the help. | 21:18 |
| JayF | np, gl | 21:20 |
| dking | JayF: I thought that was the process for manual cleaning? | 21:31 |
| rloo | JayF: No, I don't think so. to do manual clean means going to manageable. | 21:34 |
| rloo | JayF: wondering what the use case might be for doing a 'clean' when there is an instance on it | 21:35 |
| JayF | dking: when I was more ironic-y, and manual cleaning first started, you had to go to `manageable` first, which typically reuqired deleting the instance, so it'd go from manageable->cleaning->manageable | 21:35 |
| JayF | rloo: "cleaning" is just a fancy word for "run arbitrary code from IPA ramdisk" | 21:35 |
| JayF | rloo: that could include firmware upgrades for in-service machines (with downtime, obviously) | 21:35 |
| rloo | JayF: yeah, i was wondering if that was what you were thinking... | 21:36 |
| JayF | I don't have specific use cases in mind beyond that, just considering the utility that a user could get by being the "lessee" of the instance and given access to do stuff with it | 21:36 |
| dking | Oh, interesting. I've been looking into doing firmware cleaning also. I would be interested in what you're doing. Currently, I'm looking at Redfish and having an external command do the cleaning, but I would love to have it done on disk. | 21:37 |
| dking | I suppose that it requires custom code built into the IPA ramdisk? | 21:37 |
| rloo | JayF: It seems like you want it to be active -> some pause state, maybe manageable-like -> manual clean -> active ? | 21:37 |
| JayF | dking: I'm not doing anything but conjecturing :D. In-band cleaning is pretty significantly configurable, and there are examples for doing it (that last week got moved into IPA code) | 21:37 |
| JayF | rloo: I mean, or even just active -> cleaning -> active | 21:38 |
| JayF | rloo: just like it'd do for rescue, but automated | 21:38 |
| JayF | we don't mind nuking the p ower state of an active instance in the rescue case | 21:38 |
| dking | JayF: Awesome! Do you happen to know of any documentation for that? | 21:38 |
| JayF | dking: tons. search for "hardware manager" in the IPA docs, and I'm going to link you the examples | 21:38 |
| rloo | JayF: an interesting idea. I don't see why we couldn't do it. | 21:38 |
| JayF | dking: https://opendev.org/openstack/ironic-python-agent/src/branch/master/examples | 21:39 |
| dking | JayF: Thank you very much. | 21:40 |
| JayF | rloo: yeah, like I said, no real use case, just thinking about how we can make the ironic ecosystem more useful to sysadmins-outside-of-openstack-admins if they have significant access to the node via the node ownership changes | 21:40 |
| JayF | and self-serve IPA ramdisks, with custom hardware managers, used as a one-stop-shop for cleaning-between-provisions AND maintaining a node that is currently provisioned | 21:41 |
| rloo | JayF: I think 'upgrading firmware' is a great usecase. | 21:41 |
| dking | I think that's a critical piece, especially any time you might be recycling servers between customers. | 21:43 |
| rloo | dking: well, if you recycle servers between customers you want to do 'full' cleaning, but that'd be after the instance has been deleted, ie node goes from active -> deleting? -> cleaning -> available. which ironic supports now (just not the firmware upgrade part...) | 21:45 |
| dking | Exactly. We can do cleaning, but as far as I'm aware, nothing that updates firmware. | 21:46 |
| dking | Which is critical, because the firmware, unfortunately, can be vulnerable to attack from within the server. | 21:46 |
| JayF | dking: When I worked in a large public cloud that did baremetal, we had a cleaning process that included a secure erase step (often using vendor utilities in addition to upstream erase_devices), firmware upgrades and validation, device health checks (e.g. is all the ram there? are all the disks there?), and BIOS settings. All done from in-band IPA Hardware Managers. | 21:47 |
| dking | That would be the best process. I need to do some more learning about IPA. I've just been using it for the stock stuff, and never looked at Hardware Managers before. Knowing that it can be done that way means I need to do some research. | 21:50 |
| dking | Even if it's run after undeploy is fine for my use. Of course, you're right that it would be nice to be able to run it without undeploying. | 21:51 |
| JayF | An easy first step is building your own IPA instead of using the upstream-provided prebuilt images. Once you get to that step, it's pretty easy to start customizing it further. (and since most advanced cleaning would require vendor utilities, it's a prereq even before you get your code in there) | 21:52 |
| JayF | well there's automated cleaning, (enabled by [deploy]automated_clean=true) which runs cleaning on delete, before the node becomes available | 21:52 |
| JayF | what we were talking about before was manual cleaning, which is when you can run specific clean steps as desired via an API call | 21:52 |
| JayF | they use similar mechanisms though | 21:52 |
| dking | Yeah. I'm going to have to read up a bit. I'll probably be asking questions in here after. | 21:53 |
| *** mgoddard has quit IRC | 22:00 | |
| *** rcernin has joined #openstack-ironic | 22:00 | |
| *** patchbot has quit IRC | 22:02 | |
| *** mgoddard has joined #openstack-ironic | 22:05 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Detail iPXE + LACP troubleshooting information https://review.opendev.org/743787 | 22:09 |
| openstackgerrit | Christopher Dearborn proposed openstack/sushy master: WIP: Fix firmware update to return a Task https://review.opendev.org/745950 | 22:11 |
| *** chihhsin_ has quit IRC | 22:38 | |
| *** chihhsin has joined #openstack-ironic | 22:43 | |
| *** trandles has quit IRC | 22:55 | |
| *** tkajinam has joined #openstack-ironic | 22:58 | |
| *** ociuhandu has joined #openstack-ironic | 23:00 | |
| *** ociuhandu has quit IRC | 23:05 | |
| *** tosky has quit IRC | 23:09 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!