Friday, 2020-10-02

« Thursday, 2020-10-01 Index Saturday, 2020-10-03 »
*** k_mouza has joined #openstack-ironic00:06
*** k_mouza has quit IRC00:10
TheJuliaCI really  dislikes me :(00:16
stevebakerTheJulia: I have learnt what ipmitool does, it already retries all commands (including close). by default it retries 4 times, the wait before the first retry is 1 second and the wait increments by 1 second each attempt. This is controllable with cli options -N <seconds> and -R <retries>00:29
stevebakerthis is master CI, I need to see what patches are downstream00:29
stevebakermaster ipmitool00:29
*** weshay|ruck has quit IRC00:54
*** weshay has joined #openstack-ironic00:55
*** weshay is now known as weshay|ruck00:55
*** k_mouza has joined #openstack-ironic01:05
*** k_mouza has quit IRC01:09
*** zzzeek has quit IRC01:17
*** zzzeek has joined #openstack-ironic01:19
TheJuliastevebaker:  hmmm :/. Yeah, let's see what downstream has. Could be the stonith plugin need some arguments. :/01:32
*** k_mouza has joined #openstack-ironic01:33
*** k_mouza has quit IRC01:38
*** zzzeek has quit IRC01:51
*** zzzeek has joined #openstack-ironic01:51
*** k_mouza has joined #openstack-ironic01:51
*** k_mouza has quit IRC01:56
*** cdearborn has quit IRC02:15
*** k_mouza has joined #openstack-ironic02:27
*** k_mouza has quit IRC02:31
*** rcernin has quit IRC03:31
*** mkrai has joined #openstack-ironic03:50
*** zzzeek has quit IRC03:52
*** zzzeek has joined #openstack-ironic03:52
*** rcernin has joined #openstack-ironic03:59
*** rcernin has quit IRC04:13
*** zaneb has quit IRC04:20
*** rcernin has joined #openstack-ironic04:30
*** zaneb has joined #openstack-ironic04:32
*** bnemec has quit IRC05:23
*** gyee has quit IRC05:26
*** bnemec has joined #openstack-ironic05:28
*** jtomasek has joined #openstack-ironic05:39
*** mkrai has quit IRC05:52
*** mkrai has joined #openstack-ironic05:56
*** zzzeek has quit IRC05:59
*** zzzeek has joined #openstack-ironic06:01
*** mkrai has quit IRC06:01
*** mkrai has joined #openstack-ironic06:20
*** uzumaki has joined #openstack-ironic06:22
*** mkrai has quit IRC06:27
arne_wiebalckGood morning, ironic!06:33
*** hjensas|afk has joined #openstack-ironic06:38
*** yolanda has quit IRC07:04
*** yolanda has joined #openstack-ironic07:04
openstackgerritOpenStack Proposal Bot proposed openstack/ironic master: Imported Translations from Zanata  https://review.opendev.org/75571307:13
*** dsneddon has quit IRC07:26
*** rcernin has quit IRC07:27
*** dsneddon has joined #openstack-ironic07:28
*** mkrai has joined #openstack-ironic07:33
*** tosky has joined #openstack-ironic07:43
iurygregorygood morning arne_wiebalck o/07:48
iurygregorygood morning Ironic!07:48
iurygregoryhappy friday \o/07:48
*** rcernin has joined #openstack-ironic07:52
*** rcernin has quit IRC07:56
*** mkrai has quit IRC07:57
*** mkrai has joined #openstack-ironic07:57
*** lucasagomes has joined #openstack-ironic08:07
arne_wiebalckhey iurygregory o/08:17
iurygregoryo/08:17
openstackgerritMerged openstack/ironic master: Clarify power state logging  https://review.opendev.org/75565608:19
*** mkrai has quit IRC08:25
*** derekh has joined #openstack-ironic08:26
*** tkajinam has quit IRC08:29
*** tkajinam has joined #openstack-ironic08:29
*** k_mouza has joined #openstack-ironic08:36
openstackgerritIvan Kolodyazhny proposed openstack/ironic-ui stable/victoria: Add npm tox env  https://review.opendev.org/75572708:41
*** k_mouza has quit IRC08:41
*** k_mouza has joined #openstack-ironic08:42
*** rcernin has joined #openstack-ironic08:58
openstackgerritAija Jaunteva proposed openstack/ironic master: Fix idrac-wsman RAID step async error handling  https://review.opendev.org/75573108:59
*** rcernin has quit IRC09:03
openstackgerritVerification of a change to openstack/bifrost failed: Update .gitreview for stable/victoria  https://review.opendev.org/75562609:04
openstackgerritVerification of a change to openstack/bifrost failed: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/75562709:04
*** mkrai has joined #openstack-ironic09:07
*** tkajinam has quit IRC09:18
*** QianbiaoNG has joined #openstack-ironic09:18
*** mkrai has quit IRC09:20
openstackgerritMerged openstack/ironic-inspector stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75561309:24
openstackgerritMerged openstack/ironic-inspector stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/75561409:24
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic-prometheus-exporter master: Add Python3 wallaby unit tests  https://review.opendev.org/75561209:48
*** ociuhandu has joined #openstack-ironic09:48
*** hjensas|afk has quit IRC09:49
openstackgerritMerged openstack/ironic-python-agent stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75562009:49
openstackgerritMerged openstack/ironic-python-agent stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/75562209:49
openstackgerritMerged openstack/ironic-prometheus-exporter master: Update master for stable/victoria  https://review.opendev.org/75561109:49
*** rcernin has joined #openstack-ironic09:50
*** rcernin has quit IRC09:55
openstackgerritMerged openstack/bifrost master: Update master for stable/victoria  https://review.opendev.org/75562810:08
*** ricolin_ has joined #openstack-ironic10:17
openstackgerritMerged openstack/ironic-ui master: Update master for stable/victoria  https://review.opendev.org/75561910:19
openstackgerritMerged openstack/ironic-ui master: Add Python3 wallaby unit tests  https://review.opendev.org/75562110:19
openstackgerritMerged openstack/ironic-ui stable/victoria: Add npm tox env  https://review.opendev.org/75572710:30
openstackgerritMerged openstack/bifrost master: Add Python3 wallaby unit tests  https://review.opendev.org/75562910:34
iurygregoryif any core have some time to approve https://review.opendev.org/#/c/755612/ https://review.opendev.org/#/c/755609/10:39
patchbotpatch 755612 - ironic-prometheus-exporter - Add Python3 wallaby unit tests - 2 patch sets10:39
patchbotpatch 755609 - ironic-prometheus-exporter (stable/victoria) - Update .gitreview for stable/victoria - 3 patch sets10:39
openstackgerritVerification of a change to openstack/metalsmith failed: Rework jobs so failed devstack does not retry the job  https://review.opendev.org/75268011:02
*** mkrai has joined #openstack-ironic11:06
*** lmcgann_ has joined #openstack-ironic11:16
openstackgerritMerged openstack/ironic-inspector master: Update master for stable/victoria  https://review.opendev.org/75561511:23
openstackgerritMerged openstack/sushy-tools master: Fix race condition initialising persistent dict  https://review.opendev.org/75295311:27
openstackgerritMerged openstack/sushy-tools master: Add Volumes nav property to Storage template  https://review.opendev.org/75190311:39
*** sshnaidm is now known as sshnaidm|afk11:46
*** stendulker has joined #openstack-ironic11:50
*** mkrai has quit IRC11:51
*** iurygregory has quit IRC11:55
*** belmoreira has joined #openstack-ironic11:56
*** iurygregory has joined #openstack-ironic11:57
openstackgerritAija Jaunteva proposed openstack/ironic master: Fix redfish BIOS apply config error handling  https://review.opendev.org/75577612:01
openstackgerritVerification of a change to openstack/ironic failed: CI: Additional minor job changes to improve CI reliability  https://review.opendev.org/75510012:05
openstackgerritMerged openstack/ironic-inspector master: Add Python3 wallaby unit tests  https://review.opendev.org/75561612:14
openstackgerritMerged openstack/ironic master: Update master for stable/victoria  https://review.opendev.org/75563212:14
openstackgerritMerged openstack/ironic-python-agent master: Update master for stable/victoria  https://review.opendev.org/75562312:14
openstackgerritVerification of a change to openstack/ironic-python-agent failed: Add Python3 wallaby unit tests  https://review.opendev.org/75562412:43
*** rcernin has joined #openstack-ironic12:45
*** k_mouza has quit IRC12:54
*** k_mouza has joined #openstack-ironic12:55
iurygregory(╯°□°)╯︵ ┻━┻ zuul stop trolling us12:58
*** ricolin_ has quit IRC13:05
*** ricolin_ has joined #openstack-ironic13:06
*** rcernin has quit IRC13:07
openstackgerritMerged openstack/ironic-prometheus-exporter stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75560913:10
*** Goneri has joined #openstack-ironic13:13
*** dougsz has joined #openstack-ironic13:27
*** ricolin_ has quit IRC13:30
*** uzumaki has quit IRC13:33
*** rloo has joined #openstack-ironic13:34
*** tzumainn has joined #openstack-ironic13:35
*** ricolin_ has joined #openstack-ironic13:52
*** priteau has joined #openstack-ironic13:52
*** stendulker has quit IRC13:53
*** cdearborn has joined #openstack-ironic13:55
TheJuliaiurygregory: where is it trolling us today?!?13:56
iurygregorythe stable/ussuri patch =(13:57
iurygregorymetalsmith TIMED_OUT...13:57
iurygregorygood morning TheJulia =)13:57
*** priteau has quit IRC13:58
TheJuliagrrrrrrrrrr13:58
iurygregoryyeah...13:59
iurygregoryI've update a few patches that the openstack bot sent most should be merged already13:59
TheJuliak14:01
TheJuliaugh, I see the backport of the job rework14:01
TheJuliaand *boom*14:01
TheJuliaiurygregory: where shall we focus with zuul hating us today?14:04
* TheJulia engages caffination14:05
trandles:( the October bare metal SIG meeting is at the same time as a one-off meeting I have with some program managers :(14:05
iurygregorywell so far it was only the stable/ussuri and some random failures in a few projects while creating stable/victoria and updating master to wallaby14:06
iurygregorywe should be fine already in most of the stalbe/victoria and master14:06
iurygregoryI'm looking at the failures in python-ironicinspector-client14:06
iurygregorywe need to merge https://review.opendev.org/#/c/751286/ ... but CI is un happy in one job14:08
patchbotpatch 751286 - python-ironic-inspector-client - Fix l-c job - 3 patch sets14:08
* iurygregory loves fixtures._fixtures.timeout.TimeoutException14:09
iurygregory-.-'14:09
iurygregoryok it may solve if we bump the introspection timeout...14:10
iurygregorygoing to try14:10
iurygregoryI think we did this for the inspector jobs..14:10
openstackgerritJulia Kreger proposed openstack/metalsmith master: CI: Extend job timeout slightly for metalsmith jobs  https://review.opendev.org/75579814:12
TheJulia++14:13
TheJuliayeah, we need to be super mindful about timeouts  and things like build_timeout14:13
TheJuliaalso if we're in a case where we could consume lots of ram, we need to be careful14:13
TheJuliaI'm not sure that metalsmith change is the right timeout14:14
TheJuliabut we'll see14:14
iurygregorystrange the timeouts are the same from the base job of ironic-inspector14:17
iurygregorythat is working just fine14:17
arne_wiebalckAs I start to annoy my colleagues already (as I seem to repeat this), let me say this here as well then: conductor groups is a massive usability improvement for larger deployments! Thanks jroll, and everyone who worked on this!14:18
*** priteau has joined #openstack-ironic14:19
TheJulia\o/14:19
* TheJulia is super glad we did that14:19
arne_wiebalckIt is a night and day difference.14:20
* arne_wiebalck tells this to his colleagues every time there are new nodes :-D14:21
iurygregoryhehehe14:21
iurygregoryboth nodes failed to be inspected https://zuul.opendev.org/t/openstack/build/384ab391b1204b0eb51647c6dc05e31b/log/controller/logs/screen-ironic-inspector.txt O.o14:23
TheJuliai see what is going on with metalsmith14:24
openstackgerritJulia Kreger proposed openstack/metalsmith master: Handle out-of-tree openstack module_utils  https://review.opendev.org/75387014:26
jroll\o/ awesome arne_wiebalck, good to here :)14:26
jrollhear even14:26
arne_wiebalck:)14:27
*** vkmc has quit IRC14:27
TheJuliaiurygregory: https://review.opendev.org/#/c/753870/2/.zuul.yaml is going to need to get backported14:27
patchbotpatch 753870 - metalsmith - Handle out-of-tree openstack module_utils - 2 patch sets14:27
TheJuliatoo14:27
arne_wiebalckjroll: in case you're interested, we summarised how we deployed it and what the impact was here: https://techblog.web.cern.ch/techblog/post/conductor-groups/14:28
jroll<314:28
jrollvery interested, thank you14:28
iurygregoryTheJulia, ack you want to push the backport?14:28
*** vkmc has joined #openstack-ironic14:29
TheJuliaiurygregory: will once that patch passes/merges into master14:29
iurygregoryTheJulia, ack, I will review the backport when available =)14:30
TheJuliak14:30
TheJuliacdearborn: fyi, stable branch ci is not in great shape right now, so rechecking stable branch items is not advisable14:33
*** rajinir has quit IRC14:34
*** rajinir has joined #openstack-ironic14:36
TheJuliaiurygregory: I'm starting to wonder if I'm going to have to be really aggressive on stable/ussuri and just mark a bunch of stuff non-voting at this point14:39
iurygregoryI won't lie, I also had this idea ... =X14:40
TheJuliaI think part of it is the swap change in infra and just not enough reviews time to get everything fixed so as master moved along, more stuff on the older branches had to be tweeked and now we're going to have to start mashing things together on some of the branches :\14:41
* TheJulia crosses her fingers14:53
TheJuliaI've got a more aggressive version of 755100 waiting locally14:53
cdearbornTheJulia: aok - thx for the heads up14:55
*** Qianbiao has quit IRC14:58
TheJuliaiurygregory: w/r/t python-ironic-inspector-client is it just timing out?15:06
TheJuliaor is there something else going on?15:06
iurygregoryso far what I saw was timeout..15:06
iurygregoryand looking at logs it says it didnt finish the inspection15:07
iurygregoryI've put a recheck to get more fresh results since some of the configs were a bit weird when checking with ironic-inspector base job15:07
openstackgerritAija Jaunteva proposed openstack/ironic master: Fix idrac-wsman BIOS step async error handling  https://review.opendev.org/75582115:08
TheJuliayeah15:10
TheJuliaso that vm ultimately comes from ironic-base with 3GB of ram15:11
iurygregorythe jobs in ironic-inspector are fine when running the tests15:16
iurygregoryand we only have 1 VM...15:16
iurygregoryfor some reason it was showing 2 in the client15:16
iurygregoryI remember we updated some configs in ironic-inspector base job, that's why I've put a recheck15:16
TheJuliacould it be the config from another branch?15:17
iurygregorymaybe..15:18
iurygregoryI remember we had to fix some configs in the ironic-inspector to make it work15:19
iurygregorymaybe was during the same time15:19
TheJuliahmmmmm15:23
TheJuliaI think I see why we have failures on stable/train now15:23
*** HardCase has joined #openstack-ironic15:28
HardCaseAnyone here familiar with an error message like "Unable to attach VIF f1ec3cf7-5d1b-41d5-8df0-1c0b7f00a6b7, not enough free physical ports"?15:28
TheJuliagrr15:28
TheJuliai figured it out15:28
TheJuliaHardCase: unfortunately very much so15:28
TheJuliaHardCase: what versions are you running?15:29
TheJuliaiurygregory: so we have a race in the ironic-tepest-plugin where we add a hint for the root device to be /dev/md0 and never remove it after the raid test15:29
iurygregoryouch >.<15:30
iurygregoryand if we don't clean the root_device from the node we will have problem on the next test probably15:32
TheJuliaoh yes15:32
TheJuliavery yes15:32
TheJuliaan earth shattering kaboom yes15:32
iurygregoryyup15:32
TheJuliaiurygregory: can you add a map for idrac to point to redfish and we can backport it ?15:32
TheJuliaonce merged I'll re-release ironic15:33
*** k_mouza has quit IRC15:33
iurygregoryTheJulia, you think that would solve the problems? I was wondering why this only pop-up *today*15:33
*** stendulker has joined #openstack-ironic15:34
TheJuliaiurygregory: popped up downstream today because I bet the build was lagging behind15:35
iurygregoryfunny >.<15:35
TheJuliahmm that race... i've found it before15:37
TheJuliahttps://review.opendev.org/#/c/706960/2/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_cleaning.py15:37
patchbotpatch 706960 - ironic-tempest-plugin - SW RAID: Delete root device hint (MERGED) - 2 patch sets15:37
dkingI'm writing an IPA hardware manager and I'm wanting to do some unit testing. However, I'm having some trouble trying to pip install ironic-python-agent as it seems to require several system packages installed. I'm attempting the installation on CentOS 8. Is there a list somewhere of the expected environment to install IPA?15:39
TheJuliahmm... weird, this raid test shouldn't have failed15:41
HardCaseTheJulia: Sorry, multitasking, running Queens on TripleO15:41
TheJuliaouch15:42
TheJuliayeah15:42
iurygregoryTheJulia, http://paste.openstack.org/show/798654/ like this so we can backport and after we can update idrac to get the info on master to get from the idrac section right?15:42
TheJuliaHardCase: so yeah, you likely have a orphaned VIF, do openstack baremetal node vif list for each of your baremetal nodes not in use and validate that they are not present15:42
TheJuliaiurygregory: exactly15:42
iurygregoryrunning tests locally before pushing15:43
HardCaseTheJulia: Ok thanks, any suggestions for when the theorized offending host might have been deleted?15:43
iurygregorywill open a story to track the work15:43
TheJuliaHardCase: when you say deleted, do you mean removed from ironic entirely?15:43
rpiosoGood morning, ironic!15:44
TheJuliadking: if you install a tool called bindep, that should help you install the system packages15:44
HardCaseTheJulia: If memory serves, when troubleshooting a failed node, an `openstack baremetal node delete [NODE]` was issues and I can no longer see it when running `openstack baremetal node list`15:44
TheJuliadking: we maintain a bindep.txt which should help you out there15:44
TheJuliaoh jeeze15:45
HardCaseit hurts15:45
TheJuliayeah...15:45
TheJuliaso, if you re-add the node and still encounter the same failure there is one of two things going on:15:45
*** priteau has quit IRC15:45
TheJulia1) The heat templates and/or nova are trying to bind >1 VIF to the node. If you only have a single ethernet port enrolled in ironic, then you'll get that failure15:46
HardCaseK, that makes sense15:46
HardCaseI guess the tricky part is finding and removing the orphaned resource?15:47
TheJulia2) Your deployment is thrashing and heat is aborting deployment because it is taking too long and retrying (no, yelling at heat won't fix this), if you have a spare machine on your initial deployment and then scale up, typically you can avoid that sort of case because there will be less waiting around for a machine to be freed15:47
TheJuliayeah15:47
TheJuliabasically, we've got patches in queens nova/ironic/heat to help prevent this issue15:47
TheJuliabut if your running older builds, your still going to hit it15:47
HardCaseYeah I am reasonably confident I am running an older build, there plans to get up to current versions that were subverted by "competing priorities", never happens right?15:48
TheJuliaI'd start with doing an openstack port list, and look for any physical macs that shouldn't be allocated, then you can see if that machine has the VIF present15:48
TheJuliaugh, yeah15:48
TheJuliaI think the last patch to help prevent this issue was only merged to nova on queens a few months back15:48
TheJuliaeverything else merged like >1 year ago15:48
TheJuliabut still, it is the dance between the three services that makes this possible to happen15:49
TheJuliaanyway, any machine not deployed shouldn't have any attached vifs15:49
TheJuliaso you can check undeployed machines enrolled in ironic as a first step15:49
TheJuliaHardCase: hopefully that helps?!?15:50
TheJuliaHardCase: fwiw, in ?train? we forcibly go "nope, you can't keep vifs upon undeployment"15:50
HardCaseSo when you say do `openstack port list` and then "see if that machine has the VIF present", can you explain a bit more how I can see if the VIF is present since I don't have the node UUID anymore?15:50
TheJuliaokay15:51
TheJuliaokay, there are two ways to approach this15:51
HardCasesorry, this is a bit of a new issue for me15:51
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Fix inspection for idrac  https://review.opendev.org/75583515:51
TheJuliaone is `openstack baremetal node vif list <node_uuid>`15:51
TheJuliaagain, undeployed nodes should have nothing there15:51
HardCasek, makes sense15:51
TheJulianow from a neutron standpoint,  you may have an orphaned vif with a duplicate mac address15:52
TheJuliaso if you go through your `openstack port list` output and look for ports that are not actually in use or that tie to a mac address of an undeployed machine, then that port is also orphaned and can result in some errors when vif attachments attempt to occur15:52
TheJuliawhich is also another sign that somewhere along the way, a vif got orphaned and was never cleaned up15:53
TheJuliafor some reason or another15:53
*** priteau has joined #openstack-ironic15:53
HardCasek, would the appropriate action be to `openstack port delete [orphaned port]`?15:53
TheJulianow, doing openstack baremetal node delete <uuid> WILL orphan stuff in neutron if that node was in an active or transient state like provisioning or cleaning15:53
HardCaseah, that's a good one to know15:54
TheJuliato remove the orphaned vif record in ironic, it is `openstack baremetal node vif detach <node_uuid> <vif_uuid>` (you'll also likely want to see if the vif_uuid is actually in use in `openstack port list` just to be on the safe side15:55
TheJuliaand if you have something orphaned in neutron it is `openstack port delete <vif_uuid>`15:55
dkingTheJulia: Thanks! I think that's what I was looking for. I can't seem to find any good information on the tool. Would you happen to have a link to it? I did find bindep.txt, which seems helpful even by itself.15:55
lmcgann_TheJulia is it ok to submit prs as WIP before a spec is merged?15:56
TheJulialmcgann_: it is encouraged!15:56
TheJuliadking: `pip install bindep`15:56
TheJuliaI think15:56
TheJuliait has been ages since I've used it15:57
*** k_mouza has joined #openstack-ironic15:57
dkingThank you! I'll give that a try. I'm a bit skeptical as I didn't see anything in the list that looked like it would obviously fix the error I'm having now, but It's certainly the first step.15:57
openstackgerritLeo McGann proposed openstack/ironic master: Adds Security Interface to Node WIP  https://review.opendev.org/75583615:58
TheJulialmcgann_: fyi, typically we put WIP at the very beginning of the commit subject just so it is more visible :15:59
HardCaseTheJulia: Thanks for this info, it is incredibly appreciated15:59
iurygregoryI'm wondering why they changed things in the test for ilo_* if the test is for Redfish...16:01
iurygregorymaybe I'm wrong.. https://github.com/openstack/ironic/blame/master/ironic/tests/unit/drivers/modules/test_image_utils.py16:01
*** lucasagomes has quit IRC16:02
*** k_mouza has quit IRC16:02
openstackgerritLeo McGann proposed openstack/python-ironicclient master: [WIP] Adds Security Interface to Node and Driver  https://review.opendev.org/75583716:03
*** k_mouza has joined #openstack-ironic16:04
openstackgerritMerged openstack/bifrost stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75562616:06
rpiosostendulker: Thank you for reviewing https://review.opendev.org/#/c/750605/2.16:07
patchbotpatch 750605 - ironic (stable/train) - DRAC: Fix RAID create_config clean step - 2 patch sets16:07
stendulkerrpioso: you are welcome :)16:07
iurygregoryTheJulia, python-ironic-inspector-client GREEN16:08
iurygregory\o/16:08
rpiosostendulker: Please Workflow-1 it. We need more internal eyes on it, either cdearborn or me.16:08
iurygregoryhttps://review.opendev.org/#/c/751286/3 \o/16:08
patchbotpatch 751286 - python-ironic-inspector-client - Fix l-c job - 3 patch sets16:08
stendulkerrpioso: sure.16:08
rpiosostendulker: Thanks, again! :-)16:09
iurygregorywe will need to backport to stable/victoria... since is something for ubuntu-focal16:09
*** k_mouza has quit IRC16:09
dkingMaybe I should take a step back. I have hardware manager code that I would like to test with tox (unittest and flake8) on CentOS 8. However, it fails because both py38 and py36 fail to install ironic-python-agent because <crypt.h> is missing. bindep reports no missing packages if I use https://raw.githubusercontent.com/openstack/ironic-python-agent/master/bindep.txt, though I'm not completely sure that matches what's in the pip16:12
dking repos. Are there any suggestions on what I should be doing differently?16:12
dkingI'm actually testing on an IPA ramdisk, so I know that it's possible to install everything, and in fact, it works using the venv of the installed IPA, but obviously, I'm going to want to eventually have the tests run in CI.16:14
*** ricolin_ has quit IRC16:14
openstackgerritgugug proposed openstack/tenks master: Add py38 package metadata  https://review.opendev.org/75583916:16
openstackgerritzhoulinhui proposed openstack/tenks master: Add py38 package metadata  https://review.opendev.org/75584116:17
*** k_mouza has joined #openstack-ironic16:17
*** k_mouza has quit IRC16:22
*** k_mouza has joined #openstack-ironic16:23
*** QianbiaoNG has quit IRC16:27
*** k_mouza has quit IRC16:27
iurygregoryTheJulia, the fix worked \o/16:31
*** dougsz has quit IRC16:32
*** k_mouza has joined #openstack-ironic16:33
openstackgerritMerged openstack/ironic stable/ussuri: CI: Additional minor job changes to improve CI reliability  https://review.opendev.org/75510016:36
TheJulia\o/ ^^^16:37
*** k_mouza has quit IRC16:38
openstackgerritMerged openstack/ironic stable/train: Use correct error handler instead of calling process_event('fail')  https://review.opendev.org/75507116:41
TheJuliaiurygregory: approved16:42
*** ociuhandu_ has joined #openstack-ironic16:43
TheJuliawe'll need to backport it to stable/victoria and cut a new release16:44
*** k_mouza has joined #openstack-ironic16:45
iurygregoryyeah16:46
*** ociuhandu has quit IRC16:47
iurygregoryand I will work to update master after to use the information from idrac section16:47
iurygregoryI will push the backport after it merges16:48
*** ociuhandu_ has quit IRC16:48
TheJuliaiurygregory: thanks, much appreciated since then I can just +2+A it16:50
*** k_mouza has quit IRC16:50
*** stendulker has quit IRC16:50
iurygregorysure16:50
iurygregorytime to prepare the dinner now =)16:51
*** k_mouza has joined #openstack-ironic16:56
openstackgerritJulia Kreger proposed openstack/ironic stable/ussuri: CI: Making some of the older jobs more specific  https://review.opendev.org/75584916:56
openstackgerritMerged openstack/tenks master: Add py38 package metadata  https://review.opendev.org/75584116:59
*** k_mouza has quit IRC17:01
openstackgerritMerged openstack/python-ironic-inspector-client master: Fix l-c job  https://review.opendev.org/75128617:08
*** k_mouza has joined #openstack-ironic17:08
arne_wiebalckIs it possible that udev interferes during md RAID creation? I see that the udev rules are disabled for anaconda, maybe we should also make sure they do not do anything when the IPA creates md devices?17:11
* arne_wiebalck is still trying to understand why md RAID do come up in degraded mode sometimes17:11
arne_wiebalck*md RAIDs17:11
*** k_mouza has quit IRC17:13
*** derekh has quit IRC17:13
TheJuliaarne_wiebalck: hmm17:15
TheJuliaI _wouldn't_ think so, but I guess anything is possible17:15
TheJuliais that degraded after instance boot?17:15
TheJuliaiurygregory: looks like one of the jobs failed on your patch17:23
arne_wiebalckTheJulia: no, it is degraded *right* after creation17:23
arne_wiebalckTheJulia: http://paste.openstack.org/show/798657/17:24
arne_wiebalckTheJulia: I know what you think: this is a disk failure. But I can add the disk or do the creation manually and it works just fine.17:25
*** k_mouza has joined #openstack-ironic17:25
TheJuliaright after creation makes me feel like https://giphy.com/gifs/wdr-what-weird-xThtayl8IihSBoS5jy17:25
arne_wiebalckTheJulia: yep :)17:26
arne_wiebalckTheJulia: I am trying to understand this since a few days now.17:26
TheJuliaYeah, that does seem like maybe something happened there and could have been udev bouncing things around if it is actually reaching into the kernel enough that the device io becomes blocked momentarily17:27
arne_wiebalckTheJulia: I suspected the abrupt shutdown after creation (end of cleaning) to be the culprit, but it seems now the device is degraded *before*.17:27
openstackgerritVerification of a change to openstack/metalsmith failed: Handle out-of-tree openstack module_utils  https://review.opendev.org/75387017:27
arne_wiebalckTheJulia: Next things are to switch off udev right before creation, trying to re-add the device if the RAID is degraded and to check if this is happening always on the exact same physical nodes.17:28
arne_wiebalckTheJulia: Re-adding seems to be what people do.17:29
arne_wiebalckTheJulia: https://access.redhat.com/solutions/3508301 is similar17:29
*** k_mouza has quit IRC17:30
arne_wiebalckTheJulia: The reason I am thinking udev is that we just created the partitions and then the md superblock ... so udev might jump right trying to "help" us ...17:30
iurygregoryTheJulia, looking17:30
TheJuliaarne_wiebalck: that all seems reasonable I guess17:32
TheJuliajust weird :(17:32
iurygregory test_create_allocation_with_traits .-.17:32
iurygregory no suitable nodes have the requested traits CUSTOM_MEOW.17:32
TheJuliaunrelated race then17:34
iurygregoryseems to be17:34
iurygregoryI'm checking the zuul runs for this job17:34
*** k_mouza has joined #openstack-ironic17:35
iurygregoryhttps://zuul.openstack.org/builds?job_name=ironic-tempest-functional-python3&branch=master&branch=stable%2Fvictoria there are some random failures17:35
*** bnemec is now known as beekneemech17:37
openstackgerritIury Gregory Melo Ferreira proposed openstack/python-ironic-inspector-client stable/victoria: Fix l-c job  https://review.opendev.org/75585317:38
iurygregorybe back in a few hours (need to scan some documents to renew my resident permit)...17:39
*** k_mouza has quit IRC17:39
*** belmoreira has quit IRC17:47
*** jamesdenton has quit IRC18:34
*** coreycb has joined #openstack-ironic18:41
coreycbhi, does anyone happen to know where the source for python-xclarityclient lives?18:42
*** k_mouza has joined #openstack-ironic18:45
coreycbit's on pypi https://pypi.org/project/python-xclarityclient but that's all I can find. I'd like to package it up but not sure how/where we'd send patches if we needed to. and unfortunately the author's email from pypi is bouncing.18:46
TheJuliacoreycb: Great question.. hmm18:47
clarkbhttps://github.com/FinixLei/xclarity-client possibly that?18:47
TheJuliahmm18:48
TheJuliawhere is jungleboyj when we need him18:49
*** k_mouza has quit IRC18:49
*** jungleboyj has joined #openstack-ironic18:51
jungleboyjTheJulia: Sorry, didn't realize I wasn't on the channel.18:51
jungleboyjNot sure when I got booted.18:51
TheJuliaoh wow18:51
coreycbclarkb: TheJulia: that might be the right repo. thank you. I'll compare it vs what's on pypi.18:51
TheJuliaso It looks like the original author doesn't work for lenovo anymore :\18:51
jungleboyjOh yeah.  Finix.  :-(18:52
TheJuliaI wonder if we should just import it into the x/ namespace18:52
TheJulia:(18:52
clarkbthere is also a pylxca18:52
TheJuliaI could have sworn there WAS a distinct repo at one point in time18:52
clarkbthough that might all be cli stuff?18:53
jungleboyjclarkb:  pylxca is being maintained, at least internally because we use that for the project I am leading.18:53
jungleboyjpylxca is for interacting with our xClarity Administrator.18:53
jungleboyjManagement platform that interacts with the system BMCs.18:54
*** jamesdenton has joined #openstack-ironic18:56
coreycbhttps://pypi.org/project/python-xclarityclient/#files doesn't match https://github.com/FinixLei/xclarity-client unfortunately. it's the same code base but one is older than the other.18:57
TheJuliaI'm having this really worrisome feeling that the actual python-xclarityclient repo was deleted :\18:57
TheJuliawhich means at worse maybe we salvage from what is in pypi18:57
openstackgerritMerged openstack/ironic master: Add Python3 wallaby unit tests  https://review.opendev.org/75563318:58
openstackgerritMerged openstack/ironic-python-agent master: Add Python3 wallaby unit tests  https://review.opendev.org/75562418:58
coreycbuh oh.. yes that's a good idea18:58
coreycbRushil Chugh added the change to driver-requirements.txt so maybe he would know18:58
TheJuliaI think Rushil was the initial author...18:59
jungleboyjcoreycb:  Rushil is gone too.  I can text him to follow up though.18:59
coreycbjungleboyj: ok thanks :)19:00
*** k_mouza has joined #openstack-ironic19:00
*** jtomasek has quit IRC19:02
TheJuliaso I guess the plan might be see what jungleboyj gets back from his cohorts in over in Shanghai and if he doesn't hear back maybe reconsitute it somehow from pypi early next week?19:04
*** yolanda has quit IRC19:04
jungleboyjTheJulia: ++19:04
*** yolanda has joined #openstack-ironic19:04
*** k_mouza has quit IRC19:04
*** lmcgann_ has quit IRC19:05
jungleboyjI just pinged Rushil.  Though, he had called me a couple of months ago and I forgot to get back to him.  So we will see how that goes.  :-(19:05
TheJulialife gets busy, he seemed fairly laid back19:05
jungleboyjI can send a note to SH but probably won't hear from them for a while as the country is on vacation until 10/9.19:05
* TheJulia likes this idea19:06
jungleboyjHe was pretty laid back.19:06
coreycbjungleboyj: TheJulia: thank you, sounds good to me. very much appreciate the help.19:09
jungleboyjNo problem!19:09
TheJuliaIt might be worth while starting on some hacking on a repo and we can always import or those changes can get pulled in19:09
TheJuliajust thinking outloud since this doesn't seem like it is going to be quick to have full clarity19:10
*** k_mouza has joined #openstack-ironic19:17
*** priteau has quit IRC19:17
*** k_mouza has quit IRC19:22
iurygregoryTheJulia, we probably also want to backport https://review.opendev.org/#/c/751289/ for stable/victoria right?19:42
patchbotpatch 751289 - python-ironicclient - Fix l-c job (MERGED) - 2 patch sets19:42
TheJuliaiurygregory: hmm yeah19:54
openstackgerritJulia Kreger proposed openstack/python-ironicclient stable/victoria: Fix l-c job  https://review.opendev.org/75587019:54
*** tosky has quit IRC19:57
*** tosky has joined #openstack-ironic19:58
*** k_mouza has joined #openstack-ironic19:59
iurygregoryI noticed we need otherwise https://review.opendev.org/#/c/750720/ and https://review.opendev.org/#/c/750719/ won't merge20:00
patchbotpatch 750720 - python-ironicclient (stable/victoria) - Update TOX_CONSTRAINTS_FILE for stable/victoria - 1 patch set20:00
patchbotpatch 750719 - python-ironicclient (stable/victoria) - Update .gitreview for stable/victoria - 1 patch set20:00
*** k_mouza has quit IRC20:03
openstackgerritMerged openstack/bifrost stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/75562720:04
rpiosoTheJulia: I just noticed you Code-Review-1 https://review.opendev.org/#/c/750605/ just before I posted my feedback The only prospective issue  I see is with the commit message. I believe the cherry pick encountered a merge conflict in the unit test file. Would you prefer I push a new patch set with a commit message update, let Ameya respond, or ...?20:05
patchbotpatch 750605 - ironic (stable/train) - DRAC: Fix RAID create_config clean step - 2 patch sets20:05
*** k_mouza has joined #openstack-ironic20:07
rpiosoTheJulia: I shared my findings with cdearborn, who has more recent familiarity with idrac-wsman RAID.20:07
TheJuliarpioso: I'd let Ameya respond, I put in a -2 so we don't prematurely just push it through without y'all being happy with it first20:08
TheJuliaiurygregory: joy! well I clicked the button so I guess we shall see :)20:08
iurygregoryyeah20:08
rpiosoTheJulia: Works for me. I wanted to offer options, in case you're prepping to cut a new release from stable/train.20:09
openstackgerritJulia Kreger proposed openstack/metalsmith master: Replace the Centos7 image job with a Centos8 UEFI based job  https://review.opendev.org/74831520:10
openstackgerritMerged openstack/python-ironic-inspector-client master: Add Python3 wallaby unit tests  https://review.opendev.org/75112920:11
TheJuliarpioso: not today, I've mainly been trying to get CI sorted across various branches today20:11
*** k_mouza has quit IRC20:11
TheJuliarpioso: fwiw, you folks may want to setup a vmedia based job in your 3rd party CI. Some re-factoring broke it and as such I'll be releasing a new victoria release... asap https://review.opendev.org/#/c/755835/120:14
patchbotpatch 755835 - ironic - Fix inspection for idrac - 1 patch set20:14
rpiosoTheJulia: We've got plans to bolster our 3d party CI coverage of idrac HW type features, including vmedia. I believe we need to get to Zuul v3 first, right?20:19
*** tosky has quit IRC20:19
rpiosoTheJulia: I do not understand the change above ^^^. Does the broken test use the idrac HW type? Does ironic's CI test vmedia against Dell EMC hardware?20:21
TheJuliarpioso: you guys don't need to get to zuulv3 first ,but basically some image generating code refactoring broke on the hardware type since we can't actually exercise against a idrac upstream20:23
TheJuliaso while we test the code path upstream, a slight nuance was added in refactoring that broke idrac hw type when used with vmedia20:23
rpiosoTheJulia: Got it. It's late Friday here :-)20:23
TheJuliano worries20:23
TheJuliaIt is also friday in general20:23
rpiosoTheJulia: Thank you!20:23
rpioso++++++20:23
TheJuliaI actually need to go to the pet store, mr corgi overlord has decided his chew target is his cone of shame20:24
TheJuliaiurygregory: given your change just entered the gate like a half an hour ago with a recheck, By all means go ahead and call it a night. I can backport and request the release tag over the weekend if needed. Itis not like it will acctually be approved until monday20:25
iurygregoryTheJulia, ack o/20:30
iurygregorybye everyone, see you on monday!20:30
TheJuliahave a good weekend20:32
rpiosoiurygregory: Enjoy!20:47
*** bfournie has quit IRC20:59
TheJuliawow, six patches stacked for the gate21:04
* TheJulia crosses fingers21:04
openstackgerritVerification of a change to openstack/ironic failed: CI: Remove the build check for pre-build ramdisks only  https://review.opendev.org/75346721:06
NobodyCamHappy Friday Ironic folks21:49
*** tzumainn has quit IRC22:13
openstackgerritMerged openstack/metalsmith master: Handle out-of-tree openstack module_utils  https://review.opendev.org/75387022:13
*** rloo has quit IRC22:16
openstackgerritRichard G. Pioso proposed openstack/ironic stable/train: DRAC: Fix a failure to create virtual disk bug  https://review.opendev.org/75589522:17
openstackgerritMerged openstack/ironic stable/ussuri: Handle an older agent with agent_token  https://review.opendev.org/74958522:25
openstackgerritMerged openstack/ironic stable/train: Fix a race condition in the hash ring code  https://review.opendev.org/75526822:25
*** k_mouza has joined #openstack-ironic22:28
*** k_mouza has quit IRC22:32
openstackgerritMerged openstack/ironic stable/ussuri: Do not assume that prepare_image is the last command to run  https://review.opendev.org/74991322:52
openstackgerritMerged openstack/python-ironicclient master: Remove install unnecessary packages  https://review.opendev.org/75392222:52
openstackgerritMerged openstack/ironic stable/ussuri: Use correct error handler instead of calling process_event('fail')  https://review.opendev.org/75501722:52
openstackgerritVerification of a change to openstack/ironic failed: Fix inspection for idrac  https://review.opendev.org/75583522:52
openstackgerritMerged openstack/python-ironicclient stable/victoria: Fix l-c job  https://review.opendev.org/75587022:52
openstackgerritMerged openstack/ironic stable/ussuri: Fix a race condition in the hash ring code  https://review.opendev.org/75526723:19
*** yolanda has quit IRC23:23
*** yolanda has joined #openstack-ironic23:24
openstackgerritJulia Kreger proposed openstack/metalsmith stable/victoria: Handle out-of-tree openstack module_utils  https://review.opendev.org/75590223:25
*** ricolin_ has joined #openstack-ironic23:49
« Thursday, 2020-10-01 Index Saturday, 2020-10-03 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!