Thursday, 2020-07-09

*** rh-jelabarre has quit IRC00:43
*** rh-jelabarre has joined #openstack-ironic00:44
*** rh-jelabarre has quit IRC00:48
*** vkmc_ is now known as vkmc00:54
*** prometheanfire has left #openstack-ironic01:04
*** Goneri has quit IRC01:18
openstackgerritSteve Baker proposed openstack/ironic master: Add json and param parsing to args  https://review.opendev.org/70448901:26
openstackgerritSteve Baker proposed openstack/ironic master: Add wsme core types, remove WSME  https://review.opendev.org/70449001:26
openstackgerritSteve Baker proposed openstack/ironic master: Convert root path / to use plain JSON  https://review.opendev.org/74014201:26
*** gyee has quit IRC01:38
*** ricolin has joined #openstack-ironic02:02
*** mkrai has joined #openstack-ironic02:18
openstackgerritliuyuanfeng proposed openstack/ironic stable/ussuri: error word confiuration to configuration  https://review.opendev.org/74014802:26
*** mkrai has quit IRC02:49
*** tzumainn has quit IRC02:50
*** mkrai has joined #openstack-ironic02:56
*** Lucas_Gray has quit IRC02:56
*** rcernin has quit IRC03:00
*** diurnalist has quit IRC03:01
*** rcernin has joined #openstack-ironic03:03
*** yaawang has quit IRC03:10
*** yaawang has joined #openstack-ironic03:10
*** Lucas_Gray has joined #openstack-ironic03:20
*** Lucas_Gray has quit IRC03:32
*** diurnalist has joined #openstack-ironic03:33
*** diurnalist has quit IRC03:38
*** diurnalist has joined #openstack-ironic03:49
*** diurnalist has quit IRC03:54
*** ociuhandu has joined #openstack-ironic03:56
*** ociuhandu has quit IRC04:01
*** diurnalist has joined #openstack-ironic04:04
*** diurnalist has quit IRC04:10
*** diurnalist has joined #openstack-ironic04:29
*** abdysn has joined #openstack-ironic05:02
*** zzzeek has quit IRC05:19
*** zzzeek has joined #openstack-ironic05:23
*** diurnalist has quit IRC05:32
*** mkrai has quit IRC05:50
arne_wiebalckGood morning, ironic!06:04
*** yaawang has quit IRC06:32
*** yaawang has joined #openstack-ironic06:33
iurygregorygood morning arne_wiebalck and Ironic06:47
*** ildikov has quit IRC06:55
arne_wiebalckhey iurygregory o/06:56
*** ildikov has joined #openstack-ironic06:58
openstackgerrituchenily proposed openstack/networking-generic-switch master: change segment id type to int  https://review.opendev.org/74017107:03
rpittaugood morning ironic! o/07:03
iurygregorymorning rpittau o/07:03
rpittauhey iurygregory :)07:04
iurygregoryjust FYI don't try recheck yet on the patches =)07:08
rpittauwhat's broken?07:08
iurygregorywe need https://review.opendev.org/#/c/714453/ fixing it07:08
patchbotpatch 714453 - ironic - add tempest boot_mode config - 21 patch sets07:08
rpittaufun fact: I used to start a lot of my past working days with that question07:08
iurygregoryironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa07:09
iurygregoryand ironic-tempest-ipa-partition-pxe_ipmitool07:09
iurygregorythe fix is on my patch since it was introduced by my change on the tempest plugin that got merged07:09
rpittauwho breaks it, fixes it :)07:10
rpittauanother common sentence used in the past07:10
iurygregoryhehehe07:10
rpittaujust kidding of course, some memories of stuff... broken... badly :)07:11
iurygregorywell I knew that was going to break, but probably people didn't see my comment =X07:11
rpittaureading is overrated07:11
iurygregoryyeah07:12
* iurygregory grabs more coffee07:12
*** dougsz has joined #openstack-ironic07:20
arne_wiebalckgood morning, rpittau o/07:34
rpittauhey arne_wiebalck :)07:34
* arne_wiebalck missed irc most of yesterday07:34
iurygregorythis was a funny error from our multinode job https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_90b/714453/21/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/90bed6c/testr_results.html07:35
rpittauheh missed a lot of logs too :/07:35
iurygregorynever saw before =D07:35
rpittauwell I just got 5300 tests failed on 6383, something's not right07:37
iurygregorywut?!07:38
rpittaulower-constraints, Python 3.8, nightmares ahead07:40
openstackgerritliuyuanfeng proposed openstack/ironic master: error wold confiuration to configuration  https://review.opendev.org/74017407:40
iurygregoryrpittau, ouch =( if you need help let me know07:46
iurygregoryonly me or gerrit is a bit slow today?07:49
openstackgerritliuyuanfeng proposed openstack/ironic master: error word confiuration to configuration  https://review.opendev.org/74017507:53
openstackgerritliuyuanfeng proposed openstack/ironic master: error word confiuration to configuration  https://review.opendev.org/74017507:54
*** sshnaidm has joined #openstack-ironic07:55
*** Lucas_Gray has joined #openstack-ironic08:01
rpittauoh thanks iurygregory, I'm back to Failed: 0 :)08:02
*** gmann has quit IRC08:03
*** erbarr has quit IRC08:03
iurygregoryrpittau, yay!08:04
*** vkmc has quit IRC08:04
*** erbarr has joined #openstack-ironic08:04
* iurygregory resets the router brb08:04
*** v1k0d3n has quit IRC08:05
*** gmann has joined #openstack-ironic08:06
openstackgerritliuyuanfeng proposed openstack/ironic stable/ussuri: fix error word confiuration to configuration in docs  https://review.opendev.org/74014808:06
*** v1k0d3n has joined #openstack-ironic08:07
*** vkmc has joined #openstack-ironic08:07
maelkHi! dtantsur|afk I was trying to test your patch https://review.opendev.org/#/c/739964. However, the fixes you added in Metal3 make it so that IPA is behaving properly now, so there is no lookup happening anymore. So we do not need urgently this fix anymore. anyways it would be good to fix it still so I'll try again without the metal3 fix and keep08:08
maelkyou updated08:08
patchbotpatch 739964 - ironic - [WIP] Wipe agent token during reboot or power off - 1 patch set08:08
openstackgerritliuyuanfeng proposed openstack/ironic master: fix error word confiuration to configuration in docs  https://review.opendev.org/74017508:09
*** iurygregory has quit IRC08:10
*** lucasagomes has joined #openstack-ironic08:15
*** iurygregory has joined #openstack-ironic08:24
*** uzumaki has joined #openstack-ironic08:34
maelkalso, dtantsur|afk if we are using the fast track, and we have a powerloss or anything that causes the server to reboot independently of Ironic's control, IPA will need a new token, but ironic won't give it a new one because it did not wipe the old token at any point. is it right ?08:34
*** derekh has joined #openstack-ironic08:36
iurygregoryyay CI gave +108:40
iurygregorytks zuul, I will give you a cookie XD08:40
*** benj_ has quit IRC08:41
*** k_mouza has joined #openstack-ironic08:44
uzumakiiurygregory, I'd like to present myself as an ambassador to mr zuul to recive the property, thusly described as 'the cookie'08:48
iurygregoryuzumaki, hehehehehe08:48
uzumakisup iurygregory o/ xD how's it going?08:49
iurygregoryuzumaki, doing good, and you?08:49
iurygregoryhere is the cookie https://media.giphy.com/media/7GYHmjk6vlqY8/giphy.gif08:49
uzumakiI'm well!08:49
iurygregoryplease give to zuul08:49
uzumakiiurygregory, already on it! love the cookie, looks delicious xD08:50
*** benj_ has joined #openstack-ironic08:58
openstackgerritHarald Jensås proposed openstack/ironic-inspector master: Remove non-inclusive language in pxe filter  https://review.opendev.org/74019409:01
openstackgerritHarald Jensås proposed openstack/ironic-inspector master: Remove non-inclusive language in pxe filter  https://review.opendev.org/74019409:03
iurygregoryhjensas, ++09:10
*** uzumaki has quit IRC09:20
*** uzumaki has joined #openstack-ironic09:20
*** rcernin has quit IRC09:23
*** dtantsur|afk is now known as dtantsur09:29
dtantsurmorning ironic09:34
dtantsurmgoddard: hey, https://review.opendev.org/#/c/737695/ got 2x +2 and is close to getting approve, PTAL09:34
patchbotpatch 737695 - ironic - agent_base: support inserting in-band deploy steps - 5 patch sets09:34
mgoddarddtantsur: looking09:39
iurygregorymorning dtantsur09:40
dtantsuriurygregory: I'm updating the incorrect upper-constraints patches09:41
iurygregorydtantsur, ack09:42
iurygregorydo they have victoria available already?09:43
dtantsuriurygregory: yep, it's a redirect to master09:43
iurygregorydtantsur, oh awesome! otherwise we would need to update everything in the end of the cycle =X09:44
*** k_mouza has quit IRC09:47
dtantsurmaelk: re fast-track: yes, that's why we need my patch still09:52
ftarasenkoHi team!09:56
*** k_mouza has joined #openstack-ironic09:56
maelk@dtantsur, would ironic find out on its own that it needs to wipe the token ? there is nothing telling it to power off the server09:57
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Do not clone repos with source_install==false  https://review.opendev.org/73993009:57
dtantsurhi ftarasenko09:57
dtantsurmaelk: if you tell ironic to power off, it will wipe the token09:57
maelkyes, but what if the user powers it off without ironic09:58
ftarasenkoarne_wiebalck: iurygregory: dtantsur: have question regarding EFI bootloader and software RAID. Why do we put EFI bootloader only on one partition outside of mdraid?09:58
dtantsurin case of power loss.. I'm not sure09:58
dtantsurftarasenko: I thought we put it on a partition on each holder disk.. arne_wiebalck remembers it better though09:58
maelkmaybe the token would need to be wiped whenever ironic powers on a node also09:58
maelkor when ironic polls the power status and the node is off09:59
dtantsurmmm, probably the former. I need to think a bit.09:59
maelkof course that does not cover an unexpected reboot undetected by ironic09:59
ftarasenkodtantsur: I might be wrong. Checked again and see both partitions with blkid.10:00
ftarasenkoDo I correctly remember that UEFI search for EFI partition on each mounted drive?10:00
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Do not clone repos with source_install==false  https://review.opendev.org/73993010:03
openstackgerritHarald Jensås proposed openstack/ironic-inspector master: Remove non-inclusive language in pxe filter  https://review.opendev.org/74019410:03
openstackgerritHarald Jensås proposed openstack/ironic-inspector master: Remove non-inclusive language in pxe filter  https://review.opendev.org/74019410:06
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Do not clone repos with source_install==false  https://review.opendev.org/73993010:06
openstackgerritMerged openstack/ironic master: add tempest boot_mode config  https://review.opendev.org/71445310:07
iurygregoryyay CI is back10:09
iurygregoryrecheck all the patches!10:09
iurygregoryftarasenko, it will look for the efi partition on the device https://github.com/openstack/ironic-python-agent/blob/9d9a6bce5c246d18d50dc405265e865fb77142eb/ironic_python_agent/utils.py#L61310:11
*** hjensas is now known as hjensas|afk10:12
ftarasenkoiurygregory: Thank you. Still think that we need to set boot order with efibootmgr, but haven't tested yet10:13
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Get rid of ci_testing variable in roles  https://review.opendev.org/74021710:15
*** ricolin has quit IRC10:15
*** ociuhandu has joined #openstack-ironic10:16
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic stable/ussuri: add tempest boot_mode config  https://review.opendev.org/74022110:18
arne_wiebalckftarasenko: I think we put it on partitions on all drives.10:21
maelkdtantsur I verified your patch https://review.opendev.org/#/c/739964 . It fixes the issue!10:23
patchbotpatch 739964 - ironic - [WIP] Wipe agent token during reboot or power off - 1 patch set10:23
openstackgerritRiccardo Pittau proposed openstack/ironic master: migrate testing to ubuntu focal  https://review.opendev.org/73944810:23
dtantsurmaelk: great!10:23
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Add git_url_root to override root for all repositories at once  https://review.opendev.org/74022410:24
openstackgerritRiccardo Pittau proposed openstack/ironic master: migrate testing to ubuntu focal  https://review.opendev.org/73944810:24
*** janders has quit IRC10:25
*** uzumaki has quit IRC10:26
*** uzumaki has joined #openstack-ironic10:26
openstackgerritMerged openstack/ironic master: fix error word confiuration to configuration in docs  https://review.opendev.org/74017510:28
*** uzumaki has quit IRC10:33
*** uzumaki has joined #openstack-ironic10:33
*** uzumaki has quit IRC10:38
*** tkajinam has quit IRC11:02
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Account for bugfix branches in bifrost-cli  https://review.opendev.org/74024411:13
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Support extra-vars in bifrost-cli  https://review.opendev.org/74024611:22
*** jungleboyj_ is now known as jungleboyj11:46
openstackgerritIury Gregory Melo Ferreira proposed openstack/ironic master: Remove non-inclusive language  https://review.opendev.org/74025111:49
*** rh-jelabarre has joined #openstack-ironic11:57
dtantsurhmm, are we broken again? https://zuul.openstack.org/builds?job_name=ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa12:20
dtantsuryeah, same problem, I guess12:22
iurygregorydtantsur, it's fixed12:24
iurygregory=)12:24
dtantsuriurygregory: well, it still fails12:24
iurygregorystable/ussuri is probably affected12:24
iurygregorydtantsur, I didn't get your comment in https://review.opendev.org/#/c/740221/12:26
patchbotpatch 740221 - ironic (stable/ussuri) - add tempest boot_mode config - 1 patch set12:26
iurygregoryO.o12:26
dtantsuriurygregory: that job also needs to be updated (although it's not running, so let's update it separately)12:26
dtantsurokay, I'll do another round of rechecks to be sure12:27
openstackgerritDmitry Tantsur proposed openstack/bifrost master: Support extra-vars in bifrost-cli  https://review.opendev.org/74024612:28
iurygregoryoh we have  ironic-tempest-ipa-wholedisk-bios-agent_ipmitool but we don't run that job12:29
iurygregorywe run  ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa12:29
rpittauuntil ussuri it was still tinyipa12:29
iurygregory# NOTE(rpittau): converted job but not running for now as there12:31
iurygregory# could be an issue with the lookup in ironic-python-agent12:31
iurygregoryit's the dib version of it12:31
iurygregorybut we don't run because you are still testing right rpittau ?12:32
rpittauyeah12:34
rpittauthere's a little issue with networkmanager https://review.opendev.org/73475912:34
patchbotpatch 734759 - ironic - Convert ironic-tempest-ipa-wholedisk-bios-agent_ip... - 10 patch sets12:34
rpittauI confused jobs :D12:35
rpittauthat's the only one still with tinyipa12:35
iurygregoryyeah!12:35
iurygregoryXD12:36
iurygregoryjust be aware now that you may need to increase the amount of VMS12:36
iurygregory=)12:36
openstackgerritMerged openstack/ironic master: agent_base: support inserting in-band deploy steps  https://review.opendev.org/73769512:42
dtantsur\o/12:42
dtantsurnow, could I lure someone into reviewing the remaining two patches? https://review.opendev.org/#/q/topic:story/2006963+status:open12:42
openstackgerritRiccardo Pittau proposed openstack/ironic master: Convert ironic-tempest-ipa-wholedisk-bios-agent_ipmitool to dib  https://review.opendev.org/73475912:47
openstackgerritMerged openstack/ironic stable/ussuri: fix error word confiuration to configuration in docs  https://review.opendev.org/74014812:54
*** hjensas|afk is now known as hjensas13:09
*** ricolin has joined #openstack-ironic13:10
TheJuliagood morning13:10
rpittaugood morning TheJulia :)13:17
TheJuliahopefully everyone had a wonderful evening13:18
TheJuliadtantsur: I'll happilly review th elast... after I've been coffeed13:18
iurygregorygood morning TheJulia =)13:19
*** Goneri has joined #openstack-ironic13:20
dtantsurmorning TheJulia13:22
iurygregoryshould we only run test_baremetal_multitenancy+test_schedule_to_all_nodes on ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode ?13:24
TheJuliadoes anyone remember what the ansible behavior change with using | bool was that was coming in the next major release?13:27
TheJuliaiurygregory: I guess likely just that job because we're pushing nova scheduling to the maxiumum during a release, across an upgrade, I'm not sure there is value there13:28
TheJuliadtantsur: you've been around for a long time as well, any thoughts on ^^^13:28
dtantsuron which of the two questions?13:29
iurygregoryupgrade? o.o13:29
dtantsurre ansible: they'll stop treating strings as booleans eventually13:29
dtantsurso {{ variable | bool }} is safer13:29
dtantsuriurygregory: re tests, we probably should (although the ironic-tempest-plugin CI uses this job to verify the regular scenarios)13:30
iurygregorydtantsur, aha13:31
iurygregoryok now I see the reason =)13:31
dtantsurTheJulia: the good news are: we've done our intermediate releases and the world did not blow up :)13:33
dtantsurwe need branch mapping for devstack CI jobs on the new branches though (I'll look into it after some tea)13:34
TheJuliadtantsur: I remember some awesome errors recently that basically said they were going to remove a form of use, so maybe it was just uncast strings. I really don't remember13:37
TheJuliadtantsur: I was wondering about the schedule all nodes13:38
TheJuliadtantsur: \o/13:38
TheJulia++13:38
TheJuliaI have a meeting starting in ~20 minutes13:38
* TheJulia sips coffee and still tries to wake up13:38
*** tzumainn has joined #openstack-ironic13:48
dtantsurschedule all nodes... dunno. there is value in it, not sure how much.13:50
TheJuliaIt is a test that seems... easy to fail13:50
TheJuliaand by that I mean we've got a lot of moving parts going on there13:51
TheJuliaand we're trying to artificially compress everything into the CI window.13:51
TheJuliaI'd prefer minimalistic and only on the larger multinode job because that will also be less of a grenade headache in my mind13:51
iurygregorysince we started moving to dib (we only have 1 VM)13:53
dtantsurI *think* multinode jobs still use tinyIPA, no?13:53
iurygregorydtantsur, correct13:54
openstackgerritMerged openstack/ironic-inspector master: Remove non-inclusive language in pxe filter  https://review.opendev.org/74019413:54
openstackgerritMerged openstack/bifrost master: Account for bugfix branches in bifrost-cli  https://review.opendev.org/74024414:02
*** diurnalist has joined #openstack-ironic14:05
openstackgerritDmitry Tantsur proposed openstack/ironic-inspector master: Do not try to set local_gb to -1 when the matched root device size is 0  https://review.opendev.org/74030414:16
dtantsurTheJulia: 1st part of what we've just discussed downstream ^^14:16
*** abdysn has quit IRC14:24
openstackgerritMerged openstack/ironic master: Decompose the core deploy step of the direct deploy  https://review.opendev.org/69830914:30
openstackgerritDmitry Tantsur proposed openstack/ironic-python-agent master: Ignore devices with size 0 when collecting inventory  https://review.opendev.org/74030914:32
dtantsurTheJulia: 2nd part ^^14:32
dtantsurI should file a story for that, shouldn't I?14:36
openstackgerritDmitry Tantsur proposed openstack/ironic-inspector master: Do not try to set local_gb to -1 when the matched root device size is 0  https://review.opendev.org/74030414:40
openstackgerritDmitry Tantsur proposed openstack/ironic-python-agent master: Ignore devices with size 0 when collecting inventory  https://review.opendev.org/74030914:41
*** Lucas_Gray has quit IRC14:55
*** mnasiadka has joined #openstack-ironic14:58
*** Lucas_Gray has joined #openstack-ironic15:11
dtantsurarne_wiebalck: so, if RAID is broken, what exactly shows with size zero? and what are we trying to do about it?15:22
arne_wiebalckdtantsur: from what I remember, if a RAID is broken (not degraded), e.g. a RAID-0 with one disk missing, and the instance is being cleaned, the corresponding RAID device is reported as 0 size and was not cleaned properly (as it was not reported); this then broke RAID creation afterwards15:27
dtantsurarne_wiebalck: does cleaning still work for such devices?15:28
arne_wiebalckdtantsur: with cleaning I am referring to delete_configuration (our h/w manager does this on auto cleaning, hence the not very precise wording)15:30
arne_wiebalckdtantsur: the configuration on the other leg of the RAID needs to be removed15:30
arne_wiebalckdtantsur: but the other leg cannot be found since the md device is not reported15:30
dtantsurarne_wiebalck: mmm, so if I add a flag that only includes empty devices for delete_configuration, it will be fine?15:30
* dtantsur checks the code15:31
*** k_mouza has quit IRC15:31
arne_wiebalckdtantsur: I would think so, yes15:31
arne_wiebalckdtantsur: I just remember this one as it took me a while to figure out :)15:32
dtantsurarne_wiebalck: it doesn't seem like delete_configuration uses list_all_block_devices..15:32
arne_wiebalckdtantsur: that broken md devices have size 015:32
dtantsuroh, we can allow empty md devices but skip all other?15:33
arne_wiebalckdtantsur:  this one? https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/hardware.py#L178215:36
arne_wiebalckdtantsur: yep, that should also work15:36
*** k_mouza has joined #openstack-ironic15:37
dtantsuroh, how did I manage to miss it?15:43
arne_wiebalckI missed it on the first pass, too :-D15:46
*** gyee has joined #openstack-ironic15:50
openstackgerritMerged openstack/ironic master: Stop using md5 for __repr__ of objects  https://review.opendev.org/73973515:58
openstackgerritDmitry Tantsur proposed openstack/ironic stable/ussuri: Stop using md5 for __repr__ of objects  https://review.opendev.org/74032916:04
openstackgerritDmitry Tantsur proposed openstack/ironic stable/train: Stop using md5 for __repr__ of objects  https://review.opendev.org/74033016:04
*** k_mouza has quit IRC16:19
TheJuliagah corgis16:21
TheJuliadtantsur: thanks on the ironic-inspector patch16:21
dtantsurnp.. now wrapping my head around the IPA part16:21
*** ociuhandu_ has joined #openstack-ironic16:23
openstackgerritDoug Szumski proposed openstack/ironic-python-agent master: WIP: Fix bootloader install issue with MDRAID  https://review.opendev.org/74033116:24
dtantsurugh16:24
TheJulialooks like ironic-inspector-tempest-managed-non-standalone has some intermittent failures every so often16:24
dtantsurI think it's the same problem with Bad Gateway16:25
TheJuliaahh, looks like the latest... it unpacked the initramfs and nothing else was logged16:26
*** ociuhandu has quit IRC16:27
TheJulia4 cores, 3 gb of ram...16:27
*** ociuhandu_ has quit IRC16:28
openstackgerritDmitry Tantsur proposed openstack/ironic-python-agent master: Ignore devices with size 0 when collecting inventory  https://review.opendev.org/74030916:28
dtantsurarne_wiebalck: something like ^^?16:29
*** dougsz has quit IRC16:31
openstackgerritMerged openstack/ironic master: Minor agent token log handling  https://review.opendev.org/73621416:38
openstackgerritVerification of a change to openstack/ironic failed: Do not validate driver on changing non-driver fields  https://review.opendev.org/73969916:38
*** lucasagomes has quit IRC16:39
TheJuliadtantsur: yup, bad gateway...16:46
*** ricolin has quit IRC16:47
*** phrobb_ has joined #openstack-ironic16:48
*** andreaf has quit IRC16:48
*** mnasiadka_ has joined #openstack-ironic16:48
*** m1p_ has joined #openstack-ironic16:48
TheJuliadtantsur: I wonder if the proxy server needs to be primed or the config is unhappy?16:50
dtantsurI tried to figure out, but couldn't :(16:52
dtantsurI think it's coming from oslo.messaging16:54
*** phrobb has quit IRC16:55
*** m1p has quit IRC16:55
*** mnasiadka has quit IRC16:55
*** m1p_ is now known as m1p16:55
*** mnasiadka_ is now known as mnasiadka16:55
*** phrobb_ is now known as phrobb16:55
*** mkowalski has quit IRC16:56
*** derekh has quit IRC16:59
*** mkowalski has joined #openstack-ironic17:03
*** sshnaidm is now known as sshnaidm|afk17:12
TheJuliahmm17:14
* TheJulia digs17:14
arne_wiebalckdtantsur: yes, that looks good to me17:15
TheJuliadtantsur: interesting, I do see some weirdness on the message bus17:16
TheJuliawow, talk about errors not adding up17:18
*** Lucas_Gray has quit IRC17:19
TheJuliaso iptables and ipmitool commands are failing all over the place17:23
TheJuliaI see what is going on17:28
TheJuliaThe test aborted it17:30
arne_wiebalckbye everyone o/17:31
TheJuliagoodnight!17:31
TheJuliaheh, yeah, it is cascading out from the original failure17:32
dtantsurwow17:35
TheJuliait makes no sense17:38
TheJulialike the "please go inspect now, kthxbai" command is actually processed but the API is returning an error17:39
dtantsuryup :(17:42
dtantsurand oslo.messaging seems configured the same way in inspector as in ironic17:43
*** dtantsur is now known as dtantsur|afk17:45
dtantsur|afko/17:45
TheJuliai have a theory, it may be bunnies!17:48
TheJuliaI think the executor submit is blowing up17:57
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: Additional debugging around inspection start  https://review.opendev.org/74034418:10
TheJulia85% CPU wait...18:13
TheJuliayeeeahhhhhh18:13
TheJuliaNot really sure, 5 GB of ram in full use, moderate IO usage but also high CPU wait times, high new thread/process counts18:23
TheJuliaokay, not that high on wait, still odd18:34
openstackgerritRichard G. Pioso proposed openstack/ironic stable/ussuri: Correct Redfish boot once fallback conditional  https://review.opendev.org/74035620:11
*** ociuhandu has joined #openstack-ironic20:12
openstackgerritJulia Kreger proposed openstack/ironic master: Remove locks before RPC bus is started  https://review.opendev.org/74036020:14
*** diurnalist has quit IRC20:14
*** diurnalist has joined #openstack-ironic20:16
*** diurnalist has quit IRC20:22
*** diurnalist has joined #openstack-ironic20:22
*** ociuhandu has quit IRC20:31
openstackgerritRichard G. Pioso proposed openstack/ironic stable/train: Correct Redfish boot once fallback conditional  https://review.opendev.org/74036820:37
*** ociuhandu has joined #openstack-ironic20:47
TheJuliathis inspector issue is just bizzar20:55
TheJuliaits not the message bus, api does as expected returns a 20220:56
TheJuliaapache blows up20:56
*** ociuhandu has quit IRC20:59
*** yolanda has quit IRC21:16
*** yolanda has joined #openstack-ironic21:17
openstackgerritJulia Kreger proposed openstack/ironic-inspector master: WIP: Additional debugging around inspection start  https://review.opendev.org/74034421:23
TheJuliaso it seems if you return an empty body, there is no character encoding...21:25
TheJuliaand no character encoding can cause apache to get bent out of shape21:25
TheJuliaand try to return an error document, which may result in Bad Gateway errors when it tries to get the error document21:25
* TheJulia calls it a day22:04
eanderssonInteresting. Is that a 100% failure or just sometimes?22:23
eanderssonI have a Bad Gateway error for Designate  I have been trying to figure out for ages, but related to many consecutive Unauthorized calls.22:24
*** rh-jelabarre has quit IRC22:30
*** jamesdenton has quit IRC22:40
*** tkajinam has joined #openstack-ironic23:00
*** diurnalist has quit IRC23:30
*** dmellado has quit IRC23:39
openstackgerritSteve Baker proposed openstack/ironic master: Add json and param parsing to args  https://review.opendev.org/70448923:39
openstackgerritSteve Baker proposed openstack/ironic master: Add wsme core types, remove WSME  https://review.opendev.org/70449023:39
openstackgerritSteve Baker proposed openstack/ironic master: Convert root path / to use plain JSON  https://review.opendev.org/74014223:39
*** gyee has quit IRC23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!