Tuesday, 2019-08-20

« Monday, 2019-08-19 Index Wednesday, 2019-08-21 »
*** bfournie has joined #openstack-ironic00:34
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Fix networking-baremetal CI  https://review.opendev.org/67570100:36
*** gyee has quit IRC00:36
*** altlogbot_1 has quit IRC00:46
*** altlogbot_1 has joined #openstack-ironic00:47
*** gregoryo has joined #openstack-ironic00:50
*** igordc has quit IRC01:11
openstackgerritMerged openstack/ironic stable/rocky: iLO firmware update fails with 'update_firmware_sum' clean step  https://review.opendev.org/67461201:16
*** ricolin has joined #openstack-ironic02:39
*** rloo has quit IRC02:44
*** mkrai has joined #openstack-ironic02:53
*** gkadam has joined #openstack-ironic03:33
*** gkadam has quit IRC03:40
*** mkrai has quit IRC03:50
*** mkrai has joined #openstack-ironic03:52
*** sthussey has quit IRC05:12
*** persia has quit IRC05:49
*** persia has joined #openstack-ironic06:00
*** persia has quit IRC06:03
*** persia has joined #openstack-ironic06:05
*** persia has quit IRC06:05
*** persia has joined #openstack-ironic06:05
*** mkrai has quit IRC06:12
*** mkrai_ has joined #openstack-ironic06:12
arne_wiebalckGood morning, ironic!06:21
*** rcernin has quit IRC06:35
*** diga has joined #openstack-ironic06:38
*** trident has quit IRC07:16
*** trident has joined #openstack-ironic07:24
openstackgerritShivanand Tendulker proposed openstack/ironic stable/stein: Asynchronous out of band deploy steps fails to execute  https://review.opendev.org/67615207:25
*** ricolin has quit IRC07:39
*** ricolin has joined #openstack-ironic07:39
*** ianychoi has quit IRC07:46
*** ianychoi has joined #openstack-ironic07:47
*** lucasagomes has joined #openstack-ironic07:56
*** ociuhandu has joined #openstack-ironic07:58
*** dougsz has joined #openstack-ironic07:59
*** ociuhandu has quit IRC07:59
*** ociuhandu has joined #openstack-ironic08:00
*** priteau has joined #openstack-ironic08:06
*** e0ne has joined #openstack-ironic08:26
*** derekh has joined #openstack-ironic08:28
*** yolanda has quit IRC08:30
*** yolanda has joined #openstack-ironic08:43
openstackgerritMerged openstack/ironic master: Make ironic_log Ansible callback Python 3 ready  https://review.opendev.org/67726208:53
*** gregoryo has quit IRC09:06
*** rcernin has joined #openstack-ironic09:15
*** ociuhandu has quit IRC09:54
*** priteau has quit IRC10:18
*** ociuhandu has joined #openstack-ironic10:27
*** rcernin has quit IRC10:28
*** dougsz has quit IRC10:32
*** ociuhandu has quit IRC10:35
*** gkadam has joined #openstack-ironic10:38
*** dougsz has joined #openstack-ironic10:45
*** gkadam is now known as gkadam-afk11:00
*** dougsz has quit IRC11:02
*** tesseract has joined #openstack-ironic11:08
*** ociuhandu has joined #openstack-ironic11:08
*** ociuhandu has quit IRC11:09
*** ociuhandu has joined #openstack-ironic11:09
openstackgerritMaël Kimmerlin proposed openstack/sushy-tools master: fix emulator: remove image on eject and update devices  https://review.opendev.org/67744911:19
*** dougsz has joined #openstack-ironic11:19
TheJuliagood morning arne_wiebalck11:20
arne_wiebalckTheJulia: o/11:20
openstackgerritMerged openstack/ironic master: Move to unsafe caching  https://review.opendev.org/67691611:23
TheJuliaWell, it feels like this deploy failed... https://object-storage-ca-ymq-1.vexxhost.net/v1/86bbbcfa8ad043109d2d7af530225c72/logs_83/677183/1/check/ironic-standalone/2f1c532/controller/logs/ironic-bm-logs/node-5_no_ansi_2019-08-20-07:24:33_log.txt.gz11:25
openstackgerritJulia Kreger proposed openstack/ironic master: [DNM] Increase standard VM memory to 400  https://review.opendev.org/67745011:27
arne_wiebalckunsafe caching merged \o/ :-D11:28
arne_wiebalckTheJulia: Since I am fixing RAIDs since yesterday (and while I agree that the cloud solution is to delete/recreate the instance): it would be kind of handy if rebuilding a node would include rebuilding a (s/w) RAID ... in the end, it uses the same name already! ;-)11:33
*** rh-jelabarre has joined #openstack-ironic11:33
*** gkadam-afk is now known as gkadam11:42
TheJuliaarne_wiebalck: couldn't rebuild be used?11:43
openstackgerritJim Rollenhagen proposed openstack/python-ironicclient master: Add release note for bug #2006216  https://review.opendev.org/67720411:44
*** ociuhandu has quit IRC11:54
*** ociuhandu has joined #openstack-ironic11:54
arne_wiebalckTheJulia: We'd need to tell rebuild to rebuild the RAID.11:57
openstackgerritJulia Kreger proposed openstack/ironic master: Lower standalone concurrency to 3 from 4  https://review.opendev.org/67745412:04
TheJuliacustom rebuild deploy? :) rebuild with deploy_steps? :)12:04
TheJuliathe conundrum with rebuild is preservation of ephemeral data.12:04
* TheJulia is not sure anyone uses that12:04
TheJulianow...12:04
*** rpittau|afk is now known as rpittau12:05
rpittaugood afternoon ironic! o/12:05
TheJuliaGood morning rpittau12:05
arne_wiebalckHey rpittau o/12:06
rpittauhi TheJulia :)12:06
rpittauhey arne_wiebalck :)12:06
TheJuliadtantsur|afk: look at the memory consumption of the standalone job with 6 VMs, there is about always 900MB-1GB free with almost no swapping. There is however some heavy periods of 10-20 seconds of iowait which is likely noisy neighbor kind of insanity.12:08
arne_wiebalckTheJulia: The current issue is that in order to "rebuild" a node, it needs to go through cleaning to rebuild/recreate the RAID.12:08
arne_wiebalckTheJulia: That's ok if you can easily delete/recreate, but if you use these nodes as hypervisors and you need to update placement aggregates and others things afterwards, the workflow gets somewhat more complicated.12:09
*** henriqueof has quit IRC12:09
arne_wiebalckme procrastinates creating an ironic-aware reinstallation workflow12:11
* arne_wiebalck procrastinates creating an ironic-aware reinstallation workflow12:11
*** ociuhandu has quit IRC12:24
TheJuliaheh12:28
TheJuliaarne_wiebalck: would ironic publishing data into placement help address some of that headache?12:29
TheJuliaI know cdent would <3 us doing that.12:29
TheJuliaarne_wiebalck: would rebuild in your case be like... upgrading the hypervisor?12:31
*** gkadam has quit IRC12:32
*** bfournie has quit IRC12:35
arne_wiebalckTheJulia: The case I'm facing is that the RAID-0 on a hypervisor is broken due to a disk failure. The disk is replaced, now the instances needs to be re-installed.12:36
arne_wiebalckTheJulia: the physical instance (not "instances")12:37
arne_wiebalckTheJulia: "rebuild" expects a healthy device12:38
TheJuliahmmm yes12:38
*** cdearborn has joined #openstack-ironic12:38
TheJuliastructurally things have changed....12:38
arne_wiebalckTheJulia: was just an idea, maybe sth to discuss at the ptg12:39
TheJuliaIt would be a good topic, rebuild would be the right path I think and rebuild under the hood is just another verb for deploy12:39
TheJuliaso... hmm maybe a path for on deploy raid12:39
arne_wiebalckyeah, I was thinking if that was related12:46
arne_wiebalckor could be related (as it does not exist yet :) )12:46
*** henriqueof has joined #openstack-ironic12:56
openstackgerritMaël Kimmerlin proposed openstack/sushy-tools master: fix emulator: remove image on eject and update devices  https://review.opendev.org/67744912:57
*** ociuhandu has joined #openstack-ironic12:57
*** ociuhandu has quit IRC12:58
*** ociuhandu has joined #openstack-ironic12:58
openstackgerritHarald Jensås proposed openstack/networking-baremetal master: Fix networking-baremetal CI  https://review.opendev.org/67570113:00
openstackgerritMaël Kimmerlin proposed openstack/sushy-tools master: fix emulator: remove image on eject and update devices  https://review.opendev.org/67744913:01
hjensasTheJulia: I think it's working ^^, but taking too long.13:04
hjensasTheJulia: Is in inheriting tempest_concurrency: 1 from ironic-base and not utilizing the extra VMs to do concurrency?13:05
*** rloo has joined #openstack-ironic13:12
*** goldyfruit has quit IRC13:13
openstackgerritRiccardo Pittau proposed openstack/python-ironicclient master: Remove deprecated common.http.HTTPClient class  https://review.opendev.org/66627813:15
*** dougsz has quit IRC13:18
TheJuliahjensas: well, because of cleaning in some cases, we can only really run one test at a time because the test is cycling through instances13:25
openstackgerritGaëtan Trellu proposed openstack/ironic master: Pass target_raid_config field to ironic variable  https://review.opendev.org/67729713:26
*** goldyfruit has joined #openstack-ironic13:27
openstackgerritGaëtan Trellu proposed openstack/ironic master: Pass target_raid_config field to ironic variable  https://review.opendev.org/67729713:31
TheJuliawoot, standalone job passed on https://review.opendev.org/#/c/677454/13:32
patchbotpatch 677454 - ironic - Lower standalone concurrency to 3 from 4 - 1 patch set13:32
TheJuliaif any cores would like to review ^^^, that would be helpful since the gate issues on that job are becoming extremely problematic and blocking lots of things from merging13:33
*** dougsz has joined #openstack-ironic13:34
*** ociuhandu has quit IRC13:41
*** Goneri has joined #openstack-ironic13:41
*** sthussey has joined #openstack-ironic13:42
*** mkrai_ has quit IRC13:43
TheJuliaAny additional core reviewer eyes on https://review.opendev.org/#/c/677261/ would be appreciated since it is a bug in ironic-lib13:45
patchbotpatch 677261 - ironic-lib - Erase expected GPT locations in metadata wipe - 2 patch sets13:45
*** ociuhandu has joined #openstack-ironic13:46
*** henriqueof has quit IRC14:03
*** goldyfruit has quit IRC14:04
*** ricolin has quit IRC14:04
arne_wiebalckTheJulia: For the patch ^^^: doesn't dd need an "if=" param?14:08
TheJuliahmmm14:09
TheJuliamost cases....14:09
* TheJulia goes off and double checks IPA14:09
TheJuliaarne_wiebalck: uhh yeah14:10
TheJuliahttps://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/shell/write_image.sh#L4314:10
TheJuliadoh!14:10
TheJuliawithout it, I suspect it is a noop14:10
arne_wiebalckno14:10
arne_wiebalckI think it waits for stdin.14:11
arne_wiebalckyeah, stdin ... it would probably block14:12
TheJuliaI don't think it gets stdin in this case, but I'll fix it after my current meeting. Having to oddly speak some14:13
arne_wiebalckah, no? utils_execute would prevent this? anyway, I left a comment on the review so that it waits for your update14:18
TheJuliathanks arne_wiebalck14:18
*** mkrai_ has joined #openstack-ironic14:21
arne_wiebalckTheJulia: yw14:23
openstackgerritJulia Kreger proposed openstack/ironic-lib master: Erase expected GPT locations in metadata wipe  https://review.opendev.org/67726114:46
TheJuliaarne_wiebalck: ^^^14:46
mbuilguys, what package does ironic expect the deployed image to have in order to read the configdrive? I can see that the configdrive partition is correct and mounted, the info is correct too but the blade is not configuring itself with that information. I wonder what I am missing...14:50
*** mkrai_ has quit IRC14:51
TheJuliambuil: cloud-init, there is also a lighter weight tool called glean14:53
TheJuliabut it is not as extensive nor extensible14:53
TheJuliaglean is what openstack's CI instances use so cloud-init is not used.14:53
*** openstackstatus has quit IRC14:58
*** openstack has joined #openstack-ironic15:01
*** ChanServ sets mode: +o openstack15:01
mbuilTheJulia: ah ha, thanks. Problem found. I moved to python3, and apparently glean is still trying to use python2. I need to check where it gets installed15:01
TheJuliaWell that is a nice bug15:02
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: Handle floppy disk controllers  https://review.opendev.org/67728115:10
donnydI am hoping someone can help me get to the bottom of why IPA would just sit in cleaning mode and do nothing. Inspection works, if I manually clean it works, if I use automated cleaning it just sits there doing seemingly nothing15:13
TheJuliadonnyd: what does your current clean_step say?15:13
donnydI am not sure where to check that, this is literally the first time i have ever had an issue with IPA15:14
donnydprovisioning state is clean_wait15:15
TheJuliadonnyd: if you do openstack baremetal node show $uuid15:15
TheJuliait should be one of the fields printed out15:15
donnydrestarting cleaning (hit the timeout)15:16
JayFIt's nice to hear you've had IPA work for you for so long without an issue though!15:16
donnydIn my last job I used 3O almost daily on location, and never had an issue with IPA... lots of issues with not following instructions or others not knowing how vlans work... but never anything with IPA15:17
JayFThis could very easily be caused by those pesky network configs though15:18
donnydIf it would pxe to the ipa image, i was good to hook15:18
donnydwell I would say you could be correct, but inspection works fine, and manual cleaning works fine15:18
donnydjust the automated part15:18
JayFIs it possible the node has had a hardware failure?15:19
donnydnot likely15:19
donnydworks with foreman and maas just fine15:19
JayF*defers to TheJulia's more up to date knowledge* she's the pro on this15:20
TheJuliadonnyd: when doing manual cleaning, what commands are you passing?15:21
donnyd```[{"interface": "deploy", "step": "erase_devices_metadata"}]```15:22
JayFHas this particular node ever been successfully automatically cleaned? If not, is it identical hardware to other nodes that have?15:23
donnydcomes up, blows off the headers, and goes back down like its supposed to15:23
donnydin manual mode15:23
donnydno, i have not JayF15:23
JayFWhat about the second question then?15:24
donnydIts currently the only node in the pool15:24
donnydso no luck there either15:24
JayFDoes the node by any chance have a floppy disk :P /s (that's for TheJulia )15:24
TheJuliadonnyd: anything in last_error?15:24
TheJuliaJayF: yeaaaahhhhh :(15:24
TheJuliafound another bug in IPA :(15:25
donnydseriously.... it just completed successfully...  It just wanted me to reach out and waste peoples time. ... I am checking again to make sure it does the same thing15:25
JayFThat's actually pretty impressive if it recovered from a failure on it's own... our handling of those kinds of failures used to not be this good15:27
donnydIts possible that this parameter  was stuck on full wipe ```agent_cached_clean_steps```15:27
donnydnever checked it, and this node was enrolled before I changed it to metadata only15:27
JayFBe careful doing metadata only wipes unless all those who will be booting nodes are trusted :)15:28
donnydOh this thing only does Openstack  CI work (fortnebula)15:29
donnydzomg.. yea IPA works fine... as always. Sorry for the distraction15:29
TheJuliadonnyd: hmmmm.... I'd keep an eye on that machine, maybe one of the disks is being problematic?15:30
JayFIt's fine. I needed the boost this morning, I worked on the team that originally did IPA and cleaning years ago... so please tell me more about how well it works :P15:30
JayFTheJulia++ absolutely smells like a bad disk15:30
JayFthat's where I was going with the "has this node ever cleaned" question15:30
donnydthat is surely a possibility as well. I have plenty of spares so no biggie if its burned15:31
donnydI am trying to get some BM resources in FN so devs can execute jobs against metal as it would seem like there is a great need for it15:32
donnydI only have a couple pieces of metal, but its better than none15:32
JayFIt's nice to know Ironic is getting easy enough to use that it's worth spinning up only for a handful of servers :)15:36
JayFit used to be an ordeal to get it deployed for any at all15:36
donnydmaybe for NUMAy jobs and things that don't like nested virt (or need more)15:36
JayFif you give developers bare metal they'll use it and then you have to give them more :P15:36
JayFnobody likes a hypervisor except for the operators who get the easier management lol15:36
donnydJayF: well to be fair, i used it daily for many years... so maybe I just see the world different... but I am never installing from ISO - ever again.. .I can hand jam an ironic setup faster than install of 8 servers15:37
TheJuliadonnyd: I think we want t-shirts of that line :)15:39
donnydLOL  - you can have it15:40
JayFjulia and I have both been working on Ironic from the point in time15:40
JayFwhere you needed a week and a lot of brainpower to get ironic installed15:41
JayFand when like, half of the stuff you're talking about (IPA + cleaning) didn't even exist yet15:41
JayFdonnyd is like the fairy of openstack good news. Comes in for support, "just kidding it works amazing like always"15:41
JayFjust feel free to do this daily :D15:41
donnydLOL15:42
* TheJulia has all these t-shirt ideas now15:42
donnydYea, well its not something I am used to fighting with... But I am spoiled by OOO (director)15:43
* TheJulia ponders bug15:44
*** gyee has joined #openstack-ironic15:45
*** lucasagomes has quit IRC16:04
openstackgerritMerged openstack/ironic master: Add iPXE boot interface to 'ilo' hardware type  https://review.opendev.org/67685416:04
mbuilTheJulia: do you know anyone who could help me with glean? I am having DHCP - glean conflict16:05
openstackgerritDigambar proposed openstack/ironic stable/queens: Revert "Fix OOB introspection to use pxe_enabled flag in idrac driver"  https://review.opendev.org/67750216:08
arne_wiebalckbye, everyone o/16:14
rpittaugood night! o/16:17
*** rpittau is now known as rpittau|afk16:17
JayFmbuil: that's likely not a problem with glean -- it's just setting up basic networking. If you have a DHCP conflict it's most likely a core networking issue unrelated to ironic/glean.16:19
mbuilJayF I was wondering if I need to deactivate DHCP in the interface config. When glean tries to configure it, the interface has already a configuration and then glean does not do anything. Or is it glean the one doing that for me?16:24
*** e0ne has quit IRC16:25
JayFhttps://docs.openstack.org/infra/glean/ according to this (I don't know more than this doc tells me), glean looks for interface configuration based on configdrive, if it can't find one, it sets all interfaces to DHCP16:29
mbuilJayF: I can see that behavior when using python2. However, when moving to python3, DHCP gets activated :(16:35
TheJuliambuil: mordred on #openstack-infra might be able to help provide insight on glean. It has been ages since I've touched the glean code :(16:37
TheJuliaThere is a case where glean _does_ try to fire up dhcp though16:37
TheJuliaand that is codified based upon metadata and startup scripts if memory serves16:37
TheJuliaI need to go to the bank, and then get more cat litter otherwise the feline overlords will begin sharpening their claws....16:41
TheJuliambuil: just based on moving to py3 and activating seems really weird since I'm 99.95% sure (the last time I looked, that that action was all magic as part of system init scripts16:42
mbuilTheJulia: I also realized the version is different. I am trying py3 with the version I was using in py2 to doublecheck. Otherwise, I'll start asking on #openstack-infra. Thanks for the hints16:52
*** derekh has quit IRC16:59
openstackgerritRuby Loo proposed openstack/ironic master: Follow up to Option to send all portgroup data  https://review.opendev.org/67751617:02
*** gyee has quit IRC17:09
*** tesseract has quit IRC17:10
*** ociuhandu has quit IRC17:18
*** gyee has joined #openstack-ironic17:22
*** dougsz has quit IRC17:38
NobodyCamGood Morning Ironiic'ers17:39
TheJuliagood  morning NobodyCam17:42
NobodyCam:)17:42
*** tesseract has joined #openstack-ironic17:45
*** igordc has joined #openstack-ironic17:48
openstackgerritJulia Kreger proposed openstack/python-ironicclient master: Remove deprecated common.http.HTTPClient class  https://review.opendev.org/66627817:49
openstackgerritMerged openstack/python-ironicclient master: Add release note for bug #2006216  https://review.opendev.org/67720418:03
TheJuliaCI i s really starting to become depressing18:06
JayFsome things change, some stay the same18:06
TheJuliarloo: BTW ^^ I revised Ricardo's patch that hit a pep8 error. Easy fix18:07
TheJuliaThe death my a thousand different little failures that occur18:07
rlooTheJulia: looking...18:08
* TheJulia plays a fashion of the 50s youtube video while unit tests run... for sanity.18:08
rlooTheJulia: +A18:09
JayFTheJulia: I always watch videos of Ellen giving people stuff when I need my sanity restored18:09
TheJuliawhich reminds me... I need to make a salon appointment for a fundraising gala back home.....18:10
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: DNM - Trying to gather more info for failure  https://review.opendev.org/67753218:15
openstackgerritJulia Kreger proposed openstack/ironic master: Change [deploy]/default_boot_option to local  https://review.opendev.org/49211418:16
*** goldyfruit has joined #openstack-ironic18:18
JayFIs the ironic-whiteboard still used? The "developer review dashboard" link in it doesn't work anymore, at a minimum18:22
openstackgerritJulia Kreger proposed openstack/ironic master: DNM: build ramdisk for troubleshooting  https://review.opendev.org/67753518:24
TheJuliaJayF: The whiteboard is18:24
* TheJulia goes and edits18:24
TheJuliaJayF: what line was that on?18:27
openstackgerritGaëtan Trellu proposed openstack/ironic master: Pass target_raid_config field to ironic variable  https://review.opendev.org/67729718:46
*** diga has quit IRC18:48
JayFTheJulia: #4318:50
JayFdeva's alternative on #45 works, but IDK how up to date that is18:50
TheJulia43 works  just fine for me18:51
JayFand so it does for me now, too18:51
JayFit was saying "only up to 10 filters can be provided" when I sent that message earlier18:52
TheJuliaJayF: likely missing some repos, oh heh18:52
TheJuliaIronic has an absurd number of repos now18:52
JayFreposit all the things!18:52
TheJuliaWe turned it to 11 a few times and... yeah... stuff happened18:53
*** e0ne has joined #openstack-ironic19:32
openstackgerritGaëtan Trellu proposed openstack/ironic master: Pass target_raid_config field to ironic variable  https://review.opendev.org/67729719:36
openstackgerritJulia Kreger proposed openstack/ironic-lib master: Add todo note for is_block_device  https://review.opendev.org/67755819:44
openstackgerritMerged openstack/python-ironicclient master: Remove deprecated common.http.HTTPClient class  https://review.opendev.org/66627820:01
*** ash2307 has joined #openstack-ironic20:07
rlooTheJulia: wrt https://review.opendev.org/#/c/677261, I have a question. Let me know if you want to change it, or if you want +A.20:13
patchbotpatch 677261 - ironic-lib - Erase expected GPT locations in metadata wipe - 3 patch sets20:13
TheJuliaI think I figured out the vda1 failure....20:13
*** Lucas_Gray has joined #openstack-ironic20:16
TheJuliarloo: I say +A it20:18
* TheJulia works on fixing tests for fix for the vda1 headache20:18
rlooTheJulia: done!20:18
*** Goneri has quit IRC20:28
*** e0ne has quit IRC20:50
openstackgerritJulia Kreger proposed openstack/ironic-lib master: Rescan after making partition changes  https://review.opendev.org/67756621:01
TheJuliadtantsur|afk: ^^^21:01
*** bnemec has quit IRC21:27
*** tesseract has quit IRC21:32
*** Lucas_Gray has quit IRC21:32
*** Lucas_Gray has joined #openstack-ironic21:33
openstackgerritMerged openstack/ironic master: Remove ironic command bash completion  https://review.opendev.org/67718321:45
openstackgerritMerged openstack/ironic-lib master: Erase expected GPT locations in metadata wipe  https://review.opendev.org/67726121:45
openstackgerritGaëtan Trellu proposed openstack/ironic master: Pass target_raid_config field to ironic variable  https://review.opendev.org/67729721:50
openstackgerritJulia Kreger proposed openstack/python-ironicclient master: Remove the ironic command  https://review.opendev.org/67651521:51
*** rcernin has joined #openstack-ironic22:00
*** trident has quit IRC22:02
*** Lucas_Gray has quit IRC22:04
*** trident has joined #openstack-ironic22:11
*** zaneb has quit IRC23:41
*** zaneb has joined #openstack-ironic23:41
*** sthussey has quit IRC23:42
« Monday, 2019-08-19 Index Wednesday, 2019-08-21 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!