| *** gyee has quit IRC | 00:25 | |
| *** hshiina has joined #openstack-ironic | 00:52 | |
| *** dsneddon has quit IRC | 01:08 | |
| *** mjturek has quit IRC | 01:24 | |
| TheJulia | tonyb: I suspect an option | 01:25 |
|---|---|---|
| TheJulia | I was not aware openbmc implemented ipmi | 01:25 |
| tonyb | TheJulia: It does, there is an alternate driver but it isn't ready today | 01:25 |
| tonyb | TheJulia: Is theer a similar option I could consult as a template? | 01:26 |
| TheJulia | not really, but it is just the timeout flag right? | 01:27 |
| TheJulia | actually, there might be in one of the ipmi options | 01:29 |
| * TheJulia grabs link | 01:29 | |
| TheJulia | tonyb: wait, that link is from the irmc driver | 01:30 |
| tonyb | TheJulia: there may be another place we send those same bytes let me check | 01:31 |
| tonyb | ironic/drivers/modules/ipmitool.py:920 is probably the right point | 01:32 |
| * tonyb did bad grepping | 01:32 | |
| TheJulia | something kind of like http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/ipmitool.py#n95 | 01:33 |
| tonyb | TheJulia: Okay so I could add ipmi_dont_do_the_timeout_thing abnd then set that on the node? | 01:34 |
| TheJulia | yeah | 01:34 |
| TheJulia | or "ipmi_remote_bmc_openbmc" or "ipmi_the_bmc_does_not_like_us" | 01:35 |
| tonyb | TheJulia: ... would that be backportable to rocky? | 01:35 |
| *** rloo has quit IRC | 01:35 | |
| TheJulia | nein. :( | 01:35 |
| tonyb | TheJulia: okay, that was kinda what I expected | 01:35 |
| TheJulia | Nothing really would :\ | 01:35 |
| TheJulia | Since it would be a new option to support openbmc | 01:36 |
| tonyb | TheJulia: well support an otherwise fine implenentation of an IPMI based BMC ;p | 01:36 |
| *** rh-jelabarre has joined #openstack-ironic | 01:37 | |
| TheJulia | tonyb: what is it doign to the bmc anyway? | 01:37 |
| tonyb | TheJulia: The IPMI spec says: don’t automatically clear boot flag valid bit if Chassis Control command not | 01:38 |
| tonyb | and JayF added it in https://review.openstack.org/#/c/134413/ claiming that some BMCs got grumpy it it wasn't sent | 01:39 |
| patchbot | patch 134413 - ironic - Disable IPMI timeout before setting boot device (MERGED) - 1 patch set | 01:39 |
| tonyb | TheJulia: something about not resetting the device boot timeout after 60seconds | 01:39 |
| TheJulia | That was a long time ago | 01:39 |
| tonyb | TheJulia: Yeah but it'd be next to impossible to verify it isn't needed anymore | 01:40 |
| TheJulia | yup | 01:40 |
| TheJulia | Does the bmc tank, or does it just respond poorly? | 01:40 |
| tonyb | TheJulia: We could add a flag to disable it and slow transition that from fale -> true -> gone to remove it | 01:40 |
| tonyb | TheJulia: ipmitool returns non-zero and what cause the boot to abort with the appropriate exception | 01:41 |
| TheJulia | any specific error string? anythign we can use to determine it is not horribly fatal for us? | 01:41 |
| tonyb | TheJulia: Hmm I'll grep for the logs, I'm seeing it through a tripleo lens | 01:42 |
| tonyb | TheJulia: gimme 5-10 to find it | 01:42 |
| tonyb | ironic-conductor.log:2018-11-05 18:49:21.877 1 ERROR ironic.drivers.modules.ipmitool Stderr: u'Unable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8 rsp=0x80): Unknown (0x80)\n' | 01:43 |
| tonyb | that was much easier than I expected | 01:43 |
| TheJulia | fun | 01:43 |
| * TheJulia wonders if we know that... if we could just kind of go "okay" and move along | 01:44 | |
| tonyb | TheJulia: I was thinking of adding a ok_to_fail=False kwarg to send_raw and then setting that to True for this one call but adding an option seems more correct | 01:46 |
| *** gabys has joined #openstack-ironic | 01:51 | |
| * tonyb will file a bug and change upon which we can discuss the merits | 01:54 | |
| TheJulia | sounds good | 01:55 |
| TheJulia | could you add it to the meeting discussion list? | 01:56 |
| *** gabys has quit IRC | 02:01 | |
| *** gabys has joined #openstack-ironic | 02:02 | |
| tonyb | TheJulia: meeting discussion list? | 02:06 |
| tonyb | TheJulia: https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting ? | 02:07 |
| TheJulia | yup, I've not cleaned it up from today, but I'll see it when I edit it next | 02:08 |
| tonyb | Okay cool. I wont be there as it's at 0200 for me, this would be for after the summit correct? | 02:09 |
| TheJulia | yes | 02:10 |
| tonyb | cool | 02:10 |
| TheJulia | we've cancelled for the next two weeks, I've just not sent the email | 02:10 |
| *** ianychoi has quit IRC | 02:10 | |
| TheJulia | because OMG all the things | 02:10 |
| tonyb | :) | 02:10 |
| *** ianychoi has joined #openstack-ironic | 02:10 | |
| *** ajya[m] has quit IRC | 02:17 | |
| *** toabctl has quit IRC | 02:19 | |
| *** toabctl has joined #openstack-ironic | 02:23 | |
| *** ajya[m] has joined #openstack-ironic | 02:50 | |
| *** ajya[m] has quit IRC | 03:39 | |
| openstackgerrit | Merged openstack/ironic master: Fix CPU count returned by introspection in Ironic iDRAC driver https://review.openstack.org/609982 | 04:17 |
| openstackgerrit | Merged openstack/ironic master: Improve sushy mocks https://review.openstack.org/615621 | 04:17 |
| *** ajya[m] has joined #openstack-ironic | 04:35 | |
| *** jtomasek has joined #openstack-ironic | 05:14 | |
| *** rh-jelabarre has quit IRC | 05:32 | |
| *** skazi has quit IRC | 05:34 | |
| *** S4ren has quit IRC | 05:44 | |
| *** S4ren1 has joined #openstack-ironic | 05:44 | |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Add configurable Redfish client authentication https://review.openstack.org/603818 | 05:48 |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Support ip6tables for iptables pxe filter https://review.openstack.org/615728 | 05:51 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Kill misbehaving `ipmitool` process https://review.openstack.org/610007 | 06:02 |
| *** e0ne has joined #openstack-ironic | 06:24 | |
| *** jtomasek has quit IRC | 06:39 | |
| *** e0ne has quit IRC | 06:44 | |
| *** e0ne has joined #openstack-ironic | 06:48 | |
| *** e0ne has quit IRC | 07:07 | |
| *** iurygregory|home is now known as iurygregory | 07:19 | |
| *** lenka has joined #openstack-ironic | 07:22 | |
| iurygregory | good morning | 07:25 |
| *** pcaruana has joined #openstack-ironic | 07:36 | |
| *** adrianc has joined #openstack-ironic | 07:43 | |
| *** gkadam_ has joined #openstack-ironic | 07:45 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 07:54 | |
| *** gabys has quit IRC | 07:58 | |
| *** gabys has joined #openstack-ironic | 07:59 | |
| *** iurygregory is now known as iurygregory|mtg | 08:01 | |
| *** lenka has quit IRC | 08:03 | |
| *** gabys has quit IRC | 08:03 | |
| *** tssurya has joined #openstack-ironic | 08:18 | |
| *** priteau has joined #openstack-ironic | 08:39 | |
| *** jtomasek has joined #openstack-ironic | 08:41 | |
| openstackgerrit | Xing Zhang proposed openstack/bifrost stable/rocky: Fix ironic configuration for new oslo messaging https://review.openstack.org/615781 | 08:43 |
| *** betherly has joined #openstack-ironic | 09:01 | |
| *** lenka has joined #openstack-ironic | 09:10 | |
| *** dougsz has joined #openstack-ironic | 09:11 | |
| openstackgerrit | Nguyen Hai Truong proposed openstack/python-ironic-inspector-client master: Add python 3.6 unit test job https://review.openstack.org/615794 | 09:15 |
| *** priteau has quit IRC | 09:20 | |
| *** Emine has joined #openstack-ironic | 09:20 | |
| *** priteau has joined #openstack-ironic | 09:23 | |
| jiapei | Good morning all o/ | 09:25 |
| etingof | good morning jiapei, iurygregory|mtg and the rest of ironic o/ | 09:38 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Kill misbehaving `ipmitool` process https://review.openstack.org/610007 | 09:38 |
| *** derekh has joined #openstack-ironic | 09:39 | |
| jiapei | hey, morning etingof | 09:39 |
| *** hshiina has quit IRC | 09:59 | |
| *** serlex has joined #openstack-ironic | 10:01 | |
| *** iurygregory|mtg is now known as iurygregory | 10:20 | |
| iurygregory | etingof, morning o/ | 10:20 |
| openstackgerrit | Nguyen Van Trung proposed openstack/ironic master: Support firmware upgrade for irmc driver https://review.openstack.org/599992 | 10:35 |
| *** adrianc has quit IRC | 10:40 | |
| *** iurygregory is now known as iury|lunch | 10:41 | |
| *** mkrai has joined #openstack-ironic | 10:57 | |
| mkrai | Hi I am getting error while deploy where IPA fails to download image from Swift temp | 10:58 |
| mkrai | I see that there is no container in swift | 10:59 |
| mkrai | Can someone please help me with this? | 10:59 |
| mkrai | Here is the error http://paste.openstack.org/show/734197/ | 11:02 |
| *** iury|lunch is now known as iurygregory | 11:11 | |
| *** e0ne has joined #openstack-ironic | 11:11 | |
| openstackgerrit | Javier Peña proposed openstack/ironic master: Fix redfish test_get_system_resource_not_found test https://review.openstack.org/615833 | 11:23 |
| *** dtantsur|afk is now known as dtantsur | 11:33 | |
| dtantsur | morning/afternoon ironic | 11:33 |
| jroll | morning dtantsur and everyone else | 11:34 |
| dtantsur | etingof: mind joining #rdo? it seems like your sushy test update broke unit tests there | 11:34 |
| iurygregory | morning guys o/ | 11:34 |
| dtantsur | morning jroll and iurygregory | 11:34 |
| jroll | tonyb: TheJulia: at least one set of hardware that needed that raw commmand is from 2015? 2016? so I suspect we can't just drop it, though +1 for making it optional :) | 11:35 |
| *** bfournie has joined #openstack-ironic | 11:39 | |
| mkrai | Good morning Ironic!! | 11:40 |
| openstackgerrit | Javier Peña proposed openstack/ironic master: Fix redfish test_get_system_resource_not_found test https://review.openstack.org/615833 | 11:40 |
| mkrai | Hi I am getting error while deploy where IPA fails to download image from Swift temp URL | 11:40 |
| mkrai | Here is the error http://paste.openstack.org/show/734197/ | 11:40 |
| mkrai | dtantsur: jroll Can you please help? | 11:41 |
| jroll | mkrai: I saw your message earlier, if I could help I would :) | 11:42 |
| jroll | I've never seen a 409 from a temp url | 11:43 |
| dtantsur | ditto | 11:43 |
| dtantsur | maybe ask swift folks? | 11:43 |
| jroll | although | 11:43 |
| jroll | mkrai: you say you see no container in swift, are you using swift as your glance backend? | 11:43 |
| mkrai | jroll: ok :) | 11:43 |
| mkrai | jroll: I just check the container is there | 11:43 |
| mkrai | a container named glance | 11:43 |
| jroll | ah ok, no clue why the 409 then | 11:44 |
| mkrai | yes swift as glance backend | 11:44 |
| iurygregory | did you check glance and swift logs and if the services are running ok? | 11:45 |
| *** gkadam_ has quit IRC | 11:46 | |
| *** gkadam has joined #openstack-ironic | 11:46 | |
| mnasiadka | guys, once this gets merged: https://review.openstack.org/#/c/615781/ - can you plan for a release soon? bifrost in kolla stable/rocky is broken due to that, and we rely on released versions. | 11:49 |
| patchbot | patch 615781 - bifrost (stable/rocky) - Fix ironic configuration for new oslo messaging - 1 patch set | 11:49 |
| jroll | dtantsur: ^^ | 11:50 |
| * jroll just +2'd | 11:50 | |
| jroll | mnasiadka: I'll make sure we get that released soon, thanks | 11:50 |
| mnasiadka | jroll: once a release gets out - I'll be thanking then :) | 11:51 |
| jroll | :) | 11:52 |
| dtantsur | +A | 11:53 |
| jroll | thanks | 11:54 |
| dtantsur | jroll: please check https://review.openstack.org/#/c/615833/ | 11:58 |
| patchbot | patch 615833 - ironic - Fix redfish test_get_system_resource_not_found test - 2 patch sets | 11:58 |
| jroll | what happened to our job that installs driver-requirements? :( | 11:59 |
| dtantsur | it has never been created.. | 11:59 |
| jroll | +W | 11:59 |
| jroll | I thought we did | 11:59 |
| *** mkrai has quit IRC | 11:59 | |
| jroll | or rather, I thought pas-ha did | 11:59 |
| dtantsur | we only created a tox target | 12:00 |
| jroll | ah | 12:00 |
| jroll | also, why does my house smell like someone is cooking breakfast when I'm the only person/animal awake? >:( | 12:01 |
| *** adrianc has joined #openstack-ironic | 12:06 | |
| dtantsur | heh | 12:10 |
| * dtantsur thinks about lunch at this point | 12:13 | |
| etingof | your breakfast is being deployed, jroll o/ | 12:13 |
| openstackgerrit | Merged openstack/ironic master: Fix redfish test_get_system_resource_not_found test https://review.openstack.org/615833 | 12:13 |
| jroll | etingof: if only! | 12:13 |
| *** dnuka has joined #openstack-ironic | 12:16 | |
| dtantsur | does anyone remember from the top of their head if nova uses ironic port API? | 12:16 |
| dnuka | Good morning o/ | 12:16 |
| dnuka | Good morning dtantsur :) | 12:17 |
| jroll | dtantsur: I'm about 80% sure it does | 12:17 |
| dtantsur | morning dnuka | 12:17 |
| dnuka | Good morning iurygregory , etingof :) | 12:17 |
| dtantsur | jroll: sigh, I was hoping to omit it from ironic-proxy prototype :D | 12:17 |
| openstackgerrit | Merged openstack/bifrost stable/rocky: Fix ironic configuration for new oslo messaging https://review.openstack.org/615781 | 12:18 |
| jroll | https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1976 | 12:18 |
| iurygregory | dnuka, morning o/ sorry for the review some internal things showed up for me yesterday and today | 12:18 |
| dtantsur | oh, even portgroups.. | 12:19 |
| dnuka | iurygregory: It's alright :) | 12:19 |
| dtantsur | TheJulia: I'd like to make a bifrost rocky release per mnasiadka's request above. any objections? | 12:20 |
| dtantsur | jroll: right. I'll test with metalsmith for now then. proxying ports/port groups is something I have to think about | 12:21 |
| * jroll likes it when dtantsur does things like releases that are already on his todo list | 12:22 | |
| etingof | dtantsur, oh, I missed your ping :( | 12:22 |
| dtantsur | jroll: I can leave it up to you for sure ;) | 12:26 |
| *** e0ne has quit IRC | 12:29 | |
| jroll | dtantsur: okay :) https://review.openstack.org/615852 | 12:29 |
| patchbot | patch 615852 - releases - Release bifrost stable/rocky at 5.1.1 - 1 patch set | 12:29 |
| jroll | TheJulia: ^ when you're around, please :) | 12:29 |
| etingof | so the sushy@#rdo matter seems to be resolved | 12:39 |
| *** e0ne has joined #openstack-ironic | 12:43 | |
| dtantsur | yup | 12:53 |
| *** Emine has quit IRC | 12:59 | |
| *** cdearborn has joined #openstack-ironic | 13:05 | |
| *** Emine has joined #openstack-ironic | 13:15 | |
| *** emine__ has joined #openstack-ironic | 13:19 | |
| *** Emine has quit IRC | 13:20 | |
| *** jcoufal has joined #openstack-ironic | 13:23 | |
| *** dnuka has quit IRC | 13:25 | |
| *** rloo has joined #openstack-ironic | 13:27 | |
| sambetts|afk | dtantsur: the plan with /nodes/<uuid>/vifs was to remove the requirement on nova to talk the ports API, but currently it requires it to populate the mac addresses for config drive, I wanted to switch it to use the info (mac/binding) on the neutron port we update to populate the mac addresses and other info before I got pulled away from things | 13:32 |
| dtantsur | I see | 13:34 |
| * TheJulia wipes sleep from eyes | 13:37 | |
| TheJulia | jroll: is your house haunted? :) | 13:38 |
| TheJulia | Good morning everyone! | 13:38 |
| TheJulia | dtantsur: bifrost release ++ | 13:40 |
| dtantsur | TheJulia: jroll already requested :) | 13:40 |
| dtantsur | also g'morning | 13:40 |
| TheJulia | \o/ | 13:40 |
| * TheJulia greatly dislikes daylight savings time changes | 13:40 | |
| *** iurygregory has quit IRC | 13:41 | |
| *** iurygregory has joined #openstack-ironic | 13:42 | |
| TheJulia | I feel like I've seen mkrai's issue when the tempurl shared key does not match | 13:44 |
| jroll | I think that's typically a 401 or 403, but I could be wrong | 13:44 |
| TheJulia | I think that is only when it can decode it and the time stamps for authorization are no longer valid | 13:45 |
| TheJulia | I think | 13:46 |
| TheJulia | really, as suggested, swift logs need to be reviewed | 13:46 |
| *** zul has joined #openstack-ironic | 13:46 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: playbooks: Use --upgrade-strategy option for openSUSE distributions https://review.openstack.org/612632 | 13:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: [DNM] Add openSUSE Leap 15 jobs https://review.openstack.org/599521 | 13:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: Remove paste references on keystone deployments https://review.openstack.org/615877 | 13:47 |
| openstackgerrit | Dmitry Tantsur proposed openstack/metalsmith master: Use the Connection.baremetal adapter for ironicclient Session https://review.openstack.org/615878 | 13:47 |
| TheJulia | could I get some eyeballs on https://review.openstack.org/#/c/449425/ today? I'd like to add one or two more things to my "already merged into stein" list | 13:48 |
| patchbot | patch 449425 - ironic - Add Redfish inspect interface - 16 patch sets | 13:48 |
| * TheJulia misses the good old reproduce.sh.... | 13:50 | |
| *** sthussey has joined #openstack-ironic | 13:52 | |
| *** mkrai has joined #openstack-ironic | 13:57 | |
| *** yolanda has joined #openstack-ironic | 14:08 | |
| openstackgerrit | Nguyen Hai Truong proposed openstack/networking-generic-switch master: Add python 3.6 unit test job https://review.openstack.org/615888 | 14:10 |
| *** emine__ has quit IRC | 14:15 | |
| dtantsur | Failed to deploy: {u'priority': 100, u'interface': u'deploy', u'step': u'deploy', u'argsinfo': None} | 14:16 |
| dtantsur | WUT? | 14:16 |
| TheJulia | nothing else? | 14:18 |
| iurygregory | this just the output from ansible no? | 14:23 |
| dtantsur | nope, this is a deploy step | 14:25 |
| *** early has quit IRC | 14:25 | |
| dtantsur | and this is the full content of last_error | 14:25 |
| *** early has joined #openstack-ironic | 14:28 | |
| dtantsur | now last_error=None provision_state=deploy failed | 14:32 |
| dtantsur | with the root cause being outdated ironic-lib... | 14:33 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: playbooks: Use --upgrade-strategy option for openSUSE distributions https://review.openstack.org/612632 | 14:34 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: [DNM] Add openSUSE Leap 15 jobs https://review.openstack.org/599521 | 14:34 |
| TheJulia | dtantsur: perhaps we should revise how we update the last error on exit of the decorator? | 14:37 |
| dtantsur | yyeaaah | 14:51 |
| *** baha has joined #openstack-ironic | 14:53 | |
| dtantsur | \o/ ironic-proxy is now able to deploy/undeploy (using metalsmith, dunno about nova) | 14:54 |
| *** munimeha1 has joined #openstack-ironic | 14:57 | |
| *** rh-jelabarre has joined #openstack-ironic | 14:59 | |
| *** skazi has joined #openstack-ironic | 14:59 | |
| *** arne_wiebalck_ has quit IRC | 15:01 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 15:04 | |
| *** e0ne has quit IRC | 15:06 | |
| rloo | good morning. dtantsur, TheJulia: did we do a release of rocky or queens? (cuz of the rolling upgrade bug)? | 15:07 |
| mbeierl | dumb question time: I was doing a bit of a feature compare of different bare metal deployment tools. I just want to confirm: it is not possible to use Ironic for something like PXE botting from ISO with a kickstart/preseed file, correct? | 15:18 |
| mbeierl | *booting* | 15:18 |
| mbeierl | I'm thinking forward to edge use cases where there might not be a pre-built qcow cloud image for the bare metal application running at the edge. | 15:20 |
| dtantsur | mbeierl: I don't think it's actually possible. also I don't quite get how this limitation is linked to the Edge case, could you elaborate? | 15:28 |
| mbeierl | I am thinking of a fictitious product: it is a bare metal deployment with its own installer. It is shipped on an ISO and has the ability to set operating system parameters via a kickstart file. | 15:29 |
| *** cdearborn has quit IRC | 15:30 | |
| *** gkadam has quit IRC | 15:30 | |
| mbeierl | dtantsur: an option would be to create a qcow via virt-manager, but if this product decides to install components based on the presence of certain hardware, that would not work. | 15:31 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: playbooks: Use --upgrade-strategy option for openSUSE distributions https://review.openstack.org/612632 | 15:41 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: [DNM] Add openSUSE Leap 15 jobs https://review.openstack.org/599521 | 15:41 |
| * TheJulia sighs with devstack | 15:47 | |
| TheJulia | rloo: I don't think so, we should. | 15:47 |
| dtantsur | ++ | 15:47 |
| dtantsur | and ++ to sigh about devstack | 15:47 |
| TheJulia | :) | 15:47 |
| TheJulia | I think there is a race in start-up :\ | 15:47 |
| TheJulia | of vms | 15:48 |
| * TheJulia added some ||true to try and get past a point | 15:48 | |
| * dtantsur adds ||true everywhere | 15:49 | |
| *** priteau has quit IRC | 15:49 | |
| rloo | dtantsur: I assume you or your people :) tested that rolling upgrade fix? | 15:50 |
| dtantsur | rloo: I did not directly test it, but I think our folks did. in any case, we'll learn when they close/reopen that bug :D | 15:50 |
| rloo | dtantsur: heh. would be good if they tried it out, before we release! | 15:51 |
| openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Fix libvirt driver to handle domains by UUID https://review.openstack.org/614316 | 15:54 |
| *** arne_wiebalck_ has quit IRC | 15:54 | |
| *** mkrai has quit IRC | 15:56 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 16:02 | |
| *** hamzy_ has quit IRC | 16:19 | |
| *** gyee has joined #openstack-ironic | 16:21 | |
| *** mjturek has joined #openstack-ironic | 16:21 | |
| mjturek | tonyb: saw your mesage from yesterday.. why is it hitting IRMC modules on the ppc64le node? It should be using the IPMItool one (at least that's what we use) http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/ipmitool.py#n895 | 16:29 |
| *** arne_wiebalck_ has quit IRC | 16:31 | |
| *** mjturek has quit IRC | 16:36 | |
| *** andymccr has quit IRC | 16:36 | |
| *** mjturek has joined #openstack-ironic | 16:36 | |
| openstackgerrit | Will Szumski proposed openstack/tenks master: Wait for resources to become available https://review.openstack.org/615939 | 16:40 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: playbooks: Use --upgrade-strategy option for openSUSE distributions https://review.openstack.org/612632 | 16:41 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/bifrost master: [DNM] Add openSUSE Leap 15 jobs https://review.openstack.org/599521 | 16:41 |
| *** hamzy_ has joined #openstack-ironic | 16:44 | |
| openstackgerrit | Merged openstack/metalsmith master: Use the Connection.baremetal adapter for ironicclient Session https://review.openstack.org/615878 | 16:56 |
| openstackgerrit | Will Szumski proposed openstack/tenks master: Add an initial noop job to bootstrap zuul https://review.openstack.org/615945 | 17:05 |
| openstackgerrit | Merged openstack/tenks master: Add an initial noop job to bootstrap zuul https://review.openstack.org/615945 | 17:06 |
| openstackgerrit | Merged openstack/tenks master: Add a .gitreview file https://review.openstack.org/615218 | 17:08 |
| TheJulia | eureka! | 17:11 |
| openstackgerrit | Will Szumski proposed openstack/tenks master: Add an initial set of zuul jobs https://review.openstack.org/615224 | 17:16 |
| *** S4ren1 has quit IRC | 17:18 | |
| *** skazi has quit IRC | 17:27 | |
| *** gabys has joined #openstack-ironic | 17:29 | |
| openstackgerrit | Will Szumski proposed openstack/tenks master: Wait for resources to become available https://review.openstack.org/615939 | 17:31 |
| *** e0ne has joined #openstack-ironic | 17:32 | |
| *** gabys has quit IRC | 17:32 | |
| *** gabys has joined #openstack-ironic | 17:33 | |
| *** gabys has quit IRC | 17:37 | |
| rpioso | Good morning, ironicers | 17:55 |
| rpioso | TheJulia: o/ | 17:56 |
| rpioso | TheJulia: Recall the issue I encountered with iscsi deploy on Friday. | 17:56 |
| *** dougsz has quit IRC | 17:56 | |
| TheJulia | yes | 17:56 |
| rpioso | TheJulia: Installation of libguestfs-tools installs iscsi-initiator-utils-iscsiuio.x86_64 0:6.2.0.874-7.el7 | 17:58 |
| rpioso | I did that after having installed the tripleo undercloud via "openstack undercloud install". | 17:59 |
| TheJulia | afaik iscsi-initiator-utils depends on iscsi-initiator-utils-iscsiuio | 17:59 |
| rpioso | Right, it was installed along with iscsi-initiator-utils.x86_64 0:6.2.0.874-7.el7 | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| rpioso | After installing the undercloud, but before installing those tools, iscsiadm is not found on the undercloud's CentOS. | 18:01 |
| rpioso | Where does the ironic conductor container get it from? | 18:02 |
| rpioso | I'm thinking we may have been looking in the wrong place. | 18:02 |
| *** e0ne has quit IRC | 18:03 | |
| TheJulia | it should be built into the ironic conductor container | 18:04 |
| rpioso | How can I examine its version and command output? | 18:05 |
| *** skazi has joined #openstack-ironic | 18:05 | |
| TheJulia | afaik, you should be able to run commands directly against the container | 18:05 |
| rpioso | Such as "docker exec -ti $CONTAINER_ID_OR_NAME /bin/bash" | 18:06 |
| TheJulia | yup | 18:06 |
| rpioso | I'll look | 18:06 |
| *** dtantsur is now known as dtantsur|afk | 18:06 | |
| dtantsur|afk | o/ | 18:06 |
| TheJulia | goodnight dtantsur|afk | 18:06 |
| rpioso | dtantsur|afk: \o | 18:06 |
| *** e0ne has joined #openstack-ironic | 18:06 | |
| * TheJulia makes frowny faces at devstack | 18:08 | |
| *** adrianc has quit IRC | 18:17 | |
| rpioso | https://www.irccloud.com/pastebin/tIEhJF4R/ | 18:17 |
| *** jtomasek has quit IRC | 18:18 | |
| TheJulia | it needs to be executed as root | 18:18 |
| rpioso | https://www.irccloud.com/pastebin/cgwTR6k6/ | 18:23 |
| * rpioso needs to grab lunch before the cafeteria closes | 18:24 | |
| *** rpioso is now known as rpioso|afk | 18:24 | |
| *** e0ne has quit IRC | 18:24 | |
| *** tssurya has quit IRC | 18:26 | |
| openstackgerrit | Merged openstack/tenks master: Only set Ironic port attributes when changed https://review.openstack.org/615254 | 18:26 |
| openstackgerrit | Merged openstack/tenks master: Use extend-ignore in preference to ignore https://review.openstack.org/615252 | 18:26 |
| openstackgerrit | Merged openstack/tenks master: Fix lint warning concerning task names https://review.openstack.org/615253 | 18:26 |
| *** gyee has quit IRC | 18:34 | |
| openstackgerrit | Merged openstack/tenks master: Add an initial set of zuul jobs https://review.openstack.org/615224 | 18:39 |
| TheJulia | rpioso|afk: so interestingly enough, that doesn't match the errors that were appearing in the logs for the same command, granted there was an ip address, but it was throwing a syntax error if memory serves | 18:39 |
| *** hamzy_ is now known as hamzy | 18:53 | |
| openstackgerrit | Merged openstack/ironic master: Change BFV job to use ipxe interface https://review.openstack.org/614535 | 19:06 |
| *** brandor5 has joined #openstack-ironic | 19:08 | |
| *** serlex has quit IRC | 19:11 | |
| *** cxc99 has quit IRC | 19:12 | |
| *** rpioso|afk is now known as rpioso | 19:15 | |
| hwoarang | TheJulia: hello. about a month ago we were looking at some opensuse150 problems with bifrost on https://review.openstack.org/#/c/599521/ The VM log is now available there. My understanding is that is_whole_disk_image: True is the problem (http://logs.openstack.org/21/599521/28/check/bifrost-integration-tinyipa-opensuse-150/88b2eb2/logs/baremetal.txt) | 19:16 |
| patchbot | patch 599521 - bifrost - [DNM] Add openSUSE Leap 15 jobs - 28 patch sets | 19:16 |
| hwoarang | TheJulia: i have no other debug or investigation info to share at this point :/ if you have any idea please let me know | 19:17 |
| TheJulia | hwoarang: agent deploy logs or maybe the console log could help... | 19:17 |
| hwoarang | the VM console is here http://logs.openstack.org/21/599521/28/check/bifrost-integration-tinyipa-opensuse-150/88b2eb2/logs/testvm1_console.log | 19:18 |
| TheJulia | yup, I see the issue | 19:18 |
| TheJulia | looks like the kernel is now wanting a root=/dev/ram0 entry | 19:19 |
| * TheJulia looks to see where things fail | 19:21 | |
| hwoarang | looks like as if no disk is attached | 19:21 |
| *** cxc99 has joined #openstack-ironic | 19:22 | |
| rpioso | TheJulia: The output just above was immediately following the installation of the undercloud. libguestfs-tools had not been installed. | 19:23 |
| rpioso | Playing forward to the snapshot following installation of libguestfs-tools: | 19:23 |
| rpioso | https://www.irccloud.com/pastebin/zVRnjgpW/ | 19:23 |
| rpioso | Look familiar? | 19:23 |
| TheJulia | wow | 19:24 |
| TheJulia | so installing libguestfs-tools breaks it? | 19:24 |
| TheJulia | or replaces the iscsiadm binary? | 19:24 |
| TheJulia | or ?replaces? it | 19:24 |
| rpioso | Or something | 19:24 |
| TheJulia | wtf | 19:24 |
| rpioso | It was installed on the base CentOS 7.5. | 19:24 |
| TheJulia | hwoarang: still looking, looks like it did inspect | 19:24 |
| TheJulia | rpioso: give me a few and I'll fire up my centos machine and take a look | 19:25 |
| rpioso | Prior to the installation, the base didn't have iscsiadm. | 19:25 |
| rpioso | I did a minimal installation of CentOS 7.5. | 19:25 |
| TheJulia | oh jeeze, that seems really bad :( | 19:26 |
| *** brandor5 has left #openstack-ironic | 19:29 | |
| TheJulia | hwoarang: you win an award for the most cryptic failure of the week | 19:31 |
| TheJulia | :( | 19:31 |
| hwoarang | that makes me so proud :) | 19:31 |
| hwoarang | how can you tell if the image was flushed to the disk? | 19:32 |
| TheJulia | That is a great question | 19:32 |
| TheJulia | The logs should indicate the deploy was done | 19:32 |
| TheJulia | hmmm | 19:35 |
| hwoarang | maybe we should also capture the logs during the initial disk flushing by the ironic agent | 19:37 |
| TheJulia | possibly | 19:38 |
| TheJulia | looks like the ramdisk does see the inspection ramdisk contents... | 19:38 |
| TheJulia | err | 19:38 |
| hwoarang | anyway not getting anywhere right now. i am just curious why the bootlog does not say anything about a 'vda' disk compared to suse 42 or centos logs | 19:38 |
| TheJulia | so discoveyr looks broken | 19:38 |
| TheJulia | sanboot failing seems... problematic | 19:39 |
| hwoarang | i suspect i might be missing a package to make virtio disks work | 19:39 |
| TheJulia | _or_ virtio disks no longer work with ipxe | 19:39 |
| hwoarang | o_O | 19:39 |
| *** gabys has joined #openstack-ironic | 19:40 | |
| hwoarang | ok i will leave it here for today. I will try more things tomorrow :/ thank you for your help so far! | 19:40 |
| *** mjturek has quit IRC | 19:41 | |
| *** mjturek has joined #openstack-ironic | 19:43 | |
| TheJulia | okay | 19:45 |
| TheJulia | I'm looking, this is intruiging | 19:45 |
| tonyb | mjturek: Yeah I used the wrong link I opened a Story with the logs and the correct link. | 19:45 |
| mjturek | tonyb: cool, I'll take a look | 19:46 |
| tonyb | mjturek, TheJulia, jroll: I plan on pushing a change today to make it optional | 19:46 |
| TheJulia | tonyb: most excellent | 19:46 |
| mjturek | will review | 19:46 |
| jroll | 👍 | 19:46 |
| tonyb | :) | 19:46 |
| *** dsneddon has joined #openstack-ironic | 19:46 | |
| * tonyb is blocked on $other_things so it'll be nice to do somethign upstream for a bit :) | 19:47 | |
| *** luzC has quit IRC | 19:47 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Sudo is needed to copy logs https://review.openstack.org/615983 | 19:52 |
| *** luzC has joined #openstack-ironic | 19:52 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 19:54 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Mark discovery ramdisk to have an initrd https://review.openstack.org/615984 | 19:55 |
| *** ptx0 has quit IRC | 19:55 | |
| *** gabys has quit IRC | 19:56 | |
| *** gabys has joined #openstack-ironic | 19:56 | |
| *** gabys has quit IRC | 20:01 | |
| *** ptx0 has joined #openstack-ironic | 20:02 | |
| TheJulia | hwoarang: I approved two of your changes to land, and I've got another change and your change rebased on top. It might fix things, it won't solve the entire issue, but it should get things moving and from there we might want to explore alternatives to virtio if the rom loading doesn't grok the disks..... | 20:09 |
| TheJulia | rpioso: md5sum that iscsiadm file for me | 20:18 |
| TheJulia | same behavior after installing libguestfs-tools on my centos machine | 20:19 |
| jroll | blurgh, this is fun: https://storyboard.openstack.org/#!/story/2004279 | 20:21 |
| *** jtomasek has joined #openstack-ironic | 20:21 | |
| TheJulia | ugh | 20:21 |
| * TheJulia steps away to go check the pizza dough before pivoting back to the frustration of devstack | 20:22 | |
| jroll | mmmm pizza. | 20:22 |
| jroll | that reminds me, this is awesome: https://polls.pizza/ | 20:23 |
| TheJulia | That is awesome | 20:24 |
| *** jtomasek has quit IRC | 20:24 | |
| openstackgerrit | Merged openstack/bifrost master: Remove paste references on keystone deployments https://review.openstack.org/615877 | 20:25 |
| openstackgerrit | Merged openstack/bifrost master: playbooks: Use --upgrade-strategy option for openSUSE distributions https://review.openstack.org/612632 | 20:26 |
| rpioso | TheJulia: Post installation of libguestfs-tools: | 20:26 |
| rpioso | https://www.irccloud.com/pastebin/wXhu2LXU/ | 20:26 |
| rpioso | I'm glad you reproduced it :) | 20:27 |
| TheJulia | no, I didn't reproduce it | 20:28 |
| TheJulia | I get connection refused errors which is what we woudl expect | 20:28 |
| TheJulia | rpioso: `strace -f -F -o strace.out iscsiadm -m discovery -t st -p 127.0.0.2:3260` and then email me strace.out | 20:30 |
| TheJulia | something doesn't align, but I _DO_ have the exact same iscsiadm binary | 20:30 |
| TheJulia | you may need to install strace | 20:30 |
| rpioso | TheJulia: I had started the revert to before libguestfs-tools process. I'll grab the md5sum value there. | 20:32 |
| rpioso | Would strace be helpful at that point in time? | 20:32 |
| rpioso | I assume strace should be run within the ironic-conductor container. | 20:32 |
| rpioso | TheJulia: Before installationof libguestfs-tools: | 20:34 |
| rpioso | https://www.irccloud.com/pastebin/4RFOcEAm/ | 20:35 |
| *** mjturek has quit IRC | 20:38 | |
| *** mmethot_ is now known as mmethot | 20:38 | |
| *** mjturek has joined #openstack-ironic | 20:39 | |
| *** mjturek has quit IRC | 20:46 | |
| *** munimeha1 has quit IRC | 20:46 | |
| TheJulia | hmm rpioso that makes me think that they are unrelated | 20:49 |
| TheJulia | since the contents and the behavior of the container really hsouldn't be impacted by the host os on this level | 20:50 |
| *** mjturek has joined #openstack-ironic | 20:51 | |
| *** mjturek has quit IRC | 20:54 | |
| rpioso | TheJulia: In the container, the version of iscsiadm and the md5sum are identical before and after installation of libguestfs-tools. | 21:06 |
| TheJulia | indeed, and your installing it outside of the container, which should not impact container contents | 21:06 |
| TheJulia | maybe something is getting pulled in dependency wise that is changing the appearance of the underlying host holding the undercloud controller container | 21:07 |
| *** arne_wiebalck_ has quit IRC | 21:07 | |
| TheJulia | installing did seem to activate the iscsi tcp transport class, but that could have just been my first exec of iscsiadm | 21:08 |
| rpioso | Perhaps one of more of its dependencies were upgraded by the installation of those tools. | 21:09 |
| rpioso | “maybe something is getting pulled in ...” That’s what I was thinking, too :) | 21:11 |
| *** jcoufal has quit IRC | 21:25 | |
| *** mjturek has joined #openstack-ironic | 21:26 | |
| *** pcaruana has quit IRC | 21:31 | |
| *** mjturek has quit IRC | 21:31 | |
| *** dsneddon_ has quit IRC | 21:43 | |
| *** mjturek has joined #openstack-ironic | 21:44 | |
| *** hamzy has quit IRC | 21:45 | |
| rpioso | TheJulia: I sent you the strace from before and after. | 21:53 |
| jroll | dtantsur|afk: I feel like we have had conversations about things this would solve in the past. in case you hadn't seen it: https://docs.openstack.org/oslo.config/latest/reference/locations.html | 21:54 |
| *** baha has quit IRC | 22:10 | |
| *** mjturek has quit IRC | 22:12 | |
| *** rh-jelabarre has quit IRC | 22:14 | |
| * TheJulia just bangs her head into the dsk | 22:17 | |
| TheJulia | desk | 22:17 |
| *** lenka has quit IRC | 22:22 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: [DNM] Add openSUSE Leap 15 jobs https://review.openstack.org/599521 | 22:34 |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Set default_boot_option for bifrost https://review.openstack.org/616021 | 22:34 |
| *** lenka has joined #openstack-ironic | 22:50 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic master: Allow disabling instance image cache https://review.openstack.org/616031 | 23:31 |
| jroll | ^ still needs tests and a release note, but that's my thoughts on that bug | 23:32 |
| *** lenka has quit IRC | 23:40 | |
| TheJulia | jroll: ack ack | 23:50 |
| TheJulia | rpioso: sent you an email with my findings | 23:50 |
| TheJulia | tl;dr the reply over the unix socket to iscsid is different. Why... no clue | 23:50 |
| *** MattMan_ has quit IRC | 23:51 | |
| *** MattMan_ has joined #openstack-ironic | 23:52 | |
| jroll | TheJulia: no rush btw, we'll be cherry-picking it downstream | 23:52 |
| rpioso | TheJulia: Thank you so much. I'll need to dig in further. | 23:54 |
| *** rpioso is now known as rpioso|afk | 23:54 | |
| rpioso|afk | o/ | 23:54 |
| TheJulia | rpioso|afk: thanks, there really is not much else I can do right now since I can't reproduce it locally... and libguestfs-tools has an insane number of dependencies | 23:55 |
| TheJulia | rpioso|afk: let me know what you find | 23:55 |
| TheJulia | Anyway... it is 4pm local. I've been at it since sometime between 5-6 AM this morning. I think it is time to go do things like laundry and maybe start dinner | 23:56 |
| * jroll passes TheJulia a pint | 23:57 | |
| jroll | have a good night! | 23:57 |
| * TheJulia really wonders where the day went | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!