| *** cdearborn has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack/ironic stable/stein: Fix capabilities passed as string in agent prepare https://review.openstack.org/649268 | 00:15 |
|---|---|---|
| openstackgerrit | Merged openstack/ironic stable/rocky: Fix capabilities passed as string in agent prepare https://review.openstack.org/648166 | 00:50 |
| *** dsneddon has quit IRC | 01:06 | |
| *** rh-jelabarre has quit IRC | 01:15 | |
| *** tiendc has joined #openstack-ironic | 01:19 | |
| *** dsneddon has joined #openstack-ironic | 01:37 | |
| *** dsneddon has quit IRC | 01:43 | |
| *** whoami-rajat has joined #openstack-ironic | 02:03 | |
| *** dsneddon has joined #openstack-ironic | 02:14 | |
| *** dsneddon has quit IRC | 02:20 | |
| *** rloo has quit IRC | 02:26 | |
| openstackgerrit | francotseng proposed openstack/ironic-ui stable/pike: Normalize operation messages into capital case https://review.openstack.org/649776 | 02:27 |
| openstackgerrit | francotseng proposed openstack/ironic-ui stable/ocata: Normalize operation messages into capital case https://review.openstack.org/649778 | 02:34 |
| openstackgerrit | Tony Breeds proposed openstack/python-ironicclient master: Uncap jsonschema https://review.openstack.org/649797 | 03:00 |
| *** hwoarang has quit IRC | 03:01 | |
| *** hwoarang has joined #openstack-ironic | 03:06 | |
| *** sthussey has quit IRC | 03:14 | |
| *** samueldmq has quit IRC | 03:16 | |
| *** dsneddon has joined #openstack-ironic | 04:00 | |
| *** stendulker has joined #openstack-ironic | 04:52 | |
| *** pcaruana has joined #openstack-ironic | 04:58 | |
| *** dsneddon has quit IRC | 05:06 | |
| *** e0ne has joined #openstack-ironic | 05:12 | |
| *** e0ne has quit IRC | 05:18 | |
| *** hwoarang has quit IRC | 06:39 | |
| *** hwoarang has joined #openstack-ironic | 06:41 | |
| *** e0ne has joined #openstack-ironic | 06:47 | |
| rpittau|afk | good morning ironic! o/ | 06:48 |
| *** rpittau|afk is now known as rpittau | 06:48 | |
| arne_wiebalck | good morning, ironic! | 06:58 |
| rpittau | hey arne_wiebalck :) | 07:03 |
| *** dsneddon has joined #openstack-ironic | 07:03 | |
| arne_wiebalck | rpittau: o/ how is the snow situation at your place? | 07:03 |
| * arne_wiebalck was surprised by 20cm of fresh snow this morning | 07:03 | |
| rpittau | no snow but cold, -1 this morning :/ | 07:08 |
| rpittau | arne_wiebalck: I saw a couple of beautiful pictures from your colleagues :) | 07:12 |
| *** dsneddon has quit IRC | 07:16 | |
| *** dims has quit IRC | 07:22 | |
| *** dims has joined #openstack-ironic | 07:24 | |
| *** dims has quit IRC | 07:31 | |
| *** dims has joined #openstack-ironic | 07:34 | |
| *** jistr is now known as jistr|afk | 07:35 | |
| *** dsneddon has joined #openstack-ironic | 07:43 | |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Ansible module: fix partition_configdrive.sh file https://review.openstack.org/649663 | 07:44 |
| arne_wiebalck | rpittau: quite an impressive change compared to yesterday :) | 07:46 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: [WIP/DNM] Testing ubuntu bionic job https://review.openstack.org/649314 | 07:46 |
| *** dsneddon has quit IRC | 07:48 | |
| *** tssurya has joined #openstack-ironic | 07:56 | |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Ansible module fix: stream_url https://review.openstack.org/649664 | 07:57 |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Ansible module: fix clean error handling https://review.openstack.org/649665 | 08:01 |
| mgoddard | Morning all | 08:09 |
| openstackgerrit | Jerry Wang proposed openstack/ironic master: implement patch code for Enton Power SNMP driver Story: 2005317 Task: 30337 https://review.openstack.org/649913 | 08:09 |
| mgoddard | FYI I'll be out of the office today and tomorrow | 08:09 |
| *** iurygregory has joined #openstack-ironic | 08:10 | |
| iurygregory | morning all | 08:23 |
| *** dtantsur|afk is now known as dtantsur | 08:25 | |
| dtantsur | morning ironic | 08:25 |
| *** gkadam has joined #openstack-ironic | 08:27 | |
| *** e0ne has quit IRC | 08:30 | |
| *** andrein has joined #openstack-ironic | 08:35 | |
| *** e0ne has joined #openstack-ironic | 08:40 | |
| *** mkrai has joined #openstack-ironic | 08:42 | |
| *** e0ne has quit IRC | 08:42 | |
| mkrai | Hi, where can I find the Denver PTG planning etherpad? | 08:43 |
| *** dsneddon has joined #openstack-ironic | 08:44 | |
| iurygregory | mkrai, Train etherpad ? | 08:47 |
| mkrai | iurygregory: Yes | 08:47 |
| iurygregory | mkrai, fyi https://etherpad.openstack.org/p/DEN-train-ironic-brainstorming | 08:47 |
| mkrai | iurygregory: Thank you :) | 08:48 |
| iurygregory | yw | 08:48 |
| *** dsneddon has quit IRC | 08:49 | |
| *** e0ne has joined #openstack-ironic | 08:50 | |
| *** S4ren has joined #openstack-ironic | 08:50 | |
| *** S4ren has quit IRC | 08:51 | |
| openstackgerrit | Merged openstack/ironic-ui stable/stein: Normalize operation messages into capital case https://review.openstack.org/649585 | 08:56 |
| openstackgerrit | Merged openstack/ironic-ui stable/rocky: Normalize operation messages into capital case https://review.openstack.org/649586 | 09:00 |
| openstackgerrit | Merged openstack/ironic-ui stable/queens: Normalize operation messages into capital case https://review.openstack.org/649587 | 09:00 |
| openstackgerrit | Merged openstack/ironic-ui stable/pike: Normalize operation messages into capital case https://review.openstack.org/649776 | 09:00 |
| openstackgerrit | Merged openstack/ironic-ui stable/ocata: Normalize operation messages into capital case https://review.openstack.org/649778 | 09:00 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: Improve iptables test info collection https://review.openstack.org/649946 | 09:02 |
| *** dsneddon has joined #openstack-ironic | 09:03 | |
| *** dsneddon has quit IRC | 09:08 | |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Ansible module fix: stream_url https://review.openstack.org/649664 | 09:18 |
| dtantsur | rpioso: hi! the Dell CI seems to be red for a few days, are you aware of it? | 09:25 |
| *** diga has joined #openstack-ironic | 09:43 | |
| *** e0ne has quit IRC | 09:44 | |
| openstackgerrit | Digambar proposed openstack/ironic stable/rocky: [WIP] Remove nic method dependencies from dracclient==1.5.0 from stable releases https://review.openstack.org/648360 | 09:45 |
| diga | Hello Ironic | 09:47 |
| diga | Good Morning | 09:47 |
| dtantsur | morning diga | 09:49 |
| rpittau | hi diga :) | 09:51 |
| iurygregory | morning diga | 09:54 |
| *** e0ne has joined #openstack-ironic | 09:54 | |
| diga | dtantsur: rpittau iurygregory :) | 10:00 |
| *** dtantsur is now known as dtantsur|brb | 10:07 | |
| * dtantsur|brb -> short walk | 10:07 | |
| *** Qianbiao has joined #openstack-ironic | 10:12 | |
| Qianbiao | hello ironic | 10:12 |
| *** Qianbiao is now known as qianbiao|opensta | 10:13 | |
| *** qianbiao|opensta is now known as qianbiao | 10:14 | |
| *** jistr|afk is now known as jistr | 10:23 | |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: WIP: Add LED management to ManagementInterface https://review.openstack.org/649675 | 10:24 |
| *** derekh has joined #openstack-ironic | 10:35 | |
| *** diga has quit IRC | 10:37 | |
| *** mkrai has quit IRC | 10:50 | |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: [WIP/DNM] Testing ubuntu bionic job https://review.openstack.org/649314 | 10:53 |
| *** hjensas has quit IRC | 11:01 | |
| *** stendulker has quit IRC | 11:13 | |
| qianbiao | Hello, if I want to update a document file on stein branch, any gerrit document for that? | 11:20 |
| *** andrein has quit IRC | 11:27 | |
| *** bfournie has quit IRC | 11:30 | |
| jangutter | qianbiao: can it be cherry-picked from master or is it specific for stein? | 11:35 |
| ijpascual | eandersson: we are doing dhcp control when using ironic standalone | 11:35 |
| qianbiao | <jangutter> it could be cherry-picked. | 11:36 |
| qianbiao | a very small fix | 11:36 |
| qianbiao | document | 11:36 |
| jangutter | qianbiao: I recently posted this to master - https://review.openstack.org/#/c/649295/ | 11:36 |
| patchbot | patch 649295 - ironic - Slightly rephrase note in tenant networking docs (MERGED) - 2 patch sets | 11:37 |
| jroll | morning everyone | 11:37 |
| jangutter | qianbiao: generally it's a good idea to fix it there first, let it merge, then cherry-pick it to the stable branch. Patches to the stable branch need to be very well vetted. | 11:38 |
| qianbiao | ok, thanks. | 11:39 |
| qianbiao | jangutter thanks | 11:40 |
| *** samueldmq has joined #openstack-ironic | 11:40 | |
| *** tiendc has quit IRC | 11:45 | |
| *** andrein has joined #openstack-ironic | 11:45 | |
| *** e0ne has quit IRC | 11:50 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:54 | |
| jangutter | so, in the continuing saga of the disappearing baremetal network ports, I've finally finished the minor upgrade on the Queens stack and I'm still seeing them disappear :-( | 11:56 |
| *** dtantsur|brb is now known as dtantsur | 11:57 | |
| dtantsur | morning jroll | 11:57 |
| *** hjensas has joined #openstack-ironic | 11:58 | |
| *** trown|outtypewww is now known as trown | 12:01 | |
| dtantsur | etingof: did you know about https://github.com/openstack/pyghmi/tree/master/pyghmi/redfish Oo | 12:03 |
| dtantsur | also virtualbmc: https://github.com/openstack/pyghmi/blob/master/pyghmi/cmd/virshbmc.py O__o | 12:04 |
| *** bfournie has joined #openstack-ironic | 12:06 | |
| etingof | dtantsur, I encountered the virshbmc, but not redfish | 12:40 |
| *** rloo has joined #openstack-ironic | 12:42 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: [WIP] Switch to use exception from ironic-lib https://review.openstack.org/649303 | 12:47 |
| *** e0ne has joined #openstack-ironic | 13:22 | |
| *** cdearborn has joined #openstack-ironic | 13:27 | |
| jangutter | quick question, does anyone know of bad interactions with l2population without the baremetal mechanism driver? | 13:43 |
| jangutter | that's my current working theory. | 13:43 |
| *** mjturek has joined #openstack-ironic | 13:53 | |
| *** altlogbot_0 has quit IRC | 13:53 | |
| TheJulia | I do not, but with your disappearing port case that does kind of make it suspect I guess | 13:57 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-lib master: Refactor exceptions https://review.openstack.org/649277 | 13:58 |
| *** e0ne has quit IRC | 13:59 | |
| TheJulia | jangutter: maybe the neutron folks might be worth asking because it kind of sounds incompatible with the vnic_type of baermetal | 13:59 |
| TheJulia | baremetal | 13:59 |
| * TheJulia tries to wake up | 14:00 | |
| jangutter | TheJulia: thanks, I'm zapping it and retesting, soooo many blind alleys. | 14:01 |
| TheJulia | dtantsur: dell ci needs to change out their base OS, it is still trying to use xenial | 14:01 |
| openstackgerrit | QianBiao Ng proposed openstack/ironic master: doc: update ibmc driver support servers document https://review.openstack.org/650015 | 14:02 |
| *** baha has joined #openstack-ironic | 14:04 | |
| *** e0ne has joined #openstack-ironic | 14:05 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: doc: update ibmc driver support servers document https://review.openstack.org/650015 | 14:09 |
| TheJulia | ugh, so at least one ci job timed out on calling neutron for a port bind | 14:13 |
| TheJulia | woot... neutron has no record of the post | 14:21 |
| TheJulia | oh wait.. hmm | 14:21 |
| * TheJulia looked at the wrong time | 14:21 | |
| TheJulia | http://logs.openstack.org/28/648728/3/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/502b3f5/controller/logs/screen-q-svc.txt.gz#_Apr_04_08_13_03_798014 <-- 37 seconds and we only waited 30 seconds :( | 14:24 |
| rpittau | lol | 14:24 |
| * TheJulia looks into how we can fix this to make CI happy | 14:24 | |
| dtantsur | s/sleep 30/sleep 60/ :) | 14:26 |
| dtantsur | at least for now until we get the events right | 14:26 |
| arne_wiebalck | Ops question 1: is there a way to keep a node powered on after cleaning failed? | 14:33 |
| arne_wiebalck | Context is initial burn-in/stress testing where we might want to have a look at the node in the state when the test failed, i.e. before the reboot. | 14:34 |
| arne_wiebalck | Ideally, the node moves to clean_failed (so we find it), but it stays on. | 14:35 |
| *** qianbiao has quit IRC | 14:36 | |
| TheJulia | dtantsur: nah, we don't grok timeouts or handle them nicely | 14:37 |
| TheJulia | keystone timeout != http request timeout for neutron | 14:37 |
| TheJulia | :( | 14:37 |
| TheJulia | we need to scrub our deprecated config options out | 14:45 |
| *** e0ne has quit IRC | 14:46 | |
| *** e0ne has joined #openstack-ironic | 14:52 | |
| *** e0ne has quit IRC | 14:57 | |
| dtantsur | indeed | 15:01 |
| *** dsneddon has joined #openstack-ironic | 15:07 | |
| *** dsneddon has quit IRC | 15:09 | |
| *** dsneddon has joined #openstack-ironic | 15:11 | |
| arne_wiebalck | Ops question 2: We need to retire the first nodes from ironic. Is there any support for "moving nodes aside" after auto/manual cleaning beyond setting maintenance with a sensible reason (like "to be retired")? | 15:21 |
| arne_wiebalck | Context is that these nodes need to be picked up by some other tools to remove them from various databases and to add them to tools which track the remaining steps (such as removing cables, unmount them from their racks etc.), so we need some way to easily identify them. | 15:23 |
| dtantsur | arne_wiebalck: I can only think about maintenance mode.. | 15:30 |
| arne_wiebalck | dtantsur: yeah, didn't see anything else | 15:37 |
| arne_wiebalck | dtantsur: what about 'Ops question 1' ^? | 15:37 |
| dtantsur | oh, missed it | 15:37 |
| dtantsur | arne_wiebalck: I did not even know we power off nodes on failed cleaning | 15:38 |
| dtantsur | it may not be the best thing to do if e.g. we screwed up firmware upgrades | 15:38 |
| arne_wiebalck | We're trying to expand ironic more into our procurement workflows and this came up as a desired feature. Sounds sensible to me. | 15:38 |
| dtantsur | I'd say we should keep it on if it was on at the moment of failure | 15:39 |
| arne_wiebalck | hmm | 15:39 |
| dtantsur | (and power it off on moving out of this state) | 15:39 |
| arne_wiebalck | right | 15:39 |
| arne_wiebalck | maybe that's the way it is and I remember it wrongly ... | 15:39 |
| * arne_wiebalck was pretty sure nodes are off when cleaning fails | 15:40 | |
| dtantsur | maybe, then we probably should fix it | 15:42 |
| arne_wiebalck | dtantsur: let me check ... | 15:42 |
| jangutter | OK, so my l2population theory went bust. I still have no idea why the ports are getting deleted. I'm upping log level on Nova now. | 15:42 |
| dtantsur | e.g. doing power off after unsuccessful firmware update may kill the last chance to recover.. | 15:42 |
| jangutter | _something_ is directly telling Neutron to delete the port. | 15:42 |
| arne_wiebalck | dtantsur: I agree | 15:43 |
| dtantsur | jangutter: maybe try to talk to folks on #openstack-nova as well? just in case somebody recognizes it? | 15:43 |
| arne_wiebalck | dtantsur: all our nodes in clean_failed have power off | 15:43 |
| *** gyee has joined #openstack-ironic | 15:43 | |
| jangutter | dtantsur: will check when I have a bit more info. | 15:43 |
| arne_wiebalck | dtantsur: don't think we actively configured this somewhere | 15:43 |
| arne_wiebalck | dtantsur: is there an easy way to make cleaning fail, so that I can reproduce on devstack? | 15:45 |
| arne_wiebalck | dtantsur: to be sure it's not our setup somehow | 15:45 |
| dtantsur | hmmm | 15:46 |
| *** e0ne has joined #openstack-ironic | 15:46 | |
| jangutter | dtantsur: Hmmm, nova 'Received event network-vif-deleted-' <--- I see this in the logs _after_ the neutron DELETE. | 15:46 |
| dtantsur | it's probably just a notification about the deletion | 15:47 |
| *** e0ne has quit IRC | 15:48 | |
| *** e0ne has joined #openstack-ironic | 15:48 | |
| *** e0ne has quit IRC | 15:48 | |
| *** tssurya has quit IRC | 15:52 | |
| *** hjensas has quit IRC | 15:56 | |
| jangutter | dtantsur: is the ironic-conductor the only process that deletes VIFs? | 15:57 |
| jangutter | dtantsur: -- in Ironic I mean... | 15:58 |
| rpioso | dtantsur: Yes, we're well aware of it. rajinir is working to resolve it -- http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2019-04-01.log.html#t2019-04-01T15:59:18 | 15:59 |
| rpioso | dtantsur: Thanks for checking. | 16:01 |
| rpittau | arne_wiebalck: did you actually verify that the nodes are powered off and not just reported in power off status ? | 16:02 |
| arne_wiebalck | rpittau: no .. I trusted ironic :) | 16:02 |
| arne_wiebalck | rpittau: let me check ... | 16:02 |
| rpittau | arne_wiebalck: yes please doublecheck | 16:02 |
| *** e0ne has joined #openstack-ironic | 16:05 | |
| arne_wiebalck | rpittau: the state comes from ironic's power sync, no | 16:05 |
| arne_wiebalck | rpittau: which is run every 5 mins in our case | 16:05 |
| arne_wiebalck | rpittau: let me check a node | 16:05 |
| arne_wiebalck | rpittau: power seems off | 16:08 |
| arne_wiebalck | rpittau: well; I did the same as ironic (ask via IPMI) and got the same result | 16:09 |
| rpittau | arne_wiebalck: ok, I had a suspect it might be a bug | 16:10 |
| arne_wiebalck | rpittau: when developing the s/w RAID code I had quite some failed cleaning and followed things on the console, so I'm pretty confident the node is off after | 16:11 |
| *** baha has quit IRC | 16:11 | |
| arne_wiebalck | rpittau: will try to reproduce on devstack to be sure we're not looking at something local | 16:11 |
| *** e0ne has quit IRC | 16:12 | |
| rpittau | arne_wiebalck: yep, as dtantsur said before, powering off after cleaning fails can cause trouble in certain cases :) | 16:14 |
| arne_wiebalck | rpittau: dtantsur: when aborting while cleaning, the node moves to clean failed and is powered off | 16:14 |
| dtantsur | jangutter: yes, in ironic yes | 16:18 |
| NobodyCam | Good Morning Ironic'ers | 16:18 |
| dtantsur | morning NobodyCam | 16:18 |
| NobodyCam | :) | 16:18 |
| rpittau | hi NobodyCam :) | 16:19 |
| jangutter | OK, I'm seeing this: Unbinding instance ports from node 04077f5f-6b2e-4533-b820-0c31babbcd02 | 16:19 |
| NobodyCam | morning rpittau :) | 16:19 |
| jangutter | Then immediately RPC vif_detach called for the node 04077f5f-6b2e-4533-b820-0c31babbcd02 with vif_id 9eea5079-4ca9-41e7-aa55-4b3b5ca4164b vif_detach | 16:19 |
| jangutter | Is this the "smocking" gun? | 16:19 |
| dtantsur | jangutter: vif_detach itself does not delete ports. well, IIRC | 16:20 |
| dtantsur | however I wonder what called it | 16:20 |
| dtantsur | usually VIFs are detached when an instance is undeployed | 16:20 |
| rpittau | arne_wiebalck: dtantsur: https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/deploy_utils.py#L938 | 16:24 |
| jangutter | dtantsur: this just after deploy finished -> literally within the same second that ironic sent the reboot command. | 16:24 |
| * dtantsur wonders if Nova gets confused about the node state and tries to kill it | 16:25 | |
| dtantsur | jangutter: what's the status of the instances in nova? does it look alright? | 16:26 |
| jangutter | dtantsur: Nova says ACTIVE with no ports on it. | 16:26 |
| arne_wiebalck | rpittau: this https://docs.openstack.org/ironic/latest/admin/cleaning.html confirms what dtantsur suggested (no power off while cleaning) | 16:26 |
| arne_wiebalck | rpittau: Troubleshooting section | 16:26 |
| jangutter | dtantsur: this is spooooooooky. | 16:26 |
| dtantsur | it actually is | 16:27 |
| jangutter | dtantsur: machine booted up fine, doesn't get dhcp obviously, and I can attach another baremetal port to it if I want to. | 16:27 |
| dtantsur | so, nothing intriguing in nova-compute logs? | 16:29 |
| rpittau | arne_wiebalck: yeah, I was looking at the code and if you abort cleaning it will call tear_down_cleaning and power off | 16:29 |
| jangutter | dtantsur: almost everything I see happens _after_ the neutron DELETE rpc log. | 16:31 |
| * dtantsur suggests exorcism | 16:32 | |
| *** yolanda has quit IRC | 16:33 | |
| arne_wiebalck | rpittau: providing a non-existing clean step moves the node to clean_failed and powers it off as well | 16:37 |
| *** baha has joined #openstack-ironic | 16:39 | |
| rpittau | arne_wiebalck: yeah, if you look at the conductor manager, I think it depends on how cleaning_error_handler is called and if tear_down_cleaning is set to True or False | 16:40 |
| arne_wiebalck | rpittau: | 16:40 |
| arne_wiebalck | yes | 16:40 |
| arne_wiebalck | I was checking exactly this: seems it's never set to False | 16:40 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add a request_timeout to neutron https://review.openstack.org/650072 | 16:40 |
| rpittau | well, rarely | 16:40 |
| rpittau | but yeah, abort seems to be an example https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L1502 | 16:43 |
| arne_wiebalck | true, there are some occurences | 16:43 |
| rpittau | I have to run, literally :) | 16:44 |
| rpittau | see you tomorrow! o/ | 16:44 |
| *** rpittau is now known as rpittau|afk | 16:44 | |
| arne_wiebalck | bye rpittau|afk o/ | 16:47 |
| openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Use HTTP code 307 for redirects https://review.openstack.org/650074 | 16:48 |
| arne_wiebalck | From what I see, https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L1405-L1410 calls https://github.com/openstack/ironic/blob/master/ironic/conductor/utils.py#L384 relying on the default value for tear_down_cleaning (True). So, I'd say a failed cleaning step will shut the node off. | 16:55 |
| *** dtantsur is now known as dtantsur|afk | 17:00 | |
| dtantsur|afk | \o | 17:00 |
| arne_wiebalck | bye, everyone o/ | 17:04 |
| *** hjensas has joined #openstack-ironic | 17:05 | |
| *** ijw has joined #openstack-ironic | 17:07 | |
| jangutter | OK, more info on the case of the spooky disappearing VIFs. I closely watched the nova compute logs and the following happened: Nova got the network-vif-deleted event from Neutron while ironic was still in deploying state! | 17:10 |
| *** gkadam has quit IRC | 17:11 | |
| *** dsneddon has quit IRC | 17:12 | |
| *** andrein has quit IRC | 17:13 | |
| *** v12aml has quit IRC | 17:15 | |
| *** dsneddon has joined #openstack-ironic | 17:15 | |
| *** derekh has quit IRC | 17:18 | |
| *** v12aml has joined #openstack-ironic | 17:19 | |
| *** dsneddon has quit IRC | 17:21 | |
| *** dsneddon has joined #openstack-ironic | 17:30 | |
| TheJulia | wut... | 17:36 |
| jangutter | I believe I might have tracked down the culprit. And it was..... me. | 17:40 |
| *** dsneddon has quit IRC | 17:40 | |
| jangutter | Let me verify before I shame myself into oblivion. | 17:40 |
| jangutter | (hint, we've been running nodepool on the cluster) | 17:40 |
| *** dsneddon has joined #openstack-ironic | 17:40 | |
| *** hjensas has quit IRC | 17:41 | |
| TheJulia | jangutter: no shaming yourself. Not allowed for those are the rules I just made up :) | 17:43 |
| TheJulia | it happens | 17:43 |
| *** gmann is now known as gmann_afk | 17:44 | |
| *** dsneddon has quit IRC | 17:46 | |
| *** dsneddon has joined #openstack-ironic | 17:47 | |
| jangutter | TheJulia: I was literally thinking horses when it looks like it's zebras. Still checking, but it really looks like an external app is doing the nasty! | 17:47 |
| *** derekh has joined #openstack-ironic | 18:09 | |
| *** ijw has quit IRC | 18:09 | |
| jangutter | TheJulia: the current working theory is that setting min-ready for nodepool labels causes nodepool to agressively try to clean the network. | 18:10 |
| *** derekh has quit IRC | 18:10 | |
| *** andrein has joined #openstack-ironic | 18:15 | |
| *** MattMan_1 has quit IRC | 18:17 | |
| *** MattMan_1 has joined #openstack-ironic | 18:17 | |
| *** andrein has quit IRC | 18:24 | |
| *** hjensas has joined #openstack-ironic | 18:27 | |
| TheJulia | eek :( | 18:39 |
| TheJulia | I would go with bunnies for I believe they may be evil. | 18:40 |
| jangutter | TheJulia: bunnies hold no candle to cats. | 18:42 |
| *** rh-jelabarre has quit IRC | 18:43 | |
| *** rh-jelabarre has joined #openstack-ironic | 18:44 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Fix exception generation errors https://review.openstack.org/644590 | 18:57 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/stein: ipmi: Ignore sensor debug data https://review.openstack.org/650107 | 18:58 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/rocky: ipmi: Ignore sensor debug data https://review.openstack.org/650109 | 18:59 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/queens: ipmi: Ignore sensor debug data https://review.openstack.org/650110 | 18:59 |
| *** mmethot has joined #openstack-ironic | 19:03 | |
| jangutter | This. https://github.com/openstack-infra/nodepool/blob/master/nodepool/driver/openstack/provider.py#L537 <--- this was what's happening to me. | 19:03 |
| jangutter | Since the baremetal ports are always DOWN, periodically nodepool thinks "hey, they must've leaked". | 19:04 |
| jangutter | Also: https://github.com/openstack-infra/nodepool/commit/0cf8144e8ce403e6c8365f17eeb774421a7bd094 <---- I think the three minutes there coincides weirdly with spawning. | 19:06 |
| openstackgerrit | Richard G. Pioso proposed openstack/ironic stable/rocky: Revert "Fix OOB introspection to use pxe_enabled flag in idrac driver" https://review.openstack.org/650116 | 19:23 |
| *** gmann_afk is now known as gmann | 19:49 | |
| eandersson | Are hardware managers just for cleaning? or can they be used at provisioning as well? | 20:00 |
| eandersson | or rather, can you add additional provisioning steps | 20:02 |
| *** dsneddon has quit IRC | 20:14 | |
| *** dsneddon has joined #openstack-ironic | 20:17 | |
| openstackgerrit | Richard G. Pioso proposed openstack/ironic stable/queens: Revert "Fix OOB introspection to use pxe_enabled flag in idrac driver" https://review.openstack.org/650134 | 20:19 |
| *** pcaruana has quit IRC | 20:29 | |
| *** pcaruana has joined #openstack-ironic | 20:30 | |
| *** dsneddon has quit IRC | 20:31 | |
| *** dsneddon has joined #openstack-ironic | 20:32 | |
| *** andrein has joined #openstack-ironic | 20:35 | |
| TheJulia | eandersson: anything really, there is an included ?pair? for infiband adapters | 20:41 |
| TheJulia | jangutter: oh, wow, well networking-baremetal will put them up, they just are not magically forever up | 20:42 |
| *** bfournie has quit IRC | 20:51 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Split ibmc power/reboot classes https://review.openstack.org/650147 | 20:56 |
| TheJulia | ^^ complete shot in the dark | 20:56 |
| TheJulia | jangutter: By chance, have you inquired with the zuul folks? | 20:58 |
| *** jrist has quit IRC | 20:59 | |
| *** jrist has joined #openstack-ironic | 21:01 | |
| *** baha has quit IRC | 21:07 | |
| *** mjturek has quit IRC | 21:08 | |
| TheJulia | jangutter: btw, yes, cats are indeed evil. I have two who can confirm this :) | 21:14 |
| *** Sundar has joined #openstack-ironic | 21:35 | |
| *** pcaruana has quit IRC | 21:51 | |
| Sundar | TheJulia: Please ping me when you have the time | 21:56 |
| *** bnemec has quit IRC | 22:01 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/pike: Revert "Fix OOB introspection to use pxe_enabled flag in idrac driver" https://review.openstack.org/643952 | 22:19 |
| rpioso | TheJulia: I'm working on that ^^^ | 22:20 |
| TheJulia | I just fixed it | 22:20 |
| TheJulia | I had a older revert up | 22:20 |
| TheJulia | well | 22:20 |
| TheJulia | abandoned | 22:20 |
| TheJulia | Sundar: What can I do for you? | 22:20 |
| rpioso | Wasted a bit of time. | 22:21 |
| TheJulia | :( | 22:21 |
| * TheJulia feels bad | 22:21 | |
| * rpioso thinks you would feel better if you abandoned it :) | 22:22 | |
| * TheJulia smiles | 22:22 | |
| *** rcernin has joined #openstack-ironic | 22:23 | |
| Sundar | TheJulia: I think there is increased interest in accelerators with bare metal. Could we explore that topic at the PTG? | 22:28 |
| *** whoami-rajat has quit IRC | 22:32 | |
| TheJulia | Sundar: Accelerators how so? | 22:34 |
| Sundar | TheJulia: The use case is around deploying FPGAs and possibly GPUs with Ironic, using Cyborg. | 22:35 |
| TheJulia | deploying how so? | 22:36 |
| TheJulia | If they are in the physical machine that is deployed, would they not already be there? | 22:36 |
| Sundar | TheJulia: Cyborg represents devices as nested resource providers inside a compute node RP. Ironic already represents a compute node as a RP and also, IIUC, inventroies of a RC | 22:37 |
| Sundar | Yes, so in a compute cluster where some nodes have FPGAs or GPUs, and others don't, the user may want to get an instance with a specific FPGA type | 22:37 |
| Sundar | Even if all nodes have them, they will differ in their properties: model, which function is implemented etc. | 22:38 |
| Sundar | SO, the operator want to provide a ironic flavor that includes the properties of accelerators that come with the node | 22:38 |
| Sundar | *may want | 22:39 |
| Sundar | What do you think? | 22:39 |
| TheJulia | so in that case, those would just be traits within the resource class that is offered | 22:40 |
| TheJulia | user definable... | 22:40 |
| TheJulia | well, operator | 22:40 |
| TheJulia | Are there traits that Cyborg has driven to be standardized upon that ironic could list for placement to use? | 22:41 |
| Sundar | We are working out an arrangement with Nova/Placement to discover devices, created nested RPs for them, and specify their properties as traits on those RPs. | 22:42 |
| Sundar | The operator need not define or apply traits. But s/he can define a flavor which references those traits, this ensuring the VM (in Nova's case) is scheduled correctly | 22:42 |
| TheJulia | so the mechanics are different though, operators define the traits for their hardware | 22:43 |
| TheJulia | and since operators agreed they can't agree, we didn't push anything on them | 22:43 |
| TheJulia | It sounds like it would be good to be involved in that discussion, and maybe worthwhile to consider ironic's hardware introspection capabilities to record standardized traits upon hardware inspection if applicable to at least make it a little easier for operators | 22:44 |
| Sundar | For accelerators, not sure if operators would want to do that manually, though. For one, there are lots of them, and secondly FPGAs and ilk could be hard to handle | 22:44 |
| Sundar | TheJulia: Yes, agreed. We do have some standard traits -- like device model, function id/name, etc. -- which any operator should be able to use | 22:45 |
| TheJulia | Sundar: max ?50? traits I believe. Is there going to be a nova cross proejct session | 22:45 |
| TheJulia | ? | 22:45 |
| Sundar | TheJulia: I have been told that the number of traits is not a limit at all. Anyways, we are talking of a handful of traits. | 22:45 |
| Sundar | We are planning a separate Nova-Cyborg session. For Ironic, I was hoping that we could do a session where we invite somebody from placement to keep me honest | 22:46 |
| Sundar | Since there are effectively only 2 days at the PTG (many folks are off Sat, including me), this could be on Friday | 22:46 |
| Sundar | Nova's cross-project sessions are apparently all on Fri, so only some may come to Cyborg/Ironic | 22:47 |
| TheJulia | Hmm, I'm not sure how that is going to work | 22:47 |
| Sundar | TheJulia: Isnt this more of a Placement thing, in terms of RPs, RCs and traits? | 22:48 |
| TheJulia | It is, and if standards can be set then I guess we could follow-up and discuss it further as long as there are solid notes | 22:49 |
| TheJulia | I've not made a schedule for ironic yet, but it seems we are going to be crunched on time :( | 22:49 |
| Sundar | Understood. It could be a 30-min or 1-hour session, and we'll try to get some specific proposal defined in an etherpad before we get to the PTG | 22:51 |
| TheJulia | 30 minutes could work, 1 hour is going to be unschedulable | 22:53 |
| TheJulia | can you add a entry to our planning/discussion etherpad for the ptg and we'll try and figure out interest on Monday? | 22:54 |
| TheJulia | https://etherpad.openstack.org/p/DEN-train-ironic-brainstorming | 22:54 |
| Sundar | Sure. Then let's do most of the heavy lifting before we get there. | 22:54 |
| Sundar | Thank you | 22:55 |
| Sundar | Do you have a meeting on Monday that I should attend? | 22:55 |
| TheJulia | You can, but I sense it is largely going to be trying to get people to express interest on items on the etherpad and then I'll try to write up a schedule next week | 22:55 |
| Sundar | I see. | 22:56 |
| *** andrein has quit IRC | 23:13 | |
| *** keekz has joined #openstack-ironic | 23:14 | |
| *** dsneddon has quit IRC | 23:20 | |
| *** dsneddon has joined #openstack-ironic | 23:22 | |
| *** rloo has quit IRC | 23:26 | |
| *** dsneddon has quit IRC | 23:27 | |
| *** dsneddon has joined #openstack-ironic | 23:30 | |
| openstackgerrit | Richard G. Pioso proposed openstack/ironic stable/pike: Revert "Fix OOB introspection to use pxe_enabled flag in idrac driver" https://review.openstack.org/650164 | 23:30 |
| *** Sundar has quit IRC | 23:39 | |
| *** dsneddon has quit IRC | 23:40 | |
| *** dsneddon has joined #openstack-ironic | 23:47 | |
| *** dsneddon has quit IRC | 23:52 | |
| *** dsneddon has joined #openstack-ironic | 23:53 | |
| *** dsneddon has quit IRC | 23:57 | |
| *** gyee has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!