| *** vishwana_ has joined #openstack-ironic | 00:02 | |
| *** vishwana_ has quit IRC | 00:03 | |
| *** vishwanathj has quit IRC | 00:03 | |
| *** vishwanathj has joined #openstack-ironic | 00:03 | |
| *** karimb has quit IRC | 00:05 | |
| *** jayc_ has quit IRC | 00:06 | |
| *** suro-patz has quit IRC | 00:07 | |
| *** suro-patz has joined #openstack-ironic | 00:07 | |
| *** suro-patz has quit IRC | 00:10 | |
| *** vishwana_ has joined #openstack-ironic | 00:12 | |
| *** garthb has quit IRC | 00:14 | |
| *** vishwanathj has quit IRC | 00:15 | |
| *** daemontool has quit IRC | 00:27 | |
| *** mtanino_ has joined #openstack-ironic | 00:28 | |
| *** mtanino has quit IRC | 00:29 | |
| *** raildo is now known as raildo-afk | 00:33 | |
| *** moshele has joined #openstack-ironic | 00:41 | |
| *** Marga__ has quit IRC | 00:42 | |
| *** Marga_ has joined #openstack-ironic | 00:43 | |
| *** suro-patz has joined #openstack-ironic | 00:44 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ironic-inspector: use openstack cli instead of keystone cli https://review.openstack.org/307523 | 00:47 |
|---|---|---|
| *** Sukhdev has quit IRC | 00:50 | |
| *** suro-patz has quit IRC | 00:50 | |
| *** baoli has joined #openstack-ironic | 00:57 | |
| *** moshele has quit IRC | 00:57 | |
| *** hoangcx has joined #openstack-ironic | 00:58 | |
| *** moshele has joined #openstack-ironic | 01:00 | |
| *** suro-patz has joined #openstack-ironic | 01:01 | |
| *** vmud213 has joined #openstack-ironic | 01:02 | |
| *** mtanino has joined #openstack-ironic | 01:03 | |
| *** mtanino_ has quit IRC | 01:04 | |
| *** vmud213 has quit IRC | 01:04 | |
| *** vinaym213 has joined #openstack-ironic | 01:04 | |
| *** vinay50muddu has joined #openstack-ironic | 01:07 | |
| *** vinaym213 has quit IRC | 01:07 | |
| *** suro-patz has quit IRC | 01:10 | |
| *** ayoung has joined #openstack-ironic | 01:12 | |
| *** suro-patz has joined #openstack-ironic | 01:17 | |
| *** moshele has quit IRC | 01:19 | |
| *** moshele has joined #openstack-ironic | 01:19 | |
| *** suro-patz has quit IRC | 01:22 | |
| *** suro-patz has joined #openstack-ironic | 01:23 | |
| *** moshele has quit IRC | 01:35 | |
| *** moshele has joined #openstack-ironic | 01:36 | |
| *** Haomeng has joined #openstack-ironic | 01:37 | |
| *** suro-patz has quit IRC | 01:44 | |
| *** suro-patz has joined #openstack-ironic | 01:44 | |
| *** vinaym213 has joined #openstack-ironic | 01:46 | |
| *** vinay50muddu has quit IRC | 01:46 | |
| *** vinay50muddu has joined #openstack-ironic | 01:48 | |
| *** vinaym213 has quit IRC | 01:48 | |
| *** vinaym213 has joined #openstack-ironic | 01:52 | |
| *** vinay50muddu has quit IRC | 01:52 | |
| *** moshele has quit IRC | 01:56 | |
| *** Marga_ has quit IRC | 01:56 | |
| *** vmud213 has joined #openstack-ironic | 01:59 | |
| *** vinaym213 has quit IRC | 01:59 | |
| *** vmud213 has quit IRC | 02:03 | |
| *** vmud213 has joined #openstack-ironic | 02:03 | |
| *** praneshp has joined #openstack-ironic | 02:07 | |
| *** jaybeale has quit IRC | 02:07 | |
| *** keedya has quit IRC | 02:10 | |
| *** vmud213 has quit IRC | 02:11 | |
| *** vmud213 has joined #openstack-ironic | 02:11 | |
| *** vinaym213 has joined #openstack-ironic | 02:15 | |
| *** vmud213 has quit IRC | 02:15 | |
| *** suro-patz has quit IRC | 02:16 | |
| *** vinay50muddu has joined #openstack-ironic | 02:19 | |
| jroll | NobodyCam: those paramiko bugs are interesting indeed | 02:19 |
| *** ChrisAusten has joined #openstack-ironic | 02:19 | |
| *** vinaym213 has quit IRC | 02:19 | |
| *** vishwana_ has quit IRC | 02:19 | |
| *** vinay50muddu has quit IRC | 02:19 | |
| *** vinaym213 has joined #openstack-ironic | 02:19 | |
| *** vishwanathj has joined #openstack-ironic | 02:21 | |
| jroll | I added them to our LP bug btw | 02:26 |
| jroll | I'll probably be on late-ish tomorrow morning, feel free to ping me on hangouts if you need something | 02:27 |
| *** vinay50muddu has joined #openstack-ironic | 02:27 | |
| *** vinaym213 has quit IRC | 02:27 | |
| *** mtanino has quit IRC | 02:29 | |
| *** vinaym213 has joined #openstack-ironic | 02:33 | |
| *** vinay50muddu has quit IRC | 02:33 | |
| *** vinay50muddu has joined #openstack-ironic | 02:38 | |
| *** vinaym213 has quit IRC | 02:38 | |
| *** vinay50muddu has quit IRC | 02:48 | |
| openstackgerrit | Zhenguo Niu proposed openstack/ironic: Devstack: Change to use 'ovs-vsctl get port tag' https://review.openstack.org/307543 | 02:48 |
| *** sivaramakrishna has joined #openstack-ironic | 03:08 | |
| *** ijw has quit IRC | 03:13 | |
| *** amotoki has quit IRC | 03:20 | |
| openstackgerrit | Tan Lin proposed openstack/ironic: Follow-up patch of d744823639fb4382c5d631c696a8f779d291f9de https://review.openstack.org/306209 | 03:30 |
| openstackgerrit | Tan Lin proposed openstack/ironic: Follow-up patch of ec98b330b463d202be5e63b30427614b8dcf7062 https://review.openstack.org/306209 | 03:32 |
| *** baoli has quit IRC | 03:32 | |
| openstackgerrit | Tan Lin proposed openstack/ironic: Follow-up patch of 0607226fc4b4bc3c9e1738dc3f78ed99e5d4f13d https://review.openstack.org/306209 | 03:34 |
| *** morgabra has quit IRC | 03:39 | |
| *** morgabra has joined #openstack-ironic | 03:40 | |
| *** persia has quit IRC | 03:40 | |
| *** intr1nsic has quit IRC | 03:41 | |
| *** links has joined #openstack-ironic | 03:41 | |
| *** amotoki has joined #openstack-ironic | 03:41 | |
| *** Marga_ has joined #openstack-ironic | 03:43 | |
| *** ianw has quit IRC | 03:43 | |
| *** vishwanathj has quit IRC | 03:45 | |
| *** Marga_ has quit IRC | 03:47 | |
| *** persia has joined #openstack-ironic | 03:50 | |
| *** Nisha has joined #openstack-ironic | 03:56 | |
| *** vmud213 has joined #openstack-ironic | 03:56 | |
| *** Marga_ has joined #openstack-ironic | 03:57 | |
| *** Marga_ has quit IRC | 03:59 | |
| *** Marga_ has joined #openstack-ironic | 03:59 | |
| *** jaybeale has joined #openstack-ironic | 04:02 | |
| *** amotoki has quit IRC | 04:03 | |
| *** Marga_ is now known as fragatina | 04:05 | |
| *** suro-patz has joined #openstack-ironic | 04:05 | |
| *** intr1nsic has joined #openstack-ironic | 04:07 | |
| *** ianw has joined #openstack-ironic | 04:08 | |
| *** ayoung has quit IRC | 04:11 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ironic-inspector: use openstack cli instead of keystone cli https://review.openstack.org/307523 | 04:11 |
| *** praneshp has quit IRC | 04:16 | |
| *** intr1nsic has quit IRC | 04:27 | |
| *** ianw has quit IRC | 04:27 | |
| *** jaybeale has quit IRC | 04:30 | |
| *** amotoki has joined #openstack-ironic | 04:33 | |
| *** ianw has joined #openstack-ironic | 04:33 | |
| *** intr1nsic has joined #openstack-ironic | 04:33 | |
| openstackgerrit | ZhiQiang Fan proposed openstack/ironic-inspector: use openstack cli instead of keystone cli https://review.openstack.org/307523 | 04:37 |
| *** garthb has joined #openstack-ironic | 04:40 | |
| *** suro-patz has quit IRC | 04:40 | |
| *** ChrisAusten has quit IRC | 04:51 | |
| openstackgerrit | Debayan Ray proposed openstack/ironic: Add note on prerequisite of 'rpm' file extraction https://review.openstack.org/306247 | 05:07 |
| *** rcernin has joined #openstack-ironic | 05:10 | |
| *** chlong has quit IRC | 05:16 | |
| *** Sukhdev has joined #openstack-ironic | 05:30 | |
| *** garthb has quit IRC | 05:35 | |
| *** ChubYann has quit IRC | 05:37 | |
| *** e0ne has joined #openstack-ironic | 05:54 | |
| *** Nisha has quit IRC | 06:04 | |
| *** e0ne has quit IRC | 06:05 | |
| *** e0ne has joined #openstack-ironic | 06:10 | |
| *** deray has joined #openstack-ironic | 06:13 | |
| *** moshele has joined #openstack-ironic | 06:18 | |
| *** Sukhdev has quit IRC | 06:21 | |
| *** amotoki has quit IRC | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic: Imported Translations from Zanata https://review.openstack.org/307575 | 06:28 |
| *** e0ne has quit IRC | 06:30 | |
| *** e0ne has joined #openstack-ironic | 06:31 | |
| *** e0ne has quit IRC | 06:47 | |
| *** daemontool has joined #openstack-ironic | 06:55 | |
| *** hoangcx has quit IRC | 06:55 | |
| *** hoangcx has joined #openstack-ironic | 06:57 | |
| *** chopmann has joined #openstack-ironic | 07:00 | |
| *** Haomeng has quit IRC | 07:03 | |
| *** Haomeng has joined #openstack-ironic | 07:04 | |
| *** amotoki has joined #openstack-ironic | 07:06 | |
| *** pcaruana has joined #openstack-ironic | 07:16 | |
| *** amotoki_ has joined #openstack-ironic | 07:20 | |
| *** amotoki has quit IRC | 07:22 | |
| *** pcaruana has quit IRC | 07:23 | |
| *** tesseract has joined #openstack-ironic | 07:24 | |
| *** tesseract is now known as Guest22945 | 07:24 | |
| *** oberezovskyi has joined #openstack-ironic | 07:31 | |
| oberezovskyi | Hello! Do you know how to add info about new out-of-tree ironic driver to https://wiki.openstack.org/wiki/Ironic/Drivers page? | 07:33 |
| Haomeng | oberezovskyi: just login with your openid and edit it:) | 07:41 |
| *** hoangcx has quit IRC | 07:43 | |
| lucasagomes | morning all | 07:47 |
| Haomeng | lucasagomes: morning:) | 07:48 |
| *** ifarkas has joined #openstack-ironic | 07:49 | |
| *** hoangcx has joined #openstack-ironic | 07:50 | |
| *** stendulker has joined #openstack-ironic | 07:57 | |
| *** mbound has joined #openstack-ironic | 07:58 | |
| sergek | morning Ironicers! | 07:58 |
| *** test has joined #openstack-ironic | 08:01 | |
| *** yuikotakada has joined #openstack-ironic | 08:02 | |
| *** yuikotak_ has joined #openstack-ironic | 08:02 | |
| *** test is now known as Guest11946 | 08:02 | |
| *** yuikotak_ has quit IRC | 08:04 | |
| *** amotoki_ has quit IRC | 08:06 | |
| openstackgerrit | Yuriy Zveryanskyy proposed openstack/ironic-specs: Ansible deploy driver https://review.openstack.org/241946 | 08:07 |
| *** e0ne has joined #openstack-ironic | 08:13 | |
| alineb | morning all! | 08:15 |
| *** jistr has joined #openstack-ironic | 08:16 | |
| *** MattMan has quit IRC | 08:18 | |
| *** MattMan has joined #openstack-ironic | 08:18 | |
| yuikotakada | hi, ironic | 08:21 |
| *** mkovacik_ has joined #openstack-ironic | 08:30 | |
| lucasagomes | Haomeng, sergek alineb yuikotakada morning | 08:34 |
| yuikotakada | lucasagomes, morning :) | 08:35 |
| *** ohamada has joined #openstack-ironic | 08:40 | |
| *** kromanenko has joined #openstack-ironic | 08:41 | |
| *** mgould has joined #openstack-ironic | 08:47 | |
| *** derekh has joined #openstack-ironic | 08:49 | |
| *** athomas has joined #openstack-ironic | 08:50 | |
| *** Guest11946 has quit IRC | 08:50 | |
| *** lucas2 has joined #openstack-ironic | 08:54 | |
| *** lucas2 has quit IRC | 08:54 | |
| mgould | morning Ironic! | 08:56 |
| vmud213 | morning all | 08:56 |
| mkovacik_ | mgould, morning! | 08:56 |
| *** electrofelix has joined #openstack-ironic | 08:56 | |
| mkovacik_ | morning all! :) | 08:57 |
| *** lucasagomes_ has joined #openstack-ironic | 08:57 | |
| mgould | mkovacik_, vmud213 morning! | 08:57 |
| *** lucasagomes_ has quit IRC | 08:59 | |
| vmud213 | mgould o/ | 08:59 |
| *** lucasagomes_ has joined #openstack-ironic | 08:59 | |
| *** lucasagomes_ has quit IRC | 09:00 | |
| *** moshele has quit IRC | 09:05 | |
| *** lucasagomes_ has joined #openstack-ironic | 09:07 | |
| *** lucasagomes_ has quit IRC | 09:08 | |
| *** karimb has joined #openstack-ironic | 09:11 | |
| *** moshele has joined #openstack-ironic | 09:12 | |
| *** lucasagomes_ has joined #openstack-ironic | 09:13 | |
| *** lucasagomes has quit IRC | 09:13 | |
| vdrok | morning ironic! | 09:16 |
| sambetts|afk | Morning all | 09:16 |
| *** sambetts|afk is now known as sambetts | 09:17 | |
| yuikotakada | vmud213, vdrok, sambetts, morning :) | 09:17 |
| vdrok | morning yuikotakada :) | 09:17 |
| *** amotoki has joined #openstack-ironic | 09:17 | |
| vdrok | and morning sambetts | 09:17 |
| *** lucasagomes_ has quit IRC | 09:18 | |
| *** lucasagomes has joined #openstack-ironic | 09:19 | |
| vmud213 | morning yuikotakada,vdrok | 09:22 |
| vdrok | morning vmud213 | 09:23 |
| openstackgerrit | Tan Lin proposed openstack/ironic-specs: Support online upgrades https://review.openstack.org/299245 | 09:25 |
| sambetts | o/ vdrok yuikotakada | 09:26 |
| vdrok | NobodyCam: yep, issue 721 seems like the problem described here - http://stackoverflow.com/questions/443387/why-does-paramiko-hang-if-you-use-it-while-loading-a-module | 09:27 |
| vdrok | but in our case, it hangs even when run from python/ipython prompt | 09:28 |
| sambetts | vdrok: but in the tempest code its not run at module init time right? | 09:29 |
| sambetts | lucasagomes: is the gate still down? | 09:32 |
| sambetts | I assume so | 09:32 |
| lucasagomes | sambetts, apparently so :-( | 09:32 |
| lucasagomes | vdrok, btw, to run tempest locally I just need to deploy devstack and then run ./run_tempest.sh from the tempest project? I wonder how I can run a specific test | 09:33 |
| lucasagomes | the one is failing | 09:33 |
| lucasagomes | that is failing* | 09:34 |
| * lucasagomes googles | 09:34 | |
| sambetts | lucasagomes: the gate runs "tox -e all-plugin -- ironic" inside the tempest directory, to run a specifc test you just replace ironic with ironic.<the test name> | 09:34 |
| sambetts | its a regex | 09:35 |
| lucasagomes | sambetts, nice! Thanks a lot | 09:35 |
| yuikotakada | lucasagomes, you can check wheter plugin is enabled with tempest list-plugins | 09:35 |
| lucasagomes | yuikotakada, awesome! | ironic_tests | ironic_tempest_plugin.plugin:IronicTempestPlugin | | 09:36 |
| yuikotakada | lucasagomes, :) | 09:36 |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| dtantsur | Morning Ironic, hi lucasagomes, sambetts, yuikotakada, vdrok, mgould, alineb and everyone else :) | 09:41 |
| yuikotakada | dtantsur, morning :) | 09:41 |
| sambetts | o/ dtantsur | 09:41 |
| lucasagomes | dtantsur, hello there, good morning | 09:42 |
| mgould | vdrok sambetts yuikotakada lucasagomes dtantsur morning! | 09:42 |
| dtantsur | looking forward to the summit folks? I already feel like I'm flying tomorrow :D | 09:42 |
| yuikotakada | mgould, morning :) | 09:43 |
| yuikotakada | dtantsur, sure :D | 09:43 |
| *** amotoki has quit IRC | 09:43 | |
| dtantsur | good :) | 09:43 |
| sambetts | dtantsur: looking forward to the summit, not looking forward to the traveling | 09:44 |
| vdrok | morning dtantsur ! | 09:45 |
| vdrok | lucasagomes: if you want to just run it, yep, tox -eall-plugin -- ironic_tempest_plugin.tests.scenario.test_baremetal_basic_ops | 09:46 |
| vdrok | if you'll say just ironic, it will run basic ops 2 times - once from plugin and once from tempest tree | 09:46 |
| dtantsur | sambetts, +100 | 09:46 |
| vdrok | lucasagomes: if you want to use pdb, then you can use smth like ./run_tempest.sh -N -d -t -- ironic_tempest_plugin.tests.scenario.test_baremetal_basic_ops | 09:47 |
| sambetts | vdrok: how does the gate stop running it twice then? because it uses just "ironic" | 09:47 |
| vdrok | sambetts: honestly I'm not sure :) that's what happened to me | 09:48 |
| vdrok | sambetts: or, actually, iirc i was running just with test_baremetal_basic_ops | 09:48 |
| sambetts | :/ | 09:48 |
| vdrok | maybe that makes the difference | 09:48 |
| sambetts | ah, that might have been why, the ones in the tempest tree are all under a "baremetal" folder, so ironic shouldn't trigger it, but that ^ would | 09:49 |
| vdrok | lucasagomes: tho to reproduce it, you don't need to run tempest, you can just do nova boot and instantiate RemoteClient as it is done in verify_partitions | 09:50 |
| vdrok | in eg ipython | 09:50 |
| lucasagomes | sambetts, +1 about the travelling | 09:54 |
| lucasagomes | vdrok, awesome! Thanks for all the tips | 09:54 |
| vdrok | np | 09:55 |
| lucasagomes | vdrok, yeah I was curious about running tempest, I think it would be good to document it. I will submit a patch to the docs once I get it done | 09:55 |
| yuikotakada | lucasgomes, there is README in https://github.com/openstack/ironic/blob/master/ironic_tempest_plugin/README.rst , but can nobody find it?? | 09:57 |
| evgenyl | Hi ironic team, just for your information, on upcoming summit (on Thursday) we will have a track about Fuel & Ironic integration, you are welcome to participate https://etherpad.openstack.org/p/fuel-newton-summit-planning | 09:59 |
| lucasagomes | yuikotakada, nice, I was thinking about having something like that in our documentation | 10:00 |
| lucasagomes | right after the devstack run | 10:00 |
| lucasagomes | (developer quick-start) | 10:00 |
| yuikotakada | lucasagomes, it will be helpful! | 10:00 |
| *** amotoki has joined #openstack-ironic | 10:00 | |
| sambetts | lucasagomes: +1 I had to manually dig through the devstack-gate script to work it out | 10:01 |
| lucasagomes | yeah, will put a simple patch up to the docs soonish | 10:01 |
| *** ndipanov has quit IRC | 10:14 | |
| sambetts | vdrok, lucasagomes: I see someone has sent an email out on the ML where they are seeing SSH hang also | 10:14 |
| lucasagomes | :-/ | 10:14 |
| *** amotoki has quit IRC | 10:14 | |
| * lucasagomes refresh his mailbox | 10:14 | |
| sambetts | the email was sent out on Saturday | 10:15 |
| sambetts | on [openstack-dev][tempest] | 10:15 |
| sambetts | there doesn't look to have been any response | 10:15 |
| *** ndipanov has joined #openstack-ironic | 10:18 | |
| lucasagomes | I see :-/ | 10:18 |
| sambetts | unfortunatly they didn't provide much info | 10:19 |
| mgould | lucasagomes, having that documented would be enormously helpful | 10:24 |
| *** mgould has quit IRC | 10:24 | |
| *** electrofelix has quit IRC | 10:24 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: Document how to run the tempest tests https://review.openstack.org/307695 | 10:29 |
| lucasagomes | cool, first stab at it ^ I kept it simple. Lemme know if we should extend it in the reviews | 10:29 |
| *** mgould has joined #openstack-ironic | 10:29 | |
| lucasagomes | mgould, https://review.openstack.org/#/c/307695 first stab at it | 10:30 |
| mgould | lucasagomes++ | 10:30 |
| lucasagomes | it's simple, I can enhance it as I go | 10:31 |
| *** electrofelix has joined #openstack-ironic | 10:34 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: Document how to run the tempest tests https://review.openstack.org/307695 | 10:36 |
| *** thrash|g0ne is now known as thrash | 10:37 | |
| *** hoangcx has quit IRC | 10:39 | |
| *** sivaramakrishna has quit IRC | 10:42 | |
| openstackgerrit | Shivanand Tendulker proposed openstack/ironic-specs: Group management of Ironic nodes https://review.openstack.org/235344 | 10:45 |
| dtantsur | evgenyl, hey, that's great! this is 9:00, right? lemme check my schedule | 10:48 |
| dtantsur | no conflicts for me, I'll be there | 10:49 |
| dtantsur | other inspector/ironic folks filling to talk about Fuel-Ironic relationship? jroll? | 10:50 |
| dtantsur | aarefiev, ^^^ | 10:50 |
| aarefiev | I'll be there | 10:52 |
| sambetts | sounds interesting if there isn't any Ironic sessions coliding with it | 10:53 |
| dtantsur | I didn't find any | 10:54 |
| lucasagomes | vdrok, is it my understanding or exec_command never closes the connection? Making the following exec_command to hang? | 10:54 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: DO NOT MERGE https://review.openstack.org/307705 | 10:54 |
| lucasagomes | tests ^ | 10:54 |
| *** stendulker has quit IRC | 11:01 | |
| vdrok | lucasagomes: not sure, I don't see ssh.close() there, but in the gate the very first one fails. Also if run manually, sometimes 3-4 in a row are executed fine, but then it hangs | 11:01 |
| vdrok | lucasagomes: the file-like object seems to be closed tho - https://github.com/paramiko/paramiko/blob/master/paramiko/util.py#L317-L322 | 11:02 |
| lucasagomes | vdrok, yeah noticed that :-/ | 11:02 |
| evgenyl | dtantsur: correct Thursday 9:00. | 11:02 |
| vdrok | paramiko channel is inherited from that ClosingContextManager | 11:02 |
| sambetts | vdrok, lucasagomes : I'd love to know what changed on the 13th thats triggered this :/ | 11:09 |
| sambetts | vdrok, lucasagomes: related? http://stackoverflow.com/questions/16811581/python-paramiko-hangs-when-trying-to-execute-commands-over-ssh | 11:11 |
| lucasagomes | sambetts, no idea what changed either :-/ | 11:12 |
| sambetts | hmmm --> https://github.com/paramiko/paramiko/issues/109 | 11:12 |
| vdrok | sambetts: I tried to google that too yesterday, that particular case seems to be caused by http://stackoverflow.com/questions/443387/why-does-paramiko-hang-if-you-use-it-while-loading-a-module | 11:13 |
| vdrok | but in our case it's all inside functions | 11:13 |
| sambetts | yeah :/ | 11:18 |
| sambetts | lucasagomes, vdrok: http://paste.openstack.org/show/494591 this is a diff of the installed python packages between a passing test run on April 14th, and a test run that failed yesterday because of this bug | 11:31 |
| lucasagomes | vdrok, I can confirm it's the polling bits that is hanging. I rewrote it without select.poll and it seems to work | 11:32 |
| sambetts | I see a new version of testtools, could that be related? | 11:32 |
| lucasagomes | vdrok, sambetts https://review.openstack.org/307720 | 11:33 |
| sambetts | I guess thats one solution :-P | 11:34 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: DO NOT MERGE test gate with fix in tempest https://review.openstack.org/307705 | 11:35 |
| lucasagomes | added a depends-on ^ let's see... | 11:35 |
| *** mtanino has joined #openstack-ironic | 11:38 | |
| vdrok | I've inserted some prints here - https://github.com/paramiko/paramiko/blob/master/paramiko/buffered_pipe.py#L105 and https://github.com/paramiko/paramiko/blob/master/paramiko/buffered_pipe.py#L135, and it's trying to acquire that lock continuously, no matter select is used or if I patch select to not contain poll | 11:39 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: DO NOT MERGE test gate with fix in tempest https://review.openstack.org/307705 | 11:41 |
| lucasagomes | I will grab some quick food and be back to it | 11:47 |
| *** lucasagomes is now known as lucas-hungry | 11:47 | |
| sambetts | vdrok, lucasagomes: I'm just going through comparing a passed run and failed run on the same patch, and in the failed patch I see http://logs.openstack.org/42/306942/1/check/gate-tempest-dsvm-ironic-pxe_ssh/3018940/logs/tempest.txt.gz#_2016-04-18_07_38_09_469' | 11:48 |
| *** karimb has quit IRC | 11:49 | |
| *** karimb has joined #openstack-ironic | 11:49 | |
| TheJulia | Good morning everyone | 11:51 |
| sambetts | Morning TheJulia | 11:51 |
| *** Goneri has quit IRC | 11:51 | |
| *** karimb_ has joined #openstack-ironic | 11:52 | |
| *** mgould has quit IRC | 11:53 | |
| *** Guest22945 is now known as tesseract | 11:54 | |
| *** karimb has quit IRC | 11:54 | |
| *** tesseract is now known as Guest73397 | 11:54 | |
| *** karimb_ has quit IRC | 11:54 | |
| TheJulia | sambetts: looking through those, I've kind of attributed that to the machine booting up | 11:55 |
| *** dprince has joined #openstack-ironic | 11:55 | |
| vdrok | morning TheJulia | 11:55 |
| *** raildo-afk is now known as raildo | 11:58 | |
| vdrok | sambetts: testtools release does not seem to contain anything significant | 12:01 |
| sambetts | vdrok: no it doesn't look like it, and the other modules that have changed between those two don't seem related either :/ | 12:02 |
| vdrok | sambetts: as for failed to establish connection, it did happen on some successful builds too - http://logs.openstack.org/06/306006/1/check/gate-tempest-dsvm-ironic-pxe_ssh/92ef29c/logs/tempest.txt.gz | 12:02 |
| sambetts | :( | 12:02 |
| mat128 | good morning TheJulia, sambetts and vdrok | 12:03 |
| vdrok | morning mat128 | 12:04 |
| mat128 | Found this yesterday while looking at paramiko for our ssh failures: https://github.com/paramiko/paramiko/issues/612 | 12:06 |
| mat128 | "Irritating and increasingly frequent Travis-only failures" | 12:06 |
| sambetts | vdrok, lucas-hungry: We only register, to listen to POLLIN events, is it possible some other event is arriving and we're ignoring it?? | 12:11 |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** baoli has joined #openstack-ironic | 12:16 | |
| *** mtreinish has quit IRC | 12:18 | |
| *** baoli_ has joined #openstack-ironic | 12:18 | |
| *** afaranha has quit IRC | 12:19 | |
| xavierr | good morning guys :) | 12:20 |
| sambetts | o/ xavierr | 12:20 |
| *** mtreinish has joined #openstack-ironic | 12:21 | |
| xavierr | morning sambetts \o/ | 12:21 |
| *** baoli has quit IRC | 12:21 | |
| mat128 | good morning xavierr | 12:21 |
| TheJulia | sambetts: if exec_command is not spawning a shell, I would guess the remote side might be disconnecting | 12:21 |
| xavierr | countdown to summit | 12:21 |
| xavierr | morning mat128 o/ | 12:22 |
| TheJulia | sambetts: but only guessing, since cirros is slightly different from a full blown guest os | 12:22 |
| xavierr | morning TheJulia | 12:22 |
| TheJulia | good morning xavierr | 12:23 |
| sambetts | TheJulia: would you suggest adding a sh -c into the command then ? | 12:23 |
| TheJulia | sambetts: I think it would still exit | 12:23 |
| TheJulia | but I haven't dug into the paramiko code yet | 12:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic: Updated from global requirements https://review.openstack.org/306942 | 12:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-inspector: Updated from global requirements https://review.openstack.org/307752 | 12:24 |
| TheJulia | Only knowing that with cirros and ansible, I'm fairly sure I had to switch it over to paramiko with a raw command to do anything | 12:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-python-agent: Updated from global requirements https://review.openstack.org/306943 | 12:24 |
| TheJulia | although that was mainly because ansible wants python to be on the far side | 12:24 |
| vdrok | morning xavierr | 12:24 |
| vdrok | lucas-hungry: your fix works for me, both manually and in tepest test, thanks! | 12:25 |
| xavierr | morning vdrok :) | 12:26 |
| TheJulia | The only real concern that I would have for the test is if socket has the data and the connection is ready to be closed out before reacing the while loop. Admittedly the odds are kind of remote in my mind, but still a possibility | 12:34 |
| thiagop | Good morning, Ironic | 12:40 |
| mat128 | o/ thiagop | 12:41 |
| TheJulia | good morning thiagop | 12:41 |
| openstackgerrit | Julia Kreger proposed openstack/bifrost: curl is required for diskimage-builder https://review.openstack.org/305659 | 12:46 |
| *** maurosr has quit IRC | 12:46 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-specs: Promote agent vendor passthru to core API https://review.openstack.org/306418 | 12:47 |
| dtantsur | jroll, JayF ^^^ | 12:48 |
| xavierr | morning thiagop | 12:49 |
| *** lucas-hungry is now known as lucasagomes | 12:49 | |
| lucasagomes | vdrok, cool, I probably need to update the tempest unittests then | 12:50 |
| lucasagomes | the gate is so slow :-/ I wanted to wait for the results | 12:51 |
| sambetts | lucasagomes: I was thinking that now we're not using select.poll we can probably get rid of the whole self._can_system_poll() if statement | 12:52 |
| lucasagomes | sambetts, yes | 12:52 |
| lucasagomes | we can get rid of most of that code, I will update the patch | 12:52 |
| lucasagomes | sambetts, vdrok problem being, I lack context about why select.poll was used in the first place | 12:53 |
| * lucasagomes blame files | 12:54 | |
| *** mjturek1 has quit IRC | 12:54 | |
| sambetts | tbh if we're not using select.poll, then why do we need all the extra logic when we can just use the logic that was in the else in the first place | 12:54 |
| sambetts | I'm guessing select.poll was used to optimise the process by ustalising the underlying OS | 12:55 |
| sambetts | utalising* | 12:55 |
| lucasagomes | yeah | 12:56 |
| openstackgerrit | Merged openstack/ironic-webclient: Node list permits individual and group selection. https://review.openstack.org/300161 | 12:56 |
| lucasagomes | that could be it | 12:56 |
| openstackgerrit | Merged openstack/ironic-webclient: Removed setPowerState https://review.openstack.org/300162 | 12:56 |
| openstackgerrit | Merged openstack/ironic-webclient: Table header checkbox disappears when no nodes available. https://review.openstack.org/300166 | 12:56 |
| lucasagomes | trying to find the patch that introduced that anyway, just to see | 12:56 |
| sambetts | lucasagomes: I found it the other day, I think it was quite recent | 12:57 |
| lucasagomes | sambetts, problem is: https://review.openstack.org/#/c/283691/ :D | 12:57 |
| sambetts | 13:21 < xavierr> morning sambetts \o/ | 12:57 |
| sambetts | lucasagomes: ah... | 12:57 |
| sambetts | sorry for the noise xavierr | 12:57 |
| jroll | dtantsur: evgenyl: I should be able to join, but haven't made my schedule yet so no promises :) | 12:57 |
| jroll | morning from new internets at the new crib \o/ | 12:57 |
| sambetts | \o/ woop! | 12:57 |
| dtantsur | woohoo, congrats! | 12:57 |
| jroll | comcast onboarding was pleasant for once | 12:58 |
| jroll | how goes the gate troubleshooting? | 12:58 |
| *** Goneri has joined #openstack-ironic | 12:58 | |
| openstackgerrit | Merged openstack/ironic-webclient: Enabled Refresh button https://review.openstack.org/300167 | 12:58 |
| *** Goneri has quit IRC | 13:00 | |
| lucasagomes | sambetts, https://review.openstack.org/#/c/166518/ apparently it was in tempest, then moved to tempest-lib now back to tempest | 13:00 |
| lucasagomes | heh | 13:00 |
| sambetts | heh | 13:01 |
| lucasagomes | jroll, w00t for the internet, re gate troubleshooting: me and vdrok can confirm that this https://review.openstack.org/#/c/307720/ fix | 13:01 |
| lucasagomes | at least locally | 13:01 |
| sambetts | lucasagomes: https://review.openstack.org/#/c/212691/4 | 13:02 |
| jroll | lucasagomes: oh neat | 13:02 |
| jroll | lucasagomes: we should be using that code from tempest-lib though, and fix it there :/ | 13:02 |
| lucasagomes | jroll, I need to rework that patch to remove the rest of the select stuff | 13:02 |
| sambetts | jroll: they moved tempest-lib back in-tree | 13:02 |
| lucasagomes | jroll, tempest-lib? So that's what I found out | 13:02 |
| lucasagomes | they seem to have migrated from it | 13:03 |
| lucasagomes | like sambetts said | 13:03 |
| vdrok | yep, tempest_lib is now being deprecated | 13:03 |
| sambetts | jroll: and lucasagomes's patch is in the in-tree tempest-lib | 13:03 |
| lucasagomes | funny tho, it was in tempest before, then was moved to tempest-lib and now it's back to tempest | 13:03 |
| * sambetts hopes we don't have to do the same for ironic-lib | 13:03 | |
| jroll | oh, good to know | 13:05 |
| *** keedya has joined #openstack-ironic | 13:05 | |
| *** Goneri has joined #openstack-ironic | 13:05 | |
| sambetts | lucasagomes: https://review.openstack.org/#/c/41497/1/tempest/common/ssh.py | 13:06 |
| xavierr | sambetts, hahaha no problem | 13:06 |
| lucasagomes | sambetts, oh interesting | 13:07 |
| sambetts | lucasagomes: before that https://review.openstack.org/#/c/13011/3/tempest/common/ssh.py | 13:07 |
| *** divya has joined #openstack-ironic | 13:08 | |
| divya | Hello ironic folks | 13:08 |
| thiagop | hiya divya | 13:09 |
| lucasagomes | thanks | 13:09 |
| lucasagomes | divya, thiagop morning | 13:09 |
| vdrok | morning thiagop and divya | 13:10 |
| divya | very good morning guys | 13:11 |
| *** vmud213 has quit IRC | 13:12 | |
| divya | i am testing neutron integration with ironic | 13:12 |
| divya | when i try to boot nova instance on real physical bare metal server, nova list shows it is spawning | 13:14 |
| divya | $ nova list | 13:14 |
| divya | | 93162bc3-db81-41b4-91be-1531c60de491 | bm2 | BUILD | spawning | NOSTATE | provision-net=100.100.100.8 | | 13:14 |
| divya | $ ironic node-list | 13:14 |
| divya | | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | | 13:14 |
| divya | | 9a094549-d889-4010-ae31-b44051388d41 | test1 | 93162bc3-db81-41b4-91be-1531c60de491 | power on | wait call-back | False | 13:14 |
| divya | but ubuntu image is not loaded in the server | 13:15 |
| divya | using ibm m3 server, any suggestion to debug further? | 13:15 |
| sambetts | if its in wait call-back, that means that Ironic is still waiting for the deploy ramdisk to boot and phone home to Ironic | 13:15 |
| thiagop | divya: just a tip, when you paste things like that on the channel, it's very difficult to see due to formatting. You can use paste.openstack.org to paste it in a fancy way and keep formatting | 13:16 |
| lucasagomes | divya, check the console to see if the node is being booted with the deploy ramdisk | 13:16 |
| *** mgould has joined #openstack-ironic | 13:18 | |
| *** [1]cdearborn has joined #openstack-ironic | 13:18 | |
| keedya | Good morning!! | 13:19 |
| thiagop | morning keekz | 13:20 |
| thiagop | morning keedya ! | 13:20 |
| keekz | morning :) | 13:20 |
| thiagop | :) | 13:20 |
| thiagop | #autocompletefail but it's a nice thing to "good morning" someone, so I think it's ok | 13:21 |
| divya | thiagop: Thanks much | 13:21 |
| *** itamarl has joined #openstack-ironic | 13:22 | |
| jroll | keekz: \o/ | 13:22 |
| keekz | morning jroll | 13:22 |
| * jroll assumes keekz is the ironic guy now | 13:22 | |
| keekz | jroll: i'm so sad, one of the rsvp to openstack parties was full :( | 13:22 |
| jroll | nooooo | 13:23 |
| jroll | which one? | 13:23 |
| keekz | jroll: that's fine, you can be the monitoring guy | 13:23 |
| keekz | stackcity | 13:23 |
| jroll | cool | 13:23 |
| jroll | really?! | 13:23 |
| keekz | yep | 13:23 |
| jroll | :( | 13:23 |
| keekz | i got the email and went to rsvp and it was full | 13:23 |
| keekz | oh well | 13:23 |
| jroll | find me that day, we'll sneak you in | 13:23 |
| keekz | lol | 13:23 |
| jroll | srs | 13:23 |
| jroll | "omg I forgot to check +1" | 13:23 |
| * thiagop knew that jroll had superpowers | 13:23 | |
| *** karimb has joined #openstack-ironic | 13:24 | |
| *** trown is now known as trown|brb | 13:25 | |
| *** adreznec has quit IRC | 13:27 | |
| divya | lucasagomes: bare metal server's boot priority is pxe so, boot strap is loaded. | 13:30 |
| TheJulia | jroll: I think the +1 required you to submit the attendee name since it is being done by wrist bands from what I read | 13:30 |
| lucasagomes | divya, ok, so after it's loaded it will try to talk to ironic api's back | 13:31 |
| openstackgerrit | Gonéri Le Bouder proposed openstack/ironic: ipxe: retry on failure https://review.openstack.org/306196 | 13:31 |
| openstackgerrit | Merged openstack/python-ironic-inspector-client: Introspection on stored data https://review.openstack.org/302195 | 13:31 |
| *** oberezovskyi has left #openstack-ironic | 13:31 | |
| lucasagomes | divya, and then the node will be moved from wait-callback to deploying | 13:31 |
| jroll | TheJulia: we'll find a way. there's always a way. | 13:32 |
| divya | lucasagomes: approximately how long it will take? | 13:33 |
| *** adreznec has joined #openstack-ironic | 13:33 | |
| Goneri | https://review.openstack.org/#/c/306196/4 I adjusted the commit message by mistake. lucasagomes can you put back your +2? | 13:33 |
| *** trown|brb is now known as trown | 13:34 | |
| Goneri | TheJulia, I also answered your comment :D | 13:34 |
| *** piet has joined #openstack-ironic | 13:34 | |
| *** openstackstatus has joined #openstack-ironic | 13:37 | |
| *** ChanServ sets mode: +v openstackstatus | 13:37 | |
| -openstackstatus- NOTICE: We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 13:40 | |
| lucasagomes | Goneri, will do | 13:41 |
| lucasagomes | divya, this can vary from machine to machine, some of them takes few minutes to boot something | 13:41 |
| openstackgerrit | Gonéri Le Bouder proposed openstack/ironic: iscsi: wipe the disk before deployment https://review.openstack.org/303605 | 13:42 |
| lucasagomes | divya, after booted IPA (the deploy ramdisk code) should start pretty quickly | 13:42 |
| *** cdearborn has joined #openstack-ironic | 13:48 | |
| *** ayoung has joined #openstack-ironic | 13:50 | |
| *** piet has quit IRC | 13:53 | |
| keedya | Hi all, I am working on a enhacement in ironic-inspector; Can I post my questions here? https://bugs.launchpad.net/ironic-inspector/+bug/1564863 | 13:54 |
| openstack | Launchpad bug 1564863 in Ironic Inspector "[RFE] Make it possible to send/replace node data via uuid/data/unprocessed API" [Wishlist,Confirmed] - Assigned to Andre keedy (andre-keedy) | 13:54 |
| TheJulia | Goneri: awesome | 13:56 |
| lucasagomes | keedya, sure | 13:56 |
| sambetts | keedya: Sure, or ask in here and we'll discuss them :) | 13:56 |
| keedya | awesome sambetts!! thank you | 13:56 |
| openstackgerrit | Thiago Paiva Brito proposed openstack/ironic: OneView driver docs explaining Dynamic Allocation https://review.openstack.org/305441 | 13:57 |
| TheJulia | Goneri: My concern thought is that the release note indicates that both are required, but wouldn't ipxe end up retrying anyway if say, it got a 404? | 13:58 |
| openstackgerrit | Lilia Sampaio proposed openstack/ironic: Add Dynamic Allocation feature for the OneView drivers https://review.openstack.org/286192 | 13:59 |
| keedya | Just want to make sure i got it right, data passed using uuid/data/unprocessed API should overwrite discovered data for a specific UUID | 13:59 |
| *** mtanino has quit IRC | 14:00 | |
| *** moshele has quit IRC | 14:01 | |
| *** alineb has quit IRC | 14:01 | |
| *** baoli_ has quit IRC | 14:01 | |
| sambetts | keedya: yup, it should overwrite the raw discovered/stored data for a specific node, and then trigger processing over that data | 14:02 |
| *** moshele has joined #openstack-ironic | 14:03 | |
| *** [1]cdearborn has quit IRC | 14:03 | |
| keedya | processing includes running pre and post hooks and set of rules? | 14:03 |
| *** links has quit IRC | 14:03 | |
| keedya | and finallu update ironic data? | 14:03 |
| sambetts | yup, although its the post hooks and the rules do the updates to Ironic | 14:04 |
| *** jayc_ has joined #openstack-ironic | 14:04 | |
| keedya | got it!! thanks sambetts | 14:05 |
| *** piet has joined #openstack-ironic | 14:06 | |
| *** jayc_ is now known as jayc | 14:07 | |
| NobodyCam | good morning Ironic folks | 14:08 |
| mgould | morning NobodyCam | 14:08 |
| NobodyCam | morning mgould :) | 14:09 |
| thiagop | good morning, NobodyCam | 14:09 |
| NobodyCam | morning thiagop | 14:09 |
| jroll | \o NobodyCam | 14:09 |
| *** vmud213 has joined #openstack-ironic | 14:10 | |
| NobodyCam | hey morning jroll , you saw the paramiko bugs :) | 14:10 |
| jroll | ya | 14:11 |
| *** vmud213 has quit IRC | 14:12 | |
| *** vmud213 has joined #openstack-ironic | 14:12 | |
| jroll | lucasagomes: do you want to do the governance change for https://review.openstack.org/#/c/307323 | 14:13 |
| dtantsur | morning NobodyCam, thiagop | 14:13 |
| thiagop | hiya dtantsur | 14:13 |
| NobodyCam | morning dtantsur | 14:13 |
| lucasagomes | jroll, will do, I was debugging the gate stuff (now updating the tempest patch) | 14:14 |
| lucasagomes | so I didn't look at it today | 14:14 |
| jroll | lucasagomes: yeah no rush | 14:15 |
| jroll | thanks man | 14:15 |
| lucasagomes | no worries, I will try to update it today | 14:17 |
| *** vmud213 has quit IRC | 14:17 | |
| Goneri | TheJulia, in this case indeed, it will probably loop. | 14:18 |
| xavierr | morning NobodyCam :) | 14:20 |
| NobodyCam | morning xavierr | 14:20 |
| NobodyCam | :) | 14:20 |
| NobodyCam | morning lucasagomes :) | 14:21 |
| *** piet has quit IRC | 14:21 | |
| TheJulia | From what I can tell in the ipxe code, it will, hence why I feel the release note should clearly indicate that the default logic has changed, not only if the timeout is set | 14:21 |
| lucasagomes | NobodyCam, hi there, good morning | 14:21 |
| Goneri | TheJulia, agreed, I've fixing the message | 14:22 |
| *** alineb has joined #openstack-ironic | 14:22 | |
| openstackgerrit | Gonéri Le Bouder proposed openstack/ironic: ipxe: retry on failure https://review.openstack.org/306196 | 14:22 |
| TheJulia | Goneri: Awesome, thank you! :) | 14:22 |
| *** ChrisAusten has joined #openstack-ironic | 14:22 | |
| Goneri | TheJulia, +2 welcome :D | 14:22 |
| * xavierr that moment you forget change the git-review username and you commit in other person name and lose the commit | 14:23 | |
| TheJulia | xavierr: doh! | 14:24 |
| * jlvillal is unsure how that happens. | 14:24 | |
| thiagop | jlvillal: we sometimes develop in the same machine | 14:25 |
| NobodyCam | morning jlvillal | 14:25 |
| thiagop | jlvillal: kinda pair programming | 14:25 |
| jlvillal | thiagop, Ah. Okay, then I understand | 14:25 |
| jlvillal | NobodyCam, Good morning | 14:25 |
| NobodyCam | :) | 14:25 |
| thiagop | jlvillal: the only difference is that we are 6 | 14:25 |
| * jlvillal gets ready to go to work... | 14:25 | |
| jlvillal | thiagop, :) | 14:26 |
| * thiagop feels like a Borg sometimes | 14:26 | |
| *** divya has quit IRC | 14:27 | |
| TheJulia | thiagop: As long as your not refering to yourself as some number in some unit... and have no assimulation nanites, I think everything will be okay :) | 14:27 |
| thiagop | err.. nope. That would be terrible since I'm going to the summit, right?! | 14:28 |
| *** piet has joined #openstack-ironic | 14:29 | |
| TheJulia | I hope you are, although having a contingent of borg attend session swill be hilarious | 14:31 |
| TheJulia | sessions would | 14:32 |
| thiagop | Summit is a borg-like thing since we are deciding things for the good of the collective | 14:34 |
| * mgould is intrigued by the concept of "session swill" | 14:37 | |
| sambetts | lucasagomes: https://review.openstack.org/#/c/305868/ I missed this comment when I updated the devstack script | 14:37 |
| mgould | is that an after-summit party where you drink all the leftover beer? | 14:37 |
| thiagop | mgould: hahaha | 14:37 |
| sambetts | ewwww | 14:38 |
| sambetts | XD | 14:38 |
| NobodyCam | morning sambetts | 14:39 |
| sambetts | Hey NobodyCam | 14:39 |
| lucasagomes | sambetts, oh right | 14:39 |
| *** mtanino has joined #openstack-ironic | 14:39 | |
| mgould | one of the Oxford colleges has an end-of-term party called "drink the bar dry", which is exactly what it says on the tin | 14:39 |
| thiagop | lol | 14:41 |
| * xavierr planning to take a 3 h 7 m car trip to NASA Space Center Houston | 14:42 | |
| *** amotoki has joined #openstack-ironic | 14:43 | |
| lucasagomes | wow some jobs in gate are taking 23h! | 14:45 |
| lucasagomes | or more | 14:46 |
| NobodyCam | xavierr: be carful http://www.google.org/publicalerts/alert?aid=6b66172d64b8a455&hl=en&gl=US&source=web | 14:46 |
| NobodyCam | Huston is flooding badly | 14:46 |
| TheJulia | Yeah, my boot from volume spec took 12 hours to pass the two jobs on the specs repo | 14:46 |
| *** e0ne has quit IRC | 14:47 | |
| thiagop | There was a warning about an hour ago saying that guys recovered one of the clouds. I think that time tends to reduce now... | 14:48 |
| lucasagomes | TheJulia, :-/ | 14:48 |
| sambetts | NobodyCam: I saw that on the news this morning and I've checked the weather for Austin and its looks like thunder storms all week | 14:49 |
| thiagop | NobodyCam: does google now warn about these things in a reliable way? | 14:49 |
| * xavierr thinking to travel to Houston using google street view after NobodyCam link | 14:49 | |
| *** mgoddard_ has joined #openstack-ironic | 14:49 | |
| NobodyCam | thiagop: http://www.nbcnews.com/news/weather/flash-floods-hit-houston-area-storm-drenches-harris-country-n557451 | 14:49 |
| Goneri | lucasagomes, https://review.openstack.org/#/c/306196/ :D | 14:49 |
| NobodyCam | its bad there | 14:50 |
| NobodyCam | next week in Austin is just showing light rain | 14:50 |
| Goneri | lucasagomes, -openstackstatus/#openstack-ironic- NOTICE: We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 14:50 |
| sambetts | oh, when I checked this morning it said thunderstorms | 14:50 |
| lucasagomes | Goneri, will take a look | 14:50 |
| thiagop | NobodyCam: holy cow... | 14:51 |
| *** mgoddard_ has quit IRC | 14:51 | |
| krotscheck | NobodyCam: Everything is bigger in texas. Even the light rain. | 14:52 |
| *** mgoddard has joined #openstack-ironic | 14:53 | |
| sambetts | haha | 14:53 |
| JayF | I just assume I'll wanna be in AC in Texas anyway | 14:53 |
| JayF | so whether or not it's raining isn't relevant | 14:53 |
| JayF | lol | 14:54 |
| NobodyCam | krotscheck: | 14:54 |
| NobodyCam | ++ | 14:54 |
| NobodyCam | sambetts: ya looks like its changed sense last I checked | 14:54 |
| vdrok | morning jroll NobodyCam and JayF | 14:54 |
| JayF | hm | 14:55 |
| JayF | gm | 14:55 |
| NobodyCam | JayF: it says 80's that's not bad.. (says the man living in the desert) | 14:55 |
| NobodyCam | morning vdrok | 14:55 |
| vdrok | and morning jlvillal | 14:55 |
| NobodyCam | also morning JayF :) | 14:55 |
| *** mgoddard_ has joined #openstack-ironic | 14:55 | |
| JayF | this is not passing tests because the gate is unhappy, but it was earlier; if folks wouldn't mind re-reviewing this I'd appreciate it: https://review.openstack.org/#/c/263842/ --- it had a lot of momentum but seems to have died out, and I'd love to get the patch in | 14:55 |
| *** mgoddard has quit IRC | 14:55 | |
| krotscheck | Quick question on the state chart. The event to go from Available to Deploying is called 'active'. In the webclient, we've got that as "deploy". Did that change recently, or do we have a typo in the ebclient? | 14:55 |
| lucasagomes | JayF, ++ for ac | 14:56 |
| *** mgoddard__ has joined #openstack-ironic | 14:57 | |
| *** mgoddard_ has quit IRC | 14:57 | |
| *** mgoddard__ has left #openstack-ironic | 14:58 | |
| jroll | krotscheck: it hasn't changed, https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/node.py#L513 | 14:58 |
| *** dtantsur is now known as dtantsur|brb | 14:59 | |
| krotscheck | jroll: Danke | 14:59 |
| jlvillal | vdrok: Good morning | 14:59 |
| sambetts | krotscheck, jroll: in the OSC plugin, we replaced set-provision-state active with openstack baremetal node deploy https://specs.openstack.org/openstack/ironic-specs/specs/approved/ironicclient-osc-plugin.html | 15:00 |
| jroll | ametts: right, but the api is still target=active | 15:01 |
| sambetts | yup | 15:01 |
| jroll | er. | 15:01 |
| jroll | sorry allan :) | 15:01 |
| *** amotoki has quit IRC | 15:02 | |
| *** deray has quit IRC | 15:02 | |
| krotscheck | ALrightey. All I have to do now is add tests, and the webclient will officially be able to do node provisioning things. | 15:02 |
| krotscheck | Note: Only tested on fake driver. | 15:02 |
| jroll | \o/ | 15:03 |
| *** karimb has quit IRC | 15:04 | |
| devananda | morning, all | 15:05 |
| NobodyCam | good morning devananda | 15:05 |
| devananda | krotscheck: !! nice! | 15:05 |
| mgould | morning devananda | 15:05 |
| jroll | \o deva | 15:05 |
| thiagop | morning devananda | 15:06 |
| *** garthb has joined #openstack-ironic | 15:06 | |
| *** baoli has joined #openstack-ironic | 15:07 | |
| *** trown is now known as trown|afk | 15:09 | |
| *** absubram has joined #openstack-ironic | 15:10 | |
| JayF | The whiteboard says the gate is broken | 15:13 |
| JayF | as do just about every patch I look at | 15:13 |
| JayF | has anyone looked into it? Any idea what's going on? I got past the issue failing IPA gate, and it's now failing later in the tempest run :/ | 15:13 |
| * NobodyCam points finger at paramiko | 15:13 | |
| TheJulia | and tempest | 15:14 |
| TheJulia | it is both of them! | 15:14 |
| JayF | Is there a patch coming in to fix it? | 15:14 |
| JayF | Like what's the action being taken right now/ | 15:14 |
| mgould | TheJulia, they're conspiring against us | 15:14 |
| TheJulia | JayF: Yes, lucas put up a patch for tempest, looks like tempest-lib is already fixed, but apparently the wrong code is getting used | 15:14 |
| TheJulia | mgould: exactly! ;) | 15:15 |
| JayF | https://review.openstack.org/#/c/307474/ should fix IPA gate once the other pieces are working | 15:15 |
| jroll | hey y'all, in case I didn't mention it, there's a nova/ironic cross-project session: https://www.openstack.org/summit/austin-2016/summit-schedule/events/9084?goback=1 | 15:15 |
| JayF | basically httpredir, you have a much greater % chance of failing any one thing on a build | 15:15 |
| JayF | because you're using all the mirrors at once | 15:15 |
| jroll | JayF: here's the gate bug: https://bugs.launchpad.net/diskimage-builder/+bug/1571492 | 15:15 |
| openstack | Launchpad bug 1571492 in Ironic "gate-tempest-dsvm-ironic-pxe_ssh-dib failing : test_baremetal_server_ops ssh timeout" [Critical,Confirmed] | 15:15 |
| jroll | lucas' patch is in a comment there | 15:15 |
| *** chopmann has quit IRC | 15:15 | |
| JayF | changing it to http.us just for now, longer-term infra is going to put a var in devstack/images that we can use to setup the sources.list reasonably | 15:16 |
| jroll | and is sitting around in the gate | 15:16 |
| jroll | s/gate/check queue/ | 15:16 |
| vdrok | morning devananda | 15:16 |
| JayF | ty jroll | 15:16 |
| jroll | np | 15:16 |
| JayF | so sounds like there's nothing to do but wait :/ | 15:16 |
| vdrok | TheJulia: I think tempest-lib has the same problem, at least code there is the same as in tempest | 15:16 |
| jroll | JayF: yeah, assuming lucasagomes doesn't need any help with that right now | 15:17 |
| TheJulia | vdrok: my read of the conversation earlier was that it was fixed, but the code looked almost the same when I skimmed it :\ | 15:17 |
| devananda | ah, this is the paramiko issue folks were discussing yesterday? | 15:17 |
| jroll | yes | 15:17 |
| vdrok | yep :) | 15:17 |
| JayF | well my gate fix queued up behind other gate fixes, with the gate taking approximately a day to land anything | 15:17 |
| JayF | tl;dr we'll be lucky to have it working by summit | 15:18 |
| JayF | lol | 15:18 |
| jroll | ovh is back online in nodepool | 15:18 |
| jroll | things should get better today | 15:18 |
| JayF | was it just Rackspace yesterday? | 15:18 |
| jroll | I thought it was just ovh yesterday | 15:18 |
| jroll | oh | 15:18 |
| jroll | I see what you mean; I'm not sure | 15:18 |
| jroll | there were issues with ovh though | 15:18 |
| JayF | gotcha | 15:18 |
| jroll | nothing against ovh, just account things | 15:18 |
| JayF | Suuuuuuure. | 15:19 |
| jroll | srs | 15:19 |
| jroll | the CI tenant disappeared :P | 15:19 |
| jroll | whether that was an ovh thing or an infra thing, not sure | 15:19 |
| *** [1]cdearborn has joined #openstack-ironic | 15:20 | |
| *** ayoung has quit IRC | 15:20 | |
| * lucasagomes reads | 15:20 | |
| lucasagomes | JayF, jroll I'm just finishing writing the unittests | 15:21 |
| jroll | yeah, figured as much :) | 15:21 |
| lucasagomes | I rewrote the poll bits from tempest, so I had to change all the related patches | 15:21 |
| lucasagomes | plus I'm not super familiar with the tempest code base so | 15:21 |
| *** itamarl has quit IRC | 15:24 | |
| *** rama_y has joined #openstack-ironic | 15:25 | |
| *** fhubik has joined #openstack-ironic | 15:25 | |
| fhubik | jroll: Hey, created as promised yesterday - https://bugs.launchpad.net/bugs/1572201, but for some reason it assignet to network-manager even when I explicitly specified ironic ... :/ | 15:27 |
| openstack | Launchpad bug 1572201 in network-manager (Ubuntu) "Long ironic timeouts because of ServFail DNS error" [Undecided,New] | 15:27 |
| jroll | fhubik: heh, strange | 15:28 |
| * jroll fixes | 15:28 | |
| * fhubik thanks | 15:28 | |
| jlvillal | https://bugs.launchpad.net/bugs/1572201 | 15:28 |
| openstack | Launchpad bug 1572201 in network-manager (Ubuntu) "Long ironic timeouts because of ServFail DNS error" [Undecided,New] | 15:28 |
| jroll | it is fixed now | 15:29 |
| jlvillal | https://bugs.launchpad.net/bugs/1572201 | 15:29 |
| openstack | Launchpad bug 1572201 in Ironic "Long ironic timeouts because of ServFail DNS error" [Undecided,New] | 15:29 |
| jlvillal | :) | 15:29 |
| lucasagomes | JayF, jroll https://review.openstack.org/307720 that should do it | 15:29 |
| lucasagomes | vdrok, sambetts ^ FYI | 15:29 |
| fhubik | jroll: ok, thx | 15:30 |
| * jlvillal still wonders whatever happened to his little python script to query launchpad... | 15:30 | |
| *** karimb has joined #openstack-ironic | 15:31 | |
| *** cdearborn has quit IRC | 15:31 | |
| *** moshele has quit IRC | 15:32 | |
| *** dtantsur|brb is now known as dtantsur | 15:37 | |
| dtantsur | morning devananda, JayF, jlvillal | 15:38 |
| JayF | gm | 15:38 |
| devananda | mornin | 15:38 |
| jlvillal | Hey dtantsur :) | 15:38 |
| *** ppiela has quit IRC | 15:38 | |
| lucasagomes | devananda, morning | 15:38 |
| TheJulia | good morning devananda | 15:39 |
| *** dprince has quit IRC | 15:40 | |
| krotscheck | So, is there an Ironic social thing happening? | 15:43 |
| vdrok | lucasagomes: left a comment | 15:44 |
| lucasagomes | krotscheck, in austin? I would think so | 15:44 |
| lucasagomes | not sure if there's anything planned at the moment but we can come up with a plan this week | 15:45 |
| lucasagomes | if folks are up to :-) | 15:45 |
| lucasagomes | vdrok, thanks | 15:45 |
| *** links has joined #openstack-ironic | 15:45 | |
| krotscheck | lucasagomes: betherly's trying to figure out when/where to do an ironic-ui social, maybe combining it with an ironic social would be neat. | 15:45 |
| *** ppiela has joined #openstack-ironic | 15:46 | |
| jroll | it might be kind of late to do anything official-ish, might be able to pull something off though | 15:46 |
| lucasagomes | vdrok, oh right yeah I have to check that timeout | 15:46 |
| betherly | lucasagomes: krotscheck: currently thinking monday evening or thursday evening are best as per last email about it. thoughts? | 15:46 |
| * lucasagomes changes the patch | 15:46 | |
| jroll | +1 betherly | 15:47 |
| thiagop | TheJulia: thanks for the review. I'll ensure it'll be addressed later today | 15:47 |
| lucasagomes | sounds good to me | 15:47 |
| betherly | jroll: if we send out a bulk email saying where we will be and when we can leave it pretty casual | 15:47 |
| openstackgerrit | Aparna proposed openstack/ironic: Deployment vmedia operations to run when cleaning https://review.openstack.org/307225 | 15:47 |
| jroll | betherly: yeah, you'll still want to reserve at least 20-30 things if you tell the whole world about it | 15:47 |
| jroll | things/seats | 15:48 |
| TheJulia | thiagop: Thank you | 15:48 |
| betherly | jroll: true | 15:48 |
| betherly | jroll: hard to book in advance without knowing the area too though :/ | 15:49 |
| *** ppiela has quit IRC | 15:50 | |
| jroll | betherly: indeed, krtaylor may be able to help, he's local | 15:51 |
| JayF | krotscheck: https://etherpad.openstack.org/p/ironic-summit-arrivals might wanna put your info there; I don't think we've planned anything yet but that's how we're going to try and keep folks in the loop | 15:51 |
| JayF | betherly: ^ same | 15:51 |
| * krtaylor reads scrollback | 15:51 | |
| *** daemontool has quit IRC | 15:52 | |
| *** daemontool has joined #openstack-ironic | 15:53 | |
| *** rloo has joined #openstack-ironic | 15:53 | |
| krtaylor | krotscheck, lucasagomes, jroll, betherly - I have been looking at a few places within a couple of blocks of the convention center, will make some calls today and send an email | 15:53 |
| *** ppiela_ has joined #openstack-ironic | 15:54 | |
| * thiagop looks what folks will be at the same hotel | 15:54 | |
| betherly | krtaylor: ++ | 15:54 |
| jroll | krtaylor: nice, thanks | 15:55 |
| dtantsur | jroll, have you created etherpads for the summit? I could reuse the driver composition one for my notes. | 15:55 |
| lucasagomes | krtaylor, thanks! | 15:56 |
| jroll | dtantsur: yes, sec | 15:56 |
| jroll | putting all the links together now actually :) | 15:56 |
| jroll | dtantsur: https://etherpad.openstack.org/p/ironic-newton-summit and https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Ironic | 15:56 |
| *** links has quit IRC | 15:57 | |
| dtantsur | thanks! | 15:57 |
| jroll | np | 15:58 |
| jroll | TheJulia: did you see this? https://www.openstack.org/summit/austin-2016/summit-schedule/events/9306?goback=1 | 15:59 |
| TheJulia | Nope, but adding to my schedule, thank you jroll | 15:59 |
| jroll | np | 15:59 |
| *** Guest73397 has quit IRC | 16:00 | |
| *** hblixt has joined #openstack-ironic | 16:01 | |
| *** zzzeek has quit IRC | 16:02 | |
| *** zzzeek has joined #openstack-ironic | 16:03 | |
| *** __maurosr__ has joined #openstack-ironic | 16:04 | |
| cinerama | wow interesting | 16:04 |
| *** mkovacik_ has quit IRC | 16:06 | |
| *** Sukhdev has joined #openstack-ironic | 16:07 | |
| jroll | dtantsur: do you want to moderate the driver comp session? ditto for gate session? | 16:07 |
| dtantsur | jroll, yes to the first, maybe for the latter (if nobody else wants) | 16:08 |
| jroll | I can run the gate one | 16:08 |
| jroll | JayF: wanna moderate the ops tooling session? | 16:08 |
| openstackgerrit | Steve Martinelli proposed openstack/ironic-inspector: use openstack cli instead of keystone cli https://review.openstack.org/307523 | 16:09 |
| JayF | yep | 16:09 |
| JayF | if you're just looking for someone to moderate, I would be happy to help w/gate session too | 16:09 |
| openstackgerrit | Steve Martinelli proposed openstack/ironic-inspector: use openstack cli instead of keystone cli https://review.openstack.org/307523 | 16:09 |
| jroll | cool | 16:09 |
| jroll | thanks | 16:09 |
| dtantsur | devananda, sambetts, could you take a look at https://etherpad.openstack.org/p/ironic-newton-summit-driver-composition if I've captured the discussion we had more or less correctly? | 16:09 |
| dtantsur | I've reworked it a bit for simplicity | 16:09 |
| krtaylor | jroll, JayF - I can help also | 16:09 |
| jroll | krtaylor: yeah, I'll run it and pass the torch as needed (like to you for CI) | 16:10 |
| jroll | third-party CI* | 16:10 |
| krtaylor | jroll, sure, np | 16:10 |
| * dtantsur brb | 16:10 | |
| *** dtantsur is now known as dtantsur|brb | 16:10 | |
| *** ayoung has joined #openstack-ironic | 16:11 | |
| * sambetts looks at what dtantsur|brb is typing | 16:11 | |
| jroll | vdrok: aarefiev: pas-ha: who wants to moderate the ansible driver session? | 16:12 |
| pas-ha | I'm not coming to Austin :( | 16:13 |
| jroll | :( | 16:13 |
| *** cfarquhar has quit IRC | 16:14 | |
| openstackgerrit | Kyrylo Romanenko proposed openstack/python-ironicclient: Use name randomizer from tempest_lib https://review.openstack.org/307904 | 16:14 |
| sambetts | dtantsur|brb: sounds pretty good, do we want to add a list of all current/planned in-tree interfaces and hardware types so we can get an idea of the size of the matrix | 16:15 |
| sambetts | or would that be too much? | 16:16 |
| rloo | sambetts: ++ | 16:16 |
| *** cdearborn has joined #openstack-ironic | 16:17 | |
| *** cfarquhar has joined #openstack-ironic | 16:19 | |
| *** cfarquhar has quit IRC | 16:19 | |
| *** cfarquhar has joined #openstack-ironic | 16:19 | |
| aarefiev | jroll: I guess vdrok and me | 16:19 |
| *** anteaya has quit IRC | 16:20 | |
| *** ifarkas has quit IRC | 16:20 | |
| jroll | aarefiev: cool, thanks | 16:23 |
| *** jayc has quit IRC | 16:23 | |
| lucasagomes | jroll, it's funny that project-config cares about alphabetic ordering, I mean, the files have >9k lines | 16:27 |
| jroll | lucasagomes: yeah, that's probably why, else it's hard to find things | 16:27 |
| lucasagomes | I bet people would just grep for something there | 16:27 |
| *** piet has quit IRC | 16:29 | |
| *** karimb has quit IRC | 16:29 | |
| *** [1]cdearborn has quit IRC | 16:30 | |
| *** piet has joined #openstack-ironic | 16:31 | |
| *** moshele has joined #openstack-ironic | 16:34 | |
| jroll | betherly: krtaylor: can we shoot for monday for the thing? | 16:36 |
| *** jistr has quit IRC | 16:39 | |
| devananda | dtantsur|brb: looking at the 'pad now | 16:39 |
| *** kromanenko has quit IRC | 16:41 | |
| betherly | jroll: Monday sounds good. ironic ui team are planning to go for a drink after the horizon working sessions thursday also so people are welcome to keep in touch and join us then as well! | 16:44 |
| jroll | betherly: cool :) I ask because trying to plan a rackspace thing too | 16:45 |
| NobodyCam | dtantsur|brb: Just noting that you list power drivers like AMT and WOL I believe both of those are moving to the staging repo | 16:45 |
| jroll | betherly: monday or thursday is fine, just trying to lock the day down | 16:45 |
| *** cfarquhar has quit IRC | 16:45 | |
| lucasagomes | jroll, vdrok sambetts found the problem with the paramiko thing... Narrowed it down to the conditional looking for channel.closed | 16:47 |
| NobodyCam | lucasagomes: AWESOME!!!!! | 16:47 |
| lucasagomes | apparently sometimes after exec_command() the channel is still open | 16:47 |
| lucasagomes | and so it hangs | 16:47 |
| *** cfarquhar has joined #openstack-ironic | 16:47 | |
| *** cfarquhar has quit IRC | 16:47 | |
| *** cfarquhar has joined #openstack-ironic | 16:47 | |
| vdrok | lucasagomes: that's great :) thanks! | 16:47 |
| sambetts | wow... ok... I wonder why its started doing that! | 16:47 |
| lucasagomes | vdrok, right ehre https://github.com/openstack/tempest/blob/master/tempest/lib/common/ssh.py#L151 | 16:48 |
| lucasagomes | the reason why it is still open idk | 16:48 |
| lucasagomes | will investigate | 16:48 |
| sambetts | is the command we run keeping it open or something? | 16:48 |
| lucasagomes | sambetts, http://docs.paramiko.org/en/1.16/api/channel.html#paramiko.channel.Channel.exec_command | 16:48 |
| lucasagomes | so it says after the exec_command the channel should be closed | 16:49 |
| lucasagomes | wonder if it's a bug in paramiko | 16:49 |
| *** ppiela_ has quit IRC | 16:49 | |
| sambetts | lucasagomes: when the command finished executing, I wonder if the command gets stuck open, e.g. if one of the previous commands holds their pipe open | 16:50 |
| lucasagomes | sambetts, it hangs with a ls :-/ | 16:50 |
| sambetts | :/ | 16:50 |
| *** cdearborn has quit IRC | 16:50 | |
| sambetts | the version of paramiko hasn't changed in the gate requirements.txt / upper constraints since Jan when we looked, so I don't know why we're suddenly sufering unless its only just made it into the nodepool images or something | 16:51 |
| krtaylor | jroll, Wed is out? Monday is booth crawl and Rainey street Ops party | 16:51 |
| jroll | krtaylor: tuesday and wednesday are stackcity and core party (not sure which is which) | 16:52 |
| jroll | we can do thursday for ironic, that's totally fine too | 16:52 |
| JayF | all these parties are cramping our ability to have a party! | 16:52 |
| jroll | just want to decide :) | 16:52 |
| *** ppiela has joined #openstack-ironic | 16:53 | |
| thiagop | LOL | 16:53 |
| vdrok | lucasagomes: one more thing I've noticed, not related to the problem tho, self.channel_timeout that is used in the poll call, its value equals to CONF.validation.connect_timeout=60, and in description it says these are seconds, but poll expects milliseconds as timeout | 16:54 |
| lucasagomes | vdrok, ouch | 16:54 |
| lucasagomes | yeah we can add a 2nd patch-set fixing that too | 16:54 |
| lucasagomes | vdrok, great finding! | 16:54 |
| NobodyCam | oh wow so time out was effectively 60 milliseconds... nice | 16:55 |
| NobodyCam | :/ | 16:55 |
| lucasagomes | sambetts, yeah, I'm wondering about that too | 16:55 |
| krtaylor | jroll, ok, I'll try Thurs first | 16:56 |
| lucasagomes | why is the channel open after exec_command returns | 16:56 |
| jroll | krtaylor: okay cool /cc betherly | 16:56 |
| openstackgerrit | Michael Krotscheck proposed openstack/ironic-webclient: IronicNodeProvision resource https://review.openstack.org/282457 | 16:58 |
| openstackgerrit | Michael Krotscheck proposed openstack/ironic-webclient: Changed action from 'deploy' to 'active' https://review.openstack.org/307926 | 16:58 |
| openstackgerrit | Michael Krotscheck proposed openstack/ironic-webclient: Modified IronicNode mock to use single source of data https://review.openstack.org/307927 | 16:58 |
| openstackgerrit | Michael Krotscheck proposed openstack/ironic-webclient: Added provision state manipulation https://review.openstack.org/307928 | 16:58 |
| *** __maurosr__ is now known as maurosr | 17:00 | |
| krotscheck | Ironic-webclient. Now with node provision state management. | 17:00 |
| TheJulia | \o/ | 17:01 |
| *** derekh has quit IRC | 17:01 | |
| *** alineb- has joined #openstack-ironic | 17:01 | |
| *** dprince has joined #openstack-ironic | 17:02 | |
| *** cdearborn has joined #openstack-ironic | 17:02 | |
| *** ChrisAusten has quit IRC | 17:02 | |
| *** ChrisAusten has joined #openstack-ironic | 17:03 | |
| *** rcernin has quit IRC | 17:04 | |
| *** dtantsur|brb is now known as dtantsur | 17:06 | |
| * dtantsur is back sorry | 17:07 | |
| dtantsur | sambetts, that's a long task :D | 17:07 |
| *** [1]cdearborn has joined #openstack-ironic | 17:08 | |
| *** garthb has quit IRC | 17:09 | |
| *** alineb has quit IRC | 17:10 | |
| *** alineb- is now known as alineb | 17:10 | |
| rloo | jroll and others. When we talk about wanting live upgrades, what are we talking about. Upgrades between what versions, numbered versions? Between Major numbered versions? Between 'named (ie, the Mitaka and Newton versions)? | 17:10 |
| *** mbound has quit IRC | 17:11 | |
| jlvillal | rloo: My gut feeling would be all of that. | 17:11 |
| rloo | jlvillal: it makes a difference. maybe. i was wondering as i read the spec. | 17:12 |
| jlvillal | I would think there would probably be a maximum level difference. | 17:12 |
| devananda | specifically, I believe we need to support the following | 17:12 |
| jlvillal | Like probably not from Mitaka to Quixiotic | 17:13 |
| devananda | - sequential named releases (eg, L->M, M->N) | 17:13 |
| sambetts | so LTS to LTS | 17:13 |
| rloo | sambetts: what's 'LTS'? | 17:13 |
| devananda | - sequential major releases (eg, 4.x -> 5.0, 5.0 -> 5.1, ..) | 17:13 |
| sambetts | Long term support | 17:13 |
| devananda | - most recent named or numbered release -> trunk | 17:14 |
| rloo | sambetts: is LTS == named releases (Mitaka -> N, N->O)? | 17:14 |
| devananda | sambetts: Named releases are not LTS releases | 17:14 |
| *** garthb has joined #openstack-ironic | 17:14 | |
| devananda | LTS is specifically what distro's (like canonical) use to indicate multi-year support | 17:15 |
| devananda | openstack does not have LTS releases, and instead leaves that to the distros to choose if/when to do | 17:15 |
| *** Sukhdev has quit IRC | 17:15 | |
| devananda | jlvillal: openstack does not, as a whole, support or intend to support upgrades skipping over named releases. | 17:16 |
| sambetts | I thought that was the model we discussed in the meeting yesterday, we'll major release any number of times between named releases, but the one we declare as the named release is essentially a long term release | 17:16 |
| jlvillal | devananda: Good. Thanks. | 17:16 |
| sambetts | long term support release for Ironic being 6 motnhs | 17:16 |
| sambetts | months* | 17:16 |
| devananda | sambetts: you're sort of correct | 17:16 |
| *** suro-patz has joined #openstack-ironic | 17:17 | |
| sambetts | just our long term is pretty short :) | 17:17 |
| devananda | sambetts: we'll release numbered releases when ever we choose. the last minor release we cut before the OpenStack Coordinated Release becomes the basis for that Named Release | 17:17 |
| devananda | sambetts: please do not use the term "long term release" for this. it will confuse people | 17:17 |
| devananda | it is a Named release | 17:17 |
| sambetts | Sure :) | 17:17 |
| rloo | devananda: is that documented anywhere. What you just described as ironic's last minor relase .. becoming the basis for the Named release? I think it is important. | 17:18 |
| devananda | we provide stable branch support for 3 cycles after a named release | 17:18 |
| rloo | yeah, sambetts, stop confusing me :) | 17:18 |
| devananda | rloo: yea, it was part of the spec for our switch to numbered releases | 17:18 |
| sambetts | rloo: ;) | 17:18 |
| *** ChubYann has joined #openstack-ironic | 17:18 | |
| devananda | and its documented in the openstack gov. docs for "release-cycle-with-intermediary" projects | 17:18 |
| sambetts | trying to get my head around it so I can implement it on my networking-cisco repo | 17:18 |
| devananda | this is also how swift manages their releases | 17:18 |
| rloo | devananda: ok, good. all this terminology... if you don't mind, deva, i'll paraphrase/copy what you just said, into tan's spec. | 17:19 |
| devananda | however, I do not like the way we've recently discussed cutting major releases frequently | 17:19 |
| *** fragatina has quit IRC | 17:19 | |
| devananda | it goes against everything I know in software to bump major version numbers when there is really not a significant change in the project | 17:19 |
| *** garthb_ has joined #openstack-ironic | 17:19 | |
| devananda | rloo: link for context? (also thanks) | 17:20 |
| rloo | devananda: https://review.openstack.org/#/c/299245 | 17:20 |
| * sambetts is heading out | 17:20 | |
| *** sambetts is now known as sambetts|afk | 17:20 | |
| devananda | some links that might be helpful here: | 17:20 |
| devananda | https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst | 17:21 |
| *** garthb has quit IRC | 17:21 | |
| devananda | https://github.com/openstack/governance/blob/master/reference/tags/assert_follows-standard-deprecation.rst | 17:21 |
| devananda | https://github.com/openstack/governance/blob/master/reference/tags/release_cycle-with-intermediary.rst | 17:21 |
| NobodyCam | night sambetts|afk | 17:21 |
| dtantsur | devananda, lemme interrupt you please for a second before I go for a day: do the ideas on the etherpad look remotely sane? I'd put them to the spec, if it looks like something we could have | 17:21 |
| devananda | dtantsur: I do not follow the 'pad :( | 17:22 |
| devananda | dtantsur: it brings up many questions for me, and seems different from what the spec was proposing | 17:22 |
| *** cdearborn has quit IRC | 17:22 | |
| dtantsur | devananda, well, yes, that's what I took from the comments re hardware types, it's not a sum of the spec | 17:22 |
| devananda | ok | 17:22 |
| * dtantsur is good at confusing people as well, I guess | 17:22 | |
| dtantsur | I just wanted to use etherpad as a quick medium to share the very idea, then dive into longer explanations on the spec itself | 17:23 |
| devananda | gotcha | 17:23 |
| devananda | I'll put some more comments in the pad then | 17:23 |
| devananda | I was reading it as "outline for the summit discussion" | 17:24 |
| dtantsur | wannabe outline for the summit discussion :D | 17:24 |
| dtantsur | sorry, I've been thinking about this for so long, that I assume that everything is already obvious for everyone :) | 17:25 |
| devananda | dtantsur: heh, yea, it can be hard to write down one's assumptions | 17:25 |
| devananda | I will be happy to help by asking questions :) | 17:25 |
| dtantsur | sure :) I have some time here still, so you can ping me on IRC, or just dump everything to the etherpad | 17:26 |
| * devananda turns off irc notifications for a bit and digs into the 'pad | 17:26 | |
| *** chopmann has joined #openstack-ironic | 17:27 | |
| *** fhubik has quit IRC | 17:28 | |
| * dtantsur looks at letters appearing on the etherpad | 17:29 | |
| vdrok | lucasagomes: it seems to me that we can check with channel.exit_status_ready() instead of channel.closed | 17:35 |
| lucasagomes | vdrok, yeah or, eof_received | 17:36 |
| lucasagomes | vdrok, I'm talking to the paramiko guys at #paramiko | 17:36 |
| lucasagomes | vdrok, for context: http://paste.openstack.org/show/494696/ | 17:38 |
| vdrok | lucasagomes: thx :) | 17:38 |
| lucasagomes | tl;dr according to the paramiko developers it's not very reliable to look at the closed property since it does not get set to True as part of the exec_command stack | 17:39 |
| *** causten_ has joined #openstack-ironic | 17:40 | |
| *** ChrisAusten has quit IRC | 17:40 | |
| *** ohamada has quit IRC | 17:41 | |
| jroll | devananda | we provide stable branch support for 3 cycles after a named release <- this may change | 17:44 |
| jroll | devananda | it goes against everything I know in software to bump major version numbers when there is really not a significant change in the project <- I disagree, see also chrome and firefox | 17:44 |
| *** electrofelix has quit IRC | 17:44 | |
| jroll | devananda: other than that, I agree with your assertions about what live upgrades includes | 17:45 |
| *** athomas has quit IRC | 17:45 | |
| jroll | rloo: ^ | 17:45 |
| rloo | jroll: glad you agree. | 17:45 |
| rloo | jroll: am wondering what that means wrt upgrading Named->Named+1, vs Major->Major+1, when Named->Named+1 could include Major->Major+2 | 17:46 |
| *** fragatina has joined #openstack-ironic | 17:47 | |
| jroll | devananda: though, wrt major version bumps, I suppose it depends how you define "significant change"; however semver's definition is "anything that breaks a contract is a major version", and I believe dropping a deprecated config or feature meets the criteria of breaking a contract | 17:47 |
| jroll | devananda: but I'm going to send a mail on this today/tomorrow, so I don't want a large discussion here right now :) | 17:47 |
| jroll | rloo: here's what I think we should test, which also defines what we support: | 17:47 |
| devananda | jroll: ++ to discussion, ++ to it not happening here | 17:48 |
| rloo | jroll: if we add new db, or remove something from db, is that a major bump? | 17:48 |
| jroll | * last stable (named) release to master | 17:48 |
| jroll | * last release (whether x or y in x.y.z) to master | 17:48 |
| vdrok | lucasagomes: otoh, I'm not sure we should check anything regarding the channel - recv_exit_status waits self.status_event, if the function completed, then channel is closed? | 17:48 |
| dtantsur | we should probably include moar operators in the discussion | 17:48 |
| jroll | rloo: and I think if both of those stay green, that will imply anything in between | 17:48 |
| devananda | jroll: and fwiw, I'm completely OK disagreeing on how often we bump major versions. I know some projects chose to do that often. It's valid and reasonable and fine, and yet my gut still doesn't like it :) | 17:48 |
| rloo | jroll, what about stable-1 -> last-stable? | 17:49 |
| rloo | jroll: stable == Named release, right? | 17:49 |
| jroll | rloo: those will be tested in last-stable CI, yes | 17:49 |
| jroll | yes | 17:49 |
| jroll | a change on master cannot break stable-1 -> stable-0 | 17:49 |
| lucasagomes | vdrok, do you think we are better off by just removing that check? | 17:49 |
| lucasagomes | vdrok, https://review.openstack.org/#/c/307952 does work for me locally | 17:49 |
| rloo | jroll: OK, so we also need to test stable-1 -> stable-0 | 17:50 |
| lucasagomes | with the exit_status_ready() | 17:50 |
| jroll | rloo: yes, in stable-0 CI (and stable-1?) | 17:50 |
| lucasagomes | vdrok, I've moved the recv_exit_status() to be invoked after it | 17:50 |
| jroll | rloo: so for instance a stable/mitaka change would run liberty->mitaka upgrade | 17:50 |
| vdrok | lucasagomes: oh, yep, that makes sense :) | 17:50 |
| lucasagomes | vdrok, that worked for me locally | 17:50 |
| rloo | jroll: ok, that makes sense. | 17:50 |
| lucasagomes | jroll, ^ should fix the gate | 17:51 |
| lucasagomes | (don't need to rewrite) | 17:51 |
| jroll | rloo: re: DB changes being a major bump, I think that if an operator needs to do something (like run a script to do a data migration), it should be a major bump. the rest, I'm not sure; I'd want to talk with the release team at a minimum about it | 17:52 |
| *** ndipanov has quit IRC | 17:52 | |
| jroll | lucasagomes: wow, nice | 17:52 |
| jroll | mtreinish: that paramiko hang thing: https://review.openstack.org/#/c/307952/1 | 17:52 |
| rloo | jroll: so I am asking these questions in the context of what I *thought* would be an easy live-upgrade spec: https://review.openstack.org/#/c/299245 | 17:52 |
| jroll | if you're curious | 17:52 |
| lucasagomes | mtreinish, for context, I was talking to the paramiko devs as well http://paste.openstack.org/show/494696/ | 17:53 |
| rloo | jroll: so want to make sure we all know what we mean by supporting live upgrades :) | 17:53 |
| jroll | rloo: totally, thank you for asking this, I agree it should be part of the spec | 17:53 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/ironic: DO NOT MERGE test gate with fix in tempest https://review.openstack.org/307705 | 17:54 |
| lucasagomes | this should test it for us ^ but will take a while since the gate is very slow | 17:54 |
| lucasagomes | that said, I will call it a day | 17:54 |
| vdrok | lucasagomes: hmm, not working for me :( | 17:54 |
| rloo | jroll: is it worthwhile bringing it up in email. Otherwise, i am concerned we might have this discussion during the design summit (defining what we want), vs how to design what we want. | 17:54 |
| jroll | rloo: do note that the intermediate releases are not branched. so if we have 1.0.0, 1.1.0, 2.0.0, 2.1.0, 2.2.0 all in the same cycle, we don't need CI between e.g. 1.1.0 -> 2.0.0, because neither of those releases can change | 17:54 |
| lucasagomes | vdrok, oh, hanging? | 17:55 |
| NobodyCam | awesome thank you lucasagomes and have a good night | 17:55 |
| vdrok | lucasagomes: yep | 17:55 |
| vdrok | lucasagomes: I'll try to do smth, good night :) | 17:55 |
| rloo | jroll: so the live upgrade process, should it work from 1.0.0 -> 2.0.0? | 17:55 |
| jroll | rloo: yes, I think it's worth an email, I'd like to hear opinions on it | 17:55 |
| devananda | jroll: fwiw, I believe you're describing minor version bumps | 17:55 |
| jroll | rloo: right, that's the main open question, I think | 17:55 |
| devananda | jroll: patch version should be reserved for fixes that do not require operator actions, but minor version bumps, IMO, are fine | 17:56 |
| rloo | jroll: yeah, cuz to work from 1.0.0 -> 2.0.0, and 0.0.9 (if named release) -> 2.2.0 (named release) is different | 17:56 |
| devananda | major version bumps should indicate significant changes | 17:56 |
| jroll | rloo: rather, my main open question is "do we test upgrades for every existing release in the current cycle to master?" | 17:57 |
| lucasagomes | vdrok, odd, http://paste.openstack.org/show/494699/ | 17:57 |
| vdrok | lucasagomes: if not move recv_exit_status, all good | 17:57 |
| vdrok | lucasagomes: I mean leaving it in place it was | 17:57 |
| rloo | jroll: i actually want to think of it another way. because it seems like we test what we'd like to work. Not 'if we test it, that's the behaviour'. | 17:57 |
| jroll | rloo: imo test == support | 17:58 |
| rloo | jroll: yes, support == test :) | 17:58 |
| rloo | jroll: ie, we have features we support, but aren't tested (yet) | 17:58 |
| lucasagomes | vdrok, very odd, I would think that checking if it's ready() before calling recv() was the way to ggo | 17:58 |
| lucasagomes | go* | 17:58 |
| jroll | rloo: right :( | 17:58 |
| vdrok | lucasagomes: I did too | 17:58 |
| mtreinish | jroll: ok, I'll take a look | 17:58 |
| jroll | rloo: "do we support upgrades for every existing release in the current cycle to master?" | 17:58 |
| vdrok | otherwise does not make sense to add channel check | 17:58 |
| rloo | jroll: so really, i'd just like to support live upgrades from Named -> Named+1 | 17:59 |
| vdrok | lucasagomes: I think I'll add a different patch, removing channel.closed check and a depending patch in ironic, will see tomorrow for sure :) | 17:59 |
| jroll | rloo: and not from Named+.5 -> Named+.75? | 17:59 |
| rloo | jroll: yup, not that. | 17:59 |
| *** Sukhdev has joined #openstack-ironic | 18:00 | |
| JayF | That's rough. | 18:00 |
| JayF | Basically encourages/requires any serious deployers to stick to releases | 18:00 |
| jroll | rloo: seems like that should be trivial if we get named->named+1 working, no? | 18:00 |
| devananda | rloo: Named -> Named+1 && Named -> Master && (some SHA more recent than last Named) -> Master | 18:00 |
| devananda | rloo: that is what I would like | 18:00 |
| lucasagomes | vdrok, yeah just removing works as well | 18:00 |
| devananda | because some of us deploy from master, and some of us do not | 18:00 |
| *** piet has quit IRC | 18:01 | |
| jroll | rloo: I guess this is totally worth a ML post :) | 18:01 |
| * mgould -> home; good night! | 18:01 | |
| devananda | rloo: numbered releases which are not also named releases is, essentially, the same thing as an arbitrary SHA | 18:01 |
| vdrok | night mgould | 18:01 |
| lucasagomes | vdrok, oh hanged here | 18:01 |
| jroll | devananda: back to semver - do you agree that anything requiring operator intervention is not a backwards compatible change? | 18:01 |
| lucasagomes | ok let's remove .closed, | 18:02 |
| *** garthb__ has joined #openstack-ironic | 18:02 | |
| vdrok | lucasagomes: some weird shit going on :D | 18:02 |
| lucasagomes | lol hell | 18:02 |
| rloo | ok, will send out an email later. (in meeting now). | 18:02 |
| jroll | mtreinish: I guess it sounds like this isn't quite there :P | 18:02 |
| *** garthb_ has quit IRC | 18:02 | |
| devananda | jroll: upgrade process and backwards compatibility are different things | 18:02 |
| jroll | rloo: thanks | 18:02 |
| *** mgould has quit IRC | 18:02 | |
| jroll | devananda: kinda. sometimes. | 18:02 |
| lucasagomes | vdrok, there's another attribute "eol_received" | 18:02 |
| devananda | jroll: so, no, I do not agree with that statement taken in isolation | 18:03 |
| jroll | devananda: "is this version of ironic backwards compatible with the previous version's config file?" | 18:03 |
| lucasagomes | not sure if we should test that one | 18:03 |
| *** mtreinish has quit IRC | 18:03 | |
| devananda | jroll: removing some deprecated options does not REQUIRE operator intervention during an upgrade | 18:03 |
| devananda | post-upgrade, ironic will just ignore those lines | 18:03 |
| devananda | (right??!?) | 18:03 |
| jroll | devananda: right, and break their expected functionality | 18:03 |
| devananda | uh, no... | 18:04 |
| devananda | well | 18:04 |
| devananda | not necessarily | 18:04 |
| jroll | devananda: if I set make_unicorns=True, the make_unicorns config is removed and has a new name, and ironic suddenyl doesn't make unicorns, I'll be upset | 18:04 |
| devananda | if they already updated to using the new / non-deprecated option names, then nothing bad happens at all | 18:04 |
| vdrok | lucasagomes: dunno, I just look at recv_exit_status and exit_status_ready docstrings, and they say that whatever was going on is done | 18:04 |
| jroll | right, it requires intervention before or during the upgrade | 18:04 |
| devananda | that intervention could have been done 6mo prior | 18:04 |
| devananda | when we introduced the deprecation warning | 18:04 |
| jroll | sure, but it still is not backwards compatible with a config file that worked in $previous | 18:05 |
| jroll | "Configuration from release N-1 is supported in release N. Sane defaults for new configuration variables are provided in such a way that deployed code from N can be expected to run without operator intervention." | 18:05 |
| JayF | As an operator, if I used something that was semver, and the first number didn't change, and it broke because I didn't update my config file, I'd be upset | 18:05 |
| jlvillal | Isn't that what being deprecated means? It will go away and if you don't change at some point your things will break. | 18:05 |
| jroll | from http://governance.openstack.org/reference/tags/assert_supports-upgrade.html | 18:05 |
| vdrok | lucasagomes: but seems that eof_recieved is more reliable, maybe worth adding | 18:05 |
| devananda | jroll: this is where one needs to define exactly by what metric you are judging "backwards compatible" | 18:05 |
| lucasagomes | vdrok, updated the patch, let's see what reviwers had to say | 18:05 |
| lucasagomes | vdrok, yeah, that gets set as part of the exec_command() stack | 18:06 |
| lucasagomes | at least by looking at paramiko's code | 18:06 |
| jroll | devananda: yep, I agree | 18:06 |
| * devananda rereads requirements on that page | 18:06 | |
| lucasagomes | vdrok, for now I've just removed closed, checking if no more data was returned seems to be fine | 18:06 |
| jroll | devananda: I'm not saying you're wrong, I think we have a different position of the line here | 18:06 |
| lucasagomes | that's a socket anyway | 18:06 |
| *** fragatina has quit IRC | 18:07 | |
| jroll | devananda: and we all need to agree what makes sense here | 18:07 |
| *** ijw has joined #openstack-ironic | 18:07 | |
| devananda | jroll: so, my read of that paragraph is that non-deprecated configs from release N-1 are still supported in release N | 18:07 |
| jroll | devananda: that requirement can be read to... yeah what you said | 18:08 |
| devananda | jroll: not that things in N-1 which generated deprecation warnings are still there | 18:08 |
| devananda | because otherwise it would require N-* to still be supported | 18:08 |
| jroll | right | 18:08 |
| devananda | and nothing that was deprecated could ever be removed | 18:08 |
| devananda | and I do not think that is what it means | 18:08 |
| jroll | this page does not describe semver | 18:08 |
| devananda | no, it does not :) | 18:08 |
| *** mtreinish has joined #openstack-ironic | 18:08 | |
| devananda | because most openstack services are not actually following semver | 18:08 |
| jroll | well, they are actually | 18:09 |
| *** chopmann has left #openstack-ironic | 18:09 | |
| jroll | they just promise to release a major version every 6 months | 18:09 |
| jroll | and no versions in between | 18:09 |
| devananda | even if that version does not have any major changes, they bump the major version now | 18:09 |
| jroll | so they never need y or z except in stable branches | 18:09 |
| devananda | so it's not semver :) | 18:09 |
| jroll | sure, I guess, but that's with the assumption something will happen that requires it | 18:09 |
| lucasagomes | vdrok, ok calling it a day, let's see if all works tomorrow | 18:10 |
| vdrok | lucasagomes: thanks for all the findings! :) | 18:10 |
| lucasagomes | vdrok, thank you for all the help! | 18:10 |
| vdrok | lucasagomes: good night | 18:10 |
| lucasagomes | have a great evening all | 18:10 |
| * vdrok leaves too | 18:10 | |
| jroll | devananda: fwiw oslo libraries *do* follow semver, and *do* bump major version frequently, including when they drop a deprecated config | 18:10 |
| *** lucasagomes is now known as lucas-dinner | 18:10 | |
| devananda | jroll: totally | 18:13 |
| devananda | jroll: so again, my gut doesn't like bumping ironic (the service) major version super frequently, but logically, I agree that it is the right thing to do when any of those ^^ conditions are met | 18:14 |
| *** ijw has quit IRC | 18:15 | |
| jroll | devananda: right, software has a history of bumping major versions when major functionality is released | 18:15 |
| *** fragatina has joined #openstack-ironic | 18:15 | |
| JayF | honestly the interesting thing for us | 18:16 |
| jroll | and only when | 18:16 |
| dtantsur | devananda, thanks for the conversation on the etherpad, very helpful! I'm leaving it to you now, tomorrow will try to update the spec | 18:16 |
| JayF | is if we implement "major" features well | 18:16 |
| jroll | and I disagree that is always the right thing to do | 18:16 |
| JayF | it shouldn't even need a major version bump | 18:16 |
| jroll | JayF: yep | 18:16 |
| jroll | ooooo. "Deprecating existing functionality is a normal part of software development and is often required to make forward progress. When you deprecate part of your public API, you should do two things: (1) update your documentation to let users know about the change, (2) issue a new minor release with the deprecation in place. Before you completely remove the functionality in a new major release | 18:17 |
| jroll | there should be at least one minor release that contains the deprecation so that users can smoothly transition to the new API." | 18:17 |
| *** ChrisAusten has joined #openstack-ironic | 18:17 | |
| *** causten_ has quit IRC | 18:17 | |
| *** ijw has joined #openstack-ironic | 18:17 | |
| jroll | from the faq on http://semver.org/ | 18:17 |
| dtantsur | now calling it a day, see you | 18:17 |
| jroll | so it's really "what's our public API" - and I think that the config file is part of that | 18:17 |
| jroll | good night dtantsur, lucas-dinner | 18:18 |
| *** dtantsur is now known as dtantsur|afk | 18:18 | |
| JayF | you know what's funny | 18:18 |
| JayF | I bet there are a lot of operators who interact more with our config file as an "api" | 18:18 |
| JayF | than who interact with our api as an api | 18:18 |
| JayF | (meaning; op would care about the details of the config, but if using ironicclient cli, wouldn't careabout the details of the api) | 18:19 |
| *** krtaylor has quit IRC | 18:20 | |
| jroll | yeah | 18:20 |
| devananda | jroll: off the cuff, I think our "public API" is: config file, driver interfaces and entrypoints, and REST API | 18:24 |
| *** jaybeale has joined #openstack-ironic | 18:24 | |
| jroll | devananda: I agree with those, I wonder if there's more | 18:26 |
| jroll | but I'm too hungry to think about it properly | 18:26 |
| jroll | so, I'll bbl :) | 18:26 |
| *** krtaylor has joined #openstack-ironic | 18:30 | |
| *** moshele has quit IRC | 18:40 | |
| *** trown|afk is now known as trown | 18:43 | |
| *** Sukhdev has quit IRC | 19:00 | |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add system X firmware information https://review.openstack.org/307393 | 19:00 |
| *** fragatina has quit IRC | 19:01 | |
| *** fragatina has joined #openstack-ironic | 19:02 | |
| *** fragatina has quit IRC | 19:02 | |
| devananda | mmm, hungry too | 19:08 |
| devananda | also, I have put a lot of things on https://etherpad.openstack.org/p/ironic-newton-summit-driver-composition | 19:08 |
| *** Sukhdev has joined #openstack-ironic | 19:15 | |
| *** Sukhdev has quit IRC | 19:18 | |
| *** Sukhdev has joined #openstack-ironic | 19:19 | |
| openstackgerrit | Xavier proposed openstack/ironic: Add Dynamic Allocation feature for the OneView drivers https://review.openstack.org/286192 | 19:31 |
| *** mbound has joined #openstack-ironic | 19:32 | |
| *** Sukhdev has quit IRC | 19:36 | |
| *** Sukhdev has joined #openstack-ironic | 19:41 | |
| *** mtreinish has quit IRC | 19:41 | |
| *** mbound has quit IRC | 19:42 | |
| *** mtreinish has joined #openstack-ironic | 19:42 | |
| devananda | TheJulia: finally got around to filing this RFE in bifrost: https://bugs.launchpad.net/bifrost/+bug/1572227 | 19:42 |
| openstack | Launchpad bug 1572227 in bifrost "[RFE] bifrost does not assign IP to BMCs" [Undecided,New] | 19:42 |
| devananda | TheJulia: I hacked out some code to do that, but havent made time to polish it yet. if that's a thing more folks will want, pls LMK and I'll make time to push the code up | 19:43 |
| cinerama | i could see that being useful | 19:44 |
| TheJulia | devananda: I seem to remember a few people expressing that something like that would be awesome | 19:44 |
| *** baoli has quit IRC | 19:44 | |
| NobodyCam | devananda: that actually a really cool thing | 19:46 |
| NobodyCam | with the code you have are the bmc's on a different subnet then the nodes ? | 19:46 |
| devananda | NobodyCam: yes. physically separate net | 19:49 |
| NobodyCam | :) | 19:49 |
| *** mtreinish has quit IRC | 19:49 | |
| *** mkovacik has joined #openstack-ironic | 19:50 | |
| devananda | I've defined two env vars for bifrost: ipmi_net_iface and mgmt_net_iface to indicate which network devices the different dhcp configs should be applied to | 19:50 |
| devananda | I'm making the assumption that IPMI and PXE happen on different nets, whether logical or physical, and that the host running bifrost has an interface to each | 19:51 |
| NobodyCam | wonder if that works on systems that share Ethernet / and bmc on the same physical port? | 19:52 |
| TheJulia | the only case that I can think of them being the same is AMT | 19:52 |
| NobodyCam | IBM system x | 19:52 |
| jroll | devananda: we do this in our deployment, I like | 19:52 |
| NobodyCam | I think supermicro did it for awhile | 19:52 |
| jroll | (statically assign BMC IPs, that is) | 19:53 |
| devananda | jroll: what do you use to assign the IPs? | 19:53 |
| *** Sukhdev has quit IRC | 19:53 | |
| jroll | devananda: a for loop over our IP block for that cell :P | 19:53 |
| devananda | heh | 19:53 |
| devananda | I mean - are you configuring hte IP statically in the BMC, or letting it DHCP ? | 19:53 |
| jroll | oh, dhcp | 19:54 |
| devananda | cool | 19:54 |
| *** ChrisAusten has quit IRC | 19:54 | |
| jroll | separate dhcp server for BMCs only, in our case, but same idea | 19:54 |
| devananda | yea, well, that's the approach I took too -- separate dhcp server (isc-dhcpd instead of dnsmasq) | 19:54 |
| jroll | ah | 19:54 |
| devananda | whether it's colocated with bifrost's dnsmasq process then doesn't matter | 19:54 |
| jroll | yep | 19:54 |
| jroll | just set a different host for that role if you want it to be totally separate, I assume | 19:55 |
| devananda | yah | 19:55 |
| jroll | I disagree you should force a name | 19:55 |
| jroll | but like, -0.1 disagree | 19:55 |
| jroll | anyone that doesn't want one can just generate ints or something | 19:55 |
| jroll | devananda: actually, you say you use the name as the hostname for when it's deployed, why is that related to the bmc? or is that the link to the thing that defines what to deploy? | 19:57 |
| devananda | it's a link all the way down the chain, yea | 19:57 |
| devananda | I use that name when enrolling the node in ironic | 19:57 |
| devananda | here's the flow: | 19:57 |
| devananda | - dhcp sets the BMC IP | 19:57 |
| devananda | - enroll server in ironic, with no NICs and no properties | 19:57 |
| devananda | - run inspector against all nodes, which (by default) checks the BMC IP against ironic's list of known nodes | 19:58 |
| devananda | - inspector updates node.properties and creates ports | 19:58 |
| devananda | - now bifrost's dynamic inventory module can deploy the node! yea! | 19:58 |
| jroll | nice | 19:59 |
| devananda | - bifrost generates config drive , which contains hostname generated from node.name | 19:59 |
| jroll | so for me, the RFE reads as "additionally configure dhcp for the BMCs", which does not sound like the goal :P | 20:00 |
| devananda | correct | 20:00 |
| jroll | but maybe this is just one step toward that, I guess? | 20:00 |
| devananda | well, the rest is already doable | 20:00 |
| jroll | ah, I see | 20:01 |
| devananda | it's not documented well - but all the capabilities are in place to do the rest from the CLI/API | 20:01 |
| devananda | just need the first step -- hand out IPs to BMCs | 20:01 |
| jroll | so what's the input to the rest before this? a list of nodes to enroll, including their bmc info? | 20:01 |
| devananda | a file containing that BMCS variable, with a lits of Nodes | 20:02 |
| devananda | *list | 20:02 |
| JayF | So to ask a tangental question: I know inspector is working on wildcard dhcp | 20:02 |
| jroll | devananda: well, before this would just be the latter, right? | 20:02 |
| JayF | would this be smoething you'd ever want to support for BMCs? | 20:02 |
| *** piet has joined #openstack-ironic | 20:02 | |
| devananda | JayF: hypothetically, a DHCP request on the IPMI network from an unknown MAC _could_ be handed an IP and a Node created in Ironic immediately for that BMC | 20:03 |
| devananda | JayF: and ironic could either default to a likely driver, or try to determine the driver based on interrogating the BMC | 20:03 |
| JayF | or even if you could prepopulate ironic with some other matching info | 20:03 |
| JayF | like the mac addresses | 20:03 |
| jroll | devananda: anyway, I wonder why not just add a property or two to the node, manage_bmc_dhcp: True, bmc_mac: foo | 20:04 |
| devananda | JayF: and then initiate inspection on it -- asusming it was able to guess the BMC creds correctly | 20:04 |
| devananda | jroll: you mean add those properties in Ironic? | 20:04 |
| jroll | devananda: bifrost | 20:04 |
| NobodyCam | seems like it would a "type" of port attached to the node | 20:04 |
| devananda | bifrost doesn't have a data model for nodes ... | 20:04 |
| devananda | it uses Ironic as the data store | 20:04 |
| jroll | JayF: I do wonder how the wildcard dhcp thing works (I assume like regular dhcp, dynamic assignments), and if that would end in the bmc eventually getting a different IP | 20:05 |
| jroll | devananda: there's an initial list of nodes, no? | 20:05 |
| devananda | * the way I use bifrost does not have a data model | 20:05 |
| devananda | no | 20:05 |
| devananda | I mean, yes, but not for me | 20:05 |
| JayF | jroll: if I were an operator doing that, I'd setup a DHCP range that was distinct from the static range | 20:05 |
| devananda | bifrost supports passing in a JSON or YAML file with a list of nodes | 20:05 |
| JayF | jroll: wildcard stuff ets a "temporary" BMC ip, you set a perm one statically in-band in a diff subnet | 20:05 |
| devananda | it also supports a dynamic inventory plugin that is basically "poll ironic" | 20:05 |
| devananda | that is the mode I use | 20:05 |
| devananda | and so there is an initial step: populate ironic | 20:05 |
| jroll | devananda: aha, okay | 20:06 |
| jroll | devananda: still, this is essentially a list of nodes (with minimal info) | 20:06 |
| devananda | yep | 20:06 |
| jroll | so I'm proposing use the normal path of a list of nodes, with this info | 20:06 |
| jroll | instead of a different code path of passing a BMCS variable | 20:07 |
| devananda | jroll: how would that work when I do not have the other information about nodes that is usually included in that list? | 20:07 |
| devananda | like - I do not know the NICs, or how much RAM is in the host, or anything else about it | 20:07 |
| jroll | devananda: well, if manage_bmc_dhcp==True: dont_require_other_stuff_and_enroll_instead() | 20:07 |
| devananda | jroll: that file, however, is static | 20:07 |
| jroll | the separate thing/codepath you're proposing might be more straightforward, but I need to ask the question | 20:08 |
| jroll | ah | 20:08 |
| devananda | like - bifrost isn't going to edit its inventory file | 20:08 |
| jroll | ya | 20:08 |
| jroll | fair enough | 20:08 |
| devananda | so I wnt to use this in an environment where I do not have a static inventory file | 20:08 |
| devananda | it's a fair question, but ^ is why :) | 20:08 |
| jroll | right, makes sense :) | 20:08 |
| devananda | also, for batch loads of new hardware, I might have several files defining the BMCS variable, one for each import | 20:09 |
| devananda | to run different inputs through the same playbooks each time | 20:09 |
| devananda | I could, of course, do some pre-run checking against one large file: if node exists in ironic: skip | 20:10 |
| *** Goneri has quit IRC | 20:10 | |
| jroll | yep, totally fair | 20:11 |
| jlvillal | Anyone know where the 2 out of 8 for Ironic in Maturity in https://www.openstack.org/software/project-navigator/ | 20:11 |
| JayF | jlvillal: https://www.openstack.org/software/releases/liberty/components/ironic has a dig down | 20:11 |
| JayF | jlvillal: apparently not having an install guide on docs.openstack.org, % of deployments using Ironic, etc etc | 20:12 |
| *** daemontool has quit IRC | 20:12 | |
| JayF | lots of the info is wrong though | 20:12 |
| JayF | Does this project follows standard deprecation? (No) | 20:12 |
| JayF | which is true for us | 20:12 |
| JayF | and it has marked that we don't support upgrade | 20:12 |
| jlvillal | JayF: Yeah I think there is wrong info. | 20:12 |
| devananda | much wrong info | 20:13 |
| jroll | we don't support upgrade | 20:13 |
| JayF | So who manages that? | 20:13 |
| devananda | I've sent a few patches up in the past releases, but it seems like the info gets reset to being wrong with each new release | 20:13 |
| JayF | jroll: we don't support upgrades at all? | 20:13 |
| devananda | JayF: tom fifield | 20:13 |
| jroll | JayF: nope | 20:13 |
| jroll | JayF: it isn't tested | 20:13 |
| JayF | devananda: that's not a link to a code repo :( | 20:13 |
| jroll | which is why grenade is so important | 20:13 |
| devananda | JayF: one sec | 20:14 |
| jroll | also, I believe those last 3 come from tags in the governance repo | 20:14 |
| jroll | which is | 20:14 |
| jroll | https://github.com/openstack/governance/blob/master/reference/projects.yaml#L1913 | 20:14 |
| JayF | and it has "Version N/A" for all our api versions | 20:14 |
| JayF | lol | 20:14 |
| devananda | JayF: ops-tags-team | 20:14 |
| jroll | we *do* need to add the standard deprecations tag; I rabbit-holed on fixing the tag itself and forgot to come back to the tag | 20:14 |
| devananda | so that data come from a combination of places, but then gets curated in the openstack/ops-tags-team repo | 20:15 |
| jroll | ah | 20:15 |
| NobodyCam | ya and only having 2 sdk's not seven | 20:15 |
| jroll | are we in seven sdks? | 20:15 |
| devananda | not afaik | 20:15 |
| devananda | shade and OSC | 20:16 |
| devananda | that list doesn't count the project's python clients, iirc | 20:16 |
| jroll | oh, I read that as saying it was wrong | 20:16 |
| NobodyCam | no in we're in 2 | 20:16 |
| jroll | right | 20:16 |
| devananda | does anyone know of clients in non-python languages? | 20:16 |
| jroll | nope | 20:16 |
| jroll | there's a super limited crappy nodejs client that I won't show you >.> | 20:17 |
| jroll | I shouldn't say it's crappy, it works fine | 20:17 |
| devananda | oh! we have a nodejs client | 20:17 |
| jroll | but it only does node-list and node-show :P | 20:17 |
| devananda | ironic-webui | 20:17 |
| jroll | that isn't an sdk, though | 20:17 |
| jroll | that's just an app that happens to hit the api | 20:17 |
| devananda | it's got a client library in it | 20:17 |
| devananda | that could be reused, right? ;) | 20:17 |
| jroll | can you install it and use it in some other app? | 20:17 |
| jroll | heh, yeah | 20:17 |
| jroll | we'd need to pull it out of that repo, I think | 20:18 |
| devananda | ermm | 20:18 |
| jlvillal | I asked about it, since my boss asked me. And I told her, I had never heard of that page before | 20:23 |
| openstackgerrit | Merged openstack/ironic: Add note on prerequisite of 'rpm' file extraction https://review.openstack.org/306247 | 20:25 |
| NobodyCam | merged? is the gate working again? | 20:26 |
| JayF | that appears to have been docs only | 20:27 |
| JayF | but it did run, which means things are catching up | 20:27 |
| NobodyCam | :) | 20:28 |
| *** ChrisAusten has joined #openstack-ironic | 20:28 | |
| *** fragatina has joined #openstack-ironic | 20:30 | |
| krtaylor | betherly, jroll - re: Ironic social - Thursday night 6:30pm, full info on the summit arrivals etherpad | 20:34 |
| *** rloo has quit IRC | 20:34 | |
| jroll | krtaylor: nice, thanks! | 20:34 |
| krtaylor | jroll, no worries, I will confirm tomorrow | 20:34 |
| *** fragatina has quit IRC | 20:34 | |
| *** jayc_ has joined #openstack-ironic | 20:35 | |
| *** klindgren has joined #openstack-ironic | 20:37 | |
| JayF | Looking at the menu for the suggested place, I'm worried that some of the vegetarian folks might not have any choices there. | 20:37 |
| keedya | hi @here, where can i find ironic team conferance and meeting schedule for openstack summit? | 20:37 |
| krtaylor | JayF, yeah, I considered that, but it is really hard to find a veg place that can accomidate 20 people and has good beer | 20:37 |
| krtaylor | keedya, https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Ironic | 20:38 |
| JayF | krtaylor: I know it's hard, just wanted to make sure to say it in channel so that someone like devananda wouldn't show up and find themselves without food options | 20:39 |
| krtaylor | JayF, there are veg options on the menu, but few, and not totally vegan | 20:39 |
| JayF | yep | 20:39 |
| JayF | I can't have gluten or dairy, but pretty much burger-no-bun can cover me | 20:39 |
| JayF | the gf + vegetarian folks have a harder time of it | 20:40 |
| krtaylor | same, it was really hard, I'll keep looking, I confirm with them tomorrow | 20:40 |
| *** dprince has quit IRC | 20:40 | |
| krtaylor | hinestly, it is really hard in Austin downtown to get veg food | 20:40 |
| *** Sukhdev has joined #openstack-ironic | 20:40 | |
| krtaylor | BBQ is king :) | 20:40 |
| JayF | Sure. I'm not saying it's easy, just pointing it out. There's a reason I volunteered to collect info and make the hangout | 20:41 |
| JayF | and did not volunteer to find a place for dinner | 20:41 |
| JayF | lol | 20:41 |
| klindgren | Are there any plans to make openstack-ironic and openstack-inspector projects merge into something more unified? Seems like having to standup a new database + config files + PXE/TFTP environment to handle inspector on-boarding things into ironic is a bit disjointed? | 20:43 |
| JayF | klindgren: I don't believe there are any plans to that effect; at least I haven't seen it proposed by anyone. | 20:43 |
| Madasi | Duchess on the header image? I'm sold | 20:43 |
| JayF | klindgren: and honestly it was an explicit decision to have it be a separate service, I don't think the reasons for that have changed | 20:44 |
| xavierr | and the Oscar for better ironic-summit-arrivals goes to... krtaylor :) | 20:44 |
| klindgren | JayF, What are the reasons for that? | 20:44 |
| Madasi | hmm, but not on the tap list. I feel betrayed now. | 20:45 |
| JayF | klindgren: I was afraid that was the next question; I don't fully remember but I think it had something to do with many ironic users not needing/wanting inspection and keeping the core of ironic slimmer | 20:45 |
| klindgren | Fresh implementation perspective. Seems like Ironic is missing a "provisioning/on boarding network". Right now its like I have to have an onbaording network - thats totally seperate from the production network. Then I need to have a "cleaning network" that seperate from that as well. | 20:46 |
| JayF | Well for the latter part, that code is up awaiting review :) | 20:47 |
| klindgren | Where the system that we are looking at replacing - just does all that in a "staging/provisoning network" and then pivots boxes over to their production network. | 20:47 |
| JayF | For the "on boarding" network, there's two diff approaches to that: | 20:47 |
| JayF | 1) If you use inspection, the network that boots on could 'count' as the onboarding network | 20:48 |
| JayF | or, what we do at onmetal | 20:48 |
| klindgren | Yea I saw that re: the provisioning network. | 20:48 |
| NobodyCam | klindgren: can I ask what system your looking at replacing? | 20:48 |
| JayF | 2) Cleaning is idempotent. All the "onboarding" activities are done to every machine every time, not just at first-onlining | 20:48 |
| *** baoli has joined #openstack-ironic | 20:48 | |
| krtaylor | xavierr, :) | 20:50 |
| *** rloo has joined #openstack-ironic | 20:53 | |
| *** hblixt_ has joined #openstack-ironic | 20:57 | |
| *** [1]cdearborn has quit IRC | 20:59 | |
| *** hblixt has quit IRC | 21:00 | |
| *** mbound has joined #openstack-ironic | 21:01 | |
| *** fragatina has joined #openstack-ironic | 21:02 | |
| klindgren | NobodyCam, home built. Be it currently handles all of our internal infra builds @godaddy. So it handles a large number of systems and onboarding of a large amount of HW. | 21:02 |
| *** dims_ has joined #openstack-ironic | 21:02 | |
| *** dims has quit IRC | 21:02 | |
| NobodyCam | klindgren: ahh thank you :) | 21:02 |
| klindgren | Mainly trying to replace that to handle net-new builds in new infra locations via ironic. Basically trying to get all compute resource to be handled via Openstack. | 21:04 |
| klindgren | But Ironic has a ways to go to match the feature parity that the legacy system is providing. | 21:05 |
| NobodyCam | klindgren: are you going to be at the openstack summit | 21:05 |
| klindgren | yep | 21:06 |
| keedya | krtaylor:thanks:) looking forward to meet you guys | 21:06 |
| NobodyCam | I would love to meet and see where ironic is lacking in feature parity | 21:06 |
| *** baoli_ has joined #openstack-ironic | 21:06 | |
| *** baoli__ has joined #openstack-ironic | 21:08 | |
| NobodyCam | brb | 21:10 |
| klindgren | Sure. I can give you an idea off the top of my head. Mainly around swinging servers to and from production networks/provisioning networks (I know that you guys basically pass to neutron on this - but neutron doesn't have a good answer for this eaither. Further complicated in that we dont run switches from one single vendor, so neutron plugins can't really handle this). Handling bonded nics, Allowing end users to specify r | 21:10 |
| klindgren | aid config/partition config (as in its end user selectable). Handling trunk ports for servers. | 21:10 |
| *** baoli has quit IRC | 21:10 | |
| *** Sukhdev has quit IRC | 21:11 | |
| *** baoli__ has quit IRC | 21:11 | |
| *** baoli_ has quit IRC | 21:11 | |
| *** baoli has joined #openstack-ironic | 21:12 | |
| *** trown is now known as trown|outtypewww | 21:12 | |
| *** fragatina has quit IRC | 21:14 | |
| klindgren | JayF - I am trying to understand if the inspection projection is optional - then why is the workflow for onboarding servers - they must go through an "inspection" phase. I realize that I can basically create an ironic node completely via the cli. But the ironic/ironic-inpsection stuff as a guy who's jsut trying to get it going - it seems totally disjointed and I am redoing the same work for inspector that I had to do for ir | 21:16 |
| klindgren | onic. Especially since from an IPA perspective it looks like I can use the same disk image for inspection as well as deployment | 21:16 |
| JayF | klindgren: I don't use inspection, so I'm not really familiar with your pain :/, you should absolutely come to the ops session at the design summit | 21:17 |
| klindgren | What day is that? | 21:17 |
| JayF | klindgren: I run a public bare metal cloud so I think we're basically opposite sides of the use case spectrum, which is always fun :) | 21:17 |
| JayF | uh. | 21:17 |
| JayF | let me check irc logs | 21:18 |
| JayF | https://etherpad.openstack.org/p/ironic-newton-summit | 21:19 |
| JayF | Weds 1:50-2:30 is the make ops less worse session | 21:19 |
| klindgren | I would think that deploying a node on the production network would be eventually a no go from a public bare metal cloud perspective (IE someone could fake the dhcp stuff and screw up new builds) | 21:19 |
| JayF | not sure it'll be talking about use cases so much as making day to day operation of ironic less painful | 21:19 |
| klindgren | TO make it worse - testing some of this stuff out on Kilo - moving to liberty - but I see a lot of the work I care about is much later on | 21:19 |
| JayF | klindgren: we've had network-switching code like what is going upstream now downstream for about 2 years now ... just took this long for all the pieces to fall into place for the open project | 21:20 |
| JayF | you certainly want to be running mitaka if at all possible | 21:20 |
| *** ChrisAusten has quit IRC | 21:21 | |
| klindgren | yea - just dificult because we are currently planning on colocating conductor stuff on a box with some nova-cells stuff. SO having different versions makes things difficult. | 21:21 |
| *** fragatina has joined #openstack-ironic | 21:21 | |
| klindgren | jayf - network-switching code? | 21:21 |
| klindgren | got something I can follow? or you talkin about the "provisioning network" stuff for ironic | 21:22 |
| JayF | that is what I'm talking about | 21:22 |
| JayF | but yes you can follow along | 21:22 |
| JayF | there's a huge set of patchsets up, let me find the series | 21:23 |
| JayF | https://review.openstack.org/#/q/topic:bug/1526403 | 21:23 |
| JayF | https://bugs.launchpad.net/ironic/+bug/1526403 being the RFE for Ironic/Neutron integration | 21:23 |
| openstack | Launchpad bug 1526403 in Ironic "[RFE] Ironic Neutron ML2 Integration" [Wishlist,In progress] - Assigned to Vasyl Saienko (vsaienko) | 21:23 |
| JayF | http://specs.openstack.org/openstack/ironic-specs/specs/approved/ironic-ml2-integration.html is the spec describing the work in progress | 21:24 |
| *** [1]cdearborn has joined #openstack-ironic | 21:25 | |
| *** UForgotten has quit IRC | 21:25 | |
| *** hblixt_ has quit IRC | 21:26 | |
| *** UForgotten has joined #openstack-ironic | 21:29 | |
| *** ChrisAusten has joined #openstack-ironic | 21:32 | |
| *** thrash is now known as thrash|g0ne | 21:33 | |
| klindgren | JayF, I assume their is a neutron side of this? Or is it jsut assuming that a mechanism drive is going to know how to talk to the swithc in question to flip the ports. | 21:34 |
| * devananda reads scrollback | 21:34 | |
| JayF | klindgren: Neutron ML2, ML2 is a switch plugin framework thinger. </technical_explanation> | 21:34 |
| JayF | klindgren: so we tell neutron, via ml2, to do a thing to a switch | 21:34 |
| klindgren | right - but as I understanding it - ml2 then needs a mechinsim driver to know how to do thing to a switch. | 21:35 |
| klindgren | Like creds to login - commands to actually execute ect ect | 21:35 |
| JayF | yes | 21:37 |
| JayF | Those do already exist in some cases | 21:37 |
| klindgren | yea our problems is that some of our ToRs are say arista and some are say cisco | 21:38 |
| JayF | klindgren: https://wiki.openstack.org/wiki/Arista-neutron-ml2-driver for instance | 21:38 |
| JayF | klindgren: there is a plugin for cisco nexus switches too; but honestly I think homogenous switching environmenst will lead to pain here | 21:38 |
| JayF | https://wiki.openstack.org/wiki/Neutron/ML2/MechCiscoNexus | 21:39 |
| JayF | TIL an official one of those existed, not just the one Rackspace wrote | 21:39 |
| klindgren | yea - kk we have our own vlan swinging api. So I assume I could kludge something up to support an external vlan swing thing. | 21:40 |
| JayF | That's probably a pretty good idea. No idea how hard it is :) | 21:41 |
| JayF | I work more on the hardware/agent side of Ironic than coordination with other services | 21:41 |
| JayF | ask me about the agent and I'll be 100x less hand wavy, lol | 21:41 |
| klindgren | bascially was just trying to see if their was something else to handle the vlan swings - or if ironic was defering that capability to neutron | 21:41 |
| JayF | oh absolutely defering it to neutron | 21:41 |
| klindgren | which is 100% the right answer. Just :( for me :-/ | 21:42 |
| JayF | I think the interface we're making mmight be pluggable too though | 21:42 |
| klindgren | lol | 21:42 |
| JayF | yep absolutely is pluggable in ironic too | 21:42 |
| JayF | b/c the patch has noop and neutron as initial drivers | 21:42 |
| JayF | https://review.openstack.org/#/c/285852/39/ironic/drivers/modules/network/flat.py | 21:43 |
| klindgren | awesome sauce | 21:43 |
| JayF | looking at all the methods that has implemented, this will be pretty easy for you | 21:43 |
| JayF | if you already have a python client | 21:43 |
| JayF | you should review all that code if you're interested :P | 21:44 |
| klindgren | yea - going to to do that - thanks. Like I said early seems all the stuff I care about is in Mitaka+ :-/ | 21:45 |
| JayF | Honestly, we didn't even start focusing on the "ironic standalone" use case until about 12-18m ago | 21:46 |
| JayF | assuming you've seen bifrost already as well? | 21:47 |
| klindgren | the other thing that would be nice is if we could define the cleaning/provisoning network per node vs's in the config file for all of ironic. Its complicated for us even further since we run L3 networks. Where each TOR terminate L2 and L3 for that TOR. IE we dont have one vlan spanning everywhere. | 21:47 |
| JayF | no that makes a lot of sense, and probably would be a good second step | 21:48 |
| klindgren | THe other question I have is how is ironic/nova interaction going forward going to work. In kilo atleast - we can't do multiple availability-zone due to the fact that a single nova-compute acts as the proxy for all of ironic | 21:49 |
| klindgren | and we can only add that nova-compute node into a single host agrregate with an az | 21:50 |
| JayF | That's another one that's specced out and high priority | 21:50 |
| JayF | http://specs.openstack.org/openstack/nova-specs/specs/newton/approved/ironic-multiple-compute-hosts.html | 21:50 |
| devananda | klindgren: for the "vlan swinging", see the neutron integration patches. with that work, it should be possible to implement your own plugin to talk to your vlan API instead of neutron | 21:50 |
| klindgren | we abuse currently abuse host aggregates to tell nova which network to select for a VM since we have many many many neutron networks none-of-which span past the TOR. | 21:51 |
| devananda | as far as per-node clean/prov networks, that's ... interesting. I'll file that away and think about it, but don't initially see how it fits into any openstack-based deployment yet | 21:51 |
| JayF | devananda: In a perfect world, we'd dynamically provision L3 networks for each node for every cleaning/deployment | 21:52 |
| devananda | klindgren: what, if any, encapsulation do you use for traffic between VMs whose hosts are connected to different TORs ? | 21:52 |
| JayF | devananda: true network security with all the complexity that would entail :) | 21:52 |
| klindgren | we create a host aggregate per TOR and add those hosts in there with a metadata item of the networks that they support. | 21:53 |
| devananda | JayF: and when someone deletes 5000 instances at once, and neutron tries to create 5k separate networks and attach them all to ironic-conductor's host? | 21:53 |
| klindgren | Neutron is adding support for these types of networks. and the Nova-neutron spec as currently being planned is going to have neutron auto-creating similar host-aggregates in nova | 21:53 |
| devananda | I know that's extreme, but I think there will be issues with connecting that many networks to ironic-conductor | 21:54 |
| klindgren | devananda, we don't do any encapsulation at all. | 21:54 |
| klindgren | all traffic between TOR's is routed | 21:54 |
| devananda | klindgren: I see | 21:54 |
| klindgren | Bascailly we dont guarantee L2 adjacency | 21:55 |
| JayF | devananda: Heh. Didn't think about the other end of that. I can attest you can put that many IPs on a linux box though :). That many virtual network interfaces? IDK :P | 21:55 |
| devananda | klindgren: without L2 connectivity between DHCP service and the bare metal host, are you using OOB channel for deploys? | 21:55 |
| devananda | JayF: yea. it's not the IPs I'm concerned about -- it's the VLANs | 21:55 |
| klindgren | DHCP can be handled via iphelper | 21:55 |
| devananda | klindgren: DHCPBOOT? | 21:56 |
| klindgren | but mainly we are currently handling that by having dhcp servers in the network. Which is trvial to do with VM's we jsut run dhcp service on a few of the compute nodes. | 21:56 |
| devananda | I have little experience with Cisco gear, and none with doing DHCPBOOT via iphelper | 21:56 |
| JayF | devananda: it works, we do it | 21:56 |
| devananda | JayF: sorry, which "it" | 21:56 |
| klindgren | devananda, cisco's term iirc is IPhelp takes braodcast dhcp requests and unicasts it to a centralized set of dhcp servers. | 21:56 |
| JayF | dhcp proxying w/iphelper | 21:56 |
| JayF | It would be tough to get that working with dynamic dhcp though | 21:57 |
| *** ametts has quit IRC | 21:57 | |
| devananda | gotcha | 21:57 |
| JayF | because you can't point the switch to a static dhcp ip | 21:57 |
| JayF | right? Or I guess you could, it'd just have to be the host running the neutron agent? | 21:57 |
| * JayF needs to learn more about the neutron side of the stack | 21:57 | |
| devananda | then yea, everything else can be l3'd, i think | 21:57 |
| *** Sukhdev has joined #openstack-ironic | 21:58 | |
| devananda | klindgren: fwiw, there are some deploy drivers that don't need dhcp at all | 21:59 |
| klindgren | or worse case scenario on the switch side you use vxlan to span all the L2's to a centralized point to consolidate dhcp/metadata (thought metadata can be handled via routes in the network on the switches) | 21:59 |
| klindgren | devananda, I assume those make use of the virtual media stuff? | 21:59 |
| devananda | yep | 21:59 |
| klindgren | as nice as that would be - we have to do the pxe/ipxe/dhcp method :-/ | 22:00 |
| klindgren | since we use some crap dell c series boxes, with about as cheap as you can get BMC. ipmi is the best that we can do from those. | 22:02 |
| *** Sukhdev has quit IRC | 22:03 | |
| devananda | JayF: the current approach for neutron integration, AFAIUI, requires VLANs to be defined in the TORs to connect ironic-api (for ingress traffic) and ironic-conductor (for egress traffic) to the hardware running IPA | 22:03 |
| *** Sukhdev has joined #openstack-ironic | 22:04 | |
| JayF | devananda: yep, in our environment we route all that through in the network to a single "backend" interface on the conductors | 22:04 |
| *** suro-patz has quit IRC | 22:04 | |
| devananda | JayF: since there's a limit to the # of VLANs defined in any given TOR, I would be concerned trying to define one network per Node being cleaned | 22:04 |
| JayF | devananda: which is how I'd imagine most deployers would implement it rather than N interfaces for N networks on the cond | 22:04 |
| devananda | right | 22:04 |
| JayF | 4096 :) | 22:04 |
| devananda | and why, in the current spec, we have 1 VLAN for cleaning and 1 for provisioning | 22:04 |
| JayF | of course, that's until someone writes an ml2 driver to silently implement segmentation as vxlan instead :P | 22:04 |
| * JayF just kidding, he's def not going to do that | 22:05 | |
| devananda | JayF: I think nuage already did (see vidoes from vancouver) | 22:05 |
| JayF | That's not terribly surprising :) | 22:05 |
| JayF | we have that stuff somewhat hooked up now too, if you boot an onmetal server with a cloud network, the cloud network traffic is vxlan encapsulated | 22:06 |
| devananda | right | 22:07 |
| *** Goneri has joined #openstack-ironic | 22:14 | |
| *** mtreinish has joined #openstack-ironic | 22:15 | |
| *** ayoung has quit IRC | 22:16 | |
| *** Sukhdev has quit IRC | 22:16 | |
| *** alex_xu has quit IRC | 22:16 | |
| devananda | huh, I don't see the usual export-to-your-calendar links on the summit schedule | 22:17 |
| *** Sukhdev has joined #openstack-ironic | 22:17 | |
| *** alex_xu has joined #openstack-ironic | 22:18 | |
| *** intr1nsic has quit IRC | 22:20 | |
| jlvillal | jroll: devananda: As an FYI there is one more deprecated options patch outstanding: https://review.openstack.org/307456 | 22:21 |
| *** ianw has quit IRC | 22:22 | |
| jlvillal | Not sure if that matters at all in regards to the planned release. | 22:22 |
| *** ChrisAusten has quit IRC | 22:22 | |
| jlvillal | rama_y is the submitter and a co-worker of mine. | 22:22 |
| *** piet has quit IRC | 22:23 | |
| *** ianw has joined #openstack-ironic | 22:24 | |
| jlvillal | Wow, 607 jobs in the 'check' queue | 22:25 |
| NobodyCam | :( | 22:25 |
| *** intr1nsic has joined #openstack-ironic | 22:26 | |
| *** piet has joined #openstack-ironic | 22:26 | |
| jlvillal | JayF: If you get bored and want to help me figure out what is going wrong. http://www.ironicstack.com/grenade-logs/2016-04-19/ | 22:28 |
| JayF | If you wanna dig in tomorrow, I'm game | 22:29 |
| JayF | right now I'm finishing up my personal summit schedule | 22:30 |
| jlvillal | JayF: Awesome :) | 22:30 |
| devananda | JayF: if you find a way to export that to gCal, pls share | 22:35 |
| *** ianw has quit IRC | 22:38 | |
| *** suro-patz has joined #openstack-ironic | 22:39 | |
| JayF | devananda: I have a gift for you then | 22:40 |
| JayF | devananda: https://twitter.com/boot_vmlinuz/status/719149519972073473 | 22:41 |
| JayF | devananda: two ics files, I saw that link pasted in infra yesterday | 22:41 |
| JayF | I'm doing more like a personal filtered schedule, looking, deciding what to attend and making sure I do my homework for the sessions | 22:41 |
| *** mbound has quit IRC | 22:43 | |
| devananda | JayF: ohgodsno - not the ics for the whole summit | 22:44 |
| devananda | I mean the ics for my personal schedule | 22:44 |
| JayF | that implies there's a built-in way to schedule my own summit | 22:44 |
| devananda | which is way useful since my usual 'where should I be next' thought process is to check the calendar | 22:44 |
| devananda | there is | 22:44 |
| JayF | lol | 22:44 |
| devananda | there has been at every summit | 22:44 |
| JayF | I'm just like, reading the guide | 22:44 |
| JayF | and putting invites on a special calendar for myself | 22:44 |
| devananda | I see | 22:44 |
| JayF | probably not the most efficient but then it'll be stupid-easy to know what I'm going to next | 22:44 |
| devananda | I have in the past gone thorugh the summit schedule, checked the "i'm going here" on things I'm interested in, then gone to the 'export to gcal' link | 22:45 |
| devananda | and added it to google | 22:45 |
| devananda | and hten it just shows up on my phone | 22:45 |
| devananda | it's like magic :) | 22:45 |
| JayF | That sounds amazing | 22:45 |
| JayF | I'm guessing it doesn't work this time though | 22:45 |
| JayF | or else you wouldn't have asked me the question you did | 22:45 |
| devananda | i'm going to be very sad if it doesn't | 22:46 |
| *** persia has quit IRC | 22:46 | |
| *** [1]cdearborn has quit IRC | 22:46 | |
| devananda | ok - apparently it doesn't | 22:48 |
| devananda | because the whole site was rewritten | 22:48 |
| JayF | there is a "Switch to my schedule" link | 22:48 |
| JayF | which if it works on the app might be Good Enough as long as it's locally cached | 22:48 |
| JayF | this "lol my app is just the website" shit is useless though, epseically on conference wifi | 22:49 |
| *** Goneri has quit IRC | 22:51 | |
| devananda | yah | 22:54 |
| devananda | hence my reliance on gcal | 22:54 |
| * TheJulia avoids conference wifi and just relies upon the mifi in her backpack | 22:54 | |
| *** fragatin_ has joined #openstack-ironic | 22:59 | |
| JayF | devananda: so this isn't perfect. .. but "switch to my schedule" -> view and print schedule | 23:01 |
| JayF | devananda: has an option to export to PDF, but is a printable schedule of only things you clicked to add to your schedule ... so if you exported that pdf, you could just check it | 23:02 |
| JayF | not as nice as gcal, but it's functional and downloadable | 23:02 |
| *** fragatina has quit IRC | 23:03 | |
| *** persia has joined #openstack-ironic | 23:04 | |
| *** alex_xu has quit IRC | 23:05 | |
| *** alex_xu has joined #openstack-ironic | 23:06 | |
| *** absubram has quit IRC | 23:13 | |
| *** ianw_ has joined #openstack-ironic | 23:13 | |
| *** david-lyle_ has joined #openstack-ironic | 23:14 | |
| *** david-lyle has quit IRC | 23:14 | |
| *** fragatin_ has quit IRC | 23:18 | |
| *** Sukhdev has quit IRC | 23:19 | |
| *** rama_y has quit IRC | 23:20 | |
| *** Sukhdev has joined #openstack-ironic | 23:21 | |
| *** fragatina has joined #openstack-ironic | 23:23 | |
| * TheJulia ponders getting a stop watch to time to lag betwen clicking and the site being usable again | 23:30 | |
| JayF | TheJulia: IKR | 23:34 |
| JayF | TheJulia: I was wondering if it was just my firefox + extensions | 23:35 |
| *** jaybeale has quit IRC | 23:35 | |
| *** pleia2 has quit IRC | 23:43 | |
| *** pleia2 has joined #openstack-ironic | 23:43 | |
| *** HenryG has quit IRC | 23:45 | |
| *** jayc_ has quit IRC | 23:48 | |
| *** fragatina has quit IRC | 23:48 | |
| *** HenryG has joined #openstack-ironic | 23:48 | |
| *** fragatina has joined #openstack-ironic | 23:48 | |
| klindgren | One more question that came up again. How are people handling hardware that has different network interface names based upon the OS (IE I got 99 problems and systemd is one). And same question re: serial consoles. I have some hardware where Serial-over-lan will only work when configured to use com2 vs's com1 and vice versa. Which means that dpending on the node I either need to refrence ttyS0 or ttyS1. But the kernel wil | 23:50 |
| klindgren | l only allow you to specify one console of a type (ie only one serial and one normal console) | 23:50 |
| *** maratoid has quit IRC | 23:50 | |
| JayF | klindgren: liberal use of net.ifnames=0 and similar | 23:51 |
| JayF | klindgren: lol | 23:51 |
| JayF | klindgren: there's a big session re: consoles at the summit, I think that stuff we're trying to nail down | 23:51 |
| *** fragatin_ has joined #openstack-ironic | 23:51 | |
| JayF | klindgren: the images are a pain to get working well though, we have similar problems in our onmetal enviroment :/ | 23:51 |
| *** fragatina has quit IRC | 23:53 | |
| TheJulia | klindgren: w/r/t network interface names, glean has logic to read from a config drive and assert network configuration based upon that. For some, that seems to have worked really well | 23:53 |
| JayF | TheJulia: that solves ... some of it | 23:54 |
| TheJulia | indeed | 23:54 |
| TheJulia | not all of it | 23:54 |
| JayF | TheJulia: we've seen interesting interactions in some distros with timing | 23:54 |
| JayF | TheJulia: i.e. cloud-init running while interfaces are being renamed | 23:54 |
| TheJulia | oh awesome | 23:54 |
| JayF | TheJulia: so we tend to disable across the board | 23:54 |
| klindgren | JayF, kk. We were looking at moving over to using the newer network_data.json | 23:55 |
| JayF | Honestly, I think sometimes the biggest technical accomplishment of the omv2 launch was getting images working with raid :P | 23:55 |
| JayF | klindgren: yep, that's our patch, and glean uses that, and cloud-init will too soon | 23:55 |
| klindgren | working on trying to get that info back into cloud-init in all distros | 23:55 |
| JayF | klindgren: I'd suggest looking at glean depending on your needs, if you just need mainly network config, glean is a lot smaller | 23:55 |
| klindgren | we though about that - but we keep cloud-init for other stuff for end users | 23:56 |
| klindgren | so easier to jsut get cloud-init to support that | 23:56 |
| *** david-lyle_ is now known as david-lyle | 23:57 | |
| JayF | makes sense | 23:57 |
| TheJulia | JayF: now I'm seeing different things between the mobile app and the website :( | 23:57 |
| klindgren | we also do some provisioning tasks inside our corporate vms as they boot up. So we were thinking of fixing up serial consolesthat way. But it would be nice if we can specify that on a node level | 23:58 |
| JayF | TheJulia: it literally crashed my mac, lol | 23:58 |
| JayF | TheJulia: I'm going to just build a schedule and print it | | 23:58 |
| JayF | || put it on my phone as a pdf | 23:58 |
| JayF | probably both | 23:58 |
| klindgren | and the IPA agent do some of those minor fixup's on the server if possible | 23:58 |
| TheJulia | JayF: I will likely do the exact same | 23:58 |
| klindgren | worse case we will fix it on first boot of the vm | 23:58 |
| JayF | klindgren: sorta as a policy thing, Ironic doesn't mount filesystems | 23:58 |
| JayF | klindgren: so you'd want to fix it on VM boot, or by using different images for different hardware | 23:59 |
| klindgren | the issue is that it needs to be fixed in the bootloader otherwise some instances will have consoles | 23:59 |
| klindgren | and some wont | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!