rloo | TheJulia, vdrok: if you upgrade nova, it'll expect ironic with 1.<something, i forgot what>, which is in queens 9.2x release. but grenade is not upgrading API, so it won't handle that ironic API version. I think anyway. but worth trying I guess. | 00:00 |
---|---|---|
vdrok | yeah this is just a desperate attempt :) | 00:00 |
rloo | vdrok, TheJulia: OR we just don't test rolling upgrades right now? | 00:02 |
rloo | vdrok, TheJulia: I blame it on jroll, he said we needed to test or we couldn't put in the feature. we can test *after* the upgrade is done... or *after* it is done but everything pinned still? dunno. | 00:03 |
TheJulia | that and or we join the restart with it.... we should see the idfference in the log before we even restart ironic | 00:03 |
TheJulia | we'll figure it out | 00:04 |
rloo | vdrok, TheJulia: so sorry, i have to take off. thx for looking into it. talk to you on Monday. | 00:04 |
vdrok | o/ good night | 00:04 |
*** rloo has quit IRC | 00:04 | |
TheJulia | have a good weekend rloo | 00:08 |
TheJulia | :( | 00:09 |
TheJulia | I'm going to disappear for about two hours, dinner with my father... besides CI takes a long time :( | 00:09 |
openstackgerrit | yuan liang proposed openstack/ironic-python-agent master: Execute error in _detect_cna_card https://review.openstack.org/534690 | 00:24 |
*** liuzz has joined #openstack-ironic | 00:36 | |
*** penick_ has quit IRC | 00:37 | |
*** tuanla____ has joined #openstack-ironic | 00:43 | |
*** hieulq_ has joined #openstack-ironic | 01:03 | |
*** racedo_ has quit IRC | 01:04 | |
*** hieulq has quit IRC | 01:06 | |
*** crushil has joined #openstack-ironic | 01:06 | |
*** lbragstad has joined #openstack-ironic | 01:09 | |
*** tuanla____ has quit IRC | 01:24 | |
*** tuanla____ has joined #openstack-ironic | 01:25 | |
*** tiendc has joined #openstack-ironic | 01:27 | |
*** rbudden has quit IRC | 01:34 | |
*** mjturek has quit IRC | 01:35 | |
*** bfournie has joined #openstack-ironic | 01:36 | |
*** hshiina has joined #openstack-ironic | 01:55 | |
*** weshay|dentist is now known as weshay|ruck | 02:00 | |
*** crushil has quit IRC | 02:09 | |
*** gcb has joined #openstack-ironic | 02:28 | |
*** harlowja has quit IRC | 02:31 | |
TheJulia | Well, this is a good sign.. upgrading nova did the trick, minus one test failed | 02:34 |
TheJulia | different microversion. 1.35 vs 1.34 (our api endpoint) | 02:39 |
*** namnh has joined #openstack-ironic | 02:40 | |
*** threestrands_ has joined #openstack-ironic | 02:49 | |
*** threestrands_ has quit IRC | 02:49 | |
*** threestrands_ has joined #openstack-ironic | 02:49 | |
*** threestrands has quit IRC | 02:51 | |
*** andreaf has quit IRC | 03:07 | |
*** lbragstad has quit IRC | 03:08 | |
*** pmannidi has quit IRC | 03:23 | |
*** pmannidi has joined #openstack-ironic | 03:23 | |
TheJulia | dtantsur|afk: no great options at this point, https://etherpad.openstack.org/p/IronicWhiteBoard line 74 details options | 03:23 |
crushil_ | Thanks for helping out TheJulia | 03:31 |
TheJulia | I'm not really sure of a good option at this point w/r/t grenade beyond just disabling multinode for the time being | 03:32 |
TheJulia | which, in it's self is a really bad idea | 03:33 |
TheJulia | but its the best short term bad idea that I have | 03:33 |
*** andreaf has joined #openstack-ironic | 03:36 | |
* TheJulia wonders if that would be a good commit message subject | 03:38 | |
*** zshi has quit IRC | 03:38 | |
*** fragatina has quit IRC | 03:38 | |
*** fragatina has joined #openstack-ironic | 03:39 | |
*** fragatina has quit IRC | 03:41 | |
*** AlexeyAbashkin has joined #openstack-ironic | 03:44 | |
*** jappleii__ has joined #openstack-ironic | 03:44 | |
*** jappleii__ has quit IRC | 03:45 | |
*** jappleii__ has joined #openstack-ironic | 03:45 | |
*** jappleii__ has quit IRC | 03:46 | |
*** threestrands_ has quit IRC | 03:46 | |
*** lbragstad has joined #openstack-ironic | 03:46 | |
*** jappleii__ has joined #openstack-ironic | 03:47 | |
*** jappleii__ has quit IRC | 03:48 | |
*** AlexeyAbashkin has quit IRC | 03:48 | |
*** jappleii__ has joined #openstack-ironic | 03:48 | |
*** dpawar has joined #openstack-ironic | 03:49 | |
*** rcernin has quit IRC | 03:49 | |
*** dpawar has quit IRC | 03:49 | |
*** rcernin has joined #openstack-ironic | 03:49 | |
*** dpawar has joined #openstack-ironic | 03:56 | |
*** dpawar has quit IRC | 03:57 | |
*** dpawar has joined #openstack-ironic | 03:57 | |
*** fragatina has joined #openstack-ironic | 03:59 | |
*** Nisha_Agarwal has joined #openstack-ironic | 04:00 | |
*** lbragstad has quit IRC | 04:02 | |
*** hamzy_ has joined #openstack-ironic | 04:03 | |
*** fragatina has quit IRC | 04:04 | |
*** ameeda has quit IRC | 04:04 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Mark multinode job as non-voting https://review.openstack.org/535594 | 04:04 |
openstackgerrit | Rushil Chugh proposed openstack/ironic master: Add XClarity Driver https://review.openstack.org/519427 | 04:05 |
openstackgerrit | Merged openstack/bifrost master: Updated from global requirements https://review.openstack.org/534905 | 04:11 |
*** dpawar has quit IRC | 04:11 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Always attempt to restart n-cpu/n-cond https://review.openstack.org/535391 | 04:12 |
*** pmannidi_ has joined #openstack-ironic | 04:17 | |
*** pmannidi has quit IRC | 04:19 | |
*** pmannidi_ is now known as pmannidi | 04:19 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Rework upgrade to upgrade nova/ironic https://review.openstack.org/535596 | 04:29 |
*** links has joined #openstack-ironic | 04:37 | |
*** namnh has quit IRC | 04:38 | |
*** kiseok7_ has joined #openstack-ironic | 05:06 | |
*** dpawar has joined #openstack-ironic | 05:10 | |
*** harlowja has joined #openstack-ironic | 05:21 | |
openstackgerrit | Julia Kreger proposed openstack/python-ironicclient master: Facilitate latest Rest API use https://review.openstack.org/531029 | 05:36 |
openstackgerrit | Julia Kreger proposed openstack/python-ironicclient master: Allow API user to define list of versions https://review.openstack.org/531271 | 05:36 |
* TheJulia goes to sleep | 05:37 | |
*** crushil has joined #openstack-ironic | 05:43 | |
*** jappleii__ has quit IRC | 05:52 | |
*** marios has joined #openstack-ironic | 06:14 | |
*** logan- has quit IRC | 06:16 | |
*** crushil has quit IRC | 06:22 | |
*** harlowja has quit IRC | 06:29 | |
*** logan- has joined #openstack-ironic | 06:40 | |
*** pcaruana has joined #openstack-ironic | 06:44 | |
*** mcarden has quit IRC | 06:49 | |
*** charcol has quit IRC | 06:59 | |
*** namnh has joined #openstack-ironic | 07:03 | |
*** deray has joined #openstack-ironic | 07:06 | |
*** mjura has joined #openstack-ironic | 07:07 | |
deray | Hello Ironic'ers! good morning | 07:09 |
*** fragatina has joined #openstack-ironic | 07:28 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-ui master: Imported Translations from Zanata https://review.openstack.org/532436 | 07:29 |
*** fragatina has quit IRC | 07:33 | |
openstackgerrit | Ankit Kumar proposed openstack/ironic master: Adding condition for setting variable https://review.openstack.org/535651 | 07:37 |
openstackgerrit | Ankit Kumar proposed openstack/ironic-tempest-plugin master: Test cases for iLO drivers in standalone ironic https://review.openstack.org/535652 | 07:40 |
openstackgerrit | Ankit Kumar proposed openstack/ironic-tempest-plugin master: Test cases for iLO drivers in standalone ironic https://review.openstack.org/535652 | 07:43 |
*** rcernin has quit IRC | 07:43 | |
*** etingof has quit IRC | 07:55 | |
*** zshi has joined #openstack-ironic | 08:04 | |
*** AlexeyAbashkin has joined #openstack-ironic | 08:13 | |
*** Nisha_Agarwal has quit IRC | 08:15 | |
*** tesseract has joined #openstack-ironic | 08:16 | |
*** harlowja has joined #openstack-ironic | 08:17 | |
*** tuanla_____ has joined #openstack-ironic | 08:25 | |
*** namnh_ has joined #openstack-ironic | 08:26 | |
*** jmccrory has quit IRC | 08:26 | |
*** sdake has quit IRC | 08:27 | |
*** persia has quit IRC | 08:28 | |
*** mkrai has quit IRC | 08:28 | |
*** tuanla____ has quit IRC | 08:28 | |
*** tuanla_____ has quit IRC | 08:28 | |
*** namnh has quit IRC | 08:28 | |
*** tiendc has quit IRC | 08:28 | |
*** persia has joined #openstack-ironic | 08:28 | |
*** tuanla_____ has joined #openstack-ironic | 08:29 | |
*** tiendc has joined #openstack-ironic | 08:29 | |
*** deray has quit IRC | 08:30 | |
*** sdake has joined #openstack-ironic | 08:33 | |
*** sdake has quit IRC | 08:33 | |
*** sdake has joined #openstack-ironic | 08:33 | |
*** jmccrory has joined #openstack-ironic | 08:33 | |
*** deray has joined #openstack-ironic | 08:35 | |
*** harlowja has quit IRC | 08:37 | |
*** namnh has joined #openstack-ironic | 08:40 | |
*** mkrai has joined #openstack-ironic | 08:43 | |
*** namnh_ has quit IRC | 08:43 | |
*** Nisha_Agarwal has joined #openstack-ironic | 08:43 | |
*** jmccrory has quit IRC | 08:50 | |
*** jmccrory has joined #openstack-ironic | 08:52 | |
*** namnh has quit IRC | 08:54 | |
*** pmannidi has quit IRC | 08:59 | |
*** zshi has quit IRC | 09:00 | |
*** stendulker has joined #openstack-ironic | 09:01 | |
*** homeski has quit IRC | 09:02 | |
*** makowals has quit IRC | 09:06 | |
*** e0ne has joined #openstack-ironic | 09:10 | |
*** e0ne has quit IRC | 09:10 | |
*** hshiina has quit IRC | 09:17 | |
*** racedo has joined #openstack-ironic | 09:23 | |
*** Nisha_Agarwal has quit IRC | 09:24 | |
*** athomas has joined #openstack-ironic | 09:28 | |
*** fragatina has joined #openstack-ironic | 09:30 | |
*** fragatina has quit IRC | 09:35 | |
*** tuanla____ has joined #openstack-ironic | 09:40 | |
*** derekh has joined #openstack-ironic | 09:41 | |
*** makowals has joined #openstack-ironic | 09:43 | |
*** tuanla_____ has quit IRC | 09:44 | |
*** karthiks has quit IRC | 09:47 | |
*** tuanla____ has quit IRC | 09:48 | |
*** pcaruana has quit IRC | 09:50 | |
*** tuanla____ has joined #openstack-ironic | 09:50 | |
*** pcaruana has joined #openstack-ironic | 09:53 | |
*** pcaruana has quit IRC | 09:53 | |
*** pcaruana has joined #openstack-ironic | 09:54 | |
*** tuanla____ has quit IRC | 09:56 | |
*** karthiks has joined #openstack-ironic | 09:59 | |
*** lucas-pto is now known as lucasagomes | 10:03 | |
*** hieulq_ has quit IRC | 10:05 | |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Add rescuewait timeout periodic task https://review.openstack.org/353156 | 10:14 |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Agent rescue implementation https://review.openstack.org/400437 | 10:14 |
*** etingof has joined #openstack-ironic | 10:21 | |
*** tiendc has quit IRC | 10:30 | |
*** e0ne has joined #openstack-ironic | 10:32 | |
*** stendulker_ has joined #openstack-ironic | 10:33 | |
*** marlinc has quit IRC | 10:36 | |
*** stendulker has quit IRC | 10:37 | |
*** e0ne has quit IRC | 10:37 | |
*** liuzz has quit IRC | 10:39 | |
*** stendulker_ has quit IRC | 10:39 | |
*** marlinc has joined #openstack-ironic | 10:47 | |
*** makowals has quit IRC | 10:47 | |
*** makowals has joined #openstack-ironic | 10:48 | |
*** makowals has quit IRC | 10:48 | |
*** makowals has joined #openstack-ironic | 10:49 | |
*** makowals has quit IRC | 10:49 | |
*** makowals has joined #openstack-ironic | 10:49 | |
*** dtantsur|afk is now known as dtantsur | 10:54 | |
dtantsur | morning ironic | 10:55 |
*** e0ne has joined #openstack-ironic | 10:55 | |
*** deray has quit IRC | 10:59 | |
*** amarao has joined #openstack-ironic | 11:04 | |
dtantsur | TheJulia: thanks for looking into it. I'm approving making the job non-voting, but we have to recover and make it pass before the release.. | 11:09 |
*** marios has quit IRC | 11:16 | |
*** makowals has quit IRC | 11:21 | |
*** fragatina has joined #openstack-ironic | 11:32 | |
*** fragatina has quit IRC | 11:37 | |
*** dpawar has quit IRC | 11:45 | |
mgoddard | does anyone know what version the queens release will be? | 12:04 |
mgoddard | 10.1.0? | 12:04 |
*** e0ne has quit IRC | 12:12 | |
dtantsur | mgoddard: likely, unless we land a breaking change | 12:14 |
*** marios has joined #openstack-ironic | 12:19 | |
*** gmonteiro|home has joined #openstack-ironic | 12:23 | |
*** hogepodge has quit IRC | 12:26 | |
*** strigazi has quit IRC | 12:27 | |
*** dansmith has quit IRC | 12:27 | |
*** jtomasek has quit IRC | 12:27 | |
*** exodusftw has quit IRC | 12:27 | |
*** jtomasek_ has joined #openstack-ironic | 12:27 | |
*** gcb has quit IRC | 12:28 | |
*** dansmith has joined #openstack-ironic | 12:28 | |
*** dansmith is now known as Guest86649 | 12:28 | |
*** hogepodge has joined #openstack-ironic | 12:28 | |
*** strigazi has joined #openstack-ironic | 12:29 | |
*** exodusftw has joined #openstack-ironic | 12:29 | |
*** makowals has joined #openstack-ironic | 12:33 | |
*** jtomasek_ is now known as jtomasek | 12:35 | |
*** pc_m has quit IRC | 12:48 | |
*** pc_m has joined #openstack-ironic | 12:50 | |
*** yangyapeng has quit IRC | 12:55 | |
*** yangyapeng has joined #openstack-ironic | 12:55 | |
*** yangyapeng has quit IRC | 12:59 | |
openstackgerrit | Merged openstack/ironic master: Mark multinode job as non-voting https://review.openstack.org/535594 | 13:01 |
*** jaypipes is now known as leakypipes | 13:10 | |
openstackgerrit | Nikolay Fedotov proposed openstack/ironic-python-agent master: Try to find partition 3 times https://review.openstack.org/525577 | 13:11 |
*** dprince has joined #openstack-ironic | 13:11 | |
openstackgerrit | Nikolay Fedotov proposed openstack/ironic-python-agent master: Try to find partition 3 times https://review.openstack.org/525577 | 13:12 |
openstackgerrit | Hugo Nicodemos proposed openstack/ironic master: Migrate oneview hardware type to use python-hpOneView https://review.openstack.org/524310 | 13:12 |
*** jlabarre has quit IRC | 13:12 | |
*** mcarden has joined #openstack-ironic | 13:13 | |
nicodemos | hey, about https://review.openstack.org/#/c/524310/ do you want me to create a reno just to talking about the redfish import in this patch? or I put this warning in the remove python-oneviewclient release note? | 13:18 |
patchbot | patch 524310 - ironic - Migrate oneview hardware type to use python-hpOneView | 13:18 |
nicodemos | dtantsur ^ | 13:18 |
dtantsur | nicodemos: any way is fine with me | 13:20 |
nicodemos | ok, thanks. :) | 13:20 |
*** MattMan has quit IRC | 13:22 | |
*** MattMan has joined #openstack-ironic | 13:22 | |
*** crushil has joined #openstack-ironic | 13:24 | |
*** makowals has quit IRC | 13:24 | |
*** links has quit IRC | 13:28 | |
*** yangyapeng has joined #openstack-ironic | 13:29 | |
TheJulia | dtantsur: acknowledged. Sadly few good options :( What is your perspective on actually completing upgrades for nova/ironic? | 13:31 |
TheJulia | but leaving like the rpc pin in place | 13:32 |
*** yangyapeng has quit IRC | 13:32 | |
*** yangyapeng has joined #openstack-ironic | 13:32 | |
dtantsur | TheJulia: that's not quite the thing we're supposed to test.. | 13:32 |
dtantsur | I'm not sure I've fully grasped the problem, mind providing another tl;dr please? | 13:33 |
jroll | morning ironic | 13:33 |
dtantsur | morning jroll | 13:34 |
TheJulia | everything points to something gets upgraded that caues n-cond proces to spin out of control crashing on a segmentation violation, which causes other fun things like n-compute to get killed off by systemd | 13:34 |
* jroll attempts to grok grenade problems | 13:34 | |
*** fragatina has joined #openstack-ironic | 13:34 | |
gmonteiro|home | Morning jroll and all o/ | 13:34 |
dtantsur | TheJulia: isn't it something that nova folks should become concerned about as well? | 13:34 |
dtantsur | like, I don't think we expect n-cond to segfault.. | 13:34 |
TheJulia | since there is no isolation in venvs, and I believe python tries to dynamically recompile (at least I think), or that is c code that gets upgraded in some package | 13:35 |
TheJulia | no, we don't, but since we leave it running and never upgrade it.... | 13:35 |
jroll | it's even scarier that it does it without restarting n-cond | 13:36 |
TheJulia | yup.... | 13:36 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Automatically migrate nodes to hardware types https://review.openstack.org/534373 | 13:36 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow data migrations to accept options https://review.openstack.org/535772 | 13:36 |
dtantsur | TheJulia: this basically means that rolling upgrades don't work.. | 13:36 |
TheJulia | well | 13:36 |
TheJulia | no | 13:36 |
jroll | rolling upgrades don't work *if nova is on the same machine as ironic without virtualenvs or similar* :) | 13:37 |
TheJulia | The problem is that you have to have some sort of isolation, presently we don't | 13:37 |
TheJulia | we're still upgrading underlying packages that nova uses, nothing prevents that | 13:37 |
TheJulia | what jroll said | 13:37 |
jroll | feels like this is a thing to bring up with nova, or maybe better yet the requirements/release team | 13:37 |
*** edleafe is now known as figleaf | 13:37 | |
jroll | feels like if we pin down which library it is, we could do something like bump the minimum in pike, or lower the minimum in queens, or | 13:38 |
TheJulia | I was pondering that last night, for the fun of it, I went and checked bugs for oslo, nothing matched | 13:38 |
jroll | and they might have some insight on a) how to track it down, and b) the best route to take | 13:38 |
dtantsur | "if nova is on the same machine as ironic without virtualenvs or similar" is a very common situation fwiw | 13:38 |
TheJulia | I was thinking if we could get the upgrade script ot fire lsof a few times, we might be able to get a list of what it is loading.... | 13:39 |
jroll | dtantsur: I didn't mean to imply it isn't | 13:39 |
*** fragatina has quit IRC | 13:39 | |
jroll | TheJulia: and/or run an ldd on nova-conductor before and after the ironic upgrade? | 13:40 |
TheJulia | well,I did also try just restrating n-cond to see if it came up clean, I've not had coffee nor looked at that yet | 13:40 |
dtantsur | I don't think you can run ldd on python scripts | 13:40 |
TheJulia | jroll: that is only static linkings | 13:40 |
jroll | bleh, you might be right | 13:40 |
jroll | yeah | 13:40 |
TheJulia | we need to actually pull what is open in ram | 13:40 |
* jroll also needs more coffee | 13:40 | |
TheJulia | lsof -n |grep nova | 13:41 |
*** trown|outtypewww is now known as trown | 13:42 | |
*** trown is now known as trown|ruck | 13:42 | |
dtantsur | but if nova pike crashes with packages from queens, it means that their rolling upgrade does not work.. | 13:43 |
jroll | hm, yeah fair point | 13:44 |
sambetts|afk | depends if it only crashes if the package changes out during run time | 13:44 |
*** sambetts|afk is now known as sambetts | 13:44 | |
jroll | they leave conductor running in rolling upgrades | 13:44 |
jroll | only upgrade compute afaik | 13:44 |
jroll | er, I may have that backwards though | 13:45 |
jroll | I do: "For maximum safety (no failed API operations), gracefully shutdown all the services (i.e. SIG_TERM) except nova-compute." https://docs.openstack.org/nova/pike/user/upgrade.html | 13:45 |
jroll | so everything but -compute goes first | 13:45 |
TheJulia | http://logs.openstack.org/91/535391/3/check/ironic-grenade-dsvm-multinode-multitenant/41889b3/logs/screen-n-cond.txt.gz#_Jan_19_05_32_17_704998 Looks like library incompatability so just restarting n-cond doesn't work | 13:46 |
dtantsur | oh, so maybe we should do the same in the CI | 13:46 |
dtantsur | like, upgrading everything except for n-cpu | 13:46 |
sambetts | shouldn't their grenade plugin be handling all of that already? | 13:47 |
jroll | dtantsur: eh, we want the ironic upgrade to be separate from the nova upgrade, ideally | 13:47 |
TheJulia | it will upgrade nova-compute, but doesn't mean we _have_ to restart it | 13:47 |
TheJulia | but we will need to move the restart such that an ironic api endpoint is reachable | 13:47 |
jroll | dtantsur: might be a valid workaround though | 13:47 |
sambetts | so the upgrade needs to do: all nova services except compute, then ironic, then nova compute | 13:48 |
dtantsur | cam we 1. upgrade ironic, 2. restart n-cpu, 3. upgrade nova? | 13:49 |
jroll | the "upgrade ironic" is what makes everything go nuts | 13:49 |
TheJulia | dtantsur: afaik no | 13:49 |
TheJulia | dtantsur: goes back to the ordering control issues | 13:49 |
dtantsur | UGH | 13:49 |
TheJulia | ugh | 13:49 |
TheJulia | we can't prevent n-cpu from being upgraded... | 13:49 |
TheJulia | because it _will_ stop/start it | 13:50 |
TheJulia | as part of its upgrade | 13:50 |
sambetts | personally I don't think testing our rolling upgrade should test nova's rolling upgrades, so I'd be +1 to disabling upgrading any of nova | 13:50 |
dtantsur | another option: QUICKLY patch nova to stop hardcoding the latest microversion | 13:50 |
dtantsur | which may be a work around until we land TheJulia's patches | 13:50 |
TheJulia | sambetts: well, that is presently how we operate by default already | 13:50 |
TheJulia | dtantsur: we would actually have to lower the microversion | 13:51 |
TheJulia | dtantsur: OR.... get python-ironicclient paches landed | 13:51 |
* TheJulia has estimated timeline on the whiteboard for that | 13:51 | |
sambetts | can't do: upgrad all nova services except compute, then upgrade ironic, then upgrade nova compute? | 13:52 |
TheJulia | considering it is friday, there is no way to pull off a release of it, likely the earliest we could get such a patch landed is wednesday | 13:52 |
sambetts | can't we do* | 13:52 |
TheJulia | sambetts: all nova packages are upgraded in one shot, its sequence is stop/start api, compute, and conductor | 13:52 |
TheJulia | we would have to write custom nova upgrade stuffs | 13:52 |
dtantsur | TheJulia: I meant, poor man's negotiation like http://paste.openstack.org/show/647041/ | 13:53 |
*** Goneri has joined #openstack-ironic | 13:54 | |
TheJulia | dtantsur: that would do it.... | 13:54 |
*** e0ne has joined #openstack-ironic | 13:55 | |
jroll | and doing the negotiation would allow us to upgrade nova first, then ironic? | 13:55 |
jroll | thus fixing the CI issue? | 13:56 |
sambetts | well, s/fixing/avoiding | 13:56 |
*** milan_ has joined #openstack-ironic | 13:56 | |
jroll | sure | 13:56 |
jroll | just wondering if there's any pieces I'm missing | 13:56 |
jroll | (note: we'd have to caveat our upgrade docs with "if pike->queens, and nova and ironic share packages, upgrade order must be different") | 13:57 |
dtantsur | anyone wants to try such nova patch? | 13:57 |
*** rbudden has joined #openstack-ironic | 13:57 | |
dtantsur | I can, but I'm also trying to figure out upgrade to hardware types.. | 13:57 |
TheJulia | The pin still broke my upgrade ironic and nova attempt :\ http://logs.openstack.org/96/535596/1/check/ironic-grenade-dsvm-multinode-multitenant/41aa951/logs/screen-n-cpu.txt.gz#_Jan_19_06_13_51_822255 | 13:57 |
jroll | dtantsur: I can in a bit | 13:58 |
jroll | doing some irl things at the moment | 13:58 |
dtantsur | cool | 13:58 |
TheJulia | jroll: lmk when you post it, we can fire a few test jobs off with it fairly quickly | 13:59 |
jroll | TheJulia: okay, it might be 30+ minutes | 13:59 |
TheJulia | jroll: works for me, I'm still trying to wake up | 13:59 |
jroll | yes, go coffee | 13:59 |
*** sasha21 has joined #openstack-ironic | 14:00 | |
jroll | I do think it's worth letting nova know about the runtime segfaults, possibly with the bonus of nerd-sniping someone into helping | 14:03 |
*** SerenaFeng has joined #openstack-ironic | 14:07 | |
*** weshay|ruck is now known as weshay|rover | 14:08 | |
*** lbragstad has joined #openstack-ironic | 14:10 | |
TheJulia | raised in chat, lets see what people say | 14:12 |
TheJulia | another alternative, we could identify the parent nova-conductor process, and strace it for a second or two, then try and collect that process data | 14:16 |
TheJulia | that would give us a really clear picture of where in the world it is blowing up at | 14:17 |
TheJulia | Hmm... nerd sniping might not work | 14:19 |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Add API methods for [un]rescue https://review.openstack.org/350831 | 14:20 |
*** crushil has quit IRC | 14:23 | |
openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 14:27 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: DNM: Attempt to collect more info re grenade failures https://review.openstack.org/535492 | 14:29 |
*** jistr is now known as jistr|mtg | 14:30 | |
TheJulia | hopefully that will give us a little more insight into what is occuring, and maybe give us a clue of what libraries are "good" | 14:32 |
openstackgerrit | Hugo Nicodemos proposed openstack/ironic master: Migrate python-oneviewclient validations to oneview hardware type https://review.openstack.org/524599 | 14:35 |
openstackgerrit | Hugo Nicodemos proposed openstack/ironic master: Remove python-oneviewclient from oneview hardware type. https://review.openstack.org/524729 | 14:35 |
openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 14:36 |
*** lucasagomes is now known as lucas-hungry | 14:38 | |
*** jlabarre has joined #openstack-ironic | 14:42 | |
jroll | dtantsur: TheJulia: I don't need any depends-on for this nova negotiation patch, right? that'll be the base for our test patches? | 14:48 |
jroll | I don't think we do, here it is https://review.openstack.org/535786 | 14:50 |
patchbot | patch 535786 - nova - WIP: hacky negotiation of ironic microversion | 14:50 |
TheJulia | jroll: not yet :) | 14:53 |
jroll | cool | 14:53 |
dtantsur | right | 14:53 |
jroll | pretty sure that should work, I haven't executed it yet | 14:53 |
* jroll tries | 14:54 | |
jroll | appears to work fine with everything on queens | 14:57 |
jroll | or maybe not | 14:58 |
jroll | k, fixed now | 15:01 |
jroll | works with pike too | 15:01 |
jroll | er, pike ironic | 15:02 |
*** crushil_ is now known as crushil | 15:02 | |
TheJulia | jroll: a few comments on rev 1 if you haven't seen them, but later :) | 15:02 |
jroll | TheJulia: thanks, I wasn't sure if we just wanted to test this or make it an actual patch | 15:03 |
TheJulia | I'm going to fire off a job in a moment | 15:03 |
TheJulia | just trying to choose the right one :) | 15:03 |
jroll | in the latter case I could clean it up later today | 15:03 |
jroll | cool :) | 15:03 |
*** jistr|mtg is now known as jistr | 15:04 | |
*** SerenaFeng has quit IRC | 15:04 | |
dtantsur | jroll: I'm afraid we'll have to make it an actual patch, then replace with TheJulia's negotiation work | 15:06 |
*** pc_m has quit IRC | 15:08 | |
*** pc_m has joined #openstack-ironic | 15:09 | |
jroll | dtantsur: yeah, I'll clean it up and try to think of a commit message that can reasonably explain the situation :P | 15:10 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Rework upgrade to upgrade nova/ironic https://review.openstack.org/535596 | 15:10 |
TheJulia | not perfect, still upgrades ironic, we can peel that back once we have results | 15:10 |
TheJulia | marked that wf-1 and put a note in | 15:11 |
jroll | cool | 15:12 |
TheJulia | I would say that it feels like a day where I should have something tasty.... except I have to drive the RV ~250 miles later today | 15:14 |
jroll | sounds like a good reason to stop and get a dozen donuts and a gallon of coffee :P | 15:15 |
jroll | and tasty real food upon arrival | 15:16 |
TheJulia | There is not much where we're headed tonight... :( | 15:16 |
jroll | oh :/ | 15:16 |
TheJulia | We'll likely cook a pizza or maybe some chicken once we get there | 15:16 |
*** crushil_ has joined #openstack-ironic | 15:19 | |
jroll | could be worse :) | 15:19 |
TheJulia | I'm not sure zuul picked up the patch :\ | 15:22 |
jroll | it's there | 15:22 |
jroll | unless you mean the depends-on | 15:23 |
TheJulia | yeah, its in the zuul_changes string it looks like | 15:23 |
TheJulia | now we wait :( | 15:23 |
*** livelace-link has joined #openstack-ironic | 15:23 | |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Add rescue interface field to node-related notifications https://review.openstack.org/502889 | 15:26 |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Update description for config params of 'rescue' interface https://review.openstack.org/517919 | 15:26 |
openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Add documentation for rescue interface https://review.openstack.org/419606 | 15:26 |
*** mjturek has joined #openstack-ironic | 15:28 | |
TheJulia | dtantsur: fyi, if you haven't noticed, I uploaded new revs for two of the three client changes... the two changes we would need at a minimum, the third is entirely optional | 15:34 |
*** lucas-hungry is now known as lucasagomes | 15:34 | |
*** fragatina has joined #openstack-ironic | 15:36 | |
TheJulia | When I finally have a new home, I'll just have a monitor for watching CI jobs. | 15:40 |
*** fragatina has quit IRC | 15:40 | |
*** gmonteiro|home has quit IRC | 15:49 | |
*** amarao has quit IRC | 15:51 | |
dtantsur | TheJulia: thanks, will check | 15:53 |
*** jlvillal has quit IRC | 16:04 | |
*** crushil_ has quit IRC | 16:06 | |
*** mjura has quit IRC | 16:10 | |
*** jlvillal has joined #openstack-ironic | 16:14 | |
*** crushil_ has joined #openstack-ironic | 16:19 | |
*** jcoufal has joined #openstack-ironic | 16:24 | |
*** penick has joined #openstack-ironic | 16:30 | |
openstackgerrit | Sam Betts proposed openstack/ironic master: Only set default network interface flat if enabled in config https://review.openstack.org/535830 | 16:32 |
sambetts | dtantsur: just discovered that ^ while working on my CI | 16:33 |
*** AlexeyAbashkin has quit IRC | 16:35 | |
*** fragatina has joined #openstack-ironic | 16:37 | |
dtantsur | k | 16:39 |
*** absubram has joined #openstack-ironic | 16:40 | |
*** fragatina has quit IRC | 16:41 | |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add a version argument to traits DB API https://review.openstack.org/535482 | 16:42 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC object for traits https://review.openstack.org/532268 | 16:43 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC API and conductor manager for traits https://review.openstack.org/535296 | 16:43 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: API: Node Traits API https://review.openstack.org/532269 | 16:43 |
mgoddard | rloo, jroll: ^^^ | 16:43 |
*** cdearborn has joined #openstack-ironic | 16:48 | |
TheJulia | and upgrade running *crosses fingers* | 16:52 |
TheJulia | o/ cdearborn | 16:52 |
cdearborn | \o TheJulia! | 16:54 |
NobodyCam | Good Morning Ironic'ers | 16:58 |
NobodyCam | and ofc.... | 16:58 |
NobodyCam | TGIF | 16:58 |
*** trown|ruck is now known as trown|outtypewww | 16:59 | |
TheJulia | ugh, not looking good for that patch... | 17:01 |
* TheJulia waits for logs to get posted | 17:03 | |
*** absubram has quit IRC | 17:04 | |
*** etingof has quit IRC | 17:04 | |
TheJulia | looks like most of the compute tests... build failures for instances.... | 17:07 |
*** bnemec is now known as beekneemech | 17:13 | |
TheJulia | jroll: Ever seen anything like this http://logs.openstack.org/96/535596/2/check/ironic-grenade-dsvm-multinode-multitenant/079e25a/logs/screen-n-cpu.txt.gz#_Jan_19_16_55_26_003011 ? | 17:14 |
dtantsur | morning NobodyCam | 17:15 |
*** Goneri has quit IRC | 17:15 | |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Automatically migrate nodes to hardware types https://review.openstack.org/534373 | 17:16 |
dtantsur | this ^^^ was not easy at all... | 17:16 |
NobodyCam | Good Morning dtantsur :) happy Friday | 17:17 |
TheJulia | http://logs.openstack.org/96/535596/2/check/ironic-grenade-dsvm-multinode-multitenant/079e25a/logs/screen-ir-api.txt.gz#_Jan_19_17_00_40_976700 | 17:20 |
jroll | TheJulia: interesting, that placement error looks familiar | 17:22 |
TheJulia | realistically that run was with newer ironic, and it spewed some errors about a pin being present for rpc | 17:24 |
TheJulia | so, I'm going to restore the old logic to switch to an older api and not restart locally | 17:25 |
TheJulia | and go from there | 17:25 |
jroll | yeah, +1 on using the old api | 17:25 |
*** penick has quit IRC | 17:25 | |
jroll | though I guess I'm concerned the api doesn't work with a pin in it. I'm not sure if that's a good reason to be concerned. | 17:25 |
*** e0ne has quit IRC | 17:25 | |
TheJulia | It seemed to work, but not entirely sure | 17:27 |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Rework upgrade to upgrade nova/ironic https://review.openstack.org/535596 | 17:28 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Remove the resourceclasses CI job https://review.openstack.org/535862 | 17:36 |
*** pcaruana has quit IRC | 17:36 | |
*** fragatina has joined #openstack-ironic | 17:36 | |
*** fragatina has quit IRC | 17:37 | |
*** fragatina has joined #openstack-ironic | 17:37 | |
*** rpioso|afk is now known as rpioso | 17:48 | |
rpioso | dtantsur: Is https://review.openstack.org/#/c/526773/ still being worked? | 17:48 |
patchbot | patch 526773 - ironic - boot mode support at ManagementInterface | 17:48 |
*** cjloader has joined #openstack-ironic | 17:54 | |
jroll | TheJulia: your comment here was for later, not this patch, right? https://review.openstack.org/#/c/535786/1/nova/virt/ironic/client_wrapper.py@130 | 17:54 |
patchbot | patch 535786 - nova - WIP: hacky negotiation of ironic microversion | 17:54 |
*** rpioso is now known as rpioso|eat | 17:55 | |
*** penick has joined #openstack-ironic | 17:56 | |
*** milan_ has quit IRC | 17:56 | |
*** baha has joined #openstack-ironic | 17:58 | |
TheJulia | jroll: the comment on how we would down the road | 17:59 |
*** crushil_ has quit IRC | 17:59 | |
*** r-daneel has joined #openstack-ironic | 17:59 | |
jroll | TheJulia: I just linked to the RFE instead :) | 17:59 |
jroll | thanks | 18:00 |
*** SerenaFeng has joined #openstack-ironic | 18:00 | |
* jroll figures out how to test this | 18:00 | |
*** derekh has quit IRC | 18:01 | |
openstackgerrit | Merged openstack/ironic master: Adds RPC calls for rescue interface https://review.openstack.org/509336 | 18:02 |
*** Goneri has joined #openstack-ironic | 18:06 | |
*** vmud213 has joined #openstack-ironic | 18:07 | |
*** marios has quit IRC | 18:13 | |
*** AlexeyAbashkin has joined #openstack-ironic | 18:16 | |
*** AlexeyAbashkin has quit IRC | 18:20 | |
openstackgerrit | vinay kumar muddu proposed openstack/ironic master: Update proliantutils version for Queens release https://review.openstack.org/535895 | 18:22 |
openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: [WIP] Clean up CI playbooks https://review.openstack.org/535896 | 18:23 |
dtantsur | rpioso|eat: better ask etingof when he's online, but I think yes. he may appreciate your help though | 18:23 |
*** dtantsur is now known as dtantsur|afk | 18:24 | |
dtantsur|afk | happy Friday, y'all! | 18:24 |
rpioso|eat | dtantsur: Cool. So, I’ll proceed to review it and circle back with etingof next time I see him online. Thank you. | 18:27 |
*** SerenaFeng has quit IRC | 18:28 | |
dtantsur|afk | rpioso|eat: np. he's in central EU time | 18:28 |
TheJulia | I'm going to go drive for a little while, I'll check back in a couple hours :( | 18:29 |
TheJulia | looks like the gate test just got started, so ~2 hours :( | 18:29 |
vmud213 | dtantsur|afk or TheJulia, Mind taking a look at https://review.openstack.org/535895 | 18:29 |
patchbot | patch 535895 - ironic - Update proliantutils version for Queens release | 18:29 |
vmud213 | small patch on updating the driver requirements | 18:30 |
rpioso|eat | dtantsur|afk: Thanks, again. Enjoy your weekend :) | 18:30 |
*** weshay|rover is now known as weshay|bbiab | 18:31 | |
jroll | TheJulia: dtantsur|afk: updated that nova patch, should be landable ish | 18:38 |
jroll | which means I blew away julia's test :( | 18:38 |
* jroll restarts it | 18:38 | |
*** vmud213 has quit IRC | 18:43 | |
*** jaganathan has quit IRC | 18:45 | |
* sambetts --> home | 18:48 | |
sambetts | night all | 18:48 |
jroll | \o | 18:48 |
sambetts | have a great weekend | 18:48 |
sambetts | o/ | 18:48 |
jroll | you too :) | 18:48 |
*** sambetts is now known as sambetts|afk | 18:48 | |
openstackgerrit | vinay kumar muddu proposed openstack/ironic master: Update proliantutils version for Queens release https://review.openstack.org/535895 | 18:54 |
*** lucasagomes is now known as lucas-afk | 18:58 | |
*** racedo has quit IRC | 18:59 | |
*** vmud213 has joined #openstack-ironic | 18:59 | |
*** penick has quit IRC | 19:01 | |
*** penick has joined #openstack-ironic | 19:03 | |
*** e0ne has joined #openstack-ironic | 19:12 | |
vmud213 | vdrok: THere's a patch on driver-requirements here https://review.openstack.org/535895 | 19:13 |
patchbot | patch 535895 - ironic - Update proliantutils version for Queens release | 19:13 |
vmud213 | can you please look into that? | 19:13 |
*** e0ne has quit IRC | 19:17 | |
openstackgerrit | Mark Goddard proposed openstack/ironic-tempest-plugin master: Add node trait tests https://review.openstack.org/535908 | 19:23 |
openstackgerrit | Merged openstack/ironic master: Updated from global requirements https://review.openstack.org/533875 | 19:27 |
*** makowals has joined #openstack-ironic | 19:30 | |
*** harlowja has joined #openstack-ironic | 19:31 | |
openstackgerrit | Yusef Shaban proposed openstack/sushy master: Closes-Bug: 1744378 https://review.openstack.org/535910 | 19:34 |
openstack | bug 1744378 in sushy "Connector ignoring verify" [Undecided,New] https://launchpad.net/bugs/1744378 | 19:34 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add a version argument to traits DB API https://review.openstack.org/535482 | 19:35 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC object for traits https://review.openstack.org/532268 | 19:35 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC API and conductor manager for traits https://review.openstack.org/535296 | 19:35 |
openstackgerrit | Mark Goddard proposed openstack/ironic master: API: Node Traits API https://review.openstack.org/532269 | 19:35 |
*** vmud213 has quit IRC | 19:35 | |
*** AlexeyAbashkin has joined #openstack-ironic | 19:45 | |
*** etingof has joined #openstack-ironic | 19:47 | |
*** e0ne has joined #openstack-ironic | 19:47 | |
*** AlexeyAbashkin has quit IRC | 19:49 | |
openstackgerrit | John L. Villalovos proposed openstack/virtualbmc master: Change numeric constants to VARIABLE_NAME https://review.openstack.org/535916 | 19:59 |
*** manjeets has joined #openstack-ironic | 20:10 | |
*** dprince has quit IRC | 20:20 | |
*** dprince has joined #openstack-ironic | 20:20 | |
*** e0ne has quit IRC | 20:21 | |
openstackgerrit | Merged openstack/virtualbmc master: Support power reset command https://review.openstack.org/534646 | 20:25 |
*** AlexeyAbashkin has joined #openstack-ironic | 20:27 | |
*** hw_wutianwei has quit IRC | 20:31 | |
*** AlexeyAbashkin has quit IRC | 20:31 | |
*** hw_wutianwei has joined #openstack-ironic | 20:32 | |
*** fragatina has quit IRC | 20:34 | |
*** e0ne has joined #openstack-ironic | 20:35 | |
*** e0ne has quit IRC | 20:37 | |
*** weshay|bbiab is now known as weshay | 20:38 | |
*** weshay is now known as weshay|ruck | 20:38 | |
*** weshay|ruck is now known as weshay|rover | 20:38 | |
*** penick_ has joined #openstack-ironic | 20:40 | |
*** penick has quit IRC | 20:42 | |
openstackgerrit | Ihar Hrachyshka proposed openstack/ironic master: DNM testing whether lib/neutron switch breaks this repo https://review.openstack.org/535928 | 20:44 |
*** penick_ has quit IRC | 20:58 | |
*** penick has joined #openstack-ironic | 20:58 | |
*** Nisha_Agarwal has joined #openstack-ironic | 20:59 | |
*** e0ne has joined #openstack-ironic | 21:04 | |
*** e0ne has quit IRC | 21:05 | |
*** Nisha_Agarwal has quit IRC | 21:14 | |
*** jtomasek has quit IRC | 21:14 | |
*** bfournie1 has joined #openstack-ironic | 21:14 | |
*** beekneemech has quit IRC | 21:14 | |
*** Nisha_Agarwal has joined #openstack-ironic | 21:15 | |
*** crushil_ has joined #openstack-ironic | 21:15 | |
*** bfournie has quit IRC | 21:15 | |
*** bnemec has joined #openstack-ironic | 21:17 | |
*** makowals has quit IRC | 21:28 | |
*** e0ne has joined #openstack-ironic | 21:29 | |
*** e0ne has quit IRC | 21:33 | |
openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 21:38 |
openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 21:45 |
*** cjloader has quit IRC | 21:48 | |
*** cjloader has joined #openstack-ironic | 21:49 | |
jroll | TheJulia: your tests passed \o/ | 21:50 |
*** jcoufal has quit IRC | 21:53 | |
openstackgerrit | Yusef Shaban proposed openstack/sushy master: Closes-Bug: 1744378 https://review.openstack.org/535910 | 21:58 |
openstack | bug 1744378 in sushy "Connector ignoring verify" [Undecided,New] https://launchpad.net/bugs/1744378 | 21:58 |
*** crushil_ has quit IRC | 22:01 | |
*** tesseract has quit IRC | 22:01 | |
*** dprince has quit IRC | 22:01 | |
*** cjloader has quit IRC | 22:08 | |
*** fragatina has joined #openstack-ironic | 22:13 | |
*** Goneri has quit IRC | 22:14 | |
TheJulia | awww yeah \o/ | 22:16 |
*** baha has quit IRC | 22:20 | |
openstackgerrit | Julia Kreger proposed openstack/ironic master: Rework upgrade to upgrade nova/ironic https://review.openstack.org/535596 | 22:21 |
*** cjloader_ has joined #openstack-ironic | 22:21 | |
*** jlabarre has quit IRC | 22:28 | |
*** penick_ has joined #openstack-ironic | 22:28 | |
TheJulia | jroll: looks like you might have a stray change in that nova change. I marked it and gave it a +1 otherwise :) | 22:28 |
TheJulia | Anyway, going to get back on the road. 80 more miles to go and it is starting to get windy :( | 22:30 |
*** penick has quit IRC | 22:31 | |
*** penick has joined #openstack-ironic | 22:32 | |
*** penick_ has quit IRC | 22:33 | |
*** cjloader_ has quit IRC | 22:46 | |
*** figleaf is now known as edleafe | 22:55 | |
*** rbudden has quit IRC | 22:58 | |
*** Nisha_Agarwal has quit IRC | 23:02 | |
*** leakypipes has quit IRC | 23:14 | |
openstackgerrit | Yusef Shaban proposed openstack/sushy master: Closes-Bug: 1744378 https://review.openstack.org/535910 | 23:15 |
openstack | bug 1744378 in sushy "Connector ignoring verify" [Undecided,New] https://launchpad.net/bugs/1744378 | 23:15 |
*** karthiks has quit IRC | 23:16 | |
*** AlexeyAbashkin has joined #openstack-ironic | 23:22 | |
*** AlexeyAbashkin has quit IRC | 23:26 | |
openstackgerrit | Yusef Shaban proposed openstack/sushy master: Closes-Bug: 1744378 https://review.openstack.org/535910 | 23:30 |
openstack | bug 1744378 in sushy "Connector ignoring verify" [Undecided,New] https://launchpad.net/bugs/1744378 | 23:30 |
*** cdearborn has quit IRC | 23:31 | |
*** karthiks has joined #openstack-ironic | 23:35 | |
-openstackstatus- NOTICE: Zuul will be offline over the next 20 minutes to perform maintenance; active changes will be reenqueued once work completes, but new patch sets or approvals during that timeframe may need to be rechecked or reapplied as appropriate | 23:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!