| *** hamzy_ has joined #openstack-ironic | 00:02 | |
| *** itlinux has quit IRC | 00:03 | |
| *** fragatina has quit IRC | 00:04 | |
| *** fragatina has joined #openstack-ironic | 00:05 | |
| *** itlinux has joined #openstack-ironic | 00:06 | |
| *** mjturek has quit IRC | 00:07 | |
| *** hamzy has joined #openstack-ironic | 00:08 | |
| *** fragatina has quit IRC | 00:09 | |
| *** gyee has quit IRC | 00:10 | |
| *** hamzy_ has quit IRC | 00:10 | |
| *** Sukhdev has quit IRC | 00:10 | |
| *** jcoufal has joined #openstack-ironic | 00:12 | |
| *** I12S4Jaispark[m] has joined #openstack-ironic | 00:20 | |
| *** r-daneel has quit IRC | 00:21 | |
| *** I12S4Jaispark[m] has quit IRC | 00:25 | |
| openstackgerrit | Merged openstack/networking-generic-switch master: Sync hacking requirement with ironic https://review.openstack.org/560884 | 00:30 |
|---|---|---|
| *** liuzz_ has quit IRC | 00:39 | |
| *** srart has quit IRC | 00:40 | |
| *** liuzz has joined #openstack-ironic | 00:40 | |
| *** itlinux has quit IRC | 00:41 | |
| *** hshiina has joined #openstack-ironic | 00:45 | |
| *** itlinux has joined #openstack-ironic | 00:46 | |
| *** harlowja has quit IRC | 00:52 | |
| *** zhangfei has joined #openstack-ironic | 00:56 | |
| *** yangyape_ has quit IRC | 00:57 | |
| *** itlinux has quit IRC | 00:58 | |
| *** ianychoi_ is now known as ianychoi | 01:01 | |
| *** dprince has joined #openstack-ironic | 01:01 | |
| *** phuongnh has joined #openstack-ironic | 01:02 | |
| *** itlinux has joined #openstack-ironic | 01:08 | |
| *** tiendc has joined #openstack-ironic | 01:10 | |
| *** itlinux has quit IRC | 01:10 | |
| *** dprince has quit IRC | 01:12 | |
| *** itlinux has joined #openstack-ironic | 01:20 | |
| *** r-daneel has joined #openstack-ironic | 01:29 | |
| TheJulia | so... soooooo many things needing to be triaged https://storyboard.openstack.org/#!/board/67 | 01:32 |
| *** phuongnh has quit IRC | 01:33 | |
| *** yangyapeng has joined #openstack-ironic | 01:33 | |
| *** phuongnh has joined #openstack-ironic | 01:33 | |
| *** itlinux has quit IRC | 01:35 | |
| *** dprince has joined #openstack-ironic | 01:36 | |
| jroll | TheJulia: that's what I was saying, script it. anything from launchpad (ID < 2000000 I think?) definitely gets a triaged tag | 01:36 |
| TheJulia | jroll: except so many of those havne't been looked at in ages | 01:36 |
| jroll | yeah | 01:37 |
| jroll | but, they were marked as triaged on launchpad | 01:37 |
| TheJulia | I've already killed some tons of duplicates or things that we're just orphaned | 01:37 |
| jroll | so I guess it's a question of if you want to go back through | 01:37 |
| TheJulia | it should only be ~400 total | 01:37 |
| jroll | "only" :P | 01:39 |
| TheJulia | If I do like 30 a day... | 01:39 |
| TheJulia | s/do/look at/ | 01:39 |
| *** yangyapeng has quit IRC | 01:39 | |
| TheJulia | jroll: that is also pan-ironic, not just the ironic launchpad | 01:42 |
| jroll | TheJulia: yeah. just a lot of work :/ | 01:43 |
| TheJulia | Yeah :( | 01:45 |
| *** jcoufal has quit IRC | 01:53 | |
| *** dprince has quit IRC | 01:54 | |
| *** yangyapeng has joined #openstack-ironic | 01:56 | |
| openstackgerrit | Zachary Ma proposed openstack/ironic stable/queens: Check for PXE-enabled ports when creating neutron ports https://review.openstack.org/561097 | 01:57 |
| *** yangyapeng has quit IRC | 01:58 | |
| TheJulia | jroll: I guess in the end it is one of those "we've got to do it at some point" things, and it is good to re-visit as expectations/project goals change | 02:01 |
| *** phuongnh has quit IRC | 02:02 | |
| * TheJulia renames the board "Ironic Triage Panopticon" | 02:06 | |
| * TheJulia may have lost her mind | 02:09 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Use database to store introspection_data https://review.openstack.org/514552 | 02:10 |
| *** phuongnh has joined #openstack-ironic | 02:16 | |
| *** phuongnh has quit IRC | 02:21 | |
| *** phuongnh has joined #openstack-ironic | 02:22 | |
| *** d0ugal has quit IRC | 02:43 | |
| *** d0ugal has joined #openstack-ironic | 02:49 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic master: Power fault recovery: applying maintenance_type https://review.openstack.org/556758 | 02:53 |
| openstackgerrit | jiapei proposed openstack/ironic master: Add documentatin for XClarity Driver https://review.openstack.org/559960 | 03:11 |
| *** fragatina has joined #openstack-ironic | 03:16 | |
| *** fragatina has quit IRC | 03:17 | |
| *** fragatina has joined #openstack-ironic | 03:17 | |
| *** fragatina has quit IRC | 03:42 | |
| *** teclo-lg has joined #openstack-ironic | 03:48 | |
| *** itlinux has joined #openstack-ironic | 03:49 | |
| *** teclo-lg has quit IRC | 03:51 | |
| *** CrazyEddykh has joined #openstack-ironic | 04:06 | |
| *** CrazyEddykh has quit IRC | 04:08 | |
| *** rajinir has quit IRC | 04:14 | |
| *** Nisha_Agarwal has joined #openstack-ironic | 04:17 | |
| *** itlinux has quit IRC | 04:23 | |
| *** namnh has joined #openstack-ironic | 04:24 | |
| *** lbragstad has joined #openstack-ironic | 04:26 | |
| *** lbragstad has quit IRC | 04:28 | |
| *** v12aml has quit IRC | 04:40 | |
| *** v12aml has joined #openstack-ironic | 04:43 | |
| *** harlowja has joined #openstack-ironic | 04:44 | |
| *** diga has joined #openstack-ironic | 04:46 | |
| *** mvpnitesh has joined #openstack-ironic | 04:53 | |
| *** liuzz_ has joined #openstack-ironic | 04:56 | |
| *** liuzz has quit IRC | 04:59 | |
| *** links has joined #openstack-ironic | 05:12 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Use database to store introspection_data https://review.openstack.org/514552 | 05:25 |
| *** cjloader has joined #openstack-ironic | 05:27 | |
| *** mordred has quit IRC | 05:29 | |
| *** cjloader has quit IRC | 05:32 | |
| *** e0ne has joined #openstack-ironic | 05:35 | |
| *** marios has joined #openstack-ironic | 05:37 | |
| *** harlowja has quit IRC | 05:39 | |
| *** mordred has joined #openstack-ironic | 05:42 | |
| *** fragatina has joined #openstack-ironic | 05:44 | |
| *** zhangfei has quit IRC | 06:09 | |
| *** sambetts_ is now known as sambetts | 06:21 | |
| *** zhangfei has joined #openstack-ironic | 06:21 | |
| pas-ha | morning ironic :) | 06:28 |
| *** hshiina2 has joined #openstack-ironic | 06:29 | |
| *** hshiina has quit IRC | 06:32 | |
| *** phuongnh has quit IRC | 06:32 | |
| *** phuongnh has joined #openstack-ironic | 06:33 | |
| *** parlos has joined #openstack-ironic | 06:34 | |
| *** liuzz has joined #openstack-ironic | 06:35 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 06:37 | |
| *** liuzz_ has quit IRC | 06:37 | |
| *** e0ne has quit IRC | 06:38 | |
| *** phuongnh has quit IRC | 06:45 | |
| *** phuongnh has joined #openstack-ironic | 06:47 | |
| *** skazi has quit IRC | 06:54 | |
| *** skazi_ has joined #openstack-ironic | 06:54 | |
| *** phuongnh has quit IRC | 06:57 | |
| *** AlexeyAbashkin has quit IRC | 06:58 | |
| *** phuongnh has joined #openstack-ironic | 07:01 | |
| openstackgerrit | Zachary Ma proposed openstack/networking-generic-switch master: Add dynamic-portgroups spec for NGS https://review.openstack.org/561138 | 07:10 |
| *** hshiina2 has quit IRC | 07:10 | |
| *** jtomasek has joined #openstack-ironic | 07:13 | |
| *** tesseract has joined #openstack-ironic | 07:17 | |
| *** rcernin has quit IRC | 07:33 | |
| *** namnh_ has joined #openstack-ironic | 07:33 | |
| * milan is back (gone 16:42:14) | 07:35 | |
| *** namnh has quit IRC | 07:35 | |
| *** dougsz has joined #openstack-ironic | 07:39 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 07:41 | |
| openstackgerrit | Zachary Ma proposed openstack/networking-generic-switch master: Huawei vrpv8 switch support port group https://review.openstack.org/551922 | 07:41 |
| *** gcb has joined #openstack-ironic | 07:43 | |
| *** AlexeyAbashkin has quit IRC | 07:46 | |
| *** namnh_ has quit IRC | 07:53 | |
| *** hshiina has joined #openstack-ironic | 08:07 | |
| *** mgoddard has joined #openstack-ironic | 08:09 | |
| *** mvpnitesh has quit IRC | 08:10 | |
| *** gcb has quit IRC | 08:11 | |
| *** ktibi has joined #openstack-ironic | 08:16 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Introducing RPC https://review.openstack.org/497689 | 08:17 |
| *** lucas-afk is now known as lucasagomes | 08:17 | |
| Nisha_Agarwal | morning pas-ha | 08:21 |
| Nisha_Agarwal | pas-ha, or anyone else here in the IRC room, i was trying nova boot for pike stable setup. I am seeing the issue that as soon as the node is set to active, it receives the "delete" event and do_node_tear_down() gets called. Has anyone faced such issue before? The logs can be seen here http://paste.openstack.org/show/719127/ | 08:23 |
| *** ktibi_ has joined #openstack-ironic | 08:23 | |
| *** ktibi_ has quit IRC | 08:23 | |
| Nisha_Agarwal | jroll, ping | 08:23 |
| *** ktibi_ has joined #openstack-ironic | 08:24 | |
| *** ktibi has quit IRC | 08:26 | |
| *** mvpnitesh has joined #openstack-ironic | 08:35 | |
| *** derekh has joined #openstack-ironic | 08:41 | |
| Nisha_Agarwal | sambetts, ping | 08:47 |
| openstackgerrit | Debayan Ray proposed openstack/sushy master: Add storage volume https://review.openstack.org/560981 | 08:48 |
| sambetts | Nisha_Agarwal: that node never makes it to active, it goes from wait_call_back to deleting | 08:49 |
| Nisha_Agarwal | sambetts, true | 08:49 |
| Nisha_Agarwal | sambetts, but the strange part is when i use "ironic node-set-provision-state $NODE active" it works | 08:49 |
| Nisha_Agarwal | sambetts, but when i use nova boot i always see the above behaviour | 08:50 |
| Nisha_Agarwal | i am still not able to find out from where itgets the "delete" event which triggers the tear_down | 08:50 |
| sambetts | Nisha_Agarwal: nova has a instance boot timeout which if it is exceeded nova will tear down the deploy and retry on a different node | 08:50 |
| Nisha_Agarwal | could u tellme the config variable name | 08:51 |
| Nisha_Agarwal | i also thought of timeout variable | 08:51 |
| Nisha_Agarwal | but couldnt figure out the variable | 08:51 |
| sambetts | I expect your hitting that issue, for more logs I think you need to look at the ironic nova compute node logs and the nova schedular logs | 08:51 |
| Nisha_Agarwal | hmm | 08:52 |
| sambetts | [DEFAULT] instance_build_timeout is the config option in question | 08:52 |
| sambetts | that you might be exceeding | 08:53 |
| Nisha_Agarwal | sambetts, thanks ....in my setup its not even defined | 08:53 |
| Nisha_Agarwal | in any of the conf file | 08:53 |
| Nisha_Agarwal | thanks for the pointer i will check and see the value | 08:54 |
| sambetts | Nisha_Agarwal: I think the other thing that could trigger it is some error on the nova side because then it'll try to abort the instance build, so worth checking the nova compute logs for errors | 08:55 |
| Nisha_Agarwal | sambetts, yeah i checked but couldnt find anything like that. i will try setting up the timeout and see if issue get resolved | 08:56 |
| Nisha_Agarwal | sambetts, thanks for the help :) | 08:57 |
| sambetts | Nisha_Agarwal: according to the nova docs if you set it to zero it'll disable it | 08:57 |
| sambetts | (although not ideal, might be worth doing to see if that was the issue without having to work out a real timeout vlaue) | 08:57 |
| Nisha_Agarwal | sambetts, and the default value is 0 , so when it is not set explicitly in the conf file it should take 0 | 08:58 |
| Nisha_Agarwal | which means disabling timeout | 08:58 |
| sambetts | oh :( then I guess it might be hitting some other issue :/ | 08:58 |
| *** stendulker has joined #openstack-ironic | 08:59 | |
| *** ktibi_ is now known as ktibi | 09:07 | |
| openstackgerrit | Hironori Shiina proposed openstack/ironic master: Stop verifying updated driver in creating task https://review.openstack.org/561169 | 09:08 |
| *** hshiina has quit IRC | 09:11 | |
| *** pcaruana has joined #openstack-ironic | 09:12 | |
| yolanda | hi, seems that grenade jobs are failing http://logs.openstack.org/09/528609/22/check/ironic-grenade-dsvm-multinode-multitenant/b2a23b0/ | 09:15 |
| yolanda | is there some issue? | 09:15 |
| openstackgerrit | Zachary Ma proposed openstack/networking-generic-switch master: Huawei vrpv8 switch support port group https://review.openstack.org/551922 | 09:24 |
| *** liuzz has quit IRC | 09:32 | |
| olivierb- | Hello everyone, I have a potentially very basic question but after digging for a bit of time today on it I can not understand the in and outs of my issue or may be this is not an issue at all just something I missed. Here is the stuff: I have used DIB to successfully generate a ubuntu-minimal based image with LVM support that I use to provision a set of machines/VMs using Bifrost/Ironic and this works wonderfully well. However | 09:44 |
| olivierb- | when I install some packages on those machines, these packages regenerate the initramfs and kernel but very strangely (at least from my perspective) the new ramdisk is missing a lot of stuff from my original (1 third in size) but also some other stuff added in the new one. Anyways the worse part is that this new ramdisk+kernel does not boot successfully as lvm and xfs drivers are not included (at minima). My question is therefore | 09:44 |
| olivierb- | why does this behaviour occur, may be this is not relevant to DIB itself and more a generic linux issue but as a Linux user I would (naively ?) expect that rebuilding a kernel/ramdisk from a working system should lead to a working situation. Sorry for the multiple posts cross channels but even though some of you are also on the dib channel I thought on maximizing my chances here | 09:44 |
| olivierb- | any pointer, advice, explanation, hint greatly appreciated | 09:44 |
| *** e0ne has joined #openstack-ironic | 09:47 | |
| vdrok | yolanda: don't really get what's wrong there, pep8 is installed http://logs.openstack.org/09/528609/22/check/ironic-grenade-dsvm-multinode-multitenant/b2a23b0/logs/grenade.sh.txt.gz#_2018-04-13_07_48_28_179 and then it complains that it is not http://logs.openstack.org/09/528609/22/check/ironic-grenade-dsvm-multinode-multitenant/b2a23b0/logs/grenade.sh.txt.gz#_2018-04-13_07_48_50_882 | 10:05 |
| *** phuongnh has quit IRC | 10:14 | |
| *** pbourke has quit IRC | 10:15 | |
| *** pbourke has joined #openstack-ironic | 10:16 | |
| *** jiapei has joined #openstack-ironic | 10:20 | |
| jiapei | Hello everyone | 10:21 |
| sambetts | olivierb-: My guess is that your initramfs hasn't been correctly told which kernel modules it needs to enable, you can check by looking at /etc/initramfs-tools/modules, and you can add lvm by doing echo "lvm" >> /etc/initramfs-tools/modules then initramfs-update -k -c all | 10:28 |
| *** diga has quit IRC | 10:43 | |
| *** oanson has quit IRC | 10:44 | |
| *** tiendc has quit IRC | 11:07 | |
| *** niteshmvp has joined #openstack-ironic | 11:16 | |
| *** stendulker has quit IRC | 11:17 | |
| *** Nisha_Agarwal has quit IRC | 11:18 | |
| *** mvpnitesh has quit IRC | 11:20 | |
| *** lucasagomes is now known as lucas-hungry | 11:27 | |
| *** dprince has joined #openstack-ironic | 11:30 | |
| *** dprince has quit IRC | 11:30 | |
| *** dprince has joined #openstack-ironic | 11:31 | |
| * milan is away: I'm busy | 11:35 | |
| *** milan has quit IRC | 11:36 | |
| *** milan has joined #openstack-ironic | 11:36 | |
| *** jiapei has quit IRC | 11:37 | |
| olivierb- | sambetts this is what I guessed and I have been using yolanda advices to build my dib reference image but using dracut-regenerate not initramfs, may be this is (partly) where the issue may be. Do I need to provide my dib environment some extra infos so that the deployed image can be updated into idempotent image with same drivers/... ? | 11:38 |
| *** jjohnson2_ has quit IRC | 11:38 | |
| *** nizam037 has joined #openstack-ironic | 11:42 | |
| *** bstn has joined #openstack-ironic | 11:42 | |
| *** milan has quit IRC | 11:43 | |
| *** milan has joined #openstack-ironic | 11:44 | |
| nizam037 | hi.. | 11:45 |
| nizam037 | any fixes for tftp time out during deployment? | 11:45 |
| *** niteshmvp has quit IRC | 11:57 | |
| *** niteshmvp has joined #openstack-ironic | 11:58 | |
| openstackgerrit | Merged openstack/networking-generic-switch master: support to recover default vlan when delete port https://review.openstack.org/548877 | 12:00 |
| jroll | urgh | 12:07 |
| jroll | yolanda: I've been working on the grenade failure, patch incoming *should* fix it | 12:08 |
| openstackgerrit | Jim Rollenhagen proposed openstack/networking-generic-switch stable/queens: Sync hacking requirement with ironic https://review.openstack.org/561204 | 12:11 |
| *** trown|outtypewww is now known as trown | 12:11 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic master: Test n-g-s fix https://review.openstack.org/560886 | 12:13 |
| jroll | and to verify | 12:13 |
| *** Nisha_Agarwal has joined #openstack-ironic | 12:14 | |
| Nisha_Agarwal | jroll, vdrok hi | 12:16 |
| jroll | morning Nisha_Agarwal | 12:16 |
| Nisha_Agarwal | I was trying nova boot in pike setup | 12:16 |
| Nisha_Agarwal | jroll, morning | 12:17 |
| vdrok | jroll: so we'll need to backport up until ocata if we want grenade on stable branches to pass? | 12:17 |
| vdrok | also morning jroll Nisha_Agarwal | 12:17 |
| jroll | vdrok: I'm not sure, we'll need to check the stable branches | 12:17 |
| Nisha_Agarwal | jroll, i see this issue consistently getting reproduced for pike release when i do nova boot | 12:17 |
| jroll | I'm not even 100% this will work, but I'm fairly sure :) | 12:17 |
| Nisha_Agarwal | http://paste.openstack.org/show/719143/ | 12:17 |
| jroll | Nisha_Agarwal: oof, that's not good. I haven't seen that in a long time | 12:18 |
| Nisha_Agarwal | vdrok, morning! | 12:19 |
| Nisha_Agarwal | jroll, any idea why that kind of error can come? | 12:19 |
| jroll | Nisha_Agarwal: it'd be a coding error in nova | 12:19 |
| jroll | looks like we had one fixed in newton: https://bugs.launchpad.net/nova/+bug/1580987 | 12:20 |
| openstack | Launchpad bug 1580987 in OpenStack Compute (nova) ""Cannot call obj_load_attr on orphaned Instance object" in baremetal_basic_ops" [High,Fix released] - Assigned to Jim Rollenhagen (jim-rollenhagen) | 12:20 |
| jroll | Nisha_Agarwal: this is latest pike with no nova patches? | 12:20 |
| Nisha_Agarwal | because of this the node immediately gets delete event from nova as soon as it converts to wait-call-back | 12:20 |
| jroll | yes I saw scrollback | 12:20 |
| Nisha_Agarwal | this is latest pike | 12:21 |
| *** jcoufal has joined #openstack-ironic | 12:21 | |
| Nisha_Agarwal | the one i am using | 12:21 |
| Nisha_Agarwal | i stacked it just 2 days back | 12:21 |
| Nisha_Agarwal | actually i was trying creating cluster using magnum | 12:21 |
| Nisha_Agarwal | with ironic | 12:21 |
| Nisha_Agarwal | but it always failed at nova boot | 12:21 |
| jroll | did you get the same doing nova boot without magnum? | 12:22 |
| Nisha_Agarwal | jroll, yes | 12:22 |
| Nisha_Agarwal | now i doing wthout magnum | 12:22 |
| Nisha_Agarwal | s/i/i am | 12:22 |
| jroll | looks like CI on pike is mostly happy: https://review.openstack.org/#/c/560845/ | 12:23 |
| patchbot | patch 560845 - ironic (stable/pike) - Remove too large configdrive for handling error | 12:23 |
| Nisha_Agarwal | jroll, i checked the above fix and the nova code has that fix | 12:23 |
| jroll | wonder what could be going wrong there | 12:23 |
| Nisha_Agarwal | will it help if we increase config variable "api_retry_interval" as it fails at line "timer.start(interval=CONF.ironic.api_retry_interval).wait()" | 12:24 |
| Nisha_Agarwal | it currently has the value set as 5 | 12:24 |
| jroll | I don't think so | 12:25 |
| *** edleafe is now known as figleaf | 12:26 | |
| jroll | this is where it's coming from https://github.com/openstack/nova/blob/stable/pike/nova/virt/ironic/driver.py#L473 | 12:26 |
| *** mvpnitesh has joined #openstack-ironic | 12:26 | |
| *** lucas-hungry is now known as lucasagomes | 12:27 | |
| jroll | Nisha_Agarwal: I'm actually wondering if the error you posted is being caused by pdb. do you see that error in the nova-compute logs without pdb? | 12:28 |
| jroll | things like this are usually weird threading errors and such | 12:28 |
| *** Nisha_Agarwal has quit IRC | 12:31 | |
| *** marios has quit IRC | 12:31 | |
| *** marios has joined #openstack-ironic | 12:31 | |
| mvpnitesh | hi jroll, nisha got disconnected, I'm working along with Nisha | 12:31 |
| mvpnitesh | @jroll we are seeing that error with out pdb also | 12:32 |
| jroll | mvpnitesh: can you paste those logs please? | 12:34 |
| jroll | the nova-compute logs specifically | 12:34 |
| *** jjohnson2_ has joined #openstack-ironic | 12:37 | |
| jroll | ok wtf http://logs.openstack.org/86/560886/2/check/ironic-grenade-dsvm-multinode-multitenant/fabe350/logs/grenade.sh.txt.gz#_2018-04-13_12_32_50_830 | 12:47 |
| jroll | now I'm just confused | 12:47 |
| mvpnitesh | jroll: we have the same logs, i'll try to boot it again and will share the logs once i get the error | 12:50 |
| jroll | thanks | 12:51 |
| *** rloo has joined #openstack-ironic | 12:51 | |
| jroll | huh, I can't reproduce this n-g-s thing locally either | 12:54 |
| *** Goneri has joined #openstack-ironic | 12:59 | |
| *** r-daneel has quit IRC | 13:09 | |
| *** cdearborn has joined #openstack-ironic | 13:15 | |
| *** cdearborn_ has joined #openstack-ironic | 13:19 | |
| *** niteshmvp has quit IRC | 13:29 | |
| *** mvpnitesh has quit IRC | 13:30 | |
| *** r-daneel has joined #openstack-ironic | 13:30 | |
| rloo | Good Friday morning everyone. | 13:31 |
| *** dansmith is now known as superdan | 13:31 | |
| rloo | Err, at least it was good until I skimmed the logs from earlier today. So grenade multi* is broken :-( I'll update the whiteboard. | 13:31 |
| *** cdearborn has quit IRC | 13:32 | |
| * rloo sees that it was partially updated. | 13:32 | |
| jroll | yeah, was broken yesterday too :/ | 13:33 |
| * rloo is wrong, that was a diff failure from dec. | 13:33 | |
| rloo | jroll: is that the n-g-s thing? that now you're thinking isn't the fix? | 13:33 |
| jroll | rloo: right, my test patch failed | 13:33 |
| jroll | having trouble reproducing locally | 13:34 |
| rloo | probably coincidence, but the last patch that passed was the uncapping of eventlet. (that's the extent of my short investigation so far.) | 13:34 |
| jroll | hm | 13:35 |
| jroll | probably unrelated | 13:35 |
| rloo | better that it started failing yesterday than on Friday the 13th!!! | 13:35 |
| jroll | heh | 13:35 |
| *** lbragstad has joined #openstack-ironic | 13:36 | |
| olivierb- | can someone please tell me what's wrong with https://review.openstack.org/558496 CI jobs faiures ? | 13:45 |
| patchbot | patch 558496 - bifrost - Fix for proper PATH when calling keystone-manage c... | 13:46 |
| jroll | oh god | 13:47 |
| rloo | ? | 13:47 |
| jroll | just red across the baord there | 13:47 |
| jroll | seems bifrost-specific though http://logs.openstack.org/96/558496/1/check/bifrost-integration-tinyipa/4afdacf/job-output.txt.gz#_2018-04-13_12_22_14_962182 | 13:48 |
| *** hoonetorg has quit IRC | 13:48 | |
| rloo | OH. looks like it failed on apr 10 too. | 13:50 |
| rloo | ahhh, bifrost was failing on apr 6 too: https://review.openstack.org/#/c/367608/ | 13:52 |
| patchbot | patch 367608 - bifrost - Test OPNFV CI verify job. Please ignore. | 13:52 |
| * TheJulia puts a zombie sign in front of her | 13:53 | |
| jroll | TheJulia: run away, run far far away :) | 13:53 |
| TheJulia | jroll: why? | 13:53 |
| rloo | Good morning TheJulia, feel like it is a bifrost day for you? :) | 13:53 |
| jroll | TheJulia: we're just debugging multiple CI failures, nbd | 13:54 |
| TheJulia | it would be an awesome change of pace after all.... | 13:54 |
| * TheJulia offers to trade triaging | 13:54 | |
| rloo | the good news is that none of them are due to Fri the 13. | 13:54 |
| * rloo is not sure which is worse, triaging or debugging | 13:54 | |
| * rloo cuz they are both great! | 13:55 | |
| TheJulia | rloo: lol | 13:55 |
| TheJulia | the triage panopticon that I made actually has made it easy | 13:56 |
| TheJulia | the new rfe's column searches might be broken :\ | 13:56 |
| TheJulia | I might have hit a bug in the search creation | 13:56 |
| rloo | jroll: so your ngs patch (https://review.openstack.org/#/c/560884/) seems to align with the breakage? is that what you're trying to fix then? do we want the backport? | 13:58 |
| patchbot | patch 560884 - networking-generic-switch - Sync hacking requirement with ironic (MERGED) | 13:58 |
| jroll | rloo: we might want the backport just for alignment, but my test patch says it doesn't fix the issue :( | 13:58 |
| rloo | jroll: what if we revert your original patch? | 13:59 |
| jroll | rloo: no, that patch was post-breakage | 13:59 |
| TheJulia | so we have multiple distinct breakages this morning? | 13:59 |
| TheJulia | does the whiteboard have a list? | 13:59 |
| rloo | TheJulia: two that we know of. i haven't gotten around to updating with the bifrost one. i'll do that now. | 14:00 |
| jroll | TheJulia: we have our grenade failure on there, just discovered the bifrost thing, see https://review.openstack.org/558496 and http://logs.openstack.org/96/558496/1/check/bifrost-integration-tinyipa/4afdacf/job-output.txt.gz#_2018-04-13_12_22_14_962182 | 14:00 |
| patchbot | patch 558496 - bifrost - Fix for proper PATH when calling keystone-manage c... | 14:00 |
| *** bstn has quit IRC | 14:03 | |
| TheJulia | i bet it is ansible 2.5 | 14:03 |
| TheJulia | "msg": "Invalid argument given. Must be one of: seconds, minutes, prompt", | 14:03 |
| rloo | TheJulia: i updated the whiteboard, but I don't know much about bifrost so whoever looks into it, please update. | 14:03 |
| TheJulia | something changed in the parsing | 14:03 |
| rloo | olivierb-: if you are following along, hopefully bifrost will get fixed soon ^^ | 14:04 |
| * jroll asks in infra about grenade real quick, in case he's missing something obvious | 14:05 | |
| olivierb- | rloo many thanks for this | 14:05 |
| jroll | TheJulia: oh, I thought the failure was not being able to find /tmp/baremetal.json.rest | 14:06 |
| jroll | totally missed the second bit | 14:06 |
| rloo | olivierb-: I'm just the messenger :) | 14:06 |
| jroll | also trying to find the ansible version in that log is insanely hard :P | 14:06 |
| olivierb- | thanks to all involved and TheJulia in particular then ;-) | 14:08 |
| *** baha has joined #openstack-ironic | 14:08 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Fix syntax error for ansible 2.5 https://review.openstack.org/561232 | 14:10 |
| TheJulia | that _should_ fix it *crosses her fingers* | 14:10 |
| TheJulia | we'll know in like 30 minutes | 14:11 |
| TheJulia | jroll: I only thought of 2.5 because of the break it had on ironic's gate | 14:11 |
| jroll | TheJulia: where do we define the ansible requirement? | 14:12 |
| jroll | just curious | 14:12 |
| TheJulia | it used to be locked hard in the env-setup script but some contributors wanted it to accept new versions since the contract is that no new version is _really_ supposed to break old playbooks (in reality....... ehhhhh) | 14:13 |
| TheJulia | we've had to pin it a few times in the past as well | 14:13 |
| jroll | so we just try to support everything? | 14:13 |
| TheJulia | I believe it is 2.3 onward | 14:13 |
| TheJulia | but it might be 2.4 now | 14:13 |
| rloo | # Job Information | 14:14 |
| rloo | Ansible Version: 2.3.3.0 | 14:14 |
| rloo | Job: bifrost-integration-tinyipa | 14:14 |
| rloo | that's what i see in the logs ^^ | 14:15 |
| TheJulia | that might be zuul's version | 14:15 |
| jroll | that's my thought | 14:15 |
| rloo | err http://logs.openstack.org/96/558496/1/check/bifrost-integration-tinyipa/4afdacf/job-output.txt.gz#_2018-04-13_12_06_50_885650 | 14:15 |
| rloo | too many things... :-( | 14:15 |
| * TheJulia retreats to a sun puddle to wake up | 14:16 | |
| * jroll can't find where ansible is installed on the host | 14:16 | |
| *** rajinir has joined #openstack-ironic | 14:17 | |
| rloo | jroll: i see this, 2.4.4: http://logs.openstack.org/96/558496/1/check/bifrost-integration-tinyipa/4afdacf/job-output.txt.gz#_2018-04-13_12_09_45_786671 ? | 14:20 |
| jroll | rloo: aha, thanks | 14:20 |
| rloo | which isn't 2.5. well, we can wait and see. | 14:21 |
| *** mjturek has joined #openstack-ironic | 14:21 | |
| jroll | well, 2.4.4.0 was released apr 4 | 14:21 |
| rloo | jroll: ohhhh, that would time nicely with the breakages | 14:22 |
| jroll | at any rate, julia's fix is probably right | 14:22 |
| rloo | and now i wish there was a dashboard to see all the ironic projects, and see which ones might be failing. oh well. | 14:22 |
| rloo | even if julia's fix doesn't fix *this* thing, it is a good fix anyway. | 14:23 |
| *** bnemec is now known as beekneemech | 14:23 | |
| jroll | yep | 14:23 |
| jroll | https://github.com/ansible/ansible/commit/6598c8e361648c9b9066f938a9822c29d3716fa5#diff-3fbd8ab6a6e5c90497a2d84142b3a767 | 14:25 |
| jroll | bam | 14:25 |
| * jroll +2's | 14:26 | |
| *** nizam037 has quit IRC | 14:31 | |
| olivierb- | anyone available here with experience on making deboostrap dib element work with LVM ? | 14:44 |
| olivierb- | I currentl use dracut-regenerate following yolanda advices and it works at least for producing a bootable image | 14:45 |
| *** AlexeyAbashkin has joined #openstack-ironic | 14:45 | |
| olivierb- | but updating the image afterwards just fails when rebooting | 14:45 |
| *** AlexeyAbashkin has quit IRC | 14:49 | |
| *** openstackgerrit has quit IRC | 14:50 | |
| *** jcoufal_ has joined #openstack-ironic | 14:51 | |
| *** melwitt is now known as jgwentworth | 14:53 | |
| *** jcoufal has quit IRC | 14:54 | |
| derekh | Is anybody around able to look at a small ironic_staging_drivers patch? | 14:56 |
| *** cjloader has joined #openstack-ironic | 14:58 | |
| *** links has quit IRC | 14:58 | |
| *** cjloader has quit IRC | 14:58 | |
| *** cjloader has joined #openstack-ironic | 14:59 | |
| jroll | depends on the patch :) | 15:00 |
| *** livelace-link has joined #openstack-ironic | 15:10 | |
| TheJulia | dmitry and pas-ha have access to land patches in that repo | 15:10 |
| TheJulia | I can look, but I can't do anything besides +1/-1 | 15:10 |
| jroll | ah, didn't realize we weren't cores | 15:10 |
| TheJulia | no, it is out of our governance | 15:11 |
| TheJulia | for $reasons | 15:11 |
| jroll | right, that was me :P | 15:11 |
| TheJulia | heh | 15:11 |
| rloo | I just looked, vdrok is also core: https://review.openstack.org/#/admin/groups/1258,members | 15:11 |
| TheJulia | \o/ | 15:12 |
| *** pcaruana has quit IRC | 15:12 | |
| vdrok | Will do a bit later today | 15:13 |
| TheJulia | vdrok: thanks! | 15:15 |
| jroll | I can't reproduce this n-g-s thing at all :( | 15:22 |
| rloo | jroll: :-( any help from infra folks? | 15:24 |
| *** zhangfei has quit IRC | 15:24 | |
| jroll | rloo: no, I'll post again | 15:24 |
| rloo | jroll: what changed to break it though? | 15:25 |
| jroll | rloo: no clue, honestly. upstream release of *something*, can't decide what | 15:25 |
| rpioso|afk | Good morning | 15:26 |
| *** openstackgerrit has joined #openstack-ironic | 15:26 | |
| openstackgerrit | Harald Jensås proposed openstack/networking-baremetal master: ML2 Agent: Handle SIGHUP mutable config options https://review.openstack.org/561257 | 15:26 |
| *** rpioso|afk is now known as rpioso | 15:26 | |
| TheJulia | jroll: I thought it was a fluke yesterday :\ | 15:30 |
| TheJulia | but yeah :( | 15:30 |
| jroll | TheJulia: nope | 15:30 |
| TheJulia | esp when I couldn't reproduce it locally | 15:30 |
| jroll | yeah, same | 15:30 |
| jroll | have tried many iterations of pip commands :( | 15:30 |
| jroll | huh, it's already lunchtime | 15:30 |
| * jroll bbiab | 15:31 | |
| TheJulia | pas-ha: so there are two items on ironic-staging-drivers for ovirt that would be helpful to merge. vdrok is going to try to take a look as well later today, but we have some QA people that would <3 for them to merge soon | 15:36 |
| *** Nisha_Agarwal has joined #openstack-ironic | 15:37 | |
| *** srart has joined #openstack-ironic | 15:37 | |
| *** e0ne has quit IRC | 15:39 | |
| *** beekneemech has quit IRC | 15:45 | |
| *** itlinux has joined #openstack-ironic | 15:48 | |
| *** bnemec has joined #openstack-ironic | 15:49 | |
| derekh | opps just noticed I asked about that review without posting a link, vdrok this is the one I was talking about https://review.openstack.org/#/c/559353/ | 15:49 |
| patchbot | patch 559353 - ironic-staging-drivers - Convert ovirt url to a str | 15:49 |
| *** bnemec is now known as beekneemech | 15:50 | |
| *** baha has quit IRC | 15:51 | |
| *** dprince has quit IRC | 15:56 | |
| openstackgerrit | chandra shekar proposed openstack/ironic-specs master: Added new spec for L3 based Ironic deployment https://review.openstack.org/543936 | 15:58 |
| rloo | dumb question, this doesn't test grenade-multi*. Is it because only requirements.txt was changed? https://review.openstack.org/#/c/560498/ | 16:00 |
| patchbot | patch 560498 - ironic - uncap eventlet (MERGED) | 16:00 |
| *** diga has joined #openstack-ironic | 16:03 | |
| NobodyCam | Good Morning Ironic'ers... And Ofc.. | 16:06 |
| NobodyCam | TGIF. | 16:06 |
| *** r-daneel has quit IRC | 16:06 | |
| *** lucasagomes is now known as lucas-afk | 16:08 | |
| rloo | good morning NobodyCam :) | 16:08 |
| NobodyCam | Good Morning rloo | 16:08 |
| NobodyCam | :) | 16:08 |
| NobodyCam | happy Friday | 16:09 |
| *** athomas has quit IRC | 16:10 | |
| Nisha_Agarwal | jroll, ping | 16:10 |
| Nisha_Agarwal | jroll, here are the logs for the issue without pdb http://paste.openstack.org/show/719172/ | 16:10 |
| anupn | good morning NobodyCam, rloo and all others :) | 16:11 |
| NobodyCam | morning anupn :) | 16:11 |
| rloo | morning anupn | 16:11 |
| *** dprince has joined #openstack-ironic | 16:14 | |
| *** pcaruana has joined #openstack-ironic | 16:15 | |
| jroll | rloo: oof, yes, that seems very wrong | 16:18 |
| rloo | jroll: maybe an omission. i don't think it has anything to do with the grenade breakage, but noticed it. | 16:20 |
| jroll | yeah, we fail in queens, so shouldn't be related | 16:20 |
| rloo | i can take a look... that'd be easier to fix than the breakage :) | 16:20 |
| jroll | it's been a problem at least since we brought our jobs into our tree, probably longer | 16:21 |
| jroll | rloo: it's in zuul.d/legacy-ironic-jobs.yaml | 16:22 |
| rloo | oh. well, why fix it then! :) | 16:22 |
| jroll | - ^(test-|driver-)requirements.txt$ | 16:22 |
| jroll | er | 16:22 |
| rloo | damn you're fast | 16:22 |
| jroll | - ^(test-|driver-|)requirements.txt$ | 16:22 |
| jroll | that little extra | | 16:22 |
| rloo | i'm still downloading repos to my box... | 16:22 |
| jroll | though devstack installs test-requirements, so we should probably test that | 16:22 |
| jroll | heh | 16:22 |
| jroll | Nisha_Agarwal: strange, idk what would cause that, I guess I'd recommend filing a nova bug or something. it doesn't happen in upstream CI, so I guess I'm curious what is different there | 16:24 |
| *** r-daneel has joined #openstack-ironic | 16:24 | |
| Nisha_Agarwal | jroll, yeah its surprising.....i see the same error while it provisions and subsequently when it sends delete event to ironic for deleting the instance from nova. | 16:26 |
| Nisha_Agarwal | I am just cross checking other things but still strange.... | 16:26 |
| *** ktibi has quit IRC | 16:28 | |
| *** baha has joined #openstack-ironic | 16:33 | |
| *** marios has quit IRC | 16:34 | |
| TheJulia | I feel like I've seen that before, i jsut can't place it | 16:44 |
| openstackgerrit | Ruby Loo proposed openstack/ironic master: Run jobs if requirements change https://review.openstack.org/561269 | 16:47 |
| *** e0ne has joined #openstack-ironic | 16:48 | |
| jroll | TheJulia: it's kind of the generic error that pops up with threading problems in nova. we had one recurring that we fixed in newton, but no others in launchpad since | 16:50 |
| TheJulia | yeah, that feels about right | 16:50 |
| * jroll has no clue why this would be happening though | 16:51 | |
| *** gyee has joined #openstack-ironic | 16:51 | |
| *** links has joined #openstack-ironic | 16:53 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector stable/queens: Switch documentation to hardware types https://review.openstack.org/554586 | 16:54 |
| *** derekh has quit IRC | 17:00 | |
| *** mgoddard has quit IRC | 17:10 | |
| *** sambetts is now known as sambetts|afk | 17:13 | |
| sambetts|afk | night all, have great weekends o/ | 17:13 |
| *** trown is now known as trown|lunch | 17:20 | |
| Nisha_Agarwal | jroll, TheJulia we will setup one more stack setup for Pike release and try nova boot on it on Monday...if that also gives same error we will report the issue in nova. On this current setup this is consistently seen for ironic nodes | 17:30 |
| TheJulia | Nisha_Agarwal: Okay, thanks! | 17:31 |
| TheJulia | Nisha_Agarwal: have a good weekend! | 17:31 |
| Nisha_Agarwal | TheJulia, :) | 17:32 |
| openstackgerrit | Jim Rollenhagen proposed openstack/networking-generic-switch stable/queens: Pin pep8 https://review.openstack.org/561282 | 17:43 |
| jroll | I still have no idea, so I'm giving this a shot :| | 17:43 |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic master: Test n-g-s fix https://review.openstack.org/560886 | 17:49 |
| *** tesseract has quit IRC | 17:53 | |
| TheJulia | :( | 17:54 |
| *** mgoddard has joined #openstack-ironic | 17:55 | |
| *** Nisha_Agarwal has quit IRC | 17:57 | |
| *** Sukhdev has joined #openstack-ironic | 17:57 | |
| *** links has quit IRC | 18:02 | |
| *** e0ne has quit IRC | 18:12 | |
| *** cjloader_ has joined #openstack-ironic | 18:14 | |
| *** cjloader has quit IRC | 18:14 | |
| *** jaganathan has quit IRC | 18:15 | |
| *** links has joined #openstack-ironic | 18:15 | |
| *** trown|lunch is now known as trown | 18:16 | |
| *** cjloader_ has quit IRC | 18:16 | |
| *** e0ne has joined #openstack-ironic | 18:16 | |
| *** cjloader has joined #openstack-ironic | 18:17 | |
| *** cjloader_ has joined #openstack-ironic | 18:21 | |
| *** cjloader has quit IRC | 18:21 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Clean up driver_internal_info after tear_down https://review.openstack.org/561289 | 18:22 |
| jroll | yeah that didn't work | 18:22 |
| jroll | wtf | 18:22 |
| *** dougsz has quit IRC | 18:28 | |
| *** links has quit IRC | 18:31 | |
| TheJulia | hmm | 18:50 |
| *** harlowja has joined #openstack-ironic | 19:10 | |
| *** exodusftw has quit IRC | 19:12 | |
| *** exodusftw has joined #openstack-ironic | 19:17 | |
| *** mgoddard has quit IRC | 19:21 | |
| *** Sukhdev has quit IRC | 19:28 | |
| *** cjloader_ has quit IRC | 19:31 | |
| *** cjloader has joined #openstack-ironic | 19:34 | |
| *** cjloader has quit IRC | 19:36 | |
| *** cjloader has joined #openstack-ironic | 19:39 | |
| *** cjloader has quit IRC | 19:39 | |
| *** cjloader has joined #openstack-ironic | 19:39 | |
| *** e0ne has quit IRC | 19:42 | |
| *** parlos has quit IRC | 19:53 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Fix bifrost CI https://review.openstack.org/561232 | 20:08 |
| *** dprince has quit IRC | 20:19 | |
| *** MattMan has quit IRC | 20:27 | |
| *** MattMan has joined #openstack-ironic | 20:27 | |
| *** jcoufal_ has quit IRC | 20:43 | |
| TheJulia | jroll: semi-reproduced, erroring on no module named six.... I may have uninstalled TONS of stuff | 20:45 |
| jroll | heh | 20:47 |
| jroll | TheJulia: I had to do some downstream stuff, so didn't make progress, and heading out shortly | 20:47 |
| *** baha has quit IRC | 20:48 | |
| *** r-daneel has quit IRC | 20:49 | |
| TheJulia | bingo, got it | 20:49 |
| jroll | \o/ | 20:50 |
| jroll | what is/was it? | 20:50 |
| TheJulia | well, I don't have a fix yet | 20:50 |
| TheJulia | flake8 is installed and available, the code loads, but pep8 is not to be found | 20:50 |
| TheJulia | hence the error | 20:50 |
| jroll | gotcha | 20:50 |
| TheJulia | it is quite literally exactly what it is saying | 20:50 |
| *** fragatina has quit IRC | 20:50 | |
| jroll | yeah, in the gate, pep8 is pretty clearly installed though | 20:51 |
| TheJulia | as a user perhaps.... | 20:51 |
| jroll | 2018-04-13 18:17:10.369 | Requirement already satisfied: pep8==1.5.7 in /usr/local/lib/python2.7/dist-packages (from hacking>=1.0.0->-r /opt/stack/old/ironic/test-requirements.txt (line 4)) | 20:51 |
| jroll | ¯\_(ツ)_/¯ | 20:51 |
| TheJulia | frak | 20:51 |
| jroll | yeah, no clue | 20:52 |
| jroll | anyway I gotta run, sorry | 20:52 |
| jroll | have a good weekend | 20:52 |
| TheJulia | you too | 20:53 |
| *** trown is now known as trown|outtypewww | 20:58 | |
| *** d0ugal has quit IRC | 20:59 | |
| *** jjohnson2_ has quit IRC | 21:01 | |
| openstackgerrit | Merged openstack/bifrost master: Fix bifrost CI https://review.openstack.org/561232 | 21:15 |
| openstackgerrit | Julia Kreger proposed openstack/networking-generic-switch stable/queens: DNM Debugging test https://review.openstack.org/561358 | 21:16 |
| rloo | yay ^^, fixed one CI problem.. | 21:16 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Test n-g-s fix https://review.openstack.org/560886 | 21:17 |
| TheJulia | yup, I'm hoping we get more data with that job run | 21:19 |
| TheJulia | what is happening makes no sense at all | 21:19 |
| * rloo hopes TheJulia is on the right track... | 21:23 | |
| TheJulia | in the mean time, I think I'm going to go get a beer and try to not look at storyboard | 21:24 |
| *** d0ugal has joined #openstack-ironic | 21:26 | |
| * etingof +2's TheJulia's attitude from the land of beer | 21:29 | |
| TheJulia | :) | 21:30 |
| *** devananda has joined #openstack-ironic | 21:39 | |
| *** e0ne has joined #openstack-ironic | 21:44 | |
| *** fragatina has joined #openstack-ironic | 21:45 | |
| *** r-daneel has joined #openstack-ironic | 21:46 | |
| *** r-daneel has quit IRC | 22:05 | |
| *** r-daneel has joined #openstack-ironic | 22:09 | |
| *** linkmark has quit IRC | 22:12 | |
| TheJulia | Oo http://logs.openstack.org/86/560886/4/check/ironic-grenade-dsvm-multinode-multitenant/8f2f6d1/logs/grenade.sh.txt.gz#_2018-04-13_21_37_39_655 | 22:24 |
| TheJulia | pep8 is a myth | 22:24 |
| TheJulia | it makes no sense... | 22:26 |
| *** itlinux has quit IRC | 22:27 | |
| *** e0ne has quit IRC | 22:42 | |
| *** figleaf is now known as edleafe | 22:46 | |
| *** Sukhdev has joined #openstack-ironic | 22:52 | |
| openstackgerrit | Julia Kreger proposed openstack/networking-generic-switch stable/queens: Ensure pep8 is present before trying to proceed https://review.openstack.org/561358 | 23:01 |
| TheJulia | rloo: literal worst hack ever ^^^^ but it works locally just fine | 23:03 |
| TheJulia | and huge description | 23:03 |
| TheJulia | rechecked jim's test job | 23:04 |
| *** mjturek has quit IRC | 23:15 | |
| *** jgwentworth is now known as melwitt | 23:22 | |
| *** hoonetorg has joined #openstack-ironic | 23:45 | |
| *** Sukhdev has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!