| *** rpittau has quit IRC | 00:03 | |
| *** rpittau has joined #openstack-ironic | 00:03 | |
| *** iurygregory has quit IRC | 00:04 | |
| *** vabada has quit IRC | 00:10 | |
| *** strigazi has quit IRC | 00:10 | |
| *** vabada has joined #openstack-ironic | 00:10 | |
| *** patchbot has quit IRC | 00:10 | |
| *** patchbot has joined #openstack-ironic | 00:11 | |
| *** strigazi has joined #openstack-ironic | 00:11 | |
| *** devananda has quit IRC | 00:12 | |
| *** devananda has joined #openstack-ironic | 00:13 | |
| *** edleafe has quit IRC | 00:13 | |
| *** gyee has quit IRC | 00:15 | |
| *** rcernin has quit IRC | 00:29 | |
| *** hshiina has joined #openstack-ironic | 00:54 | |
| *** mjturek has joined #openstack-ironic | 00:57 | |
| *** mjturek has quit IRC | 00:58 | |
| *** lenka has quit IRC | 01:14 | |
| jiapei | Good morning Ironicers | 01:18 |
|---|---|---|
| *** tiendc has joined #openstack-ironic | 01:27 | |
| *** roger2 has quit IRC | 01:38 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-python-agent master: Enhanced checksum support https://review.openstack.org/609303 | 01:41 |
| *** MattMan_ has joined #openstack-ironic | 01:44 | |
| *** rloo has quit IRC | 01:46 | |
| *** hoangcx has quit IRC | 01:50 | |
| *** trungnv has quit IRC | 01:50 | |
| *** trungnv has joined #openstack-ironic | 01:51 | |
| *** hoangcx has joined #openstack-ironic | 01:51 | |
| *** jovial[m] has quit IRC | 01:56 | |
| *** rcernin has joined #openstack-ironic | 02:01 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Test swift python3 support https://review.openstack.org/609573 | 02:02 |
| *** rh-jelabarre has quit IRC | 02:43 | |
| *** gabys has joined #openstack-ironic | 03:54 | |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Rearrange conf and policy configuration file https://review.openstack.org/609584 | 04:11 |
| *** trungnv has quit IRC | 04:17 | |
| *** tiendc has quit IRC | 04:17 | |
| *** trungnv has joined #openstack-ironic | 04:18 | |
| *** tiendc has joined #openstack-ironic | 04:18 | |
| *** trungnv has quit IRC | 04:23 | |
| *** hoangcx has quit IRC | 04:23 | |
| *** trungnv has joined #openstack-ironic | 04:23 | |
| *** hoangcx has joined #openstack-ironic | 04:23 | |
| *** hshiina has quit IRC | 04:30 | |
| *** skazi has quit IRC | 04:55 | |
| *** hoonetorg has quit IRC | 05:09 | |
| *** olivierb has joined #openstack-ironic | 05:21 | |
| *** gabys has quit IRC | 05:36 | |
| *** gabys has joined #openstack-ironic | 05:36 | |
| *** gabys has quit IRC | 05:41 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 05:44 | |
| *** skazi has joined #openstack-ironic | 05:45 | |
| *** hshiina has joined #openstack-ironic | 05:51 | |
| *** pcaruana has joined #openstack-ironic | 06:01 | |
| *** e0ne has joined #openstack-ironic | 06:02 | |
| *** kushinauwu has joined #openstack-ironic | 06:06 | |
| *** olivierb has quit IRC | 06:11 | |
| *** olivierb has joined #openstack-ironic | 06:12 | |
| *** arne_wiebalck_ has quit IRC | 06:13 | |
| *** gabys has joined #openstack-ironic | 06:19 | |
| *** gabys has quit IRC | 06:23 | |
| *** dims has quit IRC | 06:29 | |
| *** e0ne has quit IRC | 06:29 | |
| *** cxc99 has quit IRC | 06:32 | |
| *** dims has joined #openstack-ironic | 06:33 | |
| *** jrist has joined #openstack-ironic | 06:33 | |
| *** cxc99 has joined #openstack-ironic | 06:35 | |
| *** dims has quit IRC | 06:38 | |
| *** dims has joined #openstack-ironic | 06:39 | |
| *** hoangcx has quit IRC | 06:43 | |
| *** tiendc has quit IRC | 06:43 | |
| *** trungnv has quit IRC | 06:43 | |
| *** gabys has joined #openstack-ironic | 06:43 | |
| *** trungnv has joined #openstack-ironic | 06:43 | |
| *** hoangcx has joined #openstack-ironic | 06:43 | |
| *** S4ren has joined #openstack-ironic | 06:44 | |
| *** iurygregory has joined #openstack-ironic | 06:46 | |
| *** kushinauwu has quit IRC | 06:47 | |
| *** gabys has quit IRC | 06:48 | |
| *** hoangcx has quit IRC | 06:49 | |
| *** kushinauwu has joined #openstack-ironic | 06:50 | |
| *** hoangcx has joined #openstack-ironic | 06:50 | |
| *** radez has quit IRC | 06:52 | |
| *** hoangcx has quit IRC | 06:53 | |
| *** trungnv has quit IRC | 06:53 | |
| *** hoangcx has joined #openstack-ironic | 06:54 | |
| *** gabys has joined #openstack-ironic | 06:59 | |
| *** rcernin has quit IRC | 07:01 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 07:04 | |
| iurygregory | good morning | 07:08 |
| *** jrist has quit IRC | 07:09 | |
| *** S4ren has quit IRC | 07:14 | |
| *** arne_wiebalck_ has quit IRC | 07:41 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 07:44 | |
| *** arne_wiebalck_ has quit IRC | 07:45 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 07:46 | |
| *** arne_wiebalck_ has quit IRC | 07:46 | |
| *** gabys has quit IRC | 07:48 | |
| *** gabys has joined #openstack-ironic | 07:48 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 07:52 | |
| *** gabys has quit IRC | 07:52 | |
| *** S4ren has joined #openstack-ironic | 08:02 | |
| *** arne_wiebalck_ has quit IRC | 08:07 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 08:10 | |
| *** arne_wiebalck_ has quit IRC | 08:11 | |
| *** lenka has joined #openstack-ironic | 08:12 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 08:12 | |
| *** S4ren has quit IRC | 08:19 | |
| *** kushinauwu has quit IRC | 08:30 | |
| *** kushinauwu has joined #openstack-ironic | 08:30 | |
| *** moshele has joined #openstack-ironic | 08:34 | |
| *** skazi has quit IRC | 08:35 | |
| *** arne_wiebalck_ has quit IRC | 08:36 | |
| mgoddard | Hi yolanda, would you be able to look at a DIB review when you have time? https://review.openstack.org/#/c/608187/ | 08:39 |
| patchbot | patch 608187 - diskimage-builder - Fail build due to missing kauditd only when SELinu... - 1 patch set | 08:39 |
| *** derekh has joined #openstack-ironic | 08:42 | |
| *** dtantsur|afk is now known as dtantsur| | 08:44 | |
| *** dtantsur| is now known as dtantsur | 08:44 | |
| dtantsur | morning ironic | 08:45 |
| iurygregory | o/ | 08:45 |
| *** arne_wiebalck_ has joined #openstack-ironic | 08:45 | |
| etingof | good morning iurygregory, dtantsur and mgoddard o/ | 08:46 |
| yolanda | hi mgoddard , let me take a look | 08:46 |
| yolanda | done | 08:47 |
| *** arne_wiebalck_ has quit IRC | 08:49 | |
| mgoddard | thanks yolanda :) | 08:56 |
| mgoddard | morning etingof et al. | 08:57 |
| *** arne_wiebalck_ has joined #openstack-ironic | 08:59 | |
| olivierb | morning everyone, today I have a new deployment issue due to 4d0c892e1054453774122a5aeefe0d3c26a35e91 because on ubuntu 16.04 system pip version is 8.1.1 and does not understand --upgrade-strategy only-if-needed option :-( | 09:00 |
| olivierb | failing task is TASK [bifrost-keystone-install : Install from /opt/stack/keystone using pip] | 09:00 |
| olivierb | ^ hwoarang | 09:01 |
| *** hshiina has quit IRC | 09:01 | |
| olivierb | but what I can not explain is that it seemed to work yesterday | 09:02 |
| olivierb | note that I am in keystone + venv case | 09:04 |
| *** e0ne has joined #openstack-ironic | 09:05 | |
| *** stendulker has joined #openstack-ironic | 09:10 | |
| *** jistr is now known as jistr|call | 09:12 | |
| *** gkadam has quit IRC | 09:25 | |
| *** e0ne has quit IRC | 09:27 | |
| *** arne_wiebalck_ has quit IRC | 09:28 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 09:31 | |
| *** arne_wiebalck_ has quit IRC | 09:32 | |
| *** skazi has joined #openstack-ironic | 09:33 | |
| *** dougsz has joined #openstack-ironic | 09:35 | |
| jiapei | olivierb: I'm not quite understand, do you mean you failed to deploy devstack today? but successed yesterday? | 09:39 |
| olivierb | not devstack but bifrost | 09:39 |
| olivierb | jiapei | 09:39 |
| *** e0ne has joined #openstack-ironic | 09:43 | |
| *** jtomasek has joined #openstack-ironic | 09:46 | |
| *** hjensas has joined #openstack-ironic | 09:47 | |
| olivierb | currently testing patch for this issue | 09:53 |
| olivierb | will submit very soon | 09:53 |
| *** arne_wiebalck_ has joined #openstack-ironic | 09:57 | |
| *** arne_wiebalck_ has quit IRC | 10:00 | |
| *** stendulker has quit IRC | 10:03 | |
| *** dnuka has joined #openstack-ironic | 10:08 | |
| *** dmellado has quit IRC | 10:17 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Fix ironic configuration for new oslo messaging https://review.openstack.org/609499 | 10:20 |
| *** kushinauwu has quit IRC | 10:22 | |
| *** kushinauwu has joined #openstack-ironic | 10:22 | |
| *** v12aml has quit IRC | 10:25 | |
| *** v12aml has joined #openstack-ironic | 10:26 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Fix for proper installation https://review.openstack.org/609660 | 10:27 |
| olivierb | here it is ^^ | 10:27 |
| * dtantsur waits for the CI | 10:28 | |
| *** v12aml has quit IRC | 10:34 | |
| *** e0ne has quit IRC | 10:35 | |
| *** v12aml has joined #openstack-ironic | 10:37 | |
| *** e0ne has joined #openstack-ironic | 10:38 | |
| dtantsur | mgoddard: have a second to check https://review.openstack.org/605673 ? I'd like to release ironic-lib with it soon(ish) | 10:41 |
| patchbot | patch 605673 - ironic-lib - Allow to return unpopulated root partition from wo... - 2 patch sets | 10:41 |
| mgoddard | dtantsur: sure | 10:51 |
| *** kushinauwu has quit IRC | 10:55 | |
| *** kushinauwu has joined #openstack-ironic | 10:57 | |
| *** jistr|call is now known as jistr | 11:00 | |
| jroll | morning | 11:03 |
| *** e0ne has quit IRC | 11:04 | |
| jroll | I'm surprised we don't document that system pip should be upgraded (as that's how we test) | 11:04 |
| openstackgerrit | Dhanuka proposed openstack/sushy master: Add support for the `CompositionService` resource https://review.openstack.org/608563 | 11:04 |
| dtantsur | morning jroll. upgrading system pip is not a good request, to be honest.. | 11:05 |
| dtantsur | I mean, except for throw-away VMs where you can actually do 'sudo pip install -U pip' | 11:05 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Allow streaming raw partition images https://review.openstack.org/605707 | 11:06 |
| *** belmorei_ has joined #openstack-ironic | 11:14 | |
| jroll | dtantsur: well, running python applications on system python is almost as crazy as upgrading system pip :) | 11:14 |
| jroll | especially if you can't throw the machine away | 11:14 |
| dtantsur | jroll: well, some people have been doing it in production for a lot of time.. | 11:15 |
| dtantsur | I guess I cannot see your argument here | 11:16 |
| dtantsur | what I'm saying is that updating system python packages via pip will probably screw up the package manager | 11:16 |
| jroll | agree | 11:16 |
| jroll | which is why one shouldn't install python applications on system python | 11:17 |
| *** belmoreira has quit IRC | 11:17 | |
| jroll | (I mean outside of the package manager, of course) | 11:17 |
| dtantsur | ah, right, the last bit is what I did not get | 11:17 |
| jroll | ah, sorry | 11:18 |
| dtantsur | :) | 11:18 |
| jroll | since that's what bifrost does, running it without a virtualenv is crazypants to me | 11:18 |
| dtantsur | ditto | 11:18 |
| dtantsur | or everyone's beloved containers | 11:19 |
| * dtantsur wonders why bifrost does not use virtualenv | 11:19 | |
| sambetts|afk | it has an option too, I thought with the pip version that started complaining about upgrading system packages that it was going to be the default | 11:20 |
| sambetts|afk | but maybe not | 11:20 |
| jroll | right, it's optional, dont' remember if it's default or not | 11:20 |
| *** dtantsur is now known as dtantsur|brb | 11:21 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:21 | |
| *** belmorei_ has quit IRC | 11:36 | |
| jroll | sambetts|afk: NOT ALLOWED | 11:44 |
| * sambetts|afk hides | 11:45 | |
| olivierb | dtantsur|brb jroll please remember that I am using venv (+keystone) by default but even though there are some case where system pip is used if I remember correctly | 11:45 |
| * jroll notes sam gets extra beer in berlin | 11:45 | |
| olivierb | let me get this back | 11:45 |
| * sambetts|afk unhides for beer | 11:46 | |
| jroll | aha, I found the secret | 11:46 |
| olivierb | https://github.com/openstack/bifrost/blob/1e19c45199e4fa2bf89cc6487f820b6915b926e1/playbooks/roles/bifrost-ironic-install/tasks/install.yml#L137:L148 | 11:47 |
| olivierb | but this is unrelated to the issue I am having | 11:47 |
| jroll | ugh | 11:48 |
| sambetts|afk | ah it because the ansible module uses the system python | 11:49 |
| sambetts|afk | it is possible to manually set the ansible_python_interpreter to point at the virtualenv pathg | 11:50 |
| *** e0ne has joined #openstack-ironic | 11:51 | |
| sambetts|afk | e.g. ansible_python_interpreter: "/opt/stack/bifrost/bin/python" | 11:51 |
| sambetts|afk | which will ensure that ansible uses the virtualenv even for the "remote" host (which is actually localhost but it doesn't know the difference) | 11:53 |
| *** priteau has joined #openstack-ironic | 11:55 | |
| *** derekh has quit IRC | 12:00 | |
| *** derekh has joined #openstack-ironic | 12:00 | |
| *** moshele has quit IRC | 12:04 | |
| *** dmellado has joined #openstack-ironic | 12:10 | |
| *** trown|outtypewww is now known as trown | 12:10 | |
| *** roger2 has joined #openstack-ironic | 12:15 | |
| *** rh-jelabarre has quit IRC | 12:23 | |
| *** kushinauwu has quit IRC | 12:30 | |
| olivierb | seems like https://review.openstack.org/609499 passed CI !!!! | 12:49 |
| patchbot | patch 609499 - bifrost - Fix ironic configuration for new oslo messaging - 4 patch sets | 12:49 |
| TheJulia | Good morning | 12:49 |
| jroll | olivierb: nice :D | 12:50 |
| * jroll +A | 12:51 | |
| olivierb | good morning TheJulia | 12:51 |
| jroll | hey TheJulia :) | 12:51 |
| olivierb | thanks for the W+1 jroll | 12:51 |
| TheJulia | wow, it passed | 12:51 |
| jroll | np, thanks for the patch | 12:51 |
| olivierb | yavw | 12:52 |
| TheJulia | weird.... I wonder if omit was what was throwing the conductor off | 12:52 |
| sambetts|afk | olivierb: did you see my message about setting the python interpreter? I think with that you might be able to get rid of that section that installs shade outside of the venv | 12:52 |
| TheJulia | olivierb: thanks! | 12:52 |
| *** rloo has joined #openstack-ironic | 12:53 | |
| TheJulia | sambetts|afk: it might work if explicitly pointed with ansible_python_interpreter set, however I don't remeber if that is ignored or not... or really what the exact behavior is post ssh | 12:53 |
| TheJulia | s/post/with/ | 12:53 |
| olivierb | TheJulia from all my tests earlier today the omit part was generating some stuff like transport_url = rabbit://ironic:aSecretPassword473z@127.0.0.1:5672/__omit____.* | 12:53 |
| TheJulia | oh jeeze | 12:54 |
| TheJulia | :) | 12:54 |
| TheJulia | err :( | 12:54 |
| olivierb | sambetts|afk thanks for the info however my issue is not there and I do not want to touch this part | 12:54 |
| olivierb | my issue is fixed with https://review.openstack.org/609660 | 12:54 |
| patchbot | patch 609660 - bifrost - Fix for proper installation - 1 patch set | 12:54 |
| olivierb | I still do not understand why I got this issue only today as 4d0c892e1054453774122a5aeefe0d3c26a35e91 was already intergrated :-( | 12:55 |
| olivierb | I did a lot of bifrost deploy yesterday without hitting the issue | 12:55 |
| olivierb | and based on latest version of master of course | 12:56 |
| sambetts|afk | TheJulia: from my tests locally with ssh ansible still uses the ansible_python_interpreter to process all the modules on the remote side, its requires to ensure you can switch between python2 and python3 hosts for example or virtualenvs | 12:56 |
| * TheJulia feels like is time to flip the venv switch on bifrost | 12:56 | |
| *** edleafe has joined #openstack-ironic | 12:57 | |
| TheJulia | sambetts|afk: it has been years since I looked at that :( | 12:57 |
| olivierb | TheJulia yes definitely and it will also help for https://review.openstack.org/#/c/588325/ ;-) | 12:58 |
| patchbot | patch 588325 - bifrost - Begin change over to use python3 - 20 patch sets | 12:58 |
| TheJulia | oh yeah | 12:58 |
| TheJulia | sadly half the distributions don't have a compatible python :\ | 12:59 |
| *** sai_p has joined #openstack-ironic | 13:04 | |
| olivierb | TheJulia I think https://review.openstack.org/609536 is not needed anymore, I added some comments in it but I am still puzzled on why it failed yesterday (very bad day) | 13:06 |
| patchbot | patch 609536 - bifrost - Remove pre-emptive boot.ipxe copy - 1 patch set | 13:06 |
| TheJulia | olivierb: it was a early thing to route around pxe only writing boot.ipxe when a node was deployed, but ironic changed the code a couple years ago to do so upon __init__ | 13:07 |
| TheJulia | I do feel like collecting the additional information following that... | 13:08 |
| TheJulia | olivierb: I think it was just truly bizzar that the conductor would kind of start but not actually run | 13:08 |
| *** e0ne has quit IRC | 13:11 | |
| *** dims has quit IRC | 13:12 | |
| *** dims has joined #openstack-ironic | 13:14 | |
| openstackgerrit | Merged openstack/bifrost master: Fix ironic configuration for new oslo messaging https://review.openstack.org/609499 | 13:15 |
| *** dtantsur|brb is now known as dtantsur | 13:15 | |
| dtantsur | sambetts|afk beer++ | 13:15 |
| *** arne_wiebalck_ has joined #openstack-ironic | 13:18 | |
| *** lenka has quit IRC | 13:18 | |
| *** dims has quit IRC | 13:19 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Fix for proper installation https://review.openstack.org/609660 | 13:20 |
| TheJulia | ipa CI seems so much better now :) | 13:20 |
| *** e0ne has joined #openstack-ironic | 13:21 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add voting and gating test with Keystone enabled https://review.openstack.org/599544 | 13:22 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: [DNM] Testing CI paramaters https://review.openstack.org/607480 | 13:22 |
| dtantsur | the inspector grenade is still quite unstable | 13:22 |
| TheJulia | Did the grenade fix land? | 13:23 |
| dtantsur | TheJulia: yep, so it's now not completely borken | 13:24 |
| dtantsur | but seems to fail like 60% of cases | 13:24 |
| dtantsur | last time I saw it was some kind of timeout, but I did not investigate the latest failures | 13:25 |
| *** mjturek has joined #openstack-ironic | 13:25 | |
| TheJulia | :( | 13:26 |
| *** baha has joined #openstack-ironic | 13:27 | |
| *** baha has quit IRC | 13:27 | |
| *** baha has joined #openstack-ironic | 13:27 | |
| *** lenka has joined #openstack-ironic | 13:32 | |
| *** rloo has quit IRC | 13:32 | |
| *** rloo has joined #openstack-ironic | 13:33 | |
| *** jrist has joined #openstack-ironic | 13:33 | |
| *** arne_wiebalck_ has quit IRC | 13:37 | |
| *** dnuka has quit IRC | 13:37 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 13:38 | |
| TheJulia | dtantsur: hmm, yeah, inspection timed out | 13:38 |
| TheJulia | no baremetal console logs | 13:38 |
| *** serlex has joined #openstack-ironic | 13:39 | |
| *** serlex has quit IRC | 13:39 | |
| dtantsur | the same ipxe problem as with ironic grenade? | 13:39 |
| dtantsur | although ironic setting must apply to inspector as well... | 13:40 |
| *** mjturek has quit IRC | 13:42 | |
| TheJulia | dtantsur: it dhcp's as far as I can tell from logs | 13:44 |
| TheJulia | http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector-dhcp.txt.gz#_Oct_11_06_16_22_833681 <-- about the right time, right mac address for one of the interfaces | 13:45 |
| *** mjturek has joined #openstack-ironic | 13:45 | |
| *** ijw has joined #openstack-ironic | 13:45 | |
| *** ijw has quit IRC | 13:46 | |
| *** ijw has joined #openstack-ironic | 13:46 | |
| *** arne_wiebalck_ has quit IRC | 13:47 | |
| TheJulia | bingo | 13:49 |
| TheJulia | dtantsur: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/old/ironic-bm-logs/node-4_console.txt.gz#_2018-10-11_06_19_09_971 | 13:49 |
| TheJulia | e1000 inside the VM | 13:53 |
| dtantsur | wut.. | 14:00 |
| dtantsur | yeah, now I remember seeing this. still no clue why. | 14:00 |
| *** mjturek has quit IRC | 14:00 | |
| TheJulia | http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/syslog.txt.gz#_Oct_11_06_15_43 | 14:01 |
| TheJulia | we're preemptively blocking dhcp access | 14:01 |
| dtantsur | hmmm | 14:02 |
| *** rh-jelabarre has joined #openstack-ironic | 14:06 | |
| *** mjturek has joined #openstack-ironic | 14:07 | |
| *** jrist has quit IRC | 14:20 | |
| TheJulia | weird | 14:22 |
| *** dims has joined #openstack-ironic | 14:22 | |
| TheJulia | 6:16:12 is when the vm was powered up | 14:23 |
| dtantsur | so, it closed the pxe filter while it was still on? | 14:26 |
| dtantsur | TheJulia, hjensas, I see suspicious things in inspector logs: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector.txt.gz#_Oct_11_06_16_01_444146 | 14:30 |
| TheJulia | http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/syslog.txt.gz#_Oct_11_06_16_14 <-- looks like we nuked access right after the VM powered up | 14:30 |
| TheJulia | which would explain why it never got a dhcp address | 14:30 |
| * TheJulia doesn't understand why we're nuking and rebuilding it | 14:31 | |
| *** S4ren has joined #openstack-ironic | 14:32 | |
| *** e0ne has quit IRC | 14:33 | |
| dtantsur | although no, these messages are fine | 14:34 |
| dtantsur | TheJulia: ah, I see. we're building a temporary chain, then rebuilding the main one | 14:35 |
| TheJulia | yeah | 14:36 |
| dtantsur | it should not cause issues, since the temporary chain must serve requests | 14:36 |
| *** jtomasek has quit IRC | 14:36 | |
| dtantsur | or rather no: we build an updated temporary chain, then we swap it with the old main one | 14:37 |
| dtantsur | TheJulia: https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/pxe_filter/iptables.py#L133-L146 | 14:37 |
| sambetts|afk | yeah we do that to ensure we don't have a partial chain at any point | 14:39 |
| dtantsur | what I don't get is why I don't see any white/black-listing in the syslog | 14:39 |
| TheJulia | well, some of it is there, but yeah | 14:40 |
| TheJulia | things are not adding up | 14:40 |
| *** sthussey has joined #openstack-ironic | 14:41 | |
| *** e0ne has joined #openstack-ironic | 14:42 | |
| *** dims has quit IRC | 14:43 | |
| TheJulia | end state is of course, an empty chain | 14:43 |
| TheJulia | well, just reject | 14:44 |
| dtantsur | end state yes.. I do see the correct logic being applied, I don't get why it ends up in no DHCP... | 14:44 |
| dtantsur | TheJulia: this looks weird: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector-dhcp.txt.gz#_Oct_11_06_16_11_935350 | 14:46 |
| TheJulia | that mac seems to re-dhcp without issues later on, where as the mac for node-4 doesn't seem to :( | 14:47 |
| dtantsur | okay, I'm confused. this looks like normal DHCP handshake to me: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector-dhcp.txt.gz#_Oct_11_06_17_20_023552 | 14:47 |
| *** dims has joined #openstack-ironic | 14:48 | |
| dtantsur | TheJulia: it looks like tinyipa did get DHCP around its start up time ^^^ | 14:48 |
| TheJulia | just initial ipxe | 14:49 |
| TheJulia | as far as I can tell | 14:49 |
| TheJulia | the ramdisk didn't actually begin to be executed until nearly 06:18 | 14:49 |
| olivierb | brand new bifrost deployment on baremetal node is stuck in TASK [bifrost-ironic-install : Ensure service project is present] | 14:49 |
| olivierb | no errors in any logs :-( | 14:49 |
| dtantsur | TheJulia: I see 6:17 http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/old/ironic-bm-logs/node-4_console.txt.gz#_2018-10-11_06_17_45_411 | 14:49 |
| dtantsur | unless I'm confusing something, it got DHCP once at 6:15, then at 6:17:20, then at 6:17:45 IPA started logging | 14:50 |
| dtantsur | this looks kinda normal to me.. | 14:50 |
| TheJulia | http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/old/ironic-bm-logs/node-4_console.txt.gz#_2018-10-11_06_17_49_803 | 14:51 |
| TheJulia | http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/old/ironic-bm-logs/node-4_console.txt.gz#_2018-10-11_06_18_50_512 | 14:52 |
| dtantsur | right, but it ACKed new address 30 seconds earlier: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector-dhcp.txt.gz#_Oct_11_06_17_20_023552 | 14:52 |
| * dtantsur wonders what "box" means | 14:52 | |
| sambetts|afk | dtantsur: I think thats the hostname | 14:53 |
| sambetts|afk | that tinycore uses | 14:53 |
| TheJulia | uhhh | 14:54 |
| TheJulia | I've been following mac 52:54:00:4a:fa:15 | 14:55 |
| TheJulia | that mac is 52:54:00:83:3c:15 | 14:55 |
| TheJulia | which is not on node-4 | 14:55 |
| dtantsur | omg | 14:56 |
| dtantsur | I'll run to the woods to live with owls | 14:57 |
| TheJulia | ? | 14:57 |
| dtantsur | owls are nice and soft | 14:57 |
| TheJulia | NOOOO! | 14:57 |
| dtantsur | owls do not DHCP | 14:57 |
| TheJulia | :( | 14:57 |
| dtantsur | okay, the ACK for the right MAC this time: http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/screen-ironic-inspector-dhcp.txt.gz#_Oct_11_05_17_13_176980 | 14:57 |
| olivierb | any hints on where I should be looking ? | 14:58 |
| dtantsur | wait, it's old side of grenade | 14:58 |
| *** openstackgerrit has quit IRC | 14:58 | |
| dtantsur | TheJulia: you're right, the right MAC last DHCP at 6:16 | 14:58 |
| TheJulia | olivierb: I guess keystone logging? :\ | 14:59 |
| dtantsur | olivierb: sorry, cannot fit two problems in my head at the same time :( | 14:59 |
| olivierb | dtantsur I understand no pb | 14:59 |
| olivierb | TheJulia nothing worth mentioning in keystone logs unfortunately | 14:59 |
| olivierb | last line is 127.0.0.1 - - [11/Oct/2018:16:48:26 +0200] "POST /v3/auth/tokens HTTP/1.1" 201 1221 "-" "openstacksdk/0.17.2 keystoneauth1/3.11.0 python-requests/2.19.1 CPython/2.7.12" | 15:00 |
| *** openstackgerrit has joined #openstack-ironic | 15:00 | |
| olivierb | I'll dig further on my side | 15:00 |
| TheJulia | olivierb: so it posts to the token endpoint and doesn't fulfill the next step? | 15:02 |
| olivierb | seems to be stuck in there | 15:03 |
| olivierb | TheJulia ^^ | 15:03 |
| olivierb | currently adding traces | 15:03 |
| TheJulia | olivierb: that is as bizzar as the issue we're hunting | 15:04 |
| olivierb | yes and all others since yesterday which also sometimes vanishes like the boot.ipxe file :-( | 15:04 |
| *** gyee has joined #openstack-ironic | 15:08 | |
| *** e0ne has quit IRC | 15:09 | |
| hjensas | dtantsur: going to have a look as well. Issue is one node not getting DHCP? | 15:10 |
| hjensas | dtantsur: is the grenade job running these tests on the "old" version first? Then upgrade, and run the same tests again? | 15:10 |
| olivierb | found the issue :-( | 15:11 |
| olivierb | using IP of NATed itf as public endpoint | 15:12 |
| dtantsur | hjensas: 1. rather loses DHCP after booting the OS, 2. yes | 15:12 |
| olivierb | sorry for the noise | 15:12 |
| hjensas | dtantsur: is it failing when testing the before or after the upgrade? | 15:13 |
| *** sai_p has quit IRC | 15:13 | |
| dtantsur | hjensas: after | 15:17 |
| hjensas | dtantsur: ok, thanks for context. I will look at logs as well. | 15:18 |
| *** e0ne has joined #openstack-ironic | 15:20 | |
| TheJulia | hjensas: thanks | 15:20 |
| *** e0ne has quit IRC | 15:20 | |
| *** openstackgerrit has quit IRC | 15:22 | |
| *** Chaserjim has joined #openstack-ironic | 15:32 | |
| *** ijw has quit IRC | 15:44 | |
| *** ijw has joined #openstack-ironic | 15:47 | |
| hjensas | dtantsur: afaict the last dhcp request/offer is successfull @06:16:22 - Then when ipa boots - 06:17:49.707 465 DEBUG ironic_python_agent.netutils [-] No carrier information for interface eth0 - The cable is'nt plugged. | 15:49 |
| hjensas | dtantsur: is ipa even trying to bring up the interface? | 15:49 |
| dtantsur | hjensas: IPA never does IIRCX | 15:50 |
| *** jrist has joined #openstack-ironic | 15:50 | |
| *** mjturek has quit IRC | 15:51 | |
| *** ltomasbo has joined #openstack-ironic | 15:53 | |
| *** rh-jelabarre has quit IRC | 15:55 | |
| *** jrist has quit IRC | 15:59 | |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| dtantsur|afk | see you | 16:00 |
| *** rpioso has joined #openstack-ironic | 16:09 | |
| *** openstackgerrit has joined #openstack-ironic | 16:10 | |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: [DNM] Testing CI paramaters https://review.openstack.org/607480 | 16:10 |
| rpioso | Good morning | 16:10 |
| TheJulia | good morning rpioso | 16:11 |
| TheJulia | hjensas: we don't explicitly try and bring up the interface local carrier wise, I think we expect dhcp to have done that since it is part of the process of obtaining an IP address. | 16:12 |
| hjensas | TheJulia: I just don't see anything in logs about dhcp doing that. But I don't see that in non grenade job either. | 16:13 |
| hjensas | TheJulia: https://review.openstack.org/604506 ? | 16:13 |
| patchbot | patch 604506 - ironic - Switch grenade nic driver to e1000 (MERGED) - 2 patch sets | 16:13 |
| openstackgerrit | Olivier Bourdon proposed openstack/bifrost master: Add voting and gating test with Keystone enabled https://review.openstack.org/599544 | 16:14 |
| hjensas | TheJulia: nah, I see jobs passing after that. | 16:18 |
| TheJulia | hjensas: my expression of behavior is absolutely necessary to obtain an address | 16:22 |
| *** ijw has quit IRC | 16:22 | |
| TheJulia | if link carrier is not up, you can't transmit a dhcp packet down the wire. | 16:23 |
| *** ijw has joined #openstack-ironic | 16:23 | |
| *** gabys has joined #openstack-ironic | 16:23 | |
| hjensas | TheJulia: it looks like e1000 module is loaded after echo Starting bootlocal script: | 16:23 |
| TheJulia | so I did notice that the neutron config was running in multitenant mode.... | 16:23 |
| TheJulia | is bootlocal where dhcp is started? | 16:24 |
| *** baha has quit IRC | 16:24 | |
| hjensas | TheJulia: I'm checking ... | 16:24 |
| hjensas | TheJulia: https://github.com/openstack/ironic-python-agent/blob/master/imagebuild/tinyipa/build_files/bootlocal.sh#L67 | 16:32 |
| *** trown is now known as trown|lunch | 16:34 | |
| *** dmellado has quit IRC | 16:34 | |
| *** skazi has quit IRC | 16:36 | |
| TheJulia | hjensas: that is for rescue since it has to re-dhcp. | 16:40 |
| *** mjturek has joined #openstack-ironic | 16:41 | |
| hjensas | TheJulia: ah, I see. Still in passing jobs I see the e1000 module loaded earlier in the log. | 16:45 |
| *** baha has joined #openstack-ironic | 16:53 | |
| hjensas | TheJulia: I just love how everything that would give us a clue is sent to /dev/null or commented - http://paste.openstack.org/show/731912/ | 16:54 |
| *** dougsz has quit IRC | 16:59 | |
| *** derekh has quit IRC | 16:59 | |
| hjensas | TheJulia: udevadm settle's default is 120 seconds. Maby 5 seconds is a bit aggressive? | 17:00 |
| *** S4ren has quit IRC | 17:00 | |
| *** mjturek has quit IRC | 17:01 | |
| TheJulia | it is just for disk devices really, so five is likely okay in 99% of cases. | 17:01 |
| TheJulia | Besides, adding 2 minutes to a deploy means we'll push deployments just that much longer | 17:02 |
| *** gabys has quit IRC | 17:05 | |
| *** gabys has joined #openstack-ironic | 17:05 | |
| rpioso | TheJulia: :) | 17:07 |
| *** gabys has quit IRC | 17:10 | |
| *** mbuil has quit IRC | 17:13 | |
| *** olivierb has quit IRC | 17:15 | |
| *** gabys has joined #openstack-ironic | 17:16 | |
| *** ijw has quit IRC | 17:18 | |
| *** ijw_ has joined #openstack-ironic | 17:18 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-python-agent master: Raise udevadm settle timeout frin 5 to 30 seconds https://review.openstack.org/609769 | 17:18 |
| openstackgerrit | Harald Jensås proposed openstack/ironic-python-agent master: Raise udevadm settle timeout frin 5 to 30 seconds https://review.openstack.org/609769 | 17:20 |
| *** rh-jelabarre has joined #openstack-ironic | 17:20 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector master: DNM - Testing Grenade https://review.openstack.org/609771 | 17:22 |
| openstackgerrit | Merged openstack/ironic-lib master: Allow to return unpopulated root partition from work_on_disk https://review.openstack.org/605673 | 17:31 |
| *** trown|lunch is now known as trown | 17:34 | |
| TheJulia | hjensas: you likely want to double check if the ipa image is built or downloaded on that test | 17:40 |
| *** ianychoi has quit IRC | 17:41 | |
| hjensas | TheJulia: :( it's downloaded afaict http://logs.openstack.org/84/609584/1/check/ironic-inspector-grenade-dsvm/1bb1164/logs/grenade.sh.txt.gz#_2018-10-11_04_42_02_471 | 17:45 |
| TheJulia | I think there is a flag for that someplace | 17:46 |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector master: DNM - Testing Grenade https://review.openstack.org/609771 | 17:49 |
| hjensas | TheJulia: ^^ I hope that does it. | 17:49 |
| *** skazi has joined #openstack-ironic | 17:53 | |
| *** lenka has quit IRC | 18:01 | |
| *** openstackgerrit has quit IRC | 18:20 | |
| *** dmellado has joined #openstack-ironic | 18:27 | |
| *** openstackstatus has quit IRC | 18:28 | |
| *** openstackstatus has joined #openstack-ironic | 18:29 | |
| *** ChanServ sets mode: +v openstackstatus | 18:29 | |
| *** Chaserjim has quit IRC | 18:30 | |
| * TheJulia hopes so too | 18:36 | |
| *** openstackgerrit has joined #openstack-ironic | 18:43 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-python-agent master: DNM - Debug ironic-inspector grenade job https://review.openstack.org/609782 | 18:43 |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector master: DNB - Debug grenade job https://review.openstack.org/609783 | 18:43 |
| *** rh-jelabarre has quit IRC | 19:02 | |
| *** S4ren has joined #openstack-ironic | 19:43 | |
| *** roger2 has quit IRC | 19:47 | |
| *** baha has quit IRC | 20:04 | |
| *** baha has joined #openstack-ironic | 20:18 | |
| TheJulia | so thinking outloud, does it make sense to support an HTTPClient version of ipxe since it so useful? | 20:25 |
| *** S4ren has quit IRC | 20:27 | |
| *** ijw_ has quit IRC | 20:28 | |
| *** pcaruana has quit IRC | 20:40 | |
| *** ijw has joined #openstack-ironic | 20:54 | |
| *** baha has quit IRC | 21:19 | |
| *** priteau has quit IRC | 21:22 | |
| *** ijw has quit IRC | 21:57 | |
| *** openstackgerrit has quit IRC | 22:19 | |
| *** openstackgerrit has joined #openstack-ironic | 22:21 | |
| *** rcernin has joined #openstack-ironic | 22:23 | |
| *** ijw has joined #openstack-ironic | 22:24 | |
| *** rpioso is now known as rpioso|afk | 22:26 | |
| *** openstackstatus has quit IRC | 22:28 | |
| *** openstackstatus has joined #openstack-ironic | 22:29 | |
| *** ChanServ sets mode: +v openstackstatus | 22:29 | |
| *** gabys has quit IRC | 22:50 | |
| *** gabys has joined #openstack-ironic | 22:50 | |
| *** gabys has quit IRC | 22:55 | |
| *** spotz has quit IRC | 23:17 | |
| *** mcarden has quit IRC | 23:18 | |
| *** rloo has quit IRC | 23:19 | |
| *** sthussey has quit IRC | 23:21 | |
| *** spotz has joined #openstack-ironic | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!