| *** toci-bot has joined #tripleo | 00:06 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_7js8C2k/ | 00:06 |
|---|---|---|
| *** toci-bot has quit IRC | 00:06 | |
| *** epim has joined #tripleo | 00:07 | |
| *** CaptTofu has joined #tripleo | 00:11 | |
| *** vipul-away is now known as vipul | 00:16 | |
| lifeless | SpamapS: so are you poking into the undercloud ? | 00:17 |
| lifeless | or, nuts, your EOD already | 00:17 |
| *** matsuhashi has joined #tripleo | 00:29 | |
| *** coolsvap has quit IRC | 00:30 | |
| *** tzumainn has quit IRC | 00:31 | |
| *** tzumainn has joined #tripleo | 00:31 | |
| *** CaptTofu has quit IRC | 00:42 | |
| *** CaptTofu has joined #tripleo | 00:44 | |
| *** matsuhashi has quit IRC | 00:48 | |
| *** matsuhashi has joined #tripleo | 00:49 | |
| *** matsuhashi has quit IRC | 00:53 | |
| *** jcooley_ has quit IRC | 00:55 | |
| *** jcooley_ has joined #tripleo | 00:55 | |
| *** jcooley_ has quit IRC | 01:00 | |
| *** funzo has quit IRC | 01:06 | |
| *** matsuhashi has joined #tripleo | 01:08 | |
| *** nosnos has joined #tripleo | 01:09 | |
| *** funzo has joined #tripleo | 01:11 | |
| *** funzo has quit IRC | 01:14 | |
| *** funzo has joined #tripleo | 01:15 | |
| *** funzo has quit IRC | 01:20 | |
| *** shakayumi has joined #tripleo | 01:20 | |
| *** matsuhashi has quit IRC | 01:34 | |
| *** matsuhashi has joined #tripleo | 01:35 | |
| *** jeckersb is now known as jeckersb_gone | 01:36 | |
| *** matsuhashi has quit IRC | 01:40 | |
| *** matsuhashi has joined #tripleo | 01:43 | |
| *** jcooley_ has joined #tripleo | 01:54 | |
| *** shakayumi has quit IRC | 02:06 | |
| *** toci-bot has joined #tripleo | 02:13 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_SCTNRGT/ | 02:13 |
| *** toci-bot has quit IRC | 02:13 | |
| SpamapS | lifeless: I will be poking at it later today. I'm just in the mid-phase of my day called "feeding and bathing children" | 02:21 |
| lifeless | SpamapS: what! You have children! | 02:23 |
| SpamapS | lifeless: that would be more than a little creepy if I didn't.. | 02:24 |
| lifeless | SpamapS: ewwwwwwww | 02:24 |
| *** ogelbukh has quit IRC | 02:26 | |
| *** noslzzp has quit IRC | 02:47 | |
| *** vkozhukalov has joined #tripleo | 03:02 | |
| *** epim has quit IRC | 03:02 | |
| lifeless | right, I have cycles. What to do. Oh yeah, nova bug squad writeup. Then I think I', finally caught up | 03:11 |
| *** tzumainn has quit IRC | 03:18 | |
| *** michchap has quit IRC | 03:21 | |
| *** michchap has joined #tripleo | 03:22 | |
| *** matsuhashi has quit IRC | 03:31 | |
| *** matsuhashi has joined #tripleo | 03:31 | |
| *** matsuhashi has quit IRC | 03:36 | |
| *** matsuhashi has joined #tripleo | 03:41 | |
| *** CaptTofu has quit IRC | 03:53 | |
| *** CaptTofu has joined #tripleo | 03:54 | |
| *** jcooley_ has quit IRC | 03:57 | |
| *** rushiagr has joined #tripleo | 03:59 | |
| *** boris-42 has joined #tripleo | 04:03 | |
| *** toci-bot has joined #tripleo | 04:10 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_tv97mOt/ | 04:10 |
| *** toci-bot has quit IRC | 04:10 | |
| *** coolsvap has joined #tripleo | 04:16 | |
| *** ogelbukh has joined #tripleo | 04:24 | |
| *** akuznetsov has joined #tripleo | 04:28 | |
| *** jcooley_ has joined #tripleo | 04:33 | |
| *** rushiagr has quit IRC | 04:34 | |
| *** jcooley_ has quit IRC | 04:38 | |
| mordred | lifeless: you could also fix the flaky tests in the gate | 04:42 |
| *** cwolferh has quit IRC | 04:45 | |
| lifeless | mordred: That would be great. | 04:46 |
| *** arata has joined #tripleo | 04:46 | |
| *** boris-42 has quit IRC | 04:53 | |
| *** boris-42 has joined #tripleo | 04:56 | |
| *** cwolferh has joined #tripleo | 05:04 | |
| *** sdake_ has joined #tripleo | 05:11 | |
| *** jcooley_ has joined #tripleo | 05:14 | |
| *** arata has quit IRC | 05:14 | |
| *** jcooley_ has quit IRC | 05:19 | |
| *** rushiagr has joined #tripleo | 05:20 | |
| *** coolsvap has quit IRC | 05:39 | |
| *** marun has joined #tripleo | 05:55 | |
| *** marun has quit IRC | 05:55 | |
| *** vipul has quit IRC | 05:56 | |
| *** vipul has joined #tripleo | 05:57 | |
| *** coolsvap has joined #tripleo | 05:58 | |
| *** sdake_ has quit IRC | 05:59 | |
| *** sdake_ has joined #tripleo | 05:59 | |
| *** sdake_ has quit IRC | 05:59 | |
| *** sdake_ has joined #tripleo | 05:59 | |
| *** rpodolyaka1 has joined #tripleo | 06:02 | |
| *** toci-bot has joined #tripleo | 06:03 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_Wzmpp1s/ | 06:03 |
| *** toci-bot has quit IRC | 06:03 | |
| *** michchap has quit IRC | 06:04 | |
| *** michchap has joined #tripleo | 06:05 | |
| *** matsuhashi has quit IRC | 06:08 | |
| *** matsuhashi has joined #tripleo | 06:08 | |
| *** matsuhashi has quit IRC | 06:13 | |
| *** vkozhukalov has quit IRC | 06:15 | |
| *** rushiagr has quit IRC | 06:20 | |
| *** jcooley_ has joined #tripleo | 06:21 | |
| *** jcooley_ has quit IRC | 06:24 | |
| *** nosnos_ has joined #tripleo | 06:25 | |
| *** nosnos has quit IRC | 06:29 | |
| *** matsuhashi has joined #tripleo | 06:29 | |
| *** vipul is now known as vipul-away | 06:34 | |
| *** jcooley_ has joined #tripleo | 06:35 | |
| *** marun has joined #tripleo | 06:38 | |
| *** sdake_ has quit IRC | 06:44 | |
| *** jcooley_ has quit IRC | 06:51 | |
| *** cody-somerville has quit IRC | 06:55 | |
| *** sdake_ has joined #tripleo | 07:00 | |
| *** jcooley_ has joined #tripleo | 07:03 | |
| *** nosnos_ has quit IRC | 07:05 | |
| *** arata has joined #tripleo | 07:05 | |
| *** nosnos has joined #tripleo | 07:05 | |
| *** vipul-away is now known as vipul | 07:07 | |
| *** jcooley_ has quit IRC | 07:10 | |
| *** marun has quit IRC | 07:13 | |
| *** marun has joined #tripleo | 07:14 | |
| *** matsuhashi has quit IRC | 07:22 | |
| *** matsuhashi has joined #tripleo | 07:23 | |
| *** matsuhas_ has joined #tripleo | 07:30 | |
| *** matsuhashi has quit IRC | 07:34 | |
| GheRivero | morning all | 07:35 |
| *** rpodolyaka1 has quit IRC | 07:35 | |
| *** akuznetsov has quit IRC | 07:41 | |
| *** rushiagr has joined #tripleo | 07:41 | |
| marios | rpodolyaka: ping, hey, can you please check my comment on https://review.openstack.org/#/c/56957/ - if I'm wrong please let me know and I will fix pronto | 07:49 |
| *** jprovazn has joined #tripleo | 07:52 | |
| *** vkozhukalov has joined #tripleo | 07:53 | |
| *** sdake_ has quit IRC | 07:56 | |
| *** vkozhukalov has quit IRC | 08:01 | |
| *** toci-bot has joined #tripleo | 08:04 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_2Od87JV/ | 08:04 |
| *** toci-bot has quit IRC | 08:04 | |
| *** marun has quit IRC | 08:05 | |
| *** cwolferh has quit IRC | 08:05 | |
| *** marun has joined #tripleo | 08:05 | |
| *** vkozhukalov has joined #tripleo | 08:09 | |
| *** jcooley_ has joined #tripleo | 08:11 | |
| *** matsuhas_ has quit IRC | 08:18 | |
| *** matsuhashi has joined #tripleo | 08:19 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Check existence of rhel rpm key https://review.openstack.org/57196 | 08:23 |
| *** matsuhashi has quit IRC | 08:24 | |
| *** boris-42 has quit IRC | 08:26 | |
| rpodolyaka | morning tripleo | 08:27 |
| *** boris-42 has joined #tripleo | 08:28 | |
| *** ohadlevy_ is now known as ohadlevy | 08:31 | |
| *** ohadlevy has joined #tripleo | 08:31 | |
| *** matsuhashi has joined #tripleo | 08:31 | |
| SpamapS | rpodolyaka: o/ | 08:32 |
| *** arata has left #tripleo | 08:38 | |
| * rpodolyaka should buy a new espresso machine to feel refreshed in the morning | 08:38 | |
| rpodolyaka | marios: looking | 08:38 |
| *** rushiagr has quit IRC | 08:43 | |
| *** jcoufal has joined #tripleo | 08:44 | |
| *** jcooley_ has quit IRC | 08:45 | |
| *** shardy_afk is now known as shardy | 08:46 | |
| *** epim has joined #tripleo | 08:47 | |
| marios | rpodolyaka: thanks | 08:48 |
| *** boris-42 has quit IRC | 08:49 | |
| marios | rpodolyaka: seems like ghe rivero answered my question | 08:49 |
| rpodolyaka | marios: I've been writing a comprehensive comment :) | 08:49 |
| marios | rpodolyaka: heh :) ... | 08:50 |
| * marios ducks for cover | 08:50 | |
| *** derekh has joined #tripleo | 08:54 | |
| *** akuznetsov has joined #tripleo | 08:57 | |
| *** jistr has joined #tripleo | 08:58 | |
| *** panda has joined #tripleo | 09:00 | |
| Ng | morning | 09:02 |
| *** lsmola has joined #tripleo | 09:02 | |
| *** homegrown has joined #tripleo | 09:05 | |
| GheRivero | morning Ng! | 09:05 |
| Ng | hey GheRivero | 09:06 |
| *** ifarkas has joined #tripleo | 09:06 | |
| *** ifarkas_ has joined #tripleo | 09:07 | |
| rpodolyaka | marios: GheRivero: thank for review! I left two comments inline | 09:07 |
| lifeless | morning europe :) | 09:07 |
| shadower | hey | 09:07 |
| GheRivero | looking at them | 09:07 |
| GheRivero | night NZ | 09:07 |
| *** toci-bot has joined #tripleo | 09:08 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_LOvr2hX/ | 09:08 |
| *** toci-bot has quit IRC | 09:08 | |
| marios | rpodolyaka: thanks - fwiw re 'bash skills' , i tried for like 15 mins to make parameter expansion work (to make https://review.openstack.org/#/c/56956/2/scripts/devtest_overcloud.sh a bit tidier) but finally accepted defeat that bash won't let you chain them :) | 09:09 |
| *** lucasagomes has joined #tripleo | 09:09 | |
| *** arata has joined #tripleo | 09:10 | |
| *** epim has quit IRC | 09:10 | |
| rpodolyaka | :) | 09:10 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Remove the systemctl daemon-reload commands https://review.openstack.org/57397 | 09:10 |
| marios | rpodolyaka: ok, flipped to +1 | 09:11 |
| lifeless | marios: huh, how so? | 09:11 |
| lifeless | marios: won't FOO=${FOO:-${1:-''}} DTRT ? | 09:12 |
| marios | lifeless: trying... :) | 09:12 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Use os-apply-config everywhere https://review.openstack.org/57398 | 09:12 |
| *** panda has quit IRC | 09:12 | |
| lifeless | if that doesn't, try | 09:12 |
| lifeless | marios: won't FOO=${FOO:=${1:-''}} DTRT ? | 09:12 |
| derekh | Is it just me or have we problems with neutron? | 09:12 |
| lifeless | derekh: I'm sure we do | 09:13 |
| marios | lifeless: (DTRT?) | 09:13 |
| lifeless | marios: do the right thing | 09:13 |
| rpodolyaka | derekh: I had some weird problems with neutron yesterday | 09:13 |
| marios | heh, tx | 09:13 |
| derekh | I seem to be getting random key errors | 09:13 |
| rpodolyaka | ports and networks? | 09:13 |
| derekh | running a clean run now to verify | 09:13 |
| lifeless | derekh: (random key) errors, or (random) (key errors) ? | 09:13 |
| *** marun has quit IRC | 09:13 | |
| derekh | rpodolyaka: yup, port and networks | 09:13 |
| rpodolyaka | derekh: and they are gone after restart of neutron-server | 09:13 |
| * lifeless is just stirring | 09:13 | |
| derekh | lifeless: the latter | 09:13 |
| *** marun has joined #tripleo | 09:14 | |
| *** panda has joined #tripleo | 09:15 | |
| rpodolyaka | neutron-server log seems to be ok, except neutron-server fail to connect to DB a few times on start | 09:16 |
| marios | lifeless: nice, indeed FOO=${FOO:-${1:-''}} DTRT | 09:16 |
| lifeless | rpodolyaka: the early fail to connect is before the db is created | 09:16 |
| rpodolyaka | yeah, that's why I though it was ok | 09:17 |
| derekh | rpodolyaka: I've just zapped my setup, so I can test a clean devtest | 09:17 |
| rpodolyaka | going to run devtest too | 09:17 |
| marios | rpodolyaka: gonna add a note to https://review.openstack.org/#/c/56956/2/scripts/devtest_overcloud.sh based on lifeless help above (can maybe tidy up by chaining?) | 09:18 |
| *** akuznetsov has quit IRC | 09:18 | |
| rpodolyaka | marios: sure, feel free to -1 till it looks good enough ;) | 09:18 |
| rpodolyaka | especially when I do some bash coding :) | 09:19 |
| *** arata has quit IRC | 09:19 | |
| marios | rpodolyaka: thanks, example inline | 09:22 |
| *** blues-man has joined #tripleo | 09:24 | |
| *** blues-man has quit IRC | 09:24 | |
| *** rdopieralski has joined #tripleo | 09:24 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Fix issue with Ubuntu grub pre-install step for ARM https://review.openstack.org/57064 | 09:24 |
| *** akuznetsov has joined #tripleo | 09:34 | |
| * Ng wonders if anyone has done a successful run with saucy | 09:37 | |
| *** pblaho has joined #tripleo | 09:38 | |
| *** jcooley_ has joined #tripleo | 09:41 | |
| lifeless | Ng: quite sure not | 09:42 |
| *** rushiagr has joined #tripleo | 09:42 | |
| lifeless | Ng: 'One of these has to do with the current init script decision making on how to handle udev and devtmpfs. I'll take that offline through the normal channels, though.' | 09:42 |
| lifeless | Ng: thats all the info I have | 09:42 |
| Ng | lifeless: right, so that does sound like the same issue | 09:43 |
| *** ifarkas_ has quit IRC | 09:45 | |
| lifeless | http://upstream-tracker.org/changelogs/libudev/182/changelog.html | 09:46 |
| lifeless | Ng: ^ | 09:46 |
| lifeless | Ng: ' | 09:46 |
| lifeless | The udev control socket is now created in /run/udev/control | 09:46 |
| lifeless | and no longer as an abstract namespace one. ' | 09:46 |
| *** jcooley_ has quit IRC | 09:47 | |
| derekh | Ok, I got an error http://paste.openstack.org/show/53647/ | 09:59 |
| derekh | although it wasn't the one I was looking for | 10:00 |
| derekh | rpodolyaka: ^ seen that on seed node while trying to start undercloud | 10:00 |
| rpodolyaka | derekh: this is something new | 10:01 |
| rpodolyaka | wating for seed to boot | 10:01 |
| rpodolyaka | will provide feedback in a few minutes | 10:01 |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Create newer udev control directory. https://review.openstack.org/57409 | 10:02 |
| derekh | rpodolyaka: yup, I havn't seen it befor | 10:02 |
| SpamapS | ok and with that, I bid you all goodnight | 10:05 |
| derekh | hmm, password being sent to keystone | 10:05 |
| derekh | {"auth": {"tenantName": "service", "passwordCredentials": {"username": null, "password": "unset"}}} | 10:05 |
| derekh | username == null | 10:06 |
| rpodolyaka | hmm, interesting | 10:10 |
| *** akuznetsov has quit IRC | 10:13 | |
| *** matsuhashi has quit IRC | 10:14 | |
| rpodolyaka | derekh: I'm seeing the same problem | 10:16 |
| derekh | rpodolyaka: so at least I'm not going mad :-) | 10:16 |
| rpodolyaka | :) | 10:16 |
| *** panda__ has joined #tripleo | 10:17 | |
| rpodolyaka | http://paste.openstack.org/show/53648/ | 10:18 |
| rpodolyaka | nova api log | 10:18 |
| rpodolyaka | networks?... | 10:18 |
| rpodolyaka | wondering if neutron strikes back in seed too? | 10:18 |
| derekh | rpodolyaka: this seems to be nova not using the username when trying to talk to keystone, | 10:19 |
| derekh | maybe | 10:20 |
| *** panda has quit IRC | 10:20 | |
| derekh | going to see what I can find out with several debug print statements, will let you know if I make any progress | 10:21 |
| derekh | rpodolyaka: I killed neutron and ran nova boot still seeing the null username getting sent to keystone so it doesn't looks like its coming from neutron | 10:24 |
| *** akuznetsov has joined #tripleo | 10:37 | |
| *** marun has quit IRC | 10:42 | |
| *** jcooley_ has joined #tripleo | 10:43 | |
| derekh | rpodolyaka: looks like its a problem with CONF.neutron_admin_username | 10:48 |
| rpodolyaka | derekh: yep | 10:48 |
| rpodolyaka | just found it too :) | 10:48 |
| derekh | rpodolyaka: its being set to None in | 10:49 |
| derekh | "/opt/stack/venvs/nova/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py" 69L, 2536C | 10:49 |
| *** max_lobur_afk is now known as max_lobur | 10:49 | |
| derekh | rpodolyaka: its not in nova.conf | 10:49 |
| rpodolyaka | https://review.openstack.org/#/c/57249/1/elements/nova/os-apply-config/etc/nova/nova.conf | 10:50 |
| rpodolyaka | line 45 | 10:50 |
| rpodolyaka | isn't it meant to be neutron_admin_username? | 10:50 |
| derekh | rpo yup, think so | 10:50 |
| rpodolyaka | who files a bug and makes a patch? :) | 10:52 |
| derekh | rpodolyaka: go for it | 10:52 |
| rpodolyaka | ok | 10:52 |
| *** d0ugal is now known as steg_ | 10:54 | |
| *** steg_ is now known as d0ugal | 10:54 | |
| *** coolsvap has quit IRC | 10:57 | |
| *** jtomasek has joined #tripleo | 10:58 | |
| *** blues-man has joined #tripleo | 10:59 | |
| *** toci-bot has joined #tripleo | 11:03 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_RwdW32T/ | 11:03 |
| *** toci-bot has quit IRC | 11:03 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tripleo-image-elements: Fix neutron credentials in nova config file https://review.openstack.org/57416 | 11:03 |
| *** athomas has joined #tripleo | 11:09 | |
| rpodolyaka | ran devtest once again | 11:10 |
| * rpodolyaka goes away for lunch | 11:11 | |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Detect udevd version and behave accordingly. https://review.openstack.org/57409 | 11:11 |
| derekh | yup, same here, lets see if I can find what I was origionaly looking for | 11:12 |
| Ng | argh | 11:12 |
| Ng | why did I commit -a | 11:12 |
| Ng | moron | 11:12 |
| openstackgerrit | Chris Jones proposed a change to openstack/diskimage-builder: Detect udevd version and behave accordingly. https://review.openstack.org/57409 | 11:13 |
| Ng | SpamapS: sorry, that was a WIP when I pushed it up. I think the current version is more useful | 11:13 |
| Ng | I've not done a full end-to-end test, but this does at least get /dev populated on a saucy deploy ramdisk | 11:14 |
| Ng | (as in, a full run of devtest.sh with DIB_RELEASE=saucy) | 11:14 |
| lifeless | Ng: :> | 11:15 |
| lifeless | gnight | 11:15 |
| Ng | night | 11:15 |
| hewbrocca | Ng: nice! That was quick... | 11:15 |
| *** jcooley_ has quit IRC | 11:17 | |
| *** panda has joined #tripleo | 11:17 | |
| *** panda__ has quit IRC | 11:19 | |
| *** jtomasek has quit IRC | 11:19 | |
| *** mattymo has joined #tripleo | 11:19 | |
| *** akrivoka has joined #tripleo | 11:19 | |
| *** lsmola has quit IRC | 11:24 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Use os-apply-config everywhere https://review.openstack.org/57398 | 11:29 |
| *** jtomasek has joined #tripleo | 11:32 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Run DB sync in nova-api element post configure. https://review.openstack.org/56775 | 11:33 |
| *** lsmola has joined #tripleo | 11:36 | |
| derekh | Error number 2: Nov 20 11:28:23 localhost nova-api[1993]: 2013-11-20 11:28:22,513.513 2314 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: Maximum attempts reached | 11:48 |
| rpodolyaka | derekh: which stage? | 11:50 |
| derekh | when seed tries to start the undercloud | 11:51 |
| derekh | let me get the full traceback | 11:51 |
| rpodolyaka | hmm, works for me | 11:51 |
| rpodolyaka | devtest is building overcloud images now | 11:51 |
| derekh | ya, this is the error I was thinking about originally, I think its intermittent | 11:52 |
| derekh | http://paste.openstack.org/show/53654/ | 11:53 |
| *** lsmola has quit IRC | 11:53 | |
| derekh | Ng: were you doing some kind off offline mode for devtest a while back? | 11:56 |
| Ng | derekh: I was, but that was before the thing was split out into several scripts, so it needs some work to get it relevant again | 11:57 |
| Ng | I should probably dust that off | 11:57 |
| derekh | I find myself making the same set of eddits to the devtest script every day, offline would help a lot | 11:57 |
| derekh | Ng: This is what I normally do http://paste.fedoraproject.org/55369/48696138/ | 11:59 |
| Ng | derekh: nice, thanks | 11:59 |
| *** morazi has quit IRC | 12:00 | |
| derekh | Then I normally run devtest like this | 12:00 |
| derekh | rm ~/.cache/tripleo/*cow2 ~/.cache/image-create/image-file-cache/*qcow2 ; ./scripts/devtest.sh --trash-my-machine | 12:01 |
| derekh | answering yes to any images I want to regenerate | 12:01 |
| *** lsmola has joined #tripleo | 12:06 | |
| *** nosnos_ has joined #tripleo | 12:09 | |
| *** lucasagomes is now known as lucas-hungry | 12:12 | |
| *** nosnos has quit IRC | 12:12 | |
| *** jcooley_ has joined #tripleo | 12:13 | |
| *** nosnos_ has quit IRC | 12:14 | |
| *** jcooley_ has quit IRC | 12:19 | |
| *** marun has joined #tripleo | 12:24 | |
| *** michchap_ has joined #tripleo | 12:25 | |
| *** michchap has quit IRC | 12:25 | |
| *** boris-42 has joined #tripleo | 12:30 | |
| *** marun has quit IRC | 12:34 | |
| *** marun has joined #tripleo | 12:34 | |
| derekh | That doesn't look right 2 http request in a row without a response over persistent tcp stream http://fpaste.org/55376/84951231/ is that allowed ? | 12:42 |
| rpodolyaka | sounds like a race | 12:45 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Fix neutron credentials in nova config file https://review.openstack.org/57416 | 12:49 |
| *** noslzzp has joined #tripleo | 12:55 | |
| *** dprince has joined #tripleo | 12:57 | |
| *** akuznetsov has quit IRC | 12:57 | |
| rpodolyaka | I'm wondering if this introduces a race https://github.com/openstack/nova/commit/85332012dede96fa6729026c2a90594ea0502ac5 | 13:02 |
| *** toci-bot has joined #tripleo | 13:03 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_c9NztQC/ | 13:03 |
| *** toci-bot has quit IRC | 13:03 | |
| rpodolyaka | perhaps eventlet doesn't patch threading (so thread-locals aren't really local for green threads) | 13:03 |
| derekh | maybe, I'll look into it some more after lunch | 13:04 |
| *** jprovazn has quit IRC | 13:06 | |
| *** bcrochet has quit IRC | 13:09 | |
| *** morazi has joined #tripleo | 13:09 | |
| *** bcrochet has joined #tripleo | 13:12 | |
| *** akrivoka has quit IRC | 13:12 | |
| *** akrivoka has joined #tripleo | 13:13 | |
| *** jcooley_ has joined #tripleo | 13:15 | |
| rpodolyaka | 2013-11-20 13:15:14,073.073 4565 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1198, in list | 13:16 |
| rpodolyaka | 2013-11-20 13:15:14,073.073 4565 TRACE nova.openstack.common.rpc.amqp res.extend(r[collection]) | 13:16 |
| rpodolyaka | 2013-11-20 13:15:14,073.073 4565 TRACE nova.openstack.common.rpc.amqp KeyError: 'ports' | 13:16 |
| rpodolyaka | ok, so at least it's rather easy to reproduce | 13:17 |
| *** boris-42 has quit IRC | 13:17 | |
| *** boris-42 has joined #tripleo | 13:18 | |
| *** akuznetsov has joined #tripleo | 13:23 | |
| *** jprovazn has joined #tripleo | 13:26 | |
| openstackgerrit | Marios Andreou proposed a change to openstack/tuskar: Make sure we have tripleo-heat-templates at tuskar startup https://review.openstack.org/49745 | 13:27 |
| *** max_lobur has quit IRC | 13:28 | |
| *** max_lobur has joined #tripleo | 13:29 | |
| *** jdob has joined #tripleo | 13:40 | |
| *** ogelbukh has quit IRC | 13:47 | |
| *** jdob has quit IRC | 13:48 | |
| *** jdob has joined #tripleo | 13:48 | |
| *** jcooley_ has quit IRC | 13:49 | |
| *** ogelbukh has joined #tripleo | 13:50 | |
| *** cody-somerville has joined #tripleo | 13:50 | |
| *** tzumainn has joined #tripleo | 14:03 | |
| *** julim has joined #tripleo | 14:04 | |
| *** rushiagr has quit IRC | 14:16 | |
| *** jergerber has joined #tripleo | 14:18 | |
| *** jdob has quit IRC | 14:18 | |
| *** jdob has joined #tripleo | 14:19 | |
| *** CaptTofu has quit IRC | 14:19 | |
| *** CaptTofu has joined #tripleo | 14:19 | |
| *** blues-man has quit IRC | 14:25 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Remove the systemctl daemon-reload commands https://review.openstack.org/57397 | 14:29 |
| *** dprince has quit IRC | 14:30 | |
| derekh | So looks like this is my problem https://bugs.launchpad.net/tripleo/+bug/1251784 , anybody else seen that on devtest ? | 14:33 |
| uvirtbot | Launchpad bug 1251784 in tripleo "nova+neutron scheduling error: Connection to neutron failed: Maximum attempts reached" [Critical,New] | 14:33 |
| *** dprince has joined #tripleo | 14:34 | |
| *** markmc has joined #tripleo | 14:37 | |
| *** panda has quit IRC | 14:38 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tripleo-incubator: Use separate stack-create overcloud call in docs https://review.openstack.org/57445 | 14:38 |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tripleo-incubator: Use env variables as tripleo-cd cloud params https://review.openstack.org/56956 | 14:38 |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tripleo-incubator: Fix floating ips when devtest.sh is hand executed https://review.openstack.org/56957 | 14:38 |
| rpodolyaka | derekh: haven't seen this one yet, just our old KeyError friend | 14:39 |
| *** john-n-seattle1 has quit IRC | 14:39 | |
| *** boris-42_ has joined #tripleo | 14:40 | |
| *** boris-42 has quit IRC | 14:41 | |
| *** jeckersb_gone is now known as jeckersb | 14:44 | |
| NobodyCam | Good morning TripleO | 14:45 |
| rpodolyaka | derekh: I tried to revert https://github.com/openstack/nova/commit/85332012dede96fa6729026c2a90594ea0502ac5 and restarted nova-compute/api/scheduler/conductor . Then I was able to boot/delete nova instances. But after applying of that patch again and restarting of the services, nova was still working fine, without any sign of KeyError exceptions. So this might be a red herring... | 14:47 |
| derekh | rpodolyaka: I did notice last night that I was getting the key error, I had 10 VMs running and couldn't delete any, I ended up having to reboot nova-compute once for each delete | 14:51 |
| dkehn | monring all | 14:51 |
| GheRivero | morning al | 14:52 |
| GheRivero | l | 14:52 |
| derekh | rpodolyaka: that patch you linked merged on the same date as my errors http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkNvbm5lY3Rpb24gdG8gbmV1dHJvbiBmYWlsZWQ6IE1heGltdW0gYXR0ZW1wdHMgcmVhY2hlZFwiICBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLXNjaC50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4NDU1NjE2NTY2OCwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== | 14:53 |
| *** panda_ has joined #tripleo | 14:53 | |
| derekh | same date my errors started that is | 14:53 |
| rpodolyaka | hmm | 14:59 |
| derekh | Caching the neutron client would explan who it would become possible that my http requests started happening out of order, running devtest with this reverted | 15:01 |
| *** morazi has quit IRC | 15:03 | |
| *** toci-bot has joined #tripleo | 15:04 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_WJLjxJ1/ | 15:04 |
| *** toci-bot has quit IRC | 15:04 | |
| rpodolyaka | going to try this too | 15:04 |
| *** akuznetsov has quit IRC | 15:14 | |
| *** bauzas has quit IRC | 15:14 | |
| *** morazi has joined #tripleo | 15:16 | |
| *** blamar has quit IRC | 15:16 | |
| *** coolsvap has joined #tripleo | 15:17 | |
| *** homegrown has quit IRC | 15:22 | |
| *** blamar has joined #tripleo | 15:23 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in nova-baremetal element post conf. https://review.openstack.org/57459 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in glance element post confgure. https://review.openstack.org/57460 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in heat-api element post configure. https://review.openstack.org/57461 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in cinder element post configure. https://review.openstack.org/57462 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in ironic-api element post configure. https://review.openstack.org/57463 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in tuskar-api element post configure. https://review.openstack.org/57464 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Run DB sync in keystone element post configure. https://review.openstack.org/57465 | 15:32 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Refactor reset-db so we don't sync by default. https://review.openstack.org/57466 | 15:32 |
| *** pblaho has quit IRC | 15:35 | |
| *** jeckersb has left #tripleo | 15:41 | |
| *** CaptTofu has quit IRC | 15:46 | |
| *** lucas-hungry is now known as lucasagomes | 15:46 | |
| *** CaptTofu has joined #tripleo | 15:46 | |
| *** boris-42_ is now known as boris-42 | 15:47 | |
| *** akuznetsov has joined #tripleo | 15:49 | |
| *** rdopieralski has quit IRC | 15:51 | |
| *** jcoufal has quit IRC | 16:00 | |
| *** vkozhukalov has quit IRC | 16:00 | |
| *** CaptTofu has quit IRC | 16:01 | |
| *** CaptTofu has joined #tripleo | 16:01 | |
| *** toci-bot has joined #tripleo | 16:02 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_TCLWo3I/ | 16:02 |
| *** toci-bot has quit IRC | 16:02 | |
| *** jcooley_ has joined #tripleo | 16:04 | |
| *** UtahDave has joined #tripleo | 16:05 | |
| *** marun has quit IRC | 16:06 | |
| *** markmc has quit IRC | 16:07 | |
| *** jcooley_ has quit IRC | 16:22 | |
| *** coolsvap has quit IRC | 16:29 | |
| rpodolyaka | ok, so whatever the reason is, with that commit reverted, devtest works for me again | 16:31 |
| *** jcoufal has joined #tripleo | 16:33 | |
| openstackgerrit | A change was merged to openstack/tuskar: Make sure we have tripleo-heat-templates at tuskar startup https://review.openstack.org/49745 | 16:39 |
| rpodolyaka | running devtest again with nova master | 16:41 |
| *** akuznetsov has quit IRC | 16:46 | |
| *** coolsvap has joined #tripleo | 16:48 | |
| *** coolsvap has quit IRC | 16:49 | |
| *** jcoufal has quit IRC | 16:51 | |
| *** jcooley_ has joined #tripleo | 16:53 | |
| *** jcoufal-mob has joined #tripleo | 16:54 | |
| *** john-n-seattle1 has joined #tripleo | 16:55 | |
| *** john-n-seattle1 has quit IRC | 16:59 | |
| *** akuznetsov has joined #tripleo | 17:02 | |
| rpodolyaka | hey, check this out http://paste.openstack.org/show/53686/ | 17:04 |
| rpodolyaka | am I missing something? | 17:05 |
| rpodolyaka | or we are using eventlet TLS incorrectly? | 17:06 |
| *** akuznetsov has quit IRC | 17:06 | |
| *** jcoufal-mob has quit IRC | 17:09 | |
| *** jcooley_ has quit IRC | 17:10 | |
| *** akrivoka has quit IRC | 17:13 | |
| *** jistr has quit IRC | 17:14 | |
| *** john-n-seattle1 has joined #tripleo | 17:14 | |
| *** CaptTofu has quit IRC | 17:15 | |
| *** CaptTofu has joined #tripleo | 17:17 | |
| *** cwolferh has joined #tripleo | 17:18 | |
| *** derekh has quit IRC | 17:27 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Detect udevd version and behave accordingly. https://review.openstack.org/57409 | 17:36 |
| *** toci-bot has joined #tripleo | 17:37 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_ILjtwXr/ | 17:37 |
| *** toci-bot has quit IRC | 17:37 | |
| *** vkozhukalov has joined #tripleo | 17:42 | |
| *** boris-42 has quit IRC | 17:42 | |
| *** funzo has joined #tripleo | 17:48 | |
| *** jtomasek has quit IRC | 17:50 | |
| *** max_lobur is now known as max_lobur_afk | 18:05 | |
| *** lucasagomes has quit IRC | 18:08 | |
| *** sdake_ has joined #tripleo | 18:11 | |
| *** sdake_ has quit IRC | 18:11 | |
| *** sdake_ has joined #tripleo | 18:11 | |
| *** athomas has quit IRC | 18:11 | |
| *** marun has joined #tripleo | 18:19 | |
| *** toci-bot has joined #tripleo | 18:27 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_tv8z54U/ | 18:27 |
| *** toci-bot has quit IRC | 18:27 | |
| *** rushiagr has joined #tripleo | 18:29 | |
| *** MarkAtwood has joined #tripleo | 18:31 | |
| *** MarkAtwood has quit IRC | 18:31 | |
| *** rpodolyaka1 has joined #tripleo | 18:49 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Open up gearman server port. https://review.openstack.org/57512 | 18:59 |
| *** toci-bot has joined #tripleo | 19:06 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_VIYMxkM/ | 19:06 |
| *** toci-bot has quit IRC | 19:06 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Open up gearman server port. https://review.openstack.org/57512 | 19:13 |
| rushiagr | no meeting today? | 19:19 |
| rpodolyaka1 | it was yesterday :) | 19:19 |
| rushiagr | ouch | 19:20 |
| rushiagr | will keep in mind from next time | 19:21 |
| rushiagr | good night :) | 19:21 |
| rpodolyaka1 | night! | 19:21 |
| *** rushiagr has left #tripleo | 19:21 | |
| *** jprovazn has quit IRC | 19:27 | |
| *** markmc has joined #tripleo | 19:40 | |
| *** blamar has quit IRC | 19:41 | |
| *** blamar has joined #tripleo | 19:44 | |
| *** jdob has quit IRC | 19:54 | |
| *** jdob has joined #tripleo | 19:54 | |
| *** sdake_ is now known as randallburt | 20:10 | |
| *** derekh has joined #tripleo | 20:11 | |
| *** jdob has quit IRC | 20:12 | |
| *** jdob has joined #tripleo | 20:13 | |
| pleia2 | dprince, derekh, lifeless, I'm around whenever you are | 20:15 |
| dprince | pleia2: invites just sent. | 20:16 |
| *** randallburt is now known as sdkae | 20:19 | |
| *** sdkae is now known as sdake_ | 20:19 | |
| *** vipul is now known as vipul-away | 20:19 | |
| *** vipul-away is now known as vipul | 20:19 | |
| *** asalkeld is now known as are_we_all_chang | 20:20 | |
| *** are_we_all_chang is now known as angusss | 20:20 | |
| *** sdake_ is now known as randallburt | 20:21 | |
| *** angusss is now known as asalkeld | 20:26 | |
| *** randallburt is now known as sdake_ | 20:26 | |
| Ng | ah crap, I thought I was going to be able to fix bug #1236546 with a single extra flag to the cp call, but GNU's cp still returns zero if it's had to refuse to clobber files :( | 20:31 |
| uvirtbot | Launchpad bug 1236546 in diskimage-builder "elements can be subtly wrong with no warning to developers" [Medium,Triaged] https://launchpad.net/bugs/1236546 | 20:31 |
| Ng | hmm, not that one, bug #1251952 | 20:31 |
| uvirtbot | Launchpad bug 1251952 in diskimage-builder "Hooks with the same name can override each other without triggering an error" [High,Triaged] https://launchpad.net/bugs/1251952 | 20:31 |
| rpodolyaka1 | yeah, checked that too yesterday :( | 20:37 |
| *** epim has joined #tripleo | 20:41 | |
| *** blamar has quit IRC | 20:44 | |
| *** vkozhukalov has quit IRC | 20:46 | |
| *** vipul is now known as vipul-away | 20:48 | |
| *** blamar has joined #tripleo | 20:50 | |
| *** vipul-away is now known as vipul | 20:51 | |
| *** marun has quit IRC | 20:53 | |
| *** panda_ has quit IRC | 20:54 | |
| *** panda_ has joined #tripleo | 20:56 | |
| *** jergerber has quit IRC | 20:57 | |
| *** shardy has quit IRC | 21:01 | |
| *** rpodolyaka1 has quit IRC | 21:01 | |
| *** shardy has joined #tripleo | 21:03 | |
| *** bauzas has joined #tripleo | 21:12 | |
| *** funzo has quit IRC | 21:18 | |
| *** vipul is now known as vipul-away | 21:20 | |
| *** vipul-away is now known as vipul | 21:20 | |
| *** dprince has quit IRC | 21:23 | |
| *** LarsN has quit IRC | 21:32 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Add cinder to the list of installed clients https://review.openstack.org/57529 | 21:38 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-image-elements: Use upstream glance policy.json https://review.openstack.org/57530 | 21:39 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Timeout if seed fails to get IP address https://review.openstack.org/57531 | 21:40 |
| *** noslzzp has quit IRC | 21:41 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Add swift roles to keystone https://review.openstack.org/57532 | 21:42 |
| *** noslzzp has joined #tripleo | 21:42 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Add the nova V3 api endpoint https://review.openstack.org/57533 | 21:43 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Install the cinder client on devtest host https://review.openstack.org/57535 | 21:44 |
| *** jdob has quit IRC | 21:45 | |
| *** sdake_ has quit IRC | 21:57 | |
| *** bauzas has quit IRC | 22:01 | |
| *** UtahDave has quit IRC | 22:09 | |
| openstackgerrit | Tzu-Mainn Chen proposed a change to openstack/tuskar-ui: Use baremetal node id for node detail page https://review.openstack.org/57549 | 22:52 |
| *** noslzzp has quit IRC | 22:54 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Timeout if seed fails to get IP address https://review.openstack.org/57531 | 23:00 |
| lifeless | derekh: have you tried export -f ? | 23:03 |
| *** Ng has quit IRC | 23:03 | |
| derekh | lifeless: I hadn't (and didn't know it existed) | 23:04 |
| *** epim has quit IRC | 23:04 | |
| *** michchap_ has quit IRC | 23:15 | |
| *** michchap has joined #tripleo | 23:18 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Add swift roles to keystone https://review.openstack.org/57532 | 23:23 |
| *** markmc has quit IRC | 23:34 | |
| *** sdake_ has joined #tripleo | 23:37 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-heat-templates: enable Ironic password for keystone auth https://review.openstack.org/57564 | 23:43 |
| derekh | lifeless: so export -f works for exposing the function to wait_for but then I'll need to run poll_vm again outside of wait_for to get the IP | 23:50 |
| lifeless | derekh: sure, poll_vm is fast right? | 23:53 |
| derekh | yup, | 23:53 |
| *** julim has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!