| lifeless | ahha we'd turned it off because it's pre-neutron (so has wildcard dhcpd) | 00:16 |
|---|---|---|
| lifeless | turned off nova-bm-dnsmasq I mean | 00:16 |
| *** matsuhashi has joined #tripleo | 00:33 | |
| lifeless | ok, undercloud reconfigured with ephemeral partitions | 00:44 |
| lifeless | 500 GB /, 1000 GB /mnt | 00:44 |
| lifeless | in principle | 00:44 |
| *** bauzas has quit IRC | 01:02 | |
| *** toci-bot has joined #tripleo | 01:14 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_oYcVePZ/ | 01:14 |
| *** toci-bot has quit IRC | 01:14 | |
| *** nosnos has joined #tripleo | 01:20 | |
| *** jamezpolley has joined #tripleo | 01:26 | |
| *** itooon has joined #tripleo | 01:38 | |
| *** CaptTofu has joined #tripleo | 01:57 | |
| *** rongze has joined #tripleo | 01:59 | |
| *** rongze has quit IRC | 02:18 | |
| *** rongze has joined #tripleo | 02:20 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/tripleo-ci: Drop notcompute element. https://review.openstack.org/59112 | 02:32 |
| *** toci-bot has joined #tripleo | 03:05 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_RpemykL/ | 03:05 |
| *** toci-bot has quit IRC | 03:05 | |
| *** rongze has quit IRC | 03:06 | |
| *** rongze has joined #tripleo | 03:06 | |
| *** michchap has quit IRC | 03:48 | |
| *** rbrady has quit IRC | 03:55 | |
| *** CaptTofu has quit IRC | 03:57 | |
| *** CaptTofu has joined #tripleo | 03:58 | |
| *** rongze has quit IRC | 04:11 | |
| *** rongze has joined #tripleo | 04:17 | |
| *** jcooley_ has joined #tripleo | 04:17 | |
| *** jcooley_ has quit IRC | 04:35 | |
| *** jcooley_ has joined #tripleo | 04:35 | |
| *** rushiagr has joined #tripleo | 04:40 | |
| *** boris-42 has joined #tripleo | 04:49 | |
| *** rongze has quit IRC | 04:50 | |
| *** jcooley_ has quit IRC | 04:58 | |
| *** jcooley_ has joined #tripleo | 04:59 | |
| *** rongze has joined #tripleo | 05:08 | |
| *** toci-bot has joined #tripleo | 05:15 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_MvSQsCa/ | 05:15 |
| *** toci-bot has quit IRC | 05:15 | |
| *** itooon_ has joined #tripleo | 05:19 | |
| *** itooon has quit IRC | 05:19 | |
| *** itooon_ is now known as itooon | 05:19 | |
| *** rpodolyaka1 has joined #tripleo | 05:22 | |
| *** michchap has joined #tripleo | 05:35 | |
| openstackgerrit | Jun Jie Nan proposed a change to openstack/diskimage-builder: Add an element to change root password https://review.openstack.org/58972 | 05:57 |
| *** toci-bot has joined #tripleo | 06:02 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_ZZ1LtJw/ | 06:02 |
| *** toci-bot has quit IRC | 06:02 | |
| *** sdake_ has joined #tripleo | 06:07 | |
| openstackgerrit | Jun Jie Nan proposed a change to openstack/diskimage-builder: Add an element to change root password https://review.openstack.org/58972 | 06:09 |
| *** vkozhukalov has joined #tripleo | 06:20 | |
| *** sdake_ has quit IRC | 06:20 | |
| openstackgerrit | Jun Jie Nan proposed a change to openstack/diskimage-builder: Store DIB_* only env variables https://review.openstack.org/59123 | 06:28 |
| openstackgerrit | Jun Jie Nan proposed a change to openstack/diskimage-builder: Add an element to change root password https://review.openstack.org/58972 | 06:36 |
| *** vkozhukalov has quit IRC | 06:41 | |
| *** akuznetsov has quit IRC | 06:54 | |
| *** jcooley_ has quit IRC | 07:03 | |
| *** rongze has quit IRC | 07:03 | |
| *** jcooley_ has joined #tripleo | 07:04 | |
| *** rongze has joined #tripleo | 07:06 | |
| *** jcooley_ has quit IRC | 07:08 | |
| *** jtomasek has joined #tripleo | 07:13 | |
| *** rpodolyaka1 has quit IRC | 07:17 | |
| *** rongze has quit IRC | 07:27 | |
| *** rongze has joined #tripleo | 07:28 | |
| *** akuznetsov has joined #tripleo | 07:31 | |
| *** nosnos_ has joined #tripleo | 07:33 | |
| *** nosnos has quit IRC | 07:37 | |
| lifeless | ulling/updating diskimage-builder | 07:37 |
| lifeless | error: Couldn't resolve host 'git.openstack.org' while accessing https://git.openstack.org/cgit/openstack/diskim ge-builder/info/refs | 07:37 |
| *** akuznetsov has quit IRC | 07:37 | |
| lifeless | ah bad resolv.conf after reboot | 07:37 |
| lifeless | fixed | 07:37 |
| SpamapS | lifeless: I have as one of my next todo's to build an image with the new mellanox driver and deploy it for testing. | 07:38 |
| * SpamapS says from US holiday land. | 07:38 | |
| lifeless | SpamapS: yeah, cool :) | 07:39 |
| lifeless | in principle in 60m we'll have an ephemeral mounted partition in the deployed thing | 07:39 |
| *** rdopieralski has joined #tripleo | 07:41 | |
| lifeless | lol http://osrc.dfm.io/rbtcollinshttp://osrc.dfm.io/rbtcollins | 07:48 |
| lifeless | erm http://osrc.dfm.io/rbtcollins | 07:48 |
| *** jcoufal has joined #tripleo | 07:49 | |
| *** rongze_ has joined #tripleo | 07:50 | |
| *** cd-undercloud has joined #tripleo | 07:51 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 07:51 |
| *** cd-undercloud has quit IRC | 07:51 | |
| *** rongze has quit IRC | 07:53 | |
| lifeless | huh | 07:57 |
| lifeless | You must provide a password via either --os-password or env[OS_PASSWORD] | 07:57 |
| lifeless | in devtest_overcloud.sh | 07:57 |
| SpamapS | There is also an obvious connection between rbtcollins and Devananda, SpamapS, Piotr Dobrogost, and Lennie. | 07:58 |
| lifeless | no changes to incubator that should have triggered that | 07:58 |
| rpodolyaka | morning | 08:02 |
| lifeless | so, copied the new tripleo-cd.sh in which will give us throttling and SpamapS hardware doco | 08:05 |
| lifeless | and restarting tripleo-cd | 08:05 |
| rpodolyaka | "It seems like Roman is—or should be—friends with Mike Bayer. ", hehe, nice | 08:06 |
| *** athomas has joined #tripleo | 08:09 | |
| *** rongze has joined #tripleo | 08:11 | |
| *** bauzas has joined #tripleo | 08:11 | |
| lifeless | hmm, 2 people have nominated me for a director position on the o/s board | 08:13 |
| lifeless | what did I do wrong ? :) | 08:13 |
| *** boris-42 has quit IRC | 08:13 | |
| *** toci-bot has joined #tripleo | 08:14 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_vKjvuAY/ | 08:14 |
| *** toci-bot has quit IRC | 08:14 | |
| *** rongze_ has quit IRC | 08:15 | |
| *** pblaho has joined #tripleo | 08:16 | |
| *** cd-undercloud has joined #tripleo | 08:18 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 08:18 |
| *** cd-undercloud has quit IRC | 08:18 | |
| *** bauzas has quit IRC | 08:20 | |
| rpodolyaka | :) | 08:21 |
| *** bauzas has joined #tripleo | 08:21 | |
| *** bauzas has quit IRC | 08:27 | |
| *** matsuhashi has quit IRC | 08:28 | |
| *** vkozhukalov has joined #tripleo | 08:28 | |
| hewbrocca | lifeless: no good deed goes unpunished | 08:30 |
| *** cd-undercloud has joined #tripleo | 08:30 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 08:30 |
| *** cd-undercloud has quit IRC | 08:30 | |
| rpodolyaka | http://russellbryant.net/openstack-stats/neutron-openreviews.txt (sadface) | 08:33 |
| *** akuznetsov has joined #tripleo | 08:33 | |
| marios | rpodolyaka: yah, am trying to work out what's going on over there - huge difference to how tripleo works | 08:34 |
| rpodolyaka | it's really-really cool that we are doing much better here in tripleo ;) | 08:34 |
| marios | rpodolyaka: #openstack-neutron is seriously quiet this week but it is thanksgiving | 08:34 |
| marios | rpodolyaka: i've had a review open for a few days with little activity | 08:34 |
| rpodolyaka | marios: yeah, I once had a review in neutron that was opened for about 100 days... | 08:35 |
| marios | jhchrist | 08:36 |
| rpodolyaka | marios: waiting for reviewers 99.9% of time | 08:36 |
| hewbrocca | grumble | 08:37 |
| marios | hewbrocca: aren't you supposed to be nursing a hangover and contemplating turkey sandwiches about now? | 08:39 |
| *** shardy_afk is now known as shardy | 08:39 | |
| *** bauzas has joined #tripleo | 08:42 | |
| *** cd-undercloud has joined #tripleo | 08:44 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 08:44 |
| *** cd-undercloud has quit IRC | 08:44 | |
| *** rongze has quit IRC | 08:44 | |
| *** rongze has joined #tripleo | 08:45 | |
| *** yamahata has quit IRC | 08:48 | |
| *** bauzas has quit IRC | 08:51 | |
| hewbrocca | LOL | 08:53 |
| hewbrocca | marios: still in Brno | 08:53 |
| hewbrocca | Did go out for a lovely dinner last night but it had nothing to do with thanksgiving | 08:53 |
| lifeless | ok so that status=1 is genuine | 08:54 |
| lifeless | we has a bug | 08:54 |
| marios | hewbrocca: oh long stay this time then? oh well, am sure you can make up with extra turkey in 3 weeks :) | 08:54 |
| *** ifarkas has joined #tripleo | 08:55 | |
| *** cd-undercloud has joined #tripleo | 08:56 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 08:56 |
| *** cd-undercloud has quit IRC | 08:56 | |
| *** rushiagr is now known as rushiagr_away | 08:57 | |
| *** rushiagr_away is now known as rushiagr | 08:58 | |
| hewbrocca | Yeah -- I've been here since September | 08:59 |
| *** yanhcdl has quit IRC | 09:01 | |
| *** yanhcdl has joined #tripleo | 09:02 | |
| *** bauzas has joined #tripleo | 09:06 | |
| *** jistr has joined #tripleo | 09:06 | |
| *** cd-undercloud has joined #tripleo | 09:09 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:09 |
| *** cd-undercloud has quit IRC | 09:09 | |
| *** bauzas has quit IRC | 09:15 | |
| *** derekh has joined #tripleo | 09:19 | |
| *** matsuhashi has joined #tripleo | 09:20 | |
| *** marios_ has joined #tripleo | 09:22 | |
| *** cd-undercloud has joined #tripleo | 09:22 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:22 |
| *** cd-undercloud has quit IRC | 09:22 | |
| *** pleia2_ has joined #tripleo | 09:22 | |
| Ng | morning | 09:23 |
| *** marios_ has quit IRC | 09:23 | |
| *** marios_ has joined #tripleo | 09:23 | |
| lifeless | Ng: hey | 09:24 |
| Ng | hey hey | 09:24 |
| lifeless | Ng: so, overcloud deploys are cactus | 09:24 |
| lifeless | Ng: it's failing to load a password somewhere | 09:24 |
| lifeless | Ng: 20:57 < lifeless> You must provide a password via either --os-password or env[OS_PASSWORD] | 09:25 |
| lifeless | Ng: can I tag ? | 09:25 |
| Ng | lifeless: sure | 09:25 |
| lifeless | cool | 09:25 |
| lifeless | tag :) | 09:25 |
| *** lucasagomes has joined #tripleo | 09:25 | |
| * Ng it | 09:25 | |
| *** yanhcdl has quit IRC | 09:26 | |
| *** marios has quit IRC | 09:26 | |
| *** pleia2 has quit IRC | 09:26 | |
| *** matsuhashi has quit IRC | 09:29 | |
| *** bauzas has joined #tripleo | 09:30 | |
| *** matsuhas_ has joined #tripleo | 09:31 | |
| *** boris-42 has joined #tripleo | 09:31 | |
| *** yanhcdl has joined #tripleo | 09:33 | |
| *** bauzas has quit IRC | 09:34 | |
| *** cd-undercloud has joined #tripleo | 09:36 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:36 |
| *** cd-undercloud has quit IRC | 09:36 | |
| *** nosnos_ has quit IRC | 09:36 | |
| *** itooon has quit IRC | 09:38 | |
| *** bauzas has joined #tripleo | 09:48 | |
| rpodolyaka | btw, https://review.openstack.org/#/c/58951/2 has been merged | 09:48 |
| *** cd-undercloud has joined #tripleo | 09:49 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 09:49 |
| *** cd-undercloud has quit IRC | 09:49 | |
| rpodolyaka | problem with neutron policies must be fixed now (at least partially) | 09:49 |
| *** martyntaylor has joined #tripleo | 09:54 | |
| *** martyntaylor has left #tripleo | 09:55 | |
| *** martyntaylor has joined #tripleo | 09:55 | |
| *** cd-undercloud has joined #tripleo | 10:02 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:02 |
| *** cd-undercloud has quit IRC | 10:02 | |
| *** markmc has joined #tripleo | 10:05 | |
| *** matsuhas_ has quit IRC | 10:12 | |
| *** toci-bot has joined #tripleo | 10:13 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_hlDo0RC/ | 10:13 |
| *** toci-bot has quit IRC | 10:13 | |
| *** cd-undercloud has joined #tripleo | 10:14 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:14 |
| *** cd-undercloud has quit IRC | 10:14 | |
| *** matsuhashi has joined #tripleo | 10:17 | |
| *** athomas has quit IRC | 10:19 | |
| derekh | Ng: trying to find the problem aswell, were does tripleo-cd.sh source undercloudrc ? | 10:22 |
| derekh | Ng: I don't see it and the undercloud seems to have been rebooted, relevant ? | 10:25 |
| *** cd-undercloud has joined #tripleo | 10:27 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:27 |
| *** cd-undercloud has quit IRC | 10:27 | |
| *** max_lobur_afk is now known as max_lobur | 10:27 | |
| *** athomas has joined #tripleo | 10:30 | |
| *** athomas has joined #tripleo | 10:31 | |
| derekh | Ng: I think we need to source the file /home/heat-admin/stackrc in tripleo-cd , the env variables must have somehow been in the env in the upstart job, gonna test something | 10:33 |
| *** michchap_ has joined #tripleo | 10:38 | |
| *** cd-undercloud has joined #tripleo | 10:41 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:41 |
| *** cd-undercloud has quit IRC | 10:41 | |
| Ng | derekh: yeah that's a good point | 10:42 |
| *** michchap has quit IRC | 10:42 | |
| * derekh is trying to understand how this worked in the first place | 10:43 | |
| derekh | Ng: found the problem, the heat metadata on the undercloud seems to be missing (havn't looked into why yet) | 10:49 |
| derekh | Ng: we source /root/stackrc | 10:50 |
| derekh | Ng: which is being written out by os-refresh-config | 10:50 |
| derekh | Ng: and contains | 10:50 |
| derekh | export OS_PASSWORD={{admin-password}} | 10:50 |
| derekh | os-config-applier --key admin-password | 10:50 |
| derekh | [2013/11/29 10:48:59 AM] [ERROR] key admin-password does not exist in [] | 10:50 |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/os-apply-config: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59154 | 10:51 |
| Ng | derekh: I need to pop out for a bit, feel free to leave it for me to poke at if you want to get to other things :) | 10:52 |
| derekh | Ng: ok, cool, will do | 10:53 |
| *** cd-undercloud has joined #tripleo | 10:54 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:54 |
| *** cd-undercloud has quit IRC | 10:54 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/os-collect-config: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59155 | 10:56 |
| *** cd-undercloud has joined #tripleo | 11:07 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:07 |
| *** cd-undercloud has quit IRC | 11:07 | |
| *** cd-undercloud has joined #tripleo | 11:21 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:21 |
| *** cd-undercloud has quit IRC | 11:21 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tuskar: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59158 | 11:21 |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/tuskar-ui: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59161 | 11:26 |
| *** cd-undercloud has joined #tripleo | 11:34 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:34 |
| *** cd-undercloud has quit IRC | 11:34 | |
| *** pblaho has quit IRC | 11:35 | |
| *** cd-undercloud has joined #tripleo | 11:46 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:46 |
| *** cd-undercloud has quit IRC | 11:46 | |
| openstackgerrit | Roman Podoliaka proposed a change to openstack/python-tuskarclient: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59163 | 11:51 |
| *** rushiagr has quit IRC | 11:53 | |
| *** cd-undercloud has joined #tripleo | 11:59 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 11:59 |
| *** cd-undercloud has quit IRC | 11:59 | |
| *** pblaho has joined #tripleo | 12:06 | |
| *** cd-undercloud has joined #tripleo | 12:12 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:12 |
| *** cd-undercloud has quit IRC | 12:12 | |
| *** toci-bot has joined #tripleo | 12:13 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_uTVWbeD/ | 12:13 |
| *** toci-bot has quit IRC | 12:13 | |
| *** pblaho has quit IRC | 12:15 | |
| *** matsuhashi has quit IRC | 12:19 | |
| *** matsuhashi has joined #tripleo | 12:19 | |
| *** matsuhashi has quit IRC | 12:23 | |
| *** cd-undercloud has joined #tripleo | 12:25 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:25 |
| *** cd-undercloud has quit IRC | 12:25 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Add element to install a testenv worker https://review.openstack.org/58305 | 12:29 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Add element to install a testenv client https://review.openstack.org/58306 | 12:29 |
| *** pblaho has joined #tripleo | 12:33 | |
| *** cd-undercloud has joined #tripleo | 12:38 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:38 |
| *** cd-undercloud has quit IRC | 12:38 | |
| *** rushiagr has joined #tripleo | 12:41 | |
| *** rongze has quit IRC | 12:41 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Drop default distribution root element support https://review.openstack.org/58601 | 12:50 |
| *** cd-undercloud has joined #tripleo | 12:52 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 12:52 |
| *** cd-undercloud has quit IRC | 12:52 | |
| openstackgerrit | Marios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py invokable from another script https://review.openstack.org/56947 | 12:58 |
| *** insanidade has quit IRC | 12:58 | |
| openstackgerrit | Marios Andreou proposed a change to openstack/tripleo-heat-templates: Make merge.py invokable from another script https://review.openstack.org/56947 | 13:02 |
| *** cd-undercloud has joined #tripleo | 13:05 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:05 |
| *** cd-undercloud has quit IRC | 13:05 | |
| *** boris-42 has quit IRC | 13:07 | |
| *** vkozhukalov has quit IRC | 13:11 | |
| *** rongze has joined #tripleo | 13:12 | |
| *** lucasagomes is now known as lucas-hungry | 13:12 | |
| *** CaptTofu has quit IRC | 13:16 | |
| openstackgerrit | Marios Andreou proposed a change to openstack/tripleo-heat-templates: WIP: make merge.py installable as a package using pbr https://review.openstack.org/59176 | 13:16 |
| *** CaptTofu has joined #tripleo | 13:16 | |
| *** cd-undercloud has joined #tripleo | 13:18 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:18 |
| *** cd-undercloud has quit IRC | 13:18 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Use set-os-type to set default NODE_DIST https://review.openstack.org/58426 | 13:19 |
| *** rbrady has joined #tripleo | 13:24 | |
| *** rushiagr has left #tripleo | 13:24 | |
| *** rushiagr has quit IRC | 13:24 | |
| *** vkozhukalov has joined #tripleo | 13:25 | |
| *** rongze has quit IRC | 13:28 | |
| *** cd-undercloud has joined #tripleo | 13:31 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:31 |
| *** cd-undercloud has quit IRC | 13:31 | |
| *** jtomasek has quit IRC | 13:40 | |
| *** cd-undercloud has joined #tripleo | 13:43 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:43 |
| *** cd-undercloud has quit IRC | 13:43 | |
| *** cd-undercloud has joined #tripleo | 13:57 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 13:57 |
| *** cd-undercloud has quit IRC | 13:57 | |
| *** lucas-hungry is now known as lucasagomes | 14:07 | |
| *** vkozhukalov has quit IRC | 14:07 | |
| *** cd-undercloud has joined #tripleo | 14:10 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:10 |
| *** cd-undercloud has quit IRC | 14:10 | |
| *** rbrady has quit IRC | 14:11 | |
| shadower | why do we have to set up the service endpoints manually? | 14:12 |
| shadower | that sounds like something that could and sholud be automated | 14:12 |
| shadower | either with heat or by the elements themselves | 14:13 |
| *** toci-bot has joined #tripleo | 14:14 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_JSgxtTL/ | 14:14 |
| *** toci-bot has quit IRC | 14:14 | |
| rpodolyaka | shadower: I believe, the idea was to teach tuskar to do this eventually | 14:16 |
| shadower | ah cool | 14:17 |
| *** jergerber has joined #tripleo | 14:17 | |
| * shadower worried there was a fundamental reason why we never wanted to automate this anywhere | 14:17 | |
| *** vkozhukalov has joined #tripleo | 14:20 | |
| *** jcoufal has quit IRC | 14:22 | |
| *** cd-undercloud has joined #tripleo | 14:23 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:23 |
| *** cd-undercloud has quit IRC | 14:23 | |
| *** agrz has joined #tripleo | 14:23 | |
| *** hewbrocc` has joined #tripleo | 14:33 | |
| *** pblaho1 has joined #tripleo | 14:34 | |
| *** pblaho has quit IRC | 14:34 | |
| *** lsmola_ has joined #tripleo | 14:35 | |
| *** hewbrocca has quit IRC | 14:35 | |
| *** lsmola has quit IRC | 14:35 | |
| *** cd-undercloud has joined #tripleo | 14:36 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:36 |
| *** cd-undercloud has quit IRC | 14:36 | |
| *** pblaho1 has quit IRC | 14:37 | |
| *** cd-undercloud has joined #tripleo | 14:49 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 14:49 |
| *** cd-undercloud has quit IRC | 14:49 | |
| *** pblaho has joined #tripleo | 14:50 | |
| *** lsmola has joined #tripleo | 14:51 | |
| *** lsmola_ has quit IRC | 14:53 | |
| *** hewbrocc` has quit IRC | 14:53 | |
| *** hewbrocca has joined #tripleo | 14:54 | |
| *** cd-undercloud has joined #tripleo | 15:02 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:02 |
| *** cd-undercloud has quit IRC | 15:02 | |
| openstackgerrit | A change was merged to openstack/os-apply-config: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59154 | 15:03 |
| *** boris-42 has joined #tripleo | 15:04 | |
| *** agrz2 has joined #tripleo | 15:04 | |
| *** agrz has quit IRC | 15:05 | |
| *** hewbrocc` has joined #tripleo | 15:05 | |
| *** hewbrocca has quit IRC | 15:07 | |
| openstackgerrit | A change was merged to openstack/os-collect-config: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59155 | 15:07 |
| *** agrz2 has quit IRC | 15:07 | |
| *** toci-bot has joined #tripleo | 15:07 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_Wgu0xpA/ | 15:07 |
| *** toci-bot has quit IRC | 15:07 | |
| *** agrz has joined #tripleo | 15:08 | |
| shadower | rpodolyaka: thanks for noticing the universal wheels issue, I missed that when I approved the patches :\ | 15:10 |
| rpodolyaka | shadower: np. The credit must really go to mordred and flaper87 | 15:11 |
| rpodolyaka | shadower: and I finally found time to fix this in tripleo projects before making new releases | 15:11 |
| rpodolyaka | shadower: in fact, it doesn't make any harm right now, but it would, if we started to upload wheels packages to PyPi (and AFAIK, that's what Infra is going to implement soon) | 15:12 |
| shadower | yeah | 15:13 |
| Ng | man, this cd-undercloud issue is super weird | 15:13 |
| rpodolyaka | btw, surprisingly os-refresh-config is working on Python 3 | 15:15 |
| *** cd-undercloud has joined #tripleo | 15:15 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:15 |
| *** cd-undercloud has quit IRC | 15:15 | |
| *** rushiagr has joined #tripleo | 15:17 | |
| derekh | Ng: ya, so the undercloud seems to have lost its metadata that it got from seed on firstboot (havn't looked at it beyond that point) | 15:18 |
| *** athomas has quit IRC | 15:20 | |
| *** cd-undercloud has joined #tripleo | 15:28 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:28 |
| *** cd-undercloud has quit IRC | 15:28 | |
| *** athomas has joined #tripleo | 15:29 | |
| *** rongze has joined #tripleo | 15:29 | |
| mordred | rpodolyaka: whatdid I do? | 15:33 |
| mordred | ah -neat | 15:34 |
| rpodolyaka | mordred: pointed out, that universal = 1 is for py2/py3 in-place compatibility :) | 15:34 |
| *** CaptTofu has quit IRC | 15:34 | |
| mordred | rpodolyaka: oh - should we start gating it on python3? | 15:34 |
| rpodolyaka | mordred: are you talking about os-refresh-config? | 15:34 |
| *** CaptTofu has joined #tripleo | 15:34 | |
| mordred | rpodolyaka: yeah - you said it's workong on python3 - should we add a gate to keep it that way? | 15:35 |
| rpodolyaka | mordred: we could, but it lacks a comprehensive test suite... I just tried to run it using Python 3. So I think, we should write tests first | 15:35 |
| mordred | rpodolyaka: yes. I agree- tests first would be neat | 15:36 |
| *** vkozhukalov has quit IRC | 15:37 | |
| Ng | derekh: hmm, how does it even fetch that? afaik we don't run a metadata server | 15:37 |
| rpodolyaka | shadower: could you approve wheels fixes to tuskar*? | 15:39 |
| shadower | rpodolyaka: on it! | 15:39 |
| rpodolyaka | shadower: thanks! | 15:39 |
| openstackgerrit | A change was merged to openstack/python-tuskarclient: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59163 | 15:41 |
| *** cd-undercloud has joined #tripleo | 15:41 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:41 |
| *** cd-undercloud has quit IRC | 15:41 | |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59161 | 15:42 |
| openstackgerrit | A change was merged to openstack/tuskar: Revert "Support building wheels (PEP-427)" https://review.openstack.org/59158 | 15:43 |
| derekh | Ng: I assumed it came from a seed somewhere | 15:46 |
| derekh | Ng: gonna poke around a bit more | 15:46 |
| Ng | derekh: I *think* that the only layer we inject heat stuff into the image itself, is the seed | 15:49 |
| Ng | but I am embarrasingly bad the heat part of everything | 15:49 |
| derekh | Ng: ya thats correct but the undercloud will have gotten heat metadata from the seed when it first booted | 15:51 |
| derekh | Ng: it rebooted lastnight | 15:51 |
| derekh | Ng: and it doesn't seem to be able to get metadata anymore | 15:51 |
| derekh | so I'm trying to see if I can find the seed now to see if its getting requests | 15:51 |
| Ng | derekh: yeah I was running os-collect-config by hand and it just spun on the metadata IP for a bit, gave up and tried to run with what it had (presumably just cfn-init-data, which is pretty spartan) and immediately os-apply-config explodes | 15:52 |
| derekh | yup | 15:52 |
| Ng | which distracted me for a bit, because the bit that exploded was a netfilter nat rule for talking to a metadata server ;) | 15:53 |
| *** cd-undercloud has joined #tripleo | 15:54 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 15:54 |
| *** cd-undercloud has quit IRC | 15:54 | |
| Ng | hmm | 15:54 |
| Ng | the seed is listening on the port that the nat rule was going to redirect the metadata call to, though | 15:54 |
| *** hewbrocc` has quit IRC | 15:55 | |
| Ng | I still think I'm going down a rabbit hole here, but while I'm doing that, duh, it's -j REDIRECT, so it would be sending metadata calls back to itself | 15:57 |
| Ng | no | 15:57 |
| Ng | well, whatever, this is definitely not relevant | 15:57 |
| Ng | aha, so os-collect-config will be trying to talk to 10.10.16.168 (per /var/lib/heat-cfntools/cfn-metadata-server on the undercloud) | 16:00 |
| Ng | but that IP does not appear to be up | 16:01 |
| derekh | Ng: thats the IP for the seed 10.10.16.168 | 16:03 |
| derekh | but connecting to it from the undercloud doesn't seem to work | 16:05 |
| *** itooon has joined #tripleo | 16:06 | |
| *** itooon has quit IRC | 16:07 | |
| *** cd-undercloud has joined #tripleo | 16:07 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 16:07 |
| *** cd-undercloud has quit IRC | 16:07 | |
| *** bauzas has quit IRC | 16:07 | |
| *** itooon has joined #tripleo | 16:08 | |
| Ng | derekh: hmm, good point | 16:08 |
| Ng | the IP isn't on any of the interfaces of the seed, but it clearly works because that's what I was sshing to | 16:10 |
| Ng | I don't appear to have any sudo powers on the seed host, so I can't go check out its "weird networking shit" to see how it's getting that traffic there | 16:11 |
| *** rdopieralski has quit IRC | 16:19 | |
| *** cd-undercloud has joined #tripleo | 16:20 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 16:20 |
| *** cd-undercloud has quit IRC | 16:20 | |
| *** rushiagr has left #tripleo | 16:23 | |
| *** akuznetsov has quit IRC | 16:23 | |
| derekh | Ng: sorry got distracted with a few bits | 16:27 |
| derekh | Ng: so I have root on the seed | 16:27 |
| openstackgerrit | Ralf Haferkamp proposed a change to openstack/diskimage-builder: Add support for building openSUSE images https://review.openstack.org/56157 | 16:27 |
| Ng | derekh: likewise, I meant on the seet's host | 16:27 |
| derekh | Ng: assuming I'm on the right sedd | 16:27 |
| Ng | (the seed being a VM) | 16:27 |
| derekh | Ng: ahh ok | 16:28 |
| Ng | I tracerouted to 10.10.16.168 on the bastion host and it shows 10.10.16.132 as the first hop, although its routing table suggests it should start with 10.10.16.129 | 16:29 |
| derekh | Ng: I appear to have root on the seeds host (which is 10.10.16.132 I think) | 16:31 |
| derekh | Is the infrastructure written down anywhere? | 16:32 |
| Ng | https://wiki.openstack.org/wiki/TripleO/TripleOCloud | 16:32 |
| *** jistr has quit IRC | 16:32 | |
| Ng | derekh: huh, did you get that via passwordless sudo? | 16:32 |
| *** pblaho has quit IRC | 16:33 | |
| derekh | Ng: yes but there isn't anybody else logged onto this host are you on a different host I wonder | 16:33 |
| *** cd-undercloud has joined #tripleo | 16:34 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 16:34 |
| *** cd-undercloud has quit IRC | 16:34 | |
| Ng | derekh: I had logged out, but I'm back on now | 16:34 |
| Ng | sudo -l asks for my password ;) | 16:34 |
| derekh | Ng: adding you to the sudo group | 16:35 |
| Ng | ta | 16:35 |
| derekh | Ng try now | 16:37 |
| Ng | derekh: hrm, nope. to be sure, this machine shows up as freecloud0039, right? | 16:38 |
| Ng | oh | 16:38 |
| Ng | hang on, I bet this is my ssh config | 16:38 |
| Ng | damn control sockets | 16:38 |
| Ng | derekh: working, thanks :) | 16:38 |
| SpamapS | Is anyone working on this | 16:38 |
| SpamapS | + nova list | 16:38 |
| SpamapS | ERROR: Expecting a password provided via either --os-password, env[OS_PASSWORD], or prompted response | 16:38 |
| SpamapS | ? | 16:38 |
| Ng | SpamapS: yep, the undercloud was rebooted last night and now isn't getting its heat data | 16:39 |
| Ng | we're, for want of a better word, flailing around trying to figure out why it can't talk to the cfn-metadata-server it has in /var/lib/heat-cfntools | 16:39 |
| derekh | SpamapS: so basically /root/stackrc isn't populated with the password | 16:39 |
| *** sdake_ has joined #tripleo | 16:40 | |
| *** sdake_ has joined #tripleo | 16:40 | |
| SpamapS | oh | 16:40 |
| SpamapS | I'm guessing that lifeless just editted cfn-init-data in the past | 16:41 |
| Ng | that seems unlikely I think, cfn-init-data has an mtime of 30th September | 16:43 |
| Ng | same as the other cfn-files in there | 16:43 |
| SpamapS | Ng: ah, so is cfn failing on the undercloud box? | 16:44 |
| SpamapS | Didn't we hear that the seed had been turned off because of evil DHCP? | 16:44 |
| SpamapS | Ng: ok right so perhaps _now we should edit it_ | 16:44 |
| SpamapS | oh hmm.. os-collect-config seems to be very old | 16:46 |
| SpamapS | doesn't save its state locally | 16:46 |
| Ng | SpamapS: The seed is definitely still running, fwiw | 16:46 |
| Ng | If there was talk of switching it off, I missed it | 16:46 |
| *** cd-undercloud has joined #tripleo | 16:47 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 16:47 |
| *** cd-undercloud has quit IRC | 16:47 | |
| SpamapS | Ng: maybe just dnsmasq was turned off? | 16:52 |
| SpamapS | or it has the iptables thing to stop it from being evil? | 16:52 |
| SpamapS | Ng: the connection to 169.254.16.254 seems to suggest that there is a problem with talking to the seed. | 16:53 |
| SpamapS | the timeout rather | 16:53 |
| Ng | SpamapS: hrm, I thought we didn't use the 169.254 thing | 16:54 |
| Ng | cfn-metadata-server has 10.10.16.168, which is the seed | 16:54 |
| Ng | but I am just feeling my way through trying to find what *should* be happening, so anything I say is probably wrong at the moment :) | 16:55 |
| *** CaptTofu has quit IRC | 16:56 | |
| *** CaptTofu has joined #tripleo | 16:56 | |
| SpamapS | Ng: right but 169.xxx is the ec2 bit.. we use that too. :p | 16:56 |
| Ng | SpamapS: huh, ok, for some reason I thought we didn't | 16:57 |
| SpamapS | [Errno 113] No route to host | 16:57 |
| SpamapS | Ng: when trying to connect to the seed | 16:57 |
| SpamapS | ? (10.10.16.168) at <incomplete> on eth2 | 16:57 |
| Ng | SpamapS: yep, it can't talk to the seed via 10.10 or 169.254 | 16:58 |
| *** cd-undercloud has joined #tripleo | 17:00 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:00 |
| *** cd-undercloud has quit IRC | 17:00 | |
| *** ifarkas has quit IRC | 17:01 | |
| SpamapS | Ng: ok, so that seems like "the problem" | 17:02 |
| Ng | SpamapS: that is my working assumption atm, but I'm not yet sure why it's broken | 17:03 |
| SpamapS | Ng: can you get to 10.10.16.168 from bastion? | 17:03 |
| Ng | SpamapS: yep | 17:04 |
| SpamapS | Ng: I can't ssh there but yeah I see it is reachable | 17:04 |
| Ng | you ought to be able to ssh in (as root) | 17:04 |
| Ng | I am lies, your ssh key isn't there | 17:05 |
| Ng | gah, I am still lies, your MacBookAir key is | 17:05 |
| SpamapS | oh ok I'm not trying from the air | 17:07 |
| Ng | it's a bit of a mindfuck that the 10.10.16/24 range is actually split across physical and virtual networking | 17:07 |
| SpamapS | well today is a holiday so I think I'll bug out | 17:08 |
| Ng | :) | 17:08 |
| SpamapS | Ng: seems like getting those two talking will fix it though. | 17:08 |
| SpamapS | Ng: or you can fudge it by editting cfn-init-data | 17:09 |
| *** toci-bot has joined #tripleo | 17:13 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_c8qyZYV/ | 17:13 |
| *** toci-bot has quit IRC | 17:13 | |
| *** cd-undercloud has joined #tripleo | 17:14 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:14 |
| *** cd-undercloud has quit IRC | 17:14 | |
| *** agrz has quit IRC | 17:18 | |
| *** markmc has quit IRC | 17:21 | |
| *** cd-undercloud has joined #tripleo | 17:27 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:27 |
| *** cd-undercloud has quit IRC | 17:27 | |
| *** sdake_ has quit IRC | 17:37 | |
| *** max_lobur is now known as max_lobur_afk | 17:38 | |
| *** cd-undercloud has joined #tripleo | 17:40 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:40 |
| *** cd-undercloud has quit IRC | 17:40 | |
| *** bauzas has joined #tripleo | 17:43 | |
| *** cd-undercloud has joined #tripleo | 17:52 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 17:52 |
| *** cd-undercloud has quit IRC | 17:52 | |
| *** vkozhukalov has joined #tripleo | 17:53 | |
| Ng | interesting data point, the undercloud's neutron port shows as DOWN on the seed | 17:55 |
| *** athomas has quit IRC | 18:01 | |
| *** martyntaylor has quit IRC | 18:01 | |
| *** CaptTofu has quit IRC | 18:03 | |
| *** derekh has quit IRC | 18:03 | |
| *** CaptTofu has joined #tripleo | 18:03 | |
| *** max_lobur_afk is now known as max_lobur | 18:04 | |
| *** cd-undercloud has joined #tripleo | 18:06 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:06 |
| *** cd-undercloud has quit IRC | 18:06 | |
| *** jergerber has quit IRC | 18:15 | |
| *** akuznetsov has joined #tripleo | 18:19 | |
| *** CaptTofu has quit IRC | 18:19 | |
| *** cd-undercloud has joined #tripleo | 18:20 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:20 |
| *** cd-undercloud has quit IRC | 18:20 | |
| *** CaptTofu has joined #tripleo | 18:20 | |
| * Ng dinners, back later | 18:23 | |
| Ng | I'm currently thinking the issue is networking on the undercloud machine, tcpdump -i any is showing that some packets are arriving for 10.10.16.169 and not being noticed by the higher layers | 18:24 |
| Ng | ruled out iptables so far | 18:24 |
| *** max_lobur is now known as max_lobur_afk | 18:27 | |
| *** rongze has quit IRC | 18:29 | |
| *** cd-undercloud has joined #tripleo | 18:32 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:32 |
| *** cd-undercloud has quit IRC | 18:32 | |
| *** lucasagomes has quit IRC | 18:35 | |
| *** rongze has joined #tripleo | 18:38 | |
| *** cd-undercloud has joined #tripleo | 18:44 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:44 |
| *** cd-undercloud has quit IRC | 18:44 | |
| *** pleia2_ is now known as pleia2 | 18:48 | |
| *** yanhcdl has quit IRC | 18:50 | |
| *** cd-undercloud has joined #tripleo | 18:57 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 18:57 |
| *** cd-undercloud has quit IRC | 18:57 | |
| *** CaptTofu has quit IRC | 18:58 | |
| *** CaptTofu has joined #tripleo | 18:58 | |
| *** yanhcdl has joined #tripleo | 19:03 | |
| *** toci-bot has joined #tripleo | 19:08 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_7LXGpQi/ | 19:08 |
| *** toci-bot has quit IRC | 19:08 | |
| *** cd-undercloud has joined #tripleo | 19:11 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:11 |
| *** cd-undercloud has quit IRC | 19:11 | |
| *** CaptTofu has quit IRC | 19:12 | |
| *** CaptTofu has joined #tripleo | 19:12 | |
| *** rongze has quit IRC | 19:16 | |
| *** cd-undercloud has joined #tripleo | 19:24 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:24 |
| *** cd-undercloud has quit IRC | 19:24 | |
| *** rongze has joined #tripleo | 19:34 | |
| *** cd-undercloud has joined #tripleo | 19:37 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:37 |
| *** cd-undercloud has quit IRC | 19:37 | |
| *** jergerber has joined #tripleo | 19:40 | |
| *** rongze has quit IRC | 19:41 | |
| *** CaptTofu has quit IRC | 19:41 | |
| *** CaptTofu has joined #tripleo | 19:42 | |
| *** cd-undercloud has joined #tripleo | 19:50 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 19:50 |
| *** cd-undercloud has quit IRC | 19:50 | |
| *** sdake_ has joined #tripleo | 20:01 | |
| *** sdake_ has joined #tripleo | 20:01 | |
| *** cd-undercloud has joined #tripleo | 20:03 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:03 |
| *** cd-undercloud has quit IRC | 20:03 | |
| *** rongze has joined #tripleo | 20:11 | |
| *** boris-42 has quit IRC | 20:13 | |
| *** cd-undercloud has joined #tripleo | 20:17 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:17 |
| *** cd-undercloud has quit IRC | 20:17 | |
| lifeless | morning | 20:18 |
| Ng | oh for crying out loud | 20:19 |
| Ng | hey lifeless | 20:19 |
| lifeless | You must provide a password via either --os-password or env[OS_PASSWORD] | 20:19 |
| lifeless | is that a furfy? | 20:19 |
| *** rongze has quit IRC | 20:20 | |
| Ng | lifeless: what hte heck is a furfy? :) | 20:20 |
| lifeless | irrelevant distraction from the situation at hand | 20:20 |
| Ng | I don't think so, if the current run succeeds then I would expect to not see that message | 20:21 |
| lifeless | what was changed in the current run ? | 20:21 |
| Ng | and I will go and kick something very hard for not spotting the problem *hours* sooner | 20:21 |
| Ng | lifeless: putting 10.10.16.169 on br-ctlplane (and having that interface actually be up) instead of directly on eth2 | 20:22 |
| Ng | prior to that change, seed and undercloud had no IP connectivity | 20:22 |
| Ng | I was clawing my way through netfilter, neutron and ovs trying to figure out why, didn't spot until just now that br-ctlplane on the undercloud was down | 20:23 |
| Ng | (I was thrown by tcpdump on undercloud seeing packets arriving, but userspace wasn't receiving them, so I assumed it was some of the fancy networking stuff) | 20:23 |
| lifeless | I hate linux networking sometime | 20:24 |
| lifeless | how does that cause cause the OS_PASSWORD warning | 20:24 |
| lifeless | are we missing a -e somewhere? | 20:24 |
| Ng | not as such, that comes from /root/stackrc which is written by heat. os-collect-config couldn't fetch metadata from the seed, so the templates were all empty == no OS_PASSWORD | 20:25 |
| Ng | or rather, the keys the templates were all looking for, were undefined | 20:26 |
| Ng | and we write the stackrc with a default value for that key, of '' | 20:26 |
| Ng | (of course, in a couple of minutes the bot will pop up and show the same failure and everything I've just said will be irrelevant ;) | 20:26 |
| *** rushiagr has joined #tripleo | 20:26 | |
| *** vipul-away is now known as vipul | 20:27 | |
| Ng | doh, os-collect-config isn't running on undercloud, from earlier poking. this run is not going to go well ;) | 20:28 |
| Ng | lifeless: oh, actually, one question if you have time, do we actually use the ec2 metadata service? | 20:29 |
| *** cd-undercloud has joined #tripleo | 20:30 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 20:30 |
| *** cd-undercloud has quit IRC | 20:30 | |
| Ng | yes yes, thank you bot | 20:30 |
| *** rongze has joined #tripleo | 20:46 | |
| *** sdake_ has quit IRC | 20:47 | |
| lifeless | Ng: sorry, am back | 20:51 |
| lifeless | Ng: so the undercloud was failing to talk to the seed? | 20:51 |
| lifeless | the undercloud seed, I mean ? | 20:52 |
| lifeless | Ng: we do | 20:52 |
| *** rongze has quit IRC | 20:52 | |
| Ng | lifeless: that is correct | 20:52 |
| lifeless | Ng: it bootstrapps the config for talking to heat | 20:52 |
| lifeless | but we shouldn't need it again once bootstrapped | 20:52 |
| Ng | ahh | 20:52 |
| Ng | so that's how the cfn-metadata-server gets discovered? | 20:52 |
| lifeless | yes | 20:52 |
| lifeless | + ssh key loading | 20:53 |
| lifeless | + hostname | 20:53 |
| Ng | ok, well that networking may well still be broken, os-collect-config is failing to talk to the ec2 provider | 20:53 |
| Ng | but the cfn one is succeeding | 20:53 |
| lifeless | on the u/c ? should be fine | 20:53 |
| lifeless | so what is sourcing stackrc? | 20:54 |
| lifeless | as in | 20:54 |
| Ng | tripleo-cd | 20:54 |
| Ng | .sh | 20:54 |
| lifeless | we shouldn't be using any credentials from it | 20:54 |
| Ng | I believe it's necessary for devtest_overcloud.sh to succeed? | 20:54 |
| lifeless | shouldn't be | 20:55 |
| lifeless | hmmm | 20:55 |
| lifeless | oh, it contains the undercloud creds | 20:55 |
| lifeless | <- confused | 20:55 |
| Ng | indeed | 20:55 |
| lifeless | so yeah, makes sense | 20:55 |
| Ng | so we were failing at the first load-image in devtest_overcloud.sh | 20:55 |
| lifeless | we should add an assert in devtest_overcloud that it has creds | 20:55 |
| * Ng files a bug | 20:56 | |
| lifeless | heh, file a patch | 20:56 |
| lifeless | same work | 20:56 |
| Ng | fine! | 20:56 |
| Ng | lifeless: reckon it's worth doing the same in devtest_undercloud? | 20:57 |
| *** marun has quit IRC | 20:57 | |
| *** cd-undercloud has joined #tripleo | 21:02 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 21:02 |
| *** cd-undercloud has quit IRC | 21:02 | |
| Ng | hmm | 21:02 |
| Ng | that one failed by timing out for the overcloud to be deleted | 21:02 |
| Ng | I wonder where that overcloud even came from, maybe it's vestigial from the previous boot?) | 21:02 |
| openstackgerrit | Chris Jones proposed a change to openstack/tripleo-incubator: Assert undercloud auth before building overcloud. https://review.openstack.org/59237 | 21:08 |
| Ng | right, I need a drink, then I will dig into nova on the undercloud and figure out what has it so confused about the stale overcloud instances | 21:09 |
| *** toci-bot has joined #tripleo | 21:13 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_P5xKz4I/ | 21:13 |
| *** toci-bot has quit IRC | 21:13 | |
| lifeless | Ng: okies | 21:14 |
| lifeless | | 3d66840d-6765-4a75-be33-63415b18e12d | overcloud-NovaCompute0-5npxvfqrg2pk | ACTIVE | deleting | Running | ctlplane=10.10.16.171 | | 21:14 |
| lifeless | | 3b13e7ed-fad1-4bed-a6ac-63288a268928 | overcloud-notcompute-zlnakd4qbiwm | ACTIVE | deleting | Running | ctlplane=10.10.16.174 | | 21:14 |
| lifeless | Ng: ^ that to me looks like the bm ops timed out | 21:14 |
| lifeless | Ng: I'd restart the nova-compute process | 21:15 |
| Ng | start: Job failed to start | 21:16 |
| Ng | ooh | 21:16 |
| Ng | OperationalError: (OperationalError) (1054, "Unknown column 'bm_nodes.ephemeral_mb' in 'field list'") | 21:23 |
| Ng | maybe we have a schema update to apply | 21:23 |
| Ng | right, nova-baremetal db sync fixed that up | 21:29 |
| lifeless | ahh | 21:31 |
| lifeless | foot gunned us I did | 21:31 |
| Ng | and now all the stale overcloud stuff is gone from nova/heat. hopefully the next run works :) | 21:31 |
| Ng | lifeless: should we maybe call a bunch of db sync commands after the refresh-env in tripleo-cd.sh? | 21:32 |
| lifeless | Ng: no | 21:32 |
| lifeless | Ng: because a) layers [tripleo-cd depends on the undercloud today, it doesn't maintain it] | 21:33 |
| lifeless | b) we do it in os-refresh-config hooks already, but we haven't udpated theimage on the undercloud for 60 days | 21:33 |
| Ng | aha, makes sense | 21:33 |
| *** rongze has joined #tripleo | 21:48 | |
| *** rushiagr has quit IRC | 21:48 | |
| *** marun has joined #tripleo | 21:54 | |
| *** rongze has quit IRC | 21:54 | |
| *** panda has quit IRC | 21:56 | |
| *** rpodolyaka1 has joined #tripleo | 22:03 | |
| *** cd-undercloud has joined #tripleo | 22:03 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 22:03 |
| *** cd-undercloud has quit IRC | 22:03 | |
| Ng | oh seriously | 22:05 |
| *** sdake_ has joined #tripleo | 22:12 | |
| *** rpodolyaka1 has quit IRC | 22:12 | |
| *** vkozhukalov has quit IRC | 22:18 | |
| *** bauzas has quit IRC | 22:19 | |
| *** ccrouch has joined #tripleo | 22:45 | |
| *** rongze has joined #tripleo | 22:50 | |
| *** rongze has quit IRC | 22:56 | |
| *** michchap_ has quit IRC | 22:56 | |
| *** michchap has joined #tripleo | 22:57 | |
| *** cd-undercloud has joined #tripleo | 23:03 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 23:03 |
| *** cd-undercloud has quit IRC | 23:03 | |
| Ng | lifeless: I'm out of steam for the day I'm afraid and we're not back on track yet, the overcloud isn't coming up properly atm | 23:13 |
| *** toci-bot has joined #tripleo | 23:14 | |
| toci-bot | ERROR during toci run, see http://54.228.118.193/toci/toci_logs_pGdjxmU/ | 23:14 |
| *** toci-bot has quit IRC | 23:14 | |
| *** jpeeler has quit IRC | 23:16 | |
| *** ccrouch has quit IRC | 23:43 | |
| *** rongze has joined #tripleo | 23:52 | |
| *** rongze has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!