Saturday, 2014-02-15

*** edmund has joined #tripleo00:12
*** killer_prince has joined #tripleo00:12
*** ramishra has joined #tripleo00:20
*** cd-undercloud has joined #tripleo00:24
cd-undercloud************** overcloud complete status=1 ************00:24
*** cd-undercloud has quit IRC00:24
*** ramishra has quit IRC00:25
*** cadenzajon has quit IRC00:29
*** cwolferh has quit IRC00:34
*** lifeless has quit IRC00:40
*** peoplemerge has quit IRC00:45
*** ramishra has joined #tripleo00:51
*** tzumainn has quit IRC01:10
*** ramishra has quit IRC01:15
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Nova-baremetal support for openSUSE  https://review.openstack.org/7202101:26
*** cd-undercloud has joined #tripleo01:45
cd-undercloud************** overcloud complete status=1 ************01:45
*** cd-undercloud has quit IRC01:45
*** ramishra has joined #tripleo01:46
*** ramishra has quit IRC01:51
*** wusWWW has quit IRC02:14
*** lifeless has joined #tripleo02:23
*** ChanServ sets mode: +v lifeless02:23
*** newell has quit IRC02:29
lifelessekarlso: ok, I'm confused. Start over :)02:36
*** CaptTofu has joined #tripleo02:40
*** ramishra has joined #tripleo02:47
*** ramishra has quit IRC02:52
jog0lifeless: just wrote my first testtools pull request02:55
lifelessjog0: cool02:56
jog0I was shocked to not see tox02:56
jog0and see make instead02:56
lifelesstox is meh02:56
lifelessI know openstack is in love with it02:56
lifelessbut I'm yet to understand why02:57
* jog0 blames mordred 02:57
*** cd-undercloud has joined #tripleo03:06
cd-undercloud************** overcloud complete status=1 ************03:06
*** cd-undercloud has quit IRC03:06
*** noslzzp has quit IRC03:19
*** julim has quit IRC03:44
*** ramishra has joined #tripleo03:48
*** ramishra has quit IRC03:52
openstackgerritLee Li proposed a change to openstack/tuskar: Replace assertEqual(None, *) with assertIsNone in tests  https://review.openstack.org/7377404:10
*** cd-undercloud has joined #tripleo04:29
cd-undercloud************** overcloud complete status=1 ************04:29
*** cd-undercloud has quit IRC04:29
*** e0ne has joined #tripleo04:30
*** e0ne has quit IRC04:31
*** ramishra has joined #tripleo04:32
*** ramishra has quit IRC04:42
*** ramishra has joined #tripleo05:13
*** ramishra has quit IRC05:17
*** spzala has quit IRC05:36
*** cd-undercloud has joined #tripleo05:49
cd-undercloud************** overcloud complete status=1 ************05:49
*** cd-undercloud has quit IRC05:49
*** ramishra has joined #tripleo06:14
*** ramishra has quit IRC06:18
*** cadenzajon has joined #tripleo06:25
*** saju_m has joined #tripleo06:27
*** cadenzajon has quit IRC06:32
*** ramishra has joined #tripleo06:32
*** ramishra has quit IRC06:33
*** ramishra has joined #tripleo06:40
*** ramishra has quit IRC06:45
*** saju_m has quit IRC06:51
*** AaronGr is now known as AaronGr_Zzz06:54
*** ramishra has joined #tripleo07:02
*** ramishra has quit IRC07:09
*** akuznetsov has joined #tripleo07:10
*** cd-undercloud has joined #tripleo07:11
cd-undercloud************** overcloud complete status=1 ************07:11
*** cd-undercloud has quit IRC07:11
*** akuznetsov has quit IRC07:19
*** ramishra has joined #tripleo07:22
*** rcarrillocruz1 has joined #tripleo07:23
*** rcarrillocruz has quit IRC07:25
*** ramishra has quit IRC07:44
SpamapSlooks like our keystone is too old for Heat to be happy because its v3 API is broken07:53
*** ramishra has joined #tripleo07:53
*** rcarrillocruz has joined #tripleo07:58
* SpamapS updates it... all is well again07:58
*** rcarrillocruz1 has quit IRC07:59
lifelessSpamapS: yay08:11
SpamapSlifeless: our keystone was from Sep 2908:12
SpamapSv3 was fixed about 3 weeks ago in stable/havana.. ;)08:12
SpamapSwhich led to v3-only landing in Heat08:13
SpamapSwhich led to our heat-api-cfn being 100% broken until just now :-P08:13
SpamapSalso bonus we got my faster queries ;)08:13
*** vkozhukalov has joined #tripleo08:14
*** vkozhukalov has quit IRC08:21
*** ramishra has quit IRC08:22
*** rcarrillocruz1 has joined #tripleo08:31
*** rcarrillocruz has quit IRC08:32
*** ramishra has joined #tripleo08:49
lifelessSpamapS: cool08:51
lifelessSpamapS: though 32G is 32G08:51
*** cd-undercloud has joined #tripleo08:52
cd-undercloud************** overcloud complete status=1 ************08:52
*** cd-undercloud has quit IRC08:52
openstackgerrityangxurong proposed a change to openstack/tuskar: Don't raise MySQL 2013 'Lost connection' errors  https://review.openstack.org/7379909:12
*** killer_prince has quit IRC09:21
*** akuznetsov has joined #tripleo09:24
*** ramishra has quit IRC09:31
*** cd-undercloud has joined #tripleo09:43
cd-undercloud************** overcloud complete status=1 ************09:43
*** cd-undercloud has quit IRC09:43
SpamapSTiming out after 300 seconds:09:48
SpamapSCOMMAND=neutron agent-list -f csv -c alive -c agent_type -c host | grep ":-).*Open vSwitch agent.*overcloud-novacompute"09:48
SpamapSwrong string in there09:49
SpamapSno those are right. hm09:52
SpamapS| 8be66142-6483-4910-8a56-2d355302f459 | Open vSwitch agent | overcloud-novacompute1-v6egsez2wipx | xxx   | True           |09:52
*** rcarrillocruz1 has quit IRC09:53
*** rcarrillocruz has joined #tripleo09:53
SpamapS2014-02-15 09:37:56.480 3000 TRACE neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by peer09:53
SpamapSopenvswitch agent looks like it is confused09:53
SpamapSrunning, but not actually working. :-/09:54
SpamapS2014-02-15 05:05:15.739 2911 CRITICAL neutron [-] Unexpected boolean value ''09:55
SpamapSeven weirder09:55
*** ramishra has joined #tripleo09:57
SpamapSlibvirt_type=;NovaComputeLibvirtType=kvm09:58
*** vipul has quit IRC10:04
*** vipul has joined #tripleo10:05
SpamapSoh well.. it's saturday :)10:10
lifelesshah10:19
lifelessthat might be why the overcloudis failing10:20
lifelesswe haven't copied tripleo-cd into place in a wile10:20
lifelessdone10:22
lifelessbut no, OVERCLOUD_LIBVIRT_TYPE="kvm"10:23
lifelessperhaps in a rc file ?10:23
openstackgerritDirk Mueller proposed a change to openstack/tripleo-image-elements: Nova-baremetal support for openSUSE  https://review.openstack.org/7202110:32
*** cd-undercloud has joined #tripleo10:34
cd-undercloud************** overcloud complete status=1 ************10:34
*** cd-undercloud has quit IRC10:34
*** ramishra_ has joined #tripleo10:35
*** ramishra has quit IRC10:36
*** ramishra_ has quit IRC10:40
*** ramishra has joined #tripleo10:41
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Fix up overcloud ci jobs  https://review.openstack.org/7380410:43
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Fix up the overcloud ci job  https://review.openstack.org/7380410:44
*** rcarrillocruz1 has joined #tripleo10:45
*** ramishra has quit IRC10:45
*** rcarrillocruz has quit IRC10:47
*** ramishra has joined #tripleo10:50
*** ramishra has quit IRC11:08
*** cd-undercloud has joined #tripleo11:24
cd-undercloud************** overcloud complete status=1 ************11:24
*** cd-undercloud has quit IRC11:24
lifelessSpamapS: /tmp/tmpzuS3bc/build/tripleo-image-elements/tripleo_image_elements-0.5.1.data/data/share/tripleo-image-elements/tripleo-cd/tripleo-cd.sh:    OVERCLOUD_LIBVIRT_TYPE=";NovaComputeLibvirtType=kvm" \11:28
lifelessbut thats not the copy we'd run...11:28
lifelessSpamapS: the agent-list command is correct, but I don't understand why we have to wait fror that agent...11:31
*** killer_prince has joined #tripleo11:40
*** rcarrillocruz has joined #tripleo11:47
*** rcarrillocruz1 has quit IRC11:49
*** rcarrillocruz1 has joined #tripleo12:03
*** rcarrillocruz has quit IRC12:04
*** rcarrillocruz has joined #tripleo12:08
*** rcarrillocruz1 has quit IRC12:10
*** akuznetsov has quit IRC12:16
*** akuznetsov has joined #tripleo12:16
*** d0ugal has quit IRC12:20
*** d0ugal has joined #tripleo12:20
*** akuznetsov has quit IRC12:27
*** cd-undercloud has joined #tripleo12:27
cd-undercloud************** overcloud complete status=1 ************12:27
*** cd-undercloud has quit IRC12:27
*** rcarrillocruz1 has joined #tripleo12:49
*** rcarrillocruz has quit IRC12:50
*** killer_prince has quit IRC12:53
*** killer_prince has joined #tripleo12:54
*** CaptTofu has quit IRC12:59
*** CaptTofu has joined #tripleo12:59
*** akuznetsov has joined #tripleo13:03
*** rcarrillocruz has joined #tripleo13:03
*** CaptTofu has quit IRC13:04
*** rcarrillocruz1 has quit IRC13:06
*** ramishra has joined #tripleo13:13
*** rcarrillocruz1 has joined #tripleo13:17
*** rcarrillocruz has quit IRC13:17
*** cd-undercloud has joined #tripleo13:18
cd-undercloud************** overcloud complete status=1 ************13:18
*** cd-undercloud has quit IRC13:18
*** killer_prince has quit IRC13:20
*** ramishra has quit IRC13:31
*** ramishra has joined #tripleo13:33
*** rcarrillocruz has joined #tripleo13:39
*** rcarrillocruz1 has quit IRC13:41
*** hewbrocca has quit IRC13:43
*** ramishra has quit IRC13:44
*** saju_m has joined #tripleo13:46
*** ramishra has joined #tripleo13:49
*** ramishra has quit IRC13:54
*** ramishra has joined #tripleo13:56
*** ramishra_ has joined #tripleo14:01
*** ramishra has quit IRC14:01
*** cd-undercloud has joined #tripleo14:09
cd-undercloud************** overcloud complete status=1 ************14:09
*** cd-undercloud has quit IRC14:09
*** rcarrillocruz1 has joined #tripleo14:20
*** rcarrillocruz has quit IRC14:22
*** rcarrillocruz has joined #tripleo14:31
*** rcarrillocruz1 has quit IRC14:33
*** cd-undercloud has joined #tripleo15:00
cd-undercloud************** overcloud complete status=1 ************15:00
*** cd-undercloud has quit IRC15:00
*** akuznetsov has quit IRC15:04
*** ramishra_ has quit IRC15:11
*** akuznetsov has joined #tripleo15:22
*** akuznetsov has quit IRC15:31
*** akuznetsov has joined #tripleo15:35
*** ramishra has joined #tripleo15:41
ekarlsolifeless: i'm trying to use tripleo on a cloud instance15:45
*** ramishra has quit IRC15:46
*** Qiming has joined #tripleo15:48
*** rcarrillocruz1 has joined #tripleo15:55
*** rcarrillocruz has quit IRC15:56
*** StevenD57 is now known as Lotus907efi16:15
*** akuznetsov has quit IRC16:17
*** cd-undercloud has joined #tripleo16:22
cd-undercloud************** overcloud complete status=1 ************16:22
*** cd-undercloud has quit IRC16:22
openstackgerritDan Prince proposed a change to openstack/tripleo-image-elements: Set default_ephemeral_format to ext4 for seed  https://review.openstack.org/7382416:23
*** rcarrillocruz has joined #tripleo16:29
*** ramishra has joined #tripleo16:30
*** rcarrillocruz1 has quit IRC16:31
*** akuznetsov has joined #tripleo16:44
*** rcarrillocruz1 has joined #tripleo16:48
*** rcarrillocruz has quit IRC16:50
*** Lotus907efi is now known as Lotus907efi_16:55
*** AaronGr_Zzz is now known as AaronGr17:19
*** rcarrillocruz has joined #tripleo17:22
*** rcarrillocruz1 has quit IRC17:25
*** Qiming has quit IRC17:26
*** cd-undercloud has joined #tripleo17:44
cd-undercloud************** overcloud complete status=1 ************17:44
*** cd-undercloud has quit IRC17:44
*** ramishra has quit IRC18:02
*** ramishra has joined #tripleo18:07
*** ramishra has quit IRC18:14
*** rcarrillocruz1 has joined #tripleo18:20
*** rcarrillocruz has quit IRC18:22
*** bogdando has quit IRC18:22
*** max_lobur_afk has quit IRC18:23
*** rcarrillocruz1 has quit IRC18:25
*** rcarrillocruz has joined #tripleo18:33
*** rcarrillocruz1 has joined #tripleo18:36
*** rcarrillocruz has quit IRC18:37
*** rcarrillocruz has joined #tripleo18:40
*** rcarrillocruz1 has quit IRC18:41
*** boris-42_ has quit IRC18:46
*** e0ne has joined #tripleo18:48
*** e0ne has quit IRC18:49
*** boris-42_ has joined #tripleo18:52
*** saju_m has quit IRC18:56
*** NearlyFunctiona1 has quit IRC18:58
*** NearlyFunctional has joined #tripleo18:59
*** cd-undercloud has joined #tripleo19:05
cd-undercloud************** overcloud complete status=1 ************19:05
*** cd-undercloud has quit IRC19:05
*** ramishra has joined #tripleo19:09
*** ramishra has quit IRC19:22
*** rcarrillocruz has quit IRC19:25
*** AaronGr is now known as AaronGr_Zzz19:28
lifelessSpamapS: https://review.openstack.org/#/c/73465/ needs a +A19:48
lifelessSpamapS: and https://review.openstack.org/#/c/73804/ too19:49
*** cody-somerville has quit IRC19:58
*** jcooley_ has joined #tripleo20:09
openstackgerritA change was merged to openstack/tripleo-incubator: Stop trying to copy out SSH keys from notcompute.  https://review.openstack.org/7346520:12
openstackgerritA change was merged to openstack-infra/tripleo-ci: Fix up the overcloud ci job  https://review.openstack.org/7380420:12
SpamapSlifeless: regarding the agent, if we don't wait for the agent, binding will fail. IMO it is a nova bug that I haven't gotten around to fixing (should not be available to schedule instances to until we have L2)20:13
*** ramishra has joined #tripleo20:13
*** ramishra has quit IRC20:18
lifelessSpamapS: have we even filed it ?20:18
lifelessSpamapS: point is we wait for one agent. We need to wait for them all right?20:19
lifelessSpamapS: or our test is invalid20:19
*** cd-undercloud has joined #tripleo20:26
cd-undercloud************** overcloud complete status=1 ************20:26
*** cd-undercloud has quit IRC20:26
*** akuznetsov has quit IRC20:50
*** akuznetsov has joined #tripleo20:50
*** cody-somerville has joined #tripleo21:06
lifelesshmm21:11
lifeless2014-02-15 20:44:31.258 | Waiting for keystone to initialise...21:11
lifeless2014-02-15 20:44:31.587 | Unable to establish connection to http://192.0.2.1:35357/v2.0/tenants21:11
lifelessworrisome21:12
*** ramishra has joined #tripleo21:14
*** ramishra has quit IRC21:19
lifelesseither a routing issue or failed seed build21:19
*** hewbrocca has joined #tripleo21:26
*** AaronGr_Zzz is now known as AaronGr21:45
*** cd-undercloud has joined #tripleo21:48
cd-undercloud************** overcloud complete status=1 ************21:48
*** cd-undercloud has quit IRC21:48
*** AaronGr is now known as AaronGr_Zzz21:52
*** akuznetsov has quit IRC21:54
*** derekh has joined #tripleo21:55
derekhthose ci keystone failures seem to be a keystone regression21:56
derekhkeystone log file contains21:56
derekh2014-02-15 21:00:50.386 10854 CRITICAL keystone [-] ConfigFileNotFound: The Keystone configuration file keystone-paste.ini could not be found.21:56
derekhthis commit looks suspicious https://review.openstack.org/#/c/73621/221:57
SpamapSlifeless: indeed, we'd need to wait for them all. No I don't think it is filed as such.. though I bet there is a race condition bug open somewhere to address the issue.22:00
SpamapSlifeless: I think the reason devstack doesn't hit it is that nova-compute and neutron-openvswitch-agent start up in one box with rabbit local.. with our case, we have things happening in parallel and thus are more likely to hit the problem.22:01
*** ramishra has joined #tripleo22:15
*** ramishra has quit IRC22:19
openstackgerritDerek Higgins proposed a change to openstack/tripleo-incubator: Switch default nic driver to virtio  https://review.openstack.org/7383722:29
devanandaderekh: i thought we needed e1000 or things broke?22:33
derekhdevananda: seems to be ok in my tests, the problem I believe was that ubuntu precise had an old version of dhclient that ognored dhcp packets that didn't have a checksum22:34
derekhdevananda: we no longer target that image in tripleo22:34
devanandahmm, k22:35
lifelessSpamapS: yes22:40
lifelessderekh: ah, thanks for spotting that22:40
lifelessderekh: I thought we fixed it by using ovs and not virtio - we get correct checksums22:40
lifelessderekh: I'm fairly sure 'ignore bad checksums' is not a feature in dhclient ;)22:41
derekhlifeless: we hit this about a year ago running ubuntu on rhel https://bugzilla.redhat.com/show_bug.cgi?id=910619#c622:42
lifelessderekh: and ubuntu on ubuntu22:46
lifelessderekh: you still need checksum-fill mangle rules on virbr0 for lxc for instance22:46
derekhlifeless: ok, but we don't need it for the images we support so where good ?22:48
derekh*we're22:48
lifelessderekh: I'd like to see a wider test matrix22:48
lifelessderekh: at the moment in CI we're pushing deeper not wider22:48
SpamapS| be0e3cbf-1d69-4d7b-9b11-db70dd72f6a0 | overcloud      | UPDATE_FAILED   | 2014-02-15T08:23:43Z |22:49
SpamapSsadness22:49
SpamapShave to delete it. :-/22:49
* SpamapS ponders trying out the abandon/adopt approach.. but.. not on Saturday22:50
derekhlifeless: ok, fair enough its been configurable for a few months now which should have given us a little bit of a wider matrix (although probably didn't)22:50
lifelessderekh: I've I've commented a cople of times on https://review.openstack.org/#/c/73837/22:50
SpamapSlifeless: btw, just a point that might interest you.. seems we're creating about 10,000 tokens per hour. You can see this by grepping 'cleanup-keystone-tokens' in /var/log/syslog22:51
lifelessSpamapS: doesn't surprise me22:51
SpamapSlifeless: we're also deleting 10,000 per hour .. and it takes about 7 seconds. :-P22:51
lifelessderekh: basically, I think we need a wider test matrix22:51
SpamapS    Data_length: 108637388822:51
SpamapS   Index_length: 2121728022:51
lifelessderekh: ecause this is a known WTF22:52
* SpamapS -> Saturday22:52
lifelessSpamapS: ack22:52
lifelessderekh: so how do we check that that keystone change is the breaker?22:53
lifelessI guess a CI change to add a paste_deploy line of '' would do it22:54
derekhlifeless: ok, will try and get a few people to change their default on monday for a couple of weeks and cover all options22:54
lifelessderekh: we could do an explicit test in CI by changing the oses we build for instance22:54
lifelessset NODE_DIST in a tripleo-ci patch22:54
derekhlifeless: yup, I guess config_file =  in keystone.conf would verify the problem22:55
openstackgerritlifeless proposed a change to openstack/tripleo-image-elements: Try a revert of I935c5144728d04c59abaee9a515e249ef0d24477.  https://review.openstack.org/7383822:56
derekhlifeless: we can only do fedora and ubuntu , does devtest work with any others yet?22:56
lifelessderekh: we can do fedora, ubuntu latest, ubuntu lts, centos22:57
lifelessderekh: AIUI22:57
lifelesswe have three OS's involved in a CI run - TE host, jenkins slave, built image.22:58
lifelessbut the jenkins slave OS is irrelevant for this virtio issue22:58
lifelesssince we don't interact directly with a running image from it22:58
derekhok, so once we have the fedora TE hosts we can do a more complete test matrix22:59
lifelessthe only the the jenkins slave OS affects is a) image build portability issues and b) novaclient etc portability issues22:59
lifelessderekh: right, but we can at least get the full slice for 'te host == ubuntu latest'22:59
lifelesswhich would be /much/ more comforting than just fedora and ubuntu worked23:00
derekhlifeless: ok, will give them a shot once were happy CI is up  and running properly again23:00
lifelesshttps://jenkins02.openstack.org/job/check-tripleo-seed-precise/359/console should tell us23:01
*** cd-undercloud has joined #tripleo23:09
cd-undercloud************** overcloud complete status=1 ************23:09
*** cd-undercloud has quit IRC23:09
*** ramishra has joined #tripleo23:15
*** ramishra has quit IRC23:20
lifelessderekh: early next week I think we can talk about how to do pre-setup seed/undercloud/overclouds and make things faster23:26
derekhlifeless: yup, cool, also we spoke a week ago about a pip mirror of packages to build (when I tried to add pip support to source-repositories), I put this up as a WIP https://review.openstack.org/#/c/71837/23:27
derekhlifeless: if you have a chance at some stage could you take a look23:28
lifelessok23:31
lifelessthats for holding the unreleased edited versions of things we're testing23:31
derekhlifeless: that seed job is gone green23:31
lifelessright?23:31
lifeless\o;23:31
lifeless\o/23:31
* derekh would love to see the trifecta23:31
lifelessderekh: can you file a bug + reversion in keystone?23:31
derekhlifeless: will do23:31
openstackgerritRichard Su proposed a change to openstack/tripleo-incubator: Separate OVERCLOUD_DIB_EXTRA_ARGS into two new variables  https://review.openstack.org/7384223:31
lifelessderekh: I have to run - late valentines day thing23:31
derekhlifeless: yup correct,23:33
*** ramishra has joined #tripleo23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!