15:06:03 #startmeeting XenAPI 15:06:03 Meeting started Wed Nov 26 15:06:03 2014 UTC and is due to finish in 60 minutes. The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:06:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:06:07 hello folks 15:06:08 The meeting name has been set to 'xenapi' 15:06:17 #topic CI 15:06:23 Howdy. 15:06:30 Hello 15:06:42 BobBall: matel: hows things looking in CI land? 15:06:52 Good... But... 15:06:58 https://review.openstack.org/#/c/133975/ 15:07:02 Don't like that -1 15:07:14 I've not followed up on it and I need to I know 15:07:20 but it means I'm not happy to remove the restrictions yet 15:07:38 there are still some race conditions / circumstaces where some of the excluded tests will fail 15:08:08 that's bad news 15:08:18 I did note that it's the same message we've seen many times... Details: {u'message': u'No valid host was found. There are not enough hosts available.', u'created': u'2014-11-19T16:57:33Z', u'code': 500} 15:08:23 like 10% failure I guess 15:08:24 What was the failure ratio? 15:08:32 No idea 15:08:36 BobBall: that means you have too many instances running 15:08:37 we'd need maybe 100 runs to get a ratio 15:08:55 You think it's running out of RAM then? 15:09:03 oh wait, maybe not... 15:09:16 we need the other instance faults in the DB for that instance 15:09:22 * BobBall checks screen-n-sch... 15:09:24 It's gonna be a hard nut to crack 15:09:35 its possible it was retrying, and then runs out of hosts 15:09:43 thats more likely I guess 15:10:02 screen-n-sch is taking _ages_ to get from the CDN! 15:10:39 it's only 31k but has been downloading for over 60 seconds so far... 15:10:48 * BobBall strums his fingers on the table 15:11:19 doh 15:11:26 Let's move on and see if that loads later. 15:11:29 yeah 15:11:36 any more for any more? 15:12:07 Neutron does not run in parallel for some reason. 15:12:12 But it ran in serial. 15:12:17 Let me dig out the results. 15:12:45 http://paste.openstack.org/show/137612/ 15:13:14 21 failures in parallel was that? I don't remember 15:13:30 or was that 21 failures when we went serial only? 15:13:35 That must have been serial only, sorry 15:13:39 worker 0 only :) 15:13:50 yeah, I have a feeling neutron broke in parallel 15:13:52 We got lots more failures running in parallel didn't we 15:14:07 Sure - but it's run in parallel for -infra 15:14:08 some races still in iptables, I think I was told 15:14:12 hmm, odd 15:14:18 I guess it muss be now 15:14:18 Ah - that might not be using iptables 15:14:28 anyways 15:14:32 matel: Which firewall was that with? 15:14:35 We are not using iptables either 15:14:39 Now using noop? 15:14:44 y 15:15:03 OK 15:15:15 Do you know who might have more details on the races you're thinking of johnthetubaguy ? 15:15:39 BobBall: trying to remember... 15:16:00 BobBall: we don't have the neutron call back libvirt have implemented to avoid lots of those races, it probably related 15:16:20 libvirt has instance events call back to say neutron has completed, or something like that 15:16:45 basically there is a wait for event thing in libivrt 15:17:24 I see 15:17:29 So what does that do? 15:17:43 stop neutron from running things in parallel until libvirt has said that X has finished? 15:17:58 not really 15:18:09 it waits to start the VM until neutron has set everything up 15:18:23 So it's the nova driver _listning_ to the neutron event 15:18:46 so if the issue is what you're thinking of we don't need the nova driver to emit events 15:19:02 We are experiencing failures with parallel runs 15:19:25 If we were missing something like what you said, we would see failures with serial as well, don't we? 15:19:46 yes thats true 15:19:47 No - because neutron has slowness issues 15:19:51 Or maybe the higher load... 15:19:57 so races will become more visible in parallel 15:20:04 BobBall: +1 15:20:09 you typed it before I did 15:20:30 BobBall said the magic word: race 15:20:46 yeah, it is a race 15:20:55 it might be ready when you start the VM, it might not 15:21:05 Let's just add a sleep 60. 15:21:08 so libvirt waits till its ready 15:21:10 Sleep 60's always work. 15:21:14 right... 15:21:20 ... if you have enough of them 15:21:27 And they speed up the time it takes to run the tests. 15:21:36 what are the failures looking like? 15:21:39 Paradixically there matel ... :) 15:22:24 gimme a sec 15:23:03 tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces 15:23:03 tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestXML.test_create_list_show_delete_interfaces 15:23:03 tempest.api.network.admin.test_quotas.QuotasTest.test_lbaas_quotas 15:23:03 tempest.api.network.test_service_type_management.ServiceTypeManagementTestJSON.test_service_provider_list 15:23:07 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_healthmonitor_list 15:23:10 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_member_list 15:23:13 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_pool_list 15:23:16 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_vip_list 15:23:19 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_meter_label_list 15:23:22 tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_meter_label_rule_list 15:23:25 tempest.scenario.test_load_balancer_basic.TestLoadBalancerBasic.test_load_balancer_basic 15:23:27 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic 15:23:29 tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic 15:23:32 tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_compute_with_volumes 15:23:48 I don't really mean that 15:23:50 johnthetubaguy: these are the failed ones with serial 15:23:53 test_hotplug_nic <-- we don't do that yet do we? 15:23:55 what sort of failures did those see? 15:24:11 BobBall: I have code for that, but its not finished or merged 15:24:43 OK, great. 15:24:46 are they ssh failure? 15:24:51 We might want to talk about that in the new year 15:24:59 BobBall: put it up for review a year ago or so 15:25:11 *nod* 15:25:29 coded it in some airport after some US summit, while drinking something from starbucks 15:25:57 anyways 15:26:07 any more we want to debug now? or do this offline? 15:26:10 Sounds like it doesn't even need reviewing - just merge it :D 15:26:20 That's the perfect coding environment! 15:26:38 No, I don't think there is more to talk about on that front 15:26:44 no 15:27:48 #topic Open Discussion 15:27:57 any more? 15:28:02 nope 15:28:36 yes 15:28:40 maybe 15:28:45 should we talk about DIB? 15:29:22 We'll give a try to use DIB to create the domU's root filesystem 15:30:00 So something like this: You have a qcow2 with a XenServer inside, on the SR sits a vhd containing domU's virtual disk 15:30:04 OK, yeah, they are having "fun" using that in rackspace mind 15:30:23 crazy 15:30:28 ... Use tapdisk to mount the virtual disk.. 15:30:32 anyways, any more? 15:30:46 No, thanks, thanks. 15:31:03 thanks all 15:31:07 #endmeeting