#openstack-meeting log

15:06:03 <johnthetubaguy> #startmeeting XenAPI
15:06:03 <openstack> Meeting started Wed Nov 26 15:06:03 2014 UTC and is due to finish in 60 minutes.  The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:06:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:06:07 <johnthetubaguy> hello folks
15:06:08 <openstack> The meeting name has been set to 'xenapi'
15:06:17 <johnthetubaguy> #topic CI
15:06:23 <BobBall> Howdy.
15:06:30 <matel> Hello
15:06:42 <johnthetubaguy> BobBall: matel: hows things looking in CI land?
15:06:52 <BobBall> Good... But...
15:06:58 <BobBall> https://review.openstack.org/#/c/133975/
15:07:02 <BobBall> Don't like that -1
15:07:14 <BobBall> I've not followed up on it and I need to I know
15:07:20 <BobBall> but it means I'm not happy to remove the restrictions yet
15:07:38 <BobBall> there are still some race conditions / circumstaces where some of the excluded tests will fail
15:08:08 <matel> that's bad news
15:08:18 <BobBall> I did note that it's the same message we've seen many times... Details: {u'message': u'No valid host was found. There are not enough hosts available.', u'created': u'2014-11-19T16:57:33Z', u'code': 500}
15:08:23 <johnthetubaguy> like 10% failure I guess
15:08:24 <matel> What was the failure ratio?
15:08:32 <BobBall> No idea
15:08:36 <johnthetubaguy> BobBall: that means you have too many instances running
15:08:37 <BobBall> we'd need maybe 100 runs to get a ratio
15:08:55 <BobBall> You think it's running out of RAM then?
15:09:03 <johnthetubaguy> oh wait, maybe not...
15:09:16 <johnthetubaguy> we need the other instance faults in the DB for that instance
15:09:22 * BobBall checks screen-n-sch...
15:09:24 <matel> It's gonna be a hard nut to crack
15:09:35 <johnthetubaguy> its possible it was retrying, and then runs out of hosts
15:09:43 <johnthetubaguy> thats more likely I guess
15:10:02 <BobBall> screen-n-sch is taking _ages_ to get from the CDN!
15:10:39 <BobBall> it's only 31k but has been downloading for over 60 seconds so far...
15:10:48 * BobBall strums his fingers on the table
15:11:19 <johnthetubaguy> doh
15:11:26 <BobBall> Let's move on and see if that loads later.
15:11:29 <johnthetubaguy> yeah
15:11:36 <johnthetubaguy> any more for any more?
15:12:07 <matel> Neutron does not run in parallel for some reason.
15:12:12 <matel> But it ran in serial.
15:12:17 <matel> Let me dig out the results.
15:12:45 <matel> http://paste.openstack.org/show/137612/
15:13:14 <BobBall> 21 failures in parallel was that?  I don't remember
15:13:30 <BobBall> or was that 21 failures when we went serial only?
15:13:35 <BobBall> That must have been serial only, sorry
15:13:39 <BobBall> worker 0 only :)
15:13:50 <johnthetubaguy> yeah, I have a feeling neutron broke in parallel
15:13:52 <BobBall> We got lots more failures running in parallel didn't we
15:14:07 <BobBall> Sure - but it's run in parallel for -infra
15:14:08 <johnthetubaguy> some races still in iptables, I think I was told
15:14:12 <johnthetubaguy> hmm, odd
15:14:18 <johnthetubaguy> I guess it muss be now
15:14:18 <BobBall> Ah - that might not be using iptables
15:14:28 <johnthetubaguy> anyways
15:14:32 <BobBall> matel: Which firewall was that with?
15:14:35 <matel> We are not using iptables either
15:14:39 <BobBall> Now using noop?
15:14:44 <matel> y
15:15:03 <BobBall> OK
15:15:15 <BobBall> Do you know who might have more details on the races you're thinking of johnthetubaguy ?
15:15:39 <johnthetubaguy> BobBall: trying to remember...
15:16:00 <johnthetubaguy> BobBall: we don't have the neutron call back libvirt have implemented to avoid lots of those races, it probably related
15:16:20 <johnthetubaguy> libvirt has instance events call back to say neutron has completed, or something like that
15:16:45 <johnthetubaguy> basically there is a wait for event thing in libivrt
15:17:24 <BobBall> I see
15:17:29 <BobBall> So what does that do?
15:17:43 <BobBall> stop neutron from running things in parallel until libvirt has said that X has finished?
15:17:58 <johnthetubaguy> not really
15:18:09 <johnthetubaguy> it waits to start the VM until neutron has set everything up
15:18:23 <BobBall> So it's the nova driver _listning_ to the neutron event
15:18:46 <BobBall> so if the issue is what you're thinking of we don't need the nova driver to emit events
15:19:02 <matel> We are experiencing failures with parallel runs
15:19:25 <matel> If we were missing something like what you said, we would see failures with serial as well, don't we?
15:19:46 <johnthetubaguy> yes thats true
15:19:47 <BobBall> No - because neutron has slowness issues
15:19:51 <matel> Or maybe the higher load...
15:19:57 <BobBall> so races will become more visible in parallel
15:20:04 <johnthetubaguy> BobBall: +1
15:20:09 <johnthetubaguy> you typed it before I did
15:20:30 <matel> BobBall said the magic word: race
15:20:46 <johnthetubaguy> yeah, it is a race
15:20:55 <johnthetubaguy> it might be ready when you start the VM, it might not
15:21:05 <BobBall> Let's just add a sleep 60.
15:21:08 <johnthetubaguy> so libvirt waits till its ready
15:21:10 <BobBall> Sleep 60's always work.
15:21:14 <johnthetubaguy> right...
15:21:20 <BobBall> ... if you have enough of them
15:21:27 <matel> And they speed up the time it takes to run the tests.
15:21:36 <johnthetubaguy> what are the failures looking like?
15:21:39 <BobBall> Paradixically there matel ... :)
15:22:24 <matel> gimme a sec
15:23:03 <matel> tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces
15:23:03 <matel> tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestXML.test_create_list_show_delete_interfaces
15:23:03 <matel> tempest.api.network.admin.test_quotas.QuotasTest.test_lbaas_quotas
15:23:03 <matel> tempest.api.network.test_service_type_management.ServiceTypeManagementTestJSON.test_service_provider_list
15:23:07 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_healthmonitor_list
15:23:10 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_member_list
15:23:13 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_pool_list
15:23:16 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_lb_vip_list
15:23:19 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_meter_label_list
15:23:22 <matel> tempest.cli.simple_read_only.network.test_neutron.SimpleReadOnlyNeutronClientTest.test_neutron_meter_label_rule_list
15:23:25 <matel> tempest.scenario.test_load_balancer_basic.TestLoadBalancerBasic.test_load_balancer_basic
15:23:27 <matel> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic
15:23:29 <matel> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
15:23:32 <matel> tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_compute_with_volumes
15:23:48 <johnthetubaguy> I don't really mean that
15:23:50 <matel> johnthetubaguy: these are the failed ones with serial
15:23:53 <BobBall> test_hotplug_nic <-- we don't do that yet do we?
15:23:55 <johnthetubaguy> what sort of failures did those see?
15:24:11 <johnthetubaguy> BobBall: I have code for that, but its not finished or merged
15:24:43 <BobBall> OK, great.
15:24:46 <johnthetubaguy> are they ssh failure?
15:24:51 <BobBall> We might want to talk about that in the new year
15:24:59 <johnthetubaguy> BobBall: put it up for review a year ago or so
15:25:11 <BobBall> *nod*
15:25:29 <johnthetubaguy> coded it in some airport after some US summit, while drinking something from starbucks
15:25:57 <johnthetubaguy> anyways
15:26:07 <johnthetubaguy> any more we want to debug now? or do this offline?
15:26:10 <BobBall> Sounds like it doesn't even need reviewing - just merge it :D
15:26:20 <BobBall> That's the perfect coding environment!
15:26:38 <BobBall> No, I don't think there is more to talk about on that front
15:26:44 <matel> no
15:27:48 <johnthetubaguy> #topic Open Discussion
15:27:57 <johnthetubaguy> any more?
15:28:02 <matel> nope
15:28:36 <BobBall> yes
15:28:40 <BobBall> maybe
15:28:45 <BobBall> should we talk about DIB?
15:29:22 <matel> We'll give a try to use DIB to create the domU's root filesystem
15:30:00 <matel> So something like this: You have a qcow2 with a XenServer inside, on the SR sits a vhd containing domU's virtual disk
15:30:04 <johnthetubaguy> OK, yeah, they are having "fun" using that in rackspace mind
15:30:23 <johnthetubaguy> crazy
15:30:28 <matel> ... Use tapdisk to mount the virtual disk..
15:30:32 <johnthetubaguy> anyways, any more?
15:30:46 <matel> No, thanks, thanks.
15:31:03 <johnthetubaguy> thanks all
15:31:07 <johnthetubaguy> #endmeeting