15:02:36 <BobBall> #startmeeting XenAPI
15:02:37 <openstack> Meeting started Wed Jan  7 15:02:36 2015 UTC and is due to finish in 60 minutes.  The chair is BobBall. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:40 <openstack> The meeting name has been set to 'xenapi'
15:02:46 <johnthetubaguy> BobBall: howdy
15:02:51 <BobBall> Howdy john
15:02:51 <johnthetubaguy> good holidays?
15:02:58 <BobBall> Indeed! very relaxing
15:03:03 <BobBall> and felt a lot longer than 2 weeks.
15:03:06 <BobBall> Yourself?
15:03:27 <johnthetubaguy> good stuff, a bit of running around between various groups of family, but generally quite relaxing too
15:03:44 <BobBall> Yeah - but that's the normal way with Christmas!
15:03:51 <BobBall> Well, happy new year to you and welcome back.
15:04:06 <BobBall> I have to say I do find the new year a bit of a struggle sometimes.
15:04:15 <BobBall> Two weeks off then right back in it!
15:04:21 <BobBall> OK - anyway - first topic I guess
15:04:26 <BobBall> #topic CI
15:04:34 <BobBall> Nothing much to say except it's working great
15:04:46 <BobBall> Set up a monitor with RAX to warn when the pass rate falls low
15:04:53 <johnthetubaguy> I guess there are less patches over the holidays, but thats cool
15:04:55 <BobBall> initially the trigger was for 4 hours
15:05:06 <johnthetubaguy> ah, cool, does it email you, or something like that?
15:05:10 <BobBall> which was really low... there were quite a lot of 4-hour periods with no patches and therefore no passes
15:05:16 <BobBall> Yup.  Intelligence.rackspace.com
15:05:24 <BobBall> This rackspace cloud is quite good really
15:05:29 <johnthetubaguy> :)
15:05:41 <BobBall> Anyway - I've upped the limit now to 24 hours.
15:05:55 <BobBall> It means we won't get notified that things are pear shaped until later
15:06:06 <BobBall> but it does stop all of the incorrect notifications
15:06:15 <BobBall> Might think if a middle ground (e.g. 12 hours) is better
15:06:22 <johnthetubaguy> could you not just look at a failure percentage?
15:06:27 <BobBall> Correction - I'm at 12 hours
15:06:32 <BobBall> I do
15:06:44 <BobBall> 50% or above failure rate is bad
15:07:05 <johnthetubaguy> hmm, OK
15:07:07 <BobBall> I could drop that I suppose and have a large time
15:07:21 <johnthetubaguy> nah, you need to know sooner rather than later
15:07:25 <BobBall> but it wasn't clear how to do maths in the monitoring side :)
15:07:32 <johnthetubaguy> just wondering if there is something cleverer
15:07:41 <johnthetubaguy> but not to worry
15:07:45 <BobBall> I meant drop the percentage failure rate - if you halved it to 25% then doubled the time in theory it'd tell you at the same time
15:07:54 <BobBall> Probably
15:08:03 <johnthetubaguy> yeah, possibly
15:08:09 <BobBall> But the first iteration is qorking well enough as a big red bell
15:08:22 <BobBall> It did highlight one failure (the gerrit watch failed) that I fixed
15:08:25 <johnthetubaguy> yeah, not waiting a few days is way better, so thats nice
15:08:32 <johnthetubaguy> cools
15:08:51 <BobBall> Yup - more useful than coming in after a weekend of things going bad
15:09:11 <BobBall> CI pass rates are worrying in general though
15:09:19 <BobBall> I think I mentioned last year
15:09:25 <BobBall> tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_instance
15:09:28 <BobBall> and friends
15:09:34 <BobBall> seem to be getting more failures than they should
15:09:43 <BobBall> so I think there has been some regressions in the code
15:10:01 <johnthetubaguy> yeah, that sounds nasty
15:10:04 <BobBall> Oh - actually failure rate hasn't been too bad the last couple of weeks
15:10:09 <johnthetubaguy> its odd its xenapi specific
15:10:14 <BobBall> It was a pre-christmas thing perhaps
15:10:29 <BobBall> Well not so much
15:10:33 <johnthetubaguy> I guess shelves are busy at christmas
15:10:40 <BobBall> we have some very different characteristics to other drivers
15:10:49 <BobBall> in terms of where things like race conditions in generic code hit us
15:11:09 <BobBall> Anyway, I'm not worried ATM
15:11:26 <BobBall> Will just keep a watch on it, so it's more of a heads up in case you hear or something else clicks
15:11:35 <BobBall> or perhaps someone submits a patch you notice that fixes it I'd like to know :)
15:11:53 <BobBall> How are your efforts in getting some help with CI for quark going?
15:12:27 <johnthetubaguy> no where right now I fear
15:12:40 <BobBall> OK - tis a shame
15:12:47 <BobBall> but means we'll have to focus on Neutron + OVS
15:12:49 <johnthetubaguy> not been able to hire anyone
15:13:17 <BobBall> The status of that is in limbo for a few weeks...
15:13:44 <BobBall> News on that at the end (although you may know all the news John - just for logging purposes)
15:13:53 <BobBall> Anything you want to say on CI?
15:14:18 <johnthetubaguy> nothing on CI really
15:14:29 <BobBall> okies.
15:14:32 <BobBall> #topic Bugs
15:14:42 <BobBall> Guess we've talked about a few of them already, but I wanted to talk about the coalescing bug...
15:14:58 <BobBall> That's but #1362595
15:15:04 <BobBall> That's bug #1362595
15:15:05 <uvirtbot> Launchpad bug 1362595 in nova "move_vhds_into_sr - invalid cookie" [Medium,Fix released] https://launchpad.net/bugs/1362595
15:15:09 <BobBall> thanks uvirtbot
15:15:16 <johnthetubaguy> OK
15:15:20 <BobBall> I want a backport to icehouse... :)
15:15:41 <BobBall> I needed to make an icehouse XVA and it was failing due to the coalesce issues
15:15:48 <BobBall> which I think are all self-contained really
15:15:54 <BobBall> I've got some changes up for review
15:16:00 <johnthetubaguy> ah OK
15:16:00 <BobBall> https://review.openstack.org/#/c/142543/
15:16:04 <BobBall> https://review.openstack.org/#/c/142700/
15:16:08 <BobBall> https://review.openstack.org/#/c/143109/
15:16:11 <johnthetubaguy> I am not core on stable I am afraid
15:16:12 <BobBall> https://review.openstack.org/#/c/143110/
15:16:31 <BobBall> No, not a core on stable of course, but your comments and hopefully support of the backports would be really helpful
15:16:36 <johnthetubaguy> they tend to go for very low risk changes, do we have the XenServer CI pointed at those branches?
15:16:53 <johnthetubaguy> I can certainly try give them a look at some point soon
15:16:56 <BobBall> Good point - it didn't trigger.
15:17:20 <BobBall> Not sure what needs to be done to trigger on a stable branch
15:17:22 <BobBall> I'll look at that
15:17:23 <johnthetubaguy> I duno if it would work without having another image, but its worth a whirl I guess
15:17:25 <johnthetubaguy> cools
15:17:33 <BobBall> another image?
15:17:37 <BobBall> oh
15:17:38 <BobBall> I see
15:17:52 <johnthetubaguy> more thinking about the dependencies, but it should be OK with any luck
15:18:02 <BobBall> Yes, you're right
15:18:11 <BobBall> I doubt very much it would work
15:18:23 <BobBall> I can try though but there are a whole bunch of dependencies installed
15:18:39 <johnthetubaguy> yeah, needs a try
15:18:39 <BobBall> and I thought the dependencies for the two states are quite different
15:18:44 <BobBall> particularly because icehouse was so long ago...
15:19:13 <BobBall> Anyway - the results of including those bug fixes in a Citrix build is at http://downloads.vmd.citrix.com/OpenStack/
15:19:15 <johnthetubaguy> oh, there were some stuff deprecated here and there, thinking about it
15:19:18 <BobBall> it's a devstack-icehouse XVA
15:19:25 <johnthetubaguy> hmm, interesting
15:19:37 <johnthetubaguy> its something for devs at least, which is nice
15:20:15 <BobBall> Aiming for an interesting demo at Citrix Summit.  Which I don't want to talk about quite yet because I'm blocked on a third party currently so can't commit that it will be there :)
15:20:40 <BobBall> Just a shame it has to be icehouse based rather than Juno, but that's the way things go.
15:20:57 <BobBall> You can see from the above link that the devstack XVAs are also being updated regularly again
15:21:09 <BobBall> After an internal CI rebuild there was a lot of pain getting that back working
15:21:23 <johnthetubaguy> why not juno?
15:21:45 <BobBall> Third party dependency I'm afraid :)
15:21:50 <johnthetubaguy> doh
15:22:15 <BobBall> It's for code that should be upstreamed at some point in Kilo, as far as I'm aware, but currently is only available on icehouse
15:22:49 <BobBall> Anyway, that's that for bugs from me - kind of strayed into other territories
15:22:53 <BobBall> Anything from you johnthetubaguy?
15:23:05 <johnthetubaguy> we are past the nova spec freeze now, but yeah
15:23:08 <johnthetubaguy> nothing more from me
15:23:19 <BobBall> Ah - I wonder if the spec got in... heh :)
15:23:46 <BobBall> Okay, done with bugs then
15:23:49 <BobBall> #topic AOB
15:23:55 <BobBall> You go first for AOB :)
15:24:18 <johnthetubaguy> not really got anything, I am afraid
15:24:45 <BobBall> That's fine
15:24:47 <BobBall> Oh drat
15:24:52 <BobBall> I had another bug to talk about...
15:25:17 <johnthetubaguy> doh, no worries, you got the number
15:25:32 <BobBall> http://lists.openstack.org/pipermail/openstack/2015-January/011063.html
15:25:46 <BobBall> Thought you might have some thoughts on that John?
15:25:59 <BobBall> Last time I saw this it was Ubuntu install not being able to access the internet, but John says it can
15:26:41 <BobBall> (different John!)
15:27:00 <johnthetubaguy> hmm, not sure, looking
15:27:30 <johnthetubaguy> hmm, might be a different version of XenServer not sure
15:27:56 <johnthetubaguy> the XenServer 6.0 -> 6.1 changed the need for kpartx or something like that
15:28:05 <BobBall> Oh - it did?
15:28:19 <johnthetubaguy> yeah, it wasn't needed before a certain version of xenserver
15:28:26 <johnthetubaguy> or v.v.
15:28:28 <BobBall> Hmmmz
15:28:38 <BobBall> so is your suggestion that he might be running on a very old XS?
15:28:43 <johnthetubaguy> its possible
15:28:47 <johnthetubaguy> or very new
15:29:02 <johnthetubaguy> there was a way we detected which thing to do
15:29:05 <BobBall> We regularly test on the latest version
15:29:16 <johnthetubaguy> should be fine then
15:29:17 <BobBall> OK - I'll look at that.
15:29:19 <johnthetubaguy> on that front
15:29:29 <BobBall> I did tell him how to use the XVA and the JEOS to avoid the Ubuntu installation
15:29:42 <johnthetubaguy> OK
15:29:57 <BobBall> waiting to hear back from him on whether he's willing to continue trying to find the issue while he's up and running with the XVA/JEOS
15:30:13 <johnthetubaguy> worth checking the XenServer version
15:30:23 <johnthetubaguy> it might be an error before that causing the real issue though
15:30:39 <BobBall> True but I didn't see one in the partial log he sent
15:30:44 <johnthetubaguy> network settings are a nightmare to get right for that stuff, I get it wrong at least three or four times before its working
15:31:10 <BobBall> Should be easier now with the UBUNTU_INST_BRIDGE_OR_NET_NAME
15:31:15 <johnthetubaguy> yeah, true
15:31:24 <BobBall> Anyway, that's enough of that.
15:31:35 <BobBall> If you have any other thoughts on that I'd appreciate it though
15:31:45 <johnthetubaguy> yeah, no idea really, but it does ring a very quiet bell
15:31:51 <BobBall> Ok - AOB from you?
15:31:56 <johnthetubaguy> nothing from me
15:32:03 <BobBall> Ok, final thing then...
15:32:08 <johnthetubaguy> sure
15:32:32 <BobBall> Mate Lakat unfortunately left Citrix at the end of last year so won't be returning to help out with XenAPI
15:33:01 <BobBall> We're trying to recruit backfills but until that happens we're low on resources
15:33:48 <johnthetubaguy> oh, yes, he did mention that :(
15:33:57 <BobBall> I'm not expecting Citrix to be able to achieve a huge amount in Nova this quarter with the resources we have until those backfills are on board
15:34:07 <BobBall> Bug fixes, CI, etc all will continue of course
15:34:52 <johnthetubaguy> OK, understood
15:35:11 <johnthetubaguy> most of the upstream effort on our side is cells right now
15:35:20 <johnthetubaguy> with a whole bunch of stability work as normal
15:35:21 <BobBall> *nod* makes sense
15:36:29 <BobBall> fair enough
15:36:35 <BobBall> Anyway - I think that's it from me.
15:37:10 <johnthetubaguy> and from me
15:37:10 <johnthetubaguy> thanks
15:37:23 <BobBall> As a suggestion I think we should skip next weeks' meeting and move to a fortnightly cadence
15:37:26 <BobBall> thoughts?
15:38:38 <johnthetubaguy> yes, makes sense I think
15:39:46 <BobBall> Ok good
15:39:56 <BobBall> #nextmeeting 21st January
15:40:03 <BobBall> Does it know a #nextmeeting?
15:40:04 <BobBall> No
15:40:05 <BobBall> That sucks.
15:40:15 <johnthetubaguy> info is probably the best
15:40:19 <BobBall> #info Next meeting will be 21st January
15:40:29 <BobBall> Alright.
15:40:34 <johnthetubaguy> cools
15:40:35 <BobBall> Thanks John - close there.
15:40:38 <BobBall> #endmeeting