15:02:16 <johnthetubaguy> #startmeeting XenAPI
15:02:17 <openstack> Meeting started Wed Apr 16 15:02:16 2014 UTC and is due to finish in 60 minutes.  The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:21 <openstack> The meeting name has been set to 'xenapi'
15:02:23 <BobBall> o/
15:02:32 <johnthetubaguy> how do?
15:02:34 <leifz> good morning
15:02:36 <BobBall> again.  My amazing powers of prediction on show.
15:02:53 <johnthetubaguy> #topic CI
15:03:02 <johnthetubaguy> any news from the CI world?
15:03:10 <BobBall> It's healthy :)
15:03:22 <BobBall> And https://review.openstack.org/#/c/87680/
15:03:37 <johnthetubaguy> ah, cool
15:03:43 <johnthetubaguy> so things are progressing
15:03:47 <BobBall> Did have a hiccup this morning
15:03:52 <BobBall> which I've hopefully fixed
15:04:01 <BobBall> so we lost a few jobs because the gerrit stream died
15:04:11 <johnthetubaguy> oh, bummer
15:04:18 <BobBall> https://github.com/citrix-openstack/openstack-citrix-ci/pull/26
15:04:43 <BobBall> I could always re-add them manually since the rcbops site tells me which ones it missed
15:04:56 <BobBall> but probably not worth it
15:05:01 <BobBall> we don't seem to have missed much
15:05:06 <johnthetubaguy> I duno, could be good for the stats though
15:05:08 <johnthetubaguy> but cools
15:05:15 <BobBall> sure, but I don't like gaming stats :)
15:05:27 <BobBall> we're already doing exceptionally well if you compare us to other 3rd party CIs
15:05:39 <johnthetubaguy> well, there might be a bug lurking in those changes, its not totally gaming?
15:05:54 <BobBall> highest pass rate of all 3rd party CI's
15:05:58 <BobBall> I'm happy with that ;)
15:06:01 <BobBall> True.
15:06:25 <BobBall> I suppose we could add a filter that adds jobs as they are commented on if we haven't run a test on that job
15:06:38 <BobBall> Similar to jenkins
15:07:17 <BobBall> XenServer CI voted on 327 patchsets (85.96%) passing 294 (89.91%), failing 33 (10.09%) and unparsed 0 (0.0%)  - 10% failure rate is pretty good1
15:07:20 <BobBall> -1+!
15:07:35 <johnthetubaguy> oh dear, thats a bad sign, generally
15:07:44 <BobBall> that it's so low?
15:07:51 <johnthetubaguy> that its so high
15:07:57 <BobBall> or oh dear in the context of openstack because all other tests are much higher?
15:08:01 <johnthetubaguy> 10% is massively high
15:08:04 <BobBall> jenkins has a 43% failure rate
15:08:10 <BobBall> hyper-v 47%
15:08:16 <BobBall> powerVM 25%
15:08:23 <BobBall> and docker / vmware aren't even running ATM
15:08:33 <BobBall> only turbo-hipster has a lower rate at 7.3%
15:08:39 <BobBall> http://www.rcbops.com/gerrit/reports/nova-cireport.html
15:08:44 <johnthetubaguy> yeah, not arguing with you, its just a worry
15:08:55 <johnthetubaguy> generally :)
15:09:01 <BobBall> Indeed.
15:09:18 <johnthetubaguy> jenkins has a hard job though, lots of real failures, and has many tests that fail probably less than ours
15:09:21 <BobBall> Well I'm not planning to put any effort in improving the failure rate ATM
15:09:35 <johnthetubaguy> OK, fair point
15:09:40 <johnthetubaguy> more tests first I guess
15:09:44 <BobBall> Indeed
15:09:48 <BobBall> which needs the stackforge change
15:10:02 <BobBall> I suspect we can enable quite a lot of them already
15:10:14 <BobBall> but I'm not confident enough that it won't kill the failure rate
15:10:19 <BobBall> e.g. we had a lot of volume failures
15:10:45 <johnthetubaguy> right
15:10:50 <BobBall> but a devstack change which caused it to bomb out on failing commands showed that the XFS volume wasn't big enough
15:11:18 <BobBall> we were creating it in MB rather than GB... ... ...
15:11:28 <johnthetubaguy> I keep wondering about a second setup that does on demand requests only, that can be used to try out new tests, etc
15:11:30 <BobBall> Fixing that made devstack work but should also I hope fix a lot of the volume tests
15:11:33 <BobBall> :D
15:11:39 <johnthetubaguy> cools
15:11:46 <BobBall> New tests are automatically checked with tempest
15:12:06 <BobBall> but if it's in stackforge we can easily re-submit tests when we fix things in nova etc
15:13:33 <johnthetubaguy> yeah, I was more thinking, add more tests, run it 10 times, but leave other ones running on the old config
15:13:52 <johnthetubaguy> but maybe once we test in stackforge your config repo, we get that?
15:14:09 <BobBall> hmmm
15:14:13 <BobBall> Not sure how to add that
15:14:18 <BobBall> not automatically anyway
15:14:27 <johnthetubaguy> tricky
15:14:28 <BobBall> unless we said all tempest changes get run multiple times
15:14:34 <BobBall> but that's ugly
15:14:43 <johnthetubaguy> well, recheck a few times was the plan
15:14:52 <BobBall> why call out tempest and not re-run everything in nova a few times too
15:15:05 <BobBall> in case we've added a race in nova rather than added a race from a new test
15:15:13 <BobBall> recheck works already, of course
15:15:20 <BobBall> so if there is suspicion you can recheck
15:15:30 <BobBall> I fixed a bug this week where it would overwrite results if you rechecked
15:15:33 <BobBall> now they are all archived
15:15:35 <johnthetubaguy> maybe, just wonder about testing it out without impact whats happening for peoples patches, but yeah
15:15:48 <BobBall> (don't tell Rackspace - I'm using all of their storage.  Only 150GB for now though...)
15:16:24 <johnthetubaguy> lol, thats increasing nicely
15:16:41 <BobBall> yeah...
15:16:48 <BobBall> Blame ceilometer
15:16:59 <BobBall> they have far and away the biggest log file ;)
15:17:08 <johnthetubaguy> we sell VMs with more storage than that, small amounts of data, a proper use measures in TB
15:17:09 * johnthetubaguy giggles
15:17:25 <johnthetubaguy> anyways...
15:17:31 <johnthetubaguy> #topic Summit
15:17:32 <BobBall> OK - I'll tell you when we hit the 1PB mark.
15:17:41 <johnthetubaguy> BobBall: good good
15:17:54 <johnthetubaguy> so...
15:17:59 <BobBall> oh - hang on
15:18:01 <BobBall> before we move on from CI
15:18:05 <BobBall> I've got a frustrating
15:18:08 <BobBall> frustration*
15:18:08 <johnthetubaguy> oh, ok...
15:18:18 <BobBall> servers which go into error state when deleting them
15:18:32 <BobBall> because we're running dozens a day
15:18:37 <BobBall> we hit it quite frequently
15:18:42 <BobBall> I know there is a clean up every 24 hours or so
15:18:51 <BobBall> but yesterday I had 6 servers which refused to go away
15:19:04 <johnthetubaguy> hmm, sounds annoying, can you send me some uuids?
15:19:08 <johnthetubaguy> via email
15:19:10 <BobBall> Is there a bug raised for this?  Do you know anything about what's going on etc?
15:19:13 <johnthetubaguy> I will raise a ticket
15:19:15 <BobBall> Well I've only got one ATM
15:19:22 <BobBall> but they get cleaned up as I say
15:19:29 <BobBall> and I have no idea why they fail to delete
15:19:32 <johnthetubaguy> I saw something myself once, seems to be a cells issue, as far as I can tell
15:19:45 <BobBall> uuids aren't private :)
15:19:46 <BobBall> cb6a6a40-f0d1-40a0-bbe4-29fc91eef5bc
15:19:49 <BobBall> that's the current one
15:19:56 <johnthetubaguy> like a bad sync of state, or something
15:20:12 <johnthetubaguy> OK, it was more that I am unlikely to do anything about it right now
15:20:39 <BobBall> bah :D
15:21:00 <BobBall> it'll have been cleaned up by the time you read emails ;)
15:21:52 <johnthetubaguy> yeah, I am on ETO till next wednesday from tomorrow, so you are probably right
15:22:09 <emaganap> XenAPI folks, I have a request from the docs area, let me know when will be the best time to speak up
15:22:14 <BobBall> ETO?
15:22:26 <johnthetubaguy> earned time off
15:22:27 <johnthetubaguy> holiday
15:22:55 <BobBall> Gotta love how all companies have different words for it
15:23:02 <BobBall> should we talk docs now johnthetubaguy and emaganap ?
15:23:07 <johnthetubaguy> yeah, its funncy
15:23:21 <johnthetubaguy> well, quickly about the summit...
15:23:29 <johnthetubaguy> anything we want for the summit?
15:23:35 <BobBall> A t-shirt.
15:23:40 <BobBall> But I think that's a given.
15:23:42 <emaganap> johnthetubaguy: beer
15:23:45 <johnthetubaguy> BobBall: you were going to organise some kinda beer?
15:23:52 <BobBall> beer is a better answer.
15:24:00 <BobBall> I've not done anything about beer :/
15:24:53 <johnthetubaguy> ah, well we should do something, if there is no Nova session on XenAPI
15:24:54 <johnthetubaguy> anyways
15:24:58 <johnthetubaguy> #topic Docs
15:25:01 <johnthetubaguy> fireaway
15:25:12 <emaganap> I work mostly on the Neutron area
15:25:28 <emaganap> we have a bud about undocumented work on XenAPI support
15:25:29 <emaganap> https://bugs.launchpad.net/openstack-manuals/+bug/1026745
15:25:31 <uvirtbot> Launchpad bug 1026745 in openstack-manuals "Quantum docs should describe XenAPI setup" [Medium,Confirmed]
15:26:16 <emaganap> I was wondering if you guys could take a look and give me some possible names to own this bug
15:26:23 <emaganap> It has been opened for a while
15:26:38 <johnthetubaguy> yeah, I have not played with that recently, I know matel had a wiki page
15:26:57 <BobBall> wiki page?
15:26:59 <BobBall> not blog?
15:27:08 <BobBall> I know he did http://blogs.citrix.com/2013/06/14/openstack-networking-quantum-on-xenserver-from-notworking-to-networking/
15:27:29 <BobBall> Oh - this page? https://wiki.openstack.org/wiki/QuantumDevstackOvsXcp
15:27:36 <johnthetubaguy> yeah, thats it
15:27:51 <johnthetubaguy> well, and the blog I guess
15:28:02 <BobBall> hmmm - yeah - that's devstack
15:28:07 <johnthetubaguy> I have never got it working myself, and kinda thing we need another CI setup to test it, but yeah
15:28:26 <johnthetubaguy> well, what bits run where, is probably the first bit right?
15:28:27 <BobBall> I guess the blog post is more what any docs should cover
15:28:41 <BobBall> https://raw.github.com/matelakat/shared/xs-q-v1/xenserver-quantum/deployment.png
15:28:45 <BobBall> That's what runs where
15:28:46 <BobBall> it's fun.
15:29:01 <emaganap> I was not familiar with those ones but I always hear the same statement "I have never got it working myself"
15:29:02 <johnthetubaguy> yeah, agreed
15:29:03 <emaganap> :-(
15:29:22 <BobBall> We have a CI that runs the setup at Citrix
15:29:27 <annegentle> BobBall: be sure to add that blog post in the comments for that bug
15:29:40 <BobBall> some tempest tests fail due to security groups not working properly with the OVS
15:29:42 <annegentle> BobBall: if it's not already
15:29:44 <BobBall> but I think that's being fxied ATM
15:29:46 <BobBall> will do annegentle
15:30:01 <emaganap> In Neutron we are deprecating anything that is not documented
15:30:17 <johnthetubaguy> emaganap: makes good sense
15:30:36 <emaganap> I will give it a try myself
15:30:45 <annegentle> BobBall: danke
15:30:48 <johnthetubaguy> this is probably a sticking point for nova-network removal as well right
15:31:13 <johnthetubaguy> BobBall: changes of a second XenServer CI that runs neutron?
15:32:00 <BobBall> By myself?  Quite low I think.
15:32:08 <BobBall> Not sure about the resources either - doubling up might be too much
15:32:24 <BobBall> I assume you mean a voting 3rd party CI
15:32:30 <johnthetubaguy> yeah, ideally
15:32:39 <johnthetubaguy> or switching over, if it works
15:32:44 <BobBall> it might 'just work' since all the goodness is in devstack
15:32:58 <BobBall> but the 3rd party CI is firmly nova-network ATM
15:33:02 <johnthetubaguy> anyways, maybe sorting out the testing is enough to talk about at the summit?
15:33:06 <BobBall> as I said, we have one internal at Citrix which is running
15:33:17 <BobBall> That'd be a very good topic to talk about yeah
15:33:19 <johnthetubaguy> BobBall: it must be neutron soon, else we will never remove nova-network
15:33:39 <johnthetubaguy> BobBall: so do we want a session of XenAPI driver then?
15:33:39 <BobBall> I'm eagerly anticipating a timescale for the removal of nova-network
15:33:43 <emaganap> yes, please work with Neutron folks if needed.. I can help
15:33:46 <emaganap> I am Neutron core
15:34:00 <BobBall> and I'm sure that that removal shouldn't depend on the xenserver 3rd party CI
15:34:15 <johnthetubaguy> emaganap: appreciate the heads up on deprecation
15:34:40 <johnthetubaguy> BobBall: it will do really, at least makes that CI useless, so puts XenAPI driver at risk again
15:34:47 <BobBall> Not sure it's worth a summit session - but definitely a discussion :)
15:34:49 <BobBall> indeed.
15:35:01 <BobBall> What are the prospects of nova-network being deprecated in juno?
15:35:09 <johnthetubaguy> BobBall: duno
15:35:14 <BobBall> emaganap?
15:35:18 <BobBall> are you hopeful?
15:35:29 <BobBall> *realistically hopeful? ;)
15:35:36 <emaganap> That is the 1M USD question
15:35:41 <BobBall> haha ok
15:35:50 <johnthetubaguy> BobBall: personally I am campaigning for its removal from nova, into a separate project now
15:35:54 <emaganap> we hope to make it this time
15:36:15 <johnthetubaguy> BobBall: if we don't deprecate it
15:36:16 <BobBall> btw - who gets the 1M if neutron makes it?  Because if no one has volunteered, I'd be happy to.
15:36:25 <emaganap> well, nova-network is already a separate project.. it is called Neutron
15:37:34 <emaganap> my personal take is we should deprecated ASAP.. then we will fix all issues ASAP
15:37:54 <emaganap> no body knows how many bugs are under the rocks until you move them
15:38:21 <johnthetubaguy> emaganap: yeah, it would compete with neutron, thats true
15:38:23 <emaganap> but it is just me and my aggressive, crazy view
15:38:30 <BobBall> Perhaps we should have a "Quantum week".  Break nova network, switch jenkins over to use it for a week and see how it goes
15:38:36 <johnthetubaguy> well, need to do something, anyways
15:38:36 <BobBall> sorry, sorry, neutron.
15:38:44 <johnthetubaguy> there was one I think last cycle
15:38:51 <BobBall> a neutron week?
15:38:54 <BobBall> I missed that
15:38:55 <johnthetubaguy> anyways… we are getting distracted
15:38:57 <emaganap> BobBall: I like that idea.. a breaking week
15:39:48 <johnthetubaguy> whats the plan with the docs then?
15:40:29 <emaganap> johnthetubaguy: not sure if you are asking in general or just for the bug that I asked
15:40:33 <johnthetubaguy> we use quark plugin, or at least moving that way, so I don't really know much about the other one
15:40:39 <johnthetubaguy> emaganap: kinda both
15:40:42 <BobBall> I don't have a plan currently
15:40:48 <BobBall> I don't know much about it either which is fun
15:40:54 <BobBall> What's the deprecation schedule mestery ?
15:40:56 <BobBall> emaganap*
15:41:05 <johnthetubaguy> BobBall: not that bit, just the docs plan
15:41:07 <johnthetubaguy> for us
15:41:12 <johnthetubaguy> as in XenAPI driver
15:41:19 <emaganap> BobBall: Ohhhhhhh
15:41:32 <emaganap> BobBall: what a confusion....  ha ha ah  lol
15:41:37 <johnthetubaguy> BobBall: I think I get your question, now too
15:41:42 <BobBall> what docs for XenAPI driver? *lost*
15:42:02 <johnthetubaguy> yeah… lets step back
15:42:06 <johnthetubaguy> first the bug
15:42:10 <johnthetubaguy> the doc bug
15:42:13 <emaganap> lI think it will be deprecated during Juno-3, so we have time
15:42:22 <johnthetubaguy> how do we move forward
15:42:29 <johnthetubaguy> cool, so deadline is Juno-3
15:42:51 <johnthetubaguy> BobBall: do we need to keep that code, if yes, whats the plan with the docs?
15:43:43 <BobBall> Probably, and as I said I do not have a plan
15:43:53 <BobBall> A plan needs to be made
15:43:55 <johnthetubaguy> BobBall:  OK just confirming
15:43:59 <BobBall> which is, I guess, a metaplan.
15:44:06 <johnthetubaguy> currently rackspace is using this: https://github.com/rackerlabs/quark
15:44:14 <johnthetubaguy> or at least planning to move that way
15:44:32 <BobBall> What is quark?
15:44:42 <johnthetubaguy> which is a different neutron plugin, essentially
15:44:57 <BobBall> Ah - so XenAPI is using the OVS plugin
15:45:04 <BobBall> wasn't that getting deprecated at some point?
15:45:19 <johnthetubaguy> yeah, quark is an alternative to the OVS plugin, that also uses OVS
15:45:31 <BobBall> I meant in general, from neutron.
15:45:34 <johnthetubaguy> quark currently only works with XenServer
15:46:09 <BobBall> I wonder if documenting + having a CI based on Quark would be a better long term combination
15:46:21 <BobBall> since the OVS plugin won't keep sync with the OVS from xenserver
15:46:23 <johnthetubaguy> BobBall: thats the question I think
15:46:25 <BobBall> but I guess quark is more likely to
15:46:44 <johnthetubaguy> BobBall: well, quark is using OVS v2.0 currently, I think
15:46:53 <BobBall> good point.
15:47:43 <johnthetubaguy> anyways, needs some thought
15:47:53 <BobBall> I get the link with quantum... but when I think of quark I can't help but think of low fat cheese...
15:47:58 <BobBall> strange plugin name from that perspective.
15:48:11 <johnthetubaguy> yeah, duno where the name came from
15:48:21 <BobBall> Well I assume quarks?
15:48:30 <johnthetubaguy> maybe, anyways...
15:48:33 <BobBall> as in up, down, strange, top, bottom...
15:48:37 <johnthetubaguy> yeah
15:48:43 <johnthetubaguy> its strange, lol
15:48:45 <BobBall> kinda linked to quantum stuff ;)
15:48:55 <johnthetubaguy> ah, very true...
15:48:56 <BobBall> haha
15:49:10 <johnthetubaguy> my brain is about to turn off me thinks
15:49:21 <BobBall> Let's call it then.
15:49:43 <johnthetubaguy> well...
15:49:49 <johnthetubaguy> #topic Open Discussion
15:49:53 <johnthetubaguy> are there any other bits
15:50:01 <BobBall> Didn't we just do Open Discussion? ;)
15:50:06 <johnthetubaguy> sounds like we need a plan for neutron and XenServer introp
15:50:11 <johnthetubaguy> That was docs I though
15:50:17 <johnthetubaguy> I can't remember now, oh dear
15:50:24 <BobBall> yup
15:50:35 <johnthetubaguy> I can see my band rehearsal being fun tonight
15:50:42 <BobBall> I'd happily move to quark I think
15:50:47 <johnthetubaguy> conductor: why all the wrong notes?
15:51:01 <johnthetubaguy> me: duno, just kinda stopped reading the music and fell asleep
15:51:03 <BobBall> we'd just need to document and/or package OVS
15:51:09 <johnthetubaguy> yeah, true
15:51:20 <johnthetubaguy> I am sure such packages could be provided
15:51:29 <johnthetubaguy> given they are being used already
15:51:41 <BobBall> it's easy enough to compile too
15:51:44 <johnthetubaguy> but yeah, needs some work
15:51:47 <johnthetubaguy> true true
15:52:00 <BobBall> chuck them in the supplemental pack with the plugins
15:52:03 <BobBall> and jobs a goodun
15:52:07 <johnthetubaguy> cools
15:52:25 <BobBall> getting that in a CI would be nice and close to what you use at RAX too
15:52:32 <johnthetubaguy> anyways, we need to find a way to test that upstream
15:52:54 <johnthetubaguy> BobBall: yeah, I send an email to different people asking for effort on that like every week, one day it might work
15:52:57 <BobBall> good way to get some involvement in the current CI to upgrade it to neutron + quark!
15:53:02 <BobBall> haha
15:53:03 <johnthetubaguy> by which I mean, one day I will snap and do it
15:53:10 <BobBall> *grin*
15:53:35 <johnthetubaguy> yeah, lets see, I am working on getting a copy of the CI to run XenServer + cells + quark
15:53:40 <johnthetubaguy> and using that as a good intro
15:53:47 <johnthetubaguy> maybe including cloud-cafe tests
15:53:59 <johnthetubaguy> but just one bit of those would be a step forward
15:54:10 <johnthetubaguy> so… we are out of time, almost
15:54:17 <johnthetubaguy> any last burning issues or ideas?
15:55:04 <BobBall> not that I can think of
15:55:09 <johnthetubaguy> #endmeeting