15:00:08 <zul> #startmeeting
15:00:09 <openstack> Meeting started Tue Nov 22 15:00:08 2011 UTC.  The chair is zul. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:16 <zul> ooh canonicalers and ex-canonicalers
15:00:18 <zul> ;)
15:00:24 <smoser> :)
15:00:28 <zul> so lets get this started.
15:00:33 <smoser> zul is ex-canonical now ?
15:00:37 <zul> the agenda today is the following:
15:00:40 <zul> smoser: maybe
15:00:52 <zul> 1. Bug list
15:00:54 <zul> 2. Testing
15:01:00 <zul> 3. Progress
15:01:02 <Daviey> smoser: not yet.
15:01:14 <zul> #topic Bug list
15:01:32 <zul> so last week i tagged some more bugs that are ec2 specific
15:01:44 <zul> which can be found at https://bugs.launchpad.net/nova/+bugs?field.tag=ec2
15:02:16 <zul> right now there is only 26, which is not bad but im sure there is more bugs that havent been reported or i have missed a couple
15:02:21 <Daviey> crikey
15:02:30 <Daviey> more than i hoped.
15:02:35 <Daviey> good they are discovered.
15:03:11 <zul> this doesnt include any bugs that we are going to be reporting soon due to the testing we are going to start doing for the ec2 api
15:03:43 <zul> an interesting bug for me is: https://bugs.launchpad.net/nova/+bug/889013
15:03:44 <uvirtbot> Launchpad bug 889013 in nova "nova-api not starting the EC2 API in 2012.1-dev" [Undecided,New]
15:03:53 <zul> whoops not that one
15:04:16 <zul> https://bugs.launchpad.net/nova/+bug/827569
15:04:17 <uvirtbot> Launchpad bug 827569 in nova "ec2metadata service does not include 2011-01-01" [Wishlist,Confirmed]
15:04:39 <zul> basically it doesnt exist in the ec2 metadata list
15:05:14 <smoser> so... on that one..
15:05:23 <zul> however if we just add it to the list we would be kind of lying that it is supported we should see what has been added at that date and see if we can add it to nvoa
15:05:30 <smoser> the metadata service in openstack probably poorly mimics what is in ec2
15:05:40 <smoser> so just adding another entry at the top level is only going to break it more
15:06:00 <zul> really...intresting
15:06:04 <Daviey> Does it make sense just to add it, then report bugs for each feature it is missing?
15:06:09 <smoser> it would make more sense to actually find what was in each api version and actually do it right.
15:06:19 <zul> smoser: i agree
15:06:26 <Daviey> well yes, but slowly, slowly, catchy monkey
15:06:44 <smoser> well, there are only like 8 entries there over the course of 5 years or something.
15:06:45 <Daviey> one huge merge of each part is likely going to be less fun, than incremental ones of each part, right?
15:06:51 <zul> and people have said that in the gerrit entry for that bug as well i just havent gotten a chance to do it yet
15:06:57 <smoser> that monkey in particular ismore of a sloth
15:07:18 <zul> Daviey: if we do one huge merge im afaid it would be more difficult to review
15:07:22 <smoser> Daviey, one merge of "add a string" just doesn't help anything, and doen'st make the "fix it right" merge any smaller
15:07:26 <smoser> but if you want to fix it, go for it.
15:07:35 <zul> but we are always going to be behind what amazon does
15:07:53 <Daviey> soren: Do you have thoughts?
15:07:55 <soren> As I think I argued on gerrit: Yes, it'd be lying if we said we supported 2011-01-01, but so is every other version listed there.
15:08:13 <soren> We very likely don't support any of them correctly.
15:08:29 <zul> right and thats where the testing and reporting bugs will come in
15:08:33 <soren> ...which is something we obviously should fix at some point, but nevertheless..
15:08:34 <Daviey> Yes, but claiming to - then resolving issues where we do not, sounds more viable IMO.
15:08:37 <soren> One thing I do wonder, though:
15:08:48 <soren> Why add it? Does anything depend on that particular version being listed?
15:09:06 <soren> Put differently:
15:09:43 <smoser> why add another thing that is broken.
15:09:43 <soren> How did we notice it was missing? I have a hard time believing zul was so bored one afternoon that he decided to look at that list and went "oh, dear me, 2011-01-01 is missing".
15:09:50 <Daviey> it seems closer to providing an ec2 full experience IMO.
15:09:53 <smoser> i noticed it was broken.
15:09:55 <soren> I'm fine adding it.
15:10:09 <soren> I just wonder why.
15:10:13 <Daviey> smoser: how did you notice?
15:10:21 <smoser> not because i went looking for it, but because some tool (ec2metadata or cloud-init, i dont remember which) was using it.
15:10:25 <smoser> and it wasnt there.
15:10:30 <soren> smoser: That's good enough for me. LEt's add it.
15:10:30 <zul> me thinks smoser was bored one day ;)
15:10:38 <Daviey> smoser: did it go bang, or fall back?
15:10:39 <smoser> meh.
15:10:41 <smoser> its pointless.
15:10:44 <soren> I don't think there's that much boredom in the universe.
15:10:51 <smoser> adding another thing that is wrong isn't fixing anything
15:10:53 <smoser> but who cares.
15:10:55 <smoser> i give up.
15:11:08 <soren> smoser: If you want to help actually fix it, I'd he happy to support that as well.
15:11:11 <Daviey> smoser: by adding it, we can validate what is missing, right?
15:11:35 <soren> So far, the EC2 API implemetnation doesn't really strive to be correct (which is evident to anyone who takes a look).
15:11:42 <soren> It strives to make stuff work.
15:11:57 <Daviey> Well yes, in at least one area we match the spec better than aws does :)
15:12:01 <smoser> i dont think it requires adding it to validate what is missing. clearly, adding a "symlink" is going to result in the same stuff under 2011-01-01 that is elsewhere. so you're not any closer to making it better.
15:12:08 <smoser> but i give up. do what you want.
15:12:13 <soren> If something gets upset because 2011-01-01 is missing, adding it is completely in line with how the EC2 API implementation has been done so far.
15:12:26 <soren> That is not so say that a more correct approach woulnd't be preferable, but this is how it is.
15:12:49 <smoser> i just think its better to have a missing API than a broken one.
15:12:55 <smoser> and you're adding another broken one.
15:12:58 <soren> Then disable the EC2 API altogether.
15:13:01 <soren> SEriously.
15:13:04 <soren> It's not so spec.
15:13:13 <soren> So, following the same logic, we're better off without it.
15:13:24 <soren> s/so spec/to spec/
15:13:36 <smoser> but i give up. do what you want.
15:13:45 <soren> You suck at giving up, you know that, right? :)
15:14:16 <zul> ok can we move i want to raise another issue  which is a bug
15:14:23 <soren> It's your meeting.
15:14:24 <soren> :)
15:14:35 <soren> (yes)
15:14:51 <soren> *wink* *wink* *nudge* *nudge*
15:14:52 <zul> when we were testing the ec2 api we noticed that when connecting the ec2 api metadata server the perfomance was slow
15:15:00 <zul> im slow typer :P
15:15:25 <soren> How slow?
15:15:27 <zul> it seem to just hitting the api server sequentally and somehow batch the info
15:15:47 <soren> And using which DB backend?
15:15:49 <zul> i dont have any metrics but if you look at the log file it was just constant hitting the api server
15:15:53 <zul> sqlite
15:16:12 <soren> YEah, those requests will be sequential, iirc.
15:16:20 <Daviey> erm
15:16:24 <zul> could we batch it or something
15:16:27 <Daviey> This is known issue, no?
15:16:28 <soren> And cloud-init issues a lot of requests, IIRC.
15:16:33 <Daviey> even with mysql.
15:16:47 <soren> Daviey: I don't think this will be as sequential with MySQL.
15:17:00 <Daviey> vish suggested installing nova-api onto all compute nodes to help speed it up.
15:17:17 <soren> In the short term, that's probably a good idea.
15:17:36 <smoser> cloud-init crawls the metadata service.
15:17:37 <zul> still when running on one server it seems slow but anyways
15:17:45 <smoser> and "how slow"?
15:17:47 <smoser> really freaking slow
15:18:02 <Daviey> Yeah
15:18:10 <smoser> like a new openstack, its 5 seconds to crawl it. each request takes > .3 seconds or something
15:18:10 <Daviey> it is really pretty bad, but this is known.
15:18:11 <soren> A second per request?
15:18:15 <smoser> on an openstack with ~ 2000 instances
15:18:26 <smoser> total, ever
15:18:30 <Daviey> spawning 10 concurrent instances can wedge it.
15:18:30 <smoser> then its like 45 seconds to crawl it.
15:18:35 <soren> Erk.
15:18:38 <soren> Suckage.
15:18:39 <Daviey> But this is fixed in Essex, already - right?
15:18:47 <soren> No idea.
15:18:54 <smoser> it is surely improved.
15:18:58 <soren> You're the EC2 API team. :)
15:19:03 <smoser> but each one stillr esults in a round trip to the database i think.
15:19:04 <Daviey> zul: didn't you cherry pick a patch as an SRU candidate for diablo?
15:19:12 <soren> You should know. I only joined half an hour ago. I'm the newbie.
15:19:18 <zul> Daviey: dont remember :)
15:19:35 <Daviey> soren: yeah, you are only a nova core dev, what do you know? :)
15:19:36 <soren> smoser: Yes, each one will do a round trip to the db.
15:19:53 <soren> Daviey: My point exactly.
15:20:30 <zul> heh ok...anyone wants to raising anything else on bug before we move to the next topic
15:21:10 <zul> if not we will move on the next topic
15:21:36 <zul> ok then
15:21:39 <zul> #topic Testing
15:22:21 <zul> so last week i had a look at the aws testing scripts done by cloudscaling, i need to start using them so we can start reporting more bugs on launchpad about the ec2 api
15:22:29 <soren> Link?
15:22:49 <zul> hold on
15:23:24 <zul> https://github.com/cloudscaling/aws-compat/
15:23:36 <zul> so my thought process would be:
15:23:39 <zul> 1. Run the tests
15:23:50 <zul> 2. Report the tests as bugs in launchpad
15:23:58 <zul> 3. Tag them as ec2
15:24:08 <zul> 4. Send an overview to the openstack mailing list
15:24:33 <zul> questions/comments?
15:24:50 <soren> Is this something that can be run automatically?
15:25:30 <zul> soren: yeah we are probably going to run it in the canonical test lab and mtaylor mentioned that he was interested in bringing it in the openstack-ci
15:25:37 <soren> Then we should get it integrated into the openstack-integration-tests suite.
15:25:48 <zul> we should
15:25:54 <Daviey> soren: We'll add that to our integration jenkins testing
15:26:09 <soren> Cool beans.
15:27:01 <zul> does anyone know of any other tests we could use as well so we can get a "second opinon"?
15:27:39 <zul> soren: do you know of any
15:27:58 <zul> like something you might have seen on your travels perhaps
15:27:59 <soren> Only the nova smoketests.
15:28:03 <zul> ok
15:28:13 <soren> They just do a couple of simple things like start an instance, attach a volume, kill the instance..
15:28:13 <zul> Daviey: comment/
15:28:30 <zul> gotcha
15:28:47 <zul> so im going to move on to the last topic
15:28:55 <zul> #topic Progress/Openfloor
15:29:13 <Daviey> Our Automated/CI testing has made a great start!
15:29:23 <Daviey> using juju + orchestra
15:29:24 <zul> so imho we are a being a bit slow out of the gate but i think we are finding our feet
15:30:06 <zul> and with that the floor is open for comments questions discussions etc etc etc
15:30:49 <zul> Daviey/smoser/soren/ttx anything to say or discuss or mention?
15:31:08 <smoser> i think soren should fix the metadata service suckyness immediately
15:31:20 * smoser ducks
15:31:24 <zul> hehe
15:31:32 <Daviey> hah
15:31:34 <soren> smoser: It's my top priority. It's only surpassed by all the other stuff I'm working on.
15:31:45 <zul> and get paid to work on
15:31:47 <Daviey> I think we need to produce more stats on issues we suck at.
15:32:07 <zul> like a suck-o-meter
15:32:12 <smoser> soren, if you just want to test, how sucky the MD is,
15:32:24 <zul> http://www.suckometer.com/palm/index.html
15:32:25 <smoser> time python -c 'import boto; boto.utils.get_instance_metadata()'
15:34:03 <zul> and with that
15:34:06 <zul> #endmeeting