21:12:11 <dansmith> #startmeeting nova
21:12:12 <openstack> Meeting started Thu Jan 31 21:12:11 2013 UTC.  The chair is dansmith. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:12:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:12:15 <openstack> The meeting name has been set to 'nova'
21:12:17 <dansmith> #topic vm-ensembles
21:12:19 <vishy> sorry guys
21:12:23 <dansmith> yay
21:12:25 <dansmith> #chair vishy
21:12:26 <openstack> Current chairs: dansmith vishy
21:12:28 <vishy> i just ran out for lunch an i was late
21:12:38 <garyk> bon apetite
21:12:56 <hemna> we are just complaining about reviews in here anyway
21:12:57 <hemna> :P
21:13:18 <vishy> who all is here?
21:13:18 <dansmith> hemna: man, you really know how to get people to review your stuff, huh?
21:13:32 <hemna> dansmith, I was just joking
21:13:33 <vishy> i don't see comstud or markmc
21:14:03 <vishy> well lets discuss ensembles anyway
21:14:27 <garyk> where should we start?
21:14:41 <vishy> i think the biggest request so far is to identify what you can't do with the current scheduler data
21:14:57 <vishy> i.e. why is different_host not enough
21:15:37 <garyk> as we have shown in the document it is best that scheduler has the whole picture and does not doe it in a pience meal fasion
21:15:45 <garyk> (sorry it takes me a while to write)
21:16:16 <garyk> there can be cases when the scheduling could lead to false negative admission control
21:16:46 <garyk> more specifically when the host capacity allocation is high
21:17:08 <garyk> the best way to do this is do have the scheduler make a "group" scheduling decision.
21:17:32 <kmartin> #link https://docs.google.com/document/d/1bAMtkaIFn4ZSMqqsXjs_riXofuRvApa--qo4UTwsmhw/edit  (for the new people)
21:18:19 <garyk> we provide a solution which uses anti affinity to ensure server fault tolerance
21:18:31 <garyk> sorry, host fault tolerance
21:18:41 <vishy> garyk: so the question is: is that small failure scenario enough to warrant the extra complication
21:19:18 <vishy> garyk: it will still have to have some pretty smart logic in how to schedule the whole group and deal with all sorts of weird race conditions
21:19:28 <vishy> garyk: so it may not even be feasible to do so
21:19:55 <garyk> vishy: we feel that this can occur, the false negative, quite often.
21:20:13 <garyk> especially when the host allocation capacity is high
21:20:17 <dansmith> hmm, I thought the goal of this stuff was something different, other than fault tolerance
21:20:52 <garyk> there are 2 goals:
21:21:05 <garyk> we started with high availability via anti affinity
21:21:22 <garyk> and we are strving to add network proximity to improve performance
21:21:28 <garyk> so it will be HA and performance
21:22:44 <garyk> HA and performance are the major aspects of service level or quality of service
21:23:39 <garyk> vishy: the proposed solution lowers the chance of problems. it does not prevent them complete;ly
21:23:59 <vishy> i'm not totally convinced that it is better to do this via complicating the scheduler vs doing simple hints and try, retry
21:24:16 <glikson> in fact, I think the proximity scenario (e.g., for HPC or Hadoop workloads) is a rather strong case to require group scheduling, which can not be easily achieved with sequential provisioning..
21:24:47 <vishy> glikson: but in that case why not just do a hint for host aggregates
21:25:15 <vishy> I think trying to make the scheduler too smart will lead to a lot of hard to find bugs
21:25:39 <glikson> well, how would you ensure that the first instance is provisioned in a location wit henough capacity "nearby"?
21:25:46 <vishy> glikson: you wouldn't
21:25:56 <vishy> if there isn't enough capacity it would fail and try again
21:25:56 <garyk> vishy: all of the inaformation can be passed by hints. the sceduling hould just be done at one time.
21:26:58 <glikson> vishy: how retries would help? wouldn't the weighting function return the same result?
21:27:03 <vishy> garyk: i suppose we already do group scheduling, can we do this by just using the currentmulti_boot code?
21:27:47 <vishy> as in simply change how instances are allocated when you do -n 5
21:27:47 <garyk> vishy: this is possible. one would just need to know when the group ends.
21:27:51 <vishy> that seems simple enough
21:28:23 <vishy> garyk: nova boot -n 5 --hint same-network
21:28:27 <vishy> or something like that
21:28:30 <garyk> vishy: there are 2 issues with this
21:29:16 <garyk> 1. this is done with the same type of vm's and does not cater for heterogenous groups of vms (like we provided in the document)
21:29:50 <garyk> 2. the current implementation does not have anti affinity support for the group opertaion (which is a very simple patch)
21:30:05 <vishy> so i would suggest we do 2 first
21:30:15 <garyk> i think that the first case has more weight in the sense that if thescheduler has the whole pcture then it can make te best decision.
21:30:19 <vishy> that seems like an obvious win
21:30:48 <vishy> then i guess we need to support passing multiple requests in together
21:31:02 <garyk> #2 is similar t what has already been implemented int he proposed review.
21:31:13 <vishy> russellb: isn't really here to complain :)
21:31:23 <garyk> it just extends that to be more than one group
21:31:29 <garyk> russellb: is on a plane
21:31:35 <vishy> can you link the review btw?
21:31:43 <garyk> https://review.openstack.org/#/c/19906/
21:32:17 <garyk> we are sitting at -2 due to not supporting cells. but i think that it does support cells (i need a claficication from chris)
21:35:10 <dansmith> shall we move on?
21:35:38 <garyk> dansmith: can we just continue with one more issue
21:36:00 <dansmith> garyk: if you've got one, it just seemed like the conversation had faded out
21:36:59 <garyk> i was wondering if we need to talk about storing the group information. the initial plan was to store it int he meta data
21:37:05 <vishy> aye
21:37:16 <vishy> ok we will wait for comstud
21:37:22 <garyk> what i am trying to say is that we need to store the group information
21:37:34 <vishy> garyk: why do we need to store it?
21:38:00 <garyk> vishy: we need to store it for future use.
21:38:19 <garyk> vishy: that is future rescheduling of vm's that need to comply with the policy
21:38:57 <glikson> or adding more VMs that should comply to the same policy (group elasticity)
21:39:19 <garyk> an example of this is host maintenace. when moving the avtice vm's we should still strive for the network proximity an affinity
21:39:58 <vishy> garyk: ok we need to save hints as well in that case
21:40:02 <glikson> garyk: host maintenance is a bit sensitive topic now, if you follow the review on it :-)
21:40:05 <vishy> storing it in system_metadata seems ok
21:40:33 <garyk> vishy: that works for us at the moment.
21:40:34 <vishy> #link novaclient/shell.py
21:40:43 <vishy> #link https://review.openstack.org/#/c/19906/
21:40:53 <vishy> #action comstud to give feedback on cells
21:41:11 <vishy> #action initial patch to simply extend multi-boot to schedule with hint
21:41:52 <vishy> #Info initial patch to simply extend multi-boot to schedule with hint
21:42:09 <garyk> vishy: for that we will need first and last (i.e. be able to do with heterogenous vm's)
21:42:19 <vishy> #info later patches to add support for heterogeneous guests
21:42:37 <vishy> garyk right either first and last or pass the whole list at once
21:42:48 <vishy> personally thinking the whole list at once will be cleaner
21:42:53 <vishy> ok lets move on
21:43:04 <garyk> vishy: ok. thanks!
21:43:09 <vishy> #info http://wiki.openstack.org/Meetings/Nova
21:43:13 * dansmith added to the agenda a minute ago
21:43:16 <vishy> #topic Full Tempest Gate
21:43:24 <vishy> sdague: around?
21:43:31 <dansmith> vishy: so, this is me fighting sdague's battle
21:43:38 <dansmith> vishy: he wants you to go +1 that patch :)
21:43:55 <dansmith> #link https://etherpad.openstack.org/MissingTempestTests
21:44:48 <vishy> that is very cool
21:44:58 <dansmith> dammit
21:45:05 <dansmith> what is wrong with me and my link pasting?
21:45:10 <dansmith> #link https://review.openstack.org/#/c/20762/
21:45:41 <dansmith> I'm like some sort of blabbering idiot this week.. must be the post-vacation syndrome
21:46:14 <hemna> what's a vacation?
21:46:34 <dansmith> vishy: see the +1, thanks, we can move on now :)
21:46:35 <vishy> so anything else to discuss there
21:46:40 <dansmith> not if you're good with it
21:46:53 <vishy> that is a pretty short list for missing tests
21:47:02 <vishy> tempest has come a long way
21:47:11 <dansmith> they're coverage-based missing tests :)
21:47:18 <vishy> #topic Grizzly-3 review
21:48:19 <vishy> #link https://launchpad.net/nova/+milestone/grizzly-3
21:48:25 <vishy> only 50 blueprints!
21:48:36 <vishy> :o
21:48:51 <vishy> so mainly it is review review review
21:48:57 <dansmith> no-db-compute is getting really damn close
21:49:02 <vishy> if there are any that you know aren't going to make it let me know
21:49:22 <vishy> dansmith: don't forget about set-password
21:49:23 <vishy> :)
21:49:39 <dansmith> vishy: yeah,. it's on the list
21:49:43 <hemna> I'm trying to do more reviews of other folks' stuff to help out
21:49:44 <vishy> also we lost our volunteer for https://blueprints.launchpad.net/nova/+spec/multi-boot-instance-naming
21:49:53 <vishy> so if anyone wants to grab the patch and clean it up...
21:49:54 <dansmith> vishy: help me reduce my to-be-reviewed queue and I'll get to it sooner :D
21:50:27 <vishy> i would also really like https://blueprints.launchpad.net/nova/+spec/improve-block-device-handling to make it
21:50:39 <vishy> there has been a lot of discussion on the ml
21:51:31 <vishy> i also see that there are 3 new unapproved bps
21:51:48 <kmartin> vishy: what's the process for cinder folks helping with nova reviews?
21:51:53 <vishy> this one needs reviews https://blueprints.launchpad.net/nova/+spec/optimize-nova-network
21:52:02 <vishy> kmartin: feel free to review stuff
21:52:43 <vishy> i'm going to try and do lots of reviews over the next few days
21:53:13 <hemna> thanks for the suggestions, I'll look at those as well fwiw
21:53:27 <dansmith> I did a bunch today hoping to win karma for my own patches :)
21:53:49 <dansmith> most of which are just a single tick away
21:54:02 <vishy> #topic bugs
21:54:15 <kmartin> vishy: ok, should we just comment or give out -1  or +1 without have a bunch of experience in nova I don't want to step on anyones toes?
21:54:19 <vishy> #link http://webnumbr.com/untouched-nova-bugs
21:54:23 <vishy> we are going up a bit
21:54:32 <vishy> so we need to remember to review those
21:54:41 <dansmith> not too bad for this point in the cycle though, it seems
21:54:51 <vishy> kmartin: always give +1 -1 if you feel it is appropriate
21:54:58 <vishy> also we have lots of bugs in novaclient
21:55:27 <vishy> #link https://bugs.launchpad.net/python-novaclient/+bugs?search=Search&field.status=New
21:55:36 <vishy> free cookies
21:55:38 <vishy> :)
21:56:12 <vishy> please help triage
21:56:25 <vishy> i realize we will be switching focus to bugs soon but lets not let it get too far
21:56:30 <vishy> #topic open discussion
21:56:36 <vishy> anything else?
21:56:44 <cburgess> vishy: I would like to discuss https://blueprints.launchpad.net/nova/+spec/deferred-instance-deletes if possible.
21:57:21 <cburgess> We have been working on a draft review for it based upon how we implemented it in our essex product.
21:57:46 <vishy> cburgess: ok shoot
21:57:52 <cburgess> Wanted to see if anyone has any opinions on the blueprint before we post the review.
21:57:59 <cburgess> At preset we are implementing 2 flags.
21:58:13 <cburgess> 1 to control turing it off/on and another to specifty the location to store the deleted instances.
21:58:32 <vishy> cburgess: i don't see the benefit of a new option over the current soft delete implementation
21:58:36 <cburgess> its up to each driver to implement support. We have a periodic job in the manager to do the cleanup that again each driver that wants to support it needs to implement.
21:58:56 <cburgess> Current soft deleted does not cover the 2nd and 3rd condition we outline in the blueprint.
21:59:13 <cburgess> Errors arrising from live migrations and user issuing a "stop" and "start" action via the API.
21:59:15 <vishy> condition 3 doesn't exist anymore
21:59:23 <vishy> stop and start doesn't delete the instance
21:59:26 <cburgess> Ahh ok cool. We shall remove that then.
21:59:35 <cburgess> It doesn't re-image anymore?
21:59:37 <vishy> so i would prefer solving condition 2 by another means
21:59:49 <vishy> cburgess: no it just powers off the vm
21:59:53 <cburgess> Excellent.
22:00:18 <cburgess> Hmm well I guess we could add some logic into the post migrate to do an m5sum on the disk images or something before cleaning up on the source.
22:00:26 <vishy> cburgess: i would repurpose your stuff as configuring how long to keep the vm around during migrate
22:00:33 <vishy> if it is a real issue
22:00:48 <vishy> md5 sum would be a great idea, but since it is live...
22:00:52 <vishy> they won't match
22:01:05 <cburgess> We've seen it happen a few times. Usually it involves also triggering some type of kvm/qemu or libvirt bug the result is the same, data gets nuked.
22:01:23 <vishy> cburgess: so we should keep migrated disks around for a while
22:01:28 <cburgess> Yeah thats the issue I was thinking. I don't know of a truly unique fingerprint/checksum we could use on the disk.
22:01:31 <vishy> cburgess: or...
22:01:48 <vishy> we could force a resize_confirm like we do with non-live migrate
22:01:57 <cburgess> vishy: I suppose we could extend soft delete to also cover migrated. Only issue there is we couldn't use the DB record since the instance isn't in a soft deleted state.
22:02:46 <cburgess> vishy: I will take a look at resize_confirm and see what it provides.
22:02:54 <vishy> cburgess: we've been discussing reorganizing the migrate code so resize/migrate share code with live/block migration
22:02:59 <vishy> so if we do that
22:03:09 <vishy> then we can force a confirmation step on block migrate as well
22:03:32 <vishy> although i hate having to work around qemu/libvirt bugs like that :)
22:03:39 <vishy> anyway we should probably end the meeting
22:03:44 <vishy> cburgess: you have enough to go on?
22:03:56 <cburgess> vishy: I'm not a fan of working around bugs either but we simply can't loose our clients data if we can prevent.
22:04:02 <vishy> #endmeeting