21:01:57 <russellb> #startmeeting nova
21:01:58 <openstack> Meeting started Thu Mar 28 21:01:57 2013 UTC.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:02 <openstack> The meeting name has been set to 'nova'
21:02:05 <ttx> o/
21:02:05 <russellb> #chair vishy
21:02:06 <openstack> Current chairs: russellb vishy
21:02:06 <russellb> Who's around to talk nova?
21:02:20 <alaski> o/
21:02:20 <alexpilotti> hi guys!
21:02:30 <dripton> hi
21:02:41 <dansmith> yo
21:02:41 <beagles> \o
21:02:57 <russellb> vishy: mikal: you guys here?  would be good for talking about this fixed_ip quota on ugprade issue
21:03:00 <mikal> .
21:03:06 <russellb> yay mikal
21:03:09 <vishy> hi
21:03:09 <devananda> \o
21:03:13 <mikal> Sorry, just woke up
21:03:14 <russellb> perfect
21:03:37 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova
21:03:42 <russellb> #topic grizzly status
21:03:53 <russellb> #link https://launchpad.net/nova/+milestone/grizzly-rc2
21:04:13 <russellb> need to decide on plan for bug 1161190
21:04:14 <uvirtbot> Launchpad bug 1161190 in openstack-manuals "Fixed IPs quota can break upgrades" [Undecided,New] https://launchpad.net/bugs/1161190
21:04:21 <russellb> #link https://bugs.launchpad.net/bugs/1161190
21:04:22 <uvirtbot> Launchpad bug 1161190 in openstack-manuals "Fixed IPs quota can break upgrades" [Undecided,New]
21:04:39 <russellb> vishy: i didn't catch all of your earlier discussion with ttx ...
21:05:33 <russellb> last comment on the bug is: "I think our best option here is actually to change the default to -1 and note in the release notes that providers should set it to a reasonable value."
21:05:40 <mikal> So, we need to revist the default value for that quota in essex / folsom / grizzly
21:05:42 <vishy> well the main issue is breaking existing users is really not ok
21:06:03 <vishy> so the only option that won't break anyone is setting to -1
21:06:04 <russellb> your comment there seems pretty reasonable
21:06:18 <vishy> there are some hacky ways around it that will stop breakage in most situations
21:06:29 <vishy> but they are hard to backport.
21:06:31 <ttx> and document that it should be manually set to get protection
21:06:50 <mikal> vishy: backports are probably re-writes, given the delta in quota code between releases
21:07:14 <mikal> ttx: well, we suffered documentation fail with our first attempt at this... Apparently the quota isn't mentioned in the folsom release notes?
21:07:37 <russellb> mikal: but setting to -1 by default should be easy, right?
21:07:44 <mikal> I agree by the way, we should just set it to -1 in folsom / grizzly, and make sure its actually in the release notes
21:07:47 <mikal> russellb: yes
21:07:57 <mikal> We should also look at essex. Does it support -1s?
21:08:18 <ttx> folsom/essex are a bit off-topic right now
21:08:43 <mikal> Ok, but we need to fix them too.
21:08:43 <ttx> We need a grizzly solution fast, for the others we have more time
21:08:46 <russellb> so, who's going to write the grizzly patch?
21:08:49 <mikal> I will
21:08:57 <russellb> great, today?
21:09:00 <mikal> Sure
21:09:12 <russellb> k
21:09:26 * ttx would like to close the rc2 tomorrow and switch to regression-only mode
21:09:41 <russellb> looks like that was the only outstanding issue
21:09:47 <russellb> there's one other patch that was backported, but the gate is being mean
21:10:00 <ttx> unit test fail
21:10:07 <russellb> just rekicked
21:10:31 <russellb> #note grizzly-rc2 to be closed out tomorrow, switch to regression only mode
21:10:37 <russellb> anything else on grizzly status?
21:11:27 <ttx> vishy: are you ok with RC2 publication tomorrow if the page shows everything is ok ?
21:11:28 <russellb> #topic design summit
21:11:32 <russellb> #undo
21:11:33 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x374c0d0>
21:11:49 <ttx> woo, nice, "undo"
21:11:54 <russellb> heh, that didn't quite work
21:12:11 <dripton> needs a better __repr__
21:12:24 <russellb> and needs to change the topic back :)
21:12:41 <ttx> vishy: I'll assume yes unless you scream
21:13:17 <russellb> k, feel free to come back to that topic later if needed ...
21:13:20 <ttx> russellb: you can continue, we can ack that offline
21:13:21 <russellb> #topic design summit
21:13:35 <russellb> Deadline for submissions is the end of this weekend, but i'm really hoping we have everything already
21:14:02 <russellb> as it's already quite full
21:14:28 <russellb> After today, if you would like to discuss design summit planning with me, please email me directly.  i'm going to be on vacation until Apr 10th or so
21:14:43 <russellb> but will be working on summit planning in some stolen bits of time here and there
21:14:59 <russellb> that generally applies to anything you want my attention for
21:15:36 <russellb> #note Email russellb if you want his attention over the next couple weeks, on vacation, not on IRC
21:15:45 <mikal> Do you want a flag in the subject or anything to make it easier to search?
21:15:57 <mikal> [OMGWTFBBQ] Hope you're enjoying your holiday.
21:16:01 <russellb> :)
21:16:15 <russellb> whatever, be creative
21:16:27 <russellb> just email direct, don't assume i'll see stuff on list ...
21:17:07 <russellb> and of course we have the design summit soon to discuss anything big, so i don't think it'll be an issue
21:17:26 <russellb> that's pretty much all i wanted to mention ... how to contact me during final summit planning
21:17:37 <russellb> any last minute session proposals expected to come in?
21:17:49 <russellb> or questions?  comments/suggestions?  whatever?
21:18:05 <yjiang5_> russellb: when will you finalize the approve for the sessions?
21:18:29 <russellb> yjiang5_: before the design summit?  :-)  or whenever ttx says is the deadline ...
21:18:40 <russellb> yjiang5_: shouldn't take me too much longer once I know I have the final list of sessions
21:18:47 <yjiang5_> russellb: :-)
21:18:47 <russellb> yjiang5_: have a specific one you'd like to ask about?
21:19:02 <yjiang5_> russellb: the PCI/SR-IOV related stuff
21:19:25 <ttx> russellb: "before the design summit" sounds good
21:19:28 <ttx> :)
21:19:32 <russellb> ttx: awesome.
21:19:45 <russellb> ttx: heck i can pull that off by putting it off completely until after vacation!
21:20:04 <russellb> yjiang5_: that's on my list of "very likely to accept for a full session timeslot"
21:20:11 <ttx> russellb: the goal is to publish early in the week of April 8
21:20:24 <beagles> "Key Community Members Going on Holiday Near Important Conferences and How It Can Be Prevented"
21:20:27 <russellb> ttx: ok that's reasonable
21:20:34 <yjiang5_> russellb: thanks :)
21:20:47 <vishy> russellb: did you notice if there was a topic on volume migration?
21:20:48 <russellb> beagles: yeah ... i know ... i should have thought of this when i got married 5 years ago
21:20:58 <mikal> russellb: so lame
21:21:19 <vishy> russelb: i have an idea for it but it isn't worth a full session probably
21:21:44 <russellb> vishy: there's not.  there's one on volume encryption, and there's one on making block device handling more awesome
21:21:56 <russellb> annnnd, let's see ...
21:22:04 <beagles> :)
21:22:41 <russellb> vishy: and "improving host maintenance and migrate APIs"
21:23:03 <russellb> vishy: http://summit.openstack.org/cfp/details/18
21:23:10 <russellb> sorta kinda related :-/
21:23:21 <vishy> that one might actually cover it
21:23:30 <russellb> ok cool
21:23:45 <russellb> vishy: let me know if you think of anything else
21:23:50 <vishy> will do
21:24:02 <russellb> i may have to merge a bunch of scheduler sessions as "lock scheduler people in a room parts 1 and 2"
21:24:38 <russellb> too many good ideas for things to work on :)
21:24:44 <devananda> \o/
21:25:07 <russellb> alrighty..
21:25:11 <russellb> #topic bugs
21:25:21 <russellb> mikal would like to bang the bugs drum :)
21:25:44 <mikal> Right.
21:25:50 <mikal> So, we've fallen off the bug triage wage.
21:25:52 <mikal> wagon even
21:26:03 <mikal> We got lucky we noticed the folsom fixed ip upgrade bug as quick as we did.
21:26:11 * dansmith triaged several today
21:26:11 <mikal> So y'all should remember to triage bugs please.
21:26:17 <mikal> Fixing bugs is also nice.
21:26:19 <jog0> https://bugs.launchpad.net/nova/+bug/1161175  should be in Grizzly  I think
21:26:21 <uvirtbot> Launchpad bug 1161175 in oslo "tools/conf/generate_sample.sh fails" [Undecided,New]
21:26:22 <russellb> #link http://webnumbr.com/untouched-nova-bugs
21:26:24 <mikal> dansmith: I did notice that
21:26:29 <dansmith> mikal: woohoo!
21:26:57 * jog0 is sorry for jumping at the wrong time
21:26:58 <mikal> That's all really. I just want to hassle people so they remember.
21:27:06 <mikal> jog0: I forgive you
21:27:11 <russellb> mikal: cool :)
21:27:22 <russellb> jog0: what's the effect, those options not in the sample config?
21:27:33 <jog0> russellb: yeah
21:28:05 <jog0> russellb: this is the change needed to be synced in https://review.openstack.org/#/c/25297/
21:28:09 <russellb> yar ... seems safe enough if it can beat mikal's fixed_ip quotas patch, up to vishy though
21:28:52 <jog0> russellb: I'll get the patch ready now
21:29:19 <russellb> jog0: k, just sync up with vishy to see if he's good with it going into rc2
21:29:34 <jog0> russellb: *nod*
21:29:42 <russellb> so next is open discussion, and i have one question to start
21:29:45 <russellb> #topic open discussion
21:29:59 <russellb> I was talking to beagles earlier, he was asking me about floating IP behavior with nova-network
21:30:14 <russellb> and whether it is expected that instances can't talk to each other over those IPs if they're on the same compute node
21:30:31 <mikal> There has been at least one bug for this before
21:30:31 <russellb> as that's what he sees after trying it, it was reported by someone else here internally
21:30:31 <dansmith> see,
21:30:35 <mikal> I thought it was fixed?
21:30:41 <dansmith> I don't think that the instances should know their floating IPs
21:30:54 <russellb> could be the case, i should note that this was all folsom probably
21:30:59 <russellb> beagles: around right?
21:31:06 <beagles> russellb: yes
21:31:15 <mikal> I am sure that I saw vishy handle a bug like this a while ago...
21:31:28 <beagles> as it happens, I found this bug about 10 minutes ago...
21:31:42 <beagles> https://bugs.launchpad.net/nova/+bug/1096259
21:31:43 <uvirtbot> Launchpad bug 1096259 in nova "pinging own floating ip fails with external gateway" [Low,Fix released]
21:31:45 <beagles> or rather 10 minutes before this meet started...
21:31:51 <russellb> dansmith: well they don't, directly ... but it seems broken that you may or may not be able to connect to a given public IP, depending on something you're not supposed to care about (colocation of instances or not)
21:31:56 <mikal> I think I was thinking of https://review.openstack.org/#/c/21689/
21:32:20 <russellb> mikal: nice
21:32:23 <dansmith> russellb: well, maybe
21:32:25 <russellb> and that's grizzly only
21:32:29 <mikal> https://bugs.launchpad.net/nova/+bug/1122335
21:32:31 <uvirtbot> Launchpad bug 1122335 in nova "Pinging a floating ip from an instance without floating can fail" [Low,Fix released]
21:32:35 <mikal> We could talk about backports though
21:32:43 <mikal> We know we need to do another folsom release because of fixed ip quota
21:32:45 <beagles> mikal: ooh, I think I saw this but figured I messed my config up
21:32:59 <russellb> beagles: note that patch isn't in folsom
21:33:14 * beagles nods
21:33:29 <russellb> beagles: wasn't sure if you were testing latest or not for this
21:33:39 <russellb> anyway, good references
21:34:07 <beagles> right: I was at folsom.. .for my purposes, these are good clues as to what was going on. There is a documentation note that is somewhat...
21:34:07 <beagles> discouraging
21:35:08 <russellb> beagles: k, so may be as simple as backporting that patch ... we can talk more later
21:35:37 <beagles> "Note that you cannot SSH to an instance with a public IP from within the same server as the routing configuration won't allow it. "
21:35:37 <beagles> in http://docs.openstack.org/trunk/openstack-compute/admin/content/network-troubleshooting.html
21:35:38 <beagles> granted it is somewhat ambiguous, but is this referring to a different case, or is it incorrect?
21:35:38 <beagles> russellb: ack
21:36:34 <russellb> sounds like it *should* work ... except dansmith has a fundamental disagreement that instances should ever be trying this anyway :-)
21:36:52 * dansmith stands firm
21:37:29 <russellb> dansmith: i expect nothing less, sir!
21:37:31 <beagles> :) I guess the case might be if that a client app somehow gets a URI (or similar) that has the public IP it should work without consideration of where that is
21:37:37 <beagles> <shrug>
21:37:51 <russellb> could be two different tenants entirely
21:37:52 <mikal> I think Dan is wrong to be honest
21:38:00 <russellb> that don't even know they're in the same cloud
21:38:04 <mikal> IPs should always work
21:38:08 <mikal> russellb: 'zactly
21:38:10 <russellb> mikal: that's how i feel
21:38:37 <dansmith> so, I get that,
21:39:02 <dansmith> but it seems to me that the floating IP functionality is really an edge of the network notion (regardless of where it's implemented)
21:39:33 <dansmith> that should be used for directing traffic, load balancing, etc and the mere fact that it can sometimes work means that people will treat it differently than that
21:39:46 <dansmith> they can't bind to it on the instance, so they shouldn't know anything about it, IMHO
21:39:57 <dansmith> but I definitely see the other argument, of course :)
21:40:41 * beagles does not entirely disagree with dansmith
21:40:59 <vishy> beagles: wait yes you can
21:41:11 <vishy> oh wait
21:41:33 <dripton> If the client can see the Internet, it can find its external IP no matter what we do, so obscuring it seems pointless.
21:41:35 <vishy> beagles: there are some configurations where public -> public for instances on the same host may fail
21:41:45 <beagles> but when asked why it doesn't work and trying to figure out if it can be made to work ... or not
21:42:17 <vishy> beagles: it can work in some situations
21:42:38 <russellb> a bug?  or just fundamentally not going to work?
21:43:05 <beagles> vishy: is it generally know what the key factors for not-going-to-work/should-work are?
21:43:20 <beagles> s/know/known/
21:43:28 <dansmith> dripton: it's not about whether it can determine it by colluding with an outside party, it's whether openstack's policy is that you're allowed to know it from an infrastructure point of view
21:43:37 <vishy> beagles: at one point i know most of them
21:44:26 <vishy> dansmith: i don't understand? why would we want to stop instances from knowing they have a floating ip?
21:44:58 <vishy> beagles: iirc in most situations it is a security group problem, not a problem with traffic routing
21:45:00 <russellb> and even if the instances don't know it, certainly the thing that deployed the application to the cloud knows it
21:45:14 <dansmith> vishy: well, maybe _we_ wouldn't I'm just saying it seems to _me_ that it's cleaner if they don't
21:45:19 <vishy> beagles: as in it will look like the traffic is coming from the instance's private ip instead of its public.
21:45:55 * beagles nods
21:46:08 <beagles> I suspect the DNAT/SNAT rules aren't entirely complete for this situation
21:46:08 <vishy> i suspect we expose floating ips via the metadata server
21:46:11 <vishy> but i don't remember
21:46:21 <vishy> beagles: i looked at what it would take to fix it, it is really hard
21:46:33 <vishy> we need to dynamically update ipsets for fixed and floating ips
21:46:44 <vishy> and use ipsets in the rules instead of numeric ips
21:46:58 <beagles> vishy:  I managed to get it working at one point, but I had spent a few hours trying things out.. and I suspect I didn't undo something and couldn't find what the magic ingredient was
21:47:15 <vishy> it gets really hard because security groups specify ranges
21:47:32 <vishy> so you have to have special case rules if you are within a range
21:47:40 <vishy> i ultimately decided it wasn't worth implementing
21:48:08 <vishy> you'd have to do a fanout cast every time a floating ip moves telling everything to update their rules
21:48:10 * beagles nods
21:48:21 <beagles> ugh
21:48:30 <vishy> and you hve to look through every security group and say, do i fit in the range? if so special case a rule addition for the floating ip too.
21:48:48 <russellb> if it's incredibly invasive, might be worth chalking up as a nova-network limitation, and only worrying about it in quantum
21:48:53 <vishy> the good news is
21:48:58 <vishy> yeah that is my thought
21:49:03 * beagles nods
21:49:19 <vishy> that it is just security groups
21:49:35 <vishy> so you just need to tell people that if they wish to use security groups
21:49:48 <vishy> add rules for the floating range as well
21:50:12 <vishy> or even better
21:50:18 <vishy> don't use floating ips from instance to instance
21:50:24 <beagles> :)
21:50:27 <vishy> and use something else for your vip
21:50:32 <vishy> like an haproxy instance
21:51:03 <vishy> there is a known issue in the release notes for folsom about this
21:51:25 <vishy> and some info i sent to the list about it when i realized that I can't fix it easily.
21:51:35 <russellb> https://wiki.openstack.org/wiki/ReleaseNotes/Folsom#Known_Issues_2
21:51:47 <russellb> "Security Groups do not mix well with floating ips. Users should use fixed ips in security group rules. More information here"
21:51:57 <russellb> where here is http://lists.openstack.org/pipermail/openstack-dev/2012-September/001212.html
21:51:59 <russellb> nice
21:52:16 <beagles> cool
21:52:40 <russellb> thanks a bunch, vishy
21:52:56 <beagles> yes, thanks!
21:53:23 <vishy> np
21:53:31 <russellb> 7 minutes left if anyone else has a topic
21:53:37 <jog0> when was nova synced with oslo-incubator, when I did a sync of everything (of oslo grizzly and nova milestone) I get 71 lines cahnged
21:53:50 <russellb> jog0: very recently
21:54:20 <jog0> russellb: looking at the logs I didn't see a general sync in the milestone proposed branch
21:54:37 <russellb> jog0: i did one before milestone-proposed
21:54:46 <russellb> IIRC
21:54:52 <russellb> no wait, that was after ...
21:55:03 <russellb> yes, it was after.
21:55:14 <russellb> we've only been backporting specific things we knew we needed
21:55:28 <russellb> a lockutils thing ... some setup.py things
21:55:35 <russellb> that i recall
21:57:22 <dripton> As promised last week: https://blueprints.launchpad.net/nova/+spec/convert-to-alembic
21:57:28 <jog0> russellb: ahh its just a little strange to not sync with the stable release
21:57:51 <russellb> 5 more session proposals during this meeting I think
21:58:37 <russellb> dripton: nice, short on time to go into detail ... post to the -dev list to draw attention to code/wiki stuff once you have it to look at
21:58:46 <dripton> russellb: ack
22:00:06 <russellb> alright, time's up
22:00:08 <russellb> thanks everyone!
22:00:10 <russellb> #endmeeting