21:01:57 <russellb> #startmeeting nova 21:01:58 <openstack> Meeting started Thu Mar 28 21:01:57 2013 UTC. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:02:02 <openstack> The meeting name has been set to 'nova' 21:02:05 <ttx> o/ 21:02:05 <russellb> #chair vishy 21:02:06 <openstack> Current chairs: russellb vishy 21:02:06 <russellb> Who's around to talk nova? 21:02:20 <alaski> o/ 21:02:20 <alexpilotti> hi guys! 21:02:30 <dripton> hi 21:02:41 <dansmith> yo 21:02:41 <beagles> \o 21:02:57 <russellb> vishy: mikal: you guys here? would be good for talking about this fixed_ip quota on ugprade issue 21:03:00 <mikal> . 21:03:06 <russellb> yay mikal 21:03:09 <vishy> hi 21:03:09 <devananda> \o 21:03:13 <mikal> Sorry, just woke up 21:03:14 <russellb> perfect 21:03:37 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova 21:03:42 <russellb> #topic grizzly status 21:03:53 <russellb> #link https://launchpad.net/nova/+milestone/grizzly-rc2 21:04:13 <russellb> need to decide on plan for bug 1161190 21:04:14 <uvirtbot> Launchpad bug 1161190 in openstack-manuals "Fixed IPs quota can break upgrades" [Undecided,New] https://launchpad.net/bugs/1161190 21:04:21 <russellb> #link https://bugs.launchpad.net/bugs/1161190 21:04:22 <uvirtbot> Launchpad bug 1161190 in openstack-manuals "Fixed IPs quota can break upgrades" [Undecided,New] 21:04:39 <russellb> vishy: i didn't catch all of your earlier discussion with ttx ... 21:05:33 <russellb> last comment on the bug is: "I think our best option here is actually to change the default to -1 and note in the release notes that providers should set it to a reasonable value." 21:05:40 <mikal> So, we need to revist the default value for that quota in essex / folsom / grizzly 21:05:42 <vishy> well the main issue is breaking existing users is really not ok 21:06:03 <vishy> so the only option that won't break anyone is setting to -1 21:06:04 <russellb> your comment there seems pretty reasonable 21:06:18 <vishy> there are some hacky ways around it that will stop breakage in most situations 21:06:29 <vishy> but they are hard to backport. 21:06:31 <ttx> and document that it should be manually set to get protection 21:06:50 <mikal> vishy: backports are probably re-writes, given the delta in quota code between releases 21:07:14 <mikal> ttx: well, we suffered documentation fail with our first attempt at this... Apparently the quota isn't mentioned in the folsom release notes? 21:07:37 <russellb> mikal: but setting to -1 by default should be easy, right? 21:07:44 <mikal> I agree by the way, we should just set it to -1 in folsom / grizzly, and make sure its actually in the release notes 21:07:47 <mikal> russellb: yes 21:07:57 <mikal> We should also look at essex. Does it support -1s? 21:08:18 <ttx> folsom/essex are a bit off-topic right now 21:08:43 <mikal> Ok, but we need to fix them too. 21:08:43 <ttx> We need a grizzly solution fast, for the others we have more time 21:08:46 <russellb> so, who's going to write the grizzly patch? 21:08:49 <mikal> I will 21:08:57 <russellb> great, today? 21:09:00 <mikal> Sure 21:09:12 <russellb> k 21:09:26 * ttx would like to close the rc2 tomorrow and switch to regression-only mode 21:09:41 <russellb> looks like that was the only outstanding issue 21:09:47 <russellb> there's one other patch that was backported, but the gate is being mean 21:10:00 <ttx> unit test fail 21:10:07 <russellb> just rekicked 21:10:31 <russellb> #note grizzly-rc2 to be closed out tomorrow, switch to regression only mode 21:10:37 <russellb> anything else on grizzly status? 21:11:27 <ttx> vishy: are you ok with RC2 publication tomorrow if the page shows everything is ok ? 21:11:28 <russellb> #topic design summit 21:11:32 <russellb> #undo 21:11:33 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x374c0d0> 21:11:49 <ttx> woo, nice, "undo" 21:11:54 <russellb> heh, that didn't quite work 21:12:11 <dripton> needs a better __repr__ 21:12:24 <russellb> and needs to change the topic back :) 21:12:41 <ttx> vishy: I'll assume yes unless you scream 21:13:17 <russellb> k, feel free to come back to that topic later if needed ... 21:13:20 <ttx> russellb: you can continue, we can ack that offline 21:13:21 <russellb> #topic design summit 21:13:35 <russellb> Deadline for submissions is the end of this weekend, but i'm really hoping we have everything already 21:14:02 <russellb> as it's already quite full 21:14:28 <russellb> After today, if you would like to discuss design summit planning with me, please email me directly. i'm going to be on vacation until Apr 10th or so 21:14:43 <russellb> but will be working on summit planning in some stolen bits of time here and there 21:14:59 <russellb> that generally applies to anything you want my attention for 21:15:36 <russellb> #note Email russellb if you want his attention over the next couple weeks, on vacation, not on IRC 21:15:45 <mikal> Do you want a flag in the subject or anything to make it easier to search? 21:15:57 <mikal> [OMGWTFBBQ] Hope you're enjoying your holiday. 21:16:01 <russellb> :) 21:16:15 <russellb> whatever, be creative 21:16:27 <russellb> just email direct, don't assume i'll see stuff on list ... 21:17:07 <russellb> and of course we have the design summit soon to discuss anything big, so i don't think it'll be an issue 21:17:26 <russellb> that's pretty much all i wanted to mention ... how to contact me during final summit planning 21:17:37 <russellb> any last minute session proposals expected to come in? 21:17:49 <russellb> or questions? comments/suggestions? whatever? 21:18:05 <yjiang5_> russellb: when will you finalize the approve for the sessions? 21:18:29 <russellb> yjiang5_: before the design summit? :-) or whenever ttx says is the deadline ... 21:18:40 <russellb> yjiang5_: shouldn't take me too much longer once I know I have the final list of sessions 21:18:47 <yjiang5_> russellb: :-) 21:18:47 <russellb> yjiang5_: have a specific one you'd like to ask about? 21:19:02 <yjiang5_> russellb: the PCI/SR-IOV related stuff 21:19:25 <ttx> russellb: "before the design summit" sounds good 21:19:28 <ttx> :) 21:19:32 <russellb> ttx: awesome. 21:19:45 <russellb> ttx: heck i can pull that off by putting it off completely until after vacation! 21:20:04 <russellb> yjiang5_: that's on my list of "very likely to accept for a full session timeslot" 21:20:11 <ttx> russellb: the goal is to publish early in the week of April 8 21:20:24 <beagles> "Key Community Members Going on Holiday Near Important Conferences and How It Can Be Prevented" 21:20:27 <russellb> ttx: ok that's reasonable 21:20:34 <yjiang5_> russellb: thanks :) 21:20:47 <vishy> russellb: did you notice if there was a topic on volume migration? 21:20:48 <russellb> beagles: yeah ... i know ... i should have thought of this when i got married 5 years ago 21:20:58 <mikal> russellb: so lame 21:21:19 <vishy> russelb: i have an idea for it but it isn't worth a full session probably 21:21:44 <russellb> vishy: there's not. there's one on volume encryption, and there's one on making block device handling more awesome 21:21:56 <russellb> annnnd, let's see ... 21:22:04 <beagles> :) 21:22:41 <russellb> vishy: and "improving host maintenance and migrate APIs" 21:23:03 <russellb> vishy: http://summit.openstack.org/cfp/details/18 21:23:10 <russellb> sorta kinda related :-/ 21:23:21 <vishy> that one might actually cover it 21:23:30 <russellb> ok cool 21:23:45 <russellb> vishy: let me know if you think of anything else 21:23:50 <vishy> will do 21:24:02 <russellb> i may have to merge a bunch of scheduler sessions as "lock scheduler people in a room parts 1 and 2" 21:24:38 <russellb> too many good ideas for things to work on :) 21:24:44 <devananda> \o/ 21:25:07 <russellb> alrighty.. 21:25:11 <russellb> #topic bugs 21:25:21 <russellb> mikal would like to bang the bugs drum :) 21:25:44 <mikal> Right. 21:25:50 <mikal> So, we've fallen off the bug triage wage. 21:25:52 <mikal> wagon even 21:26:03 <mikal> We got lucky we noticed the folsom fixed ip upgrade bug as quick as we did. 21:26:11 * dansmith triaged several today 21:26:11 <mikal> So y'all should remember to triage bugs please. 21:26:17 <mikal> Fixing bugs is also nice. 21:26:19 <jog0> https://bugs.launchpad.net/nova/+bug/1161175 should be in Grizzly I think 21:26:21 <uvirtbot> Launchpad bug 1161175 in oslo "tools/conf/generate_sample.sh fails" [Undecided,New] 21:26:22 <russellb> #link http://webnumbr.com/untouched-nova-bugs 21:26:24 <mikal> dansmith: I did notice that 21:26:29 <dansmith> mikal: woohoo! 21:26:57 * jog0 is sorry for jumping at the wrong time 21:26:58 <mikal> That's all really. I just want to hassle people so they remember. 21:27:06 <mikal> jog0: I forgive you 21:27:11 <russellb> mikal: cool :) 21:27:22 <russellb> jog0: what's the effect, those options not in the sample config? 21:27:33 <jog0> russellb: yeah 21:28:05 <jog0> russellb: this is the change needed to be synced in https://review.openstack.org/#/c/25297/ 21:28:09 <russellb> yar ... seems safe enough if it can beat mikal's fixed_ip quotas patch, up to vishy though 21:28:52 <jog0> russellb: I'll get the patch ready now 21:29:19 <russellb> jog0: k, just sync up with vishy to see if he's good with it going into rc2 21:29:34 <jog0> russellb: *nod* 21:29:42 <russellb> so next is open discussion, and i have one question to start 21:29:45 <russellb> #topic open discussion 21:29:59 <russellb> I was talking to beagles earlier, he was asking me about floating IP behavior with nova-network 21:30:14 <russellb> and whether it is expected that instances can't talk to each other over those IPs if they're on the same compute node 21:30:31 <mikal> There has been at least one bug for this before 21:30:31 <russellb> as that's what he sees after trying it, it was reported by someone else here internally 21:30:31 <dansmith> see, 21:30:35 <mikal> I thought it was fixed? 21:30:41 <dansmith> I don't think that the instances should know their floating IPs 21:30:54 <russellb> could be the case, i should note that this was all folsom probably 21:30:59 <russellb> beagles: around right? 21:31:06 <beagles> russellb: yes 21:31:15 <mikal> I am sure that I saw vishy handle a bug like this a while ago... 21:31:28 <beagles> as it happens, I found this bug about 10 minutes ago... 21:31:42 <beagles> https://bugs.launchpad.net/nova/+bug/1096259 21:31:43 <uvirtbot> Launchpad bug 1096259 in nova "pinging own floating ip fails with external gateway" [Low,Fix released] 21:31:45 <beagles> or rather 10 minutes before this meet started... 21:31:51 <russellb> dansmith: well they don't, directly ... but it seems broken that you may or may not be able to connect to a given public IP, depending on something you're not supposed to care about (colocation of instances or not) 21:31:56 <mikal> I think I was thinking of https://review.openstack.org/#/c/21689/ 21:32:20 <russellb> mikal: nice 21:32:23 <dansmith> russellb: well, maybe 21:32:25 <russellb> and that's grizzly only 21:32:29 <mikal> https://bugs.launchpad.net/nova/+bug/1122335 21:32:31 <uvirtbot> Launchpad bug 1122335 in nova "Pinging a floating ip from an instance without floating can fail" [Low,Fix released] 21:32:35 <mikal> We could talk about backports though 21:32:43 <mikal> We know we need to do another folsom release because of fixed ip quota 21:32:45 <beagles> mikal: ooh, I think I saw this but figured I messed my config up 21:32:59 <russellb> beagles: note that patch isn't in folsom 21:33:14 * beagles nods 21:33:29 <russellb> beagles: wasn't sure if you were testing latest or not for this 21:33:39 <russellb> anyway, good references 21:34:07 <beagles> right: I was at folsom.. .for my purposes, these are good clues as to what was going on. There is a documentation note that is somewhat... 21:34:07 <beagles> discouraging 21:35:08 <russellb> beagles: k, so may be as simple as backporting that patch ... we can talk more later 21:35:37 <beagles> "Note that you cannot SSH to an instance with a public IP from within the same server as the routing configuration won't allow it. " 21:35:37 <beagles> in http://docs.openstack.org/trunk/openstack-compute/admin/content/network-troubleshooting.html 21:35:38 <beagles> granted it is somewhat ambiguous, but is this referring to a different case, or is it incorrect? 21:35:38 <beagles> russellb: ack 21:36:34 <russellb> sounds like it *should* work ... except dansmith has a fundamental disagreement that instances should ever be trying this anyway :-) 21:36:52 * dansmith stands firm 21:37:29 <russellb> dansmith: i expect nothing less, sir! 21:37:31 <beagles> :) I guess the case might be if that a client app somehow gets a URI (or similar) that has the public IP it should work without consideration of where that is 21:37:37 <beagles> <shrug> 21:37:51 <russellb> could be two different tenants entirely 21:37:52 <mikal> I think Dan is wrong to be honest 21:38:00 <russellb> that don't even know they're in the same cloud 21:38:04 <mikal> IPs should always work 21:38:08 <mikal> russellb: 'zactly 21:38:10 <russellb> mikal: that's how i feel 21:38:37 <dansmith> so, I get that, 21:39:02 <dansmith> but it seems to me that the floating IP functionality is really an edge of the network notion (regardless of where it's implemented) 21:39:33 <dansmith> that should be used for directing traffic, load balancing, etc and the mere fact that it can sometimes work means that people will treat it differently than that 21:39:46 <dansmith> they can't bind to it on the instance, so they shouldn't know anything about it, IMHO 21:39:57 <dansmith> but I definitely see the other argument, of course :) 21:40:41 * beagles does not entirely disagree with dansmith 21:40:59 <vishy> beagles: wait yes you can 21:41:11 <vishy> oh wait 21:41:33 <dripton> If the client can see the Internet, it can find its external IP no matter what we do, so obscuring it seems pointless. 21:41:35 <vishy> beagles: there are some configurations where public -> public for instances on the same host may fail 21:41:45 <beagles> but when asked why it doesn't work and trying to figure out if it can be made to work ... or not 21:42:17 <vishy> beagles: it can work in some situations 21:42:38 <russellb> a bug? or just fundamentally not going to work? 21:43:05 <beagles> vishy: is it generally know what the key factors for not-going-to-work/should-work are? 21:43:20 <beagles> s/know/known/ 21:43:28 <dansmith> dripton: it's not about whether it can determine it by colluding with an outside party, it's whether openstack's policy is that you're allowed to know it from an infrastructure point of view 21:43:37 <vishy> beagles: at one point i know most of them 21:44:26 <vishy> dansmith: i don't understand? why would we want to stop instances from knowing they have a floating ip? 21:44:58 <vishy> beagles: iirc in most situations it is a security group problem, not a problem with traffic routing 21:45:00 <russellb> and even if the instances don't know it, certainly the thing that deployed the application to the cloud knows it 21:45:14 <dansmith> vishy: well, maybe _we_ wouldn't I'm just saying it seems to _me_ that it's cleaner if they don't 21:45:19 <vishy> beagles: as in it will look like the traffic is coming from the instance's private ip instead of its public. 21:45:55 * beagles nods 21:46:08 <beagles> I suspect the DNAT/SNAT rules aren't entirely complete for this situation 21:46:08 <vishy> i suspect we expose floating ips via the metadata server 21:46:11 <vishy> but i don't remember 21:46:21 <vishy> beagles: i looked at what it would take to fix it, it is really hard 21:46:33 <vishy> we need to dynamically update ipsets for fixed and floating ips 21:46:44 <vishy> and use ipsets in the rules instead of numeric ips 21:46:58 <beagles> vishy: I managed to get it working at one point, but I had spent a few hours trying things out.. and I suspect I didn't undo something and couldn't find what the magic ingredient was 21:47:15 <vishy> it gets really hard because security groups specify ranges 21:47:32 <vishy> so you have to have special case rules if you are within a range 21:47:40 <vishy> i ultimately decided it wasn't worth implementing 21:48:08 <vishy> you'd have to do a fanout cast every time a floating ip moves telling everything to update their rules 21:48:10 * beagles nods 21:48:21 <beagles> ugh 21:48:30 <vishy> and you hve to look through every security group and say, do i fit in the range? if so special case a rule addition for the floating ip too. 21:48:48 <russellb> if it's incredibly invasive, might be worth chalking up as a nova-network limitation, and only worrying about it in quantum 21:48:53 <vishy> the good news is 21:48:58 <vishy> yeah that is my thought 21:49:03 * beagles nods 21:49:19 <vishy> that it is just security groups 21:49:35 <vishy> so you just need to tell people that if they wish to use security groups 21:49:48 <vishy> add rules for the floating range as well 21:50:12 <vishy> or even better 21:50:18 <vishy> don't use floating ips from instance to instance 21:50:24 <beagles> :) 21:50:27 <vishy> and use something else for your vip 21:50:32 <vishy> like an haproxy instance 21:51:03 <vishy> there is a known issue in the release notes for folsom about this 21:51:25 <vishy> and some info i sent to the list about it when i realized that I can't fix it easily. 21:51:35 <russellb> https://wiki.openstack.org/wiki/ReleaseNotes/Folsom#Known_Issues_2 21:51:47 <russellb> "Security Groups do not mix well with floating ips. Users should use fixed ips in security group rules. More information here" 21:51:57 <russellb> where here is http://lists.openstack.org/pipermail/openstack-dev/2012-September/001212.html 21:51:59 <russellb> nice 21:52:16 <beagles> cool 21:52:40 <russellb> thanks a bunch, vishy 21:52:56 <beagles> yes, thanks! 21:53:23 <vishy> np 21:53:31 <russellb> 7 minutes left if anyone else has a topic 21:53:37 <jog0> when was nova synced with oslo-incubator, when I did a sync of everything (of oslo grizzly and nova milestone) I get 71 lines cahnged 21:53:50 <russellb> jog0: very recently 21:54:20 <jog0> russellb: looking at the logs I didn't see a general sync in the milestone proposed branch 21:54:37 <russellb> jog0: i did one before milestone-proposed 21:54:46 <russellb> IIRC 21:54:52 <russellb> no wait, that was after ... 21:55:03 <russellb> yes, it was after. 21:55:14 <russellb> we've only been backporting specific things we knew we needed 21:55:28 <russellb> a lockutils thing ... some setup.py things 21:55:35 <russellb> that i recall 21:57:22 <dripton> As promised last week: https://blueprints.launchpad.net/nova/+spec/convert-to-alembic 21:57:28 <jog0> russellb: ahh its just a little strange to not sync with the stable release 21:57:51 <russellb> 5 more session proposals during this meeting I think 21:58:37 <russellb> dripton: nice, short on time to go into detail ... post to the -dev list to draw attention to code/wiki stuff once you have it to look at 21:58:46 <dripton> russellb: ack 22:00:06 <russellb> alright, time's up 22:00:08 <russellb> thanks everyone! 22:00:10 <russellb> #endmeeting