14:01:13 #startmeeting nova 14:01:14 Meeting started Thu Oct 1 14:01:13 2015 UTC and is due to finish in 60 minutes. The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:16 ................o.../ 14:01:17 The meeting name has been set to 'nova' 14:01:18 hi 14:01:18 \o 14:01:19 \o/ 14:01:20 o/ 14:01:20 \..........................o 14:01:20 o/ 14:01:23 \o/ 14:01:25 bauzas: :P 14:01:25 o/ 14:01:26 o/ 14:01:27 hello 14:01:29 o/ 14:01:29 o/ 14:01:30 some long arms in here 14:01:33 o\ 14:01:36 #link https://wiki.openstack.org/wiki/Meetings/Nova 14:01:36 o/ 14:01:41 \o 14:01:45 \o 14:01:46 #topic Liberty Status 14:01:53 hello everyone 14:02:00 so we have RC1 out and being tested 14:02:19 we are waiting for RC2 to open, to include more translations, and a few stragglers 14:02:35 nothing serious enough to make us release another RC before the one we needed for translation updates 14:02:45 #link https://bugs.launchpad.net/nova/+bugs?field.tag=liberty-rc-potential 14:02:50 thats the watch list 14:02:58 any questions? 14:03:36 OK, well keep eyes peeled for problems, now is the time to find them and fix them 14:03:40 #topic Mitaka Status 14:03:48 #info master is open for mitaka 14:03:58 so the design summit sessions 14:04:04 hello 14:04:08 #info Deadline for session ideas: Tuesday 6th October, 23.59 UTC 14:04:17 #info Link for submissions: http://goo.gl/forms/D2Qk8XGhZ6 14:04:40 #info notes for those who can't get to google https://etherpad.openstack.org/p/mitaka-nova-summit-suggestions 14:04:48 OK... 14:05:06 does that date work OK for folks, there was my ML post about why I chose that date 14:05:29 basically to give us time to choose before the final lock down of sessions the following thursday 14:05:41 +1 14:05:50 so attempting to get folks to break the specs into catagories 14:05:55 makes sense 14:06:06 the idea being we make sure we don't miss any super important specs 14:06:07 #link https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking 14:06:12 thats where thats happening ^ 14:06:22 I've been using that, 14:06:25 and it's been quite helpful 14:06:35 I made it through a large chunk of the specs on that page this week 14:06:42 the urgent thing is looking for anything thats are really important, and will need in person discussion I guess 14:06:49 sweet, glad thats starting to help 14:07:04 I basically started by adding the ones that haven been around a little while 14:07:13 that's a nice break down there 14:07:17 stuff from this month mostly hasn't made it on there yet, but thats for later 14:07:38 so do add your own spec in there, if you fancy, as it gives us a helping hand 14:07:59 johnthetubaguy: oh we can ? 14:08:02 we said at the midcycle about looking at spec review priorities 14:08:03 johnthetubaguy: cool 14:08:08 gah, sawtooth comb in my browser... 14:08:22 what should we do about spec less bp's? 14:08:38 so thats next on the agenda 14:09:03 #info please ping folks via email or IRC to get your -2 removed, once your spec and/or blueprint is approved 14:09:24 so if you look on that etherpad we also have spec-less blueprints listed 14:09:43 now I am thinking we use that etherpad to collect those too, does that work for people? 14:10:22 yeah 14:10:33 should speed things up if we can just agree in the etherpad to approve rather than the nova meeting 14:10:53 or vote in the whiteboard of the bp or something like that 14:11:01 okay, so prio etherpad is for prio specs/bugs + quick hit bugs, and mitaka specs etherpad is for any other spec ? 14:11:05 yeah, then we can just ack them in the meeting 14:11:19 bauzas: thats the next bit, one sec 14:11:23 k 14:11:29 so this is just specs and blueprints for now, in that etherpad 14:11:49 so the first three vmware ones, I am going to approve those, is anyone against that? 14:12:08 I haven't looked at them yet, are they on the page now? 14:12:13 well, I will do the approve tomorrow morning, so vote in there if there is an issue I guess 14:12:18 yeah, they are in there 14:12:31 yeah, lemme run through them after the meeting 14:12:38 all of the aforemenioned are totally self contained within the driver 14:12:56 the first one I am not sure I understand, but yeah 14:13:07 so for next time, lets try vote on the etherpad before the meeting 14:13:16 so this is just a wave through and discuss dissagreement thing 14:13:20 which one? the opaque network support? 14:13:41 yeah, not sure I understand what its again 14:13:49 i mentioedn that one at the vancouver summit and it has been in review since. it is a painpoint for out liberty neutron driver :( 14:14:00 it's to support NSXv limitations, AFAIR... 14:14:07 johnthetubaguy: it is just the name os a network type support by the VC 14:14:23 jaypipes: no, that is not nsxv that is the new nsx version 14:14:36 garyk: ok 14:14:37 jaypipes: that was the pian poin in kilo :( 14:14:55 garyk: yeah, I wasn't happy with the vnic index and other things... 14:14:58 so I don't quite get how thats exposed to the user from the blueprint 14:15:04 Will this etherpad be closed later on? IOW, until which date can I add a blueprint to that? 14:15:05 anyway, don't want to go on tangent. 14:15:07 jaypipes: yes, i recall. 14:15:21 markus_z: at spec freeze I suspect, which is probably mitaka-1 14:15:30 johnthetubaguy: it is not exposed to the user 14:15:39 it is just how the internal plugin configures tings 14:15:52 until now the configuration file would have 'br-int' 14:15:55 we no longer need that 14:16:24 kind of the VC also has the notion now of an OVS bridge and that is an opaque type 14:16:33 garyk: it feels like a tiny spec for that first one would be good 14:16:43 johnthetubaguy: Ah, ok. Do I have to ping cores to have a look at it or will this be regularly looked at? 14:16:49 garyk: the support provider network port group? 14:16:57 markus_z: should be looked at without pings 14:17:03 ok, thanks 14:17:07 garyk: I think that needs a blueprint opening? 14:17:09 we're getting off in the weeds on the vmware specless bps here....can we table that? 14:17:18 garyk: if you add one, I think I can happy with that 14:17:19 johnthetubaguy: yes, agree. i need to open one 14:17:20 or do it in the nova channel later 14:17:31 mriedem: yes, was just hoping to be quick 14:17:37 lets stop with that now 14:17:39 1 is fine, half a dozen is too many for the meeting 14:17:56 give me a finger and i'll take a hand 14:18:03 pull my finger 14:18:07 yeah, agreed a plan for next time to avoid all this 14:18:11 thats the main bit 14:18:19 didn't want to leave them blocked another week though 14:18:36 #topic Regular Reminders 14:18:41 so this is the bit bauzas was asking about 14:18:53 I created the mitaka etherpad of doom 14:18:55 #link https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking 14:19:00 johnthetubaguy, can I add my spec to that etherpad? 14:19:03 same idea as before really 14:19:04 sorry, was confused :) 14:19:18 ndipanov: feel free to add it to the spec etherpad, yes 14:19:25 bauzas: no worries 14:19:49 anyways, I copied across the trivial patches, and added a few sub groups that have patches up already 14:19:54 but its there 14:19:59 #topic Bugs 14:20:14 Unfortunately I was busy with other work and don't have a good overview right now, sorry. 14:20:34 The two crits have patches for stable/liberty but ttx -2 for RC opening 14:20:54 yeah, they didn't seem bad enough to open the RC early 14:21:01 we're currently monitoring if any regressions 14:21:06 ~8 reviews on the trivial bug list 14:21:18 right, anything bug that we want to talk about 14:21:18 there is one bug that's planned to discussed during open discussion 14:21:23 That's all I can say for now 14:21:29 HI 14:21:29 there was the novaclient legacy bdm bug 14:21:31 there was a nasty one sdague and melwitt were looking at 14:21:36 yeah that 14:21:38 right, that one too 14:21:42 link? 14:21:46 we should probably do a patch release for that 14:21:46 that we just landed 14:21:58 mriedem: yeh, I've already talked with ttx on it 14:22:11 sdague: well it's just client so we an do a release right? 14:22:14 patch version release 14:22:16 I just wonder if https://bugs.launchpad.net/nova/+bug/1500688 is a regression but let's leave that for later 14:22:16 Launchpad bug 1500688 in OpenStack Compute (nova) "VNC URL of instance unavailable in CLI" [Undecided,New] 14:22:16 because we can't do liberty -> master upgrade testing until it's fixed 14:22:32 bauzas: it's definitely not good, it should be fixed 14:22:43 markus_z: https://bugs.launchpad.net/python-novaclient/+bug/1501435 14:22:43 Launchpad bug 1501435 in python-novaclient "osc 1.7 no longer can boot a server from volume" [Critical,Fix committed] - Assigned to John Garbutt (johngarbutt) 14:22:43 sdague: agreed 14:22:47 that's MHO 14:22:47 so I think we need to release off the stable branch and master, so with might need to bump the major version on master 14:23:07 master release version will depend on what's changed 14:23:10 i can look at that after the meeting 14:23:27 mriedem: yeah, just thinking we need to release of stable as well 14:23:36 so might have to ignore the rules there 14:23:44 the stable patches for the bdm stuff have been approved AFAIR 14:23:50 stable should be a patch version release 14:23:55 master can be a minor version 14:23:58 johnthetubaguy: wouldn't you do a patch release on stable and minor release on master? (rephrased, why major) 14:24:00 i'll look after the meeting 14:24:00 right, we need a release on both 14:24:10 if we backported something to stable that requires a minor version bump, we f'ed up 14:24:10 mriedem: great 14:24:15 jroll: thats my assumption, yeah 14:24:28 mriedem: oh crud, thats true, but maybe thats not allowed? 14:24:30 mriedem: it should be patch only I think, it's pretty minor 14:24:32 yeah ok 14:24:34 #action mriedem to checkout novaclient releases for stable/liberty and master 14:24:35 anyways, something for later 14:24:41 yeah, thank you! 14:24:53 Hi..all I have one review to discuss...pending since long time..don't know how can I forward thsi 14:24:54 mriedem: stable all cool? 14:24:56 anyone can put action items ? 14:25:08 i think stable is ok 14:25:22 some odd ebtables juno-only nova-net failure showed up yesterday 14:25:23 rsritesh: lets put that in open discussion, after the items we have there 14:25:25 not sure what that is yet 14:25:35 yuck 14:25:45 might be related to the ebtables failure we're seeing on master now 14:26:00 interesting, new test combo maybe 14:26:06 anyways... 14:26:07 no, not in juno 14:26:12 could be kernel updates on trusty... 14:26:14 https://bugs.launchpad.net/nova/+bug/1501366 14:26:14 Launchpad bug 1501366 in OpenStack Compute (nova) "libvirtError: Error while building firewall: Some rules could not be created for interface" [High,Confirmed] 14:26:16 #topic Open Discussion 14:26:33 mriedem: oh, true... 14:26:41 at the summit will there br driver sessions? 14:26:49 so we have an item here form abhishekk 14:26:52 hi 14:26:56 https://bugs.launchpad.net/nova/+bug/1416132 14:26:56 Launchpad bug 1416132 in OpenStack Compute (nova) "_get_instance_disk_info fails to read files from NFS due to permissions" [High,In progress] - Assigned to Eric Harney (eharney) 14:27:03 garyk: one sec, will get back to that 14:27:27 this is reported by eric and required to be fixed for cinder feature (NFS clone and snapshot) 14:27:27 abhishekk: what did you want to ask about thi? 14:27:39 nikola has suggested 2 ways to fix this bug IMHO neither of which are easy 14:28:01 I just want all members to have a look and give their opinion on the bug 14:28:07 ndipanov ^ 14:28:36 In between Eric also posted one solution but it is not a good way to fix 14:28:41 yes ndipanov 14:28:52 abhishekk: can you please post that review 14:28:57 * mriedem defers to ndipanov 14:28:58 yes 14:29:09 https://review.openstack.org/#/c/149037/ 14:29:51 honestly, I wonder about attaching BDM to the instance object, dansmith is that totally crazy? 14:30:07 like a lazy load field think 14:30:07 johnthetubaguy: we currently have the opposite, right? 14:30:19 I think it's not _that_ hard 14:30:26 johnthetubaguy: we can make a lazy-load property, but I don't think we want it as a field because it makes it a circular dependency 14:30:37 johnthetubaguy: yes lazy loading 14:31:02 cinder review: https://review.openstack.org/#/c/147186/ 14:31:09 well we can try that but I think that's gonna have a notice-able impact on the RT performance 14:31:13 dansmith: oh right... 14:32:19 we need select * from instance left join bdm on instance.uuid 14:32:23 there was someone talking about DB optimisation patches to remove excessive fetching/saving of BDMs during boot, it just feels like this could be adding lots more 14:32:44 it would add N queries to every RT run 14:32:48 N = number of instances 14:32:52 on the host 14:32:58 that sounds a bit nuts 14:33:01 it is 14:33:01 ndipanov: yes you are right 14:33:24 OK, so this feels like a good discussion for #openstack-nova, shall we take this offline? 14:33:33 we need the join query really 14:33:34 ok 14:33:36 * jroll imagines that code running in an ironic env with hundreds of instances D: 14:33:51 ok, I will ping on openstack-nova 14:34:06 so re: https://bugs.launchpad.net/nova/+bug/1500688 14:34:06 Launchpad bug 1500688 in OpenStack Compute (nova) "VNC URL of instance unavailable in CLI" [Undecided,New] 14:34:11 cool, I have a feeling there are lots of poeple in the meeting who can't add too much 14:34:16 we can hide that in a single query and rpc call wtithout much trouble 14:35:18 bauzas: Could you confirm that bug? I tried to get more info in that bug report. 14:35:35 markus_z: so, the story is that it has been changed in API v2.6 14:35:43 https://github.com/openstack/nova/blob/master/nova/api/openstack/rest_api_version_history.rst#26 14:35:54 but the client is not getting that 14:36:13 mriedem: you had a question on the client behaviour? 14:36:21 ohh, I remember 14:36:33 i was just saying the CLI shouldn't send 2.latest 14:36:42 it should send the latest version it's implemented to handle 14:36:42 right, so novaclient needs code to interpret the results 14:36:45 maybe always requesting the latest was a bad move... 14:36:46 which is 2.4 right now i think 14:36:51 mriedem: no, it's 2.11 14:36:57 sdague: in the client i mean 14:37:08 mriedem: right, in the client I thought someone implemented 2.11 14:37:10 yeah, I think the client should maybe request a high watermark? 14:37:11 https://github.com/openstack/python-novaclient/blob/master/novaclient/__init__.py#L23 14:37:21 sdague: ok, well then 2.5-2.10 are not available 14:37:22 I can see a MAX_VERSION 14:37:29 mriedem: right, that's the crux 14:37:30 ah... 14:37:42 does that mean it's not respected when calling by the CLI ? 14:37:49 we are missing some functional testing here I guess 14:37:56 yes, we are missing testing 14:38:05 and there is a hole in the microversion support 14:38:05 well you can workaround the bug by explicitly passing v2.1 or something <2.6 14:38:10 Ah, that's the reason Horizon is able to get the VNC console, but the CLI isn't. 14:38:16 markus_z: yes 14:38:28 nevermind https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L54 14:38:41 yeah, i think that's a bad idea 14:38:45 maybe it should be max API version per CLI call? or is that just silly? 14:38:56 it should be max api 14:39:06 so you just upgrade one call to a new version, rather than all calls 14:39:07 however, it would still be broken 14:39:25 imo the default should be the highest lower-bound that is supported, which i tihnk is 2.4 now 14:39:32 honestly, I think that's going to be a mess 14:39:33 regardless of if 2.11 is implemented 14:39:41 because it won't let lower bounds ever move 14:39:55 once you add 2.5 support you change the default to 2.5 14:39:56 and so on 14:39:58 it's going to cause a ton of fallout down the road that is going to be really hard to address 14:40:26 so on the server side the changes are incremental, 14:40:31 but on the client side it's willy nilly 14:40:33 changing APIs always messes up clients 14:40:39 yeah, feels like we need incremental on the client side 14:40:40 we should just be stricter on saying "you can't add 2.x to max until we're sure everything in 2.x-1 are handled 14:40:43 at least we have a sane way of addressing it 14:40:44 that's the mistake we made 14:40:47 sdague: right 14:40:49 agree with that 14:40:55 so lets just backfill that 14:41:11 and be aware in the future 14:41:24 so what should DEFAULT_OS_COMPUTE_API_VERSION be today? 2.11? 14:41:30 so to support 2.13 you must also add 2.12 support? 14:41:35 johnthetubaguy: yes 14:41:41 sdague: I guess thats OK 14:41:46 yeah, +1 to that 14:41:59 mriedem: feels like we should move that 2.4 while we fix thing up? 14:42:02 how do we test that? 14:42:21 we require tests on the boundary conditions to be submitted 14:42:30 the issue where is there are no tests for the vnc stuff either 14:42:37 no functional tests 14:42:54 yeah when i added 2.4 support there is a functional test for 2.1 and a test for 2.4 14:42:56 what about requiring a client bug be created with each microversion? 14:42:59 just like on the server side 14:43:00 I'm wondering that the old APIs got directly removed. Is that the usual approach? I thought they will be marked as deprecated but still functional. 14:43:11 *be 14:43:19 markus_z: we keep them for ever, basically 14:43:19 edleafe: it should really just be part of the nova spec 14:43:33 mriedem: okay, so it's clear from the changed code that a version is skipped? 14:43:36 mriedem: sure, but I mean as a way of tracking 14:43:55 markus_z: the resource is gone if you request that version, which is what the code is currently doing 14:44:21 alaski: i guess if we change DEFAULT_OS_COMPUTE_API_VERSION for each impl, and someone skips some versions, then it'd be clear 14:44:24 but we'd need to test that i'd think 14:44:28 unit test would be easy 14:44:47 like what we do in the nova objects version tests 14:45:04 okay. I haven't looked at microversions in the client, just wondered if it would be clear, and testable 14:45:05 'this is the last known max value, did they try skipping a version?' 14:45:12 OK, is anyone in particular able to take that one, so I can give you an action? 14:45:15 I'm a bit unclear, but why can't we default the max boundary to the same max boundary defined by novaclient? 14:45:33 bauzas: we can, and we should 14:45:33 sdague: Ah, ok, I thought it will just return the result from the new API. 14:45:34 i'm not sure what max boundary we're talking about 14:45:38 this bug would still be there 14:45:49 is there something besides this? https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L54 14:46:09 johnthetubaguy: i could take an action on this 14:46:15 id this something that we need to document - say people are starting to use liberty and their existing clients break. 14:46:28 I was looking at this I guess: https://github.com/openstack/python-novaclient/blob/master/novaclient/__init__.py#L23 14:46:29 that would be really problematic 14:46:35 sdague: sure, we also need to gracefully change the API resource to call by the client if the server is >=2.6 IIUC 14:46:45 bauzas: yes 14:46:50 * markus_z has the same thought as garyk 14:47:02 johnthetubaguy: oh hrm 14:47:04 garyk: yeah, I think we need to fix this urgently too 14:47:22 mriedem: honestly, no idea what they both do, I haven't dug enough 14:47:22 IMO the client "knows" the latest it can support. It shouldn't get that from the API 14:47:29 garyk: yup, I think this bug is High at least 14:47:30 so https://github.com/openstack/python-novaclient/blob/master/novaclient/__init__.py#L23 and https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L54 need to be tied together 14:47:40 garyk: and would need a RC2 backport 14:47:41 yeah 14:47:51 mriedem: that's MHO 14:47:56 right, the bug was that the client jumped to 2.11, but didn't handle the 2.6 switch 14:48:00 i still don't think 2.11 as the default is right 14:48:06 bauzas: this is a client issue, its a client backport, but yeah 14:48:09 2.11 as max is probably ok to opt into 14:48:14 johnthetubaguy: oh correct :) 14:48:20 but i'd think default is 2.4 14:48:35 mriedem: yeah, lets move this back to something we know works, then try add the gaps? 14:48:44 that should work I think 14:48:45 johnthetubaguy: +1 14:48:45 that's what i was thinking, keep it simple 14:48:54 especially as it will need to backport 14:49:05 we can backport the nuke it, and move forward then 14:49:06 mriedem: fwiw, in ironic we decided not to have cli as max by default after much debate 14:49:06 and we can backport a thing to stable/liberty and get it into the upcoming patch release which we know is happening b/c of the legacy bdm bug 14:49:22 jroll: geez thanks for telling us! 14:49:23 mriedem: we need the bdm release ASAP 14:49:29 sdague: ok 14:49:35 can work both today separately 14:49:37 mriedem: read the ML harder :P 14:49:49 in the client, isn't API_MAX_VERSION = x saying that the client can correctly handle any version up to and including x? 14:49:51 jroll: yes, but nova's in a different boat with multi cloud support 14:50:00 edleafe: it should mean that 14:50:08 that's the bug 14:50:08 sdague: totally. just putting that data point out there, it isn't unprecedented 14:50:23 sdague: that was my understanding; just checking 14:50:25 right, the resolution of the bug is to add support for 2.6 that's it 14:50:34 #action mriedem to sort out novaclient default/max api version mess and backport things 14:50:38 support in the client I mean 14:50:43 bauzas: well, 2.6-2.10 14:50:45 bauzas: any, related, any other potential issues in there 14:50:46 * jroll still of the opinion users should always explicitly pass a version, it's only one more env variable 14:50:54 jroll: this is just the CLI at least, when you use it as an SDK you have to be explicit with versions 14:50:56 bauzas: the immediate tactical fix though is to not default to 2.11 14:51:08 johnthetubaguy: yep 14:51:09 mriedem: agreed, there is a workaround 14:51:11 bauzas: and then we backport that 14:51:20 jroll: we disagree on that one, I don't think users should have to set any env vars 14:51:28 the cloud they are talking to knows what it can do 14:51:28 mriedem: WFM 14:51:55 sdague: oh, I know :) 14:52:00 well, I like us trying to make it work 14:52:14 I think we have a plan, we should cover the other two things before we run out of time 14:52:15 yeh, this isn't hard to fix, it just got missed 14:52:32 sdague: mriedem: you folks are both on the case here I guess? 14:52:42 I think mriedem's on lead, I'll help review 14:53:01 #action mriedem to look into the max API version in python-novaclient 14:53:04 cools 14:53:14 so garyk you had a question about driver sessions 14:53:46 so the question for me is, what do you want to debate in the session? do you have a list of specs that can't get agreement on, for example? 14:54:01 mriedem: FWIW triaged https://bugs.launchpad.net/python-novaclient/+bug/1500688 14:54:01 Launchpad bug 1500688 in python-novaclient "VNC URL of instance unavailable in CLI" [High,Confirmed] 14:54:15 johnthetubaguy: i feel that most of the time a lot of people have no idea what is going on in the various drivers and the sub teams can and should share the data. 14:54:23 it would be a good platformt o share ideas and information 14:54:43 at least enable one to give better clarity to the internals and help reviewrs undersyand things a bit better 14:54:47 is a 40 minute session every 6 months going to fix that? 14:55:01 maybe a weekly meeting would be better if there are topics to discuss 14:55:03 it feels like a monthly email might be more useful 14:55:08 yeah, something 14:55:10 yeah, I think I'm done with driver sessions at summits 14:55:14 sdague: that sounds like a good idea 14:55:19 garyk: so here is the thing, I am giving priority to things that need in person discussion 14:55:36 yeah, the monthly email is a good one 14:55:40 johnthetubaguy: ok. understood. lets leave the drivers out 14:55:48 not saying that in a bad way. 14:56:07