Thursday, 2017-05-25

*** diablo_rojo has joined #openstack-meeting-cp00:00
*** diablo_rojo has quit IRC00:17
*** edmondsw has joined #openstack-meeting-cp01:20
*** edmondsw has quit IRC01:25
*** notmyname has quit IRC02:08
*** notmyname has joined #openstack-meeting-cp02:13
*** edmondsw has joined #openstack-meeting-cp03:08
*** edmondsw has quit IRC03:13
*** gouthamr has quit IRC03:57
*** benj_ has quit IRC04:03
*** benj_ has joined #openstack-meeting-cp04:04
*** markvoelker has joined #openstack-meeting-cp04:13
*** edmondsw has joined #openstack-meeting-cp04:56
*** edmondsw has quit IRC05:01
*** markvoelker_ has joined #openstack-meeting-cp05:11
*** markvoelker has quit IRC05:14
*** markvoelker has joined #openstack-meeting-cp05:17
*** markvoel_ has joined #openstack-meeting-cp05:19
*** markvoelker has quit IRC05:19
*** markvoelker_ has quit IRC05:21
*** markvoel_ has quit IRC05:53
*** markvoelker has joined #openstack-meeting-cp05:53
*** markvoelker has quit IRC05:54
*** markvoelker has joined #openstack-meeting-cp06:27
*** markvoelker has quit IRC06:32
*** edmondsw has joined #openstack-meeting-cp06:45
*** edmondsw has quit IRC06:49
*** markvoelker has joined #openstack-meeting-cp06:59
*** markvoelker has quit IRC07:03
*** markvoelker has joined #openstack-meeting-cp07:09
*** markvoelker has quit IRC07:14
*** markvoelker has joined #openstack-meeting-cp07:29
*** markvoelker has quit IRC07:37
*** markvoelker has joined #openstack-meeting-cp07:42
*** markvoelker has quit IRC07:49
*** sheel has joined #openstack-meeting-cp08:13
*** markvoelker has joined #openstack-meeting-cp08:24
*** markvoelker has quit IRC08:30
*** markvoelker has joined #openstack-meeting-cp08:32
*** markvoelker has quit IRC08:40
*** markvoelker has joined #openstack-meeting-cp09:16
*** markvoelker has quit IRC09:21
*** sdague has joined #openstack-meeting-cp09:46
*** markvoelker has joined #openstack-meeting-cp09:54
*** markvoelker has quit IRC09:58
*** edmondsw has joined #openstack-meeting-cp10:21
*** sheel has quit IRC10:22
*** edmondsw has quit IRC10:25
*** markvoelker has joined #openstack-meeting-cp10:47
*** markvoelker has quit IRC10:52
*** markvoelker has joined #openstack-meeting-cp10:55
*** markvoelker has quit IRC10:59
*** markvoelker has joined #openstack-meeting-cp11:14
*** markvoelker has quit IRC11:19
*** markvoelker has joined #openstack-meeting-cp11:29
*** markvoelker has quit IRC11:34
*** edmondsw has joined #openstack-meeting-cp11:58
*** markvoelker has joined #openstack-meeting-cp12:00
*** markvoelker has quit IRC12:04
*** markvoelker has joined #openstack-meeting-cp12:11
*** markvoelker has quit IRC12:15
*** markvoelker has joined #openstack-meeting-cp12:17
*** markvoelker has quit IRC12:22
*** zengchen has quit IRC12:30
*** pewp has quit IRC12:39
*** pewp has joined #openstack-meeting-cp12:40
*** markvoelker has joined #openstack-meeting-cp12:56
*** gouthamr has joined #openstack-meeting-cp13:01
*** markvoelker has quit IRC13:33
*** MarkBaker has quit IRC14:09
*** markvoelker has joined #openstack-meeting-cp14:15
*** MarkBaker has joined #openstack-meeting-cp14:21
*** markvoelker has quit IRC14:23
*** felipemonteiro has joined #openstack-meeting-cp14:30
*** jaugustine has joined #openstack-meeting-cp14:44
*** markvoelker has joined #openstack-meeting-cp15:01
*** felipemonteiro has quit IRC15:14
*** cFouts is now known as nug15:44
*** nug is now known as Guest3027815:44
*** mriedem has joined #openstack-meeting-cp15:57
*** mgiles has joined #openstack-meeting-cp15:59
ildikov#startmeeting cinder-nova-api-changes16:00
openstackMeeting started Thu May 25 16:00:30 2017 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"16:00
openstackThe meeting name has been set to 'cinder_nova_api_changes'16:00
jungleboyjo/16:00
ildikovDuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood breitz jungleboyj16:00
mriedemo/16:00
hemna@!16:00
pewphemna (╯°□°)╯︵ ┻━┻16:00
jungleboyj@!h16:01
pewpjungleboyj (/ .□.) ︵╰(゜Д゜)╯︵ /(.□. )16:01
ildikovhemna: already? :)16:01
hemnait's the new hi.16:01
ildikovhemna: oh, got it :)16:01
jungleboyjhemna:  Is just angry all the time.16:02
ildikovI guess jungleboyj said smth like 'Hi All' then :)16:02
smcginnishah16:02
stvnoyeso?16:02
jungleboyjildikov:  :-)16:02
stvnoyesi guess i am here16:02
smcginnisstvnoyes: Not sure about that? :)16:03
ildikovok, so for today we have live migrate, touch on attach and swap a bit and then we might look into some other things stvnoyes found recently in the new Cinder attach/detach API16:03
stvnoyes:-)16:03
ildikovstvnoyes: I saw you put live_migrate into WIP16:03
mriedemdo we have any nova changes which aren't WIP that need review?16:04
ildikovstvnoyes: is that because of the comments for rollback?16:04
mriedemand can actually move forward?16:04
ildikovmriedem: your two changes I think still16:04
mriedemildikov: ok i thought so16:04
ildikovmriedem: and this: https://review.openstack.org/#/c/456896/16:05
mriedemwhich i co-authored16:05
mriedemi'll need to just find some helpless nova core to pick on here16:06
stvnoyesis there a proposed fix yet for this cinder bug? - https://bugs.launchpad.net/cinder/+bug/1692153 - it will be important to get that fixed16:06
openstackLaunchpad bug 1692153 in Cinder "v3 attachment connection_info formatting is different than v2" [High,New]16:06
ildikovmriedem: and we could decide on this one too: https://review.openstack.org/#/c/446671/16:06
mriedemstvnoyes: is that on 3.0?16:06
mriedemor 3.27?16:06
ildikovmriedem: 3.2716:06
stvnoyesonly with the new v3 cinder attach code16:06
hemnastvnoyes, so, I think I can put in a fix in os-brick to cast those values to ints16:07
hemnabut I still think it's a bug in the output IMHO16:07
mriedembug in the output of the api?16:07
hemnayes16:07
mriedemat this point it's in the response that way16:07
mriedemyou can't change it16:07
mriedemw/o a new microversion16:07
hemnathe OVO stuff completely reformats it16:07
hemnaeven if it's a bug ?16:08
mriedemcorrect16:08
stvnoyesi think there were boolean issues too...16:08
hemnayes16:08
hemna"False" vs. False16:08
hemnavalues.16:08
stvnoyesok16:08
hemnathe oslo versioned object stuff forces it to strings :(16:08
ildikov:/16:09
mriedemneed response schema validation in tempest16:09
mriedemhowever,16:09
mriedemgiven (1) you don't really want to test initialize_connection in tempest and (2) it's a freeform dict,16:09
mriedemthat's probably not trivial16:09
hemnaattaches won't work w/o a fix somewhere16:09
mriedemyou could do it with your in-tree tempest plugin16:09
mriedemthe fixes will have to be client-side for now16:09
ildikovso we use OVO and then convert stuff back before sending out the response?16:09
hemnaI discovered it while trying to update the brick cinderclient extension to do bare metal attaches16:09
stvnoyesi ran into it doing a v3 attach using the POC bits16:10
ildikovhemna: yeah, I remember now you were discussing it on the Cinder channel not long ago16:10
hemnanobody is using this new API yet.....16:10
mriedemheh16:11
mriedemagain,16:11
mriedemthat's not the answer16:11
mriedemonce you guys are getting used outside openstack, like you want to be, you can't make those assumptions16:11
hemna:P16:11
mriedemplus,16:11
mriedemchanging this on pike means it's still busted on ocata16:11
hemnacome on Dad!16:11
mriedemand client side doesn't know b/c you didnt bump the microversion16:12
smcginnisUntil we backport the fix.16:12
mriedemwrong answer16:12
smcginnisWhich we can't do if we bump the mv just for a bug fix.16:12
hemnaI'm not sure how to fix it on the cinder side at the moment16:12
hemnawho are the ovo "guys" in oslo ?16:12
mriedemdansmith and jaypipes16:12
mriedemi don't see how this is a problem for ovo though16:13
hemnaok, I'll raise this with them and see what their thoughts are on a fix16:13
mriedemsmcginnis: correct, you can't backport it16:13
mriedemit's a new microversion16:13
hemnain the mean time, I can patch brick to do a fix16:13
mriedemeven if you changed ovo, the response would change16:13
hemnathen we can microversion the fix on the cinder side once we figure out the right way.16:13
mriedemyup16:14
mriedemsee 2.42 in nova if you want to see how we've also had to deal with this kind of pain16:14
hemnaare we ok with me patching brick to fix at least the LUN int ?16:14
mriedembrick is client side,16:15
mriedemand can straddle releases16:15
mriedemand nova is also client side,16:15
mriedemand can require a minimum version of brick16:15
mriedemwith the fix16:15
hemnaok I'll try and get that up today16:15
ildikovcool, so brick it is16:16
ildikovand also check what we do with ovo and this type of issues long term16:16
hemnaildikov, +116:16
hemnamriedem, if you can find me a nova review that I can see how you guys handled something similar that'd help16:17
*** MarkBaker has quit IRC16:18
ildikovis there a note/warning anywhere for ovo that it does things you would not necessarily expect?16:18
hemnaildikov, well, I think it's doing exactly as you'd expect16:19
hemnahttps://github.com/openstack/cinder/blob/master/cinder/objects/volume_attachment.py#L5316:19
hemnathe connection_info was declared as a dict of strings.16:19
hemnahence the conversion16:19
mriedemhemna: https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/fix-tag-attribute-disappearing.html is a story about how we f'ed up16:19
hemnaI think you have to declare those vars as something, and as far as I can tell, there is no DictOfLeaveMeAlone16:19
ildikovhemna: hmm, I was just wondering why we only realized it now... :)16:20
hemnaildikov, because nobody was using the result coming back from the API yet16:20
hemnato try and do a real attach16:20
mriedemwell,16:20
mriedemthe long-term awesome thing to do is model ConnectionInfo as an object itself,16:20
hemnabrick puked, because "1" != 116:20
mriedemwith it's own fields that are various types16:20
mriedembut,16:20
hemnaouch16:20
ildikovhemna: I know, but still16:20
hemnayah16:20
mriedemthat means actually codifying that thing16:20
mriedemactually knowing wtf goes into connection_info would be awesome16:21
stvnoyes+116:21
mriedemeven if you needed a vendor_extras field which is a dict of strings16:21
mriedemand unvalidated16:21
mriedemanyway, that's way down the road and will probably never happen16:21
hemnabrick 2.016:21
ildikovuh, that looks a tough goal to aim for, but I definitely agree it would be nice to have some sort of a structure for that thing16:21
mriedemhemna: it's server side16:21
mriedemnot brick16:21
mriedemthink nova.objects.ImageMetaProps in nova16:22
hemnawell, there is the connector dict too16:22
mriedemcodifies image metadata properties from glance16:22
hemnaI guess I was thinking of that too16:22
mriedemyes connector should also be a versioned object16:22
mriedemthey are both a wild west of terrible16:22
mriedemanyway, wayyyyyy down the rabbit hole here16:22
mriedemcan we move on?16:22
hemnayup16:22
ildikovyes16:22
ildikovso we have a recent WIP, which is live migrate16:23
ildikovstvnoyes: do you want to discuss that here?16:23
stvnoyeshold off on rv'ing that for now. i am in the middle of functionally testing it. i just got v3 attach working yesterday.16:24
ildikovI saw your latest comments about rollback and was wondering whether that's something to raise here?16:24
stvnoyesi've been updating the POC and my rv with what I'm finding16:24
stvnoyeswell there is a odd bug in nova in rollback....16:25
stvnoyesnew or old code16:25
stvnoyeshttps://review.openstack.org/#/c/463987/3/nova/compute/manager.py - self.compute_rpcapi.remove_volume_connection has the wrong args16:26
ildikovmriedem: have you seen that ^?16:26
stvnoyes~line 575816:26
stvnoyesalso, i'm not sure fixing the args is the real answer, i don't know why we'd do this on a source node.16:27
mriedemwhy is the signature wrong?16:28
mriedemthis is the source host telling the dest host to rollback volume connections from the dest host right?16:29
mriedemb/c we're rolling back16:29
stvnoyeshmmm, re-looking at this, it seems ok. not sure what i was thinking.16:30
stvnoyesignore this for now16:30
mriedemso,16:30
mriedemlive migration is a turd16:30
mriedemjust fyi16:30
mriedem:)16:30
stvnoyesyeah, we had similar issues in the virtual iron product16:31
mriedemit's like 20 compute manager methods doing rpc calls back and forth and you rarely know which one is on the source or target host16:31
mriedemnone of it is documented,16:31
mriedemor mapped out16:31
mriedemyou're welcome16:31
stvnoyes:-)16:31
ildikovnice :)16:31
*** MarkBaker has joined #openstack-meeting-cp16:31
mriedemyou're lucky the docstring even says16:32
mriedem:param dest:16:32
mriedem            This method is called from live migration src host.16:32
mriedem            This param specifies destination host.16:32
mriedemthat's a rare bit of critical information in these methods16:32
ildikovok, then I guess whomever can make attach work with the new flow should test the live migrate patch for now to see how it works out16:33
stvnoyesthe odd part is that i was getting wrong number of args exception in here. i need to go back and re-look at where that was coming from.16:33
mriedem$5 to a new contributor to document the flow :)16:33
ildikovlol :)16:33
stvnoyesi need to check the other patches that i applied...16:34
mriedemmoving on?16:35
mriedemstvnoyes and i reviewed jgriffith's attach WIP https://review.openstack.org/#/c/330285/16:35
ildikovthe attach PoC throws warious things I would guess, the other bits should be fine16:35
ildikovmriedem: stvnoyes: we will skip the microversion discovery for now16:36
ildikovso we can have attach itself working16:36
ildikovfirst16:36
mriedemi'm fine with that for now16:36
mriedemwe16:36
mriedemwe'll want t16:36
mriedemdamn16:36
ildikovshelve should be moved out of the patch too, but as it contained more things originally than just attach I didn't bother as of yet...16:37
mriedemwe'll want to split that out into a separate patch first anyway i think16:37
mriedemwell,16:37
mriedemthis is attaching a volume to a shelved offloaded server16:37
mriedemso it's still new style attach16:37
mriedemjust a wrinkle16:37
mriedemi don't think you can split that out really16:37
ildikovI hoped we can16:38
ildikovI talked to John this morning, he's cleaning up things in the patch like using the attachment_id we have in the bdm now, etc.16:39
ildikovI also finally have a DevStack up and running so I will chime in testing and cleaning up this stuff16:39
mriedemi think there are some obvious things to fix in there, yes, which is a good start16:40
mriedemwhich should also decomplicate it16:40
ildikovmriedem: stvnoyes: also tnx for the thorough review :)16:40
stvnoyesnp, i am anxious to see migrate work16:41
ildikovmriedem: yeah, I started to fix up check and reserve for instance, but couldn't get there to clean up the rest and things like these16:41
stvnoyeshopefully shortly16:41
ildikovthe connection_info format changes in the driver are also old ones, we had a stage when the format didn't match16:41
ildikovthat needs to be checked again too16:42
ildikovand when we have attach then we will start write a new version of swap16:44
ildikovthat thing still confuses me...16:44
ildikovif someone could once describe me what migrate_volume_completion is supposed to do I would be pretty pleased :)16:45
mriedemit's nova calling back to cinder saying i'm done16:45
mriedemwhich was either successful or an error16:45
mriedemif cinder initiated the swap, i thought telling it we failed initiated a rollback16:46
mriedemon the cinder side16:46
mriedemif cinder didn't initiate the swap, i assume cinder is like, 'ok great but i don't know what you're talking about'16:46
ildikovand what does it do when Nova initiated a swap and it was successful?16:47
smcginnismriedem: I believe so: https://github.com/openstack/cinder/blob/114e3bd7223ae957bd9bb71e3fb7e114af5c9f72/cinder/volume/manager.py#L201816:47
mriedemildikov: i don't know16:48
mriedemthat's a question for cinder16:48
ildikovmriedem: ok :)16:48
mriedemi would think if cinder wasn't tracking a migration for the volume it would ignore it,16:49
mriedembut idk16:49
mriedemhttps://github.com/openstack/cinder/blob/114e3bd7223ae957bd9bb71e3fb7e114af5c9f72/cinder/volume/api.py#L144916:49
mriedemwe don't cast to the volume manager then16:49
mriedemso i think that's just updating the db on the cinder side16:50
mriedemin the case that it didn't fail16:50
mriedemwhich if nova already did that i think it's just a noop16:50
mriedembut again i'm not sure16:50
ildikovyou mean with all that attach/detach calls in the non-error state?16:50
mriedemyes16:50
ildikovok, cool16:51
mriedemhttp://logs.openstack.org/95/467995/3/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/46deb21/logs/new/tempest_conf.txt.gz16:51
mriedemoops wrong window16:51
ildikovthen we will look into how to reproduce those many noop's with the new flow then...16:52
ildikovI think mainly that's it for today16:52
mriedemcool,16:52
mriedemi've also put out the nova core call to 2 specific cores to review the bottom changes which have the +216:53
ildikovmriedem: who to ping with those few patches to merge?16:53
mriedemalex and kenichi16:53
mriedemalex is in china so he's probably done for the day already16:53
ildikovmriedem: awesome, tnx16:53
mriedembut kenichi is in california so he can't hide16:53
ildikovok :)16:53
ildikovI pinged other two guys, hopefully we will have at least one looking at those finally16:54
ildikovcool16:55
ildikovif nothing else then thank y'all for the today's meeting16:55
mriedemwhich other two guys?16:55
ildikovlyarwood and sfinucan IIRC16:55
mriedemoh, not cores16:55
mriedembut ok16:56
mriedemanyway yeah let's call it a meeting16:56
mriedemoh sfinucan is16:56
mriedemduh16:56
ildikov:)16:56
ildikovtalk to you next week the latest! :)16:57
ildikovhave a good day16:57
ildikov#endmeeting16:57
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"16:57
openstackMeeting ended Thu May 25 16:57:26 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:57
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-05-25-16.00.html16:57
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-05-25-16.00.txt16:57
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-05-25-16.00.log.html16:57
*** mriedem has left #openstack-meeting-cp16:57
*** felipemonteiro has joined #openstack-meeting-cp16:58
*** mgiles has quit IRC16:58
*** diablo_rojo has joined #openstack-meeting-cp17:43
*** david-lyle has joined #openstack-meeting-cp18:12
*** david-lyle has quit IRC18:19
*** aselius has joined #openstack-meeting-cp18:40
*** Guest30278 has quit IRC19:13
*** nug- has joined #openstack-meeting-cp19:16
*** felipemonteiro has quit IRC20:10
*** david-lyle has joined #openstack-meeting-cp20:12
*** david-lyle has quit IRC20:17
*** edmondsw has quit IRC20:37
*** kbyrne has quit IRC20:38
*** markvoelker has quit IRC20:42
*** edmondsw has joined #openstack-meeting-cp20:44
*** markvoelker has joined #openstack-meeting-cp20:56
*** markvoelker has quit IRC21:00
*** sdague has quit IRC21:24
*** markvoelker has joined #openstack-meeting-cp22:05
*** markvoelker has quit IRC22:15
*** markvoelker has joined #openstack-meeting-cp22:16
*** gouthamr has quit IRC22:29
*** jaugustine has quit IRC22:33
*** edmondsw has quit IRC22:34
*** edmondsw has joined #openstack-meeting-cp22:34
*** edmondsw has quit IRC22:39
*** gouthamr has joined #openstack-meeting-cp23:05
*** qwebirc18348 has quit IRC23:10
*** markvoelker has quit IRC23:11
*** edmondsw has joined #openstack-meeting-cp23:35
*** edmondsw has quit IRC23:40
*** diablo_rojo has quit IRC23:40
*** markvoelker has joined #openstack-meeting-cp23:44
*** markvoelker has quit IRC23:45

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!