Thursday, 2017-09-21

*** yamahata has quit IRC01:46
*** iyamahat has quit IRC01:46
*** nhelgeson has quit IRC02:14
*** gouthamr has joined #openstack-meeting-cp02:20
*** lbragstad has joined #openstack-meeting-cp03:00
*** lbragstad has quit IRC03:21
*** coolsvap has joined #openstack-meeting-cp03:37
*** yamahata has joined #openstack-meeting-cp03:50
*** iyamahat has joined #openstack-meeting-cp04:00
*** gouthamr has quit IRC04:12
*** Rockyg has joined #openstack-meeting-cp04:46
*** markvoelker has quit IRC05:16
*** iyamahat has quit IRC05:45
*** Rockyg has quit IRC06:03
*** iyamahat has joined #openstack-meeting-cp06:19
*** rarcea has joined #openstack-meeting-cp06:29
*** markvoelker has joined #openstack-meeting-cp07:18
*** iyamahat has quit IRC07:23
*** markvoelker has quit IRC07:52
*** markvoelker has joined #openstack-meeting-cp08:48
*** MarkBaker has joined #openstack-meeting-cp09:08
*** edmondsw has joined #openstack-meeting-cp09:15
*** edmondsw has quit IRC09:20
*** MarkBaker has quit IRC09:20
*** markvoelker has quit IRC09:21
*** MarkBaker has joined #openstack-meeting-cp09:32
*** markvoelker has joined #openstack-meeting-cp10:19
*** markvoelker has quit IRC10:52
*** MarkBaker has quit IRC11:20
*** MarkBaker has joined #openstack-meeting-cp11:22
*** sdague has joined #openstack-meeting-cp11:38
*** MarkBaker has quit IRC11:49
*** markvoelker has joined #openstack-meeting-cp11:49
*** markvoelker has quit IRC11:54
*** markvoelker has joined #openstack-meeting-cp11:56
*** MarkBaker has joined #openstack-meeting-cp12:02
*** yamahata has quit IRC12:12
*** edmondsw has joined #openstack-meeting-cp12:13
*** MarkBaker has quit IRC12:27
*** xyang1 has joined #openstack-meeting-cp12:51
*** MarkBaker has joined #openstack-meeting-cp13:16
*** gouthamr has joined #openstack-meeting-cp13:26
*** coolsvap has quit IRC13:39
*** felipemonteiro has joined #openstack-meeting-cp14:38
*** felipemonteiro_ has joined #openstack-meeting-cp14:39
*** felipemonteiro has quit IRC14:43
*** felipemonteiro_ has quit IRC14:59
*** felipemonteiro_ has joined #openstack-meeting-cp14:59
*** felipemonteiro has joined #openstack-meeting-cp15:11
*** felipemonteiro_ has quit IRC15:13
*** felipemonteiro_ has joined #openstack-meeting-cp15:24
*** iyamahat has joined #openstack-meeting-cp15:24
*** felipemonteiro has quit IRC15:27
*** iyamahat has quit IRC15:28
*** felipemonteiro_ has quit IRC15:32
*** felipemonteiro_ has joined #openstack-meeting-cp15:33
*** felipemonteiro has joined #openstack-meeting-cp15:34
*** felipemonteiro_ has quit IRC15:37
*** iyamahat has joined #openstack-meeting-cp15:43
*** mriedem has joined #openstack-meeting-cp15:48
*** felipemonteiro has quit IRC15:49
*** felipemonteiro_ has joined #openstack-meeting-cp15:49
*** felipemonteiro has joined #openstack-meeting-cp15:53
*** felipemonteiro_ has quit IRC15:56
ildikov#startmeeting cinder-nova-api-changes16:00
openstackMeeting started Thu Sep 21 16:00:24 2017 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"16:00
openstackThe meeting name has been set to 'cinder_nova_api_changes'16:00
ildikovjohnthetubaguy jaypipes e0ne jgriffith hemna mriedem patrickeast smcginnis diablo_rojo xyang1 raj_singh lyarwood jungleboyj stvnoyes16:00
jungleboyj@!16:00
_pewp_jungleboyj \(-_- )16:00
stvnoyeso/16:00
*** MarkBaker has quit IRC16:01
ildikovlet's wait a tiny bit, I guess people are slower joining after having the PTG last week :)16:01
mriedemo/16:02
smcginniso/16:03
jgriffitho/16:03
ildikovok, let's start16:03
jgriffithpheww16:03
ildikovjgriffith: just in time :)16:03
ildikovso we agreed on a few things last week and are about to move forward16:03
jungleboyjjgriffith:  Slides into home.16:03
ildikovI saw recent comments on the live_migration patch and test16:03
ildikovmriedem: there's only one service version check in the new attach patch that I'm aware of16:04
stvnoyeslooks like i have a grenade issue there.16:04
ildikovmriedem: which is not supposed to make the Grenade test fail16:04
mriedemit could be a problem in the test, i'm not sure16:05
stvnoyesi'll look into to see why it's failing16:05
mriedembut yes, we shouldn't be using any new style attachments if any computes are not running the latest code16:05
mriedemso i'm not sure why grenade would have issues16:05
stvnoyesi am guessing it's a test issue16:05
mriedemmaybe, i haven't grok'ed your tempest change16:05
*** yamahata has joined #openstack-meeting-cp16:07
stvnoyeswhile we're on the topic of tempest tests...16:07
ildikovthe attach patch should be up to date with the compute service version16:07
ildikovbut let me know if something went wrong there16:07
stvnoyesI've been working on debugging the migrate iscsi test.  In my local env I am seeing very occasional detach timeouts - DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.16:08
stvnoyeshappened twice yesterday, but after a bunch of debug, hasn't happened yet today16:08
stvnoyesi've been updated this bug - https://bugs.launchpad.net/nova/+bug/169612516:09
openstackLaunchpad bug 1696125 in OpenStack Compute (nova) "Detach interface failed - timeout waiting to detach tap device in linuxbridge job (pike)" [High,In progress] - Assigned to Matt Riedemann (mriedem)16:09
stvnoyesfyi16:09
stvnoyesseems to go down into libvirt and libvirt never responds with detach complete16:10
jgriffithstvnoyes I ran into that at one point a while back, but with the old code.  Problem "went away" and I never got it back16:10
jgriffithstvnoyes and never figured out what it was doing either :(16:10
*** nhelgeson has joined #openstack-meeting-cp16:10
mriedemstvnoyes: are you using ovs or linuxbridge?16:10
stvnoyesovs16:11
mriedemare you running latest nova?16:11
mriedemthere was a recent change that went into that code not too long ago16:11
stvnoyessorta, as of Aug 31, after the last change listed in that bug16:11
mriedemhttps://review.openstack.org/#/q/I8cd056fa17184a98c31547add0e9fb2d363d090816:12
mriedemok you should have that then16:12
mriedemmelwitt, mdbooth, lyarwood and kashyap are the people to ask about that weird persistent vs transient domain stuff16:13
stvnoyesok thx. the persistent  detach seems to work ok, it dies on the transient detach16:13
stvnoyesi am hoping to get a failure with my latest debug code, should learn more when that happens16:14
stvnoyesunless of course it changes things enough to prevent the failure16:14
ildikovstvnoyes: thanks for looking into it16:16
ildikovI didn't run into this one besides the Gerrit reviews16:16
ildikovis there anything else to add to this topic?16:17
stvnoyesall set here16:17
ildikovstvnoyes: cool, thanks16:18
ildikovso we're working on a couple of things to get the multi-attach prerequisites covered as well before merging the new attach patch16:18
ildikovthe reason is to wait and have the latest microversion out so we don't need to deal with bumps later if possible16:18
ildikovwe said we will have a uses_shared_connections flag16:19
* johnthetubaguy hides in the back row16:19
ildikovlooking more into it the recent plans are to have that in the volume info16:19
ildikovas opposed to the attachment or the connection_info16:19
ildikovmriedem: johnthetubaguy: any objections to that? ^^16:20
ildikovjohnthetubaguy: glad you could join :)16:20
jgriffithjohnthetubaguy mriedem for a little more detail, the idea being that you'd want *something* to use as a lock string even during attachment-reserve/create16:21
johnthetubaguyas long as when we do attachment_update and attachment_delete we have the info, I think it is fine16:21
jgriffithSo the proposal is that we have a bool added in the volume that says "uses_shared_targets"16:21
johnthetubaguyjgriffith: why on reserve?16:21
jgriffithpatrickeast raised a concern stating it needed to be there16:21
johnthetubaguydo we know why?16:21
jgriffithI'm not 100% convinced, but didn't feel like rat holing16:22
ildikovand as it is a capability it doesn't change over attachments, so it makes more sense as part of the volume info16:22
johnthetubaguyso a lock on reserve isn't really implementable in Nova right now16:22
ildikovjgriffith: didn't he say attachment_update?16:22
jgriffithildikov +1 it made sense to associate it with the volume regardless16:22
jgriffithildikov well, yes; the problem being that he'd need the info DURING the attachment_update16:22
johnthetubaguyyeah, in volume_info seems to make sense, from a data model POV16:22
jgriffithwhich means ideally he'd have it on the attachment_reserve16:22
johnthetubaguyI am guessing non-Nova users will appreciate it before attachment_create/reserve etc16:23
ildikovthat would mean the return value from reserve the latest and not using it for reserve16:23
jgriffithSo my proposal is to just put that flag in the volume, as well as the volume_backend_name which could then be used for the lock16:23
ildikovjgriffith: +116:23
johnthetubaguyyeah, we want it for attachment_update and _delete, I think that is the main bit16:24
jgriffithjohnthetubaguy yeah, and this way you can have it for *all the things* if something comes up16:24
ildikovwe retrieve the volume info before those calls I think in all cases, so it should be ok16:24
jgriffithlogically it makes sense, and the testing so far looks bueno16:24
ildikovwe could still save it in the BDM if we wanted to16:24
jgriffithhope to have it posted later this morning16:24
mriedemwhat happens if the volume is retyped?16:24
mriedemand the backend name changes16:25
jgriffithif I can figure out which Instance I had all the code on :)16:25
johnthetubaguyjgriffith: yeah, all sounds fine, my worry was there was something happening in reserve/create we don't understand16:25
jgriffithmriedem then that info gets updated if it's a backend migration16:25
jgriffithjohnthetubaguy no, I don't think there is, it's more about when/how to get the info on the caller side16:25
jungleboyjjgriffith:  That plan makes sense to me.16:26
jgriffithmriedem did that address our question?16:26
mriedemi don't really want to think about retyping a multi-attach volume16:26
jgriffithmriedem me neither :(16:26
ildikovthe info makes even more sense on the volume level considering back end migration IMHO16:26
jgriffithThere's a reason we used not allow retype of an in-use volume :)16:26
ildikovyeah, let's not ruin this nice morning with that thought... :/16:26
johnthetubaguyjgriffith: cool16:26
johnthetubaguymriedem +1 on that16:27
jungleboyjmriedem:  One step at a time.  :-)16:28
ildikovok, it seems to me that we're on an agreement here16:28
ildikovso if anyone wants to rain in this parade please do it now16:29
ildikovor don't do it at all :)16:29
mriedemshared connections are backend specific,16:30
mriedemand the volume is typed per backend,16:30
mriedemso it makes sense to model that in the volume16:30
mriedemso +116:30
ildikovgreat, so we will add the info to the volume and do the microversion bumps accordingly16:31
* johnthetubaguy nods16:32
ildikovnext update16:32
johnthetubaguyis there a spec to +1 around all that, I maybe missed the link?16:32
ildikovjohnthetubaguy: I will add this to the multi-attach spec after the meeting16:32
johnthetubaguyOK16:33
ildikovjohnthetubaguy: I added the info about the lock there, but wanted to get to an agreement on where we put the extra info before updating the spec again16:33
johnthetubaguy+116:33
ildikovjohnthetubaguy: I also added a note there about the policies I believe where I will need some double checks and suggestions as well16:33
johnthetubaguycool16:34
ildikovjohnthetubaguy: so highly appreciated if you can check the spec once I update it again shortly16:34
mriedemyou guys don't want a cinder spec for any of that?16:34
johnthetubaguyI think the policy should be mostly cinder side, but I should read through that16:34
johnthetubaguymriedem: I was wondering that too16:34
jgriffithmriedem yeah, we do/will16:34
jgriffithI'll have it posted prior to proposing the changes16:34
mriedemnova was going to have a policy rule for multi-attach boot from volume i think16:34
mriedemcinder was going to have a policy rule for multi-attach in general16:35
mriedemit was in my nova/cinder ptg recap email16:35
ildikovcovering that on both sides would be good, I agree16:35
ildikovI'm happy to type it up, but my knowledge on policies is fairly weak, so I'll need some help16:36
johnthetubaguysome of this could be called "configuration" depending on how you expose this in the API16:36
johnthetubaguyI guess I was meaning giving some control to the operators, however you feel is best16:37
jungleboyjmriedem:  Yeah, we were going to have policies for mulit-attach, read-only and read/write16:37
jgriffithjungleboyj we have some discussing to do around some of that on the Cinder side FWIW16:39
jgriffithbut anyway16:39
ildikovok, so let's discuss what we need to before the next meeting16:40
jungleboyjjgriffith: Ok.16:40
mriedemstvnoyes: yay https://photos.app.goo.gl/Q8JdpjM0PZhAzsv3216:40
ildikovI will update the Nova side spec and we can add a reference to any Cinder side document we will have16:40
mriedemrequired every time i review any live migration code16:40
stvnoyesnice, so simple :-)16:41
ildikovmriedem: nice! in the sense of existence16:42
johnthetubaguymriedem: that first one is a cast depending on the microversion now16:42
ildikovwe should have more of these honestly, I wish I wouldn't be that lazy to draw some when I go and re-understand these flows...16:42
mriedemjohnthetubaguy: thats from the API, which is not in this diagram16:42
mriedemi'm starting at the conductor live migration task16:43
johnthetubaguymriedem: good point16:43
ildikovok16:44
ildikovfurther items16:44
ildikovI updated the new attach patch with removing the tiny race conditions by changing the order of attachment_delete and attachment_create16:45
ildikovto have create first16:45
stvnoyes+116:45
ildikovwith jgriffith we're now in a smaller rabbit hole on tracking the volume and attachment state on the Cinder side16:45
mriedemthat's this right? https://review.openstack.org/#/c/504467/16:46
jgriffithmriedem yes16:46
mriedemi was confused how that played into the changes we needed on the nova side16:46
mriedemi guess they are separate issues?16:46
jgriffithpartially16:46
jgriffithit's a multi-attach issue for the most part16:46
jgriffithBut there's a problem with moving those things with making sure we have the right "states" on things16:47
ildikovmriedem: the problem by having create coming first is that it changes the volume state to 'reserved', which messes with delete, which then changes the volume state to 'available' which also doesn't work and that's not even multi-attach yet...16:47
jgriffithwhat I mean is not the enforcement/check of things, but at the end of actions, figuring out the disaster that is all the various states stashed here and there when you have multiple attachment records16:47
jgriffithand you don't treat them as the primary resource16:48
mriedemok16:48
mriedemso if a volume is reserved or in-use because there is an existing attachment,16:48
mriedemand we create another attachment to keep it reserved, that changes the volume status from in-use to reserved?16:48
jgriffithmriedem correct16:48
mriedemrather than just leaving it in-use16:48
mriedemok16:48
jgriffithoh...16:48
jgriffithno opposite16:48
jgriffithwhat happens currently is it will get updated to reserved16:49
jgriffithI don't think that's what we want16:49
jgriffithwe want to keep the in-use status16:49
mriedemand once we delete the old attachment, it should leave the volume reserved, but not in-use if that remaining attachment doesn't have a connector16:49
ildikovmriedem: exactly16:49
mriedemsomething like that16:49
jgriffithmriedem the last one you pointed out yes16:49
ildikovmriedem: yes and right now if you delete an attachment it will change the volume state to 'available' blindly16:49
mriedemok16:49
jungleboyjjgriffith:  Agreed.  Seems like in-use is the right status.16:49
mriedemjungleboyj: in-use depends on if the attachment has a connector16:50
mriedemfrom nova's pov, in-use or reserved are basically the same16:50
ildikovmriedem: +116:50
jgriffithhttps://review.openstack.org/#/c/504467/5/cinder/volume/api.py L#204816:50
jgriffithI'm open to suggestions/discussion if people think that ranking is incorrect16:50
jungleboyjmriedem:  Right, but it shouldn't get switched to reserved if there is a connector and it is still in-use in at least one location.  Right?16:51
mriedemjungleboyj: correct16:51
mriedemat least that's what i'd expect16:51
ildikovjgriffith: I couldn't finish my comments before the meeting, but will add them shortly16:51
jungleboyjmriedem:  ++16:51
jgriffithhonestly I'd like to propose a complete rewrite of all of the status mess in Cinder, but I think that's a bigger problem and don't want to pause the Nova attachment changes any more than we have to16:51
ildikovmriedem: and more importantly we also want to get 'attaching' and 'detaching' statuses correctly would be my thinking16:51
jungleboyjmriedem:  Granted I am new to this and trying to catch up.  If I make no sense you can tell me to go away.  ;-)16:51
jgriffithjungleboyj no!!  You don't get to "go away"16:52
*** iyamahat has quit IRC16:52
jungleboyjjgriffith:  Yeah, that would be a much larger effort and we should let this work finish and stabilize first.16:52
jgriffithand if something doesn't make sense, ping me or somebody else to figure it out16:52
ildikovjgriffith: that makes sense and as we don't want to introduce back status checking in Nova that much I would hope we can rewrite this later without any big mess16:52
jungleboyjWe have made so much progress.16:52
jungleboyjawww jgriffith  Wants me around!  :-)16:52
*** yamahata has quit IRC16:52
* jgriffith didn't say that16:52
mriedemthere is no attaching status right?16:52
* jungleboyj laughs16:53
jgriffithmriedem there is16:53
mriedemit's available, reserved, in-use, detaching, available16:53
ildikovjungleboyj: we're just eager to share the pain with as many fellow Stackers as possible ;)16:53
jungleboyj:-)16:53
mriedemok, i'll just leave this alone...16:53
ildikovmriedem: attachment_update puts the volume in 'attaching' state16:53
mriedemwell, tha'ts the attach_status yeah?16:54
ildikovand it's attachment_complete that moves it to 'in-use'16:54
mriedemthere is status, and attach_status16:54
mriedemlike vm and task state in nova16:54
mriedema vm can be ACTIVE and 'migrating'16:54
mriedemanywho16:54
ildikovmriedem: ah, ok16:54
mriedembtw,16:55
mriedemi should not know this much about the internal state machine in cinder16:55
jgriffithmriedem and now you're asking for my 20 minute rant on how F'd up all of this is on the Cinder side :)16:55
jgriffithmriedem these are why I feel the whole thing is "broken" in Cinder and needs fixed16:55
ildikovmriedem: I wouldn't volunteer right now to write up all the possible variations of the 4 statuses we're having...16:55
jgriffithmriedem agreed16:55
jgriffithanyway... I do plan to "fix" it16:55
jgriffithjust not before the nova attach changes16:56
mriedemok, ill go back to reviewing steve's live migration change16:56
ildikovmriedem: correct, Nova shouldn't play on that field in an ideal world16:56
jungleboyjThe fact that here is status and attach_status blew my mind when I realized it.16:56
jgriffithjungleboyj don't forget the attachment records (new and old) also has an independent attach_status column as well16:56
ildikovok, so I think we all have homeworks that we can move forward with16:56
jgriffithwe really want to make sure we record attach_status!!16:57
ildikovjgriffith: it's very important! :)16:57
jgriffithFWIW, my intent was that attach related statuses on the volume go away; and the attachment records are the only ones that matter16:57
jgriffith1 source of truth16:58
jungleboyjjgriffith:  ++ That sounds like the right goal.16:58
jgriffithanyway... we don't want to go into that right now probably16:58
ildikovnope16:58
ildikovwe have 2 minutes left16:58
ildikovso I wonder whether we have any other topics we need to discuss this week and haven't yet?16:58
ildikovok, it seems that was it for this week16:59
ildikovthank you all!16:59
ildikovlet's continue the chats on the channel(s)17:00
ildikovsee you here next week17:00
ildikov#endmeeting17:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"17:00
openstackMeeting ended Thu Sep 21 17:00:28 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-09-21-16.00.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-09-21-16.00.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-09-21-16.00.log.html17:00
jungleboyjThanks!17:00
*** aselius has joined #openstack-meeting-cp17:02
*** felipemonteiro_ has joined #openstack-meeting-cp17:22
*** felipemonteiro has quit IRC17:22
*** rarcea has quit IRC17:24
*** felipemonteiro has joined #openstack-meeting-cp17:25
*** mriedem has left #openstack-meeting-cp17:28
*** felipemonteiro_ has quit IRC17:28
*** iyamahat has joined #openstack-meeting-cp17:36
*** diablo_rojo has joined #openstack-meeting-cp17:53
*** diablo_rojo has quit IRC17:57
*** diablo_rojo has joined #openstack-meeting-cp17:57
*** yamahata has joined #openstack-meeting-cp18:36
*** xyang1 has quit IRC19:26
*** xyang1 has joined #openstack-meeting-cp19:37
*** gouthamr has quit IRC20:28
*** edmondsw has quit IRC20:51
*** edmondsw has joined #openstack-meeting-cp20:57
*** edmondsw has quit IRC21:01
*** felipemonteiro has quit IRC21:16
*** felipemonteiro_ has joined #openstack-meeting-cp21:18
*** felipemonteiro__ has joined #openstack-meeting-cp21:19
*** felipemonteiro_ has quit IRC21:23
*** felipemonteiro__ has quit IRC21:59
*** felipemonteiro__ has joined #openstack-meeting-cp22:00
*** iyamahat_ has joined #openstack-meeting-cp22:02
*** iyamahat has quit IRC22:02
*** felipemonteiro__ has quit IRC22:06
*** iyamahat_ has quit IRC22:23
*** iyamahat has joined #openstack-meeting-cp22:24
*** kbyrne has quit IRC22:33
*** sdague has quit IRC22:34
*** xyang1 has quit IRC22:38
*** edmondsw has joined #openstack-meeting-cp22:58
*** edmondsw has quit IRC23:02
*** iyamahat has quit IRC23:03
*** iyamahat_ has joined #openstack-meeting-cp23:03
*** MarkBaker has joined #openstack-meeting-cp23:34
*** gouthamr has joined #openstack-meeting-cp23:40
*** markvoelker has quit IRC23:40
*** MarkBaker has quit IRC23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!