Thursday, 2018-01-04

*** sdague has quit IRC00:18
*** edmondsw has joined #openstack-meeting-cp00:18
*** edmondsw has quit IRC00:23
*** yamahata has quit IRC01:58
*** smcginnis has quit IRC02:13
*** nhelgeson has quit IRC02:33
*** coolsvap has joined #openstack-meeting-cp03:26
*** stvnoyes has quit IRC03:41
*** edmondsw has joined #openstack-meeting-cp03:54
*** edmondsw has quit IRC03:59
*** lbragstad has quit IRC04:11
*** yamahata has joined #openstack-meeting-cp05:27
*** edmondsw has joined #openstack-meeting-cp05:43
*** edmondsw has quit IRC05:47
*** benj_ has quit IRC07:22
*** benj_ has joined #openstack-meeting-cp07:24
*** edmondsw has joined #openstack-meeting-cp07:31
*** edmondsw has quit IRC07:36
*** markvoelker has quit IRC07:55
*** diablo_rojo has quit IRC07:57
*** MarkBaker has joined #openstack-meeting-cp08:38
*** edmondsw has joined #openstack-meeting-cp09:19
*** edmondsw has quit IRC09:24
*** markvoelker has joined #openstack-meeting-cp09:56
*** MarkBaker has quit IRC10:15
*** MarkBaker has joined #openstack-meeting-cp10:15
*** yamahata has quit IRC10:16
*** markvoelker has quit IRC10:30
*** sdague has joined #openstack-meeting-cp11:01
*** markvoelker has joined #openstack-meeting-cp11:27
*** openstack has quit IRC11:28
*** openstack has joined #openstack-meeting-cp13:08
*** ChanServ sets mode: +o openstack13:08
*** edmondsw has joined #openstack-meeting-cp13:34
*** edmondsw has quit IRC13:35
*** dansmith has quit IRC14:25
*** dansmith has joined #openstack-meeting-cp14:28
*** dansmith is now known as Guest385614:28
*** stvnoyes has joined #openstack-meeting-cp14:30
*** lbragstad has joined #openstack-meeting-cp14:37
*** lbragstad has quit IRC14:39
*** lbragstad has joined #openstack-meeting-cp14:40
*** lbragstad has quit IRC14:41
*** lbragstad has joined #openstack-meeting-cp14:42
*** edmondsw has joined #openstack-meeting-cp14:47
*** felipemonteiro_ has joined #openstack-meeting-cp14:48
*** felipemonteiro__ has joined #openstack-meeting-cp14:50
*** edmondsw has quit IRC14:52
*** felipemonteiro_ has quit IRC14:54
*** felipemonteiro_ has joined #openstack-meeting-cp14:55
*** felipemonteiro__ has quit IRC14:55
*** MarkBaker has quit IRC15:07
*** MarkBaker has joined #openstack-meeting-cp15:16
*** edmondsw has joined #openstack-meeting-cp15:28
*** sdague has quit IRC15:29
*** sdague has joined #openstack-meeting-cp15:29
*** mriedem has joined #openstack-meeting-cp15:58
ildikov#startmeeting cinder-nova-api-changes16:01
openstackMeeting started Thu Jan  4 16:01:48 2018 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"16:01
openstackThe meeting name has been set to 'cinder_nova_api_changes'16:01
mriedemo/16:01
ildikovjohnthetubaguy jaypipes e0ne jgriffith hemna mriedem patrickeast smcginnis diablo_rojo xyang xyang1 raj_singh lyarwood jungleboyj stvnoyes16:02
ildikovmriedem: morning :)16:02
jungleboyj@!16:02
_pewp_jungleboyj (;-_-)ノ16:02
jungleboyjIn a workshop so I am kind of distracted.16:03
johnthetubaguyo/16:03
ildikovok, let's get started16:03
ildikovjohnthetubaguy: welcome back :)16:03
jungleboyjjohnthetubaguy: Hey new Daddy!16:03
johnthetubaguyhey good to be back :)16:03
ildikovah, right, Happy New Year and Parenthood Everyone! :)16:04
jungleboyjildikov: +216:04
johnthetubaguyheh, +116:04
ildikovand then back to the topic16:04
*** MarkBaker has quit IRC16:04
ildikovthe quicker part is the Cinder side changes as jgriffith promised on the Cinder channel to get something up by the end of this week16:04
* jungleboyj will be watching for those. :-)16:05
ildikovI started to look into it, but it would be certainly faster if it's taken care of by someone, who actually has some clue about the Cinder code... :)16:05
ildikovjungleboyj: great, thanks!16:05
ildikovthe Nova and Tempest changes currently are: https://review.openstack.org/#/q/topic:bp/multi-attach-volume+(status:open+OR+status:merged)16:06
ildikovthe very latest headache is a recent-ish Qemu change that prevents us from creating the second attachment16:07
johnthetubaguy... oh16:07
mriedemi've started documenting the laundry list of issues in https://etherpad.openstack.org/p/multi-attach-volume-queens16:07
ildikovmriedem: what's the current stage with that?16:07
mriedembecause i'm having a hard time keeping it all in my head16:07
ildikovI got lost between backporting and version checks and flags... :/16:07
johnthetubaguymriedem: awesome, that is exactly the kind of thing I was about to ask for16:08
mriedemsummary is qemu 2.10 breaks the shareable flag in the disk config,16:08
mriedemlibvirt 3.10 has a fix, but we test with pike UCA which is libvirt 3.616:08
mriedemubuntu might eventually backport the patches to libvirt, but we can't really wait for that16:08
ildikovmriedem: good idea, I hoped we don't have that much, but libvirt blowing up certainly proved me wrong16:08
mriedemalternative for testing today is to not use the pike UCA so we get qemu 2.516:09
mriedemi am going to test that in a bit,16:09
mriedemi'm also told that this isn't a problem if you're using raw images,16:09
mriedemsomething i'm investigating now too16:09
ildikovso we basically slightly ignore the fact that we cannot test it properly upstream recently?16:09
*** MarkBaker has joined #openstack-meeting-cp16:09
mriedemif it works with raw images, we have a way to test that by just configuring devstack properly16:09
mriedembut i need to test that out16:09
mriedemit would probably mean a separate CI job16:10
mriedemlong-term16:10
mriedemthe devstack patch as-is is super specific to the compute and volume types anyway, so probably not great,16:10
mriedemwhen other volume backends want to claim support for multiattach and enable it in their CI16:10
ildikovI guess we could try to group the laundry list by what needs to get landed in the upcoming 2,5 weeks and what can wait a tiny bit16:10
mriedemwell, priority 1 for me is getting a CI run where we can actually do the 2 attachments16:11
mriedemeverything falls after that16:11
jgriffithmriedem: +116:11
ildikovI think we favored turning this on in Cinder one-by-one so being a bit mroe specific for now is ok16:11
ildikovmriedem: +116:11
mriedeman override flag in devstack is easy to add,16:12
mriedemi just haven't done it yet16:12
ildikovmriedem: it doesn't sound like a famous last sentence if you say it :)16:13
mriedemthe other major things are the cinder bp for the policy rules and such, which we already mentioned,16:13
mriedemand figuring out what to do in the compute api about the new microversion16:13
mriedemcomments in https://review.openstack.org/#/c/271047/3716:13
mriedemas this code is written, i can attempt to multiattach regardless of knowing if the compute supports it at all16:14
ildikovI think we said a new microversion in Cinder too, I'm still not checking for 3.48 in the API yet in Nova16:14
mriedemand i think that needs to be tightened up16:14
mriedemwhat is 3.48?16:14
mriedemshared_targets?16:14
ildikovyes16:14
ildikovI think we wanted that for multi-attach and happy to have it otherwise16:15
ildikovbut well, I can bump the compute version referring to the libvirt and other changes16:15
ildikovand do the same thing as with adding that additional reserve_volume call a while ago16:16
mriedemi don't know that we need a minimum compute version bump though, except maybe in the boot from volume case16:16
ildikovI guess now that we're relying on qemu and libvirt versions checking the driver capabilities is even more important early on16:16
mriedemnormal attach does an rpc call to the compute using the reserve_block_device_name method, and we can pass the multiattach flag from the api to the compute and check the capability there, and fail fast if the compute doesn't support multiattach volumes16:16
mriedembut for boot from volume, we don't know the host yet, so in that case the min compute version matters...16:17
ildikovand we're talking about the second and further attachments, right?16:17
mriedemall attachments to a multiattach=True volume16:18
mriedembecause that's what the libvirt patch is checking for16:18
mriedemif the volume says multiattach and the driver doesn't support it, the attach fails16:18
mriedemregardless of it being the first or the 10th16:18
ildikovyou said something in the review that we shouldn't break the current behavior16:18
ildikovand currently we don't care whether it's mutli-attach or not16:19
ildikovI prefer failing though16:19
johnthetubaguywho is we, cinder?16:19
ildikovjohnthetubaguy: Nova16:19
johnthetubaguyah, its nova16:19
johnthetubaguyso today we allow multi-attach?16:19
* johnthetubaguy is confused again16:20
mriedemwell, since pike you can't do multiattach period16:20
mriedemno in-tree cinder drivers support it16:20
ildikovno, I mean that today we allow attaching a multi-attach volume regardless of any support16:20
ildikovobviously we allow it to be attached only once16:20
mriedemhttps://review.openstack.org/#/c/428365/16:20
mriedemsince pike, you can't even create a multiattach=True volume16:20
mriedemscheduling fails16:20
ildikovyeah, that helps16:20
johnthetubaguyright, I see16:20
ildikovwe said we turn it off to be sure16:21
ildikovwe can always argue about volumes from the ice age though...16:21
johnthetubaguyseems like cinder needs to track new vs old attachments then16:21
ildikovboot from volume will fail on the compute anyway16:21
johnthetubaguyalthough frankly a breaking API change is probably best, given its breaking zero users16:21
ildikovI mean if the env is new enough, but the hypervisor doesn't support it then it's the same as landing on an old compute16:22
mriedemso i think i want to say that in nova, you can only attempt to attach a multiattach volume using the new microversion, period.16:22
mriedemwon't work with 2.1 or 2.20 or whatever16:22
johnthetubaguymriedem: +1 that16:22
ildikovjohnthetubaguy: except those who patched both Cinder and Nova, but I guess they will figure it out by themselves how to upgrade :)16:22
johnthetubaguyway simpler test matrix16:22
mriedemso if new microversion and volume.multiattach=True, then we have to figure out what checks we make about the backend support16:23
johnthetubaguyildikov: yup, there are in crazy land16:23
johnthetubaguythey16:23
mriedem1. boot from volume could be a min compute service version check,16:23
mriedem2. normal attach could be checked via the reserve_block_device_name call to the compute16:23
mriedem3. attaching to a shelved offloaded server....i'm not sure16:23
johnthetubaguywas reading your note about shelved offloaded, oh my16:24
mriedem3 is essentially the same as 116:24
ildikovmriedem: considering it doesn't guarantee much, I'm not sure we actually want the compute version check16:24
mriedemyou don't know the host until you unshelve16:24
johnthetubaguyyeah, I think that is the best approach16:24
johnthetubaguyfail as fast as we can, but meh, you have a strange setup16:24
mriedemildikov: without a scheduler filter, it's better than nothing16:25
johnthetubaguyit stops a world of crazy during upgrade, that seems like a win16:25
mriedemotherwise we just say operators have to disable multiattach on the cinder side until all of their computes are upgraded16:25
jgriffithmriedem: that's likely the best option IMO, the other I thought we discussed was prohibit BFV for now but I don't want to go down a tangent16:26
ildikovmriedem: if it's a mixed hypervisor environment then we're screwed anyway...16:26
mriedemildikov: yes in that case you are16:26
mriedembut i don't know how many deployments actually used mixed hypervisors16:26
mriedemi would assume that's rare16:26
mriedemwithin the same region anyway16:26
stvnoyessorry, joining late...16:26
ildikovjgriffith: I removed that 5 lines of code that disables booting from a multi-attach volume, you could still attach it at boot time if I put it back16:27
johnthetubaguywell, and require multi-attach, that seems like vanishing small16:27
mriedemjgriffith: we said we'd allow bfv with multiattach and if the operator didn't like that idea they'd disable it via policy in cinder16:27
ildikovstvnoyes: no worries, we're still pretty much in the middle of everything :)16:27
johnthetubaguymriedem: +116:27
ildikovmriedem: ok, bumping a compute version is not a big deal, so I can do that16:27
jgriffithmriedem: ildikov right, I wasn't proposing anything just reviewing past discussions16:28
mriedemildikov: yeah i think we want that for the libvirt patch that adds the capabilty flag16:28
ildikovjgriffith: between the two of us, I liked it better disabled... :)16:28
ildikovmriedem: ok, I will add it there16:28
ildikovmriedem: so we should differentiate and only check it for BFV?16:28
mriedembfv and attaching to a shelved offloaded instance16:29
mriedemin both cases we don't know what the host is going to be16:29
ildikovok16:29
mriedemi don't think we need to check for cinder 3.48 because the code in https://review.openstack.org/#/c/529695/ is backward compatible and will just detach as before if it's not available16:30
mriedemit just means it's not a safe detach16:30
ildikovok, fair enough16:30
ildikovI have the workaround code to check the Cinder microversion now, so we can add it any time if we would change our mind later16:31
mriedemildikov: just a thought, but if we add a multiattach arg to the reserve_block_device_name rpc call method,16:33
mriedemthat gives us the service version bump automatically,16:33
ildikovand for other attach cases just check whether the hypervisor supports it and fail if not16:33
mriedemand that's the thing the normal attach flow from the api could fail fast on if the compute is too old or doesn't support multiattach16:33
mriedemildikov: so i think we could do that in between the libvirt patch and the api change16:34
mriedemi could write that up today16:34
ildikovwell, I think we can make it a rule that if you write it up I won't argue :)16:34
mriedemok so notes are in https://etherpad.openstack.org/p/multi-attach-volume-queens16:35
ildikovback to serious, sounds like a good enough way forward16:35
mriedemi'll start by trying to get the test stuff sorted out16:35
ildikovmriedem: is there anything stvnoyes and/or me can help out with?16:35
johnthetubaguyquick question on some comments in the patchset 37?16:35
ildikovmriedem: did you agree with stvnoyes what he could take on?16:35
johnthetubaguyyou said nova checks how many attachments the multi-attach volume already had?16:35
stvnoyesi can take a look at the new version of libvirt and test that16:36
ildikovso we take care of things we can independently and/or help each other where it makes sense16:36
mriedemjohnthetubaguy: where?16:36
johnthetubaguyhttps://review.openstack.org/#/c/271047/37/nova/compute/api.py@365916:36
ildikovjohnthetubaguy: I think we fall back to just don't attach a multi-attach volume if we cannot regardless of it's attached already or not16:37
johnthetubaguyildikov: +1 that16:37
mriedemi'm fine with that16:37
mriedemthis was before i knew that cinder blocked multiattach altogether in pike16:37
mriedemand was thinking about backward compat16:38
ildikovI think we discussed that on one of the meetings back then, but it was a while ago16:38
ildikovand I'm not really the queen of documentation, neither the Cinder guys so I will just blame them anyway :)16:38
johnthetubaguyheh, either way, sounds like we are good there16:39
ildikovcool, at least one thing :)16:39
ildikovstvnoyes: I think mriedem plans a DNM devstack patch to get an earlier qemu on the gate16:40
mriedemthat or seeing if raw images work16:40
*** sdague has quit IRC16:40
mriedemi'm going to try the raw images thing first16:40
*** sdague has joined #openstack-meeting-cp16:40
stvnoyesok, is there still a desire to test with 3.10 libvirt?16:40
ildikovmriedem: what level of test coverage we would like to have before code freeze?16:41
mriedemit would be nice to know that libvirt 3.10 actually works with qcow2 images for multiattach volume16:41
stvnoyesok, I'll take a run at that16:41
mriedemildikov: probably the test in https://review.openstack.org/#/c/266605/17/tempest/api/compute/volumes/test_attach_volume.py and the TODOs in there, plus attaching a multiattach volume to a shelved offloaded instance16:42
mriedemso like,16:42
mriedemattach a volume to an instance A,16:43
mriedemcreate instance B, shelve offload it, attach volume to B and unshelve B16:43
mriedemanyway, need to get happy path working first16:43
ildikovsure, sounds good16:44
stvnoyesa minor point - it would also be good to have the detaches be explicit in the test. otherwise detach failures in teardown are not as obvious to debug.,16:44
mriedemstvnoyes: yeah andreaf pointed that out too, i plan on adding that in16:44
ildikovstvnoyes: when the libvirt stuff seems to get together if you could take a look at the shelved offloaded case that would be pretty great16:44
stvnoyesok will do16:45
ildikovok, so summary16:46
ildikovjgriffith works on the Cinder side policy and any other related changes to enable multi-attach, target to have some code up is end of this week16:46
ildikovmriedem looks into the libvirt-qemu changes to get either a lower version qemu or test with raw image and he will add a patch on top of the libvirt one on adding a multiattach arg to the reserve_block_device_name rpc call method16:47
ildikovstvnoyes looks into libvirt 3.10 and the shelved offloaded case for Tempest after that16:48
ildikovildikov continues to work on the libvirt patch and the dependencies below to get the shared_target changes in if anything pops up16:49
ildikovwhat did I miss?16:50
mriedemthat's good for now16:50
mriedemwe'll have some api patch changes on top later16:50
mriedemfor the version checks and such16:50
jungleboyjSounds good.16:51
ildikovmriedem: you mean the BFV and other checks?16:52
ildikovI can do that once you have that additional patch16:52
*** Guest3856 is now known as dansmith16:53
mriedemyeah, and we'll need something from the actual REST API handler code to check the microversion and pass a variable down to say if multiattach is OK16:53
mriedemlike, "allow_multiattach=False"16:53
mriedemsomething like that16:53
mriedemif microversion >= 2.59: allow_multiattach=True16:53
mriedemshall we break?16:55
*** sdague has quit IRC16:55
ildikovyeah, I'm sure I will have questions to this last one, but can break for now16:56
*** sdague has joined #openstack-meeting-cp16:56
*** yamahata has joined #openstack-meeting-cp16:56
ildikovwill monitor your etherpad16:56
ildikovBTW, if anything new comes up (fingers crossed it doesn't happen) please add it to here: https://etherpad.openstack.org/p/multi-attach-volume-queens16:57
*** sheel has joined #openstack-meeting-cp16:57
ildikovthanks everyone!16:57
jungleboyjThanks ildikov  !16:57
jungleboyjand everyone else.16:57
ildikovkeep on chatting on the channels16:57
ildikov#endmeeting16:58
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"16:58
openstackMeeting ended Thu Jan  4 16:58:13 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:58
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2018/cinder_nova_api_changes.2018-01-04-16.01.html16:58
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2018/cinder_nova_api_changes.2018-01-04-16.01.txt16:58
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2018/cinder_nova_api_changes.2018-01-04-16.01.log.html16:58
*** felipemonteiro__ has joined #openstack-meeting-cp17:04
*** felipemonteiro_ has quit IRC17:07
*** mriedem has left #openstack-meeting-cp17:17
*** felipemonteiro__ has quit IRC17:24
*** felipemonteiro__ has joined #openstack-meeting-cp17:24
*** felipemonteiro__ has quit IRC17:25
*** felipemonteiro__ has joined #openstack-meeting-cp17:25
*** MarkBaker has quit IRC17:35
*** david-lyle has quit IRC18:00
*** david-lyle has joined #openstack-meeting-cp18:01
*** stvnoyes has quit IRC18:05
*** yamahata has quit IRC18:26
*** felipemonteiro_ has joined #openstack-meeting-cp18:40
*** felipemonteiro__ has quit IRC18:40
*** stvnoyes has joined #openstack-meeting-cp18:53
*** nhelgeson has joined #openstack-meeting-cp18:53
*** yamahata has joined #openstack-meeting-cp19:04
*** sheel has quit IRC19:06
*** felipemonteiro__ has joined #openstack-meeting-cp19:35
*** felipemonteiro_ has quit IRC19:39
*** felipemonteiro has joined #openstack-meeting-cp19:47
*** felipemonteiro__ has quit IRC19:50
*** edmondsw has quit IRC20:54
*** felipemonteiro has quit IRC21:18
*** felipemonteiro has joined #openstack-meeting-cp21:19
*** edmondsw has joined #openstack-meeting-cp21:19
*** felipemonteiro has quit IRC21:26
*** felipemonteiro has joined #openstack-meeting-cp21:26
*** smcginnis has joined #openstack-meeting-cp21:27
*** felipemonteiro_ has joined #openstack-meeting-cp21:53
*** felipemonteiro has quit IRC21:57
*** felipemonteiro_ has quit IRC22:46
*** felipemonteiro_ has joined #openstack-meeting-cp22:46
*** edmondsw has quit IRC22:56
*** edmondsw has joined #openstack-meeting-cp22:57
*** edmondsw has quit IRC23:01
*** felipemonteiro_ has quit IRC23:14
*** stvnoyes has quit IRC23:18
*** mtreinish has quit IRC23:39
*** mtreinish has joined #openstack-meeting-cp23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!