14:01:38 <jokke_> #startmeeting glance
14:01:39 <openstack> Meeting started Thu Oct 25 14:01:38 2018 UTC and is due to finish in 60 minutes.  The chair is jokke_. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:44 <openstack> The meeting name has been set to 'glance'
14:01:46 <abhishekk> o/
14:01:47 <jokke_> #topic roll-call
14:01:52 <jokke_> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:01:54 <jokke_> o/
14:01:57 <mhen> o/
14:02:02 <LiangFang> o/
14:02:10 <rosmaita> o/
14:02:34 <jokke_> #topic updates
14:02:52 <jokke_> First of all, we have S-1 deadline today
14:03:09 <rosmaita> that snuck up quickly
14:03:12 <jokke_> there shouldn't be anything burning waiting to be merged but now is the last moments to yell
14:03:26 <jokke_> rosmaita: indeed
14:03:41 <abhishekk> I think we should have my functional tests timeout patch in
14:04:20 <rosmaita> also, this would be the last chance to revert the image status change from queued -> active from PATCH add location
14:04:41 <jokke_> rosmaita: you mean replace?
14:05:01 <rosmaita> sorry, you are correct, replace
14:05:20 <jokke_> yes so are we going to keep that or revert it?
14:06:19 <rosmaita> i am in favor of keeping it
14:06:26 <abhishekk> me too
14:06:36 <jokke_> I still think it's unnecessary api change
14:07:01 <rosmaita> i don't see it as an api change, i see it as a bugfix
14:07:03 <smcginnis> I don't have enough knowledge there to vote one way or the other.
14:07:37 <jokke_> but but taken into account where it is, I really don't have interest to fight about it
14:08:00 <rosmaita> yeah, pretty low traffic in that part of the api
14:08:14 <jokke_> need to bump the api version for that 'though
14:08:42 <rosmaita> not for S-1, though, i wouldn't think
14:09:14 <jokke_> you recon it's nuf to do it at the end of the cycle?
14:09:46 <abhishekk> generally we do it at the end of the cycle, right?
14:09:59 <rosmaita> i really don't think it's necessary for this at all, but i think that's the usual practice, end of cycle
14:10:08 <rosmaita> no one has complained before
14:10:20 <rosmaita> or at least, i haven't payed attention if they have :)
14:10:24 <jokke_> k, I guess we should be good to go then with S-1
14:10:48 <jokke_> next thing is, Summit is approaching, fast
14:10:53 <rosmaita> abhishekk: do you have your func test update patch link handy?
14:10:56 <abhishekk> (I am talking about this, https://review.openstack.org/#/c/608856/)
14:11:03 <jokke_> which means spec freeze is approaching fast
14:11:07 <abhishekk> rosmaita, ^^
14:12:02 <jokke_> so lets get them specs reviewed and merged what we are going to accept
14:12:14 <jokke_> which gives us nice brindge to the next topic
14:12:19 <jokke_> bridge
14:12:43 <jokke_> #topic validation_data in image locations patch call
14:13:29 <jokke_> imacdonn has been chasing this actively and has new revision out
14:13:29 <abhishekk> new revision looks good to me, does it going to add new discovery call?
14:14:30 <rosmaita> abhishekk: it doesn't mention that, i think the idea is rely on the local operator docs for now
14:14:51 <abhishekk> rosmaita, ack
14:15:22 <rosmaita> since the admin has to allow use of locations anyway, should not be a big deal
14:15:27 <jokke_> we shoud definitely look to include these details when expanding the discovery endpoint
14:15:37 <rosmaita> i mean the admin/operator
14:15:44 <rosmaita> jokke_: ++
14:16:06 <jokke_> that said I don't think this will be widely used due to the fact that it opens such a can of worms
14:16:20 <rosmaita> also ++
14:16:28 <abhishekk> so it will just check it against config set in glance-api.conf and if does not match then returns 409
14:16:47 <abhishekk> agree
14:16:57 <rosmaita> right
14:17:09 <abhishekk> got it
14:17:29 <jokke_> ok, cool
14:17:53 <jokke_> lets read it through once more with good thought but I think it's getting there (again)
14:18:20 <jokke_> next up is showing store IDs in image listing
14:18:35 <jokke_> #topic show store ID in image listing with -vv
14:18:43 <jokke_> LiangFang: you're up
14:18:43 <LiangFang> hi erno
14:19:25 <LiangFang> I prepared the spec, and at this moment, the conclution is to add --include-stores option to image-list
14:19:59 <LiangFang> openstack client just have -long, not option like -vv
14:20:27 <LiangFang> so try to use --include-stores
14:20:54 <LiangFang> erno, are you agree with this?
14:21:27 <rosmaita> LiangFang: are you thinking the --include-stores will work with both "regular" image-list and also image-list -v calls?
14:21:36 <jokke_> so are you proposing thing change to python-glanceclint or openstackclient?
14:22:04 <LiangFang> the change is in python-glanceclient
14:22:19 <LiangFang> glance -v image-list --include-stores
14:22:35 <LiangFang> glance image-list --include-stores
14:22:38 <LiangFang> all can work
14:22:42 <rosmaita> excellent
14:23:05 <smcginnis> jokke_: osc comes into play from a comment I made about it not really fitting with the standardization they are doing there.
14:23:13 <jokke_> ok, I have one concern which might be just language thing in the spec
14:23:19 <smcginnis> I think this new option fits better and is probably easier for a user to discover and use.
14:23:41 <rosmaita> i think smcginnis had a good point about that
14:24:36 <jokke_> You're mentioning listing store type, I think that needs to change to store ID (and use those, "fast", "cheap", "reliable" as example) ... we are not going to leak the backend information to the user unless that's how the operator decides to set it
14:25:04 <LiangFang> OK
14:25:12 <jokke_> so we're not listing the store type, but it's ID
14:25:13 <smcginnis> jokke_: Good point.
14:25:15 <LiangFang> I will update
14:25:26 <jokke_> other than that, the proposal looks good to me
14:25:26 <rosmaita> good catch, i missed that
14:25:56 <LiangFang> thanks
14:26:07 <LiangFang> and I also updated the related code
14:26:32 <LiangFang> and the review is:https://review.openstack.org/#/c/605014/
14:27:05 <jokke_> thanks, anything else about this?
14:27:11 <LiangFang> no more
14:27:13 <LiangFang> thanks
14:27:32 <jokke_> #topic image size to prevent backend resizes
14:27:38 <jokke_> this is yours as well
14:28:08 <LiangFang> about this, last week you mentioned we should consider other component as well, not only cinder
14:28:09 <jokke_> and first of all I have to say this keeps slipping on me, so no I haven't got to test it yet :(
14:28:31 <LiangFang> that's fine:)
14:29:05 <jokke_> yes so we need to make sure that this addresses any applicable backends (if others than cinder is affected)
14:29:37 <LiangFang> considering this, I think solution 1 is better
14:29:41 <jokke_> #link https://review.openstack.org/#/c/608400/
14:29:45 <LiangFang> solution 1 is:
14:29:46 <jokke_> ^^ is the spec
14:30:06 <LiangFang> cinder calculate the image size
14:30:29 <LiangFang> python-glanceclient: convey the image size via http header
14:30:58 <LiangFang> glance: extract the image size in request deserialize phase
14:31:39 <LiangFang> Sean mentioned previously that, move the work did in cinder to python-glanceclient
14:32:11 <LiangFang> let python-glanceclient to calculate the image size
14:32:35 <LiangFang> I take a look again this week, seems this is workable
14:32:44 <rosmaita> Quick question: the point of this is to provide info that the glance backend driver can use in allocating how much space to store the image, but the 'size' in glance will still be computed as it is now?
14:34:00 <abhishekk> how this will fit in case of web-download import method?
14:34:00 <LiangFang> in current code, size is returned by backend when it finished all the data upload
14:34:51 <jokke_> So iirc we can't access the request body before all the data has been uploaded to glance (this was either webop or wsgi thing). Whic means that we should know the size before we upload it to backend
14:35:11 <LiangFang> abhishekk: let me take a think
14:35:15 <jokke_> also it's definitely the case with image import
14:35:54 <abhishekk> LiangFang, yes
14:36:06 <jokke_> so I don't think we should offload the work to glanceclient specially as there is cases when glanceclient can't determine the size before it's iterated the whole image
14:37:00 <jokke_> those being if the data is streamed to the client from command line or the fd is socket
14:37:32 <LiangFang> agree
14:37:58 <LiangFang> so should let the user of glanceclient to give the size
14:38:10 <jokke_> and these were the reasons why the decision was made not to send the size from client in v2 like it was done in v1
14:38:39 <jokke_> we should let glance-api to calculate the data before upload to the backend
14:38:56 <smcginnis> Are there any DoS issues with user supplied size input to glanceclient?
14:39:01 <jokke_> and make sure our upload is not streaming it from local disk in very small chunks
14:40:12 <jokke_> smcginnis: not really, there is the max image size parameter that eliminates any crazy calls
14:40:20 <smcginnis> OK, good.
14:40:56 <jokke_> but yet I think we have all the needed data in the api before we do the upload call, it's just not utilized
14:41:57 <LiangFang> I seems glance-api don't know the size
14:42:04 <jokke_> so we update the image size based on the info we got from the backend instead of telling the backend "We have this much data we're dumping into you"
14:42:54 <LiangFang> I did experiment with ceph as backend, if give size, it can save 30% uploading time
14:43:13 <jokke_> yeah, I thought that might be the case
14:44:30 <jokke_> one of the problems we have and this is hitting the edge usecases hard is that we always thought the backends are somewhere close by. We're using extremely small chunks when doing backend transactions. like 4k which is even small for disk writes
14:45:39 <LiangFang> the chunk size using by ceph is 8M
14:45:46 <LiangFang> lvm is 64K
14:45:55 <rosmaita> so really, for this purpose, we don't need to know the image size exactly,
14:46:13 <jokke_> yeah, well with ceph we use rdb which takes care of it
14:46:42 <jokke_> when ever we chnk it ourselves it's very very small
14:48:03 <LiangFang> sorry erno, I may not catched your point
14:48:17 <jokke_> rosmaita: well I'm not sure how cinder behaves if you try to shrink the volume ... the problem is atm, that when we don't know the size, behind the scenes we end up calling cinder resize to expand for each chunk we're sending
14:48:50 <jokke_> same happens with other backends that actually allocate the space
14:48:53 <smcginnis> Cinder will never shrink, only extend.
14:49:25 <jokke_> yeah, so we can't just estimate and shrink back to size after we get to the end of the data
14:49:26 <abhishekk> yes, downsize is not allowed
14:50:31 <jokke_> ok, we're running out of time. Lets keep investigating this
14:50:39 <LiangFang> OK
14:50:52 <jokke_> #topic python-glanceclient release
14:50:57 <jokke_> rosmaita: the stage is yours
14:51:10 <rosmaita> I send something to the ML about this
14:51:16 <rosmaita> #link http://lists.openstack.org/pipermail/openstack-dev/2018-October/135902.html
14:52:32 <rosmaita> master contains the multihash validation code
14:52:59 <abhishekk> at the moment 2nd option looks more convenient
14:53:19 <rosmaita> yes, and smcginnis already has a patch up for that
14:53:58 <rosmaita> and now that we've decided that iain's validation_data will use the configured hash algo, i'm not so worried
14:54:30 <smcginnis> We can hold the release patch or update it. There's also one up for glance_store.
14:54:41 <rosmaita> ok, this looked like it could be an issue at the end of last week, not so much now.
14:55:19 <rosmaita> let's go with option 2
14:55:32 <jokke_> so we don't have the multihash validation in rocky client and we never ended up backporting that to rocky branch
14:55:45 <rosmaita> jokke_: correct
14:56:41 <jokke_> ok, crap
14:56:54 <jokke_> I thought it had been done, just not tagged
14:57:10 <rosmaita> give me a minute to check, pretty sure i;m right though
14:57:30 <jokke_> just looked the rocky branch, didn't see it there
14:58:06 <rosmaita> yeah, not there
14:58:24 <jokke_> so yeah, I guess we need to do the same trickstery we did at the start of rocky
14:59:11 <jokke_> which is backport, cut 2.13.0 from the stable branch and then 2.14 from master before towards end of the cycle cutting 3.0.0
14:59:46 <abhishekk> last minute to go
14:59:59 <jokke_> that way we get it for those who insists using the client from rocky branch on their rocky deployment
15:00:15 <rosmaita> abhishekk: are you handling glance s-1 release, or do you want me to do it?
15:00:24 <jokke_> yup, we're out of time, lets wrap this client plan together in #os-glance
15:00:33 <jokke_> #endmeeting