14:00:26 <jokke_> #startmeeting glance
14:00:27 <openstack> Meeting started Thu Nov  8 14:00:26 2018 UTC and is due to finish in 60 minutes.  The chair is jokke_. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:30 <openstack> The meeting name has been set to 'glance'
14:00:33 <jokke_> #topic roll-call
14:00:36 <jokke_> o/
14:00:48 <rosmaita> o/
14:01:12 <jokke_> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:01:22 <jokke_> Morning rosmaita
14:01:33 <rosmaita> good afternoon to you!
14:02:15 <jokke_> lets give couple of minutes to see who else is joining us
14:02:40 <rosmaita> sounds good, full agenda, so we should have some more people soon
14:05:02 <jokke_> ok
14:05:07 <jokke_> #topic updates
14:05:28 <jokke_> So Berlin Summit is in next week, no meeting
14:05:38 <rosmaita> ok
14:05:41 <jokke_> Spec deadline is 2 weeks after summit
14:05:56 <jokke_> so 3 weeks to go to get them specs in
14:06:26 <rosmaita> jokke_: that reminds me, did you ever send out the operator survey about changing default image visibility?
14:06:41 <jokke_> FECK
14:06:48 <jokke_> It's going out today!
14:06:52 <rosmaita> i was thinking that will require a spec lite
14:06:55 <rosmaita> ok, cool
14:07:11 <jokke_> totally forgot that
14:07:27 <jokke_> yeah
14:07:30 <rosmaita> also, procedural question
14:07:42 <jokke_> shoot
14:07:43 <rosmaita> for deprecated config opts, do we need a spec-lite to remove them?
14:07:52 <rosmaita> i'm thinking of owner_is_tenant
14:08:03 <jokke_> nah, if they are deprecated we clean them out and have reno for them
14:08:17 <rosmaita> ok, that gives us some flexibility
14:08:53 <jokke_> nice, something else on this?
14:09:20 <rosmaita> nope, other than maybe i'll take an action item to make a list of deprecated opts that haven't been removed
14:09:28 <jokke_> sounds good
14:09:31 <rosmaita> could be some low-hanging fruit
14:09:42 <jokke_> #topic periodic jobs
14:09:48 <rosmaita> #action rosmaita compile list of deprecated options
14:09:53 <rosmaita> all green!
14:09:56 <jokke_> I see note on the etherpad that everything is green :)
14:10:04 <rosmaita> except occasionally the keystone tips functional py35
14:10:13 <rosmaita> http://zuul.openstack.org/builds?project=openstack%2Fglance&pipeline=periodic&job_name=glance-tox-functional-py35-keystone-tips
14:10:20 <jokke_> yah, have you had any eye on those what times out?
14:10:34 <rosmaita> no, i haven't looked
14:10:38 <jokke_> kk
14:10:46 <rosmaita> for a week it looked like abhishek's change had fixed it
14:10:51 <rosmaita> and then it failed again yesterday
14:10:57 <jokke_> yeah
14:11:28 <jokke_> might be something us related, might be something test framework related, might be something infra related. Lets keep eye on them
14:11:33 <rosmaita> yep
14:11:46 <rosmaita> that's all on periodic jobs
14:11:59 <rosmaita> we should wish Happy Diwali to everyone not here, though
14:11:59 <jokke_> #topic smuggling validation data on location add/replace
14:12:31 <jokke_> Soo I saw your review there and went ahead and merged the spec
14:12:40 <rosmaita> great
14:12:49 <jokke_> it obviously hit merge conflict but I think it finally merged this morn
14:13:41 <rosmaita> slightly off topic, the image_cache uses checksum for download validation
14:13:55 <rosmaita> i don't know if we want to take a performance hit on that to use multihash?
14:14:18 <jokke_> I think we do
14:14:34 <rosmaita> ok, i will file a bug
14:14:41 <jokke_> specially as it seems that the md5 is running away
14:14:54 <rosmaita> well, that's not completely clear
14:15:00 <rosmaita> it's ok for non-security uses
14:15:15 <rosmaita> like making sure no bit flips during transfer
14:15:22 <jokke_> yeah, but I rather be prepared for the environments where it's not available
14:15:30 <rosmaita> i am still trying to get a timeline on that
14:16:03 <rosmaita> yeah, we should probably assume not available
14:16:12 <jokke_> cool ... so this is not our first priority issue, just something we want to keep track of and be ready to act if needed
14:16:36 <rosmaita> #action rosmaita file glance bug about image cache does not use multihash to verify download
14:16:47 <jokke_> another good low hanging fruit for someone who suddenly has afternoon of free time :D
14:17:08 <rosmaita> yeah, there's some example code in glanceclient
14:18:12 <rosmaita> i'll try to carve out some time to test out imacdonn's patches, i have an S-1 devstack up
14:19:13 <jokke_> same here, I need to just reroll my containers running my devstacks
14:19:41 <rosmaita> i need to get with the containers program, i am still using VMs
14:20:11 <jokke_> I just run lxc bare and simple :D
14:20:19 <jokke_> ok lets move on
14:20:30 <jokke_> #HTTP header encoding
14:20:36 <jokke_> #topic HTTP header encoding
14:20:44 <rosmaita> i guess this is me
14:20:52 <rosmaita> well, it's my fault, anyway
14:21:02 <jokke_> :o
14:22:10 <rosmaita> we had that issue back in early rocky where some update to something broke something, and the glanceclient tests were breaking something, so the problem was sending unicode in headers which isn't strictly allowed, it's supposed to be 7bit ascii only
14:22:48 <jokke_> yes, I recall that
14:22:48 <rosmaita> so i put up a change to use urlencoding to get unicode->7 bit ascii, and then sent that over in glanceclient headers
14:23:14 <rosmaita> problem is, urlencoding may be a bit too much
14:23:36 <rosmaita> it encodes some chars c where ord(c) < 128, which you don't need to do
14:23:50 <jokke_> ok
14:23:50 <rosmaita> but the standards are a bit vague on how this should be done
14:24:05 <rosmaita> anyway, we are gradually finding a case here or there that this change breaks
14:24:26 <rosmaita> i think wxy found sometimes the token was getting corrupted and fixed that
14:24:39 <jokke_> I'm pretty sure, we're not the only ones dealing with this. Is there any safe-decode libs to just handle it?
14:24:48 <rosmaita> and ian found that the hostname part of urls are being hosed
14:24:57 <rosmaita> i haven't found anything
14:25:09 <rosmaita> the lib we are using allows you to specify a set of safe chars
14:25:17 <rosmaita> that's what wxy and ian do
14:25:27 <rosmaita> but i don;'t know of a list anywhere
14:25:33 <rosmaita> anyway, here is the actual point
14:25:52 <rosmaita> i was holding off on ian's patch until we figure out a thorough way to fix this
14:26:00 <rosmaita> but no one seems to have time ATM to tackle that
14:26:18 <rosmaita> so i have +2'd his patch to at least get this particular case he's running into fixed
14:26:44 <jokke_> can we just tell the lib "do not touch anything ord(c) < 128?
14:26:57 <jokke_> s/?/"//
14:27:04 <jokke_> 2?
14:27:07 <jokke_> "?
14:27:09 <rosmaita> i think we could, by passing it a list of 127 chars!
14:27:09 <jokke_> :P
14:27:16 <jokke_> ohhh :|
14:27:39 <rosmaita> yeah, but my idea was to scan the string first to see if any chars are out of range before doing anything
14:27:45 <rosmaita> probably the same complexity
14:28:17 <rosmaita> problem is, we still don't know what to do when we have a string with out-fo-range chars
14:28:52 <jokke_> Is there any reason why we shouldn't just reject it?
14:29:10 <rosmaita> yeah, we need to support unicode
14:30:11 <rosmaita> it's glancelient that had the tests that were using unicode, so i figured it was important to us at some point
14:30:44 <rosmaita> i think i should take an action item to write up a new bug for this and be clear about exactly what the problem is
14:30:50 <jokke_> Well it WAS at least with v1
14:31:01 <rosmaita> that's true
14:31:04 <jokke_> where all tag and property stuff happened over headers
14:32:21 <rosmaita> so i think we should either (a) find a library that handles this properly (there must be one if it's a real problem) or (b) take your string-rejection strategy
14:32:25 <jokke_> now we are getting back to the same problem again
14:32:38 <rosmaita> the codec gives a nice exception if the string is out of range
14:32:47 <jokke_> as for example the store IDs for where the image will be stored goes over headers again
14:32:53 <rosmaita> so we wont' need to scan or anything
14:33:01 <rosmaita> oh yeah
14:33:11 <rosmaita> that is still experimental
14:33:20 <rosmaita> we could require store IDs to be 7bit ascii
14:33:27 <rosmaita> the descriptions can be unicode
14:33:56 <jokke_> yeah, that's a bit of bandaid but I guess so
14:34:15 <jokke_> I think we need to keep this hard on mind before dumping any more crap to the headers
14:34:29 <jokke_> and start being more strict that the stuff needs to happen before
14:34:40 <jokke_> and use the json based calls
14:34:56 <rosmaita> #action rosmaita propose patch to multi-backend config opts saying store ID must be 7-bit ASCII
14:35:16 <rosmaita> yeah, problem is that it makes the actions extremely un-atomic
14:35:39 <rosmaita> i guess we could do multipart uploads
14:35:45 <rosmaita> json+data
14:36:10 <jokke_> jayy for HTTP based APIs ;)
14:36:49 <jokke_> I see only Imperial fecktons of ways that would go so badly wrong :D
14:36:59 <rosmaita> yeah, no kidding
14:38:58 <LiangFang> o/
14:40:13 <jokke_> ok moving on
14:40:27 <jokke_> #topic list --include-stores
14:40:30 <rosmaita> anyway, back to the point, are you ok with imacdonn's quick-fix patch?
14:40:46 <rosmaita> oops, sorry about the slow typing
14:40:58 <jokke_> #undo
14:40:59 <openstack> Removing item from minutes: #topic list --include-stores
14:41:14 <jokke_> I will need to review the patch
14:41:29 <rosmaita> sounds good, just wanted to verify you are not opposed in principle
14:41:30 <jokke_> I'll put it on my list to do before heading to berlin
14:41:36 <rosmaita> ty
14:41:40 <rosmaita> i'm done
14:41:44 <jokke_> #topic list --include-stores
14:41:51 <jokke_> there we go
14:41:58 <LiangFang> hi Jokke
14:42:01 <jokke_> The spec was also merged this morn
14:42:25 <LiangFang> my turn now?:)
14:42:28 <rosmaita> this is also on my list to test out
14:43:40 <rosmaita> sorry LiangFang, the floor is yours
14:43:49 <LiangFang> OK..
14:43:51 <LiangFang> sorry
14:44:00 <rosmaita> no my fault, i interrupted
14:44:54 <LiangFang> my English is not good, now is my topic?
14:45:01 <jokke_> LiangFang: yes
14:45:08 <LiangFang> OK..
14:45:16 <jokke_> that's what we are talking about
14:45:33 <LiangFang> sorry today I joined late
14:45:43 <LiangFang> I have implemented the code
14:46:07 <rosmaita> #link https://review.openstack.org/#/c/605014/
14:46:19 <LiangFang> someone gave comment to add UT, and I have added
14:46:53 <LiangFang> hope you can help to take a look about this
14:47:18 <LiangFang> another thing is that, I have about 10 patch to submit
14:47:38 <LiangFang> so in the following days, I will submit one by one
14:47:44 <LiangFang> sorry to bother you
14:48:06 <rosmaita> no bother, but remember that next week is the summit, so things may be a bit slow
14:48:30 <LiangFang> Ok
14:48:47 <LiangFang> some colleague will join summit there
14:49:15 <LiangFang> but the network team
14:49:24 <LiangFang> I'm the storage team
14:49:59 <rosmaita> so you are doing cinder and glance?
14:50:05 <LiangFang> yes
14:50:13 <LiangFang> starlingx glance and cinder
14:51:30 <rosmaita> ok
14:51:36 <LiangFang> so I also have lots of cinder patches to submit
14:52:41 <jokke_> ok lets continue on open discussion
14:52:47 <jokke_> #topic open discussion
14:52:56 <jokke_> Anything else we run out of time?
14:53:04 <jokke_> *before
14:53:31 <rosmaita> yeah, just want to mention that gorka has a cinder patch up about greenthread starvation caused by the image-signature-verification code in cinder's image utils
14:54:04 <rosmaita> i suspect we may be subject to a similar situation with the glance_store code
14:54:06 <jokke_> oh, how did that happen?
14:54:27 <LiangFang> glance support image cache, right?
14:54:41 <rosmaita> well, it depends on how much stuff you have running on the node
14:54:58 <rosmaita> cinder uses rabbitmq for the scheduler and volume service interactions
14:55:06 <jokke_> LiangFang: yes we have caching that can be enabled on the API nodes
14:55:27 <rosmaita> that's what was detecting starvation as service being down
14:55:50 <rosmaita> so we probably don't see that, but could see it in hyperconverged edge nodes
14:56:15 <rosmaita> found the link
14:56:23 <rosmaita> #link https://review.openstack.org/#/c/615934/
14:56:40 <jokke_> rosmaita: iirc our greenpool is by default 8000 per process ... I'd be surprised if that is the bottle neck
14:56:43 <rosmaita> gorka has an interesting comment in there about greenthread and native thread interactions
14:57:27 <rosmaita> yeah, i wonder why they're seeing this and we're not
14:57:35 <LiangFang> jokke_: thanks
14:59:29 <jokke_> rosmaita: likely because we do stream the data through all our validation rather than cache it and run it through the validators and what not
14:59:55 <rosmaita> yeah, that makes sense
15:01:42 <jokke_> like only place I really see us hitting heavily with these issue would be using fs store (which we fortunately do not support) with caching and loads of concurrent imports all going at the same time
15:02:35 <rosmaita> ok, well that's a relief
15:03:09 <jokke_> yeah, we rarely do blocking io
15:04:16 <jokke_> most of our stuff is throttled by line speeds and buffers rather than having myriads of blocking interrupts
15:04:34 <rosmaita> hooray for glance!
15:04:37 <jokke_> ok, sorry, we're out of time
15:04:48 <jokke_> #endmeeting