14:00:06 <rosmaita> #startmeeting glance
14:00:06 <openstack> Meeting started Thu Jan 11 14:00:06 2018 UTC and is due to finish in 60 minutes.  The chair is rosmaita. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:10 <openstack> The meeting name has been set to 'glance'
14:00:19 <rosmaita> #topic roll call
14:00:36 <rosmaita> hello everyone
14:00:52 <McClymontS> GM
14:00:53 <rosmaita> erno is on holiday, not sure if he'll peek in today
14:01:06 <rosmaita> hi scott
14:01:07 <jokke_> o/
14:01:11 <rosmaita> erno!
14:01:21 <jokke_> semi afk ... on dinner table atm
14:01:37 <abhishekk> o/
14:01:45 <rosmaita> ok, careful with your keyboard!
14:01:51 <rosmaita> hi abhishekk
14:01:58 <abhishekk> rosmaita, hi
14:02:14 <rosmaita> light agenda today, and i need to sign off early, so here goes
14:02:22 <rosmaita> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:02:28 <rosmaita> #topic updates
14:02:49 <rosmaita> first, a reminder that voting for Foundation Board of Directors closes at 1700 UTC tomorrow
14:03:01 <rosmaita> hopefully you have received an email
14:03:09 <rosmaita> but don't forget to vote
14:03:40 <rosmaita> smcguiness and cdent are running to get some developer voices on the board
14:03:49 <rosmaita> but all candidates have a Q&A you can read through
14:04:07 <rosmaita> second, next week: final release for non-client libraries (so for us, glance_store)
14:04:21 <rosmaita> so any glance_store outstanding issues are a priority this week
14:04:33 <rosmaita> (not sure there are any though)
14:04:46 <rosmaita> i will check before sending out the weekly email
14:04:59 <rosmaita> finally, two items about the week-after-next
14:05:12 <rosmaita> Q-3 and feature freeze is the week of Jan 22
14:05:16 <rosmaita> so coming up fast
14:05:33 <rosmaita> also, the PTL self-nomination period is the week of Jan 22
14:05:47 <rosmaita> i have talked to my management and have support to serve again
14:05:53 <rosmaita> and i'm willing to do it
14:05:58 <McClymontS> +1
14:06:04 <rosmaita> but i'm also willing to be talked out of it
14:06:23 <rosmaita> in any case, if you are interested, i'll be happy to discuss it with you and let you know what you're in for
14:06:40 <rosmaita> also, flavio wrote a blog post 2 or so years ago that has some really good info
14:07:25 <rosmaita> it is a good time to think about being PTL even if you don't want to do it for Rocky, i can help you get ready for S
14:07:35 <rosmaita> (which i think doesn't have a name yet)
14:07:52 <rosmaita> anyway, that's all i've got for updates
14:08:04 <rosmaita> #topic gate situation
14:08:19 <rosmaita> you may have noticed that it is very slow
14:08:29 <McClymontS> Definitely seeing some legitimate failures but also some non-legitimate ones rosmaita
14:08:41 <abhishekk> yes, very slow sometimes its timed out
14:08:59 <rosmaita> there have been a concatenation of circumstances, an image without pip, some zuul3 crashes, meltdown patching
14:09:03 <rosmaita> all happening at once
14:09:27 <rosmaita> i didn't see anything on the ML, so i don't have a lot of info
14:09:45 <McClymontS> Sadly its pretty obstructive right now
14:09:49 <rosmaita> but there was a lot of activity in #openstack-infra-incident yesterday
14:09:56 <rosmaita> McClymontS: you said it
14:10:22 <McClymontS> really trying to understand the blockers on the final steps of my patch and I have gotten a few timeouts, malloc type errors, and a few legitimate failures
14:10:40 <rosmaita> yeah, it's a bummer
14:10:52 <rosmaita> and not likely to get better as the load increases toward RC-time
14:11:27 <McClymontS> so whats the backup here
14:12:07 <rosmaita> one thing for abhishekk and jokke_ : sometimes when they do a zuul restart, approvals don't get picked up
14:12:28 <abhishekk> need to +A again
14:12:37 <rosmaita> right, just something to be aware of
14:12:55 <abhishekk> I will keep a eye on zuul queue
14:13:03 <jokke_> ok
14:13:14 <rosmaita> McClymontS i really don't know what the backup is, but several jobs i'm interested in have been stuck in queued for hours
14:13:34 <rosmaita> i'm not sure whether there's a way to un-queue and re-queue?
14:13:56 <abhishekk> i guess there is not
14:15:20 <rosmaita> i just asked that in #openstack-infra , though looks like they have other stuff happening
14:17:15 <rosmaita> ok, infra says if you have a stuck job, you can ask in #openstack-infra for someone to take a look ... but they also said there's a pretty big backlog from the past 2 days
14:17:30 <abhishekk> ohh
14:18:14 <rosmaita> ok, that's it from me
14:18:22 <rosmaita> #topic open discussion
14:18:30 <abhishekk> i have one
14:18:42 <rosmaita> ok, go ahead
14:18:47 <abhishekk> glanceclient: limit_param_default will never be used if set in glance-api.conf
14:19:07 <abhishekk> The reason limit_param_default is not used is that from glanceclient default limit as 20 is passed to the server side where as at server side if limit is None then only limit_param_default is used. So if glanceclient is used then limit_param_default will never be used where as if someone uses curl then only it will be used.
14:19:49 <abhishekk> is this expected behavior, or its a bug
14:19:51 <rosmaita> so it actually is used, just not all the time?
14:20:10 <jokke_> well it will e used in any case where client does not send the limit or client sends bigger limit (IIRC)
14:20:30 <abhishekk> client sends the limit all the time
14:20:42 <rosmaita> isnt there also a max_param_limit or something? or is that the one we are talking about?
14:20:49 <jokke_> abhishekk: python-glanceclient does ... not all clients necessarily
14:21:02 <rosmaita> or am i thinking about page size?
14:21:03 <abhishekk> jokke_: right
14:21:20 <rosmaita> abhishekk: what is this a limit on?
14:21:45 <abhishekk> rosmaita: listing the image per request.
14:21:59 <jokke_> yeah so it's the image list page size
14:22:25 <jokke_> which gets funny when you list like 50k images :P
14:22:25 <rosmaita> so as long as if the api says 10, and that's what the client gets, even if it asks for 20, that seems OK to me
14:22:27 <abhishekk> so if you have say 2000 images and if you use glance_client then internally it makes 100 requests
14:23:11 <rosmaita> ok, now i see what you are getting at
14:23:23 <rosmaita> that seems something that could be improved in the client
14:23:31 <rosmaita> but not necessarily a bug
14:23:31 <jokke_> IIRC you can ask glanceclient to use bigger page size and it will unless the server limit is something tiny as well
14:24:05 <abhishekk> yup,
14:24:09 <rosmaita> abhishekk are you seeing problems with this?
14:24:48 <rosmaita> this is another discovery problem, no way for the client to know what the max page size it can request is
14:24:57 <abhishekk> I guess if I made lots of list requests then it will consume my memory?
14:25:47 <rosmaita> yeah, if you ask for 10K results and they're atually there, the client will make a series of requests and try to build the response until it OOM
14:25:53 <jokke_> actually it does not make those requests parallel so the larger your page size the more memory utilization you have as you have bigger pages in the memory
14:26:31 <rosmaita> anyway, doesn't seem to be a bug
14:26:38 <jokke_> rosmaita: at least the CLI does print out per page, not build it up and dump all at once
14:27:03 <abhishekk> ok, I was just curious about the behavior
14:27:07 <rosmaita> jokke_ cool, then the 20 limit is probably on the assumption of a thin client
14:27:20 <jokke_> this very thing was one of the major performance hits when we moved to v2 API
14:27:39 <jokke_> as the client did schema verification on every page it received
14:28:02 <jokke_> and we were paging a lot when there was bit more images
14:28:49 <rosmaita> that was fixed iirc
14:28:55 <jokke_> thus it does check only the first page nowadays ;)
14:28:56 <jokke_> yup
14:29:14 <jokke_> It was Fei Long who found and fixed it at the time
14:29:23 <rosmaita> hooray for Fei Long!
14:29:32 <rosmaita> ok, anything else?
14:29:35 <jokke_> I just remember that behaviour very well as we were banging head against the wall al lot
14:30:21 <abhishekk> nope, sounds good
14:30:23 <rosmaita> i think the schema-validation-for-responses behavior is from the days when we thought other clouds would provide the openstack APIs
14:30:34 <rosmaita> not aware that ever happened, though
14:31:10 <rosmaita> ok, i have to sign off early today
14:31:13 <jokke_> anyone on track what's the state of glance store?
14:31:29 <rosmaita> i
14:31:41 <rosmaita> i'll review later, but i think it's OK
14:32:00 <jokke_> ok cool ... I think that's gonna freeze soon for the release
14:32:14 <rosmaita> yes, i'll send out an email about that
14:32:28 <rosmaita> i may need to do some monkeying with the gate jobs (hope not)
14:32:35 <abhishekk> Just to update, I got the Ireland Visa so I will be attending this PTG
14:32:41 <rosmaita> hooray!
14:32:49 <jokke_> abhishekk: \\o \o/ o//
14:32:56 <abhishekk> :D \o/
14:33:10 <jokke_> I will be there too :P
14:33:16 <rosmaita> McClymontS are you going to be able to attend?
14:33:25 <abhishekk> looking forward to see you all :D
14:33:41 <abhishekk> jokke_: you will reach first I guess :P
14:33:55 <rosmaita> no TZ problems for jokke_ this time!
14:34:00 <McClymontS> I am not going to be there rosmaita
14:34:05 <rosmaita> bummer
14:34:10 <jokke_> darn
14:34:21 <rosmaita> ok, feel free to continue discussion in the glance channel
14:34:23 <McClymontS> yeah unfortunately a mix of things just got in the way
14:34:28 <McClymontS> awesome thanks guys
14:34:28 <rosmaita> #endmeeting