13:59:34 <nikhil_k> #startmeeting Glance
13:59:35 <openstack> Meeting started Thu Apr 23 13:59:34 2015 UTC and is due to finish in 60 minutes.  The chair is nikhil_k. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:59:38 <openstack> The meeting name has been set to 'glance'
13:59:40 <kragniz> o/
13:59:43 <mfedosin> o/
13:59:43 <ativelkov> o/
13:59:46 <krykowski> o/
13:59:47 <ivasilevskaya> o/
13:59:48 <bpoulos> o/
13:59:50 <lakshmiS> o/
13:59:51 <sigmavirus24_awa> o/
13:59:52 <pkoniszewski> o/
14:00:16 <nikhil_k> welcome all :)
14:00:28 <nikhil_k> Short agenda #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:00:37 <stevelle> o/
14:00:50 <nikhil_k> Let's get started
14:00:53 <nikhil_k> #topic RC2
14:01:28 <nikhil_k> Seems like we are all set for now. Blockers were/are listed at #link https://trello.com/c/CoZ9g13d/45-kilo-rc2
14:01:52 <nikhil_k> Reviews were given and RC2 is about to be cut
14:02:11 <nikhil_k> Any last minute red flags?
14:02:25 <jokke_> o/
14:02:46 <lakshmiS> https://review.openstack.org/#/c/169813/ and https://review.openstack.org/171022 are still failing on gate. recheck going on...
14:02:54 <nikhil_k> thanks
14:03:13 <nikhil_k> We would basically use the older reviews and ninja approve if needed
14:03:39 <nikhil_k> trying to avoid any delays. Some of them need to be backported
14:05:29 <nikhil_k> yosamite just crashed
14:06:05 <nikhil_k> Backports would be done  by rel-mgrs so no need to worry about them for now
14:06:25 <nikhil_k> Sometimes if there are bunch of conflicts then it is recommended for the authors to do them
14:06:43 <jokke_> backport for what?
14:07:03 <nikhil_k> Those reviews that are target for RC2
14:07:12 <nikhil_k> and currently proposed against master
14:07:33 <jokke_> ah you mean the picks for stable/kilo branch, thanks got it
14:07:48 <nikhil_k> yeah
14:07:50 <nikhil_k> This one was brought to my attn a couple days back #link https://review.openstack.org/#/c/176411/
14:08:03 <nikhil_k> It seems unlikely to make it
14:08:20 <nikhil_k> So, let's roll that over to Liberty if there are no objections
14:09:02 <nikhil_k> Moving on..
14:09:25 <nikhil_k> #topic Gate failure blocking reviews
14:09:38 <nikhil_k> #link https://bugs.launchpad.net/glance/+bug/1447463
14:09:38 <openstack> Launchpad bug 1447463 in Glance "glance.tests.functional.v2.test_images.TestImages.test_download_random_access failed" [Critical,New]
14:09:55 <pkoniszewski> so we have small problem there, everything will keep failing
14:10:16 <pkoniszewski> I'm working with krykowski but we haven't found where the problem is yet
14:10:50 <jokke_> pkoniszewski: Just kind reminder ... if you bump bug criticality, please make sure that you set it at least confirmed as well then
14:11:11 <nikhil_k> Cool, let's keep syncing on #openstack-glance to speden up unblocking
14:11:18 <pkoniszewski> jokke_: i just wanted to bring more attention, everything with +2 is blocked and RC2 is near
14:11:24 <mclaren> I'm happy to take a look aswell
14:11:31 <pkoniszewski> thanks for any help!
14:11:48 <mclaren> (although stuck in a few meetings this afternoon :-/)
14:11:51 <jokke_> pkoniszewski: that's fine, personally I just think we do not have critical bug if no-one can verify that the fault exists
14:12:17 <jokke_> so the person marking the priority should at least verify that the bug is valid and indeed causing the issue
14:12:18 <mfedosin> I'm going to look there too
14:12:40 <jokke_> pkoniszewski: not for you only, that's for everyone
14:12:52 <nikhil_k> yeah, it would be okay to mark it confirmed by the reporter if the verification is done on the automation system like jenkins. So, please paste a link to the review and the console logs where you see the failure and mark it confirmed
14:13:36 <nikhil_k> Ok, moving on..
14:14:00 <nikhil_k> #optic py-client 0.17.1
14:14:17 <nikhil_k> #topic py-client 0.17.1
14:15:16 <nikhil_k> #link https://pypi.python.org/pypi/python-glanceclient/0.17.1
14:15:36 <mclaren> thanks Nikhil.
14:15:51 <nikhil_k> That was released yday to get it included in RC2. We are hoping that it does not break any other package and then include it
14:15:58 <mfedosin> Is going to be in global requirements?
14:16:04 <nikhil_k> ^
14:16:06 <mclaren> Should I mark my stable global-requirements patch not-WIP now?
14:16:12 <jokke_> mfedosin: nope
14:16:24 <jokke_> Doug -2'd it already
14:16:36 <nikhil_k> mclaren: I think that would be good to do it from your side (if not already)
14:16:45 <mfedosin> jokke_, oh... why?
14:16:56 <nikhil_k> The same reason as above
14:17:05 <nikhil_k> We are hoping that it does not break any other package and then include it
14:17:20 <nikhil_k> There have been some issues with libraries breaking packages recently
14:17:43 <jokke_> mfedosin: apparently the risk breaking something by touching the requirements at the moment is too high to wait that the projects gets it rolled in before cutting RC23
14:17:57 <jokke_> -21 ;)
14:18:07 <nikhil_k> and folks have worked hard to get things to a stable state. Once everything looks good Thiery is going to decide.
14:18:35 <nikhil_k> Any other questions?
14:18:36 <mclaren> So should I wait until Doug removes the -2 before doing anything?
14:19:07 <jokke_> mclaren: I think you can lift your wip so you won't be blocking it when the time comes to merge it
14:19:13 <nikhil_k> == jokke_
14:19:17 <mclaren> Ok, I'll do that.
14:19:35 <nikhil_k> Thanks.
14:19:35 <mclaren> Can you clarify when it's likely to merge? ie before kilo?
14:19:40 <nikhil_k> yes
14:19:42 <mclaren> or after?
14:19:47 <mclaren> before, ok
14:19:49 <mclaren> thanks
14:19:54 <nikhil_k> if things go all good, we will have it in RC2
14:20:06 <nikhil_k> We are working on probabilities though
14:20:20 <nikhil_k> and it looks ~1 atm
14:21:10 <nikhil_k> #topic Swift driver: 401 issue in ChunkReader
14:21:31 <mfedosin> Hi again, folks, I want to discuss with you a known issue with uploading big files (>5 Gb) to Swift.
14:21:49 <mfedosin> As you know glance_store driver has its own implementation of big files chunking.
14:22:01 <mfedosin> https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L503
14:22:27 <mfedosin> The problem is Keystone token may expire during the uploading and when we want to put another chunk to Swift we get 401 response.
14:22:41 <mfedosin> And it's a serious headache for our customers who can't store big images there.
14:23:10 <mfedosin> I'm not so familiar with Swift. So, may I ask you, what was the point of having our own implementation of Swift Large Objects and why we just can't use SLO directly?
14:23:38 <mclaren> Can I answer a question with a question?
14:23:58 <mclaren> Is there a reason SLO will help here?
14:24:43 <mfedosin> I think yes, as far as I understand SLO
14:24:49 <ativelkov> mclaren: well, in this case you will have a single call to swift
14:25:06 <mclaren> ah, I see
14:25:22 <ativelkov> the problem, as far as I can see, is the token expiring at some n-th iteration of the chunk reader
14:25:36 <mfedosin> and chunking will perform via Swift tools, not Glance's
14:25:45 <mclaren> So we use static large objects currently
14:26:07 <mclaren> oops, we use dymanic large objects currently
14:26:11 <jokke_> at least by documentation it's multiple calls to swoft
14:26:15 <jokke_> swift even
14:26:31 <ativelkov> we may either fix it by re-connecting to swift on every chunk (which is a bit ugly), or make a single call and let the swift do the cunking for us
14:26:33 <mclaren> is your proposal to switch to static large objects?
14:27:01 <mclaren> Actually, let me talk to some Swift folks so I can comment more intelligently on this :-)
14:27:07 <sigmavirus24_awa> Half-serious question: "Why not both?" (and allow operators to configure it as they need)
14:27:15 <sigmavirus24_awa> mclaren: can you loop me into thos?
14:27:18 <sigmavirus24_awa> *those discussions?
14:27:41 <jokke_> I also think some deployers might have disagreement on the swiftclient SLO functionality and it's bandwidth utilization:
14:27:44 <jokke_> This would split the large_file into 1G segments and begin uploading those segments in parallel. Once all the segments have been uploaded, swift will then create the manifest file so the segments can be downloaded as one.
14:27:47 <mclaren> sigmavirus24_awa: I can give you a summary (I'll be talking to the guy beside me :-))
14:27:58 <jokke_> ^^ that is from swift docs
14:28:12 <sigmavirus24_awa> mclaren: hah
14:28:44 <mclaren> I know the token expiration issue has been around for a while, so removing it would be more excellent
14:28:50 <mclaren> more/most
14:29:02 <jokke_> ++
14:29:16 <ativelkov> The token expiration will still remain an issue at v1
14:29:17 <mclaren> ativelkov: mfedosin  this is with the multi-tenant store?
14:29:24 <nikhil_k> followed by intelligent use of user token to save image in the DB ;)
14:29:52 <ativelkov> nikhil_k: yes, that's what I mean by v1: we still need the token to do the call to registry to change the state
14:29:53 <mfedosin> it doesn't matter, single and multi use the same 'add'
14:30:06 <ativelkov> mclaren: afaik that is single in our cases
14:30:19 <ativelkov> but multi-tenant will have the same issue
14:30:33 <mclaren> but the single-tenant store can just request a new token? (I'm missing something -- sorry)
14:30:54 <nikhil_k> he wants to avoid that retry
14:30:57 <mfedosin> mclaren, yes, in single case it's easy to fix
14:30:59 <nikhil_k> at specified intervals
14:30:59 <ativelkov> mclaren: yes, it can, but it does not do it, as it establishes a single connection for all the chunks and then reuses it
14:31:22 <mclaren> ok, thanks
14:31:30 <ativelkov> https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L490
14:31:57 <jokke_> that should be fairly easy to fix for single tenant then?
14:32:03 <ativelkov> here, it makes a connection and then uses it for all the chunks at line 526
14:32:12 <ativelkov> 536*
14:32:18 <ativelkov> yes, it should be
14:33:27 <ativelkov> for multi-tenant the problem is harder to fix, but we may try to refresh the token on every iteration
14:33:42 <mfedosin> But why we just can't send big object there and remove code in 'else' part? https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L503
14:34:19 <nikhil_k> ativelkov: mfedosin propose a spec?
14:35:02 <jokke_> mfedosin: so we ignore the swift object size limit and let swift reject them?
14:35:32 <mfedosin> swift won't reject them
14:35:45 <ativelkov> jokke_: why will it reject them if SLO is enabled?
14:36:02 <mfedosin> we uploaded files via swift client with 150G size and everything was fine
14:36:18 <jokke_> http://docs.openstack.org/developer/swift/overview_large_objects.html
14:36:44 <jokke_> it seems to be the client doing the splitting
14:36:44 <mclaren> ativelkov: 'we may try to refresh the token on every iteration' - interesting - how?
14:37:24 <nikhil_k> need some logic in glance for that
14:37:31 <nikhil_k> server
14:37:55 <mfedosin> mclaren, keystone supports this feature
14:37:57 <mclaren> are trusts helpful here in any way? https://wiki.openstack.org/wiki/Keystone/Trusts
14:38:20 <ativelkov> mclaren: keystone's authenticate API call may use existing (non-expired) token to issue a new one
14:38:31 <mfedosin> no, trust won't help, but refreshing the token is a good solution
14:38:49 <ativelkov> http://developer.openstack.org/api-ref-identity-v2.html#authenticate-v2.0
14:38:50 <mclaren> ativelkov: I didn't know that! Do both tokens continue to be valid?
14:39:26 <ativelkov> mclaren: that's the question. May be, may be not
14:39:46 <mclaren> Ok, now I need to talk to some keystone folks too :-)
14:41:19 <ativelkov> my keystone guys says it should bot expire
14:41:29 <ativelkov> should not
14:41:47 <mclaren> ok, that sounds very promising
14:41:55 <jokke_> hmm-m ... did I get this right ... we can request a new token that is valid over the current tokens validity with the curren token?
14:41:58 <ativelkov> mfedosin: trusts may help as well, but that's a bit an overkill
14:42:13 <nikhil_k> there's some overlap
14:42:21 <nikhil_k> I think  one hour
14:42:30 <dstanek> jokke_: i don't think that we can do that - i think you can only rescope a token within the same expires
14:42:56 <nikhil_k> Ok, we need to get moving here..
14:43:09 <mclaren> hmm, ok, I'll try to get  a second opinion from keystone
14:43:27 <nikhil_k> Let's create a spec/etherpad/session proposal or soemthing and let's discuss there.
14:43:42 <mfedosin> mclaren, thanks, it'll be good
14:43:42 <jokke_> dstanek: I would expect so ... if not the expiry is pretty much pointless ... but yeah I think the trust approach would be the correct one as it's designed for these use cases
14:44:23 <nikhil_k> #topic Open Discussion
14:44:50 <nikhil_k> Any thing that needs urgent attendion? (and was not on the agenda)
14:45:24 <jokke_> Again shout for people to put summit topics in!
14:45:47 <nikhil_k> If not, please take a look at the current proposals..
14:45:52 <nikhil_k> #link https://etherpad.openstack.org/p/liberty-glance-summit-topics
14:46:07 <mclaren> not urgent but couple of stable branch reviews  https://review.openstack.org/#/q/project:openstack/glance+branch:stable/kilo+status:open,n,z
14:47:19 <nikhil_k> thanks mclaren , those seem important
14:47:51 <nikhil_k> About the topics, I would encourage everyone to consider something they want discussed as the topic and that way we can prioritize the cycle better
14:48:31 <jokke_> I don't think this bug was in RC https://bugs.launchpad.net/glance/+bug/1445827
14:48:31 <openstack> Launchpad bug 1445827 in Glance "unit test failures: Glance insist on ordereddict" [High,In progress] - Assigned to Stuart McLaren (stuart-mclaren)
14:48:31 <mclaren> I think v2 nova support is probably important, but does it warrant a session?
14:49:16 <nikhil_k> It should not be the case that only a spec is proposed and there are ton of other things lined up, even for an important discussion.
14:49:37 <stevelle> mclaren: I hadn't expected it would need a session
14:49:42 <nikhil_k> mclaren: Nova usually have one session that deals with images modules
14:50:09 <nikhil_k> we can discuss it there if needed but I think we covered the base (pretty much) in the mid-cycle
14:50:17 <mclaren> sure
14:50:34 <nikhil_k> thanks jokke_
14:51:22 <nikhil_k> We shall plan to have a priority impromptu meeting sometime the week after next
14:51:45 <nikhil_k> and in the week prior to the summit, we should plan to have a video call
14:51:50 <nikhil_k> Details to be sent out soon
14:52:06 <nikhil_k> Once rc2 is cleared and things look stable)
14:53:36 <nikhil_k> So, again request for session proposals. Please add an entry for something that you need to be discussed and should be on the roadmap for Liberty
14:53:50 <nikhil_k> #info Please add an entry for something that you need to be discussed and should be on the roadmap for Liberty.
14:54:00 <nikhil_k> If nothing else ...
14:54:56 <nikhil_k> Thanks all!
14:55:00 <nikhil_k> #endmeeting