19:01:26 #startmeeting swift 19:01:27 Meeting started Wed Jan 9 19:01:26 2013 UTC. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:31 The meeting name has been set to 'swift' 19:01:54 hey there 19:01:58 swift meeting time 19:02:27 my goal for these meetings is to have a place for contributors to regularly talk about what's going on in the project 19:02:54 today's agenda is: outstanding patches, outstanding bugs, next release, and "other" 19:03:08 #topic outstanding patches 19:03:21 https://review.openstack.org/#/q/status:open+swift,n,z 19:03:41 the gerrit expiry script was fixed/enabled yesterday 19:03:51 so older patches have now fallen off 19:04:08 but there are a few I'd like to mention 19:04:21 redbo: (or other?) what's the status on the container quotas? 19:05:08 John, can we go one by one with the order of the list? 19:05:09 did it fall off? 19:05:22 ya, it fell off. and was work in progress, I think 19:05:41 tongli: I there are a few I want to highlight, but we can bring up others if needed 19:05:52 ok, 19:05:56 i have started working on a swift quota by account middleware 19:06:01 I just haven't swung back around to it yet. We still want it. 19:06:05 tongli: I'd rather do older first, or higher priority, before we can change topics if there's some newer patches that need to be discussed we can bring them up... but since it's the first week if we want to go through *all* of them? 19:06:42 that is fine. let's go. 19:07:03 redbo: ok. chmouel: is your's related to redbo's at all? 19:07:39 not at all was going to release first as external middleware and maybe propose it to swift after as this is something enovance may need 19:07:46 ah ok 19:08:07 ah. dfg. just who I was about to talk about ;-) 19:08:08 https://review.openstack.org/#/c/17878/ 19:08:39 will get some code to show probably next week if that goes up in my assigned prioriities :) 19:08:46 the bulk middleware. it has a -1 that needs addressed, but's it's a cool feature that a lot of people I mentioned it to liked 19:09:04 notmyname: yeah he's been working on it 19:09:09 notmyname: i'm about to upload a new patch with chuck's changes 19:09:16 great :-) 19:09:18 just finished up unit tests 19:09:26 cool 19:09:39 dfg: wow xml serialization too!? 19:09:56 clayg: out going, not incoming. 19:10:19 which may be lame but will wait for comments 19:10:21 as much as I don't like xml, I agree that it should be something we generate 19:10:22 on review 19:10:45 anything else on the bulk middleware? 19:11:09 ok 19:11:10 https://review.openstack.org/#/c/19296/ and https://review.openstack.org/#/c/19297/ were proposed today to add support for a separate replication network 19:11:22 yes 19:11:26 that's our stuff 19:11:47 whoa, thats new 19:11:50 one more patch is still in internal review and testing 19:11:50 I've been talking to mirantis about it a little over email. I haven't had a chance to look at them in detail, but I have a couple of high level questions 19:11:54 ok 19:12:05 dangit... what's up with tempest? :/ 19:12:22 creiht: cinder tests borked this morning or somthing 19:12:37 ogelbukh: first, does it need to be 2 dependent patches? does the first patch make sense without the 2nd? 19:12:57 notmyname: not cinder :) 19:13:06 haha 19:13:08 basically, they apply to different parts 19:13:13 jgriffith: oh, sorry. "people" are on top of it ;-) 19:13:25 I just submitted a tempest patch that should fix it 19:13:32 ok 19:13:32 first one to ring-builder and script, second to replicator processes 19:13:34 jgriffith: hi! 19:13:42 also I think sdague made a modificatio that should address as well 19:13:58 jgriffith: isn't this what reviews in gerrit are for? :) 19:14:00 and the last one to the replication-servers 19:14:10 ogelbukh: ok, that was my 2nd question 19:14:27 erm I mean notmyname -^ 19:14:32 ogelbukh: I was expecting to see the REPLICATE verb use the 2nd network 19:14:39 we didn't want to dump too many lines in one patch in the first place 19:14:49 yes, that's the last one] 19:14:52 ok 19:15:35 I'm quite interested on the RAX perspective for the 2nd replication network feature 19:16:03 it's essential for global clusters, but I think it's also quite beneficial for local clusters too 19:17:04 s/essential/nice to have/ ? 19:17:47 pandemicsyn: if you have a dedicated dark net WAN connection, it's probably not essential :-) 19:18:21 I think somehow a separate logical replication network makes actual network bandwidth increase. 19:18:52 But... I kid, I kid. Separate would be fine. Not sure if we'd use it, but it'd be fine and desirable in many cases. 19:19:47 we'll be looking at the patches at swiftstack, and I'd like some RAX people to look at it too, please. 19:20:15 the more eyes the better, so if others can look as well, that'd be swell :-) 19:20:40 speaking of global clusters... torgomatic has 2 patches up (in various states of review) https://review.openstack.org/#/c/18562/ and https://review.openstack.org/#/c/18263/ 19:20:55 So, uhm, why is the rep network essential, maybe that'd help us understand the urgency? 19:21:45 I already +2 one of those. With the dependency I was waiting to review the other. 19:21:54 gholt: in slow-wan-link land, it lets you use QoS to prioritize user traffic over replication traffic 19:21:55 because if you have a WAN connection between 2 clusters (in a global cluster scenario), you need to be able to separately control the replication traffic 19:21:55 same here 19:22:00 that's the use case I know of 19:22:07 creiht: gholt: ya, thansk :-) 19:22:22 just waiting on torgomatic to fix his stuff :) 19:22:29 I'm going to fix up https://review.openstack.org/18263 soonish. I've been pretty much useless this week, though. stupid cold. 19:22:59 Okay, I'm not sure why qos doesn't work with different logical networks, but I don't have to be sure. :) 19:23:12 <-- not a network guy 19:23:52 i think qos just advise the app to go slower it doesn't force it 19:23:59 but i am not a network guy either 19:24:03 Hehe 19:24:09 heh 19:24:10 if it is a wan, how do you have two "differnet" networks? 19:24:11 Those are all the outstanding patches I wanted to specifically bring up. any other patches to talk about before we move on to bugs? 19:24:16 * creiht also isn't a network guy 19:24:17 :) 19:24:42 https://review.openstack.org/#/c/15818/ 19:24:50 john, this one, I have a quick question. 19:25:06 ok 19:25:14 when number of segements for a file is greater than CONTAINER_LISTING_LIMIT. 19:25:32 if the requested range is not within the limit, 19:26:18 should we ignore the range or return error especially, the request contains ranges that can be met and the ranges not in the limit. 19:27:21 my first guess would be to return an error. torgomatic or gholt? 19:27:54 this is about getting the ranges of an object. 19:29:00 well, returning an error sounds reasonable to me right now, but i haven't dug through rfc2616 to see if that's allowed 19:29:08 ya, that ^ 19:29:32 isn't range requests sort of advisory to begin with? I thought you could ignore and return the whole thing if you wanted... 19:29:33 ok 19:29:38 maybe there's a 400 for that... 19:29:46 416. 19:30:07 any other patches to discuss? 19:30:39 I'm not sure why we need testr instead of nose, but I don't know if mordred is around to answer that 19:30:53 perhaps we'll bug him in the review 19:30:56 well, how much do you want me to ramble... 19:31:42 mordred: a more detailed commit message would be nice 19:32:02 notmyname: tl;dr - nose is invasive and the source of various build failures that are crazy 19:32:22 notmyname: also, testr does parallel testing 19:32:43 ya, I just read that after clicking through enough links 19:32:49 notmyname: also, we're moving everyone else to it :) - but there is no known reason yet to cause you to move to it against your will 19:33:03 notmyname: so - more verbose commit message then? :) 19:33:14 mordred: when have you known swift to do something "because everyone else is doing it"? ;-) 19:33:24 never 19:33:30 mordred: ya, please 19:33:35 I only ever expect you guys to do things because they are awesomer 19:33:39 its also worth pointing out that in the process of converting nova a lot of broken tests were fixed (it is harder to get away with broken tests) 19:33:54 Do we lose anything by switching? the code coverage report? 19:34:11 swifterdarrell: code coverage can still be generated 19:34:15 nope. code coverage stays. you lose the colorized test output is the only thing 19:34:16 yeah I looked at it a bit, and it seems reasonable 19:34:30 so if you like watching colored test scroll by, we don't have a great answer for you 19:34:40 sounds good but there is some red error when i try your review 19:34:43 cool; I could care less about colors, but the coverage report is quite useful 19:34:50 mordred: I was waiting for adding the coverage back to take a look at it again (for swift client 19:34:57 but - testr itself has _way_ better tooling - like 'testr run --failing' which re-runs the tests that failed last time 19:35:37 creiht: swiftclient doesn't have a .coverage - altohugh I would be happy to add one 19:35:54 chmouel: eek! I'll poke you about that and see if I can track it down 19:36:01 mordred: when I run the old .unittests, I see a coverage report (at least I thought I did) 19:36:12 * mordred is actually very interested to see what happens on an attempt to move swift itself 19:36:27 creiht: cool. I'll double-check - (I think I might have re-added that since we talked) 19:36:54 anything else on patches, or can we move on to bugs? 19:36:58 mordred: thanks 19:37:10 notmyname: my pleasure! 19:37:25 moving on to bugs... 19:37:35 #topic outstanding bugs 19:37:57 please take a look at https://bugs.launchpad.net/swift/+bugs?field.status=NEW&field.importance=UNDECIDED and move them out of "undecided" as you can 19:38:05 ,as approriate 19:38:21 I've got 3 bugs I want to bring up 19:38:27 first, https://bugs.launchpad.net/swift/+bug/1084762 19:38:30 Launchpad bug 1084762 in swift "error when writing to handoffs" [High,Confirmed] 19:38:54 can anyone look at that or work on it? seems that any 2 storage nodes down in a cluster could cause 500s to be returned to the client 19:39:06 I'll be happy to work on it, but I may not be able to for a few weeks 19:39:49 I suspect RAX has seen it in their logs, but I'd like confirmation on that 19:40:06 notmyname: I think if we would have seen that, it would be fixed already 19:40:12 been 19:40:33 ya, I'd think so, but I also know how many log message you see every second :-) 19:40:52 (it's somewhere above "a lot") 19:41:11 notmyname: your killing the nodes while the PUT is happening? 19:41:16 no 19:41:23 kill them first, then do the PUT 19:41:24 or just shutting them down before the proxy ever calls get_nodes? 19:41:30 well, confirmation that it's not a real thing would be good too 19:41:33 ya 19:41:48 this is on my lucid SAIO 19:42:22 notmyname: wish I remembered where I originally saw it, but it wasn't lucid 19:42:24 sounds pretty crazy... 19:42:42 anyway, an answer is something we can't get right now. but I'd love some confirmation beyond swifterdarrell and me 19:43:01 maybe on the saio it can't find two hand offs? 19:43:22 4 servers, so it should still be able to get a quorum 19:44:12 next bug is https://bugs.launchpad.net/swift/+bug/1082835. <-- this needs an answer. is the change ok or do we need to revert the functionality (manifests where the references segments are able to be listed) 19:44:14 Launchpad bug 1082835 in swift "World readable access to segmented object produces 401, even if _segments is also world readable." [High,Confirmed] 19:44:26 yeah it just looks like some eventlety barf, and RAX prod is not likely to run into "I can't come up with three nodes" 19:46:11 any comments on the .rlistings issue? 19:46:24 well, it's certainly counterintuitive 19:47:05 ya, but isn't the side effect that you can probe an account if you have write access to a manifest object? 19:47:13 is it because the proxy's listing gets rejected as anon? 19:47:13 seems to me that if you can read a manifest file and read each individual segment that you ought to be able to GET the whole object 19:48:02 notmyname: yes, there is the downside that if I can write to one of your containers, I can use manifests to produce a listing of another container that I oughtn't be able to 19:48:53 either the docs need to make this explicit (the current reported behavior) or the behavior needs to be reverted 19:49:17 we could just update a doc somewhere to point out that the use-case of anon access to segmented objects requires .rlistings on the segment container 19:49:34 torgomatic: could you acctually get the listing tho if you don't read access to those objects behind the listing? the *proxy* could get the listing, but could the client on the other side of the manifest get? 19:50:49 I'd have to check again, but does tempauth have listings as a superset of reads? or is it independent? 19:51:14 IOW, if you can do the listing then you may already have read access 19:51:34 ok, in the interest of time, I need to move on to the last item 19:51:40 #topic next release 19:51:44 clayg: it's possible; you mutate the X-Object-Manifest header and perform repeated GET requests and see if you get back an etag of an empty object or not 19:51:59 * torgomatic is just a little too slow :) 19:52:15 gholt: pandemicsyn: creiht: did you start QA yesterday on master? 19:52:56 Sorry, was afc. Uhm, no, been really busy with other stuff. :/ 19:53:03 s/yesterday/this week/ 19:53:06 ok 19:53:16 any idea when that will be able to start? 19:53:24 I'm hoping to start the packaging today, but I said that yesterday. 19:53:31 heh ok :-) 19:53:45 I need to bug our torch too, and he hasn't been onsite. 19:54:13 Anyhow, it's definitely my #2 priority and I'll ping you when it's going for real. 19:54:23 thanks 19:54:40 if that does get started this week, I think that means we can do a public release the week of Jan 21 19:54:41 Also, feel free to ping me about it daily if I haven't. :) 19:55:02 ok :-) 19:55:44 any objections to a public release (swift 1.7.6) around thursday january 24? 19:57:08 any interesting topics for coming up OS summit on Swift? 19:57:31 our next swift meeting is the 23rd, so we can have a final go/nogo then. If QA testing starts this week, I'll set the tentative release date for the 24th 19:57:47 tongli: always. but I don't knwo what they are yet :-) 19:58:01 #topic other 19:58:16 any other topics to bring up (in our last 2 minutes)? 19:58:57 v3 keystone auth? 19:59:14 clayg: seems that the mailing list thread is covering most of it 19:59:21 heh 19:59:28 I don't think there is anything affecting swift from what I've seen 19:59:50 to quote redbo: "if their version bump requires swift storage to change, they're doing it wrong" ;-) 19:59:52 i don't think either, will give a try 19:59:56 right, all these things can be done in middleware. 20:00:11 thanks for coming today. next meeting is the 23rd (two weeks from now). 20:00:15 #endmeeting