21:00:24 #startmeeting swift 21:00:25 Meeting started Wed Jan 13 21:00:24 2016 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:28 The meeting name has been set to 'swift' 21:00:32 hello, everyone 21:00:37 who's here for the swift team meeting? 21:00:41 o/ 21:00:41 Hello 21:00:41 here 21:00:43 o/ 21:00:44 o/ 21:00:45 o/ 21:00:47 o/ 21:00:47 o/ 21:00:48 o/ 21:00:50 hello 21:00:55 o/ 21:01:29 hello 21:02:08 good to see you 21:02:15 agenda for this week is 21:02:20 #link https://wiki.openstack.org/wiki/Meetings/Swift 21:02:25 . 21:02:38 a few things to discuss, so let's get started 21:02:55 o/ 21:02:58 #topic change schedule priority (patch 238799) 21:02:59 notmyname: https://review.openstack.org/#/c/238799/ - Change schedule priority of daemon/server in config 21:02:59 o/ 21:03:09 peterlisak: this is your patch 21:03:25 nad it's been discussed in this meeting now for the past two weeks 21:03:38 o/ 21:03:47 but there's a couple of things raised in the review comments 21:03:56 I've tried to summarize them on the agenda 21:04:19 first, does this actually do anything positive, or is it too blunt of an instrument for the stated goal? 21:04:30 Swift's processes are rather interconnected, so it's very difficult to isolate a logical API operation by process. eg nice'ing the container server would impact object PUTs since the container is updated as part of the object write. 21:04:44 second, assuming that yes this patch does improve things 21:04:50 Should we even be doing this in Swift itself instead of distro init scripts. On the one hand, the ionice priority "code" has to be rewritten for every distro. On the other, ops know ionice and how/when to use it and does that actually belong in Swift? If it does, why not other ops/management tools? 21:04:57 peterlisak: is that a fair summary? 21:05:03 notmyname, yes 21:05:32 so, what do you (everyone) think? peterlisak, is there anything you've got to address these? 21:05:54 firstly i think we should make some more tests if there is any improvements ... is there onovy? 21:06:58 I don't see him in here 21:07:26 donagh would be good too, but he isn't here either 21:07:47 should we table this for now then? 21:08:17 I like the idea, but i'm not an op.. and really question 1 needs to be answered before question 2 21:08:49 right. if it doesn't help, the answer to question 2 doesn't matter 21:09:07 right 21:09:17 peterlisak: since it seems there a few people I'd like to have involved who aren't here, let's table this until next week 21:09:31 peterlisak: can you work on getting some results from this change? 21:09:33 ok, we will make more tests and then we move, right? 21:10:06 right. if you can show that yes it does actually help to adjust the ionice of swift processes, then we can get to answering the second objection 21:10:28 I'll make a comment on the code review reflecting that 21:10:58 ok, thanks ... i understand it is useless without improvement proof 21:11:05 in the meanwhile, if anyone else is interested in this or think it's a really good idea, I'm sure peterlisak and onovy would appreciate other eyes on showing results 21:11:24 ok, next up.. 21:11:25 notmyname, yes 21:11:42 #topic patch 266599 object auditor patch 21:11:43 notmyname: https://review.openstack.org/#/c/266599/ - Make object-auditor storage-policy-aware 21:11:52 ok, so this one is frankly embarassing 21:11:58 timburke found it 21:12:00 if ops are already using nice/ionice it would be good to hear from them why, is it helping etc 21:12:04 mattoliverau: +1 21:12:41 it turns out that the object auditors just flat out don't work with EC policies (because they don't load the right diskfile) 21:12:47 yeah, opps. Thanks timburke for finding and pushing a patch 21:13:08 so, first, this is a critical fix that blocks a release 21:13:27 and second, we need it to land asap 21:13:30 what is timburke *not* fixing? :) 21:13:47 i can review it but not til tomorrow 21:13:51 (and figuring out how we managed to not do this or find it would be interesting) 21:13:57 acoles: thanks 21:13:59 that is true, he's a machine.. I don't know what swiftstack feeds there devs 21:14:15 i'm a little concerned about how the object replicator also only uses a (replicated) DiskFileManager, but i'm not sure it's actually a problem 21:15:12 timburke: the replicator shouldn't process EC objects, but maybe that needs checking over too 21:15:32 EC should use the reconstructor 21:15:42 not replicator 21:16:28 mattoliverau: that word "should" though ;-) 21:16:39 mattoliverau: also, the auditor should handle different policies ;-) 21:17:29 timburke: after this patch lands, it would be good to look at backporting the patch, at least to liberty (2.5.0) 21:17:44 +1 21:17:53 makes sense. just going down the list of places where i see DiskFileManager, not DiskFileRouter :) 21:18:05 I'm glad you are :-) 21:18:41 I've started on something else like that. I've got a local WIP that lets EC objects use fallocate. EC doesn't currently 21:19:09 timburke: anything else on this topic other than "go review, it's a big deal" ? 21:20:23 ok 21:20:33 #topic what's going on in swift? 21:21:07 in last week's meeting, I went on about getting a good picture of all the things going on 21:21:20 I've sent most of you, I think, an email asking a few questions 21:21:26 #link https://gist.github.com/notmyname/be49b04165928fd6662f 21:21:31 that's the 4 questions I sent out 21:21:48 and I've been getting great feedback! 21:22:15 I'll be collecting all the responses and putting those together for people 21:22:27 (anonymized. no names with anything) 21:22:58 if you haven't responded, or if you didn't get an email about this (sorry), then feel free to send me your thoughts 21:23:02 you can send it to me@not.mn 21:23:31 but I'd really love to hear what you're working on, what you think is important, and where we're going 21:24:40 I'd love to share some of the early responses (or themes from them), but I also don't want to influence what people say if they haven't responded yet ;-) 21:24:51 and what color we should paint the handbasket 21:25:16 torgomatic_: we've got very good intentions 21:25:24 notmyname: thanks for collecting and sorting all the data, it's a great idea. Getting the community's throughts 21:25:34 but here's one thing to be thinking about that more than a few people have responded with... 21:25:56 notmyname: maybe a small end date for folks to send the info your way by 21:26:19 one of the biggest pain points (and one that isn't a surprise to anyone, I think), is that patches sit in gerrit waiting for reviews for a *long* time 21:26:27 gmmaha: good thinking 21:26:39 deadline for responses: this week. send them to me by friday 21:27:39 on the topic of slow reviews, what do you think after one week of having the "small things" section on the dashboard? 21:27:47 is it good? does it help you? 21:27:55 (have you actually looked at it to review stuff? ;-) 21:28:02 * acoles confesses to not upgrading to the new dashboard yet 21:28:21 "every dashboard is a little better than the next" :) 21:28:29 used it this morning, quite useful and will probably help 21:28:30 lol 21:28:34 cschwede: cool 21:29:04 acoles: lol 21:29:18 i picked up some patches from small things ;) 21:29:22 I've looked, I think its good. Sometimes reviews take quite a while, small one are _usually_ quicker so I think it's a good idea. 21:29:40 seriously, i take note of the spark lines for size, but I am aware that I could fill a day reviewing small patches and thats another day that the big ones wait 21:29:40 Ah is that why I don't see it? You need to select the upgrade for the dashboard? How do you do that? 21:30:00 right. I can think of a few small patches that would completely break everything 21:30:16 Zyric_: yeah, a dashboard is just a link. so "upgrading" is using a different link 21:30:49 or updating with the new link in you gerrit settings, if you've added a menu item for it 21:31:02 the current link of the latest review dashboard is in the channel topic 21:31:11 mattoliverau: oooh. how's that work? 21:31:12 but on balance I think its useful to help speed some reviews through 21:32:07 mattoliverau: oh, you mean updating a bookmark? 21:32:41 notmyname: gerrit, settings > preferences then add the link #/dashboard.. to the my menu 21:32:42 notmyname: i think you can change your default dashboard in gerrit settings 21:32:51 then it appears under the My menu 21:33:15 mattoliverau: i should do that! 21:33:17 ...and if you move it to the top of the list, it becomes your default dashboard 21:33:19 ah, cool 21:34:30 on the positive side, there's been a lot of great comments about the community. everyone seems to like working on swift and working with others who work on swift 21:34:50 yeah, that’s awesome! 21:35:05 cool :) 21:35:17 please give me your feedback on those questions asap, and I'll collate everything share what I find 21:35:39 I've also got an idea for a cool visualization of it that i'm working on 21:35:52 acoles: so you'll have a different picture to roll your eyes about ;-) 21:36:02 hehe 21:36:05 sorry i'm late. ionice/nice discussion is over i think 21:36:09 notmyname, where will it be published? 21:36:16 notmyname: actually the words 'mind map' crossed my mind 21:36:16 peterlisak: of course :-) 21:36:33 peterlisak: oh. you said where, not will 21:36:52 peterlisak: yes, of course it will be published. probably informally in IRC rather than some blog post or something 21:37:27 onovy: no worries. you and peterlisak have a todo from it, and we'll address the questions at next week's meeting 21:37:41 #topic open discussion 21:37:43 notmyname: just read backlog, ok, thanks 21:37:57 acoles: yeah, "mind map" is a good phrase. or zeitgeist 21:38:12 so open discussion 21:38:16 anythign else to bring up this week? 21:38:27 Just wanted to give a really quick update as some folks have been asking; For symlink, we're trying to get post-to-target working, and we're pretty happy with post-as-copy handling (well it sucks but its no worse then .. : - ) so we're not looking at fast-post. More soon, just a heads up. 21:38:50 *we're now looking at fast-post 21:39:05 hrou: glad you clarified that :) 21:39:18 :-) 21:39:33 acoles, lol sorry I figured you of all people wouldn't have liked how that was originally stated ;) 21:39:44 if we have time: https://review.openstack.org/#/c/251151/ simple question: UTC or localtime? if UTC, it's done. if localtime, than patch1 21:40:46 in general, always UTC for anythign that matters. but translating that to localtime in CLI output or a visualization seems reasonable 21:40:55 (was that answer ambiguous enough?) 21:41:13 :) 21:41:42 for now its utc and locatime together which is wrong 21:41:54 and since operators may have (and in my experience often do) clusters in timezones that are different from themselves (and multiples in multiple TZs), IMO UTC. 21:42:21 so results of swift-recon will be same on all servers, tz doesn't matter 21:42:24 eg if I'm in california and have a cluster in sydney and one in london, what's "localtime" on my dashboard? 21:42:27 that's imho correct 21:42:34 yep, that's good point 21:42:57 so it's done i think. ready for review :) 21:43:02 and let's not even get into daylight saving time :-( 21:43:03 It'd be awesome to get more feedback on https://review.openstack.org/#/c/212824/ and https://review.openstack.org/#/c/262087/ please, if you have time. 21:43:20 ah yes. the audit watchers 21:43:26 everything should be in Sydney time 21:43:35 and if i can also ask you, i have two older patches waiting to review 253038 and 253037 ... thanks 21:44:01 Zyric_: second one have jenkins -1 21:44:14 I'm planning to re-star priority reviews after we get the current ones landed for a release 21:44:56 anything else to discuss? or shall we end now? 21:45:00 onovy: It's a new test and seems to work most of the time. I'm fixing and splitting it today. 21:45:01 mattoliverau: why syndey? use stardate! 21:45:02 reminder to those coming to hackathon, hotel group rate expires mext Monday, although we have asked for an extension 21:45:06 i just want to say thanks to acoles and joeljwright for getting patch 226897 in; I rather like that swiftclient can retry both uploads and downloads now 21:45:06 timburke: https://review.openstack.org/#/c/226897/ - Retry file uploads via SwiftService (MERGED) 21:45:16 ah yes, that hackathon! 21:45:26 onovy: cause I'm lazy and Sydney is my time :P 21:45:35 link for registration for the hackathon is https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073 21:45:39 #link https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073 21:45:43 mattoliverau: i know it :) 21:45:49 timburke: the symmetry was quite satisfying 21:46:24 timburke: I'm just sorry it took me so long to get back to that patch 21:46:37 that said, thanks acoles for organizing the hackathon and hotel reservation! 21:46:47 +1 21:46:54 yes! 21:47:03 i have a couple of EC related patches that have been around a while, patch 181407 and patch 232684 21:47:03 acoles: https://review.openstack.org/#/c/181407/ - EC: Avoid conflicts when ssync'ing fragments 21:47:04 acoles: https://review.openstack.org/#/c/232684/ - Don't ssync data when only a durable is missing 21:47:05 I'm looking forward to it. and probably need to buy a heavier coat 21:47:35 we have a great crowd signed up, I am looking forward to it 21:48:46 then if there's nothing else... 21:49:04 thanks for coming today. and thanks for working on swift 21:49:07 #endmeeting