21:00:24 <notmyname> #startmeeting swift
21:00:25 <openstack> Meeting started Wed Jan 13 21:00:24 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:28 <openstack> The meeting name has been set to 'swift'
21:00:32 <notmyname> hello, everyone
21:00:37 <notmyname> who's here for the swift team meeting?
21:00:41 <mattoliverau> o/
21:00:41 <Zyric_> Hello
21:00:41 <acoles> here
21:00:43 <minwoob> o/
21:00:44 <ho_away> o/
21:00:45 <gmmaha> o/
21:00:47 <jlhinson> o/
21:00:47 <joeljwright> o/
21:00:48 <cschwede> o/
21:00:50 <kota_> hello
21:00:55 <m_kazuhiro> o/
21:01:29 <peterlisak> hello
21:02:08 <notmyname> good to see you
21:02:15 <notmyname> agenda for this week is
21:02:20 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:25 <torgomatic_> .
21:02:38 <notmyname> a few things to discuss, so let's get started
21:02:55 <jrichli> o/
21:02:58 <notmyname> #topic change schedule priority (patch 238799)
21:02:59 <patchbot> notmyname: https://review.openstack.org/#/c/238799/ - Change schedule priority of daemon/server in config
21:02:59 <cutforth> o/
21:03:09 <notmyname> peterlisak: this is your patch
21:03:25 <notmyname> nad it's been discussed in this meeting now for the past two weeks
21:03:38 <blmartin> o/
21:03:47 <notmyname> but there's a couple of things raised in the review comments
21:03:56 <notmyname> I've tried to summarize them on the agenda
21:04:19 <notmyname> first, does this actually do anything positive, or is it too blunt of an instrument for the stated goal?
21:04:30 <notmyname> Swift's processes are rather interconnected, so it's very difficult to isolate a logical API operation by process. eg nice'ing the container server would impact object PUTs since the container is updated as part of the object write.
21:04:44 <notmyname> second, assuming that yes this patch does improve things
21:04:50 <notmyname> Should we even be doing this in Swift itself instead of distro init scripts. On the one hand, the ionice priority "code" has to be rewritten for every distro. On the other, ops know ionice and how/when to use it and does that actually belong in Swift? If it does, why not other ops/management tools?
21:04:57 <notmyname> peterlisak: is that a fair summary?
21:05:03 <peterlisak> notmyname, yes
21:05:32 <notmyname> so, what do you (everyone) think? peterlisak, is there anything you've got to address these?
21:05:54 <peterlisak> firstly i think we should make some more tests if there is any improvements ... is there onovy?
21:06:58 <notmyname> I don't see him in here
21:07:26 <mattoliverau> donagh would be good too, but he isn't here either
21:07:47 <notmyname> should we table this for now then?
21:08:17 <mattoliverau> I like the idea, but i'm not an op.. and really question 1 needs to be answered before question 2
21:08:49 <notmyname> right. if it doesn't help, the answer to question 2 doesn't matter
21:09:07 <mattoliverau> right
21:09:17 <notmyname> peterlisak: since it seems there a few people I'd like to have involved who aren't here, let's table this until next week
21:09:31 <notmyname> peterlisak: can you work on getting some results from this change?
21:09:33 <peterlisak> ok, we will make more tests and then we move, right?
21:10:06 <notmyname> right. if you can show that yes it does actually help to adjust the ionice of swift processes, then we can get to answering the second objection
21:10:28 <notmyname> I'll make a comment on the code review reflecting that
21:10:58 <peterlisak> ok, thanks ... i understand it is useless without improvement proof
21:11:05 <notmyname> in the meanwhile, if anyone else is interested in this or think it's a really good idea, I'm sure peterlisak and onovy would appreciate other eyes on showing results
21:11:24 <notmyname> ok, next up..
21:11:25 <peterlisak> notmyname, yes
21:11:42 <notmyname> #topic patch 266599 object auditor patch
21:11:43 <patchbot> notmyname: https://review.openstack.org/#/c/266599/ - Make object-auditor storage-policy-aware
21:11:52 <notmyname> ok, so this one is frankly embarassing
21:11:58 <notmyname> timburke found it
21:12:00 <mattoliverau> if ops are already using nice/ionice it would be good to hear from them why, is it helping etc
21:12:04 <notmyname> mattoliverau: +1
21:12:41 <notmyname> it turns out that the object auditors just flat out don't work with EC policies (because they don't load the right diskfile)
21:12:47 <mattoliverau> yeah, opps. Thanks timburke for finding and pushing a patch
21:13:08 <notmyname> so, first, this is a critical fix that blocks a release
21:13:27 <notmyname> and second, we need it to land asap
21:13:30 <acoles> what is timburke *not* fixing? :)
21:13:47 <acoles> i can review it but not til tomorrow
21:13:51 <notmyname> (and figuring out how we managed to not do this or find it would be interesting)
21:13:57 <notmyname> acoles: thanks
21:13:59 <mattoliverau> that is true, he's a machine.. I don't know what swiftstack feeds there devs
21:14:15 <timburke> i'm a little concerned about how the object replicator also only uses a (replicated) DiskFileManager, but i'm not sure it's actually a problem
21:15:12 <acoles> timburke: the replicator shouldn't process EC objects, but maybe that needs checking over too
21:15:32 <mattoliverau> EC should use the reconstructor
21:15:42 <mattoliverau> not replicator
21:16:28 <notmyname> mattoliverau: that word "should" though ;-)
21:16:39 <notmyname> mattoliverau: also, the auditor should handle different policies ;-)
21:17:29 <notmyname> timburke: after this patch lands, it would be good to look at backporting the patch, at least to liberty (2.5.0)
21:17:44 <kota_> +1
21:17:53 <timburke> makes sense. just going down the list of places where i see DiskFileManager, not DiskFileRouter :)
21:18:05 <notmyname> I'm glad you are :-)
21:18:41 <notmyname> I've started on something else like that. I've got a local WIP that lets EC objects use fallocate. EC doesn't currently
21:19:09 <notmyname> timburke: anything else on this topic other than "go review, it's a big deal" ?
21:20:23 <notmyname> ok
21:20:33 <notmyname> #topic what's going on in swift?
21:21:07 <notmyname> in last week's meeting, I went on about getting a good picture of all the things going on
21:21:20 <notmyname> I've sent most of you, I think, an email asking a few questions
21:21:26 <notmyname> #link https://gist.github.com/notmyname/be49b04165928fd6662f
21:21:31 <notmyname> that's the 4 questions I sent out
21:21:48 <notmyname> and I've been getting great feedback!
21:22:15 <notmyname> I'll be collecting all the responses and putting those together for people
21:22:27 <notmyname> (anonymized. no names with anything)
21:22:58 <notmyname> if you haven't responded, or if you didn't get an email about this (sorry), then feel free to send me your thoughts
21:23:02 <notmyname> you can send it to me@not.mn
21:23:31 <notmyname> but I'd really love to hear what you're working on, what you think is important, and where we're going
21:24:40 <notmyname> I'd love to share some of the early responses (or themes from them), but I also don't want to influence what people say if they haven't responded yet ;-)
21:24:51 <torgomatic_> and what color we should paint the handbasket
21:25:16 <notmyname> torgomatic_: we've got very good intentions
21:25:24 <mattoliverau> notmyname: thanks for collecting and sorting all the data, it's a great idea. Getting the community's throughts
21:25:34 <notmyname> but here's one thing to be thinking about that more than a few people have responded with...
21:25:56 <gmmaha> notmyname: maybe a small end date for folks to send the info your way by
21:26:19 <notmyname> one of the biggest pain points (and one that isn't a surprise to anyone, I think), is that patches sit in gerrit waiting for reviews for a *long* time
21:26:27 <notmyname> gmmaha: good thinking
21:26:39 <notmyname> deadline for responses: this week. send them to me by friday
21:27:39 <notmyname> on the topic of slow reviews, what do you think after one week of having the "small things" section on the dashboard?
21:27:47 <notmyname> is it good? does it help you?
21:27:55 <notmyname> (have you actually looked at it to review stuff? ;-)
21:28:02 * acoles confesses to not upgrading to the new dashboard yet
21:28:21 <acoles> "every dashboard is a little better than the next" :)
21:28:29 <cschwede> used it this morning, quite useful and will probably help
21:28:30 <mattoliverau> lol
21:28:34 <notmyname> cschwede: cool
21:29:04 <ho_away> acoles: lol
21:29:18 <kota_> i picked up some patches from small things ;)
21:29:22 <mattoliverau> I've looked, I think its good. Sometimes reviews take quite a while, small one are _usually_ quicker so I think it's a good idea.
21:29:40 <acoles> seriously, i take note of the spark lines for size, but I am aware that I could fill a day reviewing small patches and thats another day that the big ones wait
21:29:40 <Zyric_> Ah is that why I don't see it? You need to select the upgrade for the dashboard? How do you do that?
21:30:00 <notmyname> right. I can think of a few small patches that would completely break everything
21:30:16 <notmyname> Zyric_: yeah, a dashboard is just a link. so "upgrading" is using a different link
21:30:49 <mattoliverau> or updating with the new link in you gerrit settings, if you've added a menu item for it
21:31:02 <notmyname> the current link of the latest review dashboard is in the channel topic
21:31:11 <notmyname> mattoliverau: oooh. how's that work?
21:31:12 <acoles> but on balance I think its useful to help speed some reviews through
21:32:07 <notmyname> mattoliverau: oh, you mean updating a bookmark?
21:32:41 <mattoliverau> notmyname: gerrit, settings > preferences then add the link #/dashboard..  to the my menu
21:32:42 <acoles> notmyname: i think you can change your default dashboard in gerrit settings
21:32:51 <mattoliverau> then it appears under the My menu
21:33:15 <acoles> mattoliverau: i should do that!
21:33:17 <timburke> ...and if you move it to the top of the list, it becomes your default dashboard
21:33:19 <notmyname> ah, cool
21:34:30 <notmyname> on the positive side, there's been a lot of great comments about the community. everyone seems to like working on swift and working with others who work on swift
21:34:50 <cschwede> yeah, that’s awesome!
21:35:05 <peterlisak> cool :)
21:35:17 <notmyname> please give me your feedback on those questions asap, and I'll collate everything share what I find
21:35:39 <notmyname> I've also got an idea for a cool visualization of it that i'm working on
21:35:52 <notmyname> acoles: so you'll have a different picture to roll your eyes about ;-)
21:36:02 <acoles> hehe
21:36:05 <onovy> sorry i'm late. ionice/nice discussion is over i think
21:36:09 <peterlisak> notmyname, where will it be published?
21:36:16 <acoles> notmyname: actually the words 'mind map' crossed my mind
21:36:16 <notmyname> peterlisak: of course :-)
21:36:33 <notmyname> peterlisak: oh. you said where, not will
21:36:52 <notmyname> peterlisak: yes, of course it will be published. probably informally in IRC rather than some blog post or something
21:37:27 <notmyname> onovy: no worries. you and peterlisak have a todo from it, and we'll address the questions at next week's meeting
21:37:41 <notmyname> #topic open discussion
21:37:43 <onovy> notmyname: just read backlog, ok, thanks
21:37:57 <notmyname> acoles: yeah, "mind map" is a good phrase. or zeitgeist
21:38:12 <notmyname> so open discussion
21:38:16 <notmyname> anythign else to bring up this week?
21:38:27 <hrou> Just wanted to give a really quick update as some folks have been asking;  For symlink, we're trying to get post-to-target working, and we're pretty happy with post-as-copy handling (well it sucks but its no worse then .. : - ) so we're not looking at fast-post.  More soon, just a heads up.
21:38:50 <hrou> *we're now looking at fast-post
21:39:05 <acoles> hrou: glad you clarified that :)
21:39:18 <notmyname> :-)
21:39:33 <hrou> acoles, lol sorry I figured you of all people wouldn't have liked how that was originally stated ;)
21:39:44 <onovy> if we have time: https://review.openstack.org/#/c/251151/ simple question: UTC or localtime? if UTC, it's done. if localtime, than patch1
21:40:46 <notmyname> in general, always UTC for anythign that matters. but translating that to localtime in CLI output or a visualization seems reasonable
21:40:55 <notmyname> (was that answer ambiguous enough?)
21:41:13 <onovy> :)
21:41:42 <onovy> for now its utc and locatime together which is wrong
21:41:54 <notmyname> and since operators may have (and in my experience often do) clusters in timezones that are different from themselves (and multiples in multiple TZs), IMO UTC.
21:42:21 <onovy> so results of swift-recon will be same on all servers, tz doesn't matter
21:42:24 <notmyname> eg if I'm in california and have a cluster in sydney and one in london, what's "localtime" on my dashboard?
21:42:27 <onovy> that's imho correct
21:42:34 <onovy> yep, that's good point
21:42:57 <onovy> so it's done i think. ready for review :)
21:43:02 <notmyname> and let's not even get into daylight saving time :-(
21:43:03 <Zyric_> It'd be awesome to get more feedback on https://review.openstack.org/#/c/212824/ and https://review.openstack.org/#/c/262087/ please, if you have time.
21:43:20 <notmyname> ah yes. the audit watchers
21:43:26 <mattoliverau> everything should be in Sydney time
21:43:35 <peterlisak> and if i can also ask you, i have two older patches waiting to review 253038 and 253037 ... thanks
21:44:01 <onovy> Zyric_: second one have jenkins -1
21:44:14 <notmyname> I'm planning to re-star priority reviews after we get the current ones landed for a release
21:44:56 <notmyname> anything else to discuss? or shall we end now?
21:45:00 <Zyric_> onovy: It's a new test and seems to work most of the time. I'm fixing and splitting it today.
21:45:01 <onovy> mattoliverau: why syndey? use stardate!
21:45:02 <acoles> reminder to those coming to hackathon, hotel group rate expires mext Monday, although we have asked for an extension
21:45:06 <timburke> i just want to say thanks to acoles and joeljwright for getting patch 226897 in; I rather like that swiftclient can retry both uploads and downloads now
21:45:06 <patchbot> timburke: https://review.openstack.org/#/c/226897/ - Retry file uploads via SwiftService (MERGED)
21:45:16 <notmyname> ah yes, that hackathon!
21:45:26 <mattoliverau> onovy: cause I'm lazy and Sydney is my time :P
21:45:35 <notmyname> link for registration for the hackathon is https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073
21:45:39 <notmyname> #link https://www.eventbrite.com/e/swift-hackathon-bristol-sponsored-by-hpe-tickets-19994495073
21:45:43 <onovy> mattoliverau: i know it :)
21:45:49 <acoles> timburke: the symmetry was quite satisfying
21:46:24 <joeljwright> timburke: I'm just sorry it took me so long to get back to that patch
21:46:37 <cschwede> that said, thanks acoles for organizing the hackathon and hotel reservation!
21:46:47 <jrichli> +1
21:46:54 <notmyname> yes!
21:47:03 <acoles> i have a couple of EC related patches that have been around a while, patch 181407 and patch 232684
21:47:03 <patchbot> acoles: https://review.openstack.org/#/c/181407/ - EC: Avoid conflicts when ssync'ing fragments
21:47:04 <patchbot> acoles: https://review.openstack.org/#/c/232684/ - Don't ssync data when only a durable is missing
21:47:05 <notmyname> I'm looking forward to it. and probably need to buy a heavier coat
21:47:35 <acoles> we have a great crowd signed up, I am looking forward to it
21:48:46 <notmyname> then if there's nothing else...
21:49:04 <notmyname> thanks for coming today. and thanks for working on swift
21:49:07 <notmyname> #endmeeting