21:00:17 <notmyname> #startmeeting swift
21:00:18 <acoles> here
21:00:18 <openstack> Meeting started Wed Nov 23 21:00:17 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:21 <openstack> The meeting name has been set to 'swift'
21:00:29 <notmyname> who's here for the swift meeting?
21:00:41 <notmyname> I know torgomatic is out today. as is clayg
21:00:42 <mattoliverau> o/
21:00:44 <pdardeau> hi
21:00:46 <mathiasb> o/
21:00:47 <m_kazuhiro> o/
21:00:49 <mmotiani1> Hi
21:00:51 <hosanai> o/
21:00:53 <tdasilva> hello
21:01:03 <cschwede> o/
21:01:07 <notmyname> hosanai: hello! I hope you weren't negatively affected by the quake this week
21:01:08 <nadeem> o/
21:01:14 <kota_> hi
21:01:31 <mattoliverau> There are more americans in attendance then I expected :)
21:01:41 <notmyname> mattoliverau: me too ;-)
21:02:28 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:33 <notmyname> ...is the agenda
21:02:38 <joeljwright> o/
21:02:42 <notmyname> just a few things
21:02:50 <hosanai> notmyname: thanks. my area doesn't close to the quake.
21:03:01 <notmyname> hosanai: oh good
21:03:19 <notmyname> a few FYIs that will affect us all
21:03:24 <notmyname> first up
21:03:31 <notmyname> #info swift 2.11.0 has been released
21:03:42 <notmyname> thank you everyone who helped get that pushed over the line
21:04:16 <notmyname> I've proposed both 2.10.1 and 2.7.1 for stable releases as well. the changelog patches are in gerrit, and they look mostly good. a few comments on one
21:04:40 <notmyname> I'd expect those to land next week, just because tomorrow and friday are US holidays
21:05:09 <notmyname> in other news, there was a significant change to the swift gate jobs this past weekend
21:05:26 <notmyname> onovy did a great job finding, diagnosing, and fixing some issues that came up
21:05:41 <mattoliverau> onovy: great work!
21:06:03 <notmyname> the summary is that linux has a 128 character limit on the shebang lines in scripts, and the job name + the tox environment name has busted through that
21:06:22 <mattoliverau> lol, so long
21:06:24 <notmyname> the change to tox.ini is to drop the "-in-process" substring from the two jobs that had it
21:06:46 <notmyname> I'm not super happy with having to do that, but I was less happy with dropping other parts of the job/env name
21:06:46 <kota_> oh
21:07:03 <notmyname> but, if we can figure out a better name, then we can change it
21:07:27 <notmyname> here is the patch that did it https://review.openstack.org/#/c/399887/
21:07:46 <notmyname> so why this weekend instead of earlier? that part is my fault
21:07:58 <notmyname> (but there's a silver lining too)
21:08:32 <notmyname> so I added "-xfs-tmpdir" to the job name to distinguish it from the normal "tox" jobs in the gate
21:08:40 <notmyname> that's the change that made it longer
21:08:54 <notmyname> however, the good news is that now we have xfs in the gate!
21:09:51 <notmyname> which means that when https://review.openstack.org/#/c/336323/ lands, we'll have full testing
21:10:13 <notmyname> speaking of that patch, again I want to call it out this week -- it will potentially break your dev environment
21:10:34 <notmyname> the change is that we stop mocking out xattr, so you have to run tests in an environment where large xattrs are supported
21:10:51 <notmyname> the patch now indicates that in docs, and it's ready for a full review
21:11:33 <notmyname> with that patch, if there isnt' an xfs tmpdir used, many of the tests will be skipped. it will be obvious
21:12:00 <notmyname> I'd suggest that even if you don't want to do a full review, at least download it and run it on your dev setup just to get ready
21:12:02 <mattoliverau> ahh skips over failure is good :)
21:12:07 <tdasilva> just over 1k tests will be skipped :)
21:12:16 <mattoliverau> wow
21:12:33 <notmyname> IIRC is's something like 1.3k skips of 4.3k unit tests. and something like 418 of 456 in-process functional tests
21:12:41 <notmyname> so it's obvious :-)
21:13:06 <notmyname> also, tdasilva keeps making merge conflicts for me with is refactoring of tests.py ;-)
21:13:12 <tdasilva> lol
21:13:44 <tdasilva> and i'm about to -1 that patch too ;)
21:13:49 <mattoliverau> lol
21:13:50 <notmyname> mine?
21:13:51 <tdasilva> happy thanksgiving
21:13:53 <notmyname> bah
21:13:55 <notmyname> :-)
21:14:25 <notmyname> joking aside, for everyone, it would be a good idea to prep now for the test environment change
21:14:45 <notmyname> either mount your /tmp as xfs or set up an XFS partition somewhere and set TMPDIR to it
21:14:48 <mattoliverau> everyone update your saio build scripts ;)
21:15:16 <notmyname> so, also, there's a couple of things that have happened in openstack because of these changes
21:16:02 <notmyname> first, fungi has proposed a TC resolution that explicitly says test prerequirements like this are perfectly ok. this is good, because previously it was a little bit of a gray area, and someone could have said "oh swift is being different".
21:16:10 <notmyname> I'm happy to see that governance change
21:16:36 <mattoliverau> +1. fungi's awesome
21:16:44 <notmyname> second, the -infra team is working on a longer term change that will make all the work I did in CI obsolete (which is good!)
21:17:21 <notmyname> they're proposing that projects can keep a test-setup.sh script in a special place in the repo that will be called with root or sudo perms before tests are run
21:18:10 <notmyname> in general that means that all the XFS setup I did in the CI system could be done in the future in a setup file in our repo
21:18:20 <notmyname> which would have made the whole process a lot simpler
21:18:21 <mattoliverau> cool, that'll make life easier
21:18:39 <notmyname> and from the infra side, it means that every different setup combination doesn't have to have a different job template now
21:18:49 <notmyname> "now" == in the future when this is implemented
21:18:56 <notmyname> so, all that's good to see
21:19:21 <notmyname> any questions on any of that?
21:19:39 <notmyname> it's a dump of info, but it's stuff that affects all of us
21:20:17 <acoles> notmyname: might the test-setup.sh script be passed the name of the test job so it can behave differently for various jobs?
21:20:43 <mattoliverau> thats a good idea
21:20:48 <acoles> IDK why yet, just wondering
21:21:01 <notmyname> that is a good idea. I'm looking for the ML link
21:21:26 <notmyname> #link http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
21:21:38 <notmyname> acoles: probably best to raise that in the ML thread
21:22:33 <timburke> alternatively, i wonder if they could pick out the env vars from the tox env, and set them before calling test-setup.sh...
21:22:34 <acoles> notmyname: k
21:23:27 <notmyname> yeah, I can imagine needing to do different swift test setup for the same reasons we have different tox environments now
21:23:56 <acoles> so basically, "this is a neat idea, but you know swift, we'll probably want to use it differently" :P
21:23:59 <notmyname> even if the job name is the only thing passed in, that's probably a good enough starting place for us to do the right thing in our own repo
21:24:24 <notmyname> lol, no. form the ML thread, it seems there's several different setups that are needed already today. we're not at all special here :-)
21:24:41 <notmyname> different DB setups, from what I can tell
21:25:14 <notmyname> anything else on these test-related things?
21:26:08 <notmyname> ok, last thing I had on the agenda was to get some info
21:26:16 <mattoliverau> not until I forget and wonder why an old SAIO is failing :P
21:26:22 <notmyname> heh
21:26:23 <notmyname> #topic current status of changing policies
21:26:51 <notmyname> over the past couple of years, changing policies/tiering/migrations/etc has come up several times, and there's been quite a bit of work on it
21:27:16 <notmyname> I was recently asked about it, and when we'll have it upstream (yay! predictions about "done"!)
21:27:25 <notmyname> so I wanted to make sure I had my head around all the pieces
21:27:54 <notmyname> so here's what I remember:
21:27:59 <notmyname> symlinks needed first
21:28:11 <notmyname> then changing policies api
21:28:22 <notmyname> then (maybe) something that does the policy change automatically
21:28:34 <notmyname> are those the same big parts you remember, or did I leave somethign out?
21:29:32 <tdasilva> i'm not sure the changing policies is a dependency, at least to what m_kazuhiro is doing
21:29:40 <notmyname> current symlink work is at https://review.openstack.org/#/c/232162/ (led by tdasilva). then there's the change policies patch at https://review.openstack.org/#/c/209329/ from daisuke
21:30:00 <notmyname> where's m_kazuhiro's work?
21:30:08 <tdasilva> one sec
21:30:25 <tdasilva> https://review.openstack.org/#/c/287057/
21:30:39 <m_kazuhiro> notmyname: automated-tiering is independent from changing policies.
21:30:43 <acoles> tdasilva: is the distinction between policy migration vs tiering the granularity? migration means entire policy contents, tiering is per object movement between policies?
21:31:12 <acoles> m_kazuhiro: ^^
21:31:36 <tdasilva> acoles: yeah, i think that sounds like a fair description (but my understanding of the changing policies work is limited)
21:32:23 <notmyname> m_kazuhiro: is that how you understand it, too?
21:32:40 <kota_> yeah, and AFAIK, only auto-tiering depends on symlink but changing-policies doesn't
21:32:47 <acoles> kota_: +1
21:32:51 <tdasilva> right
21:33:03 <m_kazuhiro> notmyname: yes
21:33:18 <notmyname> ok, got it
21:33:39 <notmyname> well, wait :-)
21:33:48 <notmyname> we used a few different terms to talk about the same thing
21:33:55 <tdasilva> lol
21:34:04 <notmyname> assert policy migration == changing policies
21:34:15 <tdasilva> True
21:34:23 <notmyname> assert tiering == auto tiering
21:34:33 <tdasilva> True
21:34:36 <notmyname> got it :-)
21:34:37 <tdasilva> in my head
21:34:44 <notmyname> thanks, tdasilva REPL
21:34:44 <tdasilva> acoles: ?
21:34:50 <acoles> also with policy-migration as I understand it the object urls remain unchanged, with tiering objects contents are moved to new (hidden) urls
21:35:03 <tdasilva> hopefully hiddne
21:35:14 <acoles> not hiddnee
21:35:26 <notmyname> which is why tiering needs symlinks. to keep the same URLs available
21:35:28 <acoles> notmyname: assertions correct IMO
21:36:10 <m_kazuhiro> notmyname: Yes
21:36:32 <acoles> notmyname: tiering needs symlinks because containers retain original policy, so new containers are need to move content to
21:37:06 <tdasilva> notmyname: just FYI....in barcelona we also talked about some "infra" work that is somewhat generic that would be needed by auto-tiering and possibly used by other components
21:37:14 <notmyname> so, hidden or not hidden URLs with tiering. I thought I remembered hidden URLs
21:37:19 <acoles> tdasilva: yes!
21:37:23 <tdasilva> I had a TODO to write those up, but haven't yet
21:37:24 <tdasilva> sorry acoles
21:37:47 <notmyname> tdasilva: ah, ok. I wasn't in the barca conversation, so I'm definitely interested in that
21:37:49 <acoles> notmyname: I would hope for hidden urls for tiered objects
21:38:17 <tdasilva> notmyname: exactly, we all hope for hidden urls, but that's one of these "infra" discussions we had about how to tdo hidden urls
21:38:18 <notmyname> right. new hidden urls in a differnet policy with a symlink in the original policy
21:38:40 <notmyname> yeah, I could imagine myself arguing for not hidden urls ;-)
21:38:50 <notmyname> (but now's not the time to hash that out)
21:38:58 <tdasilva> well, it was more like hidden containers, but you get the idea
21:39:05 <tdasilva> agree
21:39:14 <notmyname> ok, I got what I needed to know. thanks :-)
21:39:24 <notmyname> #topic open discussion
21:39:30 <notmyname> anything else to bring up this week?
21:39:38 <acoles> tdasilva: notmyname: we discussed 1. how do we test these features? 2. is there a common "engine" for performing tasks required to implement internal object movement? 3. generic approach to hidden urls (e.g. alt reseller prefix??)
21:40:00 <tdasilva> acoles: perfect
21:40:09 <acoles> maybe timur's crawler is a step towards a common engine, IDK, I haven't reviewd it yet :/
21:40:53 <mattoliverau> In sharding I have hidden containers, but its a , (dot) prefixed account that is releted to the users account
21:41:19 <mattoliverau> but it's just for container metadata
21:41:32 <tdasilva> mattoliverau: would love to hear more about that, because that's one of the ideas we had, but there are also downsides to it
21:41:33 <acoles> mattoliverau: right - another example - and I guess I wonder if future us will find it easier to grok swift if there are some generic mechanisms/patterns
21:41:51 <mattoliverau> +1
21:42:14 <mattoliverau> in my case I don't have to worry about objects being hidden and therefore not tracked
21:42:29 <mattoliverau> and I gather stats.. maybe a reseller prefix does make sense
21:43:41 <tdasilva> notmyname: have you heard any status from the tape library guys? we had good conversations in barcelona but haven't heard anything since...
21:43:46 <notmyname> I have not
21:44:01 <tdasilva> they were also interested in the auto-tiering work
21:44:04 <acoles> tdasilva: do you remember the downsides to . prefic accounts?
21:44:04 <tdasilva> ok
21:44:13 <mattoliverau> they have a weekly phone meeting, so there still working on it
21:44:14 <notmyname> they told me they had good conversations in barca with the optical storage people
21:44:28 <tdasilva> right, panasonic, right?
21:44:33 <notmyname> yes
21:44:36 * mattoliverau gets the invite but been to busy (work and baby to attend lately)
21:44:50 <hosanai> notmyname, tdasilva: regarding tape discussion, we will have a meeting in next week
21:44:53 <notmyname> mattoliverau: ah, cool
21:45:01 <notmyname> I'm glad to hear that
21:45:03 <tdasilva> acoles: i think it's what mattoliverau alluded, tracking the objects
21:45:04 <kota_> ping m_kazuhiro, he may know something on that
21:45:22 <tdasilva> hosanai: cool!
21:45:49 <notmyname> anything else to bring up?
21:45:58 <dmorita> Sorry I'm late
21:46:15 <dmorita> It seems there was discussion about policy changingm, right?
21:46:22 <tdasilva> acoles: i'll write those items up on a etherpad and we can continue the discussions there
21:46:29 <notmyname> dmorita: yeah, just an overview of what's going on
21:46:29 <acoles> tdasilva: thanks
21:46:37 <tdasilva> acoles: sorry for the delay
21:46:44 <dmorita> I think as kota_ said, policy changing does not have dependency with symlink.
21:46:57 <dmorita> I am working on how to work with fast-POST.
21:47:10 <notmyname> cool
21:47:16 <dmorita> Actually, it works well in my local, but need to add & fix some tests.
21:47:21 <acoles> tdasilva: NP :)
21:47:23 <dmorita> not fix tests.
21:47:34 <dmorita> fix my code to work with unit tests.
21:47:49 <acoles> dmorita: apologies, I owe you an email reply re fast-post
21:48:14 <dmorita> NP. Totally your first reply makes sense to me.
21:49:14 <dmorita> Anyway, after I fix something, I will update my patch in gerrit.
21:50:12 <notmyname> are we all good for this meeting today? anything else to bring up?
21:50:21 <tdasilva> since we talked so much about it, here goes a shameless plug: symlinks is ready for review ;)
21:50:33 <notmyname> tdasilva: :-)
21:50:42 <tdasilva> notmyname: any other reviews to highlight this week?
21:50:42 <notmyname> and look at all the work dependent on it!
21:50:51 <acoles> tdasilva: lol
21:51:25 <notmyname> I've been heads-down on the infra changes and the xattr stuff. I don't have anything else to highlight this week. except maybe the libec/pyeclib stuff to fully close the ec issue
21:51:43 <timburke> thanks everybody for looking at my slo patches lately!
21:51:58 <onovy> am i late, right?
21:52:11 <mattoliverau> onovy: yup, a little :P
21:52:13 <notmyname> onovy: yeah, just ending
21:52:14 <acoles> timburke: slo sysmeta merged right?
21:52:25 <onovy> do we have time for one question?
21:52:47 <kota_> hopefully, if someone would start my global ec cluster review :P
21:52:50 <timburke> acoles: yup. and the parallel verification
21:52:54 <notmyname> onovy: yes
21:53:10 <notmyname> kota_: yes! (...so much to do...)
21:53:14 <onovy> https://review.openstack.org/#/c/401330/
21:53:21 <onovy> i think we found that strange 'rsync metrics bug'
21:53:34 <onovy> and i need to know another one opinion about it
21:53:38 <tdasilva> kota_: patch?
21:53:52 <onovy> we=Pavel
21:54:05 <onovy> fix is really simple: https://review.openstack.org/#/c/401330/2/swift/obj/diskfile.py
21:54:07 <notmyname> onovy: ah, cool
21:54:27 <onovy> Pavel is going to stress test it in lab tomorrow
21:54:28 <kota_> tdasilva: https://review.openstack.org/#/c/219165/ here
21:54:44 <onovy> but i don't want to test it in production before core review :)
21:55:26 <onovy> i think it's relativly big regression. in some situation, replicator will not replicate changed partitions
21:56:08 <notmyname> onovy: yeah, makes sense. this is a US holiday week, so reviews are a little light this week
21:56:17 <onovy> notmyname: np
21:56:36 <onovy> just want to say: don't lost your data in 2.7.0+ swift plus, good luck :)
21:56:59 <notmyname> heh
21:57:17 <cschwede> onovy: your patch is on my agenda for tomorrow morning
21:57:27 <onovy> cschwede: nice! thanks a lot
21:57:33 <cschwede> sounds scary
21:57:34 <onovy> should Pavel go to IRC for talk?
21:57:40 <onovy> (tomorrow)
21:57:47 <cschwede> let me review+test first
21:57:50 <onovy> cschwede: ok
21:58:02 <onovy> 1/10 replication phase will force-recount hashes
21:58:08 <onovy> so this is fine
21:58:24 <onovy> so every 10th phase will sync cluster correctly
21:59:09 <notmyname> we're at time, so I'm going to close the meeting
21:59:13 <mattoliverau> onovy: I've got a pretty busy day, but will also try and take a look today.
21:59:17 <acoles> notmyname: thanks for your work on infra stuff
21:59:20 <onovy> mattoliverau: thanks too
21:59:23 <mattoliverau> +1
21:59:41 <notmyname> np. it's been a ...learning... experience :-)
21:59:50 <notmyname> thanks everyone for coming. thanks for working on swift
21:59:51 <acoles> great! :P
21:59:52 <onovy> for /me too :P
21:59:56 <notmyname> #endmeeting