21:00:06 <notmyname> #startmeeting swift
21:00:06 <mmotiani1> Hi
21:00:09 <ntata> hello
21:00:11 <openstack> Meeting started Wed Nov 16 21:00:06 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:13 <kota_> o/
21:00:15 <openstack> The meeting name has been set to 'swift'
21:00:18 <notmyname> who's here fro the swift meeting?
21:00:19 <pdardeau> hi
21:00:19 <mattoliverau> o/
21:00:22 <sgundur_> hi
21:00:26 <hosanai> o/
21:00:26 <timburke> hi
21:00:28 <kota_> o/
21:00:31 <cutforth> o/
21:00:31 <dmorita> hi
21:00:33 <nadeem> o/
21:00:41 <tdasilva> hi
21:00:52 <notmyname> welcome everyone
21:01:24 <acoles> hi
21:01:25 <notmyname> like last week, not much on the agenda. but that's not a bad thing ;-)
21:01:26 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:01:32 <notmyname> #topic ptg
21:01:43 <notmyname> again, reminder to get your ticket for the ptg
21:01:47 <clayg> o/
21:01:52 <notmyname> and there were some questions that came up about hotels
21:02:12 <notmyname> there's a block in the hotel where the ptg is, but some people had pointed out that nearby hotels were cheaper
21:02:27 <notmyname> timburke: acoles: I think?
21:02:49 <timburke> i think tdasilva mentioned it first
21:02:57 <timburke> somebody
21:03:06 <notmyname> so a question that some people had was around when peopel were planning on getting there (early or mid week)
21:03:15 <notmyname> yeah, or maybe zaitcev
21:03:36 <acoles> I saw an email to -dev pointing out cheaper alternatives
21:04:13 <notmyname> timburke: what were your questions you asked me this morning?
21:05:07 <timburke> oh, mainly just whether it would make sense for a bunch of swifters to invade a cafe or something mon-tue
21:05:43 <notmyname> there will be the cross project sessions mon-tue
21:05:50 <mattoliverau> So long as there were no cross project stuff happening on those 2 days, sounds cool.
21:06:14 <notmyname> and from the ML, if we have things to address there, it's largely up to us to get on the agenda which is set by the teams meeting on those days
21:06:18 <clayg> timburke: i'm going to try to participate in some horizontal team sessions mon-tue - it's kinda the point of the ptg instead of mid-cycles?
21:06:30 <mathiasb> o/
21:06:57 <timburke> true. but how much of that time will actually be relevant to us?
21:07:20 <notmyname> ptg == the dev sessions of the summit, including cross project stuff
21:07:22 <clayg> timburke: all horizontal work is relevent to everyone by definition
21:07:41 <clayg> timburke: i'm trying - really trying
21:08:01 <notmyname> clayg: you're doing great :-)
21:08:09 <pdardeau> clayg: to a point right?
21:08:11 <timburke> i'm not opposed to sitting in on sessions that aren't directly and immediately relevant, but sometimes i want to get some shit done
21:08:26 <clayg> notmyname: i don't really buy it - as long as swift developers get to *go* summits - design will happen at summits
21:08:51 <clayg> ptg is replacing my mid-cycle travel - so it's going to have to do the job we normally do at mid-cycles - come summit time we'll make the best of what we've got
21:08:52 <notmyname> clayg: there is no summit any more. that doesn't exist. there is the ptg and the forum.
21:08:58 <clayg> if it doesn't work we re-orient
21:09:24 <notmyname> yes, that's true. and practically all that really matters
21:09:35 <clayg> sorry - yes - the forum - there will be feedback from operators and design work happen at *the forum* as long as swift operators and swift developers are *at* the forum (t.b.d.)
21:10:22 <notmyname> yes. but from what I've heard, forum == marketing talks and expo hall, and oh yeah I guess ops can have a room
21:10:40 <clayg> notmyname: be more optomistic!
21:10:57 <notmyname> we don't need to rehash the ptg/forum/summit/midcycle changes yet again this week :-)
21:11:10 <clayg> no one wants to ruin everything for everyone - if it happens - it was just an accident - we can trust everyone to observe that and correct
21:11:13 <notmyname> back to the ptg: clayg will be participating mon-tue in cross project
21:11:22 <clayg> long curve of history bends postive yada yada yada
21:11:31 <clayg> sure will!
21:11:39 <notmyname> and other people should too :-)
21:11:50 <notmyname> swift-specific sessions will be wed-fri
21:11:50 <acoles> yes I figure we should
21:11:55 <clayg> but that may acctually be a good oppertunity for the swift community to get some real work done without my constant disruption!
21:12:10 <mattoliverau> lol
21:12:16 <jrichli> +1
21:12:23 <mattoliverau> I'll be busy at least in part in the first 2 days.
21:12:38 <clayg> mattoliverau: kolla!?
21:12:40 <mattoliverau> Ansible stuff, maybe some release if Tony has his way :)
21:12:47 <notmyname> nice
21:12:48 <clayg> noice
21:13:05 <notmyname> in australia, kolla is known as "drop-containers"
21:13:11 <notmyname> /badly phrased joke
21:13:29 <mattoliverau> This time I hope to get there early and get a jetlag day, but really thats in the hands of the travel peeps, but I can nag :)
21:13:34 <notmyname> kolla->koala->drop-bears
21:13:43 <mattoliverau> notmyname: lol
21:13:50 <clayg> notmyname: it's not as fun when you have to explain it
21:14:06 <joeljwright> :D
21:14:11 <mattoliverau> I see what you did there. Shame there aren't move Aussies in the room
21:14:21 <mattoliverau> more
21:14:24 <notmyname> clayg: you just gotta go visit australia more often ;-)
21:14:26 <clayg> and here i thought dropbear was an sshd
21:14:29 <mattoliverau> +1000
21:14:48 <notmyname> ok, on to other stuff. seems ptg discussion has moved on
21:14:53 <notmyname> #topic other stuff
21:15:14 <clayg> i have a thing!
21:15:16 <notmyname> FYI https://review.openstack.org/#/c/336323/ will change test requirements
21:15:20 <clayg> oh...
21:15:26 <clayg> oh....
21:15:42 <notmyname> as in, your existing SAIO will likely break. gotta have /tmp (or TMPDIR) as an XFS mount
21:15:49 <notmyname> yay, progress!
21:15:55 <notmyname> so, be aware. and beware
21:16:00 <notmyname> clayg: what's your thing?
21:16:10 <clayg> notmyname: doesn't it ust make you saio skip a bunch of unittests until you fix it?
21:16:28 <notmyname> yep. tests will totally pass (see the current CI output). everythign is fine!
21:16:37 <timburke> notmyname: i don't see that touching the doc/saio/ tree....
21:16:40 <clayg> notmyname: oh, it's just fun stuff - you remember the openstack principles documnt?  it's actually changed quite a bit since it was first drafted -> https://review.openstack.org/#/c/398540/
21:16:44 <notmyname> just about 1/3 of unittests skip and about 98% of functests skip
21:17:03 <mattoliverau> so it also speeds up our tests.. nice ;P
21:17:06 <clayg> timburke: good point on the docs!  :)
21:17:07 <notmyname> lol
21:18:18 <notmyname> clayg: yeah, that's a much better phrasing
21:19:26 <notmyname> I have one more thing to bring up, but I want to make sure that other people have an opportunity too
21:19:45 <notmyname> anything else from anyone before we talk about EC libraries and releases?
21:20:40 <notmyname> ok, release stuff then
21:20:58 <notmyname> the swiftclient release was approved, so that's good
21:21:04 <notmyname> but we still want to do a swift release
21:21:16 <clayg> notmyname: oh yeah... what are waiting for?
21:21:43 <notmyname> if you've seen the patches in gerrit, kota_ (mostly) and clayg have been figuring out an issue related to liberasurecode
21:22:09 <notmyname> so there's been a 1.3.1 release of liberasurecode (here on out called "libec" to save typing)
21:22:32 <notmyname> so here's the deal: it seemed like the libec stuff was just about done back when we started looking at the swift release
21:22:58 <notmyname> it would be really good to release swift with requirements that force deployers to pull libec up to 1.3.1
21:23:13 <notmyname> however, nobody has a good idea on how to get that done
21:23:33 <notmyname> ideally, we'd release pyeclib to require libec>=1.3.1
21:23:57 <notmyname> however, that patch can't land because the gate tests will pull in libec from distros. which haven't packaged 1.3.1 yet
21:24:16 <notmyname> interestingly, this is one expression of a common problem
21:24:37 <notmyname> we've seen the same thing in the past for swift/swift3 and if we have to split golang into a different repo, we'll have the same issue there
21:24:45 <notmyname> so here's the options, as I see them now:
21:25:08 <notmyname> 1) redo tests to install the dependencies from source and test that way
21:25:27 <notmyname> 2) release now anyway and have a strongly-worded commit message and hope deployers read it
21:25:34 <notmyname> 3) ???
21:25:57 <notmyname> so I was hoping for some resolution that could apply to a swift release. but this is where we are
21:26:06 <notmyname> i'm not sure of the best way forward
21:26:31 <clayg> I don't think what we want out of the swift release is related to the libec 1.3.1 release?
21:26:50 <clayg> isn't there some other critical fixes in the swift code that we need to get out?  even if it doesn't help with issue in the libec code?
21:26:51 <notmyname> clayg: deployers to not have a massive footgun
21:26:52 <kota_> notmyname: thanks for bringing up it, that's what I've been fighting recently
21:27:11 <clayg> ok, go kota_ !
21:27:32 <notmyname> yeah, kota_ has been doing great work here. kota_, did I summarize it ok?
21:27:49 <kota_> notmyname: an idea for a solution, I'm draftign the hard coded dependency in-side of pyeclib, https://review.openstack.org/#/c/395998/
21:27:54 <notmyname> (rephrased) clayg: deployers to not have a massive footgun with some combination of dependencies
21:28:09 <mordred> notmyname: fwiw - the nova folks hit this with libvirt and neutron with ovs/ovn too - so it certainly raises its head in a bunch of places ... I've been trying and failing to find a general solution for a while
21:28:15 <kota_> notmyname: ah, it's kinda 1) you said above.
21:28:31 <mordred> I do not have one, mind you
21:29:09 <notmyname> oh, another option (IMO not terrible) is to combine the libec and pyeclib repos so they can be released at the same time (just with multiple deliverables). that gets rid of one layer in the puzzle in this case
21:29:28 <notmyname> mordred: any insight is appreciated
21:29:54 <mordred> notmyname: most of what I've got right now would be restating problems
21:30:25 <mordred> biggest issue with combining libec and pyeclib is that it still doens't solve the issue of pyeclib needing to be able to link against libec at pip install time AIUI
21:30:35 <notmyname> right
21:30:48 <notmyname> kota_: what do you want to do?
21:30:50 <clayg> notmyname: obviously the original intention for liberasurecode was that it would be an underlying shared library that could have bindings in different languages - when we accepted that dependency we agreed to carry that burden - i'm not sure it's fair to kevin and tushar to change that design - but obviously we have a choice about what swift does
21:30:53 <notmyname> tdasilva: any thoughts?
21:31:23 <kota_> mordred: true, that's the reason, we wanted to solve when spliting repo.
21:31:35 <tdasilva> notmyname: what clayg just said, plus i'm wondering if we are creating a artificial dependency for this upcoming release
21:31:39 <mordred> kota_: ++
21:31:51 <notmyname> tdasilva: what do you mean by artificial?
21:32:03 <kota_> notmyname: not have strongly opinon as well as you for now
21:32:07 <notmyname> kota_: ok
21:32:21 <tdasilva> notmyname: do the fixes that went into swift work with the existing libec/pyeclib?
21:32:29 <kota_> still trying to find the best way...
21:32:34 <clayg> notmyname: not to put words in tdasilva's mouth - but seriously we need to release critical fixes in swift/obj/ that have *nothing* to do with this right?
21:32:56 <tdasilva> the fix in libec is very isolated to libec right
21:33:00 <tdasilva> ?
21:33:01 <notmyname> clayg: true. I had only hesitated because I underestimated the dependency challenge for libec stuff
21:33:05 <notmyname> tdasilva: right
21:33:30 <notmyname> so, release swift now, no changes to dependencies for swift
21:33:55 * mattoliverau is sorry but has to run, have a meeting in Canberra that I need to drive in for.
21:34:05 <tdasilva> mattoliverau: o/
21:34:06 <notmyname> mattoliverau: exciting!
21:34:09 <joeljwright> enjoy!
21:34:26 <mattoliverau> :)
21:34:38 <tdasilva> notmyname: fwiw i know zaitcev is working to package new version of libec
21:34:47 <notmyname> that's great to know
21:35:43 <notmyname> so if we release swift now, without dep changes, should swift have a changelog note saying that deployers should upgrade libec? or watch out for different combos of libraries?
21:36:24 <notmyname> seems like the responsible thing to do, although it doesn't technically seem to be needed since there are zero changes in swift about it
21:36:51 <notmyname> thoughts?
21:36:52 <clayg> kota_: can you confirm that it would be *possible* to have a body of code with all the performance benifits of libec provide the ec interface that swift needs and be built/released *entirely* from code that can be tagged and released into the OpenStack gate via pypi?
21:37:10 <tdasilva> mordred: what does neutron nova do in these situations? changelog note with telling people to upgrade libvirt?
21:38:04 <mordred> tdasilva: they wait
21:38:13 <notmyname> clayg: the "via pypi" is the hard part right?
21:38:15 <mordred> until the version of whatever they want is in the distros
21:38:41 <clayg> ... granting that may not be *desirable* for various reasons - but basically just removing the extrenal linking to libec by inlining the c code that the cpython code in pyeclib calls?
21:38:51 <mordred> clayg: it should be phyiscally possible, yes
21:39:17 <kota_> either, 1. pyeclib includes libec code and built (seems hard work) or 2. devstack plugin can make it install libec from source ???(still need to make sure)
21:39:27 <kota_> not sure, right now.
21:39:33 <kota_> clayg:^^
21:39:47 <mordred> tdasilva: that is obviously not a _great_ way to write cutting-edge stuff ... thus the 'problem' - but I believe there are a bunch of feature flags of the kind "if libvirt version > blah"
21:39:55 <notmyname> mordred: interesting. one tiny difference here is that the non-openstack dependencies here are pretty much maintained by swift devs. so it's the same people waiting on distros to use their own code
21:40:12 <kota_> the deficality including libec to pyeclib is from the permission AFAIK
21:40:15 <mordred> notmyname: yes. this makes the problem more crazypants
21:40:38 <kota_> libec need to be located in sort of /usr/local/lib (means out of python package)
21:40:48 <kota_> and pyeclib will be in virtualenv
21:41:09 <kota_> to locate libec from pyeclib setupper, it requires root permition
21:41:20 <mordred> notmyname: but it's still a very similar problem - openstack wants to tell the distributors it needs a new version of $lib - we do it fine for python libs, but not for C libs
21:41:34 <notmyname> mordred: or potentially golang ;-)
21:41:36 <kota_> permission
21:41:37 <mordred> like, we certainly don't wait for distros to update python libs before we update python requirements
21:41:49 <mordred> notmyname: yah - basically anyting not-python we have no story for yet
21:41:54 <mordred> other than "wait for it to exist in the distros"
21:42:18 <notmyname> kota_: I'm currently mucking in the project config stuff to do things that require root. it's possible, but not straightforward, to install something as root before our tests get called
21:42:34 <clayg> mordred: fwiw, that exception for python libs was never obviously correct to me - i understand it being more convenient - but not more correct
21:42:37 <kota_> notmyname: how about, making 'warning log line' in pyeclib if older libec found
21:42:48 <mordred> the 'obvious' solution - which is host our own repo of distro packages that leads the release ... has the problem that it winds up becoming a production dependency for deployers even when we tell them to not rely on it for production since it's not getting security updates
21:42:57 <kota_> notmyname: it's still week but we can notify to upgrade for operators via warning log
21:42:59 <mordred> clayg: totally
21:43:01 <kota_> wak
21:43:04 <kota_> weak
21:43:47 <mordred> clayg: it just happens to be the one we have a ready mechanism to be able to communicate in a release what versions of a thing we need if someone wants to package us - definitely not indicative of having solved the problem holisitcally
21:43:54 <notmyname> kota_: and then when distros have libec 1.3.1 we can pull that out in favor of the hard version requirement?
21:44:08 <kota_> notmyname: sure
21:44:20 <notmyname> kota_: that seams reasonable, I think
21:45:16 <clayg> mordred: that's an interesting perspective - sheds a longer term attractive light on bindeps ;)
21:45:37 <notmyname> anyone else want to weigh in on kota_'s suggesting for libec and pyeclib?
21:46:28 <clayg> notmyname: we need to have a release-announce list for liberasure is my basic understanding
21:46:31 <notmyname> mordred: clayg: the thing with pypi is that we can push there, right? that's the only thing that pypi gives us that makes the python story at allr easonable
21:46:37 <clayg> then we need to tell people that package stuff to be on the list
21:47:04 <notmyname> mordred: are non-governance openstack/* projects allowed to use the release announce list?
21:47:18 <notmyname> ie pyeclib and libec
21:47:28 <mordred> notmyname: I do not know the answer to that ... swift certainly is allowed to :)
21:47:47 <notmyname> ok
21:47:56 <clayg> notmyname: idk, it makes it *work* - not sure it's "reasonable" - if a deployer is using packaged python-foobaz from distro-bar linux - us pushing to pypi doesn't really change their situation?
21:48:20 <notmyname> clayg: yeah, I agree
21:48:31 <clayg> we assume people deploy swift from "some random sha on master" (we do, we know other people that do) - but for people who use distro packages none of this effects them
21:48:39 <clayg> they're still at the whim of the release cycle of their distro
21:48:43 <mordred> clayg: it doesn't - but since we test with the version on pypi, we're signaling to the people packaing the next release of openstack that they should darned well also package the appropriate version of the python lib
21:48:58 <clayg> so we need to tell the distros - yo liberasure 1.3.0 will hurt you bad
21:49:29 <notmyname> mordred: have our own apt/yum repo that are firewalled to only allow CI nodes to access them? ;-)
21:50:35 <mordred> notmyname: yah :)
21:50:39 <acoles> notmyname: so is the proposal that we cut a release now with pyeclib warning/suggesting upgrade to libec (but not requiring), then when libec upgrade is in distros we release again and require latest libec?
21:50:44 <clayg> notmyname: honestly I've heard talk of doing some testing/ci from nightly built distro packages - if we could push a tag to liberasurecode and get new packages in the gate we'd be *pretty* close to what we need
21:51:08 <notmyname> clayg: we've already pushed the tag :-)
21:51:13 <clayg> ... still doesn't really effect the "what happens to deployers pulling from distros after the gate" question - that's a problem for liberasurecode release announce list
21:51:23 <mordred> ++
21:52:23 <mordred> we've gotten feedback at ops summits that there is actually a fairly wide range of folks deploying from distros, from pypi/source and from their own built distro packages - none of the three were a clear winner, so they're all certainly real problem spaces
21:52:27 <clayg> notmyname: i think swift works with libec 1.3.0 and swift doesn't need to change (we should do a release tho, it's unrelated and not blocked)
21:52:31 <clayg> acoles: ^
21:52:32 <notmyname> current idea is that we release swift with a changelog mentioning that deployers shoudl upgrade libec. then we also release pyeclib with a warning if libec<1.3.1. then after libec is in distros, roll those hard requirements forward through the dep chain
21:53:14 <clayg> acoles: we should *also* annouce liberasurecode packages - if you run swift + ec your realm of pain is a little bigger than *just* bugs in the swift repo unfortunately
21:53:27 <mordred> that sounds like a great plan - at least until such a time as we have a better one
21:54:20 <notmyname> acoles: kota_: clayg: tdasilva: jrichli: sound good?
21:54:23 <clayg> notmyname: yeah, it think your plan is more than the minimum required
21:54:39 <clayg> notmyname: but we *have* to figure out how to annouce releases to libec that's that the swift release notes?
21:54:57 <kota_> sounds a good way in current situation
21:55:03 <jrichli> +1
21:55:12 <acoles> got it. +1
21:55:14 <tdasilva> notmyname: just a little concerned about the pyeclib warning message??? can kota_, notmyname  explain that part again??
21:55:16 <kota_> i don't have better idea than it for now.
21:55:32 <notmyname> clayg: I think that's pretty easy. I've sent emails there myself in the past, and I think we can set up libec tags to be announced
21:55:48 <clayg> ohkay cool!
21:55:59 <notmyname> tdasilva: patch pyeclib to detect installed version of libec and emit a warning if <1.3.1 is detected
21:56:12 <kota_> tdasilva: not yet, prepared but i can make a patch for that for pyeclib. pyeclib can detect the libec version from c header
21:56:41 <notmyname> tdasilva: then once 1.3.1 is actually available in distros, we swap the warning for an actual hard dependency on >=1.3.1
21:56:42 <kota_> tdasilve: and if pyeclib find the older version, we can dump a log line to somewhere (e.g. syslog)
21:56:47 <tdasilva> is that really necessary? i'm afraid it will just raise bugs from people that see the warning and the solution is really just wait for the libec to be packaged and deployed
21:57:14 <notmyname> if the alternative is "slowly and silently corrupt data" then, yes. totally necessary :-)
21:57:59 <notmyname> and, hey, if distros bundle new versions, we can go through that process quickly
21:57:59 <kota_> tdasilva: another solution is built from source by deployer
21:57:59 <tdasilva> no no, i mean, is the warning message necessary right now if we just plan on putting the hard dependency later
21:58:16 <kota_> instead of waiting distro updated
21:58:31 <notmyname> tdasilva: well, i'm not sure that it's actually necessary, but it seems like the right thing to do
21:58:55 <tdasilva> ok, don't want to confuse matter further
21:59:16 <kota_> tdasilva: exactly, it may be redandunt though
21:59:28 <notmyname> seems we've, yet again, taken a short meeting agenda and managed to fill a full hour :-)
21:59:39 <notmyname> thank you everyone for coming. and thank you for working on swift
21:59:51 <notmyname> I'll work on the swift release process and get that finished up this week
21:59:57 <notmyname> #endmeeting