Tuesday, 2016-05-10

*** pgbridge has quit IRC00:02
openstackgerritOpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925000:05
*** Jeffrey4l_ has joined #openstack-swift00:10
openstackgerritMerged openstack/swift: go: adding a x-trans-id to repconn along with the sender device id  https://review.openstack.org/31316700:13
openstackgerritSamuel Merritt proposed openstack/swift: Skip SLO-reconciling probe test when SLO is off  https://review.openstack.org/31269800:13
notmynameopenstack clippy: "I see you've sent 4 emails to the ML today, and you started a thread that has over 75 replies. Can I order you a beer?"00:14
*** suyash has quit IRC00:20
*** diogogmt has joined #openstack-swift00:25
*** lyrrad has quit IRC00:30
mattoliveraunotmyname: +1 I think you deserve a beer00:30
mahaticgood morning00:31
mattoliveraumahatic: morning00:32
mahaticmattoliverau: o/00:34
kota_mahatic: morning00:37
kota_mattovierau: I think notmyname can get a bunch of glasses of beer from all swifters00:40
*** portante has joined #openstack-swift00:44
*** ndk has joined #openstack-swift00:45
*** dmellado has joined #openstack-swift00:49
*** tqtran has quit IRC00:54
*** sgundur- has left #openstack-swift00:57
tdasilvanotmyname: so far the best part of that long thread was: "I don't want to start a religious war...but...If you think Perl is "nice" or "easy" you better get you head checked....If you think Perl is "nice" or "easy" you better get you head checked....C++ is not worth mentioning as a language....don't force OpenStack to accept code written in $COOL_LANGUAGE because we don't all want to have to learn that language in add01:13
tdasilvaition to Python"01:13
*** klrmn has quit IRC01:15
jrichlinotmyname: What are you doing with that VM that you end up with 'Too many open files' ?01:15
jrichlire: supporting Go mail thread : all I can say is, wow.01:40
*** diogogmt has quit IRC01:44
*** arch-nemesis has quit IRC01:47
*** tqtran has joined #openstack-swift01:51
openstackgerritMerged openstack/swift: SwiftLogFormatter will log transaction IDs on INFO level  https://review.openstack.org/30952401:53
jrichlinotmyname: it looks like your test skipped the container-sync tests01:55
*** tqtran has quit IRC01:57
*** klamath has quit IRC01:59
*** klamath has joined #openstack-swift02:00
*** ndk has quit IRC02:00
*** ndk has joined #openstack-swift02:02
*** klamath has quit IRC02:03
*** klamath has joined #openstack-swift02:03
notmynameyeah, that ML thread is fun, right? :-)02:06
lifelessnotmyname: 'fun'. I'm not sure I speak the same language :)02:14
lifelessnotmyname: FWIW I'm not sure we *should* try to limit languages in the big tent; thus the conditional on my reply02:14
*** klrmn has joined #openstack-swift02:21
*** furlongm has quit IRC02:37
*** klamath has quit IRC03:00
janonymousPlease review patch 314100 , i03:06
patchbotjanonymous: https://review.openstack.org/#/c/314100/ - openstack-infra/project-config - Add non-voting gate-swift-tox-bandit job03:06
janonymousIt would require +1 from swift cores to move forward as commented by Andreas03:06
notmynamejanonymous: the entirety of what I can find out about that is "Bandit provides a framework for performing security analysis of Python source code"03:07
notmynamewhat does this test actually test for? what is it preventing, and what is it showing in the results?03:08
janonymousnotmyname: yes it is a static code analyser for security checks03:08
janonymousnotmyname: it tests against using of possible security issues which could be configured as per projects needs, as some checks might not be seen as security issue in some projs03:10
notmynamejanonymous: do you have an example of output run against swift?03:10
janonymousyou can run tox -e bandit on swift code03:11
janonymousby changing the values of n to different values will show possible medium and low severity issues , http://git.openstack.org/cgit/openstack/swift/tree/tox.ini#n6403:13
*** mingdang1 has joined #openstack-swift03:16
janonymousjust for example try: replacing s/-p gate/-lll03:16
notmynamebandit: commands succeeded03:16
notmyname$ tox -e bandit 2>&1 | grep 'Bandit internal error' | wc -l03:17
notmynamesomehow, I don't believe the first one03:17
*** Jeffrey4l_ has quit IRC03:18
janonymousm sry try from these: bandit -c bandit.yaml -r swift bin -n 5 -lll / bandit -c bandit.yaml -r swift bin -n 5 -ll / bandit -c bandit.yaml -r swift bin -n 5 -l03:19
*** Jeffrey4l_ has joined #openstack-swift03:19
*** sgundur- has joined #openstack-swift03:20
*** sgundur- has left #openstack-swift03:20
notmynamejanonymous: I'm not opposed to running more tests in the gate03:21
notmynamehowever, more tests does not always mean better03:21
notmynameI don't want to add a test that becomes the new longest-running one03:22
janonymousnotmyname: yes, you are right03:22
notmynameI don't want a test that randomly fails03:22
janonymousnotmyname: but what if we keep this non-voting03:22
janonymousnotmyname: it's a good practice though not a necessity :)03:23
notmynameI want tests that give me clear results with a path to fix them and a way to reproduce tehm locally03:23
janonymousnotmyname: bandit gives that options..03:25
notmynamejanonymous: what counts as passing of failing for bandit? right now I run these commands locally and get some interesting output and yet it says that everything passed03:26
*** _JZ_ has joined #openstack-swift03:26
notmynamejanonymous: also what about andreas's question about adding it to tox.ini?03:27
janonymousyes because issues with low/medium priority will not fail invocation03:27
janonymousnotmyname: since it is a non-voting job now, it might be not be needed at this point i think03:28
notmynameany ideas why this fails when runnign under tox but works when running locally?03:29
janonymousSome options that were tried are in this commit also ,for reference: https://review.openstack.org/#/c/196395/03:29
patchbotjanonymous: patch 196395 - swift - Adding bandit for security static analysis testing... (MERGED)03:29
*** links has joined #openstack-swift03:29
janonymousnotmyname: can you give me a case..03:30
notmynamethe 13.8k lines of errors I got when running it with `tox -e bandit`03:31
notmynameactually seems like the same error for just about every line checked: https://gist.github.com/notmyname/7b75487bf14de7e86fe30c70a65741a803:32
janonymoussomthing is wrong with env, bcs i get this as output: https://gist.github.com/codevulture/190ecc35071e69f809008227b7c733f403:36
janonymousi am checking..03:36
*** silor has joined #openstack-swift03:40
jrichlinotmyname sgundur- running probetests against different versions has shown that it was patch 270961 that broke the container_sync probetests03:40
patchbotjrichli: https://review.openstack.org/#/c/270961/ - swift - Container-Sync to perform HEAD before PUT object o... (MERGED)03:40
jrichlialthough the vm-saio-probe passed at the time.  And it did not skip the container-sync tests.  but if I run with the change just before that one, they pass in my VM.  With that change, the 3 fail.03:51
janonymousnotmyname: can you try out with bandit==0.17.3 version, maybe same as https://bugs.launchpad.net/bandit/+bug/1546777 :(03:53
openstackLaunchpad bug 1546777 in Bandit "internal error running: blacklist" [High,Fix released] - Assigned to Tim Kelsey (tim-kelsey)03:53
*** tqtran has joined #openstack-swift03:53
notmynamejanonymous: still errors with that03:54
janonymousnotmyname: same ones ?03:56
*** tqtran has quit IRC03:57
zaitcev"Secure Web Internet File Transmission Service (SWIFT) Transmitter User Guide. Overview;" WTF03:58
notmynamejanonymous: sorry. tox/venv was confusing me03:59
notmynamejanonymous: bandit 0.17.3 works in tox03:59
janonymousnotmyname: ohh great04:01
*** kei_yama has quit IRC04:05
*** kei_yama_ has joined #openstack-swift04:05
*** mingdang1 has quit IRC04:07
*** silor1 has joined #openstack-swift04:09
*** dmorita has quit IRC04:13
*** silor1 has quit IRC04:14
*** silor has quit IRC04:14
*** silor has joined #openstack-swift04:16
*** psachin has joined #openstack-swift04:32
claygjrichli: maybe al and I were running with ssync or post_as_copy = false or something?04:33
*** mingdang1 has joined #openstack-swift04:34
jrichliclayg: I think its that I had an old conf file that didn't have realms defined correctly04:34
claygjrichli: oic04:34
jrichliclayg: btw, both the fast-post and the post-as-copy tests had failed04:34
claygi'd be nice if the container_sync probetests would detect if container synt was setup and bail accordingly :\04:34
claygso are probetests passing on master now or no?04:35
claygyou don't have to answer that - i'm bringing up my vsaio now04:36
jrichliwell, i hadn't posted my results yet because I thought i had it set-up to run, but they get skipped.  however, i ran a test individually and it succeeds.04:36
jrichliclayg: clearly, I need to learn how to fully setup container-sync so the tests don't get skipped now that I changed my bad conf file.  But it will have to wait until tomorrow.  I am interested to hear if yours pass.04:40
jrichlisgundur- make sure you don't have an outdated version of container-sync-realms.conf04:42
*** sheel has joined #openstack-swift04:44
*** silor has quit IRC04:48
jrichlione thought on all this: it could be that the patch I mentioned no longer supports using what some docs say is "the older allowed_sync_hosts way of syncing."  but i haven't dug into what that means yet.04:55
* jrichli is going sleep now04:55
*** silor has joined #openstack-swift04:57
*** nadeem has joined #openstack-swift05:05
*** kei_yama has joined #openstack-swift05:07
*** kei_yama_ has quit IRC05:07
*** dmorita has joined #openstack-swift05:14
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873605:14
*** natarej has quit IRC05:16
*** natarej has joined #openstack-swift05:17
*** dmorita has quit IRC05:18
*** ukaynar has joined #openstack-swift05:22
*** mingdang1 has quit IRC05:26
*** mingdang1 has joined #openstack-swift05:26
*** mingdang1 has quit IRC05:32
*** ChubYann has quit IRC05:40
*** d0ugal has joined #openstack-swift05:52
*** _JZ_ has quit IRC05:58
*** klrmn has quit IRC06:15
*** rcernin has joined #openstack-swift06:26
*** ukaynar has quit IRC06:27
*** mmcardle has joined #openstack-swift06:27
*** jmccarthy has quit IRC06:29
*** jmccarthy has joined #openstack-swift06:29
*** dmorita has joined #openstack-swift06:36
*** dmorita has quit IRC06:42
*** SkyRocknRoll has joined #openstack-swift06:49
*** mariusv has quit IRC07:00
*** mariusv has joined #openstack-swift07:01
*** tesseract has joined #openstack-swift07:03
*** jamielennox is now known as jamielennox|away07:08
*** acoles_ is now known as acoles07:13
*** acoles is now known as acoles_07:18
*** rledisez has joined #openstack-swift07:22
*** d0ugal has quit IRC07:23
*** d0ugal has joined #openstack-swift07:26
*** d0ugal has quit IRC07:26
*** d0ugal has joined #openstack-swift07:26
*** acoles_ is now known as acoles07:34
acolesjrichli: clayg fwiw /test/probe/test_container_sync.py passes for me with post_as_copy true or false, rsync (not that I think that would make any difference)07:36
acolesI see some key errors in test_locked_container_dbs07:36
*** bil has joined #openstack-swift07:47
bilHi can someone help me with joss auth issues?07:48
*** jamielennox|away is now known as jamielennox07:49
openstackgerritAlistair Coles proposed openstack/swift: crypto - cleanup encrypter error handling  https://review.openstack.org/31266207:50
openstackgerritAlistair Coles proposed openstack/swift: crypto - cleanup decrypter exception handling  https://review.openstack.org/30480607:50
*** bil has quit IRC07:54
*** jordanP has joined #openstack-swift08:01
*** cbartz has joined #openstack-swift08:07
*** mvk has quit IRC08:16
*** daemontool__ is now known as daemontool08:30
*** nadeem has quit IRC08:36
*** jistr has joined #openstack-swift08:37
*** mvk has joined #openstack-swift08:45
*** mingdang1 has joined #openstack-swift09:05
*** eranrom has joined #openstack-swift09:14
eranromYesterday, there has been a discussion on problems with container sync probe test. I may have found a problem, however, I am still confused as why this does not happen 'at the gate'.09:18
eranromI suspect that every first call to get_info (proxy/controller/base.py) will update the cache but will return no info.09:19
eranromthus on a second call the info is returned, but not on the first.09:19
eranromthis is basically why the probe tests fail.09:19
eranromNow, the reason for the first call to always fail is the following: get_info creates an internal HEAD request for the container (in our case we are in the get_container_info code path), and does req.get_response(app) which in turn gets to GETorHEAD_base09:22
eranromThe code in get_info assumes that GETorHEAD_base would update the request environ to include the container info (which it does), however,09:23
eranromthe environment being updated by GETorHEAD_base does not seem to update the environmant of the request upon which we call req.get_resonse.09:24
eranromMy question is: when doing reg.get_response which calls the application do we expect the env of the calling request to get updated?09:24
*** daemontool has quit IRC09:28
acoleseranrom: hi! I don't have the answer to your question, but I didn't see the container sync probe tests fail (see scrollback)09:29
acoleseranrom: do you see them fail?09:29
*** chlong has quit IRC09:30
*** silor has quit IRC09:35
eranromacoles: I do see this from time to time.09:37
eranromacoles: again, it never happended on the "gate"09:37
acoleseranrom: how do you have container sync configured? the new way or "old way"?09:39
mahaticeranrom: notmyname earlier said this about gate: "probetest gates have never been "gates" (ie voting) because they are set up to run against the SAIO. and nobody has figured out how to do that with the gate images. there are probetests running in the community qa cluster though, but they aren't voting"09:40
*** mittal has joined #openstack-swift09:52
*** acoles is now known as acoles_09:52
*** acoles_ is now known as acoles09:54
mittalI have a query: In our swift deployment, with no replication, we are observing errors 'Client disconnected without sending enough data'09:58
mittalnot sure where to start tuning/debugging from ?09:59
mittalThese are observed on PUT requests only09:59
*** mvk has quit IRC10:03
*** mvk has joined #openstack-swift10:04
eranromacoles,: I am using SAIO, so I assume its the new way.10:05
eranromacoles: I guess that using SAIO means I have container sync configured...10:07
*** daemontool has joined #openstack-swift10:08
*** hosanai has quit IRC10:15
*** mittal has quit IRC11:10
*** silor has joined #openstack-swift11:16
*** ppai has joined #openstack-swift11:21
*** silor1 has joined #openstack-swift11:23
*** silor has quit IRC11:24
*** silor1 is now known as silor11:24
eranrommahatic: thanks! I am still puzzled by the inconsistency failure of those tests. Lets see what the west side of the world has to say :-)11:24
*** mingdang1 has quit IRC11:25
*** NM has joined #openstack-swift11:40
*** mingdang1 has joined #openstack-swift11:56
*** km has quit IRC12:01
*** ozialien10 has quit IRC12:08
*** ozialien10 has joined #openstack-swift12:09
acoleseranrom: i have a memory of needing to set up container-sync-realms.conf as an extra step to SAIO. But it could be the SAIO docs have been updated since then to include it.12:20
*** Lickitysplitted has quit IRC12:22
*** Lickitysplitted has joined #openstack-swift12:22
acoleseranrom: Request.get_response passes the request environ to the app, and returns a response whose environ is that of the Request instance i.e. same environ12:24
*** kei_yama has quit IRC12:34
eranromacoles,: setting up container-sync-realms.conf is now part of the SAIO setup. What you are saying about Request.get_response is also my understanding. The issue, however, is that the request env does not get changed by the call (which is the assumption made by get_info).12:36
eranrommore specifically, printing the request env before and after call_application shows the same env. Interestingly printing the env inside GETorHEAD_base does show the required change. Will try to dig dipper.12:38
*** pauloewerton has joined #openstack-swift12:50
*** [1]eranrom has joined #openstack-swift12:55
*** eranrom has quit IRC12:55
*** [1]eranrom is now known as eranrom12:55
*** asettle has joined #openstack-swift13:03
*** dmorita has joined #openstack-swift13:04
*** dmorita has quit IRC13:09
*** asettle has quit IRC13:15
*** daemontool_ has joined #openstack-swift13:21
*** links has quit IRC13:23
*** daemontool has quit IRC13:24
*** tongli has joined #openstack-swift13:30
*** esker has quit IRC13:34
*** klamath has joined #openstack-swift13:37
pdardeaugood morning13:41
*** cdelatte has joined #openstack-swift13:44
openstackgerritAndreas Jaeger proposed openstack/swift: List system dependencies for running common tests  https://review.openstack.org/29831313:50
*** mingdang1 has quit IRC13:51
*** sgundur- has joined #openstack-swift13:51
*** sgundur- has left #openstack-swift13:55
*** sgundur- has joined #openstack-swift13:55
*** ametts has joined #openstack-swift13:58
*** _JZ_ has joined #openstack-swift14:01
*** mwheckmann has joined #openstack-swift14:03
mwheckmannhello. Quick and dumb question: Any risk in *lowering* the num of replicas for a ring?14:05
*** esker has joined #openstack-swift14:05
pdardeaumwheckmann: it's a tradeoff of data durability and cost of storage. no right answer for all cases14:07
pdardeaumwheckmann: or were you asking about modifying an existing ring?14:08
mwheckmannpdardeau: exactly :)14:08
mwheckmannI wanted to know from a safety aspect.14:09
pdardeaumwheckmann: for modifying an existing ring?14:09
mwheckmannWondering if there is any limitation, that makes reducing the number of replicas (vs increasing them) more risky.14:09
mwheckmannyes, it's an existing ring in PROD14:10
mwheckmanncurrently have 9 replicas across 3 regions.14:10
mwheckmannEMEA east, EMEA west and NCSA14:11
mwheckmannI was hoping to lower my number of replicas down to 614:11
mwheckmannfrom 914:11
pdardeaumwheckmann: that's a really tricky thing. i'll defer that to others here such as acoles, cschwede, clayg14:11
mwheckmannpdardeau: I can't find any documentation that says otherwise, but I do get warnings on the rebalance operation.14:12
acolespdardeau: good morning. I'll defer to others with more experience in that area.14:12
mwheckmannLet's say I'm a little more hesitant to put that ring into PROD.14:12
*** esker has quit IRC14:20
pdardeaumwheckmann: how were you planning to get the reduction of replicas?14:20
*** esker has joined #openstack-swift14:21
mwheckmannpdardeau: what do you mean? I was just planning on running "swift-ring-builder <ring>.builder set_replicas 6"14:26
mwheckmannin fact, I did run it already but haven't put it into PROD14:26
mwheckmannpdardeau: if you're wondering why I want to do this, it's because PUT quorum for 9 replicas is 5. I'd like to get that down to 3.14:29
ahaleyou could try loading both rings up and see what changes per part - like how many replicas per region before and after14:29
mwheckmannahale: yes, I was thinking of looking into that with swift-get-nodes14:30
*** esker has quit IRC14:33
*** esker has joined #openstack-swift14:34
mwheckmannahale: thanks14:34
mwheckmannhere are some of the warnings I received on rebalance: swift/common/ring/builder.py:1369: RingValidationWarning: All devices in tier (1, 1, '') already contain a replica "contain a replica" % (tier,)))14:41
mwheckmannthe one above just happened once. The rest all reseble the following: swift/common/ring/builder.py:618: RingValidationWarning: The partition 85418 has been assigned to duplicate devices [6, 9, 1, 1, 5, 14] part, devs_for_part)))14:42
*** jamie_h has joined #openstack-swift14:42
mwheckmannof course, this is just an intermidiate state. I'll have to run rebalance again.14:42
mwheckmannso my guess is that things will fall into place.14:43
cschwede mwheckmann: note that you have to use upstream master to be able to use 6 replicas to get the quorum down to 314:47
cschwedemwheckmann: that change just landed a few days ago: https://github.com/openstack/swift/commit/29544a9e14:47
cschwedemwheckmann: so if you run one of the stable releases, you have to use 5 replicas to get down to a quorum of 414:48
cschwedemwheckmann: about the warnings - the ring-builder will only (re-)move one copy at a time, so if you change replicas in steps of 1 you probably won’t get warnings (if the ring was balanced well before)14:49
mwheckmanncschwede: understood. thanks for the info. Do you know anything about any bugs/risks when lowering the number of replicas on a ring in PROD?14:49
cschwedemwheckmann: well, besides the risk of a lower durability - i’m not aware of any bugs atm; the replicators should fix this. personally i wouldn’t be worried to have „only“ 5 or 6 replicas14:52
cschwedemwheckmann: but to be fully sure everything works as expected i would simply create another storage policy, create some data in it and then decrease the replicas of that policy14:52
*** pcaruana has joined #openstack-swift14:53
mwheckmanncschwede: good idea.14:54
*** esker has quit IRC14:54
notmynamegood morning14:54
*** esker has joined #openstack-swift14:55
mwheckmanncschwede: I'm not worried about the durability. More of any bugs or limitations in dropping the replica count on a ring in PROD since it's probably an operation that isn't done as frequently14:55
sgundur-notmyname: morning,  yesterday when I rechecked with the current master + 311899 ; I ended up getting the same errors around the container_sync15:04
*** Jeffrey4l_ has quit IRC15:05
sgundur-notmyname: also my saio setup is on rpc  VM15:06
*** psachin has quit IRC15:07
*** ukaynar has joined #openstack-swift15:08
mwheckmannnotmyname: sorry to ask you point blank, but is there any known risk in reducing replica count on a ring? (I'm talking about a production ring of course)15:09
notmynameif you reduce a replica count, you'll basically immediately get the new ring's availability characteristics (reduced number of primary nodes) for reads, you'll immediately get the (likely) faster writes from a reduced quorum, but it will take a while for the replication processes to sweep up the old replicas (ie reclaim space)15:12
notmynameI wouldn't guess that there would be any adverse affect from reducing replicas15:12
notmyname(but that's rather untested. just a guess. I don't /think/ I'm forgetting something, but I wouldn't know if I am ;-)15:13
notmynameprobably the way to test it is using the dispersion report15:13
dfg_i thought you could use the fractional replicas thing to do this incrementally- or did somebody already say that15:14
mwheckmannnotmyname: thanks.15:14
notmynamenot sure if someone said that, but yeah, use fractional replicas to gradually adjust it down15:15
notmyname4 -> 3.9 -> 3.8 -> ... -> 3.1 -> 315:15
mwheckmannok. I will try that.15:16
notmynamewhen increasing replicas, you gotta take into account the rebalancing that happens. the system thinks data should be somewhere but it hasn't made it over there yet15:16
notmynamebut with reducing replicas, you're basically jsut immediately forcing over-replication and it should be find from an anvailability point of view15:17
mwheckmanngot it15:17
notmynameso while you pretty much must use fractional replicas to increase the replica count, I'm not sure that's strictly necessary (or gains you much at all) when reducing replicas15:18
notmynamebut it certainly won't hurt anything to reduce replicas in small steps15:18
mwheckmannWondering about this commit: https://github.com/openstack/swift/commit/29544a9e While it clearly helps in the 2 region case, I don't see it helping if we have more than two. Does write affinity take precedence over the quorum?15:20
mwheckmanni.e will it write to the handoff partitions?15:20
mwheckmannAny reason that the proxy quorum couldn't be configurable?15:20
mwheckmann(as much as I dislike the idea of adding more knobs)15:21
notmynamewrite affinity and quorum are orthogonal15:22
notmynamequorum is how many must be written. affinity is where they will be written15:22
*** chuck__ is now known as zul15:24
*** arch-nemesis has joined #openstack-swift15:24
*** AJaeger has joined #openstack-swift15:24
AJaegernotmyname: are you around to discuss other-requirements?15:26
*** wshao has joined #openstack-swift15:26
notmynameAJaeger: sure15:27
AJaegersince we both worked on this, should we merge efforts? I prefer my change since it includes the tox.ini setup and the now added "test" selector - but I'm fine if you take it over and add a co-authored-by...15:28
AJaegerI can also abandon mine. What do you prefer?15:28
notmynameAJaeger: I heard about other-reqs and thought it sounded interesting and cobbled together something, but I have no particular dog in the fight. seems like yours has more stuff (tox.ini and doc updates) and, really, I was just guessing at what to put in mine15:29
notmynameso, I'm fine to go with yours15:30
AJaegernotmyname: I was guessing as well ;) But tried to minimize...15:30
notmynamemy only question is about the differences between the two files15:30
AJaegerMine seems to work - otherwise it wouldn't pass the tests.15:30
notmynameAJaeger: which patch is yours?15:30
notmynameyeah, mine too :-)15:30
AJaegernotmyname: https://review.openstack.org/29831315:30
AJaegernotmyname: you're adding a few packages that are in the base image, so don't need to be installed.15:31
AJaegerLike python15:31
notmynamewhat does the "test" thing do?15:31
notmynamemight be needed on 16.04 :-)15:31
AJaegernotmyname: see http://lists.openstack.org/pipermail/openstack-infra/2016-May/004232.html and followups15:32
notmynameand yeah, the line of what's a dependency or not is fuzzy for me. where do we stop?15:32
AJaegerAnd also https://review.openstack.org/31457415:32
AJaegernotmyname: it doesn't matter if you add more ;)15:32
AJaegerif they're installed already, it's good15:33
AJaegernotmyname: the test comes from https://review.openstack.org/314571 - we plan to run "bindep test", so that you can specify both runtime dependencies and test dependencies. See also the comments by lifeless on 29831315:33
notmynameseems weird to have "test" on everything though15:34
AJaegeryeah ;)15:35
*** pgbridge has joined #openstack-swift15:35
AJaegerbut if you add back python, it would be a base requirement15:35
notmynamebut python-dev is only for testing?15:35
*** esker has quit IRC15:35
notmynameI'm not sure that's true (but I don't have data either way)15:35
notmynamewell, no data aside from it just doesn't really seem to work without the -dev package15:36
notmynameie for building other dependencies15:36
notmynameso that seems more than a "test" thing to me15:36
*** esker has joined #openstack-swift15:36
notmynamebut now we're into the /me is guessing territory again15:36
AJaegerexactly - let's see what lifeless says when he reviews, you might want to talk with him about it - and with fungi. best on #openstack-infra15:37
AJaegerand feel free to take over my change - or merge it and update the file afterwards ;)15:37
notmynameno no. other way around :-)15:37
clarkbpython-dev is used to link against python15:37
clarkbeg when liberasure coding python lib is built15:37
notmynameAJaeger: you should update yours to include anything in mine that's good, and I should abandon mine :-)15:38
AJaegerclarkb: so, should it be in test or in the base?15:38
clarkbI would put it in base15:38
clarkbsince swift features rely on iy15:38
clarkbnot just tests15:39
AJaegernotmyname: ok, will do.15:39
AJaegerclarkb: Ok, will change.15:39
notmynameAJaeger: oh, and in yours, jerasure isn't needed at all15:39
* AJaeger will send a new version later15:39
AJaegernotmyname: great!15:40
notmynameso should any of them be test?15:40
notmynameall of them are for actual basic swift features and not just for testing15:40
AJaegernotmyname: let me send of an email on openstack-infra in reply to lifeless's latest one - are you on the list or should I CC you?15:41
notmynameAFAIK we have zero non-python dependencies that are used only for testing15:41
notmynameAJaeger: I don't think I'm on that list15:41
AJaegerok, I'll CC you...15:41
mwheckmannnotmyname: so if write affinity and quorum are orthogonal: With 6 replicas across 3 regions (2 replicas per region) quorum would be 4. Let's say I'm writing on region 1 and my affinity there is set to region 1 as well. It will write 4 replicas in region 1 (on the handoff devices?) even if normally my ring specified 2 replicas in region 1?15:45
notmynameyou'll write 6 in region 1 and require that 4 be successfully written.15:46
AJaegernotmyname: mail send, I'll be back online later and update the change...15:46
mwheckmannok. So then write affinity takes precedence over what the normal behaviour of the ring would be15:47
notmynamealso, 6 replicas + 3 regions = 2x per region if and only if the regions are exactly the same size (ie exact same total weight) and your cluster is steady-state healthy. ie only if you're lucky15:47
mwheckmannnotmyname: of course15:48
notmynameAJaeger: thanks15:48
mwheckmannI do always make sure that my rings are well balanced :)15:48
*** tesseract has quit IRC15:52
* notmyname needs to get ready and go to the office15:54
*** wshao has quit IRC16:03
openstackgerritAndreas Jaeger proposed openstack/swift: List system dependencies for running common tests  https://review.openstack.org/29831316:10
*** daemontool_ has quit IRC16:13
openstackgerritPaul Dardeau proposed openstack/swift: resurrect gholt blog posts on building consistent hashing ring  https://review.openstack.org/31427016:16
openstackgerritPaul Dardeau proposed openstack/swift: Prose lines wrapped at 70 characters  https://review.openstack.org/31466016:17
*** asettle has joined #openstack-swift16:18
*** asettle has quit IRC16:23
*** lyrrad has joined #openstack-swift16:24
*** esker has quit IRC16:25
*** ppai has quit IRC16:25
*** esker has joined #openstack-swift16:26
*** tqtran has joined #openstack-swift16:26
*** chuck__ has joined #openstack-swift16:30
*** joeljwright has quit IRC16:31
*** zul has quit IRC16:31
*** chuck__ is now known as zul16:31
*** cbartz has quit IRC16:36
*** diogogmt has joined #openstack-swift16:43
*** CaioBrentano has joined #openstack-swift16:46
*** flaper87 has quit IRC16:46
*** flaper87 has joined #openstack-swift16:46
*** cdelatte has quit IRC16:46
*** cdelatte has joined #openstack-swift16:47
*** ukaynar has quit IRC16:47
notmynamepdardeau: nice. thanks (for the line-wrapping)16:50
notmynamepdardeau: also, now if you're measured by "lines of docs proposed" your score went up by 5x or something ;-)16:50
*** nadeem has joined #openstack-swift16:50
notmynamepdardeau: I've got a lot of meetings today, but I'll try to give it a review later today16:51
*** nadeem has quit IRC16:53
*** nadeem has joined #openstack-swift16:54
*** delattec has joined #openstack-swift16:56
*** mvk has quit IRC16:57
pdardeaunotmyname: lol, thx16:58
notmynamebkeller`: ping16:58
pdardeaunotmyname: i think your comment about openstack clippy has been burned into my long-term memory banks (it's a keeper)16:59
notmynamepdardeau: lol16:59
*** klrmn has joined #openstack-swift16:59
notmynamebkeller`: so I just saw the bug you assinged yourself in LP (bug 1576889)16:59
openstackbug 1576889 in OpenStack Object Storage (swift) " change default ports for servers" [Undecided,Confirmed] https://launchpad.net/bugs/1576889 - Assigned to Bryan Keller (kellerbr)16:59
*** cdelatte has quit IRC16:59
notmynamebkeller`: that's the doc bug that was auto-created?16:59
bkeller`yeah, was that already fixed?17:00
notmynamebkeller`: AFAICT, that's actually already done in sgundur-'s original patch17:00
*** esker has quit IRC17:00
notmynameI'm guessing she added the DocImpact tag because, well, it impacted the docs ( sgundur- you did not do anything wrong there)17:00
*** daemontool_ has joined #openstack-swift17:00
bkeller`ah, ok17:00
notmynamebkeller`: but yeah, if you can give the code a cursory overview to see the updated port ranges etc, that would be great. and, if it's done, and if you can, mark it as resolved/nonissue/closed?17:01
bkeller`I think I mistook some of the other services that were still running on 60xx then17:01
*** jordanP has quit IRC17:03
*** rledisez has quit IRC17:04
*** jmccarthy1 has joined #openstack-swift17:14
*** BAKfr has quit IRC17:20
*** BAKfr has joined #openstack-swift17:20
*** jistr has quit IRC17:21
notmynamebkeller`: ah cool. it's all taken care of then?17:22
bkeller`yeah, there was nothing for me to do17:23
sgundur-notmyname: ha :) , bkeller` thanks17:23
*** ChubYann has joined #openstack-swift17:24
sgundur-notmyname, bkeller` I had added that tag , but didn't know it abt the auto-created that bug17:25
*** daemontool_ has quit IRC17:25
notmynameyeah, DocImpact used to be the way to tell the docs team "hey something changed". now it will auto-create a bug in our own project17:26
notmynamesince we're pretty good about actually enforcing "docs land with code", the docimpact is not nearly as useful17:26
*** lcurtis has joined #openstack-swift17:46
*** asettle has joined #openstack-swift17:49
*** asettle has quit IRC17:50
*** acoles is now known as acoles_17:54
*** garthb has joined #openstack-swift17:57
*** chuck__ has joined #openstack-swift18:06
*** zul has quit IRC18:07
*** chuck__ is now known as zul18:07
*** ukaynar has joined #openstack-swift18:11
*** rcernin has quit IRC18:13
*** mvk has joined #openstack-swift18:17
*** thumpba has joined #openstack-swift18:25
*** gyee has joined #openstack-swift18:34
*** janonymous_ has joined #openstack-swift18:35
lifelessAJaeger: notmyname: you wouldn't have 'test' on everything, only on those things which are not runtime deps.18:40
AJaegerlifeless: then my change https://review.openstack.org/#/c/298313 should be fine. thanks18:42
*** janonymous_ has quit IRC18:42
*** janonymous_ has joined #openstack-swift18:42
*** rcernin has joined #openstack-swift18:43
lifelessAJaeger: wel no, because the dev packages are not runtime deps18:44
lifelessAJaeger: imagine you're building wheels18:44
*** thumpba has quit IRC18:44
*** thumpba has joined #openstack-swift18:46
AJaegerlifeless: clarkb, notmyname discussed a bit earlier...18:46
AJaegerlifeless: but if I'm not building wheels, then those are required.18:46
clarkbthey are install deps18:46
clarkbnot test deps18:47
AJaegerso, how to classify these then?18:47
lifelessffi-devel is a build dep18:49
lifelesslibffi6 is a runtime dep18:49
AJaegeragreed, lifeless18:50
AJaegerSo, how do we add these to profiles?18:50
AJaegerWe have the base profile that includes runtime deps - and we discussed the test profile for gating. So, build deps go to the test profile?18:51
lifelessAJaeger: I'm saying I think that the runtime deps should be in the default profile, and build deps should not18:51
lifelessAJaeger: since testing (for us) is intrinsically a source operation, putting the nonruntime deps needed for testing in a test profile is what I have suggested18:52
clarkbbut then anyone installing from source which is the target audience will fail18:52
lifelessclarkb: how will they fail?18:52
clarkbinstall wont work18:52
lifelessclarkb: bindep isn't invoked by pip or anything else; they're going to be following docs we write18:52
clarkbif everyone using bindep must install test deps anyways why distinguish at all18:54
AJaegerclarkb, lifeless: I just updated the infra-manual section on bindep, see https://review.openstack.org/31457418:54
AJaegerThat reflects what lifeless just said IMHO - and we could make clearer that developer or people building from source need this test environment.18:55
AJaegerWhat do you think?18:55
clarkbI dislike it, I think that regardless of why someone is installing eg swift that the default bindep list should just work18:56
clarkbmaybe they are running tests, maybe this is how they deploy, I don't care. Pip install after bindep runs should just work18:57
AJaegerWe should discuss together with fungi - can we move to #openstack-infra and continue after the meeting18:57
tdasilvahere's patchset 1 to merge pyeclib/liberasurecode to openstack gerrit: https://review.openstack.org/#/c/314724/18:58
patchbottdasilva: patch 314724 - openstack-infra/project-config - Migrating liberasurecode and pyeclib projects18:58
*** esker has joined #openstack-swift18:59
notmynametdasilva: thanks!19:00
notmynameFYI, the infra team meeting is starting now in -meeting, and one of the topics is about what we'll need to work together on for testing/CI/gate/etc for golang stuff19:01
notmynamemostly I'm expecting to get a list of stuff to ask instead of get actual answers. if you're interested, it's starting now19:02
openstackgerritPaul Dardeau proposed openstack/swift: resurrect gholt blog posts on building consistent hashing ring  https://review.openstack.org/31427019:07
*** pcaruana has quit IRC19:12
*** acoles_ is now known as acoles19:13
*** jmccarthy1 has quit IRC19:19
*** SkyRocknRoll has quit IRC19:28
notmynamehere's some initial notes from the -infra meeting https://etherpad.openstack.org/p/golang-infra-issues-to-solve19:36
openstackgerritAlistair Coles proposed openstack/swift: crypto - don't break conditional GETs on unencrypted objects  https://review.openstack.org/31474019:37
acolesphew that was fun ^^ !19:38
acolesnotmyname: I updated crypto priority reviews to add that ^^19:40
openstackgerritAlistair Coles proposed openstack/swift: crypto - don't break conditional GETs on unencrypted objects  https://review.openstack.org/31474019:43
*** mmcardle has quit IRC19:44
lifelessclarkb: honestly, I don't have the energy to argue, do whatever makes sense19:44
*** acoles is now known as acoles_19:45
lifelessclarkb: I was asked for my opinion, I've given it19:45
*** rcernin has quit IRC19:51
*** daemontool_ has joined #openstack-swift19:51
*** tongli has quit IRC19:53
*** bsdkurt has quit IRC19:58
*** rcernin has joined #openstack-swift19:58
*** janonymous_ has quit IRC19:59
clarkblifeless: as did I :) out of curiosity do you not expect most bindep users to be installing from source?20:03
notmynameacoles_: thanks20:04
*** daemontool_ has quit IRC20:06
lifelessclarkb: I know a bunch of deployers that start from source but deploy binaries20:07
lifelessclarkb: (both wheels and debs)20:08
lifelessclarkb: like upper-constraints, I expect other-requirements to provide input to their deployment metadata, whether the actual things copied onto their servers are source/wheels/debs/rpms20:08
clarkbseems like we should be able to achieve those goals while also making default just work20:10
clarkbmaybe annotations that don't impact the default run?20:11
torgomaticnotmyname: I tried to weigh in on the whole golang thing. Hopefully I managed to convey that read() takes a long time and blocks the calling thread. http://lists.openstack.org/pipermail/openstack-dev/2016-May/094549.html20:12
notmynametorgomatic: I saw. thanks :-)20:12
lifelesspossibly, it would require a bit of a back-to-basic design review of the ux/minilanguage.20:12
lifelesstorgomatic: it sounds like go does exactly like what you did in Python20:14
lifelesstorgomatic: a threadpool for syscalls20:14
*** bsdkurt has joined #openstack-swift20:14
lifelesstorgomatic: (As a data point, in case its interesting, squid does exactly the same thing with one of its drivers [it has many :P]) - its C++ these days, written in a nonblocking event loop style.20:15
torgomaticlifeless: pretty much; it just does it transparently and quickly, whereas the stuff I did was neither one ;)20:15
tdasilvawow, it's hard to follow the convo in #openstack-meeting20:42
*** suyash has joined #openstack-swift20:46
*** esker has quit IRC20:47
*** silor has quit IRC20:49
notmynametdasilva: the TC meeting? yeah, I didn't realize golang would be a topic there. I happened to see my client highlight, and i'm glad I did20:52
tdasilvanotmyname: yes, interesting meeting.20:53
*** thumpba has quit IRC21:02
*** garthb has quit IRC21:09
*** mingdang1 has joined #openstack-swift21:11
jrichlitorgomatic: nice reply.  good to see some more analytical facts on the thread - less conjecture and pre-determined judgements21:12
*** pauloewerton has quit IRC21:15
*** ukaynar has quit IRC21:15
*** ametts has quit IRC21:17
*** garthb has joined #openstack-swift21:17
*** NM has quit IRC21:31
pdardeauthat's a lot to digest in TC meeting minutes21:34
*** daemontool_ has joined #openstack-swift21:35
pdardeaui'd like to ask -- how did hummingbird/go flip from not being a thing to being a big topic of interest overnight?21:35
notmynamewell sit back and grandpappy will tell you a story of how things used to be.....21:36
pdardeaubtw, i'm not opposed to go (in fact, it has some appeal if it doesn't include pip!)21:36
notmynameclarkb: did you see sam's recent message to the thread about some of the technical reasons of golang over python? I only wanted to point that out because we aren't blaming eventlet or trying to replace eventlet with golang21:38
clarkbyes, that is part of where I got that from (but also reading the designate code it seems like a huge part of the potential slowness there is just poor eventlet and related code)21:39
clarkbevery dns query uses a brand new greenthread which in turn uses a brand new db session21:39
zaitcevpdardeau: Basically redbo is a genius. He went away for a weekend and basically came back hummingbird's object server fully functional. His biggest thing was writing a pickle parser in Go (that's where my own Swift-in-C faltered).21:40
zaitcevpdardeau: from there it was a matter of gradually adding functionality step by step, such as replication.21:40
clarkbnotmyname: I interpretted that email as eventlet makes these things hard (and not necessarily python)21:41
notmynamepdardeau: the summary is that we presented our plan to use go, there's some necessary follow up from some openstack cross-project stuff (like docs, infra, etc), and the resulting "conversation" has a lot of people trying to grok the whole situation and its consequences. and so there's a lot of people who want to say something about that21:43
pdardeauzaitcev: thanks for the background21:43
pdardeauzaitcev: the Swift-in-C sounds interesting. is it in github? (not trying to stir the pot, more just from curiosity)21:44
notmynameso yeah, redbo (and then dfg_ and nadeem and others) have been working on it for a while, but the http://lists.openstack.org/pipermail/openstack-dev/2016-May/093795.html message (and the one linked there) are what started everything in the last few days21:44
zaitcevpdardeau: https://github.com/zaitcev/swift-oserver  - but there's nothing useable there, except that it has its own webserver that I stole from jgarzik's tabled.21:45
pdardeaunotmyname: right. i'm more curious in what prompted it to be a big discussion at summit21:45
notmynamepdardeau: ah. I guess you can blame that on me :-)21:45
zaitcevor, actually not even that. it was the plan21:46
clarkbnotmyname: specifically lots of discussion about IO, and read(), and slow disks, which are not python problems they exist for Go too, then the explanation of threadpool overhead (whih is an eventlet issue and not necessarily a python issue)21:46
notmynameit's something that as a community we've been talking about for a year or 18 months now. so based on the rax results + the current state of hummingbird + what other clusters are seeing as issues, it was time to make a go/no-go decision about the golang work21:46
pdardeaunotmyname: heh, go/no-go21:47
clarkbcertainly one way of addressing that is to use a different language end or VM/runtime21:50
*** cdelatte has joined #openstack-swift21:52
*** delattec has quit IRC21:54
*** cdelatte has quit IRC21:55
*** daemontool_ has quit IRC21:55
*** cdelatte has joined #openstack-swift21:59
pdardeauzaitcev: i had recently looked into doing something similar for c++, d, java, nim, and scala https://github.com/pauldardeau/swift-babel22:01
zaitcevpdardeau: trere's a pickle for C++ BTW22:04
*** cdelatte has quit IRC22:08
*** nadeem has quit IRC22:09
*** mwheckmann has quit IRC22:10
*** ukaynar has joined #openstack-swift22:11
*** cdelatte has joined #openstack-swift22:11
*** _JZ_ has quit IRC22:13
notmynamepdardeau: yeah! why aren't we rewriting swift in nim!?22:14
*** sheel has quit IRC22:15
pdardeaunotmyname: have you looked at nim? it's a cool language, especially if you like the intersection of python and pascal22:16
notmynamepdardeau: yeah, a little bit, but not enough to get much working. I was working on a swift directory uploader22:16
notmynamepdardeau: (doesn't actually work) https://gist.github.com/notmyname/bd5e01a42831b46b80109686213c784222:17
notmynamepdardeau: some of the operator overloading in nim is kinda mind-bending22:18
notmynamepdardeau: FWIW, that might compile with either num 0.10.2 or 0.11.222:18
notmynamebut I think they've made breaking updates to the language since then22:19
*** CaioBrentano has quit IRC22:19
pdardeaunotmyname: swift_pusher? spoken like a true PTL ;-)22:21
notmyname...and that's why I hadn't published it before (and why I just deleted it)22:21
notmynamehad my RAX password in it. oops. all updated now22:21
notmynamepasswords reset22:21
*** mingdang_ has joined #openstack-swift22:23
*** mingdang1 has quit IRC22:24
*** delattec has joined #openstack-swift22:24
notmynameok, sanitized version https://gist.github.com/notmyname/af8c99f04e7b8bedcff3c44827d602d722:24
*** pgbridge has quit IRC22:25
pdardeaunotmyname: in your defense, it did say 'discard' :)22:26
notmynamepdardeau: the way I do that in python scripts is like https://github.com/notmyname/cf_dropbox/blob/master/cf_drop.py#L1022:26
notmynamethen i have a global cf_auth.py in my site-packages that has my creds22:27
*** cdelatte has quit IRC22:27
notmynamepdardeau: and the nice things about using a password manager is (1) you wouldn't guess that password and (2) I'm not using it for *anything* else22:28
*** pgbridge has joined #openstack-swift22:38
*** ukaynar has quit IRC22:41
openstackgerritJohn Dickinson proposed openstack/swift: resurrect gholt blog posts on building consistent hashing ring  https://review.openstack.org/31427022:41
notmynamepdardeau: I simply added python syntax highlighting to the code blocks22:41
pdardeaunotmyname: nice! thx22:48
pdardeaugood morning mattoliverau22:48
mattoliveraupdardeau: hey man22:49
*** mingdang_ has quit IRC22:51
*** lcurtis has quit IRC22:56
*** jamielennox is now known as jamielennox|away23:13
*** delatte has joined #openstack-swift23:18
*** d0ugal has quit IRC23:21
*** delattec has quit IRC23:21
*** jamielennox|away is now known as jamielennox23:21
*** diogogmt has quit IRC23:23
*** km has joined #openstack-swift23:24
*** rcernin has quit IRC23:28
*** arch-nemesis has quit IRC23:30
*** kei_yama has joined #openstack-swift23:34
*** d0ugal has joined #openstack-swift23:36
*** hosanai has joined #openstack-swift23:47
*** ChanServ sets mode: +v hosanai23:47
*** NM has joined #openstack-swift23:52
*** chlong has joined #openstack-swift23:56
*** garthb has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!