| notmyname | PTG topic etherpad https://etherpad.openstack.org/p/swift-ptg-pike | 00:13 |
|---|---|---|
| *** ChanServ changes topic to "Topic: Let's talk, we're nice. | PTG etherpad: https://etherpad.openstack.org/p/swift-ptg-pike | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews" | 00:14 | |
| timburke | notmyname: i agree, we probably *should* have two blocks for sharding :-) | 00:15 |
| notmyname | it's the only way to scale? | 00:15 |
| timburke | one early, convince people to go look at it that evening, then later we can start having a real conversation with mattoliverau | 00:16 |
| notmyname | etherpads for the whole PTG are being collected at https://wiki.openstack.org/wiki/PTG/Pike/Etherpads | 00:18 |
| notmyname | that's where we figure out what to go to and when to be there. there is no global schedule. basically, each team gets a room and sets their own schedule | 00:19 |
| mattoliverau | right.. we should really shard our time :P | 00:19 |
| *** chsc has joined #openstack-swift | 00:31 | |
| *** chsc has joined #openstack-swift | 00:31 | |
| openstackgerrit | Merged openstack/swift: Refactor recon to use single md5_hash_for_file function https://review.openstack.org/406277 | 00:38 |
| clayg | ^ woot! wtg tdasilva! | 00:38 |
| openstackgerrit | Merged openstack/swift: remove reference to deprecated tool https://review.openstack.org/424312 | 00:39 |
| openstackgerrit | Merged openstack/python-swiftclient: Add Constraints support https://review.openstack.org/413348 | 00:39 |
| openstackgerrit | Merged openstack/python-swiftclient: Accept more types of input for headers/meta https://review.openstack.org/359477 | 00:39 |
| *** zaitcev has quit IRC | 00:41 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Break less abstractions in test_auditor https://review.openstack.org/424393 | 00:52 |
| *** mingyu has joined #openstack-swift | 00:53 | |
| *** chsc has quit IRC | 00:56 | |
| *** mingyu has quit IRC | 00:58 | |
| *** jamielennox is now known as jamielennox|away | 01:02 | |
| *** _JZ_ has quit IRC | 01:05 | |
| *** sams-gleb has joined #openstack-swift | 01:09 | |
| openstackgerrit | Jim Cheung proposed openstack/liberasurecode: Add Phazr.IO libphazr backend to liberasurecode https://review.openstack.org/424353 | 01:12 |
| JimCheung | Updated changes per Tim's suggestions. | 01:13 |
| *** sams-gleb has quit IRC | 01:14 | |
| notmyname | does anyone know what a .wsgi file is? like if someone asked you for a .wsgi file for your app, what does one look like? | 01:15 |
| *** jamielennox|away is now known as jamielennox | 01:17 | |
| clayg | notmyname: it's like a .py but it has a "application" in it's globals when you import it - and it ends in .wsgi for some reason | 01:17 |
| clayg | ... and it might be "app" instead of "application" or something else bs like that | 01:18 |
| notmyname | ah, ok. thanks. google wasn't really helping me | 01:18 |
| clayg | http://flask.pocoo.org/docs/0.12/deploying/mod_wsgi/ | 01:18 |
| clayg | WSGIScriptAlias is the bs obscure apache non-sense you're looking for | 01:19 |
| notmyname | yeah http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIScriptAlias.html | 01:19 |
| notmyname | ok, thanks | 01:19 |
| clayg | werd | 01:19 |
| openstackgerrit | Jim Cheung proposed openstack/liberasurecode: Add Phazr.IO libphazr backend to liberasurecode https://review.openstack.org/424353 | 01:23 |
| notmyname | a long time ago, there was some work to get swift working under apache/mod_wsgi. I have no idea if it still works. anyone know of anyone deploying that way (or reasons why it wouldn't work)? | 01:24 |
| clayg | notmyname: hadas documented it -> http://docs.openstack.org/developer/swift/apache_deployment_guide.html | 01:25 |
| notmyname | we've got a whole docs page: http://docs.openstack.org/developer/swift/apache_deployment_guide.html | 01:25 |
| notmyname | yeah | 01:25 |
| clayg | notmyname: no one broke it on purpose - no one deploys like that | 01:25 |
| notmyname | yeah, that was exactly my thinking :-) | 01:25 |
| clayg | wow look at that - even has an example .wsgi file! | 01:26 |
| *** mvk has quit IRC | 01:26 | |
| notmyname | cool :-) | 01:26 |
| clayg | i wonder if that's when I learned about that crap? I *have* used mod_wsgi in other contexts tho - so maybe I already knew about it | 01:26 |
| notmyname | so the reason for my questions is the proposed openstack goal for running api services under wsgi (https://review.openstack.org/#/c/419706/) | 01:26 |
| patchbot | patch 419706 - governance - OpenStack Pike Community Goal: deploy-api-in-wsgi | 01:26 |
| notmyname | so on the one hand, I think there's nothing to do in swift | 01:27 |
| clayg | specifically apache mod_wsgi? | 01:27 |
| notmyname | on the other, that document is rather apache-focussed | 01:27 |
| clayg | why not gunicorn? with nginx proxy_pass? | 01:27 |
| clayg | i'm sure the same asnwer is right for all projects and all operations teams everywhere | 01:27 |
| notmyname | it doesn't explicitly say "thou shalt use apache", but it does basically say "yeah, other stuff, but apache apache apache" | 01:27 |
| clayg | it's good to have a goal! people who hate themselves love apache web server! | 01:28 |
| notmyname | for that matter, a wsgi app running under eventlet's web server ;-) | 01:28 |
| openstackgerrit | Merged openstack/python-swiftclient: Make functests py3-compatible https://review.openstack.org/360135 | 01:28 |
| clayg | notmyname: acctually now that I think about it - EC sort of knew it only worked under eventlet.wsgi.sever | 01:29 |
| notmyname | oh yeah? | 01:30 |
| *** chosafine has joined #openstack-swift | 01:30 | |
| notmyname | because of the 100-continue shenanigans? | 01:30 |
| clayg | I think sam put something on a ML somewhere - someone said "something something apache proxy?" and sam was like "proxy is fine, only backend web services?" not sure if there was an explicit acknowledgement that we knew we were breaking deployments of backend services behind apache | 01:30 |
| notmyname | sure sure, punt the hard questions to sam ;-) | 01:31 |
| clayg | but reading the docs it seems like maybe that *was* supported - and probably still tries to work unless someone calls send_continue_headers or whatever that method on the wsgi_input is called | 01:31 |
| clayg | because yeah - basically no wsgi server is going to have that | 01:31 |
| clayg | we made that up | 01:31 |
| notmyname | wow. our proxy<->storage multipart protocol is just so awesome. | 01:32 |
| clayg | it's *so* awesome | 01:32 |
| clayg | zaitcev is going to fix it tho | 01:33 |
| notmyname | request.environ['wsgi.input'].send_hundred_continue_response() | 01:33 |
| clayg | gunna be the PUTPUT protocol | 01:33 |
| clayg | or PUTPOST maybe | 01:33 |
| clayg | the point is it'll be HTTP again | 01:33 |
| clayg | and less MIME | 01:33 |
| clayg | and moar apache! | 01:33 |
| clayg | notmyname: yeah any code that does that only runs under eventlet.wsgi.server | 01:34 |
| clayg | every other wsgi server will just attributeerror | 01:34 |
| notmyname | well, we're not *breaking* http, right? I mean, the spec does allow headers on the 100 continue responses. (note, this is not an argument saying our made-up protocol is good) | 01:34 |
| notmyname | it's just that we're using non-common corner cases of the spec | 01:34 |
| clayg | notmyname: yeah - which in general is a bad idea - but less bad if you control both ends of the socket | 01:35 |
| notmyname | totally | 01:36 |
| clayg | but it's not the headers that are breaking - the sending another 100 Continue after sending the first 100 Continue is *probably* a violation of the spec that no one ever wrote down - but the implementation of that in eventlet.wsgi is acctually broken | 01:36 |
| notmyname | someday, we'll just write some completely non-http protocol that is just for us | 01:36 |
| clayg | gRPC! | 01:36 |
| notmyname | :-) | 01:36 |
| *** openstack-swift has joined #openstack-swift | 01:36 | |
| clayg | i swear i'm going to get back to that before the PTG :\ | 01:36 |
| clayg | but I can't even get through this stupid hashes stuff to fix some of the nasty EC reconstructor shit I stepped in recently :'( | 01:37 |
| notmyname | yeah, I think we could argue (amongst ourselves, or--even better--on the openstack ML) about what's broken vs what's undocumented vs what's allowed. but the point is, we're using stuff that, even if allowed, is not common and therefore not widely implemented | 01:37 |
| notmyname | which brings me back to the wsgi-app-goal thing. if it's "write a wsgi app" then I think we're ok. if it's "totally fine under apache" then I think we'll have some issues | 01:38 |
| notmyname | but I really don't want to start a ML thread^Wflamewar about the right interpretation of all the RFCs | 01:39 |
| clayg | http://lists.openstack.org/pipermail/openstack/2014-August/008923.html | 01:39 |
| clayg | notmyname: well - again - I think the proxy can still run under apache | 01:39 |
| clayg | and honestly - the proxy is the part were we don't control both ends of the socket - so we never would have done something like this anyway | 01:40 |
| notmyname | sure | 01:40 |
| notmyname | ok, I think I have what i need for a gerrit review. thanks | 01:40 |
| clayg | notmyname: there's other daemons in openstack with a myrid of communications channels - but all the public apis are rest/http/wsgi | 01:41 |
| clayg | I think having all the public interfaces behind something ops can stand behind from a load-balancing/socket-hygine/ddos-prevention stand point is pretty friendly | 01:42 |
| *** openstack-swift has quit IRC | 01:42 | |
| notmyname | the "server must be wsgi" is another weird thing, since wsgi is only a python thing. granted, I don't think there are any non-python api endpoints, but still... (I won't be raising this argument) | 01:42 |
| notmyname | isn't the fact that swift takes over the whole path a problem for load balancing with different services? | 01:43 |
| clayg | we just happen to have some intra-service communication that looks "wsgi-ish" but can't run behind mod_wsgi - that shouldn't be a huge problem from an architecturual standpoint anymore that someone whoes internal daemons talk over rabbitmq or some other python<- <tcp> ->python protocol | 01:43 |
| clayg | notmyname: I don't follow - if you're doing *all* your layer3/4 stuff in your lb solution you don't really that swift-proxy-server just starts listening on a socket | 01:46 |
| clayg | notmyname: if you do *all* your layer3/4 stuff in apache - hopefully can't start the proxy under apache's mod_wsgi | 01:47 |
| clayg | notmyname: if you're not doing any layer3/4 stuff you probably don't expose swift the public | 01:47 |
| notmyname | ah, right, I see. I was thinking of having one apache/whatever instance under domain.com, routing domain.com/swift one place and domain.com/nova to another | 01:47 |
| clayg | notmyname: if you *want* to do layer3/4 stuff inbetween swift services ... that's an interesting use-case ;) | 01:47 |
| notmyname | but...can apache be used as an IP router? is that a thing? | 01:47 |
| notmyname | isn't http all at layer 7 and isn't apache "just" an http server? | 01:48 |
| notmyname | ah, my misreading. you're not saying use apache as a LB | 01:48 |
| notmyname | I was getting that from the proposed goal | 01:49 |
| clayg | well ... apache does all kind of stuff | 01:49 |
| notmyname | yeah, if you've just got one common LB for all the cloud stuff, that's cool. send all the swift traffic to the proxy pool | 01:49 |
| clayg | is the main point of the goal just to rehome the URL's? | 01:49 |
| notmyname | I don't think so? | 01:50 |
| clayg | apache is definately more than a webserver - and it *can* be used do proxy_pass kinda stuff so yeah you can probably use it for load balancing - not 100% that's what I was talking about | 01:50 |
| notmyname | no I don't think it was what you were saying | 01:50 |
| clayg | as far as the tcp layers - i always get that question wrong with the SRE folks want to interview me | 01:51 |
| notmyname | I read the goal as ...well... TBH there isn't a "this is why this is important and the problem we fix by implementing it" section | 01:51 |
| clayg | but like everyone cheats routers speak http, ssl terminators speak http | 01:52 |
| clayg | everyone tries to inspect the damn packets and do some kind of value add | 01:52 |
| clayg | everyone wants to have "protection" somewhere in their feature checklist | 01:52 |
| clayg | apache has a *bunch* of them - and it makes enterprise pretty happy - so "just run behind apache" is not a terrible goal | 01:52 |
| notmyname | protection Pro 9000, for Workgroups. Now with Cloud! | 01:53 |
| clayg | notmyname: if it wasn't obvious that we need to do this it wouldn't be a community wide goal (duh) | 01:53 |
| *** mingyu has joined #openstack-swift | 01:54 | |
| clayg | ok, well if I'm going to say any more about it - i'd probably have to read it - which seems counter productive - if I read it I'll comment on it - and that won't help anyone | 01:57 |
| clayg | so - good luck | 01:57 |
| notmyname | lol | 01:58 |
| notmyname | did you know that WSGI is an AM radio station in Tennessee? | 01:58 |
| *** mingyu has quit IRC | 01:58 | |
| notmyname | I wonder if this goal means that station needs to use openstack? | 01:58 |
| clayg | +2 | 01:59 |
| clayg | more whisky is OpenStack - that's what I always say | 02:00 |
| notmyname | speaking of whisky, it's time to go home. | 02:00 |
| clayg | notmyname: so I guess sam never got a response to his ML post about eventlet.wsgi | 02:01 |
| notmyname | I didn't see one | 02:02 |
| clayg | it looks like it went to the global openstack list? maybe he got some private replies? | 02:02 |
| notmyname | torgomatic: did you? ^ | 02:02 |
| clayg | torgomatic: this one http://lists.openstack.org/pipermail/openstack/2014-August/008923.html | 02:02 |
| notmyname | I left a comment and some questions on the goal. we'll see if that kicks off a 100+ message ML thread. I seem to have a knack for that in openstack | 02:02 |
| notmyname | ok, really leaving now. good night | 02:03 |
| *** mingyu has joined #openstack-swift | 02:09 | |
| *** klrmn has quit IRC | 02:14 | |
| *** chosafine has quit IRC | 02:26 | |
| *** jlwhite has quit IRC | 02:26 | |
| *** JimCheung has quit IRC | 02:37 | |
| *** jlwhite has joined #openstack-swift | 02:54 | |
| *** JimCheung has joined #openstack-swift | 03:00 | |
| *** JimCheung has quit IRC | 03:05 | |
| *** chosafine has joined #openstack-swift | 03:06 | |
| *** chosafine has quit IRC | 03:11 | |
| *** sams-gleb has joined #openstack-swift | 03:11 | |
| *** sams-gleb has quit IRC | 03:16 | |
| *** klrmn has joined #openstack-swift | 03:20 | |
| *** jlwhite has quit IRC | 03:26 | |
| *** takashi has joined #openstack-swift | 04:00 | |
| *** jerrygb_ has quit IRC | 04:02 | |
| *** takashi_ has joined #openstack-swift | 04:07 | |
| *** takashi has quit IRC | 04:07 | |
| *** takashi_ is now known as takashi | 04:08 | |
| *** jerrygb has joined #openstack-swift | 04:09 | |
| *** jerrygb has quit IRC | 04:10 | |
| *** dmorita has quit IRC | 04:13 | |
| *** jlwhite has joined #openstack-swift | 04:26 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:30 | |
| *** psachin has joined #openstack-swift | 04:39 | |
| *** winggundamth has joined #openstack-swift | 04:42 | |
| *** klrmn has quit IRC | 04:52 | |
| *** jerrygb has joined #openstack-swift | 05:11 | |
| *** sams-gleb has joined #openstack-swift | 05:14 | |
| *** jerrygb has quit IRC | 05:16 | |
| *** sams-gleb has quit IRC | 05:19 | |
| *** abhitechie has joined #openstack-swift | 05:36 | |
| openstackgerrit | Tone Zhang proposed openstack/swift: IO Priority support on the AArch64 architecture https://review.openstack.org/423783 | 05:41 |
| *** ppai has joined #openstack-swift | 05:43 | |
| *** dmorita has joined #openstack-swift | 05:44 | |
| *** dmorita has quit IRC | 05:48 | |
| clayg | the point of raising OSError when arch != 'the one and only arch we acctually test the feature on' wasn't about being too lazy to copy some list/dict from some crap we cribed off the internet - it was about being honest on what we *support* - what we *maintain* - FOSS means we don't have to block you if you need another number there - but just because you *can* return a different value doesn't mean upstream needs to "maintain" | 06:12 |
| clayg | mattoliverau: torgomatic: timburke: did I miss some context not in the review - why is everyone so keen on this patch? is it just "interesting" because linux headers - or do we have some roadmap for low power cpus - or is it just more "nice and friendly" to say "yes" | 06:15 |
| clayg | I'm happy to see it merge (we have lots of crazy hacks for random platforms we don't *really* support) - but it *really* looked like this cats problem was "I get a test error" not "I need to have this ionice feature work on this Swift deployment I'm doing on ARM" | 06:17 |
| mattoliverau | clayg: my opion is if someone wants to get swift working on ARM then we shouldn't be a blocker. Good point about not being able to test it and support it though | 06:18 |
| clayg | is he trying to "get swift working on ARM" - how is that going? | 06:18 |
| clayg | it looked like he was "trying to get swift's unittests passing on ARM" | 06:19 |
| psachin | clayg: Can you please review https://review.openstack.org/#/c/406012/ when you get time | 06:19 |
| patchbot | patch 406012 - swift - Fix swift-get-nodes arg parsing for missing ring | 06:19 |
| mattoliverau | well so far, he has run it on ARM, and on the first tox failure he's submitted a patch to fix.. that's a great start ;) | 06:19 |
| clayg | psachin: you fixed the thing!? | 06:19 |
| psachin | clayg: yep | 06:19 |
| clayg | mattoliverau: ok, fair enough - I agree we want to be supportive - for all I know there's a lot of sucess to be had using ionice options in swift on ARM processors! | 06:20 |
| mattoliverau | We might end up with little swift proxies on enbedded devices everywhere.. can your fridge speak swift, yes and it's also apart of my homes swift cluster ;P | 06:22 |
| clayg | psachin: well crap - I had some unsubmitted draft comments on the latest patch set from I don't know when :'( | 06:23 |
| clayg | psachin: i'm not sure what my status was there - i bet it's awesome | 06:23 |
| clayg | mattoliverau: lol - yeah awesome definately going to need to ioprio those fridge proxies | 06:24 |
| mattoliverau | lol, yeah, "what do you mean my icecream is melting.. I was only trying to get a listing from that sharded container" | 06:25 |
| psachin | clayg: Yes. Most things are sorted out(like you figured out issue in assertRaisesMessage(). I think the changes are ready to merge now | 06:26 |
| *** winggundamth has quit IRC | 06:28 | |
| *** takashi has quit IRC | 06:34 | |
| *** bkopilov has quit IRC | 06:47 | |
| *** jerrygb has joined #openstack-swift | 07:00 | |
| *** takashi has joined #openstack-swift | 07:04 | |
| *** jerrygb has quit IRC | 07:06 | |
| clayg | psachin: nice work | 07:12 |
| *** sams-gleb has joined #openstack-swift | 07:12 | |
| psachin | clayg: Thank clayg :) | 07:13 |
| psachin | clayg: https://review.openstack.org/#/c/415473/ | 07:14 |
| patchbot | patch 415473 - swift - Proxy server controller tests should be reorganized | 07:14 |
| clayg | mattoliverau: So I guess Zhang maybe *is* trying get swift *running* on AArch64 server - still can't quite tell if he wants to use the ioprio stuff - but it sounds like he tried his best to make sure 30 number was correct | 07:15 |
| clayg | psachin: holy smokes! look at the line count on that diff - nice work man! | 07:15 |
| psachin | It took almost 1.5 days. But I feel it can be more optimized once you review the changes. I'm egarly waiting for this to be reviews | 07:17 |
| psachin | *reviewed* | 07:17 |
| *** ChubYann has quit IRC | 07:18 | |
| clayg | test/unit/proxy/test_server.py | 2186 +-------------------------------------------- | 07:23 |
| clayg | ^ so needed to happen! | 07:23 |
| clayg | interesting, `nosetests swift/test/unit/proxy/` - branch: Ran 801 tests in 69.972s - master: Ran 772 tests in 64.175s | 07:28 |
| *** oshritf has joined #openstack-swift | 07:32 | |
| jcaron | Hi, does anyone have some news about Proxy-fs ? (http://zaitcev.livejournal.com/233432.html) like is it going to be open source ? any release date ? | 07:33 |
| clayg | `rm .coverage; nosetests swift/test/unit/proxy/controllers/test_obj.py --with-coverage --cover-package swift` -> branch: swift/proxy/controllers/obj.py 1112 55 95% ; master: swift/proxy/controllers/obj.py 1112 97 91% | 07:33 |
| clayg | 95% is bigger! | 07:33 |
| *** mvk has joined #openstack-swift | 07:35 | |
| clayg | zaitcev is now my favorite blogger | 07:36 |
| *** oshritf_ has joined #openstack-swift | 07:36 | |
| *** tesseract has joined #openstack-swift | 07:37 | |
| mattoliverau | jcaron: if only there was an SwiftStack employee around.. oh hi clayg :p | 07:37 |
| clayg | i'm trying to find some public materials to reference - but apparently we're being close lipped? | 07:38 |
| mattoliverau | jcaron: all I know is swiftstack is working on it, and yes I believe their opensourcing it. :) | 07:38 |
| *** JimCheung has joined #openstack-swift | 07:38 | |
| clayg | basically there hasn't been a public annoucement since Austin - which is weird considering the internal progress - which makes me think I need to check with Mario or risk the wrath of stealing his thunder ;) | 07:38 |
| mattoliverau | As to when I'm not sure, but if clayg can't answer then I'm sure someone can when there online tomrrow. | 07:38 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add support to increase object ring partition power https://review.openstack.org/337297 | 07:39 |
| *** oshritf has quit IRC | 07:39 | |
| *** mvk has quit IRC | 07:40 | |
| mattoliverau | cschwede: cool a new patch, I'll put it on my list to review first thing | 07:40 |
| mahatic | clayg: I think you missed a link in your commin msg here - patch 424393 | 07:41 |
| patchbot | https://review.openstack.org/#/c/424393/ - swift - Break less abstractions in test_auditor | 07:41 |
| jcaron | Ok good to know there is some progress, thanks ! | 07:41 |
| mahatic | were you referring to existing improvements to test_auditor by acoles_ | 07:42 |
| clayg | mahatic: yeah looks like it | 07:42 |
| *** cbartz has joined #openstack-swift | 07:42 | |
| clayg | oh... acctually i was going to make a comment in the commit message about _hash_suffix - but ended up changing the code and then leaving that comment | 07:42 |
| clayg | i just need to drop the [1] - there's no additional references to be made | 07:42 |
| clayg | mahatic: thanks! | 07:42 |
| *** JimCheung has quit IRC | 07:43 | |
| clayg | jcaron: yeah go to swiftstack.com find something that says "contact" and demand more information! | 07:43 |
| clayg | (I already dropped a note a product/marketing folks tho) | 07:43 |
| jcaron | I will, thx :) | 07:44 |
| mahatic | clayg: I see, thanks | 07:44 |
| *** mvk has joined #openstack-swift | 07:50 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Break less abstractions in test_auditor https://review.openstack.org/424393 | 08:02 |
| *** oshritf_ has quit IRC | 08:02 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** oshritf_ has joined #openstack-swift | 08:05 | |
| *** hseipp has joined #openstack-swift | 08:12 | |
| *** openstackgerrit has joined #openstack-swift | 08:12 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix inline tempurl/formpost signature examples https://review.openstack.org/335044 | 08:12 |
| *** CowboyPride has joined #openstack-swift | 08:20 | |
| CowboyPride | Good morning from my corner of globe. Anyone around at this time? | 08:21 |
| *** takashi has quit IRC | 08:24 | |
| *** rledisez has joined #openstack-swift | 08:32 | |
| mattoliverau | CowboyPride: there usually are some folks. Seems to be a little quiet tonight. It's my evening but around if pinged. | 08:35 |
| CowboyPride | mattoliverau: Cool I'm just trying to verfiy that a node has caught up after finding out that's volumes were not mounted but it was part of a swift cluster. There is concern on my part about data consistency and avalabiity due to the differences in the ammount of data stored. | 08:41 |
| CowboyPride | http://pastebin.com/nyHQ3GVZ | 08:41 |
| CowboyPride | I inherited this swift cluster from a previous admin. Documenation is lacking and I've been having to figure out everything on my own. | 08:41 |
| CowboyPride | You'll see the deiscripency that causes me concern on the 'df' portion of paste above. | 08:42 |
| CowboyPride | node 5 was complaining about insufficent storage in swift proxy logs, investigation revealed that data volumes were not mounted. node was part of the cluster and taking data but data wasn't able to be written because of volumes not being mounted. | 08:44 |
| mattoliverau | CowboyPride: you can look at recon and see when the last replication pass was complete after the drives where mounted. Once a replication pass has been completed all the data should be synced. | 08:44 |
| CowboyPride | having mounted the volumes, the node is now taking traffic without complains but I'm concerned about data matching up on all nodes. this is a 5 node cluster. | 08:45 |
| *** pcaruana has joined #openstack-swift | 08:45 | |
| CowboyPride | And what of the 'df' output, is it possible to be sync but have such widely diffrent 'df' reports | 08:45 |
| mattoliverau | It may serve stale data for the nodes its primary for, if there was nothing on the drives then one of the other primaries will answer | 08:46 |
| mattoliverau | CowboyPride: yeah it's possible, as not all nodes will store same partitions. Generally they'll be similar statistically speaking | 08:46 |
| CowboyPride | the other issue that is concerning for me is that node 5 was left in a partial upgraded state. | 08:47 |
| CowboyPride | node 5 has a newer kernel and newer swift version | 08:47 |
| ahale | great faith in replication matt :) maybe it might take longer if there were loads of connections , rsync limits etc. i'd take a look at the replicator logs, they have a useful status line with number of parts on a machine, can compare against the ring part assignment counts | 08:47 |
| CowboyPride | I need to bring the others up to the same version now but until till I can vett it. | 08:47 |
| ahale | or it could be something like node5 is right and the rest have a load of quarantine | 08:49 |
| CowboyPride | http://pastebin.com/eGf2JePf | 08:50 |
| CowboyPride | ahale: good thinking... | 08:51 |
| CowboyPride | dang. swift-recon is missing from node 5. I've been doing all reports from the proxy node. | 08:52 |
| CowboyPride | checking syslogs now. | 08:56 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add support to increase object ring partition power https://review.openstack.org/337297 | 08:56 |
| ahale | hm, used to have a tool, but i dont think its public, that used to get an idea of out-of-place partitions. it basically walked each drives objects dir, checked against ring and gave a % of partitions that shouldnt be on each box | 08:57 |
| *** oshritf__ has joined #openstack-swift | 08:57 | |
| *** oshritf_ has quit IRC | 08:58 | |
| CowboyPride | Interesting.... | 08:59 |
| CowboyPride | http://pastebin.com/S7gTBjm0 | 08:59 |
| CowboyPride | I found the issue I think now how to fix it. | 08:59 |
| CowboyPride | See paste above. | 08:59 |
| ahale | cool ! | 09:00 |
| *** jerrygb has joined #openstack-swift | 09:02 | |
| mattoliverau | ahale: lol, I'm always an optimist :) | 09:02 |
| CowboyPride | I thing figured it out... rsync is dead on node5 | 09:02 |
| ahale | :) | 09:03 |
| clayg | reminds me of a discussion recently on the openstack general list -> http://lists.openstack.org/pipermail/openstack/2017-January/018354.html | 09:05 |
| clayg | gotta monitor those handoff parts | 09:05 |
| openstackgerrit | Merged openstack/swift: Fix race in new partitions detecting new/invalid suffixes. https://review.openstack.org/418690 | 09:05 |
| CowboyPride | I think it's working.. seeing lots of replication messages without failure on node 1 now. | 09:07 |
| clayg | but I guess some good ol basic connectivity monitoring or daemon monitoring or some log aggregation would have also caught that - live an learn I 'spose | 09:07 |
| clayg | no one has it all in place out the gate | 09:07 |
| *** bkopilov has joined #openstack-swift | 09:08 | |
| clayg | ... or checking "failures" in recon :\ | 09:08 |
| *** jerrygb has quit IRC | 09:08 | |
| clayg | i think i'm out :D | 09:08 |
| ahale | oh interesting mail | 09:08 |
| ahale | there was another one i meant to reply to too | 09:08 |
| clayg | ahale: right... something about global clusters and dispersion report | 09:09 |
| clayg | where is that one | 09:09 |
| ahale | ahh yeah the ring copy one | 09:09 |
| clayg | http://lists.openstack.org/pipermail/openstack/2017-January/018330.html | 09:09 |
| ahale | oh hadnt seen that either | 09:10 |
| ahale | heh - i would personally run dispersion populate after each confirmed real loss of a partition | 09:10 |
| clayg | not sure if I want to laugh or cry | 09:13 |
| *** oshritf_ has joined #openstack-swift | 09:16 | |
| openstackgerrit | Karen Chan proposed openstack/swift: Return versioning name in get_container_info https://review.openstack.org/424527 | 09:19 |
| *** oshritf__ has quit IRC | 09:20 | |
| *** jcaron has left #openstack-swift | 09:26 | |
| *** jordanP has joined #openstack-swift | 09:27 | |
| *** mvk has quit IRC | 09:41 | |
| *** mingyu has quit IRC | 09:43 | |
| *** mingyu has joined #openstack-swift | 09:45 | |
| ahale | oh yeah that was it, not so much the copying rings about part, but the idea of lots of ring changes every couple of hours was slightly terrifying. idk how much people have talked before about how to decide a strategy for ring changes.. lots of small, few large, increasing weights or slam it in at full weight | 09:48 |
| *** mingyu has quit IRC | 09:48 | |
| *** mingyu_ has joined #openstack-swift | 09:48 | |
| *** mingyu_ has quit IRC | 09:52 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 09:53 |
| clayg | there was period where ring balancing had some really annoying behaviors of flopping parts around that made lots of gradual ring chnages exceptionally painful | 09:53 |
| clayg | lots of thing would move - placement/balance would only get slightly better | 09:53 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 09:53 |
| clayg | or you'd increase weight on some devices by a few points and they would gain 100 part-replicas but also loose 20 (wtf?) | 09:54 |
| ahale | lol | 09:54 |
| clayg | there is a LOT less of that non-sense on master since about a year ago (IIRC around the same time cloud files stopped taking updates from master for the rings) | 09:55 |
| clayg | we still do gradual changes - I don't ever like to see rebalance progress telling me it's 100's of hours out | 09:56 |
| clayg | I think OVH does as well | 09:56 |
| *** CowboyPride has quit IRC | 09:56 | |
| ahale | yeah, not sure where they are at anymore, I rememebr looking at those changes | 09:56 |
| clayg | but sometimes we have operational stuff where we want to just "make the weights where there done - do the rebalance - push rings - and wait" | 09:56 |
| ahale | it still kinda is 100's of hours out from where you want to get though really - and yeah that | 09:57 |
| clayg | ahale: yeah it's all process - like we won't (currently) inturrupt a running rebalance cycle with a ring_push if you want to "soft fail a device" (like set the weight to 0, that just get's enqueued in the next automatic/gradual weight change) - but like people get mad if that is not for another 3 days ;) | 09:59 |
| clayg | i'm sure just pushing out rings and inturruping the replication cycle is fine *too* - but there's some weird side effect of how that's reported in the recon data - like it'll say last_cycle_finished when the new rings come down even tho it didn't really *finish* - and of course it'll loose track of which partitions have done "update" - which is distrating when you *really* just want them to be doing "update_deleted" | 10:02 |
| clayg | or sync/revert if you're in EC-landia | 10:02 |
| rledisez | clayg, ahale; we used to do gradual increase/decrease when we were running with rsync. now that we use ssync, it's less important. we still do it, but with bigger step | 10:06 |
| rledisez | problem with rsync is the cost of trying to replicate a partition (all readdir/stats), with result being sometimes all rsync slot are used | 10:06 |
| rledisez | coest with ssync is really lower (only listdir) | 10:07 |
| rledisez | so, we can afford more pressure on the replication system, even if it fails, it's invisible for user (no more impact on latency) | 10:07 |
| clayg | why not just increase rsync max_connections then? The REPLICATE verb is just as decoupled from SSYNC request with sync_method = ssync? | 10:08 |
| clayg | rledisez: I still find it *fascinating* that you use ssync for replicated objects - none of the testing I've ever done made it look *remotely* attractive compared to rsync for large rebalances with *either* large objects *or* dense suffixes | 10:09 |
| clayg | with EC we *have* to use it of course - but my beef there is mostly with the reconstructor - acoles_ has done lots of amazing work to to make the SSYNC protocol more robust | 10:10 |
| rledisez | it's all about "time to first byte". with rsync, our customers complains about increase latency (and they are right, we see it in statsd metrics). with ssync, no impact at all, totally smooth. of course, it's slower, but we can deal with that (we just order hardware sooner) | 10:11 |
| clayg | but everytime I look at yeild_hashes and the UPDATE portion I think it can't *possibly* be as efficient as something like rsync | 10:11 |
| clayg | yeah that's fascinating - and aweomse - it makes sense - if ssync it slower it uses less resources so more resources available to object-server | 10:13 |
| rledisez | that's really the stat() done by rsync that makes the difference. if ssync were stat()ing files, it would make no sense | 10:14 |
| clayg | it's an akward way to tune/balance/alloate the available resources - but if the results are satisfactory | 10:14 |
| clayg | I seem to fight "can you make it rebalance faster" more ofthen that "can you make rebalance not hurt my ttfb so much" | 10:14 |
| ahale | yeah it is absolutely all about time to start response, that pretty much decides everything i would do | 10:15 |
| rledisez | it may be depending it's public or private cloud. on private you know when you can run rebalance (night or week end) while on public you have to give best perf all the time beause you don't know what your customers are doing and when | 10:16 |
| *** mvk has joined #openstack-swift | 10:16 | |
| clayg | I think more of my clusters are the "deep and cheap dumping grounds" and less the "public hawt web content" variety - even the people doing video stuff have a cdn sorta situation for serving content | 10:16 |
| clayg | it's still a really important consideration moving forward - if we can *finally* get suffix matching and data path in the same protocol it's important to remember "make rebalance as blazingly fast as possible" may not be the whole solution if it also "sucks up all the iops away from the object-server" | 10:20 |
| openstackgerrit | Christopher Bartz proposed openstack/swift: ISO 8601 timestamps for tempurl https://review.openstack.org/422679 | 10:22 |
| openstackgerrit | Christopher Bartz proposed openstack/python-swiftclient: ISO 8601 timestamps for tempurl https://review.openstack.org/423377 | 10:25 |
| *** sileht has quit IRC | 10:27 | |
| *** sileht has joined #openstack-swift | 10:27 | |
| ahale | yeah thats basically what I was doing with hbird replication with some messed up clusters. adjusting the balance of disk_limit and replication_disk_limit and i think ms_per_part or something? along with the out-of-place stuff i mentioned and statsd, to target the worst boxes , trading a little start response for some replication performance, but not too much | 10:32 |
| *** sams-gleb has quit IRC | 10:42 | |
| openstackgerrit | Jan Krcmar proposed openstack/python-swiftclient: Add kerberos http authentication https://review.openstack.org/424581 | 10:58 |
| *** sams-gleb has joined #openstack-swift | 11:00 | |
| *** jerrygb has joined #openstack-swift | 11:04 | |
| *** ganders has joined #openstack-swift | 11:08 | |
| *** jerrygb has quit IRC | 11:09 | |
| *** kei_yama has quit IRC | 11:44 | |
| *** catintheroof has joined #openstack-swift | 12:16 | |
| *** mingyu has joined #openstack-swift | 12:53 | |
| *** bkopilov has quit IRC | 12:55 | |
| *** mingyu has quit IRC | 12:58 | |
| *** jcaron has joined #openstack-swift | 12:59 | |
| *** jerrygb has joined #openstack-swift | 13:05 | |
| *** jerrygb has quit IRC | 13:10 | |
| *** vint_bra has joined #openstack-swift | 13:12 | |
| *** abhitechie has quit IRC | 13:15 | |
| *** jlwhite has quit IRC | 13:27 | |
| *** Jeffrey4l_ is now known as Jeffrey4l | 13:42 | |
| *** ppai has quit IRC | 13:47 | |
| *** vint_bra has quit IRC | 13:56 | |
| *** vint_bra has joined #openstack-swift | 13:59 | |
| *** _JZ_ has joined #openstack-swift | 14:01 | |
| *** jordanP has quit IRC | 14:02 | |
| *** jordanP has joined #openstack-swift | 14:02 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 14:05 | |
| *** sanchitmalhotra has quit IRC | 14:06 | |
| *** sanchitmalhotra1 is now known as sanchitmalhotra | 14:06 | |
| *** mingyu has joined #openstack-swift | 14:23 | |
| *** zaitcev has joined #openstack-swift | 14:26 | |
| *** ChanServ sets mode: +v zaitcev | 14:26 | |
| *** mingyu has quit IRC | 14:28 | |
| *** vinsh_ has quit IRC | 14:38 | |
| *** JimCheung has joined #openstack-swift | 14:39 | |
| *** JimCheung has quit IRC | 14:43 | |
| *** jerrygb has joined #openstack-swift | 14:44 | |
| *** SkyRocknRoll has quit IRC | 14:45 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:47 | |
| *** jerrygb_ has joined #openstack-swift | 14:47 | |
| *** jerrygb__ has joined #openstack-swift | 14:49 | |
| *** psachin has quit IRC | 14:50 | |
| *** jerrygb has quit IRC | 14:51 | |
| *** jerrygb_ has quit IRC | 14:52 | |
| openstackgerrit | Jan Krcmar proposed openstack/python-swiftclient: Add kerberos http authentication https://review.openstack.org/424581 | 14:57 |
| *** jerrygb has joined #openstack-swift | 15:01 | |
| tdasilva | good morning | 15:05 |
| *** jerrygb__ has quit IRC | 15:05 | |
| tdasilva | cool discussion between clayg, ahale and rledisez | 15:05 |
| *** cdelatte has joined #openstack-swift | 15:13 | |
| *** vinsh has joined #openstack-swift | 15:16 | |
| *** vint_bra has quit IRC | 15:22 | |
| *** vint_bra has joined #openstack-swift | 15:23 | |
| *** sams-gleb has quit IRC | 15:29 | |
| *** bkopilov has joined #openstack-swift | 15:29 | |
| *** sams-gleb has joined #openstack-swift | 15:29 | |
| *** dmorita has joined #openstack-swift | 15:32 | |
| *** sams-gleb has quit IRC | 15:33 | |
| *** dmorita has quit IRC | 15:37 | |
| *** jlwhite has joined #openstack-swift | 15:41 | |
| *** sams-gleb has joined #openstack-swift | 15:47 | |
| *** mvk has quit IRC | 15:49 | |
| *** briancurtin has left #openstack-swift | 15:56 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 16:01 | |
| *** sanchitmalhotra has quit IRC | 16:02 | |
| *** sanchitmalhotra1 is now known as sanchitmalhotra | 16:02 | |
| *** vinsh has quit IRC | 16:04 | |
| *** vinsh has joined #openstack-swift | 16:06 | |
| *** vinsh has quit IRC | 16:06 | |
| *** klrmn has joined #openstack-swift | 16:06 | |
| *** dmellado has quit IRC | 16:07 | |
| *** vinsh has joined #openstack-swift | 16:07 | |
| *** jerrygb has quit IRC | 16:08 | |
| *** jerrygb has joined #openstack-swift | 16:09 | |
| *** klrmn has quit IRC | 16:11 | |
| *** klrmn has joined #openstack-swift | 16:15 | |
| *** pcaruana has quit IRC | 16:16 | |
| *** klrmn has quit IRC | 16:19 | |
| *** oshritf_ has quit IRC | 16:31 | |
| *** chsc has joined #openstack-swift | 16:32 | |
| *** chsc has joined #openstack-swift | 16:32 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: func tests for tempurl iso_8601 timestamps https://review.openstack.org/424732 | 16:32 |
| *** Renich has joined #openstack-swift | 16:37 | |
| openstackgerrit | Christopher Bartz proposed openstack/swift: ISO 8601 timestamps for tempurl https://review.openstack.org/422679 | 16:38 |
| rledisez | hi tdasilva, thx for including me on the post_as_copy deprecation discussion | 16:41 |
| rledisez | related to that i'm observing a weird behavior, not sure if anyone already saw it | 16:41 |
| rledisez | default conf set object_post_as_copy in copy middleware conf | 16:41 |
| patchbot | Error: 'supybot.conf' is not a valid configuration variable. | 16:41 |
| rledisez | but it seems it is not accepted by proxy-server. it only read it from [DEFAULT]. do you have the same behavior? | 16:42 |
| tdasilva | rledisez: no, i've never seen that... | 16:44 |
| tdasilva | rledisez: just to be clear, you are saying that if you set it like this: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L809,L816 | 16:44 |
| tdasilva | then you can't start the proxy server? | 16:44 |
| *** ganders has quit IRC | 16:46 | |
| rledisez | no, i mean whatever i set for object_post_as_copy in [filter:copy], it is not honored. only what i set in [default] section is used. on current stable version, |filter:copy] object_post_as_copy = true does nothing | 16:46 |
| patchbot | Error: Spurious "]". You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands. | 16:46 |
| *** oshritf_ has joined #openstack-swift | 16:47 | |
| *** dmellado has joined #openstack-swift | 16:48 | |
| *** cbartz has left #openstack-swift | 16:48 | |
| rledisez | i'm going to open a bug on this, because if I hadn't check on object server, setting false like showed on sample config file would have done nothing | 16:49 |
| *** hseipp has quit IRC | 16:49 | |
| *** acoles_ is now known as acoles | 16:50 | |
| tdasilva | rledisez: i have not noticed that, and with the symlinks patch i've been testing that pretty often to test the interatction between copy and symlink middleware | 16:50 |
| *** ganders has joined #openstack-swift | 16:50 | |
| tdasilva | rledisez: do you remove the option from DEFAULT when adding it to the copy filter section? | 16:53 |
| *** dmellado has quit IRC | 16:53 | |
| rledisez | tdasilva: ok, got it. it's because it is auto-added in the pipeline. if i set it explicitely in the pipeline, it behave normaly | 16:54 |
| rledisez | tdasilva: fun one :) | 16:54 |
| rledisez | my pipeline need to be backward compatible with 2.7 right now as we did not upgraded every clusters, so i don't add the copy middleware yet | 16:56 |
| *** dmellado has joined #openstack-swift | 16:59 | |
| *** tqtran has joined #openstack-swift | 17:00 | |
| tdasilva | rledisez: oh i see | 17:02 |
| tdasilva | rledisez: re patch 417209 it would be great to get some ops voice on there... | 17:02 |
| patchbot | https://review.openstack.org/#/c/417209/ - swift - Default object_post_as_copy to False | 17:02 |
| *** bkopilov has quit IRC | 17:05 | |
| *** oshritf_ has quit IRC | 17:06 | |
| rledisez | tdasilva: for now, not much feedbacks as we don't use fast post where there is no container-sync (it's definitively on our todolist after upgrade to 2.12). and as POST requests are almost non-existant, no customer complaints | 17:08 |
| rledisez | tdasilva: but never had any trouble where we use it, so no opposition :) | 17:08 |
| tdasilva | rledisez: ack | 17:09 |
| *** cdelatte has quit IRC | 17:11 | |
| mahatic | do we already have a patch for post_as_copy deprecation? did I miss it? | 17:18 |
| *** JimCheung has joined #openstack-swift | 17:18 | |
| tdasilva | mahatic: patch 417209 | 17:19 |
| patchbot | https://review.openstack.org/#/c/417209/ - swift - Default object_post_as_copy to False | 17:19 |
| mahatic | tdasilva: thanks! | 17:20 |
| tdasilva | mahatic: aka: kill-it-with-fire-from-orbit | 17:21 |
| mahatic | lol | 17:22 |
| mahatic | maybe make it the commit title, folks might miss it otherwise ;) | 17:24 |
| tdasilva | lol | 17:26 |
| *** arch-nemesis has joined #openstack-swift | 17:28 | |
| *** cbartz has joined #openstack-swift | 17:37 | |
| *** cdelatte has joined #openstack-swift | 17:38 | |
| *** mvk has joined #openstack-swift | 17:44 | |
| *** rledisez has quit IRC | 17:47 | |
| *** klrmn has joined #openstack-swift | 17:47 | |
| *** dmorita has joined #openstack-swift | 17:47 | |
| *** jordanP has quit IRC | 17:52 | |
| *** SkyRocknRoll has quit IRC | 17:53 | |
| *** oshritf_ has joined #openstack-swift | 17:56 | |
| *** cbartz has quit IRC | 18:00 | |
| *** Renich has quit IRC | 18:02 | |
| *** bkopilov has joined #openstack-swift | 18:12 | |
| *** acoles is now known as acoles_ | 18:25 | |
| clayg | weee! | 18:33 |
| *** tesseract has quit IRC | 18:39 | |
| *** oshritf_ has quit IRC | 18:46 | |
| *** silor has joined #openstack-swift | 18:48 | |
| clayg | how do you tell gerrit to recheck because you think a rebase will cause test failures? | 18:50 |
| jrichli | clayg: you mean posting "recheck no bug" on the patch? that is what we did at some point. I have seen some people just put "recheck", but I dont know if that has worked. | 18:52 |
| clayg | jrichli: thanks! | 18:53 |
| *** dmorita has quit IRC | 19:00 | |
| *** lespaul has joined #openstack-swift | 19:01 | |
| *** dmorita has joined #openstack-swift | 19:02 | |
| *** ganders has quit IRC | 19:14 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Better optimistic lock in get_hashes https://review.openstack.org/419787 | 19:18 |
| *** vint_bra has quit IRC | 19:24 | |
| *** manous has joined #openstack-swift | 19:27 | |
| *** vint_bra has joined #openstack-swift | 19:28 | |
| openstackgerrit | Merged openstack/swift: Fix inline tempurl/formpost signature examples https://review.openstack.org/335044 | 19:29 |
| *** vint_bra1 has joined #openstack-swift | 19:29 | |
| *** vint_bra1 has quit IRC | 19:29 | |
| *** vint_bra has quit IRC | 19:32 | |
| *** karenc_ is now known as karenc | 19:32 | |
| *** vint_bra1 has joined #openstack-swift | 19:32 | |
| *** vint_bra has joined #openstack-swift | 19:34 | |
| *** clu_ has joined #openstack-swift | 19:35 | |
| *** vint_bra1 has quit IRC | 19:36 | |
| torgomatic | notmyname: nope, no private replies to http://lists.openstack.org/pipermail/openstack/2014-August/008923.html | 19:36 |
| notmyname | torgomatic: ack | 19:36 |
| *** ChubYann has joined #openstack-swift | 19:39 | |
| notmyname | torgomatic: this is related to https://review.openstack.org/#/c/419706/ and now I need to figure out how much of the current wording ("Switch devstack jobs to deploy control-plane API services in WSGI with Apache.") will affect swift | 19:41 |
| patchbot | patch 419706 - governance - OpenStack Pike Community Goal: deploy-api-in-wsgi | 19:41 |
| notmyname | so maybe that means the proxy will run under mod_wsgi and the storage nodes with eventlet and everything will be fine | 19:41 |
| notmyname | or maybe "control plane" excludes swift | 19:41 |
| torgomatic | notmyname: exactly my thought | 19:42 |
| zaitcev | we continue getting requests about TLS though. It's not going away. | 19:43 |
| torgomatic | Apache/mod_wsgi isn't gonna stand up to our usage of some of the more obscure corners of HTTP | 19:43 |
| zaitcev | And that means client auth with a certificate. | 19:43 |
| torgomatic | zaitcev: TLS between proxy and storage nodes? | 19:43 |
| zaitcev | yes | 19:43 |
| zaitcev | I think the biggest practical concern is if you have a (intercontinental) VPN and then your VPN somehow gets misconfigured or compromised and then everything is in the clear. Kinda like Google datacenters in Ireland. | 19:44 |
| timburke | zaitcev: those seem like orthogonal issues? | 19:44 |
| notmyname | apache/mod_wsgi isnt' really going to stand up to much of the swift traffic patterns in prod anyway (from the last I've seen on the matter). but devstack isn't prod, so it's probably fine there | 19:45 |
| zaitcev | Yes, well, Apache supports client auth | 19:45 |
| zaitcev | That's why people want it. | 19:45 |
| timburke | or are you thinking of the proxies having client certs for talking to the storage nodes? | 19:45 |
| zaitcev | yes | 19:45 |
| zaitcev | Replicators, too, somehow. | 19:45 |
| zaitcev | Sorry, didn't see your "or ...". What I mean is a storage node in Apache, client is in proxy, client presents its sertificate. Or something like that. I'm not an expert in TLS. | 19:47 |
| zaitcev | We already support proxy in Apache with that initrp() thingie. | 19:47 |
| zaitcev | So far I was able to say "I'm busy with Hummingbird, shoo, shoo" but the hour or reconing draws near | 19:48 |
| notmyname | zaitcev: but we know that storage node running under apache wont' work because we don't have the ability to handle the 100-continue responses there | 19:50 |
| timburke | ...or is it the hour of REPCONNing? | 19:50 |
| zaitcev | you slay me | 19:54 |
| *** arch-nemesis has quit IRC | 19:57 | |
| notmyname | does anyone know off the top of their heads how swift is deployed in devstack? is it under mod_wsgi or is it simply starting the swift process (ie eventlet wsgi server)? | 19:58 |
| notmyname | looks like if an envvar is set, then it will all be deployed under mod_wsgi in apache, including the object server | 20:02 |
| zaitcev | really, huh | 20:02 |
| notmyname | so either that's never used or EC is never attempted with that config | 20:02 |
| notmyname | $SWIFT_USE_MOD_WSGI | 20:04 |
| timburke | notmyname: something tells me they only ever use replicated. i wouldn't surprise me if it was only ever with a single replica | 20:06 |
| notmyname | I don't see any results for grepping for that envvar in any openstack repo I have checked out on my machine | 20:07 |
| torgomatic | in principle, it looks like one could combine nginx and iptables on proxy nodes and storage nodes to wrap connections in client-cert-authenticated TLS | 20:07 |
| torgomatic | notmyname: yeah, I'm gonna guess they don't use crypto or EC with that config. The guess is based mostly on the fact that I haven't heard complaints of everything being broken. | 20:10 |
| notmyname | torgomatic: oh right. crypto needs all that too. not just ec | 20:12 |
| *** silor has quit IRC | 20:13 | |
| clayg | notmyname: I think crypto just needs the mine wrapping - the footers - it doesn't need the commit message - which requires the mid http put response | 20:14 |
| clayg | *mime wrapping | 20:14 |
| notmyname | ah yes | 20:15 |
| timburke | looking at an object.builder from a dsvm run, i see "512 partitions, 1.000000 replicas, 1 regions, 1 zones, 1 devices" | 20:15 |
| torgomatic | clayg: I thought the MIME wrapper needed a 100 Continue header to say it was okay, otherwise the proxy thinks it's gonna get an object with a bunch of MIME in it that shouldn't be there | 20:18 |
| clayg | I think i'm going to *only* dev on swift in a devstack setup for the whole of the PTG - and everytime something annoyes me - I'll just go find some to tell me how to fix and push up a bunch of patches | 20:18 |
| notmyname | FYI the https://review.openstack.org/#/c/369749/ was just approved in the tc meeting | 20:19 |
| patchbot | patch 369749 - governance - Add Pike goal split out tempest plugins | 20:19 |
| clayg | torgomatic: yes you're 100% correct - which has often lead me to assume that probably any request to a object-server behind apache will AttributeError - but the headers in the *first* 100 continue were never really the part that broke HTTP | 20:19 |
| notmyname | I think that gives us the goals of py3 and this one for the pike release | 20:20 |
| notmyname | oops. almost typo'd "pike" as "puke" | 20:20 |
| clayg | it's only when you call that method after already having read from wsgi.input that the implementation in eventlet.wsgi acctually has a bug that makes the bytes on the wire not HTTP anymore -> https://bugs.launchpad.net/swift/+bug/1496636 | 20:21 |
| openstack | Launchpad bug 1496636 in OpenStack Object Storage (swift) "EC: Chunked transfer/commit protocol is *not* HTTP" [Undecided,New] | 20:21 |
| clayg | if you call set_hundred_continue_response before reading from wsgi.input - I think chunk_length is already -1 so it doesn't have the effect of breaking protocol | 20:25 |
| clayg | I think the 100-continue header response was a really cute way to get around the fact we forgot to put a version segment in the url of the object server API for rolling upgrades - but we could have also just down the two-step and been done with it (after upgrading all nodes turn on use_versioned_protocol=True) | 20:28 |
| clayg | and *had* we done that tdasilva would probably be proposing the patch to make it the default *right now* ;) | 20:28 |
| notmyname | clayg: I wish we were smarter back then ;-) | 20:30 |
| timburke | i don't think it's eventlet's fault; we tell the proxy to send the empty chunk at https://github.com/openstack/swift/blob/2.12.0/swift/proxy/controllers/obj.py#L1522 -- what if we just stop doing that? | 20:31 |
| clayg | notmyname: we wern't less smart - we just didn't have the requirement of keeping support for apache mod_wsgi | 20:31 |
| zaitcev | wait what | 20:32 |
| zaitcev | I thought put('') was just a way to flush data and didint' send anthing | 20:32 |
| notmyname | hang on. we do not have the requirement of running storage nodes under mod_wsgi right now, even with that pike goal landing | 20:33 |
| clayg | timburke: I tried a few things when I reported that bug (including yes deleting some of the obviously wrong things we had to do to make the damn thing work) | 20:33 |
| clayg | timburke: IIRC it didn't work, and the fix was not obvious, so I filed the bug | 20:33 |
| timburke | and go ahead and do our transition from DATA_SENT to DATA_ACKED wherever it's already getting done... eh, w/e, clayg's smarter than me, and certainly has investigated this more deeply | 20:33 |
| notmyname | with that goal, we actually don't need to do anything (perhaps other than verify that running the proxy on devstack under mod_wsgi still works, and stop using mod_wsgi for storage nodes) | 20:34 |
| clayg | zaitcev: no, when it gets transformed in the queue... it's in the bug report :P | 20:34 |
| clayg | timburke: I think "doing the right thing" required a fix in eventlet and swift - with no obvious path to make them both atomic on upgrade - so it chicken and egg'd | 20:35 |
| clayg | timburke: i'm not sure it's worth investigating again if zaitcev is going to rewrite everything so we can just use pipelined HTTP requests instead of trying to do req-resp-req-resp in the same HTTP request + MIME wrapped chuned transfer | 20:36 |
| timburke | tru | 20:36 |
| zaitcev | oi, the responsibility | 20:36 |
| timburke | no pressure ;-) | 20:36 |
| clayg | the crypto case is interesting because we don't need req-resp-req-resp we just need "PUT data + moar data" | 20:37 |
| clayg | notmyname: if we don't have an requirement/desire to run object nodes under apache then why do we care if object servers *require* wsgi.input to have a set_hundred_continue_response_headers method? | 20:39 |
| notmyname | clayg: I don't think we do, actually, aside from the sense of ought-ness that our wsgi app only runs under one wsgi server. there are zero requirements currently being placed on swift to run storage nodes under mod_wsgi | 20:40 |
| notmyname | so... *done* | 20:40 |
| clayg | nice | 20:40 |
| clayg | oh - but something devstack EC? | 20:41 |
| notmyname | oh right. yeah, that is likely broken | 20:41 |
| notmyname | the thing I'll be on the lookout for, though, is perception around "oh every openstack project just works under mod_wsgi" or "swift is different because..." etc. just typical openstack stuff | 20:41 |
| clayg | like I'm conccerend someone is saying "you have to run object-servers under apache" because "tough - that doesn't work - nehechxt" | 20:42 |
| clayg | if someone were to say "I'd really *like* to run object-servers under apaache for valid operational reasons" - that's harder because "well shit bro, that doesn't work - what are we gunna do?" | 20:42 |
| zaitcev | so I thought maybe all we need to do is something simple like this - it appears to work in e.g. the requests use the same TCP connection - https://gist.github.com/zaitcev/b5378c18c522bd695f3ac75c3b5b66f6#file-gistfile1-txt-L139 | 20:42 |
| clayg | sort *not* concered on th first one | 20:43 |
| patchbot | *not* concered first on one th | 20:43 |
| timburke | oh patchbot :P | 20:43 |
| notmyname | https://github.com/openstack-dev/devstack/blob/master/lib/swift#L196 | 20:43 |
| clayg | enable_apache_site object-server-${node_number} < probably makes all PUTs to that object-server an AttributeError | 20:45 |
| clayg | notmyname: you gunna test it!? | 20:45 |
| clayg | the gate spins up swift in devstack 1000s of times every day - i bet you could do it if you set your mind to it | 20:46 |
| clayg | less sure anyone knows how to plumb the right options/flags/vars so that method gets' called and doesn't hit some other unrelated provisioning bug | 20:46 |
| clayg | acctually just the thought of it makes me think it'd be easier to just install apaache and set it up on my saio more quickly that I could get it working devstacks mounain of shell code | 20:47 |
| zaitcev | Yessssss, with each such thought you become more servant of packaged services... apt apt apt dnf dnf dnf | 20:48 |
| zaitcev | did I say that out loud | 20:49 |
| clayg | zaitcev: i'm still looking at your gist | 20:49 |
| zaitcev | clayg: there's too much junk in it, but basically let's just POST wth our formerly footer headers. POST accidentially makes it durable, I think... I started adding a feature to it to make so but apparently it already does it anyway | 20:50 |
| zaitcev | juuuust a little more tweaking and no more 100 | 20:50 |
| clayg | zaitcev: so but it *looks* like in that gist it's still doing the MIME stuff? | 20:50 |
| notmyname | zaitcev: https://cdn.meme.am/instances/400x/75002452.jpg ? | 20:50 |
| zaitcev | yes, but | 20:50 |
| zaitcev | notmyname: LOL | 20:51 |
| clayg | zaitcev: also... is there an object server side that makes it so the PUT isn't really idk "for reazly durable/servable" until after the POST | 20:51 |
| zaitcev | details! | 20:51 |
| clayg | like we want two pipelined HTTP requests to make a single object - the proxy dying before sending the finalize POST should have the same behavior of the proxy dying before sending the MIME footers or commit MIME part on EC | 20:52 |
| zaitcev | h | 20:53 |
| notmyname | clayg: is it actually ok to tie pipelined http requests into the same transaction? ie instead of treating each request as unrelated to others? I thought that's why we did the multipart thing. also, I thought pipelining was only for reducing tcp setup times | 20:54 |
| clayg | no more... 100? presumable this protocol will only work after you upgrade your object server to some xyz version - so you either need to do the "use_versioned_protocol = True" trick - or you need to send a new HTTP header with the 100 continue to indicate "I'm a new enough object server I won't freak out if you start talking this new hawtness to me" | 20:54 |
| clayg | notmyname: it's "ok" by like "by the power of vim!!!" but like there's some purity aspects of REST symatnics you bring into it | 20:55 |
| notmyname | clayg: yeah, your point about when the proxy dies is kinda the trick with the whole thing. but yeah, it's just typing ;-) | 20:56 |
| notmyname | but also, I feel like I missed the point of this conversation. what are we solving for here? why are we changing the proxy-object protocol now? | 20:56 |
| clayg | yeah I don't really think the the MIME handling made for significantly less code there | 20:56 |
| zaitcev | yeah I can create some random key that gets stored and then POST only works with a matching key, therefore the transaction. But eh, clearly I didn't think this through. | 20:56 |
| notmyname | oh, right. the security stuff zaitcev mentioned | 20:57 |
| clayg | and *I* was never personally offended but a PUT x-requires-more-information: True | 20:57 |
| *** dmorita has quit IRC | 20:57 | |
| mattoliverau | Morning | 20:58 |
| *** dmorita has joined #openstack-swift | 20:58 | |
| notmyname | zaitcev: you're being asked to provide full TLS for internal traffic? ie instead of something simpler like signed messages | 20:58 |
| zaitcev | x-requires-more-information: 4242424242 | 20:58 |
| clayg | zaitcev: yeah that's the ticket | 20:58 |
| openstackgerrit | Merged openstack/swift: Default object_post_as_copy to False https://review.openstack.org/417209 | 20:58 |
| zaitcev | notmyname: yes, it has to be TLS. purely the paperwork | 20:58 |
| notmyname | zaitcev: ugh | 20:58 |
| notmyname | zaitcev: and stunnel won't work? | 20:58 |
| clayg | zaitcev: I'm also cool if "when the socket get's closed there is *no* request that can reference this data in any form" | 20:59 |
| notmyname | zaitcev: sounds similar to the "you can't have the characters 'm', 'd', and '5' in that order right next to each other anywhere in the code" | 20:59 |
| zaitcev | notmyname: stunnel... probably works. But I don't know. | 20:59 |
| clayg | you guys are jaded | 21:00 |
| *** jerrygb_ has joined #openstack-swift | 21:01 | |
| clayg | i always forget dnf is the new yum | 21:01 |
| clayg | dandified even | 21:01 |
| notmyname | jet-lagged? | 21:02 |
| zaitcev | notmyname: about "why are we changing the proxy-object protocol now", I started on this path because Go cannot do our current protocol without an annoyingly protracted surgery that is not likely to succeed. Please see https://groups.google.com/forum/#!topic/golang-nuts/cR5bWfLroiQ | 21:02 |
| *** dfflanders has joined #openstack-swift | 21:03 | |
| zaitcev | notmyname: coincidentally Clay found some other reason not to like the current protocol, see bug https://bugs.launchpad.net/swift/+bug/1496636 | 21:03 |
| openstack | Launchpad bug 1496636 in OpenStack Object Storage (swift) "EC: Chunked transfer/commit protocol is *not* HTTP" [Undecided,New] | 21:03 |
| zaitcev | was it py3? | 21:03 |
| *** jerrygb has quit IRC | 21:03 | |
| clayg | oh... yeah that's the *general* problem with "HTTP doesn't say we *can't* do this even though no one does ever" | 21:03 |
| notmyname | ah right. I remember you bringing up the golang issue | 21:03 |
| *** jerrygb_ has quit IRC | 21:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 21:04 |
| clayg | but that's sort of a weak argument - i'm tottaly leaverging other factors - if we wanted golang so speak our HTTP++ we could probably do that - but we should fix the protocol bug along the way and ... i'm tired let's go shopping! | 21:05 |
| notmyname | zaitcev: you never got any replies to that golang post, did you? | 21:06 |
| zaitcev | notmyname: didn't receive any replies. | 21:06 |
| *** manous has quit IRC | 21:23 | |
| *** caiobrentano_ has quit IRC | 21:26 | |
| *** caiobrentano_ has joined #openstack-swift | 21:26 | |
| *** caiobrentano_ has quit IRC | 21:30 | |
| *** Jeffrey4l_ has joined #openstack-swift | 21:34 | |
| *** catintheroof has quit IRC | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** catintheroof has joined #openstack-swift | 21:35 | |
| *** catintheroof has quit IRC | 21:35 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix performance regression with hash invalidations https://review.openstack.org/418691 | 22:13 |
| *** sams-gleb has quit IRC | 22:18 | |
| openstackgerrit | Merged openstack/swift: ISO 8601 timestamps for tempurl https://review.openstack.org/422679 | 22:22 |
| *** vint_bra has quit IRC | 22:28 | |
| *** vint_bra has joined #openstack-swift | 22:32 | |
| tdasilva | timburke: so what happened with that conversation of deprecating old container based x-version-location ;) | 22:44 |
| timburke | tdasilva: i'm all for it, but we're still going to be reading x-versions-location from old container dbs *forever* | 22:48 |
| *** david-lyle has quit IRC | 22:50 | |
| *** david-lyle has joined #openstack-swift | 22:53 | |
| tdasilva | timburke: true true | 22:57 |
| *** torgomatic has quit IRC | 23:10 | |
| *** torgomatic has joined #openstack-swift | 23:11 | |
| *** ChanServ sets mode: +v torgomatic | 23:11 | |
| *** torgomatic has quit IRC | 23:16 | |
| *** torgomatic has joined #openstack-swift | 23:17 | |
| *** ChanServ sets mode: +v torgomatic | 23:17 | |
| *** vint_bra has quit IRC | 23:18 | |
| *** sams-gleb has joined #openstack-swift | 23:19 | |
| *** sudorandom has quit IRC | 23:22 | |
| *** sams-gleb has quit IRC | 23:24 | |
| *** sudorandom has joined #openstack-swift | 23:24 | |
| *** chsc has quit IRC | 23:29 | |
| *** _JZ_ has quit IRC | 23:34 | |
| *** vint_bra has joined #openstack-swift | 23:37 | |
| *** tdasilva has quit IRC | 23:47 | |
| openstackgerrit | Cindy Lu proposed openstack/swift: Container Object Count Quota Functional Test https://review.openstack.org/314099 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
