Friday, 2014-05-02

*** shakayumi has joined #openstack-swift00:00
*** joeljwright has quit IRC00:01
*** shakamunyi has quit IRC00:05
*** shakayumi has quit IRC00:05
*** matsuhashi has joined #openstack-swift00:29
*** shri has quit IRC00:41
*** dmorita has joined #openstack-swift00:55
*** joeljwright has joined #openstack-swift00:58
*** gadb has joined #openstack-swift01:02
*** joeljwright has quit IRC01:02
*** saschpe has quit IRC01:27
*** saschpe has joined #openstack-swift01:32
*** nosnos has joined #openstack-swift01:36
*** nosnos has quit IRC01:44
*** nosnos has joined #openstack-swift01:44
*** sungju has joined #openstack-swift01:51
*** sungju has quit IRC01:53
*** joeljwright has joined #openstack-swift01:59
*** shakayumi has joined #openstack-swift02:00
*** annegentle has joined #openstack-swift02:00
*** joeljwright has quit IRC02:04
*** openstackgerrit has quit IRC02:04
*** openstackgerrit has joined #openstack-swift02:05
*** shakayumi has quit IRC02:14
*** tsg has quit IRC02:18
*** shakayumi has joined #openstack-swift02:31
*** ppai has joined #openstack-swift02:40
*** lnxnut has joined #openstack-swift02:44
*** byeager has joined #openstack-swift02:45
*** nosnos has quit IRC02:51
*** joeljwright has joined #openstack-swift03:00
*** praveenkumar has joined #openstack-swift03:03
*** joeljwright has quit IRC03:04
*** lnxnut has quit IRC03:09
*** byeager has quit IRC03:13
*** gyee has quit IRC03:25
*** matsuhashi has quit IRC03:27
openstackgerritHugo Kou proposed a change to openstack/swift: Fix the hash_suffix was never called for a non-None hash value
*** tzumainn has quit IRC03:37
*** joeljwright has joined #openstack-swift04:02
*** nosnos has joined #openstack-swift04:05
*** joeljwright has quit IRC04:06
clayggholt i still don't get it, I can shared between //realm1/cluster1 and //realm2/cluster5 just as easily as //realm1/cluster1 and //realm1/cluster2 - I don't understand the different in having clusters grouped in realms - what can two clusters in the same realm do that two clusters in different realms can't?  (maybe I'm thinking about it wrong)04:24
clayggholt: and thanks for getting back on the async!04:25
claygredbo: do you have any thoughts on this ->
openstackgerritClay Gerrard proposed a change to openstack/swift: Add probe tests for Account Reaper
*** chandan_kumar has joined #openstack-swift04:36
claygredbo: nm, I think I got it (you might still wanna sanity check me)04:43
redboclayg: sort of.  The problem is we can't just look at all the files all the time.  I was hoping to tackle that problem when I replaced hashes.pkl with a database.  Then finding tombstones to delete could just be a db query.  But that hasn't panned out performance-wise.04:44
redboit could maybe be made part of the auditor, since it's already doing the walking04:46
redboPeople have tried to put in hashes.pkl expiration before, but that scares the crap out of me.  If you ever fall behind on replication, you'll end up reindexing the whole machine in the next replication pass.  And that'll likely take long enough that they'll all be expired again the next pass.04:50
claygredbo: thanks, that's good insight04:55
redboMy plan was to put all of a partition's filenames in a database, with "chexor"-style hashes per suffix (or whatever number of hashes has the best tradeoff).  So they're continuously updated rather than re-walking the filesystem.  And then we have all the filenames in a db, so we could index and query for stuff like expired tombstones.04:59
redboOKAY, I need to get to the gymnasium!05:00
*** joeljwright has joined #openstack-swift05:02
*** matsuhashi has joined #openstack-swift05:03
*** shakayumi has quit IRC05:04
*** joeljwright has quit IRC05:07
gholtclayg: Yes you can share across realms that are available, but they will use different realms keys when the clusters communicate. So the Rackspace realm key isn't known by the Rackspace/SwiftStack realm key. So SwiftStack couldn't just start syncing to any Rackspace cluster, it'd have to be in the right realm. Does that help?05:09
gholtAnd yes, you're right that the user could sync SwiftStack->Rackspace1->Rackspace2. But they'd be allowed to sync Rackspace1->Rackspace2 regardless. Just SwiftStack->Rackspace2 wouldn't be allowed. Presumably because of route costs.05:16
gholtBut really it's so SwiftStack<->OtherCompany doesn't necessarily mean OtherCompany<->Rackspace, though obviously if one has an account on all three and there are two syncing connections (say SwiftStack<->Rackspace as well), the user can do whatever (say OtherCompany->SwiftStack->Rackspace and back).05:19
*** nshaikh has joined #openstack-swift05:19
*** changbl has joined #openstack-swift05:21
openstackgerritHugo Kou proposed a change to openstack/swift: Fix the syntax of stripping '.' for items in storage_domain list
*** shakayumi has joined #openstack-swift05:30
*** shakayumi has quit IRC05:35
*** chandan_kumar has quit IRC05:47
*** zaitcev has quit IRC05:59
*** chandan_kumar has joined #openstack-swift06:01
*** joeljwright has joined #openstack-swift06:04
*** joeljwright has quit IRC06:08
*** psharma has joined #openstack-swift06:14
*** shakayumi has joined #openstack-swift06:31
*** bvandenh has joined #openstack-swift06:36
*** shakayumi has quit IRC06:36
*** chandan_kumar has quit IRC06:51
*** gadb has quit IRC06:55
*** openstackgerrit has quit IRC06:57
*** joeljwright has joined #openstack-swift07:05
*** chandan_kumar has joined #openstack-swift07:05
*** joeljwright has quit IRC07:09
*** madhuri has quit IRC07:10
*** waterkinfe has joined #openstack-swift07:10
*** acoles_away is now known as acoles07:13
*** shakayumi has joined #openstack-swift07:32
*** mmcardle has joined #openstack-swift07:32
*** shakayumi has quit IRC07:37
*** tanee has quit IRC08:01
*** waterkinfe has quit IRC08:02
*** joeljwright has joined #openstack-swift08:06
*** haomai___ has quit IRC08:10
*** joeljwright has quit IRC08:11
*** haomaiwang has joined #openstack-swift08:11
*** joeljwright has joined #openstack-swift08:12
*** piousbox_ has joined #openstack-swift08:20
*** acoles has quit IRC08:21
*** cheri has joined #openstack-swift08:22
*** Ju_ has joined #openstack-swift08:25
*** tanee has joined #openstack-swift08:29
*** acoles has joined #openstack-swift08:32
*** ChanServ sets mode: +v acoles08:32
*** Ju_ has quit IRC08:33
*** shakayumi has joined #openstack-swift08:33
*** torgomatic_ has joined #openstack-swift08:34
*** psharma has quit IRC08:34
*** saschpe has quit IRC08:34
*** wkelly has quit IRC08:34
*** creiht has quit IRC08:34
*** torgomatic has quit IRC08:34
*** torgomatic_ is now known as torgomatic08:34
*** piousbox has quit IRC08:34
*** Ju has quit IRC08:34
*** gholt has quit IRC08:34
*** ondergetekende has quit IRC08:34
*** creiht has joined #openstack-swift08:34
*** ChanServ sets mode: +v creiht08:34
*** cheri has quit IRC08:34
*** saschpe has joined #openstack-swift08:35
*** shakayumi has quit IRC08:37
*** dmorita has quit IRC08:38
*** chandan_kumar has quit IRC08:40
*** ondergetekende has joined #openstack-swift08:47
*** cheri has joined #openstack-swift08:51
*** tzumainn has joined #openstack-swift08:52
*** chandan_kumar has joined #openstack-swift08:53
*** psharma has joined #openstack-swift08:53
*** wkelly has joined #openstack-swift08:53
*** matsuhashi has quit IRC09:04
*** nshaikh has quit IRC09:04
*** matsuhashi has joined #openstack-swift09:05
*** cheri has quit IRC09:10
*** cheri has joined #openstack-swift09:23
*** Ju has joined #openstack-swift09:33
*** shakayumi has joined #openstack-swift09:33
*** shakayumi has quit IRC09:38
*** shakayumi has joined #openstack-swift10:34
*** matsuhashi has quit IRC10:35
*** matsuhashi has joined #openstack-swift10:37
*** shakayumi has quit IRC10:39
*** nshaikh has joined #openstack-swift10:45
*** Trixboxer has joined #openstack-swift11:02
*** ppai has quit IRC11:09
*** mmcardle has quit IRC11:11
*** mmcardle has joined #openstack-swift11:13
*** gholt has joined #openstack-swift11:16
*** ChanServ sets mode: +v gholt11:17
*** matsuhashi has quit IRC11:22
*** psharma has quit IRC11:29
*** shakayumi has joined #openstack-swift11:35
*** gholt has quit IRC11:35
*** nosnos has quit IRC11:37
*** shakayumi has quit IRC11:39
*** matsuhashi has joined #openstack-swift11:48
*** nosnos has joined #openstack-swift11:53
*** vr2 has joined #openstack-swift12:21
vr2is there a specific place to add a blueprint for a new ObjectController ?12:25
*** shakamunyi has joined #openstack-swift12:36
*** nosnos has quit IRC12:39
*** shakamunyi has quit IRC12:41
*** mmcardle has quit IRC12:55
*** matsuhashi has quit IRC13:04
*** matsuhashi has joined #openstack-swift13:04
*** Midnightmyth has joined #openstack-swift13:05
*** mmcardle has joined #openstack-swift13:05
*** cheri has quit IRC13:05
*** matsuhas_ has joined #openstack-swift13:08
*** matsuhashi has quit IRC13:08
glange <-- there ?13:16
vr2yep thanks13:17
*** matsuhas_ has quit IRC13:18
vr2portante are you there ?13:18
portantevr2: back13:26
vr2portante: I'm about to create a blueprint for an integration of scality sproxyd as a backend13:30
portantecool, got code yet?13:31
vr2portante: anything special to care about ?13:31
vr2we have the code13:31
portantecan you post it?13:31
vr2it is an HTTP pass-through13:32
vr2yes I can13:32
vr2please note it requires the use of a proprietary component, anyways the backend can be hacked to point to other http based object stores13:33
vr2I mention it since I see it is the first proprietary project related to swift13:33
vr2as I've seen13:33
portanteperhaps you should consider making it general for HTTP based object stores13:34
portantebut post the code so that we can consider it and offer feedback13:34
vr2ok, where shall I put it ?13:34
portantecan you do github?13:34
portantevr2: and are you planning on attending the openstack summit in Atlanta?13:35
vr2yes I do13:35
vr2portante: yes I will be there13:35
vr2portante: so I do the blueprint + I post the source13:35
portanteyes, thanks13:36
portanteit is not likely that we'll take backends into the swift tree, though13:36
portantefor example, the gluster-swift (now being renamed to swift-on-file) is an existing backend that is public, but not in the swift ree13:36
*** shakamunyi has joined #openstack-swift13:37
portanteand I believe SwiftStack has a backend for the Seagate Kinetics drive that is not in the swift tree (though I don't know where that lives)13:37
portantethere is a proposal for a CEPH backend against the swift ree, and but that is there to help get it going, but won't land there13:37
vr2What do you prefer ? do we make a fork or do we make a branch on the repo ?13:41
*** shakamunyi has quit IRC13:41
*** zhiyan_ is now known as zhiyan13:45
vr2portante: the GlusterOnFile is not registered as a swift blueprint13:45
vr2portante: any reason for that ?13:45
portanteyes, it was initially done from within the GlusterFS community with no interactions with the Swift community13:48
portantewe at red hat then worked on the DiskFile backend work, which does have blueprints13:48
portantethe only blueprints required are those for adding things to OpenStack itself, as I understand it, though notmyname is the authority on all that13:49
portantecreiht and others know more of the history of OpenStack and the use of blueprints13:49
*** lnxnut has joined #openstack-swift13:51
*** byeager has joined #openstack-swift13:52
*** shakamunyi has joined #openstack-swift13:52
vr2portante: OK so do you recommend us to create a swift blueprint or something else ?13:56
*** krtaylor has quit IRC13:56
portanteI would post the code first, and see if there is something there that might warrant the blueprint13:56
portantebut I would defer to notmyname's judgement13:56
*** mrsnivvel has quit IRC13:59
*** openstackgerrit has joined #openstack-swift14:08
*** kevinc_ has joined #openstack-swift14:10
*** zhiyan is now known as zhiyan_14:13
vr2portante: but you did not tell me about fork or branch ?14:14
portanteI would suggest that you keep this as a separate repo, much like the gluster-swift repo14:16
portanteif you there you have code that should land in the tree, then let's propose that as a patch and we can see it and discuss it via gerrit review14:17
*** d0ugal has joined #openstack-swift14:21
*** fifieldt has quit IRC14:21
*** bill_az has joined #openstack-swift14:21
*** lpabon has joined #openstack-swift14:22
*** nshaikh has quit IRC14:23
*** Midnightmyth has quit IRC14:24
*** gholt has joined #openstack-swift14:25
*** ChanServ sets mode: +v gholt14:25
*** shakamunyi has quit IRC14:31
*** shakamunyi has joined #openstack-swift14:31
*** bill_az_ has joined #openstack-swift14:32
*** shakamunyi has quit IRC14:32
*** bill_az has quit IRC14:32
*** shakamunyi has joined #openstack-swift14:32
*** lpabon has quit IRC14:32
*** ams0 has joined #openstack-swift14:34
*** krtaylor has joined #openstack-swift14:37
*** kr4zy has joined #openstack-swift14:40
kr4zyHello, has anyone gotten chunked encoding to work when hosting Swift proxy service using Apache2 mod_wsgi?14:41
*** kevinc_ has quit IRC14:41
*** kevinc_ has joined #openstack-swift14:44
vr2portante: if you want to have a look
portantegreat, thanks14:47
vr2shall I post the link somewhere ?14:47
portanteokay, so for large objects, this is going to be memory intensive14:49
portanteit might be better to work on a new proxy ObjectController, see swift/proxy/controllers/ObjectController, which passes on the http request as it comes through rather than the store-and-forward in-memory method you have above14:51
portantebut we don't have a facility for loading something like that14:51
*** shakamunyi has quit IRC14:53
*** shakamunyi has joined #openstack-swift14:54
kr4zySo has anyone gotten chunked encoding working when hosting the proxy service using Apache2 with mod_wsgi? Thanks.14:55
portantekr4zy: I don't know14:55
kr4zyThanks protante.14:55
*** shakamunyi has quit IRC14:56
portantebut others may have, so keep asking14:56
*** shakamunyi has joined #openstack-swift14:56
*** kevinc_ has quit IRC14:57
*** shakayumi has joined #openstack-swift14:59
kr4zyanyone here beside portante that has knowledge on getting chunked encoding working when hosting swift proxy service on Apache2 mod_wsgi?15:01
*** shakamunyi has quit IRC15:01
*** shakayumi has quit IRC15:03
*** shakamunyi has joined #openstack-swift15:03
*** shakamunyi has quit IRC15:03
*** creiht has quit IRC15:06
*** ams0 has quit IRC15:10
*** corrigac has joined #openstack-swift15:11
vr2portante: initially I planned to use some kind of message passing between httplib and the diskfilewriter abstraction15:15
vr2portante: but actually the pb is that httplib requires a file compatible object because it wants to consume it at its own pace, although we don't control the swift diskfilewriter which passes chunks also at its own pace15:16
vr2I tried to use the rabbitmq from eventlet, but I had issue with greenthreads15:17
*** shakamunyi has joined #openstack-swift15:18
portantethat is why I am thinking about a proxy-controller backend, because we already have the for the proxy server15:19
vr2portante: good idea15:20
*** creiht has joined #openstack-swift15:25
*** ChanServ sets mode: +v creiht15:25
*** creiht has quit IRC15:25
*** creiht has joined #openstack-swift15:25
*** ChanServ sets mode: +v creiht15:25
vr2portante: for now, how does the proxy obj controller works when it contacts the obj server ?15:27
portantewhen the proxy server receives the initial header of a REST API request, it makes all the calculations to figure out which object/cont/acct servers to contact, and then constructs a potentially modified header of the original and starts a set of connections to the backend servers15:29
portanteif the servers, in their 100-continue response, accept the request, then the proxy server pulls bits off the wire and sends them on to the open backend connections15:30
portantethis sounds a lot like what you want to do15:30
vr2yes but we were afraid not being compatible since there is a lot of http header interpretation in the proxy code15:31
vr2e.g. x-delete-after, etc15:31
portanteyes, which is why you'd want to be sure the abstractions in the proxy code exist so that API handling semantics are properly seperated from backend controller communication15:32
vr2yes that's a good idea15:34
*** ChanServ sets mode: +v torgomatic15:34
vr2do you think we could do that ? or do you plan to do that ?15:34
portanteI had started that work a long time ago, but did not get very far15:35
*** shakayumi has joined #openstack-swift15:37
*** shakamunyi has quit IRC15:38
portantevr2: back15:40
creihtI really don't think that type of abstraction belongs in swift15:40
vr2creiht: you mean it is up to us to submit it15:41
creihtvr2: who is us?15:42
creihtsorry I had to reset my irc stuff so I lost all the history15:42
*** peluse_ has quit IRC15:43
vr2creiht: we write a swift object backend that talks to an HTTP proxy15:43
vr2we started by implementing a FileSystem but we have to do a store-and-forward in memory15:43
vr2portante thinks it is better to modify the proxy controller obj with a cleaner abstraction for managing that15:44
creihtvr2: ahh15:44
*** JelleB is now known as a1|away15:44
creihtI'm just expressing the opinion that swift needs to focus on being an object storage system15:45
vr2note that a store-and-forward + SLO make a relatively small footprint15:45
creihtnot an abstraction for other object storage systems15:45
creihtand it is just my opinion :)15:45
creihtsorry I'll go back to my corner and let yall continue :)15:46
vr2creight: that's a commendable opinion15:46
glangecreiht: but maybe this will allow ceph on our backend15:46
vr2creiht: but like any other services in OpenStack that would be cool to have contributors and vendors15:46
vr2e.g. Cinder15:46
*** Midnightmyth has joined #openstack-swift15:47
vr2Cinder has support for a variety of block devices, not only LVM15:47
vr2ideally swift shall do the same, no ?15:47
creihtvr2: right, but cinder was written specifically for that purpose15:47
creihtcinder isn't a block storage system15:47
creihtcinder is an abstraction that has a poor default implementation15:48
creihtswift was written from the beginning to be a good object storage system15:48
portantevr2: swift could do the same for storage systems, but to layer swift on another system that does the same thing just to get the api is not necessarily the best method15:48
*** gyee has joined #openstack-swift15:48
creihtif openstack wants an abstraction, my argument is that it should be something else that swift could plug into just like every other system15:48
glangethe api is the most important part :/15:48
creihtothers may not share the same opinion15:49
vr2yes, an object store system is "abstractable" so swift could be15:50
creihtmy main concern is that if swift becomes the abstraction, then we focus more on the abstraction rather than swift itself15:50
vr2yes but sysadmins want a common API, different vendors, to stimulate the offer15:51
*** bvandenh has quit IRC15:51
creihtvr2: if there is a case for that, then I'm fine, but my argument is that it shouldn't be swift15:52
vr2okay I hear it15:52
portantecreiht: if you look at the layering in the proxy server, it is almost there today15:52
creihtportante: I agree that it *could* be done15:52
creihtI'm just not certain that it is the right thing15:53
portantea few changes to break the proxy controllers into front and backend, and you get the Swift API fronting storage implementations15:53
*** peluse has joined #openstack-swift15:54
creihtswift *could* do a lot of things15:54
creihtbut my questions is what *should* it do?15:54
creihtbut I'm just one dev15:55
creihtwith his own opinions15:55
creihtand I must be a bit grumpy today15:55
*** peluse_ has joined #openstack-swift15:55
portanteI think it is a worth while discussion to have at summit15:55
*** chandan_kumar has quit IRC15:55
redbois it an evil chuck day?15:55
creihtportante: yeah I would agree15:56
portanteget out that creeper killer15:56
vr2creiht: what if another popular open-source object store wants to be there ?15:56
*** bvandenh has joined #openstack-swift15:56
creihtvr2: my opinion is, if there is to be an object storage abstraction, it should be a new project which swift could be a backend for15:57
creihtjust like any other object storage system15:57
vr2is there a particular session at openstack summit to discuss that ?15:58
creihtnot that I know of15:58
creihtnotmyname may have a different opinion15:58
creihtagain, I'm just one person on the project15:59
creihtI don't dictate for the project :)15:59
*** acoles is now known as acoles_away15:59
creihtvr2: to be clear, nobody has proposed an abstraction system16:00
creihtoutside of swift16:00
*** mkollaro has joined #openstack-swift16:02
*** csd has joined #openstack-swift16:11
*** joeljwright has quit IRC16:14
*** mwstorer has joined #openstack-swift16:20
notmynamegood morning. /me reads buffer playback. oh my.16:21
*** corrigac has quit IRC16:24
*** zaitcev has joined #openstack-swift16:24
*** ChanServ sets mode: +v zaitcev16:24
notmynameI'm writing up some bullet points of my thoughts16:25
creihtnotmyname: yeah I imagine you and I don't agree in this area16:25
notmynamecreiht: actually, I think we are very closely aligned (but perhaps not identical)16:26
notmyname* I very much think that swift is an object storage system and not a provisioner of storage systems.16:27
notmyname* while it's possible to use some different object server, I also think that isn't "Swift".16:27
notmyname* swift already supports vendor-specific implementations (gluster, zeroVM, kinetic), and it's good to encourage good abstractions so that those can be good. but swift upstream needs to focus on a very good implementation itself16:27
notmyname /end16:27
notmynamecreiht: portante: vr2: ^16:29
*** byeager has quit IRC16:30
notmynamealso, I very much think that the API and implementation aren't decoupled (eg we should never have an "api project" and a "backend project")16:30
*** corrigac has joined #openstack-swift16:34
*** byeager has joined #openstack-swift16:35
*** mmcardle has quit IRC16:36
openstackgerritChristian Berendt proposed a change to openstack/python-swiftclient: fixed typos found by RETF rules
*** vr2 has left #openstack-swift17:08
zaitcevDo people want to get into AUTHORS file _that_ bad?17:12
*** shri has joined #openstack-swift17:13
*** praveenkumar has quit IRC17:15
notmynamewhich reminds me...I wonder if the pep8 version will be bumped17:19
notmyname73 errors with new pep8, almost all are "continuation line under-indented for visual indent"17:21
notmynamein swift/ and test/17:21
portantenotmyname: thanks, sounds like a good discussion for summit17:22
notmynameportante: swift extensibility?17:22
notmynameportante: (just checking that you arent' wanting a session on pep8 and docstring typos)17:23
portanteyes, and what that means in relation to the API17:23
portantedefinitely! that would be the most attended of all!17:23
notmynamedid you see the ops rant on the openstack-operators mailing list this morning?17:24
portanteno, let me go look17:24
*** nwagner has joined #openstack-swift17:28
luisbgnotmyname, thanks for the recheck!17:29
luisbgI was about to request it and got dragged into some client work17:29
portantethanks for the link regarding the ops rant17:31
*** corrigac has quit IRC17:32
*** mkollaro has quit IRC17:35
*** shakayumi has quit IRC17:52
*** byeager has quit IRC17:54
*** ams0 has joined #openstack-swift17:56
claygis problem building netifaces for py26 a thing now?17:58
notmynameclayg: ya, I was looking at that just a while ago (before the eng meeting)18:00
notmynameclayg: not sure, shich is why I threw a recheck no bug first18:00
notmynameclayg: pypi trove info for netifaces 0.10.1 still says py25 py26 py26 and py318:02
*** byeager has joined #openstack-swift18:03
* clayg spins up lucid vm18:04
claygstupid netifaces, didn't onetime somebody write a pure python impl of whataremyips18:06
zaitcevnotmyname: Not sure if that is much relevant to Swift. Surely we are engaged with uses through the devops style companies like eNovance and Swiftstack.18:06
claygmaybe Alex_Gaynor?18:06
Alex_GaynorI wrote a pure python impl, it uses cffi though, so "pure python" gets some scare quotes18:07
zaitcevI'm going to target Clay to approve
*** joeljwright has joined #openstack-swift18:07
claygoh no!18:07
zaitcevI cannot break it up into any smaller pieces guys18:07
claygzaitcev: but what if I don't like it?  I can be sorta snobby - you may not want me to look at it18:08
zaitcevclayg: How is it worse than what's going on now?!18:08
claygzaitcev: well idk, i'm looking18:08
claygzaitcev: but having spent some time recently with DBBroker, and it's subclasses and their interaction with the base db_replicator i am pretty sure that area of the code could use a few more lines in the sand18:09
notmynamezaitcev: I think we do a decent job of listening for ops feedback. but we are tied to broader openstack issues too, so if there are tools, processes, or ideas to help improve thinking about "does this code actually work in production" when code is written, that's beneficial to everyone18:10
*** joeljwright has quit IRC18:10
notmynamezaitcev: also, while we do a decent job, I think we could do better. but a lot of that is around ongoing public QA18:10
zaitcevclayg: I'm operating under a proposition that what I'm doing will actually streamline the existing API. I keep seeing the same pattern where users of the broker have to construct a complex pattern from elementary ops. Result is, things that can be down under  with conn.get(): are not done. Result - races. For no reason!18:11
claygzaitcev: yeah i like it, ObjectController doesn't use hash_path - why should Account/ContainerController18:12
claygzaitcev: let me run it through the paces18:12
claygzaitcev: although it ends up looking mostly like a factory method for DBBroker18:13
zaitcevclayg: if you want to see how it's supposed to develop in full contex, I keep updated this: (although I gave up on having it approved whole).18:13
claygalso... what no common.DBBackend!?  :P18:13
zaitcevhmm.... Luis also said he'd prefer to have that18:14
zaitcevfully returning ErrorNotImplemented18:14
claygzaitcev: i was kidding, for better or worse I think the common base isn't helping us a ton - it's become more of a MixIn18:15
*** corrigac has joined #openstack-swift18:16
claygbut OTOH I do feel that DiskFile could have a decent shot a real base class - so it seems possible we could pull of something reasonable for the db's too18:16
notmynameluisbg: looks like your patch (and several others) were not allows because of netifaces 0.10.X on centos. the -infra team is pushing through a requirements change to exclude those versions, so maybe we'll get stuff landed later today18:18
*** ams0 has quit IRC18:18
*** corrigac_ has joined #openstack-swift18:19
claygonly on centos?  what version?18:19
notmynameclayg: 0.10.0 is reported to pass on precise and fail on centos18:20
zaitcevclayg: I'm a little irritated by the double inheritance chain AccountBackend->AccountBroker->DatabaseBroker, because when reading the code one must look up in several places to see where the method is defined.18:20
claygzaitcev: YES THAT!18:20
zaitcevclayg: I see you were somewhat facetous too, but surely what I created is not ideal.18:21
*** corrigac has quit IRC18:21
claygso i looked up "face"tous - and google figured out what you mean - but I love the definition:18:22
clayg"treating serious issues with deliberately inappropriate humor"18:22
clayg^ that's like me whole shtick18:22
kr4zyhas anyone here gotten chunked encoding to work when using Apache2 with mod_wsgi to host the proxy-service?18:27
notmynamekr4zy: I haven't looked in to that. TBH, I don't know of any production Swift clusters that run behind apache actually (although some who may run it that way in a lab)18:30
kr4zynotmyname: so how does other people host a production quality swift cluster with SSL encryption?18:31
notmynamekr4zy: use your load balancer or another tool to terminate the ssl. eg an haproxy (or commercial) load balancer, or eg stud running on each proxy redirecting over localhost to the proxy process18:32
kr4zynotmyname: we have security requirements to have ssl all the through from client to the proxy server. Is it possible to have ssl encryption using the native proxy service?18:33
zaitcevclayg: how about this idea: do not overlay methods. I identified 2 for account: get_info and put_container. So, name them and AccountBackend.put_container_2 (okay that's kernel namind style, but...). This way at least your IDE will click-through to right place for you.18:33
*** d0ugal has quit IRC18:34
*** d0ugal has joined #openstack-swift18:34
*** d0ugal has quit IRC18:34
*** d0ugal has joined #openstack-swift18:34
*** nwagner has quit IRC18:34
notmynamekr4zy: not with eventlet. does talking ssl to the proxy server box (and redirecting it locally) not meet the requirements?18:34
*** corrigac_ has quit IRC18:34
*** csd has quit IRC18:35
notmynamekr4zy: eg one thing we do at swiftstack is to send ssl to every proxy server box and use a local process to terminate the ssl and send the plain stream to another process on the same box (ie the swift proxy server process). that way, the client connection uses ssl all the way to the proxy server box18:36
kr4zynotmyname: I see what you mean.18:36
kr4zynotmyname: thanks18:36
luisbgnotmyname, cool! thanks for checking18:45
notmynameportante: in-process tests, I get 2 failures in testFileSizeLimit. swift.conf and test.conf both exist, and they have the same max_file_size (2GiB)18:53
*** csd has joined #openstack-swift18:53
luisbgnotmyname, I see you changed the filter for All patches with no core reviews in the wiki page of Priority Reviews18:54
luisbgnotmyname, do the filters need more updating?18:54
notmynameluisbg: yes, I think so. but I haven't looked at it since I saw the -infra dashboards queries they are trying to make global. but that probably won't happen for quite some time, so "we" (ie you?) should still update them18:55
luisbgnotmyname, sure. Going to look at it right now :)18:56
notmynameluisbg: thanks :-)18:56
luisbgsorry, got pulled out to some client work the other day18:56
claygzaitcev: no, i don't think that sound better?18:57
zaitcevclayg: so... Just leave as it is now?18:57
luisbgis there a public list somewhere of the core reviewers?19:03
glangecore reviewers are shrouded in mystery19:04
luisbgthat is the list I have, but want to confirm19:05
portantenotmyname: so is this an SAIO?19:05
portanteand are you running the in-process tests on that SAIO?19:05
glangeluisbg: that hast a list of people with two **'s that are core reviewers19:06
luisbgglange, that is what I am looking for, yes19:06
zaitcevthat doesn't preclude a stealth core member who never reviewes anything19:06
glangeI don't know if that is comprehenisve though, it looks good to me19:06
*** lpabon has joined #openstack-swift19:06
glangeluisbg: did we win a prize? :)19:07
glangebest core reviewers for an opensource project in 2013, maybe?19:08
zaitcevThere has to be a reporting structure of some kind, which allows Russell to populate parameters for his scripts.19:08
luisbgglange, more code to review! :P wonderful prize isn't it?19:08
glange^^ all day19:08
luisbgzaitcev, it is for filtering out the "if a patch has or not a core review". I know this doesn't sound ideal. looking for an alternative way19:09
zaitcevLooks like Sam was on a tear recently. 4x reviews over mine.19:10
luisbgzaitcev, you can catch up! :P19:16
*** ams0 has joined #openstack-swift19:17
kr4zynotmyname: I have notice communication problem with the native proxy service with keystone on wsgi ssl enabled. Have you guys notice the same problem?19:23
*** bill_az_ has quit IRC19:23
*** tzumainn has quit IRC19:26
claygif anyone has anymore info to add on the netifaces thing ->
*** csd has quit IRC19:27
luisbgclayg, reading19:29
luisbgclayg, I am getting the same
luisbgclayg, output looks exactly the same19:30
claygluisbg: yeah that seems to be the issue, going back to netifaces==0.8 seems to work19:31
*** praveenkumar has joined #openstack-swift19:31
luisbgclayg, hopefully this is fixed in Jenkins soon :)19:31
clayghopefull author can address upstream19:31
claygwell - i'm not exactly sure why we have to wait on Jenkins?  if just update our requiresments.txt will Jenkins still install something different?19:32
luisbgclayg, notmyname said the -infra teams is looking into removing those versions (or reverting to 0.8)19:34
luisbgclayg, oooh wait, I just realized you said something different19:34
luisbgclayg, netifaces>=0.5 in swift/requirements19:36
luisbgit only sets a minimum but not a maximum19:36
luisbgclayg, you could have netifaces>=0.5,<=0.819:36
openstackgerritClay Gerrard proposed a change to openstack/swift: Pin netifaces while upstream is messed up
luisbgclayg, trivial patch if you want me to do it and give it a try19:37
*** nwagner has joined #openstack-swift19:37
luisbgclayg, aaahhh there you go :)19:37
claygwell, we'll see what jenkins does iwht on the py2.6 job19:37
claygbut yeah if that works...19:38
luisbgclayg, we just need to get it through to Jenkins19:38
claygi don't think there is an 0.9?  maybe <=0.8 would have made more sense... I just don't want 0.10 :P19:38
luisbgno harm on trying19:38
luisbgclayg, potato/potatoe if there is no 0.919:39
luisbgjust thought there would be, since there was a 0.519:39
claygi'm really mostly interested to see what Jenkins does with it - then we can decide what we want to do...19:42
*** ams0 has quit IRC19:45
*** byeager has quit IRC19:45
*** ams0 has joined #openstack-swift19:45
*** byeager has joined #openstack-swift19:45
luisbgclayg, I agree19:46
*** ams0 has quit IRC19:49
notmynameleews: can you see,members19:57
notmynameportante: yes, on SAIO19:57
*** ams0 has joined #openstack-swift19:58
notmynameclayg: you'll have to wait for to land and then match that syntax exactly19:58
claygnotmyname: that's stupid20:03
claygwhat if homeboy releases 0.10.2?20:03
*** Trixboxer has quit IRC20:03
claygnotmyname:  will jenkins not run the check now that you stuck a -2 on it?20:04
portantenotmyname: so do you see a message that says you are running in-process tests?20:04
claygnotmyname: cause I honestly would like to see what jenkins will acctully do rather than what we expect it to do?20:04
luisbgclayg, filtering out specific releases is weird indeed20:06
luisbgand not maintainable, it assumes the problem will be fixed in 0.10.220:06
*** ams0 has quit IRC20:07
notmynameclayg: luisbg: I agree. the way global requirements works though is that projects can't update requirements unless it matches the global requirements file20:08
luisbgnotmyname, interesting! didn't knew20:09
luisbgnotmyname, it makes sense from a release engineering point of view, keeping consistency across all openstack modules20:09
notmynameclayg: luisbg: leave that comment on the other review ASAP otherwise that's what will be official20:10
notmynameI'll pull my -2 off so it can do it's thing andyou can see the jenkins failure20:10
*** praveenkumar has quit IRC20:13
luisbgnotmyname, want me to change my review back to +1?20:29
notmynameluisbg: on clay's patch? doesn't matter, I think20:29
luisbgnotmyname, so which one are you talking about? "leave that comment on the other review ASAP"20:30
*** nwagner has quit IRC20:30
claygluisbg: he ment this one ->
luisbgclayg, oooh right. and you commented it. great :)20:32
claygno one cares though20:32
claygi mean *I* care but only as much as "this is broken let's fix it" - i'm less caring of the "oh no no - we have to wait on other people and the politic to get them to do the right thing because *process*!"20:34
* clayg briefly wonders if he can get his picture somehwere under
luisbgclayg, I commented because I think you meant <=0.8, and not >=0.820:35
luisbgclayg, I do share the same feeling of "let's get this out of the way and not waste time on it"20:36
claygzaitcev: so I feel like there should be some testing accompanying - but everyhting I try in test_backend feels rather contrived20:37
zaitcevclayg: I thought I verified that I run normal tests, so functionally at least it all works.20:38
zaitcevclayg: Lemme see if I am making us to take a dent in coverage with this. The all-inclusive patch also has thing, which is separately excercised and I recall the coverage of the whole thing was acceptable.20:39
claygzaitcev: i'm poking at it - i'll let you know if i have any inspiration...20:40
*** lpabon has quit IRC20:46
*** csd has joined #openstack-swift20:50
*** byeager has quit IRC20:53
zaitcevclayg: I re-run coverages and it looks like exactly same numbers. I speculate it's because all these backend and broker things are common and excercised by a whole bunch of unit tests, so new code gets automatically excertised.20:54
zaitcevold:   swift.account.backend                        163      7     71      8    94%   48, 210, 216, 218, 313, 330, 38520:55
zaitcevnew:   swift.account.backend                        171      7     71      8    94%   49, 211, 217, 219, 314, 331, 38620:55
*** csd_ has joined #openstack-swift20:55
*** csd has quit IRC20:56
claygwell i wasn't specifically thinking about coverage per say20:57
*** kr4zy has quit IRC20:59
zaitcevWhat is it, then? Whenever I add code, it gets tested. In this case I'm skating using the existing tests, but the end result is to test anyway. You'll it better once new methods get introduced into the A/CBackend classes.21:01
*** tzumainn has joined #openstack-swift21:04
*** bach has joined #openstack-swift21:16
*** bach has quit IRC21:21
*** byeager has joined #openstack-swift21:24
*** nwagner has joined #openstack-swift21:24
*** byeager_ has joined #openstack-swift21:24
claygzaitcev: what was the later patch that moved some of the create/initialize stuff into the backend?21:26
claygzaitcev: i don't see it listed as a depends on 8590921:27
zaitcevclayg: everything is included into
*** byeager has quit IRC21:28
*** csd_ is now known as csd21:29
*** annegentle has quit IRC21:29
*** tzumainn has quit IRC21:31
claygzaitcev: k, lemme think21:32
*** d0ugal has quit IRC21:33
luisbgargjjjs, not being able to edit a review/comment after hitting submit21:44
openstackgerritPete Zaitcev proposed a change to openstack/swift: Pluggable Back-ends for account and container servers
zaitcevluisbg: just like e-mail huh21:54
-openstackstatus- NOTICE: Zuul is being restarted with some dependency upgrades and configuration changes; ETA 221521:59
*** bach has joined #openstack-swift22:04
*** bach has quit IRC22:09
*** bach has joined #openstack-swift22:10
*** bach has quit IRC22:12
zaitcevawww on the storage_directory() question, I rememeber that there was a super important reason to change it and I forgot what it was22:17
claygzaitcev: lol22:24
claygwell if it was *super* important i'm sure it'll come back to you :P22:24
*** lnxnut has quit IRC22:33
*** byeager_ has quit IRC22:40
claygzaitcev: not feeling super great about 85909, I acctually think having AccountBackend be a subclass of AccountBroker is a mis-step22:44
zaitcevclayg: I could refer to it by reference, just not sure it solves anything. Like... have a common class Backend, in it As far as documentation goes, it seems like a good place for docstrings. As far as code goes, I dunno. Maybe I'm getting used to this layout.22:47
claygzaitcev: I think having the class that the account controller uses delegate would acctually solve quite a bit - in that you can just stick a method on common/ and then refernce it in account/ - you have to think about it.22:48
zaitcevclayg: "delegate" is too big a word for me, I do not understand.22:48
claygbut I just stubbed that out and it looks silly (even in 47713, the list_objects_iter method that's totally a passthrough to super is pretty obviously an anti-pattern)?22:49
claygwell... like def get_info(): return
clayganyway, doesn't matter - solves one problem - creates more - I don't think it's the right way either22:49
claygI think the A/CBroker (or more specifically their base DatabaseBroker) just has too much guts to really serve the purpose of describing the interface the that the controller's need22:52
claygbut if that interface is just a subclass of them it really serves... well I care for it22:52
zaitcevNot sure if it helps, but I keep in mind that we have other implementations. In this case it's (for both a & c). Those inevitably copy. However from the API perspective the a & c share nothing, it's just convenience for our main implementation that they are so similar.22:52
zaitcevSo Luis wanted class Broker so it could contain common parts and common abstract methods22:53
claygyeah that's got us before... I think that MemDiskfile still does the wrong thing for expired objects22:54
zaitcevI suppose I can claim that if API is right, we can re-implement A/CBroker any way we like "later", but I feel that if we punt it now, we'll never get back to it.22:55
claygi agree, it the goal isn't to have the best abstraction that we can then it's not much of a goal - people can already subclass A/CController and return a thing that works like the existing implemetnation - e.g. :P22:57
claygI'd like to see (and may have to try and play with) an A/CBroker class that only implements (and documents) the interfaces needed for the REST layer and then subclass those to form an A/CBackend class that can be constructed from a path on disk and answers higher order questions like get_replication_info and their ilk23:01
*** byeager has joined #openstack-swift23:01
clayg... might be a disaster, but I'm pretty sure my concious won't let me get behind the interface being used by the controller's as a subclass of the full implementation used by the consistency engine - seems backwards23:03
zaitcevwell, this is not the road I was taking with this split. In 47713 and 85909, the primary split is that Backend is what's needed for ReST layer ( and Broker is needed for anything that has knowledge of disk layout, such as auditor.23:03
zaitcevmaybe we should fight to death in a cage23:04
claygmore like fight to the death in gerrit?23:04
zaitcevit just so happens that in existing Swift Backend inherits from Broker in the interests of not rocking the boat too much23:04
zaitcevbut notionally they are independent23:04
claygI never could get diskfile patches up fast enough to beat out portante so if you persever you'll probably beat me out ;)23:05
*** byeager has quit IRC23:05
zaitcevoh, wait23:06
zaitcevI see that you apparently flipped broker and backend around with that statement about the ilk or higher order, but otherwise they are same23:06
zaitcevokay. The reason why 47713 is like now, higher order ilk needs the path like you said. However, you can calculate path from root/dev/part/a/c/o + h + .suff, but going backwards is a major pain. Therefore at least the calls to create instances must go that way, backend calling broker. Granted it's possible to subclass in the opposite direction...23:10
*** rupsky has joined #openstack-swift23:12
zaitcevWe have subclasses providing "callback" methods today. Initialization is full of them.23:12
zaitcevPersonally I am not a fan, but perhaps I need to think in object-oriented way.23:12
*** Midnightmyth has quit IRC23:12
*** rupsky has left #openstack-swift23:12
claygwell, if nothing else I like the idea of the think called Backend being the thing that's used by backend services, but more to point I'm assuming that basically everything that isn't the wsgi server starts with a db_file - so Backend could override the Broker's __init__23:23
claygI also have a feeling that if I start with A/CBroker I'll find that most of the code tangling that comes out of the shared DBBroker base goes away - the things the wsgi server calls on A/CBroker are mostly independent implementations anyway (get_info, list_X_iter, is_status_deleted)23:25
clayg... but that's just hunch - I'd need to try it out and see23:26
-openstackstatus- NOTICE: is going down for a short database upgrade23:27
zaitcevit's true that syncs go away23:29
zaitcevinitialization stays23:29
claygzaitcev: well all the tables in _initialize_db are specific - and as you did in 47713 - it all needs to get re-written as create anyway23:30
luisbgzaitcev, gmail has a cancel feature (you can activate it and it gives you some seconds to realize you made a typo or hit submit accidentally)23:31
luisbghave a good weekend everyone!23:32
*** shri has quit IRC23:48
*** mwstorer has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!