Thursday, 2016-05-12

notmynameso 16.04 just doubled our inode memory usage? fun00:00
mattoliverauhosanai: morning00:00
timburkenotmyname: install/format with trusty, then upgrade to xenial? sounds like fun!00:01
notmynametimburke: what could go wrong?!00:01
notmynameyeah, I hadn't even thought of that (surely you'd just wipe and reinstall clean, right?)00:02
notmynamethe CRC thing in the error is interesting00:02
notmynameah ha!00:03
notmynameBy default, mkfs.xfs will enable metadata CRCs.00:03
notmynameThe value is either 0 to00:03
notmyname                          disable the feature, or 1 to enable the use of00:03
notmyname                          CRCs.00:03
notmynameso `mkfs.xfs -m crc=0 -i size=256 ...` might work00:04
notmynameyeah, that works00:05
*** lyrrad has quit IRC00:07
notmynameooo, interesting00:08
notmynameif you use crcs, then you can also turn on finobt (default on when using crc)00:09
notmyname                         The free inode btree mirrors the existing00:09
notmyname                          allocated inode btree index which indexes both00:09
notmyname                          used and free inodes. The free inode btree does00:09
notmyname                          not index used inodes, allowing faster, more00:10
notmyname                          consistent inode allocation performance as00:10
notmyname                          filesystems age.00:10
notmynamethe "faster...performance as filesystems age" sounds very interesting00:10
*** vint_bra has joined #openstack-swift00:12
mattoliveraujust +A'ed copy middleware \o/00:14
mattoliveraugo ecryption go00:15
timburkewhooo! thanks mattoliverau!00:15
kota_mattoliverau: good job!00:22
tdasilvamattoliverau: thank you!!!00:23
*** diogogmt has joined #openstack-swift00:28
*** duobei has joined #openstack-swift00:33
*** gyee has quit IRC00:36
notmynameFYI it would seem that the distro version of setuptools in xenial isn't new enough to understand the version markers in the requirements files00:38
timburkeyay! so how much work would it be to pull cname_lookup out as it's own project? :P00:40
notmynameor vendor dnspython or write a tiny dns client00:41
*** klamath has quit IRC00:42
*** lifeless has quit IRC00:46
*** lifeless has joined #openstack-swift00:47
claygnotmyname: what are you doing?00:48
claygtalking to yourself?  quoting manpages?  what's happening?00:48
claygmattoliverau: awesome!00:48
notmynameclayg: my SAIO is a precise box. I figured it was time to upgrade, so I'm building a xenial SAIO.00:48
claygnotmyname: oh oh oh are you using the saio docs!?00:49
notmynameclayg: and I was kinda hoping the xfs changes would pique your interest :-)00:49
claygnotmyname: get in rsync module per disk and servers per port while you're att it00:49
notmynameclayg: heh, not really. I should, huh? and update them00:49
claygnotmyname: nope.  i'm busy.00:49
claygnotmyname: or review one of the many patches claiming to "fix" them!00:49
timburkenotmyname: to what degree do we expect functests to pass against a non-default locale? the conversation earlier this morning got me curious about locales, so i came up with and started playing around00:50
notmynamepatch 298313 is actually going to make it a lot easier, I think. that will give you a list of non python stuff you need00:50
patchbotnotmyname: - swift - List system dependencies for running common tests00:50
claygnotmyname: I also think you sign it - everytime someome compleates the saio doc the final step should be "now open this document, add your name/distro/version/date to this list and publish the change to gerrit for review (include any fixes to commands/steps you had to figure out along the way with your change)"00:50
claygnotmyname: that *is* sorta cool?00:52
notmynameI got tox -e py27 passing, but I haven't set up configs or rings yet. I should back out some stuff now and make sure the SAIO docs are up to date. clayg's right :-)00:54
*** bill_az has quit IRC00:54
*** rcernin has quit IRC00:54
*** suyash has quit IRC01:01
clayg^ that last statement is in contrast with most of my real world experience01:03
*** Sam___ has joined #openstack-swift01:05
*** mingdang1 has quit IRC01:06
*** Sam___ has quit IRC01:07
*** ozialien10 has quit IRC01:14
*** klrmn has quit IRC01:14
jrichlimattoliverau: Awesome!  hurray!01:22
*** shakamunyi has joined #openstack-swift01:23
*** NM has quit IRC01:42
*** tqtran has quit IRC01:48
*** sgundur- has left #openstack-swift01:53
kota_back to online01:55
*** dmorita has joined #openstack-swift02:00
*** dmorita has quit IRC02:04
*** bill_az has joined #openstack-swift02:18
*** garthb has joined #openstack-swift02:25
*** klrmn has joined #openstack-swift02:32
mahaticgood morning02:33
*** daemontool has quit IRC02:38
*** garthb has quit IRC02:39
openstackgerritMerged openstack/swift: Refactor server side copy as middleware
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
openstackgerritTim Burke proposed openstack/swift: Stop complaining about auditor_status files
*** diogogmt has quit IRC03:18
mahaticCOPY merged!03:19
mahatichmm, how do we add something to "watched changes" in gerrit as opposed to starring patches?03:19
*** silor has joined #openstack-swift03:24
*** vint_bra has quit IRC03:26
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
*** mingdang1 has joined #openstack-swift03:32
*** bill_az has quit IRC03:33
*** links has joined #openstack-swift03:34
*** vint_bra has joined #openstack-swift03:34
kota_mahatic: add yourself into the review with gerrit gui, since then, you can recieve subscription when status changed.03:36
kota_into the reviewer03:36
kota_mahatic: if you want to get subscription for all patch push, set the project you want to watch via
mahatickota_: ok cool, thanks much!03:38
*** takashi has joined #openstack-swift03:46
*** vint_bra1 has joined #openstack-swift04:12
openstackgerritKota Tsuyuzaki proposed openstack/swift: Fix EC ring validation at ring reload
*** vint_bra has quit IRC04:14
jrichli#success swift just merged a large effort that started over a year ago that will facilitate new capabilities - like encryption04:16
openstackstatusjrichli: Added success to Success page04:16
mahatictorgomatic: excellent posts on go lang thread, very precise on the problem; prompted me to go look some stuff up04:17
*** nadeem has joined #openstack-swift04:18
*** ppai has joined #openstack-swift04:24
*** pcaruana has joined #openstack-swift04:25
*** pcaruana has quit IRC04:32
*** _JZ_ has quit IRC04:37
*** psachin has joined #openstack-swift04:39
*** hezhiqiang has joined #openstack-swift04:41
*** vint_bra1 has quit IRC04:59
*** hezhiqiang has quit IRC05:22
*** hezhiqiang has joined #openstack-swift05:23
*** McMurlock1 has joined #openstack-swift05:40
*** ChubYann has quit IRC05:40
*** mingdang1 has quit IRC05:47
*** ppai has quit IRC05:49
*** rcernin has joined #openstack-swift05:54
*** asettle has joined #openstack-swift06:02
*** mmcardle has joined #openstack-swift06:02
*** asettle has quit IRC06:06
*** pcaruana has joined #openstack-swift06:12
*** mmcardle has quit IRC06:13
*** ppai has joined #openstack-swift06:13
openstackgerritKota Tsuyuzaki proposed openstack/swift: Fix EC ring validation at ring reload
*** jmccarthy has quit IRC06:30
*** jmccarthy has joined #openstack-swift06:31
*** SkyRocknRoll has joined #openstack-swift06:41
openstackgerritMahati Chamarthy proposed openstack/swift: change etag (currently encypted with object key) to use container key
kota_Thanks Pete for reviewing!06:49
openstackgerritAndreas Jaeger proposed openstack/swift: List system dependencies for running common tests
*** sudorandom has quit IRC06:58
*** natarej has quit IRC06:59
*** StevenK has quit IRC07:00
*** natarej has joined #openstack-swift07:00
*** sudorandom has joined #openstack-swift07:00
*** StevenK has joined #openstack-swift07:01
*** klrmn has quit IRC07:02
*** tesseract has joined #openstack-swift07:05
*** geaaru has joined #openstack-swift07:08
*** tesseract has quit IRC07:09
*** rledisez has joined #openstack-swift07:16
*** daemontool has joined #openstack-swift07:19
*** dmorita has joined #openstack-swift07:27
openstackgerritChristian Schwede proposed openstack/swift: Don't delete misplaced dbs if not replicated
*** jmccarthy has quit IRC07:31
*** dmorita has quit IRC07:32
*** nadeem has quit IRC07:32
*** jmccarthy has joined #openstack-swift07:37
*** dmk0202 has joined #openstack-swift07:40
*** asettle has joined #openstack-swift07:52
*** acoles_ is now known as acoles07:54
*** mvk_ has joined #openstack-swift08:04
*** pchng has quit IRC08:07
acolestdasilva: ppai: congrats - great work on copy middleware08:08
*** mvk has quit IRC08:08
*** jamielennox is now known as jamielennox|away08:10
ppaiacoles, glad to see it getting merged08:13
openstackgerritMerged openstack/swift: Allow fallocate_reserve to be a percentage
*** sams-gleb has joined #openstack-swift08:21
*** Jeffrey4l has quit IRC08:22
*** asettle has quit IRC08:25
*** Jeffrey4l has joined #openstack-swift08:26
*** asettle has joined #openstack-swift08:33
*** jistr has joined #openstack-swift08:35
*** joeljwright has joined #openstack-swift08:37
*** ChanServ sets mode: +v joeljwright08:37
*** jamiehan_ has joined #openstack-swift08:43
*** jamiehan_ has quit IRC08:44
*** jamie_h_ has joined #openstack-swift08:44
*** jamie_h_ has quit IRC08:48
*** jamie_h_ has joined #openstack-swift08:48
*** jamie_h_ has quit IRC08:48
*** jamie_h_ has joined #openstack-swift08:49
*** ppai has quit IRC08:49
*** jamie_h has quit IRC08:49
*** jamie_h_ has quit IRC08:49
*** jamie_h has joined #openstack-swift08:50
*** joeljwright has quit IRC08:54
*** hosanai has quit IRC08:57
*** jamie_h has quit IRC09:00
*** pcaruana is now known as pcaruana|afk|09:01
*** ppai has joined #openstack-swift09:03
*** openstackgerrit has quit IRC09:04
*** mvk_ has quit IRC09:04
*** openstackgerrit has joined #openstack-swift09:04
openstackgerritMerged openstack/swift: crypto - cleanup decrypter exception handling
openstackgerritMerged openstack/swift: crypto - cleanup encrypter error handling
openstackgerritMerged openstack/swift: Remove unneeded setting of SO_REUSEADDR.
*** SkyRocknRoll has quit IRC09:30
*** mingdang1 has joined #openstack-swift09:34
*** JelleB is now known as a1|away09:35
*** mvk_ has joined #openstack-swift09:35
*** jamie_h has joined #openstack-swift09:36
*** jamie_h has quit IRC09:38
kota_what!!?? why patch 268993 fails at objectserver._diskfile_router?09:52
patchbotkota_: - swift - Fix EC ring validation at ring reload09:52
*** ppai has quit IRC09:52
kota_the test doesn't touch anything at object-server...09:53
kota_and it occurs only in mem_server tests. Plus, that didn't happen in my local...09:54
kota_ah, the mem_server tests looking into the filesystem look skipped and the test probably write down something as fast-post.09:58
*** ppai has joined #openstack-swift10:04
*** psachin has quit IRC10:12
*** cbartz has joined #openstack-swift10:13
*** psachin has joined #openstack-swift10:18
*** SkyRocknRoll has joined #openstack-swift10:22
mahaticacoles: is this for increased scope as well?
mahaticacoles: reduced*10:23
mahaticacoles: any of your patches covering that? Not the ones so far you've pushed i believe?10:24
acolesnot sure. commented on trello - I'm not sure why we still have crypto-id at all. I am not covering it.10:26
*** takashi has quit IRC10:30
*** jamie_h has joined #openstack-swift10:32
*** jamie_h has quit IRC10:32
*** jamie_h has joined #openstack-swift10:33
*** hezhiqiang has quit IRC10:33
mahaticacoles: there is TODO:
mahaticbut i don't completely understand the context of it10:34
kota_ah, yeah probably, mem_server doesn't support any storage-policy. That's the reason.10:35
mahaticactually i do get the context10:35
openstackgerritKota Tsuyuzaki proposed openstack/swift: Fix EC ring validation at ring reload
acolesmahatic: right, that TODO is a hint that crypto id could go once we have copy middleware. I plan to merge from master today to get that onto feature/crypto, then we can delete that code and see what happens10:36
acoleskota_: mem-server is a pain ;)10:37
mahaticacoles: yup, sounds good10:37
kota_acoles: exactly :\10:37
acoleskota_: I wonder why we have it? Maybe its a reference implementation for a pluggable backend, but unless it is maintained then it ceases to be a good reference, and I suspect we only maintain when we have to pass tests.10:40
*** duobei has quit IRC10:40
openstackgerritAlistair Coles proposed openstack/swift: crypto - wrap random body key
kota_hmm... I thought the reason is that someone is still using it but exactly I don't have the experienc to hear anyone uses it.10:45
kota_acoles: it might be good to survey in the ops session in the next summit who one uses mem_server implementation.10:46
kota_so far from now, just we done the previous austin summt a bit ago. I'm not sure I can remember to bring it to the next summit :/10:46
acoleskota_: maybe, i remember a while back you and I fixed some stuff in mem_diskfile10:47
acoleskota_: make a calendar entry ;)10:47
kota_acoles: yeah, I know.10:47
kota_make the eitherpad now for next ops session!10:48
*** mingdang1 has quit IRC10:52
*** silor has quit IRC11:03
*** diogogmt has joined #openstack-swift11:33
*** kei_yama has quit IRC11:37
*** natarej has quit IRC11:38
*** sams-gleb has quit IRC11:40
*** sams-gleb has joined #openstack-swift11:40
*** ZZelle has joined #openstack-swift11:44
*** sams-gleb has quit IRC11:49
*** ppai has quit IRC11:57
*** raildo-afk is now known as raildo12:10
*** NM has joined #openstack-swift12:11
*** sams-gleb has joined #openstack-swift12:15
*** pauloewerton has joined #openstack-swift12:17
*** hezhiqiang has joined #openstack-swift12:20
*** mingdang1 has joined #openstack-swift12:23
pdardeaugood morning12:31
pdardeauway to go on the copy middleware!12:31
*** hezhiqia_ has joined #openstack-swift12:36
*** mmcardle has joined #openstack-swift12:38
*** hezhiqiang has quit IRC12:38
*** esker has joined #openstack-swift12:42
*** hezhiqiang has joined #openstack-swift12:44
*** hezhiqia_ has quit IRC12:45
*** esker has quit IRC12:45
*** esker has joined #openstack-swift12:47
*** pcaruana|afk| is now known as pcaruana13:01
*** coreycb has quit IRC13:04
*** sams-gleb has quit IRC13:04
*** sams-gleb has joined #openstack-swift13:04
*** duobei has joined #openstack-swift13:06
*** hezhiqiang has quit IRC13:06
*** hezhiqiang has joined #openstack-swift13:07
*** sams-gleb has quit IRC13:13
*** links has quit IRC13:14
*** silor has joined #openstack-swift13:16
*** duobei has quit IRC13:16
*** sams-gleb has joined #openstack-swift13:18
openstackgerritAlistair Coles proposed openstack/swift: Merge branch 'master' into feature/crypto
*** mmcardle has quit IRC13:21
*** SkyRocknRoll has quit IRC13:22
psachinDo we have any documentation to add additional storage server to SAIO13:22
*** klamath has joined #openstack-swift13:24
*** psachin has quit IRC13:30
*** psachin has joined #openstack-swift13:31
openstackgerritSaverio Proto proposed openstack/swift: README: fix broken link
*** sams-gleb has quit IRC13:35
*** hezhiqiang has quit IRC13:35
*** sams-gleb has joined #openstack-swift13:35
*** hezhiqiang has joined #openstack-swift13:37
*** silor1 has joined #openstack-swift13:43
*** bill_az has joined #openstack-swift13:43
*** sams-gleb has quit IRC13:44
*** wanghua has quit IRC13:44
*** silor has quit IRC13:46
*** silor1 is now known as silor13:46
*** SkyRocknRoll has joined #openstack-swift13:46
glangepsachin: you mean another "fake" object server ?13:46
glangebecause a saio is swift on one server/VM, right?13:47
*** sams-gleb has joined #openstack-swift13:47
*** silor1 has joined #openstack-swift13:49
*** eranrom has quit IRC13:50
ober37mtreinish: BTW, this is the devstack patch we discussed yesterday
patchbotober37: patch 315296 - openstack-dev/devstack - WIP:  Enable Swift Account Management13:51
*** silor has quit IRC13:52
*** silor1 is now known as silor13:52
*** eranrom has joined #openstack-swift13:53
*** geaaru has quit IRC13:59
*** sgundur- has joined #openstack-swift13:59
jrichliacoles mahatic : like acoles said, now that we have copy mw, we can remove the crypto_id.  But, when we all talked about auto-tiering it made me re-think that.14:02
jrichli If we ever think some action in swift will use a different path used on the GET than what was used on the PUT/POST, then crypto-id would be necessary14:03
*** geaaru has joined #openstack-swift14:03
*** eranrom has quit IRC14:03
jrichlii know that we could try to organize the pipeline so that moves like that happen first, but ... worth a second thought14:03
openstackgerritMahati Chamarthy proposed openstack/swift: Add "disable_encryption" config option
tdasilvajrichli: sorry to jump in the convo, is crypto_id same as the hash of the key?14:05
*** esker has quit IRC14:06
mahatictdasilva: it's a header to persist object path, which is used in key derivation14:06
jrichliit's not that now, but I suppose that it could be later when BYOK is brought back14:06
*** esker has joined #openstack-swift14:06
jrichlitdasilva: we had to save the path because without copy as mw, there was a particular scenario where the path on a PUT was different than the path used on a GET14:07
acolesjrichli: if it becomes the hash then it can't be used to derive the key if an object moves. those have to be two different things14:07
jrichliacoles: yes, i meant if we decide we dont need it for decryption14:08
*** mingdang1 has quit IRC14:08
mahaticjrichli: what you mean is key_id? (I'm just using terms from overview_doc)14:08
jrichlioh, yes.  name change would be good :-)14:08
jrichliyes, if it is the hash hten it would only be for verification of supplied key14:09
mahaticjrichli: re auto-tiering usecase: good point, i'd also give it a second thought. But I'm not deeply aware of other alternatives (i.e. if there is no crypto-id, then how will we handle those cases)14:10
acolesjrichli: ok. if an object is moved without being COPYed then while a crypto_id=path would help decrypt the object, the destination container etag will be encrypted using the source container key, so we'd need to propagate the crypto_id into the listing meta, and then fetch multiple keys while decrypting a listing14:10
jrichliugh :-?14:11
mahaticacoles: mean in future we could handle it that way?14:11
mahaticdo you mean*14:11
jrichliit seems like we try to avoid copies now - want to be more like fast post.  but that leads to this situation for crypto : the crypto cant be tied to PUT path.14:13
acolesmahatic: I mean that as it is now it is not a sufficient solution for objects that move, and to make it sufficient isn't straightforward (i.e. the container listing)14:14
*** d0ugal has quit IRC14:14
mahaticacoles: ohh yeah14:15
jrichlithat last bit was just thinking out loud.14:15
jrichlii guess we will solve the problem later if needed14:16
*** d0ugal has joined #openstack-swift14:16
acolesand if/when we have wrapped container listing keys its gets harder/impossible - we cannot deal with an etag in the container db encrypted with a key that is wrapped and stored as metadata of a different container.14:16
jrichlii see your poit14:17
mahaticacoles: i thought, we'd store crypto-id as another header in listing meta14:18
mahaticwhen you said propogate crypto_id into listing meta14:19
acolesjrichli: its partly why I put that paragraph in the overview doc for developers - it's a headsup for anyone who is planning to move objects around without involving the proxy.14:20
*** AJaeger has quit IRC14:21
acolesmahatic: problem is, even if we put metadata in the listing etag that points to which key was used, if an object was moved then that key might be for a different container which would require a HEAD to that container to retrieve the key.14:22
acolesand potentially the same for every moved object in a container.14:22
mahaticacoles: oh sorry, i do see it. it's listing key meta. right, very complicated14:22
acolesthe good news is we have copy middleware on feature/crypto for real and all func tests passing14:22
*** d0ugal has quit IRC14:23
*** flaper87 has quit IRC14:24
*** flaper87 has joined #openstack-swift14:27
*** flaper87 has quit IRC14:27
*** flaper87 has joined #openstack-swift14:27
*** vint_bra has joined #openstack-swift14:28
*** _JZ_ has joined #openstack-swift14:30
*** links has joined #openstack-swift14:36
*** esker has quit IRC14:38
*** joeljwright has joined #openstack-swift14:41
*** ChanServ sets mode: +v joeljwright14:41
admin6Hi there, do you know where I’m supposed to find the logs for object-reconstructor regarding my erasure coding storage? I would expect something like object-reconstructor.log just a the replicator do with object-replicator.log. Shouldn’t I ?14:44
openstackgerritAndy McCrae proposed openstack/swift: [Docs] Document prevention of disk full scenarios
acolesjrichli: mahatic brainstorming...what if we used an account level key (wrapped) to encrypt the etags? then we'd always have the right key available regardless of which container the etag ended up in, so long as it's in the same account. Is auto-tiering going to cross accounts?14:48
acolesof course, IDK about re-suing that key so much14:48
notmynamegood morning14:49
openstackgerritAlistair Coles proposed openstack/swift: Put back missing sysmeta unit tests
acolestdasilva: ^^ i noticed this while doing the merge to feature/crypto14:50
mahaticacoles: accessing account metadata on obj/cont requests? by calling account_info etc?14:52
acolesmahatic: it already happens often when getting container info
openstackgerritAlexandra Settle proposed openstack/swift: [Docs] Document prevention of disk full scenarios
notmynamejoeljwright: is there an option in swiftclient to not validate teh md5 on download? or actually does it even do that? a cursory glance seems that it only does md5 checks when skip_identical is set14:57
notmynameacoles: now that copy middleware has landed are you (or have you already) done a merge to crypto from master?14:57
openstackgerritAndy McCrae proposed openstack/swift: [Docs] Document prevention of disk full scenarios
*** esker has joined #openstack-swift15:01
notmynameacoles: ah, nm. you have. just got to that point in my email inbox :-)15:01
*** esker has quit IRC15:02
*** esker has joined #openstack-swift15:04
*** arch-nemesis has joined #openstack-swift15:05
admin6sorry for the noise about reconstructor logs, just a stupid rsyslog misconfiguration on my side  :(15:08
*** mmcardle has joined #openstack-swift15:08
jrichliacoles: sounds like a good idea.  we dont even use an account key right now.  we used one before when we had a wrapping scheme at each level.15:11
notmynameadmin6: no worries. I'm both glad you asked and glad you figured it out :-)15:11
admin6notmyname: thanks15:12
joeljwrightnotmynname: I believe the _SwiftReader object will throw an error on the download if the received md5sum is not the expected value15:12
joeljwrightnotmyname ^^^15:12
jrichliacoles: i dont think that would be too much use.  also, if it only us used for etags - that would still make for a very lengthy attack to eventually get to object bodies or metadata15:12
notmynamejoeljwright: ah yes. there it is. thanks15:12
jrichliof course we will have crypto experts vet all this ;-)15:12
joeljwrightnotmyname: no probs15:13
notmynamejoeljwright: do you think it would be valuable to noop the md5 check there if requested?15:13
notmynamejoeljwright: someone pointed me to this morning15:14
joeljwrightnotmyname: I'll take a look15:15
notmynamejoeljwright: certainly not a priority (from my perspective), but it's an interesting consideration, I think. I wonder if swift3 does something similar or has similar issues15:15
joeljwrightnotmyname: is the problem really that we should check large objects properly?15:15
joeljwrightif we're not already15:15
notmynamejoeljwright: you mean the actual md5sum of the whole large object?15:16
mahaticacoles: okay cool. it does sound good. re reusing: we'd still have different iv's in any case i believe15:16
joeljwrightlooking at the code, it looks like it ignores the md5sum for large object downloads anyway15:16
mahaticif that's what you meant by reusing15:17
notmynamejoeljwright: yeah. but I'm guessing that in their case they actually aren't using *LOs. I mean, they don't actually have swift15:17
notmynamejoeljwright: which goes back to my first point of "not really a priority" ;-)15:17
joeljwrightnotmyname: ah, I see!15:17
*** d0ugal has joined #openstack-swift15:18
joeljwrightan option couldn't hurt I suppose15:18
notmynametimburke: how does swift3 handle multipart objects vs *LOs?15:18
joeljwrightnotmyname: and we could do with checking 'properly' anyway15:18
*** shakamunyi has quit IRC15:19
jrichlimahatic: the re-use of keys itself is considered because although we do get 256 bits, it is still a finite number of keys - if we used a random account key, then you have to factor in probability of getting same number.15:20
mahaticjrichli: oh I see15:20
jrichlibut - are we going to use the naem of the account in this case?  i suppose so.  so we still have to consider the fact that the iv for etag is still used for each time we re-use an object path15:21
jrichli(i mean derive key from name of accoutn)15:21
jrichliactually, I guess that he was speaking to the re-use of one given key - not to how many keys we were generating.15:24
mmotiani_I can see there are lot of empty docs in the swift git repository15:24
jrichlimahatic: so actually, in this case, its about the probability of the iv / nounce being repeated15:25
notmynamemmotiani_: not really empty. but they have sphinx directives in them to fill out with other generated stuff15:25
jrichlisort of along the same lines of thinking about key repetitions : its all finite so you gotta think of how many you can do before a "unique" number is not so unique anymore15:25
notmynamemmotiani_: vs
notmynamemmotiani_: check out the raw version of the .rst
*** catintheroof has joined #openstack-swift15:26
mahaticjrichli: hmm. but iv being a repetitive, that concern would apply to everything else as well. not specific to etags i guess15:26
jrichlithe etag situation is worse.  for etag, we know we repeat the iv if you repeat a path15:27
*** catintheroof has quit IRC15:27
jrichlibut for the random iv case, the argument is similar to the one about keys: you only get 16 bytes, and random is sometimes going to repeat15:28
jrichliin fact, there are fewer bytes that move in the iv case for CTR15:28
jrichli16 bytes is the nonce + counter15:28
jrichlionly the counter is incremented15:29
*** catintheroof has joined #openstack-swift15:29
jrichliif there are 2 blocks that re-use an iv, and somebody finds those two blocks, they could discover part of the key stream.15:29
*** mmcardle has quit IRC15:29
*** links has quit IRC15:32
jrichlimahatic: but every encryption has a percentage of risk like this.  I think our scheme is pretty secure (again, we will vet from a crypto expert!)15:32
mahaticacoles: jrichli since we're trying to solve moving of objects, should also think of auto-tiering being cross accounts or not? as acoles mentioned. IDK.15:33
mmotiani_notmyname, got it, thankss :)15:33
mahaticjrichli: yeah15:33
mahaticacoles: jrichli i'm calling it a night15:33
mahatichave a nice day/evening15:33
jrichlimahatic: ok, good night!15:33
*** sams-gleb has quit IRC15:35
*** dmk0202 has quit IRC15:35
*** dmk0202 has joined #openstack-swift15:36
*** sams-gleb has joined #openstack-swift15:36
*** psachin has quit IRC15:39
openstackgerritAndrew Welleck proposed openstack/python-swiftclient: Query string functionality for containers
*** mmcardle has joined #openstack-swift15:41
ZZelleHi everyone!15:43
ZZelleSome cores can look at ? it needs one more +215:43
*** lyrrad has joined #openstack-swift15:44
*** sams-gleb has quit IRC15:45
jrichliacoles: I will ask ccachin what he thinks about using the account key for each obj etag15:45
jrichliand repearhulk15:47
openstackgerritSamuel Merritt proposed openstack/swift: Clean up fallocate tests a little
*** SkyRocknRoll has quit IRC15:51
*** SkyRocknRoll has joined #openstack-swift15:52
*** dmorita has joined #openstack-swift15:54
jrichliacoles: is it possible for me to work on adding a cyrpto gate test for our branch - even without core privileges?15:55
*** garthb has joined #openstack-swift15:56
*** SkyRocknRoll has quit IRC15:57
acolesjrichli: swift cores have no privilege wrt the gate jobs, it's approved by -infra15:59
acolesjrichli: fwiw I would be inclined to add another in process test variant with a modified pipeline rather than get a devstack config option15:59
jrichliacoles: ok.  i will try to find the patch where you added that for fast post and take a look.  any info or advice would be welcome :-)16:01
acolesjrichli: so auto-tiering, if the tier containers are managed by some server side process then in principle it could be arranged for the to have to same wrapped container keys as the source container, assuming a 1:1 mapping of source to tier containers16:02
acolesjrichli: do a git blame on test/functional/ to find the change I made for fast post config16:02
acolesjrichli: the hard part will be agreeing an acceptable way to configure the in process test pipeline - i used an env var for fast post, that will be way too clunk for a pipeline. perhaps have a set of pre-config'd pipelines that selected by an env var16:04
openstackgerritRichard Hawkins proposed openstack/swift: go: middleware to allow creating test objects
acolese.g. SWIFT_IN_PROCESS_TEST_PIPELINE=default|crypto ??16:04
jrichliinteresting on the auto-tiering.  I dont know enough about the plan to know if we could expect 1:1 mapping.  but its something i will record as an idea16:09
jrichlithanks for the ideas on the in process option.  i'll take a loo16:10
*** cbartz has left #openstack-swift16:12
*** diogogmt has quit IRC16:15
*** dmk0202 has quit IRC16:15
openstackgerritAndrew Welleck proposed openstack/python-swiftclient: Query string functionality for containers
acolesjrichli: is my infra patch, you'd need something similar once you have a configurable in process test variant16:18
patchbotacoles: patch 276823 - openstack-infra/project-config - Add job gate-swift-tox-func-in-process-fast-post (MERGED)16:18
acolesjrichli: that would be great to have! and the env var referencing an alternate pipeline or conf section might be the most expedient way16:19
*** joeljwright has quit IRC16:19
acolesreminds me we should make that a voting job16:19
*** d0ugal has quit IRC16:20
*** d0ugal has joined #openstack-swift16:22
*** zaitcev has joined #openstack-swift16:22
*** ChanServ sets mode: +v zaitcev16:22
*** klrmn has joined #openstack-swift16:25
*** d0ugal has quit IRC16:27
*** asettle has quit IRC16:31
*** nadeem has joined #openstack-swift16:34
*** nadeem has quit IRC16:34
*** nadeem has joined #openstack-swift16:35
*** rledisez has quit IRC16:38
*** sams-gleb has joined #openstack-swift16:41
acolesclayg: morning!16:42
*** d0ugal has joined #openstack-swift16:42
*** esker has quit IRC16:42
acolesnotmyname: other cores, fyi patch 315665 proposes to make fast post func tests voting16:42
patchbotacoles: - openstack-infra/project-config - Make gate-swift-tox-func-in-process-fast-post voting16:42
acolesclayg: like I said to jrichli..."the hard part will be agreeing an acceptable way to configure the in process test pipeline"16:43
claygs/hard/impossible/ - ftfy16:43
*** silor has quit IRC16:44
*** McMurlock1 has quit IRC16:44
*** Jeffrey4l has quit IRC16:45
*** bill_az has quit IRC16:45
acolesright! I just didn't want to discourage her too much ;)16:46
*** dmorita has quit IRC16:47
*** gyee has joined #openstack-swift16:47
*** dmorita has joined #openstack-swift16:47
notmynamegood morning, again16:49
claygjrichli: acoles thinks your soft - I think you've stuck around too long with Swift you're not going to do something just because it's impossible.16:49
notmynamejrichli: did you say "crypto experts ... and reaperhulk"?!16:49
*** sams-gleb has quit IRC16:50
notmynameacoles: +1 to making that voting!16:52
acolesnotmyname: they are one and the same16:52
notmynameacoles: yeah, I know :-)16:52
notmynamejust made me laugh16:52
acolessometimes I'd like to go back and change my nick16:53
*** garthb_ has joined #openstack-swift16:53
notmynameto what?16:54
notmyname(you still could, of course)16:54
*** garthb has quit IRC16:55
claygnotmyname: maybe reaperhulk?16:57
*** vint_bra has quit IRC16:57
clayg"big al" apparently has some history in SF?16:58
acolesIDK, "loser" maybe16:58
clayghar har16:58
acolesclayg: really16:58
* acoles is googling16:58
acolesclayg: got it. clearly i don't get out enough!16:59
* acoles added that one to the vacation planner ;)17:01
*** diogogmt has joined #openstack-swift17:01
*** SkyRocknRoll has joined #openstack-swift17:02
*** mvk_ has quit IRC17:04
timburkegood morning17:05
*** lakshmiS has joined #openstack-swift17:06
tdasilvabig al: i 'm reviewing your patches now17:08
*** mingdang1 has joined #openstack-swift17:09
*** ChubYann has joined #openstack-swift17:09
*** mingdang1 has quit IRC17:13
openstackgerritTim Burke proposed openstack/swift: Add "history" mode to versioned_writes middleware
acolestdasilva: lol, thanks! sorry about the huge rebase noise on the wrapped key one :/17:17
tdasilvaacoles: no worries :)17:17
acolesgoo dnight17:20
*** acoles is now known as acoles_17:21
*** jistr has quit IRC17:24
openstackgerritTim Burke proposed openstack/swift: Use the same key for memcache and env['swift.infocache']
openstackgerritTim Burke proposed openstack/swift: Make info caching work across subrequests
*** garthb__ has joined #openstack-swift17:25
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
*** garthb_ has quit IRC17:26
*** SkyRocknRoll has quit IRC17:26
*** mmcardle has quit IRC17:28
openstackgerritTim Burke proposed openstack/swift: Add Vary: headers for CORS responses
openstackgerritTim Burke proposed openstack/swift: Autovivify X-Versions-Location container
timburkehurray copy middleware! hopefully all of those rebases are accurate17:33
*** d0ugal has quit IRC17:34
*** gyee has quit IRC17:35
*** NellyK has joined #openstack-swift17:36
*** SkyRocknRoll has joined #openstack-swift17:40
openstackgerritRichard Hawkins proposed openstack/swift: a well-commented skeleton middleware
*** sams-gleb has joined #openstack-swift17:47
*** d0ugal has joined #openstack-swift17:53
*** NellyK has quit IRC17:53
*** sams-gleb has quit IRC17:54
notmynamehurricanerix: thanks for keep working on that :-)17:55
hurricanerixnotmyname np, i'll follow up with those changes you mentioned.17:56
*** CaioBrentano has joined #openstack-swift17:56
tdasilvacool, step 1 - complete: and
*** tqtran has joined #openstack-swift17:58
*** suyash has joined #openstack-swift17:58
notmynametdasilva: great, thanks!17:58
notmynametdasilva: what's next?17:58
*** pcaruana has quit IRC17:58
tdasilvanotmyname: zuul jobs17:59
*** asettle has joined #openstack-swift17:59
notmynametdasilva: ack. gate-noop first? or is there actually some tests that can be run?17:59
tdasilvagate-noop first18:00
tdasilvano tests yet, planning to add that next18:00
tdasilvamy understanding is that we can only add tests for pyeclib thou, does that sound right?18:00
tdasilvasince liberasurecode is a C library18:00
*** klamath has quit IRC18:07
*** d0ugal has quit IRC18:08
*** rcernin has quit IRC18:09
openstackgerritMerged openstack/swift: Don't delete misplaced dbs if not replicated
notmynametdasilva: well...18:10
notmynametdasilva: I think it's possible to run the C tests. (actually not much different that what you might do for golang)18:10
notmynametdasilva: in this case, it's probably something like "take the bare image, install what you need, run some script in the repo that builds and run tests..."18:11
tdasilvanotmyname: ok, I was thinking more because of the whole infra only supports certain languages discussions, not so much because of the technical issues18:14
*** jistr has joined #openstack-swift18:15
tdasilvanotmyname: but I guess that's only for "offical" openstack projects...18:15
*** jistr is now known as jistr|afk18:15
tdasilvaanyway..I will play with it...18:15
jrichliclayg acoles: swift does have its share of "wicked problems"
*** openstackgerrit has quit IRC18:18
*** openstackgerrit has joined #openstack-swift18:18
*** hezhiqiang has quit IRC18:21
claygjrichli: so much of that rings so true18:23
claygjrichli: although i don't believe I'd ever seen that synopsis nor read that paper before - thank you!18:23
jrichliyw :-)18:24
jrichlinotmyname: i was including reaperhulk when i said crypto experts!18:24
claygjrichli: oh the research seems to study policy/social problems - odd that the top level principles apply so well to complex system/software design!18:25
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
jrichliclayg : yeah, seemed fitting at the surface.  i like "Every solution leaves traces that cannot be undone. You can't build a freeway to test if it works."18:26
jrichliand "Every wicked problem is a symptom of another problem. "18:26
jrichliAny solution generates waves of consequences that propagate forever.18:27
jrichliok, ill stop18:27
*** onovy has quit IRC18:28
*** onovy has joined #openstack-swift18:28
claygheh, the referenced paper is fairly short and itself short on methodolgy/references - their conclusions are less obviously correct than at first glance - still interesting18:29
*** rchurch has quit IRC18:36
*** esker has joined #openstack-swift18:38
*** esker has quit IRC18:39
*** esker has joined #openstack-swift18:39
*** awelleck has joined #openstack-swift18:46
*** rchurch has joined #openstack-swift18:46
*** dmk0202 has joined #openstack-swift18:47
torgomaticmahatic: I'm glad you found them useful :)18:47
*** sams-gleb has joined #openstack-swift18:51
notmynamejrichli: that is a great link.18:54
*** lyrrad has quit IRC18:55
jrichlinotmyname: when googling about it, there was a title i saw that mixed that concept with herding cats.  i didnt look into it, but thought that was interesting18:56
openstackgerritSamuel Merritt proposed openstack/swift: Fix up get_account_info and get_container_info
openstackgerritSamuel Merritt proposed openstack/swift: Make info caching work across subrequests
openstackgerritPaul Dardeau proposed openstack/swift: fix docs word usage for large obj copy
*** sams-gleb has quit IRC19:00
*** d0ugal has joined #openstack-swift19:02
*** SkyRocknRoll has quit IRC19:07
*** asettle has quit IRC19:09
hurricanerixzaitcev: I think these are the links you were looking for in that "supporting go" thread:
*** asettle has joined #openstack-swift19:13
zaitcevhurricanerix: The video is from panel, which didn't touch the specfic performance numbers, I think. But thanks for the presentation link, it seems like19:14
hurricanerixzaitcev np, i'll see if i can find the other video too19:14
*** bill_az has joined #openstack-swift19:18
openstackgerritMerged openstack/swift: Merge branch 'master' into feature/crypto
*** lyrrad has joined #openstack-swift19:20
openstackgerritMerged openstack/swift: README: fix broken link
hurricanerixzaitcev i am starting to wonder if that other one wasn't recorded because i can't find it either.19:20
zaitcevhurricanerix: Quite possibly... It was a Design Summit working session, not a presentation as such.19:20
*** SkyRocknRoll has joined #openstack-swift19:26
*** asettle has quit IRC19:35
*** jistr|afk has quit IRC19:52
*** pauloewerton has quit IRC19:55
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
*** sams-gleb has joined #openstack-swift19:57
*** asettle has joined #openstack-swift19:59
*** asettle has quit IRC19:59
*** asettle has joined #openstack-swift20:00
*** lyrrad has quit IRC20:00
*** asettle has quit IRC20:00
*** sams-gleb has quit IRC20:05
*** Guest28457 is now known as sc20:18
*** rcernin has joined #openstack-swift20:24
*** lyrrad has joined #openstack-swift20:32
*** raildo is now known as raildo-afk20:41
*** lyrrad_ has joined #openstack-swift20:43
*** lyrrad has quit IRC20:43
*** lyrrad_ is now known as lyrrad20:43
*** gyee has joined #openstack-swift20:44
openstackgerritMerged openstack/swift: Stop complaining about auditor_status files
openstackgerritAndrew Welleck proposed openstack/python-swiftclient: Query string functionality for containers
*** NM has quit IRC20:52
openstackgerritSamuel Merritt proposed openstack/swift: Fix up get_account_info and get_container_info
openstackgerritSamuel Merritt proposed openstack/swift: Make info caching work across subrequests
ober37Does anyone know if there are any restrictions in using the allow_account_management setting with Keystone v2 auth.    There is a comment in the functional test sample.conf file that would imply this.   I was curious if there is a limitation in this implementation or if its just something with the functional tests themselves...20:57
claygnotmyname: if you run account auto-create do you like *have* to have allow_account_management true?21:01
claygi always seem to forget how exclusive those options are21:01
*** sams-gleb has joined #openstack-swift21:01
clayglike I sorta *know* what account_autocreate does - but I always forget exactly how allow_account_management is implemented21:01
ober37clayg: I didn't think so, I thought that they were exclusive.  I think the allow* just allows for the PUT / DELETE from an authorized user21:02
ober37clayg: What confuses me is the connection with this and the comment of not allowing for v2 auth, I don't see why this restriction exists.21:02
claygober37: either way I'd guess they bot need to be true where authorized identity is provided by a seperate service - almost noone (anymore?) integrates a seperate "provision swift account" step when a new identity is added21:02
claygober37: can you ref the comment?21:03
ober37clayg: It's part of test/sample.conf lines 8-921:03
notmynameyeah, ober37 is right. allow_ isn't needed if autocreate is on21:03
claygmaybe this has some clues?
patchbotclayg: patch 262664 - swift - Add a note for functional tests with Keystone. (MERGED)21:05
ober37clayg: Thank you!   I'll take a look21:06
clayghow does test.functional.tests.TestAccount.testPUT not *always* fail when allow_account_management is true?21:07
claygdoesn't seem like keystone really matters - that test is really specific21:07
*** sams-gleb has quit IRC21:09
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
ober37clayg: Yeah I just read the patch and I think I would agree with you.    It seems that the funky behavior is that PUT will not succeed since autocreation will take care of it (at least that's my understanding).   I am not sure v2 vs v3 Keystone should make a difference.  Or at least it's not clear to me how it would21:11
ober37clayg: i guess in theory someone could disable account autocreation AND enable allow_account_management with Keystone and this test case would work...21:12
*** lakshmiS has quit IRC21:13
claygober37: i'm positive v2/v3 auth is unrelated - the comment about v2 auth and the comment about allow_account_management are orthogonal21:13
claygeither way I think the comment about allow_account_management applies to the entire suite - which is sort wonky21:13
ober37clayg: yeah.    It really seems like it's more of a conflict between using Keystone + autocreation OR Keystone + manual creation.   That if you enable autocreate + manual management, functional tests will fail.21:14
ober37(not all of them obviously)21:14
ober37clayg: Is that your impression as well?21:16
claygI think the test suite will always fail, regardless of auth system, if you change that option in the proxy config - because the test is asserting the behavior of the default configuration21:18
*** mmcardle has joined #openstack-swift21:19
*** mmcardle1 has joined #openstack-swift21:21
ober37clayg: Yeah you are right.21:22
ober37clayg: This test probably should look at the setting for allow_account_management and adjust what it verifies against based on that setting.21:24
*** mmcardle has quit IRC21:24
*** mingdang1 has joined #openstack-swift21:24
*** awelleck has quit IRC21:28
ober37clayg: Thanks for your help.  I'll look more at this tomorrow morning and will plan on providing a patch around this test case to adjust what status it is verifying based on this setting21:30
*** vint_bra has joined #openstack-swift21:31
*** ametts has quit IRC21:33
*** nadeem has quit IRC21:39
*** arch-nemesis has quit IRC21:46
*** dmk0202 has quit IRC22:00
*** cdelatte has joined #openstack-swift22:04
*** sams-gleb has joined #openstack-swift22:06
openstackgerritMerged openstack/swift: go: shuffle suffix dirs when replicating a partition
*** CaioBrentano has quit IRC22:11
*** sams-gleb has quit IRC22:15
*** mmotian__ has joined #openstack-swift22:19
*** lakshmiS has joined #openstack-swift22:24
*** jamielennox|away is now known as jamielennox22:24
*** SkyRocknRoll has quit IRC22:32
openstackgerritTim Burke proposed openstack/python-swiftclient: Accept gzip-encoded API responses
*** mmotian__ has quit IRC22:42
*** geaaru has quit IRC22:42
*** lakshmiS has quit IRC23:01
timburkenotmyname: want to +A at the same time?23:04
patchbottimburke: patch 306661 - swift - Tighten up ResumingGetter.fast_forward23:04
notmynametimburke: no. torgomatic was just standing over me threatening me about the other one ;-)23:05
notmynametimburke: so this one looks like some efficiency improvements?23:06
timburkemainly some sanity checks. shouldn't really come up in practice, but the NotSatisfiable/RangeAlreadyComplete exceptions seem appropriate if it ever came up23:08
notmynameyeah. lgtm23:08
*** sams-gleb has joined #openstack-swift23:12
notmynameI just noticed that our code coverage has gone up slightly. used to be 91%. now it's 92% (on unit tests)23:13
timburkenotmyname: looks like recent feature/crypto brings it up to 93%
timburkegood job acoles_, jrichli, mahatic!23:17
*** mingdang1 has quit IRC23:18
*** sams-gleb has quit IRC23:19
*** garthb__ has quit IRC23:25
*** cdelatte has quit IRC23:25
*** _JZ_ has quit IRC23:27
*** j_king has quit IRC23:29
*** j_king has joined #openstack-swift23:29
*** mahatic has quit IRC23:31
*** mahatic has joined #openstack-swift23:32
*** kei_yama has joined #openstack-swift23:34
*** chlong has quit IRC23:35
*** mmotian__ has joined #openstack-swift23:41
*** mmotia___ has joined #openstack-swift23:42
*** mmotian__ has quit IRC23:42
*** vint_bra has quit IRC23:43
*** mmotian__ has joined #openstack-swift23:44
*** mmotia___ has quit IRC23:46
*** mmotia___ has joined #openstack-swift23:47
*** hosanai has joined #openstack-swift23:47
*** ChanServ sets mode: +v hosanai23:47
*** mmotian__ has quit IRC23:50
*** mmcardle1 has quit IRC23:51
openstackgerritMerged openstack/swift: fix docs word usage for large obj copy
*** mmcardle has joined #openstack-swift23:52
*** NM has joined #openstack-swift23:56
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements
*** asettle has joined #openstack-swift23:56
torgomaticheh, global requirements just hit patch set 42323:57
*** mmotian__ has joined #openstack-swift23:58
torgomaticand it turned two years old last month23:58
*** mmoti____ has joined #openstack-swift23:58

Generated by 2.14.0 by Marius Gedminas - find it at!