Monday, 2017-06-05

*** tovin07_ has joined #openstack-swift00:35
*** zhengyin has joined #openstack-swift00:54
*** chsc has joined #openstack-swift00:57
*** chsc has joined #openstack-swift00:57
*** chsc has quit IRC01:07
*** vint_bra has quit IRC01:19
*** janonymous has joined #openstack-swift01:33
*** zhurong has joined #openstack-swift01:33
kota_good morning01:41
mattoliveraukota_:  morning01:43
*** zhurong has quit IRC01:44
kota_mattoliverau: o/01:48
*** tovin07_ has quit IRC01:57
*** tovin07_ has joined #openstack-swift02:05
*** zhurong has joined #openstack-swift02:32
*** winggundamth has joined #openstack-swift02:52
*** zhengyin has quit IRC03:05
*** zhengyin has joined #openstack-swift03:06
*** mingyu has joined #openstack-swift03:34
*** psachin has joined #openstack-swift03:42
*** Dinesh_Bhor has joined #openstack-swift04:03
*** mingyu has quit IRC04:10
*** mingyu has joined #openstack-swift04:11
mahaticgood morning04:11
mahatickota_: mattoliverau04:11
mahatico/04:12
kota_mahatic: o/04:12
*** mingyu has quit IRC04:15
*** klrmn has quit IRC04:15
mattoliveraumahatic: o/04:28
*** mingyu has joined #openstack-swift05:02
*** zhengyin has quit IRC05:05
*** zhengyin has joined #openstack-swift05:05
*** Fdaisuke_ has joined #openstack-swift05:10
*** Fdaisuke_ has quit IRC05:11
*** Fdaisuke has joined #openstack-swift05:11
*** Fdaisuke has left #openstack-swift05:12
*** mingyu has quit IRC05:19
*** cshastri has joined #openstack-swift05:44
* kota_ is looking at the follow up patch for proxy config per policy05:54
*** mingyu has joined #openstack-swift06:08
*** mingyu has quit IRC06:14
*** mingyu has joined #openstack-swift06:17
*** chsc has joined #openstack-swift06:22
*** chsc has joined #openstack-swift06:22
*** skudlik has joined #openstack-swift06:27
*** rcernin has joined #openstack-swift06:29
*** chsc has quit IRC06:31
*** geaaru__ has quit IRC06:50
*** mattoliverau has quit IRC06:50
*** hoonetorg has quit IRC06:50
*** samueldmq has quit IRC06:50
*** amit213 has quit IRC06:50
*** clayg has quit IRC06:50
*** geaaru__ has joined #openstack-swift06:50
*** hoonetorg has joined #openstack-swift06:50
*** mingyu has quit IRC06:50
*** samueldmq has joined #openstack-swift06:50
*** amit213 has joined #openstack-swift06:51
*** clayg has joined #openstack-swift06:51
*** ChanServ sets mode: +v clayg06:51
*** mattoliverau has joined #openstack-swift06:56
*** ChanServ sets mode: +v mattoliverau06:56
*** mingyu has joined #openstack-swift07:02
*** pcaruana has joined #openstack-swift07:04
*** oshritf has joined #openstack-swift07:16
*** mvpnitesh has joined #openstack-swift07:20
*** zhurong has quit IRC07:31
*** mingyu has quit IRC07:38
*** zhurong has joined #openstack-swift07:40
*** mingyu has joined #openstack-swift07:52
*** bkopilov has quit IRC07:53
acolesgood morning07:58
*** bkopilov has joined #openstack-swift08:02
acolesasettle: hi, you around? I wanted to confirm something I am sure I have probably asked you before (sorry!) - the install guides get published soon after new release, and then are frozen i.e. they don't get updated automatically as we patch the source, correct?08:03
mattoliverauacoles: morning08:07
acolesmattoliverau: o/08:15
*** mingyu has quit IRC08:18
*** jaosorior has joined #openstack-swift08:20
*** zhurong has quit IRC08:20
*** mingyu has joined #openstack-swift08:23
asettleacoles: heya!08:25
asettleacoles: we tend to freeze about 6 weeks *before* the release, when we start testing packages. Hard freeze probably about a month before release. All related here: https://docs.openstack.org/contributor-guide/release/taskdetail.html#installation-tutorial-testing08:26
asettleThe guides are released with the general OpenStack release time frame (so, from master, it's then published)08:26
asettleAnd then a month afterwards we cut the branch - so we then would (for example) have the Ocata branch, and new content starts being done on master08:27
acolesasettle: ah, got it. thanks. I'm just wondering how to label a bug on swift ocata install guide where it refers to the release (newton). We fixed upstream but I guess that won't get published until pike.08:29
asettleI'd probably just tag it with that install-ocata-swift or something similar. (whatever your conventions may be)08:31
*** mingyu has quit IRC08:44
*** mingyu has joined #openstack-swift08:45
*** mingyu has quit IRC08:47
*** ^andrea^ has joined #openstack-swift08:49
*** ^_andrea_^ has joined #openstack-swift08:49
*** ^andrea^ has quit IRC08:49
openstackgerritMerged openstack/swift master: Follow-up for per-policy proxy configs  https://review.openstack.org/46695208:50
*** zhurong has joined #openstack-swift08:50
*** eckesicle has joined #openstack-swift08:51
*** cshastri has quit IRC09:32
openstackgerritRomain LE DISEZ proposed openstack/swift master: Delay port binding to reduce wait at process start  https://review.openstack.org/47090309:44
*** jaosorior has quit IRC09:49
*** mingyu has joined #openstack-swift09:49
*** cshastri has joined #openstack-swift09:49
*** jaosorior has joined #openstack-swift09:53
*** zhurong has quit IRC09:59
*** tovin07_ has quit IRC10:00
*** cshastri has quit IRC10:06
*** mingyu has quit IRC10:08
*** zhurong has joined #openstack-swift10:12
*** cshastri has joined #openstack-swift10:17
*** mat128 has joined #openstack-swift10:27
*** mat128 has quit IRC10:29
*** mat128 has joined #openstack-swift10:34
*** mwheckmann has quit IRC10:38
*** mwheckmann has joined #openstack-swift10:46
hugokuoHave we addressed how object-replicator stuck there without doing anything yet? The process is not dead just keep waiting.10:51
*** zhengyin has quit IRC11:39
*** links has joined #openstack-swift12:04
*** zhurong has quit IRC12:04
*** mdrabe has joined #openstack-swift12:27
openstackgerritMatthew Oliver proposed openstack/swift master: Make eventlet.tpool's thread count configurable in object server  https://review.openstack.org/28966412:32
*** eckesicle has quit IRC12:35
*** catintheroof has joined #openstack-swift12:35
*** MVenesio has joined #openstack-swift12:38
*** mvpnitesh has quit IRC12:45
*** mvpnitesh has joined #openstack-swift12:45
*** kei_yama has quit IRC12:55
*** winggundamth has quit IRC13:01
*** mvpnitesh has quit IRC13:02
*** lucasxu has joined #openstack-swift13:17
*** klamath has joined #openstack-swift13:21
*** klamath has quit IRC13:22
*** klamath has joined #openstack-swift13:23
*** kozhukalov has quit IRC13:28
*** zhurong has joined #openstack-swift13:46
asettleacoles: heyyyyyy13:59
asettleQuestion for you this time13:59
asettleDid you guys do this? https://docs.openstack.org/admin-guide/cli-analyzing-log-files-with-swift.html13:59
asettle"you guys" = swift13:59
asettleIs this even necessary?14:00
acolesasettle: I don't recognize it14:00
asettleacoles: would there be concern about deletion?14:00
acolesasettle: you mean, if the client deleted the log files?14:02
asettleIf I deleted that page :p14:02
acolesOIC lol14:02
asettleHahaha14:02
acolesdoes git blame show likely author?14:03
asettleIt does not :(14:03
acolesasettle: I suggest checking with notmyname when he is awake, I wouldn't delete based solely on me not knowing14:04
asettleGood call ;)14:04
*** mingyu has joined #openstack-swift14:06
jrichliacoles, mahatic, rusik : have any of you started work on lp bug #1691807 ?14:16
openstackLaunchpad bug 1691807 in OpenStack Object Storage (swift) "object decryption should be able to validate that correct key is being used" [Wishlist,New] https://launchpad.net/bugs/1691807 - Assigned to Ruslan Gustomyasov (rusik)14:16
acolesjrichli: hi! not me14:16
jrichliacoles: o/ do you know Ruslan?14:17
acolesno, not seen the nick before14:17
mahaticjrichli: hello! not me either. But I've thought about it a bit, I might just drop some notes there though14:17
mahaticacoles: o/14:17
acolesjrichli: I'd like to see it progress but not had time myself14:18
*** vint_bra has joined #openstack-swift14:18
jrichlimahatic: that would be great.14:18
jrichliI know I have been a bit absent from swift, but I *really* hope that I will be able to start up again this week.14:18
jrichliso, I'll take a look and see what I can do.  I will put a comment on the bug to see if Ruslan has made some progress.  thanks!14:19
mahaticjrichli: great, thanks too14:19
acolesjrichli: great :D14:29
*** psachin has quit IRC14:31
*** psachin has joined #openstack-swift14:34
*** ukaynar has joined #openstack-swift14:41
*** psachin has quit IRC14:47
*** skudlik has quit IRC14:47
*** cshastri has quit IRC14:48
*** skudlik has joined #openstack-swift14:51
*** ukaynar_ has joined #openstack-swift14:54
*** ukaynar has quit IRC14:57
*** chsc has joined #openstack-swift15:02
*** gyee has joined #openstack-swift15:04
*** skudlik has quit IRC15:04
*** Renich has joined #openstack-swift15:13
clarkbasettle: acoles looks like joseph robinson added the file about a year ago. I ran git log -p on the file say that it got renamed, checked out the commit before the commit that renamed then checked history on the old filename. yay git15:13
clarkboh wait that may be another rename /me digs deeper15:14
RenichHello! o/15:14
asettleclarkb: oooo thanks. I couldn't find anything, but then again, I was doing a bit of a cursory glance while on a meeting15:14
RenichI am trying to set quotas to a project15:14
asettleclarkb: it just seems so out of place, and with all the doc restructuring due to happen, I question its relevance15:14
asettleAnd if so, who it belogns to15:15
Renichbut, even, though, openstack project list shows the project and user list show's the user in the project, I can't seem to show or set the quota15:15
RenichI always get "need more than 0 values to unpack15:15
Renich"15:15
*** zhurong has quit IRC15:15
*** chsc has quit IRC15:16
clarkbasettle: it seems like that file has moved many times over the last couple years... I think I have found 4 moves and still haven't found the origin of it15:16
clarkbasettle: which would support the idea that its not well placed?15:16
asettleclarkb: would explain *a lot*15:16
asettleThanks for diving into this :)15:16
*** aselius has joined #openstack-swift15:16
*** mvk has quit IRC15:19
clarkbasettle: I368af8a7b843b1f1cba0d8a2ccdaa0f40634db3d added it in 2011 as part of the swift admin guide15:23
asettleWoah dude you went so deep15:24
asettleAnd it's been moved around since then, huh15:24
asettleThank you, clarkb :)15:24
asettleI guess now I'm wondering how relevant it is to keep it.15:24
asettleCause it seems a little out of... place... still15:24
clarkbyes something like at least 10 moves15:24
clarkbI stopped counting15:25
asettle*head desk*15:25
asettleTypical docs team ;P15:25
asettleWe can't keep anything in one place15:25
*** MVenesio has quit IRC15:31
*** ukaynar has joined #openstack-swift15:34
*** lucasxu has quit IRC15:35
*** ukaynar_ has quit IRC15:37
*** mingyu has quit IRC15:37
*** mingyu has joined #openstack-swift15:39
claygredbo - do you subscribe the sqlite ML?  http://sqlite.1065341.n5.nabble.com/WAL-checkpoint-starved-td96011.html  <- about WAL checkpoint starved - I'm not sure there's a resolution suggested yet in the thread 🍿15:40
*** Renich has quit IRC15:42
*** mingyu has quit IRC15:43
*** chsc has joined #openstack-swift15:51
*** chsc has joined #openstack-swift15:51
*** rcernin has quit IRC15:52
redboWhat are we talking about?  My theory is if I grab an exclusive lock on the db, that'll make sure there aren't any readers before I checkpoint.15:53
*** chsc has quit IRC15:56
claygumm... but are you going to grab an exclusive lock before every insert?  I thought the idea was that maybe the WAL insert could eventually be better/faster/more-robust than the .pending db bulk insert stuff - but maybe we far too coupled into that now anyway...15:58
claygthe git history has the legacy of trying wal and then giving up - IIRC the issue was that the WAL would keep growing and never force the inserts - the .pending approach with it's .stat to check the size never seems to have that problem16:00
redboRight now it only forces a checkpoint during replication.16:02
*** Renich has joined #openstack-swift16:04
redboThere's no reason to force checkpoints constantly.  It just needs to not grow for forever if someone is accessing the database so much it can't auto checkpoint.16:05
redboand also when it needs to make one file to rsync in replication16:05
*** oshritf has quit IRC16:09
Renichguys, I need help setting a quota for a project or account.16:11
RenichI am using v316:11
RenichThis example: https://docs.openstack.org/ops-guide/ops-quotas.html is not helping. It doesn't work when I source the user's credentials16:12
RenichAnd I've been trying to source the admin's credentials and point it to the project, but I can't find the right combination of flags16:13
claygoh oh oh... I understood this one time...16:13
claygRenich: if you have "admin credentials" you should just need --os-storage-url pointed at the users' account16:13
claygRenich: swift post --os-storage-url https://storage.example.com/v1/AUTH_<project_id> ...16:14
Renichclayg: OK. i will try that. But, quite frankly, I think env vars are poluting the command somehow16:14
claygrun with -v or --debug to get more context16:15
Renichclayg: ah! it worked!16:15
claygnowhey!16:15
Renich... I am confused...16:15
claygs'ok - most of us are16:15
Renichclayg: but, hey, thanks a ton. I've been dealing with this for an hour or more16:15
RenichLOL16:16
claygthat sucks!16:16
Renichyeah16:16
RenichThanks a bunch, man. I appreciate the help16:16
claygwhere was the first place you looked and what can we make it say so that you don't waste your time?16:16
Renichyou mean logs? I tried -vv16:17
Renichdidn't really look at the logs16:17
RenichAlso, tried using openstack quota set/show but couldn't. It seems that it wants a networking endpoint to be avaialble, and I have none16:17
claygno... i mean... idk - presumably there's like instructions for this somewhere?  how did you know you needed quotas or that they like exited or whatever and that ... i don't know?16:17
Renich...16:18
clayg... existed ...16:18
RenichNo network endpoint in service catalog16:18
Renichneed more than 0 values to unpack16:18
Renichclean_up ShowQuota: need more than 0 values to unpack16:18
Renichclayg: ah, you mean https://docs.openstack.org/ops-guide/ops-quotas.html16:18
Renichand https://docs.openstack.org/admin-guide/cli-set-quotas.html16:18
clayggah!?  what is that?  something with the openstack cli?  I have no idea if it supports ...16:18
Renichclayg: yes, openstack cli.16:19
claygyeah, I think that's what I was looking for - maybe we can improve that doco there - thanks!16:19
Renichwell, basically, we're running a kind-of-beta for openstack-swift only. And, some clients, are testing it. So we're trying to figure out management. Kind of hard, IMHO.16:20
Renichclayg: thanks for that! ;)16:20
claygyeah, I think you'd find an interested audience here to consume your first hand experience and a stack ranking of the hardest parts16:21
claygthanks for stopping by - come ask any questions as they pop up - people are always around16:22
Renichyep, thanks a lot.16:22
RenichI will16:22
*** pcaruana has quit IRC16:23
claygasettle: you know how you and notmyname are always talking about doc trees and yaml and stuff that's in our repo and like other repos and stuff with something something docs?16:23
claygdo you have any idea where https://docs.openstack.org/ops-guide/ops-quotas.html or https://docs.openstack.org/admin-guide/cli-set-quotas.html live?  Are either of them in swift's tree?16:24
notmynamegood morning16:24
asettleclayg: yo sure do16:24
asettleclayg: https://github.com/openstack/openstack-manuals/tree/master/doc/ops-guide/source16:24
asettlehttps://github.com/openstack/openstack-manuals/blob/master/doc/ops-guide/source/ops-quotas.rst16:24
asettleandddd16:24
asettlehttps://github.com/openstack/openstack-manuals/blob/master/doc/admin-guide/source/cli-set-quotas.rst16:25
asettleclayg: openstack-manuals repo :)16:25
notmynameasettle: no, i don't knwo about https://docs.openstack.org/admin-guide/cli-analyzing-log-files-with-swift.html. analyzing logs is a common thing for ops. might be interesting to add it to https://docs.openstack.org/developer/swift/logs.html if you don't want it anymore16:25
claygah, coolio - thanks!16:25
asettlenotmyname: well, tbh, we just don't know what to do with it. If you want it, happily will pass it along. It's not not useful. Just. Dunno what it's doing there :P16:26
asettleclayg: no problemo :)16:26
notmynameasettle: after having spent 30 seconds considering it, I'd be sad to see it go away, but I have no strong feelings on where it should live16:28
*** lucasxu has joined #openstack-swift16:28
asettlenotmyname: hahaha okay. well. I'll get back to you ;)16:28
*** links has quit IRC16:35
*** ukaynar has quit IRC16:39
*** honga has joined #openstack-swift16:39
*** ukaynar has joined #openstack-swift16:40
timburkegood morning16:40
*** honga has quit IRC16:42
*** honga has joined #openstack-swift16:42
*** chsc has joined #openstack-swift16:44
*** ukaynar has quit IRC16:44
openstackgerritTim Burke proposed openstack/swift master: Add validation for sorting_method values  https://review.openstack.org/46926916:46
clarkbclayg: got it down to 16 fails with a change to test.conf that sets web_front_end to apache2 http://logs.openstack.org/74/372374/30/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/4a1d2a0/console.html16:51
clayglol - briancli1e you'd already submitted patch 185221 years ago :P16:53
patchbothttps://review.openstack.org/#/c/185221/ - swift - Add object replicator progress, heartbeat to recon16:53
claygclarkb: lolwut?16:54
claygclarkb: there's a test.conf option specifically for apache?16:54
clarkbclayg: yesm some of the tests branch on the web_front_end config value16:54
clarkblooks like portante may have added some of that?16:55
timburkei... apache2 doesn't support chunked PUTs? https://github.com/openstack/swift/blob/master/test/functional/tests.py#L2482-L248516:56
clarkbtimburke: there are some places where it asserts different responses too isntead of skipping tests. I'm not sure what all the details are, but since apache2 is terminating tls for us I just set the web_front_end config and 4 tests no longer a problem >_>16:58
openstackgerritClark Boylan proposed openstack/swift master: Only assert connection header if set in request  https://review.openstack.org/47105717:02
clarkbclayg: and I think ^ shoudl address another couple failures, though not sure if that assertion is important there and needs to be solved some other way17:02
*** jaosorior is now known as jaosorior_away17:03
*** klrmn has joined #openstack-swift17:04
*** lucasxu has quit IRC17:13
*** lucasxu has joined #openstack-swift17:14
claygclarkb: it was David Hadas -> https://review.openstack.org/#/c/23585/17:14
patchbotpatch 23585 - swift - Support tests for Apache (MERGED)17:14
claygportante: was just refactoring the pre-existing configuration17:15
clarkbah17:15
portanteclayg, clarkb: anything I can help with, or muddy, as the case may be?17:15
portante;)17:16
RenichOK, it is not working actually. I tried it again step by step and it didn't work.17:20
RenichAccount POST failed: http://myurl:8080/v1/AUTH_someID 403 Forbidden  [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc17:20
RenichFailed Transaction ID: some-transactionID17:20
openstackgerritTim Burke proposed openstack/swift master: Add validation for sorting_method values  https://review.openstack.org/46926917:22
clarkbclayg: hey for the header size limit thing, it looks like swifts default size limit is 8192, apache's is 8190. Do we just need to bump the apache limit up to the swift limit of 8192?17:29
clayglol - maybe?17:35
*** MVenesio has joined #openstack-swift17:35
*** ukaynar has joined #openstack-swift17:38
*** tonanhngo has joined #openstack-swift17:39
*** lucasxu has quit IRC17:51
*** mat128 has quit IRC17:52
*** lucasxu has joined #openstack-swift17:52
*** mat128 has joined #openstack-swift17:53
*** mat128 has quit IRC17:54
*** mat128 has joined #openstack-swift17:55
openstackgerritClark Boylan proposed openstack/swift master: Func test hacks to work under against apache2  https://review.openstack.org/47105718:01
*** lucasxu has quit IRC18:08
*** lucasxu has joined #openstack-swift18:09
*** mvk has joined #openstack-swift18:17
*** ukaynar has quit IRC18:19
*** ukaynar has joined #openstack-swift18:20
RenichI was reading https://github.com/openstack/openstack-manuals/blob/master/doc/ops-guide/source/ops-quotas.rst and it says: "To update Object Storage quotas on a project, you must have the role of ResellerAdmin in the project that the quota is being applied to.", but that role doesn't even exist here. Only _member_, testrole, admin and user.18:22
RenichIt seems odd that admin can't change quotas here and there18:22
Renich:S18:22
* Renich will try to find out how to create the ResellerAdmin role18:23
notmynameRenich: are you using keystone/18:24
notmyname?18:24
Renichnottrobin: yes18:24
Renichnotmyname: yes18:24
Renichsorry nottrobin18:24
Renichnotmyname: v318:24
*** ukaynar has quit IRC18:24
notmynameRenich: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L39818:24
notmynameRenich: that's where "reseller admin" is defined for swift+keystone18:24
Renichah! Cool. So enable that... then what?18:25
RenichDo I need to create the role or something after it?18:26
RenichOr join the role?18:26
notmynamesorry, just lost internet for a moment. not sure if I missed something18:26
Renichnotmyname: I'll repeat :)18:27
RenichSo enable that... then what?18:28
RenichDo I need to create the role or something after it?18:28
RenichOr join the role?18:28
*** ukaynar has joined #openstack-swift18:28
*** honga has quit IRC18:28
notmynameright18:32
openstackgerritThiago da Silva proposed openstack/liberasurecode master: Release 1.5.0  https://review.openstack.org/47108418:32
notmynameso that is the config var that tells swift which role (returned from keystone) to check18:32
notmynameRenich: go back up to line 372 and read the full comments. might give a fuller perspective18:33
notmynametdasilva: nice!18:33
Renichnotmyname: thanks, I will18:33
tdasilvanotmyname: yeah, tons of fixes on that release18:33
notmynameRenich: the basic flow is that swift gets a request with a token, swift asks keystone what roles that token has, then swift sets persmissions based on those returned roles. so if one of the returned roles is the same as what's configured in that reseller_admin_role setting, then swift (internally) sets the "swift owner" flag and allows the request18:34
tdasilvaas clayg likes to say: 'this will be the best libec release ever' ... or something like that.... ;)18:34
claygtdasilva: *close* - the best one *yet*18:34
Renichnotmyname: ok18:34
notmynameRenich: and with reseller admin, it just means that it will *always* set the swift owner flag, regardless of which swift account is being used18:35
tdasilvaclayg: yeah, that's better! :)18:35
notmynameRenich: well, "always, for the accounts with the configred `reseller_prefix`" ;-)18:35
Renichhah! it works!18:40
notmynameyay!!18:41
Renichhad to add reseller_admin_role and reseller_prefix18:41
Renichthen, create the role (openstack role add)18:41
Renichthen, add admin to role18:41
Renichnow, swift stat --os-storage-url some-url works!18:42
notmynamewonderful18:42
RenichIMHO, the documentation should have some kind of emphatic note regarding ResellerAdmin18:42
Renichsome warning or big, yellow note letting you know it won't work unless you have this setup18:42
Renichsince the default is not to have it18:43
notmynameRenich: (mentioning for the sake of my own conscience) using the Swift API with a reseller admin is just like logging in to linux as root. makes some things simpler, but it's also possible to *really* ruin your day. be careful, and use reseller admin only sparingly18:43
notmynameRenich: the improved docs are a great idea. where should they go?18:44
Renichnotmyname: wow, OK. Anything like sudo, then? ;D18:44
notmynameno, not quite18:44
Renichnotmyname: they should go here https://docs.openstack.org/ops-guide/ops-quotas.html18:45
Renichnotmyname: from the service provider PoV, I think there should be a way for admins to administer other accounts. Up to now, we have been keeping credential files in order to do whatever we need for the beta-test clients18:45
notmynameRenich: hmm... I think that ops guide is a repo that the docs team manages. see the bug icon in the top right? click that and write up what you want to see and it will get done :-)18:45
Renichnotmyname: will do18:46
notmynamethanks18:46
notmynamethere's three "levels" of swift accounts18:46
notmynamethe reseller admin is like root (we've covered this one)18:46
notmynamethe reseller admin is also the only one that can send a DELETE to a whole account18:46
*** ukaynar has quit IRC18:48
notmynamethe "normal" level of access in swift is where you (the api user) have full access to your account18:49
notmynamewhile not limited to this, it might be easiest to think in terms of a public cloud provider: you sign up, get an account, and do whatever you want there (but you cant' do anything in *my* account)18:49
notmynamethe third "level" of access in swift would be where you only have access to things you've specifically been granted access to18:50
Renichnotmyname: thank you for taking the time to explain. I've posted a bug already: https://bugs.launchpad.net/openstack-manuals/+bug/169596118:53
openstackLaunchpad bug 1695961 in openstack-manuals "Quotas in Operations Guide" [Undecided,New]18:53
Renichnotmyname: and thanks for all the help, really. I appreciate it.18:53
*** cschwede has quit IRC18:57
clarkbclayg: hrm going to 8192 bytes of header in apache made test_invalid_acls fail18:59
*** cschwede has joined #openstack-swift18:59
clarkboh is that for an individual header? not in aggregate?19:00
clarkbthat would explain it19:00
openstackgerritClark Boylan proposed openstack/swift master: Func test hacks to work under against apache2  https://review.openstack.org/47105719:00
*** mat128 has quit IRC19:04
notmynameclarkb: yeah, 8k per header19:04
notmynameclarkb: lots of works around the different constraints on https://github.com/openstack/swift/blob/master/etc/swift.conf-sample#L9719:04
*** zaitcev has joined #openstack-swift19:06
*** ChanServ sets mode: +v zaitcev19:06
clarkbnotmyname: ya trying to sort out why the second requset in test_invalid_acls is a 400 instead of a 204. The first one is more than 8192 bytes bu the second should be 8192 - 1024 + json overhead19:07
*** ukaynar has joined #openstack-swift19:14
*** lucasxu has quit IRC19:15
*** lucasxu has joined #openstack-swift19:34
*** Renich has quit IRC19:44
*** rcernin has joined #openstack-swift19:49
*** MVenesio has quit IRC20:05
*** lucasxu has quit IRC20:22
clarkbclayg: down to two failures now http://logs.openstack.org/74/372374/31/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/210c2f8/ using the hacks in https://review.openstack.org/#/c/471057/3/test/functional/tests.py20:40
patchbotpatch 471057 - swift - Func test hacks to work under against apache220:40
claygclarkb: that's amazing great work!20:40
openstackgerritClark Boylan proposed openstack/swift master: Func test hacks to work under against apache2  https://review.openstack.org/47105720:40
clarkbclayg: as predicted a big thing is 304's and not returning accept-ranges headers20:41
claygdoesn't seem so bad, yeah those accept headers on 304 responses are a pretty well understand - timburke will probably push towards just pulling them out of the responses altogether - I think first step is definitely fixing those20:44
claygI'm not 100% sure I understand the situation with the connection headers - I think we definitely want to validate the devstack default tls configuration is supporting connection keep-alive the way http >=1.0 servers *should* and swift clients would expect them to for the benifit of request piplineing if we're getting some signal that it might not be quite right20:45
clayg... but that may be orthogonal to getting functests passing - it may not be even be functests jobs to make sure request pipelining works... but it sorta *seems* like it would be?20:46
clayglike we've made changes to our application that can break request pipeline for better or worse - and we want to keep it working - so if functests can't validate that it feels like some sort of failure of the testing infrastructure/configuration20:47
*** rcernin has quit IRC20:47
*** lucasxu has joined #openstack-swift20:47
clarkbclayg: connection headers are explicitly set to close in devstack because python requests doesn't handle keepalives properly20:48
claygthe filesize limit could easily just be apache kicking back an error earlier than the tests were expecting to get one from the eventlet server...20:48
clarkbclayg: requests races on making new requests over keepalive'd connections that have been closed by the server20:48
clarkbwhich is why that fails right now. Generally I don't think its a good idea for swift's tests to make sure apache is configured one way or another when it comes to things like keepalives. Is fine to check the built in web server though20:49
clarkbclayg: another option there is to enable request pipelining with swift, assuming swiftclient isn't vulnerable to the issue in requests20:51
claygmaybe... I think the web_front_end flag is ridiculous personally - the functests should validate the api works the way clients expect it to - if there's differences between one configuration and another clients can't count on that behavior - so it's either not part of the api worth validating or it needs to be exposed in discoverability20:52
clarkb++20:52
clarkbworth noting that I don't expect that change of mine to merge as is. I just wanted to be able to have these conversations with some concrete code. Otherwise its harder to talk about :)20:53
claygas far as requests not working with keepalive or request pipelineing ... that's a pretty terrible claim to levy - are you sure there's maybe not a little more subtlety to the story there?  Swift really wants to ensure that request pipelining works - if that means bypassing python-requests/swiftclient and using HTTP(s)client directly so be it.20:54
clarkbclayg: my understanding of the issue is that the race exists where the server can close a keepalive'd connection due to a timeout, and if at the same time you get a new request in the web server will error saying the connection is closed. The client should then reopen a new connection and run requests over that. Instead requests just raises an exception that no one handles htemselves20:55
clarkbso the end result is whatever api call you were making fails even though it was not a fatal error, due to lack of retrying20:56
clarkbI think rquests does handle the case of connection is dropped cleanly before attempting next request though20:58
clarkbimplying it should also handle this corner case. Its a been a few months since we poked at it though20:58
claygok, that sounds a lot more reasonable than "python requests doesn't handle keepalives properly" - our functional tests are pretty robust against intermittent failures like that - maybe we can shelve that one and loop back around it20:59
clarkbclayg: I'm trying to understand what the testFileSizeLimit test is trying to do. The comments in there are confusing me. It looks like if we make a file smaller than the limit we assert that the Timeout exception happens, it should happen because copying 1GB of data over http(s) takes longer than 3 seconds?21:02
clarkbclayg: then for the >limit we should get back an error from swift because the content size is > the limit?21:03
clarkband that should happen quickly, under 3 seconds?21:03
claygi hadn't started looking at it - i was bringing up my devstack environment and working on other stuff...21:04
*** lucasxu has quit IRC21:07
claygclarkb: I agree the comments there are confusing - https://github.com/openstack/swift/blob/0d5b2a867dae30ddc6772b5ad17f6c87448a6e84/test/functional/tests.py#L213821:08
claygI think some cruft grew in for development environments that probably should have been handled differently - the comments about the fallocate configuration of the object servers are way to whitebox for these tests21:09
claygI'm not sure if that came in while the in-process functests were getting spun up - or if some default with fallocate changed and it seemed like a reasonable workaround21:09
claygidk, it may be a total red herring - the constraints and the SUT may be in total agreement and the test may be asserting the correct behavior while the comment is trying to warn at the kind of mis-config that can result in the functional failure?21:13
claygclarkb: it's notmyname's fault -> https://review.openstack.org/#/c/331318/21:14
patchbotpatch 331318 - swift - added note to testFileSizeLimit functional test (MERGED)21:14
*** ukaynar has quit IRC21:26
claygclarkb: i think swiftclient is doing the right thing with connection headers and pipeline when talking directly to the eventlet server -> https://gist.github.com/clayg/3eda31eefe8deb9dc008ca66dd6f226f21:26
*** ukaynar has joined #openstack-swift21:26
claygwhen my devstack environment comes up I'll try to decide if I think swift behind the devstack configured tls apache is doing a thing that is sane or insane21:27
clarkbclayg: ya I think in the general case it works. Where it fails is if the server (eg apache) times out its keepalive'd connection to client and starts to close it while at the same time the client makes another request on the connection. That connection will fail and should be retried but isn't21:28
clarkbI think it also works if the connection times out completely before the new request is made and python-requests will make a new connection properly in that case. its a weird corner case that our larger set of tests are able to trip. Happy to configure swift with keepalives on though21:29
*** ukaynar has quit IRC21:31
clarkbbasically its something that we can refine in the deployment made by devstack. The turn if off everywhere was the quick fix for all the job fails we hit21:31
claygclarkb: ok, makes sense, everything adds up #agree21:32
clarkbclayg: testInvalidPath change is pretty straightforward, basically we avoid just hitting / as some webservers may serve content off of / (like apache in devstack). Should I go ahead and split that one out into its own change that can merge more quickly?21:39
clarkband then the 500 vs 501 in testBadHeaders is a fun one too...21:39
claygidk, devil is in the details - asserting 5XX instead of a specific 500 error code *sounds* ok - but then again - is a "Not Implemented" response really a good substitute for a "Server Error"?  Answering "what should the tests require" normally depend on what is best/reasonable for the client - and why does one deployment to the next think it's reasonable that this be different and reasonable is it require "no, if you21:44
claygwant clients to get the expected behavior you *have* to configure your web proxies between your client and swift to XYZ"21:44
claygno one is trying to be gratuitously pedantic - but the idea is "when functests pass consumers of your cluster will generally be happy with your faithful presentation of a swift api compatible endpoint"21:45
clarkbI'm not even sure if that is expected to be configurable, which is what makse this fun. I think what is happening is apache is asserting that chunked requests without content length info is an error. Not something that could be implemented?21:45
clarkband yup as a user of these apis, I definitely get the desire to be consistent (and why switching on web_front_end is not the greatest)21:46
clarkbits possible the gzip + chunked is the problem though and we need to install mod_deflate?21:48
*** catintheroof has quit IRC21:53
timburkei wonder if the problem is that the body isn't actually gzipped? we're just sending random junk data, not a valid gzip stream...21:54
timburkekinda seems like we ought to just drop the assertion, though -- the set of supported transfer encodings is pretty decidedly up to the WSGI server, not the app21:56
timburke(which makes hacks like https://gist.github.com/tipabu/7c8ed7a713a7f51f20c2#file-compression-py all the more fun)21:57
clarkbtimburke: and instead support not 2XX?22:01
clarkbs/support/assert/?22:01
timburkeclarkb: i'm thinking drop it entirely. why shouldn't a server decide to support gzip,chunked?22:02
clarkbtimburke: oh in this case I think we neglect to pass on the length info necessary to do chunked. SO we are asserting that it fails22:02
timburkeno, chunked doesn't need length info. that's the whole point22:03
timburkenow i'm wondering if we even do chunked right in that test....22:04
timburkei'm not seeing an immediate path from write_random to chunked_write, so i'm gonna say probably not22:07
clarkbtimburke: the first one fails because content-length is X not an actual integer. Then the second one fails because content-length itself is entirely missing. Then the third one fails because server either doesn't support gzip or it does and found the gzip to be invalid?22:07
timburkeyeah -- we're getting the 400/411 just fine, right? it's the 501 that's causing a problem?22:09
clarkband only the last (third) one is requesting chunked22:09
clarkbtimburke: correct, built in webserver returns 500, apache2 returns 50122:09
clarkbI think you are likely correct in that its a server error due to the invalid gzip data22:10
clarkbso the bad header is in specifying a content type of gzip but web servers aren't consistent in what they return as the error?22:11
timburkeother way around, no? the assertion is for a 501. regardless, i just don't think https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2151-L2154 is terribly useful, so i'm in favor of dropping it entirely22:11
clarkboh yup apache2 is 500, builtin is 501. sorry22:11
*** jamielennox|away is now known as jamielennox22:13
timburkeswift itself will actually send out a 501 (https://github.com/openstack/swift/blob/2.14.0/swift/common/swob.py#L774-L777 which gets caught by https://github.com/openstack/swift/blob/2.14.0/swift/common/constraints.py#L195-L197) so i wonder if apache is translating the server error to something more generic22:15
timburkei wonder whether that's true for 503s or 507s as well... those are rather specific and useful in the context of swift... though maybe not something you want to expose to clients? idk...22:17
clarkbI think its more that apache catches things before every proxying them22:18
clarkband so you end up with the error code apache returns?22:18
clarkbgzip, chunked is a valid header value though right? swift just hasn't implemented the code to handle both together?22:20
timburkeyup, totally a valid header22:23
clarkbtimburke: clayg finally hunted down the error in the apache logs [Mon Jun 05 16:48:31.033144 2017] [proxy_http:error] [pid 27291:tid 140300542539520] [client 10.40.55.64:59438] [frontend 10.40.55.64:8080] AH01093: gzip,chunked Transfer-Encoding is not supported22:34
clarkbso I think that is a matter of not having mod_deflate installed.22:34
clarkbwhich should then proxy properly and get the 501 back?22:34
claygtimburke: why would you expect we're not doing chunked transfer correctly?!22:35
timburkeclarkb: makes sense; even shows up on https://wiki.apache.org/httpd/ListOfErrors#line-108622:36
claygclarkb: timburke: you guys are going to give me an ulcer - stuff like request pipelineing and chunked transfer are *exactly* the kind of stuff we want functests validating very carefully?22:36
timburkei don't think installing mod_deflate will fix the test, though; we'd most likely start getting back 20122:37
clarkbtimburke: you don't think it would then bubble teh 501 from swift back out?22:37
timburkeas apache unzips (now that it knows how) and passes a plain-old chunked request on to swift22:37
clarkbah22:38
*** geaaru__ has quit IRC22:38
timburkeclayg: chunked transfers, yes, we should validate those carefully. saying "this thing that's totally a reasonable thing to want to do should not be implemented" is a separate issue, and seems like a bad test22:39
timburkeclayg: i'm reasonably certain that https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2426-L2450 tests chunked transfers work. i'm not at all convinced that https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2151-L2154 has the client speaking HTTP22:49
*** vint_bra has quit IRC22:49
claygtimburke: https://github.com/openstack/swift/blob/2.14.0/swift/common/swob.py#L76722:50
claygidk, forwhatever reason we like only one te and we like it to be chunked?22:50
claygi don't really understand how the te can be unrolled - i sort of intuit that you could ... idk do a chunked encoding of a zlib compressed transfer of an entity - but I don't see who/how could unroll the zlib part first?22:52
timburkeclayg: sure, totally a reasonable thing for swift to want to do -- it's not in the business of zipping/unzipping payloads. but if eventlet were to add gzip support tomorrow, and did it by dropping the 'gzip,' from the transfer-encoding header, should the test break?22:52
claygit's probably bullshit that we return 501 there - and roughly inconsequential - but apache rejects the request before it even gets to us?22:52
claygyeah that's what i'm saying tho it can't drop the gzip from the header... just because it'd have to unchunk it first or you think it would rewrite the chunks as it deflates them?  maybe...22:53
clarkbyes apache is rejectnig the request before it gets to swift, I think because mod_deflate isn't enabled22:54
timburkeclayg: all eventlet exposes is a readable -- we shouldn't care much at all beyond that22:54
claygas far as what the test can validate... yeah that's harder to say... it's not unreasonable for the tests to assert any error that is reliably returned from the api I don't *think* - but that might be a little off the cuff22:54
claygtimburke: yeah I think i understand what you're saying - if the wsgi server can turn over the request to us in such a way that's it's transparently proxied into a request we know how to service...22:56
*** mat128 has joined #openstack-swift22:59
claygtimburke: https://www.python.org/dev/peps/pep-0333/#other-http-features <- mentions transfer-encoding related to gzip and in other places23:00
claygit's not 100% clear to me what the prescription is tho?  middleware shouldn't apply transfer-encoding gzip - but how do I know if my server is supporting it or not?  are they supposed to mess with the hop-by-hop headers to indicate they've unzipped it?  Still seems annoying for apache to decide it doesn't need to send the request to through .23:02
timburkeclayg: fwiw applying http://paste.openstack.org/show/611439/ has the test fail -- but with a 499, which is kinda strange. but it seems like eventlet partially caught the error? "ERROR WSGI: code 400, message Bad request syntax ('_\xbb..."23:02
timburkeclayg: apache as a reverse proxy to another process running on another port seems well-within rights to drop all hop-by-hop headers from the client23:04
claygtimburke: yeah if we have a reverse proxy that would take such a request to swift and transform it in such a way that swift wanted saw a normal looking chunked transfer request and wanted to respond with success that would be something indeed23:05
timburkeand i think that's exactly what we'll see if we install mod_deflate -- though i suppose we'll see what clarkb reports back :-)23:07
claygah, gotcha23:07
timburkewell, no, it'll probably 400 or something. on account of the not-really-a-chunked-body problem23:07
clarkbya should have change up momentarily now that meeting is over23:08
claygyeah cool, let's wait and see23:08
*** adriant has joined #openstack-swift23:08
*** catintheroof has joined #openstack-swift23:13
*** tonanhngo has quit IRC23:14
clarkbtimburke: clayg telnet://[2001:4800:1ae1:18:f816:3eff:fe35:2c04]:19885 should be testing it (if you can't ipv6 the logs will be copied to log server when job completes). This tests it with the 500 assertion which we no longer expect to get so those tests should fail on that check23:16
openstackgerritLingxian Kong proposed openstack/swift master: [WIP] Write-affinity aware object deletion  https://review.openstack.org/47015823:19
*** jamielennox is now known as jamielennox|away23:30
*** chsc has quit IRC23:37
*** kei_yama has joined #openstack-swift23:37
clarkbclayg timburke http://logs.openstack.org/74/372374/32/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/9289a40/console.html#_2017-06-05_23_37_47_926057 that didn't do what I expected it to do23:42
clarkb[Mon Jun 05 23:37:21.830255 2017] [proxy_http:error] [pid 27198:tid 140482566924032] [client 10.40.68.150:47120] [frontend 10.40.68.150:8080] AH01093: gzip,chunked Transfer-Encoding is not supported still failing with that, maybe it just doesn't support it at all like swift?23:42
*** jamielennox|away is now known as jamielennox23:46
claygclarkb: timburke: fwiw on the testFileSizeLimit test I'm pretty sure in all cases swift is never getting the request from apache, who's logging something and maybe buffering or something?  maybe waiting on the client connection to do something it's not going to do because the test isn't that great?  dunno yet23:57
claygI thought the 500 vs 501 assert bad headers thing was different?23:59
clayghttp://logs.openstack.org/74/372374/31/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/210c2f8/console.html#_2017-06-05_20_36_29_80824523:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!