*** tovin07_ has joined #openstack-swift | 00:35 | |
*** zhengyin has joined #openstack-swift | 00:54 | |
*** chsc has joined #openstack-swift | 00:57 | |
*** chsc has joined #openstack-swift | 00:57 | |
*** chsc has quit IRC | 01:07 | |
*** vint_bra has quit IRC | 01:19 | |
*** janonymous has joined #openstack-swift | 01:33 | |
*** zhurong has joined #openstack-swift | 01:33 | |
kota_ | good morning | 01:41 |
---|---|---|
mattoliverau | kota_: morning | 01:43 |
*** zhurong has quit IRC | 01:44 | |
kota_ | mattoliverau: o/ | 01:48 |
*** tovin07_ has quit IRC | 01:57 | |
*** tovin07_ has joined #openstack-swift | 02:05 | |
*** zhurong has joined #openstack-swift | 02:32 | |
*** winggundamth has joined #openstack-swift | 02:52 | |
*** zhengyin has quit IRC | 03:05 | |
*** zhengyin has joined #openstack-swift | 03:06 | |
*** mingyu has joined #openstack-swift | 03:34 | |
*** psachin has joined #openstack-swift | 03:42 | |
*** Dinesh_Bhor has joined #openstack-swift | 04:03 | |
*** mingyu has quit IRC | 04:10 | |
*** mingyu has joined #openstack-swift | 04:11 | |
mahatic | good morning | 04:11 |
mahatic | kota_: mattoliverau | 04:11 |
mahatic | o/ | 04:12 |
kota_ | mahatic: o/ | 04:12 |
*** mingyu has quit IRC | 04:15 | |
*** klrmn has quit IRC | 04:15 | |
mattoliverau | mahatic: o/ | 04:28 |
*** mingyu has joined #openstack-swift | 05:02 | |
*** zhengyin has quit IRC | 05:05 | |
*** zhengyin has joined #openstack-swift | 05:05 | |
*** Fdaisuke_ has joined #openstack-swift | 05:10 | |
*** Fdaisuke_ has quit IRC | 05:11 | |
*** Fdaisuke has joined #openstack-swift | 05:11 | |
*** Fdaisuke has left #openstack-swift | 05:12 | |
*** mingyu has quit IRC | 05:19 | |
*** cshastri has joined #openstack-swift | 05:44 | |
* kota_ is looking at the follow up patch for proxy config per policy | 05:54 | |
*** mingyu has joined #openstack-swift | 06:08 | |
*** mingyu has quit IRC | 06:14 | |
*** mingyu has joined #openstack-swift | 06:17 | |
*** chsc has joined #openstack-swift | 06:22 | |
*** chsc has joined #openstack-swift | 06:22 | |
*** skudlik has joined #openstack-swift | 06:27 | |
*** rcernin has joined #openstack-swift | 06:29 | |
*** chsc has quit IRC | 06:31 | |
*** geaaru__ has quit IRC | 06:50 | |
*** mattoliverau has quit IRC | 06:50 | |
*** hoonetorg has quit IRC | 06:50 | |
*** samueldmq has quit IRC | 06:50 | |
*** amit213 has quit IRC | 06:50 | |
*** clayg has quit IRC | 06:50 | |
*** geaaru__ has joined #openstack-swift | 06:50 | |
*** hoonetorg has joined #openstack-swift | 06:50 | |
*** mingyu has quit IRC | 06:50 | |
*** samueldmq has joined #openstack-swift | 06:50 | |
*** amit213 has joined #openstack-swift | 06:51 | |
*** clayg has joined #openstack-swift | 06:51 | |
*** ChanServ sets mode: +v clayg | 06:51 | |
*** mattoliverau has joined #openstack-swift | 06:56 | |
*** ChanServ sets mode: +v mattoliverau | 06:56 | |
*** mingyu has joined #openstack-swift | 07:02 | |
*** pcaruana has joined #openstack-swift | 07:04 | |
*** oshritf has joined #openstack-swift | 07:16 | |
*** mvpnitesh has joined #openstack-swift | 07:20 | |
*** zhurong has quit IRC | 07:31 | |
*** mingyu has quit IRC | 07:38 | |
*** zhurong has joined #openstack-swift | 07:40 | |
*** mingyu has joined #openstack-swift | 07:52 | |
*** bkopilov has quit IRC | 07:53 | |
acoles | good morning | 07:58 |
*** bkopilov has joined #openstack-swift | 08:02 | |
acoles | asettle: hi, you around? I wanted to confirm something I am sure I have probably asked you before (sorry!) - the install guides get published soon after new release, and then are frozen i.e. they don't get updated automatically as we patch the source, correct? | 08:03 |
mattoliverau | acoles: morning | 08:07 |
acoles | mattoliverau: o/ | 08:15 |
*** mingyu has quit IRC | 08:18 | |
*** jaosorior has joined #openstack-swift | 08:20 | |
*** zhurong has quit IRC | 08:20 | |
*** mingyu has joined #openstack-swift | 08:23 | |
asettle | acoles: heya! | 08:25 |
asettle | acoles: we tend to freeze about 6 weeks *before* the release, when we start testing packages. Hard freeze probably about a month before release. All related here: https://docs.openstack.org/contributor-guide/release/taskdetail.html#installation-tutorial-testing | 08:26 |
asettle | The guides are released with the general OpenStack release time frame (so, from master, it's then published) | 08:26 |
asettle | And then a month afterwards we cut the branch - so we then would (for example) have the Ocata branch, and new content starts being done on master | 08:27 |
acoles | asettle: ah, got it. thanks. I'm just wondering how to label a bug on swift ocata install guide where it refers to the release (newton). We fixed upstream but I guess that won't get published until pike. | 08:29 |
asettle | I'd probably just tag it with that install-ocata-swift or something similar. (whatever your conventions may be) | 08:31 |
*** mingyu has quit IRC | 08:44 | |
*** mingyu has joined #openstack-swift | 08:45 | |
*** mingyu has quit IRC | 08:47 | |
*** ^andrea^ has joined #openstack-swift | 08:49 | |
*** ^_andrea_^ has joined #openstack-swift | 08:49 | |
*** ^andrea^ has quit IRC | 08:49 | |
openstackgerrit | Merged openstack/swift master: Follow-up for per-policy proxy configs https://review.openstack.org/466952 | 08:50 |
*** zhurong has joined #openstack-swift | 08:50 | |
*** eckesicle has joined #openstack-swift | 08:51 | |
*** cshastri has quit IRC | 09:32 | |
openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Delay port binding to reduce wait at process start https://review.openstack.org/470903 | 09:44 |
*** jaosorior has quit IRC | 09:49 | |
*** mingyu has joined #openstack-swift | 09:49 | |
*** cshastri has joined #openstack-swift | 09:49 | |
*** jaosorior has joined #openstack-swift | 09:53 | |
*** zhurong has quit IRC | 09:59 | |
*** tovin07_ has quit IRC | 10:00 | |
*** cshastri has quit IRC | 10:06 | |
*** mingyu has quit IRC | 10:08 | |
*** zhurong has joined #openstack-swift | 10:12 | |
*** cshastri has joined #openstack-swift | 10:17 | |
*** mat128 has joined #openstack-swift | 10:27 | |
*** mat128 has quit IRC | 10:29 | |
*** mat128 has joined #openstack-swift | 10:34 | |
*** mwheckmann has quit IRC | 10:38 | |
*** mwheckmann has joined #openstack-swift | 10:46 | |
hugokuo | Have we addressed how object-replicator stuck there without doing anything yet? The process is not dead just keep waiting. | 10:51 |
*** zhengyin has quit IRC | 11:39 | |
*** links has joined #openstack-swift | 12:04 | |
*** zhurong has quit IRC | 12:04 | |
*** mdrabe has joined #openstack-swift | 12:27 | |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Make eventlet.tpool's thread count configurable in object server https://review.openstack.org/289664 | 12:32 |
*** eckesicle has quit IRC | 12:35 | |
*** catintheroof has joined #openstack-swift | 12:35 | |
*** MVenesio has joined #openstack-swift | 12:38 | |
*** mvpnitesh has quit IRC | 12:45 | |
*** mvpnitesh has joined #openstack-swift | 12:45 | |
*** kei_yama has quit IRC | 12:55 | |
*** winggundamth has quit IRC | 13:01 | |
*** mvpnitesh has quit IRC | 13:02 | |
*** lucasxu has joined #openstack-swift | 13:17 | |
*** klamath has joined #openstack-swift | 13:21 | |
*** klamath has quit IRC | 13:22 | |
*** klamath has joined #openstack-swift | 13:23 | |
*** kozhukalov has quit IRC | 13:28 | |
*** zhurong has joined #openstack-swift | 13:46 | |
asettle | acoles: heyyyyyy | 13:59 |
asettle | Question for you this time | 13:59 |
asettle | Did you guys do this? https://docs.openstack.org/admin-guide/cli-analyzing-log-files-with-swift.html | 13:59 |
asettle | "you guys" = swift | 13:59 |
asettle | Is this even necessary? | 14:00 |
acoles | asettle: I don't recognize it | 14:00 |
asettle | acoles: would there be concern about deletion? | 14:00 |
acoles | asettle: you mean, if the client deleted the log files? | 14:02 |
asettle | If I deleted that page :p | 14:02 |
acoles | OIC lol | 14:02 |
asettle | Hahaha | 14:02 |
acoles | does git blame show likely author? | 14:03 |
asettle | It does not :( | 14:03 |
acoles | asettle: I suggest checking with notmyname when he is awake, I wouldn't delete based solely on me not knowing | 14:04 |
asettle | Good call ;) | 14:04 |
*** mingyu has joined #openstack-swift | 14:06 | |
jrichli | acoles, mahatic, rusik : have any of you started work on lp bug #1691807 ? | 14:16 |
openstack | Launchpad bug 1691807 in OpenStack Object Storage (swift) "object decryption should be able to validate that correct key is being used" [Wishlist,New] https://launchpad.net/bugs/1691807 - Assigned to Ruslan Gustomyasov (rusik) | 14:16 |
acoles | jrichli: hi! not me | 14:16 |
jrichli | acoles: o/ do you know Ruslan? | 14:17 |
acoles | no, not seen the nick before | 14:17 |
mahatic | jrichli: hello! not me either. But I've thought about it a bit, I might just drop some notes there though | 14:17 |
mahatic | acoles: o/ | 14:17 |
acoles | jrichli: I'd like to see it progress but not had time myself | 14:18 |
*** vint_bra has joined #openstack-swift | 14:18 | |
jrichli | mahatic: that would be great. | 14:18 |
jrichli | I know I have been a bit absent from swift, but I *really* hope that I will be able to start up again this week. | 14:18 |
jrichli | so, I'll take a look and see what I can do. I will put a comment on the bug to see if Ruslan has made some progress. thanks! | 14:19 |
mahatic | jrichli: great, thanks too | 14:19 |
acoles | jrichli: great :D | 14:29 |
*** psachin has quit IRC | 14:31 | |
*** psachin has joined #openstack-swift | 14:34 | |
*** ukaynar has joined #openstack-swift | 14:41 | |
*** psachin has quit IRC | 14:47 | |
*** skudlik has quit IRC | 14:47 | |
*** cshastri has quit IRC | 14:48 | |
*** skudlik has joined #openstack-swift | 14:51 | |
*** ukaynar_ has joined #openstack-swift | 14:54 | |
*** ukaynar has quit IRC | 14:57 | |
*** chsc has joined #openstack-swift | 15:02 | |
*** gyee has joined #openstack-swift | 15:04 | |
*** skudlik has quit IRC | 15:04 | |
*** Renich has joined #openstack-swift | 15:13 | |
clarkb | asettle: acoles looks like joseph robinson added the file about a year ago. I ran git log -p on the file say that it got renamed, checked out the commit before the commit that renamed then checked history on the old filename. yay git | 15:13 |
clarkb | oh wait that may be another rename /me digs deeper | 15:14 |
Renich | Hello! o/ | 15:14 |
asettle | clarkb: oooo thanks. I couldn't find anything, but then again, I was doing a bit of a cursory glance while on a meeting | 15:14 |
Renich | I am trying to set quotas to a project | 15:14 |
asettle | clarkb: it just seems so out of place, and with all the doc restructuring due to happen, I question its relevance | 15:14 |
asettle | And if so, who it belogns to | 15:15 |
Renich | but, even, though, openstack project list shows the project and user list show's the user in the project, I can't seem to show or set the quota | 15:15 |
Renich | I always get "need more than 0 values to unpack | 15:15 |
Renich | " | 15:15 |
*** zhurong has quit IRC | 15:15 | |
*** chsc has quit IRC | 15:16 | |
clarkb | asettle: it seems like that file has moved many times over the last couple years... I think I have found 4 moves and still haven't found the origin of it | 15:16 |
clarkb | asettle: which would support the idea that its not well placed? | 15:16 |
asettle | clarkb: would explain *a lot* | 15:16 |
asettle | Thanks for diving into this :) | 15:16 |
*** aselius has joined #openstack-swift | 15:16 | |
*** mvk has quit IRC | 15:19 | |
clarkb | asettle: I368af8a7b843b1f1cba0d8a2ccdaa0f40634db3d added it in 2011 as part of the swift admin guide | 15:23 |
asettle | Woah dude you went so deep | 15:24 |
asettle | And it's been moved around since then, huh | 15:24 |
asettle | Thank you, clarkb :) | 15:24 |
asettle | I guess now I'm wondering how relevant it is to keep it. | 15:24 |
asettle | Cause it seems a little out of... place... still | 15:24 |
clarkb | yes something like at least 10 moves | 15:24 |
clarkb | I stopped counting | 15:25 |
asettle | *head desk* | 15:25 |
asettle | Typical docs team ;P | 15:25 |
asettle | We can't keep anything in one place | 15:25 |
*** MVenesio has quit IRC | 15:31 | |
*** ukaynar has joined #openstack-swift | 15:34 | |
*** lucasxu has quit IRC | 15:35 | |
*** ukaynar_ has quit IRC | 15:37 | |
*** mingyu has quit IRC | 15:37 | |
*** mingyu has joined #openstack-swift | 15:39 | |
clayg | redbo - do you subscribe the sqlite ML? http://sqlite.1065341.n5.nabble.com/WAL-checkpoint-starved-td96011.html <- about WAL checkpoint starved - I'm not sure there's a resolution suggested yet in the thread 🍿 | 15:40 |
*** Renich has quit IRC | 15:42 | |
*** mingyu has quit IRC | 15:43 | |
*** chsc has joined #openstack-swift | 15:51 | |
*** chsc has joined #openstack-swift | 15:51 | |
*** rcernin has quit IRC | 15:52 | |
redbo | What are we talking about? My theory is if I grab an exclusive lock on the db, that'll make sure there aren't any readers before I checkpoint. | 15:53 |
*** chsc has quit IRC | 15:56 | |
clayg | umm... but are you going to grab an exclusive lock before every insert? I thought the idea was that maybe the WAL insert could eventually be better/faster/more-robust than the .pending db bulk insert stuff - but maybe we far too coupled into that now anyway... | 15:58 |
clayg | the git history has the legacy of trying wal and then giving up - IIRC the issue was that the WAL would keep growing and never force the inserts - the .pending approach with it's .stat to check the size never seems to have that problem | 16:00 |
redbo | Right now it only forces a checkpoint during replication. | 16:02 |
*** Renich has joined #openstack-swift | 16:04 | |
redbo | There's no reason to force checkpoints constantly. It just needs to not grow for forever if someone is accessing the database so much it can't auto checkpoint. | 16:05 |
redbo | and also when it needs to make one file to rsync in replication | 16:05 |
*** oshritf has quit IRC | 16:09 | |
Renich | guys, I need help setting a quota for a project or account. | 16:11 |
Renich | I am using v3 | 16:11 |
Renich | This example: https://docs.openstack.org/ops-guide/ops-quotas.html is not helping. It doesn't work when I source the user's credentials | 16:12 |
Renich | And I've been trying to source the admin's credentials and point it to the project, but I can't find the right combination of flags | 16:13 |
clayg | oh oh oh... I understood this one time... | 16:13 |
clayg | Renich: if you have "admin credentials" you should just need --os-storage-url pointed at the users' account | 16:13 |
clayg | Renich: swift post --os-storage-url https://storage.example.com/v1/AUTH_<project_id> ... | 16:14 |
Renich | clayg: OK. i will try that. But, quite frankly, I think env vars are poluting the command somehow | 16:14 |
clayg | run with -v or --debug to get more context | 16:15 |
Renich | clayg: ah! it worked! | 16:15 |
clayg | nowhey! | 16:15 |
Renich | ... I am confused... | 16:15 |
clayg | s'ok - most of us are | 16:15 |
Renich | clayg: but, hey, thanks a ton. I've been dealing with this for an hour or more | 16:15 |
Renich | LOL | 16:16 |
clayg | that sucks! | 16:16 |
Renich | yeah | 16:16 |
Renich | Thanks a bunch, man. I appreciate the help | 16:16 |
clayg | where was the first place you looked and what can we make it say so that you don't waste your time? | 16:16 |
Renich | you mean logs? I tried -vv | 16:17 |
Renich | didn't really look at the logs | 16:17 |
Renich | Also, tried using openstack quota set/show but couldn't. It seems that it wants a networking endpoint to be avaialble, and I have none | 16:17 |
clayg | no... i mean... idk - presumably there's like instructions for this somewhere? how did you know you needed quotas or that they like exited or whatever and that ... i don't know? | 16:17 |
Renich | ... | 16:18 |
clayg | ... existed ... | 16:18 |
Renich | No network endpoint in service catalog | 16:18 |
Renich | need more than 0 values to unpack | 16:18 |
Renich | clean_up ShowQuota: need more than 0 values to unpack | 16:18 |
Renich | clayg: ah, you mean https://docs.openstack.org/ops-guide/ops-quotas.html | 16:18 |
Renich | and https://docs.openstack.org/admin-guide/cli-set-quotas.html | 16:18 |
clayg | gah!? what is that? something with the openstack cli? I have no idea if it supports ... | 16:18 |
Renich | clayg: yes, openstack cli. | 16:19 |
clayg | yeah, I think that's what I was looking for - maybe we can improve that doco there - thanks! | 16:19 |
Renich | well, basically, we're running a kind-of-beta for openstack-swift only. And, some clients, are testing it. So we're trying to figure out management. Kind of hard, IMHO. | 16:20 |
Renich | clayg: thanks for that! ;) | 16:20 |
clayg | yeah, I think you'd find an interested audience here to consume your first hand experience and a stack ranking of the hardest parts | 16:21 |
clayg | thanks for stopping by - come ask any questions as they pop up - people are always around | 16:22 |
Renich | yep, thanks a lot. | 16:22 |
Renich | I will | 16:22 |
*** pcaruana has quit IRC | 16:23 | |
clayg | asettle: you know how you and notmyname are always talking about doc trees and yaml and stuff that's in our repo and like other repos and stuff with something something docs? | 16:23 |
clayg | do you have any idea where https://docs.openstack.org/ops-guide/ops-quotas.html or https://docs.openstack.org/admin-guide/cli-set-quotas.html live? Are either of them in swift's tree? | 16:24 |
notmyname | good morning | 16:24 |
asettle | clayg: yo sure do | 16:24 |
asettle | clayg: https://github.com/openstack/openstack-manuals/tree/master/doc/ops-guide/source | 16:24 |
asettle | https://github.com/openstack/openstack-manuals/blob/master/doc/ops-guide/source/ops-quotas.rst | 16:24 |
asettle | andddd | 16:24 |
asettle | https://github.com/openstack/openstack-manuals/blob/master/doc/admin-guide/source/cli-set-quotas.rst | 16:25 |
asettle | clayg: openstack-manuals repo :) | 16:25 |
notmyname | asettle: no, i don't knwo about https://docs.openstack.org/admin-guide/cli-analyzing-log-files-with-swift.html. analyzing logs is a common thing for ops. might be interesting to add it to https://docs.openstack.org/developer/swift/logs.html if you don't want it anymore | 16:25 |
clayg | ah, coolio - thanks! | 16:25 |
asettle | notmyname: well, tbh, we just don't know what to do with it. If you want it, happily will pass it along. It's not not useful. Just. Dunno what it's doing there :P | 16:26 |
asettle | clayg: no problemo :) | 16:26 |
notmyname | asettle: after having spent 30 seconds considering it, I'd be sad to see it go away, but I have no strong feelings on where it should live | 16:28 |
*** lucasxu has joined #openstack-swift | 16:28 | |
asettle | notmyname: hahaha okay. well. I'll get back to you ;) | 16:28 |
*** links has quit IRC | 16:35 | |
*** ukaynar has quit IRC | 16:39 | |
*** honga has joined #openstack-swift | 16:39 | |
*** ukaynar has joined #openstack-swift | 16:40 | |
timburke | good morning | 16:40 |
*** honga has quit IRC | 16:42 | |
*** honga has joined #openstack-swift | 16:42 | |
*** chsc has joined #openstack-swift | 16:44 | |
*** ukaynar has quit IRC | 16:44 | |
openstackgerrit | Tim Burke proposed openstack/swift master: Add validation for sorting_method values https://review.openstack.org/469269 | 16:46 |
clarkb | clayg: got it down to 16 fails with a change to test.conf that sets web_front_end to apache2 http://logs.openstack.org/74/372374/30/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/4a1d2a0/console.html | 16:51 |
clayg | lol - briancli1e you'd already submitted patch 185221 years ago :P | 16:53 |
patchbot | https://review.openstack.org/#/c/185221/ - swift - Add object replicator progress, heartbeat to recon | 16:53 |
clayg | clarkb: lolwut? | 16:54 |
clayg | clarkb: there's a test.conf option specifically for apache? | 16:54 |
clarkb | clayg: yesm some of the tests branch on the web_front_end config value | 16:54 |
clarkb | looks like portante may have added some of that? | 16:55 |
timburke | i... apache2 doesn't support chunked PUTs? https://github.com/openstack/swift/blob/master/test/functional/tests.py#L2482-L2485 | 16:56 |
clarkb | timburke: there are some places where it asserts different responses too isntead of skipping tests. I'm not sure what all the details are, but since apache2 is terminating tls for us I just set the web_front_end config and 4 tests no longer a problem >_> | 16:58 |
openstackgerrit | Clark Boylan proposed openstack/swift master: Only assert connection header if set in request https://review.openstack.org/471057 | 17:02 |
clarkb | clayg: and I think ^ shoudl address another couple failures, though not sure if that assertion is important there and needs to be solved some other way | 17:02 |
*** jaosorior is now known as jaosorior_away | 17:03 | |
*** klrmn has joined #openstack-swift | 17:04 | |
*** lucasxu has quit IRC | 17:13 | |
*** lucasxu has joined #openstack-swift | 17:14 | |
clayg | clarkb: it was David Hadas -> https://review.openstack.org/#/c/23585/ | 17:14 |
patchbot | patch 23585 - swift - Support tests for Apache (MERGED) | 17:14 |
clayg | portante: was just refactoring the pre-existing configuration | 17:15 |
clarkb | ah | 17:15 |
portante | clayg, clarkb: anything I can help with, or muddy, as the case may be? | 17:15 |
portante | ;) | 17:16 |
Renich | OK, it is not working actually. I tried it again step by step and it didn't work. | 17:20 |
Renich | Account POST failed: http://myurl:8080/v1/AUTH_someID 403 Forbidden [first 60 chars of response] <html><h1>Forbidden</h1><p>Access was denied to this resourc | 17:20 |
Renich | Failed Transaction ID: some-transactionID | 17:20 |
openstackgerrit | Tim Burke proposed openstack/swift master: Add validation for sorting_method values https://review.openstack.org/469269 | 17:22 |
clarkb | clayg: hey for the header size limit thing, it looks like swifts default size limit is 8192, apache's is 8190. Do we just need to bump the apache limit up to the swift limit of 8192? | 17:29 |
clayg | lol - maybe? | 17:35 |
*** MVenesio has joined #openstack-swift | 17:35 | |
*** ukaynar has joined #openstack-swift | 17:38 | |
*** tonanhngo has joined #openstack-swift | 17:39 | |
*** lucasxu has quit IRC | 17:51 | |
*** mat128 has quit IRC | 17:52 | |
*** lucasxu has joined #openstack-swift | 17:52 | |
*** mat128 has joined #openstack-swift | 17:53 | |
*** mat128 has quit IRC | 17:54 | |
*** mat128 has joined #openstack-swift | 17:55 | |
openstackgerrit | Clark Boylan proposed openstack/swift master: Func test hacks to work under against apache2 https://review.openstack.org/471057 | 18:01 |
*** lucasxu has quit IRC | 18:08 | |
*** lucasxu has joined #openstack-swift | 18:09 | |
*** mvk has joined #openstack-swift | 18:17 | |
*** ukaynar has quit IRC | 18:19 | |
*** ukaynar has joined #openstack-swift | 18:20 | |
Renich | I was reading https://github.com/openstack/openstack-manuals/blob/master/doc/ops-guide/source/ops-quotas.rst and it says: "To update Object Storage quotas on a project, you must have the role of ResellerAdmin in the project that the quota is being applied to.", but that role doesn't even exist here. Only _member_, testrole, admin and user. | 18:22 |
Renich | It seems odd that admin can't change quotas here and there | 18:22 |
Renich | :S | 18:22 |
* Renich will try to find out how to create the ResellerAdmin role | 18:23 | |
notmyname | Renich: are you using keystone/ | 18:24 |
notmyname | ? | 18:24 |
Renich | nottrobin: yes | 18:24 |
Renich | notmyname: yes | 18:24 |
Renich | sorry nottrobin | 18:24 |
Renich | notmyname: v3 | 18:24 |
*** ukaynar has quit IRC | 18:24 | |
notmyname | Renich: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L398 | 18:24 |
notmyname | Renich: that's where "reseller admin" is defined for swift+keystone | 18:24 |
Renich | ah! Cool. So enable that... then what? | 18:25 |
Renich | Do I need to create the role or something after it? | 18:26 |
Renich | Or join the role? | 18:26 |
notmyname | sorry, just lost internet for a moment. not sure if I missed something | 18:26 |
Renich | notmyname: I'll repeat :) | 18:27 |
Renich | So enable that... then what? | 18:28 |
Renich | Do I need to create the role or something after it? | 18:28 |
Renich | Or join the role? | 18:28 |
*** ukaynar has joined #openstack-swift | 18:28 | |
*** honga has quit IRC | 18:28 | |
notmyname | right | 18:32 |
openstackgerrit | Thiago da Silva proposed openstack/liberasurecode master: Release 1.5.0 https://review.openstack.org/471084 | 18:32 |
notmyname | so that is the config var that tells swift which role (returned from keystone) to check | 18:32 |
notmyname | Renich: go back up to line 372 and read the full comments. might give a fuller perspective | 18:33 |
notmyname | tdasilva: nice! | 18:33 |
Renich | notmyname: thanks, I will | 18:33 |
tdasilva | notmyname: yeah, tons of fixes on that release | 18:33 |
notmyname | Renich: the basic flow is that swift gets a request with a token, swift asks keystone what roles that token has, then swift sets persmissions based on those returned roles. so if one of the returned roles is the same as what's configured in that reseller_admin_role setting, then swift (internally) sets the "swift owner" flag and allows the request | 18:34 |
tdasilva | as clayg likes to say: 'this will be the best libec release ever' ... or something like that.... ;) | 18:34 |
clayg | tdasilva: *close* - the best one *yet* | 18:34 |
Renich | notmyname: ok | 18:34 |
notmyname | Renich: and with reseller admin, it just means that it will *always* set the swift owner flag, regardless of which swift account is being used | 18:35 |
tdasilva | clayg: yeah, that's better! :) | 18:35 |
notmyname | Renich: well, "always, for the accounts with the configred `reseller_prefix`" ;-) | 18:35 |
Renich | hah! it works! | 18:40 |
notmyname | yay!! | 18:41 |
Renich | had to add reseller_admin_role and reseller_prefix | 18:41 |
Renich | then, create the role (openstack role add) | 18:41 |
Renich | then, add admin to role | 18:41 |
Renich | now, swift stat --os-storage-url some-url works! | 18:42 |
notmyname | wonderful | 18:42 |
Renich | IMHO, the documentation should have some kind of emphatic note regarding ResellerAdmin | 18:42 |
Renich | some warning or big, yellow note letting you know it won't work unless you have this setup | 18:42 |
Renich | since the default is not to have it | 18:43 |
notmyname | Renich: (mentioning for the sake of my own conscience) using the Swift API with a reseller admin is just like logging in to linux as root. makes some things simpler, but it's also possible to *really* ruin your day. be careful, and use reseller admin only sparingly | 18:43 |
notmyname | Renich: the improved docs are a great idea. where should they go? | 18:44 |
Renich | notmyname: wow, OK. Anything like sudo, then? ;D | 18:44 |
notmyname | no, not quite | 18:44 |
Renich | notmyname: they should go here https://docs.openstack.org/ops-guide/ops-quotas.html | 18:45 |
Renich | notmyname: from the service provider PoV, I think there should be a way for admins to administer other accounts. Up to now, we have been keeping credential files in order to do whatever we need for the beta-test clients | 18:45 |
notmyname | Renich: hmm... I think that ops guide is a repo that the docs team manages. see the bug icon in the top right? click that and write up what you want to see and it will get done :-) | 18:45 |
Renich | notmyname: will do | 18:46 |
notmyname | thanks | 18:46 |
notmyname | there's three "levels" of swift accounts | 18:46 |
notmyname | the reseller admin is like root (we've covered this one) | 18:46 |
notmyname | the reseller admin is also the only one that can send a DELETE to a whole account | 18:46 |
*** ukaynar has quit IRC | 18:48 | |
notmyname | the "normal" level of access in swift is where you (the api user) have full access to your account | 18:49 |
notmyname | while not limited to this, it might be easiest to think in terms of a public cloud provider: you sign up, get an account, and do whatever you want there (but you cant' do anything in *my* account) | 18:49 |
notmyname | the third "level" of access in swift would be where you only have access to things you've specifically been granted access to | 18:50 |
Renich | notmyname: thank you for taking the time to explain. I've posted a bug already: https://bugs.launchpad.net/openstack-manuals/+bug/1695961 | 18:53 |
openstack | Launchpad bug 1695961 in openstack-manuals "Quotas in Operations Guide" [Undecided,New] | 18:53 |
Renich | notmyname: and thanks for all the help, really. I appreciate it. | 18:53 |
*** cschwede has quit IRC | 18:57 | |
clarkb | clayg: hrm going to 8192 bytes of header in apache made test_invalid_acls fail | 18:59 |
*** cschwede has joined #openstack-swift | 18:59 | |
clarkb | oh is that for an individual header? not in aggregate? | 19:00 |
clarkb | that would explain it | 19:00 |
openstackgerrit | Clark Boylan proposed openstack/swift master: Func test hacks to work under against apache2 https://review.openstack.org/471057 | 19:00 |
*** mat128 has quit IRC | 19:04 | |
notmyname | clarkb: yeah, 8k per header | 19:04 |
notmyname | clarkb: lots of works around the different constraints on https://github.com/openstack/swift/blob/master/etc/swift.conf-sample#L97 | 19:04 |
*** zaitcev has joined #openstack-swift | 19:06 | |
*** ChanServ sets mode: +v zaitcev | 19:06 | |
clarkb | notmyname: ya trying to sort out why the second requset in test_invalid_acls is a 400 instead of a 204. The first one is more than 8192 bytes bu the second should be 8192 - 1024 + json overhead | 19:07 |
*** ukaynar has joined #openstack-swift | 19:14 | |
*** lucasxu has quit IRC | 19:15 | |
*** lucasxu has joined #openstack-swift | 19:34 | |
*** Renich has quit IRC | 19:44 | |
*** rcernin has joined #openstack-swift | 19:49 | |
*** MVenesio has quit IRC | 20:05 | |
*** lucasxu has quit IRC | 20:22 | |
clarkb | clayg: down to two failures now http://logs.openstack.org/74/372374/31/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/210c2f8/ using the hacks in https://review.openstack.org/#/c/471057/3/test/functional/tests.py | 20:40 |
patchbot | patch 471057 - swift - Func test hacks to work under against apache2 | 20:40 |
clayg | clarkb: that's amazing great work! | 20:40 |
openstackgerrit | Clark Boylan proposed openstack/swift master: Func test hacks to work under against apache2 https://review.openstack.org/471057 | 20:40 |
clarkb | clayg: as predicted a big thing is 304's and not returning accept-ranges headers | 20:41 |
clayg | doesn't seem so bad, yeah those accept headers on 304 responses are a pretty well understand - timburke will probably push towards just pulling them out of the responses altogether - I think first step is definitely fixing those | 20:44 |
clayg | I'm not 100% sure I understand the situation with the connection headers - I think we definitely want to validate the devstack default tls configuration is supporting connection keep-alive the way http >=1.0 servers *should* and swift clients would expect them to for the benifit of request piplineing if we're getting some signal that it might not be quite right | 20:45 |
clayg | ... but that may be orthogonal to getting functests passing - it may not be even be functests jobs to make sure request pipelining works... but it sorta *seems* like it would be? | 20:46 |
clayg | like we've made changes to our application that can break request pipeline for better or worse - and we want to keep it working - so if functests can't validate that it feels like some sort of failure of the testing infrastructure/configuration | 20:47 |
*** rcernin has quit IRC | 20:47 | |
*** lucasxu has joined #openstack-swift | 20:47 | |
clarkb | clayg: connection headers are explicitly set to close in devstack because python requests doesn't handle keepalives properly | 20:48 |
clayg | the filesize limit could easily just be apache kicking back an error earlier than the tests were expecting to get one from the eventlet server... | 20:48 |
clarkb | clayg: requests races on making new requests over keepalive'd connections that have been closed by the server | 20:48 |
clarkb | which is why that fails right now. Generally I don't think its a good idea for swift's tests to make sure apache is configured one way or another when it comes to things like keepalives. Is fine to check the built in web server though | 20:49 |
clarkb | clayg: another option there is to enable request pipelining with swift, assuming swiftclient isn't vulnerable to the issue in requests | 20:51 |
clayg | maybe... I think the web_front_end flag is ridiculous personally - the functests should validate the api works the way clients expect it to - if there's differences between one configuration and another clients can't count on that behavior - so it's either not part of the api worth validating or it needs to be exposed in discoverability | 20:52 |
clarkb | ++ | 20:52 |
clarkb | worth noting that I don't expect that change of mine to merge as is. I just wanted to be able to have these conversations with some concrete code. Otherwise its harder to talk about :) | 20:53 |
clayg | as far as requests not working with keepalive or request pipelineing ... that's a pretty terrible claim to levy - are you sure there's maybe not a little more subtlety to the story there? Swift really wants to ensure that request pipelining works - if that means bypassing python-requests/swiftclient and using HTTP(s)client directly so be it. | 20:54 |
clarkb | clayg: my understanding of the issue is that the race exists where the server can close a keepalive'd connection due to a timeout, and if at the same time you get a new request in the web server will error saying the connection is closed. The client should then reopen a new connection and run requests over that. Instead requests just raises an exception that no one handles htemselves | 20:55 |
clarkb | so the end result is whatever api call you were making fails even though it was not a fatal error, due to lack of retrying | 20:56 |
clarkb | I think rquests does handle the case of connection is dropped cleanly before attempting next request though | 20:58 |
clarkb | implying it should also handle this corner case. Its a been a few months since we poked at it though | 20:58 |
clayg | ok, that sounds a lot more reasonable than "python requests doesn't handle keepalives properly" - our functional tests are pretty robust against intermittent failures like that - maybe we can shelve that one and loop back around it | 20:59 |
clarkb | clayg: I'm trying to understand what the testFileSizeLimit test is trying to do. The comments in there are confusing me. It looks like if we make a file smaller than the limit we assert that the Timeout exception happens, it should happen because copying 1GB of data over http(s) takes longer than 3 seconds? | 21:02 |
clarkb | clayg: then for the >limit we should get back an error from swift because the content size is > the limit? | 21:03 |
clarkb | and that should happen quickly, under 3 seconds? | 21:03 |
clayg | i hadn't started looking at it - i was bringing up my devstack environment and working on other stuff... | 21:04 |
*** lucasxu has quit IRC | 21:07 | |
clayg | clarkb: I agree the comments there are confusing - https://github.com/openstack/swift/blob/0d5b2a867dae30ddc6772b5ad17f6c87448a6e84/test/functional/tests.py#L2138 | 21:08 |
clayg | I think some cruft grew in for development environments that probably should have been handled differently - the comments about the fallocate configuration of the object servers are way to whitebox for these tests | 21:09 |
clayg | I'm not sure if that came in while the in-process functests were getting spun up - or if some default with fallocate changed and it seemed like a reasonable workaround | 21:09 |
clayg | idk, it may be a total red herring - the constraints and the SUT may be in total agreement and the test may be asserting the correct behavior while the comment is trying to warn at the kind of mis-config that can result in the functional failure? | 21:13 |
clayg | clarkb: it's notmyname's fault -> https://review.openstack.org/#/c/331318/ | 21:14 |
patchbot | patch 331318 - swift - added note to testFileSizeLimit functional test (MERGED) | 21:14 |
*** ukaynar has quit IRC | 21:26 | |
clayg | clarkb: i think swiftclient is doing the right thing with connection headers and pipeline when talking directly to the eventlet server -> https://gist.github.com/clayg/3eda31eefe8deb9dc008ca66dd6f226f | 21:26 |
*** ukaynar has joined #openstack-swift | 21:26 | |
clayg | when my devstack environment comes up I'll try to decide if I think swift behind the devstack configured tls apache is doing a thing that is sane or insane | 21:27 |
clarkb | clayg: ya I think in the general case it works. Where it fails is if the server (eg apache) times out its keepalive'd connection to client and starts to close it while at the same time the client makes another request on the connection. That connection will fail and should be retried but isn't | 21:28 |
clarkb | I think it also works if the connection times out completely before the new request is made and python-requests will make a new connection properly in that case. its a weird corner case that our larger set of tests are able to trip. Happy to configure swift with keepalives on though | 21:29 |
*** ukaynar has quit IRC | 21:31 | |
clarkb | basically its something that we can refine in the deployment made by devstack. The turn if off everywhere was the quick fix for all the job fails we hit | 21:31 |
clayg | clarkb: ok, makes sense, everything adds up #agree | 21:32 |
clarkb | clayg: testInvalidPath change is pretty straightforward, basically we avoid just hitting / as some webservers may serve content off of / (like apache in devstack). Should I go ahead and split that one out into its own change that can merge more quickly? | 21:39 |
clarkb | and then the 500 vs 501 in testBadHeaders is a fun one too... | 21:39 |
clayg | idk, devil is in the details - asserting 5XX instead of a specific 500 error code *sounds* ok - but then again - is a "Not Implemented" response really a good substitute for a "Server Error"? Answering "what should the tests require" normally depend on what is best/reasonable for the client - and why does one deployment to the next think it's reasonable that this be different and reasonable is it require "no, if you | 21:44 |
clayg | want clients to get the expected behavior you *have* to configure your web proxies between your client and swift to XYZ" | 21:44 |
clayg | no one is trying to be gratuitously pedantic - but the idea is "when functests pass consumers of your cluster will generally be happy with your faithful presentation of a swift api compatible endpoint" | 21:45 |
clarkb | I'm not even sure if that is expected to be configurable, which is what makse this fun. I think what is happening is apache is asserting that chunked requests without content length info is an error. Not something that could be implemented? | 21:45 |
clarkb | and yup as a user of these apis, I definitely get the desire to be consistent (and why switching on web_front_end is not the greatest) | 21:46 |
clarkb | its possible the gzip + chunked is the problem though and we need to install mod_deflate? | 21:48 |
*** catintheroof has quit IRC | 21:53 | |
timburke | i wonder if the problem is that the body isn't actually gzipped? we're just sending random junk data, not a valid gzip stream... | 21:54 |
timburke | kinda seems like we ought to just drop the assertion, though -- the set of supported transfer encodings is pretty decidedly up to the WSGI server, not the app | 21:56 |
timburke | (which makes hacks like https://gist.github.com/tipabu/7c8ed7a713a7f51f20c2#file-compression-py all the more fun) | 21:57 |
clarkb | timburke: and instead support not 2XX? | 22:01 |
clarkb | s/support/assert/? | 22:01 |
timburke | clarkb: i'm thinking drop it entirely. why shouldn't a server decide to support gzip,chunked? | 22:02 |
clarkb | timburke: oh in this case I think we neglect to pass on the length info necessary to do chunked. SO we are asserting that it fails | 22:02 |
timburke | no, chunked doesn't need length info. that's the whole point | 22:03 |
timburke | now i'm wondering if we even do chunked right in that test.... | 22:04 |
timburke | i'm not seeing an immediate path from write_random to chunked_write, so i'm gonna say probably not | 22:07 |
clarkb | timburke: the first one fails because content-length is X not an actual integer. Then the second one fails because content-length itself is entirely missing. Then the third one fails because server either doesn't support gzip or it does and found the gzip to be invalid? | 22:07 |
timburke | yeah -- we're getting the 400/411 just fine, right? it's the 501 that's causing a problem? | 22:09 |
clarkb | and only the last (third) one is requesting chunked | 22:09 |
clarkb | timburke: correct, built in webserver returns 500, apache2 returns 501 | 22:09 |
clarkb | I think you are likely correct in that its a server error due to the invalid gzip data | 22:10 |
clarkb | so the bad header is in specifying a content type of gzip but web servers aren't consistent in what they return as the error? | 22:11 |
timburke | other way around, no? the assertion is for a 501. regardless, i just don't think https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2151-L2154 is terribly useful, so i'm in favor of dropping it entirely | 22:11 |
clarkb | oh yup apache2 is 500, builtin is 501. sorry | 22:11 |
*** jamielennox|away is now known as jamielennox | 22:13 | |
timburke | swift itself will actually send out a 501 (https://github.com/openstack/swift/blob/2.14.0/swift/common/swob.py#L774-L777 which gets caught by https://github.com/openstack/swift/blob/2.14.0/swift/common/constraints.py#L195-L197) so i wonder if apache is translating the server error to something more generic | 22:15 |
timburke | i wonder whether that's true for 503s or 507s as well... those are rather specific and useful in the context of swift... though maybe not something you want to expose to clients? idk... | 22:17 |
clarkb | I think its more that apache catches things before every proxying them | 22:18 |
clarkb | and so you end up with the error code apache returns? | 22:18 |
clarkb | gzip, chunked is a valid header value though right? swift just hasn't implemented the code to handle both together? | 22:20 |
timburke | yup, totally a valid header | 22:23 |
clarkb | timburke: clayg finally hunted down the error in the apache logs [Mon Jun 05 16:48:31.033144 2017] [proxy_http:error] [pid 27291:tid 140300542539520] [client 10.40.55.64:59438] [frontend 10.40.55.64:8080] AH01093: gzip,chunked Transfer-Encoding is not supported | 22:34 |
clarkb | so I think that is a matter of not having mod_deflate installed. | 22:34 |
clarkb | which should then proxy properly and get the 501 back? | 22:34 |
clayg | timburke: why would you expect we're not doing chunked transfer correctly?! | 22:35 |
timburke | clarkb: makes sense; even shows up on https://wiki.apache.org/httpd/ListOfErrors#line-1086 | 22:36 |
clayg | clarkb: timburke: you guys are going to give me an ulcer - stuff like request pipelineing and chunked transfer are *exactly* the kind of stuff we want functests validating very carefully? | 22:36 |
timburke | i don't think installing mod_deflate will fix the test, though; we'd most likely start getting back 201 | 22:37 |
clarkb | timburke: you don't think it would then bubble teh 501 from swift back out? | 22:37 |
timburke | as apache unzips (now that it knows how) and passes a plain-old chunked request on to swift | 22:37 |
clarkb | ah | 22:38 |
*** geaaru__ has quit IRC | 22:38 | |
timburke | clayg: chunked transfers, yes, we should validate those carefully. saying "this thing that's totally a reasonable thing to want to do should not be implemented" is a separate issue, and seems like a bad test | 22:39 |
timburke | clayg: i'm reasonably certain that https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2426-L2450 tests chunked transfers work. i'm not at all convinced that https://github.com/openstack/swift/blob/2.14.0/test/functional/tests.py#L2151-L2154 has the client speaking HTTP | 22:49 |
*** vint_bra has quit IRC | 22:49 | |
clayg | timburke: https://github.com/openstack/swift/blob/2.14.0/swift/common/swob.py#L767 | 22:50 |
clayg | idk, forwhatever reason we like only one te and we like it to be chunked? | 22:50 |
clayg | i don't really understand how the te can be unrolled - i sort of intuit that you could ... idk do a chunked encoding of a zlib compressed transfer of an entity - but I don't see who/how could unroll the zlib part first? | 22:52 |
timburke | clayg: sure, totally a reasonable thing for swift to want to do -- it's not in the business of zipping/unzipping payloads. but if eventlet were to add gzip support tomorrow, and did it by dropping the 'gzip,' from the transfer-encoding header, should the test break? | 22:52 |
clayg | it's probably bullshit that we return 501 there - and roughly inconsequential - but apache rejects the request before it even gets to us? | 22:52 |
clayg | yeah that's what i'm saying tho it can't drop the gzip from the header... just because it'd have to unchunk it first or you think it would rewrite the chunks as it deflates them? maybe... | 22:53 |
clarkb | yes apache is rejectnig the request before it gets to swift, I think because mod_deflate isn't enabled | 22:54 |
timburke | clayg: all eventlet exposes is a readable -- we shouldn't care much at all beyond that | 22:54 |
clayg | as far as what the test can validate... yeah that's harder to say... it's not unreasonable for the tests to assert any error that is reliably returned from the api I don't *think* - but that might be a little off the cuff | 22:54 |
clayg | timburke: yeah I think i understand what you're saying - if the wsgi server can turn over the request to us in such a way that's it's transparently proxied into a request we know how to service... | 22:56 |
*** mat128 has joined #openstack-swift | 22:59 | |
clayg | timburke: https://www.python.org/dev/peps/pep-0333/#other-http-features <- mentions transfer-encoding related to gzip and in other places | 23:00 |
clayg | it's not 100% clear to me what the prescription is tho? middleware shouldn't apply transfer-encoding gzip - but how do I know if my server is supporting it or not? are they supposed to mess with the hop-by-hop headers to indicate they've unzipped it? Still seems annoying for apache to decide it doesn't need to send the request to through . | 23:02 |
timburke | clayg: fwiw applying http://paste.openstack.org/show/611439/ has the test fail -- but with a 499, which is kinda strange. but it seems like eventlet partially caught the error? "ERROR WSGI: code 400, message Bad request syntax ('_\xbb..." | 23:02 |
timburke | clayg: apache as a reverse proxy to another process running on another port seems well-within rights to drop all hop-by-hop headers from the client | 23:04 |
clayg | timburke: yeah if we have a reverse proxy that would take such a request to swift and transform it in such a way that swift wanted saw a normal looking chunked transfer request and wanted to respond with success that would be something indeed | 23:05 |
timburke | and i think that's exactly what we'll see if we install mod_deflate -- though i suppose we'll see what clarkb reports back :-) | 23:07 |
clayg | ah, gotcha | 23:07 |
timburke | well, no, it'll probably 400 or something. on account of the not-really-a-chunked-body problem | 23:07 |
clarkb | ya should have change up momentarily now that meeting is over | 23:08 |
clayg | yeah cool, let's wait and see | 23:08 |
*** adriant has joined #openstack-swift | 23:08 | |
*** catintheroof has joined #openstack-swift | 23:13 | |
*** tonanhngo has quit IRC | 23:14 | |
clarkb | timburke: clayg telnet://[2001:4800:1ae1:18:f816:3eff:fe35:2c04]:19885 should be testing it (if you can't ipv6 the logs will be copied to log server when job completes). This tests it with the 500 assertion which we no longer expect to get so those tests should fail on that check | 23:16 |
openstackgerrit | Lingxian Kong proposed openstack/swift master: [WIP] Write-affinity aware object deletion https://review.openstack.org/470158 | 23:19 |
*** jamielennox is now known as jamielennox|away | 23:30 | |
*** chsc has quit IRC | 23:37 | |
*** kei_yama has joined #openstack-swift | 23:37 | |
clarkb | clayg timburke http://logs.openstack.org/74/372374/32/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/9289a40/console.html#_2017-06-05_23_37_47_926057 that didn't do what I expected it to do | 23:42 |
clarkb | [Mon Jun 05 23:37:21.830255 2017] [proxy_http:error] [pid 27198:tid 140482566924032] [client 10.40.68.150:47120] [frontend 10.40.68.150:8080] AH01093: gzip,chunked Transfer-Encoding is not supported still failing with that, maybe it just doesn't support it at all like swift? | 23:42 |
*** jamielennox|away is now known as jamielennox | 23:46 | |
clayg | clarkb: timburke: fwiw on the testFileSizeLimit test I'm pretty sure in all cases swift is never getting the request from apache, who's logging something and maybe buffering or something? maybe waiting on the client connection to do something it's not going to do because the test isn't that great? dunno yet | 23:57 |
clayg | I thought the 500 vs 501 assert bad headers thing was different? | 23:59 |
clayg | http://logs.openstack.org/74/372374/31/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/210c2f8/console.html#_2017-06-05_20_36_29_808245 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!