| openstackgerrit | James E. Blair proposed openstack/pyeclib master: Zuul: add file extension to playbook path https://review.openstack.org/523618 | 00:32 |
|---|---|---|
| *** ukaynar has quit IRC | 00:36 | |
| *** ukaynar has joined #openstack-swift | 00:36 | |
| *** ukaynar has quit IRC | 00:41 | |
| openstackgerrit | James E. Blair proposed openstack/pyeclib master: Zuul: add file extension to playbook path https://review.openstack.org/523618 | 00:41 |
| *** tovin07_ has joined #openstack-swift | 00:48 | |
| *** two_tired has joined #openstack-swift | 01:49 | |
| *** mat128 has joined #openstack-swift | 01:52 | |
| *** geaaru has quit IRC | 02:05 | |
| *** geaaru has joined #openstack-swift | 02:17 | |
| kota_ | good morning | 02:24 |
| *** threestrands has joined #openstack-swift | 03:05 | |
| mattoliverau | kota_: morning | 03:20 |
| kota_ | mattoliverau: o/ | 03:20 |
| *** ukaynar has joined #openstack-swift | 03:29 | |
| *** kei_yama has quit IRC | 03:39 | |
| *** links has joined #openstack-swift | 03:42 | |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** mat128 has quit IRC | 04:10 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:13 | |
| *** SkyRocknRoll has quit IRC | 04:17 | |
| *** d0ugal has quit IRC | 04:18 | |
| *** gkadam_ has joined #openstack-swift | 04:23 | |
| *** d0ugal has joined #openstack-swift | 04:25 | |
| *** threestrands has quit IRC | 04:29 | |
| *** ukaynar has quit IRC | 04:33 | |
| *** ukaynar has joined #openstack-swift | 04:34 | |
| *** armaan has joined #openstack-swift | 04:37 | |
| *** threestrands has joined #openstack-swift | 04:41 | |
| *** threestrands has quit IRC | 04:41 | |
| *** threestrands has joined #openstack-swift | 04:41 | |
| *** ukaynar has quit IRC | 05:01 | |
| *** ukaynar has joined #openstack-swift | 05:03 | |
| *** two_tired has quit IRC | 05:06 | |
| *** psachin has joined #openstack-swift | 05:12 | |
| *** threestrands has quit IRC | 05:32 | |
| *** geaaru has quit IRC | 05:54 | |
| *** cshastri has joined #openstack-swift | 06:06 | |
| *** psachin has left #openstack-swift | 06:13 | |
| *** gkadam_ has quit IRC | 06:58 | |
| *** armaan has quit IRC | 07:03 | |
| *** armaan has joined #openstack-swift | 07:19 | |
| *** m_kazuhiro has joined #openstack-swift | 07:26 | |
| *** rcernin has quit IRC | 07:43 | |
| *** tovin07__ has joined #openstack-swift | 08:01 | |
| *** tovin07_ has quit IRC | 08:03 | |
| *** pcaruana has joined #openstack-swift | 08:04 | |
| *** oshritf has quit IRC | 08:20 | |
| *** geaaru has joined #openstack-swift | 08:20 | |
| *** oshritf has joined #openstack-swift | 08:21 | |
| *** gkadam_ has joined #openstack-swift | 08:22 | |
| *** oshritf has quit IRC | 08:29 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:33 | |
| *** oshritf has joined #openstack-swift | 08:35 | |
| *** kei_yama has quit IRC | 08:35 | |
| *** SkyRocknRoll has quit IRC | 08:38 | |
| *** rcernin has joined #openstack-swift | 08:40 | |
| *** tovin07_ has joined #openstack-swift | 08:47 | |
| *** tovin07__ has quit IRC | 08:50 | |
| *** oshritf has quit IRC | 08:51 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:59 | |
| *** msno has joined #openstack-swift | 09:03 | |
| *** spectr has joined #openstack-swift | 09:03 | |
| msno | i am getting error from swift when adding big images to glance | 09:04 |
| msno | this is the error message | 09:04 |
| msno | http://paste.openstack.org/show/627678/ | 09:04 |
| msno | but the servers in question is working fine | 09:04 |
| msno | any pointers | 09:05 |
| *** oshritf has joined #openstack-swift | 09:08 | |
| *** oshritf has quit IRC | 09:12 | |
| *** armaan has quit IRC | 09:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 09:23 |
| *** ^andrea^ has joined #openstack-swift | 09:30 | |
| *** oshritf has joined #openstack-swift | 09:39 | |
| *** oshritf has quit IRC | 09:41 | |
| *** oshritf has joined #openstack-swift | 09:46 | |
| *** armaan has joined #openstack-swift | 09:46 | |
| *** msno_ has joined #openstack-swift | 09:49 | |
| *** oshritf has quit IRC | 09:52 | |
| *** msno has quit IRC | 09:53 | |
| *** oshritf has joined #openstack-swift | 09:55 | |
| *** oshritf has quit IRC | 10:00 | |
| *** oshritf has joined #openstack-swift | 10:07 | |
| *** tovin07_ has quit IRC | 10:14 | |
| *** m_kazuhiro has quit IRC | 10:23 | |
| *** mat128 has joined #openstack-swift | 10:26 | |
| *** oshritf has quit IRC | 10:26 | |
| *** spectr has quit IRC | 10:33 | |
| *** oshritf has joined #openstack-swift | 10:38 | |
| *** oshritf has quit IRC | 10:42 | |
| *** oshritf has joined #openstack-swift | 10:45 | |
| *** mvk has quit IRC | 10:56 | |
| *** mvk has joined #openstack-swift | 10:58 | |
| *** armaan has quit IRC | 11:42 | |
| *** armaan has joined #openstack-swift | 11:42 | |
| *** mvk has quit IRC | 11:47 | |
| *** armaan has quit IRC | 12:16 | |
| *** armaan_ has joined #openstack-swift | 12:16 | |
| *** mvk has joined #openstack-swift | 12:19 | |
| *** armaan_ has quit IRC | 12:32 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Follow up symlink patch https://review.openstack.org/523859 | 12:37 |
| *** armaan has joined #openstack-swift | 12:38 | |
| *** threestrands has joined #openstack-swift | 12:42 | |
| *** threestrands has quit IRC | 12:42 | |
| *** threestrands has joined #openstack-swift | 12:42 | |
| kota_ | msno: 499 response in your log means client closed the connection before all content arrived at your swift. could you check your glance settings? | 12:47 |
| *** psachin has joined #openstack-swift | 12:48 | |
| *** armaan has quit IRC | 12:48 | |
| kota_ | msno: i don't suppose the container-replicator error logs are related to the PUT behavior but it seems like container-server is also in something trouble. | 12:49 |
| *** armaan has joined #openstack-swift | 12:49 | |
| *** ukaynar has quit IRC | 13:00 | |
| *** threestrands has quit IRC | 13:01 | |
| *** ukaynar has joined #openstack-swift | 13:05 | |
| *** ukaynar has quit IRC | 13:09 | |
| *** SkyRocknRoll has quit IRC | 13:18 | |
| *** links has quit IRC | 13:21 | |
| *** hseipp has joined #openstack-swift | 13:35 | |
| *** zhurong has joined #openstack-swift | 13:38 | |
| *** openstackstatus has quit IRC | 13:43 | |
| *** openstack has joined #openstack-swift | 13:45 | |
| *** ChanServ sets mode: +o openstack | 13:45 | |
| *** openstackstatus has joined #openstack-swift | 13:45 | |
| *** ChanServ sets mode: +v openstackstatus | 13:45 | |
| *** abhitechie has joined #openstack-swift | 13:50 | |
| *** zhurong has quit IRC | 13:56 | |
| *** zhurong has joined #openstack-swift | 13:58 | |
| *** oshritf has quit IRC | 14:08 | |
| *** oshritf has joined #openstack-swift | 14:11 | |
| *** abhitechie has quit IRC | 14:14 | |
| *** mat128 has quit IRC | 14:19 | |
| *** rcernin has quit IRC | 14:27 | |
| *** psachin has quit IRC | 14:28 | |
| *** cshastri has quit IRC | 14:37 | |
| *** ukaynar has joined #openstack-swift | 14:40 | |
| *** zhurong has quit IRC | 14:46 | |
| *** openstack has quit IRC | 14:46 | |
| *** openstack has joined #openstack-swift | 14:50 | |
| *** ChanServ sets mode: +o openstack | 14:50 | |
| *** d0ugal has quit IRC | 14:56 | |
| *** ukaynar has quit IRC | 14:59 | |
| *** ukaynar has joined #openstack-swift | 14:59 | |
| msno_ | any experts who can make me under http://paste.openstack.org/show/627725/ | 15:00 |
| msno_ | i am fiddling with glance upload to swift for 2 days | 15:00 |
| msno_ | the services of swift is working fine | 15:00 |
| *** armaan has joined #openstack-swift | 15:01 | |
| *** mat128 has joined #openstack-swift | 15:01 | |
| *** d0ugal has joined #openstack-swift | 15:04 | |
| msno_ | kota_, to answer your query .. ssmall oblects are success in glance | 15:04 |
| tdasilva | msno_: are you able to test swift isolated from glance? just to start eliminating variables, i'd start with trying to upload small object and large objects into swift directly | 15:11 |
| msno_ | tdasilva, its failing | 15:12 |
| msno_ | let me check once more\ | 15:12 |
| msno_ | a 2 GB file and a small file | 15:12 |
| acoles | msno_: the container-replicator errors suggest that container server is not running or is binding a different port than in the ring | 15:12 |
| *** psachin has joined #openstack-swift | 15:13 | |
| msno_ | tdasilva, i have a glance contianer | 15:13 |
| msno_ | you want me to try to upload there or some other container | 15:13 |
| tdasilva | msno_ it shouldn't matter, but acoles has a good point, i'd also checkout the rings | 15:14 |
| tdasilva | msno_: looks like you are using tripleo and there were some fixes done recently to tripleo newton that fixed some issues with rings | 15:14 |
| msno_ | tdasilva, acoles , for only 2 IPS i get this error.. for .64 its fine | 15:14 |
| msno_ | let me check the difference of service status between 64 and other ctrls | 15:15 |
| *** d0ugal_ has joined #openstack-swift | 15:17 | |
| msno_ | tdasilva, is it fine to delete the /var/cache/swift/* | 15:17 |
| *** mat128 has quit IRC | 15:17 | |
| *** d0ugal has quit IRC | 15:20 | |
| *** mat128 has joined #openstack-swift | 15:20 | |
| msno_ | tdasilva, its an old version .. | 15:21 |
| msno_ | not mitaka | 15:21 |
| tdasilva | msno_: why do you need to delete /var/cache/swift? | 15:23 |
| *** mat128 has quit IRC | 15:24 | |
| *** mat128 has joined #openstack-swift | 15:25 | |
| msno_ | tdasilva, ok.. i am directly uploading a 2gb qcow2 to glance contianer now | 15:25 |
| msno_ | swift upload glance --os-project-name services <filename> | 15:25 |
| msno_ | a 2GB upload to swift fauiled | 15:31 |
| msno_ | ientException: Object HEAD failed: https://14.191.184.140:8080/v1/AUTH_eaaccf6d4e7344e187862924d30322ee/glance/base.qcow2 404 Not Found | 15:31 |
| msno_ | what does this mean | 15:31 |
| tdasilva | need more context, can you share more of the log file? | 15:34 |
| tdasilva | msno_: have you checked the rings to see if they are ok? | 15:34 |
| *** d0ugal has joined #openstack-swift | 15:35 | |
| msno_ | tdasilva,how to test if rings are ok | 15:35 |
| msno_ | i guess its rinz issue only | 15:35 |
| tdasilva | swift-recon --md5 | 15:36 |
| tdasilva | might need `sudo swift-recon --md5` | 15:36 |
| *** d0ugal_ has quit IRC | 15:37 | |
| tdasilva | msno_: also, make sure to run that from one of the controller nodes, not the undercloud | 15:37 |
| msno_ | tdasilva, thansk .. let me chk | 15:37 |
| *** psachin has quit IRC | 15:37 | |
| msno_ | tdasilva, the rings are fine http://paste.openstack.org/show/627731/ | 15:40 |
| *** armaan has quit IRC | 15:45 | |
| *** armaan has joined #openstack-swift | 15:45 | |
| tdasilva | msno_: just to be on the safe side, can you check the account, container rings too, `swift-recon container --md5`, same for account... if they are all good, then it would be good to look at logs | 15:46 |
| msno_ | tdasilva, sure | 15:46 |
| *** Sukhdev has joined #openstack-swift | 15:47 | |
| msno_ | tdasilva, 3 hosts matched.. 0 errors for account .. object and container | 15:50 |
| msno_ | ccount-server: 14.191.182.63 - - [29/Nov/2017:15:34:21 +0000] "HEAD /d1/713/.misplaced_objects" 404 - "HEAD http://localhost/v1/.misplaced_objects" "tx39398603b7664b589ff8d-005a1ed37d" "proxy-server 23012" 0.0003 "-" 25432 - | 15:51 |
| msno_ | all misplaced errors are for object 713 | 15:51 |
| msno_ | tdasilva, swift.log = http://paste.openstack.org/show/627732/ | 15:56 |
| msno_ | tdasilva, ignore this log .. let me clear it and make a new swift upload | 15:59 |
| msno_ | then i will share | 15:59 |
| *** psachin has joined #openstack-swift | 16:03 | |
| *** psachin has quit IRC | 16:08 | |
| *** psachin has joined #openstack-swift | 16:10 | |
| *** armaan has quit IRC | 16:13 | |
| *** mat128 has quit IRC | 16:14 | |
| *** armaan has joined #openstack-swift | 16:14 | |
| msno_ | tdasilva, acoles can you help me understand this just to be on the safe side, can you check the account, container rings too, `swift-recon container --md5`, same for account... if they are all good, then it would be good to look at logs | 16:23 |
| msno_ | tdasilva, acoles can you help me understand this http://paste.openstack.org/show/627734/ | 16:23 |
| msno_ | this has more details : http://paste.openstack.org/show/627735/ | 16:26 |
| *** psachin has quit IRC | 16:31 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:33 | |
| *** psachin has joined #openstack-swift | 16:39 | |
| *** dc588 has joined #openstack-swift | 16:40 | |
| *** chsc has joined #openstack-swift | 16:41 | |
| dc588 | Is there an easy way to find out storage utilization on individual storage nodes. The swift-recon -d gives information about the cluster as a whole. | 16:44 |
| *** psachin has quit IRC | 16:45 | |
| tdasilva | msno_: jumping between a bunch of things, will get back to you in a bit | 16:52 |
| *** Sukhdev has quit IRC | 17:06 | |
| *** vinsh_ has quit IRC | 17:14 | |
| *** gkadam_ has quit IRC | 17:27 | |
| msno_ | tdasilva, sure .. np | 17:33 |
| timburke | good morning | 17:36 |
| *** silor has joined #openstack-swift | 17:46 | |
| timburke | msno_: looking at http://paste.openstack.org/show/627735/ it seems the proxy had trouble contacting backend servers. if you grep for tx55f3f0fe2d0740beaf629 in the server logs, it should have more information | 17:52 |
| timburke | something like `proxy-server: ERROR with Object server 127.0.0.1:6040/sdb4 re: Expect: 100-continue on /AUTH_test/c/o: Connection refused` maybe | 17:53 |
| timburke | culminating with something like `Object PUT returning 503, 0/5 required connections` | 17:53 |
| timburke | the 404's a red herring -- if you upgrade swiftclient, i think https://github.com/openstack/python-swiftclient/commit/481f74c fixed it | 17:54 |
| msno_ | timburke, thanks.. let me debug further | 17:55 |
| *** cshastri has joined #openstack-swift | 17:57 | |
| msno_ | timburke, http://paste.openstack.org/show/627746/ | 17:57 |
| msno_ | when you say proxy server issue.. i need to check if swift proxy has any issue in communication with my controllers .. right? | 17:58 |
| msno_ | will it help if i increase timeout value in proxy conf of swift | 18:01 |
| tdasilva | ccccccdkkgthhrlglitcebvifklejgfuvdfculhdnitt | 18:02 |
| tdasilva | ccccccdkkgthcigrtuunjktcilkfbirktfclvfcvbkni | 18:02 |
| tdasilva | sorry | 18:02 |
| timburke | so that's the log for the object server... which is handy, it tells us a bit about what's going on there... but where's the proxy-server? should be ip 14.191.184.140 | 18:03 |
| timburke | (i'm right in thinking you're currently grepping on 14.191.182.64, yeah?) | 18:04 |
| timburke | increasing timeouts seems like a not-crazy thing to try... but i would have thought the defaults should be ok | 18:05 |
| *** hseipp has quit IRC | 18:06 | |
| *** cshastri has quit IRC | 18:06 | |
| timburke | it *does* kinda smell like a timeout -- looks like after ~100s the object-server gave up. how big's the object? | 18:07 |
| clayg | lazy object server | 18:08 |
| timburke | i know slo/dlo have max_get_time -- are there any similar total-request timeouts during PUTs? or is it down to just client_timeout and node_timeout (which are renewed every client_chunk_size bytes or something like that)? | 18:15 |
| msno_ | timburke, the file size is 2.1 GB | 18:20 |
| msno_ | timburke, i did a grep from from master server with ip 14.191.182.62 | 18:23 |
| msno_ | timburke, how would i check timeout between each client_chunk_size push | 18:27 |
| *** ukaynar has quit IRC | 18:27 | |
| *** armaan has quit IRC | 18:35 | |
| *** gkadam_ has joined #openstack-swift | 18:35 | |
| *** mvk has quit IRC | 18:36 | |
| *** SkyRocknRoll has quit IRC | 18:36 | |
| *** SkyRocknRoll has joined #openstack-swift | 18:37 | |
| *** mat128 has joined #openstack-swift | 18:40 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 18:50 | |
| *** Sukhdev has joined #openstack-swift | 18:59 | |
| *** mvk has joined #openstack-swift | 19:09 | |
| msno_ | timburke, thanks for the pointer .. i increased the timeout to node_timeout = 120 | 19:13 |
| msno_ | restarted swift | 19:13 |
| msno_ | and it worked.. | 19:13 |
| msno_ | thanks for everyone support .. you guyz are awesome...!! | 19:13 |
| *** msno_ has quit IRC | 19:15 | |
| *** armaan has joined #openstack-swift | 19:15 | |
| *** armaan has quit IRC | 19:17 | |
| *** armaan has joined #openstack-swift | 19:17 | |
| *** SkyRocknRoll_ has quit IRC | 19:27 | |
| *** anupn has joined #openstack-swift | 19:41 | |
| anupn | Hi Folks, i am curious to know if are there any plans of supporting py3 in swift soon? I have a patch set https://review.openstack.org/#/c/509652/13 for supporting py3 inside ironic-python-agent and right now the jobs that need swift to run with py3 are failing | 19:43 |
| patchbot | patch 509652 - ironic-python-agent - [Test]DNM Testing if TinyIPA builds with py3 | 19:43 |
| *** linkmark has joined #openstack-swift | 19:50 | |
| *** SkyRocknRoll has quit IRC | 19:53 | |
| timburke | anupn: it's a pretty low priority for most of the community -- we've been bitten by a few separate bugs in trying to get some modules to be py3-compatible, so i think most everyone's a little gun-shy to either write new py3 patches or review the handful of outstanding ones | 20:04 |
| timburke | between that and the hope to eventually have a golang object-server (at least), i feel like we're approaching a fork in the road and there's a definite possibility that we'll wind up in a field :-/ | 20:06 |
| anupn | timburke, ah I see, he he. | 20:06 |
| anupn | timburke, thanks for the update | 20:06 |
| timburke | sure. fwiw, there's https://review.openstack.org/#/c/401397/ (though it needs to be rebased) and https://gist.github.com/tipabu/833b03a865dba96e9fa2230b82f5d075 has been collecting thoughts about some known pitfalls | 20:07 |
| patchbot | patch 401397 - swift - py3: port common/ring/ and common/utils.py | 20:07 |
| anupn | timburke, that's what i wondered how come community has not thought of supporting py3 as like mine there might be other bugs depedning upon swift | 20:08 |
| * anupn hopes to see the support | 20:09 | |
| *** spectr has joined #openstack-swift | 20:18 | |
| *** spectr has quit IRC | 20:20 | |
| *** Sukhdev has quit IRC | 20:26 | |
| *** threestrands has joined #openstack-swift | 20:47 | |
| *** threestrands has quit IRC | 20:47 | |
| *** threestrands has joined #openstack-swift | 20:47 | |
| *** silor has quit IRC | 20:49 | |
| mattoliverau | So by the ajenda I assume no meeting today? | 20:53 |
| mattoliverau | If that's the case, that's great cause I wanna go back to bed and get rid of this here damn flu :( | 20:54 |
| timburke | yeah, no meeting | 20:56 |
| timburke | http://eavesdrop.openstack.org/meetings/swift/2017/swift.2017-11-22-21.00.log.html#l-58 | 20:57 |
| timburke | get well soon, mattoliverau! | 20:57 |
| *** pcaruana has quit IRC | 21:00 | |
| *** d0ugal has quit IRC | 21:07 | |
| *** mat128 has quit IRC | 21:37 | |
| jungleboyj | Ah, ok, good to know. | 22:01 |
| *** rcernin has joined #openstack-swift | 22:03 | |
| *** ejat has quit IRC | 22:25 | |
| *** ejat has joined #openstack-swift | 22:26 | |
| *** ejat has quit IRC | 22:26 | |
| *** ejat has joined #openstack-swift | 22:26 | |
| *** dc588 has quit IRC | 22:29 | |
| *** d0ugal has joined #openstack-swift | 23:04 | |
| *** hoonetorg has joined #openstack-swift | 23:10 | |
| *** JimCheung has joined #openstack-swift | 23:21 | |
| *** linkmark has quit IRC | 23:25 | |
| *** kei_yama has joined #openstack-swift | 23:31 | |
| *** chsc has quit IRC | 23:34 | |
| *** geaaru has quit IRC | 23:39 | |
| *** tesseract has joined #openstack-swift | 23:48 | |
| *** tesseract has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!