| *** mtreinish has quit IRC | 00:12 | |
| *** kei_yama has joined #openstack-swift | 00:21 | |
| *** mtreinish has joined #openstack-swift | 00:37 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: wip: delete parts as we go https://review.openstack.org/541083 | 00:45 |
|---|---|---|
| *** Supun has joined #openstack-swift | 00:50 | |
| *** Supun has quit IRC | 01:23 | |
| *** ukaynar has quit IRC | 01:32 | |
| *** ukaynar has joined #openstack-swift | 01:33 | |
| *** itlinux has joined #openstack-swift | 02:13 | |
| *** itlinux has quit IRC | 02:43 | |
| *** ukaynar has quit IRC | 02:47 | |
| *** bkopilov has quit IRC | 02:50 | |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift master: Refactor expirer's task round robin implementation https://review.openstack.org/538171 | 03:07 |
| *** d0ugal_ has joined #openstack-swift | 03:17 | |
| *** d0ugal has quit IRC | 03:19 | |
| *** gyee has quit IRC | 03:29 | |
| *** cshastri has joined #openstack-swift | 03:46 | |
| *** bkopilov has joined #openstack-swift | 04:08 | |
| *** psachin has joined #openstack-swift | 04:26 | |
| kota_ | hello world | 04:36 |
| kota_ | looking at my twitter stream, it looks the FOSDEM is in this week, acoles, tdasilva, and cschwede seem to join it. | 04:44 |
| * kota_ is looking at https://fosdem.org/2018/schedule/track/software_defined_storage/ | 04:44 | |
| mattoliverau | kota_: o/ | 04:50 |
| mattoliverau | I hope they were recorded :) | 04:50 |
| *** links has joined #openstack-swift | 05:12 | |
| *** itlinux has joined #openstack-swift | 05:17 | |
| *** m_kazuhiro has joined #openstack-swift | 05:20 | |
| *** ukaynar has joined #openstack-swift | 05:23 | |
| *** armaan has quit IRC | 05:25 | |
| *** gkadam has joined #openstack-swift | 05:25 | |
| *** armaan has joined #openstack-swift | 05:28 | |
| m_kazuhiro | good morning | 05:39 |
| *** itlinux has quit IRC | 05:55 | |
| *** d0ugal_ has quit IRC | 06:14 | |
| kota_ | mattoliverau: yeah, it looks like they have videos for the presentations | 06:14 |
| mattoliverau | \o/ | 06:14 |
| kota_ | mattoliverau: https://fosdem.org/2018/news/2018-02-04-first-videos-online/ | 06:15 |
| kota_ | and hopefully, all of them would be uploaded to the youtube channel. | 06:15 |
| kota_ | i added the channel to my list ;) https://www.youtube.com/user/fosdemtalks | 06:15 |
| mattoliverau | kota_: thanks! | 06:17 |
| kota_ | I'd like to hear how's worth from acoles, tdasilva, cschwede. If it's nice place to discuss the Software Defined Storage, I'd like to put it into my radar that I should go in the next time. | 06:18 |
| mattoliverau | yeah, especually with your storlets hat on too | 06:18 |
| kota_ | indeed, I put myself as candidacy to the next Storlets PTL role. | 06:19 |
| *** d0ugal_ has joined #openstack-swift | 06:24 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Add and pipe reconstructor stats through recon https://review.openstack.org/541141 | 06:27 |
| *** threestrands has quit IRC | 06:40 | |
| *** ianychoi_ has quit IRC | 06:47 | |
| *** ianychoi_ has joined #openstack-swift | 06:48 | |
| *** rcernin has quit IRC | 07:37 | |
| *** hseipp has joined #openstack-swift | 07:47 | |
| *** hseipp has quit IRC | 07:55 | |
| *** hseipp has joined #openstack-swift | 07:58 | |
| *** m_kazuhiro has quit IRC | 07:59 | |
| *** neonpastor has quit IRC | 08:00 | |
| *** neonpastor has joined #openstack-swift | 08:02 | |
| *** pcaruana has joined #openstack-swift | 08:14 | |
| *** tesseract has joined #openstack-swift | 08:22 | |
| *** ianychoi_ has quit IRC | 08:27 | |
| *** ianychoi_ has joined #openstack-swift | 08:28 | |
| *** kei_yama has quit IRC | 08:34 | |
| *** armaan has quit IRC | 08:37 | |
| *** armaan has joined #openstack-swift | 08:37 | |
| *** frti has joined #openstack-swift | 08:48 | |
| acoles | good morning | 08:51 |
| acoles | kota_: mattoliverau the swift talk videos (and other software defined storage talks) from FOSDEM are here https://ftp.heanet.ie/mirrors/fosdem-video/2018/H.2213/ | 08:55 |
| acoles | there are still some to come - e.g. Thiago's is not there yet | 08:56 |
| *** d0ugal_ has quit IRC | 08:57 | |
| *** d0ugal has joined #openstack-swift | 08:57 | |
| *** d0ugal has quit IRC | 08:57 | |
| *** d0ugal has joined #openstack-swift | 08:57 | |
| acoles | wait, this is a much better link https://fosdem.org/2018/schedule/track/software_defined_storage/ | 08:58 |
| mattoliverau | acoles: cool! ta. How'd it go? have a good time at fosdem? | 08:59 |
| acoles | mattoliverau: yes, good weekend, and good to have some swift talks on the agenda. The conference is big and unfortunately it can be impossible to get in to the more popular talks, so I'll be catching up on videos. | 09:02 |
| *** cbartz has joined #openstack-swift | 09:02 | |
| *** threestrands has joined #openstack-swift | 09:06 | |
| acoles | also bumped into Jan from Seznam who played pool with some of us in Boston | 09:06 |
| *** threestrands has quit IRC | 09:21 | |
| *** frti_ has joined #openstack-swift | 09:21 | |
| mattoliverau | Nice, small world | 09:24 |
| *** frti_ has quit IRC | 09:25 | |
| *** frti has quit IRC | 09:25 | |
| *** hoonetorg has quit IRC | 09:28 | |
| *** hoonetorg has joined #openstack-swift | 09:41 | |
| *** amrith has left #openstack-swift | 10:07 | |
| *** ukaynar has quit IRC | 10:11 | |
| *** kallenp has joined #openstack-swift | 10:24 | |
| kallenp | Hallo. Please, I have SAIO and now, I am need to add replication server and replicate my SAIO to this new server. SAIO will be still primary for me. Can I ask for a help howto to do it ? Or URL with instructions ? Thanks, Petr | 10:25 |
| *** bkopilov has quit IRC | 10:27 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Merge master into feature/deep https://review.openstack.org/541233 | 10:29 |
| -openstackstatus- NOTICE: Our Zuul infrastructure is currently experiencing some problems and processing jobs very slowly, we're investigating. Please do not approve or recheck changes for now. | 10:30 | |
| *** kallenp has quit IRC | 10:33 | |
| *** kallenp has joined #openstack-swift | 10:34 | |
| *** mrjk_ has quit IRC | 10:36 | |
| *** mrjk_ has joined #openstack-swift | 10:36 | |
| *** kallenp has quit IRC | 10:55 | |
| *** kallenp has joined #openstack-swift | 11:04 | |
| *** kallenp1 has joined #openstack-swift | 11:13 | |
| *** kallenp has quit IRC | 11:17 | |
| *** kallenp1 has left #openstack-swift | 11:20 | |
| *** gkadam is now known as gkadam-brb | 11:27 | |
| kota_ | acoles: thanks for sharing those info! I'll catch up the slides and videos | 11:34 |
| *** links has quit IRC | 11:41 | |
| *** gkadam-brb is now known as gkadam | 11:49 | |
| bob_cheesey | morning all - i was wondering if anyone could point me in the right direction as to why i see a fair bit of 'proxy server Handoff requested' being repeated 3 times across all three proxies in a cluster? the cluster is fairly busy (public cloud provider) and those log entries occurred ~1 million times in total across all three proxies yesterday, so there were ~330k handoffs in total | 11:49 |
| kota_ | notmyname: just as notification, OpenStack PTL nomination will end in about 1 day and 12 hours, I found no one in the Swift directory in the https://github.com/openstack/election/tree/master/candidates/rocky/Swift | 11:50 |
| kota_ | no one is in, yet | 11:50 |
| * kota_ expects notmyname will be keeping to run his position as well. | 11:51 | |
| bob_cheesey | kota_: i assumed as much, just thought i'd get it down whilst it was fresh in my mind | 11:51 |
| kota_ | bob_cheesey: it depends on the reason why handoff requested. I think various reasons can exist according to your cluster health. | 11:53 |
| *** links has joined #openstack-swift | 11:54 | |
| kota_ | e.g. connection timeout, full disks, affinity settings, .... etc | 11:54 |
| kota_ | bob_cheesey: did you find any error/warning or 5xx responses from object-servers? | 11:55 |
| bob_cheesey | storage boxes seem fine - that's the first place i looked | 11:57 |
| bob_cheesey | no 500s, a ton of 404s from one particular customer but that's about it | 11:57 |
| *** Supun has joined #openstack-swift | 11:57 | |
| kota_ | i don't have clear memory for now on the area for now but if i remember correctly, GET 404s should cause handoff requests to find the objects | 12:08 |
| openstackgerrit | Ondřej Nový proposed openstack/swift master: WIP: Quarantine DB without *_stat row https://review.openstack.org/541252 | 12:09 |
| *** fyxim has quit IRC | 12:11 | |
| *** fyxim has joined #openstack-swift | 12:11 | |
| kota_ | but ... i'm not sure the logs happens on the GET... let me check the code. | 12:12 |
| bob_cheesey | thanks kota_: appreciate it! | 12:12 |
| bob_cheesey | if i look at the file causing the 404 I can see it attempt to HEAD the file on 3 different devices, then finally it PUTSs it and then a successful HEAD occurs | 12:15 |
| bob_cheesey | so i think the problem is more the client's software rather than the cluster, but i'm not 100% certain yet | 12:15 |
| kota_ | bob_cheesey: if you can find the correct file in another device than the first 3 primary candidates, you may doubt your replication status that should work to move the file into the primary nodes. | 12:18 |
| kota_ | i checked, 404 will be redirected to other candidacies in my cluster but handoff logs were not found. perhaps, that is from my misconfiguration on rsyslog and warning logs | 12:19 |
| bob_cheesey | kota_: it's not finding it which i think means it doesn't exist at all, replication seems fine | 12:19 |
| bob_cheesey | ok thanks, appreciate you looking! | 12:19 |
| kota_ | bob_cheesey: sorry, i couldn't reach the root cause. | 12:22 |
| bob_cheesey | no thank you, you've been very helpful! | 12:22 |
| kota_ | oic, handoffs under (request_node_count - num_primaries) won't be logged so in my tiny env (with small number of devices) I won't see the log message anyway. | 12:54 |
| kota_ | timeup, i have to left my office. | 12:54 |
| *** links has quit IRC | 12:55 | |
| *** takamatsu has joined #openstack-swift | 12:57 | |
| *** zhurong_ has joined #openstack-swift | 13:01 | |
| *** links has joined #openstack-swift | 13:08 | |
| *** cshastri has quit IRC | 13:16 | |
| *** Supun has quit IRC | 13:27 | |
| *** bkopilov has joined #openstack-swift | 13:27 | |
| *** Supun has joined #openstack-swift | 13:28 | |
| *** zhurong_ has quit IRC | 13:52 | |
| *** psachin has quit IRC | 14:04 | |
| *** links has quit IRC | 14:38 | |
| *** Aravind has joined #openstack-swift | 14:38 | |
| Aravind | hi team i am facing problem with swift during upload of files | 14:38 |
| Aravind | i am able to create containres | 14:38 |
| Aravind | but during upload i am getting below error on proxy server | 14:38 |
| Aravind | swift-proxy-server: ERROR with Object server 192.168.47.129:6201/sda re: Trying to write to /AUTH_admin/cbd/abc: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/proxy/controllers/obj.py", line 1611, in _send_file#012 self.conn.send(to_send)#012 File "/usr/lib64/python2.7/httplib.py", line 840, in send#012 self.sock.sendall(data)#012 File "/usr/lib/python2.7/site-packages/even | 14:39 |
| Aravind | (data)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 393, in sendall#012 tail += self.send(data[tail:], flags)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send#012 return self._send_loop(self.fd.send, data, flags)#012 File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop#012 return send_method(data, *args)#012 | 14:39 |
| Aravind | ANy help will be appreciated? | 14:39 |
| *** gkadam has quit IRC | 14:44 | |
| *** Aravind has quit IRC | 14:52 | |
| *** derekjhyang has quit IRC | 14:55 | |
| *** derekjhyang has joined #openstack-swift | 14:56 | |
| *** Supun has quit IRC | 15:08 | |
| openstackgerrit | Ondřej Nový proposed openstack/swift master: Quarantine DB without *_stat row https://review.openstack.org/541252 | 15:09 |
| *** jungleboyj has quit IRC | 15:36 | |
| *** jungleboyj has joined #openstack-swift | 15:36 | |
| *** Supun has joined #openstack-swift | 15:39 | |
| *** armaan has quit IRC | 15:40 | |
| *** armaan has joined #openstack-swift | 15:41 | |
| *** Supun has quit IRC | 15:44 | |
| *** pcaruana has quit IRC | 15:51 | |
| *** samueldmq has quit IRC | 15:52 | |
| *** samueldmq has joined #openstack-swift | 15:52 | |
| *** armaan has quit IRC | 16:07 | |
| *** armaan has joined #openstack-swift | 16:08 | |
| *** clayg has quit IRC | 16:13 | |
| *** clayg has joined #openstack-swift | 16:14 | |
| *** ChanServ sets mode: +v clayg | 16:14 | |
| *** ukaynar has joined #openstack-swift | 16:26 | |
| *** itlinux has joined #openstack-swift | 16:34 | |
| notmyname | good morning | 16:52 |
| *** gyee has joined #openstack-swift | 17:02 | |
| *** derekjhyang has quit IRC | 17:05 | |
| *** cbartz has left #openstack-swift | 17:17 | |
| *** hseipp has quit IRC | 17:22 | |
| *** mtreinish has quit IRC | 17:43 | |
| *** mtreinish has joined #openstack-swift | 17:50 | |
| *** tesseract has quit IRC | 18:00 | |
| *** ukaynar has quit IRC | 18:01 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Don't lose objects when db is rsync replicated during sharding https://review.openstack.org/536943 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Remove special case for mismatched last cleaved pointer https://review.openstack.org/538009 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: make get_own_shard_range() always return something https://review.openstack.org/538010 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Make find_shard_ranges cope with misplaced objects https://review.openstack.org/538011 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Add an end_marker property to ShardRange https://review.openstack.org/538326 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Fix misplaced_objects method and align with cleave method https://review.openstack.org/538327 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Reduce scope: do not merge misplaced objects into container listing https://review.openstack.org/541388 | 18:06 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Avoid object update redirection loops https://review.openstack.org/541389 | 18:06 |
| *** ukaynar has joined #openstack-swift | 18:17 | |
| *** ukaynar has quit IRC | 18:27 | |
| onovy | notmyname: good morning, can you help me with #541252 please? | 18:27 |
| onovy | https://review.openstack.org/541252 | 18:28 |
| patchbot | patch 541252 - swift - Quarantine DB without *_stat row | 18:28 |
| *** ukaynar has joined #openstack-swift | 18:28 | |
| notmyname | onovy! good to see you :-) | 19:07 |
| onovy | :) | 19:07 |
| notmyname | I heard Jan was at fosdem and saw acoles and tdasilva and cschwede | 19:07 |
| tdasilva | we had lunch together, pretty awesome! | 19:08 |
| notmyname | onovy: looks like that patch is a small refactor and an extra check on a container db. I'm assuming you've run across this issue in your clusters? | 19:11 |
| onovy | notmyname: look into bug | 19:17 |
| notmyname | yeah. I did after I typed that | 19:18 |
| onovy | :) | 19:18 |
| notmyname | (note to self: read first, then type) | 19:18 |
| onovy | i have that corrupted DB and this patch show quarantine this kind of "wrong db" | 19:18 |
| onovy | problem is, that tests are failing | 19:18 |
| onovy | show->should | 19:18 |
| onovy | i can't explain how this situation happends, probably incosistency of sqlite somewhere | 19:19 |
| notmyname | in the current patch, the tests should fail? | 19:20 |
| notmyname | I'm running tests on the patch in my own saio now | 19:20 |
| onovy | https://8b86aea46fb38e6450f2-0e5f4c086da474abc1df58826577db2f.ssl.cf1.rackcdn.com/541252/9885/ | 19:20 |
| onovy | unit tests work | 19:20 |
| onovy | func tests don't | 19:20 |
| notmyname | ah right. that's... not accurate. I had turned those off, but somehow they got turned on. I don't think those failures are because of your patch | 19:21 |
| onovy | hmm some of them fails in update_meta | 19:22 |
| onovy | which is code i just changed | 19:22 |
| timburke | let's see the results from zuul | 19:28 |
| notmyname | all passed locally on my saio | 19:28 |
| notmyname | I woulnd't worry about the qa cluster tests (and I disabled them again because they're reporting bad info) | 19:28 |
| timburke | gd, the check queue is 400+ deep! | 19:29 |
| notmyname | oh wow | 19:29 |
| timburke | could be a while... onovy, go enjoy your evening, sleep, and see what the morning brings :-) | 19:31 |
| timburke | we'll try to get some +2s on it in the mean time | 19:31 |
| *** peterlisak has quit IRC | 19:46 | |
| *** onovy has quit IRC | 19:47 | |
| *** onovy has joined #openstack-swift | 20:03 | |
| *** onovy has quit IRC | 20:07 | |
| *** onovy has joined #openstack-swift | 20:20 | |
| *** onovy has quit IRC | 20:25 | |
| *** peterlisak has joined #openstack-swift | 20:32 | |
| *** onovy has joined #openstack-swift | 20:33 | |
| *** takamatsu has quit IRC | 21:01 | |
| *** threestrands has joined #openstack-swift | 21:32 | |
| timburke | torgomatic: the migration actually runs in a transaction, though, right? so if it fails, it should roll back. i'm fairly certain p 541252 came up in the context of a failed *initialization* | 21:40 |
| patchbot | https://review.openstack.org/#/c/541252/ - swift - Quarantine DB without *_stat row | 21:40 |
| timburke | (at least, that's the only way *i* can imagine it coming up) | 21:40 |
| torgomatic | timburke: true; I'm also considering a failure of imagination to include a failed migration | 21:40 |
| torgomatic | so if we failed to imagine that some code path might need a migration, then that could be blowing up because we missed a spot in the code | 21:41 |
| timburke | maybe it'd be worth us initializing in a temp dir, then mv'ing when we think it's ready? | 21:41 |
| timburke | ah, hmm... | 21:41 |
| torgomatic | the DB in the bug has a container_stat table and a container_info table, but they're both empty | 21:43 |
| *** peterlisak has quit IRC | 21:44 | |
| torgomatic | or maybe it has a container_info table and a container_stat view | 21:45 |
| torgomatic | yeah, definitely a table and a view, so that makes more sense | 21:45 |
| *** onovy has quit IRC | 21:45 | |
| *** itlinux has quit IRC | 21:50 | |
| torgomatic | timburke: looks like _initialize is already taking place inside a transaction, and it always inserts a row into container_info, so I'm not sure how this wound up empty | 21:51 |
| torgomatic | Maybe I'm paranoid, but I worry that there's some code path that could empty out container_info and screw things up | 21:51 |
| torgomatic | of course, it could be your garden-variety FS corruption | 21:52 |
| *** itlinux has joined #openstack-swift | 21:55 | |
| *** openstackgerrit has quit IRC | 22:16 | |
| *** peterlisak has joined #openstack-swift | 22:22 | |
| *** onovy has joined #openstack-swift | 22:25 | |
| *** rcernin has joined #openstack-swift | 22:28 | |
| *** ukaynar has quit IRC | 22:41 | |
| *** ukaynar has joined #openstack-swift | 22:42 | |
| *** itlinux has quit IRC | 22:53 | |
| mattoliverau | oh I forgot to day morning here... so morning :) | 23:16 |
| *** armaan has quit IRC | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!