Friday, 2019-02-22

*** mikecmpbll has quit IRC00:45
*** gyee has quit IRC00:54
*** takamatsu_ has joined #openstack-swift01:26
*** takamatsu has quit IRC01:26
openstackgerritMerged openstack/python-swiftclient master: Update hacking version
*** ccamacho has quit IRC02:48
*** two_tired has joined #openstack-swift03:01
*** psachin has joined #openstack-swift03:04
*** two_tired has quit IRC05:02
*** gmann has quit IRC05:28
openstackgerritMerged openstack/swift master: py3: port some more middleware tests
zaitcevSo, this went in, but versioned writes didn't.05:56
mattoliverauversion writes needed another recheck :(06:30
*** e0ne has joined #openstack-swift07:29
*** e0ne has quit IRC07:43
*** e0ne has joined #openstack-swift07:44
*** e0ne has quit IRC07:44
*** e0ne has joined #openstack-swift07:50
*** e0ne has quit IRC08:10
*** tkajinam has quit IRC08:13
*** hseipp has joined #openstack-swift08:18
*** mikecmpbll has joined #openstack-swift09:06
*** hseipp has quit IRC09:31
*** mikecmpbll has quit IRC09:38
*** mikecmpbll has joined #openstack-swift09:39
*** takamatsu_ has quit IRC10:04
*** takamatsu has joined #openstack-swift10:05
*** early has quit IRC10:15
*** e0ne has joined #openstack-swift10:19
*** early has joined #openstack-swift10:21
*** takamatsu_ has joined #openstack-swift10:23
*** takamatsu has quit IRC10:24
*** rcernin has quit IRC10:31
*** hseipp has joined #openstack-swift10:42
*** takamatsu_ has quit IRC10:48
*** takamatsu has joined #openstack-swift10:49
*** ccamacho has joined #openstack-swift10:54
*** ccamacho has quit IRC10:54
*** takamatsu has quit IRC10:54
*** takamatsu has joined #openstack-swift10:58
kota_alecuyer: sorry I couldn't make my time about LOSF today but you seems to find the newer grpcio would work with eventlet.11:13
kota_if the newer (and lower from 1.15.0) is fine, we may change the patch to set just lower requirement of grpcio in requirements.txt rather than mangle the constarints settings.11:14
kota_i.e. `grpcio >= <enough version to able to run>` is easier.11:15
*** gmann has joined #openstack-swift11:49
alecuyerkota_: no worries. It does seem to work with 1.15.0 and i'm running git bisect to find where it started working, but that takes some time12:09
*** guimaluf has joined #openstack-swift12:26
guimalufhi folks, I've a production cluster running swift and keystone at kilo version. My ring is splited in two regions with 4 nodes each. One of our regions is completely off and we found that one file is lost cause all the three copies are in the same region, the one is out... why is this happen? swift should not put a file within the same region, I always trust in that premise... my storage nodes are far away from been full... I12:39
guimalufwoudl appreciate if someone could shed light in the subject12:39
*** ccamacho has joined #openstack-swift12:39
guimalufchecking dispersion report I realize that 189 objects are lost12:51
alecuyerguimaluf: I am not familiar with multi region, but are you using write affinity ? (I understand this will make copies to other regions asynchronously)12:57
DHEunfortunately it's not a guarantee, just a best effort. maybe the ring needed periodic rebalancing, or the regions were not well balanced in terms of [sum of] device weights13:11
DHEthere is a way to make a guarantee in the current version, but it's an app marked as "experimental" and you'll always have exactly the same number of copies of an object in each region13:11
guimalufthank you DHE and alecuyer. now, how can I investigate which objects or partitions were affected?13:23
guimalufcan I namely the partitions that are within the same region?13:23
guimalufI imagine that is impossible to figure out which or how many files were there, right?13:24
DHEhow many zones per region?13:25
guimalufr1z{1..4} and r2z113:26
guimalufzones were servers, that's why when I add r2 I set up only z113:26
DHEand r1 is down?13:26
guimalufr2 was deleted13:27
guimalufnot down :/ removed... blown up13:27
DHEoh... that's not expected...13:27
guimalufyes... :/ that's why I'm a little bit in dispair13:28
guimalufI've never realize that 0.58 dispersion could do this kind of thing...13:29
DHEwell, the report command you want is probably: swift-ring-builder [filename.builder] dispersion --verbose13:29
guimalufnow I need to know how many objects were affected..13:29
guimalufI did this already13:29
guimalufI know that 189 partitions were at r213:29
guimalufswift-ring-builder /etc/swift/object.builder list_parts r2 | grep '3$'13:32
*** psachin has quit IRC13:35
guimalufhow can I find more info about partition? which server/device/zone its store?13:35
*** ccamacho has quit IRC13:50
*** ccamacho has joined #openstack-swift13:51
*** ccamacho has quit IRC15:01
*** e0ne has quit IRC16:11
*** e0ne has joined #openstack-swift16:20
*** hseipp has quit IRC16:21
notmynamegood morning16:28
*** e0ne has quit IRC16:31
*** gyee has joined #openstack-swift16:40
*** itlinux has joined #openstack-swift16:47
*** mikecmpbll has quit IRC17:04
alecuyerkota_: It seems that the introduction of a background python poller thread broke grpc client with gevent (8f7bc54ff2256179ede6d4a6d1a8792b7820fc28). Per the comments, this was a "temporary workaround". This was removed in a75712a71f01c06d7e5868a75cb62b26d775e5df , as the thread is now in C core. With that patch, it does not hang anymore.   Im not done yet but I wanted to tell you what I found for now.  Ill put more info on the g17:08
alecuyeroogle drive doc17:08
*** hseipp has joined #openstack-swift17:09
timburkemattoliverau, huh. well there goes my theory -- landed in 3.5.3 ...17:41
*** takamatsu_ has joined #openstack-swift17:48
*** takamatsu has quit IRC17:48
*** mikecmpbll has joined #openstack-swift17:59
*** takamatsu_ has quit IRC18:03
*** takamatsu_ has joined #openstack-swift18:06
*** e0ne has joined #openstack-swift19:18
*** hseipp has quit IRC19:24
*** itlinux has quit IRC19:41
* zaitcev is cheering for versioned writes19:50
zaitcevI'm asking myself if we want to make probe tests more tolerant in some ways.19:50
zaitcevIf I become the PTL, I'll need to sit down with the infra guys and study how all that machinery works, what are problems.19:51
zaitcevMaybe it's not just the sponsor money that contributes to this.19:51
zaitcever, the lack of19:51
*** e0ne has quit IRC20:08
*** itlinux has joined #openstack-swift20:24
*** itlinux has quit IRC20:51
*** itlinux has joined #openstack-swift21:01
*** itlinux has quit IRC21:02
*** itlinux has joined #openstack-swift21:13
*** itlinux has quit IRC21:20
*** itlinux has joined #openstack-swift21:39
openstackgerritTim Burke proposed openstack/swift master: Make test_multiphase_put_drains_extra_commit_junk_disconnect less flakey
openstackgerritMerged openstack/swift master: py3: port the container
openstackgerritMerged openstack/swift master: py3: port versioned_writes
*** thurloat has joined #openstack-swift23:25
*** itlinux has quit IRC23:35
*** itlinux has joined #openstack-swift23:57

Generated by 2.15.3 by Marius Gedminas - find it at!