Friday, 2018-06-01

*** early has quit IRC00:03
openstackgerritMerged openstack/swift master: Fix suffix-byte-range responses for zero-byte EC objects.  https://review.openstack.org/52656500:07
*** early has joined #openstack-swift00:16
openstackgerritMerged openstack/python-swiftclient master: Make swiftclient respect region_name when using sessions  https://review.openstack.org/56897900:18
*** links has joined #openstack-swift00:44
*** d0ugal__ has quit IRC00:58
*** d0ugal__ has joined #openstack-swift00:59
*** m_kazuhiro has joined #openstack-swift01:08
m_kazuhirogood morning01:09
*** lifeless has quit IRC01:15
*** gyankum has joined #openstack-swift01:18
*** itlinux has joined #openstack-swift01:30
*** itlinux has quit IRC01:38
*** itlinux has joined #openstack-swift01:42
*** itlinux has quit IRC01:45
*** itlinux has joined #openstack-swift01:52
*** lifeless has joined #openstack-swift01:58
*** amandap has quit IRC01:59
*** amandap has joined #openstack-swift02:00
*** amandap has quit IRC02:04
kota_good morning02:08
*** lifeless_ has joined #openstack-swift02:14
*** lifeless has quit IRC02:14
*** psachin has joined #openstack-swift02:36
openstackgerritMerged openstack/swift master: display swift services in apache2  https://review.openstack.org/44272402:54
*** itlinux has quit IRC03:06
*** klamath has quit IRC03:10
*** amandap has joined #openstack-swift03:20
*** dcourtoi_ has joined #openstack-swift03:39
*** lifeless_ has quit IRC03:44
*** DHE has quit IRC03:44
*** AndyWojo has quit IRC03:44
*** timburke has quit IRC03:44
*** StevenK has quit IRC03:44
*** kota_ has quit IRC03:44
*** MooingLemur has quit IRC03:44
*** dcourtoi has quit IRC03:44
*** tristanC has quit IRC03:44
*** amandap has quit IRC03:45
*** amandap has joined #openstack-swift03:45
*** m_kazuhiro has quit IRC03:52
*** d0ugal__ has quit IRC03:52
*** m_kazuhiro has joined #openstack-swift04:09
*** d0ugal__ has joined #openstack-swift04:14
*** itlinux has joined #openstack-swift04:20
*** cbartz has joined #openstack-swift04:33
*** d0ugal__ has quit IRC04:49
*** gyee has quit IRC04:51
*** d0ugal__ has joined #openstack-swift04:57
*** klamath has joined #openstack-swift05:10
*** itlinux has quit IRC05:11
openstackgerritKota Tsuyuzaki proposed openstack/swift master: WIP: Add ceph-s3 test non-voting job  https://review.openstack.org/56470005:18
*** cshastri has joined #openstack-swift05:41
*** d0ugal__ has quit IRC06:26
*** amandap has quit IRC06:40
*** pcaruana has joined #openstack-swift06:40
*** hseipp has joined #openstack-swift06:41
*** amandap has joined #openstack-swift06:44
openstackgerritKota Tsuyuzaki proposed openstack/swift master: WIP: Add ceph-s3 test non-voting job  https://review.openstack.org/56470006:44
*** d0ugal__ has joined #openstack-swift06:48
*** lifeless has joined #openstack-swift07:07
*** rcernin has quit IRC07:08
*** m_kazuhiro has quit IRC07:13
*** ccamacho has quit IRC07:16
*** ccamacho has joined #openstack-swift07:16
*** geaaru has joined #openstack-swift07:20
*** mvk has joined #openstack-swift07:24
*** tesseract has joined #openstack-swift07:25
openstackgerritKota Tsuyuzaki proposed openstack/swift master: WIP: Add ceph-s3 test non-voting job  https://review.openstack.org/56470007:29
acolesgood morning07:42
*** cbartz has quit IRC07:45
*** DHE has joined #openstack-swift07:47
*** AndyWojo has joined #openstack-swift07:47
*** timburke has joined #openstack-swift07:47
*** StevenK has joined #openstack-swift07:47
*** kota_ has joined #openstack-swift07:47
*** MooingLemur has joined #openstack-swift07:47
*** tristanC has joined #openstack-swift07:47
*** barjavel.freenode.net sets mode: +vv timburke kota_07:47
openstackgerritAlistair Coles proposed openstack/swift master: Verify diff stat is unchanged when syncing only shard ranges  https://review.openstack.org/57017107:50
*** mikecmpbll has joined #openstack-swift08:03
*** d0ugal__ has quit IRC08:03
*** d0ugal has joined #openstack-swift08:03
*** amandap has quit IRC08:21
*** amandap has joined #openstack-swift08:21
*** mvk has quit IRC08:29
*** lifeless has quit IRC08:37
*** mvk has joined #openstack-swift08:37
*** armaan has joined #openstack-swift08:39
*** lifeless has joined #openstack-swift08:44
openstackgerritMerged openstack/swift master: Fix marker and end_marker descriptions in API ref  https://review.openstack.org/57146008:46
*** lifeless_ has joined #openstack-swift08:52
*** lifeless has quit IRC08:52
openstackgerritMerged openstack/swift master: Verify diff stat is unchanged when syncing only shard ranges  https://review.openstack.org/57017108:56
*** amandap has quit IRC09:06
*** prasen has joined #openstack-swift09:09
*** d0ugal has quit IRC09:33
*** mvk has quit IRC09:33
*** corvus has quit IRC09:33
*** corvus has joined #openstack-swift09:34
*** mvk has joined #openstack-swift09:34
*** mvk has quit IRC09:42
*** mvk has joined #openstack-swift09:42
remix_tjhello, anyone can help me with a stupid thing? I'm very noob. I'm writing a middleware and i need to change an header a user puts called "X-Something" into an header called X-Object-Meta-Somethingelse09:43
remix_tji'm using in my __call__ function req = Request(env). At the end of the __call__ function if something has changed, how to i pass the altered environment to the next middleware in the pipeline?09:45
remix_tji see several self.app(env, start_response) but seems that env gets not altered by me and environment is passed clearly. I've seen in some middleware Request.blank(req.path_info, environ=req.environ) is used. Is it right?09:47
*** amandap has joined #openstack-swift09:52
*** d0ugal has joined #openstack-swift09:52
acolesremix_tj: there's an example of something similar in encrypter.py middleware https://github.com/openstack/swift/blob/3a1a198780433e6ebf2bcc3862bd63c7d033930d/swift/common/middleware/crypto/encrypter.py#L207-L28009:52
remix_tjacoles: thank you09:53
acolesnote how the middleware uses a subclass of WSGIContext to handle requests - that's a good pattern to follow09:53
remix_tjuhm, it's totally different from samples i've found, where middleware was derived directly by object10:00
remix_tji was looking at https://docs.openstack.org/swift/latest/development_middleware.html#creating-your-own-middleware and at https://github.com/redhat-cip/swift-middleware-sample10:01
remix_tjok, i'll try to rewrite in an easier way10:01
*** d0ugal has quit IRC10:16
*** d0ugal has joined #openstack-swift10:24
acolesremix_tj: are you using the wsgify decorator? back to you original question - if you modify req.headers then the modified headers should be passed to next middleware10:35
*** sajjadg has joined #openstack-swift10:37
*** bkopilov_ has quit IRC10:39
*** cbartz has joined #openstack-swift10:48
remix_tjacoles: yes i am11:01
*** d0ugal has quit IRC11:06
*** armaan has quit IRC11:06
*** armaan has joined #openstack-swift11:06
*** armaan has quit IRC11:09
*** armaan has joined #openstack-swift11:09
*** armaan has quit IRC11:15
*** armaan has joined #openstack-swift11:16
*** bkopilov_ has joined #openstack-swift11:16
*** d0ugal has joined #openstack-swift11:23
*** d0ugal has quit IRC11:53
*** d0ugal has joined #openstack-swift11:58
*** armaan has quit IRC12:12
*** armaan has joined #openstack-swift12:13
*** d0ugal has quit IRC12:16
*** hoonetorg has quit IRC12:18
*** amandap has quit IRC12:29
remix_tjacoles: solved, i was confused by mixing several samples with different setups. Now i'm able to get an X-Something and convert to X-Object-Sysmeta-Somethingelse12:32
*** kei_yama has quit IRC12:33
*** d0ugal has joined #openstack-swift12:34
acolesremix_tj: great12:36
remix_tjat the moment i have a stupid middleware allowing to set X-Retention: $seconds header when creating object. Subsequent POST DELETE PUT requests are denied if request time is lower than the value of retention12:37
*** mikecmpb_ has joined #openstack-swift12:46
*** mikecmpbll has quit IRC12:48
acolesremix_tj: at some point you are going to run into the problem of inconsistent object replicas i.e. when you check X-Object-Sysmeta-Something you may get an old replica of the object that doesn't have that metadata.12:54
remix_tjdoes data and metadata get synced in a different time? I can also decide that retention can be set only at creation time, so if the object has been created with retention can be retained, in alternative you cannot13:00
*** mikecmpb_ has quit IRC13:02
*** mikecmpbll has joined #openstack-swift13:02
*** pcaruana has quit IRC13:15
acolesthe data and metadata of a PUT request are written at same time to same file. you'd need to think about cases when sampling the existing object state returns a 404 but there are in fact existing replicas of the object, they are just not currently available.13:23
*** armaan has quit IRC13:28
*** armaan has joined #openstack-swift13:28
*** prasen has quit IRC13:30
*** armaan has quit IRC13:32
acolesI haven't thought about this much, it's probably crazy, but your middleware could run time backwards i.e. assign decreasing x-timestamps to every request. Then swift's consistency protocols would favour the first request to an object url i.e. the first PUT. But...of course, you'd then *never* be able to delete an object :/ ...except by setting an x-delete-at on the first PUT13:34
*** links has quit IRC13:37
*** klamath has quit IRC13:40
*** klamath has joined #openstack-swift13:40
*** armaan has joined #openstack-swift13:40
*** klamath_ has joined #openstack-swift13:41
remix_tjacoles: you say the file has been PUT at time 0, i make a query at time 1 where replica hasn't been completed and when i try to get object info i get 404?13:41
*** klamath has quit IRC13:45
acolesthat might happen under failure conditions when replicas are not available at time 113:46
*** klamath_ has quit IRC13:48
*** klamath has joined #openstack-swift13:48
remix_tjacoles: but if i'm not able to get the object metadata from the backend i deny the operation, to be conservative13:52
acolesthat helps, but the PUT request may be the first and should be allowed, but only if it is the first?13:54
remix_tjthat's right13:59
remix_tjbut how i know if it is?13:59
remix_tjgood question!13:59
acolesyeah, it's a tough problem13:59
acoles:(13:59
acolesso it may be that the consistency mechanisms need to be modified. for example, here's where the 'correct' on disk file for a replica policy is chosen https://github.com/openstack/swift/blob/7a7677868d4ec7693132014779909ac70d4f391d/swift/obj/diskfile.py#L2767-L2793 and you can see the most recent timestamp file is chosen14:01
remix_tjwe can limit the application of this middleware to archival purposes14:01
acolesbut a different implementation might open up each .data file and look for retention metadata and make a different decision14:02
remix_tjand saying that doesn't work if you're making an high number of requests14:02
acoles^^setting appropriate user expectation is another way to deal with tough problems14:02
remix_tjhehe14:03
acoles:)14:03
remix_tjis repliacing something i can't do with "you're using it in the bad way"14:03
*** klamath_ has joined #openstack-swift14:04
acolesa.k.a. "education" ;)14:05
*** klamath has quit IRC14:05
acolesthis will help enable more 'custom' backends https://review.openstack.org/44712914:06
patchbotpatch 447129 - swift - Configure diskfile per storage policy14:06
*** pcaruana has joined #openstack-swift14:10
remix_tjthis what? your commit?14:12
acolesthe change proposed in patch 447129 enables different backend diskfile implementation to be specified for different storage policies, which makes it a little easier to introduce new diskfile implementations14:14
patchbothttps://review.openstack.org/#/c/447129/ - swift - Configure diskfile per storage policy14:14
acolesbut that would still be a lot of work compared to your middleware14:15
remix_tjwhat you propose is a backend change, i, instead want simply to filter out some kind of requests14:16
*** gyankum has quit IRC14:18
acolesremix_tj: yes, I understand. I'm just sharing info. And it may well be that your middleware is sufficient for your use case. It comes down to the probability of failure conditions that would allow a retained object to be erroneously overwritten. It sounds like you are thinking about how to minimise that probability.14:19
*** armaan has quit IRC14:27
*** d0ugal has quit IRC14:31
*** hoonetorg has joined #openstack-swift14:31
*** d0ugal has joined #openstack-swift14:35
*** klamath_ has quit IRC14:35
openstackgerritAlistair Coles proposed openstack/swift master: Always fetch listings from shards when possible  https://review.openstack.org/57153114:35
remix_tjacoles: is there a way to have the time of the current request? i'm relying on datetime.now() but i think is not the right way to do it14:42
*** cbartz has quit IRC14:44
acolesremix_tj: do you mean how to get the request x-timestamp value in your middleware?14:48
*** d0ugal has quit IRC14:49
acolesx-timestamp is not actually set until a PUT request reaches the proxy-server app (here https://github.com/openstack/swift/blob/26418140108281ae5ac11004ebf33b7b3e08c74d/swift/proxy/controllers/base.py#L1525) , so after middleware has handled the request.14:50
acolesx-timestamp is the 'time' of the object used internally to swift14:50
acolesBUT IIRC middleware can set this value and the proxy then respects whatever the middleware set.14:51
acolesand if you do that then the right way is exactly as that link above i.e. use headers.setdefault('x-timestamp', Timestamp.now().internal)14:53
acolesand remember that a middleware to the left of yours may have already set the timestamp so if you are going to use the value then do ts = Timestamp(headers.setdefault('x-timestamp', Timestamp.now().internal))14:54
DHEis there a recommended (or optimal?) number of disks per server? swiftstack sells machines with 60+ but that seems excessively large...14:54
*** d0ugal has joined #openstack-swift14:56
*** gkadam has joined #openstack-swift15:05
*** itlinux has joined #openstack-swift15:16
*** gkadam_ has joined #openstack-swift15:20
notmynamegood morning15:23
*** lifeless_ has quit IRC15:23
*** lifeless has joined #openstack-swift15:24
*** gkadam has quit IRC15:24
openstackgerritAlexandre Lécuyer proposed openstack/swift master: Configure diskfile per storage policy  https://review.openstack.org/44712915:24
openstackgerritAlexandre Lécuyer proposed openstack/swift master: WIP - LOSF  https://review.openstack.org/57176315:24
*** amandap has joined #openstack-swift15:26
*** sajjadg has quit IRC15:34
*** d0ugal has quit IRC15:34
*** sajjadg has joined #openstack-swift15:35
*** sajjadg has quit IRC15:36
*** d0ugal has joined #openstack-swift15:41
*** gkadam__ has joined #openstack-swift15:46
*** d0ugal has quit IRC15:48
*** gkadam_ has quit IRC15:49
*** d0ugal has joined #openstack-swift15:51
*** gkadam_ has joined #openstack-swift15:52
*** gkadam__ has quit IRC15:55
*** SkyRocknRoll has joined #openstack-swift15:58
*** tesseract has quit IRC16:01
*** SkyRocknRoll has quit IRC16:08
*** SkyRocknRoll has joined #openstack-swift16:10
*** lifeless_ has joined #openstack-swift16:11
*** lifeless has quit IRC16:12
*** mikecmpbll has quit IRC16:16
*** armaan has joined #openstack-swift16:17
*** hseipp has quit IRC16:24
*** gyee has joined #openstack-swift16:29
*** gkadam__ has joined #openstack-swift16:33
*** gkadam_ has quit IRC16:36
*** gkadam_ has joined #openstack-swift16:37
*** gkadam__ has quit IRC16:40
*** prasen has joined #openstack-swift16:41
*** prasen has quit IRC16:42
*** armaan has quit IRC16:50
timburkegood morning16:55
*** gkadam__ has joined #openstack-swift16:57
*** gkadam has joined #openstack-swift17:00
*** gkadam_ has quit IRC17:00
openstackgerritJohn Dickinson proposed openstack/swift master: made changelog section breaks more consistent  https://review.openstack.org/54097817:01
*** gkadam__ has quit IRC17:02
*** psachin has quit IRC17:08
*** amandap has quit IRC17:16
*** gkadam has quit IRC17:18
*** cshastri has quit IRC17:20
*** mikecmpbll has joined #openstack-swift17:36
*** mvk has quit IRC17:46
*** SkyRocknRoll has quit IRC18:29
*** geaaru has quit IRC18:38
*** itlinux has quit IRC18:42
*** itlinux has joined #openstack-swift18:46
*** itlinux has quit IRC18:46
*** bkopilov_ has quit IRC18:48
*** openstackgerrit has quit IRC19:04
timburkelooking at https://bugs.launchpad.net/swift/+bug/1774238, i'm up to 18 files changed, 173 insertions(+), 146 deletions(-) and down to just 8 failures!19:06
openstackLaunchpad bug 1774238 in OpenStack Object Storage (swift) "Troubles with unicode account names" [High,Confirmed] - Assigned to Tim Burke (1-tim-z)19:06
timburkeside note: i think "%ff" may be my new favorite substring to include in names...19:06
timburkeof course, at some point i'll have to think more about how my changes handle already-written data... which may get depressing19:07
*** amandap has joined #openstack-swift19:16
*** amandap has quit IRC19:20
*** itlinux has joined #openstack-swift19:30
*** itlinux has quit IRC19:35
*** jlvillal is now known as jlvacation19:43
*** d0ugal has quit IRC19:47
*** lifeless_ has quit IRC20:00
*** lifeless has joined #openstack-swift20:01
*** pcaruana has quit IRC20:03
*** d0ugal has joined #openstack-swift20:04
*** itlinux has joined #openstack-swift20:08
*** lifeless has quit IRC20:50
*** lifeless has joined #openstack-swift20:51
timburkeman, tempauth does *not* handle users like user_te,st_tester very well...20:55
notmynametimburke: definitely sounds like a "closed as 'unfortunate'" bug20:56
timburkenotmyname: yeah, i'm fine with that kind of a resolution there. but i *will* make user_tést%ff_tester work!20:57
*** d0ugal has quit IRC21:08
*** klamath has joined #openstack-swift21:09
*** klamath has quit IRC21:09
*** klamath has joined #openstack-swift21:09
*** d0ugal has joined #openstack-swift21:15
*** itlinux_ has joined #openstack-swift21:41
*** itlinux has quit IRC21:44
*** geaaru has joined #openstack-swift21:57
*** lifeless has quit IRC22:38
*** lifeless has joined #openstack-swift22:40
*** Sukhdev has joined #openstack-swift22:47
*** openstackgerrit has joined #openstack-swift23:13
openstackgerritPete Zaitcev proposed openstack/swift master: Make ShardRanges and its CLI py3-compliant  https://review.openstack.org/57032023:13
*** amandap has joined #openstack-swift23:20
*** amandap has quit IRC23:24
*** lifeless has quit IRC23:36
*** lifeless has joined #openstack-swift23:43
openstackgerritTim Burke proposed openstack/swift master: Improve user logging  https://review.openstack.org/57105123:56
openstackgerritTim Burke proposed openstack/swift master: Make ACLs work with Unicode in user/account names  https://review.openstack.org/57104823:56
openstackgerritTim Burke proposed openstack/swift master: Make COPY work with unicode account names  https://review.openstack.org/57104923:56
openstackgerritTim Burke proposed openstack/swift master: functests: Clean up objects better  https://review.openstack.org/57105023:56
openstackgerritTim Burke proposed openstack/swift master: copy: Stop hanging per-request data on middleware instance  https://review.openstack.org/57190323:56
openstackgerritTim Burke proposed openstack/swift master: tempauth: Send back url-encoded account names  https://review.openstack.org/57190423:56
openstackgerritTim Burke proposed openstack/swift master: Fix quoting for large objects  https://review.openstack.org/57190523:56
openstackgerritTim Burke proposed openstack/swift master: Make symlink work with Unicode account names  https://review.openstack.org/57190623:56
openstackgerritTim Burke proposed openstack/swift master: Make staticweb return URL-encoded Location headers  https://review.openstack.org/57190723:56
openstackgerritTim Burke proposed openstack/swift master: Support Unicode in account and user names during func tests  https://review.openstack.org/57190823:56
timburkethere are about 5 more commits on the way related to that stupid bug...23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!