| *** vinsh has joined #openstack-swift | 00:05 | |
| *** vinsh has quit IRC | 00:05 | |
| *** briancli1e has quit IRC | 00:12 | |
| *** briancline has joined #openstack-swift | 00:13 | |
| *** vinsh has joined #openstack-swift | 00:32 | |
| *** vinsh has quit IRC | 00:43 | |
| *** zhurong has joined #openstack-swift | 00:45 | |
| *** vinsh has joined #openstack-swift | 00:50 | |
| *** vinsh has quit IRC | 00:52 | |
| *** zhengyin has joined #openstack-swift | 00:55 | |
| *** vinsh has joined #openstack-swift | 00:55 | |
| *** vinsh has quit IRC | 00:56 | |
| *** tovin07_ has joined #openstack-swift | 00:59 | |
| *** lucasxu has joined #openstack-swift | 01:00 | |
| *** lucasxu has quit IRC | 01:01 | |
| *** vinsh has joined #openstack-swift | 01:09 | |
| *** ianychoi has joined #openstack-swift | 01:14 | |
| *** vinsh has quit IRC | 01:15 | |
| *** zhurong has quit IRC | 01:33 | |
| mattoliverau | tdasilva: I know your sleeping, but while playing with patch 289664 on a centos based SAIO (ansible-saio) I have problems with servers_per_port which i don't have on Ubuntu. Do you have problems with server_per_port on centos/redhat, or am I doing something stupid (which I probably am)? | 01:34 |
|---|---|---|
| patchbot | https://review.openstack.org/#/c/289664/ - swift - Make eventlet.tpool's thread count configurable in... | 01:34 |
| mattoliverau | Maybe something isn't working on my centos vbox'ed saio. | 01:35 |
| *** zhurong has joined #openstack-swift | 01:35 | |
| clarkb | mattoliverau: are multipl servers listening on the aame port using SO_REUSEPORT? | 01:37 |
| clarkb | thats in newer kernels and may not be available on centos | 01:38 |
| mattoliverau | clarkb: ahh that could be it :) | 01:38 |
| mattoliverau | I shall go look at the kernel version :) | 01:38 |
| mattoliverau | its ok for now, as I have an Ubuntu SAIO I can use to actually test some wierdness I'm finding in the patch. Just wanted to check with centos support. | 01:40 |
| mattoliverau | tho that does make sense to what I'm seeing | 01:40 |
| *** mingyu has joined #openstack-swift | 01:46 | |
| kota_ | morning | 02:06 |
| kota_ | clayg: I'll be on https://review.openstack.org/#/c/473673/ today | 02:06 |
| patchbot | patch 473673 - swift - More Global EC doc updates (MERGED) | 02:06 |
| kota_ | oh | 02:06 |
| kota_ | it has been merged | 02:07 |
| kota_ | thanks patchbot | 02:07 |
| clayg | lol, yeah wtg patchbot! | 02:07 |
| clayg | kota_: you should look for your own patches that might have slipped through the crack that you have pending/open related to EC/reconstructor and see if there's anything you need to dust off and get folks to review | 02:08 |
| clayg | kota_: unless acoles and rledisez come online and tell us all the important reconstructor fixes are already to merge :P | 02:08 |
| kota_ | clayg: i see, thanks for poking me | 02:09 |
| *** SkyRocknRoll has quit IRC | 02:21 | |
| *** mingyu has quit IRC | 02:39 | |
| *** zhengyin has quit IRC | 02:50 | |
| *** zhengyin has joined #openstack-swift | 02:50 | |
| openstackgerrit | Merged openstack/swift master: Ring doc cleanups https://review.openstack.org/465184 | 02:54 |
| openstackgerrit | Merged openstack/swift master: Follow up for affinity config per policy https://review.openstack.org/467920 | 02:54 |
| *** links has joined #openstack-swift | 03:14 | |
| *** mingyu has joined #openstack-swift | 03:18 | |
| *** vinsh has joined #openstack-swift | 03:34 | |
| *** vinsh has quit IRC | 03:34 | |
| *** itlinux_ has joined #openstack-swift | 03:42 | |
| *** abhitechie has joined #openstack-swift | 03:47 | |
| *** klrmn has quit IRC | 03:55 | |
| openstackgerrit | Lingxian Kong proposed openstack/swift master: Write-affinity aware object deletion https://review.openstack.org/470158 | 04:03 |
| openstackgerrit | Lingxian Kong proposed openstack/swift master: Write-affinity aware object deletion https://review.openstack.org/470158 | 04:04 |
| *** winggundamth has joined #openstack-swift | 04:08 | |
| *** gkadam has joined #openstack-swift | 04:08 | |
| *** hoonetorg has quit IRC | 04:11 | |
| *** psachin has joined #openstack-swift | 04:14 | |
| *** abhinavtechie has joined #openstack-swift | 04:20 | |
| *** hoonetorg has joined #openstack-swift | 04:21 | |
| *** abhitechie has quit IRC | 04:24 | |
| *** zhurong has quit IRC | 04:28 | |
| *** alecuyer has quit IRC | 04:28 | |
| *** alecuyer has joined #openstack-swift | 04:28 | |
| notmyname | thanks mattoliverau for the reviews | 04:29 |
| *** EmilienM has quit IRC | 04:30 | |
| *** EmilienM has joined #openstack-swift | 04:35 | |
| *** zhurong has joined #openstack-swift | 04:39 | |
| *** mingyu has quit IRC | 04:46 | |
| *** qwertyco has joined #openstack-swift | 04:58 | |
| *** abhinavtechie has quit IRC | 04:59 | |
| *** abhinavtechie has joined #openstack-swift | 04:59 | |
| *** bkopilov has quit IRC | 04:59 | |
| *** ChubYann has quit IRC | 05:01 | |
| *** zhengyin has quit IRC | 05:06 | |
| *** zhengyin has joined #openstack-swift | 05:07 | |
| *** itlinux_ has quit IRC | 05:12 | |
| *** zhurong has quit IRC | 05:17 | |
| *** zhurong has joined #openstack-swift | 05:21 | |
| *** ghs39ghsggf has joined #openstack-swift | 05:25 | |
| *** ghs39ghsggf has left #openstack-swift | 05:25 | |
| *** skudlik has joined #openstack-swift | 05:32 | |
| mattoliverau | notmyname: nps :) | 05:35 |
| *** mingyu has joined #openstack-swift | 05:48 | |
| mahatic | notmyname: mattoliverau o/ | 05:48 |
| mattoliverau | mahatic: morning | 05:48 |
| mahatic | notmyname: are you gonna be up until our meeting time? :) | 05:48 |
| *** abhinavtechie has quit IRC | 05:50 | |
| *** abhitechie has joined #openstack-swift | 05:50 | |
| *** aselius has quit IRC | 05:54 | |
| *** abhitechie has quit IRC | 05:54 | |
| mattoliverau | notmyname: yeah, you still have another hour :) | 05:59 |
| *** zhengyin has quit IRC | 05:59 | |
| *** zhengyin has joined #openstack-swift | 05:59 | |
| *** jeffli has joined #openstack-swift | 06:07 | |
| *** rcernin has joined #openstack-swift | 06:08 | |
| *** bkopilov has joined #openstack-swift | 06:10 | |
| *** silor has joined #openstack-swift | 06:13 | |
| *** winggundamth has quit IRC | 06:18 | |
| *** abhitechie has joined #openstack-swift | 06:21 | |
| *** cshastri has joined #openstack-swift | 06:42 | |
| *** hseipp has joined #openstack-swift | 06:44 | |
| *** m_kazuhiro has joined #openstack-swift | 06:46 | |
| m_kazuhiro | good morning | 06:46 |
| *** jaosorior has joined #openstack-swift | 06:49 | |
| acoles | meeting in #openstack-meeting in 2 minutes | 06:58 |
| mattoliverau | \o/ | 06:59 |
| mattoliverau | m_kazuhiro: o/ | 06:59 |
| *** silor has quit IRC | 07:11 | |
| *** silor has joined #openstack-swift | 07:11 | |
| *** d0ugal has joined #openstack-swift | 07:11 | |
| *** tesseract has joined #openstack-swift | 07:16 | |
| m_kazuhiro | mattoliverau: o/ | 07:17 |
| *** gyee has quit IRC | 07:22 | |
| *** mingyu has quit IRC | 07:30 | |
| *** mingyu has joined #openstack-swift | 07:30 | |
| *** mingyu has quit IRC | 07:39 | |
| *** mingyu has joined #openstack-swift | 07:41 | |
| *** mingyu has quit IRC | 07:55 | |
| *** m_kazuhiro has quit IRC | 08:00 | |
| acoles | and now, to use mattoliverau's usual words, I am off to get some breakfast :) | 08:00 |
| mattoliverau | :) | 08:01 |
| mattoliverau | k, I gotta go bath the troublesome toddler o/ | 08:01 |
| mahatic | acoles: :D enjoy your breakfast! | 08:01 |
| acoles | mattoliverau: have fun | 08:01 |
| mahatic | mattoliverau: enjoy too | 08:01 |
| mahatic | "I'll go get some lunch" :) | 08:02 |
| *** hseipp has quit IRC | 08:02 | |
| acoles | mahatic: enjoy | 08:03 |
| * kota_ and dinner for me | 08:03 | |
| *** oshritf_ has joined #openstack-swift | 08:08 | |
| *** cbartz has joined #openstack-swift | 08:14 | |
| *** tovin07_ has quit IRC | 08:14 | |
| *** geaaru has joined #openstack-swift | 08:17 | |
| *** jeffli has quit IRC | 08:18 | |
| *** tovin07_ has joined #openstack-swift | 08:24 | |
| *** winggundamth has joined #openstack-swift | 08:28 | |
| *** ianychoi has quit IRC | 08:43 | |
| *** hseipp has joined #openstack-swift | 08:43 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:56 | |
| *** kei_yama has quit IRC | 09:01 | |
| *** sanchitmalhotra has quit IRC | 09:07 | |
| *** sanchitmalhotra has joined #openstack-swift | 09:07 | |
| *** gkadam is now known as gkadam-afk | 09:20 | |
| *** zhurong has quit IRC | 09:40 | |
| *** ianychoi has joined #openstack-swift | 09:49 | |
| *** mathiasb has quit IRC | 09:58 | |
| *** tovin07_ has quit IRC | 10:02 | |
| *** mvk has quit IRC | 10:15 | |
| *** mvk has joined #openstack-swift | 10:26 | |
| openstackgerrit | Hieu LE proposed openstack/swift master: OSprofiler in OpenStack Swift https://review.openstack.org/468316 | 10:26 |
| *** winggundamth has quit IRC | 10:48 | |
| *** hseipp has quit IRC | 11:06 | |
| *** zhengyin has quit IRC | 11:09 | |
| *** qwertyco has quit IRC | 11:25 | |
| *** jaosorior has quit IRC | 11:29 | |
| *** gkadam-afk is now known as gkadam | 11:45 | |
| *** abhitechie has quit IRC | 11:46 | |
| *** abhitechie has joined #openstack-swift | 11:53 | |
| *** zhengyin has joined #openstack-swift | 11:58 | |
| *** qwertyco has joined #openstack-swift | 12:01 | |
| *** zhengyin has quit IRC | 12:17 | |
| *** chlong has quit IRC | 12:20 | |
| *** links has quit IRC | 12:46 | |
| tdasilva | good morning | 12:47 |
| acoles | tdasilva: o/ | 12:48 |
| * tdasilva goes to catch up on this morning's meeting... | 12:48 | |
| tdasilva | acoles: o/ | 12:48 |
| *** catintheroof has joined #openstack-swift | 12:52 | |
| *** mwheckmann has quit IRC | 12:55 | |
| *** cnf has joined #openstack-swift | 12:59 | |
| tdasilva | acoles: good job chairing the meeting... | 13:03 |
| acoles | tdasilva: thanks | 13:03 |
| acoles | tdasilva: your turn next time ;) | 13:03 |
| tdasilva | mattoliverau: any luck with patch 289664? | 13:03 |
| patchbot | https://review.openstack.org/#/c/289664/ - swift - Make eventlet.tpool's thread count configurable in... | 13:03 |
| tdasilva | acoles: lol | 13:04 |
| tdasilva | acoles: really cool to see folks that don't typically join the 2100 meeting joining there | 13:05 |
| acoles | tdasilva: yes | 13:05 |
| *** mingyu has joined #openstack-swift | 13:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 13:09 |
| *** skudlik has quit IRC | 13:09 | |
| rledisez | hey, does anyone already though of an auto-throttle feature in the proxy? when the proxy feels it is getting overflowed, it starts returning an error like 429 Too Many Requests. I don't know how to do it, but that would be awesome… | 13:12 |
| *** lucasxu has joined #openstack-swift | 13:15 | |
| *** klamath has joined #openstack-swift | 13:16 | |
| *** klamath has quit IRC | 13:16 | |
| *** gkadam has quit IRC | 13:17 | |
| *** klamath has joined #openstack-swift | 13:17 | |
| *** openstackgerrit has quit IRC | 13:18 | |
| clayg | Overloaded in what way? | 13:20 |
| rledisez | clayg: getting a huge amount of requests (bulk-delete, but i'm not sure it's relevant), starting to get terrible ttfb, timeout connecting to memcache, and increasing number of errors | 13:22 |
| *** dmsimard has joined #openstack-swift | 13:27 | |
| *** zhurong has joined #openstack-swift | 13:42 | |
| *** ^andrea^ has quit IRC | 13:57 | |
| *** chlong has joined #openstack-swift | 14:01 | |
| *** chlong has quit IRC | 14:06 | |
| *** ^andrea^ has joined #openstack-swift | 14:10 | |
| *** dmsimard has quit IRC | 14:10 | |
| *** qwertyco has quit IRC | 14:13 | |
| *** qwertyco has joined #openstack-swift | 14:13 | |
| *** links has joined #openstack-swift | 14:19 | |
| *** qwertyco has quit IRC | 14:22 | |
| *** qwertyco has joined #openstack-swift | 14:22 | |
| *** jaosorior has joined #openstack-swift | 14:28 | |
| *** chlong has joined #openstack-swift | 14:34 | |
| *** chlong has quit IRC | 14:35 | |
| *** links has quit IRC | 14:36 | |
| *** chlong has joined #openstack-swift | 14:38 | |
| *** cshastri has quit IRC | 14:38 | |
| *** skudlik has joined #openstack-swift | 14:44 | |
| *** psachin has quit IRC | 14:45 | |
| *** zhurong has quit IRC | 14:49 | |
| *** mingyu has quit IRC | 14:52 | |
| *** vinsh has joined #openstack-swift | 14:53 | |
| *** aselius has joined #openstack-swift | 14:56 | |
| *** mingyu has joined #openstack-swift | 14:57 | |
| *** abhitechie has quit IRC | 14:57 | |
| *** ujjain has joined #openstack-swift | 15:02 | |
| *** ujjain has joined #openstack-swift | 15:02 | |
| *** qwertyco has quit IRC | 15:03 | |
| *** qwertyco has joined #openstack-swift | 15:04 | |
| *** mingyu has quit IRC | 15:07 | |
| *** oshritf_ has quit IRC | 15:07 | |
| *** hseipp has joined #openstack-swift | 15:08 | |
| *** openstackgerrit has joined #openstack-swift | 15:09 | |
| openstackgerrit | Janie Richling proposed openstack/swift master: Validate the object key when decrypting https://review.openstack.org/473149 | 15:09 |
| *** qwertyco has quit IRC | 15:12 | |
| *** mingyu has joined #openstack-swift | 15:13 | |
| *** vinsh has quit IRC | 15:16 | |
| *** mingyu has quit IRC | 15:18 | |
| *** mingyu has joined #openstack-swift | 15:22 | |
| *** mingyu has quit IRC | 15:25 | |
| *** klrmn has joined #openstack-swift | 15:25 | |
| *** gyee has joined #openstack-swift | 15:32 | |
| *** chsc has joined #openstack-swift | 15:38 | |
| *** chsc has joined #openstack-swift | 15:38 | |
| clayg | Bulk requests definitely have a dos risk because of amplification | 15:44 |
| clayg | I think bulk delete in particular offers some concurrency - which could make matters worse | 15:45 |
| *** mvk has quit IRC | 15:45 | |
| clayg | Ratelimiting mw (which is per a/c) should be the typical goto answer but it's probably not affecting sub requests | 15:47 |
| clayg | I think unless the 429 response is tied to what appears to be a bad actor it has an equally likely chance to make things worse... and client support could take awhile | 15:49 |
| rledisez | yeah, i'm thinking maybe of just playing with max_clients as a first step. i'll just need a way to detect when I hit the limit | 15:50 |
| clayg | ... which might be a reason to get basic support in now? What does RL respond now? Does swiftclient like it? | 15:50 |
| clayg | Is it just one proxy that sucks? Like when it's eating a request with a lot of amplification? | 15:51 |
| *** oshritf_ has joined #openstack-swift | 15:51 | |
| rledisez | there were so many request, the whole cluster when down :/ | 15:51 |
| *** mathiasb has joined #openstack-swift | 15:52 | |
| rledisez | i just put the cluster up as fast as i could. i'll take some time to re-play that on a dev cluster | 15:52 |
| clayg | Ok. So under extreme duress - how to degrade more gracefully. Gotcha. | 15:54 |
| rledisez | yeah, exactly :) | 15:55 |
| clayg | The eventlet wsgi server has some concept tunable about backlog.. or yeah max_clients. I wonder if we could dig it out and put something on statsd... | 15:56 |
| openstackgerrit | Janie Richling proposed openstack/swift master: Validate the object key when decrypting https://review.openstack.org/473149 | 15:57 |
| *** mingyu has joined #openstack-swift | 15:57 | |
| *** chsc has quit IRC | 15:57 | |
| *** itlinux_ has joined #openstack-swift | 16:00 | |
| *** klrmn has quit IRC | 16:01 | |
| notmyname | good morning | 16:11 |
| *** Sukhdev_ has joined #openstack-swift | 16:12 | |
| *** vinsh has joined #openstack-swift | 16:21 | |
| *** vinsh has quit IRC | 16:28 | |
| timburke | good morning | 16:29 |
| *** vinsh has joined #openstack-swift | 16:31 | |
| *** oshritf_ has quit IRC | 16:34 | |
| *** ChubYann has joined #openstack-swift | 16:38 | |
| *** lucasxu has quit IRC | 16:43 | |
| *** abhitechie has joined #openstack-swift | 16:48 | |
| *** rcernin has quit IRC | 16:49 | |
| notmyname | acoles: sorry about patchbot not being in the meeting. -infra hates it aparantly, and they have the ability to kick/ban it from channels. so when patchbot is in channel and says something in an -infra meeting, it gets kickbanned | 16:50 |
| acoles | notmyname: it wasn't a problem | 16:51 |
| notmyname | also, for everyone, AFAIK, the "UpgradeImpact" word in a commit message does precisely nothing. it's *only* for a human to notice if they happen to read the full commit message | 16:53 |
| notmyname | and yes, that's what acoles said in the meeting :-) | 16:54 |
| timburke | handy for doing things like `git log --grep UpgradeImpact 2.10.0..2.14.0` | 16:55 |
| notmyname | sure. *I've* never run that command before though ;-) | 16:55 |
| *** tesseract has quit IRC | 16:56 | |
| *** silor has quit IRC | 16:56 | |
| *** hseipp has quit IRC | 16:57 | |
| notmyname | timburke: but I totally should :-) | 16:58 |
| *** mvk has joined #openstack-swift | 16:59 | |
| timburke | of course, it's only as good as our ability to remember to include it | 16:59 |
| acoles | timburke: but you are very good at remembering :) | 17:00 |
| timburke | i try to be :-) | 17:00 |
| acoles | so it's only as good as our ability to get timburke to review *everything* | 17:00 |
| *** catintheroof has quit IRC | 17:02 | |
| *** klrmn has joined #openstack-swift | 17:02 | |
| timburke | like i said, i try to :P | 17:02 |
| *** catintheroof has joined #openstack-swift | 17:03 | |
| *** klrmn has quit IRC | 17:03 | |
| *** klrmn has joined #openstack-swift | 17:06 | |
| clayg | I'm getting* really* close to spiking on "building a replica2part2dev table from {d: os.listdir(path_for_disk(d)) for d in all_devices_on_all_nodes}" - it's a significantly more io intensive problem on EC - but even still it might be worth it to read all the hashes.pkl and regroup - compared with marching forward and then having to change placement again. | 17:08 |
| *** chsc has joined #openstack-swift | 17:14 | |
| *** oshritf_ has joined #openstack-swift | 17:16 | |
| *** pcaruana has quit IRC | 17:29 | |
| *** oshritf_ has quit IRC | 17:30 | |
| *** lucasxu has joined #openstack-swift | 17:39 | |
| *** itlinux_ has quit IRC | 17:41 | |
| *** Sukhdev_ has quit IRC | 17:47 | |
| *** itlinux_ has joined #openstack-swift | 17:57 | |
| *** geaaru has quit IRC | 18:03 | |
| *** silor has joined #openstack-swift | 18:11 | |
| *** itlinux_ has quit IRC | 18:14 | |
| *** qwertyco has joined #openstack-swift | 18:16 | |
| *** itlinux has joined #openstack-swift | 18:16 | |
| *** SkyRocknRoll has quit IRC | 18:32 | |
| *** hseipp has joined #openstack-swift | 18:36 | |
| *** ujjain has quit IRC | 18:38 | |
| *** Sukhdev_ has joined #openstack-swift | 18:46 | |
| *** qwertyco has quit IRC | 18:47 | |
| *** mingyu has quit IRC | 18:49 | |
| *** silor has quit IRC | 19:03 | |
| *** rcernin has joined #openstack-swift | 19:10 | |
| openstackgerrit | Merged openstack/swift master: Make mount_check option usable in containerized environments https://review.openstack.org/466255 | 19:49 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 20:00 |
| tdasilva | mattoliverau: I'm also seeing those issues you mentioned on my ansible-saio env with centos running...not sure yet what's going on yet. My understanding is that SO_REUSEPORT is supported since Centos 6.5 should I'm not sure that's the reason... | 20:03 |
| tdasilva | anybody else runs server_per_port on centos and has run into problems before? | 20:03 |
| *** abhitechie has quit IRC | 20:07 | |
| *** itlinux has quit IRC | 20:13 | |
| mattoliverau | tdasilva: I'll try on my Fedora desktop, that at least has a newer kernel. But still a Redhat based system, so might tell us if it is a kernel issue or Redhat system problem. | 20:16 |
| mattoliverau | I was setting it on a ansible-saio as well so I guess could be something there. But it's really just saio | 20:17 |
| notmyname | what was the issue? | 20:17 |
| mattoliverau | s/setting/seeing | 20:17 |
| mattoliverau | notmyname: starting services, get connection refused and seemed to start just a bunch of 1 server. But this is from memory and too early :p is need to go find my laptop | 20:18 |
| mattoliverau | Apparently my English also doesnt work 'p | 20:19 |
| *** skudlik has quit IRC | 20:22 | |
| *** tonanhngo has joined #openstack-swift | 20:23 | |
| *** tonanhngo has quit IRC | 20:34 | |
| *** hseipp has quit IRC | 20:40 | |
| *** itlinux has joined #openstack-swift | 20:45 | |
| *** itlinux has quit IRC | 20:46 | |
| notmyname | the 2100 swift team meeting is in about 10 minutes in #openstack-meeting | 20:47 |
| *** joeljwright has joined #openstack-swift | 20:51 | |
| *** ChanServ sets mode: +v joeljwright | 20:51 | |
| notmyname | hello joeljwright | 20:51 |
| joeljwright | evening | 20:52 |
| joeljwright | (or afternoon I guess) | 20:52 |
| notmyname | greetings appropriate in your own timezone are entirely ok :-) | 20:53 |
| *** m_kazuhiro has joined #openstack-swift | 20:53 | |
| joeljwright | :) | 20:53 |
| tdasilva | mattoliverau: notmyname: this is what I see http://paste.openstack.org/show/612595/ | 20:55 |
| tdasilva | I changed the settings to use server_per_port = 1 | 20:56 |
| notmyname | and it's only on centos? | 20:56 |
| tdasilva | mattoliverau: I haven't tried to run ansible-saio on Fedora for a while, maybe I should try that...I think zaitcev goes through a lot of effort to make sure swift runs on Fedora | 20:57 |
| kota_ | good morning | 20:57 |
| tdasilva | notmyname: correct | 20:57 |
| notmyname | hello kota_ | 20:57 |
| kota_ | notmyname: o/ | 20:57 |
| mattoliverau | kota_: o/ | 20:57 |
| kota_ | mattoliverau: o/ | 20:58 |
| mattoliverau | tdasilva: yup same. Then I looked at ps to see what servers where running. | 20:58 |
| * notmyname goes to #openstack-meeting | 21:00 | |
| mattoliverau | tdasilva: I'll try and find time to do some debugging. However cannot bind does sound very SO_REUSEPORT.. but yeah if the kernel is new enough then it could definitely be something else. | 21:02 |
| *** itlinux_ has joined #openstack-swift | 21:04 | |
| *** lucasxu has quit IRC | 21:06 | |
| *** mingyu has joined #openstack-swift | 21:31 | |
| *** hoonetorg has quit IRC | 21:34 | |
| clayg | tdasilva: along with that gist can I see that ring? | 21:35 |
| clayg | tdasilva: swift-init all stop; pkill -f swift is a thing sometimes if you loose some pids fwiw? | 21:35 |
| clayg | in prod sometimes I have to use a swift-orphans like | 21:35 |
| notmyname | acoles: who's the next 0700 chair? | 21:36 |
| acoles | notmyname: cschwede was next on list I believe | 21:37 |
| notmyname | kk | 21:37 |
| notmyname | cschwede: then in about a week, I'll start trying to hunt you down to talk about the agenda (assuming you're ok with being chair) | 21:38 |
| acoles | he's off changing his name to zak ;) | 21:38 |
| clayg | I'm going to look at patch 455470 | 21:41 |
| patchbot | https://review.openstack.org/#/c/455470/ - python-swiftclient - Stop sending X-Static-Large-Object headers | 21:41 |
| tdasilva | clayg: http://paste.openstack.org/show/612598/ | 21:42 |
| clayg | tdasilva: so.... in each of those 1.conf, 2.conf, 3.conf, 4.conf - I think they *all* think those devices in the ring with "127.0.0.1" are "for me!" | 21:44 |
| clayg | tdasilva: on vsaio when you turn on servers_per_port it configures the rings with 127.0.0.1, 127.0.0.2, 127.0.0.3, etc. | 21:44 |
| tdasilva | clayg: ahhh! cool, i should go checkout vsaio | 21:45 |
| clayg | well, i mean it's not centos | 21:45 |
| tdasilva | clayg: no, i get it | 21:45 |
| tdasilva | i meant checkout vsaio to see exactly how you are setting up vsaio | 21:46 |
| tdasilva | for servers_per_port | 21:46 |
| mattoliverau | clayg: I thought it looked at the server and port to identify different servers per port. Though I chould be wrong. | 21:48 |
| clayg | tdasilva: I think this is the skinny -> https://gist.github.com/clayg/8d4196c332f802c9856898b0eeee1678 | 21:48 |
| mattoliverau | I initially tried just setting servers_per_port manually on a vsaio and seemed to work. | 21:48 |
| mattoliverau | without changing the ips | 21:49 |
| mattoliverau | unless you have already changed ips | 21:49 |
| clayg | mattoliverau: yeah... no | 21:49 |
| clayg | mattoliverau: so... if you use servers_per_port the wsgi strategy BindPortCache guy basically does is_local_device with port=None | 21:49 |
| mattoliverau | oh ok | 21:50 |
| *** joeljwright has quit IRC | 21:50 | |
| mattoliverau | cool, so maybe that's all it is.. that'll be cool | 21:50 |
| clayg | because typically you're setting each device to it's own port - because you want to have X wsgi server workers for each device | 21:50 |
| tdasilva | clayg: can you share your object-2 ring? | 21:50 |
| tdasilva | i'm assuming that's EC? what do you with the ports there? | 21:50 |
| clayg | on your newish kernal you're mis-configured - each object-server is spinning up a process for each device on the four ports listed - but because eventlet started to SO_REUSE_PORT by default it's all crazy | 21:51 |
| *** m_kazuhiro has quit IRC | 21:51 | |
| clayg | I mean.. if it works at all in dev... it's crazy... i mean saio's are crazy to begin with - you can't have real rings with 127.0.0.1 in them - proxy isn't not going to find any object servers on local host ;) | 21:52 |
| tdasilva | which reminds me, mattoliverau we should continue with swift-ansible ;) | 21:53 |
| mattoliverau | tdasilva: yup ;) | 21:53 |
| clayg | no it's all about the runway now -> https://github.com/swiftstack/runway | 21:53 |
| mattoliverau | anyway, thanks clayg, I'll look a little closer at servers_per_port. I'm off to find some breakfast now :) | 21:53 |
| clayg | why is that repo 62% shell - notmyname what are you doing!? | 21:53 |
| tdasilva | "A project to help you take flight with a storage system" | 21:54 |
| clayg | and there's like 10 top level python scripts | 21:54 |
| acoles | he's winging it | 21:56 |
| clayg | NOICE | 21:56 |
| *** mingyu has quit IRC | 21:57 | |
| *** klamath has quit IRC | 21:57 | |
| tdasilva | clayg: thanks for the tips | 21:58 |
| timburke | it's really like 6.2% shell 3.7% python, but github doesn't know how to count YAML | 21:58 |
| tdasilva | gotta run | 21:58 |
| timburke | (without introducing some arbitrary-code-execution bug) | 21:58 |
| clayg | timburke: I think for lp bug #1680083 I should also file a swiftclient bug that will cause --use-slo to .... error? ... if the /info check doesn't have SLO? Is that a thing? | 22:12 |
| openstack | Launchpad bug 1680083 in OpenStack Object Storage (swift) "SLO missing in pipeline because pipeline is confusing" [Medium,Confirmed] https://launchpad.net/bugs/1680083 | 22:13 |
| timburke | maybe? or just fall back to dlo? see also p 258800 | 22:14 |
| patchbot | https://review.openstack.org/#/c/258800/ - python-swiftclient - Add validation of -S option for SLO upload | 22:14 |
| timburke | seems like doing a xLO upload *probably* gives us enough context that doing a /info check is worth it? | 22:15 |
| timburke | i *so* want some form of caching! | 22:15 |
| clayg | didn't we discuss a --use-dlo option with the intention of making --use-slo the default when neither is specified? | 22:16 |
| clayg | timburke: a'ight, i'm cool with dat | 22:20 |
| timburke | i'd love to switch to a --use-dlo option. haven't gotten around to it. having some try-slo-but-fall-back-to-dlo logic will surely ease the transition | 22:21 |
| *** ^andrea^ has quit IRC | 22:32 | |
| *** chlong has quit IRC | 22:37 | |
| *** itlinux_ has quit IRC | 22:59 | |
| *** mvk has quit IRC | 23:07 | |
| *** mvk has joined #openstack-swift | 23:07 | |
| *** catintheroof has quit IRC | 23:09 | |
| *** Sukhdev_ has quit IRC | 23:10 | |
| openstackgerrit | Lingxian Kong proposed openstack/swift master: Write-affinity aware object deletion https://review.openstack.org/470158 | 23:21 |
| *** kei_yama has joined #openstack-swift | 23:25 | |
| *** chsc has quit IRC | 23:34 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Stop sending X-Static-Large-Object headers https://review.openstack.org/455470 | 23:35 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!