Thursday, 2019-12-05

*** diablo_rojo has quit IRC00:06
timburkehmmm... that's right, there were p 695780 and p 695781 that i was playing with last week....00:23
patchbot - swift - s3api: Fix blank delimiter handling - 1 patch set00:23
patchbot - swift - s3api: Fix prefix/delimiter/marker quoting - 3 patch sets00:23
tdasilvaclayg, timburke: so my theory for the `DELETE ?version-id=null` when history-location is enabled is because legacy is handling the delete, but during the PUT (while moving data) to archive container, it doesn't strip query parameters from the request, so object versioning is now getting a PUT `?version-id=null` which triggers a 400, legacy then converts it to a 41200:33
tdasilvawondering if we should avoid handling `?versions` and `?version-id` requests all together to containers with history-location to avoid confusion from users.00:35
tdasilvaalthough for situations like slo IIRC we ignore the manifest queries for non-slo objects00:37
donnydSo I am hoping there is some hope for my swift servers. I had a drive fail a while back and it would seem that this particular drive held an object for 95% of what is stored in swift01:20
donnydI tried to run an audit, but after day 5 it was still not complete01:21
openstackgerritTim Burke proposed openstack/swift master: Latch shard-stat reporting
donnydthis is the error that was thrown for all of the missing objects01:21
donnydFailed fo fetch object /AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b6b/periodic/ at all!01:21
donnydThere are 10 drives in each server and there are two servers. I only had a single drive failure01:22
donnydI am not even sure where to being with it. Any pointers would be super appreciated01:24
timburkesure sounds like a ghost-listing -- object's deleted but still shows up in listings01:25
donnydWell the objects have an expiration timer set01:25
timburkefirst thing i'd do is run something like `swift-get-nodes /etc/swift/object.ring.gz AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b6b/periodic/` to figure out where swift *thinks* that object should be01:26
timburkepoke around on disk to see if there are any data files around -- if it's expired but not yet cleaned up, that could certainly do it01:27
donnydServer:Port Device10.0.10.12:6000 sde01:27
donnydServer:Port Device10.0.10.11:6000 sde01:27
donnydServer:Port Device10.0.10.12:6000 sdc [Handoff]01:27
donnydServer:Port Device10.0.10.11:6000 sdf [Handoff]01:27
timburkethere should be some ls commands toward the end that'll give you (give or take) the path to look in01:28
timburkesomething like `ls -lah ${DEVICE:-/srv/node*}/sde/objects/318/776/4fb3c4be96a096b79904e1daf750e776`, though your particular swift_hash_path_prefix/suffix should mean it's different01:29
donnydroot@swift1:/home/ubuntu# ls -lah ${DEVICE:-/srv/node*}/sde/objects/561/f32/8c7a67c6d86b56f9627ad8ae62d5ef3201:29
donnydtotal 16K01:29
donnyddrwxr-xr-x  2 swift swift 4.0K Oct  1 08:18 .01:29
donnyddrwxr-xr-x 35 swift swift 4.0K Nov 12 08:17 ..01:29
donnyd-rwxr-xr-x  1 swift swift  746 Oct  1 08:18 1569917935.31814.data01:29
donnydit looks to me like there is a thing there01:30
timburkecool, we've got a .data! swift-object-info /srv/.../ should tell you some info about it01:31
timburkeincluding the expiration (which i'm guessing is in the past)01:31
donnyddate +%s01:33
timburkeright, so it should've expired more than a month ago :-/01:34
timburkelet's check on the expiring-objects queue01:34
timburkeswift-get-nodes /etc/swift/account.ring.gz .expiring_objects01:34
timburkeshould give you some curl commands01:34
timburkeheh, sorry -- was getting my dev environment into a state that more-closely resembles yours ;-) we'll edit them a bit, get them more like `curl '' | python -m json.tool`01:39
timburke7000 expiry buckets! wow!01:40
timburkei suppose that happens when it's *all* supposed to be expiring ;-)01:40
donnydWe knew the logs in swift thing would be interesting01:40
timburkeso the containers are also done as timestamps -- i'm trying to figure out how far behind we are by looking at just the first few containers01:41
timburkeok, so ~2 months behind...01:43
donnydSo it never actually expired anything01:43
donnydThis whole swift log thing has only been running for like 501:43
timburkeare the object-expirers running?01:44
donnydUmmm.. is that a separate thing.. I am swift dumb, so I do apologize01:45
timburkeyep, gonna need another service running01:46
timburkeis there a swift-object-expirer unit?01:46
donnydsystemctl enable --now swift-object-expirer01:47
donnydFailed to enable unit: Unit file swift-object-expirer.service does not exist.01:47
donnydSo maybe I should read this ^^^^01:48
donnydthere is nothing about object expirations in the object-server.conf01:49
donnydDoes it need to be there or is it on by default?01:49
timburkesorry, this is a bit of a rough edge: historically there's been a /etc/swift/object-expirer.conf but recently we've been trying to move toward having it configured via object-server.conf01:51
donnydok, well I can give the object-server.conf way a go first01:52
timburkejust make sure you've got dequeue_from_legacy_queue enabled01:52
donnydSo I need to set that value to true?01:53
donnydhow will i know if its working?01:55
timburkeyou'll want it on both servers, so processes=2, and set process=0 on one, =1 on the other. probably want to be pretty aggressive on the concurrency value for both01:55
donnydi set that to 1201:56
timburkeif you've got a graph for object-server requests, you'll see a good spike in DELETEs. i know you've got good disk usage graphs; that'll definitely come down01:56
timburkeyou can also watch logs to verify that it's making progress; by default every 5 minutes it should be dropping some stats01:57
timburkeyou might want to kick up the reclaim_age for now until things have caught up01:58
donnydhigher or lower number then?01:58
donnydI assume lower01:59
timburkehigher reclaim age -- that way we'll only remove work items if we actually did the work01:59
timburkekeep an eye on the expiring objects account stats -- the object count there doesn't represent data on disk, but objects to be expired02:00
timburkeit'll come down, but not to zero -- there should be some plateau02:00
donnydIs there a good way to check that02:01
timburkebasically, we want that GET to have containers for today, maybe even the last couple days, but not two months ago02:01
donnydcheck across the whole cluster02:02
tdasilvawould `swift-recon --expirer` help?02:02
donnyd```swift-recon --expirer02:03
donnyd--> Starting reconnaissance on 2 hosts (object)02:03
donnyd[2019-12-05 02:02:40] Checking on expirers02:03
donnyd-> HTTP Error 400: Bad Request02:03
donnyd-> HTTP Error 400: Bad Request02:03
donnyd[object_expiration_pass] - No hosts returned valid data.02:03
donnyd[expired_last_pass] - No hosts returned valid data.02:03
timburkeprobably! honestly, i haven't used swift-recon much :P i need to look more at what tools we provide02:03
timburkeonce you've got the expirers running, it'll probably provide more-useful info02:03
donnydhow can I know for sure they are running?02:04
timburkewatch logs02:04
tdasilvadonnyd: are you running tripleo swift?02:04
donnydis there a seperate process that is spawned?02:04
donnydno tdasilva02:04
tdasilvaor just upstream?02:04
donnydthat one02:04
donnydjust vanilla swift02:04
timburkei'm afraid i've gotta go, though -- kids waiting at home! i'm sure tdasilva can help :-)02:04
donnydthanks for the help timburke02:05
donnydits much appreciated02:05
tdasilvadonnyd: `swift-init all start` is kinda of brute force, but at least makes sure it's running everything02:05
tdasilvahow do you start other processes?02:06
tdasilvabut upstream swift doesn't provide systemd configuration files for swift, does it? what am i missing?02:07
donnydthey were installed via package02:13
donnydand the packages do ship unit files02:14
mattoliverauif you're using packages, are is there an expirer package then?  or does the exirer appear in systemctl?02:17
mattoliverauie, does it appear in  systemctl list-units02:18
mattoliveraujust so we can see what it's called to start it02:18
tdasilvaiirc, in RH packages, object expirer was packaged with container-server package. zaitcev, cschwede would know for sure02:20
mattoliverauoh, of course it is :P02:20
mattoliverauwell that wont work when we introduce the general task queue ;)02:20
tdasilvaactually it's in the proxy02:27
tdasilvadonnyd: so you will probably need to run expirer from your proxy nodes, in case you are not running paco nodes02:27
tdasilvaso i guess `systemctl openstack-swift-object-expirer` start should work?02:28
tdasilvawell, you can also install the proxy package in all your other nodes02:29
donnydSo there needs to be a separate unit and should not run on the swift nodes themselves02:36
tdasilvadonnyd: no no, i was just trying to figure out why you couldn't start object-expirer02:37
tdasilvayou should be able to run expirer in your object-server nodes02:37
donnydok, then maybe i just need to install that package02:37
tdasilvawelll...are you able to run:  `systemctl start openstack-swift-object-expirer` ?02:38
tdasilvaok, just checking, 'cause before you had tried `systemctl start swift-object-expirer`, looks like you were missing openstack in the name02:39
tdasilvathen try install the proxy package02:40
donnydI am missing the openstack name if I am on centos02:40
tdasilvaah, i see02:40
donnydoh yes, having the package installed is super helpful02:41
donnydso now its good02:41
tdasilvadonnyd: can you check what other swift services you have running02:42
tdasilvajust afraid if you ddidn't have expirer running, you might not have other consistency engine services running02:42
donnydtdasilva: I am afraid its likely I am a terrible swift operator02:43
donnydI have been a ceph guy forever and this is my first go with swift02:44
donnydmuch mo betta02:45
donnyd ps aux |grep expire02:45
donnydswift    47575  0.2  0.0 120236 36508 ?        Ss   02:41   0:00 /usr/bin/python2 /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf02:45
tdasilvaand you have rsync and memcache also running, right?02:45
donnydOh yes and now I see a bunch of deletes coming in02:46
tdasilvahopefully `swift-recon --expirer` returns better info this time ??02:46
donnydI am using memcached from my controller. is that ok?02:46
tdasilvayou want to run it on the same nodes as the proxy nodes, possibly you are running proxy nodes from controller nodes too ?02:47
donnydYes, my controller has the proxy02:47
tdasilvamm...not sure why recon is returning that02:48
donnydOh well i ran the command from a swift storage node, not the proxy02:49
donnydthe proxy gives me a totally different error02:50
donnydI will give it a little bit to catch up02:51
donnydAt least we are making some progress :) so thanks a bunch for helping me02:51
tdasilvanp, hope it works out02:54
donnydtdasilva: Me too. I have a bunch of build logs from the community in this swift and would like to get it normalized and back to work02:55
donnydI am going to let this thing work at expiring all the old data and then run the audit again to see whats the what02:56
tdasilvaok, sounds good, definetely recommend playing with swift-recon tool more, and swift-dispersion-report might also be helpful02:58
tdasilvayou want to make sure replicators and updaters are doing their work02:59
donnydi do see them running in the logs and on the machines03:00
donnydwell replicator for sure03:00
*** gyee has quit IRC03:03
*** diablo_rojo has joined #openstack-swift03:06
*** tkajinam is now known as tkajinam|lunch03:40
mattoliverauSo now that we have the null hidden containers merged. I wonder when we should think about moving sharding over to using it. Of course we'll always support the legacy. But seeing as the sharder just looks at containers on disk, there is no harm in updating stats for legecy over to the root container. and otherwise, when using null prefixed containers, the starts is magically done via the updaters, which will be much03:43
mattoliverauI don't mean we should jump on it.. but it would clean parts up a bit :)03:44
mattoliverauanyway that can wait until someone feels inclined or I find more upstream time.03:44
* mattoliverau was reading p 682138 great work!03:45
patchbot - swift - Allow internal clients to use reserved namespace (MERGED) - 38 patch sets03:45
*** diablo_rojo has quit IRC03:46
* mattoliverau should stop reading cool patches, cause it makes me want to get distracted and planning things I want to hack on :P03:46
tdasilvamattoliverau: I think there's a good distinction between .account and hidden containers that we might want to maintain03:47
tdasilvai guess one could start using hidden container in .accounts, but i'm not sure what the benefit of that would be03:47
tdasilvagood distinction in use cases that is03:48
tdasilvasince sharded containers stats get rolled up into the root container, i wonder if you moved them to hidden container, if you would get double counting03:50
mattoliverauoh yeah04:06
mattoliverauthat's true04:07
mattoliverauyour right. we want the stats to migrate to the root container not just the account04:07
mattoliverauso null containers don't work04:07
mattoliverauwithout horrible hacks.. and we already have hacks for the .accounts04:07
mattoliverautdasilva: thanks, I can sigh some relief :)04:08
mattoliverautdasilva: where you hacking from today? find a nice place in Syd?04:09
tdasilvayeah, found a co-working place in Surry Hills, know the area?04:09
mattoliverauoh a little. that's near central isn't it? china town is near there. If head over that way theres some great food! including a great Korean BBQ, just ask notmyname ;)04:12
mattoliverauand csmart if he's online :)04:13
* csmart waves04:13
csmartalso ask tonyb04:13
mattoliverauhaha yeah :)04:15
tonybsay want now?04:15
csmarttonyb: we're talking about Korean BBQ in Sydney...04:15
tonybcsmart: "Mikal's paying!"04:16
tdasilvaYang San Park ?04:16
tonybtdasilva: Correct04:16
csmarttonyb: haha04:17
*** zaitcev has joined #openstack-swift04:17
*** ChanServ sets mode: +v zaitcev04:17
tonybtdasilva: is kinda funky04:20
tdasilvanice, that's not too far04:22
tdasilvatonyb: saved!04:23
tonybtdasilva: I have a few faves for coffee but not super close to where you are04:23
tdasilvatonyb: please send my way, I can move around :)04:24
tonybtdasilva: Will do04:24
tdasilvatonyb: especially if you have anything in the northern beaches area04:24
tonybtdasilva: Sadly not04:28
mattoliverauStevenK: Your a Sydney local. Maybe you have good recommendations for tdasilva :)04:29
StevenKCoffee? But I don't drink it :-P04:31
tdasilvawhat? how come I didn't know about some many aussies in the swift channel?04:34
tdasilvawe should definitely have a hackathon here04:34
tonybtdasilva: If you like coffee then beanhunter is good as a reasearch tool04:34
mattoliverauStevenK: not just coffee, but just general places for tdasilva to visit while he's in town. Ie good food etc? :)04:37
tdasilvayeah, any place to grab a bite during lunch, break from work type of thing04:37
tonybtdasilva: is one of my faves for cofee and for brekfast04:38
StevenKGood food? I recommend Melbourne04:39
tonybStevenK: be nice04:39
mattoliverauyou can tell the Rackspace Australia office is in Circular Quay, or at least was when we were all employed there :P04:39
tonybmattoliverau: true04:39
StevenKtonyb: I'm allowed to scoff at Sydney's food scene :-P04:39
tonybStevenK: You are04:39
StevenKThere's a good Japanese hot pot in the city which is amazing04:40
tonybStevenK: being a git ... I didn't actually say it was good food only that it was a fave for breakfast ;P04:40
tonybStevenK: I like plenty of things are are bad ;p04:40
StevenKI was speaking in general :-P04:40
tonybStevenK: fair04:41
tdasilvammm...seems like everyone i talk to prefers melbourne over sydney04:41
* StevenK reaches for the smashed avo joke04:42
StevenKtdasilva: There's Muglan near Town Hall, which is pretty great04:42
tdasilvaah, that's close to the korean bbq and thai place notmyname mentioned04:43
mattoliverautdasilva: there is a just a city rivalry between the 2.04:44
tdasilvai can tell :) it's funny04:44
mattoliverauoh the thai place was nice.. if it's the one I think he means. Glad he remembers where it was :P04:44
StevenKMy external memory is not co-operating about where this hot pot restaurant is, but it's probably fairly close by to the places you've found04:47
mattoliverautdasilva will be there for a few weeks I suspect, so you have time :)04:48
StevenKAlso, there's a good reason to visit out west, but mattoliverau can't say publically :-P04:48
mattoliverautdasilva: is swiftstack sends you down to canberra to meet with csmart (a customer of yours) let me know so I can come up ;)04:50
mattoliverauif I can get past these fires :P04:50
tdasilvamattoliverau: what town do you live in again?04:50
mattoliverausurf beach04:50
StevenKIt's about four hours in a car from Sydney04:51
mattoliveraumore with the northern roads closed atm :P04:51
StevenKUgh, I keep forgetting there are a bunch of fires around04:51
tdasilvasmoke is crazy in Sydney today04:53
mattoliverauyeah, it was really bad here this morning. seems wind is pushing it out to the sea now (at least for the fire close to me)04:54
mattoliveraubut there seems to just a few near sydney.. just a few :P04:55
mattoliverauyeah, we're kinda on fire. and summer has only just started.05:04
mattoliverautdasilva: at least there's no shoveling snow for you this time of year ;)05:05
tdasilvatrue that, back home they just got their first snow fall of that year and it was a mere 35cm ;)05:09
tdasilvamove a few km to the west and it was close to 5005:10
zaitcevI'm in central Texas, we only get the frozen water rarely.05:23
*** threestrands has joined #openstack-swift05:24
*** pcaruana has joined #openstack-swift05:38
*** pcaruana has quit IRC05:42
mattoliverauzaitcev: sounds like here ;)05:45
*** rcernin has quit IRC06:09
*** ccamacho has quit IRC06:54
*** pcaruana has joined #openstack-swift07:17
*** rcernin has joined #openstack-swift08:02
*** tesseract has joined #openstack-swift08:21
*** rpittau|afk is now known as rpittau08:32
*** mattoliverau has quit IRC08:34
*** mattoliverau has joined #openstack-swift08:38
*** sets mode: +v mattoliverau08:38
*** threestrands has quit IRC09:07
*** tkajinam|lunch has quit IRC09:09
*** ccamacho has joined #openstack-swift09:42
*** mattoliverau has quit IRC09:54
*** ccamacho has quit IRC09:54
*** sorrison has quit IRC09:54
*** mugsie has quit IRC09:54
*** donnyd has quit IRC09:54
*** ccamel has quit IRC09:54
*** ormandj has quit IRC09:54
*** beisner has quit IRC09:54
*** rpittau has quit IRC09:54
*** gregwork has quit IRC09:54
*** toanster has quit IRC09:54
*** rledisez has quit IRC09:54
*** godog has quit IRC09:54
*** seongsoocho has quit IRC09:54
*** abelur has quit IRC09:54
*** d34dh0r53 has quit IRC09:54
*** openstackgerrit has quit IRC09:54
*** hugokuo has quit IRC09:54
*** evrardjp has quit IRC09:54
*** dasp has quit IRC09:54
*** StevenK has quit IRC09:54
*** manuvakery has quit IRC09:54
*** onovy has quit IRC09:54
*** kota_ has quit IRC09:54
*** peluse has quit IRC09:54
*** Anticimex has quit IRC09:54
*** coreycb has quit IRC09:54
*** clayg has quit IRC09:54
*** nottrobin has quit IRC09:54
*** spotz has quit IRC09:54
*** dabukalam has quit IRC09:54
*** timur has quit IRC09:54
*** tesseract has quit IRC09:54
*** pcaruana has quit IRC09:54
*** Jeffrey4l has quit IRC09:54
*** hoonetorg has quit IRC09:55
*** irclogbot_3 has quit IRC09:55
*** benj_ has quit IRC09:55
*** corvus has quit IRC09:55
*** edausq has quit IRC09:55
*** efried has quit IRC09:55
*** dosaboy has quit IRC09:55
*** szaher has quit IRC09:55
*** MooingLemur has quit IRC09:55
*** rcernin has quit IRC09:55
*** persia has quit IRC09:55
*** joeljwright has quit IRC09:55
*** fungi has quit IRC09:55
*** gmann has quit IRC09:55
*** zaitcev has quit IRC09:55
*** paladox has quit IRC09:55
*** jistr has quit IRC09:55
*** mvkr has quit IRC09:55
*** patchbot has quit IRC09:55
*** tonyb has quit IRC09:55
*** kukacz_ has quit IRC09:55
*** aluria has quit IRC09:55
*** rickflare has quit IRC09:55
*** dcourtoi has quit IRC09:55
*** cschwede has quit IRC09:55
*** timburke has quit IRC09:55
*** tristanC has quit IRC09:55
*** mathiasb has quit IRC09:55
*** noonedeadpunk has quit IRC09:55
*** baffle has quit IRC09:55
*** viks___ has quit IRC09:55
*** csmart has quit IRC09:55
*** f0o has quit IRC09:55
*** jrosser has quit IRC09:55
*** tdasilva has quit IRC09:55
*** mgagne has quit IRC09:55
*** lifeless has quit IRC09:55
*** mahatic has quit IRC09:55
*** early has quit IRC09:55
*** ab-a has quit IRC09:55
*** DHE has quit IRC09:55
*** alecuyer has quit IRC09:55
*** cjloader has quit IRC09:55
*** ChanServ has quit IRC09:55
*** mattoliverau has joined #openstack-swift10:04
*** sorrison has joined #openstack-swift10:04
*** ccamacho has joined #openstack-swift10:04
*** abelur has joined #openstack-swift10:04
*** godog has joined #openstack-swift10:04
*** rledisez has joined #openstack-swift10:04
*** seongsoocho has joined #openstack-swift10:04
*** toanster has joined #openstack-swift10:04
*** gregwork has joined #openstack-swift10:04
*** rpittau has joined #openstack-swift10:04
*** beisner has joined #openstack-swift10:04
*** ccamel has joined #openstack-swift10:04
*** donnyd has joined #openstack-swift10:04
*** mugsie has joined #openstack-swift10:04
*** ormandj has joined #openstack-swift10:04
*** evrardjp has joined #openstack-swift10:04
*** hugokuo has joined #openstack-swift10:04
*** d34dh0r53 has joined #openstack-swift10:04
*** openstackgerrit has joined #openstack-swift10:04
*** tesseract has joined #openstack-swift10:04
*** rcernin has joined #openstack-swift10:04
*** pcaruana has joined #openstack-swift10:04
*** zaitcev has joined #openstack-swift10:04
*** timur has joined #openstack-swift10:04
*** spotz has joined #openstack-swift10:04
*** dabukalam has joined #openstack-swift10:04
*** efried has joined #openstack-swift10:04
*** paladox has joined #openstack-swift10:04
*** jistr has joined #openstack-swift10:04
*** f0o has joined #openstack-swift10:04
*** Jeffrey4l has joined #openstack-swift10:04
*** persia has joined #openstack-swift10:04
*** jrosser has joined #openstack-swift10:04
*** mvkr has joined #openstack-swift10:04
*** hoonetorg has joined #openstack-swift10:04
*** baffle has joined #openstack-swift10:04
*** csmart has joined #openstack-swift10:04
*** joeljwright has joined #openstack-swift10:04
*** sets mode: +vvvv mattoliverau rledisez zaitcev joeljwright10:04
*** ab-a has joined #openstack-swift10:04
*** patchbot has joined #openstack-swift10:04
*** tonyb has joined #openstack-swift10:04
*** irclogbot_3 has joined #openstack-swift10:04
*** benj_ has joined #openstack-swift10:04
*** corvus has joined #openstack-swift10:04
*** edausq has joined #openstack-swift10:04
*** tdasilva has joined #openstack-swift10:04
*** mgagne has joined #openstack-swift10:04
*** kukacz_ has joined #openstack-swift10:04
*** lifeless has joined #openstack-swift10:04
*** mahatic has joined #openstack-swift10:04
*** early has joined #openstack-swift10:04
*** rickflare has joined #openstack-swift10:04
*** aluria has joined #openstack-swift10:04
*** dosaboy has joined #openstack-swift10:04
*** viks___ has joined #openstack-swift10:04
*** fungi has joined #openstack-swift10:04
*** dcourtoi has joined #openstack-swift10:04
*** szaher has joined #openstack-swift10:04
*** DHE has joined #openstack-swift10:04
*** MooingLemur has joined #openstack-swift10:04
*** mathiasb has joined #openstack-swift10:04
*** cschwede has joined #openstack-swift10:04
*** alecuyer has joined #openstack-swift10:04
*** timburke has joined #openstack-swift10:04
*** sets mode: +vvvv tdasilva mahatic cschwede timburke10:04
*** tristanC has joined #openstack-swift10:04
*** noonedeadpunk has joined #openstack-swift10:04
*** cjloader has joined #openstack-swift10:04
*** gmann has joined #openstack-swift10:04
*** ChanServ has joined #openstack-swift10:04
*** sets mode: +o ChanServ10:04
*** peluse has joined #openstack-swift10:04
*** dasp has joined #openstack-swift10:04
*** StevenK has joined #openstack-swift10:04
*** manuvakery has joined #openstack-swift10:04
*** onovy has joined #openstack-swift10:04
*** kota_ has joined #openstack-swift10:04
*** Anticimex has joined #openstack-swift10:04
*** coreycb has joined #openstack-swift10:04
*** clayg has joined #openstack-swift10:04
*** nottrobin has joined #openstack-swift10:04
*** sets mode: +vv kota_ clayg10:04
*** rdejoux has joined #openstack-swift10:15
*** pcaruana has quit IRC10:42
*** rcernin has quit IRC12:52
*** pcaruana has joined #openstack-swift12:54
*** diablo_rojo has joined #openstack-swift14:00
*** pcaruana has quit IRC14:38
*** rpittau is now known as rpittau|afk15:03
ormandjclayg: timburke: moving ratelimit after s3token fixed the PUT issue, so we're in the home stretch now. last remaining issue is how the 498 is handled that swift returns. we're seeing an exception16:06
ormandjDec  5 15:21:06 sw-proxy01.blah proxy-server: Returning 498 for PUT to AUTH_40a4e27e114a40b0829395945766c9f8/ormandj/test86 . Ratelimit (Max Sleep) Max Sleep Time Exceeded: 60.71 (txn: tx2d6d88fe30e54bb7a1ab6-005de92062)16:06
ormandjDec  5 15:21:06 sw-proxy01.blah proxy-server: 500 Internal Server Error: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/", line 229, in __call__#012    resp = self.h16:07
ormandjandle_request(req)#012  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/", line 265, in handle_request#012    res = handler(req)#012  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/controllers/", l16:07
ormandjine 128, in PUT#012    resp = req.get_response(  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/s3api/", line 1378, in get_response#012    headers, body, query)#012  File "/usr/lib/python2.7/dist-packages/swift/co16:07
ormandjmmon/middleware/s3api/", line 1362, in _get_response#012    raise InternalError('unexpected status code %d' % status)#012InternalError: 500 Internal Server Error (txn: tx2d6d88fe30e54bb7a1ab6-005de92062)16:07
ormandjyuck, sorry on paste.16:07
clayg😂. We’re not in good shape when a 500 & a traceback is considered “progress”16:08
ormandjhaha, i've been using swift a while, my yardstick has changed dimension :p16:08
ormandj503 Slow Down <- appropriate s3 response when rate limiting, so somehow 498 from swift should be getting translated to this. looking at it looks like that should be accounted for16:11
ormandji'm not sure where the 500 is coming from, i haven't spent a ton of time tracking that down yet.16:11
ormandjwhat we end up seeing is a 500 issued to the client for that request16:16
*** tesseract has quit IRC16:39
*** gyee has joined #openstack-swift16:41
*** openstackgerrit has quit IRC16:42
timburkeyeah, it's the 'unexpected status code' traceback -- i should really try to quiet that down...16:47
timburkei think we'd want to add the 498-handling around
claygtimburke: 👍17:06
ormandjyeah, what's odd to me, it's showing 50017:30
ormandji would have expected 503, as that's what the 498 slow down should be xlated to in s3response17:30
*** openstackgerrit has joined #openstack-swift17:35
openstackgerritTim Burke proposed openstack/swift master: Latch shard-stat reporting
*** ccamacho has quit IRC17:46
openstackgerritTim Burke proposed openstack/swift master: s3api: Better handle 498/429 responses
*** mvkr has quit IRC19:31
timburke:-( one of these things is not like the other...
timburkeNameError waiting to happen19:59
*** rcernin has joined #openstack-swift20:03
*** dosaboy has quit IRC20:04
ormandjtimburke: hilariously, your proposed solution is what we just figured out to implement a few minutes ago after hours of adding log lines because i have no idea what i'm doing :p20:08
ormandjtimburke: on the note, the rate limiting seems to have to be after the s3api middle ware, or it breaks entirely for s320:09
ormandji should be more clear, the clients of s3 do not get notification of rate limiting20:09
ormandjso they can't implement exponential backoff/etc20:09
ormandjalso, does rate limiting even work if not after s3api, for s3api requests? it didn't seem to in our testing, at least we never saw the rate limiting logs being produced in our debug output20:10
*** rcernin has quit IRC20:18
*** zaitcev_ has joined #openstack-swift20:18
*** ChanServ sets mode: +v zaitcev_20:18
*** zaitcev has quit IRC20:22
ormandjtimburke: FWIW, we just backported to stein and deployed in our dev env and verified the rate limiting is functional20:43
ormandjclient is seeing the 503s and behaving20:43
ormandj(was doing 1000 parallel puts)20:47
timburkeglad to hear it's doing better :-D21:00
*** dosaboy has joined #openstack-swift21:09
*** rdejoux has quit IRC21:26
*** persia has quit IRC21:43
*** persia has joined #openstack-swift21:45
*** dasp_ has joined #openstack-swift21:52
*** dasp has quit IRC21:55
openstackgerritTim Burke proposed openstack/swift master: Set swift_source in account_quotas middleware
openstackgerritTim Burke proposed openstack/swift master: Set swift_source more in s3api middleware
*** rcernin has joined #openstack-swift22:28
openstackgerritTim Burke proposed openstack/swift master: Set swift_source more in versioned_writes
mattoliverauset swift_source in all the things :P ^22:38
*** zaitcev_ is now known as zaitcev22:42
zaitceva good practice22:42
timburkeall of them!22:52
*** tkajinam has joined #openstack-swift23:07
*** mvkr has joined #openstack-swift23:55

Generated by 2.15.3 by Marius Gedminas - find it at!