Friday, 2020-09-18

« Thursday, 2020-09-17 Index Saturday, 2020-09-19 »
timburkehuh. i guess the lock didn't get greened? i wonder how that happened...00:35
DHEI skipped all the green threads because there's a LOT and the file is 200+ kb00:43
DHEand no references to current_thread except in the new bit at the end00:43
openstackgerritTim Burke proposed openstack/swift master: Replace threading._active_limbo_lock with a re-entrant lock  https://review.opendev.org/75259300:53
openstackgerritGhanshyam Mann proposed openstack/python-swiftclient master: [goal] Migrate testing to ubuntu focal  https://review.opendev.org/74460500:56
timburkeDHE, do you have an idea of how often you run into this trouble? one of the frustrating things for me was that i couldn't repro the deadlock at will, and my home cluster sees little enough use that it'd only come up every day or two at most -- so i'd need to be running with p 752593 for a while before i'd feel confident that it addressed the issue01:01
patchbothttps://review.opendev.org/#/c/752593/ - swift - Replace threading._active_limbo_lock with a re-ent... - 1 patch set01:01
timburkeif you see it more frequently and can get that patch applied (and it seems to resovle the issue), i'd feel a lot more confident in the proposed fix01:01
DHEtimburke: I have 16 processes running on this system for nearly 2 months now. about half of them needed a kick01:02
*** gyee has quit IRC01:03
DHEit's a video platform serving segmented videos. if a user seeks or closes the video it will abort a transfer in progress. so I suspect I actually have an abnormally high occurrence of aborted transactions01:04
timburkecoincidentally, i largely use swift as a media server at home...01:06
timburkedo you feel like you have good tooling for detecting the deadlock?01:07
DHEit's pretty simple. grep futex /proc/$PID/wchan01:07
timburkeTIL about wchan01:09
DHEyeah it's an obscure feature but potentially useful01:14
*** MooingLemur has quit IRC01:19
*** MooingLe1ur has joined #openstack-swift01:19
*** gregwork has quit IRC01:27
*** rcernin has quit IRC02:29
*** rcernin has joined #openstack-swift03:15
*** psachin has joined #openstack-swift03:37
openstackgerritTim Burke proposed openstack/swift master: Replace threading._active_limbo_lock with a re-entrant lock  https://review.opendev.org/75259303:52
*** ianychoi has quit IRC04:29
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-swift04:33
openstackgerritMerged openstack/swift master: Authors/ChangeLog for 2.26.0  https://review.opendev.org/75053704:34
*** shanu_ has quit IRC05:01
openstackgerritlikui proposed openstack/swift master: Remove babel.cfg  https://review.opendev.org/75262006:11
*** samueldmq has quit IRC06:52
*** seongsoocho has quit IRC06:53
*** coreycb has quit IRC06:53
*** rpittau|afk has quit IRC06:53
*** fyx has quit IRC06:53
*** donnyd has quit IRC06:53
*** abelur has quit IRC06:54
*** samueldmq has joined #openstack-swift06:55
*** seongsoocho has joined #openstack-swift06:56
*** coreycb has joined #openstack-swift06:56
*** rpittau|afk has joined #openstack-swift06:56
*** abelur has joined #openstack-swift06:56
*** donnyd has joined #openstack-swift06:57
*** fyx has joined #openstack-swift06:57
openstackgerritlikui proposed openstack/swift master: Remove babel.cfg  https://review.opendev.org/75262008:52
*** psachin has quit IRC10:09
*** rcernin has quit IRC10:09
*** psachin has joined #openstack-swift10:11
*** psachin has quit IRC10:16
*** psachin has joined #openstack-swift10:18
*** rcernin has joined #openstack-swift10:45
*** rcernin has quit IRC10:54
*** cschwede has joined #openstack-swift11:41
*** ChanServ sets mode: +v cschwede11:41
*** rcernin has joined #openstack-swift11:54
*** rcernin has quit IRC11:59
*** ianychoi has joined #openstack-swift14:59
openstackgerritGhanshyam Mann proposed openstack/python-swiftclient master: [goal] Migrate testing to ubuntu focal  https://review.opendev.org/74460515:29
openstackgerritGhanshyam Mann proposed openstack/python-swiftclient master: Clean up some requirements  https://review.opendev.org/75047515:29
*** tkajinam has quit IRC15:44
*** rcernin has joined #openstack-swift15:58
*** zaitcev has quit IRC15:59
*** rcernin has quit IRC16:03
*** cschwede has quit IRC16:17
*** zaitcev has joined #openstack-swift16:18
*** ChanServ sets mode: +v zaitcev16:18
*** psachin has quit IRC16:49
*** Qianbiao has joined #openstack-swift16:56
Qianbiaohello, swift stat shows: "Account HEAD failed: http://192.168.43.145:8080/v1/AUTH_64654e4b99a3459d982ed1907bd9df71 503 Service Unavailable"16:57
Qianbiaohow should i fix it.16:57
*** Qianbiao has quit IRC16:58
*** Qianbiao has joined #openstack-swift17:00
openstackgerritMerged openstack/swift master: Run swift-tox-func-encryption-py37 job in the gate  https://review.opendev.org/75258017:12
DHEcheck that your account servers are reachable and working from the proxy you queried (firewall? service not running?)17:21
DHEdisks mounted? permissions on disks okay?17:22
Qianbiao<DHE> yes account-server is active17:26
QianbiaoDHE account-server: 127.0.0.1 - - [18/Sep/2020:17:25:36 +0000] "HEAD /sdb1/37/AUTH_64654e4b99a3459d982ed1907bd9df71" 507 - "HEAD http://192.168.43.145:8080/v1/AUTH_64654e4b99a3459d982ed1907bd9df71?format=json" "tx710c024c2591450ebef95-005f64ed90" "proxy-server 11212" 0.0003 "-" 11208 -17:28
Qianbiaois this means account-server is bind to 127.0.0.1?17:28
DHEI'm more interested in the '507' bit which I believe is an error code17:31
DHEinsufficient storage, which I would take as meaning check out the disks17:32
DHE/srv/node/sdb1/ should be owned and accessible by swift, and you should see a directory in there named account17:32
Qianbiaowill check17:33
QianbiaoDHE http://paste.openstack.org/show/798082/17:35
*** gyee has joined #openstack-swift17:35
Qianbiaoseems it is empty. the owner is right17:35
DHEso you edited the config files to point /srv/node elsewhere17:36
QianbiaoDHE update container-server conf  devices = xxxx ?17:43
DHEyes17:44
QianbiaoDHE Then I just restart services?17:46
*** klamath_atx has joined #openstack-swift17:47
DHEthe account server? yes17:47
DHEyou had devices on default?17:47
Qianbiaoyes17:47
Qianbiaonothing changes still 50317:48
QianbiaoAccount GET failed: http://192.168.43.145:8080/v1/AUTH_64654e4b99a3459d982ed1907bd9df71?format=json 503 Service Unavailable17:48
openstackgerritTim Burke proposed openstack/python-swiftclient master: Clean up some requirements  https://review.opendev.org/75047517:54
openstackgerritTim Burke proposed openstack/python-swiftclient master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/73274217:56
QianbiaoDHE may it possible that the auth account is wrong?18:11
DHEthe account server itself has no authentication required. so if it's returning errors besides 404 then something's wrong18:12
QianbiaoDHE http://paste.openstack.org/show/798084/18:13
Qianbiaothe "Node error limited 127.0.0.1:6612 (sdb1)" may hint you something?18:14
DHEproxy server logs aren't interesting. definitely want account-server logs18:16
klamath_atxHowdy, I was looking for some advise on two things, one was enabling s3 credentials caching with region awarenesses, that seems to be missing out of the current U branch of swift.  Is there a means to rate limit s3 clients in any version of swift, or would rate limiting of s3 be handled up stream further then swift?18:17
*** irclogbot_2 has quit IRC18:19
*** irclogbot_0 has joined #openstack-swift18:24
*** irclogbot_0 has quit IRC18:31
*** Qianbiao has quit IRC18:31
*** irclogbot_3 has joined #openstack-swift18:36
openstackgerritOpenStack Release Bot proposed openstack/swift stable/victoria: Update .gitreview for stable/victoria  https://review.opendev.org/75275019:17
openstackgerritOpenStack Release Bot proposed openstack/swift stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria  https://review.opendev.org/75275119:18
openstackgerritOpenStack Release Bot proposed openstack/swift master: Update master for stable/victoria  https://review.opendev.org/75275219:18
openstackgerritTim Burke proposed openstack/python-swiftclient master: Clean up some requirements  https://review.opendev.org/75047519:22
openstackgerritTim Burke proposed openstack/python-swiftclient master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/73274219:22
openstackgerritTim Burke proposed openstack/swift master: Get func tests passing with a minimal proxy pipeline  https://review.opendev.org/73055820:05
openstackgerritGhanshyam Mann proposed openstack/python-swiftclient master: [goal] Migrate testing to ubuntu focal  https://review.opendev.org/74460521:02
*** rcernin has joined #openstack-swift21:27
*** rcernin has quit IRC21:35
*** rcernin has joined #openstack-swift21:36
*** rcernin has quit IRC21:42
openstackgerritAde Lee proposed openstack/swift master: replace md5 with oslo_utils version  https://review.opendev.org/75196621:56
openstackgerritAde Lee proposed openstack/swift master: replace md5 with swift utils version  https://review.opendev.org/75196621:57
*** rcernin has joined #openstack-swift22:01
*** rcernin has quit IRC23:06
openstackgerritClay Gerrard proposed openstack/swift master: Log error processing manifest as ServerError  https://review.opendev.org/75277023:08
*** patchbot has quit IRC23:10
*** patchbot has joined #openstack-swift23:18
*** rcernin has joined #openstack-swift23:39
« Thursday, 2020-09-17 Index Saturday, 2020-09-19 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!