| *** rlandy|ruck is now known as rlandy|out | 00:58 | |
| *** pojadhav- is now known as pojadhav | 05:02 | |
| *** ysandeep|out is now known as ysandeep | 05:45 | |
| *** cloudnull5 is now known as cloudnull | 06:09 | |
| *** ysandeep is now known as ysandeep|afk | 07:30 | |
| *** ysandeep|afk is now known as ysandeep | 07:44 | |
| *** akahat|out is now known as akahat|ruck | 08:27 | |
| *** ysandeep is now known as ysandeep|lunch | 09:07 | |
| *** ysandeep|lunch is now known as ysandeep | 09:28 | |
| opendevreview | Merged openstack/project-config master: Revert "Disable nodepool temporarily" https://review.opendev.org/c/openstack/project-config/+/823357 | 09:52 |
|---|---|---|
| *** rlandy_ is now known as rlandy|ruck | 11:06 | |
| *** dviroel|rover|afk is now known as dviroel|rover | 11:16 | |
| *** pojadhav is now known as pojadhav|afk | 11:36 | |
| *** pojadhav|afk is now known as pojadhav | 13:05 | |
| *** ysandeep is now known as ysandeep|dinner | 13:05 | |
| *** ysandeep|dinner is now known as ysandeep | 14:54 | |
| *** arxcruz is now known as arxcruz|ruck | 15:21 | |
| *** akahat|ruck is now known as akahat | 15:22 | |
| *** dviroel|rover is now known as dviroel | 15:26 | |
| *** ysandeep is now known as ysandeep|out | 16:46 | |
| *** rlandy|ruck is now known as rlandy|ruck|lunch | 16:58 | |
| *** rlandy|ruck|lunch is now known as rlandy|ruck | 17:27 | |
| clarkb | I note that we've got a few hosts that have LE certs expiring in 29 days or less. Not likely to be able to dig into that today or this week, but calling it out as noticed and should be able to inspect next week | 17:37 |
| fungi | i must have missed the notifications about those | 17:39 |
| fungi | i saw some for openstackid, but hopefully we can move translate.o.o to the new openinfra idp before that becomes critical | 17:40 |
| fungi | infra-root: how do folks feel about putting copies of the release sdists/wheels for tools like bindep and git-review on tarballs.opendev.org? | 17:54 |
| fungi | i feel like that was our intent all along, but we never got around to implementing it after moving the projects into the new namespace | 17:55 |
| fungi | i'm happy to work on solving that in the near future unless there were obvious reasons we avoided that and i've merely forgotten them | 17:55 |
| clarkb | no objections from me | 17:58 |
| fungi | it also came to my attention that when we moved tarballs subdirs for other projects to the new namespace, we missed moving git-review (possibly others, i haven't looked closely, we did move bindep though) | 18:00 |
| fungi | that's probably on me, planning to fix that as well | 18:00 |
| *** dviroel is now known as dviroel|afk | 18:29 | |
| clarkb | jpic: hrw has a similar problem in reverse. I'm under a mountain of paperwork the next few days but can hopefully look at it next week. | 18:54 |
| clarkb | jpic: I think if you can login with your old ubuntu one openid and use the old account that would work without any intervention from us. However, if that isn't possible we can retire the old account and remove the conflicting attributes allowing a new account to be created | 18:54 |
| fungi | i am in the same boat, but happy to help once no longer swamped | 18:54 |
| clarkb | jpic: Another option is for you to create a new account with a different email today, then in the future (early next week?) we can retire the old account and you can add the prefered email to the new account | 18:55 |
| *** artom__ is now known as artom | 18:55 | |
| *** rlandy is now known as rlandy|ruck | 20:29 | |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 21:10 | |
| timburke__ | i've been seeing a fair number of timeouts lately; seems like it's usually around https://github.com/openstack/swift/blob/master/test/unit/common/test_utils.py#L5565-L5615 -- how easy would it be for me to run something like https://github.com/tipabu/python-stack-xray/blob/master/python-stack-xray while the test is hung, so i can mock out whatever part of the socket library seems to be blocking? | 21:18 |
| timburke__ | fwiw, https://zuul.openstack.org/stream/64cb0246c8554f05a8559db731b78563?logfile=console.log seems to be currently in that problematic state | 21:18 |
| clarkb | timburke__: give me a few minutes and I can take a look at it | 21:19 |
| timburke__ | thanks! | 21:19 |
| *** timburke__ is now known as timburke | 21:19 | |
| fungi | it the hang reproducible? i'm guessing it's unpredictable based on your description | 21:19 |
| fungi | one tactic we've used is to propose a dnm change which adds a bunch of copies of the problem job (removing all the other jobs), and instrument the data collection tool in that change or a parent change | 21:21 |
| fungi | that way you don't spend all day rechecking in hopes of tripping the right conditions | 21:21 |
| timburke | unpredictable -- eventually, a recheck will get me going, but it's a pain on both sides to keep doing that | 21:21 |
| clarkb | timburke: it looks like that tool expects to be run from outside of the process | 21:21 |
| clarkb | which makes this more complicated since you really only get the one process via the ansible exec | 21:22 |
| clarkb | what we can do is hold the node and you can get on it though | 21:22 |
| clarkb | let me work on doing that | 21:22 |
| fungi | does it hang ~forever? if so then, yeah, an autohold would probably get you what you need | 21:23 |
| timburke | clarkb, yes -- it's something i've used before when debugging customer issues. ssh onto a box, copy in the script, find the process you want to trace, and run against it. if anybody has better ideas, i'm all ears :-) | 21:23 |
| fungi | job fails with a timeout, zuul holds the node, you log in and inspect the hung process | 21:23 |
| timburke | fungi, yes, afaict -- the job either completes in <20min, or timeouts out at 90min | 21:24 |
| fungi | if it just runs too long and tanks the job but vanishes before you can log in to take a look then that gets a lot harder | 21:24 |
| clarkb | timburke: do you have your ssh pubkey? I've put a hold in and if I get that now you can ssh in now and run your tool | 21:24 |
| clarkb | you may want to try and get in now while it is running in case the job cancellation kills the running process. If that happens you might have to rerun tox and try to reproduce on the host | 21:25 |
| timburke | https://paste.opendev.org/raw/811956/ | 21:25 |
| clarkb | but if we move quickly maybe you can just get what you need right now :) | 21:25 |
| clarkb | timburke: ssh root@198.72.124.64 | 21:25 |
| fungi | worst case do the many-jobs dnm change and bump the timeout to the maximum, then find one which exhibits the issue and that may buy longer to troubleshoot | 21:25 |
| clarkb | the node is held too so when the job ends yuo can keep poking at it. Let us know when you are done and we'll delete it | 21:26 |
| clarkb | (done might mean "this host is no longer useful for debugging but we haven't figured it out yet" too) | 21:26 |
| timburke | got it! thanks. seems like it may actually be something down in logging :-/ but i've got enough to go digging some more | 21:31 |
| timburke | (https://paste.opendev.org/show/811957/ for the curious) | 21:31 |
| fungi | that's a very neat tool, didn't know about it before now | 21:34 |
| timburke | torgomatic (who doesn't work on swift anymore, unfortunately for us) wrote it a while ago and i've found it *incredibly* useful | 21:34 |
| fungi | we've implemented https://pypi.org/project/yappi in zuul and nodepool to similar ends, and it's been indispensible | 21:35 |
| clarkb | timburke: zuul has similar tooling built into it. It accepts sigusr2 to toggle on a profiler and print stack traces if you've got the right stuff installed. | 21:35 |
| clarkb | then you let it run in slow mode with the profiler on for a few minutes, hit it with sigusr2 again and you get the stack traces and profiling data | 21:36 |
| fungi | yeah, the signal handler does thread dumps, and we've got it set to also toggle yappi's profiler | 21:36 |
| timburke | i'm sure other people have written similar things. his trick to writing both bash and python in the same file is pretty clever -- if it were for anything more substantial than a debugging tool like this, it'd never pass review ;-) | 21:36 |
| fungi | that's definitely an amusing hack for abusing string quoting semantics in the two languages | 21:38 |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 22:22 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!