| *** haleyb|out is now known as haleyb | 00:31 | |
| frickler | #status notice zuul processing is broken since about 03:00 UTC, investigation is in progress, please be patient | 08:01 |
|---|---|---|
| opendevstatus | frickler: sending notice | 08:01 |
| -opendevstatus- NOTICE: zuul processing is broken since about 03:00 UTC, investigation is in progress, please be patient | 08:02 | |
| opendevstatus | frickler: finished sending notice | 08:04 |
| *** tobias-u1 is now known as tobias-urdin | 08:19 | |
| *** mikal0 is now known as mikal | 08:57 | |
| stephenfin | frickler: fungi: Just because we've already had a few issues here, would it be worth a status update about pbr 7.1.3 being released? | 11:54 |
| stephenfin | something like: pbr 7.1.3 has been released. If you see packaging-related issues in CI, particularly for stable branches, please report them on #openstack-oslo ? | 11:55 |
| fungi | stephenfin: maybe, though i did mention during the tc meeting yesterday to be on the lookout for it too | 13:42 |
| stephenfin | With any luck, we won't see any issues and things will be rosy, but the fact we're on attempt no. 4 suggests we should be on the lookout | 13:44 |
| stephenfin | The TC notification is probably good enough though. I'm sure people will find us if things do break | 13:44 |
| tkajinam | could anyone add me to https://launchpad.net/~zaqar-drivers/+members as an admin ? | 14:12 |
| tkajinam | fungi, I'd appreciate it if you have a few minutes ^^^ | 14:12 |
| tkajinam | I think none of the current admins is around the community these days | 14:13 |
| fungi | tkajinam: your kajinamit account, not the takashi-kajinami one? | 14:14 |
| fungi | i can't remember which one you're using now | 14:14 |
| tkajinam | fungi, yes kajinamit is the right one | 14:14 |
| tkajinam | I don't remember with which email address I create the other one but I believe that's also mine :-P | 14:14 |
| fungi | the other one is associated with a gmail address rather than ntt | 14:15 |
| fungi | anyway, done, please double-check | 14:15 |
| tkajinam | fungi, thanks. it works | 14:17 |
| tkajinam | I assume we don't need the release team as its member so I'll deactivate it too | 14:17 |
| fungi | tkajinam: yeah, unnecessary these days. once upon a time the release managers ran scripts that updated bug target milestones but that hasn't been the case for many, many years now | 14:27 |
| bauzas | folks, do we still have zuul processing issue ? I want to recheck some change | 14:47 |
| bauzas | when looking at https://zuul.opendev.org/t/openstack/status I see new changes in the check pipeline | 14:48 |
| frickler | bauzas: there are capacity issues that affect only some special kinds of large nodes, but for general changes it should be fine to recheck | 14:58 |
| melwitt | a while back I added a patch to devstack to collect guru meditation report on failed job runs (https://review.opendev.org/c/openstack/devstack/+/997443) and I thought to check on it today and find it's erroring out but there is almost no information in a job log for example https://zuul.opendev.org/t/openstack/build/72d39e4f417b4a2881d1382d803e8266/log/job-output.txt#40151 | 15:11 |
| melwitt | does anyone have any suggestion how to debug something like this? | 15:12 |
| bauzas | frickler: ty, I'll recheck then | 15:17 |
| fungi | melwitt: i'd look at what commands `TASK [Collect Guru Meditation Reports from OpenStack services]` and then see if manpages for anything in there document an exit code of `-12` (which seems unusual so probably oddly specific to one particular command?) | 16:33 |
| melwitt | good point | 16:42 |
| melwitt | thanks :) | 16:42 |
| fungi | there's also some chance that the exit code belongs to ansible rather than a command it ran, i don't really know | 16:43 |
| fungi | melwitt: oh, there's also task-specific details zuul records which you can browse in the console view, more than simple stdout/stderr, which may help speed up the investigation: https://zuul.opendev.org/t/openstack/build/72d39e4f417b4a2881d1382d803e8266/console#8/0/38/compute1 | 16:45 |
| fungi | no idea if any of the additional detail there helps | 16:45 |
| melwitt | a-ha, cool thanks | 16:46 |
| fungi | it at least records what was passed to ansible, which for a shell action includes the shell commands it would try to run, so less digging around in playbook/role definitions in git | 16:48 |
| clarkb | melwitt: fungi that script doesn't look like it emits anything and isn't set -e. Basically the lack of a set -e and the last command being sleep make me suspicious it is running at all with an rc of -12. But without having some indication of it running at all (set -x may be a good idea) its hard to say | 16:53 |
| clarkb | so maybe add set -x and see if that sheds light on where it is going wrong? My suspicion is that maybe it is /bin/sh or even just cmd not shell and we're not able to run the command at all /me looks for the task definition | 16:54 |
| melwitt | yeah, ok that is my bad. I'll propose a change to add set -x | 16:55 |
| melwitt | I'm sorry I didn't do it originally 😩 | 16:55 |
| clarkb | https://opendev.org/openstack/devstack/src/branch/master/playbooks/post.yaml#L30-L31 ok it is bash so it should be capable of processing hte array | 16:55 |
| clarkb | so ya I would start with making the task a little more debuggable to try and track down where the rc -12 is coming from | 16:55 |
| fungi | maybe also consider `-e` and `-o pipefail` to short-circuit the script asap if it hits a problem rather than possibly resulting in misleading cascade failures for subsequent commands | 16:55 |
| melwitt | good call | 16:56 |
| melwitt | thanks for the good ideas :) | 16:56 |
| fungi | though if you do that, you need to safeguard commands you expect to exit nonzero normally within the script too | 16:56 |
| melwitt | ++ | 17:10 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Undefine zuul_site_mirror_fqdn on non x86_64 hosts https://review.opendev.org/c/openstack/project-config/+/1009325 | 17:33 |
| *** priteau_ is now known as priteau | 21:38 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!