| *** rlandy|bbl is now known as rlandy | 00:03 | |
| *** rlandy is now known as rlandy|out | 01:19 | |
| *** rlandy is now known as rlandy|out | 01:24 | |
| *** ysandeep|out is now known as ysandeep | 06:14 | |
| opendevreview | Rodolfo Alonso proposed openstack/project-config master: Remove lower-constraints and tox-py36 from Neutron Grafana https://review.opendev.org/c/openstack/project-config/+/844254 | 07:43 |
|---|---|---|
| *** ysandeep is now known as ysandeep|lunch | 08:09 | |
| opendevreview | Merged openstack/ci-log-processing master: Move container images to quay.io https://review.opendev.org/c/openstack/ci-log-processing/+/843107 | 08:37 |
| dpawlik | fungi: hey, I need to remove one indices due opensearch is out of space. I see messages with "DEBUG'" might be a good solution, but the results I will see after 14 days... | 09:00 |
| *** ysandeep|lunch is now known as ysandeep | 09:26 | |
| *** jpena|off is now known as jpena | 09:45 | |
| *** rlandy|out is now known as rlandy | 10:18 | |
| *** rlandy_ is now known as rlandy__ | 10:24 | |
| fungi | dpawlik: that sounds fine, are you asking for help? | 11:46 |
| dpawlik | fungi: no :) I should handle that. Thanks for asking | 11:47 |
| *** dviroel|afk is now known as dviroel | 12:21 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Add new information for Opensearch cluster configuration https://review.opendev.org/c/openstack/ci-log-processing/+/839507 | 12:34 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Parse performance.json file and send to separate index https://review.opendev.org/c/openstack/ci-log-processing/+/840466 | 12:45 |
| *** dviroel_ is now known as dviroel | 13:15 | |
| opendevreview | Merged openstack/project-config master: Add a repository for the Large Scale SIG https://review.opendev.org/c/openstack/project-config/+/843534 | 14:01 |
| sean-k-mooney | clarkb: fungi am i imagining it or are the nodepool images hosted somewhere as part of the buidl output | 14:07 |
| sean-k-mooney | like on nl01.opendev.org or something | 14:08 |
| sean-k-mooney | ah yes | 14:08 |
| sean-k-mooney | nb03.opendev.org | 14:08 |
| sean-k-mooney | oh those are just the logs for the arm build | 14:10 |
| sean-k-mooney | 2 has the x86 logs | 14:10 |
| *** rlandy__ is now known as rlandy | 14:16 | |
| fungi | sean-k-mooney: nb01 and nb02 host images and image build logs for x86_64 (amd64), they grab the image build requests at random between them as available | 14:19 |
| fungi | nb03 is for aarch64 (arm64) as you discovered | 14:20 |
| fungi | the image build logs are served from the top-level directories, but if you append /images you get the qcow2 images and their checksums | 14:21 |
| sean-k-mooney | fungi: cool am on stable train we seam to be missing bash in a job | 14:24 |
| sean-k-mooney | or rather | 14:24 |
| sean-k-mooney | the devstack_ceph_plugin | 14:24 |
| sean-k-mooney | is useing ansible raw with execuable /bin/bash | 14:24 |
| sean-k-mooney | but we are seeign an issue with source not found | 14:24 |
| sean-k-mooney | in /bin/sh | 14:24 |
| sean-k-mooney | so we were just trying to figureout whats goign on in the image | 14:25 |
| sean-k-mooney | i.e. is /bin/sh symlinked to /bin/dash instead | 14:25 |
| sean-k-mooney | it was working for years but started failing | 14:25 |
| fungi | on what node label/distro? | 14:26 |
| fungi | ubuntu i guess? | 14:26 |
| fungi | stable/train presumably tests on ubuntu-bionic | 14:26 |
| fungi | (18.04 lts) | 14:26 |
| sean-k-mooney | ya i can see if i can get the link | 14:27 |
| sean-k-mooney | elodilles: do you have it ^ | 14:28 |
| fungi | yeah, pti says to use bionic for train: https://governance.openstack.org/tc/reference/runtimes/train.html | 14:28 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/9c886d2f459143cdaf587ba0b93c7cd4/log/job-output.txt | 14:29 |
| sean-k-mooney | that was the last succesful run i think | 14:29 |
| fungi | okay, so failures started about a week ago | 14:30 |
| sean-k-mooney | https://paste.opendev.org/show/bxCRqAFCVzM28hDqlWix/ | 14:31 |
| elodilles | and this is the error: https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 | 14:31 |
| sean-k-mooney | yep gibi was looking at itt | 14:31 |
| elodilles | and we are around here when it happens: https://opendev.org/openstack/nova/src/branch/stable/train/gate/live_migration/hooks/ceph.sh#L10 | 14:31 |
| sean-k-mooney | fungi: so i was considering pulling the image to see what was going on but i suspect i has to be related to some kind fo image/package change | 14:32 |
| sean-k-mooney | as you said its at /images https://nb02.opendev.org/images/ubuntu-bionic-0000229942.qcow2 its 12G but that not that bad | 14:34 |
| fungi | yeah, most of that is the git cache | 14:34 |
| sean-k-mooney | yep thats about 8G | 14:34 |
| sean-k-mooney | for the repos | 14:34 |
| fungi | sean-k-mooney: while i haven't checked a job node, just looking on a random bionic server i see that /bin/sh is indeed symlinked to dash by default | 14:38 |
| fungi | that error looks like it might be coming from a package maintscript, do you happen to know? | 14:41 |
| fungi | my first guess would be that package updated a week ago and added a bashism in a maintscript but didn't correct the shebang on it | 14:41 |
| fungi | i doubt this would have changed in the images | 14:42 |
| fungi | debian and ubuntu have traditionally linked /bin/sh to dash since many years, expecting you to call /bin/bash instead if you want bash builtins like "source" | 14:43 |
| sean-k-mooney | fungi: ya i was not sure if that was a recent change | 14:50 |
| sean-k-mooney | the issue here is we are pasing /bin/bash | 14:50 |
| sean-k-mooney | as the executable | 14:50 |
| fungi | where is that "source" command being invoked? | 14:51 |
| sean-k-mooney | in the "sciprt" invoke by ansble raw | 14:51 |
| sean-k-mooney | its notre realy a script but inline commands | 14:51 |
| fungi | have a link to that task? | 14:51 |
| sean-k-mooney | looking for it now | 14:53 |
| fungi | just curious to see the inline commands since the error doesn't say what file it's trying to source | 14:53 |
| sean-k-mooney | its in nova i think | 14:53 |
| sean-k-mooney | fungi: https://github.com/openstack/nova/blob/stable/train/gate/live_migration/hooks/ceph.sh#L10= | 14:56 |
| sean-k-mooney | fungi: so the not found is coming from https://github.com/openstack/nova/blob/stable/train/gate/live_migration/hooks/ceph.sh#L16= | 14:57 |
| sean-k-mooney | but it shoudl be provded by the previous source | 14:57 |
| sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L889= | 14:58 |
| *** ysandeep is now known as ysandeep|out | 14:58 | |
| fungi | okay, so it's "source $BASE/new/devstack/functions" in the function prepare_ceph definition | 15:05 |
| fungi | what calls the prepare_ceph function? since it's only on a stable branch codesearch isn't being much help | 15:07 |
| *** dviroel is now known as dviroel|lunch | 15:08 | |
| fungi | my expectation, given the error, is that prepare_ceph is being called (directly or indirectly) from some call under sh or from some script with a shebang which isn't bash | 15:08 |
| fungi | ahh, in nova's gate/live_migration/hooks/run_tests.sh | 15:11 |
| fungi | which gets invoked by a cmd task in playbooks/legacy/nova-live-migration/run.yaml | 15:12 |
| fungi | and that task does seem to set executable: /bin/bash | 15:13 |
| fungi | so, yes, i'm similarly stumped | 15:15 |
| clarkb | its running as a d-g hook? | 15:23 |
| clarkb | if so then none of what the script and ansible do matter. d-g is what matters | 15:23 |
| fungi | oh, true it's in a post_test_hook definition | 15:24 |
| fungi | executed as `with_timeout call_hook_if_defined "post_test_hook"` | 15:25 |
| clarkb | devstack-vm-gate-wrap.sh also sets a shebang of /bin/bash though | 15:25 |
| fungi | and with_timeout is defined as: | 15:26 |
| fungi | timeout -s 9 ${REMAINING_TIME}m bash -c "source $WORKSPACE/devstack-gate/functions.sh && $cmd" | 15:26 |
| fungi | oh! | 15:26 |
| fungi | no, nevermind, the && $cmd should be executing the hook script, which also has #!/bin/bash | 15:27 |
| fungi | er, though $cmd is call_hook_if_defined | 15:27 |
| fungi | which calls the hook command thusly: | 15:28 |
| fungi | tsfilter $hook_name | tee $filename | 15:28 |
| fungi | though tsfilter is calling $@ and then passing the resulting stdout/stderr through awk | 15:29 |
| fungi | so should still come down to the shebang in the hook script | 15:29 |
| *** dviroel|lunch is now known as dviroel | 16:20 | |
| *** jpena is now known as jpena|off | 17:10 | |
| opendevreview | Alex Kavanagh proposed openstack/openstack-zuul-jobs master: Update 'openstack-python3-charm-jobs' to latest Python https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/844322 | 17:47 |
| opendevreview | Alex Kavanagh proposed openstack/openstack-zuul-jobs master: Update 'openstack-python3-charm-jobs' to latest Python https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833330 | 17:48 |
| *** rlandy is now known as rlandy|mtg | 18:13 | |
| *** rlandy|mtg is now known as rlandy | 19:12 | |
| *** timburke_ is now known as timburke | 20:59 | |
| *** dviroel is now known as dviroel|out | 21:34 | |
| *** rlandy is now known as rlandy|out | 22:44 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!