| *** ykarel_ is now known as ykarel | 05:15 | |
| *** pojadhav|out is now known as pojadhav | 05:38 | |
| *** marios is now known as marios|ruck | 05:41 | |
| *** amoralej|off is now known as amoralej | 06:26 | |
| zbr | sshnaidm: do you have any idea on how to get more infor about the stopped container with podman? the one with no logs? I am afraid that this some kind of ticking bomb for us using the molecule-podman driver as I seen lots of users reporting the same kind of problem. I really want to find a wayt to improve the UX on these failures, making clear what | 07:04 |
|---|---|---|
| zbr | happened. If the containers failed to start, create should not report success, and coverage should never start, agree? | 07:04 |
| sshnaidm | zbr, paste "podman inspect container_name" | 07:05 |
| zbr | yeah, that is what i did yesterday, any interesting things on https://gist.github.com/ssbarnea/80941778d05dbd9060f6a446a8ca951c ? | 07:05 |
| zbr | that bash command should create an infinite loop, i doubt is broken considering it was not touched in years | 07:06 |
| zbr | that is amazing, "state: running" but "podman ps" does not see it, going to try again, just to be sure. | 07:08 |
| zbr | but how can a container be running, and still have and exitcode=0 ? | 07:08 |
| sshnaidm | zbr, well, the container is running | 07:12 |
| * zbr trying to reproduce it... | 07:13 | |
| sshnaidm | maybe sudo podman ps? | 07:13 |
| sshnaidm | if you don't see it in ps, something bad happened with your local podman setup | 07:13 |
| sshnaidm | I ran the command just now and see container | 07:13 |
| zbr | now it started, but gather facts still fails, with interesting error: https://gist.github.com/ssbarnea/57736b3fcd2841b9fbe909d2c6a43c8c | 07:15 |
| zbr | that is from ansible 2.9.9 (the one on ubuntu) | 07:16 |
| zbr | that line "/usr/bin/python3.6 && sleep 0" looks weird, it does succeed but we still get setup module failure. | 07:19 |
| sshnaidm | that's the usual command | 07:22 |
| sshnaidm | and container is still running? | 07:22 |
| zbr | yeah | 07:22 |
| sshnaidm | and what happens when you run podman exec centos8 /bin/sh -c "/usr/bin/python3.6 && sleep 0" | 07:24 |
| zbr | returns 0, no output at all. | 07:24 |
| zbr | can it have anything to do with the warning above, "Failed to mount container centos8" | 07:26 |
| sshnaidm | no, it's fine | 07:27 |
| sshnaidm | enable -vvvvv | 07:27 |
| sshnaidm | 5 v's | 07:27 |
| sshnaidm | as I see you use vfs, maybe it's related.. | 07:28 |
| sshnaidm | with 5 v's you should see rc, stdout, stderr | 07:28 |
| zbr | updated https://gist.github.com/ssbarnea/57736b3fcd2841b9fbe909d2c6a43c8c with 5v's | 07:29 |
| zbr | it confirms what happens to me, no stdout/err output and exit code 0. | 07:29 |
| sshnaidm | idk, do you have a molecule file to reproduce? | 07:32 |
| zbr | yeah, i am now trying to do the same on a totally different system, with fedora 34, just to eliminate some questions. | 07:33 |
| zbr | sshnaidm: https://github.com/openstack/ansible-role-collect-logs/blob/master/roles/collect_logs/molecule/infrared/molecule.yml#L3 -- the only change was to switchi from docker to podman. | 07:36 |
| zbr | i get the same behavior on fedora, so is not a broken system or podman. | 07:36 |
| zbr | i think found it! | 07:37 |
| zbr | almost sure is https://github.com/openstack/ansible-role-collect-logs/blob/master/roles/collect_logs/molecule/infrared/molecule.yml#L25-L26 | 07:37 |
| zbr | that explains why so many people report this error, as this is a very popular ansible performance optimization | 07:38 |
| zbr | now I need to find a way to detect if pipelining is enabled at runtime, so I can fail before even trying to gather. | 07:40 |
| zbr | ansible did mention that is using pipelining but who reads every debug line ;) | 07:41 |
| sshnaidm | yes, we always said pipelining should be disabled in podman connection | 07:49 |
| sshnaidm | idk why to enable it in docker connection as well, tbh | 07:49 |
| zbr | probably is a leftover from when it was delegated, is not uncommon to see leftovers in molecule files. | 07:52 |
| zbr | now i am trying to find a ~smart way to detect this at runtime | 07:53 |
| zbr | i do not want to force people to disable pipeline but i need to detect if someone sets it wrong for podman containers to prevent that | 07:53 |
| sshnaidm | if [[ pipeline detected ]] then <fail with message> ? | 07:59 |
| sshnaidm | for podman driver | 07:59 |
| sshnaidm | zbr, any idea how to install additional RPMs in tox jobs? | 08:02 |
| sshnaidm | I need to install collections, but they're not python modules, I can't use requirements file | 08:02 |
| sshnaidm | otherwise linters don't understand "from ansible_collections import ..." | 08:03 |
| *** ykarel is now known as ykarel|lunch | 08:24 | |
| soniya|rover | marios|ruck, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks is failing on wallaby, is this a known issue? | 09:26 |
| marios|ruck | soniya|rover: link? | 09:28 |
| soniya|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/384226d/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 09:28 |
| marios|ruck | soniya|rover: did you check cix? sounds familiar but not sure | 09:28 |
| soniya|rover | marios|ruck, on cix we have different test i.e test_port_security_macspoofing_port varies the above one is test_connectivity_between_vms_on_different_networks | 09:29 |
| marios|ruck | soniya|rover: so i don't think we have enough for a bug yet because we only have one example | 09:30 |
| marios|ruck | soniya|rover: it fails twice in the last two runs, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby | 09:30 |
| soniya|rover | marios|ruck, two failures for today | 09:31 |
| marios|ruck | soniya|rover: the one you point to here is the latest, the one before it is not the sam e | 09:31 |
| marios|ruck | soniya|rover: all sorts failing there https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/5c177c6/logs/undercloud/var/log/tempest/stestr_results.html.gz | 09:31 |
| marios|ruck | soniya|rover: so you can run a testproject to get a test of periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby | 09:31 |
| marios|ruck | soniya|rover: if it reproduces there then we can say we have a bug | 09:31 |
| marios|ruck | soniya|rover: otherwise you can wait for wallaby to run again and see | 09:31 |
| marios|ruck | soniya|rover: make sense? | 09:32 |
| soniya|rover | marios|ruck, okay, thanks :) | 09:32 |
| *** ykarel|lunch is now known as ykarel | 09:51 | |
| soniya29|rover | marios|ruck, I am going out for half an hour | 10:54 |
| marios|ruck | soniya29|rover: thanks | 10:58 |
| frenzy_friday | Has anyone faced this: InsecureRequestWarning: Unverified HTTPS request is being made to host 'mirror.kna1.airship-citycloud.opendev.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings ? | 11:57 |
| frenzy_friday | ^ https://zuul.opendev.org/t/openstack/build/72738d1c0db8457c90a3c477812b1052 | 11:58 |
| marios|ruck | frenzy_friday: not come across that | 12:00 |
| chandankumar | rlandy: Hello, Does PSI is up now for running testproject? | 12:02 |
| rlandy | chandankumar: yep PSI is back | 12:02 |
| chandankumar | rlandy: cool, thanks :-) | 12:02 |
| rlandy | marios|ruck: soniya|rover: hey - saw your notes ... just trying to get downstream in shape | 12:03 |
| rlandy | ping if you need me to look upstream | 12:03 |
| marios|ruck | rlandy: ack | 12:03 |
| *** chem is now known as Guest2699 | 12:12 | |
| *** rlandy is now known as rlandy|ruck | 12:19 | |
| *** amoralej is now known as amoralej|lunch | 12:31 | |
| soniya|rover | rlandy|ruck, ack | 12:52 |
| sshnaidm | frenzy_friday, saw something similar with different proxy hosts, but seems like doesn't affect anything? | 13:00 |
| rlandy|ruck | chandankumar: zbr: akahat: scrum time | 13:01 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34334 | 13:07 |
| rlandy|ruck | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_727/787502/9/check/tox-ansible-test-sanity/72738d1/job-output.txt | 13:11 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34688 | 13:14 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34273 | 13:14 |
| * zbr still rebooting after an unexpected upgrade… | 13:16 | |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| marios|ruck | akahat: o/ hey can you pleas go make the latest_hashes 100 on the master promoter ? | 13:24 |
| marios|ruck | akahat: pretty please & cherries? | 13:24 |
| marios|ruck | akahat: let me check if i can login | 13:24 |
| akahat | marios|ruck, okay. | 13:25 |
| akahat | marios|ruck, are you able to login? | 13:28 |
| marios|ruck | akahat: are you doing that? vim is telling me there is a sswapfile already | 13:29 |
| marios|ruck | akahat: Swap file "ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/.master.yaml.swp | 13:29 |
| akahat | marios|ruck, yes.. please join tmux session | 13:29 |
| marios|ruck | " already exists! | 13:29 |
| marios|ruck | akahat: ah right thx sec | 13:29 |
| marios|ruck | akahat: thanks :D | 13:30 |
| rlandy|ruck | zbr: <rlandy|ruck> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_727/787502/9/check/tox-ansible-test-sanity/72738d1/job-output.txt | 13:30 |
| akahat | marios|ruck, restarting service. | 13:30 |
| akahat | marios|ruck, master is promoting :) | 13:32 |
| marios|ruck | akahat: thank you | 13:32 |
| marios|ruck | akahat: :D | 13:32 |
| rlandy|ruck | akahat: what number of hashes did you use? | 13:32 |
| marios|ruck | akahat: sorry for fighting you for the cursor on tmux ;) | 13:32 |
| rlandy|ruck | if that changed | 13:32 |
| marios|ruck | rlandy|ruck: 100 | 13:32 |
| rlandy|ruck | perfect | 13:32 |
| akahat | 100 | 13:32 |
| zbr | rlandy|ruck: sanity checks are not expected to pass in initial review, too many changes are needed for a single review. still, i need to fix the infrared issue before considering that ready for review. | 13:38 |
| rlandy|ruck | ok | 13:38 |
| sshnaidm | zbr, do you remember how to include "ensure-docker" in job definition? like roles: zuul: something../ensure-docker | 13:41 |
| zbr | sshnaidm: with some help from codesearch, i supposed is something like https://opendev.org/openstack/cinder/src/branch/master/.zuul.yaml#L153 | 13:44 |
| zbr | likele just `- zuul: zuul-jobs/ensure-docker` ? | 13:44 |
| sshnaidm | oh, forgot "opendev" | 13:45 |
| zbr | clearly we never used it like this, | 13:45 |
| sshnaidm | I'm trying to use ansible-test with --docker | 13:45 |
| sshnaidm | it should solve problem with packaging | 13:46 |
| sshnaidm | and it doesn't support podman yet | 13:46 |
| zbr | i only used it with --venv, w/o containers. | 13:46 |
| zbr | and it worked, mainly underlined a ton of issues within out code | 13:47 |
| zbr | probably you want to try "tox-ansible", it should make it much easier to run ansible-test. | 13:47 |
| sshnaidm | zbr, the problem is with ansible-test | 13:48 |
| *** pojadhav- is now known as pojadhav | 13:50 | |
| rlandy|ruck | marios|ruck: you mentioned a few promoter issues in the chat ... anything else needed for promotions? | 14:26 |
| marios|ruck | rlandy|ruck: i think we're good. akahat said master started promoting \o/ got some success with the others too victoria 9d53349f9c48ea9802b5f5f4ed74b243 https://review.rdoproject.org/r/c/testproject/+/34733 train 2155fc11a6d9458255c9c0f28833ea8d https://review.rdoproject.org/r/c/testproject/+/34742 should promote | 14:35 |
| marios|ruck | rlandy|ruck: not sure if we need to increase the fetched hashes to 100 for the other branches too probably should | 14:35 |
| marios|ruck | rlandy|ruck: on the other promoter | 14:35 |
| marios|ruck | rlandy|ruck: ussuri failed but it should run OK now after we remerged the skip https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/802874 (https://logserver.rdoproject.org/41/34741/1/check/periodic-tripleo-ci-centos-8-scenario004-standalone-ussuri/7238660/logs/undercloud/var/log/tempest/stestr_results.html.gz) rechecking that | 14:36 |
| marios|ruck | akahat: around? | 14:37 |
| akahat | marios|ruck, yes | 14:38 |
| akahat | marios|ruck, i'll update it on other promoter as well. | 14:38 |
| marios|ruck | akahat: hey can we do the same (100 hashes) on the other promoter for branches? | 14:38 |
| marios|ruck | akahat: right | 14:38 |
| marios|ruck | akahat: thank you is there tmux there too? | 14:39 |
| marios|ruck | akahat: am there | 14:40 |
| akahat | marios|ruck, yes. full screen please. :) | 14:40 |
| marios|ruck | akahat: ack | 14:40 |
| akahat | marios|ruck, can | 14:40 |
| marios|ruck | akahat: victoria promoting? | 14:40 |
| akahat | marios|ruck, can't restart it now.. | 14:41 |
| marios|ruck | akahat: yup | 14:41 |
| marios|ruck | akahat: thanks for checking! | 14:41 |
| akahat | yes. victoria is promoting | 14:41 |
| marios|ruck | akahat: i am wathcing it on tmux | 14:41 |
| marios|ruck | akahat: ok lets leave till tomorrow | 14:41 |
| marios|ruck | akahat: thank you amol! | 14:41 |
| akahat | marios|ruck, okay. i"ll update it tomorrow. | 14:41 |
| marios|ruck | akahat: ack | 14:41 |
| marios|ruck | rlandy|ruck: ^^ won't switch 100 for other branches now v is promoting, (maybe also T ) | 14:41 |
| marios|ruck | rlandy|ruck: bump tomorrow | 14:41 |
| rlandy|ruck | marios|ruck: ack ok | 14:43 |
| marios|ruck | rlandy|ruck: akahat: nice now i see master running container-push too for 6392edc0eaefb612986de9809a0195ba | 14:49 |
| marios|ruck | feels_good_man.jpg | 14:50 |
| akahat | \o/ | 14:51 |
| marios|ruck | train too :D | 14:54 |
| marios|ruck | https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5 train 2155fc11a6d9458255c9c0f28833ea8d https://review.rdoproject.org/r/c/testproject/+/34742 | 14:54 |
| zbr | sshnaidm: got some feedback from ansible core, podman collection must be sure that it has `has_pipelining = False` | 14:54 |
| sshnaidm | hmm.. and how to do it? | 14:56 |
| zbr | https://github.com/containers/ansible-podman-collections/blob/ba763e2988c4d3f94940695a5699f26fb94ed31f/plugins/connection/podman.py#L82 | 14:58 |
| zbr | apparently buildah has the same setting, not sure if it does suppor it. | 14:58 |
| *** ykarel is now known as ykarel|away | 15:50 | |
| marios|ruck | rlandy|ruck: please add to your reviews https://review.opendev.org/q/topic:upstream-periodic-run-once workflow if happy some have votes but no workflow | 15:51 |
| marios|ruck | zbr: chandankumar: please add to your review queue https://review.opendev.org/q/topic:upstream-periodic-run-once | 15:52 |
| rlandy|ruck | marios|ruck: ack ... marios - ok by me | 15:55 |
| rlandy|ruck | want me to vote or w+? | 15:55 |
| rlandy|ruck | both? | 15:55 |
| marios|ruck | rlandy|ruck: all of the above please | 15:55 |
| marios|ruck | rlandy|ruck: thank you | 15:55 |
| marios|ruck | chandankumar: zbr: rlandy|ruck: sshnaidm: please add this to your queue when you next have time https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34751 | 15:57 |
| rlandy|ruck | done | 15:57 |
| marios|ruck | rlandy|ruck: thanks | 15:58 |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-undercloud-upgrade-ussuri | 15:58 |
| rlandy|ruck | results not great | 15:58 |
| rlandy|ruck | did something change? | 15:58 |
| rlandy|ruck | one success | 15:58 |
| marios|ruck | rlandy|ruck: yes see https://bugs.launchpad.net/tripleo/+bug/1936825/comments/5 | 15:58 |
| marios|ruck | rlandy|ruck: the fix merged yesterday and latest periodic run didn't have it, but my testproject did | 15:59 |
| rlandy|ruck | marios|ruck: srry missed that | 15:59 |
| rlandy|ruck | voted | 15:59 |
| marios|ruck | rlandy|ruck: no problem | 15:59 |
| marios|ruck | rlandy|ruck: thank you | 15:59 |
| rlandy|ruck | k - getting | 15:59 |
| rlandy|ruck | there downstream | 15:59 |
| rlandy|ruck | one promotion on 16.2 | 15:59 |
| rlandy|ruck | still working on 17 | 15:59 |
| marios|ruck | rlandy|ruck: `nice | 15:59 |
| rlandy|ruck | still debug issues | 15:59 |
| marios|ruck | rlandy|ruck: so "its back" at least sounds like? | 16:00 |
| rlandy|ruck | but want to clear the decks before next program call | 16:00 |
| marios|ruck | rlandy|ruck: i mean psi | 16:00 |
| rlandy|ruck | container push on 16.2 is broken now though | 16:00 |
| marios|ruck | rlandy|ruck: starting shutdown in a few minutes | 16:04 |
| rlandy|ruck | marios|ruck: k - anything on the watch list? | 16:07 |
| marios|ruck | rlandy|ruck: not really looks like the promotion chasing was frutiful ... maybe checkout wallaby but no urgent can do tomorrow it is only 2 days old | 16:08 |
| marios|ruck | rlandy|ruck: nothing then ;) | 16:08 |
| marios|ruck | rlandy|ruck: hope it is a quiet day | 16:08 |
| rlandy|ruck | ok - sure | 16:08 |
| rlandy|ruck | marios|ruck: hang in there - tomorrow is friday | 16:08 |
| rlandy|ruck | and we get days off!! | 16:09 |
| marios|ruck | rlandy|ruck: yes :D | 16:09 |
| *** amoralej is now known as amoralej|off | 16:10 | |
| *** marios|ruck is now known as marios|out | 16:18 | |
| rlandy|ruck | akahat: hey | 17:41 |
| akahat | rlandy|ruck, hello | 17:42 |
| rlandy|ruck | http://10.0.148.74/promoter_logs/centos8_master.log | 17:42 |
| rlandy|ruck | looks like we have a successful promotion for 6392edc0eaefb612986de9809a0195ba | 17:42 |
| rlandy|ruck | but that's not the hash in https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 | 17:43 |
| rlandy|ruck | is it still busy promoting? | 17:43 |
| akahat | looking | 17:43 |
| akahat | rlandy|ruck, container promote role failed again for manifest push. | 17:46 |
| akahat | http://10.0.148.74/promoter_logs/centos8_master_2021-07-29T17:01.log | 17:46 |
| akahat | I'll disable manifest push. And re run it.. it will promote dlrn | 17:47 |
| rlandy|ruck | akahat: thank you | 17:47 |
| akahat | done. | 17:52 |
| akahat | https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 | 17:52 |
| rlandy|ruck | akahat: thank you, sir | 17:52 |
| rlandy|ruck | akahat: here's another mystery | 18:14 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-victoria/component/tripleo/ | 18:14 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-victoria/component/tripleo/current-tripleo/commit.yaml | 18:15 |
| rlandy|ruck | is marked 07/06 | 18:15 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-victoria/current-tripleo/ 07/29 | 18:15 |
| rlandy|ruck | maybe we need another promotion | 18:21 |
| akahat | rlandy|ruck, yes. current-tripleo promotion is not happend recently, https://trunk.rdoproject.org/centos8-victoria/component/tripleo/current-tripleo/?C=M;O=D | 18:24 |
| rlandy|ruck | k - rekciking failed jobs | 18:24 |
| *** sshnaidm is now known as sshnaidm|afk | 18:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!