| *** Goneri has quit IRC | 00:14 | |
| *** jmasud has joined #oooq | 03:48 | |
| *** jmasud has quit IRC | 04:06 | |
| *** ykarel|away is now known as ykarel | 04:15 | |
| *** ratailor has joined #oooq | 04:20 | |
| *** jmasud has joined #oooq | 04:40 | |
| *** skramaja has joined #oooq | 05:11 | |
| *** jmasud has quit IRC | 05:14 | |
| *** jmasud has joined #oooq | 05:16 | |
| *** jfrancoa has joined #oooq | 05:22 | |
| *** chem has quit IRC | 05:23 | |
| *** chem has joined #oooq | 05:24 | |
| *** udesale has joined #oooq | 05:41 | |
| *** ysandeep|away is now known as ysandeep | 05:42 | |
| *** marios has joined #oooq | 05:57 | |
| akahat|ruck | chandankumar, o/ | 06:10 |
|---|---|---|
| *** jmasud has quit IRC | 06:16 | |
| *** jmasud has joined #oooq | 06:18 | |
| *** bogdando has joined #oooq | 06:27 | |
| *** jmasud has quit IRC | 06:57 | |
| *** jmasud has joined #oooq | 06:59 | |
| *** amoralej|off is now known as amoralej | 07:20 | |
| marios | panda|ruck: akahat|ruck: o/ hey folks rocky/queens gate borked?both blocking there TripleO CI Scrum | 07:23 |
| marios | Thursday, 30 April⋅16:00 – 17:00 | 07:23 |
| marios | Weekly on Monday, Thursday, until 31 Dec 2020 | 07:23 |
| marios | meet.google.com/oiv-geho-mai | 07:23 |
| marios | panda|ruck: akahat|ruck: sorry, wrong paste ;) there https://review.opendev.org/#/c/722562/ | 07:24 |
| akahat|ruck | marios, hello. | 07:26 |
| akahat|ruck | marios, yes few jobs are still broken | 07:26 |
| akahat|ruck | BZ is already there for it:https://bugs.launchpad.net/tripleo/+bug/1875833 | 07:27 |
| openstack | Launchpad bug 1875833 in tripleo "The WebSocket timed out before the Workflow completed in rocky/stain jobs" [Critical,New] - Assigned to amolkahat (amolkahat) | 07:27 |
| *** tosky has joined #oooq | 07:28 | |
| marios | thanks akahat|ruck | 07:30 |
| *** matbu has quit IRC | 07:39 | |
| *** matbu has joined #oooq | 07:46 | |
| ykarel | akahat|ruck, i have triggered openstack-periodic-wednesday-weekend pipeline as puppet-pacemaker is available in consistent repo | 07:46 |
| ykarel | i see rocky job failed in -latest-released pipeline, didn't checked the reason | 07:47 |
| akahat|ruck | ykarel, great. :) | 07:47 |
| ykarel | can u check, if it's real failure | 07:47 |
| ykarel | if not a real failure, good to run a testproject change with rocky periodic jobs | 07:48 |
| akahat|ruck | ykarel, could you please pass the link | 07:48 |
| akahat|ruck | ? | 07:48 |
| ykarel | https://review.rdoproject.org/zuul/status | 07:48 |
| *** jmasud has quit IRC | 07:48 | |
| *** jmasud has joined #oooq | 07:49 | |
| akahat|ruck | ykarel, thanks :) | 07:50 |
| *** ratailor is now known as ratailor|lunch | 07:50 | |
| *** jpena|off is now known as jpena | 07:57 | |
| zbr | marios: i guess i could increase my productivity by starting the day very early, zuul queues are decent till ~noon GTM. | 07:58 |
| zbr | i managed to find an older CR that would make install-deps use venv under py3, which avoids virtualenv bugs: https://review.opendev.org/#/c/707215/8 | 08:02 |
| zbr | any reasons for not doing it? it would have saved us a good number of man-work-days with recent virtualenv issues | 08:03 |
| marios | zbr: since venv is the 'standard' way for py3 ... personally don't object to that... i see you have included a test so it will fall back to virtualenv if not available | 08:06 |
| zbr | in fact the implementation is doing the opposite | 08:07 |
| zbr | marios: haha! you did not read the code. that is what it does. | 08:07 |
| marios | zbr: if ! $(python_cmd) -m venv ${OPTS:-} ${OPT_WORKDIR}; then virtuaelenv... | 08:07 |
| marios | zbr: i see you have included a test so it will fall back to virtualenv if not available | 08:08 |
| marios | zbr: ? don't forr what you mean by 11:07 < zbr> marios: haha! you did not read the code. that is what it does. | 08:08 |
| *** ysandeep is now known as ysandeep|lunch | 08:08 | |
| zbr | marios: ahh, i was the one not reading carefully the ms :p | 08:09 |
| marios | zbr: ack | 08:09 |
| zbr | if that code would not work, we would se a lot of red from builds. | 08:10 |
| *** ccamacho has joined #oooq | 08:12 | |
| *** jbadiapa has joined #oooq | 08:14 | |
| *** ratailor|lunch is now known as ratailor | 08:30 | |
| *** ykarel is now known as ykarel|lunch | 08:39 | |
| *** yolanda has joined #oooq | 08:41 | |
| *** apetrich has joined #oooq | 08:59 | |
| chem | hey, so I'm in pain with https://review.opendev.org/#/c/721292/18 the molecule testing (docker) and role addition check are consistently failing | 08:59 |
| chem | akahat|ruck: ^ | 08:59 |
| chem | akahat|ruck: hey, what are my options here ? | 08:59 |
| chem | akahat|ruck: can we move them out of the way https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-8-role-addition | 09:01 |
| chem | and that https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-8-molecule-tripleo_redhat_enforce# | 09:02 |
| *** ysandeep|lunch is now known as ysandeep | 09:04 | |
| zbr | arxcruz: what is happening with https://review.opendev.org/#/c/681211/ ? | 09:06 |
| zbr | weshay|ruck: is https://review.opendev.org/#/c/712105/7 still needed? | 09:11 |
| zbr | marios: https://review.opendev.org/#/c/708380/ still need to be open or we can abandon? | 09:16 |
| zbr | never mind, looked at it and decided to abandon, i am sure you will agree. | 09:17 |
| akahat|ruck | chem, looking. | 09:18 |
| chem | akahat|ruck: thanks | 09:18 |
| marios | zbr: ack thanks agree | 09:20 |
| akahat|ruck | marios, zbr do you have an idea what's going on with tox ^^ | 09:22 |
| *** dtantsur|afk is now known as dtantsur | 09:23 | |
| zbr | akahat|ruck: i do not see any link to a tox job | 09:24 |
| *** derekh has joined #oooq | 09:25 | |
| marios | akahat|ruck: i don't but looking | 09:28 |
| marios | zbr: in there https://review.opendev.org/#/c/721292/18 | 09:28 |
| marios | module 'setuptools.build_meta' has no attribute '__legacy__ | 09:29 |
| zbr | that has nothing to do with tox | 09:31 |
| marios | akahat|ruck: zbr: maybe that https://github.com/pypa/setuptools/issues/1694#issuecomment-466023567 and https://github.com/cloudnull/ansible-tripleo_sdk/commit/0bdafbd21118540e970b74a34adb282feb11002d | 09:31 |
| zbr | it is a pip/setuptools issue | 09:31 |
| zbr | LMGFY sends me to https://github.com/pypa/setuptools/issues/1694 | 09:33 |
| zbr | i am inclined to believe the error has nothing to do with the tested change | 09:34 |
| marios | akahat|ruck: so if its blocking us then file a new bug for starters... looks like that change from cloudnull can provide a hint for the fix cc panda|ruck | 09:36 |
| akahat|ruck | marios, yes it's blocking us | 09:36 |
| akahat|ruck | chem, ^^ that might help. | 09:39 |
| akahat|ruck | marios, zbr thanks.. I'll talk to cloudnull. :) | 09:39 |
| akahat|ruck | marios, cloudnull is he online now? | 09:40 |
| zbr | akahat|ruck: i bet the reason is sitepackages = True | 09:41 |
| chem | akahat|ruck: not sure what could help, should I fill the lp ? | 09:41 |
| akahat|ruck | chem, it's blocking us from merging then yes.. this will help. | 09:43 |
| akahat|ruck | zbr, did not got your point. :( | 09:46 |
| zbr | https://review.opendev.org/#/c/724627/ | 09:48 |
| akahat|ruck | zbr, oh.. got it. :) | 09:50 |
| arxcruz | zbr: nothing, it just not priority right now | 09:52 |
| zbr | arxcruz: mark it somehow that it does not need attention, -W or a "wip" prefix. | 09:53 |
| marios | zbr: arxcruz: chem: i think we just need to disable pep517? | 09:53 |
| marios | zbr: --no-use-pep517 but i can't find the right pip task to add it to | 09:53 |
| marios | for the fix i mean. | 09:53 |
| marios | well 'fix' at least workaround | 09:53 |
| zbr | marios: hah, pep517 is not something you can disable | 09:53 |
| marios | zbr: well apparently there is a --no-use-pep517 for pip extra_args | 09:54 |
| zbr | or remove it, the future is that everyone will switch to pep517, and drop support for old setup.py | 09:54 |
| zbr | is not something you can afford to ignore | 09:55 |
| marios | zbr: well you can see here it is where the error is coming from | 09:55 |
| marios | zbr: 2020-04-30 03:41:04.887579 | centos-8 | File "/home/zuul/src/opendev.org/openstack/tripleo-ansible/.tox/role-addition/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 76, in _build_backend | 09:55 |
| marios | 2020-04-30 03:41:04.887592 | centos-8 | AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__' | 09:55 |
| marios | https://f90bbe801532b1321206-1f2089e8a25dfa4acbcd47153d041690.ssl.cf2.rackcdn.com/721292/18/check/tripleo-ansible-centos-8-role-addition/f39b625/job-output.txt | 09:55 |
| marios | akahat|ruck: did you file a LP ? | 09:55 |
| zbr | please file a bug and add me to it, if that comes from one of our repos/packages i could help, i happen to know the pep517 subject | 09:57 |
| chem | akahat|ruck: marios https://bugs.launchpad.net/tripleo/+bug/1876073 | 09:57 |
| openstack | Launchpad bug 1876073 in tripleo "Zuul CI is giving false positive on role-addition and molecule consistently " [Critical,New] | 09:57 |
| marios | thanks chem ;) | 09:57 |
| akahat|ruck | chem, thank you :) | 09:57 |
| akahat|ruck | panda|ruck, ^^ | 09:58 |
| chem | akahat|ruck: marios it's only basic reporting, I never run molecule and everything in that job is from the default templates | 09:58 |
| chem | akahat|ruck: marios beside the 15line of ansible I needed | 09:58 |
| chem | akahat|ruck: marios On my defense, docker doensn't run out of the box on fedora 31 ... podman is coming :) | 09:59 |
| marios | chem: akahat|ruck: panda|ruck: adding alert and critical to that one | 10:00 |
| zbr | it works even on fedora 32, but with some hacking needed, any redhat OS after centos-7 needed hacks to make it work. | 10:00 |
| marios | zbr: akahat|ruck: panda|ruck: looks like it is coming from the invocation there https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml ie zuul-jobs so we can't workaround it i think | 10:15 |
| *** jmasud has quit IRC | 10:16 | |
| *** jaosorior has quit IRC | 10:17 | |
| zbr | i do not see a bug there, what is wrong on siblings? | 10:18 |
| marios | zbr: no that is where the invocation starts i mean and where we'd have to disable pep517 | 10:18 |
| marios | zbr: just from going on the logs trace | 10:18 |
| marios | zbr: https://bugs.launchpad.net/tripleo/+bug/1876073/comments/2 | 10:19 |
| openstack | Launchpad bug 1876073 in tripleo "Zuul CI is giving false positive on role-addition and molecule consistently " [Critical,Triaged] | 10:19 |
| zbr | marios: here is my proof: https://review.opendev.org/#/c/724627/ | 10:23 |
| *** ykarel|lunch is now known as ykarel | 10:25 | |
| *** derekh has quit IRC | 10:41 | |
| *** derekh has joined #oooq | 10:42 | |
| akahat|ruck | zbr, your patch worked: https://review.opendev.org/#/c/724627/ | 10:44 |
| zbr | not surprised, there is another failed build, which is supposed has similar reason. | 10:45 |
| zbr | akahat|ruck: can you take care of making a fix on this and ping me if getting stuck? | 10:46 |
| akahat|ruck | zbr, yeah sure. | 10:47 |
| akahat|ruck | arxcruz, hey o/ | 10:59 |
| *** udesale_ has joined #oooq | 11:12 | |
| *** udesale has quit IRC | 11:15 | |
| *** ysandeep is now known as ysandeep|brb | 11:18 | |
| arxcruz | akahat|ruck: hey, ho, let's go | 11:27 |
| arxcruz | akahat|ruck: sorry, was in a meeting, what's up ? | 11:27 |
| akahat|ruck | arxcruz, it's okay. I have few jobs from openstack-periodic-wednesday-weekend pipeline. | 11:28 |
| akahat|ruck | those jobs are breaking on one specific test. | 11:29 |
| akahat|ruck | arxcruz, this is test: - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps | 11:29 |
| akahat|ruck | all of them are queens jobs. | 11:29 |
| akahat|ruck | arxcruz, - https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/998a119/logs/undercloud/home/zuul/tempest/tempest.html.gz | 11:29 |
| akahat|ruck | ust nee your help to verify this.. is it really issue ? | 11:30 |
| arxcruz | akahat|ruck: yes, it is, I sugest to open a promotion blocker, so the bug will be added on the cix board, and the network team can take a look | 11:32 |
| arxcruz | the vm is up, running, but the network is failing | 11:32 |
| akahat|ruck | arxcruz, okay. I'll open bug. | 11:33 |
| panda|ruck | akahat|ruck: they're forming a straight line | 11:34 |
| panda|ruck | arxcruz: they're forming a straight line | 11:34 |
| *** pojadhav is now known as pojadhav|brb | 11:34 | |
| arxcruz | panda|ruck: didn't get it... | 11:35 |
| akahat|ruck | panda|ruck, what are you trying to say? | 11:35 |
| *** jpena is now known as jpena|lunch | 11:36 | |
| panda|ruck | arxcruz: you can't start with "hey, oh, let's go" without knowing the rest of the lyrics. | 11:39 |
| weshay|ruck | bhagyashris, http://dashboard-ci.tripleo.org/d/Z_UNB29Wz/third-party-dependency-check?orgId=1 | 11:39 |
| arxcruz | panda|ruck: lol, i know the music, but i wasn't connectiong it, sorry, i still in my fist cup of coffee :D | 11:40 |
| arxcruz | even though it's late here already | 11:40 |
| panda|ruck | arxcruz: I hope you're on you are in your third can of beer | 11:41 |
| arxcruz | panda|ruck: Last time I drink was when marios set a meeting | 11:42 |
| marios | weshay|ruck: fyi tomorrow i am out cos may 1st public holiday so taking the day | 11:43 |
| marios | weshay|ruck: (noted during planning, i think quite a few folks are out tomorrow ) | 11:44 |
| akahat|ruck | arxcruz, there is on more thing: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/d6660dd/logs/undercloud/home/zuul/tempest/tempest.html.gz | 11:47 |
| akahat|ruck | looks like simple :P | 11:47 |
| arxcruz | akahat|ruck: seems to be the same issue, but without the console log output | 11:48 |
| akahat|ruck | arxcruz, it's RHEL8 job | 11:49 |
| akahat|ruck | arxcruz, sorry.. centos8 | 11:49 |
| arxcruz | akahat|ruck: probably the console output option in temepst is off | 11:49 |
| arxcruz | tempest* | 11:49 |
| arxcruz | but pretty much the same thing | 11:49 |
| weshay|ruck | bhagyashris, https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit | 11:49 |
| weshay|ruck | bhagyashris, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/development_script.sh | 11:49 |
| akahat|ruck | arxcruz, okay. | 11:50 |
| arxcruz | akahat|ruck: oh, wait | 11:50 |
| arxcruz | akahat|ruck: it's a different test | 11:50 |
| arxcruz | well, still, it tries to ssh to the vm, and get a timeout | 11:50 |
| arxcruz | This might be impact other tests as well | 11:50 |
| akahat|ruck | arxcruz, yes.. it says password is None. | 11:50 |
| akahat|ruck | arxcruz, yes. | 11:50 |
| rfolco | ykarel, hi o/ | 11:51 |
| ykarel | rfolco, hi, was about to ping you :) | 11:51 |
| rfolco | ykarel, link for the demo: https://meet.google.com/fgn-rnkp-sqt | 11:51 |
| ykarel | rfolco, ack i joined | 11:52 |
| rfolco | ykarel, if you have any slides or material, please paste here or add to the training page https://hackmd.io/OXNWWIShSBaPXPC8d7kFgQ | 11:53 |
| ykarel | adding | 11:54 |
| *** ysandeep|brb is now known as ysandeep | 11:54 | |
| rfolco | akahat|ruck, bhagyashris pojadhav|brb soniya29 ysandeep >> ping virtual training session - https://meet.google.com/fgn-rnkp-sqt | 11:55 |
| rfolco | please join 2 min earlier so we can get started on time | 11:55 |
| arxcruz | akahat|ruck: also, there's no errors in nova log | 11:58 |
| arxcruz | i would sugest you to create a promotion blocker to discuss this on cix call | 11:58 |
| *** pojadhav|brb is now known as pojadhav | 11:59 | |
| akahat|ruck | arxcruz, for this centos8: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops | 12:00 |
| akahat|ruck | ? | 12:00 |
| *** ratailor has quit IRC | 12:10 | |
| *** rlandy has joined #oooq | 12:22 | |
| rlandy | pojadhav: hi there | 12:25 |
| pojadhav | rlandy, in training | 12:25 |
| rlandy | pojadhav: can we meet after scrum? | 12:26 |
| pojadhav | rlandy, yup | 12:26 |
| weshay|ruck | rfolco, rec | 12:29 |
| weshay|ruck | is that on? | 12:29 |
| weshay|ruck | can't tell | 12:29 |
| weshay|ruck | :) | 12:29 |
| rfolco | weshay|ruck, recording is on | 12:29 |
| rfolco | :) | 12:29 |
| *** derekh has quit IRC | 12:32 | |
| *** amoralej is now known as amoralej|lunch | 12:33 | |
| weshay|ruck | panda|ruck, akahat|ruck looking at queens failures.. | 12:36 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/5449d05/logs/tempest.html.gz | 12:36 |
| weshay|ruck | probably going to have to bug / skip | 12:36 |
| weshay|ruck | queens is fooked | 12:36 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/998a119/logs/tempest.html.gz | 12:36 |
| rfolco | ykarel, don't know if you can hear us, there is one more question from soniya29 | 12:37 |
| rfolco | ykarel, okay, so why do we even need to add packages for non openstack dependency? | 12:37 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/2d7d8bd/logs/tempest.html.gz | 12:37 |
| ykarel | rfolco, i can't hear right now | 12:37 |
| akahat|ruck | weshay|ruck, alredy file for network error: https://bugs.launchpad.net/tripleo/+bug/1876087 | 12:37 |
| openstack | Launchpad bug 1876087 in tripleo "Network tests are failing on queens jobs" [Critical,Triaged] - Assigned to amolkahat (amolkahat) | 12:37 |
| akahat|ruck | weshay|ruck, will file for the volume_boot_pattern. | 12:38 |
| weshay|ruck | akahat|ruck, k.. updated the bug name https://bugs.launchpad.net/tripleo/+bug/1876087 | 12:38 |
| openstack | Launchpad bug 1876087 in tripleo "Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout" [Critical,Triaged] - Assigned to amolkahat (amolkahat) | 12:38 |
| *** derekh has joined #oooq | 12:38 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| weshay|ruck | akahat|ruck++ | 12:38 |
| arxcruz | ykarel: we can't hear you | 12:39 |
| weshay|ruck | ykarel, question in chat... re-review current -> consistent | 12:39 |
| arxcruz | akahat|ruck: weshay|ruck both octavia and this are the same issue | 12:39 |
| arxcruz | ssh timeout | 12:39 |
| weshay|ruck | arxcruz, ya.. we're talking about queens though | 12:39 |
| arxcruz | the message are different because they are different tests, but the root cause seems to be equals | 12:39 |
| arxcruz | ok | 12:40 |
| weshay|ruck | arxcruz, will need ur help in queens as most issues are in tempest atm soniya29 ^ | 12:40 |
| weshay|ruck | arxcruz, octavia is train+ no? | 12:40 |
| weshay|ruck | I'm not clear on the connection you are drawing other than the error message is the same.. but it's generic.. right? | 12:41 |
| weshay|ruck | or am I misunderstanding | 12:41 |
| weshay|ruck | akahat|ruck, add these failed tempest tests to the skip list | 12:41 |
| arxcruz | weshay|ruck: i check the logs, haven't see any error on neutron side | 12:41 |
| arxcruz | the message are different because it uses different methods to ssh into the machine, but both are timeing out | 12:42 |
| weshay|ruck | akahat|ruck, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_queens.yml | 12:42 |
| arxcruz | ssh into the vm* | 12:42 |
| weshay|ruck | arxcruz++ please assist akahat|ruck | 12:42 |
| arxcruz | weshay|ruck: akahat|ruck oh, found something in openvswitch... | 12:43 |
| arxcruz | 2020-04-30 10:06:37.256 80095 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory | 12:43 |
| akahat|ruck | arxcruz, those packages are missing again? | 12:44 |
| arxcruz | https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queens/998a119/logs/subnode-1/var/log/containers/neutron/openvswitch-agent.log.txt.gz | 12:44 |
| arxcruz | akahat|ruck: i remember i saw these errors before... | 12:44 |
| akahat|ruck | those packages did not get pulled as dependency for neutron? | 12:44 |
| akahat|ruck | s/neutron/openvswith/ | 12:45 |
| *** dpawlik has joined #oooq | 12:46 | |
| *** dpawlik has quit IRC | 12:46 | |
| *** dpawlik has joined #oooq | 12:46 | |
| chem | marios: I'm wondering if the last comment in the lp means that something must be changed in the review ? | 12:49 |
| weshay|ruck | akahat|ruck, panda|ruck \0/ https://review.rdoproject.org/zuul/build/521209cf497444a98144fd0eca84c9b3 | 12:50 |
| chem | marios: basically, we are downporting this today ... so if it needs another round, the earlier I know the better :) | 12:50 |
| weshay|ruck | chem, the job should be marked nv imho | 12:53 |
| weshay|ruck | panda|ruck, can you review train for promotion | 12:53 |
| weshay|ruck | panda|ruck, let's pull the trigger it to fix pacemaker | 12:54 |
| marios | chem: looking | 12:54 |
| weshay|ruck | zbr, marios mark it non-voting w/ a bug included in the patch | 12:54 |
| marios | akahat|ruck: zbr: didn't zbr fix that? | 12:55 |
| zbr | akahat|ruck: marios: wait, cloudnull is working on a fix | 12:55 |
| marios | https://review.opendev.org/#/c/724627/ with that | 12:55 |
| marios | chem: sorry i thought it was fixed cos i saw that discussion earlier here ^^^ | 12:55 |
| marios | panda|ruck: akahat|ruck: so even if cloudnull is working on a fix we can mark non voting to unblock chem for now | 12:57 |
| akahat|ruck | marios, yes. i think we can do that as quick fix. | 12:57 |
| weshay|ruck | panda|ruck, ? | 12:57 |
| weshay|ruck | you with me? | 12:57 |
| marios | akahat|ruck: are you on that or would you like me to? | 12:58 |
| akahat|ruck | marios, could you please.. I'm busy with some other stuff. | 12:58 |
| marios | akahat|ruck: k doing | 12:59 |
| akahat|ruck | marios, thanks | 13:00 |
| panda|ruck | weshay|ruck: I am now | 13:02 |
| weshay|ruck | panda|ruck, see what you think about train.. periodic results | 13:02 |
| rfolco | scrum is delayed a few min | 13:03 |
| panda|ruck | weshay|ruck: looking | 13:05 |
| rfolco | ykarel, you are muted | 13:06 |
| ysandeep | rfolco, do you have the hackmd link handy where yatin have created for demo. | 13:09 |
| pojadhav | ykarel, thanks for the session :) | 13:10 |
| ykarel | ysandeep, it's in presentation also, u can check https://hackmd.io/lzjXK_FmQfeL7Wr_J-3V9A | 13:10 |
| rfolco | ysandeep, https://hackmd.io/OXNWWIShSBaPXPC8d7kFgQ click on ykarel's presentation | 13:10 |
| rfolco | scrum time | 13:10 |
| ysandeep | ykarel, thank you for the session, It was very informative o/ | 13:10 |
| ysandeep | ykarel++ | 13:10 |
| ykarel | please go through exercises, it shouldn't take much time, but will help to understand better :) | 13:11 |
| ysandeep | rfolco, thanks! | 13:11 |
| pojadhav | ykarel, sure | 13:11 |
| rfolco | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 13:12 |
| rfolco | ping scrum | 13:12 |
| rfolco | https://meet.google.com/oiv-geho-mai?authuser=1 | 13:12 |
| weshay|ruck | in another mtg atm | 13:12 |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| panda|ruck | weshay|ruck: I think we are good to go on train, want me to propose the waive | 13:15 |
| panda|ruck | ? | 13:15 |
| weshay|ruck | panda|ruck, yes.. please propose a waive :) | 13:17 |
| weshay|ruck | thank you | 13:17 |
| weshay|ruck | I'll review and merge :) | 13:18 |
| *** amoralej|lunch is now known as amoralej | 13:19 | |
| *** skramaja has quit IRC | 13:22 | |
| *** skramaja has joined #oooq | 13:23 | |
| *** TrevorV has joined #oooq | 13:23 | |
| panda|ruck | weshay|ruck: that job is not in promotion criteria, so train should have promoted by now. | 13:27 |
| marios | 10:27 < akahat|ruck> BZ is already there for it: https://bugs.launchpad.net/tripleo/+bug/1875833 | 13:30 |
| openstack | Launchpad bug 1875833 in tripleo "The WebSocket timed out before the Workflow completed in rocky/stain jobs" [Critical,New] - Assigned to amolkahat (amolkahat) | 13:30 |
| marios | rfolco: ^ | 13:30 |
| panda|ruck | weshay|ruck: but it didn't | 13:34 |
| marios | arxcruz: 10:27 < akahat|ruck> BZ is already there for it: https://bugs.launchpad.net/tripleo/+bug/1875833 | 13:47 |
| openstack | Launchpad bug 1875833 in tripleo "The WebSocket timed out before the Workflow completed in rocky/stain jobs" [Critical,New] - Assigned to amolkahat (amolkahat) | 13:47 |
| arxcruz | marios: efcharistó | 13:48 |
| marios | arxcruz: bitteschon | 13:51 |
| *** ysandeep is now known as ysandeep|afk | 13:56 | |
| *** ykarel|afk is now known as ykarel | 14:09 | |
| panda|ruck | weshay|ruck: I was mistaken, so the fs039 failed last, but a bunch of other jobs failed 1 hour before. fs002 failed, so we can't promote without overcloud images.. | 14:09 |
| * weshay|ruck looks | 14:10 | |
| *** Goneri has joined #oooq | 14:10 | |
| weshay|ruck | panda|ruck, ah.. the current one is passing https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload/57f75bd/ | 14:11 |
| weshay|ruck | so perhaps hold for that one | 14:11 |
| panda|ruck | weshay|ruck: yep | 14:11 |
| weshay|ruck | panda|ruck, and fs20 passed | 14:11 |
| weshay|ruck | panda|ruck, fs001/35 also passing | 14:12 |
| panda|ruck | weshay|ruck: ah ye the current one looks good | 14:12 |
| weshay|ruck | might promote on it's own | 14:12 |
| weshay|ruck | soniya29, arxcruz fyi.. https://trello.com/c/niYGK63i/1475-cixlp1876087tripleociproa-queens-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-timeout | 14:16 |
| arxcruz | weshay|ruck: I saw it | 14:17 |
| weshay|ruck | danka | 14:17 |
| soniya29 | weshay|ruck, I will have a look over it | 14:19 |
| *** dtantsur has quit IRC | 14:24 | |
| weshay|ruck | thanks! | 14:24 |
| rfolco | akahat|ruck, do we have a fix for https://bugs.launchpad.net/tripleo/+bug/1875833 already? | 14:26 |
| openstack | Launchpad bug 1875833 in tripleo "The WebSocket timed out before the Workflow completed in rocky/stain jobs" [Critical,New] - Assigned to amolkahat (amolkahat) | 14:26 |
| akahat|ruck | rfolco, no not yet. | 14:26 |
| rfolco | workarounds? anything? marios panda|ruck ^ | 14:27 |
| weshay|ruck | akahat|ruck, panda|ruck train is promoting | 14:27 |
| marios | rfolco: was not involved there at all i just got the bug from seeing those jobs fail on a few reviews this morning | 14:27 |
| rfolco | marios, ack | 14:28 |
| marios | rlandy: so do you think i should file a new BZ for https://sf.hosted.upshift.rdu2.redhat.com/logs/66/198466/1/check/periodic-tripleo-build-containers-rhel-8-master/845202a/logs/containers-failed-to-build.log | 14:33 |
| rlandy | arxcruz: tls and tempest ... I see in validate tempest there is an option to work with tls - is there something similar in os_tempest? | 14:33 |
| marios | rlandy: maybe just one for all of them ? | 14:33 |
| *** dtantsur has joined #oooq | 14:33 | |
| rlandy | marios: there are a few | 14:33 |
| marios | rlandy: yeah its like 4 | 14:33 |
| rlandy | marios: one per BZ is probably clearer | 14:33 |
| rlandy | if you have the time | 14:33 |
| marios | rlandy: ack | 14:34 |
| rlandy | marios: wrt fs039, are you running os_tempest there? | 14:34 |
| marios | rlandy: yeah | 14:34 |
| marios | rlandy: the problem right now though isn't tempest there its that https://bugs.launchpad.net/tripleo/+bug/1875353 which is a race | 14:34 |
| openstack | Launchpad bug 1875353 in tripleo "(inconsistent/race) periodic centos-8-ovb featureset039-master fails overcloud ipa-client install No such file or directory: /var/lib/ipa-client/sysrestore" [High,Triaged] | 14:34 |
| rlandy | marios: are you adding anything special? | 14:34 |
| marios | rlandy: i.e. fails on deployment | 14:35 |
| rlandy | marios: ack - but standalone ipa has a tempest failure ... TASK [os_tempest : Ping router ip address] | 14:35 |
| rlandy | fails | 14:35 |
| rlandy | and I wanted to know if I needed to add another option to deal with tls | 14:35 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/98/195798/5/check/tripleo-ci-centos-8-standalone-on-multinode-ipa-internal/a66d9a5/job-output.txt | 14:36 |
| rlandy | marios: arxcruz: ^^? | 14:36 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/run-tempest.sh.j2#L94 | 14:36 |
| zbr | rlandy: marios see https://review.opendev.org/#/c/724664/ | 14:36 |
| marios | rlandy: i updated fs39 to use the os-tempest stuff https://github.com/openstack/tripleo-quickstart/blob/701271b11c8f793600726434b168f3cea44feb7b/config/general_config/featureset039.yml#L140 | 14:36 |
| zbr | panda|ruck: ^ | 14:37 |
| marios | rlandy: ack hadn't seen that one in centos8/fs39 i.e. the ping fail from os-tempest | 14:37 |
| rlandy | marios: ok - and that's working fine for you | 14:38 |
| marios | rlandy: well when deployment passes then yeah we get green job with tempest | 14:38 |
| rlandy | ok | 14:38 |
| marios | rlandy: but some kind of race cos it is inconsistent | 14:38 |
| rlandy | I'll look at that later | 14:38 |
| rlandy | arxcruz: what does tempest_os_cloud refer to? | 14:44 |
| arxcruz | rlandy: in which context? | 14:51 |
| arxcruz | from the name, i believe it's the oscloud name for the yaml file containing the credentials | 14:52 |
| rlandy | arxcruz: in creating projects ... https://github.com/openstack/openstack-ansible-os_tempest/blob/ad4841f70a0ad4ed8090fc3f4c83812204cb59c0/tasks/tempest_resources.yml#L83 | 14:52 |
| rlandy | hmmm ... where do I find that? | 14:53 |
| marios | rlandy: k going to put ceilo-base and swift-proxy-server together cos they have the same dependency missing | 14:53 |
| rlandy | ok | 14:53 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/98/195798/5/check/tripleo-ci-centos-8-standalone-on-multinode-ipa-internal/a66d9a5/logs/undercloud/home/zuul/.config/openstack/clouds.yaml | 14:54 |
| rlandy | arxcruz: ^^ that? | 14:54 |
| rlandy | arxcruz: what I think I need is something like: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/run-tempest.sh.j2#L94 | 14:55 |
| arxcruz | rlandy: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L20 | 14:56 |
| arxcruz | basically, it's the name of the credential in the clouds.yaml | 14:56 |
| rlandy | arxcruz: ok - that seems fine | 14:57 |
| rlandy | arxcruz: just trying to figure out why it can't ping router | 14:57 |
| *** bogdando has quit IRC | 14:59 | |
| rlandy | marios: sorry - another tempest question ... | 15:11 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset039.yml - the tempest setting you added here ... | 15:11 |
| rlandy | they are not in fs052 | 15:11 |
| rlandy | I think those settings are elsewhere? | 15:12 |
| marios | rlandy: filed filed new https://bugzilla.redhat.com/show_bug.cgi?id=1829918 https://bugzilla.redhat.com/show_bug.cgi?id=1829921 https://bugzilla.redhat.com/show_bug.cgi?id=1829924 https://bugzilla.redhat.com/show_bug.cgi?id=1829927 | 15:12 |
| openstack | bugzilla.redhat.com bug 1829918 in releng "Cannot build aodh-base container for rhel8-osp17 - missing python3-ceilometerclient" [Unspecified,New] - Assigned to rhos-maint | 15:12 |
| openstack | bugzilla.redhat.com bug 1829921 in releng "Cannot build designate-base container for rhel8-osp17 - missing python3-edgegrid" [Unspecified,New] - Assigned to rhos-maint | 15:12 |
| openstack | bugzilla.redhat.com bug 1829924 in releng "Cannot build octavia-base container for rhel8-osp17 - missing python3-tenacity" [Unspecified,New] - Assigned to rhos-maint | 15:12 |
| openstack | bugzilla.redhat.com bug 1829927 in releng "Cannot build ceilometer-base swift-proxy-server container for rhel8-osp17 - missing python3-tenacity" [Unspecified,New] - Assigned to rhos-maint | 15:12 |
| marios | so that was fun | 15:12 |
| marios | rlandy: added to https://tree.taiga.io/project/tripleo-ci-board/task/1656 & back to in progress | 15:12 |
| marios | i'll refrain from sending the cix mails though unless lon tells us he need them? wdyt | 15:13 |
| marios | do we need those on the trello board is the question | 15:13 |
| marios | rlandy: reading back on fs52 | 15:13 |
| marios | rlandy: 18:11 < rlandy> they are not in fs052 | 15:13 |
| marios | rlandy: not sure what you mean? | 15:14 |
| marios | rlandy: the fs39 things are not in fs 52 ? | 15:14 |
| rlandy | marios: thanks re: BZ's ... I'll chat with lon as soon as I am done debugging tempest here | 15:14 |
| marios | rlandy: i'm out tomorrow its a public holiday here | 15:16 |
| rlandy | marios: ah - May 1st - labor day everywhere expect here | 15:17 |
| panda|ruck | almost everywhere. | 15:30 |
| marios | rlandy: and then on monday we get 'upgraded' we get 3 sms per day \o/ for outing :D - joking aside its progress... stage 1 of lifting restrictions some shops will open etc | 15:31 |
| rlandy | marios: enjoy - go do three fun things | 15:31 |
| *** skramaja has quit IRC | 15:32 | |
| *** matbu has quit IRC | 15:34 | |
| rfolco | akahat|ruck, panda|ruck: this was the last time tripleo-ci-centos-7-containers-multinode-rocky reported success: 2020-04-27T16:17:31 | 15:54 |
| *** marios is now known as marios|out | 16:09 | |
| weshay|ruck | panda|ruck, akahat|ruck before you guys go.. do we have patches up to the skip list? | 16:14 |
| *** matbu has joined #oooq | 16:20 | |
| *** jmasud has joined #oooq | 16:24 | |
| *** marios|out has quit IRC | 16:30 | |
| *** ykarel is now known as ykarel|afk | 16:31 | |
| rlandy | weshay|ruck: hey ... | 16:33 |
| rlandy | weshay|ruck: we hit build dependency failures again in downstream containers per the BZs marios added above | 16:33 |
| rlandy | weshay|ruck: chatted with lon about fixing those missing deps - some of which we need to wait until monday to check with tvignaud | 16:34 |
| weshay|ruck | fun | 16:34 |
| rlandy | weshay|ruck: what do you think about adding those failing containers to the exclude list and just pushing the ones that succeed | 16:34 |
| weshay|ruck | rlandy, any luck w/ pushing? | 16:35 |
| rlandy | so that we can test push? | 16:35 |
| weshay|ruck | rlandy, ya.. that's fine as a temp | 16:35 |
| weshay|ruck | +1 | 16:35 |
| rlandy | weshay|ruck: you don't get to push if build fails | 16:35 |
| weshay|ruck | ya.. I know unless you try it manually | 16:35 |
| weshay|ruck | :) | 16:35 |
| rlandy | weshay|ruck: alter, if easy enough, I'll patch openstack-tripleo-common to exclude those deps | 16:35 |
| rlandy | but as lon said, they may actually get added | 16:36 |
| rlandy | chicken-egg problem | 16:36 |
| *** ykarel|afk is now known as ykarel|away | 16:36 | |
| weshay|ruck | zbr, | 16:38 |
| weshay|ruck | molecule-container-push-delegated-centos-7FAILURE in 4m 08s | 16:38 |
| weshay|ruck | molecule-tripleo-common-delegated-centos-7FAILURE in 3m 23s | 16:38 |
| weshay|ruck | molecule-delegated-promote-images-delegated-centos-7FAILURE in 2m 00s | 16:38 |
| weshay|ruck | those have been red for several days now | 16:38 |
| weshay|ruck | panda|ruck, ^ | 16:39 |
| *** udesale_ has quit IRC | 16:39 | |
| weshay|ruck | rlandy, https://review.rdoproject.org/r/#/c/27117/ | 16:42 |
| rlandy | fine by me | 16:43 |
| weshay|ruck | https://review.opendev.org/724703 https://review.opendev.org/724704 <--- keep an eye on | 16:45 |
| zbr | going to have a brief look now | 16:50 |
| panda|ruck | zbr: it's clearer in this job https://logserver.rdoproject.org/22/27022/1/check/molecule-delegated-promote-images-delegated-centos-7/da19134/job-output.txt | 17:04 |
| zbr | panda|ruck: yep, remove the pinning of molecule, latest version does not have this bug (caused by a bad release of "sh") | 17:05 |
| zbr | use >3.0,<3.1 | 17:05 |
| zbr | 3.0.4 fixed that | 17:05 |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| zbr | i already mentioned that twice yesterday | 17:06 |
| panda|ruck | mentioned to whom ? | 17:08 |
| panda|ruck | zbr: ^ ? | 17:08 |
| zbr | ahh, I forgot that ruck does not join scrums | 17:20 |
| *** chem has quit IRC | 17:22 | |
| *** jpena is now known as jpena|off | 17:24 | |
| zbr | panda|ruck: see https://pypi.org/project/sh/#history -- for molecule 2.x you must manually add <1.13 | 17:24 |
| *** chem has joined #oooq | 17:24 | |
| *** amoralej is now known as amoralej|off | 17:36 | |
| *** dpawlik has quit IRC | 17:38 | |
| *** jmasud has quit IRC | 17:43 | |
| *** jbadiapa has quit IRC | 18:03 | |
| *** ccamacho has quit IRC | 18:08 | |
| *** jmasud has joined #oooq | 18:11 | |
| zbr | weshay|ruck: rlandy fix for promoter https://review.rdoproject.org/r/#/c/27118/ (passed) | 18:14 |
| *** jmasud has quit IRC | 18:17 | |
| *** jmasud has joined #oooq | 18:18 | |
| rfolco | akahat|ruck, panda|ruck: filed this bug as it seems to be different issue: https://bugs.launchpad.net/tripleo/+bug/1876160 | 18:27 |
| openstack | Launchpad bug 1876160 in tripleo "pacemaker: Failed to call refresh on containers-multinode-{rocky,queens}" [Critical,Triaged] | 18:27 |
| weshay|ruck | zbr, thanks.. move to nv abandonded 27118 wf | 18:49 |
| weshay|ruck | rfolco, what are you hitting? | 18:50 |
| rfolco | weshay|ruck, bug above is blocking release file patch | 18:50 |
| weshay|ruck | rfolco, dupe | 18:51 |
| rfolco | weshay|ruck, https://review.opendev.org/#/c/721842 | 18:51 |
| rfolco | dupe of ? | 18:51 |
| *** jmasud has quit IRC | 18:53 | |
| *** rlandy is now known as rlandy|biab | 19:00 | |
| *** jmasud has joined #oooq | 19:08 | |
| weshay|ruck | rfolco, we know :) | 19:09 |
| rfolco | weshay|ruck, any fixes or workarounds ? | 19:10 |
| *** jfrancoa has quit IRC | 19:25 | |
| *** derekh has quit IRC | 19:25 | |
| *** rlandy|biab is now known as rlandy | 19:32 | |
| rlandy | weshay|ruck: do you still have not merged patches for get-hash or are all your changes in? | 19:32 |
| rlandy | fixing for downstream | 19:33 |
| weshay|ruck | rfolco, rlandy https://review.rdoproject.org/r/27119 | 19:37 |
| weshay|ruck | threading needles here | 19:38 |
| weshay|ruck | rlandy, I don't know :) | 19:38 |
| rlandy | weshay|ruck: well - this is where we have come to - running one job in promotion | 19:39 |
| rlandy | +2 | 19:39 |
| weshay|ruck | heh.. weeeeeee | 19:39 |
| weshay|ruck | rlandy, and turning tempest off | 19:39 |
| weshay|ruck | rlandy, https://review.opendev.org/#/c/724703/ | 19:40 |
| weshay|ruck | shit hit the fan | 19:40 |
| rfolco | weshay|ruck, I don't want to slow you down, I just don't understand how relaxing promotion criteria helps to fix the issue | 19:40 |
| rlandy | weshay|ruck: ho4it | 19:43 |
| rlandy | go4it | 19:43 |
| weshay|ruck | rfolco, ya.. well sign me up for another section in https://hackmd.io/OXNWWIShSBaPXPC8d7kFgQ and I'll walk through it | 19:43 |
| weshay|ruck | because this will happen again | 19:43 |
| rfolco | unless we switch with some pre-scheduled one, next available slot is June 18th | 19:45 |
| rfolco | weshay|ruck, ^ | 19:45 |
| weshay|ruck | fine.. it shouldn't happen again before that | 19:45 |
| rfolco | k | 19:46 |
| rfolco | weshay|ruck, how do you entitle this ? | 19:47 |
| rfolco | living dangerous | 19:47 |
| rfolco | do not do this at home | 19:47 |
| rfolco | I still know what you did last summer | 19:48 |
| *** jmasud has quit IRC | 19:48 | |
| weshay|ruck | What to do went CentOS breaks your shit | 19:48 |
| rfolco | k | 19:48 |
| *** saneax has quit IRC | 19:51 | |
| *** jmasud has joined #oooq | 19:52 | |
| rlandy | weshay|ruck: we're close here ... tls deploy is passing ... falling over on tempest now - can't ping the router... I tried adding the ca_cert, changing to use the public interface ... no dice | 20:16 |
| rlandy | any other ideas? | 20:16 |
| rlandy | would love to close this out | 20:16 |
| weshay|ruck | rlandy, this is master? | 20:17 |
| rlandy | weshay|ruck: ack | 20:17 |
| rlandy | https://zuul.openstack.org/stream/e16e76d5c8ab49fd8b4077a5f95b5417?logfile=console.log | 20:17 |
| rlandy | https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_resources.yml#L283 | 20:18 |
| rlandy | dies there ^^ | 20:18 |
| weshay|ruck | rlandy, will have to look at the logs | 20:18 |
| rlandy | weshay|ruck: https://zuul.opendev.org/t/openstack/build/fbdf8a89dda8464d8938b5d53bb6210b | 20:19 |
| rlandy | need to chat with chandankumar when he gets back | 20:19 |
| rlandy | there were settings to deal with tls in validate tempest | 20:19 |
| weshay|ruck | + exec /usr/bin/networking-ovn-metadata-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini --log-file=/var/log/neutron/ovn-metadata-agent.log | 20:25 |
| weshay|ruck | net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory | 20:25 |
| weshay|ruck | net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) | 20:25 |
| weshay|ruck | PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory | 20:25 |
| weshay|ruck | PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) | 20:25 |
| weshay|ruck | rlandy, ^ | 20:25 |
| weshay|ruck | https://802b28d13e59f2e68859-fef50fbf80bf7a8da07d35d3242576b8.ssl.cf1.rackcdn.com/706288/25/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/fbdf8a8/logs/undercloud/var/log/extra/podman/containers/ovn_metadata_agent/stdout.log | 20:25 |
| weshay|ruck | comparing to https://0743aa731054346d94ba-c5bb35ae23b4f1517e6ef394b01c7761.ssl.cf1.rackcdn.com/722109/4/check/tripleo-ci-centos-8-standalone/c5504de/logs/undercloud/var/log/containers/neutron/ovn-metadata-agent.log | 20:25 |
| rlandy | weshay|ruck: hmmm ...not following | 20:26 |
| rlandy | how would that be diff in tls? | 20:27 |
| weshay|ruck | rlandy, well.. I would at least run it twice | 20:31 |
| weshay|ruck | it could be you just had bad luck that go | 20:31 |
| rlandy | weshay|ruck: I did | 20:31 |
| weshay|ruck | same error both times? | 20:31 |
| rlandy | the second run failed in the same place | 20:31 |
| rlandy | and this run on internal | 20:32 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/98/195798/5/check/tripleo-ci-centos-8-standalone-on-multinode-ipa-internal/a66d9a5/logs/ | 20:32 |
| rlandy | weshay|ruck: somehow fs039 runs | 20:32 |
| weshay|ruck | rlandy, this is the internal port version? | 20:33 |
| rlandy | weshay|ruck: what do you mean by internal port version? | 20:33 |
| weshay|ruck | the work around you referred to yesterday | 20:34 |
| rlandy | weshay|ruck: no, this is the proper - no-hack-me version | 20:35 |
| rlandy | the hacked version passed tempest | 20:36 |
| rlandy | so here's another question ... | 20:36 |
| rlandy | tempest_interface_name | 20:36 |
| rlandy | public or internal? | 20:36 |
| rlandy | I think that should be public | 20:36 |
| rlandy | I tried setting it but no idea if that setting took | 20:36 |
| weshay|ruck | hrm | 20:38 |
| weshay|ruck | rlandy, it shouldn't be different than the normal tempest job.. | 20:38 |
| weshay|ruck | normal standalone rather | 20:39 |
| rlandy | https://2bd0765af3260bed3084-e3e0586d119e861007b7a5c43dc05934.ssl.cf2.rackcdn.com/706288/26/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/e16e76d/logs/undercloud/var/log/containers/neutron/ovn-metadata-agent.log | 20:40 |
| rlandy | ^^ clean log - still fails to ping | 20:40 |
| rlandy | https://review.opendev.org/#/c/724729/ made no diff | 20:41 |
| rlandy | weshay|ruck: ^^ | 20:41 |
| rlandy | trying turning off ssl validation altogether | 20:41 |
| rlandy | to see if that helps | 20:41 |
| rlandy | https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tasks/tempest_resources.yml#L254 | 20:42 |
| weshay|ruck | rlandy, afaict neutron did not come up properly | 20:42 |
| * weshay|ruck looks at internal | 20:42 | |
| weshay|ruck | ah | 20:42 |
| weshay|ruck | ur saying that one is clean | 20:42 |
| rlandy | yep | 20:43 |
| rlandy | second run | 20:43 |
| weshay|ruck | rlandy, nope | 20:44 |
| weshay|ruck | rlandy, ur looking at the wrong log :) https://2bd0765af3260bed3084-e3e0586d119e861007b7a5c43dc05934.ssl.cf2.rackcdn.com/706288/26/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/e16e76d/logs/undercloud/var/log/extra/podman/containers/ovn_metadata_agent/stdout.log | 20:44 |
| weshay|ruck | same issue | 20:45 |
| weshay|ruck | https://2bd0765af3260bed3084-e3e0586d119e861007b7a5c43dc05934.ssl.cf2.rackcdn.com/706288/26/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/e16e76d/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 20:45 |
| rlandy | weshay|ruck> comparing to https://0743aa731054346d94ba-c5bb35ae23b4f1517e6ef394b01c7761.ssl.cf1.rackcdn.com/722109/4/check/tripleo-ci-centos-8-standalone/c5504de/logs/undercloud/var/log/containers/neutron/ovn-metadata-agent.log | 20:46 |
| rlandy | ^^ two diff logs :) - followed your comment | 20:46 |
| rlandy | weshay|ruck: same problem here https://320c106ad4452b72d3ea-9dead46848380cd5333c94a2972396ab.ssl.cf5.rackcdn.com/724436/2/gate/tripleo-ci-centos-8-standalone/1df4a52/logs/undercloud/var/log/extra/podman/containers/ovn_metadata_agent/stdout.log | 20:48 |
| rlandy | job passes fine | 20:48 |
| weshay|ruck | rlandy, hold the nodes in a test project | 20:49 |
| weshay|ruck | will be easier | 20:49 |
| rlandy | I wish - will have to wait until tomorrow when internal clears from its phantom jobs - blocked there | 20:50 |
| rlandy | nhicher said he'd bounce that tomorrow | 20:50 |
| *** cgoncalves has quit IRC | 20:53 | |
| *** kopecmartin has quit IRC | 20:53 | |
| *** whoami-rajat has quit IRC | 20:53 | |
| *** pojadhav has quit IRC | 20:53 | |
| *** beagles has quit IRC | 20:53 | |
| *** whoami-rajat has joined #oooq | 20:54 | |
| *** b3nt_pin has joined #oooq | 20:54 | |
| *** pojadhav has joined #oooq | 20:55 | |
| *** cgoncalves has joined #oooq | 20:55 | |
| *** jmasud has quit IRC | 21:21 | |
| rlandy | weshay|ruck: rfolco: https://review.rdoproject.org/r/27120 Upadte dlrn_report role for downstream usage | 21:36 |
| weshay|ruck | rlandy, can we remove - fedora_centos_mixed_distribution_job|default(false) | 21:37 |
| weshay|ruck | that's long gone | 21:37 |
| rlandy | weshay|ruck: k | 21:37 |
| rlandy | still in the upstream playbooks though | 21:37 |
| rlandy | weshay|ruck: https://review.rdoproject.org/r/#/c/27120/ updated | 21:48 |
| weshay|ruck | rlandy, one more | 21:54 |
| rlandy | weshay|ruck: link? | 21:55 |
| weshay|ruck | comment | 21:55 |
| *** TrevorV has quit IRC | 22:00 | |
| rlandy | weshay|ruck: is train rhel-8 still a thing? | 22:09 |
| weshay|ruck | rlandy, if you could kill those jobs, I'd appreciate it | 22:09 |
| rlandy | weshay|ruck: I ma just asking for the conditions on using get-hash | 22:10 |
| rlandy | either way, this role is not yet used upstream | 22:10 |
| *** Tengu has quit IRC | 22:11 | |
| rlandy | https://review.rdoproject.org/r/#/c/27120/3/roles/dlrn-report/tasks/dlrn-vars-setup.yml | 22:11 |
| rlandy | weshay|ruck: ^^ that will fail with rhle-8 train ... but it's fine | 22:11 |
| rlandy | this role is never used there | 22:11 |
| *** Tengu has joined #oooq | 22:12 | |
| *** rfolco has quit IRC | 22:37 | |
| *** jmasud has joined #oooq | 22:38 | |
| *** jmasud has quit IRC | 22:39 | |
| *** jmasud has joined #oooq | 22:43 | |
| *** tosky has quit IRC | 23:11 | |
| *** jmasud has quit IRC | 23:44 | |
| *** jmasud has joined #oooq | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!