| *** dviroel|afk is now known as dviroel|out | 00:46 | |
| *** dasm is now known as dasm|out | 03:11 | |
| soniya29 | chandankumar, most of the jobs are skipping on cs8 master | 04:45 |
|---|---|---|
| chandankumar | soniya29: we have dropped jobs from cs8 master long time ago | 04:46 |
| chandankumar | in main line, we run cs9 jobs | 04:46 |
| soniya29 | chandankumar, sorry cs9 | 04:46 |
| chandankumar | soniya29: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main there was a node failure | 04:47 |
| chandankumar | that's why it got skipped | 04:47 |
| soniya29 | chandankumar, should we approach infra for this issue? | 04:48 |
| soniya29 | chandankumar, yesterday we didn't had master in such a shape | 04:49 |
| chandankumar | soniya29: in other pipeline other jobs have started | 04:49 |
| chandankumar | it might be a blip | 04:49 |
| soniya29 | chandankumar, okay | 04:49 |
| chandankumar | ykarel: hello, please have a look at https://bugs.launchpad.net/tripleo/+bug/1964269 when free, thanks! | 04:52 |
| ykarel | chandankumar, issue is consistent? | 04:53 |
| ykarel | looks more of compute specific | 04:54 |
| chandankumar | ykarel: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario004-standalone-master last 2 run has the same issue | 04:55 |
| chandankumar | ignore the skipped one | 04:56 |
| ykarel | only some scenarios? or all are impacted | 04:56 |
| ykarel | would be worth to get a pattern, actually have some meetings in morning, so can check later, but better to tag someone from compute | 04:57 |
| chandankumar | ykarel: sure, will ping someone from compute | 04:57 |
| *** soniya29 is now known as soniya29|ruck | 04:57 | |
| ykarel | ack Thanks | 04:59 |
| chandankumar | ykarel: can you rekick the master line? it got skipped due to node failure if you have the access. | 05:04 |
| chandankumar | it will give better results | 05:05 |
| chandankumar | in wallaby line, mentioned jobs in the bug is passing | 05:05 |
| * soniya29|ruck back in 15 mins | 05:20 | |
| *** ykarel is now known as ykarel|mtg | 05:26 | |
| * ykarel|mtg in meeting will check in some time | 05:26 | |
| chandankumar | ykarel|mtg: we got good success with this patch https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640 | 06:21 |
| chandankumar | two tests failure but with different issue, let's get this merged | 06:21 |
| ykarel|mtg | chandankumar, ok good | 07:01 |
| *** ykarel|mtg is now known as ykarel | 07:01 | |
| ykarel | chandankumar, need to trigger some pipeline? | 07:01 |
| chandankumar | ykarel: in one hour, main line will trigger anyway so leave that | 07:02 |
| ykarel | chandankumar, k | 07:02 |
| chandankumar | ykarel: can you take a look at this https://logserver.rdoproject.org/15/151734d416f03f3b5c9c93b1cb999f490c49589c/openstack-periodic-integration-stable1-cs9/periodic-tripleo-ci-centos-9-standalone-wallaby/43072dc/logs/undercloud/var/log/tempest/stestr_results.html.gz | 07:02 |
| chandankumar | network basics ops failurew | 07:02 |
| chandankumar | it is seen on all most all cs9 wallaby jobs | 07:03 |
| ykarel | looking | 07:03 |
| *** amoralej|off is now known as amoralej | 07:06 | |
| ykarel | chandankumar, that's happening as tempest is pinned in wallaby and missing commits from master that's needed for centos 9-stream | 07:08 |
| ykarel | like https://github.com/openstack/tempest/commit/6ded070b511263df2b6c06905615a951b29e2035 | 07:09 |
| ykarel | same will apply for neutron-tempest-plugin after it's merged in master | 07:09 |
| chandankumar | ykarel: for neutron tempest plugin you are going to do a new release? | 07:11 |
| chandankumar | or will I pin at that commit? | 07:12 |
| ykarel | chandankumar, it's release time so i think all tempest plugin releases are upcoming | 07:12 |
| ykarel | i see qa team has already patch up for tempest-30.0.0 | 07:12 |
| chandankumar | but currently our integration line is red | 07:12 |
| ykarel | yes can pin to commit both tempest and neutron-tempest-plugin for time being | 07:13 |
| ykarel | and later switch to yoga tags | 07:13 |
| chandankumar | ykarel: 40228: Pin tempest and neutron-tempest-plugin for Wallaby and Xena | https://review.rdoproject.org/r/c/rdoinfo/+/40228 | 07:27 |
| chandankumar | ykarel: thanks :-) | 07:27 |
| * soniya29|ruck lunch break | 07:36 | |
| chandankumar | ykarel: can we kill these jobs running here https://review.rdoproject.org/zuul/status#827867 ? it is showing more than 43 hrs | 07:40 |
| ykarel | chandankumar, yes can kill | 08:03 |
| ykarel | killing it, but seems it's about to finish as 3rd attemp | 08:03 |
| ykarel | hmm something wrong as stuck in 3rd attempt from 2022-03-07 15:50:10.593287 | 08:04 |
| soniya29|ruck | kopecmartin, chandankumar, rlandy|out, we are cancelling today's tempest meeting since arxcruz|off and ysandeep|PTO both are out and many of us are busy with ruck/rovering stuffs | 08:04 |
| ykarel | chandankumar, dequed | 08:07 |
| *** arxcruz|off is now known as arxcruz | 08:12 | |
| kopecmartin | soniya29|ruck: ack | 08:19 |
| chandankumar | ykarel: thanks! | 08:20 |
| *** jpena|off is now known as jpena | 08:36 | |
| arxcruz | soniya29|ruck i'm not out today btw :) | 09:42 |
| soniya29|ruck | arxcruz, yeah, i know | 10:06 |
| marios | chandankumar: o/ do you already have tests for https://review.rdoproject.org/r/c/rdoinfo/+/40228/ | 10:32 |
| marios | chandankumar: literally everything that failed in the last wallaby 9 buildset is tempest https://review.rdoproject.org/zuul/buildset/3ae5ce4e79784fab81260584860766a2 | 10:32 |
| marios | https://review.rdoproject.org/r/c/rdoinfo/+/40228/ also amoralej o/ | 10:32 |
| marios | please thank you :) | 10:33 |
| amoralej | checking | 10:37 |
| amoralej | marios, chandankumar see my comment please | 10:46 |
| marios | amoralej: thanks for checking | 10:53 |
| marios | amoralej: it's the depends-on | 10:53 |
| marios | i replied there | 10:53 |
| amoralej | then you can use Upstream-Id: | 10:54 |
| amoralej | not depends-on | 10:54 |
| marios | amoralej: you mean in the commit message s/Depends-On/Upstream-Id: ? | 10:55 |
| amoralej | yes | 10:55 |
| marios | amoralej: thank you doing | 10:55 |
| amoralej | that'd test the package | 10:55 |
| amoralej | but anyway, we need to wait for the upstream before merging it | 10:55 |
| marios | amoralej: matter if we have both? | 10:55 |
| amoralej | i think not | 10:55 |
| amoralej | actually, it'd be good | 10:55 |
| chandankumar | marios: amoralej we need both | 10:56 |
| amoralej | to prevent merging before upstream | 10:56 |
| chandankumar | marios: amoralej https://review.rdoproject.org/r/c/testproject/+/40212 as per this it clears most of the neutron tests | 10:56 |
| marios | chandankumar: thanks i updated | 10:56 |
| marios | for Upstream-Id | 10:56 |
| chandankumar | I have added depends on to avoid merging | 10:56 |
| marios | chandankumar: yeah i kept it depends-on i mean | 10:57 |
| marios | chandankumar: checking testproject thanks | 10:57 |
| amoralej | chandankumar, yes, that's good | 10:57 |
| marios | chandankumar: yes all of the jobs in latest wallaby are tempest | 10:57 |
| marios | chandankumar: k your testproject is master only | 10:57 |
| marios | chandankumar: i can post something for the wallaby tests then | 10:57 |
| chandankumar | marios: but for wallaby we will be using old tempest there as it is pinned | 10:58 |
| chandankumar | let me find the distgit job to test it there directly | 10:59 |
| marios | chandankumar: don't we need source-branch: 6ded070b511263df2b6c06905615a951b29e2035 i.e. your patch? | 10:59 |
| marios | chandankumar: ugh looks like merge conflict on https://review.rdoproject.org/r/c/rdoinfo/+/40228 | 11:00 |
| chandankumar | sorry I mean to say on wallaby testproject, if we use neuton temepst plugin depends on, it will be using the pinned tempest version | 11:01 |
| chandankumar | marios: you are updating the rdoinfo merge conflict patch? | 11:04 |
| marios | chandankumar: i did not update it | 11:08 |
| marios | chandankumar: would you like me to | 11:08 |
| chandankumar | marios: sure go ahead | 11:08 |
| marios | chandankumar: heh | 11:08 |
| marios | chandankumar: rebase button worked ;) | 11:08 |
| marios | chandankumar: :D | 11:08 |
| *** rlandy|out is now known as rlandy|ruck | 11:14 | |
| marios | chandankumar: wanna join review time soniya29 join us ? | 11:19 |
| rlandy|ruck | chandankumar: soniya29: pls join review time | 11:19 |
| soniya29 | marios yupp | 11:19 |
| *** dviroel|out is now known as dviroel | 11:19 | |
| *** soniya29 is now known as soniya29|ruck | 11:20 | |
| chandankumar | https://review.opendev.org/c/openstack/tripleo-ci/+/831151 | 11:50 |
| chandankumar | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/316474 - finish image build | 11:51 |
| chandankumar | cleanup suggested by marios | 11:51 |
| chandankumar | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/316782 | 11:51 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-jobs/+/40112 | 11:57 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-jobs/+/40152 | 11:57 |
| rlandy|ruck | marios: let's catch up - in 10? | 11:59 |
| * rlandy|ruck reading relevant emails | 11:59 | |
| marios | rlandy|ruck: sure thanks | 12:01 |
| chandankumar | rlandy|ruck: marios test network basic ops are also failing in upstream check on wallaby | 12:01 |
| marios | chandankumar: ack | 12:02 |
| chandankumar | fyi https://review.opendev.org/c/openstack/tripleo-common/+/832342/1#message-6dedb63e7ae10bbbc45a321acc4b46503f95b46f | 12:02 |
| dviroel | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832670 - this change solves DNS issue - I got one job passing IPA client setup. I'm running testproject again. | 12:09 |
| rlandy|ruck | soniya29|ruck: are you on to report at program call - doc is updated | 12:10 |
| soniya29|ruck | rlandy|ruck, sure | 12:10 |
| rlandy|ruck | dviroel: ++ nice - left one suggestion on patch | 12:13 |
| rlandy|ruck | jm1: hey can you update the link to internal cockpit in tripleo-infra doc | 12:15 |
| rlandy|ruck | soniya29|ruck: ^^ | 12:16 |
| rlandy|ruck | it moved | 12:16 |
| rlandy|ruck | marios: https://meet.google.com/uoi-evnk-vre?pli=1&authuser=0 | 12:17 |
| marios | reviews please https://review.opendev.org/c/openstack/tripleo-ci/+/831932 | 12:17 |
| marios | rlandy|ruck: ack sec joining | 12:18 |
| rlandy|ruck | chandankumar: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640 merged | 12:32 |
| rlandy|ruck | can we go forward with https://review.rdoproject.org/r/c/rdoinfo/+/40228? | 12:32 |
| rlandy|ruck | amoralej: ^^? | 12:32 |
| chandankumar | yes | 12:33 |
| rlandy|ruck | bhagyashris: joining | 12:34 |
| bhagyashris | ack | 12:34 |
| amoralej | chandankumar, check my comment and the one from ykarel | 12:49 |
| rlandy|ruck | bhagyashris: pls also cancel scrum tomorrow | 12:58 |
| rlandy|ruck | if we don't need three hours | 12:58 |
| rlandy|ruck | two is probably as many as we can expect people to sit through | 12:58 |
| bhagyashris | hello tripleo ci team akahat arxcruz chandankumar dasm|out dviroel marios pojadhav soniya29|ruck ysandeep|PTO rlandy|ruck rcastillo jm1 sshnaidm we are going to cancel the todays retro meeting because most of the people are out today so will club thr retro meeting in tomorrows Tripleo CI planning meeting (1st one hr retro and second one hr planning) | 13:00 |
| marios | bhagyashris: thanks sounds good | 13:00 |
| soniya29|ruck | bhagyashris, ack | 13:00 |
| arxcruz | ack | 13:00 |
| chandankumar | ack | 13:00 |
| dviroel | lgtm | 13:01 |
| jm1 | ack | 13:01 |
| bhagyashris | i will also going to clean the board after tomorrows planning meeting so guys plz upto-date your jira cards | 13:01 |
| bhagyashris | will need some modification on the board so it would be good if you can upto date your jira cards so that i can clean up properly... | 13:03 |
| rlandy|ruck | soniya29|ruck: pls look into status on https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available | 13:10 |
| rlandy|ruck | patch looks to be merged | 13:10 |
| soniya29|ruck | rlandy|ruck, i will test the patch with the failing jobs | 13:13 |
| rlandy|ruck | bhagyashris: pls cancel retro meeting if we are not having it today | 13:15 |
| rlandy|ruck | and scrum tomorrow | 13:16 |
| * rlandy|ruck can't change those meetings | 13:16 | |
| bhagyashris | rlandy|ruck, done | 13:19 |
| rlandy|ruck | thank you | 13:20 |
| rlandy|ruck | soniya29|ruck: rerunning the failed train c7 jobs | 13:23 |
| rlandy|ruck | we need to promote this | 13:23 |
| soniya29|ruck | rlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/40248 | 13:23 |
| *** amoralej is now known as amoralej|lunch | 13:24 | |
| rlandy|ruck | soniya29|ruck: ^^ we will need tempest component promotion to mak ethose work iiuc | 13:25 |
| soniya29|ruck | rlandy|ruck, yes, i forgot | 13:26 |
| soniya29|ruck | rlandy|ruck, we need to manually promote tempest right? | 13:26 |
| rlandy|ruck | soniya29|ruck: ack - we will need to run testproject jobs | 13:27 |
| soniya29|ruck | rlandy|ruck, ack | 13:28 |
| *** ykarel is now known as ykarel|away | 13:32 | |
| *** dasm|out is now known as dasm | 13:35 | |
| chandankumar | marios: rlandy|ruck https://review.rdoproject.org/zuul/status#40228 does this sounds ok? | 13:35 |
| chandankumar | for testing | 13:35 |
| chandankumar | any more job to add there? | 13:35 |
| rlandy|ruck | chandankumar: looks ok to me ... | 13:36 |
| rlandy|ruck | I don;t think any of our tests will help | 13:36 |
| rlandy|ruck | at this poin | 13:36 |
| rlandy|ruck | point | 13:37 |
| rlandy|ruck | they won't pick up the bump iiuc | 13:37 |
| chandankumar | rlandy|ruck: it will pick up the bump | 13:37 |
| rlandy|ruck | with depends-on? | 13:41 |
| chandankumar | rdoinfo-DLRN-check-centos9 will build the rpm from pinned commit | 13:42 |
| chandankumar | and the dependent job will reuse the rpm | 13:42 |
| chandankumar | we used to test tempest bump that way | 13:42 |
| rlandy|ruck | as you think | 13:49 |
| rlandy|ruck | but I am ok to merge as is | 13:49 |
| rlandy|ruck | I don;t have commit rights there | 13:49 |
| chandankumar | I have access to that repo | 13:49 |
| chandankumar | if amoralej|lunch is ok, then we can merge it without those jobs | 13:50 |
| chandankumar | rlandy|ruck: wallaby c8s line is also looking good except ovb | 13:53 |
| marios | chandankumar: in call will check in 2 thanks | 13:54 |
| *** amoralej|lunch is now known as amoralej | 14:00 | |
| marios | chandankumar: ack wallaby tempest scenario sounds good | 14:00 |
| dasm | fyi: https://bugs.launchpad.net/nova/+bug/1962726 openssl issue in nova | 14:07 |
| dasm | it might be affecting our services in a similar way | 14:07 |
| dasm | o/ | 14:08 |
| rcastillo | dasm: o/ | 14:10 |
| chandankumar | amoralej: please have a look at this https://logserver.rdoproject.org/28/40228/5/check/rdoinfo-DLRN-check-centos9/0d8200c/ I am not sure buildset does not have rpms | 14:18 |
| rlandy|ruck | chandankumar: ack - we have two open bugs against fs001 and fs035 | 14:19 |
| rlandy|ruck | was just on meeting | 14:19 |
| rlandy|ruck | checking if those are still an issue | 14:19 |
| amoralej | i'm in a meeting | 14:21 |
| amoralej | gimme a while chandankumar | 14:21 |
| rlandy|ruck | chandankumar: for c8 - 039 and 064 we know about | 14:22 |
| rlandy|ruck | dviroel, has a fix | 14:22 |
| rlandy|ruck | fs001 passed - nice | 14:22 |
| rlandy|ruck | checking others | 14:22 |
| jm1 | rlandy|ruck: Please approve https://bugzilla.redhat.com/show_bug.cgi?id=2062309 :) | 14:26 |
| chandankumar | rlandy|ruck: fs035 failing with same issue | 14:28 |
| chandankumar | rlandy|ruck: need someone from compute team to take a look | 14:28 |
| rlandy|ruck | jm1: done | 14:29 |
| rlandy|ruck | chandankumar: ok - we have two open cards - will comment there | 14:29 |
| rlandy|ruck | soniya29|ruck: ^^ | 14:30 |
| rlandy|ruck | for fs035 | 14:30 |
| chandankumar | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1964199 | 14:31 |
| chandankumar | failing with readtimeout | 14:31 |
| rlandy|ruck | chandankumar: ack - thanks - commenting that CIX card | 14:32 |
| rlandy|ruck | closing the fs001 one | 14:32 |
| chandankumar | yes | 14:32 |
| rlandy|ruck | that one looks to be passing now | 14:32 |
| chandankumar | Do we want to promote cs8 wallaby by removing fs035 from criteria? | 14:32 |
| marios | rlandy|ruck: chandankumar: dviroel: please review when you next get a chance https://review.opendev.org/c/openstack/tripleo-ci/+/831932 | 14:36 |
| rlandy|ruck | chandankumar: we would need to remove fs020, 039 as well | 14:36 |
| rlandy|ruck | checking fs020 | 14:37 |
| rlandy|ruck | if it's the same tests, we could skip and promote | 14:37 |
| rlandy|ruck | we'd need to remove fs039 as well | 14:37 |
| dviroel | marios: ok | 14:37 |
| jm1 | rlandy|ruck: thanks :) | 14:41 |
| dviroel | rlandy|ruck: https://review.rdoproject.org/r/c/testproject/+/38646/7#message-df953bbfac46e9d06f8d8d3a89cdd5ec8324484b \o/ | 14:42 |
| rlandy|ruck | dviroel: awesome!!! | 14:43 |
| rlandy|ruck | let's get that depends merged! | 14:43 |
| dviroel | rlandy|ruck: will update with debug msg | 14:44 |
| chandankumar | fs039 wallaby cleared \o/ | 14:45 |
| chandankumar | amoralej: ok | 14:48 |
| rlandy|ruck | chandankumar: fs020 is a node prov failure: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-wallaby/996aa92/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz | 14:49 |
| rlandy|ruck | chandankumar: ack so fs035 is the only test missing from criteria according to rr tool | 14:51 |
| rlandy|ruck | chandankumar: so proposing patch to skip that in criteria to promote | 14:52 |
| rlandy|ruck | then will revert | 14:52 |
| chandankumar | sounds good | 14:52 |
| soniya29|ruck | rlandy|ruck, ack | 14:53 |
| chandankumar | rlandy|ruck: on fs020 ironic.common.exception.NodeNotLocked: Node 3d0bd7ab-c839-4f85-806e-66136a7a7df8 found not to be locked on release | 14:57 |
| chandankumar | it does not have much logs | 14:57 |
| rlandy|ruck | chandankumar; must have passed in prev run | 14:57 |
| rlandy|ruck | was not in failed jobs | 14:57 |
| chandankumar | bmc logs are also empty | 14:57 |
| chandankumar | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-wallaby yes last one passed | 14:58 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40256 Temp exclude fs035 from wallaby c8 [NEW] | 14:58 |
| rlandy|ruck | soniya29|ruck: ^^ | 14:58 |
| rlandy|ruck | chandankumar: ^^ | 14:58 |
| rlandy|ruck | dviroel: reverting the change to exclude fs039 from u, v | 14:59 |
| chandankumar | rlandy|ruck: thanks :-) | 14:59 |
| rlandy|ruck | depends on your patch | 14:59 |
| dviroel | ok | 15:00 |
| chandankumar | rlandy|ruck: for cs9 master except fs020, 035 and 064, all are green https://review.rdoproject.org/zuul/buildset/4bdec68d62d24f0cb42bb35be0f29406 | 15:00 |
| chandankumar | Want to promote this also? | 15:00 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39347 | 15:00 |
| rlandy|ruck | chandankumar: dviroel: soniya29|ruck: ^^ | 15:01 |
| rlandy|ruck | chandankumar: there we really need the neutron fix, right? | 15:01 |
| chandankumar | rlandy|ruck: yes,yes | 15:01 |
| rlandy|ruck | so if we promote, we will break stuff | 15:02 |
| chandankumar | rdoinfo-DLRN-check-centos9 is broken amoralej is working on fixing it | 15:02 |
| rlandy|ruck | ok - np | 15:02 |
| rlandy|ruck | at least we have a path forward now | 15:02 |
| rlandy|ruck | chandankumar: we are doing better than two days ago :) | 15:03 |
| rlandy|ruck | where we had 9 red lines | 15:03 |
| chandankumar | yes yes | 15:03 |
| rlandy|ruck | now at least we have 16.2, 17, train, ussuri, victoria and soon w8 promoting | 15:04 |
| rlandy|ruck | checking on c7 | 15:04 |
| rlandy|ruck | one test left to finish and then that one will promote | 15:04 |
| rlandy|ruck | chandankumar: question ... since https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/832640 merged and we ar enot waiting for any rdo bump on master .... | 15:12 |
| rlandy|ruck | should I just promote master tempest or network component? | 15:12 |
| rlandy|ruck | which ever master only needs? | 15:12 |
| *** dviroel is now known as dviroel|lunch | 15:14 | |
| rlandy|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master failing | 15:14 |
| rlandy|ruck | will have to check criteria | 15:14 |
| chandankumar | rlandy|ruck: for master we can just promote network component | 15:48 |
| rlandy|ruck | chandankumar: will testproject that | 15:48 |
| rlandy|ruck | after meeting | 15:48 |
| chandankumar | rlandy|ruck: for wallaby https://review.rdoproject.org/r/c/testproject/+/40260 | 15:48 |
| rlandy|ruck | cool | 15:49 |
| chandankumar | once we have the results we can merge the rdoinfo patch and take the network and tempest component reoute for wallaby | 15:50 |
| rlandy|ruck | chandankumar: to confirm: for master - no tempest component promotion needed, right? | 15:53 |
| chandankumar | rlandy|ruck: yes correct | 15:54 |
| rlandy|ruck | k - creating testproject | 15:54 |
| chandankumar | rlandy|ruck: for wallaby, we are only bumping tempest | 16:10 |
| chandankumar | neutron tempest plugin bump is breaking stuff so we will deal with that later for wallaby | 16:10 |
| chandankumar | i think tempest bump is enough to clear cs9 wallaby | 16:10 |
| rlandy|ruck | c7 should promote | 16:18 |
| rlandy|ruck | chandankumar: https://review.rdoproject.org/r/c/testproject/+/36255 | 16:18 |
| rlandy|ruck | chandankumar: going to dequeue main | 16:19 |
| rlandy|ruck | will rerun after network is promoted | 16:19 |
| rlandy|ruck | chandankumar: "i think tempest bump is enough to clear cs9 wallaby" - ok ping me when ready and I'll testproject tempest component | 16:21 |
| * soniya29|ruck leaving for the day | 16:38 | |
| *** dviroel|lunch is now known as dviroel | 16:42 | |
| rlandy|ruck | chandankumar: dviroel: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39348 | 16:45 |
| rlandy|ruck | adding fs035 back | 16:45 |
| rlandy|ruck | 2022-03-09 11:32:45 | [FAILED] NetworkManager-ovs-1.36.0-2.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success | 16:47 |
| rlandy|ruck | hmmm | 16:47 |
| rlandy|ruck | https://logserver.rdoproject.org/55/36255/17/check/periodic-tripleo-ci-centos-9-standalone-network-master/45536e6/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz | 16:48 |
| chandankumar | rlandy|ruck: https://review.rdoproject.org/r/c/rdoinfo/+/40228 +wed | 16:48 |
| chandankumar | once merges, please promote tempest component for wallaby | 16:48 |
| chandankumar | it will unblock most of the stuff for cs9 | 16:48 |
| rlandy|ruck | chandankumar: thanks - will do | 16:48 |
| chandankumar | see ya! | 16:48 |
| rlandy|ruck | chandankumar++ awesome | 16:48 |
| rlandy|ruck | see you tomorrow | 16:49 |
| rlandy|ruck | amoralej++ thanks | 16:49 |
| dviroel | tks chandankumar o/ | 16:49 |
| dviroel | rlandy|ruck: "[Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]" | 16:51 |
| dviroel | the rpm is there | 16:51 |
| *** marios is now known as marios|out | 16:51 | |
| rlandy|ruck | dviroel; yeah - rerunning | 16:52 |
| rlandy|ruck | there must have been a hitch | 16:54 |
| rlandy|ruck | all stable 1 is read as well | 16:54 |
| rlandy|ruck | dlrn build | 16:54 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-wallaby/de81829/logs/undercloud/home/zuul/repo_setup.log.txt.gz | 16:54 |
| rlandy|ruck | repo is there | 16:55 |
| dviroel | might be | 16:57 |
| dviroel | ade_lee_: we made a progress in standalone-on-multinode-ipa | 17:07 |
| dviroel | ade_lee_: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_335/824479/17/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/3356ffc/logs/undercloud/home/zuul/undercloud-ipa-install.log | 17:07 |
| dviroel | ade_lee_: "Kerberos authentication failed: kinit: KDC has no support for encryption type while getting initial credentials" | 17:08 |
| dviroel | is the current error | 17:08 |
| dviroel | ade_lee_: btw, we have Zuul +1 - https://review.opendev.org/c/openstack/tripleo-ci/+/824479/17#message-d32b1d468f6b8215f1a325e9824c143192ee1ca6 (standalone-on-multinode-ipa is still non-voting due to openssl issue) | 17:11 |
| *** amoralej is now known as amoralej|off | 17:12 | |
| rlandy|ruck | lunch - brb | 17:16 |
| dasm | dviroel: thanks for +2. | 17:27 |
| dasm | dviroel: if you're on it, could you review other patches too? https://review.rdoproject.org/r/q/topic:rr_refactor | 17:28 |
| *** jpena is now known as jpena|off | 17:52 | |
| dviroel | ok | 17:57 |
| ade_lee_ | dviroel, nice -- seems like the rabbit issue went away | 18:03 |
| ade_lee_ | dviroel, is the ipa job not running with a downgraded openssl? | 18:04 |
| rlandy|ruck | oh good - tempest kicked right on time | 18:04 |
| dasm | dviroel: thanks. I'm trying to refactor this script a bit more, but I'm going through changes slowly. I don't want to nuke everything at once. | 18:05 |
| dasm | ade_lee_: multinode-ipa was running yesterday | 18:05 |
| ade_lee_ | dviroel, nice - I see the secondary enable_fips: false flag you added | 18:05 |
| dviroel | ade_lee_: i think that should be running with downgraded openssl already - will check | 18:06 |
| dviroel | ade_lee_: oh, it is not openssl.x86_64 1:3.0.1-14.el9 @baseos | 18:07 |
| dviroel | ade_lee_: will check that in a minute | 18:07 |
| ade_lee_ | dviroel, yeah - so theoretically, once we have that set up, we should be golden | 18:08 |
| dviroel | yes | 18:08 |
| ade_lee_ | excellent! | 18:10 |
| dviroel | rlandy|ruck: dasm: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832670 is ready for review | 18:11 |
| dasm | dviroel: +1 from me | 18:12 |
| rlandy|ruck | thanks - merged | 18:23 |
| rlandy|ruck | dasm: after master promotes, you can try uefi boot again | 18:24 |
| rlandy|ruck | rcastillo: ^^ | 18:24 |
| dasm | ack | 18:24 |
| rlandy|ruck | we can give this a shot on ovb | 18:24 |
| rcastillo | rlandy|ruck: ack | 18:24 |
| rlandy|ruck | give us a half a day to promote | 18:24 |
| dasm | i'm leaving in a bit, so either i'll try tonight or tomorrow morning (after retro) | 18:24 |
| rlandy|ruck | promoting network component | 18:25 |
| *** dasm is now known as dasm|bbl | 18:26 | |
| rlandy|ruck | here we go master | 18:33 |
| dviroel | ade_lee_: about standalone-on-multinode-ipa: the openssl fix is applied, but it just dio the downgrade on ipa-server node (secondary which has fips disabled) | 19:02 |
| dviroel | ade_lee_: this issue doesn't exists without fips enabled on upstream | 19:03 |
| dviroel | ade_lee_: make sense downgrade openssl in the client-side too? (primary node) | 19:03 |
| dviroel | rlandy|ruck: how are we now? something to debug atm? | 19:29 |
| dviroel | main line has all dependencies needed already? | 19:29 |
| ade_lee_ | dviroel, so ... | 19:29 |
| ade_lee_ | dviroel, I thought that the issue was -- independently of fips - that ipa server was not working with the newest openssl | 19:30 |
| rlandy|ruck | dviroel: rekicked master - running now | 19:30 |
| rlandy|ruck | waiting for tempest component on wallaby 9 | 19:31 |
| ade_lee_ | which means that the temporary fix is to downgrade openssl on the ipa server - ie. secondary node | 19:31 |
| rlandy|ruck | once that completed | 19:31 |
| rlandy|ruck | completes, I need to promote that component and kick wallaby c9 line | 19:31 |
| rlandy|ruck | wallaby c8 promoted | 19:31 |
| rlandy|ruck | train 7 promoted | 19:31 |
| rlandy|ruck | as did rhos-17 | 19:32 |
| rlandy|ruck | u, v, t promoted yesterday | 19:32 |
| rlandy|ruck | gate is ok | 19:32 |
| rlandy|ruck | breathe, breathe | 19:32 |
| rlandy|ruck | we will see how master does | 19:32 |
| ade_lee_ | once we have fips, we need both the older openssl on the secondary master as well as not turning fips on -- on the secndary node | 19:32 |
| rlandy|ruck | and wallaby | 19:32 |
| rlandy|ruck | dviroel: ^^ is that the answer you wanted? | 19:32 |
| ade_lee_ | dviroel, is that what we have? or am I confused? | 19:32 |
| dviroel | rlandy|ruck: yes - do we need votes to revert something wrt wallaby-c8? | 19:33 |
| ade_lee_ | dviroel, if there is still an issue with fips turned off and openssl downgraded on the secondaty node, then we need to look further | 19:34 |
| rlandy|ruck | merged that ... I think | 19:34 |
| ade_lee_ | and maybe pull in an ipa dev | 19:34 |
| rlandy|ruck | so fs035 bug needs investigating | 19:34 |
| rlandy|ruck | dviroel: ^^ | 19:34 |
| rlandy|ruck | getting | 19:34 |
| dviroel | rlandy|ruck: ok | 19:34 |
| rlandy|ruck | and then we will have to try deal with the component lines | 19:34 |
| rlandy|ruck | but they are so old atm | 19:35 |
| rlandy|ruck | dviroel: https://bugs.launchpad.net/bugs/1964199 | 19:35 |
| dviroel | ade_lee_: latest run has: primary (client) fips enabled + latest openssl; secondary (server) fips disabled + older openssl - fails happens on primary atm | 19:39 |
| rlandy|ruck | dviroel: here is the revert: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39348 | 19:41 |
| ade_lee_ | dviroel, ok -- and we also know that without fips, with primary (client) + latest openssl, everything works? | 19:41 |
| ade_lee_ | dviroel, well more specifically, primary no-fips + latest openssl, secondary no-fips + older openssl | 19:43 |
| ade_lee_ | that this works, right? ^^ | 19:43 |
| dviroel | ade_lee_: correct | 19:44 |
| ade_lee_ | dviroel, ok - I gotta take a look then and see whats breaking under fips -- may be another ipa fips bug (on the client side) | 19:45 |
| ade_lee_ | dviroel, did you see what the error was? | 19:46 |
| dviroel | ade_lee_: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_335/824479/17/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/3356ffc/logs/undercloud/home/zuul/undercloud-ipa-install.log | 19:47 |
| dviroel | ade_lee_: "kinit: KDC has no support for encryption type while getting initial credentials" | 19:47 |
| dviroel | ade_lee_: https://github.com/freeipa/ansible-freeipa/blob/master/roles/ipaclient/tasks/install.yml#L197 | 19:48 |
| ade_lee_ | dviroel, ok - cool , let me talk to some ipa folks | 19:50 |
| ade_lee_ | dviroel, at this point, though, I agree - it makes sense to try to downgrade the primary node (client) and see if that fixes the problem (under fips) | 19:51 |
| dviroel | ade_lee_: i know rjeffman, but he is on pto | 19:53 |
| ade_lee_ | dviroel, ack -- I'll ask rcrit and maybe cheimes | 19:53 |
| ade_lee_ | dviroel, but lets try downgrading the client too to make sure things work under fips | 19:54 |
| dviroel | ade_lee_: ack - lets try that | 19:54 |
| ade_lee_ | (we'll still hae the logs for ipa folks to look at) | 19:55 |
| dviroel | ade_lee_: hum, by downgrading openssl we will miss some FIPS fixes right? lets try anyway | 19:58 |
| ade_lee_ | dviroel, yeah , but that fips fix was because of openssl changes | 20:03 |
| rlandy|ruck | hmmm .... master not looking great | 20:45 |
| dviroel | - multinode-ipa: python3-linux-procfs-0.7.0-1.el9.noarch: Cannot download | 20:56 |
| rlandy|ruck | lots of download issues | 20:56 |
| rlandy|ruck | dviroel: we should look at chandankumar's review requests for today | 20:57 |
| dviroel | rlandy|ruck: ok | 20:57 |
| dviroel | rlandy|ruck: lets do it | 20:57 |
| * rlandy|ruck gets | 20:57 | |
| dviroel | rlandy|ruck: full-tempest-scenario: "buildah run 192.168.24.1-working-container-1 yum list installed " again | 20:57 |
| rlandy|ruck | yeah - figure that is transient | 20:58 |
| rlandy|ruck | going through review list | 20:59 |
| dviroel | all other are download issues | 21:00 |
| rlandy|ruck | dviroel: am I missing something here? https://review.opendev.org/c/openstack/tripleo-ci/+/831932? | 21:04 |
| rlandy|ruck | more reparenting will be required | 21:05 |
| dviroel | where is tripleo-ci-base-multinode-centos-9? | 21:06 |
| dviroel | oh, will merge https://review.opendev.org/c/openstack/tripleo-ci/+/831151/19/zuul.d/multinode-jobs.yaml | 21:08 |
| dviroel | so yes, we can hold marios patch | 21:09 |
| dviroel | not easy to review the container ones | 21:21 |
| rlandy|ruck | where is the multinode one? | 21:22 |
| dviroel | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/831151/19/zuul.d/multinode-jobs.yaml | 21:22 |
| dviroel | this one? | 21:22 |
| rlandy|ruck | ok - so that is all then | 21:23 |
| rlandy|ruck | thought I was missing another one | 21:23 |
| rlandy|ruck | going to rekick failing main jobs | 21:27 |
| rlandy|ruck | then try w c9 | 21:27 |
| dviroel | ok | 21:28 |
| * dviroel needs to step out | 21:28 | |
| * dviroel o/ | 21:29 | |
| *** dviroel is now known as dviroel|out | 21:29 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 23:26 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!