| soniya29 | frenzyfriday|rover, good morning :) | 04:39 |
|---|---|---|
| *** soniya29 is now known as soniya29|ruck | 04:39 | |
| soniya29|ruck | frenzyfriday|rover, shall we sync in half hour or so? | 04:41 |
| frenzyfriday|rover | soniya29|ruck, Good morning. Yep, sure | 04:41 |
| rlandy__ | soniya29|ruck: frenzyfriday|rover: pls ask arxcruz to copy back containers for master c9 and wallaby c9 | 05:03 |
| rlandy__ | arxcruz: we need that script set up asap - please | 05:04 |
| rlandy__ | running on a cron | 05:04 |
| rlandy__ | that is hosing the jobs in the component lines | 05:04 |
| rlandy__ | frenzyfriday|rover: soniya29|ruck: ^^ | 05:04 |
| rlandy__ | frenzyfriday|rover: soniya29|ruck: start with one integration line at a time ... | 05:05 |
| rlandy__ | master looks not too bad | 05:05 |
| frenzyfriday|rover | rlandy__, ack | 05:05 |
| rlandy__ | look to find dasm's fix for ipa | 05:05 |
| rlandy__ | see if it is merged | 05:05 |
| rlandy__ | and if the same fix will work for fs-30 and 064 | 05:05 |
| rlandy__ | frenzyfriday|rover: get infra to cleat the stuck ceph lines | 05:06 |
| frenzyfriday|rover | rlandy__, yep, 62 hrs | 05:06 |
| rlandy__ | look into fs001 sce01 and temo | 05:06 |
| rlandy__ | tempest | 05:07 |
| rlandy__ | frenzyfriday|rover: also clear periodic-ci-config-molecule-python | 05:07 |
| rlandy__ | frenzyfriday|rover: soniya29|ruck: will see you in a few hours | 05:07 |
| rlandy__ | soniya29|ruck: I left you notes re: rhos-16.2 | 05:07 |
| rlandy__ | pls check in with attilla there and get 16.2 to promote | 05:07 |
| rlandy__ | frenzyfriday|rover: once you have master c9 under control start with wallaby c8 cand c9 | 05:08 |
| rlandy__ | we will try clear these lines one at a time | 05:08 |
| *** rlandy__ is now known as rlandy|out | 05:08 | |
| rlandy|out | chandankumar: marios is out today - pls try get multinode in shape for merge with dviroel|out | 05:09 |
| chandankumar | rlandy|out: sure | 05:22 |
| *** ysandeep|out is now known as ysandeep | 05:22 | |
| soniya29|ruck | rlandy_ ack | 05:37 |
| ysandeep | pojadhav: good morning, lets meet when you are available to sync about 17 status. | 05:49 |
| pojadhav | ysandeep, good morning sure.. lets do it now.. | 05:55 |
| chandankumar | ysandeep: frenzyfriday|rover hello, let's remove ceph-ansible jobs https://review.rdoproject.org/r/c/config/+/40081 | 05:56 |
| ysandeep | chandankumar, ack | 05:58 |
| ysandeep | pojadhav, ack.. 2mins | 05:59 |
| frenzyfriday|rover | chandankumar, ack thanks | 05:59 |
| ysandeep | chandankumar, pojadhav review request o/ add ovb jobs for 17 on rhel-9 https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/305869 | 06:00 |
| ysandeep | testproject results here: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/305869/16#message-1ca1179c6646fd605b9eb90e959271f419ea2468 | 06:00 |
| ysandeep | pojadhav, https://meet.google.com/ydf-eqzj-mfh?authuser=0&hl=en | 06:04 |
| ysandeep | pojadhav, container build with workarounds: https://code.engineering.redhat.com/gerrit/c/testproject/+/210197 | 06:11 |
| pojadhav | ysandeep, https://code.engineering.redhat.com/gerrit/q/topic:%2522rhos17_rhel9%2522+(status:open) | 06:16 |
| *** amoralej|off is now known as amoralej | 07:29 | |
| frenzyfriday|rover | soniya29|ruck, I updated the hackmd with failures from integration master - loads of failures. Do you want to sync and see if some of them are common to what you found? | 07:31 |
| soniya29|ruck | frenzyfriday|rover, sure | 07:32 |
| soniya29|ruck | frenzyfriday|rover, lets sync | 07:32 |
| soniya29|ruck | frenzyfriday|rover, meet.google.com/yab-mkqz-dpg | 07:32 |
| *** jpena|off is now known as jpena | 08:02 | |
| *** ysandeep is now known as ysandeep|lunch | 08:29 | |
| *** gthiemon1e is now known as gthiemonge | 08:39 | |
| *** ysandeep|lunch is now known as ysandeep | 08:52 | |
| *** poojajadhav is now known as pojadhav | 09:01 | |
| *** chem is now known as Guest1522 | 09:48 | |
| ysandeep | pojadhav, I have merged your other patches, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315857 went in merge conflict | 09:55 |
| ysandeep | chandankumar, ykarel review request when time permits https://code.engineering.redhat.com/gerrit/q/topic:%22bug%252F2060932%22+(status:open%20OR%20status:merged) | 10:20 |
| pojadhav | ysandeep, ack looking | 10:36 |
| pojadhav | ysandeep, rebased : https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315857 | 10:37 |
| *** arxcruz is now known as arxcruz|off | 11:04 | |
| *** rlandy is now known as rlandy|ruck | 11:13 | |
| chandankumar | https://review.opendev.org/c/openstack/tripleo-ci/+/831151 | 11:18 |
| chandankumar | rlandy|ruck: ^^ | 11:18 |
| * frenzyfriday|rover going afk for 30 mins | 11:21 | |
| rlandy|ruck | ysandeep: dviroel|out: https://review.opendev.org/c/openstack/tripleo-ci/+/831151 - pls we need another +2 and w+ if you are happy by EoD | 11:22 |
| rlandy|ruck | most important patch | 11:22 |
| *** dviroel|out is now known as dviroel | 11:23 | |
| chandankumar | rlandy|ruck: https://review.rdoproject.org/r/c/config/+/40081 | 11:33 |
| ysandeep | rlandy|ruck, https://code.engineering.redhat.com/gerrit/q/topic:%22bug%252F2060932%22+(status:open%20OR%20status:merged) | 11:33 |
| ysandeep | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/315860 | 11:35 |
| ysandeep | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315488 | 11:35 |
| rlandy|ruck | frenzyfriday|rover: soniya29|ruck: pls ping when you are back - we need to ruck rover sync asap | 11:41 |
| rlandy|ruck | ysandeep: is the ovb stack clean up script back up and running? | 11:43 |
| rlandy|ruck | vexx | 11:43 |
| ysandeep | rlandy|ruck, yes its running | 11:44 |
| rlandy|ruck | k - thank you | 11:44 |
| frenzyfriday|rover | rlandy|ruck, hey, I am back | 11:47 |
| soniya29|ruck | rlandy|ruck, i am also there | 11:48 |
| rlandy|ruck | frenzyfriday|rover: soniya29|ruck: https://meet.google.com/wqn-sdre-inh?pli=1&authuser=0 | 11:49 |
| ysandeep | rlandy|ruck, chandankumar https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315488 updated, when you have time to revisit again. | 12:03 |
| chandankumar | dviroel: ++ thanks for artifact question | 12:04 |
| chandankumar | dviroel: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/316264 and https://review.rdoproject.org/r/c/config/+/40160/ when free, thanks! | 12:04 |
| dviroel | chandankumar: o/ | 12:05 |
| chandankumar | ysandeep: thanks! | 12:09 |
| dviroel | chandankumar: hey - have a question https://review.rdoproject.org/r/c/config/+/40160/1#message-8ceab353594b95f46ccca3bcbd242cd916d81e51 | 12:18 |
| ysandeep | dviroel, :D I have a similiar comment on a downstream patch: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/315923/2/zuul.d/tripleo-build-containers.yaml#31 | 12:22 |
| chandankumar | dviroel: ysandeep testing here https://code.engineering.redhat.com/gerrit/c/testproject/+/301974 | 12:24 |
| chandankumar | let's see | 12:24 |
| dviroel | ysandeep: chandankumar ack | 12:24 |
| rlandy|ruck | frenzyfriday|rover: rekicked https://review.rdoproject.org/r/c/testproject/+/40103 | 12:41 |
| rlandy|ruck | with periodic | 12:41 |
| rlandy|ruck | now that cloud is better | 12:41 |
| frenzyfriday|rover | rlcool, thanks! | 12:42 |
| rlandy|ruck | looks like it does not address the issue | 12:42 |
| rlandy|ruck | but dasm is on it | 12:42 |
| rlandy|ruck | dasm: you around? | 12:42 |
| rlandy|ruck | frenzyfriday|rover: soniya29|ruck: have the two failing victoria jobs in rerun | 12:47 |
| rlandy|ruck | https://review.rdoproject.org/r/c/testproject/+/35663 | 12:48 |
| frenzyfriday|rover | rlandy|ruck, ack, thanks | 12:48 |
| rlandy|ruck | soniya29|ruck; https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/9cd2263/logs/undercloud/var/log/tempest/stestr_results.html.gz only two failures here | 12:50 |
| rlandy|ruck | rerunning | 12:51 |
| soniya29|ruck | rlandy|ruck, shall I update the bug accordingly | 12:52 |
| rlandy|ruck | soniya29|ruck: ack pls close out https://bugs.launchpad.net/tripleo/+bug/1963874 | 12:52 |
| rlandy|ruck | I am rerunning the current failure | 12:52 |
| rlandy|ruck | to se if it's a real issue | 12:52 |
| rlandy|ruck | diff tempest failure from last run | 12:52 |
| soniya29|ruck | rlandy|ruck, okay | 12:53 |
| frenzyfriday|rover | rlandy|ruck, soniya29|ruck : https://bugs.launchpad.net/tripleo/+bug/1963907 - bug for tempest master | 12:56 |
| soniya29|ruck | frenzyfriday|rover, ack | 12:57 |
| * soniya29|ruck looking into bug | 12:57 | |
| rlandy|ruck | frenzyfriday|rover: soniya29|ruck; thank you - let;s get some useful debug info in that | 12:59 |
| rlandy|ruck | ysandeep: do you still have a patch to kick 16,2 once a day? | 13:19 |
| rlandy|ruck | I think we should dequeue it when it starts | 13:20 |
| rlandy|ruck | so baremetal can finish | 13:20 |
| ysandeep | rlandy|ruck, I added in list of items to do, but haven't posted the patch yet - As I want to do the math first based on resource calculation - what will be the best time to start those lines. | 13:21 |
| *** amoralej is now known as amoralej|lunch | 13:22 | |
| ysandeep | rlandy|ruck, I am sorting something with delivery for 17 currently, but I will post something today after some resource calculation math | 13:24 |
| chandankumar | ysandeep: dviroel https://code.engineering.redhat.com/gerrit/c/testproject/+/301974 | 13:54 |
| chandankumar | as per zuul we cannot reuse the playbook from upstream in an inherited job | 13:54 |
| chandankumar | need to think something else | 13:55 |
| ysandeep | for container build may be we add one more layer in upstream to hold that pre | 13:55 |
| ysandeep | parenting layer* | 13:56 |
| chandankumar | ysandeep: yes | 13:56 |
| dviroel | chandankumar: yeah, the problem is that zuul always looks for playbooks in current repo - "openstack/tripleo-ci-internal-jobs/playbooks/tripleo-ci/post-artifact-mult.yml" | 13:56 |
| chandankumar | let me check with zuul people | 13:57 |
| dviroel | had that issue once in rdo config | 13:57 |
| dasm | o/ | 13:57 |
| dasm | rlandy|ruck: hey, just joined. the revert wasn't the right fix. hence we merged frenzyfriday|rover non-voting patcn | 13:59 |
| dasm | *patch | 13:59 |
| rlandy|ruck | dasm: I updated your patch | 13:59 |
| dasm | joining rbac meeting. will join scrum after that | 13:59 |
| rlandy|ruck | frenzyfriday|rover: soniya29|ruck: akahat: pojadhav: scrum now | 14:03 |
| soniya29|ruck | rlandy|ruck, will join in 5 minutes | 14:03 |
| akahat | rlandy|ruck, in cibyl meeting will join in 5 - 10 mins. | 14:04 |
| chandankumar | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/831151 | 14:06 |
| ysandeep | rlandy|ruck, https://hackmd.io/NbfR-zG3T9GokDO3IZ0FeA?view#Full-Integration-pipeline-Status | 14:10 |
| *** amoralej|lunch is now known as amoralej | 14:19 | |
| dviroel | https://review.opendev.org/c/openstack/tripleo-ci/+/831015 | 14:19 |
| rlandy|ruck | dasm: hi - let's touch base re: fs039 | 14:27 |
| dasm | ack | 14:28 |
| rlandy|ruck | https://meet.google.com/noj-fsyn-yyn?pli=1&authuser=0 | 14:28 |
| rlandy|ruck | dasm: ^^ | 14:28 |
| frenzyfriday|rover | create-scenario010-trigger-for-downstream - again node failure | 14:31 |
| chandankumar | ysandeep: dviroel https://review.opendev.org/c/openstack/tripleo-ci/+/832254 | 14:46 |
| rlandy|ruck | frenzyfriday|rover: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset064-wallaby is not in criteria | 14:49 |
| rlandy|ruck | not sure if it should be | 14:50 |
| rlandy|ruck | ysandeep: ^^? | 14:50 |
| rlandy|ruck | c8 | 14:50 |
| ysandeep | rlandy|ruck, it can be added as its a supported config for tripleo - job was stable for a while (before the recent bug - i think fs039 hitting same issue) | 14:52 |
| ysandeep | chandankumar, ack | 14:52 |
| rlandy|ruck | ysandeep: wonder if we still have old nodes on downstream | 14:52 |
| frenzyfriday|rover | ysandeep, rlandy|ruck ack, adding 64 in criteria | 14:52 |
| rlandy|ruck | jobs waiting to start | 14:53 |
| ysandeep | rlandy|ruck, looking | 14:54 |
| rlandy|ruck | ysandeep: I think it's the additional jobs in the component lines | 14:54 |
| rlandy|ruck | we need to up the server numbers | 14:54 |
| ysandeep | rlandy|ruck, we got our increased quota? | 14:55 |
| ysandeep | yes network component line running there have number of jobs.. | 14:55 |
| frenzyfriday|rover | rlandy|ruck, ysandeep pls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40167 | 14:56 |
| reviewbot | I have added your review the to Review list | 14:56 |
| frenzyfriday|rover | We can merge it after the bug is fixed | 14:56 |
| *** dviroel is now known as dviroel|lunch | 14:56 | |
| rlandy|ruck | soniya: pls nick | 14:56 |
| *** soniya is now known as soniya|ruck | 14:56 | |
| ysandeep | rlandy|ruck, no old nodes there - confirmed | 14:56 |
| chandankumar | rlandy|ruck: let's merge container multinode tomorrow | 14:57 |
| rlandy|ruck | chandankumar: ok - get marios and ysandeep ti review | 15:00 |
| rlandy|ruck | don;t need to wait for me | 15:00 |
| frenzyfriday|rover | rlandy|ruck, soniya|ruck do you want to sync now? | 15:27 |
| soniya|ruck | frenzyfriday|rover, sure | 15:27 |
| rlandy|ruck | frenzyfriday|rover: soniya|ruck: ack ... | 15:28 |
| frenzyfriday|rover | rlandy|ruck, soniya|ruck https://meet.google.com/ijp-uuch-vep | 15:29 |
| dasm | ysandeep: can I get +w on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40058 and https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40060 Both had +W but were stuck due to merge conflict | 15:30 |
| ysandeep | dasm, done | 15:30 |
| dasm | ysandeep: ack, thank you sir! | 15:32 |
| ysandeep | dasm: my pleasure sir! :D | 15:32 |
| *** ysandeep is now known as ysandeep|dinner | 15:36 | |
| soniya|ruck | slaweq, hello, we have neutron tempest tests failing with 'instance not found error' on cs9 fs20 master and cs9 standalone full tempest scenario master | 15:38 |
| soniya|ruck | slaweq, here is the relevant bug:- https://bugs.launchpad.net/tripleo/+bug/1963907 | 15:38 |
| frenzyfriday|rover | rlandy|ruck, soniya|ruck I am leaving for the day in ~20 mins | 15:39 |
| soniya|ruck | frenzyfriday|rover, sure :) | 15:39 |
| chandankumar | dviroel|lunch: rlandy|ruck if things work fine with testproject https://review.rdoproject.org/r/c/testproject/+/40169 feel free to merge this https://review.opendev.org/c/openstack/tripleo-ci/+/832254/ and https://review.opendev.org/c/openstack/tripleo-ci/+/831151 in your time, thanks! | 15:41 |
| chandankumar | seeya! | 15:42 |
| rlandy|ruck | chandankumar: rdo zuul is down | 15:42 |
| rlandy|ruck | testproject will not run atm | 15:42 |
| frenzyfriday|rover | zuul is back now | 15:54 |
| *** ysandeep|dinner is now known as ysandeep | 15:55 | |
| soniya|ruck | rlandy|ruck, frenzyfriday|rover, the neutron tempest tests failing on c9 fs20 master and full tempest api master are throwing 'SSHTimeout' errors | 15:59 |
| soniya|ruck | rlandy|ruck:, frenzyfriday|rover:, this seems relevant to the effort going on in neutron-tempest plugin | 16:00 |
| soniya|ruck | rlandy|ruck:, frenzyfriday|rover: there are patches related to SSH interminent failures | 16:00 |
| soniya|ruck | rlandy|ruck: frenzyfriday|rover: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/821067 | 16:01 |
| rlandy|ruck | soniya|ruck: can you test those pls | 16:01 |
| rlandy|ruck | with full tempest | 16:01 |
| soniya|ruck | rlandy|ruck: frenzyfriday|rover: https://etherpad.opendev.org/p/neutron-ci-improvements | 16:01 |
| rlandy|ruck | frenzyfriday|rover: zuul is back | 16:02 |
| rlandy|ruck | frenzyfriday|rover: can you recheck your patches | 16:02 |
| soniya|ruck | rlandy|ruck, having dinner | 16:08 |
| soniya|ruck | rlandy|ruck, will test that patch right after it | 16:08 |
| *** dviroel|lunch is now known as dviroel | 16:13 | |
| chandankumar | rlandy|ruck: dviroel ysandeep https://review.opendev.org/c/openstack/tripleo-ci/+/832278 needed to quick for testing downstream container and image build | 16:14 |
| ysandeep | chandankumar: ack, can check first thing in my morning. | 16:21 |
| dviroel | is upstream zuul down? | 16:28 |
| * dviroel it seems so | 16:40 | |
| soniya|ruck | rlandy|ruck:, https://review.rdoproject.org/r/c/testproject/+/40171 | 16:42 |
| * soniya|ruck leaving for the day | 16:44 | |
| soniya|ruck | rlandy|ruck, leaving for today | 16:44 |
| *** ysandeep is now known as ysandeep|out | 17:18 | |
| *** jpena is now known as jpena|off | 17:52 | |
| rlandy|ruck | dasm: still failing on https://logserver.rdoproject.org/03/40103/3/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/d0e9a3f/job-output.txt | 17:53 |
| rlandy|ruck | pls review | 17:53 |
| dasm | hmmm... openssl downgrade still has > openssl.x86_64 1:3.0.1-14.el9 @baseos | 17:53 |
| dasm | i'm checking that now | 17:53 |
| rlandy|ruck | dasm: if it gets updated later - still an issue | 17:58 |
| rlandy|ruck | you may need to remove it from repos | 17:58 |
| rlandy|ruck | when you downgrade | 17:58 |
| dasm | checking | 17:59 |
| rlandy|ruck | dasm: wallaby and victor c8 fs039 also still failing | 18:30 |
| dasm | :/ | 18:31 |
| *** amoralej is now known as amoralej|off | 18:37 | |
| dviroel | rlandy|ruck: dasm: fs038/064 have the same openssl issue? | 18:45 |
| dviroel | dasm: iirc is a different setup role | 18:45 |
| dasm | wallaby fs39 is different | 18:47 |
| dasm | > IPA client test | undercloud | error={"changed": false, "msg": "Unable to discover domain, not provided on command line"} | 18:47 |
| dviroel | https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/ca79791/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz | 18:53 |
| dviroel | fatal: [localhost]: FAILED! => {"changed": false, "msg": "login: HTTP Error 500: Internal Server Error"} | 18:55 |
| dviroel | found this one | 18:55 |
| dviroel | TASK [add nova host manager role] | 18:55 |
| dasm | afair i saw similar thing. lemme check libvirt logs | 18:56 |
| dasm | (facepalm). above, i was checking cs8 wallaby, not cs9 | 18:58 |
| dviroel | dasm: about your multinode patch: https://logserver.rdoproject.org/03/40103/3/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/d0e9a3f/logs/38.102.83.93/var/log/dnf.log.txt.gz | 18:59 |
| dviroel | search for "downgrade" | 19:00 |
| dasm | > 2022-03-07T12:12:25-0500 INFO No package 1:3.0.1-5 available. | 19:00 |
| dasm | heh. | 19:00 |
| dviroel | =/ | 19:00 |
| dasm | dviroel++ thanks | 19:00 |
| dviroel | o/ np | 19:02 |
| rlandy|ruck | dasm: need help with ipa and fs039 failures? | 19:10 |
| dasm | probably. currently checking repo wrt available openssl versions | 19:11 |
| rlandy|ruck | sorry - just on another chat | 19:28 |
| dasm | ade_lee_: wrt kinit issues and downgrading openssl version go 3.0.1-5 -- it's unavailable anymore. Latest version is 1-12 Any other ideas how we can try to solve this issue? | 19:28 |
| rlandy|ruck | we can look at it in a few | 19:28 |
| dasm | ade_lee_: for reference: https://logserver.rdoproject.org/03/40103/3/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/d0e9a3f/job-output.txt | 19:28 |
| dasm | rlandy|ruck: no worries. thanks | 19:28 |
| rlandy|ruck | dasm: ok - let me know | 19:29 |
| rlandy|ruck | those two failures are blocking master promotion now | 19:29 |
| dviroel | dasm: http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ - openssl-3.0.1-5.el9...>2022-01-27 18:271.2M | 19:30 |
| rlandy|ruck | dasm: easiest ... run the job | 19:31 |
| rlandy|ruck | hold the nodes | 19:31 |
| rlandy|ruck | get on the nodes and debug | 19:31 |
| dasm | rlandy|ruck: i don't know how :) | 19:34 |
| dasm | dviroel: oh, i saw only 1-12 Thanks for this finding! | 19:35 |
| dviroel | dasm: np - you may need to lock versions - to avoid other process to upgrade this package (dnf version lock OR lock in dnf.conf) | 19:36 |
| rlandy|ruck | dasm: dviroel: let's get on a call and sort this out | 19:38 |
| dasm | k | 19:38 |
| dviroel | sure | 19:38 |
| dasm | https://meet.google.com/oyg-rkjh-bfz | 19:38 |
| rlandy|ruck | joining | 19:38 |
| rlandy|ruck | dviroel: ^^ | 19:38 |
| dasm | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116/3/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml | 19:40 |
| dasm | https://logserver.rdoproject.org/03/40103/3/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/d0e9a3f/logs/38.102.83.93/var/log/dnf.log.txt.gz | 19:43 |
| ade_lee_ | dasm, are you sure it just isn't that you used the wrong command? does dnf downgrade foo version work? | 20:12 |
| ade_lee_ | dasm, https://unix.stackexchange.com/questions/266888/can-i-force-dnf-to-install-an-old-version-of-a-package | 20:13 |
| dasm | ade_lee_: i think so. checking that now | 20:13 |
| ade_lee_ | dasm, dnf install foo-version | 20:13 |
| ade_lee_ | dnf downgrade foo-version | 20:13 |
| dasm | it didn't work for "dnf downgrade openssl 1:3.1.0-5" but worked for downgrade openssl-3.1.0-5.el9 | 20:14 |
| ade_lee_ | right foo-version not foo version | 20:14 |
| rlandy|ruck | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/freeipa-setup/templates/ipa_prep.sh.j2 | 20:14 |
| rlandy|ruck | periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master | 20:15 |
| ade_lee_ | going to get kids - back in a little bit | 20:15 |
| rlandy|ruck | ade_lee_: pls ping when you get back - re: fs039 on c8, victoria, ussuri, wallaby etc. https://bugs.launchpad.net/tripleo/+bug/1962727 | 20:26 |
| dviroel | rlandy|ruck: found this https://opendev.org/openstack/tripleo-heat-templates/commit/d56a191590b669249569e01d8507b00849c85fb4 | 20:32 |
| rlandy|ruck | yatinkarel 1 year ago | 20:33 |
| dviroel | fix lives in master and wallaby | 20:33 |
| dviroel | if is failing on wallaby too - might not be related | 20:34 |
| rlandy|ruck | dviroel: alos failing wallaby c8 | 20:34 |
| rlandy|ruck | https://logserver.rdoproject.org/62/40162/5/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby/0d09301/ | 20:34 |
| dviroel | ok - will investigate a bit more | 20:35 |
| rlandy|ruck | dviroel: whatever it is, it's not in train | 20:58 |
| rlandy|ruck | but in all branches other than master | 20:58 |
| rlandy|ruck | dviroel: could be in master and wallaby as well and we are failing before | 20:58 |
| dviroel | if we are failing in ipa-server setup in master and wallaby - we are failing before | 20:59 |
| dviroel | ipa-client is already testing connection with the server right | 21:00 |
| dviroel | and failing | 21:00 |
| dviroel | maybe this output can help on debug: https://github.com/freeipa/ansible-freeipa/blob/master/roles/ipaclient/tasks/install.yml#L30-L53 | 21:02 |
| dasm | dviroel: can you re+w tnis one? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40058 resolved merge conflict | 21:23 |
| * dviroel llooking | 21:24 | |
| dviroel | easy one, done | 21:25 |
| dasm | thx | 21:25 |
| dviroel | rlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/831569 - didn't work? | 21:34 |
| ade_lee_ | rlandy|ruck, back | 21:38 |
| rlandy|ruck | dviroel: ade_lee_: let's discuss ... dasm you can join if you want as well | 21:38 |
| dasm | joining | 21:39 |
| ade_lee_ | link? | 21:39 |
| rlandy|ruck | https://meet.google.com/tyh-egcr-pmo | 21:39 |
| rlandy|ruck | https://logserver.rdoproject.org/63/35663/19/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria/056d30f/logs/undercloud/home/zuul/ | 22:04 |
| rlandy|ruck | https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/baremetal-full-freeipa.yml | 22:08 |
| dasm | https://logserver.rdoproject.org/75/40175/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/9d2162d/logs/supplemental/home/cloud-user/deploy_freeipa.log.txt.gz | 22:15 |
| dasm | > openssl-perl-1:3.0.1-14.el9.x86_64 | 22:15 |
| dasm | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116/4/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml | 22:16 |
| dasm | ade_lee_: ^ | 22:16 |
| rlandy|ruck | https://logserver.rdoproject.org/75/40175/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/9d2162d/logs/supplemental/home/cloud-user/deploy_freeipa.sh.txt.gz | 22:22 |
| rlandy|ruck | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/freeipa-setup/templates/deploy_freeipa.sh.j2#L57 | 22:23 |
| * dviroel going out - dinner o/ | 22:25 | |
| *** dviroel is now known as dviroel|out | 22:25 | |
| rlandy|ruck | woohoo - train promoting | 22:25 |
| rlandy|ruck | http://promoter.rdoproject.org/promoter_logs/container-push/20220303-112452.log-20220304 | 22:25 |
| rlandy|ruck | one release - it's a start | 22:25 |
| ade_lee_ | rlandy|ruck, this is where ipa_prep is being run -- https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/ipa_prep.yml | 22:29 |
| ade_lee_ | its in the role undercloud-deploy | 22:29 |
| rlandy|ruck | chandankumar, https://review.rdoproject.org/r/c/testproject/+/40169 is a noop, https://review.opendev.org/c/openstack/tripleo-ci/+/832254/ fails | 22:32 |
| ade_lee_ | rlandy|ruck, which is called from https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L32-L35 | 22:32 |
| ade_lee_ | rlandy|ruck, I'm guessing that this is causing the resolv.conf to be changed -- https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L65-L66 | 22:33 |
| rlandy|ruck | possibly | 22:34 |
| rlandy|ruck | let | 22:34 |
| rlandy|ruck | s review that | 22:34 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-operator-ansible/commits/branch/master/roles/tripleo_undercloud_install | 22:35 |
| rlandy|ruck | not much changed | 22:35 |
| rlandy|ruck | https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_undercloud_install | 22:35 |
| ade_lee_ | rlandy|ruck, so this is interesting .. | 22:41 |
| ade_lee_ | rlandy|ruck, so the first thing we do is https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L32-L35 | 22:42 |
| ade_lee_ | rlandy|ruck, then we do this -- https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml#L39-L41 | 22:43 |
| ade_lee_ | rlandy|ruck, actually -- I'm not sure .. maybe the thing to do is to add some "printfs" :) | 22:44 |
| ade_lee_ | rlandy|ruck, that is add some tasks to print out the /etc/resolv.conf | 22:44 |
| ade_lee_ | rlandy|ruck, in https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/main.yml | 22:44 |
| dasm | rlandy|ruck: multinode ipa failed badly on tempest: https://logserver.rdoproject.org/03/40103/3/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/4215d08/logs/undercloud/var/log/tempest/stestr_results.html.gz | 22:45 |
| ade_lee_ | say at line 36, 42, | 22:45 |
| dasm | > Failed to establish a new connection: [Errno 111] Connection refused')) | 22:45 |
| ade_lee_ | 61 | 22:46 |
| dasm | we might need to change something with the way how we run tests. | 22:46 |
| ade_lee_ | at least we'll know if we're going into the undercloud install with the right /etc/resolv.conf | 22:46 |
| ade_lee_ | dasm, is keystone actually up? | 22:48 |
| rlandy|ruck | dasm: re: tempest | 22:48 |
| rlandy|ruck | idk | 22:48 |
| dasm | i see some of tests passing | 22:48 |
| rlandy|ruck | you can try the full downgrade | 22:48 |
| * rlandy|ruck is so tired of all these failures | 22:48 | |
| rlandy|ruck | you'll need to update the condition | 22:50 |
| dasm | rlandy|ruck: https://i.pinimg.com/originals/c6/80/ea/c680eafa53543372915f778e67a78b7c.jpg | 22:50 |
| dasm | hmm... there were plenty of issues with mysql and amqp: client unexpectedly closed TCP connection | 22:55 |
| dasm | still checking logs | 22:55 |
| rlandy|ruck | ade_lee_: all these roles have not changed in ages though | 23:10 |
| rlandy|ruck | looking through changes in wallaby | 23:11 |
| rlandy|ruck | http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos8-wallaby/component/tripleo/3a/d4/3ad413310b84b038157b2d02ad7b0ff10264f152_bfb6fe68/?C=M;O=D | 23:11 |
| rlandy|ruck | on that day | 23:11 |
| rlandy|ruck | dasm: do they look related to ipa? | 23:13 |
| rlandy|ruck | dequeuing master | 23:14 |
| rlandy|ruck | it's hanging | 23:14 |
| rlandy|ruck | dasm: pls try one more run on the multinode-ipa job to see if tempest tests failure repeat | 23:18 |
| rlandy|ruck | maybe we will get lucky | 23:18 |
| rlandy|ruck | if they do - pls leave notes on tripleo-ci chat for ruck/rovers | 23:18 |
| rlandy|ruck | on both that failure and fs039 | 23:18 |
| rlandy|ruck | for master | 23:18 |
| rlandy|ruck | dasm: they start earlier than I do | 23:18 |
| rlandy|ruck | so pls pass info there | 23:18 |
| rlandy|ruck | 2022-03-07 22:05:52.626483 | fa163e2c-1505-77b1-3465-0000000001f1 | TIMING | tripleo_bootstrap : Reload NetworkManager | undercloud | 0:00:10.104783 | 0.35s | 23:23 |
| rlandy|ruck | ade_lee_: ^^ wonder if that is still an issue | 23:23 |
| rlandy|ruck | resetting dns | 23:23 |
| rlandy|ruck | ade_lee_: https://opendev.org/openstack/tripleo-ansible/commit/9d0da2b383239d296b28ecb3b2458543dc0f0cb4 | 23:26 |
| dasm | k | 23:26 |
| dasm | done. let's hope for better test run | 23:32 |
| dasm | rlandy|ruck: multinode ipa (non-voting) passed here: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832116 | 23:33 |
| * dasm afk-ing | 23:36 | |
| *** dasm is now known as dasm|off | 23:36 | |
| rlandy|ruck | ade_lee_: https://composes.centos.org/latest-CentOS-Stream-8/compose/BaseOS/x86_64/os/Packages/?C=M;O=A | 23:39 |
| rlandy|ruck | network manager rpms got an update on c8 recently | 23:40 |
| rlandy|ruck | nah - scratch that - same in passing and failing jobs | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
