| *** bhagyashris is now known as bhagyashris|ruck | 04:38 | |
| *** pojadhav|out is now known as pojadhav | 05:26 | |
| * bhagyashris|ruck brb | 06:10 | |
| *** ysandeep|out is now known as ysandeep | 06:28 | |
| ysandeep | bhagyashris|ruck, o/ please let me know if you need a hand with rr. | 06:30 |
|---|---|---|
| *** amoralej|off is now known as amoralej | 07:11 | |
| * pojadhav stepping out for urgent work for few hours. | 08:01 | |
| *** jpena|off is now known as jpena | 08:05 | |
| * soniya29 dr appt | 08:06 | |
| marios | biab errand o/ | 09:49 |
| *** bhagyashris_ is now known as bhagyashris|ruck | 09:55 | |
| *** ysandeep is now known as ysandeep|afk | 10:02 | |
| *** chem is now known as Guest468 | 10:09 | |
| *** bhagyashris_ is now known as bhagyashris|ruck | 10:44 | |
| *** rlandy|out is now known as rlandy|ruck | 10:48 | |
| rlandy|ruck | chandankumar: hey | 10:50 |
| chandankumar | rlandy|ruck: hello | 10:50 |
| rlandy|ruck | chandankumar: need to meet with you for a few pls | 10:50 |
| chandankumar | rlandy|ruck: sure | 10:50 |
| rlandy|ruck | chandankumar: https://meet.google.com/dpx-usxm-ffd?pli=1&authuser=0 | 10:50 |
| chandankumar | akahat: review time | 11:17 |
| rlandy|ruck | bhagyashris|ruck: hey https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone&skip=0 | 11:18 |
| bhagyashris|ruck | rlandy|ruck, yeah looking on the same | 11:19 |
| rlandy|ruck | bhagyashris|ruck: anything else burning? | 11:19 |
| rlandy|ruck | bhagyashris|ruck: on review time - will look in a bit with you if need be | 11:21 |
| bhagyashris|ruck | rlandy|ruck, ack | 11:22 |
| bhagyashris|ruck | rlandy|ruck, hey i have already filled bug for ^ issue https://bugs.launchpad.net/tripleo/+bug/1962298 | 11:26 |
| bhagyashris|ruck | looks like tempest tests failure failing most of the jobs | 11:26 |
| rlandy|ruck | bhagyashris|ruck: we need to see which review caused this | 11:39 |
| rlandy|ruck | the failure | 11:39 |
| rlandy|ruck | so we can get a fix | 11:39 |
| rlandy|ruck | we can skip the test to clear the gates | 11:39 |
| bhagyashris|ruck | rlandy|ruck, ack | 11:39 |
| rlandy|ruck | but there must be a change that makes this fail | 11:40 |
| rlandy|ruck | bhagyashris|ruck: hi - let's sync | 11:52 |
| bhagyashris|ruck | rlandy|ruck, sure | 11:52 |
| rlandy|ruck | https://meet.google.com/tav-psyj-nfh?pli=1&authuser=0 | 11:52 |
| bhagyashris|ruck | rlandy|ruck, giv me min | 11:53 |
| *** ysandeep|afk is now known as ysandeep | 12:00 | |
| bhagyashris|ruck | rlandy|ruck, https://trello.com/c/2CAk8C4p/2330-cixlp1959328tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicopstestnetworkbasicops-failing-on-periodic-tripleo-c | 12:10 |
| frenzyfriday|rover | hey rlandy|ruck, I couldnt find anything more on https://bugs.launchpad.net/tripleo/+bug/1960661 | 12:11 |
| bhagyashris|ruck | rlandy|ruck, https://bugs.launchpad.net/tripleo/+bug/1958531 | 12:11 |
| rlandy|ruck | soniya29: kopecmartin: hello ... we're looking at a master c9 failure on tempest: https://bugs.launchpad.net/tripleo/+bug/1962298 - did anything change recently on the tempest side? | 12:25 |
| bhagyashris|ruck | rlandy|ruck, fyi https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/830978 | 12:33 |
| rlandy|ruck | https://opendev.org/openstack/puppet-tempest/commit/4f7566fe806a75bba2c43e993f4ed9d81c961661? | 12:34 |
| rlandy|ruck | bhagyashris|ruck: pls send an mail to openstack-discuss saying we have a gate blocker | 12:37 |
| rlandy|ruck | example: | 12:37 |
| rlandy|ruck | see openstack-discuss for [TripleO] Gate blocker - ansible version | 12:38 |
| rlandy|ruck | title it [TripleO] Gate blocker - centos-9 standalone and multinode | 12:38 |
| rlandy|ruck | marios: ^^ fyi | 12:38 |
| rlandy|ruck | your patches will need a recheck after we resolve this | 12:38 |
| bhagyashris|ruck | rlandy|ruck, ack | 12:39 |
| marios | rlandy|ruck: thanks | 12:46 |
| rlandy|ruck | soniya29" hey - you around?? | 12:48 |
| *** frenzyfriday|rover is now known as frenzyfriday | 12:49 | |
| soniya29 | rlandy|ruck, yupp | 12:51 |
| rlandy|ruck | soniya29: we need some tempest help | 12:52 |
| rlandy|ruck | pls see https://bugs.launchpad.net/tripleo/+bug/1962298 | 12:52 |
| rlandy|ruck | note the failure started today at 5am UTC | 12:52 |
| rlandy|ruck | centos 9 only | 12:52 |
| rlandy|ruck | tempest only | 12:52 |
| soniya29 | rlandy|ruck, thanks for letting me know, i will check | 12:53 |
| rlandy|ruck | soniya29: major gate blocker | 12:53 |
| rlandy|ruck | soniya29: we need to know if anything changed on the tempest side | 12:54 |
| rlandy|ruck | images used | 12:54 |
| rlandy|ruck | login required etc. | 12:54 |
| rlandy|ruck | soniya29: looking for any change in plugins/login etc | 12:55 |
| chandankumar | rlandy|ruck: marios https://review.opendev.org/c/openstack/tripleo-ci/+/830965/ is ready for review | 12:56 |
| chandankumar | container build is seriously confusing | 12:56 |
| chandankumar | *build parenting | 12:56 |
| *** ysandeep is now known as ysandeep|away | 12:58 | |
| soniya29 | rlandy|ruck, give me some time, i will let know | 13:01 |
| marios | chandankumar: remove WIP | 13:04 |
| marios | chandankumar: ack adding to reviews list | 13:04 |
| marios | chandankumar: maybe add 2 lines explanation please | 13:04 |
| *** amoralej is now known as amoralej|lunch | 13:21 | |
| rlandy|ruck | dviroel: hey ... can you help me run a standalone c9 test on a previous compose? | 13:26 |
| dviroel | rlandy|ruck: sure | 13:30 |
| rlandy|ruck | dviroel: https://meet.google.com/vtw-cuzb-cnj?pli=1&authuser=0 | 13:31 |
| dviroel | rlandy|ruck: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/dependencies-jobs.yaml#L19-L23 | 13:31 |
| *** ysandeep|away is now known as ysandeep | 13:42 | |
| rlandy|ruck | bhagyashris|ruck; trying some experiments with older compose and downgrading openssl | 13:44 |
| * bhagyashris|ruck out for 2 hrs | 13:55 | |
| *** amoralej|lunch is now known as amoralej | 14:03 | |
| soniya29 | frenzyfriday, pojadhav, is python-tempestconf gate unstable, did you find any issues related to it? | 14:13 |
| pojadhav | soniya29, i didnt came across because was mostly focusing on downstream. frenzyfriday might have some clues.. | 14:15 |
| soniya29 | pojadhav, okay | 14:15 |
| frenzyfriday | soniya29, nope, I didnt find anything | 14:25 |
| soniya29 | frenzyfriday, okay | 14:25 |
| ysandeep | afaik... we don't monitor python-tempestconf gates, i don't think that's tracked in our cockpit. | 14:26 |
| soniya29 | rlandy|ruck, please give me 5 mins for rbac meeting | 14:28 |
| rlandy|ruck | dviroel: hey - can you add Tengu to https://github.com/orgs/tripleo-ci-director-op/people? | 14:29 |
| rlandy|ruck | or whatever else he needs to login | 14:29 |
| Tengu | my github username is "cjeanner" :) | 14:29 |
| dviroel | adding | 14:30 |
| Tengu | thanks | 14:30 |
| dviroel | invitation sent | 14:31 |
| * dviroel biab | 14:31 | |
| soniya29 | rlandy|ruck, i am in | 14:32 |
| rlandy|ruck | soniya29: sorry - on call with Tengu - will ping you later | 14:32 |
| soniya29 | rlandy|ruck, sure | 14:32 |
| Tengu | it's working, I get access to the board now :) | 14:44 |
| Tengu | thanks! | 14:44 |
| dviroel | \o/ | 14:45 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-wallaby/93b38f3/logs/undercloud/var/log/tempest/stestr_results.html.gz | 14:46 |
| rlandy|ruck | dviroel: ^^ ha - the node we messed with | 14:46 |
| rlandy|ruck | soniya29: sorry - getting on call now | 14:46 |
| dviroel | rlandy|ruck: oh, nice catch | 14:49 |
| dviroel | Tengu: you are cluster-admin now | 14:49 |
| rlandy|ruck | trying the same on the master nodw | 14:50 |
| rlandy|ruck | node | 14:50 |
| rlandy|ruck | soniya29: dropped - let me know when you are back | 14:50 |
| rlandy|ruck | amoralej: hello ... think the latest c9 compose is killing us again :( ... | 14:51 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1962298 | 14:52 |
| rlandy|ruck | testing with the previous compose | 14:52 |
| rlandy|ruck | hacked up one node with downgraded openssl | 14:52 |
| *** ysandeep is now known as ysandeep|out | 14:52 | |
| rlandy|ruck | not 100% sure yet if that is the real issue | 14:52 |
| *** dviroel is now known as dviroel|lunch | 14:57 | |
| amoralej | rlandy|ruck, i need to leave now, sorry | 14:59 |
| amoralej | i've asked joel to check | 14:59 |
| amoralej | you can also ask apevec | 15:00 |
| rlandy|ruck | amoralej: np - adding an exclude to our jobs | 15:00 |
| rlandy|ruck | to start with | 15:00 |
| *** amoralej is now known as amoralej|off | 15:00 | |
| rlandy|ruck | putting in excludes patch to test | 15:03 |
| rlandy|ruck | frenzyfriday: ok - sorry - working on c9 issue | 15:05 |
| rlandy|ruck | can you put back the clean up scripts | 15:05 |
| rlandy|ruck | for OVB stacks | 15:05 |
| rlandy|ruck | will deal with it next week | 15:05 |
| * soniya29 dinner | 15:08 | |
| *** ykarel is now known as ykarel|away | 15:22 | |
| soniya29 | rlandy|ruck, let's meet | 15:46 |
| rlandy|ruck | on call with phil | 15:47 |
| soniya29 | rlandy|ruck, okay | 15:49 |
| rlandy|ruck | https://review.rdoproject.org/r/c/testproject/+/36256 | 15:53 |
| rlandy|ruck | bhagyashris|ruck: dviroel|lunch: ^^ point made | 15:53 |
| rlandy|ruck | patch in test | 15:53 |
| rlandy|ruck | dviroel|lunch: thanks for the help | 15:53 |
| *** dviroel|lunch is now known as dviroel | 16:04 | |
| dviroel | rlandy|ruck: both passed? | 16:05 |
| rlandy|ruck | dviroel: I hacked the original node | 16:06 |
| dviroel | ah ok | 16:06 |
| rlandy|ruck | Exclude openssl 3.0.1-12 from C9 stream install https://review.opendev.org/c/openstack/tripleo-quickstart/+/830996 | 16:06 |
| rlandy|ruck | dviroel: ^^ | 16:06 |
| rlandy|ruck | openssl comes from the nodepool image | 16:06 |
| rlandy|ruck | so need to remove it from repos and downgrade | 16:06 |
| rlandy|ruck | dviroel: https://review.opendev.org/c/openstack/tripleo-quickstart/+/830996 under test now | 16:08 |
| rlandy|ruck | if it passes, we will need to merge that asap | 16:08 |
| rlandy|ruck | there is probably some way to work with new openssl | 16:08 |
| rlandy|ruck | but we need to clear gate and work it out | 16:08 |
| dviroel | rlandy|ruck: agree - we can work on a fix in meantime | 16:09 |
| ade_lee | dviroel, hey - how things going? | 16:15 |
| dviroel | ade_lee: o/ | 16:16 |
| ade_lee | rlandy|ruck, did I see something about you guys needing to work with a new openssl? | 16:16 |
| rlandy|ruck | ade_lee: yes :( | 16:17 |
| dviroel | ade_lee: scenario004 succeeded - cenario007-multinode-oooq-container also succeeded with featureset030 hack | 16:17 |
| rlandy|ruck | ade_lee: killed all our c9 work | 16:17 |
| rlandy|ruck | with newest version | 16:17 |
| ade_lee | rlandy|ruck, dviroel the reason I ask is because I was investigating a failure in one of the centos 9 fips jobs due to certmonger not being able to create its creds file under fips | 16:18 |
| ade_lee | turns out it fails due to this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2049265 | 16:18 |
| ade_lee | which is fixed by a newer version of openssl | 16:19 |
| ade_lee | openssl-3.0.1-11.el9 | 16:19 |
| rlandy|ruck | ade_lee: ack 3.0.1-5 worked | 16:19 |
| rlandy|ruck | new compose has 3.0.1.-12 | 16:19 |
| rlandy|ruck | hosed | 16:20 |
| * rlandy|ruck gets | 16:20 | |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1962298 | 16:20 |
| rlandy|ruck | ade_lee: hacked a fix to downgrade the version | 16:20 |
| rlandy|ruck | but I gather there is a right way to work with the newer version | 16:20 |
| rlandy|ruck | reading bug | 16:20 |
| ade_lee | rlandy|ruck, so 3.0.1-5 works and 3.0.1-12 is broken? | 16:21 |
| ade_lee | dviroel, sweet :) | 16:22 |
| rlandy|ruck | ade_lee: ack | 16:23 |
| ade_lee | rlandy|ruck, and of course I need somthing that went into 3.0.1-11 to fix the BZ I pointed you to in fips | 16:24 |
| rlandy|ruck | ade_lee: may be related in how we set up tempest - maybe we need to do something to work with latest | 16:24 |
| rlandy|ruck | you can try run your jobs now | 16:25 |
| rlandy|ruck | your stuff could work | 16:25 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart/+/830996/ | 16:25 |
| ade_lee | rlandy|ruck, when did the openssl newest version land? | 16:25 |
| rlandy|ruck | ade_lee: this morning 5am | 16:26 |
| rlandy|ruck | from stream compose | 16:26 |
| rlandy|ruck | I'd happily use the new version if I knew what to edit to make it work | 16:26 |
| * rlandy|ruck starting to dread stream | 16:26 | |
| marios | have a good weekend tripleo-ci o/ | 16:30 |
| dviroel | marios: o/ | 16:36 |
| *** marios is now known as marios|out | 16:42 | |
| dviroel | ade_lee: rlandy|ruck https://centos.pkgs.org/9-stream/centos-baseos-x86_64/openssl-3.0.1-12.el9.x86_64.rpm.html - all fixes from Feb are new in 3.0.1-12 | 16:44 |
| dviroel | maybe: - Disable SHA1 signature creation and verification by default | 16:45 |
| dviroel | https://bugzilla.redhat.com/show_bug.cgi?id=2031742 | 16:47 |
| dviroel | last comment ^ | 16:47 |
| rlandy|ruck | but not in compose | 16:52 |
| *** jpena is now known as jpena|off | 16:56 | |
| dviroel | yeah, not yet, merged hours ago | 16:59 |
| * dviroel coffee | 17:02 | |
| rlandy|ruck | ade_lee: if we merge the openssl downgrade, will that mess with your plans??? | 19:13 |
| rlandy|ruck | if so, we need to work out how to use this newer version | 19:13 |
| ade_lee | rlandy|ruck, let me see if the bug is present with the lower openssl | 19:26 |
| rlandy|ruck | ugh node provision failures in check | 19:57 |
| rlandy|ruck | dviroel: rcastillo: going to w+ https://review.opendev.org/c/openstack/tripleo-quickstart/+/830996/ | 19:59 |
| rlandy|ruck | check in OVB is hosed | 19:59 |
| rlandy|ruck | with node provision errors | 19:59 |
| rlandy|ruck | that started yesterday so probably unrelated | 20:00 |
| dviroel | rlandy|ruck: ack | 20:00 |
| rcastillo | rlandy|ruck: ack | 20:00 |
| rlandy|ruck | integration lines get past node provision | 20:00 |
| rlandy|ruck | hopefully promotion will clear it up | 20:00 |
| ade_lee | rlandy|ruck, dviroel actually, I'm sure that 3.0.1-5 definitely contains the bug I have for fips | 20:07 |
| ade_lee | rlandy|ruck, so ideally we'll need the later version at some point | 20:08 |
| rlandy|ruck | ade_lee: ack- we will need to work on a real fix on monday | 20:08 |
| ade_lee | ack | 20:08 |
| rlandy|ruck | still testing this with image and container builds | 20:08 |
| * dviroel hopes that a new openssl release will fix that issue | 20:09 | |
| *** dviroel is now known as dviroel|afk | 20:21 | |
| kopecmartin | rlandy|ruck: hi, sorry for not replying, i was half day (afternoon) on pto (i really should started updating my nick) ... i see the failure was in openssl at the end? | 21:12 |
| rlandy|ruck | kopecmartin: yeah - I think so | 21:15 |
| rlandy|ruck | patch in the gate to downgrade the version | 21:15 |
| rlandy|ruck | hopefully will hold us over | 21:15 |
| rlandy|ruck | rcastillo: ho ware you doing with the virt deploy? | 21:19 |
| rlandy|ruck | if you need help, please reach out | 21:19 |
| rcastillo | rlandy|ruck: added this fix for c9 libvirt https://review.opendev.org/c/openstack/tripleo-quickstart/+/831053 | 21:35 |
| rcastillo | affected me locally, can't tell whether it affects CI yet | 21:35 |
| rlandy|ruck | ok | 21:35 |
| rlandy|ruck | trying to get your other patch in | 21:35 |
| rlandy|ruck | after the openssl change merges | 21:36 |
| rlandy|ruck | with ^^ are you further along? locally? | 21:36 |
| rcastillo | yeah | 21:39 |
| rcastillo | it's currently running, waiting for it to fail | 21:39 |
| rlandy|ruck | ok | 22:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!