Tuesday, 2021-12-14

*** rlandy|ruck is now known as rlandy|out00:54
*** ysandeep|out is now known as ysandeep05:27
jfrancoaysandeep: good morning Sandeep, sorry to ask again but I need to ask you again to hold the node from this job https://review.rdoproject.org/r/c/testproject/+/36859 ... could you do it, please? we were pretty sure we found out where the issue lays at, but the job is not collecting any logs at all now...not even in the rdo testproject05:56
jfrancoaysandeep: so I would like to jump into the node and have a look05:57
ysandeepjfrancoa, sure, let me see how we can push your keys using testproject itself. 05:58
jfrancoaysandeep: thanks a lot05:58
jfrancoaysandeep++05:58
ysandeepjfrancoa, can we remove the node from hold which we put yesterday?05:59
jfrancoaysandeep: yes please05:59
ysandeepack, thanks!05:59
ysandeepjfrancoa, I have added node on hold and pinged infra to add your keys, Once they add your keys.. I will ping you credentials.. 06:13
jfrancoaysandeep: thanks a lot, let's see if I can find out this time whats going on06:15
*** sshnaidm|afk is now known as sshnaidm06:57
ysandeepfolks o/ please add https://review.rdoproject.org/r/c/rdo-jobs/+/37139 on your review list07:16
bhagyashrisfolks kindly add into your review list https://review.rdoproject.org/r/c/rdo-jobs/+/36994 thanks :)07:22
*** ysandeep is now known as ysandeep|lunch07:26
*** amoralej|off is now known as amoralej08:15
*** ysandeep|lunch is now known as ysandeep08:41
akahatysandeep, o/09:09
ysandeephello akahat 09:10
akahatysandeep, looks like rlandy found way to run sc-10 in downstream.09:12
ysandeepakahat, :) I saw her comments on chat, but was not sure where she left off, Lets wait for her to come online then we can discuss09:13
akahatysandeep, sure np.09:14
chandankumaranbanerj|ruck: hello, please have a look https://review.opendev.org/c/openstack/tripleo-ci/+/780950, may be rebase will help, when free, thanks!09:22
marioschandankumar: looks like that revert helped ? 09:24
marioschandankumar: i think provision passed now in your test 09:24
marioshttps://logserver.rdoproject.org/13/36713/7/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/3cef704/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz09:24
chandankumarmarios: yes, but there are objections from Ian 09:25
marioschandankumar: k thanks i'll add it for fs1 for now09:25
chandankumarmarios: https://bugs.launchpad.net/tripleo/+bug/1954456/comments/709:25
marioschandankumar: ah but fs1 isn't building though hmm 09:25
marioschandankumar: no it is actually ... k 09:26
*** ysandeep is now known as ysandeep|afk09:35
anbanerj|ruckchandankumar, ack, checking09:37
chandankumaranbanerj|ruck: please recheck this patch also https://review.rdoproject.org/r/c/testproject/+/36508 so that we can clear sc7 cs909:40
jpodivinI've found a duplicate definition for one of VF jobs, no idea how we missed it, but it didn't cause any problems so far. https://review.rdoproject.org/r/c/rdo-jobs/+/3719609:41
anbanerj|ruckchandankumar, ack, rebased the job def and restarted the testP09:45
chandankumaranbanerj|ruck: thanks!09:45
anbanerj|ruckhey chandankumar, I saw this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821106 . But the C8 standalone-upgrade jobs ussuri/victoria are again hitting https://bugs.launchpad.net/tripleo/+bug/195332610:11
anbanerj|ruckchandankumar, https://logserver.rdoproject.org/95/37195/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/a99e761/job-output.txt I can see Downgrade pcs running. Why are we running Return current pcs version after that?10:11
chandankumaranbanerj|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82152110:12
chandankumaranbanerj|ruck: can you test the ussuri/victoria job with above patch?10:13
anbanerj|ruckthanks, putting depends on10:13
jbadiapahi, I've just tried to deploy tripleo-quickstart with cs9. 10:21
jbadiapabut it failed10:21
jbadiapaon the provisioning stage10:21
jbadiapaIt failed on the "modify-image : Run virt-customize on the provided image"10:21
jbadiapathe error is "command: mount '-o' 'ro' '/dev/sda' '/sysroot//'"10:22
jbadiapa'[    3.024065] XFS (sda): SB validate failed with error -22.'10:22
jbadiapais there any package that is needed to be able to run cs9 as guest on cs8 host?10:23
jbadiapashould I file a launchpad? 10:23
mariosjbadiapa: o/ hey i don't know if it is the same issue but, we had some problems with virt-customize for cs9 (no solution yet/it has been low priority) bug there... https://bugs.launchpad.net/tripleo/+bug/1950453 ... in that case the command it tries to run is https://opendev.org/openstack/tripleo-ci/src/commit/e6a036545aa0d3dd543562703b0f87f3267dacd2/roles/oooci-build-images/tasks/image_sanity.yaml#L24 ... 10:24
mariosfyi10:25
jbadiapamarios, thanks10:26
*** ysandeep|afk is now known as ysandeep10:32
*** dviroel|rover|afk is now known as dviroel|rover11:01
chandankumarsshnaidm: I am creating a new review on top of yours to put remaining standalone jobs11:11
*** rlandy is now known as rlandy|ruck11:13
dviroel|roveroh, train c7 promoted11:16
rlandy|ruckdviroel|rover: ussuri still not11:17
rlandy|ruckdviroel|rover: did you rekick the testproject?11:17
dviroel|roverrlandy|ruck: yes, yesterday, but with no sucess11:18
dviroel|roveranbanerj|ruck: is tp'ing atm11:18
dviroel|roveranbanerj|ruck: i think that you already tested with pcs revert too, right?11:19
rlandy|ruckwe may need to pull marios back into this one11:19
dviroel|roverrlandy|ruck: ^11:19
rlandy|ruckno luck11:19
anbanerj|ruckdviroel|rover, rlandy|ruck Hey 0/ Yes, testing with pcs revert - https://review.rdoproject.org/r/c/testproject/+/3719511:19
mariosrlandy|ruck: o/ ? ok i can have a look after i finish what i'm currently doing11:20
dviroel|roverrlandy|ruck: will dig into error to see if i can spot something. Try to check diffs between friday and monday, since it was passing on friday11:21
mariosussuri blocked again? 11:21
dviroel|rovermarios: standalone-upgrade-ussuri is fighting a war with us11:22
mariossounds about right . k dviroel|rover i am working on arxcruz upgrade patch right now but will have a look after this11:23
dviroel|rovermarios: sure, tks11:23
*** jpena|off is now known as jpena11:42
sshnaidmchandankumar, replied in https://review.opendev.org/c/openstack/tripleo-ci/+/821241 - we don't vote now, so it's fine to run on wallaby11:43
sshnaidmchandankumar, and we can know when it's ok if job starts passing11:43
jfrancoaanbanerj|ruck: hey, as the ruck...have you seen any ussury deployment work fail with https://bugs.launchpad.net/tripleo/+bug/1871703 recently? I've triggered my upgrades job from ussuri to victoria and it has failed due to exactly that bug: https://zuul.opendev.org/t/openstack/build/d0dd2d838f9840d2bcc42dcd58580779/log/logs/undercloud/home/zuul/standalone_deploy.log11:48
jfrancoaduring ussuri deployment11:48
anbanerj|ruckjfrancoa, not yet, these jobs are failing today with https://bugs.launchpad.net/tripleo/+bug/1953326. But lemme check11:51
rlandy|ruckbhagyashris: pls put back the planning meeting for thursday11:52
rlandy|ruckit's our next year's planning11:52
rlandy|ruck^^11:52
rlandy|ruckanbanerj|ruck: failed again11:56
rlandy|ruckjfrancoa: we;ve been going on a few bugs for ussuri upgrades11:57
rlandy|ruckblocked our promotion for 13 days11:57
rlandy|ruckonly the standalone11:57
rlandy|ruckthe current failure is logging now11:58
rlandy|ruckussuri itself should be passing11:58
anbanerj|ruckrlandy|ruck, checking after the logs are generated11:58
jfrancoarlandy|ruck: I saw that you had to revert the pcs package (I though for a moment that was the issue I was facing, but I'm still stuck on finding out why are my attempts to improve the stanadlone-upgrade failing and generating so much logs that the disk gets full)11:59
jfrancoaeverytime I say, I think I got it! It's not it.. :-D 11:59
rlandy|ruckjfrancoa: https://trunk.rdoproject.org/centos8-ussuri/component/tripleo/current/ - has marios revert on what was the previous bug12:00
rlandy|ruckso this testproject run should include both12:00
bhagyashrisrlandy|ruck, ack12:01
rlandy|ruckjfrancoa: anbanerj|ruck: ussuri just kicked again12:02
rlandy|ruckthe whole line12:02
rlandy|ruckdviroel|rover: do pcs was upgraded12:03
rlandy|ruckso we need to revert that workaround12:03
rlandy|ruckanbanerj|ruck included that patch12:03
jfrancoarlandy|ruck: oh..good to know. I'll wait for the revert and do the same on my changes12:04
jfrancoarlandy|ruck: thanks for hte heads up12:04
rlandy|ruckjfrancoa: even with the revert we have a problem12:04
dviroel|roverit seems that even with the pcs downgrade fix, we have the jobs running with pcs-0.10.11-2.el8.x86_64 (instead of pcs-0.10.11-1.el8.x86_64)12:04
*** soniya29 is now known as soniya29|afk12:04
dviroel|roverwhich version we have with the revert? checking..12:05
anbanerj|ruckdviroel|rover, with revert pcs-0.10.1212:05
rlandy|ruckdviroel|rover: anbanerj|ruck: jfrancoa: checking those logs now12:05
anbanerj|ruckhttps://logserver.rdoproject.org/95/37195/3/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/48079dd/logs/undercloud/var/log/extra/rpm-list.txt.gz12:05
rlandy|ruckhttps://logserver.rdoproject.org/95/37195/3/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/48079dd/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz12:06
rlandy|ruckstill issue has pcs 12:06
dviroel|roverpcs-0.10.12-1.el8.x86_64 is the newer, still not working12:06
rlandy|ruckdviroel|rover: anbanerj|ruck: instead of reverting the workaround12:07
rlandy|ruckwe need to change it12:07
rlandy|ruckto install the exact version we know works12:07
anbanerj|ruckI tried with version https://logserver.rdoproject.org/95/37195/2/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/12dd58b/logs/undercloud/var/log/extra/rpm-list.txt.gz as well12:07
anbanerj|ruckin case it was fixed12:08
rlandy|ruckanbanerj|ruck: dviroel|rover: let's meet12:08
anbanerj|ruckchecking which version worked last12:08
rlandy|ruckhttps://meet.google.com/smf-tjbw-wjt?pli=1&authuser=012:08
rlandy|ruckanbanerj|ruck: dviroel|rover: ^^12:08
rlandy|ruckjfrancoa: ^^ if you are interested12:09
rlandy|ruckpcs-0.10.11-1.el8.x86_6412:15
bhagyashrisrlandy|ruck, hey for logrotate testing is it ok to do the ussuri promotion using old promoter server?12:18
bhagyashrisold promoter server http://10.0.148.74/promoter_logs/12:20
rlandy|ruckbhagyashris: pls hold on using ussuri - we are trying to get that release promoted12:20
rlandy|ruckcan you try victoria?12:20
*** ysandeep is now known as ysandeep|brb12:21
bhagyashrisrlandy|ruck, np i will try victoria12:21
rlandy|ruckbhagyashris: thanks12:22
rlandy|ruckbhagyashris: can we move the retro an hour later?12:24
rlandy|ruckvery early for daruisz and rafael12:24
bhagyashrisrlandy|ruck, ok will do that12:24
rlandy|ruckbhagyashris: also - do we usually use 2 hours for planning?12:25
rlandy|ruck3 hours?12:25
bhagyashris3 hr12:25
rlandy|ruckbhagyashris: do we usually use all that time?12:25
bhagyashrisnope12:25
rlandy|ruckbhagyashris: do you think we could do it in 2 hours?12:25
rlandy|ruckand start an hour later?12:26
bhagyashrisin reality it's just 2 hrs12:26
bhagyashrisrlandy|ruck, sure12:26
rlandy|ruckbhagyashris: thanks - just trying to accommodate the newer people12:26
rlandy|ruckysandeep|brb: hey - pls ping when you have time to sync12:27
bhagyashrisrlandy|ruck, yeah12:27
rlandy|ruckbhagyashris: lastly - we added c9 topics to today's community call 12:27
rlandy|ruckand invited Tengu to join us12:27
bhagyashrisrlandy|ruck, ack12:27
rlandy|ruckakahat: so think we have a way to run sc01012:28
rlandy|ruckjust need to check with ysandeep|brb that we are not hosing anything12:28
rlandy|ruckbhagyashris++ thank you12:29
* Tengu waves12:31
dviroel|roverrlandy|ruck: anbanerj|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82169912:38
dviroel|roversanity check please12:38
dviroel|roveranbanerj|ruck: do you mind to testproject in one of your patches?12:38
anbanerj|ruckdviroel|rover, thanks! adding to the original tp of ussuri12:39
rlandy|ruckdviroel|rover++ nice12:41
rlandy|ruckthank you12:41
anbanerj|ruckhttps://review.rdoproject.org/r/c/testproject/+/37195 - testing here12:41
* anbanerj|ruck tea12:43
dviroel|rovertks12:43
* dviroel|rover coffee12:44
* bhagyashris tea12:48
*** ysandeep|brb is now known as ysandeep12:48
ysandeeprlandy|ruck, back12:49
rlandy|ruckysandeep: let's chat for 10 ... https://meet.google.com/ydx-sfhb-bpn?pli=1&authuser=012:50
Tengurlandy|ruck: fyi, the next LADA release will support package lookup for osp17-rhel913:03
rlandy|rucknice13:03
Tenguand this kind of things will be based on ospinfo, using a new "translator" I've provided for PCD (osp_release.yml).13:03
Tengutook me a while, with a dead brain, but.... it's almost in.13:04
dviroel|roveranbanerj|ruck: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa - 3 consecutive failures, same package issue13:04
anbanerj|ruckdviroel|rover, checking 13:06
mariosdviroel|rover: so do we have a new bug for ussuri or is it still this one https://bugs.launchpad.net/tripleo/+bug/1953326 13:10
mariosdviroel|rover: i see... the workaround fails and we are trying now with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 specific version instead 13:10
dviroel|rovermarios: it is still pcs issue, now we created a patch that downgrades to a specific pcs version13:10
dviroel|rovermarios: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82169913:11
dviroel|roveryes13:11
dviroel|roveranbanerj|ruck has a testproject running13:11
jfrancoarlandy|ruck: sorry, I was having lunch...I missed the party13:13
mariosanbanerj|ruck: sanity check did you mean to use that specific hash (i guess you need it for promotion?)                 dlrn_hash_tag: 0c303674a982c099b14bcb4198325ec713:15
mariosdviroel|rover: anbanerj|ruck: so you mean, it failed in the periodic line, even though we have the workaround in place? i guess this is why the revert was posted, and finally we are instead trying to downgrade to a specific version13:15
*** amoralej is now known as amoralej|lunch13:16
anbanerj|ruckmarios, yep - 0c30.. is the current hash under test. I copied directly from the rick rover script :D13:16
mariosanbanerj|ruck: thanks i see it http://promoter.rdoproject.org/promoter_logs/centos8_ussuri.log 13:17
dviroel|rovermarios: the revert that test newer versions didn't work13:18
anbanerj|ruckmarios, with the downgrade it passed in the original (sandeeps) testproject - there pcs was 10.11-1. Right now with downgrade the version is 10.11-2 which is not working. With revert the version is 10.12-1 which is not working either. So we are trying to go back to 10.11-113:18
mariosanbanerj|ruck: dviroel|rover: k so it sounds like it didnt' fail in the periodic line just in testproject (i.e. the workaround is still working right now?)13:22
mariosanbanerj|ruck: dviroel|rover: any idea why it didjn't run yesterday though? https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri&pipeline=openstack-periodic-integration-stable3 i see run from 12th (failing on different issue)13:22
mariosanbanerj|ruck: dviroel|rover: i was just trying to find if it fails in periodic with the workaround but sounds like no 13:22
* marios fetch coffee 13:22
anbanerj|ruckmarios, yes, the job initially failed on a different issue in the periodic line. I testprojected the job after https://review.opendev.org/c/openstack/tripleo-heat-templates/+/821264 i merged - then it started failing on the pcs issue13:23
bhagyashrisarxcruz, sshnaidm, rlandy, rlandy|ruck, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm13:27
bhagyashrisTengu, 13:27
bhagyashrisTripleO CI community call in 3 mins13:27
Tenguyup13:27
Tengupreparing my side :)13:27
bhagyashrisfolks feel free to add topic at line no 29 here https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-12-14-Community-Call13:28
bhagyashrishttps://meet.google.com/bqx-xwht-wky?authuser=013:28
dviroel|rovermarios: yesterday rdo zuul was restarted, maybe ussuri was affected13:28
mariosdviroel|rover: thanks13:29
ysandeeprlandy|ruck, approach i tried worked, details in https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826/2#message-f367e18389b84a7a2a910a87316a7260cfa25e75 , lets merge that if you agree13:29
mariosanbanerj|ruck: i see.. so it *would* have failed in the next periodic run i guess ? 13:29
marios:D13:29
ysandeepbhagyashris, will join few mins late today13:30
bhagyashrisysandeep, np13:30
anbanerj|ruckmarios, yes, i think so13:30
bhagyashrisrlandy|ruck, pojadhav- anbanerj|ruck chandankumar akahat 13:30
bhagyashris^13:31
bhagyashrissoniya29|afk, ^13:31
Tenguchandankumar: sorry, forgot to re-check about that nsenter thing -.-. As said, I'll take some time now.13:48
chandankumarTengu: I am reading about nsenter 13:48
Tengufor that I need a local reproducer though.13:48
Tenguchandankumar: there's a "--cgroup[=file]" param, but I don't think this is what we're looking for.13:49
Tenguchandankumar: do you happen to have a simple reproducer?13:49
chandankumarTengu: Nope13:49
Tengudang..13:49
Tengugetting a centos 9 env shortly.13:50
rlandy|ruckanbanerj|ruck: hey - any word on what's killing master/wallaby?13:50
rlandy|rucklots of failures there13:50
rlandy|rucknetwork line is also red13:51
rlandy|ruckdviroel|rover: ^^13:51
anbanerj|ruckmaster errors:13:51
anbanerj|ruck- periodic-tripleo-ci-centos-8-scenario002-standalone-master - barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningSnapshotTest, .ImageSigningTest, test_volume_encryption.VolumeEncryptionTest failed - server didnt reach active state - maybe this is a one time issue13:51
anbanerj|ruck- periodic-tripleo-ci-centos-8-scenario001-standalone-master - couldnt reach keystone endpoint - but no errors in keystone containet logs - rerunning13:51
anbanerj|ruck- periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master - package sssd-common-pac-2.5.2-2.el8_5.1.x86_64 requires libndr.so.1()(64bit), but none of the providers can be installed", "2021-12-14 10:05:29 |   - package sssd-common-pac-2.5.2-2.el8_5.1.x86_64 requires libndr.so.1(NDR_0.0.1)(64bit), but none of the providers can be installed", "2021-12-14 10:05:29 |   - cannot install both samba-client-libs-4.15.2-2.el8.x86_64 and 13:51
anbanerj|rucksamba-client-libs-4.14.5-2.el8.x86_64", "2021-12-14 10:05:29 | 13:51
anbanerj|ruck- cannot install both libipa_hbac-2.4.0-9.el8.x86_64 and libipa_hbac-2.5.2-2.el8_5.1.x86_64", "2021-12-14 10:05:29 |   - cannot install both libipa_hbac-2.4.0-9.el8_4.1.x86_64 and libipa_hbac-2.5.2-2.el8_5.1.x86_64", "2021-12-14 10:05:29 13:51
anbanerj|ruck^ this might be a real issue13:51
anbanerj|rucktestproj:  https://review.rdoproject.org/r/c/testproject/+/3719913:51
chandankumarTengu: may be recheck the testproject and get a node hold?13:51
* anbanerj|ruck checking the rpm failures13:51
rlandy|ruck2021-12-14 11:11:58.183395 | fa163e20-c6f4-4011-a861-000000003745 |      FATAL | Check Keystone admin endpoint status | undercloud | item=neutron | 13:52
dviroel|roveranbanerj|ruck: yes 13:52
rlandy|ruck FATAL | Check Keystone admin endpoint status | undercloud | item=swift |13:52
sshnaidmmarios, for running on all ci repos we'll need to add c9 jobs to every repo separately, or to include it in template "tripleo-multinode-branchful" which is only one that runs on CI repos only13:53
sshnaidmmarios, although it's neither multinode and not branchful :)13:53
sshnaidmmarios, I think we can include in this template for a few days? Just to save on patches13:54
sshnaidmrlandy|ruck, ^13:54
mariossshnaidm: yeah sure 13:54
mariossshnaidm: i think its fine... mainly just to see them running at least :)13:54
rlandy|rucksshnaidm: yep13:54
rlandy|rucksounds fine13:54
ysandeeprlandy|ruck, any objections merging https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826 ?13:54
rlandy|rucklet's get the minimal run in ci repos merged13:54
rlandy|ruckysandeep: works?13:55
rlandy|ruckif so, let's merge that13:55
ysandeeprlandy|ruck, yes https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/298826/2#message-f367e18389b84a7a2a910a87316a7260cfa25e75 13:55
Tenguchandankumar: nah - I'd rather get a simple, stupid container and try to talk to another one with a simple command in order to understand better the whole issue.13:55
Tenguchandankumar: shouldn't take much.13:55
chandankumarok13:55
rlandy|ruck ovn-2021                            x86_64  21.06.0-29.el8fdp rhosp-rhel-8.4-fdp    13:56
rlandy|ruckgood13:56
ysandeeprlandy|ruck, thanks for review ++13:56
*** soniya29|afk is now known as soniya2913:58
chandankumarmarios: sshnaidm: for standalone scenario will I move to tripleo-multinode-branchful ?13:58
marioschandankumar: yeah that's what sshnaidm proposed for now so it is easier to wire it up for ci repos14:01
sshnaidmchandankumar, yeah, I'll move sa job there14:01
chandankumarsshnaidm: can we nuke for wallaby for now? Since nothing is ready there14:01
chandankumarit does not looks good by seeing failing job14:02
sshnaidmchandankumar, I'll put them only for ci repos for now14:02
chandankumarok14:02
sshnaidmchandankumar, ok14:02
rlandy|ruckdviroel|rover: anbanerj|ruck: still not on ussuri upgrade14:03
rlandy|ruckjfrancoa: no worries the party is still on going14:04
rlandy|ruckhttps://review.rdoproject.org/zuul/stream/f8617286044c44f29b657b1f4d5eff53?logfile=console.log14:04
rlandy|ruck^^ still no pass on ussuri upgrades14:04
*** amoralej|lunch is now known as amoralej14:04
dviroel|roverrlandy|ruck: anbanerj|ruck     "msg": "Last metadata expiration check: 0:02:16 ago on Tue 14 Dec 2021 01:35:05 PM UTC.\nNo package pcs-0.10.11-1 available.\npcs-0.10.12-1.el8.x86_64"14:05
dviroel|roverpcs-0.10.11-1 is not available anymore14:06
rlandy|ruckdviroel|rover: anbanerj|ruck: this will fail gates as well14:06
rlandy|ruckmarios: ^^14:07
rlandy|ruckI think we need to remove this job from criteria and promote now14:07
rlandy|ruckthe error is in gates as well14:07
rlandy|ruckand mark this job non-voting again14:07
rlandy|ruckthis is going on too long14:08
mariosrlandy|ruck: thanks, waiting for that to report so we can get the logs 14:08
rlandy|ruckopen it again on CIX14:08
mariosrlandy|ruck: so we got a promotion but the workaround only lasted a day or two 14:08
rlandy|ruckmarios: no promotion14:08
rlandy|ruckthat hash failed on OVB14:08
mariosrlandy|ruck: but the ussuri promotion is 'only' 4 days old at this point right? 14:08
mariosrlandy|ruck: ah :/14:08
rlandy|ruck13 days agoussuricurrent-tripleocentos-83 days ago14:09
mariosrlandy|ruck: we have a bug for that ovb issue ? 14:09
rlandy|ruckmarios: that was fix14:09
rlandy|ruckussuri is running now14:09
mariosrlandy|ruck: so... in the time frame the workaround *was* working, ovb failed. now ovb is fixed but the workaround expired :/14:09
marios:/14:09
marioslavly14:09
rlandy|ruckexactly14:09
rlandy|rucklooking at the line now14:10
rlandy|ruckwe'll have to see what's the story14:10
anbanerj|ruckdviroel|rover, do we need maybe dnf downgrade pcs-0.10.12-1.el8 -y instead of cs-0.10.12-1 ?14:10
mariosanbanerj|ruck: can you let the test report before you update it though so we can get th elogs14:10
mariosah it reported already anbanerj|ruck k 14:11
* ysandeep out for the day14:12
*** ysandeep is now known as ysandeep|out14:13
dviroel|rovermarios: actually, your testproject that had upgrade-ussuri green was with report_dlrn: false14:14
mariosdviroel|rover: :/14:16
dviroel|roverhttps://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/HighAvailability/x86_64/os/Packages/14:19
mariosrlandy|ruck: 2021-12-14 13:37:22.929736 | primary |     "msg": "Last metadata expiration check: 0:02:16 ago on Tue 14 Dec 2021 01:35:05 PM UTC.\nNo package pcs-0.10.11-1 available.\npcs-0.10.12-1.el8.x86_64"14:19
dviroel|roverpcs-0.10.11-1 is still there14:19
rlandy|ruckmarios: no longer carried14:19
rlandy|ruckyou see it?14:19
dviroel|roveranbanerj|ruck: lets try with pcs-0.10.11-1.el8 instead, might be it14:28
* anbanerj|ruck crosses finger and prays to the ci god14:28
mariosdviroel|rover: :) wrote something similar just now https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699/1#message-3bd316ec872e38470bb01f109b4c31b869fc36da 14:28
rlandy|ruckamoralej: hello - touching base re:  weirdo jobs in master14:31
amoralejrlandy|ruck, working on it, https://review.opendev.org/c/x/packstack/+/821594 will fix it14:31
rlandy|ruckamoralej: great  - thank you14:31
amoraleji hope...14:31
dviroel|rovermarios: just checking in my centos env here14:32
mariosdviroel|rover: same but still not working anbanerj|ruck hold a sec not sure yet14:32
dviroel|rover0.10.8-1.el8 - is the version here, so we need to add el814:33
anbanerj|ruckfor my fedora sudo  dnf install pcs-0.10.8-1.fc32.x86_64 worked14:33
mariosdviroel|rover: is it working for you with ^^ ? .el8? if so lets try it 14:33
mariosanbanerj|ruck: so that is another thing we could try you know good idea anbanerj|ruck :D14:33
mariosdviroel|rover: i mean anbanerj|ruck 14:33
mariosinstead of downgrade 14:34
marioswe can remove and install specific version we want14:34
dviroel|rovermarios: thats what dnf downgrade does, if you try there14:34
mariosmaybe rpm -e --no-deps for the removal so we don't get rid of stuffs14:34
mariosdviroel|rover: ah i didn't realise it actually removes and installs the one you want14:34
dviroel|rovermarios: downgrade of my telegraf here https://paste.opendev.org/show/811663/14:36
dviroel|roverwith dnf downgrade14:36
dviroel|roverlets add full package name14:36
mariosk lets try it 14:37
dviroel|roverwill not add arch since the OS should solve that right14:37
mariosack14:37
dviroel|roverseems not right to force an arch14:37
amoralejmarios, sorry for interfere but,  why do we need to downgrade pcs? any known issue?14:38
amoralejor what's the issue?14:38
mariosamoralej: https://bugs.launchpad.net/tripleo/+bug/195332614:38
mariosamoralej: its a workaround for now14:38
amoralejups14:39
amoralejit may be better to blacklist in the repo config?14:39
mariosamoralej: ah pin to the version we want ? yes rdo-info i think?14:40
amoralejno, i mean in the .repo file created by oooq ?14:40
mariosamoralej: ah i see14:40
amoralejwe don't manage OS packages in rdoinfo14:40
mariosamoralej: yes this is also a good idea 14:40
mariosamoralej: k ... dviroel|rover ^^ we can try that as a next step update the ha .repo file in the release config14:41
mariosthanks amoralej 14:41
dviroel|roveryes, the right way to do it, this workaround was suppose to be very temp14:42
amoralejah i see you put the feedback in the bz, nice14:42
anbanerj|ruckdviroel|rover, thanks for rechecking https://review.rdoproject.org/r/c/testproject/+/37195. I am eod in ~10 mins. Updated the master tp in gchat14:43
dviroel|roveranbanerj|ruck: np, thanks anbanerj|ruck14:43
dasmo/14:54
*** dviroel|rover is now known as dviroel|rover|lunch14:59
chandankumaranbanerj|ruck: can I update your sc7 cs9 testproject?15:01
anbanerj|ruckchandankumar, sure15:02
chandankumarwe have got fix for sc7 cs9 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/82170915:02
mariostripleo-ci o/ please add to your reviews list https://review.opendev.org/c/openstack/tripleo-ci/+/816991/16#message-199f53afaae2f5a30fe54be0bdf3c57b348e8b54 15:13
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-component-network - network line is pretty bad15:20
marioschandankumar: well, fs1 also gets passed provision with https://review.opendev.org/c/openstack/diskimage-builder/+/821526 15:25
marioschandankumar: waiting for it to report and will add pointer in the bug 15:26
chandankumarmarios: awesome ++15:26
marioschandankumar: also fails in tempest like fs215:26
rlandy|ruckdviroel|rover|lunch: hey  - going to rekick the network line when it is done15:38
rlandy|ruckvery red on so many branches15:38
dviroel|rover|lunchrlandy|ruck: ack, thanks15:59
*** dviroel|rover|lunch is now known as dviroel|rover15:59
dviroel|roverDowngraded:\n  pcs-0.10.11-1.el8.x86_64                                                      \n\nComplete!\npcs-0.10.11-1.el8.x86_6416:08
dviroel|rover:)16:08
dviroel|rovermarios ^16:08
dviroel|roverlets see if the job will pass now16:08
mariosdviroel|rover: ack good 16:10
marios'good' ;) dviroel|rover 16:10
dviroel|rover:)16:10
mariostripleo-ci o/ please add to your reviews list https://review.opendev.org/c/openstack/tripleo-ci/+/816991/16#message-199f53afaae2f5a30fe54be0bdf3c57b348e8b54 16:27
mariosrlandy|ruck: when you next have time thanks ^ 16:27
*** marios is now known as marios|out16:46
rlandy|ruckdviroel|rover: upgrade done??16:49
rlandy|ruckfinally :)16:49
dviroel|roveroh yes, running tempest tests now16:50
dviroel|rover:)16:50
rlandy|ruckdviroel|rover: great  - let's get https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 merged17:07
dviroel|roverrlandy|ruck: +117:10
dviroel|rover0c303674a982c099b14bcb4198325ec7 is good for the promoter now :)17:13
rlandy|ruckdviroel|rover++ yep17:14
rlandy|ruckpromoting now17:14
rlandy|ruckdviroel|rover: k ... will merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/821699 when it passes check17:15
rlandy|ruckgates must be breaking now17:15
dviroel|rovercomponent-network looks bad17:16
*** amoralej is now known as amoralej|off17:26
*** jpena is now known as jpena|off17:37
akahatmoni_l18:00
rlandy|ruckdviroel|rover: and now zuul is gone18:47
rlandy|ruckdviroel|rover: on the upside ussuri promoted18:47
dviroel|roveryes :) and yes :(18:50
rlandy|ruckdviroel|rover: meeting19:04
dviroel|roverah, true, joining19:05
rlandy|ruckdviroel|rover++ rcastillo++ looking very nice - thanks for that work19:40
dviroel|rover\o/19:41
rlandy|ruckdviroel|rover: updating the program doc for tomorrow's meeting  you can decide with ananya who wants to present19:46
dviroel|roverrlandy|ruck: ok19:46
dviroel|roverrechecking pcs fix =/19:46
rlandy|ruckugh - zuul still awol20:05
rlandy|ruckdviroel|rover: pls ping when you are EoD - I have one meeting left - then I can pick up what's left to rerun20:05
dviroel|roverok20:05
dviroel|roverrlandy|ruck: going eod, there isn't too much to see while we didn't kick missed pipeline20:26
dviroel|roverrlandy|ruck: we need to run master and train c820:27
dviroel|roverrlandy|ruck: network-component has different failures, but I've seen pacemaker error more than once20:27
dviroel|roverrlandy|ruck: Error: Execution of '/bin/dnf -d 0 -e 1 -y install pacemaker' returned 1: Error: Error downloading packages:\n<13>Dec 14 13:05:25 puppet-user:   libwbclient-4.15.2-2.el8.x86_64: Cannot download, all mirrors were already tried without success\n20:28
rlandy|ruckdviroel|rover: ok - I'll log the network one20:28
rlandy|ruckand rerun the lines20:29
rlandy|ruckthanks dviroel|rover 20:29
dviroel|rovernetwork component error: https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-master/bc0b0b5/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz20:29
dviroel|rovero/20:31
*** dviroel|rover is now known as dviroel|out20:31
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/1954778 - I see master bug20:39
rlandy|rucktrain 8 rekicked20:44
rlandy|ruckmaster rekicked20:46
rlandy|ruckugh - we hosed container builds on wallaby21:57
rlandy|ruckchandankumar: ^^21:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!