Thursday, 2022-07-21

« Wednesday, 2022-07-20 Index Friday, 2022-07-22 »
*** dviroel|afk is now known as dviroel|out00:37
*** rlandy is now known as rlandy|out01:09
dasm|offthanks rlandy|out 02:53
dasm|offhmm. i don't see promotion yet02:54
dasm|offchkumar|rover: please check if i didn't mess up anything. I believe we should have cs9 master already but i don't see it.02:55
*** undefined is now known as Guest559803:35
dasm|offchkumar|rover: YES! WE HAVE IT!03:40
dasm|offWE GOT IT! Nice!03:42
*** ysandeep|out is now known as ysandeep05:00
ysandeepfrenzy_friday, I am changing topic of https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/417017 to add-17.1-rhel-9, imho.. lets keep single topic fpr 17.1 job so that we can easily track reviews.05:06
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.05:06
chkumar|roverrlandy|out: dasm|off well done, we got master promotion thank you thank you :-)05:59
chkumar|roverand osp 17 also promoted :-)06:03
chkumar|roversweet sweet06:03
jm1hello folks :)06:52
frenzy_fridayysandeep, ack, thanks!06:57
*** ysandeep is now known as ysandeep|lunch07:35
chkumar|rovergthiemonge: hello08:37
chkumar|rovergthiemonge: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_435/847237/1/gate/tripleo-ci-centos-9-scenario010-standalone/435f65c/logs/undercloud/var/log/tempest/tempest_run.log08:38
chkumar|rover{0} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest) [0.000000s] ... FAILED started failing Policy does not allow this request to be performed.08:38
chkumar|roverDoes octavia distgit revert caused it?08:38
chkumar|roveror do we need to manipulate some settings in tempest.conf to fix it?08:38
chkumar|roverit is blocking upstream check and gate wallaby patches08:39
gthiemongechkumar|rover: hi, the revert should have fixed this "Policy does not allow" error08:39
chkumar|roverchecking the integration line08:41
chkumar|roveramoralej: hello08:59
chkumar|roveramoralej: opened this bug https://bugs.launchpad.net/tripleo/+bug/198246408:59
chkumar|roverI have seen these issue ipa jobs multiple times, so filed a bug to track it08:59
chkumar|roverfeel free to investigate when free, thanks! not urgent08:59
amoralejchkumar|rover, lemme check09:05
amoralejchkumar|rover, only in ipa jobs?09:06
amoralejweird ...09:06
dpawlikchkumar|rover, ysandeep|lunch, arxcruz: hey, all is fine with quay.rdoproject.org ? I mean pruner does not remove any unnecessary image, right? 09:08
arxcruzdpawlik it seems it doesn't, but maybe chkumar|rover can tell you better, since he's rr this week09:08
chkumar|roveramoralej: yes, only in ipa jobs09:11
chkumar|roverdpawlik: arxcruz I have not seen any job complaining about missing container images09:12
chkumar|roverif I get any, will let you know09:12
dpawlikcool, thanks09:12
marioschkumar|rover: sorry i missed your ping see pvt is that what you meant? 09:15
marioseasy one please needs votes when you have time https://review.opendev.org/c/openstack/tripleo-heat-templates/+/850616 thank you 09:18
amoralejchkumar|rover, so, iiuc that job is installing ipa correctly and fails the next package installation09:19
amoralejgiven that ipa installation is probably changing dns config, i'd say something is wrong with the ipa installation?09:20
amoralejit'd be good to hold a node09:20
amoraleji just sent a comment in the LP09:22
chkumar|rovermarios: thank you :-)09:23
chkumar|roveramoralej: might be related to ipa installation not sure, 09:25
chkumar|roverIt is a 50 -50 chance for this error to pop up09:25
amoraleji'd say so09:25
amoralejmmmm weirdo09:25
amoralejso random?09:25
chkumar|roverI will hold a node09:25
chkumar|roverit 's a common random issue09:25
amoralejbut only in ipa09:26
amoralejand always in the same step, i'd say it's related to some of the changes done by the ipa install script09:26
chkumar|roveramoralej: Yes, I will run these jobs seperatley and see If I can reproduce the issue09:32
chkumar|roveramoralej: thank you for looking into this09:32
mariosnp chkumar|rover 09:41
chkumar|rovergthiemonge: I think I need to skip few tests to fix the upstream gate jobs until we get a new promotion for master and wallaby09:45
gthiemongechkumar|rover: ack, I'm still lookin at the logs. do you know how I can download the openstack-octavia-common RPM used in this job? I would like to check if the revert was applied correctly09:55
*** ysandeep|lunch is now known as ysandeep09:59
chkumar|rovergthiemonge: here is the repo config https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_435/847237/1/gate/tripleo-ci-centos-9-scenario010-standalone/435f65c/logs/undercloud/etc/yum.repos.d/delorean.repo09:59
mariosthanks ysandeep (tht patch )09:59
chkumar|roverhttp://mirror.bhs1.ovh.opendev.org:8080/rdo/centos9-wallaby/component/octavia/94/e5/94e5a6f99d752848a0bbd16ddb10e76fc53e0179_a0f4722a octavia rpms10:00
chkumar|rovergthiemonge: http://mirror.bhs1.ovh.opendev.org:8080/rdo/centos9-wallaby/component/octavia/94/e5/94e5a6f99d752848a0bbd16ddb10e76fc53e0179_a0f4722a/openstack-octavia-common-8.0.2-0.20220713082359.94e5a6f.el9.noarch.rpm10:00
gthiemongechkumar|rover: the openstack-octavia-common package does not include the revert10:01
chkumar|rovergthiemonge: yes, we need promotion10:01
chkumar|roverit is going to take time10:01
gthiemongeok10:01
chkumar|roverSo i will be skipping those tests 10:01
ysandeepmarios: No problem, It was an easy one :D10:02
chkumar|roverhello #oooq people I need one help on debugging gather_facts unreachable issue10:26
chkumar|roverit is a kind of weird one10:26
chkumar|roverhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/720a18920749415ab504183a297933ed/log/job-output.txt#694510:27
chkumar|roverpassing logs ^^10:27
chkumar|roverfailing one - https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/9e6959ef65fb4118b26ad5ef7729d6ba/log/job-output.txt#696410:27
chkumar|rover[overcloud-controller-1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.\r\nWarning: Permanently added 'overcloud-controller-1' (ECDSA) to the list of known hosts.\r\ntripleo-admin@overcloud-controller-1: Permission denied (publickey).", "unreachable": true}10:28
chkumar|roverthese jobs have successfull tempest run but due to this playbook failed10:28
*** rlandy|out is now known as rlandy10:32
rlandyfrenzy_friday: bhagyashris: ysandeep: can we merge: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/41702010:35
rlandyhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/417017 was merged10:36
rlandydepends10:36
ysandeepWhen I checked in my morning, you had a comment there.. let me check if patch was updated after that10:36
chkumar|roverrlandy: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626 merge please10:37
frenzy_fridayrlandy, yep. Container build passed and I removed  repoclosure10:37
rlandychkumar|rover: hi - let's sync10:37
chkumar|roversure10:37
ysandeepack, frenzy_friday updated the patch lets +w it10:37
rlandyysandeep: pls vote and I'll merge10:38
rlandyfrenzy_friday: you need a config patch to add the 17.1 line10:39
rlandychkumar|rover: https://meet.google.com/rok-mfgz-pjw?pli=1&authuser=010:40
frenzy_fridayrlandy, sorry which patch?10:40
rlandyfrenzy_friday: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/417020 - need sto merge10:41
rlandyI'll explain what you need next in a few10:41
frenzy_fridayrlandy, yep we need to merge the release file patch10:41
bhagyashrisrlandy, yup we can merge10:48
mariosrlandy: bhagyashris: ysandeep: missing scrum today have clash with mixed-rhel call same time10:50
chkumar|roverrlandy: https://review.rdoproject.org/zuul/status#4411510:50
bhagyashrismarios, ack10:50
ysandeepmarios: thanks for info. 10:52
*** ysandeep is now known as ysandeep|afk10:53
chkumar|roverrlandy: https://bugs.launchpad.net/tripleo/+bug/198246410:56
chkumar|roverrlandy: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4410010:59
chkumar|roverdasm|off: I have removed all the commented jobs from master criteria11:01
rlandychkumar|rover: thanks11:05
bhagyashrisrlandy, we already have this patch to add 17.1 line https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/41697111:05
akahatmarios, have you seen this error before: https://logserver.rdoproject.org/48/44048/9/check/periodic-tripleo-ci-mixed-os-8-9-wallaby/fcfd87f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz ?11:06
rlandybhagyashris++ thanks11:12
rlandymerging11:12
rlandychkumar|rover: arxcruz: https://review.opendev.org/c/openstack/tempest/+/84956211:17
rlandywe need that to merge11:17
rlandyto close out a cix11:17
rlandyhttps://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft11:17
rlandycan you check into getting that sorted11:17
mariosakahat: looking11:18
arxcruzrlandy asked on o-qa channel for core reviewers 11:18
rlandythanks11:18
mariosakahat: not seen that one str object has no element ?11:18
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0 flat out failing11:18
rlandychkumar|rover: ^^ node provision thing is still there11:19
chkumar|roverrlandy: needs new overcloud image with latest kernel11:19
chkumar|roverupcoming master promotion will fix it11:19
chkumar|roversame for wallaby11:19
rlandychkumar|rover++ nice11:19
rlandylet's push for that promo - great11:19
akahatmarios, okay.. 11:20
*** dviroel|out is now known as dviroel11:24
* bhagyashris tea brb11:35
mariosrlandy: ysandeep|afk: when you have time please re-add your votes @ https://review.opendev.org/c/openstack/tripleo-ci/+/841583 (I updated merged depends-on, added new one from today and rebase. no change to content) thank you11:35
rlandyk11:35
mariosand chkumar|rover please ^^^ 11:35
rlandybhagyashris: frenzy_friday: any other patches to review?11:36
rlandyfor 17.1?11:36
frenzy_fridayno I think11:38
rlandylooks like you have standalone working?11:43
frenzy_fridaybhagyashris had some patches for standalone and scenarios. Lemme check with her11:55
rlandyarxcruz: akahat: bhagyashris: frenzy_friday: ysandeep|afk: chkumar|rover: anyone else working downstream ...11:56
rlandypls note the maintenance work this weekend11:57
rlandylikely all jobs will fail11:57
rlandyarxcruz: akahat: pls check all systems on monday morning11:57
rlandyand ping on rhos-ops if you need help11:57
arxcruzrlandy ack11:57
akahatrlandy, ok11:57
arxcruzis akahat next rr with me?11:57
akahatarxcruz, yes. 11:57
rlandydasm|off: dviroel: also note email .. IT has scheduled the Westford Lab4 Power Outage for Friday, July 22 @ 4 pm ET (10 pm CET). This work is an 8-hour outage for all Westford Lab4 hardware.11:58
rlandywill start friday afternoon11:58
rlandywill impact all dev systems11:58
rlandybhagyashris: frenzy_friday: running some ceph check jobs now - if those work, will ask you to upgrade for 17.112:00
rlandywill let you know12:00
dviroelrlandy: ack 12:02
frenzy_fridayack12:03
bhagyashrisrlandy, yeah standalone working12:06
bhagyashrisrunning scenarios12:07
rlandybhagyashris: nice work :)12:10
chkumar|roverrlandy: dasm|off rhos-16.2 issue https://bugzilla.redhat.com/show_bug.cgi?id=210949512:15
chkumar|rovercix email sent12:15
rlandyty12:16
*** amoralej is now known as amoralej|lunch12:20
chkumar|roverrlandy: dasm|off arxcruz akahat I have made the handover notes in advance https://hackmd.io/KrGAxws2QiaHuJmj8TH6Jg?view#Handover-notes12:26
chkumar|roverplease go through it when free, thanks!12:26
akahatchkumar|rover, thanks !!12:26
rlandychkumar|rover: dasm|off: arxcruz: akahat: rhos-17 on rhel-9 promoted yesterday - but is failing today 12:31
rlandyI need to test the new ceoh12:31
rlandyceph12:31
rlandyI am going to rebuild container and overcloud images12:31
rlandywith a test depends on12:31
rlandychkumar|rover: rerun failed12:31
rlandybut I need that line now12:31
rlandyif it's no good, tomorrow's run will go back to original12:32
rlandychkumar|rover: 17 n 9 has no provision failures12:32
rlandymaybe the same on wallaby?12:33
rlandycf5c-406e-ba2a-db8e6d7b209a)\n", "msg": "Node 861d2118-234e-4385-aa7a-24735c3f0be1 reached failure state \"deploy failed\"; the last error is Failed to prepare to deploy: IPMI call failed: raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00."}12:33
rlandy2022-07-21 12:11:15.119064 | fa163e74-f3c6-ec6e-05ad-00000000001a |     TIMING | Provision instances | localhost | 0:04:01.258373 | 224.92s12:33
chkumar|roverrlandy: oh12:33
chkumar|roverrlandy: log please12:33
chkumar|roverrlandy: cs9 wallaby will promote soon12:33
chkumar|roverrlandy: wallaby node provisioning is different issyue12:34
rlandychkumar|rover: ok12:34
arxcruzakahat https://hackmd.io/lfpJ3NdRTXaTah6t3C1oGw for our week :)12:34
rlandyeither way - I need to play with that line now12:34
chkumar|roverrlandy: I think you can test with ceph-5 12:34
akahatarxcruz, thanks :)12:35
chkumar|roverrlandy: if fs01 passes, master will also promote12:36
rlandychkumar|rover: awesome12:36
chkumar|roverakahat: arxcruz just filed this https://trello.com/c/9Lc7Z8Xw/2634-cixbz2109495osp162rhel8hosts-unreachable-during-gathering-facts-leading-to-ovb-job-failure12:38
chkumar|roverit is an interesting bug but I have no idea why it is coming12:39
chkumar|roverrest of all downstream bugs are taken care12:39
chkumar|roverrlandy: dasm|off stepping out for an hour 30 mins12:43
chkumar|roverakahat: arxcruz rr sync invite sent for today12:43
chkumar|roveryou can start from tomorrow12:43
frenzy_fridayfolks, I might be a little late to scrum12:59
bhagyashrisfolks, plz review https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/400946 when you have time12:59
bhagyashrisrlandy, arxcruz chkumar|rover scrum time13:01
bhagyashrisakahat, ^13:01
akahatbeagles, joining13:01
akahatbhagyashris, joining13:01
akahatbeagles, ignore.13:01
arxcruzchkumar|rover unless you're busy, we can do it after the scrum meeting13:04
*** dasm|off is now known as dasm|ruck13:05
dasm|rucko/13:05
*** Guest5598 is now known as rcastillo13:05
*** ysandeep|afk is now known as ysandeep13:20
*** amoralej|lunch is now known as amoralej13:28
akahatdasm|ruck, another way you can check hashes using dlrnapi client13:45
akahatdasm|ruck, dlrnapi --url https://trunk.rdoproject.org/api-centos9-master-uc promotion-get --promote-name tripleo-ci-testing13:45
chkumar|roverwallaby promoted13:45
dasm|ruckakahat++ perfect! I was looking for something like that!13:47
dasm|ruckftr: dlrnapi --url https://trunk.rdoproject.org/api-centos9-master-uc promotion-get --promote-name tripleo-ci-testing | jq .[].aggregate_hash | uniq  gives last 19 hashes13:48
chkumar|roverrlandy: dasm|ruck arxcruz akahat rr sync https://meet.google.com/mzv-zjdu-uhc?authuser=014:00
chkumar|roverdasm|ruck: we are waiting for you14:01
chkumar|roverrlandy: dasm|ruck arxcruz akahat https://hackmd.io/KrGAxws2QiaHuJmj8TH6Jg14:02
arxcruzrlandy akahat https://hackmd.io/lfpJ3NdRTXaTah6t3C1oGw14:03
ykarelchkumar|rover, wrt https://review.opendev.org/c/openstack/tripleo-quickstart/+/850444 abandon14:07
ykarelno longer due to promotion resolved it?14:07
ykarelif yes i think better to root cause that as next kernel updates may break it again14:08
ykarelso good to know why we didn't hit with previous kernel updates in c8 or c9, and why only know, or it used to happen in previous releases too14:09
chkumar|roverarxcruz: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/85015514:11
chkumar|roverarxcruz: akahat https://bugzilla.redhat.com/show_bug.cgi?id=210949514:12
chkumar|roverykarel: yes promotion is going to resolve it14:27
chkumar|roverykarel: I will pass this bug to hardprov to take a look14:27
ykarelchkumar|rover, yes please that would be helpful to avoid such issues in future14:27
rlandychkumar|rover: hey - have 5 mins to help me with image builds?14:30
chkumar|roverrlandy: yes14:31
rlandychkumar|rover: https://meet.google.com/vfg-bpwu-uor14:32
ysandeepakahat++ for the dlrnapi client hint14:43
mariosanyone have a workflow for me please? https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841764 14:52
mariosotherwise i'll do in a bit14:52
chkumar|roverno way14:57
chkumar|roverrlandy: dasm|ruck thank you, we made it :-)15:02
chkumar|roverthis week15:02
mariosthank you chkumar|rover :)15:02
rlandydasm|ruck++ chkumar|rover++15:02
chkumar|roverrlandy++15:02
rlandythanks  it wasn;t easy15:02
dasm|ruckchkumar|rover: i'm still a day short :D but yeah. cs9 master did it :)15:03
rlandyfrenzy_friday: hey - you arounnd?15:03
chkumar|roveryes cs9 master, that line was crazy one15:03
frenzy_fridayrlandy, yep15:03
rlandyfrenzy_friday: meet for a few? got some docs for you15:03
frenzy_fridayyep sure15:04
rlandyfrenzy_friday: https://meet.google.com/noi-kqxz-xfz?pli=1&authuser=015:04
ysandeepdviroel_, fyi.. I have asked few questions(our doubts) on testplatform slack, lets see if someone responds.15:25
dviroel_ysandeep: nice, i will take a look15:26
rlandydasm|ruck: stepping out now15:26
* dviroel_ lunch15:26
rlandythanks holding down the fort here15:26
*** dviroel_ is now known as dviroel15:26
rlandydasm|ruck: if fs01 master run fails, https://review.rdoproject.org/r/c/testproject/+/42319 - please prep patch to skip and promo15:27
rlandythank you15:27
*** rlandy is now known as rlandy|afk15:28
dasm|ruckrlandy|afk: ack15:29
*** ysandeep is now known as ysandeep|out15:30
*** marios is now known as marios|out15:40
*** amoralej is now known as amoralej|off15:57
dasm|ruckseems like we're gonna have cs9 master promo :)16:07
dasm|ruck> last_promotion=2022-07-21 16:31:1917:01
dasm|ruckcs9 master \o/17:01
dviroel\o/17:03
* jm1 out for today 👋20:23
dasm|ruckakahat: arxcruz handover summary: https://hackmd.io/KrGAxws2QiaHuJmj8TH6Jg#2022-07-22---handover21:04
dasm|ruckgood luck!21:04
* dasm|ruck => offline21:04
*** dasm|ruck is now known as dasm|off21:04
*** dviroel is now known as dviroel|out21:07
« Wednesday, 2022-07-20 Index Friday, 2022-07-22 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!