Wednesday, 2022-04-27

« Tuesday, 2022-04-26 Index Thursday, 2022-04-28 »
*** ysandeep|out is now known as ysandeep04:31
*** jpena|off is now known as jpena07:27
*** ysandeep is now known as ysandeep|lunch08:12
*** ysandeep|lunch is now known as ysandeep08:52
* ysandeep around but not feeling great, suffering with cold and cough.09:51
chandankumarysandeep: take rest & get well soon :-)09:55
ysandeepchandankumar: thanks! 09:56
*** rlandy|out is now known as rlandy10:21
rlandybhagyashris|ruck: hello10:23
bhagyashris|ruckrlandy, hi10:23
rlandybhagyashris|ruck: any news on the image build failures?10:23
bhagyashris|ruckimage build is failing on master and wallaby cs910:23
rlandyI asked dviroel|rover|out to leave you a note to look into that10:23
rlandybhagyashris|ruck; I know10:23
rlandywhat is the cause?10:24
bhagyashris|ruckhere is the fix https://review.opendev.org/c/openstack/tripleo-ci/+/83947210:24
bhagyashris|rucki just change CentOS-Stream-9 image 10:24
rlandybhagyashris|ruck: great  - is that tested?10:24
bhagyashris|ruckand here it's passing https://review.rdoproject.org/zuul/status/change/39932,2510:24
bhagyashris|ruckyup testproject is in progress10:25
bhagyashris|ruckbut image jobs got passed10:25
rlandybhagyashris|ruck: pls add that info to the review10:25
bhagyashris|ruckwaiting for fs00210:25
rlandyand I'll w+ it10:25
bhagyashris|ruckyes added 10:25
rlandybhagyashris|ruck: ok - looks like it passed10:25
bhagyashris|ruckyeah10:26
rlandyw+ing10:26
rlandybhagyashris|ruck; I updated the program call doc10:26
rlandyyou can read it off there10:27
bhagyashris|ruckrlandy, yeah thanks 10:27
rlandyI see wallaby c8 is still failing10:27
rlandydviroel|rover|out promoted up the components yesterday10:27
rlandylooking at results10:27
rlandyand victoria is also still behind10:27
rlandyany news there?10:27
ysandeeprlandy, you created some playbook somewhere(I think in downstream) that will help gather the latest image.. May be we should use that in upstream as well.10:29
bhagyashris|ruckyes actually dviroel|rover|out re-kicked the line and just fs035 and fs001 was in progress and rhos-ops started the upgrade infra10:29
rlandybhagyashris|ruck: downstream component jobs look to have failures in jenkins10:29
rlandybhagyashris|ruck: ah10:29
bhagyashris|ruckso the line cleared up without any results10:29
rlandybhagyashris|ruck: ok - so let's rekick that10:29
rlandybhagyashris|ruck: do you know how to rekick lines?10:29
bhagyashris|ruckrlandy, yeah we can. i just hit the testprject patch for fs035 and fs001 10:30
bhagyashris|ruckrlandy, nope i mean i never did that10:30
rlandybhagyashris|ruck: ok - let me show you10:30
bhagyashris|rucksure10:30
rlandyhttps://meet.google.com/xob-brxn-pnd?pli=1&authuser=010:31
soniyaarxcruz, kopecmartin, chandankumar, ysandeep, do you have anything to discuss for today's tempest meeting?11:01
arxcruzsoniya might update on tempest-allow list if you want11:01
arxcruzi already talked with pojadhav but we can sync again with you if you want 11:01
arxcruzup to you 11:01
arxcruzotherwise, i don't have anything else 11:02
soniyaarxcruz, okay11:09
rlandysoniya: meeting is a clash with engineering all hands11:09
pojadhavsoniya, I will sync up with on allow list, arxcruz will explained me.11:10
pojadhavsoniya, let me know your availability11:10
soniyaarxcruz, pojadhav can sync with me regarding allow list, you can carry on with your stuffs or join us, as you wish :)11:12
soniyarlandy, ack11:12
arxcruzsoniya pojadhav sure, let me know if you need me 11:12
soniyaarxcruz, okay, thanks :)11:12
*** dviroel|rover|out is now known as dviroel|rover11:16
dviroel|rovero/11:16
pojadhavarxcruz, yep11:16
bhagyashris|ruckrlandy, hey just one question  --newrev means recent aggregate hash right?11:16
bhagyashris|rucki mea tripleo-ci-testing one11:16
bhagyashris|ruckwhile en-queue the pipeline11:18
dviroel|roverbhagyashris|ruck: most recent commit in tripleo-ci repo11:19
dviroel|rover"--project openstack/tripleo-ci --ref refs/heads/master --newrev 15761b77d91ab3e398f6fa1d10d2f5da267b7931"11:20
* dviroel|rover at least, is the way that I have being doing11:20
bhagyashris|ruckdviroel|rover, re-kicking victoria and c8 wallaby11:23
bhagyashris|ruckdviroel|rover, thanks11:23
dviroel|rovergreat, tks11:25
kopecmartinsoniya29: nothing from my side11:31
rlandybhagyashris|ruck: no - head of triple-ci repo11:33
bhagyashris|ruckrlandy, yes did11:33
rlandydviroel|rover: wallaby c8 rerun got disrupted by infra11:34
rlandyasked bhagyashris|ruck  to rerun11:34
rlandydviroel|rover: major failure on check ovb  node provision11:34
rlandymaster11:34
rlandybhagyashris|ruck logged bug11:34
rlandypls follow up with harold11:34
bhagyashris|ruckdviroel|rover, hey we can sync up when you free11:35
rlandybhagyashris|ruck: steve left notes on https://bugs.launchpad.net/tripleo/+bug/197040011:37
rlandyAccording to [1] the baremetal_image is ipxe-boot. Shouldn't it be ipxe-boot-uefi?11:37
rlandy[1] https://logserver.rdoproject.org/63/839163/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/ad39bde/job-output.txt11:37
rlandybhagyashris|ruck: can you try that11:38
dviroel|rover ack, thanks for the updates11:40
rlandybhagyashris|ruck: pls join review time11:41
rlandydviroel|rover and I are on for ruck rover sync11:41
bhagyashris|ruckrlandy, dviroel|rover https://bugs.launchpad.net/tripleo/+bug/197040011:42
rlandydasm|off: https://bugs.launchpad.net/tripleo/+bug/1970400 - pls see steve's last comment11:43
rlandycan we use your patch to test???11:43
rlandyhow far out are we from using that?11:43
*** pojadhav is now known as pojadhav|afk11:48
soniya29kopecmartin, arxcruz, chandankumar, ysandeep, rlandy, we can cancel the meeting then, if everybody is okay11:53
rlandyok11:53
chandankumarsoniya29: yuo11:53
kopecmartinack11:53
chandankumaryup11:53
arxcruzok11:53
ysandeepsoniya29, ack.. I think we should move that meeting to biweekly from weekly, we have less topic these days.11:53
soniya29ysandeep, yupp11:54
dviroel|roverbhagyashris|ruck: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83118511:56
dviroel|roverthe revert11:56
*** pojadhav|afk is now known as pojadhav12:26
*** amoralej is now known as amoralej|lunch12:30
*** ysandeep is now known as ysandeep|afk12:42
rlandydviroel|rover: you ok to work on the higher timeouts for c8 node provision?12:51
dviroel|roverrlandy: yes, checking wher I can change this parameter12:52
rlandydviroel|rover: no idea :) ... would have to look12:52
rlandyping if you need help12:52
dviroel|roverack12:52
rlandydviroel|rover: marios: chandankumar: ysandeep|afk: kieran apologized for the rushed timing at yesterday's meeting - said you guys did a great job:)12:53
mariosrlandy: ack 12:56
dviroel|roverrlandy: ack, tks12:59
*** ysandeep|afk is now known as ysandeep13:00
rlandyall: engineering all hands meeting now13:01
rlandyysandeep: starting work on the new base image for 16.213:07
rlandywill hit the nameserver issue after that13:07
rlandydviroel|rover: bhagyashris|ruck: ^^13:08
ysandeeprlandy: ack 13:09
*** amoralej|lunch is now known as amoralej13:12
rlandychandankumar: marios: ysandeep: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/sf-config.git;a=blob;f=nodepool/virt_images/roles/discover-latest-image/tasks/main.yaml;h=13deb96155e953620c7103789adb34766d7076a5;hb=HEAD13:19
rlandythat is what we need upstream in RDO13:19
rlandyto get latest image13:19
rlandyproposing there13:20
ysandeeprlandy, ++13:20
dviroel|rovernice ++13:20
mariosrlandy: ack thanks13:22
rlandyhmm  - why is rhel-9 not using https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/sf-config.git;a=blob;f=nodepool/virt_images/roles/discover-rhel/tasks/main.yaml;h=29fc82610436aab1fe8c035c0d591689b5f2058a;hb=HEAD?13:26
dasm|offo/13:48
*** dasm|off is now known as dasm13:48
rlandydasm: hello - pls see messages above re: uefi13:50
rlandysent you meeting request for later today13:50
dasmrlandy: this is my periodic run: https://logserver.rdoproject.org/28/41128/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/d9d8bf1/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz13:50
dasmit passed with ipxe-uefi13:50
dasmrlandy: this is the patch for all jobs: https://review.rdoproject.org/r/c/testproject/+/4112813:51
rlandyk -let's review at meeting13:52
dasmack13:52
dviroel|roverrlandy: changing values here https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/83952614:04
dviroel|roverdidn't find any puppet config for that14:04
dviroel|rovertesting here https://review.rdoproject.org/zuul/status#3864614:05
dasmhttps://lists.openstack.org/pipermail/openstack-discuss/2022-April/028330.html14:20
dasm\o/14:20
*** ysandeep is now known as ysandeep|out14:21
jm1arxcruz, frenzy_friday, rcastillo: will be there in 5 mins14:27
dviroel|rover++14:27
frenzy_fridayjm1, cool14:28
rlandyjm1: pls see gchat question14:28
rlandydasm: are you switching with soniya for next rr cycle?14:37
dasmrlandy: yes14:37
rlandyok14:38
bhagyashris|ruckrlandy, dviroel|rover hey leaving for the day anything you want from me ...14:52
dviroel|roverbhagyashris|ruck: i think that we are ok, thanks, have a good eod14:54
bhagyashris|ruckdviroel|rover, ack thanks14:54
rlandybhagyashris|ruck: should be good - thanks - pls carry on with tempest investigations tomorrow14:59
rlandyso we can try clear some off the CIX board14:59
rlandydviroel|rover: need help with victoria?14:59
rlandywe are way out there as well15:00
dviroel|roverrlandy: checking those failures now15:00
rlandythanks15:01
dviroel|roverrlandy: will rerun failed jobs, none of those are consistent atm15:04
rlandyk15:06
dviroel|roverdasm: question for your tp https://review.rdoproject.org/r/c/testproject/+/41128 - your depends on only changes master and wallaby images, right? why are you testing against all other branches?15:15
dasmchecking15:17
dasmdviroel|rover: yes, it's only wallaby and master. no particular reason to test against all of them right now.15:18
dasmbefore i changed ovb-manage so i had to verify if i'm not breaking anything15:18
dasms/before/earlier15:18
dviroel|roverdasm: ack, next run can you comment those that are not relevant? just to save resources15:20
dasmofc15:22
dviroel|rovertyvm15:22
dasmdviroel|rover: are we waiting on jobs because of me?15:22
jm1Tengu: fyi the server which hosts your lada vm has been shut down because its getting a ram replacement15:24
jm1sshnaidm, rlandy: ^15:24
jm1this also means our downstream cockpit is down atm15:25
jm1bhagyashris|ruck, dviroel|rover: ^15:25
dviroel|roverdasm: no, not atm 15:26
dviroel|roverjm1: ack thanks for letting us know15:28
*** marios is now known as marios|out15:28
* dviroel|rover lunch - biab15:29
*** dviroel|rover is now known as dviroel|rover|lunch15:29
dasmdviroel|rover|lunch: ack. thanks15:29
jm1sshnaidm, rlandy, bhagyashris|ruck, dviroel|rover, Tengu: ram replacement succesful, downstream cockpit and ladavm are up again15:30
rlandyjm1: awesome - thanks15:31
*** amoralej is now known as amoralej|off15:57
rlandylunch - brb16:17
*** dviroel|rover|lunch is now known as dviroel|rover16:18
*** jgilaber is now known as jgilaber|away16:35
Tenguwoohoo16:58
Tengumeaning i dont need to move away?16:58
rlandyyep - you can stay on our box17:01
*** jpena is now known as jpena|off17:01
dviroel|roverrlandy: i think that we need a skiplist for https://bugs.launchpad.net/tripleo/+bug/196873217:12
dviroel|roveradded a comment to LP17:12
dviroel|roverat least 3 dif tests are failing with this error17:13
dviroel|roveroh, we already have some jobs skipping - will update17:16
rlandyok17:25
dasmdviroel|rover: rlandy there is no gmeet link, so please join here: https://meet.google.com/zsf-tmvf-gaw18:00
rlandyupdated invite18:01
dasmk18:01
dviroel|roverack. tks for the reminder :)18:01
dasmdviroel|rover: https://meet.google.com/hdk-gueg-vnb18:01
dasmrlandy: https://review.rdoproject.org/r/c/rdo-jobs/+/4210118:01
dasmdviroel|rover: rlandy  https://review.rdoproject.org/r/c/rdo-jobs/+/42101/3/zuul.d/integration-jobs-cs9-master.yaml#5118:03
rlandynameserver test patch is in18:15
rlandyworking on base image update18:15
dviroel|roverrlandy: wrt Wallaby C8, increasing ironic conductor times 18:30
dviroel|roverrlandy: better results on first run https://review.rdoproject.org/r/c/testproject/+/38646/29#message-68fe5558a6dcc8864cc72249bcae99b37dbe56a318:31
dviroel|roverboth passed node_provision but failed due to timeouts18:31
dviroel|roverlot of ovb jobs were failing with timeouts18:32
dviroel|roverhere: https://logserver.rdoproject.org/46/38646/29/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/3a72c5b/logs/undercloud/var/lib/config-data/puppet-generated/ironic/etc/ironic/ironic.conf.txt.gz 18:32
dviroel|roverironic conductor config ^18:32
dasmrlandy: dviroel|rover fyi: https://review.rdoproject.org/r/q/topic:pxe_uefi+is:open+NOT+DNM18:46
rlandydviroel|rover: nice find on that18:52
rlandyat leadt passed node provision18:52
dviroel|roveryes18:52
rlandydviroel|rover: overcloud deploy timeouts are only on the tests that ran with condictor timeout changes or all?18:53
dviroel|roverrlandy: all18:53
rlandywhy are they timing out?18:53
rlandysame all over?18:53
dviroel|roverovercloud deploy, not sure which step18:53
dviroel|roversome jobs take too long to deploy the overcloud and timeout on tempest18:54
dviroel|roverthat was my first impression18:54
rlandydviroel|rover: sowe can increase the job timeout18:54
rlandybut there is probably a reason for the change18:54
rlandydviroel|rover: there was some maintenance work done earlier today18:54
rlandywould check in with nhicher about what that ws and if it's related18:55
rlandythe timeouts were not there yesterday, right?18:55
dviroel|roveri dont think so18:55
rlandydviroel|rover: I would start there18:56
rlandynhciher may know what they changed18:56
dviroel|roverdo we have more quota/resources now? 18:57
* dviroel|rover checking logs 18:59
rlandyonly on doenstream18:59
rlandychecking logs as well18:59
rlandydviroel|rover: all ovb, right?19:00
rlandylooking at stable 119:00
dviroel|roveryes19:00
dviroel|roverack19:00
dviroel|roverfs001 on victoria: 04/25 overcloud-deploy: 3096.28s; 04/26: overcloud-deploy: 4698.85s (failed); 04/27: overcloud-deploy: 7609.92s(failed)19:06
dviroel|roverhum, TIMING | Wait for stack status | localhost | 0:30:18.711431 19:08
dasmcould it be a network issue?19:08
rlandyyeah19:14
rlandyperiodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby - success19:15
rlandyso that's one point19:15
rlandyis node provision taking longer?19:25
rlandydasm: nice - pls ping when you have results we can merge19:26
dviroel|roverrlandy: yes, 342.92s vs 736.45s - in fs001-victoria that i was looking19:27
dviroel|roverin the same proportion as overcloud deploy19:28
dviroel|roversame thing for undercloud-install19:29
rlandyhmm - major slow down19:29
rlandydviroel|rover: did you check with infra?19:29
rlandydviroel|rover: pls log those results in a bug and let's pass to #rhos-ops19:29
rlandyif there are enough examples19:30
dasmack19:31
dviroel|roverrlandy: no, not yet - current run looks better19:32
rlandyk - see how you go19:32
dviroel|roverrlandy: dasm https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839608 - can we merge this one today?19:38
dviroel|roveri will trigger some component jobs on oed19:38
dviroel|roverfor wallaby c919:39
dasmdviroel|rover: +1 from me. rlandy can you +W?19:39
dasmdviroel|rover: minor thing, you could add "related-bug" to commit msd19:39
dasm*msg19:39
dviroel|roverdasm: sure, missed that, tks19:41
dviroel|roverdon19:42
* rlandy looks19:42
dasm+1ed19:42
rlandydone19:42
dasmthx19:43
rlandydownstream gerrit is sooooo slow20:03
dasmrlandy: hmm... ipxe-uefi is also exhibiting the issue with node provisioning: "192.168.24.24 port 22: Connection refused" https://logserver.rdoproject.org/28/41128/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/0d0101f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz20:21
rlandydasm: dviroel|rover was working on node provision failures20:21
dviroel|roverdasm: this is the failure that we were expecting to fix with uefi-boot20:23
dviroel|roverdasm: see https://bugs.launchpad.net/tripleo/+bug/197040020:23
dasmyeah20:23
dasmand it's not really working20:24
dviroel|roverdasm: see console logs - seems that we have some progress with your patch20:26
dasmcheckind20:26
dasmdviroel|rover: where can i find them?20:27
dviroel|roverhttps://logserver.rdoproject.org/28/41128/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/0d0101f/logs/baremetal_6_86371_0-console.log20:27
dviroel|roverfrom bug we have: https://logserver.rdoproject.org/63/839163/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/ad39bde/logs/baremetal_2_86483_0-console.log20:28
dasmoh right20:28
*** rlandy is now known as rlandy|mtg20:28
dasmack. yeah, that's definitely improvements, because it's booting up and being configured.20:29
dasmbut nevertheless, we're still hitting provisioning issue.20:29
dasminteresting -- i didn't see it 2 days ago20:29
dviroel|rovererrors is different too20:30
dviroel|roverwithout your patch "192.168.24.10 port 22: No route to host"20:30
dviroel|roverwith your patch is "Connection refused"20:30
dasmOh, right. I thought we had "Connection refused" but we didn't even had the route.20:31
dasmSo it sounds like we didn't provide valid ssh key20:32
dasmbut cloud init is working20:32
dasmwe have20:33
dasm> ci-info: | ssh-rsa | a6:17:89:19:69:f8:45:f9:9c:1f:3f:1f:27:7e:eb:cd:88:29:c9:fe:42:5f:23:1f:76:4c:f5:8b:1c:7d:55:ed |    -    | zuul-build-sshkey20:33
dasmso, are we missing other keys?20:33
* dasm checking how it worked before20:33
dasmdviroel|rover: in that case, i think it would be useful to merge this patch. wdyt?20:45
dviroel|roverdasm: yeah, maybe, we can provide these new logs to steve, and get some feedback20:48
dasmftr: https://review.rdoproject.org/r/c/rdo-jobs/+/4210120:49
dasmdviroel|rover: is the issue with uefi boot only on fs01? I see fs02 passed: https://review.rdoproject.org/zuul/status/change/41128,620:58
dviroel|roverdasm: no, https://review.rdoproject.org/zuul/build/2e4b2b4844e744fd9584531a40cf423121:01
dasmdviroel|rover: in that case, i see passes for fs02 master and wallaby!21:02
dasm\o/21:02
dviroel|roverdasm: see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master 21:02
dasmwhoa. it sometimes passes, and other times it fails?21:03
dviroel|roverseems to be affecting more fs00121:03
dviroel|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master21:03
dasmbut this werked: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/2e4b2b4/logs/baremetal_0-console.log21:03
dviroel|roverdasm: ack, only fixing boot won't fix the node_provision issue21:06
dviroel|roverbut it is a good progress21:06
dasmindeed21:07
*** dviroel|rover is now known as dviroel|rover|biab21:12
rlandy|mtgperiodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-mastersuccess (non-voting)21:26
rlandy|mtgshocker - that passed21:26
*** rlandy|mtg is now known as rlandy21:27
dasmrlandy: after talking to dviroel|rover|biab we can merge this: https://review.rdoproject.org/r/c/rdo-jobs/+/42101 It passes overcloud provisioning: https://review.rdoproject.org/r/c/testproject/+/41128 even though not fully resolving the issue.22:07
rlandyok22:09
rlandydviroel|rover|biab: dasm: I +2'e it22:09
rlandywaiting for zuul to vote22:10
rlandythen will w+22:10
* rlandy need sto step out bbl22:10
rlandydviroel|rover|biab: you can merge if yo get back first22:10
*** rlandy is now known as rlandy|bbl22:10
dasmthx. yeah, i added "Related-Bug" to keep it in order.22:11
* dasm signs off for tonight22:19
dasmsee you!22:19
*** dasm is now known as dasm|off22:19
*** dviroel|rover|biab is now known as dviroel|rover22:31
* dviroel|rover w+1 dasm patch22:44
*** dviroel|rover is now known as dviroel|rover|out22:54
rcastilloleaving as well22:58
« Tuesday, 2022-04-26 Index Thursday, 2022-04-28 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!