Tuesday, 2022-03-01

*** rlandy|ruck is now known as rlandy|out00:20
ysandeeppojadhav, good morning, when you are around lets sync about 17 on rhel9 current status.05:42
pojadhavysandeep, hello.. good morning. lets sync now. I am available.06:06
ysandeeppojadhav, meet.google.com/bqi-ayfc-ymu06:08
*** ysandeep is now known as ysandeep|afk06:29
dpawlikarxcruz, marios: hey, could you check https://review.opendev.org/c/openstack/ci-log-processing/+/830941 , https://review.opendev.org/c/openstack/ci-log-processing/+/83016907:15
mariosdpawlik: k adding to reviews 07:18
mariosysandeep|afk: o/ needs rebase fyi https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/830060 07:20
soniya29kopecmartin, any idea why is this patch blocked and what can we do get this merged:- https://review.opendev.org/c/openstack/requirements/+/82037807:34
*** ysandeep|afk is now known as ysandeep07:41
kopecmartinsoniya29: there is a merge conflict, but mainly the problem is that we weren't sure whether is safe to use that dependency - therefore -1 on workflow07:52
kopecmartini wouldn't merge that, in the comments we can see that there is a concern regarding the stability of the lib , it also lacks testing 07:52
kopecmartinwe can figure out a different way hot to test ping , without that lib07:53
ysandeepmarios: I thought this was merged, thanks for notifying - I have rebased the patch.07:55
mariosack ysandeep 08:05
soniya29kopecmartin, let's discuss this today in office hours, and let's take this discussion to the patch itself08:05
*** jpena|off is now known as jpena08:33
jm1soniya29: while adding my ssh keys to ci-config repo I saw two old commits from 2020 where you tried to add you key, too. do you still need/want SSH access to cockpit etc.?11:05
soniya29jm1: yeah, i will need them11:06
jm1soniya29: still the correct SSH key? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/2472711:07
soniya29jm1, i will check and correct those11:13
soniya29thanks for pointing, i forgot about them :)11:13
jm1soniya29: they have not been merged yet and they wont because i changed the format 🙈 If those keys are still correct i will create a patch for you11:14
soniya29jm1, please go ahead11:15
jm1soniya29: ack, will ping you when its done11:15
soniya29jm1, sure11:16
*** rlandy|out is now known as rlandy|ruck11:18
rlandy|ruckchandankumar: hello ... how are you with merging parenting patches?11:21
rlandy|ruckdo you want the review meeting now?11:21
chandankumarrlandy|ruck: I have not yet updated the patch11:21
chandankumarrlandy|ruck: today not needed11:21
rlandy|ruckok11:21
chandankumarrlandy|ruck: but I need one help11:21
chandankumarto clear out cs9 patches11:21
chandankumarotherwise rdo reviews are ending with merge conflicts11:22
rlandy|ruckysandeep: is downstream all set now?11:22
rlandy|ruckjobs starting ok?11:22
rlandy|ruckarxcruz: hey ...11:24
arxcruzrlandy|ruck hi11:24
chandankumarrlandy|ruck: review below11:24
arxcruzrlandy|ruck still working on the openssl 11:24
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/830836/11:24
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/82965511:24
chandankumarhttps://review.rdoproject.org/r/c/rdo-jobs/+/3998111:24
chandankumarhttps://review.rdoproject.org/r/c/rdo-jobs/+/3993411:24
chandankumarhttps://review.rdoproject.org/r/c/rdo-jobs/+/3979311:24
rlandy|ruckarxcruz: let's touch base11:24
arxcruzok 11:25
rlandy|ruckwe need the key change11:25
rlandy|ruckhttps://meet.google.com/odn-unmu-vam?pli=1&authuser=011:25
rlandy|ruckarxcruz: ^^11:25
chandankumarit will finish the audit of both master and wallaby11:25
ysandeeprlandy|ruck, no same situation but we know what's the cause, There are 11 instances which are stuck in deletion and nodepool is trying to delete them before creating new instances.11:29
ysandeeprlandy|ruck, I have opened a ticket with PSI, you are in cc.11:29
rlandy|ruckbhagyashris: pls nick11:30
*** bhagyashris is now known as bhagyashris|ruck11:30
rlandy|ruckbhagyashris|ruck: will meet with you when done with arxcruz11:30
arxcruzrlandy|ruck https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L24811:30
bhagyashris|ruckrlandy|ruck, sure11:30
chandankumarmarios: please have a look when free  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/831198/1#message-d198642e2ff42822a9c247baafc4b36e8d6c5f0711:31
chandankumarthanks!11:31
chandankumarysandeep: hello can you +w https://review.opendev.org/c/openstack/tripleo-ci/+/830836 it ? Thanks!11:32
rlandy|ruckarxcruz: https://review.rdoproject.org/r/c/testproject/+/39960/1/.zuul.yaml#1011:33
ysandeepyes11:33
marioschandankumar: ack thanks11:34
rlandy|ruckarxcruz: https://review.opendev.org/c/openstack/tripleo-quickstart/+/83118411:34
rlandy|ruckmanually rebase ^^11:34
rlandy|ruckbhagyashris|ruck: can you join https://meet.google.com/odn-unmu-vam?pli=1&authuser=011:34
* marios relocate biab11:40
arxcruzrlandy|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/83136211:43
rlandy|ruckysandeep: hey - can you join https://meet.google.com/odn-unmu-vam?pli=1&authuser=011:43
ysandeeprlandy|ruck, ack11:44
rlandy|ruck        - create-scenario010-trigger-for-downstream:11:50
rlandy|ruck            vars:11:50
rlandy|ruck              trigger_branch: master11:50
rlandy|ruck              trigger_os_distro: centos_911:50
rlandy|ruckarxcruz: thanks - checking what jobs kick there11:55
rlandy|ruckarxcruz: so no c8 jobs kicked on that change11:55
arxcruzrlandy|ruck i'm updating testproject11:56
rlandy|ruckarxcruz: thanks  - we will need c7 and c811:56
arxcruzrlandy|ruck https://review.rdoproject.org/r/c/testproject/+/3994411:56
bhagyashris|ruckrlandy|ruck, fyi  https://review.rdoproject.org/r/c/testproject/+/39932 [DNM] Test cs9 master failing jobs12:00
* bhagyashris|ruck tea brb12:02
rlandy|ruckbhagyashris|ruck: thanks - making small edit12:02
rlandy|ruckarxcruz: thanks ... standalone 8 master does not exist anymore - and we'll need force_periodic: true on those12:03
rlandy|ruckunder the vars12:03
arxcruzrlandy|ruck if i use force_periodic it might affect the promotion no ?12:04
ysandeeppojadhav, around?12:05
rlandy|ruckarxcruz: report_dlrn: false12:06
rlandy|ruckin vars12:06
rlandy|ruckarxcruz: ^^ make sense?12:06
pojadhavysandeep, yes12:06
arxcruzrlandy|ruck done 12:07
arxcruzrlandy|ruck yes, it does12:07
rlandy|ruckthank you12:07
ysandeeppojadhav, lets discuss parenting on that your patch: meet.google.com/dxd-wbke-ccu , I see some issues there.12:08
pojadhavysandeep, ack12:08
chandankumarysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/829655 and this also, thanks!12:11
ysandeepchandankumar: sure will check in sometime, In a mtg with pooja12:16
chandankumarysandeep: aye, no problem :-)12:17
chandankumarysandeep: Do you know where we use this job https://code.engineering.redhat.com/gerrit/plugins/gitiles/openstack/tripleo-ci-internal-jobs/+/refs/heads/master/zuul.d/required-projects-overrides.yaml#4 [tripleo-ci-base-required-projects] as a parent in downstream?12:18
rlandy|ruckhmmm ... problem on the wallaby line12:19
rlandy|ruckcreate has no dependencies12:19
ysandeepchandankumar, yes.. will explain in few mins12:20
ysandeepchandankumar, want to chat on gmeet?12:30
soniya29rlandy|ruck, meeting?12:30
rlandy|ruckjoining 1 mi12:31
chandankumarysandeep: yes12:31
ysandeepchandankumar, meet.google.com/nzz-hyxf-stm12:31
rlandy|ruckchandankumar: https://review.rdoproject.org/r/c/rdo-jobs/+/39990 - is there a better fix here?12:32
chandankumarrlandy|ruck: let me update the patch12:37
*** ysandeep is now known as ysandeep|break12:38
jm1soniya29: ssh key merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3998212:41
chandankumarrlandy|ruck: https://review.rdoproject.org/r/c/rdo-jobs/+/39990/3/zuul.d/integration-pipeline-wallaby-centos9.yaml can you check now12:42
chandankumarbhagyashris|ruck: rlandy|ruck I might skip community call, Need to drop my brother to station12:44
bhagyashris|ruckchandankumar, ack12:45
rcastilloo/13:02
rlandy|ruckchandankumar: looks good on https://review.rdoproject.org/r/c/rdo-jobs/+/39990/13:08
frenzy_fridayhey folks and reviewbot, pls add to your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3999413:09
reviewbotI have added your review the to Review list13:09
mariosfrenzy_friday: nice ;) ^ 13:12
rlandy|ruckmarios: hey - had to move 1-1 meeting  - clash with prod chain13:13
mariosrlandy|ruck: k np13:13
rlandy|ruckrcastillo: arxcruz: akahat: chandankumar: marios: dasm|rover: soniya29: frenzy_friday: jm1: pojadhav: ysandeep|break: community call in 9 mins13:21
rlandy|ruckhttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both to add agenda items13:21
rlandy|ruckmarios: adding the updates/upgrades item13:21
mariosrlandy|ruck: thank you :) i just fetched the mailing list link to do the same13:23
*** ysandeep|break is now known as ysandeep13:29
rcastilloneed help figuring out an overcloud node provision issue we're hitting on centos ci for the c9 libvirt job13:51
rcastillohttps://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal-29/undercloud/home/stack/overcloud_node_provision.log13:51
ade_leerlandy|ruck, thats great!  we should do that then!13:57
rlandy|ruckrcastillo: pls reach out to hjensas on #tripleo from hardware prov team14:00
rlandy|ruckrcastillo: you can share the logs and setup14:00
rcastillorlandy|ruck: ack14:00
rlandy|ruckif you is unavailable, try steve in afternoon hours14:00
rlandy|ruckrcastillo: ^^14:01
rlandy|ruckyou can use hardware prov chat to leave steve notes14:01
dasm|rovero/14:05
rlandy|ruckdasm|rover: we're looking for master and wallaby c9 to promote14:07
rlandy|ruckalso to see if that cleans up the node provisioning errors in check14:08
dasm|roverack14:08
rlandy|ruckdasm|rover: so you have patches from yesterday skiplist reverts that need to be merged?14:08
dasm|roverchecking on them now14:08
dasm|roverrlandy|ruck: all of them failed tests. Nothing to merge14:09
rlandy|ruckdasm|rover: ok  ...14:12
rlandy|ruckbhagyashris_,: pls pass your work to dasm|rover 14:12
rlandy|ruckpls also nick14:12
*** pojadhav is now known as pojadhav|afk14:19
*** pojadhav|afk is now known as pojadhav14:57
pojadhavrlandy|ruck, chandankumar : kindly review rhel9 standalone component jobs patches https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/312030/ and https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/31491215:10
ysandeepchandankumar: +wed https://review.opendev.org/c/openstack/tripleo-ci/+/829655 lgtm15:18
*** pojadhav is now known as pojadhav|afk15:18
rlandy|ruckdasm|rover: rekicking failed master periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master test15:29
rlandy|ruckwaiting on that one and internal kvm15:29
rlandy|ruckcan meet with you later re: what's still outstanding on CIX board15:29
rlandy|ruckalso we need to check the component lines15:29
rlandy|ruckupstream pls15:29
chandankumarysandeep: thanks!15:32
*** ysandeep is now known as ysandeep|dinner15:34
dasm|roverack15:35
chandankumarrlandy|ruck: ysandeep|dinner: please +w these patches https://review.rdoproject.org/r/c/rdo-jobs/+/39934 https://review.rdoproject.org/r/c/rdo-jobs/+/39981 https://review.rdoproject.org/r/c/rdo-jobs/+/3979315:44
*** ysandeep|dinner is now known as ysandeep15:46
ysandeepchandankumar, looking15:47
dasm|roverrlandy|ruck: when you're free, can I meet with you for a sec?16:14
rlandy|ruckdasm|rover: ack - can meet now16:17
dasm|roverk16:17
dasm|roverhttps://meet.google.com/iro-jmng-nyz16:17
dasm|roverrlandy|ruck: ^16:17
*** ysandeep is now known as ysandeep|out16:21
rlandy|ruckarxcruz: can https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/831362/1/playbooks/tasks/tempest.yml merge?16:36
rlandy|rucklooks like your testproject passes16:36
arxcruzchecking 16:37
arxcruzrlandy|ruck yes, it can be merged, but it has a depends-on on the revert patch, so needs to merge both 16:38
arxcruzi can remove the depends-on to merge it first 16:38
rlandy|ruckin the right order16:38
arxcruzthen the revert, since it will not affect 16:39
rlandy|ruckarxcruz: ack - pls remove the depends on16:39
arxcruzok16:39
rlandy|ruckwe will merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/831362 first16:39
rlandy|ruckthen the revert16:39
rlandy|ruckdasm|rover: openstack-periodic-integration-stable4-centos7 16:40
arxcruzrlandy|ruck done 16:42
*** bhagyashris_ is now known as bhagyashris|ruck16:43
bhagyashris|ruckrlandy|ruck, hey sorry due to network disconnection nick got changed 16:44
rlandy|ruckbhagyashris|ruck: just met with dasm|rover16:47
rlandy|ruckbhagyashris|ruck: pls pass anything you are working on to him since it's late for you16:47
rlandy|ruckhave 1 failed master c9 test in rerun16:47
rlandy|ruckwatching the two kvm tests (wallaby and master) in downstrea,16:48
rlandy|ruckbhagyashris|ruck: you ok to report on the program call tomorrow?16:48
rlandy|ruckbhagyashris|ruck: dasm|rover: we will need to check on component and integration lines in downstream early tomorrow16:53
rlandy|ruckand later today16:53
dasm|roverack16:53
bhagyashris|ruckrlandy|ruck, ok 16:53
bhagyashris|rucki can attend tomorrows program call16:54
bhagyashris|rucknp16:54
bhagyashris|ruckrlandy|ruck, done handover 16:55
bhagyashris|ruckrlandy|ruck, dasm|rover looking at the promoter logs http://promoter.rdoproject.org/promoter_logs/centos9_wallaby_2022-03-01T16:51.log centos-9-scenario010-kvm-internal-standalone-wallaby job is missing so will need to recheck that job for promotion16:58
bhagyashris|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby16:58
bhagyashris|ruckhere all jobs are green but centos-9-scenario010-kvm-internal-standalone-wallaby failed due to NODE_FAILURE17:00
bhagyashris|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1-cs917:00
* bhagyashris|ruck out17:02
dasm|roverack, thanks17:08
rlandy|ruckbhagyashris|ruck: it's running now17:09
rlandy|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/27a60f57a6344df5b775c5253c656757?logfile=console.log17:09
rlandy|ruckno need to recheck17:09
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master17:12
rlandy|ruckreal failure on ipa master17:12
rlandy|ruckwill need a bug there17:12
rlandy|rucklunch brb17:12
rlandy|ruckmaybe skip17:12
rlandy|ruckbhagyashris|ruck: dasm|rover: ^^ rerun won;t help17:13
rlandy|rucklunch brb17:15
dasm|roverack17:18
*** jpena is now known as jpena|off17:38
rlandy|rucktripleowallabycentos9 promoting :)17:49
dasm|rovernice17:50
dasm|roverchecking issue with multinode ipa master17:53
*** rlandy|ruck is now known as rlandy|ruck|biab18:35
*** rlandy|ruck|biab is now known as rlandy|ruck18:53
*** jparker__ is now known as jparker19:06
jm1rlandy|ruck: who is maintaining our promoter instance on vexxhost?19:34
rlandy|ruckjm1: we are19:34
jm1rlandy|ruck: there seem to be mismatches between promoter in our infra-setup and the live instance. i will add a Jira card to fix that19:36
rlandy|ruckjm1: ack - pls raise that Thursday's call19:37
jm1rlandy|ruck: today we discovered that multiple instances on vexxhost are based on centos8, not stream. since c8 mirrors are down, our infra-setup playbooks are broken because yum repos cannot be updated. this has to be fixed as well.19:38
jm1atm we cannot apply any updates to those c8 hosts except manually, which would then again break our infra-as-code setup19:39
jm1with "updates" i mean config changes19:39
jm1rlandy|ruck: i will add jira cards and then you can reevaluate priorities, e.g. if i should fix this or skip it for now and continue working on reproducer19:41
rlandy|ruckhttps://review.opendev.org/q/topic:c9_wallaby_gates19:41
rlandy|ruckjm1: rcastillo is already on the centos-8 to stream update19:42
jm1rlandy|ruck: did not know, will ping him19:45
rcastillojm1: I'm working on upgrading the promoter server to stream atm19:49
jm1rcastillo: awesome :)19:49
rlandy|ruckdasm|rover: hello20:53
rlandy|ruckdasm|rover: you busy atm?20:53
rlandy|rucktripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallabyopenstack/tripleo-cimasteropenstack-regular2 hrs 5 mins 31 secs2022-03-01 18:23:28SUCCESS20:58
rlandy|ruckfirst clear run on check20:59
dasm|roverrlandy|ruck: 'sup?20:59
dasm|roverwelp21:03
rlandy|ruckdasm|rover: no worries - just put the wallaby c8 tests in rerun21:03
dasm|roverack21:03
rlandy|ruckdasm|rover: something wrong?21:04
dasm|roverno, everything is fine21:04
rlandy|ruckok - need to do some review prep 21:04
dasm|roverjust checking bhagyashris|ruck patches21:04
dasm|roverack21:04
rlandy|ruckchecking back in a bit21:04
dasm|roverk21:05
dasm|roverrlandy|ruck: fyi, openstack-periodic-integration-stable4-centos7 succeeded22:11
rlandy|ruckdasm|rover: yeah22:12
rlandy|ruckso you can close the bug and the card22:12
rlandy|ruckthank you22:12
dasm|roverdone22:19
rlandy|ruckgreat22:20
rlandy|ruckrcastillo: any luck with wallaby?22:20
rcastillorlandy|ruck: sadly no22:24
rcastilloovercloud doesn't seem to be coming up22:25
rcastilloI'll double-check my makeshift wallaby release file22:25
rlandy|ruckrcastillo: k - no worries22:30
rlandy|ruckhopefully tomorrow we will have a better master build for you to try22:31
dasm|roverbhagyashris|ruck: i recheck your testproject patches. One or two jobs passed, but others are still red. Sorry, but I didn't move forward with those. But at least we had a green cs7 train run \o/22:37
* dasm|rover is slowly departing...22:38
*** dasm|rover is now known as dasm|off22:39
rcastillorlandy|ruck: yeah. Fingers crossed for master c9 promotion22:41
rcastillorlandy|ruck: also added a couple patches for the centosci c9 wallaby jobs to reviewlist22:42
* rcastillo out22:42
dasm|offo/22:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!