Wednesday, 2022-07-06

*** rlandy is now known as rlandy|out01:17
*** ysandeep|out is now known as ysandeep|ruck03:12
pojadhavysandeep|ruck, good morning :)03:14
ysandeep|ruckgood morning o/03:15
ysandeep|ruckrlandy|out, rcastillo|rover thanks for keeping an eye on component promotion, looks like everything which merged is in on time.04:09
*** raukadah is now known as chandankumar04:22
chandankumarpojadhav: ysandeep|ruck Good morning :-)04:22
ysandeep|ruckchandankumar, good morning and congrats o/ 04:32
*** ysandeep|ruck is now known as ysandeep|ruck|afk04:32
chandankumarysandeep|ruck: thank you :-)04:32
*** beagles_ is now known as beagles04:45
*** ysandeep|ruck|afk is now known as ysandeep|ruck05:08
ysandeep|ruckrlandy|out, rcastillo|rover fyi.. new regression in 17 - ceph related: https://bugzilla.redhat.com/show_bug.cgi?id=2104372 05:27
jm1o/06:36
jm1marios: hello :) saw your comment in rr schedule. how about switching our slots?06:55
jm1marios: i am okay with switching (a) both our shifts or (b) only the marked shifts06:57
mariosjm1 cant swap as i am on the week before you07:09
mariosah yeah switching both could work ok 07:09
mariosjm1: ^^ 07:09
jm1marios: ack, so switch both?07:13
mariosjm1: sure go ahead07:19
mariosjm1: hmmm actually i cannot switch with you for the one i need in august 07:21
mariosjm1: because i am not planning on being back until Monday 5th 07:22
mariosjm1: so it doesn't work :/07:22
*** amoralej|off is now known as amoralej07:22
jm1marios: okay, np. we will find others to switch ^^ maybe we can discuss this in scrum meeting tomorrow07:22
mariosjm1: sure07:22
*** ysandeep|ruck is now known as ysandeep|ruck|lunch08:43
jm1rcastillo, rlandy, ysandeep: we have a patch for aoc which will probably break tripleo. how to proceed? https://review.opendev.org/c/openstack/ansible-collections-openstack/+/83964009:24
*** jgilaber|away is now known as jgilaber|testing09:42
*** jgilaber|testing is now known as jgilaber|testing|testing09:43
*** jgilaber|testing|testing is now known as jgilaber09:44
mariosysandeep|ruck|lunch: aware of any mirror issues in rdo/vexx? i have hit a few content provider dlrn build issue with mirror like https://logserver.rdoproject.org/58/43558/10/check/tripleo-ci-centos-9-content-provider/f76725d/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/c1/52/c15223c528707677b78e7028d830759cdd6f8a9e_dev/rpmbuild.log.txt.gz ( from test @ 09:45
marioshttps://review.rdoproject.org/r/c/testproject/+/43558 (last few recheck there are for the same) fyi/fwiw... maybe i'm just really unlucky? 09:45
*** ysandeep|ruck|lunch is now known as ysandeep|ruck09:53
ysandeep|ruckmarios, from above failure.. looks like job is not using vexx mirror09:55
ysandeep|ruck~~~09:55
ysandeep|ruckDEBUG: Errors during downloading metadata for repository 'quickstart-centos-appstreams':09:56
ysandeep|ruckDEBUG:   - Curl error (28): Timeout was reached for http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/repodata/repomd.xml [Failed to connect to mirror.stream.centos.org port 80: Connection timed out]09:56
ysandeep|ruckDEBUG: Error: Failed to download metadata for repo 'quickstart-centos-appstreams': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried09:56
ysandeep|ruck~~~09:56
ysandeep|ruckmirror.stream.centos.org 09:56
ysandeep|ruckWhen I checked periodic line this morning - I haven't noticed jobs failing with mirror issues.09:57
ysandeep|ruckmarios, may be you can include a playbook this this: https://review.rdoproject.org/r/c/testproject/+/43803/6/playbooks/mirror.yaml 09:58
ysandeep|ruckso that job uses vexx mirror09:59
mariosysandeep|ruck: ah thanks - didn't notice that - will see why it isn't using the mirror. but i've been running it for few weeks just lots of issues yesterday/today 09:59
mariosysandeep|ruck: thank you09:59
*** rlandy|out is now known as rlandy10:24
rlandyysandeep|ruck: hey - want to sync?10:25
ysandeep|rucksure.. let me grab my headphones10:25
rlandyysandeep|ruck: https://meet.google.com/adj-ceau-qpz?pli=1&authuser=010:27
ysandeep|ruckrlandy, fyi.. rhel8 tripleo component triggered here: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#21164310:56
rlandygreat - thanks10:58
*** amoralej is now known as amoralej|lunch11:05
marioso/ any one want to join us? 11:16
mariosreview time now11:16
rlandychandankumar: ^^?11:17
*** dviroel|out is now known as dviroel11:25
rlandyarxcruz: there are a few patches on the review list that need your help  - from dasm11:27
arxcruzrlandy k, i'll check11:27
rlandyhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4381511:27
rlandyarxcruz: ^^ nobody else will w+ that - go ahead if you are happy with it11:28
arxcruzrlandy done 11:28
rlandyarxcruz: one more: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4378011:31
ysandeep|ruckrlandy, dviroel does it look familiar: https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/160/check/periodic-tripleo-ci-rhel-8-scenario004-standalone-tripleo-rhos-17/8b69359/job-output.txt ?11:39
rlandylooking11:39
ysandeep|ruck"/bin/podman: stderr Error: error getting default registries to try: error loading registries configuration \"/etc/containers/registries.conf\": mixing sysregistry v1/v2 is not supported"11:40
rlandyysandeep|ruck: ack11:40
rlandyysandeep|ruck: dviroel fixed that in the containers job11:41
* pojadhav coffee break...11:43
ysandeep|ruckrlandy, sc04 is not in tripleo rhel8 criteria.. /me checking why11:44
rlandyysandeep|ruck: was fixed for 16.211:45
* rlandy gets11:45
ysandeep|ruckrlandy, https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/418587 adding sc04 to avoid tripleo promotion - we should fix this issue in tripleo component itself.11:48
rlandymerging that11:48
* ysandeep|ruck filing a bug - then we can investigate11:49
rlandyysandeep|ruck: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/415882/1/playbooks/configure-rhel.yaml11:50
rlandydviroel patch to deal with mixed v1/v2 containers11:50
rlandyysandeep|ruck: can you try depends-on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/415882 and see if it helps11:50
ysandeep|ruckthanks, will check after program call 11:50
ysandeep|ruckit was working so far.. not sure what changed all of sudden11:51
ysandeep|rucklet me file a bz so that we can track issue11:52
* dviroel reads11:57
rlandydviroel: it's the n \"/etc/containers/registries.conf\": mixing sysregistry v1/v2 is not supported" - error11:58
dviroelysandeep|ruck: so, what was happening is that some playbooks might be forcing container v1  configuration, and after that, another role (like tripleo_podman install) is making containers v2 configuration, and both can't leave together11:58
rlandyysandeep|ruck: looks like tripleo on rhel-9 passed11:59
rlandyyou want me to promote that hash and rekick line?11:59
ysandeep|ruckrlandy, sure11:59
rlandyk11:59
ysandeep|ruckdviroel, so job was working so far: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario004-standalone-tripleo-rhos-17&skip=0 , looks like some changes in code side12:00
ysandeep|ruckwe need to debug what changed12:00
dviroelysandeep|ruck: ack, let me help with this debug12:01
rlandywhat? we have to do this all again on July 20???12:03
rlandyysandeep|ruck: you're reporting on call?12:03
rlandyysandeep|ruck: pls mention we promoted all components12:03
ysandeep|ruckyes12:03
ysandeep|ruck:) hah fun.. july 20th now 12:05
dviroel=/12:05
rlandyysandeep|ruck: not upstream though12:05
rlandyat least it's just one env12:05
ysandeep|ruckyeah no import atleast12:05
rlandyysandeep|ruck++ nice report12:22
* rlandy needs to run up to complex maintenance office - brb12:23
*** rlandy is now known as rlandy|brb12:23
*** amoralej|lunch is now known as amoralej12:39
*** rlandy|brb is now known as rlandy12:42
rlandyysandeep|ruck: need help with anything else?12:43
ysandeep|ruckjust investigating sc04, found something.. please check rhos-dev in internal12:43
ysandeep|ruckregistries.conf content differs in current and last good run12:44
rlandywe found that with 16.212:49
rcastillo|rovero/13:06
rlandyrcastillo|rover: pls see #rhos-dev on internal13:10
rlandyysandeep|ruck is chatting there with ceph team13:10
rcastillo|roverk13:10
rlandyrcastillo|rover: also - pls see https://bugs.launchpad.net/tripleo/+bug/1980255/comments/513:12
rlandythere is a suggestion there from xek13:12
rlandyon how to fix ipa job13:12
rlandy(master) on cix board13:12
rcastillo|roveralright13:13
rcastillo|rovercatching up13:13
rlandyysandeep|ruck: let's sync with rcastillo|rover before you are EoD - before or after CIX call? 13:22
rlandydequeing network line and requeuing - failed on nodes13:25
ysandeep|ruckwe can sync even now if you want - I will also attend cix - so whatever works for you guys.13:31
rcastillo|roverI can sync now as well13:32
ysandeep|ruckrlandy, ^^ want to sync now?13:34
*** dasm|off is now known as dasm13:39
dasmo/13:40
rlandyysandeep|ruck: sure13:44
ysandeep|ruckrlandy, rcastillo|rover meet.google.com/jhf-gthx-fvj13:45
dasmarxcruz: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43815/4 done13:47
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4390313:55
dasmarxcruz: could you take a look here too? https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84847213:59
rlandyrcastillo|rover: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master&pipeline=openstack-periodic-integration-main&skip=014:03
arxcruzdasm done, do you have an example for the skiplist that is returning empty ?14:20
dasmarxcruz: wrt tempest-skip?14:21
arxcruzdasm yes 14:21
dasm(venv) [dasm@fedora vars]$ tempest-skip list-skipped --file tempest_skip.yml 14:22
dasm(venv) [dasm@fedora vars]$ 14:22
dasmarxcruz: ^14:22
dasmmaster branch, without the fix14:22
mariosysandeep|ruck: you must know about rdo node_failure ? last couple hours ish 14:22
rlandymarios: ack14:23
marioshad a couple and i see more in https://review.rdoproject.org/zuul/status 14:23
mariosthanks rlandy 14:23
ysandeep|ruckmarios, yes we are aware14:23
mariosk thanks ysandeep|ruck 14:23
rlandyrekicked network line 14:23
rlandybut appears it's still there14:23
rlandyysandeep|ruck: rcastillo|rover: ^^ pining rhos-ops14:23
ysandeep|ruckrcastillo|rover, rlandy I am in remote with ceph team, skipping cix - could you please take care of cix14:28
rlandyysandeep|ruck: ack14:28
*** dviroel is now known as dviroel|lunch14:59
arxcruzdasm i'm having problems to compile the v6 pretty table on that python3.6 version :/ 15:02
dasmarxcruz: ack.15:03
dasmshould we update Dockerfile to newer one? 15:03
arxcruzdasm let me try update to python 3.8 the container15:03
arxcruzmaybe it will fix the problem15:03
arxcruzdasm or maybe get rid of the table, since nobody is using 15:03
dasmwhy not py3.10?15:03
arxcruzdasm that also work :) 15:04
dasmarxcruz: v6 adds "Separator" which is not used, hence we can go back to old one15:04
arxcruzdasm removing the pretty-table works 15:12
arxcruzit seems they added some new stuff and it's failing to build 15:12
arxcruzspecially in a small container that doesn't have all dependences 15:12
dasmarxcruz: k. do you want to update the patch, or should i do it?15:14
arxcruzdasm i'll do it if you don't mind :) 15:14
dasmgo for it15:14
arxcruzok, just doing some more tests 15:14
dasmack15:14
*** ysandeep|ruck is now known as ysandeep|dinner15:28
pojadhavfolks, need +w on this https://review.opendev.org/c/openstack/tripleo-quickstart/+/84235415:43
mariosdone pojadhav 15:50
pojadhavmarios, thank you :)15:50
marioswelcome 15:54
*** marios is now known as marios|out15:54
pojadhavfolks, kindly revisit once and if no objections please +W this to close out pending patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/843840 I have added summarize comment on the patch to recall easily in future if needed.16:00
*** ysandeep|dinner is now known as ysandeep16:03
rlandymarios|out: rcastillo|rover: all c9 rdo jobs are now in node_failure16:03
rlandyfollowing up on rhos-ops16:03
rlandyysandeep: ^^ fyi for your morning16:04
ysandeepack16:05
ysandeeprlandy, rcastillo|rover fyi.. incase john v1 thing don't work - I was thinking if we can do something like this https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/418625/1/playbooks/install_podman.yaml so that tripleo_podman role - configure coorect registries.cong16:06
marios|outthanks rlandy .. could not get an environment going today at all first mirror issues then multiple node failures seemed to be pretty consistent since lunchtime ish 16:06
marios|outwell lunchtime here like ~7 hours ago 16:07
ysandeeprlandy, rcastillo|rover but it hit ERROR! the role 'tripleo_podman' was not found - but we can fix that..16:07
rlandyysandeep: think john's approach sounds the safest at this point16:07
rlandybut feel free to try out your approach16:08
pojadhavneed reviews : https://review.opendev.org/c/openstack/tripleo-ci/+/84809716:08
*** dviroel|lunch is now known as dviroel16:08
ysandeeprlandy, ack please testproject his change once he post it.. 16:08
* ysandeep out16:08
rlandyysandeep: ack - on it16:08
*** ysandeep is now known as ysandeep|out16:08
ysandeep|outrlandy, check this out - looks like we are running everything on rhos-release change: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#417047 16:12
ysandeep|outlooks like we need to tune the file filters16:13
ysandeep|out16.2 running on ceph5 is wrong, should only trigger on ceph4 changes16:13
rlandyack16:15
rlandybhagyashri is working on that?16:15
ysandeep|outrlandy, hopefully this will fix that: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/418630 16:18
ysandeep|outhttps://code.engineering.redhat.com/gerrit/c/rhos-release/+/417047/3/rhos-release this file update in every patch16:18
* ysandeep|out signing off, see you all tomorrow o/16:22
rlandysee you16:24
dviroelo/16:27
*** amoralej is now known as amoralej|off16:28
dviroeloh, 17-rhel9 will promote :)17:07
dviroelrcastillo|rover: ^17:07
dviroel~o~17:07
rcastillo|roverdviroel: nice!17:13
rlandynow we have to promote all components17:18
rcastillo|roverrlandy: want me to run any?17:18
rlandyrcastillo|rover: let me check if promote happened yet17:19
rcastillo|roverok17:19
rcastillo|roverjust for reference, where do we keep downstream criteria?17:19
rlandycurrent-tripleo/2022-07-04 13:31 17:19
rlandyrcastillo|rover: https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=tree;f=ci-scripts/dlrnapi_promoter;h=4f081e40258d8efee8b395f61d38b18700b300c6;hb=HEAD17:20
rcastillo|roverrlandy: ack thanks17:20
rlandynot promoted yet17:20
rlandykicks once an hour17:20
rlandyrcastillo|rover: should kick in 10 mins17:21
rlandyrcastillo|rover: here are the components we need to rekick ... so testprojects with the all the jobs that run in the rhel 9 line ...17:21
rlandynetwork17:22
rlandycinder17:22
rlandybaremetal17:22
rlandycompute 17:22
rlandycommon17:22
rlandyclients17:22
rlandytripleo17:22
rlandyrcastillo|rover: so you want to take the top three (network, cinder, baremetal)? I'll take compute common clients tripleo?17:23
rlandyI'd suggest one patch per component17:24
rlandyso we can rerun if needed17:24
rlandyrcastillo|rover: ^^ ok?17:24
rcastillo|roverrlandy: sounds good17:24
rlandyonly rhel-917:24
rlandywe can prep the patches - I'll ping you when I see the promo to submit17:24
rlandyhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=ci-scripts/dlrnapi_promoter/config/RedHat-9/component/rhos-17.yaml;h=cf63558ec5f5573ba04754d5418d8c3a2c9abbc0;hb=HEAD17:26
rlandyrcastillo|rover: ^^ that file17:26
rlandycriteria17:26
rlandyand definitions are in:17:26
rcastillo|roverI found the definitions I think17:27
rcastillo|roverin internal-jobs17:27
rlandyhttps://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/component-pipeline-rhos-17-rhel9.yaml;h=ea46b2aed64b96d0eb3db05cdcdefb939ec3740b;hb=HEAD17:27
rlandyI restarted the service17:33
rlandyshoudl run now17:33
rlandycurrent-tripleo/2022-07-06 13:32 17:33
rlandydone17:33
rlandyrcastillo|rover: and tempest17:38
rlandy^^ forgot that one17:38
rcastillo|roverI'll get that one17:38
rlandygreat17:39
rlandyok - my four are in17:39
rcastillo|roverjust sent the tempest tp17:42
dasmrcastillo|rover: qq. are we having node_failures issues? I don't want to blindly recheck the job: https://review.rdoproject.org/r/c/testproject/+/4136718:11
rlandyrcastillo|rover++18:11
rlandynow we watch and wait18:11
rlandyrcastillo|rover: c9 is all hosed o rdo so nothing we can do there18:12
dasmhttps://review.rdoproject.org/zuul/builds?result=NODE_FAILURE&skip=018:16
rlandyrcastillo|rover: is there a train criteria revert?18:22
rlandyhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4368218:25
rlandy^^ done18:25
rcastillo|roverrlandy ack18:57
rlandyrcastillo|rover: you all set with the ipa investigation?19:03
rlandystill waiting on the tqe fix from john for rhel-819:04
rcastillo|roverrlandy: got hit by node failures for the ipa job19:06
rlandyah19:06
rlandyyou can try that on opendev19:06
rcastillo|roverand no response from ade yet19:06
rlandydnm patch19:06
rcastillo|roverrlandy: ahh, right19:06
rlandywith depends-on19:06
* rlandy works on requirements - will check in on rhle-9 components19:07
rcastillo|roverrlandy: testing here https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84889619:20
arxcruzdasm https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4381519:21
arxcruzdasm just tested, the container is being built properly 19:21
rlandyok19:32
dasmarxcruz: did you test my recent update? 19:38
rlandyrcastillo|rover: can you rerun the missing fs001 to promote wallaby c8?19:51
rlandyrcastillo|rover: rhel-8 in test with tqe patch19:53
rcastillo|roverrlandy: sure19:55
rlandyrcastillo|rover: thanks19:57
arxcruzdasm yes, i did something similar, but go-pretty didn't compile anyway 20:03
dasmarxcruz: ack20:05
*** dviroel is now known as dviroel|out20:50
* dasm => offline21:36
*** dasm is now known as dasm|off21:36
rlandyrcastillo|rover: c9 nodes are back 21:45
rlandynhicher put new build on pause21:45
rlandyrcastillo|rover: going out for a bit - will check status when back21:46
*** rlandy is now known as rlandy|biab21:50
*** rlandy|biab is now known as rlandy23:01
rlandyysandeep|out: node_failures in rhel-9 nodes now23:36
rlandypls check in with rhos-ops23:36
*** rlandy is now known as rlandy|out23:37

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!