Wednesday, 2020-05-06

*** jmasud has joined #oooq00:07
*** rlandy has quit IRC00:20
*** rfolco has quit IRC00:35
*** jmasud has quit IRC01:17
*** jmasud has joined #oooq01:39
*** matbu has quit IRC01:50
*** jmasud has quit IRC02:11
*** jmasud has joined #oooq02:49
*** jmasud has quit IRC03:16
weshay|ruckakahat|ruck, sending you an email about the program call tomorrow w/ what to do and say03:20
weshay|ruckakahat|ruck, feel free to drop that call after reporting status and join the retrospective.03:21
weshay|ruckthanks03:21
*** ykarel|away is now known as ykarel03:54
*** openstack has joined #oooq04:21
*** ChanServ sets mode: +o openstack04:21
*** matbu has quit IRC04:29
*** ratailor has joined #oooq04:37
*** udesale has joined #oooq04:40
*** jmasud has joined #oooq05:00
*** sanjayu__ has joined #oooq05:14
*** sanjayu_ has quit IRC05:17
*** skramaja has joined #oooq05:44
*** sanjayu__ has quit IRC05:48
*** ysandeep|away is now known as ysandeep05:51
*** saneax has joined #oooq05:54
*** marios has joined #oooq05:58
*** dpawlik has joined #oooq06:03
*** ysandeep is now known as ysandeep|away06:12
*** jmasud has quit IRC06:17
*** jmasud has joined #oooq06:19
*** ysandeep|away has quit IRC06:29
*** jfrancoa has joined #oooq06:34
*** matbu has joined #oooq06:35
*** ysandeep has joined #oooq06:35
*** ysandeep is now known as ysandeep|brb06:38
*** ysandeep|brb is now known as ysandeep06:39
*** jmasud has quit IRC06:51
*** pojadhav has quit IRC06:53
*** cgoncalves has joined #oooq06:55
*** jmasud has joined #oooq06:55
*** jbadiapa has joined #oooq06:59
*** ccamacho has joined #oooq07:04
*** saneax has quit IRC07:06
*** pojadhav has joined #oooq07:10
*** saneax has joined #oooq07:11
*** yolanda has joined #oooq07:14
*** jmasud has quit IRC07:14
*** sanjayu_ has joined #oooq07:17
*** jmasud has joined #oooq07:19
*** saneax has quit IRC07:19
*** bhagyashris has quit IRC07:23
*** amoralej|off is now known as amoralej07:23
*** tosky has joined #oooq07:30
*** bhagyashris has joined #oooq07:31
*** cgoncalves has quit IRC07:34
*** cgoncalves has joined #oooq07:40
*** jmasud has quit IRC07:40
*** dtantsur|afk is now known as dtantsur07:40
*** bogdando has joined #oooq07:43
*** openstackstatus has quit IRC07:46
*** openstackstatus has joined #oooq07:46
*** openstack has joined #oooq07:47
*** ChanServ sets mode: +o openstack07:47
*** bogdando_ has joined #oooq07:56
*** bogdando has quit IRC07:57
*** bogdando_ is now known as bogdando07:57
*** jpena|off is now known as jpena07:58
*** dmellado has quit IRC07:58
*** ysandeep is now known as ysandeep|lunch08:03
*** dmellado has joined #oooq08:04
*** dpawlik has quit IRC08:09
*** dpawlik has joined #oooq08:09
*** ykarel is now known as ykarel|lunch08:45
akahat|ruckmarios, sshnaidm, chandankumar o/08:49
akahat|ruckhttps://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens/bb68d90/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz08:49
akahat|ruckseen this issue before?08:50
mariosakahat|ruck: o/ perhaps a bad promotion or a promotion happened while the job was running?08:50
mariosakahat|ruck: 2020-05-06 07:52:29 | docker pull failed: Error: image tripleoqueens/centos-binary-rabbitmq:d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708 not found08:50
mariosakahat|ruck: is likely transient have you seen it more than once ?08:50
akahat|ruckmarios, no found only in 1 job.08:50
akahat|ruckrest of the pipeline is running..08:50
sshnaidmakahat|ruck, hi, check if it was built08:59
akahat|rucksshnaidm, where we can check it? We jsut can not browse to trunk.registry.rdoproject.org/tripleoqueens .09:02
sshnaidmakahat|ruck, wait, is it update job? so d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708 is the previous hash actually09:02
*** jmasud has quit IRC09:02
sshnaidmakahat|ruck, the image is not there, you can try to pull it and it's not found09:03
akahat|rucksshnaidm, no. It's Wednesday pipeline09:03
sshnaidmakahat|ruck, firstly need to understand why this job tries to get image with hash d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb47009:05
sshnaidmakahat|ruck, the hash that is trying to promote now is 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91 https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-queens-promote-consistent-to-tripleo-ci-testing/0b12606/job-output.txt09:06
sshnaidmakahat|ruck, featureset037-updates-queens is updates job, marios can tell more about it :)09:06
mariossshnaidm: akahat|ruck: i checked tripleo-ci-testing was updated yesterday to that                   d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb47009:07
marios12:06 < sshnaidm> akahat|ruck, the hash that is trying to promote now is 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e9109:07
mariossshnaidm: sorry to that https://trunk.rdoproject.org/centos7-queens/tripleo-ci-testing/delorean.repo09:07
mariosi.e. 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e9109:07
sshnaidmyep, so 037 job firstly tries to deploy previous hash (d50..) and then to update it to promoting one (9c5..), marios - correct?09:08
mariosakahat|ruck: try a testproject run for that job and see if it happens again then we can know if its cos of some transient issue (i.e. tripleo-ci-testing update) or somethin gelse09:08
mariossshnaidm: yeah sounds right ;)09:09
mariossshnaidm: you can confirm it there https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens/bb68d90/logs/quickstart_files/releases.sh09:09
mariosexport OVERCLOUD_DEPLOY_RELEASE="promotion-testing-hash-queens"09:10
mariosexport OVERCLOUD_DEPLOY_HASH="d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708"09:10
mariosexport OVERCLOUD_TARGET_RELEASE="promotion-testing-hash-queens"09:10
mariosexport OVERCLOUD_TARGET_HASH="9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91"09:10
sshnaidmyep, and d50... images were built in 30 Apr: https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-queens-containers-build/ef572f2/job-output.txt09:10
sshnaidm06 May - 30 Apr = 1 week09:11
akahat|ruckonly OVERCLOUD_DEPLOY_HASH is old.09:11
sshnaidmakahat|ruck, pruning script of infra deleted them09:11
sshnaidmthis is my assumption ^09:11
sshnaidmyou can check it in #rhos-ops with on-duty folks09:12
akahat|rucksshnaidm, okay.09:12
sshnaidmakahat|ruck, it should be old, we deploy previous hash and then deploy an update to current hash, this what job should do09:13
sshnaidmakahat|ruck, this is usual with updates/upgrades job09:13
mariosakahat|ruck: sshnaidm: confirm the hash is not there @ https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleoqueens/imagestreamtags/09:14
sshnaidmakahat|ruck, good way to check ^09:15
sshnaidmmarios, but current release for queens is 05f87142c5b81c53e6200c6bf1936df3d5a9e991_bcf3c00b and not d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb470809:18
sshnaidmmarios, why does it take d50.. then?09:18
sshnaidmoh, I see d501b27616cec68ce1b35dd5345ebb220e73d3c3 is "previous-current-tripleo": https://trunk.rdoproject.org/centos7-queens/previous-current-tripleo/delorean.repo09:19
*** ysandeep|lunch is now known as ysandeep09:20
* dpawlik reading09:20
akahat|rucksshnaidm, i'm still confused with the baseurl hash and name hash in this repo file ^^09:25
*** ykarel|lunch is now known as ykarel09:27
akahat|rucksshnaidm, sorry.. got it.09:29
sshnaidmakahat|ruck, do you understand what happened?09:30
akahat|rucksshnaidm, no not exactly what's happening in the job.09:31
sshnaidmakahat|ruck, so 037 is updates job which firstly deploys all with "previous-current-tripleo" hash, then make update to "tripleo-ci-testing" hash09:32
sshnaidmakahat|ruck, we check that updates work in this job, so we can update openstack to a newer version09:33
sshnaidmfor doing this we deploy old version and run updates to a current version09:34
sshnaidmakahat|ruck, that's why you see old hash in  OVERCLOUD_DEPLOY_RELEASE09:34
sshnaidma newer hash is in  OVERCLOUD_TARGET_HASH09:35
akahat|rucksshnaidm, okay. Got it.09:35
sshnaidmakahat|ruck, but we have a pruning script in registry that deletes week old images09:35
sshnaidmakahat|ruck, because "previous-current-tripleo" wasn't whitelisted there, script just deleted all containers with this hash09:36
akahat|ruckoh.. that's bad...09:36
sshnaidmakahat|ruck, so now dpawlik added this type of hashes to whitelist and we hopefully won't hit it again :)09:36
akahat|rucksshnaidm, that's great :)09:38
akahat|rucksshnaidm, marios thank you :) Now it's more clear.09:38
*** owalsh has quit IRC09:46
akahat|rucksshnaidm, there is one more issue with the upgrades. https://bugs.launchpad.net/tripleo/+bug/187703109:55
openstackLaunchpad bug 1877031 in tripleo "queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version" [High,Triaged]09:55
akahat|ruckit also because of the same reason?09:55
*** owalsh has joined #oooq10:09
sshnaidmakahat|ruck, nope, no containers error there10:24
sshnaidmakahat|ruck, arxcruz is there a bug about tempest failures in ovb stable branches?10:32
arxcruzsshnaidm: https://bugs.launchpad.net/tripleo/+bug/1876972 ?10:32
openstackLaunchpad bug 1876972 in tripleo "tempest failing across RDO sf stein jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)10:32
sshnaidmhttps://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch10:32
sshnaidmit's also rocky and queens ^10:32
arxcruzsshnaidm: yeah10:33
arxcruzsshnaidm: https://trello.com/c/niYGK63i/1475-cixlp1876087tripleociproa-queens-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-timeout10:33
arxcruzjust wrote this10:33
sshnaidmarxcruz, cool10:33
chandankumarsshnaidm, marios please have a look at c8 train container build job https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/build.log10:33
chandankumarcontainers are getting builts but it is not showing , and the job is passing10:33
arxcruzsshnaidm: the main difference between upstream and the failing one in rdo, is the kernel version that is slightly different, and the failing one is running on AMD, while the passing one is running on Intel10:34
sshnaidmBuilt...................... 0 SUCCESS: Congratulations! All containers have been built :)10:34
sshnaidmchandankumar, neat ^10:34
sshnaidmarxcruz, I see, wonder why it pass on train10:35
marioschandankumar: middle sthing right now will check in bit10:35
arxcruzsshnaidm: did it pass? lol, it's so many failures that i didn't "catch the train" yet10:35
arxcruzlol10:36
sshnaidmarxcruz, all green https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch#10:36
sshnaidmmaybe something wasn't backported..10:37
chandankumarsshnaidm, if you check this file https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/containers-consolidated-builds.log containers are actually built10:40
marioschandankumar: indeed .. you can see logs for the builds at each container e.g. https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/buildah-builds/kolla-builds/1da3495d-007a-4b08-827c-1b545d9a24ea/docker/aodh/aodh-base/aodh-base-build.log10:44
marioschandankumar: are containers pushed?10:45
chandankumarmarios, nope, [Retrieve built images] is empty, these is no stdout, as it prints in master job10:46
marioschandankumar: wonder if it is related to https://review.opendev.org/#/c/718433/ merged 9 hours ago10:46
*** derekh has joined #oooq10:46
chandankumarmarios, let me check other builds10:47
marioschandankumar: hmm push false? https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/conf/kolla-build.conf10:48
marioschandankumar: no the push/false is unrelated i think10:49
chandankumarmarios, https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/tripleoci.yaml#l39210:49
chandankumarit is set to true so unrelated10:49
marioschandankumar: but master is same https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/660759a/logs/build.log10:50
chandankumarmarios, but master errors are legit10:50
chandankumarmarios, from success job https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/2d6fb0c/job-output.txt 2020-05-06 00:43:38.221773 | TASK [Retrieve built images]10:51
chandankumar2020-05-06 00:43:38.814114 | primary | trunk.registry.rdoproject.org/tripleomaster/centos-binary-sahara-api:93c75d0c65654d0ce6ba51da601f9e9b10:51
*** sanjayu__ has joined #oooq11:03
ykarelchandankumar, need to handle spacial namespace, tripleotraincentos8, buildcontainers role not handling that currently11:03
ykarelit assumes tripleo{{ release }} currently11:04
*** sanjayu_ has quit IRC11:05
chandankumarykarel, but if you check consolidated build log the namespace is used correctly11:11
ykarelchandankumar, building is fine, but the way it detects built containers, is missing handling of special namespace11:14
ykarelsee https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/build.sh.j2#L5611:15
ykarelsudo {{ container_cli }} images | grep tripleo{% if osp_branch is defined %}{{ osp_branch }}{% else %}{{ ci_branch }}{% endif %}/{{ kolla_base }}-binary > containers-successfully-built.log11:15
ykarelit search tripleotrain/centos11:16
*** ysandeep is now known as ysandeep|brb11:16
chandankumarykarel, ah, let me fix that11:18
ykarelack11:20
*** rfolco has joined #oooq11:30
chandankumarykarel, https://review.opendev.org/#/c/725831/11:33
ykarelchandankumar, ack11:35
*** jpena is now known as jpena|lunch11:39
chandankumarmarios, sshnaidm it will fix c8 train container issue https://review.opendev.org/#/c/725831/11:39
*** ysandeep|brb is now known as ysandeep11:54
*** rlandy has joined #oooq11:58
weshay|ruckakahat|ruck, 0/11:58
weshay|ruckakahat|ruck, you know what to do and when you can drop from the program call?11:59
chandankumarweshay|ruck, I will be there with him12:00
rfolcoarxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar12:02
weshay|ruckchandankumar, you can join us if you like.. Amol's GOT THIS... :)12:02
rfolcoping retro12:02
rfolcoarxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar12:02
rfolcoping retro12:02
rfolcohttps://meet.google.com/bop-gsjm-nhp12:02
chandankumarweshay|ruck, ok, joining retro12:02
rfolcohttps://trello.com/b/0VFswmht/pcci-retrospective?menu=filter&filter=label:UniSprint2612:02
weshay|ruckakahat|ruck, phil and mike burns can help.. if you need it12:03
*** amoralej is now known as amoralej|lunch12:03
*** holser_ has joined #oooq12:10
akahat|ruckweshay|ruck, yeah sure. Thanks :)12:18
*** udesale_ has joined #oooq12:23
*** udesale has quit IRC12:26
*** ykarel is now known as ykarel|afk12:38
*** Goneri has joined #oooq12:38
*** jpena|lunch is now known as jpena12:45
*** sanjayu_ has joined #oooq12:47
*** sanjayu__ has quit IRC12:50
mariosrfolco: 1 min left ... picking ruck|rover12:59
mariosweshay|ruck: ^12:59
rfolcothx marios12:59
*** chem_ is now known as chem13:01
rfolcobhagyashris, sorry, you'll be fine :)13:02
bhagyashrisrfolco, np13:02
*** ratailor has quit IRC13:02
chandankumarweshay|ruck, you have few mins for train c8?13:04
rfolcobhagyashris, you want to be ruck or rover first ?13:05
weshay|ruckchandankumar, locked in a 2hr meeting..13:07
weshay|ruckchandankumar, we got past get_hash afaict13:07
bhagyashrisrfolco, i will first go with ruck13:07
weshay|ruckbut deployment failed13:07
chandankumarweshay|ruck, yes, new issue13:08
rfolcobhagyashris, clever13:08
rfolco:)13:08
chandankumarfs01 and 10 failed right now13:08
chandankumarweshay|ruck, I think I need to https://review.opendev.org/#/c/712289/ backport it for train for sc113:08
weshay|ruckchandankumar, let's focus on merging hte deps.. because those jobs reset several times13:08
weshay|ruckw/ rebases etc13:08
weshay|ruckchandankumar, I can do irc now13:08
rfolcoweshay|ruck, akahat|ruck let us know when we switch nicks, call me and bhagyashris for the handoff/sync13:08
chandankumarweshay|ruck, https://review.opendev.org/#/c/725831/ needed for container build13:08
weshay|ruckchandankumar, that's a good change13:09
chandankumarweshay|ruck, fs010 I think there is a patch already13:10
chandankumarlet me find that13:10
akahat|ruckrfolco, it's depend on weshay|ruck schedule. about nick.. whenever you want.13:12
akahat|ruckbhagyashris, ^^13:12
weshay|ruckakahat|ruck, and bhagyashris's day is ending shortly.. so you folks can unnick13:13
weshay|ruckrfolco, you should renick to rover13:13
*** rfolco is now known as rfolco|rover13:13
*** akahat|ruck is now known as akahat13:13
rfolco|roverakahat, how is the feeling ?13:13
chandankumarweshay|ruck, and https://review.opendev.org/#/c/725852/1 and https://review.opendev.org/#/c/722284/313:13
rfolco|rover:)13:13
chandankumarto clear sc 1 and 10 for c8 train13:14
chandankumarweshay|ruck, updating the reviews13:14
akahatrfolco|rover, good. Learn lot's of new things... random discussion.. prodchain flow.. etc. :)13:14
akahatExciting..13:14
rlandymarios: weshay|ruck: we got the deps we were missing in trunk-candidate. Building and pushing containers again13:16
*** pojadhav is now known as pojadhav|afk13:16
akahatrlandy, pojadhav bhagyashris o/13:16
chandankumarweshay|ruck, https://review.rdoproject.org/r/#/c/26920/ tracks all the patches for c8 train13:16
weshay|ruckrlandy, nice13:16
rlandyakahat: hey13:16
akahatrlandy, pojadhav|afk bhagyashris this is came up today. https://bugs.launchpad.net/tripleo/+bug/187703113:16
openstackLaunchpad bug 1877031 in tripleo "queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version" [High,Triaged]13:16
akahatrlandy, are you the right person of contact?13:16
rlandyakahat: I think you are looking for the new ruck/rovers?13:17
rlandyrfolco|rover: ^^13:17
rlandyakahat: rfolco|rover will be taking over from you13:17
rfolco|roverakahat, hit me13:17
akahatrlandy, okay. Thanks :)13:17
akahatrfolco|rover, this is failing because of the lower version of ansible.13:18
rlandymarios: want to meet today - or ok if I carry on with standalone?13:18
rlandyalso ... https://code.engineering.redhat.com/gerrit/#/c/198838/ can merge13:18
rfolco|roverakahat, assign to me and I'll take a look13:18
*** skramaja has quit IRC13:18
akahatrfolco|rover, okay13:18
*** bhagyashris is now known as bhagyashris|ruck13:18
*** cgoncalves has quit IRC13:19
mariosrlandy: yes lets meet13:23
mariosrlandy: give me couple mins (got the get_hash stuff ok but i am hitting the release file issue now)13:23
chandankumarmarios, does my changes to get_hash broke your stuff?13:23
*** cgoncalves has joined #oooq13:25
rlandymarios: cool - ping me when ready - the release file is what I wanted to discuss13:25
marioschandankumar: no no it was more like people working on things at the same time and it changes... e.g. you look at log files but the code is different cos it was changed e.g. last night and you didn't notice, so you get mismatch13:26
marioschandankumar: don't worry i about it i am just feeling low13:26
mariosrlandy: now ok for you?13:27
rlandymarios; yep - 3 mins?13:28
mariosrlandy: ack13:28
*** TrevorV has joined #oooq13:31
rlandymarios: joining community call13:32
mariosrlandy: ack13:32
weshay|ruckgate failed on https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/13:36
weshay|ruckchandankumar, need a favor13:36
weshay|ruckhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ac/717850/24/gate/openstack-tox-molecule/3aca407/job-output.txt13:36
chandankumarweshay|ruck, where is the image?13:36
weshay|rucklet's change those molecule tests.. to test for current-tripleo vs. tripleo-ci-testing13:36
weshay|ruckchandankumar, missing13:36
chandankumarweshay|ruck, there is an option to disable that check13:37
chandankumarweshay|ruck, https://review.opendev.org/#/c/717850/24/config/release/tripleo-ci/CentOS-8/train.yml@1213:38
chandankumarweshay|ruck, validate_on: rdo13:38
chandankumarlet me update the patch13:38
chandankumarthe option is there in both files why it is failing then13:40
*** amoralej|lunch is now known as amoralej13:44
weshay|ruckrfolco|rover, can you prep a new hackmd for the sprint13:45
rfolco|roverweshay|ruck, planning ? okay13:45
chandankumarweshay|ruck, there is an issue with c8 master image build https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/30ab58a/build.log13:45
weshay|ruckruck rover13:45
chandankumarFailed to download metadata for repo 'centos7-rt'13:45
weshay|ruckchandankumar, aye.. and tripleo-ci-testing should not block the upstream gate imho13:46
chandankumarweshay|ruck, in c8 from where c7 repo is coming13:46
weshay|ruckrepo?13:48
chandankumarweshay|ruck, https://review.opendev.org/72585813:50
chandankumarit will unblock molecule tests13:50
weshay|ruckchandankumar, comment13:52
*** ykarel|afk is now known as ykarel13:54
chandankumarweshay|ruck, commented and updated13:59
chandankumarweshay|ruck, please donot recheck train patches, as rdo review c8 train jobs running14:00
*** ysandeep is now known as ysandeep|away14:02
*** bogdando_ has joined #oooq14:04
rfolco|roverweshay|ruck, taiga board and meeting notes created14:05
rfolco|roverbrb14:06
*** bogdando has quit IRC14:06
*** bogdando_ is now known as bogdando14:06
weshay|ruckrfolco|rover, rdo mtg14:14
rfolco|roverweshay|ruck, ack14:14
weshay|ruckchandankumar, interesting.. that skips the curl check completely https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c2e/725858/2/check/openstack-tox-molecule/c2e16a9/job-output.txt14:16
weshay|ruckI don't understand that test I guess14:17
chandankumarweshay|ruck, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/tests/validate-release-config.yml#L914:18
weshay|ruckya.. I see that.. don't get it14:19
weshay|ruckit's defined14:19
weshay|ruckwhat does "rdo" mean14:19
weshay|ruckdoes it mean.. only check from 3rd party?14:19
chandankumarweshay|ruck, it was initially added just for rdo urls14:21
chandankumarby zbr14:21
weshay|ruckwhat's a rdo url?14:22
chandankumarweshay|ruck, i mean for images.rdorproject.org14:23
* weshay|ruck is confused.. I'll raise a bug and assign it to zbr14:23
chandankumarweshay|ruck, ok14:23
weshay|ruckI think promote_source could be set to current-tripleo for established releases14:24
weshay|rucka new release w/o a prmotion to current-tripleo.. tripleo-ci-testing makes sense14:24
rlandychandankumar: hey is https://code.engineering.redhat.com/gerrit/#/c/196373/ ready to go?14:26
rlandychandankumar: we're almost ready to merge the mini-pipeline which includes those jobs14:27
chandankumarrlandy, yes, good to go14:30
*** sanjayu__ has joined #oooq14:30
chandankumarrlandy, do we want to run build images jobs in check pipeline?14:32
weshay|ruckrfolco|rover, can you please join the cix call.. I'm caught in a perf review mtg14:32
chandankumarweshay|ruck, you mean this part https://review.opendev.org/#/c/717850/24/config/release/tripleo-ci/CentOS-8/train.yml@11 ?14:32
weshay|ruckrfolco|rover, just do your best14:32
*** sanjayu_ has quit IRC14:33
rfolco|roverweshay|ruck, joined14:35
weshay|ruckrfolco|rover++14:35
*** udesale_ has quit IRC14:38
*** ccamacho has quit IRC14:39
rlandychandankumar: we'll do that after we get integration going14:40
chandankumarrlandy, aye thanks :-)14:40
rlandymarios: ^^ pls vote on https://code.engineering.redhat.com/gerrit/#/c/196373 so we can merge14:40
rlandychandankumar: marios: weshay|ruck: anyone else .. pls review TLS reviews ... would love to get those in ...14:48
rlandyhttps://review.opendev.org/#/c/70741914:48
rlandy https://review.opendev.org/#/c/72490714:48
rlandyhttps://review.opendev.org/#/c/720757/14:48
mariosrlandy: ack will do14:48
rlandyhttps://review.opendev.org/#/c/706288/14:48
rlandychandankumar: marios: weshay|ruck: thanks ^^ 4 reviews above ... results in test review https://review.opendev.org/#/c/72485014:49
mariosrlandy: ack but not right now will check them more carefully tomorrow concentrating on the release stuff now14:51
mariosrlandy: so is anything using tripleo-environments/rhos-17 release file currently? git blame says you added that in december14:57
mariosrlandy: wondering if i should re-use it or make new one14:57
* marios starts editing in place14:57
rlandymarios: per meeting .. pls move that fil eto rhos-17-compose14:57
rlandyand add two new ones14:57
mariosrlandy: ack14:57
rlandyfor check and promotion14:58
mariosrlandy: but does something use it?14:58
mariosrlandy: so i'll have to edit whatever uses it to instead take rhos-17 compose14:58
* marios grep14:58
rlandymarios: there is one test that uses it14:58
rlandybut let's delete it14:58
rlandythere has not been a compose yet14:58
mariosrlandy: in which case why not just use that one?14:58
mariosrlandy: and keep the upstream pattern.. rhos-17 for check and then the testing-hash one for promotions14:59
rlandymarios: becuase at somepoint we may want to compare14:59
mariosrlandy: k k going with new files14:59
rlandyie: why compose works14:59
rlandyor not14:59
rlandychandankumar: https://code.engineering.redhat.com/gerrit/#/c/196373/ failed the two tests15:04
rlandyshould they pass now?15:04
rlandyif not, can we make them non-voting so we can merge the code15:04
chandankumarrlandy, legit failure, fixed now it will pass15:08
rlandychandankumar++15:09
mariosrlandy: so actually all the component rhel8/17 jobs are using that rhos-17 ?15:14
mariosrlandy: ah no as 'osp_release' not release-file sorry15:15
rlandymarios: we need to rework those jobs15:15
rlandythey are not running now15:15
rlandyfrom old pipeline15:15
rlandydon't worry about it15:15
mariosrlandy: well i think we need to do _something_ for our sanity/future generations... like documenting a lot of this stuff cos its a completely different world tooling-wise15:15
rlandymarios: sure15:15
mariosrlandy: but of course that is secondary :) /me shutsup carry one15:15
marioscarry on even15:16
rlandymarios: we're trying to make it look more like upstream15:16
rlandyso to your point15:16
mariosrlandy: like e.g. this get_build_command business...15:16
mariosrlandy: yeah 18:16 < rlandy> marios: we're trying to make it look more like upstream15:16
rlandywe could move the files we want to use to RedHat815:16
mariosrlandy: but the problem is there are these cases that we mix the two worlds15:16
rlandylike we have in upstrea,15:16
rlandyget_build will go away15:17
rlandywe needed thta for compose15:17
rlandythat15:17
*** ccamacho has joined #oooq15:17
*** ykarel is now known as ykarel|away15:23
rlandymarios: you can just use https://code.engineering.redhat.com/gerrit/#/c/196373/38/playbooks/pre-rhos.yaml15:25
rlandyas is chandan's review15:26
rlandyexcept don't hard code tripleo-ci-testing15:26
mariosrlandy: i was going to add into the release file with includepkgs as we discussed for delorean/deps ie like upstream?15:27
rlandysure15:27
mariosrlandy: ack well adding another pre might be easier - we can see what the release file looks like and revisit on the review?15:28
rlandyok15:28
rlandymarios: ugh - we're missing one more dep now - we didn't build nova - logging BZ15:30
rlandypackage openstack-nova-common-1:21.1.0-0.20200505172042.dddc0dc.el8ost.noarch requires python3-nova = 1:21.1.0-0.20200505172042.dddc0dc.el8ost, but none of the providers can be installed]15:30
mariosrlandy: ack thanks15:38
mariosrlandy: we kinda expected this :/ i mean dep after dep but the cadence is kinda worrying here... if only we had a job that would prevent us from doing that :D15:39
rlandymarios: coming soon to a pipeline near you15:39
*** jmasud has joined #oooq15:41
rlandymarios: before you go, please remind me how to create a CIX for  https://bugzilla.redhat.com/show_bug.cgi?id=183237815:44
openstackbugzilla.redhat.com bug 1832378 in distribution "Cannot build nova-base and mistral-executor containers for rhel8-osp17 - missing python3-nova" [Unspecified,New] - Assigned to lhh15:44
mariosrlandy: search "[CIX]" emails in rhos-dev... you need to send one of those and then lon has tooling that picks it up15:45
mariosrlandy: not sure we want the includepackages stuff cos it should come from delorean we won't have delorean-current for this job i think ?15:48
rlandyyep - i think so15:49
*** ccamacho has quit IRC15:51
*** jmasud has quit IRC15:55
rfolco|roverbhagyashris|ruck, weshay|ruck rr notes created https://hackmd.io/2MdkNAUuT7aBcM0Yck4xnw?both15:55
rlandychandankumar: you have a post-failure https://sf.hosted.upshift.rdu2.redhat.com/logs/73/196373/38/check/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/7302520/job-output.txt16:03
*** panda|pto has quit IRC16:03
rlandyshould be a minor fix16:03
*** jmasud has joined #oooq16:04
*** panda has joined #oooq16:06
*** dtantsur is now known as dtantsur|afk16:11
mariosrlandy: k it will be wildly wrong but its a start @ lucky v13 @ https://code.engineering.redhat.com/gerrit/#/c/198789/ and environment-file in depends-on16:13
mariosrlandy: i'll have to pick it up tomorrow again16:13
rlandymarios: cool - will carry on from @1316:13
rlandymarios: hoping to merge the image build jobs as well16:14
rlandyas then pipeline can merge16:14
mariosrlandy: yeah i'll get to those if they aren't merged by morning16:15
rlandymarios: also go a fix for the dep failures16:15
rlandywaiting for an updated candidate repo16:15
marios rlandy: that was quick :)16:15
rlandymarios: sent link - the latest build was 6am16:16
rlandyfix came in at 10ish16:16
*** jmasud has quit IRC16:21
*** jmasud has joined #oooq16:23
*** jmasud has quit IRC16:23
*** marios is now known as marios|out16:27
rlandychandankumar: thanks - merging https://code.engineering.redhat.com/gerrit/#/c/196373/16:28
chandankumarrlandy, aye thanks!16:28
*** sanjayu__ has quit IRC16:38
chandankumarrlandy, weshay|ruck rfolco|rover I think we can delete this card https://tree.taiga.io/project/tripleo-ci-board/task/1626?kanban-status=1447274 and will cover as a part of integration pipeline?16:42
weshay|ruckchandankumar, sure16:42
*** bogdando has quit IRC16:43
*** marios|out has quit IRC17:01
*** jpena is now known as jpena|off17:08
*** holser_ has quit IRC17:09
*** holser has joined #oooq17:09
*** holser_ has joined #oooq17:13
*** holser has quit IRC17:14
*** pojadhav|afk is now known as pojadhav17:15
*** amoralej is now known as amoralej|off17:15
bhagyashris|ruckrfolco|rover, ack17:20
*** irclogbot_0 has quit IRC17:20
*** irclogbot_3 has joined #oooq17:22
*** derekh has quit IRC17:26
*** sshnaidm is now known as sshnaidm|afk17:26
*** hamzy_ has quit IRC17:52
*** hamzy has joined #oooq17:54
weshay|ruckrlandy, interestingly enough.. I think the issue ur hitting w/ nova in component builds17:55
weshay|ruckis also failing upstream components17:56
weshay|ruck  - package openstack-nova-common-1:21.1.0-0.20200505172042.dddc0dc.el8ost.noarch requires python3-nova = 1:21.1.0-0.20200505172042.dddc0dc.el8ost, but none of the providers can be installed17:56
weshay|ruckhttp://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&fullscreen&panelId=42317:56
rlandyweshay|ruck: ack17:57
weshay|ruck cannot install the best update candidate for package openstack-nova-common-1:21.1.0-0.20200502034918.fdcb1e3.el8.noarch\n      - nothing provides python3-oslo-policy >= 3.1.017:57
rlandywas fixed in compose17:57
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/e1190f4/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz17:57
weshay|ruckrlandy, ah k17:57
rlandywaiting for updated rmp to hit the candidate repo17:57
rlandyweshay|ruck: https://review.opendev.org/#/c/724907/ - can you w+ that?18:01
rfolco|roveris review.rdoproject.org ok for you weshay|ruck18:08
rfolco|roveractually... zuul is down18:09
rlandychandankumar: where do downstream images get pushed?18:11
rlandyovercloud_image_url?18:11
rlandyipa_image_url?18:11
chandankumarrlandy, we have not got any places to push images to image server18:15
chandankumarrlandy, apevec is working on that18:16
rlandychandankumar: ok - will update that setting later18:16
chandankumarrlandy, we can use to_build flag with in ovb job to rebuild the image and use it18:16
chandankumarrlandy, weshay|ruck https://review.rdoproject.org/r/27289 and https://review.rdoproject.org/r/27307 final reviews for fs0218:25
*** hamzy_ has joined #oooq18:39
*** hamzy has quit IRC18:42
*** jmasud has joined #oooq19:04
*** jmasud has quit IRC19:18
*** yolanda has quit IRC19:33
*** jfrancoa has quit IRC19:59
weshay|ruckrfolco|rover any updates from docker.io?20:09
rfolco|roverignored20:11
rfolco|roverweshay|ruck, we'll have to find a workaround.... or create the entire alphatbet and change the code to pull from triplea, tripleob, tripleoc...20:12
weshay|ruckcan you cc me20:12
rfolco|roverweshay|ruck, its a web form20:12
rfolco|roverthats why you can't cc20:12
rfolco|roverlet me get the link20:13
weshay|ruckk20:13
rfolco|roverhttps://hub.docker.com/support/contact/20:13
rfolco|roverweshay|ruck, do you want to try ?20:13
rfolco|roverweshay|ruck, did you change rr notes I created ? https://hackmd.io/2MdkNAUuT7aBcM0Yck4xnw20:27
weshay|ruckyes20:28
rfolco|roverweshay|ruck, ok so that was intentional. ok20:29
weshay|ruckrfolco|rover, I liked some of your changes.. but keeping the layout consistent is also important.. so I integrated a bit and changed a bit20:30
*** Goneri has quit IRC21:05
rlandyweshay|ruck: ping re: director images21:18
weshay|ruckrlandy, 0/21:20
weshay|ruckrfolco|rover, I don't think https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml#L1721:21
weshay|ruckis going to be a big deal21:21
rlandyweshay|ruck: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-17.yml includes a bunch of lines about director images21:21
weshay|ruckI think the release is defined in the dlrn-promoter.sh21:21
rlandyI assume we don;t have those with the component setup21:21
weshay|ruckrlandy, ya.. we don't want to use those21:22
rlandywe just download them from the server that will one day be set up21:22
rlandyk - deleting21:22
weshay|ruckso we can change it more of the upstream method21:22
weshay|ruck:)21:22
*** jmasud has joined #oooq21:28
rlandyweshay|ruck: any idea why if delorean.repo and delorean-current.repo are installed in https://sf.hosted.upshift.rdu2.redhat.com/logs/89/198789/17/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/969a4de/logs/undercloud/home/zuul/repo_setup.log21:54
rlandythat they don;t show up in:21:54
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/89/198789/17/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/969a4de/logs/undercloud/etc/yum.repos.d/21:55
rlandy2020-05-06 17:35:41 | Error: Unable to find a match: python3-tripleoclient ceph-ansible21:56
* weshay|ruck looks at release file21:56
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/199471/5/config/release/rhos-17-periodic.yml21:58
weshay|ruckrlandy, could be rhos-release -x22:00
rlandyweshay|ruck: ok - let me do that in a before command22:01
rfolco|roverweshay|ruck, okay, but its not (only) about pushing... but also pulling from...22:09
rlandyweshay|ruck: thanks - that worked to keep the delorean repos ... still missing  ceph-ansible22:17
rlandymaybe it's in build deps22:17
*** jmasud has quit IRC22:17
*** jmasud has joined #oooq22:18
weshay|ruckceph-ansible ah crud..22:18
weshay|ruckI don't think we add that right..22:18
weshay|ruckmeaning not in dlrn22:18
weshay|ruckrlandy, 2020-05-05 20:15:35 |  ceph-ansible                            noarch  4.0.14-1.el8cp                                  rhelosp-ceph-4-tools      196 k22:20
rlandyweshay|ruck: yep - those repos used to come with rhos-release -p 1722:21
weshay|ruckso /me wonders if rhos-release supports just ceph ansible22:21
rlandynow I need to install them manually22:21
weshay|ruckrlandy, or run it.. and disable repos except for base and -ceph-r22:21
weshay|ruckrlandy, or run it.. and disable repos except for base and -ceph-422:21
weshay|ruckrlandy, let's do this22:22
weshay|rucklet's update rhos-release to just support base + ceph22:22
rlandyhow?22:22
weshay|ruckrlandy, http://git.app.eng.bos.redhat.com/git/rhos-release.git/tree/rhos-release22:23
rlandyweshay|ruck: sorry - still not seeing it :(22:24
rlandyEnable testing (pending) repositories. Comma separated.22:25
weshay|ruckhttps://meet.google.com/det-vzdp-bpp22:25
weshay|ruckrm -f /etc/yum.repos.d/rhos-release-17-trunk.repo22:31
*** jmasud has quit IRC22:31
*** jmasud has joined #oooq22:32
rlandyweshay|ruck: https://review.opendev.org/#/c/724907/22:38
*** jmasud has quit IRC22:46
*** TrevorV has quit IRC22:54
*** tosky has quit IRC22:56
*** jmasud has joined #oooq23:03
*** holser_ has quit IRC23:09

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!