*** jmasud has joined #oooq | 00:07 | |
*** rlandy has quit IRC | 00:20 | |
*** rfolco has quit IRC | 00:35 | |
*** jmasud has quit IRC | 01:17 | |
*** jmasud has joined #oooq | 01:39 | |
*** matbu has quit IRC | 01:50 | |
*** jmasud has quit IRC | 02:11 | |
*** jmasud has joined #oooq | 02:49 | |
*** jmasud has quit IRC | 03:16 | |
weshay|ruck | akahat|ruck, sending you an email about the program call tomorrow w/ what to do and say | 03:20 |
---|---|---|
weshay|ruck | akahat|ruck, feel free to drop that call after reporting status and join the retrospective. | 03:21 |
weshay|ruck | thanks | 03:21 |
*** ykarel|away is now known as ykarel | 03:54 | |
*** openstack has joined #oooq | 04:21 | |
*** ChanServ sets mode: +o openstack | 04:21 | |
*** matbu has quit IRC | 04:29 | |
*** ratailor has joined #oooq | 04:37 | |
*** udesale has joined #oooq | 04:40 | |
*** jmasud has joined #oooq | 05:00 | |
*** sanjayu__ has joined #oooq | 05:14 | |
*** sanjayu_ has quit IRC | 05:17 | |
*** skramaja has joined #oooq | 05:44 | |
*** sanjayu__ has quit IRC | 05:48 | |
*** ysandeep|away is now known as ysandeep | 05:51 | |
*** saneax has joined #oooq | 05:54 | |
*** marios has joined #oooq | 05:58 | |
*** dpawlik has joined #oooq | 06:03 | |
*** ysandeep is now known as ysandeep|away | 06:12 | |
*** jmasud has quit IRC | 06:17 | |
*** jmasud has joined #oooq | 06:19 | |
*** ysandeep|away has quit IRC | 06:29 | |
*** jfrancoa has joined #oooq | 06:34 | |
*** matbu has joined #oooq | 06:35 | |
*** ysandeep has joined #oooq | 06:35 | |
*** ysandeep is now known as ysandeep|brb | 06:38 | |
*** ysandeep|brb is now known as ysandeep | 06:39 | |
*** jmasud has quit IRC | 06:51 | |
*** pojadhav has quit IRC | 06:53 | |
*** cgoncalves has joined #oooq | 06:55 | |
*** jmasud has joined #oooq | 06:55 | |
*** jbadiapa has joined #oooq | 06:59 | |
*** ccamacho has joined #oooq | 07:04 | |
*** saneax has quit IRC | 07:06 | |
*** pojadhav has joined #oooq | 07:10 | |
*** saneax has joined #oooq | 07:11 | |
*** yolanda has joined #oooq | 07:14 | |
*** jmasud has quit IRC | 07:14 | |
*** sanjayu_ has joined #oooq | 07:17 | |
*** jmasud has joined #oooq | 07:19 | |
*** saneax has quit IRC | 07:19 | |
*** bhagyashris has quit IRC | 07:23 | |
*** amoralej|off is now known as amoralej | 07:23 | |
*** tosky has joined #oooq | 07:30 | |
*** bhagyashris has joined #oooq | 07:31 | |
*** cgoncalves has quit IRC | 07:34 | |
*** cgoncalves has joined #oooq | 07:40 | |
*** jmasud has quit IRC | 07:40 | |
*** dtantsur|afk is now known as dtantsur | 07:40 | |
*** bogdando has joined #oooq | 07:43 | |
*** openstackstatus has quit IRC | 07:46 | |
*** openstackstatus has joined #oooq | 07:46 | |
*** openstack has joined #oooq | 07:47 | |
*** ChanServ sets mode: +o openstack | 07:47 | |
*** bogdando_ has joined #oooq | 07:56 | |
*** bogdando has quit IRC | 07:57 | |
*** bogdando_ is now known as bogdando | 07:57 | |
*** jpena|off is now known as jpena | 07:58 | |
*** dmellado has quit IRC | 07:58 | |
*** ysandeep is now known as ysandeep|lunch | 08:03 | |
*** dmellado has joined #oooq | 08:04 | |
*** dpawlik has quit IRC | 08:09 | |
*** dpawlik has joined #oooq | 08:09 | |
*** ykarel is now known as ykarel|lunch | 08:45 | |
akahat|ruck | marios, sshnaidm, chandankumar o/ | 08:49 |
akahat|ruck | https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens/bb68d90/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 08:49 |
akahat|ruck | seen this issue before? | 08:50 |
marios | akahat|ruck: o/ perhaps a bad promotion or a promotion happened while the job was running? | 08:50 |
marios | akahat|ruck: 2020-05-06 07:52:29 | docker pull failed: Error: image tripleoqueens/centos-binary-rabbitmq:d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708 not found | 08:50 |
marios | akahat|ruck: is likely transient have you seen it more than once ? | 08:50 |
akahat|ruck | marios, no found only in 1 job. | 08:50 |
akahat|ruck | rest of the pipeline is running.. | 08:50 |
sshnaidm | akahat|ruck, hi, check if it was built | 08:59 |
akahat|ruck | sshnaidm, where we can check it? We jsut can not browse to trunk.registry.rdoproject.org/tripleoqueens . | 09:02 |
sshnaidm | akahat|ruck, wait, is it update job? so d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708 is the previous hash actually | 09:02 |
*** jmasud has quit IRC | 09:02 | |
sshnaidm | akahat|ruck, the image is not there, you can try to pull it and it's not found | 09:03 |
akahat|ruck | sshnaidm, no. It's Wednesday pipeline | 09:03 |
sshnaidm | akahat|ruck, firstly need to understand why this job tries to get image with hash d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb470 | 09:05 |
sshnaidm | akahat|ruck, the hash that is trying to promote now is 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91 https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-queens-promote-consistent-to-tripleo-ci-testing/0b12606/job-output.txt | 09:06 |
sshnaidm | akahat|ruck, featureset037-updates-queens is updates job, marios can tell more about it :) | 09:06 |
marios | sshnaidm: akahat|ruck: i checked tripleo-ci-testing was updated yesterday to that d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb470 | 09:07 |
marios | 12:06 < sshnaidm> akahat|ruck, the hash that is trying to promote now is 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91 | 09:07 |
marios | sshnaidm: sorry to that https://trunk.rdoproject.org/centos7-queens/tripleo-ci-testing/delorean.repo | 09:07 |
marios | i.e. 9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91 | 09:07 |
sshnaidm | yep, so 037 job firstly tries to deploy previous hash (d50..) and then to update it to promoting one (9c5..), marios - correct? | 09:08 |
marios | akahat|ruck: try a testproject run for that job and see if it happens again then we can know if its cos of some transient issue (i.e. tripleo-ci-testing update) or somethin gelse | 09:08 |
marios | sshnaidm: yeah sounds right ;) | 09:09 |
marios | sshnaidm: you can confirm it there https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens/bb68d90/logs/quickstart_files/releases.sh | 09:09 |
marios | export OVERCLOUD_DEPLOY_RELEASE="promotion-testing-hash-queens" | 09:10 |
marios | export OVERCLOUD_DEPLOY_HASH="d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708" | 09:10 |
marios | export OVERCLOUD_TARGET_RELEASE="promotion-testing-hash-queens" | 09:10 |
marios | export OVERCLOUD_TARGET_HASH="9c51e1238dced8653eb40c6b6189a30c76d398e5_77cf1e91" | 09:10 |
sshnaidm | yep, and d50... images were built in 30 Apr: https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-queens-containers-build/ef572f2/job-output.txt | 09:10 |
sshnaidm | 06 May - 30 Apr = 1 week | 09:11 |
akahat|ruck | only OVERCLOUD_DEPLOY_HASH is old. | 09:11 |
sshnaidm | akahat|ruck, pruning script of infra deleted them | 09:11 |
sshnaidm | this is my assumption ^ | 09:11 |
sshnaidm | you can check it in #rhos-ops with on-duty folks | 09:12 |
akahat|ruck | sshnaidm, okay. | 09:12 |
sshnaidm | akahat|ruck, it should be old, we deploy previous hash and then deploy an update to current hash, this what job should do | 09:13 |
sshnaidm | akahat|ruck, this is usual with updates/upgrades job | 09:13 |
marios | akahat|ruck: sshnaidm: confirm the hash is not there @ https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleoqueens/imagestreamtags/ | 09:14 |
sshnaidm | akahat|ruck, good way to check ^ | 09:15 |
sshnaidm | marios, but current release for queens is 05f87142c5b81c53e6200c6bf1936df3d5a9e991_bcf3c00b and not d501b27616cec68ce1b35dd5345ebb220e73d3c3_bfcb4708 | 09:18 |
sshnaidm | marios, why does it take d50.. then? | 09:18 |
sshnaidm | oh, I see d501b27616cec68ce1b35dd5345ebb220e73d3c3 is "previous-current-tripleo": https://trunk.rdoproject.org/centos7-queens/previous-current-tripleo/delorean.repo | 09:19 |
*** ysandeep|lunch is now known as ysandeep | 09:20 | |
* dpawlik reading | 09:20 | |
akahat|ruck | sshnaidm, i'm still confused with the baseurl hash and name hash in this repo file ^^ | 09:25 |
*** ykarel|lunch is now known as ykarel | 09:27 | |
akahat|ruck | sshnaidm, sorry.. got it. | 09:29 |
sshnaidm | akahat|ruck, do you understand what happened? | 09:30 |
akahat|ruck | sshnaidm, no not exactly what's happening in the job. | 09:31 |
sshnaidm | akahat|ruck, so 037 is updates job which firstly deploys all with "previous-current-tripleo" hash, then make update to "tripleo-ci-testing" hash | 09:32 |
sshnaidm | akahat|ruck, we check that updates work in this job, so we can update openstack to a newer version | 09:33 |
sshnaidm | for doing this we deploy old version and run updates to a current version | 09:34 |
sshnaidm | akahat|ruck, that's why you see old hash in OVERCLOUD_DEPLOY_RELEASE | 09:34 |
sshnaidm | a newer hash is in OVERCLOUD_TARGET_HASH | 09:35 |
akahat|ruck | sshnaidm, okay. Got it. | 09:35 |
sshnaidm | akahat|ruck, but we have a pruning script in registry that deletes week old images | 09:35 |
sshnaidm | akahat|ruck, because "previous-current-tripleo" wasn't whitelisted there, script just deleted all containers with this hash | 09:36 |
akahat|ruck | oh.. that's bad... | 09:36 |
sshnaidm | akahat|ruck, so now dpawlik added this type of hashes to whitelist and we hopefully won't hit it again :) | 09:36 |
akahat|ruck | sshnaidm, that's great :) | 09:38 |
akahat|ruck | sshnaidm, marios thank you :) Now it's more clear. | 09:38 |
*** owalsh has quit IRC | 09:46 | |
akahat|ruck | sshnaidm, there is one more issue with the upgrades. https://bugs.launchpad.net/tripleo/+bug/1877031 | 09:55 |
openstack | Launchpad bug 1877031 in tripleo "queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version" [High,Triaged] | 09:55 |
akahat|ruck | it also because of the same reason? | 09:55 |
*** owalsh has joined #oooq | 10:09 | |
sshnaidm | akahat|ruck, nope, no containers error there | 10:24 |
sshnaidm | akahat|ruck, arxcruz is there a bug about tempest failures in ovb stable branches? | 10:32 |
arxcruz | sshnaidm: https://bugs.launchpad.net/tripleo/+bug/1876972 ? | 10:32 |
openstack | Launchpad bug 1876972 in tripleo "tempest failing across RDO sf stein jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 10:32 |
sshnaidm | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch | 10:32 |
sshnaidm | it's also rocky and queens ^ | 10:32 |
arxcruz | sshnaidm: yeah | 10:33 |
arxcruz | sshnaidm: https://trello.com/c/niYGK63i/1475-cixlp1876087tripleociproa-queens-tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-timeout | 10:33 |
arxcruz | just wrote this | 10:33 |
sshnaidm | arxcruz, cool | 10:33 |
chandankumar | sshnaidm, marios please have a look at c8 train container build job https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/build.log | 10:33 |
chandankumar | containers are getting builts but it is not showing , and the job is passing | 10:33 |
arxcruz | sshnaidm: the main difference between upstream and the failing one in rdo, is the kernel version that is slightly different, and the failing one is running on AMD, while the passing one is running on Intel | 10:34 |
sshnaidm | Built...................... 0 SUCCESS: Congratulations! All containers have been built :) | 10:34 |
sshnaidm | chandankumar, neat ^ | 10:34 |
sshnaidm | arxcruz, I see, wonder why it pass on train | 10:35 |
marios | chandankumar: middle sthing right now will check in bit | 10:35 |
arxcruz | sshnaidm: did it pass? lol, it's so many failures that i didn't "catch the train" yet | 10:35 |
arxcruz | lol | 10:36 |
sshnaidm | arxcruz, all green https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch# | 10:36 |
sshnaidm | maybe something wasn't backported.. | 10:37 |
chandankumar | sshnaidm, if you check this file https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/containers-consolidated-builds.log containers are actually built | 10:40 |
marios | chandankumar: indeed .. you can see logs for the builds at each container e.g. https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/logs/buildah-builds/kolla-builds/1da3495d-007a-4b08-827c-1b545d9a24ea/docker/aodh/aodh-base/aodh-base-build.log | 10:44 |
marios | chandankumar: are containers pushed? | 10:45 |
chandankumar | marios, nope, [Retrieve built images] is empty, these is no stdout, as it prints in master job | 10:46 |
marios | chandankumar: wonder if it is related to https://review.opendev.org/#/c/718433/ merged 9 hours ago | 10:46 |
*** derekh has joined #oooq | 10:46 | |
chandankumar | marios, let me check other builds | 10:47 |
marios | chandankumar: hmm push false? https://logserver.rdoproject.org/20/26920/15/check/periodic-tripleo-centos-8-train-containers-build-push/fa6e2bc/conf/kolla-build.conf | 10:48 |
marios | chandankumar: no the push/false is unrelated i think | 10:49 |
chandankumar | marios, https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/tripleoci.yaml#l392 | 10:49 |
chandankumar | it is set to true so unrelated | 10:49 |
marios | chandankumar: but master is same https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/660759a/logs/build.log | 10:50 |
chandankumar | marios, but master errors are legit | 10:50 |
chandankumar | marios, from success job https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/2d6fb0c/job-output.txt 2020-05-06 00:43:38.221773 | TASK [Retrieve built images] | 10:51 |
chandankumar | 2020-05-06 00:43:38.814114 | primary | trunk.registry.rdoproject.org/tripleomaster/centos-binary-sahara-api:93c75d0c65654d0ce6ba51da601f9e9b | 10:51 |
*** sanjayu__ has joined #oooq | 11:03 | |
ykarel | chandankumar, need to handle spacial namespace, tripleotraincentos8, buildcontainers role not handling that currently | 11:03 |
ykarel | it assumes tripleo{{ release }} currently | 11:04 |
*** sanjayu_ has quit IRC | 11:05 | |
chandankumar | ykarel, but if you check consolidated build log the namespace is used correctly | 11:11 |
ykarel | chandankumar, building is fine, but the way it detects built containers, is missing handling of special namespace | 11:14 |
ykarel | see https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/build.sh.j2#L56 | 11:15 |
ykarel | sudo {{ container_cli }} images | grep tripleo{% if osp_branch is defined %}{{ osp_branch }}{% else %}{{ ci_branch }}{% endif %}/{{ kolla_base }}-binary > containers-successfully-built.log | 11:15 |
ykarel | it search tripleotrain/centos | 11:16 |
*** ysandeep is now known as ysandeep|brb | 11:16 | |
chandankumar | ykarel, ah, let me fix that | 11:18 |
ykarel | ack | 11:20 |
*** rfolco has joined #oooq | 11:30 | |
chandankumar | ykarel, https://review.opendev.org/#/c/725831/ | 11:33 |
ykarel | chandankumar, ack | 11:35 |
*** jpena is now known as jpena|lunch | 11:39 | |
chandankumar | marios, sshnaidm it will fix c8 train container issue https://review.opendev.org/#/c/725831/ | 11:39 |
*** ysandeep|brb is now known as ysandeep | 11:54 | |
*** rlandy has joined #oooq | 11:58 | |
weshay|ruck | akahat|ruck, 0/ | 11:58 |
weshay|ruck | akahat|ruck, you know what to do and when you can drop from the program call? | 11:59 |
chandankumar | weshay|ruck, I will be there with him | 12:00 |
rfolco | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 12:02 |
weshay|ruck | chandankumar, you can join us if you like.. Amol's GOT THIS... :) | 12:02 |
rfolco | ping retro | 12:02 |
rfolco | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 12:02 |
rfolco | ping retro | 12:02 |
rfolco | https://meet.google.com/bop-gsjm-nhp | 12:02 |
chandankumar | weshay|ruck, ok, joining retro | 12:02 |
rfolco | https://trello.com/b/0VFswmht/pcci-retrospective?menu=filter&filter=label:UniSprint26 | 12:02 |
weshay|ruck | akahat|ruck, phil and mike burns can help.. if you need it | 12:03 |
*** amoralej is now known as amoralej|lunch | 12:03 | |
*** holser_ has joined #oooq | 12:10 | |
akahat|ruck | weshay|ruck, yeah sure. Thanks :) | 12:18 |
*** udesale_ has joined #oooq | 12:23 | |
*** udesale has quit IRC | 12:26 | |
*** ykarel is now known as ykarel|afk | 12:38 | |
*** Goneri has joined #oooq | 12:38 | |
*** jpena|lunch is now known as jpena | 12:45 | |
*** sanjayu_ has joined #oooq | 12:47 | |
*** sanjayu__ has quit IRC | 12:50 | |
marios | rfolco: 1 min left ... picking ruck|rover | 12:59 |
marios | weshay|ruck: ^ | 12:59 |
rfolco | thx marios | 12:59 |
*** chem_ is now known as chem | 13:01 | |
rfolco | bhagyashris, sorry, you'll be fine :) | 13:02 |
bhagyashris | rfolco, np | 13:02 |
*** ratailor has quit IRC | 13:02 | |
chandankumar | weshay|ruck, you have few mins for train c8? | 13:04 |
rfolco | bhagyashris, you want to be ruck or rover first ? | 13:05 |
weshay|ruck | chandankumar, locked in a 2hr meeting.. | 13:07 |
weshay|ruck | chandankumar, we got past get_hash afaict | 13:07 |
bhagyashris | rfolco, i will first go with ruck | 13:07 |
weshay|ruck | but deployment failed | 13:07 |
chandankumar | weshay|ruck, yes, new issue | 13:08 |
rfolco | bhagyashris, clever | 13:08 |
rfolco | :) | 13:08 |
chandankumar | fs01 and 10 failed right now | 13:08 |
chandankumar | weshay|ruck, I think I need to https://review.opendev.org/#/c/712289/ backport it for train for sc1 | 13:08 |
weshay|ruck | chandankumar, let's focus on merging hte deps.. because those jobs reset several times | 13:08 |
weshay|ruck | w/ rebases etc | 13:08 |
weshay|ruck | chandankumar, I can do irc now | 13:08 |
rfolco | weshay|ruck, akahat|ruck let us know when we switch nicks, call me and bhagyashris for the handoff/sync | 13:08 |
chandankumar | weshay|ruck, https://review.opendev.org/#/c/725831/ needed for container build | 13:08 |
weshay|ruck | chandankumar, that's a good change | 13:09 |
chandankumar | weshay|ruck, fs010 I think there is a patch already | 13:10 |
chandankumar | let me find that | 13:10 |
akahat|ruck | rfolco, it's depend on weshay|ruck schedule. about nick.. whenever you want. | 13:12 |
akahat|ruck | bhagyashris, ^^ | 13:12 |
weshay|ruck | akahat|ruck, and bhagyashris's day is ending shortly.. so you folks can unnick | 13:13 |
weshay|ruck | rfolco, you should renick to rover | 13:13 |
*** rfolco is now known as rfolco|rover | 13:13 | |
*** akahat|ruck is now known as akahat | 13:13 | |
rfolco|rover | akahat, how is the feeling ? | 13:13 |
chandankumar | weshay|ruck, and https://review.opendev.org/#/c/725852/1 and https://review.opendev.org/#/c/722284/3 | 13:13 |
rfolco|rover | :) | 13:13 |
chandankumar | to clear sc 1 and 10 for c8 train | 13:14 |
chandankumar | weshay|ruck, updating the reviews | 13:14 |
akahat | rfolco|rover, good. Learn lot's of new things... random discussion.. prodchain flow.. etc. :) | 13:14 |
akahat | Exciting.. | 13:14 |
rlandy | marios: weshay|ruck: we got the deps we were missing in trunk-candidate. Building and pushing containers again | 13:16 |
*** pojadhav is now known as pojadhav|afk | 13:16 | |
akahat | rlandy, pojadhav bhagyashris o/ | 13:16 |
chandankumar | weshay|ruck, https://review.rdoproject.org/r/#/c/26920/ tracks all the patches for c8 train | 13:16 |
weshay|ruck | rlandy, nice | 13:16 |
rlandy | akahat: hey | 13:16 |
akahat | rlandy, pojadhav|afk bhagyashris this is came up today. https://bugs.launchpad.net/tripleo/+bug/1877031 | 13:16 |
openstack | Launchpad bug 1877031 in tripleo "queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version" [High,Triaged] | 13:16 |
akahat | rlandy, are you the right person of contact? | 13:16 |
rlandy | akahat: I think you are looking for the new ruck/rovers? | 13:17 |
rlandy | rfolco|rover: ^^ | 13:17 |
rlandy | akahat: rfolco|rover will be taking over from you | 13:17 |
rfolco|rover | akahat, hit me | 13:17 |
akahat | rlandy, okay. Thanks :) | 13:17 |
akahat | rfolco|rover, this is failing because of the lower version of ansible. | 13:18 |
rlandy | marios: want to meet today - or ok if I carry on with standalone? | 13:18 |
rlandy | also ... https://code.engineering.redhat.com/gerrit/#/c/198838/ can merge | 13:18 |
rfolco|rover | akahat, assign to me and I'll take a look | 13:18 |
*** skramaja has quit IRC | 13:18 | |
akahat | rfolco|rover, okay | 13:18 |
*** bhagyashris is now known as bhagyashris|ruck | 13:18 | |
*** cgoncalves has quit IRC | 13:19 | |
marios | rlandy: yes lets meet | 13:23 |
marios | rlandy: give me couple mins (got the get_hash stuff ok but i am hitting the release file issue now) | 13:23 |
chandankumar | marios, does my changes to get_hash broke your stuff? | 13:23 |
*** cgoncalves has joined #oooq | 13:25 | |
rlandy | marios: cool - ping me when ready - the release file is what I wanted to discuss | 13:25 |
marios | chandankumar: no no it was more like people working on things at the same time and it changes... e.g. you look at log files but the code is different cos it was changed e.g. last night and you didn't notice, so you get mismatch | 13:26 |
marios | chandankumar: don't worry i about it i am just feeling low | 13:26 |
marios | rlandy: now ok for you? | 13:27 |
rlandy | marios; yep - 3 mins? | 13:28 |
marios | rlandy: ack | 13:28 |
*** TrevorV has joined #oooq | 13:31 | |
rlandy | marios: joining community call | 13:32 |
marios | rlandy: ack | 13:32 |
weshay|ruck | gate failed on https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/ | 13:36 |
weshay|ruck | chandankumar, need a favor | 13:36 |
weshay|ruck | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ac/717850/24/gate/openstack-tox-molecule/3aca407/job-output.txt | 13:36 |
chandankumar | weshay|ruck, where is the image? | 13:36 |
weshay|ruck | let's change those molecule tests.. to test for current-tripleo vs. tripleo-ci-testing | 13:36 |
weshay|ruck | chandankumar, missing | 13:36 |
chandankumar | weshay|ruck, there is an option to disable that check | 13:37 |
chandankumar | weshay|ruck, https://review.opendev.org/#/c/717850/24/config/release/tripleo-ci/CentOS-8/train.yml@12 | 13:38 |
chandankumar | weshay|ruck, validate_on: rdo | 13:38 |
chandankumar | let me update the patch | 13:38 |
chandankumar | the option is there in both files why it is failing then | 13:40 |
*** amoralej|lunch is now known as amoralej | 13:44 | |
weshay|ruck | rfolco|rover, can you prep a new hackmd for the sprint | 13:45 |
rfolco|rover | weshay|ruck, planning ? okay | 13:45 |
chandankumar | weshay|ruck, there is an issue with c8 master image build https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/30ab58a/build.log | 13:45 |
weshay|ruck | ruck rover | 13:45 |
chandankumar | Failed to download metadata for repo 'centos7-rt' | 13:45 |
weshay|ruck | chandankumar, aye.. and tripleo-ci-testing should not block the upstream gate imho | 13:46 |
chandankumar | weshay|ruck, in c8 from where c7 repo is coming | 13:46 |
weshay|ruck | repo? | 13:48 |
chandankumar | weshay|ruck, https://review.opendev.org/725858 | 13:50 |
chandankumar | it will unblock molecule tests | 13:50 |
weshay|ruck | chandankumar, comment | 13:52 |
*** ykarel|afk is now known as ykarel | 13:54 | |
chandankumar | weshay|ruck, commented and updated | 13:59 |
chandankumar | weshay|ruck, please donot recheck train patches, as rdo review c8 train jobs running | 14:00 |
*** ysandeep is now known as ysandeep|away | 14:02 | |
*** bogdando_ has joined #oooq | 14:04 | |
rfolco|rover | weshay|ruck, taiga board and meeting notes created | 14:05 |
rfolco|rover | brb | 14:06 |
*** bogdando has quit IRC | 14:06 | |
*** bogdando_ is now known as bogdando | 14:06 | |
weshay|ruck | rfolco|rover, rdo mtg | 14:14 |
rfolco|rover | weshay|ruck, ack | 14:14 |
weshay|ruck | chandankumar, interesting.. that skips the curl check completely https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c2e/725858/2/check/openstack-tox-molecule/c2e16a9/job-output.txt | 14:16 |
weshay|ruck | I don't understand that test I guess | 14:17 |
chandankumar | weshay|ruck, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/tests/validate-release-config.yml#L9 | 14:18 |
weshay|ruck | ya.. I see that.. don't get it | 14:19 |
weshay|ruck | it's defined | 14:19 |
weshay|ruck | what does "rdo" mean | 14:19 |
weshay|ruck | does it mean.. only check from 3rd party? | 14:19 |
chandankumar | weshay|ruck, it was initially added just for rdo urls | 14:21 |
chandankumar | by zbr | 14:21 |
weshay|ruck | what's a rdo url? | 14:22 |
chandankumar | weshay|ruck, i mean for images.rdorproject.org | 14:23 |
* weshay|ruck is confused.. I'll raise a bug and assign it to zbr | 14:23 | |
chandankumar | weshay|ruck, ok | 14:23 |
weshay|ruck | I think promote_source could be set to current-tripleo for established releases | 14:24 |
weshay|ruck | a new release w/o a prmotion to current-tripleo.. tripleo-ci-testing makes sense | 14:24 |
rlandy | chandankumar: hey is https://code.engineering.redhat.com/gerrit/#/c/196373/ ready to go? | 14:26 |
rlandy | chandankumar: we're almost ready to merge the mini-pipeline which includes those jobs | 14:27 |
chandankumar | rlandy, yes, good to go | 14:30 |
*** sanjayu__ has joined #oooq | 14:30 | |
chandankumar | rlandy, do we want to run build images jobs in check pipeline? | 14:32 |
weshay|ruck | rfolco|rover, can you please join the cix call.. I'm caught in a perf review mtg | 14:32 |
chandankumar | weshay|ruck, you mean this part https://review.opendev.org/#/c/717850/24/config/release/tripleo-ci/CentOS-8/train.yml@11 ? | 14:32 |
weshay|ruck | rfolco|rover, just do your best | 14:32 |
*** sanjayu_ has quit IRC | 14:33 | |
rfolco|rover | weshay|ruck, joined | 14:35 |
weshay|ruck | rfolco|rover++ | 14:35 |
*** udesale_ has quit IRC | 14:38 | |
*** ccamacho has quit IRC | 14:39 | |
rlandy | chandankumar: we'll do that after we get integration going | 14:40 |
chandankumar | rlandy, aye thanks :-) | 14:40 |
rlandy | marios: ^^ pls vote on https://code.engineering.redhat.com/gerrit/#/c/196373 so we can merge | 14:40 |
rlandy | chandankumar: marios: weshay|ruck: anyone else .. pls review TLS reviews ... would love to get those in ... | 14:48 |
rlandy | https://review.opendev.org/#/c/707419 | 14:48 |
rlandy | https://review.opendev.org/#/c/724907 | 14:48 |
rlandy | https://review.opendev.org/#/c/720757/ | 14:48 |
marios | rlandy: ack will do | 14:48 |
rlandy | https://review.opendev.org/#/c/706288/ | 14:48 |
rlandy | chandankumar: marios: weshay|ruck: thanks ^^ 4 reviews above ... results in test review https://review.opendev.org/#/c/724850 | 14:49 |
marios | rlandy: ack but not right now will check them more carefully tomorrow concentrating on the release stuff now | 14:51 |
marios | rlandy: so is anything using tripleo-environments/rhos-17 release file currently? git blame says you added that in december | 14:57 |
marios | rlandy: wondering if i should re-use it or make new one | 14:57 |
* marios starts editing in place | 14:57 | |
rlandy | marios: per meeting .. pls move that fil eto rhos-17-compose | 14:57 |
rlandy | and add two new ones | 14:57 |
marios | rlandy: ack | 14:57 |
rlandy | for check and promotion | 14:58 |
marios | rlandy: but does something use it? | 14:58 |
marios | rlandy: so i'll have to edit whatever uses it to instead take rhos-17 compose | 14:58 |
* marios grep | 14:58 | |
rlandy | marios: there is one test that uses it | 14:58 |
rlandy | but let's delete it | 14:58 |
rlandy | there has not been a compose yet | 14:58 |
marios | rlandy: in which case why not just use that one? | 14:58 |
marios | rlandy: and keep the upstream pattern.. rhos-17 for check and then the testing-hash one for promotions | 14:59 |
rlandy | marios: becuase at somepoint we may want to compare | 14:59 |
marios | rlandy: k k going with new files | 14:59 |
rlandy | ie: why compose works | 14:59 |
rlandy | or not | 14:59 |
rlandy | chandankumar: https://code.engineering.redhat.com/gerrit/#/c/196373/ failed the two tests | 15:04 |
rlandy | should they pass now? | 15:04 |
rlandy | if not, can we make them non-voting so we can merge the code | 15:04 |
chandankumar | rlandy, legit failure, fixed now it will pass | 15:08 |
rlandy | chandankumar++ | 15:09 |
marios | rlandy: so actually all the component rhel8/17 jobs are using that rhos-17 ? | 15:14 |
marios | rlandy: ah no as 'osp_release' not release-file sorry | 15:15 |
rlandy | marios: we need to rework those jobs | 15:15 |
rlandy | they are not running now | 15:15 |
rlandy | from old pipeline | 15:15 |
rlandy | don't worry about it | 15:15 |
marios | rlandy: well i think we need to do _something_ for our sanity/future generations... like documenting a lot of this stuff cos its a completely different world tooling-wise | 15:15 |
rlandy | marios: sure | 15:15 |
marios | rlandy: but of course that is secondary :) /me shutsup carry one | 15:15 |
marios | carry on even | 15:16 |
rlandy | marios: we're trying to make it look more like upstream | 15:16 |
rlandy | so to your point | 15:16 |
marios | rlandy: like e.g. this get_build_command business... | 15:16 |
marios | rlandy: yeah 18:16 < rlandy> marios: we're trying to make it look more like upstream | 15:16 |
rlandy | we could move the files we want to use to RedHat8 | 15:16 |
marios | rlandy: but the problem is there are these cases that we mix the two worlds | 15:16 |
rlandy | like we have in upstrea, | 15:16 |
rlandy | get_build will go away | 15:17 |
rlandy | we needed thta for compose | 15:17 |
rlandy | that | 15:17 |
*** ccamacho has joined #oooq | 15:17 | |
*** ykarel is now known as ykarel|away | 15:23 | |
rlandy | marios: you can just use https://code.engineering.redhat.com/gerrit/#/c/196373/38/playbooks/pre-rhos.yaml | 15:25 |
rlandy | as is chandan's review | 15:26 |
rlandy | except don't hard code tripleo-ci-testing | 15:26 |
marios | rlandy: i was going to add into the release file with includepkgs as we discussed for delorean/deps ie like upstream? | 15:27 |
rlandy | sure | 15:27 |
marios | rlandy: ack well adding another pre might be easier - we can see what the release file looks like and revisit on the review? | 15:28 |
rlandy | ok | 15:28 |
rlandy | marios: ugh - we're missing one more dep now - we didn't build nova - logging BZ | 15:30 |
rlandy | package openstack-nova-common-1:21.1.0-0.20200505172042.dddc0dc.el8ost.noarch requires python3-nova = 1:21.1.0-0.20200505172042.dddc0dc.el8ost, but none of the providers can be installed] | 15:30 |
marios | rlandy: ack thanks | 15:38 |
marios | rlandy: we kinda expected this :/ i mean dep after dep but the cadence is kinda worrying here... if only we had a job that would prevent us from doing that :D | 15:39 |
rlandy | marios: coming soon to a pipeline near you | 15:39 |
*** jmasud has joined #oooq | 15:41 | |
rlandy | marios: before you go, please remind me how to create a CIX for https://bugzilla.redhat.com/show_bug.cgi?id=1832378 | 15:44 |
openstack | bugzilla.redhat.com bug 1832378 in distribution "Cannot build nova-base and mistral-executor containers for rhel8-osp17 - missing python3-nova" [Unspecified,New] - Assigned to lhh | 15:44 |
marios | rlandy: search "[CIX]" emails in rhos-dev... you need to send one of those and then lon has tooling that picks it up | 15:45 |
marios | rlandy: not sure we want the includepackages stuff cos it should come from delorean we won't have delorean-current for this job i think ? | 15:48 |
rlandy | yep - i think so | 15:49 |
*** ccamacho has quit IRC | 15:51 | |
*** jmasud has quit IRC | 15:55 | |
rfolco|rover | bhagyashris|ruck, weshay|ruck rr notes created https://hackmd.io/2MdkNAUuT7aBcM0Yck4xnw?both | 15:55 |
rlandy | chandankumar: you have a post-failure https://sf.hosted.upshift.rdu2.redhat.com/logs/73/196373/38/check/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/7302520/job-output.txt | 16:03 |
*** panda|pto has quit IRC | 16:03 | |
rlandy | should be a minor fix | 16:03 |
*** jmasud has joined #oooq | 16:04 | |
*** panda has joined #oooq | 16:06 | |
*** dtantsur is now known as dtantsur|afk | 16:11 | |
marios | rlandy: k it will be wildly wrong but its a start @ lucky v13 @ https://code.engineering.redhat.com/gerrit/#/c/198789/ and environment-file in depends-on | 16:13 |
marios | rlandy: i'll have to pick it up tomorrow again | 16:13 |
rlandy | marios: cool - will carry on from @13 | 16:13 |
rlandy | marios: hoping to merge the image build jobs as well | 16:14 |
rlandy | as then pipeline can merge | 16:14 |
marios | rlandy: yeah i'll get to those if they aren't merged by morning | 16:15 |
rlandy | marios: also go a fix for the dep failures | 16:15 |
rlandy | waiting for an updated candidate repo | 16:15 |
marios | rlandy: that was quick :) | 16:15 |
rlandy | marios: sent link - the latest build was 6am | 16:16 |
rlandy | fix came in at 10ish | 16:16 |
*** jmasud has quit IRC | 16:21 | |
*** jmasud has joined #oooq | 16:23 | |
*** jmasud has quit IRC | 16:23 | |
*** marios is now known as marios|out | 16:27 | |
rlandy | chandankumar: thanks - merging https://code.engineering.redhat.com/gerrit/#/c/196373/ | 16:28 |
chandankumar | rlandy, aye thanks! | 16:28 |
*** sanjayu__ has quit IRC | 16:38 | |
chandankumar | rlandy, weshay|ruck rfolco|rover I think we can delete this card https://tree.taiga.io/project/tripleo-ci-board/task/1626?kanban-status=1447274 and will cover as a part of integration pipeline? | 16:42 |
weshay|ruck | chandankumar, sure | 16:42 |
*** bogdando has quit IRC | 16:43 | |
*** marios|out has quit IRC | 17:01 | |
*** jpena is now known as jpena|off | 17:08 | |
*** holser_ has quit IRC | 17:09 | |
*** holser has joined #oooq | 17:09 | |
*** holser_ has joined #oooq | 17:13 | |
*** holser has quit IRC | 17:14 | |
*** pojadhav|afk is now known as pojadhav | 17:15 | |
*** amoralej is now known as amoralej|off | 17:15 | |
bhagyashris|ruck | rfolco|rover, ack | 17:20 |
*** irclogbot_0 has quit IRC | 17:20 | |
*** irclogbot_3 has joined #oooq | 17:22 | |
*** derekh has quit IRC | 17:26 | |
*** sshnaidm is now known as sshnaidm|afk | 17:26 | |
*** hamzy_ has quit IRC | 17:52 | |
*** hamzy has joined #oooq | 17:54 | |
weshay|ruck | rlandy, interestingly enough.. I think the issue ur hitting w/ nova in component builds | 17:55 |
weshay|ruck | is also failing upstream components | 17:56 |
weshay|ruck | - package openstack-nova-common-1:21.1.0-0.20200505172042.dddc0dc.el8ost.noarch requires python3-nova = 1:21.1.0-0.20200505172042.dddc0dc.el8ost, but none of the providers can be installed | 17:56 |
weshay|ruck | http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&fullscreen&panelId=423 | 17:56 |
rlandy | weshay|ruck: ack | 17:57 |
weshay|ruck | cannot install the best update candidate for package openstack-nova-common-1:21.1.0-0.20200502034918.fdcb1e3.el8.noarch\n - nothing provides python3-oslo-policy >= 3.1.0 | 17:57 |
rlandy | was fixed in compose | 17:57 |
weshay|ruck | https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/e1190f4/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 17:57 |
weshay|ruck | rlandy, ah k | 17:57 |
rlandy | waiting for updated rmp to hit the candidate repo | 17:57 |
rlandy | weshay|ruck: https://review.opendev.org/#/c/724907/ - can you w+ that? | 18:01 |
rfolco|rover | is review.rdoproject.org ok for you weshay|ruck | 18:08 |
rfolco|rover | actually... zuul is down | 18:09 |
rlandy | chandankumar: where do downstream images get pushed? | 18:11 |
rlandy | overcloud_image_url? | 18:11 |
rlandy | ipa_image_url? | 18:11 |
chandankumar | rlandy, we have not got any places to push images to image server | 18:15 |
chandankumar | rlandy, apevec is working on that | 18:16 |
rlandy | chandankumar: ok - will update that setting later | 18:16 |
chandankumar | rlandy, we can use to_build flag with in ovb job to rebuild the image and use it | 18:16 |
chandankumar | rlandy, weshay|ruck https://review.rdoproject.org/r/27289 and https://review.rdoproject.org/r/27307 final reviews for fs02 | 18:25 |
*** hamzy_ has joined #oooq | 18:39 | |
*** hamzy has quit IRC | 18:42 | |
*** jmasud has joined #oooq | 19:04 | |
*** jmasud has quit IRC | 19:18 | |
*** yolanda has quit IRC | 19:33 | |
*** jfrancoa has quit IRC | 19:59 | |
weshay|ruck | rfolco|rover any updates from docker.io? | 20:09 |
rfolco|rover | ignored | 20:11 |
rfolco|rover | weshay|ruck, we'll have to find a workaround.... or create the entire alphatbet and change the code to pull from triplea, tripleob, tripleoc... | 20:12 |
weshay|ruck | can you cc me | 20:12 |
rfolco|rover | weshay|ruck, its a web form | 20:12 |
rfolco|rover | thats why you can't cc | 20:12 |
rfolco|rover | let me get the link | 20:13 |
weshay|ruck | k | 20:13 |
rfolco|rover | https://hub.docker.com/support/contact/ | 20:13 |
rfolco|rover | weshay|ruck, do you want to try ? | 20:13 |
rfolco|rover | weshay|ruck, did you change rr notes I created ? https://hackmd.io/2MdkNAUuT7aBcM0Yck4xnw | 20:27 |
weshay|ruck | yes | 20:28 |
rfolco|rover | weshay|ruck, ok so that was intentional. ok | 20:29 |
weshay|ruck | rfolco|rover, I liked some of your changes.. but keeping the layout consistent is also important.. so I integrated a bit and changed a bit | 20:30 |
*** Goneri has quit IRC | 21:05 | |
rlandy | weshay|ruck: ping re: director images | 21:18 |
weshay|ruck | rlandy, 0/ | 21:20 |
weshay|ruck | rfolco|rover, I don't think https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml#L17 | 21:21 |
weshay|ruck | is going to be a big deal | 21:21 |
rlandy | weshay|ruck: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-17.yml includes a bunch of lines about director images | 21:21 |
weshay|ruck | I think the release is defined in the dlrn-promoter.sh | 21:21 |
rlandy | I assume we don;t have those with the component setup | 21:21 |
weshay|ruck | rlandy, ya.. we don't want to use those | 21:22 |
rlandy | we just download them from the server that will one day be set up | 21:22 |
rlandy | k - deleting | 21:22 |
weshay|ruck | so we can change it more of the upstream method | 21:22 |
weshay|ruck | :) | 21:22 |
*** jmasud has joined #oooq | 21:28 | |
rlandy | weshay|ruck: any idea why if delorean.repo and delorean-current.repo are installed in https://sf.hosted.upshift.rdu2.redhat.com/logs/89/198789/17/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/969a4de/logs/undercloud/home/zuul/repo_setup.log | 21:54 |
rlandy | that they don;t show up in: | 21:54 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/89/198789/17/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17/969a4de/logs/undercloud/etc/yum.repos.d/ | 21:55 |
rlandy | 2020-05-06 17:35:41 | Error: Unable to find a match: python3-tripleoclient ceph-ansible | 21:56 |
* weshay|ruck looks at release file | 21:56 | |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/199471/5/config/release/rhos-17-periodic.yml | 21:58 |
weshay|ruck | rlandy, could be rhos-release -x | 22:00 |
rlandy | weshay|ruck: ok - let me do that in a before command | 22:01 |
rfolco|rover | weshay|ruck, okay, but its not (only) about pushing... but also pulling from... | 22:09 |
rlandy | weshay|ruck: thanks - that worked to keep the delorean repos ... still missing ceph-ansible | 22:17 |
rlandy | maybe it's in build deps | 22:17 |
*** jmasud has quit IRC | 22:17 | |
*** jmasud has joined #oooq | 22:18 | |
weshay|ruck | ceph-ansible ah crud.. | 22:18 |
weshay|ruck | I don't think we add that right.. | 22:18 |
weshay|ruck | meaning not in dlrn | 22:18 |
weshay|ruck | rlandy, 2020-05-05 20:15:35 | ceph-ansible noarch 4.0.14-1.el8cp rhelosp-ceph-4-tools 196 k | 22:20 |
rlandy | weshay|ruck: yep - those repos used to come with rhos-release -p 17 | 22:21 |
weshay|ruck | so /me wonders if rhos-release supports just ceph ansible | 22:21 |
rlandy | now I need to install them manually | 22:21 |
weshay|ruck | rlandy, or run it.. and disable repos except for base and -ceph-r | 22:21 |
weshay|ruck | rlandy, or run it.. and disable repos except for base and -ceph-4 | 22:21 |
weshay|ruck | rlandy, let's do this | 22:22 |
weshay|ruck | let's update rhos-release to just support base + ceph | 22:22 |
rlandy | how? | 22:22 |
weshay|ruck | rlandy, http://git.app.eng.bos.redhat.com/git/rhos-release.git/tree/rhos-release | 22:23 |
rlandy | weshay|ruck: sorry - still not seeing it :( | 22:24 |
rlandy | Enable testing (pending) repositories. Comma separated. | 22:25 |
weshay|ruck | https://meet.google.com/det-vzdp-bpp | 22:25 |
weshay|ruck | rm -f /etc/yum.repos.d/rhos-release-17-trunk.repo | 22:31 |
*** jmasud has quit IRC | 22:31 | |
*** jmasud has joined #oooq | 22:32 | |
rlandy | weshay|ruck: https://review.opendev.org/#/c/724907/ | 22:38 |
*** jmasud has quit IRC | 22:46 | |
*** TrevorV has quit IRC | 22:54 | |
*** tosky has quit IRC | 22:56 | |
*** jmasud has joined #oooq | 23:03 | |
*** holser_ has quit IRC | 23:09 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!