Monday, 2022-01-31

*** ysandeep|out is now known as ysandeep05:58
*** marios is now known as marios|ruck06:02
*** ysandeep is now known as ysandeep|brb06:42
*** amoralej|off is now known as amoralej07:02
*** ysandeep|brb is now known as ysandeep07:22
Tenguhello there - can anyone push this in the gate? https://review.opendev.org/c/openstack/tripleo-ci/+/82543707:56
Tenguthat would allow to re-use existing container or env when we have a scheduled build..07:57
*** jpena|off is now known as jpena08:38
ysandeepTengu, looking ^^ 09:27
ysandeepokay, already +wed09:28
Tenguysandeep: yep :). but thanks for looking09:30
amoralejare you seeing issues with opstools repo?09:31
ysandeep^^ may be marios|ruck noticed anything? 09:34
marios|ruckysandeep: there are a few issues ongoing but haven't come across opstools issue yet09:41
marios|ruckamoralej: we have issue possibly with appstream there https://bugs.launchpad.net/tripleo/+bug/1959524 09:41
amoralejyeah, that's different09:42
ysandeepmarios|ruck: ack, thank you!09:42
amoralejyou may start getting issues with opstools09:42
amoralejbackground https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/3867309:42
marios|ruckamoralej: what issue? we have one for pacemaker-cluster-libs not sure if that is opstools09:42
amoraleji'm setting manually09:42
amoralejin delorean-deps.repo09:42
amoralejbut you may hit it for repos configured via oooq config09:43
marios|ruckamoralej: noting 09:43
ysandeephttps://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master/a3dff07/job-output.txt09:47
ysandeep2022-01-31 09:17:06.521213 | primary |   "msg": "Failed to download metadata for repo 'RDO-CentOS8-deps': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried",09:47
ysandeepI noticed some mirror issues as well09:47
ysandeepmarios|ruck, amoralej ^^ 09:48
marios|ruckysandeep: ack 09:48
ysandeeplooks like started recently, happening with lot of running jobs now - https://review.rdoproject.org/zuul/status#openstack-component-glance 09:48
amoralejysandeep, i'm not sure, tbh09:53
amoralejcould you look where that repo is created?09:55
amoralejhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/templates/zuul-multi-node-bridge-ovs.repo.j209:56
amoralejaaahhh09:56
amoralejalso related09:56
amoralejsending the patch09:56
bhagyashrisfolks please add in your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3866709:57
bhagyashrisfrenzyfriday, will need your help here https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3866709:57
bhagyashrisvictoria and ussuri promotion is blocked 09:57
ysandeepamoralej, sry missed your ping, https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/templates/zuul-multi-node-bridge-ovs.repo.j2#L17 - need to move to cs8 stream?10:01
marios|ruckamoralej: any ideas about that one https://bugs.launchpad.net/tripleo/+bug/1959526/comments/2 appstream repo 10:01
marios|ruckamoralej: but the package is there 10:01
marios|ruckso maybe mirror issue not sure10:01
*** ysandeep is now known as ysandeep|coffee10:04
amoralejmarios|ruck, we are still landing patches in stable/train, right?10:06
marios|ruckamoralej: yes10:06
amoralejysandeep|coffee, yes, i'm sengind a patch10:06
marios|ruckamoralej: so then we'll need a patch for quickstart too? 10:07
amoralejmarios|ruck, quickstart install repos from config or pull delorean-deps.repo?10:07
marios|ruckamoralej: https://opendev.org/openstack/tripleo-quickstart/src/commit/358d8da7877a5cb408bbf79c80ddd42edf9ef553/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml#L175 10:07
marios|ruckwhere      export NODEPOOL_CENTOS_MIRROR={{ ansible_env['NODEPOOL_CENTOS_MIRROR']|default('http://mirror.centos.org/centos', true) }} 10:08
amoralejmarios|ruck, we have a couple of different cases10:08
amoralejgimme a while please10:08
marios|ruckamoralej: ack 10:08
frenzyfridaybhagyashris, checking10:10
amoralejmarios|ruck, ysandeep|coffee https://review.opendev.org/c/zuul/zuul-jobs/+/827032 that is fix for the issue in multi-node-bridge10:12
amoralejmarios|ruck, yes, we need to fix that10:13
amoralejuntil we have collectd properly in the mirrors for cs8, i've created https://trunk.rdoproject.org/centos8-master/deps/collectd/ as workaround10:13
frenzyfridaybhagyashris, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-ussuri - ussuri sc10 kvm internal seems to be running, but I think it is not reporting to dlrn maybe?10:16
bhagyashrisfrenzyfriday, yeah10:17
frenzyfridayhm..  checking Meanwhile ++ to the patch ^^10:17
bhagyashrisfrenzyfriday, ack thanks :)10:17
*** chem is now known as Guest126010:21
marios|ruckamoralej: ack 10:21
*** ysandeep|coffee is now known as ysandeep10:51
ysandeepamoralej: ++ thanks for the patch10:51
dviroelo/11:14
*** rlandy is now known as rlandy|ruck11:14
* dviroel will rebase https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38694 - 1m11:19
rlandy|ruckslaweq: hi - can you review harold's comments on https://bugs.launchpad.net/tripleo/+bug/1958955/comments/11 and see if you agree. We will try a revert on testproject11:21
rlandy|ruckgthiemonge: hello ... per https://bugs.launchpad.net/tripleo/+bug/1958531 - last comment - your patches are working - thank you - juts waiting on merge there11:24
gthiemongerlandy|ruck: amazing! thanks for testing them!11:26
rlandy|ruckTengu: hey ... looking at https://bugs.launchpad.net/tripleo/+bug/1958230 - does proposed patch work for you?11:26
rlandy|ruckakahat: hey is https://review.opendev.org/c/openstack/tripleo-ci/+/823688 ready for review? on review list but DNM11:36
marios|ruckysandeep: https://review.rdoproject.org/r/c/rdo-jobs/+/38602 & https://review.opendev.org/c/openstack/tripleo-ci/+/82631711:37
marios|ruckrlandy|ruck: ^^ 11:39
marios|ruckrlandy|ruck: ysandeep: & https://review.rdoproject.org/r/c/config/+/38601 11:41
akahatrlandy|ruck, no not yet. 11:41
bhagyashrisrlandy|ruck, ysandeep kindly add in your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3866711:42
bhagyashrisadded in the review list hackmd 11:42
ysandeepbhagyashris, merged11:43
bhagyashrisysandeep, thanks 11:43
bhagyashrisananya is looking into that 11:43
ysandeepack, got that from ananya's comment11:44
dviroelrlandy|ruck: hi, fyi, i was pinning c9 ovb to 01/19 compose - got same results - https://logserver.rdoproject.org/74/35174/42/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compose-master-pinning/2fad731/logs/11:55
Tengurlandy|ruck: nope - yet another issue apparently.11:56
Tengumissing file or something, I've added a comment to the gchat + patch11:56
Tengurlandy|ruck: though - if the overcloud-full.qcow2 is being deprecated, it might be good to know about the proposed timeline for the actual deprecation - and I can switch to the overcloud-hardened-uefi-full instead.11:57
Tengurlandy|ruck: though I'm hitting some issues while trying to build that hardened-uefi-full image :). cf https://bugs.launchpad.net/tripleo/+bug/1959568 + https://bugs.launchpad.net/tripleo/+bug/195909311:58
Tenguwould be too easy.11:58
TenguI'll modify tripleo-lab to allow the hardened-uefi one, apparently it's not delivered as a targz, and there's probably a need for some option such as wholedisk or something.11:58
rlandy|ruckdviroel: thanks12:08
slaweqrlandy|ruck: yes, I already pushed some DNM patch to neutron to add some logs there and to try to understand what is going on there12:08
rlandy|ruckslaweq: awesome - thanks12:09
rlandy|ruckhttps://trello.com/c/hzuFEfxF/2320-cixlp1958955tripleociproa-cs9-master-fs001-fs002-fs020-fs035-fs039-and-fs064-is-failing-to-provision-overcloud-node-the-last-err12:10
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195895512:10
rlandy|ruck https://opendev.org/openstack/neutron/commit/1562f9141ba9c90c69c1c851dd9a50c8108ab15f12:10
rlandy|ruckysandeep: hey  pls join review time again12:17
amoralejmarios|ruck, wrt opstools change, did you send any patch?12:18
ysandeeprlandy|ruck, ack, joining back in 1 min12:18
marios|ruckamoralej: not yet (for quickstart you mean) sorry just got off a call and chasing sthing else will do in few mins12:22
amoralejno problem, i'm fixing in rdo, in some minutes i can also help on that12:23
marios|ruckamoralej: thx12:23
amoralejbtw, isn't this breaking all the cs8 jobs?12:23
amoralejwhat's the impact in tripleo-ci?12:23
marios|ruckamoralej: haven't seen it in gates... promotions are failing on sthing else so possibly we don't see it at that point? 12:23
marios|ruckamoralej: we have 2 centos8 appstream related/deps(?) issues12:24
amoralejmarios|ruck, the other one is the gcc one?12:24
amoralejwhere i can see logs?12:24
marios|ruckamoralej: yeah gcc and pacemaker-cluster-libs logs on bugs @ https://bugs.launchpad.net/tripleo/+bug/1959524 https://bugs.launchpad.net/tripleo/+bug/1959526 12:24
amoralejmmm pacemaker-cluster-libs?12:25
amoralejlemme check12:25
marios|ruckslaweq: can you point me to your dnm patch i don't see it quickly https://review.opendev.org/q/project:openstack/neutron 12:25
marios|ruckslaweq: for https://bugs.launchpad.net/tripleo/+bug/1958955/comments/11 12:26
slaweqmarios|ruck: https://review.opendev.org/c/openstack/neutron/+/82699912:26
marios|ruckslaweq: thank you .. ok so it isn't revert you are debugging for now... i may post the revert so i can testproject and see if it is OK? 12:27
rlandy|ruckdviroel: sshnaidm: hey - can you guys vote on https://review.opendev.org/c/openstack/tripleo-ci/+/826521 - to spring us on downstream? it fails tox now due to opstools  - will recheck later12:29
marios|ruckslaweq: fyi https://review.opendev.org/c/openstack/neutron/+/827014 & https://review.rdoproject.org/r/c/testproject/+/38830 12:34
amoralejmarios|ruck, i commented in https://bugs.launchpad.net/tripleo/+bug/1959526/ it should work now12:40
amoraleji proposed a fix12:40
amoraleji mean, didn't send a patch, just in the comment12:41
arxcruzmarios|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/815654 12:41
arxcruzcan you review it?12:41
arxcruzyou add some comments, but I think we are good to go12:41
rlandy|ruckmarios|ruck: so reading back on amoralej's comments on https://bugs.launchpad.net/tripleo/+bug/1959526/ - it looks like we do not disable the centos repos laid down in image builds12:42
arxcruzmarios|ruck nevermind12:42
amoralejrlandy|ruck, actually, it seems baseos and appstream are disabled12:44
amoralejbut not others as highavailability12:44
rlandy|ruckamoralej: ack - looking into why that is12:44
amoralejrlandy|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/12:44
slaweqmarios|ruck: 2 things - first, I don't want to revert that patch as it fixes some other CI issue which we had few times12:45
slaweqand 2nd - are You sure it will work with such depends-on? I proposed https://review.rdoproject.org/r/c/testproject/+/38582 which depends on https://review.opendev.org/c/openstack/neutron/+/826999 and I don't see my extra logs in the dhcp agent's logs in the undercloud (that's where the issue is)12:46
rlandy|ruckamoralej: are we looking at the same problem on https://bugs.launchpad.net/tripleo/+bug/1959524?12:46
* rlandy|ruck checks those repos12:46
amoralejlemme check12:46
amoralejrlandy|ruck, that's probably different12:47
amoralejlemme see12:47
rlandy|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/logs/undercloud/etc/yum.repos.d/12:47
rlandy|ruckack12:47
rlandy|ruckthose are all disabled properly12:47
* rlandy|ruck looks into fixing the first one12:47
marios|ruckslaweq: ack .. more so we can say if it is the root or not... cheap to find out if it works (running now https://review.rdoproject.org/zuul/stream/9478b678c7e646f0a6aba3da67cb9175?logfile=console.log i think it should work lets see it has ZUUL_CHANGE_IDS=827014,1 38830,1 12:49
marios|ruckslaweq: i have DNM and workflow-1 on the neutron patch 12:49
marios|ruckslaweq: let me check logs from your job sec12:50
slaweqmarios|ruck: I think that depends-on works correctly, but maybe the culprit is different12:50
slaweqI have node on hold now and I'm investigating it12:50
marios|ruckslaweq: should be there (the neutron change i mean) looking at https://logserver.rdoproject.org/82/38582/3/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/cd78e7f/job-output.txt 2022-01-31 04:11:15.574423 | primary | included: /home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/build-test-packages/tasks/dlrn-build.yml for undercloud => (item={'host': 12:51
marios|ruck'https://review.opendev.org/c/openstack/neutron/+', 'project': 'openstack/neutron', 'branch': 'master', 'refspec': 'refs/changes/99/826999/1'}) 12:51
marios|ruckslaweq: ack thanks12:51
amoralejrlandy|ruck, that's strange, i see AFS synchronization for gcc and gcc-c++ happened fine, in the same sync and repos are configured to use AFS mirrors12:54
amoralejit's just that occurrence?12:54
amoralejactually gcc-8.5.0-8 was synced on centos.log.1:2022-01-28T16:4312:55
amoralejbut the log is from 30th12:55
rlandy|ruckamoralej: nah - been going on since friday12:55
rlandy|ruckonly that rpm12:55
rlandy|ruckbut it's been happening for a while12:56
amoralejit's strange i see in that log that it pulled http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/BaseOS/x86_64/os/Packages/libgfortran-8.5.0-7.el8.x86_64.rpm13:01
amoralejbut by then, 8.5.0-8 was available in the repo13:01
rlandy|ruckyep - it is13:02
amoralejmmm, has failed 29, 30 and 3113:03
amoralejrlandy|ruck, only for victoria?13:03
Tengurlandy|ruck: quick feedback: using the hardened-uefi-full image available on images.rdoproject, I get a successful node provisioning. So, if the overcloud-full.qcow2 is to be deprecated, I can live with that.13:06
Tengurlandy|ruck: tripleo-lab is able to eat both now.13:06
marios|ruckamoralej: cant we use http://mirror.centos.org/centos/8-stream/opstools/x86_64/collectd-5/ instead of https://trunk.rdoproject.org/centos8-master/deps/collectd or it doesn't have collectd yet in mirror. ? 13:09
amoralejmarios|ruck, collectd is still not in that repo13:10
amoralejsee http://mirror.centos.org/centos/8-stream/opstools/x86_64/collectd-5/Packages/13:10
amoralejthat's the actual problem13:10
marios|ruckamoralej: yes sorry just checked ack 13:10
marios|ruckamoralej: k posting patch in a few 13:10
*** amoralej is now known as amoralej|lunch13:10
marios|ruckamoralej|lunch: but will you create the same for wallaby... train or just master ? 13:14
marios|ruckamoralej|lunch: i mean create teh https://trunk.rdoproject.org/centos8-master/deps/ 13:14
*** beagles is now known as eagles13:16
*** dasm|off is now known as dasm13:19
dasmo/13:19
*** dasm is now known as dasm|rover13:19
marios|ruckamoralej|lunch: no ... stream will be the same regardless of openstack branch i think so you'll probably just use one repo as the temp 13:22
rlandy|ruckbhagyashris: hello ... https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38814/1/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh13:24
rlandy|ruckbhagyashris: merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3881413:25
rlandy|rucklet me know when we can out it back13:26
marios|ruckrlandy|ruck: amoralej|lunch: i think we aren't seeing the issue with collectd because we aren't enabling that repo anywhere that we are running (the featuresets here aren't used I think) 13:26
marios|ruckamoralej|lunch: its why we dind't see it in ci 13:26
dviroelrlandy|ruck: hi, https://bugs.launchpad.net/tripleo/+bug/1959582 - for centos9-scenario007-standalone-wallaby - didn't find any related LP13:26
marios|ruckamoralej|lunch: rlandy|ruck: i think we would have seen it in th eintegration line but that is broken earlier so isn't reaching that point13:27
marios|ruckamoralej|lunch: anyway posting the patch and we can discuss there13:27
rlandy|ruckdviroel: there probably wasn't one - thanks13:27
marios|ruckamoralej|lunch: going to point to https://trunk.rdoproject.org/centos8-master/deps/collectd/ for all the branches with TODO13:27
Tengudang..... some more selinux issues.13:30
bhagyashrisrlandy|ruck, ack13:37
rlandy|ruckamoralej|lunch: not just victoria13:39
rlandy|ruckthat was just the first place we saw it13:40
marios|ruckamoralej|lunch: what is [1] please i want to reference it too https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673/6#message-a01d8c3a9b8dfa9e1ff66e34f99952fb22440fb2 13:43
frenzyfridayHey marios|ruck 0/ have you seen this error: "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n  - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n  - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n  - cannot install the best candidate for the job" ?13:44
frenzyfridayscenario010-kvm-internal-standalone wallaby and ussuri are hitting this. bhagyashris : Maybe thats what was blocking the promotions?13:45
bhagyashrisfrenzyfriday, fyi https://trello.com/c/AFB8lJuR/2331-cixlp1959524tripleociproa-ostempest-pythonvenvbuild-failing-with-depsolve-error-occured-n-problem-cannot-install-both-gcc-850-7e13:45
marios|ruckfrenzyfriday: yeah 13:46
marios|ruckfrenzyfriday: it blocks all the c8 lines 13:46
marios|ruckfrenzyfriday: on cix call 13:46
frenzyfridayack, thanks13:47
slaweqmarios|ruck: rlandy|ruck I think that the issue isn't caused by my patch https://opendev.org/openstack/neutron/commit/1562f9141ba9c90c69c1c851dd9a50c8108ab15f but I more or less know what is going on ther13:54
slaweqI will send small patch in few minutes so we can test it in test project13:54
slaweqI hope it will be better with that fix13:54
*** amoralej|lunch is now known as amoralej13:54
amoralejamoralej, that make sense, but i think collectd is used to build containers, didn't we hit the issue in the jobs building containers?13:55
slaweqmarios|ruck: rlandy|ruck FYI: https://review.opendev.org/c/openstack/neutron/+/82710113:59
slaweqand I just updated https://review.rdoproject.org/r/c/testproject/+/38582 to test it13:59
slaweqlets see how it will work now13:59
marios|ruckslaweq: thanks in cix call now will check in a bit13:59
slaweqykarel_: hi, You can clean the node which You hold for me earlier (IP 38.102.83.85)14:00
slaweqthx a lot, it was very helpful14:00
bhagyashrisrlandy|ruck, ysandeep marios|ruck dam dviroel rcastillo 14:01
bhagyashrisscrum time14:01
rlandy|ruckwe;re on cix14:01
rlandy|ruckwill join in a few14:01
bhagyashrisack14:02
dasm|roverin srbac. will join in a bit14:02
dviroeljoinning14:02
ysandeepsshnaidm: I would like to join today's meeting as well with my UA hat on so that I can understood how much work is there, please invite me as well.14:10
amoralejthere may be some way to hold a node for https://bugs.launchpad.net/tripleo/+bug/1959524 ?14:36
marios|ruckamoralej|lunch: what is [1] please i want to reference it too https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673/6#message-a01d8c3a9b8dfa9e1ff66e34f99952fb22440fb2 14:37
marios|ruckamoralej: ah thanks you upated 14:37
marios|ruck(sorry still in call :D)14:37
marios|ruck[1] https://www.centos.org/centos-linux-eol/14:37
amoralejnp14:37
amoraleji can't reproduce it by myself14:37
sshnaidm** I created an event for talk today with SDK folks, 16.30 UTC: https://meet.google.com/teo-fhxt-irr - if I didn't add someone and you would like to join - ping me, or just join :)14:38
ysandeepsshnaidm: thanks!14:41
* bhagyashris dinner brb14:44
marios|ruckrlandy|ruck: https://opendev.org/openstack/tripleo-repos/src/branch/master/plugins/module_utils/tripleo_repos/main.py 14:47
rlandy|ruckmarios|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/8128424/yum.repos.d/14:50
rlandy|ruckmarios|ruck" dasm|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/82368814:53
marios|ruckdasm|rover: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8 14:53
marios|ruckdasm|rover: https://opendev.org/openstack/tripleo-repos/src/branch/master/plugins/module_utils/tripleo_repos/main.py#L4114:55
* frenzyfriday will be back in 2 hrs14:57
rlandy|ruckdasm|rover: https://review.rdoproject.org/r/c/testproject/+/3806514:57
rlandy|ruckdasm|rover: https://bugs.launchpad.net/tripleo/+bug/195952415:00
rlandy|ruckmarios|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 ready to merge?15:13
marios|ruckrlandy|ruck: yeah i think so15:19
marios|ruckrlandy|ruck: i don't think it affects us in ci 15:19
marios|ruckrlandy|ruck: because https://opendev.org/openstack/tripleo-quickstart/src/commit/358d8da7877a5cb408bbf79c80ddd42edf9ef553/config/release/tripleo-ci/CentOS-8/master.yml#L20215:19
marios|ruckrlandy|ruck: and https://codesearch.opendev.org/?q=enable_opstools&i=nope&literal=nope&files=&excludeFiles=&repos= (none of the featuresets are used i think)15:19
marios|ruckrlandy|ruck: but lets merge it anyway since the 8 one is gone and until  we get the 8-stream opstools 15:20
rlandy|rucksshnaidm: can you add me to the sdk meeting15:21
sshnaidmrlandy|ruck, done15:22
rlandy|ruckmarios|ruck: assuming https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 won;t fix the tox issues we're seeing15:24
marios|ruckrlandy|ruck: not that am aware of :)15:25
marios|ruckrlandy|ruck: link to tox issue?15:25
rlandy|ruckmarios|ruck: also https://review.opendev.org/c/openstack/tripleo-ci/+/823688/ is not passing check15:26
rlandy|ruckdasm|rover: ^^15:26
rlandy|ruckwe need to fix tripleo-repos15:26
dasm|roverrlandy|ruck: do you mean exceeded "retry_limit" on centos-8?15:33
dasm|rover> ERROR: This script does not work on Python 3.6 The minimum supported Python version is 3.7. Please use https://bootstrap.pypa.io/pip/3.6/get-pip.py instead.15:36
dasm|roverhttps://zuul.opendev.org/t/openstack/build/83bd7f6201114b7fbd3274cd7faa152f15:36
dasm|roverpip released today 22.0 stopped supporting python 3.615:37
dasm|rover> Requires: Python >=3.715:37
dasm|roverhttps://pypi.org/project/pip/15:37
beagleswas just about to ask about a missing opstools repo for CentOS.. but I thnk this is being discussed already?15:38
marios|ruckbeagles: yeah are you seeing it in ci now? 15:39
*** dviroel is now known as dviroel|lunch15:39
beaglesmarios|ruck: no I'm seeing problems with image configi using local quickstart deployment15:39
marios|ruckbeagles: k this might help you https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 15:39
rlandy|ruckysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/826521/8..9/roles/oooci-build-images/add_cert.yaml15:40
marios|ruckbeagles: or edit the relevant file in your env 15:40
rlandy|ruckd /etc/pki/ca-trust/source/anchors/rh.crt:/etc/pki/ca-trust/source/anchors/rh.crt15:40
beaglesmarios|ruck: perfect! thanks!15:40
marios|ruckbeagles: ack np15:40
ysandeeprlandy|ruck, thanks for highlight that15:40
rlandy|ruckwould aonly make sense for path 115:41
ysandeeprlandy|ruck, I wanted to move to --upload {{ cert_path }}:{{ cert_path }} but missed that15:41
*** jpodivin_ is now known as jpodivin15:43
rlandy|ruckysandeep: let's meet in 1515:44
ysandeeprlandy|ruck, ack15:44
ysandeeprlandy|ruck, trying https://review.opendev.org/c/openstack/tripleo-ci/+/826521/10/roles/oooci-build-images/add_cert.yaml#48 15:45
rlandy|ruckdasm|rover: marios|ruck: ugh - may need to fix tripleo-repos15:45
rlandy|ruckquickest path15:45
dasm|roverrlandy|ruck: you mean "add pip to install"?15:45
dasm|roverit seems that we're checking for the pip presence, which tries to download it, when it's missing.15:46
dasm|roverand it's pulling recent release.15:46
*** ysandeep is now known as ysandeep|dinner15:46
* ysandeep|dinner back in ~15 mins15:47
marios|ruckrlandy|ruck: yeah i see it... not sure if it is the quickest though maybe we can fix the issue with the akahat patch 15:48
marios|ruckrlandy|ruck: i think we have another/new issue fs39 train digging a bit 15:48
rlandy|ruckmarios|ruck: /o\ too man issues16:00
rlandy|ruckmany16:00
rlandy|ruckdasm: ack - if you can fix that review16:04
rlandy|ruckysandeep|dinner: when you are back ... https://meet.google.com/igg-cjeb-fhi?pli=1&authuser=016:04
marios|ruckrlandy|ruck: luckily this one is nit/easy fix dealing with now 16:06
*** ysandeep|dinner is now known as ysandeep16:06
ysandeeprlandy|ruck, joining16:06
*** amoralej is now known as amoralej|off16:30
marios|ruckrlandy|ruck: dasm|rover: going in a few mins folks need sthing before i do? otherwise same again tomorrow ;)16:31
marios|ruckwew monday 16:31
rlandy|ruckmarios|ruck: just want we can pick up16:32
marios|ruckrlandy|ruck: should be on the reviews list16:32
marios|rucklemme check 16:32
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/1959524 - says still investigating16:32
rlandy|ruckso pacemaker-libs fix16:33
marios|ruckrlandy|ruck: dont have any updates on those cix16:33
marios|ruckrlandy|ruck: only https://review.opendev.org/c/openstack/tripleo-quickstart/+/827143 for train 16:33
marios|ruckrlandy|ruck: and https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 but not urgent/should not be seeing that in ci 16:34
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/827143 - thanks for that16:36
marios|ruckrlandy|ruck: will pickup in the morning - try get some progress on the cix :/16:37
rlandy|ruckmarios|ruck: sure dasm|rover and I will leave you nots16:38
rlandy|rucknotes16:38
*** dviroel|lunch is now known as dviroel16:38
rlandy|ruckmarios|ruck: no worries - we are all broken on gcc16:40
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195952416:40
rlandy|ruckamoralej|off is gone already16:40
rlandy|ruckso we'll see tomorrow16:40
marios|ruckrlandy|ruck: good news from slaweq at least ;)16:41
dviroelyeah?16:41
marios|ruckdviroel: for the ovb provision issue see #tripleo just now16:41
* marios|ruck out16:41
*** marios|ruck is now known as marios|out16:42
dviroelwill take a look o/16:42
rlandy|ruckdviroel:  https://logserver.rdoproject.org/82/38582/4/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/ae3536a/ 16:43
dasm|roverhmm... marios is out :/16:45
rlandy|ruckdasm|rover: what do you need?16:46
dviroelinteresting, small fix on neutron16:46
rlandy|ruckdasm|rover: still testing amol's patch?16:46
dviroelwe should tp against more ovb jobs, just to make sure16:46
dasm|roverrlandy|ruck: i'm not testing amol's patch. the issue is we're pulling latest pip from upstream16:47
dasm|rovermarios introduced that change here: https://review.opendev.org/c/openstack/tripleo-ci/+/81822216:47
dasm|roverand i wanted to double check with him if that's actually what he intended to do16:47
dasm|roverhere: https://review.opendev.org/c/openstack/tripleo-ci/+/818222/49/roles/tripleo-build-jobs-repos/tasks/main.yaml#3816:47
dasm|roverrlandy|ruck: maybe you remember what was the intend here? why he was uninstalling pip from repo and installing from the upstream?16:48
rlandy|ruckdasm|rover: not really16:48
dasm|roverack16:48
dasm|roverlemme try this change and i'm gonna rerun jobs to see if that's what we need to do16:48
rlandy|ruckdasm|rover: amoralej|off said mirrors should be fixed atm16:49
dasm|roverack16:49
rlandy|ruckso yeah - maybe line rekick16:49
rlandy|ruckbhagyashris: any new on c7?16:49
rlandy|ruckdasm|rover: so you can rekick openstack-periodic-integration-stable4 16:55
rlandy|ruckand openstack-periodic-integration-stable216:55
rlandy|ruckand see if we spring free16:55
rlandy|ruckwe need to fix the issue16:55
rlandy|ruckanyways16:55
rlandy|ruckbut we may be able to promote16:56
dasm|roverrlandy|ruck: i can do that, but i'm not sure if the issue is going to be fixed.16:58
dasm|roverrlandy|ruck: we're pulling upstream pip, which dropped support for python 3.616:58
dasm|roverhowever centos8 has python 3.6, hence we're unable to install it.16:59
dasm|roveri submitted change https://review.opendev.org/c/openstack/tripleo-ci/+/827157 and i'm going to run it with my fix.16:59
rlandy|ruckok - dropping meeting - will chat with you16:59
dasm|roverit should fix the issue with c8, but i don't know why marios needed that in first place. in the commit message, he mentioned it as needed for c917:00
rlandy|ruckyou can switch case17:00
dasm|roverswitch case? what do you mean?17:00
dasm|roverhmm.. i see there is another issue, which you just mentioned, about broken mirrors.17:01
dasm|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-train/4844be3/job-output.txt17:01
dasm|roverbrb17:02
dasm|roverback17:05
rlandy|ruckdasm|rover: ok - https://meet.google.com/rrx-nbyd-gfb?pli=1&authuser=017:05
*** ysandeep is now known as ysandeep|out17:30
*** jpena is now known as jpena|off17:32
ysandeep|outrlandy|ruck, /me out, please keep an eye on https://review.opendev.org/c/openstack/tripleo-ci/+/826521 thanks!17:36
* frenzyfriday afk17:41
rlandy|ruck823688,1517:42
dasm|roverrlandy|ruck: fyi, promotion is failing due to "primary | /bin/sh: /home/zuul/workspace/hash_info.sh: No such file or directory"17:50
dasm|roveri'm gonna look into that as a next thing17:50
rlandy|ruckdasm|rover: taking few minutes for lunch - brb17:52
dasm|roverk, no worries17:54
rlandy|ruckdasm|rover: back - looking into line failures18:13
dasm|roverk18:13
rlandy|ruckdasm|rover: thanks for creating https://review.rdoproject.org/r/c/testproject/+/38844/1/.zuul.yaml18:14
dasm|rover:) i'm looking at its progress now18:14
rlandy|ruckrcastillo: hey ... how about some more tox molecule fun?18:18
rlandy|ruckrcastillo: want to debug this one for us? https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-molecule&skip=018:18
rcastillooh boy18:19
rcastillolooking18:19
rlandy|ruckrcastillo: tripleo-ci as well18:20
rlandy|ruckrepo layout there18:21
rlandy|ruckrcastillo: needs to be running on a stream node18:27
rcastilloyeah I saw18:28
rcastilloneed to mount the vars I believe18:28
rlandy|ruck2022-01-31 17:33:22.448391 | localhost | Distro: Ubuntu 20.0418:28
rlandy|ruckrcastillo: it's defined in a templates18:28
rlandy|ruckrcastillo": https://meet.google.com/exu-fomx-knp?pli=1&authuser=018:29
dviroelrlandy|ruck: pls, take a look at this LP later https://bugs.launchpad.net/tripleo/+bug/1956475 - hitting wallaby too, no CIX card openned I think18:43
dviroelrlandy|ruck: tests being skipped: https://opendev.org/openstack/openstack-tempest-skiplist/commit/85a70a8691010b82f1a509a313f4aa2bc829bb7d18:43
dasm|roverrlandy|ruck: fyi, https://bugs.launchpad.net/tripleo/+bug/195961118:51
rlandy|ruckdasm|rover: k - need to get on meeting now ...18:57
rlandy|ruckspoke to rcastillo about molecule failures18:57
rlandy|ruckneed ti fix linter failure on marios patch18:57
rlandy|ruckdasm|rover: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-train/abccb60/job-output.txt - need marios patch to fix18:59
dasm|roveri think the issue i just reported is a red herring. i see it being marked as "ignore_errors: true"18:59
rlandy|ruckdasm|rover: it;s ok  just close it18:59
dasm|roverk18:59
rlandy|ruckperiodic-tripleo-centos-8-buildimage-overcloud-hardened-full-train need to testproject that with https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111/19:00
rlandy|ruckand fix linter ussue there ^^19:00
rlandy|ruckdasm|rover: ^^ you can give that a shot19:00
rlandy|ruckussuri white space19:00
dasm|roverk19:02
dasm|roverfixed linter issue, started test job with ^19:11
dasm|roverbrb19:33
rlandy|ruckdasm|rover: thanks19:43
dasm|roverback19:47
rlandy|ruckok - out of meeting20:18
rlandy|ruckrcastillo: how are things looking?20:18
rlandy|ruckdasm|rover: here's the next issue: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=020:19
dasm|rovermhm20:20
dviroelDEBUG util.py:596:  Errors during downloading metadata for repository 'centos-opstools':20:21
dviroelDEBUG util.py:596:    - Status code: 404 for http://mirror.centos.org/centos/8/opstools/x86_64/collectd-5/repodata/repomd.xml (IP: 85.236.43.108)20:21
rlandy|ruck - Status code: 404 for http://mirror.centos.org/centos/8/opstools/x86_64/collectd-5/repodata/repomd.xml (IP: 85.236.43.108)20:21
rlandy|ruckugh20:21
rlandy|ruckwe need a bug for the opstools issue20:21
rlandy|rucksee https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111/20:22
* rlandy|ruck logs20:23
rlandy|ruckdasm|rover: dviroel: rcastillo: at least we have a bug for it now: https://bugs.launchpad.net/tripleo/+bug/195961920:32
dasm|roverack20:32
dasm|roverrlandy|ruck: this one https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=0 is also affected by opstools20:35
dasm|roverhttps://2af8965ad28e49781e91-ba7281a549a418e9d574e043a1b402b5.ssl.cf5.rackcdn.com/827088/2/gate/tripleo-ci-centos-8-content-provider/b05d748/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/ec/33/ec33ab40488efacaf98e31852b280947ee73bff8_dev/rpmbuild.log20:35
dasm|roverthis particular change: https://review.opendev.org/82708820:35
dasm|roverchecking others20:35
rlandy|ruckdasm|rover: yep - pinging also rhos-ops to get https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673 merged20:35
dasm|rovermy hunch says -- they're the same20:35
rcastillomolecule job on quickstart failing with a 404 on an image now https://d5816c7856271a4bd827-9262fa59dca60616978c458f4dd8b484.ssl.cf2.rackcdn.com/827166/4/check/tripleo-tox-molecule/e790e3a/20:36
rcastilloso https://review.opendev.org/c/openstack/tripleo-quickstart/+/827166 clears the repo issue we were getting at least20:37
rcastillorlandy|ruck: ^^20:38
rlandy|ruckrcastillo: posting out about gate blocket20:39
rlandy|ruckblocker20:39
* rcastillo lunch, brb20:40
rlandy|ruckoh gee20:41
dasm|rover?20:41
rlandy|ruckrcastillo: k- let me see if I can fix that20:42
rlandy|ruckmaybe we can get this in20:42
*** dviroel is now known as dviroel|brb20:48
* rcastillo back21:01
rcastilloso c8 is actually gone for realsies now huh?21:01
rlandy|ruckrcastillo: no it exists21:04
rlandy|ruckthis is what happened21:04
rlandy|ruckthere was a space issue on the images server21:04
rlandy|ruckso https://images.rdoproject.org/centos8/wallaby/rdo_trunk/current-tripleo/ promoted empty21:05
rlandy|ruckfs039 is the only test outstanding to promote wallaby21:05
rlandy|ruckso I am rerunning21:05
rlandy|ruckrcastillo: maybe make that test non-voting21:06
rlandy|ruckjust to get marios patch in to spring everything else???21:06
rlandy|ruckdasm|rover: rcastillo: this is literally the craziet ruck/rover day ever21:07
dasm|rover#TODO: [x] - attend craziest day ever21:08
dasm|roverchecked!21:08
rlandy|ruckdasm|rover: so sorry 21:08
rcastillogeez21:08
dasm|roverno worries, rlandy|ruck! you didn't plan it21:09
dasm|rover(at least i hope so ;P) 21:09
rlandy|ruckrcastillo: dasm|rover: let's meet and try get the order of merge here21:09
dasm|roverk21:09
rcastillosure21:10
rlandy|ruckhttps://meet.google.com/rcf-veof-car?pli=1&authuser=021:11
dasm|rover> Failed to download metadata for repo 'centos-opstools'", "2022-01-31 21:59:08 | Error: Failed to download metadata for repo 'centos-opstools'"]}22:13
dasm|roverhttps://logserver.rdoproject.org/44/38844/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby/94fff24/job-output.txt22:13
dasm|roverfor https://review.rdoproject.org/r/c/testproject/+/3884422:13
rlandy|ruckoff call22:27
rlandy|ruckchecking on jobs in a few22:27
dasm|roverrlandy|ruck: i'm out of steam for today. i'm gonna disconnect.22:28
rlandy|ruckdasm|rover: sure - see you tomorrow22:28
dasm|roverrlandy|ruck: sorry for leaving you in the middle of the storm22:28
*** dasm|rover is now known as dasm|off22:29
rlandy|ruckdasm|off; no worries - have a good night22:35
rlandy|ruckrcastillo: looking into failure22:38
rlandy|ruckmay need to update the patch22:39
rlandy|ruckrcastillo: merging https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3885623:05
rlandy|ruckto promote23:05
rcastillorlandy|ruck: ack23:05
*** ysandeep|out is now known as ysandeep23:30
ysandeeprlandy|ruck, need help with anything for check/gate till marios comes online? 23:58
rlandy|ruckysandeep: hey23:58
rlandy|ruckysandeep: we're dealing with so many failures23:59
rlandy|ruckI need to get marios patch though23:59
rlandy|ruckwith also disables molecule23:59
ysandeepI can keep an eye if you can share the patch?23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!