| *** ysandeep|out is now known as ysandeep|rover | 04:58 | |
| ysandeep|rover | chandankumar: good morning o/ When you have a free slot could you please take a look on why dlrn is not building rpm in testproject with depends-on diskimage-builder patch on https://review.rdoproject.org/r/c/testproject/+/39357 | 05:12 |
|---|---|---|
| ysandeep|rover | https://logserver.rdoproject.org/57/39357/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/d67c11f/job-output.txt | 05:12 |
| ysandeep|rover | I was trying to test Tengu 's patch https://review.opendev.org/c/openstack/diskimage-builder/+/840825 | 05:13 |
| marios | ysandeep|rover: from quick look i think it may be because of branches/release (train vs master) we have a conditional in there that tries to match it unless that changes | 05:22 |
| marios | changed | 05:22 |
| chandankumar | ysandeep|rover: https://review.rdoproject.org/zuul/builds?change=840825&skip=0 | 05:22 |
| chandankumar | ysandeep|rover: marios is correct | 05:23 |
| ysandeep|rover | chandankumar, marios so I should choose train branch in testproject. | 05:23 |
| marios | ysandeep|rover: https://github.com/openstack/tripleo-quickstart-extras/blob/cb5ba7c682c0a158584143272380c5cafe3a93b5/roles/build-test-packages/tasks/main.yml#L202 | 05:24 |
| marios | something like that ^^^ | 05:24 |
| marios | is blocking it i think | 05:24 |
| marios | ysandeep|rover: i think the problem is that 'release' is defined for the train job | 05:24 |
| marios | but item.branch is master | 05:24 |
| marios | ysandeep|rover: so not sure what you propose would work has to be the other way around? not sure but dig at that conditional maybe can come up with somehtin ghtere | 05:25 |
| marios | o/ chandankumar | 05:25 |
| marios | :D morning | 05:25 |
| chandankumar | marios: Good morning \o | 05:25 |
| ykarel | marios, ysandeep|rover dib is branchless so need to add it to artg_branchless_projects so it get's build in non master runs | 05:25 |
| ykarel | or other option is to run job in testproject stable/train branch | 05:26 |
| ykarel | iirc correctly dib is also pinned in train | 05:27 |
| * ykarel checks | 05:27 | |
| ykarel | yes source-branch: 3.9.0 | 05:27 |
| ysandeep|rover | ykarel: add dib in default artg_branchless_projects list - https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/defaults/main.yml#L99 here? | 05:28 |
| ykarel | but for train c8 depends-on should be ok, issue is with c7 | 05:28 |
| ykarel | ysandeep|rover, yes | 05:28 |
| ysandeep|rover | ykarel++ thanks! let me try both ways - testproject for testing and also add dib in artg_branchless_projects default list so that we don't hit this issue again. | 05:29 |
| ysandeep|rover | testproject(train* branch) | 05:29 |
| ykarel | testproject master should work if added to artg_branchless_project | 05:29 |
| marios | ykarel: nice thanks | 05:30 |
| ysandeep|rover | ykarel: ack, understood thanks! | 05:30 |
| chandankumar | ysandeep|rover: ykarel thanks! | 05:30 |
| ysandeep|rover | marios++ chandankumar++ ykarel++ thanks guys! | 05:30 |
| ykarel | ysandeep|rover, that dib change just merged, so depends-on will no longer work now | 05:31 |
| ysandeep|rover | yeah, i will propose a dnm patch in dib for testing | 05:31 |
| chandankumar | ykarel: when we add it to artg_branchless_project and then Does rdoinfo pin does affect the dib depends-on while runing in test project? | 05:31 |
| ykarel | k that should work | 05:31 |
| ykarel | chandankumar, no rdoinfo pin do not impacte building in CI | 05:32 |
| chandankumar | ykarel: ok | 05:32 |
| ykarel | but as it's pinned in rdoinfo dlrn will not build master commits for train | 05:32 |
| chandankumar | yup, thanks for clarifing it | 05:33 |
| soniya29 | ysandeep|rover, hello | 06:09 |
| ysandeep|rover | hello o/ | 06:10 |
| soniya29 | ysandeep|rover, rdo promotions are lagging behind | 06:11 |
| ysandeep|rover | we are doing okay, except train | 06:11 |
| soniya29 | ysandeep|rover, gate seems okay :) | 06:11 |
| soniya29 | yeah cs8 and cs9 train are of concern | 06:12 |
| ysandeep|rover | cs9 train? we don't have cs9 for train - do you mean wallaby | 06:12 |
| *** soniya29 is now known as soniya29|ruck | 06:13 | |
| soniya29|ruck | s/cs9/cs7 | 06:13 |
| soniya29|ruck | sorry | 06:13 |
| soniya29|ruck | cs7 train i mean | 06:13 |
| ysandeep|rover | okay, don't worry about c7 - I think we already did our last promotion for c7 train | 06:14 |
| soniya29|ruck | ysandeep|rover, anything you want me to look before i go looking for downstream? | 06:15 |
| ysandeep|rover | soniya29|ruck, vexx host did some changes on friday- we should be in better condition now but if you see same tempest issues please let me know. | 06:15 |
| ysandeep|rover | soniya29|ruck, please start with component pipelines in rdo first - and see if any component is behind. | 06:16 |
| soniya29|ruck | ysandeep|rover, okay | 06:16 |
| Tengu | marios, chandankumar, ysandeep|rover heya! isn't DIB branchless? | 06:20 |
| Tengu | ah, my patch merged anyways. | 06:20 |
| Tengu | :) | 06:20 |
| ysandeep|rover | Tengu: it is , https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841064 will fix the issue with build-test-package role. | 06:20 |
| Tengu | ah, good. | 06:21 |
| Tengu | ysandeep|rover: my patch merged though, so it should be in for next promotion I guess? | 06:21 |
| Tengu | oh, ok. +2 for that correction as well | 06:21 |
| Tengu | always good ensuring we're able to test things properly :) | 06:22 |
| ysandeep|rover | Tengu: yes, promotion via component line first and then integration line | 06:22 |
| Tengu | at least, I'm happy we found out the RCA for that issue. | 06:22 |
| Tengu | wasn't an easy one. Thank you for pointing the cloud-init update in cs8! | 06:22 |
| ysandeep|rover | Tengu, :) , C9 jobs already ran against your patch: https://review.rdoproject.org/zuul/builds?change=840825&skip=0 | 06:24 |
| ysandeep|rover | I was checking image build logs: overcloud-hardened-uefi-full.log | 06:25 |
| ysandeep|rover | https://logserver.rdoproject.org/25/840825/2/openstack-check/tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001/69ab4f1/logs/undercloud/home/zuul/overcloud-hardened-uefi-full.log.txt.gz | 06:25 |
| Tengu | c9 didn't have that issue though - but at least, it didn't break things more :) | 06:25 |
| Tengu | which is already a really good thing | 06:25 |
| ysandeep|rover | but I don't see 22-cloud-init-ssh_genkeytypes anywhere in logs - is that expected? | 06:26 |
| Tengu | err... don't tell me we have to point to the file in some other location.... | 06:26 |
| Tengu | nah. | 06:27 |
| Tengu | should be fine. | 06:27 |
| Tengu | we don't see the 20-enable-cloud-init either | 06:27 |
| Tengu | nor the 21-cloud-init-allow-password-auth | 06:27 |
| Tengu | it's kind of wrapped in some higher level thing. | 06:27 |
| Tengu | though..... https://logserver.rdoproject.org/25/840825/2/openstack-check/tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001/69ab4f1/logs/overcloud-controller-0/etc/cloud/cloud.cfg.txt.gz this one looks as if the script didn't kick in | 06:28 |
| Tengu | the ssh_genkeytypes should be at the very end of the file. | 06:28 |
| ysandeep|rover | yeah, i don't see post-install.d running | 06:31 |
| ysandeep|rover | Tengu, may be you can test image build in your builder to confirm that. | 06:32 |
| Tengu | hmm. lemme check if we do activate the cloud-init element.... | 06:32 |
| Tengu | but yeah. I can deploy a quick UC and see how it goes in. | 06:33 |
| ysandeep|rover | ykarel: fyi.. hit an issue during rpm build: https://logserver.rdoproject.org/57/39357/8/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/168da26/logs/undercloud/home/zuul/DLRN/data/repos/component/common/22/26/22261ef23f99eaf96d32d4fd0d20663ea42a599d_dev/mock.log.txt.gz | 06:37 |
| ysandeep|rover | but yes its trying to build dib rpm now after adding in artg_branchless_projects | 06:37 |
| Tengu | ysandeep|rover: is there a generated script for the build-image? | 06:38 |
| Tengu | oh. no. it's embedded in oooq, isn't it? | 06:38 |
| Tengu | iirc I have something here. | 06:38 |
| ysandeep|rover | Tengu: https://logserver.rdoproject.org/25/840825/2/openstack-check/tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001/69ab4f1/logs/undercloud/home/zuul/build_images.sh.txt.gz | 06:38 |
| Tengu | oh | 06:39 |
| Tengu | I missed it. | 06:39 |
| Tengu | thanks! | 06:39 |
| ysandeep|rover | :) | 06:39 |
| Tengu | bingo. | 06:40 |
| Tengu | we don't call the cloud-init element! | 06:40 |
| Tengu | https://opendev.org/openstack/tripleo-common/src/branch/master/image-yaml/overcloud-hardened-images-uefi-python3.yaml | 06:41 |
| marios | Tengu: o/ yeah ykarel pointed that out earlier but the conditional there was still blocking it | 06:41 |
| Tengu | we should add the cloud-init in here | 06:41 |
| ysandeep|rover | ahh, that's explain the behavior | 06:41 |
| Tengu | ysandeep|rover: I'll propose a patch to enable this, and see what will happen for CS9. A testjob would be good to ensure it doesn't break things. | 06:43 |
| Tengu | (I'll add a depends-on to show the link between both patches, even if the DIB one is already in) | 06:44 |
| ysandeep|rover | Tengu: thanks! ack for testjob , I will prepare it | 06:45 |
| ykarel | ysandeep|rover, looking | 06:45 |
| Tengu | hmmmmmm iirc... lemme check something, I'd rather avoid duplicating things. | 06:45 |
| Tengu | yeah, ok. I can push in overcloud-hardened-images-uefi-python3.yaml | 06:45 |
| ykarel | ysandeep|rover, there is conflict in master requirements and patch applied in train | 06:46 |
| ykarel | https://github.com/rdo-packages/diskimage-builder-distgit/blob/train-rdo/0001-Revert-Drop-six-usage.patch#L66-L76 | 06:48 |
| ysandeep|rover | ykarel: ack, and maybe we didn't hit this issue in train because of dib pinning | 06:48 |
| Tengu | ysandeep|rover: https://review.opendev.org/c/openstack/tripleo-common/+/841067 | 06:48 |
| ykarel | yes exactly, in train it's pinned so no new builds | 06:48 |
| ysandeep|rover | ykarel++ thanks! Atleast we know build-test-package role works now for stable branches of dib with testproject. | 06:54 |
| ysandeep|rover | Tengu, thanks for https://review.opendev.org/c/openstack/tripleo-common/+/841067 , I will testproject it | 06:57 |
| *** soniya29 is now known as soniya29|ruck | 06:58 | |
| Tengu | ysandeep|rover: I'll also do it on my UC once it's deployed. | 07:01 |
| Tengu | 'k, launched with tripleo-common + diskimage-builder patches. let's see how the UC deploys, and how the image is built. | 07:06 |
| ysandeep|rover | soniya: I am dropping from the call but it looks like tempest component failure.. | 07:11 |
| ysandeep|rover | you can find the diff using rr script: http://pastebin.test.redhat.com/1050296 | 07:12 |
| *** soniya29 is now known as soniya29|ruck | 07:19 | |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 07:20 | |
| *** jpena|off is now known as jpena | 07:30 | |
| *** soniya29 is now known as soniya29|ruck | 07:38 | |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 07:57 | |
| Tengu | ysandeep|rover|lunch: hmmm, build failed here, but not sure about the actual cause. | 08:09 |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 08:23 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:23 | |
| ysandeep|rover | frenzy_friday, hey o/ what was the resolution of this bug: https://bugs.launchpad.net/tripleo/+bug/1961811 ? Did it resolved on itself? | 08:26 |
| ysandeep|rover | I am again seeing this today: https://logserver.rdoproject.org/64/841064/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/d84b9cf/logs/baremetal_2_22143_3-console.log | 08:26 |
| frenzy_friday | ysandeep|rover, checking | 08:26 |
| frenzy_friday | ysandeep|rover, https://trello.com/c/TpHRXYv5/2366-cixlp1961811tripleociproa-diskfilter-writes-are-not-supported-timeout-waiting-for-provisioned-nodes-to-become-available-in-fs1-w this was the activity on the bug | 08:29 |
| frenzy_friday | https://review.opendev.org/c/openstack/diskimage-builder/+/830900, https://review.opendev.org/q/topic:bootloader-nopart | 08:30 |
| ysandeep|rover | frenzy_friday: thanks! checking | 08:30 |
| soniya29|ruck | ysandeep|rover, baremetal, compute and network components are behind w.r.t promotions..i am starting with compute | 08:42 |
| ysandeep|rover | soniya29|ruck, for master branch? | 08:42 |
| soniya29|ruck | yes | 08:43 |
| ysandeep|rover | ack, fyi.. I have reopened: https://bugs.launchpad.net/tripleo/+bug/1961811 | 08:43 |
| soniya29|ruck | ysandeep|rover, on compute component..not all jobs have run but the one who have failed are transient | 08:47 |
| soniya29|ruck | ysandeep|rover, i am rekicking those jobs again | 08:48 |
| ysandeep|rover | soniya29|ruck, ack, fyi.. new bug for downstream container build: https://bugzilla.redhat.com/show_bug.cgi?id=2083067 | 08:49 |
| soniya29|ruck | ysandeep|rover, ack | 08:52 |
| Tengu | ysandeep|rover: do you happen to have the testproject link for the tripleo-common patch? | 09:09 |
| soniya29|ruck | ysandeep|rover, cs9 full-tempest-api-compute- master is failing on tempest for different test cases and different failures | 09:10 |
| ysandeep|rover | Tengu, ovb job for c9 ran against your patch itself: https://review.opendev.org/c/openstack/tripleo-common/+/841067/1#message-63763d24375052084aea5844781a955128ba65d3 | 09:11 |
| Tengu | oh. good. | 09:12 |
| Tengu | andd... faile. | 09:12 |
| Tengu | failed. | 09:12 |
| ysandeep|rover | failed to boot: https://logserver.rdoproject.org/67/841067/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/3ee489c/logs/baremetal_1_60753_2-console.log | 09:12 |
| Tengu | o_O | 09:12 |
| Tengu | again that disk thingy ?! | 09:12 |
| Tengu | no way. | 09:12 |
| Tengu | oh. | 09:13 |
| Tengu | wait. | 09:13 |
| Tengu | that cloud-init element only supports gentoo in addition | 09:13 |
| Tengu | grumpf. | 09:13 |
| ysandeep|rover | Tengu, https://logserver.rdoproject.org/67/841067/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/3ee489c/logs/undercloud/home/zuul/overcloud-hardened-uefi-full.log.txt.gz | 09:14 |
| ysandeep|rover | not seeing 22-cloud-init | 09:14 |
| Tengu | yeah... | 09:14 |
| Tengu | that's.... weird. | 09:14 |
| ysandeep|rover | but this time i see 21 running | 09:14 |
| Tengu | yup - and 20 as well. | 09:15 |
| Tengu | oh. maybe... errr. no, I've even added the depends-on | 09:16 |
| Tengu | what the heck | 09:16 |
| ysandeep|rover | Tengu, hah.. may be that's because diskimage patch is now merged | 09:17 |
| ysandeep|rover | diskimage-builder.noarch 3.20.4-0.20220421140835.adc40db.el9 @delorean-component-common | 09:18 |
| ysandeep|rover | https://logserver.rdoproject.org/67/841067/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/3ee489c/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 09:18 |
| ysandeep|rover | zuul doesn't take depends-on into account if patch is merged | 09:18 |
| Tengu | hmmmm so we're in a state where the patch is merged, but package not yet available in the right repo for CI? | 09:19 |
| Tengu | though CI was using the lowest level, such as ci-testing or something like that | 09:19 |
| ysandeep|rover | Tengu: correct | 09:19 |
| Tengu | humpf. | 09:19 |
| Tengu | brb, need some water. | 09:19 |
| ysandeep|rover | Tengu, let me see if patch is build by dlrn - if yes I can run that job in common component line. | 09:20 |
| ysandeep|rover | hah openstack-tacker which is also in common component is failing , diskimage-builder rpm can't promote further till that resolves | 09:23 |
| Tengu | meh. anything I can help with? | 09:23 |
| ysandeep|rover | Tengu, I will reach out to rdo folks: deps issue | 09:24 |
| ysandeep|rover | https://trunk.rdoproject.org/centos9-master/component/common/e2/db/e2db1c7a4032d316e5b655b858b023ace8435cd8_3e6f89cb/rpmbuild.log | 09:24 |
| Tengu | ok! | 09:24 |
| Tengu | lemme know if I can do anything. | 09:24 |
| ysandeep|rover | Tengu, thanks, i will let you know when we are in situation to test your patch. | 09:25 |
| soniya29|ruck | ysandeep|rover, https://bugs.launchpad.net/tripleo/+bug/1972163 | 09:26 |
| ysandeep|rover | soniya29|ruck, ack o/ please also set importance and milestone flag on the bug- i think these flags are needed if we decides to mark the bug promotion blocker - needed by cix script. | 09:30 |
| soniya29|ruck | ysandeep|rover, yupp | 09:30 |
| soniya29|ruck | ysandeep|rover, added | 09:31 |
| soniya29|ruck | ysandeep|rover, shall we remove full-tempest-api-compute-master from criteria? | 09:31 |
| soniya29|ruck | its a promotion-blocker and is failing 5th of May | 09:32 |
| ysandeep|rover | no, this is only blocking a particular component | 09:32 |
| ysandeep|rover | for compute related bugs - I think we need to give proper reproducer steps to nova team - how to reproduce a particular bug | 09:33 |
| ysandeep|rover | iirc.. agreement was we also report nova bug with correct service logs - not tempest log | 09:34 |
| ysandeep|rover | lot of keystone/ nova related error in logs ttps://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/dd92c65/logs/undercloud/var/log/extra/errors.txt.gz | 09:37 |
| soniya29|ruck | ysandeep|rover, ack | 09:37 |
| soniya29|ruck | ysandeep|rover, i will update the bug accordingly | 09:38 |
| ykarel | https://github.com/rdo-packages/diskimage-builder-distgit/blob/train-rdo/0001-Revert-Drop-six-usage.patch#L66-L76 | 09:40 |
| soniya29|ruck | ysandeep|rover, updated! | 10:08 |
| pojadhav | marios, could you please help me in this https://review.opendev.org/c/openstack/tripleo-ci/+/835460/25#message-4346afab13e74355781b373511e916d1129d81ef | 10:12 |
| pojadhav | on 4th may at least ran with failure. after that getting syntax error with some secret. | 10:18 |
| *** rlandy|out is now known as rlandy | 10:24 | |
| rlandy | ysandeep|rover: soniya29|ruck: hello ... | 10:25 |
| soniya29|ruck | rlandy, hello | 10:26 |
| ysandeep|rover | o/ | 10:26 |
| rlandy | ysandeep|rover: soniya29|ruck: let's touch base ... | 10:26 |
| ysandeep|rover | ack | 10:27 |
| soniya29|ruck | ack | 10:27 |
| ysandeep|rover | meet.google.com/vnh-ynyb-ghh | 10:28 |
| marios | pojadhav: o/ will check | 10:35 |
| marios | pojadhav: yes i have the same :) https://review.opendev.org/c/openstack/tripleo-ci/+/839518/8#message-23c7677c86b0457399ecc41e41ff97834a3f361b | 10:36 |
| marios | pojadhav: don't know why it is happening as i didn't touch that code but i also think we might just remove that play that is referenced in the error | 10:36 |
| marios | pojadhav: we can discuss in review time in half hour if you're around | 10:36 |
| rlandy | ysandeep|rover: soniya29|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart/+/840917 | 10:39 |
| pojadhav | marios, sure | 10:47 |
| rlandy | soniya29|ruck: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2&skip=0 | 11:03 |
| *** dviroel_ is now known as dviroel\ | 11:14 | |
| *** dviroel\ is now known as dviroel | 11:14 | |
| rlandy | dasm|off: https://review.rdoproject.org/r/c/config/+/42619/1/zuul.d/tripleo-rdo-base.yaml ... this needs to start on c9 check ASAP pls | 11:16 |
| rlandy | not all jobs | 11:16 |
| rlandy | bug still open on this | 11:16 |
| marios | review time? i dropped please someone ping me if the call is still on and you're on it ;) | 11:17 |
| ysandeep|rover | https://bugs.launchpad.net/tripleo/+bug/1961811 | 11:17 |
| rlandy | marios: sorry - we were on rr sync | 11:28 |
| rlandy | marios: can join now | 11:28 |
| marios | rlandy: k joining - 2 mins just got my food from microwave :) | 11:29 |
| *** ysandeep|rover is now known as ysandeep|rover|break | 11:57 | |
| chandankumar | rlandy: meeting time | 12:01 |
| rlandy | chandankumar: on review time - few minutes | 12:01 |
| chandankumar | rlandy: ok | 12:01 |
| *** pojadhav is now known as pojadhav|break | 12:22 | |
| * pojadhav|break stepping out... | 12:22 | |
| rlandy | bhagyashris: the TC/UA/PM sync is a clash for me - can we move it one hour later? | 12:37 |
| rlandy | ysandeep|rover|break: ok - so we did get a legit promo - wallaby c8 on friday with patch change | 12:42 |
| rlandy | will run one last testproject there and we can merge | 12:43 |
| *** ysandeep|rover|break is now known as ysandeep|rover | 12:57 | |
| ysandeep|rover | rlandy, ack | 12:57 |
| dviroel | marios: rlandy: do we still care about that one too: https://review.rdoproject.org/zuul/builds?job_name=mol-get_hash-centos-7&project=rdo-infra/ci-config | 13:03 |
| dviroel | now failing on my refactoring change | 13:03 |
| rlandy | https://review.rdoproject.org/r/c/testproject/+/36255 - let's see how that goes | 13:07 |
| rlandy | w c8 | 13:07 |
| rlandy | dviroel: not resolving | 13:07 |
| rlandy | but my guess is no | 13:08 |
| rlandy | if it c7 | 13:08 |
| marios | dviroel: yeah but probably we want to move it to c8 | 13:08 |
| marios | dviroel: i mean, it is not a bad test to have ansible invocation of get-hash | 13:09 |
| rlandy | marios: ysandeep|rover: chandankumar: frenzy_friday: anyone else .... topics to raise at prod chain council? | 13:10 |
| rlandy | added the vexx isssue | 13:10 |
| rlandy | internal reg | 13:10 |
| ysandeep|rover | rlandy, marios please review https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840735 , tht tox for wallaby is still affected. | 13:10 |
| rlandy | elastic search | 13:11 |
| dviroel | rlandy: marios https://review.rdoproject.org/zuul/builds?job_name=mol-get_hash-centos-7 - removing project from search | 13:13 |
| dviroel | marios: we already have mol-get_hash-centos-8 | 13:14 |
| dviroel | we don't have the c9 version | 13:14 |
| ysandeep|rover | rlandy: prod chain topic - vexx issue - that's all I can think of currently. | 13:14 |
| ysandeep|rover | Tengu: hey o/ could you please +w https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840735 , needed to unblock tht wallaby gate | 13:15 |
| ysandeep|rover | I only have core on ci repos | 13:15 |
| marios | dviroel: ack then kill the 7 one :) | 13:16 |
| marios | dviroel: fine by me | 13:16 |
| dviroel | will propose a patch | 13:17 |
| rlandy | soniya29|ruck: pls add the testproject for the victoria jobs we discussed | 13:17 |
| marios | ysandeep|rover: ack | 13:17 |
| marios | rlandy: just the c7 stuffs happening this week maybe | 13:18 |
| ysandeep|rover | marios, thanks and apologies I missed to discuss this in review time. | 13:18 |
| marios | ysandeep|rover: np | 13:18 |
| Tengu | ysandeep|rover: marios: did it :) | 13:19 |
| ysandeep|rover | marios++ Tengu++ | 13:19 |
| marios | wasnt me | 13:19 |
| *** dasm|off is now known as dasm | 13:27 | |
| dasm | o/ | 13:27 |
| ysandeep|rover | rlandy, cix time | 13:31 |
| soniya29|ruck | rlandy, ack | 13:31 |
| ysandeep|rover | you are here, ignore | 13:32 |
| rlandy | dasm: hello ... pls see comments on your patch to change all OVB to use uefi | 13:32 |
| rlandy | not desired atm | 13:32 |
| dasm | rlandy: ack | 13:32 |
| rlandy | we do need check c9 change ASAP | 13:32 |
| rlandy | failing all check | 13:32 |
| rlandy | can you get that change in? | 13:33 |
| rlandy | dasm: ^^ | 13:33 |
| dasm | rlandy: yes | 13:33 |
| rlandy | ysandeep|rover, has bugs out against that | 13:33 |
| rlandy | ysandeep|rover: hmmm ... not encouraging https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/4a6e217/logs/undercloud/var/log/tempest/stestr_results.html.gz | 13:38 |
| ysandeep|rover | rlandy, i was waiting for this rerun - i think its legit bug.. please see my notes on rr hackmd ->> Wallaby - rerunning failed jobs - fs001 failure may be real neutron bug. | 13:41 |
| ysandeep|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/4a6e217/logs/overcloud-controller-2/var/log/extra/errors.txt.gz | 13:41 |
| ysandeep|rover | TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' | 13:41 |
| ysandeep|rover | rlandy, I will report wallaby failure now | 13:42 |
| rlandy | ysandeep|rover: setup failures in tempest on 17 on rhel-8 | 13:42 |
| rlandy | soniya29|ruck was looking into those | 13:42 |
| rlandy | soniya29|ruck: ^^ | 13:42 |
| rlandy | pls check if we have the same error | 13:42 |
| rlandy | do you have an analysis on that yet? | 13:42 |
| dasm | ysandeep|rover: rlandy please take a look -- CS9 checks with uefi boot https://review.rdoproject.org/r/c/rdo-jobs/+/42673 | 13:43 |
| rlandy | dasm: looks right to me ... https://review.rdoproject.org/r/c/rdo-jobs/+/42673 - pls add testproject to run on those | 13:45 |
| rlandy | and see if we get by the node provision | 13:46 |
| dasm | ysandeep|rover: do you have the bug # handy? | 13:47 |
| dasm | ^ | 13:48 |
| ysandeep|rover | dasm, https://trello.com/c/TpHRXYv5/2366-cixlp1961811tripleociproa-diskfilter-writes-are-not-supported-timeout-waiting-for-provisioned-nodes-to-become-available-in-fs1-w | 13:50 |
| ysandeep|rover | dasm: https://bugs.launchpad.net/bugs/1961811 | 13:50 |
| dasm | thx | 13:50 |
| soniya29|ruck | rlandy, setup failures on which cinder, neutron tests? | 13:56 |
| soniya29|ruck | s/which/what | 13:56 |
| *** pojadhav|break is now known as pojadhav | 13:57 | |
| dasm | will join ooo scrum after rbac meeting | 13:59 |
| bhagyashris | akahat, arxcruz frenzy_friday marios dasm rcastillo pojadhav rlandy chandankumar ysandeep|rover soniya29|ruck | 14:00 |
| bhagyashris | scrum time | 14:00 |
| bhagyashris | dviroel, ^ | 14:00 |
| bhagyashris | jm1, | 14:01 |
| ysandeep|rover | bhagyashris, will join in few.. In prod chain | 14:01 |
| bhagyashris | rlandy, ^ | 14:02 |
| *** soniya29 is now known as soniya29|ruck | 14:06 | |
| ysandeep|rover | slaweq, ykarel we are seeing some tempest failures in wallaby fs001 job, and observed a neutron traceback: ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context' | 14:17 |
| ysandeep|rover | we have reported: https://bugs.launchpad.net/tripleo/+bug/1972660 | 14:18 |
| slaweq | ysandeep|rover: if ykarel will not have time to check it, I can try to look at this tomorrow probably | 14:18 |
| slaweq | now I'm in the meeting | 14:19 |
| ysandeep|rover | slaweq: thanks, nw | 14:19 |
| ysandeep|rover | I will need investigate it more to see if tempest failure and neutron traceback is related. | 14:20 |
| ykarel | ysandeep|rover, slaweq /me will check, traceback atleast seems related to my patch | 14:20 |
| ykarel | but need to dig as that was specifically for port bindings | 14:20 |
| ysandeep|rover | ykarel: ack, thanks | 14:20 |
| chandankumar | dasm: https://review.rdoproject.org/r/q/topic:cs-parenting-ovb | 14:22 |
| ykarel | ysandeep|rover, not seeing those consistently, right? | 14:23 |
| ykarel | if yes, that's a known issue | 14:23 |
| ykarel | me shares link | 14:23 |
| ykarel | https://bugs.launchpad.net/tripleo/+bug/1964940 | 14:24 |
| soniya29|ruck | ykarel, slaweq, we are seeing neutron tempest tests skipping in rhel8 rhos-17 full-tempest-api/full-tempest-sceanario with traceback 'l3_agent_scheduler extensions not enabled', 'network-segment-range extension not enabled.' | 14:24 |
| soniya29|ruck | we have reported the bug here:- https://bugzilla.redhat.com/show_bug.cgi?id=2082632 | 14:25 |
| ysandeep|rover | soniya29|ruck, i think that's expected if a particular plugin is not enabled. | 14:26 |
| jm1 | chandankumar++ vault looks great! | 14:28 |
| soniya29|ruck | ysandeep|rover, so many extensions being disabled is something suspicious, isn't it | 14:30 |
| soniya29|ruck | ysandeep|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-tempest-rhos-17/798e141/logs/undercloud/var/log/tempest/stestr_results.html | 14:31 |
| ysandeep|rover | ykarel: yeah it seems be related: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/4a6e217/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz - nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed | 14:31 |
| ysandeep|rover | ykarel, I got distracted by the neutron failure "ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context'" | 14:31 |
| ysandeep|rover | ykarel, I will mark new one as duplicate | 14:32 |
| ykarel | ysandeep|rover, yes i will check the traceback issue, but as said it's duplicate of others | 14:32 |
| ykarel | soniya29|ruck, ysandeep|rover will also check osp 17 issue | 14:32 |
| ykarel | will comment, would need testtools update after updating tempest | 14:33 |
| soniya29|ruck | ykarel, ack | 14:33 |
| bhagyashris | https://review.opendev.org/c/openstack/tripleo-ci/+/839323 | 14:41 |
| dviroel | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42669 | 14:43 |
| dviroel | https://review.rdoproject.org/r/c/rdo-jobs/+/42674 | 14:43 |
| dasm | arxcruz: seems like bitwarden has an option to add more accounts: https://bitwarden.com/help/account-switching/ | 14:44 |
| arxcruz | dasm cool | 14:44 |
| dasm | arxcruz: but i don't see it working in a browser. maybe that's only mobile/desktop? | 14:45 |
| dasm | id | 14:45 |
| dasm | idk | 14:45 |
| arxcruz | dasm seems to be the case, i'm not seeing option in the browser | 14:45 |
| soniya29|ruck | rlandy, https://review.rdoproject.org/r/c/testproject/+/42676 | 14:47 |
| jm1 | chandankumar, rlandy: auditing and exporting organization vaults is possible with bitwarden. added links to jira card. maybe we need additional permissions on our vault? | 14:48 |
| dasm | jm1: nice. | 14:49 |
| jm1 | chandankumar: or maybe organizational vaults are something different than our shared vault? | 14:49 |
| dasm | jm1: by "organization" it might mean "red hat" which might be too much | 14:49 |
| dasm | but worth checking | 14:49 |
| chandankumar | jm1: yes, may be, I will raise a ticket to helpdesk regarding the same tomorrow. | 14:49 |
| jm1 | dasm: as far as i can see, bitwarden docs only speak of personal vaults and organization vaults. So i guess our shared collection is a organization vault? | 14:51 |
| dasm | maybe | 14:51 |
| *** pojadhav is now known as pojadhav|afk | 14:54 | |
| *** soniya29|ruck is now known as soniya29|ruck|dinner | 14:54 | |
| jm1 | chandankumar++ thanks and thanks for moving aaaaall credentials over! much better (and safer) now :) | 14:55 |
| marios | rlandy: prod chain council cancelled? | 15:01 |
| * dviroel lunch | 15:10 | |
| *** dviroel is now known as dviroel|lunch | 15:10 | |
| ysandeep|rover | rlandy, fyi.. I think the node provisioning issue which we are seeing is still because of vexx infra - general slowness in environment | 15:22 |
| rlandy | ysandeep|rover: k .. | 15:23 |
| ysandeep|rover | from reproducer: http://pastebin.test.redhat.com/1050454 , ssh failed during node provision but worked when I login - looks like ssh took long to be ready | 15:23 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/840917 may promote | 15:23 |
| rlandy | so I will add wallaby.yaml file to the review as well | 15:23 |
| rlandy | when test job completes | 15:24 |
| ysandeep|rover | rlandy, I am going to ping mnaser on #vexxhost to see if they are still seeing high load on compute nodes today. | 15:30 |
| rlandy | ysandeep|rover: yep -pls do | 15:31 |
| rlandy | I owe him feedback | 15:31 |
| rlandy | ysandeep|rover: tell him we're somewhat better | 15:31 |
| rlandy | but still seeing errors | 15:31 |
| *** soniya29|ruck|dinner is now known as soniya29|ruck | 15:36 | |
| rlandy | arxcruz: hey - want to meet now? | 15:37 |
| rlandy | move 1-1 forward? | 15:37 |
| arxcruz | rlandy sure | 15:37 |
| arxcruz | rlandy joining | 15:38 |
| rlandy | joining | 15:39 |
| *** marios is now known as marios|out | 15:50 | |
| ysandeep|rover | rlandy, fyi... the wallaby fs001 issue I have reported today was duplicate of https://bugs.launchpad.net/tripleo/+bug/1964940 | 15:58 |
| ysandeep|rover | ^^ is intermittent, against resurfaced today | 15:58 |
| ysandeep|rover | again* | 15:58 |
| rlandy | yatin's bug? | 15:59 |
| ysandeep|rover | rlandy, yes | 16:00 |
| rlandy | ysandeep|rover: dviroel|lunch: chandankumar: https://review.opendev.org/c/openstack/tripleo-quickstart/+/840917 - I think this can go ... ser current testproject with image build work | 16:00 |
| ysandeep|rover | rlandy, voted +2 | 16:00 |
| rlandy | will submit a separate patch for check release file | 16:01 |
| rlandy | dviroel|lunch: ^^ will send your way | 16:01 |
| rlandy | ysandeep|rover: need anything taken care of in my afternoon? | 16:01 |
| rlandy | ysandeep|rover: what did we do re: jon's comment on fixed dep? | 16:02 |
| rlandy | need to promote 17 on rhel-8? | 16:02 |
| rlandy | rebuild containers? | 16:02 |
| rlandy | still failing atm | 16:02 |
| ysandeep|rover | rlandy: let me rekick the 17 on rhel-8 line, I think last run was before jon fixed it | 16:04 |
| rlandy | ysandeep|rover: hmm ... what time? | 16:04 |
| rlandy | may kick again shortly | 16:04 |
| rlandy | 18:07:13 | 16:05 |
| ysandeep|rover | true.. next run in few hours: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-integration-rhos-17 | 16:05 |
| rlandy | ysandeep|rover; leave it - will kick in two hours | 16:05 |
| rlandy | will watch it | 16:05 |
| rlandy | and leave you notes | 16:05 |
| ysandeep|rover | rlandy: thanks! | 16:06 |
| ysandeep|rover | rlandy, I am out soon, need something? | 16:07 |
| rlandy | ysandeep|rover: no - should be fine | 16:07 |
| rlandy | will try promo w c8 | 16:07 |
| rlandy | will leave master and w c9 | 16:07 |
| rlandy | pls see if you can promo those tomorrow | 16:07 |
| rlandy | and will try 17 on rhel-8 | 16:08 |
| rlandy | the rest should be fine | 16:08 |
| rlandy | soniya29|ruck: any luck on the tempest component investigation? | 16:08 |
| ysandeep|rover | rlandy, yeah i just hope vexx will be in better situation tomorrow :D | 16:09 |
| *** ysandeep|rover is now known as ysandeep|out | 16:09 | |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/841140 | 16:09 |
| rlandy | ysandeep|out: there's always hope | 16:09 |
| rlandy | dviroel|lunch: ^^ check patch | 16:09 |
| rlandy | rebased on integration one | 16:10 |
| ysandeep|out | +2 | 16:10 |
| rlandy | ysandeep|out: have a good night | 16:10 |
| * rlandy lunch - brb | 16:10 | |
| ysandeep|out | thanks o/ see you tomorrow | 16:10 |
| soniya29|ruck | rlandy, i have asked yatin and slaweq to have look on neutron tests skipping | 16:11 |
| *** soniya29|ruck is now known as soniya29|out | 16:12 | |
| rlandy | soniya: thanks | 16:12 |
| dasm | rlandy: fyi, we're going further with cs9 checks. I see them booting up: https://review.rdoproject.org/r/c/rdo-jobs/+/42673 but there are other provision failures. cc ysandeep|out soniya | 16:23 |
| *** dviroel|lunch is now known as dviroel | 16:24 | |
| dviroel | rlandy: ack - checking results and we can merge that | 16:26 |
| dviroel | both* | 16:26 |
| rlandy | dviroel: thanks fs001 in rerun | 16:29 |
| rlandy | but w/o it does not even build images | 16:29 |
| rlandy | https://review.rdoproject.org/zuul/stream/65231d8a1d444a59abff1aee09e0f435?logfile=console.log | 16:30 |
| dviroel | ack - that dependency issue, right | 16:30 |
| rlandy | dviroel: pls also review https://review.rdoproject.org/r/c/rdo-jobs/+/42673/2/zuul.d/ovb-jobs.yaml from dasm | 16:31 |
| rlandy | dasm: if your testproject jobs past nodeprovision we are better off | 16:31 |
| dviroel | ok | 16:31 |
| rlandy | dasm: ^^ let us know when we are ok to merge | 16:31 |
| dasm | dviroel: checking the bug: https://review.rdoproject.org/r/c/testproject/+/28446 wrt ovb issues. Failing logs: https://logserver.rdoproject.org/64/841064/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/d84b9cf/logs/baremetal_2_22143_0-console.log ; logs after introducing uefi | 16:58 |
| dasm | https://logserver.rdoproject.org/74/42374/10/check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/c3828c0/logs/baremetal_10_61290_0-console.log | 16:58 |
| dasm | dviroel: it's working through initial provisioning. | 16:58 |
| dasm | I'm +1 on merging it. | 16:58 |
| dasm | https://review.rdoproject.org/r/c/rdo-jobs/+/42673 | 16:59 |
| dviroel | dasm: ack - will check the patch | 17:15 |
| *** jpena is now known as jpena|off | 17:17 | |
| dasm | thanks. ysandeep|out uefi is enabled for checks ^ | 17:25 |
| rlandy | great | 17:58 |
| rlandy | let's see if that helps | 17:58 |
| rlandy | k - 17 on rhel 8 started | 18:06 |
| rlandy | let's hope | 18:06 |
| dasm | nice | 18:06 |
| rlandy | dasm: pls keep a check on the check jobs for ovb c9 | 18:07 |
| rlandy | let's see the data by tomorrow | 18:07 |
| rlandy | can you leave comments on the rr hackmd by your EoD? | 18:07 |
| rlandy | dasm: ^^ | 18:07 |
| dasm | k | 18:08 |
| rlandy | thank you | 18:10 |
| rlandy | dviroel: dasm: can you access https://etherpad.opendev.org/p/tripleo-zed-ptg-topics? 500 for me | 18:21 |
| rlandy | jm1: mike burns said not to worry about hardware IDs yet - he'll let us know when | 18:22 |
| rlandy | the hardware used in zuul cab fall under zuul codes | 18:22 |
| dasm | the same rlandy | 18:22 |
| dasm | 503 on etherpad | 18:23 |
| rlandy | dasm: thanks for confirming | 18:23 |
| dviroel | https://etherpad.opendev.org/ same here | 18:31 |
| rlandy | anyone need anything from me? | 18:31 |
| rlandy | otherwise going to work on 16.2 base image again | 18:31 |
| rlandy | dasm: dviroel: rcastillo: ^^? need anything? | 18:32 |
| dviroel | i'm fine - will add chances to review etherpad | 18:32 |
| rcastillo | nothing from me either, thanks | 18:33 |
| rlandy | k | 18:33 |
| jm1 | rlandy: ack, thank you! | 19:02 |
| rlandy | chandankumar: hey - pls see pvt for response to email | 19:37 |
| rlandy | dviroel: can you spot check if this makes basic sense: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/406943/2/playbooks/tripleo-rdo-base/container-login-test.yaml | 20:12 |
| rlandy | for two registry logins | 20:13 |
| * rlandy will have to merge to check | 20:13 | |
| * dviroel looking | 20:14 | |
| * rlandy needs to fix linter failure | 20:19 | |
| dviroel | why didn't work in the previous patch? | 20:20 |
| rlandy | dviroel: I added another var | 20:21 |
| rlandy | if it looks basically ok | 20:22 |
| rlandy | I'll merge | 20:22 |
| rlandy | and pray | 20:22 |
| rlandy | and try | 20:22 |
| dviroel | rlandy: can't spot anything wrong, lgtm | 20:27 |
| rlandy | 3 time lucky with spacing | 20:30 |
| * dviroel afk | 20:43 | |
| *** dviroel is now known as dviroel|afk | 20:43 | |
| rcastillo | lunch brb | 20:45 |
| rlandy | k - trying merge | 20:47 |
| rlandy | import httpcore", "ModuleNotFoundError: No module named 'httpcore'", | 20:52 |
| rlandy | ugh | 20:52 |
| rlandy | 17 on rhel 8 | 20:52 |
| dasm | O/ | 20:53 |
| rcastillo | apparently I'm off tomorrow, holiday in Mexico | 21:00 |
| dasm | rcastillo: have a great time then! | 21:04 |
| rlandy | rcastillo: enjoy | 21:51 |
| rlandy | arxcruz: ysandeep|out: fyi ... https://bugzilla.redhat.com/show_bug.cgi?id=2083390 | 21:51 |
| *** rlandy is now known as rlandy|bbl | 22:13 | |
| * dasm is leaving | 22:16 | |
| dasm | have a great afternoon/evening! | 22:16 |
| *** dasm is now known as dasm|off | 22:16 | |
| rcastillo | o/ | 22:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!