| *** jmasud has quit IRC | 00:32 | |
| *** tosky has quit IRC | 00:36 | |
| *** jmasud has joined #oooq | 01:24 | |
| bhagyashri|rover | weshay|ruck, Hi, anything you want me to handover? | 02:46 |
|---|---|---|
| bhagyashri|rover | anything that i can take care ? | 02:47 |
| * bhagyashri|rover looking at mails... | 02:49 | |
| ysandeep|away | bhagyashri|rover, hey good morning! | 03:03 |
| bhagyashri|rover | ysandeep|away, good morning ! | 03:03 |
| ysandeep|away | Do we already have a bug for this failure - https://logserver.rdoproject.org/47/778447/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/1bf601d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 03:03 |
| ysandeep|away | FATAL | Check Keystone internal endpoint status | undercloud | 03:03 |
| *** ysandeep|away is now known as ysandeep | 03:04 | |
| ysandeep | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 03:04 |
| *** jmasud has quit IRC | 03:04 | |
| bhagyashri|rover | ysandeep, i saw this issue yesterday and i was trying to test here https://review.rdoproject.org/r/#/c/29351/ just regression test and yeah now confirmed | 03:05 |
| bhagyashri|rover | so yeah that is issue | 03:05 |
| bhagyashri|rover | ysandeep, i am going to report the bug for that | 03:05 |
| bhagyashri|rover | i9t's hitting fs001 and fs039 | 03:05 |
| bhagyashri|rover | it's | 03:05 |
| ysandeep | bhagyashri|rover, thanks! | 03:06 |
| *** apetrich has quit IRC | 03:12 | |
| bhagyashri|rover | ysandeep, fyi https://bugs.launchpad.net/tripleo/+bug/1918366 | 03:14 |
| openstack | Launchpad bug 1918366 in tripleo "FATAL | Check Keystone project status | undercloud | item=service failing on fs001, fs035 and fs039" [High,Triaged] | 03:14 |
| ysandeep | bhagyashri|rover, fyi.. we have faced something similiar in 16.2 couple of days back.. but we didn't find cause at that time https://bugzilla.redhat.com/show_bug.cgi?id=1926301 .. issue was intermittent and cleared after couple of days | 03:19 |
| openstack | bugzilla.redhat.com bug 1926301 in openstack-keystone "[RHOS 16.2] OVB overcloud deployments are failing with "FATAL | Check Keystone user assignment to roles status ... error"" [Unspecified,Closed: notabug] - Assigned to lbragsta | 03:19 |
| ysandeep | different async ansible tasks fails.. | 03:21 |
| ysandeep | bhagyashri|rover, can i update your test patch to get more debug output? | 03:22 |
| ysandeep | bhagyashri|rover, or can you add a depends on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/775833 on your testproject and run only fs001 | 03:23 |
| *** jmasud has joined #oooq | 03:26 | |
| *** udesale has joined #oooq | 03:29 | |
| ysandeep | bhagyashri|rover, https://review.rdoproject.org/r/#/c/28537/ | 03:29 |
| ysandeep | we can provide debug logs from ^^ on bug, once job reports back.. | 03:30 |
| *** ysandeep is now known as ysandeep|afk | 03:50 | |
| weshay|ruck | 0/ | 04:15 |
| weshay|ruck | hey folks | 04:15 |
| weshay|ruck | FYI.. I'm trying to merge https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315 running tox -re linters locally atm to see what's up | 04:15 |
| *** ykarel has joined #oooq | 04:16 | |
| weshay|ruck | https://d05783eeba6f8fb45435-8e9a9f995b4a6012c53e1bcc573a9fc8.ssl.cf1.rackcdn.com/779315/1/check/openstack-tox-linters/715da5b/job-output.txt | 04:25 |
| weshay|ruck | trying to run this.. | 04:25 |
| weshay|ruck | Looking in indexes: http://mirror.regionone.rdo-cloud.rdoproject.org/pypi/simple, http://mirror.regionone.rdo-cloud.rdoproject.org/wheel/fedora-30-x86_64 | 04:25 |
| weshay|ruck | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7f0b8c208470>: Failed to establish a new connection: [Errno 113] No route to host',)': /pypi/simple/pre-commit/ | 04:25 |
| weshay|ruck | WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7f0b8c208710>: Failed to establish a new connection: [Errno 113] No route to host',)': /pypi/simple/pre-commit/ | 04:25 |
| weshay|ruck | WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection bro | 04:26 |
| weshay|ruck | bhagyashri|rover, ykarel can you please help me merge https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315 | 04:26 |
| ykarel | weshay|ruck, looking | 04:27 |
| weshay|ruck | thanks | 04:27 |
| ykarel | weshay|ruck, also commented on https://bugs.launchpad.net/tripleo/+bug/1918325 | 04:28 |
| openstack | Launchpad bug 1918325 in tripleo "victoria updates fail w/ Problem: package libknet1-compress-bzip2-plugin-1.16-1.el8.x86_64 requires libknet1(x86-64) = 1.16-1.el8, but none of the providers can be installed" [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 04:28 |
| weshay|ruck | ah.. thank you! | 04:28 |
| ykarel | weshay|ruck, okk so there is already victoria promotion for c8-stream? | 04:29 |
| weshay|ruck | ykarel, yes sir | 04:29 |
| ykarel | okk cool | 04:29 |
| weshay|ruck | and we'll shoot for ussuri tomorrow | 04:29 |
| ykarel | okk will clear above issue for ussuri to clear gate | 04:29 |
| weshay|ruck | thank you!! | 04:30 |
| weshay|ruck | I have to go to sleep :) | 04:31 |
| ykarel | yeap have good sleep, will clear that patch | 04:38 |
| *** jmasud has quit IRC | 04:46 | |
| chandankumar | ykarel: does https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315 also fixed https://bugs.launchpad.net/tripleo/+bug/1918325 | 04:56 |
| openstack | Launchpad bug 1918325 in tripleo "victoria updates fail w/ Problem: package libknet1-compress-bzip2-plugin-1.16-1.el8.x86_64 requires libknet1(x86-64) = 1.16-1.el8, but none of the providers can be installed" [Critical,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 04:56 |
| chandankumar | ? | 04:56 |
| *** jmasud has joined #oooq | 04:57 | |
| *** ratailor has joined #oooq | 05:09 | |
| ykarel | chandankumar, looking if that alone is sufficient | 05:09 |
| ykarel | seems issue with nodes used too, centos-8 vs centos8-stream | 05:10 |
| chandankumar | ykarel: ok I will wait for results before merging it | 05:12 |
| bhagyashri|rover | ysandeep|afk, sure | 05:20 |
| *** ysandeep|afk is now known as ysandeep | 05:20 | |
| bhagyashri|rover | weshay|ruck, ack | 05:20 |
| bhagyashri|rover | ysandeep, sorry i was away | 05:21 |
| ysandeep | bhagyashri|rover, nw :) | 05:21 |
| bhagyashri|rover | i saw your patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/775833 | 05:23 |
| ykarel | chandankumar, yes that should clear the issue as libknet1 will be installed from stream repo | 05:34 |
| ykarel | i mean https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315, but let's see how job goes | 05:34 |
| ykarel | added depends-on on https://review.opendev.org/c/openstack/tripleo-heat-templates/+/778832 | 05:37 |
| *** jmasud has quit IRC | 06:02 | |
| *** jmasud has joined #oooq | 06:06 | |
| *** saneax has joined #oooq | 06:09 | |
| chandankumar | ykarel: ack | 06:17 |
| chandankumar | ykarel: content provider victoria job is looking good | 06:17 |
| ykarel | actually content provider don't have issue, issue will be on job where pacemaker get's installed and try to get updated | 06:21 |
| *** marios has joined #oooq | 06:27 | |
| bhagyashri|rover | chandankumar, regarding stein: the quay.io login issue has been resolved | 06:53 |
| bhagyashri|rover | chandankumar, but still it's stuck-ed at some point while pushing the containers on quay.io | 06:54 |
| bhagyashri|rover | chandankumar, i did the manual pull an push of containers on quay.io and it's working fine. but not sure why it's getting stucked through promoter code | 06:55 |
| bhagyashri|rover | arxcruz|off, ^ | 07:00 |
| *** jmasud has quit IRC | 07:02 | |
| *** saneax has quit IRC | 07:08 | |
| *** jmasud has joined #oooq | 07:08 | |
| chandankumar | bhagyashri|rover: at which container it is getting stuck? | 07:17 |
| bhagyashri|rover | chandankumar, earlier it was stuck at nova-compute-ironic | 07:18 |
| bhagyashri|rover | http://10.0.111.180/promoter_logs/container-push/20210310-025400.log | 07:19 |
| bhagyashri|rover | and recently it got stuck at no-api | 07:19 |
| bhagyashri|rover | http://10.0.111.180/promoter_logs/container-push/20210310-045932.log | 07:19 |
| *** jbadiapa has joined #oooq | 07:23 | |
| *** slaweq has quit IRC | 07:42 | |
| *** jpodivin has joined #oooq | 07:43 | |
| *** slaweq has joined #oooq | 07:46 | |
| *** slaweq has quit IRC | 07:52 | |
| *** saneax has joined #oooq | 07:52 | |
| *** slaweq has joined #oooq | 07:54 | |
| *** ysandeep is now known as ysandeep|lunch | 07:58 | |
| *** amoralej|off is now known as amoralej | 08:01 | |
| arxcruz|off | bhagyashri|rover: hey | 08:24 |
| arxcruz|off | bhagyashri|rover: what's the problem? | 08:24 |
| *** arxcruz|off is now known as arxcruz | 08:24 | |
| *** jmasud has quit IRC | 08:30 | |
| *** jmasud has joined #oooq | 08:31 | |
| *** ysandeep|lunch is now known as ysandeep | 08:37 | |
| *** tosky has joined #oooq | 08:46 | |
| bhagyashri|rover | arxcruz, hi, regarding stein: the quay.io login issue has been resolved | 08:46 |
| arxcruz | bhagyashri|rover: \o/ | 08:46 |
| arxcruz | bhagyashri|rover: do you need help with something? | 08:46 |
| bhagyashri|rover | arxcruz, while pushing the containers on quay.io through promoter it's getting stuck-ed at some point | 08:47 |
| bhagyashri|rover | i did the manual pull an push of containers on quay.io and it's working fine. but not sure why it's getting stucked through promoter code | 08:47 |
| arxcruz | bhagyashri|rover: do you have logs ? | 08:48 |
| bhagyashri|rover | http://10.0.111.180/promoter_logs/container-push/20210310-025400.log | 08:48 |
| bhagyashri|rover | http://10.0.111.180/promoter_logs/container-push/20210310-045932.log | 08:48 |
| bhagyashri|rover | arxcruz, yes ^ | 08:48 |
| *** jpena|off is now known as jpena | 08:58 | |
| ykarel | chandankumar, depends-on failed with different error, let's get https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315 +Wed | 09:03 |
| ykarel | commented in patch too | 09:03 |
| *** saneax has quit IRC | 09:09 | |
| *** derekh has joined #oooq | 09:09 | |
| chandankumar | ykarel: done, thanks! | 09:10 |
| ykarel | in meanwhile marios testproject change will also have results | 09:11 |
| *** jmasud has quit IRC | 09:12 | |
| *** apetrich has joined #oooq | 09:15 | |
| sshnaidm | is OVB broken on master? | 09:15 |
| sshnaidm | soniya29|ruck, bhagyashri|rover ^ | 09:16 |
| bhagyashri|rover | sshnaidm, https://bugs.launchpad.net/tripleo/+bug/1918366 | 09:17 |
| openstack | Launchpad bug 1918366 in tripleo "FATAL | Check Keystone project status | undercloud | item=service failing on fs001, fs035 and fs039" [High,Triaged] | 09:17 |
| sshnaidm | bhagyashri|rover, ack | 09:17 |
| bhagyashri|rover | sshnaidm, ysandeep i has pushed patch https://review.rdoproject.org/r/#/c/28537/ to debug the issue | 09:18 |
| sshnaidm | bhagyashri|rover, same for victoria | 09:18 |
| bhagyashri|rover | sshnaidm, i dont see issue in Victoria https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1# | 09:19 |
| sshnaidm | bhagyashri|rover, https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&branch=stable%2Fvictoria | 09:20 |
| ysandeep | bhagyashri|rover, fyi.. https://review.rdoproject.org/r/#/c/28537/ is already running with debug.. last run failed with different error so i have rechecked | 09:20 |
| sshnaidm | bhagyashri|rover, ovb is running in check jobs | 09:20 |
| bhagyashri|rover | sshnaidm, fs001 and fs002 is failing here^ but it passed in the testproject patch https://review.rdoproject.org/r/#/c/32266/ and here | 09:20 |
| bhagyashri|rover | periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria https://review.rdoproject.org/zuul/build/c0578e32687b4ca9bca194eafc7e0d46 : SUCCESS in 3h 39m 03s | 09:20 |
| bhagyashri|rover | sshnaidm, ^ | 09:22 |
| bhagyashri|rover | ysandeep, ack, i see your comment on testproject patch | 09:22 |
| sshnaidm | bhagyashri|rover, last failure is from today: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/779121/ | 09:26 |
| sshnaidm | bhagyashri|rover, probably worth to run OVB job on a real victoria branch in testprojec | 09:26 |
| sshnaidm | bhagyashri|rover, like that: https://review.rdoproject.org/r/#/c/32270/ | 09:29 |
| ysandeep | fyi.. master job reported back - debug logs output https://logserver.rdoproject.org/37/28537/12/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/6b24c0f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 09:30 |
| bhagyashri|rover | sshnaidm, ack , | 09:31 |
| *** ykarel is now known as ykarel|lunch | 10:00 | |
| bhagyashri|rover | ykarel|lunch, chandankumar marios tripleo-ci-centos-8-scenario000-multinode-oooq-container-upgrades-victoria https://zuul.opendev.org/t/openstack/build/2a054733b69742379d0124b3f3ff6b24 : SUCCESS in 1h 37m 17s | 10:03 |
| bhagyashri|rover | we have one success | 10:04 |
| bhagyashri|rover | one failure | 10:04 |
| * bhagyashri|rover lunch brb | 10:04 | |
| soniya29|ruck | kopecmartin, arxcruz, please edit/add the today's agenda for tempest meeting:- https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?both | 10:08 |
| marios | nice thanks bhagyashri|rover | 10:16 |
| chandankumar | sshnaidm: please have a look https://review.rdoproject.org/r/#/c/32271/ when free, thanks! | 10:17 |
| bhagyashri|rover | ysandeep, hi, i think we are hitting with different issue for fs001 right? | 10:34 |
| ysandeep | bhagyashri|rover, async ansible tasks are failing after retries.. failure is not on a particular task | 10:36 |
| marios | hello tripleo-ci please add to your reviews list https://review.opendev.org/q/7746b2e57f546a04231b490143085a1157b14c27 - ussuri is waiting for zuul check (but blocked) the others are good to go please thank you | 10:39 |
| marios | chandankumar: thank you and sorry (#tripleo) | 10:44 |
| bhagyashri|rover | ysandeep, ack, my browser hanged while loading the logs :D | 10:48 |
| ysandeep | bhagyashri|rover, yeah that's a 19MB file :) i wget it locally | 10:49 |
| *** ysandeep is now known as ysandeep|away | 10:50 | |
| *** dtantsur|afk is now known as dtantsur | 11:07 | |
| *** ysandeep|away is now known as ysandeep | 11:09 | |
| *** ykarel|lunch is now known as ykarel | 11:18 | |
| zbr | are we affected by OVH fires? | 11:19 |
| *** udesale_ has joined #oooq | 11:24 | |
| *** udesale has quit IRC | 11:28 | |
| *** saneax has joined #oooq | 11:41 | |
| *** derekh has quit IRC | 11:52 | |
| *** derekh has joined #oooq | 11:54 | |
| bhagyashri|rover | ysandeep, fyi on the on going integration line run it fails with the same issue https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/b6b7abb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:57 |
| *** frenzy_friday has quit IRC | 12:08 | |
| *** frenzy_friday has joined #oooq | 12:08 | |
| *** frenzy_friday has quit IRC | 12:15 | |
| *** frenzy_friday has joined #oooq | 12:15 | |
| *** ykarel_ has joined #oooq | 12:17 | |
| *** ykarel has quit IRC | 12:20 | |
| ysandeep | bhagyashri|rover, ack.. failure of stable branches looks different than the master | 12:30 |
| ysandeep | bhagyashri|rover, the testpatch which sshnaidm posted failed with a different error | 12:31 |
| *** ratailor has quit IRC | 12:31 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| bhagyashri|rover | ysandeep, ack | 12:57 |
| chandankumar | arxcruz: soniya29|ruck akahat tempest meeting https://meet.google.com/zvw-hnds-ncu?authuser=0 | 13:01 |
| akahat | ykarel_, hey.. could you please hold node from this review: https://review.rdoproject.org/r/#/c/32007/, job: tripleo-ci-promotion-staging-single-pipeline-centos-8 and keys: github.com/amolkahat.keys | 13:02 |
| *** amoralej is now known as amoralej|lunch | 13:04 | |
| ykarel_ | akahat, doing | 13:07 |
| chandankumar | https://review.opendev.org/q/topic:%22remove-tempest-container%22+(status:open%20OR%20status:merged) | 13:08 |
| akahat | chandankumar, weshay|ruck bhagyashri|rover whenever you are free: https://review.rdoproject.org/r/#/c/32069/ | 13:11 |
| ysandeep | chandankumar, weshay|ruck soniya29|ruck fyi.. https://bugzilla.redhat.com/show_bug.cgi?id=1937355 i am writing a mail to cix the issue | 13:12 |
| openstack | bugzilla.redhat.com bug 1937355 in openstack-tempest "RHOSP17 Tempest Component line jobs are failing with ImportError: cannot import name 'manager'" [Urgent,New] - Assigned to chkumar | 13:12 |
| chandankumar | kopecmartin: ^^ | 13:13 |
| weshay|ruck | hrm.. thanks ysandeep .. soniya29|ruck fyi.. https://bugzilla.redhat.com/show_bug.cgi?id=1937355 | 13:13 |
| chandankumar | akahat: commented | 13:14 |
| akahat | chandankumar, looking | 13:14 |
| ykarel_ | akahat, requested node hold | 13:15 |
| *** ykarel_ is now known as ykarel | 13:15 | |
| bhagyashri|rover | weshay|ruck, frenzy_friday jfyi, testproject patch https://review.rdoproject.org/r/#/c/29562/ for ussuri failing jobs | 13:20 |
| akahat | chandankumar, responded | 13:21 |
| ykarel | akahat, zuul@38.102.83.248 | 13:23 |
| *** jpodivin has quit IRC | 13:24 | |
| kopecmartin | chandankumar: i commented in the BZ, they need to update the neutron-tempest-plugin | 13:26 |
| soniya29|ruck | weshay|ruck, ack | 13:26 |
| weshay|ruck | soniya29|ruck, nice job on the program call | 13:27 |
| akahat | ykarel, thanks. i'm in. :) | 13:27 |
| weshay|ruck | ysandeep, ok.. getting on the prmoter to force bm 17 | 13:27 |
| soniya29|ruck | weshay|ruck, thanks :) | 13:27 |
| ysandeep | weshay|ruck, thanks! | 13:28 |
| *** jpodivin has joined #oooq | 13:29 | |
| *** skramaja has quit IRC | 13:32 | |
| weshay|ruck | frenzy_friday, stat prepping for ussuri centos-8-stream please | 13:35 |
| ysandeep | weshay|ruck, as per kopecmartin this is the fix https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/767715 for tempest bug: https://bugzilla.redhat.com/show_bug.cgi?id=1937355, we need force promote of network component also | 13:38 |
| openstack | bugzilla.redhat.com bug 1937355 in python-neutron-tests-tempest "RHOSP17 Tempest Component line jobs are failing with ImportError: cannot import name 'manager'" [Urgent,New] - Assigned to skaplons | 13:38 |
| ysandeep | http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17/component/network/current-tripleo/ | 13:38 |
| ysandeep | [ ]python-neutron-tests-tempest-1.2.1-0.20210115183004.5ff3a43.el8ost.src.rpm2021-01-15 13:38302K | 13:38 |
| *** jpena|lunch is now known as jpena | 13:40 | |
| weshay|ruck | ykarel++ thanks for picking up that patch :) | 13:41 |
| weshay|ruck | frenzy_friday, hello? | 13:47 |
| frenzy_friday | weshay|ruck, yep, I'll start the ussuri patches in around 30 mins | 13:48 |
| weshay|ruck | frenzy_friday, k.. marios fyi ^ we'll try to get stream going today for ussuri to resolve the update/upgrade errors there | 13:48 |
| marios | weshay|ruck: ack | 13:49 |
| weshay|ruck | marios, should be pretty straight forward https://review.rdoproject.org/r/#/c/32159/ | 13:50 |
| marios | weshay|ruck: ack voted on https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555 | 13:54 |
| weshay|ruck | frenzy_friday, ya.. let's go 4 merge on https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555 ussuri is not imported to internal.. so why not :) | 13:56 |
| weshay|ruck | FYI... folks.. when victoria moves to c8-stream upstream... I expect some timeouts as mirrors/proxies are populated | 13:57 |
| frenzy_friday | weshay|ruck, should we waith for https://review.rdoproject.org/r/#/c/29562/ to finish? | 13:57 |
| weshay|ruck | frenzy_friday, that's on c8 | 13:58 |
| *** amoralej|lunch is now known as amoralej | 13:59 | |
| frenzy_friday | weshay|ruck, oh, ok! Then should I recheck this first https://review.rdoproject.org/r/#/c/32159/ ? | 14:00 |
| sshnaidm | frenzy_friday, weshay|ruck fyi ocv jobs are broken in victoria and master | 14:03 |
| sshnaidm | s/ocv/ovb/ | 14:03 |
| sshnaidm | so don't expect them to pass | 14:03 |
| sshnaidm | jpena, seems like we have same problem on ovb that we had in upstream, neutron db sync takes more than 12 minutes: https://logserver.rdoproject.org/14/759414/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/ba2bd21/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz | 14:07 |
| ysandeep | sshnaidm, master bug could be also related to slow infra.. https://bugs.launchpad.net/tripleo/+bug/1918366 if not some ansible async task shenanigan | 14:16 |
| openstack | Launchpad bug 1918366 in tripleo "FATAL | Check Keystone project status | undercloud | item=service failing on fs001, fs035 and fs039" [High,Triaged] | 14:16 |
| ysandeep | weshay|ruck, Could you please also promote network component for 17 to clear tempest line issues when you have time.. fyi.. I am analysing other failing jobs as well.. details here https://hackmd.io/YW31LNNFQSm7PZKbYJZRzQ#17-Component-line-failing-jobs | 14:21 |
| marios | grateful for sanity check and votes @ master https://review.opendev.org/c/openstack/os-collect-config/+/779757 victoria https://review.opendev.org/c/openstack/os-collect-config/+/779758 ussuri https://review.opendev.org/c/openstack/os-collect-config/+/779759 (fix issue with tags @ https://bugs.launchpad.net/os-collect-config/+bug/1918311 ) thank you for your reviews | 14:21 |
| openstack | Launchpad bug 1918311 in os-collect-config " Multiple tags on the same commit break PBR " [Undecided,In progress] - Assigned to Marios Andreou (marios-b) | 14:21 |
| sshnaidm | ysandeep, yeah, probably | 14:34 |
| sshnaidm | isn't https://bugs.launchpad.net/tripleo/+bug/1918366 escalated to CIX yet? | 14:40 |
| openstack | Launchpad bug 1918366 in tripleo "FATAL | Check Keystone project status | undercloud | item=service failing on fs001, fs035 and fs039" [High,Triaged] | 14:40 |
| bhagyashri|rover | sshnaidm, https://trello.com/c/YYXA7Npb/1863-cixlp1918366tripleociproa-fatal-check-keystone-project-status-undercloud-itemservice-failing-on-fs001-fs035-and-fs039 | 14:41 |
| bhagyashri|rover | sshnaidm, fyi i have updated chat in the comments | 14:42 |
| sshnaidm | bhagyashri|rover, ack, thanks | 14:42 |
| sshnaidm | bhagyashri|rover, I'll attend cix today if need update | 14:43 |
| bhagyashri|rover | sshnaidm, sure thanks :) | 14:43 |
| chandankumar | frenzy_friday: can we also send a seperate patch https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/ussuri.yml for moving to c8 stream | 14:48 |
| chandankumar | so we donot forgot it | 14:48 |
| frenzy_friday | the other release file right? yes, updating that now | 14:48 |
| frenzy_friday | https://hackmd.io/9Xve-rYpRaKbk5NMe7kukw?both | 14:48 |
| chandankumar | frenzy_friday: yes | 14:49 |
| weshay|ruck | ysandeep, ok w/ bm promotion now? | 14:54 |
| weshay|ruck | bhagyashri|rover, soniya29|ruck fyi.. when logging bugs.. it's very useful to paste in the /var/log/extra/errors.log for others reference | 14:55 |
| bhagyashri|rover | weshay|ruck, ack | 14:56 |
| ysandeep | weshay|ruck, yes thanks! for promoting bm.. | 14:56 |
| ysandeep | weshay|ruck, fyi.. i am seeing duplicate entries in grafana https://paste.opensuse.org/10026415 http://tripleo-cockpit.usersys.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?viewPanel=483&orgId=1 , want to bring into your notice.. incase it is not known | 15:02 |
| weshay|ruck | ysandeep, ya... I've seen it | 15:02 |
| ysandeep | ack | 15:02 |
| *** baha has joined #oooq | 15:09 | |
| *** ysandeep is now known as ysandeep|away | 15:11 | |
| soniya29|ruck | weshay|ruck, ack | 15:27 |
| weshay|ruck | bhagyashri|rover, speak to ysandeep|away re: baremetal | 15:34 |
| weshay|ruck | it may make sense to try an upstream job on real baremetal | 15:34 |
| weshay|ruck | to compare against ovb | 15:34 |
| weshay|ruck | soniya29|ruck, can you dig into https://trello.com/c/7GY5Jvrn/1864-cixbz1937355componentosp17rhel8tempestrhosp17-tempest-component-line-jobs-are-failing-with-importerror-cannot-import-name-manage | 15:35 |
| bhagyashri|rover | weshay|ruck, ack | 15:35 |
| soniya29|ruck | weshay|ruck, sure | 15:35 |
| chandankumar | weshay|ruck: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/767715 will fix the issue | 15:36 |
| frenzy_friday | weshay|ruck, updated https://hackmd.io/9Xve-rYpRaKbk5NMe7kukw?both with ussuri and train checklists | 15:48 |
| weshay|ruck | chandankumar, which issue? | 15:50 |
| sshnaidm | jpena, wrt https://bugs.launchpad.net/tripleo/+bug/1918366 - do you need anything else for opening the ticket for vexx? | 15:55 |
| openstack | Launchpad bug 1918366 in tripleo "FATAL | Check Keystone project status | undercloud | item=service failing on fs001, fs035 and fs039" [High,Triaged] | 15:55 |
| *** jpodivin has quit IRC | 15:57 | |
| jpena | sshnaidm: we'd need some clue of where they should start looking at. For example, when it started happening, if there's some correlation between the failed jobs (like running on the same host_id)... | 15:59 |
| chandankumar | frenzy_friday: https://review.rdoproject.org/r/#/c/32276/1/zuul.d/integration-pipeline-ussuri.yaml what about scenario jobs? | 15:59 |
| jpena | sshnaidm: if we just send them a list of tripleo logs, it'll be quite hard to get an answer | 15:59 |
| *** derekh has quit IRC | 16:00 | |
| frenzy_friday | chandankumar, for now we are moving only the buildimage and push jobs to c8 stream node, the rest have only stream content, not node. I think after train is also merged with stream content we would change label in nodeset to move all c8 to c8 stream - or something like that | 16:02 |
| *** derekh has joined #oooq | 16:03 | |
| sshnaidm | jpena, I think they have an experience of investigating it.. | 16:03 |
| *** jpodivin has joined #oooq | 16:03 | |
| sshnaidm | jpena, I can extract host0-ids, but anyone can take it from jobs logs | 16:03 |
| chandankumar | frenzy_friday: ok, thanks! | 16:03 |
| chandankumar | bhagyashri|rover: soniya29|ruck please keep an eye on ussuri promotion if it explodes approving https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555 | 16:05 |
| *** ysandeep|away is now known as ysandeep | 16:08 | |
| ysandeep | weshay|ruck, soniya29|ruck https://trello.com/c/7GY5Jvrn/1864-cixbz1937355componentosp17rhel8tempestrhosp17-tempest-component-line-jobs-are-failing-with-importerror-cannot-import-name-manage will be fixed by network component force promotion for 17 line, weshay|ruck i think you missed my earlier pings. | 16:09 |
| *** jpodivin has quit IRC | 16:10 | |
| ysandeep | http://eavesdrop.openstack.org/irclogs/%23oooq/%23oooq.2021-03-10.log.html#t2021-03-10T13:38:40 | 16:10 |
| weshay|ruck | pojadhav, http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?editPanel=398&orgId=1 | 16:16 |
| *** udesale_ has quit IRC | 16:18 | |
| ysandeep | bhagyashri|rover, weshay|ruck internal gerrit is currently down, we cannot try master on real baremetal till its back.. But we haven't tried upstream master on real baremetal from long, so we can expect to hit other issues.. | 16:18 |
| weshay|ruck | pojadhav, https://zuul.openstack.org/api/builds | 16:19 |
| weshay|ruck | you'll see a [ as the first character | 16:19 |
| weshay|ruck | pojadhav, ^ | 16:19 |
| weshay|ruck | ysandeep, hey.. pooja is going to explore why we're getting duplicate zuul entries | 16:25 |
| ysandeep | weshay|ruck, pojadhav++ thanks! | 16:26 |
| *** marios is now known as marios|out | 16:37 | |
| *** marios|out has quit IRC | 16:43 | |
| weshay|ruck | frenzy_friday, should we chat? | 16:45 |
| frenzy_friday | weshay|ruck, yes sure https://meet.google.com/ycc-gtvs-mim | 16:46 |
| weshay|ruck | frenzy_friday, https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315/ | 16:47 |
| *** derekh has quit IRC | 17:02 | |
| *** derekh has joined #oooq | 17:02 | |
| ysandeep | chandankumar, https://code.engineering.redhat.com/gerrit/#/c/230433/ | 17:03 |
| soniya29|ruck | ysandeep, ack | 17:06 |
| *** jmasud has joined #oooq | 17:08 | |
| weshay|ruck | chandankumar, can you merge https://review.rdoproject.org/r/#/c/32276/ | 17:14 |
| weshay|ruck | I don't have +2 | 17:14 |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| weshay|ruck | sshnaidm, ^ | 17:15 |
| weshay|ruck | sshnaidm, also fyi https://review.rdoproject.org/r/#/c/32271/ | 17:16 |
| *** jmasud has quit IRC | 17:20 | |
| ysandeep | weshay|ruck, please add https://code.engineering.redhat.com/gerrit/#/c/230433/ in your review list.. | 17:22 |
| weshay|ruck | sshnaidm, https://review.rdoproject.org/r/#/c/32276/ | 17:29 |
| *** jmasud has joined #oooq | 17:33 | |
| *** amoralej is now known as amoralej|off | 17:36 | |
| weshay|ruck | frenzy_friday, ok.. https://review.rdoproject.org/r/#/c/32276/ is merging soon | 17:39 |
| weshay|ruck | bhagyashri|rover, soniya29|ruck fyi.. stable/victoria is now on centos-8-stream, promoter is on for master && stable/victoria.. today I am working on stable/ussuri | 17:40 |
| *** ykarel has quit IRC | 17:50 | |
| *** saneax has quit IRC | 17:53 | |
| ysandeep | weshay|ruck, bhagyashri|rover https://code.engineering.redhat.com/gerrit/#/c/230451/ to test master on real baremetal.. will see how it goes.. I don't expect it will straight away work.. last we tested was on c7 node.. a year ago.. | 17:55 |
| weshay|ruck | ysandeep, sounds like sshnaidm may not have needed it.. so before you do a ton of work... sync w/ him | 17:56 |
| sshnaidm | weshay|ruck, ysandeep well, worth to see the results anyway | 17:57 |
| ysandeep | weshay|ruck, ack | 17:57 |
| *** jpena is now known as jpena|off | 17:58 | |
| ysandeep | sshnaidm, I am off for the day, please feel free to update ^^ if needed | 18:00 |
| sshnaidm | ysandeep, sure | 18:00 |
| * ysandeep leaves for the days.. fyi.. tomorrow is public Holiday in India. | 18:00 | |
| ysandeep | sshnaidm, fyi.. this is last green run from a year ago: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-7-bm_envE-3ctlr_1comp-featureset001-master , testpatch: https://code.engineering.redhat.com/gerrit/#/c/190672/6/.zuul.yaml | 18:01 |
| sshnaidm | promising.. | 18:01 |
| *** ysandeep is now known as ysandeep|holiday | 18:03 | |
| *** chem has quit IRC | 18:07 | |
| *** chem has joined #oooq | 18:16 | |
| *** dsneddon has quit IRC | 18:45 | |
| *** dsneddon has joined #oooq | 18:52 | |
| *** jmasud has quit IRC | 19:18 | |
| *** dpawlik6 has joined #oooq | 19:34 | |
| *** jmasud has joined #oooq | 19:35 | |
| *** beagles has quit IRC | 19:36 | |
| *** tosky_ has joined #oooq | 19:37 | |
| *** dsneddon_ has joined #oooq | 19:38 | |
| *** dsneddon has quit IRC | 19:42 | |
| *** tosky has quit IRC | 19:42 | |
| *** mjturek has quit IRC | 19:42 | |
| *** baha has quit IRC | 19:42 | |
| *** dpawlik has quit IRC | 19:42 | |
| *** dpawlik6 is now known as dpawlik | 19:42 | |
| *** irclogbot_2 has quit IRC | 19:44 | |
| *** tosky_ is now known as tosky | 20:16 | |
| *** jmasud has quit IRC | 20:23 | |
| *** jmasud has joined #oooq | 20:32 | |
| *** jmasud has quit IRC | 20:46 | |
| *** tosky has quit IRC | 20:49 | |
| *** tosky has joined #oooq | 20:49 | |
| *** jmasud has joined #oooq | 20:52 | |
| *** derekh has quit IRC | 20:52 | |
| *** jmasud has quit IRC | 21:08 | |
| *** dsneddon_ has quit IRC | 21:27 | |
| *** saneax has joined #oooq | 21:40 | |
| *** saneax has quit IRC | 21:46 | |
| *** johnsom has quit IRC | 21:46 | |
| *** saneax has joined #oooq | 21:46 | |
| *** johnsom has joined #oooq | 21:47 | |
| *** slaweq has quit IRC | 21:55 | |
| *** jmasud has joined #oooq | 22:13 | |
| *** saneax has quit IRC | 22:17 | |
| *** dmellado_ has joined #oooq | 23:47 | |
| *** dmellado has quit IRC | 23:48 | |
| *** dmellado_ is now known as dmellado | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!