| *** dviroel|afk is now known as dviroel | 00:26 | |
| rcastillo|rover | still timing out :/ | 00:44 |
|---|---|---|
| rlandy|bbl | I'm out | 01:11 |
| *** rlandy|bbl is now known as rlandy|out | 01:11 | |
| ysandeep|out | rcastillo|rover, rlandy|out so we have a new issue with tox jobs? | 01:44 |
| ysandeep|out | I see you are modifying marios patch to disable voting on undercloud job to include a change in tox.ini | 01:46 |
| ysandeep|out | Have we reported any bug for tox job failures? I don't see anything in the latest hackmd | 01:47 |
| *** ysandeep|out is now known as ysandeep | 01:53 | |
| ysandeep | I believe you were trying to solve "TypeError: 'NoneType' object is not iterable" initially | 02:00 |
| ysandeep | seen here: https://5e6d48fe25fecf1b2eda-cc994f454b3da94a07b55a097da7db60.ssl.cf5.rackcdn.com/852880/3/check/openstack-tox-linters/0abde65/job-output.txt | 02:00 |
| ysandeep | but now openstack-tox-py39 are timing out | 02:00 |
| ysandeep | I agree with dviroel here: https://review.opendev.org/c/openstack/tripleo-ci/+/852880/4#message-cb3ee44d565d7570c713adc8b5ee61581010d5fb that a separate patch would be nice | 02:01 |
| ysandeep | and I don't find any reported bug for tox issue: https://launchpad.net/tripleo | 02:01 |
| ysandeep | Let me remove tox.ini from marios patch, if linters failed - report a bug and propose a separate patch for linters issue | 02:02 |
| ysandeep | interesting, now linters passed on marios patch even without tox.ini changes | 02:37 |
| ysandeep | good news.. https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html seems fixed | 02:43 |
| * ysandeep let me trigger undercloud job to confirm if its fixed actually | 02:44 | |
| * ysandeep updated bz and hackmd with ^^ above info | 03:11 | |
| *** ysandeep is now known as ysandeep|breakfast | 03:14 | |
| *** ysandeep|breakfast is now known as ysandeep|afk | 03:35 | |
| jm1 | happy friday #oooq | 05:28 |
| chandankumar | jm1: Happy friday o/ | 05:29 |
| chandankumar | ysandeep|afk: o/ you are around ? | 05:29 |
| marios | ysandeep|afk: o/ morning - thanks for sorting out the tox issue on https://review.opendev.org/c/openstack/tripleo-ci/+/852880 | 05:32 |
| *** ysandeep|afk is now known as ysandeep | 05:35 | |
| ysandeep | marios, chandankumar jm1 good morning | 05:35 |
| chandankumar | ysandeep: marios jm1 good morning | 05:36 |
| ysandeep | marios, we don't need https://review.opendev.org/c/openstack/tripleo-ci/+/852880 anymore.. mirror issue is solved now | 05:36 |
| ysandeep | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&project=openstack/tripleo-heat-templates is passing now | 05:36 |
| chandankumar | ysandeep: thank you keeping an eye on undercloud upgrade issue | 05:36 |
| ysandeep | chandankumar, my pleasure :D | 05:37 |
| ysandeep | marios, could you please add https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790 in your review list.. to clear master promotion | 05:37 |
| reviewbot | I have added your review to the Review list | 05:37 |
| * ysandeep updating openstack-dicuss email thread for upgrade job | 05:40 | |
| chandankumar | marios: we need to recheck this https://review.rdoproject.org/r/c/testproject/+/36254/ last run failed at overcloud deploy step's Check Keystone project status due to 500 error, for testing https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790 | 05:41 |
| marios | chandankumar: ysandeep: jm1: o/ | 05:42 |
| marios | ysandeep: thanks i saw update from chandankumar couple minutes ago | 05:42 |
| marios | great i mean better if the issue was resolved but >24 hours wew | 05:43 |
| marios | ysandeep: ack checking ripleo-heat-templates/+/852790 (we were waiting on the test is it ok now/ looking) | 05:44 |
| marios | ysandeep: still not green on the test ack chandankumar commented just now about that (so holding merge ysandeep until we get at least up to the failing step) | 05:47 |
| ysandeep | ah okay, I was not aware we were waiting for tests, I thought we are holding +w because of zuul -1 | 05:47 |
| marios | ysandeep: np see https://bugs.launchpad.net/tripleo/+bug/1984184/comments/5 (comment from yesterday same situation ) | 05:47 |
| marios | chandankumar: want to remove your vote @ https://review.opendev.org/c/openstack/tripleo-ci/+/852880 i am going to abandon | 05:49 |
| marios | chandankumar: i can remove it apparently :D | 05:50 |
| * marios deletes all the votes \o/ | 05:50 | |
| ysandeep | thanks for the additional context, /me rechecked the testproject - lets see how it goes | 05:50 |
| ysandeep | fips jobs is non-voting, not sure why dviroel this one as promotion-blocker, https://bugs.launchpad.net/tripleo/+bug/1984237 - may be to get security folks attention. | 06:16 |
| ysandeep | marked* this one | 06:16 |
| marios | ysandeep: yes we didn't get involved in that one at all i think its as you say for cix/tracking but not on ruck|rover just for FYI | 06:17 |
| ysandeep | marios, thanks! | 06:18 |
| marios | np | 06:19 |
| chandankumar | ysandeep: on fips one, I have just rechecked the dviroel review. there was no logs in the last run | 06:32 |
| ysandeep | chandankumar, thanks! | 06:33 |
| ysandeep | chandankumar, I have checked all upstream bugs - We can't do much atm - need to wait for testproject for neutron tht fix. | 06:33 |
| chandankumar | ysandeep: ack | 06:33 |
| ysandeep | chandankumar, I have some ideas about how we can workaround downstream, but I am doubtful about something.. Do you have few mins to discuss. | 06:34 |
| ysandeep | basically enabling config-drive for downstream ovb jobs | 06:34 |
| chandankumar | ysandeep: please | 06:35 |
| chandankumar | yes | 06:35 |
| chandankumar | give me 5 mins | 06:36 |
| ysandeep | ack | 06:36 |
| chandankumar | need to bring my earphone | 06:36 |
| marios | ysandeep: o/ check pvt replied | 06:38 |
| chandankumar | back now | 06:38 |
| ysandeep | chandankumar, meet.google.com/sxt-jrfh-cgc | 06:41 |
| marios | chandankumar: ysandeep: let me know if you want any clarifications about any of the things i can join | 06:46 |
| ysandeep | marios, thanks! | 06:55 |
| ysandeep | chandankumar, https://logserver.rdoproject.org/78/78a0506b67463f18e0650c2602387164bb92fe30/openstack-periodic-integration-stable1-cs8/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/5d085bf/logs/undercloud/var/log/extra/disk.txt.gz | 06:55 |
| ysandeep | vs https://logserver.rdoproject.org/55/36255/106/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/dc9700b/logs/undercloud/var/log/extra/disk.txt.gz | 06:55 |
| ysandeep | https://review.rdoproject.org/r/c/config/+/33831 | 06:56 |
| ysandeep | chandankumar, https://docs.openstack.org/nova/queens/user/config-drive.html | 07:01 |
| *** ysandeep is now known as ysandeep|lunch | 07:31 | |
| marios | ysandeep|lunch: fyi maybe that is why the bug was fixed https://bugs.launchpad.net/tripleo/+bug/1984175/comments/18 they switched the mirrors amoralej|off patch | 08:34 |
| *** ysandeep|lunch is now known as ysandeep | 08:34 | |
| marios | ysandeep: or both things happened | 08:34 |
| marios | :) | 08:34 |
| marios | mirror was fixed. and we switched to rax anyway | 08:34 |
| marios | ysandeep: i updated the comment with [EDIT] | 08:36 |
| ysandeep | wget -qO- http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | grep NetworkManager-ovs | sed 's/.*href="\([^"]*\)".*/\1/' now have NetworkManager-ovs-1.39.12-1.el9.x86_64.rpm | 08:36 |
| ysandeep | ^^ so to me looks like fb should fixed the mirror as well. | 08:36 |
| marios | ack yes i see ysandeep | 08:36 |
| ysandeep | also fs002 wallaby passed periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby https://review.rdoproject.org/zuul/build/ffe38d9ed80e4a9ba58d32d6920ab138 : SUCCESS in 2h 06m 46s | 08:37 |
| ysandeep | So i think i can close another bug, just waiting for master fs002 green run too | 08:37 |
| marios | ack yes thanks ysandeep happy now at least we captured all the info there | 08:43 |
| marios | like switching the mirrors to rax is a significant thing we should make sure it is called out there that actually happened too regardless if the mirror was also fixed | 08:44 |
| marios | ysandeep: yes it was two duplicate bugs there should be fixed by the same thing | 08:44 |
| marios | and associated cix cards | 08:44 |
| marios | \o/ | 08:44 |
| marios | :_ | 08:44 |
| ysandeep | marios, https://bugs.launchpad.net/tripleo/+bug/1984184 - https://review.rdoproject.org/r/c/testproject/+/36254/148,edit#message-0f0c699b43bfd1146e68017993fca0669ccfdaef reported, fs001 passed / fs035 failed with unrelated issues | 08:46 |
| * ysandeep rekicking fs035 again | 08:46 | |
| ysandeep | last run for master looks great! https://review.rdoproject.org/zuul/buildset/b73491013ad740bfa17d09af6ea54040 | 08:49 |
| ysandeep | just waiting on fs002/35/64 | 08:49 |
| marios | ysandeep: cool... fs35 doesn't need to pass just get to the setup tempest step | 08:50 |
| ysandeep | marios, yeah hopefully soon | 08:50 |
| marios | afk for bit should be <1 hour o/ | 09:29 |
| ysandeep | https://bugs.launchpad.net/tripleo/+bug/1984453 closed and cix updated - Not marking cix as done incase infra want to discuss setting RH own mirror. | 10:01 |
| ysandeep | folks o/ stepping out for sometime, Need to drop my Aunt to her place.. will be back in ~1.5 hr. | 10:04 |
| *** ysandeep is now known as ysandeep|bbl | 10:05 | |
| chandankumar | ysandeep|bbl: last night we move cs8 wallaby component line on IBM cloud, there were few node failure, i have fired a testproject to re-run them https://review.rdoproject.org/r/c/testproject/+/44522 | 11:02 |
| *** ysandeep|bbl is now known as ysandeep | 11:16 | |
| ysandeep | chandankumar, ack, thanks! | 11:16 |
| ysandeep | wohoo.. fs035 passed the earlier failing step: Ensure private network exists | 11:22 |
| ysandeep | https://review.rdoproject.org/zuul/stream/840ac77c7085429cb6b0b7da9059d7d3?logfile=console.log | 11:22 |
| ysandeep | in tempest run | 11:22 |
| chandankumar | ysandeep: yes, I think we can merge that patch | 11:23 |
| ysandeep | chandankumar, I agree with you.. we need marios to +w that | 11:24 |
| ysandeep | marios, fs035 in tempest now: https://review.rdoproject.org/zuul/stream/840ac77c7085429cb6b0b7da9059d7d3?logfile=console.log , passed the earlier failing step | 11:24 |
| ysandeep | folks o/ sc010 which run in rdo is stable nowadays.. https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master&skip=0 | 11:28 |
| ysandeep | do we still need internal sc010 job? | 11:28 |
| ysandeep | I am thinking about removing from criteria.. because downstream infra not in great shape and that will block promotion. | 11:29 |
| chandankumar | ysandeep: sounds good about removing from criteria currently, but i would suggest keep the job running internally | 11:30 |
| marios | ysandeep: great need me to vote/workflow? | 11:30 |
| ysandeep | marios, yes please | 11:30 |
| marios | ysandeep: k | 11:30 |
| ysandeep | marios++ thanks! | 11:31 |
| marios | np ysandeep thanks for watching it | 11:32 |
| marios | and chandankumar | 11:32 |
| marios | o/ | 11:32 |
| marios | \o/ | 11:32 |
| ysandeep | chandankumar, ack for keep it running | 11:32 |
| chandankumar | \o/ | 11:32 |
| marios | :D | 11:32 |
| marios | ysandeep: can we keep the job definition though? | 11:33 |
| marios | ysandeep: are you talkinga bout removing it from downstream line? | 11:33 |
| marios | ysandeep: because it is useful to have for rerun when we have issue in rdo for that job | 11:33 |
| marios | ah right you are talking about criteria | 11:33 |
| marios | so fine i think | 11:33 |
| ysandeep | marios, yes criteria for now but we can follow up to remove later.. I don't see a reason to keep job running in both places if rdo one is stable. | 11:34 |
| marios | ack ysandeep sounds sane to me | 11:35 |
| marios | long as we can run if needed | 11:36 |
| ysandeep | periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master https://review.rdoproject.org/zuul/build/840ac77c7085429cb6b0b7da9059d7d3 : SUCCESS in 3h 03m 32 :) | 11:54 |
| * marios prepares multiple sacrifices for zuul | 11:54 | |
| ysandeep | interesting, internal sc010 fails on Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory" but passes in rdo | 11:56 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/103/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/d907207/logs/undercloud/home/zuul/ansible.log | 11:56 |
| ysandeep | ahh, probably because of missing catatonit.. (it’s only a weak dependency for podman). | 11:58 |
| ysandeep | https://unix.stackexchange.com/questions/619212/podman-run-with-init-gives-me-error-container-init-binary-not-found-on-the-h | 11:58 |
| ysandeep | I see catatonit not installed in internal job: https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/103/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/d907207/logs/undercloud/var/log/extra/package-list-installed.txt | 11:59 |
| ysandeep | but present here: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master/97aa4c5/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 11:59 |
| ysandeep | noting: we need to check with #rdo folks (may be amoralej|off ) on monday on why catatonit is a weak dependency of podman: causing https://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/103/check/r/d907207/logs/undercloud/home/zuul/ansible.logs | 12:12 |
| ysandeep | good news.. we will promote master today.. we have two candidates now - just waiting for few jobs.. | 12:54 |
| * ysandeep bbl to check results | 12:54 | |
| *** ysandeep is now known as ysandeep|afk | 12:55 | |
| chandankumar | good | 13:01 |
| marios | great ysandeep|afk finally | 13:04 |
| marios | wicked wicked | 13:11 |
| marios | jungle is massive | 13:11 |
| *** ysandeep|afk is now known as ysandeep | 13:52 | |
| ysandeep | wohooo master should promote now for hash.. rr --release master --aggregate_hash tripleo-ci-testing/12/0b/120b06809f1abde369d06cef81ded6ab | 14:03 |
| * ysandeep out | 14:03 | |
| ysandeep | marios, chandankumar have a great weekend! | 14:03 |
| *** ysandeep is now known as ysandeep|out | 14:04 | |
| chandankumar | ysandeep: great work on promoting master | 14:04 |
| chandankumar | ysandeep|out: you too :-) | 14:04 |
| marios | chandankumar: ysandeep|out: have a good one folks o/ | 14:04 |
| marios | take care | 14:04 |
| marios | ah we got master chandankumar ? | 14:04 |
| marios | \o/ | 14:04 |
| chandankumar | marios: yes | 14:05 |
| marios | not yet in grafana but cool \o/ | 14:05 |
| marios | nice one ysandeep|out | 14:05 |
| chandankumar | marios: sorry will promote soon | 14:05 |
| marios | chandankumar: ack | 14:05 |
| * marios prepares backup zuul sacrifices | 14:06 | |
| dviroel | ysandeep|out: chandankumar: i added fips as promotion-blocker just to create the CIX card, we want that card on Degraded Column | 14:29 |
| dviroel | not a blocker | 14:29 |
| marios | dviroel: all good discussed with ysandeep|out earlier | 14:29 |
| marios | o/ | 14:30 |
| dviroel | nice | 14:30 |
| dviroel | ok, back to recharge o/ | 14:31 |
| *** dviroel is now known as dviroel|out | 14:31 | |
| * marios off in few chandankumar have a good evening | 14:32 | |
| chandankumar | marios: master promoted now \o/ | 14:39 |
| chandankumar | marios: have a nice evening and weekend, see ya on monday :-) | 14:39 |
| marios | chandankumar: nice | 14:42 |
| marios | chandankumar: mojito on me ;) | 14:42 |
| *** marios is now known as marios|out | 15:00 | |
| jm1 | what a day... but finally toy fcos is booting via network with a signed boot chain 🥳 | 17:54 |
| jm1 | now i need a break.. | 17:54 |
| jm1 | have a nice weekend #oooq! | 17:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!