| rdogerrit | rdo-trunk proposed rdoinfo master: Bump rdoinfo zed-uc tags to latest upper-constraints https://review.rdoproject.org/r/c/rdoinfo/+/43643 | 00:05 |
|---|---|---|
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for xena-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43652 | 00:06 |
| rdogerrit | rdo-trunk proposed rdoinfo master: Promote CBS tags update for wallaby-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43608 | 00:08 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for yoga-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43653 | 00:10 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for yoga-9s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43654 | 00:11 |
| *** ysandeep|out is now known as ysandeep | 04:34 | |
| *** pojadhav is now known as pojadhav|ruck | 05:11 | |
| rdogerrit | User bshewale proposed rdo-jobs master: Repareting sc010 jobs using -options job https://review.rdoproject.org/r/c/rdo-jobs/+/43146 | 05:53 |
| *** amoralej|off is now known as amoralej | 06:34 | |
| *** ysandeep is now known as ysandeep|afk | 06:42 | |
| *** marios_ is now known as marios | 06:53 | |
| rdogerrit | User karolinku proposed rdoinfo master: Crosstag python-tomli https://review.rdoproject.org/r/c/rdoinfo/+/43650 | 07:15 |
| rdogerrit | Merged rdoinfo master: Promote dependencies updates for zed cs9 https://review.rdoproject.org/r/c/rdoinfo/+/43563 | 07:25 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Crosstag python-tomli https://review.rdoproject.org/r/c/rdoinfo/+/43650 | 07:30 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Update puppet and facter to last versions for CS9 in Zed https://review.rdoproject.org/r/c/rdoinfo/+/42273 | 07:30 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 07:44 | |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for xena-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43652 | 07:54 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for yoga-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/43653 | 08:02 |
| rdogerrit | Alfredo Moralejo proposed nfvinfo master: Promote openvswitch2.17 and ovn22.03 to NFV SIG testing tags https://review.rdoproject.org/r/c/nfvinfo/+/43651 | 08:11 |
| amoralej | ykarel_, jcapitao[m] https://review.rdoproject.org/r/c/nfvinfo/+/43651 if you have a chance | 08:30 |
| ykarel_ | looking | 08:31 |
| *** pojadhav|lunch is now known as pojadhav|ruck | 08:38 | |
| *** ysandeep|afk is now known as ysandeep | 09:10 | |
| *** ykarel_ is now known as ykarel|lunch | 09:19 | |
| ykarel|lunch | amoralej, fyi https://bugs.launchpad.net/tripleo/+bug/1979276 | 09:20 |
| ykarel|lunch | new ovs/ovn release looks suspecious | 09:20 |
| amoralej | ykarel|lunch, ups | 09:21 |
| amoralej | ykarel|lunch, feel free to propose the revert if that's the problem | 09:22 |
| ykarel|lunch | yes sure will dig post lunch | 09:23 |
| ykarel|lunch | but likely would need revert as affecting gate | 09:23 |
| rdogerrit | Merged nfvinfo master: Promote openvswitch2.17 and ovn22.03 to NFV SIG testing tags https://review.rdoproject.org/r/c/nfvinfo/+/43651 | 09:31 |
| dulek | Hey folks! So we see checksum mismatch for "CentOS-9 - NFV OpenvSwitch" in our CI. Any idea how to work this around? | 10:04 |
| rdogerrit | User bshewale created rdo-infra/ci-config master: Add bshewale's public key https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43656 | 10:04 |
| rdogerrit | User bshewale proposed rdo-jobs master: Reparented standalone-full-tempest-api and scenarios jobs https://review.rdoproject.org/r/c/rdo-jobs/+/43149 | 10:08 |
| *** rlandy|out is now known as rlandy | 10:25 | |
| rdogerrit | Merged rdoinfo master: Bump rdoinfo zed-uc tags to latest upper-constraints https://review.rdoproject.org/r/c/rdoinfo/+/43643 | 10:39 |
| rdogerrit | Merged rdoinfo master: Crosstag python-tomli https://review.rdoproject.org/r/c/rdoinfo/+/43650 | 10:39 |
| rdogerrit | rdo-trunk created config master: Create project for RDO dependency python-tomli https://review.rdoproject.org/r/c/config/+/43657 | 10:41 |
| rdogerrit | rdo-trunk created config master: Create project info for RDO dependency python-tomli in RDO https://review.rdoproject.org/r/c/config/+/43658 | 10:41 |
| amoralej | jcapitao[m], we have a mistery https://kojipkgs.fedoraproject.org//work/tasks/2451/88342451/build.log | 11:00 |
| amoralej | dulek, where? | 11:00 |
| amoralej | may you paste the exact error? | 11:00 |
| maysams | amoralej: I was checking the issue with dulek and this is it https://zuul.opendev.org/t/openstack/build/3ce48b90974b474893ee8189726493a2 | 11:02 |
| amoralej | maysams, looks like infra issue | 11:03 |
| amoralej | let me try it | 11:03 |
| jcapitao[m] | amoralej: is it the rebuild for py3.11 ? | 11:03 |
| amoralej | jcapitao[m], yep | 11:03 |
| maysams | amoralej: I also tried locally and it didn't work | 11:03 |
| amoralej | the error seems related to tempita | 11:03 |
| jcapitao[m] | lemme check | 11:03 |
| jcapitao[m] | I missed the mass rebuild, do we have a tracker ? | 11:04 |
| amoralej | what is strange is that, in the tempita build for fc35 i see a patch | 11:04 |
| amoralej | when importing urllib quote | 11:04 |
| amoralej | which is not upstream, and i don't find where it was patched | 11:04 |
| amoralej | maysams, correct, i can reproduce it locally too, let me ask centos team | 11:05 |
| amoralej | maysams, reported https://pagure.io/centos-infra/issue/819 | 11:11 |
| maysams | dulek: fyi ^ | 11:11 |
| ykarel|lunch | amoralej, recently similar was seen for other centos repos too, in that case it was issue related to mirror manager | 11:11 |
| maysams | thank you, amoralej | 11:12 |
| jcapitao[m] | amoralej: what is upstream repo of tempita ? | 11:12 |
| amoralej | maysams, as temporary workaround you can use the testing repo | 11:12 |
| amoralej | --disablerepo=centos-nfv-openvswitch --enablerepo=centos-nfv-openvswitch-testing | 11:12 |
| amoralej | i checked it's working fine now | 11:13 |
| amoralej | jcapitao[m], i took the tarball from the srpm directly | 11:13 |
| amoralej | ykarel|lunch, yep, that's mirror manager having sync problems | 11:13 |
| *** ykarel|lunch is now known as ykarel_ | 11:14 | |
| amoralej | jcapitao[m], must be use_2to3 | 11:17 |
| amoralej | jcapitao[m], any reason to remove it? | 11:18 |
| amoralej | probably that code is old | 11:18 |
| jcapitao[m] | I think it was related to last mass rebuild | 11:20 |
| amoralej | jcapitao[m], the repo may be https://github.com/agramfort/tempita/commits/master | 11:21 |
| amoralej | i'm not sure | 11:21 |
| amoralej | from 2013 ... | 11:21 |
| amoralej | what is weird is that it doesn't fail in upstream | 11:21 |
| amoralej | ah, because of 2to3 is in setup.cfg | 11:21 |
| amoralej | or something | 11:21 |
| jcapitao[m] | yeah it coincide with the f36 mass rebuild in January https://koji.fedoraproject.org/koji/packageinfo?packageID=6339 | 11:24 |
| amoralej | yep i see it ... | 11:24 |
| jcapitao[m] | so I had to remove use_2to3 to be able to build it again | 11:24 |
| amoralej | yep, i see the problem now | 11:26 |
| jcapitao[m] | ah | 11:26 |
| amoralej | in upstream, when you install from pip ig pulls the wheel which is built for py3 so it carries the patch | 11:26 |
| jcapitao[m] | the main problem you mean ? | 11:26 |
| jcapitao[m] | ah | 11:27 |
| jcapitao[m] | okk | 11:27 |
| * jcapitao[m] checking | 11:29 | |
| *** dviroel|out is now known as dviroel | 11:30 | |
| amoralej | jcapitao[m], yes, i understand why we don't have the patch and why the problem doesn't happen in upstream | 11:32 |
| amoralej | i wonder if we may use the whl package as source instead | 11:32 |
| jcapitao[m] | ah setuptools is 57.4 when pull with pip | 11:33 |
| jcapitao[m] | so there is the 2to3 support | 11:34 |
| amoralej | there are other fixes needed | 11:38 |
| amoralej | to make it work with python3 | 11:38 |
| amoralej | we may implement it as a patch | 11:38 |
| jcapitao[m] | ah I think I got it now | 11:38 |
| jcapitao[m] | files in wheel are from sept 2021 | 11:39 |
| amoralej | yes | 11:41 |
| amoralej | wheel was created in pypi with py3 | 11:41 |
| amoralej | and executed 2to3 | 11:41 |
| amoralej | so the content on it is fine | 11:41 |
| amoralej | said this, it seems clear that tempita should be avoided | 11:42 |
| rdogerrit | Yatin Karel created nfvinfo master: Revert "Promote CBS tags update for nfvsig-2.15-release" https://review.rdoproject.org/r/c/nfvinfo/+/43673 | 11:42 |
| amoralej | ykarel_, so it seems ovs is the root cause? | 11:43 |
| jcapitao[m] | amoralej: I was not aware of this dynamic part, I assumed it was static since 2013 | 11:43 |
| ykarel_ | amoralej, it's still being investigated, but to clear gate i proposed ^ | 11:43 |
| ykarel_ | the thing i don't understand is why we not seeing it in rdo jobs | 11:44 |
| jcapitao[m] | amoralej: so we should get the content of the wheel, get the diff with current tarball, and apply it as a patch ? | 11:44 |
| ykarel_ | i mean in promotion pipelines | 11:44 |
| amoralej | jcapitao[m], that's exactly what i'm doing | 11:44 |
| amoralej | actually, instead of the wheel, i'm taking the content from the last rpm that hat 2to3 in fedora | 11:44 |
| amoralej | the one for f35 | 11:44 |
| amoralej | ykarel_, is the problem consistent? | 11:45 |
| rdogerrit | User karolinku proposed config master: Create project for RDO dependency python-tomli https://review.rdoproject.org/r/c/config/+/43657 | 11:45 |
| ykarel_ | amoralej, random | 11:45 |
| ykarel_ | atleast in devstack,puppet-openstack jobs seeing both success/failures | 11:46 |
| ykarel_ | as per bug in tripleo-standalone it's consistent | 11:46 |
| ykarel_ | https://zuul.opendev.org/t/openstack/builds?job_name=puppet-glance-tripleo-standalone&project=openstack/puppet-glance | 11:46 |
| amoralej | it's strange is consistent in oly one particular job | 11:48 |
| amoralej | ykarel_, those are standalone oooq jobs? | 11:48 |
| ykarel_ | yes it's standalone job | 11:49 |
| jcapitao[m] | I missed the whole thing in my patch a couple of months ago | 11:49 |
| jcapitao[m] | ok good, thank you for the clarification | 11:49 |
| ykarel_ | i have not checedk other tripleo jobs if that's consistent there too, but just read bug | 11:49 |
| amoralej | ykarel_, the problem seems more related to ovn that to ovs? | 11:52 |
| ykarel_ | like other job it's inconstent https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-standalone | 11:52 |
| amoralej | maybe https://review.rdoproject.org/r/c/nfvinfo/+/43620 ? | 11:52 |
| ykarel_ | amoralej, actually i proposed ovs first as ovn version was getting tested in downstream jobs and was found good | 11:53 |
| ykarel_ | but yes you are right just seeing the symptoms it's looks more ovn related | 11:53 |
| amoralej | ah, that makes sense too | 11:53 |
| amoralej | let me untag latest ovs2.15 from -release | 11:54 |
| ykarel_ | yes let's try that, and if it didn't help then we can check ovn one | 11:54 |
| amoralej | ykarel_, you see it only in master? | 11:55 |
| amoralej | i mean, only in centos9 ? | 11:55 |
| ykarel_ | amoralej, yes till now seen only in centos9 and master/wallaby | 11:56 |
| jcapitao[m] | what are the logs you're looking ? I didn't find the reason for standalong oooq job earlier | 11:56 |
| ykarel_ | jcapitao[m], not sure you also looking at same issue | 11:57 |
| ykarel_ | but we checking https://bugs.launchpad.net/tripleo/+bug/1979276 | 11:57 |
| ykarel_ | https://zuul.opendev.org/t/openstack/build/4757380fddac4d59a02f778887727c0e/log/logs/undercloud/var/log/extra/journal.txt#46904 | 11:58 |
| jcapitao[m] | ah, not the same :) | 11:58 |
| amoralej | ykarel_, untagged | 11:58 |
| amoralej | both for cs8 and cs9 | 11:58 |
| ykarel_ | amoralej, ok let's see how it goes, should be clear in next few hours | 11:58 |
| amoralej | ykarel_, new builds will keep coming | 11:59 |
| amoralej | so it's important to find the issue and involve fdp guys if needed | 11:59 |
| amoralej | mmm | 11:59 |
| amoralej | if it's cs9 only, maybe we should switch to ovs2.17 | 11:59 |
| ykarel_ | yes i am checking with ovn guys in internal channel | 11:59 |
| amoralej | ykarel_, i have builds ready for ovs2.17 and ovn22.03 | 12:00 |
| ykarel_ | yes we can consider the switch but once heat settles down :) | 12:00 |
| amoralej | yep :) | 12:00 |
| rdogerrit | Merged nfvinfo master: Revert "Promote CBS tags update for nfvsig-2.15-release" https://review.rdoproject.org/r/c/nfvinfo/+/43673 | 12:03 |
| rdogerrit | Alfredo Moralejo created nfvinfo master: Promote openvswitchr-2.17 and ovn-22.03 to release https://review.rdoproject.org/r/c/nfvinfo/+/43660 | 12:04 |
| amoralej | ykarel_, ^ | 12:04 |
| *** amoralej is now known as amoralej|lunch | 12:04 | |
| ykarel_ | k let's promote atleast | 12:04 |
| *** jschluet is now known as jschlueter|dad_taxi | 12:20 | |
| *** jschlueter|dad_taxi is now known as jschlueter | 12:23 | |
| rdogerrit | Merged openstack/tripleo-validations-distgit ussuri-rdo: Remove the tripleo-ansible-inventory script https://review.rdoproject.org/r/c/openstack/tripleo-validations-distgit/+/43616 | 12:58 |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| rdogerrit | User karolinku proposed deps/python-hatchling c9s-zed-rdo: Update to 1.3.1 https://review.rdoproject.org/r/c/deps/python-hatchling/+/43472 | 13:27 |
| rdogerrit | Merged config master: Create project for RDO dependency python-tomli https://review.rdoproject.org/r/c/config/+/43657 | 13:30 |
| *** dasm|off is now known as dasm | 13:33 | |
| amoralej | jcapitao[m], https://src.fedoraproject.org/rpms/python-tempita/c/9b628793889c39e318bfd0bb5058353d4c8f88fb?branch=rawhide | 13:33 |
| amoralej | i just built it | 13:33 |
| jcapitao[m] | great, you tested it with oslo-db ? | 13:35 |
| amoralej | jcapitao[m], i'm on it | 13:40 |
| amoralej | https://koji.fedoraproject.org/koji/taskinfo?taskID=88549998 | 13:40 |
| jcapitao[m] | nice amoralej++ | 13:43 |
| amoralej | it passed, nice | 13:43 |
| rdogerrit | Merged deps/python-hatchling c9s-zed-rdo: Update to 1.3.1 https://review.rdoproject.org/r/c/deps/python-hatchling/+/43472 | 13:47 |
| ykarel_ | amoralej, the same job passing when run in rdo-infra https://review.rdoproject.org/r/c/testproject/+/43551 | 14:06 |
| amoralej | ykarel_, with which version of ovs? | 14:06 |
| ykarel_ | -99 | 14:07 |
| amoralej | the "bad" one | 14:07 |
| ykarel_ | yes | 14:07 |
| amoralej | no, the old one | 14:07 |
| amoralej | let me check version, i forot | 14:07 |
| amoralej | forgot | 14:07 |
| ykarel_ | -99 is the latest, that we reverted | 14:07 |
| amoralej | yep | 14:08 |
| amoralej | weird | 14:08 |
| amoralej | may it be related to the image? | 14:08 |
| ykarel_ | no idea :(. i am running upstream two jobs one excluding latest ovn and other excluding latest ovn | 14:08 |
| ykarel_ | https://review.opendev.org/c/openstack/puppet-glance/+/847014 | 14:09 |
| amoralej | ack | 14:09 |
| ykarel_ | https://review.opendev.org/c/openstack/puppet-glance/+/847012 | 14:09 |
| amoralej | you mentioned devstack before | 14:10 |
| ykarel_ | exclude ovs already failed, waiting for results to see if patch really worked, if that's true we would need to revert ovn patch | 14:10 |
| amoralej | devstack uses nfv sig? | 14:10 |
| ykarel_ | amoralej, yes delorean-deps | 14:11 |
| ykarel_ | there nfv sig repos are set | 14:11 |
| amoralej | and fails consistently? | 14:11 |
| ykarel_ | nope random | 14:11 |
| amoralej | that may be easier to reproduce | 14:11 |
| amoralej | but same issue? | 14:11 |
| ykarel_ | yes same | 14:11 |
| ykarel_ | let me try that too on vm | 14:13 |
| ykarel_ | patch worked, so it failed even with old ovs | 14:19 |
| *** ysandeep is now known as ysandeep|afk | 14:19 | |
| ykarel_ | patch with old ovn passing, reached tempest | 14:19 |
| ykarel_ | tempest too passed | 14:19 |
| amoralej | ykarel_which ovs version? | 14:20 |
| amoralej | you are testing ovn or ovs there? | 14:21 |
| rdogerrit | Yatin Karel created nfvinfo master: Revert "Promote CBS tags update for nfvsig-2021-release" https://review.rdoproject.org/r/c/nfvinfo/+/43674 | 14:21 |
| ykarel_ | amoralej, https://00d84cadba7dd9fbd396-fc598bfe0fee75dcfd56d86b8dd3e9bc.ssl.cf2.rackcdn.com/847014/1/check/puppet-glance-tripleo-standalone/332c14c/logs/undercloud/home/zuul/container-builds/f6837a1a-616b-486f-9b4e-feb8635e3764/base/ovn-base/ovn-base-build.log | 14:22 |
| ykarel_ | in ^ old ovs + latest ovn tested, and failed | 14:23 |
| amoralej | good | 14:23 |
| ykarel_ | in other one old ovn + latest ovs, and that passed | 14:23 |
| amoralej | ykarel_, | 14:24 |
| amoralej | nop | 14:24 |
| amoralej | ah, yes | 14:24 |
| ykarel_ | so ovn one is bad, but likely something else combined with it causing issues, as we not seeing in rdo jobs, and neither seen in downstream jobs | 14:24 |
| amoralej | :) | 14:24 |
| amoralej | it may be the mix of latest ovn and latest ovs | 14:24 |
| amoralej | i mean, latest ovs in nfv sig | 14:24 |
| ykarel_ | but with latest ovs and latest ovn, the same jobs passing in rdo-infra | 14:26 |
| ykarel_ | amoralej, can you untag ovn-2021-21.12.0-46 too | 14:27 |
| ykarel_ | will continue troubleshooting with testing repo | 14:27 |
| ykarel_ | revert https://review.rdoproject.org/r/c/nfvinfo/+/43674 | 14:28 |
| amoralej | ykarel_, untagged | 14:28 |
| ykarel_ | amoralej, Thanks, let's see infra mirrors picks both, infra mirrors not yet updated | 14:29 |
| ykarel_ | centos mirrors already have untagged ovs removed | 14:29 |
| ykarel_ | amoralej, in ovn one, both ovn and ovs were downgraded, as mirror.stream already have remove ovs2.15-99 | 14:44 |
| amoralej | ok | 14:45 |
| ykarel_ | updating lp with the latest info | 14:46 |
| ykarel_ | amoralej, devstack deployed successfully in local tests | 15:06 |
| amoralej | ykarel_, with old ovs/ovn? | 15:06 |
| ykarel_ | with the bad ones | 15:07 |
| amoralej | damn | 15:07 |
| ykarel_ | amoralej, i noticed upstream nodes have old kernel, can that be related? | 15:07 |
| ykarel_ | 5.14.0-96 vs 5.14.0-105 | 15:07 |
| amoralej | who knows ... | 15:07 |
| amoralej | maybe | 15:07 |
| ykarel_ | yeap, next i will try with same kernel as upstream nodes | 15:08 |
| ykarel_ | or the last thing tomorrow to hold a upstream node if nothing works | 15:08 |
| *** ysandeep|afk is now known as ysandeep | 15:12 | |
| rdogerrit | Merged nfvinfo master: Revert "Promote CBS tags update for nfvsig-2021-release" https://review.rdoproject.org/r/c/nfvinfo/+/43674 | 15:23 |
| rdogerrit | Merged nfvinfo master: Promote openvswitchr-2.17 and ovn-22.03 to release https://review.rdoproject.org/r/c/nfvinfo/+/43660 | 15:39 |
| *** dviroel is now known as dviroel|lunch | 15:52 | |
| *** amoralej is now known as amoralej|off | 16:09 | |
| *** marios is now known as marios|out | 16:11 | |
| *** pojadhav|ruck is now known as pojadhav|out | 16:11 | |
| *** jpena is now known as jpena|off | 16:24 | |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| rdogerrit | rdo-trunk created openstack/nova-distgit rpm-master: openstack-nova: failed to build 84a84f7f2f https://review.rdoproject.org/r/c/openstack/nova-distgit/+/43661 | 17:24 |
| rdogerrit | rdo-trunk created openstack/ironic-tempest-plugin-distgit rpm-master: python-ironic-tests-tempest: failed to build 3af6253 https://review.rdoproject.org/r/c/openstack/ironic-tempest-plugin-distgit/+/43662 | 17:29 |
| *** ysandeep is now known as ysandeep|out | 17:37 | |
| rdogerrit | rdo-trunk created openstack/designate-distgit rpm-master: openstack-designate: failed to build 260e1ddd https://review.rdoproject.org/r/c/openstack/designate-distgit/+/43663 | 17:49 |
| rdogerrit | rdo-trunk created openstack/designate-dashboard-distgit rpm-master: openstack-designate-ui: failed to build 50bb2c1 https://review.rdoproject.org/r/c/openstack/designate-dashboard-distgit/+/43664 | 17:49 |
| rdogerrit | rdo-trunk created openstack/cinder-tempest-plugin-distgit rpm-master: python-cinder-tests-tempest: failed to build 141dace https://review.rdoproject.org/r/c/openstack/cinder-tempest-plugin-distgit/+/43665 | 17:49 |
| *** dviroel is now known as dviroel|afk | 20:06 | |
| *** dasm is now known as dasm|off | 21:38 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!