| *** soniya29 is now known as soniya29|ruck | 04:16 | |
| *** ysandeep|out is now known as ysandeep | 04:58 | |
| ysandeep | chandankumar, soniya29|ruck 17 on rhel9 container build is failing, left log url for you on internal channel. | 05:01 |
|---|---|---|
| chandankumar | ysandeep: thanks, checking | 05:52 |
| *** chandankumar is now known as chkumar|rover | 05:52 | |
| ysandeep | chandankumar: ignore, I have proposed a fix, under testing currently. | 05:53 |
| chkumar|rover | ysandeep: ansible one? | 05:53 |
| ysandeep | chkumar|rover, yes https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059 | 05:53 |
| ysandeep | funny thing is it was working so far, from yesterday it started complaining that ansible package is not available.. may be earlier it was installing ansible-core even if someone gives "dnf install ansible" :D | 05:53 |
| chkumar|rover | soniya29|ruck: new hackmd https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view | 05:55 |
| *** ysandeep is now known as ysandeep|afk | 06:13 | |
| marios | soniya29|ruck: o/ hi morning anything you need help with? | 06:29 |
| *** ysandeep|afk is now known as ysandeep | 07:01 | |
| ysandeep | marios: enjoy your pto, We will help soniya29|ruck if she needs help with anything. | 07:01 |
| marios | o/ morning thanks ysandeep :) | 07:02 |
| marios | am going to take the kids out in a bit just doing some reviews and check if everything ok | 07:03 |
| *** amoralej|off is now known as amoralej | 07:10 | |
| ysandeep | Getting a node in downstream is really really slow today, I have pinged infra on internal channel. | 07:17 |
| marios | chkumar|rover: soniya29|ruck: do you have a hackmd? | 07:33 |
| ysandeep | marios: https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view | 07:40 |
| ysandeep | marios, fyi.. this is what chandan pinged to soniya before you joined channel | 07:41 |
| ysandeep | ~~~ | 07:41 |
| ysandeep | chkumar|rover> soniya29|ruck: new hackmd https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view | 07:41 |
| ysandeep | ~~~ | 07:41 |
| marios | ysandeep: ah thank you | 07:43 |
| marios | we have a new gate blocker i want to check if they have it filed already | 07:43 |
| ysandeep | takashi filed this today: https://bugs.launchpad.net/tripleo/+bug/1966382 - incase you are seeing similiar issue | 07:44 |
| marios | ysandeep: heh yeah i just filed a duplicate :/ | 07:45 |
| marios | ysandeep: https://bugs.launchpad.net/tripleo/+bug/1966384 ... k putting them on the hackmd thanks | 07:46 |
| soniya29|ruck | chkumar|rover, ack | 07:46 |
| soniya29|ruck | marios: not any thing as such, will let you know if required | 07:46 |
| ysandeep | marios: okay | 07:46 |
| marios | i have to go in a bit soniya29|ruck (will try come back later) but there is a gate blocker there will need some attention fyi ^^^ | 07:50 |
| marios | soniya29|ruck: i added it to the hackmd under 25 mar | 07:50 |
| *** ysandeep is now known as ysandeep|lunch | 08:04 | |
| *** marios is now known as marios|out | 08:07 | |
| *** arxcruz|ruck is now known as arxcruz | 08:10 | |
| *** jpena|off is now known as jpena | 08:11 | |
| ysandeep|lunch | marios|out, chkumar|rover looks like broke by latest setuptools: https://github.com/pypa/setuptools/issues/3197 | 08:13 |
| marios|out | ack ysandeep|lunch yeah i found that one but it was the only report i could find so must be a new thing | 08:18 |
| *** marios|out is now known as marios | 08:18 | |
| ysandeep|lunch | yes setuptools release 14 hours ago.. I think we will see more bugs soon | 08:21 |
| ysandeep|lunch | someone just posted more details in https://github.com/pypa/setuptools/issues/3197 | 08:22 |
| marios | ack ysandeep|lunch looks like there is a suggested workaround maybe chkumar|rover or soniya can try it i will really have to go in a sec | 08:24 |
| ysandeep|lunch | marios, don't worry you go, chkumar|rover is out for lunch.. we will take care of this. | 08:24 |
| * ysandeep|lunch also checking | 08:24 | |
| marios | thanks ysandeep|lunch | 08:25 |
| marios | thanks chkumar|rover | 08:25 |
| ysandeep|lunch | Pune team is meeting first time after long, they went out for lunch together. | 08:26 |
| marios | i see :) | 08:26 |
| marios | nice | 08:26 |
| *** marios is now known as marios|out | 08:27 | |
| ysandeep|lunch | I think you and me are the only one here for now, and here you go as well :D | 08:27 |
| * ysandeep|lunch stepping away for lunch for sometime | 08:27 | |
| ysandeep|lunch | marios|out, chkumar|rover trying https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835178 as a workaround. Let's see if it helps | 08:35 |
| jpodivin_ | soniya29|ruck: I'm seeing some strange build errors on centos 9 jobs. "error: Multiple top-level packages discovered in a flat-layout: ['roles', 'plugins', 'infrared_plugin']." https://zuul.opendev.org/t/openstack/build/1ce0e0dd27df433687346bab240df465 | 09:20 |
| soniya29|ruck | jpodivin_, i am watching the same issue | 09:21 |
| jpodivin_ | soniya29|ruck: I see, any idea what could it be? | 09:21 |
| *** jpodivin_ is now known as jpodivin | 09:21 | |
| soniya29|ruck | jpodivin_, i am testing these jobs to confirm whether this issue is consistent | 09:22 |
| soniya29|ruck | jpodivin, nope :) | 09:22 |
| jpodivin | soniya29|ruck: we are on the same page then :D | 09:22 |
| soniya29|ruck | jpodivin, yeah | 09:23 |
| jpodivin | I've hit recheck on the patch and we'll see if it persists. Worst case scenario we will have to implement some fix in the package concerned. I've found some examples of this error online. | 09:23 |
| jpodivin | I shouldn't happen unless the package structure changed considerably. | 09:24 |
| jpodivin | s/I/It/ | 09:24 |
| chkumar|rover | jpodivin: soniya29|ruck fyi https://bugs.launchpad.net/tripleo/+bug/1966382 | 09:25 |
| jpodivin | chkumar|rover++ cool, thanks | 09:25 |
| soniya29|ruck | chkumar|rover, thanks | 09:28 |
| chkumar|rover | ysandeep|lunch: proposed similar fix on https://review.opendev.org/c/openstack/tripleo-quickstart/+/835192 | 09:30 |
| jpodivin | I have to say, and I'm bit ashamed of it, makes for a nice change to have something from the outside break our CI :D | 09:30 |
| *** ysandeep|lunch is now known as ysandeep | 09:32 | |
| jpodivin | chkumar|rover: are we going to fix in in the ansible-role-collect-logs as well? | 09:32 |
| ysandeep | chkumar|rover, job freeze: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835192/1#message-b239fc34a592880f359f5872459535ad2b67c2f8 | 09:34 |
| ysandeep | jpodivin: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835178 | 09:35 |
| jpodivin | ysandeep: I see, thanks. | 09:35 |
| ysandeep | chkumar|rover, job freeze: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835192/1#message-b239fc34a592880f359f5872459535ad2b67c2f8 | 09:35 |
| chkumar|rover | akahat: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835192 | 09:36 |
| chkumar|rover | ysandeep: checking that one, figuring out how to fix it | 09:36 |
| jpodivin | ysandeep, chkumar|rover: could we keep the directive in one place? In either pyproject.toml or setup.cfg file I mean. Both work fine obviously but just for consistency sake? | 09:37 |
| ysandeep | looking at https://github.com/pypa/setuptools/issues/3197, it broke multiple projects.. someone reported one more broken project. | 09:39 |
| ysandeep | jpodivin: I agree with you.. let us test if fix works, then we will update to keep in one place. | 09:40 |
| chkumar|rover | ysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/835193 | 09:41 |
| jpodivin | ysandeep: thanks. As for the number of projects broken, I imagine we are only hitting tip of the iceberg. It has been published ... 24 hrs ago? And it's hitting build process, so I suspect that others will soon come knocking on the pypa door as their periodic builds fail. | 09:42 |
| jpodivin | they might even bring torches | 09:42 |
| jpodivin | Should we join them? | 09:43 |
| ysandeep | jpodivin, true ... people will complain as they woke up today.. | 09:43 |
| ysandeep | chkumar|rover: ack, looking | 09:47 |
| chkumar|rover | ysandeep: working on the fix | 09:48 |
| chkumar|rover | ysandeep: now it works https://zuul.openstack.org/status#835192 | 09:49 |
| chkumar|rover | I mean the freeze issue is gone | 09:49 |
| ysandeep | chkumar|rover, nice ++ | 09:52 |
| soniya29|ruck | chkumar|rover, ysandeep, we are hitting this same issue on centos-8 ussuri, wallaby and victoria as well | 09:52 |
| ysandeep | soniya29|ruck, everything uses setuptools, that's expected | 09:53 |
| soniya29|ruck | ysandeep, okay | 09:53 |
| ysandeep | chkumar|rover, can you +w this https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059 , fixed downstream contianer build - tested already. | 10:03 |
| chkumar|rover | ysandeep: onit | 10:08 |
| ysandeep | chkumar|rover: thanks! | 10:10 |
| chkumar|rover | ysandeep: soniya29|ruck dropping an email to list | 10:15 |
| chkumar|rover | so that people can avoid merging patches | 10:15 |
| ysandeep | chkumar|rover, +1 I also left a message on #opendev incase other projects see this as well. | 10:16 |
| chkumar|rover | ysandeep: soniya29|ruck http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027832.html | 10:20 |
| chkumar|rover | ysandeep: can you update the collect-logs patch https://opendev.org/openstack/ansible-role-collect-logs with setup.cfg changes | 10:28 |
| *** rlandy|PTO is now known as rlandy | 10:28 | |
| chkumar|rover | linters failing https://56ebfee43ce7e28457e8-7037332330f80d2400043986fc1e8a59.ssl.cf2.rackcdn.com/835178/3/check/openstack-tox-linters/ae11ee9/job-output.txt | 10:28 |
| marios|out | thanks ysandeep chkumar|rover looks like you got that one sorted | 10:29 |
| ysandeep | chkumar|rover, that patch didn't work as well | 10:29 |
| ysandeep | chkumar|rover, i just tried adding in setup.py - setuptools.setup(setup_requires=["pbr"], pbr=True, py_modules=[]) | 10:30 |
| ysandeep | ^^ works | 10:30 |
| ysandeep | chkumar|rover, let me try putting in setup.cfg if that works as well | 10:30 |
| chkumar|rover | setup.cfg works | 10:30 |
| rlandy | chkumar|rover: soniya29|ruck: hello | 10:32 |
| rlandy | how are things today? | 10:32 |
| ysandeep | chkumar|rover, changes in setup.py works but not in setup.cfg http://pastebin.test.redhat.com/1039857 | 10:33 |
| marios|out | rlandy: setuptools release broke the gate c9 blocker ;) | 10:33 |
| ysandeep | chkumar|rover, is [metadata] correct section, have you already tried in your local setup? | 10:33 |
| rlandy | ok - do we have a bug on it? | 10:34 |
| ysandeep | chkumar|rover, Do you want to jump on a call, let me show what i am facing. | 10:34 |
| chkumar|rover | rlandy: https://bugs.launchpad.net/tripleo/+bug/1966382 | 10:34 |
| chkumar|rover | ysandeep: yes sure | 10:34 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1966382 | 10:34 |
| ysandeep | chkumar|rover, ack 1 min | 10:34 |
| rlandy | yea - I see it | 10:34 |
| rlandy | chkumar|rover: let me know when you are leaving so we can sync before | 10:35 |
| chkumar|rover | rlandy: sure | 10:35 |
| ysandeep | chkumar|rover, meet.google.com/eqb-bghy-nkd | 10:36 |
| rlandy | soniya29|ruck: ysandeep: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059 fixes container builds? | 10:36 |
| ysandeep | rlandy, yes | 10:36 |
| rlandy | soniya29|ruck: how are things in downstream? | 10:40 |
| chkumar|rover | https://6bdcbd79c131f54e644b-cf46b97128df9f2b5d8420d3b89377a5.ssl.cf1.rackcdn.com/835086/2/check/tripleo-puppet-ci-centos-9-content-provider/685b30d/job-output.txt | 10:40 |
| chkumar|rover | error: subprocess-exited-with-error | 10:41 |
| chkumar|rover | ysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/835193 | 10:54 |
| chkumar|rover | ysandeep: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/quickstart.sh#L167 | 11:04 |
| chkumar|rover | https://opendev.org/openstack/tripleo-quickstart/src/branch/master/requirements.txt#L7 | 11:08 |
| * soniya29|ruck stepping out for an hour...dr appt | 11:16 | |
| chkumar|rover | ysandeep: https://zuul.openstack.org/status#835193 and https://zuul.openstack.org/stream/1c0ec35cf89440faa57210d4772001fe?logfile=console.log | 11:17 |
| rlandy | chkumar|rover: ysandeep: anyone joining review time? otherwise will drop | 11:17 |
| ysandeep | rlandy, joining | 11:17 |
| chkumar|rover | rlandy: https://review.opendev.org/q/topic:lp%252F1966382 | 11:25 |
| amoralej | there has been a new version of jinja that seems to broke ansible when installing from pip | 11:28 |
| amoralej | did you hit any issue? | 11:28 |
| amoralej | messate is cannot import name 'environmentfilter' from 'jinja2.filters' | 11:28 |
| chkumar|rover | amoralej: not yet | 11:30 |
| chkumar|rover | amoralej: setuptools broke the world | 11:30 |
| amoralej | i just saw your lp :( | 11:30 |
| rlandy | chkumar|rover++ ysandeep++ for quick resolutions | 11:44 |
| ysandeep | I just hope pining works :) | 11:45 |
| rlandy | yep - that's the cleanest shot atm | 11:47 |
| ysandeep | chkumar|rover, https://zuul.openstack.org/stream/f7bf82b424794ea798204639c354b01a?logfile=console.log - didn't work | 11:48 |
| rlandy | https://zuul.opendev.org/t/openstack/stream/f7bf82b424794ea798204639c354b01a?logfile=console.log | 11:48 |
| rlandy | yep | 11:48 |
| rlandy | no show | 11:48 |
| rlandy | ysandeep: can you downgrade | 11:49 |
| rlandy | maybe it's already installed | 11:49 |
| ysandeep | rlandy, yes can try that | 11:51 |
| rlandy | ysandeep: would it also help to hold a node on rdo zuul? | 11:52 |
| ysandeep | chkumar|rover: want to meet again? | 11:54 |
| chkumar|rover | ysandeep: going with another approach https://review.opendev.org/c/openstack/tripleo-quickstart/+/835192/ | 11:57 |
| chkumar|rover | making wallaby jobs voting | 11:57 |
| chkumar|rover | wait trying one more hack | 11:58 |
| ysandeep | ack, me checking why pinning didn't work | 11:58 |
| chkumar|rover | ysandeep: rlandy https://zuul.opendev.org/t/openstack/status#835192 hope this works | 12:00 |
| chkumar|rover | otherwise I have no options | 12:01 |
| chkumar|rover | ysandeep: need to leave now | 12:01 |
| chkumar|rover | see ya! | 12:01 |
| *** chkumar|rover is now known as raukadah | 12:01 | |
| ysandeep | chkumar|rover, ack, just to confirm you are out for the day | 12:01 |
| ysandeep | raukadah, ^^ | 12:01 |
| raukadah | ysandeep: I am out till thursday | 12:02 |
| raukadah | will check emails on monday | 12:02 |
| ysandeep | raukadah: :D o/ enjoy | 12:02 |
| ysandeep | rlandy, ^^ chandan is out | 12:02 |
| rlandy | ysandeep: yep I see | 12:02 |
| rlandy | ysandeep: ok - let's meet | 12:03 |
| rlandy | ysandeep: https://meet.google.com/zky-zevq-opd | 12:04 |
| arxcruz | i'm having connection issues, trying to check what's going on | 12:07 |
| ysandeep | rlandy, https://1ba43f288a45a3eff87a-0193a6f8a337e845d0f23740e999ed07.ssl.cf1.rackcdn.com/835207/1/check/tripleo-ci-centos-9-content-provider/f7bf82b/ | 12:09 |
| ysandeep | rlandy, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d09/835101/1/check/tripleo-ci-centos-9-content-provider/d09808b/logs/undercloud/var/log/extra/pip.txt | 12:13 |
| ysandeep | setuptools 57.4.0 | 12:13 |
| ysandeep | soniya29|ruck, >> <soniya29|ruck> chkumar|rover, ysandeep, we are hitting this same issue on centos-8 ussuri, wallaby and victoria as well | 12:23 |
| ysandeep | soniya29|ruck, can you share an example for c8 failure. | 12:23 |
| rlandy | soniya29|ruck: can you join https://meet.google.com/zky-zevq-opd | 12:25 |
| *** rlandy is now known as rlandy|rover | 12:30 | |
| rlandy|rover | ysandeep: ^^ guess I'm back | 12:32 |
| ysandeep | hahaha, i would not say it suits you.. but yeah you are really great at it :D | 12:33 |
| rlandy|rover | ysandeep: I wish I was great at other things | 12:34 |
| ysandeep | and here it goes: https://zuul.openstack.org/stream/527d63eb95794088a2c8167d60403615?logfile=console.log lets see in few mins | 12:35 |
| rlandy|rover | say a little prayer for pip | 12:35 |
| ysandeep | "/o\" | 12:36 |
| jpodivin | Regarding the setup error, would it be possible, as a stop gap solution, to restrict the version of setuptools to <61? | 12:37 |
| ysandeep | jpodivin, trying that here: https://review.opendev.org/c/openstack/tripleo-quickstart/+/835207 | 12:37 |
| ysandeep | otherwise we have to patch every repo setup.py | 12:38 |
| ysandeep | and merge order is not clear | 12:38 |
| rlandy|rover | jpodivin: you can join our little prayer session that it works | 12:38 |
| jpodivin | rlandy|rover, ysandeep: I really hope it does. | 12:39 |
| jpodivin | ysandeep, rlandy|rover: I guess we should update the requirements project as well. | 12:40 |
| rlandy|rover | :( | 12:41 |
| rlandy|rover | ysandeep: no show | 12:41 |
| ysandeep | rlandy|rover, let's get in reproducer env once available | 12:42 |
| dpawlik | rlandy|rover, marios|out, arxcruz: hey, is some logs that will be helpful to include in logs that will be pushed to Opensearch that are not mentioned here: https://opendev.org/openstack/ci-log-processing/src/branch/master/logscraper/config.yaml.sample ? | 12:42 |
| rlandy|rover | dpawlik: hey ... thanks - we can take a look in a but - we're just investigating a gate blocker atm | 12:43 |
| dpawlik | rlandy|rover: sure | 12:43 |
| arxcruz | dpawlik checking | 12:43 |
| arxcruz | dpawlik https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce5/830870/2/gate/tripleo-ci-centos-9-containers-multinode/ce5f591/logs/undercloud/var/log/tempest/ for example | 12:43 |
| arxcruz | tempest_run.log | 12:44 |
| arxcruz | tempestconf.log would be interesting, but not so critical | 12:44 |
| dpawlik | tempest_log.txt is there | 12:44 |
| arxcruz | dpawlik https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ce5/830870/2/gate/tripleo-ci-centos-9-containers-multinode/ce5f591/logs/undercloud/home/zuul/tempest/ | 12:44 |
| dpawlik | ah, its from undercloud | 12:44 |
| arxcruz | but ours is in undercloud/home/zuul/tempest | 12:45 |
| rlandy|rover | ysandeep: this is not happening in rdo | 12:46 |
| rlandy|rover | https://review.rdoproject.org/zuul/stream/bb2dfabbdd214b38acec5a04671ad12c?logfile=console.log | 12:46 |
| rlandy|rover | only opendev | 12:47 |
| soniya29|ruck | ysandeep, https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-victoria/d6e5443/job-output.txt | 12:47 |
| ysandeep | fun :D | 12:47 |
| ysandeep | soniya29|ruck, tempest failure? i thought you mean same issue as https://launchpad.net/bugs/1966382 | 12:49 |
| rlandy|rover | soniya29|ruck: hi - yeah we are looking for setuptools failures | 12:49 |
| rlandy|rover | ysandeep: checking version of setuptools in rdo | 12:49 |
| rlandy|rover | ysandeep: tetsproject started | 12:52 |
| soniya29|ruck | rlandy|rover, ack | 12:53 |
| rlandy|rover | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-master/e54ae99/job-output.txt | 12:54 |
| rlandy|rover | ah no - it is | 12:54 |
| rlandy|rover | it;s just quickstart | 12:54 |
| rlandy|rover | https://github.com/openstack/tripleo-quickstart/blob/master/install-deps.sh#L240 | 12:56 |
| rlandy|rover | ysandeep: ^^ | 12:56 |
| rlandy|rover | 2022-03-25 05:05:48.835127 | primary | Requirement already satisfied: distro>=1.3.0 in /home/zuul/workspace/.quickstart/lib/python3.9/site-packages (from selinux) (1.7.0) | 12:56 |
| rlandy|rover | 2022-03-25 05:05:48.836058 | primary | Requirement already satisfied: setuptools>=39.0 in /home/zuul/workspace/.quickstart/lib/python3.9/site-packages (from selinux) (53.0.0) | 12:56 |
| ysandeep | rlandy|rover, we need to confirm setuptools version in the working one | 12:57 |
| ysandeep | rlandy|rover, give me few mins - trying something | 12:58 |
| rlandy|rover | ysandeep: got to go into meetings now | 12:59 |
| ysandeep | rlandy|rover, no worries, I will keep checking | 12:59 |
| rlandy|rover | https://logserver.rdoproject.org/55/36255/38/check/tripleo-ci-centos-9-content-provider/4716af7/ | 13:00 |
| rlandy|rover | ysandeep: ^^ failed | 13:00 |
| rlandy|rover | node is held | 13:00 |
| rlandy|rover | you can ask to add your keys | 13:00 |
| rlandy|rover | soniya29|ruck: ^^ | 13:00 |
| soniya29|ruck | rlandy|rover, should i add my keys? | 13:01 |
| rlandy|rover | soniya29|ruck: only of ysandeep and you want ro investigate that | 13:01 |
| rlandy|rover | pojadhav: 1-1? | 13:01 |
| pojadhav | rlandy|rover, just 5 mins | 13:02 |
| pojadhav | just back from office... | 13:02 |
| soniya29|ruck | rlandy|rover, ack | 13:02 |
| ysandeep | rlandy|rover, pinged infra to add my keys to debug... checking original approach as well. | 13:02 |
| rlandy|rover | https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835222 | 13:03 |
| rlandy|rover | jm1 is also getting in on the act here | 13:03 |
| jm1 | rlandy|rover: you dont even give me some time to explain :P | 13:04 |
| rlandy|rover | soniya29|ruck: pls work with ysandeep here | 13:04 |
| pojadhav | rlandy|rover, hey joining | 13:04 |
| soniya29|ruck | rlandy|rover, ack | 13:04 |
| jm1 | ysandeep: https://github.com/pypa/setuptools/issues/3197 suggests to set include_package_data to False which is what my patch tries to test https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835222 | 13:04 |
| ysandeep | jm1: check next comment - I tried setting include_package_data to False but it does not help. | 13:06 |
| ysandeep | jm1: this on the other hand worked: one way to disable auto discovery is to explicitly set py_modules=[] in setup.py | 13:06 |
| ysandeep | jm1, but the thing is we have multiple repos so we have to patch all of them. | 13:07 |
| ysandeep | jm1: let me try - include_package_data to False in my local env as well | 13:08 |
| jm1 | ysandeep: the author of this comment on github is not clear about what he really did because the patch he linked does something else | 13:08 |
| soniya29|ruck | ysandeep, how can i help you with this issue? | 13:09 |
| jm1 | ysandeep: dont be distracted by my patch, i did not want to raise awareness until i got a result ^^ | 13:10 |
| ysandeep | soniya29|ruck: have you already read https://github.com/pypa/setuptools/issues/3197 and our bug? | 13:10 |
| ysandeep | soniya29|ruck: https://review.opendev.org/q/topic:lp%252F1966382 | 13:10 |
| ysandeep | we are trying two different approach | 13:10 |
| soniya29|ruck | ysandeep, i have gone through bug | 13:11 |
| ysandeep | 1) pin setuptools to older good known - didn't work for us - need to investigate why that happened | 13:11 |
| ysandeep | 2) trying workaround mentioned in github issue -> disable auto discovery is to explicitly set py_modules=[] in setup.py | 13:11 |
| soniya29|ruck | ysandeep: okay | 13:13 |
| ysandeep | jm1, thanks man, i tried your patch, facing same issue: http://pastebin.test.redhat.com/1039905 | 13:19 |
| jpodivin | ysandeep: it's really strange it didn't work, the right setuptools were installed, but were not used. I'm guessing that package from another source must have got into the env. | 13:25 |
| ysandeep | jpodivin, yeah that's what I am wondering, I have pinged infra to add my keys on node.. pip list will reveal | 13:26 |
| jpodivin | ysandeep: if site packages are allowed ... all bets are off. That way you can get pretty much anything. | 13:26 |
| jm1 | ysandeep: your patch worked: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835178/2 | 13:26 |
| jm1 | ..latest patchset | 13:27 |
| ysandeep | jm1: yes | 13:27 |
| ysandeep | jm1, problem is now failing on tqe.. and i am patching tqe now.. so we might need to fix multiple repos | 13:28 |
| ysandeep | jm1: to fix early was thinking to pin setuptools instead | 13:28 |
| jm1 | ysandeep: saw the discussion ^^ | 13:31 |
| rlandy|rover | ysandeep: ok - back - I guess we're down to patching all the repos? | 13:32 |
| ysandeep | rlandy|rover, yes | 13:32 |
| rlandy|rover | if so, let's get started with a merge order | 13:32 |
| rlandy|rover | ysandeep: soniya29|ruck: let's meet and get a merge order sorted | 13:32 |
| rlandy|rover | jpodivin: ^^ feel free to join us | 13:32 |
| rlandy|rover | jm1 | 13:32 |
| rlandy|rover | https://meet.google.com/jzs-rtrx-nvi?pli=1&authuser=0 | 13:33 |
| *** amoralej is now known as amoralej|lunch | 13:36 | |
| jm1 | ysandeep: github confirms "I can confirm that settings include-package-data in pyproject.toml does indeed work." | 13:48 |
| ysandeep | jm1: http://pastebin.test.redhat.com/1039929 | 14:04 |
| *** amoralej|lunch is now known as amoralej | 14:17 | |
| ysandeep | ohh man, when this will start: https://zuul.openstack.org/status#835180 | 14:21 |
| rlandy|rover | ysandeep: kicked testproject: https://review.rdoproject.org/r/c/testproject/+/36255 | 14:25 |
| rlandy|rover | with tq and tqe changes | 14:25 |
| rlandy|rover | we can take our chances on which one will start first | 14:26 |
| rlandy|rover | jpodivin: ^^ fyi | 14:26 |
| ysandeep | ack, i don't have tq one in current one - https://zuul.openstack.org/status#835180 - will add that when i hit issue on tq.. going step by step | 14:26 |
| jpodivin | The zuul does use some randomization to assign resources ... but I think it also tries to balance projects so it wouldn't be a fair bet. | 14:27 |
| jm1 | ysandeep: imho your approach to disable automatic discovery is the first and most important step to apply in all affected projects. setting include-package-data to false will not help us. We actually want to include data, e.g. we install roles in quickstart-extras | 14:28 |
| ysandeep | jm1: ack, thanks man! | 14:30 |
| rlandy|rover | cool | 14:30 |
| rlandy|rover | testproject running now | 14:30 |
| ysandeep | I just hope we don't break anything else.. | 14:30 |
| jm1 | ysandeep: additionally we probably want to get rid of pyproject.toml in its current form because its violating pep621, as you can see in your logs: http://pastebin.test.redhat.com/1039905 | 14:30 |
| * ysandeep afk for few mins to order some food.. Got too busy today to cook :D | 14:31 | |
| ysandeep | jm1: +1 to get rid of http://pyproject.toml | 14:31 |
| jpodivin | jm1: agreed, either we should use it properly or not at all | 14:32 |
| ysandeep | pyproject.toml | 14:32 |
| rlandy|rover | https://review.rdoproject.org/zuul/stream/db415cd0fbe4456ea7c51f191f45524a?logfile=console.log | 14:32 |
| rlandy|rover | OMG | 14:32 |
| rlandy|rover | we have a winner | 14:32 |
| rlandy|rover | ysandeep: ^^ yay you!! | 14:33 |
| ysandeep | wohooo | 14:33 |
| rlandy|rover | jpodivin: ^^ jm1: ^^ | 14:33 |
| rlandy|rover | happy dance!!! | 14:33 |
| jpodivin | yay | 14:33 |
| ysandeep | upstream triggered as well... let's see if just tqe patch can fix it | 14:33 |
| rlandy|rover | yep | 14:33 |
| rlandy|rover | just saw that | 14:33 |
| ysandeep | marios|out knewed it was going to happen, he took PTO on right day :D just kidding | 14:35 |
| ysandeep | rlandy|rover, wohooo. just tqe will fix things | 14:36 |
| ysandeep | but we should add workaround in tq still for sanity | 14:36 |
| rlandy|rover | ysandeep: awesome - will get that merged | 14:36 |
| ysandeep | rlandy|rover, merge order -> tqe then tq(just for sanity - not causing issues currently) | 14:36 |
| rlandy|rover | ysandeep: got it | 14:37 |
| rlandy|rover | voting on both | 14:37 |
| ysandeep | :D | 14:37 |
| jm1 | ysandeep, jpodivin: first patch to get rid of pyproject.toml https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835222/ | 14:37 |
| ysandeep | next week - when we have time and more comments on actual github issue - I think worth to investigate from where setuptools:61.0 is coming. | 14:38 |
| *** dasm|off is now known as dasm | 14:38 | |
| dasm | o/ | 14:38 |
| jpodivin | ysandeep: agreed, let's hope it's really just confined to that one version. | 14:39 |
| rlandy|rover | dasm: hey | 14:39 |
| raukadah | ysandeep: with this https://zuul.openstack.org/status#835192 and tq patch will fix content provider https://zuul.openstack.org/stream/cab8e9f43e124b25b08e23f39faead6c?logfile=console.log | 14:39 |
| rlandy|rover | dasm: looks like you made uefi boot progress | 14:40 |
| ysandeep | raukadah, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835203 just need this. | 14:40 |
| jpodivin | jm1: voted | 14:40 |
| ysandeep | raukadah, tripleo-ci and ansible-collect already merged | 14:41 |
| rlandy|rover | https://review.opendev.org/q/topic:lp%252F1966382 | 14:41 |
| raukadah | ysandeep: cool! | 14:41 |
| rlandy|rover | ysandeep: ^^ +2 voted on tq and tqe | 14:41 |
| rlandy|rover | raukadah: hey | 14:41 |
| ysandeep | rlandy|rover, raukadah okay if I abandon pinning one to avoid confusion? | 14:41 |
| raukadah | ysandeep: yes please | 14:41 |
| rlandy|rover | ysandeep: ack pls | 14:41 |
| raukadah | rlandy|rover: hello | 14:41 |
| rlandy|rover | let's keep this mess as simple as possible | 14:41 |
| rlandy|rover | raukadah: how's the wedding? | 14:42 |
| raukadah | rlandy|rover: wedding is on 29th Mar | 14:42 |
| raukadah | today i will be going to Amol's home town | 14:42 |
| raukadah | will reach tomorrow morning there | 14:42 |
| rlandy|rover | ok | 14:42 |
| * raukadah out now | 14:43 | |
| rlandy|rover | removing the node hold | 14:43 |
| rlandy|rover | ysandeep: ^^ | 14:44 |
| ysandeep | rlandy|rover, wait | 14:44 |
| rlandy|rover | ok - we can keep it | 14:44 |
| ysandeep | rlandy|rover, I want to debug there | 14:44 |
| rlandy|rover | sure | 14:44 |
| rlandy|rover | sorry | 14:44 |
| ysandeep | rlandy|rover: thanks! | 14:44 |
| ysandeep | rlandy|rover, /me afk.. will check back in an hour before logging off.. I am happy we managed to fix this before weekend. | 14:45 |
| ysandeep | thanks everyone.. team-work++ | 14:45 |
| rlandy|rover | ysandeep+++++++++++ | 14:45 |
| rlandy|rover | I'll merge those two | 14:45 |
| rlandy|rover | when zuul reports | 14:45 |
| ysandeep | sure | 14:45 |
| *** ysandeep is now known as ysandeep|dinner | 14:46 | |
| jm1 | jpodivin, ysandeep: for tripleo-ci we wont be able to get rid of pyproject.toml because black only supports TOML file https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html | 14:46 |
| dasm | rlandy|rover: yeah! atfer Steve's help, it all worked well. I'm gonna check few more things, but it looks like we're good to go. I saw it booting, picking up pxe bits and executing tests. | 14:46 |
| soniya29|ruck | *leaving for the day | 14:47 |
| jpodivin | jm1: hm, well that's unfortunate. But at least we know this now. | 14:47 |
| dasm | jm1: do we use black for tripleo-ci? til | 14:48 |
| rlandy|rover | dasm: OMG awesome news!!! | 14:48 |
| jpodivin | jm1: we can adjust it so it works properly | 14:48 |
| rlandy|rover | dasm++ thank you for sticking with this | 14:48 |
| dasm | no worries | 14:51 |
| jm1 | dasm: tripleo-ci is using black in .pre-commit-config.yaml ^^ | 14:54 |
| dasm | ack. | 14:55 |
| dasm | i see i missed some 'fun'? https://bugs.launchpad.net/tripleo/+bug/1966382 | 15:23 |
| jpodivin | dasm: I wouldn't say missed. More like arrived in the middle of . | 15:27 |
| dasm | ack. i saw bunch of packages. btw, setuptools reverted change. | 15:28 |
| dasm | https://github.com/pypa/setuptools/pull/3207/files | 15:28 |
| rlandy|rover | lol | 15:28 |
| dasm | btw2 who uses private api? hm? ;) | 15:28 |
| rlandy|rover | after all that | 15:28 |
| dasm | > 9 minutes ago | 15:28 |
| rlandy|rover | we're just going with these patches | 15:29 |
| rlandy|rover | in case they put it back | 15:29 |
| dasm | s/packages/changes | 15:29 |
| dasm | makes sense. it's deprecated. so it can go away any time soon | 15:29 |
| jpodivin | dasm: out of curiosity, do you know any places where we are using pyproject.toml? I tried looking with hound and such but had far fewer hit than I thought. | 15:31 |
| dasm | jpodivin: no idea. i just learned we use it through black, when jm1 mentioned about that | 15:31 |
| jpodivin | jm1: btw your patch just passed the check https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/835222 | 15:32 |
| jpodivin | dasm: ok. There don't seem to be that many, actually I've only found those we already know about. | 15:33 |
| dasm | i don't see it being used in a lot of places too. | 15:33 |
| jpodivin | There is pbr, but only hypothetically. It's kept there only as a future starting point. Still, as the project doesn't have any modules it could cause problems. | 15:38 |
| *** ysandeep|dinner is now known as ysandeep | 15:38 | |
| jpodivin | I'm pushing patch with the workaround rather than removal for the pbr, feels more appropriate than removing a file that's not being used. | 15:39 |
| ysandeep | rlandy|rover: green on complete run: so we know it works :) | 15:42 |
| ysandeep | tripleo-ci-centos-9-content-provider https://zuul.opendev.org/t/openstack/build/385fd57947914890b4032a4477325561 : SUCCESS in 1h 06m 57s | 15:42 |
| rlandy|rover | ysandeep: both w+'ed | 15:43 |
| ysandeep | nice, have a good weekend guys.. See you on Monday o/ | 15:44 |
| *** ysandeep is now known as ysandeep|out | 15:44 | |
| jpodivin | ysandeep|out: see ya | 15:45 |
| rlandy|rover | lunch brb | 15:53 |
| *** amoralej is now known as amoralej|off | 16:10 | |
| *** jpena is now known as jpena|off | 17:06 | |
| dasm | rlandy|rover: o/ any pros/cons regarding switching all ovb jobs from legacy boot (ipxe-boot) to uefi (ipxe-uefi)? The change is simple on vexx side: https://paste.opendev.org/show/bV5gCGrucPDIeg5QZ1i2/ | 18:21 |
| dasm | if we're gonna swith globally -- we're immediately testing new uefi image | 18:21 |
| dasm | cons: it can get borked immediately killing all our ovb jobs | 18:22 |
| dasm | based on my hound-fu i only found ipxe-boot image being used in ovb jobs. It's also simpler on vexx side than updating all references in repos | 18:26 |
| rcastillo | dasm: we could do a tp on the jobs we'd change with a different baremetal_image set, right? | 18:28 |
| dasm | according to what i found -- that was it: https://review.rdoproject.org/r/c/testproject/+/40332/5/.zuul.yaml | 18:29 |
| dasm | baremetal_image_name | 18:29 |
| dasm | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/generate_template.yml#L22 and other places with baremetal_image_name | 18:30 |
| dasm | default value is 'ipxe-boot' | 18:30 |
| dasm | we could implement the change at Monday to deal with (let's hope not) any issues | 18:32 |
| rlandy|rover | dasm: a few things ... | 18:34 |
| rlandy|rover | ovb jobs run on PSI as well | 18:35 |
| * dasm listening | 18:35 | |
| rlandy|rover | uefi boot is not supported in train/ussuri/victoria | 18:35 |
| dasm | hmmm | 18:35 |
| rlandy|rover | for master/wallaby whole sale switching should be ok | 18:35 |
| dasm | that complicates my easy switch | 18:35 |
| rlandy|rover | if you have tested fss01, 02, 039 064 and 035 | 18:35 |
| rlandy|rover | dasm; not really | 18:36 |
| rlandy|rover | just switch case it for releases | 18:36 |
| dasm | no, i just ran fs01 for c9-master and c8-wallaby | 18:36 |
| rlandy|rover | you will see that in a lot of fs files | 18:36 |
| dasm | mhm | 18:36 |
| rlandy|rover | dasm: suggestion: | 18:36 |
| rlandy|rover | qualify all above fss01, 02, 039 064 and 035 for master and wallaby | 18:37 |
| rlandy|rover | if they all pass | 18:37 |
| rlandy|rover | then switch the image for those release | 18:37 |
| rlandy|rover | on vexx only | 18:37 |
| rlandy|rover | until psi on 17 is qualified | 18:37 |
| rlandy|rover | it's not that compliacted ... | 18:37 |
| rlandy|rover | the settings per cloud are in one place | 18:38 |
| dasm | but we're still running train, ussuri and victoria on vexx? | 18:38 |
| rlandy|rover | dasm: we are | 18:38 |
| rlandy|rover | and train is a long support release | 18:38 |
| dasm | so i would need to check it on fs-basis | 18:38 |
| rlandy|rover | train ->16.2 | 18:38 |
| dasm | ack | 18:38 |
| rlandy|rover | dasm: do above steps make sense? | 18:38 |
| dasm | yeah | 18:39 |
| dasm | lemme see what i can do with that | 18:39 |
| dasm | thanks rlandy|rover++ | 18:40 |
| rlandy|rover | dasm: great - and awesome news on getting this sorted | 18:40 |
| rlandy|rover | it was nt easy | 18:40 |
| rlandy|rover | and you cracked it | 18:40 |
| rlandy|rover | nicely done!!!! | 18:40 |
| dasm | ~o~ | 18:40 |
| rlandy|rover | really made my day | 18:40 |
| rlandy|rover | and it was a rough day | 18:40 |
| dasm | one fix at a time | 18:41 |
| rlandy|rover | vexx is still unhappy :( | 18:43 |
| rlandy|rover | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/06616fc/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 18:43 |
| dasm | :/ | 18:43 |
| rlandy|rover | whatever - have some ptg things to do | 19:05 |
| rlandy|rover | now seems like a good time - the cloud decided | 19:05 |
| dasm | The Cloud Has Spoken | 19:09 |
| dasm | i started bunch of jobs. some of them will definitely fail due to non-uefi. However I want to get the confirmation for all of them. | 19:12 |
| dasm | By Monday I should have better understanding of these relations/requirements. | 19:13 |
| dasm | Have a wonderful weekend Team! | 19:13 |
| dasm | o/ | 19:13 |
| jm1 | dasm: Have a nice weekend :) | 19:13 |
| * dasm slowly disappears | 19:14 | |
| dasm | jm1: it's Friday night. You don't have better things to do? :) | 19:14 |
| rcastillo | have a good weekend dasm | 19:14 |
| dasm | Like Netflix and chill? | 19:14 |
| dasm | rcastillo: you as well. And you too jm1! | 19:14 |
| *** dasm is now known as dasm|off | 19:14 | |
| * dasm|off is going camping! First time in a camper! | 19:15 | |
| * jm1 out for today, happy friday ^^ | 20:55 | |
| rlandy|rover | see you all monday | 21:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!