| *** rlandy|bbl is now known as rlandy|out | 00:03 | |
| *** ysandeep|out is now known as ysandeep|rover | 01:21 | |
| ysandeep|rover | rlandy|out, wohoo we know why only master was affected - https://opendev.org/openstack/tripleo-ansible/commit/b4e23529891eccd83e1e313e871e299b16ed93e7 only in master | 02:07 |
|---|---|---|
| ysandeep|rover | rlandy|out, https://review.opendev.org/c/openstack/tripleo-ansible/+/841360 - will fix master hopefully , looks good based on tests in my env. | 02:08 |
| * ysandeep|rover woke up early to catch with steve, afk - back in few hours, | 02:08 | |
| *** ysandeep|rover is now known as ysandeep|rover|bbl | 02:08 | |
| *** dasm|bbl is now known as dasm|off | 03:40 | |
| *** bhagyashris|out is now known as bhagyashris | 03:58 | |
| bhagyashris | is downstream gerrit working? | 04:16 |
| *** ysandeep|rover|bbl is now known as ysandeep|rover | 04:23 | |
| ysandeep|rover | bhagyashris, let me check | 04:23 |
| ysandeep|rover | bhagyashris, yeah not working | 04:26 |
| bhagyashris | yeah | 04:26 |
| ysandeep|rover | bhagyashris, forwared an email, i think that schedule maintenance is in play here. | 04:27 |
| bhagyashris | ysandeep|rover, ohh i see | 04:27 |
| *** soniya29 is now known as soniya29|ruck | 04:51 | |
| soniya29|ruck | ysandeep|rover, can you connect with vpn? | 04:51 |
| ysandeep|rover | soniya29|ruck, yes I am already connected | 04:53 |
| soniya29|ruck | ysandeep|rover, i am facing certain issues with it | 04:53 |
| ysandeep|rover | soniya29|ruck, there is an ongoing scheduled maintenance that might have caused it | 04:54 |
| ysandeep|rover | soniya29|ruck, check outage list | 04:54 |
| soniya29|ruck | ysandeep|rover, yeah, many folks are unable to connect to vpn and sso services it seems | 04:55 |
| ysandeep|rover | yeah sso broke by maintenance | 04:56 |
| ysandeep|rover | soniya29|ruck, Could you please report this and investigate https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&project=openstack%2Ftripleo-heat-templates&branch=stable%2Fwallaby&skip=0 , I need to track master promotion. | 05:02 |
| soniya29|ruck | ysandeep|rover, sure | 05:03 |
| ysandeep|rover | soniya29|ruck, looks like failing only for wallaby in tht | 05:03 |
| soniya29|ruck | ysandeep|rover, okay | 05:05 |
| soniya29|ruck | ysandeep|rover, it is getting timed_out | 05:06 |
| ysandeep|rover | yup | 05:06 |
| ysandeep|rover | I have retrigger pep8 on one job - let's see how that goes:- https://zuul.opendev.org/t/openstack/status#840194 | 05:07 |
| ysandeep|rover | https://zuul.opendev.org/t/openstack/stream/b5914211bfaf4562ab98bb67a43a9cbf?logfile=console.log | 05:08 |
| soniya29|ruck | ysandeep|rover, is it a promotion blocker? | 05:13 |
| ysandeep|rover | soniya29|ruck, yes as its affecting patches from getting merged. | 05:13 |
| soniya29|ruck | ysandeep|rover, https://bugs.launchpad.net/tripleo/+bug/1972940 | 05:14 |
| ysandeep|rover | soniya29|ruck, thanks left a comment - INFO: pip is looking at multiple versions of passlib to determine which version is compatible with other requirements. This could take a while. - this was not seen in last good run | 05:17 |
| soniya29|ruck | ysandeep|rover, okay | 05:19 |
| ysandeep|rover | soniya29|ruck, hey o/ yesterday during rr sync did you mentioned hitting during this task: FATAL | Collect OpenStack Keystone domains infos | 05:55 |
| ysandeep|rover | https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-clients-master/e03f708/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 05:55 |
| ysandeep|rover | hitting error* during this task | 05:56 |
| soniya29|ruck | ysandeep|rover, yeah its the same thing | 05:59 |
| soniya29|ruck | ysandeep|rover, nAttributeError: 'Munch' object has no attribute 'to_dict'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n | 05:59 |
| ysandeep|rover | soniya29|ruck, did you reported a bug already for this, If not I can. | 05:59 |
| soniya29|ruck | ysandeep|rover, nope, since it was since just in 1 run, i didn't reported it | 06:00 |
| soniya29|ruck | s/since/seen | 06:00 |
| ysandeep|rover | soniya29|ruck: ack, I will report it, thanks! | 06:00 |
| soniya29|ruck | ysandeep|rover, fs01 and fs35 are failing on wallaby c8 | 06:12 |
| soniya29|ruck | ysandeep|rover, should we remove them from criteria? | 06:13 |
| soniya29|ruck | ysandeep|rover, rekicked those jobs here:- https://review.rdoproject.org/r/c/testproject/+/40101 | 06:16 |
| ysandeep|rover | soniya29|ruck, trying to fix tht bug with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841280 , let see how it goes | 06:16 |
| ysandeep|rover | soniya29|ruck: should we remove them from criteria >> no if we don't have a bug and valid reason to skip the criteria | 06:17 |
| soniya29|ruck | ysandeep|rover, they are blocking promotion | 06:18 |
| ysandeep|rover | soniya29|ruck, do we have a bug on why there are failing? | 06:19 |
| soniya29|ruck | ysandeep|rover, nope, i will report that, fs01 wallaby is failing on tempest tests wit Server e299c58d-5d9d-412c-866f-1a9f03077cf4 failed to reach ACTIVE status | 06:23 |
| soniya29|ruck | s/wit/with | 06:24 |
| ysandeep|rover | soniya29|ruck, before reporting please check its not duplicate of https://bugs.launchpad.net/tripleo/+bug/1964940 | 06:27 |
| soniya29|ruck | ysandeep|rover, its a duplicate, thanks for pointing it out | 06:30 |
| soniya29|ruck | let me comment on the bug | 06:30 |
| soniya29|ruck | ysandeep|rover, shall we sync? | 06:39 |
| ysandeep|rover | soniya29|ruck, meet.google.com/bme-ysrm-wrn | 06:40 |
| ysandeep|rover | soniya29|ruck, https://zuul.openstack.org/stream/a60f389edb224537bb0e569efee78845?logfile=console.log | 06:47 |
| jm1 | ysandeep|rover: hey hey :) do you have an idea why c8 standalone jobs fail with "Failed to pull image trunk.registry.rdoproject.org/tripleotraincentos8/..."? | 06:56 |
| ysandeep|rover | jm1, logs | 06:56 |
| jm1 | ysandeep|rover: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_057/periodic/opendev.org/openstack/ansible-collections-openstack/stable/1.0.0/tripleo-ci-centos-8-standalone-train-osa/057882f/ | 06:57 |
| jm1 | ysandeep|rover: or maybe you could give me a hint where to search | 06:57 |
| ysandeep|rover | looks like you don't use content-provider jobs here and pulling from rdo registry | 07:00 |
| jm1 | ysandeep|rover: our jobs are based on tripleo-ci-base-standalone-centos-8 and we do not alter that much https://opendev.org/openstack/ansible-collections-openstack/src/commit/acf64a1f72bf10e0e332899168e468db5c8ba034/.zuul.yaml#L488 | 07:02 |
| jm1 | ysandeep|rover: by looking at the job def in .zuul.yaml can you spot something odd at first glance? | 07:02 |
| jm1 | maybe "consumer_job: false"? | 07:03 |
| ysandeep|rover | jm1, that hash container are not present in rdo registry anymore.. | 07:03 |
| ysandeep|rover | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_057/periodic/opendev.org/openstack/ansible-collections-openstack/stable/1.0.0/tripleo-ci-centos-8-standalone-train-osa/057882f/logs/undercloud/home/zuul/containers-prepare-parameters.yaml - 2932024d76f6fdfa72b5f5b44a487074 is not current-tripleo hash for tripleo | 07:03 |
| ysandeep|rover | https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5 - a70f6f792d4efabf990f657dfc9c084e | 07:03 |
| ysandeep|rover | let me check if that's the right namespace for train | 07:04 |
| ysandeep|rover | jm1, ahh looks like we are trying to pull container with wrong prefix:- http://pastebin.test.redhat.com/1050960 | 07:07 |
| ysandeep|rover | openstack vs centos | 07:07 |
| ysandeep|rover | jm1: in rdo we are building container with centos prefix: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/6b29efe/logs/containers-built.log | 07:07 |
| * ysandeep|rover comparing a check job | 07:08 | |
| ysandeep|rover | jm1: https://dcf277031c0d2af0f2e0-e099c5c03695c7198c297e75ec3f8d05.ssl.cf5.rackcdn.com/841347/2/check/tripleo-ci-centos-8-standalone/ab2447e/logs/undercloud/home/zuul/containers-prepare-parameters.yaml : name_prefix: centos-binary | 07:09 |
| jm1 | ysandeep|rover: good catch! we dont set the prefix in our jobs, do we? | 07:12 |
| jm1 | ysandeep|rover: we are talking about container_name_prefix, dont we? https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/defaults/main.yaml | 07:20 |
| ysandeep|rover | that's while building container, we need to check what creates containers-prepare-parameters.yaml | 07:21 |
| ysandeep|rover | jm1, docker_prep_prefix | 07:24 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 07:29 | |
| jm1 | ysandeep|rover: hmm.. has not changed for 1 year.. dont think our job failed for one year but who knows https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build/defaults/main.yaml#L23 | 07:29 |
| bhagyashris | marios, hey once you have retro board ready let me know i will update the retro board link in the invitation | 07:29 |
| jm1 | ysandeep|rover|brb: the last related commit which changed docker_prep_prefix says that "openstack" is used since victoria. does this mean it must not be openstack for older releases? If so, shouldn't it be "and release not in" instead of "or release not in"? | 07:40 |
| *** soniya is now known as soniya|ruck | 07:40 | |
| jm1 | chandankumar: ^ | 07:40 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 07:40 | |
| * chandankumar reads | 07:41 | |
| jm1 | ysandeep|rover, chandankumar: here is the commit : https://opendev.org/openstack/tripleo-quickstart-extras/commit/b71b15167cac4b137667d9c88af6b273df08b072 | 07:42 |
| ysandeep|rover | jm1: In middle of something, will try to check soon | 07:42 |
| jm1 | ysandeep|rover: ack, thanks for helping out! | 07:43 |
| ysandeep|rover | soniya|ruck, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841282 will fix tht. | 07:48 |
| marios | bhagyashris: ack https://miro.com/app/board/uXjVO1hv5Uk=/ thank you | 07:48 |
| soniya|ruck | ysandeep|rover, thats great | 07:49 |
| chandankumar | jm1: Let me play with the job definitions, I think parenting might screwed up | 07:49 |
| chandankumar | jm1: https://opendev.org/openstack/ansible-collections-openstack/src/commit/acf64a1f72bf10e0e332899168e468db5c8ba034/.zuul.yaml#L437 | 07:49 |
| chandankumar | we need to fix container prepare parameter like this https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-train/39a7ab2/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 07:50 |
| *** soniya|ruck is now known as soniya|ruck|dinner | 07:51 | |
| *** soniya|ruck|dinner is now known as soniya|ruck|lunch | 07:53 | |
| bhagyashris | marios, thanks! Updated | 07:55 |
| marios | thanks bhagyashris | 07:56 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:01 | |
| jm1 | chandankumar: hm... not sure what to do exactly. | 08:06 |
| chandankumar | jm1: in these jobs, container are built with in the job itself due to https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/.zuul.yaml#L303 | 08:08 |
| chandankumar | and https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_057/periodic/opendev.org/openstack/ansible-collections-openstack/stable/1.0.0/tripleo-ci-centos-8-standalone-train-osa/057882f/logs/undercloud/home/zuul/container_image_build.log | 08:09 |
| chandankumar | as ysandeep|rover|lunch said, we first need to fix the container namespace | 08:10 |
| jm1 | chandankumar: afaiu it has to be changed from openstack to centos-*. | 08:10 |
| jm1 | chandankumar: ..which is set by docker_prep_prefix | 08:11 |
| chandankumar | jm1: can you change the condition https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build/defaults/main.yaml#L23 like this | 08:12 |
| chandankumar | if ansible distribution is c7 then use openstack prefix else centos-binary | 08:12 |
| chandankumar | openstack namespace was used with kolla and with tcib it is used with centos-binary | 08:13 |
| chandankumar | * if ansible distribution is c7 and relase is train then use openstack prefix else centos-binary | 08:13 |
| chandankumar | jm1: can you propose a fix then we can dig further | 08:13 |
| jm1 | chandankumar: you mean "or release not in" to "or (ansible_distribution_version == "7" and release not in ...)"? | 08:15 |
| chandankumar | jm1: if (ansible_distribution_major_version is version(7, '==') and release in ['train']) | 08:18 |
| chandankumar | you can fix the ansible syntax | 08:18 |
| jm1 | chandankumar: but should we leave the "build_container_images|default(false)|bool and ..." conditional? | 08:20 |
| chandankumar | jm1: it will not break anything, if we keep those conditions or not | 08:21 |
| jm1 | chandankumar: ack | 08:23 |
| jm1 | chandankumar: if i leave the conditions, then we will still use "openstack" because "build_container_images: true" and "provider_job" is not defined (hence not provider_job is true) | 08:28 |
| jm1 | chandankumar: so maybe we have to change the "or" operator to an "and" operator? | 08:29 |
| chandankumar | jm1: https://paste.opendev.org/raw/bqDmqiwPrv2LWvPnpAIB/ it will handle all the case of content provider | 08:32 |
| chandankumar | or the container images built with in the job or not | 08:32 |
| chandankumar | for c7 train, we used kolla method where namespace with openstack and apart from other release it was centos-binary | 08:33 |
| chandankumar | due to tcib (tripleo container image build cli) | 08:33 |
| jm1 | chandankumar: okay, thanks! | 08:34 |
| jm1 | chandankumar: feel free to edit https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841381 | 08:39 |
| *** soniya|ruck|lunch is now known as soniya|ruck | 08:56 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:02 | |
| chandankumar | jm1: testing here: https://review.opendev.org/c/openstack/ansible-collections-openstack/+/841283/ | 09:06 |
| jm1 | chandankumar: hehe, thanks ^^ | 09:11 |
| * ysandeep|rover reading back | 09:14 | |
| ysandeep|rover | if we are building container in job itself then prefix is right. | 09:15 |
| ysandeep|rover | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_057/periodic/opendev.org/openstack/ansible-collections-openstack/stable/1.0.0/tripleo-ci-centos-8-standalone-train-osa/057882f/logs/undercloud/home/zuul/container_image_build.log | 09:15 |
| ysandeep|rover | job is building container with openstack prefix. | 09:15 |
| ysandeep|rover | namespace: trunk.registry.rdoproject.org/tripleotraincentos8 ---> this is wrong in that case | 09:16 |
| ysandeep|rover | namespace should be 127.0.0.1:5001/tripleotraincentos8 | 09:16 |
| ysandeep|rover | docker_registry_host | 09:17 |
| ysandeep|rover | jm1, chandankumar https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/train.yml#L13 | 09:18 |
| ysandeep|rover | soniya|ruck, We have cix call today, please update the cards on which you are working, I will do the same. | 09:21 |
| soniya|ruck | ysandeep|rover, sure | 09:21 |
| jm1 | ysandeep|rover: oha, ok so something else in tripleo ci jobs is broken | 09:27 |
| ysandeep|rover | worth to check which release file we use in osa jobs | 09:29 |
| ysandeep|rover | is it really https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/train.yml ? | 09:29 |
| ysandeep|rover | aha, i found something | 09:31 |
| ysandeep|rover | jm1, chandankumar https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_057/periodic/opendev.org/openstack/ansible-collections-openstack/stable/1.0.0/tripleo-ci-centos-8-standalone-train-osa/057882f/job-output.txt | 09:31 |
| ysandeep|rover | (./toci_quickstart.sh:46): main(): export 'QUICKSTART_DEFAULT_RELEASE_ARG=--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-train.yml' | 09:32 |
| ysandeep|rover | we are wrong release file for check - promotion-testing-hash-train.yml instead of train.yml | 09:32 |
| ysandeep|rover | we are using* | 09:32 |
| ysandeep|rover | jm1, chandankumar https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-train.yml#L7 , docker_registry_host: trunk.registry.rdoproject.org is hardcoded for promotion-testing-hash-train.yml case | 09:33 |
| ysandeep|rover | promotion-testing-hash-train.yml should only be used for integration line jobs | 09:34 |
| *** jpena|off is now known as jpena | 09:39 | |
| ysandeep|rover | jm1: compare with tripleo-ci job - https://dcf277031c0d2af0f2e0-e099c5c03695c7198c297e75ec3f8d05.ssl.cf5.rackcdn.com/841347/2/check/tripleo-ci-centos-8-standalone/ab2447e/job-output.txt - export 'QUICKSTART_DEFAULT_RELEASE_ARG=--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-8/train.yml' | 09:39 |
| jm1 | ysandeep|rover: maybe because of this? https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-build-jobs-repos/tasks/load-vars.yaml#L13 | 09:40 |
| jm1 | ysandeep|rover: we run it as a periodic job | 09:40 |
| ysandeep|rover | ohh, you running this job in periodic line | 09:42 |
| ysandeep|rover | wait there is a var to fix that | 09:42 |
| ysandeep|rover | jm1, force_non_periodic https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/ci-common-vars/vars/main.yaml#L24 | 09:43 |
| ysandeep|rover | jm1: so you need to do something like this: https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/periodic.yaml#L27 | 09:43 |
| ysandeep|rover | that's what we are already doing in our periodic jobs. | 09:44 |
| jm1 | ysandeep|rover: omg what a hack.. | 09:44 |
| ysandeep|rover | :) | 09:44 |
| jm1 | ysandeep|rover: okay will update all jobs. thank you very much for figureing that out :) | 09:44 |
| jm1 | chandankumar: thank you as well for jumping on this :) | 09:45 |
| ysandeep|rover | jm1: happy to help o/, let me know if it does the trick | 09:45 |
| jm1 | ysandeep|rover: sure i will ;) | 09:45 |
| jm1 | ysandeep|rover: the last successful tripleo jobs in our periodic pipeline has been so long ago that zuul's build history has no entries about it anymore😱 | 09:58 |
| ysandeep|rover | :) | 10:07 |
| marios | folks please see email "retrospective today - please read and respond asap" thanks | 10:18 |
| marios | arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, dviroel, rcastillo, jm1, rlandy|ruck, dasm please see email "retrospective today - please read and respond asap" thank you o/ | 10:19 |
| ysandeep|rover | marios, o/ replied already | 10:19 |
| marios | ysandeep|rover: ack thanks point was to ping everybody to not reveal who already replied as i added them on the board :D | 10:20 |
| marios | thanks to all those who already replied o/ | 10:20 |
| ysandeep|rover | marios: in that case i haven't replied please disregard.. I thought you are talking about some other email :D | 10:22 |
| marios | ysandeep|rover: right i thought so thanks for clarifying | 10:23 |
| marios | ysandeep|rover: i'll look into it and will raise a ticket with support if needed | 10:23 |
| marios | ysandeep|rover: ;) | 10:23 |
| ysandeep|rover | hahaha, lol! | 10:23 |
| ysandeep|rover | soniya|ruck, do you want to represent at program call? otherwise I can. | 10:24 |
| soniya|ruck | ysandeep|rover, you can go ahead :) | 10:25 |
| ysandeep|rover | ack | 10:25 |
| rlandy|out | ysandeep|rover: sounds good | 10:27 |
| rlandy|out | ysandeep|rover: soniya|ruck: let's sync ... | 10:28 |
| rlandy|out | ysandeep|rover: soniya|ruck: https://meet.google.com/rno-jhjr-bme?pli=1&authuser=0 | 10:29 |
| soniya|ruck | ykarel, can you have a look on this https://bugs.launchpad.net/tripleo/+bug/1972163? | 10:34 |
| soniya|ruck | slaweq, ^^ | 10:38 |
| chandankumar | jm1: I was checking check jobs | 10:38 |
| chandankumar | jm1: can we restore this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841381 ? | 10:39 |
| chandankumar | jm1: we need to fix the prefix | 10:40 |
| soniya|ruck | slaweq, neutron tempest tests are failing on centos-8-standalone-full-tempest-api-compute-victoria with deadlock kind of issue | 10:40 |
| slaweq | soniya|ruck: sorry but I don't think I will have time today to check it | 10:51 |
| slaweq | but I will try to find someone else who can check it | 10:51 |
| soniya|ruck | slaweq, sure | 10:56 |
| slaweq | soniya|ruck: I pinged mlavalle on our internal channel about it | 10:58 |
| slaweq | he should be available in some time as he's based in US | 10:58 |
| slaweq | so I hope he will check it | 10:58 |
| soniya|ruck | arxcruz, chandankumar, kopecmartin, ysandeep|rover, do you have anything to discuss for tempest meeting? | 11:01 |
| arxcruz | soniya|ruck not really | 11:02 |
| *** bhagyashris_ is now known as bhagyashris | 11:03 | |
| chandankumar | soniya|ruck: nope | 11:09 |
| soniya|ruck | arxcruz, chandankumar, kopecmartin , ysandeep|rover, shall we meet next wednesday if that is okay with everyone? | 11:18 |
| arxcruz | soniya|ruck sure | 11:18 |
| ysandeep|rover | soniya|ruck, nope, I would vote to make that meeting biweekly | 11:18 |
| kopecmartin | soniya|ruck: sure | 11:18 |
| ysandeep|rover | soniya|ruck, nope for >> do you have anything to discuss for tempest meeting? | 11:18 |
| ysandeep|rover | soniya|ruck, ack for next wednesday | 11:19 |
| rlandy|out | ysandeep|rover: are you ok if we merge https://review.opendev.org/c/openstack/tripleo-ci/+/839323? | 11:19 |
| rlandy|out | or do you want to get a master promo through first? | 11:19 |
| soniya|ruck | ysandeep|rover, let's discuss that in next meeting -> I would vote to make that meeting biweekly | 11:20 |
| ysandeep|rover | rlandy|out, if ananya have green runs in testprojects - I would not block that. | 11:21 |
| *** rlandy|out is now known as rlandy | 11:21 | |
| rlandy | ysandeep|rover: frenzy_friday: ok - I'll w+ that at my EoD so you can get through the promoted up tripleo component | 11:23 |
| rlandy | chandankumar: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42669 | 11:28 |
| ysandeep|rover | rlandy, soniya|ruck https://bugs.launchpad.net/tripleo/+bug/1972986 - clients component | 11:28 |
| ysandeep|rover | rlandy, looks like recent ansible-collections-openstack-1.7.2-0.20220510202938.80ef209.el9 broke us | 11:28 |
| ysandeep|rover | ansible-collections-openstack-1.7.2-0.20220510202938.80ef209.el9 - dfg-jm1? | 11:28 |
| ysandeep|rover | jm1: https://bugs.launchpad.net/tripleo/+bug/1972986 when you have time could you please check that. | 11:29 |
| *** dviroel|out is now known as dviroel | 11:31 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 11:32 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 11:37 | |
| ysandeep|rover | rlandy, rcastillo jm1 looks like we broke by https://github.com/openstack/ansible-collections-openstack/commit/b31fdf8320af1a39ecee9277f90213a26b81a5b1 | 11:38 |
| ysandeep|rover | jm1: we definetely need tripleo jobs in ansible-collection-openstack check/gate soon :D | 11:39 |
| chandankumar | marios: https://review.opendev.org/c/openstack/tripleo-ci/+/839518/10#message-859101237f55ba7b7219a5ef2621c03e86535405 one comment | 11:39 |
| rlandy | frenzy_friday; what order should these merge in: https://review.opendev.org/c/openstack/tripleo-ci/+/839323 and https://review.opendev.org/c/openstack/tripleo-quickstart/+/839305 | 11:40 |
| frenzy_friday | rlandy, probably https://review.opendev.org/c/openstack/tripleo-ci/+/839323 and then https://review.opendev.org/c/openstack/tripleo-quickstart/+/839305 Not sure if the merge order will break something | 11:43 |
| chandankumar | frenzy_friday: rlandy let's +w both of them | 11:45 |
| chandankumar | for quay registry | 11:45 |
| rlandy | chandankumar: will do both at my eod | 11:45 |
| chandankumar | ok | 11:45 |
| jm1 | ysandeep|rover: yeah, we made it! | 11:46 |
| *** dasm|off is now known as dasm | 11:46 | |
| jm1 | ysandeep|rover: will have a look | 11:46 |
| dasm | o/ | 11:46 |
| rlandy | ysandeep|rover: which failure was from the collections change? | 11:46 |
| ysandeep|rover | rlandy, failure in clients component | 11:47 |
| rlandy | ah | 11:47 |
| ysandeep|rover | all releases are affected for clients component | 11:47 |
| ysandeep|rover | jm1, thanks dfg-jm1++ | 11:48 |
| jm1 | ysandeep|rover: do you pull the master branch of ansible openstack collection? | 11:49 |
| jm1 | ysandeep|rover: do you use master branch of openstacksdk? | 11:50 |
| marios | chandankumar: thanks i think we can tackle on followup | 11:51 |
| jm1 | ysandeep|rover: regarding tripleo in check gate jobs: our periodic jobs did not run for a while but our tripleo check jobs in ansible openstack collection do run. they fail as expected since we are working on compatibility to new sdk | 11:52 |
| dasm | frenzy_friday: i respawn my patch https://review.opendev.org/c/opendev/elastic-recheck/+/841172 with your changes and few additional. | 11:52 |
| frenzy_friday | dasm, cool, thanks! | 11:53 |
| ysandeep|rover | jm1: I am not sure, probably check rpm spec file or ping someone in rdo. | 11:53 |
| dasm | frenzy_friday: btw. Elastic Recheck is moving nicely forward. I hope to see it running in next few days.' | 11:54 |
| dasm | s/moving nicely/nicely moving | 11:54 |
| *** ysandeep|rover is now known as ysandeep|rover|mtg | 11:55 | |
| ysandeep|rover|mtg | soniya|ruck, please ping when you have patch to ignore fs001 and fs035 for c8 wallaby | 11:56 |
| soniya|ruck | ysandeep|rover|mtg, yeah | 11:56 |
| dviroel | rlandy: https://review.rdoproject.org/r/c/config/+/42585 - depends-on merged - you can now remove w-1 | 12:01 |
| rlandy | yep - w+'ed | 12:02 |
| jm1 | ysandeep|rover|mtg: asked in #rdo, dunno why rdo is using master branch. this must fail. | 12:09 |
| rlandy | ysandeep|rover|mtg: soniya|ruck: periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 failed - will rerun to promo | 12:11 |
| ysandeep|rover|mtg | jm1: thanks! | 12:12 |
| ysandeep|rover|mtg | rlandy, bm is not in criteria. 17/9 already promoted | 12:12 |
| rlandy | ysandeep|rover|mtg: thank s- was just in progress on checking criteria :) | 12:12 |
| ysandeep|rover|mtg | rlandy, there was a QOW - What percentage of your 16.2 tests are capable of running on the OSP 17.0 compose today? but that doesn't apply to us as we don't run our test on compose. | 12:18 |
| rlandy | ysandeep|rover|mtg: exactly - we run parallel tests on both releases | 12:21 |
| rlandy | as it is | 12:21 |
| jm1 | ysandeep|rover|mtg: submitted a patch and updated lp bug. https://bugs.launchpad.net/tripleo/+bug/1972986 | 12:24 |
| soniya|ruck | ysandeep|rover|mtg, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42735 | 12:25 |
| ysandeep|rover|mtg | jm1, t/u/v is also affected | 12:25 |
| soniya|ruck | rlandy, ^^ | 12:25 |
| rlandy | soniya|ruck: thanks - will revert | 12:26 |
| rlandy | chandankumar: marios: want me to w+ https://review.opendev.org/c/openstack/tripleo-ci/+/839518? | 12:28 |
| rlandy | there is one follow up needed | 12:28 |
| chandankumar | rlandy: yes good to go | 12:28 |
| rlandy | marios: can you remove -1 from https://review.rdoproject.org/r/c/config/+/42442 | 12:30 |
| rlandy | w+'ed the previous patches | 12:30 |
| marios | rlandy: looking | 12:31 |
| marios | rlandy: voted and yes thanks on tripleo-ci/+/839518 | 12:32 |
| ysandeep|rover|mtg | jm1: do you want to also pin w/v/t branches. | 12:32 |
| jm1 | ysandeep|rover|mtg: i am trying to understand why w/v/t are affected althougth ansible-collections-openstack is pined in rdoinfo/buildsys-tags | 12:33 |
| jm1 | ysandeep|rover|mtg: i would ask amorelej but he is not here today. no idea whom to ask | 12:34 |
| ysandeep|rover|mtg | jm1: may be chandankumar | 12:35 |
| jm1 | chandankumar: ^ | 12:35 |
| jm1 | ;) | 12:35 |
| chandankumar | jm1: I am reading the bug | 12:35 |
| *** ysandeep|rover|mtg is now known as ysandeep|rover | 12:37 | |
| ysandeep|rover | soniya|ruck, thanks! | 12:37 |
| jm1 | chandankumar: the package version says 1.7.2 but inside is the latest code from our master branch. that cannot work https://trunk.rdoproject.org/centos8-train/component/clients/current/ansible-collections-openstack-1.7.2-0.20220511083902.a6805cd.el8.noarch.rpm | 12:40 |
| chandankumar | jm1: asnible collection is not pinned for train https://github.com/redhat-openstack/rdoinfo/blob/master/tags/train.yml#L4 so it is getting built from master | 12:46 |
| chandankumar | check second row https://trunk.rdoproject.org/centos8-train/report.html | 12:46 |
| jm1 | chandankumar: what do we use build-tags for, then? | 12:47 |
| chandankumar | buildsys tags are used to build a package in cbs | 12:49 |
| jm1 | chandankumar: i read the readme but still no idea what the connection between buildtags and tags is | 12:53 |
| jm1 | chandankumar: so if we pin ansible-collections-openstack in buildtags, it seems to be ignored by tags. so what is the purpose of both? | 12:54 |
| chandankumar | tags/release.yml file is used for building the packages using dlrn and control it | 12:54 |
| chandankumar | if it is not pinned, then it will build from respective release branch | 12:54 |
| chandankumar | buildsys-tags are used to build centos dependendencies on CBS | 12:55 |
| chandankumar | jm1: let's chat on #rdo | 12:57 |
| jm1 | chandankumar: if buildsys-tags is for cbs, where will ansible-collections-openstack packages pop up? in which repo? | 13:04 |
| rlandy | bhagyashris_: can you cancel the scrum call tomorrow? | 13:11 |
| ysandeep|rover | rlandy, soniya|ruck fyi.. I spoke with rodolfo , created a wallaby cherry-pick https://review.opendev.org/c/openstack/ovsdbapp/+/841429 for testing | 13:15 |
| soniya|ruck | ysandeep|rover, ack | 13:15 |
| rlandy | ysandeep|rover: cool going to run w8 on that fs001 and fs035 | 13:16 |
| rlandy | ysandeep|rover: would we need to rebuild containers? | 13:17 |
| ysandeep|rover | rlandy: yes | 13:17 |
| rlandy | ysandeep|rover: hmmm ... idk if periodic would update | 13:18 |
| rlandy | with a depends-on | 13:18 |
| bhagyashris_ | rlandy, sure | 13:18 |
| rlandy | ysandeep|rover: for c8 it's probably fin - the line is short if we rebuilt | 13:19 |
| ysandeep|rover | we can do that in a testproject with dependency chain 1) container build 2) ovb depending on container build | 13:19 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 13:21 | |
| rlandy | ysandeep|rover|brb: ack but there are jobs running in the line now | 13:21 |
| rlandy | will wait for them to complete pulling containers | 13:22 |
| rlandy | then start | 13:22 |
| dviroel | ok - compose-id promotion is working after refactoring merge: https://review.rdoproject.org/zuul/build/6a8ff5d2b74b4148b7ed1b3f352f2fc8 and https://images.rdoproject.org/centos_compose/centos8/master/ | 13:22 |
| dviroel | FYI marios rlandy chandankumar ^ | 13:22 |
| marios | dviroel: thanks | 13:23 |
| marios | folks please see email "retrospective today - please read and respond asap" thanks | 13:24 |
| rlandy | dviroel: great | 13:24 |
| bhagyashris_ | rcastillo, dviroel rlandy soniya|ruck ysandeep|rover|brb | 13:30 |
| bhagyashris_ | retro time | 13:30 |
| marios | arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, dviroel, rcastillo, jm1, rlandy|ruck, dasm retro | 13:30 |
| bhagyashris_ | https://meet.google.com/nft-kdnz-ivn?authuser=0 | 13:30 |
| soniya|ruck | bhagyashris_, i am in | 13:30 |
| bhagyashris_ | https://miro.com/app/board/uXjVO1hv5Uk=/ | 13:31 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 13:31 | |
| *** bhagyashris_ is now known as bhagyashris | 13:33 | |
| chandankumar | jm1: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L2673 it comes as a part of client component | 13:33 |
| chandankumar | dlrn builds it as another openstack package | 13:33 |
| jpodivin | soniya|ruck: I'm seeing subsequent failures on periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation, during the ovb-manage task. Is it being tracked? | 13:38 |
| jpodivin | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation | 13:38 |
| jpodivin | Similar pattern of failures in other components using the same abstract job https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master | 13:40 |
| soniya|ruck | jpodivin, yeah, we are dealing with node_provisioning issues and baremetal issues | 13:40 |
| jpodivin | soniya|ruck: ack. | 13:59 |
| jpodivin | soniya|ruck: Is there something I can do to help? | 13:59 |
| rlandy | jpodivin: pls come to the CIX call | 14:10 |
| rlandy | you'll get a full run down of all our issues | 14:10 |
| chandankumar | thank you marios nicely done :-) | 14:30 |
| dviroel | marios++ | 14:30 |
| marios | thanks folks ... fyi i locked the board so you cannot edit any more | 14:31 |
| marios | let me know if you want to add something i can unlock it | 14:31 |
| rlandy | ysandeep|rover: https://review.rdoproject.org/r/c/testproject/+/36140 w c8 with depends-on fix | 14:35 |
| rlandy | marios++ for taking retro on at short notice | 14:35 |
| marios | np rlandy | 14:36 |
| ysandeep|rover | rlandy, marked fixed released for https://bugs.launchpad.net/tripleo/+bug/1959328 | 14:36 |
| ysandeep|rover | rlandy, is it okay to close the card too https://trello.com/c/2CAk8C4p/2330-cixlp1959328tripleociproa-tempestscenariotestnetworkbasicopstestnetworkbasicopstestnetworkbasicops-failing-on-periodic-tripleo-c or do you guys want to rekick the tempest discussion on this. | 14:37 |
| rlandy | ysandeep|rover: pls close it | 14:37 |
| ysandeep|rover | ack | 14:37 |
| rlandy | discussion is going on on ML | 14:38 |
| ysandeep|rover | done | 14:38 |
| rlandy | soniya|ruck: pls raise that with kopecmartin | 14:57 |
| soniya|ruck | rlandy, sure | 14:57 |
| rlandy | soniya|ruck: thank you - let's try clear that before monday's call | 14:58 |
| soniya|ruck | rlandy, yupp | 14:58 |
| soniya|ruck | rlandy, will start with that issue tommorrow morning | 15:01 |
| soniya|ruck | rlandy: ysandeep|rover: anything else you want me to go in morning? | 15:03 |
| ysandeep|rover | soniya|ruck: nope, have a good rest and a great evening! | 15:04 |
| rlandy | soniya|ruck: thanks | 15:04 |
| soniya|ruck | ysandeep|rover, need to make up with ruck/rover hangover :) | 15:04 |
| * ysandeep|rover reporting new provision issue - failing on boot now - kernel panic | 15:04 | |
| ysandeep|rover | soniya|ruck: :D | 15:05 |
| *** soniya|ruck is now known as soniya|out | 15:05 | |
| ysandeep|rover | rlandy, fyi.. reporting new bug for boot issue now: https://logserver.rdoproject.org/37/28537/41/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/21b13d8/logs/baremetal_41_49511_0-console.log | 15:05 |
| rlandy | ysandeep|rover: always more fun :( | 15:06 |
| rlandy | ysandeep|rover: anything you need me to track in my afternoon? | 15:06 |
| ysandeep|rover | rlandy, no thanks.. hjensas is aware of that issue.. I will ping him before leaving | 15:07 |
| ysandeep|rover | rlandy, just the c8 wallaby - if rodolfo patch works | 15:07 |
| rlandy | ysandeep|rover: ack - will do | 15:07 |
| ysandeep|rover | ++ thanks! | 15:08 |
| *** dviroel is now known as dviroel|lunch | 15:09 | |
| ysandeep|rover | rlandy, wallaby c8 promoted.. lets merge the criteria revert: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42537 | 15:26 |
| *** ysandeep|rover is now known as ysandeep|rover|out | 15:26 | |
| * ysandeep|rover|out shutdown sequence... | 15:26 | |
| rlandy | ysandeep|rover|out: w+'ed thanks | 15:27 |
| ysandeep|rover|out | rlandy: Have a good rest of your day, see you tomorrow o/ | 15:28 |
| rlandy | ysandeep|rover|out: thanks - see you tomorrow | 15:28 |
| rlandy | periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby shocker - passed | 15:28 |
| ysandeep|rover|out | rlandy, with rodolfo patch or without? | 15:28 |
| rlandy | without | 15:29 |
| rlandy | fs035 still failed | 15:29 |
| ysandeep|rover|out | ack, that neutron bug is a transient thing - possible | 15:29 |
| ysandeep|rover|out | but frequency of occurence is high | 15:29 |
| ysandeep|rover|out | good that it passed we know now the recent promotion is legit :D | 15:31 |
| * ysandeep|rover|out out | 15:31 | |
| *** marios is now known as marios|out | 15:44 | |
| rlandy | frenzy_friday: need 5 - just logging a bug | 16:00 |
| frenzy_friday | rlandy, sure, np | 16:00 |
| *** dviroel|lunch is now known as dviroel | 16:03 | |
| *** jpena is now known as jpena|off | 17:01 | |
| * dviroel getting coffee | 17:19 | |
| dviroel | rlandy: 1-1 | 17:32 |
| rlandy | joining | 17:34 |
| dasm | dviroel: can you check if you're seeing new folder with entry in bitwarden? | 19:46 |
| dviroel | rlandy: we can add this to 'repo_cmd_after' section | 20:01 |
| dviroel | sudo yum-config-manager --save --setopt=delorean-master-testing.exclude=python3-smmap-5.* | 20:01 |
| rlandy | dviroel: something like that - will edit | 20:02 |
| rlandy | we can try that | 20:02 |
| rlandy | editing | 20:02 |
| dviroel | ack | 20:03 |
| dasm | weshay: are you around? | 20:05 |
| dasm | weshay: i need a confirmation for login | 20:05 |
| rlandy | dasm: what do you need? | 20:05 |
| rlandy | wes is not active on this channel anymore | 20:06 |
| dviroel | dasm: i see: No Folders | 20:06 |
| dasm | dviroel: can you try refreshing it? there should be a button there | 20:06 |
| dasm | rlandy: trying to login to tripleo gmail account which is created by Wes | 20:06 |
| dasm | and associated with his phone # and email | 20:06 |
| dasm | rlandy: i need to update ssh keys to rdoproject for elastic recheck | 20:07 |
| rlandy | dasm: sending you in in pvt | 20:07 |
| dviroel | dasm: nothing here | 20:08 |
| dasm | O/ | 20:08 |
| dasm | :/ | 20:08 |
| dasm | dviroel: thanks for checking. it worries me tho, because i added ssh keys | 20:09 |
| dasm | dviroel: try now | 20:09 |
| dasm | i forced syncing on my side | 20:09 |
| dviroel | dasm: what should I see? | 20:11 |
| dasm | > os-tripleo-ci Gerrit SSH access | 20:12 |
| dviroel | i can see that 'item' | 20:13 |
| dasm | \( ﾟヮﾟ)/ | 20:13 |
| dviroel | dasm: i was looking under Folders. Your item is still under Folder: No Folder for me | 20:19 |
| dasm | booo ;( | 20:19 |
| dasm | i thought i made it working with going extra hoops with folders. | 20:20 |
| dasm | either way -- it's stored there. that's good | 20:20 |
| dviroel | hum, maybe folders are personal only, so anyone can organize on its own way | 20:21 |
| dviroel | if you click on the Organization, there is no Folders there | 20:22 |
| dviroel | only items and collections | 20:22 |
| dasm | ¯\_(ツ)_/¯ | 20:24 |
| * rcastillo lunch brb | 20:32 | |
| rlandy | ysandeep|rover|out: https://review.rdoproject.org/r/c/testproject/+/36140 - no success with networking patch on w c8 | 20:38 |
| rlandy | dviroel: this is current change is woorking | 20:38 |
| dviroel | rlandy: yes :) | 20:40 |
| * dviroel going afk | 20:53 | |
| *** dviroel is now known as dviroel|afk | 20:53 | |
| * dasm leaves too | 21:02 | |
| dasm | o/ | 21:02 |
| *** dasm is now known as dasm|off | 21:02 | |
| rlandy | oh gee ... now everything is failing on [undercloud]: FAILED! => {"changed": false, "msg": "Failed to pull following containers: ['quay.ceph.io/prometheus/prometheus:v2.7.2', 'quay.ceph.io/prometheus/alertmanager:v0.16.2', 'quay.ceph.io/prometheus/node-exporter:v0.17.0', 'quay.ceph.io/app-sre/grafana:6.7.4']"} | 22:01 |
| *** rlandy is now known as rlandy|bbl | 22:08 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!