| rdogerrit | rdo-trunk proposed rdoinfo master: Bump rdoinfo xena-uc tags to latest upper-constraints https://review.rdoproject.org/r/c/rdoinfo/+/34603 | 00:10 |
|---|---|---|
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for wallaby-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/34613 | 00:14 |
| rdogerrit | Steve Baker proposed rdo-jobs master: Upload overcloud-hardened-uefi-full in main integration https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 03:25 |
| rdogerrit | Steve Baker created rdo-jobs master: Upload both overcloud-full and overcloud-hardened-uefi-full https://review.rdoproject.org/r/c/rdo-jobs/+/34614 | 03:25 |
| rdogerrit | Steve Baker proposed rdo-jobs master: Upload both overcloud-full and overcloud-hardened-uefi-full https://review.rdoproject.org/r/c/rdo-jobs/+/34614 | 03:37 |
| rdogerrit | Steve Baker proposed rdo-jobs master: Build overcloud-hardened-uefi-full in main integration https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 03:37 |
| rdogerrit | Yatin Karel proposed rdoinfo master: Promote CBS tags update for wallaby-8s-testing https://review.rdoproject.org/r/c/rdoinfo/+/34613 | 05:40 |
| opendevreview | yatin proposed x/packstack master: Include placement::api class https://review.opendev.org/c/x/packstack/+/801596 | 06:30 |
| *** amoralej|off is now known as amoralej | 06:51 | |
| -opendevstatus- NOTICE: Due to a configuration error unfortunately the Zuul queue was lost. Please recheck any in-flight changes | 06:57 | |
| ykarel | amoralej, jcapitao i have installed Nested View Plugin in new jenkins | 07:05 |
| ykarel | to have view like https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/puppet-promotion-pipelines/ | 07:06 |
| amoralej | ykarel, lemme check | 07:06 |
| ykarel | instead of all the views in top layer | 07:06 |
| amoralej | ok, lgtm | 07:06 |
| amoralej | ykarel, btw i was discussing yesterday with jcapitao wrt gerrit plugin | 07:06 |
| ykarel | similary can add for phase 1 , and all releases as nested | 07:06 |
| amoralej | we need to put a ssh key in the jenkins container filesystem, we'll add it as a openshift secret | 07:07 |
| ykarel | amoralej, okk u mean the issue with tripleo jobs not reporting to gerrit? | 07:07 |
| amoralej | i'm not sure if that's related | 07:07 |
| amoralej | but we need to add it to configure it | 07:07 |
| jcapitao | good | 07:07 |
| amoralej | we'll check today with centos-ci guys if there is any other way | 07:07 |
| ykarel | okk sounds good | 07:07 |
| ykarel | for the gerrit issue in ci.centos i tried reach out to centos ci guys months back, but never followed back | 07:08 |
| ykarel | so can check again on that | 07:09 |
| amoralej | anyway, i think it'd be good to check with oooq team if they really want/need those third party | 07:09 |
| amoralej | as it has not been reported for months iiuc | 07:09 |
| ykarel | i think i saw some discussion b/w jcapitao and rlandy for those jobs | 07:10 |
| jcapitao | ykarel: ah, so it's not working since a while ? | 07:10 |
| ykarel | jcapitao, yes not reporting from long | 07:10 |
| jcapitao | IIUC it's libvirt based job | 07:10 |
| ykarel | atleast 6 monthis i think | 07:10 |
| rdogerrit | Merged puppet/puppet-vswitch-distgit rpm-master: Add puppet requirements https://review.rdoproject.org/r/c/puppet/puppet-vswitch-distgit/+/34609 | 07:10 |
| ykarel | jcapitao, yes it's libvirt based jobs, it creates a vm on host it's running and deploy tripleo standalone | 07:11 |
| rdogerrit | Merged openstack/tripleoclient-distgit victoria-rdo: Fix heat epoch https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/34065 | 07:11 |
| amoralej | i guess it'd be good to have it as it's phase2 promotion criteria | 07:11 |
| jcapitao | ykarel: ok, I think the plugin is broken, and I want to test it on our Jenkins as we can access logs | 07:11 |
| amoralej | but it's probably good to confirm | 07:11 |
| ykarel | jcapitao, ack | 07:12 |
| ykarel | yes good to confirm | 07:13 |
| jcapitao | amoralej: I asked rlandy last monday on #tripleo and he replied it's needed | 07:21 |
| ykarel | amoralej, also fyi c9 dlrn was stuck since Friday, i cleared that and pending builds are in queue https://trunk.rdoproject.org/centos9-master-preview/queue.html | 07:23 |
| *** rpittau|afk is now known as rpittau | 07:27 | |
| amoralej | jcapitao, ok let's go with it then | 07:39 |
| amoralej | ykarel, i think there should be alarm in https://prometheus.monitoring.softwarefactory-project.io/prometheus/alerts for it | 07:41 |
| jcapitao | amoralej: I asked on #centos-ci for Gerrit topic | 07:49 |
| amoralej | i'm not sure what's the threshold for RDOTrunk | 07:49 |
| amoralej | jcapitao, ack | 07:49 |
| amoralej | i just joined | 07:49 |
| jcapitao | amoralej: https://paste.centos.org/view/9ffa0428 | 07:54 |
| amoralej | ack, thanks | 07:55 |
| ykarel | amoralej, yes there should be alarm | 07:59 |
| ykarel | amoralej, seems c9 don't have it configured https://github.com/softwarefactory-project/sf-infra/blob/master/monitoring/rules-dlrn.yaml | 08:00 |
| ykarel | c8 is 1 day and c7 3 days | 08:00 |
| amoralej | ykarel, i though https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/22280 should have added it | 08:02 |
| ykarel | amoralej, seems it was missed for c9 | 08:02 |
| amoralej | yes, looks like ti | 08:03 |
| amoralej | it | 08:03 |
| ykarel | or was waiting for to be it non preview | 08:03 |
| amoralej | maybe | 08:05 |
| amoralej | as per conversation with jpena before he left, i understood he was preparing to add it | 08:06 |
| amoralej | anyway, let's check the worker from time to time | 08:06 |
| ykarel | okk | 08:07 |
| ykarel | amoralej, jcapitao u already aware about https://zuul.opendev.org/t/openstack/builds?job_name=packstack-centos8s-multinode-scenario002-tempest&project=x/packstack failures ? | 08:10 |
| ykarel | noticed while cleare puppet promotion blocker https://review.opendev.org/c/x/packstack/+/801596 | 08:10 |
| ykarel | looks like it's failing from long, in c9 prep review | 08:11 |
| amoralej | i saw it in the change for c9, right | 08:11 |
| amoralej | but didn't dig too much | 08:11 |
| amoralej | let me check | 08:12 |
| ykarel | ohkk | 08:13 |
| amoralej | packstack.installer.exceptions.ScriptRuntimeError: Failed to run remote script, stdout: skipping creation of hiera.yaml symlink | 08:15 |
| amoralej | mmm | 08:15 |
| amoralej | that's connection issue? | 08:15 |
| ykarel | amoralej, ahh seems due to hiera coming from epel | 08:15 |
| amoralej | mmm | 08:15 |
| ykarel | and in epel build i don't see that file | 08:15 |
| amoralej | it's unclear to me | 08:15 |
| ykarel | rpm -qpl https://kojipkgs.fedoraproject.org//packages/hiera/3.6.0/2.el8/noarch/hiera-3.6.0-2.el8.noarch.rpm | 08:15 |
| ykarel | vs https://cbs.centos.org/kojifiles/packages/hiera/3.6.0/1.el8/noarch/hiera-3.6.0-1.el8.noarch.rpm | 08:15 |
| ykarel | /etc/puppet/hiera.yaml is part of our build | 08:16 |
| amoralej | and, why it deploy from epel? | 08:16 |
| ykarel | higher version then us, and epel might be enabled | 08:17 |
| amoralej | ykarel, https://5388d16ce1623680ba18-cb5427740da0927fbab3d70711de4f05.ssl.cf5.rackcdn.com/801596/1/check/packstack-centos8s-integration-scenario003/93298a0/logs/etc/yum.repos.d/epel.repo.txt | 08:20 |
| amoralej | that's single node | 08:20 |
| amoralej | so it seems we have epel enabled too | 08:20 |
| amoralej | what hiera are we getting there? | 08:20 |
| ykarel | hiera-3.6.0-2 | 08:21 |
| amoralej | the one from epel, right? | 08:21 |
| amoralej | we should disable epel | 08:21 |
| amoralej | everywhere | 08:21 |
| ykarel | yes | 08:21 |
| ykarel | mmm i see we have rm -rf /etc/yum.repos.d/epel* | 08:22 |
| amoralej | where? | 08:23 |
| ykarel | in playbooks/packstack-centos8-pre.yaml | 08:23 |
| ykarel | ahh it's getting reinstalled | 08:25 |
| ykarel | i mean upgrade | 08:25 |
| ykarel | epel-release noarch 8-11.el8 extras | 08:25 |
| ykarel | so we would need to clear rpm itself | 08:25 |
| amoralej | where it's getting reinstalled? | 08:26 |
| ykarel | dnf update -y | 08:26 |
| ykarel | https://17dfaf0b65f837c850bf-9c94c5208d9eb7cfc1f62a5c8e77c9f1.ssl.cf2.rackcdn.com/801596/1/check/packstack-centos8s-multinode-scenario002-tempest/379e96b/job-output.txt | 08:26 |
| amoralej | ah, yes let's uninstall "epel*" | 08:26 |
| ykarel | yes including in my patch | 08:27 |
| amoralej | ack | 08:27 |
| rdogerrit | Sagi Shnaidman created rdo-infra/ci-config master: Fix error in zuul jobs build file https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34615 | 08:30 |
| sshnaidm | pojadhav, ^ | 08:30 |
| pojadhav | sshnaidm, ack | 08:30 |
| opendevreview | yatin proposed x/packstack master: Include placement::api class https://review.opendev.org/c/x/packstack/+/801596 | 08:31 |
| *** ykarel is now known as ykarel|lunch | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 09:46 | |
| ykarel | amoralej, jcapitao https://review.opendev.org/c/x/packstack/+/801596 | 09:57 |
| ykarel | multinode too green with epel removal | 09:57 |
| amoralej | good | 09:58 |
| jcapitao | great, I spent some time with my c9 patch but didn't find it | 10:00 |
| rdogerrit | Sagi Shnaidman created rdo-infra/ci-config master: Fix zuul builds script according to API changes https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34616 | 10:23 |
| sshnaidm | pojadhav, ^^ | 10:23 |
| rdogerrit | Merged rdo-infra/ci-config master: Fix error in zuul jobs build file https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34615 | 10:27 |
| *** jcapitao is now known as jcapitao_lunch | 10:38 | |
| rdogerrit | rdo-trunk created openstack/tripleo-repos-distgit rpm-master: tripleo-repos: failed to build 0ecc6e0 https://review.rdoproject.org/r/c/openstack/tripleo-repos-distgit/+/34617 | 11:02 |
| rdogerrit | rdo-trunk created openstack/tripleo-repos-distgit victoria-rdo: tripleo-repos: failed to build 0ecc6e0 https://review.rdoproject.org/r/c/openstack/tripleo-repos-distgit/+/34618 | 11:03 |
| rdogerrit | rdo-trunk created openstack/tripleo-repos-distgit wallaby-rdo: tripleo-repos: failed to build 0ecc6e0 https://review.rdoproject.org/r/c/openstack/tripleo-repos-distgit/+/34619 | 11:03 |
| rdogerrit | rdo-trunk created openstack/tripleo-repos-distgit ussuri-rdo: tripleo-repos: failed to build 0ecc6e0 https://review.rdoproject.org/r/c/openstack/tripleo-repos-distgit/+/34620 | 11:04 |
| ykarel | amoralej, needs +W too https://review.opendev.org/c/x/packstack/+/801596 | 11:31 |
| amoralej | merging it | 11:31 |
| ykarel | Thanks | 11:31 |
| ykarel | zbr, sshnaidm FTBFS post tripleo-repos https://review.opendev.org/c/openstack/tripleo-repos/+/801029 ^ | 11:37 |
| ykarel | can u check | 11:37 |
| ykarel | strange CI jobs there didn't catched it | 11:37 |
| rdogerrit | Merged rdo-infra/ci-config master: Add undercloud upgrade and minor update job to criteria https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34598 | 11:38 |
| ykarel | ahhk no deploy job ran | 11:38 |
| sshnaidm | No module named 'ansible' ? | 11:44 |
| ykarel | yes | 11:49 |
| *** jcapitao_lunch is now known as jcapitao | 12:05 | |
| rdogerrit | Merged rdoinfo master: Bump rdoinfo xena-uc tags to latest upper-constraints https://review.rdoproject.org/r/c/rdoinfo/+/34603 | 12:13 |
| *** rlandy is now known as rlandy|ruck | 12:17 | |
| rdogerrit | Sagi Shnaidman proposed rdo-infra/ci-config master: Fix zuul builds script according to API changes https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34616 | 12:23 |
| *** amoralej is now known as amoralej|lunch | 12:41 | |
| opendevreview | Merged x/packstack master: Include placement::api class https://review.opendev.org/c/x/packstack/+/801596 | 12:43 |
| ykarel | mwhahaha, some days back you reverted tripleo-repos patch that added cliff as requirement, with https://review.opendev.org/c/openstack/tripleo-repos/+/801029 ansible is added which created above FTBFS | 12:56 |
| ykarel | can u check what to do in this case | 12:57 |
| mwhahaha | ykarel: yea i saw the ftb was going to look at it | 12:57 |
| mwhahaha | i thought it just added yaml | 12:57 |
| mwhahaha | if it adds ansible that's a non-starter again | 12:57 |
| ykarel | mwhahaha, ansible imports in https://review.opendev.org/c/openstack/tripleo-repos/+/801029/8/tripleo_repos/get_hash/tripleo_get_hash.py#18 | 12:58 |
| mwhahaha | thouse shold only get invoked when running the ansible modules | 12:59 |
| mwhahaha | anyway i'm going to look at it today | 12:59 |
| sshnaidm | zbr, ^^ | 12:59 |
| ykarel | ack | 12:59 |
| mwhahaha | yup 2021-07-21 11:09:54.363578 | rdo-centos-stream | 2021-07-21 11:09:54,363 INFO:dlrn-build-mock:DEBUG: from ansible.module_utils.basic import AnsibleModule | 13:00 |
| mwhahaha | cause the unit tests | 13:00 |
| * mwhahaha reverts again | 13:00 | |
| sshnaidm | probably we can add "try: except: " for import ansible ? | 13:00 |
| mwhahaha | or we could stop adding all this randoms tuff into tripleo-repos | 13:00 |
| sshnaidm | yeah, maybe modules should move to oooq repos | 13:01 |
| mwhahaha | they were there originally | 13:01 |
| sshnaidm | although nice to have all together | 13:01 |
| mwhahaha | the ansible bits need to not be in tripleo-repos | 13:01 |
| mwhahaha | or could be a sub package | 13:01 |
| sshnaidm | like plugin | 13:02 |
| mwhahaha | https://review.opendev.org/c/openstack/tripleo-repos/+/801583 revert | 13:02 |
| sshnaidm | dviroel, zbr probably we'll need to move ansible modules to oooq repos from tripleo-repos ^ | 13:02 |
| mwhahaha | the modules can exist and even be a sub package | 13:02 |
| mwhahaha | they just can't be in the tripleo_repos python bits | 13:02 |
| sshnaidm | the idea was installing them together with tripleo-repos iirc | 13:03 |
| mwhahaha | a subpackage could do that | 13:03 |
| mwhahaha | ansible-tripleo-repos would pull in all the bits | 13:03 |
| mwhahaha | tripleo-repos really should just be a cli tool. anything beyond that (e.g. ansible modules) should be a subpackage | 13:04 |
| mwhahaha | the ansible bits would go on the ansible executor and not the source system realisitically | 13:04 |
| zbr | mwhahaha: and that change make those two subpackages. what is the problem? is that the data_files section? | 13:04 |
| mwhahaha | https://review.opendev.org/c/openstack/tripleo-repos/+/801029/8/tripleo_repos/get_hash/tripleo_get_hash.py#18 | 13:05 |
| mwhahaha | your unit tests invoke that | 13:05 |
| mwhahaha | the ansible module should not be in tripleo_repos/* | 13:05 |
| mwhahaha | it should be elsewhere and packaged differently | 13:05 |
| ykarel | RDO meeting in next 1 hour, add topics to agenda https://etherpad.opendev.org/p/RDO-Meeting | 13:06 |
| zbr | mwhahaha: yeah, in fact i was about to move them inside plugins/modules -- i totally agree they should not be inside the python package. | 13:06 |
| mwhahaha | zbr: if you move then we can skip the revert | 13:06 |
| zbr | yes, i can obviously move. i was already planning to do that. | 13:07 |
| mwhahaha | IMHO the ansible modules should be elsewhere (within the repo is fine) and packaged as a subpackage | 13:07 |
| mwhahaha | i was concerned about that patch but since it only included pyyaml in the requirements i thought it was ok. we really need a job that builds the rpm on that repo | 13:07 |
| mwhahaha | this is the 2nd time we've done this | 13:07 |
| zbr | clearly not a bad idea. | 13:08 |
| zbr | i always hated using data_files to install ansible modules and i was telling marios that i would like to avoid that. | 13:09 |
| mwhahaha | i think the issue is the consumption of tripleo-repos via pip | 13:10 |
| mwhahaha | but yea | 13:10 |
| mwhahaha | needs thought | 13:10 |
| zbr | either rely on ansible-galaxy native collection install method, or a workaround i was considering was to add a command like `triple-repo --install-ansible-modules` that does install the two modules. | 13:10 |
| *** pojadhav is now known as pojadhav|afk | 13:10 | |
| zbr | consuming tripleo-repo from pip is not really a problem, in fact I am glad it works. | 13:11 |
| mwhahaha | yea normal consumption, it's the from pip ansible side | 13:11 |
| mwhahaha | i think the expectation should be that tripleo-repos (the cli) should work from pip and a single rpm | 13:11 |
| mwhahaha | anything beyond that needs extra packaging or additional things | 13:12 |
| zbr | that bit is pure evil, and i would be glad to remove that bit | 13:12 |
| mwhahaha | :D | 13:12 |
| rdogerrit | Merged rdo-infra/ci-config master: Fix zuul builds script according to API changes https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34616 | 13:15 |
| *** amoralej|lunch is now known as amoralej | 13:33 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:33 | |
| spotz | Meeting in 25 minutes, add your agenda items - https://etherpad.opendev.org/p/RDO-Meeting | 13:34 |
| *** rlandy is now known as rlandy|ruck | 13:35 | |
| ykarel | Meeting time | 14:00 |
| ykarel | #startmeeting RDO meeting - 2021-07-21 | 14:00 |
| opendevmeet | Meeting started Wed Jul 21 14:00:48 2021 UTC and is due to finish in 60 minutes. The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'rdo_meeting___2021_07_21' | 14:00 |
| ykarel | Please add topic to agenda https://etherpad.opendev.org/p/RDO-Meeting | 14:01 |
| ykarel | #topic roll call | 14:01 |
| spotz | o/ | 14:02 |
| ykarel | #chair spotz | 14:02 |
| opendevmeet | Current chairs: spotz ykarel | 14:02 |
| amoralej | o/ | 14:02 |
| ykarel | #chair amoralej | 14:02 |
| opendevmeet | Current chairs: amoralej spotz ykarel | 14:02 |
| jcapitao | o/ | 14:02 |
| ykarel | #chair jcapitao | 14:02 |
| opendevmeet | Current chairs: amoralej jcapitao spotz ykarel | 14:02 |
| ykarel | Ok let's start with topics in agenda | 14:03 |
| ykarel | #topic C9 Stream Updates | 14:03 |
| ykarel | #link https://review.rdoproject.org/r/c/testproject/+/33878 | 14:05 |
| ykarel | #link https://review.rdoproject.org/r/c/testproject/+/34549 | 14:06 |
| ykarel | #info some tests were run with devstack + c9 | 14:06 |
| ykarel | amoralej, any other updates apart from ^? | 14:07 |
| amoralej | i think that's the main progress | 14:07 |
| amoralej | i tried to debug some issues in devstack runs | 14:07 |
| amoralej | but i think i need someone to check | 14:07 |
| ykarel | amoralej, related to tempest failures? | 14:07 |
| amoralej | yes | 14:07 |
| ykarel | i noticed some of those were random | 14:07 |
| amoralej | i'm not sure if it's just random tbh | 14:08 |
| ykarel | okk | 14:08 |
| amoralej | tosky, ^ we are running devstack on Centos9 and got some errors in cinder | 14:08 |
| spotz | Wasn;t there a post about tempest failures? Might have been last week | 14:08 |
| amoralej | may we get some help from some cinder expert? | 14:08 |
| ykarel | like in https://logserver.rdoproject.org/49/34549/20/check/devstack-platform-centos-9-stream/d2b0b81/testr_results.html there was just 1 failure | 14:08 |
| ykarel | and the run before and after ^ there were more failrues | 14:09 |
| amoralej | after adding swift we got more errors | 14:09 |
| amoralej | https://logserver.rdoproject.org/49/34549/20/check/devstack-platform-centos-9-stream/793602b/testr_results.html | 14:09 |
| amoralej | but i'm not sure if those are related to swift | 14:10 |
| amoralej | or maybe performance related | 14:10 |
| amoralej | but there seems to be some pattern | 14:10 |
| amoralej | and be mainly related to cinder | 14:10 |
| tosky | amoralej: is it really cinder or something that looks cinder but it's really, say, nova? No :) | 14:10 |
| amoralej | in fact it's nova complaining but seems to be cinder :) | 14:11 |
| amoralej | it's unclear to me, tbh | 14:13 |
| amoralej | Jul 19 07:59:29.720234 node-0001435902 devstack@c-api.service[107162]: CRITICAL cinder [None req-3be6ea4f-0dfb-4686-9ad3-3ad58f157fc6 tempest-ServerStableDeviceRescueTest-1149582697 tempest-ServerStableDeviceRescueTest-1149582697-project] Unhandled error: OSError: write error | 14:13 |
| amoralej | that looks suspictious | 14:13 |
| amoralej | https://logserver.rdoproject.org/49/34549/20/check/devstack-platform-centos-9-stream/793602b/controller/logs/screen-c-api.txt | 14:14 |
| amoralej | anyway, we can follow up later | 14:14 |
| ykarel | yeap ok let's move to next | 14:15 |
| ykarel | #topic Jenkins Migration Updates | 14:15 |
| ykarel | #link https://review.rdoproject.org/r/q/topic:jenkins-v2 | 14:15 |
| ykarel | jcapitao, anything to add in this | 14:16 |
| jcapitao | so the patch to migrate promotion jobs for tripleo master is ready | 14:16 |
| jcapitao | #link https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34419 | 14:16 |
| jcapitao | as well for per releases and distro jobs (based on master patch) | 14:17 |
| jcapitao | #link https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34560 | 14:17 |
| jcapitao | now we're working on tripleo-quickstart 3rd party job which needs some specific Jenkins configuration | 14:17 |
| ykarel | ok Good, Thanks jcapitao | 14:18 |
| ykarel | weshay|ruck, rlandy|ruck can u have a look ^ | 14:19 |
| rlandy|ruck | ykarel: ack - will look after meetings | 14:19 |
| rlandy|ruck | weshay|ruck is PTO | 14:19 |
| rlandy|ruck | jcapitao: thanks for setting this up | 14:19 |
| ykarel | rlandy|ruck, okk Thanks | 14:19 |
| ykarel | Ok let's move to next topic | 14:20 |
| ykarel | #topic RDO Trunk repos for centos-[queens|rocky|stein] from last meeting | 14:20 |
| ykarel | amoralej, you got some updates on ^? | 14:21 |
| amoralej | yes, sorry, i realized about i missed sending the mail | 14:21 |
| amoralej | i was writing it right before the meeting, i'll send it asap | 14:21 |
| ykarel | ok np | 14:21 |
| ykarel | can followup next week | 14:21 |
| ykarel | #topic chair for next week | 14:22 |
| ykarel | any volunteer? | 14:22 |
| jcapitao | I can take it | 14:22 |
| spotz | Thhanks jcapitao | 14:22 |
| ykarel | Thanks jcapitao | 14:22 |
| ykarel | #action jcapitao to chair next week | 14:23 |
| ykarel | #topic Open Floor | 14:23 |
| ykarel | Feel free to bring any topic now | 14:23 |
| tosky | amoralej: back to that cinder issue (so that's in the log) that error you reported (OSError) can be seen also in different tests (look for tempest-ServerStableDeviceRescueTest), so it may not be critical | 14:23 |
| tosky | there is a proxy error here: https://logserver.rdoproject.org/49/34549/20/check/devstack-platform-centos-9-stream/793602b/controller/logs/screen-n-cpu.txt | 14:23 |
| tosky | look for Jul 19 07:59:29 | 14:24 |
| amoralej | tosky, iiuc the actual error there is: | 14:25 |
| amoralej | ERROR nova.virt.libvirt.driver [None req-bc98f5f5-5bdf-4839-8e92-54e818ebbfca tempest-ServerStableDeviceRescueTest-1149582697 tempest-ServerStableDeviceRescueTest-1149582697-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ca12fb3e-0e7c-4460-ab03-c77a593ccc75 is timed out. | 14:25 |
| amoralej | that's how i was to check cinder | 14:25 |
| amoralej | but i'm not sure who is responsible of that dettaching | 14:26 |
| amoralej | is that nova issue? | 14:26 |
| tosky | it's on the boundary and that's the part I'm not sure about | 14:28 |
| tosky | I'm sure geguileo knows if it's cinder or not | 14:28 |
| amoralej | yeah, we are in the same page then :) | 14:29 |
| ykarel | ok i just rechecked, and will request node hold, so can check on live env | 14:29 |
| amoralej | good | 14:29 |
| geguileo | tosky: let me catch up on the conversation... | 14:29 |
| rdogerrit | Joel Capitao proposed config master: Update script which sends review based on u-c changes https://review.rdoproject.org/r/c/config/+/33518 | 14:31 |
| geguileo | tosky: amoralej that issue is in Nova's domain, because the timeout is coming from libvirt | 14:32 |
| geguileo | it's failing before they even call os-brick (telling the instance to stop using the block device) | 14:33 |
| amoralej | ok, i'll look for some nova friend now | 14:33 |
| geguileo | amoralej: no, I may be wrong... | 14:34 |
| geguileo | let me have another look... | 14:34 |
| geguileo | looking at request req-bc98f5f5-5bdf-4839-8e92-54e818ebbfca which is the one with the error pasted above | 14:35 |
| geguileo | that error was temporary, because on the next try it succeeded | 14:35 |
| geguileo | so, it failed calling cinder :-( | 14:36 |
| ykarel | yes seems so, both the test where it was seeing timeout, have actually passed. | 14:38 |
| ykarel | failing one is ERROR nova.compute.manager [None req-70164f48-e90d-494a-8ccd-fbcade5428d4 tempest-AttachVolumeMultiAttachTest-1454555876 tempest-AttachVolumeMultiAttachTest-1454555876-project] [instance: b725072c-0ddf-41ba-9ffd-ad61aa4e54fd] Failed to attach deaabc20-29d2-4ec4-a8a0-ba0fc827b0d7 at /dev/vdb: cinderclient.exceptions.ClientException: Proxy Error (HTTP 502) | 14:38 |
| geguileo | the scary part is that the actual cinder code returns 200 | 14:39 |
| geguileo | Jul 19 07:59:34.690149 node-0001435902 devstack@c-api.service[107162]: INFO cinder.api.openstack.wsgi [req-bc98f5f5-5bdf-4839-8e92-54e818ebbfca req-86510417-b70f-4423-83b6-cceb0e70caab tempest-ServerStableDeviceRescueTest-1149582697 tempest-ServerStableDeviceRescueTest-1149582697-project] | 14:39 |
| amoralej | it may be even uwsgi itself | 14:39 |
| geguileo | https://38.102.83.14/volume/v3/170ba9baa97a46baac4c99ef456e5d84/attachments/beac4203-1e49-472f-8a1e-2bb3d7d8fe3c returned with HTTP 200 | 14:39 |
| amoralej | geguileo, the log is written before the request result is passed to uwsgi? | 14:40 |
| geguileo | it would seem so, because it fails after cinder API code is "done" | 14:40 |
| amoralej | https://stackoverflow.com/questions/36156887/uwsgi-raises-oserror-write-error-during-large-request | 14:41 |
| geguileo | amoralej: no I believe it's wsgi who returns the log return value | 14:42 |
| amoralej | let's see if we can reproduce it and check | 14:43 |
| geguileo | amoralej: https://github.com/openstack/cinder/blob/81f2aaeea91bce6455f9b09cc8795855200e75e1/cinder/api/openstack/wsgi.py#L938-L954 | 14:44 |
| geguileo | that's the INFO logging and what cinder does afterwards... | 14:46 |
| geguileo | doesn't seem to do any OS calls | 14:47 |
| amoralej | uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] | 14:47 |
| amoralej | yes, looks like it's uwsgi itself | 14:48 |
| ykarel | Lock acquired by "attachment_update" | 14:50 |
| ykarel | doesn't seem this released ^ | 14:50 |
| ykarel | ^ seems related geguileo ^? | 14:50 |
| ykarel | following req-70164f48-e90d-494a-8ccd-fbcade5428d4 in cinder api log | 14:51 |
| ykarel | https://logserver.rdoproject.org/49/34549/20/check/devstack-platform-centos-9-stream/d2b0b81/controller/logs/screen-c-api.txt | 14:51 |
| ykarel | Ok let's have this offline, /me ends meeting | 14:52 |
| ykarel | Thanks all | 14:52 |
| ykarel | #endmeeting | 14:52 |
| opendevmeet | Meeting ended Wed Jul 21 14:52:53 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:52 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/rdo_meeting___2021_07_21/2021/rdo_meeting___2021_07_21.2021-07-21-14.00.html | 14:52 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/rdo_meeting___2021_07_21/2021/rdo_meeting___2021_07_21.2021-07-21-14.00.txt | 14:52 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/rdo_meeting___2021_07_21/2021/rdo_meeting___2021_07_21.2021-07-21-14.00.log.html | 14:52 |
| geguileo | ykarel: lock not likely to be related | 14:52 |
| jcapitao | thank you ykarel | 14:53 |
| ykarel | geguileo, ack, not directly i meant if that could fallback to proxy error | 14:54 |
| geguileo | not releasing it would just block another request... | 14:54 |
| geguileo | and that would show as a timeout | 14:54 |
| ykarel | okk | 14:55 |
| spotz | Thanks ykarel for leading and everyone for coming | 15:00 |
| *** ykarel is now known as ykarel|away | 15:07 | |
| rdogerrit | Yatin Karel created rdo-website master: RDO Meeting minutes for 2021-07-21 https://review.rdoproject.org/r/c/rdo-website/+/34622 | 15:12 |
| rdogerrit | Merged openstack/ironic-distgit rpm-master: Update minimum requirements https://review.rdoproject.org/r/c/openstack/ironic-distgit/+/34553 | 15:23 |
| spotz | amoralej jcapitao - I think passwords are being deprecated on GitHub, will that affect out logins for Software Factory? | 15:27 |
| amoralej | mmm i'm not sure | 15:28 |
| amoralej | spotz, where was it announced? | 15:28 |
| spotz | You recently used a password to access the repository at spotz/Calendar with git using git/2.25.2. | 15:30 |
| spotz | Basic authentication using a password to Git is deprecated and will soon no longer work. Visit https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information around suggested workarounds and removal dates. | 15:30 |
| spotz | Thanks, | 15:30 |
| spotz | The GitHub Team | 15:30 |
| spotz | 15:30 | |
| spotz | 15:30 | |
| spotz | 15:30 | |
| spotz | amoralej: I got an email yesterday when I logged in to GitHub do the Cloud SiG Calendar chhange - | 15:30 |
| spotz | ^ | 15:30 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for victoria-8-release https://review.rdoproject.org/r/c/rdoinfo/+/34606 | 15:30 |
| spotz | First II'd hheard of it as well but I mainly do stuff from opendev and rdoproject:) | 15:31 |
| amoralej | i hope that doesn't affect us | 15:32 |
| amoralej | tristanC, ^ | 15:32 |
| spotz | I kow one persoon had asked abouot us using users and passwords as they didn't want to have a shared login situation for a patch. I ended up doing it for them | 15:33 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for ussuri-8-release https://review.rdoproject.org/r/c/rdoinfo/+/34604 | 15:35 |
| tristanC | amoralej: i don't think we are using password with git | 15:46 |
| amoralej | yep, that's also my guess | 16:01 |
| *** amoralej is now known as amoralej|off | 16:22 | |
| rdogerrit | Merged rdo-website master: RDO Meeting minutes for 2021-07-21 https://review.rdoproject.org/r/c/rdo-website/+/34622 | 16:28 |
| *** marios is now known as marios|out | 16:31 | |
| *** rpittau is now known as rpittau|afk | 17:21 | |
| rlandy|ruck | jcapitao[m]: hi - question about ci.centos | 20:04 |
| rlandy|ruck | our login got updated? | 20:04 |
| rlandy|ruck | nvm - found it | 20:05 |
| *** jjoyce|away is now known as jjoyce | 20:12 | |
| rdogerrit | Steve Baker proposed rdo-jobs master: Upload both overcloud-full and overcloud-hardened-uefi-full https://review.rdoproject.org/r/c/rdo-jobs/+/34614 | 20:27 |
| rdogerrit | Steve Baker proposed rdo-jobs master: Build overcloud-hardened-uefi-full in main integration https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 20:27 |
| stevebaker | rlandy|ruck: Hey, I'm close to finishing whole-disk overcloud image support, its mostly CI changes now. The next task is to get overcloud-hardened-uefi-full.qcow2 published alongside overcloud-full.tar. I think this change will do that https://review.rdoproject.org/r/c/rdo-jobs/+/34614 | 20:30 |
| rlandy|ruck | stevebaker: I'm a little out of the loop here ... for master ( and v, u, t releases), we build and upload images in the integration line with a different job: | 20:34 |
| rlandy|ruck | periodic-tripleo-centos-8-buildimage-overcloud-full-master | 20:34 |
| rlandy|ruck | we used to use the fs002 job | 20:35 |
| rlandy|ruck | periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master | 20:35 |
| rlandy|ruck | periodic-tripleo-centos-8-buildimage-ironic-python-agent-master | 20:35 |
| rlandy|ruck | ^^ are you after those jobs? | 20:36 |
| rlandy|ruck | or an equivalent for overcloud-hardened-uefi-full.qcow | 20:36 |
| stevebaker | rlandy|ruck: oh yeah I'm making guesses about what is actually doing the upload. I also have this change https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 20:36 |
| rlandy|ruck | that's correct ^^ | 20:37 |
| stevebaker | rlandy|ruck: should I drop the fs002 change to avoid confusion? | 20:37 |
| rlandy|ruck | stevebaker: yep | 20:37 |
| rlandy|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/34251 has a zuul error | 20:38 |
| * rlandy|ruck looks | 20:38 | |
| rdogerrit | Steve Baker proposed rdo-jobs master: Upload overcloud-hardened-uefi-full in main integration https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 20:38 |
| rlandy|ruck | stevebaker: we'd need to merge https://review.rdoproject.org/r/c/config/+/34250 for you first | 20:39 |
| stevebaker | yes I think that one is ready | 20:39 |
| rlandy|ruck | stevebaker: ok w+ing that | 20:40 |
| stevebaker | awesome, thanks | 20:40 |
| rlandy|ruck | stevebaker: once that goes through, we can recheck the master promotion line one | 20:40 |
| stevebaker | rlandy|ruck: much appreciated | 20:41 |
| rdogerrit | Merged config master: Upload job for overcloud-hardened-uefi-full https://review.rdoproject.org/r/c/config/+/34250 | 20:42 |
| rlandy|ruck | stevebaker: k- going to leave a note for tomorrow's ruck/rovers to check that this job is running ok | 20:43 |
| stevebaker | rlandy|ruck: ok. The intention is for this job to remove the depends-on, so that fs001 downloads the image instead of building it | 20:45 |
| stevebaker | rlandy|ruck: this change I mean https://review.opendev.org/c/openstack/tripleo-quickstart/+/801409 | 20:45 |
| rlandy|ruck | stevebaker: ^^ the control of whether to build or not is usually set on the job directly ... | 20:46 |
| rlandy|ruck | getting | 20:46 |
| rlandy|ruck | to_build setting | 20:47 |
| rlandy|ruck | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ovb-jobs.yaml#L613 | 20:47 |
| stevebaker | I think I tracked to_build being set when there was a depends-on to one of a list of projects | 20:47 |
| rlandy|ruck | so correct | 20:48 |
| rlandy|ruck | if there is a change that could impact the image, we may need to rebuild it | 20:48 |
| rlandy|ruck | but, we have an overriding setting to_build ... | 20:48 |
| rlandy|ruck | which will build images rather than download them even if there is no change | 20:48 |
| stevebaker | I just wanted prove fs001 to be green for whole-disk images, once it is downloadable I'll remove the depends-on | 20:49 |
| rlandy|ruck | for the periodic jobs, we should downlosd always | 20:49 |
| rlandy|ruck | as there is no possibility of a change | 20:49 |
| stevebaker | ok | 20:49 |
| rlandy|ruck | with no depends-on fs001 should download | 20:49 |
| rlandy|ruck | I assume we have to do this in order ... | 20:50 |
| rlandy|ruck | merge your config change ( in progress) | 20:50 |
| rlandy|ruck | then merge the job to add the change to the line | 20:50 |
| stevebaker | I think land this next https://review.rdoproject.org/r/c/rdo-jobs/+/34251 | 20:51 |
| rlandy|ruck | then we can depends-on your https://review.opendev.org/c/openstack/tripleo-quickstart/+/801409 patch in a tetsproject | 20:51 |
| rlandy|ruck | and confirm it works | 20:51 |
| rlandy|ruck | ^^ ok? | 20:51 |
| stevebaker | then wait for published images to be promoted enough so that https://review.opendev.org/c/openstack/tripleo-quickstart/+/801409 is green when it downloads the image | 20:52 |
| rlandy|ruck | https://review.rdoproject.org/r/c/config/+/34250 merged | 20:52 |
| stevebaker | yep, I"ve rechecked | 20:52 |
| * rlandy|ruck checks what is running in master now | 20:53 | |
| rlandy|ruck | let's testproject your job to upload the image | 20:54 |
| stevebaker | yes lets | 20:54 |
| rlandy|ruck | so we can get a start on testing | 20:54 |
| rlandy|ruck | sec | 20:54 |
| rlandy|ruck | stevebaker: https://review.rdoproject.org/r/c/testproject/+/16882 | 20:58 |
| rlandy|ruck | https://review.rdoproject.org/r/c/testproject/+/16882/35/.zuul.yaml | 20:58 |
| rlandy|ruck | ^^ should build and upload your image | 20:58 |
| rlandy|ruck | and once that passes, run fs001 with depends-on your quickstart change | 20:59 |
| * stevebaker gets popcorn | 20:59 | |
| * rlandy|ruck goes to deal with pnt hardware requirements ... pls ping if any problems | 21:00 | |
| stevebaker | rlandy|ruck: thanks very much! | 21:00 |
| rlandy|ruck | sure | 21:00 |
| rlandy|ruck | stevebaker: I don;t see any image upload https://images.rdoproject.org/centos8/master/rdo_trunk/9babf57c9c44edca718cb41887ed4936/ | 21:41 |
| rlandy|ruck | the image built fine | 21:41 |
| rlandy|ruck | probably need a new dir to upload | 21:42 |
| rlandy|ruck | ah missing the tar file https://review.rdoproject.org/r/c/config/+/34250/2/ci-scripts/tripleo-upstream/upload-cloud-images.sh#36 | 21:53 |
| stevebaker | rlandy|ruck: yeah I've got a fix, hang on | 21:54 |
| rdogerrit | Steve Baker created config master: Don't assume a tar file will be uploaded https://review.rdoproject.org/r/c/config/+/34623 | 21:55 |
| stevebaker | rlandy|ruck: ^^ | 21:58 |
| rlandy|ruck | stevebaker: why is there no tar file in this case? | 21:58 |
| stevebaker | rlandy|ruck: we're uploading the actual qcow2 file. It doesn't need to be a tar because there is only one file, not three | 21:59 |
| rlandy|ruck | also the md5 | 21:59 |
| stevebaker | no ramdisk or kernel file is required for a whole-disk image | 21:59 |
| rlandy|ruck | ok | 21:59 |
| rlandy|ruck | let's try this | 22:00 |
| rlandy|ruck | it's config | 22:00 |
| rlandy|ruck | so we have to merge and pray | 22:00 |
| stevebaker | I think the md5 is seperate | 22:00 |
| rlandy|ruck | no speculative testing on config repos :( | 22:00 |
| rlandy|ruck | yep - the md5 is separate | 22:00 |
| rlandy|ruck | merging | 22:01 |
| rdogerrit | Merged config master: Don't assume a tar file will be uploaded https://review.rdoproject.org/r/c/config/+/34623 | 22:03 |
| * rlandy|ruck rekicks testproject | 22:04 | |
| rlandy|ruck | running again | 22:06 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:36 | |
| stevebaker | rlandy|ruck|bbl: oh hey https://images.rdoproject.org/centos8/master/rdo_trunk/9babf57c9c44edca718cb41887ed4936/ | 22:51 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!