| *** rlandy has quit IRC | 00:00 | |
| *** holser has joined #oooq | 00:14 | |
| *** holser has quit IRC | 00:28 | |
| *** tosky has quit IRC | 00:46 | |
| *** rlandy has joined #oooq | 00:52 | |
| *** Goneri has quit IRC | 01:12 | |
| *** jmasud has joined #oooq | 01:22 | |
| *** rlandy has quit IRC | 02:09 | |
| *** apetrich has quit IRC | 03:09 | |
| *** ykarel|away is now known as ykarel | 04:14 | |
| *** saneax has joined #oooq | 04:53 | |
| *** jmasud has quit IRC | 04:54 | |
| *** jmasud has joined #oooq | 04:56 | |
| *** udesale has joined #oooq | 04:57 | |
| *** skramaja has joined #oooq | 05:18 | |
| *** ratailor has joined #oooq | 05:20 | |
| *** gchamoul has quit IRC | 06:10 | |
| *** holser has joined #oooq | 06:49 | |
| *** dpawlik has joined #oooq | 06:57 | |
| *** marios has joined #oooq | 07:06 | |
| *** jfrancoa has joined #oooq | 07:06 | |
| *** jfrancoa is now known as jfrancoa|lunch | 07:07 | |
| *** dpawlik has quit IRC | 07:41 | |
| *** dpawlik has joined #oooq | 07:42 | |
| *** bogdando has joined #oooq | 07:43 | |
| *** dpawlik has quit IRC | 07:48 | |
| *** raukadah is now known as chandankumar | 07:58 | |
| *** ccamacho has joined #oooq | 08:06 | |
| *** danpawlik has joined #oooq | 08:15 | |
| *** tesseract has joined #oooq | 08:17 | |
| *** apetrich has joined #oooq | 08:19 | |
| *** dtantsur|afk is now known as dtantsur | 08:24 | |
| *** amoralej|off is now known as amoralej | 08:31 | |
| *** holser has quit IRC | 08:38 | |
| *** jtomasek has joined #oooq | 08:43 | |
| *** holser has joined #oooq | 08:54 | |
| *** jpena|off is now known as jpena | 08:56 | |
| arxcruz|rover | apetrich: https://bugs.launchpad.net/tripleo/+bug/1867744 if you have time to take a look | 09:06 |
|---|---|---|
| openstack | Launchpad bug 1867744 in tripleo "Mistral failed command manually execute the following script: /var/lib/mistral/overcloud/ansible-playbook-command.sh" [Critical,Triaged] | 09:06 |
| *** holser has quit IRC | 09:10 | |
| *** holser has joined #oooq | 09:13 | |
| *** tosky has joined #oooq | 09:17 | |
| apetrich | arxcruz|rover, looking | 09:18 |
| *** marios has quit IRC | 09:19 | |
| *** jfrancoa|lunch has quit IRC | 09:32 | |
| *** jfrancoa has joined #oooq | 09:47 | |
| chandankumar | arxcruz|rover, Hello | 10:14 |
| chandankumar | arxcruz|rover, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/6498437/logs/undercloud/var/log/containers/stdouts/container-puppet-neutron.log.txt.gz | 10:14 |
| chandankumar | arxcruz|rover, 17 09:58:26 puppet-user: Error: Could not set 'present' on ensure: No such file or directory @ rb_sysopen - /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini (file: /etc/puppet/modules/neutron/manifests/agents/ovn_metadata.pp, line: 150) | 10:14 |
| arxcruz|rover | chandankumar: ok, opening a bug | 10:14 |
| arxcruz|rover | was looking into mistral ones | 10:15 |
| ykarel | i think bug already exist | 10:15 |
| chandankumar | ykarel, link? | 10:15 |
| ykarel | iirc rlandy created last night | 10:15 |
| arxcruz|rover | ykarel: can you point me? | 10:15 |
| ykarel | need to find | 10:15 |
| ykarel | but should be one of top bugs | 10:15 |
| ykarel | https://bugs.launchpad.net/tripleo/+bug/1867664 | 10:16 |
| openstack | Launchpad bug 1867664 in tripleo "Master periodic jobs are failing overcloud deploy with ''Container(s) with bad ExitCode: [''container-puppet-neutron''], check logs in /var/log/containers/stdouts/'" [Critical,Triaged] | 10:16 |
| arxcruz|rover | yup | 10:16 |
| arxcruz|rover | chandankumar: it's on ruck and rover notes under integration pipeline | 10:16 |
| arxcruz|rover | chandankumar: https://hackmd.io/7MBqFHurTA2e5H8kYRwgag?both#integration-pipeline-failuresbugs | 10:16 |
| chandankumar | arxcruz|rover, ah, missed that, thanks! | 10:17 |
| *** marios has joined #oooq | 10:17 | |
| *** skramaja has quit IRC | 10:18 | |
| holser | hey folks | 10:51 |
| holser | one of our job failed due to https://zuul.opendev.org/t/openstack/build/ec5ac32649f1433780ff02f0375a215f/log/logs/undercloud/home/zuul/standalone_deploy.log | 10:51 |
| holser | 2020-03-16 21:31:57 | tripleo_common.image.exception.ImageNotFoundException: Not found image: docker://docker.io/tripleomaster/centos-binary-aodh-api:e03f7a0753e60c3263a20e4e6abf8512 | 10:52 |
| holser | 356 | 10:52 |
| holser | looks like we don't have all images uploaded to docker.io | 10:52 |
| marios | holser: o/ sounds like bad promotion arxcruz|rover did we have any issues last night you are aware of ? | 10:56 |
| marios | holser: master? link to change please? | 10:56 |
| holser | sec | 10:56 |
| holser | marios - https://review.opendev.org/#/c/713330/1 | 10:56 |
| holser | standalone job | 10:56 |
| marios | holser: ack.. scen2 but that is strange that the others pass that one doesnt wtf | 10:58 |
| marios | holser: try a recheck it looks pretty green https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario002-standalone | 10:58 |
| marios | holser: i wonder if promotion happened during that zuul run.. e.g. scen1 is using a different hash : https://4a5e3592766e68d7e4c2-3dc3e41248caa5c56d618587d77ea9dd.ssl.cf1.rackcdn.com/713330/1/check/tripleo-ci-centos-8-scenario001-standalone/c2d84ae/logs/undercloud/home/zuul/standalone_deploy.log (cant see e03f7a0753e60c3263a20e4e6abf8512 ) | 11:01 |
| holser | marios I'll try to rebase it | 11:02 |
| holser | thanks | 11:02 |
| ykarel | marios, there was some bad promotion last night as per irc logs | 11:06 |
| ykarel | http://eavesdrop.openstack.org/irclogs/%23oooq/%23oooq.2020-03-16.log.html#t2020-03-16T20:34:06 | 11:07 |
| marios | ykarel: ah thanks ... but that one was weird its like scen2 was using different hash... may be related to that ^ thank you | 11:08 |
| *** jbadiapa has joined #oooq | 11:19 | |
| *** holser has quit IRC | 11:45 | |
| *** ysandeep is now known as ysandeep|afk | 11:46 | |
| *** holser has joined #oooq | 11:56 | |
| arxcruz|rover | test | 11:58 |
| marios | sshnaidm: thanks for checking https://review.rdoproject.org/r/#/c/25932/1/roles/ovb-manage/defaults/main.yml think i'll try branch instead | 12:22 |
| marios | sshnaidm: but its a tricky one cos its config repo so difficult to test it | 12:22 |
| amoralej | chandankumar, you know if there are jobs with octavia and ovn backend for centos8 already running? | 12:23 |
| marios | amoralej: we hav scen10-ovn-provider there https://review.opendev.org/#/c/711507/ for running only in testproject assuming this is what you're looking for sec fetching | 12:25 |
| cgoncalves | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone | 12:25 |
| amoralej | chandankumar, cgoncalves https://review.rdoproject.org/r/#/c/24462/ will break it | 12:26 |
| marios | cgoncalves: ack yeah they are all runs from that review (adds the job there) https://review.opendev.org/#/c/711507/ | 12:26 |
| amoralej | until there is proper package for ovn-octavia-provider and fix in tripleo-common | 12:26 |
| marios | periodic there (we merged it non voting/not in criteria, until we move to os-tempest, it currently does validate tempest) https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master | 12:27 |
| marios | amoralej: looking (they made some packaging changees?) | 12:27 |
| marios | amoralej: danger is that this job is non voting/gating/promoting so it would be very easy for us to miss something like that change thanks for pointing to it | 12:28 |
| *** rlandy has joined #oooq | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| weshay|ruck | arxcruz|rover, 0/ | 12:38 |
| rlandy | jpena|lunch: ping (when you get back) re: tracking aggregate hashes that were in current | 12:40 |
| rlandy | jpena|lunch: the reproducer picks up dlrn_hash_tag_newest - which is current ... but when the developer comes to use that hash, it's nowhere we can see it ... is there a way to track that hash? query the dlrn api? | 12:41 |
| weshay|ruck | arxcruz|rover, can you debug https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-train-containers-build-push-vexxhost/5ad3f9a/logs/containers-failed-to-build.log | 12:41 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-train-containers-build-push-vexxhost/5ad3f9a/logs/buildah-builds/kolla-995vksng/docker/collectd/collectd-build.log | 12:42 |
| weshay|ruck | arxcruz|rover, looks like we need a kolla patch update | 12:42 |
| weshay|ruck | or rebase | 12:42 |
| sshnaidm | marios, maybe, but isn't ansible distro is better for this variable? I mean if it's running on centos8, so external image should be c8 as well | 12:42 |
| rlandy | chandankumar: pls review ... https://review.opendev.org/#/c/713333 | 12:47 |
| arxcruz|rover | weshay|ruck: ack | 12:48 |
| marios | sshnaidm: ack makes sense will try/consider | 12:49 |
| arxcruz|rover | weshay|ruck: i think chandankumar was working on something related to collectd | 12:50 |
| arxcruz|rover | nevermind | 12:51 |
| *** udesale_ has joined #oooq | 12:53 | |
| chandankumar | arxcruz|rover, collectd was fixed long time ago | 12:53 |
| chandankumar | rlandy, I fixed fs030 also | 12:54 |
| chandankumar | https://review.rdoproject.org/zuul/build/248d80517d0d43c3ba68db5ef4e9d9f7 | 12:54 |
| *** udesale has quit IRC | 12:55 | |
| weshay|ruck | chandankumar, thanks for following https://review.rdoproject.org/r/#/c/24462/ | 12:56 |
| weshay|ruck | arxcruz|rover, chandankumar re: collectd.. we carry patches for rhel builds on train | 12:56 |
| weshay|ruck | they may just need a rebase https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/vars/main.yaml#L7 | 12:57 |
| weshay|ruck | chandankumar, any progress on 3rd party dep jobs? | 12:57 |
| chandankumar | weshay|ruck, https://review.rdoproject.org/r/#/q/topic:c8-podman+(status:open+OR+status:merged) | 12:58 |
| chandankumar | merge these two and I am done with podman | 12:58 |
| weshay|ruck | reviews please ^ | 12:59 |
| *** ysandeep|afk is now known as ysandeep | 12:59 | |
| *** arxcruz|rover has quit IRC | 13:00 | |
| rlandy | chandankumar: weshay|ruck: panda|off: https://meet.google.com/bqx-xwht-wky - if you want to centos8 sync | 13:00 |
| rlandy | marios and I are one | 13:00 |
| rlandy | on | 13:00 |
| chandankumar | rlandy, weshay|ruck https://review.rdoproject.org/r/#/c/24462/ | 13:01 |
| *** sshnaidm is now known as sshnaidm|afk | 13:02 | |
| *** arxcruz has joined #oooq | 13:03 | |
| arxcruz | test | 13:03 |
| arxcruz | test | 13:03 |
| *** arxcruz is now known as arxcruz|rover | 13:03 | |
| arxcruz|rover | weshay|ruck: so you want me to rebase https://review.opendev.org/#/c/693159/ ? | 13:04 |
| weshay|ruck | arxcruz|rover, https://review.opendev.org/#/c/693390/4 is under it | 13:05 |
| arxcruz|rover | weshay|ruck: but https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/vars/main.yaml#L7 is pointing to 693159 | 13:07 |
| weshay|ruck | arxcruz|rover, they are on top of each other | 13:13 |
| *** amoralej is now known as amoralej|lunch | 13:13 | |
| weshay|ruck | so they are both pulled | 13:13 |
| zbr | https://github.com/openstack/tripleo-quickstart/blob/master/tox.ini#L10 | 13:18 |
| weshay|ruck | arxcruz|rover, so looking at https://review.opendev.org/#/c/693159/ | 13:20 |
| weshay|ruck | which does NOT need a rebase | 13:20 |
| weshay|ruck | but look at the patch's parent | 13:20 |
| arxcruz|rover | weshay|ruck: so all it's need is click on rebase button ? | 13:24 |
| marios | * https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/54a83b8/job-output.txt | 13:25 |
| marios | * 2020-03-17 02:15:15.792080 | primary | 2020-03-17 02:15:15.549 227783 ERROR tempest urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable> | 13:25 |
| arxcruz|rover | marios: is glance available? | 13:26 |
| *** Goneri has joined #oooq | 13:27 | |
| marios | arxcruz|rover: didn't check that (in call right now we are discussing) | 13:27 |
| arxcruz|rover | marios: seems to be iptables issue the endpoint isn't being reached | 13:30 |
| *** holser has quit IRC | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** holser has joined #oooq | 13:34 | |
| chandankumar | weshay|ruck, https://review.opendev.org/#/c/710473/ | 13:35 |
| jpena | rlandy: hm, so for current/consistent we are not currently tracking all previous hashes (as we do for other promotions). If this can be useful, it's something I need to add in DLRN (should be easy code-wise) | 13:37 |
| rlandy | jpena: we discussed it a little with the team ... I am not sure it's something we need right away - is there any way to query the dlnr api for the aggregate hash? | 13:38 |
| rlandy | it really only impacts the reproducer | 13:38 |
| jpena | rlandy: currently not. current/consistent are not considered as promotions, so we don't store any aggregate hash there | 13:41 |
| rlandy | jpena: if it's not a major work item for you, being able to track the exact current hash would be useful - but it's not urgent. I will get back to stevebaker who requested this tracking and let him know. | 13:43 |
| rlandy | thanks | 13:43 |
| jpena | rlandy: ack, I'll work on it | 13:43 |
| rlandy | jpena: thanks for your awesome support | 13:44 |
| *** chem has quit IRC | 13:44 | |
| *** chem has joined #oooq | 13:46 | |
| rlandy | marios: thanks for fixing https://review.opendev.org/#/c/711507/ - +2 | 13:48 |
| *** ysandeep is now known as ysandeep|afk | 14:01 | |
| *** jtomasek has quit IRC | 14:12 | |
| *** jtomasek has joined #oooq | 14:14 | |
| *** jtomasek has quit IRC | 14:15 | |
| *** amoralej|lunch is now known as amoralej | 14:15 | |
| *** ykarel is now known as ykarel|afk | 14:17 | |
| *** pojadhav is now known as pojadhav|away | 14:21 | |
| rlandy | chandankumar: are we ok to merge on https://review.opendev.org/#/c/713277? | 14:24 |
| chandankumar | rlandy, yes if molecule job passed | 14:26 |
| chandankumar | *passes | 14:26 |
| *** TrevorV has joined #oooq | 14:27 | |
| chandankumar | rlandy, +w | 14:27 |
| rlandy | chandankumar: ok - what other jobs need reviews/w+? | 14:30 |
| rlandy | let's try get done here | 14:30 |
| rlandy | looking at fs030 reviews | 14:30 |
| chandankumar | rlandy, check rrpad | 14:30 |
| chandankumar | rlandy, https://review.opendev.org/712013 and https://review.opendev.org/713397 | 14:31 |
| chandankumar | not there on etherpad | 14:31 |
| rlandy | https://review.opendev.org/#/c/712013/ w+ | 14:33 |
| chandankumar | depends on https://review.opendev.org/713397 | 14:33 |
| rlandy | chandankumar: fs030 reviews all w+ | 14:35 |
| rlandy | https://review.rdoproject.org/r/25910 also w+ | 14:37 |
| rlandy | just job def | 14:37 |
| rlandy | chandankumar: ok to w+ https://review.rdoproject.org/r/#/c/25916/? | 14:39 |
| rlandy | podman | 14:39 |
| chandankumar | rlandy, podman yes | 14:39 |
| rlandy | ok - here goes | 14:39 |
| * rlandy cleaning the review house | 14:39 | |
| chandankumar | rlandy, I went for grocerry, mini maggi packets are getting out of stock | 14:40 |
| rlandy | chandankumar: it's crazy here as well | 14:40 |
| rlandy | chandankumar: trying to get out of New York next week | 14:40 |
| chandankumar | rlandy, vegetables are easily available | 14:40 |
| rlandy | I have a flight but I don't know if it will go | 14:41 |
| rlandy | family lives in St Louis - its' much calmer there | 14:41 |
| rlandy | I might be driving the 15 hours to get there | 14:41 |
| chandankumar | rlandy, same here, we are thinking of hiring a cab driver and drive directly to home town via car by covering 1500 kms | 14:41 |
| chandankumar | if I go to airport, govt went sent to isolation ward for 14 days in india | 14:42 |
| marios | :( | 14:43 |
| rlandy | chandankumar: yeah - I am worried about the same thing | 14:43 |
| marios | same here everyone that arrives goes for 14 days isolation | 14:43 |
| rlandy | for domestic flights there is no isolation - yet | 14:43 |
| *** jbadiapa has quit IRC | 14:45 | |
| rlandy | chandankumar: have you seen some passes on scenario12 in rdocloud? | 14:46 |
| rlandy | if so, will merge the addition to pipeline | 14:47 |
| rlandy | I merged the job definition | 14:47 |
| *** panda|off has quit IRC | 14:49 | |
| *** panda has joined #oooq | 14:50 | |
| *** ratailor has quit IRC | 14:55 | |
| marios | sshnaidm|afk: thanks for the suggestion wdyt https://review.rdoproject.org/r/#/c/25932/2/roles/ovb-manage/defaults/main.yml | 14:56 |
| marios | cc rlandy weshay|ruck when you next have time please ^^ context in the commit message ... tricky bit is that is trusted/config repo | 14:57 |
| *** sshnaidm|afk is now known as sshnaidm | 14:57 | |
| *** dtantsur is now known as dtantsur|brb | 15:00 | |
| rlandy | chandankumar: hi - ping re: scenario12 question above ^^ | 15:00 |
| rlandy | closing out all these open reviews | 15:00 |
| rlandy | woohoo - component compute promoted | 15:01 |
| rlandy | and manila | 15:01 |
| rlandy | assume that unpinned now | 15:01 |
| rlandy | network promoted | 15:02 |
| *** jmasud has quit IRC | 15:20 | |
| chandankumar | rlandy, sorry was in a meeting with alan | 15:21 |
| chandankumar | rlandy, nope, not passing yet | 15:22 |
| chandankumar | rlandy, we can treat scneario 12 like ovn one | 15:23 |
| rlandy | chandankumar: fine - will add it to the pipeline - commented out in criteria | 15:24 |
| chandankumar | rlandy, current state is here https://a25c63f93d767ad4a9bb-ef02a6c5e2cf8d78c9c91d8a83ee626c.ssl.cf1.rackcdn.com/713202/1/check/tripleo-ci-centos-8-scenario012-standalone/1fc7b77/logs/undercloud/home/zuul/tempest.log | 15:25 |
| chandankumar | sc 12 | 15:25 |
| rlandy | chandankumar: as long as the scenario is still valid for testing, I'd rather see it fail so we know what our damage is | 15:26 |
| rlandy | no point in keeping these reviews floating out there | 15:27 |
| rlandy | merging up and we can fix afterwards | 15:27 |
| rlandy | fs039 is next | 15:28 |
| rlandy | weshay|ruck: just fyi - we're having a merge party now | 15:28 |
| rlandy | and arxcruz|rover | 15:28 |
| rlandy | if you see some new damage | 15:29 |
| chandankumar | marios, all fs039 patches rebased | 15:29 |
| arxcruz|rover | rlandy: do you have beer? If so I want to participate | 15:29 |
| marios | chandankumar: k thx | 15:29 |
| weshay|ruck | rlandy, merging centos-8 reviews? | 15:29 |
| chandankumar | marios, sorry forgot ?, rebasing now | 15:29 |
| chandankumar | rlandy, https://review.opendev.org/#/c/713202/ good to go | 15:29 |
| marios | chandankumar: ?? you lost me | 15:30 |
| marios | chandankumar: were you asking me to rebase, or telling me you rebased, with 17:29 < chandankumar> marios, all fs039 patches rebased | 15:30 |
| * marios checks | 15:30 | |
| marios | chandankumar: ack i see they need rebases | 15:31 |
| marios | chandankumar: will do | 15:31 |
| chandankumar | marios, aye thanks | 15:31 |
| rlandy | arxcruz|rover: sending you virtual beer | 15:31 |
| rlandy | weshay|ruck: ack | 15:31 |
| * chandankumar opens up the beer | 15:31 | |
| marios | chandankumar: right now i need reviews there please https://review.rdoproject.org/r/25932 (for fs39) | 15:32 |
| marios | chandankumar: when you next hav time | 15:32 |
| chandankumar | marios, on it | 15:32 |
| rlandy | chandankumar: k - w+'ing https://review.opendev.org/#/c/713202/ | 15:33 |
| weshay|ruck | arxcruz|rover, it just triggered again fyi https://review.rdoproject.org/zuul/stream/2c22be2fe1484b03aa1cf785e9f12f8c?logfile=console.log | 15:38 |
| weshay|ruck | arxcruz|rover, hopefully you got the rebase in time :)) | 15:38 |
| arxcruz|rover | weshay|ruck: let's pray to zuul god | 15:38 |
| chandankumar | rlandy, https://review.rdoproject.org/r/#/c/25793/ | 15:39 |
| chandankumar | marios, more rebase needed https://review.rdoproject.org/r/#/c/25794/ and https://review.rdoproject.org/r/#/c/25795/ | 15:40 |
| marios | chandankumar: yeah am doing the config one was a bit of mess seems weshay|ruck updated those last night | 15:43 |
| chandankumar | rlandy, marios https://review.rdoproject.org/r/#/c/25794/ for fs039 since it is a ovb job, we need to put depends on build images na? | 15:46 |
| zbr | i made a new release of pytest-molecule which is not affected by pytest-html bug, new builds should pick it automatically unless someone manually pinned it. | 15:47 |
| zbr | https://pypi.org/project/pytest-molecule/#history | 15:47 |
| rlandy | sec - will start looking at those reviews as soon as all the others are in | 15:47 |
| chandankumar | aye | 15:49 |
| chandankumar | marios, thanks! | 15:49 |
| *** ykarel|afk is now known as ykare | 15:51 | |
| *** ykare is now known as ykarel | 15:51 | |
| *** jbadiapa has joined #oooq | 16:10 | |
| weshay|ruck | arxcruz|rover, missed the window | 16:23 |
| *** danpawlik has quit IRC | 16:25 | |
| weshay|ruck | chandankumar, you still w/ us? can you update https://hackmd.io/7MBqFHurTA2e5H8kYRwgag w/ fs001 | 16:32 |
| weshay|ruck | or anyone who has the patches, bugs /me looks around | 16:32 |
| marios | weshay|ruck: am filing a bug for that now | 16:32 |
| weshay|ruck | marios, thank you! | 16:32 |
| marios | weshay|ruck: will add to the hackmd | 16:32 |
| weshay|ruck | thanks | 16:33 |
| weshay|ruck | marios, do you know if we're still seeing https://bugs.launchpad.net/tripleo/+bug/1866204 | 16:36 |
| openstack | Launchpad bug 1866204 in tripleo "OVB is stuck on providing nodes" [Critical,In progress] - Assigned to Luke Short (ekultails) | 16:36 |
| chandankumar | weshay|ruck, nope | 16:37 |
| weshay|ruck | rlandy, can we close https://bugs.launchpad.net/tripleo/+bug/1866696 | 16:37 |
| openstack | Launchpad bug 1866543 in tripleo "duplicate for #1866696 All CentOS-8 jobs fail on missing various packages from component repos" [Critical,Fix released] | 16:37 |
| marios | weshay|ruck: we doing promote blocker/alert on those now (component centos8) assume yes | 16:37 |
| marios | chandankumar: weshay|ruck: filed https://bugs.launchpad.net/tripleo/+bug/1867807 | 16:38 |
| openstack | Launchpad bug 1867807 in tripleo "periodic- centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master fails to download tempest cirros image" [Critical,Triaged] | 16:38 |
| weshay|ruck | marios, sorry.. yes.. if there is something blocking component pipeline, it's cix and assigned to the dfg who owns the component | 16:39 |
| weshay|ruck | marios, maybe list the comopnent name in the bug.. so it's easier to triage in cix | 16:39 |
| marios | weshay|ruck: ack i added critical /ci-blocker | 16:39 |
| marios | weshay|ruck: will remove the alert now | 16:39 |
| weshay|ruck | and the introspection issues? providing nodes | 16:40 |
| weshay|ruck | anyone see that in fs001? | 16:40 |
| *** dtantsur|brb is now known as dtantsur | 16:40 | |
| marios | weshay|ruck: done (18:39 < weshay|ruck> marios, maybe list the comopnent name in the bug.. so it's easier to triage in cix | 16:40 |
| weshay|ruck | thanks | 16:40 |
| weshay|ruck | chandankumar, want help w/ https://bugs.launchpad.net/tripleo/+bug/1867602 | 16:45 |
| openstack | Launchpad bug 1867602 in tripleo "overcloud deploy failed due to Systemd start for pcsd failed" [Critical,Triaged] | 16:45 |
| weshay|ruck | ? | 16:45 |
| *** ykarel is now known as ykarel|away | 16:45 | |
| chandankumar | weshay|ruck, we tried not able to reproduce | 16:45 |
| weshay|ruck | chandankumar, k :) | 16:47 |
| chandankumar | marios, thanks, will get it fixed tomorrow | 16:48 |
| chandankumar | see ya | 16:48 |
| *** chandankumar is now known as raukadah | 16:48 | |
| weshay|ruck | raukadah, 0/ | 16:48 |
| weshay|ruck | thanks | 16:48 |
| marios | raukadah: ack let's talk in the morning i will work on that too | 16:48 |
| * marios shutdown sequence | 16:52 | |
| *** udesale_ has quit IRC | 16:57 | |
| weshay|ruck | rlandy, marios looking at the last several runs of fs001 c8.. in check.. it's getting to tempest and failing on basic-ops | 16:58 |
| * weshay|ruck wonders if we should disable the tests in check | 16:58 | |
| weshay|ruck | so we get some better visibility of the deployment.. | 16:58 |
| marios | weshay|ruck: is it a particular test? if we are reducing coverage lets make it as small as possible and just use skip list for that test | 17:01 |
| weshay|ruck | there are 9 tests in network basic ops.. | 17:01 |
| weshay|ruck | and 3-4 or inconsistently passing/failing | 17:01 |
| * weshay|ruck puts up patch to review | 17:04 | |
| rlandy | weshay|ruck: sorry lunch ... https://bugs.launchpad.net/tripleo/+bug/1866696 is a duplicate | 17:05 |
| openstack | Launchpad bug 1866543 in tripleo "duplicate for #1866696 All CentOS-8 jobs fail on missing various packages from component repos" [Critical,Fix released] | 17:05 |
| rlandy | weshay|ruck: it | 17:05 |
| * marios off | 17:05 | |
| *** marios is now known as marios|out | 17:05 | |
| rlandy | 's not consistent | 17:05 |
| rlandy | for fs001 | 17:05 |
| *** jfrancoa has quit IRC | 17:06 | |
| weshay|ruck | something like https://review.opendev.org/713489 | 17:07 |
| *** bogdando has quit IRC | 17:07 | |
| weshay|ruck | rlandy, I checked the last $x runs in fs001 mostly seeing tempest fails | 17:07 |
| rlandy | weshay|ruck: was that master integration and baremetal component? | 17:08 |
| weshay|ruck | rlandy, https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001 | 17:09 |
| rlandy | 2020-03-17 02:15:15.792080 | primary | 2020-03-17 02:15:15.549 227783 ERROR tempest urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable> | 17:09 |
| weshay|ruck | in check | 17:09 |
| rlandy | oh | 17:09 |
| rlandy | ok - been following the periodics | 17:09 |
| weshay|ruck | np | 17:09 |
| rlandy | weshay|ruck: tbh it's also tempest | 17:09 |
| weshay|ruck | ya.. two different root causes though | 17:10 |
| rlandy | in periodics - same issue marios raised | 17:10 |
| weshay|ruck | this patch reduces coverage in check for centos-8 | 17:10 |
| weshay|ruck | which I think is appropriate right now | 17:10 |
| weshay|ruck | aye | 17:10 |
| rlandy | weshay|ruck: ack | 17:10 |
| weshay|ruck | in periodic leave the coverage.. fix the tempest download.. etc | 17:10 |
| rlandy | for sure | 17:10 |
| rlandy | in fact - I want to enable fs001 on criteria | 17:11 |
| rlandy | so we DON'T promote w/o it | 17:11 |
| rlandy | thoughts? | 17:11 |
| *** marios|out has quit IRC | 17:12 | |
| rlandy | wrt https://review.opendev.org/#/c/713489/ - were we not going to deal with that in the job definitions? | 17:12 |
| zbr | DEFAULT NODESET ON RDO JUST GOT SWITCHED TO CENTOS-8, please keep an eye open for surprises starting from now. | 17:13 |
| weshay|ruck | zbr, what patch | 17:13 |
| zbr | https://review.rdoproject.org/r/#/c/25904/ | 17:13 |
| zbr | look at #rdo last notif | 17:13 |
| rlandy | weshay|ruck: also network promoted | 17:14 |
| weshay|ruck | rlandy, tempest is not defined in the job-defs at least atm.. w/ fs001 | 17:14 |
| weshay|ruck | only standalone.. | 17:14 |
| rlandy | and manila | 17:14 |
| weshay|ruck | rlandy, /me looking for positive feedback loop on introspection | 17:14 |
| rlandy | so assuming we got the unpinned content now? | 17:14 |
| rlandy | ack | 17:15 |
| weshay|ruck | zbr, ykarel|away well done guys!! | 17:16 |
| rlandy | zbr: good - about time | 17:16 |
| zbr | weshay|ruck: well,... only time will tell, likely if we do not have to revert it before monday we can call migration a success. | 17:16 |
| rlandy | raukadah: wrt https://bugs.launchpad.net/tripleo/+bug/1867807 did you contact dpawlik to upload the image? | 17:20 |
| openstack | Launchpad bug 1867807 in tripleo "periodic- centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master fails to download tempest cirros image" [Critical,Triaged] | 17:20 |
| rlandy | probably anyone from infra | 17:20 |
| rlandy | in fact so can we | 17:20 |
| rlandy | weshay|ruck: ^^ should we just upload that image? | 17:20 |
| rlandy | we upload images there all the time | 17:21 |
| weshay|ruck | rlandy, ya.. let's do it.. chat and do it? | 17:21 |
| weshay|ruck | we have access | 17:21 |
| rlandy | weshay|ruck: ack | 17:21 |
| rlandy | weshay|ruck: getting in tmux a on promoter | 17:21 |
| weshay|ruck | https://meet.google.com/bdg-pfkr-hwh?authuser=1 | 17:21 |
| weshay|ruck | rlandy, you know which image? | 17:22 |
| *** jfrancoa has joined #oooq | 17:23 | |
| rlandy | weshay|ruck: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img | 17:23 |
| rlandy | https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/54a83b8/job-output.txt | 17:27 |
| rlandy | weshay|ruck: ^^ | 17:27 |
| weshay|ruck | "tempest_test_image_path": "/opt/cache/files/cirros-0.3.5-x86_64-disk.img", | 17:27 |
| weshay|ruck | arxcruz|rover, you still around by chance? looking for tempest_test_image_path that's pulling the 4.0 cirros | 17:33 |
| weshay|ruck | it's no where to be found | 17:33 |
| arxcruz|rover | weshay|ruck: hey | 17:33 |
| weshay|ruck | arxcruz|rover, hey.. so fs001 can't download the cirros 4 image | 17:34 |
| arxcruz|rover | weshay|ruck: network issues? the cirros image is there | 17:36 |
| *** jmasud has joined #oooq | 17:36 | |
| arxcruz|rover | https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img | 17:36 |
| weshay|ruck | arxcruz|rover, ya.. we know WHERE it is | 17:36 |
| weshay|ruck | arxcruz|rover, we just found the def.. in os-tempest/vars/main.yml | 17:37 |
| *** tesseract has quit IRC | 17:37 | |
| weshay|ruck | 2020-03-17 02:04:59.136306 | primary | TASK [os_tempest : Image(s) download] ****************************************** | 17:38 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/54a83b8/job-output.txt | 17:38 |
| weshay|ruck | arxcruz|rover, can we override the url that python-temestconf uses | 17:42 |
| weshay|ruck | ? | 17:42 |
| arxcruz|rover | weshay|ruck: yes, i'm checking right now | 17:43 |
| weshay|ruck | rlandy, 2020-03-17 02:04:59.136306 | primary | TASK [os_tempest : Image(s) download] ****************************************** | 17:43 |
| weshay|ruck | arxcruz|rover, rlandy we could also enable os-ansible to download the image | 17:44 |
| weshay|ruck | which is skipping | 17:44 |
| weshay|ruck | so.. I guess os-ansible skips the download because of - tempest_service_available_glance | bool | 17:47 |
| *** jmasud has quit IRC | 17:47 | |
| arxcruz|rover | weshay|ruck: do we have a cirros file in the vm already ? | 17:48 |
| arxcruz|rover | like /opt/cache/files/cirros-bla | 17:48 |
| weshay|ruck | arxcruz|rover, upstream we do | 17:48 |
| weshay|ruck | arxcruz|rover, agree we should have it | 17:48 |
| arxcruz|rover | weshay|ruck: asking because we can configure it to use it instead of download | 17:48 |
| weshay|ruck | but atm.. we're trying to upload the cirros to our image server | 17:48 |
| weshay|ruck | and then will look at dib | 17:48 |
| weshay|ruck | arxcruz|rover, well.. python tempestconf tries to download if the image is not found | 17:49 |
| arxcruz|rover | weshay|ruck: so, the problem is that tempestconf is trying to download | 17:49 |
| weshay|ruck | arxcruz|rover, agree.. the rdo nodepool image should have it | 17:50 |
| weshay|ruck | and it's a diff between upstream and downstream | 17:50 |
| arxcruz|rover | yeah, so, if there's no image on the vm, it must get it downloaded, and the download is failing, is that a known issue ? | 17:51 |
| arxcruz|rover | so we either have the cirros already in the vm, or it needs to be downloaded | 17:51 |
| *** jmasud has joined #oooq | 17:55 | |
| weshay|ruck | rlandy, https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-baremetal-master/247dcf4/job-output.txt | 17:56 |
| weshay|ruck | arxcruz|rover, ya.. this is odd.. ovb uses the same image as standalone | 17:57 |
| weshay|ruck | standalone appears to have it built in | 17:57 |
| arxcruz|rover | the other option is fix the download issue | 17:58 |
| arxcruz|rover | we can even download from another source | 17:58 |
| *** jmasud has quit IRC | 17:58 | |
| *** dtantsur is now known as dtantsur|afk | 17:59 | |
| weshay|ruck | arxcruz|rover, ok.. for tomorrow :) | 18:00 |
| weshay|ruck | thanks for popping on late | 18:00 |
| weshay|ruck | arxcruz|rover, we also need to clean up some old tempest cirros config | 18:00 |
| arxcruz|rover | weshay|ruck: you can set here for example: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset001.yml#L190 image.image_path: 'https://download.another.cirros.repo' | 18:03 |
| weshay|ruck | arxcruz|rover, thank you | 18:04 |
| *** jpena is now known as jpena|off | 18:04 | |
| weshay|ruck | legit blocked | 18:10 |
| weshay|ruck | rlandy, http://images.rdoproject.org/cirros/cirros-0.4.0-x86_64-disk.img | 18:10 |
| *** saneax has quit IRC | 18:35 | |
| *** rfolco has joined #oooq | 18:44 | |
| * rfolco is alive | 18:45 | |
| rfolco | how is everybody? | 18:46 |
| rlandy | rfolco: you back?? | 18:46 |
| rfolco | not yet, next monday | 18:46 |
| rlandy | we're all alive - semi-quarantined but alive | 18:47 |
| rfolco | same here | 18:47 |
| rfolco | ok cool ttyl | 18:48 |
| *** rfolco has quit IRC | 18:48 | |
| weshay|ruck | missed him | 19:00 |
| *** amoralej is now known as amoralej|off | 19:02 | |
| *** dsneddon has quit IRC | 19:22 | |
| *** dsneddon has joined #oooq | 19:22 | |
| *** jmasud has joined #oooq | 19:48 | |
| weshay|ruck | rlandy, I have the kids for a bit.. I'll be back on later tonight if you need reviews | 20:17 |
| rlandy | weshay|ruck: sure - just working to prep baremetal | 20:17 |
| rlandy | nothing urgent | 20:18 |
| *** jfrancoa has quit IRC | 20:18 | |
| *** TrevorV has quit IRC | 20:25 | |
| rlandy | https://review.rdoproject.org/zuul/stream/24177c535c80457ea22af1cd61427d00?logfile=console.log | 20:40 |
| rlandy | tempest failed but nor due to download | 20:40 |
| *** jmasud has quit IRC | 20:55 | |
| *** jmasud has joined #oooq | 20:56 | |
| rlandy | going to dr's appt | 21:01 |
| rlandy | bbl | 21:02 |
| rlandy | weshay|ruck: ^^ | 21:02 |
| *** rlandy is now known as rlandy|afk | 21:02 | |
| *** rlandy|afk is now known as rlandy | 22:48 | |
| rlandy | weshay|ruck: weird ... https://logserver.rdoproject.org/12/25412/11/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/24177c5/job-output.txt | 22:50 |
| rlandy | doesn't look like it downloaded any image | 22:50 |
| rlandy | but it didn't fail there | 22:50 |
| *** Goneri has quit IRC | 22:50 | |
| *** holser has quit IRC | 22:56 | |
| *** tosky has quit IRC | 23:38 | |
| *** holser has joined #oooq | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!