| rlandy|bbl | "tripleowallabycentos9" promoting | 01:18 |
|---|---|---|
| rlandy|bbl | dasm|off: looks stuck on cloud-init | 01:23 |
| *** rlandy|bbl is now known as rlandy|out | 01:23 | |
| *** soniya29 is now known as soniya29|ruck | 02:50 | |
| * soniya29|ruck need to step out | 04:29 | |
| soniya29|ruck | ysandeep|out, i need to step out for a while | 04:30 |
| *** ysandeep|out is now known as ysandeep | 04:43 | |
| *** ysandeep is now known as ysandeep|rover | 04:43 | |
| ysandeep|rover | soniya29|ruck, ack o/ Let's sync when you are back. | 04:44 |
| ysandeep|rover | soniya29|ruck, new ruck/rover hackmd: https://hackmd.io/zjCkCnVeQ06bnBq836iF8g | 04:51 |
| ysandeep|rover | dasm|off, rlandy|out thanks for cloud-init pinning patches, /me checking what's the status currently. | 04:53 |
| ysandeep|rover | rlandy|out, I see a note on hackmd that we can skip fs035 and promote and you were rerunning testproject for fs035- I see your testproject failed but fs035 passed later in periodic run - /me triggering fs035 for older hash and other failed jobs for current hash - hopefully we will get better results now. | 05:08 |
| ysandeep|rover | soniya29|ruck, I am looking into the c8 train cloud-init issue. | 05:10 |
| soniya29|ruck | ysandeep|rover, shall we sync? | 06:33 |
| ysandeep|rover | soniya29|ruck, I am in a debug session with cedric, I will ping back after few mins | 06:34 |
| soniya29|ruck | ysandeep|rover, sure | 06:34 |
| soniya29|ruck | ysandeep|rover, i will start looking on downstream or do you want me to look at something else? | 06:35 |
| ysandeep|rover | soniya29|ruck, please start with monitoring gate first and then go to rdo promotion pipelines - see if we have any new issue. | 06:36 |
| soniya29|ruck | ysandeep|rover, ansible-collections-openstack-functional-devstack-octavia is failing with error - 'Provided object does not match schema' but it seems it has been made non-voting | 06:42 |
| soniya29|ruck | cockpit reports it as a gate failure | 06:42 |
| soniya29|ruck | ysandeep|rover, ansible-collections-openstack-functional-devstack-releases is green with latest run on stable/1.0.0, earlier it was also failing with above error | 06:44 |
| ysandeep|rover | soniya29|ruck, leave ansible-collections to jm1 , please check tripleo repos instead | 06:45 |
| soniya29|ruck | ysandeep|rover, cs9 standalone is failing because of ssh connection issues | 06:46 |
| soniya29|ruck | though cs8 standalone has passed | 06:46 |
| soniya29|ruck | ysandeep|rover, cs9 standalone multinode ipa is failing on tempest test - TestNetworkBasicOps..it seems a race condition with the resources | 06:48 |
| ysandeep|rover | ack please report and debug if they are consistent if we don't have a bug already | 06:49 |
| soniya29|ruck | ysandeep|rover, sure | 06:50 |
| jm1 | soniya29|ruck: thank you :) We have a couple of failing jobs in ansible openstack collection but this is expected. I will talk about our CI in next ci community meeting. I am continously monitoring our aom jobs, so you can just ignore them ;) | 06:53 |
| soniya29|ruck | jm1: sure, thanks :) | 06:53 |
| *** jpena|off is now known as jpena | 07:01 | |
| soniya29|ruck | ysandeep|rover, cs9 master has been promoted 8 days ago..looking into it | 07:08 |
| ysandeep|rover | soniya29|ruck, ovb issues | 07:09 |
| soniya29|ruck | ysandeep|rover, should i go looking into it or chase different stuffs? | 07:10 |
| ysandeep|rover | You can leave ovb debug on me and check new failures if any | 07:13 |
| soniya29|ruck | ysandeep|rover, okay, thanks | 07:16 |
| ysandeep|rover | rlandy|out, dasm|off hopefully https://review.opendev.org/c/openstack/tripleo-ci/+/840826 will fix the c8 train issue, /me testing | 07:37 |
| ysandeep|rover | soniya29|ruck, I am available to sync now | 07:40 |
| * ysandeep|rover looking into ovb tempest issue now, i got a reproducer | 07:45 | |
| soniya29|ruck | ysandeep|rover, give me 10 min | 08:15 |
| ysandeep|rover | soniya29|ruck, let's meet at 02:30 pm - I will also head out for lunch soon | 08:15 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:20 | |
| soniya29|ruck | ysandeep|rover|lunch, ack | 08:36 |
| *** soniya29|ruck is now known as soniya29|ruck|lunch | 08:36 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:59 | |
| *** soniya29|ruck|lunch is now known as soniya29|ruck | 09:09 | |
| soniya29|ruck | ysandeep|rover, meet.google.com/twa-avjs-dtq | 09:10 |
| rlandy|out | soniya29|ruck: ysandeep|rover: hello | 10:21 |
| soniya29|ruck | rlandy|out, hello | 10:21 |
| rlandy|out | soniya29|ruck: ysandeep|rover: let's touch base pls | 10:21 |
| *** rlandy|out is now known as rlandy | 10:21 | |
| rlandy | soniya29|ruck: ysandeep|rover: re: status of OVB | 10:22 |
| ysandeep|rover | rlandy, ack | 10:22 |
| soniya29|ruck | rlandy, ack | 10:22 |
| rlandy | ysandeep|rover: soniya29|ruck: https://meet.google.com/kcm-dyso-xbq?pli=1&authuser=0 | 10:22 |
| rlandy | soniya29|ruck: https://bugs.launchpad.net/tripleo/+bug/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors | 10:29 |
| ysandeep|rover | https://review.opendev.org/q/topic:discover-latest-image | 10:35 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-compute-rhos-17/fedcc2e/logs/undercloud/home/zuul/install_packages.sh.log | 10:44 |
| rlandy | Problem: package python3-tripleoclient-16.4.1-0.20220428005553.095182c.el8osttrunk.noarch requires python3-tripleo-common >= 15.2.0, but none of the providers can be installed | 10:44 |
| ysandeep|rover | rlandy, http://pastebin.test.redhat.com/1050013 | 10:53 |
| soniya29|ruck | rlandy, ysandeep|rover https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42601 | 11:13 |
| rlandy | soniya29|ruck: thank you - merging - pls revert once master promotes | 11:14 |
| soniya29|ruck | rlandy, sure | 11:14 |
| soniya29|ruck | ysandeep|rover, cs8 standalone victoria is failing on tempest because of timed_out/connection issues | 11:17 |
| ysandeep|rover | soniya29|ruck, failure consistent? | 11:17 |
| soniya29|ruck | ysandeep|rover, yes | 11:17 |
| soniya29|ruck | it is consistently failing for 3 runs | 11:17 |
| ysandeep|rover | on same test? | 11:17 |
| rlandy | chandankumar: you joining review time? | 11:18 |
| soniya29|ruck | ysandeep|rover, let me cross-verify for same tests | 11:18 |
| soniya29|ruck | ysandeep|rover, not on same tests particularly but on compute, network and in one of the runs all the tests are failing | 11:19 |
| soniya29|ruck | but failure reason seems similar | 11:19 |
| rlandy | dviroel|out: we're looking at your review https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42378 | 11:24 |
| rlandy | can you explain when you are in? | 11:25 |
| *** dviroel|out is now known as dviroel | 11:35 | |
| * soniya29|ruck takes a break | 11:49 | |
| *** soniya29|ruck is now known as soniya29|ruck|afk | 11:49 | |
| *** ysandeep|rover is now known as ysandeep|afk | 11:56 | |
| *** soniya is now known as soniya|ruck|afk | 11:57 | |
| *** pojadhav is now known as pojadhav|break | 11:58 | |
| chandankumar | dviroel: please approve this one https://review.opendev.org/c/openstack/tripleo-ci/+/839149 , thanks! | 11:59 |
| rlandy | soniya29|ruck|afk: ysandeep|afk: going to BZ this https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario004-standalone-glance-rhos-17/f6483d5/logs/undercloud/home/zuul/install_packages.sh.log | 12:01 |
| * dviroel looks | 12:03 | |
| *** pojadhav|break is now known as pojadhav | 12:29 | |
| rlandy | chandankumar: let's sync | 12:31 |
| rlandy | https://meet.google.com/ggp-fbnq-xea?pli=1&authuser=0 | 12:31 |
| *** ysandeep|afk is now known as ysandeep|rover | 12:42 | |
| ysandeep|rover | rlandy, ack | 12:47 |
| rlandy | ysandeep|rover: soniya|ruck|afk: master promoted - reverting criteria patch | 12:48 |
| ysandeep|rover | nice :) | 12:48 |
| rlandy | done | 12:48 |
| ysandeep|rover | perf issue on vexx is not consistent, podman ps is now returning in micro seconds now.. | 12:49 |
| rlandy | ysandeep|rover: welcome to all of last week :) | 12:51 |
| rlandy | master not looking great | 12:52 |
| rlandy | soniya|ruck|afk: ysandeep|rover: let's rekick c8 line when your patch merges | 12:53 |
| ysandeep|rover | ack | 12:53 |
| ysandeep|rover | we don't create dstat.html on overcloud nodes, would have helped us with performance issue. | 12:58 |
| * ysandeep|rover checking if we can enable that for overcloud nodes | 12:58 | |
| rlandy | pojadhav: just logging a bug - will be there in 5 | 13:00 |
| pojadhav | rlandy, yep | 13:00 |
| rlandy | pojadhav: https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/ | 13:11 |
| rlandy | https://github.com/rdo-infra/ci-config | 13:11 |
| dviroel | ysandeep|rover: would help a lot - yes | 13:14 |
| *** chem_ is now known as chem | 13:17 | |
| *** chem is now known as Guest268 | 13:18 | |
| *** chem_ is now known as chem | 13:23 | |
| rcastillo | o/ | 13:25 |
| rlandy | rcastillo: hello | 13:26 |
| *** soniya|ruck|afk is now known as soniya|ruck | 13:43 | |
| rlandy | ysandeep|rover: soniya|ruck: added BZ to the rr hackmd | 13:43 |
| soniya|ruck | rlandy, ack | 13:43 |
| rlandy | pinged lon and jon on it | 13:43 |
| ysandeep|rover | rlandy++ | 13:43 |
| rlandy | ysandeep|rover: soniya|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/840826 merged | 13:43 |
| rlandy | rekicking wallby c8 line | 13:43 |
| ysandeep|rover | ack | 13:43 |
| rlandy | will deuque enqueue | 13:44 |
| rlandy | ok - let's see what that does | 13:45 |
| rlandy | ysandeep|rover: soniya|ruck: wil rekick victoria afterwards | 13:45 |
| rlandy | soniya|ruck: you testproject failed on rhos-17 on rhel-9 | 13:48 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/99/407599/3/check/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17/8885df0/logs/undercloud/home/zuul/standalone_deploy.log | 13:48 |
| soniya|ruck | rlandy, on it..checking them | 13:48 |
| rlandy | you want to see if that is a consistent failure | 13:49 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17&project=testproject&skip=0 | 13:49 |
| rlandy | not too bad | 13:49 |
| soniya|ruck | though seems unstable but | 13:50 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17/459bc65/logs/undercloud/home/zuul/standalone_deploy.log | 13:50 |
| rlandy | soniya|ruck: ^^ passed deploy - pls recheck again | 13:50 |
| rlandy | and watch to see if it is consistent | 13:50 |
| soniya|ruck | rlandy, yupp | 13:51 |
| rlandy | soniya|ruck: https://code.engineering.redhat.com/gerrit/c/testproject/+/407601 - a rekcik won't help | 13:51 |
| rlandy | please read the BZ logged | 13:51 |
| rlandy | rekick eth rhel-9 one not the rhel-8 one | 13:51 |
| soniya|ruck | rlandy, https://code.engineering.redhat.com/gerrit/c/testproject/+/407599 | 13:52 |
| soniya|ruck | rekicked rhel9 one | 13:52 |
| rlandy | yep - that one - thank you | 13:52 |
| *** pojadhav is now known as pojadhav|afk | 13:57 | |
| dasm|off | o/ | 13:59 |
| *** dasm|off is now known as dasm | 13:59 | |
| rlandy | oh great - now all the component jobs are red | 14:00 |
| ysandeep|rover | folks, happy friday call o/ | 14:01 |
| * ysandeep|rover starts the cleanup script, last reproducer env - not replicating issue anymore | 14:04 | |
| rlandy | ysandeep|rover: thanks - just talking with reldel | 14:05 |
| ysandeep|rover | rlandy, For Monday i have following plan if vexx doesn't solve the issue by then, validate-perf test (which only run on undercloud right now) - make it run on overcloud node as well so that we have some data to show to vexxhost. | 14:07 |
| rlandy | ysandeep|rover: ok - let's touch base before you are EoD | 14:07 |
| rlandy | are you out now? | 14:07 |
| ysandeep|rover | I am on happy friday call atm, but can sync in few mins before logging off | 14:08 |
| rlandy | ok | 14:10 |
| rlandy | pls ping | 14:10 |
| ysandeep|rover | rlandy: meet.google.com/fny-kpzc-seh | 14:14 |
| soniya|ruck | shall i join? | 14:14 |
| soniya|ruck | ysandeep|rover, rlandy ^^ | 14:16 |
| rlandy | soniya|ruck' hey - we just dropped | 14:34 |
| rlandy | soniya|ruck: ysandeep|rover wil carry on with vexx investigation | 14:35 |
| rlandy | soniya|ruck: can you keep an eye on the component lines? | 14:35 |
| rlandy | thank you | 14:35 |
| soniya|ruck | rlandy, ack | 14:35 |
| rlandy | I am checking into the c8 image build failure | 14:35 |
| soniya|ruck | rlandy, sure | 14:35 |
| rlandy | we have a real bug in c8 image builds | 14:41 |
| rcastillo | package libguestfs-1:1.44.0-6.module_el8.7.0+1140+ff0772f9.x86_64 requires libvirt-daemon-kvm >= 8.0.0, but none of the providers can be installed | 14:45 |
| rcastillo | rlandy: we're excluding libvirt* | 14:45 |
| rlandy | rcastillo: ack - in process of testprojecting putting that back | 14:47 |
| rlandy | we excluded it because the last build was bad | 14:47 |
| soniya|ruck | rlandy, standalone rhel8 full tempest scenario rhos-17 is failing on tempest test and is a consistent issue | 14:49 |
| soniya|ruck | i think we need to file a bug for it | 14:49 |
| rlandy | soniya|ruck: pls paste links and I'll check | 14:50 |
| soniya|ruck | http://pastebin.test.redhat.com/1050121 | 14:53 |
| rlandy | soniya|ruck: one sec - starting testproject - then will review | 14:54 |
| soniya|ruck | rlandy, okay | 14:55 |
| rlandy | rcastillo: trying https://review.rdoproject.org/r/c/testproject/+/36254 | 14:57 |
| rlandy | soniya|ruck: k- looking | 14:57 |
| *** ysandeep|rover is now known as ysandeep|out | 14:58 | |
| *** ysandeep|out is now known as ysandeep | 14:58 | |
| rlandy | soniya|ruck: looking through the tests ... let's sync on this ... | 15:04 |
| rlandy | https://meet.google.com/sgv-aafe-tuz?pli=1&authuser=0 | 15:04 |
| ysandeep | rlandy, fyi.. https://user-images.githubusercontent.com/435815/166965207-7279a94e-3361-498d-a206-3f89e7be954f.png load on vexx compute going as high as 500 - I am almost certain its because of that. | 15:12 |
| ysandeep | to me looks like host is stealing vm cpu cycles | 15:12 |
| *** ysandeep is now known as ysandeep|out | 15:18 | |
| rlandy | ysandeep|out: ack- thanks for collecting that | 15:19 |
| *** dviroel is now known as dviroel|lunch | 15:40 | |
| soniya|ruck | rlandy, https://bugzilla.redhat.com/show_bug.cgi?id=2082632 | 16:10 |
| rlandy | rcastillo: soniya|ruck: https://review.rdoproject.org/r/c/testproject/+/36254 - depends on should fix wallaby c8 image builds | 16:12 |
| rlandy | testing OVB jobs now with those images | 16:12 |
| rlandy | https://review.rdoproject.org/r/c/testproject/+/39960 | 16:13 |
| rlandy | let's see what that does | 16:13 |
| soniya|ruck | rlandy, ack | 16:14 |
| rlandy | checking your bug now | 16:15 |
| rcastillo | rlandy: nice | 16:15 |
| rcastillo | do we not need the cloud init downgrade after all? | 16:15 |
| rcastillo | or testing that later? | 16:15 |
| rlandy | soniya|ruck: editing to include rhel 8 rhos-17 | 16:16 |
| rlandy | rcastillo; no - sandeep merged a patch to upgarade the cloud image | 16:16 |
| rcastillo | rlandy: ack | 16:16 |
| rlandy | soniya|ruck: looks good - pls send an email to CIX that | 16:18 |
| soniya|ruck | rlandy, sure | 16:18 |
| soniya|ruck | rlandy, done | 16:24 |
| rlandy | soniya|ruck: k - great - next step would be to compare the versions of the rpms in the tempest component | 16:28 |
| rlandy | in component-ci-testing | 16:28 |
| rlandy | vs tripleo-ci-testing | 16:28 |
| rlandy | and see what is different | 16:28 |
| rlandy | may help you find the root cause | 16:28 |
| soniya|ruck | rlandy, okay..digging it | 16:29 |
| rlandy | soniya|ruck++ | 16:29 |
| *** jpena is now known as jpena|off | 16:29 | |
| soniya|ruck | rlandy, i see some failures in cinder component as well..will check after above investigation | 16:30 |
| rlandy | soniya|ruck: cinder component is probably what we logged earlier | 16:30 |
| rlandy | install depenedcy failure | 16:30 |
| rlandy | pinged lon about tat | 16:30 |
| rlandy | that | 16:30 |
| rlandy | alan commented in bug | 16:30 |
| soniya|ruck | rlandy, okay | 16:30 |
| soniya|ruck | rlandy, these are only issues with 17 rhel8 | 16:31 |
| soniya|ruck | apart from that all good | 16:32 |
| *** dviroel|lunch is now known as dviroel | 16:42 | |
| soniya|ruck | rlandy, can you pass the link for component-ci-testing repo? | 16:51 |
| soniya|ruck | i am not getting it | 16:52 |
| rlandy | https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17/component/tempest/component-ci-testing/ | 17:00 |
| rlandy | soniya|ruck: ^^ | 17:00 |
| soniya|ruck | rlandy, thanks | 17:06 |
| soniya|ruck | rlandy, 17 rhel9 component line seems good | 17:29 |
| soniya|ruck | rhel9 multinode-octavia-rhos-17 is failing because of ssh time_out issue but is inconsistent | 17:32 |
| soniya|ruck | rlandy, leaving for today :) | 17:39 |
| *** soniya|ruck is now known as soniya|out | 17:39 | |
| rlandy | thanks, soniya | 17:41 |
| * dasm stepping away for a bit. bbl | 18:07 | |
| dasm | back | 19:22 |
| *** dviroel is now known as dviroel|afk | 20:11 | |
| * dasm is calling it a week | 21:09 | |
| dasm | take care Team! | 21:09 |
| *** dasm is now known as dasm|off | 21:09 | |
| * rcastillo is also leaving | 21:33 | |
| rcastillo | I'll see you on monday o/ | 21:33 |
| *** dviroel|afk is now known as dviroel | 23:53 | |
| *** dviroel is now known as dviroel|out | 23:56 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
