| *** bhagyashris_ has joined #oooq | 00:31 | |
| *** bhagyashris has quit IRC | 00:38 | |
| *** ysandeep|away has joined #oooq | 01:19 | |
| *** ysandeep|away is now known as ysandeep | 01:19 | |
| *** ysandeep is now known as ysandeep|ruck | 01:19 | |
| ysandeep|ruck | rlandy|bbl, weshay|ruck how is it going? | 01:19 |
|---|---|---|
| ysandeep|ruck | rlandy|bbl, anything to pick up in downstream? looks like we have new tripleo-ansible | 02:29 |
| ysandeep|ruck | weshay|ruck: i am looking at https://review.rdoproject.org/r/c/rdo-jobs/+/33982 , looks like old image also didn't help | 02:36 |
| ysandeep|ruck | https://logserver.rdoproject.org/82/33982/1/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/6ef26c1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 02:36 |
| ysandeep|ruck | weshay|ruck: Even after pinning to old image, overcloud node got the latest Networkmanager, https://logserver.rdoproject.org/82/33982/1/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/6ef26c1/logs/overcloud-controller-0/var/log/extra/rpm-list.txt.gz | 02:45 |
| ysandeep|ruck | May be we are triggering "yum update" somewhere ? | 02:45 |
| weshay|ruck | ysandeep|ruck, 0/ | 02:49 |
| weshay|ruck | ysandeep|ruck, correct... the old image did not work... 1/3 passed | 02:49 |
| weshay|ruck | hrm | 02:50 |
| ysandeep|ruck | trying to find how old image got latest NM | 02:50 |
| weshay|ruck | ysandeep|ruck, so what is logged isn't neccessarily what is running when the nodes boot | 02:51 |
| weshay|ruck | ysandeep|ruck, we have to get passwd in on root.. and we need to hold an environment | 02:51 |
| weshay|ruck | ysandeep|ruck, I'd also like to keep working your patch to pin NetworkManager | 02:52 |
| weshay|ruck | ysandeep|ruck, perhaps we need to edit... yum.conf | 02:52 |
| ysandeep|ruck | weshay|ruck, just hold will work.. after reboot node becomes accessible.. so we can login and check | 02:52 |
| ysandeep|ruck | weshay|ruck, i just updated my test patch.. with depends-on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/794700 | 02:53 |
| * weshay|ruck looks | 02:53 | |
| weshay|ruck | ysandeep|ruck, I'm going to kill all the centos-7 train ovb jobs btw and promote train | 02:53 |
| ysandeep|ruck | weshay|ruck, pm | 02:53 |
| weshay|ruck | ysandeep|ruck++ versionlock ++ | 02:53 |
| *** rlandy|bbl is now known as rlandy | 03:14 | |
| *** rlandy has quit IRC | 03:14 | |
| ysandeep|ruck | weshay|ruck, fyi.. failed at later step because dib tries to update the package.. http://paste.openstack.org/show/806347/ . I will try version lock in dib instead.. | 03:35 |
| weshay|ruck | ++ | 03:36 |
| weshay|ruck | ysandeep|ruck, didn't work | 03:42 |
| weshay|ruck | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bb8/791033/20/gate/tripleo-ci-centos-8-scenario007-standalone/bb8377a/logs/undercloud/var/log/tempest/tempest_run.log | 03:42 |
| weshay|ruck | oops | 03:42 |
| weshay|ruck | requests.exceptions.SSLError: HTTPSConnectionPool(host='sf.hosted.upshift.rdu2.redhat.com', port=443): Max retries exceeded with url: /zuul/api/info (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))) | 03:42 |
| ysandeep|ruck | weshay|ruck, thanks for trying, i will post testpatch | 03:43 |
| ysandeep|ruck | i hope we are in promotable state for 16.2 by the time you wake up.. | 03:44 |
| *** ysandeep|ruck is now known as ysandeep|afk | 03:49 | |
| weshay|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/33990 | 03:54 |
| ysandeep|afk | ack for ^^ o/ | 04:00 |
| weshay|ruck | 0/ | 04:20 |
| *** ykarel|away has joined #oooq | 04:37 | |
| *** ysandeep|afk has quit IRC | 04:38 | |
| *** ysandeep|afk has joined #oooq | 04:51 | |
| *** ysandeep|afk is now known as ysandeep|ruck | 05:03 | |
| *** ykarel|away is now known as ykarel | 05:14 | |
| *** marios has joined #oooq | 05:18 | |
| *** slaweq_ has joined #oooq | 06:00 | |
| *** slaweq has left #oooq | 06:02 | |
| *** slaweq_ has quit IRC | 06:03 | |
| *** slaweq_ has joined #oooq | 06:03 | |
| *** jbadiapa_ is now known as jbadiapa | 06:09 | |
| *** jbadiapa has joined #oooq | 06:09 | |
| *** soniya29 has joined #oooq | 06:21 | |
| *** soniya29 has quit IRC | 06:22 | |
| *** soniya29 has joined #oooq | 06:25 | |
| *** soniya29 has joined #oooq | 06:26 | |
| *** bhagyashris_ is now known as bhagyashris | 06:33 | |
| *** amoralej|off has joined #oooq | 06:35 | |
| *** amoralej|off is now known as amoralej | 06:38 | |
| *** soniya29 has quit IRC | 06:50 | |
| *** soniya29 has joined #oooq | 06:55 | |
| *** tosky has joined #oooq | 07:23 | |
| *** slaweq has joined #oooq | 07:34 | |
| * ysandeep|ruck stepping away for lunch | 07:35 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:35 | |
| *** slaweq_ has quit IRC | 07:41 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** slaweq_ has joined #oooq | 08:05 | |
| *** slaweq_ has quit IRC | 08:11 | |
| *** soniya29 has quit IRC | 08:57 | |
| *** jpward has quit IRC | 09:11 | |
| *** ysandeep|lunch is now known as ysandeep | 09:16 | |
| *** ysandeep is now known as ysandeep|ruck | 09:16 | |
| zbr | fyi, i am working on fixing tox jobs on collect logs, broken due to nodeset change. | 09:47 |
| ysandeep|ruck | zbr++ thanks! | 09:55 |
| *** ChanServ sets mode: +o marios | 10:19 | |
| zbr | ysandeep|ruck: arxcruz|rover: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/794679/ already passed, that is blocking other patches. | 10:22 |
| * ysandeep|ruck looking | 10:26 | |
| *** soniya29 has joined #oooq | 10:28 | |
| amoralej | ysandeep|ruck, we are re-adding ovn-2021 to the nfvsig repos | 10:29 |
| amoralej | now, it shouldn't affect anything | 10:29 |
| amoralej | but as last time we broke some jobs | 10:30 |
| amoralej | let me know if you fins something abnormal | 10:30 |
| amoralej | it will take some time to be synced across all mirrors | 10:30 |
| ysandeep|ruck | amoralej: thanks for headsup, but i hope i don't have to reach you this time :) | 10:31 |
| amoralej | hopefully | 10:31 |
| *** amoralej is now known as amoralej|afk | 10:55 | |
| *** chem has quit IRC | 11:00 | |
| ysandeep|ruck | arxcruz|rover, commented on https://review.opendev.org/c/openstack/tripleo-ci/+/794750 | 11:05 |
| *** derekh has joined #oooq | 11:05 | |
| *** soniya29 has quit IRC | 11:22 | |
| frenzy_friday | Hey zbr, weshay|ruck When you get some time pls lemme know if this makes sense: https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/794744 | 11:27 |
| marios | zbr: when you next have some time can you have a quick look at https://bugs.launchpad.net/tripleo/+bug/1930416 see if something jumps at you (possibly related to https://bugs.launchpad.net/tripleo/+bug/1921679 which you fixed) thank you | 11:31 |
| opendevmeet | Launchpad bug 1930416 in tripleo "master undercloud upgrade fails "not an openstack command"" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 11:31 |
| opendevmeet | Launchpad bug 1921679 in tripleo "openstack: 'undercloud install' is not an openstack command. See 'openstack --help'." [Critical,Triaged] | 11:31 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| zbr | marios: to me it looks like that mentioned fix was not backported to stable/wallaby | 11:36 |
| zbr | marios: in fact i am wrong, it was fixed on wallaby. not sure what is causing the failure. | 11:40 |
| zbr | frenzy_friday: have you run tox locally, apparent it does alter one tracked file which you probably failed to include in review. am i wrong? | 11:42 |
| frenzy_friday | zbr, yes sorry, forgot to add queries-schema . Updated | 11:44 |
| zbr | on that repo, if you just run "tox" and passed it should also pass on zuul. | 11:44 |
| *** rlandy has joined #oooq | 11:49 | |
| marios | zbr: yeah it is on wallaby it merged on master before we branched i checked/commented on that bit | 11:52 |
| marios | zbr: k thanks for checking | 11:54 |
| *** cgoncalves has quit IRC | 11:59 | |
| *** cgoncalves has joined #oooq | 12:01 | |
| * dviroel taking my son to see the pediatrician, be back soon | 12:15 | |
| *** dviroel is now known as dviroel|away | 12:15 | |
| ysandeep|ruck | weshay|ruck, rlandy we need this patch in container build to pick new ovn - https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/245375 please review when you have time.. had to mark jobs non-voting because check jobs are failing currently. | 12:19 |
| *** jpena|lunch is now known as jpena | 12:20 | |
| rlandy | ysandeep|ruck: we need the new ovn for 17 as well, right? | 12:21 |
| ysandeep|ruck | rlandy, correct | 12:22 |
| rlandy | ysandeep|ruck: then merging that | 12:22 |
| rlandy | ysandeep|ruck: also I am cutting down the 16.2 line run frequency | 12:23 |
| ysandeep|ruck | rlandy, once in a day? | 12:23 |
| rlandy | if the jobs are red and nobody cares | 12:23 |
| rlandy | we can use the resources better for rhel 9 | 12:23 |
| ysandeep|ruck | rlandy, would be better if we change timing.. otherwise by the time you guys login.. hash changes | 12:24 |
| rlandy | ysandeep|ruck: Ill think about timing | 12:25 |
| ysandeep|ruck | may be keep in my morning hours.. so that i can rekick in case something fails.. and we can promote in your morning hours. | 12:25 |
| rlandy | I very much doubt the 16.2 hash will change often | 12:25 |
| rlandy | if it was, we would need the line | 12:26 |
| *** chem has joined #oooq | 12:26 | |
| *** chem is now known as Guest849 | 12:26 | |
| ysandeep|ruck | rlandy: i have a plan to not trigger 16.2 component if nothing changes.. will work on it next sprint | 12:26 |
| * rlandy moves on to 17 | 12:28 | |
| marios | please needs reviews when you have time: https://review.opendev.org/c/openstack/tripleo-repos/+/792126 & https://review.opendev.org/c/openstack/tripleo-ci/+/791908 | 12:30 |
| *** ysandeep|ruck is now known as ysandeep|mtg | 12:33 | |
| *** kapoios_allos has joined #oooq | 12:41 | |
| *** kapoios_allos has quit IRC | 12:42 | |
| *** bhagyashris_ has joined #oooq | 12:50 | |
| *** soniya29 has joined #oooq | 12:52 | |
| *** rlandy is now known as rlandy|training | 12:54 | |
| *** bhagyashris has quit IRC | 12:56 | |
| *** bhagyashris_ is now known as bhagyashris | 12:58 | |
| *** amoralej|afk is now known as amoralej | 13:00 | |
| ysandeep|mtg | weshay|ruck, https://bugs.launchpad.net/tripleo/+bug/1930849 | 13:05 |
| opendevmeet | Launchpad bug 1930849 in tripleo "OVB CI - Virtual baremetals boot fail to bring up network - Ordering cycle found, skipping Network Manager" [High,Triaged] | 13:05 |
| *** cgoncalves has quit IRC | 13:14 | |
| *** cgoncalves has joined #oooq | 13:15 | |
| weshay|ruck | arxcruz|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/00ea393/logs/overcloud-controller-0/var/log/extra/journal.txt | 13:19 |
| bhagyashris | weshay|ruck, hey you still in meeting? | 13:32 |
| bhagyashris | weshay|ruck, me and amol joined here https://meet.google.com/bjs-zvvr-ibi?authuser=0 kindly join when you will free | 13:32 |
| weshay|ruck | ya.. coming | 13:33 |
| weshay|ruck | bhagyashris, sec | 13:33 |
| bhagyashris | ok | 13:33 |
| bhagyashris | np | 13:34 |
| akahat | weshay|ruck, bhagyashris https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33913 | 13:38 |
| *** rlandy|training is now known as rlandy | 13:43 | |
| *** ysandeep|mtg is now known as ysandeep | 13:44 | |
| *** ysandeep is now known as ysandeep|ruck | 13:44 | |
| zbr | joys of having too many jobs... https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/794679 | 13:46 |
| zbr | tripleo-ci-centos-8-scenario001-standalone failed | 13:47 |
| ysandeep|ruck | weshay|ruck, code.eng is down.. i can't post baremetal job now.. will work on it on monday. | 13:48 |
| ysandeep|ruck | weshay|ruck, arxcruz|rover, issue reproduced with old NM too :( | 14:00 |
| weshay|ruck | grrrr | 14:08 |
| weshay|ruck | ysandeep|ruck, ok... baremetal | 14:08 |
| ysandeep|ruck | no code eng.. no baremetal for today :) | 14:09 |
| *** soniya29 has quit IRC | 14:14 | |
| weshay|ruck | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/0a9bee93077a4274ac654eb06b8126b6?logfile=console.log | 14:19 |
| weshay|ruck | that's that 16.2 baremetal integration job.. | 14:19 |
| weshay|ruck | :) | 14:19 |
| rlandy | "the last known survivor" | 14:20 |
| weshay|ruck | if we put a couple tempest skips in for scenario01/02.. 16.2 should be ok | 14:20 |
| rlandy | weshay|ruck: baremetal could survive a nuclear war | 14:20 |
| weshay|ruck | :) | 14:20 |
| weshay|ruck | rlandy, it's pretty good https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status | 14:20 |
| ysandeep|ruck | weshay|ruck: i haven't promoted 16.2 yet.. not sure if we should in between westford power outage.. | 14:21 |
| weshay|ruck | Queue: testproject | 14:21 |
| weshay|ruck | ya.. we'll check the infra.. but I think we'll promote 16.2 again today | 14:22 |
| rlandy | no OSP Trunk | 14:22 |
| rlandy | all new jobs are in error now | 14:23 |
| *** dviroel|away is now known as dviroel | 14:26 | |
| rlandy | weshay|ruck: guess no time like the present to get back to tripleo os next | 14:48 |
| bhagyashris | weshay|ruck, akahat ussuri promoted successfully on new promoter server https://trunk.rdoproject.org/centos8-ussuri/current-tripleo/delorean.repo.md5 :) | 15:04 |
| bhagyashris | weshay|ruck, akahat so this change https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33913 unblock us from promotion, we can safely merge this change | 15:06 |
| * bhagyashris Dinner brb | 15:10 | |
| ysandeep|ruck | weshay|ruck, arxcruz|rover so what we know - cloud-init and NM not directly related | 15:13 |
| ysandeep|ruck | we know it's systemd ordering, but don't know yet why it started happening now | 15:14 |
| ysandeep|ruck | we can focus on kernel update.. but may be before that we can run our ovb job in different cloud | 15:14 |
| ysandeep|ruck | weshay|ruck, when internal gerrit comes back.. can you run master ovb job in internal? | 15:15 |
| ysandeep|ruck | weshay|ruck, or psi-public cloud to rule out its not infra issue. | 15:16 |
| ysandeep|ruck | weshay|ruck, psi-public live for us? | 15:16 |
| ysandeep|ruck | i will get on baremetal on Monday, but if we can shoot ovb master job in different cloud.. that will help. | 15:18 |
| weshay|ruck | k.. sec | 15:18 |
| ysandeep|ruck | weshay|ruck, ykarel have posted recheck here https://review.rdoproject.org/r/c/testproject/+/30669 to see if psi is working. | 15:19 |
| weshay|ruck | ok.. back | 15:25 |
| weshay|ruck | reading | 15:25 |
| weshay|ruck | ysandeep|ruck, yes.. but I think the ovb jobs in psi have been down | 15:26 |
| weshay|ruck | but we can double check that before going the full bm route | 15:26 |
| *** ykarel is now known as ykarel|away | 15:27 | |
| ysandeep|ruck | weshay|ruck, lets see jobs running here https://review.rdoproject.org/r/c/testproject/+/30669 | 15:29 |
| weshay|ruck | probably going to fail | 15:30 |
| ysandeep|ruck | if it fail i am going to try adding "enable_config_drive: true" in cloud setting.. and see if that helps | 15:31 |
| ysandeep|ruck | things will be better when we will have 2 clouds to compare.. | 15:32 |
| *** ysandeep|ruck is now known as ysandeep|away | 15:49 | |
| bhagyashris | weshay|ruck, hey ^ | 15:55 |
| weshay|ruck | bhagyashris, I'm reviewing it now | 15:56 |
| bhagyashris | weshay|ruck, ok , it's working thanks | 15:56 |
| weshay|ruck | marios, seems like we could use some of these functions... to check the sanity of the hash.. in the tripleo-repos module | 15:57 |
| weshay|ruck | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33913/5/ci-scripts/dlrnapi_promoter/dlrn_hash.py | 15:57 |
| weshay|ruck | promoter creates a hash object.. though.. | 15:58 |
| marios | weshay|ruck: sure can check/discuss more next week | 16:01 |
| marios | weshay|ruck: could be added into the code sure | 16:02 |
| marios | weshay|ruck: to verify | 16:02 |
| ykarel|away | weshay|ruck, marios when u get chance https://review.opendev.org/c/openstack/tripleo-quickstart/+/794737 | 16:03 |
| marios | ykarel|away: sure adding to reviews list but monday | 16:04 |
| *** marios is now known as marios|out | 16:04 | |
| ykarel|away | rlandy, may be u can check^ | 16:04 |
| weshay|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/94/211494/41/check/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-16.2/a8295c9/logs/undercloud/var/log/tempest/stestr_results.html | 16:06 |
| weshay|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/94/211494/41/check/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-16.2/82d0690/logs/undercloud/var/log/tempest/stestr_results.html | 16:06 |
| weshay|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/94/211494/41/check/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/0a9bee9/logs/undercloud/var/log/tempest/stestr_results.html | 16:06 |
| weshay|ruck | I'm going to add these failures to tempest-skip | 16:06 |
| weshay|ruck | ykarel|away, rlandy once these skips are merged..afaict.. we can promote 16.2 based on http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=be961f86077a159733558b85764595ca | 16:06 |
| weshay|ruck | and things should return to normal | 16:07 |
| * rlandy looks | 16:07 | |
| rlandy | idk we were promoting 16.2 anymore | 16:10 |
| *** rlandy is now known as rlandy|brb | 16:10 | |
| weshay|ruck | rlandy, they may not import it.. who cares... for our own sake... to restore sanity | 16:11 |
| ysandeep|away | weshay|ruck, you don't need to add them in skip, i think just merge of my ovn patch will solve the issue | 16:15 |
| ysandeep|away | weshay|ruck, they are failing because of old ovn issues | 16:16 |
| ysandeep|away | weshay|ruck, psi public is not working.. | 16:16 |
| * ysandeep|away out | 16:16 | |
| *** ykarel|away has quit IRC | 16:18 | |
| *** marios|out has quit IRC | 16:18 | |
| *** eagles_ has joined #oooq | 16:22 | |
| weshay|ruck | arxcruz|rover, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/794861 | 16:23 |
| *** eagles has quit IRC | 16:28 | |
| *** ysandeep|away has quit IRC | 16:33 | |
| *** derekh has quit IRC | 16:34 | |
| *** jpena is now known as jpena|off | 16:38 | |
| *** rlandy|brb is now known as rlandy | 16:42 | |
| weshay|ruck | rlandy, one patch on top of this one https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/794864 | 16:51 |
| rlandy | looks fine | 16:53 |
| *** amoralej is now known as amoralej|off | 17:00 | |
| *** amoralej|off has quit IRC | 17:42 | |
| weshay|ruck | rlandy, ok. unblocked in collect logs https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/794679 | 17:43 |
| rlandy | cool | 17:43 |
| *** amoralej|off has joined #oooq | 18:56 | |
| weshay|ruck | 0. | 19:17 |
| weshay|ruck | 0/ | 19:17 |
| rlandy | o/ | 19:35 |
| *** amoralej|off has quit IRC | 19:45 | |
| *** tosky has quit IRC | 22:01 | |
| *** rlandy has quit IRC | 23:01 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!