| *** frenzy_friday is now known as frenzyfriday|ruck | 03:45 | |
| *** ysandeep|away is now known as ysandeep | 05:36 | |
| ysandeep | chkumar|rover, ++ thanks for zuul-jobs patch | 05:59 |
|---|---|---|
| ysandeep | chkumar|rover, Could you please re-review https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/302181 when time permits, I have replied back on your comment. | 05:59 |
| chkumar|rover | slaweq: ykarel: frenzyfriday|ruck https://review.opendev.org/c/openstack/neutron/+/825425 please have a look when free, thanks! | 06:07 |
| chkumar|rover | ysandeep: sure | 06:24 |
| ysandeep | chkumar|rover, sshnaidm marios I have added you for reviews on 17 rhel9 container build patches: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/300678 and https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/299521 as well. | 06:27 |
| marios | ysandeep: o/ k | 06:27 |
| *** ysandeep is now known as ysandeep|brb | 06:30 | |
| slaweq | chkumar|rover: hi, done | 06:31 |
| chkumar|rover | slaweq: thanks :-) | 06:31 |
| *** ysandeep|brb is now known as ysandeep | 07:14 | |
| ykarel | chkumar|rover, ack | 07:26 |
| ysandeep | marios: ++ thanks! | 07:40 |
| marios | np ysandeep :) | 07:54 |
| *** amoralej|off is now known as amoralej | 08:16 | |
| ysandeep | chkumar|rover, frenzyfriday|ruck content-provider start failing https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&project=openstack/tripleo-common | 08:29 |
| ysandeep | looks like mirror issue | 08:30 |
| ysandeep | Status code: 404 for https://mirror.bhs1.ovh.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/09255ba7c10e01afeb0d343667190f9c3e42d0a6099f887619abcb92ea0378db-filelists.xml.gz (IP: 158.69.73.218) | 08:30 |
| chkumar|rover | ouch, asked on #infra | 08:32 |
| chkumar|rover | ysandeep: thanks for reporting :-) | 08:32 |
| *** jpena|off is now known as jpena | 08:37 | |
| chkumar|rover | ysandeep: left some comments on internal reviews | 08:53 |
| ysandeep | chkumar|rover, thanks, Will update/reply back | 09:17 |
| marios | chkumar|rover: ysandeep: please vote @ https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/825220 (https://bugs.launchpad.net/tripleo/+bug/1957060/comments/12 ) | 09:48 |
| ysandeep | checking | 09:49 |
| ysandeep | for some reason, launchpad not opening for me, will retry in sometime | 09:51 |
| marios | ysandeep: ack thanks in your own time | 09:51 |
| *** ysandeep is now known as ysandeep|lunch | 09:57 | |
| arxcruz | marios is it okay to merge https://review.opendev.org/825220 ? | 10:45 |
| marios | arxcruz: yeah i had a good run see Reason for revert: https://bugs.launchpad.net/tripleo/+bug/1957060/comments/12 for link | 10:46 |
| marios | coffee brb | 10:46 |
| *** ysandeep|lunch is now known as ysandeep | 10:48 | |
| marios | thanks arxcruz | 10:57 |
| *** rlandy|out is now known as rlandy|ruck | 11:15 | |
| rlandy|ruck | ysandeep: meeting? | 11:17 |
| ysandeep | rlandy|ruck, joining | 11:17 |
| *** dviroel__ is now known as dviroel | 11:25 | |
| marios | https://bugs.launchpad.net/tripleo/+bug/1957060/comments/12 | 11:29 |
| marios | rlandy|ruck: ^^^ | 11:29 |
| marios | https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/825220 | 11:30 |
| marios | periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-master | 11:36 |
| marios | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/master.yaml | 11:37 |
| rlandy|ruck | dviroel: good morning | 11:40 |
| rlandy|ruck | dviroel: pinged yesterday about a patch alex had to update ceph repos | 11:40 |
| rlandy|ruck | can you review pls | 11:40 |
| rlandy|ruck | frenzy_friday: hello | 11:40 |
| rlandy|ruck | frenzy_friday: let's sync meet.google.com/vzm-nrah-qqf | 11:40 |
| frenzy_friday | rlandy|ruck, hey, joining | 11:41 |
| *** frenzy_friday is now known as frenzyfriday|ruck | 11:42 | |
| dviroel | rlandy|ruck: missed that ping I think, let me check | 11:43 |
| rlandy|ruck | dviroel: thanks - check #tripleo | 11:43 |
| rlandy|ruck | arxcruz: hey can you join https://meet.google.com/vzm-nrah-qqf?authuser=0 | 11:45 |
| rlandy|ruck | 10.0.109.56 | 11:48 |
| arxcruz | rlandy|ruck https://github.com/arxcruz.keys | 11:49 |
| *** bhagyashris__ is now known as bhagyashris | 11:49 | |
| ysandeep | rlandy|ruck, fyi.. wallaby tripleo didn't promote last night because of failing periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby , I will trigger the line. | 11:54 |
| rlandy|ruck | ysandeep: does that job usually pass? | 11:55 |
| ysandeep | rlandy|ruck, yes https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby | 11:55 |
| rlandy|ruck | ok | 11:55 |
| ysandeep | retriggering via testproject | 11:55 |
| rlandy|ruck | ok | 11:56 |
| rlandy|ruck | thanks | 11:56 |
| *** ysandeep is now known as ysandeep|brb | 12:06 | |
| rlandy|ruck | gthiemonge: hello ... for tracking frenzyfriday|ruck created https://bugs.launchpad.net/tripleo/+bug/1958384 for the c9 nested-virt node | 12:11 |
| rlandy|ruck | so we can unskip those tests when available | 12:11 |
| rlandy|ruck | thanks for looking into this | 12:11 |
| frenzyfriday|ruck | rlandy|ruck, , arxcruz : last time I checked these didnt run: http://pastebin.test.redhat.com/1020277 | 12:16 |
| marios | food brb | 12:27 |
| rlandy|ruck | bhagyashris: hello ... we merged a bunch of tests in wallaby c9 | 12:28 |
| rlandy|ruck | dasm did some great work | 12:28 |
| bhagyashris | rlandy|ruck, yeah i saw that | 12:29 |
| bhagyashris | now i am working on adding standalone wallaby component jobs | 12:29 |
| bhagyashris | rlandy|ruck, will add the sc001 2, 3.....in the follow up patches soon | 12:30 |
| rlandy|ruck | bhagyashris: are you tempest jobs passing? | 12:30 |
| rlandy|ruck | api and scenario? | 12:31 |
| rlandy|ruck | ipa? | 12:31 |
| bhagyashris | rlandy|ruck, yes | 12:31 |
| rlandy|ruck | bhagyashris: so let's review in scrum and merge up | 12:31 |
| bhagyashris | let me pass the link | 12:31 |
| bhagyashris | rlandy|ruck, sure | 12:31 |
| bhagyashris | rlandy|ruck, i have added the patch links those are ready to merge in the review list hackmd | 12:32 |
| * rlandy|ruck looks | 12:32 | |
| rcastillo | good morning #oooq o/ | 12:39 |
| dviroel | rcastillo: morning | 12:45 |
| dviroel | marios: o/ quick question - https://review.opendev.org/c/openstack/tripleo-ci/+/820584/4/roles/tripleo-build-jobs-repos/tasks/main.yaml#38 | 12:47 |
| dviroel | marios: Need to install it as root, in order to configure repos (/etc/yum.repos.d access) - so you see any other solution that installing 2 times? | 12:47 |
| dviroel | marios: cause the task need 'become: true' | 12:48 |
| marios | dviroel: o/ looking | 12:54 |
| marios | dviroel: not following though you pointed to the "copy to zuul root code" | 12:56 |
| marios | dviroel: did you mean to point at the install somewhere else? | 12:56 |
| marios | dviroel: lets talk in 2 mins on scrum then after if needed? | 12:56 |
| marios | dviroel: oh... line 38... k lets talk in a minute | 12:57 |
| dviroel | marios: ok | 12:59 |
| bhagyashris | arxcruz, chkumar|rover frenzyfriday|ruck ysandeep|brb | 13:02 |
| arxcruz | bhagyashris me and frenzyfriday|ruck will be al ittle bit late | 13:03 |
| bhagyashris | ack | 13:03 |
| *** amoralej is now known as amoralej|lunch | 13:04 | |
| rlandy|ruck | arxcruz: frenzyfriday|ruck: hey - can you join us in scrum | 13:04 |
| rlandy|ruck | I know you're working on the bug :) | 13:05 |
| rlandy|ruck | sorry | 13:05 |
| frenzyfriday|ruck | rlandy|ruck, 10 mins, creating the bug | 13:05 |
| frenzyfriday|ruck | rlandy|ruck, is that ok> | 13:05 |
| frenzyfriday|ruck | ? | 13:05 |
| rlandy|ruck | yep | 13:06 |
| frenzyfriday|ruck | coo, thanks | 13:06 |
| *** dasm|off is now known as dasm | 13:09 | |
| dasm | o/ | 13:09 |
| bhagyashris | review list | 13:17 |
| bhagyashris | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38262 | 13:17 |
| frenzyfriday|ruck | https://review.rdoproject.org/r/c/config/+/38090 | 13:18 |
| dasm | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38335 | 13:19 |
| rlandy|ruck | bhagyashris: rcastillo: https://review.opendev.org/c/openstack/tripleo-ansible/+/824961 | 13:40 |
| frenzyfriday|ruck | https://review.opendev.org/c/opendev/system-config/+/825446 | 13:42 |
| *** ysandeep|brb is now known as ysandeep | 13:47 | |
| ysandeep | bhagyashris, rlandy|ruck sry I couldn't join today's scrum, I was not feeling well. | 13:49 |
| rlandy|ruck | marios: dviroel: can one of you guy be the other +2/ w on https://review.rdoproject.org/r/c/rdo-jobs/+/38208 | 13:49 |
| bhagyashris | ysandeep, ok np. Take care. | 13:49 |
| dviroel | rlandy|ruck: sure | 13:49 |
| rlandy|ruck | then we can merge criteria | 13:50 |
| rlandy|ruck | ysandeep: we suspected you were not well - hope you feel better | 13:50 |
| rlandy|ruck | ysandeep: anything you want me t pick up from downstream? | 13:50 |
| rlandy|ruck | or rhel 9? | 13:50 |
| ysandeep | rlandy|ruck, thanks, just eye on tripleo wallaby component run - https://review.rdoproject.org/r/c/testproject/+/38348 | 13:51 |
| rlandy|ruck | ok | 13:51 |
| marios | rlandy|ruck: sure will check in a minute finishing something here | 13:52 |
| rlandy|ruck | marios: sure when you are ready | 13:52 |
| rlandy|ruck | marios: dviroel: also not sure what we want to do with https://review.rdoproject.org/r/c/rdo-jobs/+/38338 | 13:53 |
| rlandy|ruck | dasm: bhagyashris: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1-cs9 - failures look due to chrony | 13:56 |
| rlandy|ruck | may be a real issue | 13:56 |
| rlandy|ruck | or mirror story | 13:56 |
| *** amoralej|lunch is now known as amoralej | 14:00 | |
| dviroel | rlandy|ruck: i can update https://review.rdoproject.org/r/c/rdo-jobs/+/38338 - to fix the current issue | 14:01 |
| bhagyashris | rlandy|ruck, ack | 14:02 |
| rlandy|ruck | dviroel: sure | 14:02 |
| dviroel | ok, will do | 14:02 |
| bhagyashris | rlandy|ruck, dasm fyi i have rebased my patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38344 on dasm's patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38335 | 14:02 |
| rlandy|ruck | bhagyashris: great | 14:05 |
| frenzyfriday|ruck | hey rcastillo, dasm, jm1 : Could you pl send me your pub keys. I'll add to https://review.rdoproject.org/r/c/config/+/38090/1/playbooks/tripleo-rdo-base/pre.yaml - this adds the keys to zuul nodes - useful when you want to ssh after holding the nodes | 14:05 |
| rlandy|ruck | bhagyashris: dasm: I think the line has a real issue | 14:05 |
| rlandy|ruck | the chrony problem | 14:05 |
| dasm | rlandy|ruck: which line? | 14:05 |
| rlandy|ruck | wallaby-c9 | 14:06 |
| rlandy|ruck | so we should get the criteria in | 14:06 |
| dasm | ok, i see the link. checking | 14:06 |
| rlandy|ruck | let me check those patches | 14:06 |
| dasm | > false, "msg": "Unable to start service chrony-online.service: Job for chrony-online.service failed because the control process exited with error code.\nSee \"systemctl status chrony-online.service\" and \"journalctl -xeu chrony-online.service\" for details | 14:07 |
| dasm | hmm | 14:07 |
| rcastillo | frenzyfriday|ruck: here's mine https://review.rdoproject.org/paste/show/251/ | 14:08 |
| rlandy|ruck | bhagyashris: wrt https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38344/2/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-9/wallaby.yaml - not sure image builds are working yet w/o depends | 14:08 |
| rlandy|ruck | rcastillo: ^^ pls confirm | 14:08 |
| dasm | frenzyfriday|ruck: https://review.rdoproject.org/paste/show/252/ | 14:09 |
| rlandy|ruck | https://review.opendev.org/c/openstack/ansible-role-chrony/+/825383 | 14:09 |
| rcastillo | rlandy|ruck: bhagyashris correct, needs the depends on from https://review.rdoproject.org/r/c/testproject/+/37713 | 14:10 |
| rcastillo | some are already merged | 14:10 |
| rlandy|ruck | bhagyashris: let's hold on that | 14:10 |
| rcastillo | just missing the one ipa patch actually | 14:10 |
| frenzyfriday|ruck | rcastillo, dasm thanks | 14:10 |
| bhagyashris | rlandy|ruck, ack | 14:11 |
| bhagyashris | pushed the image build patches because i saw the definition got merged . will hold on the patches by doing the -W | 14:12 |
| bhagyashris | rlandy|ruck, -W on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38344 https://review.rdoproject.org/r/c/rdo-jobs/+/38342 | 14:15 |
| bhagyashris | dasm, rcastillo ^ | 14:15 |
| * bhagyashris cooking and dinner brb | 14:18 | |
| dviroel | marios: rlandy|ruck - hum, by removing '-branch' we end with this issue https://review.rdoproject.org/r/c/rdo-jobs/+/38338/4#message-0582b2dd224124e0a40e846606ce5da41275903f, in rdo-config repo. chicken and egg situation | 14:22 |
| rlandy|ruck | I know | 14:22 |
| rlandy|ruck | we may need to dup the job definitions | 14:22 |
| dviroel | rlandy|ruck: hum, yeah, should work | 14:24 |
| marios | rlandy|ruck: dviroel: i guess we would have to remove them from the layout first... in tripleo.yaml (config repo) ... checking | 14:24 |
| dviroel | marios: works too. by dupplicating, we merge duplicate definitions in rdo-jobs, fix config repo and delete -branch ones from rdo-jobs | 14:26 |
| marios | dviroel: the problem is there i think https://github.com/rdo-infra/review.rdoproject.org-config/blob/f92fb27ed76c033993fa3d10a24a3b7414f059e2/zuul.d/tripleo.yaml#L39 | 14:26 |
| marios | at least this is what zuul is complaining about | 14:26 |
| marios | dviroel: rlandy|ruck: so you'd need to update that to use the new one without -branch... but you'd have to first remove those ^^^ , merge the new job changes in https://review.rdoproject.org/r/c/rdo-jobs/+/38338 then add them back with new name in tripleo.yaml | 14:27 |
| marios | dviroel: k i see what you mean by duplicating now sure whatever is easier to work with | 14:27 |
| rlandy|ruck | yeah marios suggestion is better | 14:28 |
| rlandy|ruck | but whatever works easiest is fine | 14:28 |
| marios | rlandy|ruck: right whatever is easier dviroel | 14:28 |
| * dviroel thinks | 14:29 | |
| dviroel | rlandy|ruck: marios: will proceed with the dup method, because we can continue the work in chandan's config patch (https://review.rdoproject.org/r/c/config/+/38298) without too much changes. After that, delete the duplicates. | 14:33 |
| rlandy|ruck | sure - thanks | 14:33 |
| marios | dviroel: ack | 14:34 |
| frenzyfriday|ruck | rlandy|ruck, /me will be eod in ~30 mins. Anything that I can check? | 14:40 |
| rlandy|ruck | frenzyfriday|ruck: think we should be good - if you testprojected victoria again | 14:41 |
| rlandy|ruck | gates are rolling again | 14:41 |
| frenzyfriday|ruck | rlandy|ruck, yeah, testprojecting victoria and c7 | 14:41 |
| rlandy|ruck | frenzyfriday|ruck++ thank you | 14:42 |
| rlandy|ruck | dviroel: one more for your new merge powers: https://review.rdoproject.org/r/c/config/+/38090 | 14:44 |
| rlandy|ruck | jm1: ^^ when you get a chance, we need your keys there as well | 14:45 |
| jm1 | gimme a sec | 14:45 |
| jm1 | frenzyfriday|ruck: https://github.com/jm1.keys | 14:46 |
| rlandy|ruck | jm1: hey - let's chat for a few https://meet.google.com/sqz-srtx-hhm | 14:54 |
| jm1 | ok | 14:55 |
| rlandy|ruck | dasm: do you want to rekick the wallaby c9 line | 15:11 |
| rlandy|ruck | to see if we are cleared now | 15:11 |
| frenzyfriday|ruck | Hey rlandy|ruck, updated victoria and c7 train testproj in tripleo-ci. jm1: Added your keys https://review.rdoproject.org/r/c/config/+/38359 - pl review when you get some time | 15:14 |
| dasm | yeah. how can i do that rlandy|ruck ? "recheck"? | 15:14 |
| rlandy|ruck | dasm: sending you link | 15:15 |
| dasm | k | 15:15 |
| frenzyfriday|ruck | Hey bhagyashris, rcastillo I added the current important patches in tripleo-ci chat. Happy ruck rovering \0/ | 15:16 |
| * dviroel lunch + manila community meeting | 15:16 | |
| * frenzyfriday|ruck going afk | 15:16 | |
| *** frenzyfriday|ruck is now known as frenzyfriday | 15:16 | |
| rcastillo | thanks frenzyfriday|ruck, have a good one | 15:17 |
| bhagyashris | frenzyfriday, ack | 15:17 |
| *** bhagyashris is now known as bhagyashris|ruck | 15:17 | |
| *** rcastillo is now known as rcastillo|rover | 15:18 | |
| ysandeep | rlandy|ruck, wallaby upgrade job failed in testproject as well - https://logserver.rdoproject.org/48/38348/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby/5f5a606/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz | 15:22 |
| ysandeep | bhagyashris|ruck, rcastillo|rover ^^ | 15:23 |
| ysandeep | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby | 15:23 |
| rlandy|ruck | k - on meeting | 15:24 |
| rlandy|ruck | I think we made that change with dviroel a while back | 15:24 |
| rlandy|ruck | maybe we need to look at it | 15:24 |
| rlandy|ruck | to fix ussuri | 15:25 |
| ysandeep | ack | 15:25 |
| * ysandeep out for the day | 15:25 | |
| *** ysandeep is now known as ysandeep|away | 15:25 | |
| dasm | rlandy|ruck: i believe i managed to restart Q | 15:42 |
| rlandy|ruck | dasm: you did indeed | 15:43 |
| rlandy|ruck | you have the power | 15:43 |
| rlandy|ruck | pls use responsibly | 15:43 |
| dasm | [mandatory spider-man reference here] | 15:43 |
| *** marios is now known as marios|out | 16:03 | |
| ade_lee_ | rlandy|ruck, dviroel dasm -- fultonj and I would like to try and diagnose the ceph failures on the fips job | 16:09 |
| ade_lee_ | can we run that job on rdocloud and hold the node so we can get in and investigate | 16:09 |
| ade_lee_ | ? | 16:09 |
| fultonj | https://github.com/fultonj.keys | 16:10 |
| * dasm is not sure. waiting for others to respond | 16:10 | |
| ade_lee_ | we're talking about https://review.opendev.org/c/openstack/tripleo-ci/+/808215 | 16:11 |
| ade_lee_ | fultonj, which scenario was it again? scenario 2? | 16:12 |
| dviroel | ade_lee_: sure, go for it, start a testproject, so we can hold and add your keys | 16:12 |
| fultonj | 001 | 16:12 |
| ade_lee_ | dviroel, ack -- I always have to re-figure out how to create the relevant test project - so I may need some guidance shortly | 16:14 |
| ade_lee_ | fultonj, the last scenario 1 passed , but I think maybe we're using swift there? | 16:14 |
| fultonj | ade_lee_: right | 16:14 |
| fultonj | so switch ceph env files | 16:14 |
| fultonj | one sec | 16:14 |
| fultonj | replace this | 16:15 |
| fultonj | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/cephadm/cephadm-rbd-only.yaml | 16:15 |
| fultonj | with this | 16:15 |
| fultonj | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/cephadm/cephadm.yaml | 16:15 |
| fultonj | so RGW is dployed and not swift | 16:15 |
| ade_lee_ | fultonj, looking to see where we modified that .. | 16:18 |
| ade_lee_ | fultonj, I think I remember modifying it back -- so scenario 1 can't be right .. | 16:19 |
| fultonj | 004 also deploys rgw | 16:19 |
| ade_lee_ | yeah - thats right - it was 004 | 16:20 |
| ade_lee_ | fultonj, 001 deploys ceph but doesn't test it iirc | 16:20 |
| fultonj | 001 tests ceph | 16:20 |
| ade_lee_ | or at least doesn't run the tests that cause it to segfualt | 16:20 |
| fultonj | could be | 16:21 |
| fultonj | 001 it tests ceph block services | 16:21 |
| fultonj | rgw is ceph object services | 16:21 |
| fultonj | anyway... | 16:21 |
| ade_lee_ | but I do remember 004 being the thing that fails -- can you confirm that https://zuul.opendev.org/t/openstack/build/90fb078a9eb94b36967a45b7585391a9 runs ceph rgw? | 16:22 |
| fultonj | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_90f/808215/68/check/tripleo-ci-centos-8-scenario004-standalone/90fb078/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-97h4xjif/cephadm/cephadm_command.log | 16:23 |
| fultonj | rgw is deployed as per ^ | 16:24 |
| ade_lee_ | fultonj, ok - creating testproject -- give me a few minutes to get it right .. | 16:25 |
| fultonj | thx | 16:30 |
| rlandy|ruck | ade_lee_: sorry - revolving meetings but I see dviroel has you sorted | 16:30 |
| ade_lee_ | dviroel, does this look right? https://review.rdoproject.org/r/c/testproject/+/38368 | 16:35 |
| ade_lee_ | rlandy|ruck, ^^ | 16:38 |
| dviroel | ade_lee_: yeah, looks correct to me | 16:43 |
| ade_lee_ | dviroel, ok cool - we need the node held for that then, please | 16:44 |
| ade_lee_ | and fultonj keys added -- https://github.com/fultonj.keys | 16:44 |
| fultonj | thanks | 16:44 |
| dviroel | ade_lee_: will do in a bit | 16:45 |
| ade_lee_ | dviroel, thanks | 16:45 |
| fultonj | ade_lee_: can we add your keys too? | 16:45 |
| fultonj | i want to make sure if i re-run that the fips parts are correct | 16:45 |
| ade_lee_ | fultonj, yes - when you log in, you can add my keys too | 17:12 |
| ade_lee_ | dviroel, although - it seems the job failed .. | 17:12 |
| dviroel | ade_lee_: looking | 17:13 |
| ade_lee_ | dviroel, is fips enabled on this content-provider job - coz its not supposed to be .. | 17:15 |
| dviroel | Status code: 404 for http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/678056e5b64153ca221196673208730234dd72f03397a3ab2d30fea01392bd87-primary.xml.gz (IP: 192.240.166.165) | 17:16 |
| ade_lee_ | dviroel, that looks like something I saw elsewhere today too .. on tripleo-validations jobs.. | 17:17 |
| ade_lee_ | dviroel, is mirror down? or needs to be changed? | 17:17 |
| dviroel | ade_lee_: it is skipping enable_fips tasks, so it is ok | 17:18 |
| dviroel | ade_lee_: no, mirror is ok\ | 17:19 |
| ade_lee_ | dviroel, yeah . I didn't see fips in kernel command line so .. | 17:19 |
| dviroel | http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/ | 17:19 |
| ade_lee_ | dviroel, outdated repo info ? mirror is ok, but files are different than what was attempted | 17:21 |
| ade_lee_ | dviroel, recheck? | 17:21 |
| dviroel | ade_lee_: yeah, rlandy|ruck do you know why this happens? | 17:22 |
| dviroel | ade_lee_: sure, lets try again | 17:22 |
| ade_lee_ | once more into the breach .. | 17:22 |
| * rlandy|ruck reads back | 17:22 | |
| rlandy|ruck | mirror were down earlier today | 17:23 |
| rlandy|ruck | infra re-freshed them | 17:23 |
| dviroel | ok, might be related | 17:24 |
| rlandy|ruck | https://logserver.rdoproject.org/68/38368/1/check/tripleo-ci-centos-8-content-provider/843c476/logs/undercloud/home/zuul/DLRN/data/repos/component/tempest/0b/7f/0b7f92d29ce034105ce1c0946176bca1046c2e48_dev/rpmbuild.log.txt.gz | 17:25 |
| rlandy|ruck | error there | 17:26 |
| dviroel | yeah | 17:26 |
| rlandy|ruck | 022-01-20 16:50:26,886 ERROR:dlrn:Received exception Error in build_rpm_wrapper for openstack-tempest: | 17:26 |
| rlandy|ruck | error building tempest change | 17:26 |
| rlandy|ruck | https://logserver.rdoproject.org/68/38368/1/check/tripleo-ci-centos-8-content-provider/843c476/logs/undercloud/home/zuul/dlrn.log.txt.gz | 17:27 |
| dviroel | Cannot download, all mirrors were already tried without success | 17:28 |
| dviroel | DEBUG: Child return code was: 1 | 17:28 |
| rlandy|ruck | let's watch this run | 17:28 |
| ade_lee_ | ack | 17:29 |
| dviroel | rlandy|ruck: question about node holding, since --list doesn't give too much info | 17:29 |
| dviroel | rlandy|ruck: the node hold is associated with a change + job name | 17:29 |
| dviroel | rlandy|ruck: it will hold any node that matches this two? | 17:29 |
| rlandy|ruck | yeah | 17:30 |
| rlandy|ruck | sending example | 17:30 |
| dviroel | so, i don't need to delete + hold again, since the job failed/didn't start | 17:31 |
| * dviroel brb | 17:31 | |
| rlandy|ruck | sent | 17:32 |
| rlandy|ruck | checking hold status | 17:32 |
| rlandy|ruck | do you want to hold the content-provider or scenario004? | 17:34 |
| ade_lee_ | scenario 4 | 17:34 |
| rlandy|ruck | ok - sent dviroel command | 17:36 |
| rlandy|ruck | there is an autohold | 17:36 |
| rlandy|ruck | but I don;t think it works until the job starts | 17:36 |
| *** jpena is now known as jpena|off | 17:37 | |
| rlandy|ruck | content-provider is running now | 17:38 |
| rlandy|ruck | few minutes for lunch | 17:39 |
| rlandy|ruck | brb | 17:39 |
| * dviroel back | 17:40 | |
| dviroel | yeah, make sense to me only work after job starts, so it can get the actual node_id | 17:41 |
| * dviroel remember holding node on old nodepool instances, but specifying the "node_id" | 17:42 | |
| * dviroel zuulv2 :P | 17:42 | |
| rlandy|ruck | dviroel: so you can remove current hold and resubmit when job starts and node is assigned | 17:45 |
| dviroel | rlandy|ruck: yeah, already removed the current hold, now waiting | 17:45 |
| dviroel | rlandy|ruck: using ysandeep tool | 17:46 |
| dviroel | ade_lee_: content-provider is already ahead from previous failure | 17:52 |
| * fultonj standing by | 18:18 | |
| dasm | rlandy|ruck: https://trunk.rdoproject.org/centos9-wallaby/component/tripleo/0d/d2/0dd2ea062d732f2c5715eb1144d4b6cd49f96b97_fb1adbb4/ | 18:35 |
| *** amoralej is now known as amoralej|off | 18:35 | |
| rcastillo|rover | are the hashes for centos 7 different in some way? the rr script gives a len==40 hash, which fails the validation in repo-setup's get_hash | 18:39 |
| rcastillo|rover | I've tried switching to a hash of the format 0b31308815c2e05a6fec5a56a6522bc70083c7d6_55400e15 but I'm getting a different error | 18:40 |
| rcastillo|rover | not sure if I'm doing something wrong or if I've hit a bug | 18:40 |
| dviroel | rcastillo|rover: not sure, we may need marios help on that, can be a setup-repo get-hash issue, not sure | 19:01 |
| dviroel | rcastillo|rover: https://trunk.rdoproject.org/centos7-train/tripleo-ci-testing/commit.yaml | 19:03 |
| dviroel | rcastillo|rover: for now, you can just remove the hash from your testproject, and it will test against the same hash ^ | 19:04 |
| rcastillo|rover | oh, ok, easy enough | 19:04 |
| dviroel | rcastillo|rover: but we should investigate that | 19:04 |
| rcastillo|rover | yeah, I'll look into it a bit | 19:04 |
| dviroel | rlandy|ruck: can't ssh into standalone-scenario004 instance, I think that this one does't run the task to add our keys - will ask someone to add | 19:06 |
| ade_lee_ | a | 19:09 |
| rlandy|ruck | dviroel: was on meeting | 19:16 |
| rlandy|ruck | looking | 19:16 |
| rlandy|ruck | dviroel: ack - you are correct | 19:16 |
| rlandy|ruck | upstream defined job do not add out keys | 19:17 |
| dviroel | fultonj: ssh zuul@38.102.83.44 | 19:36 |
| fultonj | dviroel: zuul@38.102.83.44: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). | 19:40 |
| fultonj | ssh -v ----> https://paste.opendev.org/show/812268/ | 19:40 |
| dviroel | fultonj: let me check | 19:45 |
| dviroel | https://github.com/fultonj.keys - both are there | 19:45 |
| fultonj | journalctl -u sshd | 19:48 |
| ade_lee_ | fultonj, dviroel the ed25519 key will fail , because fips .. | 19:51 |
| fultonj | the other key is rsa | 19:51 |
| ade_lee_ | fultonj, the rsa one should succeed -- is it getting that far? | 19:52 |
| ade_lee_ | dviroel, fultonj maybe remove the ed25519 key , just to be sure .. | 19:52 |
| dviroel | removed from server side | 19:53 |
| * fultonj tries the RSA key | 19:53 | |
| fultonj | rsa works | 19:53 |
| fultonj | sorry i should have know | 19:54 |
| ade_lee_ | fultonj, well at least we know fips is on :) | 19:54 |
| dviroel | :) | 19:54 |
| fultonj | :) | 19:54 |
| ade_lee_ | dviroel, fultonj - please add my key too -- https://github.com/vakwetu.keys | 19:54 |
| * fultonj installed tmux | 19:55 | |
| dviroel | ade_lee_: try now | 19:55 |
| fultonj | ade_lee_: try a 'tmux a' when you get in | 19:55 |
| ade_lee_ | dviroel, working - thanks | 19:55 |
| dviroel | ++ | 19:55 |
| fultonj | thank you dviroel | 19:55 |
| dviroel | yw | 19:55 |
| ade_lee_ | fultonj, tmux exited -- but gotta go get kids from school - back shortly | 19:57 |
| fultonj | ack , ttyl | 19:57 |
| * dviroel wondering what vakwetu means | 20:02 | |
| * dviroel a style? | 20:03 | |
| rlandy|ruck | dviroel: lol - you need to be South African to appreciate ade_lee_'s github name choice | 20:16 |
| dviroel | yeah, google isn't helping | 20:17 |
| rlandy|ruck | I'm not sure google is very good with african languages | 20:17 |
| ade_lee_ | dviroel, vakwetu means 'friend' in rukwangali , which was the local language near Rundu Namibia, where I worked for awhile as a peace corps volunteer | 21:29 |
| dviroel | ade_lee_: oh, nice :) - thanks for letting me know | 21:30 |
| rlandy|ruck | rcastillo|rover: I stopped ananya's testproject for c7 | 21:46 |
| rlandy|ruck | the neutron patch has not yet merged | 21:46 |
| rlandy|ruck | so didn't want to promote a broken build and break check/gate | 21:46 |
| ade_lee_ | rlandy|ruck, dviroel any idea how I would track when a fix got merged into centos-9-stream? | 21:55 |
| dasm | rlandy|ruck: fyi. forced run of "openstack-periodic-integration-stable1-cs9" is wrapping up. all succeeded. still waiting form multinode-wallaby results (but it seems to be passing) | 21:56 |
| ade_lee_ | rlandy|ruck, dviroel specifically, I'd like to retry fips as soon as the fix to https://bugzilla.redhat.com/show_bug.cgi?id=1942527 shows up | 21:57 |
| fultonj | that's very interesting ade_lee_ | 21:58 |
| ade_lee_ | which according to the comments is supposed to happen soon -- how often are builds done and made available to us? | 21:58 |
| ade_lee_ | fultonj, yeah - centos 9 + python + fips broken pending this fix | 21:59 |
| fultonj | i meant that you were a peace corps volunteer in Rundu Namibia | 22:00 |
| ade_lee_ | fultonj, oh yeah :) | 22:00 |
| rcastillo|rover | rlandy|ruck: ack | 22:00 |
| rlandy|ruck | dasm: nice | 22:01 |
| ade_lee_ | seems like a lifetime ago | 22:01 |
| rlandy|ruck | ade_lee_: yep ... check dlrn hash | 22:01 |
| rlandy|ruck | what fix are you looking for | 22:01 |
| fultonj | ade_lee_: i downloaded the downstream container and i'm looking to start the upgrade to it, but there's an issue i need to workaround. | 22:01 |
| fultonj | ade_lee_: i'm going to resume tomorrow though as it's now family time | 22:02 |
| fultonj | hopefully we can get some results tomorrow so we can return the machine to the pool | 22:02 |
| ade_lee_ | rlandy|ruck, https://bugzilla.redhat.com/show_bug.cgi?id=1942527 -- which has fix here -- https://github.com/stratakis/cpython/commits/fips_3.9.10_rhel9 | 22:02 |
| * fultonj going AFK | 22:02 | |
| ade_lee_ | fultonj, ack - sounds good | 22:02 |
| rlandy|ruck | ah- ok what component does that go into? | 22:03 |
| ade_lee_ | python? | 22:03 |
| * rlandy|ruck gets composes | 22:03 | |
| dviroel | ade_lee_: rlandy|ruck: as soon it gets to latest-compose we can test it | 22:03 |
| rlandy|ruck | ack | 22:03 |
| rlandy|ruck | checking compose | 22:04 |
| rlandy|ruck | need to know what repo it lands in | 22:04 |
| ade_lee_ | well its in cpython .. | 22:04 |
| rlandy|ruck | https://composes.stream.centos.org/production/ | 22:05 |
| rlandy|ruck | https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/ | 22:05 |
| rlandy|ruck | baseos? appstream? | 22:05 |
| ade_lee_ | I would guess baseos .. looking .. | 22:05 |
| rlandy|ruck | https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/ | 22:06 |
| rlandy|ruck | don't see it | 22:06 |
| ade_lee_ | rlandy|ruck, ack - asking the python devs where that would show up | 22:09 |
| dviroel | this is python base code, should land together with new python versions | 22:10 |
| dviroel | https://devguide.python.org/ | 22:10 |
| ade_lee_ | dviroel, yeah -- just wondering how we could tell it landed | 22:11 |
| rlandy|ruck | doens't look like it's in centos base repos | 22:11 |
| ade_lee_ | rlandy|ruck, dviroel do we need to do anything special - or do we automatically get the latest compose in centos-9-stream? | 22:12 |
| rlandy|ruck | ade_lee_: the mirrors should update to latest | 22:12 |
| rlandy|ruck | the process of moving from compose to release in stream is a mystery to me | 22:12 |
| rlandy|ruck | but they update regularly | 22:13 |
| ade_lee_ | rlandy|ruck, dviroel ok -- I just rechecked https://review.opendev.org/c/openstack/tripleo-ci/+/824479 | 22:13 |
| rlandy|ruck | you can check the mirror | 22:13 |
| ade_lee_ | lets see if the problem goes away ... | 22:13 |
| ade_lee_ | rlandy|ruck, I would check the mirror if I knew what to look for :) | 22:14 |
| rlandy|ruck | ade_lee_: in any job ... | 22:14 |
| rlandy|ruck | look at logs | 22:14 |
| rlandy|ruck | undercloud | 22:14 |
| rlandy|ruck | etc/yum.repos.d | 22:14 |
| rlandy|ruck | you'll find some repos with quickstart* | 22:14 |
| rlandy|ruck | open those files | 22:14 |
| rlandy|ruck | see baseurl | 22:14 |
| rlandy|ruck | we remove the standard centos repos and replace them with ones that reference the mirrors | 22:15 |
| rlandy|ruck | ping us if you don't find them | 22:15 |
| rlandy|ruck | dviroel; hey - did you get a chance to look at the upgrades job for c9 master? - I didn't yet | 22:16 |
| rlandy|ruck | can try a patch now if not | 22:17 |
| dviroel | rlandy|ruck: yeah, i tried | 22:17 |
| dviroel | rlandy|ruck: and it failed | 22:17 |
| dviroel | rlandy|ruck: missing wallaby current-tripleo I think | 22:18 |
| * dviroel gets tp | 22:18 | |
| rlandy|ruck | dviroel: oh - we can promote that if you need it | 22:18 |
| dviroel | rlandy|ruck: https://review.rdoproject.org/r/c/testproject/+/36976 | 22:18 |
| dviroel | rlandy|ruck: Failed to retrieve repo file from http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos9-wallaby/current-tripleo/delorean.repo.md5 after 20 retries | 22:19 |
| dviroel | rlandy|ruck: yeah, need a promotion | 22:19 |
| rlandy|ruck | dviroel: k - so we can do a promotion tomorrow | 22:19 |
| rlandy|ruck | it's a component by component promotion | 22:19 |
| dviroel | rlandy|ruck: ok, then we can move foward | 22:19 |
| rlandy|ruck | or add it to the promoter with empty criteria file | 22:19 |
| rlandy|ruck | dviroel: ^^ easier | 22:19 |
| rlandy|ruck | dviroel: let me know - can help with that tomorrow | 22:20 |
| * rlandy|ruck needs to check failing ovb now | 22:20 | |
| dviroel | rlandy|ruck: sure, thanks | 22:21 |
| ade_lee_ | rlandy|ruck, dviroel actually probably not there -- the patches still under review | 22:22 |
| rlandy|ruck | ok - but dviroel is right - will probably land with python | 22:23 |
| rlandy|ruck | no trace in centos repos | 22:23 |
| dviroel | ade_lee_: i was expecting a PR to python repo and then a release, but not sure the flow | 22:24 |
| dviroel | that repo in last BZ comment is a fork | 22:24 |
| ade_lee_ | dviroel, rlandy|ruck -- yeah, I missed a key phrase in the bz comment -- "currently under review" | 22:30 |
| dviroel | lol | 22:30 |
| * dviroel dinner and out - seeya o/ | 22:34 | |
| *** dviroel is now known as dviroel|out | 22:34 | |
| rcastillo|rover | neutron patch didn't make it through gate, some tox failure | 23:00 |
| rcastillo|rover | getting something to eat and signing off for the day | 23:01 |
| *** rcastillo|rover is now known as rcastillo|out | 23:01 | |
| rlandy|ruck | rcastillo|out: see you tomorrow | 23:05 |
| * dasm => afk | 23:13 | |
| *** dasm is now known as dasm|off | 23:13 | |
| *** ysandeep|away is now known as ysandeep | 23:52 | |
| rlandy|ruck | ysandeep: how are you? | 23:58 |
| ysandeep | rlandy|ruck, better than last night, thank you for asking | 23:58 |
| rlandy|ruck | ysandeep: submitted a revert for patch break wallaby upgrade | 23:59 |
| rlandy|ruck | dvd is working on a fix | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!