| *** ykarel|away is now known as ykarel | 03:34 | |
| *** ysandeep|out is now known as ysandeep | 04:32 | |
| ysandeep | Hey o/ chandankumar good morning. Yesterday, Were you referring about this issue "/home/zuul/workspace/hash_info.sh: No such file or directory" .. When you mentioned fix is coming. | 04:48 |
|---|---|---|
| chandankumar | ysandeep: thought it was easy but does not seems to be easy | 04:51 |
| chandankumar | ysandeep: adding some debug hack | 04:52 |
| chandankumar | https://review.rdoproject.org/r/ | 04:52 |
| chandankumar | https://review.rdoproject.org/r/c/config/+/36193 | 04:52 |
| * ysandeep looking | 04:52 | |
| chandankumar | I am merging this to see what is going on there | 04:53 |
| ysandeep | chandankumar, you have my symbolic +1 | 04:54 |
| chandankumar | ysandeep: thanks! | 04:56 |
| * ysandeep wondering why we are changed ssername to zuul-worker in cs9.. we had consistent name b/w c7 and c8. | 05:09 | |
| ysandeep | username* | 05:09 |
| chandankumar | ysandeep: good question, good to check with jpena|off | 05:16 |
| ysandeep | chandankumar: sure, I will ping him once he comes online | 05:16 |
| chandankumar | in zuul legacy vars, they used to create symlink with /home/zuul https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=playbooks/tripleo-rdo-base/pre.yaml#l164 | 05:17 |
| ysandeep | we are not calling ^^ this pre in our job | 05:19 |
| chandankumar | yes, we are not calling it | 05:20 |
| * ysandeep checking if we are calling this in consistent-to-compenent-ci-testing job | 05:20 | |
| ysandeep | ahh.. this is failing while checking reports from DLRN, consistent-to-compenent-ci-testing doesn't need to check dlrn | 05:22 |
| chandankumar | https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=roles/promote-hash/tasks/check_for_previous_promotions.yml;h=1acf3abab50a2569a687d672112ebb5b5f6c82b7;hb=HEAD#l15 | 05:24 |
| ysandeep | ci_config_repo - is set to /home/zuul , i think that's the issue | 05:29 |
| chandankumar | ysandeep: where? | 05:32 |
| chandankumar | I was looking for that last night with last night but not able to find out | 05:33 |
| ysandeep | chandankumar, ^^ ignore my bad.. | 05:36 |
| ysandeep | but we are looking at wrong place.. its failing here https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=ci-scripts/tripleo-upstream/check-commit-hash-promote-target.sh;h=cdd170be8f21ee01ed9b96101f188dd1806d367b;hb=HEAD#l8 | 05:36 |
| ysandeep | hmm.. workspace value inside the shell task is different | 05:39 |
| chandankumar | trying one more hack | 05:40 |
| chandankumar | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36194 | 05:40 |
| ysandeep | thanks! I think that will work.. | 05:43 |
| chandankumar | nope | 05:45 |
| chandankumar | not working | 05:45 |
| chandankumar | ysandeep: let's go with this https://review.rdoproject.org/r/c/config/+/36195 | 05:55 |
| *** ksambor_ is now known as ksambor | 06:02 | |
| ysandeep | chandankumar, ack we can give it a shot.. I am not sure and still trying to figure out why inside shell workspace value is /home/zuul. | 06:02 |
| chandankumar | it worked at that place one more fix needed | 06:27 |
| chandankumar | ysandeep: https://review.rdoproject.org/r/c/config/+/36196 | 06:32 |
| ysandeep | chandankumar, ack +1 | 06:35 |
| chandankumar | ysandeep: ok now it worked, but it is blocked on api endpoint while creating symlink | 07:07 |
| ysandeep | chandankumar, checking | 07:07 |
| chandankumar | ysandeep: I am not sure api-centos9-master-uc is fully functional | 07:08 |
| chandankumar | Symlink creation failed with error: [Errno 13] Permission denied: | 07:09 |
| chandankumar | https://logserver.rdoproject.org/54/35554/11/check/periodic-tripleo-centos-9-master-component-baremetal-promote-to-promoted-components/b4fee77/job-output.txt | 07:09 |
| ysandeep | chandankumar: yes would be good to check with infra, jpena|off have off today(spain public holiday) | 07:09 |
| chandankumar | ysandeep: so for current testing, let's do this | 07:10 |
| chandankumar | use consistent repo instead of tripleo-ci-testing via release file | 07:10 |
| chandankumar | and then pass the dlrn md5 hash with in the job | 07:11 |
| chandankumar | I think it might work | 07:11 |
| chandankumar | ykarel_: hello, do you have access to cs9 dlrn server? | 07:12 |
| ykarel_ | chandankumar, is internal server error happening only in c9? | 07:12 |
| chandankumar | ykarel_: yes | 07:12 |
| ykarel_ | you seen latest runs for c8? | 07:12 |
| ykarel_ | i think if issue is there it should be for all releases | 07:12 |
| ykarel_ | afaik c9 is not special here, should work fine | 07:12 |
| ykarel_ | recently we switched selinux to enforcing, that might have caused the permission issue, if true all release api should be impacted | 07:13 |
| chandankumar | ykarel_: it is from recent run https://logserver.rdoproject.org/openstack-promote-component/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-baremetal-promote-to-promoted-components/5753362/job-output.txt | 07:13 |
| ykarel_ | and yes i have access | 07:13 |
| chandankumar | 1 hour back | 07:14 |
| ykarel_ | and c9 failure was before that or after that? | 07:14 |
| chandankumar | ykarel_: the job has started working now | 07:14 |
| chandankumar | earlier it was broken at other place | 07:14 |
| ykarel_ | chandankumar, okk good than, it might be possible that the failure were at the time of selinux changes were applied | 07:14 |
| ykarel_ | https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/35933 was the change and it merged at 11:32 IST | 07:16 |
| ykarel_ | ohkk or you meant job started running now instead of working, /me checks | 07:17 |
| chandankumar | ykarel_: that is a new job, right now it started working at hitting dlrn api step | 07:18 |
| chandankumar | for the first time | 07:18 |
| chandankumar | and it failed with internal server error | 07:19 |
| ysandeep | last promote jobs ran ~1.5 hours ago.. selinux changed was merged after that | 07:20 |
| ysandeep | change* | 07:20 |
| ykarel_ | ok so that likely not related | 07:22 |
| ykarel_ | will check | 07:22 |
| ysandeep | ykarel_, sry I mean last c8 promote jobs ran ~1.5 hours ago, but c9 job was run recently | 07:30 |
| * chandankumar is stepping out, will back in evening. | 07:30 | |
| ykarel_ | ysandeep, ok thanks for confirming, i see selinux change was applied around 11:38 IST | 07:32 |
| ykarel_ | if failing job was around that good to rerun and see if all good now | 07:32 |
| ysandeep | ykarel_, 12:37pm https://review.rdoproject.org/r/c/testproject/+/35554/11#message-a0ff8f93a030e393a3cd411cba0e7af2dec752cd | 07:33 |
| ykarel_ | ok it's too later, and no c8 promote job ran after 11:38, right? | 07:35 |
| ysandeep | ykarel_, few jobs just ran and failed | 07:37 |
| ysandeep | ykarel_, https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-train-component-compute-promote-consistent-to-component-ci-testing/7189065/job-output.txt | 07:37 |
| ysandeep | another example: https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-compute-promote-consistent-to-component-ci-testing/76506ae/job-output.txt | 07:38 |
| ysandeep | failing with internal server error | 07:38 |
| ykarel_ | Okk Good | 07:39 |
| ykarel_ | that confirms it's not c9 specific | 07:39 |
| ysandeep | yes, all the branches are impacted | 07:39 |
| ykarel_ | okk seems api case was not considered at all with that patch as selinux context applied only to workers | 07:44 |
| ykarel_ | did manual permissive and triggered job in https://review.rdoproject.org/r/c/testproject/+/35554/ | 07:45 |
| ykarel_ | let's see | 07:45 |
| ysandeep | ykarel_: ack | 07:46 |
| *** ykarel_ is now known as ykarel | 07:47 | |
| ysandeep | ykarel, do we have single instance of dlrn that takes care of all branches or we have multiple instance for each branch? | 07:47 |
| ykarel | ysandeep, we have two 1 for centos7 ones and other for centos8 and centos9 | 07:48 |
| ykarel | and for public access there is seperate server | 07:49 |
| ykarel | api hits on that seperate server and there we have issue as selinux contexts are not applied to repos | 07:49 |
| ykarel | trunk.rdoproject.org is public one, trunk-centos7 and trunk-centos8 are where dlrn workers are running | 07:50 |
| ykarel | the repos are copied to public server using rsync and i think there we are not synching selinux context | 07:50 |
| ykarel | need to check with amoralej|off ^ | 07:51 |
| ysandeep | ykarel, okay | 07:51 |
| ykarel | my laptop battery is about to die, power outage here | 07:51 |
| ysandeep | ykarel, it worked, thanks! | 07:51 |
| ykarel | okk good | 07:51 |
| ysandeep | can we keep selinux as permissive till we figure out the next steps | 07:52 |
| ykarel | yes | 07:52 |
| ysandeep | great thanks! ykarel++ | 07:53 |
| *** ysandeep is now known as ysandeep|lunch | 08:16 | |
| *** ysandeep|lunch is now known as ysandeep | 08:37 | |
| *** rlandy is now known as rlandy|ruck | 10:32 | |
| rlandy|ruck | arxcruz|ruck: bhagyashris|rover: how are things today? | 10:38 |
| rlandy|ruck | arxcruz|ruck: any progress with the update/upgrade jobs? | 10:53 |
| rlandy|ruck | got another one for you | 10:53 |
| rlandy|ruck | ade_lee: you got a clean run on https://review.rdoproject.org/r/c/testproject/+/36139 | 10:58 |
| rlandy|ruck | with fips enabled image | 10:58 |
| rlandy|ruck | thanks to dpawlik | 10:58 |
| dpawlik | np rlandy|ruck :) | 10:59 |
| rlandy|ruck | ysandeep: for rhos-17 ... | 10:59 |
| rlandy|ruck | fs020 is the only red job atm | 11:00 |
| rlandy|ruck | going to rerun | 11:00 |
| ysandeep | rlandy|ruck, ack | 11:00 |
| rlandy|ruck | let's see if we get an automatic promotion | 11:00 |
| rlandy|ruck | ysandeep: ^^ | 11:01 |
| ysandeep | rlandy|ruck, fyi.. we have created hashes till tripleo-c-testing for c9.. build container successfully as well with tripleo-ci-testing hash | 11:03 |
| rlandy|ruck | pojadhav: how are we looking on the internal components? | 11:03 |
| rlandy|ruck | ysandeep: that's awesome | 11:03 |
| rlandy|ruck | is chandankumar out atm? | 11:03 |
| ysandeep | rlandy|ruck, yes he is out for couple of hours | 11:03 |
| rlandy|ruck | k - np | 11:04 |
| ysandeep | rlandy|ruck, we are good on components, 17 ovb jobs are failing, need promotion to clear those(fix in tripleo component) | 11:04 |
| rlandy|ruck | ysandeep: we need to promote the tripleo component or we need to promote the 17 integration line? | 11:05 |
| ysandeep | rlandy|ruck, just need integration line promotion. | 11:05 |
| rlandy|ruck | ysandeep: ok - I'll work on getting that through today | 11:05 |
| ysandeep | yeah, I was watching 17 queue so that we can get promotion before wednesday, fs020 failed few mins back | 11:06 |
| rlandy|ruck | marios: sshnaidm: https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 11:07 |
| rlandy|ruck | we got a -1 there | 11:07 |
| rlandy|ruck | any other DIB patch we need to get through? | 11:07 |
| rlandy|ruck | otherwise we can add our testing to comments there and will ping ian later | 11:07 |
| rlandy|ruck | ysandeep: no worries | 11:08 |
| rlandy|ruck | should be fine | 11:08 |
| rlandy|ruck | you are doing a lot of other things now:) | 11:08 |
| sshnaidm | rlandy|ruck, I'll look how we can test it in CI as ianw asks | 11:08 |
| rlandy|ruck | ysandeep: and the jenkins jobs and minimal feature set are taking a lot of attention | 11:08 |
| rlandy|ruck | sshnaidm++ thanks | 11:09 |
| ysandeep | rlandy|ruck, yeah, should be fine after rabi's latest patch to fix ussuri. | 11:09 |
| rlandy|ruck | ysandeep: we always seem to have three releases passing and three failing | 11:09 |
| rlandy|ruck | they just change which ones pass and fail | 11:10 |
| rlandy|ruck | yep - waiting to w+ that patch | 11:10 |
| rlandy|ruck | wallaby and master are still failing | 11:10 |
| rlandy|ruck | looking at those | 11:10 |
| ysandeep | hmm.. they are fixed at one point of time.. https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/ | 11:11 |
| rlandy|ruck | ysandeep: always a challenge!! | 11:11 |
| rlandy|ruck | ysandeep: no worries - you can focus on c9 | 11:11 |
| rlandy|ruck | I will look into those | 11:12 |
| ysandeep | rlandy|ruck, ack thanks! yeah need to take care of python3-virtualenv for quickstart now :) https://logserver.rdoproject.org/29/35229/5/check/periodic-tripleo-ci-centos-9-standalone-master/17bbe8d/job-output.txt | 11:12 |
| pojadhav | rlandy|ruck, component jobs are good except jenkins jobs. | 11:13 |
| rlandy|ruck | arxcruz|ruck: we still have https://bugs.launchpad.net/tripleo/+bug/1946641 | 11:15 |
| rlandy|ruck | bhagyashris|rover: ^^ can you take a look at that | 11:15 |
| rlandy|ruck | note similar fix in the upstream | 11:15 |
| sshnaidm | amoralej|off, can you please take a look at https://review.opendev.org/c/openstack/diskimage-builder/+/806819 ? | 11:19 |
| sshnaidm | rlandy|ruck, there is a testing only for nodepool images, like here amoralej|off sets: https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 11:19 |
| sshnaidm | I don't see any other testing | 11:20 |
| sshnaidm | I'm not even sure the tests for nodepool image touch this code | 11:20 |
| sshnaidm | let's wait for amoralej|off to clear this | 11:21 |
| ykarel | sshnaidm, Spain has holiday today, so Alfredo will be back tomorrow | 11:25 |
| sshnaidm | ykarel, ack | 11:25 |
| marios | rlandy|ruck: not aware of somethingb | 11:25 |
| marios | rlandy|ruck: but didn't see that -1 yet checking | 11:25 |
| rlandy|ruck | marios: no worries - sshnaidm is on it | 11:26 |
| ykarel | sshnaidm, also currently in RDO nodepool we are not using dib images, but using cloud image + some virt-customize | 11:26 |
| marios | rlandy|ruck: for image build at least green with just that one you pointed at | 11:26 |
| sshnaidm | ykarel, yeah, but his patch in DIB repo is related to nodepool image I think | 11:26 |
| rlandy|ruck | we need it for the upstream nodepool image | 11:27 |
| *** pojadhav is now known as pojadhav|brb | 11:27 | |
| marios | sshnaidm: do you know what is the 'other change' that ian is referring to? | 11:27 |
| arxcruz|ruck | bhagyashris|rover: rlandy|ruck back | 11:27 |
| ykarel | sshnaidm, actually the time Alfredo proposed the patch it was too early for c9 as repos and cloud images were missing | 11:27 |
| rlandy|ruck | arxcruz|ruck: one sec | 11:27 |
| marios | sshnaidm: " I would like to see this combined with the other change to make sure we have some sort of coverage. " | 11:28 |
| ykarel | so that patch is outdated now | 11:28 |
| sshnaidm | marios, https://review.opendev.org/c/openstack/diskimage-builder/+/811392 | 11:28 |
| sshnaidm | ykarel, I see | 11:28 |
| marios | sshnaidm: thanks, i don't see any tests there i thought that's what he meant. | 11:28 |
| marios | sshnaidm: so we can point to https://review.rdoproject.org/r/c/testproject/+/35465/10#message-b90fbe5330a14c6d4507916864917620a5858b60 | 11:29 |
| sshnaidm | marios, there are no any tests, only for nodepool images | 11:29 |
| marios | sshnaidm: even though i think he means probably check/aget jobs | 11:29 |
| sshnaidm | marios, yeah, but won't hurt | 11:29 |
| marios | sshnaidm: do we even have a tripleo job there? /me checking | 11:30 |
| marios | https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/jobs.yaml | 11:30 |
| sshnaidm | marios, nope | 11:32 |
| marios | yeah | 11:32 |
| marios | sshnaidm: so not sure what he means but i am adding a comment there sec | 11:32 |
| bhagyashris|rover | arxcruz|ruck, ack | 11:33 |
| marios | sshnaidm: https://review.opendev.org/c/openstack/diskimage-builder/+/806819/8#message-ad2c97f979238e19d0497d150defc3c50abb3a22 | 11:34 |
| rlandy|ruck | arxcruz|ruck: bhagyashris|rover: let's sync in 5 | 11:37 |
| bhagyashris|rover | rlandy|ruck, ack | 11:38 |
| marios | rlandy|ruck: arxcruz|ruck: bhagyashris|rover: i am going to workflow https://review.opendev.org/c/openstack/tripleo-ci/+/810261/6#message-5f8efc42d650f7b021fcdaf2e3c4ecb30dbd23e6 as discussed in yesterday scrum | 11:41 |
| marios | arxcruz|ruck: bhagyashris|rover: fyi in case you see any issues for 3rd party rdo jobs we can revert it | 11:42 |
| bhagyashris|rover | marios, ack thanks :) | 11:42 |
| rlandy|ruck | marios: ack - sure | 11:43 |
| rlandy|ruck | arxcruz|ruck: bhagyashris|rover: https://meet.google.com/etr-zncc-tgs?pli=1&authuser=0 | 11:43 |
| rlandy|ruck | arxcruz|ruck: https://bugs.launchpad.net/tripleo/+bug/1945682 | 11:52 |
| marios | rlandy|ruck: chandankumar: do we have an upstream nodeset yet for 9? not rdo i mean but opendev infra ? | 11:59 |
| marios | probably not yet | 12:00 |
| rlandy|ruck | marios: not yet - that is what amoralej|off was working on | 12:00 |
| marios | so we cant define the check/gate jobs | 12:00 |
| marios | rlandy|ruck: ack thanks | 12:00 |
| rlandy|ruck | marios: hence all the push to get the DIB patches through :) | 12:00 |
| marios | ack | 12:01 |
| arxcruz|ruck | rlandy|ruck: according fungi is not the same error | 12:07 |
| rlandy|ruck | arxcruz|ruck: hmmm - ok | 12:09 |
| rlandy|ruck | arxcruz|ruck: pls work on fixing the update/upgrades jobs first | 12:09 |
| rlandy|ruck | since that is killing the gate | 12:09 |
| rlandy|ruck | bhagyashris|rover: can you pick up the sibling failure investigation from arxcruz|ruck? | 12:10 |
| bhagyashris|rover | rlandy|ruck, sure | 12:17 |
| rlandy|ruck | bhagyashris|rover: thanks | 12:19 |
| marios | sshnaidm: fyi there is a template wired up but we are blocked on the nodeset https://review.opendev.org/c/openstack/diskimage-builder/+/806819/8#message-5f559648e070777daec8c30a1c0bcc8cfc625e73 | 12:19 |
| sshnaidm | marios, ack | 12:23 |
| soniya29 | rlandy|ruck, meeting? | 12:31 |
| rlandy|ruck | yep joining | 12:31 |
| *** pojadhav|brb is now known as pojadhav | 12:34 | |
| rlandy|ruck | chandankumar: arxcruz|ruck: FYI ... pls review https://review.opendev.org/c/openstack/tempest/+/813590 | 12:45 |
| rlandy|ruck | soniya29: ^^ | 12:45 |
| arxcruz|ruck | rlandy|ruck: soniya29 this will not be executed on our side, only on tempest side, you can either add it on os_tempest role, or in our tempest.yml playbook | 12:47 |
| chandankumar | ysandeep: awesome , we have got the stuff ready for cs9 | 12:48 |
| chandankumar | frenzy_friday: \o | 12:48 |
| soniya29 | arxcruz|ruck, i have added this in run_tempest role | 12:48 |
| ysandeep | chandankumar, yeah need to take care of python3-virtualenv for quickstart now :) https://logserver.rdoproject.org/29/35229/5/check/periodic-tripleo-ci-centos-9-standalone-master/17bbe8d/job-output.txt | 12:49 |
| frenzy_friday | chandankumar, o/ | 12:49 |
| ysandeep | chandankumar, trying.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/813624 | 12:49 |
| arxcruz|ruck | soniya29: tripleo doesn't uses run_tempest but os_tempest role to execute tempest | 12:49 |
| chandankumar | frenzy_friday: please update this patch https://review.rdoproject.org/r/c/rdo-jobs/+/35831 and suggestion | 12:49 |
| chandankumar | we can merge it and get the line up for cs9 | 12:49 |
| chandankumar | ysandeep: checking | 12:49 |
| rlandy|ruck | arxcruz|ruck: thanks - soniya29 will update patch | 12:49 |
| frenzy_friday | chandankumar, ack, checking | 12:51 |
| frenzy_friday | chandankumar, new patch https://review.rdoproject.org/r/c/config/+/36220 | 13:00 |
| chandankumar | rlandy|ruck: bhagyashris|rover rlandy|ruck UAPMTC sync | 13:01 |
| chandankumar | soniya29: ^^ | 13:01 |
| rlandy|ruck | bhagyashris|rover: ^^ | 13:02 |
| Tengu | hello there! | 13:05 |
| Tengu | I'm trying to understand how sealert is supposed to work - and it seems there's an associated "dbus service" named setroubleshootd. | 13:05 |
| Tengu | apparently, we're NOT installing that tool at the very beginning of the CI job (i.e. when we prepare the undercloud env), so I'm pretty sure we're not taking advantage of that "service" at all. This might lead to the very long sealert run in the collect-logs step. | 13:06 |
| Tengu | soooo I'm wondering if, instead of pushing some hard timeout on the "sealert -a" command, we shouldn't just install the setroubleshoot package at the very start. | 13:07 |
| Tengu | Especially with: "setroubleshootd is a system daemon which runs under setroubleshoot user and listens for audit events emitted from the kernel related to SELinux." - so it's working "live", and it "records the results of the analysis" | 13:08 |
| Tengu | meaning sealert should be way faster. | 13:08 |
| Tengu | rlandy|ruck: what do you think? | 13:08 |
| Tengu | that might make https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809987 useless. | 13:09 |
| Tengu | sealert, as currently used, may take a long time if there are a lot of AVC. | 13:09 |
| rlandy|ruck | sec - in meeting | 13:13 |
| Tengu | np. I'm trying to get a better understanding of the intend of setroubleshootd (and sealert) | 13:14 |
| Tengu | aha. seems we'd need to install setroubleshoot-server | 13:15 |
| Tengu | that's the server part. | 13:15 |
| Tengu | fun. sealert is supposed to run in a GTK env. | 13:18 |
| Tengu | or, at least, seems to. | 13:19 |
| bhagyashris|rover | Tripleo CI community meeting in 2 mins | 13:28 |
| bhagyashris|rover | feel free to add the agenda here https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-10-12-Community-Call @ line 31 | 13:29 |
| bhagyashris|rover | arxcruz, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday, anbanerj, dviroel ^ | 13:29 |
| bhagyashris|rover | arxcruz|ruck, ^ | 13:32 |
| rlandy|ruck | Tengu: reading back {{ meetings }} | 13:59 |
| Tengu | rlandy|ruck: np. I also think we don't use the sealert output in fact... | 14:00 |
| ade_lee | rlandy|ruck, happy dance :) | 14:00 |
| Tengu | I mean, we don't redirect it to any file | 14:00 |
| Tengu | and sealert doesn't write anythin. | 14:00 |
| rlandy|ruck | ade_lee: yep - you're all set in RDO world | 14:00 |
| rlandy|ruck | we will need downstream equivalent | 14:00 |
| ade_lee | rlandy|ruck, yup plan to talk to lon this week | 14:00 |
| rlandy|ruck | Tengu: there are a bunch of comments above | 14:01 |
| rlandy|ruck | ade_lee: that image is built with virt-customize | 14:01 |
| rlandy|ruck | we can change that one ourselves | 14:01 |
| rlandy|ruck | ade_lee: pls ping this afternoon and we can work on it | 14:02 |
| rlandy|ruck | T<engu> rlandy|ruck: np. I also think we don't use the sealert output in fact... | 14:02 |
| ade_lee | rlandy|ruck, will do | 14:02 |
| rlandy|ruck | Tengu: ^^ are you all set - or do you have an action item for CI on this? | 14:02 |
| Tengu | rlandy|ruck: I'm looking into that whole thing - pretty sure we can make sealert faster. Also discovering we might want to get "sesearch" in order to list currently allowed things, for instance. | 14:03 |
| Tengu | that might help a bit | 14:03 |
| Tengu | rlandy|ruck: I'll produce an LP with details once I'm all set | 14:03 |
| rlandy|ruck | Tengu: ok - sounds good - then we can pick up tasks/changes if needed | 14:03 |
| Tengu | exactly | 14:04 |
| rlandy|ruck | Tengu: BTW - used your LADA/ where's my patch tool the other day - was a great help - thank you | 14:04 |
| ysandeep | Tengu: fyi.. selinux is in permissive mode in upstream jobs | 14:05 |
| rlandy|ruck | following multiple neutron patches around | 14:05 |
| rlandy|ruck | ysandeep: should be on in downstream, no? | 14:05 |
| Tengu | ysandeep: it's "intended" - but in permissive, it will still log denials | 14:05 |
| ysandeep | rlandy|ruck, yes in downstream we have selinux in enforcing mode | 14:05 |
| rlandy|ruck | we were supposed to be keep track of those | 14:06 |
| rlandy|ruck | bit nobody really did | 14:06 |
| rlandy|ruck | ie: so they didn't just arrive downstream | 14:07 |
| Tengu | rlandy|ruck: there might be better commands than sealert :). audit2allow -e might be nice. faster and better. | 14:30 |
| Tengu | rlandy|ruck: I might produce some custom parser that will create a kind of "consolidated" denial list. | 14:30 |
| chandankumar | ysandeep: rlandy|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/35831 | 14:32 |
| Tengu | rlandy|ruck: also discussing with SELinux folks for a nice solution. might end in a contrib to setroubleshoot :). | 14:33 |
| rlandy|ruck | Tengu: they don't serve much of a purpose in the usptream | 14:34 |
| rlandy|ruck | so a compact solution would be bst | 14:34 |
| rlandy|ruck | best | 14:34 |
| Tengu | yeah. trying to gather intel' | 14:34 |
| rlandy|ruck | cool | 14:35 |
| Tengu | aha! | 14:35 |
| Tengu | rlandy|ruck: so. yeah. installing setroubleshoot-server at the very beginning will allow to run `sealert -l '*'` at the end, and should be really, really faster | 14:36 |
| Tengu | because everything will be ready. | 14:36 |
| rlandy|ruck | chandankumar: frenzy_friday: ysandeep: ok to merge https://review.rdoproject.org/r/c/rdo-jobs/+/35831? | 14:37 |
| ysandeep | rlandy|ruck, chandankumar, frenzy_friday looks good to me | 14:37 |
| rlandy|ruck | and will merge the next patch | 14:37 |
| chandankumar | rlandy|ruck: +1 | 14:38 |
| ysandeep | rlandy|ruck, we already tested in testproject.. those 2 jobs which frenzy_friday have added are passing | 14:38 |
| rlandy|ruck | nice | 14:38 |
| * Tengu creating an LP | 14:39 | |
| chandankumar | rlandy|ruck: please hit +w https://review.rdoproject.org/r/c/rdo-jobs/+/35831 as we merge the template name | 14:40 |
| rlandy|ruck | chandankumar: w+ on both | 14:41 |
| chandankumar | cool thanks! | 14:41 |
| rlandy|ruck | chandankumar: anything else that needs review? | 14:42 |
| chandankumar | rlandy|ruck: ysandeep might be interested in this https://review.opendev.org/c/openstack/tripleo-ci/+/813619 | 14:43 |
| Tengu | rlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1946763 | 14:44 |
| chandankumar | rlandy|ruck: once zuul passes then https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/813137 | 14:44 |
| chandankumar | rlandy|ruck: I need to take a look at these https://review.opendev.org/q/topic:%2522bp/whole-disk-default%2522+status:open | 14:45 |
| rlandy|ruck | Tengu: thanks - will comment | 14:45 |
| Tengu | rlandy|ruck: also commented on https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809987 with another proposal. | 14:46 |
| Tengu | rlandy|ruck: after a discussion with SELinux folks, the setroubleshoot isn't really container-aware. | 14:47 |
| rlandy|ruck | ugh | 14:47 |
| Tengu | so we'll end up with a really, really huge report for close to nothing. | 14:47 |
| Tengu | maybe using `audit2allow -e -i /var/log/extras/...' is better. | 14:47 |
| Tengu | it will output some interesting data. | 14:47 |
| Tengu | and it's really faster. | 14:47 |
| *** ykarel is now known as ykarel|away | 14:48 | |
| rlandy|ruck | akahat: thanks for starting with https://review.opendev.org/c/openstack/tripleo-ci/+/813619 | 14:52 |
| sshnaidm | arxcruz|ruck, do we still use tempest-sendmail.tripleo.org ? | 14:54 |
| tosky | Tengu: interesting data, or the same data? - sorry for my reply, but this seems to be going in a direction of "this is never going to happen" | 15:18 |
| Tengu | tosky: in a mtg for 40 more minutes | 15:20 |
| tosky | sure, np | 15:21 |
| *** poojajadhav is now known as pojadhav | 15:31 | |
| chandankumar | ysandeep: left some comment here https://review.opendev.org/c/openstack/tripleo-quickstart/+/813624 if around | 15:34 |
| chandankumar | we can take a look at this tomorrow | 15:34 |
| *** marios is now known as marios|out | 15:35 | |
| chandankumar | rlandy|ruck: \o. please get it merged https://review.opendev.org/c/openstack/tripleo-common/+/800580 once it shows green https://review.rdoproject.org/r/c/testproject/+/18953 | 15:42 |
| chandankumar | see ya! | 15:42 |
| rlandy|ruck | sure | 15:45 |
| rlandy|ruck | chandankumar: will do - tanks | 15:45 |
| rlandy|ruck | thanks | 15:45 |
| rlandy|ruck | ysandeep: ^^ pls vote on the above before you are EoD | 15:47 |
| Tengu | tosky: back. Do you have time now? or we can have a quick call tomorrow (iirc you're EMEA) | 16:05 |
| tosky | Tengu: I can't have a call now, just chat, sorry :/ | 16:09 |
| tosky | tomorrow is fine if the calendar says so | 16:09 |
| Tengu | tosky: I might inject a thing at 10am CET apparently. | 16:10 |
| Tengu | a 15 minutes call should be enough - and faster than typing :) | 16:10 |
| tosky | oki | 16:11 |
| Tengu | tosky: booked :). | 16:11 |
| ysandeep | chandankumar, rlandy|ruck latest run failed: https://logserver.rdoproject.org/53/18953/106/check/tripleo-build-containers-stream9-development/d453cc0/logs/build.log | 16:12 |
| Tengu | so... have a great one! | 16:12 |
| ysandeep | chandankumar, rlandy|ruck to mee seems like we are missing logs... https://logserver.rdoproject.org/53/18953/106/check/tripleo-build-containers-stream9-development/d453cc0/logs/container-builds/d2bc4b53-d224-4b2f-8e5e-c356c75f9078/base/os/ I don't see os-build.log | 16:16 |
| rlandy|ruck | ysandeep: in meeting - will look in a few | 16:16 |
| ysandeep | ack o/ | 16:18 |
| * ysandeep out for the day | 16:18 | |
| *** ysandeep is now known as ysandeep|out | 16:18 | |
| rlandy|ruck | ysandeep|out: thanks for the heads up | 17:45 |
| rlandy|ruck | looking now | 17:45 |
| ysandeep|out | rlandy|ruck, thanks! | 17:47 |
| ysandeep|out | rlandy|ruck, side note.. which stream we need to enable on c9? - latest? | 17:48 |
| rlandy|ruck | ysandeep|out: ack - latest | 17:49 |
| rlandy|ruck | per c9 meeting today | 17:49 |
| ysandeep|out | thanks for confirmation. | 17:50 |
| rlandy|ruck | ysandeep|out: if we need to lock - that will be a good discussion to have with alfredo | 17:52 |
| ysandeep|out | i will check with amoralej|off tomorrow. | 17:53 |
| rlandy|ruck | ysandeep|out: that will put more emphasis on the stream locking project | 17:56 |
| ysandeep|out | agree, stream locking work will play crucial part in centos9 stream stability. | 17:58 |
| ysandeep|out | rlandy|ruck, don't we have extras repos? https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/extras/x86_64/os/ | 18:06 |
| ysandeep|out | didn't find powertools also: https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/PowerTools/x86_64/os/ | 18:06 |
| rlandy|ruck | https://composes.centos.org/latest-CentOS-Stream-8/compose/ | 18:09 |
| rlandy|ruck | no extras there either | 18:09 |
| rlandy|ruck | powertools ... | 18:10 |
| rlandy|ruck | idk - maybe they excluded that now | 18:11 |
| ysandeep|out | rlandy|ruck, ack o/ will exclude that.. and see if we have any missing rpms. | 18:12 |
| rlandy|ruck | only way to find out :) | 18:14 |
| rlandy|ruck | ha - failed container job didn't even build containers | 18:15 |
| rlandy|ruck | ade_lee: you all set now? | 18:25 |
| rlandy|ruck | doug is out today - public holiday in Brazil | 18:25 |
| ade_lee | rlandy|ruck, did we need to do anything together to work on downstream testing with fips enabled image? | 18:26 |
| ade_lee | rlandy|ruck, looking good right now though - at least for multinode - both with the fips enabled image and without | 18:26 |
| rlandy|ruck | ade_lee: ah yes ... | 18:27 |
| rlandy|ruck | downstream - will work on that a bit later | 18:27 |
| rlandy|ruck | will ping with review | 18:27 |
| ade_lee | rlandy|ruck, I'm revisiting the other tests now that were failing | 18:27 |
| ade_lee | rlandy|ruck, thanks works for me | 18:27 |
| rlandy|ruck | http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/providers/tripleo-ci.dhall#n37 | 18:33 |
| rlandy|ruck | http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/elements/virt-customize/cloud-rhel-8-4.yaml | 18:34 |
| rlandy|ruck | we need another one of these | 18:34 |
| rlandy|ruck | 2021-10-12 11:54:27 | 2021-10-12 11:54:27.517009 | fa163e1b-33a0-edfa-4586-000000000073 | FATAL | Create a container file | base | error={ | 19:07 |
| rlandy|ruck | 2021-10-12 11:54:27 | "changed": false, | 19:07 |
| ade_lee | rlandy|ruck, hey - do you know where I can find the srpm for python-paramiko in rdo? | 20:33 |
| rlandy|ruck | sprm? | 20:35 |
| rlandy|ruck | srpm | 20:35 |
| ade_lee | rlandy|ruck, yeah - want to build the latest version | 20:36 |
| ade_lee | rlandy|ruck, I had built a version of 2.7.2 but it seems master has moved on to 2.8.0 | 20:37 |
| rlandy|ruck | more than the github repo, not sure | 20:38 |
| rlandy|ruck | oh ... | 20:38 |
| rlandy|ruck | if that builds with dlrn | 20:39 |
| ade_lee | yeah - its in dlrn somewhere | 20:39 |
| rlandy|ruck | python3-paramiko.noarch 2.7.2-1.el8 @delorean-master-testing | 20:40 |
| rlandy|ruck | ok - sec - finding | 20:40 |
| rlandy|ruck | name=dlrn-master-testing | 20:42 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-master/deps/latest/noarch/ | 20:44 |
| rlandy|ruck | python3-paramiko-2.7.1-1.el8.noarch.rpm2020-02-27 10:07 312K | 20:44 |
| rlandy|ruck | ade_lee: ^^ | 20:44 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-master/deps/ | 20:45 |
| rlandy|ruck | all other previous versions | 20:45 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-master/deps/latest/SRPMS/ | 20:46 |
| ade_lee | rlandy|ruck, thanks -- yeah - it looks like https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt changed a couple of days ago | 20:47 |
| ade_lee | and the version there is 2.8.0 | 20:47 |
| ade_lee | rlandy|ruck, maybe that will trigger a build in the next day or two | 20:49 |
| rlandy|ruck | https://trunk.rdoproject.org/centos8-master/deps/202110081310/SRPMS/ | 20:49 |
| rlandy|ruck | previous days | 20:49 |
| * rlandy|ruck building fips image - sec | 20:50 | |
| ade_lee | rlandy|ruck, https://opendev.org/openstack/requirements/commit/6714d78911a1f4ceeb453d99b2e37aadb466abf6 | 20:51 |
| rlandy|ruck | ade_lee: for fips-enabled downstream ... | 20:52 |
| rlandy|ruck | where in this file would we add the line to enable fips: | 20:52 |
| rlandy|ruck | http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/elements/virt-customize/cloud-rhel-8-4.yaml | 20:52 |
| rlandy|ruck | http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/elements/virt-customize/cloud-rhel-8-4.yaml#n176 | 20:53 |
| rlandy|ruck | ade_lee: https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/281397 | 21:01 |
| rlandy|ruck | https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/281397/1/nodepool/elements/virt-customize/cloud-rhel-8-4-fips.yaml | 21:01 |
| rlandy|ruck | see line 178 | 21:02 |
| rlandy|ruck | did I need to do anything else? | 21:02 |
| rlandy|ruck | install something? | 21:02 |
| ade_lee | sorry .. looking | 21:10 |
| ade_lee | rlandy|ruck, that looks good -- lets see if you run into an error if you need to install something to get fips-mode-setup | 21:13 |
| ade_lee | rlandy|ruck, in the infrared work I was testing before, I ended up doing this -- https://review.gerrithub.io/c/redhat-openstack/infrared/+/515913/36/plugins/virsh/tasks/vms_1_create_disk.yml | 21:15 |
| ade_lee | rlandy|ruck, not sure if you need all that though -- I think fips-mode-setup --enable will add the fips=1 to the kernel command line | 21:15 |
| rlandy|ruck | ade_lee: not sure - pls comment on the review | 21:16 |
| rlandy|ruck | I just added what was in the rdo image | 21:16 |
| ade_lee | ack and that might be sufficient | 21:16 |
| rlandy|ruck | we will have to get that built and added to the tenant | 21:17 |
| rlandy|ruck | ysandeep|out: http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp17/api/civotes_agg_detail.html?ref_hash=81bfaba5e1acfd62305aae1ff3b22bb0 | 21:18 |
| rlandy|ruck | 17 has enough criteria tests passing to promote | 21:19 |
| rlandy|ruck | let's see if the automated promoter kicks in | 21:19 |
| rlandy|ruck | otherwise - will kick it manually | 21:19 |
| rlandy|ruck | akahat: ^^ fyi | 21:19 |
| ade_lee | rlandy|ruck, actually -- I'm wondering if it is sufficient | 21:22 |
| ade_lee | rlandy|ruck, I'm looking mor eclosely at the passing run .. | 21:22 |
| rlandy|ruck | ade_lee: k - feel free to edit the review and/or comments | 21:22 |
| rlandy|ruck | comment | 21:22 |
| rlandy|ruck | and we can update | 21:22 |
| ade_lee | rlandy|ruck, https://review.rdoproject.org/zuul/build/0005d261e5cb4e4aaf0fa80d0c8ab020/log/logs/undercloud/var/log/extra/journal.txt.gz | 21:22 |
| ade_lee | in line 3, I would expect to see fips=1 | 21:23 |
| rlandy|ruck | that's in rdo | 21:23 |
| ade_lee | right | 21:23 |
| rlandy|ruck | oh the rdo change might not be enough | 21:24 |
| ade_lee | yes | 21:24 |
| ade_lee | so let me look at this with jpena in the morning - and if we get that right - then we'll tackle downstream | 21:24 |
| rlandy|ruck | ade_lee: sounds good | 21:26 |
| rlandy|ruck | ade_lee: you can show jpena the downstream review - and edit it concurrently | 21:26 |
| ade_lee | ack | 21:27 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:24 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!