| opendevreview | Merged openstack/tripleo-upgrade stable/wallaby: Make sure patch reports issues and ignores test files https://review.opendev.org/c/openstack/tripleo-upgrade/+/866468 | 00:01 |
|---|---|---|
| *** dviroel is now known as dviroel|out | 00:28 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible stable/wallaby: Fix compatibility with Ansible 2.14 https://review.opendev.org/c/openstack/tripleo-ansible/+/865089 | 00:35 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Ceilometer: Enable oslo.cache backend https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855126 | 01:14 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/wallaby: Replace remaining usage of /var/run https://review.opendev.org/c/openstack/tripleo-heat-templates/+/858845 | 01:20 |
| opendevreview | Merged openstack/tripleo-ansible master: Handle child groups when building ceph spec from inventory https://review.opendev.org/c/openstack/tripleo-ansible/+/866503 | 01:59 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 02:28 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 03:31 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 03:36 |
| *** yadnesh|away is now known as yadnesh | 04:45 | |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 04:59 |
| opendevreview | Merged openstack/tripleo-heat-templates stable/zed: Improve volumes format check of yaml-validate.py https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865800 | 05:20 |
| opendevreview | Merged openstack/tripleo-puppet-elements stable/zed: Move tuned_profiles_cpu_partitioning dependency in overcloud-base https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/863460 | 05:20 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 05:37 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 05:41 |
| *** ysandeep is now known as ysandeep|afk | 05:47 | |
| *** marios is now known as marios|ruck | 06:08 | |
| opendevreview | Luca Miccini proposed openstack/puppet-pacemaker stable/1.5.x: Add fence_watchdog (sbd) stonith agent https://review.opendev.org/c/openstack/puppet-pacemaker/+/866650 | 06:12 |
| *** ysandeep|afk is now known as ysandeep | 07:00 | |
| marios|ruck | need votes please https://review.opendev.org/c/openstack/tripleo-ansible/+/866563 train minor update promotion blocker (need to get master/zed/wallaby/train) | 07:03 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-tripleo master: Adapt to the heartbeat_key parameter migration https://review.opendev.org/c/openstack/puppet-tripleo/+/866648 | 07:41 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-heat-templates master: Implementing basic pre-commit hooks for local testing https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859085 | 07:56 |
| opendevreview | Yadnesh Kulkarni proposed openstack/tripleo-heat-templates master: Rectify service_name in ceph monitor's hieradata https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866670 | 07:57 |
| *** yadnesh is now known as yadnesh|afk | 08:13 | |
| opendevreview | Cedric Jeanneret proposed openstack/puppet-tripleo stable/wallaby: Cinder: Manage quota options https://review.opendev.org/c/openstack/puppet-tripleo/+/866651 | 08:16 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 08:32 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-common stable/wallaby: By default, unset everything proxy related for healthcheck https://review.opendev.org/c/openstack/tripleo-common/+/866652 | 08:37 |
| tkajinam | is any core around who can review this ? https://review.opendev.org/c/openstack/puppet-tripleo/+/866648 | 08:43 |
| tkajinam | this fixes unit tests failure in master | 08:43 |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 08:46 |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 08:48 |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 08:59 |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 09:17 |
| *** yadnesh|afk is now known as yadnesh | 09:20 | |
| *** ysandeep is now known as ysandeep|lunch | 09:21 | |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 09:26 |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: DNM testing new doc job overrides for stable branches https://review.opendev.org/c/openstack/validations-common/+/866343 | 09:42 |
| opendevreview | Merged openstack/tripleo-ansible master: Fix ovs check pkg_has_restart error. https://review.opendev.org/c/openstack/tripleo-ansible/+/866563 | 09:48 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: DNM testing molecule job for oslo config validator https://review.opendev.org/c/openstack/tripleo-validations/+/866677 | 09:51 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 10:11 |
| *** jpodivin_ is now known as jpodivin | 10:14 | |
| *** ysandeep|lunch is now known as ysandeep | 10:18 | |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: DNM testing molecule job for oslo config validator https://review.opendev.org/c/openstack/tripleo-validations/+/866677 | 10:36 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-heat-templates master: Implementing basic pre-commit hooks for local testing https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859085 | 10:59 |
| *** dviroel|out is now known as dviroel | 11:12 | |
| marios|ruck | tkajinam: o/ can you please also vote there https://review.opendev.org/c/openstack/tripleo-ansible/+/866564 when you have aminute (master merged) cc chem | 11:12 |
| opendevreview | Merged openstack/tripleo-quickstart-extras master: undercloud: Remove logic for unsupported old releases https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/861692 | 11:14 |
| *** rlandy|out is now known as rlandy|rover | 11:18 | |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations stable/wallaby: Changing mount parameter of podman task to volume https://review.opendev.org/c/openstack/tripleo-validations/+/866687 | 11:21 |
| opendevreview | Yadnesh Kulkarni proposed openstack/tripleo-heat-templates master: Add ceph cluster's fsid to collectd hieradata https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866688 | 11:22 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-heat-templates master: Implementing basic pre-commit hooks for local testing https://review.opendev.org/c/openstack/tripleo-heat-templates/+/859085 | 11:23 |
| marios|ruck | Tengu: can you please vote on that one https://review.opendev.org/c/openstack/tripleo-ansible/+/866564 when you have time (master merged) | 11:29 |
| marios|ruck | actually... chem is core so expediting merge here | 11:30 |
| marios|ruck | thanks sorry for noise Tengu | 11:30 |
| Tengu | :) | 11:31 |
| Tengu | marios|ruck: while you're around: https://review.opendev.org/c/openstack/tripleo-common/+/866652 care to have a quick look? | 11:31 |
| Tengu | trying to get it backported. | 11:31 |
| marios|ruck | Tengu: in call will check in bit | 11:31 |
| chem | marios|ruck: is self merging that https://review.opendev.org/c/openstack/tripleo-ansible/+/866565 ok ? | 11:33 |
| Tengu | chem: merged. | 11:33 |
| chem | I guess it should be | 11:33 |
| chem | Tengu: thanks | 11:33 |
| Tengu | #lunch | 11:33 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates master: Fix list_concat for Nfs mounts https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866691 | 11:40 |
| marios|ruck | ack Tengu voted but waiting on zuul report | 11:55 |
| marios|ruck | chem: ack i think we should get it today. looks like we are promoting train anyway based on our test result (https://review.rdoproject.org/r/c/testproject/+/46346/1#message-c47492d8b09e23d15b522f705a025de0318f5ce0 it took this result and is promoting that hash... not ideal but also not a problem as long as we get that train patch merged ;) | 11:57 |
| *** yadnesh is now known as yadnesh|afk | 12:00 | |
| opendevreview | Jiri Podivin proposed openstack/validations-common stable/1.6: Tox docs job moved to ubuntu-focal nodeset https://review.opendev.org/c/openstack/validations-common/+/866700 | 12:13 |
| *** frenzy_friday is now known as frenzy_friday|food | 12:21 | |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Refactoring default-node-count validation to use openstack-collection modules https://review.opendev.org/c/openstack/tripleo-validations/+/864421 | 12:27 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations stable/wallaby: Refactoring ironic-boot-configuration validation to use openstack collection modules https://review.opendev.org/c/openstack/tripleo-validations/+/864561 | 12:29 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adjusting node-disks validation for new authentication procedures https://review.opendev.org/c/openstack/tripleo-validations/+/861396 | 12:32 |
| *** yadnesh|afk is now known as yadnesh | 13:10 | |
| *** frenzy_friday|food is now known as frenzy_friday | 13:30 | |
| *** ysandeep is now known as ysandeep|dinner | 13:34 | |
| *** dhill is now known as Guest1131 | 13:38 | |
| opendevreview | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Convert Nova httpd configurations to new role https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866177 | 13:49 |
| opendevreview | John Fulton proposed openstack/tripleo-ansible stable/zed: Handle child groups when building ceph spec from inventory https://review.opendev.org/c/openstack/tripleo-ansible/+/866653 | 13:53 |
| opendevreview | Joel Capitao proposed openstack/puppet-pacemaker stable/1.1.x: Add fence_watchdog (sbd) stonith agent https://review.opendev.org/c/openstack/puppet-pacemaker/+/866717 | 14:03 |
| *** dasm|off is now known as dasm | 14:04 | |
| opendevreview | Merged openstack/tripleo-ansible stable/zed: Fix ovs check pkg_has_restart error. https://review.opendev.org/c/openstack/tripleo-ansible/+/866564 | 14:08 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-ansible master: Fix issues with connectivity check https://review.opendev.org/c/openstack/tripleo-ansible/+/866638 | 14:09 |
| *** jpodivin_ is now known as jpodivin | 14:17 | |
| lmiccini | o/ rlandy|rover: hi! not sure if someone from your team can help with https://review.opendev.org/c/openstack/puppet-pacemaker/+/866650 ? looks like zuul is not picking up these cherry picks | 14:20 |
| lmiccini | tkajinam: ^ fyi | 14:21 |
| rlandy|rover | lmiccini: sorry - meeting - will check in a few | 14:26 |
| lmiccini | rlandy|rover: sure thank you | 14:26 |
| rlandy|rover | lmiccini: the branch is likely not resolving | 14:27 |
| rlandy|rover | so master will match a branch | 14:27 |
| rlandy|rover | but stable/1.1.x stc. does not | 14:27 |
| rlandy|rover | jpodivin had a similar issue with validations releases a while back | 14:27 |
| rlandy|rover | lmiccini: so if this is a branch that will need to be used going forward, we need to adjust the branch matching | 14:28 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 14:35 |
| lmiccini | rlandy|rover: ack thank you | 14:35 |
| tkajinam | rlandy|rover, do you have any examples which we can mimic ? I guess the same and was looking for good existing solutions but failed to find any. | 14:37 |
| rlandy|rover | tkajinam: lmiccini: yeah we went down this road at some point with jpodivin and validations | 14:38 |
| rlandy|rover | with a number based release | 14:38 |
| lmiccini | I found https://github.com/openstack/tripleo-ci/blob/cd037774f195faf00ed54d8761431b14a2d4304f/roles/run-test/templates/common_vars.bash.j2 | 14:38 |
| rlandy|rover | lmiccini: tkajinam: pretty much ... https://github.com/openstack/tripleo-ci/blob/cd037774f195faf00ed54d8761431b14a2d4304f/roles/run-test/templates/common_vars.bash.j2#L25 | 14:39 |
| rlandy|rover | # For validations-common repo "stable/1.6" is a support branch | 14:39 |
| rlandy|rover | if [ "$ZUUL_BRANCH" = "stable/1.6" -a "$ZUUL_PROJECT" = "openstack/validations-common" ]; then | 14:39 |
| rlandy|rover | export ZUUL_BRANCH="stable/train" | 14:39 |
| rlandy|rover | export STABLE_RELEASE="train" | 14:39 |
| rlandy|rover | so we'd need something like that for your repos | 14:40 |
| rlandy|rover | if they kicked via quickstart | 14:40 |
| rlandy|rover | these are puppet tests | 14:40 |
| * dviroel joining fips meeting | 14:41 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Glance: Fix wrong format of depeneds_on key https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866720 | 14:41 |
| *** ysandeep|dinner is now known as ysandeep__ | 14:46 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Gnocchi: Ensure NFS mount is ready https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845180 | 14:47 |
| rlandy|rover | lmiccini: tkajinam: so ... https://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/common_vars.bash.j2#L10 should trigger train jobs | 14:47 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-pacemaker stable/1.1.x: DNM: Checking CI https://review.opendev.org/c/openstack/puppet-pacemaker/+/866721 | 14:48 |
| rlandy|rover | lmiccini: tkajinam: what branch do you want to trigger on stable/1.5.x? | 14:48 |
| rlandy|rover | we can add that clause and see if it does anything | 14:48 |
| tkajinam | probably zed ? | 14:48 |
| lmiccini | jcapitao[m]: do we need to have entried for 1.1.x ,1.3.x 1.4.x ? | 14:48 |
| lmiccini | entries* | 14:49 |
| tkajinam | we don't have stable/1.3.0 or stable/1.4.0 | 14:49 |
| tkajinam | only 1.1.0 and 1.5.0 | 14:49 |
| jcapitao[m] | I'd say just 1.1.x and 1.5.x | 14:49 |
| lmiccini | ok | 14:49 |
| tkajinam | so what is the plan. will we use 1.1.x for tran and wallaby and 1.5.x for zed ? | 14:49 |
| jcapitao[m] | yes, I'd go with this plan | 14:50 |
| tkajinam | rlandy|rover, hmm. I'm afraid this patch in stable/1.1.0 does not trigger any job https://review.opendev.org/c/openstack/puppet-pacemaker/+/866721 | 14:50 |
| rlandy|rover | tkajinam: correct - that would work if you expected quickstart related jobs to run | 14:52 |
| rlandy|rover | puppet is something else | 14:52 |
| opendevreview | Luca Miccini proposed openstack/tripleo-ci master: Match puppet-pacemaker branches to respective releases https://review.opendev.org/c/openstack/tripleo-ci/+/866722 | 14:53 |
| lmiccini | ah damn :( | 14:53 |
| opendevreview | Chris Sibbitt proposed openstack/tripleo-ansible stable/wallaby: Fix collectd path replacement bug https://review.opendev.org/c/openstack/tripleo-ansible/+/865601 | 14:54 |
| jpodivin | lmiccini: yeh, the one you saw only works for quickstart based jobs | 14:54 |
| jpodivin | anything else needs to be solved in other ways. | 14:54 |
| jpodivin | I'm dealing with one such case right now. | 14:55 |
| lmiccini | jcapitao[m]: how about we bump everything to 1.5 :D | 14:55 |
| jpodivin | lmiccini: what jobs do you want to run on that branch btw? | 14:56 |
| jcapitao[m] | lmiccini: yes we can, but still we need to have https://review.opendev.org/c/openstack/puppet-pacemaker/+/866650 to be merged and then release 1.5.1 | 14:57 |
| lmiccini | jpodivin: afaics from https://review.opendev.org/c/openstack/puppet-pacemaker/+/863036 it's lint + syntax + unit | 14:58 |
| lmiccini | maybe we need to patch https://github.com/openstack/puppet-openstack-integration/ ? | 14:59 |
| jpodivin | lmiccini: so what you need to do is to change the branches clause in the job definitions https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.branches | 14:59 |
| jpodivin | lmiccini: yeah, there. | 14:59 |
| jpodivin | but you'd need buy in from other projects that use this infa. | 15:00 |
| jpodivin | That's why we didn't do this for validations and instead relied on overrides. Which wasn't a good choice, as it appears they don't do exactly what I thought they should do. | 15:01 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-pacemaker stable/1.5.x: Update .gitreview for stable/1.5.x https://review.opendev.org/c/openstack/puppet-pacemaker/+/863375 | 15:02 |
| tkajinam | ^^^ lmiccini | 15:02 |
| tkajinam | someting like this ? | 15:03 |
| lmiccini | ah interesting | 15:03 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-pacemaker stable/1.5.x: Update TOX_CONSTRAINTS_FILE for stable/1.5.x https://review.opendev.org/c/openstack/puppet-pacemaker/+/863376 | 15:03 |
| opendevreview | Takashi Kajinami proposed openstack/puppet-pacemaker stable/1.5.x: Add fence_watchdog (sbd) stonith agent https://review.opendev.org/c/openstack/puppet-pacemaker/+/866650 | 15:03 |
| tkajinam | I also have to look into puppet jobs because we don't have that stable/1.5.x branch in puppet repos | 15:04 |
| rlandy|rover | tkajinam; lmiccini: ha - it's triggering third party jobs: https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0 | 15:06 |
| rlandy|rover | failing but triggering | 15:06 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 15:11 |
| opendevreview | Bogdan Dobrelya proposed openstack/python-tripleoclient master: Also export OVN DBs IPs for multi-stack inputs https://review.opendev.org/c/openstack/python-tripleoclient/+/866593 | 15:13 |
| opendevreview | Bogdan Dobrelya proposed openstack/python-tripleoclient stable/wallaby: Also export OVN DBs IPs for multi-stack inputs https://review.opendev.org/c/openstack/python-tripleoclient/+/866724 | 15:13 |
| opendevreview | Bogdan Dobrelya proposed openstack/python-tripleoclient stable/zed: Also export OVN DBs IPs for multi-stack inputs https://review.opendev.org/c/openstack/python-tripleoclient/+/866725 | 15:14 |
| opendevreview | Merged openstack/tripleo-ansible stable/wallaby: Fix ovs check pkg_has_restart error. https://review.opendev.org/c/openstack/tripleo-ansible/+/866565 | 15:15 |
| tkajinam | lmiccini, we have to also fix puppet jobs because now no puppet jobs are triggered by this change but I'll look into that tomorrow | 15:16 |
| tkajinam | ... or today precisely speaking | 15:17 |
| opendevreview | Bogdan Dobrelya proposed openstack/python-tripleoclient stable/wallaby: Also export OVN DBs IPs for multi-stack inputs https://review.opendev.org/c/openstack/python-tripleoclient/+/866724 | 15:17 |
| lmiccini | tkajinam: just forget about this and go to sleep :) | 15:17 |
| tkajinam | lmiccini, :-) | 15:18 |
| tkajinam | we can merge it as long as the job passes. | 15:19 |
| opendevreview | mbu proposed openstack/tripleo-ansible master: Start ephemeral heat with the correct params for config-download https://review.opendev.org/c/openstack/tripleo-ansible/+/866726 | 15:22 |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 15:27 |
| *** dviroel is now known as dviroel|lunch | 15:31 | |
| opendevreview | Jiri Podivin proposed openstack/tripleo-validations master: Adding graceful failure condition to options dictionary lookup https://review.opendev.org/c/openstack/tripleo-validations/+/866581 | 15:31 |
| *** pojadhav is now known as pojadhav|afk | 15:32 | |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-ansible master: Configure ansible-galaxy to use caching proxies https://review.opendev.org/c/openstack/tripleo-ansible/+/866031 | 15:41 |
| *** ysandeep__ is now known as ysandeep|out | 15:50 | |
| *** yadnesh is now known as yadnesh|away | 15:51 | |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-ansible master: Configure ansible-galaxy to use caching proxies https://review.opendev.org/c/openstack/tripleo-ansible/+/866031 | 16:31 |
| opendevreview | John Fulton proposed openstack/tripleo-ansible master: Configure Ceph clients in standalone playbooks for a compute node https://review.opendev.org/c/openstack/tripleo-ansible/+/859197 | 16:33 |
| fultonj | mciecierski: ^ fyi | 16:33 |
| *** dviroel|lunch is now known as dviroel | 16:34 | |
| mciecierski | thank you fultonj | 16:34 |
| opendevreview | John Fulton proposed openstack/tripleo-ansible master: Configure Ceph clients in standalone playbooks for a compute node https://review.opendev.org/c/openstack/tripleo-ansible/+/859197 | 16:36 |
| *** marios|ruck is now known as marios|out | 16:55 | |
| rlandy|rover | fultonj: hello ... any recent ceph-related change that could have caused: | 17:20 |
| rlandy|rover | File \\\"/usr/sbin/cephadm\\\", line 1636, in call_throws\", \" raise RuntimeError(f'Failed command: {\\\" \\\".join(command)}: {s}')\", \"RuntimeError: Failed command: systemctl start ceph-b03d550a-899e-57f6-ae90-b3f8ed340e63@mon.standalone.localdomain: Job for ceph-b03d550a-899e-57f6-ae90-b3f8ed340e63@mon.standalone.localdomain.service failed because the control process exited with error code.\" | 17:20 |
| rlandy|rover | as in https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/job-output.txt | 17:21 |
| rlandy|rover | seeing scn001 and 004 fail now | 17:21 |
| rlandy|rover | https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/logs/_ceph-b03d550a-899e-57f6-ae90-b3f8ed340e63@mon_standalone_localdomain_service_failed_because_the_cont.log | 17:21 |
| fultonj | fpantano: ^ fyi | 17:21 |
| * rlandy|rover logs bug | 17:21 | |
| fultonj | rlandy|rover: i'm looking. | 17:22 |
| rlandy|rover | ty | 17:22 |
| rlandy|rover | showing up in zed, wallaby c9 lines atm | 17:22 |
| fultonj | rlandy|rover: do you know when this started? | 17:27 |
| rlandy|rover | fultonj: recently - we had a zed pass earlier ... busy creating bug with logs and details | 17:28 |
| rlandy|rover | will pass here shortly | 17:28 |
| rlandy|rover | dviroel: ^^ need a few - just logging this | 17:30 |
| dviroel | ack | 17:31 |
| rlandy|rover | fultonj: seems like it just started - saw it on sc001 ans cs004 in zed and wallaby lines ... https://bugs.launchpad.net/tripleo/+bug/1998954 | 17:32 |
| rlandy|rover | not the best bug description - sorry | 17:33 |
| rlandy|rover | need to jump onto meeting - then will refine the bug | 17:33 |
| rlandy|rover | fpantano: ^^ | 17:33 |
| fpantano | rlandy|rover: fultonj o/ | 17:33 |
| * fpantano looking | 17:33 | |
| fpantano | rlandy|rover: so this is failing in wallaby -> pacific | 17:34 |
| rlandy|rover | fpantano: ack | 17:35 |
| fpantano | dviroel: I'm not sure we recently promoted new containers | 17:35 |
| fpantano | we bumped master to quincy, but I guess that's green | 17:35 |
| fpantano | rlandy|rover: do we have consistent failures on sc001 ? | 17:35 |
| * fpantano trying to find the dashboard | 17:36 | |
| rlandy|rover | fpantano; I see two so far - notes are in the bug | 17:36 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1998954 | 17:36 |
| rlandy|rover | not the best description | 17:36 |
| rlandy|rover | just on call - can refine later | 17:36 |
| rlandy|rover | fpantano: want me to rerun and recheck? | 17:36 |
| fpantano | rlandy|rover: | 17:37 |
| fpantano | yes | 17:37 |
| fpantano | http://dashboard-ci.tripleo.org/d/yEn51gd4k/ceph-jobs-and-promotions?orgId=1 | 17:37 |
| fpantano | fultonj: FYI ^ we can monitor from here ^ the status of the jobs | 17:37 |
| * fultonj bookmarked | 17:38 | |
| rlandy|rover | https://review.rdoproject.org/r/c/testproject/+/36254 | 17:40 |
| rlandy|rover | rekicked tp | 17:42 |
| rlandy|rover | let's see if we have a reproducer | 17:42 |
| fultonj | owalsh: has an unmerged patch affecting network | 17:45 |
| fultonj | but that should be just for one patch, not all of ci | 17:45 |
| fultonj | that failed for network reasons as expected in the unmerged patch | 17:45 |
| fultonj | https://review.opendev.org/c/openstack/tripleo-ansible/+/866410 | 17:46 |
| fultonj | oh you said wally | 17:46 |
| fultonj | wallaby | 17:46 |
| * fultonj trying to do this while on a call :/ | 17:46 | |
| fpantano | rlandy|rover: so, the ceph bits are the same, but looking at the logs I see the mon is trying to start | 17:59 |
| fpantano | then dies because of: "Dec 06 11:37:59 standalone.localdomain bash[68635]: Error: open /etc/containers/networks/netavark.lock: permission denied" | 18:00 |
| rlandy|rover | fpantano: I have a tp running | 18:00 |
| rlandy|rover | I can get you on that node | 18:00 |
| fultonj | podman or selinx change? | 18:00 |
| fpantano | fultonj: yeah, something like that | 18:00 |
| fultonj | nice find fpantano | 18:00 |
| fpantano | fultonj: I suspect it's more podman this time | 18:01 |
| fpantano | fultonj: let's try to get that info from the logs | 18:02 |
| opendevreview | Oliver Walsh proposed openstack/tripleo-ansible master: WIP: fix ceph_spec_bootstrap inventory parsing https://review.opendev.org/c/openstack/tripleo-ansible/+/866410 | 18:07 |
| fpantano | rlandy|rover: and it's basically selinux that prevents the ceph container to access to "/etc/containers/networks/netavark.lock"-> https://logserver.rdoproject.org/b6/b63515f8d458cd4599a36d0f60a050c09b8fbfce/openstack-periodic-integration-stable1/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/b62e851/logs/undercloud/var/log/extra/selinux_denials.txt.gz | 18:13 |
| fpantano | fultonj: FYI ^ | 18:13 |
| rlandy|rover | fpantano: I am on the currently running node - pls send keys if you want to get on that node | 18:14 |
| fpantano | rlandy|rover: ack thanks -> https://github.com/fmount.keys | 18:15 |
| fpantano | I can try something on the node | 18:15 |
| * fpantano back in 5 minutes | 18:15 | |
| rlandy|rover | fpantano: ssh zuul@38.102.83.8 | 18:16 |
| rlandy|rover | 0000000154 | rdoproject.org | review.rdoproject.org/testproject | periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby | .* | 1 | ceph debug | | 18:19 |
| rlandy|rover | | 0000000155 | rdoproject.org | review.rdoproject.org/testproject | periodic-tripleo-ci-centos-9-scenario004-standalone-wallaby | .* | 1 | ceph debug | 18:19 |
| rlandy|rover | autohold | 18:19 |
| * rlandy|rover checks selinux related versions | 18:21 | |
| fpantano | yeah, let's check if there's a change in selinux | 18:22 |
| opendevreview | Merged openstack/tripleo-common stable/wallaby: By default, unset everything proxy related for healthcheck https://review.opendev.org/c/openstack/tripleo-common/+/866652 | 18:24 |
| fpantano | rlandy|rover: so, deleted the ceph cluster -> setenforce 0 -> redeploy ceph -> it worked | 18:26 |
| fpantano | so we known that the root cause is something that changed on that front | 18:26 |
| rlandy|rover | fpantano: | 18:27 |
| rlandy|rover | selinux-policy.noarch 38.1.2-1.el9 @anaconda | 18:27 |
| rlandy|rover | selinux-policy-targeted.noarch 38.1.2-1.el9 @anaconda | 18:27 |
| rlandy|rover | vs | 18:27 |
| rlandy|rover | selinux-policy.noarch 38.1.1-1.el9 @anaconda | 18:28 |
| rlandy|rover | selinux-policy-targeted.noarch 38.1.1-1.el9 @anaconda | 18:28 |
| rlandy|rover | maybe? | 18:28 |
| rlandy|rover | diff'ed installed packages on the last passing and first failing jobs | 18:28 |
| rlandy|rover | setenforce should always be 0 on centis | 18:28 |
| rlandy|rover | centos | 18:29 |
| fpantano | rlandy|rover: on that node it was "Enforcing", so I just setenforce 0 to double check it wasn't ceph messing up w/ things | 18:29 |
| rlandy|rover | weird | 18:29 |
| fpantano | but it wasn't "0" there | 18:30 |
| rlandy|rover | let me check the other node | 18:30 |
| fpantano | ack | 18:30 |
| owalsh | rlandy|rover: container-selinux version? | 18:30 |
| opendevreview | Joel Capitao proposed openstack/tripleo-quickstart master: Ignore pip constraint file for bindep installation https://review.opendev.org/c/openstack/tripleo-quickstart/+/866789 | 18:31 |
| rlandy|rover | container-selinux.noarch 3:2.193.0-1.el9 @appstream | 18:31 |
| rlandy|rover | container-selinux.noarch 3:2.193.0-1.el9 @appstream | 18:31 |
| rlandy|rover | on both passing and failing | 18:31 |
| rlandy|rover | owalsh: ^^ | 18:31 |
| rlandy|rover | [zuul@standalone ~]$ getenforce | 18:32 |
| rlandy|rover | Enforcing | 18:32 |
| rlandy|rover | fpantano: ^^ you're right | 18:32 |
| rlandy|rover | I checked the other node | 18:33 |
| rlandy|rover | I had held both | 18:33 |
| owalsh | rlandy|rover: ack related to https://github.com/containers/container-selinux/issues/198 maybe? | 18:33 |
| rlandy|rover | yeah - I remember this | 18:33 |
| rlandy|rover | and then it just went away | 18:34 |
| rlandy|rover | interesting that we are running enforcing though | 18:34 |
| rlandy|rover | https://bugzilla.redhat.com/show_bug.cgi?id=2150283 | 18:35 |
| opendevreview | Joel Capitao proposed openstack/puppet-pacemaker stable/1.5.x: Update .gitreview for stable/1.5.x https://review.opendev.org/c/openstack/puppet-pacemaker/+/863375 | 18:35 |
| rlandy|rover | fpantano: fultonj: owalsh: thanks - going to check on the enforcing issue | 18:36 |
| rlandy|rover | https://bugzilla.redhat.com/show_bug.cgi?id=2150283 claims selinux-policy-38.1.2-1 is better | 18:37 |
| fpantano | rlandy|rover: ack, I think we need to double check what changed on that front | 18:37 |
| *** pojadhav|afk is now known as pojadhav | 18:37 | |
| opendevreview | Merged openstack/puppet-tripleo stable/wallaby: Cinder: Manage quota options https://review.opendev.org/c/openstack/puppet-tripleo/+/866651 | 19:17 |
| opendevreview | Merged openstack/tripleo-ansible master: Fix molecule failures in tripleo_collectd https://review.opendev.org/c/openstack/tripleo-ansible/+/865465 | 19:17 |
| opendevreview | Merged openstack/tripleo-validations stable/wallaby: Changing mount parameter of podman task to volume https://review.opendev.org/c/openstack/tripleo-validations/+/866687 | 19:17 |
| opendevreview | Merged openstack/tripleo-ansible stable/train: Fix ovs check pkg_has_restart error. https://review.opendev.org/c/openstack/tripleo-ansible/+/866566 | 19:18 |
| opendevreview | Merged openstack/tripleo-ansible stable/zed: Handle child groups when building ceph spec from inventory https://review.opendev.org/c/openstack/tripleo-ansible/+/866653 | 19:48 |
| *** dviroel is now known as dviroel|biab | 19:48 | |
| opendevreview | Martin Magr proposed openstack/tripleo-heat-templates master: Switch to Ansible for script download https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866799 | 20:04 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-ci master: WIP: Ensure CentOS nodes are selinux Permissive https://review.opendev.org/c/openstack/tripleo-ci/+/866806 | 21:09 |
| opendevreview | Oliver Walsh proposed openstack/tripleo-ansible master: Generalize ceph_spec_bootstrap inventory parsing https://review.opendev.org/c/openstack/tripleo-ansible/+/866410 | 21:22 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-ci master: Ensure CentOS nodes are selinux Permissive https://review.opendev.org/c/openstack/tripleo-ci/+/866806 | 21:34 |
| rlandy|rover | Tengu: owalsh: ^^ trying this for https://bugs.launchpad.net/tripleo/+bug/1998954 - may be the wrong approach to change the selinux state in CI | 21:37 |
| rlandy|rover | but it does work in tp | 21:37 |
| opendevreview | David Wilde proposed openstack/tripleo-heat-templates master: Remove external from KeystoneAuthMethods https://review.opendev.org/c/openstack/tripleo-heat-templates/+/864956 | 21:38 |
| opendevreview | Oliver Walsh proposed openstack/tripleo-common master: WIP: Stop overwriting undercloud host vars when merging inventory https://review.opendev.org/c/openstack/tripleo-common/+/866809 | 22:04 |
| opendevreview | Merged openstack/tripleo-heat-templates stable/zed: Remove no longer used libvirt log hostpaths https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866020 | 22:05 |
| opendevreview | Oliver Walsh proposed openstack/python-tripleoclient master: Use mergable inventory format for openstack tripleo deploy https://review.opendev.org/c/openstack/python-tripleoclient/+/866447 | 22:06 |
| *** dasm is now known as dasm|off | 22:51 | |
| *** rlandy|rover is now known as rlandy|out | 23:01 | |
| opendevreview | Merged openstack/tripleo-heat-templates stable/zed: Fix logs permissions for swtpm https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866015 | 23:02 |
| opendevreview | Merged openstack/puppet-tripleo stable/zed: Support deploying additional internal glance-api service https://review.opendev.org/c/openstack/puppet-tripleo/+/865874 | 23:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!