Wednesday, 2021-10-06

opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations master: oslo-config-validator: Removing merged patches  https://review.opendev.org/c/openstack/tripleo-validations/+/81242200:36
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations master: oslo-config-validator: Removing merged patches  https://review.opendev.org/c/openstack/tripleo-validations/+/81242201:14
opendevreviewMerged openstack/tripleo-heat-templates stable/ussuri: Enable keystone_authtoken/memcach_use_advanced_pool for Sahara  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80690801:39
*** pmannidi is now known as pmannidi|brb03:07
*** pmannidi|brb is now known as pmannidi03:18
opendevreviewMerged openstack/puppet-tripleo master: Revert Horizon's Memecached config to use IP addresses  https://review.opendev.org/c/openstack/puppet-tripleo/+/81234904:04
opendevreviewMerged openstack/os-net-config stable/train: Updating sriov_config.service to run before driverctl.slice and network-pre  https://review.opendev.org/c/openstack/os-net-config/+/81178004:16
opendevreviewMerged openstack/tripleo-common master: Revert "Allow for OCI manifests"  https://review.opendev.org/c/openstack/tripleo-common/+/81244104:16
*** ykarel|away is now known as ykarel04:18
*** pmannidi is now known as pmannidi|AFK04:23
*** pojadhav- is now known as pojadhav04:43
*** ysandeep|out is now known as ysandeep05:27
*** pmannidi|AFK is now known as pmannidi05:27
opendevreviewSandeep Yadav proposed openstack/tripleo-quickstart master: [WIP] centos 9 periodic release file  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80691006:02
opendevreviewSandeep Yadav proposed openstack/tripleo-quickstart master: [WIP] centos 9 periodic release file  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80691006:14
opendevreviewBrendan Shephard proposed openstack/tripleo-common master: Remove six library  https://review.opendev.org/c/openstack/tripleo-common/+/81263106:18
Tenguhjensas: heya! starting my env over (trashed it yesterday). Saw your comment on the change request, I'd say we're one step further..06:37
opendevreviewJiri Podivin proposed openstack/tripleo-validations master: Multinode and provider jobs  https://review.opendev.org/c/openstack/tripleo-validations/+/81210506:59
opendevreviewBrendan Shephard proposed openstack/tripleo-common master: Remove six library  https://review.opendev.org/c/openstack/tripleo-common/+/81263107:00
opendevreviewBrendan Shephard proposed openstack/tripleo-heat-templates master: Remove six library  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81263507:02
opendevreviewJuan Badia Payno proposed openstack/tripleo-ansible master: bnr Grouped tasks in a loop and added OUTPUT_PREFIX_PXE  https://review.opendev.org/c/openstack/tripleo-ansible/+/81195307:28
*** jpena|off is now known as jpena07:29
opendevreviewJiri Podivin proposed openstack/tripleo-validations master: Multinode and provider jobs  https://review.opendev.org/c/openstack/tripleo-validations/+/81210507:31
opendevreviewJuan Badia Payno proposed openstack/tripleo-ansible master: bnr Grouped tasks in a loop and added OUTPUT_PREFIX_PXE  https://review.opendev.org/c/openstack/tripleo-ansible/+/81195307:31
opendevreviewVijayalakshmi proposed openstack/os-net-config stable/ussuri: Update sriov config service to handle nic partitioned PF  https://review.opendev.org/c/openstack/os-net-config/+/81247507:32
Tenguhjensas: hmm. hitting the issue for haproxy-external-cert - have to double-check the generated content.07:42
*** amoralej|off is now known as amoralej07:42
Tenguok. so that's yet another file we didn't edit until now. yay.07:45
opendevreviewmbu proposed openstack/validations-libs master: Expose skip list mechanism via the CLI  https://review.opendev.org/c/openstack/validations-libs/+/80928907:49
opendevreviewRabi Mishra proposed openstack/tripleo-quickstart master: Cleanup minimal/fs001/fs002 featuresets  https://review.opendev.org/c/openstack/tripleo-quickstart/+/81263807:54
opendevreviewMarios Andreou proposed openstack/diskimage-builder master: Correct path for CentOS 9 stream base image  https://review.opendev.org/c/openstack/diskimage-builder/+/80681907:54
opendevreviewSandeep Yadav proposed openstack/tripleo-quickstart-extras master: Fix jenkins ssl issue  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81263907:55
opendevreviewJiri Podivin proposed openstack/tripleo-validations master: Multinode and provider jobs  https://review.opendev.org/c/openstack/tripleo-validations/+/81210507:59
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates stable/train: [TRAIN-ONLY] Restart ceph-nfs after the Ceph Upgrade stage  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81252608:05
*** ykarel is now known as ykarel|lunch08:15
Tenguoh.......08:16
TenguI made a typo in my oc0-domain.yaml XD08:16
Tenguhjensas: trashing and re-creating.08:16
opendevreviewTakashi Kajinami proposed openstack/tripleo-heat-templates stable/train: Enable keystone_authtoken/memcach_use_advanced_pool for Sahara  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81264008:17
opendevreviewMerged openstack/puppet-tripleo stable/wallaby: Add a new bind-mount for vTPM logs  https://review.opendev.org/c/openstack/puppet-tripleo/+/81243008:20
opendevreviewMerged openstack/python-tripleoclient stable/ussuri: Remove ipxe_enabled ppc64le validation  https://review.opendev.org/c/openstack/python-tripleoclient/+/81183008:23
opendevreviewMarios Andreou proposed openstack/tripleo-ci master: [wip] Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/tripleo-ci/+/80817708:27
*** ysandeep is now known as ysandeep|lunch08:28
opendevreviewMerged openstack/python-tripleoclient stable/train: Remove ipxe_enabled ppc64le validation  https://review.opendev.org/c/openstack/python-tripleoclient/+/81183108:48
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: placement: Adding Debug parameter  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81244408:48
opendevreviewMerged openstack/tripleo-heat-templates master: Implement a tool for converting policy.yaml files to heat templates  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/78157108:48
opendevreviewMerged openstack/tripleo-common stable/wallaby: Fix awk expression  https://review.opendev.org/c/openstack/tripleo-common/+/81232908:49
opendevreviewMerged openstack/tripleo-common stable/victoria: Fix awk expression  https://review.opendev.org/c/openstack/tripleo-common/+/81237008:49
opendevreviewMerged openstack/tripleo-common stable/ussuri: Fix awk expression  https://review.opendev.org/c/openstack/tripleo-common/+/81237108:49
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Changing default tuned profile for ComputeVdpa  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81115808:49
bogdandoykarel|lunch, ysandeep|lunch, weshay: hi, JFYI https://review.opendev.org/c/openstack/tripleo-ci/+/810672 and its topic is somewhat related to https://bugs.launchpad.net/tripleo/+bug/1895248/comments/608:53
opendevreviewBrendan Shephard proposed openstack/tripleo-common master: Remove six library  https://review.opendev.org/c/openstack/tripleo-common/+/81263108:54
bogdandowe should come up with whatever tweaking approach for workers counts that works the best08:54
bogdandoor we'll keep receiving similar bugs08:54
bogdandocgoncalves: ^^08:55
bogdandojohnsom: ^^08:55
bogdandoade_lee: ^^09:09
opendevreviewRabi Mishra proposed openstack/tripleo-quickstart master: Provision networks before overcloud deploy  https://review.opendev.org/c/openstack/tripleo-quickstart/+/81242309:17
opendevreviewRabi Mishra proposed openstack/tripleo-quickstart master: Cleanup minimal/fs001/fs002 featuresets  https://review.opendev.org/c/openstack/tripleo-quickstart/+/81263809:17
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates master: DNM/DNR WIP Switch to rabbitmq fqdn hostnames and pass proper CA info  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81239009:29
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240109:30
hjensasTengu: I wonder if we should revert this https://opendev.org/openstack/tripleo-heat-templates/commit/f708ab7a827cc0db211b4709447f77126087347e and try just that.09:30
Tenguhjensas: ok. so. in my case, I still get a missing host "overcloud0.mydomain.tld", preventing to get a certificate for the CloudNameExternalCloud0 fqdn. I suspect something is defaulting on overcloud0.ctlplane.mydomain.tld somewhere when we create the host in freeIPA. I'm digging.09:30
Tenguhjensas: hmmm... it will do what we're doing aleady in your patch, more or less. But won't be sufficient, since we're also affecting some other files.09:32
hjensasTengu: I think reverting ^^ would fix your initial issue with service_net_map_replace. And I think using name_lower.replace('_', '') is'nt a problem. 09:32
Tenguhjensas: we can try, and rebase the WIP one onto the revert?09:33
hjensasTengu: I think removing the underscores has nothing to do with it being used in a fqdn. It's just to make this split work https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/ansible_plugins/filter/service_metadata.py#L7209:33
Tenguuho09:33
Tenguok09:33
Tenguhjensas: but I don't think this will correct the IPA host creation with the wrong name... ? 09:34
opendevreviewAnanya proposed openstack/tripleo-ci-health-queries master: removing "Nova failure" from no valid host was found description  https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/80957709:35
opendevreviewHarald Jensås proposed openstack/tripleo-heat-templates master: Revert "krb-service-principals support service_net_map_replace"  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81256109:36
Tenguhjensas: if you want to hop on the env, same connection data as yesterday :)09:37
Tenguhjensas: will you rebase the other WIP thing? or do you want to test as-is first?09:39
opendevreviewMerged openstack/tripleo-ansible master: Align tasks between config-download playbooks  https://review.opendev.org/c/openstack/tripleo-ansible/+/80361209:40
opendevreviewMerged openstack/tripleo-ansible master: bnr - Reduce the default parameters of the /etc/rear/local.conf  https://review.opendev.org/c/openstack/tripleo-ansible/+/80920309:40
*** ykarel|lunch is now known as ykarel09:45
hjensasTengu: having some trouble rebasing. I will squash the revert in instead.09:45
Tenguhjensas: yeah, just saw that. Lemme known when ready, and I'll pull it on my env.09:45
TenguI've clean freeIPA content and removed the overcloud.09:45
*** ysandeep|lunch is now known as ysandeep09:46
opendevreviewHarald Jensås proposed openstack/tripleo-heat-templates master: WIP - Fix TLS-e with custom network names  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81228609:50
hjensasTengu: Let's try ^ and see how it goes.09:51
Tenguhjensas: ok. lemme get it and push to the env.09:51
Tenguhjensas: deploy started. let's see.09:53
Tenguyou can join the env. I'm about to get lunch.09:53
Tenguhjensas: ah, and thanks to mwhahaha we know why the freeIPA cleaning isn't working anymore.09:54
Tengubbl - lunch09:54
hjensasTengu: ack09:55
*** hjensas is now known as hjensas|lunch09:58
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP drop non-bundle manila  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81265410:02
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: WIP drop non-bundle manila  https://review.opendev.org/c/openstack/puppet-tripleo/+/81265510:04
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: WIP drop resource_restart_flag  https://review.opendev.org/c/openstack/puppet-tripleo/+/81265610:04
opendevreviewMerged openstack/tripleo-ansible master: Remove mistral  https://review.opendev.org/c/openstack/tripleo-ansible/+/80946810:21
opendevreviewMerged openstack/tripleo-ansible master: Remove zaqar  https://review.opendev.org/c/openstack/tripleo-ansible/+/80946910:21
opendevreviewMerged openstack/tripleo-ansible master: Fix octavia deployment tasks for FIPS environment  https://review.opendev.org/c/openstack/tripleo-ansible/+/80917410:21
opendevreviewMerged openstack/tripleo-ansible master: Fix tripleo_nova_image_cache to not use md5 for validation  https://review.opendev.org/c/openstack/tripleo-ansible/+/81078310:21
opendevreviewDamien Ciabrini proposed openstack/puppet-tripleo master: WIP haproxy config to favour local backends  https://review.opendev.org/c/openstack/puppet-tripleo/+/80523010:24
opendevreviewMerged openstack/tripleo-ansible stable/ussuri: Make Galera desync operation more resilient  https://review.opendev.org/c/openstack/tripleo-ansible/+/81232610:28
opendevreviewMerged openstack/tripleo-ansible master: bnr restore wirh Ironic  https://review.opendev.org/c/openstack/tripleo-ansible/+/81172610:28
*** rlandy is now known as rlandy|ruck10:31
opendevreviewRabi Mishra proposed openstack/tripleo-ansible stable/wallaby: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256210:36
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Remove mistral  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81119410:36
opendevreviewRabi Mishra proposed openstack/tripleo-ansible stable/victoria: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256310:36
opendevreviewRabi Mishra proposed openstack/tripleo-ansible stable/ussuri: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256410:36
opendevreviewMerged openstack/puppet-tripleo stable/wallaby: Remove mistral  https://review.opendev.org/c/openstack/puppet-tripleo/+/81119810:37
ysandeepbogdando: thanks! o/ will check 10:39
opendevreviewRabi Mishra proposed openstack/tripleo-ansible stable/train: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81266010:40
opendevreviewVijayalakshmi proposed openstack/os-net-config stable/victoria: Update sriov config service to handle nic partitioned PF  https://review.opendev.org/c/openstack/os-net-config/+/81235910:45
*** ysandeep is now known as ysandeep|away11:00
Tenguhjensas: uho. it seems to be far, far further!!11:00
hjensasTengu: yes, looking ok so far I see. Unless it now times out puppet host config11:01
Tenguapparently. let's see.11:01
ykarelbogdando, ack11:03
Tenguhumpf... don't we have any puppet logs ?!11:04
hjensasTengu: in the log/containers/stdouts dir?11:05
Tenguhmmm..... we're on the host config, not container11:05
Tengubingo11:05
Tenguno container.11:05
Tenguwell.... errr..... that's a bit empty.11:08
Tenguhjensas: seems to block on pacemaker? though I didn't include HA afaik11:09
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations master: oslo-config-validator: container plugin is replaced with collections  https://review.opendev.org/c/openstack/tripleo-validations/+/80419311:09
*** pojadhav is now known as pojadhav|brb11:11
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations master: oslo-config-validator: invalidating configs only when necessary  https://review.opendev.org/c/openstack/tripleo-validations/+/81049111:11
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations master: oslo-config-validator: Removing merged patches  https://review.opendev.org/c/openstack/tripleo-validations/+/81242211:12
Tenguhmm.... as if the play didn't send anything back stating "hey, I'm done".11:13
Tengunot sure if "puppet-user[49031]: error: Could not connect to controller: Connection refused" is the actual error. Might be. though.... no puppetmaster, so I'm a bit surprised.11:15
hjensasTengu: I have no idea ... there is also ipa-submit[25452]: JSON-RPC error: 4001: The host 'overcloud0.mydomain.tld' does not exist to add a service to.11:18
Tenguthat was a previous run11:18
Tenguthat host now exists, else we wouldn't go that far.11:18
hjensasTengu: ah, ok.11:19
* Tengu is stuck11:20
*** pojadhav|brb is now known as pojadhav11:21
Tenguhjensas: hmmm there's mention of a container... ansible-container_startup_config Invoked with config_base_dir=/var/lib/tripleo-config/container-startup-config  but.. that's weird.11:24
Tengubut yeah. no log. nothing.11:25
* Tengu relaunch thing11:25
cloudnullgood mornings all 11:25
Tenguhey cloudnull :)11:26
rlandy|ruckgood morning11:26
cloudnullhows it ?11:27
Tengucloudnull: hitting walls at almost every step ^^'11:28
cloudnullTengu sounds productive :D 11:28
opendevreviewMerged openstack/tripleo-ci-health-queries master: removing "Nova failure" from no valid host was found description  https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/80957711:28
Tengucloudnull: mostly painful ;)11:28
Tenguhjensas: if it fails once more, I'll clear the whole thing up once more, just to be sure I don't have some weird dangling things in the way - though I doubt it, we went far past the initial certmonger issues.11:29
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations stable/wallaby: Use collections in validations for Ansible 2.11  https://review.opendev.org/c/openstack/tripleo-validations/+/81256511:32
opendevreviewDavid Vallee Delisle proposed openstack/tripleo-validations stable/wallaby: Use collections in validations for Ansible 2.11  https://review.opendev.org/c/openstack/tripleo-validations/+/81256511:34
Tengugrumpf.11:41
*** dviroel|out is now known as dviroel11:41
opendevreviewMerged openstack/diskimage-builder master: Drop lower version requirement for networkx  https://review.opendev.org/c/openstack/diskimage-builder/+/81245311:42
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240111:48
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates master: DNM/DNR WIP Switch to rabbitmq fqdn hostnames and pass proper CA info  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81239011:49
opendevreviewMerged openstack/puppet-tripleo master: haproxy: Add missing default_backend option  https://review.opendev.org/c/openstack/puppet-tripleo/+/81245711:51
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo stable/wallaby: haproxy: Add missing default_backend option  https://review.opendev.org/c/openstack/puppet-tripleo/+/81256711:52
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates master: Fix swift "Could not enable rsyncd" error  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81230711:52
*** amoralej is now known as amoralej|lunch11:55
*** jpena is now known as jpena|lunch12:00
opendevreviewDouglas Viroel proposed openstack/tripleo-quickstart master: Add 'generic' repos setup using yum_config module  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80840812:40
opendevreviewDouglas Viroel proposed openstack/tripleo-quickstart master: Enable use of yum-config module in wallaby  https://review.opendev.org/c/openstack/tripleo-quickstart/+/81267212:40
*** ysandeep|away is now known as ysandeep12:44
gfidenterlandy|ruck the other day you mentioned adding scenario004 to changes for lvmfilter in tripleo-ansible12:58
gfidentebut I can't find that submission anymore :(12:58
rlandy|ruckgfidente: getting12:58
gfidenterlandy++ thanks, sorry!12:58
rlandy|ruckgfidente: no problem - https://github.com/openstack/tripleo-ci/commit/443cff5d1cd46f3412fac532e62adaf1a577a16112:59
gfidenteah so it's merged already!12:59
gfidentedah, thanks!12:59
lbragstadcloudnull Tengu mwhahaha thanks for the reviews on the enable-secure-rbac.yaml patch13:01
lbragstadthese are some quick follow-on patches that have +1s from their respective service team members13:02
lbragstadhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/810466 (nova)13:02
lbragstadhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/810458 (designate)13:02
lbragstadhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/810456 (ironic)13:03
lbragstadhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/810455 (octavia)13:03
Tenguhjensas: ok. sooooo... yeah. that puppet host configuration step_1 is failing for unknown reason.13:04
Tenguyay.13:04
* Tengu flips some tables and grunts13:04
opendevreviewGiulio Fidente proposed openstack/tripleo-ansible master: Allow LVM filter to work when either allowlist or denylist are empty  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256813:07
opendevreviewGael Chamoulaud proposed openstack/tripleo-validations stable/wallaby: [validation_init] add new metadata keys in the playbook template  https://review.opendev.org/c/openstack/tripleo-validations/+/80799513:08
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240113:12
opendevreviewLance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 10 to deploy with secure RBAC's project personas  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80974913:12
opendevreviewLance Bragstad proposed openstack/tripleo-quickstart master: Update featureset 20 to use secure RBAC project personas  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80981113:12
opendevreviewGael Chamoulaud proposed openstack/tripleo-validations stable/wallaby: remove using containers-prepare-parameter.yaml  https://review.opendev.org/c/openstack/tripleo-validations/+/80689613:12
lbragstadrlandy|ruck (for when you're free from meetings) ^13:12
lbragstadcc dviroel ^13:12
dviroellbragstad: thanks13:12
opendevreviewGiulio Fidente proposed openstack/tripleo-ansible master: Allow LVM filter to work when either allowlist or denylist are empty  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256813:13
dviroelysandeep: rbac updates ^13:13
opendevreviewMerged openstack/tripleo-ansible stable/victoria: Make Galera desync operation more resilient  https://review.opendev.org/c/openstack/tripleo-ansible/+/81232513:13
Tengucloudnull: does this error ring any bells? "error: Could not connect to controller: Connection refused"  (it's comming from puppet)13:13
ysandeepdviroel, lbragstad ack thanks!13:14
cloudnullTengu no bells from me 13:14
cloudnullwhat is generating that?13:15
Tengucloudnull: no real idea so far.... it's on a tls-e with freeipa and custom networks. It's showing up during the "puppet host configuration". and that task is ending with a nice timeout.13:16
Tenguit's on a controller.13:16
Tengucloudnull: https://paste.openstack.org/show/809818/ that's ALL I can get out of the logs (for now)13:16
*** amoralej|lunch is now known as amoralej13:17
*** jcapitao1 is now known as jcapitao13:19
cloudnullmaybe a red herring but Warning: Unknown variable seems sus 13:19
Tenguyeah.... there's also that. but no actual puppet error :/.13:20
TenguI'm adding some more debug (Debug: true, ConfigDebug: true) and will restart things.13:20
opendevreviewMerged openstack/diskimage-builder master: Fix cron not installed in debian  https://review.opendev.org/c/openstack/diskimage-builder/+/80699013:23
*** jpena|lunch is now known as jpena13:24
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: WIP Remove stack_update check in rabbit manifest  https://review.opendev.org/c/openstack/puppet-tripleo/+/81268213:41
opendevreviewHaresh proposed openstack/tripleo-common master: add kernel-modules-extra in overcloud image needed for HWOL  https://review.opendev.org/c/openstack/tripleo-common/+/81268313:43
Tenguhjensas: so, we can not see what's taking so long: /sbin/pcs host auth oc0-controller-0 -u hacluster -p <PASSWORD>13:45
Tenguit's looping and failing.13:45
Tengunow the question is: why.13:45
opendevreviewmbu proposed openstack/validations-libs master: Expose skip list mechanism via the CLI  https://review.opendev.org/c/openstack/validations-libs/+/80928913:45
Tengu.... also, why do I have pacemaker...13:46
Tengunon-sense13:46
Tengubandini: heya! is pacemaker installed by default on a 1-controller OC?13:46
mwhahahayes13:46
mwhahahait's always enabled13:46
bandiniyes13:47
bandinisince ussuri13:47
Tenguperfect...13:47
Tenguso it's crashing my deploy :)13:47
opendevreviewMerged openstack/tripleo-ci master: puppet-tripleo: Run sc001/002 for changes in redis manifests  https://review.opendev.org/c/openstack/tripleo-ci/+/81246213:48
opendevreviewMerged openstack/tripleo-validations master: Use collections in validations for Ansible 2.11  https://review.opendev.org/c/openstack/tripleo-validations/+/80676313:48
mwhahaharlandy|ruck: scneario001 on master is still broke after the oci patch revert13:48
mwhahahahttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario001-standalone13:48
rlandy|rucklooking13:49
Tengubandini: what might make this loop endless? https://paste.openstack.org/show/809821/13:49
rlandy|ruck"Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'internal_api' (file: /etc/puppet/modules/tripleo/manifests/haproxy.pp, line: 827) on node standalone.localdomain"13:49
rlandy|ruckdifferent error13:50
mwhahahai've also seen gnonchi and step3 hangs13:50
rlandy|ruckgnochi fix should have merged 13:50
mwhahahaok then just step3 hang was the latest fail13:50
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/1946045/13:51
Tenguah..... corosync can't start. no config file. what the hell13:51
bandiniTengu: run it by hand and see what it says13:51
rlandy|ruckmwhahaha: https://review.opendev.org/c/openstack/puppet-tripleo/+/81247713:51
bandinino corosync will be configured later13:51
mwhahahafun13:51
rlandy|ruckhttps://review.opendev.org/c/openstack/puppet-tripleo/+/81245713:51
rlandy|ruckmergd13:51
Tengubandini: err... so pacemaker isn't started..13:52
Tengubandini: the pcs command returns:  Error: Unable to communicate with oc0-controller-013:52
Tengusounds legit13:52
Tengusince no pacemaker13:52
bandiniit only needs pcsd to be up13:52
rlandy|ruckshould have merged this morning13:52
Tenguwokay. so that one is up13:52
mwhahaharlandy|ruck: k i'll try and propose additional coverage in puppet-tripleo later today13:53
rlandy|ruckmwhahaha: k - but we still have an error13:53
mwhahahayea i can look into that as well13:53
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/812506/13:54
rlandy|ruckmwhahaha: ^^ your coverage patch is in recheck now13:54
* rlandy|ruck looks if we need a new bug13:54
rlandy|rucktripleo-ci-centos-8-scenario001-standaloneopenstack/puppet-tripleomastergate812457,313:55
rlandy|ruck1 hr 34 mins 33 secs2021-10-06 10:03:25SUCCESS13:55
rlandy|ruckthat looks good13:55
mwhahahawe shall see13:55
rlandy|ruckk - thanks for the heads up - will watch it13:56
rlandy|ruckso much fun - with multiple failures on the same job13:56
mwhahahadat lack o' coverage13:57
rlandy|ruckthe problem with editing templates for coverage is that you don't see the results until merge13:58
mwhahahayup13:58
Tenguhjensas: want to know a good news? we might need to re-re-re-work the /etc/hosts generation horrible thing. Again.14:03
* mwhahaha not it14:03
Tengushort story: the "shortname" wasn't associated to the (custom) internalapi host.14:04
Tenguand since pacemaker uses shortnames for historical reason, while listening on the internalapi network, we must ensure we're associating that shortname properly.14:05
Tenguin my case, it was associated to the external (CloudNameExternalCloud0)14:05
mwhahahathere was a bug around the shortname ending up on the wrong network14:06
Tenguthere were many of them, but mostly for the UC14:06
Tenguwhere the shortname is, indeed, associated to the internalapi, but the UC has no access to that network, so ppl wanted to get it associated to the ctlplane14:06
Tenguit was nop'ed14:06
mwhahahayea the uc hosts file likely needs to be different than the oc one14:08
Tenguyup14:09
Tengustill.... have to have a new look at that terrible /etc/hosts generation14:09
Tenguand I'm scared.14:09
mwhahahaenjoy14:09
* mwhahaha ninja vanishes14:09
* Tengu shows up his GPS tracker14:10
opendevreviewMerged openstack/tripleo-heat-templates stable/victoria: Add SendQueueLimit to collectd-amqp1  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/76110314:10
TenguI know where you're at, mwhahaha :)14:10
mwhahahaa likely story14:10
* mwhahaha goes to live in the mountains14:10
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Check for passwords at old path during tripleo deploy  https://review.opendev.org/c/openstack/python-tripleoclient/+/81252914:10
*** pojadhav is now known as pojadhav|afk14:13
Tenguhjensas: pff. I'm lost with that thing... would you have a moment for a "quick" call in order to work on that?14:18
opendevreviewTakashi Kajinami proposed openstack/puppet-tripleo stable/wallaby: haproxy: Use healthcheck middleware to monitor service availability  https://review.opendev.org/c/openstack/puppet-tripleo/+/81017914:23
opendevreviewTakashi Kajinami proposed openstack/puppet-tripleo stable/wallaby: haproxy: Use healthcheck API for Ironic and Octavia  https://review.opendev.org/c/openstack/puppet-tripleo/+/81018014:23
opendevreviewTakashi Kajinami proposed openstack/puppet-tripleo stable/wallaby: haproxy: Use healthcheck middleware to monitor service availability  https://review.opendev.org/c/openstack/puppet-tripleo/+/81017914:26
opendevreviewTakashi Kajinami proposed openstack/puppet-tripleo stable/wallaby: haproxy: Use healthcheck API for Ironic and Octavia  https://review.opendev.org/c/openstack/puppet-tripleo/+/81018014:26
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240114:30
opendevreviewJulia Kreger proposed openstack/tripleo-common stable/wallaby: Remove tftp-server package from ironic-pxe container  https://review.opendev.org/c/openstack/tripleo-common/+/81269014:35
Tenguhjensas: after some more git-grep'ing, it seems the issue is deeper than the generation itself.14:39
Tengufor some reason, all of the HostnameResolveNetwork point to the "ctlplane" network in my env. afaik it should be internal_api_cloud_0 or something like that..14:40
opendevreviewLance Bragstad proposed openstack/tripleo-heat-templates master: Add a note about how glance currently supports project personas  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81043814:41
*** dviroel is now known as dviroel|afk14:41
lbragstadTengu cloudnull mwhahaha ^ updated glance patch for project personas, which is straight forward, too14:42
Tenguhjensas: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/overcloud-resource-registry-puppet.j2.yaml#L430-L448  that probably also needs some love.14:42
Tenguyeah, that one isn't affected by the WIP patch14:43
Tengubandini: -^^ fyi  that's the reason I think: hard-coded network names that won't match custom networks.14:43
Tenguysandeep: -^^  you'll probably face that kind of issue, even without TLS-e14:44
Tenguysandeep: basically "shortname" not matching the right network on the overcloud. yay14:44
Tenguat least.... there's probably no need to rework the /etc/hosts generation code.14:45
Tengu.... though I'm not sure that other thing is better to edit -.-'14:45
mwhahaharlandy|ruck: jfyi i went looking and scenario001 runs when haproxy.pp changes happen so i have no idea how it suddenly broke14:49
rlandy|ruckchecking scenario001 on master14:50
rlandy|ruckwe had one gate pass14:50
rlandy|ruckdoesn't mean we done here14:51
mwhahahak14:51
Tenguhjensas, ysandeep fyi: https://bugs.launchpad.net/tripleo/+bug/194623915:00
Tengumwhahaha: -^^  if you want to follow ;)15:00
mwhahahafun15:00
opendevreviewCedric Jeanneret proposed openstack/tripleo-heat-templates master: WIP - Fix TLS-e with custom network names  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81228615:14
Tenguhjensas: -^^   sounds to be the right thing to do.15:16
Tengu(removing nested IF from the overcloud-resource-registry-puppet.j2.yaml)15:16
Tenguw00t! seems to work!15:18
Tenguwaiting for the next failure now.15:18
Tenguramishra: so, regarding network names, nope, they aren't fixed, and tripleo-lab uses this feature for proper multi-overcloud support. And, in fact, this feature is broken for some time now, and I'm seeing it "only" now that I want to really get TLS-e with freeIPA. Now, whether using custom network names is a good idea or not, that's another story ;).15:29
Tenguthere's another location where we namely check for "default" network names - overcloud.j2.yaml - but it doesn't have any bad consequence because there's a safe "else" allowing to catch everything.15:31
Tenguhjensas: my env reached "external deployment step_4" before crashing! It's getting better :)15:32
Tenguo_O there's really something fishy with the ctlplane vs public network...15:34
*** ykarel is now known as ykarel|away15:38
Tengunow the generated haproxy config seems to mix ctlplane and public, and is configured to listen on the ctlplane instead of public in the "public_keystone" frontend.15:38
Tengushould listen on "external". probably something with some hardcoded name, again.15:39
Tenguenvironments/network-environment.j2.yaml:{%-     if network.name == 'External' %}   guess we're really not helping ourselves.15:40
ramishraTengu: AFAIK you can't change the 'name' and it's not even supported.. see the docs for multicloud https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/multiple_overclouds.html#deploying-additional-overclouds15:41
mwhahahawell the service_net_map_repalce was supposed to allow for name changes15:42
Tenguramishra: https://slagle.fedorapeople.org/tripleo-docs/install/advanced_deployment/custom_networks.html15:42
ramishrayes it does15:42
mwhahahaso that Foo would be replaced where we use internal_api but we aren't using that consistently i guess15:42
Tengubut it doesn't work :)15:42
Tengumwhahaha: even worst, but yeah, that's the idea.15:42
Tenguand the service_net_map_replace was "just" in order to avoid rewriting the whole ServiceNetMap thing15:43
mwhahahaits kinda like our old Controller role where you couldn't have something named otherwise but we addressed it by having a 'controller' tag to allow us to lookup the correct role name15:43
Tenguexactly15:43
*** jpena is now known as jpena|off15:43
Tengubut for network, you also have the name_lower that might not match the name.lower() :D15:43
Tenguit's...15:44
mwhahahanetworks really need to express their intended use were clouds have to have a ctlplane/internal/external maybe?15:44
Tengucrazy.15:44
* mwhahaha runs away from this one15:44
Tengumwhahaha: issue is: we're using the actual "network.name" in the role-data15:44
*** jpena|off is now known as jpena15:44
Tenguso it's a bit more complicated to do the "math" with the service_net_map_replace15:44
Tenguramishra: that said, a clean solution would be to ensure those default names are present in the network-data, and crash the deploy if they aren't (with a clear statement). for instance, tripleo-lab is using custom names (you even saw it when you added support for network-v2) and it was mostly working or, at least, not crashing.15:47
Tengu.. maybe it's a PTG topic...15:47
Tengufor "default" usually means "you can do something else".15:48
opendevreviewRonelle Landy proposed openstack/tripleo-ci master: DNM - testing scenario001  https://review.opendev.org/c/openstack/tripleo-ci/+/81272215:48
Tenguanyway... getting late in here. guess I'll discuss with hjensas tomorrow.15:48
Tengu<- off.15:49
*** chem is now known as Guest200015:51
*** ysandeep is now known as ysandeep|out16:15
slagleTengu: that docs link you pasted from my fedorapeople is likely ancient :)16:36
shyambmwhahaha: Hi16:49
shyambDo we have any other approach to register service in keystone https://github.com/trilioData/triliovault-cfg-scripts/blob/master/redhat-director-scripts/rhosp16.1/services/trilio-datamover-api.yaml#L166-L17616:49
shyambservice and endpoints.16:49
shyambI mean haproxy endpoints16:50
mwhahahai don't think so16:51
shyambmwhahaha: Okay.16:51
shyambWith this approach, haproxy pacemaker resource is getting restarted16:51
shyamband sometimes it's not coming up. getting in FAILED state16:52
shyambCausing deployment to fail.16:52
opendevreviewMerged openstack/os-net-config stable/victoria: sriov_config: light refactoring  https://review.opendev.org/c/openstack/os-net-config/+/81235816:54
mwhahahasounds like a bug, it shouldn't keep restarting it16:55
shyambmwhahaha: Okay.16:56
shyambI can open bug.16:57
shyambThis was not the case earlier. We are facing this issue from last 3-4 months16:57
opendevreviewBrent Eagles proposed openstack/tripleo-ansible master: Don't try to set unbound address when already set  https://review.opendev.org/c/openstack/tripleo-ansible/+/81273116:59
opendevreviewBrent Eagles proposed openstack/tripleo-ansible master: Don't try to set unbound address when already set  https://review.opendev.org/c/openstack/tripleo-ansible/+/81273117:00
*** jpena is now known as jpena|off17:00
opendevreviewBrent Eagles proposed openstack/tripleo-ansible master: Don't try to set unbound address when already set  https://review.opendev.org/c/openstack/tripleo-ansible/+/81273117:01
*** amoralej is now known as amoralej|off17:01
hjensasTengu: mwhahaha: ramishra: Been reading the scrollback. The intended use of a network is in the ServiceNetMap. I think the issue is we are trying to use all custom networks without also passing a custom ServiceNetMap, i.e using service_net_map_replace. Unfortunately service_net_map_replace only takes us 80% there. I think atleast the role specific entries and the redis/ovn_dbs VIP entries must be manually set in ServiceNetMap - I put 17:03
hjensasa comment on, https://review.opendev.org/c/openstack/tripleo-heat-templates/+/812286/6/overcloud-resource-registry-puppet.j2.yaml#b440. Will catch up with Tengu tomorrow.17:03
mwhahahatha tbit of code is terrible because of stf, we should probably just revise that17:06
mwhahahaand ceph i think because of the storage role17:06
mwhahahaif resolvenetwork == short hostname, we need to understand which services depend only on thsoe and why 17:07
lbragstaddoes tripleo have an http extra configuration variable for particular services?17:20
shyambmwhahaha: opened this bug: https://bugs.launchpad.net/tripleo/+bug/194625217:23
hjensasmwhahaha: yeah, if we could get everything to use fqdn we likely would'nt need the resolvnetwork.17:24
hjensasmwhahaha: If we add a option in role data, for resolve network and qdr_metrics_network it might be cleaner.17:25
hjensas    {{role.name}}HostnameResolveNetwork: {{ _service_nets.get(role.resolve_network, 'ctlplane') }}17:28
hjensas    {{role.name}}MetricsQdrNetwork: {{ _service_nets.get(role.qdr_metrics_network, 'ctlplane') }}17:28
*** dviroel|afk is now known as dviroel17:41
mwhahahalbragstad: what do you mean?17:51
*** ysandeep|out is now known as ysandeep18:03
opendevreviewSandeep Yadav proposed openstack/tripleo-ci master: [wip] test install pip  https://review.opendev.org/c/openstack/tripleo-ci/+/81273718:55
*** atarakt is now known as nhicher18:58
*** fmount is now known as fpantano19:04
opendevreviewDanni Shi proposed openstack/diskimage-builder master: Update keylime-agent and tpm-emulator elements  https://review.opendev.org/c/openstack/diskimage-builder/+/81025419:05
rlandy|ruckmwhahaha: scenario001 is doing better now ... https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario001-standalone19:14
mwhahahacool19:14
rlandy|ruckwe should have all associated fixes from yesterday merged19:14
rlandy|ruckexpect maybe the template one19:14
*** ysandeep is now known as ysandeep|out19:24
lbragstadmwhahaha if i wanted to supply a custom Location directive in keystone wsgi configuration, is there a way to do that via THT parameters?19:26
mwhahahano19:26
lbragstadok - i didn't see anything, but wanted to double check 19:26
lbragstadthanks19:26
mwhahahathe wsgi configs come from puppet-apache via puppet-keystone19:27
mwhahahawe don't expose extra bits for those configs19:27
lbragstadi did see this https://github.com/openstack/puppet-keystone/blob/master/manifests/wsgi/apache.pp#L16419:27
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240119:27
mwhahahayea you could try that19:27
mwhahahathough don't know if it's in or out of the vhost19:28
mwhahahahttps://github.com/openstack/puppet-openstacklib/blob/master/manifests/wsgi/apache.pp#L37719:28
mwhahahaso looks liek that can add a location19:29
mwhahahayou'd have to do a hieradata override as i don't think we expose that via heat19:29
mwhahahahttps://github.com/puppetlabs/puppetlabs-apache/blob/main/manifests/vhost.pp#L138119:29
lbragstadright - so, not a supported thing, right?19:30
mwhahahadefinately not19:30
mwhahahatesting sure, production no19:30
* lbragstad nods19:30
mwhahahanow if you put it into puppet-tripleo that'd be fine but not something we'd expose as a thing for folks to use19:31
opendevreviewMerged openstack/tripleo-quickstart-extras master: Fix jenkins ssl issue  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81263919:31
mwhahahalbragstad: what specifically are you trying to configure?19:32
lbragstadmwhahaha it's kinda similar to (or a workaround for) https://bugs.launchpad.net/keystone/+bug/180404219:33
lbragstadso - only allowing access to certain APIs based on IP19:33
mwhahahaterrible idea19:33
lbragstad:) 19:34
lbragstadyeah - keystone v2.0 allow that because admin functionality was a separate application - so you could put it where ever19:34
lbragstadand i think the user wants that same kind of strategy with v219:34
lbragstadv3*19:34
mwhahahai don't think apache is the best place to solve for that19:36
lbragstadno - probably not19:36
lbragstadit's a policy problem 19:36
opendevreviewMichele Baldessari proposed openstack/puppet-tripleo master: DNM/DNR WIP rabbitmq tls tests  https://review.opendev.org/c/openstack/puppet-tripleo/+/81240120:09
opendevreviewAlex Schultz proposed openstack/tripleo-common master: [WIP] task-core file generation  https://review.opendev.org/c/openstack/tripleo-common/+/79872120:21
opendevreviewSteve Baker proposed openstack/tripleo-quickstart-extras master: Support setting growvols_args in topology_map  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81241420:29
opendevreviewSteve Baker proposed openstack/tripleo-quickstart-extras master: Use tripleo-mount-image for modify-image  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80628420:29
opendevreviewSteve Baker proposed openstack/tripleo-quickstart-extras master: In modify-image don't convert qcow2<->raw  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80948720:29
rlandy|ruckstevebaker: hi - we can w+ https://review.opendev.org/c/openstack/tripleo-quickstart/+/801409 when the growvols patches are merged20:38
rlandy|rucktests are fine 20:38
stevebakerrlandy|ruck: ok, I just made the change Marios asked for on this one so I think its ok to +w https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81241420:38
rlandy|ruckstevebaker: yes - juts +2'ed that one again20:39
rlandy|ruckjust20:39
stevebakerrlandy|ruck: I'm on PTO from tomorrow to my monday20:39
rlandy|ruckstevebaker: k - if all is well, marios and I can merge those20:40
rlandy|ruckif there is an issue, we will email you20:40
stevebakerok thanks20:40
rlandy|ruckenjoy your time off20:40
opendevreviewRonelle Landy proposed openstack/tripleo-ci master: Edit underlying review to test comments  https://review.opendev.org/c/openstack/tripleo-ci/+/81274720:45
opendevreviewMerged openstack/tripleo-common stable/wallaby: Shrink whole disk image from 40G to 6G  https://review.opendev.org/c/openstack/tripleo-common/+/81153421:02
*** rlandy|ruck is now known as rlandy|ruck|bbl22:20
opendevreviewMerged openstack/tripleo-heat-templates master: Implement project personas in custom nova policy file  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81046622:39
opendevreviewSteve Baker proposed openstack/python-tripleoclient stable/wallaby: Add --boot-mode to import and configure  https://review.opendev.org/c/openstack/python-tripleoclient/+/81153922:59
opendevreviewSteve Baker proposed openstack/python-tripleoclient stable/wallaby: Always run growvols playbook after provision  https://review.opendev.org/c/openstack/python-tripleoclient/+/81227022:59
opendevreviewMerged openstack/tripleo-heat-templates master: Implement project personas in custom designate policy file  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81045823:04
opendevreviewMerged openstack/tripleo-heat-templates master: Enable support for Libvirt modular daemons  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80391523:04
opendevreviewMerged openstack/validations-libs master: Expose skip list mechanism via the CLI  https://review.opendev.org/c/openstack/validations-libs/+/80928923:04
opendevreviewMerged openstack/tripleo-heat-templates master: Add a note about how glance currently supports project personas  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81043823:04
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256223:04
opendevreviewMerged openstack/tripleo-ansible stable/victoria: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256323:07
opendevreviewMerged openstack/tripleo-ansible stable/ussuri: Disable TraceEnable to satisfy security scanners  https://review.opendev.org/c/openstack/tripleo-ansible/+/81256423:07
opendevreviewMerged openstack/tripleo-common stable/wallaby: Remove tftp-server package from ironic-pxe container  https://review.opendev.org/c/openstack/tripleo-common/+/81269023:07
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Don't write <stack_name>-passwords.conf  https://review.opendev.org/c/openstack/python-tripleoclient/+/81253023:13
opendevreviewMerged openstack/tripleo-common stable/wallaby: Removal mistral  https://review.opendev.org/c/openstack/tripleo-common/+/81119623:13

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!