Tuesday, 2019-12-17

cloudnullmwhahaha looking00:04
*** chem_ has quit IRC00:05
cloudnullmwhahaha remote_src would expect /usr/share/python-tripleoclient/undercloud.conf.sample to exist on the target node00:05
*** ysandeep has joined #tripleo00:06
*** chem_ has joined #tripleo00:07
*** holser has quit IRC00:07
cloudnullif the playbook is being run on what I expect is the "undercloud"  then you wouldn't need remote_src00:07
cloudnullthe remote_src flag is the same as the copy module - https://docs.ansible.com/ansible/latest/modules/copy_module.html00:08
cloudnullconfig_template is an action plugin and uses the copy module, that flag is just passed through.00:09
cloudnulldo you have the playbook example handy ?00:09
openstackgerritMerged openstack/python-tripleoclient master: TripleO Validator CLI Refactor  https://review.opendev.org/69355200:10
openstackgerritSagi Shnaidman proposed openstack/python-tripleoclient master: WIP: support standalone ansible.cfg addition and extra args  https://review.opendev.org/69913700:11
mwhahahacloudnull: it does exist00:11
cloudnullis it a local playbook ?00:12
mwhahahacloudnull: that snippet is it. Just trying to run that on a node that has tripleoclient installed00:12
mwhahahaNo just host: undercloud00:12
mwhahahaNothing else00:12
openstackgerritMerged openstack/tripleo-common master: Incorrectly derives NeutronPhysnetNUMANodesMapping  https://review.opendev.org/69853300:13
*** dsneddon has joined #tripleo00:13
cloudnullmwhahaha https://pasted.tech/pastes/91554430258ca13da847509cab91ee202b8ece1d.raw00:15
cloudnullthat works for me00:15
mwhahahacloudnull: not connection local00:15
cloudnullit raises another error because there's jinja in that sample file00:15
mwhahahaDo it remotely00:16
mwhahahaIt fails on the mtime bits cause the file is not local00:16
* cloudnull trying from my workstation now 00:17
*** dsneddon has quit IRC00:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: WIP: run ansible plugins on undercloud  https://review.opendev.org/69931400:21
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates stable/train: Optional enable undercloud nova-metadata-api  https://review.opendev.org/69859300:24
cloudnullmwhahaha running py 3?00:24
mwhahahaYea00:25
mwhahaha2.9.200:25
cloudnullwe're not trapping FileNotFoundError00:25
cloudnullwhich is new in py300:25
mwhahahaIt should not be running that code if remote src is true00:25
mwhahahaCause the fallback forces connection local00:26
cloudnullyou're right but it doesn't fall back becuase we're not trapping that error00:26
mwhahahaSeems that it doesn't properly support remote_src00:26
cloudnullwe add the fallback to use the stat module00:26
mwhahahaBut that won't work either atm00:26
*** ysandeep has quit IRC00:29
cloudnullmwhahaha https://review.opendev.org/69931500:31
cloudnullthat fixes it00:31
mwhahahaYes looks like it. Thanks00:32
mwhahahaThere's probably some improvements to be had00:33
mwhahahaBecause it should skip right to using the module if remote00:33
mwhahahaCause if you have the file locally it'll not work right00:33
mwhahahaI'll poke at it tomorrow00:33
cloudnull++00:35
cloudnullthis was the playbook I used and it seemed to be working - https://pasted.tech/pastes/0e531169324517a4d2bf4d9848c9ff3f59c4234e.raw00:37
cloudnullwill poke at it more in a bit, for now its dinner time00:37
*** dsneddon has joined #tripleo00:41
*** dsneddon has quit IRC00:50
*** rlandy|bbl is now known as rlandy00:58
openstackgerritTakashi Kajinami proposed openstack/puppet-pacemaker master: Add support to configure pcsd bind address  https://review.opendev.org/69794201:12
openstackgerritTakashi Kajinami proposed openstack/puppet-pacemaker master: DNM: Just test the parameter  https://review.opendev.org/69931701:12
openstackgerritTakashi Kajinami proposed openstack/puppet-pacemaker master: Add support to configure pcsd bind address  https://review.opendev.org/69794201:14
openstackgerritTakashi Kajinami proposed openstack/puppet-pacemaker master: DNM: Just test the parameter  https://review.opendev.org/69931701:14
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Add support to configure pcsd bind address  https://review.opendev.org/69794301:15
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Make pcsd listen on PacemakerNetwork/PacemakerRemoteNetwork  https://review.opendev.org/69931801:16
*** ysandeep has joined #tripleo01:22
*** dsneddon has joined #tripleo01:36
*** dsneddon has quit IRC01:41
openstackgerritMerged openstack/tripleo-ansible master: Update override for keystone resources  https://review.opendev.org/69928202:05
*** rlandy has quit IRC02:08
*** bnemec has quit IRC02:10
*** bnemec has joined #tripleo02:12
*** dsneddon has joined #tripleo02:15
*** dsneddon has quit IRC02:20
*** Goneri has quit IRC02:21
*** dsneddon has joined #tripleo02:30
*** dsneddon has quit IRC02:35
*** mschuppert has quit IRC02:43
*** rfolco|bbl has quit IRC02:46
*** dsneddon has joined #tripleo02:47
*** dsneddon has quit IRC02:52
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo-keystone-resources: handle multiple roles per user  https://review.opendev.org/69928602:54
openstackgerritAntony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31  https://review.opendev.org/69771302:56
*** dsneddon has joined #tripleo02:56
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/rocky: Remove configuration related to Horizon  https://review.opendev.org/69621502:59
*** dsneddon has quit IRC03:01
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698003:03
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/rocky: Remove trove firewall rule  https://review.opendev.org/69932803:03
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698003:06
*** apetrich has quit IRC03:09
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: actions/ansible: catch errors when cleaning the work directory  https://review.opendev.org/69923403:11
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Revert "Added for support for generating overcloud clouds.yaml"  https://review.opendev.org/69932903:13
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Revert "Added for support for generating overcloud clouds.yaml"  https://review.opendev.org/69932903:13
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Revert "Added support of overcloud os_cloud"  https://review.opendev.org/69920603:13
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Revert "Added support of overcloud os_cloud"  https://review.opendev.org/69920603:13
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/rocky: Remove configuration related to Horizon  https://review.opendev.org/69621503:26
*** dsneddon has joined #tripleo03:32
*** psachin has joined #tripleo03:38
*** dsneddon has quit IRC03:44
*** ykarel|away has joined #tripleo03:46
*** dsneddon has joined #tripleo04:01
*** dsneddon has quit IRC04:06
*** dsneddon has joined #tripleo04:10
openstackgerritSteve Baker proposed openstack/python-tripleoclient stable/train: Add undercloud.conf enable_nova_metadata  https://review.opendev.org/69930804:12
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM test undercloud_enable_nova_metadata  https://review.opendev.org/69832704:13
*** dsneddon has quit IRC04:15
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM test undercloud_enable_nova_metadata  https://review.opendev.org/69832704:21
openstackgerritSteve Baker proposed openstack/python-tripleoclient stable/train: Add undercloud.conf enable_nova_metadata  https://review.opendev.org/69930804:26
*** epoojad1 has joined #tripleo04:26
*** ykarel|away has quit IRC04:32
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM test undercloud_enable_nova_metadata  https://review.opendev.org/69832704:36
*** dsneddon has joined #tripleo04:42
*** dsneddon has quit IRC04:49
*** ykarel|away has joined #tripleo04:52
*** ykarel|away is now known as ykarel04:53
*** dsneddon has joined #tripleo04:56
*** rh-jelabarre has quit IRC05:00
*** dsneddon has quit IRC05:01
*** ramishra has joined #tripleo05:10
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/rocky: Remove configuration related to Horizon  https://review.opendev.org/69621505:11
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Use olso to parse size strings  https://review.opendev.org/50357405:20
*** dsneddon has joined #tripleo05:23
*** dsneddon has quit IRC05:24
*** dsneddon has joined #tripleo05:25
*** shyamb has joined #tripleo05:30
*** raukadah is now known as chkumar|rover05:33
*** slaweq has joined #tripleo05:36
*** shyamb has quit IRC05:36
*** yog_ has joined #tripleo05:41
*** slaweq has quit IRC05:43
*** shyamb has joined #tripleo05:59
*** marios|ruck has joined #tripleo06:04
*** shyamb has quit IRC06:05
*** shyamb has joined #tripleo06:05
*** waleedm_ has joined #tripleo06:12
*** epoojad1 has quit IRC06:14
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common stable/stein: OvS DPDK parameters failing in NIC Partitioning  https://review.opendev.org/69934306:20
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common stable/train: Incorrectly derives NeutronPhysnetNUMANodesMapping  https://review.opendev.org/69934406:21
*** holser has joined #tripleo06:22
openstackgerritMerged openstack/diskimage-builder master: Install rng-tools in Red Hat family distro images  https://review.opendev.org/69718306:25
*** bhagyashris has joined #tripleo06:26
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/queens: [Rocky and older] Actually install tmpwatch on overcloud nodes  https://review.opendev.org/69934706:32
*** yprokule has joined #tripleo06:44
shyambHi06:50
shyambrhosp14 deploy failed at ansible execution06:50
shyambit's saying "Control socket connect(/var/lib/mistral/overcloud/ansible-ssh/2d02ace92a): Permission denied"06:51
shyambwhat is expected here?06:51
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Use osp_release flag to set selinux mode  https://review.opendev.org/69917006:52
*** kozhukalov has joined #tripleo06:59
shyambHi07:05
*** lmiccini has joined #tripleo07:05
shyambAny idea on how can we we deploy tripleo openstack for CI/CD?07:05
shyambWe want to automate deployment of rhosp14 and perform some tests07:06
shyambas part of CI/CD07:06
shyambHow we do this in upstream?07:06
chkumar|rovershyamb: It is driven by zuul, Here is the jobs defined https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d which depends on tripleo-quickstart https://opendev.org/openstack/tripleo-quickstart07:08
*** pkopec has joined #tripleo07:09
shyambchkumar: thank you07:09
shyambchkkumar: I hope this works smooth without any issues07:12
shyambin upstream, we deploy the openstack + test + cleanup?07:13
shyambor we keep it deployed all the time?07:13
chkumar|rovershyamb: once the job finishes, the node gets deleted07:13
*** bandini has joined #tripleo07:14
*** florianf has joined #tripleo07:14
shyambchkkumar: great07:14
shyambthank you07:14
*** jfrancoa has joined #tripleo07:15
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/queens: Make sure glance_api_tls_proxy logs are persisted on the host  https://review.opendev.org/69935507:15
chkumar|roverTengu: Hello07:15
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/queens: Fix neutron api TLS logging  https://review.opendev.org/69935607:15
Tenguchkumar|rover: heya07:15
chkumar|roverTengu: please have a look at this one https://review.opendev.org/#/c/699161/07:15
Tenguchkumar|rover: how much do you pay me? beer accepted XD07:16
shyambchkkumar: We have a openstack based service get installed on compute07:16
*** jbadiapa has joined #tripleo07:16
shyambwe have integrated it with tripleo  also, not in the upstream07:16
Tengumeh...07:16
chkumar|roverTengu: works for me, n numbers of beer will work? But I will be deciding the place07:16
shyambchkkumar: for tripleo-quickstart, do we need to write a ansible role to install our product along with openstack?07:16
Tenguchkumar|rover: will have to wait for slagle poping up today in order to get some details.07:18
chkumar|rovershyamb: I think yes, you can check the tripleo-quickstart07:21
chkumar|roverTengu: ack!07:21
chkumar|roverTengu: thanks for looking into it07:22
shyambchkumar: okay, thank you07:24
*** bhagyashris has quit IRC07:26
*** whoami-rajat__ has joined #tripleo07:33
*** bhagyashris has joined #tripleo07:41
*** ykarel is now known as ykarel|lunch07:41
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Ensure blacklisted nodes are not included in server_names  https://review.opendev.org/69935907:42
*** shyamb has quit IRC07:44
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Create a dedicated log file for healthchecks for collectd  https://review.opendev.org/69922607:47
*** holser has quit IRC07:47
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Ensure /var/log/journal exists as soon as possible  https://review.opendev.org/69916107:50
*** pcaruana has joined #tripleo07:52
*** slaweq has joined #tripleo07:59
*** rpittau|afk is now known as rpittau08:05
*** bhagyashris has quit IRC08:06
openstackgerritDaniel Bengtsson proposed openstack/tripleo-quickstart-extras master: The file option should not require environment variable.  https://review.opendev.org/69402108:12
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/rocky: Apply firewall rule for tripleo ui if ui is really enabled  https://review.opendev.org/69936608:14
*** tkajinam has quit IRC08:18
*** tosky has joined #tripleo08:20
*** saneax has joined #tripleo08:22
*** tmazur has joined #tripleo08:24
*** shyamb has joined #tripleo08:24
*** tesseract has joined #tripleo08:25
*** paramite has joined #tripleo08:27
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/stein: Force facts cache refreshing after OS upgrade.  https://review.opendev.org/69889408:29
*** ykarel|lunch is now known as ykarel08:31
*** tmazur has quit IRC08:31
*** hberaud|gone is now known as hberaud08:33
*** tmazur has joined #tripleo08:37
*** amoralej|off is now known as amoralej08:38
*** jpalanis has joined #tripleo08:39
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/train: TripleO Validator CLI Refactor  https://review.opendev.org/69937108:45
openstackgerritMerged openstack/tripleo-common stable/rocky: Quote output values for overcloudrc  https://review.opendev.org/69748308:46
openstackgerritMerged openstack/python-tripleoclient stable/queens: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890308:46
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Enable additional Swift healtchecks  https://review.opendev.org/69889708:46
*** apetrich has joined #tripleo08:47
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Handle package builds when testing in rdoinfo  https://review.opendev.org/64941808:48
*** jpena|off is now known as jpena08:49
*** shyamb has quit IRC08:53
*** lucasagomes has joined #tripleo08:54
openstackgerritCarlos Goncalves proposed openstack/puppet-tripleo master: Add octavia::nova to Octavia services  https://review.opendev.org/69907409:00
*** jpich has joined #tripleo09:00
*** bhagyashris has joined #tripleo09:01
*** suuuper has joined #tripleo09:01
*** dtantsur has joined #tripleo09:02
*** rpioso_ has joined #tripleo09:03
*** ildikov_ has joined #tripleo09:03
*** bcrochet has joined #tripleo09:03
*** ykarel_ has joined #tripleo09:03
*** tosky_ has joined #tripleo09:04
*** skramaja_ has joined #tripleo09:04
*** cyberpear_ has joined #tripleo09:04
*** akahat has joined #tripleo09:04
*** jistr_ has joined #tripleo09:05
*** dasp has joined #tripleo09:05
*** StevenK_ has joined #tripleo09:06
*** shyamb has joined #tripleo09:07
shyambHi09:08
shyambIn RHOSP14, on rerun on overcloud deploy command, duplicate containers are getting created for horizon and couple of other services09:08
shyambIs it a defect or something I am doing wrong?09:09
shyambIn first deployment, stack created successfully but deployment failed at step409:10
shyambI executed overcloud deploy command with --config-download-only option, and I see duplicate horizon containers09:10
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Delegate and run once debug start-at-task messages  https://review.opendev.org/69926609:10
*** tesseract has quit IRC09:10
*** tosky has quit IRC09:10
*** florianf has quit IRC09:10
*** kozhukalov has quit IRC09:10
*** ykarel has quit IRC09:10
*** psachin has quit IRC09:10
*** rcernin has quit IRC09:10
*** thrash has quit IRC09:10
*** osmanlicilegi has quit IRC09:10
*** jeremy__bouncer has quit IRC09:10
*** baha has quit IRC09:10
*** tbarron has quit IRC09:10
*** fultonj has quit IRC09:10
*** skramaja has quit IRC09:10
*** jistr has quit IRC09:10
*** bfournie has quit IRC09:10
*** redrobot has quit IRC09:10
*** kambiz has quit IRC09:10
*** jamesdenton has quit IRC09:10
*** dasp_ has quit IRC09:10
*** StevenK has quit IRC09:10
*** dtantsur|afk has quit IRC09:10
*** mcarden has quit IRC09:10
*** radez has quit IRC09:10
*** rpioso has quit IRC09:10
*** cyberpear has quit IRC09:10
*** ildikov has quit IRC09:10
*** bcrochet is now known as thrash09:10
*** rpioso_ is now known as rpioso09:12
*** ildikov_ is now known as ildikov09:12
*** cyberpear_ is now known as cyberpear09:12
*** fultonj has joined #tripleo09:12
shyambTengu: owalsh: ykarel: any idea?09:12
*** skramaja_ is now known as skramaja09:13
*** jeremy__bouncer has joined #tripleo09:17
*** tesseract has joined #tripleo09:17
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/stein: TripleO Validator CLI Refactor  https://review.opendev.org/69937509:17
*** rcernin has joined #tripleo09:19
*** tosky_ is now known as tosky09:19
ykarel_shyamb, i doubt if duplicate containers can exist as deployment would try to create container with same name always09:20
ykarel_https://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/docker/services/horizon.yaml#L12209:20
*** bfournie has joined #tripleo09:20
ykarel_shyamb, can u share output of docker ps where u see duplicated containers?09:20
ykarel_and name of container u think is duplicated09:20
*** ykarel_ is now known as ykarel09:20
ykarelbut if they exist then possibly it can be a bug09:20
shyambykarel: yes, let me share the o/p09:21
shyambit's using system generated name09:21
shyambhere is the o/p: http://paste.openstack.org/show/787658/09:22
shyambthese duplicate containers were in "Restarting" state09:22
shyambI just manually stopped it09:23
shyambI saw "Address already in use" error in logs of these containers09:23
shyambit means they are created for the same purpose using same ip and port09:23
ykarelshyamb, looks like u started those containers manually, the exited one09:25
shyambykarel: same issue with keystone containers09:25
shyambno09:26
shyambI have not started them manually09:26
ykarellike horizon_fix_perms name09:26
shyambno09:26
*** florianf has joined #tripleo09:26
shyambhttp://paste.openstack.org/show/787659/09:26
shyambwhen I observed this issue I stopped the duplicate container which were in restarting state09:27
ykarelok horizon_fix_perms is started by tripleo, but the random names ones are created if we do not specify name09:29
*** bandini has quit IRC09:29
ykarelI have seen that when using paunch to start those for debugging09:29
shyambokay09:30
*** derekh has joined #tripleo09:33
*** sshnaidm|afk is now known as sshnaidm09:34
shyambykarel: what should we do now?09:36
shyamblog a defect?09:36
ykarelshyamb, yes u can report by specifying reproduction steps and issue u faced, and someone can confirm what could have gone wrong09:37
*** jbadiapa has quit IRC09:37
ykarelshyamb, so u have not used any paunch command, right?09:38
ykarelfor debugging09:38
owalshykarel: looking at the source I think it's a paunch bug....09:40
ykarelowalsh, hmm possible, u got it in what cases it can happen?09:41
owalshykarel: cleans up container here https://github.com/openstack/paunch/blob/stable/rocky/paunch/builder/compose1.py, but generates a unique name after that https://github.com/openstack/paunch/blob/stable/rocky/paunch/builder/compose1.py#L6109:41
shyambykarel: I have not used any paunch commands for debugging09:41
*** bandini has joined #tripleo09:41
owalshso when the cleanup happens it sees the horizon container but the config hasn't changed so it doesn't remove it09:42
ykarelshyamb, ack, then in that case it can be a pauch bug as owalsh said09:42
shyambykarel: okay, I will log the bug09:42
owalsh... then it tries to start it again, generating a unique name. Bug is that it shouldn't try to start it again09:43
owalshalthough this should have skipped it? https://github.com/openstack/paunch/blob/stable/rocky/paunch/builder/compose1.py#L5309:44
owalshykarel, shyamb: ^^^ that check should have skipped it AFAICT09:47
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates master: Fix Octavia to use correct Puppet class  https://review.opendev.org/69937809:50
*** jbadiapa has joined #tripleo09:51
*** shyamb has quit IRC09:51
*** alexmcleod has joined #tripleo09:52
owalshshyamb: paunch checks the container_name label and only run one 'horizon' container, i.e: docker ps -a --format '{{.Names}} {{.Label "container_name"}}'09:52
ykarelowalsh, looks like issue happens when container is stuck in restarting state09:53
owalshykarel: but the first horizon container is good?09:54
owalshso it shouldn't have tried to start the 2nd, which is restarting09:54
*** dsneddon has quit IRC09:57
*** dsneddon has joined #tripleo09:57
ykarelowalsh, hmm right09:58
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates stable/train: Fix Octavia to use correct Puppet class  https://review.opendev.org/69937909:58
owalshykarel: also I think docker ps -a should still list it when it's restarting09:58
owalshykarel: hrmm, so what does this return when docker ps fails? https://github.com/openstack/paunch/blob/stable/rocky/paunch/runner.py#L10509:58
owalshprobably None, but then will fail with a 'NoneType is not iterable' error10:00
owalshykarel: ah, I'd bet the paunch logic hasn't been backported to OSP1410:06
ykarelowalsh, u mean commits missing from stable/rocky in pauch shipped with osp14?10:08
owalshykarel: I mean upsteam fixes haven't landed in downstream osp1410:08
owalshykarel: shyamb is running osp1410:08
owalshno rocky10:09
ykarelowalsh, yes right10:09
owalshykarel: looking at rocky code I don't see how it's possible, but I've not checked osp1410:10
jfrancoaodyssey4me: hey, about https://review.opendev.org/698891 I was doing some testing in my env10:10
jfrancoaodyssey4me: and it looks like the problem is only with one of the facts, the discovered_python_interpreter fact10:11
jfrancoaodyssey4me: it doesn't get updated unless we flush all the facts10:11
jfrancoaodyssey4me: I'm testing this right now: https://review.opendev.org/#/c/698894/10:11
jfrancoaodyssey4me: right after performing the OS upgrade, force a facts refresh for that node. Like this next time we try to run any playbook the facts are the right ones10:12
odyssey4mejfrancoa: I've just commented much the same in review10:14
odyssey4mejfrancoa: off-hand, do you know which version of ansible we're using10:15
*** shyamb has joined #tripleo10:15
odyssey4meideally we should use ansible 2.8 to gain the new interpreter discovery feature... but if we're stuck with ansible 2.7 then things will get a little hairy10:15
jfrancoaodyssey4me: 2.8 downstream10:15
odyssey4mejfrancoa: really? then this behaviour is odd10:16
owalshykarel: hrmm, missing some patches but it should still work I think10:18
owalshshyamb: hi, can you check that the labels set correctly on the horizon containers?10:19
*** ysandeep has quit IRC10:19
shyambowalsh: okay10:19
shyambhow can I check it? command10:19
jfrancoaodyssey4me: yep, it's quite weird...let me find you the logs. and you'll see in the cache that all references to the python binary get updated exceot the discovered_python_interpreter one10:20
owalshshyamb: docker ps -a --format '{{.Names}} {{.Label "container_name"}}'10:20
ykarelowalsh, ack10:20
shyambowalsh: okay10:22
hrwhttps://bugs.launchpad.net/tripleo/+bug/1856678 reported so it maybe will not be forgotten10:25
openstackLaunchpad bug 1856678 in tripleo "Use Python 3 in CentOS 7 overcloud image" [Undecided,New]10:25
*** shyam89 has joined #tripleo10:25
*** shyamb has quit IRC10:27
shyam89owalsh: here is the labels list for horizon container:10:29
shyam89http://paste.openstack.org/show/787660/10:29
*** yog_ has quit IRC10:29
owalshshyamb: run the exact command, don't put a container_name in it (you want it to return the container name label)10:29
owalshshyam89: ^^^ i.e run this exact command: docker ps -a --format '{{.Names}} {{.Label "container_name"}}'10:30
shyam89okay10:32
shyam89got it10:32
shyam89owalsh: let me check10:32
shyam89bug logged for this issue: https://bugs.launchpad.net/tripleo/+bug/185668010:33
openstackLaunchpad bug 1856680 in tripleo "duplicate horizon, keystone containers getting launced during overcloud deploy" [Undecided,New]10:33
shyam89owalsh: ykarel: ^^10:33
shyam89owalsh: here is the labels list http://paste.openstack.org/show/787661/10:34
owalshshyam89: I only see one horizon container there, where did the others go?10:35
shyam89one quick question: If I want to proceed on deployment now, can I clean all the containers of overcloud and try --config-download-only option to apply ansible config?10:35
shyam89or any other option we have?10:35
shyam89owalsh: just now We cleaned it manually10:35
owalshshyam89: alright, well the label is set correctly on the original horizon container at least so paunch shouldn't have started the other 210:36
*** holser has joined #tripleo10:36
shyam89horizon_fix_perms container is there10:37
shyam89but it is in exited state10:37
*** holser has quit IRC10:37
shyam89owalsh: ^^10:37
owalshshyam89: yea, that's a different container10:37
*** holser has joined #tripleo10:38
shyam89owalsh: so overall it's not a bug?10:38
owalshshyam89: the fact that it started multiple horizon containers looks like a bug, but the logic in paunch looks ok to me, so not sure how it occurred10:38
shyam89owalsh: okay10:39
*** hberaud is now known as hberaud|lunch10:39
shyam89owalsh: how can I get out of this situation for now?10:39
shyam89any suggestion?10:39
shyam89delete all containers or overcloud stack delete?10:40
*** hrw has left #tripleo10:40
shyam89overcloud deploy failed with most of the containers is exited state10:44
shyam89rabbitmq, galera, haproxy10:44
shyam89in system logs we saw error related to horizon only10:44
*** holser has quit IRC10:44
shyam89and that was the last error10:45
shyam89of ansible10:45
shyam89at step510:45
shyam89ykarel: owalsh: ^^10:45
*** holser has joined #tripleo10:45
owalshshyam89: worth grabbing the docker inspect output from the affected containers10:47
owalshshyam89: but since this is osp14, and not rocky, really need to raise a Red Hat bugzilla not a launchpad bug10:48
shyam89owalsh: okay10:49
owalshshyam89: also FYI OSP14 is out of support next month10:49
shyam89okay10:49
shyam89I will log the bug10:49
shyam89any way to get out of this situation?10:49
shyam89We tried to clean the stack and redploy but again we are falling into same situation10:50
owalshshyam89: what did you do to clean it?10:50
shyam89openstack stack delete overcloud10:51
shyam89openstack plan delete10:52
shyam89owalsh: ^^10:52
*** holser has quit IRC10:54
*** holser has joined #tripleo10:54
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Create a dedicated log file for healthchecks for collectd  https://review.opendev.org/69922610:55
owalshshyam89: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/director_installation_and_usage/chap-performing_tasks_after_overcloud_creation#sect-Removing_the_Overcloud10:55
owalshshyam89: but AFAIK stack delete + plan delete should work10:55
*** kozhukalov has joined #tripleo10:56
shyam89owalsh: okay10:56
shyam89thank you10:57
*** hkominos has quit IRC11:06
*** shyam89 has quit IRC11:12
owalshshyam89: if deleting/redeploying gave the same result then I expect paunch will continue doing the same every time, so deleting containers etc.. isn't likely to work11:12
openstackgerritDaniel Bengtsson proposed openstack/tripleo-upgrade master: Fix python command.  https://review.opendev.org/69939111:14
odyssey4mejfrancoa: hmm, it seems from tripleo-quickstart/requirements.txt:ansible>=2.8,<2.9 that upstream is using ansible 2.8 as well... maybe there's a bug in thenew functionality, or maybe it's something like a broken symlink/binary in the build after the upgrade?11:14
odyssey4mejfrancoa: perhaps it'd be best to try using the new setting 'interpreter_python: auto' rather than the default to see if it works better. ref: https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html11:17
*** raildo has joined #tripleo11:19
*** bhagyashris has quit IRC11:19
jfrancoaodyssey4me: yep, that's something I was also thinking about...maybe setting that in the ansible.cfg would help11:20
jfrancoaodyssey4me: I'll give it a try, thanks for the tip11:20
*** lucasagomes has quit IRC11:24
*** florianf has quit IRC11:24
openstackgerritSoniya Murlidhar Vyas proposed openstack/tripleo-quickstart-extras master: [Rocky] Removed passing tests from skiplist  https://review.opendev.org/69939411:25
*** bhagyashris has joined #tripleo11:25
openstackgerritMerged openstack/tripleo-ci master: Replace ostempest job in favor of plain standalone  https://review.opendev.org/69799211:32
*** surpatil has joined #tripleo11:41
*** SurajPatil has joined #tripleo11:41
*** shyamb has joined #tripleo11:45
*** raildo has quit IRC11:47
openstackgerritMerged openstack/tripleo-common master: actions/ansible: catch errors when cleaning the work directory  https://review.opendev.org/69923411:48
*** raildo has joined #tripleo11:48
*** rfolco|bbl has joined #tripleo11:57
chkumar|roverTengu: Hello12:00
*** shyamb has quit IRC12:00
chkumar|roverTengu: I got the logs http://logs.rdoproject.org/89/24189/2/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/78ba3f9/logs/undercloud/var/log/audit/12:00
chkumar|roverfor selinux with permissive but the problem is in CI, log.1.gz file is there but audit.log but another file contains less log12:00
chkumar|roverTengu: here is all the denials http://logs.rdoproject.org/89/24189/2/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/78ba3f9/logs/undercloud/var/log/extra/denials.txt.gz12:02
openstackgerritGael Chamoulaud proposed openstack/python-tripleoclient stable/stein: TripleO Validator CLI Refactor  https://review.opendev.org/69937512:03
Tenguchkumar|rover: lemme check that.12:04
*** raildo has quit IRC12:04
Tenguo___O12:04
Tengupuppet shouldn't raise any AVC for about a month now.... jpich -^^^ ?!12:04
Tengusame for certmonger12:04
Tenguan old openstack-selinux is probably used there.12:04
chkumar|roverfinding out that also12:05
*** hberaud|lunch is now known as hberaud12:05
Tengupff... log is unusable.12:05
chkumar|roveropenstack-selinux.noarch                      0.8.20-0.20191105094256.6578483.el8            @delorean12:05
Tenguabout 90% of the AVCs are already corrected.12:05
Tengufinding the one we "need" is terrible.12:06
Tenguoh.12:06
Tengujpich: I think I'm right with the "domain transition" - we'd need to allow-errr... init_t to run container_t I think12:07
jpichNovember RPM looks too old yeah12:07
Tengusame thing as we did with certmonger. would make sense, since neutron side-cars are now managed with some systemd voodoo12:07
chkumar|roverok looking for fresh one12:07
Tengujpich: http://logs.rdoproject.org/89/24189/2/check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/78ba3f9/logs/undercloud/var/log/extra/denials.txt.gz  search for "execute" or "execute_no_trans"12:08
Tenguwell, maybe not a full domain transition.....12:08
Tengubut at least access to the files12:08
openstackgerritSergii Golovatiuk proposed openstack/tripleo-common stable/train: Add horizon health check  https://review.opendev.org/69900512:08
jpichA bunch related to Neutron as well?12:08
*** jbadiapa has quit IRC12:09
Tengujpich: bunch related to certmonger (OK), puppet (OK), dbus (we should drop them) mostly.12:09
Tenguneutron is the init_t thing12:09
chkumar|roverTengu: jpich let me re run the job and let;s see what is there12:09
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates stable/train: Enable horizon healthcheck  https://review.opendev.org/69900612:09
Tengujpich: now we're starting the side-cars (network dhcp and so on) using a systemd "check" instead of plain wrappers from withinthe containers.12:09
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/queens: Apply firewall rule for tripleo ui if ui is really enabled  https://review.opendev.org/69940112:10
Tengujpich: grosso-merdo, systemd is instructed to check a location, and if a file is changed|added|removed|other, it has to take a certain action (start or drop a container)12:10
jpichTengu: I'll take your word for it ;)12:10
Tengujpich: systemd runs as init_t - and it wants to sync (?) container_file_t12:11
Tengujpich: ah, apparently, it would be "just" allow init_t container_file_t:file { execute execute_no_trans };12:11
chkumar|roverTengu: jpich running fs01 train & master in periodic https://review.rdoproject.org/r/#/c/24189/ it will get fresh contents and then see12:11
TenguI can add a new os-systemd.te12:11
chkumar|roverI mean we have the results after 5 hours12:12
Tenguchkumar|rover: I think we should get what we need, but it would be good to ensure openstack-selinux is latest in the jobs12:12
jpichSeems reasonable12:12
jpichThanks, chkumar|rover12:12
Tengujpich: lemme provide a patch :)12:12
jpichSounds good, thanks Tengu :D12:13
Tengujpich: care to point me the BZ again?12:13
chkumar|roveropenstack-selinux.noarch         0.8.20-0.20191204092526.4b7dae8.el7   @delorean12:13
chkumar|roverit is coming from master12:13
chkumar|rovercentos job12:13
Tenguchkumar|rover: ah, that's better. really better12:13
jpichchkumar|rover: that looks like the latest :)12:14
jpichTengu: Was thee a BZ for the init_t stuff? I only recall the dontaudit send_msg one from yesterday?12:14
Tengujpich: hmmm I think there is one?12:15
Tengu1s12:15
Tenguor was it an LP?12:15
chkumar|roverTengu: jpich https://bugs.launchpad.net/tripleo/+bug/185365212:15
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged]12:15
Tenguchkumar|rover: thanks dude!12:15
jpichAh! :)12:15
*** amoralej is now known as amoralej|lunch12:16
Tengujpich: is it OK to point to an LP in the change?12:16
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893612:16
jpichTengu: Sure! Though this LP is pretty long so if you could include the explanation summary from above too that would be cool :D Thank you!!12:16
jpichI have to step away for a bit but will catch up on PRs etc when back12:17
Tengujpich: I'll make a commit message with the needed blah (context and so on)12:17
jpich\o/12:17
chkumar|roverTengu: jpich openstack-selinux-0.8.20-0.20191204091803.4b7dae8.el8.noarch.rpm is available in https://trunk.rdoproject.org/rhel8-master/tripleo-ci-testing/12:18
chkumar|rovermay be upcomign run can give more picture12:18
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Derive Pci passthrough whitelist disable option  https://review.opendev.org/69762312:18
*** jbadiapa has joined #tripleo12:19
Tengujpich: chkumar|rover https://github.com/redhat-openstack/openstack-selinux/pull/5212:20
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-heat-templates stable/queens: Derive Pci passthrough whitelist disable option  https://review.opendev.org/69940312:22
*** jpich has quit IRC12:24
*** jpich has joined #tripleo12:24
*** rfolco|bbl is now known as rfolco12:27
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-heat-templates stable/queens: Derive Pci passthrough whitelist disable option  https://review.opendev.org/69940312:28
Tengujpich: is my PR valid? since we apparently don't need to include the class anymore..12:29
*** raildo has joined #tripleo12:36
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Move ostempest job definition with in featureset 52  https://review.opendev.org/69821112:37
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Remove tempest fsoverrides in favor of featureset vars  https://review.opendev.org/69821412:37
*** tesseract has quit IRC12:38
*** florianf has joined #tripleo12:38
*** tesseract has joined #tripleo12:38
*** jpena is now known as jpena|lunch12:38
*** lucasagomes has joined #tripleo12:39
*** ksambor_ has joined #tripleo12:43
*** ksambor has quit IRC12:45
*** ksambor_ is now known as ksambor12:45
*** jpalanis has quit IRC12:48
*** dsneddon has quit IRC12:51
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69862812:54
*** gfidente has joined #tripleo12:54
*** surpatil has quit IRC12:54
*** SurajPatil has quit IRC12:55
*** surpatil has joined #tripleo12:55
*** SurajPatil has joined #tripleo12:55
*** dsneddon has joined #tripleo12:57
*** sshnaidm is now known as sshnaidm|afk12:58
*** rlandy has joined #tripleo12:59
*** SurajPatil has quit IRC13:00
*** morazi has joined #tripleo13:00
*** SurajPatil has joined #tripleo13:01
openstackgerritMerged openstack/tripleo-heat-templates master: Try deleting container for failures too  https://review.opendev.org/69862213:01
*** surpatil has quit IRC13:01
*** redrobot has joined #tripleo13:01
*** surpatil has joined #tripleo13:02
*** dsneddon has quit IRC13:02
jpichTengu: Should be fine, looking now13:05
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Fix neutron api TLS logging  https://review.opendev.org/69935613:07
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Make sure glance_api_tls_proxy logs are persisted on the host  https://review.opendev.org/69935513:07
Tengujpich: thanks - lemme know if you want some more details in the commit message.13:07
jpichTengu: Text looks fine to me! Though maybe it'd be good to reference the LP there as well13:08
*** lbragstad_ is now known as lbragstad13:08
Tenguah, yeah, maybe :). lemme correct it.13:08
*** dsneddon has joined #tripleo13:09
Tengujpich: updated.13:09
*** whoami-rajat__ has quit IRC13:09
*** rh-jelabarre has joined #tripleo13:11
*** udesale has joined #tripleo13:11
*** ykarel is now known as ykarel|afk13:12
*** udesale has quit IRC13:12
*** udesale has joined #tripleo13:13
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates master: WIP: Change default dns domain in neutron  https://review.opendev.org/69940713:14
*** dsneddon has quit IRC13:14
*** hberaud has quit IRC13:15
*** ykarel|afk has quit IRC13:17
*** hberaud has joined #tripleo13:18
*** boazel has joined #tripleo13:20
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/train: actions/ansible: catch errors when cleaning the work directory  https://review.opendev.org/69940913:20
openstackgerritJose Luis Franco proposed openstack/tripleo-common master: Change the python interpreter discovery mode.  https://review.opendev.org/69941113:22
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI/fs010: disable Paunch on the overcloud  https://review.opendev.org/69407613:23
openstackgerritJose Luis Franco proposed openstack/tripleo-common master: Change the python interpreter discovery mode.  https://review.opendev.org/69941113:25
*** bhagyashris has quit IRC13:28
Tengujpich: updated the PR with a new commit so that it's clean in case of revert (and for the history - better to keep 1-change 1-commit)13:29
jpichTengu: I like it! Poking around some more but should be good13:30
Tengujpich: cool :). Lemme know otherwise.13:30
cloudnullmornings13:31
*** ade_lee has quit IRC13:33
*** ade_lee has joined #tripleo13:33
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Replace ansible shell with python runner  https://review.opendev.org/67187813:35
openstackgerritMerged openstack/tripleo-upgrade stable/stein: Fix the undefined operation_type variable.  https://review.opendev.org/69925413:35
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Overload stdout callback plugin  https://review.opendev.org/69801513:36
*** ade_lee has quit IRC13:36
*** ade_lee has joined #tripleo13:36
*** udesale has quit IRC13:39
*** jpena|lunch is now known as jpena13:41
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Ensure blacklisted nodes are not included in server_names  https://review.opendev.org/69935913:44
*** dsneddon has joined #tripleo13:45
*** amoralej|lunch is now known as amoralej13:47
*** ade_lee has quit IRC13:47
*** ksambor has quit IRC13:57
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: CiscoNexus: Update and Remove deprecated vars  https://review.opendev.org/69942013:57
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: Remove incorrect mapped_data key from cisco ml2 hieradata  https://review.opendev.org/69942113:58
*** mcornea has joined #tripleo13:58
EmilienMI'm supposed to run the tripleo meeting13:59
*** akahat has quit IRC13:59
*** dpawlik has joined #tripleo14:00
Tengu:]14:00
EmilienM#startmeeting tripleo14:00
openstackMeeting started Tue Dec 17 14:00:52 2019 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: tripleo)"14:00
openstackThe meeting name has been set to 'tripleo'14:00
EmilienM#topic agenda14:00
*** openstack changes topic to "agenda (Meeting topic: tripleo)"14:01
EmilienM* Review past action items14:01
EmilienM* One off agenda items14:01
EmilienM* Squad status14:01
EmilienM* Bugs & Blueprints14:01
EmilienM* Projects releases or stable backports14:01
EmilienM* Specs14:01
EmilienM* Other14:01
EmilienMAnyone can use the #link, #action and #info commands, not just the moderator14:01
EmilienMwho is around today?14:01
EmilienMdon't be shy14:01
Tengu«o/14:02
sshnaidm|afko/14:02
matbuo/14:02
*** sshnaidm|afk is now known as sshnaidm14:02
*** ksambor has joined #tripleo14:02
EmilienMskipping the past action items, there is none14:02
EmilienM#topic one off agenda items14:02
*** openstack changes topic to "one off agenda items (Meeting topic: tripleo)"14:02
EmilienM#link https://etherpad.openstack.org/p/tripleo-meeting-items14:02
EmilienMnothing noted in the agenda today14:03
marios|ruck\o14:03
Tenguguess it will be a short mtg :)14:03
EmilienMthe best ones14:03
openstackgerritDaniel Bengtsson proposed openstack/tripleo-upgrade master: Fix python command.  https://review.opendev.org/69939114:03
EmilienM#topic Squad status14:03
*** openstack changes topic to "Squad status (Meeting topic: tripleo)"14:03
EmilienMAny squad wanting to bring up their status, or a topic for the general public?14:03
chkumar|rover\o/14:03
EmilienMI'm lazy to post all the link14:03
*** tkajinam has joined #tripleo14:04
EmilienM#topic bugs & blueprints14:04
*** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)"14:04
EmilienMchkumar|rover: do we have any outstanding bug to report here?14:04
EmilienManything we need to make visible?14:04
holsero/14:05
chkumar|roverhttps://bugs.launchpad.net/tripleo/+bug/185365214:05
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Fix committed] - Assigned to Cédric Jeanneret (cjeanner)14:05
chkumar|roverEmilienM: https://bugs.launchpad.net/tripleo/+bug/1856278 this one needs to be sorted out14:05
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,In progress] - Assigned to Martin Mágr (mmagr)14:05
chkumar|roverit is blocking train promotion14:05
*** waleedm_ has quit IRC14:06
openstackgerritDaniel Bengtsson proposed openstack/tripleo-upgrade stable/rocky: Fix the undefined operation_type variable.  https://review.opendev.org/69942314:06
EmilienMcjloader, paramite ^ any updates14:06
EmilienMoops14:06
EmilienMTengu: ^14:06
Tengua fix seems to be proposed for the second one14:06
chkumar|roverThat's it, I think, marios|ruck any other bug?14:06
Tengufirst one is sorted out already.14:06
Tenguah. meh. the fix is on hold due to slagle in fact.14:07
Tenguhave to check with him...14:07
EmilienMTengu: link?14:07
Tenguhttps://review.opendev.org/#/c/699161/14:07
marios|ruckchkumar|rover: all there as usual https://etherpad.openstack.org/p/ruckroversprint1914:07
marios|ruckchkumar|rover: nothing major today gates green afaik chasing rhel promotion still14:08
Tenguhe's not happy with the way it's done, although it's a backport from master... so I don't know if we should force it and come later, or find another way to do the same thing...14:08
EmilienMI would fix master14:09
EmilienMand backport the new patch only14:09
EmilienMand revert the previous one on master14:09
Tenguand there, I don't know what's wrong with it either....14:09
EmilienMthey shouldn't conflict14:09
Tengubecause I'm just doing what was already done :/14:09
Tengui.e. adding ansible stuff in there14:10
EmilienMTengu: I understand slagle's point where every time we add tasks here, it makes our deployment slower, we need to find more efficient ways to do it14:10
EmilienM#action Tengu to find consensus with slagle wrt https://review.opendev.org/#/c/699161/14:11
Tenguwell, that task should be done on every node anyway. Soooo... well, we'll see.14:11
*** ksambor has quit IRC14:11
*** ksambor has joined #tripleo14:12
EmilienMshould we move on14:12
Tenguyup. we'll take it offline14:12
*** surpatil has quit IRC14:12
EmilienMI wish paramite would give an update on https://bugs.launchpad.net/tripleo/+bug/185627814:13
openstackLaunchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,In progress] - Assigned to Martin Mágr (mmagr)14:13
EmilienMbut we'll see later14:13
*** SurajPatil has quit IRC14:13
EmilienM#topic projects releases or stable backports14:13
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:13
EmilienMis there anything about release management? Any release is needed this week?14:13
EmilienM#link https://releases.openstack.org/ussuri/schedule.html14:13
EmilienMUssuri-1 milestone was last week14:14
EmilienMI suspect weshay missed it, I haven't seen any patch14:14
EmilienMhttps://review.opendev.org/#/q/project:openstack/releases14:14
EmilienMI don't see it14:14
EmilienM#action EmilienM to release TripleO Ussuri-1 milestone14:15
Tengu(AH! I think I've found where to push that: common/deploy-steps-tasks-step-0.j2.yaml)14:15
EmilienManything about stable releases otherwise?14:15
EmilienMTengu: yes, better14:15
EmilienM#topic specs14:15
*** openstack changes topic to "specs (Meeting topic: tripleo)"14:15
EmilienManyone wants to bring up their specs here?14:16
EmilienMit's time for free marketing14:16
EmilienMnote that 2 specs were approved this month14:16
EmilienM#link https://specs.openstack.org/openstack/tripleo-specs/specs/ussuri/scaling-with-ansible-inventory.html from slagle14:17
EmilienMand14:17
EmilienM#link https://specs.openstack.org/openstack/tripleo-specs/specs/ussuri/tripleo-operator-ansible.html from mwhahaha14:17
holserEmilienM we are also working on FFWD2 spec14:17
EmilienMholser: cool!14:17
EmilienM#info FFWD2 specs coming soon!14:17
EmilienManything else regarding specs?14:17
*** ade_lee has joined #tripleo14:18
EmilienM#topic open discussion14:18
*** openstack changes topic to "open discussion (Meeting topic: tripleo)"14:18
marios|ruckholser: not ffu2? has a certain ring to it ;)14:18
holser:)14:18
EmilienMfeel free to ask anything now or bring a last minute topic14:18
EmilienMlike... what is the meaning of life?14:18
EmilienMor... is anyone actually reading me?14:18
marios|ruckwill i finally no longer be ruck tomorrow?14:18
EmilienMthese kind of questions14:18
EmilienMmarios|ruck: just change your nickname14:19
marios|ruckthanks EmilienM !14:19
EmilienMI suspect we won't handle the meeting next week14:19
EmilienMbut who knows14:19
EmilienMhave fun everyone14:19
EmilienM#endmeeting14:19
*** openstack changes topic to "CI Status: green | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"14:19
openstackMeeting ended Tue Dec 17 14:19:40 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:19
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-12-17-14.00.html14:19
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-12-17-14.00.txt14:19
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo/2019/tripleo.2019-12-17-14.00.log.html14:19
*** ade_lee has quit IRC14:20
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Create /var/log/journal directory during step-0  https://review.opendev.org/69942514:21
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Revert "Ensure /var/log/journal exists as soon as possible"  https://review.opendev.org/69942614:21
TenguEmilienM: -^^14:21
Tengushould be better I guess.14:21
*** ade_lee has joined #tripleo14:21
*** whoami-rajat__ has joined #tripleo14:23
EmilienMI think this patch is enough for the ussuri m1 https://review.opendev.org/#/c/696147/14:23
EmilienMto be confirmed when mwhahaha is around14:23
EmilienMTengu: yes better I think14:24
*** ade_lee_ has joined #tripleo14:24
EmilienMslagle: https://review.opendev.org/#/c/69942514:24
cjloaderEmilienM: ?14:25
cloudnullanyone around want to give - https://review.opendev.org/#/c/698620 - a nudge14:26
EmilienMcjloader: wrong ping14:26
openstackgerritJose Luis Franco proposed openstack/tripleo-common master: Change the python interpreter discovery mode.  https://review.opendev.org/69941114:27
*** ade_lee has quit IRC14:27
cjloaderok np14:27
cjloadercloudnull: o/14:27
cloudnullo/14:27
openstackgerritMerged openstack/tripleo-quickstart-extras master: [Train] Move list_agent and traffic tests to skip list.  https://review.opendev.org/69843514:32
openstackgerritMerged openstack/puppet-tripleo master: Convert all class usage to relative names  https://review.opendev.org/69793014:37
openstackgerritDaniel Bengtsson proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893614:41
*** panda has quit IRC14:44
*** panda has joined #tripleo14:44
*** tkajinam has quit IRC14:47
openstackgerritTobias Urdin proposed openstack/puppet-pacemaker master: Convert all class usage to relative names  https://review.opendev.org/69792014:47
*** TrevorV has joined #tripleo14:47
*** ykarel|afk has joined #tripleo14:47
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo-keystone-resources: reduce permissions for clouds.yaml  https://review.opendev.org/69943014:47
*** ykarel|afk is now known as ykarel14:51
*** artom has joined #tripleo15:01
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Adapt overcloud upgrade workflow to Train.  https://review.opendev.org/69925615:02
*** ykarel has quit IRC15:03
*** ykarel has joined #tripleo15:03
openstackgerritTobias Urdin proposed openstack/puppet-pacemaker master: Convert all class usage to relative names  https://review.opendev.org/69792015:05
*** kozhukalov has quit IRC15:07
openstackgerritDaniel Bengtsson proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893615:09
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-ansible master: Add SyslogIdenfier to healthcheck systemd unit  https://review.opendev.org/69921515:10
openstackgerritCédric Jeanneret (Tengu) proposed openstack/paunch master: Add SyslogIdenfier to healthcheck systemd unit  https://review.opendev.org/69921615:10
openstackgerritDaniel Bengtsson proposed openstack/tripleo-heat-templates master: Run update without yum update to apply hotfixes.  https://review.opendev.org/69893615:10
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates master: Create a dedicated log file for healthchecks for collectd  https://review.opendev.org/69922615:12
*** jaosorior has joined #tripleo15:26
paramiteEmilienM, the update is that Tengu is working on it.15:26
EmilienMparamite: ok15:26
Tenguyup - and it would be good to get the "correct" patch merged asap so that we can move forward :)15:29
Tengui.e. backport it15:29
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Create /var/log/journal directory during step-0  https://review.opendev.org/69943815:31
TenguEmilienM, paramite : just pushed the backport -^^15:32
openstackgerritJose Luis Franco proposed openstack/tripleo-common stable/stein: Change the python interpreter discovery mode.  https://review.opendev.org/69943915:32
*** skramaja has quit IRC15:36
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Move ostempest job definition with in featureset 52  https://review.opendev.org/69821115:36
*** waleedm has joined #tripleo15:39
chkumar|roverabishop: Hello, if around, please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1856723 , we check the glance logs does not found anyhting15:39
openstackLaunchpad bug 1856723 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master fails during tempest run ImagesTestJSON:test_delete_saving_image" [Critical,Triaged]15:39
chkumar|roverit does not seems to be a network issue, we saw it twice15:39
chkumar|roveror thrice15:39
chkumar|roverit is blocking master promotion15:40
chkumar|roverthanks :-)15:40
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Add CephGrafana resource to the ControllerStorageNFS role  https://review.opendev.org/69884015:41
chkumar|roverksambor: Hi, please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/1856016 it seems to be an issue15:47
openstackLaunchpad bug 1856016 in tripleo "Tempest basic ops test_cross_tenant_traffic tests failed on fs020 train with Timed out waiting for 10.0.0.102 to become reachable from 10.0.0.117" [Critical,Confirmed]15:47
chkumar|roverthanks!15:47
*** hkominos has joined #tripleo15:48
hkominosHi all. Just a sanity check on THT Is this a typo ? https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/octavia-underc... . I was expecting stat: path: and not path= . I did not see that annotation in the ansible docs.15:48
ksamborhey chkumar|rover thanks I will take a look15:49
*** chkumar|rover is now known as raukadah15:50
*** saneax has quit IRC15:52
abishopraukadah: ack, I'll take a look15:53
*** leanderthal has joined #tripleo15:54
openstackgerritAlex Schultz proposed openstack/tripleo-docs master: Remove fedora28 repo info  https://review.opendev.org/69944415:55
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/train: Add LEAPP_SKIP_CHECK_OS_RELEASE variable in UC upgrade.  https://review.opendev.org/69875815:56
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Whitelist the profile_tasks callback by default  https://review.opendev.org/69809615:58
abishopraukadah: looks like a tempest timing issue, perhaps a repeat of https://bugs.launchpad.net/tempest/+bug/171316315:58
openstackLaunchpad bug 1713163 in tempest "test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed]15:58
raukadahabishop: any setting we can tweak there, like build time or something?15:59
*** lucasagomes has quit IRC16:00
abishopraukadah: I don't think so. I'm not a tempest dev, but would look to update the test so it doesn't require observing the transitional SAVING state16:01
abishopthis is pretty much what this says: https://bugs.launchpad.net/tempest/+bug/1713163/comments/516:02
openstackLaunchpad bug 1713163 in tempest "test_delete_saving_image fails because image hasn't transitioned to SAVING" [Medium,Confirmed]16:02
raukadahabishop: thanks, I will take a look in morning16:02
*** ykarel is now known as ykarel|away16:04
*** gkadam has joined #tripleo16:06
*** whoami-rajat__ has quit IRC16:06
*** baha has joined #tripleo16:06
*** mcornea has quit IRC16:07
*** mcornea has joined #tripleo16:09
*** leanderthal has quit IRC16:09
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: WIP: Allow full path to be specified for amphora destination  https://review.opendev.org/69285816:10
*** leanderthal has joined #tripleo16:10
*** B_dgfebba has joined #tripleo16:11
*** yprokule has quit IRC16:15
*** B_dgfebba has quit IRC16:15
*** gkadam has quit IRC16:15
*** fmount has quit IRC16:17
*** fmount has joined #tripleo16:18
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Implement utils.copy_clouds_yaml  https://review.opendev.org/69944816:20
*** lmiccini has quit IRC16:20
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Implement utils.copy_clouds_yaml  https://review.opendev.org/69944816:21
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639016:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639016:23
odyssey4mejfrancoa: does https://review.opendev.org/#/q/I87974e88c38b42e90bc3cd801fcf1deaf268720c depend on https://review.opendev.org/#/q/Iaef4839bb15ec398537b5c57a441c8e28a552bc0 ?16:24
*** paramite|clone has joined #tripleo16:24
*** paramite has quit IRC16:24
openstackgerritDaniel Alvarez proposed openstack/tripleo-heat-templates stable/train: ovn standalone: Start OVN ovsdb-servers using the kolla startup script  https://review.opendev.org/69723716:25
jfrancoaodyssey4me: nope, I think they are independent. I am testing the first patch and that would work (the ansible facts cache gets rebuilt and we don't fail anymore when running the upgrade step)16:25
odyssey4mejfrancoa: yep, looks like the new discovery method works much better16:26
jfrancoaodyssey4me: but once the job finishes I want to test if the second patch would also solve the issue...if it does I think it's a better solution16:26
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Revert "Added for support for generating overcloud clouds.yaml"  https://review.opendev.org/69932916:27
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Revert "Added support of overcloud os_cloud"  https://review.opendev.org/69920616:27
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo-keystone-resources: fix a typo for domains creation  https://review.opendev.org/69927416:28
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo-keystone-resources: handle multiple roles per user  https://review.opendev.org/69928616:28
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698016:28
*** leanderthal has quit IRC16:31
openstackgerritAntony Messerli proposed openstack/diskimage-builder master: Adds support for Fedora 31  https://review.opendev.org/69771316:32
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-validations master: New validation: validate-selinux  https://review.opendev.org/69800616:33
*** jfrancoa has quit IRC16:33
*** artom has quit IRC16:37
*** lhinds has quit IRC16:44
*** lhinds has joined #tripleo16:45
*** jpena is now known as jpena|brb16:45
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/rocky: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865316:47
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/queens: Properly set loopback_users via rabbitmq::loopback_users  https://review.opendev.org/69865516:47
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/stein: Update environment var for keystone bootstrap  https://review.opendev.org/69901916:47
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/train: Increase rabbitmq tcp backlog  https://review.opendev.org/69946216:48
*** jeremy__bouncer has quit IRC16:49
*** stevebaker has quit IRC16:49
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/rocky: Fix the undefined operation_type variable.  https://review.opendev.org/69942316:50
*** jfrancoa has joined #tripleo16:53
*** hberaud is now known as hberaud|school-r16:53
*** mcornea has quit IRC16:55
*** ykarel|away has quit IRC16:59
*** waleedm has quit IRC17:00
*** marios|ruck is now known as marios|out17:00
*** tesseract has quit IRC17:00
*** marios|out has quit IRC17:03
*** hkominos has quit IRC17:04
*** mcornea has joined #tripleo17:08
*** artom has joined #tripleo17:08
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: HA: increase resource default op timeout for podman bundles  https://review.opendev.org/69766017:13
*** bandini has quit IRC17:15
*** jpich has quit IRC17:17
*** Goneri has joined #tripleo17:20
*** hberaud|school-r is now known as hberaud17:21
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Replace Fluentd and SensuClient from roles_data.  https://review.opendev.org/69924517:22
*** jpena|brb is now known as jpena17:23
*** ramishra has quit IRC17:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: keystone/bootstrap: fix identity service name  https://review.opendev.org/69946817:23
*** rpittau is now known as rpittau|afk17:25
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/stein: [stein only] WIP Make podman update from 1.0.5 to 1.4.2.  https://review.opendev.org/69925517:27
*** tmazur has quit IRC17:43
openstackgerritAlex Schultz proposed openstack/puppet-tripleo master: Use keystone::bootstrap  https://review.opendev.org/69581917:46
openstackgerritMerged openstack/tripleo-common stable/train: Incorrectly derives NeutronPhysnetNUMANodesMapping  https://review.opendev.org/69934417:48
openstackgerritMerged openstack/tripleo-common stable/stein: OvS DPDK parameters failing in NIC Partitioning  https://review.opendev.org/69934317:48
openstackgerritMerged openstack/tripleo-heat-templates master: Set octavia services' stop grace period to 300sec  https://review.opendev.org/69801417:57
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: deploy-steps: use lookup() to get pre|post_deployments_role variables  https://review.opendev.org/69907317:57
*** lbragstad_ has joined #tripleo17:58
*** lbragstad has quit IRC18:00
*** derekh has quit IRC18:00
*** jaosorior has quit IRC18:04
abishopowalsh, slagle: is there a tht parameter to control whether volumes can be attached across AZ's?18:06
*** amoralej is now known as amoralej|off18:06
openstackgerritMerged openstack/tripleo-quickstart-extras master: Use osp_release flag to set selinux mode  https://review.opendev.org/69917018:14
*** rlandy has quit IRC18:15
*** rlandy has joined #tripleo18:16
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: [Rocky and older] Actually install tmpwatch on overcloud nodes  https://review.opendev.org/69934718:17
cloudnullhas anyone seen this error before - https://pasted.tech/pastes/33c09cb07119badf6ddb8a6072876a3a8c30ed2e.raw ?18:20
cloudnull> no attribute 'ansible_ssh_host_key_rsa_public'18:20
cloudnullmaybe bad facts?18:21
mwhahahacached fact problem?18:22
slagleabishop: not that I know of18:22
cloudnullwould that be something within the mistral container ?18:22
mwhahahacloudnull: no18:23
mwhahahacloudnull: was this a job? or what18:24
cloudnullno this was running locally18:24
abishopslagle: ack, though I recall discussion about preventing dcn compute from attaching volumes in another edge site18:24
mwhahahacloudnull: do you not have sshd running (or no host keys?)18:24
cloudnullyes sshd is running18:25
* cloudnull checking hostkeys 18:25
mwhahahano rsa pub key tho?18:25
cloudnullthe overcloud nodes all have host keys and I can ssh to them.18:27
cloudnullnone of the overcloud nodes have their own pub/private user keys. its just setup where the undercloud can access the hosts18:28
cloudnulldoes the non-root user need to have keys generated on all overcloud nodes ?18:28
mwhahahathis is an ansible fact18:29
mwhahahamy assumption is that it failed to fetch it18:29
mwhahahaand it's a host key18:29
cloudnullodd. will try and purge facts and rerun18:30
*** jpena is now known as jpena|off18:33
slagleabishop: we may have just recommended setting the hieradata18:36
*** pcaruana has quit IRC18:38
Tengumwhahaha: thanks for the votes :)18:47
Tenguslagle: heya! so I've reworked the patch on master (for the /var/log/journal directory): https://review.opendev.org/#/c/699425/  I guess it's more suited, isn't it?18:48
*** panda has quit IRC18:49
Tenguslagle: and reverted the other one, abandonned the backport and.. well, guess it's clean now :)18:49
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Revert "Convert firewall rules to use TripleO-Ansible"  https://review.opendev.org/69947718:49
Tenguhjensas: why that? -^^  (ping cloudnull )18:49
*** panda has joined #tripleo18:49
hjensasTengu: see bug.18:49
hjensasTengu: I can't figure out why the role does'nt persist the rules so that they survive a reboot.18:50
Tenguhjensas: ah, some rules are added by services directly, right?18:51
openstackgerritJames Slagle proposed openstack/tripleo-docs master: Update docs for ManageNetworks  https://review.opendev.org/69947918:51
hjensasTengu: it does'nt persist the rules. Reboot, and you end up with the default tables ... It's not just the special stuff with rules managed by services such as neutron and ironic-inspector.18:53
Tenguhjensas: I think it's mainly due to the firewalld being active.18:53
slagleTengu: yes, that's a much better spot for it. thanks18:53
Tenguhjensas: pretty sure disabling it will help. Like, a lot18:53
Tenguslagle: great :). Happy to make better changes, thanks for your review!18:53
Tenguhjensas: would you be able to test that somewhere?18:54
cloudnullslagle while you're around https://review.opendev.org/#/c/698620/ ;)18:54
cloudnullTengu ++18:54
Tenguhjensas: I can probably take some time tomorrow if you're OK to wait (a bit late today for me, my wife will probably be angry if she knows I'm here :D)18:54
slaglecloudnull: done18:54
cloudnullhjensas in the job outputs we can see the rules being saved, so something else must be overwriting them on reboot18:55
Tengucloudnull: as said: pretty sure it's firewalld18:55
cloudnull+118:55
Tenguif it's running, we're doomed.18:55
cloudnullshould we disable firewalld in the tripleo-firewall role ?18:55
Tenguso it would be good to ensure it's disabled18:55
Tengucloudnull: I think so, yes.18:55
cloudnullwe can add a task for that18:56
cloudnullthanks slagle18:56
Tengucloudnull: ah, also.... would be interesting to ensure we don't have any of the "initial rules" in the firewall - I remember having a big, big headache with that some months ago18:56
hjensasTengu: on the system I have with the issue now there is no firewalld service.18:56
Tenguhjensas: I would take some time to investigate the issue instead of that "dry" revert - pretty sure cloudnull is in the same mood18:57
openstackgerritBrent Eagles proposed openstack/tripleo-ansible master: Scan existing controllers on update to pick up existing CA/private key pairs  https://review.opendev.org/69259918:57
cloudnullTengu commented that in the review :D18:57
Tengucloudnull: :)18:58
hjensasTengu: well, the bug's been open for a couple of weeks. I'm tierd of re-running the undercloud installer after each reboot. ;)18:58
Tengucloudnull: I can try to help you tomorrow in your morning if you want? (that is, if you didn't find the issue before)18:58
Tenguhjensas: were there some calls for help on ML|meeting? true it's "old", but raising some attention earlier might have helped :/. let's see what cloudnull can do during his day19:00
*** jaosorior has joined #tripleo19:00
cloudnullhjensas from the bug it looks like firewalld was running ?19:00
cloudnull> Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled;19:01
Tengucloudnull: you can probably already send a patch in order to disable that service.19:01
hjensascloudnull: yes firewalld was running on the system I had 2019-12-06, the one i deployed yesterday does not have firewalld.19:01
Tenguhjensas: manually disabled? if so, when? before or after the deploy?19:02
cloudnullisn't firewalld installed by default on rhel7/8 ?19:03
Tengucloudnull: it is for 8 at least. Not sure about 719:03
hjensasTengu: not manually, something must have changed since 06-12. Now I get:19:03
ade_lee_EmilienM, are scale_tasks executed on the node being scaled out, or on the undercloud?19:03
hjensas[root@undercloud ~]# systemctl status firewalld.service19:03
hjensasUnit firewalld.service could not be found.19:03
hjensas[root@undercloud ~]# cat /etc/redhat-release19:03
hjensasCentOS Linux release 7.7.1908 (Core)19:03
openstackgerritAlex Schultz proposed openstack/tripleo-ansible master: zuul: all functional jobs now depend on linter job first  https://review.opendev.org/69335519:03
EmilienMade_lee_: from the undercloud19:04
Tenguhjensas: hmmm.... centos 7 doesn't have firewalld, I'm sure of that19:04
Tengucan't check anything right now since I'm on my private laptop - but I'll give a look tomorrow and check what's going on with a clean deploy.19:04
hjensas[root@undercloud ~]# rpm -q firewalld.noarch19:05
hjensaspackage firewalld.noarch is not installed19:05
Tenguhjensas: yeah, so centos-7 doesn't have firewalld19:05
Tenguthat's integrated since centos-819:05
Tenguand probably the same for rhel, cloudnull -^19:05
cloudnullhjensas I just rebooted an undercloud I had running deployed from master. the rules didn't disappear for me.19:07
hjensascloudnull: ip6tables -L INPUT -v ?19:07
cloudnulli diffed iptables-save19:07
* cloudnull will rebuild on rhel8 to verify 19:08
hjensascloudnull: did you check what's in runtime? for both iptables and ip6tables?19:10
cloudnullno just iptables19:10
cloudnullI have the diff if you want to take a look19:11
hjensascloudnull: check ip6tables as well.19:11
cloudnullI didn't make a backup of ip6tables19:11
cloudnullbefore the reboot that is19:11
*** florianf has quit IRC19:11
hjensasbut after reboot, whats in ip6tables?19:11
cloudnullthere's a mess of rules. lots of have the same comments as iptables19:12
cloudnullbut I dont have the diff19:12
* cloudnull re-running the deployment on that node now 19:12
cloudnullhjensas - https://pasted.tech/pastes/115b7486bda2e2e051ad631a3d283f91585d68fe19:15
cloudnulllooking at the save file19:16
cloudnullfor ip6tables19:16
cloudnullit seems to be working ?19:16
ade_lee_EmilienM, so looking at the scale_tasks in https://github.com/openstack/tripleo-heat-templates/blob/dd6cc8e3ad40c05673c30e6cba09c102ff6fa40f/deployment/nova/nova-compute-container-puppet.yaml ..19:18
*** pcaruana has joined #tripleo19:18
hjensascloudnull: yes, that looks ok. And if you reboot the node and run ip6tables -L to list the rules. Are they there? In my case they are not. I'm redoing my lab from scratch atm.19:19
ade_lee_it looks like these are tasks that are running on the undercloud (delegate: localhost) , but that have the host_vars for the host being deleted (ansible_fqdn)?19:19
*** TrevorV has quit IRC19:20
cloudnullhjensas yes those saved rules are present.19:20
* cloudnull now looking how we can better ensure that they're restored. 19:20
cloudnullhjensas can you check if ip6tables.service is running ?19:22
hjensascloudnull: I'm redeploying, will check once it's complete.19:23
openstackgerritBrent Eagles proposed openstack/tripleo-ansible master: Scan existing controllers on update to pick up existing CA/private key pairs  https://review.opendev.org/69259919:24
*** holser has quit IRC19:30
*** alexmcleod has quit IRC19:33
openstackgerritSagi Shnaidman proposed openstack/python-tripleoclient master: WIP: support standalone ansible.cfg addition and extra args  https://review.opendev.org/69913719:34
*** hberaud is now known as hberaud|gone19:37
*** TrevorV has joined #tripleo19:37
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Implement utils.copy_clouds_yaml  https://review.opendev.org/69944819:40
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Revert "Added for support for generating overcloud clouds.yaml"  https://review.opendev.org/69932919:40
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639019:40
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role handlers  https://review.opendev.org/69948619:42
cloudnull^ hjensas makes the handlers more explicit. mind giving that a spin ?19:42
mwhahahawe need to disable container image prepare debug in ci19:45
mwhahahaat least the part that outputs it to the std out19:45
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Add undercloud.conf enable_nova_metadata  https://review.opendev.org/69832519:47
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart-extras master: WIP: Allow full path to be specified for amphora destination  https://review.opendev.org/69285819:50
*** ianw is now known as ianw_pto19:50
*** pcaruana has quit IRC19:59
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Optionally set undercloud.conf enable_nova_metadata  https://review.opendev.org/69832620:00
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM test undercloud_enable_nova_metadata  https://review.opendev.org/69832720:01
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo-keystone-resources: handle multiple roles per user  https://review.opendev.org/69928620:11
hjensascloudnull: thanks, I will.20:12
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update firewall role handlers  https://review.opendev.org/69948620:12
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698020:13
*** mcornea has quit IRC20:14
*** mcornea has joined #tripleo20:15
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update regex for presistent save roles  https://review.opendev.org/69948920:15
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Update regex for presistent save roles  https://review.opendev.org/69948920:15
cloudnullhjensas mind taking a look at that one too20:15
cloudnullyou mentioned that bug report we were pulling out the wrong rules in the persistent rule prune: I think update should do what we need .20:16
hjensascloudnull: right, looks promising. I'll verify it. Thanks!20:18
cloudnulllooking at https://regex101.com/r/CyTfKz/1/20:19
cloudnullit seems to do what I'd expect given the comment in the bug report, but I'd appreciate your feedback20:19
*** rascasoft has quit IRC20:24
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Implement utils.copy_clouds_yaml  https://review.opendev.org/69944820:24
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy /etc/openstack/clouds.yaml with Ansible  https://review.opendev.org/69639020:24
*** rascasoft has joined #tripleo20:24
cloudnullmwhahaha ran into that same error again20:25
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698020:25
cloudnullthe ansible_ssh_host_key_rsa_public option not being defined20:25
* mwhahaha shrugs20:25
cloudnullI can see that there are three cached fact files, all of them have the key20:25
mwhahahais that an old var name? did soemthing change?20:26
cloudnullnope, brand new deployment20:26
mwhahahait's also a loop20:26
mwhahahaperhaps needs a check20:26
*** holser has joined #tripleo20:28
openstackgerritMartin Mágr proposed openstack/puppet-tripleo master: Fix typo in sslProfile conditional  https://review.opendev.org/69949120:28
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Adjust install path  https://review.opendev.org/69949220:35
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Fix typos in hiera values  https://review.opendev.org/69949320:36
openstackgerritMerged openstack/python-tripleoclient stable/rocky: Retry the connection to heat-api when 500 code  https://review.opendev.org/69890220:37
*** Chaserjim has joined #tripleo20:38
openstackgerritMerged openstack/tripleo-upgrade stable/rocky: Update: Ensure parallel execution  https://review.opendev.org/69637620:39
openstackgerritMerged openstack/tripleo-upgrade stable/rocky: Update: Run all roles update at once  https://review.opendev.org/69637720:39
openstackgerritMerged openstack/tripleo-heat-templates stable/train: HA: minor update of arbitrary container image name  https://review.opendev.org/69827420:39
openstackgerritJohn Fulton proposed openstack/tripleo-validations master: Introduce Ceph placement group validation  https://review.opendev.org/69467320:40
*** pcaruana has joined #tripleo20:42
*** rascasoft has quit IRC20:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Manage all Keystone resources with Ansible  https://review.opendev.org/69698020:45
*** jaosorior has quit IRC20:45
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Adjust install path  https://review.opendev.org/69949220:47
*** gfidente has quit IRC20:48
openstackgerritRonelle Landy proposed openstack/ansible-role-collect-logs master: WIP: Add functionality to sanitize collected logs  https://review.opendev.org/69199420:52
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix tenancy lower constrain  https://review.opendev.org/69949920:52
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451220:52
*** pkopec has quit IRC20:55
*** kozhukalov has joined #tripleo21:02
*** kopecmartin is now known as kopecmartin|off21:04
*** mcornea has quit IRC21:04
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Programatically generate ssh hosts lines  https://review.opendev.org/69950221:07
*** mcornea has joined #tripleo21:07
openstackgerritMerged openstack/tripleo-operator-ansible master: Adjust install path  https://review.opendev.org/69949221:07
*** jaosorior has joined #tripleo21:09
cloudnullmwhahaha did you happen to give https://review.opendev.org/#/c/699315/ another spin ?21:10
mwhahahano let me grab it real quick21:10
mwhahahafail creating teh dest21:12
mwhahahaoh wrong host21:12
mwhahahanm21:12
mwhahahajinja'd21:13
mwhahahawe need to fix that21:13
mwhahahaworked once i set teh var21:14
mwhahahakinda annoying that it parses the jinja bits21:14
*** jaosorior has quit IRC21:15
cloudnull++21:15
cloudnullits treating it as a template21:15
cloudnull:)21:15
cloudnulldo we use that jinja21:15
cloudnullor is that just as an example ?21:15
* cloudnull sees that its all commented 21:16
* cloudnull curious if we render that data ever? 21:16
mwhahahait's just an example21:16
mwhahahaif a user uses that content it can render21:16
mwhahahait's an override bit, we should move it to real documentation instead of comments21:17
cloudnullwe might be able to make config_template not render the template data ?21:17
mwhahahawould be nice21:17
cloudnulllike an option to ignore template data and just copy ?21:17
mwhahahathat'd be ideal for this usage21:18
cloudnullby default it uses the template module but maybe we can make that configurable ...21:18
* cloudnull goes to look 21:19
*** suuuper has quit IRC21:21
mwhahahawould be useful for older files where we wouldn't be able to cleanup the jinja bits21:24
*** paramite|clone has quit IRC21:26
*** raildo has quit IRC21:27
*** kozhukalov has quit IRC21:29
openstackgerritMerged openstack/puppet-pacemaker master: Add support to configure pcsd bind address  https://review.opendev.org/69794221:32
*** TrevorV has quit IRC21:40
*** pcaruana has quit IRC21:41
openstackgerritMerged openstack/tripleo-ansible master: Update firewall role to use an action plugin  https://review.opendev.org/69862021:43
*** morazi has quit IRC21:46
*** rcernin has quit IRC21:46
openstackgerritJohn Fulton proposed openstack/tripleo-common master: Increase randomness of generated CephFSID  https://review.opendev.org/69950521:48
*** dpawlik has quit IRC21:53
cloudnullmwhahaha - https://review.opendev.org/69950621:53
cloudnullgive that a spin21:53
cloudnullset render_template false21:53
cloudnullwithin the task21:53
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix lower constraints  https://review.opendev.org/69949921:53
*** stevebaker_ is now known as stevebaker21:54
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451221:54
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/stein: [stein only] WIP Make podman update from 1.0.5 to 1.4.2.  https://review.opendev.org/69925521:55
*** rascasoft has joined #tripleo21:55
mwhahahacloudnull: hey that worked21:57
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates stable/train: HA: enable cluster-common-tag naming scheme by default  https://review.opendev.org/69951122:01
cloudnullmwhahaha jinja should still work in the overrides, however it wont render anything in the src file22:05
mwhahahayea that's fine22:06
*** panda has quit IRC22:07
*** mcornea has quit IRC22:09
*** panda has joined #tripleo22:10
*** ade_lee__ has joined #tripleo22:18
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix lower constraints  https://review.opendev.org/69949922:19
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451222:19
*** ade_lee_ has quit IRC22:21
*** ade_lee__ has quit IRC22:24
*** ade_lee has joined #tripleo22:24
*** holser has quit IRC22:25
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix lower constraints  https://review.opendev.org/69949922:26
*** rh-jelabarre has quit IRC22:26
*** jfrancoa has quit IRC22:26
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451222:26
*** ade_lee has quit IRC22:30
*** holser has joined #tripleo22:31
*** slaweq has quit IRC22:37
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix lower constraints  https://review.opendev.org/69949922:38
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451222:41
*** rcernin has joined #tripleo22:42
openstackgerritMerged openstack/tripleo-heat-templates master: Create /var/log/journal directory during step-0  https://review.opendev.org/69942522:48
*** ade_lee has joined #tripleo22:51
*** ade_lee has quit IRC22:56
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Fix lower constraints  https://review.opendev.org/69949922:56
openstackgerritAlex Schultz proposed openstack/tripleo-validations master: Stop configuring install_command in tox.  https://review.opendev.org/69451222:56
*** ade_lee has joined #tripleo22:56
*** saneax has joined #tripleo23:03
openstackgerritMerged openstack/tripleo-ansible master: Create Validations logging directory  https://review.opendev.org/69625923:04
openstackgerritMerged openstack/tripleo-ansible master: tripleo-keystone-resources: reduce permissions for clouds.yaml  https://review.opendev.org/69943023:07
openstackgerritMerged openstack/tripleo-ansible master: tripleo-keystone-resources: fix a typo for domains creation  https://review.opendev.org/69927423:07
*** tkajinam has joined #tripleo23:13
*** holser has quit IRC23:19
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Programatically generate ssh hosts lines  https://review.opendev.org/69950223:21
*** jbadiapa has quit IRC23:38

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!