Wednesday, 2018-08-22

« Tuesday, 2018-08-21 Index Thursday, 2018-08-23 »
*** yolanda has quit IRC00:08
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676400:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791000:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)00:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)00:10
*** moshele has quit IRC00:25
*** lblanchard has joined #tripleo00:34
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM - Add another level of parent jobs for zuul v3  https://review.openstack.org/59306300:36
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: [PoC] Move extra playbooks to job definition  https://review.openstack.org/59458300:36
*** tzumainn has quit IRC00:40
*** gbarros has quit IRC01:01
*** dmellado has quit IRC01:22
*** rcernin_ has joined #tripleo01:23
*** gouthamr has quit IRC01:23
*** mschuppert has quit IRC01:24
*** stevebaker has quit IRC01:24
*** rcernin has quit IRC01:25
*** gbarros has joined #tripleo01:44
*** rcernin has joined #tripleo02:00
*** lblanchard has quit IRC02:01
*** rcernin_ has quit IRC02:03
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676402:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791002:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)02:10
openstackgerritMerged openstack/tripleo-quickstart master: Use sudo when working with root home as non root  https://review.openstack.org/59310402:15
openstackgerritMerged openstack/tripleo-heat-templates master: Fix enabled tripleo-validations check  https://review.openstack.org/59023702:15
openstackgerritMerged openstack/python-tripleoclient master: Update reno for stable/rocky  https://review.openstack.org/59426302:15
openstackgerritMerged openstack/tripleo-heat-templates master: Enable collectd to connect to metrics QDR  https://review.openstack.org/57605702:18
openstackgerritMerged openstack/tripleo-quickstart master: we need to allow tripleo-roles to be updated in ci  https://review.openstack.org/59431402:18
*** khyr0n has joined #tripleo02:19
*** rlandy has quit IRC02:25
*** apetrich has quit IRC02:26
*** dhill_ has quit IRC02:34
*** gouthamr has joined #tripleo02:36
*** dmellado has joined #tripleo02:38
*** rpioso is now known as rpioso|afk02:43
*** edmondsw has quit IRC02:45
*** mmethot has quit IRC02:46
*** stevebaker has joined #tripleo02:48
*** mmethot has joined #tripleo02:49
*** gbarros has quit IRC02:49
*** psachin has joined #tripleo02:51
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Fix HASH for periodic jobs(use tripleo-ci-testing)  https://review.openstack.org/59338303:01
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676403:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791003:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)03:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)03:10
*** sanjayu_ has joined #tripleo03:20
*** khyr0n has quit IRC03:31
openstackgerritMerged openstack/tripleo-quickstart master: rocky is not yet promoted, adjust release file  https://review.openstack.org/59435703:42
*** Petersingh has joined #tripleo03:44
*** Petersingh is now known as Petersingh|afk03:56
*** morazi has quit IRC04:05
*** khyr0n has joined #tripleo04:09
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676404:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791004:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)04:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)04:10
*** khyr0n has quit IRC04:11
*** khyr0n has joined #tripleo04:12
*** shyamb has joined #tripleo04:15
*** shyamb has quit IRC04:32
*** shyamb has joined #tripleo04:32
shyambHi04:33
shyambIs it good idea to call hiera from puppet code?04:33
shyambfrom puppet template?04:34
*** Petersingh|afk is now known as Petersingh04:35
*** aufi has joined #tripleo04:47
openstackgerritmelissaml proposed openstack/tripleo-specs master: Remove the redundant word  https://review.openstack.org/59479904:47
openstackgerritGoutham Pacha Ravi proposed openstack/tripleo-heat-templates master: Fix bind-mount to manila's bootstrap container  https://review.openstack.org/59480104:52
openstackgerritGoutham Pacha Ravi proposed openstack/tripleo-heat-templates master: Fix bind-mount to manila's bootstrap container  https://review.openstack.org/59480104:52
*** moshele has joined #tripleo05:00
*** moshele has quit IRC05:02
gouthamrhi bug czars, need to mark this triaged please: https://review.openstack.org/#/c/59480105:08
gouthamrsorry, meant the LP: https://bugs.launchpad.net/tripleo/+bug/1788337 :)05:09
openstackLaunchpad bug 1788337 in tripleo "container manila_api_db_sync fails with TLS everywhere" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr)05:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676405:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791005:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)05:10
*** sdake has quit IRC05:13
*** sdake has joined #tripleo05:13
*** aufi has quit IRC05:25
*** mschuppert has joined #tripleo05:25
*** shyamb has quit IRC05:30
*** palkar has joined #tripleo05:30
*** sanjayu_ has quit IRC05:34
*** shyamb has joined #tripleo05:36
*** sdake has quit IRC05:50
openstackgerritCédric Jeanneret proposed openstack/tripleo-specs master: Validation Framework specifications  https://review.openstack.org/58916905:50
*** sdake has joined #tripleo05:51
*** hjensas has quit IRC05:52
*** ykarel has joined #tripleo05:54
*** prometheanfire has left #tripleo05:57
*** ksambor has joined #tripleo05:59
*** sdake has quit IRC06:02
*** sdake has joined #tripleo06:04
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates stable/ocata: DNM: test ocata  https://review.openstack.org/59481006:04
*** moshele has joined #tripleo06:06
openstackgerritKamil Sambor proposed openstack/python-tripleoclient master: Add fixtures instead mock  https://review.openstack.org/57953806:07
*** shyamb has quit IRC06:08
*** shyamb has joined #tripleo06:09
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676406:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791006:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)06:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)06:10
*** jfrancoa has joined #tripleo06:18
*** ccamacho has joined #tripleo06:20
*** ccamacho has joined #tripleo06:20
*** yprokule has joined #tripleo06:20
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: [WIP] logging test DO NOT MERGED  https://review.openstack.org/59483606:22
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates stable/queens: [WIP] logging test DO NOT MERGED  https://review.openstack.org/59483806:30
*** gkadam has joined #tripleo06:31
*** saneax has joined #tripleo06:31
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates stable/pike: [WIP] logging test DO NOT MERGED  https://review.openstack.org/59484006:39
*** marios|rover has joined #tripleo06:40
*** pcaruana has joined #tripleo06:41
*** shyamb has quit IRC06:51
*** shyamb has joined #tripleo06:52
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Abstract container backend  https://review.openstack.org/58466106:52
*** Petersingh is now known as Petersingh|afk06:54
*** hjensas has joined #tripleo06:56
*** agurenko has joined #tripleo06:57
*** amoralej|off is now known as amoralej07:01
*** dmellado has quit IRC07:04
*** Petersingh|afk is now known as Petersingh07:05
*** dmellado has joined #tripleo07:06
*** rcernin has quit IRC07:07
*** mschuppert has quit IRC07:07
*** noama has joined #tripleo07:08
*** bogdando has joined #tripleo07:10
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676407:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791007:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)07:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)07:10
*** dmellado has quit IRC07:12
*** tosky has joined #tripleo07:13
*** Tengu has joined #tripleo07:15
*** sshnaidm|afk is now known as sshnaidm07:17
*** dmellado has joined #tripleo07:19
*** jpich has joined #tripleo07:20
*** dtantsur|afk is now known as dtantsur07:22
*** f1 has joined #tripleo07:23
*** f1 is now known as florianf07:23
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Composable networks - L3 routed networks (1/3)  https://review.openstack.org/58218007:26
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Composable networks - L3 routed networks (2/3)  https://review.openstack.org/58218107:26
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Composable networks - L3 routed networks (3/3)  https://review.openstack.org/58230107:26
*** rdopiera has joined #tripleo07:26
*** shyamb has quit IRC07:27
*** moguimar has joined #tripleo07:30
*** ccamacho has quit IRC07:33
*** Petersingh is now known as Petersingh|lunch07:34
*** akrivoka has joined #tripleo07:34
*** moguimar has quit IRC07:36
*** shardy has joined #tripleo07:38
*** assassin has joined #tripleo07:40
*** jtomasek has joined #tripleo07:41
*** aufi has joined #tripleo07:43
*** tosky has quit IRC07:43
*** tosky has joined #tripleo07:43
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Collect overcloud statistics with ARA  https://review.openstack.org/57846207:44
*** dxiri has joined #tripleo07:46
*** mschuppert has joined #tripleo07:47
*** jpena|off is now known as jpena07:47
*** jaganathan has quit IRC07:52
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676408:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791008:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)08:10
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Increase timeout from 180 mins to 200 mins  https://review.openstack.org/59490508:16
*** amoralej is now known as amoralej|brb08:16
*** ccamacho has joined #tripleo08:16
*** Petersingh|lunch is now known as Petersingh08:18
*** akrivoka has quit IRC08:20
*** ccamacho has quit IRC08:25
*** ccamacho has joined #tripleo08:26
*** holser_ has joined #tripleo08:26
*** pcaruana has quit IRC08:28
*** pcaruana has joined #tripleo08:30
*** shyamb has joined #tripleo08:41
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Remove uneececary conditionals in network-environment jinja  https://review.openstack.org/59490608:43
*** dxiri has quit IRC08:46
*** amoralej|brb is now known as amoralej08:47
openstackgerritSteven Hardy proposed openstack/tripleo-quickstart-extras master: WIP tet changing default for PREPARE_ARGS  https://review.openstack.org/59490708:48
shardymarios: Hey, I pushed ^^ to see if it works, will comment again on the review08:48
*** sri_ has joined #tripleo08:49
sshnaidmdoes anybody know which use runs ansible-playbook when installing overcloud? is it root or tripleo-admin?08:50
shardysshnaidm: it's heat-admin by default IIRC, with some become: true blocks08:51
*** akrivoka has joined #tripleo08:52
shardyyou can see it via grep ansible_ssh_user in tripleo-common FYI08:52
shardyand in workbooks/deployment.yaml (ssh_user)08:52
sri_shardy, quick question is there way i can skip ntp for simple overcloud deployment in stable/queens ?08:53
*** paramite_ has joined #tripleo08:54
shardysri_: you could try either setting the NtpServer parameter to an empty list:08:55
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/time/ntp.yaml#L3608:55
shardyor08:55
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L20708:55
shardyyou could set the OS::TripleO::Services::Ntp service to OS::Heat::None in your resource registry08:56
shardyor you could pass a custom ControllerServices parameter with only the services you want08:56
sri_shardy, got it,   thank you  :)08:57
sri_shardy, the reason I wanted to do this is ntp port 123/udp  is disabled in my network, it will take a 3 to 4 days to complete that firewall request08:58
sshnaidmshardy, hmm.. I see tripleo-admin here in inventory class: https://github.com/openstack/tripleo-common/blob/dfb9c24d86cbdeb95f8ea82d04eb32bb48d54bbf/tripleo_common/actions/ansible.py#L59508:59
*** tosky has quit IRC08:59
*** tosky has joined #tripleo09:00
mariosack thanks shardy will check09:00
sshnaidmshardy, I look at files that are created in mistral folder and they're of user "42430".. is it being deleted..?09:02
sshnaidmbogdando, Tengu maybe you know? ^^09:02
shardysshnaidm: which mistral folder is that?09:03
shardythe inventory output etc is probably generated as the mistral user09:03
jaosorior_bandini: could you check this out https://review.openstack.org/#/c/593491/ ?09:04
shardydtantsur: Hey if you have any notes re https://review.openstack.org/#/c/576856/ I plan to try it today, overall looks good to me09:05
*** moshele has quit IRC09:05
sshnaidmshardy, /var/lib/mistral/overcloud/09:05
shardydid you rebuild all the kolla containers or just the mistral one to test this?09:05
*** moshele has joined #tripleo09:05
sshnaidmd0ugal, do you know if ansible-playbook in overcloud deployment runs with mistral user tripleo-admin or heat-admin..?09:06
shardysshnaidm: interesting, I see the same, but that's created by mistral not ansible IIRC09:06
shardysshnaidm: I wonder if it's a uid mapping issue because mistral is now running in a container09:07
sshnaidmshardy, yeah, container user will have this09:07
shardythe ansible mistral action will now run inside the mistral-executor container, so I guess that explains it09:07
openstackgerritKamil Sambor proposed openstack/python-tripleoclient master: Add fixtures instead mocks  https://review.openstack.org/58134209:08
shardyIIRC we updated the kolla uid mappings in a few places before to avoid it09:08
sshnaidmshardy, I see, so ansible-playbook is actually running from within mistral container..09:08
shardysshnaidm: yes, which is why we recently had to mount some dirs to ensure roles etc from the host are available09:08
shardyit'd be good in due course to move away from that though09:09
sshnaidmshardy, do you know when I can see volume mappings of this container?09:09
shardysshnaidm: sec09:09
*** paramite_ has quit IRC09:09
sshnaidmshardy, now I understand why creating files didn't work :) they were created inside the container..09:09
shardysshnaidm: docker inspect mistral_executor shows them under HostConfig/Binds09:10
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676409:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791009:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)09:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)09:10
*** aufi has quit IRC09:10
sshnaidmshardy, thanks, found it here: http://logs.openstack.org/62/578462/22/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3d98420/logs/undercloud/var/log/extra/docker/containers/mistral_executor/docker_info.log.txt.gz09:12
shardysshnaidm: ah nice, I wasn't sure if we captured that in CI09:13
sshnaidmshardy, yeah, we do docker inspect on all containers09:13
*** akrivoka has quit IRC09:15
*** chem has joined #tripleo09:19
*** akrivoka has joined #tripleo09:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Collect overcloud statistics with ARA  https://review.openstack.org/57846209:20
sshnaidmd0ugal, do you know maybe if user (tripleo-admin) that runs ansible-playbook from mistral executor container have permissions to create files in /var/lib/mistral ?09:21
*** shyam89 has joined #tripleo09:22
d0ugalsshnaidm: Sorry, was in a meeting. ansible-playbook should run as the mistral user.09:22
d0ugalsshnaidm: I believe they should have permissions - isn't that the reason that folder exists? but I'm not sure.09:23
*** amoralej is now known as amoralej|brb09:24
*** akrivoka has quit IRC09:24
sshnaidmd0ugal, yeah, seems like so, thanks09:25
*** leanderthal has joined #tripleo09:25
sshnaidmd0ugal, but if I want to use some special plugin for ansible, seems like I need to install it in container, right? I don't see mapping of python module there..09:26
*** shyamb has quit IRC09:27
chandankumard0ugal: Hello09:27
chandankumard0ugal: Can we do a tag release for mistral-tempest-plugin?09:27
*** ykarel_ has joined #tripleo09:28
*** akrivoka has joined #tripleo09:29
dtantsurshardy: will post something today; got stuck yesterday with trying to update code in a container.. ended up screwing my undercloud twice09:30
*** ykarel has quit IRC09:31
*** akrivoka has quit IRC09:33
shardydtantsur: ack yeah that was the part I was unsure about, happy to help figure it out if needed though09:33
*** moshele has quit IRC09:34
d0ugalsshnaidm: correct, I am not sure how you would install it in the container09:34
d0ugalchandankumar: Sure09:35
chandankumard0ugal: I am prepating the patch then for release09:35
d0ugalchandankumar: great, let me know and I'll +1 it09:35
dtantsurshardy: I ended up with this script (run from tripleo-common checkout): http://paste.openstack.org/show/728578/09:35
d0ugalchandankumar: I'm going out for a bit, back in ~1hr09:35
dtantsurI'll post the remainign instructions later09:35
d0ugaldtantsur: neat.09:36
d0ugalwell, messy but also neat ;)09:36
dtantsurno, it's NOT neat :D it's horrible, but it works09:36
d0ugallol09:36
dtantsurI especially like retries around 'docker cp', because it fails without clear reason A LOT09:36
d0ugalugh09:36
* d0ugal will bbiab09:37
shardywow, yeah, err, well whatever works I guess :)09:37
shardyI was thinking we'd build a new tripleo-common rpm with dlrn then rebuild the container referencing the local repo09:38
shardybut not got any step-by-step notes on that09:38
*** aevoo has joined #tripleo09:38
shardyhaving it automated somewhere would be helpful, since we do that in CI and all developers must need to do the same sometimes09:38
dtantsurshardy: this is going to be much longer, I suspect.. but I'd be glad to see a script09:38
shardydtantsur: yeah, I'll take a look at what CI does when I've had more coffee ;)09:39
shardyI know the rpm build part, it's the container rebuild with repo override that I'm not sure on09:39
shardydtantsur: the other option is to just test it with mistral on baremetal I guess ;)09:40
shardyI'll take a look at the container rebuild thing though09:41
*** Petersingh_ has joined #tripleo09:41
*** akrivoka has joined #tripleo09:42
*** Mantorok has quit IRC09:42
sshnaidmd0ugal, if ara has rpm, I suppose it should be done somewhere here? https://github.com/openstack/tripleo-common/blob/master/container-images/tripleo_kolla_template_overrides.j2#L4809:42
*** Mantorok has joined #tripleo09:43
*** moshele has joined #tripleo09:45
*** Petersingh has quit IRC09:45
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Replace call to map-project-name script with rdopkg  https://review.openstack.org/55261209:48
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Replace call to map-project-name script with rdopkg  https://review.openstack.org/55261209:49
*** akrivoka has quit IRC09:53
*** shyam89 has quit IRC09:54
*** ccamacho has quit IRC09:54
*** ccamacho has joined #tripleo09:55
shardydtantsur: FYI https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html#testing-a-code-fix-in-a-container shows how to rebuild the container (kudos bandini!)09:55
*** amoralej|brb is now known as amoralej09:56
*** ykarel_ is now known as ykarel09:58
dtantsurshardy: sure, but involved rebuilding the undercloud.. and from my experience, you also need to manually kill the existing container, since the installer may not replace it.09:58
*** akrivoka has joined #tripleo09:59
*** akrivoka has quit IRC10:01
*** shyam89 has joined #tripleo10:02
dtantsurshardy: re "testing mistral on baremetal", building non-containerized underclouds with quickstart is no longer possible..10:03
*** shyamb has joined #tripleo10:03
*** jaosorior_ has quit IRC10:05
*** akrivoka has joined #tripleo10:06
*** akrivoka has quit IRC10:06
*** akrivoka has joined #tripleo10:06
*** shyam89 has quit IRC10:07
shardydtantsur: I'm pretty sure it would be possible, but just not using instack-undercloud, I'll give it a try later10:09
shardye.g you could kill the container then re-run the deploy with the heat config pointing at the puppet/services/ files for mistral10:09
shardyI know that's become a poorly tested path in CI lately though10:10
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676410:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791010:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)10:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)10:10
dtantsurshardy: wonderful, now hitting a missing dependency of ironicclient10:11
shardydtantsur: Hmm, don't we already use that for the various baremetal actions?10:12
dtantsurshardy: I don't think we build configdrives10:13
shardydtantsur: Hmm, well all the nova deployed nodes have config drives, but I'm not sure how/where that happens10:13
dtantsurshardy: it may be a dependency of the nova package10:14
shardyyeah I guess it must be as it contains the nova user/meta-data10:14
shardybut I was thinking of all the provide workflow etc10:14
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates stable/pike: DNM: test pike 004  https://review.openstack.org/59492510:19
dtantsurshardy: this stuff does not use configdrive building feature of ironicclient, but metalsmith does10:20
*** Petersingh_ is now known as Petersingh10:20
*** aufi has joined #tripleo10:21
shardydtantsur: ack10:22
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Replace call to map-project-name script with rdopkg  https://review.openstack.org/55261210:22
dtantsurshardy: posted a comment with rough testing instructions10:23
shardydtantsur: thanks!10:23
Tenguhmm. that selinux thing in container is a pain.10:28
Tenguthere are some weird things.10:28
*** ykarel is now known as ykarel|lunch10:29
*** ykarel|lunch is now known as ykarel|away10:34
d0ugalsshnaidm: Interesting, I hadn't see that file before! That does look like the correct place.10:37
d0ugalchandankumar: Did you create the release patch?10:37
*** sanjayu_ has joined #tripleo10:53
*** saneax has quit IRC10:53
*** sanjayu__ has joined #tripleo10:55
*** apetrich has joined #tripleo10:58
*** sanjayu_ has quit IRC10:58
*** boazel has joined #tripleo10:59
*** electrichead has quit IRC10:59
*** slagle has quit IRC11:01
*** Petersingh is now known as Petersingh|afk11:04
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676411:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791011:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)11:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)11:10
*** shyamb has quit IRC11:11
*** dxiri has joined #tripleo11:12
*** akrivoka_ has joined #tripleo11:14
*** akrivoka_ has quit IRC11:14
*** jaosorior has joined #tripleo11:15
*** shyamb has joined #tripleo11:17
chandankumard0ugal: https://review.openstack.org/#/c/594930/11:17
*** ccamacho has quit IRC11:18
*** ykarel|away has quit IRC11:19
*** jpena is now known as jpena|lunch11:20
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart master: allow tripleo-roles to be updated in ci  https://review.openstack.org/59493311:24
*** akrivoka_ has joined #tripleo11:24
*** ccamacho has joined #tripleo11:25
ssbarnea|ruckmarios: ^^ i hope that is what was missing.11:25
tbarronjaosorior: bogdando: bandini: pls review https://review.openstack.org/#/c/594801 which11:27
tbarronjaosorior: bogdando: bandini is like https://review.openstack.org/#/c/539498/ but for manila rather than cinder11:28
jaosoriortbarron: sure, have you had a chance to test it out?11:28
tbarronjaosorior: gouthamr deployed with it and the bind mount is there11:28
*** abishop has quit IRC11:28
tbarronjaosorior: his deployment has some issue with the api connection so I'm redeploying fresh this morning11:29
tbarronjaosorior: overcloud deploy completed, manila api service is running fine with the bind mount for the db11:29
tbarronjaosorior: but he may not have the free ipa setup and connection just right, seems independent of this fix though11:30
*** akrivoka_ has quit IRC11:30
tbarronjaosorior: also he had to redeploy with telemetry disabled since it appears to have the same issue :(11:32
marios|roverssbarnea|ruck: ack lets see what ci makes of it (the master job was also failing but for different reason so it might not make them green yet ;) )11:32
marios|roverssbarnea|ruck: but yeah that's what i meant in the comment11:33
*** rh-jelabarre has joined #tripleo11:33
*** dhill_ has joined #tripleo11:33
*** edmondsw has joined #tripleo11:33
tbarronjaosorior: bogdando: ty11:34
*** Petersingh|afk is now known as Petersingh11:37
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: test fs008 tempest  https://review.openstack.org/59493411:38
sshnaidmtosky, just interesting if it will work ^^11:38
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates stable/ocata: DNM: test ocata  https://review.openstack.org/59481011:39
*** saneax has joined #tripleo11:39
*** sanjayu__ has quit IRC11:42
*** saneax has quit IRC11:42
openstackgerritAdriano Petrich proposed openstack/tripleo-quickstart-extras master: WIP validate that the mistral error is still happening.  https://review.openstack.org/59493511:43
*** morazi has joined #tripleo11:45
bandinitbarron: ops was too slow ;)11:51
sshnaidmmwhahaha, hi, in which step do we update containers in CI now with that new ansible role?11:52
sshnaidmEmilienM, stevebaker ^^11:53
shardysshnaidm: which new ansible role?  For overcloud deployments the containers are always started/restarted via paunch in whatever step the templates define them to start11:53
sshnaidmshardy, I mean https://github.com/openstack/ansible-role-tripleo-modify-image11:54
*** leanderthal has quit IRC11:55
sshnaidmseems like I'm not familiar with that new workflow..11:55
sshnaidmweshay, ^^11:55
*** raildo has joined #tripleo11:56
weshaysshnaidm, that is executed in the undercloud install11:56
weshayhttp://git.openstack.org/cgit/openstack/tripleo-docs/tree/doc/source/install/advanced_deployment/container_image_prepare.rst#n8311:58
*** boazel has quit IRC11:58
sshnaidmweshay, trying to find logs of that..11:58
tbarronbandini: ty anyways!11:59
jaosorior#startmeeting TripleO Security Squad12:00
openstackMeeting started Wed Aug 22 12:00:22 2018 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.12:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.12:00
*** openstack changes topic to " (Meeting topic: TripleO Security Squad)"12:00
openstackThe meeting name has been set to 'tripleo_security_squad'12:00
*** lhinds has joined #tripleo12:00
jaosoriorhey lhinds !12:01
weshaysshnaidm, it's in the stinkin journal12:01
weshaysshnaidm, https://review.openstack.org/#/c/593716/12:01
jaosoriorraildo, redrobot, are you around?12:01
lhindsjaosorior: hey12:01
raildoo/12:01
jaosoriorI'll wait a bit to see if more folks log in12:02
jaosoriorTengu: ^^12:02
Tengujaosorior: :)12:02
*** abishop has joined #tripleo12:03
*** lblanchard has joined #tripleo12:03
*** moguimar has joined #tripleo12:04
moguimaro/12:04
jaosorioro/12:04
jaosoriorAlright, lets start12:04
jaosorior#topic SELinux for containers12:04
*** openstack changes topic to "SELinux for containers (Meeting topic: TripleO Security Squad)"12:04
TenguEmilienM: -^  might be interesting for you as well :)12:05
jaosoriorSo, Tengu has been involved with work related to moving from docker to podman, and on the way, he started checking out the SELinux integration for containers, a topic which we had in our view12:05
jaosoriorTengu: IIRC, you started looking at this by running the containerized undercloud with podman, and SELinux enabled, right?12:05
Tenguindeed. Following EmilienM first steps in order to get an undercloud running on podman instead of docker.12:06
jaosoriorSo, Tengu run into some SELinux issues, which he tracked down in this trello card https://trello.com/c/hNkI15a7/1-selinux-issues12:06
*** amoralej is now known as amoralej|lunch12:06
jaosoriorand they're not as much issues as I expected :D12:06
chandankumarRuck/Rover https://review.rdoproject.org/r/#/c/15732/ we are updating tempestconf to 2.0.0 in queens if you see any failures related to temepstconfiguraiton let us know12:06
*** electrichead has joined #tripleo12:06
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Fix network lines rendering  https://review.openstack.org/59493812:06
Tenguindeed, but they are nasty :).12:06
jaosoriorthey are12:07
jaosoriorSo, whoever is interested in working on this, please contact me and Tengu about it, so we can put you up to speed. It's quite interesting work :D12:07
Tengusome of the issues are "normal" and require a specific selinux policy.12:07
Tengubut most of them exists only because of bad practices :]12:08
jaosoriorright, we need to divide which of these issues should we rectify ASAP, and which should have an "exception" in the selinux policy (and hopefully get fixed later)12:08
jaosoriorthe first of the issues is the way we use docker/podman with puppet in order to generate the configurations; basically we bind-mount the /etc/puppet directory (which has a selinux label of etc_t) into the container to a temporary directory, and then attempt to copy that into the /etc/puppet directory in the container12:09
jaosoriorthis is not allowed, as containers only have access to reading and executing stuff with etc_t12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676412:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791012:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)12:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)12:10
jaosoriorcreating stuff from it, and writing to it, isn't allowed12:10
jaosoriorwith the hopes of following an approach where we'll have an immutable host, we're going with the approach of having dedicated hieradata for the containers12:10
Tengualso, the first thing docker-puppet.sh does is an rm -rf /etc/puppet/ssl directory in the container.12:10
Tengusecond step is to add a file in the /etc/puppet/hieradata directory12:11
Tenguboth actions are forbidden.12:11
Tengufirst one can be avoided by copying only wanted files. second one is trickier.12:11
*** ssbarnea|ruck has quit IRC12:13
jaosoriorright, though having dedicated hieradata for the containers would solve these issues (I think) since we would then have those files with the needed selinux labels, and copy them as needed, trying to keep docker-puppet.py's functionality12:13
jaosoriorSo, either we copy the hieradata to a temp location on the host, and relabel that, or we straight generate the hieradata on a container volume12:13
*** leanderthal has joined #tripleo12:14
jaosoriorAnyway, this is the stuff that we've been discussing lately, and we'll keep this trello card updated: https://trello.com/c/hNkI15a7/1-selinux-issues12:15
jaosoriorAny questions/feedback/interest in this topic?12:15
weshaysshnaidm, that patch is failing on file not found12:16
weshaynot sure why12:16
sshnaidmweshay, because you use "shell: |" and need to use "shell: >"12:17
weshaybah..12:17
jaosorior#topic Secret Management update12:18
openstackgerritSagi Shnaidman proposed openstack/ansible-role-tripleo-modify-image master: log modify image to a log file for humans  https://review.openstack.org/59371612:18
*** openstack changes topic to "Secret Management update (Meeting topic: TripleO Security Squad)"12:18
jaosoriorredrobot, moguimar, raildo: anything you wanna bring up on this topic?12:18
sshnaidmweshay, ^^12:18
moguimaro/12:18
moguimarstarted working on the castellan drive12:18
moguimarfound out today that the castellan-vault tests are not working12:18
moguimaralready diagnosed the cause12:18
moguimarfrom vault 0.10.0 forward there is a change in the API12:19
raildojaosorior, not from my side12:19
openstackgerritSagi Shnaidman proposed openstack/ansible-role-tripleo-modify-image master: log modify image to a log file for humans  https://review.openstack.org/59371612:19
moguimarso castellan fails to talk to a vault server >= 0.10.012:19
jaosoriormoguimar: so, is the castellan driver broken as well?12:19
moguimaryep, I filed a bug on launchpad12:20
jaosoriorcrap12:20
jaosoriorthanks for filing it12:20
moguimar#link https://bugs.launchpad.net/castellan/+bug/178837512:20
openstackLaunchpad bug 1788375 in castellan "API changes in vault 0.10.0 causes test to fail." [Undecided,New]12:20
*** jpena|lunch is now known as jpena12:20
openstackgerritSagi Shnaidman proposed openstack/ansible-role-tripleo-modify-image master: log modify image to a log file for humans  https://review.openstack.org/59371612:20
moguimarbut the fix is quite simple12:20
*** dprince has joined #tripleo12:20
moguimarI already have a fix in progress12:20
moguimartests passing and all, just need to make it backward compatible with vault < 0.10.012:21
jaosoriorade_lee: are we supposed to have access to castellan's launchpad?12:21
jaosoriormoguimar: when you have a fix let me know. thanks for working on this.12:21
*** trown|outtypewww is now known as trown12:21
moguimarI've analyzed the wireshark logs and the vault client itself does some http request to fetch API version12:21
jaosoriormoguimar: right, it does discovery. Maybe we can do that upon first interaction and cache the result.12:22
moguimarso I'll add the same behaviour in the castellan driver12:22
moguimaryep12:22
jaosoriorawesome12:22
jaosoriormoguimar: thanks for this work12:22
moguimaro/12:22
jaosorior#topic Any other business12:24
*** openstack changes topic to "Any other business (Meeting topic: TripleO Security Squad)"12:24
jaosoriorAnything else folks want to bring up to the meeting?12:24
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: WIP: Update default for THT resource registry  https://review.openstack.org/59494412:27
jaosoriorAlright folks! thanks for joining!12:28
moguimaro/12:28
electricheadthanks jaosorior12:28
jaosoriorJust a reminder, the security squad meeting is now every two weeks12:28
jaosoriorso, talk to you here in two weeks!12:28
jaosorior#endmeeting12:28
*** openstack changes topic to "Welcome to Rocky | CI Status: GREEN | https://docs.openstack.org/tripleo-docs/latest/"12:28
moguimaryup12:28
openstackMeeting ended Wed Aug 22 12:28:48 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)12:28
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-08-22-12.00.html12:28
weshayanyone have the link to the fedora tripleo etherpad?12:28
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-08-22-12.00.txt12:28
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-08-22-12.00.log.html12:28
*** agopi has quit IRC12:28
*** dxiri has quit IRC12:29
*** thrash|g0ne is now known as thrash12:30
*** akrivoka_ has joined #tripleo12:30
*** akrivoka has quit IRC12:31
*** rlandy has joined #tripleo12:32
*** psachin has quit IRC12:32
*** akrivoka has joined #tripleo12:32
*** akrivoka has quit IRC12:33
*** shyamb has quit IRC12:35
*** toure has joined #tripleo12:38
*** rnoriega_ has joined #tripleo12:38
*** shyamb has joined #tripleo12:42
*** akrivoka has joined #tripleo12:44
openstackgerritKamil Sambor proposed openstack/python-tripleoclient master: Add fixtures instead mock  https://review.openstack.org/57953812:44
Tengujaosorior: I was right from the very start in fact.12:47
*** tzumainn has joined #tripleo12:47
jaosoriorTengu: ?12:47
Tengujaosorior: cp: failed to restore the default file creation context: Permission denied12:47
Tengu+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /etc/puppet/12:47
Tenguso it's not even the fact we actually push the "docker.json" with the step.12:48
Tenguit's the fact we can't restore properly the labels.12:48
Tengui.e. the cp itself12:48
Tenguand doing temporary directory will probably NOT do any good12:48
jaosoriorwhat context do they have?12:48
Tenguwait, modifing.12:48
*** agopi has joined #tripleo12:56
*** tbonds has joined #tripleo12:56
Tengujaosorior: http://paste.openstack.org/show/728590/12:58
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Collect overcloud statistics with ARA  https://review.openstack.org/57846212:58
Tengujaosorior: this is from the container directly.12:58
*** Petersingh has quit IRC12:58
TenguMAYBE it's an issue with the container and libpod.12:58
dtantsurfolks, I have some bad news. It seems that the very basic and default deployment of a rocky overcloud fails with: sudo: no tty present and no askpass program specified13:00
dtantsurthis is on running: Command: /usr/bin/sudo /bin/nova-manage cell_v2 discover_hosts --verbose13:00
*** jcoufal has joined #tripleo13:01
openstackgerritBrent Eagles proposed openstack/puppet-tripleo master: Use logrotate copytruncate with neutron services  https://review.openstack.org/58266113:01
*** aevoo has quit IRC13:02
*** ssbarnea has joined #tripleo13:03
*** boazel has joined #tripleo13:03
Tengujaosorior: replacing the "cp -a" with a "cp -dR" seems to make it go far, far further.13:03
*** amoralej|lunch is now known as amoralej13:05
jaosorioruhm13:06
jaosoriorinteresting13:06
jaosoriormaybe this is just the way it's supposed to be13:07
jaosoriorand we just didn't take into account the container's SELinux context13:07
*** ssbarnea is now known as ssbarnea_13:07
Tengu^^13:07
jaosoriorTengu: notice c266,c558 for /etc/puppet13:07
dtantsurshardy: are you aware of any problems with rocky deployment? (see above)13:07
Tengujaosorior: yep, because it's create from within the container13:07
jaosoriorwhile /tmp/puppet-etc has an c0,c0 (which doesn't show up cause it's the default... which comes from the host)13:07
Tenguyep13:08
Tengujaosorior: and the error is: can't restore.13:08
jaosoriorI completely disregarded that13:08
jaosorioryeah, that's the expected behavior then13:08
Tenguyep13:08
Tengusooo13:08
Tengudrop -a13:08
Tenguadd -dR13:08
Tenguand we're good.13:08
jaosoriorsounds good13:08
Tenguin fact -a == -dR --preserve=ALL13:08
Tenguwe can --preserve some other things, but not 100% sure it's needed.13:09
Tengurunning a new deploy with the modified code.13:09
Tenguwill check what's the next error.13:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676413:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,In progress] - Assigned to Sorin Sbarnea (ssbarnea)13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791013:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)13:10
shardydtantsur: not aware, CI looks OK and I deployed yesterday and things seemed to be working - when do you see that error?13:10
jaosoriorshardy, bandini could I get a review for this https://review.openstack.org/#/c/593491/ ?13:12
*** boazel has quit IRC13:12
*** rh-jelabarre has quit IRC13:13
*** ssbarnea_ has quit IRC13:13
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates master: Bind mount the he database client settings in sahara_db_sync  https://review.openstack.org/59498013:13
*** ssbarnea_ has joined #tripleo13:13
*** mjturek has joined #tripleo13:14
Tengujaosorior: crossing fingers, I'm at the "step 1". Hopefully it will do what I want :).13:15
Tenguthere's also an "rsync" command, will check what's going on with that one.13:15
*** ssbarnea_ has quit IRC13:16
*** ssbarnea|ruck has joined #tripleo13:16
dtantsurshardy: on trying to deploying anything on a clean undercloud.. literally just openstack overcloud deploy --templates13:17
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates master: Bind mount the database client settings in sahara_db_sync  https://review.openstack.org/59498013:18
*** ssbarnea|ruck has quit IRC13:18
*** ssbarnea|ruck has joined #tripleo13:18
Tengujaosorior: rsync: opendir "/etc/pki/ca-trust/extracted" failed: Permission denied (13)  next error :).13:19
openstackgerritDaniel Alvarez proposed openstack/tripleo-quickstart master: DNM: OVN tempest test  https://review.openstack.org/58783013:20
jaosoriorTengu: funky, I wonder what happened there.13:20
jaosorioroh yeah13:20
jaosoriorcert_t13:20
jaosoriorso that I do think we should add to the policy13:20
*** etingof has joined #tripleo13:20
Tenguprobably yes.13:20
Tenguwill audit2allow in order to check.13:21
*** shyamb has quit IRC13:21
Tenguallow container_t cert_t:dir read;    allow container_t cert_t:file read;13:21
Tenguseems fine.13:21
*** etingof has quit IRC13:21
Tengujaosorior: hmmm. allow container_t security_t:file write;   ??13:21
jaosoriorno13:21
Tenguah, wait.13:21
jaosoriorsecurity_t is for selinux13:21
Tenguyes13:21
jaosoriorcert_t is for certs13:21
Tenguwill truncate the audit.13:22
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates master: Move nova-metadata api to httpd wsgi  https://review.openstack.org/58262313:26
*** akrivoka_ has quit IRC13:28
*** hjensas has quit IRC13:28
openstackgerritMartin Schuppert proposed openstack/puppet-tripleo master: Do not create metadata ssl proxy if we have metadata api via httpd wsgi  https://review.openstack.org/59500113:30
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Avoid SELinux issues with the "cp -a" command  https://review.openstack.org/59500313:31
Tengujaosorior: EmilienM -^^13:31
*** shardy is now known as shardy_mtg13:31
sri_dsneddon, beagles, i am trying to deploy overcloud with dvr enabled in stable/qeens , I've created my roles with  /roles/ComputeDVR.yaml and I've generated network templates using  ./process-templates.py , but in compute node os-net-config not generating based on my computedvr.yaml config  it's generating generic compute templates(default). here is my config http://paste.openstack.org/show/728594/13:33
*** jcoufal has quit IRC13:33
Tengujaosorior: so basically, now, the policy looks like: http://paste.openstack.org/show/728595/13:35
TenguI'm not happy with the security_t and setfscreate.13:36
*** sri_ has quit IRC13:37
*** sri_ has joined #tripleo13:37
jaosoriorTengu: neither am I13:38
*** boazel has joined #tripleo13:38
Tengu;)13:41
Tengujaosorior: in mtg - will check that after13:41
openstackgerritFlorian Fuchs proposed openstack/tripleo-validations stable/queens: Fix node health validation  https://review.openstack.org/59500613:42
dtantsurshardy_mtg: rebased and now my script stopped working >_< I guess I'll put this work on hold until somebody comes with a working procedure of developing on the undercloud.13:43
openstackgerritMerged openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes add ceph loop device  https://review.openstack.org/58319513:43
openstackgerritMerged openstack/tripleo-heat-templates master: Group fast_forward_upgrade_tasks tasks into blocks  https://review.openstack.org/58723513:43
*** dtrainor has quit IRC13:44
Tengujaosorior: will create a temporary policy with the cert_t thingy + ovs socket, and see what's crashing.13:46
jaosoriorTengu: sounds good to me13:47
*** jcoufal has joined #tripleo13:47
Tengujaosorior: hm, do you have a good cheat-sheet for building custom selinux modules?13:50
jaosoriorI don't :13:50
jaosorior:/13:50
Tenguhmm. ok. I just known the audit2allow -M + semodule -i13:51
dtantsurshardy_mtg, mwhahaha, posted https://bugs.launchpad.net/tripleo/+bug/1788414 so that we track the problems with development we have now. Feel free to contribute ideas, suggestions, etc.13:51
openstackLaunchpad bug 1788414 in tripleo "No easy way for local debugging on the containerized undercloud" [Critical,Confirmed]13:51
*** akrivoka has quit IRC13:51
*** akrivoka has joined #tripleo13:52
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Fix standalone home dir variable  https://review.openstack.org/59501013:52
jaosoriorbandini: thanks for the review13:53
mwhahahadtantsur: so i guess it assumes openstack projects have their own way of developing their features.  So being able to test things in a non-deployment fashion would help reduce the criticality of that. I do agree there needs to be a better way to test tripleo-common and such in containers but that also points to our inability to validate changes outside of a live environment13:54
openstackgerritMerged openstack/tripleo-heat-templates master: undercloud: revert to using the iscsi deploy interface by default  https://review.openstack.org/58537013:57
openstackgerritMerged openstack/tripleo-quickstart-extras master: add delorean-current to repolist for updates  https://review.openstack.org/59257713:59
*** gouthamr has quit IRC14:00
*** dmellado has quit IRC14:01
*** stevebaker has quit IRC14:01
*** ksambor has quit IRC14:01
*** shardy_mtg is now known as shardy14:01
shardydtantsur: there should be no need to rebuild the undercloud, only restart the container, as you mention re-running paunch via the undercloud install should be enough14:01
shardydtantsur: thanks for the bug, can you please share your quickstart config, and I'll reproduce and add some ideas on how we can streamline things14:02
shardyor even just restarting the container with the new image I guess14:02
dtantsurshardy: well, any quickstart config, I don't see why it would matter.14:03
dtantsureven without quickstart14:03
shardydtantsur: well you said there was a nova error - my local quickstart env built yesterday is working fine14:04
dtantsurshardy: I realized that it may be because I updated tripleo-common from an older checkout14:04
shardydtantsur: Ok cool, let me take a pass at the container restart steps then14:04
dtantsurafter rebasing my ugly script stopped working :(14:04
shardyOk lets see if we can come up with a slightly less ugly and working one :)14:05
dtantsurshardy: thanks! we need something (semi)automatic, otherwise the life of occasional contributors like me becomes too miserable14:05
shardydtantsur: yeah I agree14:05
openstackgerritMerged openstack/tripleo-heat-templates master: Fix bind-mount to manila's bootstrap container  https://review.openstack.org/59480114:06
openstackgerritMerged openstack/tripleo-heat-templates master: Clarify ironic classic driver removal reno  https://review.openstack.org/59304114:06
*** rh-jelabarre has joined #tripleo14:06
openstackgerritTom Barron proposed openstack/tripleo-heat-templates stable/queens: Fix bind-mount to manila's bootstrap container  https://review.openstack.org/59501414:07
*** dtrainor has joined #tripleo14:08
*** hjensas has joined #tripleo14:09
*** haleyb has joined #tripleo14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791014:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)14:10
*** ooolpbot has quit IRC14:10
*** moshele has quit IRC14:10
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Bind mount tripleo.cnf for tls everywhere work for gnocchi  https://review.openstack.org/59501814:12
*** mwhahaha changes topic to "Welcome to Rocky | CI Status: ORANGE - RDO Cloud unavailable| https://docs.openstack.org/tripleo-docs/latest/"14:13
weshaymarios|rover, ssbarnea|ruck join #rdo please14:15
weshaymarios|rover, ssbarnea|ruck rdo cloud just went down14:15
sri_dsneddon, beagles, nerver mind i pass the network-environment.yaml as last that's fixed the issue, but little bit tricky to get the order right14:16
openstackgerritMerged openstack/tripleo-heat-templates master: Assert container's ssl cert exists before upgrade.  https://review.openstack.org/58908214:18
Tengujaosorior: puppet wants to apply selinux labels.14:20
Tengujaosorior: but the two policies I've created drastically reduce the audit.log content :).14:21
Tenguso I'm really happy now. Will update the trello.14:22
*** ssbarnea|ruck has quit IRC14:23
*** toure is now known as toure|biab14:24
jaosoriorTengu: can we track which labels it tries to apply? maybe we can move that out of puppet.14:25
Tengujaosorior: http://paste.openstack.org/show/728602/14:28
Tengujaosorior: fact is, puppet MODULES might want to set the labels on the fly for either new or existing files.14:29
Tengujaosorior: file { "...": ensure => (file|directory), selabel => ...} for instance.14:29
Tengu(not sure about the option name, but you get it)14:29
weshayrascasoft, rasca?14:31
jaosoriorTengu: isn't that handled by a puppet provider? could we set that provider to be noop?14:40
*** raildo has quit IRC14:40
*** raildo has joined #tripleo14:42
bogdandoPTAL https://review.openstack.org/#/c/587431, jaosorior ?14:44
bogdandoit was 2 +2, just rebased14:45
*** jrist has joined #tripleo14:45
*** gouthamr has joined #tripleo14:50
*** bugzy has joined #tripleo14:51
*** raildo_ has joined #tripleo14:51
*** morazi has quit IRC14:51
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Include overcloud-deploy to the default tags  https://review.openstack.org/59413314:52
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Collect overcloud statistics with ARA  https://review.openstack.org/57846214:53
*** bugzy_ has quit IRC14:53
Tengujaosorior: hm. nope.14:53
d0ugalWorkflow squad meeting in 5 mins. https://etherpad.openstack.org/p/tripleo-workflows-squad-status /cc rbrady,apetrich,thrash,toure,jtomasek14:54
Tengujaosorior: the "file" type has the selinux support directly embedded, and check the puppet facts for that.14:54
Tenguoh.14:54
Tenguwait.14:54
TenguMAYBE.... we can override that fact locally in the container?14:54
*** raildo has quit IRC14:54
Tengujaosorior: selinux => true  if we set it to "false", the file resources should not try to enforce the labels/context.14:55
shardydtantsur: FYI I did a first pass on some notes here https://etherpad.openstack.org/p/tripleo-rpm-container-image-notes14:55
*** mmedvede is now known as mmedvede_14:55
Tengubut it may create some other issues.14:55
Tengulet's try.14:55
shardyprobably we need a script and some optimization of those steps, but hopefully it helps understand the overview14:55
*** mmedvede_ is now known as mmedvede14:55
dtantsurshardy: will read after the meetings, thansk14:55
*** tzumainn has quit IRC14:56
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Make sure we properly set container related facts during upgrade.  https://review.openstack.org/59514314:57
chemjfrancoa: ^14:58
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: WIP: Update default for THT resource registry  https://review.openstack.org/59494414:58
*** moguimar has quit IRC14:58
*** apetrich has quit IRC14:58
dtantsurshardy: oh yeah, we do need a script for that. I can barely understand it, let alone remember :)14:58
jfrancoachem: thank you, I'll check it14:58
chemjfrancoa: untested, unrun, but sure it works!14:59
shardydtantsur: heh yeah a script would be good, but hopefully that helps in the meantime - we can probably cut/paste a script to get you building tripleo-common and restarting all the mistral containers14:59
dtantsurshardy: btw, how long does it long take?15:00
shardydtantsur: not sure as I was cut/pasting notes between steps - the longest is the dlrn-setup (which should be 1-time) and the dlrn build15:00
shardythe docker part is very fast15:00
dtantsurk, I wonder if we can do it within 2 minutes15:01
shardylet me try to create a script and run it end to end15:01
*** ssbarnea|ruck has joined #tripleo15:02
mwhahahashardy: i've been reworking the oooq build test package ansible role to be a standalone thing which may help wrap that build thing15:02
mwhahahahttps://github.com/mwhahaha/tripleo-f28-testbed/tree/master/roles/bootstrap-packages15:03
*** dxiri has joined #tripleo15:03
mwhahahaaccepts a yaml with gerrit changes and gives you a repo to use15:03
shardymwhahaha: ack nice, yeah that may be easier, I just wanted to capture the steps initially15:03
*** dxiri has quit IRC15:03
*** dxiri has joined #tripleo15:04
*** tonyb has quit IRC15:04
*** aufi has quit IRC15:05
*** morazi has joined #tripleo15:05
*** aufi has joined #tripleo15:05
Tengujaosorior: faking the fact using the FACTER_selinux=false doesn't seem to do it - at least not completly.... getting closer though.15:06
shardydtantsur: the dlrn build takes just over 2mins for me, but that can possibly be improved as atm it's e.g cloning rdoinfo every time15:07
*** rnoriega_ is now known as rnoriega15:07
dtantsurshardy: ideally yes. 2+ minutes is a bit too much..15:07
mwhahaha...15:08
shardyyeah, better than redeploying the undercloud though ;)15:08
dtantsurabsolutely15:08
dtantsurbut when working on my patch I had 10+ iteration a day. it would be 20 minutes just waiting for DLRN :)15:08
shardyanother option might be to just setup.py install in the Dockerfile I guess15:09
mwhahahacompared to everything else we do on a regular basis 20 mins spread over the course of the day is not that much15:09
mwhahahai think i spent 20 mins waiting for images to download15:09
dtantsurwell, it adds to everything, not replaced it15:09
dtantsur20 mins here, 20 mins there, y'know..15:09
shardysec let me try the hack-install-from source via docker build15:10
*** pcaruana has quit IRC15:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791015:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)15:10
*** rh-jelabarre has quit IRC15:11
dtantsurisn't ^^^ already fixed in nova?15:11
mwhahahait's probably not availabel yet in promotion15:12
dtantsurI thought it's blocking promotion. Maybe I'm confusing something.15:12
mwhahahait has to make it into packaging, etc15:13
mwhahahait should be fixed but with rdo cloud dead, who knows15:13
dtantsurCI of Schroedinger15:19
shardydtantsur: I added an alternative that does setup.py install intead15:20
shardyshould be pretty fast, seems to build but I've not tested the resulting image15:20
dtantsurI guess it's better if it works15:21
shardyseems to start so worth trying I think15:23
EmilienMweshay: anyone looking at stable/rocky failures in tripleoclient,15:27
EmilienM?15:27
EmilienMhttp://logs.openstack.org/61/594261/1/check/tripleo-ci-centos-7-undercloud-containers/8e99210/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-08-22_00_29_5015:27
EmilienMamoralej: ^15:27
dsneddonsri_, you might have run into this bug? https://review.openstack.org/#/c/59265215:28
weshayEmilienM, the lord is watching everything at all times15:29
EmilienMyup15:29
* weshay gets coffee15:29
EmilienMwe are waiting for this branch to work in CI15:29
EmilienMso we can release15:29
weshayEmilienM, rdo is down man15:30
EmilienMI know15:30
weshaycan we release rocky w/o a promotion?15:30
EmilienMno15:30
weshayEmilienM, I think NOT15:30
*** openstackgerrit has quit IRC15:31
amoralejEmilienM, that job is trying to pull from /docker.io/tripleorocky/centos-binary-barbican-api:current-tripleo15:32
amoralejhave we uploaded containers there?15:32
amoraleji can't see them there15:32
EmilienMsince we didn't have a first promotion, I suspect no15:32
amoralejyes, that's also my guess15:33
EmilienMbut we could workaround by pushing the containers from latest promotion in master no?15:33
*** openstackgerrit has joined #tripleo15:33
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Avoid ignoring failure to collect missing folders  https://review.openstack.org/59407715:33
amoralejEmilienM, latest before branching15:33
weshayEmilienM, /me gets patch15:33
weshayEmilienM, rocky jobs should be using master containers15:34
amoralejthere is a https://trunk.rdoproject.org/centos7-rocky/current-tripleo/15:34
Tengujaosorior: ok. so now I have the var_lib_t thingy: Error: Could not set 'present' on ensure: Permission denied - /var/lib/ironic/httpboot/inspector.ipxe20180822-11-11oty8d.lock at /etc/puppet/modules/ironic/manifests/inspector.pp:33615:34
Tengujaosorior: guess we can't avoid adding the var_lib_t write access... at least for now.15:34
weshayEmilienM, amoralej  https://review.openstack.org/#/c/594357/15:34
amoralejack15:35
EmilienMweshay: ok let me 'recheck'15:35
weshayEmilienM, ya.. your patch ran on old code15:37
weshayEmilienM, come on bro15:37
weshayhive mind15:37
EmilienMI put high expectations in you weshay15:37
*** aufi has quit IRC15:44
*** dmellado has joined #tripleo15:45
*** sri_ has quit IRC15:46
openstackgerritmelissaml proposed openstack/tripleo-heat-templates master: Remove the duplicated word  https://review.openstack.org/59523415:49
*** sri_ has joined #tripleo15:49
*** pcaruana has joined #tripleo15:49
jaosoriorTengu: the option there is to remove the security labeling option from that specific container15:51
Tengujaosorior: for the security_t thingy?15:51
*** tzumainn has joined #tripleo15:51
jaosoriorTengu: --security-opt label:disable15:51
jaosoriorTengu: no, for the ironic container15:52
Tenguah15:52
Tengujaosorior: so that's for the var_lib_t then? or is it unrelated?15:52
jaosoriorTengu: correct, for the var_lib_t thing15:52
jaosoriorTengu: I had a blog post on that at some point...15:52
jaosoriorlet me fetch it15:52
*** gkadam is now known as gkadam-afk15:52
jaosoriorTengu: http://jaormx.github.io/2018/selinux-and-docker-notes/15:53
Tengujaosorior: hm, what's the best to do in fact? disabling this flag, or...?15:53
jaosoriorTengu: write, if a container REALLY needs to do something extra (like the ironic container) then we disable selinux labeling for that specific container15:53
jaosoriorthis way, the rest of the containers stay secure, while this special one... well... remains doing it's special thing :)15:53
*** leanderthal has quit IRC15:54
Tenguok. makes sense.15:54
jaosoriorTengu: at some point we can come back to it and figure out if we can change the behavior of the container15:54
Tengujaosorior: still, while checking the logs of the containers, I get error with rsync and the PKI sync. will dig into this a bit more.15:54
*** paramite_ has joined #tripleo15:58
Tengudamn. that's the archive done at the end of the puppet initial run....15:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044516:00
*** rpioso|afk is now known as rpioso16:02
Tengufact is, we don't need to archive the PKI - it's a mounted volume.16:04
*** yprokule has quit IRC16:04
*** rdopiera has quit IRC16:04
*** gouthamr has quit IRC16:05
Tengujaosorior: guess your proposal won't be OK, since all containers actually write in /var/lib: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py#L264-L26616:06
Tenguand also L25716:06
jaosoriorTengu: yeah; we talked about that with dprince, and shardy; ideally we should write to docker volumes instead of /var/lib/config-data/...16:06
Tengu;)16:07
jaosoriorTengu: but I guess dealing with that is out of scope for now16:07
*** morazi has quit IRC16:07
Tenguso we might want to enable the var_lib_t write directly.16:07
jaosoriorso, for now, the only thing we can do is add the var_lib_t write permission16:07
jaosorioryeah16:07
Tengu"cool".16:07
jaosorior:(16:07
TenguI have three policies in order to get a clean way to disable them later.16:07
Tenguoh wow. already that late?16:08
*** jpena is now known as jpena|off16:09
weshaymwhahaha, EmilienM do I need to make sure ntpdate is just available on the undercloud or do I need to ensure it's installed.. I think installed16:09
mwhahahaweshay: so both16:10
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791016:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178842616:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)16:10
openstackLaunchpad bug 1788426 in tripleo "Tracking bug: RDO Cloud is down" [Critical,In progress] - Assigned to Marios Andreou (marios-b)16:10
mwhahahaweshay: i was going to propose a patch to get it into the fedora stable repo for rdo but then rdo cloud ate it16:10
openstackgerritDavid Peacock proposed openstack/tripleo-common master: add Python 3 to requirements  https://review.openstack.org/59524216:10
chandankumarmwhahaha: is there a way to create reproducable env using standalone mode in offline mode?16:11
mwhahahaweshay: so if you configure ValidateNtp to false, then you don't need it installed and just need it available for install16:11
*** paramite_ has quit IRC16:11
mwhahahachandankumar: if you pull down all the containers a head of time to a registry, maybe?16:11
chandankumarmwhahaha: i need to try that16:12
weshaychandankumar, what do you mean offline16:13
mwhahahachandankumar: it uses a similar method to the overcloud so as along as you can use the container prepare to point to a local registry, it should work16:13
mwhahahachandankumar: it would also assume a local copy of the tripleo rpm repo(s) as well16:13
chandankumarweshay: in devstack, we have a offline mode, like we installed devstack do ubstack.sh then shut down the vm then restsrat the vm and re-run stack.sh everything is back to normal16:14
chandankumarweshay: I am looking for something that like scenario16:14
weshaywouldn't that be super16:15
Tengujaosorior: in fact, we should use named volumes, no bind-mount, when we want persistent data to be written... Right?16:15
chandankumarit would be useful for developer developing stuff on a laptop16:15
*** raildo has joined #tripleo16:16
*** raildo_ has quit IRC16:19
*** morazi has joined #tripleo16:19
Tengujaosorior: ok. and apparently, we don't have a selinux relabelling issued by puppet... the file /var/lib/ironic/httpboot/inspector.ipxe20180822-11-17p9bsk.lock comes from puppet, as it's wanting to create /var/lib/ironic/httpboot/inspector.ipxe from a template. The audit.log shows we can't change the file group to "ironic-inspector" apparently.16:21
Tenguand apparently, that part is managed by a "write" with security_t OR setfscreate. that's the last issues I get for now.16:22
*** rh-jelabarre has joined #tripleo16:28
*** moshele has joined #tripleo16:28
jaosoriorTengu: lets talk about this tomorrow; gotta head off16:34
jaosoriorTengu: and the assertion about named volumes is correct :) that's the ideal thing.16:34
Tengujaosorior: ok :)16:37
Tengujaosorior: see you tomorrow then!16:37
Tenguhave a good one16:37
Tenguwill also sign off.16:37
Tengupretty long day.16:37
*** toure|biab is now known as toure16:38
*** jpich has quit IRC16:40
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Bind mount tripleo.cnf for tls everywhere work for telemetry  https://review.openstack.org/59501816:44
*** gouthamr has joined #tripleo16:46
*** gkadam-afk is now known as gkadam16:49
*** ykarel has joined #tripleo16:51
*** trown is now known as trown|lunch16:52
*** agurenko has quit IRC16:53
*** gkadam has quit IRC16:54
*** trozet has quit IRC17:05
*** gbarros has joined #tripleo17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791017:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178842617:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)17:10
openstackLaunchpad bug 1788426 in tripleo "Tracking bug: RDO Cloud is down" [Critical,In progress] - Assigned to Marios Andreou (marios-b)17:10
*** rh-jelabarre has quit IRC17:15
*** ykarel has quit IRC17:20
sri_dsneddon, ack17:21
*** ykarel has joined #tripleo17:21
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Use become when doing recursive chown  https://review.openstack.org/59526517:21
weshaymwhahaha, EmilienM do you want a lp tracking the progress of f28? I don't care.. just let me know how you guys want to iterate17:21
weshayhttp://paste.openstack.org/show/728619/17:21
*** vinaykns has joined #tripleo17:21
weshayguess the step is PODMAN17:21
weshay< . . . .17:22
*** vinaykns has left #tripleo17:22
EmilienMbandini: why are we overriding FACTER_hostname in docker-puppet.py17:22
EmilienM?17:22
EmilienMfacter should be able to get $::hostname by itself, no?17:22
*** morazi has quit IRC17:23
*** ykarel_ has joined #tripleo17:24
mwhahahaweshay: that's already covered in my etherpad17:24
mwhahahaweshay: w/ a bug and patches17:24
*** ykarel_ has quit IRC17:24
* weshay looks again17:24
*** ykarel_ has joined #tripleo17:24
*** morazi has joined #tripleo17:25
mwhahahaweshay: https://review.openstack.org/593963 and https://review.openstack.org/#/c/59449917:25
*** rnoriega has quit IRC17:25
*** bcafarel has quit IRC17:25
*** mmedvede has quit IRC17:25
*** rf0lc0 has quit IRC17:25
*** redrobot has quit IRC17:25
*** mburned has quit IRC17:25
*** jpena|off has quit IRC17:25
*** panda|off has quit IRC17:25
*** marios has quit IRC17:25
*** EmilienM has quit IRC17:25
*** jschlueter has quit IRC17:25
*** rakhmerov has quit IRC17:25
*** weshay has quit IRC17:25
*** ericyoung has quit IRC17:25
*** Damjanek has quit IRC17:25
*** corvus has quit IRC17:25
bandiniEmilienM: sec lemme find it17:25
*** ykarel has quit IRC17:26
*** ykarel_ has quit IRC17:27
*** gouthamr has quit IRC17:29
*** holser_ has quit IRC17:30
*** EmilienM has joined #tripleo17:30
openstackgerritLuigi Toscano proposed openstack/tripleo-heat-templates stable/queens: Add FFU upgrade steps to Sahara services  https://review.openstack.org/59527017:32
*** ChanServ sets mode: +v EmilienM17:33
*** jpena has joined #tripleo17:36
*** bogdando has quit IRC17:41
*** rodrigods has joined #tripleo17:45
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM - docker-puppet: don't force FACTER_hostname  https://review.openstack.org/59527417:45
*** rfolco has joined #tripleo17:50
*** weshay has joined #tripleo17:51
shardymwhahaha: Hi, do you happen to know if there are any other examples of standalone where you only have one nic?17:55
shardyI'm debugging some issues where the only physical nic gets added to br-ctlplane but then the networking into and out of the (baremetal) box is broken from that point17:55
shardywanted to check if there are known issues before I dig into it17:56
mwhahahashardy: yes it's in the docs17:56
*** rfolco has quit IRC17:56
mwhahahashardy: oh wait no  i just have the one17:56
mwhahahashardy: you have to configure the single nic with the equivalent of what was on the box previously17:56
mwhahahashardy: so you have to add the default route into the standalone params you pass in and stuff17:57
shardymwhahaha: ack I'm working on a config based from the docs, must be getting something wrong17:57
shardyI couldn't see where $IP was being set for the public interface, only NeutronPublicInterface17:57
mwhahahashardy: we basically move the network config from the existing nic, add a br-ctrplane on that nic and setup the ip/routes via that.17:58
weshayshardy, I think the upstream standalone job uses br-ex17:58
mwhahahashardy: it's in tripleoclient, let me dig it up17:58
shardyOk so we take down e.g em1, add it to the bridge then bring the same ip back up17:58
mwhahahashardy: yea17:58
shardyweshay: ack thanks, I suspect I'm using the wrong bridge, just about to reproduce17:59
*** mmedvede has joined #tripleo18:00
mwhahahashardy: right so we create the NeutronPhysicalBridge on the NeutronPublicInterface18:01
weshayman the upstream images are 8.3G18:01
mwhahahashardy: the deployed-server portmap we create is done here https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/tripleo_deploy.py#L39018:02
mwhahahaand of course our network config is https://github.com/openstack/tripleo-heat-templates/blob/master/net-config-standalone.j2.yaml#L8418:03
mwhahahai think we basically are hardcoding br-ctlplane18:03
*** rfolco has joined #tripleo18:05
shardymwhahaha: ah thanks for the links - I've rebuilt the box and trying again with the exact config from the docs18:07
mwhahahashardy: yea i think out of the box you have to use br-ctlplane, probably should document that better18:08
mwhahahayou could create a new br to use instead of like em118:08
mwhahahaas the NeutronPublicInterface18:08
mwhahahabut the NeutronPublicBridge has to be br-ctlplane18:08
shardyOk thanks for the help all, seems better this time \o/18:14
*** shardy is now known as shardy_afk18:15
*** sri_ has quit IRC18:15
*** ChanServ has quit IRC18:16
*** khyr0n has quit IRC18:18
*** stevebaker has joined #tripleo18:18
*** trown|lunch is now known as trown18:21
*** ChanServ has joined #tripleo18:22
*** barjavel.freenode.net sets mode: +o ChanServ18:22
*** dmellado has quit IRC18:22
*** stevebaker has quit IRC18:23
*** dxiri has quit IRC18:24
*** bcafarel has joined #tripleo18:27
*** tbonds has quit IRC18:30
EmilienMweshay: stable/rocky jobs look green so far18:31
EmilienMsee 594261 in zuul18:32
EmilienMso far it's green except update/upgrades18:32
EmilienMbut we can figure these ones later18:32
EmilienMso if that pass, the only blocker to release is to get a promotion18:32
EmilienM(for that we need RDO cloud to be back I guess)18:32
weshaysec18:33
*** electrichead has quit IRC18:34
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: Test dryrun  https://review.openstack.org/59530818:40
*** assassin has quit IRC18:42
*** dtantsur is now known as dtantsur|afk18:43
*** moshele has quit IRC18:46
*** tosky has quit IRC18:47
*** mburned has joined #tripleo18:47
weshayEmilienM, /me looks18:49
weshayEmilienM, ah.. this is looking good18:50
*** amoralej is now known as amoralej|off18:51
*** redrobot has joined #tripleo18:52
EmilienMweshay: yes.18:54
EmilienMnow we need this @#! promotion18:54
*** jfrancoa has quit IRC19:08
*** mjturek has quit IRC19:11
weshayade_lee, I added an alert bug here19:34
weshaybut I don't see the bot running19:34
weshayade_lee, check out rh-openstack-dev and tripleo bugs w/ alert tags19:35
ade_leeweshay, ack thanks19:35
*** assassin has joined #tripleo19:43
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Move toci BASH variables to ansible  https://review.openstack.org/58944819:43
*** pcaruana has quit IRC19:47
*** panda has joined #tripleo19:55
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes from toci_gate_test.sh  https://review.openstack.org/58701219:56
*** gbarros has quit IRC20:01
mwhahahaweshay, EmilienM: i deployed an centos7 overcloud on f28 \o/20:08
weshaymwhahaha, nice20:08
weshaymwhahaha, so we just have to wait on a few of these patches to land20:09
mwhahahayea20:09
mwhahahamost are approved20:09
mwhahahaso soonish20:09
weshayk. cool.. I'm trying locally w/ the upstream fedora28 image.. fk. uuug 8.2gb20:09
mwhahahathe cloud image?20:10
weshayya20:11
weshayhttps://nb01.openstack.org/images/20:11
weshay8.620:11
mwhahahaweshay: wow they are all really big, wonder why20:13
*** Damjanek has joined #tripleo20:14
*** dprince has quit IRC20:16
mwhahahaso i guess the next step would be to try and build a fedora overcloud image20:20
*** morazi has quit IRC20:21
weshaymwhahaha, ya.. so let me see how far I can get w/ this image.. the stock image will NOT work20:21
mwhahahaweshay: what are you using that image for? the overcloud?20:21
weshayand the partitions are a little different and some other small changes between centos and fed20:21
mwhahahaah20:22
weshaymwhahaha, atm.. I'm using it for the undercloud20:22
mwhahahathe standard fedora28 cloud image worked fine for the undercloud20:22
mwhahahamy tests were with my python3 packages tho20:22
mwhahahawhich have yet to land in rdo20:22
weshaymwhahaha, ya.. I have to customize and expand the partitions etc20:23
weshayin a way that is slightly different than centos20:24
weshayso I'm just checking to see if the upstream image is any different20:24
mwhahahaweshay: that's weird i didn't run into that20:26
mwhahahacloud-init issues maybe?20:26
mwhahahaweshay: oh wait you're using libvirt right? does cloud-init run?20:26
*** tonyb has joined #tripleo20:26
weshayno sir20:26
*** itlinux has joined #tripleo20:27
mwhahahano to which part20:28
mwhahahayou running against ovb then?20:28
weshaymwhahaha, actually I take that back.. but that is not what is used for libvirt20:28
weshaymwhahaha, you were running on the cloud though20:28
itlinuxhello guys, I found this command in the docs.openstack.org openstack overcloud update --nodes controller-0 will this command work on a bm in pike? no containers used. Talking to some of the great guys in this channel told me that they did not test it against not container pike. Thanks20:29
mwhahahaweshay: right so you'd have to inject fake cloud-init user config20:29
weshayya20:29
mwhahahaweshay: so if you use libvirt you can just tell cloud-init to expand the disk20:29
weshaymwhahaha, also trying to rip out a lot of this stuff20:29
weshayso I'm not very interested in changing much of it20:29
weshaymwhahaha, ya20:29
mwhahahawell i'd assume that's the same problem with using the centos cloud images on libvirt? :D20:30
* mwhahaha shrugs20:30
mwhahahait's a minor image tweak in comparison to the rest of our hacks20:30
weshaymwhahaha, we do. .but virt-customize is used20:30
weshayI need to tear most of this out..20:30
mwhahahayea20:30
weshaystand up a couple nodes.. run toci20:30
weshaythat's all20:30
mwhahahawouldn't be a problem if our provisioning system wasn't so tightly coupled with the rest20:31
* mwhahaha hides20:31
weshayrun infra playbooks, run tripleo playbooks20:31
*** stevebaker has joined #tripleo20:31
weshaymwhahaha, hey.. at least we have one20:31
*** khyr0n has joined #tripleo20:31
weshaymwhahaha, if upstream nodepool had an answer to libvirt multinode.. I'd take it20:31
weshayso :P20:32
mwhahahaweshay: https://i.imgur.com/t96UURT.jpg20:32
*** ericyoung has joined #tripleo20:33
weshayheh.. true20:33
*** gbarros has joined #tripleo20:33
mwhahahastevebaker: btw i figured out my problem from yesterday. old containers that don't properly support push_destination: true20:33
mwhahahastevebaker: so we just have to wait on a promotion20:33
mwhahahaspeaking of promotions, weshay what are we blocked on for master promotions (other than rdo cloud being dead)20:33
* weshay shows20:34
weshaysec20:34
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: [PoC] Enable featureset override  https://review.openstack.org/59451120:34
*** tbonds has joined #tripleo20:34
weshaymwhahaha, 59338320:34
mwhahahak20:35
weshaythe upstream image appeears to be working20:35
mwhahahaweshay: looks like the ntp getting added to the fedora stable repo soonish (patch is merging)20:45
mwhahahahttps://review.rdoproject.org/r/1582320:45
*** raildo has quit IRC20:45
openstackgerritAakarsh proposed openstack/tripleo-quickstart master: DNM: Testing browbeat's sysmetric collection  https://review.openstack.org/59535120:45
*** jcoufal has quit IRC20:49
openstackgerritMerged openstack/tripleo-common master: nova_metadata healthcheck script  https://review.openstack.org/59432020:51
openstackgerritMerged openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/59404920:51
openstackgerritAlan Bishop proposed openstack/tripleo-common stable/queens: Add missing service for DockerCinderVolumeImage  https://review.openstack.org/59535720:53
*** trown is now known as trown|outtypewww20:54
*** florianf has quit IRC20:57
*** jpena is now known as jpena|off20:57
*** jtomasek has quit IRC20:57
*** toure is now known as toure|gone20:59
*** abishop has quit IRC21:04
*** morazi has joined #tripleo21:08
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791021:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178842621:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)21:10
openstackLaunchpad bug 1788426 in tripleo "Tracking bug: RDO Cloud is down" [Critical,In progress] - Assigned to Marios Andreou (marios-b)21:10
*** gouthamr has joined #tripleo21:11
openstackgerritMerged openstack/python-tripleoclient master: Allow custom UC/standalone passwords  https://review.openstack.org/58743121:12
openstackgerritMerged openstack/python-tripleoclient master: Use binary io for object saving  https://review.openstack.org/59456321:12
openstackgerritMerged openstack/tripleo-common master: Open tarball in binary mode  https://review.openstack.org/59456021:15
*** itlinux has quit IRC21:15
*** holser_ has joined #tripleo21:18
*** dprince has joined #tripleo21:21
*** ubijtsa has joined #tripleo21:29
*** sri_ has joined #tripleo21:30
*** assassin has quit IRC21:31
*** boazel has quit IRC21:33
*** dansmith is now known as htimsnad21:34
openstackgerritMerged openstack-infra/tripleo-ci master: Fix HASH for periodic jobs(use tripleo-ci-testing)  https://review.openstack.org/59338321:43
openstackgerritMerged openstack/tripleo-quickstart-extras master: Updated skip list for Tempest  https://review.openstack.org/59198221:43
*** agopi has quit IRC21:52
EmilienMbandini: FWIW, https://review.openstack.org/#/c/595274/ seems to pass21:56
mwhahahaa likely story22:01
* mwhahaha tries a deploy of a fedora28 overcloud image22:05
* mwhahaha thinks this will not go well22:05
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: standalone support for quickstart on libvirt  https://review.openstack.org/59154022:08
weshaymwhahaha, waste of time22:08
weshaymwhahaha, did you build the images?22:08
mwhahahaSHHHHHH DON'T RUIN MY FUN22:08
mwhahahaweshay: yes22:08
mwhahahaonly had to extract the kernel by hand and do some hacking of tripleo-common22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791022:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178842622:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)22:10
openstackLaunchpad bug 1788426 in tripleo "Tracking bug: RDO Cloud is down" [Critical,In progress] - Assigned to Marios Andreou (marios-b)22:10
EmilienMfun times22:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: standalone support for quickstart on libvirt  https://review.openstack.org/59154022:11
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Podman support for docker-puppet  https://review.openstack.org/58865522:12
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM - test podman  https://review.openstack.org/59058722:12
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Avoid SELinux issues with the "cp -a" command  https://review.openstack.org/59500322:13
EmilienMideally we would wait to get a rocky promotion before we release rc122:14
EmilienMbut what if it doesn't happen next week?22:14
EmilienMI think it's tolerable to release rc1 in our current state22:14
EmilienMunless we have a critical bug that really prevents the deployment of Rocky version22:15
mwhahahalet's see what happens this week22:16
mwhahahabtw my fedora image ironic'd22:16
mwhahahaso the provisioning worked22:16
* mwhahaha is shocked22:16
* mwhahaha waits for ansible to fail horribly22:17
EmilienMyou're so positive22:17
mwhahahai think if this works i'll just take tomorrow off and buy a lottery ticket22:17
*** tzumainn has quit IRC22:19
mwhahaha{"msg": "No package openstack-heat-agents available.", "failed": true ...22:20
mwhahahawell that's solvable22:20
mwhahahaok so my image build was just bad, but good to know it provisions fine and ansible ran22:21
EmilienM:-O22:21
EmilienMtest with podman if you need failures22:21
mwhahahaNO22:21
mwhahahaYOU KEEP YOUR DIRTY PODMAN TO YOURSELF22:22
EmilienMourselves*22:22
mwhahahanaw uh22:22
mwhahahait's all yours22:22
mwhahahaoh fiddle sticks my image build failed because i'm using a file:/// repo so disk image builder doesn't like that22:23
*** holser_ has quit IRC22:26
*** rcernin has joined #tripleo22:34
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: WIP,DNM fedora28 standalone  https://review.openstack.org/59537422:34
*** holser_ has joined #tripleo22:34
weshaymwhahaha, wouldn't we be better off.. just focusing on the standalone?22:35
mwhahahaweshay: in the short term yea, but at some point we're going to need a python3 overcloud node22:35
mwhahahaand if it works...22:35
weshaymwhahaha, will we NEEED it22:36
weshayand for how long22:36
mwhahaharight now i'm doing a basic pass to see where our gaps are22:36
weshayanyway.. ya it's good idea22:36
mwhahahagiven what i've found i'm not that concerned on the standalone22:36
weshaymwhahaha, ideally I'd like to get a fed-28 standalone running22:36
mwhahahasince i managed to deploy an overcloud succesfully22:36
weshayok22:37
*** sri_ has quit IRC22:37
mwhahahastandalone is basically an undercloud which we got working but that's not where most of our hidden python bits are22:37
weshayk22:37
mwhahahai'm more concerned about our mistral workflow bits22:38
mwhahahaand hidden tripleo-common stuff22:38
*** sri_ has joined #tripleo22:38
mwhahahaweshay: do we have stable/queens scenario timeouts tracked somewhere?22:40
mwhahahaweshay: https://review.openstack.org/#/c/595014/22:40
mwhahahaor is that related to the one alert22:40
weshaysorry.. your message timed out, please call back later22:40
mwhahahaweshay: https://www.youtube.com/watch?v=dIiZ3vvZ78s22:41
* weshay looks for the bug22:41
weshaykr22:41
weshayfkr22:41
weshayeven22:41
weshaymwhahaha, ah good..22:42
weshayur patch is on queens22:42
weshaysoooooooooooooooooooooo22:42
weshaymwhahaha, https://review.openstack.org/#/c/594933/22:42
weshaymwhahaha, so I don't think we should use wild cards in the include_pkgs22:43
weshaybecause as soon as you stand up a new repo.. boom22:43
weshayit's in c22:43
weshayci22:43
weshayso your yum update patch is not in mix yet22:43
mwhahahaweshay: hmmm22:44
mwhahahabut we shouldn't have that many packages22:44
weshayso you are still hitting https://bugs.launchpad.net/tripleo/+bug/178676422:44
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Fix released] - Assigned to Sorin Sbarnea (ssbarnea)22:44
mwhahahaweshay: why did this suddenly become a problem22:45
weshaymwhahaha, meaning.. why did containers all of a sudden become a problem?22:46
mwhahahayea22:46
weshaythe update issue has been there22:46
weshaymwhahaha, don't make me paste that irc blob from infra about dns servers and such22:47
mwhahaharight so i'll be coming back to my own little world22:47
mwhahahawhen you solve it, let me know22:47
* mwhahaha runs22:47
weshay:)22:47
*** ChanServ has quit IRC22:49
mwhahahaturns out i just built a blank fedora cloud image23:01
mwhahahaso i guess it would have worked if i enabled package installs23:01
* mwhahaha goes and builds an overcloud-full23:01
*** ChanServ has joined #tripleo23:03
*** barjavel.freenode.net sets mode: +o ChanServ23:03
*** dprince has quit IRC23:09
*** holser_ has quit IRC23:10
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178676423:10
openstackLaunchpad bug 1786764 in tripleo "tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates times out on prepare" [Critical,Triaged] - Assigned to Sorin Sbarnea (ssbarnea)23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/178791023:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)23:10
*** slagle has joined #tripleo23:17
*** rpioso is now known as rpioso|afk23:18
*** sri_ has quit IRC23:19
*** sri_ has joined #tripleo23:20
*** rlandy has quit IRC23:39
*** gbarros has quit IRC23:46
*** gouthamr has quit IRC23:48
« Tuesday, 2018-08-21 Index Thursday, 2018-08-23 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!