Friday, 2019-01-11

*** gyee has joined #openstack-ansible00:08
*** lbragstad has quit IRC00:10
*** gyee has quit IRC00:12
*** macza has joined #openstack-ansible00:25
*** macza has quit IRC00:25
*** macza has joined #openstack-ansible00:26
*** macza has quit IRC00:33
*** openstackgerrit has quit IRC00:34
*** hwoarang_ has joined #openstack-ansible00:57
*** hwoarang has quit IRC00:59
cloudnullanyone around want to do some reviews on the smart_sources bits https://review.openstack.org/#/q/topic:smart-sources+(status:open)00:59
*** TxGirlGeek has quit IRC01:00
*** gyee has joined #openstack-ansible01:01
cloudnullI think the pattern is finally in a good place, especially considering all the learnings we had with neutron. I've not tackled all the roles, but this should lead the way for us to easily get it in this cycle allowing us to stop carrying templates unnecessarily .01:02
*** tosky has quit IRC01:05
*** markvoelker has joined #openstack-ansible01:15
*** gyee has quit IRC01:37
*** gyee has joined #openstack-ansible01:38
*** cshen has joined #openstack-ansible01:48
*** openstackgerrit has joined #openstack-ansible01:48
openstackgerritMerged openstack/openstack-ansible-ops master: remove dynamic ns.enable generators  https://review.openstack.org/62993901:48
*** gyee has quit IRC01:49
*** cshen has quit IRC01:53
*** markvoelker has quit IRC02:42
*** markvoelker has joined #openstack-ansible02:52
*** lbragstad has joined #openstack-ansible02:53
*** TxGirlGeek has joined #openstack-ansible02:56
*** hwoarang has joined #openstack-ansible03:09
*** hwoarang_ has quit IRC03:09
*** vnogin has joined #openstack-ansible03:14
*** vnogin has quit IRC03:18
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Use the inventory to enable/disable services by default  https://review.openstack.org/62897903:25
*** markvoelker has quit IRC03:28
*** markvoelker has joined #openstack-ansible03:49
*** cshen has joined #openstack-ansible03:49
cloudnullevenings all03:50
*** cshen has quit IRC03:54
kaiokmocloudnull: o/04:05
*** udesale has joined #openstack-ansible04:14
openstackgerritMerged openstack/openstack-ansible-galera_server master: Tidy yum repository setup  https://review.openstack.org/62993404:18
*** markvoelker has quit IRC04:26
*** lbragstad has quit IRC04:35
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Add libpython2.7 as a required package  https://review.openstack.org/61904004:44
*** radeks has joined #openstack-ansible04:56
cloudnullhows it kaiokmo?04:58
kaiokmothings are fine. and you?05:02
*** markvoelker has joined #openstack-ansible05:05
*** markvoelker has quit IRC05:11
*** markvoelker has joined #openstack-ansible05:11
*** markvoelker has quit IRC05:15
*** markvoelker has joined #openstack-ansible05:17
cloudnullgenerally good :)05:17
*** fatdragon has quit IRC05:25
*** fatdragon has joined #openstack-ansible05:25
*** dave-mccowan has quit IRC05:28
*** fatdragon has quit IRC05:31
*** chkumar|out is now known as chandankumar05:40
*** TxGirlGeek has quit IRC05:41
*** cshen has joined #openstack-ansible05:49
*** cshen has quit IRC05:54
*** radeks has quit IRC05:58
openstackgerritVieri proposed openstack/openstack-ansible-os_aodh master: fix tox python3 overrides  https://review.openstack.org/60682406:09
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove tempest_image_dir_owner var  https://review.openstack.org/62941906:13
*** markvoelker has quit IRC06:20
*** markvoelker has joined #openstack-ansible06:31
*** markvoelker has quit IRC06:36
*** czunker_ccc has joined #openstack-ansible06:38
*** mathlin has quit IRC06:44
*** markvoelker has joined #openstack-ansible06:51
*** radeks has joined #openstack-ansible06:57
*** markvoelker has quit IRC06:58
chandankumarodyssey4me: Hello07:02
*** markvoelker has joined #openstack-ansible07:02
chandankumarodyssey4me: what is the use of tempest_service_setup_host in os_tempest https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L35 ?07:03
chandankumarodyssey4me: As per comment, It says it will execute shade module and host have already clouds.yaml configured07:04
chandankumarodyssey4me: but shade is deprecated in favor of openstacksdk07:04
*** pcaruana has joined #openstack-ansible07:05
*** kopecmartin|off is now known as kopecmartin07:07
*** markvoelker has quit IRC07:07
*** jawad_axd has joined #openstack-ansible07:08
*** markvoelker has joined #openstack-ansible07:08
*** mathlin has joined #openstack-ansible07:10
*** markvoelker has quit IRC07:13
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: [DNM] testing without tempest_service_setup_host  https://review.openstack.org/63004007:14
*** vnogin has joined #openstack-ansible07:14
*** cshen has joined #openstack-ansible07:15
*** vnogin has quit IRC07:18
*** fatdragon has joined #openstack-ansible07:24
*** mathlin has quit IRC07:27
*** fatdragon has quit IRC07:29
jrosserchandankumar: all the osa roles have a ...._setup_host07:36
jrosserThere are times when the deploy host cannot directly do http against the admin endpoint, for example when there is an ssh bastion in between07:42
jrosserThat var allows service setup tasks to be delegated to a host that the deployer chooses, which can hit the admin endpoint07:43
*** klamath has joined #openstack-ansible07:57
*** radeks_ has joined #openstack-ansible08:01
fnpanichi08:02
fnpanicusing rocky and mimic is fine or should i stick with luminous for rocky?08:02
*** radeks has quit IRC08:03
*** mathlin has joined #openstack-ansible08:05
*** luksky has joined #openstack-ansible08:06
*** gkadam has joined #openstack-ansible08:09
*** eumel8 has joined #openstack-ansible08:10
*** markvoelker has joined #openstack-ansible08:16
jrosserfnpanic: rocky+mimic is working for me08:18
*** cshen has quit IRC08:18
*** dcdamien has joined #openstack-ansible08:18
jrosserI use my own tooling to deploy ceph, you get to choose if you have osa do the ceph deploy or if you make it separate08:19
*** gkadam_ has joined #openstack-ansible08:20
*** jbadiapa has joined #openstack-ansible08:22
*** gkadam has quit IRC08:23
*** markvoelker has quit IRC08:23
*** cshen has joined #openstack-ansible08:24
*** mathlin has quit IRC08:24
*** rgogunskiy has joined #openstack-ansible08:28
jrosserodyssey4me: can you take a look at this - ceph finally fixed with brute force permissions change https://review.openstack.org/#/c/629317/08:29
*** rgogunskiy has quit IRC08:31
*** cshen has quit IRC08:31
*** tosky has joined #openstack-ansible08:40
*** gkadam__ has joined #openstack-ansible08:45
*** gkadam_ has quit IRC08:47
*** thuydang has joined #openstack-ansible08:47
*** ygk_12345 has joined #openstack-ansible08:53
ygk_12345hi all my heat playbook of rocky 18.1.2 release is failing08:59
*** mathlin has joined #openstack-ansible09:00
*** dcdamien has quit IRC09:01
*** vollman has quit IRC09:07
*** thuydang has left #openstack-ansible09:13
mbuilcould anybody check https://review.openstack.org/#/c/621249/ please?09:16
ygk_12345hi all my heat playbook of rocky 18.1.2 release is failing09:17
*** mcela has joined #openstack-ansible09:19
*** DanyC has joined #openstack-ansible09:19
*** shardy has joined #openstack-ansible09:20
*** mathlin has quit IRC09:23
openstackgerritFrank Kloeker proposed openstack/openstack-ansible-haproxy_server master: Add feature Letsencrypt SSL certification  https://review.openstack.org/58677409:24
ygk_12345shardy: Hi. heat playbook of rocky 18.1.2 release is failing09:25
ygk_12345shardy: can u help me please09:26
shardyygk_12345: sorry I've never deployed heat (or anything else) with openstack-ansible09:28
shardyhopefully some other folks around here can help - I'd put the error into paste.openstack.org and add the link here09:28
ygk_12345can someone check this error of heat playbook of rocky 18.1.2 please09:30
ygk_12345http://paste.openstack.org/show/741763/09:30
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Fix git tracking of networking OVN  https://review.openstack.org/63009909:46
*** DanyC has quit IRC09:47
*** DanyC has joined #openstack-ansible09:48
*** mathlin has joined #openstack-ansible09:54
evrardjpygk_12345: hey, thanks for the bug09:56
evrardjpwe'll triage this appropriately09:57
fnpanicjrosser: thanks for the info!09:59
*** dcdamien has joined #openstack-ansible10:04
ygk_12345evrardjp: attached those files to the bug10:10
*** shardy has quit IRC10:12
*** shardy has joined #openstack-ansible10:15
*** luksky has quit IRC10:16
*** mathlin has quit IRC10:20
MiougeGood morning folks!10:20
*** shardy_ has joined #openstack-ansible10:26
*** cshen has joined #openstack-ansible10:27
*** shardy has quit IRC10:29
ygk_12345evrardjp: any workaround possible in the meantime to overcome this issue ?10:32
*** cshen has quit IRC10:33
*** shardy_ has quit IRC10:36
*** mathlin has joined #openstack-ansible10:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-hardening stable/rocky: Switch to using import_tasks for static inclusion  https://review.openstack.org/63013910:47
*** electrofelix has joined #openstack-ansible10:48
*** shardy has joined #openstack-ansible10:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/rocky: Tidy yum repository setup  https://review.openstack.org/63014010:50
*** mathlin has quit IRC10:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/rocky: Switch galera_install_dnf back to being a symlink  https://review.openstack.org/63014110:51
openstackgerritJakob Englisch proposed openstack/ansible-hardening stable/rocky: Chrony: make ntp server options configurable  https://review.openstack.org/63014210:54
*** udesale has quit IRC10:54
openstackgerritJakob Englisch proposed openstack/ansible-hardening stable/queens: Chrony: make ntp server options configurable  https://review.openstack.org/63014310:55
*** deployer2 has joined #openstack-ansible10:56
openstackgerritJakob Englisch proposed openstack/ansible-hardening stable/rocky: Chrony: add an option to sync the hardware clock  https://review.openstack.org/63014410:56
openstackgerritJakob Englisch proposed openstack/ansible-hardening stable/queens: Chrony: add an option to sync the hardware clock  https://review.openstack.org/63014510:57
*** shardy has quit IRC10:58
odyssey4meevrardjp jrosser what do you think about https://review.openstack.org/629499 to help cut down the number of overrides needed for tests?11:10
odyssey4mejrosser guilhermesp ygk_12345 interesting, it appears that https://review.openstack.org/629544 shows that, at least for master, adding heat doesn't give any problems11:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: zuul: Add required project openstack/openstack-ansible  https://review.openstack.org/62954211:12
odyssey4mejrosser thanks for figuring out https://review.openstack.org/629317 :)11:13
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove tempest_image_dir_owner var  https://review.openstack.org/62941911:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Fixes empty metal_query  https://review.openstack.org/62993011:17
MiougeI would like to add a “haproxy_whitelist_networks” to a couple of API services (heat_api, magnum, octavia). Do I need to copy the whole “haproxy_default_services” from upstream into my group_var?11:19
*** mathlin has joined #openstack-ansible11:20
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove tempest_image_dir_owner var  https://review.openstack.org/62941911:23
*** vnogin has joined #openstack-ansible11:31
chandankumarodyssey4me: Hello11:33
chandankumarodyssey4me: https://review.openstack.org/#/c/627482/35/.zuul.yaml@189 I can set the value here directly https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L35 to inventory_hostname ?11:34
odyssey4mechandankumar no, for OSA we want to use local host by default11:41
*** vnogin has quit IRC11:41
chandankumarodyssey4me: ok, I think i need to take care of that in playbook itself11:43
odyssey4mechandankumar just change the var as I suggested11:43
chandankumarodyssey4me: yup already updated the patch11:44
odyssey4meno need to do anything in the playbook - just change the var11:44
*** mathlin has quit IRC11:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Cleanup files and templates using smart sources  https://review.openstack.org/58895311:52
*** cshen has joined #openstack-ansible11:55
*** luksky has joined #openstack-ansible11:56
*** cshen has quit IRC12:01
*** mathlin has joined #openstack-ansible12:16
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide support for ovs-sfc  https://review.openstack.org/62124912:17
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide support for ovs-sfc  https://review.openstack.org/62124912:18
*** czunker_ccc has quit IRC12:27
jamesdentonmornin12:27
ygk_12345evrardjp: is the fix released for this bug ?12:29
openstackgerritMerged openstack/openstack-ansible-os_barbican master: Only implement policy.json if an override is configured  https://review.openstack.org/62928012:34
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63019912:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63019912:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican stable/queens: Only implement policy.json if an override is configured  https://review.openstack.org/63020012:36
*** cshen has joined #openstack-ansible12:38
*** hwoarang has quit IRC12:40
*** mathlin has quit IRC12:41
guilhermespodyssey4me: that's interesting. ygk_12345 for now I'd suggest you to checkout the master branch of heat and try to run the playbook again just to see the the issue persists12:42
odyssey4meguilhermesp I don't think he should do that because it's his production environment12:44
guilhermespHe told yesterday was a fresh install?12:44
guilhermespAt least the requests I tried to follow12:45
*** cshen has quit IRC12:45
guilhermespIf it is not a fresh I agree with odyssey4me12:46
odyssey4meguilhermesp ah, fair enough - if it's a test environment that can be dstroyed and recreated, then yeah try master and see if that works - if it does, then maybe we have an ansible bug, or something needs backporting12:46
odyssey4memaster uses ansible 2.7, and rocky uses ansible 2.5 IIRC12:47
ygk_12345guilhermesp: odyssey4me i will try with the master branch of rocky then12:47
odyssey4meI'm suspecting an ansible bug at this point12:47
guilhermespYep, we could have a clue that could make easy for us to build a pr for this12:47
ygk_12345guilhermesp: odyssey4me i will let you know how it goes12:47
odyssey4meygk_12345 master is not rocky - it is stein, and it should not be used for production... but is fine for testing12:47
ygk_12345odyssey4me: so which version is stable as of now for rocky ?12:48
guilhermespygk_12345: yep, be sure to checkout master in a test env12:48
ygk_1234518.1.1 ? or 18.1.2 ?12:48
odyssey4meygk_12345 whatever the latest tag is, but nothing's changed for rocky so the experience will be the same regardless12:48
chandankumarmnaser: cloudnull jrosser https://review.openstack.org/#/c/629419/ is good to go12:49
odyssey4meI suspect an ansible bug.12:49
*** strobelight has joined #openstack-ansible12:51
*** strobelight_ has joined #openstack-ansible12:54
*** strobelight is now known as Guest6910112:54
*** strobelight_ is now known as strobelight12:55
*** Guest69101 has quit IRC12:58
openstackgerritMerged openstack/openstack-ansible master: Call ceph-facts role as required by changes to ceph-ansible  https://review.openstack.org/62931713:03
*** vnogin has joined #openstack-ansible13:06
odyssey4mejrosser could you take a peek at https://review.openstack.org/629542 ?13:07
odyssey4mecores - I need another review for https://review.openstack.org/624773 please13:08
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Update inventory generator to support container gateway override  https://review.openstack.org/62930913:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Make OVN track master branch  https://review.openstack.org/62991413:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Spice console doesn't work on aarch64+kvm.  https://review.openstack.org/62659313:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Add http proxy test scenario  https://review.openstack.org/62552313:11
*** mathlin has joined #openstack-ansible13:12
*** vnogin has quit IRC13:27
*** jawad_axd has quit IRC13:32
*** mathlin has quit IRC13:36
*** cshen has joined #openstack-ansible13:41
*** priteau has joined #openstack-ansible13:47
*** mkuf has quit IRC13:52
*** cshen has quit IRC13:52
*** shardy has joined #openstack-ansible13:57
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide support for ovs-sfc  https://review.openstack.org/62124913:57
*** lbragstad has joined #openstack-ansible13:57
deployer2where in OSA rocky are policy.json files located? Looking into nova-api-container and only file /etc/nova/policy.json contains just { }13:58
odyssey4medeployer2 the default policy is on code - the policy.json file only contains overrides of the defaults13:59
deployer2odyssey4me hmm, if I want to understand how default permissions work - where to look for policy rules?14:01
odyssey4medeployer2 I don't know - either the keystone docs, or the docs of the service itself14:02
*** strattao has joined #openstack-ansible14:04
*** hwoarang has joined #openstack-ansible14:04
*** dave-mccowan has joined #openstack-ansible14:08
*** mathlin has joined #openstack-ansible14:08
*** DanyC has quit IRC14:08
*** DanyC has joined #openstack-ansible14:09
*** cshen has joined #openstack-ansible14:19
*** deployer2 has quit IRC14:20
*** cshen has quit IRC14:24
*** mathlin has quit IRC14:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Cleanup files and templates using smart sources  https://review.openstack.org/58895114:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Cleanup files and templates using smart sources  https://review.openstack.org/58895314:33
*** vnogin has joined #openstack-ansible14:40
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove centos copy-on-write backed tests  https://review.openstack.org/63027614:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/62940914:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat stable/queens: Only implement policy.json if an override is configured  https://review.openstack.org/62941014:42
*** kopecmartin is now known as kopecmartin|off14:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican master: Remove legacy policy.json file  https://review.openstack.org/63027914:48
ygk_12345odyssey4me: evrardjp i tried with 18.1.1 branch and again the heat failed with the same error14:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Remove centos copy-on-write backed tests  https://review.openstack.org/63027614:49
odyssey4meygk_12345 yes, as expected14:49
ygk_12345odyssey4me: so what could be the problem ?14:49
odyssey4meygk_12345 the suggestion was to build using master to validate that it works for you with your test config14:49
ygk_12345odyssey4me: ok14:50
odyssey4meif it does, then it's most likely an ansible bug14:50
ygk_12345odyssey4me: ok i will try master then and let you know14:50
ygk_12345odyssey4me: evrardjp thanks for your support guys. Much appreciated. I will try master and let you know.14:50
evrardjpyw14:51
*** ygk_12345 has quit IRC14:51
*** cshen has joined #openstack-ansible14:51
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Cleanup files and templates using smart sources  https://review.openstack.org/58895314:53
jamesdentonany opposition to extending the MNAIO to lay down something besides LinuxBridge? Or does that functionality already exist?14:54
odyssey4mejamesdenton it doesn't exist, and no objection14:54
jamesdentoncool14:54
*** cshen has quit IRC14:56
*** vnogin has quit IRC15:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Remove legacy policy.json file  https://review.openstack.org/63028215:02
*** mathlin has joined #openstack-ansible15:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63028215:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63028315:10
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Remove tempest_image_dir_owner var  https://review.openstack.org/62941915:15
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Automatically select the correct tempest plugins  https://review.openstack.org/62949915:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/rocky: Remove legacy policy.json file  https://review.openstack.org/63028915:22
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63028315:23
*** cshen has joined #openstack-ansible15:24
cloudnullmornings15:28
*** cshen has quit IRC15:29
jamesdentonhello15:29
*** mathlin has quit IRC15:30
cloudnullo/ hows it ?15:31
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide support for ovs-sfc  https://review.openstack.org/62124915:32
jamesdentonit's goin15:33
openstackgerritMerged openstack/openstack-ansible master: zuul: Add required project openstack/openstack-ansible  https://review.openstack.org/62954215:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: zuul: Add required project openstack/openstack-ansible  https://review.openstack.org/63029515:42
noonedeadpunkfolks, need 1 more +2 for the patch https://review.openstack.org/#/c/629930/15:43
*** cshen has joined #openstack-ansible15:50
*** vollman has joined #openstack-ansible15:56
*** cshen has quit IRC15:57
*** markvoelker has joined #openstack-ansible15:58
*** mathlin has joined #openstack-ansible16:00
*** priteau has quit IRC16:02
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova master: Cleanup files and templates using smart sources  https://review.openstack.org/58895116:09
*** evrardjp has quit IRC16:10
*** evrardjp has joined #openstack-ansible16:11
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance master: Cleanup files and templates using smart sources  https://review.openstack.org/58895916:13
cloudnullwhile folks are in a reviewing mood :) https://review.openstack.org/#/q/topic:smart-sources+(status:open)16:13
odyssey4mecloudnull pretty quiet today - no major drama... I pushed up a few patches to remove policy.json, even from stable branches - it seems we've been carrying them for a long time and a lot of services dropped them ages ago, some even in queens16:17
*** gkadam__ has quit IRC16:18
odyssey4meplenty of patches need review16:19
odyssey4me;)16:19
prometheanfireso... it looks like the ssl/tls vars are not used at all in os_barbican (defaults) should I just remove them?16:21
*** cshen has joined #openstack-ansible16:21
cloudnullodyssey4me looking now :)16:25
*** mathlin has quit IRC16:26
*** cshen has quit IRC16:27
cloudnullfor reference:16:28
cloudnullrocky policy changes need to reviewed - https://review.openstack.org/#/q/topic:smart-sources-stable/rocky+(status:open+OR+status:merged)16:28
cloudnullqueens policy changes need to be reviewed - https://review.openstack.org/#/q/topic:smart-sources-stable/queens+(status:open+OR+status:merged)16:28
*** luksky has quit IRC16:31
*** pcaruana has quit IRC16:36
hwoaranghey everybody. just so you know i am investigating the distro install failures on opensuse. sorry it took so long!16:38
odyssey4methanks hwoarang  - appreciate the time you're taking!16:39
*** shardy has quit IRC16:40
cloudnull++16:40
*** macza has joined #openstack-ansible16:40
cloudnullappreciate it hwoarang!16:41
*** gyee has joined #openstack-ansible16:47
pabelangerq: about updating services (specifically venvs) with OSA.  I can see, with nova you have a nova_venv_tag variable.  If, I have set that to 18.0.0, that will install that version.  However, if I rev that up to 18.0.1, I am assuming a new venv gets created with that version number16:50
pabelangerif that is right, does the nova service keep running when the new venv is created? If so, I am guessing when you restart the services the old copy stops and new venv starts16:51
pabelangercloudnull: odyssey4me: mnaser: ^16:51
cloudnullyes the venvs are tagged and on upgrade you'll get a new one. the role will restart the service using the new venv at the completion of the upgrade using a handler.16:53
cloudnullif the upgrade fails for any reason you should be able to roll the service back to the old venv16:53
pabelangercloudnull: okay, so I guess you've never had an issue with say systemd stopping the running service, using new venv16:53
cloudnullnope, i've never had that issue.16:54
cloudnullsystemd should keep track of the pid and kill it on service restart.16:54
*** mathlin has joined #openstack-ansible16:56
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/rocky: Resolve btrfs backing store variable inconsistency  https://review.openstack.org/62913716:59
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/queens: Resolve btrfs backing store variable inconsistency  https://review.openstack.org/62913816:59
jamesdentonthere are some nuances to that, for neutron anyway. jrosser and i observed some neutron agents still running on old venv post-upgrade. Not sure where that went16:59
pabelangercloudnull: okay cool, going to test it our for some local plays I am writing.  thanks for info17:02
cloudnullanytime!17:03
*** dcdamien has quit IRC17:05
pabelangercloudnull: if running from stable/rocky, does the venv become venv/nova-stable/rocky ?17:08
pabelangeror do you substring /17:08
*** cshen has joined #openstack-ansible17:11
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Remove legacy policy.json file  https://review.openstack.org/63028917:16
*** cshen has quit IRC17:17
*** DanyC has quit IRC17:17
*** hamzaachi has joined #openstack-ansible17:18
openstackgerritMerged openstack/openstack-ansible-galera_server stable/rocky: Tidy yum repository setup  https://review.openstack.org/63014017:18
*** DanyC has joined #openstack-ansible17:18
*** stuartgr has quit IRC17:21
*** mathlin has quit IRC17:22
*** DanyC has quit IRC17:23
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide support for ovs-sfc  https://review.openstack.org/62124917:23
*** fatdragon has joined #openstack-ansible17:28
openstackgerritMerged openstack/openstack-ansible master: Spice console doesn't work on aarch64+kvm.  https://review.openstack.org/62659317:28
*** ostackz has joined #openstack-ansible17:31
*** electrofelix has quit IRC17:31
*** vicky84 has joined #openstack-ansible17:31
ostackzAnyone using domains in rocky? For me domains worked in queens but cannot login to domains in rocky17:32
*** hamzaachi_ has joined #openstack-ansible17:33
*** lbragstad has quit IRC17:35
*** hamzaachi has quit IRC17:36
*** hamzaachi_ has quit IRC17:37
*** tosky has quit IRC17:40
cloudnullpabelanger it should be XX.0.0-devYYY17:42
cloudnullI think?17:43
jenglischi previously opened a bug https://bugs.launchpad.net/openstack-ansible/+bug/1782195 i'd like to fix properly, hence upstream17:43
openstackLaunchpad bug 1782195 in openstack-ansible "neutron l3 agent fails to spawn HA router" [High,In progress]17:43
jenglischthe l3-agent invokes a custom python script in a subprocess https://github.com/openstack/neutron/blob/9ad2e05088400a57f5e1fe0246006b9e213f101f/neutron/agent/l3/ha_router.py#L36817:43
jenglischbut l3-agent does not include it's own venv in it's path as i can see17:44
jenglisch cat /proc/17403/environ17:44
jenglischLANG=de_AT.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binHOME=/var/lib/neutronLOGNAME=neutronUSER=neutronSHELL=/bin/false17:44
jenglischin my issue i symlinked the binary, i don't thin thats the proper solution17:44
jenglischi'd rather add  /openstack/venvs/neutron-$VERSION/bin/ to the PATH in the systemd unit17:45
jenglischWDYT?17:45
cloudnullthat sounds reasonable17:47
jenglischfuthermore the l3-agent tries to kill the process again aftermore, however even with (imo) a generous rootwrap.d it isn't able to kill it17:47
jenglischhttp://paste.openstack.org/show/oJFFypq14ttbEt3K8Nmb/17:47
cloudnullyou should be able to add additional options to the system provided units under `/etc/systemd/system/<service_name>.d/`17:48
jenglischL90-94 got added by myself, i just saw afterwards that the python* kill thingy is already present for metadata anyway17:48
jenglischcloudnull: i know, but i'd like to provide an upstream fix17:48
jenglischor is this the prefered fix already?17:48
cloudnullah a fix into neutron specifically17:48
jenglischthis issue should hit all users with l3 ha routers currently17:49
jenglisch(in all releases from queens on)17:49
jenglischalthough i only tested it on queens, but the subprocess thingy is still in neutron-l3-agent master and i haven't found a (related) fix in openstack-ansible17:50
*** mathlin has joined #openstack-ansible17:51
cloudnullso is it a rootwarp command issue with the path being wrong?17:51
* cloudnull sorry may be being dense 17:51
openstackgerritMerged openstack/openstack-ansible master: Fixes empty metal_query  https://review.openstack.org/62993017:52
openstackgerritMerged openstack/openstack-ansible master: Make OVN track master branch  https://review.openstack.org/62991417:52
jenglischsorry for the confusion, those are actually two seperate bugs17:52
*** hwoarang has quit IRC17:52
jenglischthe first one is, that l3-agent isn't able to invoke neutron-keepalived-state-change at first, since it's not in the $PATH17:52
jenglisch(it's located in the venv)17:53
jenglischthe second one is: if the router gets deleted, neutron tries to kill it again, that doesn't work beacuse of rootwrap(?)17:53
jenglischi resolved the first one, initially, by symlinking the script from the venv env to /usr/local/bin/17:54
jenglischneutron expects this script in the $PATH as far as i can see17:54
jenglischhttps://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L36617:54
cloudnullah ok.17:54
*** dcdamien has joined #openstack-ansible17:55
cloudnullso the path issue is something we'd need to fix by making sure its in the service unit we (OSA) creates and that the neutron_sudoers file has the correct path.17:55
cloudnullthe additional rootwrap conf should go into upstream neutron though we can override it for a time, until if goes upstream17:56
cloudnull**it goes17:56
cloudnullthis is where we create the sudoers files https://github.com/openstack/openstack-ansible-os_neutron/blob/ef07cd02c5f469f7e5899109b132a93069463100/tasks/neutron_pre_install.yml#L113-L119 and this is where we'd need to make sure the path is included in the systemd unit file we create for neutron https://github.com/openstack/openstack-ansible-os_neutr17:57
cloudnullon/blob/ee5f750859fd0bb458b99493f15ebab1612362eb/tasks/main.yml#L92-L12317:57
cloudnullI'm not sure the missing PATH issue is a specific upstream neutron problem, it sounds like its something we're just not feeding into the service correctly17:58
openstackgerritMerged openstack/openstack-ansible-os_heat stable/queens: Only implement policy.json if an override is configured  https://review.openstack.org/62941017:58
openstackgerritMerged openstack/openstack-ansible-os_heat stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/62940917:58
*** hwoarang has joined #openstack-ansible17:59
*** stuartgr has joined #openstack-ansible17:59
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/rocky: Only implement policy.json if an override is configured  https://review.openstack.org/63019918:00
jenglischcloudnull: the sudoers file is actually fine, since it accepts everything in the secure_path where the venv's bin directory is already listed18:01
jenglischi think it's an OSA problem, since it only affects venv installations imo18:05
jenglischis upstream neutron venv compatible ?18:05
jenglischi'd like to add 'Environment=PATH=${venv}:$PATH' into the systemd unit. it seems like i need to extend the OSA systemd_service for that18:06
cloudnullyes neutron works in a venv just fine. at least it has for a few years now.18:07
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/queens: Only implement policy.json if an override is configured  https://review.openstack.org/63020018:09
*** mathlin has quit IRC18:10
jenglischhm, neutron spawns external processes in multiple modules, however it seems that this helper script is the only shell-out command which is shipped with neutron itself18:10
cloudnullthat entry into the systemd unit file should be able to be added by setting the init_config_overrides for the given service18:10
jenglischother process invokes includes radvd, keepalived and other software which is typically installed into $PATH18:10
cloudnullhowever, i think it would be a good addition to extend the systemd_service role to add Environment options when defined18:11
*** mmercer has quit IRC18:23
*** hamzaachi has joined #openstack-ansible18:27
openstackgerritJakob Englisch proposed openstack/ansible-role-systemd_service master: Add an option to configure a service environment  https://review.openstack.org/63037018:29
jenglischcloudnull: https://review.openstack.org/#/c/630370/2/templates/systemd-service.j2 like this?18:30
jenglischcould be used like this http://paste.openstack.org/show/741834/18:31
openstackgerritJakob Englisch proposed openstack/ansible-role-systemd_service master: Add an option to configure a service environment  https://review.openstack.org/63037018:32
jenglisch* https://review.openstack.org/#/c/630370/3/templates/systemd-service.j2 ; got a typo18:32
*** mathlin has joined #openstack-ansible18:34
jrosserjenglisch: interesting stuff - is it related to this? https://bugs.launchpad.net/openstack-ansible/+bug/178073318:38
openstackLaunchpad bug 1780733 in openstack-ansible "Neutron rootwrap error" [High,Triaged] - Assigned to David Wilde (david-wilde-rackspace)18:38
*** cshen has joined #openstack-ansible18:39
openstackgerritDamian Cikowski (dcdamien) proposed openstack/openstack-ansible-os_designate stable/rocky: Set project that owns managed resources  https://review.openstack.org/63037118:41
jenglischprobably, but this 'only' affects my second problem that rootwarp isn't able to kill the HA router helper script anymore18:41
jenglischthe first problem is the invocation itself, that's missing in the mentioned log, unfortunately18:42
jenglischhowever your bug could be a subsequent fault of the first one18:42
jrosseryeah, it felt very much in the same area18:43
*** cshen has quit IRC18:44
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_service master: Add the ability to set additional Environment settings  https://review.openstack.org/63037318:53
cloudnulljenglisch ^ that should give us an interface to add environment options as needed18:53
openstackgerritJakob Englisch proposed openstack/openstack-ansible-os_neutron master: Fix the subprocess invocation of neutron-l3-agent  https://review.openstack.org/63037418:59
jenglisch^ this would be the full fix of the first problem, depending on the systemd role addition18:59
*** cshen has joined #openstack-ansible19:00
openstackgerritJakob Englisch proposed openstack/ansible-role-systemd_service master: Add an option to configure a service environment  https://review.openstack.org/63037019:00
openstackgerritJakob Englisch proposed openstack/ansible-role-systemd_service master: Add an option to configure a service environment  https://review.openstack.org/63037019:04
*** cshen has quit IRC19:06
openstackgerritJames Denton proposed openstack/openstack-ansible-ops master: Add mnaio_data_disk_suffix var to support NVMe partition naming  https://review.openstack.org/63037519:14
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_service master: Add the ability to set additional Environment settings  https://review.openstack.org/63037319:15
cloudnullmy bad, jenglisch. ill abandon my patch and build off of yours19:17
jenglischactually yours is better documented and adds env file aswell, so i can just depend on your's as well :)19:17
*** cshen has joined #openstack-ansible19:19
jenglischis there a possibility to tell zuul to do another try, if just the dependency got updated?19:20
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_service master: Add docs and tests to the environment capabilities  https://review.openstack.org/63037619:24
cloudnullI abandoned mine and depended on yours :)19:25
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_service master: Add docs and tests to the environment capabilities  https://review.openstack.org/63037619:26
openstackgerritJames Denton proposed openstack/openstack-ansible-ops master: Add support for OVN in MNAIO deploy  https://review.openstack.org/63037719:27
*** Soopaman has joined #openstack-ansible19:31
jenglischfunfact: the only other executable which neutron expects (but not by default) in the path and ships itself is neutron-rootwrap which had already the full path added instead of the relvative one19:33
jenglischhttps://github.com/openstack/neutron/blob/7b850a9f6054ea640e262efc7fa1a843fd3c2b65/neutron/conf/agent/common.py#L8119:33
jenglischhttps://github.com/openstack/openstack-ansible-os_neutron/blob/614e3bd1d6c61124410b1e5ea7f999b73a227bdb/templates/neutron.conf.j2#L24519:33
jenglischbut changing the relative path for neutron-keepalived-state-change to the full qualified path is quite hard, if the thing is hardcoded19:35
jenglischhttps://github.com/openstack/neutron/blob/9ad2e05088400a57f5e1fe0246006b9e213f101f/neutron/agent/l3/ha_router.py#L36719:35
*** openstackstatus has quit IRC19:43
*** openstackstatus has joined #openstack-ansible19:43
*** ChanServ sets mode: +v openstackstatus19:43
jenglischjrosser: i guess i analyzed your problem as well19:46
jenglischfirst of all, the fix i proposed (by adding the venv's bin to PATH) is required, since the KillFilter depends on PATH19:46
jenglischhttps://github.com/openstack/oslo.rootwrap/blob/99f6383e97f59c110211bdbc60894befc3fdc4e9/oslo_rootwrap/filters.py#L15719:46
jenglischthe funny thing is, you can define an exec_path in rootwrap.conf, but rootwrap decides to just use PATH19:47
jenglischthe exec_path is only for the CommandFilter19:47
jenglischthe second problem is, that the KillFilter is entirely missing in neutron upstream19:47
jenglischhttps://github.com/openstack/neutron/blob/master/etc/neutron/rootwrap.d/l3.filters#L5919:47
jenglischbut they're sending SIGTERM explicitly in the code https://github.com/openstack/neutron/blob/8db1a47fa8d4a245ea7163599a10b6699761de6d/neutron/agent/l3/ha_router.py#L39419:48
mnaserjenglisch: you need to look at differences, recently we started using a new system in openstack called privsep19:50
mnaserwhich is a python native way of making calls instead of using rootwrap19:50
mnaseralso, that neutron keepalived thing, you're using centos right?19:50
mnaserme and guilhermesp ran into it, i think he's going to submit a patch for that soon19:50
mnaseralways good to check what codebase you're looking at, so stable/rocky is the branch you wanna read19:51
mnaseror whatever series you'r egoing for19:51
jenglischmnaser: yep, centos. currently queens, but as i checked privsep is just getting introduced, and rootwrap is still used for l3-agent on master19:56
*** DanyC has joined #openstack-ansible19:56
*** DanyC has quit IRC20:00
jenglischhm. actually in queens, it seems like extending rootwrap.d/l3.filters isn't necessary by accident20:01
jenglischonly the PATH fix is needed20:02
jenglischthey still have the python-yolo approach in there https://github.com/openstack/neutron/blob/stable/queens/etc/neutron/rootwrap.d/l3.filters#L2620:02
jenglischand the full cmdline is '/openstack/venvs/neutron-17.0.6/bin/python /usr/local/bin/neutron-keepalived-state-change [...]'20:03
jenglischso this should be the fix already, mnaser - https://review.openstack.org/#/c/630374/120:04
jenglischi'd be glad if guilhermesp could do a review :)20:05
openstackgerritJakob Englisch proposed openstack/openstack-ansible-os_neutron master: Fix the subprocess invocation of neutron-l3-agent  https://review.openstack.org/63037420:07
*** cshen has quit IRC20:08
*** luksky has joined #openstack-ansible20:12
*** openstack has joined #openstack-ansible20:18
*** ChanServ sets mode: +o openstack20:18
*** sreejithp has joined #openstack-ansible20:39
*** sreejithp has quit IRC20:39
*** sreejithp has joined #openstack-ansible20:41
*** cshen has joined #openstack-ansible20:42
jamesdentonanyone here work with networking-vpp for Neutron?20:42
*** sreejithp has left #openstack-ansible20:46
*** cshen has quit IRC20:46
*** DanyC has joined #openstack-ansible20:49
*** sreejithp has joined #openstack-ansible20:50
cloudnull^ jrosser20:50
cloudnullhe was working on (looking at?) some FIDO things back in the day20:50
jrosserwell i keep looking and wishing it was easier to try20:50
* cloudnull ran FIDO on a nuc for a min 20:51
jrosserimho it has great potential particularly for network nodes20:52
jrosserbut it is sad you have(had?) to go all-in with it and couldnt do a bit of mix/match20:52
jamesdentonany interest in getting it in OSA for dev?20:52
cloudnullhttps://i.stack.imgur.com/MNeE7.jpg20:53
jrosserits been a while since i looked but iirc there were a number of approaches20:54
jrosseryou could run it 'native' with etcd acting as a shim between neutron and vpp20:54
jrosseror you could go full SDN with opendaylight and so on and a shim between ODL and VPP20:54
*** Soopaman has quit IRC20:55
jamesdentonyeah i was just thinking the ML2 route.. native20:58
jrosseri'm interested to see how that would go20:58
jrosserit's properly innovative stuff underneath that has the potential for extreme performance, on a par with hardware routers20:59
jenglischi think we'll see a lot of change in the near future in routing/networking on linux and in software21:01
jenglischalso eBPF and XDP is a very hot topic currently21:01
openstackgerritMerged openstack/ansible-role-systemd_service master: Add an option to configure a service environment  https://review.openstack.org/63037021:02
*** corvus is now known as thecount21:02
*** thecount is now known as corvus21:02
jenglischi think cumulus neutron is currently a better chance as ODL21:03
jenglischhttps://docs.cumulusnetworks.com/display/DOCS/OpenStack+Neutron+ML2+and+Cumulus+Linux21:03
openstackgerritMerged openstack/ansible-role-systemd_service master: Add docs and tests to the environment capabilities  https://review.openstack.org/63037621:03
jenglischthe last time i checked ODL, i thought it would be faster and more stable if i implemented the stuff i needed myself21:05
jrosserjenglisch: lots of options around for l2, but much less so for l321:07
jenglischindeed, at least as core router21:08
jenglischcumulus switches may do l3 as well, but only on edges i think (i had not yet the chance to test them in depth)21:09
jrosseri mean l3 in openstack terms - no whitebox l2/l3 switch running cumulus is going to do the SNAT/DNAT you need21:10
jenglischhttps://docs.cumulusnetworks.com/display/DOCS/Routing 32k :/21:10
jenglischno DNAT/SNAT needed for ipv6 :321:11
jrosserstateful firewall, blah blah theres tons that a real router does that merchant switching silicon doenst21:11
jrosserhaving said that there will be a new breed of whitebox router this year21:12
jrosseras opposed to whitebox switch21:12
jenglischyep, XDP and eBPF is currently on a good road and they yet don't require to give your nic fully to DPDK21:14
jenglischhttps://people.netfilter.org/hawk/presentations/OpenSourceDays2017/XDP_DDoS_protecting_osd2017.pdf21:14
jenglischhttps://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/21:14
jenglisch^ just the first two links i remembered and could google quite fast21:15
*** cshen has joined #openstack-ansible21:18
jenglisch[22:12] <jrosser> having said that there will be a new breed of whitebox router this year << can you name a decent one already, or are they still all WIP ?21:22
*** cshen has quit IRC21:23
jrossernothing yet - but im told there will be silicon vendors selling routing chipsets this year rather than those being proprietary to single manufacturers21:25
*** radeks_ has quit IRC21:41
*** radeks_ has joined #openstack-ansible21:41
*** tosky has joined #openstack-ansible21:42
*** hamzaachi has quit IRC21:45
*** strattao has quit IRC21:50
jenglischhm. https://review.openstack.org/630374 i guess zuul marked this one as failed since the ODL xenial build timed out ; is there a re-run functionallity?21:50
*** radeks_ has quit IRC22:04
mnaserjenglisch: yes, you wanna comment with the words 'recheck' :)22:07
*** hamzaachi has joined #openstack-ansible22:10
*** vollman has quit IRC22:13
*** mmercer has joined #openstack-ansible22:16
*** hamzaachi has quit IRC22:17
*** mcela has quit IRC22:19
*** noonedeadpunk has quit IRC22:26
*** macza has quit IRC22:28
*** strobelight has quit IRC22:31
*** dcdamien has quit IRC22:32
*** markvoelker has quit IRC22:32
*** strobelight has joined #openstack-ansible22:33
*** mathlin has quit IRC22:41
*** DanyC has quit IRC22:55
*** mathlin has joined #openstack-ansible23:13
*** rjgibson has quit IRC23:17
*** cshen has joined #openstack-ansible23:19
*** cshen has quit IRC23:24
*** hwoarang has quit IRC23:46
*** mathlin has quit IRC23:47
*** hwoarang has joined #openstack-ansible23:47
*** pabelanger has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!