Monday, 2019-03-18

*** DanyC has quit IRC00:01
*** shyamb has quit IRC00:28
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest stable/rocky: Set tempest concurrency to int only  https://review.openstack.org/64378900:52
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible stable/rocky: Add support to deploy mistral  https://review.openstack.org/64261400:54
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible stable/rocky: nova: restart instead of reloading services  https://review.openstack.org/64284200:54
*** marst has joined #openstack-ansible01:31
*** DanyC has joined #openstack-ansible02:02
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest stable/rocky: Set tempest concurrency to int only  https://review.openstack.org/64378902:03
*** DanyC has quit IRC02:07
*** gkadam_ has joined #openstack-ansible02:11
*** gkadam_ has quit IRC02:17
*** BjoernT has joined #openstack-ansible02:19
*** irclogbot_0 has quit IRC02:23
openstackgerritMohammed Naser proposed openstack/openstack-ansible-galera_server master: defaults: bump deployed version to 10.3.13  https://review.openstack.org/64379802:25
*** marst has quit IRC02:40
*** BjoernT has quit IRC02:48
*** BjoernT has joined #openstack-ansible02:49
*** nicolasbock has quit IRC02:58
*** shyamb has joined #openstack-ansible03:05
*** BjoernT has quit IRC03:06
*** shyam89 has joined #openstack-ansible03:11
*** shyam89 has quit IRC03:14
*** shyamb has quit IRC03:14
*** BjoernT has joined #openstack-ansible03:16
*** bgmccollum has quit IRC03:56
*** bgmccollum has joined #openstack-ansible04:00
guilhermespchandankumar: https://review.openstack.org/#/c/643789/04:05
guilhermespbackported but seems we have two tempest tests failing. Have you seen this http://logs.openstack.org/89/643789/2/check/openstack-ansible-functional-opensuse-423/3873aa6/logs/ara-report/result/1b1e5b56-23b9-494c-817d-8a6be46b2e45/?04:07
*** udesale has joined #openstack-ansible04:08
*** BjoernT has quit IRC04:20
*** chhagarw has joined #openstack-ansible04:45
*** ivve has quit IRC05:22
*** bgmccollum has quit IRC05:28
*** bgmccollum has joined #openstack-ansible05:31
*** mathlin has quit IRC05:34
*** mathlin has joined #openstack-ansible05:36
*** jbadiapa has joined #openstack-ansible06:06
*** miloa has joined #openstack-ansible06:10
openstackgerritMerged openstack/openstack-ansible master: Bump SHAs for master  https://review.openstack.org/64220806:18
*** ivve has joined #openstack-ansible06:30
*** ChosSimbaOne has quit IRC06:48
*** ChosSimbaOne has joined #openstack-ansible06:48
*** mkuf_ has joined #openstack-ansible06:51
*** mkuf_ is now known as mkuf06:51
*** phasespace has quit IRC06:54
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.openstack.org/64386406:55
*** kopecmartin|off is now known as kopecmartin07:03
*** ivve has quit IRC07:07
*** luksky has joined #openstack-ansible07:08
*** DanyC has joined #openstack-ansible07:08
*** luksky has quit IRC07:18
*** pcaruana has joined #openstack-ansible07:19
*** DanyC has quit IRC07:24
*** cshen has joined #openstack-ansible07:29
jrosserMorning07:34
*** ivve has joined #openstack-ansible07:38
fnpanicgood morning07:44
*** hamzaachi has joined #openstack-ansible07:57
*** hamzaachi has quit IRC07:59
*** DanyC has joined #openstack-ansible08:00
*** luksky has joined #openstack-ansible08:01
*** mkuf has quit IRC08:04
*** tosky has joined #openstack-ansible08:05
*** fnpanic has quit IRC08:06
*** tosky has quit IRC08:07
*** hamzaachi has joined #openstack-ansible08:08
*** phasespace has joined #openstack-ansible08:10
jrosserchandankumar: has anything changed in tempest/os_tempest with the amount of concurrency? We see jobs failing tempest in the last couple of weeks with lots of load on keystone....08:15
*** tosky has joined #openstack-ansible08:25
*** priteau has joined #openstack-ansible08:25
chandankumar jrosser yes08:38
*** DanyC has quit IRC08:39
chandankumarjrosser: https://review.openstack.org/#/c/643423/08:39
chandankumarjrosser: what was the default concurrency set in the jobs?08:39
chandankumardefault is 008:39
jrosserwell that is a good question :)08:39
jrosseri think the answer is that it was never set http://codesearch.openstack.org/?q=tempest_run_concurrency&i=nope&files=&repos=08:40
chandankumarjrosser: from here I am seeing basically two for 4 http://codesearch.openstack.org/?q=tempest_concurrency&i=nope&files=&repos=08:40
chandankumarlet me propose a pathc to set it to 208:41
jrosserwhat happens when it is 0? N-CPU or something?08:41
chandankumarjrosser: https://github.com/mtreinish/stestr/blob/master/stestr/commands/run.py#L75 -> no. of cpus count08:43
jrosserchandankumar: hmmm right yes it would certainly be good to define it as a small number by default then08:44
chandankumarin os_tempest itself or in openstack-ansible-tests?08:44
jrosserthere should be some sensible default behaviour in os_tempest imho08:46
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Set tempest_run_concurrency defaults to 2  https://review.openstack.org/64388008:48
chandankumarjrosser: I need some help on this review https://review.openstack.org/#/c/643573/08:48
chandankumarjrosser: currently giving pip_install_upper_constraints is undefined  http://logs.openstack.org/73/643573/1/check/openstack-ansible-deploy-aio_distro_lxc-opensuse-423/c015fad/job-output.txt.gz#_2019-03-15_14_17_53_50866008:49
*** luksky has quit IRC08:55
*** luksky has joined #openstack-ansible08:57
jrosserchandankumar: i think that pip_install_upper_constraints is normally only defined for source installs08:57
chandankumarjrosser: but there tempest is installed from source?08:58
chandankumar*.08:58
chandankumarfor rocky distro support is not there08:59
chandankumarlet me make a single modification there08:59
chandankumarby setting tempest_install_method == source But not sure it will work08:59
jrosserhmmmm ok we had some discussion about this before and it's a but ugly09:01
jrosser*bit09:01
*** simon-AS559 has joined #openstack-ansible09:02
* chandankumar waits for odyssey4me to come.09:04
*** mkuf has joined #openstack-ansible09:11
*** DanyC has joined #openstack-ansible09:18
*** fnpanic has joined #openstack-ansible09:27
*** luksky has quit IRC09:31
*** rgogunskiy has joined #openstack-ansible09:34
*** CeeMac has quit IRC09:34
*** CeeMac has joined #openstack-ansible09:40
CeeMacmorning09:40
*** shyamb has joined #openstack-ansible09:41
*** tsi has joined #openstack-ansible09:51
tsiHello . I have an AIO for testing ( release 18.1.4) and i would like to add some of the openstack services such as telemetry . From https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html#bootstrap-the-aio-configuration as I understand it all  i have to do is copy the correct yml.aio to /etc/openstack_deploy/conf.d/ and then do i have to re run bootstrap-aio  and the rest of the openstack-ansible setup comma09:59
*** shyamb has quit IRC10:04
openstackgerritMerged openstack/openstack-ansible master: Pass ceph_mons as a list of inventory hosts  https://review.openstack.org/64369510:13
*** shyamb has joined #openstack-ansible10:14
*** luksky has joined #openstack-ansible10:21
*** udesale has quit IRC10:28
*** shyamb has quit IRC10:30
*** shyamb has joined #openstack-ansible10:30
*** nicolasbock has joined #openstack-ansible10:35
*** jbadiapa has quit IRC10:44
*** simon-AS5591 has joined #openstack-ansible11:00
*** simon-AS559 has quit IRC11:04
*** mkuf has quit IRC11:08
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Set tempest_run_concurrency defaults to 2  https://review.openstack.org/64388011:10
*** flaviosr has joined #openstack-ansible11:18
openstackgerritChandan Kumar proposed openstack/openstack-ansible stable/rocky: Always set pip vars for tempest install  https://review.openstack.org/64357311:24
chandankumarjrosser: ^^ switched it to source for tempest let's see how it goes!11:24
*** mkuf has joined #openstack-ansible11:29
*** shyamb has quit IRC11:31
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Install user supplied CA certificates into system trust store  https://review.openstack.org/64144511:36
*** kaiokmo has joined #openstack-ansible11:39
*** kaiokmo has quit IRC11:39
*** kaiokmo has joined #openstack-ansible11:40
*** udesale has joined #openstack-ansible11:45
*** maxbab has joined #openstack-ansible11:47
*** ansmith has quit IRC11:49
*** shyamb has joined #openstack-ansible11:56
*** udesale has quit IRC12:01
*** udesale has joined #openstack-ansible12:02
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_tempest stable/rocky: Revert "Pin stestr to 2.3.0 for rocky"  https://review.openstack.org/64392312:04
odyssey4mechandankumar jrosser o/12:07
jrosserhello!!12:07
*** cshen has quit IRC12:14
*** cshen has joined #openstack-ansible12:22
*** markvoelker has quit IRC12:22
jrosserchandankumar: there is something I don't understand12:27
jrosserpreviously we didnt define tempest_run_concurrency12:27
*** udesale has quit IRC12:27
jrosserso according to this line http://git.openstack.org/cgit/openstack/openstack-ansible-os_tempest/tree/tasks/tempest_run.yml#n24 we should either have --concurrency <N> or --serial on the tempest run command12:28
*** udesale has joined #openstack-ansible12:28
jrosserbut here for example, we have neither http://logs.openstack.org/95/643695/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/d7458c2/logs/ara-report/12:28
tsiHello . I have an AIO for testing ( release 18.1.4) and i would like to add some of the openstack services such as telemetry . From https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html#bootstrap-the-aio-configuration as I understand it all  i have to do is copy the correct yml.aio to /etc/openstack_deploy/conf.d/ rename them,and then do i have to re run bootstrap-aio  and the rest of the openstack-ansible12:31
DimGRhi , i have opened this issue but not sure if its a bug or not https://bugs.launchpad.net/openstack-ansible/+bug/182059112:32
openstackLaunchpad bug 1820591 in openstack-ansible "openstack-ansible AIO , heat role detects wrong IP " [Undecided,New]12:32
chandankumarodyssey4me: \o/12:32
jrossertsi: i think you can just drop those new files into conf.d and the run the playbooks for the new components you want to deploy12:32
chandankumarodyssey4me: https://review.openstack.org/#/c/643573/ I have updated this patch, not sure that is the correct fix12:33
chandankumarplease have a look@12:33
odyssey4metsi no need to re-run bootstrap-aio, just run the playbooks that deploy openstack12:33
odyssey4metsi ie setup-hosts, setup-infrastructure, setup-openstack12:33
tsiodyssey4me: ok thanks .12:33
DimGRjrosser thanks12:33
jrosserDimGR: that is indeed a bug - sort of12:34
odyssey4mejrosser chandankumar yeah, IIRC we use --serial for tempest runs because we found that when using concurrency we'd end up with differing amounts of tests executing on each run12:34
jrosserbut i see --serial is missing12:34
jrosserlike here http://logs.openstack.org/95/643695/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/d7458c2/logs/ara-report/12:35
chandankumarjrosser: yes12:35
jrossersorry here http://logs.openstack.org/95/643695/1/gate/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/d7458c2/logs/ara-report/result/1d75b12f-02d2-404a-bf56-4fa4d5a0d285/12:35
chandankumarjrosser: something wrong12:35
chandankumarjrosser: https://review.openstack.org/#/c/643423/2/tasks/tempest_run.yml got merged12:35
chandankumarfor concurrency12:36
jrosseroh12:36
chandankumarjrosser: http://logs.openstack.org/38/640138/2/check/openstack-ansible-functional-centos-7/44fd417/logs/ara-report/result/6ebae9b6-5ccd-45fd-9949-b361c3f63821/ it is from old12:37
chandankumarhere also --serial is missing12:37
noonedeadpunkHi everyone12:37
jrosserhmmm ok i think we need to get this back to --serial because it pretty much DoS on keystone12:37
noonedeadpunkHave a question - do I need to add tag to every task inside included play, or it's enough to place tag on the include?12:38
chandankumarjrosser: yes , looking in the jinja12:38
jamesdentonmornin folks12:39
odyssey4menoonedeadpunk the answer is, it depends - I believe there are some bugs in ansible depending on whether import or include are used, and whether they are included/imported plays/tasks - best is to try what you think is good and to test it12:41
*** aedc has joined #openstack-ansible12:41
nicolasbocknoonedeadpunk: Depending on how you include, you can apply tags to he included tasks: https://docs.ansible.com/ansible/latest/modules/include_tasks_module.html#include-tasks-module12:43
noonedeadpunkodyssey4me nicolasbock: I see. Actually the question is regarding https://review.openstack.org/#/c/639960/2 - tasks/horizon_translations_update.yml is included like any other task in this role. And Frank says, that horizon-translations is not working to him.12:44
noonedeadpunkSo for me it looks like that I need to test every tag for the role12:44
odyssey4menoonedeadpunk I can confirm that using import_tasks and tags like https://github.com/openstack/openstack-ansible-os_glance/blob/master/tasks/main.yml#L51-L68 definitely works12:44
jamesdentonIf it help, the changes here allowed existing tags to work again: https://review.openstack.org/#/c/637026/12:45
noonedeadpunkjamesdenton: Yep, I think I need to do the same for horizon12:47
jrosserDimGR: by making the changes in your bug report you will have made the basic heat cli work, but unless your created servers/vm can contact your internal VIP (possibly for a lab, highly unlikely for production) then heat (and magnum) wont work properly12:47
noonedeadpunkSo the question now - should I do it before mentioned patch, or right after it?12:48
*** ansmith has joined #openstack-ansible12:48
*** shyamb has quit IRC12:48
noonedeadpunkThanks for your answers:)12:53
noonedeadpunkI mean, for pointing me to import/include12:53
*** pcaruana|afk| has joined #openstack-ansible12:54
*** pcaruana has quit IRC12:56
jamesdentonTrying to rerun os-neutron-install.yml and getting hung up on the python_venv_build tasks: https://pasted.tech/pastes/88f0b29f9df256c74cd78a6b32a71b29a4235d03. Haven't OSA'd in a while. Any known issue here, or should I keep digging?13:09
guilhermespmornings!13:12
jamesdentonhello!13:12
guilhermespjamesdenton: o/13:13
*** BjoernT has joined #openstack-ansible13:13
odyssey4mejamesdenton I'm guessing that's a stale fact issue - force the regathering of facts, or just wipe the fact cache.13:13
guilhermesphum ok chandankumar so that case just wait the new tempest release to be able to unblock some of the rocky jobs? these are some examples https://review.openstack.org/#/c/642614/7 https://review.openstack.org/#/c/642842/13:13
chandankumarguilhermesp: for rocky release we have a work in progress patch https://review.openstack.org/#/c/643573/13:14
guilhermesphuuum ok chandankumar watching it13:24
chandankumarodyssey4me: http://logs.openstack.org/73/643573/2/check/openstack-ansible-deploy-aio_distro_lxc-opensuse-423/f78a1a5/job-output.txt.gz#_2019-03-18_12_35_37_745883 still failing with same error13:26
chandankumarodyssey4me: 'pip_install_upper_constraints' is undefined13:26
odyssey4mechandankumar yep, will look into it a bit later13:26
*** marst has joined #openstack-ansible13:26
odyssey4meI have a call in a bit which I need to prep for.13:27
chandankumarodyssey4me: sure13:27
jamesdentonodyssey4me Turns out I had to rebuild facts for the repo container. Of course I purged them beforehand. Would have been nice to compare.13:30
openstackgerritMerged openstack/openstack-ansible-os_nova master: setup: allow custom conductor host  https://review.openstack.org/62874313:33
*** hwoarang has quit IRC13:37
*** hwoarang has joined #openstack-ansible13:38
*** marst has quit IRC13:41
*** jwitko has joined #openstack-ansible13:47
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: [DNM] serial testing Syntex is correct, need to dig why serial flag is not showing  https://review.openstack.org/64394713:48
*** pcaruana|afk| has quit IRC13:48
mnaseri am not sure what to do about opensuse in rocky13:51
odyssey4memnaser it is a little odd that it's only opensuse hitting the issue, but I'll spend some time after my call looking into it13:59
mnaserthanks odyssey4me and good luck :)14:01
nicolasbockI am pulling up an env to test the rocky/opensuse issues14:03
nicolasbockI am seeing a different failure though14:04
jrossermnaser: i thought this may be some positive movement becasue it looks like an opensuse/rocky test has worked there https://review.openstack.org/#/c/643550/14:06
jrosserbut not in the integrated repo though14:07
DimGRjrosser in other words the changes i did were not enough for heat to work properly ? this is a test lab and the created instances can contact the internal vip14:07
*** marst has joined #openstack-ansible14:07
jrosserDimGR: ok, that should be OK then I think14:08
DimGRi  have had instaces working fine for few weeks , its only now that i needed heat and found this bug14:09
DimGRhope it gets resolved with a newer release14:09
jrosserDimGR: there is a bit of ambiguity in the heat code with how keystone endpoints are handled, see the details here http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002925.html14:12
jrosserso it's not really an OSA problem as such, we can configure either internal or external endpoints in heat.conf right now, just different things break depending on which you pick, sadly14:13
* DimGR checking14:13
*** mgariepy has quit IRC14:18
mkufHey there, I'd like to serve all my keystone endpoints (public/internal/admin) via https. Setting 'keystone_service_proto: https' in user_variables.yaml and running haproxy-install.yml and os-keystone-install.yml works fine and all Keystone-Endpoints are available via https. Running another playbook (e.g. os-glance-install) fails at the 'ensure glance service' stage, as it is still trying to14:24
mkufconnect via http. http://paste.openstack.org/show/747967/  Is this behaviour intended or am I missing something essential here?14:24
mkuf^ this occures in an AIO setup on Queens 17.1.714:26
evrardjpguilhermesp: jrosser can you vote on https://review.openstack.org/#/c/642738/1 ?14:31
evrardjpthe freeze has merged I can finally release14:31
*** nurdie has joined #openstack-ansible14:31
jrosserevrardjp: done14:31
evrardjpthanks14:32
jrossermkuf: it looks to me like something is making an https request but finding a certificate that is not trusted, but there is not much context in your error message really14:33
*** pcaruana has joined #openstack-ansible14:33
*** BjoernT has quit IRC14:35
*** maxbab has quit IRC14:37
*** BjoernT has joined #openstack-ansible14:40
*** shyamb has joined #openstack-ansible14:41
cloudnullmornings14:42
evrardjpmorning14:43
*** xenefix has joined #openstack-ansible14:50
cloudnullhows it on this fine day of days :)14:52
mkufjrosser: would a full output of the playbook run be more helpful or are there any other switches I can add to assist troubleshooting except -vvv?14:52
mkufjrosser: the url at the end of line 8 uses http, if this is a certificate error shuldn't there be any ssl exception?14:52
spotzhey cloudnull evrardjp14:53
evrardjphey14:53
jrossermkuf: oh right, missed that :/14:53
cloudnullhola14:54
cloudnullevrardjp have you messed much with the nova_api db things?14:54
cloudnullor anyone for that matter?14:54
spotzcloudnull: Beyond manual setups I have not14:55
cloudnullwith cell rabbit connection information stored in the db, i'm wondering what the official steps to update the db are, besides running an `update` query14:56
cloudnullIE: https://pasted.tech/pastes/09b13478ca0dddd56b0779ee42f35a6a318a4d3315:01
cloudnulland a service restart does not update the fields15:01
cloudnullkinda of a bummer when trying to move/scale rabbitmq nodes.15:02
*** BjoernT_ has joined #openstack-ansible15:02
cloudnullsame story for db connections15:02
*** shyamb has quit IRC15:03
*** phasespace has quit IRC15:03
spotzcloudnull: I think it's changed with a restart of the nova api, etc. Least when doing installs if you need to change that's the proceedure. I do not believe there is a reload function15:03
cloudnullon another note, anyone want to give this a shove through https://review.openstack.org/#/c/643612 ?15:03
spotzlooking15:04
*** BjoernT has quit IRC15:05
cloudnullspotz I restarted the services a few times, in fact went through a little hair pulling event last weekend trying to figure out why my cluster was so unstable: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2019-03-17.log.html#t2019-03-17T04:42:1115:06
cloudnullI removed the venvs, rebuilt all the things, and it wasn't until I started hacking on oslo messaging and logging all kinds of random things that I caught the bad rabbitmq connection string.15:07
*** BjoernT has joined #openstack-ansible15:07
DimGRjrosser i read the pipermail list about it . Its the mother of all bugs this heat bug15:07
spotzcloudnull: oh oh.... I think there's a second setting if you're having issues let me think@15:07
cloudnull++ would greatly appreciate it :)15:07
spotzcloudnull: Bringing up old notes give me a few15:09
*** BjoernT_ has quit IRC15:10
chandankumarjrosser: hello15:13
chandankumarjrosser: the serial syntex was correct15:13
jrossermnaser: i dug a bit at your galera upgrade patch and it can't find wsrep_sst_xtrabackup-v215:13
chandankumarjrosser: but on this patch https://review.openstack.org/#/c/643947/ -> serial is alive now http://logs.openstack.org/47/643947/1/check/openstack-ansible-functional-centos-7/aa78849/logs/ara-report/result/ba5c68f4-2c97-4835-998f-30bcb61707fe/15:13
jrosseri'm not sure where that is supposed to be installed from15:14
mnaserjrosser: i looked so much and i couldn't find it either15:15
mnaser:\15:15
jrosseridk which package that is supposed to be in15:15
mnaseri looked and couldn't find it :\15:15
jrosserthere are a ton of other percona-* packages and i was going to just throw some of those on to see what they contained15:15
jrosseri've also seen those galera "this is probably a bug" error messages15:16
spotzcloudnull: What port is rabbit? Not finding what I'm looking for but I may be misremembering what I need to look for:)15:16
*** miloa has quit IRC15:17
*** tsi has quit IRC15:17
cloudnullspotz: 567115:18
cloudnullthough could also be on 567215:18
cloudnullssl vs non-ssl15:18
chandankumarjrosser: I am modifying the dnm patch15:18
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Bring back the --serial flag alive  https://review.openstack.org/64394715:21
spotzcloudnull: this is going to drive me insane as I've told people this setting before and can't find it!15:24
*** BjoernT has quit IRC15:37
*** BjoernT_ has joined #openstack-ansible15:37
*** mgariepy has joined #openstack-ansible15:38
odyssey4mejrosser mnaser looks like the heat discussion on the ML has yielded some good advice and a few actionable options - one is a workaround we can implement (adding '[clients_keystone]/auth_uri') in rocky, and the other is a more suitable fix for the future15:46
jrosserodyssey4me: we got a patch already https://review.openstack.org/#/c/642812/15:47
jrosseri think there are two parts, a new config variable which shouldn't change any behaviour unless set15:47
jrosserthen a later revisit of the whole keystone endpoint handling at some point later15:47
odyssey4meboom gshippey is rocking it!15:48
jrosseri don't think anything can be done right now with the existing config options15:48
chandankumarodyssey4me: jrosser https://review.openstack.org/#/c/643947/ will bring serial flag alive15:49
chandankumarodyssey4me: jrosser I am thinking of doing the same for all long shell scripts, if it is ok?15:49
odyssey4mejrosser it seems that adding '[clients_keystone]/auth_uri' should work?15:49
jrosseradjusting any of the existing vars away from internal endpoint breaks our deployment here, in some way15:50
odyssey4mechandankumar anything to make things readable is good - assuming that works... not sure if you might hit some whitespace shenanigans there15:50
odyssey4mejrosser ah ok, then this might get a bit complicated because it'll involve a stable backport... but let's get something into master first and worry about that later15:51
jrosseryes, we've tried to keep that patch super super minimal and not change any existing (maybe wrong) behaviour so that it will backport15:51
mnaserwoot.  Yeah, maybe might be good15:54
mnaserTo update the ML15:54
*** cshen has quit IRC15:56
tridentcloudnull: Just a heads up if you have messaging related instabilities: https://bugs.launchpad.net/oslo.messaging/+bug/180095715:57
openstackLaunchpad bug 1800957 in oslo.messaging (Ubuntu) "Upgrading to pike version causes rabbit timeouts with ssl" [Undecided,New]15:57
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_heat master: [DNM] heat tempest tests finding  https://review.openstack.org/63069415:57
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_heat master: [DNM] heat tempest tests finding  https://review.openstack.org/63069415:57
cloudnulltrident: oslo.messaging is working great. Sadly the issue I have at this time is related to the rabbit connection string being stored in the nova-api db and not being updated when I change things.15:59
cloudnullI guess maybe that is something that should be handled in oslo?15:59
*** weshay|rover is now known as weshay16:02
tridentcloudnull: Yeah, that's not dynamic at all. But might make sense to make it so...16:02
tridentAs far as I have seen that is. Have always had to change it manually...16:02
tridentOr with nova-manage16:02
tridentcloudnull: I am not sure if it is supposed to be dynamically updated though. Might make more sense to have osa update it if rabbit settings have changed?16:05
*** mkuf has quit IRC16:11
openstackgerritLogan V proposed openstack/openstack-ansible stable/rocky: Pass ceph_mons as a list of inventory hosts  https://review.openstack.org/64426516:12
*** cmart has joined #openstack-ansible16:12
jrossermnaser: magnum related, if anyone ever needs to use magnum.conf/openstack_ca_file=<blah> I did this https://review.openstack.org/#/c/641445/16:13
openstackgerritLogan V proposed openstack/openstack-ansible stable/queens: Pass ceph_mons as a list of inventory hosts  https://review.openstack.org/64426716:14
openstackgerritLogan V proposed openstack/openstack-ansible stable/pike: Pass ceph_mons as a list of inventory hosts  https://review.openstack.org/64426816:15
odyssey4mecloudnull trident I dunno why it's in the DB, but it makes sense to me that if it changes in the .conf file, then on the next service load it should update it in the DB... or it should just not be in the DB. :p Maybe this is another one of those things that really should go to the ML for discussion and to figure out a solution. Either that, or perhaps a discussion in #openstack-nova at least.16:15
tridentcloudnull: The case here I think is that you might want to use different settings for each cell. The transport_url there is set when creating the cell with nova-manage cell__v2 create_cell16:19
odyssey4metrident if that is the case, then perhaps we should use nova-manage to always set the transport URL to whatever's current as a task in the nova role? That way it would always be the current one and we could forget about the hassle16:21
*** udesale has quit IRC16:21
*** BjoernT_ has quit IRC16:21
odyssey4meideally we should only set it again if it is changing, but then we'd have to figure out a way to determine the change - it might just be simpler to force it every time16:22
tridentcloudnull: Hm, the behaviour seem to have changed in OSA though. It doesn't seem to set transport-url when creating cells any longer. IIRC if you don't specify transport_url in cell db, it will use the default from config.16:22
tridenthttps://github.com/openstack/openstack-ansible-os_nova/blob/ab9642f95ca870f2616dfab3f9e22937646cc648/tasks/nova_db_setup.yml#L12116:22
chandankumarodyssey4me: I will propose a seperate patch to fix all long shell script for os_tempest tomorrow!16:22
*** cmart has quit IRC16:23
*** DanyC has quit IRC16:26
tridentcloudnull, odyssey4me: Changed here: https://github.com/openstack/openstack-ansible-os_nova/commit/2448db8064a6ef019f4fa75d700345ca65a905e1#diff-794858b9a821f0ebca63fffea0e0143b16:27
*** DanyC has joined #openstack-ansible16:27
*** shyamb has joined #openstack-ansible16:28
tridentSo, for transport_url this should no longer be an issue for new installations as transport-url is no longer provided when creating the cells so it will use the default from config which in the majority of cases will make sense.16:30
*** BjoernT has joined #openstack-ansible16:31
*** DanyC has quit IRC16:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Always set pip vars for tempest install  https://review.openstack.org/64357316:32
tridentHowever I'd think for an existing installation the easiest way to go if you just want it to use default is to just clear it out of the db (with the reservation that I have not tested it).16:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Always set pip vars for tempest install  https://review.openstack.org/64357316:33
odyssey4mechandankumar mnaser ^ I think that will sort out rocky's current integrated build failure16:34
chandankumarodyssey4me: ah nice thanks guilhermesp ^^ will fix tempest issue!16:37
*** shyamb has quit IRC16:46
*** SmearedBeard has joined #openstack-ansible16:48
guilhermespcool! thanks chandankumar odyssey4me16:48
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Always set pip vars for tempest install  https://review.openstack.org/64357316:51
*** mmercer has joined #openstack-ansible16:52
openstackgerritMerged openstack/openstack-ansible master: Revert "Freeze for milestone release"  https://review.openstack.org/64273816:55
*** hamzaachi has quit IRC17:04
*** ivve has quit IRC17:13
*** DanyC has joined #openstack-ansible17:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Update role for new source build process  https://review.openstack.org/62034017:13
openstackgerritMerged openstack/openstack-ansible-ceph_client master: Add additional linked libs for cephfs  https://review.openstack.org/64361217:17
*** BjoernT has quit IRC17:19
*** DanyC has quit IRC17:21
*** DanyC has joined #openstack-ansible17:22
*** simon-AS5591 has quit IRC17:22
*** kopecmartin is now known as kopecmartin|off17:22
*** jwitko has left #openstack-ansible17:24
evrardjpjrosser: are you there?17:34
evrardjpwell not only jrosser, people that are really old in here, for example cloudnull odyssey4me....17:34
chandankumarodyssey4me: jrosser mnaser cloudnull https://review.openstack.org/#/c/643947/ can we get this merged, it fixes one of the oldest bug17:35
evrardjpIf you were to redo multi os distro support, would you make sure all the distros are configured the same, or like we do it today, embracing the differences per distro?17:35
evrardjpI would do it like we do, but I am not strongly in favour or not17:35
chandankumarbug == never filed17:36
odyssey4meevrardjp https://media.giphy.com/media/K6aZlaZEJjt4s/giphy.gif17:36
evrardjphahaha17:37
evrardjpyou're totally like that now17:37
odyssey4meevrardjp well, we do a weird mix now, which is a bit confusing - and I find that people who try it, and the rest of us get all confused all the time too17:37
evrardjpyeah17:37
evrardjpI am not surprised17:37
evrardjpI wonder what we can do better17:38
odyssey4mesome packages assume the setup which they use for their 'native' tooling17:38
evrardjpor more17:38
odyssey4mewe try to block that and smash it into what we do17:38
evrardjp"what did we learn"17:38
odyssey4methe trouble is that every distro does things differently17:38
odyssey4meso, quite frankly... just don't do it IMO17:38
evrardjpyeah, but I wonder if we should say "no let's do it OUR way, and disregard ALL distro" or the other way around17:39
odyssey4meeither pick one distro and only support it on that, or have each distro setup in the way each distro packager expects things setup17:39
cloudnullevrardjp I'd probably keep things using the current pattern of embracing the differences. early on we tired to do it all ONE way and that was fraught with peril.17:39
cloudnulli think we're still paying the price for some of those decisions17:39
odyssey4methe cost of maintaining it all is pretty high though, because you're always context switching17:39
chandankumardistro == how they compile the stuff, restart the services and where they dump the file, but distro makes some things stable so that one can use it17:39
evrardjpodyssey4me: I agree with you, it's harder, because it means more pressure on maintainers17:40
evrardjp"I don't know opensuse way"17:40
odyssey4meyeah - we've also got a few things where we symlink things so that the differences between distributions are immaterial, but that's more maintenance17:40
evrardjpyet, I believe cloudnull is right too: There is a price of doing everything the same way: "Oh god selinux broke"17:40
odyssey4meso, I think it really comes down to what you're trying to achieve17:41
evrardjpodyssey4me: to be honest, I kinda hate that, it's less obvious for the operations at first sight17:41
evrardjpodyssey4me: yeah, nothing perfect in here :)17:41
odyssey4meI would think that if we only had one way of deploying - eg: with nspawn containers - and we made a rule that the containers would only be one distro - then life would be quite simple.17:42
odyssey4meBut alas, distro vendor licensing makes that near impossible.17:42
evrardjpWell I am thinking of simplifying things for opensuse -- I want to have less moving parts and get opensuse back on track on my own spare time17:42
evrardjpI believe the problems would still happen: people don't know this and maybe don't want to maintain it17:43
evrardjpnext to that, I have $dayjob which kinda has the same kind of problem, and I wonder what are the best way forward17:43
evrardjpI don't want to repeat mistakes :)17:43
odyssey4meIf there was an easy way to map out the differences and just carry them like some sort of constant, that'd be nice.17:45
evrardjpyou mean a lib that projects like ansible/puppet/anything could use?17:46
evrardjpEVIL17:46
evrardjpor you mean a standard?17:46
*** SmearedBeard has quit IRC17:46
evrardjphttps://xkcd.com/927/17:46
odyssey4melol, yeah17:46
odyssey4melet's not do that17:46
*** bgmccollum has quit IRC17:47
chandankumarjrosser: cloudnull thanks!17:48
odyssey4mechandankumar arxcruz I've hit a bit of a block with regards to stackviz and I wonder if you're available to talk it through a bit?17:49
arxcruzodyssey4me: sure17:49
arxcruzwhatsup ?17:49
odyssey4mecurrently stackviz is installed by default in the tempest role (which I wonder if appropriate, given that stackviz is more of a test tool than a production tool)17:50
chandankumarodyssey4me: /me will help tomorrow. mind is not working, in sleeping/watching movie mode17:50
odyssey4meit's installed from the tarball: https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L33417:50
chandankumarbut arxcruz can help you! He is a super awesome guy!17:51
arxcruzodyssey4me: okay...17:51
*** chandankumar is now known as raukadah17:51
odyssey4meI'm working on changing the way that the python builds are done to get rid of our 'repo build' and exclusively use the venv build role, but I'm hitting the situation where I can build a stackviz wheel but I have no way to then tell pip to use that wheel, because there is no constraint to work with17:52
*** cmart has joined #openstack-ansible17:52
odyssey4meOf course I can change the way it installs to use git+ like we do for everything else, then it would install using the same mechanism and model and life can carry on - but then I wonder how that would affect TripleO's test process.17:52
arxcruzodyssey4me: it won't17:53
arxcruzwe are starting to use os_tempest17:53
arxcruzas far as stackviz generate the data, in the right directory, we are collecting it17:53
odyssey4meok, so if I switch it up to use git instead, that would be fine?17:53
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-rabbitmq_server master: Add the ability to set the rabbitmq bind address  https://review.openstack.org/64437917:53
arxcruzodyssey4me: yeah17:53
arxcruzwe used tarball, because... i don't remember why17:53
arxcruzah, yeah, because the git comes with the whole node stuff17:54
arxcruzwe just wanted to not waste too much time on that17:54
arxcruzodyssey4me: oh, yeah, so, the tarball have the html already generated17:55
arxcruzthe git doesn't17:55
arxcruzif you use git, you need to generate all those node stuff to generate the html17:55
arxcruzthat's why we weren't using it17:55
odyssey4meoh dear :/17:55
arxcruzbefore, I had generate it, and added on tripleo side17:55
arxcruzodyssey4me: maybe we can ask to have in the tarball the requirements17:56
odyssey4meok, so y best option here is to just work out a way around this instead17:56
odyssey4me*my17:56
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: [DNM] Convert dynamic includes to static imports  https://review.openstack.org/64438017:56
odyssey4meok, I think I have an idea which is pretty simple and will allow us to just keep using it as-is.17:57
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-rabbitmq_server master: Add the ability to set the rabbitmq bind address  https://review.openstack.org/64437917:58
*** gyee has joined #openstack-ansible17:58
arxcruzodyssey4me: cool :)18:06
*** cshen has joined #openstack-ansible18:06
arxcruzrender the html takes like 10 minutes18:06
arxcruzwhich is too much for us :/18:06
*** luksky has quit IRC18:10
* mnaser thinks take things as distros suggest is the path to least resistance18:12
mnasertypically, distros ship pretty reliable and well integrated/tested things like mariadb/rabbit/etc18:12
mnaserso install from source should work regardless of distro, in the samy way18:13
odyssey4medoing something like that is easier to maintain from a code standpoint, but stifles innovation somewhat18:14
odyssey4mewe've managed to innovate quite a bit and do thing well before any distro has even tried in many cases, but perhaps that's all now tech debt and we'll have to change it18:14
mnaserodyssey4me: well this is only for the infrastructure components and i doubt we'd have much innovation out of rabbit/galera/etc.18:16
mnaserwe deploy openstack from source the same way across all environments, and when doing distro installs, we do them the way the distro wants it18:17
*** cshen has quit IRC18:18
*** BjoernT has joined #openstack-ansible18:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add toggle for python wheel build process  https://review.openstack.org/64439118:34
NobodyCamGood Morning OSA folks18:34
NobodyCamoff the wall question, does octavia require a flat network?18:35
*** BjoernT_ has joined #openstack-ansible18:36
NobodyCamI attempted to add as vlan and get `Invalid input for operation: Flat provider networks are disabled.` it is attempting to add as a flt18:36
*** BjoernT has quit IRC18:37
*** DanyC has quit IRC18:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Ensure stackviz installs from tarball URL only  https://review.openstack.org/63750318:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Update role for new source build process  https://review.openstack.org/62034018:43
*** ivve has joined #openstack-ansible18:45
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_tempest stable/rocky: Set tempest concurrency to int only  https://review.openstack.org/64378918:47
*** BjoernT_ has quit IRC19:00
*** dave-mccowan has joined #openstack-ansible19:05
*** chhagarw has quit IRC19:06
*** BjoernT has joined #openstack-ansible19:10
*** BjoernT has quit IRC19:11
*** BjoernT has joined #openstack-ansible19:14
*** bgmccollum has joined #openstack-ansible19:18
*** DanyC has joined #openstack-ansible19:53
*** DanyC_ has joined #openstack-ansible19:58
*** luksky has joined #openstack-ansible20:01
*** DanyC has quit IRC20:02
*** irclogbot_0 has joined #openstack-ansible20:12
*** cshen has joined #openstack-ansible20:15
*** BjoernT has quit IRC20:16
*** fresta_ has quit IRC20:17
*** cshen has quit IRC20:19
*** irclogbot_0 has quit IRC20:25
*** irclogbot_0 has joined #openstack-ansible20:27
*** hamzaachi has joined #openstack-ansible20:39
*** dave-mccowan has quit IRC20:39
*** BjoernT has joined #openstack-ansible20:40
*** cmart has quit IRC20:42
*** dave-mccowan has joined #openstack-ansible20:44
guilhermespodyssey4me nice, already unblocked some of my PRs https://review.openstack.org/#/c/643573/520:49
guilhermespand to unblock some others, once the above one is merged, I appreciate any review https://review.openstack.org/#/c/643789/20:50
*** pcaruana has quit IRC20:54
*** ansmith has quit IRC20:57
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible stable/rocky: Pass ceph_mons as a list of inventory hosts  https://review.openstack.org/64426520:58
*** irclogbot_0 has quit IRC21:05
*** irclogbot_0 has joined #openstack-ansible21:06
*** priteau has quit IRC21:12
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-rabbitmq_server master: Add the ability to set the rabbitmq bind address  https://review.openstack.org/64437921:25
*** irclogbot_0 has quit IRC21:26
*** irclogbot_0 has joined #openstack-ansible21:28
*** hamzaachi has quit IRC21:28
*** BjoernT has quit IRC21:30
*** cmart has joined #openstack-ansible21:49
*** cmart has quit IRC21:54
*** nurdie has quit IRC22:00
*** DanyC_ has quit IRC22:08
*** cshen has joined #openstack-ansible22:15
*** marst has quit IRC22:17
*** cshen has quit IRC22:19
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add OpenStack Zun support to OSA  https://review.openstack.org/57140022:25
*** cmart has joined #openstack-ansible22:30
*** fresta has joined #openstack-ansible23:05
openstackgerritMerged openstack/openstack-ansible stable/rocky: Always set pip vars for tempest install  https://review.openstack.org/64357323:06
*** ivve has quit IRC23:09
NobodyCamis there a trick to getting Osa to completely recreate a container? I made a mistake with the network config in openstack_user_config.yml I corrected that destroyed the container removed the it from the inventory, However when I rerun the playbooks the container is not getting a ip address23:23
cloudnullNobodyCam are any of the containers getting an IP address when you stop/restart them ?23:34
cloudnullI'd check to see if lxc-dnsmasq is running / restart the process/23:34
NobodyCamcloudnull: !!! Thank you !23:34
cloudnulli've seen that process get wedged on a host23:34
cloudnullresulting in a container not getting an address on eth0 and that blocks just about everything else23:35
NobodyCamI think I had a couple issues, main one being I was one space off in the user_config yaml23:41
cloudnullah that sucks23:41
NobodyCam:p23:42
NobodyCamtook me a couple tries to catch that one23:42
cloudnullthere have been aspirations in the past to build a general config validator, its come close but nothing concrete quite yet23:42
cloudnullsomething that would validate and catch little things like that could be super useful for folks23:42
* cloudnull IMHO23:42
nicolasbock+1 I agree23:43
cloudnullo/ nicolasbock23:43
nicolasbock\o :)23:43
NobodyCam:) we be super helpful if --check-syntact just printed out what was parsed.. Ie, network configs, type, range, ip from, etc...23:44
cloudnullone off errors are pretty common when building the config.23:44
cloudnullyea That would be a good thing to have23:44
*** tosky has quit IRC23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!