Tuesday, 2019-08-06

*** owalsh has quit IRC00:01
*** Wasaac has joined #openstack-kolla00:15
*** Wasaac has quit IRC00:19
*** trungnv has quit IRC00:48
*** igordc has quit IRC01:00
*** spsurya has joined #openstack-kolla01:17
*** kevinz has joined #openstack-kolla01:24
*** altlogbot_0 has quit IRC01:37
*** altlogbot_1 has joined #openstack-kolla01:38
*** kevinz has quit IRC01:53
*** weshay_dentist is now known as weshay02:13
*** gregoryo has joined #openstack-kolla02:25
*** BjoernT has joined #openstack-kolla02:41
*** kevinz has joined #openstack-kolla02:46
*** hongbin has joined #openstack-kolla02:49
*** kevinz has quit IRC02:55
*** BjoernT_ has joined #openstack-kolla02:58
*** BjoernT has quit IRC02:59
*** whoami-rajat has joined #openstack-kolla03:07
*** erijpkema has quit IRC03:08
*** hongbin has quit IRC04:11
*** BjoernT_ has quit IRC04:12
*** skramaja has joined #openstack-kolla04:28
*** janki has joined #openstack-kolla05:04
*** Luzi has joined #openstack-kolla05:05
Tenguhello there05:10
*** jchhatbar has joined #openstack-kolla05:38
*** jchhatbar has quit IRC05:41
*** janki has quit IRC05:41
*** jchhatbar has joined #openstack-kolla05:42
*** jchhatbar has quit IRC05:43
*** gkadam_ has joined #openstack-kolla05:54
*** gkadam_ has quit IRC05:54
*** gkadam has quit IRC05:56
*** cah_link has joined #openstack-kolla06:12
yoctozeptomorning06:19
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473206:21
*** kevinz has joined #openstack-kolla06:30
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473206:33
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473206:35
mnasiadkamorning07:08
*** jbadiapa has joined #openstack-kolla07:18
*** jbadiapa has quit IRC07:27
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473207:40
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473207:42
openstackgerritjacky06 proposed openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473207:44
*** rpittau|afk is now known as rpittau07:46
*** gregoryo has quit IRC07:50
*** dougsz has joined #openstack-kolla07:52
*** kevinz has quit IRC07:52
*** kevinz has joined #openstack-kolla07:52
*** JangwonLee has joined #openstack-kolla08:23
hrwmorning08:24
openstackgerritMerged openstack/kolla-cli master: Add reno for release notes management  https://review.opendev.org/67473208:32
openstackgerritcaoyuan proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67240608:45
*** rgogunskiy has joined #openstack-kolla08:52
*** Wasaac has joined #openstack-kolla08:57
*** gfidente has joined #openstack-kolla09:08
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Allow simple building of stable tarballs based source images  https://review.opendev.org/67460809:08
stingrayzao/09:08
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Allow simple building of stable tarballs based source images  https://review.opendev.org/67460809:09
*** owalsh_ is now known as owalsh09:10
*** dr_feelgood has joined #openstack-kolla09:11
*** shyamb has joined #openstack-kolla09:22
hrwmnasiadka: 'point'?09:56
mnasiadkahrw: in stable branches we use point releases, 7.0.0, 7.0.1, etc09:57
hrwmnasiadka: ok09:57
hrwmnasiadka: calling 'master' stable sounds tricky ;D09:57
mnasiadkahrw: I’ll run a name contest after fixing tests :)09:58
hrw;D09:59
hrwmnasiadka: and there was a tool to generate sources list for stable branches.10:00
mnasiadkahrw: yeah, I’ll look into that as well, just wanted to have openstack_release var for downloading upper-constraints.txt, but now I have some more ideas :)10:02
hrwsure10:02
hrwmnasiadka: I just wanted to get rid of /somewhere/systemd/something/kolla.conf and now I have to learn 'to_nice_json' ;D10:02
*** mgoddard has quit IRC10:03
hrwmnasiadka: commented10:04
yoctozeptohrw, mnasiadka: I just wanted to integrate ldap with keystone and now I'm fixing keystone10:04
*** mgoddard has joined #openstack-kolla10:07
*** dr_feelgood has quit IRC10:07
hrw~curse bigdata 'devs;10:09
*** dr_feelgood has joined #openstack-kolla10:09
yoctozeptobigdata? whoa, whoa, you shouldn't have rebuilt containers thousand times10:11
hrwyoctozepto: I am build engineer. kolla is one of my tasks10:11
*** shyamb has quit IRC10:19
*** dr_feelgood has quit IRC10:21
*** dr_feelgood has joined #openstack-kolla10:21
*** dr_feelgood has quit IRC10:25
hrwuf. my playbook is now able to inject registry into existing docker.json10:39
*** rgogunskiy has quit IRC10:45
*** dr_feelgood has joined #openstack-kolla10:49
*** shyamb has joined #openstack-kolla10:51
*** kevinz has quit IRC10:54
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477510:57
*** kevinz has joined #openstack-kolla11:08
hrwmnasiadka: does https://paste.fedoraproject.org/paste/90C0uivUEktN-1jDZoA9hA have sense?11:09
openstackgerritIsaac Prior proposed openstack/kolla master: Ensures monasca-thresh data dir exists and is empty on start up  https://review.opendev.org/67477911:12
*** shyamb has quit IRC11:12
*** shyam89 has joined #openstack-kolla11:12
*** dr_feelgood has quit IRC11:13
*** dr_feelgood has joined #openstack-kolla11:13
hrwlet CI check ;d11:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270011:13
*** dr_feelgood has quit IRC11:14
*** dr_feelgood has joined #openstack-kolla11:15
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Profile performance and task execution time  https://review.opendev.org/67466411:16
*** shyamb has joined #openstack-kolla11:17
*** shyam89 has quit IRC11:17
*** kplant has joined #openstack-kolla11:19
*** ivve has joined #openstack-kolla11:22
mnasiadkahrw: yeah, it's nice - question is what happens with MountFlags != shared :)11:31
hrwmnasiadka: read docker 18.09 release notes please. and https://review.opendev.org/67458711:31
mnasiadkahrw: I know the release notes, just haven't experienced them on myself :)11:32
hrwmnasiadka: my latest deployment has empty mountflags11:32
mnasiadkahrw: and neutron-openvswitch-agent works?11:32
hrwmnasiadka: not present11:32
mnasiadkahrw: I think neutron-ovs-agent had a problem without mountflags=shared, so we would need to test that11:33
hrwmnasiadka: I would say that this is a problem with those agents11:33
mnasiadkahrw: that's a bold statement that is equally true and nobody will give a shit about it :)11:34
hrw~curse yaml11:34
mnasiadkaI hate json more :)11:36
hrwjson ignores indents. yaml is not11:37
hrw- name: get configured insecure registries11:37
hrw  vars:11:37
hrw    registries: [{{docker_registry}}]11:37
hrwpep8 shouts at 3rd line11:38
kplantlanguages that don't ignore whitespace are annoying... cough python11:38
mnasiadkahrw: my json love comes from integration efforts with one ITSM tool, it vomitted huge json, from which I needed at most 5 variables :)11:39
hrwmnasiadka: ;D11:39
hrwmnasiadka: I did some JSON API hacking in past with some game.11:40
hrwmnasiadka: can you take a look at https://review.opendev.org/672700 and tell me how to fix post-install.yml syntax so our pep8 check with shut?11:41
*** Wasaac has quit IRC11:42
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add Debian/source CI job  https://review.opendev.org/67424011:43
*** Wasaac has joined #openstack-kolla11:43
mgoddardhrw: registries: ["{{docker_registry}}"]11:50
hrwmgoddard: thx11:50
mgoddardhrw: Jinja confuses YAML, you often need quots11:50
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270011:51
mnasiadkaseems mgoddard has a nice parser in his eyes :)11:52
mgoddardtoo many hours spent staring at ansible syntax erros11:52
yoctozeptomgoddard: hi, https://review.opendev.org/673165 waiting for you :-)11:53
yoctozeptowith mnasiadka decided to wait for you11:53
mgoddardyoctozepto: yeah, will do it shortly11:53
mgoddardthanks for waiting11:53
yoctozeptosure, I value quality, not quantity11:54
*** dr_feelgood has quit IRC12:01
*** zbr is now known as zbr|lunch12:01
hrwmgoddard: thanks for docker.io patch comments12:01
mgoddardhrw: do they make sense?12:02
*** dr_feelgood has joined #openstack-kolla12:02
hrwmgoddard: replied there - docker.io from debian/ubuntu gives us kind of 'docker lts' as we do not have to worry about any changes for a while12:02
mnasiadkadamn, my change will get bigger...12:03
mnasiadkamgoddard: thanks for comments, integrating that with version-check.py makes sense12:03
mgoddardmnasiadka: yeah, I think that's a key use case12:04
mgoddardhrw: I see. Have we seen problems with docker CE being unstable?12:04
hrwmgoddard: for most of time I avoid updating it.12:05
hrwmgoddard: https://review.opendev.org/674587 is fun12:05
hrwmgoddard: and related bug12:05
*** dr_feelgood has quit IRC12:07
*** shyamb has quit IRC12:10
*** henriqueof has joined #openstack-kolla12:10
*** shyamb has joined #openstack-kolla12:11
mgoddardheading out for a bit. yoctozepto I will look at your review when I'm back12:12
*** dr_feelgood has joined #openstack-kolla12:15
*** Wasaac has quit IRC12:16
*** Wasaac has joined #openstack-kolla12:17
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270012:24
hrwfixed another error12:25
hrw    "msg": "'dict object' has no attribute 'insecure-registries'"12:26
*** shyamb has quit IRC12:27
mgoddardhrw: why don't we see that mountflags issue on centos?12:32
mgoddarddo we see it on ubuntu when using docker CE (rather than docker.io)?12:32
hrwmgoddard: no idea.12:33
mgoddardlatest docker CE is 19.0312:33
mgoddardwhen did you see the issue?12:33
hrwmgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/183383512:33
openstackLaunchpad bug 1833835 in kolla-ansible "[debian] MountFlags=shared fail for docker 18.09" [Undecided,In progress] - Assigned to Marcin Juszkiewicz (hrw)12:33
hrwmgoddard: all my machines were on Debian stretch with 18.06 before I got that bug12:33
mgoddardusing docker CE or docker.io?12:34
hrwCE12:34
hrwstretch did not had docker at all12:34
*** hongbin has joined #openstack-kolla12:34
openstackgerritMerged openstack/kolla-ansible stable/stein: CI: fix checks for upgrade and multinode jobs  https://review.opendev.org/67398912:36
hrwand most of my machines are not supported in buster so upgrading them was tricky12:36
*** zbr|lunch is now known as zbr12:38
mgoddardyoctozepto: thanks for updating, approved the ceph patch12:49
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270012:54
hrwand another error ;D12:55
*** kevinz has quit IRC12:56
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062612:56
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:01
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Add Debian/source CI job  https://review.opendev.org/67424013:01
hrwmaking pep8 test happier13:02
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:02
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:05
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:09
*** dr_feelgood has quit IRC13:09
*** dr_feelgood has joined #openstack-kolla13:09
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:10
hrwkolla-cli became Chinese project13:11
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:11
openstackgerritIsaac Prior proposed openstack/kolla master: Ensures monasca-thresh data and worker dirs exist and are empty on start up  https://review.opendev.org/67477913:12
openstackgerritPierre Riteau proposed x/kayobe master: Remove storage management network from controllers  https://review.opendev.org/67248213:13
goldyfruit_hrw, more a 99cloud.net project13:13
*** Wasaac has quit IRC13:15
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477513:15
*** Wasaac_ has joined #openstack-kolla13:15
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270013:18
hrwshit. need to rebase13:19
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:21
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480613:22
*** hongbin has quit IRC13:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270013:24
hrwnow it sits on top of mountflags one13:25
hrwso you guys probably will ignore it ;D13:25
*** dr_feelgood has quit IRC13:39
*** dr_feelgood has joined #openstack-kolla13:40
*** BjoernT has joined #openstack-kolla13:40
*** mflynn has joined #openstack-kolla13:42
*** dr_feelgood has quit IRC13:42
*** dr_feelgood has joined #openstack-kolla13:43
*** BjoernT has quit IRC13:44
mflynnWhat combination of options are recommended for infrastructure monitoring? telegraf, prometheus, grafana?13:46
hrwGFW#$@#$@%^@#$^@$(^@$&^(@#$&%@13:46
openstackgerritIsaac Prior proposed openstack/kolla master: Ensures monasca-thresh data and worker dirs exist and are empty on start up  https://review.opendev.org/67477913:47
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270013:47
*** lemko has joined #openstack-kolla13:48
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477513:48
*** BjoernT has joined #openstack-kolla13:48
hrwmflynn: prometheus+grafana or monasca? but I never deployed any of them13:48
goldyfruit_mosnasca will be more to monitor instances, no ?13:50
kplant^ was just typing that13:50
*** dr_feelgood has quit IRC13:51
hrwI only know that our previous admin was prometheus+grafana fan13:51
goldyfruit_hrw, I'm!13:52
mflynnI was hoping someone might have a grafana dashboard already built with all kinds of pretty graphs.13:53
goldyfruit_mflynn, https://grafana.com/grafana/dashboards13:54
mflynngoldyfruit_, been looking at those. Found some nice ones.  thanks13:56
*** Luzi has quit IRC13:56
openstackgerritMerged openstack/kolla-ansible master: ceph: fixes to deployment and upgrade  https://review.opendev.org/67316513:59
openstackgerritMerged openstack/kolla-ansible master: Support mon and osd to be named with hostname  https://review.opendev.org/65441713:59
*** kevinz has joined #openstack-kolla14:00
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477514:06
openstackgerritgugug proposed openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480614:07
openstackgerritIsaac Prior proposed openstack/kolla master: Ensures monasca-thresh data and worker dirs exist and are empty on start up  https://review.opendev.org/67477914:07
*** JangwonLee has quit IRC14:08
mnasiadkayoctozepto: the monster merged :)14:09
*** dpawlik has quit IRC14:10
*** altlogbot_1 has quit IRC14:12
henriqueofI added a new service on globals.yml, should I use reconfigure or deploy option to enable it?14:13
*** altlogbot_3 has joined #openstack-kolla14:14
*** cah_link has quit IRC14:16
goldyfruit_henriqueof, deploy, you will have trouble to reconfigure something that doesn't exist :)14:16
henriqueofgoldyfruit_: hahah, thanks, will stop and use deploy.14:17
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Moves monasca-thresh java.io.tmpdir to existing docker volume  https://review.opendev.org/67477514:18
henriqueofgoldyfruit_: So deploy won't cause problems to my runnning services, right?14:19
goldyfruit_Nop, except if you change configuration manually14:20
goldyfruit_if that is the case then kolla, will override the changes and then the container(s) will be restarted14:20
henriqueofI changed some configs on Octavia, since kolla-ansible won't configure it (create networks and stuff), how can I override the kolla scripts so I don't need to add it manually again later?14:21
goldyfruit_Override the config into /etc/kolla/config/*14:23
goldyfruit_but you could use tag to deploy14:23
goldyfruit_kolla-ansible deploy -t barbican14:23
goldyfruit_which will just touch barbican14:23
henriqueofAh man, I used deploy just to add barbican, if I knew the tag feature before. hahaha14:24
henriqueofIs there any risks if I stop the deploy script now?14:24
goldyfruit_yeah, I would wait just in case14:24
*** kevinz has quit IRC14:26
yoctozeptomgoddard, mnasiadka: thanks, now onto stein14:27
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: ceph: fixes to deployment and upgrade  https://review.opendev.org/67444314:28
*** cah_link has joined #openstack-kolla14:28
yoctozepto^ this14:28
yoctozeptoand will need more of me for rocky so be done later14:28
*** KeithMnemonic has joined #openstack-kolla14:36
*** BjoernT has quit IRC14:42
openstackgerritMerged openstack/kolla-cli master: Update kolla command to kolla-cli action command  https://review.opendev.org/67480614:43
hrwlooks like docker.io change goes nicely on CI14:53
hrwyoctozepto: so ceph jobs are good now and upgrade-ceph ones need stein change?15:07
*** BjoernT has joined #openstack-kolla15:10
mgoddardyoctozepto: WDYT about this notifications patch? Do we really want to configure notifications for all services, even if we don't know that they produce them?15:13
*** skramaja has quit IRC15:14
*** BjoernT has quit IRC15:23
*** BjoernT has joined #openstack-kolla15:33
yoctozeptohrw: yep15:45
yoctozeptoupgrades left without fancy rebootstrapping, it is saner I think15:46
*** ivve has quit IRC15:46
yoctozeptomgoddard: mhm, that's the painful point of it, would be better to know, still this is something that can be orchestrated nicely because it is popular part of oslo15:46
yoctozeptoand Rafael seems to have only changed the ones that we already "tried" to configure15:47
mgoddardyoctozepto: I found the list of services ceilometer listens to15:47
mgoddardwe can restrict it to that for anything using enable_ceilometer15:47
yoctozeptomgoddard: sounds reasonable15:47
*** igordc has joined #openstack-kolla15:48
yoctozeptohave you linked it?15:48
yoctozeptok, refreshed, I see15:48
yoctozeptomaybe we should only orchestrate those ceilometer-relevant15:59
yoctozeptootherwise it could be confusing yeah15:59
yoctozeptothough the list is maybe not the best reference16:01
yoctozeptohttps://blueprints.launchpad.net/blazar/+spec/notifications16:01
yoctozeptoreads as if blazar has integrated the notifications16:01
yoctozeptothough definitely not all16:02
yoctozeptoargh, decisions, decisions :-)16:02
kdeanyoctozepto: Btw just wanted to say thanks for your help16:03
kdeanyoctozepto: I'm now up and running16:04
yoctozeptokdean: you are welcome, that's so nice to hear! :-)16:06
yoctozepto(or read in fact)16:06
yoctozeptomgoddard: what if we orchestrated notification_control_exchanges too?16:07
yoctozeptoseems logic to do this from kolla, huh?16:07
yoctozepto(I mean k-a ofc)16:07
mgoddardyoctozepto: so blazar is a good example of where a separate flag would be useful. ceilometer is clearly not interested, but users might be16:08
yoctozeptowell, ceilometer can be configure to listen16:09
yoctozeptowhatever that would mean though16:09
yoctozeptobut yeah, plainly configuring everything by enable_ceilometer without configuring ceilometer is actually pointless and confusing16:09
kdeanyoctozepto: :-)16:09
kdeanDo you know if there is anyway to get designate to use the internal URL for reverse lookup zones this is the only bit I have left to get configured, should I just turn the setting off in the neutron.conf as its currently causing this error.16:10
kdean Its due to HAproxy speaking HTTP on the internal address and neutron trys to connect to it using https16:10
mgoddardOh, I see notification_control_exchanges is the default, not allowed values16:10
kdeanNo details.: SSLError: SSL exception connecting to https://example.com:9001/zones?name=0.21.10.in-addr.arpa.: HTTPSConnectionPool(host='example.com', port=9001):16:10
kdeanMax retries exceeded with url: /zones?name=0.21.10.in-addr.arpa. (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')],)",),))16:10
yoctozeptoyes, what I meant exactyl :-)16:10
yoctozeptoI love my typos16:10
yoctozeptosounds like a name for an e-dinosaur16:10
mgoddardok. Well either way, I think we need to know which services actually produce notifications rather than just randomly adding variables16:11
yoctozeptokdean: I have little knowledge of designate16:11
kdeanno worries. thanks anyway16:12
*** igordc has quit IRC16:12
*** dougsz has quit IRC16:12
yoctozeptomgoddard: k, left note with our thoughts on change16:14
*** owalsh_ has joined #openstack-kolla16:17
*** owalsh has quit IRC16:22
*** hongbin has joined #openstack-kolla16:24
*** dougsz has joined #openstack-kolla16:25
*** hongbin has quit IRC16:29
*** owalsh_ has quit IRC16:31
*** owalsh has joined #openstack-kolla16:31
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062616:33
openstackgerritPierre Riteau proposed openstack/kolla-ansible master: Run task only if heat-engine is enabled on this host  https://review.opendev.org/67486716:37
*** gfidente has quit IRC16:38
*** rpittau is now known as rpittau|afk16:41
yoctozeptomgoddard: argh, Rafael must be very confused as we agree only partially ;D16:44
mgoddardyoctozepto: we as in you + me?16:45
yoctozeptoyeah16:45
yoctozeptosee my response16:45
yoctozeptodid not see yours before posting16:45
mgoddardI'm trying to reduce the scope of the change. It's already too large16:46
mgoddardfactor out the less well understood parts into another patch16:46
mgoddardmake sense?16:46
*** BjoernT has quit IRC16:48
yoctozeptomakes16:50
yoctozeptobut better put it16:51
yoctozepto"what to do"16:51
yoctozeptoin this particular change16:51
yoctozeptorather that what not to ;p16:51
openstackgerritMerged openstack/kolla-ansible master: Add mon address to ceph release version check  https://review.opendev.org/66926917:04
mgoddardyoctozepto: except I want some of the current change to be removed17:08
*** BjoernT has joined #openstack-kolla17:08
yoctozeptomgoddard: yes, yes, but it is more understandable when you say what you want the scope of the current change to be rather what not17:09
yoctozeptorather than*17:09
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Add support for using custom Logstash patterns  https://review.opendev.org/67488417:18
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Profile performance and task execution time  https://review.opendev.org/67466417:27
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove support for Docker legacy packages  https://review.opendev.org/67488617:34
*** dcapone2004 has joined #openstack-kolla17:41
dcapone2004Is it possible to do an AIO install for evaluation purposes using kolla ansible and a virtual environment?  I am getting some errors and I am not finding much help online17:42
kplantdcapone2004: absolutely17:42
kplantwhat errors are you getting and at what stage?17:43
dcapone2004prechecks it is complaining about importerror no module named docker found17:43
dcapone2004I googled that and came across this...17:43
kplantright, okay17:44
kplantso before you can run prechecks, you need a prepared environment17:44
dcapone2004https://ask.openstack.org/en/question/101298/docker-module-importerror-with-kolla-ansible/   but it seemed to imply that using a virtual environment wasn't recommended which is contrary to the kolla ansible quickstart which does recommend it17:44
kplantif you run: kolla-ansible -i all-in-one bootstrap-servers17:44
kplantthat will pop docker on there for you17:44
dcapone2004that runs perfect no error17:44
dcapone2004run prechecks it fails17:45
*** dougsz has quit IRC17:45
kplantyou're using the provided 'all-in-one' ?17:45
yoctozeptosounds like virtualenv is not picked up17:45
yoctozeptowhat release of kolla-ansible is this?17:45
dcapone2004yep, I don't think I made any changes whatsoever to the all-in-one file....I customized globals.yml though17:45
yoctozeptoprobably worth reporting as bug17:46
dcapone2004for stein...I am running kolla-ansible from within the virtual environment is that correct?17:46
kplantpip or git?17:46
dcapone2004pip17:46
dcapone2004end of bootstrap output:17:47
dcapone2004PLAY RECAP **************************************************************************************************************************************************************************************17:47
dcapone2004localhost                  : ok=37   changed=1    unreachable=0    failed=0    skipped=24   rescued=0    ignored=017:47
kplantright so, regarding venvs, it gets a little hairy17:47
kplantyou can use venvs on the deploying host and/or on the target hosts17:47
kplantbut you're using aio.... heh17:47
dcapone2004and the docker tasks seem to run successfully17:47
dcapone2004TASK [baremetal : Add kolla user to docker group] ***********************************************************************************************************************************************17:47
dcapone2004ok: [localhost]17:47
dcapone2004TASK [baremetal : Start docker] *****************************************************************************************************************************************************************17:47
dcapone2004ok: [localhost]17:47
dcapone2004TASK [baremetal : Restart docker] ***************************************************************************************************************************************************************17:47
dcapone2004skipping: [localhost]17:47
dcapone2004TASK [baremetal : Enable docker] ****************************************************************************************************************************************************************17:47
dcapone2004ok: [localhost]17:47
kplantif you run `docker ps` do you get output?17:47
dcapone2004kplant: yeah I thought the virtualenv issue would be on multimode deployments not aio17:48
dcapone2004from within the virtualenv or outside of it?17:48
kplantoutside17:48
kplantthat's where ansible is going to run, unless specified otherwise in inventory or globals.yml17:48
dcapone2004get output from both actually17:49
dcapone2004(virtualaio) [root@dev-lx-os-stein-aio bin]# docker ps17:49
dcapone2004CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES17:49
dcapone2004(virtualaio) [root@dev-lx-os-stein-aio bin]# deactivate17:49
dcapone2004[root@dev-lx-os-stein-aio bin]# docker ps17:49
dcapone2004CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES17:49
dcapone2004[root@dev-lx-os-stein-aio bin]#17:49
dcapone2004oh crap, should I not be running this as root and running as the user kolla instead?17:49
kplantroot is fine17:50
kplantfor a poc aio anyway :-)17:50
dcapone2004lol17:50
kplantwhat distro is this17:50
dcapone2004in what config file does the kolla user get set?   CentOS 717:51
dcapone2004using centos for the container distro17:51
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: ceph: fixes to deployment and upgrade  https://review.opendev.org/67488917:51
kplantand you're running `kolla-ansible -i all-in-one prechecks` ?17:52
dcapone2004correct17:52
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Add mon address to ceph release version check  https://review.opendev.org/67489117:52
*** spsurya has quit IRC17:52
dcapone2004from within the virtual environment....is that correct?17:52
kplantif you're using venv on the deploying host yes, otherwise kolla-ansible won't exist17:53
dcapone2004very true17:53
dcapone2004:-)17:53
kplantcan you try without a venv?17:53
kplantthis is just a vm right? you don't care about it?17:53
dcapone2004it is bare metal but I don't care about it as it is a complete dev machine so I can whack and reload easily17:54
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67447917:54
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67447917:54
kplantif there's nothing else on here i would try it without a venv17:55
dcapone2004that was my planned next step17:55
kplantsomething sounds... screwy17:55
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67330217:55
kplanti use venvs in production (on both build and target hosts) without trouble17:55
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67330217:55
dcapone2004centos or ubuntu?  Also, I configured it to use binary vs source but I don't think that would make any difference at this stage of the game17:56
kplantcentos from source17:57
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Sanity check that nodepool.private_ipv4 is assigned  https://review.opendev.org/67489818:04
*** igordc has joined #openstack-kolla18:08
dcapone2004so from a documentation standpoint when not using a virtual environment.....you need to install kolla-ansible using pip BEFORE installing ansible as yum installs PyYAML with ansible and it creates a dependency issue when using pip18:18
dcapone2004that is probably not an issue for multimode though18:18
kplantyou can just pip install ansible --ignore-installed18:19
dcapone2004pip install kolla-ansible is where the error is generated if ansible is already installed with yum18:19
kplantsorry18:21
kplantmeant pip install kolla-ansible --ignore-installed18:21
kplantbut the documentation calls out installing ansible from pypi iirc18:22
kplantso it doesn't expect that18:22
dcapone2004docs say use yum to install ansible and then pip for kolla-ansible18:23
kplantaah it does, for venv it gets it right18:23
kplantbut without venv it says to use yum18:24
dcapone2004correct18:24
dcapone2004that is why I was just mentioning it from a documentation standpoint here18:24
kplantyeah that should probably be a pip install ansible OR pip install kolla-ansible --ignore-installed18:24
kplantthe former would probably be better18:24
*** igordc has quit IRC18:25
*** henriqueof has quit IRC18:38
*** igordc has joined #openstack-kolla18:46
*** jbadiapa has joined #openstack-kolla18:47
*** Wasaac_ has quit IRC18:55
dcapone2004alrightly, so install completed but neutron did not appear to come up....I'm coming from an RDO/packstack world and I would usually be checking neutron processes using systemctl and config files for neutron, but since kolla uses docker containers I'm guessing I would need to enter the correct container and make these similar checks?18:57
dcapone2004looking at docker, it appears like the neutron-server:stein container just keeps restarting on its own18:59
*** jbadiapa has quit IRC19:03
dcapone2004trying to redeploy from source and see what happens19:06
kplant`docker logs [containername]`19:14
kplantthat will give you a nice output of what's going on19:14
*** jbadiapa has joined #openstack-kolla19:16
*** shaan9 has joined #openstack-kolla19:25
*** jbadiapa has quit IRC19:31
*** ivve has joined #openstack-kolla19:32
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062619:35
*** atmark has joined #openstack-kolla19:41
*** henriqueof has joined #openstack-kolla19:42
*** Wasaac has joined #openstack-kolla19:45
*** Wasaac has quit IRC19:49
dcapone2004well, the source didn't help and nor did the logs really give much info19:50
dcapone2004and I can't get into the container fast enough to look into much more detail before it restarts itself19:50
dcapone2004ok, hitting enter just at the right time to get me the neutron-server.log file has shed some light....the issue is with Fwaas....it cannot find the plugin inside of the container....is Fwaas not supported via Kolla Ansible?19:59
dcapone2004this is what is getting put in the neutron-server container....20:02
dcapone20042019-08-06 15:56:09.234 6 INFO neutron.manager [req-ca8c9369-22bb-48b0-bbcf-1680810e735b - - - - -] Loading Plugin: firewall20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime [req-ca8c9369-22bb-48b0-bbcf-1680810e735b - - - - -] Error loading class by alias: NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall'20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime Traceback (most recent call last):20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/utils/runtime.py", line 114, in load_class_by_alias_or_classname20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime     namespace, name, warn_on_missing_entrypoint=False)20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/stevedore/driver.py", line 61, in __init__20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime     warn_on_missing_entrypoint=warn_on_missing_entrypoint20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/stevedore/named.py", line 89, in __init__20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime     self._init_plugins(extensions)20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/stevedore/driver.py", line 113, in _init_plugins20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime     (self.namespace, name))20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime NoMatches: No 'neutron.service_plugins' driver found, looking for 'firewall'20:02
dcapone20042019-08-06 15:56:09.236 6 ERROR neutron_lib.utils.runtime20:02
dcapone20042019-08-06 15:56:09.237 6 ERROR neutron_lib.utils.runtime [req-ca8c9369-22bb-48b0-bbcf-1680810e735b - - - - -] Error loading class by class name: ValueError: Empty module name20:02
dcapone20042019-08-06 15:56:09.237 6 ERROR neutron_lib.utils.runtime Traceback (most recent call last):20:02
dcapone20042019-08-06 15:56:09.237 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/utils/runtime.py", line 120, in load_class_by_alias_or_classname20:02
dcapone20042019-08-06 15:56:09.237 6 ERROR neutron_lib.utils.runtime     class_to_load = importutils.import_class(name)20:02
dcapone20042019-08-06 15:56:09.237 6 ERROR neutron_lib.utils.runtime   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class20:02
openstackgerritMerged openstack/kolla-ansible stable/stein: ceph: fixes to deployment and upgrade  https://review.opendev.org/67444320:11
kplanti would recommend sharing large output like that with something like paste.openstack.org :-)20:13
kplantlooks like you enabled neutron fwaas?20:14
kplantoh, you said that. i only read your output and not messages earlier20:14
kplanti have not personally tried fwaas20:15
dcapone2004redeploying without it now20:17
dcapone2004is there a way to have kolla-ansible only redeploy what changed?  using deploy again appears to be rerunning and rebuilding everything which is different than ceph-ansible20:18
dcapone2004nevermind, looks like it only redeployed the neutron containers20:22
*** factor has joined #openstack-kolla20:22
*** Wasaac has joined #openstack-kolla20:25
*** lemko has quit IRC20:27
*** Wasaac has quit IRC20:29
*** priteau has joined #openstack-kolla20:51
*** mflynn has quit IRC20:55
*** whoami-rajat has quit IRC20:56
*** strigazi has joined #openstack-kolla21:01
*** kplant has quit IRC21:02
*** Wasaac has joined #openstack-kolla21:03
dcapone2004in which container is br-ex defined?21:04
*** Wasaac has quit IRC21:07
*** igordc has quit IRC21:12
dcapone2004I am having issues with the horizon interface....a lot of things are not working quite right....when creating a network, I am unable to select anything in any downdrops, clicking them closes the dropdown but does not change the value, I have tried edge and chrome, about to try firefox21:17
dcapone2004has anyone else seen anything like this?21:17
dcapone2004"tabs" also do not appear to function properly, if I create networks and subnets via the command line, I can see the networks in horizon, but if I select a network and then click "subnets", I am left with a blank screen21:18
*** igordc has joined #openstack-kolla21:31
*** henriqueof has quit IRC21:45
*** Wasaac has joined #openstack-kolla21:50
*** Wasaac has quit IRC21:55
dcapone2004enabled debugging in the horizon config and there appears to be some permissions issues....where does this project maintain bug reports?22:07
priteaudcapone2004: https://bugs.launchpad.net/kolla-ansible22:11
priteauI've never seen the issues you're mentioning though22:11
dcapone2004with offline compression enabled (default from the kolla install) inside the container22:12
dcapone2004I get this22:12
dcapone2004OfflineGenerationError at /project/22:12
dcapone2004You have offline compression enabled but key "3dcdc14a426856978c36be7fea160fa32caa3b3167f519795a0c3e31bbe119f8" is missing from offline manifest. You may need to run "python manage.py compress". Here is the original content:22:12
dcapone2004disable offline compression, I am getting:22:13
dcapone2004OSError at /project/server_groups/22:13
dcapone2004[Errno 13] Permission denied: '/var/lib/kolla/venv/lib/python2.7/site-packages/static/dashboard/js/1ba732d085e1.js'22:13
dcapone2004Request Method:GET22:13
dcapone2004this existed as an issue at some point in the past as well in horizon https://bugs.launchpad.net/horizon/+bug/115576522:19
openstackLaunchpad bug 1155765 in OpenStack Dashboard (Horizon) "Offline compression enabled but key is missing from offline manifest" [Medium,Fix released] - Assigned to Yves-Gwenael Bourhis (yves-gwenael-bourhis)22:19
priteauI've seen this kind of issue with static assets a while ago but never with kolla. You could try deleting the horizon container and redeploying.22:21
priteauSorry I don't have a better answer. If you can, try asking during European daytime, there will be more people in this channel.22:21
*** priteau has quit IRC22:21
*** dcapone2004 has quit IRC22:23
*** cah_link has quit IRC22:35
*** shaan9 has quit IRC22:42
*** henriqueof has joined #openstack-kolla22:43
*** igordc has quit IRC22:56
*** igordc has joined #openstack-kolla23:20
henriqueofI enabled neutron_vpnaas but there is no neutron_vpnaas_agent container and all the vpn services keeps on pending create, is this right?23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!