Tuesday, 2018-09-18

*** gyee has quit IRC00:04
*** spatel has joined #openstack-ansible00:04
*** spatel has quit IRC00:09
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client master: Find the list of packages to remove using rpm  https://review.openstack.org/60320500:13
openstackgerritcaoyuan proposed openstack/openstack-ansible master: fix tox python3 overrides  https://review.openstack.org/57431300:23
*** dxiri has joined #openstack-ansible00:43
*** markvoelker has quit IRC00:44
*** cshen has joined #openstack-ansible00:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/queens: Clean up role tests  https://review.openstack.org/60186400:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/queens: import zuul job settings from project-config  https://review.openstack.org/59512300:47
*** cshen has quit IRC00:49
*** mschuppert has quit IRC01:37
*** dave-mccowan has quit IRC01:42
*** dave-mccowan has joined #openstack-ansible02:01
*** dave-mccowan has quit IRC02:20
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add capability to set node role  https://review.openstack.org/60327002:22
openstackgerritjacky06 proposed openstack/openstack-ansible-os_horizon master: Add watcher dashboard into horizon  https://review.openstack.org/60315602:27
*** dxiri has quit IRC02:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client master: Find the list of packages to remove using rpm  https://review.openstack.org/60320502:44
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add capability to set node role  https://review.openstack.org/60327003:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Revisit conflicting package removal  https://review.openstack.org/60321503:01
*** lbragstad has quit IRC03:04
*** lbragstad has joined #openstack-ansible03:11
openstackgerritMerged openstack/openstack-ansible-os_octavia stable/rocky: Update branch to point to Rocky  https://review.openstack.org/59980303:20
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add capability to set node role  https://review.openstack.org/60327003:48
*** lbragstad has quit IRC03:48
*** udesale has joined #openstack-ansible03:53
*** hw_wutianwei_ has quit IRC04:07
*** shyamb has joined #openstack-ansible05:02
*** shyamb has quit IRC05:14
*** shyamb has joined #openstack-ansible05:20
*** shyamb has quit IRC05:36
*** shyamb has joined #openstack-ansible05:38
*** DanyC has joined #openstack-ansible05:41
*** DanyC has quit IRC05:47
*** shyamb has quit IRC06:07
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.openstack.org/60328506:14
*** rgogunskiy has joined #openstack-ansible06:20
*** cshen has joined #openstack-ansible06:26
*** shyamb has joined #openstack-ansible06:38
*** DanyC has joined #openstack-ansible06:43
*** DanyC has quit IRC06:47
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support  https://review.openstack.org/60310006:48
*** olivierb has joined #openstack-ansible07:08
*** ekan has joined #openstack-ansible07:10
*** fghaas has joined #openstack-ansible07:18
*** shyamb has quit IRC07:22
mbuil_good morning! I need a bit of help with the gate problems ==> https://review.openstack.org/#/c/603165/, I am not sure why it cannot find the ceilometer dependency if it is part of https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L107:23
ekanHi,  Currently sitting with a problem regarding host aggregates / AZ.  I have a environment where I want the majority of compute nodes in different a host aggregate than the one used with AZ `nova` and leave a few computes as default.  As I keep all configuration regarding openstack-ansible config in git (user_variables openstack_user_config), my question is really if it's supported to handle this within the installation with07:24
ekan#2: penstack_user_config or similar). The alternative would be to handle this in a seperate playbook (with potentially seperate vars) after everything is up and running.  Thankful for any pointers to documentation or tips.07:29
*** gkadam has joined #openstack-ansible07:39
*** priteau has joined #openstack-ansible07:41
*** hamzaachi has joined #openstack-ansible07:45
hwoarangmbuil_: shouldn't ceilometer be added in vars/main.yml in os_tempest since this is the part that fails in that log?07:49
hwoarangoh wait07:50
hwoarangor in the tests/ overrides files in the os_neutron role?07:50
mbuil_hwoarang: it seems the dependency is defined but can't find the version that dependency sets "Could not satisfy constraints for 'ceilometer': installation from path or url cannot be constrained to a version" or?07:51
hwoarangmbuil_: i am just looking at the log and it seems to fail on the tempest plugins installation and then lookded at the role and ceilometer is not defined in tempest plugins07:53
mbuil_hwoarang: where are you looking in tempest pluging? os_tempest/defaults/main.yml?07:53
hwoarangos_tempest/vars/main is the defaults07:54
hwoarangbut if you look at os_neutron/tests/ there are a bunch of os_neutron-overrides* files07:55
hwoarangfor each scenario07:55
hwoarangwhere they define their own tempest plugins07:55
mbuil_hwoarang: but this test has nothing to do with os_neutron07:55
mbuil_hwoarang: need to leave 20 minutes, sorry! brb07:56
hwoaranghmm true. and this message is giving a clue 'Could not satisfy constraints for 'ceilometer': installation from path or url cannot be constrained to a version' but needs more investigation because i am not very familiar with this piece of code07:58
jrosserekan: i have a completely seperate playbook / vars for setting host aggregates. it uses the same inventory as OSA though so it's quite simple07:59
*** persia has quit IRC08:06
ekanjrosser: Ok, fair enough. Seems to be quite simple if using same inventory. Thank you08:06
*** persia has joined #openstack-ansible08:07
jrosserekan: http://paste.openstack.org/show/730236/08:09
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support  https://review.openstack.org/60310008:10
ekanjrosser: Great, I'll have a look. Thanks!08:11
*** tosky has joined #openstack-ansible08:16
*** luksky has joined #openstack-ansible08:19
*** DanyC has joined #openstack-ansible08:27
gokhanihi team, is there anyone who use vlan tenant networks ? When I use vlan tenant networks, all l3 agents are master and it causes network connection problem. But when I use vxlan tenant networks, it is working properly and one L3 agent is master and others are BACKUP.  What is your thoughts about this problem ? ? my environment is on openstack pike 16.0.13 on ubuntu 16.04.508:33
mbuil_hwoarang: back :)08:33
*** markvoelker has joined #openstack-ansible08:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Remove python-keystoneclient package install  https://review.openstack.org/58772408:36
*** shyamb has joined #openstack-ansible08:45
*** lbragstad has joined #openstack-ansible08:59
mbuil_hwoarang: there is no ceilometer-tempest-plugin :/08:59
mbuil_you are right08:59
odyssey4mecores, I'd appreciate some reviews for the patches I have backing up: https://goo.gl/k9cCfF09:00
hwoarangmbuil_: i dont know what to tell you really. maybe something in upper-constrains that we build forces ceilometer to a version and according to the error message you can't use specific version if you install from git or path09:01
odyssey4mehopefully https://review.openstack.org/602927 will pass this time around to unblock the integrated gate09:01
hwoarangbut i am not familiar with this part of the code tbh09:01
hwoarangodyssey4me: this centos thing looks a bit fragile :/09:02
odyssey4membuil_ hwoarang yeah, I saw those yesterday and it's a bit odd - haven't had much time to figure it out09:02
odyssey4mehwoarang: yeah, it sucks - however the good news is that we're removing the use of that role almost everywhere09:03
odyssey4meI am finding myself tempted to switch to using the RDO packages (mariadb 10.2) for the distro installation on CentOS given that RDO has their deps very well tuned.09:04
chkumar|ruckmbuil_: the all telemetry projects has a singel tempest plugin -> telemetry-tempest-plugin09:04
mbuil_chkumar|ruck: ok, thanks. That one is part of vars/main.yml, any idea why all gates fail with the same dependency problem? ==> https://review.openstack.org/#/c/603165/09:06
odyssey4membuil_: ok, let's take a step back a bit here09:08
odyssey4membuil_: why does networking-odl require ceilometer? I don't understand that requirement at all.09:09
odyssey4meSurely it should be an extra or something, or is it intrinsically required for the basic functioning of the driver?09:10
mbuil_chkumar|ruck --> perhaps I should add the telemetry-tempest-plugin here... https://github.com/openstack/openstack-ansible-os_ceilometer/blob/master/tests/os_ceilometer-overrides.yml#L48-L5109:10
*** markvoelker has quit IRC09:11
chkumar|ruckmbuil_: yes this one needs to be updated09:11
chkumar|ruckmbuil_: and the test name also09:11
mbuil_odyssey4me: your question makes a lot of sense. In parallel we are working on defining it as extra. It is not required for the basic functioning of the driver: https://review.openstack.org/#/c/602622/09:12
mbuil_chkumar|ruck: name should be tempest-telemetry?09:13
odyssey4membuil_: oh fantastic, thanks - because adding it into requirements for neutron is going to break the integrated build unless we modify some groups09:13
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307809:13
chkumar|ruckmbuil_: the name of the plugin is telemetry-tempest-plugin09:13
odyssey4membuil_: https://github.com/openstack/openstack-ansible-os_aodh/blob/master/tests/os_aodh-overrides.yml#L27-L3009:13
odyssey4menot sure what an appropriate test whitelist would be09:14
chkumar|ruckodyssey4me: mbuil_ list of telemetry tests http://paste.openstack.org/show/730240/09:17
*** idlemind has quit IRC09:18
openstackgerritMerged openstack/openstack-ansible-os_trove stable/pike: Remove mysql-python  https://review.openstack.org/60295909:19
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_ceilometer master: Bug fix: tempest plugin is not well defined  https://review.openstack.org/60333709:19
mbuil_chkumar|ruck: ok, but according to that output ceilometer.tests.tempest.api.test_telemetry_notification_api does not exist, right? In fact, there is no ceilometer test09:21
chkumar|ruckmbuil_: let me check the history09:22
odyssey4membuil_: yeah, there will need to be a different test set exercised09:23
odyssey4membuil_: actually, it looks like there're no ceilometer tests - it's not an API any more, so that kinda makes sense09:24
odyssey4meit may make more sense to remove tempest from there and do a simple assertion task to validate that the poller is running or something like that09:24
*** aludwar has quit IRC09:28
*** aludwar has joined #openstack-ansible09:28
mbuil_odyssey4me: I feel like building a domino row of patches. I already have 6 or 7 waiting for the next one to be finished in order to merge. Do you have an example of assertion task in another role? Maybe I can use it as base for ceiloemter tests09:30
mbuil_odyssey4me: I have just noticed the comment on top of this https://github.com/openstack/openstack-ansible-os_ceilometer/blob/master/tests/os_ceilometer-overrides.yml#L44. If tempest_run is "no", it should not run the tempest test cases or?09:33
*** shardy has joined #openstack-ansible09:34
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307809:35
*** sum12 has quit IRC09:35
odyssey4membuil_: yeah, but it still installs everything and the fail is in the install, not the execution of tempest itself09:40
chkumar|ruckmbuil_: you can simply replace the test is TelemetryAlarming instead of ceilometer09:40
odyssey4melemme put a quick patch to address this up09:41
chkumar|ruck*with09:41
openstackgerritMerged openstack/openstack-ansible-nspawn_container_create stable/rocky: Cleanup mounts in containers  https://review.openstack.org/60286809:43
*** shardy has quit IRC09:46
*** shyamb has quit IRC09:46
*** shyamb has joined #openstack-ansible09:47
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Switch to simple service test instead of tempest  https://review.openstack.org/60334609:51
mbuil_odyssey4me: thanks ^. Let's wait for the tests09:55
odyssey4membuil_: yeah, the right thing to do would be to do panko as suggested in the override file, but I don't know how any of that stuff works - I guess if anyone is interested, then they can pick it up later09:55
openstackgerritMerged openstack/openstack-ansible-os_trove stable/rocky: Remove mysql-python  https://review.openstack.org/60295709:58
*** Emine has joined #openstack-ansible09:58
openstackgerritMerged openstack/openstack-ansible-os_panko stable/rocky: Remove pip_install meta dependency  https://review.openstack.org/60307609:59
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/pike: Define cache_timeout variable  https://review.openstack.org/60296410:00
*** gkadam has quit IRC10:02
*** gkadam has joined #openstack-ansible10:04
*** markvoelker has joined #openstack-ansible10:07
*** FuzzyFerric has joined #openstack-ansible10:09
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307810:14
* hwoarang kicks apparmor10:16
FuzzyFerricHi. I am installing Openstack Queens using Ansible in a HA setup with SSL and have nearly completed the list, I am having problems with Glance using Cinder as the backend. Tempest installs ok, downloads the Cirros issue, but then fails to upload it to the Glance service, reporting an "Internal Error" and failing on retries. I know the Cinder service is working as I can create Volumes without issue. Looking at the Glance contai10:17
FuzzyFerricner in the Glance-api log I can see an error relating to "Unable to Establish a connection" (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))) - I am at a bit of a loss here as I cannot find any config which could override this, or what SSL connection it is referring to as it uses the same Keystone and certificates as all other services whi10:17
FuzzyFerricch appear to be running correctly.. Is there any additional config location I can look into, or is Glance with a Cinder back end not working?.. Thanks10:17
ArchiFleKsevrardjp: So i'm not sure about what happened but it was unrelated to the error message I posted yesterday (it was just cosmetic). The problem was that after the upgrade the neutron DB tables lbaasagentbinding was empty so all my existing haproxy LBAAS were agentless and since the migration from container to bare metal they were not recreated10:20
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307810:20
*** gkadam has quit IRC10:21
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/queens: Define cache_timeout variable  https://review.openstack.org/60296510:21
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/rocky: Define cache_timeout variable  https://review.openstack.org/60296210:21
ArchiFleKsI don't know if this happened when we removed the old agent but I think it was the case, I though it would behave juste like the l3-agent migration10:21
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/pike: Define cache_timeout variable  https://review.openstack.org/60296610:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Switch to simple service test instead of tempest  https://review.openstack.org/60334610:21
ArchiFleKsI inserted the binding manually in the agentbinding table et was able to recover et the old LB got recreated after an lbaas-agent restart10:21
*** shyamb has quit IRC10:26
odyssey4mejroll: mind taking a peek at https://review.openstack.org/602927 ? with that approved and merged, the integrated gate will be unblocked10:31
odyssey4meoops, I meant jrosser10:31
jrosserdone10:32
odyssey4metyvm10:32
odyssey4meFuzzyFerric: are you using ssl for internal/admin endpoints, or only for public endpoints? it seems like the services are trying to use the public endpoint for some reason, and are failing because the ssl cert is likely self-signed or something10:33
jrosserstill looking for review on this https://review.openstack.org/#/c/601534/ hwoarang or evrardjp....10:34
evrardjpI have a quite busy day, I will have trouble finding time today. Will prep the bug triage and all, but that will be hard for me to do something else10:35
evrardjpPTG made me late on my other duties10:35
hwoarangjrosser: lgtm10:38
FuzzyFerricodyssey4me: I am using SSL only for public endpoints which I can verify, the internal and admin endpoints are all http://. You are correct that the certificate is a self signed one, but should the glance server not be using the internal or admin interfaces? Is there somewhere in then ansible setup i can tell it to do this?..10:38
odyssey4meFuzzyFerric: it should do it by default, so something has gone wrong10:39
*** markvoelker has quit IRC10:41
FuzzyFerricodyssey4me: Running all the playbooks appears to work fine, until this one at the end which is the only error I am having. I have been trying to complete this by rerunning the os-tempest-install.yml individually10:41
odyssey4meFuzzyFerric: yeah, that should be fine - which branch/tag are you using?10:42
*** udesale has quit IRC10:47
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support  https://review.openstack.org/60310010:47
openstackgerritMerged openstack/openstack-ansible stable/rocky: Bump requirement SHA  https://review.openstack.org/60323310:48
FuzzyFerricodyssey4me: Not sure, I did not initiate the install on this one, is there a way I can find out?10:48
odyssey4meFuzzyFerric: cat /etc/openstack-release on any of the hosts, or check that the git branch/tag is10:49
odyssey4me(in /opt/openstack-ansible on the deploy host)10:50
FuzzyFerricDISTRIB_ID="OSA"10:50
FuzzyFerricDISTRIB_RELEASE="17.1.1"10:50
FuzzyFerricDISTRIB_CODENAME="Queens"10:50
FuzzyFerricDISTRIB_DESCRIPTION="OpenStack-Ansible"10:50
odyssey4meok, so sounds like it's the head of the stable/queens branch as that's the next planned release version10:50
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Define cache_timeout variable  https://review.openstack.org/60296110:51
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/queens: Define cache_timeout variable  https://review.openstack.org/60296310:51
FuzzyFerricodessy4me: this is running on Ubuntu 16.04.5 LTS10:52
odyssey4meFuzzyFerric: what's the first SHA shown when you do 'git log' in /opt/openstack-ansible10:52
odyssey4meSHA or title10:52
FuzzyFerricMerge "Revert "Revert "Work around nested virt issues on OVH test nodes""" into stable/queens10:55
FuzzyFerriccommit 0ceb08dee21f00625a1bad0a9cf5c8d8f722647210:55
odyssey4meFuzzyFerric: ok, it might be good to update that checkout to pull in https://github.com/openstack/openstack-ansible/commit/9435dc9d0eff1094ccc187d7fb392cb015c2169710:56
*** shyamb has joined #openstack-ansible10:56
odyssey4megit fetch --all; git reset --hard origin/stable/queens10:57
FuzzyFerricDone, HEAD is now at 9435dc9 Bump plugins SHA to include recent bugfixes10:57
odyssey4meok, however that doesn't explain the issue you're seeing10:57
*** gkadam has joined #openstack-ansible10:57
odyssey4mewhat overrides have you implemented in user_*.yml ?10:58
odyssey4mecould you share them in pastebin/gist ?10:58
odyssey4mewell, leave out your secrets of course10:58
openstackgerritMerged openstack/openstack-ansible-galera_client master: Find the list of packages to remove using rpm  https://review.openstack.org/60320511:00
FuzzyFerrichttps://gist.github.com/suggestable/94ac20e8944658fd0de1bb29ca82798e11:01
odyssey4meFuzzyFerric: it looks like you have conflicting configs there - haproxy_ssl is off, but the public protocol is ssl - what's up with that?11:02
FuzzyFerricodyssey4me:  Apologies, That was a previous one and it looks like my live config has already been updated. Checking the file on the live server shows that haproxy_ssl is set to true.. I will update the Gist11:07
odyssey4meFuzzyFerric: so openstack_service_publicuri_proto: https is the default, so no need to set it11:08
odyssey4me'openstack_external_ssl: true' is also the default, so no need to set it11:08
FuzzyFerricok, should I comment out those two lines int he config?11:09
odyssey4me'haproxy_ssl: true' is also the default, so no need to set it11:09
odyssey4me'haproxy_bind_on_non_local' will be set to 'true' dynamically if you have more than one haproxy host, which you do, so again no need to set it11:10
odyssey4me'galera_cluster_name: openstack_galera_cluster' is also the default, so no need to set it11:10
odyssey4meso yeah, remark those out or remove them entirely - perhaps comment them out and add a comment about why you did so, to ensure the rest of your team knows ;)11:11
odyssey4meok, so next I see 'glance_default_store: cinder' which is something I don't think is well tested... so we need to look there, especially given that it's glance throwing those errors11:12
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Convert lxc2 to lxc3 config keys dynamically  https://review.openstack.org/60153411:13
odyssey4meyeah, I'm guessing we're missing some settings in glance.conf which inform glance to use the internal endpoint11:13
FuzzyFerricThank you.. It is glance, against the Cinder back end which is on a SAN (and works fine for the Voume service)11:13
*** hamzaachi has quit IRC11:15
openstackgerritMerged openstack/openstack-ansible-os_glance stable/ocata: Clean up role tests  https://review.openstack.org/60188011:16
odyssey4meFuzzyFerric: given https://ebalduf.github.io/configure-glance-for-cinder/ it seems there's quite a bit of missing config11:17
openstackgerritMerged openstack/openstack-ansible-os_glance master: Remove python-keystoneclient package install  https://review.openstack.org/58772411:17
odyssey4memore in https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-backed-image.html too11:17
*** noonedeadpunk has quit IRC11:18
*** noonedeadpunk has joined #openstack-ansible11:18
odyssey4meFuzzyFerric: you can use https://docs.openstack.org/openstack-ansible/latest/reference/configuration/using-overrides.html#overriding-openstack-configuration-defaults to set just about anything in glance.conf - if you can work out all the bits that need to go in there and share those, we can push up a patch to make it work by default11:19
chkumar|ruckodyssey4me: hello11:21
chkumar|ruckodyssey4me: I am going to break this patch into smaller stuff https://review.openstack.org/59142411:21
FuzzyFerricodyssey4me: Great, Thank you… I will go and read that and see what I am missing here11:21
chkumar|ruckodyssey4me: like: 1. Installing from rpm and runnign tempest from that11:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/rocky: Remove python-keystoneclient package install  https://review.openstack.org/60336111:21
chkumar|ruck2. using tempestconfiguations11:21
chkumar|ruck3. using tempest container11:22
chkumar|rucklike this way11:22
chkumar|ruckI will abandon the first patch11:22
odyssey4mechkumar|ruck: sure, sounds good11:22
odyssey4mesmaller patches are always helpful11:22
chkumar|ruckodyssey4me: where is the centos functional tempest job defined?11:23
odyssey4methere is plenty of prior art in the compute kit roles for installing via distro packages instead of from source if you'd like to see existing conventions - of course if you see anything that's totally bat crazy then please feel free to question it or change it :)11:23
odyssey4mechkumar|ruck: the job is defined in openstack-ansible-tests, as is the playbook - each repo has tox config to add some env vars or extra vars where necessary for driving different types of installs11:24
odyssey4metake a peek at the keystone role for example, it has many different configs tested11:25
odyssey4meall role tests also use tests/test.yml as the top-level meta playbook11:25
chkumar|ruckodyssey4me: https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_install.yml source and distro logic are in same task11:26
odyssey4mechkumar|ruck: yes, they share many things - but the source install diverges here: https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_install.yml#L109-L11111:27
chkumar|ruckodyssey4me: can we made this something like https://github.com/openstack/ansible-role-nodepool/tree/master/tasks/install ?11:28
chkumar|ruckfor tempest?11:28
chkumar|ruckodyssey4me: or I follow the keystone role methodology?11:28
odyssey4mechkumar|ruck: yes, a task routing system would be fine for tempest given that it's a lot less complex11:28
odyssey4meI'd suggest making that change in its own patch11:29
chkumar|ruckyou mean resusing the previous patch, I can do that11:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift master: Remove python-keystoneclient package install  https://review.openstack.org/58773011:29
odyssey4meyeah, that's fine too11:29
chkumar|ruckok, great, thanks :-)11:30
odyssey4mechkumar|ruck: I wouldn't bother with the git/pip thing though, just focus on source vs distro. I'm doing a bunch of work on the source builds, and pip/git will just be automagically handled when I'm done.11:30
chkumar|ruckodyssey4me: ok11:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Remove python-keystoneclient package install  https://review.openstack.org/58772211:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat master: Remove python-keystoneclient package install  https://review.openstack.org/58772511:33
openstackgerritMerged openstack/openstack-ansible-os_swift stable/rocky: Update branch to point to Rocky  https://review.openstack.org/59980811:34
*** sum12 has joined #openstack-ansible11:35
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Switch to simple service test instead of tempest  https://review.openstack.org/60334611:38
*** gkadam has quit IRC11:39
*** gkadam has joined #openstack-ansible11:40
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307811:54
*** markvoelker has joined #openstack-ansible12:04
kaiokmomorning everyone12:06
*** gkadam has quit IRC12:06
*** shyamb has quit IRC12:06
*** shyamb has joined #openstack-ansible12:07
*** priteau has quit IRC12:14
*** priteau has joined #openstack-ansible12:17
openstackgerritMerged openstack/openstack-ansible-os_nova master: Rename SUSE vars file to cater for any version  https://review.openstack.org/60292712:20
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307812:21
*** dave-mccowan has joined #openstack-ansible12:23
errrodyssey4me: you there?12:23
odyssey4meerrr: yep12:23
errrodyssey4me: I have a federation question :)12:24
odyssey4meerrr: happy to try and help12:24
errrI have newton-eol setup and using ADFS as the IDP. I have a valid session and have checked the logs and it shows to be mapping my user to the set group. When I get to horizon and hit the ADFS button it gets back to horizon saying "Login failed: An error occurred authenticating. Please try again later."12:25
errrbut in the keystone logs it is a success12:25
errrand at /Shibboleth.sso/Session its a valid session12:25
errrIm not sure what else to check for why it wont let me into horizon12:26
odyssey4meerrr: hmm, so shib's authorizing it - are the mappings working then? there was a way to see the attributes seen by shib - not sure if it was /Session or another URL12:27
errrwell I had to decrypt the assertion because its all encrypted.12:28
*** dave-mccowan has quit IRC12:28
errrbut, once I did that the 1 attribute ADFS is sending is defined propery in the mappings.xml file and keystone shows its mapping my user based on that to my fedgroup in the Default domain12:29
guilhermespmorning12:29
errrhttps://gist.github.com/michaelrice/b65140e08b6582b2d620e73d6539204c12:30
errrthats the mapping file12:30
guilhermespso odyssey4me I didn't look at the PR yet but are they working now ?12:30
guilhermespPRs*12:30
odyssey4meguilhermesp: it's in the gate now :)12:30
guilhermespsweet odyssey4me ! awesome work12:30
*** chkumar|ruck is now known as chkumar|off12:32
odyssey4meerrr: hmm, it's been too long - I wonder if cmurphy/lbragstad or any of the other keystone folks can help (in #openstack-keystone)12:32
errrIll ask, thanks :)12:32
odyssey4meguilhermesp: oh, it just merged :) https://review.openstack.org/60292712:32
guilhermespI just saw now :D12:33
openstackgerritMerged openstack/openstack-ansible-os_nova master: Remove galera_client meta-dependency  https://review.openstack.org/60290712:33
guilhermesp🤟🏼12:33
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Ensure that SSHD is installed, enabled and running  https://review.openstack.org/60311912:38
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307812:39
*** shyamb has quit IRC12:45
*** shyamb has joined #openstack-ansible12:45
*** ansmith has joined #openstack-ansible12:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Switch to simple service test instead of tempest  https://review.openstack.org/60334612:49
*** hamzaachi has joined #openstack-ansible12:49
*** shyamb has quit IRC12:50
openstackgerritMerged openstack/openstack-ansible-os_heat master: Remove python-keystoneclient package install  https://review.openstack.org/58772513:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client stable/rocky: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337313:08
*** spatel has joined #openstack-ansible13:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client stable/queens: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337513:12
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client stable/pike: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337713:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat stable/rocky: Remove python-keystoneclient package install  https://review.openstack.org/60337813:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift master: Remove python-keystoneclient package install  https://review.openstack.org/58773013:19
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_apparmor: Drop lxc-start openSUSE workaround  https://review.openstack.org/60337913:24
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_apparmor: Use aa-disable to disable apparmor profiles  https://review.openstack.org/60338013:24
spateljamesdenton: let me know when you there13:26
spatelno rush13:26
spateloops i have meeting in 5 min too...13:27
spatelI have few results to share13:27
openstackgerritMerged openstack/openstack-ansible-galera_server master: Revisit conflicting package removal  https://review.openstack.org/60321513:27
jamesdenton i'm around13:29
spatelLet me finish my meeting and i will ping you back..13:30
spatelwhat do you think about Linuxbridge vs ovs performance?13:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/rocky: Revisit conflicting package removal  https://review.openstack.org/60338313:30
spateldo you think OVS will perform better in this case?13:30
jamesdentonno13:31
spatelYesterday i ran couple of test and found TCP doing good job so far compare to UDP13:32
*** ekan has quit IRC13:32
guilhermespare we going to have a meeting today?13:33
spatelIf i set "ethtoot -L eth0 combined 8"  then TCP rocks and i get 200kpps rate13:33
spatelbut UDP doing very bad if i set queue 8 only getting 40kpps13:33
openstackgerritMerged openstack/openstack-ansible-os_zun stable/rocky: Update branch to point to Rocky  https://review.openstack.org/59981413:34
spatelbut UDP doing very good with queue #2 size but still getting drops on tap interface13:34
spateli am still test multiple workload and capturing all data13:35
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server stable/queens: Revisit conflicting package removal  https://review.openstack.org/60338813:36
jamesdentonif i have some time today i will try to run that test13:36
spatelthat would be great!!13:36
spateli have one more cloud running on OVS i am running test on it..13:36
spatelI google everywhere and didn't find any performance data related PPS rate.. everybody talking about throughput and bandwidth rate... I think PPS does matter too..13:38
spatelhow fast your vm processing packets13:38
mbuil_odyssey4me: what is the integrated build in the context of your comment: https://review.openstack.org/#/c/603068/1?13:41
openstackgerritMerged openstack/openstack-ansible-os_sahara master: Remove galera_client meta-dependency  https://review.openstack.org/60291213:41
odyssey4membuil_: the openstack-ansible repo is often termed 'the integrated build'13:41
hwoarangmbuil_: integrated build is the openstack-ansible repo where all roles come together and tested13:41
*** goldenfri has joined #openstack-ansible13:42
mbuil_odyssey4me, hwoarang: ok, thanks!13:43
spateljamesdenton: this is my iperf look like 5Gbps rate without packet loss for TCP13:46
spatelhttp://paste.openstack.org/show/730270/13:46
openstackgerritMerged openstack/openstack-ansible-os_trove stable/queens: Update trove tempest plugin path  https://review.openstack.org/60310913:47
openstackgerritMerged openstack/openstack-ansible-os_trove stable/queens: Remove mysql-python  https://review.openstack.org/60295813:47
jamesdentonabove 5Gbps you see loss?13:47
odyssey4mehwoarang: mind reviewing this quickly? https://review.openstack.org/60311213:47
jamesdentonDid you adjust the scaling_governor?13:47
spatelYes i have changed it to performance but result is pretty much same13:49
jamesdentonk13:49
-spatel- [root@foo ~]# iperf -c 10.10.10.10 -u13:49
-spatel- ------------------------------------------------------------13:49
-spatel- Client connecting to 10.10.10.10, UDP port 500113:49
-spatel- Sending 1470 byte datagrams, IPG target: 11215.21 us (kalman adjust)13:49
-spatel- UDP buffer size: 208 KByte (default)13:49
-spatel- ------------------------------------------------------------13:49
-spatel- [ 3] local 10.10.10.10 port 48961 connected with 10.10.10.10 port 500113:49
-spatel- [ ID] Interval Transfer Bandwidth13:49
jamesdentoncareful with the paste13:49
-spatel- [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec13:49
-spatel- [ 3] Sent 892 datagrams13:49
-spatel- [ 3] Server Report:13:49
-spatel- [ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.026 ms 0/ 892 (0%)13:49
spateljamesdenton: this is interesting UDP result is very poor http://paste.openstack.org/show/730271/13:49
spatelsorry by mistake i copy paste13:49
spatel1.25 MBytes UDP & 5Gbps for TCP13:50
jamesdentonDoesn't the server also need to be started with -u? can't recall13:50
spatelthat is terrible result13:50
spateli did -u13:50
spateli am very good at iperf because that is what i am running last 48 hour :)13:50
spateliperf -c 10.10.10.10 -u  ( on server side)13:51
spatelsorry client side...13:51
spatelserver side "iperf -s -u"13:51
spatelDo you have any tool you want me to try based on your past experience ?13:51
jamesdentonnot at the moment. You might also try disabling port security if you're testing between VMs. Also, test between the two compute nodes so you have a reference point13:52
hwoarangodyssey4me: done13:52
*** lbragstad has quit IRC13:53
spatelport security?13:53
spatelis that iptables?13:53
jamesdentonneutron port security. it affects iptables, yes13:53
spateli will try to disable it and see13:53
jamesdentondisable GRO inside the VMs13:54
*** lbragstad has joined #openstack-ansible13:54
spatelethtool -K eth0 gro off   -- running test again13:55
spatelsame result [  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.028 ms    0/  892 (0%)13:55
spatelpoor UDP13:56
odyssey4metyvm hwoarang13:57
*** radeks_ has joined #openstack-ansible13:58
mnaserodyssey4me: the thing about centos is temporarily i assume right14:01
mnaseruntil we get rid of galera_client this cycle (i say we converge to galera_server for all the delegated db creates)14:01
odyssey4memnaser: we already delegate to galera_server for all actions, and have been doing for ages14:01
odyssey4meso I'm removing MySQL client all the way back to pike, then will get rid of galera_client all the way back to pike too14:02
mnaserodyssey4me: right, but i mean we install the client side things in galera_server rather than meta dependency of galera_client in galera_server14:02
odyssey4mewell, not totally rid of - galera_client still runs against the utility container and the galera_server14:02
mnaserah okay14:03
odyssey4mewe still need galera_client on the utility container, and running the whole server role seems a little wasteful for that don't you think?14:03
mnaserodyssey4me: yeah -- didn't think of the utility container14:03
odyssey4mewe could do that, and just disable a bunch of stuff that's done in the server role... but we can take that decision later14:03
mnaseryeah if it was galera server only it would hve been a much more straight forward decision14:04
*** vakuznet has joined #openstack-ansible14:04
odyssey4meagreed, I initially thought that too - then remember the utility container14:04
mnaserim surprised that it's conflicting the way it is though in centos14:04
*** shyamb has joined #openstack-ansible14:04
mnaserdoes that mean anyone with a fresh centos install has that issue? i'd think its a packaging bug in maraidb14:04
odyssey4meof course we could consider getting rid of the utility container in the future, and switch to leaving a deployer to do their own thing, or just use the deploy host14:05
odyssey4memnaser: the conflict seems to be something in openstack-ci, rather than base centos14:05
odyssey4meit's only on the host, not in the containers14:06
*** Emine has quit IRC14:06
odyssey4mehaving the removal of the conflicts is good though - they were originally implemented long ago due to issues that arose in CI and reported in bugs14:06
mnaserodyssey4me: so the nodepool images are the ones that have this problem?14:07
mnaserthat's why we hit it in baremtal i assume14:07
odyssey4meI think so, yes - but there are also deps in RDO which bring them in, and that affects distro installs.14:08
odyssey4meThat's why it had to be removal without deps.14:08
openstackgerritManuel Buil proposed openstack/openstack-ansible master: Ensure ceilometer repo gets installed with neutron  https://review.openstack.org/60340014:09
*** radeks_ has quit IRC14:14
*** KeithMnemonic has quit IRC14:14
*** KeithMnemonic has joined #openstack-ansible14:14
hwoarangodyssey4me: do you understand this problem http://logs.openstack.org/78/603078/10/check/openstack-ansible-linters/c663e4d/job-output.txt.gz#_2018-09-18_13_24_29_516218 caused by line 34 on https://review.openstack.org/#/c/603078/10/tasks/haproxy_apparmor.yml14:16
hwoarangit's exactly the same with your galera_client code14:17
odyssey4mehwoarang: I've seen that before. I've not worked out why - it might be because it's in a block?14:17
odyssey4meIt's a bug in ansible-lint for sure.14:18
hwoarangright14:18
odyssey4meadd the skip_ansible_lint tag if you can't work out why - aint nobody got time for that nonsense14:18
hwoaranglol yeah ok14:18
*** Emine has joined #openstack-ansible14:19
hwoarangor possibly a comment as first line14:19
openstackgerritMerged openstack/openstack-ansible-os_trove stable/pike: Use the correct branch for tempest tests  https://review.openstack.org/60311114:19
odyssey4mehwoarang: it would be nice to know the root cause, because I have a few TODO's littered around to figure it out. :p14:20
hwoarangyeah.... i may look at it if i find the right amount of motivation14:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/ocata: Use the correct branch for tempest tests  https://review.openstack.org/60340514:20
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307814:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove stable/ocata: Use the correct branch for tempest tests  https://review.openstack.org/60340514:21
cloudnullmornings14:22
mnasergood morning14:23
cloudnullo/14:24
cloudnullhows it post PTG ?14:24
cloudnullback to normal yet ? :)14:24
openstackgerritMerged openstack/openstack-ansible-os_glance stable/rocky: Remove python-keystoneclient package install  https://review.openstack.org/60336114:25
openstackgerritMerged openstack/openstack-ansible stable/ocata: Improve get-pip.py reliability  https://review.openstack.org/60288514:27
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_apparmor: Use aa-disable to disable apparmor profiles  https://review.openstack.org/60338014:29
openstackgerritMerged openstack/openstack-ansible-os_neutron stable/rocky: Update branch to point to Rocky  https://review.openstack.org/59980114:31
openstackgerritMerged openstack/openstack-ansible stable/ocata: Bump plugins SHA to include recent bugfixes  https://review.openstack.org/60299414:32
openstackgerritMerged openstack/openstack-ansible master: fix tox python3 overrides  https://review.openstack.org/57431314:32
spatelcloudnull: hey!!14:32
spatelcloudnull: Question how do i disable port security ?14:33
cloudnullI believe there's an option in the os_neutron role to do that14:33
* cloudnull goes to look 14:33
spatellet me see14:33
spatelis port security means iptables ?14:35
spatelis it possible i can disable port security for specific instance?14:39
hwoarangodyssey4me: maybe you want to track this https://github.com/willthames/ansible-lint/issues/27514:39
cloudnullwelp, it looks like the port sec driver is added by default14:39
cloudnullhttps://github.com/openstack/openstack-ansible-os_neutron/blob/master/templates/plugins/ml2/ml2_conf.ini.j2#L814:39
hwoarangit's a known problem14:39
jamesdentonyou can disable it on a port-by-port basis using the api14:40
*** dxiri has joined #openstack-ansible14:40
*** rgogunskiy has quit IRC14:40
cloudnull^ said it before I could :)14:40
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-haproxy_server master: Disable HAProxy apparmor profile if present  https://review.openstack.org/60307814:40
spateljamesdenton: how can i do that..14:40
jamesdentonopenstack port set -h14:40
spatelsorry asking very basic question14:40
jamesdenton:D14:40
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: tasks: lxc_apparmor: Use aa-disable to disable apparmor profiles  https://review.openstack.org/60338014:41
spatelreading...14:42
spatel"neutron port-update ccbd0ed6-3dfd-4431-af29-4a2d921abb38 --port_security_enabled=False"  this is what you talking about?14:42
jamesdentonyes14:43
spatellet me try14:43
spateli have to remove security group right if i disable port security?14:44
jamesdentonyou might need to, yes14:45
openstackgerritMerged openstack/openstack-ansible-os_cinder master: Remove python-keystoneclient package install  https://review.openstack.org/58772214:46
*** cshen has quit IRC14:47
spateljamesdenton: i disabled port-security but now i can't SSH in because i have removed security group also14:48
openstackgerritMerged openstack/openstack-ansible-os_trove stable/queens: Pin the tempest test plugin  https://review.openstack.org/60311214:48
openstackgerritMerged openstack/openstack-ansible-os_trove stable/queens: Clean up role tests  https://review.openstack.org/60186414:48
openstackgerritMerged openstack/openstack-ansible-os_trove stable/queens: import zuul job settings from project-config  https://review.openstack.org/59512314:48
jamesdentonit should let you in, since there are no rules applies at all14:48
spatelOh! wait... i am in but it took very long time to ssh in14:49
spatellook like won't able to DNS or something anyway i will look into that later first let me run performance test14:49
*** FuzzyFerric has quit IRC14:55
spatelCompte <---iperf test-----> Compte    ( TCP: 8Gbps & UDP 1.25Mbps ) something is wrong with UDP14:56
*** hamzy has quit IRC14:56
spatelI ran test between compute node.. (no instance)14:56
spatellet me run test on standard box... openstack..14:57
*** hamzy has joined #openstack-ansible14:58
*** luksky has quit IRC14:59
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Rename SUSE vars file to cater for any version  https://review.openstack.org/60293215:00
openstackgerritMerged openstack/openstack-ansible-os_congress master: Fix the wrong home-page  https://review.openstack.org/60301915:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Remove python-keystoneclient package install  https://review.openstack.org/60340915:02
openstackgerritMerged openstack/openstack-ansible-os_tacker master: Fix the wrong home-page  https://review.openstack.org/60302015:07
*** strattao has joined #openstack-ansible15:10
*** dxiri has quit IRC15:12
odyssey4membuil_: we only cherry-pick once it merges in master15:14
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_ceilometer stable/rocky: Switch to simple service test instead of tempest  https://review.openstack.org/60341515:14
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Set default ssl protocols according to used webserver  https://review.openstack.org/60312015:16
*** dariko has joined #openstack-ansible15:17
*** shyamb has quit IRC15:18
*** Leo_m has joined #openstack-ansible15:22
mbuil_odyssey4me: ok, sorry!15:24
*** dariko has quit IRC15:24
*** dariko has joined #openstack-ansible15:25
*** MrWatson is now known as NostawRm15:26
*** Emine has quit IRC15:31
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support  https://review.openstack.org/60310015:33
*** cshen has joined #openstack-ansible15:34
odyssey4mecores, FYI master ic unblocked - so any failures you see should be looked at more carefully, and integrated build patches can be rechecked15:38
*** tomtom001 has quit IRC15:41
*** cshen has quit IRC15:44
cloudnullwoot!15:44
prometheanfireguess that means it's time to break master right?15:45
*** luksky has joined #openstack-ansible15:45
mnaserodyssey4me: thank you for your work15:50
mnasercloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397,15:51
mnaserqwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, Tahvok, mnaser, nicolasbock, jrosser15:51
mnasermeeting in 10.  anyone around? :)15:51
d34dh0r53o/15:51
jrossero/15:51
spotzI'm always aroundish:)15:51
guilhermespo/15:51
odyssey4meo/15:51
nicolasbocko/15:51
hwoarangi will be on an internal meeting but i will keep an eye here as well15:52
prometheanfireI am a meat popsicle15:52
evrardjpo/15:54
evrardjpprometheanfire: wrong air conditioning?15:54
prometheanfireevrardjp: don't get the ref?15:55
evrardjpI cannot link it to anything that was just said15:59
*** dxiri has joined #openstack-ansible16:00
cjloadero/16:01
evrardjpo/16:01
*** hamzy has quit IRC16:01
openstackgerritGerman Eichberger proposed openstack/openstack-ansible-os_octavia master: Break out config files for each octavia service  https://review.openstack.org/60322816:02
*** gkadam has joined #openstack-ansible16:02
*** johnduarte has joined #openstack-ansible16:02
prometheanfiremnaser: 11?16:02
mnaserugrh16:03
mnasersorry16:03
mnasertc stuff16:03
mnaser#startmeeting openstack_ansible_meeting16:03
openstackMeeting started Tue Sep 18 16:03:58 2018 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.16:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:04
prometheanfire:P16:04
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:04
mnaser#topic rollcall16:04
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:04
*** openstack changes topic to "rollcall (Meeting topic: openstack_ansible_meeting)"16:04
prometheanfireo/16:04
cjloadero/16:04
mnasero/16:04
spotzo/ - but ping if needed in case I squirrel off:)16:05
mnaser#topic Last week highlights16:05
*** johnduarte has quit IRC16:05
*** openstack changes topic to "Last week highlights (Meeting topic: openstack_ansible_meeting)"16:05
mnaserwe were at the ptg last week16:05
mnaserso we didn't hold a meeting16:06
mnaserit was super productive, most of the stuff we tried to talk about were jotted down here -- https://etherpad.openstack.org/p/osa-stein-ptg16:06
mnaseri see a lot of green there which was me taking notes, if anyone wants to go over those that'd be good in case i misunderstood things :)16:06
odyssey4me:)16:07
mnaserother than that i'm super excited for teh upcoming cycle16:07
mnaserit will be a cycle of a lot of deleted code :D16:07
mnaserhaha16:07
guilhermespmnaser: how about kafka stuff?16:07
mnaserguilhermesp: sure!  ansmith16:08
mnasermight want to comment on that16:08
mnaserwe can help out with getting the stuff imported into openstack gerrit, get some basic gating in place16:08
mnaserand then that can we be used for an alternative messaging job16:08
guilhermespas I mentioned in the ehterpad. That one that we use for monasca is in my github just because other are pretty stale16:08
mnasermaybe wait for ansmith's +1 and if he sees value then we can add it16:09
guilhermespbut if you guys thinks will be good to more to osa namespace we can work on it16:09
mnaseri'm not opposed to adding it under the group of things we manage, it makes it easier, it also means that we can add it to CI and have it as a deliverable, we can add it with it's own core team that's a subset of osa-core and you'd be on it guilhermesp16:09
*** johnduarte has joined #openstack-ansible16:09
mnaserthoughts?16:10
odyssey4meyup, ansible-role-kafka - then keep it generally useful and not OSA specific16:10
evrardjpagreed16:10
guilhermespI think is a better approach16:10
guilhermespI will just need to use that one as a start point16:11
mnaserguilhermesp: yes, it's possible to import an existing project16:11
guilhermespas it is designed to work along monasca rola16:11
guilhermesprole*16:11
waynro/16:11
mnaserguilhermesp: do you want to work on importing the project? https://docs.openstack.org/infra/manual/creators.html16:11
odyssey4meguilhermesp: sure, it'll likely need some rework, but that can happen in time16:11
guilhermespmnaser: sure I do16:12
*** fghaas has quit IRC16:12
mnasercool, please ping me on any reviews on that (it'l llikely require a ptl+1 anyways)16:12
guilhermespk mnaser16:12
odyssey4meguilhermesp: you'll find, for example, the os_neutron role has an independent core team - so you can model from it16:12
odyssey4mefor the gerrit config I mean16:13
mnaser^ yep, that too16:13
guilhermespcool I will need some guidance  to do it but I'm pretty sure can be done by my side16:13
TahvokHey guys! Just wanted to say a few words. Unfortunately, I have moved to other company, and won't be dealing with openstack anymore. Anyway, I just wanted to say thank you all for supporting me, and guiding me trought this awesome project. It was really nice working with you, and even meating some of you at the ptg event, and hopefully we'll meet someday again, maybe it'll be openstack event again,16:14
Tahvokwho knows. Thanks for everything!16:14
mnaserTahvok: thank you <316:15
mnaserdon't be a stranger!16:15
odyssey4meo/ Tahvok - thanks for helping us out and best wishes as you move on to your next adventure!16:15
mnaserguilhermesp: cool well please let me know if you need any help with moving it16:16
*** hamzaachi has quit IRC16:16
mnaser#topic bug triage16:16
*** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)"16:16
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179205016:16
openstackLaunchpad bug 1792050 in openstack-ansible "SELinux context overrides fail for neutron install on metal" [Undecided,New]16:16
mnasereh16:17
mnasershould we just remove the selinux stuff because we don't support it right now anyways?16:17
odyssey4meargh, just delete it and port all the deletes back as far as they go16:17
mnaserjust like swift16:17
mnaserguilhermesp: can you take this please? i know it's affected us a few times16:17
guilhermespsure16:18
mnaserso confirmed/medium/assign to guilhermesp16:18
odyssey4meit's not like mhayden even likes having setenforce 1 ;)16:18
mnaseryeah we have to ping him everytime we pull selinux code out16:18
mnaserha16:18
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179187316:19
openstackLaunchpad bug 1791873 in openstack-ansible "Keystone LDAP tasks fail because haproxy pool is drained during install" [Undecided,New]16:19
odyssey4megiven he runs openstack on centos, it's the least we can do ;)16:19
odyssey4meoh bother, that's a pain16:20
odyssey4mewe'll have to make it do that set of tasks on the last of the keystone hosts so that it has an endpoint to speak to16:20
odyssey4meI'll pick that up16:20
mnasercool, thanks odyssey4me16:20
mnaserdone!16:20
odyssey4meoh nice, it's using our old module - so this is another little bit of tech debt to clean up16:21
mnasersweet16:22
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179125816:22
openstackLaunchpad bug 1791258 in openstack-ansible "Make use of ansible-config_template repo" [Undecided,New]16:22
mnaserugh darn16:22
mnaserthe code is diverged too16:22
mnaserno bueno16:22
mnaserjrosser: given you commented do you have some spare time? :p16:23
odyssey4meevrardjp: had a plan for this16:23
*** cshen has joined #openstack-ansible16:24
mnaserevrardjp: a plan with time to do it or? :p16:25
evrardjpgood point16:25
evrardjpI will document the plan if I cannot work on this next week-end16:25
evrardjpis that okay for you?16:26
mnaserevrardjp: seems reasonable, can we assign to you and then if time is a thing we can reassign?16:26
* jrosser back16:27
evrardjpmnaser: fine for me16:27
jrosserit all blew up for me with the elk stuff which calls up the seperate repo, that had/has a bug with json templating16:27
mnaserjrosser: i had the same issue i think16:27
jrosserneed to be careful when bringing the code into line not to pollute the good code with that16:28
jrosserand also it probably means there is a missing test16:28
mnasergod dang, my mouse battery is dying aha16:28
* mnaser will be slower16:28
mnaserum16:28
mnaserim getting a big error16:28
mnasersetting that bug to confirmed16:28
mnaseranyone else can do it?16:28
*** dariko has quit IRC16:29
*** spatel has quit IRC16:30
mnaserplease? :>16:30
mnaseri'm getting a big blob of red error text.16:30
jrosserdone16:31
mnaserthats weird16:31
mnaseri guess its something on my side16:31
mnaseryeah i cant set anything it keeps giving me errors16:31
mnasergah16:31
mnaserok switcihng browsers worked16:31
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179108516:32
openstackLaunchpad bug 1791085 in openstack-ansible "OVN Metadata Service Broken" [Undecided,New]16:32
* mnaser ping jamesdenton 16:32
prometheanfirerelated to ceilometer?16:33
mnaserand maybe cloudnull because that is related to teh thing that broke it16:33
mnaserno OVN + os_neutron16:33
mnaserhttps://github.com/openstack/openstack-ansible-os_neutron/commit/d6481ef9fcfc85ff931317b6f16bdac43cbb6488#diff-2444ad0870f91f17ca6c2a5e96b2682316:33
prometheanfirenope, shutting-up :D (there was another ovn issue upstream)16:33
prometheanfirereqs wise16:33
*** cshen has quit IRC16:35
mnaseri think given that both who are involved aren't around16:35
mnaserill leave it there so we review it next week16:35
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179093316:35
openstackLaunchpad bug 1790933 in openstack-ansible "Implement OVSDB clustering for OVN" [Undecided,New] - Assigned to James Denton (james-denton)16:35
mnaserlooks like this is to track a bug16:36
mnaserand already assigned16:36
mnaserto maybe in progress / medium16:36
*** cshen has joined #openstack-ansible16:37
*** hamzaachi has joined #openstack-ansible16:37
*** Darcidride has joined #openstack-ansible16:38
*** gyee has joined #openstack-ansible16:38
evrardjpmnaser: jamesdenton generally works like that -- let him the time and new feature shall appear :)16:38
mnaseryeah i guess no one else opposes to that16:39
mnaser#link https://bugs.launchpad.net/openstack-ansible/+bug/179077916:39
openstackLaunchpad bug 1790779 in openstack-ansible "delegate_to on deploying host uses dns?" [Undecided,New]16:39
mnaserhwoarang: or logan- might chime on this?16:39
hwoaranghmm16:40
mnaserafaik this should use the physical_mapping thing no?16:40
logan-yes one sec16:40
logan-i hit this on pike last week16:41
evrardjpinteresting, is it the case of delegation to something not in the inventory that triggers?16:41
logan-it is broken as of https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f16:41
logan-you can see it on a prod deploy here: http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/16:41
evrardjpoh god it's loop_var that causing the issue?16:41
hwoarangreporter does not say what version it is16:41
logan-http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/logs/ara-report/result/45ed4626-65ca-4b65-a4c4-d39cf32eaadf/16:41
jamesdentonmnaser pong16:41
hwoarangi thought the interpolation of loop_var was fixed by the recent commit in the plugins repo16:42
logan-hwoarang: i was hoping that backporting your change to the other branches would solve this, but it did not in pike16:42
openstackgerritJacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy Ceph into a Multinode AIO  https://review.openstack.org/58500316:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Implement LDAP domains using last keystone host  https://review.openstack.org/60344016:42
hwoarangwe should clarify on the bug what is the OSA version that he is using16:43
evrardjpdebug of https://github.com/openstack/openstack-ansible-plugins/blob/master/strategy/linear.py#L138-L147 would help I guess16:43
logan-that build used dc4abed00960c0123d45e9e67e666a5e9a1bb51f from plugins http://logs.openstack.org/e9/e957a7dc2fd07f86bc2ab60dd25e00504c705eca/post/limestone-ci-deploy/705562f/job-output.txt.gz#_2018-09-17_19_32_59_21741616:43
hwoarangbecause master and rocky should be ok16:43
logan-which is your fix: https://github.com/openstack/openstack-ansible-plugins/commits/stable/pike16:43
*** Darcidride has quit IRC16:43
evrardjphwoarang: do we have the loop_var in testing?16:43
hwoarangprob not16:43
odyssey4mewe do not, because there is only one cluster member16:44
hwoarangbut in opnfv we use rocky with multinode and it's ok (so we are using the loop_var thing)16:44
odyssey4mewe should add one to the plugins repo I guess16:44
*** Darcidride has joined #openstack-ansible16:44
mnaserjamesdenton: we can discuss after this issue :>16:44
jamesdentonyep - will sync up later16:44
evrardjphwoarang: ok16:44
*** Darcidride has quit IRC16:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Implement LDAP domains using last keystone host  https://review.openstack.org/60344016:45
mnaserso what do we do about this bug?16:45
*** dariko has joined #openstack-ansible16:45
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/60311416:46
logan-i don't know the fix yet, but it clearly exists in pike, not sure about ocata or queens, hwoarang says rocky is fixed and we can presume master is probably ok if rocky works16:46
hwoarangmnaser: well based on what logan said it's ay it's confirmed but I would ask the reporter to at least provide the OSA version so we know what we are looking for16:46
odyssey4mewe either backout multiple fixes, and add a release note notifying people - or we add a test to expose it and fix it, and add a release note16:46
logan-https://github.com/openstack/openstack-ansible/commit/a8a809839484105d9cd27463defc19a8a617c64b#diff-db999e390dd84f2a8c2a48b19aa9533f could be reverted and implemented differently16:47
logan-rather than trying to fix plugins16:47
*** olivierb has quit IRC16:47
*** dariko has quit IRC16:47
odyssey4meor that16:47
mnaserincomplete = ask for information such as release?16:47
evrardjpI'd say so16:48
logan-i will append the bug with this log so we can mark it confirmed on pike at least16:48
evrardjpok16:48
mnaserlogan-: cool wfm16:48
logan-would be nice to know if queens is broken also16:49
mnaserjamesdenton: going back to the ovn breakage, do you want to talk this over with cloudnull maybe to see if he can adjust the patch?16:49
* hwoarang needs to run for a bit. bbl16:49
jamesdentonyes, that's probably best. It looks like some of the other distros are handling it a little differently, by running OVS as non-root16:50
jamesdentonso long term we may need to figure out something different for OSA16:50
jamesdentoni can get with cloudnull16:50
*** dariko has joined #openstack-ansible16:50
mnaserthat'd be nice, ill leave it as is and we can follow up next week16:51
jamesdentonsure16:51
mnaser#topic open discussion16:51
*** openstack changes topic to "open discussion (Meeting topic: openstack_ansible_meeting)"16:51
mnaserany pressing issues? :)16:51
cjloaderI have a follow up proposal16:51
mnasersure16:51
cjloaderfor the https://review.openstack.org/#/c/556586/ designate spec we discussed last week16:52
spotzThat was quick cjloader:)16:52
cjloaderwhat if we provide an option to use bind and powerdns, to where we allow users to switch from one to another, we alrready have playbooks to install bind, but then can work to add powerdns later?16:54
prometheanfirecjloader: those are test playbooks16:54
*** dariko has quit IRC16:55
mnaseralso afaik mugsie mentioned powerdns is the 'better' driver subjectively16:55
evrardjpI am not surprised16:55
jrosseri linked to a WIP of mine that does a good deal of what the spec talks about, it consumes an external role for bind16:55
*** skiedude has joined #openstack-ansible16:56
jrosserwhich could be plugged for $dns-server of choice16:56
skiedudeFun fact running the ./scripts/run-upgrade.sh in any branch (at least for queens, rockey, master) it always says " This script will perform a Pike to Queens upgrade."16:56
cjloaderyes, but not fully tested though, bind has been tested on a PR basis16:57
evrardjpskiedude: haha I see a patch incoming then :)16:57
skiedudeI assume you'd need to patch that in every branch, no? evrardjp16:57
evrardjpthanks skiedude for noticing and the patch :)16:57
evrardjpskiedude: yup16:58
mnaserprometheanfire: what do you feel about what cjloader proposes?16:58
evrardjpbut let's start with master :)16:58
evrardjpI have to run16:58
evrardjpthanks mnaser and everyone16:58
prometheanfiremnaser: about choice?  I personally feel like powerdns should be focussed on first, simply because working with upstream is better, once that works move to bind (which role is not published iirc)16:59
noonedeadpunkevrardjp: if you wish to save your time, I can manage with that16:59
mnaseri agree with prometheanfire on this16:59
odyssey4mePersonally, I don't really see how someone would 'switch from one to the other' simply - but having an option for either is fine. If it's destined to be on the same host, then including it in the same role is fine - but I suspect people may want to put DNS on its own host, different to the designate service... so it probably makes better sense to have the role execute in a playbook instead.16:59
prometheanfireiirc designate will repopulate the server if the backend is switched17:00
cjloaderokay, powerdns first then17:00
prometheanfirebut I don't see switching 'live' as a common thing17:00
cjloaderi'm not saying switch live17:01
cjloaderi'm saying have a variable that will pull each respective role17:01
cjloaderif set17:01
mnaseryeah i think thats' what we agreed on / thats what odyssey4me suggested17:02
odyssey4meI'm suggesting that making that promise is a hard one to keep, so don't make it. Support one back-end first, then support a second. If people want to migrate from one to the other, it's up to them to test and do so.17:02
mnaserbut the first one we'd do would be powerdns17:02
odyssey4meyes, definitely17:02
cjloaderokay17:02
cjloadersounds good17:02
mnasercool, any other subjcets? :)17:02
mnaser(we're up on time anyways)17:03
odyssey4meI'd like to propose pulling https://review.openstack.org/#/q/topic:remove-mysql-python+(status:open+OR+status:merged) as far back as pike.17:03
mnaserim +117:03
odyssey4meWe've been using pymsql a long time, and most of that is just cleaning/tidying up.17:03
prometheanfiredidn't we change to pymysql?17:03
prometheanfireya17:03
odyssey4meBut doing this will also probably reduce run time considerably.17:03
odyssey4meI think we moved to pymysql as far back as liberty.17:04
mnaserand increase stabiltiy imho. i'm all for it17:04
odyssey4meok, I'll do the backport proposals tomorrow - please look out for them, it'd be best if we get them all in quickly rather than end up with a mix and match in a release.17:05
mnaser++ i will do that too17:05
mnaseralright, that is pretty much it i think17:05
mnaserthats everyone17:05
mnaser#endmeeting17:05
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs"17:05
openstackMeeting ended Tue Sep 18 17:05:35 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:05
* mnaser off to find lunch17:05
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-09-18-16.03.html17:05
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-09-18-16.03.txt17:05
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-09-18-16.03.log.html17:05
prometheanfireyep, lunching17:05
odyssey4meas an aside, some good news is that with the clean up we've been doing, the MNAIO images for several of the hosts are down under 1GB now, down from around 2GB... so it's working :)17:06
*** gkadam has quit IRC17:07
spotznice17:07
odyssey4memore cleaning up will be coming as I explore more things :)17:07
odyssey4mewe're still installing a truck load more distro packages than necessary17:07
odyssey4meand cleaning those up will make for faster build times :)17:07
prometheanfireuse gentoo17:09
prometheanfire:D17:09
openstackgerritLogan V proposed openstack/openstack-ansible master: Avoid using loop_var in endpoint manage  https://review.openstack.org/60345017:10
logan-that's great odyssey4me17:12
odyssey4meAlright, I'm out for the night. Have a great day!17:15
*** hamzaachi has quit IRC17:15
mgariepyhey logan- are you using Image-Volume cache with ceph ?17:19
logan-99% sure we are not17:20
logan-are you? ive looked at it before a little bit.17:20
mgariepyi'm trying to configure it but it doesn't seems to work.17:21
*** olivierb has joined #openstack-ansible17:21
logan-:/17:22
mgariepyit seems nice, caching image from qcow > raw so a volume can be created quickly17:22
mgariepyi'll try to debug it a bit.17:22
jrossercjloader: in that designate stuff have you ideas about the dns server being authoritative/recursive/both... ?17:22
logan-mgariepy: yeah i could see it saving a lot of space if you have lots of ppl uploading qcow and launching volume backed images17:24
*** DanyC has quit IRC17:24
logan-for my stuff we skipped it because all of our provider images are raw and most people just use them17:25
mgariepywe need to qcow image for local storage on the computes, and it would be nice to have the caching when ppl boot from a volume.17:27
logan-ahh yeah makes sense17:27
mgariepynext one should be ceph epehemeral only.17:28
logan-do you run any regions where you have a mix of HVs split between local storage and ceph ephemeral?17:28
mgariepyno region,17:29
mgariepyit's going to be a new cluster at some point.17:29
cjloaderjrosser: not yet. still have alot to consider17:29
logan-we use host aggregates to separate the ceph ephemeral from the local storage hvs, and just found that if you resize from a local storage flavor to a ceph-backed flavor, it throws the resize into error state and deletes the block device17:30
jrossercjloader: even though my bind example uses an unpublished role that just boils down to a trivial templating of the config file. The important thing for me is being able to deploy the external dns onto a host of my choice with the networking done how I need it.17:33
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add capability to set node role  https://review.openstack.org/60327017:35
skiedudeHas anyone ever run into duplicate yum repos before doing an ugprade? I ran the upgrade script from queens to rocky, and 3 of my containers failed because a yum repo was defined twice. https://pastebin.com/Mjmsfkfu17:36
skiedudeand if I manually fix this, can I just rerun the upgrade script again ?17:36
*** mmercer has joined #openstack-ansible17:40
mgariepyskiedude, please open a bug for this ?17:42
skiedudecan do, just wanted to check first17:42
skiedudeforgive me, but where do you open them at17:43
skiedudemgariepy:17:43
mgariepyLaunchpad: https://launchpad.net/openstack-ansible17:44
mgariepyskiedude, https://bugs.launchpad.net/openstack-ansible/+filebug17:44
*** electrofelix has quit IRC17:45
openstackgerritMerged openstack/openstack-ansible master: Bootstrap ansible virtualenv with --never-download  https://review.openstack.org/59968217:46
openstackgerritMerged openstack/openstack-ansible master: Move ara to scripts-library  https://review.openstack.org/60246117:52
cjloadermnaser: odyssey4me are BIND or PowerDNS currently gating for upstream?17:54
*** DanyC has joined #openstack-ansible18:04
openstackgerritMerged openstack/openstack-ansible master: Add nspawn to experimental jobs  https://review.openstack.org/60073918:07
*** DanyC has quit IRC18:09
d34dh0r53cloudnull: http://paste.openstack.org/show/730302/ seen this?18:20
*** morphmenow65 has joined #openstack-ansible18:21
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Tune-up logrotate config  https://review.openstack.org/60346418:21
cloudnulld34dh0r53 nope18:22
cloudnullrecent pull ?18:22
d34dh0r53yeah, this morning18:22
cloudnullive not seen that myself18:23
d34dh0r53hmm18:24
*** priteau has quit IRC18:28
*** spatel has joined #openstack-ansible18:36
*** johnduarte has quit IRC18:37
d34dh0r53ahh, looks like _logstash_block_device is not being defined18:37
*** morphmenow65 has quit IRC18:39
*** luksky11 has joined #openstack-ansible18:42
*** luksky has quit IRC18:46
openstackgerritJacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy Ceph into a Multinode AIO  https://review.openstack.org/58500318:48
*** fghaas has joined #openstack-ansible18:54
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_nova master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58445018:55
*** hamzaachi has joined #openstack-ansible18:55
mnaserwow cool news18:57
mnaserhttps://review.openstack.org/#/c/602193/18:57
mnaserthis is after removing majority of galera_client18:57
openstackgerritAndy Smith proposed openstack/openstack-ansible-os_cinder master: Setup oslo.messaging extra packages for optional drivers  https://review.openstack.org/58263218:59
mnaseri wonder if we should start adapting include_role instead of roles: foo19:01
mnaserto minimize memory usage19:01
*** hamzy has joined #openstack-ansible19:04
*** dxiri has quit IRC19:06
jamesdentonspatel i have two VMs here with 4 vCPU and 8GB RAM and DPDK. Anything you want me to try?19:07
spatelYes, install iptraf one one of VM19:08
spateland other VM install hping319:08
jamesdentondone19:08
spatelstart iptraf-ng tool > Detailed interface statistic19:09
spatelselect your NIC19:09
jamesdentonk19:09
spatelon other VM start hping3 <ip_address_of_other_VM> --flood19:09
spatelhping3 192.168.1.1 --flood   (this will use TCP for packet flooding)19:10
spatelhping3 192.168.1.1 --flood --udp ( this will use UDP)19:10
jamesdentonok it's going19:10
spatelcheck PPS rate and on compute node ifconfig tap<UUID>  | grep drop19:10
spatelyou will see TX drop on tap interface19:11
jamesdentonhmm, i won't be able to see it in that way, as these are dpdkvhostuserclient ports not tap ports. let me see if i can glean that info19:12
spatelok19:12
spateli have two VM to attack to increase PPS rate19:12
spatelhow much PPS rate you getting?19:13
jamesdentoni don't know yet p19:13
spatel:)19:13
spateli am little excited as you can see ;)19:14
jamesdentonIs this is right: sudo ovs-vsctl get Interface vhubacdbff5-a8 statistics19:15
jamesdentonthe counters show approx 11% loss19:15
jamesdentontx_dropped=2789347, tx_packets=237536919:15
openstackgerritMerged openstack/openstack-ansible-os_magnum master: Remove mysql-python  https://review.openstack.org/60287619:16
spatelI don't know about DPDK so don't know how to look that stats but any drop is bad19:16
spatelTry TCP and UDP both test and see19:17
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_neutron stable/queens: Drop SELinux support for CentOS 7  https://review.openstack.org/60347219:17
openstackgerritMerged openstack/openstack-ansible-os_magnum master: Remove galera_client meta-dependency  https://review.openstack.org/60290319:18
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-os_neutron stable/queens: Drop SELinux support for CentOS 7  https://review.openstack.org/60347219:19
spatelI am not using NUMA, i think i should try with NUMA and see result..19:19
skiedudemgariepy: in investigating further the yum issue was a red herring. The issue is that the interfaces on the infra1 host were removed, and not remade, so the container can't resolev anything, including yum repos19:19
skiedudethe container interfaces on the infra1 host for the 3 containers*19:20
guilhermespodyssey4me mnaser any reviews on that ? https://review.openstack.org/#/c/603472/19:20
jamesdentonyou can try. i'm still seeing drops according to the counters.19:21
spatelhmm! I am trying numa19:21
spatellook like this drops are all because CPU can't picking packet from queue fast enough....19:22
spatelif you run iperf test you will good throughput... but throughput is different then packet rate19:22
spatel"will get"19:23
*** priteau has joined #openstack-ansible19:29
*** olivierb has quit IRC20:04
*** dave-mccowan has joined #openstack-ansible20:09
*** fghaas has quit IRC20:14
*** fghaas has joined #openstack-ansible20:15
*** KeithMnemonic has quit IRC20:16
*** NostawRm has quit IRC20:17
openstackgerritMerged openstack/openstack-ansible master: playbooks: healthcheck-infrastructure: Ensure netcat is installed  https://review.openstack.org/60204320:19
openstackgerritGerman Eichberger proposed openstack/openstack-ansible-os_octavia master: Break out config files for each octavia service  https://review.openstack.org/60322820:20
jrosserspatel: you could take a read of this for some tips on the NUMA setup https://www.stackhpc.com/vxlan-ovs-bandwidth.html20:20
jrosseralthough that is talking in general about an OVS setup I think that the principles around CPU pinning, reservation and numa passthrough should be relevant in general20:20
spatelBut i am using LinuxBridge does that matter?20:20
jrosserOVS is just a piece of software, its not magic :)20:21
spatelreading... my problem is PPS rate issue ( because we are media company and dealing with realtime data)20:22
*** hamzaachi has quit IRC20:22
jrosserregardless of the actual networking implementation the same fundamental things about having the VM understand the underlying NUMA architecture, and ensuring there is sufficient CPU available for the kernel to run the network stack in a way thats not colliding all the time with your VM..... thats all got to be moving you in the right direction20:22
*** hamzaachi has joined #openstack-ansible20:23
jrosseryou could even try to ensure the CPUs that the kernel have are on the same socket as the PCIe lanes of the NIC...... lots and lots to go at here i think20:24
spatelI have seen my VM running on Cell0 zone of numa so that is good20:30
spatelhow to check PCI lanes i think that is for SR-IOV20:30
jrosseryou'd need a manual for your server and the output from numactl --hardware to figure out how stuff is arranged20:32
jrosserit's up to the hardware vendor20:33
jamesdentonYou would want to make sure the VF you're using is on a NIC using the same numa node as the VM20:33
jrosserbut if it were me i'd leave that till after looking at everthgin else20:33
logan-if anyone has a moment for a review, https://review.openstack.org/#/c/603450 thanks20:33
spatelI am learning about SR-IOV to implement, soon i am going to configure SRIOV20:34
jamesdentonwhat are the specs of your compute nodes, anyway20:34
spatelHP DL360p G8 (32 core 2.9GHz / 32G memroy)20:35
jamesdentonk20:35
jamesdentonand the VMs themselves? vcpu/ram?20:35
spatel15vCPU and 14GB memory20:36
spateli have run test on multiple flavor20:36
spatelresult is no good :(20:37
jamesdentonthat 32GB is split between the CPUs, probably?20:37
spatelno matter how big your VM is20:37
spatelYes20:37
spateljamesdenton: look at this http://paste.openstack.org/show/730312/20:38
jamesdentonk20:40
*** cshen has quit IRC20:41
spatelwhat about your compute hardware?20:42
spatelis it G8 ?20:42
jamesdentonI'm using DL360e G8s20:42
spatelI am planning to test on G9 next week to see if i get boost20:42
jamesdentonBut these are quite a bit slower. 2x E5-2450L @ 1.8Ghz20:42
spatelG9 has DDR 420:42
jrosseryou could check /proc/interrupts to see if the NIC interrupts are being handled on the CPU reserved for the kernel20:42
jrosserand fiddle smp_affinity if not20:42
spatelhmm! i will try that too20:43
spatelI am surprised no one seeing this issue in community :)20:43
spatelor may be whatever i am trying to do is something not normal and people won't do this on cloud20:44
jamesdenton^^^20:44
jamesdentonit's done (you're doing it on AWS) i just don't see it much20:44
*** ansmith has quit IRC20:45
jrosserthe telco/mobile operators probably do this *all* the time20:45
spatelWe are running 300 servers in AWS with SR-IOV20:45
spateland they are running smooth20:45
spatelmy AWS bill is over million dollar per year :) that is why i am building private cloud in datacenter20:46
jrosserlogan-: out of interest is there an underlying ansible bug here https://review.openstack.org/#/c/603450 ?20:46
jamesdentonReal quick, can you post the output of `lspci -vvv | grep Ethernet` to pastebin?20:46
spatelI think i have use SR-IOV to get around this solution, low latency and high packet rate..20:47
spatelok hold on20:47
logan-jrosser: it is a bug in the connection plugin i think20:47
*** hamzy has quit IRC20:47
jamesdentonoh wait, you're on a 360p, nevermind, i know the answer to my question20:47
logan-i see it on my env where haproxy is containerized20:47
spateljamesdenton: http://paste.openstack.org/show/730314/20:47
jamesdentonIIRC, the 360p G8 has two numa nodes when both CPUs are installed, but the PCI slots are all bound to numa020:47
logan-seems like we're only seeing it on old branches also jrosser20:48
jamesdentonHow about `cat /sys/bus/pci/devices/0000\:03\:00.1/numa_node`20:48
spatel020:48
logan-there is an irc log in the bug but the consensus is: not sure about ocata or queens, pike is broken, rocky and master are probably unaffected. I just did this fix against master anyway so we can backport it all the way thru20:48
jamesdentonyeah, ok.20:49
jamesdentonyou already have the VFs enabled so sr-iov isn't much of a stretch20:49
spatelYes i bought 100 servers, for openstack with that VF support20:50
spatelI am going to enable SRIOV and give it a try20:51
spatelit would be very interesting to see resilt20:51
spatelresult*20:51
*** mmercer has quit IRC20:51
jamesdentongood luck20:51
spatelI will let you know soon, stay tune :)20:51
spatelI gotta go now so i will catch you later..20:51
jamesdentonin AWS - are you able to do similar testing between VMs to get a baseline?20:52
spatelThanks jamesdenton for your great support :)20:52
spatellove it...20:52
jamesdentonapples-to-apples iperf/hping3 etc?20:52
jamesdentonsure, glad to help.20:52
spatelon aws we run our own application to find benchmark20:52
spateli didn't run hping320:53
spatelproblem is i can't see TX DROP in aws :)20:53
jamesdentonright, no access.20:53
spatelno tap interface in aws20:53
jamesdentonperhaps iperf is the way then?20:53
spatelbut we notice customer getting great experience as soon as we move them over SR-IOV20:53
jamesdentonthere needs to be a consistent test besides the app, if at all possible.20:53
jamesdentonsure.20:53
*** mmercer has joined #openstack-ansible20:53
jamesdentongotta run. good luck and keep us posted20:54
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: playbooks: healthcheck-infrastructure: Ensure netcat is installed  https://review.openstack.org/60349120:54
spateli will try to run hping3 and iptraf test on AWS find out20:54
spatelThanks!! good night20:54
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum stable/rocky: Remove mysql-python  https://review.openstack.org/60349220:55
*** spatel has quit IRC20:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum stable/rocky: Remove mysql-python  https://review.openstack.org/60349220:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Add nspawn to experimental jobs  https://review.openstack.org/60349320:56
*** priteau has quit IRC20:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_container_create stable/rocky: Convert lxc2 to lxc3 config keys dynamically  https://review.openstack.org/60349420:58
*** priteau has joined #openstack-ansible20:58
openstackgerritMerged openstack/openstack-ansible master: Add LXC COPR cache  https://review.openstack.org/60219321:00
*** cshen has joined #openstack-ansible21:07
*** hamzaachi has quit IRC21:07
*** cshen has quit IRC21:11
*** DanyC has joined #openstack-ansible21:12
*** vakuznet has quit IRC21:15
*** mmercer has quit IRC21:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Add LXC COPR cache  https://review.openstack.org/60349721:18
*** mmercer has joined #openstack-ansible21:20
*** Emine has joined #openstack-ansible21:28
*** ansmith has joined #openstack-ansible21:37
*** fghaas has quit IRC21:37
*** Emine has quit IRC21:40
*** fghaas has joined #openstack-ansible21:53
openstackgerritMerged openstack/openstack-ansible-os_horizon stable/rocky: Remove mysql-python  https://review.openstack.org/60307721:59
*** skiedude has quit IRC21:59
openstackgerritMerged openstack/openstack-ansible-galera_client stable/pike: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337721:59
openstackgerritMerged openstack/openstack-ansible-galera_client stable/rocky: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337321:59
*** goldenfri has quit IRC22:00
openstackgerritMerged openstack/openstack-ansible-galera_client stable/queens: RedHat: Remove packages with conflict with MariaDB  https://review.openstack.org/60337522:03
*** priteau has quit IRC22:05
*** DanyC has quit IRC22:08
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add playbook to automatically refresh fields in kibana  https://review.openstack.org/60351722:09
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: WIP - Enable stackviz support  https://review.openstack.org/60310022:13
openstackgerritMerged openstack/openstack-ansible-openstack_hosts stable/rocky: SUSE: Switch cloud repository to Rocky  https://review.openstack.org/60151722:14
*** luksky11 has quit IRC22:16
*** fghaas has quit IRC22:17
openstackgerritMerged openstack/openstack-ansible-os_ceilometer master: Switch to simple service test instead of tempest  https://review.openstack.org/60334622:30
openstackgerritMerged openstack/openstack-ansible-ops master: Add capability to set node role  https://review.openstack.org/60327022:32
openstackgerritMerged openstack/openstack-ansible-ops master: Tune-up logrotate config  https://review.openstack.org/60346422:32
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: Replace port 35357 with 5000  https://review.openstack.org/60311822:35
openstackgerritMerged openstack/openstack-ansible-os_nova stable/rocky: Update branch to point to Rocky  https://review.openstack.org/59980223:00
openstackgerritMerged openstack/openstack-ansible-os_swift master: Remove python-keystoneclient package install  https://review.openstack.org/58773023:06
*** Leo_m has quit IRC23:11
*** strattao has quit IRC23:13
*** tosky has quit IRC23:18
*** spatel has joined #openstack-ansible23:25
spatelcloudnull: ^^23:25
spatelyou there?23:25
spatelI have quick question23:26
spatelfor SR-IOV i should add these line in config file and keep whatever there right? http://paste.openstack.org/show/730325/23:28
openstackgerritMerged openstack/openstack-ansible-os_zun master: Remove rabbit_max_retries option  https://review.openstack.org/58778123:32
*** spatel has quit IRC23:41
*** mandeer has joined #openstack-ansible23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!