Wednesday, 2018-01-10

strgouthowdy..00:15
strgoutceph .. when i deploy with ceph what globals options enable creating volumes for VM?00:18
strgoutyou know.. a better question would be where are are global options documented00:20
logan-strgout: the best place to look for ceph-ansible options is https://github.com/ceph/ceph-ansible/tree/master/group_vars00:21
logan-with regard to the openstack integration, there is a doc here with example configs: https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/app-config-prod-ceph.html00:22
logan-look for "cinder_backends" at the bottom00:23
*** SmearedBeard has quit IRC00:31
*** andymccr has quit IRC00:36
*** _nyloc_ has joined #openstack-ansible00:36
*** nyloc has quit IRC00:36
*** chyka has quit IRC00:40
*** andymccr has joined #openstack-ansible00:40
*** markvoelker has joined #openstack-ansible00:43
*** markvoelker has quit IRC00:45
*** markvoelker has joined #openstack-ansible00:49
*** markvoelker has quit IRC00:50
*** andymccr has quit IRC00:51
*** andymccr has joined #openstack-ansible00:55
*** kaiokassiano has joined #openstack-ansible00:58
*** dave-mccowan has joined #openstack-ansible01:07
*** markvoelker has joined #openstack-ansible01:08
strgouthmm so do i need all the options in cinder_backends?01:08
*** markvoelker has quit IRC01:09
*** rmcall has joined #openstack-ansible01:10
*** dave-mccowan has quit IRC01:42
*** kaiokassiano has quit IRC01:45
*** gfa has joined #openstack-ansible01:50
*** jafeha__ has joined #openstack-ansible01:52
*** jafeha has quit IRC01:54
*** pbandark has quit IRC02:10
*** chyka has joined #openstack-ansible02:12
strgoutoh i see how it is.... so if I really want volumes I have to enable cinder. pfft.02:12
strgoutso one thing i'm unsure about. If i enable a new module (in this case cinder) can i just flip to yes then deploy or do i need to destroy then deploy again?02:15
*** dave-mccowan has joined #openstack-ansible02:15
*** chyka has quit IRC02:17
*** rmcall has quit IRC02:21
*** pbandark has joined #openstack-ansible02:28
*** pmannidi has quit IRC02:44
*** dave-mccowan has quit IRC02:47
*** pmannidi has joined #openstack-ansible02:48
*** lbragstad has quit IRC02:54
*** strgout has quit IRC03:10
*** pbandark has quit IRC03:16
cloudnullevenings03:17
cloudnullstrgout if you want to enable a new service you simply add the required config and run the playbooks.03:21
*** udesale has joined #openstack-ansible03:59
*** chhavi__ has joined #openstack-ansible04:25
masbergood afternoon, I am trying to setup my first OSA deployment04:33
masberdo I need br-mgmt on the compute nodes?04:33
pradiprwtmasber: yes, you need br-mgmt in every node04:56
masberthank you!04:57
pradiprwtthis interface is used for container management04:57
masberand also bind the external VIP?04:57
pradiprwtyeah, you are right04:58
pradiprwtinternal VIP as well04:58
masberbut not all services runs on LXC right? nova and cinder and bare metal04:58
pradiprwtyes, but ansible needs to deploy the services on compute and storage also, so it is using br-mgmt for that05:00
pradiprwtI think you can keep IP to interface05:01
masbersorry this is my first OSA installation, I went throught the documentation but I have the feeling taht I need help, I tried before packstack and kolla-ansible and they all configures the underlines network in different way plus I am not an expert in OVS or linux bridges etc...05:01
masberpradiprwt, one question, external VIP is same as the external_lb_vip_address and you said I need to set it up on the compute nodes, then are we talking about a DVR installation?05:08
*** poopcat1 has quit IRC05:09
*** brad[] has quit IRC05:10
pradiprwtmasber : Sorry I did't get you.. I was saying about management network for compute host...05:17
pradiprwtyou can assign any network IP to external_lb_vip_address, it should be accessible from infra nodes05:19
masberso I put the br-mgmt on the compute node, and then do I also need to bind the external VIP into the br-mgmt:0?05:20
pradiprwtNo need to bind external VIP with compute node, just provide as a variable to external_lb_vip_address  in openstack_user_config.yml file, OSA will bind it with HAproxy05:25
*** nshetty has joined #openstack-ansible05:26
pradiprwtmasber: the steps will be 1) prepare the deployment host which is your ansible node  2) Prepare the target host which are infra, compute and storage nodes then apply the playbooks05:28
pradiprwtmake sure networks are configured properly05:28
*** chyka has joined #openstack-ansible05:29
pradiprwtfor configuration you can take reference of given example  https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-test.html#test-environment-config05:29
*** admin0 has quit IRC05:33
masberhahaha ok! I will try05:33
*** chyka has quit IRC05:33
*** nshetty is now known as nshetty|afk05:35
*** woodard has quit IRC05:37
*** markvoelker has joined #openstack-ansible05:39
*** bhujay has joined #openstack-ansible05:42
*** admin0 has joined #openstack-ansible05:47
*** nshetty|afk is now known as nshetty05:59
*** jbadiapa has quit IRC06:02
*** oanson has joined #openstack-ansible06:06
*** oanson has quit IRC06:18
*** oanson has joined #openstack-ansible06:22
*** jbadiapa has joined #openstack-ansible06:26
*** pcaruana has joined #openstack-ansible06:27
*** jbadiapa has quit IRC06:30
*** pcaruana has quit IRC06:33
*** pcaruana has joined #openstack-ansible06:34
*** udesale__ has joined #openstack-ansible06:57
*** udesale has quit IRC06:59
*** arbrandes1 has joined #openstack-ansible07:01
*** arbrandes has quit IRC07:03
*** sar has joined #openstack-ansible07:06
*** sar is now known as sbr07:06
*** sbr is now known as sbra07:06
*** masber has quit IRC07:08
*** jbadiapa has joined #openstack-ansible07:18
*** udesale has joined #openstack-ansible07:19
*** udesale has quit IRC07:20
*** pmannidi has quit IRC07:21
*** udesale__ has quit IRC07:21
*** SmearedBeard has joined #openstack-ansible07:27
*** armaan has joined #openstack-ansible07:30
*** SmearedBeard has quit IRC07:32
*** sbra has quit IRC07:34
*** threestrands has quit IRC07:36
*** mardim has quit IRC07:38
*** sbra has joined #openstack-ansible07:45
*** mardim has joined #openstack-ansible07:51
*** _nyloc_ is now known as nyloc07:52
*** rgogunskiy has joined #openstack-ansible07:57
*** mbuil has joined #openstack-ansible08:04
*** gkadam has joined #openstack-ansible08:04
*** shardy has joined #openstack-ansible08:09
*** ianychoi has quit IRC08:35
*** mbuil has quit IRC08:35
*** mbuil has joined #openstack-ansible08:35
*** epalper has joined #openstack-ansible09:04
*** markvoelker has quit IRC09:08
*** sxc731 has joined #openstack-ansible09:11
*** pbandark has joined #openstack-ansible09:19
*** SmearedBeard has joined #openstack-ansible09:32
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_keystone master: Use a venv with previous ansible version for upgrades  https://review.openstack.org/53247009:35
*** cshen has quit IRC09:35
*** nshetty has quit IRC09:56
*** nshetty has joined #openstack-ansible09:56
*** cshen has joined #openstack-ansible09:56
*** nshetty_ has joined #openstack-ansible10:03
*** nshetty has quit IRC10:04
*** nshetty_ is now known as nshetty10:05
*** threestrands has joined #openstack-ansible10:21
*** stuartgr has joined #openstack-ansible10:36
*** rpittau has joined #openstack-ansible10:54
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_keystone master: DNM - test to see if linters are working  https://review.openstack.org/53248910:56
*** rgogunskiy has quit IRC10:57
pradiprwtHi Everyone, I have integrated the ceph with OSA, everything is working fine but I am not able to delete images.. any suggestion for this issue >>Traceback >>http://paste.openstack.org/show/642228/11:02
*** Smeared_Beard has joined #openstack-ansible11:02
*** SmearedBeard has quit IRC11:05
*** sbra has quit IRC11:06
*** markvoelker has joined #openstack-ansible11:08
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: cinder_qos: Use exact match for QoS rules  https://review.openstack.org/53249411:18
openstackgerritPeriyasamy Palanisamy proposed openstack/openstack-ansible-os_neutron master: add ml2 config for networking bgpvpn  https://review.openstack.org/52259811:31
*** armaan has quit IRC11:33
*** armaan has joined #openstack-ansible11:33
openstackgerritPeriyasamy Palanisamy proposed openstack/openstack-ansible-os_neutron master: add ml2 config for networking bgpvpn  https://review.openstack.org/52259811:33
*** armaan has quit IRC11:34
*** armaan has joined #openstack-ansible11:34
*** chhavi__ has quit IRC11:35
*** markvoelker has quit IRC11:42
*** ThomasSch has joined #openstack-ansible11:50
ThomasSchhi11:50
ThomasSchi know it is not OSA specific but does anyone know where to configure the action triggered by horizon when you click migrate host?11:50
ThomasSchi would love to have it trigger host-evacuate-live11:51
*** drifterza has joined #openstack-ansible11:53
*** shardy is now known as shardy_lunch12:03
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: tasks: cinder_qos: Use exact match for QoS rules  https://review.openstack.org/53249412:18
*** rgogunskiy has joined #openstack-ansible12:20
*** smatzek has joined #openstack-ansible12:20
*** brad[] has joined #openstack-ansible12:29
*** lihi has joined #openstack-ansible12:29
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible-os_congress master: Introduce os_congress role in gerrit  https://review.openstack.org/52249112:36
*** markvoelker has joined #openstack-ansible12:39
bhujaypradiprwt: home on the glance pool permission is proper . Some hint for the permission  - { name: client.glance, key: "$(ceph-authtool --gen-print-key)", mon_cap: "allow r", osd_cap: "allow class-read object_prefix rbd_children, allow rwx pool={{ openstack_glance_pool.name }}", mode: "0600", acls: [] }12:42
bhujay*hope12:43
bhujaypradiprwt: connect  from shell using glance key and  try creating and deleting some rbd image with glance pool  to  see if the issue is with ceph or openstack12:44
*** jwitko_ has quit IRC12:46
*** shardy_lunch is now known as shardy12:47
mhaydengood morning, folks12:52
evrardjpmorning mhayden12:56
*** prasen has joined #openstack-ansible12:56
*** yifei has quit IRC12:59
*** yifei has joined #openstack-ansible13:00
*** DELL has joined #openstack-ansible13:03
*** DELL is now known as Guest6994513:04
*** Guest69945 has quit IRC13:06
*** Sadipan has joined #openstack-ansible13:09
openstackgerritPeriyasamy Palanisamy proposed openstack/openstack-ansible-os_neutron master: add ml2 config for networking bgpvpn  https://review.openstack.org/52259813:09
*** smatzek has quit IRC13:09
* mhayden is deploying master on his centos 7 environment13:10
*** markvoelker has quit IRC13:12
*** ivveh has joined #openstack-ansible13:13
*** smatzek has joined #openstack-ansible13:14
*** drifterza has quit IRC13:14
odyssey4memhayden https://media.giphy.com/media/kYkQYXkO3XyRa/giphy.gif13:17
mhaydenhaha13:18
odyssey4memhayden you have seen that all the upgrade jobs are failing, right?13:23
*** yifei1 has joined #openstack-ansible13:24
*** yifei has quit IRC13:26
*** yifei1 is now known as yifei13:26
*** woodard has joined #openstack-ansible13:26
*** shardy is now known as shardy_mtg13:28
*** sxc731 has quit IRC13:30
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Fix 'status' typo  https://review.openstack.org/53252613:30
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts stable/pike: Allow NetworkManager to work (if present)  https://review.openstack.org/53220113:31
mhaydenevrardjp / odyssey4me: if y'all could help me tidy up my mess, i'd be much obliged https://review.openstack.org/#/c/532526/13:31
*** woodard has quit IRC13:32
*** Sadipan has quit IRC13:32
*** woodard has joined #openstack-ansible13:33
odyssey4meoh dear :/ done13:33
*** bhujay has quit IRC13:33
mhaydenthanks sir13:34
*** hw_wutianwei has joined #openstack-ansible13:35
*** sbra has joined #openstack-ansible13:35
*** prasen has quit IRC13:36
mhaydenoof i botched it again13:37
mhaydeni knew quitting coffee was a bad idea13:37
evrardjpmhayden: you remember what I said about FOCUS? :p13:37
*** SerenaFeng has joined #openstack-ansible13:38
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Fix typo for wait-online service  https://review.openstack.org/53252613:38
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts stable/pike: Allow NetworkManager to work (if present)  https://review.openstack.org/53220113:39
mhaydeni guess i'm glad i'm deploying master ;)13:39
*** prasen has joined #openstack-ansible13:39
evrardjpWhat would be cool, is that we stop introducing features this week, and just fix the whole series of bug we have.13:40
evrardjplike the upgrade ones.13:40
evrardjp:)13:40
evrardjpJust saying!13:40
mhaydenhaha13:40
*** SerenaFeng has quit IRC13:40
mhaydenwell i am doing greenfield testing of master on centos this morning in the background while doing other things13:40
mhaydendo we plan to address the ansible 2.4 deprecations around "import_tasks/include_tasks" in this cycle?13:42
mhaydennot sure if i missed a discussion there13:43
*** prasen has quit IRC13:46
evrardjpwe will but that's a minor bug, right now we have so much bigger on our plate :)13:47
evrardjplike real ansible 2.4 regressions and stuff!13:47
evrardjp(working on it)13:47
evrardjpand our inventory issues :/13:47
evrardjpand our upgrade issues :/13:47
evrardjp(working on it too)13:47
evrardjpand the opened bugs!13:47
evrardjpI think there is one assigned to you with sudo13:48
evrardjplong story short: I am sure we can find you places to help if you want :D13:48
*** shardy_mtg is now known as shardy13:59
*** drifterza has joined #openstack-ansible14:00
*** armaan has quit IRC14:00
*** armaan has joined #openstack-ansible14:00
*** hw_wutianwei has quit IRC14:01
*** sxc731 has joined #openstack-ansible14:04
*** hw_wutianwei has joined #openstack-ansible14:08
*** markvoelker has joined #openstack-ansible14:10
*** cmart has joined #openstack-ansible14:11
cmartGood morning OSA. Looks like Ubuntu just released kernel patches for the "Meltdown" vulnerability. Has anyone applied these to their deployments yet, and if so, run into any issues?14:12
*** jwitko_ has joined #openstack-ansible14:19
*** smatzek has quit IRC14:22
*** smatzek has joined #openstack-ansible14:22
*** smatzek has quit IRC14:24
*** chhavi__ has joined #openstack-ansible14:34
*** lbragstad has joined #openstack-ansible14:38
*** markvoelker has quit IRC14:43
mhaydencmart: mattt just pointed me at #20 here -> https://blog.online.net/2018/01/03/important-note-about-the-security-flaw-impacting-arm-intel-hardware/14:44
*** afred312 has quit IRC14:46
*** afred312 has joined #openstack-ansible14:46
cmart"Just got the Meltdown update to kernel linux-image-4.4.0-108-generic but this doesnt boot at all. It just hangs after grub."14:46
cmartwow. oops. glad I didn't apply that one. sounds like a fix was released, 4.4.0-109?14:47
*** markvoelker has joined #openstack-ansible14:51
*** prasen has joined #openstack-ansible14:52
*** rgogunskiy has quit IRC14:56
-openstackstatus- NOTICE: Gerrit is being restarted due to slowness and to apply kernel patches14:57
*** openstackgerrit has quit IRC15:01
*** markvoelker has quit IRC15:03
*** tone_z has joined #openstack-ansible15:03
*** smatzek has joined #openstack-ansible15:05
*** smatzek_ has joined #openstack-ansible15:06
*** tone_zrt has quit IRC15:07
*** smatzek has quit IRC15:08
*** woodard has quit IRC15:08
*** dave-mccowan has joined #openstack-ansible15:09
*** dave-mcc_ has joined #openstack-ansible15:12
*** ianychoi has joined #openstack-ansible15:14
*** dave-mccowan has quit IRC15:14
*** pbandark has quit IRC15:16
*** pbandark has joined #openstack-ansible15:20
*** sxc731 has quit IRC15:25
*** bhujay has joined #openstack-ansible15:27
*** hw_wutianwei has quit IRC15:29
*** openstackgerrit has joined #openstack-ansible15:36
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Fix typo for wait-online service  https://review.openstack.org/53252615:36
*** markvoelker has joined #openstack-ansible15:37
*** germs has joined #openstack-ansible15:48
*** foutatoro has joined #openstack-ansible15:50
*** sxc731 has joined #openstack-ansible15:50
*** nshetty has quit IRC15:50
*** sbra has quit IRC16:04
*** mfisch` has joined #openstack-ansible16:06
*** mfisch has quit IRC16:10
*** woodard has joined #openstack-ansible16:10
*** woodard has quit IRC16:10
*** woodard has joined #openstack-ansible16:10
*** john51 has quit IRC16:17
*** markvoelker has quit IRC16:19
*** markvoelker has joined #openstack-ansible16:22
*** epalper has quit IRC16:23
*** john51 has joined #openstack-ansible16:23
openstackgerritMerged openstack/openstack-ansible stable/ocata: Avoid retrieving IP from a missing bridge  https://review.openstack.org/53224416:25
*** markvoelker has quit IRC16:27
*** armaan has quit IRC16:27
*** armaan has joined #openstack-ansible16:28
*** armaan has quit IRC16:28
*** armaan has joined #openstack-ansible16:28
*** chyka has joined #openstack-ansible16:31
*** prasen has quit IRC16:41
mhaydenevrardjp: i slid you a little more keepalived love https://github.com/evrardjp/ansible-keepalived/pull/5616:41
evrardjp[m]woot16:42
evrardjp[m]I can't check rn, so it will probably be for tomorrow.16:43
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: Use full paths for keepalived scripts  https://review.openstack.org/53257316:45
*** bhujay has quit IRC16:46
*** RandomTech has joined #openstack-ansible16:49
RandomTechhey, is it normal for rabbitMQ to have an error during setup-infrastructure that doesnt happen after rerunning it16:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/ansible-hardening master: tasks: auth: Pass --unrestricted to Linux Grub2 entries  https://review.openstack.org/53257416:50
hwoarangmhayden: ^ what do you think16:50
mhaydenlookin16:51
mhaydenhmm, does that work okay on other distros?16:53
mhaydeni've not tested16:53
hwoarangworks on ubuntu16:53
hwoarangyou need password to edit an entry but you can boot without a password16:53
hwoarangi believe that's the desired behavior16:53
mhaydenyeah, i'd certainly hope so16:54
mhaydensheesh16:54
mhayden:P16:54
mhayden71961 requires auth before single-user and maintenance mode booting16:55
mhaydenso i think that's sufficient16:55
hwoaranglet me check again what happens in the advanced boot entries16:56
hwoarangbut if there is a problem there that affects all distros and it's unrelated :/16:56
hwoarang*unrelated to this bug16:56
mhaydenhwoarang: btw, i backported that fix for networkmanager -> https://review.openstack.org/#/c/532201/16:58
hwoarangsounds like all entries can be booted without password so a separate fix is needed16:59
*** sbra has joined #openstack-ansible17:01
*** dave-mcc_ has quit IRC17:11
*** pcaruana has quit IRC17:11
prometheanfiredo we have more info than https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/app-ceph.html for integrating ceph into osa?17:11
logan-https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/app-config-prod-ceph.html17:11
*** pradiprwt has quit IRC17:12
prometheanfireah, thanks17:14
*** pbandark has quit IRC17:15
*** phalmos has joined #openstack-ansible17:15
mhaydenthat feeling when you fight OSA for a half hour only to release you forgot to trunk a vlan17:17
mhayden:|17:17
* mhayden razzafrazzas17:17
*** chyka has quit IRC17:18
*** chyka has joined #openstack-ansible17:19
*** dave-mccowan has joined #openstack-ansible17:19
prometheanfirelogan-: that link makes it look like osa is deploying ceph-ansible17:19
*** phalmos has quit IRC17:20
logan-that is what the example is for, yes17:20
logan-are you lookign to integrate existing ceph cluster into osa?17:20
prometheanfireyep17:20
*** Smeared_Beard has quit IRC17:20
*** mfisch` has quit IRC17:23
*** germs1 has joined #openstack-ansible17:23
*** mbuil has quit IRC17:24
*** phalmos has joined #openstack-ansible17:27
logan-prometheanfire: i don't know if there is much info about that in the docs pages, but there is a lot of example info on how to setup the ceph client vars in https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/user_variables.yml17:30
prometheanfirelogan-: true, it's just a mater of figuring out exactly which bits to flip17:33
openstackgerritAntony Messerli proposed openstack/openstack-ansible-repo_server stable/newton: Ensure the /var/log/apt-cacher-ng directory created  https://review.openstack.org/53258117:35
openstackgerritAntony Messerli proposed openstack/openstack-ansible-repo_server stable/ocata: Ensure the /var/log/apt-cacher-ng directory created  https://review.openstack.org/53258217:35
openstackgerritAntony Messerli proposed openstack/openstack-ansible-repo_server stable/pike: Ensure the /var/log/apt-cacher-ng directory created  https://review.openstack.org/53258317:36
*** cjloader_ has joined #openstack-ansible17:36
*** zerick has joined #openstack-ansible17:39
cloudnullafternoons !17:40
*** zerick has quit IRC17:41
*** zerick has joined #openstack-ansible17:42
*** zerick has quit IRC17:45
*** zerick has joined #openstack-ansible17:46
*** gkadam has quit IRC17:47
SamYapleo/17:47
*** armaan has quit IRC17:47
*** armaan has joined #openstack-ansible17:48
*** drifterza has quit IRC17:50
spotzHey SamYaple17:50
spotzhey cloudnull17:51
cloudnullyo yo17:54
*** sxc731 has quit IRC18:10
*** RandomTech has quit IRC18:18
*** openstacking_123 has joined #openstack-ansible18:19
DimGRyo18:23
*** openstack has joined #openstack-ansible18:32
*** ChanServ sets mode: +o openstack18:32
*** sxc731 has joined #openstack-ansible18:32
*** SmearedBeard has joined #openstack-ansible18:44
*** stuartgr has quit IRC18:47
*** SmearedBeard has quit IRC18:52
*** TxGirlGeek has joined #openstack-ansible18:55
mhaydeni deployed master and xtrabackup's xinetd listener isn't configured to allow traffic from my haproxy node :/19:01
mhaydenso the health checks fail each time19:01
*** poopcat has joined #openstack-ansible19:06
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/pike: Allow NetworkManager to work (if present)  https://review.openstack.org/53220119:07
*** shardy has quit IRC19:08
sxc731Hi team, I'm trying to figure out why my OSA-deployed Keystone performs poorly... I've found that the uWSGI processes complain with "!!! no internal routing support, rebuild with pcre support !!!"19:08
sxc731(this is 16.0.2 on Xenial)19:09
sxc731Is this serious?  Should I raise a defect? According to https://stackoverflow.com/a/22645915/645016 it's just a matter of compiling uWSGI after installing libpcre3-dev, which seems to be missing from the repo containers19:10
*** poopcat1 has joined #openstack-ansible19:10
*** SmearedBeard has joined #openstack-ansible19:11
spotzsxc731: If you would that'd be nice, we can go over it at the next triage19:12
*** poopcat has quit IRC19:13
*** poopcat1 is now known as poopcat19:14
openstackgerritMajor Hayden proposed openstack/openstack-ansible-haproxy_server master: Make haproxy stats refresh interval configurable  https://review.openstack.org/53260719:14
sxc731spotz: thank you!  Will do.  Alas, I've ruled this out as the cause of my poor KS performance after installing a "fixed" uWSGI (getting ~150ms token validation with some weird degenerate cases at 5000+ms, on lightly loaded metal).  I'm trying to profile the processes but uWSGI doesn't seem to make this easy.  Any idea as to what could cause this poor performance?19:15
spotzsxc731: I haven't done much with metal in a long time. i do know evrardjp and mauybe cloudnull were doing somoe work on it but not sure what aspect19:16
*** SmearedBeard has quit IRC19:16
sxc731spotz: sorry for confusion KS is running in containers but the containers are on decent metal as opposed to some VMs (which should yield decent perf)19:17
cloudnulllbragstad: ^19:17
cloudnullsxc731: I've not seen anything in particular that would create a slow performance situation with Keystone and uWSGI19:18
cloudnullis it only token validation ?19:19
lbragstadsxc731: o/19:19
lbragstadit is strange that you're seeing this with uwsgi only (if that is the case), were any other configuration changes made?19:20
sxc731thank you guys!  I haven't done any tweaking and uWSGI was the default deployment from what I can tell?19:20
sxc731Basically deployed OpenStack perf is crap all round and I figured if token validation doesn't perform, then pretty much nothing will, right?  Upward from 150ms for each token validation oin lightly loaded HW can't be right, can it?19:21
sxc731I'm using this from Dolph Matthews to benchmark KS: https://gist.github.com/dolph/02c6d37f49596b3f429819:22
lbragstadunfortunately, it depends on your configuration... do you know if you've enabled caching (osa might by default but I haven't checked in a while)19:22
cloudnullsxc731: I've seens act like that when caching was busted?19:22
cloudnullha.19:22
sxc731Yes, caching is enabled by default and I've left it at that19:22
* cloudnull deffers to lbragstad19:23
cloudnull:)19:23
sxc731lbragstad: is there an easy way to figure out if memcached is actually working as it should?19:23
sxc731Agree, cache was my first thought19:24
lbragstadsxc731: i believe oslo.cache exposes debug logging (which is super noisy) but it should tell you in the service logs if there are hits or misses in the cache19:24
lbragstadi'd also just double check the caching configuration if you know there is traffic going to memcache19:25
cloudnullsxc731: is debug enabled and are you seeing a lot of IO on your hosts?19:25
lbragstadfor one reason or another, we offer a million knobs for caching in keystone19:25
sxc731lbragstad: cool, I suppose enabling global "debug = True" in KS's main conf file isn't sufficient to get this?  I'll see if I can figure how to turn it on19:25
lbragstadlet me see if i can find the oslo.cache bits19:25
cloudnulli've seen a notable slowdown in openstack when debug is enabled.19:26
lbragstadhttps://github.com/openstack/oslo.cache/blob/master/oslo_cache/_opts.py#L69-L7519:26
cloudnullnot to say it wasn't slow before debug was enabled.19:26
lbragstad^ that probably won't fix your performance problems, but it could help diagnose if your cache backend is seeing traffic19:26
cloudnullbut it can exacerbate the issue.19:27
lbragstadkeystone allows you to toggle caching *per* keystone sub-system, which gives you the flexibility to cache identity data but not cache catalog data (or whatever your case is)19:27
sxc731Thanks for the suggestion: the issue first appeared when debugging was turned off.  Enabling it to diagnose didn't actually make a noticeable difference; suggesting my perf was cr*p to begin with...19:27
lbragstadI usually recommend caching everything though19:28
lbragstadthis is a pike deployment?19:28
sxc731Yes, I did study the caching config; seems that so long as the main toggle is on then all subsystems not explicitly disabled will be on too.  But I'm sure you knew this ;-)19:28
sxc731It is Pike yes19:28
lbragstadok - yep, that sounds right19:28
lbragstadso long as your cache backend is seeing activity you should be caching everything19:29
*** rmcall has joined #openstack-ansible19:30
sxc731OK let me try to see if I can prove caching is working using your debug toggle19:31
lbragstadtoken validation is also strictly made up of read-only database queries (if caching is disabled)19:31
sxc731FWIW this is keystone.conf (generated by OSA with just some minor tweak by myself): http://paste.openstack.org/show/642486/19:32
spotzhey lbragstad:)19:32
cloudnullsxc731: also which token type are you using? fernet (the default)?19:33
sxc731My tweak was adding caching = true for [catalog] and  [role] (before I understood they were on by default).  Perhaps the longer cache_time for [catalog] would be a nice enhancement for OSA?19:33
sxc731cloudnull: yep, I have actually not tweaked any of the config, so Fernet it is19:33
cloudnullcool19:33
cloudnulljust checking19:33
lbragstadspotz: o/19:33
lbragstadis this argument the same across all keystone containers? backend_argument = url:172.20.120.247,172.20.68.222,172.20.112.234:1121119:34
sxc731At some point I was wondering if this could be caused by invalid certs or not running the KS internal & admin endpoints with TLS19:34
sxc731lbragstad: yes it's the same19:34
lbragstadok - cool19:34
sxc731lbragstad: OSA-generated; I've compared them19:35
lbragstadsxc731: do you have an idea of what the data looks like in keystone (how many users, projects, role assignments, etc..)19:35
*** SmearedBeard has joined #openstack-ansible19:36
sxc731lbragstad: it's a pretty new default install with just a handful of users/projects.  I did set Tempest loose on it and it created some addtn'l shtuff19:36
lbragstadack - so nothing out of the ordinary there19:37
sxc731perf was (subjectively, as in, as witnessed in Horizon) crap straight out of the box and I've been trying to figure this out for a good few weeks now :(19:37
sxc731lbragstad: can you confirm that ~150ms time to validate tokens is substandard?  Dolph's gist'ed b-m results have order of 5-6ms!19:39
*** mpranjic has joined #openstack-ansible19:39
sxc731And also what could be causing these 1-2/100 spikes to over 5s to validate?  This is single-threaded benchmarking on a system that's not commissioned yet, ie: no background load.19:40
lbragstadwhen we did the benchmarking, we found it to be totally subjective based on the environment19:40
*** SmearedBeard has quit IRC19:41
sxc731Here's a typical run with 100 requests:19:41
sxc731Percentage of the requests served within a certain time (ms):19:41
sxc73150%    16319:41
sxc73166%    57919:41
sxc73175%    58819:41
* lbragstad digs up performance bot results19:41
sxc73180%    59619:41
sxc73190%   557219:41
sxc73195%   559219:41
sxc73198%   559219:41
sxc73199%   559219:41
sxc731BTW, I'm running the b-m's directly against the KS back-ends as opposed to going through haproxy (which didn't make a difference but I wanted to rule out haproxy shenanigans)19:42
lbragstadsure19:43
lbragstadi did have some on-metal performance infrastructure so that we could run performance benchmarks on patches in review19:43
*** rmcall has quit IRC19:43
lbragstadand i dumped all the results https://github.com/lbragstad/keystone-performance/tree/master/results19:43
sxc731OK just looking at one of those (https://github.com/lbragstad/keystone-performance/blob/master/results/03ba1324c2/20161122103227/benchmark-results.txt) they don't seem too far off mine; albeit you don't see those degenerate cases at 5s!19:45
lbragstadright19:45
lbragstadsxc731: if you deploy keystone with a higher affinity do you see differences?19:45
sxc731I noticed the validation API spits back the complete catalog (which can be quite big).  Presumably this isn't the case everytime an OS service validates through keystonemiddleware?19:46
lbragstadthere is a switch in keystonemiddleware for that i believe19:46
lbragstadbut that catalog is also cache-able19:46
*** SmearedBeard has joined #openstack-ansible19:46
*** rmcall has joined #openstack-ansible19:46
sxc731Sure, and it should be cached if my cache is working well... that's still feels like quite a bit of data that flows back to the client (~7k in my case)19:47
sxc731What's the affinity?19:48
sxc731(sorry, not familiar with that one)19:48
lbragstadfor reference - dolph and i benchmarked a PoC of fernet in a global deployment (~36 app servers backed by ~18 galera nodes) and we saw anywhere from 110 - 160 ms response times for 120k RPM19:48
lbragstadsxc731: affinity is just the number of containers19:48
sxc731Ah... well I'm running a single b-m client against a single container so presumably this wouldn't make a huge difference19:49
lbragstadthat was about the hardest we were able to push it, but certainly noticed it scaled linearly19:49
sxc731Presumably the disk perf only matters if not hitting cache?  Mine are on SSDs anyway...19:50
lbragstadcorrect19:50
lbragstadin which case, you might notice when the cache expires19:50
*** cshen_ has joined #openstack-ansible19:50
sxc731But if I summarize what you're saying, order of 150ms is ok; therefore I must look elsewhere as to what's causing every single Horizon page to take order of 10s to load?19:51
*** zerick has quit IRC19:51
*** zerick_ has joined #openstack-ansible19:51
lbragstadsxc731: yeah19:52
*** SmearedBeard has quit IRC19:52
ashakcan anyone help with a bit of coaching? I've been looking at why a bunch of logging doesn't work on CentOS, particularly syslog and haproxy, but I think it affects a few of teh other default logs. I know how to fix it but i'm a bit conflicted on how I should fix it.19:52
lbragstadsxc731: but - we do have oslo.cache working into keystonemiddleware and it can be setup the same way you do caching in keystone19:52
lbragstadwhich means you should only see round-trip performance hits once from the service19:52
lbragstadand that should help19:53
sxc731lbragstad: now that's quite interesting. Does this need to be set in each service's config?19:53
lbragstadthe options would be exposed through keystonemiddleware via oslo.cache, so yeah..19:53
lbragstadcorrect, it would need to be defined at the service19:53
lbragstadhttps://github.com/openstack/keystonemiddleware/commit/9d8e2836fe7fca186e0380d8a532540ff5cc521519:54
sxc731Do you happen to know if this is enabled by default with OSA deployments?  I don't suppose the Horizon performance I'm seeing is "works as expected" for a pretty much out-of-box OSA deployment using Fernet?19:55
sxc731I'm coming from a Fuel-deployed Kilo deployment (UUID tokens) on weaker hardware that performs significantly better19:55
lbragstadsxc731: yeah - the last time i deployed osa i didn't notice that19:56
lbragstadit could be something specific to horizon, too?19:56
sxc731lbragstad: I agree.  Seems like I've been barking up the wrong tree (KS) all along; it was dolph's gist that lead me to believe that my ~150ms validations were off (though these 5s degenerate cases are also intriguing but don't explain my bad overall perf of itself).  Thank you _very_ much for all the help, I really appreciate it!19:58
spotzhehe blaiming dolph:)19:59
sxc731lbragstad: With your pointers, I'm still going to check that the cache works properly and that the inter-service cache is properly enabled19:59
lbragstadsxc731: anytime... i mean, there are probably things we can do in keystone19:59
lbragstadand yeah - i'd start by confirming cache traffic20:00
sxc731spotz: really didn't mean to blame anyone in the amazing community!20:00
spotzsxc731: We love Dolph no worries:)20:01
lbragstadsxc731: IME if you have 150 ms token validation responses, you shouldn't be having 10s page load issues in horizon20:01
sxc731lbragstad: one last q: if I want to hunt down those 5s degenerate cases, how would you go about profiling KS running under uWSGI; seems really hard...  I've played with --profiler=pycall but this just outputs counts of call trees; not particularly useful20:01
lbragstaddolph had a good writeup using repoze20:02
lbragstadi've used plop in the past20:02
lbragstadhttps://github.com/bdarnell/plop20:02
sxc731OK ta, I'll google those20:02
lbragstadhttp://blog.dolphm.com/performance-profiling-openstack-services-with-repoze-profile/20:03
sxc731lbragstad: brilliant, thank you again! (and sorry for hijacking the OSA channel with all these KS q's)20:03
lbragstadsxc731: no worries - come find me in #openstack-keystone if you have any more issues20:04
sxc731lbragstad: will do, thank you.  I'll also raise the defect on "!!! no internal routing support, rebuild with pcre support !!!"; this can't be improving things; though I've confirmed this isn't the cause of my issue20:05
*** cshen_ has quit IRC20:05
lbragstadsounds good20:05
*** hybridpollo has joined #openstack-ansible20:10
mhaydensxc731: which log had that message?20:11
*** cshen_ has joined #openstack-ansible20:17
sxc731mhayden: sorry was chatting with lbragstad on the KS channel.  The msg appears in both keystone-wsgi-admin.log and keystone-wsgi-public.log (KS container obviously). This is on Xenial.  Believe the fix is here: https://stackoverflow.com/a/22645915/645016 (ie: make sure libpcre3-dev is installed when building the wheels on the repo container?)20:23
mhaydenah okay, thanks for that20:23
*** rmcall has quit IRC20:25
*** dave-mccowan has quit IRC20:27
*** dave-mccowan has joined #openstack-ansible20:38
sxc731uWSGI PCRE issue: https://bugs.launchpad.net/openstack-ansible/+bug/174253820:43
openstackLaunchpad bug 1742538 in openstack-ansible "OSA-deployed uWSGI used by Keystone warns: "!!! no internal routing support, rebuild with pcre support !!!"" [Undecided,New]20:43
SamYaplesxc731: thats not an issue, just a warning20:49
*** chyka has quit IRC20:49
SamYaplewith or without pcre isnt changing anything20:50
*** chyka has joined #openstack-ansible20:50
SamYaple(not for the way we use uwsgi that is)20:50
sxc731SamYaple: sure, I daid as much in the ticket.  So Keystone doesn't rely in "internal routing"?20:50
sxc731^in^on20:51
SamYaplesxc731: forgive me, its been a while since i reeaserched it, but because of the way we run things (single service and all) the pcre stuff wouldnt get triggered anyway20:51
SamYaplethat was my takeaway at the time20:51
SamYaplethat said, i still build with libpcre3-dev20:52
SamYaplenot discouraging adding it20:52
sxc731SamYaple: OK ta.  Indeed I suppose if there's a reasonably cheap way to fix the issue maybe it should be done (as a low prio) just to avoid causing false alarm?20:52
*** Jeffrey4l has quit IRC20:52
SamYapleagreed 100%20:53
sxc731Good stuff.  I'm sure there's like a gazillion more pressing issues...20:54
SamYapleyea but none with such easy solutions :P20:55
sxc731Well then, cool low hanging fruit.  I have to admit I looked into how to fix this in repo_build (assuming that's even the right place) and couldn't figure it out; so at least for me, not trivial...   Ended up installing uWSGI from PyPi to test if it made a difference...20:57
*** chyka has quit IRC20:59
*** chyka_ has joined #openstack-ansible20:59
*** cshen_ has quit IRC21:00
*** Jeffrey4l has joined #openstack-ansible21:03
SamYaplesxc731: you just need to add the dev libs to the repo build package list21:07
sxc731SamYaple: ok ta. I must be a little thick because I still can't seem to fins this and I'm sure it's obvious... but presumably this should be in there by default, correct?21:11
SamYapleyea21:12
SamYapleprobably21:12
*** kylek3h has joined #openstack-ansible21:16
*** smatzek_ has quit IRC21:19
*** smatzek has joined #openstack-ansible21:20
*** smatzek_ has joined #openstack-ansible21:22
*** smatzek has quit IRC21:24
*** smatzek_ has quit IRC21:26
*** sbra has quit IRC21:27
*** dave-mccowan has quit IRC21:29
*** sxc731 has quit IRC21:31
*** kylek3h has quit IRC21:38
*** chhavi__ has quit IRC21:41
mhaydencloudnull: i like neutron agents on the hosts, but dang, i'm in SELinux hell with that right now ;)21:41
jrosserlooking at a deploy log i see that pretty much every non skipped task takes a minimum of ~6 seconds, regardless of how trivial it is21:41
*** pcaruana has joined #openstack-ansible21:42
jrosseris this normal? that minimum time seems to entirely dominate the deploy time of ~5hours21:42
*** pcaruana has quit IRC21:43
*** armaan has quit IRC21:48
*** woodard_ has joined #openstack-ansible21:50
*** woodard has quit IRC21:50
*** woodard_ has quit IRC21:51
*** threestrands_ has joined #openstack-ansible21:52
*** threestrands_ has quit IRC21:53
*** threestrands_ has joined #openstack-ansible21:53
*** threestrands has quit IRC21:54
*** chyka_ has quit IRC21:57
*** armaan has joined #openstack-ansible22:00
*** smatzek has joined #openstack-ansible22:00
*** masber has joined #openstack-ansible22:02
*** cjloader_ has quit IRC22:03
*** smatzek has quit IRC22:05
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_neutron master: Add SELinux policies for bare metal agents  https://review.openstack.org/53264622:05
mhaydencloudnull: ^^22:05
mhaydenantonym: on those glance backports, did you use the cherry pick button in gerrit?22:17
-openstackstatus- NOTICE: The zuul system is being restarted to apply security updates and will be offline for several minutes. It will be restarted and changes re-equeued; changes approved during the downtime will need to be rechecked or re-approved.22:22
*** afred312 has quit IRC22:27
*** threestrands_ has quit IRC22:41
*** threestrands has joined #openstack-ansible22:55
*** afred312 has joined #openstack-ansible22:58
*** afred312 has quit IRC23:03
*** afred312 has joined #openstack-ansible23:07
*** rmcall has joined #openstack-ansible23:09
*** threestrands has quit IRC23:11
*** threestrands has joined #openstack-ansible23:20
*** openstacking_123 has quit IRC23:39
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-tests stable/ocata: Skip and echo out gzip log compression error  https://review.openstack.org/53269223:40
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-tests stable/newton: Skip and echo out gzip log compression error  https://review.openstack.org/53269323:41
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-tests stable/pike: Skip and echo out gzip log compression error  https://review.openstack.org/53269423:43
*** cjloader_ has joined #openstack-ansible23:46
cloudnulljrosser: yup. we've an execution plugin that will evaluate tasks and not even load them if we know for certain nothing will be executed for this very issue.23:47
*** phalmos has quit IRC23:47
cloudnullhowever we can't fully evaluate all tasks so the plugin is quite conservative23:48
*** cjloader_ has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!