Monday, 2018-06-25

*** tux_ has joined #openstack-ansible00:01
*** jmnk has joined #openstack-ansible00:07
*** jmnk has quit IRC00:14
*** spsurya has joined #openstack-ansible00:16
*** vnogin has joined #openstack-ansible00:34
*** vnogin has quit IRC00:38
*** aspiers has quit IRC00:41
openstackgerritMerged openstack/openstack-ansible master: Add LXC and PIP mirrors  https://review.openstack.org/57688500:53
*** gregoryo has joined #openstack-ansible01:13
*** openstack has joined #openstack-ansible01:20
*** ChanServ sets mode: +o openstack01:20
*** harryhuang_ has joined #openstack-ansible01:58
openstackgerritzhongshengping proposed openstack/openstack-ansible-os_magnum master: Deprecate auth_uri option  https://review.openstack.org/55838002:07
*** dxiri has joined #openstack-ansible02:21
*** dxiri has quit IRC02:27
*** vurtz has quit IRC02:40
*** jmnk has joined #openstack-ansible03:05
openstackgerritMerged openstack/openstack-ansible master: Fix loop variable name for nested loop  https://review.openstack.org/57243303:11
*** dxiri has joined #openstack-ansible03:20
*** dxiri has quit IRC03:26
*** jmnk has quit IRC03:32
*** dxiri has joined #openstack-ansible03:35
*** dxiri has quit IRC03:41
*** dxiri has joined #openstack-ansible03:41
*** udesale has joined #openstack-ansible03:46
*** jmnk has joined #openstack-ansible04:02
*** dxiri has quit IRC04:08
*** jmnk has quit IRC04:09
*** jmnk has joined #openstack-ansible04:46
*** jmnk has quit IRC04:56
*** cshen has joined #openstack-ansible04:58
*** cshen has quit IRC05:02
*** tux_ has quit IRC05:05
*** nicolasbock has joined #openstack-ansible05:21
*** aspiers has joined #openstack-ansible05:23
*** armaan has joined #openstack-ansible05:29
*** hamzy_ has quit IRC05:36
*** Bhujay has joined #openstack-ansible05:37
*** hamzy has joined #openstack-ansible05:38
*** hamzy has quit IRC05:40
*** hamzy has joined #openstack-ansible05:42
*** Bhujay has quit IRC05:44
*** Bhujay has joined #openstack-ansible05:44
*** Bhujay has quit IRC05:53
*** yolanda has joined #openstack-ansible05:53
*** DanyC has joined #openstack-ansible05:54
*** DanyC_ has joined #openstack-ansible05:55
*** DanyC has quit IRC05:58
*** DanyC_ has quit IRC06:02
*** DanyC has joined #openstack-ansible06:02
*** DanyC has quit IRC06:17
*** lvdombrkr has joined #openstack-ansible06:17
*** cshen has joined #openstack-ansible06:20
*** jbadiapa has joined #openstack-ansible06:35
*** mbuil has joined #openstack-ansible06:36
*** lvdombrkr89 has joined #openstack-ansible06:37
*** lvdombrkr has quit IRC06:39
*** Bhujay has joined #openstack-ansible06:41
*** armaan has quit IRC07:05
*** armaan has joined #openstack-ansible07:06
*** pcaruana has joined #openstack-ansible07:07
*** DanyC has joined #openstack-ansible07:08
*** armaan has quit IRC07:12
*** epalper has joined #openstack-ansible07:23
openstackgerritNguyen Hai proposed openstack/openstack-ansible master: Trivial: Update pypi url to new url  https://review.openstack.org/56338507:23
*** DanyC has quit IRC07:40
*** luksky has joined #openstack-ansible07:46
*** armaan has joined #openstack-ansible07:51
*** radeks has quit IRC08:00
hwoaranggood morning08:20
hwoarangcores i'd appreciate another +2 on https://review.openstack.org/#/c/577330/ since it fixes the connection plugin for ansible 2.508:20
hwoarangjmccrory: ^^ great work thank you very much08:20
*** armaan has quit IRC08:21
*** shardy has joined #openstack-ansible08:24
*** gregoryo has quit IRC08:26
andymccrnice work jmccrory!08:29
andymccrlgtm08:29
*** dxiri has joined #openstack-ansible08:34
odyssey4meo/ all back from holiday08:35
MiougeAnybody doing letsencrypt with OSA? I found this article on google: http://openpower.ic.unicamp.br/blog/integrating-openstack-ansible-with-lets-encrypt.html but kind of hacky with scp etc… I suppose that the difficulty is to share the state around the LE challenge with multiple haproxy nodes08:37
*** dxiri has quit IRC08:38
*** upgrader6 has joined #openstack-ansible08:50
*** upgrader6 has left #openstack-ansible08:54
*** vnogin has joined #openstack-ansible08:57
*** electrofelix has joined #openstack-ansible08:58
*** upgrader6 has joined #openstack-ansible09:11
*** vnogin has quit IRC09:14
*** brad[] has joined #openstack-ansible09:15
upgrader6Hi! Can anyone suggest how to troubleshoot cinder? I have upgraded from Pike to Queens and now when creating volumes on ceph or LVM volumes hang on creating state09:18
upgrader6have switched cinder.conf to debug= True, but I see no errors09:18
*** vnogin has joined #openstack-ansible09:28
hwoarangwelcome back odyssey4me09:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican master: Move database creation into role  https://review.openstack.org/57484709:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican master: Move test-vars into overrides file  https://review.openstack.org/57484809:37
*** cshen has quit IRC09:44
*** cshen has joined #openstack-ansible09:45
hwoaranganyone up for an easy +2 https://review.openstack.org/#/c/575455/ ? :)09:47
*** cshen has quit IRC10:03
odyssey4mehwoarang: done but it looks like that patch is based on another unapproved one10:09
hwoarangoh true10:11
*** tux_ has joined #openstack-ansible10:12
*** tux_ has quit IRC10:17
odyssey4mejrosser: about those queens patches for the links removal - I've not yet managed to figure out why the install fails when trying to install mysql-python... it's odd... I thought it might be a DNS issue (due to trying to use ipv6) but it turns out that doesn't seem to be the case... if you've any ideas, please share10:18
odyssey4mein master we don't install that package from pip any more, so those are all passed and merged...10:19
jrosserodyssey4me: i will do - theres been an awful lot fixed in the last week so i figured it was worth another look at these10:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: Update os_neutron role SHA for OVS fix  https://review.openstack.org/57365010:26
upgrader6After upgrade Pike->Queens(ansible playbooks run successfully) cannot create ceph,lvm volumes - hanging on creating state. No errors in cinder logs. Any hint how to troubleshoot is really appreciated.10:34
upgrader6I just see successfull cinder connects to ceph: DEBUG cinder.volume.drivers.rbd [req-f0f74c0e-3c2f-4889-9e7e-374bb1bc91a4 - - - - -] connecting to ceph (timeout=-1). _do_conn /openstack/venvs/cinder-17.0.5/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:33010:37
*** vnogin has quit IRC10:41
jrosserupgrader6: the only thing i have seen is the cinder volumes services being down after a deployment with multiple backends10:42
*** vnogin has joined #openstack-ansible10:42
jrosserrestarting those services by hand got things going again10:42
upgrader6jrosser, thanks, have restarted cinder-colume on one host, but will check if there are any more related services to restart10:44
*** shardy has quit IRC10:44
ivvehey guys10:50
ivvei have a question regarding l3_agent.ini in two different setups10:51
ivveone is configured with send_arp_for_ha = 310:51
ivveother doesn't have this, which means its diabled?10:51
*** lvdombrkr89 has quit IRC10:51
ivvedisabled*10:51
upgrader6Is it safe to run openstack-ansible queens install playbooks after upgrade playbooks have finished? In hope to resolve those cinder issues10:57
*** vnogin has quit IRC11:01
*** vnogin has joined #openstack-ansible11:08
jrosserodyssey4me: this feels like a silly quesstion but why does repo_build need galera_client? https://github.com/openstack/openstack-ansible-repo_build/blob/81dc44903a00fd11f79ef2444d0aadec63ad4fbb/meta/main.yml#L4411:09
odyssey4mejrosser: it's building wheels which require those binaries there to fulfill the sql client needs11:10
odyssey4meit's basically just to ensure that it has the sql client binaries which are used in the wheel build for those wheels11:10
*** maybebuggy has joined #openstack-ansible11:11
*** nsingh has quit IRC11:19
*** lvdombrkr89 has joined #openstack-ansible11:20
*** lvdombrkr has joined #openstack-ansible11:27
*** udesale has quit IRC11:27
*** tosky has joined #openstack-ansible11:28
*** lvdombrkr89 has quit IRC11:28
*** vnogin has quit IRC11:30
*** koukou has joined #openstack-ansible11:40
*** vnogin has joined #openstack-ansible11:40
openstackgerritMerged openstack/openstack-ansible-os_monasca-agent master: Add release note link in README  https://review.openstack.org/57744711:44
openstackgerritMerged openstack/openstack-ansible-os_zun master: Add release note link in README  https://review.openstack.org/57745311:45
openstackgerritMerged openstack/openstack-ansible-os_molteniron master: Add release note link in README  https://review.openstack.org/57744611:45
openstackgerritMerged openstack/openstack-ansible-os_monasca-ui master: Add release note link in README  https://review.openstack.org/57745011:47
*** cshen has joined #openstack-ansible11:49
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Add release note link in README  https://review.openstack.org/57745111:50
upgrader6confused, on volume create cinder-api returns success, cinder-volume has no sign of doing something at all, both have debug=True. Volume status remain creating. Where to those calls go? Some rabbitmq issue? Any hints?11:59
upgrader6how to enable debugging so that I can see real ceph/lvm commands/api calls issued for backend? Currently api calls just vanish12:01
openstackgerritMerged openstack/openstack-ansible-os_blazar master: Add project home link in README  https://review.openstack.org/57744212:02
openstackgerritMerged openstack/openstack-ansible-ops master: Switch to using project-templates  https://review.openstack.org/57754612:05
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/56753012:07
hwoarangodyssey4me: do you have 10 min for a question about os_nova? ansible is doing something sneaky and i need another pair of eyes12:08
*** tux_ has joined #openstack-ansible12:09
mnaserhttps://review.openstack.org/#/c/577322/12:10
mnaserThis unblocks centos 7 please12:10
*** vnogin has quit IRC12:11
*** shardy has joined #openstack-ansible12:12
hwoarangmnaser: what happens if all the repos share the same file (eg rdo) ?12:13
*** tux_ has quit IRC12:13
hwoarangi guess it's ok to define multiple repos in a single file?12:14
mnaserhwoarang: nothing. They’re just all listed in their own section similar to how CentOS-Base.repo usually has base and updates etc12:14
hwoarangok12:14
mnaserYup. It is12:14
openstackgerritzhulingjie proposed openstack/openstack-ansible-os_tempest master: Add description into README like others  https://review.openstack.org/57744112:18
*** ansmith has quit IRC12:18
*** harryhuang_ has quit IRC12:23
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/56753012:25
*** upgrader6 has quit IRC12:28
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/56753012:30
*** vnogin has joined #openstack-ansible12:38
mnaserhwoarang: thanks for your review. Addressed your comments.12:41
hwoarangmnaser: ok maybe i didn't express myself correctly. what i meant was that the extras repo will be added to the host too. not just containers12:43
*** vnogin has quit IRC12:46
*** evrardjp_ has joined #openstack-ansible12:47
*** vnogin has joined #openstack-ansible12:48
*** evrardjp has quit IRC12:50
*** rmart04 has joined #openstack-ansible12:52
*** evrardjp_ has quit IRC12:53
*** prasen has joined #openstack-ansible12:58
hwoaranganother easy +2 up for grabs https://review.openstack.org/#/c/575708/13:00
hwoarangnicolasbock: hello :) what is not working in leap42.3?13:03
*** aludwar has joined #openstack-ansible13:03
hwoarang42.3 is a gate on every role + integrated repo so it works as far as i know13:03
hwoarangif you have a specific problem tell me or file a bug so we can look at it :)13:03
*** vakuznet has joined #openstack-ansible13:04
nicolasbockHi hwoarang !13:04
nicolasbockOk, I'll try it again and see what error I get exactly13:05
mnaserhwoarang: right that’s correct as well!  However centos ships with extras enabled so it’ll be largely a nooo13:05
mnaserNoop***13:05
hwoaranghmm13:06
hwoarangi recall that we have an rsync in place to copy /etc/yum.d/* from host to containers13:06
hwoarangmaybe that's changed13:07
hwoaranganyway it's good to be explicit13:07
*** evrardjp_ has joined #openstack-ansible13:07
*** prasen has quit IRC13:08
*** KeithMnemonic has joined #openstack-ansible13:08
mnaserhwoarang: by the way, uwsgi is in rdo and you’ll notice centos jobs started passing for distros :)13:08
hwoarangnice :)13:08
odyssey4memnaser: excellent, thanks for getting that done!13:11
*** evrardjp_ has quit IRC13:16
*** evrardjp_ has joined #openstack-ansible13:17
*** ansmith has joined #openstack-ansible13:17
*** luksky has quit IRC13:19
openstackgerritMerged openstack/openstack-ansible-galera_client master: Fix the home-page link  https://review.openstack.org/57760213:19
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Add release note link in README  https://review.openstack.org/57744413:21
openstackgerritMerged openstack/openstack-ansible-os_searchlight master: Fix the ansible role aim to searchlight  https://review.openstack.org/57743613:22
mnaserAnd mariadb mirrors are up on infra13:22
mnaserSo I’ll be updating that which will eliminate one more source of external failures13:23
openstackgerritMerged openstack/openstack-ansible master: Bump setuptool to 39.2.0  https://review.openstack.org/57593513:28
openstackgerritMerged openstack/openstack-ansible master: Trivial: Update pypi url to new url  https://review.openstack.org/56338513:28
*** throwsb1 has joined #openstack-ansible13:30
*** rmart04 has quit IRC13:46
*** vnogin has quit IRC13:47
*** maybebuggy has quit IRC13:48
*** maybebuggy has joined #openstack-ansible13:50
*** vnogin has joined #openstack-ansible13:51
*** Brew has joined #openstack-ansible13:54
*** cjloader has joined #openstack-ansible13:57
*** tux_ has joined #openstack-ansible14:00
*** lvdombrkr89 has joined #openstack-ansible14:01
*** lvdombrkr has quit IRC14:04
*** tux__ has joined #openstack-ansible14:08
*** dxiri has joined #openstack-ansible14:10
*** tux_ has quit IRC14:11
mnaserodyssey4me: https://review.openstack.org/#/c/577322/ if you don't mind helping unbreak centos :)14:15
*** gillesMo has joined #openstack-ansible14:16
odyssey4memnaser: done - was just waiting to see the response from hwoarang's question14:17
mnaserhwoarang: https://review.openstack.org/#/c/576883/6 addressed your comment and this should unblock os_cinder14:19
mnaserandymccr: ^ needed some extra work so i can borrow your +2 back :)14:20
hwoarangmnaser: but openstackclient depends on cinder so the package should pull it in14:20
andymccrcool will take a look14:20
mnaserhwoarang: which might change at some point later14:20
mnaserso i think it's safer to have both14:20
hwoarangbut if that changes, that means you dont need cinderclient anymore14:20
gillesMoHello stackers. I'm planning a new deployment with Ceph and I wonder if I should deploy ceph appart, with ceph-ansible and just connect openstack to it with openstack-ansible, or, deploy Ceph with openstack-ansible... It feels like I could be more free with a separate deployment like using Mimic with bluestore on LVM. I don't know what's possible w14:20
gillesMoith openstack-ansible. As a drawback, without OSA, I will not have ceph mons in LXC containers...14:20
hwoarangand you would pull it in for no reason14:20
mnaserhwoarang: can we cross that bridge when we get there or can i do a follow up? i've iterated like 8 times on that patch14:21
hwoarangwell we will never get there because nobody will notice that the depnedency is not needed anymore :)14:22
hwoarangbut ok will comment and if others dont object i will switch to +214:22
mnaseri'll push up a patch right now14:22
mnaseri just don't want to have this go through check/gate and take more time to merge14:22
hwoarangit's ok cinder has been broken for weeks now ;p14:23
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop python-cinderclient usage  https://review.openstack.org/57783814:24
mnaserhwoarang: ^14:24
mnaser(i also rebased a bunch of os_cinder patches over the weekend to make sure they were passing and ready for review)14:25
mnaserso it would have affected like 5 patches :)14:25
*** jmnk has joined #openstack-ansible14:25
mnaserdoing this is a lot less touch14:25
hwoarangok thanks14:28
tux__cloudnull: openstack successfully built and ready14:29
tux__thanks for your help man!!!14:29
*** jmnk has quit IRC14:30
tux__I think related networkd issue i am sure it was systemd 2.19 bug14:30
tux__Just wondering what version you are using on your environment ?14:30
tux__cloudnull: i think we should update official document with warning or soft notes, that if you using CentOS 7.x then systemd-networkd style networking won't work with LXC14:33
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Add CentOS extras repository inside containers  https://review.openstack.org/57732214:35
*** chyka has joined #openstack-ansible14:36
tux__cloudnull: i have report this bug https://bugs.launchpad.net/openstack-ansible/+bug/177853714:40
openstackLaunchpad bug 1778537 in openstack-ansible "LXC bridge issue with networkd on CentOS 7" [Undecided,New]14:40
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/56753014:43
*** udesale has joined #openstack-ansible14:45
*** aludwar has quit IRC14:46
*** FrankZhang has joined #openstack-ansible14:50
*** cshen has quit IRC14:52
*** vnogin has quit IRC15:01
*** cmart has joined #openstack-ansible15:04
*** vnogin has joined #openstack-ansible15:06
*** maybebuggy has quit IRC15:06
hwoaranganyone up for a quick +2 https://review.openstack.org/#/c/575708/ ?15:08
openstackgerritMerged openstack/openstack-ansible-openstack_openrc master: Fix the home-page  https://review.openstack.org/57760015:08
*** kukacz_ has quit IRC15:09
openstackgerritFrank Zhang proposed openstack/openstack-ansible master: Add example host confd file for barbican  https://review.openstack.org/57784715:10
*** kukacz_ has joined #openstack-ansible15:10
*** Bhujay has quit IRC15:12
*** pcaruana has quit IRC15:13
openstackgerritMerged openstack/openstack-ansible-os_searchlight master: Fix the home-page link  https://review.openstack.org/57759515:19
ansmithHello, with linters and functional passing, was hoping to land https://review.openstack.org/#/c/575505/ if anyone could add to their review list, thanks15:25
*** tux__ has quit IRC15:26
*** tux_ has joined #openstack-ansible15:26
openstackgerritMerged openstack/openstack-ansible-os_almanach master: Fix the home-page  https://review.openstack.org/57759315:27
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Cleanup mounts in containers  https://review.openstack.org/57409615:28
cloudnullmornings15:29
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/56753015:29
openstackgerritMerged openstack/openstack-ansible-plugins master: Fix connection plugin for Ansible 2.5  https://review.openstack.org/57733015:32
openstackgerritMerged openstack/openstack-ansible-plugins master: Remove the dist_sort module  https://review.openstack.org/57546915:34
*** rmart04 has joined #openstack-ansible15:41
tux_cloudnull: morning15:42
*** electrofelix has quit IRC15:44
*** cmart has quit IRC15:46
*** koukou has quit IRC15:49
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Add required packages to build mysql-python  https://review.openstack.org/57689115:54
*** rmart04 has quit IRC15:57
mnasercloudnull: i'm noticing a common pattern of jobs timing in nspawn world after networkd is restarted16:00
cloudnullmnaser https://review.openstack.org/#/c/577568/ - i think that solves that16:01
cloudnullhowever I need to circle up on that work later on today16:01
*** epalper has quit IRC16:01
cloudnullo/ tux_16:01
mnasercloudnull: no worries, just wanted to make sure you had were aware of it :)16:02
tux_cloudnull: now working on to load images and going to setup neutron for VLAN base network16:02
tux_cloudnull: I have question i am not using shared storage for glance so if i load image it will go to one of node out of 3 infra node right?16:04
*** cmart has joined #openstack-ansible16:08
*** vnogin has quit IRC16:09
*** lvdombrkr89 has quit IRC16:09
tux_cloudnull: what MTU setting you have set on your NIC and if i want to set MTU 9000 then where should i add that in OSA16:12
cloudnulltux_ if your not using shared storage for images I would recommend disabling 2 of the three glance nodes.16:17
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Add description into README like others  https://review.openstack.org/57744116:17
cloudnullyou could also setup a small swift deployment on your infra nodes and use it only for glance storage16:17
*** vnogin has joined #openstack-ansible16:17
*** Guest63 has joined #openstack-ansible16:18
tux_I have future plan to move them on Ceph, this is just temporary setup16:18
cloudnullbut not having shared storage for glance images will result in the image hitting one of many glance nodes which could (likely will) be problematic16:18
tux_cloudnull: what if i rsync /var/lib/glance to all infra node?16:18
cloudnullthat works, you'll just need to make sure the images all stay in sync16:19
cloudnullwe've nothing that does that naively16:19
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: Add description into README like others  https://review.openstack.org/57743916:19
tux_Ok got it, i will sync them all by manually16:19
tux_later i will move them to Ceph16:19
cloudnullmy team and I made a tool a VERY long time ago https://github.com/rcbops/glance-image-sync - it works ok16:20
cloudnullat least it did16:20
cloudnullthough I would not depend on that tool for prod16:20
*** Bhujay has joined #openstack-ansible16:21
tux_lets say i want to build swift in that case what i need to do ?16:22
tux_i think i can take example of AIO right?16:23
FrankZhanghey, anybody knows how to use tempest only test some certain role functions? Like I want to test barbican role on my environment but don't plan to test creating vm or creating images.16:24
*** shardy has quit IRC16:25
cloudnulltux_ yup. the aio would work just fine.16:28
tux_cloudnull: thanks!16:29
cloudnullif you have LVM on your infra nodes, you can create three LVs mount them at /srv/ and use the following config to enable swift, https://gist.github.com/cloudnull/69bde16011d1ea94edf926252ea93933#file-openstack_user_config-yml-L60-L7116:30
tux_Thanks16:31
tux_3 LVM per infra node or 1 LVM per infra node?16:32
*** Guest63 has quit IRC16:33
cloudnull3 lvs per infra node16:34
tux_hmm16:34
tux_cloudnull: will do that... what MTU size you set on production and where so you set in OSA ?16:36
tux_cloudnull: my LAN using jumboframe so i am planning to use MTU 900016:36
*** Guest63 has joined #openstack-ansible16:37
cloudnullwe sent 9000 whenever possible16:39
tux_Do you set them in openstack_user_config.yml?16:40
*** gillesMo has quit IRC16:40
cloudnullyou need that set on the interfaces themselves and then in your user_variables with the following options:  lxc_net_mtu, lxc_container_default_mtu, neutron_network_device_mtu16:44
cloudnullthe "neutron_network_device_mtu" is only used if your running ironci16:44
cloudnull**ironic16:44
tux_I am not running ironic16:45
tux_after setting all those stuff i need to re-run setup-openstack.yml right?16:45
mnasercloudnull: https://github.com/rcbops/glance-image-sync i kinda wanna rewrite that using glance_store16:49
*** cjloader_ has joined #openstack-ansible16:52
*** cjloader has quit IRC16:52
*** vnogin has quit IRC16:52
*** gyee has joined #openstack-ansible16:55
cloudnullmnaser ++16:55
*** cjloader_ has quit IRC16:55
cloudnullby all means, if i can be of any help let me know.16:56
*** cjloader has joined #openstack-ansible16:56
*** udesale has quit IRC16:56
openstackgerritMerged openstack/openstack-ansible-tests master: get-ansible-role-requirements: Use refspec when outside of OpenStack CI  https://review.openstack.org/57570817:00
tux_cloudnull: lets say i want to disable two other glance node in that case what i need to do? i am thinking until i implement ceph i should disable 2 node17:05
*** elbragstad is now known as lbragstad17:16
*** cshen has joined #openstack-ansible17:18
*** Guest63 has quit IRC17:23
mnaserjmccrory, andymccr, hwoarang: i think we broke upgrade jobs with the ansible 2.5 change17:30
mnaserAttributeError: 'Connection' object has no attribute 'container_user'17:31
jmccrorymnaser have a link to failing job?17:31
mnaserwhen it tries to deploy the old release17:31
mnaserwarning, giant file because we run the upgrade job with -vvv apparently17:31
mnaserhttp://logs.openstack.org/83/576883/6/gate/openstack-ansible-upgrade-ubuntu-xenial/d090866/job-output.txt.gz17:31
mnaserjmccrory: i wonder if the issue here is that we're pulling master plugins with older release17:34
jmccrorysounds likely17:34
mnaserwhen we should be actually pulling all the deps from the older branch17:34
mnaseryeah that does seem to be the case17:35
mnaseri really don't think we should be pulling down dependencies inside the test part17:35
mnaserwe should make use of the actual dependencies of the role17:36
odyssey4memnaser: is this integrated, or role test?17:37
mnaserodyssey4me: upgrade role test17:37
mnaserso it's combining stable/queens os_previous_cinder with what i assume masters plugins17:37
odyssey4mebother, the whole approach there needs a rewrite because it tries to only do the upgrade for a single component... but that approach is getting unwieldy17:38
mnaserodyssey4me: very, and it's also localized inside the repo too17:38
odyssey4methe plan has been to move to use the integrated repo for upgrade tests instead, as it does all the right things... but we've not been in a position to do that yet17:38
mnaseryeah.. just barely been trying to unbreak it to CI normal deploys17:39
odyssey4meso for now I'd recommend just making the upgrade test non-voting for roles... it's pretty useless in the current state and unusable until we're in a position to curate the integrated repo test to be useful for the roles17:39
*** cjloader has quit IRC17:39
mnaserodyssey4me: my ideal world is running integrated repo test for roles for deploy and upgrade (which will avoid much of our huge breakages) as well as functional tests to see if they work "on their own"17:40
odyssey4mewe've already had to disable it due to UCA things in stable branches - the only useful upgrade test is the integrated one17:40
mnaseri agree17:40
mnaser+ you might be in a state where one service is newer than the other but depends on the other being newer or something17:40
mnaserit doesn't actually mean much right now, i think17:40
*** cjloader has joined #openstack-ansible17:40
odyssey4meyes, but as we've discussed before we have to be able to make the integrated repo only run the compute kit (or nothing), then have pluggable extras per role17:40
mnaser++17:41
mnaserok so17:41
mnaserinterim solution = cinder upgrade goes nv? (i hate that but i cant imagine much better options for now)17:41
odyssey4meI'd suggest making the upgrade test non-voting in the tests repo so that it applies to all role tests that use it17:41
odyssey4methe issue in cinder is going to be the same for neutron, glance, nova, etc17:41
mnasertrue17:42
mnaserspam inc because i have a big local branch17:44
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Run all tests without containers  https://review.openstack.org/57596117:44
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter  https://review.openstack.org/57572717:44
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Switch upgrade jobs to be non-voting  https://review.openstack.org/57788417:44
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add MariaDB infrastructure mirrors  https://review.openstack.org/57788517:44
mnaserodyssey4me, jmccrory: https://review.openstack.org/#/c/577884/1 and https://review.openstack.org/#/c/576884/3 parent if possible17:45
mnaseroh17:46
mnaseri broke something17:46
mnaserone second17:46
mnasersheesh openstack-ansible-upgrade-centos-7 must be defined somewhere weird17:46
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Switch upgrade jobs to be non-voting  https://review.openstack.org/57788417:50
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add MariaDB infrastructure mirrors  https://review.openstack.org/57788517:50
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Run all tests without containers  https://review.openstack.org/57596117:50
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter  https://review.openstack.org/57572717:50
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Switch upgrade jobs to be non-voting  https://review.openstack.org/57788417:52
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add MariaDB infrastructure mirrors  https://review.openstack.org/57788517:52
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Run all tests without containers  https://review.openstack.org/57596117:52
openstackgerritMohammed Naser proposed openstack/openstack-ansible-tests master: Add retry install packages linter  https://review.openstack.org/57572717:52
mnaserok, im actually done now.17:52
mnaserodyssey4me: addressed your comment too, with way too many detail imho17:52
mnaser:p17:52
odyssey4methanks lgtm17:53
*** mbuil has quit IRC17:56
*** cshen has quit IRC17:58
*** poopcat has joined #openstack-ansible18:00
*** cmart has quit IRC18:05
*** Guest63 has joined #openstack-ansible18:08
*** Brew has quit IRC18:08
*** Guest63 has quit IRC18:14
*** Guest63 has joined #openstack-ansible18:16
*** poopcat has quit IRC18:17
*** poopcat has joined #openstack-ansible18:17
*** guilhermesp has quit IRC18:19
*** guilhermesp has joined #openstack-ansible18:20
openstackgerritMerged openstack/openstack-ansible-nspawn_container_create master: Enable network cleanup on role run  https://review.openstack.org/57455118:25
ivvehey guys18:27
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Correct networking setup for macvlan  https://review.openstack.org/57756818:27
ivvethere is a problem with neutron 16.0.1518:27
cloudnullivve what are  you seeing ?18:28
*** dxiri has quit IRC18:29
ivvetalking in #openstack-neutron18:30
ivvebasically its a GARP bug18:31
ivveits not sending it out but .py is crashing18:31
ivvehttps://hastebin.com/jivobonuqa.vbs18:31
ivvecloudnull: basically keepalived in the hosts are failing and the .py for keepalived state change is crashing18:32
ivveline 19 /openstack/venvs/neutron-16.0.15/lib/python2.7/site-packages/neutron/agent/l3/keepalived_state_change.py18:34
ivveops wrong link18:35
ivvethis one https://github.com/openstack/neutron/blob/master/neutron/agent/l3/keepalived_state_change.py18:36
ivveline 8018:37
ivvei can successfully get the switch(es) to get two arp responses from different macs if arp timeout is long enough and i spawn machines on same ip or switch FLIP18:38
openstackgerritMerged openstack/openstack-ansible-os_nova master: Ensure the database for cell0 is created  https://review.openstack.org/57693218:42
*** SimAloo has joined #openstack-ansible18:42
*** epalper has joined #openstack-ansible18:44
*** cjloader has quit IRC18:49
*** FrankZhang has quit IRC18:53
*** FrankZhang has joined #openstack-ansible18:54
*** cjloader has joined #openstack-ansible18:54
*** cjloader has quit IRC18:56
*** cjloader has joined #openstack-ansible18:57
*** cshen has joined #openstack-ansible19:01
openstackgerritMaksim Malchuk proposed openstack/openstack-ansible stable/queens: [DNM] test  https://review.openstack.org/57790219:02
*** Guest63 is now known as mattinator19:06
*** cjloader has quit IRC19:07
*** Brew has joined #openstack-ansible19:08
ivvecloudnull: observed another thing, the ha-xyz-interface doesn't seem to spawn19:08
*** dxiri has joined #openstack-ansible19:08
xgerman_cores https://review.openstack.org/#/c/576234/19:08
*** poopcat is now known as poopcat119:09
*** poopcat1 is now known as poopcat19:09
xgerman_^^ cloudnull19:10
ivveinterface ha-7624e76f-34 in the ha_confs in neutron_agents doesn't seem to exist19:11
ivveand i can also spot some "can't find interface ha-7624e76f-34" in neutron.log19:12
tux_folks, I am create network but physnet1 keyword is missing in /etc/neutron/plugins/ml2/ml2_conf.ini file so i need to add that?19:14
tux_something like network_vlan_ranges=physnet1:171:17219:15
tux_cloudnull:19:15
*** poopcat is now known as poopcat119:15
*** poopcat1 is now known as poopcat19:15
ivvenvm its in the namespace19:15
*** mattinator has quit IRC19:19
*** epalper has quit IRC19:25
*** pcaruana has joined #openstack-ansible19:27
*** Brew has quit IRC19:32
*** vnogin has joined #openstack-ansible19:38
*** cmart has joined #openstack-ansible19:44
*** Guest63 has joined #openstack-ansible19:47
*** yolanda has quit IRC20:01
*** yolanda has joined #openstack-ansible20:01
*** FrankZhang has quit IRC20:03
openstackgerritMerged openstack/openstack-ansible-os_nova master: Setup service users and endpoints before database setup  https://review.openstack.org/57697620:07
openstackgerritMerged openstack/openstack-ansible master: Add example host confd file for barbican  https://review.openstack.org/57784720:09
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts master: Correct networking setup for macvlan  https://review.openstack.org/57756820:11
*** FrankZhang has joined #openstack-ansible20:15
*** cjloader has joined #openstack-ansible20:18
*** vakuznet has quit IRC20:21
*** ansmith has quit IRC20:33
mnasercloudnull: im working on infra to figure out the timeouts20:36
mnaserand i noticed a pattern in the restart systemd failures20:37
mnaserit seems to happen almost always on limestone which infra seems to talk to over ipv620:37
mnaserthat info was not visible in ara-report20:37
mnaserhttp://logs.openstack.org/92/577492/1/check/openstack-ansible-functional-ubuntu-xenial/0bae802/job-output.txt.gz#_2018-06-25_15_31_00_25219420:37
mnasersomething to keep in mind20:37
mnasercloudnull: and found a job that ran on our cloud and same thing happened over ipv620:39
*** Guest63 has quit IRC20:39
*** KeithMnemonic has quit IRC20:40
cloudnullmnaser with the recent merges in nspawn_hosts/container_create I wonder if that issue is still happening ?20:41
mnasercloudnull: let me see if we had a job run on any of those clouds20:43
*** pcaruana has quit IRC20:43
mnasercloudnull: http://logs.openstack.org/68/577568/2/gate/openstack-ansible-functional-centos-7/a0953fa/ ran on vexxhost with ipv6 and didn't break20:43
mnaserso i think thats a positive sign20:43
cloudnullivve tux_ sorry - been busy most of the day .20:45
*** tux_ has quit IRC20:45
ivvecloudnull: no worries20:45
cloudnullivve is the keepalived issues getting sorted ?20:45
ivveim trying a redeploy with latest pike/stable but no it wasn't20:46
*** tux_ has joined #openstack-ansible20:46
cloudnullok.20:46
ivvethere was an apparent problem with GARP20:46
cloudnullmnaser ++ I think the issue was being caused by the duplicate ip assignment that was happening on the macvlan interface.20:46
ivvewith high arp timeout i was easily able to replicate multiple mac on single ip20:46
*** vurtz has joined #openstack-ansible20:47
cloudnullthe duplicate assignment was messing the routing table up20:47
ivvehigh meaning more than 300 seconds20:47
cloudnullinteresting.20:47
ivvethe problem caused keepalived to go into "unknown state" resulting in that "Netlink: filter function error"20:48
ivvethe keepalived problem occured on host level20:48
*** cmart has quit IRC20:48
ivvewhile the error was noticed first in mac addr table & in neutron20:48
ivveis dvr recommended or not? im only running l3_ha20:49
ivvewhich is default20:49
*** tux_ has quit IRC20:50
*** dxiri has quit IRC20:50
cloudnullivve are you running OvS?20:50
ivveLB20:51
cloudnullDVR is only available when running OvS20:51
ivveah20:51
* cloudnull talking to my network folks about it to see if they have some insight on the issue 20:51
ivvei still have one environment left20:52
ivveit was OK in 16.0.020:52
ivvewhich i also have one of20:52
ivvelet me check the second20:52
*** dxiri has joined #openstack-ansible20:52
cloudnullthe network folks say this is a known issue that they've seen before.20:52
ivveaye happens in 16.0.11 as well20:53
cloudnullthere's anecdotal information that updating the version of the keepalived can help the issue20:53
ivveaye it seems so20:53
*** cshen has quit IRC20:53
cloudnullthey also say that things work a lot better if there are only 2 routers20:53
cloudnullinstead of the default 320:53
ivvei see20:54
ivvewell everything else is in 3's20:54
ivve:)20:54
ivveno issues with ipv6 then?20:54
cloudnullthey say that the ubuntu version of keepalived is anchient and has a number of known bugs20:55
ivvecheck20:55
ivverealised that20:55
ivveis there any reason why osa not using latest?20:55
cloudnullwe're just using what the distro provides20:56
cloudnullthe network folks are also saying that if you can trigger any egress traffic out of the router things will work better too20:57
* cloudnull is giving a stream of consensus from the network people 20:58
ivve:D20:58
ivveits appreciated!20:58
ivvethe issue was experienced both assigning flips or just rebuilding machines directly on the vlan20:58
ivvewhich was the provider net in the testcase20:58
guilhermesphello!! Finally I finshed making the my first custom dashboard in kibana. It is focused on an old version of kibana in rpc repo https://github.com/rcbops/rpc-openstack/blob/newton/rpcd/playbooks/roles/kibana/tasks/kibana_dashboards.yml20:59
ivvei wonder what sort of trouble i would get into manually using different keepalived20:59
guilhermespops, lost the link of the dashboard, let me show ya20:59
guilhermesphttps://imgur.com/a/1Ewj0e420:59
cloudnullhttps://ihrachyshka.com/2017/05/17/gratuitous-arp-for-openstack-neutron/20:59
ivveread about that earlier today21:00
guilhermespalready have everything in a ansible playbook for elk_metrics_6x. I will try to upload the path today21:00
cloudnullthe network people linked me to that21:00
*** Guest63 has joined #openstack-ansible21:01
cloudnullok. so they're saying that keepalived 1.3.2+ solves the issue21:02
*** cmart has joined #openstack-ansible21:02
ivvealright, i guess ill try that21:02
ivveas usual, thanks a bunch cloudnull :)21:02
*** EmilienM has joined #openstack-ansible21:02
*** kukacz_ has quit IRC21:03
*** EmilienM_PTO has quit IRC21:04
*** EmilienM has quit IRC21:04
*** EmilienM has joined #openstack-ansible21:04
*** kukacz_ has joined #openstack-ansible21:05
ivvethis was the version that worked /openstack/venvs/neutron-16.0.0/bin/neutron-keepalived-state-change version 11.0.1.dev4621:08
ivvecombined with 1.2.19 (from ub 16.04)21:08
ivveill check the neutron options used in combo and diff it vs a fresh env21:09
ivvebut im guessing its not solving anything :P21:09
jrosseri've just tested this locally https://review.openstack.org/#/c/568637/ but it stubbornly fails in CI along with a couple others in the same way, any suggestions about whats making it fail?21:13
*** Guest63 has quit IRC21:14
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible stable/queens: Remove default pip_links value  https://review.openstack.org/56863721:15
jrossercloudnull: :) doesnt it get put on top of stable/queens anyway when the test runs?21:16
cloudnulljrosser, i'm not really sure why it was failing, however it was able to rebase so I'll see if it fails the same way with a code refresh .21:16
jrossersure, lets see21:16
cloudnulljrosser it should get updated on recheck21:16
jrosserthats essentially what i've just done here, cherry picked it onto stable/queens21:16
cloudnullbut, idk. change wise everything seems like it should work21:17
*** throwsb1 has quit IRC21:17
*** ansmith has joined #openstack-ansible21:26
cloudnullivve http://mirrors.kernel.org/ubuntu/pool/main/k/keepalived/keepalived_1.3.9-1build1_amd64.deb this package seems to install just fine21:29
ivveaye got it, just waiting for the deploy to finish :)21:31
cloudnullhttp://paste.openstack.org/show/724262/21:32
cloudnullah, excellent.21:32
cloudnullif that works we may be able to pull the main repo and install that package when l3_ha is enabled.21:33
*** cjloader has quit IRC21:33
*** nicolasbock has quit IRC21:37
*** FrankZhang has quit IRC21:40
*** aludwar has joined #openstack-ansible21:41
ivvecloudnull: ill let you know when i wake up21:41
cloudnullcool!21:43
cloudnullIt looks like we can just make sure cloudarchive is installed on our network nodes (agents containers) and we'll get 1.3.9 out of the box21:43
cloudnullat least we do in queens21:43
*** cshen has joined #openstack-ansible21:52
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_horizon master: Add developer panels to horizon  https://review.openstack.org/57331821:52
*** cshen has quit IRC21:57
*** SimAloo has quit IRC21:58
cloudnullif any cores are around and could give these a nudge https://review.openstack.org/#/q/topic:osprofiler+(status:open) I'd greatly appreciate it22:00
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Switch to using project-templates  https://review.openstack.org/57749222:01
*** yolanda_ has joined #openstack-ansible22:21
*** yolanda has quit IRC22:22
*** yolanda__ has joined #openstack-ansible22:28
*** tux_ has joined #openstack-ansible22:29
*** yolanda_ has quit IRC22:31
*** tux_ has quit IRC22:34
*** cmart has quit IRC22:35
*** tomcsi has quit IRC22:38
*** tomcsi has joined #openstack-ansible22:38
*** chyka has quit IRC22:38
*** chyka_ has joined #openstack-ansible22:38
*** pmannidi has joined #openstack-ansible23:14
*** tux_ has joined #openstack-ansible23:17
tux_cloudnull: i sent you email anyway!23:18
mnasercloudnull: cleated our as much as I could from those23:25
mnaserCleared**23:25
mnaserA lot I’m voting on already :p23:26
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-ops master: Add kibana custom dashboard  https://review.openstack.org/57794223:29
guilhermesp^^ really appreciate any improvement of this patch. I made a simple dashboard that attends my needs. Feel free to suggest, comment or make revisions.23:31
*** tosky has quit IRC23:34
*** chyka_ has quit IRC23:41
*** gyee has quit IRC23:51
openstackgerritMerged openstack/openstack-ansible-os_tacker master: Add packages required for osprofiler  https://review.openstack.org/57351723:56
tux_cloudnull: do you know why my instance not getting ip ? http://paste.openstack.org/show/724270/23:58
tux_anyone else here you can help me out http://paste.openstack.org/show/724270/23:58
*** vnogin has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!