Wednesday, 2019-07-17

*** weifan has quit IRC00:11
*** weifan has joined #openstack-ansible00:12
openstackgerritMerged openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests  https://review.opendev.org/67106400:12
*** poopcat has quit IRC00:13
*** poopcat has joined #openstack-ansible00:14
*** weifan has quit IRC00:16
*** TxGirlGeek has quit IRC00:21
*** gyee has quit IRC00:48
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Updated from OpenStack Ansible Tests  https://review.opendev.org/67106001:05
*** TxGirlGeek has joined #openstack-ansible01:09
*** TxGirlGeek has quit IRC01:11
*** TxGirlGeek has joined #openstack-ansible01:16
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Convert dynamic includes to static imports  https://review.opendev.org/67103701:40
*** BjoernT has joined #openstack-ansible02:39
*** BjoernT_ has joined #openstack-ansible02:40
*** BjoernT has quit IRC02:44
*** jbadiapa has quit IRC03:21
*** jbadiapa has joined #openstack-ansible03:21
*** BjoernT_ has quit IRC03:27
openstackgerritmelissaml proposed openstack/openstack-ansible-os_aodh master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65724303:28
*** gkadam has joined #openstack-ansible03:40
*** gkadam has quit IRC03:45
*** chhagarw has joined #openstack-ansible03:55
*** poopcat has quit IRC03:58
*** udesale has joined #openstack-ansible04:03
*** chhagarw has quit IRC04:16
*** chhagarw has joined #openstack-ansible04:16
*** chhagarw has quit IRC04:27
*** shyamb has joined #openstack-ansible04:46
*** kopecmartin|off is now known as kopecmartin04:54
*** TxGirlGeek has quit IRC05:22
*** cshen has joined #openstack-ansible05:33
*** udesale has quit IRC05:35
*** udesale has joined #openstack-ansible05:35
*** udesale has quit IRC05:36
*** udesale has joined #openstack-ansible05:36
*** shyamb has quit IRC05:59
*** shyamb has joined #openstack-ansible06:06
*** raukadah is now known as chandankumar06:19
*** pcaruana has joined #openstack-ansible06:21
*** udesale has quit IRC06:22
*** udesale has joined #openstack-ansible06:22
*** udesale has quit IRC06:24
*** udesale has joined #openstack-ansible06:25
*** shyam89 has joined #openstack-ansible06:35
*** gokhani has joined #openstack-ansible06:36
*** shyamb has quit IRC06:36
*** luksky11 has joined #openstack-ansible06:37
openstackgerritMerged openstack/openstack-ansible-os_placement master: Use systemd-journald instead of log files  https://review.opendev.org/67099606:42
openstackgerritMerged openstack/openstack-ansible-os_heat master: Use systemd-journald instead of log files  https://review.opendev.org/67111206:42
*** shyam89 has quit IRC06:44
*** shyamb has joined #openstack-ansible06:52
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Use systemd-journald instead of log files  https://review.opendev.org/67101906:57
*** rpittau|afk is now known as rpittau07:08
*** joshualyle has joined #openstack-ansible07:13
openstackgerritMerged openstack/openstack-ansible-os_gnocchi master: Use systemd-journald instead of log files  https://review.opendev.org/67112307:15
*** shyamb has quit IRC07:22
*** shyam89 has joined #openstack-ansible07:22
*** cshen has quit IRC07:23
*** tosky has joined #openstack-ansible07:26
*** shyam89 has quit IRC07:32
*** udesale has quit IRC07:56
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Convert dynamic includes to static imports  https://review.opendev.org/67112208:09
*** spatel has joined #openstack-ansible08:10
*** yolanda has quit IRC08:12
*** yolanda has joined #openstack-ansible08:12
*** yolanda has quit IRC08:12
*** spatel has quit IRC08:14
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: Set nodepool mirror PIP_OPTS consistently  https://review.opendev.org/67120708:19
*** shyam89 has joined #openstack-ansible08:29
*** jawad_axd has joined #openstack-ansible08:36
jawad_axdDoes anyone familiar with error " AttributeError: 'NoneType' object has no attribute 'Rados'" while integrating glance to ceph ? I have installed python-rados package in glance container, why am getting it still same error?Please help08:38
jrosserjawad_axd: you should not need to install packages manually to integrate ceph with glance08:42
ionimost probably the virtual env wasn't built with the right dependencies08:43
*** electrofelix has joined #openstack-ansible08:44
*** udesale has joined #openstack-ansible08:46
*** luksky11 has quit IRC08:47
jawad_axdokay..thanks. Should I expect ceph.conf and key file in /etc/ceph directory of glance container after running playbooks with glance parameters set  in user_variables.yml or it works differently with openstack-ansible?08:48
*** vnogin has joined #openstack-ansible08:49
*** gillesMo has quit IRC08:52
jawad_axdActually I have already running openstack and want to integrate with ceph now. Parameters seem straight forward in openstack_user_config.yml and user_variable.yml  but having no success at the moment.08:53
jrosserioni: if you set stuff up properly OSA will run the ceph_client role against the necessary containers / hosts and install everything needed for ceph support08:54
jawad_axdOkay..its not triggering ceph_client role then for some reason, i guess.08:56
jrosseri have a meeting right now but will be back latr08:56
jawad_axdthanks08:56
jrosserjawad_axd: are you deploying ceph with OSA or integrating with an existing ceph ckuster?08:56
jawad_axdintegrating with existing ceph cluster (deployed through ceph-ansible)08:57
jrosserok, thats exactly what i do, so it does work08:58
jawad_axdok, I hope it ll work for me too.M gonna try again and recheck everything.thanks09:03
ionii'09:03
ioniupsy09:03
ionihow is better to add ceph into OSA? deploy using openstack ansible the entire ceph or deploy ceph maually and integrate it in OSA?09:04
ionihttps://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html09:05
ionii dont see any way to set up OSD which disks to be used by OSA09:05
jawad_axdYeah, thats true. I found not enough material on setting up OSDs while deploying OSA and ceph together. Thus, I decided to deploy it separate and integrate later.09:07
*** luksky11 has joined #openstack-ansible09:22
*** shyam89 has quit IRC09:24
*** shyamb has joined #openstack-ansible09:33
*** shyamb has quit IRC09:57
*** shyamb has joined #openstack-ansible09:59
jrosserioni: jawad_axd: there are no specific variables in OSA for OSD setup, becasue OSA calls the ceph-ansible ansible roles10:01
jrosserso you just use exactly the same ceph-ansible vars in your OSA host/group_vars as you would with native ceph-ansible10:02
jrosserjawad_axd: this is some config i have for ceph in my user_variables.yml http://paste.openstack.org/show/754481/10:11
*** udesale has quit IRC10:38
*** owalsh has quit IRC10:42
*** shyamb has quit IRC10:49
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Use systemd-journald instead of log files  https://review.opendev.org/67103511:01
jawad_axdjrosser: Thanks.I have similar config and getting these while uploading image  http://paste.openstack.org/show/754487/ and glance-api log http://paste.openstack.org/show/754486/ . Just to mention,, listing images works fine on command line.11:08
*** shyamb has joined #openstack-ansible11:12
*** shyam89 has joined #openstack-ansible11:16
*** shyamb has quit IRC11:17
*** gtarnaras has joined #openstack-ansible11:17
*** owalsh has joined #openstack-ansible11:18
*** jawad_axd has quit IRC11:25
*** shyamb has joined #openstack-ansible11:29
*** shyam89 has quit IRC11:29
*** ansmith has quit IRC11:29
*** kplant has joined #openstack-ansible11:31
*** shyamb has quit IRC11:40
*** ygk_12345 has joined #openstack-ansible11:41
*** gtarnaras has quit IRC11:47
*** jawad_axd has joined #openstack-ansible11:49
*** gtarnaras has joined #openstack-ansible11:51
*** jawad_axd has quit IRC11:52
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Drop legacy neutron L3 HA tool  https://review.opendev.org/67124712:16
* jamesdenton salutes the legacy L3 HA tool12:19
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-config_template master: Update setup for easier packaging and publish a tarball  https://review.opendev.org/67124812:20
*** cshen has joined #openstack-ansible12:23
* cloudnull legacy HA tool - https://www.youtube.com/watch?v=WChTqYlDjtI12:27
jamesdentonexactly12:28
jamesdenton"2013 - 2019 -- You sucked, but you were better than nothing. Sometimes."12:28
*** cshen has quit IRC12:28
*** cshen has joined #openstack-ansible12:29
cloudnull^12:29
*** goldyfruit has quit IRC12:30
mgariepyhmm12:31
jamesdenton"RabbitMQ problems? Hold my beer. I got you, bro" - Legacy L3 HA Tool12:31
mgariepyis the ha stuff working correctly at scale now ?12:31
jamesdentondefine "working"12:31
mgariepynot taking 15 minutes to re-align router when a node crash.12:31
jamesdentonthat may have been keepalived's fault12:32
mgariepythe issue came around with 300ish and more router12:32
jamesdentonahh12:32
mgariepysome routers were all standby for quite a while and some other when active on all the remaining node.s.12:33
jamesdentonand using HA routers?12:33
*** udesale has joined #openstack-ansible12:33
mgariepyyep.12:33
mgariepyduring tests with a few dozen router wasn't an issue tho..12:33
mgariepyit was working great actually. but when you add a bunch of router to that it started to add a lot of lags.. and stuff took for ever to come back.12:34
mgariepyhmm. maybe it was rabbitmq fault tho.12:34
mgariepyor maybe not..12:34
noonedeadpunkwow, I haven't written ML for this ^ yet12:34
noonedeadpunk* https://review.opendev.org/67124712:35
jamesdentoni see. not sure i've seen that many routers and tested failure. The most recent VRRP election issues i was aware of was solved by newer keepalived package. The vrouter config drops the status file and the agent picks that up. So API 'status' may not always align to real status12:35
mgariepyi guess i'll see when upgrading to train ;) haha12:36
jamesdentonthat's the spirit!12:36
mgariepylol12:36
mgariepyworst case i'll do a mistral job to replace the legacy ha tool12:36
mgariepyhaha12:36
jamesdenton:D12:36
mgariepyi don't care much if the api status isn't quite aligned as long as the network works :)12:38
*** shyamb has joined #openstack-ansible12:41
*** cshen has quit IRC12:43
*** ygk_12345 has quit IRC12:49
*** ansmith has joined #openstack-ansible12:52
*** cshen has joined #openstack-ansible12:59
*** shyamb has quit IRC13:00
*** hamzy has quit IRC13:05
noonedeadpunkFolks, how do you feel about backporting https://review.opendev.org/#/q/topic:static_imports+(status:open+OR+status:merged) to stein? I think that stein worth having working tags...13:09
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum master: Use systemd-journald instead of log files  https://review.opendev.org/67126013:12
*** jamesdenton has quit IRC13:16
*** jamesdenton has joined #openstack-ansible13:17
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum master: Convert dynamic includes to static imports  https://review.opendev.org/67126513:19
*** jawad_axd has joined #openstack-ansible13:23
*** vnogin has quit IRC13:24
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Use systemd-journald instead of log files  https://review.opendev.org/67128513:30
*** ygk_12345 has joined #openstack-ansible13:32
*** hamzy has joined #openstack-ansible13:36
jrossermnaser: i think we need this for the upgrades patch https://review.opendev.org/#/c/671207/13:38
*** goldyfruit has joined #openstack-ansible13:38
jrosserbecasue it checks out the older branch from inside the gate-check-commit script so all the library functions need to exist on both branches13:38
jrosserperhaps thats unclear wording, but it breaks here http://logs.openstack.org/05/671105/1/check/openstack-ansible-upgrade-aio_metal-ubuntu-bionic/018f559/ara-report/result/43d2e698-411c-45fb-83f2-c7fd5218dec2/13:40
mnaserjrosser: was that an unclean cherry pick?13:41
mnaserit seems to have some extra stuff alongside13:41
mnasernoonedeadpunk: i think its probably worth backporting13:41
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_tacker master: Use systemd-journald instead of log files  https://review.opendev.org/67129913:43
*** udesale has quit IRC13:43
*** udesale has joined #openstack-ansible13:43
*** dr_feelgood has joined #openstack-ansible13:44
*** gtarnaras has quit IRC13:45
jrossermnaser: oh hmm let me look, yes it wasnt clean13:46
jrosserwhoops yes let me fix that13:46
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: Set nodepool mirror PIP_OPTS consistently  https://review.opendev.org/67120713:50
*** spatel has joined #openstack-ansible13:51
*** gtarnaras has joined #openstack-ansible13:51
*** vnogin has joined #openstack-ansible13:53
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_blazar master: Convert dynamic includes to static imports  https://review.opendev.org/67130113:56
*** dr_feelgood has quit IRC14:03
cloudnullhey all, long time no chat :) - https://review.opendev.org/#/c/671248/ - If folks have a moment to review that change it'd be greatly appreciated.14:05
cloudnullI'm trying to get that plugin packaged within RDO - https://review.rdoproject.org/r/#/c/21519 - and that change would be super helpful for getting that done14:06
mnasercloudnull: lgtm14:07
cloudnullthanks mnaser14:08
*** altlogbot_1 has quit IRC14:09
cloudnullwhile I've been lurking, i've not really been following along. how have things been going?14:09
mnasercloudnull: stein is out, jrosser doing cool work on binding to management interface instead of 0.0.0.0 which improves security and metal stuff, noonedeadpunk working on cleanups and moving all services to use journald14:11
cloudnullawesome!14:11
mnasera lot of clean-up, i also think our jobs are soooo much more stable now14:11
* cloudnull really likes journald 14:11
cloudnullalso great to hear about the work jrosser is doing!14:12
cloudnullmnaser - I was looking over the user survey bits just the other day and it looks like OSA is still killing it (~50% of all deployments in the survey use OSA). I suspect the numbers will be even better this round.14:15
*** vnogin has quit IRC14:15
jrosserthats amazing14:15
jawad_axdHi all, m having hard time while integrating glance with ceph. While uploading image, getting this http://paste.openstack.org/show/754501/ and glance-api log http://paste.openstack.org/show/754500/ . In user_variable.yml  glance variable are these http://paste.openstack.org/show/754502/ . When I change default store to file, it works fine. Please help14:16
*** altlogbot_3 has joined #openstack-ansible14:17
*** irclogbot_0 has quit IRC14:17
*** joshualyle has quit IRC14:18
jrossercloudnull: whilst you're around do you remember the history of 127.0.1.1? https://github.com/openstack/openstack-ansible-lxc_container_create/commit/ccddfc7731a72627e4fda919e56d2061741bbc3f14:19
jrosserjust ripping that out and wondering what i'm breaking :)14:19
guilhermespheeeeeeeey cloudnull o/14:19
*** joshualyle has joined #openstack-ansible14:19
cloudnulljrosser it was just a very likely available loopback address.14:20
cloudnullno significant meaning throughout the playbooks (that I remember)14:20
cloudnullo/ guilhermesp14:20
*** TxGirlGeek has joined #openstack-ansible14:21
*** altlogbot_3 has quit IRC14:22
*** altlogbot_1 has joined #openstack-ansible14:25
*** udesale has quit IRC14:25
*** altlogbot_1 has quit IRC14:26
*** altlogbot_2 has joined #openstack-ansible14:26
*** ygk_12345 has quit IRC14:28
*** gtarnaras has quit IRC14:29
mgariepyjawad_axd, in your glance container which version of rados is installed ?14:30
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_congress master: Convert dynamic includes to static imports  https://review.opendev.org/67130814:30
jawad_axd13.2 same as on ceph nodes14:30
mgariepyis the rados/rbd linked in /openstack/venv/undefined/14:31
*** altlogbot_2 has quit IRC14:32
jawad_axdmgariepy: How to verify that ?14:33
*** altlogbot_3 has joined #openstack-ansible14:33
mgariepyfind /openstack/venvs/glance-*/ |grep "rados\|rbd"14:33
jawad_axdmgariepy: http://paste.openstack.org/show/754504/14:35
mgariepyhttps://github.com/openstack/openstack-ansible-ceph_client/blob/master/tasks/ceph_install_python_libs.yml14:36
mgariepyseems like this tasks didn't run14:36
jawad_axdmgariepy: Should I run it manually or ?14:37
mgariepyyou can re-run the os-glance install it and look if it does the linking when it's installing ceph stuff14:37
*** altlogbot_3 has quit IRC14:38
jawad_axdmgariepy: ok, thanks!14:38
*** altlogbot_0 has joined #openstack-ansible14:39
mgariepylet me know it it fixes when re-running14:40
jawad_axdmgariepy: sure14:42
openstackgerritMerged openstack/ansible-config_template master: Update setup for easier packaging and publish a tarball  https://review.opendev.org/67124814:43
*** altlogbot_0 has quit IRC14:44
*** altlogbot_0 has joined #openstack-ansible14:45
jawad_axdmgariepy: No success. Just to mention, I am integrating with existing ceph cluster.14:49
*** altlogbot_0 has quit IRC14:50
*** luksky11 has quit IRC14:51
*** altlogbot_2 has joined #openstack-ansible14:51
mgariepyjawad_axd, it shouldn't matter14:54
jawad_axdmgariepy: Just wondering how to run https://github.com/openstack/openstack-ansible-ceph_client/blob/master/tasks/ceph_install_python_libs.yml if I am integrating with existing ceph cluster.14:55
mgariepyin the ansible output of os_glance you should see it loading the tasks from ceph_client14:55
*** altlogbot_2 has quit IRC14:56
*** altlogbot_0 has joined #openstack-ansible14:57
jawad_axdmgariepy: http://paste.openstack.org/show/754506/14:59
*** gyee has joined #openstack-ansible14:59
mgariepyhmm it has not run.15:01
*** altlogbot_0 has quit IRC15:01
mgariepycan you re-run with -v ? or -vv ?15:01
jawad_axdmgariepy: yeah ok15:02
*** altlogbot_3 has joined #openstack-ansible15:03
jrosserjawad_axd: this is the task in the os_glance code https://github.com/openstack/openstack-ansible-os_glance/blob/fc071c769e7a676a061c63122d6ebc87d19dc8c2/tasks/main.yml#L124-L13415:04
jrosseri would suggest checking those two conditions on L131,13215:05
jrosserthat you have variables defined that make those be true15:05
jawad_axdmgariepy: http://paste.openstack.org/show/754507/15:05
jawad_axdjrosser: I am on it15:06
jrosserjawad_axd: this looks bad "[u'ceph_mons != []', u'ceph_mons | list == ceph_mons']" was evaluated as "False"15:07
jrosserhave you defined a variable ceph_mons in your user variables that is a list of the IP of your mons?15:07
*** altlogbot_3 has quit IRC15:08
jawad_axdjrosser: yes   http://paste.openstack.org/show/754502/15:08
mgariepytry  to add space between - and " in your ceph_mons15:09
mgariepylist.15:09
jrosserjawad_axd: ^ yes that15:09
jrosserbeat me to it :)15:09
mgariepyjrosser, sorry ;p15:09
jrosser:)15:09
mgariepyyaml it valid tho.15:10
mgariepywhich is kinda bad haha15:10
jrosserwhat does it make? it's not a list.....15:10
mgariepymultiline string15:11
mgariepyfun.15:11
*** altlogbot_2 has joined #openstack-ansible15:11
mgariepyceph_mons: "-\"10.10.10.18\" -\"10.10.10.19\" -\"10.10.10.20\""15:11
jrosseroh my15:11
jawad_axdI am running os-glance again after adding spaces15:11
*** altlogbot_2 has quit IRC15:12
*** gtarnaras has joined #openstack-ansible15:13
*** altlogbot_3 has joined #openstack-ansible15:13
*** tosky has quit IRC15:13
jawad_axdmgariepy: jrosser, Holy Cow, It triggered ceph_client roles. Although I got some dependency error which I will fix it. But I can not believe it was space issue. Fu** YAML. But thank you so much guys, you are awesome. I think I can proceed further now. Have a great  day!15:16
mgariepyyep, yaml is fun.15:17
*** altlogbot_3 has quit IRC15:18
mgariepywould be nice to have some kind of validator but seems like a lot of work for not much benefit in the long run. (once your config is good it's simple to manage on upgrade)15:18
*** altlogbot_1 has joined #openstack-ansible15:19
jrosserjawad_axd: \o/ great news15:19
jawad_axdmgariepy: Thats true. But guess what, i validated code online and it showed all ok (valid code).I am never gonna visit that site again..hihi15:21
jrosserstein sha bump is ready to go https://review.opendev.org/#/c/670728/15:21
mgariepyjawad_axd, a valid syntax isn't a valid config.15:22
mgariepyit's not validating that you enterred the right config keys and options.15:22
jawad_axdmgariepy: ah ok. got it!15:22
*** jawad_axd has quit IRC15:23
*** altlogbot_1 has quit IRC15:24
jrosserlogan-: did you want to look at anything bionic/lxb?15:30
*** joshualyle has quit IRC15:30
*** altlogbot_1 has joined #openstack-ansible15:31
*** altlogbot_1 has quit IRC15:32
grabesjrosser: Do your neutron created vlan bridges have ageing of 0, or 30015:33
*** altlogbot_3 has joined #openstack-ansible15:33
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Drop legacy neutron L3 HA tool  https://review.opendev.org/67124715:34
cshenHey guys, does openstack-ansible stein still support Ubuntu 16.04?15:34
logan-jrosser: yeah show brctl showstp <vlan bridge name> on a hv and see what the ageing timer is15:34
grabesjrosser: Also, is it Stein?15:35
goldenfriIs there a way to add multiple alias lines in my nova override? I see something about it at the bottom of this file, but I'm not exactly sure how to use that then I am using container vars in the openstack_user_config for my infra nodes https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml15:35
jrosserlogan-: grabes: how about this http://paste.openstack.org/show/754509/15:36
jrosserthis is stein on 18.0415:36
logan-nice, reprod15:37
jrosserso whats going on here?15:37
*** altlogbot_3 has quit IRC15:38
*** altlogbot_3 has joined #openstack-ansible15:39
jrosseri see ageing time 300.00 on a bridge for a vxlan network15:39
grabesjrosser: Can you create a vlan based network, and check that one..15:41
logan-jrosser: what ageing do you see if you  brctl addbr testbr; brctl addif testbr <upstream interface for your vlan network>; brctl showstp testbr; brctl delbr testbr;15:41
jrosserthe first paste is a vlan based network15:41
logan-next thing i want to do is get a startup log on a clean reboot from neutron-linuxbridge-agent with debug enabled so we can see all the ip/bridge commands it runs15:41
logan-to get us to that state15:42
*** siqbal has joined #openstack-ansible15:42
grabesjrosser: Thanks, yikes ageing is 0 on that one which is what I was seeing15:42
jrosserlogan-:  i'm doing this wrong? device bond1.2501 is already a member of a bridge; can't enslave it to bridge testbr.15:42
jrosseroh hrrm15:43
logan-ah15:44
jrosserok i got it15:44
*** altlogbot_3 has quit IRC15:44
logan-well that makes sense i guess if possible just skip to debug mdoe reboot of linuxbridge agent15:44
jrosserhttp://paste.openstack.org/show/754510/15:44
grabesLogan, couldnt we just enabled debugging on neutron, restart it, and then add a new net?15:44
logan-yep that should work15:45
jrosserbond1 is a member of br-vlan which gets made with ifupdown at boot http://paste.openstack.org/show/754510/15:45
*** altlogbot_1 has joined #openstack-ansible15:45
jrosserbond1.2501 is a member of a bridge created by neutron and there we see ageing time 015:45
logan-so it really seems like it is something that neutron is doing to get us that broken bridge15:45
jrosserbond1.2501 in this case is a flat network for lbaas15:46
jrosserlogan-: ^ that is the culmination of my adventure with nxos BGP-EVPN, finally got a l2-over-l3 running on the switches for br-lbass and friends15:47
grabesLogan if you want you, and have time you are welcome to do this on a system on my side.  I can probably get around to it later afternoon, or tonight15:48
*** altlogbot_1 has quit IRC15:50
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_mistral master: Convert dynamic includes to static imports  https://review.opendev.org/67132515:50
logan-jrosser haha nice :)15:51
*** altlogbot_0 has joined #openstack-ansible15:51
logan-grabes: ok will do. probably wont be able to till late this afternoon or evening.15:56
grabesso now we have 3 people having the same issue15:57
*** altlogbot_0 has quit IRC15:58
jrosseri think i am lucky becasue i do all vxlan with the exception of special things like br-lbaas which are on vlans16:00
logan-yep and spatel who is not, so it seems specific to bionic or at least ubuntu16:00
jrosserthe the leakage is very much more limited than if i had vlan project networks16:01
grabesjrosser: yes, unfortunately there a probably a lot of unlucky deployments who are broadcasting unicast traffic in their VMs16:01
logan-i wonder if it occurs on bionic rocky installs16:01
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_mistral master: Use systemd-journald instead of log files  https://review.opendev.org/67132616:01
jrosseraltough i do plan to do more vlan stuff now i can do EVPN on my switches, so yeah, this is not cool16:02
spatelin my case i have centOS and its doing right thing16:02
grabeslogan-: I would hope someone would have seen this given there are probablty a lot more deployments on that, but thats simply an assumption16:02
spatelmnaser: running on CentOS, he may provide some input..16:02
*** altlogbot_1 has joined #openstack-ansible16:03
Nick_A<logan-> i wonder if it occurs on bionic rocky installs <-- not for us afaik16:04
logan-thanks Nick_A16:04
*** rpittau is now known as rpittau|afk16:04
Nick_Ahttp://paste.openstack.org/show/m5WozwDR5KhysfQepyFX/16:06
*** altlogbot_1 has quit IRC16:08
*** altlogbot_3 has joined #openstack-ansible16:09
*** irclogbot_2 has joined #openstack-ansible16:13
*** altlogbot_3 has quit IRC16:14
*** altlogbot_0 has joined #openstack-ansible16:14
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_mistral master: Use systemd-journald instead of log files  https://review.opendev.org/67132616:14
spotzlogan-: can we even still install on xenial?16:16
logan-on rocky, yes. on stein xenial is no longer supported16:16
*** irclogbot_2 has quit IRC16:17
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_panko master: Use systemd-journald instead of log files  https://review.opendev.org/67132716:18
cshenlogan-: so what's written here https://docs.openstack.org/project-deploy-guide/openstack-ansible/stein/deploymenthost.html is wrong?16:18
*** irclogbot_3 has joined #openstack-ansible16:22
*** gtarnaras has quit IRC16:22
spotzcshen: would seem that way according to what logan- just said. I can patch that real quick if that's the case16:23
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_swift master: Convert dynamic includes to static imports  https://review.opendev.org/67132916:26
*** irclogbot_3 has quit IRC16:26
*** irclogbot_0 has joined #openstack-ansible16:26
cshenspotz: also the next page https://docs.openstack.org/project-deploy-guide/openstack-ansible/stein/targethosts.html has Ubuntu 16.0416:28
spotzcshen thanks16:28
cshennp16:28
jrosseri have a patch for this i think16:29
jrosserhere https://review.opendev.org/#/c/668236/16:29
noonedeadpunkCan anyone share why do we do this? https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/master/tasks/swift_pre_install.yml#L86-L9216:29
*** irclogbot_0 has quit IRC16:30
*** irclogbot_3 has joined #openstack-ansible16:30
noonedeadpunkjust to reduce influence created by updatedb?16:30
openstackgerritAmy Marrich (spotz) proposed openstack/openstack-ansible master: Remove Ubuntu Xenial from suported Operating Systems  https://review.opendev.org/67133016:31
spotzbah jrosser:)16:31
spotzlet me go abandon mine then I'll review yours16:32
jrosseri think i found more references to xenial16:32
spotzyeah16:32
*** irclogbot_3 has quit IRC16:34
*** altlogbot_0 has quit IRC16:34
spotzjrosser: Didn't workflow it but +2d16:34
*** irclogbot_0 has joined #openstack-ansible16:34
spotznoonedeadpunk just did though:)16:34
*** altlogbot_3 has joined #openstack-ansible16:35
*** irclogbot_0 has quit IRC16:38
*** irclogbot_3 has joined #openstack-ansible16:38
*** irclogbot_3 has quit IRC16:42
*** irclogbot_3 has joined #openstack-ansible16:42
*** cshen has quit IRC16:42
*** irclogbot_3 has quit IRC16:46
*** irclogbot_1 has joined #openstack-ansible16:46
*** irclogbot_1 has quit IRC16:50
*** irclogbot_1 has joined #openstack-ansible16:50
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_trove master: Use systemd-journald instead of log files  https://review.opendev.org/67133116:52
*** irclogbot_1 has quit IRC16:54
*** irclogbot_0 has joined #openstack-ansible16:54
*** jbadiapa has quit IRC16:55
*** irclogbot_0 has quit IRC16:58
*** strattao has joined #openstack-ansible16:58
*** irclogbot_1 has joined #openstack-ansible16:58
openstackgerritMerged openstack/openstack-ansible-os_barbican master: Convert dynamic includes to static imports  https://review.opendev.org/67112217:00
*** irclogbot_1 has quit IRC17:02
*** irclogbot_0 has joined #openstack-ansible17:02
*** spatel has quit IRC17:02
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_trove master: Convert dynamic includes to static imports  https://review.opendev.org/67133417:03
*** irclogbot_0 has quit IRC17:06
*** irclogbot_3 has joined #openstack-ansible17:06
*** irclogbot_3 has quit IRC17:10
*** irclogbot_3 has joined #openstack-ansible17:10
*** irclogbot_3 has quit IRC17:14
*** irclogbot_2 has joined #openstack-ansible17:14
*** electrofelix has quit IRC17:15
mnasernoonedeadpunk: that updatedb thing looks so weird17:17
*** irclogbot_2 has quit IRC17:18
*** irclogbot_0 has joined #openstack-ansible17:18
jrosserIdk if swift leverages “locate” in some way but I couldn’t really find anything17:19
*** siqbal has quit IRC17:19
noonedeadpunkmnaser:  yeah, I think it just replace native one and disable updatedb for zfs and place ionice-3 for the rest...17:20
mnaseri dont think swift uses locate afaik17:21
noonedeadpunkI don't think that swift relies on it in anyway, but locate may took a lot of time if it start sceanning object storage (in case it's local??). Tbh I'm not really common with swift:(17:21
mnaseri wonder if that is a rackspaceism17:21
mnaserhttps://bugs.launchpad.net/openstack-ansible/+bug/157230717:21
openstackLaunchpad bug 1572307 in openstack-ansible "mlocate indexes swift objects" [Undecided,Fix released] - Assigned to Bjoern Teipel (bjoern-teipel)17:21
mnaserthis seems to be the source17:22
mnaseri think this is not something that the OSA role should be touching/fixing17:22
*** irclogbot_0 has quit IRC17:22
*** irclogbot_2 has joined #openstack-ansible17:22
noonedeadpunkbut since we're already fixing this...17:23
noonedeadpunkwondering if issue still here17:23
jrosserI might expect to see a so17:24
jrosser*similar thing for ceph filestore osd17:24
jrosserYou want to keep any kind of indexing out of that too17:25
openstackgerritMerged openstack/openstack-ansible-os_aodh master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65724317:26
*** irclogbot_2 has quit IRC17:26
*** chandankumar is now known as raukadah17:26
*** irclogbot_1 has joined #openstack-ansible17:26
noonedeadpunkyeah, that will also be a case...17:26
mnaserwell17:27
mnaserlets say i was using ext4 instead of xfs for my partitions17:27
mnaserthat doesnt really solve it17:27
mnaserand if im using centos i just removed ALL my updatedb-ing because it uses xfs for root17:27
*** irclogbot_1 has quit IRC17:30
*** irclogbot_2 has joined #openstack-ansible17:30
noonedeadpunkyep, probably it's more about ops stuff than swift....17:30
*** weifan has joined #openstack-ansible17:32
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_zun master: Use systemd-journald instead of log files  https://review.opendev.org/67133917:32
*** irclogbot_2 has quit IRC17:34
*** irclogbot_2 has joined #openstack-ansible17:34
noonedeadpunkmnaser: btw, before creating role for uwsgi I'd migrate services from apache to uwsgi then to unify things.17:36
mnasernoonedeadpunk: i think the only one that is using apache still might be horizon and keystone?17:36
noonedeadpunkNope.17:36
noonedeadpunkit's also aodh, gnocchi, panko and trove17:36
noonedeadpunkand I probably missed one (don't remember what exactly) :(17:37
*** irclogbot_2 has quit IRC17:38
mnaserah.. that seems like most of the telemetry stack, probably not too hard to port it17:38
mnaserbut maybe its easier to just make it use the new role (instead of converting to old way THEN moving to role)17:38
*** irclogbot_0 has joined #openstack-ansible17:38
*** goldyfruit has quit IRC17:39
*** goldyfruit has joined #openstack-ansible17:40
noonedeadpunkyeah, maybe. Actually order doesn't matter I'd say17:40
noonedeadpunkjust migration to role will be more straightforward17:41
noonedeadpunkwhen we have just 1 way everything is configured17:41
mnaseryeah17:41
mnasernoonedeadpunk: btw, masakari api container seems to be missing from metal.yml file17:42
*** irclogbot_0 has quit IRC17:42
mnaserwhich means that it's creating a container for it in our ci17:42
*** irclogbot_2 has joined #openstack-ansible17:42
mnaserhttps://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/env.d/aio_metal.yml.example17:42
mnasermasakari_monitors_container is there17:42
noonedeadpunkmasakari-api might be in container.17:42
mnaserbut masakari_api_container17:42
mnaseris not there17:42
mnaserfound this out whe ni was hcecking the systemd logs here http://logs.openstack.org/85/671285/1/check/openstack-ansible-deploy-aio_metal-debian-stable/e89bd9b/logs/openstack/aio1_masakari_api_container-f180ea6f/17:43
noonedeadpunkmnaser: for masakari it's ok that api is placed in container. Or you mean, that once we're running metal CI everything should be runned on metal?17:44
mnasernoonedeadpunk: when we are running metal CI, everything should be in metal17:44
mnaserbecause the entry is missign there, in metal ci, it's creating a container for it17:44
*** cshen has joined #openstack-ansible17:44
noonedeadpunkahh, yep...17:44
noonedeadpunkgood catch17:45
*** irclogbot_2 has quit IRC17:46
mnaser:)17:46
*** irclogbot_2 has joined #openstack-ansible17:46
jrossernoonedeadpunk: maybe designate too needs some uwsgi-ness adding17:48
noonedeadpunkjrosser: don't see anything apache-related for designate...17:50
*** irclogbot_2 has quit IRC17:50
*** cshen has quit IRC17:50
*** irclogbot_2 has joined #openstack-ansible17:50
jrosserNo but isnt is a service that never got ported to uwsgi in the first place maybe17:51
jrosserIt all looked very odd when I was looking at the bind address stuff17:51
noonedeadpunkthan this list is much bigger as not only designate is running api with internal service17:52
noonedeadpunkmentioned masakari does the same17:53
*** irclogbot_2 has quit IRC17:54
*** irclogbot_2 has joined #openstack-ansible17:54
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add masakari-api to aio_metal  https://review.opendev.org/67134217:54
mnaseryeah i think designate still uses the 'good ol' eventlet builtin server17:55
noonedeadpunkmnaser: addresses masakari api stuff ^17:55
*** vnogin has joined #openstack-ansible17:55
mnasernice nice17:55
*** vnogin has quit IRC17:57
*** irclogbot_2 has quit IRC17:58
*** irclogbot_2 has joined #openstack-ansible17:58
*** cshen has joined #openstack-ansible18:01
*** poopcat has joined #openstack-ansible18:02
*** irclogbot_2 has quit IRC18:02
*** irclogbot_1 has joined #openstack-ansible18:02
openstackgerritMohammed Naser proposed openstack/openstack-ansible stable/stein: Set nodepool mirror PIP_OPTS consistently  https://review.opendev.org/67134318:03
mnaserjrosser: ^ this was your change18:03
mnaseryou pushed to rocky and forgot stein :P18:03
mnasermaster upgrades test stable/stein => master18:04
*** irclogbot_1 has quit IRC18:06
*** cshen has quit IRC18:06
*** irclogbot_0 has joined #openstack-ansible18:06
*** irclogbot_0 has quit IRC18:10
*** irclogbot_0 has joined #openstack-ansible18:10
*** irclogbot_0 has quit IRC18:14
*** irclogbot_2 has joined #openstack-ansible18:14
openstackgerritMerged openstack/openstack-ansible-ops master: Use raw url for pip deps  https://review.opendev.org/67018318:16
*** irclogbot_2 has quit IRC18:18
*** irclogbot_0 has joined #openstack-ansible18:18
jrossermnaser: was t the change already in stein?18:22
*** irclogbot_0 has quit IRC18:22
jrosserWtf then :)18:22
*** irclogbot_3 has joined #openstack-ansible18:22
jrosserBecause your test patch was checking out stable/rocky :)18:22
*** irclogbot_3 has quit IRC18:26
*** irclogbot_1 has joined #openstack-ansible18:26
*** irclogbot_1 has quit IRC18:30
*** irclogbot_0 has joined #openstack-ansible18:30
*** irclogbot_0 has quit IRC18:34
mnaserjrosser: then our master branch upgrade job is broken18:34
*** irclogbot_2 has joined #openstack-ansible18:34
mnaserand its trying to upgrade from rocky to master18:34
mnaser1 sec18:34
openstackgerritMohammed Naser proposed openstack/openstack-ansible master: Add metal upgrade jobs  https://review.opendev.org/67110518:35
mnaserjrosser: now it should do the right thing(tm)18:35
*** irclogbot_2 has quit IRC18:38
*** irclogbot_3 has joined #openstack-ansible18:38
*** irclogbot_3 has quit IRC18:42
*** irclogbot_2 has joined #openstack-ansible18:42
*** irclogbot_2 has quit IRC18:46
*** irclogbot_1 has joined #openstack-ansible18:46
*** irclogbot_1 has quit IRC18:50
*** irclogbot_0 has joined #openstack-ansible18:50
*** weifan has quit IRC18:52
*** irclogbot_0 has quit IRC18:54
*** irclogbot_3 has joined #openstack-ansible18:54
*** ianychoi has quit IRC18:56
*** ianychoi has joined #openstack-ansible18:56
*** irclogbot_3 has quit IRC18:58
*** irclogbot_0 has joined #openstack-ansible18:58
*** irclogbot_0 has quit IRC19:02
*** irclogbot_0 has joined #openstack-ansible19:02
*** irclogbot_0 has quit IRC19:06
*** irclogbot_0 has joined #openstack-ansible19:06
*** irclogbot_0 has quit IRC19:10
*** irclogbot_1 has joined #openstack-ansible19:10
*** irclogbot_1 has quit IRC19:14
*** irclogbot_3 has joined #openstack-ansible19:14
*** irclogbot_3 has quit IRC19:18
*** irclogbot_1 has joined #openstack-ansible19:18
jrossermnaser: yes looks like thats totally needed in stein too http://logs.openstack.org/05/671105/2/check/openstack-ansible-upgrade-aio_metal-ubuntu-bionic/4221bcd/ara-report/result/d5eb7aea-b61d-4646-9c2a-99b00fc4ac49/19:20
*** irclogbot_1 has quit IRC19:22
*** irclogbot_3 has joined #openstack-ansible19:22
*** irclogbot_3 has quit IRC19:26
*** irclogbot_2 has joined #openstack-ansible19:26
openstackgerritMerged openstack/openstack-ansible-os_masakari master: Use systemd-journald instead of log files  https://review.opendev.org/67128519:27
*** irclogbot_2 has quit IRC19:30
*** irclogbot_1 has joined #openstack-ansible19:30
*** cshen has joined #openstack-ansible19:32
*** irclogbot_1 has quit IRC19:34
*** irclogbot_1 has joined #openstack-ansible19:34
*** poopcat1 has joined #openstack-ansible19:34
*** weifan has joined #openstack-ansible19:35
redkriegwe've just tried changing some nova policies via nova_policy_overrides and after running playbooks the new policies are in policy.json-18.1.8 instead of policy.json.  should policy.json be a symlink there or is there some additional step we need to do?19:36
*** kopecmartin is now known as kopecmartin|off19:36
*** poopcat has quit IRC19:37
*** irclogbot_1 has quit IRC19:38
*** irclogbot_2 has joined #openstack-ansible19:38
*** poopcat1 is now known as poopcat19:42
*** irclogbot_2 has quit IRC19:42
*** irclogbot_1 has joined #openstack-ansible19:42
*** luksky11 has joined #openstack-ansible19:45
*** cshen has quit IRC19:46
*** irclogbot_1 has quit IRC19:46
*** weifan has quit IRC19:46
*** irclogbot_3 has joined #openstack-ansible19:46
*** kaiokmo has quit IRC19:48
*** gyee has quit IRC19:49
*** gyee has joined #openstack-ansible19:50
*** irclogbot_3 has quit IRC19:50
*** irclogbot_2 has joined #openstack-ansible19:50
*** kplant has quit IRC19:52
*** irclogbot_2 has quit IRC19:54
*** irclogbot_0 has joined #openstack-ansible19:54
*** cshen has joined #openstack-ansible19:57
*** irclogbot_0 has quit IRC19:58
*** irclogbot_2 has joined #openstack-ansible19:58
grabesjrosser, logan- Here is the debug of neutron creating the bridge of a vlan http://paste.openstack.org/show/754520/20:01
grabesjamesdenton: you may want to take a look at it as well20:01
*** cshen has quit IRC20:02
*** irclogbot_2 has quit IRC20:02
*** irclogbot_3 has joined #openstack-ansible20:02
*** weifan has joined #openstack-ansible20:03
mnasergrabes: might wanna forward that to #openstack-neutron too20:04
openstackgerritMerged openstack/openstack-ansible master: Drop support for Ubuntu Xenial  https://review.opendev.org/66823620:05
openstackgerritMerged openstack/openstack-ansible stable/stein: Bump SHAs for stable/stein  https://review.opendev.org/67072820:05
*** irclogbot_3 has quit IRC20:06
*** irclogbot_1 has joined #openstack-ansible20:06
grabesmnaser: done20:06
*** irclogbot_1 has quit IRC20:09
logan-i don't see where a bridge is being created there20:10
*** irclogbot_3 has joined #openstack-ansible20:11
logan-if you only created the network, I think you'll also need to add that network to an instance before the bridge will be initialized20:11
grabesthe bridge got initialized and the vm booted20:12
grabesYou are correct though the bridge doesn’t get created on the host until the vm is booted20:14
*** irclogbot_3 has quit IRC20:14
grabesbridge brq79450c5c-3920:16
*** goldyfruit has quit IRC20:20
*** joshualyle has joined #openstack-ansible20:21
*** ansmith has quit IRC20:38
*** irclogbot_0 has joined #openstack-ansible20:43
*** irclogbot_0 has quit IRC20:46
*** pcaruana has quit IRC20:48
*** ansmith has joined #openstack-ansible21:12
*** BjoernT has joined #openstack-ansible21:21
*** irclogbot_3 has joined #openstack-ansible21:22
*** BjoernT has quit IRC21:23
*** vnogin has joined #openstack-ansible21:23
*** vnogin has quit IRC21:29
*** luksky11 has quit IRC22:24
*** logan- has quit IRC22:38
*** logan- has joined #openstack-ansible22:39
*** strattao has quit IRC22:45
*** TxGirlGeek has quit IRC23:01
*** hwoarang has quit IRC23:37
*** hwoarang has joined #openstack-ansible23:38
*** goldyfruit has joined #openstack-ansible23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!