Friday, 2020-02-28

*** jawad_axd has joined #openstack-ansible00:01
*** macz_ has quit IRC00:02
*** jawad_axd has quit IRC00:05
*** errr has quit IRC00:17
*** errr has joined #openstack-ansible00:18
*** tosky has quit IRC00:18
*** jawad_axd has joined #openstack-ansible00:22
*** jawad_axd has quit IRC00:26
*** mpjetta has joined #openstack-ansible00:50
*** mpjetta has quit IRC01:21
*** mpjetta has joined #openstack-ansible01:24
openstackgerritMerged openstack/openstack-ansible stable/stein: Bump SHAs for stable/stein  https://review.opendev.org/70670701:48
*** gyee has quit IRC01:52
*** errr has quit IRC02:17
*** errr has joined #openstack-ansible02:18
*** joshualyle has joined #openstack-ansible03:50
*** d34dh0r53 has joined #openstack-ansible04:16
*** errr has quit IRC04:17
*** errr has joined #openstack-ansible04:17
*** rh-jelabarre has quit IRC05:08
*** udesale has joined #openstack-ansible05:13
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-ansible05:35
*** jamesdenton has quit IRC05:38
*** jamesdenton has joined #openstack-ansible05:39
*** joshualyle has quit IRC05:41
*** joshualyle has joined #openstack-ansible05:49
*** joshualyle has quit IRC05:53
*** shyamb has joined #openstack-ansible06:03
*** errr has quit IRC06:17
*** errr has joined #openstack-ansible06:18
*** pcaruana has joined #openstack-ansible06:22
*** joshualyle has joined #openstack-ansible06:50
*** joshualyle has quit IRC06:55
*** shyamb has quit IRC06:56
*** ansmith has quit IRC07:06
*** ansmith has joined #openstack-ansible07:07
*** jawad_axd has joined #openstack-ansible07:08
*** shyamb has joined #openstack-ansible07:13
*** errr has quit IRC07:17
*** errr has joined #openstack-ansible07:18
*** shyamb has quit IRC07:20
*** pcaruana has quit IRC07:39
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add magnum tempest test  https://review.opendev.org/71024507:54
*** this10nly has joined #openstack-ansible08:03
*** joshualyle has joined #openstack-ansible08:05
*** fghaas has joined #openstack-ansible08:07
*** joshualyle has quit IRC08:09
*** cshen has joined #openstack-ansible08:15
*** tosky has joined #openstack-ansible08:21
*** rpittau|afk is now known as rpittau08:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_magnum master: Add ability to create COE template  https://review.opendev.org/70809708:23
*** joshualyle has joined #openstack-ansible08:26
*** joshualyle has quit IRC08:30
*** pcaruana has joined #openstack-ansible08:34
*** DanyC has joined #openstack-ansible08:34
*** DanyC has quit IRC08:36
*** ivve has joined #openstack-ansible08:36
*** DanyC has joined #openstack-ansible08:36
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/71041208:38
*** DanyC has quit IRC08:41
*** DanyC has joined #openstack-ansible08:42
*** pcaruana has quit IRC08:48
*** Neurognostic has joined #openstack-ansible08:50
*** fghaas has left #openstack-ansible08:52
*** shyamb has joined #openstack-ansible09:00
*** joshualyle has joined #openstack-ansible09:06
*** joshualyle has quit IRC09:10
openstackgerritMerged openstack/openstack-ansible-tests master: Clean CI virtualenv installation  https://review.opendev.org/70972009:17
*** errr has quit IRC09:17
*** errr has joined #openstack-ansible09:18
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_hosts master: Updated from OpenStack Ansible Tests  https://review.opendev.org/71041409:22
*** cshen has quit IRC09:29
Adri2000hello, anyone interested in a playbook to manage haproxy backends... https://review.opendev.org/#/c/708679/ ? :-)09:30
jrosserAdri2000: nice! gshippey ^ you might be interested in that09:35
*** lkoranda has joined #openstack-ansible09:38
mnaserredrobot: sure -- you can push up a release patch and i can review it :)09:40
*** lkoranda has quit IRC09:42
openstackgerritMerged openstack/openstack-ansible-tests stable/train: Do not include docs/requirements.txt for functional tests  https://review.opendev.org/70979509:50
*** DanyC_ has joined #openstack-ansible09:50
*** alti_17 has joined #openstack-ansible09:53
*** DanyC has quit IRC09:54
*** DanyC_ has quit IRC09:57
*** DanyC has joined #openstack-ansible09:58
openstackgerritMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/71041210:01
openstackgerritMerged openstack/openstack-ansible-tests stable/train: Use /tmp for ansible temporary directory  https://review.opendev.org/70979610:03
alti_17Hello, I've just noticed that magnum + fedora coreos was discussed recently, just would like to share my experience. In general, it works, in my case cluster provisioning two times faster than atomic (5min vs 10). But there two things to consider, currently you can't use PVC because of a bug in Kubernetes10:04
alti_17https://github.com/kubernetes/kubernetes/pull/86027 - workaround disable selinux. Second - fedora coreos has auto-update feature which updates OS and reboot system immediately without any confirmations https://docs.fedoraproject.org/en-US/fedora-coreos/auto-updates/  and looks like magnum driver doesn't handle this behavior, and it causes failed10:04
alti_17clusters if you boot cluster with yesterday's image and new image was released zincati service update and reboot server what often breaks cluster creation/scaling - workaround disable zincati in driver.10:04
*** shyamb has quit IRC10:14
*** cshen has joined #openstack-ansible10:16
jrosseralti_17: i wonder if we should start collecting 'known good' magnum configs as part of the os_magnum role documentation10:17
*** errr has quit IRC10:17
jrosserthey might have a relatively short lifetime but would be helpful to record what was known to work at a particular point in time10:17
*** errr has joined #openstack-ansible10:18
*** joshualyle has joined #openstack-ansible10:22
openstackgerritMerged openstack/openstack-ansible-tests stable/train: Update ansible to 2.8.8  https://review.opendev.org/70979710:24
*** joshualyle has quit IRC10:27
*** shyamb has joined #openstack-ansible10:31
*** trident has quit IRC10:31
*** DanyC has quit IRC10:34
*** trident has joined #openstack-ansible10:34
*** DanyC has joined #openstack-ansible10:35
*** DanyC has quit IRC10:39
*** joshualyle has joined #openstack-ansible10:52
*** joshualyle has quit IRC10:57
*** DanyC has joined #openstack-ansible11:14
*** cshen has quit IRC11:31
*** DanyC has quit IRC11:38
*** DanyC has joined #openstack-ansible11:44
*** shyamb has quit IRC11:54
*** cshen has joined #openstack-ansible12:00
*** Neurognostic has quit IRC12:10
alti_17jrosser I can take care of it if you think it might be useful, but could you point me location where you prefer to see it?12:11
*** shyamb has joined #openstack-ansible12:13
*** joshualyle has joined #openstack-ansible12:15
*** errr has quit IRC12:17
*** errr has joined #openstack-ansible12:18
*** joshualyle has quit IRC12:21
*** shyamb has quit IRC12:32
*** nicolasbock has joined #openstack-ansible12:34
noonedeadpunkjrosser: do you have any idea how we can set template_id here https://review.opendev.org/#/c/710245/3/tests/roles/bootstrap-host/templates/user_variables_magnum.yml.j2 while it's created here https://review.opendev.org/#/c/708097/6/tasks/magnum_resources.yml?12:36
noonedeadpunkoh, sorry, clean forgot that you are not available today12:36
*** joshualyle has joined #openstack-ansible12:37
*** pcaruana has joined #openstack-ansible12:39
*** joshualyle has quit IRC12:42
openstackgerritMerged openstack/openstack-ansible master: Bump SHAs for master  https://review.opendev.org/70670812:45
noonedeadpunkalti_17: I think it may be linked as a separate page to https://opendev.org/openstack/openstack-ansible-os_magnum/src/branch/master/doc/source/index.rst12:45
*** fghaas has joined #openstack-ansible12:59
*** rh-jelabarre has joined #openstack-ansible13:00
*** nicolasbock has quit IRC13:11
*** udesale_ has joined #openstack-ansible13:13
*** nicolasbock has joined #openstack-ansible13:15
*** udesale has quit IRC13:15
*** errr has quit IRC13:17
*** errr has joined #openstack-ansible13:18
*** udesale_ has quit IRC13:33
*** mgariepy has quit IRC13:33
*** joshualyle has joined #openstack-ansible14:15
*** errr has quit IRC14:17
*** errr has joined #openstack-ansible14:18
*** joshualyle has quit IRC14:19
CeeMacIs there any way that adding new compute/storage nodes into an OSA environment can break the ability to deploy instances in the rest of the environment?14:23
CeeMaci thought the problem was localised to the new nodes I put in yesterday, but now I'm getting the same issue on the existing nodes too :|14:24
*** oligau has quit IRC14:26
*** oligau has joined #openstack-ansible14:27
CeeMaci've enabled debug on nova-compute like melwitt suggested incase os-brick is having issues, but I don't see any errors from os-brick, just nova-compute14:30
*** dave-mccowan has joined #openstack-ansible14:32
*** jawad_axd has quit IRC14:48
*** jawad_axd has joined #openstack-ansible14:53
*** errr has quit IRC14:54
*** jawad_axd has quit IRC14:57
*** nicolasbock has quit IRC14:58
*** nicolasbock has joined #openstack-ansible14:59
*** this10nly has quit IRC15:00
*** this10nly has joined #openstack-ansible15:01
strattaohas anyone ever had issues running l3_ha with OVS+DVR? I see that l3_ha = True is hard coded in the os_neutron role.15:02
strattaoand has been for... well... for a really long time. We were running into an issue and one of the recommendations we ran across was to disable l3_ha because it performs poorly at scale. Was looking for a knob to tune in OSA, but now I'm having second thoughts...15:06
*** this10nly has quit IRC15:10
strattaoI know jrosser doesn't use ovs+dvr, jamesdenton hopefully isn't around. I haven't seen any chatter in the backlogs since the cloudnull days... Anybody have any recent experience with ovs+dvr at a large scale?15:10
*** joshualyle has joined #openstack-ansible15:12
*** joshualyle has quit IRC15:12
strattaoCeeMac, you aren't running ovs+dvr are you?15:15
jamesdentonthe l3_ha piece only impacts the snat routers, if you use em15:19
strattaoyeah, we use the snat routers, but when so many ports show DOWN, no  traffic flowing, etc. disabling the l3 ha was one of the things we have been trying out. When I went to add a code override, seeing the default l3_ha=True in there gave me pause because we typically trust the OSA defaults to be sane. Just wanted to get some other opinions before we decide we really are special snowflakes with our workloads.15:26
strattaoand jamesdenton, when do you ever rest?15:27
jamesdentoni lurk15:28
jamesdentonwe try to avoid DVR, routers, vxlan, etc if we can. most performant and reliable are provider networks+vlans15:29
jamesdentonbut that requires planning15:29
*** alti_17 has quit IRC15:30
strattaoyeah, probably not too many people hit up against the max network vlan limit on a regular basis...15:31
CeeMacstrattao: OVS no DVR currently15:34
strattaothx CeeMac15:35
CeeMacnw15:36
strattaowell, how about this OVS question then, when OVS is DEAD, what is the best process to get things started again? I came in this morning and saw in the logs that OVS is dead, in all three of our network nodes running the agent containers... not good. It still isn't entirely clear to me the ramifications of restarting neutron-openvswitch-agent vs ovs-vswitchd. And do you restart all at once, one at a time?15:37
strattaoReboot all the network nodes? :D15:37
strattaoI guess that's probably more of a question for the openstack-neutron channel, but any thoughts jamesdenton, CeeMac?15:40
jamesdentonum, so OVS should never DIE15:40
jamesdentonthat's indicative of a bigger issue, i'm afraid15:40
jamesdentonmy observation in the last few weeks of troubleshooting Stein, is that if you restart openvswitch-switch (ovs process) then it wipes the flows and your traffic will stop15:41
jamesdentonyou will need to restart the neutron ovs agent to rebuild that and gets things going again15:41
jamesdentonit is rare to need to restart ovs proper. the agent, too, for that matter. but if you ever need to restart OVS, restart the agent 2nd15:41
strattaoyeah, we're working on our root OVS issues :) still have that issue we've been tracking down since last week15:42
jamesdentonright. certainly, if your ovs is dying then that can cause connectivity issues15:44
jamesdentonanything interesting in the logs? /var/log/openvswitch/ovs-vswitchd.log or neutron agent log?15:44
strattaofor sure! We still haven't figured out what the root cause is. There are a lot of timeouts in the logs. The big errors seem to be things like ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 10 seconds, disconnecting. same for br-provider and br-int in the ovs-vswitchd.log15:50
strattaoIn the ovs-dbserver.log plenty of ERR|tcp:127.0.0.1:{PORT}: no response to inactivity probe after 5 seconds, disconnecting15:50
jamesdentonubuntu? centos?15:59
strattaoubuntu15:59
jrosserCeeMac: re sharing networks with specific projects have you seen this? https://docs.openstack.org/neutron/train/admin/config-rbac.html16:00
*** cshen has quit IRC16:00
*** jawad_axd has joined #openstack-ansible16:01
noonedeadpunkcores, can we merge https://review.opendev.org/#/c/708083/5 to fix and set rocky to EM?16:02
*** jawad_axd has quit IRC16:05
*** ivve has quit IRC16:14
*** macz_ has joined #openstack-ansible16:16
*** gyee has joined #openstack-ansible16:16
CeeMacjrosser: yes, i had the rocky version on a link somewhere.  i've used rbac before for access_as_shared for 'transport' network connections to customer MPLS networks, just not for access_as_external16:22
CeeMacso i wasn't sure if there was a difference in how to set it up and add interfaces etc16:23
CeeMacfor the record, I've got the FW instance working with floating IP now so I don't need to worry about the shared network for the moment.  Althrough I like the idea of it so I'll add that to my list of "things I'd like to get done when I get 5 minutes!"16:32
*** joshualyle has joined #openstack-ansible16:40
*** DanyC has quit IRC16:42
*** DanyC has joined #openstack-ansible16:43
jamesdentonCeeMac what did you do to get it working?16:43
*** DanyC has quit IRC16:47
CeeMacchanged the outbound nat rule from an 'any' source to 'lan-net' source16:47
*** DanyC has joined #openstack-ansible16:47
CeeMaccuriously enough16:47
*** fghaas has quit IRC16:50
*** pcaruana has quit IRC16:56
noonedeadpunkso need votes on https://review.opendev.org/#/c/703379 and https://review.opendev.org/#/c/706709/5 :)16:59
*** spatel has joined #openstack-ansible17:01
spateljamesdenton: are you there?17:01
jamesdentoninteresting. was that source port change unrelated, then?17:01
jamesdentonspatel yes but headed to lunch while i can!17:02
jamesdentonwhats up?17:02
spatelCould you please give me this command output from your CPU pinning vm "lstopo-no-graphics --no-io --no-legend --of txt"17:03
spatelIf you have one..17:03
spatelI am seeing very big issue with CPUTopology in my cloud17:03
spatelI have compare with AWS and they have real and true CPU topology compare to what i am seeing in my cloud17:04
jamesdentoni don't have one at the moment, sorry17:04
spateljamesdenton: no worry when you come back just ping me17:04
jamesdentoni had to tear down my lab to prepare for move17:04
spateloh!! no worry if anyone else here.. i would appreciate17:05
*** rpittau is now known as rpittau|afk17:09
*** tosky has quit IRC17:09
*** DanyC_ has joined #openstack-ansible17:29
*** DanyC has quit IRC17:32
*** DanyC_ has quit IRC17:34
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-ansible17:35
spatelHow do i tell openstack to use "<cache mode='passthrough'/>" in kvm for virtual machine?18:26
strattaohey jamesdenton when you get back from lunch, I could really use some help. neutron-openvswitch-agent is reporting that OVS is dead in two of my network nodes...18:37
jamesdentonin the logs?18:43
jamesdentonand systemctl status openvswitch-switch says what? can you run ovs-vsctl show and get output?18:43
strattaoyeah ovs-vsctl show works and returns port info, systemctl status openvswittch-switch looks normal18:46
strattaoas far as I can tell, OVS is actually up... no idea why neutron-openvswitch-agent thinks otherwise18:46
strattaothere are errors in the ovs logs that there is no response to inactivity probe after 10 seconds, disconnecting18:53
strattaoI want to try and increase that inactivity probe timeout, but I haven't figured out where that even gets set.18:54
*** dasp_ has joined #openstack-ansible18:55
*** mgariepy has joined #openstack-ansible18:56
*** dasp has quit IRC18:57
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add magnum tempest test  https://review.opendev.org/71024519:04
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Use debian OpenStack repos  https://review.opendev.org/70628519:14
jamesdentonstrattao ovs-vsctl set controller <bridge> inactivity_probe=<millisecs>19:15
jamesdentonthere is mention of it here: https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/tshooting/tshoot-mcp-openstack/ovs-timeouts.html19:16
jamesdenton"Insufficient OVS timeouts causing instance traffic losses"19:16
openstackgerritMerged openstack/openstack-ansible stable/rocky: Drop virtualenv pip package for CI  https://review.opendev.org/70808319:24
*** EmilienM is now known as EvilienM19:30
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/train: Fix federation scenario assurances os_user usage  https://review.opendev.org/71025619:31
openstackgerritMerged openstack/openstack-ansible stable/rocky: Set fixed version for networking-calico  https://review.opendev.org/70337919:37
openstackgerritMerged openstack/openstack-ansible stable/rocky: Bump SHAs for stable/rocky  https://review.opendev.org/70670919:37
*** DanyC has joined #openstack-ansible19:37
CeeMacjamesdenton: It was weird, I thought because the nat was bound to the wan interface with any source it was basically double NATing which rewrote the source port on the reply19:39
*** joshualyle has quit IRC19:40
jamesdentonwell, glad it's fixed!19:40
CeeMacMe too!19:41
CeeMacI fixed my nova/cinder issue too so quite a productive day!19:42
jamesdentoni like those fridays19:42
openstackgerritMagnus Bergman proposed openstack/openstack-ansible-haproxy_server master: Add missing X-Forwarded-Prot for extra_lb_tls_vip_addresses  https://review.opendev.org/71051519:49
*** Neurognostic has joined #openstack-ansible19:55
*** DanyC has quit IRC19:56
strattaojamesdenton, thank you for that find. Since the default was 10 seconds I bumped it up to 30 seconds, and it is taking longer to fail now! However, I think that neutron-openvswitch-agent will still eventually complain that OVS is dead. The neutron-openvswitch-agent logs are still following the same pattern as before.20:05
*** jawad_axd has joined #openstack-ansible20:06
*** jawad_axd has quit IRC20:10
*** DanyC has joined #openstack-ansible20:14
*** Neurognostic has quit IRC20:32
*** DanyC has quit IRC20:33
*** joshualyle has joined #openstack-ansible20:45
joshualyleI destroyed all the containers on a infra node and am attempting to rebuild them now that I've upgraded the host from 16.04->18.04 but I keep getting "msg": "'dict object' has no attribute 'ansible_hostname'" on the new rabbit container. Galera appears to have setup correctly but rabbit won't setup correctly. Any ideas?20:46
joshualyleI've cleared the entries for all of the containers and the host from /etc/openstack_deploy/ansible_facts20:47
joshualyleappears that it didn't work if I limited the install to just that one infra node and its containers? Maybe it gets the info about the other nodes during the play20:53
jrosserquite often you need to include localhost in the —limit20:57
*** nicolasbock has quit IRC21:07
joshualyleweird jrosser, thanks. I'll give it a try21:16
jrossertake care with --limit, some tasks run across the various hosts distributing ssh keys and so on21:21
joshualyleI don't typically run with --limit, I'm only giving it a try because these upgrades are taking forever and you mentioned it might be an option when we talked about rebuilding from scratch yesterday21:22
joshualylebut thanks for the heads up. I can always fallback to setup-everything without --limit21:22
jrosseryou can run the playbook for each service individually21:23
jrossertake a quick look in the setup-<...>.yml playbooks and you'll see they just include a bunch of others21:23
*** spatel has quit IRC21:35
*** spatel has joined #openstack-ansible21:48
*** spatel has quit IRC22:01
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add magnum tempest test  https://review.opendev.org/71024522:01
*** tosky has joined #openstack-ansible22:24
strattaojamesdenton I don't see a way to programmatically add the of_inactivity_probe. At least, I assume that override would have to live in the plugins/ml2/openvswitch_agent.ini config, but the only OSA overrides seem to be for the ml2_conf.ini22:38
strattaowait, I missed the neutron_openvswitch_agent_ini_overrides sitting there right in front of my face!22:42
jamesdentonnot sure if that's an override setting or an actual ovs bridge setting22:46
*** idlemind has joined #openstack-ansible22:50
strattaojust tested it in my test region, and it populated the correct value in the plugins/ml2/openvswitch_agent.ini config! I love OSA22:50
strattaoverified the probe limit was set correctly with an ovs-vsctl list controller <bridge>22:52
*** EvilienM is now known as EmilienM22:56
jamesdentonvery good, very good23:01
*** macz_ has quit IRC23:44
openstackgerritMerged openstack/openstack-ansible-os_swift master: Use py2 shebang for centos distro installs  https://review.opendev.org/70524723:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!