Tuesday, 2020-06-30

*** markvoelker has quit IRC00:02
*** cshen has joined #openstack-ansible00:46
*** cshen has quit IRC00:50
*** cshen has joined #openstack-ansible00:59
*** Berthe01 has quit IRC01:02
*** cshen has quit IRC01:04
*** mgariepy has quit IRC01:29
*** mgariepy has joined #openstack-ansible01:43
*** gyee has quit IRC02:04
*** mubix has quit IRC02:36
*** vblando has quit IRC02:36
*** Open10K8S has quit IRC02:37
*** Open10K8S has joined #openstack-ansible02:37
*** mubix has joined #openstack-ansible02:39
*** vblando has joined #openstack-ansible02:40
*** udesale has joined #openstack-ansible03:52
*** markvoelker has joined #openstack-ansible04:23
*** markvoelker has quit IRC04:28
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-ansible04:33
*** rh-jelabarre has quit IRC04:50
*** luksky has joined #openstack-ansible06:00
*** miloa has joined #openstack-ansible06:05
*** NewJorg has quit IRC06:19
*** markvoelker has joined #openstack-ansible06:24
*** NewJorg has joined #openstack-ansible06:25
*** markvoelker has quit IRC06:29
*** cshen has joined #openstack-ansible06:54
*** yolanda has joined #openstack-ansible07:26
*** sshnaidm|afk is now known as sshnaidm|ruck07:28
*** tosky has joined #openstack-ansible07:28
*** stingrayza has joined #openstack-ansible07:35
*** also_stingrayza has quit IRC07:38
admin0morning07:43
admin0in one platform, still on rocky, neutron plugin base has -vpnaas listed.. but i don't see the vpn button in horizon ..07:46
admin0and nothing failed in the deployment07:47
admin018.1.8 tag07:47
*** arkan has joined #openstack-ansible08:06
arkanhi guys, I have my running openstack, everything was ok. I wanted to install Load Balancer from this page https://docs.openstack.org/openstack-ansible/12.2.6/install-guide/configure-lbaas.html08:11
arkanso I added the code for v2, and now in Horizon my vms disappears08:12
arkanand some warning messages appeared08:12
arkanI'm using openstack-ansible "train"08:13
*** arkan has quit IRC08:15
*** arkan_ has joined #openstack-ansible08:16
arkan_and in "/project/" url it says "Something went wrong!, unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator."08:16
arkan_when I ran the os-neutron-install.yml I did not have errors08:17
arkan_I also tried to run os-horizon-install.yml after I ran os-neutron-install.yml and also I did not have any errors08:18
CeeMacadmin0: did you also add 'horizon_enable_neutron_vpnaas: True' to user_variables?08:25
*** markvoelker has joined #openstack-ansible08:25
CeeMaci think there are some kernel specific modules to be added too iirc08:26
CeeMaci started looking at this in my test cluster but didn't get around to validating it by setting up an actual VPN08:26
*** markvoelker has quit IRC08:30
*** ioni has quit IRC08:38
*** masterpe has quit IRC08:38
arkan_I removed the code from https://docs.openstack.org/openstack-ansible/12.2.6/install-guide/configure-lbaas.html for load balancer v2 and re-run os-neutron-install.yml and now my vms appeared08:40
arkan_I will try to install octavia https://docs.openstack.org/openstack-ansible-os_octavia/train/configure-octavia.html08:41
*** namrata has joined #openstack-ansible08:43
*** masterpe has joined #openstack-ansible08:47
masterpeLast week we did a upgrade from rocky to stein to OSA version 19.1.0, now when we deploy a neutron router, it sometimes does not get it configured competently.08:47
jrosserarkan_: that is a very old set of documenation08:47
arkan_jrosser: I did not know about it https://docs.openstack.org/openstack-ansible/12.2.6/install-guide/configure-lbaas.html08:48
jrosserarkan_: that is from when the lbaas was part of neutron, which does not exist any more. these days you should be looking at the octavia service08:48
masterpeIs there a known bug that causes this?08:48
arkan_jrosser: is this ok https://docs.openstack.org/openstack-ansible-os_octavia/train/configure-octavia.html ?08:48
*** arkan_ is now known as arkan08:49
jrosserarkan_: yes and there are 3 parts there, you need a provider network for octavia, the service to be setup, and then you need the loadbalancer VM image (amphora)08:50
jrosserso theres a few more moving parts than just deploying the API endpoints08:50
arkanjrosser: is the page sufficient to follow, or should I read other docs, related to installing the vm image (amphora)08:51
arkanjrosser: setting the provider net is easy08:51
jrosserthe OSA page is not instructions for octavia really08:52
jrosserthe os_octavia role can deal with it all for you i think08:52
*** ioni has joined #openstack-ansible08:52
jrosserif you want to understand a working deployment then there is an 'octavia' scenario in the OSA all-in-one08:53
arkanjrosser: I mean the default values in the os-role is sufficient08:53
arkanjrosser: ok I will check08:53
arkanjrosser: thank you as always08:53
jrosseryou have the choice of os_octaiva doing the provider network and amphora image for you, or you can make that setup yourself08:54
jrosserdepends what is appropriate in your environment08:54
arkanjrosser: I want to choose os_octavia to do it for me08:57
arkanI can provide in openstack_user_config.yml the provider network for LB08:57
arkanjrosser: so If I provide the provider network for octavia and run os_octavia, is this sufficient ?08:59
jrosserthe network config is here https://github.com/openstack/openstack-ansible-os_octavia/blob/master/defaults/main.yml#L326-L35408:59
jrosserand it will use br-lbaas to connect to octavia containers on the infra hosts so you'll need to create those09:00
jrosserand its up to you however you want vlan or flat or other networking type on the network/compute nodes09:00
arkanjrosser: I will create "- network: " (I will use vlan) section in openstack_user_config.yml and set the needed values from what you've given me, and run os-octavia ansible role09:02
namrataHi, I am at stable/train branch and doing a fresh installation and and when I run os-placement-install.yml my installation fails on Task Create database of serviceTASK [os_placement : Create database for service] ********************************************************************************************************************fatal:09:06
namrata[alice_placement_container-d59c2be8]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'None' has no attribute 'name'\n\nThe error appears to be in '/etc/ansible/roles/os_placement/tasks/db_setup.yml': line 28, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe09:06
namrataoffending line appears to be:\n\n  block:\n    - name: Create database for service\n      ^ here\n"}09:06
*** sshnaidm|ruck has quit IRC09:14
*** sshnaidm has joined #openstack-ansible09:23
*** cshen has quit IRC09:29
*** cshen has joined #openstack-ansible09:30
admin0CeeMac, i think i have not done that :) .. will do and check09:36
*** sshnaidm has quit IRC09:42
arkanjrosser: I set the required vars, now I want to add octavia in the inventory in order to run os-octavia09:42
arkanjrosser: is it ok to run openstack-ansible setup-infrastructure.yml --syntax-check ?09:43
CeeMacadmin0: good luck :)09:43
CeeMacyou'll need to run the horizon playbook again after setting the variable of course09:43
arkanjrosser: If I run os-octavia it will result "skipping: no hosts matched", because it is not in the inventory09:44
admin0CeeMac, does it appear under  neutron_plugin_base: -vpnaas ..09:44
admin0or it should be - neutron_vpnaas09:45
CeeMac- vpnaas'09:46
admin0main.yml says # Please add the 'vpnaas' to the neutron_plugin_base list .. so i think i did it correctly .. ran the neutron and the horizon .. still don't see the vpn under network09:46
CeeMacyes, plugin list looks fine. and you added the horizon variable too?09:47
admin0yep09:47
admin0horizon_enable_neutron_vpnaas: True09:47
CeeMacdo you have anything set for openstack_host_specific_kernel_modules:09:48
CeeMacjust wondering if its failed on a dependency09:48
admin0if I do this: grep -ri horizon_enable_neutron  /etc/ansible/roles/*        . i do not see that variable anywhere09:49
admin0all i see is _fwaas and _lbaas09:49
admin0 grep -ri horizon_enable  /etc/ansible/roles/* | grep vpn -- no hits09:49
CeeMacadmin0: https://pasted.tech/pastes/2cba21450a3d416e2f92736a088722b534294fae09:50
CeeMacis what I had set09:50
CeeMaclet me dig back through my notes a second09:51
CeeMacI was using this as a reference : https://docs.openstack.org/openstack-ansible-os_neutron/rocky/configure-network-services.html09:52
admin0i was in the same page :)09:52
CeeMacstep 3 has the kernel overrides :)09:52
admin0ERROR! the playbook: openstack_hosts-config could not be found .. wrong tags i guess09:54
*** sshnaidm has joined #openstack-ansible09:55
CeeMacah09:57
CeeMacone minute09:57
*** markvoelker has joined #openstack-ansible09:58
CeeMacfound my docs, there is some work to do, just need to dive into a meeting then will pull together something for you admin010:00
admin0thanks CeeMac  .. will wait10:01
*** markvoelker has quit IRC10:03
*** mgariepy has quit IRC10:04
*** mugsie has quit IRC10:07
*** mugsie has joined #openstack-ansible10:10
CeeMacadmin0: https://pasted.tech/pastes/09e022aef5eea50ad7f25e5217f96b6b895641b910:23
CeeMacpretty sure thats a condensed list of what I had to do to get it installed10:23
CeeMacactually you can skip the other neutron settings at the top there, I should have pruned them out10:24
CeeMachttps://pasted.tech/pastes/2881be560e445801598cd14b7c831b8a4a06c21010:25
CeeMacis better10:25
admin0CeeMac, thanks .. i will follow this10:38
admin0quick quesiton .. is this workaround only for rocky .... if i  want to do vpn for the latest one, is this the same workaround ?10:39
CeeMacadmin0: I think maybe just rocky10:54
CeeMacI haven't tried any newer install yet10:55
CeeMacAlso not sure of the neutron xaas plugins now, they lost a lot of support on them10:55
*** udesale_ has joined #openstack-ansible11:04
*** udesale has quit IRC11:06
*** NewJorg has quit IRC11:08
*** NewJorg has joined #openstack-ansible11:14
namrataCan somebody help me with this error while installation of os_placement-install.yml I am seeing11:20
namratahttp://paste.openstack.org/show/795347/11:20
namratai am working on stable/train branch11:21
*** markvoelker has joined #openstack-ansible11:55
*** gregwork has quit IRC12:00
*** rh-jelabarre has joined #openstack-ansible12:06
arkanjrosser: I updated my inventory to include octavia container host, then I ran openstack-ansible setup-hosts.yml but it failed, I log the errors, and I can see it's using openvswitch12:09
arkanopenstack_user_config.yml: http://paste.openstack.org/show/795349/12:10
arkanuser_variables.yml: http://paste.openstack.org/show/795350/12:10
arkanerror log: http://paste.openstack.org/show/795348/12:10
*** mgariepy has joined #openstack-ansible12:15
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible master: Add integrated test for Ubuntu Bionic using ML2/OVN driver  https://review.opendev.org/73301712:25
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible master: Add integrated test for Ubuntu using ML2/OVN driver  https://review.opendev.org/73301712:26
jrosserarkan: i dont think it is using openvswitch. did you create br-lbaas?12:32
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible master: Add integrated test for Ubuntu using ML2/OVN driver  https://review.opendev.org/73301712:32
arkanjrosser: I added it in the openstack_user_config.yml, I will use the same method and use br-ext as I've done it for the external network provider12:33
jrosserarkan: you know that you have to create any bridges that you specify in openstack_user_config?12:34
arkanjrosser: yes, and I think I made a mistake because br-lbass is not created12:35
arkanjsrosser: I will create it now12:35
jrosserok12:35
mgariepyanything that needs review ?12:35
jrossermgariepy: train is broken somehow, i don't know where we are with centos8 really12:36
arkanjrosser: I will create the bridge only on the machines that use octavia specified in my openstack_user_config.yml (my controller node), and I will leave the other node (compute node) without creating the bridge12:39
namratajrosser mgariepy can you help me with the failed installation of os-placement-install.yml this is the error:http://paste.openstack.org/show/795347/12:41
jrossernamrata: if you comment out the no_log here https://github.com/openstack/openstack-ansible-os_placement/blob/master/tasks/db_setup.yml#L3212:44
jrosserand rerun with -vv then you should see which variable is empty12:44
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: WIP - lxd containers  https://review.opendev.org/73859912:51
*** namrata has quit IRC12:55
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Remove support for Ubuntu 16.04  https://review.opendev.org/73797813:00
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Do not install python2 or its dev package on Ubuntu Bionic hosts  https://review.opendev.org/73797913:00
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Do not install python2 or its dev package on Ubuntu Bionic hosts  https://review.opendev.org/73860713:02
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Remove support for Ubuntu 16.04  https://review.opendev.org/73860813:05
*** namrata has joined #openstack-ansible13:28
arkanjrosser: to include octavia13:42
arkan1. Add bridge br-lbaas (ok)13:42
arkan2. openstack-ansible setup-hosts.yml (success)13:42
arkan3. openstack-ansible os-octavia-install.yml (failed)13:42
arkan4. openstack-ansble os-neutron-install.yml (success)13:42
arkan5. openstack-ansible os-octavia-install.yml (success)13:42
arkanThe problem was after adding 'lbaas' it could not be found by octavia install, so I need to update13:42
arkanneutron plugin which is  /etc/neutron/plugins/ml2# vi ml2_conf.ini by running os-neutron-install.yml13:42
arkanAnd after running it (os-neutron-install.yml), the file updated to includ lbaas vlan13:42
arkan[ml2_type_vlan]13:42
arkannetwork_vlan_ranges = provider:40:40,lbaas:60:6013:42
ioni you only need to create br-lbaas on controller13:44
jrosserif you edit the neutron ml2_conf.ini your changes will be removed next time you run the os_neutron playbook13:44
*** tacco has joined #openstack-ansible13:45
ioniyou want lbaas network to be flat and vlan ?13:45
arkanjrosser: this is what I've done as I mentioned in step nr. 413:45
taccohi there.. anyone knows where the volumes_attached from "openstack server show " comes from?13:45
arkanioni: it's vlan with id 6013:45
ioniwhat type of network do you have now into the cloud13:45
taccobecause i have a volume that still apear there.. but is marked as deleted in the database and does not exists anymore13:45
arkanioni: I can see in the 'Admin' section there is something new added13:46
arkanProject: service, Network Name: br-lbaas, br-lbaas-subnet 192.168.60.0/2413:47
arkanioni: DHCP Agents: 0, Shared: No, External: No, Status: Active, Admin State: UP, Availability Zones: -13:48
ioniyou do not need br-lbass on compute nodes13:48
ionithe idea of br-lbaas is to have containers with ips from the same network/subnet13:48
ionicontainers ip(api octavia) needs to talk with amphora instances that have ips from that subnet13:49
arkanioni: In my openstack_user_config.yml, I included octavia only to use Infra host not compute nodes13:49
arkanioni: also compute node does not have this bridge13:49
arkanioni: it's only available on the controller node13:50
jrosserarkan: octavia created virtual machines that run the loadbalancers13:50
ionibr-lbaas configuration https://paste.xinu.at/uTm/13:50
jrosserthe octavia service is on the infra nodes, the lxc on those nodes connects to br-lbaas13:50
ioniin your case is tagged 60 not 669 like in my configuration13:50
jrosserbr-lbaas must be connected to the provider network that goes to the compute nodes so that the octavia service can commicate with the loadbalancer VM13:51
arkanioni: yes and I'm not using bond, and it's only on my controller node (my infra host)13:51
jrosserarkan: if you have used the values from os_octavia defaults/main.yml without overriding them, it is set up for a flat network, not vlan13:52
jrosserhttps://github.com/openstack/openstack-ansible-os_octavia/blob/master/defaults/main.yml#L33113:52
arkanI've used mine13:52
arkanI overridden some of them13:52
jrosseri'm not sure you need to be adjusting ml2.conf them13:52
jrosser*then13:52
jrosserit's just another vlan tag on your existing vlan type provider network?13:53
ioniarkan: https://paste.xinu.at/G2Attd/13:56
*** spatel has joined #openstack-ansible14:24
namratajrosser the error of placement_api install after -vv is http://paste.openstack.org/show/795361/14:28
arkanjrosser, ioni: sorry I was outside with the workers14:32
jrosserarkan: you have an example from ioni now of a proper config for using the existing vlan provider network14:33
arkanioni: openstack_user_config.yml http://paste.openstack.org/show/795363/14:33
*** cshen has quit IRC14:33
arkanioni: user_variables.yml http://paste.openstack.org/show/795364/14:34
arkanioni: thanks I will use your example14:36
arkanbut I will change vlan seg: to be 60, yours is 66914:36
*** miloa has quit IRC14:37
arkanalso the subnet, 192.168.60.0/24 instead of 10.100.103.0/2414:37
namratajrosser http://paste.openstack.org/show/795361/ is this a bug14:59
jrossernamrata: i don't know :) you will need to determine which variable is undefined using normal ansible debugging techniqes15:03
jrosserbut we run that code many many times in CI so it does work with a standard AIO config15:03
namrata@jrosser I removed no_log and tried with -vvv15:04
namrataI am not able to find the variable15:04
noonedeadpunknamrata: maybe you have _oslodb_databases overriden somewhere?15:06
*** andrea15 has quit IRC15:07
spatelnamrata: if you run with -vvvv might give you some hints15:07
namratanoonedeadpunk this is a fresh installation I haven;t made any changes15:08
namrataspatel the error log which I pasted is with -vvv only15:08
jrossernamrata: everyone is trying to show you the different things available to help debug15:09
noonedeadpunkso obviously it fails here https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/master/tasks/db_setup.yml#L3015:09
jrosserif there isnt much useful info in the output then you should increase the debug/verbose level and try again15:09
noonedeadpunkbut for placement we have name explicitly set https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/master/tasks/main.yml#L5015:10
*** mgariepy has quit IRC15:34
*** gyee has joined #openstack-ansible15:38
arkanIoni: I've done your setup for octavia, just some modifications for vlan seg. http://paste.openstack.org/show/795375/15:44
arkanthen I've run os-neutron-install, os-octavia-install and os-horizon-install15:45
arkannow I can see the load balancer in my horizon15:45
arkanbut it is giving Alerts: "Error: Unable to retrieve load balancers."15:46
arkanioni: and this is the added part in the network section http://paste.openstack.org/show/795376/15:49
*** sshnaidm has quit IRC15:51
*** sshnaidm has joined #openstack-ansible15:53
arkan[wsgi:error] [pid 18497:tid 139770147088128] [remote 172.29.236.10:50092] urllib3.exceptions.15:54
arkanMaxRetryError: HTTPConnectionPool(host='172.29.236.210', port=9876): Max retries exceeded with url: /v2.0/lbaas/loadbalancers?project_id=d491172334d84e1c8ad5d231b1935e20 (Caused by NewConn15:54
arkanectionError('<urllib3.connection.HTTPConnection object at 0x7f1eaa57a3c8>: Failed to establish a new connection: [Errno 111] Connection refused',))15:54
arkanalso16:00
arkan[remote 172.29.236.10:54524] No policy rules for16:00
arkan service 'load-balancer' in /openstack/venvs/horizon-20.1.1/lib/python3.6/site-packages/openstack_dashboard/conf/octavia_policy.yaml16:00
noonedeadpunk#startmeeting openstack_ansible_meeting16:03
openstackMeeting started Tue Jun 30 16:03:45 2020 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.16:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:03
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:03
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:03
noonedeadpunk#topic office hours16:03
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"16:03
noonedeadpunk\o/16:03
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-plugins master: WIP add lxd support  https://review.opendev.org/73865716:04
noonedeadpunkso16:05
noonedeadpunkhave we backported everything we wanted regarding focal?16:05
noonedeadpunkjrosser: btw regarding this... I was thinking if we should just rename containers in dynamic_inventory?16:07
jrossero/ hello16:07
jrosseroh you mean _ vs - ?16:08
noonedeadpunkyeah16:08
jrosseroh yes there is probably something nice we need to do there16:08
jrosseri sort of put it to one side because theres a bigger piece of work to do with the group names too16:08
jrosseri did a reasonable look across the master/ussuri branches for stuff to backport for focal16:10
jrosserit looked in reasonable shape16:10
noonedeadpunkI also had a brief look and didn't find anything at glance16:10
noonedeadpunkok, cool, so will place a bump today16:11
noonedeadpunka nasty thing is that train bump seems broken...16:11
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Bump SHAs for stable/train  https://review.opendev.org/73546916:11
*** mgariepy has joined #openstack-ansible16:12
noonedeadpunkmaybe it's because of some ceph-ansible update...16:12
jrosseryes i was looking at that a bit16:15
jrosserand i was also thinking that there was something not right with radosgw authentication16:16
noonedeadpunkI thougnth it might be swift tempest plugin, but it seems that's a part of tempest core.16:16
jrosserthere is ceph 14.2.10 i think16:16
jrosserthats a very recent release which is what could have broken16:16
jrossertheres an error in the tempest log about 404 instead of 401, then you can see the auth trail looking all reasonable through radosgw logs and keystone log16:17
jrosserand then i didnt really know what was breaking16:17
*** sshnaidm is now known as sshnaidm|ruck16:18
noonedeadpunkLike interesting part that tempest was expecting unathorized anyway16:18
noonedeadpunkbut got 404 instead.16:19
noonedeadpunkso this thing wasn't expected to finish with 200 or smth16:19
noonedeadpunkso yeah, probably ceph has changed smth in terms of return codes and now it's not fully compatible with swift in terms of return codes...16:20
jrosserthere was an interesting string in one of the log messages - don't have that open right now16:23
jrosserand it took me to a RH bug about ResellerAdmin needing to be in the radosgw config16:24
noonedeadpunkso here where it fails https://opendev.org/openstack/tempest/src/branch/master/tempest/scenario/test_object_storage_basic_ops.py#L6516:26
*** mgariepy has quit IRC16:27
noonedeadpunkso we're checking that things fail here eventually16:27
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: WIP - lxd containers  https://review.opendev.org/73859916:28
jrosserand that code is not touched in several years16:29
jrosseri don't know if it's possible to pin ceph back to the previous point release and see what happens16:30
noonedeadpunkyeah, so I think it's ceph who just returns 404 instead of 401 for unathorized16:30
*** cshen has joined #openstack-ansible16:30
noonedeadpunkbtw I guess that's not happening for O16:31
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: WIP - lxd containers  https://review.opendev.org/73859916:32
jrosserright, which is wierd, as you'd expect all the changes on N to be backports from O16:32
*** cshen has quit IRC16:35
noonedeadpunkI don't see anything obvious here https://github.com/ceph/ceph/compare/v14.2.9...v14.2.1016:38
*** udesale_ has quit IRC16:39
*** mgariepy has joined #openstack-ansible16:41
noonedeadpunkalso interesting if old bump is going to pass now https://review.opendev.org/#/c/735469/16:42
*** jbadiapa has quit IRC16:51
openstackgerritMerged openstack/openstack-ansible-os_rally master: Do not install development packages on the target  https://review.opendev.org/73145916:52
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-os_rally stable/ussuri: Do not install development packages on the target  https://review.opendev.org/73867616:52
noonedeadpunk#endmeeting17:06
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://bit.ly/2SAcGAn"17:06
openstackMeeting ended Tue Jun 30 17:06:01 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:06
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-06-30-16.03.html17:06
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-06-30-16.03.txt17:06
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2020/openstack_ansible_meeting.2020-06-30-16.03.log.html17:06
ioniarkan: run the haproxy install playbook, pretty sure it doesn't have an entry point for octavia service17:22
*** namrata has quit IRC17:36
arkanioni: ok17:36
*** cshen has joined #openstack-ansible17:43
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: WIP - lxd containers  https://review.opendev.org/73859917:45
arkanioni: now it has, but horizon it is not working as expected, some pages give 503 Service Unavailable17:48
arkanioni: in my controller node with journalctl I'm seeing this: "2020-06-30 17:52:20.697 568784 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Parsing physical_interface_mappings failed: Value eno1 in mapping: 'provider:eno1' not unique. Agent terminated!"17:53
arkanioni: this is my current file in neutron container http://paste.openstack.org/show/795395/17:55
jamesdentonwhat does linuxbridge_agent.ini look like17:56
arkanioni: path /etc/neutron/plugins/ml2/ml2_conf.ini17:56
arkanjamesdenton: where to find this file ? in which container ?17:57
arkanjamesdenton: in ls /etc/neutron17:59
jamesdentonyes17:59
arkanonly these are :17:59
arkanapi-paste.ini  api-paste.ini.original  neutron.conf  plugins  rootwrap.conf  rootwrap.conf.original  rootwrap.d17:59
jamesdentonin plugins/ml218:00
arkanls /etc/neutron/plugins/ml2/18:00
arkanml2_conf.ini18:00
arkanthere is no such file18:00
jamesdentonis this the neutron-server container? or a compute?18:00
arkanit's a container18:01
arkanroot@controller1-neutron-server-container-24d28d2718:01
arkanin the controller node18:01
jamesdentonok, so look for the file where you got that log message from18:01
jamesdentonprobably on the controller itself, not inside the neutron-server container18:02
arkanjamesdenton: that message was in the journalctl from the controller node (not container)18:02
arkanjamesdenton: yes it is there18:03
arkanin the controller node (not container)18:03
arkanI will paste it18:03
jamesdentonyep18:03
arkanhttp://paste.openstack.org/show/795397/18:04
arkanI can see there are 2 mappings18:04
arkanthis is because of my - network section in my openstack_user_config.yml18:04
arkanthe provider network is using it18:05
arkanand lbaas is also using it18:05
jamesdentonyes, you won't be able to use eno1 for lbaas. but you could use eno1.100, if vlan 100 was configured18:05
arkanas Host_bind_override: "eno1"18:05
arkanI made eno1.6018:05
jamesdentonyou can configure, say eno1.100, map it to lbaas, and then the corresponding neutron neutron for lbaas mgmt would be a 'flat' network18:05
arkanit's also configured in the router18:06
jamesdentonfor lbaas?18:06
arkanyes18:06
arkanbut I need to update my openstack_user_config18:06
jamesdentonok, so change your mapping to eno1.60 on the openstack_user_config.yml18:06
arkanjamesdenton: this is my current config http://paste.openstack.org/show/795398/18:07
jamesdentonchange host_bind_override: "eno1" to host_bind_override: "eno1.60" for the lbaas section18:07
arkanyes18:08
arkanafter that what to run ?18:08
arkanos-neutron-install.yml18:08
arkanos-octavia-install.yml18:08
arkanos-horizon-install.yml18:08
arkanshould I run also haproxy-install.yml?18:09
jamesdentonos-neutron-install18:20
arkanjamesdenton: os-neutron-install run successfully18:20
jamesdentoncheck your agent log now18:20
arkanI'm running now os-octavia-install18:20
arkanok18:20
arkanjournalctl -xf18:21
arkanthere are no errors18:21
-openstackstatus- NOTICE: Due to a flood of connections from random prefixes, we have temporarily blocked all AS4837 (China Unicom) source addresses from access to the Git service at opendev.org while we investigate further options.18:21
arkanjamesdenton: I've finished installing os-neutron, os-octavia, haproxy-install, os-horizon18:27
arkannow some of the working pages are not working18:27
arkanlike floating ips18:28
arkanok, now it worked18:28
arkanmaybe it needed time to work18:28
jamesdentonyes, it can take a little bit of time to be ready18:29
arkanthe Load Balancer page, gave 1 warning "Warning: Policy check failed."18:29
arkanand "Error: Unable to retrieve load balancers."18:29
jamesdentonthere may be a hint in octavia logs18:29
arkanI will check18:29
jamesdentonyou might also try cli18:29
arkanI saw in journalctl one big red error18:30
admin0does osa work on ubuntu 20.04 ?18:30
arkanhttp://paste.openstack.org/show/795399/18:31
arkanjamesdenton: this is a repeated error (periodic)18:32
arkanjamesdenton: do you have some cli commands to get more info about why there is error ?18:33
arkanI'm inside the container controller1_octavia_server_container-2e9b4f3118:34
jamesdentonnot really, no. can you try to spin up a vm normally?18:34
arkanI will check18:34
arkanjamesdenton: hummm now I can see strange message in the Launch Instance dialog box18:35
arkanin the Source* section18:36
arkan"There are no allowed boot sources. If you think this is wrong please contact your administrator."18:36
jamesdentonsee if you can upload an image, and create a flavor18:36
arkanjamesdenton: my openstack was fully functional, just today I wanted to install os-octavia18:36
arkanafter I tried it, then I got sort of errors18:37
jamesdentonwhat did that process look like? Sorry, i think missed that here18:37
arkanjamesdenton: no even I can not create flavors18:38
jamesdentonare you familiar w/ the openstack cli? can you try there?18:38
arkanjamesdenton: yes18:38
arkanI had created 7 vms with terraform, and other vms18:39
arkanplaying around18:39
arkanalso teleport for teleporting with ssh between the machines18:40
arkanand bastion vm18:40
arkanall were working18:40
arkantoday I needed to install octavia18:40
arkanbecause I need a load balancer18:40
arkanI also installed kubernetes18:40
arkanI don't know why it's giving errors after installing os-octavia18:41
jamesdentonultimately, what did the process look like for installing octavia? updating openstack_user_config.yml and maybe user_overrides.yml and then installing which playbooks?18:41
jamesdentoni assume setup-hosts and setup-infra were run, too? to create the octavia api container?18:41
arkanjamesdenton: on moment I will explain you18:42
arkan1. I've added the bridge18:43
arkanhttp://paste.openstack.org/show/795401/18:43
arkan2. I have the variables into user_variables.yml http://paste.openstack.org/show/795403/18:44
arkan(section octavia above)18:44
arkan3. I added the cidr, network section, and octavia-infra_hosts http://paste.openstack.org/show/795404/18:46
arkan4. I ran inventory/dyanmic_inventory.py18:46
arkan5. I ran openstack-ansible setup-hosts.yml18:47
arkanthen I tried to run other os-* like os-neutron, os-octavia, os-horizon, os-keyston, haproxy-install18:48
arkanjamesdenton: I did not run setup-infrastructure.yml18:49
arkanjamesdenton: should I run setup-infrastructure.yml ?18:50
jamesdentonno, its ok18:50
jamesdentonthere is some mismatch here for the octavia bits, around vlan tagging and whatnot. i need find a working example18:50
jamesdentoni'm just a bit tied up at the moment18:50
arkanok18:50
arkanjamesdenton: ioni gave me this one https://paste.xinu.at/G2Attd/ and I adapted it to my situation18:52
jrosserarkan: i think you are getting several different sets if info telling you different things, and your config is now mixed up18:53
arkanjrosser: I did not copy paste the values, I adapted it in my current setup, like I've changed vlan, cidr, network name, ...etc18:54
jrosserright, but now you have eno1.60 and type=vlan, thats not right because it's a mix of flat and vlan18:55
arkanbut I don't know why octavia is not satisfied today18:55
jamesdentondid ioni happen to provide the openstack_user_config.yml provider network definition, too?18:55
arkanI don't have flat18:55
jrosseryes18:55
arkanjrosser: I did not use flat in my setup18:56
jrosserif you don't want to use a flat network then don't specify eno1.60 becasue that is a de-tagged vlan 60 -> flat18:56
jrosserand previously the issue was that you had eno1 twice in the neutron config18:57
jrosserimho this can be simpler18:57
jrosserjust one vlan provider network, that you already have18:57
jrosserand make the *neutron* lbaas network be a vlan with segmentation_id 6018:58
arkanjrosser: hummm this is strange eno1.60 about flat, look into my current netplan file http://paste.openstack.org/show/795405/18:58
noonedeadpunkjrosser: yeah old train bump also fails with 14.2.10 ceph...18:58
jrosserarkan: you have host_bind_override: "eno1.60"18:59
arkanall the vlan section are used through sub interface eno1.10, eno1.20 ...etc and they are tagged, and communicate through the router18:59
jrosserthat is the neutron pyhsical network config18:59
arkanjrosser: so that one "host_bind_override" is for flat ?18:59
jrosserno, it's a pyhsical interface for neutron19:00
arkanok19:00
arkanI used it for the provider network and I specified "eno1" which is working19:00
*** cshen has quit IRC19:00
arkanbut I don't know what to do now for octavia lbaas19:01
arkanI have only one cable NIC to the router19:01
*** sshnaidm|ruck is now known as sshnaidm|bbl19:02
jrosserthats fine19:02
arkanjrosser: my physical network in my controller is "eno1"19:02
noonedeadpunkand we have no option to use 14.2.9 at the moment:(19:03
arkanthis is where the cable is attached19:03
*** Julos20 has joined #openstack-ansible19:03
jrosserarkan: on the controller you have br-lbaas which is hooked to vlan 60 through your netplan config19:03
*** cshen has joined #openstack-ansible19:03
arkanyes19:03
jrosserthe octavia containers are connected to that becasue of *part* of the config in openstack_user_config19:04
arkanyes19:04
jrossercontainer_interface: "eth14" for example, makes eth14 in the octavia container get connected to br-lbaas19:04
ebbexnoonedeadpunk: https://review.opendev.org/#/c/673778/ should I be seeing two "admin" projects in my project list, and three domains (Default, heat, magnum) on a fresh train install?19:04
*** yolanda has quit IRC19:05
jrosserthen quite separately neutron needs to know how to connect octavia VM to the lbaas network, i.e vlan 6019:05
jrosserand there are many ways to do that depending on your setup, which is why there is no "correct" answer for this19:05
jrosserarkan: do you have the same eno1 on your compute node?19:06
arkanjrosser: so  container (eth14) <--------> br-lbaas bridge19:06
arkanjrosser: yes19:06
arkanI mean compute node is another physical host (separate host)19:07
arkanalso it has one NIC which is named "eno1"19:07
jrosserok, so you already have a config in neutron for the vlan provider network19:07
arkanthe provider network is running perfectly19:07
arkanno doubt about it19:07
jrosserbut you can't have a second config for it19:08
arkani can even create floating ips19:08
arkanand access the machines19:08
arkanthe problem is happened today when I tried to install octavia by adding vlan 6019:08
jrosseri think you need to remove some of the config from the openstack_user_config for the lbaas network19:08
jrosserso that it does not try to make a second neutron physical network mapping to eno1, becasue you already have one19:09
arkanwhat string to use for host_bind_override: ?19:09
jrosserthat field is optional19:09
arkanyeah ?19:09
arkangood19:10
arkanthen I can delete it19:10
arkanso now we will have container eth14 <--------connected to ------> br-lbaas19:10
arkanand br-lbaas is inside the controller node (infra)19:11
jrosserthere is an annotated example file https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.example#L153-L16219:11
arkangreat this is optional19:12
jrosserarkan: and you should also have eth14 <----> controller br-lbaas <----> compute node br-ext(vlan tag 60)19:12
arkanjrosser: aha19:13
arkanjrosser: I though that I needed only for controller node19:13
arkanok, I will create br-lbaas in compute node also19:14
jrosserno no :)19:14
jrosserbr-lbaas is only on the controller19:14
arkanjrosser: aha gr819:15
jrosserwhen octavia creates a amphora VM (i think) there is a network ID in the config file (i forget exactly what)19:15
jrosserneutron needs to know what to connect the vm to19:15
jrosserso in your case the neutron network/subnet would be on physical network name "provider" and segmentation id 6019:16
arkanyes19:16
arkanno19:16
arkanmy working OS is using "provider" network with vlan seg. 4019:17
arkanNetwork Type: vlan19:18
arkanPhysical Network: provider19:18
arkanSegmentation ID: 4019:18
arkanName19:18
arkanprovider19:18
arkanID19:18
arkancbc85df0-b906-4912-9b2a-bfb7dc60ccc719:18
arkanthis is the one that I create floating ips from19:18
jrosserthat is a logical network created in neutron called "provider"19:19
jrosserand it's using a physical network also (confusingly) called "provider"19:19
jrosseryou can have another logical network on a different vlan tag on the same neturon physical network19:19
arkanand I have it in my netplan19:20
arkanand it's configured in my router also as tagged vlan 4019:20
arkanjrosser: and how to do it ? :)19:21
noonedeadpunkebbex: 2 admins yes, but they're gonna be in diferent domains. https://docs.openstack.org/heat/pike/install/install-ubuntu.html19:21
noonedeadpunkas well as 2 domains19:22
noonedeadpunkcant instantly recall about magnum, but probably yes19:22
*** viks____ has quit IRC19:22
jrosserarkan: you allow vlan 60 on whatever trunk interface between the compute and controller19:22
noonedeadpunkbut I believe there's a bug somewhere, as some permissions are missing19:22
noonedeadpunkso like http://paste.openstack.org/show/794358/ fixes them (thanks to dpaclt)19:22
jrosserthen you make sure that the neutron octavia network is using physical network "provider" and segmentation id 6019:22
jrosserand you make sure that network is referenced in the octavia config file19:23
noonedeadpunkBut I didn't manage to verify which exact is missing and why19:23
arkanjrosser: my question is now I've deleted "host_bind_override" for lbaas and ran os-neutron19:24
noonedeadpunkbtw we can totally simplify that now https://review.opendev.org/#/c/673778/7/tasks/heat_service_setup.yml19:24
arkancan I keep my old setup and use lbaas without using host_bind_override ?19:24
arkanbecause now I saw that if I have 2 places in my openstack_user_config that use host_bind_override with the same value, then I will see errors19:26
jrosserhopefully i've just shown you that you can remove one of those19:27
arkanand now I know that host_bind_override is a physical interface (the NIC where the cable is attached)19:27
jrosserthat ends up in the physical network mappings in the neutron config19:28
arkanjrosser: yes and it can not have values to the same value like provider:eno1 and lbaas:eno1, this will give errors19:29
ebbexnoonedeadpunk: Thanks, I'll look into these docs some more :) Never actually managed to get magnum to work, and never seen two "admin" projects before train (messed up some scripts i had adding stuff to the now conflictingly named "admin" project), but the domain stuff seems legit.19:29
noonedeadpunkit's really pretty weird comparing to other project and totally not readable in terms of that patch...19:30
noonedeadpunkwe can improve somehow though...19:30
noonedeadpunkbut really worth checking these permissions as dpaclt claimed they fixed heat for him19:31
noonedeadpunkmagnum from the other side is pretty tough itself and you'll need to pickup it's specific version to make it work...19:31
noonedeadpunk(you can probably ask guilhermesp about it)19:32
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/train: Bump SHAs for stable/train  https://review.opendev.org/73835719:33
noonedeadpunkjrosser: I've just blacklisted this swift test as for now... ^19:33
*** cshen has quit IRC19:42
arkanjrosser: I caught this error on controller node, I think I need to create the bridge also on compute node19:43
arkanon host compute1 for vnic_type normal using segments [{'id': 'cac5b1a5-b0e9-4883-9a27-fe301bce6c0c', 'network_type': 'vlan', 'physical_network': 'lbaas', 'segmentation_id': 60, 'network_id': 'a681a436-36ea-44de-b1b8-672c45d936a0'}]19:43
jrosser'physical_network': 'lbaas',19:44
jrosserthat doesnt look right according to what you pasted before19:44
arkanyes, why it is physical ?19:44
arkannow my config is modified19:44
arkanand I ran os-neutron, os-octavia, haproxy-install and os-horizon19:45
arkanI've deleted the line host_bind_override: "eno1"19:45
arkanso why it's saying "physical_network"19:46
jrosserthere is a logical network defined in neutron19:47
arkanand also it's mentioning the compute node19:47
jrosseropenstack network list19:47
jrosseropenstack network show <id-of-octavia-network>19:47
jrosseryou need to get that correct as well19:47
arkanjrosser: my openstack now got sick, now it is not working19:47
arkanit was working before running os-octavia19:48
arkannow it throws some bunch of error with ImportError: No module named queue19:48
arkanin my horizon, I can see lbaas-mgmt19:49
arkanhummm19:49
arkanNetwork Type: vlan19:49
arkanPhysical Network: lbaas19:49
arkanSegmentation ID: 6019:49
jrosseryes this is wrong19:49
arkanit's there with physical network19:49
arkanI will delete it19:49
jrosseryou don't have a pysical network called lbaas19:50
jrosseroverride the variables in os_octavia defaults/main.yml to represent the configuration you have19:50
arkanwell my openstack horizon, is destroyed now19:51
arkan:)))19:51
arkana big black circle19:51
arkanok, now it got logged in again after some refreshings19:52
arkanI overrided with these19:54
arkanoctavia_enable_anti_affinity: True19:54
arkanoctavia_legacy_policy: True19:54
arkanoctavia_provider_network_name: lbaas19:54
arkanoctavia_provider_network_type: vlan19:54
arkanoctavia_provider_segmentation_id: 6019:54
arkanoctavia_container_interface: "eth14"19:54
arkanoctavia_container_network_name: "lbaas_address"19:54
arkanoctavia_management_net_dhcp: "False"19:54
arkanoctavia_management_net_subnet_cidr: 192.168.60.0/2419:54
arkanoctavia_management_net_subnet_allocation_pools: "192.168.60.20-192.168.60.200"19:54
*** dave-mccowan has quit IRC19:55
arkanis there a magic value that I need to use ? :)19:56
arkanon octavia container I caught an error19:57
arkan2020-06-30 19:57:17.991 18300 ERROR octavia.controller.worker.v1.controller_worker [-] Failed to create an amphora due to: Failed to build compute instance due to: Network a681a436-36ea-44de-b1b8-672c45d936a0 could not be found. (HTTP 400) (Request-ID: req-2467c352-69fe-4a80-bac9-0e163f471e8d): octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: Network a681a436-36ea-44de-b1b8-672c45d936a0 could not19:58
arkan be found. (HTTP 400) (Request-ID: req-2467c352-69fe-4a80-bac9-0e163f471e8d)19:58
arkanbecause I've deleted it the network19:58
*** dave-mccowan has joined #openstack-ansible19:58
arkannow octavia container is crying19:59
arkanif I will give it the network, then I will cry19:59
*** noonedeadpunk has quit IRC20:02
guilhermespyeah i mean.. i'd say working with magnum is an adventure :P20:03
guilhermespwe used to be pointing to specific shas for whatever version mangum works20:03
guilhermespwhat we should keep in mind is that we need to use coreos images ( can generate them here https://github.com/stackhpc/magnum-terraform/blob/master/upload-coreos.sh )20:04
*** yolanda has joined #openstack-ansible20:11
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/ussuri: Bump SHAs for stable/ussuri  https://review.opendev.org/73868920:19
*** sshnaidm|bbl has quit IRC20:34
*** sshnaidm|bbl has joined #openstack-ansible20:35
*** cshen has joined #openstack-ansible20:35
*** sshnaidm|bbl is now known as sshnaidm|afk20:46
*** jbadiapa has joined #openstack-ansible20:54
*** markvoelker has quit IRC21:32
*** rh-jelabarre has quit IRC21:37
*** luksky has quit IRC21:55
*** spatel has quit IRC21:57
*** Berthe01 has joined #openstack-ansible21:57
*** Julos20 has quit IRC22:00
openstackgerritErik Berg proposed openstack/openstack-ansible-os_heat master: add description to domain during creation  https://review.opendev.org/73871222:33
*** tosky has quit IRC22:57
*** markvoelker has joined #openstack-ansible23:33
*** markvoelker has quit IRC23:38
*** markvoelker has joined #openstack-ansible23:48
*** cshen has quit IRC23:53
*** markvoelker has quit IRC23:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!