Friday, 2020-12-11

*** macz_ has quit IRC00:14
*** cshen has quit IRC02:04
*** spatel has joined #openstack-ansible02:38
*** spatel has quit IRC02:42
-openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to enable support for Git protocol v2, downtime should be less than 5 minutes02:53
*** pto has joined #openstack-ansible05:31
*** pto has quit IRC05:31
*** pto has joined #openstack-ansible05:32
*** evrardjp_ has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** pto has quit IRC05:38
*** pto_ has joined #openstack-ansible05:38
*** pto_ has quit IRC05:45
*** pto has joined #openstack-ansible05:46
*** lemko0 has joined #openstack-ansible05:47
*** lemko has quit IRC05:50
*** lemko0 is now known as lemko05:50
*** maharg101 has joined #openstack-ansible07:34
*** maharg101 has quit IRC07:38
*** jbadiapa has joined #openstack-ansible07:42
*** pcaruana has joined #openstack-ansible07:56
*** pcaruana has quit IRC07:57
*** rpittau|afk is now known as rpittau07:57
*** pcaruana has joined #openstack-ansible08:04
*** pcaruana has quit IRC08:04
*** pcaruana has joined #openstack-ansible08:05
*** pcaruana has quit IRC08:05
*** pcaruana has joined #openstack-ansible08:12
*** pcaruana has quit IRC08:12
*** cshen has joined #openstack-ansible08:15
*** evrardjp has quit IRC08:15
*** evrardjp has joined #openstack-ansible08:16
*** miloa has joined #openstack-ansible08:19
*** newtim has quit IRC08:19
*** andrewbonney has joined #openstack-ansible08:22
*** maharg101 has joined #openstack-ansible08:26
admin0\o08:27
*** tosky has joined #openstack-ansible08:47
noonedeadpunkmornings09:42
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_keystone master: Add security.txt file hosting to keystone  https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/76643709:59
*** macz_ has joined #openstack-ansible10:09
*** macz_ has quit IRC10:14
admin0applying octavia in prod, i get fatal: [c3_octavia_server_container-692ae034]: FAILED! => {"attempts": 5, "changed": false, "msg": "openstacksdk is required for this module"}10:16
admin010:16
admin0what does that mean ?10:16
admin0do i need to rebuild utility . or the containers itself10:17
noonedeadpunkcan you at least post what task is failing?10:18
admin0sorry .. https://gist.github.com/a1git/1c2364a0fd8b7573908d6c87c66c5d9710:19
noonedeadpunkuh10:20
noonedeadpunklet me patch this out....10:21
admin0it didn't happen in the lab . but applying from lab -> prod ;(10:22
admin0is it an easy patch that i can manually apply for the time being ?10:22
noonedeadpunkyep, you will be aple to copy cherry-pick command from gerrit "Download" menu and just run it inside /etc/ansible/roles/os_octavia10:23
admin0ok .. waiting for it :)10:31
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Delegate info gathering to setup host  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/76669310:35
noonedeadpunkadmin0: ^10:35
admin0thanks ..re-running10:37
noonedeadpunkjrosser: (not critical, when you will be around - monday or smth) - do you recall issue with live migrations and apparmour when we were about to patch some rule for nova but this has been fixed either by libvirt package or smth?10:37
noonedeadpunkor we've added some variable to provide custom path for libvirt....10:38
admin0noonedeadpunk, nope: https://gist.githubusercontent.com/a1git/e689f3cea602e718c40a91c715c9fe1a/raw/f242a882359e7444037557bf2e9702f4232ec509/gistfile1.txt10:41
admin0failed with something else10:42
noonedeadpunkah10:42
noonedeadpunkfair10:42
admin0fair :D ? lol10:42
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Delegate info gathering to setup host  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/76669310:44
noonedeadpunkyeah, my bad, fixed10:44
*** macz_ has joined #openstack-ansible10:45
*** macz_ has quit IRC10:49
admin0re-running10:52
*** kukacz has quit IRC10:53
admin0noonedeadpunk, still something there: https://gist.githubusercontent.com/a1git/8e448e3c54ebfff81fde6e4bb048ccd9/raw/c4099077d34e312532590b367d08c746b2328c80/gistfile1.txt10:56
admin0is there any info you want me to do/check noonedeadpunk ?11:08
*** kukacz has joined #openstack-ansible11:16
noonedeadpunkhm11:19
noonedeadpunksorry, was a bit busy11:19
admin0if there is any checks you want me to do .. manual ones for path or etc, i can do it11:21
admin0before it fails again, for you to validate anything11:21
noonedeadpunkand what oa version are you running?11:23
noonedeadpunk*osa11:24
noonedeadpunkis it train?11:24
*** odyssey4me has joined #openstack-ansible11:37
admin0ussuri11:45
*** SecOpsNinja has joined #openstack-ansible11:47
SecOpsNinjahi to all. where can i find the relation list/information regaridng nova and other databases? im having gosts vms that i cant delete using dashboard and cli so the only solution whould be to delete it from database11:49
*** sshnaidm is now known as sshnaidm|off11:59
noonedeadpunkadmin0: hm. can you kindly check that you don't have overrides of `octavia_service_setup_host_python_interpreter` or `openstack_service_setup_host_python_interpreter` or just `ansible_python_interpreter` for host/group_vars11:59
admin0none noonedeadpunk11:59
noonedeadpunkah wairt12:01
noonedeadpunklol12:02
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Delegate info gathering to setup host  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/76669312:03
noonedeadpunkI'm just blind12:03
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Fix libsystemd version for Centos  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/76603012:14
*** pto has quit IRC12:21
*** pto_ has joined #openstack-ansible12:21
*** newtim_ has joined #openstack-ansible12:51
noonedeadpunkadmin0: does ^ work for you now?13:13
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Make container_bridge optional for provider networks  https://review.opendev.org/c/openstack/openstack-ansible/+/76517413:16
admin0noonedeadpunk, trying now sorry .. had been in 2 meetings13:19
admin0how do i tell bootstrap-ansible to download/clone only the os_octavia role13:23
admin0i accidently deleted the branch13:23
*** pto_ has quit IRC13:24
*** pto has joined #openstack-ansible13:24
*** macz_ has joined #openstack-ansible13:25
noonedeadpunkyou cant13:28
noonedeadpunkyou can just manually clone and checkout to specific sha though13:28
*** pto has quit IRC13:29
*** pto has joined #openstack-ansible13:29
*** macz_ has quit IRC13:30
admin0noonedeadpunk, its running now :)13:33
admin0with the latest patch13:33
admin0and -vvv13:33
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts stable/ussuri: Fix libsystemd version for Centos  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/76670713:42
admin0noonedeadpunk, no errors with that step and it passed ( but 1 container failed) so redoing that one13:43
admin0noonedeadpunk, i logged into reviews .. can't i just +1 it ?13:45
*** pto has quit IRC13:46
noonedeadpunkI thoink you should be able to +1 it13:46
admin0i don't see that option in the ui13:49
noonedeadpunkThere should be blue Reply button at13:50
noonedeadpunkonce you press it opens window with voting and you can leave some comment as well13:50
*** miloa has quit IRC13:56
*** spatel has joined #openstack-ansible14:01
openstackgerritlikui proposed openstack/openstack-ansible-os_nova master: Reuse the docs deps to benefit from constraints  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/76673714:05
*** cshen has quit IRC14:25
*** kukacz has quit IRC14:46
*** rpittau is now known as rpittau|afk15:15
*** cshen has joined #openstack-ansible15:19
*** kukacz has joined #openstack-ansible15:20
spatelHow difficult is to run ipv6 based openstack cloud?15:21
noonedeadpunknot so much I'd say. But all depends on the network setup you have. As it might start from just adding ipv6 network to the public net and end with neutron-bgp15:22
*** cshen has quit IRC15:23
*** macz_ has joined #openstack-ansible15:26
spatelneutron-bgp?15:28
spatelIf i setup IPv6 public IP pool to my router and setup VLAN like ipv4, then all i need to just create ipv6 network and thats all right?15:29
*** cshen has joined #openstack-ansible15:29
*** macz_ has quit IRC15:31
noonedeadpunkyeah, it would be the easiest way:) you can even add just another ipv6 pool to existing public network, in case you allow VMs just to get public IP15:31
noonedeadpunk(ie some might require to have floating ip for having public ipv4 - then public vlan is not accessible for vms and you need some workaround to have)15:32
spatelI don't have floating IPs, I have pure VLAN provider15:33
spatelDo i need to tell OSA anywhere for ipv6 or its transparent ?15:33
noonedeadpunkyou do in case you want API endpoints to be accessible via ipv415:34
noonedeadpunkthen you need to have ipv6 VIP in addition to ipv415:34
noonedeadpunk*to be accessible via ipv615:35
noonedeadpunkI think we need some doc page describing this...15:37
noonedeadpunkso sepcificly I can recall about `extra_lb_tls_vip_addresses` (which should be your public ipv6 vip) and I think rewrite keepalived_instances to include ipv6 into `vips_excluded` key....15:46
spatelI don't think i need ipv5 endpoint15:46
spatelipv6*15:46
noonedeadpunkwe totally should make this cleaner and document....15:46
spatelall i need ipv6 public network for VM to access from public15:47
noonedeadpunkthen you should not need osa changes I think15:47
noonedeadpunkbut again - you know who is netwroking expert here :p15:47
spatelnoonedeadpunk: thank you! just trying to understand if i am missing something here. but look like its very simple as far as network handle ipv6 traffic15:53
spatelnoonedeadpunk: any word on Victoria release :)15:54
spatelsorry for pushing but its kind of holding my deployment :(15:54
noonedeadpunkI'd love to do it, but techincaly can't until gates are broken15:55
noonedeadpunkI really was ready on monday but then 8.3 released...15:55
noonedeadpunkhttps://review.opendev.org/c/openstack/openstack-ansible/+/766244 has failed in gates....15:56
noonedeadpunkso we need to recheck and pray actually...15:57
spatel:(16:02
admin0spatel, yes ..16:03
admin0just add the ipv6 like any ipv616:03
admin0with the gw in the router and the range in a vlan16:03
admin0and it will just work out of the box16:03
spatel+1 that is what i want16:04
admin0spatel, could we not have just used br-vlan.27 directly ?16:04
admin0instead of creating that v-eth-pair ?16:04
admin0moving from poc -> prod, I see packets going out of the box .. but not arriving back .. network guys say its OK in their end16:04
spatelYou can do that directly also in that case you need to create br-lbaas on each compute nodes and create lb-lbaas-mgmt network to tell use br-lbaas bridge to wire up amphora16:05
spatelIn short make sure you amphora directly talk to octavia container.. whatever technology you use..16:06
admin0aah ..  create br-lbaas on each compute nodes -- did not worked .. as the error is cannot add a bridge to a bridge . then used mgariepy method .. and it was all mixed up16:06
spatelYou need to tell neutron to attach that amphora to br-lbaas bridge instead of br-vlan16:07
spatelIn my method neutron create VLAN.27 port inside br-vlan and attach that to amphora16:08
spatelbut if you want separate bridge then just tell neutron create VLAN.27 port on br-lbaas bridge and attach that to amphora16:09
spatelFinally i created monitoring script to monitoring openstack control plane, 1. create VM 2. ping VM 3. destroy VM16:14
spatelevery 5 minute its running.16:15
*** chkumar|ruck is now known as raukadah16:21
admin0spatel, you should also create a canary in every hypervisor and monitor that canary16:24
admin0canary = very lightweight small instance16:24
spatelwhy every hypervisor ?16:24
spatelThis script it to just check my control plane is functional, mainly rabbitMQ16:25
noonedeadpunkspatel: well I'd rather leveraged rally for that - it can also give you SLA report17:15
noonedeadpunkand you can define more complex tests - eventually anything supported by tempest (which is almost everything I guess)17:16
noonedeadpunkhttps://rally.readthedocs.io/en/latest/overview/overview.html#use-cases17:18
noonedeadpunkit's useful to have vm on every HV in case you do eed to check if billing is correct I guess...17:19
spatelnoonedeadpunk: let me take a look17:20
spatelI wrote dirty script for my monitoring tool but sure will look into see how rally can fit inside my monitoring17:20
admin0well, if you are a public cloud and selling stuff, you want to be on top of the game .. being not reactive but proactive .. so you need to know that a customer has issue before they notify you via phone call/ticket17:21
admin0i come from a public cloud background . so its been a habit17:21
admin0so monitoring a small instance in the vm tells you that the hypervisor is good and that its networking agents are good and routing is good as well17:22
admin0this would be on top of your api monitoring to create, delete, instances, network, volume, attach volume also17:23
spatelWe are running private cloud so pressure is low but yes.. i do need smart way to check all components17:23
spateladmin0: are you talking about rally17:23
admin0nah17:24
admin0terraform does it just well17:24
admin0let me share what i use17:24
spateli need something interactive, create vm, grab IP address, try to ssh or ping, etc..17:25
spatelterraform is good for deployment but not sure about fetching IPs and do some task17:25
spatelnoonedeadpunk: rally looks good if we can put that metrics in influx/grafana :)17:28
*** dave-mccowan has joined #openstack-ansible17:31
admin0https://gist.github.com/a1git/1fddc213d030901051564fb01039a13e  - i use something like this to quick-test an aio // can be expanded17:31
spatellet me check17:32
*** gyee has joined #openstack-ansible17:33
spatelis this script part of your monitoring tool?17:34
spatellike alert folks about something wrong etc..17:34
admin0this is just to test .. if the output is good, means all apis are good  ..  we use zabbix in some, check_mk in some to monitor the canaries and do alerting17:35
admin0this is not a part of monitoring script .. it was just my someting to quick test a new osa install ..17:35
admin0but terraform is well documented to expand this17:35
spatelThat is what i did. my alert script create VM / Ping VM / Delete VM  (if it fail in any step report NOC)17:36
noonedeadpunkspatel: well I had results in prometheus, so I think you can put in influx as well17:36
spatelwe are heavily using terraform for all kind of application deployment. GCP/AWS/Tencent/AliCloud/Openstack (all these cloud providers we are running our stuff)17:37
spatelnoonedeadpunk: look like rally going to be my next goal to monitor API performance17:38
noonedeadpunkany core around - would be greate to merge https://review.opendev.org/c/openstack/openstack-ansible-galera_client/+/76577917:38
*** tosky has quit IRC17:46
admin0kleini, https://www.openstackfaq.com/openstack-octavia/17:53
openstackgerritMerged openstack/openstack-ansible-galera_client master: Deprecate openstack-ansible-galera_client role  https://review.opendev.org/c/openstack/openstack-ansible-galera_client/+/76577917:55
*** SecOpsNinja has left #openstack-ansible17:56
jrosseradmin0: it would be better to document the right things to put in used_ips and the dhcp allowed range to avoid a conflict17:57
admin0i tried to work it out .. but could not17:58
admin0they eventually overlap17:58
noonedeadpunkI will  be deploying octavia pretty soon - will try to follow our docs and will adjust them with missing parts17:58
*** yann-kaelig has joined #openstack-ansible17:59
jrosserthey should not, that makes it like there is a bug you have to workaround17:59
admin0when we have time, can we take a 10.0.0.0/24 example range and work it out17:59
jrosseri believe you have used_ips the wrong way round18:00
jrosserused_ips: "10.62.0.1,10.62.0.99"18:00
jrosser^ this says to OSA "do not use these addresses for container interfaces"18:01
admin0aha ..18:01
admin0jrosser, where do i send you a pizza ?18:01
jrosserthe clash is almost guaranteed combining tat with octavia_management_net_subnet_allocation_pools: 10.62.0.101-10.62.7.25018:01
admin0i got it now18:01
*** MickyMan77 has quit IRC18:02
jrosserawesome :)18:02
*** jbadiapa has quit IRC18:03
admin0so  basically octavia_management_net_subnet_allocation_pools = used_ips :)18:05
jrosserpretty much, yes18:05
jrosserin other circumstances you would add your routers/firewalls/other hardare to used_ips18:06
admin0updated .. thanks18:07
*** dave-mccowan has quit IRC18:12
*** dave-mccowan has joined #openstack-ansible18:16
*** dave-mccowan has quit IRC18:20
*** dave-mccowan has joined #openstack-ansible18:21
*** andrewbonney has quit IRC18:24
spatelI have random question related DHCP.  (we have 3 controller nodes means 3 DHCP in that case which one handover IP address to VM?)18:43
spateldoes dnsmasq DHCP shared lease database between 3 DHCP?18:44
*** maharg101 has quit IRC19:29
*** mgariepy has quit IRC19:58
*** dave-mccowan has quit IRC20:03
*** dave-mccowan has joined #openstack-ansible20:07
*** lemko3 has joined #openstack-ansible20:14
*** lemko has quit IRC20:14
*** lemko3 is now known as lemko20:14
*** cshen has quit IRC20:36
*** cshen has joined #openstack-ansible20:58
*** rfolco has quit IRC21:05
*** tosky has joined #openstack-ansible21:05
admin0spatel, if you are targeting public, then don't make controllers your network node .. what we do is make compute node also network node .. in that regard, any 1 node going down does not take your whole customers down.. and in that regard, you can have N number of dhcp per network .. ( in an old non osa openstack) we found upto 5 dhcp per network .. imagine a public cloud with 10,000+ tenants,  and if even if 10% is actively using it,    100021:18
admin0networks with 4-5 dhcp each = 5000 dhcp in the whole21:18
admin0we ran cron to calculate dhcp per network and delete > 221:18
admin0in osa you can fix that via user_variables21:19
spatelcurrently we have 3 controller node and they are running DHCP service21:19
admin0also if public services, also offer direct-dhcp public ip and not via floating IP21:20
admin0that way, instances can get direct dhcp public ip ..  and there is no need to create network, create router add 1:1 NAt etc21:20
admin0as those waste IP address21:20
admin0this way => https://www.openstackfaq.com/openstack-add-direct-attached-dhcp-ip/21:21
admin0you have to ensure that the .1 is added to the vlan specified in the router21:21
admin0this is a way that is used in public clouds21:21
admin0and customers are also happy ( cpanel/directadmin/voip licensing) that they get a direct public ip21:21
spatellet me understand what you trying to say21:23
admin0the only thing is, via direct dhcp method, unlike floating ip,  if they delete the instance, the ip is also gone21:23
spatelwhat is direct dhcp method?21:23
admin0:D21:23
admin0i just explained above :)21:23
admin0let me elaborate21:24
admin0ipv4 is scarce  you don't get it .. so yuo want to maxmize it .. now , via floating ip .. router = .1 .. then in our HA setup  .2 .3 .4 = dhcp ..  then customer router = .5 .. and then his floating ip = .6 .. another customer,  his router = .7,  his floating ip = .8 . .. so for every router you creae, you lose an IP address21:25
admin0via direct dhcp method, your router is .1, the instance get direct public IP .. no need to create network or add floating ip ..21:25
admin0that way, in terms of scale,  you are saving resources .. coz for every private network = 3 dhcp namespaces, 3 router in HA = lot of stuff in network21:26
admin0if you have 10,000 tenants , its 10,000 x 3 of routers and dhcp21:26
admin0thats huge21:26
admin0overhead21:26
admin0via the direct dhcp method, you get an IP in the instance directly .. no floating ip, no network creation,21:26
admin0its used in public cloud provider .. so not properly documented21:27
admin0 i started in public cloud, so i documented it21:27
spatelWe have Physical router is my openstack gateway for all VM21:27
spatelwe don't run L3 on software layer.21:27
admin0you are stil creating network right ?21:27
spatelYes21:27
admin0that spawns up 3 dhcp namespaces and 3 router namespaces in HA21:27
spatelNO21:28
admin0every (private network) will have a dhcp and a router to connect to ext-21:28
spatelOh wait.. i know what you saying..21:28
spatelwhen i create subnet it reserve 3 public IP DHCP namespace.21:29
spatelreading your doc21:30
spateladmin0: explain me this line (via direct dhcp method, your router is .1, the instance get direct public IP .. no need to create network or add floating ip )21:31
admin0:D21:31
spatelHow my instance get direct public IP?21:31
admin0let me try21:31
spatelthere must be DHCP somewhere in network21:31
admin0you have only 1-3 dhcp process for that specific public network21:31
admin0let me try to simplify21:31
admin0in a public cloud, the typical customer is who wants to do web hosting correct -- take a vm, and run a control panel like cpanel or directadmin .. he wants a server with IP address right ?21:32
admin0so without direct-dhcp, you create a tenant, then as teannt create 1. network, and then 2. router, then 3. assign floating ip and 4. assign ip -> instance21:33
admin0the end result is .. the instance is mapped to a public ip21:33
admin0with direct dhcp, you don't need to create networks or router, the customer gets direct IP21:33
admin0bro.. just trust me .. follow my notes .. and enjoy the "magic"  :D21:34
spatelOk i think i am following you now.21:34
admin0this is only done when you want to grow as public cloud provider and want to have like 10k tenants21:34
spatelIn my case i am running private cloud and all my tenants using shared subnet.21:35
admin0wtih 10k tenants, with 1 router reach, you are losing 1k public IP just for router, and then inviting 3x dhcp process for dhcp = 30,000 dhcp process and 30,000 routers21:35
admin0the end result is.. the vm gets a public iP21:35
admin0everything else remains the same21:35
spateltotally agreed21:35
admin0exactly .. so this direct dhcp method is not documented .. outside of running openstack at scale in a public domain, not even people know or think about it21:36
spatelinstead of direct DHCP why not create subnet and shared with tenants21:36
admin0i think the end result is the same isn't it ?21:37
admin0" why not create subnet and shared with tenants " document this man :)21:37
spatelWhen you create network/subnet you can tell it --shared or --private21:38
spatelif you shared then it will be visible to all tenants and they can use it to create VM21:39
spatelwe have handful tenant and project because of private cloud. but in public it would be mess to deal21:39
admin0my past 3 companies were public cloud providers :)21:40
admin0like at scale :)21:40
*** newtim_ has quit IRC21:41
admin0preparing myself for trove from monday :)21:41
admin0my belief is, if osa provides a project as stable, we ( operators) should be able to run it21:42
admin0trove -> designate -> magnum21:43
admin0you can give the dhcp method a try if you have an acceptane/lab env on how it works21:44
spatelAfter successfully deploy senlin (my next goal is to use designate)21:44
spateldoes trove support postgresql ?21:44
admin0designate with osa,  i already have it implemented in office21:44
admin0postgresql is my 1st goal21:44
admin0mariadb second21:44
admin0i think it does21:45
spatellet me know how it goes21:45
admin0btw, octavia is not complete..i found out that for https, and to have letsencrypt/zerossl, it needs babrican21:45
spateli want to do magnum but with vlan provider where my container get direct IP from my VLAN21:46
admin0so have to setup babrican, get https working and can finally mark it as done21:46
admin0the direct dhcp method i documented is exacly what it does.. provides a direct IP from the vlan21:46
admin0have a great weekend everyone21:47
spatelThank you man!!!21:48
spatelhave a great weekend and stay safe21:48
*** spatel has quit IRC22:05
*** dave-mccowan has quit IRC22:31

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!