Thursday, 2019-11-14

johnsomrm_work I am also seeing a bunch of grenade failures. They seem to all be nova or cinder failures outside our stuff.00:15
johnsomVery annoying00:15
johnsomI have seen:00:16
johnsom[ERROR] /opt/stack/new/grenade/projects/70_cinder/ SSH to the client did not work, something very wrong00:16
johnsomsetUpClass (tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON) [0.000000s] ... FAILED00:16
johnsomkeystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to ('Connection aborted.', OSError("(104, 'ECONNRESET')",))00:17
johnsomtempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip [69.753469s] ... FAILED00:17
johnsomAll on different jobs00:18
johnsomNone of which are related to the patches under test00:18
johnsomJust rechecked another grenade: {0} tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip [71.884997s] ... FAILED00:24
johnsomHere is a new one: {0} [98.141138s] ... FAILED00:31
johnsomurllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='', port=9696): Read timed out. (read timeout=60)00:32
*** yamamoto has joined #openstack-lbaas00:57
*** goldyfruit has joined #openstack-lbaas00:58
*** yamamoto has quit IRC01:02
johnsomLooks like neutron is just *slow* creating networks:01:06
johnsomINFO neutron.wsgi [None req-90e1fd30-b3df-4ab0-914e-39988864a1b7 tempest-RoutersIpV6Test-381827999 tempest-RoutersIpV6Test-381827999] "POST /v2.0/networks HTTP/1.1" status: 201  len: 769 time: 77.273855001:06
johnsomIt took 77 seconds to respond, tempest timeout was 6001:07
johnsomubuntu-bionic-rax-ord-0012796682 was the host, so known slow host01:09
rm_workALL of the ones i have seen were the nova no hosts issue01:10
rm_workwhich can fail on whatever test happens to come first that requires a VM I think01:10
johnsomWell, I at least feel better that I RCA'd one of my grenade failures01:11
*** ricolin has joined #openstack-lbaas01:25
*** ltomasbo has quit IRC01:50
*** yamamoto has joined #openstack-lbaas02:47
*** yamamoto has quit IRC02:49
*** yamamoto has joined #openstack-lbaas02:49
*** abaindur has quit IRC03:04
*** abaindur has joined #openstack-lbaas03:04
*** abaindur has quit IRC03:04
*** ramishra has joined #openstack-lbaas03:08
*** yamamoto has quit IRC03:11
*** yamamoto has joined #openstack-lbaas03:12
*** psachin has joined #openstack-lbaas03:38
*** yamamoto has quit IRC04:45
*** goldyfruit has quit IRC04:54
*** spatel has joined #openstack-lbaas05:15
*** yamamoto has joined #openstack-lbaas05:19
*** spatel has quit IRC05:20
*** yamamoto has quit IRC05:24
*** yamamoto has joined #openstack-lbaas05:27
*** ricolin has quit IRC06:00
*** gcheresh_ has joined #openstack-lbaas06:11
*** rcernin has quit IRC06:53
*** AlexStaf has quit IRC07:01
*** yamamoto has quit IRC07:04
*** ccamposr__ has joined #openstack-lbaas07:32
*** ccamposr has quit IRC07:35
*** yamamoto has joined #openstack-lbaas07:43
*** yamamoto has quit IRC07:48
*** maciejjozefczyk has joined #openstack-lbaas08:12
*** tesseract has joined #openstack-lbaas08:17
*** ivve has joined #openstack-lbaas08:20
*** maciejjozefczyk has quit IRC08:21
*** dmellado has quit IRC08:21
*** dmellado has joined #openstack-lbaas08:23
*** dmellado has quit IRC08:25
*** dmellado has joined #openstack-lbaas08:27
*** tkajinam has quit IRC08:29
*** AlexStaf has joined #openstack-lbaas08:30
*** rpittau|afk is now known as rpittau08:57
*** chungpht has quit IRC09:11
*** abaindur has joined #openstack-lbaas09:15
*** trident has quit IRC09:16
*** dmellado has quit IRC09:20
*** dmellado has joined #openstack-lbaas09:24
*** trident has joined #openstack-lbaas09:26
*** yamamoto has joined #openstack-lbaas09:40
*** spatel has joined #openstack-lbaas09:50
*** pcaruana has joined #openstack-lbaas09:54
*** spatel has quit IRC09:55
*** yamamoto has quit IRC10:06
*** yamamoto has joined #openstack-lbaas10:10
*** yamamoto has quit IRC10:15
*** gcheresh_ has quit IRC10:39
*** gcheresh_ has joined #openstack-lbaas10:43
*** henriqueof has joined #openstack-lbaas11:55
*** maciejjozefczyk has joined #openstack-lbaas11:57
*** maciejjozefczyk has quit IRC12:02
*** trident has quit IRC12:09
*** trident has joined #openstack-lbaas12:18
*** tesseract has quit IRC12:28
*** tesseract has joined #openstack-lbaas12:29
*** ccamposr__ has quit IRC12:51
*** psachin has quit IRC12:51
*** vishalmanchanda has quit IRC12:52
*** dulek has quit IRC12:52
*** amotoki has quit IRC12:52
*** coreycb has quit IRC12:52
*** openstackstatus has quit IRC12:53
*** ccamposr__ has joined #openstack-lbaas12:54
*** psachin has joined #openstack-lbaas12:54
*** vishalmanchanda has joined #openstack-lbaas12:54
*** dulek has joined #openstack-lbaas12:54
*** amotoki has joined #openstack-lbaas12:54
*** coreycb has joined #openstack-lbaas12:54
*** yamamoto has joined #openstack-lbaas13:16
*** yamamoto has quit IRC13:20
*** spatel has joined #openstack-lbaas13:30
*** maciejjozefczyk has joined #openstack-lbaas13:32
*** yamamoto has joined #openstack-lbaas13:33
*** spatel has quit IRC13:34
*** maciejjozefczyk has quit IRC13:39
*** henriqueof has quit IRC13:40
*** yamamoto has quit IRC13:58
*** psachin has quit IRC14:14
*** henriqueof has joined #openstack-lbaas14:20
*** goldyfruit has joined #openstack-lbaas14:30
*** yamamoto has joined #openstack-lbaas14:32
*** yamamoto has quit IRC14:38
*** gcheresh_ has quit IRC14:46
*** ricolin has joined #openstack-lbaas14:52
*** TrevorV has joined #openstack-lbaas14:56
*** gcheresh_ has joined #openstack-lbaas15:14
*** ccamposr has joined #openstack-lbaas15:26
*** maciejjozefczyk has joined #openstack-lbaas15:28
*** ccamposr__ has quit IRC15:29
*** goldyfruit_ has joined #openstack-lbaas15:43
*** goldyfruit has quit IRC15:45
*** ivve has quit IRC16:02
*** Trevor_V has joined #openstack-lbaas16:12
*** TrevorV has quit IRC16:15
*** ricolin has quit IRC16:20
*** AlexStaf has quit IRC16:28
*** KeithMnemonic has joined #openstack-lbaas17:00
*** yamamoto has joined #openstack-lbaas17:02
*** rpittau is now known as rpittau|afk17:04
*** yamamoto has quit IRC17:06
*** tesseract has quit IRC17:15
guilhermesphello all17:18
guilhermespdo we expect octavia to create neutron resources or just consume them?17:18
guilhermespfor example, users wants to attach a FIP into their LB, but you need previously to allocate a FIP to the project17:18
guilhermespusers that wants to create a LB with a fixed ip of a vxlan network, but we need to previously create a port for it17:19
johnsomOctavia does create an manage resources in neutron, but floating IPs in not one of them. If the user wants a floating IP they will have to attach it after the load balancer is created.17:19
johnsomAs for fixed IP vxlan, they can either create the load balancer directly specifying that, or they can pass in a pre-created neutron port.17:20
guilhermesphum  I see... I do see a LB with Error status and has a fixed ip defined, but it doesn't exists in the subnet17:20
guilhermespyep, pre-created port17:20
guilhermespif an user wants a fixed ip, it needs to create beforehand17:20
guilhermespthat makes sense then17:21
johnsomProvisioning_status error means the cloud failed in some action Octavia was requesting of it. I.e. neutron is broken, etc.17:21
johnsomYou can specify a fixed IP at LB create time, no need to pre-create it17:21
guilhermespalso, I've been seeing in all my funcional LBs the Operating Status as offline17:21
johnsomThat likely means either the load balancer is not currently passing traffic or your Octavia deployment is not working.17:22
johnsomIf the load balancer is working, check your lb-mgmt-net. It is likely that the amphorae cannot connect back to the health manager processes.17:22
guilhermespbut that's the thing... my listener responds to members, like it is working as expected, but status is offline17:23
johnsomYeah, ok, so your lb-mgmt-net is not working17:23
johnsomamphroae send heartbeats back to the controllers on UDP 5555. You should see messages in the health manager log if you enable debug logging on it.17:24
johnsomDEBUG [-] Received packet from ('', 46000) {{(pid=14074) dorecv /opt/stack/octavia/octavia/amphorae/drivers/health/}}17:24
*** ivve has joined #openstack-lbaas17:24
johnsomDEBUG octavia.controller.healthmanager.health_drivers.update_db [-] Health Update finished in: 0.00808617501752451 seconds {{(pid=16283) update_health /opt/stack/octavia/octavia/controller/healthmanager/health_drivers/}}17:25
guilhermespgotcha. I will do some debugging later on17:25
colin-if i have 10 spares and i tell octavia to loadbalancer amphora failover each of them, do the first five just eat the last five?18:01
johnsomNo, it will delete the amphora you list and the housekeeping process will boot a replacement fresh18:05
*** yamamoto has joined #openstack-lbaas18:13
*** maciejjozefczyk has quit IRC18:15
*** yamamoto has quit IRC18:18
*** baffle has quit IRC18:22
*** gcheresh_ has quit IRC18:34
*** yamamoto has joined #openstack-lbaas18:53
*** yamamoto has quit IRC18:58
*** henriqueof has quit IRC19:05
*** abaindur has quit IRC19:37
*** abaindur has joined #openstack-lbaas20:13
*** henriqueof has joined #openstack-lbaas20:26
*** abaindur has quit IRC20:28
*** abaindur has joined #openstack-lbaas20:28
openstackgerritMerged openstack/octavia master: Fix listeners with SNI certificates
*** gcheresh_ has joined #openstack-lbaas20:47
*** henriqueof has quit IRC21:10
*** henriqueof has joined #openstack-lbaas21:19
*** goldyfruit_ has quit IRC21:23
*** rcernin has joined #openstack-lbaas21:28
*** henriqueof has quit IRC21:33
*** henriqueof has joined #openstack-lbaas21:34
*** servagem has quit IRC21:37
rm_worksorrison: you had a chance to look at rebasing your patch on top of mine yet?21:42
sorrisonJust looking at it now21:42
sorrisonAbout to -1 you for your patch as found a few bugs :-)21:42
rm_workYeah please do: P21:42
rm_workOnly have minimal unit testing yet, was going to start on tempest today21:43
rm_workAnd it was basically just a giant copy/paste/find/replace job lol21:43
sorrisonyeah I picked up on that, looks good though21:46
*** gcheresh_ has quit IRC21:47
*** yamamoto has joined #openstack-lbaas21:48
*** goldyfruit_ has joined #openstack-lbaas21:55
*** henriqueof has quit IRC21:58
*** henriqueof has joined #openstack-lbaas22:00
*** henriqueof has quit IRC22:00
*** Trevor_V has quit IRC22:03
sorrisonrm_work: question when returning a LB from the API should it return as availability_zone or availability_zone_id and then up to the client to go and resolve that?22:11
sorrisonI'm leaning towards returning the name22:12
rm_workErr, probably ID the way it's designed now? Though name would be more in line with other AZ implementations22:14
rm_workBut different from flavors22:14
rm_workName IS unique for both, but22:22
rm_workWe allow deleting a zone and making a new zone with the same name, so22:23
rm_workThat could POSSIBLY lead to confusion? But I don't know if that would happen often enough for it to be a design concern22:23
rm_workJust rambling some thoughts22:23
rm_workIf we're using ID as the key in the AZ objects, then we need to return ID22:24
rm_workI think that's going to be my assertion for the moment22:25
sorrisonI could use the name as the key in az objects then?22:47
sorrisonI just thinking from a client side the ID of an AZ is pretty useless22:47
sorrisonAn AZ is basically just a name in all other projects too22:48
*** yamamoto has quit IRC22:49
rm_workI mean, if we do that, we should commit fully22:53
*** tkajinam has joined #openstack-lbaas22:54
rm_workJust remove the id column and use name as PK22:54
rm_workThoughts johnsom ?22:54
rm_workIt's a divergence from the flavor model22:55
rm_workWhich I was trying to avoid22:55
johnsomWell, everything we have with names is returned as the ID in our  API.22:55
rm_workBut otherwise it's a divergence from the Openstack AZ model22:55
johnsomTypically for uniqueness reasons22:55
rm_workSo... Which one is more important22:55
rm_workI'm actually kinda ok with removing id and using name as the pk22:56
rm_workIn this specific case because it would be consistent with other AZ usage?22:56
sorrisonmy vote would be to use id as the link internally but from the API perspective hide availability_zone_id and show availability_zone which would be the name22:56
johnsomGive me a minute to look around before I vote.22:56
rm_workAt least nova does it that way, need to look at neutron and cinder22:57
sorrisoncinder is same as nova22:57
rm_workHmm yeah that's possible too though kinda pointless maybe22:57
rm_workWell, as long as name is unique I guess it's fine22:57
rm_workJust change all the lookups to be name based22:57
johnsomYeah, nova isn't exactly the poster child for how to do things right either....22:58
sorrisonYeah I guess because name is unique then we can use name as the foreign key and use that everywhere22:59
rm_workRight, but the question is, do we stay consistent to the existing model, even if it's a little wonky?22:59
rm_workYeah then name also becomes immutable IMO22:59
sorrisonI think that would be fine22:59
sorrisonalso loadbalancer.operating_status is linked by the name23:00
sorrisonand topology23:00
rm_workYeah I don't think linking by name is inherently bad23:00
rm_workJust inconsistent with the flavor model23:00
johnsomYeah, I think name as ID is fine. It will always have to be unique, so you can't have duplicate AZs across regions, etc.23:08
johnsomBut I guess in OpenStack region is a complete duplicate control plane.23:09
sorrisonyeah another region would mean another octavia23:10
johnsomPersonally I always default to using abstract IDs, that way I don't paint myself in a corner, but it's true the other projects just pass names around23:11
*** ivve has quit IRC23:14
*** KeithMnemonic has quit IRC23:15
rm_workk, i can make changes today23:37
rm_workyou think keep ID in our DB and just change stuff to *return* names? or ... actually remove ID23:37
johnsomLess headache to just use the name as the primary key IMO. If you aren't going to expose the ID.23:59
johnsomOtherwise you will be adding/dropping the ID column all of the time23:59

Generated by 2.15.3 by Marius Gedminas - find it at!