Friday, 2019-05-17

« Thursday, 2019-05-16 Index Saturday, 2019-05-18 »
rm_workcolin-: ;00:00
rm_workcolin-: ;)00:00
*** goldyfruit has joined #openstack-lbaas00:07
*** ccamposr has joined #openstack-lbaas00:19
*** ccamposr__ has quit IRC00:22
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Cleanup requirements.txt  https://review.opendev.org/64265100:28
johnsomAh, still getting used to the new requirements scheme. Ok, got that fixed.00:28
*** goldyfruit has quit IRC00:30
rm_workjohnsom: lower-constraints still broke there00:45
*** yamamoto has joined #openstack-lbaas00:47
rm_workjohnsom: commented00:52
rm_work^^ on the ampv2 patch00:52
rm_worklooks generally good00:52
rm_worki need to get back to multi-vip... tomorrow maybe00:53
*** hongbin has joined #openstack-lbaas01:11
*** ricolin has joined #openstack-lbaas01:22
*** ricolin has quit IRC01:30
johnsomThank you. Want to push something up early to get feedback. I will try to make it actually work tomorrow. Lol01:41
*** yamamoto has quit IRC01:49
*** ricolin has joined #openstack-lbaas01:56
*** ianychoi has joined #openstack-lbaas02:00
*** ricolin has quit IRC02:04
*** yamamoto has joined #openstack-lbaas02:50
*** gthiemon1e has quit IRC02:52
*** gthiemonge has joined #openstack-lbaas02:54
*** hongbin has quit IRC03:24
*** gcheresh_ has joined #openstack-lbaas03:37
*** psachin has joined #openstack-lbaas03:41
*** gcheresh_ has quit IRC03:46
*** ianychoi has quit IRC03:49
*** ianychoi has joined #openstack-lbaas03:49
*** ianychoi has quit IRC03:52
*** ianychoi has joined #openstack-lbaas03:52
openstackgerritMerged openstack/octavia-dashboard stable/rocky: Pin stable/rocky to xenial nodeset  https://review.opendev.org/64885204:29
openstackgerritMerged openstack/octavia-dashboard stable/queens: Pin stable/queens to xenial nodeset  https://review.opendev.org/65963604:30
openstackgerritMerged openstack/octavia-dashboard stable/queens: Add the missing protocol for listener  https://review.opendev.org/65958004:30
*** gcheresh_ has joined #openstack-lbaas05:08
*** pcaruana has joined #openstack-lbaas05:20
*** gcheresh_ has quit IRC05:25
*** ivve has quit IRC05:50
*** vishalmanchanda has joined #openstack-lbaas05:54
*** sapd1_x has joined #openstack-lbaas06:10
openstackgerritCarlos Goncalves proposed openstack/octavia-dashboard stable/queens: Fix auth url for Barbican client  https://review.opendev.org/64157806:26
openstackgerritCarlos Goncalves proposed openstack/octavia-dashboard stable/rocky: Fix auth url for Barbican client  https://review.opendev.org/64157706:26
openstackgerritReedip proposed openstack/octavia-tempest-plugin master: Support skipping APP_COOKIE and HTTP_COOKIE  https://review.opendev.org/64549506:49
*** ivve has joined #openstack-lbaas06:58
*** luksky has joined #openstack-lbaas06:59
*** ccamposr has quit IRC07:04
*** ccamposr has joined #openstack-lbaas07:04
*** tesseract has joined #openstack-lbaas07:08
*** rcernin has quit IRC07:19
*** rpittau|afk is now known as rpittau07:22
*** luksky has quit IRC07:45
*** luksky has joined #openstack-lbaas08:19
*** sapd1_x has quit IRC08:21
*** sapd1_x has joined #openstack-lbaas09:13
*** mkuf_ has quit IRC09:30
*** sapd1_x has quit IRC09:36
*** mkuf has joined #openstack-lbaas09:37
*** gthiemonge has quit IRC09:47
*** gthiemon1e has joined #openstack-lbaas09:47
*** sapd1_x has joined #openstack-lbaas09:49
*** rcernin has joined #openstack-lbaas09:50
*** sapd1_x has quit IRC09:59
*** yamamoto has quit IRC10:08
*** yamamoto has joined #openstack-lbaas10:35
*** vishalmanchanda has quit IRC10:44
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Jobboard based controller  https://review.opendev.org/64740610:51
*** henriqueof has joined #openstack-lbaas11:06
*** mkuf has quit IRC11:27
*** mkuf_ has joined #openstack-lbaas11:27
*** mkuf has joined #openstack-lbaas11:28
*** mkuf_ has quit IRC11:32
*** rcernin has quit IRC11:36
*** boden has joined #openstack-lbaas11:38
*** yamamoto has quit IRC11:59
*** yamamoto has joined #openstack-lbaas12:02
*** goldyfruit has joined #openstack-lbaas13:31
*** yamamoto has quit IRC14:18
*** yamamoto has joined #openstack-lbaas14:18
*** luksky has quit IRC14:19
*** yboaron_ has joined #openstack-lbaas14:23
*** goldyfruit has quit IRC14:31
*** gthiemon1e has quit IRC14:41
*** gthiemonge has joined #openstack-lbaas14:42
*** yamamoto has quit IRC14:47
*** yamamoto has joined #openstack-lbaas14:48
*** yamamoto has quit IRC14:53
openstackgerritCarlos Goncalves proposed openstack/octavia master: Delete unused files in tests/contrib  https://review.opendev.org/65981514:55
*** goldyfruit has joined #openstack-lbaas15:08
*** Vorrtex has joined #openstack-lbaas15:16
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Cleanup requirements.txt  https://review.opendev.org/64265115:18
*** ivve has quit IRC15:21
*** sapd1_x has joined #openstack-lbaas15:21
openstackgerritMerged openstack/octavia-dashboard stable/queens: Fix auth url for Barbican client  https://review.opendev.org/64157815:41
openstackgerritMerged openstack/octavia-dashboard stable/rocky: Fix auth url for Barbican client  https://review.opendev.org/64157715:41
johnsomOk, now the cleanup requirements patch is good.15:51
*** ut2k3 has joined #openstack-lbaas15:51
ut2k3Hi guys, how can I recreate the amphora instances on a lb in case of the amphora-table is empty?15:52
ut2k3openstack loadbalancer failover ... isn't helping.15:52
johnsomI think rm_work knows the procedure.  It's something like creating fake amphora instance records.15:57
ut2k3could you maybe help me rm_work?15:59
johnsomOut of curiosity, what version of Octavia are you running and what happened to your cloud that got you here?16:00
*** rpittau is now known as rpittau|afk16:01
ut2k3version 3.0.2, we rebooted the compute nodes ... and it seems that the cleanup killed my records somehow. So I've ended up with empty amphora table.16:04
ut2k3My mistake is here: not having a backup around...16:05
johnsomDid you reboot them all at once or in quick series?16:06
johnsomut2k3 I thought this was fixed, but looking at the code it is not, so I opened a bug for you: https://storyboard.openstack.org/#!/story/200573116:09
ut2k3The reboot happened all at once, due to electricity outage in the Rack :/16:09
johnsomSince rm_work doesn't seem to be around, let me see if I can help you. I would recommend making a backup of the DB before we start messing with it though, just in case something goes wrong.16:10
johnsomWhat provisioning status are the LBs in?  ERROR?16:11
ut2k3Yep I did that. So far from your information of creating a fake entry, I found this post: https://openstack.nimeyo.com/122174/openstack-dev-octavia-how-to-recreate-amphora-instances16:11
ut2k3So they have been at ERROR state, I've put them to ACTIVE, and did then a failover again .. it seems I have to recreate the PORTS16:12
ut2k3Since it bootet up then two new amphora instances. One is in READY State with no loadbalancer_id and one is now in ERROR state16:13
ut2k3From logs: > ERROR octavia.controller.worker.controller_worker [-] LB 9e950a88-af30-4eeb-8fb6-d8c20388db17 failover exception: port not found (port id: None).: PortNotFound: port not found (port id: None).16:13
ut2k3Just an Idea: What about setting the LB to PENDING_CREATE would that work?16:14
johnsomNo, that will not help you.16:15
ut2k3Ok16:15
johnsomHow many LBs do you have like this?16:16
johnsomAlso, in the DB, let's pick one that is in ERROR status. Look up the LB ID, then look in the "vip" table and make sure it has a full record for the LB.16:16
*** tesseract has quit IRC16:17
*** psachin has quit IRC16:17
ut2k3I have currently 3 LB that I have to fix. So if you help me with one I can for sure fix the others.16:17
johnsomAlso, let's shut down the housekeeping process.16:18
johnsomJust to make sure a purge process doesn't run16:18
*** tesseract has joined #openstack-lbaas16:18
johnsomWe don't want it to delete any DB records at the moment16:18
ut2k3Ok16:19
openstackgerritMerged openstack/python-octaviaclient master: Cleanup requirements.txt  https://review.opendev.org/64265116:20
ut2k3Done and: https://pastebin.com/vN67NHzA here that's the vip-table.16:20
ut2k3So one entry per LB as I have only 3 LBs running.16:21
johnsomOk nice!  So let's pick one LB record by ID.  Give me a second and I will write the SQL for you.16:22
johnsomAre these active/standby or standalone LBs?16:22
ut2k3Topology: SINGLE16:25
ut2k3If you mean that.16:25
johnsomYes, ok that will make this easier16:26
johnsomOne minute16:26
ut2k3Thanks :)16:26
johnsominsert into amphora (id, compute_id, status, load_balancer_id, ha_ip, ha_port_id, role, cert_busy) VALUES (uuid(), uuid(), 'ALLOCATED', '<LB ID GOES HERE>', '<IP address from VIP table for LB goes here>', '<Port ID from VIP table for LB goes here>', 'STANDALONE', 0);16:28
johnsomAlso make sure the LB is in provisioning_status='ERROR'16:28
ut2k3OK16:30
ut2k3sec16:30
*** luksky has joined #openstack-lbaas16:33
*** goldyfruit has quit IRC16:33
*** goldyfruit has joined #openstack-lbaas16:33
ut2k3Done that: https://pastebin.com/t1gLR2yu16:37
ut2k3I would expect that now 3 new amphora instances would be created or not?16:39
johnsomOk, now make the load balancer failover call while watching the worker logs.16:39
johnsomIt should remove your record and build a new functional amphora16:40
ut2k3Ok16:40
johnsomI haven't tested this yet, my devstack is still building.16:40
ut2k3When I now trigger the failover: Invalid state ERROR of loadbalancer resource 9e950a88-af30-4eeb-8fb6-d8c20388db17 (HTTP 409) (Request-ID: req-ccb41e86-482b-4acb-b136-13142cae310f)16:40
ut2k3All 3 LB are currently in the provisioning_status = ERROR state.16:41
ut2k3The command was working on ACTIVE, so I can try that by changing them in the DB to ACTIVE .16:41
johnsomArgh, I thought that was fixed too.  Ok, set it to ACTIVE then do the failover16:41
johnsomLet's do one at a time as well. Not all three at once.16:42
ut2k3yep16:43
ut2k3Its being created now, lets see :)16:45
* johnsom Hopes for success!16:45
ut2k3Its changing into  transitioned into state 'REVERTED' from state 'REVERTING'16:47
ut2k3> 2019-05-17 16:47:10.911 99 ERROR octavia.controller.worker.controller_worker [-] LB 9e950a88-af30-4eeb-8fb6-d8c20388db17 failover exception: port not found (port id: None).: PortNotFound: port not found (port id: None).16:47
johnsomThat is not good. That means we have something else wrong that needs fixed.16:47
*** sapd1_x has quit IRC16:48
johnsomCan you do a "openstack port show 10389c9a-3250-41a3-a398-b3027c0ad1c8" ?16:48
ut2k3Thats the trace (I removed the key stuff)> https://pastebin.com/3CqtKRNx16:49
ut2k3Thats the output from the command: https://pastebin.com/yLsTmYRd16:50
johnsomHmm, ok, give me a minute16:51
johnsomOk, we are going to need to fill in another field16:53
ut2k3Ok16:54
johnsomJust a minute or two more, sorry.16:57
*** tesseract has quit IRC16:58
*** tesseract has joined #openstack-lbaas16:59
johnsomOk, can you do a "openstack port list | grep vrrp"17:00
johnsomIf we are lucky there will be three ports listed17:01
ut2k3Yep 3 there :)17:01
johnsomOk, one of those should have a uuid that matches our load balancer ID we are attempting to fix.17:02
ut2k3Nope, unfortunately not17:03
ut2k3https://pastebin.com/QXjKtZD217:03
johnsomOh! right, sorry, that was my mistake. It's going to match the old amphora ID.  That is ok. Are all three on the same subnet?17:04
ut2k3yep17:04
johnsomOk, cool, we will just pick one by one. The order doesn't matter for that port given they are all on the same subnet.17:04
johnsomupdate amphora set vrrp_ip='<vrrp port IP goes here>', vrrp_port_id='<vrrp port ID goes here>' where id = '<our fake amphora record ID>'17:05
johnsomSo, for the fake amphora record we created before, let's update it to fill in the VRRP port fields with one of the ports we found in neutron.17:05
johnsomThen set the LB back to active, then failover17:06
johnsomOh, the ending ';' didn't paste. The SQL line needs the ; on the end17:06
ut2k3Question: I can pick ANY  vrrp_port id to this amphora or should it be a certain one?17:07
johnsomJust pick the first one on your list.  You can't re-use them, but it doesn't matter which one you use for this.17:08
johnsomIt will get deleted and rebuilt during the failover.17:08
ut2k3Instance is spawning, fingers crossed :)17:10
ut2k3Awesome, it was working :)17:15
johnsom+117:15
johnsomThere you go. Glad we could help.17:15
*** tesseract has quit IRC17:17
*** yboaron_ has quit IRC17:21
*** yboaron_ has joined #openstack-lbaas17:22
johnsomI took some notes too so hopefully we can get that all cleaned up so it's not so hard to recover.17:24
*** ivve has joined #openstack-lbaas17:30
*** ut2k3 has quit IRC17:35
*** yamamoto has joined #openstack-lbaas17:41
*** yamamoto has quit IRC17:47
colin-morning, i'm trying to get a sense for the volume of gratuitous ARP requests we should see for a given number of LBs17:58
colin-is there anywhere in the project that describes when this occurs, how frequently, etc? as a specific topic?17:58
johnsomcolin- Hi. It's pretty aggressive as we found we need to beat neutron over the head with them in some clouds.17:59
colin-yeah, i'm concerned about the ability of <generic switch> that sits between those services since that will vary greatly from one cloud to the next18:00
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/keepalived/jinja/templates/keepalived_base.template#L3018:01
colin-so i'm hoping to quantify the footprint18:01
colin-ah thanks18:01
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/keepalived/jinja/jinja_cfg.py#L9918:01
johnsomAny switch worth it's weight should be able to handle that without trouble.18:02
colin-which switches are worth their weight :)?18:04
colin-a never ending search for the best one18:04
colin-this will help, thanks a lot18:04
johnsomI have opinions, but will constrain myself18:04
*** yboaron_ has quit IRC18:30
*** yboaron_ has joined #openstack-lbaas18:31
*** yboaron_ has quit IRC18:32
*** yboaron_ has joined #openstack-lbaas18:33
*** yboaron_ has quit IRC18:34
*** yboaron_ has joined #openstack-lbaas18:34
*** yboaron_ has quit IRC18:38
*** yboaron_ has joined #openstack-lbaas18:39
*** yboaron_ has quit IRC18:47
*** yboaron_ has joined #openstack-lbaas18:48
*** ut2k3 has joined #openstack-lbaas18:52
*** yboaron_ has quit IRC18:52
*** Vorrtex has quit IRC18:52
*** yboaron_ has joined #openstack-lbaas18:52
*** ut2k3 has quit IRC18:55
*** gcheresh_ has joined #openstack-lbaas19:21
*** yboaron_ has quit IRC19:26
*** yboaron_ has joined #openstack-lbaas19:27
*** gthiemonge has quit IRC19:38
*** gthiemonge has joined #openstack-lbaas19:39
*** goldyfruit has quit IRC19:39
*** goldyfruit has joined #openstack-lbaas19:53
*** gcheresh_ has quit IRC20:00
*** pcaruana has quit IRC20:02
openstackgerritMichael Johnson proposed openstack/octavia master: Create Amphora V2 provider driver  https://review.opendev.org/65968920:08
openstackgerritMichael Johnson proposed openstack/octavia master: Create Amphora V2 provider driver  https://review.opendev.org/65968920:08
*** ut2k3 has joined #openstack-lbaas20:37
*** ut2k3 has quit IRC20:37
*** KeithMnemonic has quit IRC21:08
openstackgerritMerged openstack/octavia master: db: add missing primary key in spares_pool table  https://review.opendev.org/65645921:34
*** boden has quit IRC21:35
*** henriqueof has quit IRC21:55
*** gthiemonge has quit IRC22:00
*** gthiemonge has joined #openstack-lbaas22:01
*** gthiemonge has quit IRC22:21
*** gthiemonge has joined #openstack-lbaas22:22
*** luksky has quit IRC22:50
*** AlexStaf has joined #openstack-lbaas23:07
xgermanjohnsom: HPE bought Cray! WTF?23:42
johnsomYeah, saw that. Funny given they bought SGI a few years ago.23:43
xgermanit’s almost like they are now the place great comouter manufacturers got o die23:43
johnsomI'm still not sure what they really got out of SGI23:44
xgermanYep, real estate?23:46
*** goldyfruit has quit IRC23:48
johnsomWell, there are a few more on the list than I expected: https://www.top500.org/statistics/sublist/23:50
colin-had the same reaction as you two about that news23:50
colin-what on earth are they going to do with that acq23:50
colin-don't understand the market appetite for these buys specifically right _now_ especially with Uber's IPO drama23:51
johnsomDesperate attempt to compete with IBM?23:51
colin-maybe so, i certainly can't attribute it to anything else specifically23:52
colin-curious what fruit the f5/nginx thing will bear23:52
*** AlexStaf has quit IRC23:55
johnsomI made a mistake picking listener for my demo patch. It touches nearly everything.... sigh23:57
« Thursday, 2019-05-16 Index Saturday, 2019-05-18 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!