Wednesday, 2016-06-29

*** kbyrne has joined #openstack-lbaas00:01
*** diogogmt has quit IRC00:02
*** piet has quit IRC00:03
*** diogogmt has joined #openstack-lbaas00:03
*** ajmiller has quit IRC00:03
*** ajmiller has joined #openstack-lbaas00:04
*** bana_k has joined #openstack-lbaas00:05
*** piet has joined #openstack-lbaas00:12
*** diogogmt has quit IRC00:16
*** diogogmt has joined #openstack-lbaas00:21
*** bana_k has quit IRC00:32
*** chenli has joined #openstack-lbaas00:50
*** kbyrne has quit IRC00:57
*** Purandar has quit IRC00:58
*** kevo has quit IRC00:59
*** rudrajit_ has quit IRC01:08
*** openstack has joined #openstack-lbaas01:25
*** somerville32 has joined #openstack-lbaas01:27
*** piet has quit IRC01:37
*** woodster_ has quit IRC01:39
*** daneyon has quit IRC01:50
*** fnaval has quit IRC02:02
*** fnaval has joined #openstack-lbaas02:07
*** fnaval has quit IRC02:18
*** fnaval has joined #openstack-lbaas02:18
*** Purandar has joined #openstack-lbaas02:29
*** armax_ has joined #openstack-lbaas02:42
*** armax has quit IRC02:44
*** armax_ is now known as armax02:44
*** numans has quit IRC02:49
*** yuanying has quit IRC02:50
*** eranra has quit IRC03:06
*** eranra has joined #openstack-lbaas03:06
*** Purandar has quit IRC03:17
*** Purandar has joined #openstack-lbaas03:26
*** numans has joined #openstack-lbaas03:30
*** yamamot__ has joined #openstack-lbaas03:32
*** eranra has quit IRC03:32
*** eranra has joined #openstack-lbaas03:32
*** ducttape_ has joined #openstack-lbaas03:35
*** ajmiller has quit IRC03:36
*** ajmiller has joined #openstack-lbaas03:36
*** gongysh has joined #openstack-lbaas03:39
*** yuanying has joined #openstack-lbaas03:47
*** ducttape_ has quit IRC04:07
*** chenli has quit IRC04:08
*** chenli has joined #openstack-lbaas04:08
*** rudrajit has joined #openstack-lbaas04:09
*** brad_behle has quit IRC04:13
*** rudrajit_ has joined #openstack-lbaas04:13
*** rudrajit has quit IRC04:13
*** links has joined #openstack-lbaas04:22
*** armax has quit IRC04:29
*** gongysh has quit IRC04:43
*** rcernin has joined #openstack-lbaas04:53
*** pcaruana has quit IRC04:58
*** yamamot__ has quit IRC04:59
*** bana_k has joined #openstack-lbaas05:03
*** M00nr41n has quit IRC05:06
*** eranra has quit IRC05:07
*** numans has quit IRC05:13
*** numans has joined #openstack-lbaas05:27
*** rcernin has quit IRC05:33
*** sputnik13 has joined #openstack-lbaas05:34
*** ducttape_ has joined #openstack-lbaas05:38
*** yamamot__ has joined #openstack-lbaas05:38
*** ducttape_ has quit IRC05:42
*** outofmemory has quit IRC05:47
*** numans has quit IRC05:48
*** somerville32 has quit IRC05:49
*** numans has joined #openstack-lbaas06:00
*** chenli has quit IRC06:01
*** kobis has joined #openstack-lbaas06:01
*** chenli has joined #openstack-lbaas06:01
*** rcernin has joined #openstack-lbaas06:06
*** M00nr41n has joined #openstack-lbaas06:08
*** Purandar has quit IRC06:15
*** pcaruana has joined #openstack-lbaas06:16
*** pcaruana is now known as pcaruana|afk|06:19
*** eranra has joined #openstack-lbaas06:19
*** eranra_ has joined #openstack-lbaas06:19
openstackgerritKaiLi proposed openstack/neutron-lbaas: Add status in Octavia
*** M00nr41n has quit IRC06:22
*** M00nr41n has joined #openstack-lbaas06:23
*** eranra has quit IRC06:23
*** M00nr41n has quit IRC06:33
*** M00nr41n has joined #openstack-lbaas06:34
*** rudrajit_ has quit IRC06:37
*** rudrajit has joined #openstack-lbaas06:37
*** amotoki has joined #openstack-lbaas06:39
*** rudrajit has quit IRC06:42
*** pcaruana|afk| is now known as pcaruana06:49
*** jsheeren has joined #openstack-lbaas07:01
*** piet has joined #openstack-lbaas07:02
*** tesseract- has joined #openstack-lbaas07:09
openstackgerritKaiLi proposed openstack/neutron-lbaas: Add status in Octavia
*** eezhova has joined #openstack-lbaas07:18
*** chenli has quit IRC07:18
*** nmagnezi has joined #openstack-lbaas07:20
*** chenli has joined #openstack-lbaas07:21
*** saju_m has joined #openstack-lbaas07:26
*** piet has quit IRC07:29
*** ajmiller has quit IRC07:33
*** ajmiller has joined #openstack-lbaas07:34
*** amotoki has quit IRC07:37
*** ducttape_ has joined #openstack-lbaas07:40
*** amotoki has joined #openstack-lbaas07:43
*** ducttape_ has quit IRC07:44
openstackgerritElena Ezhova proposed openstack/octavia: Cleanup deleted load balancers in housekeeper's db_cleanup
openstackgerritElena Ezhova proposed openstack/octavia: Decrease default resource expiry age in DevStack setup
*** saju_m has quit IRC07:51
*** saju_m has joined #openstack-lbaas07:52
*** noshankus has joined #openstack-lbaas07:54
*** ihrachys has joined #openstack-lbaas08:03
*** sbalukoff_ has quit IRC08:20
*** sbalukoff_ has joined #openstack-lbaas08:20
*** dmk0202 has joined #openstack-lbaas08:39
*** ducttape_ has joined #openstack-lbaas08:41
*** anilvenkata has joined #openstack-lbaas08:45
*** ducttape_ has quit IRC08:46
*** sputnik13 has quit IRC08:55
*** gongysh has joined #openstack-lbaas08:58
*** chlong has quit IRC09:04
openstackgerritElena Ezhova proposed openstack/octavia: Cleanup deleted load balancers in housekeeper's db_cleanup
openstackgerritElena Ezhova proposed openstack/octavia: Decrease default resource expiry age in DevStack setup
*** chlong has joined #openstack-lbaas09:17
*** gongysh has quit IRC09:27
jsheerenhi all, so i've been trying to configure octavia09:29
jsheereni'm at the point where my amphora image gets booted09:29
jsheerenips are assigned from the lbnet and my private net09:30
jsheerenbut the loadbalancer does not become active, it goes to error state09:30
jsheereni'm seeing this in the logs: WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying09:30
jsheereni created the amphora image with the diskimage-create script09:31
jsheereni can't ping the amphora instance from my private network. i should be able to do this, correct?09:32
*** amotoki has quit IRC09:37
*** chenli has quit IRC09:37
rm_workjsheeren: your timeouts may be too low?09:40
rm_workthat WARNING is normal, as octavia will poll for the rest agent09:40
rm_workand it will fail many times until the boot completes09:40
jsheerenbut i can see in the console of the amphora instance that the boot has completed?09:40
rm_workit can take up to 10 minutes or so, if you don't have nested-virtualization (most clouds do not include this if you are running devstack in a VM, and also virtualbox does not have it)09:40
rm_workfrom "ACTIVE" in nova, to *actually responding* can be another while09:41
rm_workthough usually only a minute or two09:41
*** ducttape_ has joined #openstack-lbaas09:41
jsheereni would assume that when i can see the login prompt, that it should respond to ping?09:42
rm_workit depends a lot on what hardware or host VM you're using to run the VM09:42
rm_workyeah, if you can *log in* it should be working09:42
jsheereni cannot ping the instance from the router namespace09:42
jsheerenas well09:42
rm_workdid you follow a guide? or, is this devstack?09:42
jsheereni did not find a  guide .. using devstac/openstack-ansible as a "guide" on configureing it09:43
rm_workunfortunately if it's a networking problem, that is my weakness :/ I am not good at debugging low-level networking failures09:43
jsheerenwhere can i find a guie?09:43
rm_workIf you prefer to look at *code* to see how to set it up, there is our devstack plugin09:43
rm_workI think someone has written a guide for basic setup too but I am not sure where it lives... I think it was a blog post by ajmiller? though it may also be out of date09:45
rm_workunfortunately the rest of the guides I can think of actually just use our devstack plugin T_09:45
*** ducttape_ has quit IRC09:46
jsheerenrm_work: thanks for the link to the plugin09:48
jsheereni followed that more or less09:48
jsheereni did not do this though:
*** amotoki has joined #openstack-lbaas09:50
rm_workThe network stuff seems very particular... It needs to be exactly right, and unfortunately I don't understand it well enough to advise you. :(09:51
rm_workIf you are still on much later (unfortunately, about 6 hours from now) there will be more people online that can help...09:51
*** ajmiller has quit IRC09:52
jsheerenok thx if i don't figure it out by then, i'll be here :)09:52
*** ajmiller has joined #openstack-lbaas09:52
rm_workgood luck!09:52
*** eezhova has quit IRC09:53
*** eezhova has joined #openstack-lbaas09:59
*** eranra_ has quit IRC10:09
*** nokes has joined #openstack-lbaas10:17
*** noshankus has quit IRC10:18
*** nokes is now known as noshankus10:18
*** eezhova has quit IRC10:23
*** amotoki has quit IRC10:23
*** eezhova has joined #openstack-lbaas10:26
*** eezhova has quit IRC10:31
*** amotoki has joined #openstack-lbaas10:33
*** ajmiller has quit IRC10:37
*** ajmiller has joined #openstack-lbaas10:37
*** ducttape_ has joined #openstack-lbaas10:42
*** amotoki has quit IRC10:44
*** ducttape_ has quit IRC10:47
*** yamamot__ has quit IRC10:49
*** amotoki has joined #openstack-lbaas11:02
*** eranra has joined #openstack-lbaas11:20
*** noshankus has quit IRC11:29
*** noshankus has joined #openstack-lbaas11:30
*** svinota has joined #openstack-lbaas11:31
*** eezhova has joined #openstack-lbaas11:33
*** eranra has quit IRC11:34
*** eezhova has quit IRC11:40
*** ducttape_ has joined #openstack-lbaas11:43
*** amotoki has quit IRC11:44
*** eranra has joined #openstack-lbaas11:46
*** ducttape_ has quit IRC11:47
*** numans has quit IRC11:48
*** yamamoto has joined #openstack-lbaas11:57
*** eranra has quit IRC11:59
*** Frito has joined #openstack-lbaas12:00
*** ducttape_ has joined #openstack-lbaas12:04
*** ducttape_ has quit IRC12:05
*** ducttape_ has joined #openstack-lbaas12:06
*** eranra has joined #openstack-lbaas12:18
*** ebagdasa has quit IRC12:19
*** ducttape_ has quit IRC12:23
*** brad_behle has joined #openstack-lbaas12:35
*** diogogmt has quit IRC12:38
*** matt-borland has joined #openstack-lbaas12:44
*** gongysh has joined #openstack-lbaas12:48
*** diogogmt has joined #openstack-lbaas12:52
*** eranra has quit IRC12:59
*** woodster_ has joined #openstack-lbaas13:05
*** M00nr41n has quit IRC13:11
*** ihrachys has quit IRC13:11
*** ihrachys has joined #openstack-lbaas13:13
*** eranra has joined #openstack-lbaas13:22
*** eranra_ has joined #openstack-lbaas13:23
*** ducttape_ has joined #openstack-lbaas13:24
*** eranra has quit IRC13:27
*** anilvenkata has quit IRC13:27
*** ducttape_ has quit IRC13:29
*** diogogmt has quit IRC13:32
*** links has quit IRC13:36
*** eezhova has joined #openstack-lbaas13:40
*** eezhova has quit IRC13:45
*** bana_k has quit IRC13:46
*** eezhova has joined #openstack-lbaas13:47
ihrachysdougwig: are there any docs on *using* octavia?14:00
ihrachysnot API, not configuration options, but things like upgrades or HA14:00
*** ajo_ has joined #openstack-lbaas14:00
*** yamamoto has quit IRC14:02
*** ajo_ has quit IRC14:03
*** ajo_ has joined #openstack-lbaas14:04
*** jsheeren has quit IRC14:06
*** ajo_ has quit IRC14:07
*** ajo_ has joined #openstack-lbaas14:07
*** fnaval has quit IRC14:08
*** ajo_ has quit IRC14:09
*** Purandar has joined #openstack-lbaas14:12
*** eezhova has quit IRC14:12
nmagnezidougwig, could you please check this one out? waiting for it's second +2 vote for a while now
*** kobis has quit IRC14:20
*** ducttape_ has joined #openstack-lbaas14:21
*** fnaval has joined #openstack-lbaas14:21
*** zw has joined #openstack-lbaas14:26
zwHi there14:26
zwAny octavia users online? :)14:26
*** evgenyf has joined #openstack-lbaas14:31
*** eezhova has joined #openstack-lbaas14:31
*** amotoki has joined #openstack-lbaas14:31
FritoI've done a couple of patches for Octavia but I'm fairly new to it.14:32
Fritozw: --^14:33
*** sputnik13 has joined #openstack-lbaas14:33
zwFrito: ever set it up?14:33
zwIn a DVR env?14:33
FritoNegative. I've only worked with it on DevStack. Sorry :-(. Some of the others should be getting online within the next hour or so though.14:34
zwOur instances are booting up and so on14:35
zwBut then : 2016-06-29 16:34:34.475 25979 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying14:35
zwIf we ssh to the instance from within the correct namespace, it is listening14:35
zwBut is looks like it tries to connect from the default eth0 interface towards the instance, not from the correct namespace14:36
Fritotagging dougwig, johnsom, sbalukoff_ in case they can help.14:36
*** amotoki has quit IRC14:38
johnsomZw The communication with the controller is over the non-namespace eth0.  Only the tenant traffic is in the namespace.14:38
johnsomzw It is normal to see some warnings about connecting to the amphora.  Typically the vm takes a while to boot.  30 seconds with nested virtualization, 8-10 minutes without nested virt (virtualbox for example)14:41
johnsomzw Also note DVR has a nasty bug where floating IPs don't work right.14:41
openstackgerritElena Ezhova proposed openstack/neutron-lbaas: Set up hooks for the functional job and add test_migrations test
johnsomBut that won't stop creating an lb, etc14:42
zwjohnsom: Well, the instace is already booted for 4 hours :)14:43
*** yamamoto has joined #openstack-lbaas14:43
*** armax has joined #openstack-lbaas14:43
zwSo there must be something wrong :) That DVR bug, is that in mitaka also?14:43
johnsomYes, last I checked it has not yet been fixed in newton either14:44
zwjohnsom do you have a link to that bug report for me please?14:45
zwThanks already for responding to my question14:45
zwjohnsom: about octavia, "controller" = the instance running haproxy?14:46
*** amotoki has joined #openstack-lbaas14:46
johnsomSure. If you can ssh into the amp, check the agent log, /var/log/upstart/amphora-agent.log14:46
zwNevermind, you mean the communication between controller->amphorae instance14:46
zwlet me see14:47
johnsomController == the controller worker process14:47
johnsomzw This is the bug:
openstackLaunchpad bug 1583694 in neutron "[RFE] DVR support for Allowed_address_pair port that are bound to multiple ACTIVE VM ports" [Wishlist,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)14:50
zwjohnsom: 2016-06-29 14:50:47.488 1377 ERROR octavia.amphorae.backends.health_daemon.health_sender [-] No controller address found. Unable to send heartbeat14:51
*** TrevorV has joined #openstack-lbaas14:51
johnsomzw Ok, that is a configuration file error, but should not impact the ability to create an amphora14:51
*** evgenyf has quit IRC14:52
johnsomihrachys We are rebooting the docs:
zwIndeed, but it does not get in the correct provision status14:53
zwIt stays on Error14:53
zwBut it is booted and so on14:53
ihrachysjohnsom: the list is very promising!14:54
*** anilvenkata has joined #openstack-lbaas14:55
zwjohnsom: I'm on the instance, can't really see something wrong, maybe ssl is the issue14:58
zwrequest url request /opt/osp/mitaka/octavia/octavia/amphorae/drivers/haproxy/
zw192.168.246.10 is the instance I'm logged into14:58
ihrachysjohnsom: is this bug the right one to include coverage for LB upgrade?
openstackLaunchpad bug 1558378 in octavia "Docs: Octavia operator maintenance guide needed" [High,New]14:59
*** piet has joined #openstack-lbaas15:00
ihrachysjohnsom: remember we talked at the last summit about octavia amphora upgrades? you were mentioning there is some mechanism in the work to upgrade with a quick-failover. is it ready to try?15:04
ihrachysback then I was also mentioning the approach with using FIPs to switch between old and new versions of a LB. I wonder if that approach could find its place in the official docs in addition to failover based approach.15:05
*** jsheeren has joined #openstack-lbaas15:08
*** sputnik13 has quit IRC15:09
*** gongysh has quit IRC15:09
*** daneyon has joined #openstack-lbaas15:11
*** tesseract- has quit IRC15:12
johnsomihrachys Yes, you can use active/standby topology and do rolling upgrades of the amphora15:13
johnsomIt would go in that guide.15:13
johnsomFIPs typically update too slowly to be of much use15:14
ihrachysjohnsom: that assumes HA. is it a reasonable limitation?15:14
johnsomWell, the process still works in non-act/stndby mode, but the user will incur up to 30 seconds of outage15:14
*** rcernin has quit IRC15:14
ihrachysjohnsom: not sure I follow an issue with FIPs. the idea is to start a clone of the old LB (using the new image), then once it's all ready to process requests, flip the FIP switch to using the new clone; after that deprovision the old LB.15:15
johnsomYeah, but updating the backend to a FIP via neutron is a slow process15:15
ihrachysno need to update backend, the FIP points to VIP15:15
johnsomIt is also essentially the same as the failover scenario I just mentioned, which incurs outage15:16
ihrachysnothing changes for amphora, it handles whatever router sends its direction15:16
*** piet has quit IRC15:16
johnsomYes, that is what I meant by backend, the VIP the FIP points to15:16
ihrachysso what's the deal with the downtime then? I may not follow the exact concern.15:16
johnsomNo need to update the amphora member list15:16
*** catintheroof has joined #openstack-lbaas15:16
*** pcaruana has quit IRC15:17
ihrachysthere is no member list involved, it's a completely new LB resource15:17
ihrachysthat by chance serves the same members15:17
openstackgerritBernard Cafarelli proposed openstack/octavia: Add nogroup in RHEL based images
johnsomIf you aren't running act/standby you are not syncing the flows.  Updating the FIP to point to a different VIP or doing a failover, both cause an outage window to the flows through the load balancer15:18
ihrachysjohnsom: at least for new flows there is no extended downtime. right?15:20
*** bana_k has joined #openstack-lbaas15:20
johnsomWell, they would be rejected for a period of time15:21
ihrachysjohnsom: not if the clone is ready to process new flows. also, with the clone approach, you have LB that has not degraded to single node on failover.15:22
johnsomI assume the FIP would fail connections while it is updating15:22
ihrachysjohnsom: not really, updating FIP will be atomic, it will immediately redirect traffic to the clone15:22
ihrachyswe handle FIPs with iptables, and afaik it's atomic using iptables-save15:23
openstackgerritElena Ezhova proposed openstack/neutron-lbaas: Set up hooks for the functional job and add test_migrations test
*** somerville32 has joined #openstack-lbaas15:26
nmagnezijohnsom, hi :) sent a mail about the meeting today. re:
johnsomnmagnezi Hmm, I'm not seeing the e-mail.  What was the subject?15:30
*** rcernin has joined #openstack-lbaas15:30
*** amotoki has quit IRC15:31
*** sputnik13 has joined #openstack-lbaas15:31
*** somerville32 has quit IRC15:31
*** pcaruana has joined #openstack-lbaas15:32
johnsomAh, just got it15:32
*** nmagnezi has quit IRC15:33
*** saju_m has quit IRC15:34
ihrachysjohnsom: how should I proceed with the FIP idea? maybe discussing its merit on mailing list, and if we agree there is some value in it in addition to failover approach, we can include that into official docs? or you think it's dead end?15:36
*** eezhova has quit IRC15:36
*** eezhova has joined #openstack-lbaas15:36
johnsomihrachys Mailing list sounds like a good idea.  I don't think it's a dead end, but I am also not convinced we have all of the details figured out.15:38
ihrachysjohnsom: ok, I will craft something hopefully today.15:39
johnsomnmagnezi - Agenda for today:
*** piet has joined #openstack-lbaas15:45
*** amotoki has joined #openstack-lbaas15:45
*** sputnik13 has quit IRC15:45
*** svinota has quit IRC15:45
*** gongysh has joined #openstack-lbaas15:46
*** somerville32 has joined #openstack-lbaas15:54
*** diogogmt has joined #openstack-lbaas15:55
openstackgerritElena Ezhova proposed openstack/neutron-lbaas: Set up hooks for the functional job and add test_migrations test
*** amotoki has quit IRC15:59
*** _ducttape_ has joined #openstack-lbaas16:00
*** ducttape_ has quit IRC16:03
*** gongysh has quit IRC16:07
*** ajmiller has quit IRC16:09
*** ajmiller has joined #openstack-lbaas16:09
ihrachysjohnsom: fyi I sent I won't be able to join the meeting today in case you would like to discuss it there in more details, but I will read logs tomorrow.16:15
*** HenryG has quit IRC16:16
*** HenryG has joined #openstack-lbaas16:16
*** _ducttape_ has quit IRC16:20
*** piet has quit IRC16:21
johnsomihrachys Thanks!16:21
*** ducttape_ has joined #openstack-lbaas16:23
*** ihrachys has quit IRC16:27
*** dmk0202 has quit IRC16:31
*** piet has joined #openstack-lbaas16:35
*** bana_k has quit IRC16:36
*** pcaruana has quit IRC16:36
*** rudrajit has joined #openstack-lbaas16:36
*** rudrajit has quit IRC16:39
*** rudrajit has joined #openstack-lbaas16:40
*** rcernin has quit IRC16:45
*** bana_k has joined #openstack-lbaas16:47
*** rudrajit has quit IRC16:47
*** greghaynes has joined #openstack-lbaas16:47
*** kobis has joined #openstack-lbaas16:52
*** noshankus has quit IRC16:54
*** krotscheck is now known as krotscheck_vaca17:01
*** krotscheck_vaca is now known as krot_vaca_jul1917:01
eezhovajohnsom, hi! I just hit this bug both on stable/mitaka and master. Is it possible to re-open it or I have to file a new bug?17:03
openstackLaunchpad bug 1558934 in octavia "Healthmonitor fails to spawn new amphora from spare pool" [Critical,Fix released] - Assigned to Michael Johnson (johnsom)17:03
johnsomeezhova Hmm, give me a second to read this.17:04
johnsomeezhova Could it be this issue:
openstackLaunchpad bug 1577963 in octavia "Failover fails with haproxy network namespaces" [Critical,In progress] - Assigned to Michael Johnson (johnsom)17:06
openstackgerritgreghaynes proposed openstack/octavia: Modernize amphora-agent element
*** saju_m has joined #openstack-lbaas17:06
*** kobis has quit IRC17:09
openstackgerritgreghaynes proposed openstack/octavia: Use git.o.o rather than review.o.o for cloning
eezhovajohnsom, not sure. what are the symptoms for this bug? the traces I see are just like in bug 1558934 description on stable/mitaka and like in Kevin's comment on master17:10
openstackbug 1558934 in octavia "Healthmonitor fails to spawn new amphora from spare pool" [Critical,Fix released] - Assigned to Michael Johnson (johnsom)17:10
greghaynesquick question - do you all have some type of gating test which builds amphorae images?17:10
greghaynesand actually tests the image creation?17:10
johnsomgreghaynes The scenario test builds an image17:10
greghaynesawesome. Does it run the image at all or just build?17:11
johnsomgreghaynes It runs it17:11
greghaynesI'm wondering how careful I have to be with these dib cleanup patches17:11
johnsomcareful == good.  The scenario tests are a bit light at the moment since the gates struggle to boot nested VMs17:12
greghaynesYea, we have simlar issues in dib ;)17:12
johnsomeezhova So, you are seeing "Unrecognized attribute(s) 'dns_name'" in the log?  The other issue, with failover and namespace driver is that after an amphora failover, the network interfaces are not coming up reliably.17:13
johnsomgreghaynes Thanks for helping us out with our DIB elements BTW!17:14
*** dmk0202 has joined #openstack-lbaas17:16
greghaynesI had no idea you all used dib17:17
greghaynestheres a lot of stuff you all fixed that i'd like to fix in DIB proper so you all don't have to carry the code17:17
johnsomSounds great.  Yeah, I think we are one of the few projects that include gates that run DIB.17:18
*** piet has quit IRC17:19
greghaynesTheres a few who are just starting to and it seems like everyone is running in to the same issues17:19
openstackgerritgreghaynes proposed openstack/octavia: Use git.o.o rather than review.o.o for cloning
openstackgerritgreghaynes proposed openstack/octavia: Condense amphora-agent-ubuntu in to amphora-agent
greghaynestrove is the other big one17:19
*** sbalukoff_ has quit IRC17:21
eezhovajohnsom, yes, I see "Unrecognized attribute(s) 'dns_name'".  As for the second, there is no amphora-haproxy namespase on a failover node17:21
*** svinota has joined #openstack-lbaas17:26
eezhovajohnsom, As for dns_name problem I think we can just check if dns-integration extension is enabled before calling port_update here
johnsomeezhova  Hmmm, ok.  Let's open a new bug to track this issue.17:31
eezhovajohnsom, Ok, I will file a new bug17:32
*** rudrajit has joined #openstack-lbaas17:32
johnsomThanks.  Yeah, checking for the extension might be the right answer17:33
*** kevo has joined #openstack-lbaas17:38
*** piet has joined #openstack-lbaas17:38
*** rcernin has joined #openstack-lbaas17:44
*** anilvenkata has quit IRC17:44
eezhovajohnsom, Here:
openstackLaunchpad bug 1597451 in octavia "Failover to an amphora from a spares pool fails" [Undecided,New]17:46
openstackgerritgreghaynes proposed openstack/octavia: Merge haproxy-octavia-ubuntu into haproxy-octavia
johnsomeezhova Thanks!17:47
*** eezhova has quit IRC17:50
*** chlong has quit IRC17:52
*** permalac has quit IRC17:57
*** _ducttape_ has joined #openstack-lbaas17:57
*** ducttape_ has quit IRC18:00
*** cody-somerville has joined #openstack-lbaas18:04
*** csomerville has joined #openstack-lbaas18:05
*** somerville32 has quit IRC18:07
*** yamamoto has quit IRC18:07
*** cody-somerville has quit IRC18:08
*** saju_m has quit IRC18:17
*** saju_m has joined #openstack-lbaas18:18
*** ajmiller has quit IRC18:22
*** ajmiller has joined #openstack-lbaas18:22
*** alhu has joined #openstack-lbaas18:29
*** dmk0202 has quit IRC18:32
*** dmk0202 has joined #openstack-lbaas18:34
*** dmk0202 has quit IRC18:37
*** csomerville has quit IRC18:39
*** amotoki has joined #openstack-lbaas18:41
*** csomerville has joined #openstack-lbaas18:42
*** _ducttape_ has quit IRC18:45
*** ducttape_ has joined #openstack-lbaas18:46
*** amotoki has quit IRC18:47
*** sbalukoff has joined #openstack-lbaas18:47
*** nmagnezi has joined #openstack-lbaas18:55
*** ajmiller has quit IRC18:56
*** ajmiller has joined #openstack-lbaas18:56
nmagnezijohnsom, hi19:01
nmagnezijohnsom, meeting is now?19:02
johnsomnmagnezi It starts in an hour19:04
nmagnezijohnsom, oh, my bad19:04
johnsomIt's at 20:00 UTC time19:04
nmagnezijohnsom, got the timezone diff wrong19:04
nmagnezijohnsom, ack19:04
*** yamamoto has joined #openstack-lbaas19:08
*** yamamoto has quit IRC19:19
*** ihrachys has joined #openstack-lbaas19:25
*** csomerville has quit IRC19:25
*** Guest51388 has joined #openstack-lbaas19:30
*** ajmiller has quit IRC19:31
*** ajmiller has joined #openstack-lbaas19:32
jsheerenhi everyone, i have a question about the octavia health manager listen port19:32
jsheereni'm following for reference19:33
jsheerenwe create port in neutron and attach it on the br-int19:33
jsheerenthis is ok if we only have one controller.  we have 3 controllers19:33
jsheerenis it ok if i create an octavia health manager listen port on each controller?  i assume this would not be an issue?19:34
jsheerenam i correct?19:34
*** csomerville has joined #openstack-lbaas19:34
rm_worki have no idea if multi-controller is actually *working* at the moment19:35
rm_worki can't think of exactly what would break, but i don't think we've specifically tested it19:35
rm_workand i remember there being discussion of it being a concern19:35
jsheerendo you remember anything specific on that concern? i'm curious :-)19:36
*** piet has quit IRC19:38
sbalukoffI'm not sure that existing amphorae would fail-over to a different IP presently. However, it's possible to write your own method for taking over a VRRP IP that the health monitor would bind to.19:40
sbalukoffI'm not sure anyone has contributed code to do this at this point, yet, though.19:40
sbalukoff(It would be handy to have, if you feel like tackling that problem.)19:41
*** catintheroof has quit IRC19:43
jsheeren:-) i would like to, but my python coding skills are a little on the low end of the curve unfortunately19:43
jsheerenhmz, we are using neutron dvr, wouldn't the necessary namespaces be recreated by neutron when a controller goes out?19:45
jsheerenand if we have a octavia health listening port active on the controller, it should work in theory?19:47
*** piet has joined #openstack-lbaas19:49
*** perelman has joined #openstack-lbaas19:53
*** eezhova_ has joined #openstack-lbaas19:57
johnsomjsheeren Multi-controller should work fine.  There is at least one vendor shipping with three controllers19:59
jsheerenjohnsom: good to hear! thx19:59
jsheerenis the fact that we're using neutron dvr an issue?20:00
johnsomDVR has a bug with floating IPs20:00
jsheerenthis one?
openstackLaunchpad bug 1583694 in neutron "[RFE] DVR support for Allowed_address_pair port that are bound to multiple ACTIVE VM ports" [Wishlist,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)20:02
*** amotoki has joined #openstack-lbaas20:02
*** Frito has quit IRC20:05
*** kobis has joined #openstack-lbaas20:06
*** amotoki has quit IRC20:06
*** Frito has joined #openstack-lbaas20:07
jsheerenjohnsom: isn't this only the case when using active/standby amphorae? is it an issue when providing single amphora?20:09
johnsomYes, it impacts single amphora topology deploys as well20:10
*** ajmiller_ has joined #openstack-lbaas20:23
*** ajmiller has quit IRC20:25
*** kobis has quit IRC20:25
jsheerenjohnsom: do i need to create an octavia health manager port on each node that has router namespaces (with dvr, controllers and compute)?20:27
*** _ducttape_ has joined #openstack-lbaas20:32
*** ducttape_ has quit IRC20:34
johnsomjsheeren Just the ones with the octavia health manager process on them20:35
*** rudrajit_ has joined #openstack-lbaas20:35
jsheerenjohnsom: thanks20:35
*** rudrajit has quit IRC20:38
*** piet has quit IRC20:44
*** _ducttape_ has quit IRC20:50
*** ducttape_ has joined #openstack-lbaas20:50
*** ajmiller_ has quit IRC20:51
*** dmk0202 has joined #openstack-lbaas20:54
*** _ducttape_ has joined #openstack-lbaas20:59
*** ducttape_ has quit IRC21:02
*** TrevorV has quit IRC21:04
*** nmagnezi has quit IRC21:09
*** _ducttape_ has quit IRC21:14
*** Frito has quit IRC21:14
*** ducttape_ has joined #openstack-lbaas21:14
*** eezhova_ has quit IRC21:15
greghaynesHey, does anyone know the patch Adam is referring to here?
greghaynesOr Adam's nick, for that matter? :)21:19
blogangreghaynes: rm_work is his nick21:19
greghaynesblogan: ah, ty21:20
blogangreghaynes: and he's talking about what we just talked about in our meeting21:20
greghaynesrm_work: Hey, gerrit fails at showing a reply to a -2 so I just wanted to poke you :)21:20
blogangreghaynes: no patch yet that i know, just a decision to be made21:20
greghaynesah, maybe Ill find a log21:20
blogansorry a decision made, but a review shortly we can assume21:20
greghaynesbut FWIW, infra really hates when people use review.o.o21:21
greghaynesfor git clone21:21
blogangreghaynes: yeah dougwig just told us that in the meeting :)21:21
johnsomgreghaynes That was what Adam mentioned in the meeting today21:21
johnsomI think it is WIP21:22
*** amotoki has joined #openstack-lbaas21:22
greghaynesI can find the meeting logs and try and catch up21:22
johnsomHappened to have that handy21:23
rm_workgreghaynes: one sec, it's going up NOW21:25
greghaynesoh hey, there was some convo about using an amphora container21:26
rm_workgreghaynes: the update was in a combo patch21:26
greghaynesneat thing about that - you can make containers with DIB21:26
rm_worksplitting it out21:26
greghaynesso you should be able to just use the same elements and tell dib to output a container21:26
rm_workand would be uploaded but my firewall is being dumb21:26
johnsomYeah, we have the flag for the docker thing.21:26
*** amotoki has quit IRC21:26
greghaynesjohnsom: ok, awesome21:26
greghaynesjohnsom: its not used a ton but if you all hit issues its something I am really interested in making sure works21:27
johnsomI haven't looked at it or tried it in a very long time, but it's there21:27
greghaynesso just let me know21:27
johnsomCool, thanks!21:27
greghaynesrm_work: What I am mostly wondering is - what are you switching to instead of git?21:27
rm_workgreghaynes: local filesystem21:27
rm_workgreghaynes: just copying the files *from the current review*21:27
rm_workwhich is actually the reason21:28
johnsomI think the proposal is to copy the already cloned repo into the amp21:28
rm_workusing git that way is a but21:28
greghaynesoh, so, thats the whole point of source-repositories21:28
rm_workit means changes to the amp agent are never actually tested in the current review21:28
greghaynesyou dont have to edit that file, its just a default - as a user you can override where it hits to get that data21:28
greghaynesso you can just export DIB_REPOREF_amphora=zuul_ref21:28
rm_workok, do you want to just update your patch to do cause that to happen?21:29
rm_workthat'd be easier than what I was doing21:29
*** saju_m has quit IRC21:29
rm_work(very slightly)21:29
greghayneswell, I think you want to keep the code the way it is and then in your CI job definitions export that var21:29
rm_workok, the alternative is what i am TRYING to submit but my VPN is blocking me21:30
*** daneyon has quit IRC21:30
blogangreghaynes: so if DIB_REPOREF_amphora isn't set then it'll use that value in that file?21:31
greghaynesblogan: yep21:31
blogangreghaynes: ah good to know!21:31
greghaynesCheck out for the varios vars you can set21:31
bloganso we can set that to just a local path and it works?21:31
greghaynesfor that youll want to set DIB_REPOLOCATION_amphora=/path21:31
rm_workok, my solution is probably not as good then21:32
rm_worki was doing actual copying21:32
rm_worki'll submit it anyway21:32
rm_workbut can -2 in favor of the other approach21:33
openstackgerritAdam Harwell proposed openstack/octavia: [WIP] Use amphora-agent code from the current review
rm_work^^ that21:33
rm_workis much more effort than what you describe, if it works automatically with that variable set :P21:33
greghaynesrm_work: ok, Ill have a look21:33
greghaynesthe source-repositories stuff isnt obvious but once you grok what its doing it can be pretty powerful21:34
greghaynesrm_work: the other concern I have is we should still change the default git location to git.o.o21:34
blogangreghaynes: ah REPOREF is just the branch/reference21:34
rm_workgreghaynes: oh, we can't change the source-repo to the local filesystem?21:34
greghaynesblogan: yep21:34
rm_workthere's really zero point to using git21:34
rm_workor rather, pulling the files again21:35
rm_workin fact in that case, i'd rather do my method21:35
greghaynesrm_work: I dont think you want to - by having git be the default the element will work on its own rather than relying on local state21:35
rm_workbecause it will work for testing21:35
rm_workwe WANT local state21:35
rm_workat least, I do21:35
greghaynesin some cases, only in testing21:35
greghaynesso for those cases you override21:35
rm_worki was going to say "this is for devstack"21:35
rm_workbut i guess it's theoretically for real images too21:35
rm_worki dunno21:36
rm_worki'm a little bit on the fence21:36
bloganthe devstack script can possibly override that no?21:36
greghaynesyes, it can21:36
greghaynesthis is how tripleo and other dib users do it21:36
greghaynessince they have the same exact issue21:36
rm_workso the devstack script can override it to point to the local-fs git?21:37
*** alhu has quit IRC21:37
greghaynesall you ahve to do is export that var before you run the image build21:37
*** alhu has joined #openstack-lbaas21:37
rm_workok, so we can do that in...21:37
blogangreghaynes: but will it recognize just a local fs path? or does it need to be an actual git repo?21:37
greghaynesblogan: it can do both21:38
blogangreghaynes: awesome21:38
greghaynesblogan: will the local fs path not be a git repo too?21:38
rm_workDIB_REPOREF_amphora is for ref, DIB_REPOLOCATION_amphora is for local21:38
bloganrm_work: no they can be used both for remote21:38
bloganref is like stable/mitaka, location is git url21:38
bloganor in our case the local path21:39
greghaynesyea, so unless there is something I am missing - the typical way to do this is only export DIB_REPOREF_amphora=zuuul_ref21:39
blogangreghaynes: it will be, but won't that mean it'll clone it again locally?21:39
greghaynesblogan: oh, no, itll run a clone from there in to the image21:39
blogangreghaynes: what is the zuul_ref usually pointing to?21:39
*** matt-borland has quit IRC21:40
greghaynesblogan: that identifies a ref in the git repo which is where zuul stores the current change being tested21:40
blogangreghaynes: ah okay21:40
greghaynesbasically, if you do it that way you dont even need to clone locally first - itll just grab the current change being tested directly from zuul21:40
blogansounds like the way to go to me21:40
*** ducttape_ has quit IRC21:41
*** ducttape_ has joined #openstack-lbaas21:42
*** saju_m has joined #openstack-lbaas21:42
greghaynesI think this can even live in the element, actually. Let me try and work up a patch real wuick21:42
blogangreghaynes: won't that be a problem if people are trying to build it independently?21:43
bloganlike say i just wanted to build the image locally21:43
greghaynesblogan: We can have it check the env var that zuul sets and only set the zuul ref if that var is set21:43
greghaynesso if zuul isnt running the build then it shouldnt do anything different21:44
johnsomIt has to be "DIB_REPOREF_amphora-agent" BTW as that is what we "named" the element in the svc-map.  I think....21:44
greghaynesjohnsom: ah, ty21:44
blogangreghaynes: sounds like that more belongs in the devstack/ script than in the element21:44
bloganjohnsom: yes thats how its named21:44
greghaynesblogan: yea, that works for me too21:44
*** ducttape_ has quit IRC21:44
rm_workgreghaynes: i am fighting serious internal fires at the moment so can't focus on this for like ... at least an hour21:44
rm_workgreghaynes: are you just "doing it"?21:44
bloganrm_work: just change your -221:44
*** ducttape_ has joined #openstack-lbaas21:44
rm_workgreghaynes: if so i'll just remove that -2 and abandon my thing21:44
greghaynesthe devstack deal I don't really understand so someone else might need to do that... I can supply the line to export if someone can find where to put it ;)21:45
blogangreghaynes: i'm looking into it now21:45
rm_workswitched to a -121:45
greghaynesbased on we need to set the reporef to ZUUL_REF, so it should just be DIB_REPOREF_amphora_agent=$ZUUL_REF21:47
*** rudrajit has joined #openstack-lbaas21:47
greghaynes(you cant have -'s in env vars so dib replaces them with _'s)21:47
johnsomYeah, that is what I was going to do.  It will git clone remote though21:47
greghaynesYep, but itll grab the current patch under review so that should be fine21:48
rm_workI personally prefer local, but since that supposedly works too with a quick override, i'm fine with it21:49
rm_workbut we should document EXPLICITLY how to make it do that21:49
rm_worklike, line numbers or something21:49
greghaynesif you want to do local then youll need to jsut do DIB_REPOLOCATION_amphora_agent=/path/to/local21:49
greghaynesand obviously make sure the local checkout is of the correct ref21:50
johnsomYeah, I would also make a note in the sources file that this is happening for the gates21:50
rm_workI really liked just copying, then i didn't have to constantly be committing every time i did a typo fix or something, but i guess it's not a huge deal21:50
openstackgerritBrandon Logan proposed openstack/octavia: Use correct code version for amphora agent image
*** rudrajit_ has quit IRC21:51
bloganthat is just a first stab at it, i may have totally missed it21:51
rm_workah we're not just sticking it all in one patch?21:51
rm_workprobably they shouldn't both be $ZUUL_REF21:51
rm_workbecause ... ...21:51
bloganoh good point lol21:52
greghaynesblogan: commented21:52
bloganim dumb21:52
johnsomYeah, that doesn't look right.21:52
bloganmy brain is fried today21:52
rm_workok if we're doing this as separate patches, we may as well merge your other one21:52
openstackgerritBrandon Logan proposed openstack/octavia: Use correct code version for amphora agent image
greghaynesrm_work: for local dev I just export the DIB_REPOLOCATION_...=/my/repo21:52
bloganrm_work: thats why i +2'ed it21:53
greghaynesrm_work: and then dev that way21:53
bloganmakes sense as separate patches21:53
*** diogogmt has quit IRC21:53
rm_workfine by me, as long as the end result is that this works :P21:53
bloganit fixes everything21:54
bloganall bugs, features requested, etc21:54
*** rcernin has quit IRC21:54
greghaynesYea, this whole system was made because when we were doing tripleo we had this exact problem and wanted an easier way to deal with it21:54
greghaynese.g. I want to dev on nova locally and build an image with it21:54
blogangreghaynes: i'm not sure if i'm glad that we're running into teh same issues yall solved a long time ago, or sad21:55
greghaynesIt just means we should have better docs I think21:55
bloganwell the ZUUL_REF i didn't see in teh docs21:55
bloganbut that kind of stuff is just hard to doc21:55
greghaynesYea, that is infra dark magic21:55
bloganinfra and dark magic are synonymous21:56
blogani'm going to set that DIB_REPOLOCATION_amphora_agent in my bashrc so that i'll never have to deal with that again, and also never remember21:57
*** piet has joined #openstack-lbaas21:57
johnsomYeah, I just added it to my notes21:58
blogangreghaynes: thanks for hitting us with some knowledge21:58
bloganmost productive -2 ever21:58
greghaynesyep, np21:59
greghayneswoo code review!21:59
*** dmk0202 has quit IRC22:08
*** ihrachys has quit IRC22:08
*** ducttape_ has quit IRC22:11
jsheerenhi all, i'm currently seeing this in my octavia-worker log after createing a loadbalancer: WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.22:15
jsheerenthe amphora is booted and has an ip in the lb-net and the private net22:15
jsheerenthis does not get through: request url
*** chlong has joined #openstack-lbaas22:17
*** ajmiller has joined #openstack-lbaas22:19
jsheerenthe amphora agent is running inside the amphora and listening on port 944322:19
jsheerenthe correct sec group is attached to the amphora instance22:20
*** perelman has quit IRC22:21
*** rtheis has quit IRC22:22
*** saju_m has quit IRC22:23
jsheerenhow do the octavia processes contact the amphora instance?  it's trying to contact the lb-net ip, to also activate the private net ip inside the amphora instance; but how can it reach the amphora instance?22:25
jsheerendoes it use the network namespaces?22:26
*** ajmiller has quit IRC22:28
*** ajmiller has joined #openstack-lbaas22:28
*** minwang2 has joined #openstack-lbaas22:34
*** saju_m has joined #openstack-lbaas22:36
*** minwang2 has quit IRC22:37
jsheerenalso a quick question about the health manager port: after attaching the neutron port to the openvswitch br-int, do i need to set an ip on the physical network for it?22:38
*** amotoki has joined #openstack-lbaas22:42
*** amotoki has quit IRC22:47
*** Guest51388 has quit IRC22:55
*** diogogmt has joined #openstack-lbaas22:56
*** mixos has joined #openstack-lbaas23:07
*** csomerville has quit IRC23:09
*** piet has quit IRC23:11
*** jsheeren has left #openstack-lbaas23:17
*** piet has joined #openstack-lbaas23:18
*** saju_m has quit IRC23:29
*** ajmiller has quit IRC23:33
*** ajmiller has joined #openstack-lbaas23:34
*** svinota has quit IRC23:37
*** daneyon has joined #openstack-lbaas23:38
*** ajmiller has quit IRC23:39
*** daneyon_ has joined #openstack-lbaas23:39
*** ajmiller has joined #openstack-lbaas23:39
*** daneyon has quit IRC23:43
*** mixos has quit IRC23:54

Generated by 2.14.0 by Marius Gedminas - find it at!