Tuesday, 2018-08-21

*** hongbin has joined #openstack-lbaas00:26
*** sapd1 has joined #openstack-lbaas00:46
*** abaindur has quit IRC01:05
*** bzhao__ has joined #openstack-lbaas01:10
rm_workjohnsom: anything else?01:11
johnsomrm_work: there are two rocky backports, after that we are good01:21
rm_work#1 and 2?01:21
rm_workah those merged?01:21
rm_workoh, *are merging*01:22
rm_workso don't need me :)01:22
bzhao__Sorry, team. Just back to office after a business trip with miguel.01:27
rm_workplz to be ignoring this01:48
openstackgerritAdam Harwell proposed openstack/octavia master: DNM: two dumb downstream things to fix, IGNORE ME  https://review.openstack.org/59398601:48
rm_work100% ignore01:48
openstackgerritAdam Harwell proposed openstack/octavia master: Experimental multi-az support  https://review.openstack.org/55896201:51
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: AZ Evacuation resource  https://review.openstack.org/55987301:51
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561201:55
sapd1johnsom: I know why my octavia-api is too slow. Because I added a new columns to load_balancer table so when listing from database it takes more times. But I can't know how to solve the problem?02:10
rm_workyou added DB columns? O_o03:15
sapd1rm_work: I knew why?03:17
sapd1because LoadBalancerRepository extend from BaseRepository, and use get_all function. in get_all function it joins all table.03:18
rm_workyeah but like03:26
rm_workwhy did you have to add custom DB columns?03:26
sapd1I would like to use multiple nova flavor for amphora instance.03:30
sapd1In my case, We would like allow use to select flavor of load balancer.03:30
sapd1rm_work: https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L133 why we need join all in here?03:31
rm_workah yeah, so that is the flavor framework03:34
rm_worklike, that's exactly what it will do03:34
rm_workare you working on the patches for that?03:34
rm_workthat is there because it actually REDUCES the trips to the DB a ton03:34
rm_workwhich is important because actually we found that the round-trip overhead was WAY worse than the delay from doing the joins03:35
rm_workeven with very large data sets03:35
sapd1rm_work: I used cProfile to tracing source code. And Total time to get from database  is too large.03:39
sapd1rm_work: http://paste.openstack.org/show/728470/03:40
*** ramishra has joined #openstack-lbaas03:40
rm_workyeah something is wrong with your DB I think :/03:40
rm_worki mean, how many LBs do you have?03:41
rm_workit should be subsecond even when joining every other table, with thousands of LBs...03:42
rm_workmaybe there's a case in the data we missed?03:42
sapd1rm_work: I have 4 LB and on one LB a have 4 L7 Policy. When create more L7 Policy load time is increased.03:44
rm_workyeah that's insane03:44
rm_work80s makes NO sense03:44
sapd1rm_work: normally It takes 20 seconds, But when I add cProfile, It takes more.03:45
rm_workright, more than 0.5s or so is nuts for that03:45
sapd1rm_work: Have you checked my story yet? https://storyboard.openstack.org/#!/story/200293303:45
rm_workso is it ONLY with a ton of L7 policies?03:46
rm_worki wonder if there's some sort of weird interaction with the way those are linked in?03:46
sapd1rm_work: I commented query.options(joinedload('*')) and re-run unit test in python-octaviaclient and no errors.03:46
*** abaindur has joined #openstack-lbaas04:04
*** hongbin has quit IRC04:10
*** yboaron_ has joined #openstack-lbaas04:20
johnsomdayou Happen to be around?  https://review.openstack.org/#/c/591295 A translation for nlbaas.  Then I will cut our RC204:52
johnsomThank you.  RC2 patch posted.  Catch you all tomorrow.05:18
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Drop nose dependencies  https://review.openstack.org/59314705:20
*** abaindur has quit IRC05:38
*** abaindur has joined #openstack-lbaas05:38
*** yboaron_ has quit IRC06:16
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Add listener and pool protocol validation.  https://review.openstack.org/59404006:27
*** rcernin has quit IRC06:38
*** rcernin has joined #openstack-lbaas06:40
*** pcaruana has joined #openstack-lbaas06:42
*** luksky has joined #openstack-lbaas06:50
*** rcernin has quit IRC06:51
*** velizarx has joined #openstack-lbaas07:16
*** abaindur has quit IRC07:22
*** velizarx has quit IRC07:49
cgoncalvesjohnsom, thanks for cutting queens. we could have made to get https://review.openstack.org/#/c/592569/ in too but okay07:53
cgoncalvesnmagnezi, https://review.openstack.org/#/c/592569/ pretty please :)07:53
nmagnezicgoncalves, +208:08
openstackgerritYang JianFeng proposed openstack/octavia master: [WIP] Add listener and pool protocol validation.  https://review.openstack.org/59404008:13
*** velizarx has joined #openstack-lbaas08:23
openstackgerritNir Magnezi proposed openstack/octavia master: DNM: Leave VIP NIC plugging for keepalived  https://review.openstack.org/58929208:38
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: WIP: Add octavia-v2-dsvm-scenario-ipv6  https://review.openstack.org/59407808:56
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: WIP: Add octavia-v2-dsvm-scenario-ipv6  https://review.openstack.org/59407809:00
openstackgerritCarlos Goncalves proposed openstack/octavia master: Temporarily remove octavia-v2-dsvm-scenario-ubuntu.bionic  https://review.openstack.org/58888309:07
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Gate on CentOS 7 and check on Ubuntu Bionic  https://review.openstack.org/58741409:09
openstackgerritCarlos Goncalves proposed openstack/octavia master: Gate on octavia-dsvm-base based jobs and housekeeping  https://review.openstack.org/58744209:15
openstackgerritKobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: do not change SG owned by user  https://review.openstack.org/59247109:28
*** luksky has quit IRC09:28
openstackgerritKobi Samoray proposed openstack/neutron-lbaas master: nlbaas2octavia: improve member error log message  https://review.openstack.org/59361009:28
*** luksky has joined #openstack-lbaas10:02
*** dolly has joined #openstack-lbaas10:22
dollyjohnsom: yesterday you were saying something about making a "stable" release for queens and that it would be ready today, did you manage to do so ?10:23
dollyis it the "stable/queens" branch in git repo ?10:23
dollycgoncalves: you there ?10:26
*** hvhaugwitz has quit IRC10:45
*** hvhaugwitz has joined #openstack-lbaas10:45
*** velizarx has quit IRC10:46
*** velizarx has joined #openstack-lbaas10:46
jitekaHello, I'm still doing some testing on our deployement of octavia in queens (with ACTIVE/STANDBY topology)11:10
jitekaI have a scenario where, one of my amphora went in error while trying to failover.11:10
jitekaI tried to delete the amphora VM (Backup) in error to see if health-manager would catch it and generate a new VM11:10
jitekaBut now I end up with only 1 amphora MASTER and the loadbalancer show as ONLINE (operating_status)11:10
jitekaWhat could be done to come back to an ACTIVE/STANDBY haproxy on that VIP ?11:10
jitekaMore generally how to deal with amphora in ERROR ?11:10
jitekajohnsom: looking at https://github.com/openstack/octavia/tree/stable/queens11:12
jitekaI see that latest commit was 3 days ago 69beadc7a8a14c2fedee79227b38bc37153b5dce (Merge "Fix neutron "tenat_id" compatibility" into stable/queens)11:12
jitekaIs it the correct way to double check if a new version can be used ?11:12
*** crazik has joined #openstack-lbaas11:35
crazikI had some issues with DB and octavia. Tried to fix this, and now I have only MASTER amphora for LB.11:36
crazikhow to force octavia to add BACKUP amphorae?11:36
craziktried loadbalancer failover, but with no effect.11:36
crazikany ideas?11:38
rm_workjiteka: that is a thing i've run into a lot -- i believe we did have a fix recently that improved handling of that a bit11:47
rm_workif you try a manual failover of that amp via the amphora-api, it might perform better maybe11:48
cgoncalvesyes. most of failover issues should be fixed in rocky/master and backported to queens11:48
crazikhm. how can I do that?11:49
cgoncalvesdolly, yes11:49
cgoncalvesdolly, still open https://review.openstack.org/#/c/593954/11:49
rm_workcrazik: sorry was still talking to jiteka -- seems you ran into basically the same issue at the same time11:50
rm_workif it is GONE, i think the only way is manual DB hackery11:50
crazikI did manual DB cleanup11:50
nmagnezijohnsom, do you want to have this in RC2? https://review.openstack.org/#/c/589408/11:50
rm_workwhich I have done but requires you to be very careful and know what you're doing exactly :P11:51
crazikand at the end, I have only one amphora in DB. LB is working, but in ERROR state11:51
rm_workyou can ... kinda fix it manually11:51
rm_workif it's absolutely necesary11:51
crazikneed a way to recreate backup amphora11:51
rm_workcopy the entry in the amp table for the master, and change the role to BACKUP, and all of the ID fields to made-up uuids11:51
rm_workerr, when I say "all", I mean11:52
crazikI have db backup, I can try to re-add previous one11:52
rm_workcompute_id, vrrp_port_id11:53
rm_workah yeah that works too11:53
rm_workre-add the previous line11:53
rm_workand THEN trigger a failover via the amp-api, of that amp11:53
rm_worki am used to not knowing the old info11:53
rm_workcongrats on having a backup :P11:53
crazikok, I will try.11:53
dollycgoncalves: ok cool. trying to build my own container as we speak :p11:54
*** numans_ has joined #openstack-lbaas12:07
*** velizarx has quit IRC12:08
*** numans has quit IRC12:10
*** velizarx has joined #openstack-lbaas12:18
jitekarm_work: just saw that https://ghosthub.corp.blizzard.net/openstack/octavia/commit/72715ba6197105db352779e0236ac108a710f72d12:24
jitekaand we are running  a version without that change12:24
jitekaI will try to get updated our deployement and see how it behave12:24
rm_worklol internal git12:28
rm_workbut yeah that one is important12:29
jitekarm_work: I can't agree or deny that we have an internal git12:32
rm_workyou definitely can't deny :P12:33
rm_workit's basically required tho, github breaks builds like crazy with timeouts12:33
jitekarm_work: ahaha yes that's good practice in CI/CD12:34
jitekarm_work: never said we are following it (or not)12:34
jitekarm_work: secrets everywhere12:34
*** Krast has joined #openstack-lbaas12:44
*** velizarx has quit IRC12:45
dollycgoncalves: you there ?13:50
cgoncalvesdolly, yep13:51
dollycgoncalves: so, I went out on a lim here and made some custom octavia containers. All I did was take the branch stable/queens from github and overwrote the octavia-sitepackage in the container. Got everything up and running, can talk to the api and so forth. But when deploying an LB I get this, http://paste.openstack.org/show/728523/13:54
dollynow I understand that this is not supported in any way13:54
dollyI just figure I might as well do some testing and see what I come up with13:54
dollyDo you think it has to do with the amphora-image ? Because I haven't rebuilt that one that is used for building the amphoras13:55
cgoncalvesdolly, might be, yes. I reckon I fixed some things in the amphora agent side for centos13:56
dollycgoncalves: Is this image available somewhere ? Or can I build it easily myself13:56
cgoncalvesdolly, for testing purposes, there are a few options13:58
cgoncalvesdolly, I would say download http://tarballs.openstack.org/octavia/test-images/test-only-amphora-x64-haproxy-centos-7.qcow213:58
cgoncalvesit is based on master but should be okay13:59
cgoncalvesother option is to build your own. use diskimage-create: https://github.com/openstack/octavia/tree/master/diskimage-create13:59
dollyAh, well if you have one build already I gladly try that one =)14:00
dollyJust upload it with a specific name and put that name in the octavia-conf right ?14:00
cgoncalvesupload it under the 'service' project and tag it14:00
dollygot it14:01
cgoncalvesoctavia will use latest uploaded/created image filtering by tag name14:01
dollyperfect =)14:02
cgoncalves"amphora-image" is the tag used by default in TripleO-based envs14:02
dollygot it14:02
dolly518 234 586 2,10KB/s  eta 23s     <- whats going on here :D14:03
dollydownloads like 70% of the image and then croaks14:03
cgoncalvesdownloading. full speed so far14:03
dollyah now it started again :)14:03
cgoncalves100% downloaded at full speed14:04
dollyyep, the reason why I got confused was because I started on my desktop, it croaked at ~50-60%.. Then from a server, same thing happend.. Not sure why, but started to work anyway so..14:05
dollyNow lets upload it to our cloud =)14:05
cgoncalvesjohnsom, thanks! :)14:07
johnsomcgoncalves The priority bug list is your friend...14:08
dolly$ > openstack loadbalancer list14:17
dolly$ > | 13e711ee-daad-4dee-b1e4-5f9ae23a45a6 | pm-lb | 91cf3955df114256870c20b7737b3a41 |  | ACTIVE              | octavia  |14:17
dollyWould you look at that =)14:17
dollyRunning with both the multi_az patch and branch stable/queens from the git repo14:18
dollyOn a standard OSP 13 installation14:18
dollyACTIVE_STANDBY mode btw.14:18
dollyLet me see what happens when I destroy the activve amphora :p14:18
cgoncalvesand amp from master (built today)14:18
dollyyes, perfect =)14:18
openstackgerritJacky Hu proposed openstack/neutron-lbaas-dashboard master: Remove obsolete gate hooks  https://review.openstack.org/59424314:23
dollyOh, btw, the octavia_health_manager container is unhealthy, same with octavia_housekeeping...14:23
dollyI reckon that is not suppose to be like that ?14:23
dollyAre all containers suppose to be healthy ?14:24
dolly(just so I know)14:24
cgoncalveswhat is because the container doesn't include a health check14:24
cgoncalvesshould be fixed in OSP13 z2 https://bugzilla.redhat.com/show_bug.cgi?id=151750014:24
openstackbugzilla.redhat.com bug 1517500 in openstack-tripleo-common "OPS Tools | Availability Monitoring | Octavia dockers monitoring support" [Medium,Assigned] - Assigned to mmagr14:24
dollyHm ok, well I deleted the master amp. But no new amp got created. Switchover worked though, so backends still available through LB VIP.14:26
dolly2018-08-21 14:27:07.343 82 WARNING stevedore.named [-] Could not load health_db: NoMatches: No 'octavia.amphora.health_update_drivers' driver found, looking for 'health_db''14:27
dolly2018-08-21 14:27:07.343 83 WARNING stevedore.named [-] Could not load stats_db: NoMatches: No 'octavia.amphora.stats_update_drivers' driver found, looking for 'stats_db'14:27
dollyNot sure if that is good ?14:27
dollyHm, where are the octavia.amphora.health_update_drivers and octavia.amphora.health_update_drivers suppose to be ? Maybe I missed something when building my containers ?14:31
cgoncalvesthere are defaults set for both: http://git.openstack.org/cgit/openstack/octavia/tree/octavia/common/config.py#n21314:32
cgoncalvesso I'm not sure why it is not loading14:32
dollyBut what is health_db ?14:34
johnsomIt a health manager driver that stores the results in the mysql database (there is also a logging only driver)14:35
dollyBut if I do a "grep health_db" in site-packages/octavia-directory I only find the reference from the config-file... Nothing more.. Shouldn't I see more references to it ?14:36
johnsomdolly: https://github.com/openstack/octavia/tree/stable/queens/octavia/controller/healthmanager/health_drivers14:37
johnsomAssuming you are running queens14:37
johnsomSame path for Rocky/master14:37
dolly2018-08-21 14:39:38.057 23 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('', 30401) dorecv /usr/lib/python2.7/site-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:187 2018-08-21 14:39:38.101 30 WARNING stevedore.named [-] Could not load health_db 2018-08-21 14:39:38.103 31 WARNING stevedore.named [-] Could not load stats_db14:39
dollyOk, so there is something fishy going on with the healthmanager for sure then...14:40
johnsomYeah, it looks like you have version mis-match somehow14:41
dollywell, since I "manually hack this" - that is definitely possible.14:42
dollyversion mismatch between what you mean ?14:42
johnsomI would not expect heartbeat_udp.py to exist on queens or rocky14:42
johnsomI would update your version of Octavia inside that container. Be sure to pip uninstall then pip install.14:43
dollyHm, ok. Well what I did to "update my octavia", was to clone the git-repo, check out the stable/queens branch, and then create a docker file where I copied all the content from the folder octavia into the containers /usr/lib/python2.7/site-packages/14:45
dollyBut maybe that will only partially work then.14:46
cgoncalvesdolly, there's an easier and better way. 1 sec14:46
dollycgoncalves, I'm sure there is :D14:46
cgoncalvesI ran it once months ago. I don't remember exactly which steps I took but definitely based on that page14:48
jitekaJust deployed the latest version of stable/queens in our environment and hit a bug :14:53
jitekaException during message handling: InvalidRequestError: Entity '<class 'octavia.db.models.Amphora'>' has no property 'show_deleted'14:53
jitekalog : http://paste.openstack.org/show/728529/14:53
jitekais it something that was fixed in rocky but not backported yet ?14:53
johnsomNo, again something is wrong with your versioning.  Did you run the DB migration?14:54
jitekajohnsom: hello, no I didn't14:55
jitekajohnsom: I though it was only needed when changing major version14:55
johnsomI would recommend also reading the upgrade guide: https://docs.openstack.org/octavia/latest/admin/guides/upgrade.html14:55
jitekajohnsom: was in stable/queens and I'm still in stable/queens14:55
johnsomWell, I'm not sure what state your environment is in. I'm not sure that is really a schema issue or not, but it's a thought.14:56
jitekajohnsom: but it makes sense if changes were made on the DB scheme within the same release14:56
johnsomWe don't really have a column for show deleted, so it might be completely wrong.  I'm just not sure how you could get that error with an install.....14:57
jitekano I didn't got that with an install I got that on a lb failover after refreshing our docker image and restart service with latest version of stable/queens14:58
jitekajohnsom: including these changes http://paste.openstack.org/show/728531/14:59
johnsomjiteka Is your API process the same version of stable/queens as your octavia-worker process?15:02
jitekajohnsom: yes15:06
johnsomjiteka Ok, confirmed we have a problem on queens somehow.  Looking into that now. Likely a missing backport15:07
johnsomnmagnezi Are you around?15:09
*** pcaruana has quit IRC15:09
*** dlundquist has quit IRC15:39
*** strigazi has quit IRC15:46
*** strigazi has joined #openstack-lbaas15:46
*** luksky has quit IRC15:48
openstackgerritMerged openstack/octavia master: Remove user_group option  https://review.openstack.org/58940815:51
*** ramishra has quit IRC15:54
johnsomCores, please review https://review.openstack.org/594332 to fix a bad backport impacting failover15:55
*** velizarx has joined #openstack-lbaas15:58
*** sapd1_ has joined #openstack-lbaas16:01
johnsomjiteka https://review.openstack.org/594332 will fix your queens issue16:01
jitekathanks johnsom, lgtm16:12
jitekawill share it with other colleague to have them taking a look as well16:13
johnsomI will try to get that into the 2.0.2 queens release16:13
*** sapd1_ has quit IRC16:19
jitekajohnsom: did that new version you told me about yesterday was released ?16:28
jitekajohnsom: or it's 2.0.2 ?16:28
johnsomIt's 2.0.2, it didn't get merged last night for whatever reason, so I will slip it in that version.16:29
jitekajohnsom: ok I was correct then assuming that it wasn't available yet16:32
johnsomYeah, just didn't get reviewed last night16:32
*** sapd1_ has joined #openstack-lbaas16:32
sapd1_johnsom: I don't know why This have to join all table.16:32
sapd1_ https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L9616:32
johnsomsapd1_ Yeah, I have started to look at some of these issues. It is definitely a regression in API performance.16:33
johnsomI know that there was some concern with the number of DB connections sqlalchemy was making, which led to some of these changes. But we need to re-evaluate and work on some optimizations.16:34
sapd1_johnsom: When I remove the options. API performance is increased16:34
johnsomYes, it's just not that simple sadly. It's going to take some work16:34
jitekajohnsom: Applying the current stable/queens with running LB was a bit scary, all active LB transitioned their state to PENDING_UPDATE. Hopefully VIP were still reachable but not sure how I would recover from this state if it was production vip16:35
johnsomjiteka Did you follow the upgrade guide steps?16:36
jitekajohnsom: forcing a state change via CLI is not a thing for the moment right ? something like loadbalancer set --state error LB_ID16:36
jitekajohnsom: no that was my mistake when I did the db upgrade head octavia service were running16:36
johnsomPENDING_UPDATE means the controllers have taken ownership and are actively doing something with the LBs. Forcing will toast your cloud16:36
jitekajohnsom: but restarting services kept them in PENDING_UPDATE16:37
jitekajohnsom: as I was on an unstable version (not including your fix), I destroyed all LB (after moving their state in error on the DB itself)16:37
jitekajohnsom: were just testing LBs16:37
johnsomYou restarted gracefully right?  Not a kill -9?16:37
jitekajohnsom: yes it was not a kill16:38
johnsomThen if they were in PENDING_* the processes should not exit until they are back in ACTIVE or ERROR16:38
johnsomI should also comment, I don't use OSP, so things might be different in OSP16:39
jitekajohnsom: in our deployement systemctl stop octavia-{api,housekeeping,worker,health-manager} is actually a stop on a docker container16:40
johnsomI'm not really sure why they would flip to PENDING_UPDATE either, other than somehow it decided they all needed to failover16:40
jitekait's a "docker stop"16:40
johnsomjiteka OH! If that does not gracefully shutdown the processes you are going to have a lot of pain16:40
jitekajohnsom: actually each restart is a bit painfull because it takes something like 3 attempt on any call that do a POST to get it pass to the handler (most of the time octavia-worker) to get the action done16:41
jitekajohnsom: was thinking about worker not connecting faster enough to the rabbitMQ16:42
jitekajohnsom: I'm not using OSP16:42
jitekajohnsom: using source with in-house CI/CD with control plane living in docker containers and configuration managed via custom puppet module16:42
johnsomOh, ok, sorry I got confused with all of the new installations16:43
jitekaahaha yes I understand16:43
jitekabut no, no ansible-kolla or OSA or OSP distro16:43
johnsomTell me more about this 3 attempts?  What is going on?16:43
jitekaso for example I'm restarting the service because I changed a value in conf or just pushed a new image version of my dockers images (that will be used at the next start)16:44
jitekaif I try to create a new LB16:44
jitekait will always fail 3 times before working16:44
jitekasame with trying to delete an existing LB16:44
jitekabut anything that is handle at the API level work16:45
johnsomRight after the restart our just always?16:45
jitekalike loadbalancer list or loadbalancer amphora list16:45
jitekajust after the restart16:45
jitekaas soon as the first action reach octavia-worker16:45
johnsomAre you fronting the API with a load balancer? like haproxy?16:45
jitekaall the others will also get to them (octavia-workers)16:45
jitekayes I have 2 controller nodes running the docker containers with a haproxy balancing api traffic between them16:46
johnsomlist calls don't go to the worker, they are serviced by the API layer directly. So no rabbit there16:46
jitekaI see my delete/create call on api logs16:46
jitekabut they don't get to octavia-worker16:46
jitekaand after insisting16:46
jitekait works16:46
johnsomSo the sequence is:16:47
jitekalist always works16:47
johnsomOk. hmm, and you are on queens?16:47
jitekaonly POST call that need to be handle from octavia-worker fail 2 time then works the 3rd16:47
jiteka(or fail 3 times then work the 4rd I don't remember)16:47
jitekayes running everything in stable/queens16:47
johnsomAnd by fail, the user gets an error back on the command line?16:48
jitekano the user get a 20016:48
jitekabut nothing happened on the backend16:48
johnsomSo it's going onto the queue then. And the controllers doen't run all three calls?16:48
jitekaonly 116:49
jitekabut that's only happening after a restart16:49
jitekaeach call get process otherwise16:49
johnsomThat is super odd, if the user got 200 it should mean it posted the action to the rabbit queue. When a controller comes up it should pop that from the queue.  Three calls should mean three  executions16:50
johnsomIs rabbit somehow getting restarted too and not running with durrable queues?16:50
*** velizarx has quit IRC16:50
jitekaI don't restart the rabbit when restarting my docker containers with octavia services inside (it's 1 container per service)16:51
jitekaif it help here are the few parameter relative to rabbitMQ http://paste.openstack.org/show/728545/16:52
jitekabut I'm not sure thats the culprit, was just a guess16:53
jitekasome of these value may come from config we run for other modules (like nova, neutron etc.)16:53
jitekawe are still tweaking the service16:54
johnsomjiteka Yeah, you have me stumped on that one.16:55
johnsomI guess open a story on it with steps to reproduce16:55
rm_workdolly: you're using my multi-az patch? :P16:56
johnsomWe don't set any of those rabbit specific settings, so I can't talk to those, but hmmm16:56
jitekajohnsom: first I will see if it's still a problem when deploying 2.0.216:56
johnsomCould be this: kombu_reconnect_delay = 1516:57
sapd1_jiteka: Could you share steps to reproduce?16:57
johnsomIf it's waiting like 15 seconds before reconnecting to the queue16:57
johnsombug still, should get all three calls I thingk16:57
jitekajohnsom: just as precision, I'm not trying to delete or create LB right after the restart. I let sometime few minutes but it still happening17:00
jitekajohnsom sapd1 : I'm retrying it now and will share step in detail and logs17:00
jitekaactually I'm running out of time, will take care of this tomorrow morning17:03
sapd1_jiteka: are you running Octavia in production now?17:04
jiteka only cloud admin have access to APIs for the moment17:05
jitekawouldn't leave it like that if it was in production ^^17:05
johnsomOk, please open a story with the steps17:06
jitekajohnsom: ok I will17:07
rm_workyeah i'm unclear why all your LBs went to PENDING_UPDATE still O_o17:21
rm_workthere is no reason for that to happen automatically unless like, all of them tried to failover17:21
rm_workor did we figure that out already and i missed it17:21
johnsomNo, no idea on that one17:21
johnsomrm_work Can you take a quick look at: https://review.openstack.org/#/c/594332/17:22
rm_workyeah i saw17:22
rm_workwhich patch didn't we backport?17:22
johnsomthe "DELETED" as 404 patch as it changed the API version17:22
rm_workah <_<17:23
johnsomThis one: https://review.openstack.org/#/c/564430/17:23
rm_worki wish we could have merged that17:24
rm_worki only -1'd it because i thought we should have the discussion17:24
rm_worknot because i didn't want to do it :P17:24
johnsomYeah, it's a good point though that we don't want to bump that version on queens.... We need that to fix the tests (which I am working on right now)17:25
*** velizarx has joined #openstack-lbaas17:29
*** numans_ has quit IRC17:54
*** luksky has joined #openstack-lbaas18:02
*** sapd1_ has quit IRC18:08
*** dolly_ has joined #openstack-lbaas18:31
dolly_Hi again guys, sorry to bother this much but I feel that I'm pretty close to get this up and running. As I said earlier was to use the github stable/queens branch and put it into a container. This seems to be all fine and dandy and most things seems to work. Except for the health-manager, doesn't say much except this, http://paste.openstack.org/show/728549/18:34
dolly_Now, 1 ) is the healthmanager responsible for detecting if an amphora dissapears and thus trigger a build of a new one ?18:35
dolly_Cause that would explain why that doesn't happen in my current setup =)18:35
*** abaindur has joined #openstack-lbaas18:36
*** abaindur has quit IRC18:36
*** abaindur has joined #openstack-lbaas18:37
*** dougwig has quit IRC18:42
*** fyx has quit IRC18:42
*** mnaser has quit IRC18:42
johnsomdolly_ That is correct, those errors mean that stats are not getting update and the health monitoring of the amphroa is not active. (note, those should probably be ERROR and not warning)18:43
dolly_ok cool. then I just need to understand why the health-manager cant find that health_db/stats_db driver, right ?18:45
dolly_ok cool18:45
johnsomIt should be using setuptools entrypoints, loaded from the setup.cfg in octavia18:45
johnsomat install time18:45
dolly_awesome, I'll dig deeper =)18:58
*** mnaser has joined #openstack-lbaas19:09
*** luksky11 has joined #openstack-lbaas19:14
*** luksky has quit IRC19:17
*** celebdor has joined #openstack-lbaas19:22
*** luksky has joined #openstack-lbaas19:36
*** luksky11 has quit IRC19:38
*** PagliaccisCloud has quit IRC19:39
*** cgoncalves has quit IRC19:39
*** PagliaccisCloud has joined #openstack-lbaas19:46
*** dolly_ has quit IRC19:47
*** luksky11 has joined #openstack-lbaas19:56
*** luksky has quit IRC20:00
*** beisner_ has joined #openstack-lbaas20:12
*** velizarx has quit IRC20:17
*** crazik has quit IRC20:19
*** beisner has quit IRC20:19
*** beisner_ is now known as beisner20:19
nmagnezijohnsom, thanks for the vote :)20:23
nmagnezijohnsom, now that list looks great :D https://etherpad.openstack.org/p/octavia-priority-reviews20:24
johnsomnmagnezi It didn't make RC2 as I didn't have enough cores for the backport vote, but if we can get that merged back in stable I will do an RC3 for you.20:25
nmagnezijohnsom, I don't think we need an RC3 just for that one20:25
nmagnezijohnsom, I'll propose a backports anyways, but don't cut another RC just for that one20:26
johnsomOk, if we can turn it around it's not that big of deal, I just didn't want to hold everything else20:26
johnsomif I couldn't get a core quorum20:26
nmagnezijohnsom, got it20:28
*** blake has joined #openstack-lbaas20:29
openstackgerritMerged openstack/neutron-lbaas-dashboard master: Remove obsolete gate hooks  https://review.openstack.org/59424320:44
*** blake has quit IRC20:53
*** blake has joined #openstack-lbaas20:54
*** harlowja has joined #openstack-lbaas21:02
*** luksky11 has quit IRC21:29
KeithMnemonicsmcginnis: I think this one is ready now https://review.openstack.org/#/c/589576/21:50
johnsomKeithMnemonic Wrong channel?21:52
KeithMnemonichow are you doing today johnsom? I hope you are doing well21:55
*** blake_ has joined #openstack-lbaas21:55
*** blake_ has quit IRC21:55
johnsomHa, yeah, it's going along....  Fun with tempest tests today. You?21:55
KeithMnemonicso my octavia, designate, neutron troubleshooting session got accepted ;-).21:55
johnsomI won't be in Berlin, but will be cheering for you remote21:56
KeithMnemonicthanks, i think we will focus on amphora launch, maybe HA21:59
*** blake has quit IRC21:59
KeithMnemonici have about 25 minutes or so to cover it21:59
johnsomLet me know if you want me to review something or you have any questions22:00
*** rcernin has joined #openstack-lbaas22:10
johnsomnmagnezi dayou If you have a minute, I have fix for Queens: https://review.openstack.org/#/c/594332/ then we can cut 2.0.222:38
*** celebdor has quit IRC23:14

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!