16:01:32 <johnsom> #startmeeting Octavia
16:01:33 <openstack> Meeting started Wed Aug 21 16:01:32 2019 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:37 <openstack> The meeting name has been set to 'octavia'
16:01:40 <johnsom> Sorry I was late...
16:01:40 <ataraday_> hi
16:01:43 <cgoncalves> hi
16:02:00 <colin-> o/
16:02:15 <johnsom> I was working to try to put an agenda together...
16:02:17 <johnsom> #link https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2019-08-21
16:02:25 <gthiemonge> hi
16:02:26 <johnsom> Hi everyone
16:02:36 <johnsom> #topic Announcements
16:03:11 <johnsom> First up, the ranked list for the "U" cycle name is posted. Once the legal review is done the "U" name will be announced.
16:03:39 <johnsom> FYI, the zuul log archive is now different. I mentioned this last week, but I have seen some questions about it.
16:03:45 <johnsom> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008313.html
16:04:10 <johnsom> What I have found is that if you use the top "Logs" tab, then the logs still have links and can be filtered.
16:04:25 <johnsom> It seems slower IMO, but at least it still works.
16:05:18 <johnsom> Personally I think the old way was better, but....
16:05:29 <cgoncalves> I had not noticed the "Logs" tab. thank you!
16:05:33 <johnsom> Finally my weekly reminder:
16:05:35 <cgoncalves> yeah, me too
16:05:39 <johnsom> Feature freeze is rapidly approaching. We must have features merged by Sept 2nd for library changes and Sept. 9th for everything else.
16:06:09 <johnsom> Any other announcements today?
16:06:43 <johnsom> #topic Brief progress reports / bugs needing review
16:06:48 <cgoncalves> more like a personal announcement that I will be on PTO and miss the next two meetings
16:06:59 <johnsom> Ok, enjoy!
16:07:19 <ataraday_> cgoncalves, have a good vacation!
16:07:48 <johnsom> I have been focusing on re-working the failover flow. We know there are some pretty major issues in the flow and now is the time to fix those.
16:08:38 <johnsom> I have added sub-flow retries, task retries, passive failures, support for missing amps, and code to fix the VIP so far, but there is much left to do....
16:09:23 <johnsom> Also, ataraday_ I poked the Oslo folks and your fix for the mysql column storage mereged:
16:09:26 <johnsom> #link https://review.opendev.org/675388
16:09:29 <ataraday_> I continue work on taskflow to db obj to dicts, bug in taskflow is merged so this is ready for review again https://review.opendev.org/#/c/662791/
16:09:40 <ataraday_> johnsom, thanks a lot!
16:09:49 <johnsom> It may still be a week or two before it is released in a package and upper constraints updated
16:10:00 <cgoncalves> it is a very good start! are you going to push it all in one patch?
16:10:23 <johnsom> There are many patches open last time I checked
16:10:37 <johnsom> #link https://review.opendev.org/#/c/662791/
16:10:58 <cgoncalves> johnsom, I meant for the failover flow work
16:11:02 <johnsom> Added a link so it is highlighted in the meeting notes
16:11:09 <colin-> https://review.opendev.org/#/c/673518/ saw adam's comment in here about testing, going to try cherry picking this down for some basic workflow validation this week
16:11:18 <johnsom> Ah, failover. A strong maybe.....
16:11:24 <colin-> but, don't wait on us :)
16:11:53 <cgoncalves> colin-, awesome!
16:12:25 <johnsom> colin- Please also pull down the child patch:
16:12:26 <johnsom> #link https://review.opendev.org/#/c/675063/1
16:12:27 <cgoncalves> we have not yet been able to test it down due to last minute CI/infra issues
16:12:56 <colin-> ah interesting, good note thx
16:13:29 <johnsom> ataraday_ I did not get a chance to look at the listener patch yet. Still on my list.
16:13:49 <cgoncalves> the VIP ACL patch is now ready for reviews
16:13:51 <cgoncalves> #link https://review.opendev.org/#/c/659626/
16:13:58 <colin-> nice
16:14:00 <johnsom> Yay!
16:14:31 <cgoncalves> I know the AAP driver misses some test coverage, hence the Workflow-1. but don't feel discourage to review it
16:15:06 <cgoncalves> next I will be working on a tempest test, but will take me some time till I get to it
16:16:00 <johnsom> Any other updates today?
16:16:20 <cgoncalves> the amphora image size should now be noticable smaller
16:16:22 <cgoncalves> #link https://review.opendev.org/#/c/639155/
16:16:31 <cgoncalves> *noticeable
16:16:38 <ataraday_> johnsom, I checked today - rebase when well, there was an issue with my devstack. But there is an issue with cascade delete, which I point on the 9th patch set. And probably the fix for it should a bit bigger than I though.
16:17:23 <cgoncalves> we also switched taskflow engine to parallel. that should accelerate some flow operations like active-standby LB creation
16:17:25 <cgoncalves> #link https://review.opendev.org/#/c/676379/
16:17:30 <johnsom> ataraday_ Ok, do you still want me to look at it, or do you have that covered?
16:17:46 <johnsom> #link http://tarballs.openstack.org/octavia/test-images/
16:18:04 <johnsom> FYI, this is how I look at image sizes
16:18:27 <cgoncalves> I have some patches open in DIB to reduce the footprint of yum-minimal images
16:18:31 <cgoncalves> #link https://review.opendev.org/#/q/topic:yum-reduce-footprint
16:19:02 <johnsom> Cool
16:19:26 <ataraday_> johnsom, I think I will make it work and than ask for review, no worries for now
16:19:46 <johnsom> ataraday_ Ok, thank you. Sorry I have been so busy on other tasks.
16:20:23 <johnsom> #topic Community goals
16:20:53 <johnsom> There are three community goals for Train. I wanted to review those real quick as we are getting close to feature freeze, etc.
16:21:07 <johnsom> First up is the python3 (3.7) goal.
16:21:14 <johnsom> #link https://governance.openstack.org/tc/goals/train/python3-updates.html
16:21:33 <johnsom> I think we are done/good here. Does anyone think otherwise?
16:22:09 <johnsom> We have had python3 gates for some time, I think the main change here was adding 3.7
16:23:11 <johnsom> Ok then, we are already marked as done there, so happy dance.
16:23:17 <johnsom> Second is PDF docs
16:23:24 <johnsom> #link https://governance.openstack.org/tc/goals/train/pdf-doc-generation.html
16:24:06 <johnsom> I took lead on this, but the job infrastructure for this goal is still not really ready.
16:24:18 <johnsom> I have created an etherpad to track our test results:
16:24:20 <johnsom> #link https://etherpad.openstack.org/p/pdf-goal-train-octavia
16:25:11 <johnsom> There is also a overall tracking etherpad:
16:25:13 <johnsom> #link https://etherpad.openstack.org/p/train-pdf-support-goal
16:25:29 <johnsom> I will continue to track this work, but I consider this goal at-risk for train.
16:26:01 <johnsom> Any questions/comments on this goal?
16:26:06 <cgoncalves> does it need to be completed by feature freeze?
16:26:39 <johnsom> Technically, probably not as it's just a docs job.
16:26:41 <cgoncalves> what I am understanding is that it is at risk but not blocked on us
16:27:10 <johnsom> Correct, it is blocked on the infrastructure/jobs being functional for us to try/use
16:27:20 <cgoncalves> right
16:27:40 <johnsom> This is part of what is blocking:
16:27:42 <johnsom> #link https://review.opendev.org/#/c/664555/
16:28:22 <johnsom> Finally the IPv6 goal:
16:28:30 <johnsom> #link https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
16:28:39 <johnsom> I think we had some proposed patches for this.
16:28:58 <cgoncalves> #link https://review.opendev.org/#/c/594078/
16:29:01 <johnsom> As you know, we have IPv6 tempest scenarios already. This goal, for us, is about the control plane.
16:29:31 <johnsom> I.e. calling out to the other services using IPv6   and running the lb-mgmt-net as pure IPv6.
16:30:05 <johnsom> How is that going?
16:30:52 * johnsom hears crickets
16:30:52 <cgoncalves> it isn't from my side. haven't had cycles to work on that. rm_work seemed to have picked up the work for a while
16:31:18 <johnsom> Ok, maybe we can get an update from him when he is back online.
16:31:59 <johnsom> Any other questions/comments/updates on the community goals?
16:32:37 <johnsom> #topic Open Discussion
16:32:45 <johnsom> Ok, any other topics for today?
16:32:58 <colin-> what is the best way to visualize the data a healthmonitor is receiving from a given member?
16:33:17 <colin-> are the GETs/POSTs and corresponding replies logged somewhere i am missing?
16:33:52 <johnsom> Some of that data is logged at the debug level
16:34:19 <colin-> is the amphora-agent carying it out? i checked its log in hopes of finding it (not debug) and didn't see
16:34:34 <johnsom> I.e.
16:34:35 <johnsom> Aug 12 07:04:33 devstack octavia-health-manager[14967]: DEBUG octavia.controller.healthmanager.health_drivers.update_db [-] Listener 34e6feee-6ced-4296-8652-4668a87d2350 / Amphora 8b556645-e8b0-4101-a69e-6e8c5f5a70c4 stats: {'bytes_in': 146, 'bytes_out': 157, 'active_connections': 0, 'total_connections': 2, 'request_errors': 0} {{(pid=29660) _update_stats
16:34:36 <johnsom> /opt/stack/octavia/octavia/controller/healthmanager/health_drivers/update_db.py:543}}
16:35:20 * cgoncalves has a hard stop now. o/
16:35:27 <johnsom> There is also acknowledgment of a packet received:
16:35:28 <johnsom> Aug 12 07:04:43 devstack octavia-health-manager[14967]: DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('192.168.0.74', 64717) {{(pid=15515) dorecv /opt/stack/octavia/octavia/amphorae/drivers/health/heartbeat_udp.py:189}}
16:35:43 <johnsom> But I don't think we dump the status payload into the debug log.
16:36:02 <colin-> so for context i'm setting up a healthmonitor and playing around with url_path trying to get my monitor healthy
16:36:09 <colin-> and it's just challenging when i'm not sure what the monitor is seeing
16:36:29 <johnsom> Oh, sorry, my bad. I was thinking health manager.... sigh
16:36:32 <colin-> ;)
16:37:31 <johnsom> Health monitor.... This is done by the haproxy engine. It is not done by the amphora agent.  Any messages about those will be in the tenant flow logs in the new log offloading.
16:38:10 <colin-> absent the offloading (not runnint it locally yet) do i have any other options?
16:38:17 <colin-> i guess crank up the debug/logging on haproxy
16:38:59 <johnsom> Yeah, you can look directly at the haproxy logs inside the amphora. All health monitoring results and state transitions are listed there.
16:39:20 <colin-> got it, thanks
16:39:56 <johnsom> If you want to see the content of the health monitor check, you would need to run tcpdump inside the network namespace. (remember to bring up lo interface)
16:40:22 <johnsom> Actually for tcpdump, you may not need the lo up
16:41:48 <johnsom> Ok, any other topics today?
16:43:04 <johnsom> Alright. Thank you folks! Have a great week.
16:43:06 <johnsom> #endmeeting