| *** Liang__ has quit IRC | 00:03 | |
| *** Liang__ has joined #openstack-meeting | 00:04 | |
| *** rcernin has joined #openstack-meeting | 00:08 | |
| *** artom has quit IRC | 00:10 | |
| *** slaweq has joined #openstack-meeting | 00:11 | |
| *** ykatabam has quit IRC | 00:14 | |
| *** artom has joined #openstack-meeting | 00:14 | |
| *** artom has quit IRC | 00:15 | |
| *** artom has joined #openstack-meeting | 00:15 | |
| *** slaweq has quit IRC | 00:16 | |
| *** ykatabam has joined #openstack-meeting | 00:18 | |
| *** macz has quit IRC | 00:38 | |
| *** armax has joined #openstack-meeting | 00:43 | |
| *** rf0lc0 has quit IRC | 00:50 | |
| *** rcernin has quit IRC | 00:55 | |
| *** rcernin has joined #openstack-meeting | 00:55 | |
| *** masayukig is now known as migawa|AFK | 01:08 | |
| *** migawa|AFK is now known as migawa | 01:09 | |
| *** slaweq has joined #openstack-meeting | 01:11 | |
| *** ykatabam has quit IRC | 01:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:12 | |
| *** ykatabam has joined #openstack-meeting | 01:12 | |
| *** slaweq has quit IRC | 01:16 | |
| *** jmasud has joined #openstack-meeting | 01:17 | |
| *** brinzhang has quit IRC | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:36 | |
| *** TomStappaerts has quit IRC | 01:42 | |
| *** TomStappaerts has joined #openstack-meeting | 01:43 | |
| *** ykatabam has quit IRC | 01:44 | |
| *** ykatabam has joined #openstack-meeting | 01:49 | |
| *** brinzhang has joined #openstack-meeting | 01:51 | |
| *** yamamoto has joined #openstack-meeting | 01:57 | |
| *** ykatabam has quit IRC | 02:02 | |
| *** ykatabam has joined #openstack-meeting | 02:05 | |
| *** slaweq has joined #openstack-meeting | 02:11 | |
| *** ykatabam has quit IRC | 02:11 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** yamamoto has joined #openstack-meeting | 02:13 | |
| *** slaweq has quit IRC | 02:16 | |
| *** yamamoto has quit IRC | 02:17 | |
| *** ricolin has joined #openstack-meeting | 02:19 | |
| *** ykatabam has joined #openstack-meeting | 02:23 | |
| *** ykatabam has quit IRC | 02:28 | |
| *** ykatabam has joined #openstack-meeting | 02:29 | |
| *** ykatabam has joined #openstack-meeting | 02:30 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** macz has joined #openstack-meeting | 02:40 | |
| *** yamamoto has joined #openstack-meeting | 02:40 | |
| *** macz has quit IRC | 02:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:53 | |
| *** yamamoto has quit IRC | 03:01 | |
| *** yamamoto has joined #openstack-meeting | 03:03 | |
| *** rcernin has quit IRC | 03:06 | |
| *** ykatabam has quit IRC | 03:06 | |
| *** ykatabam has joined #openstack-meeting | 03:07 | |
| *** slaweq has joined #openstack-meeting | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| *** psachin has joined #openstack-meeting | 03:21 | |
| *** rcernin has joined #openstack-meeting | 03:23 | |
| *** masahito_ has joined #openstack-meeting | 03:24 | |
| *** senrique_ has quit IRC | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:55 | |
| *** gyee has quit IRC | 04:08 | |
| *** jmasud has quit IRC | 04:10 | |
| *** slaweq has joined #openstack-meeting | 04:11 | |
| *** vishalmanchanda has joined #openstack-meeting | 04:13 | |
| *** slaweq has quit IRC | 04:16 | |
| *** rcernin has quit IRC | 04:16 | |
| *** rcernin has joined #openstack-meeting | 04:16 | |
| *** masahito_ has quit IRC | 04:28 | |
| *** psachin has quit IRC | 04:33 | |
| *** artom has quit IRC | 04:35 | |
| *** macz has joined #openstack-meeting | 04:41 | |
| *** macz has quit IRC | 04:46 | |
| *** migawa is now known as migawa|lunch|AFK | 04:50 | |
| *** rcernin is now known as rcernin|lunch | 05:01 | |
| *** slaweq has joined #openstack-meeting | 05:11 | |
| *** slaweq has quit IRC | 05:16 | |
| *** migawa|lunch|AFK is now known as migawa|lunch | 05:18 | |
| *** TomStappaerts has quit IRC | 05:32 | |
| *** TomStappaerts has joined #openstack-meeting | 05:33 | |
| *** mmethot_ has joined #openstack-meeting | 05:35 | |
| *** mmethot has quit IRC | 05:37 | |
| *** macz has joined #openstack-meeting | 05:41 | |
| *** macz has quit IRC | 05:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:57 | |
| *** macz has joined #openstack-meeting | 05:58 | |
| *** jamesmcarthur has quit IRC | 06:01 | |
| *** macz has quit IRC | 06:03 | |
| *** slaweq has joined #openstack-meeting | 06:11 | |
| *** TomStappaerts has quit IRC | 06:15 | |
| *** TomStappaerts has joined #openstack-meeting | 06:15 | |
| *** slaweq has quit IRC | 06:16 | |
| *** slaweq has joined #openstack-meeting | 07:08 | |
| *** slaweq has quit IRC | 07:15 | |
| *** jmasud has joined #openstack-meeting | 07:17 | |
| *** rsimai has joined #openstack-meeting | 07:20 | |
| *** rpittau|afk is now known as rpittau | 07:21 | |
| *** macz has joined #openstack-meeting | 07:24 | |
| *** macz has quit IRC | 07:29 | |
| *** brinzhang has quit IRC | 07:29 | |
| *** lpetrut has joined #openstack-meeting | 07:31 | |
| *** kopecmartin|afk is now known as kopecmartin | 07:44 | |
| *** TomStappaerts has quit IRC | 07:46 | |
| *** ianychoi has joined #openstack-meeting | 07:46 | |
| *** TomStappaerts has joined #openstack-meeting | 07:46 | |
| *** slaweq has joined #openstack-meeting | 07:51 | |
| *** maciejjozefczyk has joined #openstack-meeting | 07:54 | |
| *** slaweq has quit IRC | 07:57 | |
| *** hammad_ has joined #openstack-meeting | 07:57 | |
| *** slaweq has joined #openstack-meeting | 07:58 | |
| *** lajoskatona has joined #openstack-meeting | 08:00 | |
| *** hammad_ has quit IRC | 08:07 | |
| *** hammad_ has joined #openstack-meeting | 08:07 | |
| *** witek has joined #openstack-meeting | 08:10 | |
| *** jmasud has quit IRC | 08:13 | |
| *** jmasud has joined #openstack-meeting | 08:14 | |
| *** Liang__ has quit IRC | 08:22 | |
| *** Liang__ has joined #openstack-meeting | 08:23 | |
| *** yamamoto has quit IRC | 08:30 | |
| *** ralonsoh has joined #openstack-meeting | 08:38 | |
| *** yamamoto has joined #openstack-meeting | 08:44 | |
| *** TomStappaerts has quit IRC | 08:47 | |
| *** TomStappaerts has joined #openstack-meeting | 08:47 | |
| *** etp has joined #openstack-meeting | 08:52 | |
| *** mdelavergne has joined #openstack-meeting | 08:56 | |
| *** oneswig has joined #openstack-meeting | 08:58 | |
| *** masahito has joined #openstack-meeting | 08:59 | |
| ttx | #startmeeting large_scale_sig | 09:00 |
|---|---|---|
| openstack | Meeting started Wed Feb 12 09:00:39 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| ttx | #topic Rollcall | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 09:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 09:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 09:00 | |
| oneswig | hello | 09:00 |
| mdelavergne | Hi! | 09:00 |
| ttx | Hello hello everyone | 09:01 |
| ttx | Agenda at: | 09:01 |
| witek | good morning | 09:01 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 09:01 |
| *** dougsz has joined #openstack-meeting | 09:01 | |
| ttx | Let's wait for our Asian friends to join | 09:01 |
| dougsz | o/ | 09:01 |
| masahito | o/ | 09:02 |
| etp | o/ | 09:02 |
| ttx | OK, let's get started and maybe others will join | 09:02 |
| ttx | #topic Show and tell | 09:02 |
| *** openstack changes topic to "Show and tell (Meeting topic: large_scale_sig)" | 09:02 | |
| oneswig | Thanks ttx | 09:03 |
| ttx | We'll start with oneswig presenting | 09:03 |
| ttx | oneswig: floor is yours | 09:03 |
| oneswig | I put a link in the etherpad but here's a kind-of presentation... http://www.stackhpc.com/resources/HAproxy-telemetry.pdf | 09:03 |
| oneswig | just a few slides as talking points. We do this in the Scientific SIG sometimes and it mostly works | 09:03 |
| * ttx clicks | 09:03 | |
| oneswig | So I wanted to share some work my colleague dougsz has been doing :-) | 09:04 |
| oneswig | We noticed that HAproxy had sprouted some new telemetry data and wanted to use it. | 09:04 |
| oneswig | It's published in prometheus protocol. | 09:04 |
| oneswig | Doug has done some useful work recently with Monasca-Agent (our monitoring infra) to extend support for drawing data from prometheus sources | 09:05 |
| oneswig | this is desirable because there's a lot of new work coming in that format - Ceph, HAproxy, etc. | 09:06 |
| *** lucasagomes has joined #openstack-meeting | 09:06 | |
| ttx | and oslo.metric | 09:07 |
| oneswig | ttx: indeed :-) It works nicely for us because we get the innovation in prometheus coupled with the advantages of Monasca | 09:07 |
| oneswig | OK, so on slide 2 we have a diagram of Monasca services and how the agent pulls data from prometheus endoints, does some preliminary sanitisation and pushes them to the Monasca API | 09:08 |
| oneswig | Slide 4 is an example end-to-end. That's the API response latencies, as measured by haproxy, sampled and averaged, collated by API endpoint | 09:09 |
| oneswig | Its interesting - to a point. | 09:09 |
| oneswig | One problem with the prometheus data as it stands is that it doesn't appear to split latency out by (eg) response code or http operation | 09:10 |
| ttx | that glance_api_external does not seem to be on a good trends | 09:10 |
| oneswig | the big red ascending saw-tooth looks bad. To make it do that I was repeatedly creating and deleting images. | 09:11 |
| oneswig | (this is only a single-node control plane, it's a dev environment) | 09:11 |
| ttx | oneswig: so success latency is mixed with error latency? | 09:12 |
| oneswig | What it doesn't show is that this is a combination of POST, PUT, GET, DELETE | 09:12 |
| oneswig | ttx: exactly, which isn't perfect. | 09:12 |
| ttx | yes because I read the golden signals chapter | 09:12 |
| oneswig | and brings us to the next part of the work - slide 5 | 09:12 |
| oneswig | :-) | 09:12 |
| oneswig | Doug noticed that haproxy can log copious amounts of useful data, and the log messages appear designed to be parsed into semi-structure ddata | 09:13 |
| oneswig | Forgot to mention at the top - we use Kolla-Ansible, which standardises on a fluentd log aggregator container on each node | 09:13 |
| oneswig | After parsing the logs with some epic regex, we get json structured data about each api request | 09:14 |
| oneswig | This goes into monasca's log pipeline. The box in the bottom right corner can be used to convert log messages into dimensioned time series telemetry (monasca log metrics) | 09:15 |
| oneswig | That gives us individual samples which can be aggregated at various points in the pipeline into histograms | 09:15 |
| oneswig | currently we do this in grafana, but it could be pushed further upstream, even to fluentd | 09:16 |
| *** lajoskatona has left #openstack-meeting | 09:16 | |
| oneswig | The second grafana screenshot shows api responses for one service (keystone in this case, because the data's prettiest:-) | 09:17 |
| ttx | Hmm... y is response time, but what is x in this graph? | 09:18 |
| oneswig | But what we can see is data by operation, interface and response | 09:18 |
| oneswig | frequency density, iirc | 09:18 |
| oneswig | it's binned into 10ms intervals | 09:18 |
| ttx | ah ok | 09:18 |
| oneswig | Unfortunately there weren't errors for this service but if there were, we might see 500 response codes on the graph too. | 09:19 |
| ttx | Keystone never fails | 09:19 |
| oneswig | One shortcoming I guess is that if the number of errors is small, or the timing of them scattered, they'd be lost in the noise along the x-xis | 09:20 |
| ttx | yeah for errors you might want to aggregate on longer periods of time | 09:20 |
| ttx | (or for other rare events) | 09:21 |
| witek | and create a separate dashboard for errors only | 09:21 |
| dougsz | Re. the axis labelling, unfortunately the histogram plugin doesn't support it | 09:21 |
| ttx | dougsz: as long as the reader knows what's on them, I guess it does not matter that much | 09:22 |
| oneswig | a good sample would certainly be more useful. We also have such detailed data here, we could potentially trace every datapoint to a single api request | 09:22 |
| ttx | dougsz, oneswig: That's interesting integration work! Thanks for sharing | 09:22 |
| oneswig | BTW all this work is going / has gone upstream | 09:22 |
| oneswig | It should be useful at various levels - even the general principles for folks not using any of these components | 09:23 |
| ttx | It shows that once you start reducing the parameters (like say you start from Kolla-Ansible so you can rely on fluentd/monasca being present), it simplifies things a lot | 09:23 |
| ttx | (in the telemetry gathering space) | 09:23 |
| oneswig | I think so. One thing missing is a nice starting point in terms of dashboards. It's pretty basic what you get as standard | 09:24 |
| ttx | questions on this presentation? | 09:24 |
| oneswig | doug did I miss anything? | 09:24 |
| dougsz | Indeed, we have a goal to pre-configure monitoring out of the box in Kolla-Ansible | 09:24 |
| dougsz | I think you captured it nicely, thanks oneswig | 09:25 |
| ttx | oneswig: do you mind if I include a link to your slides on the meeting summary sent to the ML? | 09:25 |
| dougsz | It might be worth mentioning that Prometheus users can take advantage of the log parsing via the Fluentd Prometheus plugin in Kolla | 09:25 |
| oneswig | BTW the glance API - I ran that test overnight, it reached a steady state around 3.5s :-) | 09:25 |
| ttx | OK, if no more questions we'll switch to our next topic... | 09:26 |
| ttx | #topic Progress on "Documenting large scale operations" goal | 09:26 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 09:26 | |
| ttx | amorin: around? | 09:26 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 09:26 |
| *** tetsuro has joined #openstack-meeting | 09:27 | |
| ttx | No amorin today, so we'll get how things are going on the config default documentation effort in a future meeting | 09:27 |
| ttx | We also had a bunch a links added to the etherpad, which is great. I briefly scanned them and feel like they all belong. | 09:28 |
| ttx | #topic Progress on "Scaling within one cluster" goal | 09:28 |
| *** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 09:28 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 09:28 |
| ttx | The oslo.metric blueprint got its first round of reviews, mostly from outside the SIG | 09:28 |
| ttx | #link https://review.opendev.org/#/c/704733/ | 09:28 |
| ttx | Would be great to get more feedback from the SIG participants themselves, before masahito does another round. | 09:29 |
| ttx | So I propose to add a TODO for the group to review that spec ASAP | 09:29 |
| ttx | (as soon as possible) | 09:29 |
| masahito | I don't have much update this week. I'll update the patch without adding actual schema in this week. | 09:30 |
| masahito | s/patch/draft/ | 09:30 |
| ttx | #action masahito to update spec based on initial feedback | 09:30 |
| ttx | #action everyone to review and comment on https://review.opendev.org/#/c/704733/ | 09:30 |
| ttx | Regarding golden signals, did anyone read the recommended read ? | 09:31 |
| ttx | #link https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/#xref_monitoring_golden-signals | 09:31 |
| ttx | Basically the summary is that any monitoring system should watch latency, traffic, errors, and saturation as early detectors of impeding failure | 09:31 |
| ttx | So making sure this group has a common understanding of those terms is useful. | 09:31 |
| *** oneswig_ has joined #openstack-meeting | 09:32 | |
| ttx | Please read it (it's short) if you haven't had the chance yet | 09:32 |
| ttx | On the scaling stories front, thanks to Albert Braden who posted to the doc recently | 09:32 |
| ttx | #link https://etherpad.openstack.org/p/scaling-stories | 09:32 |
| ttx | I feel like some of it should find its way to amorin's documentation efforts | 09:33 |
| ttx | comments on that? | 09:33 |
| ttx | Sounds like a no | 09:34 |
| ttx | #topic Other topics | 09:34 |
| *** openstack changes topic to "Other topics (Meeting topic: large_scale_sig)" | 09:34 | |
| ttx | So... we have an "Opendev" event coming up in Vancouver, June 8-11 | 09:34 |
| ttx | I wanted to talk briefly about it and how this group can get involved | 09:35 |
| *** oneswig has quit IRC | 09:35 | |
| ttx | For those not familiar with the format, it's a workshop-style event where we deep-dive into a number of predefined topics, with lots of flexibility in the exact event format | 09:35 |
| ttx | One of the 4 themes selected for the event is "Large-scale Usage of Open Source Infrastructure Software" | 09:35 |
| ttx | Which is directly aligned with this SIG | 09:35 |
| ttx | So I think we should definitely participate to the programming committee and help select the content that will be discussed there | 09:36 |
| ttx | Like make sure we have some part of the event discussing our "scaling one cluster" and "documenting large scale use defaults" goals | 09:36 |
| ttx | And also take the opportunity to recruit more people to this SIG. | 09:36 |
| ttx | I saw that Belmiro applied to be on the programming committee, I'll make sure to support that application. I myself will be focused on other tracks at the same event | 09:37 |
| ttx | So if others (masahito? oneswig? amorin? jiaopingju?) are interested in driving the content that will be discussed there, please let me know | 09:37 |
| masahito | Thank. I plan to go to the Opendev. But not decided yet. | 09:38 |
| mdelavergne | I might be able to help, but I am not sure I will physically go there | 09:38 |
| masahito | If I can go, I'm okay to be a moderator. | 09:39 |
| ttx | That would be great | 09:39 |
| ttx | OK, we still have some tie, but I know the event organizers want to select the programming committees ASAP | 09:39 |
| ttx | some time* | 09:39 |
| masahito | Is there room or talk submission for the event now? | 09:40 |
| ttx | masahito: It depends on the choice of each programming committee. The general idea is to have a limited number of formal presentations, so in most cases the programming committee will just invite a specific speaker | 09:41 |
| ttx | But some tracks might do a more classic CFP | 09:41 |
| masahito | Got it. Thanks. | 09:41 |
| ttx | and some others might do more of a short-presentations session | 09:41 |
| ttx | (this is why it's important to have members of this SIG in the programming committee) | 09:42 |
| ttx | masahito: in case you can confirm your presence soon, I'd be happy to pass your name to be a part of the committee | 09:42 |
| ttx | if you'll know later, we can always have you moderate some topic | 09:43 |
| ttx | like a workshop on gathering metrics on bottlenecks in your OpenStack clusters | 09:43 |
| masahito | okay. | 09:43 |
| ttx | Other questions on this event? | 09:44 |
| ttx | Other topics to discuss? | 09:44 |
| mdelavergne | Not from myself right now | 09:44 |
| ttx | #topic Next meeting | 09:44 |
| masahito | Nothing from my side. | 09:44 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 09:44 | |
| ttx | Our next meeting should be Feb 26, 9 utc | 09:44 |
| ttx | Hopefully we'll have more people around | 09:44 |
| ttx | Anything else before we close the meeting? | 09:45 |
| ttx | If not, thank you for attending! | 09:45 |
| mdelavergne | Thanks everyone! | 09:46 |
| ttx | #endmeeting | 09:46 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 09:46 | |
| openstack | Meeting ended Wed Feb 12 09:46:18 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:46 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-12-09.00.html | 09:46 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-12-09.00.txt | 09:46 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-02-12-09.00.log.html | 09:46 |
| masahito | thanks | 09:46 |
| *** e0ne has joined #openstack-meeting | 09:54 | |
| *** ociuhandu has joined #openstack-meeting | 09:56 | |
| *** mdelavergne has quit IRC | 10:02 | |
| *** jiaopengju2 has joined #openstack-meeting | 10:04 | |
| *** TomStappaerts has quit IRC | 10:08 | |
| *** TomStappaerts has joined #openstack-meeting | 10:08 | |
| *** ykatabam has quit IRC | 10:22 | |
| *** rcernin|lunch has quit IRC | 10:22 | |
| *** kaisers has quit IRC | 10:27 | |
| *** kaisers has joined #openstack-meeting | 10:35 | |
| *** rcernin|lunch has joined #openstack-meeting | 10:36 | |
| *** vishalmanchanda has quit IRC | 11:12 | |
| *** armax has quit IRC | 11:14 | |
| *** armax has joined #openstack-meeting | 11:19 | |
| *** Lucas_Gray has joined #openstack-meeting | 11:27 | |
| *** oneswig_ has quit IRC | 11:31 | |
| *** rpittau is now known as rpittau|bbl | 11:33 | |
| *** hammad_ has quit IRC | 11:35 | |
| *** armax has quit IRC | 11:44 | |
| *** raildo has joined #openstack-meeting | 11:47 | |
| *** nicolasbock has joined #openstack-meeting | 12:06 | |
| *** ociuhandu has quit IRC | 12:09 | |
| *** ociuhandu has joined #openstack-meeting | 12:10 | |
| *** rfolco has joined #openstack-meeting | 12:10 | |
| *** jmasud has quit IRC | 12:13 | |
| *** jmasud has joined #openstack-meeting | 12:14 | |
| *** jiaopengju2 has quit IRC | 12:21 | |
| *** jiaopengju2 has joined #openstack-meeting | 12:22 | |
| *** TomStappaerts has quit IRC | 12:24 | |
| *** TomStappaerts has joined #openstack-meeting | 12:24 | |
| *** ianychoi has quit IRC | 12:25 | |
| *** masahito has quit IRC | 12:25 | |
| *** masahito has joined #openstack-meeting | 12:26 | |
| *** rfolco has quit IRC | 12:26 | |
| *** rfolco has joined #openstack-meeting | 12:26 | |
| *** masahito has quit IRC | 12:26 | |
| *** masahito has joined #openstack-meeting | 12:27 | |
| *** masahito has quit IRC | 12:27 | |
| *** TomStappaerts has quit IRC | 12:29 | |
| *** TomStappaerts has joined #openstack-meeting | 12:30 | |
| *** ykatabam has joined #openstack-meeting | 12:37 | |
| *** TomStappaerts has quit IRC | 13:03 | |
| *** TomStappaerts has joined #openstack-meeting | 13:03 | |
| *** Liang__ has quit IRC | 13:03 | |
| *** Liang__ has joined #openstack-meeting | 13:04 | |
| *** rpittau|bbl is now known as rpittau | 13:05 | |
| *** senrique_ has joined #openstack-meeting | 13:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:09 | |
| *** ociuhandu has quit IRC | 13:11 | |
| *** manuvakery has quit IRC | 13:13 | |
| *** TrevorV has joined #openstack-meeting | 13:16 | |
| *** Lucas_Gray has quit IRC | 13:31 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:34 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** rh-jelabarre has joined #openstack-meeting | 13:39 | |
| *** nicolasbock has quit IRC | 13:43 | |
| *** TomStappaerts has quit IRC | 13:44 | |
| *** TomStappaerts has joined #openstack-meeting | 13:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:45 | |
| *** nicolasbock has joined #openstack-meeting | 13:47 | |
| *** ociuhandu has joined #openstack-meeting | 13:48 | |
| *** rcernin|lunch has quit IRC | 13:50 | |
| *** liuyulong has joined #openstack-meeting | 13:51 | |
| *** ociuhandu has quit IRC | 13:52 | |
| *** eharney has quit IRC | 13:55 | |
| liuyulong | #startmeeting neutron_l3 | 13:58 |
| openstack | Meeting started Wed Feb 12 13:58:47 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:58 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:58 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 13:58 | |
| openstack | The meeting name has been set to 'neutron_l3' | 13:58 |
| liuyulong | #topic Announcements | 13:58 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 13:58 | |
| slaweq | hi | 13:59 |
| liuyulong | hi | 13:59 |
| liuyulong | Ussuri-2 is this week. | 13:59 |
| liuyulong | And we have this list: | 14:00 |
| liuyulong | #link https://launchpad.net/neutron/+milestone/ussuri-2 | 14:00 |
| *** Liang__ is now known as LiangFang | 14:00 | |
| *** sfernand has joined #openstack-meeting | 14:00 | |
| *** TomStappaerts has quit IRC | 14:01 | |
| liuyulong | Two RFE related bugs are not finished. So maybe we should move it to next U-3 | 14:02 |
| *** TomStappaerts has joined #openstack-meeting | 14:02 | |
| *** lbragstad has joined #openstack-meeting | 14:02 | |
| liuyulong | OVN migrations are quite good I guess. | 14:02 |
| liuyulong | OK, I just read the book, haha. | 14:03 |
| liuyulong | No more from me now. | 14:03 |
| liuyulong | Any updates? | 14:03 |
| *** senrique_ is now known as enriquetaso | 14:04 | |
| liuyulong | OK, let's move on. | 14:04 |
| liuyulong | #topic Bugs | 14:04 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:04 | |
| ralonsoh | hi | 14:04 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012477.html | 14:04 |
| liuyulong | ralonsoh, hi | 14:04 |
| *** ociuhandu has joined #openstack-meeting | 14:04 | |
| liuyulong | First one | 14:05 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1862315 | 14:05 |
| openstack | Launchpad bug 1862315 in neutron "Sometimes VMs can't get IP when spawned concurrently" [High,New] - Assigned to Oleg Bondarev (obondarev) | 14:05 |
| liuyulong | It has a fix now. | 14:05 |
| liuyulong | #link https://review.opendev.org/#/c/707077/ | 14:06 |
| *** lbragstad has quit IRC | 14:08 | |
| liuyulong | sorry, a bit latency on networking... | 14:09 |
| liuyulong | Yes, the fix is fine to me, we can go with that. | 14:09 |
| *** jamesmcarthur has quit IRC | 14:11 | |
| *** vishalmanchanda has joined #openstack-meeting | 14:11 | |
| liuyulong | We have that original fix https://review.opendev.org/#/c/626830/ in our queens branch for a long time, in general it works fine. | 14:11 |
| *** jamesmcarthur has joined #openstack-meeting | 14:12 | |
| liuyulong | I will check our local logstash to see if it has such issue. | 14:13 |
| liuyulong | OK, no more bugs in the deputy list, let's give a quick scan of the the LP. | 14:14 |
| liuyulong | ++,sorry, no more bugs related to L3 in the deputy list | 14:15 |
| *** ociuhandu has quit IRC | 14:15 | |
| *** witek has left #openstack-meeting | 14:15 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1862851 | 14:17 |
| openstack | Launchpad bug 1862851 in neutron "update_floatingip_statuses: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched." [Undecided,New] | 14:17 |
| *** jamesmcarthur has quit IRC | 14:17 | |
| haleyb | hi, sorry was in a downstream chat | 14:17 |
| ralonsoh | this is not the first time we have this bug | 14:17 |
| liuyulong | This is a really common exception in DB related actions. | 14:18 |
| haleyb | we've seen those errors before ^^^, and that bug is against rocky | 14:18 |
| ralonsoh | btw, a patch for a previous bug related to this | 14:19 |
| ralonsoh | https://review.opendev.org/#/c/503517/ | 14:19 |
| ralonsoh | https://bugs.launchpad.net/neutron/+bug/1716321 | 14:19 |
| openstack | Launchpad bug 1716321 in neutron "StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched." [Undecided,Fix released] | 14:19 |
| *** tesseract has quit IRC | 14:20 | |
| liuyulong | Yes, we have met this sometimes also, as far as we can see the exception does not have big effect to the neutron-api, the DB retry action could work. | 14:20 |
| ralonsoh | exactly | 14:21 |
| liuyulong | OK, next | 14:22 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1861876 | 14:22 |
| openstack | Launchpad bug 1861876 in neutron "[Neutron API] Neutron Floating IP not always have 'port_details'" [Critical,New] | 14:22 |
| liuyulong | A critical one, but I did not read the detail yet... | 14:22 |
| slaweq | I think it's fixed now (at least not so critical anymore) | 14:23 |
| liuyulong | I don't think floating IP has such "port_details" attribute. It is another L3 service_providers ? | 14:23 |
| ralonsoh | that was because OVN L3 pluging didn't have the correct mixin | 14:24 |
| ralonsoh | the nova bug is solved | 14:24 |
| ralonsoh | and neutron one | 14:24 |
| ralonsoh | https://review.opendev.org/#/c/705982/ | 14:24 |
| liuyulong | Alright, maybe we can get it merged. | 14:26 |
| liuyulong | OK, that's all from me. | 14:27 |
| liuyulong | Any other bugs? | 14:27 |
| slaweq | I want to ask You to review https://review.opendev.org/#/c/702856/ | 14:28 |
| ralonsoh | sure | 14:28 |
| slaweq | liuyulong: I was playing with putting interface down on standby nodes but it didn't work as I wanted | 14:28 |
| slaweq | there were some errors in keepalived logs during failover | 14:29 |
| liuyulong | : ), maybe I should submit our local fix. | 14:29 |
| slaweq | liuyulong: sure | 14:30 |
| slaweq | if You have something which really works without problems than we can review it | 14:30 |
| liuyulong | I will submit it today. | 14:30 |
| slaweq | liuyulong: thx | 14:31 |
| liuyulong | slaweq, np | 14:33 |
| liuyulong | OK, let's move on | 14:33 |
| liuyulong | #topic OVN_L3 | 14:33 |
| *** openstack changes topic to "OVN_L3 (Meeting topic: neutron_l3)" | 14:33 | |
| *** anastzhyr has joined #openstack-meeting | 14:33 | |
| ralonsoh | maciejjozefczyk, ping | 14:34 |
| ralonsoh | lucasagomes, | 14:34 |
| ralonsoh | ^^ any relevant update this week about OVN L3? | 14:34 |
| lucasagomes | hi there, not from my part | 14:34 |
| liuyulong | We have a bug related to OVN gateway I remember. | 14:35 |
| ralonsoh | btw, we are still dealing with the CI | 14:35 |
| liuyulong | #link https://review.opendev.org/#/c/705660/ | 14:35 |
| liuyulong | This one | 14:35 |
| *** Lucas_Gray has quit IRC | 14:35 | |
| lucasagomes | yeah, specially on the func tests... I know maciejjozefczyk is looking into it. He's uploaded a new patch-set today | 14:35 |
| maciejjozefczyk | liuyulong, Yes we have an OVN gateway related patch: https://review.opendev.org/#/c/705660/ | 14:36 |
| maciejjozefczyk | I needs to be on top of functional one, and those functional test failed on something else. | 14:37 |
| maciejjozefczyk | I pushed today a fix to stablize part of functional tests related failures | 14:37 |
| liuyulong | Yes, I noticed that, it is based on 3 parts of functional test fix. | 14:38 |
| maciejjozefczyk | liuyulong, yes, actually the functional test that I needed to change is in the third part. | 14:39 |
| liuyulong | So I wonder if the stable branch could have the fix reviewing in parallel ? | 14:40 |
| liuyulong | Stable branch does not need to fix the functional test, right? | 14:40 |
| maciejjozefczyk | liuyulong, potentially we could do a review in a stable networking-ovn branch and then apply it on master, but that breaks workflow a bit ;/ | 14:41 |
| *** eharney has joined #openstack-meeting | 14:41 | |
| maciejjozefczyk | liuyulong, or we can review this patch now, have agreement on its shape, then I can cherry-pick it, what do you think? | 14:42 |
| maciejjozefczyk | liuyulong, actually tests that are failing are unrelated. | 14:42 |
| *** jamesmcarthur has joined #openstack-meeting | 14:42 | |
| liuyulong | Both make sense for me, but IMO if it get merged first in networking-ovn, it will be more easily to backport to T/Q/R stable branches. | 14:44 |
| liuyulong | For important patches, my idea is to quickly merge and benefits the end users. | 14:45 |
| maciejjozefczyk | liuyulong, Ok, I'm gonna propose it in stalble/train first, we gonna have a discussion there. Then, if functional tests will be in place, we gonna merge it in Neutron master, agreed? | 14:46 |
| slaweq | I think that we should propose in master first and then do "cherry-pick" to networking-ovn | 14:47 |
| *** masahito has joined #openstack-meeting | 14:47 | |
| *** jamesmcarthur has quit IRC | 14:47 | |
| slaweq | I don't think we should change our normal workflow as maciejjozefczyk already said | 14:47 |
| maciejjozefczyk | slaweq, liuyulong I'm ok with both, this change is important, but not a blocker. | 14:49 |
| liuyulong | Sure, both works | 14:49 |
| maciejjozefczyk | Would be great to have somebody eyes on it first :) | 14:50 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1862648 | 14:51 |
| openstack | Launchpad bug 1862648 in neutron "[OVN] Reduce the number of tables watched by MetadataProxyHandler " [High,Fix released] - Assigned to Lucas Alvares Gomes (lucasagomes) | 14:51 |
| lucasagomes | yeah that's an optimization to the ovn metadata anget | 14:52 |
| lucasagomes | agent* | 14:52 |
| liuyulong | After reading this bug, I have an idea, if we can ovn-metedata-agent someday by using some distributed proxies locally in the compute node. | 14:52 |
| liuyulong | sorry, my idea is to replace, "if we can replace" | 14:53 |
| *** ociuhandu has joined #openstack-meeting | 14:53 | |
| liuyulong | let the ovn-controller do something instead of a python agent. | 14:54 |
| lucasagomes | it's something we could bring up with the OVN team itself, I've jsut heard of the idea so I don't know much about it atm | 14:56 |
| liuyulong | Np, we have a local topic which is trying to replace python neutron-metadata-agent with distributed proxies locally. | 14:57 |
| *** ociuhandu has quit IRC | 14:58 | |
| liuyulong | OK, time is up. | 14:58 |
| liuyulong | Thank you guys, bye | 14:58 |
| liuyulong | #endmeeting | 14:58 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:58 | |
| openstack | Meeting ended Wed Feb 12 14:58:56 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:58 |
| ralonsoh | bye | 14:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-02-12-13.58.html | 14:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-02-12-13.58.txt | 14:59 |
| maciejjozefczyk | bye! | 14:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-02-12-13.58.log.html | 14:59 |
| *** jamesmcarthur has joined #openstack-meeting | 15:00 | |
| *** masahito has quit IRC | 15:10 | |
| *** artom has joined #openstack-meeting | 15:11 | |
| *** priteau has joined #openstack-meeting | 15:15 | |
| *** lbragstad has joined #openstack-meeting | 15:15 | |
| *** jamesmcarthur has quit IRC | 15:16 | |
| *** lpetrut has quit IRC | 15:24 | |
| *** ociuhandu has joined #openstack-meeting | 15:25 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:31 | |
| *** rpittau is now known as rpittau|afk | 15:31 | |
| *** armax has joined #openstack-meeting | 15:45 | |
| *** ysandeep is now known as ysandeep|away | 15:50 | |
| *** jmasud has quit IRC | 15:50 | |
| *** jmasud has joined #openstack-meeting | 15:51 | |
| *** Trevor_V has joined #openstack-meeting | 15:55 | |
| *** TrevorV has quit IRC | 15:59 | |
| *** tesseract has joined #openstack-meeting | 16:02 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 16:03 | |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** maciejjozefczyk has quit IRC | 16:06 | |
| *** jmasud has quit IRC | 16:12 | |
| *** jmasud has joined #openstack-meeting | 16:13 | |
| *** jiaopengju2 has quit IRC | 16:19 | |
| *** jiaopengju2 has joined #openstack-meeting | 16:20 | |
| *** ykatabam has quit IRC | 16:20 | |
| *** priteau has quit IRC | 16:38 | |
| *** priteau has joined #openstack-meeting | 16:39 | |
| *** macz has joined #openstack-meeting | 16:43 | |
| *** macz has quit IRC | 16:43 | |
| *** macz has joined #openstack-meeting | 16:44 | |
| *** jiaopengju2 has quit IRC | 16:46 | |
| *** jiaopengju2 has joined #openstack-meeting | 16:46 | |
| *** lucasagomes has quit IRC | 16:47 | |
| *** macz has quit IRC | 16:50 | |
| *** priteau has quit IRC | 16:51 | |
| *** gyee has joined #openstack-meeting | 16:52 | |
| *** jiaopengju2 has quit IRC | 17:00 | |
| *** jiaopengju2 has joined #openstack-meeting | 17:00 | |
| *** igordc has joined #openstack-meeting | 17:17 | |
| *** tesseract has quit IRC | 17:32 | |
| *** ociuhandu_ has joined #openstack-meeting | 17:32 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** ociuhandu_ has quit IRC | 17:37 | |
| *** anastzhyr has quit IRC | 17:43 | |
| *** jmasud has quit IRC | 17:43 | |
| *** jamesmcarthur_ has quit IRC | 17:47 | |
| *** vishalmanchanda has quit IRC | 17:51 | |
| *** dougsz has quit IRC | 17:59 | |
| *** igordc has quit IRC | 18:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:18 | |
| *** eharney has quit IRC | 18:18 | |
| *** ijw has joined #openstack-meeting | 18:23 | |
| *** igordc has joined #openstack-meeting | 18:23 | |
| *** e0ne has quit IRC | 18:26 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:31 | |
| *** jamesmcarthur has quit IRC | 18:42 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:43 | |
| *** jamesmcarthur has quit IRC | 18:48 | |
| *** ijw has quit IRC | 18:53 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:54 | |
| *** jamesmcarthur has quit IRC | 18:57 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:58 | |
| *** yamamoto has quit IRC | 19:03 | |
| *** ralonsoh has quit IRC | 19:04 | |
| *** manuvakery has joined #openstack-meeting | 19:05 | |
| *** jamesmcarthur has quit IRC | 19:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:09 | |
| *** jamesmcarthur has quit IRC | 19:13 | |
| *** eharney has joined #openstack-meeting | 19:17 | |
| *** jamesmcarthur has joined #openstack-meeting | 19:30 | |
| *** anastzhyr has joined #openstack-meeting | 19:31 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 19:31 | |
| *** jamesmcarthur has quit IRC | 19:35 | |
| *** yamamoto has joined #openstack-meeting | 19:39 | |
| *** e0ne has joined #openstack-meeting | 19:40 | |
| *** e0ne has quit IRC | 19:51 | |
| *** yamamoto has quit IRC | 19:52 | |
| *** jamesmcarthur_ has quit IRC | 19:54 | |
| *** igordc has quit IRC | 19:58 | |
| *** LiangFang has quit IRC | 20:05 | |
| *** jmasud has joined #openstack-meeting | 20:37 | |
| *** armax has quit IRC | 20:45 | |
| *** armax has joined #openstack-meeting | 20:48 | |
| *** ChanServ has quit IRC | 20:50 | |
| *** zaitcev has joined #openstack-meeting | 20:50 | |
| *** artom has quit IRC | 21:12 | |
| *** ChanServ has joined #openstack-meeting | 21:13 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 21:13 | |
| *** rfolco has quit IRC | 21:18 | |
| *** rcernin has joined #openstack-meeting | 21:18 | |
| *** sluna has quit IRC | 21:23 | |
| *** ociuhandu has joined #openstack-meeting | 21:37 | |
| *** zaitcev has left #openstack-meeting | 21:44 | |
| *** ociuhandu has quit IRC | 21:48 | |
| *** ociuhandu has joined #openstack-meeting | 21:49 | |
| *** ociuhandu has quit IRC | 21:52 | |
| *** manuvakery has quit IRC | 21:55 | |
| *** ykatabam has joined #openstack-meeting | 21:57 | |
| *** raildo_ has joined #openstack-meeting | 22:05 | |
| *** raildo has quit IRC | 22:08 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:20 | |
| *** jamesmcarthur has quit IRC | 22:24 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:25 | |
| *** slaweq has quit IRC | 22:26 | |
| *** slaweq has joined #openstack-meeting | 22:38 | |
| *** slaweq has quit IRC | 22:42 | |
| *** jamesmcarthur has quit IRC | 23:02 | |
| *** Trevor_V has quit IRC | 23:04 | |
| *** rh-jelabarre has quit IRC | 23:04 | |
| *** Adri2000 has quit IRC | 23:19 | |
| *** jmasud has quit IRC | 23:22 | |
| *** yamamoto has joined #openstack-meeting | 23:49 | |
| *** yamamoto has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!