22:00:05 <mlavalle> #startmeeting neutron_drivers
22:00:06 <openstack> Meeting started Thu Oct 19 22:00:05 2017 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:09 <openstack> The meeting name has been set to 'neutron_drivers'
22:00:16 <yamamoto> morning
22:00:56 <mlavalle> good Friday morning Yamamoto, from Thursday evening :-)
22:01:33 <mlavalle> armax said that he is not going to make it today
22:02:13 <mlavalle> amotoki, ihrachys: ping
22:03:20 * mlavalle checking if got email from ihrachys or amotoki
22:03:48 <mlavalle> nope
22:03:59 <mlavalle> let's give them a couple of minutes
22:05:51 <mlavalle> yamamoto: I went through all the triaged RFEs and saw that you left comments in a lot of them
22:06:18 <mlavalle> I commented on the one about adding segment_id to subnets
22:10:27 <mlavalle> yamamoto: I wanted to start the meeting today with the last RFE: https://bugs.launchpad.net/neutron/+bug/1718605
22:10:29 <openstack> Launchpad bug 1718605 in neutron "[RFE] Support regex matching when filtering port with IP address" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu)
22:11:00 <mlavalle> The reason is that we have a volunteer who wants to do the work in Neutron and Nova
22:11:29 <mlavalle> he pinged me a couple of days ago, because Nova's feature freeze is today
22:12:11 <mlavalle> so he wants to know if he can move ahead. On the Nova side, mriedem supports this RFE
22:12:15 <yamamoto> i haven't read it yet.  i was assuming the list is ordered. :-)
22:12:43 <mlavalle> yamamoto: it is, I am making an exception given the Nova feature freeze
22:13:00 <mlavalle> would you take a look and tell me what you think?
22:13:13 <mriedem> nova has already approved the nova spec
22:13:27 <mriedem> it's assumed the neutron RFE will be approved - if not, we go back to the drawing board
22:13:48 <mriedem> the neutron RFE might be useful regardless of the nova need for it
22:14:14 <mlavalle> yeah, I agree and armax indicated he supports it
22:15:43 <yamamoto> i suspect SQL LIKE and python regex are incompatible
22:16:05 <mriedem> the reason nova can't do a regex filter on IPs,
22:16:11 <mriedem> is because nova stores those in a serialized json blob in the db
22:16:23 <mriedem> so we have to pull the instances and then filter via regex in python,
22:16:29 <mriedem> once we have deserialized the json
22:16:35 <mriedem> hence the performance issue
22:16:48 <mriedem> so depending on how IP values are stored in the neutron db, you could use LIKE
22:17:34 <mriedem> but yeah things like ^192.[foobars]+$ probably not :)
22:17:57 <mriedem> i'd have to see what nova actually allows for the filter
22:18:50 <mriedem> https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L51
22:18:55 <mriedem> i guess we allow a regex
22:21:19 <yamamoto> so, the rfe is about 1. allow the same set of regex as nova and 2. optimize the common case using SQL LIKE, fallback to python regex for complex cases?
22:21:41 <mriedem> that's unclear
22:22:25 <mriedem> so probably requires some thought
22:22:38 <mriedem> i thought we did a substring match,
22:22:41 <mriedem> not a full regex compare
22:23:04 <mriedem> this is where the nova code happens https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2318
22:23:43 <mriedem> and apply it https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2468
22:23:56 <yamamoto> i've always considered ip addresses stored as strings a bug in neutron. it's surprising it can be useful. :-)
22:24:17 <mriedem> the compute API reference doesn't say anything about valid forms of the regex
22:24:20 <mriedem> it just says, "An IPv4 address to filter results by."
22:24:33 <mriedem> we take what's given and shove into ^%s$
22:24:55 <mriedem> so if you literally passed ^192.168.*$ it would turn out wrong
22:25:07 <mriedem> so i think it's really just a subset
22:25:47 <mriedem> note we allow filtering on both ipv4 and ipv6
22:25:53 <mlavalle> yeah, it doesn't seem that it would work
22:27:03 <mriedem> so,
22:27:13 <mriedem> we could table it for now and ask operators in the ML how they use the ip filters
22:27:21 <mriedem> and figure out limitations
22:27:21 <mlavalle> the other thing is that it seems it is done with data in memory
22:27:24 <mlavalle> right?
22:27:42 <mriedem> yes, as noted, we pull the instance records out of the database and then have to deserialize the json blob that contains the ips
22:27:43 <mlavalle> network_info
22:27:51 <mriedem> then we apply the python regex match on those
22:27:57 <mriedem> which is the performance issue
22:28:01 <mlavalle> right
22:28:04 <mriedem> because we can't filter in sql
22:29:05 <mlavalle> ok, would Zhenyu follow up in the ML?
22:29:17 <mriedem> if you tell him to sure :)
22:29:27 <mriedem> i assume you guys reply in the RFE bug report with next steps?
22:29:37 <mlavalle> LOL, I'll update the RFE
22:29:45 <mriedem> heh ok
22:29:50 <mriedem> don't let me define your process :)
22:30:02 <mlavalle> no, that is exactly the process
22:30:08 <yamamoto> is the nova functionality documented in eg. api-ref?
22:30:53 <mlavalle> mriedem: ^^^^
22:31:21 <mriedem> it just says, "An IPv4 address to filter results by."
22:31:26 <mriedem> and IPv6
22:31:48 <mlavalle> well, maybe some testing will be in order as well
22:31:49 <mriedem> https://developer.openstack.org/api-ref/compute/#list-servers
22:32:07 <mriedem> i think tempest might have a test for this,
22:32:12 <mriedem> i know we have a functional test internally
22:33:59 <mriedem> https://github.com/openstack/nova/blob/master/nova/tests/functional/test_list_servers_ip_filter.py
22:35:19 <yamamoto> it tests only exact matches right?
22:35:48 <mriedem> no
22:36:00 <mriedem> 2 servers
22:36:04 <mriedem> one with 10.1.1.1 and one with 10.1.1.10
22:36:08 <mriedem> tests that filtering by 10.1.1.1 returns both
22:36:14 <mriedem> and filtering by 10.1.1.10 returns only 1
22:36:16 <mlavalle> eactly
22:36:21 <mriedem> it's not complicated by any means
22:36:27 <mlavalle> it's testing a substring
22:36:39 <mriedem> and it's not passing "10.1[1234]+"
22:36:45 <mlavalle> correct
22:36:53 <mlavalle> not really regex
22:36:56 <mriedem> but, i don't know if operators rely on passing a complicated regex
22:37:02 <mriedem> i'd hope not
22:37:18 <mriedem> i suppose you could pass a regex to do OR type filtering on IPs
22:37:26 <mriedem> or maybe ranges of IPs?
22:37:41 <mlavalle> yeah, I think that for the time being we can say we have no evidence that regex is possible
22:37:53 <mlavalle> but operators might by doing it
22:38:02 <mriedem> on option is,
22:38:06 <mriedem> and i hate to say this,
22:38:12 <mriedem> but neutron implements a strict substring match,
22:38:25 <mriedem> and nova-api has a config option that either makes nova use neutron or the existing thing we have
22:38:42 <mriedem> so if you really depend on wacky regex filtering, you don't do the faster neutron thing
22:38:53 <mriedem> i don't like config-driven api but it's an option,
22:38:59 <mriedem> i'd like to see what operators think about it
22:39:10 <mlavalle> cool
22:39:37 <mriedem> ultimately it comes down to you guys tell us what you're willing to support
22:39:39 <mriedem> and we'll figure out the rest
22:40:40 <yamamoto> why need a config?  nova-api can tell if the given filter is just a subtring match or not and fallback to the old method i guess.
22:41:00 <mriedem> i was thinking about that,
22:41:09 <mriedem> but wasn't sure how possible it is to determine if something is a regex or not
22:41:19 <mriedem> like, is 10.1 a valid regex?
22:41:44 <mriedem> especially when you have ipv6 filters
22:41:47 <mriedem> with brackets
22:42:50 <yamamoto> i thought nova escaped "." in the code you mentioned above
22:43:10 <mriedem> yeah it does here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2318
22:43:47 <yamamoto> so 10.1 is just a substring match right?
22:44:34 <mriedem> yeah but how do we determine what isn't a substring?
22:45:19 <yamamoto> i agree it might not be trivial in general.
22:45:37 <mriedem> if the re module has something that tells us programmatically what the special characters are maybe that would work
22:48:25 <mriedem> maybe we use re.match to check the filter string :)
22:48:38 <mriedem> should only contain numbers, dots and brackets
22:48:50 <mriedem> so if * and - are in there it's a regex
22:48:54 <mriedem> just an idea
22:49:28 <mlavalle> but we still need feedback from the operators right?
22:49:36 <mriedem> i think that would be useful
22:49:44 <mriedem> as far as, here is the problem,
22:49:46 <yamamoto> does ip6 filter use the same _remap_fixed_ip_filter?
22:49:47 <mriedem> here are some options,
22:49:49 <mriedem> what do you think
22:50:11 <mriedem> yamamoto: looking
22:50:43 <mlavalle> yeah, I agree in seeking feedback
22:52:30 <mriedem> i'm not even sure if https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2332 is used anymore
22:52:39 <mriedem> i'd have to dig into that
22:52:54 <mriedem> this is what we key on https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2362
22:53:06 <mriedem> https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2468
22:53:13 <mriedem> and we just pull the straight filter value
22:53:15 <mriedem> and compile it
22:53:36 <yamamoto> yea, i was wondering why it was a substring match while it shoves into "^%s$"
22:53:54 <mriedem> i think that is some really old code
22:55:55 <mriedem> so with 5 minutes left on your meeting,
22:56:04 <mriedem> i assume next steps are updating the RFE bug,
22:56:08 <mriedem> asking about the regex problem,
22:56:19 <mriedem> and how that's going to work with LIKE in a SQL query in neutron
22:56:27 <mriedem> and probably bring this up in the ops ML
22:56:48 <mlavalle> yeah
22:57:21 <mlavalle> ok, I will update the RFE
22:57:49 <mlavalle> yamamoto: despite not having quorum, we made progress :-)
22:58:07 <mlavalle> some anyway
22:59:02 <yamamoto> mriedem: have you (or someone) confirmed LIKE query is that fast to make this worth to tackle?
22:59:41 <mriedem> yamamoto: Kevin_Zheng should be POCing this and testing the difference at scale
22:59:43 <mriedem> with 1000 instances
22:59:56 <yamamoto> i'm not sure if we have a proper index to make it fast right now.
23:00:10 <mriedem> maybe that would need to be considered on the neutron side,
23:00:27 <mriedem> for the nova side, i said we'll need to gauge this based on the poc and benchmark perf results
23:00:38 <mriedem> i think neutron should also do that before merging code
23:00:56 <yamamoto> iirc postgres uses btree index for like queries but i'm not sure how other backends do.
23:01:01 <yamamoto> i agree
23:02:44 <mlavalle> yamamoto, mriedem: thanks for a lovely Thursday evening
23:02:50 <mlavalle> #endmeeting