16:00:30 <slaweq_> #startmeeting neutron_ci
16:00:31 <openstack> Meeting started Tue Apr 17 16:00:30 2018 UTC and is due to finish in 60 minutes.  The chair is slaweq_. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:34 <openstack> The meeting name has been set to 'neutron_ci'
16:00:38 <slaweq_> hi
16:01:06 <mlavalle> o/
16:01:31 <jlibosva> o/
16:01:43 <slaweq_> jlibosva: haleyb: are You there? :)
16:01:51 <jlibosva> I am here
16:01:53 <ihar> o/
16:01:56 <haleyb> hi, but i need to run home so will miss :20
16:02:03 <slaweq_> yes :)
16:02:11 <slaweq_> ihar: new nick?
16:02:16 <haleyb> assign me all the bugs :)
16:02:22 <slaweq_> haleyb: sure
16:02:37 <jlibosva> all the bugs, not just the Neutron ones
16:02:56 <slaweq_> ok, let's start
16:03:01 <slaweq_> #topic Actions from previous meetings
16:03:08 <slaweq_> haleyb to continue testing why router migrations tests fails
16:04:04 <slaweq_> haleyb: any update on this one?
16:05:09 <slaweq_> ok, so I guess not, let's move on then
16:05:16 <slaweq_> next one was
16:05:17 <slaweq_> haleyb to continue testing why router migrations tests fails
16:05:21 <slaweq_> undo
16:05:29 <slaweq_> slaweq will check old gate-failure bugs
16:05:46 <slaweq_> so, again I didn't have time to go through this list yet
16:06:03 <slaweq_> but I will try to do it this week
16:06:40 <slaweq_> #action slaweq will check old gate-failure bugs
16:06:50 <slaweq_> next one
16:06:52 <slaweq_> yamahata to fix issues with openstack-tox-py35-with-neutron-lib-master periodic job
16:07:30 <yamahata> I think it's fixed and it can be confirmed health check
16:07:48 <slaweq_> yes, I saw that periodic jobs are passing now
16:07:51 <slaweq_> thx a lot for help
16:08:15 <slaweq_> ok, next one
16:08:16 <slaweq_> haleyb will mark router migration tests are unstable
16:08:29 <slaweq_> I think it was done already
16:08:39 <mlavalle> I +2ed that patch yesterday
16:08:58 <slaweq_> https://review.openstack.org/#/c/561322/
16:08:58 <patchbot> patch 561322 - neutron-tempest-plugin - Mark DVR/HA migration tests unstable (MERGED)
16:09:04 <slaweq_> it is this one probably
16:09:11 <slaweq_> so it's done
16:09:17 <mlavalle> yes
16:09:27 <slaweq_> and last but not least
16:09:37 <slaweq_> agreed after meeting with mlavalle :)
16:09:38 <slaweq_> mlavalle to makes ovsfw scenario job voting
16:09:52 <mlavalle> that was taken care of by jlibosva
16:09:54 <slaweq_> I think that jlibosva did it and it's now voting
16:10:00 <jlibosva> yep it is voting now
16:10:07 <slaweq_> thx jlibosva
16:10:17 <mlavalle> jlibosva: I told you, he was going to come after me ;-)
16:10:21 <slaweq_> today I also pushed https://review.openstack.org/#/c/561930/ related to this one
16:10:22 <patchbot> patch 561930 - openstack-infra/project-config - Change label for neutron-tempest-ovsfw to "voting"
16:10:28 <jlibosva> hehe :)
16:10:33 <slaweq_> to change job name in grafana
16:11:10 <slaweq_> ok, so that's all about actions from previous week
16:11:22 <slaweq_> #topic Grafana
16:11:28 <slaweq_> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:12:08 <slaweq_> I checked today graphs from last 7 days and it looks quite good
16:12:35 <jlibosva> I like how dvr-multinode goes down after router migration tests are skipped :)
16:12:49 <slaweq_> except problems with pip 10 and lower-constraints which caused (almost) all tests failing all the time for last few days
16:13:08 <slaweq_> but those problems should be already fixed and jobs are getting better today again
16:13:31 <slaweq_> jlibosva: yes, dvr-multinode job is much better now :)
16:13:32 <ihar> why gate tempest jobs so bad? is it because not many patches merged?
16:14:21 <slaweq_> ihar: might be - please not that in last 3 days nothing was merged probably
16:14:35 <slaweq_> because of this issue with lower-constraints job and ryu package
16:14:55 <jlibosva> yeah, we had one patch merged on 14th and then 2 patches merged today
16:15:04 <jlibosva> and that's it
16:15:12 <ihar> yeah. that's what I am saying, maybe that's because the number of data points is so low so average was skewed by an outlier.
16:15:18 <ihar> ok
16:15:48 <slaweq_> ok then :)
16:16:12 <slaweq_> do You have anything to add?
16:16:35 <slaweq_> or we can go to talk about some specific job types?
16:16:57 <jlibosva> looks like we're good here
16:17:15 <slaweq_> ok then
16:17:20 <slaweq_> #topic Functional
16:17:59 <slaweq_> today I saw failure like http://logs.openstack.org/67/556667/20/check/neutron-functional/71b2acc/logs/testr_results.html.gz in functional tests
16:18:14 <jlibosva> bollocks
16:18:17 <slaweq_> and I think that I saw same issue at least one more time last week
16:18:37 <slaweq_> maybe it's not big problem yet but I wanted to mention about it
16:18:39 <jlibosva> that might mean the firewall blink is not fixed
16:19:17 <jlibosva> I'll have a look
16:19:29 <slaweq_> or at least there is some corner case when it is failing
16:19:35 <slaweq_> thx jlibosva
16:19:55 <slaweq_> #action jlibosva take a look on failing ovsfw blink functional test
16:20:41 <slaweq_> ok, next topic
16:20:46 <slaweq_> #topic Fullstack
16:20:59 <slaweq_> nothing very urgent on my side for fullstack currently
16:21:30 <slaweq_> but I found today one failed security groups test  http://logs.openstack.org/12/470912/38/check/neutron-fullstack/ba16b38/logs/testr_results.html.gz so I will take a look on it during the week
16:21:52 <slaweq_> #action slaweq will check failed SG fullstack test
16:22:03 <slaweq_> do You have anything to add according to fullstack?
16:23:10 <ihar> no
16:23:36 <slaweq_> so next topic
16:23:38 <slaweq_> #topic Scenarios
16:24:00 <slaweq_> as jlibosva told already dvr-multinode job is getting better with skipped migration tests :)
16:24:23 <slaweq_> let's see this week what failure rate it will have without those tests
16:24:49 <slaweq_> other scenario jobs looks quite good now IMO
16:24:53 <mlavalle> ++
16:24:55 * ihar thrilled
16:24:56 <jlibosva> we still have the trunk failing occasionally that I'm looking at. I deployed multinode dvr environment and I'm not able to reproduce the issue locally. I ran the tests probably around 100 times
16:25:52 <slaweq_> I know that it wasn't so common failure reason for this job so it might be hard to reproduce
16:26:40 <slaweq_> jlibosva: it might be hard to reproduce it locally :/
16:27:05 <jlibosva> it never failed for me
16:28:09 <slaweq_> did You checked logs from such failed job? Do You have any idea what could be the problem there?
16:29:08 <jlibosva> I suspect wrong order of rpc messages about remote security group and prepare_port_filter. the logs contain error where vlan tag can't be found in ovsdb for given subport/trunk
16:29:31 <jlibosva> that means ovs firewall cannot get correct zone to be used in conntrack as the zone number corresponds with the local vlan tag of given port
16:29:38 <jlibosva> so then all traffic is dropped
16:29:46 <jlibosva> which would explain the SSH connection issue
16:30:13 <jlibosva> but I can't find a reason why the vlan tag is not present in the ovsdb
16:31:38 <slaweq_> maybe You should add some additional debug logs in agent and wait for new failure to check it?
16:31:45 <jlibosva> it also seems that the scenario job doesn't have indexed console output in logstash
16:32:18 <jlibosva> yeah, I plan to log snapshot of openflows per ovs firewall action, so we can see the state of openflows at the time of failure
16:32:24 <slaweq_> yeah, I was talking about some time ago but when I didn't need it anymore I forgot about it :/
16:33:06 <slaweq_> ok, so You will continue work or this, right?
16:33:13 <ihar> jlibosva: why isn't it in logstash?
16:33:30 <jlibosva> ihar: I do not know :)
16:33:48 <ihar> you mean console.html not indexed?
16:33:54 <ihar> or whatever it is named now
16:33:59 <jlibosva> it's now the job-output.txt or something like that
16:34:00 <jlibosva> but yeah, that one
16:34:12 <ihar> weird. I can take a look at that one
16:34:20 <slaweq_> it's job-output.txt.gz
16:34:28 <jlibosva> ihar: first step would be to make sure I'm not lying :)
16:34:41 <ihar> yeah sure :)
16:34:53 <ihar> I will keep you honest and shame in public if you aren't!
16:35:04 <jlibosva> thanks
16:35:07 <ihar> :))
16:35:16 <ihar> slaweq_: add an action
16:35:55 <slaweq_> #action jlibosva will check if job output are indexed in logstash
16:36:02 <jlibosva> slaweq_: ihar :)
16:36:04 <slaweq_> ihar: here You go :)
16:36:19 <jlibosva> slaweq_: ihar will check
16:36:28 <ihar> yes
16:36:29 <slaweq_> ah, sorry
16:36:35 <jlibosva> I mean, I just checked and I still cannot see it
16:36:44 <slaweq_> #action ihar will check if job output are indexed in logstash
16:36:54 <slaweq_> better?
16:36:56 <slaweq_> :)
16:37:12 <ihar> yes sir
16:37:20 <slaweq_> ok :)
16:38:21 <slaweq_> so, moving on?
16:38:26 <mlavalle> yes
16:38:29 <slaweq_> #topic Rally
16:38:49 <slaweq_> as from grafana it looks that rally is fine now
16:39:14 <slaweq_> so I think that we don't need to talk about it too much :)
16:39:25 <slaweq_> at least You have something to add here
16:39:59 <ihar> there was an email from rally folks about openstack plugin being spun off rally. shouldn't affect us but worth being aware.
16:40:20 <ihar> apparently rally is more than openstack :)
16:40:54 <slaweq_> thx ihar
16:41:02 <mlavalle> good to know
16:42:18 <slaweq_> ok, so can we move to next topic then?
16:42:36 <mlavalle> I'd say so
16:42:37 <ihar> YES
16:42:41 <slaweq_> #topic Periodic
16:42:55 <slaweq_> here I just wanted to mention that it looks fine currently
16:43:11 <slaweq_> looks that yamahata's fix works fine :)
16:43:40 <slaweq_> thx once again yamahata
16:43:50 <yamahata> :)
16:44:54 <slaweq_> and the last topic is
16:44:55 <slaweq_> #topic Gate
16:45:23 <slaweq_> here I also don't have anything to talk for today - except this tempest job which ihar mentioned before it looks good IMO
16:45:50 <slaweq_> do You want to talk anything here?
16:46:06 <slaweq_> or do You anything else to talk about today?
16:46:37 <ihar> I don't. I think if the chair believes there is nothing to cover we can as well skip sections and leave the rest for open discussion
16:47:20 <slaweq_> #topic Open Discussion
16:47:50 <slaweq_> so do You want to talk about something else related to CI?
16:47:57 <jlibosva> I don't have anything
16:48:21 <ihar> I have nothing
16:48:36 <slaweq_> mlavalle: ?
16:48:39 <mlavalle> nope
16:48:49 <slaweq_> ok, so thank You
16:48:56 <jlibosva> 11 minutes back, yay :)
16:48:59 <slaweq_> and enjoy Your free 11 minutes :)
16:49:00 <slaweq_> #endmeeting