16:00:28 <slaweq> #startmeeting neutron_ci
16:00:28 <openstack> Meeting started Tue Apr 23 16:00:28 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:32 <slaweq> hi
16:00:33 <openstack> The meeting name has been set to 'neutron_ci'
16:00:36 <mlavalle> o/
16:00:38 <njohnston> o/
16:00:45 <bcafarel> \o
16:00:53 <haleyb> hi
16:01:28 <slaweq> please give me 2 minutes, I need to help my wife
16:02:01 <mlavalle> taking care of wife is the most important thing for your survival
16:03:54 * njohnston refrains from making a Betty Ross joke
16:03:58 <bcafarel> :)
16:04:08 <bcafarel> I can help by saying we can open http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1 in the meantime
16:04:14 <slaweq> I'm back
16:04:24 <slaweq> sorry :)
16:04:27 <slaweq> so lets start
16:04:29 <slaweq> #topic Actions from previous meetings
16:04:39 <slaweq> and thx bcafarel :)
16:04:46 <bcafarel> np :)
16:05:03 <slaweq> ok, so first action from last week
16:05:05 <slaweq> mlavalle to debug reasons of neutron-tempest-plugin-dvr-multinode-scenario failures
16:05:18 <mlavalle> I spent some time doing this
16:05:39 <mlavalle> dug in a lot of patches. Some conclusions:
16:06:54 <mlavalle> 1) in all the failures most tests pass. It is usually 1 to 3 tests that fail
16:07:15 <mlavalle> 2) Obviously most of the failures are ssh timeouts
16:07:29 <mlavalle> not all the time, but 95% of the time
16:08:30 <mlavalle> 3) The one test that fails most of the time is test_connectivity_through_2_routers
16:08:47 <mlavalle> this is a good example: http://logs.openstack.org/94/654394/2/check/neutron-tempest-plugin-dvr-multinode-scenario/9de8bce/testr_results.html.gz
16:09:46 <mlavalle> we added it in late last summer
16:10:02 <mlavalle> so I am digging deeper on this one
16:10:14 <mlavalle> that's all so far
16:10:21 <slaweq> yes, I added this test
16:10:40 <mlavalle> please keep me on the hook with this action item
16:10:45 <slaweq> in in Your example it failed on check connectivity from one vm to the other one
16:11:03 <mlavalle> yes, that's most of the time
16:11:48 <slaweq> for this one You can check if routers are properly configured in both routers
16:12:14 <mlavalle> good point. I'll continue from there
16:12:23 <slaweq> ok
16:12:32 <slaweq> thx mlavalle for working on it and for this update
16:12:46 <slaweq> #action mlavalle to continue debuging reasons of neutron-tempest-plugin-dvr-multinode-scenario failures
16:12:59 <slaweq> next one then
16:13:02 <slaweq> slaweq will send DNM patch to tempest to dump router's namespace state when ssh will fail
16:13:10 <slaweq> and I still didn't have time to get into it
16:13:25 <slaweq> but I will do it this week (tomorrow as first thing on my todo list)
16:13:27 <slaweq> sorry for that
16:13:46 <mlavalle> well, it was a difficult week with the Holidays
16:14:12 <slaweq> mlavalle: yep, I wasn't working on friday and monday so I didn't have a lot of time
16:14:20 <slaweq> and also some internal tasks todo :/
16:14:31 <bcafarel> chocolate does not boost productivity that much
16:14:37 <slaweq> but I will do it this week (before summit) for sure
16:14:39 <mlavalle> yeah, I got cought up last week in downstream stuff as well
16:14:52 <slaweq> bcafarel: LOL, I agree
16:14:58 <slaweq> ok, next one then
16:15:01 <slaweq> mlavalle to recheck tcpdump patch and analyze output from ci jobs
16:15:11 <slaweq> it's in fact related to the previous one
16:15:33 <mlavalle> I had to fix some of the code
16:15:57 <mlavalle> I made a newby mistake, passing a strin to asynch process
16:16:11 <mlavalle> the command has to be a list
16:16:23 <mlavalle> so I fixed that last night
16:16:41 <mlavalle> and now I am getting tcpdump output in the logs
16:17:09 <mlavalle> next step is to use that output to analyze some ssh failures
16:17:55 <mlavalle> and of course I was unlucky: dvr multinode got green in the last run
16:18:05 <slaweq> LOL
16:18:54 <slaweq> ok, so can I assign it to You for next week too?
16:19:01 <mlavalle> yes pelase
16:19:09 <mlavalle> keep on the hook for this one
16:19:10 <slaweq> #action mlavalle to recheck tcpdump patch and analyze output from ci jobs
16:19:12 <slaweq> thx
16:19:20 <slaweq> next one is
16:19:22 <slaweq> njohnston move wsgi jobs to check queue nonvoting
16:19:44 <njohnston> I did not get to that since I was on PTO last week; I'll try to get it going today
16:19:57 <slaweq> sure njohnston
16:20:04 <slaweq> #action njohnston move wsgi jobs to check queue nonvoting
16:20:47 <slaweq> that isn't urgent for sure but would be good to have some data before PTG as there is such topic in etherpad IIRC
16:20:56 <njohnston> definitely
16:21:23 <slaweq> if You will do it, please also add it to grafana dashboard, ok?
16:21:30 <njohnston> absolutely
16:22:09 <slaweq> thx
16:22:20 <slaweq> next one was
16:22:22 <slaweq> ralonsoh to debug issue with neutron_tempest_plugin.api.admin.test_network_segment_range test
16:22:41 <slaweq> as ralonsoh is on PTO today I will assign it to him for next week also
16:23:09 <slaweq> unless You know about some patch which fixes this problem already
16:23:43 <bcafarel> I don't think I have seen one pass by
16:24:06 <slaweq> #action ralonsoh to debug issue with neutron_tempest_plugin.api.admin.test_network_segment_range test
16:24:21 <slaweq> lets keep it here for next week and ask ralonsoh when he will come back
16:24:29 <slaweq> next one was
16:24:31 <mlavalle> ++
16:24:33 <njohnston> I haven't seen one either
16:24:34 <slaweq> slaweq to switch periodic jobs from py35 to py36
16:25:05 <slaweq> this was already done in https://review.opendev.org/#/c/654571/
16:25:26 <slaweq> so I only pushed today patch to update grafana: https://review.opendev.org/#/c/655178/
16:25:37 <slaweq> and we should be good in fact
16:26:08 <bcafarel> there are some pending reviews for stadium (AJaeger pointed to https://review.opendev.org/#/q/topic:periodic-py36+status:open)
16:26:10 <njohnston> +1
16:26:32 <slaweq> that was all related to actions from last week
16:26:34 <haleyb> slaweq: thanks for that, will need similar for ovn when it merges there
16:26:59 <slaweq> haleyb: yep, if ovn runs periodic jobs, we should add it there too
16:27:12 <slaweq> next topic then
16:27:14 <slaweq> #topic Stadium projects
16:27:24 <slaweq> (related to ovn) :)
16:28:01 <slaweq> according to bcafarel's proposal from last week, I changed topics a bit and now we will have only one topic related to stadium projects
16:28:42 <slaweq> we have 2 things related to stadium:
16:28:47 <slaweq> python 3 migration
16:28:53 <slaweq> etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status
16:28:58 <slaweq> njohnston: any updates?
16:29:28 <njohnston> Nope, but I am hoping to get a few of the conversions in before the PTG
16:30:14 <slaweq> anyone else have any updates here?
16:30:24 <mlavalle> not me
16:30:32 <bcafarel> not on this topic
16:30:47 <slaweq> ok, thx njohnston for taking care of this
16:30:54 <slaweq> lets move on to the next topic then
16:31:00 <slaweq> *subtopic
16:31:02 <slaweq> tempest-plugins migration
16:31:05 <bcafarel> :)
16:31:10 <slaweq> Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo
16:31:14 <slaweq> bcafarel: go on :)
16:31:40 <bcafarel> for sfc, 1st part patch looks good for reviews: https://review.opendev.org/#/c/653012
16:31:47 <bcafarel> (all moved in and tests passing)
16:32:13 <mlavalle> good jib bca
16:32:20 <mlavalle> bcafarel^^^
16:32:31 <slaweq> bcafarel++
16:32:32 <bcafarel> 2nd patch I do not see tempest tests running, Depends-On may not be enough? https://review.opendev.org/#/c/653747 if anyone has an idea :)
16:32:36 <njohnston> I had proposed that we use top-level directories in neutron-plugin-stadium; I got negative feedback on that from Sergey Vilgelm on my part 1 patch: https://review.opendev.org/#/c/643662/
16:34:16 <mlavalle> but that doesn't reflect the reality of Neutron stadium
16:34:39 <mlavalle> we have several projects contributing tests
16:34:50 <mlavalle> and I would like to keep visibility of that
16:34:53 <slaweq> we talked about it last week also and we decided to do it in separate top level modules
16:35:09 <mlavalle> yes, that's the way we want to go with this
16:35:10 <njohnston> that was my thought, but I wanted to air it here; sorry I missed the discussion last week
16:35:22 <njohnston> ok, I'll respond with a pointer to the logs of this discussion
16:35:24 <njohnston> thanks!
16:35:44 <slaweq> bcafarel: one thing I see in Your patch
16:35:54 <slaweq> I think You should add new job for stable branches also
16:36:21 <slaweq> as neutron-tempest-plugin is branchless we need job which will run agains stable branches of networking-sfc repo
16:36:46 <mlavalle> I also made a bit of progress last week with https://review.opendev.org/#/c/649373/
16:37:25 <bcafarel> slaweq: good point, else a new test could break stable branches in a stadium repo
16:37:39 <slaweq> bcafarel: yes, exactly
16:38:15 <mlavalle> I got a job running for master branch. All the APi tests pass, but the scenario ones are failling, so I must be missing something. any obvious suggestions?
16:39:42 <bcafarel> probably missing extensions, or tweaks done in the existing job (is it zuulv3 or legacy?)
16:40:09 <mlavalle> yeah, I'll compare with what they have in vpnaas
16:41:10 <bcafarel> I was scared of similar issue with sfc (which used a legacy playbook), but converting to zuulv3 was easy
16:41:36 <slaweq> yes, probably missing some extension/driver/config option
16:41:38 <mlavalle> vpnaas actually uses zuulv3 for this job
16:42:42 <slaweq> regarding to https://review.opendev.org/#/c/652991/ I didn't make big progress
16:42:59 <slaweq> job for new tests is failing due to devstack error
16:43:16 <slaweq> it says that bagpipe-bgp did not start and I will have to check why
16:43:52 <slaweq> I now see such error http://logs.openstack.org/91/652991/2/check/neutron-tempest-plugin-bgpvpn-bagpipe/ea3dd4f/controller/logs/screen-neutron-bagpipe-bgp.txt.gz
16:44:09 <slaweq> but I have no idea about bagpipe-bgp process
16:44:23 <slaweq> maybe I will be able to catch tmorin somehow to get some help on this
16:44:40 <bcafarel> I saw him comment on some recent bgpvpn reviews, so he may be around these days
16:44:47 * bcafarel has to run, sorry
16:45:10 <slaweq> I will try to catch him on irc this week
16:45:34 <slaweq> maybe it's just some python3 - python2.7 issue
16:46:03 <slaweq> that's all from my side regarding stadium projects
16:46:12 <slaweq> anything else You want to add?
16:47:05 <slaweq> ok, I take it as no
16:47:09 <slaweq> so let's move on
16:47:11 <slaweq> #topic Grafana
16:47:17 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:47:50 <slaweq> one thing worth to mention is that due to last migration to opendev.org we don't have old data in grafana
16:48:30 <slaweq> in https://review.opendev.org/#/c/654026/ all jobs were changed to be "opendev_org" jobs
16:48:54 <slaweq> thus we don't have now old data for jobs with git_openstack_org in names
16:49:16 <njohnston> Is that related to why we don't have any datapoints for some graphs, like functional/fullstack in gate?
16:49:25 <slaweq> njohnston: yes
16:49:33 <slaweq> I think so
16:49:40 <mlavalle> do we need to take any action?
16:50:03 <slaweq> mlavalle: no, new data should be filled in now
16:50:08 <mlavalle> cool
16:50:13 <slaweq> so next week we should have more data I hope
16:50:25 <mlavalle> other than that, I don't see anything worrisome
16:50:37 <slaweq> me neighter
16:50:52 <haleyb> patch merged 4/21 so that's day zero now for stats
16:51:19 <slaweq> haleyb: yep, kind of "fresh start" for Train :)
16:52:11 <slaweq> anyone wants to add/ask something relatd to grafana?
16:52:16 <slaweq> *related
16:52:18 <mlavalle> not me
16:52:42 <slaweq> ok
16:52:53 <slaweq> so that's all from me for this week
16:53:09 <mlavalle> I don't have anything to add
16:53:14 <slaweq> I was today checking results of some failed jobs from last couple of days
16:53:24 <mlavalle> other than I am looking forward to see you next week
16:53:26 <slaweq> but I didn't found anything new/interesting there
16:53:35 <slaweq> mlavalle: yes, me too
16:53:37 <slaweq> :)
16:53:39 <mlavalle> and survving the sharks
16:53:46 <slaweq> so one last thing for today:
16:53:57 <slaweq> I will cancel next week's meeting
16:54:02 <slaweq> are You ok with it?
16:54:05 <njohnston> +1
16:54:08 <mlavalle> yes
16:54:15 <slaweq> ok :)
16:54:29 <mlavalle> if that's ok with you, we can do I quick grafana check on Tuesday
16:54:39 <slaweq> #action slaweq to cancel next week meeting
16:54:50 <mlavalle> just to make sure nothing is burning
16:54:51 <slaweq> mlavalle: sure, we can do it in person even :)
16:55:00 <mlavalle> yes, you and I can do it together
16:55:04 <slaweq> sure
16:55:15 <slaweq> ok, thanks for attending
16:55:27 <slaweq> safe traveling to Denver
16:55:38 <slaweq> and safe diving with sharks mlavalle :)
16:55:45 <slaweq> see You
16:55:47 <mlavalle> Thanks
16:55:49 <slaweq> #endmeeting