00:01:42 <gmann> #startmeeting qa 00:01:43 <openstack> Meeting started Thu Sep 12 00:01:42 2019 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:01:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 00:01:47 <openstack> The meeting name has been set to 'qa' 00:02:02 <gmann> who all here today? 00:02:05 <masayukig> hi 00:02:11 <bigdogstl> Hello 00:03:23 <gmann> this is our first office hour in new time. 00:03:27 <gmann> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours 00:03:31 <gmann> agenda ^^ 00:03:53 <gmann> #topic Announcement and Action Item 00:04:52 <gmann> nothing to announce here except feature freeze is coming in this week(tomorrow) and we need to keep monitoring on gate and requirement freeze things. 00:04:53 <bigdogstl> AT&T is now using tempest as a deploy gate on all of our deployments 00:05:13 <gmann> bigdogstl: perfect, that is good news 00:05:13 <bigdogstl> in downstream CI thank you to all for the hardwork. 00:05:20 <masayukig> bigdogstl: cool 00:05:35 <bigdogstl> we are running over 100 test every deployment with no leaks 00:05:51 <bigdogstl> we deploy at least 5 time a day 00:06:05 <bigdogstl> that is why you are see high activity from new people 00:06:07 <gmann> nice, how you use Tempest. master or release tagged version ? 00:06:11 <bigdogstl> 1000 not 100 00:06:30 <bigdogstl> master against ocata with some code patches 00:06:39 <bigdogstl> 100% containerized 00:06:41 <gmann> +1 thanks for adding new intern. 00:07:04 <masayukig> it's a very good usecase 00:07:05 <gmann> code patch for internal stuff or upstream bugs ? 00:07:08 <bigdogstl> they have gone back but I have full time engineers working upstream now 00:07:25 <bigdogstl> miss match of ocata and master 00:07:36 <gmann> ohk 00:07:47 <bigdogstl> we blacklist test that do not work in house 00:07:50 <bigdogstl> not many 00:08:03 <gmann> i see. 00:08:20 <gmann> let's move 00:08:23 <bigdogstl> we are trying to get patrole working but we are seeing to many leaks for it to be a offical gate yet 00:08:29 <bigdogstl> sure 00:08:47 <gmann> ok. let's discuss that in detail on later topic 00:08:53 <gmann> #topic Train Priority Items progress 00:09:05 <gmann> #link https://etherpad.openstack.org/p/qa-train-priority 00:09:33 <gmann> we will quickly go through the items which has updates to share 00:09:38 <gmann> How to make tempest-full stable - gmann 00:10:17 <gmann> only glance patch is left which i have replied the comment and pinged glance PTL to check. 00:10:32 <gmann> OpenStack-health improvement - masayukig 00:10:37 <gmann> masayukig: anything to share on this ^^ 00:11:11 <masayukig> heh, nope actually.. 00:11:11 <gmann> or bigdogstl 00:11:29 <gmann> ok 00:11:31 <gmann> Planning for Patrole Stable release - gmann 00:11:48 <bigdogstl> no progress at this time 00:11:51 <gmann> bigdogstl has list of patches for resource leake thngs 00:12:11 <bigdogstl> https://review.opendev.org/#/q/owner:rick.bartra%2540att.com+status:open+project:+openstack/patrole 00:12:27 <gmann> bigdogstl: one question on patrole use- how many role you run ? 00:12:40 <bigdogstl> 10 at this time 00:12:52 <gmann> admin and member or other also 00:13:22 <bigdogstl> yes 00:13:39 <bigdogstl> we have 3 admins 00:14:06 <bigdogstl> full, support read only 00:14:32 <bigdogstl> and more for type of roles 00:14:35 <gmann> ok 00:14:49 <bigdogstl> we containerize the test suite and 00:15:23 <bigdogstl> run them in parallel 00:15:34 <gmann> ok 00:15:41 <bigdogstl> in test labs and green field prod and serial in brown field prod 00:16:13 <bigdogstl> once we have all of the leaks done we will start address missing test in our deployments 00:17:04 <bigdogstl> again we are using master against ocata 00:17:06 <gmann> bigdogstl: ok, I will get those review done by this week before my flight back to CT TZ 00:17:19 <bigdogstl> but we are moving to stein this soon 00:17:28 <gmann> perfect, that was my question :) 00:17:30 <bigdogstl> thank you 00:17:36 <gmann> one thing i was thinking to add patrole job in Tempest gate as many of service client are being used by patrole only and it can break patrole gate. one case was volume schema change. 00:17:58 <bigdogstl> that will be nice 00:18:00 <gmann> may be as admin and member role which should cover most of the cases 00:18:26 <bigdogstl> we are finding other plugin doing or wanting to do patrole like neutron and taas 00:18:34 <gmann> #action gmann to add patrole job to Tempest gate 00:18:38 <bigdogstl> I agree 00:18:45 <masayukig> +1 00:19:07 <gmann> bigdogstl: yeah we need to discuss how we extend the patrole for other services 00:19:24 <gmann> it can be good discussion for coming PTG. 00:19:29 <bigdogstl> +1 00:19:35 <gmann> bigdogstl: are you planning to come to Shanghai ? 00:19:53 <bigdogstl> I will not be there in person but will try to be on the etherpad 00:20:23 <bigdogstl> I wish but cost is too high for everyone 00:20:25 <gmann> ok. i will make sure to get you and your team into the discussion. 00:20:36 <bigdogstl> thank you 00:20:36 <gmann> next item: 00:20:38 <gmann> Keystone system-scope testing - gmann 00:20:56 <gmann> no progress on this. I will take this up after feature freeze and IPv6 goal things. 00:21:12 <gmann> Document the QA process or todo things for releases, stable branch cut etc - gmann 00:21:18 <gmann> ditto 00:21:27 <gmann> new whitebox plugin for tempest - artom 00:21:56 <bigdogstl> I have look at it but not much activity 00:22:00 <gmann> we talked about it few days back and he will finish the few pending code things first and then push the spec 00:22:07 <gmann> yeah 00:22:17 <gmann> Tempest Plugin sanity check among cross tempest plugins - masayukig 00:22:21 <gmann> masayukig: ^^ 00:22:30 <bigdogstl> we are still holding our expected values downstreaam 00:22:46 <bigdogstl> but it is growing and will talk to them to see if it is a fix later 00:23:05 <masayukig> The sanity check job is already voting now. And I think it's stable. 00:23:13 <bigdogstl> I think we need a gate process for plugins 00:23:39 <bigdogstl> so this gate an issue in the future 00:23:53 <bigdogstl> the x/ranger team worked on this 00:24:09 <bigdogstl> we hope they get back tot he commit comments soon 00:24:22 <bigdogstl> I think we need a vision here 00:24:33 <gmann> masayukig: nice thanks. 00:24:46 <bigdogstl> right now we have cookie cuter smaple but not sure if we can use tempest sheel script to drive it 00:24:57 <bigdogstl> shell 00:25:27 <gmann> you mean to create tempest plugin ? 00:25:56 <bigdogstl> let me get the ps 00:26:18 <gmann> ok 00:26:30 <bigdogstl> no a tox.ini in the plugin as a gate 00:26:54 <bigdogstl> here is the doc change https://review.opendev.org/#/c/674121/ 00:27:21 <bigdogstl> please review and add comments 00:27:37 <bigdogstl> Felipe was trrying to get it as a tool over docs 00:27:51 <gmann> bigdogstl: i did not notice that. that is good initiative. 00:27:55 <bigdogstl> may need input for direction on that 00:28:00 <gmann> i will check that 00:28:05 <gmann> sure 00:28:27 <gmann> bigdogstl: can you add that link under sanity job item in etherpad 00:28:36 <gmann> Data plane tester - 00:28:36 <bigdogstl> will do 00:28:43 <gmann> bigdogstl ^^ 00:28:45 <artom> gmann, btw, does Tempest have the same spec freeze/feature freeze thing as Nova? 00:28:51 <artom> Or can I propose the spec anytime? 00:29:57 <gmann> artom: no, anytime is ok. 00:30:11 <artom> gmann, aha, good to know 00:30:26 <artom> My NUMA live migration thing will land in time for Nova FF 00:30:32 <artom> So I have time to think about other things :) 00:30:40 <bigdogstl> we are just using shaker wight now and are starting small changes to do harbinger 00:30:55 <gmann> artom: thanks, that is same way i plan things for Tempest. 00:31:11 <gmann> bigdogstl: ok. ping us once spec is ready 00:31:15 <bigdogstl> we are allo doing long live test and signal to stop for resiliency 00:31:41 <bigdogstl> run east west and north south while playing with control plane node 00:31:59 <bigdogstl> hope to get spec started soon 00:32:06 <gmann> ok, that is really good to add in upstream. thx 00:32:12 <gmann> let's move next 00:32:15 <gmann> #topic OpenStack Events Updates and Planning 00:32:25 <gmann> Next: Shanghai Summit & PTG 00:32:46 <gmann> we need to prepare the forum and PTG planning soon. 00:33:27 <gmann> masayukig: if ok for you, can you start the etherpad for forum brainstorming ideas and advertise to ML 00:33:39 <gmann> and we can use the same for PTG topic also. 00:33:52 <gmann> forum brainstorming+PTG 00:34:07 <masayukig> gmann: ok, I'll do 00:34:15 <gmann> thx 00:34:28 <gmann> #action masayukig to plan the forum brainstorming+PTG etherpad 00:34:58 <gmann> #topic Sub Teams highlights 00:35:04 <gmann> Tempest 00:35:40 <gmann> we have stestr version up in req freeze which need Tempest workaround due to teststool issue 00:35:50 <gmann> #link https://review.opendev.org/#/c/681340/4 00:36:09 <gmann> I have workaround up and need to get this in by today or tomorrow. this is high priority for us 00:36:36 <gmann> bigdogstl: ^^ this is to watch if anything break on AT&T testing 00:37:25 <gmann> Patrole 00:37:46 <gmann> for patrole, we need to get those resource leak patch in and add job in tempest gate 00:38:19 <gmann> anything else on patrole as I remember i mentioned few things to discuss in later topic 00:38:19 <bigdogstl> Once we have the downstream gate running we will be able to spotlight more upstream work 00:38:20 <masayukig> gmann: sure, I'll put my eyes on that patch. 00:38:26 <gmann> thanks 00:38:31 <bigdogstl> thanks 00:38:52 <gmann> one case needs to handle which i am waiting for gate result 00:39:52 <masayukig> Tim Burke gave a comment about the workaround 00:41:17 <gmann> i see. i tried to monkey patching that but it did not work. But i can check Tim suggestion which seems much better 00:41:24 <gmann> let me test that locally and then on gate 00:41:40 <masayukig> ++ 00:42:11 <masayukig> it looks better if it works 00:42:19 <gmann> yeah. 00:42:32 <gmann> any other subteam to dicuss anything 00:42:34 <gmann> discuss 00:42:59 <gmann> #topic Bug Triage 00:43:02 <gmann> #link https://etherpad.openstack.org/p/openstack_qa_tempest_2019_bug_review 00:43:26 <bigdogstl> What is the best doc for bug lifecycle 00:43:57 <bigdogstl> I am not sure how to close items like 00:44:11 <gmann> #link https://wiki.openstack.org/wiki/BugTriage 00:44:19 <gmann> bigdogstl: this is what we follow ^^ 00:44:35 <gmann> i think etherpad you have created is good way to start 00:44:41 <openstackgerrit> Clark Boylan proposed openstack/devstack master: Fix worlddump log collection https://review.opendev.org/680340 00:44:56 <bigdogstl> ok I will try to catch up on the process 00:45:05 <gmann> only thing we need to care about is even old bug but we should triage them if that still exist and not fixed 00:45:17 <bigdogstl> like line 41 00:45:23 <bigdogstl> how do we close it 00:45:37 <bigdogstl> https://bugs.launchpad.net/tempest/+bug/1596458 - preprovision creds leaking.ISSUE/NotesThe stack trace is pointing to tempest.common.preprov_creds.PreProvisionedCredentialProvider:and this class has been movedWe also use pre-provision creds and we do not have any leak at this timePROPOSAL Close this bug 00:45:38 <openstack> Launchpad bug 1596458 in tempest "Preprovisioned cred provider may leak credentials" [High,Confirmed] 00:46:21 <gmann> ok, i think this should be confirmed by unit test only 00:47:28 <gmann> because this is case of error during get creds 00:47:33 <bigdogstl> what about a tool to see if the oslo concurrency directory have hash loack left over 00:48:07 <bigdogstl> we have a piece of code that check that and we fix them as we see them 00:48:16 <bigdogstl> must are plugin test 00:48:58 <bigdogstl> but from tempest we use preprovision and we check lock directory and set a yellow flag if it happens 00:49:30 <bigdogstl> most of the time it is negative test that do not have bullet proof tear down 00:49:49 <gmann> ok, that kind of stuff we can run as part of account job in the end 00:49:51 <bigdogstl> another word do not use tempest framework 00:50:00 <bigdogstl> +1 00:50:30 <gmann> bigdogstl: can you push that code up somewhere to see and then we can add in account job? 00:50:43 <bigdogstl> but we know of not master cred leaks at this time so I would like to close this since the bug was filed against the non lib version 00:51:03 <bigdogstl> give me an action on that 00:51:08 <gmann> bigdogstl: but code is not changed from old->new lb version 00:51:22 <bigdogstl> we had to add logic to not list tempest fixture locks 00:51:43 <gmann> #action bigdogstl to bring up the pre-provisioned cred resource leak code 00:51:43 <bigdogstl> ok 00:51:57 <bigdogstl> but the test have changed over the year and i do not know of any issue 00:52:18 <bigdogstl> if we close it and there are issue we will get a new bug 00:52:35 <gmann> yeah, bug seems on basis of static code analysis not actual. 00:52:47 <gmann> let me check it again and then we can take action 00:52:49 <bigdogstl> some are so old the data is not very good and the person that reported it is not responding 00:53:08 <bigdogstl> ok 00:53:24 <gmann> yeah if author does not provide required info which is must to proceed we can mark it incomplete 00:53:36 <bigdogstl> if we have a well define set of bug I will try too get other engineers to address them 00:53:43 <bigdogstl> ok 00:53:53 <gmann> bigdogstl: we have latest new bugs also for them 00:53:53 <bigdogstl> does that close it over time 00:54:20 <gmann> we used to maintain in single digit but now it is 45 00:54:22 <gmann> #link https://bugs.launchpad.net/tempest/+bugs?search=Search&field.status=New 00:55:00 <gmann> those are good to target and keep triage and add in same etherpad which we further discuss during office hour 00:55:11 <gmann> those are latest bugs reported 00:55:13 <bigdogstl> I was trying to effect the 160 number before storyboard move 00:55:35 <bigdogstl> ok 00:55:40 <gmann> yeah we should keep doing those also 00:56:04 <gmann> I will get sometime to add the comment in etherpad but not in this or next week 00:56:14 <bigdogstl> ok 00:56:15 <gmann> but let's check one by one and add comments there 00:56:21 <gmann> 4 min left 00:56:35 <gmann> anything on bug triage ? 00:56:52 <bigdogstl> not here 00:57:06 <gmann> ok, thanks 00:57:10 <gmann> #topic Critical Reviews 00:57:24 <gmann> any other critical review than we already discussed 00:57:45 <masayukig> https://review.opendev.org/#/c/668823/ This should be easy one 00:58:19 <gmann> masayukig: so badges works fine ? 00:58:31 <masayukig> yeah, it works now 00:58:51 <gmann> perfect, i will check. thanks masayukig for getting this goal. 00:59:01 <gmann> #topic Open Discussion 00:59:15 <masayukig> no worries :) 00:59:44 <bigdogstl> This was my first office hour I will be better prepared for the next one. Now that I understand the format. Thanks 01:00:08 <gmann> we have 2 item in open discussion 1. swift/ceph bug which i will comment 2. devstack speedy thing which we are discussing/discussed over patch and ML. 01:00:32 <gmann> bigdogstl: thanks a lot for showing up and working on many items. 01:00:40 <masayukig> ++ 01:01:00 <gmann> bigdogstl: we have very simple format here first 30 min for status things and 30 min for technical/bug discussions 01:01:02 <clarkb> I wanted to point out that it feels like the openstack gate needs an air traffic controller. We've got lots of resets happening across many projects and the queues are long 01:01:02 <bigdogstl> Your are welcome and thank you to you 2 and the other reviewers 01:01:44 <clarkb> some sort of coordination around getting bug fixes ready and enqueued/promoted would likely be helpful if someone is able to dig into that 01:02:19 <gmann> clarkb: oh. do you have links or bugs ? 01:02:28 <clarkb> gmann: thats the problem we need someone to do that work 01:02:43 <clarkb> I've been unable to dig into why openstack jobs are failing as I've been debugging infra issues 01:03:04 <bigdogstl> which gates? 01:03:14 <clarkb> the openstack integrated gate queue 01:03:29 <clarkb> keystone + nova + cinder + neutron + glance (and probably others) 01:03:58 <gmann> grenade and openstack-tox-py27 seems failing frequently 01:04:29 <bigdogstl> ok 01:04:55 <clarkb> basically we need someone to identify issues, coordinate getting them fixed, then point out when they are ready so that we can promote them and fix them quicker 01:05:15 <gmann> seems like upper-constraints issue on py27 01:05:17 <clarkb> mriedem was looking at failures today and filed some e-r bugs 01:05:18 <gmann> ok 01:05:33 <bigdogstl> openstack-tox-py27 should be easy to create bugs and gather information about 01:06:05 <clarkb> gmann: cmurphy points out it is relatedto configparser changes maybe? over in #openstack-infra 01:06:16 <bigdogstl> Sorry for the silly question I am new and trying to see when myself or my team mates can help out 01:06:19 <cmurphy> hi 01:07:21 <gmann> yeah i am afraid that is not caught while version up in req gate 01:07:41 <clarkb> bigdogstl: I think the first thing we need is someone to spend some time identifiying the common errors we are seeing so that we can file bugs and get the appropriate teams working to fix them. Then as fixes happen let the infra team know so that we can promote those changes ahead of changes that are likely to fail due to the bugs that these fixes fix :) 01:08:01 <bigdogstl> +1 01:08:02 <clarkb> bigdogstl: its about the end of my day but I'll be around starting at about 1500UTC tomorrow if people want a quick walkthrough 01:08:44 <gmann> sure. 01:08:54 <gmann> I will try to ding into those today if get time 01:09:15 <gmann> let's close the office hour as time is over. 01:09:18 <gmann> #endmeeting