15:00:22 <kumarmn> #startmeeting trove
15:00:23 <openstack> Meeting started Wed Dec 13 15:00:22 2017 UTC and is due to finish in 60 minutes.  The chair is kumarmn. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:27 <openstack> The meeting name has been set to 'trove'
15:01:02 <kumarmn> #topic rollcall
15:01:55 <kumarmn> who all do we have today?
15:03:10 <fanzhang> o/
15:03:10 <songjian> hi kumarmn,i am here
15:03:42 <kumarmn> ok smatzek let me know that he can only join later.
15:05:27 <kumarmn> #topic crossproject_liaison
15:05:51 <kumarmn> amrith let me know he has not been attending the oslo meetings.
15:06:07 <kumarmn> We need to identify who within trove will be the oslo liaison:
15:06:10 <kumarmn> https://wiki.openstack.org/wiki/CrossProjectLiaisons
15:06:49 <kumarmn> fanzhang or songjian, would you like to be the oslo liaison?
15:07:34 <fanzhang> kumarmn: what's oslo liaison?
15:10:22 <kumarmn> each openstack project needs to have someone representing them in oslo meetings.
15:10:52 <kumarmn> i believe they are on mondays, i will need to check the time. either 15:00 or 16:00 UTC
15:11:24 <fanzhang> it's 14:00 UTC which is 22:00 in Beijing
15:11:49 <fanzhang> I just saw it in https://wiki.openstack.org/wiki/Meetings/Oslo
15:12:29 <kumarmn> The other requirements are on the link I sent
15:13:17 <songjian> http://eavesdrop.openstack.org/#Oslo_Team_Meeting
15:13:36 <songjian> look like is 15:00
15:14:33 <fanzhang> Changbo guo is the PTL of oslo?
15:15:02 <songjian> yes
15:15:08 <fanzhang> it looks like songjian is the perfect liaison candidate
15:16:23 <songjian> Well, I think I can try
15:16:37 <fanzhang> Actually I'm looking for a new job opportunity recently, I am sure I have the time, energy for that.
15:17:06 <kumarmn> you are sure, OR you are not sure?
15:17:58 <fanzhang> I am not, sorry. I propose songjian being one for now. And if you want, I can be alternate for that role.
15:18:30 <kumarmn> thanks songjian
15:19:09 <kumarmn> i am sure there will be other opportunities fanzhang.
15:19:31 <kumarmn> #topic queens-2
15:19:55 <fanzhang> thx kumarmn,
15:19:55 <kumarmn> I was hoping to tag queens-2 for trove last weekend. I pushed for a lot of things to merge.
15:19:59 <songjian> Nothing, I can try, changbo is my leader, communication will be more convenient.
15:20:18 <zhanggang> Beijing full of opportunities :)
15:20:38 <kumarmn> I see, I did not know the connection. Where are you based zhanggang?
15:21:57 <fanzhang> Suzhou, I think
15:22:06 <zhanggang> My company is in SuZhou
15:22:27 <zhanggang> yes, a beautiful city. Welcome
15:24:06 <fanzhang> smatzek is here. Let's come back to the queens-2, unfortunately, a mess of gate jobs failed in the past two or three days
15:25:00 <fanzhang> It looks like we fall behind the schedule?
15:25:03 <kumarmn> yes, for queens-2 I would have liked to get your redis change, and the bunch of client changes that Yao has been pushing.
15:25:18 <kumarmn> However we had some gate failures, related to the nova drop.
15:25:32 <kumarmn> smatzek has been working with the nova team.
15:25:40 <kumarmn> looks like to two changes are under review.
15:25:56 <kumarmn> https://review.openstack.org/#/c/527228/
15:26:10 <kumarmn> https://review.openstack.org/#/c/527479/
15:26:38 <kumarmn> Anything else smatzek?
15:27:39 <smatzek> that's about it.  We have https://review.openstack.org/#/c/527230/1 to test with the Nova patches to ensure they fix Nova and our gates.
15:27:40 <zhanggang> I also commit some *_delete method to troveclient. And will commit *_create methods when the gate fix.
15:28:34 <smatzek> Zuul crashed or was at least restarted yesterday so we lost the recheck job.  I submitted another recheck this morning.  We know one of the fixes works but need to test out the other which was just created yesterday
15:29:22 <fanzhang> it's about to find out. It costs about 3 hours to finish a recheck job I think
15:29:22 <smatzek> on the gate front, we need to do some skill building here so I'm not the only one who has the skill to help get it working.  It's not good to be single threaded / single point of failure.
15:30:54 <fanzhang> all the tests in zuul are based on devstack, right?
15:30:57 <smatzek> I gave some tips on how to do that on my Monday morning in the trove channel
15:31:03 <smatzek> yes they are all based on devstack.
15:31:31 <smatzek> well not *all* the tests.the scenario and functional ones are, py27 obviously isn't.
15:32:55 <fanzhang> yes, that's right. LOL. I can try to keep up, since I've talked with you that night in the trove channel. It's all about how to find out the currect logs
15:33:08 <kumarmn> smatzek, good point. can you look through smatzek's comments on Monday and ask him questions on how to debug?
15:33:47 <fanzhang> definitely
15:34:48 <smatzek> beyond the pointers to the correct logs, which are in the irclogs from Monday, after that you just need to have some working knowledge of the things Trove uses (Cinder, Nova, etc), and start debugging by investigating the logs and the Trove, etc code.
15:35:37 <smatzek> for this past failure, I noted that both types of errors (there were two) that we were seeing were volume related. So I looked at the commit history of Nova and saw they changed their volume attach code right about the same time the Trove gate broke so I focused on that specific area.
15:37:51 <fanzhang> cool, really appreciate your hard work
15:38:40 <fanzhang> should we press the nova team to have these two patches merged in order to recover our gate jobs?
15:39:04 <songjian> The same gratitude for the sharing of these debug knowledge
15:39:19 <smatzek> I don't think we have to press them anymore.  The Nova PTL owns the bugs and worked directly with me the last two days on investigating and getting the fixes up.
15:39:35 <smatzek> The fixes are listed as High priority and Matt is pushing the fixes.
15:40:18 <fanzhang> oh, i see. the test patch's gate job just failed ...:(
15:44:19 <fanzhang> I've got the Quota exceeded for ram: Requested 131072, but already used 2048 of 51200 ram (HTTP 403) error inside http://logs.openstack.org/30/527230/1/check/legacy-trove-scenario-dsvm-mysql-single/8d14c7d/logs/screen-tr-tmgr.txt.gz
15:45:44 <smatzek> http://logs.openstack.org/30/527230/1/check/legacy-trove-scenario-dsvm-mysql-single/8d14c7d/logs/devstack-gate-post_test_hook.txt.gz#_2017-12-13_14_07_34_097
15:46:21 <smatzek> the way these test suites work is an instance will be created in one test case and then re-used in later test cases.  A failure early will cause a lot of downstream tests to fail.
15:46:43 <smatzek> This one failed because the inital instance failed to become active.  Why exactly?  I don't know yet
15:46:57 <zhanggang> I have a question. Is the same reason for trove zuul crash and troveclient zuul crash.
15:47:27 <smatzek> yes, the whole zuul system for openstack was restarted yesterday
15:47:50 <fanzhang> it seems like the quota exceeded for ram. and I've got another erorr Server error: Build of instance cffd6903-cbee-402c-8c0e-754bfa83867b aborted: Flavor's disk is too small for requested image. Flavor disk is 1073741824 bytes, image is 3194093568 bytes.
15:47:56 <zhanggang> OK, I know , thanks smatzek
15:48:26 <fanzhang> there are many error infos in http://logs.openstack.org/30/527230/1/check/legacy-trove-scenario-dsvm-mysql-single/8d14c7d/logs/screen-tr-tmgr.txt.gz
15:50:17 <smatzek> let's walk through this one if you have time. Sorry, I'm also in another chat client meeting and on a teleconference, 3 meetings at once :)
15:50:32 <smatzek> so this upgrade test failed, http://logs.openstack.org/30/527230/1/check/legacy-trove-scenario-dsvm-mysql-single/8d14c7d/logs/devstack-gate-post_test_hook.txt.gz#_2017-12-13_15_28_08_858
15:50:53 <smatzek> sorry, not upgrade, wait_for_instances, initial DB instance creation.
15:51:19 <smatzek> you can find the DB uuid following, 6d225c7a-097c-400c-b0f2-4ee84d4c6921
15:52:13 <smatzek> so if you look in that file you can see the test suite was able to pull out the guest agent log from the DB instance.  It's not always able to do this:  http://logs.openstack.org/30/527230/1/check/legacy-trove-scenario-dsvm-mysql-single/8d14c7d/logs/devstack-gate-post_test_hook.txt.gz#_2017-12-13_15_28_11_609
15:52:27 <smatzek> you can look in there, or start in trove manager.  I'm going to look in trove manager.
15:53:34 <smatzek> a quick find on the uuid with highlighting all the occurrences in the trove manager log makes it easy to follow.  I can see the Nova instance went to ACTIVE, which is good.
15:53:56 <fanzhang> yes, I get there too
15:55:07 <smatzek> Trove polls until the Nova instance goes active and then polls until the guest agent reports active/up.
15:57:13 <kumarmn> we might have to continue discussion on openstack-trove
15:57:24 <kumarmn> as our meeting hour is coming to an end.
15:57:31 <smatzek> k
15:57:36 <fanzhang> let's do that..
15:57:38 <kumarmn> any other topics?
15:59:08 <kumarmn> ok, see you all on openstack-trove.
15:59:14 <songjian> I was trying to add trove to openstack-helm,if finish,hope to help me to review
15:59:59 <songjian> In addition, nothing else
16:00:04 <kumarmn> #endmeeting