| *** gouthamr has quit IRC | 00:09 | |
| *** kumarmn has joined #openstack-trove | 00:11 | |
| *** kumarmn has quit IRC | 00:17 | |
| *** tosky has quit IRC | 00:28 | |
| *** kumarmn has joined #openstack-trove | 00:38 | |
| *** kumarmn has quit IRC | 01:00 | |
| *** kumarmn has joined #openstack-trove | 01:02 | |
| *** kumarmn has quit IRC | 01:02 | |
| *** kumarmn has joined #openstack-trove | 01:03 | |
| *** kumarmn has quit IRC | 01:07 | |
| *** gouthamr has joined #openstack-trove | 01:08 | |
| *** kumarmn has joined #openstack-trove | 01:12 | |
| *** kumarmn has quit IRC | 01:19 | |
| *** hui has joined #openstack-trove | 01:40 | |
| *** zhaochao has joined #openstack-trove | 01:42 | |
| *** kumarmn has joined #openstack-trove | 02:33 | |
| fanzhang | kumarmn: there are lots of zuul jobs failed, is there anything we can do to fix that? | 02:36 |
|---|---|---|
| *** kei_yama has quit IRC | 03:35 | |
| *** gouthamr has quit IRC | 03:45 | |
| *** daidv has joined #openstack-trove | 03:48 | |
| *** daidv_ has joined #openstack-trove | 03:48 | |
| *** daidv_ has quit IRC | 03:49 | |
| *** kei_yama has joined #openstack-trove | 04:01 | |
| *** links has joined #openstack-trove | 04:07 | |
| *** links has quit IRC | 04:07 | |
| *** kumarmn has quit IRC | 04:17 | |
| *** kei_yama has quit IRC | 05:02 | |
| *** kumarmn has joined #openstack-trove | 05:09 | |
| *** kumarmn has quit IRC | 05:14 | |
| *** kei_yama has joined #openstack-trove | 05:22 | |
| *** kumarmn has joined #openstack-trove | 06:10 | |
| *** kumarmn has quit IRC | 06:14 | |
| *** kumarmn has joined #openstack-trove | 07:11 | |
| *** kumarmn has quit IRC | 07:16 | |
| *** hoonetorg has quit IRC | 07:18 | |
| *** hoonetorg has joined #openstack-trove | 07:31 | |
| *** rcernin has quit IRC | 07:43 | |
| *** b_bezak has joined #openstack-trove | 08:13 | |
| *** damien_r has joined #openstack-trove | 08:27 | |
| *** damien_r has quit IRC | 08:28 | |
| *** damien_r has joined #openstack-trove | 08:29 | |
| *** magicboiz has joined #openstack-trove | 08:35 | |
| *** rcernin has joined #openstack-trove | 08:48 | |
| *** Jack_Iv has joined #openstack-trove | 09:07 | |
| *** damien_r has quit IRC | 09:10 | |
| *** damien_r has joined #openstack-trove | 09:11 | |
| *** kumarmn has joined #openstack-trove | 09:12 | |
| *** kumarmn has quit IRC | 09:17 | |
| *** magicboiz has quit IRC | 09:43 | |
| *** magicboiz has joined #openstack-trove | 09:51 | |
| *** tosky has joined #openstack-trove | 10:37 | |
| openstackgerrit | zhanggang proposed openstack/python-troveclient master: Modify database-list in OSC https://review.openstack.org/527056 | 10:47 |
| *** Jack_Iv has quit IRC | 11:08 | |
| *** kumarmn has joined #openstack-trove | 11:14 | |
| *** Jack_Iv has joined #openstack-trove | 11:18 | |
| *** kumarmn has quit IRC | 11:19 | |
| *** Jack_Iv has quit IRC | 11:23 | |
| *** kei_yama has quit IRC | 11:34 | |
| *** Jack_Iv has joined #openstack-trove | 12:08 | |
| *** Jack_Iv_ has joined #openstack-trove | 12:23 | |
| *** Jack_Iv has quit IRC | 12:27 | |
| *** smatzek has joined #openstack-trove | 12:45 | |
| fanzhang | hi smatzek you there? | 12:47 |
| fanzhang | there are lots of zuul jobs failed, you are the expert at gate jobs, would you please take a look? | 12:48 |
| *** ChanServ has quit IRC | 13:08 | |
| smatzek | fanzhang: I'm here. I'll take a look and see if there is anything I can do. | 13:14 |
| *** kumarmn has joined #openstack-trove | 13:16 | |
| fanzhang | smatzek: thx...I tried to seek help from infra team, but they suggested it was some deeply unit tests failure. I am not sure why the jobs fail so suddenly.. | 13:16 |
| fanzhang | it seems we can not create a mysql instance successfully, then related jobs fail | 13:17 |
| *** ChanServ has joined #openstack-trove | 13:18 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 13:18 | |
| *** kumarmn has quit IRC | 13:20 | |
| smatzek | fanzhang: have you looked into this error at all yet? http://logs.openstack.org/05/526605/4/check/legacy-trove-functional-dsvm-mysql/f493108/logs/screen-n-cpu.txt.gz#_Dec_10_15_54_33_712268 | 13:23 |
| *** rcernin has quit IRC | 13:23 | |
| fanzhang | smatzek: I have not checked this log..I only checked the job-output...http://logs.openstack.org/96/522696/14/check/legacy-trove-functional-dsvm-mysql/73f665c/ | 13:27 |
| smatzek | the way I debug these errors for most of the gates is look in the logs/ directory. I open the *-post_test_hook.txt.gz file, which contains the run of all the tests. You get to see status of all the tests. | 13:29 |
| smatzek | Then find the test that is failing, typically one or two tests fail and a bunch of others fail due to dependencies on the instance that failed. | 13:29 |
| smatzek | Next, take the instance UUID from that file and search for it in logs/screen-tr-mgr.txt.gz, which is Trove manager's log file. From there, you can find entries that correlate the Trove ID with the Nova UUID and you can search the screen-nova-cpu.txt.gz log if necessary. That file is the nova compute host log. | 13:31 |
| fanzhang | Oh I see, I had no clue where I could find the task manager's log and like nova's log. You just make that clear :) thanks so much | 13:33 |
| fanzhang | I usually check the output log, and then look through the related code to see if there were anything wrong. But in my env, I can access to the trove node, controller node, it's easier to get the log | 13:36 |
| fanzhang | smatzek: really appreciate that you tell me all these tricks :) | 13:36 |
| smatzek | no problem. Its good if other people can learn how to do this debug. | 13:37 |
| smatzek | When many reviews are failing I typically look at multiple reviews to see if there is a pattern to their failures. | 13:38 |
| smatzek | I just looked down another one and found this error: http://logs.openstack.org/39/524539/1/check/legacy-trove-scenario-dsvm-mysql-single/e96ebe5/logs/screen-tr-tmgr.txt.gz#_Dec_11_03_10_39_478892 | 13:38 |
| smatzek | it's not the same volume error but mysql is complaining about mounting a device path. At the moment I'm thinking something in Nova or Cinder changed how storage is handled. | 13:38 |
| fanzhang | Yeah I do look at multiple reviews but I never get the right log, LOL | 13:42 |
| fanzhang | How can we fix that then? Seek help from nova and cinder? | 13:42 |
| smatzek | first you have to root cause it. I just found another run fail wit the volume driver error in the nova compute log. | 13:43 |
| smatzek | so I look at the code involved, see if it has changed recently, and why the changes were made, etc. Try to figure out why it is failing... Then if Nova or Cinder changes are needed I approach those teams. | 13:44 |
| smatzek | The error may be that Trove needs additional config changes in its devstack setup. It may not be a functional issue in Nova, Cinder, etc. | 13:45 |
| fanzhang | I am trying to do like you said...really learn a lot. | 13:48 |
| smatzek | I'm going to be in work meetings for the next few hours but will try to look more later today. | 13:52 |
| fanzhang | ok really appreciate that | 13:54 |
| *** pmackinn has joined #openstack-trove | 13:54 | |
| smatzek | fanzhang: do you have a window of time where these failures started happening? That will help locate the breaking change. on Sunday, Saturday, Friday, etc? | 13:55 |
| fanzhang | smatzek: it seems like Dec.08 or Dec 09 | 13:55 |
| fanzhang | smatzek: my bp failed yesterday 04:05 the first time | 13:56 |
| fanzhang | Dec 10 04:05 UTC+8 | 13:57 |
| *** kumarmn has joined #openstack-trove | 13:58 | |
| *** kumarmn has quit IRC | 14:03 | |
| openstackgerrit | Yao Wang proposed openstack/trove-dashboard master: Fix incorrect example of mysql connection https://review.openstack.org/527096 | 14:06 |
| *** gcb has joined #openstack-trove | 14:24 | |
| *** gouthamr has joined #openstack-trove | 14:26 | |
| *** smatzek has quit IRC | 14:30 | |
| *** smatzek has joined #openstack-trove | 14:30 | |
| *** smatzek has quit IRC | 14:30 | |
| *** kumarmn has joined #openstack-trove | 14:50 | |
| *** b_bezak has quit IRC | 14:59 | |
| *** b_bezak has joined #openstack-trove | 15:00 | |
| *** b_bezak has quit IRC | 15:04 | |
| *** smatzek has joined #openstack-trove | 15:06 | |
| *** magicboiz has quit IRC | 15:13 | |
| *** magicboiz has joined #openstack-trove | 15:14 | |
| amrith | kumarmn please identify a new oslo-trove liaison. I've not been attending the oslo meetings on a regular basis. | 15:26 |
| chandankumar | kumarmn: need acks on these review https://review.openstack.org/#/c/527119/ and https://review.openstack.org/#/c/527122/ for tempest plugin split goal | 15:28 |
| chandankumar | kumarmn: here is the imported tests in a seperate repo https://github.com/chkumar246/trove-tempest-plugin | 15:30 |
| kumarmn | amrith: I can nominate someone after this week's trove meeting. | 15:34 |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: Fix Increase guest agent cmd process timeout https://review.openstack.org/527135 | 15:41 |
| *** gcb has quit IRC | 15:44 | |
| chandankumar | kumarmn: i have commented on the reviews, i am not sure i answered properly. | 15:45 |
| kumarmn | chandrankumar: thanks, I believe you have answered amrith's question, but will let him clarify. | 15:48 |
| *** gouthamr has quit IRC | 16:01 | |
| *** dkehn has quit IRC | 16:08 | |
| *** dkehn has joined #openstack-trove | 16:08 | |
| *** vipul has quit IRC | 16:10 | |
| *** vipul has joined #openstack-trove | 16:10 | |
| *** damien_r has quit IRC | 16:50 | |
| *** Jack_Iv_ has quit IRC | 16:54 | |
| *** Jack_Iv has joined #openstack-trove | 16:55 | |
| *** Jack_Iv_ has joined #openstack-trove | 17:06 | |
| *** Jack_Iv has quit IRC | 17:09 | |
| *** damien_r has joined #openstack-trove | 17:44 | |
| *** Jack_Iv_ has quit IRC | 18:28 | |
| *** magicboiz has quit IRC | 18:32 | |
| *** Jack_Iv has joined #openstack-trove | 18:38 | |
| *** Jack_Iv has quit IRC | 18:43 | |
| *** tosky has quit IRC | 18:46 | |
| smatzek | kumarmn: I'm asking about the trove breakage in Nova's channel right now. | 18:49 |
| kumarmn | thanks smatzek | 18:49 |
| *** damien_r has quit IRC | 18:53 | |
| *** Jack_Iv has joined #openstack-trove | 19:06 | |
| *** tosky has joined #openstack-trove | 19:15 | |
| *** itlinux has joined #openstack-trove | 19:17 | |
| *** itlinux has quit IRC | 19:22 | |
| *** itlinux has joined #openstack-trove | 19:27 | |
| *** itlinux has quit IRC | 19:29 | |
| openstackgerrit | Samuel Matzek proposed openstack/trove master: DNM: Test connection_info fix https://review.openstack.org/527230 | 19:53 |
| *** Jack_Iv has quit IRC | 19:59 | |
| *** smatzek has quit IRC | 21:23 | |
| *** tosky__ has joined #openstack-trove | 21:48 | |
| *** tosky has quit IRC | 21:49 | |
| *** tosky__ is now known as tosky | 21:49 | |
| *** tonyb_ has joined #openstack-trove | 21:51 | |
| *** tonyb has quit IRC | 21:56 | |
| *** gouthamr has joined #openstack-trove | 22:00 | |
| *** rcernin has joined #openstack-trove | 22:19 | |
| kumarmn | smatzek: I see that https://review.openstack.org/#/c/527228/ is in process. From the openstack-nova history, it looks like there is more than one issue. | 22:29 |
| *** gouthamr has quit IRC | 22:40 | |
| *** kumarmn has quit IRC | 22:47 | |
| *** kumarmn has joined #openstack-trove | 23:06 | |
| *** kei_yama has joined #openstack-trove | 23:07 | |
| *** kumarmn has quit IRC | 23:11 | |
| *** tosky has quit IRC | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!