08:58:25 #startmeeting powervm_driver_meeting 08:58:26 Meeting started Wed Aug 7 08:58:25 2019 UTC and is due to finish in 60 minutes. The chair is VedaAnnayappa. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:58:27 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:58:29 The meeting name has been set to 'powervm_driver_meeting' 08:59:01 Hi 08:59:09 Hi 09:00:16 #VedaAnnayappa ci-infra is back online 09:01:11 #anuragmahanto243 good news 09:01:43 in the last meeting we were discussing on mysql connection issue.. what was the troubleshoot done 09:01:45 :] 09:02:13 what are the other issues resolved to bring back the infra to working state 09:03:46 concurrent connections on mysql was more, but killing processes wasn`t helping. 09:04:02 so I redeployed the CI 09:05:16 Ok.. is there any other workaround applied? 09:05:19 at the time of redeployment , I found 2 git links were moved to opendev. and nodepool was internally calling pyyaml load() 09:07:14 any idea which version of pyyaml used? 09:07:17 which is deprecated. so manually deleted stale nodes 09:07:38 python ignore warning did`nt help. 09:09:04 Version: 5.1.2 09:09:23 load() is depricated 09:09:28 so this issue can reappear in future. we would need some generic solution to look at recovering from mysql connection failure and to delete stale nodes 09:10:14 we would have to upgrade to python 3 09:11:35 ok.. i still see the test failing 09:12:02 or patch "yaml.load(input, Loader=yaml.FullLoader)" 09:12:41 yes right now I am looking into , IOError: [Errno 28] No space left on device in out-of-tree 09:12:44 anurag: did you find any failures here: http://ciwatch.mmedvede.net/project?project=nova&time=7+days 09:14:00 space constraints are in the VM that runs the test bucket? 09:14:48 what is the VM size, Do we need to increase the size of the VM as new tests keeps accumulated as part of tempest bucket run. 09:15:02 mujahid: yes apart from that I checked time out exception. but did`nt dig deep into those 09:16:08 veda: I am trying 2 approachs, 1) clean tmp and 2) with disk space of 50 gb 09:16:17 current disk space is 30 gb 09:16:53 ok. how many tests are failing? 09:17:22 5 (out-of-tree) 09:20:07 ok.. 09:23:38 I dont see any merge to be done in nova to nova-powervm. 09:24:18 anurag, Thanks for the updates, keep up the good work, hope to see all green by next week 09:24:27 any other points to discuss? 09:24:41 sure veda. 09:24:53 no thats all from my side 09:25:03 thanks all 09:25:18 #endmeeting