08:58:25 <VedaAnnayappa> #startmeeting powervm_driver_meeting
08:58:26 <openstack> Meeting started Wed Aug  7 08:58:25 2019 UTC and is due to finish in 60 minutes.  The chair is VedaAnnayappa. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:58:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:58:29 <openstack> The meeting name has been set to 'powervm_driver_meeting'
08:59:01 <VedaAnnayappa> Hi
08:59:09 <anuragmahanto243> Hi
09:00:16 <anuragmahanto243> #VedaAnnayappa ci-infra is back online
09:01:11 <VedaAnnayappa> #anuragmahanto243 good news
09:01:43 <VedaAnnayappa> in the last meeting we were discussing on mysql connection issue.. what was the troubleshoot done
09:01:45 <anuragmahanto243> :]
09:02:13 <VedaAnnayappa> what are the other issues resolved to bring back the infra to working state
09:03:46 <anuragmahanto243> concurrent connections on mysql was more, but killing processes wasn`t helping.
09:04:02 <anuragmahanto243> so I redeployed the CI
09:05:16 <VedaAnnayappa> Ok.. is there any other workaround applied?
09:05:19 <anuragmahanto243> at the time of redeployment , I found 2 git links were moved to opendev. and nodepool was internally calling pyyaml load()
09:07:14 <VedaAnnayappa> any idea which version of pyyaml used?
09:07:17 <anuragmahanto243> which is deprecated. so manually deleted stale nodes
09:07:38 <anuragmahanto243> python ignore warning did`nt help.
09:09:04 <anuragmahanto243> Version: 5.1.2
09:09:23 <anuragmahanto243> load() is depricated
09:09:28 <VedaAnnayappa> so this issue can reappear in future. we would need some generic solution to look at recovering from mysql connection failure and to delete stale nodes
09:10:14 <anuragmahanto243> we would have to upgrade to python 3
09:11:35 <VedaAnnayappa> ok.. i still see the test failing
09:12:02 <anuragmahanto243> or  patch "yaml.load(input, Loader=yaml.FullLoader)"
09:12:41 <anuragmahanto243> yes right now I am looking into , IOError: [Errno 28] No space left on device in out-of-tree
09:12:44 <MujahidAli> anurag: did you find any failures here: http://ciwatch.mmedvede.net/project?project=nova&time=7+days
09:14:00 <VedaAnnayappa> space constraints are in the VM that runs the test bucket?
09:14:48 <VedaAnnayappa> what is the VM size, Do we need to increase the size of the VM as new tests keeps accumulated as part of tempest bucket run.
09:15:02 <anuragmahanto243> mujahid: yes apart from that I checked time out exception. but did`nt dig deep into those
09:16:08 <anuragmahanto243> veda: I am trying 2 approachs, 1) clean tmp and 2) with disk space of 50 gb
09:16:17 <anuragmahanto243> current disk space is 30 gb
09:16:53 <VedaAnnayappa> ok. how many tests are failing?
09:17:22 <anuragmahanto243> 5 (out-of-tree)
09:20:07 <VedaAnnayappa> ok..
09:23:38 <VedaAnnayappa> I dont see any merge to be done in nova to nova-powervm.
09:24:18 <VedaAnnayappa> anurag, Thanks for the updates, keep up the good work, hope to see all green by next week
09:24:27 <VedaAnnayappa> any other points to discuss?
09:24:41 <anuragmahanto243> sure veda.
09:24:53 <anuragmahanto243> no thats all from my side
09:25:03 <VedaAnnayappa> thanks all
09:25:18 <VedaAnnayappa> #endmeeting