09:01:53 <anuragmahanto243> #startmeeting openstack-powervm
09:01:54 <openstack> Meeting started Wed Jul 31 09:01:53 2019 UTC and is due to finish in 60 minutes.  The chair is anuragmahanto243. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:01:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:01:57 <openstack> The meeting name has been set to 'openstack_powervm'
09:07:10 <VedaAnnayappa4> Hi Anurag
09:07:49 <VedaAnnayappa4> what the current status of the CI, looks like it has broken again?
09:09:29 <anuragmahanto243> #VedaAnnayappa4 Yes, on monday I was seeing the devstack services failing , i tried restarting the devstack services, but keystone was always failing
09:10:09 <anuragmahanto243> so then I re-deployed the control node and management nodes.
09:10:55 <MujahidAli> +1
09:10:59 <MujahidAli> joined
09:11:02 <anuragmahanto243> yesterday, I could see in nodepool logs the templates were forming.
09:12:06 <anuragmahanto243> but today , I recheck the nodepool log and hit "Max retries exceeded with url: /crumbIssuer/api/json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f39080f2690>:"
09:13:00 <VedaAnnayappa4> #anuragmahanto243 is the mysql right state?
09:13:50 <VedaAnnayappa4> also if there any firewall that is blocking the call
09:14:18 <anuragmahanto243> #VedaAnnayappa4 yes mysql service status is fine.
09:15:18 <VedaAnnayappa4> #MujahidAli  had you encountered these issues before.. in your experience, why are the system so unstable
09:16:42 <MujahidAli> I think it's problem with mysql.
09:17:49 <MujahidAli> I looked into the logs and found that mysql reached maximum connection, the problem might occurred due to excessive restart of mysql or all the ports exhausted for mysql.
09:18:42 <VedaAnnayappa4> how can we resolve this day-in day-out issues of reaching the max connection?
09:19:15 <VedaAnnayappa4> can we have some cleanup solution is there is any dangling connections?
09:19:30 <VedaAnnayappa4> if*
09:20:24 <VedaAnnayappa4> what is the main cause of mysql reaching the max limit, is there any leak that is not caught ?
09:24:47 <anuragmahanto243> #VedaAnnayappa4 restarted mysql service once after playbook run. I encounter the issue while 2nd-template creation.
09:25:26 <VedaAnnayappa4> #MujahidAli #anuragmahanto243 then in that case it should not reach the max limit.. what is the current limit, is there any scope to increase it?
09:25:31 <MujahidAli> I think it's a problem with MYSQL
09:26:27 <MujahidAli> It need to be updated
09:27:11 <VedaAnnayappa4> #MujahidAli do u mean that it need to be updated to latest version?
09:28:33 <MujahidAli> we need to update, but before doing that we need to make sure that our current version of nodepool is not broken by updating it.
09:29:26 <anuragmahanto243> #VedaAnnayappa4 our current is "mysql  Ver 14.14 Distrib 5.7.27" and latest is 8.0, we are 3 versions far.
09:30:16 <anuragmahanto243> #MujahidAli any work around for , current mysql issue.
09:31:25 <VedaAnnayappa4> #anuragmahanto243 then thats a quite old versions that we are using it. this might also have an impacts on other modules and components
09:31:35 <MujahidAli> yes
09:31:54 <MujahidAli> but if we upgrade it then it might break the other running services.
09:33:12 <VedaAnnayappa4> so looks like its quite a dead lock situation and to be handled carefully.
09:37:43 <anuragmahanto243> #MujahidAli how can I solve the current mysql issue.
09:38:24 <MujahidAli> Google is the solution to almost every problem :)
09:38:52 <MujahidAli> I think we need to manually cleanup the connections
09:39:07 <MujahidAli> and then see if that helps.
09:41:02 <anuragmahanto243> Sure, I will do this (y) .  Any suggestions on a permanent solution.
09:43:15 <MujahidAli> Permanent solution should be: Upgrade nodepool, jenkins, zuul and mysql to the version that community is using.
09:44:15 <MujahidAli> You can try upgrading all of the above services on staging env and see if that worked then you are good to replicate the same on prod.
09:46:07 <VedaAnnayappa4> fine.. lets target it stage by stage.. Anurag, try manual cleanup.. we will sync up next week for positive signs.
09:46:22 <anuragmahanto243> sure
09:46:23 <VedaAnnayappa4> any other topics for discussion
09:46:59 <MujahidAli> nothing from me.
09:47:03 <anuragmahanto243> nothing more from myside
09:49:40 <anuragmahanto243> #endmeeting openstack-powervm