| *** ricolin has joined #openstack-powervm | 01:01 | |
| *** ricolin has quit IRC | 01:31 | |
| *** ricolin has joined #openstack-powervm | 08:40 | |
| *** dikonoor has joined #openstack-powervm | 08:43 | |
| *** anuragmahanto243 has joined #openstack-powervm | 08:53 | |
| anuragmahanto243 | #startmeeting | 09:01 |
|---|---|---|
| openstack | anuragmahanto243: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 09:01 |
| anuragmahanto243 | #startmeeting openstack-powervm | 09:01 |
| openstack | Meeting started Wed Jul 31 09:01:53 2019 UTC and is due to finish in 60 minutes. The chair is anuragmahanto243. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:01 |
| *** openstack changes topic to " (Meeting topic: openstack-powervm)" | 09:01 | |
| openstack | The meeting name has been set to 'openstack_powervm' | 09:01 |
| *** VedaAnnayappa4 has joined #openstack-powervm | 09:07 | |
| VedaAnnayappa4 | Hi Anurag | 09:07 |
| VedaAnnayappa4 | what the current status of the CI, looks like it has broken again? | 09:07 |
| anuragmahanto243 | #VedaAnnayappa4 Yes, on monday I was seeing the devstack services failing , i tried restarting the devstack services, but keystone was always failing | 09:09 |
| anuragmahanto243 | so then I re-deployed the control node and management nodes. | 09:10 |
| *** MujahidAli has joined #openstack-powervm | 09:10 | |
| MujahidAli | +1 | 09:10 |
| MujahidAli | joined | 09:10 |
| anuragmahanto243 | yesterday, I could see in nodepool logs the templates were forming. | 09:11 |
| anuragmahanto243 | but today , I recheck the nodepool log and hit "Max retries exceeded with url: /crumbIssuer/api/json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f39080f2690>:" | 09:12 |
| VedaAnnayappa4 | #anuragmahanto243 is the mysql right state? | 09:13 |
| VedaAnnayappa4 | also if there any firewall that is blocking the call | 09:13 |
| anuragmahanto243 | #VedaAnnayappa4 yes mysql service status is fine. | 09:14 |
| VedaAnnayappa4 | #MujahidAli had you encountered these issues before.. in your experience, why are the system so unstable | 09:15 |
| MujahidAli | I think it's problem with mysql. | 09:16 |
| MujahidAli | I looked into the logs and found that mysql reached maximum connection, the problem might occurred due to excessive restart of mysql or all the ports exhausted for mysql. | 09:17 |
| VedaAnnayappa4 | how can we resolve this day-in day-out issues of reaching the max connection? | 09:18 |
| VedaAnnayappa4 | can we have some cleanup solution is there is any dangling connections? | 09:19 |
| VedaAnnayappa4 | if* | 09:19 |
| VedaAnnayappa4 | what is the main cause of mysql reaching the max limit, is there any leak that is not caught ? | 09:20 |
| anuragmahanto243 | #VedaAnnayappa4 restarted mysql service once after playbook run. I encounter the issue while 2nd-template creation. | 09:24 |
| VedaAnnayappa4 | #MujahidAli #anuragmahanto243 then in that case it should not reach the max limit.. what is the current limit, is there any scope to increase it? | 09:25 |
| MujahidAli | I think it's a problem with MYSQL | 09:25 |
| MujahidAli | It need to be updated | 09:26 |
| VedaAnnayappa4 | #MujahidAli do u mean that it need to be updated to latest version? | 09:27 |
| MujahidAli | we need to update, but before doing that we need to make sure that our current version of nodepool is not broken by updating it. | 09:28 |
| anuragmahanto243 | #VedaAnnayappa4 our current is "mysql Ver 14.14 Distrib 5.7.27" and latest is 8.0, we are 3 versions far. | 09:29 |
| anuragmahanto243 | #MujahidAli any work around for , current mysql issue. | 09:30 |
| VedaAnnayappa4 | #anuragmahanto243 then thats a quite old versions that we are using it. this might also have an impacts on other modules and components | 09:31 |
| MujahidAli | yes | 09:31 |
| MujahidAli | but if we upgrade it then it might break the other running services. | 09:31 |
| VedaAnnayappa4 | so looks like its quite a dead lock situation and to be handled carefully. | 09:33 |
| *** dikonoor has quit IRC | 09:36 | |
| anuragmahanto243 | #MujahidAli how can I solve the current mysql issue. | 09:37 |
| MujahidAli | Google is the solution to almost every problem :) | 09:38 |
| MujahidAli | I think we need to manually cleanup the connections | 09:38 |
| MujahidAli | and then see if that helps. | 09:39 |
| anuragmahanto243 | Sure, I will do this (y) . Any suggestions on a permanent solution. | 09:41 |
| MujahidAli | Permanent solution should be: Upgrade nodepool, jenkins, zuul and mysql to the version that community is using. | 09:43 |
| MujahidAli | You can try upgrading all of the above services on staging env and see if that worked then you are good to replicate the same on prod. | 09:44 |
| VedaAnnayappa4 | fine.. lets target it stage by stage.. Anurag, try manual cleanup.. we will sync up next week for positive signs. | 09:46 |
| anuragmahanto243 | sure | 09:46 |
| VedaAnnayappa4 | any other topics for discussion | 09:46 |
| MujahidAli | nothing from me. | 09:46 |
| anuragmahanto243 | nothing more from myside | 09:47 |
| anuragmahanto243 | #endmeeting openstack-powervm | 09:49 |
| *** openstack changes topic to "This channel is for PowerVM-related development and discussion. For general OpenStack support, please use #openstack." | 09:49 | |
| openstack | Meeting ended Wed Jul 31 09:49:40 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_powervm/2019/openstack_powervm.2019-07-31-09.01.html | 09:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_powervm/2019/openstack_powervm.2019-07-31-09.01.txt | 09:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_powervm/2019/openstack_powervm.2019-07-31-09.01.log.html | 09:49 |
| *** VedaAnnayappa4 has quit IRC | 10:04 | |
| *** MujahidAli has quit IRC | 10:10 | |
| *** anuragmahanto243 has quit IRC | 10:29 | |
| *** ricolin_ has joined #openstack-powervm | 11:01 | |
| *** ricolin has quit IRC | 11:03 | |
| *** ricolin_ is now known as ricolin | 12:50 | |
| *** efried is now known as efried_rollin | 15:57 | |
| *** ricolin has quit IRC | 16:50 | |
| *** altlogbot_3 has quit IRC | 18:16 | |
| *** altlogbot_0 has joined #openstack-powervm | 18:20 | |
| *** efried_rollin is now known as efried | 19:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!