| *** kranthikirang has joined #openstack-helm | 01:42 | |
| *** kranthikirang has quit IRC | 01:47 | |
| *** sdake has joined #openstack-helm | 01:59 | |
| *** cfriesen has joined #openstack-helm | 02:03 | |
| *** sdake has quit IRC | 02:18 | |
| *** sdake has joined #openstack-helm | 02:22 | |
| *** sdake has quit IRC | 02:44 | |
| *** kranthikirang has joined #openstack-helm | 03:30 | |
| *** kranthikirang has quit IRC | 03:35 | |
| *** kranthikirang has joined #openstack-helm | 05:19 | |
| *** kranthikirang has quit IRC | 05:23 | |
| *** rchurch has quit IRC | 05:52 | |
| *** cfriesen has quit IRC | 06:36 | |
| *** sdake has joined #openstack-helm | 06:37 | |
| *** belmoreira has joined #openstack-helm | 06:49 | |
| *** kranthikirang has joined #openstack-helm | 07:07 | |
| *** kranthikirang has quit IRC | 07:11 | |
| *** aojea has joined #openstack-helm | 07:19 | |
| *** sdake has quit IRC | 07:30 | |
| *** nmimi has joined #openstack-helm | 07:33 | |
| *** witek has joined #openstack-helm | 08:00 | |
| *** gkadam has joined #openstack-helm | 08:06 | |
| *** gkadam has quit IRC | 08:09 | |
| *** jsuchome has joined #openstack-helm | 08:16 | |
| *** dimitris_ has joined #openstack-helm | 08:28 | |
| *** aojea has quit IRC | 08:57 | |
| *** JangwonLee_ has quit IRC | 09:06 | |
| *** roman_g has joined #openstack-helm | 09:11 | |
| *** lemko has joined #openstack-helm | 09:14 | |
| *** sdake has joined #openstack-helm | 09:14 | |
| *** nick_kar has joined #openstack-helm | 09:21 | |
| *** tone_zrt has joined #openstack-helm | 09:52 | |
| *** kranthikirang has joined #openstack-helm | 10:12 | |
| *** kranthikirang has quit IRC | 10:16 | |
| *** sdake has quit IRC | 11:16 | |
| *** sdake has joined #openstack-helm | 11:21 | |
| *** kranthikirang has joined #openstack-helm | 12:00 | |
| *** kranthikirang has quit IRC | 12:04 | |
| *** sdake has quit IRC | 12:12 | |
| *** sdake has joined #openstack-helm | 12:15 | |
| *** hemanth_n has joined #openstack-helm | 13:25 | |
| *** sdake has quit IRC | 13:28 | |
| *** leakypipes is now known as jaypipes | 13:39 | |
| *** JangwonLee has joined #openstack-helm | 13:42 | |
| *** kranthikirang has joined #openstack-helm | 13:48 | |
| *** kranthikirang has quit IRC | 13:52 | |
| *** sdake has joined #openstack-helm | 13:55 | |
| *** kranthikirang has joined #openstack-helm | 14:16 | |
| *** howell has joined #openstack-helm | 14:26 | |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm master: WIP/DNM Implement argo workflows into the keystone chart https://review.openstack.org/636346 | 14:45 |
|---|---|---|
| *** aojea has joined #openstack-helm | 14:53 | |
| *** dustinspecker has joined #openstack-helm | 14:55 | |
| *** aojea has quit IRC | 14:57 | |
| *** sdake has quit IRC | 14:58 | |
| *** kranthikirang has quit IRC | 15:03 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: Ceph Provisioners helm tests https://review.openstack.org/636735 | 15:03 |
| *** aaronsheffield has joined #openstack-helm | 15:03 | |
| *** kranthikirang has joined #openstack-helm | 15:03 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [CEPH] Switch to using ceph-volume for ceph-osd chart https://review.openstack.org/633981 | 15:03 |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [CEPH] Add support for creating custom EC profiles https://review.openstack.org/627197 | 15:04 |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm-infra master: This adds the ability to specify custom resource dependencies https://review.openstack.org/634037 | 15:05 |
| *** aojea has joined #openstack-helm | 15:06 | |
| *** hemanth_n has quit IRC | 15:07 | |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm master: WIP/DNM Implement argo workflows into the keystone chart https://review.openstack.org/636346 | 15:09 |
| openstackgerrit | Ian Howell proposed openstack/openstack-helm master: WIP/DNM Implement argo workflows into the keystone chart https://review.openstack.org/636346 | 15:10 |
| *** munimeha1 has joined #openstack-helm | 15:13 | |
| openstackgerrit | Scott Hussey proposed openstack/openstack-helm-infra master: (postgresql) Background process to set password https://review.openstack.org/635070 | 15:14 |
| *** happyhemant has joined #openstack-helm | 15:28 | |
| *** lemko has quit IRC | 15:39 | |
| happyhemant | hello, I was trying to install openstack-helm but had a problem with ceph-osd pods . Its crashLoopBackoff all the time. Does anybody know why is it like this or someone can help me solve this error. Refer the logs for better understanding | 15:40 |
| supamatt | can you run the following: kubectl log <ceph osd pod name> -n ceph? | 15:42 |
| happyhemant | https://www.irccloud.com/pastebin/PteR6vEa/ | 15:43 |
| happyhemant | yes have a look i was about to post it | 15:43 |
| supamatt | Are you mon pods healthy? | 15:44 |
| *** sdake has joined #openstack-helm | 15:44 | |
| supamatt | run 'ceph status' from inside a mon pod | 15:44 |
| happyhemant | https://www.irccloud.com/pastebin/Hl5hzZ3w/ | 15:46 |
| happyhemant | https://www.irccloud.com/pastebin/0fBHF25G/ | 15:47 |
| happyhemant | also have a look at the mon pods | 15:47 |
| supamatt | kubectl get pods -n ceph -o wide | 15:49 |
| supamatt | you may have a network problem | 15:49 |
| happyhemant | https://www.irccloud.com/pastebin/qqVcO4Ef/ | 15:50 |
| supamatt | Do you have mon endpoints showing here? kubectl get endpoints ceph-mon -n ceph | 15:51 |
| happyhemant | https://www.irccloud.com/pastebin/GR3HD5cj/ | 15:52 |
| happyhemant | yes i have it | 15:53 |
| supamatt | This is an issue I've heard about, but when I was told about it. I couldn't look into it at the time. | 15:59 |
| supamatt | Let me see if I can dig up that PS. | 15:59 |
| happyhemant | ok thanks :) | 15:59 |
| supamatt | You can try this PS, https://review.openstack.org/#/c/633981/ | 16:08 |
| happyhemant | ok thanks alot | 16:08 |
| supamatt | your dns may not be working, which is why the process is failing | 16:08 |
| supamatt | incidentaly this PS solves for that condition | 16:10 |
| happyhemant | sure i will try this and will get back to you tomorrow | 16:12 |
| happyhemant | if it works | 16:12 |
| happyhemant | thanks alot for the help | 16:12 |
| happyhemant | :) | 16:12 |
| *** lemko has joined #openstack-helm | 16:27 | |
| *** jsuchome has quit IRC | 16:39 | |
| *** sdake has quit IRC | 17:00 | |
| *** aojea has quit IRC | 17:04 | |
| *** sdake has joined #openstack-helm | 17:05 | |
| openstackgerrit | John Haan proposed openstack/openstack-helm-infra master: Fix for absent link packages in ceph deployment shell https://review.openstack.org/637592 | 17:27 |
| *** sdake has quit IRC | 18:29 | |
| *** sdake_ has joined #openstack-helm | 18:29 | |
| todin | Hi, does the cinder chart support different backends than ceph? | 19:23 |
| *** happyhemant has quit IRC | 19:28 | |
| *** nishant_ has joined #openstack-helm | 19:33 | |
| kranthikirang | supamatt: please take a look into https://bugs.launchpad.net/openstack-helm-infra/+bug/1816478 | 19:49 |
| openstack | Launchpad bug 1816478 in openstack-helm-infra "Mimic version active ceph-mgr memory leak" [Undecided,New] | 19:49 |
| openstackgerrit | chinasubbareddy mallavarapu proposed openstack/openstack-helm-infra master: ceph-rgw: Add network policy for ceph-rgw pods https://review.openstack.org/632567 | 19:49 |
| openstackgerrit | chinasubbareddy mallavarapu proposed openstack/openstack-helm master: OSH: Add ingress netpol for ceph-rgw pods https://review.openstack.org/633045 | 19:49 |
| *** sdake_ has quit IRC | 19:54 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add ingress network policy to kube-state-metrics and openstack-exporter https://review.openstack.org/637621 | 20:13 |
| *** lemko has quit IRC | 20:49 | |
| *** dustinspecker has quit IRC | 20:53 | |
| openstackgerrit | Meghan Heisler proposed openstack/openstack-helm-infra master: Add ingress network policy to kube-state-metrics and openstack-exporter https://review.openstack.org/637621 | 21:09 |
| supamatt | kranthikirang: that specific log snippet you posted is specific to network problems | 21:25 |
| kranthikirang | supamatt: that doesn't seem right to me; We have been using those settings from long time and its the same calico CNI and the network settings are correct; I am not sure what to say but I am pretty sure there is something wrong | 21:26 |
| supamatt | kranthikirang: A lossy channel in the logs is network related, so probably a port switch, cable, nic. | 21:27 |
| supamatt | As for the memory leak, there are memory limits set for the mgr pod. 500MB I think. | 21:28 |
| kranthikirang | supamatt: OK; I have been trying this in multiple setups and in all I am seeing the same problem | 21:28 |
| supamatt | You will need to send a ps output of the process when you see it do this. | 21:29 |
| kranthikirang | supamatt: its eating up all the host memory; I don't see its limiting | 21:29 |
| kranthikirang | I can do right away | 21:29 |
| supamatt | ah | 21:29 |
| supamatt | it's because | 21:30 |
| supamatt | resources: | 21:30 |
| supamatt | enabled: false | 21:30 |
| kranthikirang | OK; | 21:30 |
| kranthikirang | Do you want process tree inside the ceph-mgr container? | 21:30 |
| kranthikirang | or in the host | 21:30 |
| kranthikirang | ? | 21:30 |
| supamatt | the host | 21:30 |
| kranthikirang | [root@dsl-compute4 /]# ps -ef | 21:31 |
| kranthikirang | UID PID PPID C STIME TTY TIME CMD | 21:31 |
| kranthikirang | ceph 1 0 0 Feb11 ? 00:16:06 /usr/bin/ceph-mgr --cluster ceph --setuser ceph --setgroup ceph -d -i dsl-compute4 | 21:31 |
| kranthikirang | root 8446 0 0 21:30 pts/0 00:00:00 bash | 21:31 |
| kranthikirang | root 8676 8446 0 21:31 pts/0 00:00:00 ps -ef | 21:31 |
| kranthikirang | [root@dsl-compute4 /]# | 21:31 |
| kranthikirang | oh ok | 21:31 |
| supamatt | you can override that value, and enable it. You may want to do that and then redeploy the ceph-client chart | 21:31 |
| kranthikirang | yeah, that make sense; but what do you think will happen if we limit and still there is a session failure? | 21:32 |
| supamatt | ps aux | grep mgr | 21:32 |
| supamatt | ^ need that | 21:32 |
| supamatt | it oom's the application in the container, and it will restart | 21:32 |
| kranthikirang | root@dsl-compute4:~# ps aux | grep mgr | 21:33 |
| kranthikirang | 167 25188 0.1 1.5 1654116 1036328 ? Ssl Feb11 16:09 /usr/bin/ceph-mgr --cluster ceph --setuser ceph --setgroup ceph -d -i dsl-compute4 | 21:33 |
| kranthikirang | root 25331 0.0 0.0 12944 1024 pts/1 S+ 21:33 0:00 grep --color=auto mgr | 21:33 |
| kranthikirang | root@dsl-compute4:~# | 21:33 |
| kranthikirang | above output is from the host | 21:33 |
| supamatt | Yea go ahead and enable that resource limit | 21:34 |
| supamatt | you can see the example in the values.yaml file for ceph-client chart | 21:34 |
| kranthikirang | Ok; will do that | 21:35 |
| kranthikirang | and post my observation | 21:35 |
| kranthikirang | supmatt: I simply get OOMKilled and k8s restarting the container | 21:39 |
| kranthikirang | supamatt: I simply get OOMKilled and k8s restarting the container | 21:39 |
| supamatt | is the service coming back up? | 21:40 |
| kranthikirang | no | 21:40 |
| kranthikirang | k8s STATUS shows as OOMKilled | 21:40 |
| supamatt | is it just looping OOM? | 21:41 |
| kranthikirang | yes | 21:42 |
| supamatt | the values are not large enough :doh: | 21:42 |
| supamatt | can you double them? | 21:42 |
| supamatt | just the memory ones for the mgr service | 21:42 |
| supamatt | I'll have to review these limits, and likely see if we can put them back on. | 21:43 |
| kranthikirang | ok | 21:43 |
| kranthikirang | Let me do that | 21:43 |
| kranthikirang | I did double the requests to 10Mi and limits to 100Mi for mgr and I see the same behavior | 21:48 |
| kranthikirang | increasing requests to 100Mi and limits to 500Mi worked | 21:54 |
| kranthikirang | supamatt: i see the same logs in active ceph-mgr; If this is a network issue all other would have failed; I have a completed openstack running and VM running including prometheus alerts | 21:54 |
| kranthikirang | supamatt: If not me then someone might have reported if this is a calico or network issue; Only I see issue with ceph-mgr | 21:55 |
| *** witek has quit IRC | 22:23 | |
| *** sdake has joined #openstack-helm | 22:25 | |
| *** sdake has quit IRC | 22:27 | |
| *** howell has quit IRC | 22:28 | |
| *** sdake has joined #openstack-helm | 22:32 | |
| *** sdake has quit IRC | 22:46 | |
| *** sdake has joined #openstack-helm | 22:47 | |
| *** munimeha1 has quit IRC | 23:12 | |
| *** sdake has quit IRC | 23:24 | |
| *** sdake has joined #openstack-helm | 23:25 | |
| openstackgerrit | Matthew Heler proposed openstack/openstack-helm-infra master: [CEPH] Ensure ceph-rbd-pool job runs with MON IPs https://review.openstack.org/637649 | 23:27 |
| *** sdake has quit IRC | 23:32 | |
| *** aaronsheffield has quit IRC | 23:51 | |
| *** kranthikirang has quit IRC | 23:51 | |
| *** spsurya has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!