| *** yamamoto has joined #openstack-helm | 03:07 | |
| *** yamamoto has quit IRC | 03:17 | |
| *** yamamoto has joined #openstack-helm | 03:20 | |
| *** hachi_ has joined #openstack-helm | 04:26 | |
| *** yamamoto has quit IRC | 04:37 | |
| *** gouthamr has joined #openstack-helm | 04:44 | |
| *** gouthamr has quit IRC | 04:44 | |
| *** gouthamr has joined #openstack-helm | 04:45 | |
| *** hachi_ has quit IRC | 05:32 | |
| *** yamamoto has joined #openstack-helm | 05:37 | |
| *** yamamoto has quit IRC | 05:46 | |
| *** gouthamr has quit IRC | 05:51 | |
| openstackgerrit | Merged openstack/openstack-helm master: openvswitch: wait for ovs socket readiness https://review.openstack.org/511976 | 06:10 |
|---|---|---|
| *** Sean-md has joined #openstack-helm | 08:02 | |
| Sean-md | Hi all, I followed the multinode installation guide (http://openstack-helm.readthedocs.io/en/latest/install/multinode.html) | 08:25 |
| Sean-md | but, I can't deploy ceph using "helm install..." | 08:26 |
| Sean-md | Some Pods got errors like this "MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found" | 08:26 |
| Sean-md | Full logs is "http://paste.openstack.org/show/623614/" | 08:26 |
| Sean-md | Do I need preparation for ceph? like this "https://docs.openstack.org/kolla-ansible/latest/reference/ceph-guide.html#preparation" | 08:33 |
| Sean-md | For example, adding a additional disk, labeling to the disk... and so on | 08:36 |
| Sean-md | I labeled nodes like this | 08:42 |
| Sean-md | kubectl label node helm00 openstack-control-plane=enabled, kubectl label node helm01 openstack-compute-node=enabled, kubectl label node helm02 openvswitch=enabled, kubectl label node helm03 ceph-mon=enabled, kubectl label node helm04 ceph-osd=enabled, kubectl label node helm05 ceph-mds=enabled, kubectl label node helm06 ceph-rgw=enabled | 08:43 |
| *** yamamoto has joined #openstack-helm | 11:24 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-helm | 12:32 | |
| *** yamamoto has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-helm | 13:44 | |
| *** gouthamr has joined #openstack-helm | 13:57 | |
| mattmceuen | Hey @Sean-md , can you try `kubectl -n ceph get secrets`, and also describe any that seem to be having issues? | 14:18 |
| *** yamamoto has quit IRC | 14:26 | |
| *** yamamoto has joined #openstack-helm | 14:26 | |
| *** yamamoto has joined #openstack-helm | 15:28 | |
| *** yamamoto has quit IRC | 15:38 | |
| anticw | Sean-md: or: kubectl get secrets --all-namespaces | grep ceph | 15:56 |
| anticw | in the off chance they ended up somewhere else | 15:56 |
| *** Sean-mdt has joined #openstack-helm | 15:56 | |
| portdirect | could you run kubectl describe on ceph-mds-keyring-generator-gjl59 | 15:58 |
| portdirect | did you run: kubectl label nodes openstack-control-plane=enabled --all ? | 15:59 |
| portdirect | looks like the reason the keys cannot be found is that they have not been generated | 16:00 |
| Sean-mdt | mattmceuen, the result is http://paste.openstack.org/show/623632/ | 16:17 |
| Sean-mdt | I don't know if there is a problem or not.. | 16:18 |
| Sean-mdt | anticw, result is same as "kubectl -n ceph get secrets" | 16:18 |
| Sean-mdt | portdirect, I can't enter the pod using "kubectl exec..." because a main container doesn't run. | 16:18 |
| portdirect | Sean-mdt: you shouldnt need to exec into the container | 16:19 |
| Sean-mdt | I labeled to each node, not to all, like this.."kubectl label node helm00 openstack-control-plane=enabled" | 16:19 |
| portdirect | but can you run `kubectl describe -n ceph ceph-mds-keyring-generator-gjl59` | 16:19 |
| portdirect | it being stuck in pending means that the secrets have never been generated | 16:20 |
| Sean-mdt | ok let me check.. | 16:20 |
| Sean-mdt | Warning FailedScheduling 1s (x1252 over 6h) default-scheduler No nodes are available that match all of the following predicates:: MatchNodeSelector (6), PodToleratesNodeTaints (1). | 16:22 |
| portdirect | how many nodes do you have? | 16:23 |
| Sean-mdt | http://paste.openstack.org/show/623634/ | 16:23 |
| Sean-mdt | 6 | 16:24 |
| portdirect | can you dump `kubectl describe nodes` into a paste | 16:24 |
| Sean-mdt | hold on please... | 16:26 |
| Sean-mdt | http://paste.openstack.org/show/623635/ | 16:30 |
| Sean-mdt | sorry it's 7 not 6 | 16:30 |
| portdirect | ah - so only k8s master has openstack-control-plane applied | 16:32 |
| portdirect | do you mind hosting pods on that node? | 16:32 |
| Sean-mdt | my labels | 16:32 |
| Sean-mdt | http://paste.openstack.org/show/623636/ | 16:32 |
| *** lukepatrick has joined #openstack-helm | 16:32 | |
| portdirect | can you run `kubectl taint nodes --all node-role.kubernetes.io/master-` | 16:33 |
| Sean-mdt | Yes wait please.. | 16:34 |
| portdirect | you have `Taints: node-role.kubernetes.io/master:NoSchedule` | 16:34 |
| portdirect | on the node that you are trying to run the jobs on | 16:34 |
| portdirect | so if we remove this they should be able to come up | 16:35 |
| portdirect | regarding the other labels - are you sure thats what you want? | 16:35 |
| Sean-mdt | I see, | 16:36 |
| portdirect | for nova/neutron to work, you will need to have openvswitch enabled on helm01 | 16:36 |
| Sean-mdt | Thank you so much.. | 16:36 |
| portdirect | and you probably want more than a single osd | 16:36 |
| portdirect | that work? | 16:36 |
| Sean-mdt | yes I want multi osd | 16:37 |
| Sean-mdt | you mean I shouldn't run any pods on master except for kube-system pods | 16:38 |
| portdirect | thats up to you | 16:38 |
| portdirect | i do it the whole time - but it depends on your setup and needs | 16:39 |
| Sean-mdt | I understood | 16:39 |
| portdirect | honestly if your just kicking the tyres -id recommend labeling all the nodes like in the multinode guide | 16:40 |
| portdirect | but if not you may find somthing like this works for you | 16:41 |
| portdirect | https://www.irccloud.com/pastebin/PXfam8Qd/ | 16:41 |
| portdirect | lol - 2 sec | 16:41 |
| portdirect | https://www.irccloud.com/pastebin/C964GqNU/ | 16:41 |
| portdirect | which would make nodes 0 and 6 run the control plane, 3,4,5 run ceph leaving 1 & 2 to be compute hosts | 16:42 |
| Sean-mdt | Thank you, I will re-label nodes as you recommend | 16:45 |
| portdirect | just a suggestion ;) | 16:45 |
| portdirect | but did removing that taint work for you? | 16:45 |
| Sean-mdt | pending pods disappear, but the rest pods remain "init" | 16:48 |
| Sean-mdt | Warning FailedMount 10m (x185 over 6h) kubelet, helm03 Unable to mount volumes for pod "ceph-mon-1dvn5_ceph(16710e29-b0c7-11e7-9dca-5254009a2265)": timeout expired waiting for volumes to attach/mount for pod "ceph"/"ceph-mon-1dvn5". list of unattached/unmounted volumes=[ceph-client-admin-keyring ceph-mon-keyring ceph-bootstrap-osd-keyring ceph-bootstrap-mds-keyring ceph-bootstrap-rgw-keyring] | 16:48 |
| Sean-mdt | should I reinstall using helm? | 16:49 |
| portdirect | give it some time | 16:49 |
| portdirect | they will have backed off loads over the last 6 hours | 16:49 |
| Sean-mdt | ok, Thanks :) | 16:49 |
| portdirect | I'd wait 10 minutes and see what things look like then | 16:50 |
| portdirect | I'll still be around | 16:50 |
| Sean-mdt | I see.. | 16:50 |
| Sean-mdt | can I designate a specific hdd as an osd? | 16:52 |
| portdirect | not yet | 16:52 |
| portdirect | but very soon :) | 16:52 |
| Sean-mdt | I hope.. :) | 16:53 |
| portdirect | renmak on here is doing great work along with davekorman on that right now | 16:54 |
| Sean-mdt | I am grateful to all of you | 16:56 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Bash: remove tabs in openvswitch script https://review.openstack.org/512028 | 16:59 |
| *** gouthamr has quit IRC | 18:10 | |
| *** gouthamr has joined #openstack-helm | 19:56 | |
| openstackgerrit | Merged openstack/openstack-helm master: Added resolv.conf example to multinode doc https://review.openstack.org/511378 | 20:56 |
| openstackgerrit | Merged openstack/openstack-helm master: Bash: remove tabs in openvswitch script https://review.openstack.org/512028 | 21:37 |
| openstackgerrit | Merged openstack/openstack-helm master: Fix port reference name https://review.openstack.org/511947 | 21:41 |
| *** lukepatrick has quit IRC | 22:13 | |
| openstackgerrit | Merged openstack/openstack-helm master: Nova: Use endpoint lookups for NoVNC proxy https://review.openstack.org/511706 | 22:15 |
| openstackgerrit | Merged openstack/openstack-helm master: Specification: OSH logging combining fluentbit-fluentd https://review.openstack.org/505491 | 22:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!