| *** jascott1 has joined #openstack-kolla | 00:00 | |
| *** jascott1 has quit IRC | 00:04 | |
| *** dardelean has joined #openstack-kolla | 00:08 | |
| *** dardelean has quit IRC | 00:12 | |
| vskilet_ | kfox1111 : http://paste.openstack.org/show/639291/ | 00:15 |
|---|---|---|
| *** vskilet_ has quit IRC | 00:17 | |
| *** vskilet_ has joined #openstack-kolla | 00:17 | |
| vskilet_ | sorry i'm not really reactive ^^ | 00:19 |
| vskilet_ | http://paste.openstack.org/show/639291/ | 00:19 |
| *** goldyfruit has quit IRC | 00:22 | |
| kfox1111 | kubectl logs keystone-6b75955498-jspl2 -c initialize-keystone-logs -n kolla | 00:25 |
| *** mwynne has quit IRC | 00:26 | |
| *** masuberu has joined #openstack-kolla | 00:26 | |
| *** masuberu has quit IRC | 00:28 | |
| *** masuberu has joined #openstack-kolla | 00:28 | |
| *** Masber has quit IRC | 00:29 | |
| *** masuberu has quit IRC | 00:30 | |
| *** masuberu has joined #openstack-kolla | 00:31 | |
| *** mwynne has joined #openstack-kolla | 00:31 | |
| vskilet_ | mkdir: cannot create directory '/var/log/kolla/keystone': File exists | 00:32 |
| *** masuberu has quit IRC | 00:32 | |
| *** masuberu has joined #openstack-kolla | 00:33 | |
| kfox1111 | hmm.... | 00:36 |
| kfox1111 | that wuld be a problem... | 00:37 |
| kfox1111 | kubectl edit the deployment for kube-api, change the mkdir to mkdir -p in the init container and delete the pod. | 00:38 |
| kfox1111 | seems like we need to patch a few other services too. but do check to see if that fixes it. if so we can come up with a more perminant patch. | 00:39 |
| *** dardelean has joined #openstack-kolla | 00:40 | |
| vskilet_ | kfox1111 : so how I can do that ? | 00:48 |
| *** goldyfruit has joined #openstack-kolla | 00:48 | |
| vskilet_ | kfox1111 : docker exec ? | 00:49 |
| kfox1111 | kubectl get deployments -n kolla | 00:49 |
| kfox1111 | kubectl edit deployment <name of keystone deployment> | 00:50 |
| kfox1111 | should drop you in vi so you can edit. | 00:50 |
| kfox1111 | delete the pod like 'kubectl delete pod <podname> -n kolla' | 00:50 |
| vskilet_ | kubectl edit deployment keystone : Error from server (NotFound): deployments.extensions "keystone" not found | 00:51 |
| kfox1111 | -n kolla | 00:52 |
| vskilet_ | http://paste.openstack.org/show/639297/ | 00:52 |
| vskilet_ | ok sorry i should guest | 00:52 |
| vskilet_ | should I redeploy keystone with helm ? | 00:55 |
| kfox1111 | just edit the deployment, add the -p and delete the old pod. | 00:55 |
| kfox1111 | should be able to test that way. | 00:56 |
| vskilet_ | ok so it's run | 00:56 |
| vskilet_ | this should provoc something ? | 00:57 |
| kfox1111 | maybe. | 00:58 |
| *** jascott1 has joined #openstack-kolla | 01:01 | |
| vskilet_ | kfox1111 http://paste.openstack.org/show/639298/ | 01:02 |
| kfox1111 | looks a little better. | 01:04 |
| vskilet_ | :) | 01:05 |
| *** goldyfruit has quit IRC | 01:06 | |
| *** pramodrj07 has quit IRC | 01:07 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fix missing -p to allow pre-existing directory to exist. https://review.openstack.org/531526 | 01:09 |
| kfox1111 | that ps should fix that issue I think. | 01:10 |
| *** ManoX has quit IRC | 01:13 | |
| vskilet_ | ok | 01:13 |
| vskilet_ | why on some pods i can't have logs (kubectl logs) ? | 01:15 |
| kfox1111 | what do you mean? | 01:17 |
| vskilet_ | for example "kubectl logs openvswitch-ovsdb-network-bqp95 -n kolla" return nothing | 01:18 |
| vskilet_ | in fact this comportement stop me in my investigation | 01:20 |
| vskilet_ | no log from kubectl, no log in /var/log/kolla, ... so nothing to read | 01:20 |
| kfox1111 | there should be logs in /var/log/kolla/openvswitch/ovsdb-server.log I think. | 01:21 |
| kfox1111 | well, but your kubectl get pods shows it was crashing in an init. | 01:22 |
| kfox1111 | so try 'kubectl logs openvswitch-ovsdb-network-bqp95 -c initialize-ovs-db -n kolla | 01:23 |
| kfox1111 | if you don't specify the -c <container name> you get the default container for the pod, which might not be the right one to look at. | 01:24 |
| vskilet_ | ok, so I must search for every init script in each pods to have logs error return ? | 01:25 |
| vskilet_ | http://paste.openstack.org/show/639300/ | 01:25 |
| kfox1111 | its like the system partiallly inited, died, and then is having problems recovering. | 01:26 |
| kfox1111 | edit the daemonset for that pod, and add the following right after: | 01:28 |
| kfox1111 | DB=/etc/openvswitch/conf.db; | 01:28 |
| kfox1111 | the new line: | 01:29 |
| kfox1111 | [ -f "$DB" ] || exit 0 | 01:29 |
| kfox1111 | and delete the pod. | 01:29 |
| kfox1111 | I think that will probably fix it and allow things to continue. | 01:29 |
| kfox1111 | as for your other question, the plan is to ship all logs to a central place so they can be much more easily analized. | 01:29 |
| kfox1111 | I have a prototype of fluent-bit, elasticsearch-operator, elasticsearch and kibana working together to do so. | 01:30 |
| vskilet_ | ok | 01:31 |
| vskilet_ | so to edit daemonset i need to do kubectl edit openvswitch-ovsdb-network-bqp95 -n kolla ? | 01:32 |
| *** pengdake has joined #openstack-kolla | 01:32 | |
| kfox1111 | kubectl edit openvswitch-ovsdb-network -n kolla | 01:32 |
| kfox1111 | and kubectl delete pod openvswitch-ovsdb-network-bqp95 -n kolla | 01:32 |
| kfox1111 | gotta head out in a couple minutes | 01:33 |
| vskilet_ | the server doesn't have a resource type "openvswitch-ovsdb-network-bqp95" | 01:34 |
| kfox1111 | 'pod' | 01:35 |
| vskilet_ | ok | 01:35 |
| *** dardelean has quit IRC | 01:37 | |
| vskilet_ | My changes arn't accepted | 01:39 |
| kfox1111 | it wont save, or the pod won't delete? | 01:40 |
| vskilet_ | i can't save my modifications with vi | 01:41 |
| kfox1111 | hmm... it is spaced out to be at the same indent level as the DB=? | 01:41 |
| kfox1111 | also probably needs a ';' at the end of the line. | 01:42 |
| kfox1111 | like: http://paste.openstack.org/ | 01:42 |
| kfox1111 | gotta head out. will be back tomorrow. | 01:43 |
| kfox1111 | good luck. I think your prety close now. | 01:43 |
| vskilet_ | yes thx | 01:45 |
| vskilet_ | mmm nothing in your paste | 01:45 |
| *** vskilet_ has quit IRC | 01:52 | |
| *** dardelean has joined #openstack-kolla | 01:53 | |
| *** dardelean has quit IRC | 01:57 | |
| *** goldyfruit has joined #openstack-kolla | 02:04 | |
| *** pengdake has quit IRC | 02:16 | |
| *** spsurya_ has joined #openstack-kolla | 02:19 | |
| *** karlamrhein_ has joined #openstack-kolla | 02:19 | |
| *** ntpttr_ has joined #openstack-kolla | 02:20 | |
| *** kklimonda_ has joined #openstack-kolla | 02:20 | |
| *** hogepodge_ has joined #openstack-kolla | 02:21 | |
| *** SamYaple_ has joined #openstack-kolla | 02:22 | |
| *** mrostecki1 has joined #openstack-kolla | 02:22 | |
| *** dims_ has joined #openstack-kolla | 02:25 | |
| *** ljjjustin has joined #openstack-kolla | 02:26 | |
| *** wirehead_1 has joined #openstack-kolla | 02:26 | |
| *** goldyfruit has quit IRC | 02:26 | |
| *** spsurya has quit IRC | 02:26 | |
| *** mrostecki has quit IRC | 02:26 | |
| *** dims has quit IRC | 02:26 | |
| *** hogepodge has quit IRC | 02:26 | |
| *** karlamrhein has quit IRC | 02:26 | |
| *** kklimonda has quit IRC | 02:26 | |
| *** wirehead_ has quit IRC | 02:26 | |
| *** ntpttr__ has quit IRC | 02:26 | |
| *** SamYaple has quit IRC | 02:26 | |
| *** spsurya_ is now known as spsurya | 02:26 | |
| *** hogepodge_ is now known as hogepodge | 02:27 | |
| *** karlamrhein_ is now known as karlamrhein | 02:27 | |
| *** kklimonda_ is now known as kklimonda | 02:27 | |
| *** unicell has quit IRC | 02:32 | |
| *** goldyfruit has joined #openstack-kolla | 02:35 | |
| *** harlowja has quit IRC | 02:38 | |
| *** ljjjustin has quit IRC | 02:42 | |
| *** ljjjustin has joined #openstack-kolla | 02:51 | |
| *** ljjjustin has quit IRC | 03:03 | |
| *** unicell has joined #openstack-kolla | 03:05 | |
| *** harlowja has joined #openstack-kolla | 03:10 | |
| *** harlowja has quit IRC | 03:19 | |
| *** M4g1c5t0rM has joined #openstack-kolla | 03:22 | |
| *** dave-mccowan has joined #openstack-kolla | 03:42 | |
| *** dave-mccowan has quit IRC | 04:08 | |
| *** M4g1c5t0rM has quit IRC | 04:40 | |
| *** david-lyle has quit IRC | 04:58 | |
| *** david-lyle has joined #openstack-kolla | 04:59 | |
| *** goldyfruit has quit IRC | 04:59 | |
| *** david-lyle has quit IRC | 05:00 | |
| *** david-lyle has joined #openstack-kolla | 05:01 | |
| *** masuberu has quit IRC | 05:36 | |
| *** pengdake has joined #openstack-kolla | 05:53 | |
| *** masuberu has joined #openstack-kolla | 06:24 | |
| *** harlowja has joined #openstack-kolla | 06:26 | |
| *** masuberu has quit IRC | 06:30 | |
| *** zhouya has quit IRC | 06:36 | |
| *** unicell has quit IRC | 07:07 | |
| *** unicell has joined #openstack-kolla | 07:07 | |
| *** unicell1 has joined #openstack-kolla | 07:18 | |
| *** unicell has quit IRC | 07:19 | |
| *** kbaegis has joined #openstack-kolla | 07:24 | |
| *** kbaegis has joined #openstack-kolla | 07:24 | |
| *** kbaegis has joined #openstack-kolla | 07:25 | |
| *** Masber has joined #openstack-kolla | 07:45 | |
| *** harlowja has quit IRC | 08:00 | |
| *** pengdake_ has joined #openstack-kolla | 08:23 | |
| *** unicell has joined #openstack-kolla | 08:25 | |
| *** unicell1 has quit IRC | 08:25 | |
| *** pengdake has quit IRC | 08:26 | |
| *** mgoddard has joined #openstack-kolla | 08:38 | |
| *** shardy_afk has quit IRC | 08:43 | |
| *** jascott1 has quit IRC | 09:06 | |
| *** ipsecguy_ has quit IRC | 09:08 | |
| *** ipsecguy has joined #openstack-kolla | 09:08 | |
| *** numans has quit IRC | 09:15 | |
| *** numans has joined #openstack-kolla | 09:16 | |
| *** numans has quit IRC | 09:20 | |
| *** dardelean has joined #openstack-kolla | 09:25 | |
| *** numans has joined #openstack-kolla | 09:25 | |
| *** dardelean has quit IRC | 09:29 | |
| *** mgoddard has quit IRC | 10:09 | |
| *** pbourke has quit IRC | 10:11 | |
| *** pbourke has joined #openstack-kolla | 10:12 | |
| *** vskilet_ has joined #openstack-kolla | 10:30 | |
| *** dardelean has joined #openstack-kolla | 10:36 | |
| vskilet_ | kfox1111 : when I want to modified some pods there this message when I save with vim : | 10:44 |
| vskilet_ | [ -f "$DB" ] || exit 0 | 10:44 |
| vskilet_ | euh sorry this : error: pods "openvswitch-ovsdb-network-bqp95" is invalid | 10:44 |
| *** genek has quit IRC | 10:47 | |
| *** genek has joined #openstack-kolla | 10:49 | |
| *** vskilet__ has joined #openstack-kolla | 10:51 | |
| *** vskilet has joined #openstack-kolla | 10:52 | |
| *** vskilet__ has quit IRC | 10:52 | |
| *** vskilet_ has quit IRC | 10:53 | |
| *** livelace-link has quit IRC | 11:05 | |
| *** dciabrin_ has joined #openstack-kolla | 11:06 | |
| *** livelace-link has joined #openstack-kolla | 11:07 | |
| *** dardelean has quit IRC | 11:07 | |
| *** dardelean has joined #openstack-kolla | 11:08 | |
| *** dciabrin has quit IRC | 11:08 | |
| *** pengdake_ has quit IRC | 11:10 | |
| *** dardelean has quit IRC | 11:12 | |
| *** vskilet has quit IRC | 11:52 | |
| *** vskilet has joined #openstack-kolla | 11:53 | |
| *** vskilet has quit IRC | 11:57 | |
| *** vskilet has joined #openstack-kolla | 11:58 | |
| *** numans has quit IRC | 12:00 | |
| *** numans has joined #openstack-kolla | 12:00 | |
| *** vskilet has quit IRC | 12:01 | |
| *** vskilet has joined #openstack-kolla | 12:01 | |
| *** numans has quit IRC | 12:04 | |
| *** numans has joined #openstack-kolla | 12:11 | |
| *** vskilet has quit IRC | 12:19 | |
| *** vskilet has joined #openstack-kolla | 12:21 | |
| *** vskilet has quit IRC | 12:50 | |
| *** vskilet has joined #openstack-kolla | 12:51 | |
| *** vskilet has quit IRC | 13:01 | |
| *** vskilet has joined #openstack-kolla | 13:02 | |
| *** vskilet has quit IRC | 13:10 | |
| *** vskilet has joined #openstack-kolla | 13:10 | |
| *** dave-mccowan has joined #openstack-kolla | 13:14 | |
| *** wojdec has joined #openstack-kolla | 13:19 | |
| *** dave-mcc_ has joined #openstack-kolla | 13:23 | |
| *** dave-mccowan has quit IRC | 13:26 | |
| *** dciabrin__ has joined #openstack-kolla | 13:29 | |
| *** dciabrin_ has quit IRC | 13:31 | |
| *** dave-mcc_ has quit IRC | 14:02 | |
| *** vskilet has quit IRC | 14:14 | |
| *** wojdec has quit IRC | 14:19 | |
| *** Masber has quit IRC | 14:33 | |
| *** rhallisey has quit IRC | 14:47 | |
| *** dardelean has joined #openstack-kolla | 15:00 | |
| *** jascott1 has joined #openstack-kolla | 15:03 | |
| *** jascott1 has quit IRC | 15:08 | |
| *** vskilet has joined #openstack-kolla | 15:17 | |
| *** dardelean has quit IRC | 15:43 | |
| *** vskilet has quit IRC | 15:45 | |
| *** vskilet has joined #openstack-kolla | 15:50 | |
| *** vskilet has quit IRC | 15:59 | |
| *** vskilet has joined #openstack-kolla | 16:00 | |
| *** dardelean has joined #openstack-kolla | 16:04 | |
| *** dave-mccowan has joined #openstack-kolla | 16:08 | |
| *** dardelean has quit IRC | 16:17 | |
| *** dave-mccowan has quit IRC | 16:17 | |
| *** dardelean has joined #openstack-kolla | 16:17 | |
| *** dave-mccowan has joined #openstack-kolla | 16:18 | |
| *** dave-mcc_ has joined #openstack-kolla | 16:22 | |
| *** dardelean has quit IRC | 16:22 | |
| *** dave-mccowan has quit IRC | 16:24 | |
| *** kbaegis has joined #openstack-kolla | 16:27 | |
| *** kbaegis has joined #openstack-kolla | 16:28 | |
| *** vskilet has quit IRC | 16:29 | |
| *** vskilet has joined #openstack-kolla | 16:29 | |
| *** vskilet has quit IRC | 16:43 | |
| *** dave-mcc_ has quit IRC | 16:43 | |
| *** vskilet has joined #openstack-kolla | 16:46 | |
| *** zzzeek has quit IRC | 16:50 | |
| *** zzzeek has joined #openstack-kolla | 16:51 | |
| *** zzzeek has quit IRC | 16:53 | |
| *** zzzeek has joined #openstack-kolla | 16:55 | |
| *** kbaegis has quit IRC | 17:41 | |
| *** hrw has quit IRC | 17:47 | |
| *** mgoddard has joined #openstack-kolla | 18:03 | |
| *** EmilienM_ has joined #openstack-kolla | 18:23 | |
| *** EmilienM_ is now known as EmilienM | 18:28 | |
| *** EmilienM has joined #openstack-kolla | 18:28 | |
| *** vskilet has quit IRC | 18:31 | |
| *** vskilet has joined #openstack-kolla | 18:53 | |
| *** kbaegis has joined #openstack-kolla | 19:19 | |
| *** mgoddard has quit IRC | 19:23 | |
| *** numans has quit IRC | 19:34 | |
| *** numans has joined #openstack-kolla | 19:39 | |
| *** EmilienM has quit IRC | 20:10 | |
| *** vskilet has quit IRC | 20:12 | |
| *** EmilienM has joined #openstack-kolla | 20:13 | |
| *** EmilienM has quit IRC | 20:13 | |
| *** EmilienM has joined #openstack-kolla | 20:13 | |
| *** vskilet has joined #openstack-kolla | 20:15 | |
| *** jascott1 has joined #openstack-kolla | 21:04 | |
| *** jascott1 has quit IRC | 21:08 | |
| *** hrw has joined #openstack-kolla | 21:22 | |
| *** vskilet has quit IRC | 22:09 | |
| *** vskilet has joined #openstack-kolla | 22:22 | |
| *** vskilet has quit IRC | 22:41 | |
| *** igordc has quit IRC | 22:42 | |
| *** unicell has quit IRC | 23:18 | |
| *** dardelean has joined #openstack-kolla | 23:19 | |
| *** dardelean has quit IRC | 23:23 | |
| *** dardelean has joined #openstack-kolla | 23:33 | |
| *** dardelean has quit IRC | 23:38 | |
| *** tvignaud has quit IRC | 23:40 | |
| *** tvignaud has joined #openstack-kolla | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!