| *** igordc has quit IRC | 02:17 | |
| *** EricRen has joined #kata-dev | 02:25 | |
| *** EricRen has quit IRC | 04:46 | |
| *** EricRen has joined #kata-dev | 05:20 | |
| *** EricRen has quit IRC | 05:24 | |
| *** EricRen has joined #kata-dev | 06:16 | |
| *** dklyle has quit IRC | 07:15 | |
| *** dklyle has joined #kata-dev | 07:16 | |
| *** lpetrut has joined #kata-dev | 07:20 | |
| *** sameo has joined #kata-dev | 07:22 | |
| kata-irc-bot | <xwlpt> Is there anyway to check the agent’s log on the host? | 07:32 |
|---|---|---|
| kata-irc-bot | <xwlpt> @graham.whaley @archana.m.shinde | 07:38 |
| *** EricRen has quit IRC | 07:52 | |
| *** EricRen has joined #kata-dev | 07:54 | |
| *** sgarzare has joined #kata-dev | 08:14 | |
| *** pcaruana has joined #kata-dev | 08:29 | |
| *** davidgiluk has joined #kata-dev | 09:04 | |
| *** gwhaley has joined #kata-dev | 09:11 | |
| kata-irc-bot | <graham.whaley> @xwlpt - I think so - if you enable agent debug (in the config file) and also proxy or shim debug (depending if you are using vsock or not), then I think you should get the agent logs mixed in with the proxy or shim log - I think..... | 09:19 |
| kata-irc-bot | <graham.whaley> so, I will ask you a favour @xwlpt - if you work it out, and it is not in the documentation already, maybe you can make a PR to add it? :slightly_smiling_face: | 09:20 |
| kata-irc-bot | <xwlpt> I am using shimv2, do I still need to enable shim debug? | 09:20 |
| kata-irc-bot | <xwlpt> Let me have a try, if it works, I will have a PR for it. | 09:21 |
| kata-irc-bot | <xu> I think, with shim-v2, the logs could be feed to containerd's log, right? | 09:21 |
| kata-irc-bot | <xwlpt> @xu Not found any agent log from containerd’s log | 09:22 |
| kata-irc-bot | <xwlpt> Maybe I am missing something | 09:23 |
| kata-irc-bot | <graham.whaley> @xu - do you mean they *could* be passed to containerd logs, or they *are* passed to containerd logs? :slightly_smiling_face: | 09:23 |
| kata-irc-bot | <xu> @graham.whaley let me check :,) | 09:25 |
| *** pcaruana has quit IRC | 09:30 | |
| kata-irc-bot | <xwlpt> Not works for me :crying_cat_face: | 09:35 |
| kata-irc-bot | <fupan> you should enabled it by add “agent.log=debug ” to kernel’s boot parameter in kata config | 09:39 |
| kata-irc-bot | <xwlpt> kernel_params = ” agent.log=debug,initcall_debug ” | 09:39 |
| kata-irc-bot | <xwlpt> I add these two | 09:40 |
| kata-irc-bot | <fupan> and then set enable_debug = true in runtime section | 09:40 |
| kata-irc-bot | <fupan> oh, I remember, the containerd’s debug setting will overrides the kata’s config setting, you should make your containerd enabled debug | 09:42 |
| kata-irc-bot | <xu> @fupan I think we should have a document on this | 09:47 |
| kata-irc-bot | <fupan> yeah, I will send a PR to address this issue | 09:48 |
| kata-irc-bot | <xu> cool, thanks @fupan | 09:48 |
| kata-irc-bot | <graham.whaley> :thumbsup: | 09:55 |
| kata-irc-bot | <xwlpt> Feb 18 03:04:23 tess-node-sbtxw-3867512 kata[10883]: time=“2020-02-18T03:04:23.210290459-07:00” level=error msg=“Failed to read agent logs” ID=41be98283d17320282031ae7aa43c01c4079c36aea99302c3a3c3db2c7f14ec0 console-protocol=unix console-socket=/run/vc/vm/41be98283d17320282031ae7aa43c01c4079c36aea99302c3a3c3db2c7f14ec0/console.sock error=“read unix @->/run/vc/vm/41be98283d17320282031ae7aa43c01c4079c36aea99302c3a3c3db2c7f14ec0/con | 10:06 |
| kata-irc-bot | use of closed network connection” sandbox=41be98283d17320282031ae7aa43c01c4079c36aea99302c3a3c3db2c7f14ec0 source=virtcontainers subsystem=kata_agent | 10:06 |
| kata-irc-bot | <xwlpt> I get this error | 10:06 |
| kata-irc-bot | <xwlpt> > oh, I remember, the containerd’s debug setting will overrides the kata’s config setting, you should make your containerd enabled debug | 10:07 |
| kata-irc-bot | <xwlpt> What you mean is: ```[plugins.linux] shim_debug = true``` | 10:08 |
| kata-irc-bot | <xwlpt> @fupan | 10:08 |
| kata-irc-bot | <xwlpt> If I add agent.debug_console to the boot param, there is no such kind of error. But I still can’t found any agent logs. | 10:09 |
| kata-irc-bot | <fupan> what hypervisor did you use? qemu or firecracker? | 10:10 |
| kata-irc-bot | <xwlpt> qemu | 10:10 |
| kata-irc-bot | <fupan> that’s weird , it seems shimv2 try to fetch the agent log from /run/vc/vm/41be98283d17320282031ae7aa43c01c4079c36aea99302c3a3c3db2c7f14ec0/console.sock, but it closed. | 10:13 |
| kata-irc-bot | <xwlpt> @fupan Can you have a try whether this works in your env? If it works, I will check from my side. Thanks | 10:14 |
| kata-irc-bot | <fupan> yes, it worked from my side | 10:18 |
| kata-irc-bot | <fupan> sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=virtcontainers subsystem=kata_agent vmconsole=“time=\“2020-02-18T10:18:17.663412385Z\” level=debug msg=\“new request\” debug_console=false name=kata-agent pid=41 req= request=/grpc.Health/Check sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=agent” time=“2020-02-18T18:18:17.675113410+08:00” level=debug msg=“read | 10:18 |
| kata-irc-bot | ID=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=virtcontainers subsystem=kata_agent vmconsole=“time=\“2020-02-18T10:18:17.663486439Z\” level=debug msg=\“request end\” debug_console=false duration=533ns name=kata-agent pid=41 request=/grpc.Health/Check resp=\“status:SERVING \” sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6c | 10:18 |
| kata-irc-bot | source=agent” time=“2020-02-18T18:18:18.674550051+08:00” level=info msg=“{\“execute\“:\“query-status\“}” ID=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=virtcontainers subsystem=qmp time=“2020-02-18T18:18:18.674721155+08:00" level=info msg=“{\“return\“: {\“status\“: \“running\“, \“singlestep\“: false, \“running\“: true}}” ID=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6cc | 10:18 |
| kata-irc-bot | subsystem=qmp time=“2020-02-18T18:18:18.674767336+08:00" level=debug msg=“sending request” ID=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 name=grpc.CheckRequest req= source=virtcontainers subsystem=kata_agent time=“2020-02-18T18:18:18.675051391+08:00" level=debug msg=“reading guest console” ID=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 | 10:18 |
| kata-irc-bot | sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=virtcontainers subsystem=kata_agent vmconsole=“time=\“2020-02-18T10:18:18.66339434Z\” level=debug msg=\“new request\” debug_console=false name=kata-agent pid=41 req= request=/grpc.Health/Check sandbox=b36d2ca71b45de8a493dc44321c3f4bd95463e440e00565d9b6ccb07d9a2c069 source=agent” | 10:18 |
| kata-irc-bot | <fupan> the source=agent lines are the logs from the agent | 10:19 |
| kata-irc-bot | <xwlpt> Thanks | 10:21 |
| kata-irc-bot | <fupan> did you do any change to the kernel boot parameters except adding agent.log=debug? | 10:22 |
| kata-irc-bot | <xwlpt> Not any other changes. | 10:23 |
| kata-irc-bot | <xwlpt> I found that there is no level=debug info in the logs. | 10:23 |
| kata-irc-bot | <xwlpt> I need to enable this? ```[plugins.linux] shim_debug = true``` right? | 10:24 |
| kata-irc-bot | <xwlpt> I see that our containerd’s log is info but not debug. | 10:25 |
| kata-irc-bot | <xwlpt> Let me change it. | 10:25 |
| *** pcaruana has joined #kata-dev | 10:28 | |
| *** pcaruana has quit IRC | 10:45 | |
| kata-irc-bot | <fupan> hmm, yes, you should make the three option enabled: containerd debug enabled, kata runtime debug enabled and passed agent.log=debug to guest kernel boot parameters. | 10:48 |
| kata-irc-bot | <fupan> I think you can enable containerd debug by ```[debug] level = "debug"``` | 10:49 |
| kata-irc-bot | <fupan> please see https://github.com/containerd/containerd/blob/master/docs/getting-started.md | 10:49 |
| *** pcaruana has joined #kata-dev | 10:55 | |
| *** EricRen has quit IRC | 11:25 | |
| *** EricRen has joined #kata-dev | 11:33 | |
| kata-irc-bot | <xwlpt> Seems it not works for kata shim v2. | 11:40 |
| kata-irc-bot | <xwlpt> I can see the docker shim has -debug parameter | 11:40 |
| kata-irc-bot | <xwlpt> But kata shim v2 doesnot has it. | 11:40 |
| kata-irc-bot | <xwlpt> root 12453 1 0 03:21 ? 00:00:06 /var/usrlocal/bin/containerd-shim-kata-v2 -namespace k8s.io -address /var/containerd/containerd.sock -publish-binary /var/usrlocal/bin/containerd -id d414cb4b17bb8edf675f3e67b363bf07596bc08cd060ef93d3a0b7bbae191d5c root 13919 1 0 Feb12 ? 00:00:37 containerd-shim -namespace k8s.io -workdir | 11:41 |
| kata-irc-bot | /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/10e6b6f25b2e9508ec86ea9ba641107b27278d82db274e6ea8eaa5955dc3c354 -address /var/containerd/containerd.sock -containerd-binary /var/usrlocal/bin/containerd -debug | 11:41 |
| *** gwhaley has quit IRC | 12:02 | |
| *** EricRen has quit IRC | 13:01 | |
| *** gwhaley has joined #kata-dev | 13:15 | |
| *** fuentess has joined #kata-dev | 13:21 | |
| *** canyounot has joined #kata-dev | 13:35 | |
| *** EricRen has joined #kata-dev | 13:35 | |
| *** sameo has quit IRC | 13:55 | |
| *** igordc has joined #kata-dev | 14:04 | |
| *** EricRen has quit IRC | 14:06 | |
| *** pcaruana has quit IRC | 14:14 | |
| *** devimc has joined #kata-dev | 14:19 | |
| *** sameo has joined #kata-dev | 14:24 | |
| *** igordc has quit IRC | 15:00 | |
| *** pcaruana has joined #kata-dev | 15:03 | |
| *** lpetrut has quit IRC | 15:11 | |
| *** pcaruana has quit IRC | 15:53 | |
| *** pcaruana has joined #kata-dev | 15:55 | |
| *** pcaruana has quit IRC | 15:58 | |
| *** pcaruana has joined #kata-dev | 16:00 | |
| fidencio | davidgiluk: devimc: What would be the best way to approach https://github.com/kata-containers/runtime/issues/2464? | 16:27 |
| fidencio | davidgiluk: devimc: the options currently passed do not make sense for upstream virtiofs. we could just leave everything empty | 16:27 |
| fidencio | davidgiluk: devimc: my guess is the way to approach that would be "leave everything empty by default", and provide options or users to just override that from the configuration file | 16:28 |
| fidencio | but how hard would that break existing deployments? | 16:28 |
| davidgiluk | fidencio: or should it be a flag day | 16:29 |
| fidencio | davidgiluk: flag as in for compiling those options off? | 16:31 |
| davidgiluk | fidencio: No, I mean a commit where we flip everything | 16:32 |
| fidencio | davidgiluk: that would be ideal | 16:32 |
| fidencio | davidgiluk: I'm just wondering whether we really want to flip everything | 16:32 |
| fidencio | davidgiluk: let me re-phrase, does make sense to offer the users options to set any of those options? | 16:33 |
| davidgiluk | fidencio: It does to me | 16:34 |
| fidencio | davidgiluk: okay, cool, so I'll (try to) do this in two steps: 1) Add those new options in the config file; 2) Remove the current flags | 16:35 |
| fidencio | davidgiluk: and we can (try to) have those plus the other virtiofs patches merged at once | 16:35 |
| fidencio | possibly after the 1.11 release? | 16:35 |
| devimc | fidencio, currently there is an option in the conf file to pass extra option, I think you should deprecated it, and allow users have full control over the virtiofsd option | 16:36 |
| devimc | *options | 16:36 |
| *** pcaruana has quit IRC | 16:36 | |
| devimc | fidencio, but there are some lazy users (like me) that may not want to investigate all virtiofsd flags | 16:37 |
| devimc | fidencio, so, we should at least use a default virtiofsd flags, just in case the user doesn't want to deal with that | 16:38 |
| fidencio | devimc: davidgiluk can correct if I'm mistaken, but not using any options may be the default one | 16:38 |
| fidencio | I mean, may be good enough for the majority of the cases | 16:38 |
| devimc | fidencio, yes | 16:39 |
| devimc | virtio_fs_extra_args is just to appends extra options | 16:39 |
| devimc | not to overwrite default options | 16:40 |
| devimc | maybe we should deprecate that option and add a new one (virtiofs_args) | 16:41 |
| fidencio | devimc: let's take it in parts. I totally agree with deprecating that option and adding a new one | 16:42 |
| devimc | if virtiofs_args == "" then use default args (args defined by kata) | 16:42 |
| fidencio | devimc: what I said that may be good enough for the majority of the cases is not having any of the sharedDirVirtioFS*Options set by default | 16:42 |
| devimc | if virtiofs_args == " " then don't use args | 16:43 |
| devimc | because " " overrides default args | 16:43 |
| fidencio | devimc: aha! | 16:44 |
| devimc | fidencio, why? | 16:44 |
| fidencio | devimc: I didn't know about the difference of "" and " " | 16:44 |
| devimc | "" is empty (nothing) | 16:45 |
| * davidgiluk notes he's out next week (check mail occasionally) | 16:45 | |
| devimc | " " is a space (something that overwrite default args) | 16:45 |
| fidencio | devimc: sure, it does make sense | 16:45 |
| devimc | fidencio, IMO, we should at least define a list of default args | 16:46 |
| fidencio | devimc: okay, I'll work on this in parts then: 1) deprecate virtio_fs_extra_args; 2) add virtio_fs_args (documenting what's already set as default); | 16:47 |
| devimc | fidencio, yes | 16:47 |
| fidencio | devimc: one thing that bothers me about the current options is that they do not work with upstream stuff | 16:48 |
| devimc | fidencio, but maybe you want to name it virtiofs_args | 16:48 |
| devimc | hah | 16:48 |
| fidencio | devimc: and that's the reason of my bug report in the first place :-) | 16:48 |
| fidencio | or request for enhancement | 16:48 |
| devimc | yes, that make sense, the same thing happen with qemu, we have to update QMP and command line option to support the latest version | 16:49 |
| *** devimc has quit IRC | 16:52 | |
| *** devimc has joined #kata-dev | 16:52 | |
| kata-irc-bot | <xu> @eric.ernst Just discussed with @bergwolf, we could write a series of issues as proposals for 2.0. Then we may discuss them in the following AC meetings. | 16:54 |
| kata-irc-bot | <eric.ernst> +1 | 16:55 |
| kata-irc-bot | <eric.ernst> Sounds great. | 16:55 |
| kata-irc-bot | <eric.ernst> Looking forward to these -- let me know if you need help on this. | 16:55 |
| kata-irc-bot | <eric.ernst> in particular. | 16:55 |
| *** eernst has joined #kata-dev | 17:02 | |
| *** ailan has joined #kata-dev | 17:06 | |
| *** eernst has quit IRC | 17:07 | |
| *** igordc has joined #kata-dev | 17:09 | |
| *** sgarzare has quit IRC | 17:33 | |
| *** ailan has quit IRC | 18:10 | |
| *** eernst has joined #kata-dev | 18:44 | |
| *** sameo has quit IRC | 18:55 | |
| *** devimc has quit IRC | 19:16 | |
| *** devimc has joined #kata-dev | 19:35 | |
| *** eernst has quit IRC | 19:48 | |
| *** eernst has joined #kata-dev | 20:05 | |
| *** davidgiluk has quit IRC | 20:06 | |
| *** gwhaley has quit IRC | 20:08 | |
| *** eernst has quit IRC | 20:09 | |
| *** jugs has quit IRC | 20:11 | |
| *** jugs has joined #kata-dev | 20:13 | |
| *** canyounot has quit IRC | 21:49 | |
| *** fuentess has quit IRC | 22:08 | |
| *** devimc has quit IRC | 23:07 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!