| *** igordc has quit IRC | 01:06 | |
| kata-irc-bot | <manohar.r.castelino> FYI: https://gist.github.com/mcastelino/9b9634226cb732474a4535391eb32cfe | 01:10 |
|---|---|---|
| kata-irc-bot | <manohar.r.castelino> | 01:10 |
| *** igordc has joined #kata-dev | 02:40 | |
| *** auk has quit IRC | 04:00 | |
| *** igordc has quit IRC | 05:11 | |
| *** sameo has joined #kata-dev | 05:28 | |
| *** auk has joined #kata-dev | 05:30 | |
| *** auk has quit IRC | 05:35 | |
| *** sameo has quit IRC | 06:47 | |
| *** sgarzare has joined #kata-dev | 06:58 | |
| *** EricRen has joined #kata-dev | 07:27 | |
| *** sameo has joined #kata-dev | 07:29 | |
| *** tmhoang has joined #kata-dev | 07:39 | |
| *** sgarzare has quit IRC | 07:45 | |
| *** sgarzare has joined #kata-dev | 07:48 | |
| *** davidgiluk has joined #kata-dev | 08:01 | |
| *** gwhaley has joined #kata-dev | 08:18 | |
| *** EricRen has quit IRC | 08:27 | |
| *** EricRen has joined #kata-dev | 08:27 | |
| *** tmhoang_ has joined #kata-dev | 08:37 | |
| *** tmhoang has quit IRC | 08:39 | |
| *** EricRen has quit IRC | 09:01 | |
| *** tmhoang_ is now known as tmhoang | 09:02 | |
| kata-irc-bot | <graham.whaley> hi @yan632 - depends which sort of benchmarks you ran :slightly_smiling_face: The most prominent thing you will normally notice is that (by default in many systems), file access will be slower as it goes over the 9pfs filesystem out to the host. That can be mitigated by using block based graph drivers, like devicemapper, and there are some other soon-to-land items (virtio-fs, lvm snapshotter) that sould mitigate this as well... | 09:06 |
| kata-irc-bot | and then | 09:06 |
| kata-irc-bot | <graham.whaley> that 9pfs can affect other benchmarks that you might not expect - such as we have seen the default web server benchmarks grind to a halt, not because the networking is slow, but they are serving up an uncached file that lives on a 9pfs share.... | 09:07 |
| kata-irc-bot | <graham.whaley> for actual virtualisation costs/overheads, normally you would only notice this (as you would with any VM), for very memory intensive benchmarks afaik. CPU/computer overheads I think you probably will not notice. | 09:08 |
| kata-irc-bot | <graham.whaley> Hope that helps. And you can always let us know which benchmarks you are running, and we can try and predict/diagnose/help configure them etc. | 09:08 |
| *** gwhaley has quit IRC | 11:04 | |
| *** devimc has joined #kata-dev | 12:06 | |
| *** gwhaley has joined #kata-dev | 12:07 | |
| kata-irc-bot | <eric.ernst> Heads up: let’s skip the AC call this week. | 12:08 |
| davidgiluk | eric.ernst: You were asking about formal stuff for virtio-fs ; how/what formal were you thinking off? | 12:21 |
| stefanha | davidgiluk: There is a document here that we can contribute to: https://docs.google.com/document/d/1oc467vctWcxcfzuoXWCFnjPrLXTbDfJzVv67HbGAUrw/edit?ts=5c9c12f5#heading=h.greyp0m0li6d | 13:22 |
| *** lpetrut has joined #kata-dev | 13:42 | |
| *** eernst has joined #kata-dev | 13:44 | |
| kata-irc-bot | <harshal.patil> anyone has detail steps on how to test https://github.com/kata-containers/runtime/pull/1412 ? | 13:56 |
| kata-irc-bot | <harshal.patil> https://github.com/kata-containers/runtime/pull/1412#issuecomment-478209407 @archana.m.shinde any help is appreciated. I tried to test it but no matter which snapshotter I use in my case kata ended up mounting 9pfs always. | 13:57 |
| *** eernst has quit IRC | 13:57 | |
| kata-irc-bot | <harshal.patil> maybe I am missing some step | 13:58 |
| kata-irc-bot | <harshal.patil> @fupan | 13:59 |
| kata-irc-bot | <graham.whaley> @archana.m.shinde @gmmaharaj don't know if you can help with that ^^ ? | 14:03 |
| *** lpetrut has quit IRC | 14:04 | |
| kata-irc-bot | <harshal.patil> I want to setup device mapper support and understand if arguments how command line arguments to qemu differ in case of devicemapper vs regular 9pfs mount | 14:06 |
| kata-irc-bot | <harshal.patil> `./bin/ctr run --snapshotter devmapper --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh` The qemu arguments still show 9pfs mount, ``` /usr/bin/qemu-lite-system-x86_64 -name sandbox-hello -uuid 60ccd470-d469-4e85-bc47-27161656f128 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/hello/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=8707M -device | 14:25 |
| kata-irc-bot | pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/hello/console.sock,server,nowait -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device | 14:25 |
| kata-irc-bot | virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/hello/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/hello,security_model=none -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel | 14:25 |
| kata-irc-bot | /usr/share/kata-containers/vmlinuz-4.19.28.31-22.container -initrd /usr/share/kata-containers/kata-containers-initrd-2019-04-09-17:59:58.516216312+0530-edd7d9c -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 quiet panic=1 nr_cpus=4 agent.use_vsock=false -pidfile | 14:25 |
| kata-irc-bot | /run/vc/vm/hello/pid -smp 1,cores=1,threads=1,sockets=1,maxcpus=4 ``` | 14:25 |
| *** igordc has joined #kata-dev | 14:42 | |
| *** lpetrut has joined #kata-dev | 14:48 | |
| *** devimc has quit IRC | 15:08 | |
| *** devimc has joined #kata-dev | 15:08 | |
| *** devimc_ has joined #kata-dev | 15:10 | |
| *** devimc has quit IRC | 15:12 | |
| *** tmhoang has quit IRC | 15:25 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** sameo has quit IRC | 15:58 | |
| *** sgarzare has quit IRC | 16:05 | |
| kata-irc-bot | <gmmaharaj> @harshal.patil which version of containerd are you running now? | 16:07 |
| kata-irc-bot | <gmmaharaj> let me bring up the latest tree and check it out again. | 16:07 |
| kata-irc-bot | <harshal.patil> ``` # ./bin/containerd -v containerd github.com/containerd/containerd v1.2.0-458-gaaca11c aaca11c40d6540bfccf823485c1dea20106ba7d7 ``` | 16:08 |
| kata-irc-bot | <gmmaharaj> and just to make sure we are all set, can you share the contents of your `/etc/containerd/config.toml` file? | 16:08 |
| kata-irc-bot | <harshal.patil> ``` [plugins] [plugins.devmapper] pool_name = "vg001-mythinpool" base_image_size = "128MB" [plugins.cri.containerd] no_pivot = false [plugins.cri.containerd.runtimes] [plugins.cri.containerd.runtimes.runc] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.runc.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" | 16:09 |
| kata-irc-bot | IoUid = 0 IoGid = 0 BinaryName = "runc" Root = "" CriuPath = "" SystemdCgroup = false [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" [plugins.cri.containerd.runtimes.katacli] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.katacli.options] NoPivotRoot = false NoNewKeyring = | 16:09 |
| kata-irc-bot | false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "/usr/bin/kata-runtime" Root = "" CriuPath = "" SystemdCgroup = false ``` | 16:09 |
| kata-irc-bot | <gmmaharaj> @harshal.patil give me a bit and let me try and bring up my system | 16:10 |
| kata-irc-bot | <gmmaharaj> @harshal.patil ouch! yeah, i am hitting 9p as well. i guess my test was botched and i didn't notice this. apologies. | 16:39 |
| kata-irc-bot | <gmmaharaj> @archana.m.shinde sorry about that too. | 16:40 |
| kata-irc-bot | <gmmaharaj> will try to get a fix for it up and running asap | 16:40 |
| davidgiluk | for kata, where would people like us to put virtiofsd? At the moment we've got it in qemu's contrib - but would that be ok to go in nemu's contrib or is it better elsewhere? | 16:40 |
| stefanha | One reason why we put it into QEMU's contrib/ directory is it will allow testing without cloning multiple git repos. | 16:44 |
| stefanha | Maybe that's a good reason for NEMU to keep it there, too. | 16:44 |
| kata-irc-bot | <archana.m.shinde> @gmmaharaj were you seeing 9p on the command line? | 16:54 |
| kata-irc-bot | <gmmaharaj> @archana.m.shinde snip from my qemu command. ``` -device virtio-scsi-pci,id=scsi0,disable-modern=true,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/hello/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=true,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= | 16:54 |
| kata-irc-bot | -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/hello,security_model=none -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/kata-containers/vmlinuz-4.19.28.31-22.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 ```` | 16:54 |
| kata-irc-bot | <archana.m.shinde> can you run a `mount` command inside your container | 16:55 |
| kata-irc-bot | <archana.m.shinde> and check `/` mount | 16:55 |
| kata-irc-bot | <gmmaharaj> ``` / # mount /dev/sda on / type ext4 (rw,relatime,stripe=64) ``` | 16:55 |
| kata-irc-bot | <archana.m.shinde> cool, then its block :slightly_smiling_face: | 16:56 |
| kata-irc-bot | <gmmaharaj> wait, i thought we will use `virtio-scsi` instead of 9p | 16:56 |
| kata-irc-bot | <gmmaharaj> @harshal.patil ^^ | 16:56 |
| kata-irc-bot | <archana.m.shinde> the 9p on the qemu command line is just a shared directory where we place all rootfs for a containers | 16:56 |
| kata-irc-bot | <graham.whaley> I see a virtio-scsi-pci in that qemu line | 16:56 |
| kata-irc-bot | <graham.whaley> but, we still use 9p for some other bits iirc? | 16:56 |
| kata-irc-bot | <gmmaharaj> aaaah | 16:57 |
| kata-irc-bot | <archana.m.shinde> @graham.whaley thats right | 16:57 |
| kata-irc-bot | <gmmaharaj> @archana.m.shinde let me create the container again. | 16:57 |
| kata-irc-bot | <gmmaharaj> give me 2. | 16:57 |
| kata-irc-bot | <archana.m.shinde> ok | 16:57 |
| kata-irc-bot | <archana.m.shinde> in any case we hotplug the scsi devices except for the scsi controller | 16:58 |
| kata-irc-bot | <gmmaharaj> yup. i am testing 1.7.0_alpha0 and it is mounted as well | 16:58 |
| kata-irc-bot | <gmmaharaj> aaah.. | 16:58 |
| kata-irc-bot | <archana.m.shinde> so you would not see the rootfs on the qemu command line | 16:58 |
| kata-irc-bot | <gmmaharaj> so nothing shows in the command line. | 16:58 |
| kata-irc-bot | <archana.m.shinde> yes | 16:58 |
| kata-irc-bot | <archana.m.shinde> its all hotplugged | 16:59 |
| kata-irc-bot | <gmmaharaj> gotcha. i guess it would help if we document it. and steps for folks to see that we are indeed using blk mount & hotplug./ | 16:59 |
| kata-irc-bot | <gmmaharaj> @harshal.patil you should also be seeing blk then. when you get a chance, could you check. | 16:59 |
| kata-irc-bot | <archana.m.shinde> @harshal.patil Can you perform the above check as well, by running `mount` inside your container | 16:59 |
| *** EricRen has joined #kata-dev | 17:00 | |
| gmmaha | stefanha: davidgiluk: not sure if folks who are working with nemu are around here. i will let them comment and if i don't hear anything, i will reach out to folks to get an answer. | 17:02 |
| stefanha | gmmaha: thanks | 17:43 |
| *** EricRen has quit IRC | 18:20 | |
| *** lpetrut has joined #kata-dev | 18:25 | |
| *** lpetrut has quit IRC | 18:29 | |
| kata-irc-bot | <yan632> Thank you very much @graham.whaley. I was running a Redis server benchmark using memtier and noticed about 50% slowdown in terms of ops/sec for the default configuration, compared to running natively. | 18:45 |
| kata-irc-bot | <graham.whaley> Right. So, that could be the files are coming off 9p. Or.... the default container setup for Kata is 1 or 2 cpus and 2Gb of RAM - which, is unlike the default for say runc. Sooo, if your container has not had cpu and memory constraints set, then that could be a disparity that could also affect the results ;) Kata honours the docker command line and k8s cpu and memory constraint requests, so you can set that on the command line or | 18:48 |
| kata-irc-bot | in k8s yaml for instance. | 18:48 |
| kata-irc-bot | <graham.whaley> and then... heh, if you are running with a lot of cpu's in the container, then the networking (iirc) goes through some qemu worker threads. We set the number to '8' by default I think - but - for heavy traffic workloads you might need more to keep up. That can be set in the kata config file iirc. | 18:50 |
| kata-irc-bot | <graham.whaley> (/me) thinks we need some whitepapers on how to tune these things for kata.... | 18:50 |
| *** davidgiluk has quit IRC | 19:07 | |
| *** gwhaley has quit IRC | 19:36 | |
| kata-irc-bot | <claire> Hey y’all. I added this to the AC call agenda for next week too - but for anyone planning to attend the Project Teams Gathering (PTG) that’s co-located with the Open Infrastructure Summit in Denver in a couple of weeks - it’s now time to start firming up the agenda and discussion topics. https://etherpad.openstack.org/p/katacontainers-2019-ptg-denver-agenda. Also, if you haven’t yet registered or booked a hotel, it’s time to do | 20:49 |
| kata-irc-bot | Links for both are in the etherpad. Thanks! | 20:49 |
| *** fuentess has quit IRC | 21:14 | |
| *** devimc_ has quit IRC | 21:17 | |
| *** igordc has quit IRC | 21:25 | |
| kata-irc-bot | <eric.ernst> @fupan @xu around? | 22:26 |
| *** dklyle has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!