Wednesday, 2018-11-28

« Tuesday, 2018-11-27 Index Thursday, 2018-11-29 »
*** stackedsax has quit IRC02:53
*** zerocoolback has joined #kata-dev04:15
*** zerocoolback has quit IRC04:16
*** lpetrut has joined #kata-dev04:51
*** lpetrut has quit IRC05:28
*** mtudosoiu has quit IRC07:48
*** kailun has joined #kata-dev08:08
*** jodh has joined #kata-dev08:34
*** gwhaley has joined #kata-dev08:52
*** davidgiluk has joined #kata-dev09:00
*** sameo has joined #kata-dev09:08
*** kata-irc-bot has quit IRC09:10
*** kata-irc-bot has joined #kata-dev09:11
*** zerocoolback has joined #kata-dev10:20
*** zerocoolback has quit IRC10:20
*** shrasool has joined #kata-dev10:32
*** shrasool has quit IRC10:38
*** shrasool has joined #kata-dev10:38
*** jodh has quit IRC11:28
*** jodh has joined #kata-dev11:29
kata-irc-bot<niteshkonkar007> Hi, I am unable to boot 1.3 and 1.4 Kata release on x86 as I get the following error: `docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown.`  Tried with centos/fedora initrd with/without seccomp.            Kata release 1.2 boots fine.  From the logs: ``` Record 35: {Count:35 TimeDelta:18162411:45
kata-irc-botFilename:/home/nitesh/TST/kata/src/github.com/kata-containers/runtime/proxy.log Line:6 Time:2018-11-28 16:13:23.480807377 +0530 IST Pid:3261 Level:fatal Msg:failed to handle exit signal Source:proxy Name:kata-proxy Container: Sandbox:5bdb03fdda00adc1be18316c888a25ee3706366c40a08ccf654f642a2b4b5790 Data:map[error:close unix @->/run/vc/vm/5bdb03fdda00adc1be18316c888a25ee3706366c40a08ccf654f642a2b4b5790/kata.sock: use of closed network connection]}11:45
kata-irc-bot30763 +0530 IST Pid:2563 Level:error Msg:Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing Source:runtime Name:kata-runtime Container:5bdb03fdda00adc1be18316c888a25ee3706366c40a08ccf654f642a2b4b5790 Sandbox: Data:map[arch:amd64 command:create]} ```11:45
kata-irc-bot<graham.whaley> hi @niteshkonkar007 - hmm, not a lot of clues in those messages.  When I saw that last week it was my qemu version that was out of date (it was on a CI machine, and it was trying to use a qemu-lite built from source). But, it could also be the agent version in your image is out of sync - check image version as well. I saw you updating github on 'enable full debug' link, so I guess you did that already :slightly_smiling_face: If you12:11
kata-irc-botdon't get further, open a github Issue and others can chip in there12:11
kata-irc-bot<mvedovati> @niteshkonkar007 I agree with @graham.whaley, make sure that qemu is up to date, as that may not be done automatically by your package manager because of this: https://github.com/kata-containers/packaging/issues/19312:13
*** gwhaley has quit IRC12:19
kata-irc-bot<niteshkonkar007> @graham.whaley @mvedovati: I just followed the developer guide and didn't install it from package. Lemme check my qemu version and also check whats the min required one.12:38
*** gwhaley has joined #kata-dev13:21
*** fuentess has joined #kata-dev13:24
*** jodh has quit IRC13:57
*** jodh has joined #kata-dev13:58
*** gwhaley has quit IRC14:28
*** dklyle has quit IRC14:47
*** gwhaley has joined #kata-dev14:53
kata-irc-bot<niteshkonkar007> @graham.whaley @mvedovati: Qemu version was the problem. Thanks.14:59
*** eernst has joined #kata-dev15:00
kata-irc-bot<graham.whaley> @niteshkonkar007 np - it took me a while to figure that out as well... and I really wish the error message could be clearer, but I suspect that it is quite hard to detect early..15:06
*** sameo has quit IRC15:17
*** sameo has joined #kata-dev15:23
davidgilukwhat qemu problem are you hitting?15:23
kata-irc-bot<niteshkonkar007> @davidgiluk: I was using qemu-lite-system-x86_64 ( QEMU emulator version 2.11.0) when I hit the issue. Then switched to qemu-system-x86_64 (version 2.10.2) and I didn't hit the issue then . (The issue is listed above in the chat)15:31
gwhaleydavidgiluk: so, I'm not sure if it a qemu issue per-se - more a 'this qemu version not working with this kata version' issue - and a little annoyingly, I can't think of and I don't know what would cause that. I don't think we added any features that required a specific option for instance...15:33
gwhaleywe probably want a github Issue to discuss it - I suspect somebody will know what/why (maybe @sboeuf or @bergwolf or @eernst for instance)15:33
*** eernst has quit IRC16:00
*** shrasool has quit IRC16:52
*** eernst has joined #kata-dev16:57
*** eernst has quit IRC17:03
*** gwhaley has quit IRC17:59
*** fuentes0 has joined #kata-dev18:01
*** fuentess has quit IRC18:02
*** jodh has quit IRC18:06
*** shrasool has joined #kata-dev18:29
kata-irc-bot<mnaser> good news -- i got it to work18:30
kata-irc-bot<mnaser> bad news -- it crashed after trying a few commands18:30
kata-irc-bot<mnaser> ```[  872.460820] general protection fault: 0000 [#1] SMP PTI [  872.463617] Modules linked in: overlay vhost_net vhost macvtap macvlan tap veth br_netfilter bridge stp llc tun nf_conntrack_netlink nfnetlink xt_statistic xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark xt_comment xt_conntrack xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison kvm_intel18:35
kata-irc-botcirrus kvm ttm drm_kms_helper irqbypass drm snd_pcsp snd_pcm i2c_piix4 virtio_balloon snd_timer snd joydev soundcore qemu_fw_cfg xfs libcrc32c virtio_net virtio_blk crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_generic virtio_pci serio_raw virtio_ring virtio pata_acpi sunrpc scsi_transport_iscsi [  872.489792] CPU: 4 PID: 6599 Comm: qemu-lite-syste Not tainted 4.15.17-300.fc27.x86_64 #1 [  872.494398] Hardware name: OpenStack18:35
kata-irc-botFoundation OpenStack Nova, BIOS 1.11.0-2.el7 04/01/2014 [  872.498964] RIP: 0010:native_write_cr4+0x0/0x10 [  872.501896] RSP: 0018:ffffb559c16cfbf0 EFLAGS: 00010006 [  872.505063] RAX: 00000000001626e0 RBX: 0000000000000046 RCX: ffff9fa53fd00000 [  872.509110] RDX: ffff9fa53fd14020 RSI: ffff9fa53fd23030 RDI: 00000000001606e0 [  872.513151] RBP: 0000000000000004 R08: ffffffffffffff80 R09: 00000000000000ef [  872.517285] R10: fffffc6688d75380 R11:18:35
kata-irc-bot0000000000000778 R12: 0000000000023030 [  872.521366] R13: ffff9fa4ed9509a0 R14: ffff9fa4ed9509a0 R15: ffff9fa4ed95a778 [  872.525459] FS:  00007fc823216700(0000) GS:ffff9fa53fd00000(0000) knlGS:0000000000000000 [  872.530061] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [  872.533609] CR2: 00007fc7fba14e90 CR3: 000000019e20a004 CR4: 00000000001626e0 [  872.537715] Call Trace: [  872.539849]  hardware_disable+0xa5/0xc0 [kvm_intel] [18:35
kata-irc-bot872.543104]  ? vcpu_stat_get_per_vm_open+0x20/0x20 [kvm] [  872.545944]  kvm_arch_hardware_disable+0x15/0x40 [kvm] [  872.548856]  on_each_cpu+0x41/0x60 [  872.551410]  kvm_put_kvm+0x206/0x290 [kvm] [  872.553884]  kvm_vm_release+0x1d/0x30 [kvm] [  872.556335]  __fput+0xd0/0x1e0 [  872.558425]  task_work_run+0x84/0xa0 [  872.560739]  do_exit+0x2dc/0xbb0 [  872.563470]  ? smp_call_function_many+0x23f/0x250 [  872.566223]  do_group_exit+0x3a/0xa0 [18:35
kata-irc-bot872.568535]  get_signal+0x272/0x580 [  872.570859]  do_signal+0x36/0x610 [  872.573020]  ? __audit_syscall_exit+0x231/0x2b0 [  872.575594]  exit_to_usermode_loop+0x69/0xa0 [  872.578055]  do_syscall_64+0x172/0x180 [  872.580358]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [  872.583095] RIP: 0033:0x7fc82e528b99 [  872.585259] RSP: 002b:00007fc823213568 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca [  872.588671] RAX: fffffffffffffe00 RBX:18:35
kata-irc-bot0000561e78e33fe8 RCX: 00007fc82e528b99 [  872.592089] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: 0000561e78e33fe8 [  872.595391] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [  872.598687] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe9bb21a1e [  872.602023] R13: 00007ffe9bb21a1f R14: 0000000000000000 R15: 00007ffe9bb21ab0 [  872.605309] Code: 90 66 2e 0f 1f 84 00 00 00 00 00 0f 20 d8 c3 66 90 66 2e 0f 1f 8418:35
kata-irc-bot00 00 00 00 00 0f 22 df c3 66 90 66 2e 0f 1f 84 00 00 00 00 00 <0f> 22 e7 c3 66 90 66 2e 0f 1f 84 00 00 00 00 00 44 0f 20 c0 c3  [  872.614036] RIP: native_write_cr4+0x0/0x10 RSP: ffffb559c16cfbf0 [  872.617174] ---[ end trace de2cf6ca9471eb5c ]--- [  872.619755] Kernel panic - not syncing: Fatal exception [  872.622933] Kernel Offset: 0x18000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [  872.627366]18:35
kata-irc-botRebooting in 10 seconds.. [  882.633372] invalid opcode: 0000 [#2] SMP PTI```18:35
davidgilukhmm native_write_cr4 - that wrings a bell, why does it ring a bell....18:37
kata-irc-bot<mnaser> following this thread, it looks like folks ran into the same issue with gcp till they patched it upstream18:38
kata-irc-bot<mnaser> https://lkml.org/lkml/2017/12/20/6718:38
kata-irc-bot<mnaser> apparently https://lkml.org/lkml/2017/10/31/43218:38
davidgilukoh yeh, that one again18:38
kata-irc-bot<mnaser> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=8eb3f87d903168bdbd1222776a6b1e281f50513e18:40
davidgilukyep, you need the fixed kernel18:41
kata-irc-bot<mnaser> let met see if the host kernel has that patch included18:41
kata-irc-bot<mnaser> from what i understand, that happens on the host, right?18:41
davidgilukYeh I think so18:42
kata-irc-botAction: mnaser needs to figure out if its included in centos 7.518:42
davidgilukIsnt this the discussion we had the other day when I said it was in 7.6?18:43
*** dklyle has joined #kata-dev19:03
*** dklyle has quit IRC19:27
kata-irc-bot<mnaser> davidgiluk: perhaps.. i am digging in to see if its teh same one19:29
kata-irc-bot<mnaser> i can't find that bug or where it was from bleh19:32
kata-irc-bot<mnaser> yeah indeed that was the fix19:39
kata-irc-bot<mnaser> urgh bummed out we cant do anything about this till 7.619:40
davidgilukmnaser: Well, you could backport the fix, it's tiny19:44
kata-irc-bot<mnaser> davidgiluk: looks like the kernel in centos-rolling is `3.10.0-957.el7`19:47
kata-irc-bot<mnaser> so i think i could bump to that19:48
*** shrasool has quit IRC20:04
*** shrasool has joined #kata-dev20:07
*** davidgiluk has quit IRC20:13
*** shrasool has quit IRC20:16
kata-irc-bot<mnaser> ok, got a machine with an updated kernel, time to test20:26
kata-irc-bot<mnaser> @salvador.fuentes if this goes well, is it okay if i update the host that runs the jenkins workers right now?20:33
kata-irc-bot<mnaser> with the new kernel which should hopefully allow for fedora to stop crashing and maybe bionic start working again20:34
kata-irc-bot<salvador.fuentes> @mnaser, sure, there is no problem, we will have them on queue until the host come back20:36
kata-irc-bot<mnaser> @salvador.fuentes ok, the vms might go down, so do we want to place jenkins in shutdown mode?20:36
kata-irc-bot<salvador.fuentes> @mnaser, you are right, done :slightly_smiling_face:20:38
kata-irc-bot<mnaser> im testing in a fedora atomic vm with new kernel20:39
kata-irc-bot<mnaser> woo, working20:41
kata-irc-bot<mnaser> ok, we have a reboot party ahead of us to get this addressed.20:41
kata-irc-bot<mnaser> ```[fedora@my-cluster-pojfxyiqeafi-minion-0 ~]$ time sudo docker run --runtime kata -it --rm centos:7 echo noop noop  real0m3.699s user0m0.019s sys0m0.020s [fedora@my-cluster-pojfxyiqeafi-minion-0 ~]$ time sudo docker run --runtime oci -it --rm centos:7 echo noop noop  real0m1.073s user0m0.015s sys0m0.029s [fedora@my-cluster-pojfxyiqeafi-minion-0 ~]$ ```20:42
kata-irc-bot<mnaser> are these okay numbers or do i have something possibly miswired?20:42
kata-irc-bot<salvador.fuentes> I think they are ok, not sure what is the boot time we have right now... @graham.whaley should have more info, but he is already off for today. @sebastien.boeuf @julio.montes, any idea if that number is ok?20:46
kata-irc-bot<salvador.fuentes> @mnaser oci == runc?20:46
kata-irc-bot<mnaser> @salvador.fuentes `--add-runtime oci=/usr/libexec/docker/docker-runc-current` correct20:47
kata-irc-bot<salvador.fuentes> ok20:47
kata-irc-bot<mnaser> now trying to see why k8s containers arent going up20:47
kata-irc-bot<mnaser> ```[root@my-cluster-pojfxyiqeafi-minion-0 fedora]# docker ps -a CONTAINER ID        IMAGE                                                                                                                                     COMMAND                  CREATED              STATUS              PORTS               NAMES 2ab98f15c1e120:47
kata-irc-botgcr.io/google_containers/kubernetes-dashboard-amd64@sha256:dc4026c1b595435ef5527ca598e1e9c4343076926d7d62b365c44831395adbd0               "/dashboard --inse..."   About a minute ago   Created                                 k8s_kubernetes-dashboard_kubernetes-dashboard-846b8b6844-8t5xg_kube-system_6ec3b24e-f34c-11e8-a5e2-fa163e1f20de_3 6f722cc5eb5a20:47
kata-irc-botdocker.io/coredns/coredns@sha256:54e6d446cf14791230000eb9259e65717e147187f01f0534052a60f50b96186f                                         "/coredns -conf /e..."   About a minute ago   Created                                 k8s_coredns_coredns-5864cfd79d-hxgr6_kube-system_6ec09372-f34c-11e8-a5e2-fa163e1f20de_3 b9de85b90c11        gcr.io/google_containers/heapster-amd64@sha256:f58ded16b56884eeb73b1ba256bcc489714570bacdeca43d4ba3b91ef9897b2020:47
kata-irc-bot                  "/heapster --sourc..."   About a minute ago   Created                                 k8s_heapster_heapster-77984c85b6-fh2kf_kube-system_6ec018a6-f34c-11e8-a5e2-fa163e1f20de_3 4cd7122f8c2f        docker.io/googlecontainer/cluster-proportional-autoscaler-amd64@sha256:621d77fe109a75e1381b10c2525d8d62656b0192ecdbab7e5d26a9bd0dfbb76a   "/cluster-proporti..."   About a minute ago   Created20:47
kata-irc-botk8s_autoscaler_kube-dns-autoscaler-579c464c94-lpkmr_kube-system_6ec1cd3c-f34c-11e8-a5e2-fa163e1f20de_3 7136d061e480        gcr.io/google_containers/pause:3.0                                                                                                        "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_heapster-77984c85b6-fh2kf_kube-system_6ec018a6-f34c-11e8-a5e2-fa163e1f20de_1 8cc44b482c3f20:47
kata-irc-botgcr.io/google_containers/pause:3.0                                                                                                        "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_coredns-5864cfd79d-hxgr6_kube-system_6ec09372-f34c-11e8-a5e2-fa163e1f20de_1 7e073152d576        gcr.io/google_containers/pause:3.020:47
kata-irc-bot              "/pause"                 2 minutes ago        Up 2 minutes                            k8s_POD_kube-dns-autoscaler-579c464c94-lpkmr_kube-system_6ec1cd3c-f34c-11e8-a5e2-fa163e1f20de_1 24c0e907a1bf        gcr.io/google_containers/pause:3.0                                                                                                        "/pause"                 2 minutes ago        Up 2 minutes20:47
kata-irc-botk8s_POD_kubernetes-dashboard-846b8b6844-8t5xg_kube-system_6ec3b24e-f34c-11e8-a5e2-fa163e1f20de_2```20:47
kata-irc-bot<mnaser> the pause ones are up fine, but the other ones arent going up for some reason20:47
kata-irc-bot<mnaser> is there a way to get kata containers debug logs? i dont see the qemu procs or anything like that up now20:48
kata-irc-bot<mnaser> ok also i see no jobs running now20:49
kata-irc-bot<mnaser> so im going to get updating20:49
kata-irc-bot<salvador.fuentes> @mnaser, you will need cri-o or containerd instead of using docker20:51
kata-irc-bot<sebastien.boeuf> @salvador.fuentes because those run inside a VM, the numbers don't look crazy to me. @eric.ernst any input on this?20:51
kata-irc-bot<mnaser> i think docker on atomic uses containerd?20:51
*** sameo has quit IRC20:51
kata-irc-bot<mnaser> ```root      7580  0.1  0.1 1277192 11732 ?       Ssl  20:44   0:00 /usr/libexec/docker/docker-containerd-current --listen unix:///run/container root      8606  0.0  0.0 675532  2900 ?        Sl   20:44   0:00  \_ /usr/libexec/docker/docker-containerd-shim-current 24c0e907a1bf5801982cd root      8725  0.8  2.1 3601900 178996 ?      Sl   20:44   0:03  |   \_ /usr/bin/qemu-lite-system-x86_64 -name sandbox-24c0e907a1bf5801982cd root20:52
kata-irc-bot8745  0.0  0.0 678744  2888 ?        Sl   20:44   0:00  |   \_ /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/v root      8807  0.0  0.1 758712  8724 ?        Sl   20:45   0:00  |   \_ /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/24c root      9004  0.0  0.0 601800  2956 ?        Sl   20:45   0:00  \_ /usr/libexec/docker/docker-containerd-shim-current 7e073152d5767775094fc root      9063  0.8  2.2 332745220:52
kata-irc-bot180816 ?      Sl   20:45   0:03  |   \_ /usr/bin/qemu-lite-system-x86_64 -name sandbox-7e073152d5767775094fc root      9069  0.0  0.0 676824  3156 ?        Sl   20:45   0:00  |   \_ /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/v root      9096  0.0  0.1 684724  8652 ?        Sl   20:45   0:00  |   \_ /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/7e0 root      9230  0.0  0.0 601800  2508 ?        Sl   20:4520:52
kata-irc-bot 0:00  \_ /usr/libexec/docker/docker-containerd-shim-current 8cc44b482c3f7c1689d75 root      9300  0.9  2.2 3536364 180604 ?      Sl   20:45   0:03  |   \_ /usr/bin/qemu-lite-system-x86_64 -name sandbox-8cc44b482c3f7c1689d75 root      9326  0.0  0.0 677080  3400 ?        Sl   20:45   0:00  |   \_ /usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/v root      9449  0.0  0.1 758456  8704 ?        Sl   20:45   0:00  |   \_20:52
kata-irc-bot/usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/8cc root      9359  0.0  0.0 749008  2680 ?        Sl   20:45   0:00  \_ /usr/libexec/docker/docker-containerd-shim-current 7136d061e4804561406cc root      9415  0.9  2.2 3539312 181176 ?      Sl   20:45   0:03      \_ /usr/bin/qemu-lite-system-x86_64 -name sandbox-7136d061e4804561406cc root      9421  0.0  0.0 677080  3224 ?        Sl   20:45   0:00      \_20:52
kata-irc-bot/usr/libexec/kata-containers/kata-proxy -listen-socket unix:///run/v root      9479  0.0  0.1 683316  8956 ?        Sl   20:45   0:00      \_ /usr/libexec/kata-containers/kata-shim -agent unix:///run/vc/sbs/713 root      7651  1.2  0.7 1234744 64472 ?       Ssl  20:44   0:05 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-c```20:52
kata-irc-bot<mnaser> @salvador.fuentes you can put it out of shutdown mode21:11
kata-irc-bot<salvador.fuentes> @mnaser, it is on shutdown mode now, you can go ahead and make your changes21:18
kata-irc-bot<mnaser> @salvador.fuentes oh my work is done :slightly_smiling_face:21:18
kata-irc-bot<mnaser> we can try and restore fedora-28 jobs too21:18
kata-irc-bot<salvador.fuentes> ok, cool, can you create a fedora28 flavor for that? Last time I checked, I didnt saw a fedora28 image21:19
kata-irc-bot<mnaser> @salvador.fuentes it should all be setup, no? i just see it disabled thats all21:27
kata-irc-bot<salvador.fuentes> @mnaser, but if I go to the cloud setup, there is no image for fedora 28, the last time I tested fedora 28 on vexxhost was using a VM from a zuul job21:29
kata-irc-bot<mnaser> ah oh yes21:29
kata-irc-bot<salvador.fuentes> but in jenkins we do not have it available21:29
kata-irc-bot<mnaser> let me find an image to upload21:29
kata-irc-bot<salvador.fuentes> thanks21:29
kata-irc-bot<mnaser> @salvador.fuentes is f29 okay?21:30
kata-irc-bot<mnaser> or do we want f2821:30
kata-irc-bot<salvador.fuentes> @mnaser f28 please21:37
kata-irc-bot<salvador.fuentes> we still have issues with f29 because OBS has not yet support for f29 and the docker version we use21:37
kata-irc-bot<mnaser> @salvador.fuentes added both fedora 28 and fedora 29 anyways :slightly_smiling_face:21:39
kata-irc-bot<mnaser> so there should be templates fo rboth21:39
kata-irc-bot<salvador.fuentes> ahh great, thank you, will re-enable the f28 and lets see how it goes21:40
*** shrasool has joined #kata-dev21:51
kata-irc-bot<salvador.fuentes> @mnaser, seems like the f28 machines cannot be created http://jenkins.katacontainers.io/cloud-stats/21:54
kata-irc-bot<mnaser> "java.lang.NullPointerException"21:55
kata-irc-bot<mnaser> very helpful21:55
kata-irc-bot<mnaser> lol21:55
kata-irc-bot<mnaser> i gotta run, ill bbl shortly21:55
*** shrasool has quit IRC22:00
kata-irc-bot<salvador.fuentes> @mnaser, hehe, yeah, helpless message, sure, no problem22:25
clarkbyou should have a full traceback in the jenkins log on disk22:30
*** shrasool has joined #kata-dev22:39
*** fuentes0 has quit IRC22:42
*** shrasool has quit IRC22:56
*** shrasool has joined #kata-dev23:24
*** jugs has quit IRC23:57
*** jugs has joined #kata-dev23:57
« Tuesday, 2018-11-27 Index Thursday, 2018-11-29 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!