Thursday, 2020-10-29

« Wednesday, 2020-10-28 Index Friday, 2020-10-30 »
*** Yarboa has quit IRC02:32
*** Yarboa has joined #kata-dev02:43
*** auk has quit IRC02:45
*** auk has joined #kata-dev02:47
*** auk_ has joined #kata-dev02:58
*** auk has quit IRC03:00
*** Yarboa has quit IRC03:22
*** auk_ has quit IRC03:30
*** Yarboa has joined #kata-dev03:33
kata-irc-bot<dgibson> @bergwolf hello, are you around?03:46
kata-irc-bot<dgibson> @bergwolf wondering if you could help me debug a CI failure03:47
*** Yarboa has quit IRC04:22
*** Yarboa has joined #kata-dev04:23
*** Yarboa has quit IRC05:32
*** Yarboa has joined #kata-dev05:33
kata-irc-bot<bergwolf> @dgibson which one? can you post the link?05:34
kata-irc-bot<dgibson> @bergwolf http://jenkins.katacontainers.io/job/kata-containers-runtime-vfio-PR/255/console05:35
kata-irc-bot<bergwolf> ```08:59:01 rpc error: code = DeadlineExceeded desc = Timeout reached after 10s waiting for device 0000:00:02.0/0000:01:01.0 08:59:01 Failed at 86: sudo kata-runtime --kata-config "${CONFIG_FILE}" run --detach -b "${bundle_dir}" --pid-file="${tmp_data_dir}/pid" "${container_id}" 08:59:01 ++ handle_error 86```05:38
kata-irc-bot<bergwolf> looks like vfio hotplug is failing?05:39
kata-irc-bot<dgibson> yes, but I've no idea how to investigate deeper05:39
kata-irc-bot<dgibson> it doesn't fail on a simple case I run myself05:39
kata-irc-bot<dgibson> and whatever is different in the CI case is not obvious to me05:40
kata-irc-bot<dgibson> @bergwolf ^05:48
kata-irc-bot<bergwolf> If you cannot reproduce locally, it is a bit nasty. maybe enable all debug options and retry CI to get more logs?05:49
kata-irc-bot<bergwolf> the guest kernel log might have more hints on vfio hotplug05:51
kata-irc-bot<dgibson> @bergwolf where are debug options for the CI controlled?06:20
*** Yarboa has quit IRC06:21
*** Yarboa has joined #kata-dev06:24
kata-irc-bot<dgibson> guest kernel log very likely has useful information, but I don't know how to get to it06:25
kata-irc-bot<bergwolf> https://github.com/kata-containers/documentation/blob/3269ba1f9e57acfc81650177b4b127bbf5b88c12/Developer-Guide.md#enable-full-debug06:32
kata-irc-bot<bergwolf> follow here to enable it06:32
kata-irc-bot<bergwolf> https://github.com/kata-containers/documentation/blob/3269ba1f9e57acfc81650177b4b127bbf5b88c12/Developer-Guide.md#capturing-kernel-boot-logs here is how to get guest dmesg logs06:32
*** dklyle has quit IRC07:10
*** jodh has joined #kata-dev07:38
*** Yarboa has quit IRC07:41
*** sgarzare has joined #kata-dev07:43
*** Yarboa has joined #kata-dev07:54
*** sameo has joined #kata-dev08:18
*** Yarboa has quit IRC08:52
*** Yarboa has joined #kata-dev09:05
*** davidgiluk has joined #kata-dev09:09
*** fgiudici has joined #kata-dev09:16
*** snir has quit IRC12:10
*** snir has joined #kata-dev12:11
*** crobinso has joined #kata-dev12:28
*** sameo has quit IRC13:00
*** sameo has joined #kata-dev13:00
*** devimc has joined #kata-dev13:12
*** vgoyal has joined #kata-dev13:52
*** fuentess has joined #kata-dev14:13
*** snir has quit IRC14:46
*** snir has joined #kata-dev14:47
kata-irc-bot<cmeadors> @bergwolf @dgibson An issue for this CI failure would be helpful.14:47
*** dklyle has joined #kata-dev14:47
*** devimc has quit IRC14:56
*** devimc has joined #kata-dev15:03
kata-irc-bot<dgibson> @cmeadors I'm not really sure what you mean by that15:28
kata-irc-bot<dgibson> @bergwolf those instructions are if you have something local.  I don't, so far, have a way to reproduce locally15:29
kata-irc-bot<cmeadors> @dgibson We need to know more about CI failures: how often they fail and what the root cause it.  Especially if the failure is in something other than the software under test.  Having an issue reported about the failure would be a vehicle to help understand how good our CI is.15:36
kata-irc-bot<cmeadors> especially if it is unclear where the bug is:: software under test, the tests, CI, or infrastructure.15:38
kata-irc-bot<dgibson> well, I don't know the root cause here - the whole point is that figuring out the root cause is really hard16:35
kata-irc-bot<dgibson> I mean intermittent CI failures unrelated to the code under test is *also* a problem16:35
kata-irc-bot<dgibson> but what I16:35
kata-irc-bot<dgibson> but what I'm getting at here is when there's a CI failure due to a bug in submitted code, how do you actually debug it16:36
kata-irc-bot<dgibson> if you can guess the problem from the log messages, well great, but if not...16:36
kata-irc-bot<dgibson> anyway, I'm up far too alte at the moemnt16:37
*** fgiudici has quit IRC17:43
*** jodh has quit IRC18:06
kata-irc-bot<wmoschet> hi guys, how can I trigger the `obs-packaging-ci` job in a PR?18:27
*** sgarzare has quit IRC18:45
kata-irc-bot<eric.ernst> @jose.carlos.venegas.m ^^18:49
kata-irc-bot<wmoschet> it seems to be `/AzurePipelines run` but  I don't have privilege to run it. @jose.carlos.venegas.m mind to trigger it in https://github.com/kata-containers/packaging/pull/1162 for me?18:54
kata-irc-bot<jose.carlos.venegas.m> @wmoschet done :slightly_smiling_face:19:11
kata-irc-bot<wmoschet> great! thanks @jose.carlos.venegas.m @eric.ernst!19:12
kata-irc-bot<wmoschet> @dgibson If I understood correctly the CI job runs in a bare metal in packet.net. Starts with a calling kata-containers/tests/.ci/vfio_jenkins_job_build.sh, which in turn create/setup/boot a either Fedora or Ubuntu VM, then log in the VM to finally run `make vfio` (from kata-containers/tests directory)19:36
kata-irc-bot<wmoschet> that said, did you try to simply run `make vfio` in your local machine?19:36
kata-irc-bot<wmoschet> I copied the script from the CI job, did some modifications. I believe (no guarantee it won't damage your env!) you can run the follow script (from runtime repo) to exactly reproduce the job locally: ```#!/bin/bash  set -ex  export ghprbPullId #export ghprbTargetBranch export ghprbTargetBranch=master export kata_repo=github.com/kata-containers/runtime  #export GOROOT="/usr/local/go" #export GOPATH=${WORKSPACE}/go export19:45
kata-irc-botPATH=${GOPATH}/bin:/usr/local/go/bin:/usr/sbin:/sbin:${PATH}  #pr_number="${ghprbPullId}" pr_number=255 pr_branch="PR_${pr_number}"  mkdir -p ${GOPATH}/src/github.com/kata-containers/tests git clone https://github.com/kata-containers/tests.git ${GOPATH}/src/github.com/kata-containers/tests cd ${GOPATH}/src/github.com/kata-containers/tests  .ci/vfio_jenkins_job_build.sh```19:45
devimc@wmoschet @dgibson I use this https://paste.centos.org/view/bb58efdb19:52
devimcworks in fedora and maybe ubuntu19:52
*** davidgiluk has quit IRC20:03
*** crobinso has quit IRC21:04
*** vgoyal has quit IRC22:05
*** devimc has quit IRC22:08
*** auk has joined #kata-dev22:54
*** crobinso has joined #kata-dev23:42
« Wednesday, 2020-10-28 Index Friday, 2020-10-30 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!