| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Skip backports for buster https://review.openstack.org/648293 | 00:01 |
|---|---|---|
| ianw | fungi / zigo : ^ might be the minimum viable thing at this point ... | 00:01 |
| *** wolverineav has quit IRC | 00:02 | |
| fungi | doesn't seem all that hacky to me | 00:08 |
| *** hongbin has joined #openstack-infra | 00:18 | |
| *** betherly has joined #openstack-infra | 00:28 | |
| *** betherly has quit IRC | 00:33 | |
| *** lseki has quit IRC | 00:53 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** betherly has joined #openstack-infra | 01:01 | |
| ianw | hrm, "Get https://registry-1.docker.io/v2/library/registry/manifests/2: unauthorized: incorrect username or password" | 01:05 |
| *** betherly has quit IRC | 01:05 | |
| *** jamesmcarthur has quit IRC | 01:07 | |
| *** ricolin has joined #openstack-infra | 01:07 | |
| *** gyee has quit IRC | 01:16 | |
| *** betherly has joined #openstack-infra | 01:33 | |
| *** betherly has quit IRC | 01:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Skip backports for buster https://review.openstack.org/648293 | 01:53 |
| *** rlandy|bbl is now known as rlandy | 01:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:03 | |
| *** gregoryo has joined #openstack-infra | 02:04 | |
| *** gouthamr has quit IRC | 02:10 | |
| *** jhesketh has joined #openstack-infra | 02:12 | |
| *** gouthamr has joined #openstack-infra | 02:12 | |
| *** yamamoto has joined #openstack-infra | 02:15 | |
| *** betherly has joined #openstack-infra | 02:15 | |
| *** betherly has quit IRC | 02:20 | |
| *** diablo_rojo has joined #openstack-infra | 02:24 | |
| *** bhavikdbavishi has quit IRC | 02:25 | |
| *** rh-jelabarre has quit IRC | 02:25 | |
| *** raukadah is now known as chandankumar | 02:27 | |
| *** wolverineav has joined #openstack-infra | 02:37 | |
| *** wolverineav has quit IRC | 02:42 | |
| ianw | oh ffs reprepo ... now it's unhappy the thing that was never there went away ... Error: packages database contains unused 'buster-backports|main|amd64' database. | 02:46 |
| *** diablo_rojo has quit IRC | 02:47 | |
| *** psachin has joined #openstack-infra | 02:51 | |
| *** ykarel|away has joined #openstack-infra | 02:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:56 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:57 | |
| *** whoami-rajat has joined #openstack-infra | 02:58 | |
| *** bhavikdbavishi has quit IRC | 03:01 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 03:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: implement prep-workspace https://review.openstack.org/607082 | 03:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add configuration schema https://review.openstack.org/640672 | 03:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 03:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add command line interface https://review.openstack.org/644770 | 03:04 |
| *** betherly has joined #openstack-infra | 03:08 | |
| *** betherly has quit IRC | 03:13 | |
| *** apetrich has quit IRC | 03:14 | |
| *** yamamoto has quit IRC | 03:18 | |
| *** rkukura has joined #openstack-infra | 03:27 | |
| *** hongbin has quit IRC | 03:28 | |
| *** ramishra has joined #openstack-infra | 03:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 03:41 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: implement prep-workspace https://review.openstack.org/607082 | 03:41 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add configuration schema https://review.openstack.org/640672 | 03:41 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: runner: add execute sub-command https://review.openstack.org/630944 | 03:41 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add command line interface https://review.openstack.org/644770 | 03:41 |
| *** yamamoto has joined #openstack-infra | 03:42 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Minor clarifications in centos7 element docs https://review.openstack.org/644913 | 03:50 |
| *** betherly has joined #openstack-infra | 04:02 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** betherly has quit IRC | 04:07 | |
| *** Lucas_Gray has joined #openstack-infra | 04:14 | |
| *** yamamoto has quit IRC | 04:25 | |
| *** yamamoto has joined #openstack-infra | 04:29 | |
| *** janki has joined #openstack-infra | 04:41 | |
| *** Lucas_Gray has quit IRC | 04:43 | |
| *** Lucas_Gray has joined #openstack-infra | 04:45 | |
| *** rcernin has quit IRC | 04:47 | |
| *** betherly has joined #openstack-infra | 04:55 | |
| *** ykarel|away has quit IRC | 04:58 | |
| *** betherly has quit IRC | 04:59 | |
| *** rcernin has joined #openstack-infra | 05:01 | |
| *** markvoelker has joined #openstack-infra | 05:04 | |
| *** whoami-rajat has quit IRC | 05:07 | |
| *** ykarel|away has joined #openstack-infra | 05:13 | |
| *** ruijing has joined #openstack-infra | 05:13 | |
| ruijing | Hi, Naser | 05:14 |
| ruijing | OpenStack CI runs on VM, which is not nested virtualization. how can we make it to support nested virtualization | 05:15 |
| *** Lucas_Gray has quit IRC | 05:22 | |
| *** betherly has joined #openstack-infra | 05:48 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config master: Trove: use storyboard https://review.openstack.org/648320 | 05:52 |
| *** betherly has quit IRC | 05:52 | |
| *** kjackal has joined #openstack-infra | 05:53 | |
| *** ykarel|away is now known as ykarel | 05:58 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Add nofollow attribute to user-submitted links https://review.openstack.org/648322 | 06:01 |
| *** rcernin has quit IRC | 06:03 | |
| *** jbadiapa has quit IRC | 06:07 | |
| *** slaweq has quit IRC | 06:07 | |
| *** fatema_ has quit IRC | 06:11 | |
| ruijing | Nested virt is not available in all clouds. And in clouds where it is enabled we have observed a higher rate of crashed test VMs when using it. As a result we enforce qemu when running devstack and may further restrict the use of nested virt. | 06:13 |
| *** ccamacho has quit IRC | 06:23 | |
| *** wolverineav has joined #openstack-infra | 06:24 | |
| *** whoami-rajat has joined #openstack-infra | 06:31 | |
| *** wolverineav has quit IRC | 06:36 | |
| *** wolverineav has joined #openstack-infra | 06:37 | |
| *** betherly has joined #openstack-infra | 06:40 | |
| *** betherly has quit IRC | 06:45 | |
| *** rcernin has joined #openstack-infra | 06:47 | |
| *** rcernin has quit IRC | 06:47 | |
| *** rcernin has joined #openstack-infra | 06:48 | |
| *** quiquell|off is now known as quiquell | 06:56 | |
| ianw | zigo: btw it's still cloning | 06:58 |
| AJaeger | infra-root, did anybody clean up the content after https://review.openstack.org/#/c/648237/ merged (see there for explanation by corvus)? I still do not see the new repos... | 06:58 |
| *** AJaeger has quit IRC | 07:02 | |
| *** AJaeger has joined #openstack-infra | 07:05 | |
| *** chandankumar has quit IRC | 07:06 | |
| *** gfidente has joined #openstack-infra | 07:07 | |
| *** chandankumar has joined #openstack-infra | 07:08 | |
| dirk | infra-root: is there a new error in rsync loga for opensuse? The timestamp http://mirror.dfw.rax.openstack.org/opensuse/ here is still old.. | 07:13 |
| *** jbadiapa has joined #openstack-infra | 07:14 | |
| *** slaweq has joined #openstack-infra | 07:14 | |
| AJaeger | Yeah, finally https://mirror.us.leaseweb.net/opensuse/update/leap/15.0/oss/ has content now | 07:16 |
| *** e0ne has joined #openstack-infra | 07:17 | |
| *** e0ne has quit IRC | 07:19 | |
| *** pcaruana has joined #openstack-infra | 07:20 | |
| *** yboaron_ has joined #openstack-infra | 07:22 | |
| mordred | AJaeger: /var/lib/git/zuul looks empty on the git backends, so i'm not sure what's up with the new repos | 07:23 |
| *** ykarel is now known as ykarel|lunch | 07:26 | |
| dirk | jhesketh: can you help? For almost a week all of the opensuse stuff in infra is broken | 07:26 |
| *** amorin has quit IRC | 07:28 | |
| *** amorin has joined #openstack-infra | 07:28 | |
| AJaeger | mordred: thanks for checking. Did we hit the next bug? There's no urgency that we need this now, so go back to sleep ;) | 07:29 |
| mordred | AJaeger: oh - going back to sleep isn't the right life choice - I'm in Athens for a conference, so staying awake is today's main task | 07:29 |
| mordred | AJaeger: but - given the circumstances we might not want to trust any output from my brain today either :) | 07:30 |
| AJaeger | mordred: hope you see more than the conference location in Athens ;) | 07:35 |
| AJaeger | mordred: good morning to you! | 07:35 |
| mordred | AJaeger: good morning to you too! | 07:37 |
| *** rpittau|afk is now known as rpittau | 07:41 | |
| *** rcernin has quit IRC | 07:48 | |
| *** apetrich has joined #openstack-infra | 07:49 | |
| *** kopecmartin|off is now known as kopecmartin | 07:52 | |
| *** wolverineav has quit IRC | 08:01 | |
| *** wolverineav has joined #openstack-infra | 08:02 | |
| *** pgaxatte has joined #openstack-infra | 08:04 | |
| *** tosky has joined #openstack-infra | 08:04 | |
| *** rcernin has joined #openstack-infra | 08:04 | |
| *** ccamacho has joined #openstack-infra | 08:05 | |
| *** betherly has joined #openstack-infra | 08:06 | |
| *** wolverineav has quit IRC | 08:08 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** iurygregory_ has joined #openstack-infra | 08:10 | |
| *** wolverineav has joined #openstack-infra | 08:11 | |
| *** betherly has quit IRC | 08:11 | |
| *** iurygregory has quit IRC | 08:12 | |
| *** iurygregory_ is now known as iurygregory | 08:12 | |
| *** Lucas_Gray has joined #openstack-infra | 08:13 | |
| *** florianf has joined #openstack-infra | 08:16 | |
| *** wolverineav has quit IRC | 08:24 | |
| *** wolverineav has joined #openstack-infra | 08:26 | |
| *** helenafm has joined #openstack-infra | 08:29 | |
| openstackgerrit | Natal Ngétal proposed openstack/diskimage-builder master: [Configuration] Switch to stestr. https://review.openstack.org/629414 | 08:35 |
| *** jpena|off is now known as jpena | 08:39 | |
| *** ykarel|lunch is now known as ykarel | 08:40 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 08:40 | |
| *** wolverineav has quit IRC | 08:42 | |
| *** wolverineav has joined #openstack-infra | 08:43 | |
| *** jpich has joined #openstack-infra | 08:49 | |
| *** wolverineav has quit IRC | 08:52 | |
| *** Wryhder has joined #openstack-infra | 08:54 | |
| frickler | dirk: the mirroring is in progress, but didn't finish yet due to timeout. it is picking up on the next run where it was interrupted, though, so it hopefully should finish soon. or maybe some infra-root has time to let it run without timeout manually | 08:56 |
| *** Lucas_Gray has quit IRC | 08:57 | |
| *** Wryhder is now known as Lucas_Gray | 08:57 | |
| frickler | dirk: actually, that was when I looked earlier, now it's running into another error, this time with tumbleweed: http://paste.openstack.org/show/748517/ | 08:59 |
| *** betherly has joined #openstack-infra | 08:59 | |
| frickler | dirk: I'm assuming the new status will only be released when the mirror job completed successfully | 09:00 |
| jhesketh | dirk: sorry, was making dinner.. back for some meetings, but looks like frickler is on it :-) | 09:00 |
| *** rossella_s has joined #openstack-infra | 09:02 | |
| *** betherly has quit IRC | 09:04 | |
| *** oanson has quit IRC | 09:10 | |
| *** snapiri has quit IRC | 09:10 | |
| *** snapiri has joined #openstack-infra | 09:18 | |
| *** shachar has joined #openstack-infra | 09:21 | |
| *** snapiri has quit IRC | 09:22 | |
| *** betherly has joined #openstack-infra | 09:23 | |
| *** lucasagomes has joined #openstack-infra | 09:25 | |
| *** ricolin has quit IRC | 09:27 | |
| *** betherly has quit IRC | 09:28 | |
| *** gregoryo has quit IRC | 09:28 | |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** oanson has joined #openstack-infra | 09:41 | |
| *** ginopc has joined #openstack-infra | 09:43 | |
| *** e0ne has joined #openstack-infra | 09:47 | |
| *** ginux has joined #openstack-infra | 09:51 | |
| *** ginopc has quit IRC | 09:51 | |
| *** ginux is now known as ginopc | 09:51 | |
| *** lpetrut has joined #openstack-infra | 09:54 | |
| *** priteau has joined #openstack-infra | 09:54 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/openstack-zuul-jobs master: Extend project translation template for Stein https://review.openstack.org/648362 | 09:56 |
| ianychoi | AJaeger, would you kindly review ^? (I will check more later - after dinner...) | 09:57 |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** tobias-urdin has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:07 | |
| *** florianf has quit IRC | 10:12 | |
| *** ginopc has quit IRC | 10:21 | |
| *** ginopc has joined #openstack-infra | 10:23 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/system-config master: Add monitoring for filesystems other than root https://review.openstack.org/648365 | 10:25 |
| *** ginopc has quit IRC | 10:26 | |
| frickler | infra-root: ^^ I tested this manually for nb01, if you are unsure about it, I could also give it a test about all our hosts | 10:26 |
| *** melwitt has quit IRC | 10:26 | |
| *** ginopc has joined #openstack-infra | 10:30 | |
| *** e0ne has quit IRC | 10:31 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: Fix typos https://review.openstack.org/648368 | 10:32 |
| *** melwitt has joined #openstack-infra | 10:32 | |
| *** e0ne has joined #openstack-infra | 10:36 | |
| *** ramishra has quit IRC | 10:37 | |
| *** yamamoto has quit IRC | 10:37 | |
| *** ramishra has joined #openstack-infra | 10:37 | |
| *** bhavikdbavishi has quit IRC | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:41 | |
| *** ginopc has quit IRC | 10:58 | |
| *** e0ne has quit IRC | 10:58 | |
| *** ginopc has joined #openstack-infra | 10:59 | |
| *** udesale has quit IRC | 11:01 | |
| *** e0ne has joined #openstack-infra | 11:05 | |
| *** ginopc has quit IRC | 11:11 | |
| *** lennyb has quit IRC | 11:12 | |
| *** lennyb has joined #openstack-infra | 11:14 | |
| stephenfin | If I'm reading the code correctly, this URL should exist and return something useful but it doesn't. Am I missing something obvious? http://zuul.openstack.org/api/tenant/openstack/info | 11:17 |
| stephenfin | http://zuul.openstack.org/api/info does work | 11:17 |
| *** janki has quit IRC | 11:18 | |
| stephenfin | Also, the fact that it sometimes HTML pages for things under '/api' and redirects invalid pages is annoying when trying to explore the API. I assume that's intentional (i.e. I shouldn't try to "fix" it)? | 11:19 |
| *** bhavikdbavishi has joined #openstack-infra | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:37 | |
| *** janki has joined #openstack-infra | 11:43 | |
| *** Lucas_Gray has joined #openstack-infra | 11:49 | |
| *** jtomasek has joined #openstack-infra | 11:51 | |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** weshay_pto is now known as weshay | 11:59 | |
| *** jtomasek has quit IRC | 12:00 | |
| *** rh-jelabarre has joined #openstack-infra | 12:01 | |
| iurygregory | morning infra team, we are trying to backport a zuulv3 change to stable/stein for python-ironicclient, but the jobs for some reason aren't running https://review.openstack.org/#/c/647774/ does anyone have an idea why? in the master the jobs are fine | 12:05 |
| *** Lucas_Gray has quit IRC | 12:09 | |
| *** Wryhder has joined #openstack-infra | 12:09 | |
| *** whoami-rajat has quit IRC | 12:10 | |
| *** Wryhder is now known as Lucas_Gray | 12:10 | |
| *** trown|outtypewww is now known as trown | 12:11 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** whoami-rajat has joined #openstack-infra | 12:20 | |
| *** yamamoto has quit IRC | 12:21 | |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| *** electrofelix has joined #openstack-infra | 12:30 | |
| *** jtomasek has joined #openstack-infra | 12:32 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** kgiusti has joined #openstack-infra | 12:40 | |
| *** e0ne has quit IRC | 12:41 | |
| frickler | iurygregory: devstack branched only yesterday, so devstack-minimal that you use as parent didn't exist until then. I think a recheck should run those jobs now | 12:43 |
| *** e0ne has joined #openstack-infra | 12:43 | |
| frickler | iurygregory: if it doesn't, we would need into the job definitions im more detail | 12:43 |
| iurygregory | frickler, ack going to try a recheck o/ | 12:44 |
| frickler | +insert"to look" | 12:44 |
| iurygregory | https://zuul.openstack.org/status \o/ | 12:44 |
| iurygregory | yay it shows the jobs | 12:44 |
| frickler | o.k., great | 12:45 |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| *** jtomasek has quit IRC | 12:46 | |
| *** zul has joined #openstack-infra | 12:47 | |
| AJaeger | iurygregory: you can always insert "debug: true" in your zuul.yaml to see why jobs do not run, see the Zuul docs for details | 12:47 |
| AJaeger | iurygregory: https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug | 12:47 |
| iurygregory | AJaeger, tks for the tip o/ | 12:48 |
| iurygregory | AJaeger++ | 12:48 |
| AJaeger | but glad that frickler was able to find it without that ;) | 12:48 |
| fungi | ruijing: kashyap and some others were trying to get kernel maintainers connected with qemu maintainers and systems operators to work out the pain points with nested virt. our experience has been that most people who consider it "stable" are running newer (no more than 5-year-old) hardware and in private clouds where they control the kernel version both on the hosts and in the guests. last i heard, | 12:48 |
| fungi | nested virt acceleration for intel cpus was even still disabled by default in upstream (kernel.org) linux kernel configs | 12:48 |
| *** wolverineav has joined #openstack-infra | 12:49 | |
| fungi | stephenfin: you may have more luck with http://zuul.opendev.org/api/tenant/openstack/info | 12:50 |
| fungi | zuul.openstack.org has some legacy redirects in place from when it was primarily single-tenant | 12:50 |
| *** udesale has joined #openstack-infra | 12:50 | |
| fungi | (they're both the same zuul server, just different apache vhosts with different redirects/rewrites) | 12:51 |
| *** jcoufal has joined #openstack-infra | 12:53 | |
| *** wolverineav has quit IRC | 12:54 | |
| AJaeger | fungi: kernel 4.20 in OCtober '18 finally enabled nested virtualization by default... | 12:55 |
| AJaeger | fungi: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e58e5e59148916fa43444a406335a990783fb78 | 12:55 |
| fungi | oh, wow--that's quite a milestone! | 12:56 |
| fungi | so i guess as long as all hosts and guests are booting 4.20 or later on hardware from the end of last year, everything should be peachy ;) | 12:57 |
| AJaeger | so, let's wait another 5 years ;) | 12:57 |
| fungi | roughly | 12:57 |
| *** mujahidali has joined #openstack-infra | 13:05 | |
| *** lseki has joined #openstack-infra | 13:06 | |
| *** jtomasek has joined #openstack-infra | 13:06 | |
| *** jtomasek has quit IRC | 13:08 | |
| *** aaronsheffield has joined #openstack-infra | 13:09 | |
| *** jtomasek has joined #openstack-infra | 13:12 | |
| frickler | fungi: would you have time to look into the opensuse mirror error I pasted earlier? it is still happening and I'm only half sure that it's an issue with the upstream mirror and nothing on our side | 13:12 |
| fungi | frickler: i'll try to take a look shortly | 13:13 |
| openstackgerrit | Natal Ngétal proposed openstack-dev/pbr master: Add python3.7 job on Stein+ https://review.openstack.org/610659 | 13:14 |
| *** rkukura has quit IRC | 13:15 | |
| *** jpena|lunch is now known as jpena | 13:16 | |
| *** jhesketh has quit IRC | 13:18 | |
| *** e0ne has quit IRC | 13:20 | |
| *** quiquell is now known as quiquell|lunch | 13:23 | |
| *** amansi26 has joined #openstack-infra | 13:23 | |
| *** derekh has quit IRC | 13:24 | |
| *** lpetrut has quit IRC | 13:25 | |
| *** electrofelix has quit IRC | 13:26 | |
| *** electrofelix has joined #openstack-infra | 13:26 | |
| openstackgerrit | Serena Ziviani proposed openstack/diskimage-builder master: Also use selinuxenabled to check selinux status https://review.openstack.org/643367 | 13:27 |
| stephenfin | fungi++ Of course :) Thanks | 13:32 |
| *** electrofelix has quit IRC | 13:32 | |
| stephenfin | fungi: Would returning a 404 for something like http://zuul.opendev.org/api/foobar make sense? | 13:33 |
| *** imacdonn has quit IRC | 13:33 | |
| stephenfin | (as opposed to the current 'Fetching info' HTML page) | 13:34 |
| *** udesale has quit IRC | 13:41 | |
| *** udesale has joined #openstack-infra | 13:41 | |
| fungi | stephenfin: some of that may have to do with what we do and don't proxy through to the api service from apache | 13:43 |
| *** tobias-urdin has joined #openstack-infra | 13:43 | |
| zigo | ianw: There's indeed no buster-backports yet until it's released. | 13:43 |
| zigo | fungi: ianw: What's the status of the mirror? | 13:44 |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul master: web: Add '/' API route https://review.openstack.org/648399 | 13:44 |
| fungi | stephenfin: though it looks like we do rewrite all of ^/api/(.*)$ to the api | 13:44 |
| stephenfin | fungi: Aye, I _think_ it might be intentional https://github.com/openstack-infra/zuul/blob/master/tests/unit/test_web.py#L262-L265 | 13:45 |
| stephenfin | though I can't fathom why | 13:45 |
| stephenfin | Not a big deal though. More of an "oh, this is odd" thing :) | 13:46 |
| *** mujahidali has quit IRC | 13:51 | |
| *** jamesmcarthur has quit IRC | 13:52 | |
| tosky | hi, I have a small review for the infra-manual which has been waiting for a while: https://review.openstack.org/#/c/630981/ | 13:52 |
| *** yamamoto has quit IRC | 13:53 | |
| *** quiquell|lunch is now known as quiquell | 13:57 | |
| *** dpawlik has quit IRC | 13:58 | |
| *** mujahidali has joined #openstack-infra | 13:58 | |
| amansi26 | Ian Wienand proposed a patch set with change ID [I277cbdccf8ea0679d53e449474d7aa8911c999ef] which fails at Zuul check with an error build/temp.linux-x86_64-2.7/_openssl.c:493:10: fatal error: openssl/opensslv.h: No such file or directory. Can anyone help me resolving the issue? | 13:59 |
| *** ginopc has joined #openstack-infra | 14:00 | |
| *** yamamoto has joined #openstack-infra | 14:01 | |
| *** derekh has joined #openstack-infra | 14:04 | |
| *** ginopc has quit IRC | 14:05 | |
| *** yamamoto has quit IRC | 14:05 | |
| frickler | amansi26: that error looks unrelated to the patch, probably more related to the job running on bionic, which shouldn't happen on stable/ocata. | 14:07 |
| fungi | frickler: rsync: opendir "/tumbleweed/repo/oss/i686/.~tmp~" (in opensuse) failed: Permission denied (13) | 14:08 |
| zbr | fungi: AJaeger : if you can help with small nit at https://review.openstack.org/#/c/639951/ bindep feature | 14:08 |
| fungi | frickler: looks like there are unreadable files in the tree | 14:08 |
| amansi26 | frickler: so what fix do you suggest? | 14:09 |
| frickler | fungi: is that remote or local? I couldn't find that .~tmp~ thing looking at the remote via the webserver, nor locally | 14:09 |
| fungi | that was one example but there are also some with the same name under noarch, repodata and x86_64 | 14:10 |
| fungi | frickler: were you looking with rsync or http/ftp? the latter likely hide dotfiles | 14:10 |
| frickler | fungi: with http and yes it may just hide that. dirk seems to have access to the origin, maybe he can crosscheck | 14:11 |
| frickler | amansi26: tie that job to running on xenial instead of bionic. though I'm not sure why that isn't happening globally. maybe someone involved in the migration can take a closer look. gmann? | 14:12 |
| dirk | frickler: that looks like an issue on the mirror | 14:12 |
| dirk | those ~.tmp files are created when rsync is running | 14:12 |
| dirk | frickler: so did it finish with the leap 15 mirroring? we can ignore tw mirroring failure atm | 14:12 |
| dirk | but not having working nodepool images for days is a large problem | 14:13 |
| fungi | i think it's all one afs volume and we only publish the changed state of the volume if the mirror script completes successfully | 14:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a job that runs the upload-git-mirror role https://review.openstack.org/648230 | 14:14 |
| dirk | fungi: ok, so any failure prevents a full update? wow. I didn't realize that | 14:14 |
| dirk | this is a major problem, tumbleweed is far less stable than others | 14:14 |
| fungi | we don't want to publish a broken mirror | 14:14 |
| *** michael-beaver has joined #openstack-infra | 14:15 | |
| fungi | so this method prefers a stale but intact mirror over a partial and possibly broken one | 14:15 |
| fungi | just unfortunate that the last time it successfully mirrored anything it was copying from an empty source, or something like that | 14:15 |
| amansi26 | frickler:the job is running at openstack environment. How should I tie it with xenial? | 14:16 |
| dirk | fungi: are you okay iwth ignoring the rsync errors for tumbleweed? | 14:16 |
| *** rkukura has joined #openstack-infra | 14:16 | |
| frickler | fungi: could you publish the current state manually? seems it will still be in a better state than the old empty mirror | 14:16 |
| dirk | there aren't active users of tumbleweed yet afaik so this is mostly harmless atm | 14:17 |
| fungi | dirk: frickler: i'm still trying to rule out a problem on this end, so let me continue digging first | 14:17 |
| fungi | we do have, for example, a /afs/.openstack.org/mirror/opensuse/tumbleweed/repo/oss/i686/.~tmp~ which is owned 10015:root and with drwx------ perms so it's possible afs is rejecting the attempt to delete | 14:18 |
| fungi | i'm trying to figure out if someone maybe tried to manually update the mirror but didn't use the correct permissions | 14:18 |
| *** mujahidali has quit IRC | 14:18 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Be permissive with tumbleweed mirroring for now https://review.openstack.org/648408 | 14:19 |
| dirk | fungi: ah, its a local thing, ok, sorry, didn't get that. thanks for chasing the problem | 14:19 |
| dirk | fungi: ^^ as a stop gap (untested) | 14:19 |
| fungi | well, it may be both | 14:19 |
| fungi | i don't know yet | 14:19 |
| fungi | so still trying to research, but i need to refresh my memory of kerberos first | 14:20 |
| *** jtomasek has quit IRC | 14:20 | |
| fungi | hrm, nope, i can remove those with the same principal the script is using | 14:21 |
| fungi | so i think the local empty directories may be a symptom of trying and failing to copy them from the remote when it has no permission to allow that | 14:22 |
| fungi | i've cleaned out all four of the local copies of the directories rsync is complaining about | 14:23 |
| fungi | trying to rerun the sync now and seeing if i get the same errors from before | 14:24 |
| *** jtomasek has joined #openstack-infra | 14:26 | |
| fungi | yep, it recreated them with the same opendir errors | 14:26 |
| *** michaelbeaver has joined #openstack-infra | 14:26 | |
| fungi | according to rsync output it creates i686/ and then i686/.~tmp~/ and then complains that it can't read the contents under there | 14:26 |
| *** jamesmcarthur has joined #openstack-infra | 14:27 | |
| dtantsur | folks, is it possible to make zuul retry the jobs when a dependency is updated rather then failing them with the incorrect "failed to merge" error? | 14:28 |
| fungi | dtantsur: that's a topic for #zuul but i think there was discussion of someone wanting to work on making that happen | 14:29 |
| fungi | might have been pabelanger but i don't recall now | 14:29 |
| dtantsur | cool, just wanted to make sure it's on the radar | 14:29 |
| *** michael-beaver has quit IRC | 14:29 | |
| fungi | i mean, the "depends on a change which failed to merge" error technically isn't incorrect | 14:29 |
| fungi | the version of the change it was depending on will no longer be allowed to merge | 14:30 |
| fungi | there is a new version it could depend on instead, and will when the jobs are reenqueued | 14:30 |
| *** whoami-rajat has quit IRC | 14:30 | |
| fungi | but i agree it's a lot more user friendly to restart those builds | 14:30 |
| dtantsur | yeah, from a naive user it doesn't sound quite right | 14:31 |
| fungi | for depends-on specifically. if it's a git parent of course you'd need to rebase | 14:32 |
| fungi | zigo: i need to tackle another chore real quick and then can take a look at the debian mirroring | 14:33 |
| *** Dobroslaw is now known as Dobroslaw_afk | 14:34 | |
| *** e0ne has joined #openstack-infra | 14:38 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a job that runs the upload-git-mirror role https://review.openstack.org/648230 | 14:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Fix typo in upload-git-mirror README https://review.openstack.org/648423 | 14:39 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/openstack-zuul-jobs master: Extend project translation template for Stein https://review.openstack.org/648362 | 14:41 |
| corvus | AJaeger: i manually ran create-cgitrepos on the git backends and replicated the two new projects, so i think everything should be good now | 14:43 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/openstack-zuul-jobs master: Add Stein to stable project translation https://review.openstack.org/648362 | 14:43 |
| *** yboaron_ has quit IRC | 14:44 | |
| *** sthussey has joined #openstack-infra | 14:44 | |
| corvus | fungi: can you +3 this dns change when you have a sec? that should be the last thing needed for docs.opendev https://review.openstack.org/647941 | 14:45 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/openstack-zuul-jobs master: Add Stein to stable project translation https://review.openstack.org/648362 | 14:47 |
| AJaeger | corvus: thanks - yes, I can check them out now | 14:47 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/openstack-zuul-jobs master: Add Stein to stable project translation https://review.openstack.org/648362 | 14:48 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add docs.opendev.org https://review.openstack.org/647941 | 14:50 |
| AJaeger | corvus: do you want to setup the ACL groups for the two new project-config repos as well? | 14:51 |
| corvus | AJaeger: yes i do! | 14:52 |
| fungi | zigo: looks like reprepro is "exporting indices" at the moment... not sure how long it's been saying that | 14:55 |
| zigo | :) | 14:56 |
| frickler | fungi: could you add this other suse mirror update patch to your list, too? https://review.openstack.org/648097/ | 14:56 |
| zigo | It may take a while, though it's finishing ... | 14:56 |
| * frickler will be away for a bit now | 14:56 | |
| zigo | Great. | 14:56 |
| *** ricolin has joined #openstack-infra | 14:59 | |
| *** Lucas_Gray has quit IRC | 15:01 | |
| openstackgerrit | James E. Blair proposed openstack/gertty master: gitrepo DiffFile: convert tab to » + spaces https://review.openstack.org/608154 | 15:03 |
| openstackgerrit | James E. Blair proposed openstack/gertty master: Fix tab expansion in inter-patchset diff https://review.openstack.org/644601 | 15:03 |
| openstackgerrit | James E. Blair proposed openstack/gertty master: Fix change view display of inline comments in python3 https://review.openstack.org/648433 | 15:03 |
| openstackgerrit | Merged openstack/gertty master: gitrepo DiffFile: convert tab to » + spaces https://review.openstack.org/608154 | 15:05 |
| openstackgerrit | Merged openstack/gertty master: Fix tab expansion in inter-patchset diff https://review.openstack.org/644601 | 15:05 |
| openstackgerrit | Merged openstack/gertty master: Fix change view display of inline comments in python3 https://review.openstack.org/648433 | 15:05 |
| *** yboaron_ has joined #openstack-infra | 15:05 | |
| *** tdasilva has joined #openstack-infra | 15:06 | |
| *** bgmccollum has quit IRC | 15:09 | |
| *** hamzy has quit IRC | 15:10 | |
| *** e0ne has quit IRC | 15:14 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Use yaml.safe_load instead of load https://review.openstack.org/648436 | 15:17 |
| *** jtomasek has quit IRC | 15:19 | |
| *** e0ne has joined #openstack-infra | 15:20 | |
| *** jtomasek has joined #openstack-infra | 15:21 | |
| *** jhesketh has joined #openstack-infra | 15:28 | |
| *** michaelbeaver has quit IRC | 15:28 | |
| *** michael-beaver has joined #openstack-infra | 15:29 | |
| *** ykarel is now known as ykarel|away | 15:29 | |
| openstackgerrit | Merged openstack-infra/system-config master: Stop trying to mirror Master openSUSE_Leap_42.3 https://review.openstack.org/648097 | 15:33 |
| *** jhesketh has quit IRC | 15:34 | |
| *** kgiusti has left #openstack-infra | 15:35 | |
| *** hamzy has joined #openstack-infra | 15:35 | |
| *** quiquell is now known as quiquell|off | 15:37 | |
| *** wolverineav has joined #openstack-infra | 15:38 | |
| *** e0ne has quit IRC | 15:40 | |
| *** wolverineav has quit IRC | 15:42 | |
| *** ykarel|away has quit IRC | 15:43 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Fix redirect loop in docs promotion https://review.openstack.org/648446 | 15:43 |
| corvus | AJaeger: ^ does that look like an improvement? | 15:43 |
| *** pgaxatte has quit IRC | 15:45 | |
| corvus | i'm going to try making that change manually in afs to verify | 15:46 |
| mnaser | is opendev.org search broken by any chance | 15:50 |
| corvus | mnaser: not afaik | 15:50 |
| mnaser | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L39 => https://opendev.org/openstack/openstack-ansible-os_neutron/search?q=neutron_package_state | 15:50 |
| mnaser | maybe it works on word level matching or the indexing isn't the greatest? | 15:51 |
| mnaser | can't match `neutron_package_state:` either | 15:51 |
| corvus | mnaser: oh, hrm. i did run into another similar problem the other day | 15:51 |
| mnaser | I haven't had any successful searches yet | 15:52 |
| corvus | i think we'll have to dig into the underlying search index to see if we can spot the problem | 15:52 |
| tosky | oh, similar to the issue that I hit | 15:54 |
| tosky | maybe the same :) | 15:54 |
| dmsimard | infra-root: upload-git-mirror is good to go I think but we need to land the patches so I can properly test it in post with a secret https://review.openstack.org/#/q/topic:upload-git-mirror | 15:55 |
| openstackgerrit | Merged openstack-infra/system-config master: Be permissive with tumbleweed mirroring for now https://review.openstack.org/648408 | 15:59 |
| *** yboaron_ has quit IRC | 15:59 | |
| *** yboaron_ has joined #openstack-infra | 16:00 | |
| *** udesale has quit IRC | 16:01 | |
| *** jbadiapa has quit IRC | 16:02 | |
| gmann | frickler: fungi you mean openstack-tox-docs job on ocata using the bionic ? | 16:02 |
| *** diablo_rojo has joined #openstack-infra | 16:02 | |
| gmann | frickler: fungi i am not sure why openstack-tox-docs is being used for ocata also. what we can do is to extend the xenial pin till ocata here - https://github.com/openstack-infra/openstack-zuul-jobs/blob/b65b5bdfa2062ea7473944c50fba53895dc5337b/zuul.d/jobs.yaml#L679 | 16:06 |
| *** janki has quit IRC | 16:07 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Add nofollow attribute to user-submitted links https://review.openstack.org/648322 | 16:07 |
| mordred | dmsimard: looks great to me - thanks for the great work on that! | 16:08 |
| dmsimard | itshappening.gif | 16:10 |
| fungi | gmann: i'm having to refresh my memory on that, but i believe the reason it's like that is that the openstack-tox-docs job was introduced in the rocky cycle so shouldn't have been added to any older branches than stable/rocky anyway (those use build-openstack-sphinx-docs instead) | 16:10 |
| gmann | fungi: exactly . i am surprise openstack-tox-docs is being used in ocata also | 16:11 |
| fungi | ((and build-openstack-sphinx-docs already sets nodeset: ubuntu-xenial for all branches) | 16:11 |
| fungi | gmann: where did you see it run on ocata changes? | 16:12 |
| gmann | fungi: https://review.openstack.org/#/c/646690/ | 16:12 |
| fungi | i feel like i'm probably forgetting some earlier context from discussions which took place while i was still asleep | 16:12 |
| *** gyee has joined #openstack-infra | 16:12 | |
| *** chandankumar is now known as raukadah | 16:13 | |
| *** jtomasek has quit IRC | 16:13 | |
| fungi | gmann: http://logs.openstack.org/90/646690/1/check/openstack-tox-docs/59b4acb/zuul-info/inventory.yaml shows the inheritence path | 16:15 |
| fungi | gmann: aha! says it's coming from the docs-on-readthedocs template https://opendev.org/openstack-infra/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L876 | 16:17 |
| fungi | i guess we have that set to use openstack-tox-docs on any branch for which it's applied | 16:18 |
| gmann | humm | 16:18 |
| *** wolverineav has joined #openstack-infra | 16:18 | |
| fungi | so options are to make variants which use build-openstack-sphinx-docs on older branches and openstack-tox-docs on newer branches, or just roll the dice and assume openstack-tox-docs will work on older branches if it's set to use xenial, or remove docs-on-readthedocs from branches older than stable/rocky | 16:20 |
| fungi | there are likely more options i've not thought of, those were just off the top of my head | 16:20 |
| gmann | i think just make openstack-tox-docs running with xenial on older branch should be ok for safer side | 16:22 |
| fungi | in that case we need to update the two existing variants to take additional branches into account | 16:23 |
| gmann | yeah | 16:23 |
| fungi | i'm in meetings at the moment but can review if you propose it | 16:23 |
| gmann | ok i can push the change | 16:23 |
| fungi | thanks! | 16:24 |
| *** e0ne has joined #openstack-infra | 16:27 | |
| *** mattw4 has joined #openstack-infra | 16:30 | |
| *** ramishra has quit IRC | 16:32 | |
| *** whoami-rajat has joined #openstack-infra | 16:32 | |
| *** ginopc has joined #openstack-infra | 16:36 | |
| *** wolverineav has quit IRC | 16:37 | |
| *** dayou has quit IRC | 16:37 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Use xenial for openstack-tox-docs for all older stable branch https://review.openstack.org/648462 | 16:38 |
| gmann | fungi: ^^ | 16:38 |
| *** helenafm has quit IRC | 16:40 | |
| *** ginopc has quit IRC | 16:45 | |
| *** smcginnis is now known as smcginnis_pto | 16:48 | |
| *** jtomasek has joined #openstack-infra | 16:49 | |
| *** ginopc has joined #openstack-infra | 16:51 | |
| *** dayou has joined #openstack-infra | 16:55 | |
| *** ginopc has quit IRC | 16:55 | |
| *** lucasagomes has quit IRC | 17:00 | |
| fungi | zigo: reprepro finished running, so now we're just waiting on the afs vos release to complete (started at 16:48z) | 17:02 |
| fungi | not sure how long it'll take to sync the volumes | 17:03 |
| *** psachin has quit IRC | 17:05 | |
| *** trown is now known as trown|lunch | 17:07 | |
| *** e0ne has quit IRC | 17:07 | |
| *** e0ne has joined #openstack-infra | 17:08 | |
| *** ginopc has joined #openstack-infra | 17:12 | |
| *** e0ne has quit IRC | 17:17 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:17 | |
| *** chrisyang_0660 has quit IRC | 17:18 | |
| corvus | fungi: there's a suggestion on https://github.com/go-gitea/gitea/issues/6450 that we should redirect short shas to long ones. what do you think about that? i was thinking of replying with http://paste.openstack.org/raw/748546/ but i think you may be better placed to think about use cases. | 17:19 |
| *** e0ne has joined #openstack-infra | 17:21 | |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** ccamacho has quit IRC | 17:23 | |
| *** rpittau is now known as rpittau|afk | 17:24 | |
| dmsimard | there are failing opensuse jobs in the openstack-zuul-jobs gate :( | 17:26 |
| *** ricolin has quit IRC | 17:26 | |
| fungi | corvus: thanks for the heads up, i also just commented on the issue, but i agree with your points as well, it does result in more work manually shortening the urls again after browsing | 17:27 |
| dmsimard | they're failing on validate-host: http://logs.openstack.org/35/643435/12/gate/openstack-infra-base-integration-opensuse15/21ebf0e/ara-report/result/e9e06976-5e00-4bae-a760-0f4a66fba9db/ | 17:27 |
| fungi | dmsimard: yes, i think we may be nearly through the problems with the opensuse mirror | 17:27 |
| dmsimard | fungi: doesn't look like a mirror issue to me -- dns resolution failure on git.o.o ? | 17:27 |
| fungi | the problem seems to be that they incorrectly ran an empty opensuse package repository briefly which we then mirrored | 17:28 |
| fungi | as a result our opensuse images are built with an older unbound with stale dnssec root keys | 17:28 |
| dmsimard | I see | 17:29 |
| dmsimard | Should we remove those jobs from the gate while this is ironed out ? | 17:29 |
| fungi | so when the job node booted from that tries to resolve things, unbound filters out the records it's unable to validate | 17:29 |
| dmsimard | Would really love to land the upload-git-mirror stack :( | 17:29 |
| fungi | dmsimard: i'm fine with removing the opensuse jobs until things are back to normal. we shouldn't let issues with those block us from getting things merged | 17:30 |
| dmsimard | ack | 17:30 |
| *** jpich has quit IRC | 17:30 | |
| dmsimard | fungi: is it just opensuse15 ? or tumbleweed too ? | 17:34 |
| dmsimard | It looks like it's just opensuse15 that is failing right now but tumbleweed was non-voting already | 17:35 |
| fungi | dmsimard: opensuse15 i think | 17:35 |
| fungi | though they all share a common afs volume so we need to be able to mirror them all to update opensuse15 | 17:36 |
| fungi | and they have since broken something with the tumbleweed mirror by leaving tempdirs in it which aren't world-readable, so rsync was choking on them | 17:36 |
| fungi | but we merged an override for that an hour or two ago | 17:36 |
| dmsimard | ok I'll leave tumbleweed non-voting for now | 17:37 |
| dmsimard | I'll put up a patch to re-enable them so we can recheck to see if the issue has been resolved | 17:38 |
| *** ricolin has joined #openstack-infra | 17:38 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Temporarily set opensuse-15 and opensuse-tumbleweed jobs to non-voting https://review.openstack.org/648481 | 17:42 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Re-enable voting for opensuse15, opensuse423 and opensuse-tumbleweed https://review.openstack.org/648482 | 17:42 |
| *** wolverineav has joined #openstack-infra | 17:44 | |
| *** wolverineav has quit IRC | 17:44 | |
| *** e0ne has quit IRC | 17:45 | |
| *** wolverineav has joined #openstack-infra | 17:46 | |
| *** ricolin has quit IRC | 17:46 | |
| *** wolverineav has quit IRC | 17:51 | |
| *** wolverineav has joined #openstack-infra | 17:52 | |
| *** wolverineav has quit IRC | 17:52 | |
| *** wolverineav has joined #openstack-infra | 17:53 | |
| *** wolverineav has quit IRC | 17:53 | |
| *** wolverineav has joined #openstack-infra | 17:53 | |
| *** jbadiapa has joined #openstack-infra | 17:54 | |
| *** ykarel|away has joined #openstack-infra | 17:54 | |
| *** wolverineav has quit IRC | 18:00 | |
| *** wolverineav has joined #openstack-infra | 18:01 | |
| *** wolverineav has quit IRC | 18:01 | |
| *** ginopc has quit IRC | 18:01 | |
| *** wolverineav has joined #openstack-infra | 18:01 | |
| *** derekh has quit IRC | 18:02 | |
| *** gfidente has quit IRC | 18:05 | |
| *** trown|lunch is now known as trown | 18:06 | |
| *** dtantsur is now known as dtantsur|afk | 18:08 | |
| *** jpena is now known as jpena|off | 18:08 | |
| dmsimard | ^ hrm, the opensuse jobs don't actually run if we only modify zuul.yaml, I'll fix that | 18:11 |
| *** kopecmartin is now known as kopecmartin|off | 18:11 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: test patch for opensuse jobs https://review.openstack.org/648494 | 18:13 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add job and integration tests for the upload-git-mirror role https://review.openstack.org/643435 | 18:16 |
| AJaeger | config-core, here're two stein patches : https://review.openstack.org/648362 for adding translations and https://review.openstack.org/647915 for periodic jobs. Please review | 18:18 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Temporarily set opensuse-15 and opensuse-tumbleweed jobs to non-voting https://review.openstack.org/648481 | 18:23 |
| AJaeger | thanks, corvus for the reviews! | 18:25 |
| corvus | i'm performing an online full-reconfigure in zuul; that should cause it to pick up the new tenant config without losing any data, but it'll appear frozen for a few minutes. | 18:26 |
| fungi | thanks for the heads up (and for adding new tenants!) | 18:26 |
| *** quiquell|off is now known as quiquell | 18:28 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: stage-output: fix the archiving of all files https://review.openstack.org/645239 | 18:29 |
| corvus | it's done: http://zuul.opendev.org/tenants | 18:29 |
| pabelanger | \o/ | 18:29 |
| corvus | i'll start in on some changes for those after lunch | 18:30 |
| AJaeger | dmsimard: question on https://review.openstack.org/#/c/643435 - I think the change is not complete | 18:30 |
| dmsimard | AJaeger: o-z-j can't set jobs to run on z-j, it needs to go in project-config | 18:30 |
| dmsimard | but I do need to put up a patch for that | 18:30 |
| *** quiquell is now known as quiquell|off | 18:30 | |
| dmsimard | the job was tested with a child patch in z-j | 18:31 |
| AJaeger | dmsimard: Ah, should run on z-j - ok. Please push the patch... | 18:31 |
| *** jtomasek has quit IRC | 18:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Add job to test upload-git-mirror on zuul-jobs https://review.openstack.org/648501 | 18:34 |
| dmsimard | AJaeger: ^ | 18:34 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add Stein to stable project translation https://review.openstack.org/648362 | 18:38 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add stable/stein to periodic-stable templates https://review.openstack.org/647915 | 18:38 |
| AJaeger | dmsimard: +2 | 18:46 |
| AJaeger | dmsimard: thanks | 18:46 |
| *** jamesmcarthur_ has quit IRC | 18:47 | |
| amansi26 | gmann:Ian Wienand proposed a patch set with change ID [I277cbdccf8ea0679d53e449474d7aa8911c999ef] which fails at Zuul check with an error build/temp.linux-x86_64-2.7/_openssl.c:493:10: fatal error: openssl/opensslv.h: No such file or directory. Can anyone help me resolving the issue? | 18:52 |
| *** MasterofJOKers has quit IRC | 18:52 | |
| *** yamamoto has joined #openstack-infra | 18:54 | |
| gmann | amansi26: this can fix - https://review.openstack.org/#/c/648462/ | 18:54 |
| amansi26 | gmann:Thanks | 18:55 |
| AJaeger | gmann: are you going to update the change? I'd love to +2 it ;) | 18:57 |
| gmann | AJaeger: yeah, doing that, i missed the explanation and description part. | 18:57 |
| *** yamamoto has quit IRC | 18:58 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Use xenial for openstack-tox-docs for all older stable branch https://review.openstack.org/648462 | 19:01 |
| gmann | AJaeger: fungi ^^ | 19:01 |
| *** ykarel|away has quit IRC | 19:05 | |
| *** wolverineav has quit IRC | 19:07 | |
| *** wolverineav has joined #openstack-infra | 19:08 | |
| *** wolverineav has quit IRC | 19:13 | |
| *** apetrich has quit IRC | 19:16 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use xenial for openstack-tox-docs for all older stable branch https://review.openstack.org/648462 | 19:19 |
| *** kjackal has quit IRC | 19:21 | |
| *** priteau has quit IRC | 19:27 | |
| dmsimard | doh... | 19:32 |
| dmsimard | the opensuse issue failed the opensuse423 job too and hadn't set that one to non-voting | 19:33 |
| *** e0ne has joined #openstack-infra | 19:33 | |
| fungi | fun | 19:35 |
| fungi | well, the opensuse mirror finally successfully updated so i think we just need new images now | 19:35 |
| fungi | perhaps i can delete the existing ones and get a new build going for them to make that happen sooner | 19:35 |
| fungi | or maybe it's already underway if the builds were actually failing | 19:36 |
| fungi | though i don't think they were | 19:36 |
| *** rcernin has quit IRC | 19:36 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 19:40 | |
| fungi | looks like we built new opensuse-150 roughly 1.5 hours ago and new opensuse-423 ~3 hours ago | 19:41 |
| fungi | probably not recent enough to have working unbound | 19:42 |
| fungi | deleted the oldest of each of the three opensuse images types now | 19:42 |
| redrobot | ohai infra friends! | 19:44 |
| redrobot | i was wondering what version of Fedora is used in fedora-latest gate jobs? | 19:45 |
| pabelanger | redrobot: fedora-29 | 19:45 |
| redrobot | pabelanger, server? | 19:45 |
| pabelanger | no, we build them from scratch... more like fedora-cloud | 19:46 |
| pabelanger | however, you have the ability to install anything on the nodes | 19:46 |
| redrobot | ack, thanks pabelanger | 19:46 |
| pabelanger | https://nb01.openstack.org/fedora-29-0000038652.log is an example build log | 19:47 |
| *** wolverineav has joined #openstack-infra | 19:48 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** wolverineav has quit IRC | 19:53 | |
| *** bhavikdbavishi has quit IRC | 19:55 | |
| *** jtomasek has joined #openstack-infra | 19:57 | |
| *** e0ne has quit IRC | 20:00 | |
| *** wolverineav has joined #openstack-infra | 20:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Gerritbot: add zuul/project-config to #zuul https://review.openstack.org/648524 | 20:03 |
| openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Enabling suggestions on all user fields https://review.openstack.org/648525 | 20:06 |
| *** pcaruana has quit IRC | 20:08 | |
| *** kjackal has joined #openstack-infra | 20:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add some projects to zuul zuul tenant https://review.openstack.org/648527 | 20:20 |
| corvus | pabelanger, AJaeger: ^ 2 changes are next steps for zuul tenant | 20:21 |
| *** efried has quit IRC | 20:22 | |
| *** amansi26 has quit IRC | 20:23 | |
| *** jtomasek has quit IRC | 20:24 | |
| *** efried has joined #openstack-infra | 20:26 | |
| *** wolverineav has quit IRC | 20:32 | |
| *** wolverineav has joined #openstack-infra | 20:33 | |
| AJaeger | corvus: can we have zuul-sphinx in both tenants? Or will we get tests twice now? | 20:35 |
| AJaeger | pabelanger: could you +2A https://review.openstack.org/#/c/648524/ so that we can merge 527 which is on top of it, please? | 20:35 |
| corvus | AJaeger: we'll get both. which will be really weird, and we generally shouldn't do it. but zuul-sphinx moves slowly enough that if we're unable to land changes for a day or two, it's no big deal. we could go ahead and take the plunge and remove it from openstack though, i guess there's no reason not to. i'll do that. | 20:36 |
| tonyb | I ran my 'update openstack script' and it failed with: | 20:36 |
| tonyb | Cloning into 'zuul/project-config'... | 20:36 |
| tonyb | fatal: remote error: access denied or repository not exported: /zuul/project-config | 20:36 |
| tonyb | when will the new namespaces/orgs be accessible? | 20:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add some projects to zuul zuul tenant https://review.openstack.org/648527 | 20:37 |
| *** eharney has quit IRC | 20:37 | |
| corvus | AJaeger, pabelanger: ^ updated | 20:37 |
| corvus | tonyb: can you clarify which system reported an error? | 20:37 |
| *** wolverineav has quit IRC | 20:37 | |
| AJaeger | tonyb: I was able t oclone that repo earlier today - how can we reproduce? | 20:37 |
| corvus | (due to the transition, that could be an error on one of 17 different hosts) | 20:39 |
| tonyb | AJaeger: This was riagh as I gave up last night so it might be fixed this morning | 20:40 |
| corvus | tonyb: yes, there was an error, it should be fixed as of a few hours ago. | 20:41 |
| tonyb | AJaeger, corvus: I'll try again and then replort back | 20:41 |
| tonyb | I *did* just clone opendev/project-config so fingers crossed y'all already fixed it | 20:41 |
| corvus | tonyb: the error was specifically with zuul/project-config (we previously had a hidden zuul namespace on the git farm; it was disused, but we still needed to clean it up) | 20:42 |
| corvus | i'm guessing we'll need to wait till the end of your script to confirm :) | 20:43 |
| *** efried has quit IRC | 20:44 | |
| * tonyb adds a sort -r | 20:46 | |
| tonyb | success | 20:46 |
| AJaeger | yeah | 20:46 |
| tonyb | Thanks all, sorry for the noise | 20:47 |
| corvus | tonyb: sorry for the screwup :) | 20:47 |
| *** rkukura has quit IRC | 20:48 | |
| tonyb | corvus: no biggie | 20:48 |
| *** wolverineav has joined #openstack-infra | 20:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Gerritbot: add zuul/project-config to #zuul https://review.openstack.org/648524 | 20:54 |
| *** hamzy has quit IRC | 20:54 | |
| johnsom | I am seeing some POST_FAILUREs claiming the host identification changed when it was doing rsync over ssh. Thought I would mention it. http://logs.openstack.org/04/648504/1/check/openstack-tox-py27/fd76640/job-output.txt.gz#_2019-03-28_18_56_13_817133 | 20:55 |
| fungi | thanks johnsom, that's almost always a rogue vm in the provider which nova has lost track of winning an arp fight against the node we booted there | 20:56 |
| johnsom | Ha, interesting | 20:57 |
| *** efried has joined #openstack-infra | 20:58 | |
| fungi | yes, odds are 198.72.124.72 is an old vm from some tenant in that same region which is no longer known to nova so the ip address got recycled and assigned to the one we used for that job | 20:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Add some projects to zuul zuul tenant https://review.openstack.org/648527 | 20:58 |
| fungi | when you use virtual machines at the scale we do, these systemic issues in openstack start to become readily apparent | 20:59 |
| fungi | in this case http://logs.openstack.org/04/648504/1/check/openstack-tox-py27/fd76640/zuul-info/inventory.yaml says it ran in inap-mtl01 so mgagne probably wants to hunt it down and clean it up when he's around next | 20:59 |
| johnsom | Yeah, we had to put some code in to handle zombie instances as well. I had hoped it was limited to just certain deployments. | 21:01 |
| fungi | yes, only affects deployments running openstack | 21:01 |
| johnsom | The team had a good time with the commit messages, "Implement zombie killer", etc. | 21:01 |
| *** openstackgerrit has quit IRC | 21:07 | |
| *** trown is now known as trown|outtypewww | 21:08 | |
| *** efried has quit IRC | 21:09 | |
| *** efried has joined #openstack-infra | 21:23 | |
| *** wolverineav has quit IRC | 21:24 | |
| *** wolverineav has joined #openstack-infra | 21:25 | |
| *** wolverineav has quit IRC | 21:30 | |
| *** jcoufal has quit IRC | 21:31 | |
| *** rkukura has joined #openstack-infra | 21:32 | |
| *** mattw4 has quit IRC | 21:36 | |
| *** mattw4 has joined #openstack-infra | 21:37 | |
| fungi | okay, so the first opensuse-423 build failed (i haven't checked the error log yet) but we have successful opensuse-150 and opensuse-tumbleweed builds as of roughly half an hour ago | 21:39 |
| fungi | and another opensuse-423 is building as of a few minutes now | 21:40 |
| *** wolverineav has joined #openstack-infra | 21:40 | |
| fungi | hrm, though looks like a couple of them failed in a row so prognosis is not good | 21:40 |
| fungi | as soon as i free myself up from my current task i'll check the log | 21:41 |
| *** openstackgerrit has joined #openstack-infra | 21:43 | |
| fungi | yeah, now several have failed at around the 15-minute mark | 21:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Forward artifacts to child jobs within buildset https://review.openstack.org/642857 | 21:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Perform per repo locking on the executor https://review.openstack.org/635495 | 21:43 |
| *** rcernin has joined #openstack-infra | 21:44 | |
| *** kjackal has quit IRC | 21:44 | |
| *** michael-beaver has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack-infra/zuul master: Parallelize update threads https://review.openstack.org/635496 | 21:45 |
| *** rcernin has quit IRC | 21:55 | |
| ianw | zigo: http://mirror.iad.rax.openstack.org/debian/dists/buster/ | 21:58 |
| fungi | yep, that completed a few hours ago | 21:58 |
| fungi | i joined your screen session for a good chunk of the day to keep an eye on the progress | 21:58 |
| ianw | cool; ssh keepalive seems to keep my connection through this cgnat i'm behind for a while, but it didn't survive overnight | 22:00 |
| fungi | likely because we don't also set keepalive on the server | 22:01 |
| fungi | and it almost certainly went long pauses of sending nothing | 22:02 |
| *** rcernin has joined #openstack-infra | 22:04 | |
| *** yamamoto has joined #openstack-infra | 22:08 | |
| *** wolverineav has quit IRC | 22:08 | |
| *** wolverineav has joined #openstack-infra | 22:10 | |
| openstackgerrit | Ian Wells proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-networking-vpp-smoke https://review.openstack.org/648549 | 22:11 |
| *** wolverineav has quit IRC | 22:14 | |
| openstackgerrit | Ian Wells proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-networking-vpp-smoke https://review.openstack.org/648549 | 22:17 |
| corvus | fungi: "topic:letsencrypt" has 2x+2 on the top 3 changes which ianw needs before performing final testing if you want to push those through | 22:27 |
| *** yamamoto has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add monitoring for filesystems other than root https://review.openstack.org/648365 | 22:35 |
| openstackgerrit | Ian Wells proposed openstack-infra/project-config master: Remove legacy-networking-vpp-smoke https://review.openstack.org/648553 | 22:35 |
| *** yamamoto has joined #openstack-infra | 22:36 | |
| *** yamamoto has quit IRC | 22:39 | |
| fungi | corvus: thanks, will take a look now | 22:40 |
| *** wolverineav has joined #openstack-infra | 22:40 | |
| *** yamamoto has joined #openstack-infra | 22:43 | |
| *** wolverineav has quit IRC | 22:45 | |
| *** mattw4 has quit IRC | 22:45 | |
| *** yamamoto has quit IRC | 22:52 | |
| *** whoami-rajat has quit IRC | 22:52 | |
| *** yamamoto has joined #openstack-infra | 22:54 | |
| *** wolverineav has joined #openstack-infra | 22:56 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| *** tkajinam has joined #openstack-infra | 22:59 | |
| *** wolverineav has quit IRC | 22:59 | |
| *** wolverineav has joined #openstack-infra | 23:00 | |
| *** wolverineav has quit IRC | 23:01 | |
| *** wolverineav has joined #openstack-infra | 23:01 | |
| *** yamamoto has quit IRC | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:04 | |
| *** wolverineav has quit IRC | 23:07 | |
| *** wolverineav has joined #openstack-infra | 23:08 | |
| *** aaronsheffield has quit IRC | 23:08 | |
| *** wolverineav has quit IRC | 23:09 | |
| *** wolverineav has joined #openstack-infra | 23:09 | |
| *** yamamoto has quit IRC | 23:14 | |
| *** jamesmcarthur has joined #openstack-infra | 23:14 | |
| *** slaweq has quit IRC | 23:17 | |
| *** adriant has quit IRC | 23:34 | |
| ianw | thanks, with that i can stick some TXT records in acme.opendev.org and test the cert generation directly | 23:39 |
| *** jamesmcarthur has quit IRC | 23:42 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add fake DNSSEC keys for zones https://review.openstack.org/647952 | 23:42 |
| openstackgerrit | Merged openstack-infra/system-config master: Use adns group in base.yaml https://review.openstack.org/647993 | 23:42 |
| openstackgerrit | Merged openstack-infra/system-config master: master-nameserver: Add unmanaged domains; add acme.opendev.org https://review.openstack.org/641155 | 23:42 |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/project-config master: Add openstack/freezer-tempest-plugin build job https://review.openstack.org/648559 | 23:43 |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| *** jamesmcarthur has quit IRC | 23:48 | |
| *** adriant has joined #openstack-infra | 23:49 | |
| *** yamamoto has joined #openstack-infra | 23:53 | |
| *** yamamoto has quit IRC | 23:55 | |
| *** gyee has quit IRC | 23:57 | |
| *** jhesketh has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!