| *** markvoelker has quit IRC | 00:00 | |
| jamesmcarthur | fungi, rpioso: The drivers page is updated. Moving forward, drivers with multiple vendors will show up as expected: https://www.openstack.org/marketplace/drivers/#project=ironic%20(bare%20metal%20provisioning) | 00:00 |
|---|---|---|
| rpioso | jamesmcarthur: \o/ It looks great. Thank you so much, Jimmy! | 00:02 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:02 |
| jamesmcarthur | rpioso: np. happy to help! | 00:03 |
| cloudnull | ianw updated to remove the fallback install on upstream | 00:03 |
| ianw | cloudnull: readme variable can go then too? | 00:05 |
| *** hwoarang has joined #openstack-infra | 00:05 | |
| cloudnull | yup, one sec :) | 00:05 |
| ianw | i'm open to discussion too :) maybe it just makes sense to have it variable-ised with turned off by default? so if you know you've got it on a base image, you enable purging all the old versiosn? | 00:06 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add multi-distro support to install-docker https://review.opendev.org/664455 | 00:11 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:11 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 00:12 |
| cloudnull | ianw I honestly think we should expect the CI images to come clean and for the tasks to fail in expected ways instead of masking failure, initially I was trying to implement whats in the docker docs, for good measure, but I now think the role should simply expect a clean target. | 00:13 |
| *** michael-beaver has quit IRC | 00:17 | |
| *** jcoufal has joined #openstack-infra | 00:18 | |
| *** ianychoi_ has joined #openstack-infra | 00:18 | |
| *** jcoufal has quit IRC | 00:20 | |
| *** ianychoi has quit IRC | 00:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: linters: ensure openstack/inventory.yaml parses https://review.opendev.org/664751 | 00:25 |
| *** rkukura has quit IRC | 00:26 | |
| cloudnull | thanks ianw and clarkb for the reviews ! | 00:27 |
| cloudnull | one more on https://review.opendev.org/#/c/664711/ and im in like flin :) | 00:28 |
| fungi | in like kevin flynn? | 00:29 |
| ianw | Errol i believe .. i thought it was an Australian thing. or you watch too much eevblog | 00:33 |
| cloudnull | something like that :) | 00:34 |
| * cloudnull hopes one day to make it into tron | 00:35 | |
| cloudnull | **the grid and meet tron | 00:35 |
| *** jamesmcarthur has quit IRC | 00:39 | |
| fungi | yeah, i was twisting it into a tron reference | 00:39 |
| *** mhu has quit IRC | 00:40 | |
| fungi | one of the films which molded my youth | 00:40 |
| *** pabelanger has quit IRC | 00:40 | |
| *** jpena|off has quit IRC | 00:40 | |
| cloudnull | 100% | 00:42 |
| cloudnull | i actually liked the new one, was kinda bummed they didn't make a second one | 00:43 |
| openstackgerrit | Merged opendev/system-config master: Fix syntax for new mirror hosts https://review.opendev.org/664745 | 00:44 |
| *** mhu has joined #openstack-infra | 00:45 | |
| fungi | yes, the recent-ish sequel was quite good, though i still have a soft spot for the original, now-dated but then-groundbreaking computer animation, wendy carlos soundtrack and all | 00:45 |
| openstackgerrit | Merged zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 00:46 |
| fungi | at least daft punk did an excellent job from drawing on some original carlos themes | 00:46 |
| fungi | (then again, i'm a long-time daft punk fan going back to their early loop stuff on homework) | 00:47 |
| *** zzzeek has quit IRC | 00:47 | |
| *** zzzeek has joined #openstack-infra | 00:47 | |
| *** jpena|off has joined #openstack-infra | 00:48 | |
| cloudnull | ++ the original was super good! | 00:49 |
| * cloudnull may rewatch that tonight (cant decide on that or rambo first blood III) | 00:49 | |
| *** jcoufal has joined #openstack-infra | 00:50 | |
| *** zhurong has quit IRC | 00:52 | |
| *** armax has quit IRC | 00:52 | |
| *** markvoelker has joined #openstack-infra | 00:57 | |
| *** happyhemant has quit IRC | 01:03 | |
| *** diablo_rojo has joined #openstack-infra | 01:05 | |
| *** Goneri has quit IRC | 01:13 | |
| *** rh-jelabarre has quit IRC | 01:22 | |
| *** bobh has joined #openstack-infra | 01:26 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add certificates for IAD/ORD opendev.org mirrors https://review.opendev.org/664755 | 01:26 |
| *** jcoufal has quit IRC | 01:26 | |
| *** dayou has quit IRC | 01:27 | |
| *** rosmaita has left #openstack-infra | 01:28 | |
| *** igordc has quit IRC | 01:28 | |
| *** dayou has joined #openstack-infra | 01:29 | |
| *** markvoelker has quit IRC | 01:31 | |
| *** dave-mccowan has quit IRC | 01:34 | |
| *** dave-mccowan has joined #openstack-infra | 01:40 | |
| corvus | this account of the sfx in legacy is very good: http://www.talisman.org/~erlkonig/misc/tron-legacy-effects-blog/ | 01:40 |
| fungi | heh, emacs shell | 01:43 |
| fungi | great article | 01:44 |
| *** hwoarang has quit IRC | 01:45 | |
| *** hwoarang has joined #openstack-infra | 01:47 | |
| *** yamamoto has joined #openstack-infra | 01:47 | |
| *** spsurya has joined #openstack-infra | 01:52 | |
| *** gregoryo has joined #openstack-infra | 01:54 | |
| *** zzzeek has quit IRC | 01:57 | |
| *** zzzeek has joined #openstack-infra | 01:58 | |
| *** pcrews has quit IRC | 02:05 | |
| *** pcrews has joined #openstack-infra | 02:09 | |
| *** zzzeek has quit IRC | 02:13 | |
| *** zzzeek has joined #openstack-infra | 02:16 | |
| *** pabelanger has joined #openstack-infra | 02:26 | |
| *** markvoelker has joined #openstack-infra | 02:27 | |
| *** diablo_rojo has quit IRC | 02:46 | |
| *** hongbin has joined #openstack-infra | 02:47 | |
| *** yamamoto has quit IRC | 02:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:49 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 02:54 |
| *** markvoelker has quit IRC | 03:01 | |
| *** ramishra has joined #openstack-infra | 03:10 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| openstackgerrit | Liping Mao proposed openstack/logstash-filters master: ERROR TRACE log will merged with previous log https://review.opendev.org/664770 | 03:13 |
| auristor | hi fungi: openafs does not interact with the Linux kernel afs (kafs) implementation. openafs and kafs are completely independent. | 03:20 |
| openstackgerrit | Merged opendev/system-config master: Add certificates for IAD/ORD opendev.org mirrors https://review.opendev.org/664755 | 03:22 |
| auristor | The Heimdal libkafs also has no relationship to Linux kernel afs (kafs). Heimdal libkafs implements a subset of the OpenAFS pioctls necessary to set tokens | 03:22 |
| *** jcoufal has joined #openstack-infra | 03:22 | |
| *** ykarel|away has joined #openstack-infra | 03:25 | |
| auristor | Linux kernel afs doesn't use pioctls to set tokens but instead uses keyrings | 03:25 |
| auristor | Debian and Ubuntu build their kernels with AF_RXRPC and KAFS but as far as I am aware none of the userland tooling for enabling the kernel modules or activating the /afs virtual mount is present. | 03:27 |
| auristor | The current status on Fedora is that up until last week the publication of the kafs-client package was blocked because creating the /afs directory for the mount was forbidden. The packaging committee ruled last week that in rawhide and F31 that /afs would be created as part of the filesystem package. | 03:28 |
| auristor | But that means that we do not have a solution for F29 or F30. | 03:29 |
| *** psachin has joined #openstack-infra | 03:29 | |
| *** jcoufal has quit IRC | 03:30 | |
| auristor | The Debian openafs packages create /afs. Since Ubuntu uses the Debian openafs packages as upstream they are sure to do the same. | 03:30 |
| ianw | auristor: yeah, i just got fedora kafs working in our CI ... http://logs.openstack.org/74/623974/10/check/system-config-zuul-role-integration-fedora/befd359/ara-report/ | 03:31 |
| ianw | it mounts openafs.org and can see the file | 03:31 |
| ianw | a file | 03:31 |
| auristor | Jonathan Billings provides a full set of the required packages https://copr.fedorainfracloud.org/coprs/jsbillings/kafs/packages/ | 03:32 |
| ianw | yep, using that ... last time i tried it didn't work, but a bit of fiddling it has now | 03:32 |
| auristor | note that there are byte range locking changes for compatibility with openafs that were not accepted by Linus until the 5.2 merge window https://www.phoronix.com/scan.php?page=news_item&px=AFS-For-Linux-5.2 | 03:34 |
| ianw | ok ... fixed letsencrypt certs ... so ... | 03:35 |
| ianw | https://mirror.iad.rax.opendev.org is now running kafs | 03:35 |
| ianw | auristor: ^ (note openDEV.org, not openSTACK.org) | 03:36 |
| auristor | There is a pending request to add /afs to FHS 3.1 which we believe will be approved. If so, we can push more Linux distributions to add /afs to the base installation. | 03:36 |
| auristor | what is opendev.org? | 03:36 |
| ianw | auristor: it's a whole thing where we're moving to a much more generic hosting approach, where openstack is one of many sub-projects | 03:37 |
| auristor | is opendev.org another afs cell? | 03:37 |
| ianw | yes, but there is a long transition as there's a *lot* of stuff to migrate in various ways. this mirror is using the openstack.org cell and all the various mirror volumes under that | 03:38 |
| ianw | auristor: to catch you up, we moved our mirror hosts to bionic and the 1.8.0~pre5 version of openafs it comes with. in testing, we're seeing what appears to be corrupt files cached and served | 03:39 |
| ianw | the blame is *not* clearly placed on openafs yet ... we're investigating. one avenue we're going to try is running a kafs mirror in parallel too, and see if we have any similar sorts of issues | 03:40 |
| *** dayou has quit IRC | 03:40 | |
| ianw | we are right in the middle of it all -- terse notes being kept @ https://etherpad.openstack.org/p/opendev-mirror-afs | 03:40 |
| *** dayou has joined #openstack-infra | 03:41 | |
| auristor | I believe openafs pre-1.8.1 clients have a bug that prevents callbacks from being processed correctly | 03:42 |
| auristor | Let me double check when it was fixed | 03:42 |
| ianw | auristor: yeah, another step is to build our own latest openafs packages and try that out too | 03:43 |
| ianw | #status log mirror01.iad.rax.opendev.org in emergency file, and serving mirror files via kafs which is currently hand-configured; see https://etherpad.openstack.org/p/opendev-mirror-afs | 03:44 |
| openstackstatus | ianw: finished logging | 03:44 |
| auristor | yes, volume callbacks created when new RO or BK snapshots are created and volumes are released or moved were ignored by the client pre-1.8.1 | 03:45 |
| *** dave-mccowan has quit IRC | 03:46 | |
| auristor | There are also as yet unresolved issues which handling pending signals that cause EINTR and ERESTARTSYS errors to be converted to ENOENT which causes negative dentry objects to be cached by the vfs. | 03:47 |
| auristor | The 1.6.x openafs clients did not experience the volume callback processing failures | 03:47 |
| ianw | auristor: ok, so it sounds like not worth investigating unless we're seeing the same thing with >=1.8.1 packages | 03:48 |
| *** lsell has quit IRC | 03:49 | |
| *** evgenyl has quit IRC | 03:49 | |
| *** evgenyl has joined #openstack-infra | 03:49 | |
| auristor | There are security fixes in 1.8.2 and 1.8.3 is the most recent release. I wish you could use the auristorfs client and server but I understand why that isn't possible | 03:49 |
| auristor | the symptoms described in opendev-mirror-afs are consistent with callbacks being ignored | 03:51 |
| auristor | then the old version of the file will continued to be served from the cache | 03:51 |
| auristor | it can also result in corruption if some chunks of a file are ejected from the cache and then re-populated from a different version of the file | 03:52 |
| *** dayou has quit IRC | 03:52 | |
| ianw | all sounds exactly like what's going on :) | 03:52 |
| ianw | I think that bionic should really upgrade ... it does not sound like it has chosen a good package for it's LTS distro | 03:53 |
| *** signed8bit has quit IRC | 03:53 | |
| *** lsell has joined #openstack-infra | 03:53 | |
| *** dayou has joined #openstack-infra | 03:54 | |
| auristor | I would never deploy Ubuntu specifically because they do not ship release packages from upstream. Why are they distributing a release candidate as their production stable version? | 03:54 |
| auristor | I've had the same issues with them and the version of Heimdal they stip | 03:55 |
| auristor | ship | 03:55 |
| ianw | i think it's just what was there at release. i'll put it in my todo list to get a bug on it at least | 03:56 |
| auristor | exactly. for Heimdal they shipped whatever the tip of the master branch was on the day they required a new version. | 03:57 |
| auristor | If upstream hasn't declared a branch production ready and tagged a new release, the Linux distributions have no business shipping the code | 03:58 |
| *** markvoelker has joined #openstack-infra | 03:58 | |
| auristor | looks to me as if openafs-client 1.6.21 is still packaged for bionic https://launchpad.net/ubuntu/bionic/amd64/openafs-client/1.6.21-3 | 04:01 |
| *** signed8bit has joined #openstack-infra | 04:02 | |
| auristor | you might want to install that instead | 04:02 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:04 |
| *** markmcd has quit IRC | 04:05 | |
| ianw | ^ this has a very hard-coded .service file for bionic, but when it works should be able to test it with mirror hosts in CI | 04:06 |
| auristor | you might find https://gitlab.umich.edu/jsbillin/ansible-kafs-workstation useful | 04:08 |
| *** ykarel|away has quit IRC | 04:08 | |
| *** georgk has quit IRC | 04:08 | |
| *** georgk has joined #openstack-infra | 04:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:09 |
| *** ykarel|away has joined #openstack-infra | 04:11 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:14 |
| ianw | auristor: yep, top comment :) https://review.opendev.org/#/c/623974/13/roles/kafs-client/tasks/kafs-client/Fedora.yaml | 04:15 |
| *** udesale has joined #openstack-infra | 04:16 | |
| auristor | great | 04:16 |
| auristor | there was some comment earlier about initial reads from openstack.org being slow? | 04:18 |
| auristor | from a cold cache its taking my client 1.8s to "ls -l /afs/openstack.org/mirror" | 04:19 |
| ianw | yeah, i think that was when clarkb brought up a kafs test environment; i did not see anything like that. and the kafs CI job seemed fine too | 04:20 |
| auristor | ok. then if there is nothing else. I'm going to find a pillow | 04:21 |
| ianw | auristor: is cachefilesd likely to cause any issues? i can't find a lot of discussion about it -- not sure if that's because it's a bit inactive, or it "just works" | 04:21 |
| ianw | oh, find a pillow, don't stay up for me :) | 04:21 |
| auristor | cachefilesd is supposed to work with kafs. Its maintained by David Howells who also maintains kafs | 04:23 |
| auristor | Although it probably hasn't seen a lot of real world use | 04:23 |
| ianw | well, we're pretty close to being able to give it a pretty good stress test, at least in the R/O case (for now) ... we'll keep you updated :) | 04:25 |
| *** dayou has quit IRC | 04:28 | |
| ianw | as usual, thanks for all the advice, always helpful and saves us a lot of time going down wrong paths | 04:28 |
| *** bhavikdbavishi has quit IRC | 04:31 | |
| *** markvoelker has quit IRC | 04:32 | |
| *** ykarel|away has quit IRC | 04:32 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:33 | |
| *** kjackal has joined #openstack-infra | 04:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:36 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:38 |
| *** dayou has joined #openstack-infra | 04:41 | |
| *** ykarel|away has joined #openstack-infra | 04:50 | |
| *** ykarel|away is now known as ykarel | 04:50 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 04:51 |
| *** whoami-rajat has joined #openstack-infra | 04:51 | |
| *** pcaruana has joined #openstack-infra | 04:54 | |
| *** pcaruana has quit IRC | 04:59 | |
| *** hongbin has quit IRC | 05:01 | |
| *** raukadah is now known as chandankumar | 05:01 | |
| *** noama has joined #openstack-infra | 05:21 | |
| *** markvoelker has joined #openstack-infra | 05:29 | |
| *** kopecmartin|off is now known as kopecmartin | 05:32 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 05:37 |
| *** piotrowskim has joined #openstack-infra | 05:42 | |
| *** yboaron_ has joined #openstack-infra | 05:44 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [DNM] kafs testing https://review.opendev.org/623974 | 05:47 |
| *** e0ne has joined #openstack-infra | 05:51 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: opendev.org mirrors: switch out DFW, switch in IAD https://review.opendev.org/664784 | 05:54 |
| ianw | infra-root: ^ i'm not going to merge that and disappear, but that will get us testing the kafs install on IAD | 05:54 |
| *** igordc has joined #openstack-infra | 05:58 | |
| *** markvoelker has quit IRC | 06:02 | |
| *** zhangfei has joined #openstack-infra | 06:18 | |
| *** rpittau|afk is now known as rpittau | 06:20 | |
| *** e0ne has quit IRC | 06:22 | |
| *** pgaxatte has joined #openstack-infra | 06:26 | |
| *** ricolin has joined #openstack-infra | 06:27 | |
| *** zhangfei has quit IRC | 06:30 | |
| *** takamatsu has quit IRC | 06:46 | |
| *** zhurong has joined #openstack-infra | 06:47 | |
| *** dayou has quit IRC | 06:48 | |
| *** dayou has joined #openstack-infra | 06:51 | |
| *** xinranwang has quit IRC | 06:54 | |
| *** pcaruana has joined #openstack-infra | 06:56 | |
| *** markvoelker has joined #openstack-infra | 06:59 | |
| *** gfidente has joined #openstack-infra | 07:07 | |
| ianw | http://logs.openstack.org/74/623974/18/check/system-config-run-mirror/595517d/ <- kafs working on the mirror ci jobs too | 07:08 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:13 |
| *** ginopc has joined #openstack-infra | 07:14 | |
| *** tesseract has joined #openstack-infra | 07:14 | |
| *** dmellado has quit IRC | 07:17 | |
| *** psachin has quit IRC | 07:19 | |
| *** nicolasbock has joined #openstack-infra | 07:20 | |
| *** nicolasbock has quit IRC | 07:22 | |
| *** rcernin has quit IRC | 07:22 | |
| *** nicolasbock has joined #openstack-infra | 07:22 | |
| *** nicolasbock has quit IRC | 07:23 | |
| *** nicolasbock has joined #openstack-infra | 07:24 | |
| *** nicolasbock has quit IRC | 07:26 | |
| *** iurygregory has joined #openstack-infra | 07:26 | |
| *** bhavikdbavishi has quit IRC | 07:26 | |
| *** dmellado has joined #openstack-infra | 07:29 | |
| *** gregoryo has quit IRC | 07:31 | |
| *** dayou_ has joined #openstack-infra | 07:33 | |
| *** markvoelker has quit IRC | 07:33 | |
| *** dayou has quit IRC | 07:34 | |
| *** dtantsur|afk is now known as dtantsur | 07:34 | |
| *** nicolasbock has joined #openstack-infra | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:36 | |
| *** takamatsu has joined #openstack-infra | 07:37 | |
| *** nicolasbock has quit IRC | 07:37 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 07:37 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 07:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] Docker compose example: add keycloak authentication https://review.opendev.org/664813 | 07:40 |
| *** nicolasbock has joined #openstack-infra | 07:41 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:41 |
| *** ralonsoh has joined #openstack-infra | 07:42 | |
| *** dayou has joined #openstack-infra | 07:43 | |
| *** slaweq has quit IRC | 07:44 | |
| *** dayou_ has quit IRC | 07:45 | |
| *** slaweq has joined #openstack-infra | 07:48 | |
| *** ykarel is now known as ykarel|lunch | 07:51 | |
| *** dayou has quit IRC | 07:54 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** ccamacho has joined #openstack-infra | 07:58 | |
| *** jpich has joined #openstack-infra | 07:59 | |
| *** guilhermesp has quit IRC | 08:01 | |
| *** seyeongkim has quit IRC | 08:02 | |
| *** guilhermesp has joined #openstack-infra | 08:03 | |
| *** seyeongkim has joined #openstack-infra | 08:03 | |
| *** tdasilva_ has quit IRC | 08:04 | |
| *** dayou has joined #openstack-infra | 08:07 | |
| *** dayou_ has joined #openstack-infra | 08:10 | |
| *** dayou has quit IRC | 08:12 | |
| *** dayou_ has quit IRC | 08:16 | |
| *** igordc has quit IRC | 08:17 | |
| *** slaweq has quit IRC | 08:19 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** dayou_ has joined #openstack-infra | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:30 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 08:32 |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** imacdonn has quit IRC | 08:38 | |
| *** nicolasbock has quit IRC | 08:38 | |
| *** imacdonn has joined #openstack-infra | 08:39 | |
| *** takamatsu has quit IRC | 08:39 | |
| *** happyhemant has joined #openstack-infra | 08:39 | |
| *** takamatsu has joined #openstack-infra | 08:40 | |
| *** lucasagomes has joined #openstack-infra | 08:41 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:49 | |
| *** jaosorior has quit IRC | 08:51 | |
| *** cyberpear has quit IRC | 08:59 | |
| zbr|rover | ianw: can you help wf https://review.opendev.org/#/c/663599/ ? its sister already went in yesterday. | 08:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 08:59 |
| *** nicolasbock has joined #openstack-infra | 09:00 | |
| *** markvoelker has quit IRC | 09:03 | |
| *** slaweq has joined #openstack-infra | 09:03 | |
| *** slaweq has quit IRC | 09:09 | |
| *** takamatsu has quit IRC | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:17 | |
| *** ykarel|lunch is now known as ykarel | 09:37 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:44 |
| *** takamatsu has joined #openstack-infra | 09:51 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 09:55 |
| *** markvoelker has joined #openstack-infra | 10:00 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** takamatsu has quit IRC | 10:16 | |
| *** bhavikdbavishi has quit IRC | 10:18 | |
| *** dayou_ has quit IRC | 10:21 | |
| *** rpittau is now known as rpittau|afk | 10:22 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 10:27 |
| *** markvoelker has quit IRC | 10:34 | |
| *** dayou_ has joined #openstack-infra | 10:35 | |
| *** lucasagomes has quit IRC | 10:39 | |
| *** lucasagomes has joined #openstack-infra | 10:40 | |
| *** dayou_ has quit IRC | 10:43 | |
| *** takamatsu has joined #openstack-infra | 10:49 | |
| *** dayou_ has joined #openstack-infra | 10:55 | |
| *** adriancz has quit IRC | 10:58 | |
| *** ykarel is now known as ykarel|afk | 11:03 | |
| *** ykarel|afk has quit IRC | 11:08 | |
| *** yamamoto has joined #openstack-infra | 11:11 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:12 |
| *** yamamoto has quit IRC | 11:12 | |
| *** dayou_ has quit IRC | 11:13 | |
| *** tosky has joined #openstack-infra | 11:14 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:17 |
| *** adrianreza_ has joined #openstack-infra | 11:26 | |
| *** dayou_ has joined #openstack-infra | 11:26 | |
| *** electrofelix has joined #openstack-infra | 11:28 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** rh-jelabarre has joined #openstack-infra | 11:37 | |
| *** cyberpear has joined #openstack-infra | 11:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:44 | |
| *** ykarel|afk has joined #openstack-infra | 11:46 | |
| *** yamamoto has joined #openstack-infra | 11:49 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:50 |
| *** ykarel|afk has quit IRC | 11:50 | |
| *** ykarel has joined #openstack-infra | 11:51 | |
| *** tosky has quit IRC | 11:51 | |
| *** tosky has joined #openstack-infra | 11:51 | |
| *** rlandy has joined #openstack-infra | 11:55 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** dayou_ has quit IRC | 11:59 | |
| *** ekultails has joined #openstack-infra | 12:04 | |
| *** rlandy is now known as rlandy|ruck | 12:05 | |
| *** boden has joined #openstack-infra | 12:08 | |
| boden | howdy. looking for any pointer in resolving the stable/stein release notes job for neutron-lbaas... its consistently failing with http://logs.openstack.org/30/660930/5/check/build-openstack-releasenotes/32e27d6/job-output.txt.gz#_2019-06-10_04_25_01_002244 | 12:08 |
| boden | I'm not seeing any obvious reason why... any help is appreciated | 12:09 |
| *** udesale has quit IRC | 12:10 | |
| boden | actually maybe I see the problem | 12:10 |
| *** udesale has joined #openstack-infra | 12:11 | |
| *** dayou_ has joined #openstack-infra | 12:11 | |
| *** auristor has quit IRC | 12:11 | |
| *** dave-mccowan has joined #openstack-infra | 12:17 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** goldyfruit has quit IRC | 12:19 | |
| *** yboaron_ has quit IRC | 12:24 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** iurygregory has quit IRC | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Adds openstack-tox-molecule template https://review.opendev.org/663599 | 12:33 |
| *** rpittau|afk is now known as rpittau | 12:36 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 12:39 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 12:41 |
| *** jamesmcarthur has joined #openstack-infra | 12:41 | |
| *** michael-beaver has joined #openstack-infra | 12:42 | |
| *** kaiokmo has joined #openstack-infra | 12:42 | |
| *** bobh has quit IRC | 12:48 | |
| *** _erlon_ has joined #openstack-infra | 12:50 | |
| *** bhavikdbavishi has quit IRC | 12:50 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:51 | |
| *** tdasilva has joined #openstack-infra | 12:52 | |
| *** bhavikdbavishi has quit IRC | 12:52 | |
| *** ykarel is now known as ykarel|afk | 12:52 | |
| *** zbr|rover is now known as zbr|flow | 12:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:54 | |
| *** gfidente has quit IRC | 12:55 | |
| *** sthussey has joined #openstack-infra | 12:56 | |
| *** mattmceuen has joined #openstack-infra | 12:56 | |
| *** auristor has joined #openstack-infra | 12:57 | |
| *** gfidente has joined #openstack-infra | 12:58 | |
| *** rfarr_ has joined #openstack-infra | 12:58 | |
| *** rfarr__ has joined #openstack-infra | 13:01 | |
| *** markvoelker has joined #openstack-infra | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:02 |
| *** rfarr_ has quit IRC | 13:04 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:04 | |
| mattmceuen | hi all, GM/GE! qq -- do project additions that pull from an upstream git seed repo happen pretty quick after the `project-config` change merges, or do they get batched up for later? | 13:05 |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** smarcet has joined #openstack-infra | 13:06 | |
| mattmceuen | asking as my change is to add an election repo for airship which has schedule dates baked into it - I just want to tweak those dates to be realistic | 13:06 |
| *** yamamoto has quit IRC | 13:06 | |
| *** jtomasek_ has joined #openstack-infra | 13:07 | |
| *** lseki has joined #openstack-infra | 13:08 | |
| *** jtomasek has quit IRC | 13:08 | |
| arxcruz | hello guys, I have a yaml file with some variables there, is it possible to read set this file as variables in zuul job definition? | 13:12 |
| *** aaronsheffield has joined #openstack-infra | 13:12 | |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| *** eharney has joined #openstack-infra | 13:14 | |
| *** florianf has joined #openstack-infra | 13:15 | |
| *** smarcet has quit IRC | 13:17 | |
| *** smarcet has joined #openstack-infra | 13:19 | |
| *** pcaruana has quit IRC | 13:30 | |
| *** pcaruana|afk| has joined #openstack-infra | 13:30 | |
| *** iurygregory has joined #openstack-infra | 13:32 | |
| *** bhavikdbavishi has quit IRC | 13:32 | |
| *** dklyle has quit IRC | 13:34 | |
| *** goldyfruit has joined #openstack-infra | 13:34 | |
| *** florianf has quit IRC | 13:34 | |
| *** markvoelker has quit IRC | 13:34 | |
| *** electrofelix has quit IRC | 13:37 | |
| openstackgerrit | Joshua Hesketh proposed zuul/zuul master: WIP Expose date time as facts https://review.opendev.org/664674 | 13:38 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 13:39 | |
| *** smarcet has quit IRC | 13:39 | |
| *** Goneri has joined #openstack-infra | 13:45 | |
| *** yamamoto has quit IRC | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| *** liuyulong has joined #openstack-infra | 13:49 | |
| *** mriedem has joined #openstack-infra | 13:49 | |
| *** ricolin_ has joined #openstack-infra | 13:52 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/project-config master: Retire independent ansible role https://review.opendev.org/664932 | 13:54 |
| *** whoami-rajat has quit IRC | 13:55 | |
| *** ricolin has quit IRC | 13:55 | |
| cloudnull | ^ hey all, we're retiring a bunch of things, is the use of the noop-jobs template the right thing to do ? | 13:55 |
| *** whoami-rajat has joined #openstack-infra | 14:01 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Remove the rhel 8 check for xfs https://review.opendev.org/663998 | 14:01 |
| *** goldyfruit has quit IRC | 14:04 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: debootstrap: make default network interface names configurable https://review.opendev.org/664936 | 14:09 |
| *** smarcet has joined #openstack-infra | 14:09 | |
| *** factor has quit IRC | 14:12 | |
| *** factor has joined #openstack-infra | 14:15 | |
| clarkb | mattmceuen: they should happen within about 45 minutes (the time it takes to run the next puppet pulse) | 14:16 |
| *** Lucas_Gray has joined #openstack-infra | 14:17 | |
| *** kjackal has quit IRC | 14:17 | |
| clarkb | cloudnull: ya add noop jobs, push updates to retire in repo, then clean up zuul config. Should be documented in the infra manual | 14:17 |
| clarkb | arxcruz: not via the job definition in the zuul config, but on the ansible config side your playbook can load the vars | 14:18 |
| *** takamatsu has quit IRC | 14:18 | |
| clarkb | boden if you scroll up a bit yhe build failed so thr artifacts dont exist for copying | 14:19 |
| *** jtomasek_ has quit IRC | 14:20 | |
| *** ianychoi_ is now known as ianychoi | 14:22 | |
| *** jtomasek_ has joined #openstack-infra | 14:22 | |
| arxcruz | clarkb thanks :) | 14:22 |
| fungi | boden: on the neutron-lbaas release notes builds, the problem is that release notes jobs always run from the master branch no matter what branch is involved in triggering them. since the repo has its master branch retired that job is no longer going to be able to work. this is probably worth discussing with the release team, but i've been of the opinion for a while that we simply shouldn't retire | 14:24 |
| fungi | projects until we're ready to retire *all* their branches | 14:24 |
| *** igordc has joined #openstack-infra | 14:26 | |
| *** iurygregory has quit IRC | 14:27 | |
| *** takamatsu has joined #openstack-infra | 14:31 | |
| *** markvoelker has joined #openstack-infra | 14:32 | |
| *** takamatsu has quit IRC | 14:38 | |
| *** pgaxatte has quit IRC | 14:39 | |
| *** tdasilva has quit IRC | 14:39 | |
| *** tdasilva has joined #openstack-infra | 14:40 | |
| *** ykarel|afk is now known as ykarel | 14:40 | |
| *** dklyle has joined #openstack-infra | 14:45 | |
| *** iurygregory has joined #openstack-infra | 14:45 | |
| *** smarcet has quit IRC | 14:46 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 14:49 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Fix typo in artifact promote job https://review.opendev.org/664944 | 14:50 |
| corvus | clarkb, fungi: ^ progress since yesterday! on to the next oops | 14:50 |
| fungi | heh, getting closer! | 14:52 |
| *** rfolco has quit IRC | 14:52 | |
| fungi | here's hoping it's not an asymptotic approach ;) | 14:52 |
| *** smarcet has joined #openstack-infra | 14:56 | |
| openstackgerrit | Merged zuul/zuul master: Fix minor typos in Nodepool OpenStack instructions https://review.opendev.org/664665 | 14:57 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Unify gearman worker handling https://review.opendev.org/664948 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Move fingergw config to fingergw https://review.opendev.org/664949 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Support ssl encrypted fingergw clients https://review.opendev.org/664950 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Add ssl support to fingergw client https://review.opendev.org/664951 | 15:00 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Register finger gateway if it's zoned https://review.opendev.org/664952 | 15:00 |
| openstackgerrit | Merged opendev/base-jobs master: Fix typo in artifact promote job https://review.opendev.org/664944 | 15:04 |
| *** markvoelker has quit IRC | 15:05 | |
| *** kjackal has joined #openstack-infra | 15:06 | |
| zbr|flow | was zuul restarted in the last ~4h? i had a change that received the workflow about that time and it was not in the queue. I had add a | 15:08 |
| zbr|flow | a recheck and now is on check again. | 15:08 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/project-config master: Retire independent ansible role https://review.opendev.org/664932 | 15:11 |
| clarkb | zbr|flow: I dont see any mention of a restart in scrollback so probably not | 15:17 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 15:17 | |
| zbr|flow | clarkb: that is why I asked because I did not see any msg/email. not sure what could have caused it to be missed. See https://review.opendev.org/#/c/652971/ -- it did had one depends on on a change that merged 2-3h ago, so it did make sense to wait. but apparently it did not reach the queue. | 15:19 |
| corvus | zbr|flow: i think you answered your own question; it wasn't enqueued because the dependency had not merged | 15:21 |
| zbr|flow | corvus: but once the dependency is merge, is not added automatically to the queue? | 15:22 |
| zbr|flow | if not, how can I trigger that? clearly "recheck" does sent it back to the check, instead of gate, which means many hours delay. | 15:22 |
| corvus | zbr|flow: no, we don't have our pipelines configured to do that (that's the "zuul" trigger) | 15:23 |
| corvus | zbr|flow: another workflow+1 would do it (or clearing it and re-adding it) | 15:24 |
| clarkb | I dont think you have ti clear and readd it anymore | 15:24 |
| zbr|flow | corvus: ahh, thanks. i will remember that trick. I guess that even a rebase does that, where is possible. | 15:24 |
| clarkb | just recomment +A | 15:24 |
| *** nicolasbock has quit IRC | 15:24 | |
| *** smarcet has quit IRC | 15:26 | |
| *** smarcet has joined #openstack-infra | 15:29 | |
| *** pcaruana|afk| has quit IRC | 15:30 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 15:33 |
| corvus | fungi: https://review.opendev.org/664387 could use a nudge if you have a minute; i don't expect it to interfere with your other base-test work (if it does break something, i promise to fix it fast) | 15:42 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Safely add Ansible password lookup plugin https://review.opendev.org/662870 | 15:42 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Increase test timeout to 6 minutes https://review.opendev.org/664961 | 15:42 |
| mordred | we officially no longer own real estate \o/ | 15:52 |
| corvus | mordred: congratulations on finishing the american dream! | 15:53 |
| mordred | corvus: it was a good dream while it lasted :) | 15:53 |
| mordred | now we're on to the modern american dream - perpetual homelessness! | 15:53 |
| clarkb | mordred: nice | 15:53 |
| *** iurygregory has quit IRC | 15:54 | |
| clarkb | mordred: so yesterday I discovered a flaw with our control plane images plan | 15:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Register finger gateway if it's zoned https://review.opendev.org/664952 | 15:54 |
| mordred | yes - I see this from scrollback | 15:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway https://review.opendev.org/664965 | 15:54 |
| clarkb | mordred: vexxhost is boot from volume and youcannot delete images from glance if that are being used in a boot from volume setup | 15:54 |
| mordred | clarkb: yeah. I like the thoughts about letting it build/upload once and copying/freezing or similar - we don't actually need daily up to date images | 15:55 |
| clarkb | beginning to think we probably want to revert the control plane images changes in nodepool. Upload the latest build manually for now, then reapproach this maybe with a dedicated builder and some less strict must delete older images behavior from nodepool | 15:55 |
| mordred | ++ | 15:55 |
| mordred | maybe add a feature to nodepool to allow how frequently someone wants new images made / old ones deleted ... something something handwave design something | 15:56 |
| mattmceuen | clarkb: (belatedly) awesome, thanks! | 15:58 |
| *** tdasilva has quit IRC | 15:59 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 15:59 |
| *** tdasilva has joined #openstack-infra | 16:00 | |
| *** markvoelker has joined #openstack-infra | 16:02 | |
| *** yamamoto has quit IRC | 16:02 | |
| *** ykarel is now known as ykarel|away | 16:03 | |
| *** gyee has joined #openstack-infra | 16:03 | |
| boden | fungi thanks.. I didn't realize that was the case... I guess for now we'll have to remove the release notes from patches or disable that job | 16:04 |
| *** Goneri has quit IRC | 16:05 | |
| fungi | boden: i also asked in #openstack-release for ideas since they're most familiar with the release notes builds | 16:07 |
| boden | fungi thank you | 16:07 |
| clarkb | fwiw I am able to navigate http://mirror.iad.rax.opendev.org so I think the kafs stuff is working there and I've +2'd https://review.opendev.org/#/c/664784/1 | 16:08 |
| mordred | infra-root: I'm going to be afk for most of today - we're behind in packing and I need to crank on that for a while - will check in from time to time. sorry for late notice on that | 16:09 |
| corvus | mordred: should we all come over and help? | 16:09 |
| *** pcaruana|afk| has joined #openstack-infra | 16:09 | |
| clarkb | its actually cooler in dallas today than here | 16:10 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 16:10 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 16:10 |
| *** tosky has quit IRC | 16:10 | |
| boden | fungi FYI for now I have proposed https://review.opendev.org/#/c/664874/ | 16:10 |
| mordred | corvus: YES PLEASE | 16:11 |
| mordred | corvus: something about moving from a 5-bedroom suburban texas house to some suitcases and a storage unit is more complex than a normal move | 16:11 |
| mordred | who would have guessed? | 16:11 |
| openstackgerrit | Merged opendev/base-jobs master: base-test: Move log-inventory to localhost https://review.opendev.org/664387 | 16:12 |
| Shrews | mordred: feel free to park your tesla at my place | 16:12 |
| Shrews | i will watch it | 16:12 |
| *** bhavikdbavishi has joined #openstack-infra | 16:12 | |
| fungi | feel free to park your boat at my place | 16:12 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:12 | |
| corvus | Shrews: he can just tell it to drive itself over there! | 16:12 |
| fungi | once the dock is back in working order | 16:12 |
| *** mattw4 has joined #openstack-infra | 16:13 | |
| corvus | in 50 years, mordred can park his boat at my place too! | 16:13 |
| clarkb | corvus: ooh you did the long term beach front play | 16:13 |
| fungi | in 50 years, mordred can go scuba divnig in the remains of my house | 16:13 |
| mordred | fungi: I will enjoy swimming through your livingroom | 16:14 |
| fungi | shipwreck diving here will be replaced by townwreck diving | 16:14 |
| *** kopecmartin is now known as kopecmartin|off | 16:16 | |
| *** Goneri has joined #openstack-infra | 16:18 | |
| clarkb | and maybe eventually it will be townreef diving | 16:18 |
| mordred | mmm. townreef | 16:18 |
| *** rkukura has joined #openstack-infra | 16:18 | |
| clarkb | https://en.wikipedia.org/wiki/Nan_Madol is the island ruin reef diving spot :) | 16:19 |
| mordred | https://www.sportdiver.com/what-its-like-to-dive-norways-hauntingly-preserved-underwater-village | 16:19 |
| mordred | of course, diving fungi's house in NC will have more pleasant water temperatures | 16:19 |
| corvus | "what it's like..." it's cold that's what it's like | 16:22 |
| *** lucasagomes has quit IRC | 16:23 | |
| clarkb | mordred: ok so no concerns from you if we revert/clean up the control plane image building and upload the latest image as a more set in stone image? | 16:24 |
| *** Goneri has quit IRC | 16:24 | |
| clarkb | re kafs navigating the dir paths seems slower, but files seem to load quick | 16:25 |
| clarkb | I expect that for our use as a mirror thats fine as indexes are at known locations | 16:25 |
| clarkb | (and we don't rely on mod autoindex on top of afs) | 16:25 |
| mordred | clarkb: nope - I think that's best for now | 16:25 |
| clarkb | ok I'll work on that in a bit. I promised kids some time outside this morning before it gets super hot so I should do that soon | 16:26 |
| mordred | clarkb: there's a part of me that thinks we should have a zuul job that builds control plane images and copies them somewhere - then a playbook kind of like cloud-launcher that just makes sure that image is in the given cloud - since we don't really need most of the nodepool-builder complexity | 16:29 |
| mordred | but that would need more thought than I have space for atm | 16:29 |
| *** jpich has quit IRC | 16:29 | |
| *** ginopc has quit IRC | 16:30 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 16:31 | |
| *** bhavikdbavishi has quit IRC | 16:32 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:32 | |
| corvus | clarkb: are you trying kafs out on a mirror node? | 16:33 |
| corvus | clarkb: ah yes, i see in scrollback now, sorry | 16:33 |
| corvus | clarkb: did that get the cache daemon added? | 16:33 |
| corvus | root 977 0.0 0.0 4876 1568 ? Ss 05:44 0:00 /sbin/cachefilesd | 16:34 |
| corvus | looks like it | 16:34 |
| clarkb | corvus: yup and there is data in the cache | 16:35 |
| clarkb | loading files is actually quite quick. I expect its mod autoindex and dir listing that is slow and not the actual data reads | 16:35 |
| *** markvoelker has quit IRC | 16:36 | |
| *** Goneri has joined #openstack-infra | 16:36 | |
| *** Lucas_Gray has quit IRC | 16:37 | |
| corvus | clarkb: it's possible that cachefilesd doesn't work the same way as the openafs client cache manager, and maybe has to do more round-trips to verify indexes? | 16:39 |
| corvus | 664784+3 | 16:39 |
| *** e0ne has quit IRC | 16:44 | |
| johnsom | Hi infra folks, I have a zuul puzzle that codesearch is not revealing the answer to. We have a "devstack_plugins" setting that points to https://opendev.org/openstack/octavia.git, this works fine, but if we do a depends-on to an octavia repo patch, the devstack plugin.sh code that is run is from master, not the depends-on patch. | 16:44 |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| johnsom | It seems like I need some magic ref variable here: https://opendev.org/openstack/octavia-tempest-plugin/src/branch/master/zuul.d/jobs.yaml#L84 | 16:45 |
| johnsom | Any advice if that is the right approach or if I'm doing something else wrong that it's not picking up the depends-on version? Should we set that to a local path instead of git? I couldn't find any project doing that in codesearch. | 16:47 |
| *** derekh has quit IRC | 16:50 | |
| clarkb | johnsom: https://opendev.org/openstack/octavia-tempest-plugin/src/branch/master/zuul.d/jobs.yaml#L38-L39 is the first thing I would check but octavia is already a required project | 16:50 |
| johnsom | Yep | 16:50 |
| clarkb | johnsom: can you link to the job logs that doesn't do the right thing? | 16:50 |
| johnsom | It is this patch that didn't honor the depends-on: https://review.opendev.org/626406 | 16:50 |
| openstackgerrit | Merged openstack/project-config master: opendev.org mirrors: switch out DFW, switch in IAD https://review.opendev.org/664784 | 16:51 |
| johnsom | Specifically this one: http://logs.openstack.org/06/626406/8/check/octavia-v2-dsvm-py2-scenario/30e303f/ | 16:51 |
| clarkb | johnsom: http://logs.openstack.org/06/626406/8/check/octavia-v2-dsvm-py2-scenario/30e303f/zuul-info/inventory.yaml shows that octavia is checked out to master | 16:53 |
| clarkb | but I think that is supposed to be the future master | 16:54 |
| clarkb | however as you point out http://logs.openstack.org/06/626406/8/check/octavia-v2-dsvm-py2-scenario/30e303f/controller/logs/devstacklog.txt.gz#_2019-06-12_03_58_34_853 seems to be the commit chosen, not the depeds on commit | 16:56 |
| johnsom | I didn't see the commit ID from the depends-on in the job-output log either, which I thought was odd | 16:56 |
| *** rpittau is now known as rpittau|afk | 16:56 | |
| clarkb | oh I see it :) | 16:58 |
| clarkb | your depends on uses openstack.org and not opendev.org | 16:58 |
| clarkb | since the zuul gerrit connection is for opendev.org it doesn't know ow to handle the openstack.org depends on | 16:58 |
| johnsom | I wondered about that, but the link works when you click on it. | 16:58 |
| clarkb | this was something we sent email about after the switch | 16:59 |
| fungi | that's because it's treated as a url when you click on it | 16:59 |
| clarkb | ya the link is a valid link | 16:59 |
| johnsom | Ah, ok, sorry I missed that one. | 16:59 |
| clarkb | but it isn't someting that zuul understands in the context of dependency chains | 16:59 |
| fungi | but zuul doesn't resolve the url, it merely sees it as a set of fields | 16:59 |
| fungi | it parses the url to get the connection hostname and the change number | 16:59 |
| johnsom | It is an old patch I am updating. Thanks clarkb! I will give that a go and maybe go through our patches and update as necessary. | 17:00 |
| fungi | it doesn't actually follow the url though to see any redirects which might be there | 17:00 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 17:01 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 17:01 |
| fungi | johnsom: you can search gerrit for them... https://review.opendev.org/#/q/is:open+message:%22depends-on:+https://review.openstack%22 | 17:01 |
| johnsom | fungi Thanks! Just a few.... lol | 17:02 |
| *** rfolco has joined #openstack-infra | 17:02 | |
| fungi | looks like ~275 open changes with pre-migration depends-on urls | 17:02 |
| fungi | but editing their commit messages means creating new patchsets of course | 17:03 |
| johnsom | Yeah, I think I will handle the octavia repos and not impose on other projects. | 17:03 |
| fungi | makes sense | 17:04 |
| fungi | we expected folks would probably fix them over time | 17:04 |
| *** goldyfruit has joined #openstack-infra | 17:04 | |
| fungi | just explaining why we didn't fix them at the time of the migration | 17:04 |
| *** ramishra has quit IRC | 17:04 | |
| *** tosky has joined #openstack-infra | 17:05 | |
| fungi | many of those also may simply have a depends-on to merged changes, and so wouldn't need fixing | 17:05 |
| *** mattw4 has quit IRC | 17:05 | |
| johnsom | Yep, makes sense | 17:05 |
| *** mattw4 has joined #openstack-infra | 17:07 | |
| *** rh-jelabarre has quit IRC | 17:10 | |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| *** yamamoto has quit IRC | 17:11 | |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| *** jamesmcarthur has quit IRC | 17:12 | |
| *** rh-jelabarre has joined #openstack-infra | 17:13 | |
| *** jpena is now known as jpena|off | 17:13 | |
| *** yamamoto has quit IRC | 17:13 | |
| *** yamamoto has joined #openstack-infra | 17:14 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 17:14 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test base-test https://review.opendev.org/664981 | 17:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: test base-test https://review.opendev.org/664981 | 17:16 |
| *** spsurya has quit IRC | 17:19 | |
| *** weifan has joined #openstack-infra | 17:23 | |
| *** ykarel|away has quit IRC | 17:24 | |
| *** ianychoi has quit IRC | 17:25 | |
| *** diablo_rojo has joined #openstack-infra | 17:26 | |
| *** eharney has quit IRC | 17:27 | |
| *** eharney has joined #openstack-infra | 17:29 | |
| *** markvoelker has joined #openstack-infra | 17:33 | |
| *** adriant has quit IRC | 17:35 | |
| *** iokiwi has quit IRC | 17:35 | |
| *** adriant has joined #openstack-infra | 17:36 | |
| *** iokiwi has joined #openstack-infra | 17:36 | |
| *** tdasilva has quit IRC | 17:36 | |
| *** tdasilva has joined #openstack-infra | 17:37 | |
| corvus | fungi, clarkb: the log-inventory test looks good so https://review.opendev.org/664388 is ready for realz | 17:37 |
| *** ricolin_ has quit IRC | 17:44 | |
| mgagne | We are performing major changes and maintenances on inap-mtl01 (hence why it has been disabled for weeks). We would like to be able to delete all servers in that region. the mirror is still up and running. We need to delete that server too. We can however preserve the data found on the volumes attached to the instance. If you want to preserve them, can you please detach them and backup any relevant LVM configs? | 17:45 |
| *** jamesmcarthur has joined #openstack-infra | 17:45 | |
| *** jamesmcarthur has quit IRC | 17:45 | |
| *** ociuhandu has quit IRC | 17:45 | |
| *** jamesmcarthur has joined #openstack-infra | 17:45 | |
| corvus | i think we can just recreate it from scratch... clarkb, fungi: ^? | 17:48 |
| corvus | (we're starting to cycle them out for a new build anyway) | 17:48 |
| clarkb | mgagne: corvus ya we don't need the data there | 17:51 |
| clarkb | its all cache of http proxies and afs | 17:51 |
| *** noama has quit IRC | 17:51 | |
| clarkb | so will just be repopulated when used after being rebuilt | 17:51 |
| *** gfidente has quit IRC | 17:52 | |
| mgagne | clarkb: ok, we will be deleting all servers in inap-mtl01. I think only mirror01 is left. | 17:52 |
| *** ekultails has quit IRC | 17:54 | |
| *** owalsh has quit IRC | 17:54 | |
| *** owalsh has joined #openstack-infra | 17:55 | |
| clarkb | I think we caused mirror.iad.rax.opendev.org to fall over by using it | 17:55 |
| *** ekultails has joined #openstack-infra | 17:57 | |
| clarkb | I'll get a revert up in a sec | 17:58 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "opendev.org mirrors: switch out DFW, switch in IAD" https://review.opendev.org/664990 | 18:00 |
| corvus | clarkb: ack; i want to poke at it under load for just a sec | 18:00 |
| clarkb | corvus: sure | 18:00 |
| corvus | it has a LA of 140 | 18:01 |
| corvus | and is 100% idle | 18:01 |
| clarkb | apache2 seems to peg a cpu off and on | 18:01 |
| corvus | i'm guessing something is deadlocked, but who knows what | 18:02 |
| corvus | +3 | 18:02 |
| *** tdasilva has quit IRC | 18:03 | |
| corvus | strace says cachefilesd isn't doing anything. i wonder if that's normal. | 18:05 |
| *** markvoelker has quit IRC | 18:06 | |
| *** yamamoto has quit IRC | 18:06 | |
| corvus | clarkb: should we force-merge 990? | 18:06 |
| clarkb | the most recent data in /var/cache/openafs seems to be from today at ~17:22UTC | 18:07 |
| clarkb | corvus: probably a good idea | 18:07 |
| clarkb | corvus: are you doing that or should I? | 18:07 |
| corvus | i will | 18:07 |
| corvus | cd/ls in /afs/openstack.org is working but is very slow | 18:07 |
| openstackgerrit | Merged openstack/project-config master: Revert "opendev.org mirrors: switch out DFW, switch in IAD" https://review.opendev.org/664990 | 18:08 |
| corvus | we still need ansible to plop that onto the executors, but at least that's the only thing we're waiting for | 18:08 |
| clarkb | ++ | 18:08 |
| clarkb | it was worth an experiment | 18:09 |
| clarkb | and maybe even further debugging :) | 18:09 |
| corvus | it did not crash, which i find promising. | 18:09 |
| *** e0ne has joined #openstack-infra | 18:10 | |
| clarkb | we might try the bionic hwe kenrel next | 18:10 |
| clarkb | oh we might be on the hwe kernel already? | 18:11 |
| *** goldyfruit has quit IRC | 18:11 | |
| *** e0ne has quit IRC | 18:12 | |
| corvus | i'm a little confused as to why we have 2 mount entries | 18:12 |
| *** e0ne has joined #openstack-infra | 18:12 | |
| corvus | #openstack.org:root.afs 5000 214 4786 5% /afs | 18:12 |
| corvus | #openstack.org:root.cell 5000 9 4991 1% /afs/openstack.org | 18:12 |
| *** eernst has joined #openstack-infra | 18:14 | |
| corvus | [ 23.357060] FS-Cache: Netfs 'afs' registered for caching | 18:14 |
| corvus | that seems to suggest they should be talking to each other | 18:15 |
| corvus | and the cache usage is only at 1% | 18:15 |
| clarkb | corvus: on my 5.1 kernel I get a mount per afs volume as well as the cell | 18:16 |
| clarkb | the root.afs mount goes away seems like | 18:16 |
| fungi | what was the indication that it has fallen over? | 18:16 |
| clarkb | fungi: logstash showing jobs timing out talking to that mirror | 18:16 |
| corvus | clarkb: if it does mount-per-volume, then i wonder why we *only* have 2... | 18:16 |
| clarkb | and when I tried to browse it via web browser it wasn't loading pages | 18:16 |
| corvus | i would expect 1 mount or... like 10. | 18:16 |
| clarkb | corvus: ya | 18:17 |
| *** e0ne has quit IRC | 18:17 | |
| *** goldyfruit has joined #openstack-infra | 18:17 | |
| *** e0ne has joined #openstack-infra | 18:19 | |
| openstackgerrit | Merged zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 18:19 |
| clarkb | fungi: I was just checking if it was working fwiw. I didn't notice anything until I checked directly | 18:20 |
| *** udesale has quit IRC | 18:20 | |
| fungi | yeah, i seem to be unable to fetch things through its apache which are served from afs | 18:20 |
| fungi | though also things which should be proxied are equally unresponsive for me | 18:21 |
| fungi | for example i can fetch http://mirror.iad.rax.openSTACK.org/pypi/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl but not http://mirror.iad.rax.openDEV.org/pypi/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl | 18:23 |
| fungi | so i suspect whatever's going on has locked up apache | 18:23 |
| fungi | browser eventually reports a connection timeout | 18:24 |
| fungi | 2019-06-12 17:15:13.094656] [mpm_event:error] [pid 979:tid 140134352444352] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting | 18:26 |
| *** Goneri has quit IRC | 18:26 | |
| *** dims has quit IRC | 18:29 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 18:30 |
| *** e0ne has quit IRC | 18:35 | |
| *** igordc has quit IRC | 18:43 | |
| *** diablo_rojo has quit IRC | 18:49 | |
| *** bhavikdbavishi has quit IRC | 18:54 | |
| clarkb | fungi: all the slots filled up with afs requestes probably | 18:54 |
| *** tesseract has quit IRC | 18:54 | |
| *** weifan has quit IRC | 18:54 | |
| *** smarcet has quit IRC | 18:55 | |
| *** eharney has quit IRC | 18:57 | |
| clarkb | I'm copying the latest control plane image (which is on nb01) into my home dir there. I'll then upload it to vexxhost (and attempt to rax but that will require figuring out using sdk as a client) | 18:57 |
| clarkb | Then I can sort out what changes need to happen to stop having nodepool build those images | 18:58 |
| *** markvoelker has joined #openstack-infra | 19:03 | |
| *** smarcet has joined #openstack-infra | 19:06 | |
| fungi | yeah, i'm mostly wondering if sluggish response times coupled with high request volume left the server unable to allocate workers for further requests leading to timeouts | 19:07 |
| fungi | speculation | 19:07 |
| *** dims has joined #openstack-infra | 19:09 | |
| *** yamamoto has joined #openstack-infra | 19:13 | |
| *** Goneri has joined #openstack-infra | 19:14 | |
| *** diablo_rojo has joined #openstack-infra | 19:14 | |
| *** factor has quit IRC | 19:16 | |
| *** weifan has joined #openstack-infra | 19:17 | |
| *** yamamoto has quit IRC | 19:17 | |
| *** smarcet has quit IRC | 19:21 | |
| *** weifan has quit IRC | 19:22 | |
| *** e0ne has joined #openstack-infra | 19:23 | |
| *** xek_ has joined #openstack-infra | 19:24 | |
| openstackgerrit | Merged zuul/zuul master: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 19:25 |
| *** weifan has joined #openstack-infra | 19:25 | |
| *** e0ne has quit IRC | 19:26 | |
| *** tosky has quit IRC | 19:26 | |
| clarkb | infra-root http://paste.openstack.org/show/752834/ is what I think I want to run on nb01 to upload the image to rax. Anyone want ot double check the script before I run it? | 19:28 |
| fungi | i find it interesting you prefer sdk in an interactive session over cli | 19:33 |
| clarkb | fungi: sdk cannot upload to rax | 19:33 |
| clarkb | er other way around | 19:33 |
| clarkb | the openstack cli tool cannot upload images to rax | 19:34 |
| clarkb | I uploaded to vexxhost using openstack client | 19:34 |
| fungi | huh | 19:34 |
| fungi | that's... fun | 19:34 |
| clarkb | fungi: rax requires you upload to swift first then submit a glance task request to import the image from swift into glance. openstack client can do the first bit but I don't think the task api is exposed anywhere sane because its a schemaless api | 19:35 |
| clarkb | sdk (shade really) knows how to handle the rax specific case | 19:35 |
| fungi | what clouds.yaml are you using on nb01? | 19:36 |
| *** jamesmcarthur has quit IRC | 19:36 | |
| *** markvoelker has quit IRC | 19:37 | |
| clarkb | the one for the nodepool user | 19:37 |
| fungi | i guess this https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/nodepool_builder_clouds.yaml.j2#L31 | 19:39 |
| fungi | i guess we don't put that in /etc/openstack on the builders | 19:39 |
| clarkb | it is in the users homedir under .config | 19:39 |
| fungi | aha | 19:41 |
| fungi | i never think to look there because it's weird to hide server configuration in dotfiles | 19:41 |
| *** eharney has joined #openstack-infra | 19:41 | |
| fungi | or dotdirs in this case | 19:42 |
| *** jamesmcarthur has joined #openstack-infra | 19:43 | |
| fungi | yeah, that looks right to me, but i don't use the sdk heavily | 19:44 |
| corvus | clarkb: i'm not qualified to review all of the technical details of that paste, but i support it in spirit and the things i do understand look right. | 19:44 |
| *** eernst has quit IRC | 19:44 | |
| corvus | clarkb: with 'wait', make sure you run it in screen :) | 19:44 |
| *** Goneri has quit IRC | 19:45 | |
| fungi | oh, yeah, you could be 'wait'ing a while | 19:45 |
| corvus | you know, just in case your power company decided to pre-emptively cut power for safety reasons. or something. | 19:46 |
| clarkb | ++ | 19:47 |
| fungi | like that would ever happen | 19:47 |
| *** slaweq has joined #openstack-infra | 19:48 | |
| *** ralonsoh has quit IRC | 19:50 | |
| openstackgerrit | Dirk Mueller proposed openstack/project-config master: Add openSUSE 15.1 to the nodepool building as opensuse-15 https://review.opendev.org/660131 | 19:50 |
| *** weifan has quit IRC | 19:51 | |
| *** weifan has joined #openstack-infra | 19:51 | |
| *** weifan has quit IRC | 19:52 | |
| *** weifan has joined #openstack-infra | 19:52 | |
| *** weifan has quit IRC | 19:53 | |
| *** weifan has joined #openstack-infra | 19:53 | |
| *** ChosSimbaOne has joined #openstack-infra | 19:53 | |
| *** weifan has quit IRC | 19:54 | |
| *** weifan has joined #openstack-infra | 19:54 | |
| *** raissa has joined #openstack-infra | 19:54 | |
| *** weifan has quit IRC | 19:54 | |
| *** weifan has joined #openstack-infra | 19:55 | |
| *** whoami-rajat has quit IRC | 19:55 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 19:57 |
| *** mattw4 has quit IRC | 19:58 | |
| *** weifan has quit IRC | 20:00 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Allow artifact promote jobs a little more control over renaming https://review.opendev.org/665009 | 20:01 |
| *** factor has joined #openstack-infra | 20:08 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 20:10 |
| *** aedc has joined #openstack-infra | 20:10 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Allow artifact promote jobs a little more control over renaming https://review.opendev.org/665009 | 20:12 |
| clarkb | ok I made a couple edits to the script. I added debug logging and will upload to 'rax' first as a test case | 20:19 |
| *** factor has quit IRC | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:23 | |
| clarkb | the logging change is nice as now I get verbose output for what it is doing | 20:24 |
| *** kjackal has quit IRC | 20:27 | |
| clarkb | woot ok test upload to the test node tenant succeeded and the metadata was set properly (baseed on defaults in the cloud config) | 20:28 |
| clarkb | I'm updating the cloud name now and rerunning | 20:28 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:31 |
| *** rf0lc0 has joined #openstack-infra | 20:33 | |
| *** slaweq has quit IRC | 20:33 | |
| *** weifan has joined #openstack-infra | 20:33 | |
| *** xek_ has quit IRC | 20:34 | |
| *** markvoelker has joined #openstack-infra | 20:34 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add OpenAPI documentation https://review.opendev.org/535541 | 20:35 |
| clarkb | ok infra-ubuntu-bionic-minimal-20190612 exists in both vexxhost sjc1 and rax dfw now. Going to clean up my test upload. Then that bit is done. Next up pulling this stuff out of the nodepool running config | 20:36 |
| *** rfolco has quit IRC | 20:36 | |
| *** weifan has quit IRC | 20:36 | |
| *** mattw4 has joined #openstack-infra | 20:36 | |
| openstackgerrit | Merged opendev/base-jobs master: Allow artifact promote jobs a little more control over renaming https://review.opendev.org/665009 | 20:38 |
| corvus | i love not having clean check | 20:38 |
| *** hwoarang has quit IRC | 20:38 | |
| *** hwoarang has joined #openstack-infra | 20:39 | |
| openstackgerrit | Merged zuul/nodepool master: Add Debian Buster boot tests https://review.opendev.org/649214 | 20:40 |
| *** ekultails has quit IRC | 20:41 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove nodepool builder config for control plane images https://review.opendev.org/665012 | 20:42 |
| clarkb | ok there is the first step | 20:42 |
| clarkb | now to clean up the system-config side | 20:43 |
| clarkb | er actually do I need to set it to an empty list first? | 20:43 |
| clarkb | I think so | 20:43 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:46 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove nodepool builder config for control plane images https://review.opendev.org/665012 | 20:46 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Stop building control plane images https://review.opendev.org/665014 | 20:46 |
| clarkb | there | 20:46 |
| clarkb | (any concern that nodepool might delete too much?) | 20:47 |
| openstackgerrit | Merged zuul/zuul master: Add retry logic to github event handler https://review.opendev.org/664843 | 20:52 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 20:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup nodepool builder clouds.yaml https://review.opendev.org/665018 | 21:01 |
| clarkb | ok I think that is the stack we need. May want mordred to take a look at it too. I don't think it is super urgent as we just won't boot off those nodepool images | 21:01 |
| *** diablo_rojo has quit IRC | 21:06 | |
| openstackgerrit | Merged zuul/zuul master: Fix test-logs.sh utility for multi-ansible https://review.opendev.org/661435 | 21:06 |
| *** markvoelker has quit IRC | 21:08 | |
| *** weifan has joined #openstack-infra | 21:08 | |
| *** rf0lc0 has quit IRC | 21:09 | |
| *** boden has quit IRC | 21:10 | |
| *** pcaruana|afk| has quit IRC | 21:16 | |
| *** weifan has quit IRC | 21:17 | |
| ianw | clarkb / corvus : thanks for the test on the kafs stuff ... the hwe kernel is still a 4 series, i'll build a custom 5.1.9 and we can see. i'll also look at openafs 1.8.3 packages to cover that angle; when i confirm they build ok we can request bionic update | 21:19 |
| clarkb | ianw: sounds like a plan | 21:19 |
| corvus | ++ | 21:19 |
| *** mriedem has quit IRC | 21:24 | |
| *** smarcet has joined #openstack-infra | 21:28 | |
| openstackgerrit | Merged zuul/zuul master: Remove unused code https://review.opendev.org/661436 | 21:30 |
| *** rh-jelabarre has quit IRC | 21:32 | |
| *** weifan has joined #openstack-infra | 21:33 | |
| corvus | clarkb: https://review.opendev.org/664388 is an easy +3 | 21:34 |
| *** raissa has quit IRC | 21:34 | |
| clarkb | done | 21:35 |
| *** raissa has joined #openstack-infra | 21:36 | |
| *** raissa has quit IRC | 21:36 | |
| *** raissa has joined #openstack-infra | 21:37 | |
| *** raissa has quit IRC | 21:37 | |
| *** raissa has joined #openstack-infra | 21:38 | |
| *** raissa has quit IRC | 21:38 | |
| *** weifan has quit IRC | 21:39 | |
| *** raissa has joined #openstack-infra | 21:39 | |
| *** raissa has quit IRC | 21:39 | |
| *** weifan has joined #openstack-infra | 21:42 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:46 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:47 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup nodepool builder clouds.yaml https://review.opendev.org/665018 | 21:47 |
| clarkb | yay testing | 21:48 |
| *** weifan has quit IRC | 21:49 | |
| openstackgerrit | Merged zuul/zuul master: Increase test timeout to 6 minutes https://review.opendev.org/664961 | 21:49 |
| *** weifan has joined #openstack-infra | 21:54 | |
| openstackgerrit | Merged opendev/base-jobs master: base: Move log-inventory to localhost https://review.opendev.org/664388 | 22:03 |
| *** weifan has quit IRC | 22:04 | |
| *** markvoelker has joined #openstack-infra | 22:05 | |
| *** diablo_rojo has joined #openstack-infra | 22:05 | |
| openstackgerrit | Merged zuul/zuul master: executor: use node python path https://review.opendev.org/637339 | 22:08 |
| auristor | corvus: kafs is different from openafs in that each afs volume is a unique device | 22:09 |
| *** weifan has joined #openstack-infra | 22:10 | |
| corvus | auristor: ah, that does make sense. then that makes me wonder why we saw too few volumes mounted during our trial earlier. we should have crossed a few more mounts, but only ended up with two volumes visibly mounted (root.afs and root.cell; we should have also seen 'mirror' and 'mirror.ubuntu' after accessing the path /afs/openstack.org/mirror/ubuntu) | 22:12 |
| auristor | if there is a belief that requests to /afs are deadlocked somewhere, output from /proc/<pid>/stack for each apache process would be useful | 22:12 |
| auristor | the volumes will auto-mounted as required and auto-dismounted when idle | 22:13 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 22:13 |
| corvus | ianw: i'm not sure what the state is on iad now -- is it still running kafs? if so, i'm guessing the slowdown/deadlock from earlier cleared up once we removed the load? | 22:14 |
| fungi | yeah, if whatever went sideways led to apache ceasing to be able to allocate new workers (as the subsequent apache errors indicated) then it's possible by the time we went looking the mounts had basically all gone idle because no new requests for them were coming in | 22:14 |
| ianw | corvus: it is ... it hasn't rebooted or anything | 22:15 |
| fungi | it may be that we need to observe it more closely when it's first put under load before whatever it is happens to apache to cause it to stop handling new requests | 22:15 |
| fungi | though apache seems to have recovered at least enough to serve content again now | 22:16 |
| fungi | i can retrieve http://mirror.iad.rax.opendev.org/pypi/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl through its caching proxy whereas before i couldn't and just got timeouts (even though that doesn't hit the afs backend) | 22:17 |
| *** weifan has quit IRC | 22:18 | |
| ianw | yeah, and pulling up bits of /debian also seems to work ok | 22:18 |
| ianw | interesting ... #openstack.org:mirror.centos on /afs/openstack.org/mirror/centos type afs (rw,relatime) | 22:19 |
| ianw | but debian isn't in there ... despite me looking at it via apache | 22:19 |
| fungi | perhaps auto-unmounted very quickly after your request was fulfilled? | 22:20 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove apport package from ubuntu servers https://review.opendev.org/652143 | 22:20 |
| ianw | now it is, when i go in via cd and look ... this was definitely things not cached with apache, so yeah, it must disappear quickly | 22:21 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:21 |
| corvus | as long as you're cwd in the directory it can't unmount i guess? | 22:22 |
| ianw | coruvs: yeah, i'm out of the dirs | 22:23 |
| ianw | like i pulled up https://mirror.iad.rax.opendev.org/fedora/updates/30/Modular/x86_64/Packages/m/ | 22:23 |
| ianw | no fedora mount | 22:23 |
| fungi | i wasn't able to get the ubuntu volume to show up in the mount output either, even after retrieving esoteric files no client would likely have caused to get cached | 22:24 |
| fungi | and checking within a second or so | 22:24 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Finish removing git.openstack.org references https://review.opendev.org/654040 | 22:28 |
| clarkb | I'm trying to get a few outstanding changes that we probably should get in, well in except they are failing tests so pushing updates to them | 22:29 |
| openstackgerrit | Merged zuul/zuul master: Safely add Ansible password lookup plugin https://review.opendev.org/662870 | 22:29 |
| *** rpioso is now known as rpioso|afk | 22:30 | |
| clarkb | fungi: re apache recovering I'm guessing as enough connections timed out workers were freed up? | 22:32 |
| fungi | my assumption yes | 22:32 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:35 |
| ianw | fungi: here's apache looking through things ... http://paste.openstack.org/show/752849/ | 22:35 |
| ianw | ok, so when i go to /var/www/mirror/fedora on command line, the mount appears | 22:36 |
| clarkb | why is it looking for a readme there? | 22:38 |
| *** markvoelker has quit IRC | 22:38 | |
| fungi | apache mod_autoindex will display a readme if the directory has one | 22:38 |
| fungi | emulating traditional ftp servers | 22:39 |
| ianw | ok, i downloaded ubuntu/db/release.caches.db | 22:39 |
| ianw | apache grabbed it | 22:39 |
| ianw | 2614 22:37:53 openat(AT_FDCWD, "/var/www/mirror/ubuntu/db/release.caches.db", O_RDONLY|O_CLOEXEC) = 30 | 22:39 |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:39 | |
| ianw | there is no afs mount for ubuntu volume ... ? | 22:39 |
| fungi | even though /var/www/mirror/ubuntu is a symlink to /afs/openstack.org/mirror/ubuntu | 22:40 |
| clarkb | if you read the file without cd'ing into the dir do you get a mount? | 22:40 |
| clarkb | automount can be weird like that iirc | 22:40 |
| ianw | hrm, let's see | 22:40 |
| ianw | yep! | 22:41 |
| ianw | cat /var/www/mirror/ubuntu/db/release.caches.db > /tmp/file.txt | 22:41 |
| ianw | now we have a mount for ubuntu volume | 22:41 |
| clarkb | weird | 22:41 |
| *** elod has quit IRC | 22:41 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:41 |
| *** weifan has joined #openstack-infra | 22:41 | |
| *** rcernin has joined #openstack-infra | 22:41 | |
| ianw | proc/fs/afs/openstack.org# cat volumes | wc -l | 22:48 |
| ianw | 20 | 22:48 |
| ianw | i'm not sure how to correlate those volumeids | 22:48 |
| *** weifan has quit IRC | 22:48 | |
| *** ChosSimbaOne has quit IRC | 22:49 | |
| ianw | ahh, it's the volumeid in hex, where "fs" shows it in dec | 22:49 |
| openstackgerrit | Merged zuul/zuul master: Add opendev tarball jobs https://review.opendev.org/664006 | 22:51 |
| clarkb | ianw: left some notes on the backup server/client ansible change. No rush though | 22:54 |
| ianw | echo 'obase=16;536871042' | bc | 22:54 |
| ianw | 20000082 | 22:54 |
| ianw | that's ubuntu-ports | 22:54 |
| ianw | root@mirror01:/proc/fs/afs/openstack.org# cat volumes | grep 82 | 22:54 |
| ianw | 1 20000082 RO | 22:54 |
| ianw | so it clearly knows about it, even if it doesn't show up as a mount | 22:54 |
| *** tkajinam has joined #openstack-infra | 22:56 | |
| *** goldyfruit has quit IRC | 22:59 | |
| *** jamesmcarthur has joined #openstack-infra | 22:59 | |
| *** lifeless has quit IRC | 23:05 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Link development.rst to contributing.rst https://review.opendev.org/645960 | 23:07 |
| *** jamesmcarthur has quit IRC | 23:08 | |
| *** smarcet has quit IRC | 23:10 | |
| *** goldyfruit has joined #openstack-infra | 23:11 | |
| *** michael-beaver has quit IRC | 23:12 | |
| *** aedc has quit IRC | 23:12 | |
| *** diablo_rojo has quit IRC | 23:14 | |
| *** hwoarang has quit IRC | 23:15 | |
| *** hwoarang has joined #openstack-infra | 23:16 | |
| *** weifan has joined #openstack-infra | 23:22 | |
| *** lseki has quit IRC | 23:24 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 23:28 |
| ianw | ok, goign to reboot iad opendev mirror with 5.1.9 kernel | 23:34 |
| *** weifan has quit IRC | 23:35 | |
| *** markvoelker has joined #openstack-infra | 23:35 | |
| ianw | it still does the same thing of not showing hte mounts. i'm not sure what triggers the automount callbacks but i guess apache doesn't do it | 23:38 |
| *** elod has joined #openstack-infra | 23:42 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Revert "Revert "opendev.org mirrors: switch out DFW, switch in IAD"" https://review.opendev.org/665037 | 23:43 |
| ianw | infra-root: ^ any objections? I can monitor for a few hours, then revert if issues | 23:43 |
| clarkb | ianw: none from me, +2'd | 23:43 |
| corvus | +3 | 23:45 |
| ianw | i guess the jinja has the change number there, we could do something like load-balance it in by only setting to opendev mirror on odd changes or something else similarly giving ~50% (current seconds odd/even maybe) | 23:48 |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| *** diablo_rojo has joined #openstack-infra | 23:50 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 23:57 |
| *** aaronsheffield has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!