| *** ociuhandu has quit IRC | 00:01 | |
| *** owalsh has quit IRC | 00:03 | |
| *** owalsh has joined #openstack-infra | 00:03 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project: ansible-role-airship https://review.opendev.org/698114 | 00:04 |
|---|---|---|
| *** rcernin_ has joined #openstack-infra | 00:04 | |
| *** owalsh_ has joined #openstack-infra | 00:05 | |
| *** rcernin has quit IRC | 00:07 | |
| *** owalsh has quit IRC | 00:08 | |
| *** ociuhandu has joined #openstack-infra | 00:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 00:10 |
| *** ociuhandu has quit IRC | 00:14 | |
| *** ociuhandu has joined #openstack-infra | 00:17 | |
| *** rcernin_ has quit IRC | 00:19 | |
| *** ociuhandu has quit IRC | 00:21 | |
| *** ociuhandu has joined #openstack-infra | 00:23 | |
| *** gyee has quit IRC | 00:24 | |
| *** rcernin has joined #openstack-infra | 00:27 | |
| *** ociuhandu has quit IRC | 00:28 | |
| *** jamesmcarthur has joined #openstack-infra | 00:28 | |
| *** dave-mccowan has joined #openstack-infra | 00:29 | |
| *** Goneri has joined #openstack-infra | 00:29 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Also build sibling container images https://review.opendev.org/697393 | 00:30 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 00:30 |
| *** ociuhandu has joined #openstack-infra | 00:31 | |
| *** dave-mccowan has quit IRC | 00:33 | |
| *** sshnaidm is now known as sshnaidm|afk | 00:35 | |
| *** ociuhandu has quit IRC | 00:36 | |
| *** dave-mccowan has joined #openstack-infra | 00:36 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openafs master: Increase udp buffer sizes when running afs https://review.opendev.org/697588 | 00:41 |
| clarkb | lets see if that makes the linter happy | 00:41 |
| *** ociuhandu has joined #openstack-infra | 00:41 | |
| *** kjackal has quit IRC | 00:44 | |
| *** ociuhandu has quit IRC | 00:46 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** rf0lc0 has joined #openstack-infra | 00:51 | |
| *** ociuhandu has joined #openstack-infra | 00:51 | |
| *** rfolco has quit IRC | 00:51 | |
| *** dave-mccowan has quit IRC | 00:52 | |
| *** Goneri has quit IRC | 00:55 | |
| *** ricolin has joined #openstack-infra | 00:56 | |
| *** ociuhandu has quit IRC | 00:57 | |
| *** yamamoto has joined #openstack-infra | 01:01 | |
| ianw | clarkb: i think i caught another mirror job error | 01:01 |
| ianw | https://zuul.opendev.org/t/openstack/build/7469f5925fc04750b600c0e5ef9ed835/log/mirror01.openafs.provider.opendev.org/acme.sh/acme.sh.log | 01:01 |
| ianw | not exactly sure what that means ... | 01:01 |
| *** ociuhandu has joined #openstack-infra | 01:02 | |
| clarkb | I think that is saying authz failed against le staging? | 01:03 |
| ianw | In short, the cURL error code 35 denotes an SSL connection error | 01:03 |
| clarkb | is it possible the remote simply told us to go away? | 01:03 |
| ianw | maybe; it is the staging environment | 01:04 |
| clarkb | ianw: maybe we should have ~3 retries with a wait between attemps if we fail in that manner? I don't know how to distinguish that from "no new cert needed" | 01:04 |
| ianw | ... and then the openafs client failed to install for unknown reasons too https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8fa/697587/7/check/system-config-run-static/8fabb26/bridge.openstack.org/ara-report/result/e747fab8-e775-4b05-9a55-6a48b78370cf/ | 01:05 |
| ianw | actually: Cache size (50000000) must be less than 95% of partition size (which is 36856200). Lower cache size | 01:06 |
| ianw | are we not growing disks properly in some cases? | 01:06 |
| *** ociuhandu has quit IRC | 01:06 | |
| clarkb | ianw: did that job run on rax? it has a smaller / then we put big /opt in place | 01:06 |
| *** jamesmcarthur has quit IRC | 01:07 | |
| ianw | it did run in rax (https://zuul.opendev.org/t/openstack/build/8fabb262c9f24cee9a1f399c90ec8b77/log/zuul-info/inventory.yaml) ... but i've never seen that before | 01:08 |
| ianw | surely this can't be the first time an openafs system-config job ahs run in rax | 01:08 |
| clarkb | ianw: if you grep the build uuid on zuul executors the cleanup playbook should capture a df | 01:09 |
| clarkb | to confirm this hunch | 01:09 |
| *** goldyfruit_ has joined #openstack-infra | 01:09 | |
| ianw | on a fresh rax-ord node : /dev/xvda1 37G 4.0G 32G 12% / | 01:11 |
| *** ociuhandu has joined #openstack-infra | 01:12 | |
| clarkb | is 50000000 5GB? | 01:12 |
| clarkb | or 50GB? | 01:12 |
| clarkb | not sure what unit we are working with | 01:12 |
| ianw | should be 50gb ... how has this never failed before? | 01:12 |
| clarkb | maybe it didn't check before? | 01:12 |
| clarkb | or we always got lucky and ran on not rax? | 01:12 |
| ianw | why am i always so lucky! :) | 01:13 |
| ianw | zuul/templates/group_vars/mirror_opendev.yaml.j2:afs_client_cache_size: 500000 # 500MiB | 01:15 |
| clarkb | now that I think about this more I want to say we do ya that | 01:15 |
| clarkb | are we not applying that properly? | 01:16 |
| ianw | ohhhhh ... that's only applying to mirror node | 01:16 |
| clarkb | ah | 01:16 |
| ianw | not the new static.opendev.org node | 01:16 |
| ianw | ok, that makes sense | 01:16 |
| clarkb | ianw: we can set that in group_vars/all.yaml | 01:16 |
| clarkb | under zuul/templates | 01:16 |
| ianw | yeah, or for all afs-clients | 01:16 |
| *** ociuhandu has quit IRC | 01:16 | |
| clarkb | then it should apply to all hosts | 01:16 |
| clarkb | ++ | 01:16 |
| *** ociuhandu has joined #openstack-infra | 01:21 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** jamesmcarthur has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** ociuhandu has quit IRC | 01:26 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Enable devstack force=true for xenial based jobs https://review.opendev.org/698122 | 01:29 |
| *** ociuhandu has joined #openstack-infra | 01:29 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Enable devstack force=yes for xenial based jobs https://review.opendev.org/698122 | 01:32 |
| *** jamesmcarthur has quit IRC | 01:32 | |
| ianw | hrm,i thought we had a change out to updat ethat to bionic | 01:34 |
| *** ociuhandu has quit IRC | 01:37 | |
| *** larainema has joined #openstack-infra | 01:38 | |
| *** ociuhandu has joined #openstack-infra | 01:42 | |
| *** goldyfruit_ has quit IRC | 01:43 | |
| *** ociuhandu has quit IRC | 01:47 | |
| larainema | hello infra-core, question here: how we can fork opendev source code? like to github | 01:48 |
| *** ociuhandu has joined #openstack-infra | 01:49 | |
| ianw | larainema: do you mean to mirror a repository, so that the changes are in sync with opendev? | 01:52 |
| *** ociuhandu has quit IRC | 01:57 | |
| *** lastmikoi has quit IRC | 02:01 | |
| larainema | ianw, yes | 02:06 |
| ianw | larainema: you need to define a job to push changes; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html | 02:08 |
| *** ociuhandu has joined #openstack-infra | 02:08 | |
| *** lastmikoi has joined #openstack-infra | 02:08 | |
| *** nicolasbock has quit IRC | 02:10 | |
| *** Xuchu has joined #openstack-infra | 02:11 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 02:12 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: afs-client: move reduced cache to group variable https://review.opendev.org/698128 | 02:12 |
| *** ociuhandu has quit IRC | 02:13 | |
| *** ociuhandu has joined #openstack-infra | 02:14 | |
| *** slaweq has joined #openstack-infra | 02:17 | |
| larainema | thanks ianw, will look at it | 02:17 |
| *** slaweq has quit IRC | 02:22 | |
| *** ociuhandu has quit IRC | 02:22 | |
| *** igordc has quit IRC | 02:24 | |
| *** ykarel|away has joined #openstack-infra | 02:34 | |
| *** ociuhandu has joined #openstack-infra | 02:38 | |
| *** dmellado has quit IRC | 02:45 | |
| *** vesper11 has quit IRC | 02:45 | |
| *** tridde has joined #openstack-infra | 02:45 | |
| *** trident has quit IRC | 02:46 | |
| *** vesper11 has joined #openstack-infra | 02:46 | |
| *** ociuhandu has quit IRC | 02:47 | |
| *** dmellado has joined #openstack-infra | 02:48 | |
| *** diablo_rojo has quit IRC | 02:51 | |
| *** yamamoto has quit IRC | 02:55 | |
| *** yamamoto has joined #openstack-infra | 03:01 | |
| *** slaweq has joined #openstack-infra | 03:06 | |
| *** slaweq has quit IRC | 03:10 | |
| *** Xuchu_ has joined #openstack-infra | 03:20 | |
| *** Xuchu has quit IRC | 03:23 | |
| *** Xuchu_ is now known as Xuchu | 03:23 | |
| *** slaweq has joined #openstack-infra | 03:25 | |
| *** slaweq has quit IRC | 03:30 | |
| *** psachin has joined #openstack-infra | 03:31 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** ociuhandu has joined #openstack-infra | 03:47 | |
| *** diablo_rojo has joined #openstack-infra | 03:48 | |
| *** hongbin has joined #openstack-infra | 03:50 | |
| *** ociuhandu has quit IRC | 03:51 | |
| *** ramishra has joined #openstack-infra | 03:55 | |
| *** nicolasbock has joined #openstack-infra | 04:02 | |
| *** rh-jelabarre has quit IRC | 04:08 | |
| openstackgerrit | Sampath Priyankara (samP) proposed opendev/irc-meetings master: Change Masakari meeting frequency to biweekly-even https://review.opendev.org/698135 | 04:15 |
| *** dchen has quit IRC | 04:19 | |
| *** dchen has joined #openstack-infra | 04:21 | |
| *** slaweq has joined #openstack-infra | 04:29 | |
| *** rishabhhpe has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:32 | |
| *** tsukasasa has joined #openstack-infra | 04:32 | |
| *** slaweq has quit IRC | 04:36 | |
| *** ociuhandu has joined #openstack-infra | 04:41 | |
| *** hongbin has quit IRC | 04:41 | |
| *** udesale has joined #openstack-infra | 04:41 | |
| *** ykarel|away has quit IRC | 04:43 | |
| *** ociuhandu has quit IRC | 04:46 | |
| *** nicolasbock has quit IRC | 04:46 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 04:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 04:49 |
| *** tsukasasa has quit IRC | 04:53 | |
| *** ykarel|away has joined #openstack-infra | 04:56 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 04:57 |
| *** diablo_rojo has quit IRC | 05:00 | |
| openstackgerrit | Jens Harbott (frickler) proposed opendev/gerritbot master: WIP: Add Python 3 support https://review.opendev.org/545607 | 05:00 |
| *** surpatil has joined #openstack-infra | 05:07 | |
| *** raukadah is now known as chkumar|ruck | 05:23 | |
| *** ociuhandu has joined #openstack-infra | 05:35 | |
| *** slaweq has joined #openstack-infra | 05:35 | |
| *** ykarel|away is now known as ykarel | 05:36 | |
| *** ociuhandu has quit IRC | 05:40 | |
| *** slaweq has quit IRC | 05:40 | |
| *** slaweq has joined #openstack-infra | 05:46 | |
| *** slaweq has quit IRC | 05:50 | |
| openstackgerrit | Sampath Priyankara (samP) proposed openstack/project-config master: Add horizon-core to masakari-dashboard https://review.opendev.org/698141 | 05:51 |
| *** SurajPatil has joined #openstack-infra | 05:52 | |
| *** surpatil has quit IRC | 05:54 | |
| *** surpatil has joined #openstack-infra | 06:01 | |
| *** slaweq has joined #openstack-infra | 06:01 | |
| *** jamesmcarthur has joined #openstack-infra | 06:01 | |
| *** SurajPatil has quit IRC | 06:03 | |
| *** slaweq has quit IRC | 06:05 | |
| *** jamesmcarthur has quit IRC | 06:06 | |
| *** kjackal has joined #openstack-infra | 06:08 | |
| *** dpawlik has joined #openstack-infra | 06:12 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** slaweq has joined #openstack-infra | 06:18 | |
| *** dpawlik has quit IRC | 06:18 | |
| *** dpawlik has joined #openstack-infra | 06:20 | |
| *** yamamoto has joined #openstack-infra | 06:21 | |
| *** slaweq has quit IRC | 06:23 | |
| *** dpawlik has quit IRC | 06:27 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop Xenial from functional test https://review.opendev.org/698136 | 06:27 |
| *** ociuhandu has joined #openstack-infra | 06:29 | |
| *** jtomasek has joined #openstack-infra | 06:31 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: enqueue: make trigger deprecated https://review.opendev.org/695446 | 06:34 |
| *** slaweq has joined #openstack-infra | 06:34 | |
| *** ociuhandu has quit IRC | 06:34 | |
| *** slaweq has quit IRC | 06:39 | |
| *** psachin has quit IRC | 06:42 | |
| larainema | hi ianw, if the failed post pipline build can be re-triggered? | 06:42 |
| *** dpawlik has joined #openstack-infra | 06:43 | |
| *** psachin has joined #openstack-infra | 06:49 | |
| rishabhhpe | Hi .. Openstack is not supporting for ubuntu xenial release ? | 07:00 |
| *** lpetrut has joined #openstack-infra | 07:00 | |
| *** diablo_rojo has joined #openstack-infra | 07:01 | |
| *** ociuhandu has joined #openstack-infra | 07:02 | |
| *** ociuhandu has quit IRC | 07:06 | |
| AJaeger | rishabhhpe: older OpenStack releases do. But this is not the right place to ask, we run the Infrastructure here to develop and test OpenStack. Better ask on #openstack if you have generic questions | 07:08 |
| AJaeger | larainema: why not merge another change? | 07:08 |
| larainema | AJaeger, yes, that's what I am doing, just curious if we have a way to do it | 07:12 |
| rishabhhpe | Ajaeger: Thank you | 07:13 |
| *** ociuhandu has joined #openstack-infra | 07:26 | |
| *** pgaxatte has joined #openstack-infra | 07:28 | |
| *** ociuhandu has quit IRC | 07:32 | |
| AJaeger | larainema: an admin can do it for you - but that's a manual step, so we very rarely do it | 07:37 |
| *** pcaruana has joined #openstack-infra | 07:39 | |
| openstackgerrit | Merged zuul/nodepool master: Dockerfile: add user to shadow file too https://review.opendev.org/693647 | 07:46 |
| *** ociuhandu has joined #openstack-infra | 07:46 | |
| *** ykarel is now known as ykarel|lunch | 07:47 | |
| *** yamamoto has quit IRC | 07:52 | |
| larainema | AJaeger, understood, thanks, no worry, I am merging another change. | 07:54 |
| *** rcernin has quit IRC | 07:54 | |
| *** rishabhhpe has quit IRC | 07:55 | |
| *** slaweq has joined #openstack-infra | 08:01 | |
| *** tkajinam has quit IRC | 08:06 | |
| *** factor has quit IRC | 08:10 | |
| *** tosky has joined #openstack-infra | 08:11 | |
| *** witek has joined #openstack-infra | 08:13 | |
| openstackgerrit | Sampath Priyankara (samP) proposed openstack/project-config master: Add horizon-core to masakari-dashboard https://review.opendev.org/698141 | 08:13 |
| *** rishabhhpe has joined #openstack-infra | 08:15 | |
| *** apetrich has joined #openstack-infra | 08:16 | |
| *** dchen has quit IRC | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:27 | |
| *** kjackal has quit IRC | 08:34 | |
| *** cgoncalves has quit IRC | 08:40 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** hashar has joined #openstack-infra | 08:43 | |
| *** kjackal has joined #openstack-infra | 08:46 | |
| *** iurygregory has joined #openstack-infra | 08:46 | |
| *** ociuhandu has quit IRC | 08:48 | |
| *** ralonsoh has joined #openstack-infra | 08:48 | |
| *** jpena|off is now known as jpena | 08:53 | |
| *** pkopec has joined #openstack-infra | 08:54 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:58 | |
| *** priteau has joined #openstack-infra | 09:00 | |
| kevinz | ianw: ping | 09:01 |
| sshnaidm | please, remind - how can I use jobs in project that are defined in other project? | 09:04 |
| *** yamamoto has joined #openstack-infra | 09:06 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** gfidente|afk has quit IRC | 09:13 | |
| *** lucasagomes has joined #openstack-infra | 09:15 | |
| *** ociuhandu has joined #openstack-infra | 09:17 | |
| frickler | sshnaidm: job definitions should be global (within the tenant, e.g. openstack), so you should be able to use them just like any other job | 09:20 |
| *** ociuhandu has quit IRC | 09:22 | |
| sshnaidm | frickler, not sure I understand.. I need to use the job in different project: https://review.opendev.org/#/c/698085/ | 09:23 |
| frickler | sshnaidm: you may not use duplicate job names, precisely because they are global. if you define a job within project ansible-collections-openstack, it's name should start with the project name | 09:25 |
| *** spsurya has joined #openstack-infra | 09:25 | |
| frickler | sshnaidm: otherwise omit the job definition part in your change and just do the project part | 09:25 |
| *** derekh has joined #openstack-infra | 09:37 | |
| *** owalsh_ is now known as owalsh | 09:37 | |
| *** ociuhandu has joined #openstack-infra | 09:42 | |
| *** gfidente has joined #openstack-infra | 09:47 | |
| openstackgerrit | Merged opendev/irc-meetings master: Change Masakari meeting frequency to biweekly-even https://review.opendev.org/698135 | 09:52 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: remove connectors burden and simplify code https://review.opendev.org/696134 | 09:59 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional max_token_age, skew https://review.opendev.org/642408 | 10:04 |
| *** priteau has quit IRC | 10:05 | |
| *** dtantsur|afk is now known as dtantsur | 10:15 | |
| *** slaweq has quit IRC | 10:17 | |
| *** slaweq has joined #openstack-infra | 10:19 | |
| *** derekh has quit IRC | 10:20 | |
| *** ykarel|lunch is now known as ykarel | 10:22 | |
| *** pgaxatte has quit IRC | 10:22 | |
| *** etingof has quit IRC | 10:26 | |
| openstackgerrit | Merged zuul/zuul master: enqueue: make trigger deprecated https://review.opendev.org/695446 | 10:30 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Refresh token and recheck sign when token regenerated https://review.opendev.org/698208 | 10:33 |
| *** hashar has quit IRC | 10:48 | |
| *** kambiz has quit IRC | 11:09 | |
| *** kjackal has quit IRC | 11:11 | |
| *** ociuhandu has quit IRC | 11:13 | |
| *** ociuhandu has joined #openstack-infra | 11:13 | |
| *** kambiz has joined #openstack-infra | 11:14 | |
| *** kjackal has joined #openstack-infra | 11:23 | |
| *** ociuhandu has quit IRC | 11:28 | |
| *** rishabhhpe has quit IRC | 11:34 | |
| *** rf0lc0 has quit IRC | 11:40 | |
| *** ijw has joined #openstack-infra | 11:43 | |
| *** ykarel is now known as ykarel|away | 11:43 | |
| *** ijw has quit IRC | 11:47 | |
| *** dtantsur is now known as dtantsur|ikea | 11:54 | |
| *** jklare has quit IRC | 11:58 | |
| *** nicolasbock has joined #openstack-infra | 12:00 | |
| *** SurajPatil has joined #openstack-infra | 12:00 | |
| *** mtreinish has joined #openstack-infra | 12:00 | |
| *** jklare has joined #openstack-infra | 12:01 | |
| *** udesale has quit IRC | 12:01 | |
| *** lucasagomes is now known as lucas-hungry | 12:02 | |
| *** weshay_pto is now known as weshay | 12:03 | |
| *** surpatil has quit IRC | 12:03 | |
| *** cgoncalves has joined #openstack-infra | 12:09 | |
| *** owalsh has quit IRC | 12:14 | |
| *** ociuhandu has joined #openstack-infra | 12:23 | |
| *** ociuhandu has quit IRC | 12:27 | |
| *** surpatil has joined #openstack-infra | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| *** SurajPatil has quit IRC | 12:34 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 12:34 |
| *** spsurya has quit IRC | 12:35 | |
| *** larainema has quit IRC | 12:44 | |
| *** owalsh has joined #openstack-infra | 12:46 | |
| *** cgoncalves has quit IRC | 12:47 | |
| openstackgerrit | Albin Vass proposed zuul/nodepool master: Keys must be defined for host-key-checking: false https://review.opendev.org/698029 | 12:48 |
| *** Xuchu has quit IRC | 12:51 | |
| *** rh-jelabarre has joined #openstack-infra | 12:55 | |
| *** gfidente has quit IRC | 13:00 | |
| *** dpawlik has quit IRC | 13:01 | |
| *** rlandy has joined #openstack-infra | 13:01 | |
| *** hashar has joined #openstack-infra | 13:06 | |
| *** lucas-hungry is now known as lucasagomes | 13:06 | |
| *** owalsh has quit IRC | 13:08 | |
| *** owalsh has joined #openstack-infra | 13:08 | |
| *** aaronsheffield has joined #openstack-infra | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** cgoncalves has joined #openstack-infra | 13:23 | |
| *** jamesmcarthur has quit IRC | 13:28 | |
| *** owalsh has quit IRC | 13:29 | |
| *** jamesmcarthur has joined #openstack-infra | 13:30 | |
| *** jamesmcarthur has quit IRC | 13:34 | |
| *** owalsh has joined #openstack-infra | 13:35 | |
| *** goldyfruit_ has joined #openstack-infra | 13:37 | |
| *** jpena|lunch is now known as jpena | 13:37 | |
| *** goldyfruit___ has joined #openstack-infra | 13:41 | |
| *** Goneri has joined #openstack-infra | 13:43 | |
| *** goldyfruit_ has quit IRC | 13:44 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** Xuchu has joined #openstack-infra | 13:56 | |
| *** tkajinam has joined #openstack-infra | 14:01 | |
| *** mriedem has joined #openstack-infra | 14:01 | |
| *** liuyulong has joined #openstack-infra | 14:02 | |
| *** ykarel|away has quit IRC | 14:04 | |
| *** gfidente has joined #openstack-infra | 14:06 | |
| *** eharney has quit IRC | 14:07 | |
| *** Xuchu has quit IRC | 14:23 | |
| *** chkumar|ruck is now known as raukadah | 14:26 | |
| *** liuyulong has quit IRC | 14:32 | |
| *** ociuhandu has joined #openstack-infra | 14:35 | |
| *** pcaruana has quit IRC | 14:36 | |
| *** surpatil has quit IRC | 14:36 | |
| *** hashar has quit IRC | 14:38 | |
| *** ykarel|away has joined #openstack-infra | 14:40 | |
| *** ykarel|away is now known as ykarel | 14:40 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Update Neutron's Grafana dashboard https://review.opendev.org/698264 | 14:41 |
| *** eharney has joined #openstack-infra | 14:47 | |
| openstackgerrit | Monty Taylor proposed opendev/storyboard master: Build container images https://review.opendev.org/611191 | 14:48 |
| *** tkajinam has quit IRC | 14:52 | |
| openstackgerrit | Merged zuul/nodepool master: Aws cloud-image is referred to from pool labels section https://review.opendev.org/697998 | 14:58 |
| *** igordc has joined #openstack-infra | 14:58 | |
| *** lbragsta_ has joined #openstack-infra | 14:59 | |
| *** pcaruana has joined #openstack-infra | 15:01 | |
| *** jamesmcarthur has quit IRC | 15:03 | |
| *** rosmaita has quit IRC | 15:06 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 15:09 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [DNM][WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 15:09 |
| *** rosmaita has joined #openstack-infra | 15:09 | |
| *** ijw has joined #openstack-infra | 15:10 | |
| *** rascasoft has joined #openstack-infra | 15:12 | |
| *** jamesmcarthur has joined #openstack-infra | 15:15 | |
| *** jamesmcarthur has quit IRC | 15:15 | |
| *** bnemec has quit IRC | 15:15 | |
| *** jamesmcarthur_ has joined #openstack-infra | 15:15 | |
| fungi | infra-root: heads up, we got a notification from inap about a job node with an exposed portmapper service (this can be found by crawlers and exploited to mount reflection attacks against victims elsewhere on the internet. they gave us an ip address and timestamp, so i've been able to track it back to this build: http://zuul.opendev.org/t/openstack/build/93f6432502b0428486d24a97fe30e824 | 15:21 |
| fungi | slaweq: that ^ seems to be a build for a change in a neutron deliverable | 15:24 |
| Shrews | fungi: is that node still active? | 15:27 |
| fungi | this is the detail from the report we received: http://paste.openstack.org/show/787376/ | 15:27 |
| fungi | Shrews: heavens no, it just happened to get scanned by inap's vulnerability scanner while a job was running | 15:28 |
| Shrews | oh, i thought it would be odd to be long-lived | 15:28 |
| Shrews | unless we held it | 15:29 |
| fungi | in the past we've used incidents like this to help teams fix their jobs to not expose dangerous services to the internet | 15:29 |
| frickler | hmm, "udp 0 0 0.0.0.0:111 0.0.0.0:* 1/systemd" is what I see on a centos8 node I happen to have held ... :-/ | 15:31 |
| fungi | yeah, is iptables/netfilter allowing access to it from the world? | 15:33 |
| fungi | in most cases, we see this happen in jobs where someone has decided that figuring out firewall rules is "hard" and so they just turn them off | 15:33 |
| slaweq | fungi: I don't see anything "special" in the patch on which this runs | 15:34 |
| fungi | slaweq: neither did i, it may be systemic to networking-ovn-tempest-dsvm-ovs-release jobs, or an even broader class of jobs from which that one inherits | 15:34 |
| slaweq | fungi: this job inherits from networking-ovn-base: https://opendev.org/openstack/networking-ovn/src/branch/master/zuul.d/networking-ovn-jobs.yaml#L2 | 15:35 |
| slaweq | and this one from devstack-tempest | 15:35 |
| fungi | most of our providers don't run vulnerability scanners across servers, and the intersection of an infrequent (perhaps daily) scan with specific jobs running in that provider could result in few opportunities for them to spot and notify us | 15:36 |
| *** lbragsta_ has quit IRC | 15:36 | |
| slaweq | and I don't see anything "special" in this job definition | 15:36 |
| fungi | probably best to stop assuming there's anything special in the change or the job definition, and instead think in terms of some class of jobs may be incorrectly handling firewall configuration | 15:37 |
| *** lbragsta_ has joined #openstack-infra | 15:37 | |
| *** ijw has quit IRC | 15:40 | |
| *** iokiwi has quit IRC | 15:40 | |
| *** adriant has quit IRC | 15:40 | |
| corvus | i don't know when this was generated, but i'd guess pretty late in the job: http://zuul.opendev.org/t/openstack/build/93f6432502b0428486d24a97fe30e824/log/controller/logs/iptables.txt.gz | 15:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional max_validity_time, skew https://review.opendev.org/642408 | 15:41 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [DNM][WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 15:42 |
| fungi | er, also i said inap earlier, but this was ovh not inap | 15:43 |
| frickler | fungi: how certain can we be that the timestamp is indeed utc? | 15:44 |
| fungi | er, sorry, node was in inap, logs are in ovh | 15:47 |
| fungi | moar coffee | 15:47 |
| * Shrews shovels coffee beans on top of fungi :) | 15:47 | |
| fungi | frickler: yeah, it's a fair point, we only have a few times where nodes were booted with that ip address by nl03 that day though | 15:47 |
| fungi | the devstack log indicates the time of the scan (if we assume utc and accurate clocks) occurred in the middle of `pip2.7 install -c /opt/stack/requirements/upper-constraints.txt -r /opt/stack/neutron-tempest-plugin/test-requirements.txt -e /opt/stack/neutron-tempest-plugin` | 15:48 |
| mordred | Shrews: can I have some of those? | 15:48 |
| Shrews | beans for all! | 15:49 |
| fungi | mgagne: do you have any idea if inap's network scanner reports are timestamped in utc? they don't say: http://paste.openstack.org/show/787376/ | 15:50 |
| *** ykarel is now known as ykarel|afk | 15:50 | |
| fungi | at 04:05:06.947 the devstack log identifies that iptables is already installed, so that's good at least | 15:53 |
| corvus | curious the event report doesn't agree on the date | 15:53 |
| corvus | if the subject date is EST, and the eventtime is UTC, that is plausible considering utc offset | 15:55 |
| corvus | still, sticking a "Z" on the eventtime would be nice | 15:55 |
| *** lbragsta_ has quit IRC | 15:55 | |
| frickler | I tested against a currently running job of the same type and don't see anything open. confirmed by logging into 158.69.70.17 http://zuul.openstack.org/stream/d495ccb08d094839a0c5a6ef21afaea5?logfile=console.log | 15:56 |
| frickler | that job is currently in the tempest run phase | 15:56 |
| *** goldyfruit_ has joined #openstack-infra | 15:57 | |
| *** jklare has quit IRC | 15:59 | |
| fungi | corvus: yeah, i too found that interesting. i also suspect that they may date the scans by when they initiate them, so it could have wrapped over the midnight boundary between when the scan started and when that particular problem was identified | 15:59 |
| *** goldyfruit___ has quit IRC | 16:00 | |
| corvus | fungi: oh yep, then that gives a plausible explanation for both being local time | 16:00 |
| fungi | or both being utc for that matter | 16:01 |
| *** jklare has joined #openstack-infra | 16:02 | |
| *** ociuhandu has quit IRC | 16:02 | |
| fungi | anyway, might make sense to put a pin in this investigation and keep an eye out for anything we think could contribute to that behavior or see if we get more notices which will allow us to better correlate | 16:02 |
| *** dtantsur|ikea is now known as dtantsur | 16:02 | |
| fungi | i mostly wanted to jump on it quickly because they took nearly a week to let us know and in the past we've gotten reports from them just at the edge of our nodepool log retention | 16:03 |
| fungi | now we at least have something to compare subsequent reports against | 16:03 |
| *** ijw has joined #openstack-infra | 16:09 | |
| *** igordc has quit IRC | 16:11 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 16:12 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.9 https://review.opendev.org/674854 | 16:12 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 16:16 |
| mordred | fungi: https://review.opendev.org/#/c/611191/ is green if you wanna re-+2 it | 16:17 |
| *** iurygregory has quit IRC | 16:18 | |
| *** iurygregory has joined #openstack-infra | 16:19 | |
| *** ricolin has quit IRC | 16:19 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Set correct python version for non-chroot scripts https://review.opendev.org/697211 | 16:23 |
| fungi | mordred: was ps10 just a rebase? | 16:23 |
| mordred | fungi: yes | 16:24 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Extract an abstract base Parser class https://review.opendev.org/545610 | 16:28 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 16:29 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.9 https://review.opendev.org/674854 | 16:29 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 16:29 |
| *** lpetrut has quit IRC | 16:29 | |
| mgagne | fungi: sorry, I would have to ask the abuse department about that one. | 16:31 |
| *** gyee has joined #openstack-infra | 16:37 | |
| *** lpetrut has joined #openstack-infra | 16:39 | |
| fungi | mgagne: no worries, we may be able to better confirm it if we end eventually up with more reports | 16:40 |
| *** goldyfruit___ has joined #openstack-infra | 16:41 | |
| mgagne | fungi: yeah, I wouldn't worry about them for now. I'm trying to track down who manages the scanner. There was a lot of staff movements in the last months/years. | 16:41 |
| *** goldyfruit_ has quit IRC | 16:44 | |
| *** jpena is now known as jpena|brb | 16:45 | |
| *** lpetrut has quit IRC | 16:50 | |
| *** jaosorior has joined #openstack-infra | 16:50 | |
| fungi | thanks! | 16:51 |
| mordred | infra-root: got an update from RH IT on the email issue from yesterday - there is investigation underway as to what's going on, and they also indicated that they're putting in a workaround in the meantime | 16:51 |
| fungi | thanks for the update mordred! | 16:53 |
| *** ykarel|afk is now known as ykarel|away | 16:53 | |
| *** ijw_ has joined #openstack-infra | 16:53 | |
| *** ijw has quit IRC | 16:58 | |
| *** tesseract has quit IRC | 16:58 | |
| *** iurygregory has quit IRC | 17:02 | |
| *** lucasagomes has quit IRC | 17:04 | |
| *** ykarel|away has quit IRC | 17:08 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [DNM][WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 17:08 |
| mgagne | fungi: timezone is UTC. | 17:19 |
| mgagne | fungi: I suggested adding a Z to indicate UTC timezone. | 17:20 |
| openstackgerrit | Merged opendev/storyboard master: Build container images https://review.opendev.org/611191 | 17:22 |
| *** witek has quit IRC | 17:23 | |
| *** jpena|brb is now known as jpena | 17:24 | |
| openstackgerrit | Merged zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 17:24 |
| *** ociuhandu has joined #openstack-infra | 17:25 | |
| *** tosky has quit IRC | 17:27 | |
| *** ociuhandu has quit IRC | 17:29 | |
| *** rpittau is now known as rpittau|afk | 17:29 | |
| fungi | thanks mgagne!!! we suspected that was the case but it's good to be sure | 17:32 |
| *** rlandy is now known as rlandy|brb | 17:36 | |
| *** AR2018 has joined #openstack-infra | 17:37 | |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| *** AR2018 has quit IRC | 17:39 | |
| openstackgerrit | Merged zuul/zuul master: Support Ansible 2.9 https://review.opendev.org/674854 | 17:40 |
| *** rlandy|brb is now known as rlandy | 17:45 | |
| corvus | we'd like to restart zuul again to smoketest the new 2.9 support before cutting a release | 17:48 |
| corvus | i'll let release know | 17:49 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| *** armax has quit IRC | 17:51 | |
| corvus | restarting now | 17:53 |
| corvus | fungi, clarkb: it looks like we may actually want to be running something like "python /opt/zuul/tools/zuul-changes.py http://zuul.opendev.org >queue.sh" | 17:54 |
| corvus | that appears to do all pipelines of all tenants | 17:54 |
| corvus | without the weird "None" tenant thing | 17:54 |
| corvus | 15afed554e2bbac87aad83b0a42cd47c66858cfd is the commit | 17:55 |
| *** jaosorior has quit IRC | 17:59 | |
| fungi | corvus: i concur. that's where i arrived at with my analysis yesterday | 17:59 |
| fungi | if you're pointing it at a multi-tenant api url the tenant parameter is effectively ignored | 18:00 |
| fungi | at which point filtering across tenants by common pipeline names seems a little strange | 18:00 |
| fungi | but as you observed, you can just omit them and get a dump of everything, then filter the result by tenant/pipeline as needed when reenqueuing | 18:01 |
| AJaeger | mnaser: want to merge https://review.opendev.org/681584 as next step for static.o.o? That's the final patch I pushed, so rest is for you... | 18:03 |
| corvus | merger stopping is very slow ... | 18:04 |
| *** ijw_ has quit IRC | 18:07 | |
| fungi | i don't remember seeing that in yesterday's restart | 18:07 |
| corvus | i think we're waiting on zm03 | 18:08 |
| *** ijw has joined #openstack-infra | 18:09 | |
| corvus | looks like it's still doing something with nova | 18:09 |
| corvus | we probably need to clean out those repos | 18:09 |
| clarkb | ya executor shutdown was the slow thing yesterday ~10 minutes | 18:09 |
| corvus | we could kill it, or, it's still running, so we could just wait a bit more | 18:10 |
| johnsom | "Something went wrong." I guess there is some kind of maintenance going on? | 18:10 |
| corvus | it seems to take 3 minutes to create each merge commit | 18:11 |
| corvus | and it's a bkc nova stack | 18:11 |
| corvus | johnsom: yes, we're restarting zuul; it's a bit slow | 18:11 |
| mnaser | AJaeger: what would be the next steps? | 18:15 |
| *** lbragstad_ has joined #openstack-infra | 18:16 | |
| *** hrybacki has joined #openstack-infra | 18:17 | |
| *** rf0lc0 has joined #openstack-infra | 18:22 | |
| *** goldyfruit_ has joined #openstack-infra | 18:22 | |
| *** goldyfruit___ has quit IRC | 18:25 | |
| *** jamesmcarthur_ has quit IRC | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| TheJulia | should zuul.opendev.org still be returning 500s? | 18:26 |
| *** ijw has quit IRC | 18:26 | |
| corvus | i reckon we should kill zm03 | 18:27 |
| *** ijw has joined #openstack-infra | 18:27 | |
| noonedeadpunk | btw, it looks like some patches don't appear in zuul at all - for instance https://review.opendev.org/#/c/695877/ | 18:28 |
| *** ijw has joined #openstack-infra | 18:28 | |
| corvus | okay it finished the restart | 18:28 |
| fungi | yeah, let's just ungracefully stop the merger daemon on 03 | 18:28 |
| fungi | did you wind up manually killing it or did it just now wrap up on its own | 18:28 |
| fungi | ? | 18:28 |
| corvus | i killed it | 18:28 |
| corvus | the scheduler is not running | 18:29 |
| fungi | noonedeadpunk: yeah, we're in the middle of a restart, the earlier queued changes will be readded from a dump which was created just prior to the restart | 18:29 |
| corvus | restarting that | 18:29 |
| corvus | i'm assuming the scheduler timed out waiting for mergers | 18:30 |
| Shrews | yay for sorted hold requests | 18:32 |
| corvus | okay it's back up now | 18:35 |
| corvus | re-enqueing | 18:35 |
| *** jamesmcarthur has quit IRC | 18:36 | |
| *** ociuhandu has joined #openstack-infra | 18:39 | |
| *** kopecmartin is now known as kopecmartin|off | 18:41 | |
| *** jpena is now known as jpena|off | 18:42 | |
| *** ociuhandu has quit IRC | 18:44 | |
| corvus | #status log restarted all of zuul at commit 15afed554e2bbac87aad83b0a42cd47c66858cfd | 18:45 |
| AJaeger | mnaser: best check the spec, I don't remember all details. AFAIK you have two options as next step: 1) Add jobs for remaining sites ; 2) Shutdown old site, remove old jobs and use new static site. I think ianw lays the groundwork for 2 right now. | 18:45 |
| openstackstatus | corvus: finished logging | 18:45 |
| *** ijw_ has joined #openstack-infra | 18:45 | |
| corvus | pabelanger: you should be clear to test 2.9 now | 18:45 |
| *** gfidente is now known as gfidente|afk | 18:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move security site to AFS publishing https://review.opendev.org/681584 | 18:47 |
| *** ijw has quit IRC | 18:48 | |
| pabelanger | corvus: \o/ | 18:48 |
| pabelanger | tyty | 18:48 |
| mordred | infra-root: today's plague of the stomach is enworsening so I'm going to step AFK for a bit. I don't have much progress to report in meeting due to last week's plague | 18:49 |
| clarkb | mordred: the USDA recommends you cook turkey to a minimum of 165F | 18:50 |
| *** psachin has quit IRC | 19:02 | |
| corvus | infra-root: there seems to be a lot of failing tox related jobs | 19:03 |
| EmilienM | hi all, we have our pep8 jobs failing | 19:03 |
| EmilienM | "msg": "a bytes-like object is required, not 'str'" | 19:04 |
| corvus | http://paste.openstack.org/show/787397/ | 19:04 |
| EmilienM | yeah | 19:04 |
| corvus | pabelanger: could tox-siblings be incompatible with the version bump? | 19:04 |
| corvus | maybe we should set the default back to 2.7 | 19:05 |
| pabelanger | did python interpreter get changed to python3? | 19:05 |
| pabelanger | with auto setting now | 19:05 |
| corvus | pabelanger: would that have changed between the last 2 restarts? | 19:05 |
| pabelanger | checking | 19:05 |
| corvus | or are you thinking python3 + 2.8 may be a problem? | 19:06 |
| pabelanger | IIRC, that is 2.8 default now | 19:06 |
| pabelanger | ya | 19:06 |
| pabelanger | if jobs were upgraded to ansible 2.8, ithink that is default now | 19:06 |
| pabelanger | looking at logs | 19:06 |
| corvus | either way, pinnig to 2.7 should probably be our first step? | 19:06 |
| pabelanger | https://review.opendev.org/#/c/676697/ | 19:06 |
| *** efried is now known as efried_afk | 19:06 | |
| pabelanger | we should do that, until openstack has the ability to test ansible 2.8 | 19:07 |
| pabelanger | ansible_python_interpreter: auto | 19:07 |
| pabelanger | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_824/696042/4/gate/openstack-tox-py36/8242e97/zuul-info/inventory.yaml | 19:07 |
| pabelanger | so, maybe that is the issue | 19:07 |
| corvus | right, but that changed a long time ago | 19:07 |
| pabelanger | Oh, I didn't realize that | 19:08 |
| corvus | we restarted zuul yesterday | 19:08 |
| corvus | the only changes between then and now are sorting autohold lists, adding 2.9, and defaulting to 2.8 | 19:08 |
| corvus | i will force-merg 697 | 19:09 |
| clarkb | is 697 a pin to 2.7? | 19:10 |
| openstackgerrit | Merged openstack/project-config master: Set default-ansible-version to 2.7 for all tenant (minus zuul) https://review.opendev.org/676697 | 19:10 |
| fungi | clarkb: on 3 out of 4 zuul tenants, yes | 19:10 |
| corvus | clarkb: ^ yes | 19:10 |
| corvus | now i wonder if we should restart the scheduler again | 19:11 |
| fungi | only the zuul tenant is keeping the version increase from default | 19:11 |
| corvus | just to re-run all those failing jobs | 19:11 |
| mnaser | https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html seems to exist in 2.8 docs but not in 2.7 | 19:11 |
| corvus | which means, likely, that the first change the zuul tenant needs to merge is a fix to the siblings thing | 19:11 |
| corvus | clarkb, fungi: should i do a restart to re-run currently failing jobs? | 19:12 |
| fungi | it's a tough call, but yeah i guess if you're up for it | 19:12 |
| corvus | will do | 19:12 |
| clarkb | corvus: ya I think that is probably the friendliest user thing | 19:12 |
| clarkb | tyty | 19:12 |
| fungi | i'm happy to help if there's anything for me to pitch in on there | 19:12 |
| corvus | i manually edited the file in 676697 as well | 19:14 |
| pabelanger | clarkb: corvus: fungi: mnaser: EmilienM: https://etherpad.openstack.org/p/CQgvHEzs3x should show reason for fails. Jobs bumped to ansible 2.8, which now has ansible_python_interpreter as auto (ansible 2.8) and likely first time openstack content is running under python3 for jobs | 19:14 |
| EmilienM | pabelanger: ohai | 19:15 |
| mnaser | i fixed a few py3 things the other time in tox_install_sibling_packages but i guess that wasnt all | 19:15 |
| fungi | pabelanger: thanks, that helps | 19:15 |
| EmilienM | pabelanger++ corvus++ thanks for the fixes :) | 19:15 |
| pabelanger | another option, is to force ubuntu-bionic in nodepool to python2, until we know python3 ansible works there | 19:16 |
| corvus | re-enqueueing | 19:18 |
| corvus | pabelanger: to be clear, tox-siblings isn't openstack, it's in zuul-jobs. so it seems whatever testing of that which happens in the zuul tenant is insufficient | 19:19 |
| pabelanger | +1 can look to see why it didn't break | 19:20 |
| corvus | we may not typically use any siblings | 19:20 |
| corvus | it looks like the 2.7 pin is good | 19:21 |
| corvus | i've seen at least one tox job pass the siblings point | 19:21 |
| EmilienM | can we recheck already or should we wait? | 19:22 |
| corvus | EmilienM: the re-enqueue is still in progress, so maybe wait just a bit? unless you want to jump the queue :) | 19:23 |
| EmilienM | no no, we'll wait | 19:23 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Update tox-docs to ansible 2.9 job https://review.opendev.org/698316 | 19:30 |
| TheJulia | is zuul's web interface happy? | 19:33 |
| fungi | TheJulia: it ought to be now | 19:34 |
| johnsom | Yeah, down again | 19:34 |
| johnsom | Stuck "Fetching info..." for me | 19:34 |
| clarkb | possible that zuul-web needed to be restarted after the renqueue | 19:34 |
| TheJulia | likewise | 19:34 |
| fungi | oh, maybe zuul-web needed a restart following the | 19:34 |
| fungi | yeah | 19:34 |
| TheJulia | It was working like 10 minutes ago | 19:34 |
| TheJulia | maybe... 15, I'm loosing track of time due to a conference call | 19:34 |
| corvus | no it's just slow | 19:35 |
| johnsom | Looks like all the jobs restarted again too | 19:35 |
| TheJulia | there it goes! | 19:35 |
| corvus | it managed to win the opportunity to do a full reconfig during a re-enqueue | 19:35 |
| corvus | EmilienM: re-enqueue is done | 19:35 |
| EmilienM | ok thanks | 19:36 |
| *** lbragstad_ is now known as lbragstad | 19:36 | |
| *** igordc has joined #openstack-infra | 19:39 | |
| *** gmann is now known as gmann_afk | 19:40 | |
| *** eharney has quit IRC | 19:55 | |
| *** jklare has quit IRC | 19:57 | |
| *** jklare has joined #openstack-infra | 20:01 | |
| clarkb | ianw: there is a dib question on openstack-discuss. I know we successfully install grub things on debuntu with our images and they are bootable. Do you know if this is a rhel/fedora problem? | 20:02 |
| clarkb | ianw: hrm looks like bionic images run grub-mkconfig not grub2-mkconfig though I expect they are actually the same system under the hood? | 20:05 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add upgrade note about ansible_python_interpreter https://review.opendev.org/698318 | 20:07 |
| *** hashar has joined #openstack-infra | 20:07 | |
| fungi | clarkb: probably. i can't imagine bionic has grub v1 even available for it | 20:07 |
| clarkb | ya | 20:08 |
| clarkb | I'm working on a response now to get more info | 20:08 |
| *** ralonsoh has quit IRC | 20:10 | |
| *** rcernin has joined #openstack-infra | 20:16 | |
| *** pcaruana has quit IRC | 20:16 | |
| openstackgerrit | Merged openstack/project-config master: Move security site to AFS publishing https://review.opendev.org/681584 | 20:21 |
| * clarkb finds lunch | 20:21 | |
| *** pkopec has quit IRC | 20:26 | |
| ianw | clarkb: hrmmm, i feel like there might be a bugzilla on this too | 20:29 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add upgrade note about ansible_python_interpreter https://review.opendev.org/698318 | 20:34 |
| *** Goneri has quit IRC | 20:38 | |
| ianw | i'm thinking of rhbz#1647915 | 20:40 |
| *** gfidente|afk has quit IRC | 20:51 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add additional info for executor.merge_jobs release note https://review.opendev.org/698324 | 20:59 |
| *** eharney has joined #openstack-infra | 21:02 | |
| *** jtomasek has quit IRC | 21:02 | |
| *** smcginnis has joined #openstack-infra | 21:05 | |
| *** jaosorior has joined #openstack-infra | 21:05 | |
| *** kjackal has quit IRC | 21:13 | |
| *** goldyfruit_ has quit IRC | 21:24 | |
| *** adriant has joined #openstack-infra | 21:26 | |
| *** mriedem has quit IRC | 21:29 | |
| *** goldyfruit_ has joined #openstack-infra | 21:30 | |
| *** hashar has quit IRC | 21:59 | |
| *** jaosorior has quit IRC | 22:01 | |
| ianw | "2019-12-10 20:52:24,434 DEBUG nodepool.NodeLauncher: [node: 0000000000] Serious errors were found while checking the disk drive for /." | 22:02 |
| ianw | this seems to be on trusty nodes for https://zuul.opendev.org/t/openstack/build/93729cbea2144cf68528cf8256e0645c/log/nodepool/nodepool-launcher.log ... | 22:02 |
| ianw | https://zuul.opendev.org/t/openstack/build/2fe5bae4a17245cfaf708015b4f3a0ed/log/nodepool/nodepool-launcher.log another instance | 22:03 |
| *** mriedem has joined #openstack-infra | 22:05 | |
| ianw | hrrm, it looks like the idea it is actaully a fsck error is a bit of red herring, and is probably related to mounting the root disk | 22:06 |
| *** ramishra has quit IRC | 22:06 | |
| *** slaweq has quit IRC | 22:10 | |
| clarkb | ianw: I got that locally too it had to do with the grub install iirc | 22:11 |
| clarkb | I basically had to manually rerun the grub install then it worked iirc | 22:11 |
| *** nicolasbock has quit IRC | 22:12 | |
| ianw | hrmmm, so this is switching the base builder to bionic from xenial ... the build logs look exactly the same | 22:14 |
| clarkb | ya I want to say my problem originated in a similar spot | 22:17 |
| clarkb | but was a while back | 22:17 |
| *** efried_afk is now known as efried | 22:25 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Fix python3 compat in tox siblings handling https://review.opendev.org/698335 | 22:44 |
| ianw | well i can recreate it at least :/ | 22:47 |
| *** apetrich has quit IRC | 22:54 | |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| *** gmann_afk is now known as gmann | 23:06 | |
| ianw | this is somehow on the right track -- https://askubuntu.com/questions/468846/permanent-grub-edit-from-ro-to-rw | 23:06 |
| ianw | switching to rw does let it boot | 23:06 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Fix python3 compat in tox siblings handling https://review.opendev.org/698335 | 23:07 |
| *** slaweq has joined #openstack-infra | 23:08 | |
| *** rlandy is now known as rlandy|bbl | 23:13 | |
| *** slaweq has quit IRC | 23:14 | |
| *** rh-jelabarre has quit IRC | 23:15 | |
| *** mriedem has quit IRC | 23:17 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Fix python3 compat in tox siblings handling https://review.opendev.org/698335 | 23:27 |
| *** goldyfruit_ has quit IRC | 23:30 | |
| *** Goneri has joined #openstack-infra | 23:35 | |
| *** Goneri has quit IRC | 23:40 | |
| *** dchen has joined #openstack-infra | 23:42 | |
| ianw | the grub.cfg between a bionic build and a xenial build looks the same :/ | 23:42 |
| *** rf0lc0 has quit IRC | 23:47 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: Add Google Cloud provider https://review.opendev.org/698342 | 23:47 |
| clarkb | ianw: could it be something to do with the filesystem layout or partition table? | 23:49 |
| clarkb | I guess if changing the kernel flag to rw fixes it then probably not? | 23:49 |
| *** rf0lc0 has joined #openstack-infra | 23:49 | |
| ianw | i wonder if it's initrd generation, and something leaking through from the host's /sys or /proc | 23:49 |
| clarkb | oh that would be fun | 23:49 |
| ianw | my desire to spend hours debugging this is limited :/ | 23:50 |
| ianw | can we ditch trusty yet? | 23:50 |
| clarkb | just about I think wiki and static are the main things | 23:50 |
| clarkb | static is on its way out and I'm not sure how much testing we even do for wiki on trusty? | 23:51 |
| clarkb | I think all of that testing side is pivotted to xenial now | 23:51 |
| *** harlowja has quit IRC | 23:54 | |
| ianw | clarkb: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml has an unfortunate number of hits for "-trusty" | 23:55 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch nodes to xenial https://review.opendev.org/698343 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!