| *** wolverineav has quit IRC | 00:03 | |
| *** wolverineav has joined #openstack-infra | 00:04 | |
| *** pahuang has quit IRC | 00:05 | |
| *** xinliang has quit IRC | 00:07 | |
| *** odyssey4me has quit IRC | 00:10 | |
| *** odyssey4me has joined #openstack-infra | 00:10 | |
| *** germs has joined #openstack-infra | 00:11 | |
| *** germs has quit IRC | 00:11 | |
| *** germs has joined #openstack-infra | 00:11 | |
| *** germs has quit IRC | 00:15 | |
| *** pahuang has joined #openstack-infra | 00:17 | |
| *** wolverineav has quit IRC | 00:18 | |
| *** xinliang has joined #openstack-infra | 00:19 | |
| *** wolverineav has joined #openstack-infra | 00:21 | |
| corvus | clarkb: it'll take a lot of test infrastructure to handle that case. is it worth it? | 00:26 |
|---|---|---|
| clarkb | corvus: mayhe not? it just seema like the main reason for having that ordering system? A substitute for unittest setup may be to just add a couple plugins to a devstack only job that have a dep order and that eill test that it works at an integration level | 00:28 |
| *** wolverineav has quit IRC | 00:29 | |
| *** wolverineav has joined #openstack-infra | 00:30 | |
| *** r-daneel has quit IRC | 00:31 | |
| corvus | clarkb: i can give it a shot. i'll just note we're establishing a very high bar of additional testing for a self-testing change. | 00:37 |
| clarkb | corvus: ya thats what I mean about just aving a devstack up job that includes some plugins that sue the featur | 00:38 |
| clarkb | none would use it now but we could add that once they do | 00:39 |
| clarkb | rather than write an explicit test for it | 00:39 |
| corvus | the change has been sitting for 4 months and i've forgotten who needed the feature. :( | 00:39 |
| clarkb | but then it would be self testing for consumers of the feature | 00:39 |
| corvus | i know that someone couldn't use the new devstack job without this. i can't remember who. | 00:39 |
| *** wolverineav has quit IRC | 00:40 | |
| corvus | anyway, i'll try adding a test tomorrow | 00:41 |
| clarkb | I want to say one group was magnum/zun and depending on the generic docker plugin for devstack | 00:41 |
| *** yamamoto has joined #openstack-infra | 00:43 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** claudiub has quit IRC | 00:50 | |
| *** hongbin has joined #openstack-infra | 00:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 00:52 | |
| *** wolverineav has joined #openstack-infra | 00:53 | |
| *** felipemonteiro__ has joined #openstack-infra | 00:54 | |
| *** wolverin_ has joined #openstack-infra | 00:55 | |
| *** felipemonteiro_ has quit IRC | 00:58 | |
| *** wolverineav has quit IRC | 00:59 | |
| *** diablo_rojo has quit IRC | 01:04 | |
| *** harlowja has quit IRC | 01:07 | |
| *** gcb has joined #openstack-infra | 01:08 | |
| *** andreww has quit IRC | 01:09 | |
| *** wolverin_ has quit IRC | 01:13 | |
| *** wolverineav has joined #openstack-infra | 01:13 | |
| *** wolverineav has quit IRC | 01:17 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:32 | |
| *** felipemonteiro__ has quit IRC | 01:32 | |
| *** pahuang has quit IRC | 01:33 | |
| *** pahuang has joined #openstack-infra | 01:34 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] testing 554684 https://review.openstack.org/554685 | 01:42 |
| *** yamamoto has joined #openstack-infra | 01:45 | |
| *** pahuang_ has joined #openstack-infra | 01:45 | |
| *** pahuang has quit IRC | 01:45 | |
| *** dingyichen has joined #openstack-infra | 01:47 | |
| *** yamamoto has quit IRC | 01:51 | |
| *** cshastri has joined #openstack-infra | 02:00 | |
| *** agopi has joined #openstack-infra | 02:03 | |
| *** mriedem has quit IRC | 02:06 | |
| *** pahuang_ has quit IRC | 02:13 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| *** myoung|afk is now known as myoung | 02:23 | |
| *** myoung is now known as myoung|afk | 02:27 | |
| *** pahuang_ has joined #openstack-infra | 02:30 | |
| ianw | OSError: [Errno 28] No space left on device: '/home/zuul/.ansible/tmp/ansible-local-12482nyahLb' ... i feel like i'm seeing this a lot | 02:33 |
| *** yamamoto has joined #openstack-infra | 02:37 | |
| *** psachin has joined #openstack-infra | 02:38 | |
| ianw | it's not on the executor is it? ... | 02:38 |
| fungi | cruft on executors? | 02:39 |
| clarkb | I think it might be | 02:39 |
| fungi | we can run out of space if we overrun them with load or leak cruft | 02:39 |
| *** zhurong has joined #openstack-infra | 02:40 | |
| ianw | hmm, two failures are | 02:41 |
| ianw | http://logs.openstack.org/05/554705/2/check/tripleo-ci-centos-7-containers-multinode/57bb822/ | 02:41 |
| ianw | http://logs.openstack.org/05/554705/2/check/tripleo-ci-centos-7-undercloud-containers/dfe438c | 02:41 |
| ianw | one was z02, the other ze06 i think ... neither seems that full | 02:42 |
| fungi | then probably the nodes | 02:42 |
| fungi | ran in the same provider? | 02:43 |
| openstackgerrit | YumengBao proposed openstack-infra/project-config master: Set up cyborg-specs repository https://review.openstack.org/553976 | 02:43 |
| clarkb | could be the remotes then? it is possible the jobs do run out if 80gb isnt enough | 02:43 |
| *** andreww has joined #openstack-infra | 02:44 | |
| ianw | ohhh, i wonder if https://review.openstack.org/#/c/553784/ is related. i just noticed that looking at my particular failure | 02:46 |
| *** salv-orl_ has joined #openstack-infra | 02:48 | |
| ianw | no, i can't see that the test has even run long enough that it could fill up the disk | 02:48 |
| *** andreas_s has joined #openstack-infra | 02:49 | |
| *** salv-orlando has quit IRC | 02:51 | |
| ianw | here's more out of disk errors ... http://logs.openstack.org/85/554685/3/check/dib-dsvm-functests-python2-centos-7-image/22ffbf6/job-output.txt.gz#_2018-03-21_01_55_38_503649 | 02:53 |
| *** rosmaita has quit IRC | 02:53 | |
| *** andreas_s has quit IRC | 02:53 | |
| *** yamamoto has quit IRC | 02:55 | |
| *** felipemonteiro_ has quit IRC | 02:55 | |
| *** wolverineav has joined #openstack-infra | 02:56 | |
| ianw | http://logs.openstack.org/85/554685/3/check/dib-dsvm-functests-python2-centos-7-image/22ffbf6/job-output.json.gz ... no configure-swap role? | 02:57 |
| *** pahuang_ has quit IRC | 02:57 | |
| clarkb | happening in different clouds too | 02:58 |
| clarkb | size_available: 3730972672 according to ansible | 02:59 |
| clarkb | http://logs.openstack.org/05/554705/2/check/tripleo-ci-centos-7-containers-multinode/57bb822/zuul-info/host-info.primary.yaml | 02:59 |
| clarkb | thats less than 4GB | 03:00 |
| clarkb | ianw: is it all centos 7? maybe something on the image isn't growfsing properly? | 03:00 |
| ianw | yeah ... i logged into a few and they seemed ok ,but maybe a bad image is rolling out | 03:00 |
| clarkb | and only 12GB or so total avaialble it thinks | 03:01 |
| clarkb | so ya I think it must be a growfs problem on boot | 03:01 |
| ianw | Mar 21 02:49:00 centos-7-rax-dfw-0003093035 growroot[732]: + growpart /dev/xvda 1 | 03:01 |
| ianw | Mar 21 02:49:00 centos-7-rax-dfw-0003093035 growroot[732]: WARN: sector size not found in sfdisk output, assuming 512 | 03:01 |
| ianw | Mar 21 02:49:00 centos-7-rax-dfw-0003093035 growroot[732]: FAILED: failed to get start and end for /dev/xvda1 in /dev/xvda | 03:02 |
| ianw | sigh ... | 03:02 |
| clarkb | we can rollbcak centos7 and pause image builds. At that point concern is it affecting the other images too | 03:03 |
| clarkb | ianw: oh did we switch to gpt ? I wonder if growpart doesn't understand gpt | 03:03 |
| ianw | we shouldn't have ... but new dib release is suspect i guess | 03:05 |
| ianw | Partition Table: msdos | 03:06 |
| clarkb | https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1087526 even if it was gpt looks like support was added ~5 years ago | 03:06 |
| openstack | Launchpad bug 1087526 in cloud-utils (Ubuntu) "need support for gpt partition tables" [Medium,Fix released] | 03:06 |
| clarkb | so would be surprised if that broke it | 03:06 |
| pabelanger | Hmm, seeing errors on fedora-27 | 03:06 |
| pabelanger | http://logs.openstack.org/95/554695/31/check/ansible-role-statsd-fedora-27/e3ae030/ara/result/5a8e4e24-6cfa-42c6-960b-6ed013f00910/ | 03:06 |
| pabelanger | any change with repos recently? | 03:07 |
| ianw | -bash-4.2# sfdisk --unit=S --dump /dev/vda | 03:07 |
| ianw | sfdisk: detected Disk Manager - unable to handle that | 03:07 |
| clarkb | pabelanger: that host has a proper 80GB at least | 03:07 |
| ianw | https://github.com/mmalecki/util-linux/blob/master/fdisks/sfdisk.c#L1524 wtf | 03:10 |
| ianw | /dev/vda1 * 2048 26664575 13331264 53 OnTrack DM6 Aux3 | 03:10 |
| *** pahuang_ has joined #openstack-infra | 03:10 | |
| openstackgerrit | megan guiney proposed openstack-infra/project-config master: initial config for getting-started-with-openstack project https://review.openstack.org/554768 | 03:11 |
| ianw | that's 0x51 | 03:11 |
| clarkb | line 1521 is excellent | 03:11 |
| clarkb | lets just pointer math at a magical offset | 03:11 |
| clarkb | ianw: so it is a DM6 like it is mad about? | 03:11 |
| ianw | http://paste.openstack.org/show/707073/ ... parted seems to be ok, but fdisk is reporting this weird type | 03:12 |
| ianw | but we didn't change the mbr portions of the code in dib | 03:13 |
| clarkb | its almost like they see two different partition tables | 03:15 |
| *** rlandy|bbl is now known as rlandy | 03:15 | |
| *** sree has joined #openstack-infra | 03:16 | |
| *** wolverineav has quit IRC | 03:16 | |
| ianw | https://review.openstack.org/#/c/533490/21/diskimage_builder/block_device/level1/partition.py | 03:16 |
| *** sree has quit IRC | 03:16 | |
| *** dave-mccowan has quit IRC | 03:16 | |
| ianw | line 59, what's 83 in hex | 03:16 |
| *** wolverineav has joined #openstack-infra | 03:17 | |
| *** sree has joined #openstack-infra | 03:17 | |
| ianw | 0x53 ... which is this "OnTrack DM6 Aux3" type | 03:17 |
| clarkb | 53 | 03:17 |
| clarkb | so its an encoding error in the int? | 03:18 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/irc-meetings master: Add Shengqin Feng as a chair of Zun meetings https://review.openstack.org/554769 | 03:18 |
| *** sree_ has joined #openstack-infra | 03:18 | |
| clarkb | oh it shows up in the diff too so ya that seems like the likely problem | 03:18 |
| *** sree_ is now known as Guest21977 | 03:19 | |
| ianw | yep ... and we would have gotten away with it too except for this meddling check for ancient weird disk managers in sfdisk | 03:19 |
| *** hongbin has quit IRC | 03:19 | |
| *** Guest21977 has quit IRC | 03:20 | |
| clarkb | I'll happily review a fix for that before bed | 03:21 |
| *** wolverineav has quit IRC | 03:21 | |
| *** sree has quit IRC | 03:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix default partition type https://review.openstack.org/554771 | 03:22 |
| *** sree has joined #openstack-infra | 03:22 | |
| ianw | clarkb: ^ i think that this stage, a dib point release with this is probably the way forward | 03:23 |
| clarkb | I guess integration testing didn't catch it because the image was big enough on disk to function for ssh | 03:23 |
| *** dhajare has joined #openstack-infra | 03:23 | |
| ianw | yeah, i'll add testing growroot to the todo list. have to grep the logs on the booted system or something | 03:24 |
| *** andreww has quit IRC | 03:24 | |
| clarkb | I've +2'd the change and I agree a bugfix release seems in order | 03:24 |
| ianw | thx! it's always something :/ | 03:25 |
| ianw | ohhhhh, but the dib gate is broken at the moment for another triple-o issue | 03:27 |
| ianw | and is also likely to hit this problem as it merges | 03:28 |
| clarkb | we can pause image building across the board then delete the new images | 03:28 |
| openstackgerrit | shangxdy proposed openstack-infra/project-config master: Fix ZUUL_USER_SSH_PUBLIC_KEY to support ssh key content https://review.openstack.org/467919 | 03:28 |
| clarkb | then unpause image building once a dib release is out | 03:28 |
| ianw | yeah, doing that now | 03:28 |
| clarkb | assuming our old image of the pair is still working (I think it is) | 03:28 |
| *** ramishra has joined #openstack-infra | 03:30 | |
| *** dhajare has quit IRC | 03:31 | |
| clarkb | also current nodepool no longer runs ready scripts so we may want to modify our tests to ssh explicitly if they don't aalready | 03:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Pause builds for dib 2.12.0 https://review.openstack.org/554785 | 03:33 |
| *** iyamahat_ has quit IRC | 03:33 | |
| *** dchen has joined #openstack-infra | 03:35 | |
| clarkb | and I guess nb03 is fine because it uses gpt | 03:35 |
| *** pahuang_ has quit IRC | 03:35 | |
| clarkb | I've approved ^ | 03:35 |
| ianw | thanks, once in i'll delete all the new images. that should get us to stability, tripleo can fix the dib job, and we can do a point release | 03:36 |
| *** ykarel has joined #openstack-infra | 03:36 | |
| clarkb | probably worth an email to the dev list to let people know why their jobs had a sad and also to avoid 2.12.0 and wait for 2.12.1 | 03:41 |
| clarkb | I think its under control now though so I'm going to find a late dinner and bed | 03:41 |
| ianw | clarkb: thanks as always, ttyl | 03:41 |
| *** jamesmcarthur has quit IRC | 03:44 | |
| *** zhurong has quit IRC | 03:44 | |
| *** yamamoto has joined #openstack-infra | 03:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pause builds for dib 2.12.0 https://review.openstack.org/554785 | 03:46 |
| *** pahuang_ has joined #openstack-infra | 03:52 | |
| *** links has joined #openstack-infra | 04:00 | |
| *** links has quit IRC | 04:00 | |
| *** yamamoto has quit IRC | 04:07 | |
| *** yamamoto has joined #openstack-infra | 04:08 | |
| *** udesale has joined #openstack-infra | 04:09 | |
| ianw | alright, puppet's rolled that config out hopefully | 04:14 |
| ianw | deleting less used like fedora first to make sure we're ok | 04:19 |
| *** eernst has joined #openstack-infra | 04:22 | |
| *** andreww has joined #openstack-infra | 04:27 | |
| *** yamamoto has quit IRC | 04:33 | |
| *** pgadiya has joined #openstack-infra | 04:35 | |
| *** harlowja has joined #openstack-infra | 04:39 | |
| *** rlandy has quit IRC | 04:39 | |
| *** eernst has quit IRC | 04:42 | |
| ianw | #status log all today's builds deleted, and all image builds on hold until dib 2.12.1 release. dib fix is https://review.openstack.org/554771 ; however requires a tripleo fix in https://review.openstack.org/554705 to first unblock dib gate | 04:44 |
| openstackstatus | ianw: finished logging | 04:44 |
| *** dhajare has joined #openstack-infra | 04:50 | |
| *** sree has quit IRC | 04:54 | |
| *** sree has joined #openstack-infra | 04:55 | |
| *** harlowja has quit IRC | 04:56 | |
| *** sree has quit IRC | 04:59 | |
| *** pahuang_ has quit IRC | 05:02 | |
| *** sree has joined #openstack-infra | 05:05 | |
| *** dchen has quit IRC | 05:05 | |
| *** dchen has joined #openstack-infra | 05:06 | |
| *** lpetrut has joined #openstack-infra | 05:06 | |
| *** dchen has quit IRC | 05:08 | |
| *** sree has quit IRC | 05:09 | |
| *** imacdonn has quit IRC | 05:14 | |
| *** imacdonn has joined #openstack-infra | 05:14 | |
| *** pahuang_ has joined #openstack-infra | 05:16 | |
| *** sree has joined #openstack-infra | 05:27 | |
| *** sree has quit IRC | 05:31 | |
| *** claudiub has joined #openstack-infra | 05:43 | |
| *** dsariel has joined #openstack-infra | 05:45 | |
| *** masuberu has quit IRC | 06:04 | |
| *** zhurong has joined #openstack-infra | 06:05 | |
| *** sree_ has joined #openstack-infra | 06:07 | |
| *** sree_ is now known as Guest83294 | 06:08 | |
| *** Guest83294 has quit IRC | 06:12 | |
| *** jcoufal has joined #openstack-infra | 06:12 | |
| *** sree_ has joined #openstack-infra | 06:12 | |
| *** sree_ is now known as Guest56076 | 06:13 | |
| *** germs has joined #openstack-infra | 06:13 | |
| *** germs has quit IRC | 06:13 | |
| *** germs has joined #openstack-infra | 06:13 | |
| *** ihrachys has quit IRC | 06:15 | |
| *** germs has quit IRC | 06:18 | |
| *** lpetrut has quit IRC | 06:19 | |
| *** jcoufal_ has joined #openstack-infra | 06:19 | |
| *** e0ne has joined #openstack-infra | 06:20 | |
| *** udesale has quit IRC | 06:21 | |
| *** jcoufal has quit IRC | 06:21 | |
| *** udesale has joined #openstack-infra | 06:21 | |
| *** jcoufal has joined #openstack-infra | 06:27 | |
| *** Guest56076 has quit IRC | 06:27 | |
| tobiash | AJaeger, mordred: I'm +2 on https://review.openstack.org/554297 but didn't hit +3 in case someone else wants/should look on this | 06:28 |
| *** yamamoto has joined #openstack-infra | 06:28 | |
| *** armaan has joined #openstack-infra | 06:28 | |
| *** jcoufal_ has quit IRC | 06:30 | |
| *** masber has joined #openstack-infra | 06:31 | |
| *** dbecker has quit IRC | 06:31 | |
| *** vaidy has quit IRC | 06:34 | |
| *** isviridov_away has quit IRC | 06:34 | |
| *** gus has quit IRC | 06:34 | |
| *** lpetrut has joined #openstack-infra | 06:35 | |
| *** StevenK has quit IRC | 06:35 | |
| *** sdake has quit IRC | 06:35 | |
| *** gus has joined #openstack-infra | 06:36 | |
| *** jbadiapa has joined #openstack-infra | 06:36 | |
| *** StevenK has joined #openstack-infra | 06:36 | |
| *** sdake has joined #openstack-infra | 06:37 | |
| *** sdake has quit IRC | 06:37 | |
| *** sdake has joined #openstack-infra | 06:37 | |
| *** isviridov_away has joined #openstack-infra | 06:40 | |
| *** e0ne has quit IRC | 06:40 | |
| *** jamesmcarthur has joined #openstack-infra | 06:44 | |
| *** pcichy has joined #openstack-infra | 06:44 | |
| *** dbecker has joined #openstack-infra | 06:46 | |
| *** vaidy has joined #openstack-infra | 06:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Pass NODEPOOL_ZK_HOST variable for py35 test https://review.openstack.org/554810 | 06:46 |
| *** jamesmcarthur has quit IRC | 06:48 | |
| *** agopi has quit IRC | 06:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow external zookeeper in tox py35 runs https://review.openstack.org/554810 | 06:52 |
| *** gongysh has joined #openstack-infra | 06:55 | |
| *** lpetrut has quit IRC | 06:55 | |
| *** logan- has quit IRC | 06:58 | |
| *** logan- has joined #openstack-infra | 06:58 | |
| AJaeger | tobiash: thanks. I'm fine with it, just doing one sanity check... | 07:04 |
| *** kiennt26 has joined #openstack-infra | 07:04 | |
| *** jaosorior has quit IRC | 07:05 | |
| *** masber has quit IRC | 07:05 | |
| *** masber has joined #openstack-infra | 07:06 | |
| *** salv-orl_ has quit IRC | 07:12 | |
| *** sree_ has joined #openstack-infra | 07:13 | |
| *** sree_ is now known as Guest70027 | 07:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Switch to stestr https://review.openstack.org/536882 | 07:14 |
| *** alexchadin has joined #openstack-infra | 07:14 | |
| *** salv-orlando has joined #openstack-infra | 07:16 | |
| *** Guest70027 has quit IRC | 07:18 | |
| *** rcernin has quit IRC | 07:21 | |
| *** andreas_s has joined #openstack-infra | 07:26 | |
| *** yamamoto has quit IRC | 07:32 | |
| *** hashar has joined #openstack-infra | 07:33 | |
| *** diablo_rojo has joined #openstack-infra | 07:34 | |
| *** kjackal has joined #openstack-infra | 07:40 | |
| *** jaosorior has joined #openstack-infra | 07:44 | |
| *** ralonsoh has joined #openstack-infra | 07:46 | |
| *** priteau has joined #openstack-infra | 07:52 | |
| *** yamahata has joined #openstack-infra | 07:55 | |
| *** yamamoto has joined #openstack-infra | 08:00 | |
| *** yamamoto has quit IRC | 08:03 | |
| *** HeOS has joined #openstack-infra | 08:07 | |
| *** priteau has quit IRC | 08:07 | |
| *** danpawlik has joined #openstack-infra | 08:10 | |
| *** florianf has joined #openstack-infra | 08:13 | |
| *** yamamoto has joined #openstack-infra | 08:13 | |
| AJaeger | garyk, boden, is https://review.openstack.org/#/c/554292/ and https://review.openstack.org/#/c/554245 working fine? IT looks to me fine - and if you agree, I'll merge https://review.openstack.org/554297 and you can merge 554292 | 08:14 |
| *** yamamoto has quit IRC | 08:16 | |
| *** yamamoto has joined #openstack-infra | 08:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix zuul-web port in zuul-from-scratch doc https://review.openstack.org/554829 | 08:20 |
| *** dingyichen has quit IRC | 08:24 | |
| *** krenczewski has quit IRC | 08:28 | |
| *** tesseract has joined #openstack-infra | 08:31 | |
| *** krenczewski has joined #openstack-infra | 08:35 | |
| *** amoralej|off is now known as amoralej | 08:36 | |
| *** masber has quit IRC | 08:40 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** tosky has joined #openstack-infra | 08:43 | |
| *** lpetrut has joined #openstack-infra | 08:45 | |
| *** lpetrut has quit IRC | 08:48 | |
| *** lpetrut_ has joined #openstack-infra | 08:48 | |
| *** claudiub has quit IRC | 08:50 | |
| *** tesseract has quit IRC | 08:51 | |
| *** tesseract has joined #openstack-infra | 08:52 | |
| *** tesseract has quit IRC | 08:54 | |
| *** tesseract has joined #openstack-infra | 08:57 | |
| *** lucas-afk is now known as lucasagomes | 08:59 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** arxcruz|off is now known as arxcruz | 09:04 | |
| *** zhurong has quit IRC | 09:04 | |
| *** zhurong has joined #openstack-infra | 09:14 | |
| *** electrofelix has joined #openstack-infra | 09:14 | |
| *** duobei has joined #openstack-infra | 09:17 | |
| *** duobei has left #openstack-infra | 09:17 | |
| *** masber has joined #openstack-infra | 09:20 | |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 09:20 |
| *** jesusaur has quit IRC | 09:21 | |
| *** jesusaur has joined #openstack-infra | 09:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add trigger driver https://review.openstack.org/554839 | 09:25 |
| *** gongysh has quit IRC | 09:30 | |
| *** efoley has joined #openstack-infra | 09:33 | |
| *** derekh has joined #openstack-infra | 09:34 | |
| *** yamamoto has quit IRC | 09:42 | |
| *** yamamoto has joined #openstack-infra | 09:43 | |
| *** pgaxatte has joined #openstack-infra | 09:45 | |
| pgaxatte | hello | 09:45 |
| *** markmcd has left #openstack-infra | 09:46 | |
| pgaxatte | coreycb: I noticed a problem with mistral's pike release on ubuntu cloud archive | 09:47 |
| *** yamamoto has quit IRC | 09:48 | |
| *** yamamoto has joined #openstack-infra | 09:48 | |
| *** yamamoto has quit IRC | 09:48 | |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 09:49 |
| *** panda|off is now known as panda | 09:52 | |
| *** pbourke has joined #openstack-infra | 09:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add trigger driver https://review.openstack.org/554839 | 09:54 |
| *** gfidente has joined #openstack-infra | 09:54 | |
| *** gfidente has joined #openstack-infra | 09:54 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add reenqueue button https://review.openstack.org/554856 | 09:54 |
| *** claudiub has joined #openstack-infra | 09:55 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add Shengqin Feng as a chair of Zun meetings https://review.openstack.org/554769 | 09:56 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove WOS Mentoring Meeting https://review.openstack.org/554723 | 09:56 |
| *** dizquierdo has joined #openstack-infra | 09:57 | |
| *** armaan has quit IRC | 10:09 | |
| *** armaan has joined #openstack-infra | 10:10 | |
| dmellado | Hi, I've started seeing a few POST_FAILURES again | 10:10 |
| dmellado | is there something off in the infra? | 10:10 |
| dmellado | AJaeger: rcarrillocruz ? | 10:10 |
| *** priteau has joined #openstack-infra | 10:11 | |
| *** dhajare has quit IRC | 10:16 | |
| stephenfin | AJaeger: Could you point me to the job definition that decides if we run the legacy docs build (setup.py build_sphinx) or not? I can't find it | 10:17 |
| *** dhajare has joined #openstack-infra | 10:17 | |
| stephenfin | It seems a few of the merged 'topic:updated-pti' patches have inadvertently broken local docs builds and they weren't picked up in the gate because of that job's magic | 10:17 |
| stephenfin | smcginnis: ^ | 10:18 |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 10:23 |
| AJaeger | stephenfin: it's in zuul-jobs, let me give you a link... | 10:25 |
| AJaeger | stephenfin: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/tasks/main.yaml | 10:26 |
| AJaeger | dmellado: for CentOS images? There was an email to the dev mailing list | 10:26 |
| dmellado | AJaeger: this I've seen with Ubuntu image | 10:27 |
| *** alexchadin has quit IRC | 10:27 | |
| dmellado | i.e. https://review.openstack.org/#/c/548309/ | 10:27 |
| *** alexchadin has joined #openstack-infra | 10:28 | |
| *** alexchadin has quit IRC | 10:30 | |
| AJaeger | dmellado: there was a timeout at http://logs.openstack.org/09/548309/8/check/kuryr-kubernetes-tempest-lbaasv2/1b908b5/job-output.txt.gz#_2018-03-21_09_37_04_786322 - not sure why. | 10:33 |
| dmellado | AJaeger: I'll recheck and see | 10:34 |
| *** alexchadin has joined #openstack-infra | 10:35 | |
| *** boden has joined #openstack-infra | 10:39 | |
| *** dizquierdo has quit IRC | 10:39 | |
| *** alexchadin has quit IRC | 10:46 | |
| *** yamamoto has joined #openstack-infra | 10:48 | |
| stephenfin | AJaeger: Ta. Email sent | 10:49 |
| *** zoli is now known as zoli|lunch | 10:49 | |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| *** yamahata has quit IRC | 10:52 | |
| boden | AJaeger hi, FYI left you a response here https://review.openstack.org/#/c/554245/2 seems we still may have some issues | 10:53 |
| kashyap | Hey folks, just a quick thank you note: paste.openstack.org is super fast and reliable. | 10:53 |
| kashyap | Nice work there! | 10:54 |
| *** yamamoto has quit IRC | 10:54 | |
| *** dizquierdo has joined #openstack-infra | 10:55 | |
| *** zhurong has quit IRC | 10:55 | |
| AJaeger | boden: indeed - one step forward, one more problem found ;) | 10:56 |
| *** yamamoto has joined #openstack-infra | 10:56 | |
| AJaeger | boden: best discuss with mordred how to handle the vmware-api tests. He should be around soon | 10:56 |
| boden | AJaeger: ack, I just need a pointer in the right direction… then I can go off and break more stuff :) | 10:56 |
| AJaeger | hope you and mordred find a solution. | 10:58 |
| *** e0ne has joined #openstack-infra | 10:59 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| *** udesale_ has joined #openstack-infra | 11:04 | |
| *** numans is now known as numans_afk | 11:05 | |
| *** yamamoto has quit IRC | 11:06 | |
| *** udesale has quit IRC | 11:07 | |
| *** sshnaidm|sick is now known as sshnaidm | 11:08 | |
| *** dhajare has quit IRC | 11:08 | |
| *** gyankum has joined #openstack-infra | 11:10 | |
| *** udesale_ has quit IRC | 11:13 | |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 11:14 |
| *** yamamoto has joined #openstack-infra | 11:16 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** alexchadin has joined #openstack-infra | 11:16 | |
| *** katkapilatova has joined #openstack-infra | 11:21 | |
| *** pcichy has quit IRC | 11:27 | |
| *** adarazs is now known as adarazs_lunch | 11:27 | |
| *** dhajare has joined #openstack-infra | 11:27 | |
| *** cshastri has quit IRC | 11:27 | |
| *** snapiri has quit IRC | 11:28 | |
| *** snapiri has joined #openstack-infra | 11:28 | |
| *** numans_afk is now known as numans | 11:29 | |
| coreycb | pgaxatte: hi, what's happening? we should move to #ubuntu-server for package issues. | 11:32 |
| *** claudiub has quit IRC | 11:35 | |
| *** claudiub has joined #openstack-infra | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| *** jpena is now known as jpena|off | 11:39 | |
| *** jpena|off is now known as jpena | 11:40 | |
| ssbarnea | any gerritbot expert around? I observed that it fails to spot stalled connections and do a reconnect. | 11:40 |
| *** e0ne has quit IRC | 11:42 | |
| ssbarnea | https://storyboard.openstack.org/#!/story/2001714 | 11:46 |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** dhajare has quit IRC | 11:54 | |
| *** dhajare has joined #openstack-infra | 11:54 | |
| *** dsariel has quit IRC | 11:58 | |
| *** jlabarre has quit IRC | 11:58 | |
| *** tpsilva has joined #openstack-infra | 11:59 | |
| *** e0ne has joined #openstack-infra | 12:00 | |
| *** adarazs_lunch is now known as adarazs | 12:01 | |
| *** e0ne has quit IRC | 12:02 | |
| *** yamamoto has joined #openstack-infra | 12:03 | |
| *** odyssey4me has quit IRC | 12:03 | |
| *** odyssey4me has joined #openstack-infra | 12:03 | |
| *** rfolco has joined #openstack-infra | 12:05 | |
| *** dprince has joined #openstack-infra | 12:06 | |
| *** rosmaita has joined #openstack-infra | 12:08 | |
| *** yamamoto has quit IRC | 12:08 | |
| *** zoli|lunch is now known as zoli | 12:11 | |
| *** dsariel has joined #openstack-infra | 12:13 | |
| *** e0ne has joined #openstack-infra | 12:15 | |
| *** lucasagomes is now known as lucas-hungry | 12:16 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** jpena is now known as jpena|lunch | 12:20 | |
| *** dsariel has quit IRC | 12:21 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** efried has quit IRC | 12:23 | |
| *** sambetts|afk is now known as sambetts | 12:24 | |
| *** efried has joined #openstack-infra | 12:24 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: WIP Retry merge jobs https://review.openstack.org/554890 | 12:27 |
| *** dprince has quit IRC | 12:29 | |
| *** panda is now known as panda|lunch | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** trown|outtypewww is now known as trown|ruck | 12:34 | |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** yamamoto has quit IRC | 12:38 | |
| *** edmondsw has joined #openstack-infra | 12:40 | |
| *** VW has joined #openstack-infra | 12:41 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: WIP: Rewrite launch-node.py in Ansible playbooks/roles https://review.openstack.org/554894 | 12:42 |
| *** gcb has quit IRC | 12:42 | |
| dmsimard | infra-root: ^ I couldn't sleep last night so I did something | 12:42 |
| *** dhajare has quit IRC | 12:42 | |
| *** pgadiya has quit IRC | 12:44 | |
| *** panda|lunch is now known as panda | 12:45 | |
| *** yamamoto has joined #openstack-infra | 12:48 | |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| *** florianf_ has joined #openstack-infra | 12:51 | |
| *** yamamoto has quit IRC | 12:53 | |
| *** rosmaita has quit IRC | 12:53 | |
| *** jamesmcarthur has quit IRC | 12:53 | |
| *** florianf has quit IRC | 12:53 | |
| *** dizquierdo has quit IRC | 12:59 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:01 | |
| *** eharney has joined #openstack-infra | 13:01 | |
| *** felipemonteiro__ has joined #openstack-infra | 13:02 | |
| *** adarazs is now known as adarazs_afk | 13:02 | |
| *** kgiusti has joined #openstack-infra | 13:03 | |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| *** germs has joined #openstack-infra | 13:04 | |
| *** germs has quit IRC | 13:04 | |
| *** germs has joined #openstack-infra | 13:04 | |
| *** dprince has joined #openstack-infra | 13:05 | |
| *** felipemonteiro_ has quit IRC | 13:06 | |
| frickler | infra-root: google found out for me that we also publish the sources for zuul docs, doesn't look like it should be that way: https://docs.openstack.org/infra/zuul/_sources/admin/client.rst.txt | 13:06 |
| dmsimard | frickler: errr that's inside _sources | 13:07 |
| dmsimard | how did it even find that link ? | 13:07 |
| dmsimard | do we need to add a robots.txt in there ? | 13:07 |
| *** yamamoto has quit IRC | 13:08 | |
| frickler | dmsimard: not sure, put I also don't think the sources should even exist at that place on docs.o.o ? | 13:08 |
| frickler | dmsimard: other question: I tried to autohold a node for debugging, how can I find out if there is indeed a held node for that? | 13:09 |
| dmsimard | frickler: what I've been doing is doing a nodepool list --detail |grep hold (or held?) on one of the nodepool launchers | 13:09 |
| dmsimard | there might be a better way but I feel there's a gap between zuul and nodepool for that | 13:10 |
| AJaeger | frickler: sphinx publishes them... | 13:11 |
| AJaeger | frickler: that's a sphinx variable to set | 13:11 |
| *** udesale has joined #openstack-infra | 13:12 | |
| frickler | AJaeger: so is that published intentionally? it is the 4th hit when searching "openstack zuul autohold" btw | 13:14 |
| AJaeger | frickler: html_copy_source and html_show_sourcelink handle this... | 13:14 |
| frickler | dmsimard: that command seems to work, thx. waiting for the proper node to show up now | 13:15 |
| frickler | mordred: fyi, there is a held node attributed to you 19 days old. please check if you still need that one | 13:16 |
| *** myoung|afk is now known as myoung | 13:18 | |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| *** snapiri has quit IRC | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:20 | |
| *** yamamoto has quit IRC | 13:22 | |
| *** alexchad_ has joined #openstack-infra | 13:24 | |
| *** jpena|lunch is now known as jpena | 13:25 | |
| *** alexchadin has quit IRC | 13:25 | |
| *** amoralej is now known as amoralej|lunch | 13:27 | |
| kashyap | Any way to reduce the time for `git-review`? This is just abysmal :-( | 13:28 |
| kashyap | $> time git review | 13:28 |
| kashyap | remote: Processing changes: updated: 1, refs: 1, done | 13:28 |
| kashyap | [...] | 13:28 |
| kashyap | remote: https://review.openstack.org/534384 libvirt: Allow to specify granular CPU feature flags | 13:28 |
| kashyap | [...] | 13:28 |
| kashyap | real 7m1.107s | 13:28 |
| kashyap | user 0m1.102s | 13:28 |
| kashyap | sys 0m0.816s | 13:28 |
| *** lucas-hungry is now known as lucasagomes | 13:29 | |
| *** eharney has quit IRC | 13:30 | |
| frickler | kashyap: please use paste.openstack.org for multiline pastes. also gerrit seems to be a bit slow for me at times, too, that might be related | 13:33 |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| kashyap | frickler: Yeah, I normally do use paste.o.o extensively. Posted here as it was under 8 lines, saving people to open yet another URL | 13:33 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add trigger driver https://review.openstack.org/554839 | 13:33 |
| kashyap | That's I even trimmed the output before I pasted here. | 13:34 |
| kashyap | s/That's/That's why/ | 13:34 |
| *** jlabarre has joined #openstack-infra | 13:34 | |
| *** adarazs_afk is now known as adarazs | 13:37 | |
| *** yamamoto has quit IRC | 13:38 | |
| frickler | kashyap: o.k., I agree that it is a borderline case ;) regarding the timing, do you see that every time or was it a one off? how long does a "git review -d" take for you in comparison? | 13:38 |
| kashyap | frickler: (No worries; I myself correct people on other channels to paste.) Yes, I saw that last night too :-( About 6 minutes | 13:39 |
| kashyap | Made me put a fork in my eye & turn it until everything came out | 13:39 |
| * kashyap tries `git review -d` | 13:41 | |
| frickler | kashyap: I agree that this is unreasonably long. could you also check the timings for "for i in 4 6;do time curl -I -$i https://review.openstack.org; done" please? | 13:42 |
| kashyap | Will try; first letting the `git review -d` run finish | 13:42 |
| kashyap | frickler: BTW, my network speed is ... let's say "less than stellar" | 13:43 |
| kashyap | Download: 3.72 Mbits/s | 13:43 |
| kashyap | Upload: 3.85 Mbits/s | 13:43 |
| *** dklyle has joined #openstack-infra | 13:44 | |
| *** david-lyle has quit IRC | 13:44 | |
| frickler | kashyap: that still sounds pretty reasonable IMO. from your timing I feared you would come up with some kbit/s only | 13:45 |
| kashyap | frickler: `git review -d 534384` is still running | 13:45 |
| * kashyap goes to spend some time on other mailing lists based projects where I use a `git-send-email` + 'mutt' to apply 100s of patches instantenously | 13:46 | |
| kashyap | (To regain some sanity) | 13:46 |
| frickler | kashyap: good or rather not good. but at least confirms that the other direction is affected, too | 13:46 |
| kashyap | frickler: http://paste.openstack.org/show/707550/ | 13:47 |
| kashyap | (The `curl` thing you asked for.) | 13:47 |
| AJaeger | frickler: for the doc issue from earlier, see also http://sphinx.readthedocs.io/en/master/config.html#confval-html_copy_source - we need the sources for searching in the docs and proper display. | 13:47 |
| AJaeger | frickler: we could add to the global robots.txt a line to disable _sources | 13:48 |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| AJaeger | frickler: want to send a patch for http://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/static/robots.txt ? | 13:48 |
| *** psachin has quit IRC | 13:48 | |
| frickler | AJaeger: ah, didn't know that the sources are used for that. I'll take a look at that, thx | 13:49 |
| frickler | kashyap: o.k., that looks pretty normal to me, so I'm out of clue for now, sorry. maybe some other infra-root has more ideas | 13:50 |
| *** ihrachys has joined #openstack-infra | 13:51 | |
| kashyap | frickler: No problem. Meanwhile, your `git-review -d` is just finished: | 13:51 |
| kashyap | $> time git review -d 534384 | 13:51 |
| kashyap | [...] | 13:51 |
| kashyap | real 6m30.754s | 13:51 |
| kashyap | user 0m0.729s | 13:51 |
| kashyap | sys 0m0.528s | 13:51 |
| dmsimard | kashyap: what repository ? | 13:53 |
| *** yamamoto has quit IRC | 13:53 | |
| kashyap | Nova | 13:53 |
| frickler | kashyap: just to confirm, this is a new situation for you in the last couple of days? or has it always been that bad for you? | 13:53 |
| kashyap | frickler: It's new in the past few days. In the past it was under 10 seconds | 13:53 |
| dmsimard | kashyap: is that a fresh repository clone ? or something you've been carrying for a while ? | 13:54 |
| kashyap | dmsimard: The latter | 13:54 |
| kashyap | For a while. It has multiple remotes | 13:54 |
| *** alexchad_ is now known as alexchadin | 13:54 | |
| kashyap | $> du -sh .git | 13:54 |
| kashyap | 325M .git | 13:54 |
| dmsimard | kashyap: can you try and reproduce from a fresh git.o.o clone ? It would probably be a good hint to point us in the right direction. | 13:54 |
| kashyap | dmsimard: Will this test be okay: | 13:54 |
| kashyap | (1) Do a fresh clone | 13:55 |
| dmsimard | Nova is definitely one of those bigger repositories too | 13:55 |
| kashyap | (2) Time `git review -d` | 13:55 |
| kashyap | ? | 13:55 |
| dmsimard | kashyap: sure -- and if you could compare with the pure git commands it would be helpful too. Like: git fetch https://git.openstack.org/openstack/nova refs/changes/84/534384/7 && git checkout FETCH_HEAD | 13:56 |
| dmsimard | (please use paste.o.o :D) | 13:56 |
| AJaeger | kashyap: "real 0m38.751" for me | 13:57 |
| kashyap | dmsimard: As I said earlier, if it's under 6 lines, I normally just paste here; it's less work for everyone. And ~6 lines is completely fine on IRC, IMHO | 13:57 |
| kashyap | But yeah, anything above, I do use pastebin (extensively) | 13:57 |
| *** zhipeng has joined #openstack-infra | 13:57 | |
| dmsimard | kashyap: yeah but these are going to be a couple times ~6 lines, let's capture everything in the same paste :p | 13:58 |
| kashyap | dmsimard: Of course, I'll use paste for such entries. Really, I myself correct pepole on other channels I help out on. So no worries. | 13:59 |
| *** bobh has joined #openstack-infra | 14:03 | |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| kashyap | dmsimard: I'm in a bit of a rush; I'll get it sometime tonight | 14:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 14:04 |
| dmsimard | kashyap: okay -- my general train of thought was to try and isolate if the issue was coming from gerrit, from git-review, or from something else (git pack/garbage collection/etc?) | 14:04 |
| dmsimard | kashyap: git review is more or less a wrapper so just the test between "git-review -d" and pure git checkout is a good test | 14:05 |
| *** dizquierdo has joined #openstack-infra | 14:06 | |
| kashyap | Yeah | 14:06 |
| kashyap | Noted; that's a good tip to remember | 14:06 |
| *** yamamoto has quit IRC | 14:07 | |
| *** hongbin has joined #openstack-infra | 14:08 | |
| frickler | AJaeger: do you know whether /_sources/ in robots.txt would also match subdirectories? or does it need a full path like /infra/zuul/_sources/ ? if we need to the latter for all the projects we publish, I guess we would need to write a tool to autogenerate it | 14:08 |
| mordred | frickler: I do not still need the held node - lemme go delete it | 14:09 |
| *** dsariel has joined #openstack-infra | 14:11 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow external zookeeper in tox py35 runs https://review.openstack.org/554810 | 14:13 |
| *** yamamoto has joined #openstack-infra | 14:18 | |
| *** myoung is now known as myoung|rover | 14:19 | |
| *** esberglu has joined #openstack-infra | 14:19 | |
| *** myoung|rover is now known as myoung|rover|mtg | 14:20 | |
| *** yamamoto has quit IRC | 14:23 | |
| *** cshastri has joined #openstack-infra | 14:24 | |
| *** derekh has quit IRC | 14:25 | |
| *** r-daneel has joined #openstack-infra | 14:25 | |
| *** ykarel is now known as ykarel|away | 14:25 | |
| *** derekh has joined #openstack-infra | 14:27 | |
| AJaeger | frickler: it starts from root, so /_sources/ will not help, you need /infra/zuul/_sources/ | 14:28 |
| AJaeger | mordred: did you see boden's comment earlier on how to setup vmware-api so that it now checks out from git? | 14:29 |
| *** ykarel|away has quit IRC | 14:30 | |
| *** gouthamr has joined #openstack-infra | 14:32 | |
| *** yamamoto has joined #openstack-infra | 14:33 | |
| mordred | AJaeger: I did not - looking | 14:37 |
| pabelanger | morning | 14:37 |
| pabelanger | I'm going to try launching review-dev01.o.o again | 14:37 |
| *** yamamoto has quit IRC | 14:38 | |
| *** hashar is now known as hasharAway | 14:39 | |
| *** amoralej|lunch is now known as amoralej | 14:40 | |
| smcginnis | pabelanger: 2.14 testing? | 14:40 |
| AJaeger | mordred: so, https://review.openstack.org/#/c/554292/ is fine on the OpenStack CI side - but now vmware-api installs from pypi instead of from git. They need some guideance/tools - and developers as well - on how to test locally with those packages from git. | 14:41 |
| pabelanger | smcginnis: first upgrade to xenial, but yah eventually gerrit testing | 14:41 |
| AJaeger | mordred: want to +A https://review.openstack.org/554297 ? I think it's good to go and has 2 +2s | 14:41 |
| smcginnis | pabelanger: Nice! | 14:42 |
| mordred | AJaeger: gotcha. so - local testing of siblings things is definitely high on the todo list - there isn't a GREAT story for it this instant | 14:42 |
| dansmith | I'm not the only one seeing post_failures again recently, right? | 14:43 |
| mordred | AJaeger, boden: I've got a patch up to add a helper to pbr - although it might be better to add such a helper as a separate repo | 14:43 |
| dansmith | looks like unreachable workers: http://logs.openstack.org/90/547990/6/check/legacy-tempest-dsvm-multinode-live-migration/28c2873/job-output.txt.gz#_2018-03-21_14_38_29_837702 | 14:44 |
| dansmith | I saw at least one yesterday too | 14:44 |
| mordred | AJaeger: ok. pulling the trigger- hold on to your hats :) | 14:44 |
| AJaeger | ;) | 14:44 |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| boden | AJaeger mordred I’m still a little confused on how things work… how can we still install our dependenat projects like neutron, sfc, etc. from git when running tox locally (outside of the gate)? | 14:46 |
| *** felipemonteiro__ has quit IRC | 14:47 | |
| frickler | dansmith: saw a few of those, but not enough yet to establish a pattern. if you have multiple occurrences, you may want to check zuul-info/inventory.yaml whether they all fail on a particular provider | 14:47 |
| *** dave-mccowan has joined #openstack-infra | 14:47 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:47 | |
| dansmith | frickler: okay the one from just now is rax, FYI | 14:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Revert "Switch to stestr" https://review.openstack.org/554943 | 14:50 |
| *** alexchadin has quit IRC | 14:50 | |
| *** germs has quit IRC | 14:51 | |
| *** gyankum has quit IRC | 14:51 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:52 | |
| *** germs has joined #openstack-infra | 14:52 | |
| AJaeger | boden, mordred , did you see http://lists.openstack.org/pipermail/openstack-dev/2018-March/128328.html ? | 14:54 |
| *** ykarel|away has joined #openstack-infra | 14:55 | |
| *** ykarel|away is now known as ykarel | 14:55 | |
| *** felipemonteiro__ has quit IRC | 14:55 | |
| mordred | AJaeger: I didn't - but yeah, that's basically a good summary of the current state - we need it in places that aren't neutron/horizon related too | 14:56 |
| boden | AJaeger I missed that detail.. I’ll have to spend some time munking with it to see if I can get it to work | 14:57 |
| mordred | AJaeger, boden: we also have a similar thing in python-openstackclient and made tox envs like this: http://git.openstack.org/cgit/openstack/python-openstackclient/tree/tox.ini#n58 | 14:57 |
| mordred | that's not ideal either though - which is why I started in on pbr siblings | 14:58 |
| pabelanger | hmm, fedora-27 nodes still having dnf issues | 14:58 |
| boden | mordred AJaeger ok thanks.. I’ll parse that info… is this doc’d anywhere? | 14:58 |
| pabelanger | going to hold one and see why that is | 14:58 |
| pabelanger | doh | 14:59 |
| pabelanger | mordred: dmsimard: could I get a +3 on: https://review.openstack.org/554624 | 14:59 |
| pabelanger | needed so we can 2 step review-dev01.o.o online | 14:59 |
| *** agopi has joined #openstack-infra | 15:00 | |
| *** yamahata has joined #openstack-infra | 15:00 | |
| mordred | pabelanger: wfm | 15:00 |
| *** iyamahat has joined #openstack-infra | 15:00 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Uninstall and reinstall siblings one at a time https://review.openstack.org/554297 | 15:03 |
| *** felipemonteiro__ has joined #openstack-infra | 15:07 | |
| clarkb | kashyap: considering it was happening to github as well the other day I am guessing it is a problem local to you. You may want to tcpdump a fetch to see what is going on (and possibly fetch via http so that you can read a bit more of what is going on than if it were https) | 15:07 |
| *** cshastri has quit IRC | 15:08 | |
| kashyap | clarkb: Yeah, will investigate. Under some duress to finish something else, before I run to my Dutch class in a few minutes | 15:08 |
| kashyap | First thing in the morning. | 15:08 |
| *** felipemonteiro_ has quit IRC | 15:11 | |
| *** kien-ha has joined #openstack-infra | 15:12 | |
| *** electrofelix has quit IRC | 15:12 | |
| AJaeger | boden: saw your comment - can you do a change on top of mine first and iterate on that? Once that works, we can disucss merging them in one - or approve both. | 15:16 |
| AJaeger | I'd like to keep a baseline ;) | 15:16 |
| *** eernst has joined #openstack-infra | 15:22 | |
| *** jamesdenton has quit IRC | 15:24 | |
| *** zhipeng has quit IRC | 15:26 | |
| openstackgerrit | Merged openstack-infra/zuul master: Revert "Switch to stestr" https://review.openstack.org/554943 | 15:27 |
| *** VW_ has joined #openstack-infra | 15:28 | |
| boden | AJaeger ok | 15:29 |
| openstackgerrit | Merged openstack-infra/system-config master: Add gerrit_configure flag to review-dev01.o.o https://review.openstack.org/554624 | 15:30 |
| *** VW has quit IRC | 15:30 | |
| *** agopi is now known as agopi|lunch | 15:31 | |
| frickler | infra-root: I'll be afk now, but in case someone has time to continue investigating post failures, it does look like we have some significant increase in the last 48h or so: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22POST-RUN%20END%20RESULT_TIMED_OUT%5C%22 | 15:31 |
| clarkb | frickler: thanks, I'll probably dig into that once properly awake | 15:31 |
| clarkb | I was looking at a post failure for the tripleo change that dib needs to get the 2.12.1 release out and it appears to be different than the post failure linked by dansmith above | 15:32 |
| *** VW_ has quit IRC | 15:32 | |
| *** VW has joined #openstack-infra | 15:32 | |
| *** yamamoto has joined #openstack-infra | 15:33 | |
| frickler | clarkb: yes, I saw two patterns, one the ssh hostkey changed during post and the other a timeout without further logs during fetch-devstack-log-dir | 15:33 |
| clarkb | I wonder if we had a zookeeper disconnection and nodepool recycled nodes (and their IPs) | 15:34 |
| clarkb | pabelanger: ^ what is the easiest way to check for that? | 15:34 |
| corvus | clarkb: nodepool doesn't recycle nodes | 15:35 |
| clarkb | corvus: right it would be the cloud recycle IPs, nodepool would delete and make new ones | 15:35 |
| clarkb | zuul scheduler memory use is still looking good so I don't think we had a swapping situation result in zk disconnects | 15:35 |
| frickler | clarkb: oh, now that you mention that, I was seeing this error when looking at held nodes earlier on nl01: "WARNING kazoo.client: Connection dropped: socket connection error: Permission denied" | 15:36 |
| frickler | clarkb: I ignored it because the command seemed to succeed anyway, but it might be related | 15:36 |
| corvus | clarkb: ah yes. it's possible the scheduler won't cancel a job if it loses the node. | 15:36 |
| corvus | which would cause weird behavior like this | 15:36 |
| pabelanger | clarkb: I usually grep debug log on scheduler looking for kazoo.client logging | 15:36 |
| clarkb | I think that would explain why some of the jobs basically timeout and others hit ssh key errors | 15:36 |
| clarkb | the timeouts happen when cloud doesn't recycle the IP and the key errors when it does | 15:37 |
| * clarkb checks the zk server first | 15:37 | |
| corvus | kazoo doesn't appear in the last 3 log files | 15:38 |
| clarkb | plenty of disk and free memory on nodepool.o.o. The zookeeper log itself doesn't complain about anything since last october | 15:38 |
| clarkb | and process has been running since january 28 | 15:38 |
| clarkb | I think zk itself is fine | 15:38 |
| *** dizquierdo has quit IRC | 15:39 | |
| *** florianf_ has quit IRC | 15:39 | |
| pabelanger | clarkb: I've used http://status.openstack.org/elastic-recheck/#1721093 too to help track them. Last time we lost zookeeper connection, there was a huge spike since all nodes were being deleted | 15:40 |
| corvus | yeah, if this is a trend vs an event, it's less likely to be a zk issue | 15:40 |
| corvus | logstash says they're all ovh-bhs1 | 15:41 |
| clarkb | corvus: dansmith's was rax ord | 15:41 |
| corvus | logstash says they're nearly all ovh-bhs1 | 15:41 |
| clarkb | http://logs.openstack.org/90/547990/6/check/legacy-tempest-dsvm-multinode-live-migration/28c2873/job-output.txt.gz#_2018-03-21_14_38_29_837702 is the log for that one | 15:41 |
| clarkb | (possible that dansmiths is separate issue) | 15:42 |
| *** florianf has joined #openstack-infra | 15:42 | |
| *** kien-ha has quit IRC | 15:42 | |
| corvus | like maybe 5 out of 100 are not bhs1 | 15:42 |
| corvus | in fact, exactly 5 out of the 100 i'm looking at are not bhs1 | 15:43 |
| clarkb | reading nl01 logs for the IPs involved in dansmith's failure I don't think nodepool deleted the node early or booted the reused IP nodes quickly enough to cause a conflict | 15:46 |
| *** kiennt26_ has joined #openstack-infra | 15:46 | |
| clarkb | the timestamps all line up in a run dansmiths job to completion, a minute or two later ask for cloud to delete nodes. Than half an hour later boot new instances with those same IPs | 15:47 |
| clarkb | so I think that does rule out the zk theory | 15:47 |
| *** eharney has joined #openstack-infra | 15:48 | |
| *** VW has quit IRC | 15:50 | |
| *** VW has joined #openstack-infra | 15:50 | |
| *** eernst has quit IRC | 15:52 | |
| *** eernst_ has joined #openstack-infra | 15:53 | |
| clarkb | in the dansmitch case I wonder if something in the devstack process is restarting networking and sshd is finding a new host key? | 15:54 |
| clarkb | fail at typing | 15:54 |
| clarkb | I'm going to look at a bhs1 case now | 15:54 |
| *** eernst_ has quit IRC | 15:54 | |
| clarkb | ara renders these funny, I guess because zuul is preemting it | 15:56 |
| *** eernst has joined #openstack-infra | 15:56 | |
| clarkb | ah yup the helpful ara float over tooltip thing says that it was interrupted | 15:57 |
| *** eernst has quit IRC | 15:57 | |
| *** eernst has joined #openstack-infra | 15:57 | |
| *** rosmaita has joined #openstack-infra | 15:58 | |
| *** iyamahat has quit IRC | 15:58 | |
| *** yamahata has quit IRC | 16:00 | |
| *** efried is now known as efried_rollin | 16:01 | |
| clarkb | corvus: 2018-03-21 15:15:52,503 WARNING nodepool.CleanupWorker: Deleting leaked instance ubuntu-xenial-ovh-bhs1-0003103290 (e30d69e9-ec6b-4f51-8a18-3fee2e13b2c2) in ovh-bhs1 (unknown node id 0003103290) | 16:01 |
| clarkb | corvus: nodepool seems ot think the node leaked. It is still leaking after the job failed though | 16:02 |
| openstackgerrit | Saju M proposed openstack/python-jenkins master: pypy is not checked at gate https://review.openstack.org/554971 | 16:03 |
| clarkb | oh wait it appears to do the normal deletion first then the cleanup thread runs on its every minute run or whatever and catches it because it is still around | 16:03 |
| corvus | so maybe just a slow delete | 16:03 |
| clarkb | ya I don't think that is odd anymore. Just a race for how quickly cloud can delete an instance | 16:04 |
| *** eernst has quit IRC | 16:06 | |
| *** jlabarre has quit IRC | 16:07 | |
| *** kien-ha has joined #openstack-infra | 16:07 | |
| *** katkapilatova has left #openstack-infra | 16:07 | |
| *** kien-ha has quit IRC | 16:10 | |
| *** danpawlik has quit IRC | 16:11 | |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thanks! | 16:13 |
| *** eernst has joined #openstack-infra | 16:16 | |
| *** andreas_s has quit IRC | 16:16 | |
| clarkb | short of networking trouble between the executor(s) and bhs1 I'm stumped. Running a ping with high packet count between ze07 (where one job timed out) to the bhs1 mirror lost no packets | 16:16 |
| *** yolanda_ has joined #openstack-infra | 16:19 | |
| *** eernst has quit IRC | 16:19 | |
| *** yolanda has quit IRC | 16:19 | |
| *** derekh has quit IRC | 16:21 | |
| *** derekh has joined #openstack-infra | 16:22 | |
| *** wolverineav has joined #openstack-infra | 16:22 | |
| *** dsariel has quit IRC | 16:23 | |
| *** andreas_s has joined #openstack-infra | 16:26 | |
| *** masber has quit IRC | 16:27 | |
| *** trown|ruck has quit IRC | 16:28 | |
| dirk | Wasn't there some way of doing a mixin? E.g I want to inherit Openstack-tox but change the nodeset.. e.g. build on bionic instead of the default xenial - for background of the question see https://review.openstack.org/#/c/554824/ | 16:29 |
| clarkb | dirk: the zuul docs call it a variant. | 16:30 |
| *** andreas_s has quit IRC | 16:30 | |
| corvus | dirk: wow, that syntax would be surprisingly easy to implement at this point. but no, it's not supported. | 16:31 |
| corvus | dirk: maybe just parent to cross-test and then change the nodeset? | 16:31 |
| corvus | (i hesitate to say this, but, to directly answer the question, if you just duplicated that job with a different parent line on each (as clarkb says -- variants), you'd get the mixin behavior. i say i hesitate because i don't know if that behavior is going to be too confusing) | 16:36 |
| dirk | corvus: yeah, there a ways to avoid multiple parents.. is that the best way? | 16:36 |
| *** dizquierdo has joined #openstack-infra | 16:37 | |
| *** kiennt26_ has quit IRC | 16:37 | |
| *** danpawlik has joined #openstack-infra | 16:37 | |
| *** armaan has quit IRC | 16:37 | |
| *** ramishra has quit IRC | 16:37 | |
| dirk | Hmm, yeah. Does bionic support multiple python 3.x versions? E.g. is it likely that we end up with a single distro that can do all tox jobs at some point? | 16:38 |
| dirk | I'll try duplicate of the job for now | 16:38 |
| *** gyee has joined #openstack-infra | 16:38 | |
| corvus | dirk: i honestly don't know which way is best. i've spend enough time with the algorithm that it seems reasonable to me; i'd like to know if others think mixins are reasonable, or too confusing/unmaintainable. | 16:38 |
| corvus | dirk: perhaps if folks do like the idea of mixins, we should legitimize it by adopting your 'list' syntax. | 16:39 |
| dirk | corvus: the other possibility is a yaml variable reference | 16:39 |
| dirk | E.g. &openstack_py36_nodeset | 16:40 |
| dirk | That is centrally defined to expqnd to whatever base distro is the choice for that tox flavor | 16:40 |
| corvus | dirk: yaml references will only work within the same file, so you could do that in that file, but not in project-config and use it in requirements | 16:40 |
| corvus | dirk: but we could define nodesets by purpose... ie, an actual nodeset called 'openstack-py36-nodeset'. | 16:41 |
| persia | Isn't https://review.openstack.org/#/c/550235/ the current way to do that sort of thing? | 16:41 |
| *** agopi|lunch has quit IRC | 16:41 | |
| persia | So one would just define a new base nodeset (with properties) if one wanted to have specific environments? | 16:42 |
| *** agopi|lunch has joined #openstack-infra | 16:42 | |
| *** danpawlik has quit IRC | 16:42 | |
| fungi | dirk: depends on what you mean by "support" but like xenial and trusty before it, the main archive for bionic only comes with a single python 3 interpreter (3.6.x). there are plenty of ways to install other versions of python on any of the platforms we offer depending on how complicated you want to make your jobs | 16:43 |
| corvus | persia: well, we have a bunch of nodesets defined here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/nodesets.yaml | 16:43 |
| corvus | persia: so right now, we're generally saying things like "openstack-tox-py36 requires bionic". i think dirk wants to say something like "requirements-py36 requires whatever openstack is using for py36" | 16:44 |
| corvus | so, another layer of indirection | 16:44 |
| corvus | one way of doing that is a 'mixin' of the openstack py36 job. another is indirection in the nodeset reference. | 16:44 |
| persia | Right. Based on what I've been learning to try to understand our nodepool config, I would expect the to use the "label" construct for that indirection. | 16:45 |
| corvus | persia: i think we want to keep the labels descriptive of what the nodes actually are (eg 'bionic'), and only add the indirection for their use to either jobs or nodesets in zuul | 16:45 |
| persia | Then I think I'll let this conversation conclude without adding more, and will want to have a different conversation about what to use when "bionic" isn't enough information to explain what a node *is*. | 16:46 |
| persia | (but I'm not prepared for the latter conversation yet) | 16:46 |
| persia | Broad gloss being that "bionic" describes a set of versions of software, installed in a way, but may not specify behaviour, specific packages installed, or even the ABI of the platform, if we support multiple architectures. | 16:47 |
| *** VW has quit IRC | 16:47 | |
| *** myoung|rover|mtg is now known as myoung | 16:48 | |
| fungi | in this case, as a policy we've (openstack infra) decided to only support one image for any given distro+release | 16:48 |
| *** VW has joined #openstack-infra | 16:48 | |
| corvus | persia: indeed; i believe the thought is that we can expand that as needed (eg, bionic-x86_64, bionic-arm [which is an awesome label name], etc) | 16:48 |
| fungi | but yes, we're i suppose extending that to distro+release+arch | 16:48 |
| dirk | corvus: nodeset name by purpose sounds good to me as well | 16:49 |
| * fungi wants a bionic arm now | 16:49 | |
| persia | corvus: Yes. I'm not currently prepared, but there was talk about "bionic" being some-random-arch-of-bionic in the future. | 16:49 |
| persia | fungi: You can run jobs against ubuntu-xenial-arm now, if you like. They work. Bionic is mostly waiting for bionic to finish (there are some wrinkles) and/or for us to have more capacity. | 16:49 |
| fungi | persia: i know, i was joking about wishing i had an appendage made of a mix of meat and artificial technology | 16:50 |
| *** eernst has joined #openstack-infra | 16:50 | |
| corvus | persia: perhaps we should anticipate this and go ahead and use ubuntu-bionic-x86 instead of ubuntu-bionic. | 16:51 |
| corvus | so, going forward, start encoding arch into labels | 16:51 |
| * persia wishes there were better ways to acheive the physical equivalent of "straight man" humor on iRC | 16:51 | |
| fungi | persia: i think i just did? ;) | 16:51 |
| *** e0ne has quit IRC | 16:51 | |
| corvus | basically, it's just that until recently, distro+release was sufficiently descriptive; now we have a second axis | 16:51 |
| persia | corvus: There are migration issues. I hope to have enough time to think through enough to propose a early draft spec in the next couple weeks. I would strongly suggest keeping just ubuntu-bionic as the node label for now. | 16:52 |
| fungi | yeah, i am wholeheartedly in favor of going ahead and extending our labels to include architecture for any new labels we add, and planning to go back and correct the old ones as well | 16:52 |
| persia | But, anyway, I am really interested in the best solution for purpose-based nodes (dirk's thing). We should discuss that now, as I think we have the right input information for it :) | 16:52 |
| fungi | for bionic there shouldn't be any migration concerns there | 16:53 |
| fungi | as we have said it's not officially supported and don't use it for voting jobs | 16:53 |
| *** myoung is now known as myoung|food | 16:53 | |
| persia | We already do, but only in a couple places, so it isn't actually that important for now. There is migration work to do, but it is small for bionic. Ideally, we'll sort the arch thing before we suggest projects test on bionic. | 16:53 |
| fungi | as long as we decide to have ubuntu-bionic-x86_64 and ubuntu-bionic-aarch64 or whatever before we officially support its use then i don't see a problem | 16:54 |
| fungi | "migration" may need to be done, but should be entirely non-impacting as far as ongoing software development is concerned | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to @zuul-ci/dashboard https://review.openstack.org/551999 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Stop falling back to job name for missing url https://review.openstack.org/554056 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use requests instead of urllib.request in tests https://review.openstack.org/554057 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/550978 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 16:55 |
| fungi | since those jobs should remain non-voting for another month-ish | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift html templates into components https://review.openstack.org/551327 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove dashboard workaround for missing log_url https://review.openstack.org/554066 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 16:55 |
| corvus | persia: so if we wanted to do that, then i think we should define purpose nodesets. a "python" nodeset we use for pep8 jobs, and a "python36" nodeset we use for openstack-tox-py36 and requirements-tox-py36 | 16:55 |
| persia | corvus: That was the thought I had, as it makes job definitions look like the project-config change I linked above. | 16:56 |
| persia | And that means users can target specific nodesets, which can be composed of whatever infra thinks is a good idea at the time. | 16:56 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint to delete RSVP Question Value https://review.openstack.org/554989 | 16:57 |
| corvus | persia: i'm having trouble connecting https://review.openstack.org/550235 to that suggestion (because it seems to be an example of a job specifying a complete anonymous nodeset, rather than using one which is purpose named) | 16:57 |
| *** camunoz has joined #openstack-infra | 16:57 | |
| *** hasharAway is now known as hashar | 16:57 | |
| corvus | persia: but yes, otherwise i agree we could would accomplish the thing you just said | 16:58 |
| persia | corvus: Ah. Apologies. That nodeset is one that is capable of building wheels that match the cpython running for a bionic mirror. That it happens to be the same nodeset that would be used by people testing with ubuntu-bionic is somewhat of a coincidence. | 16:58 |
| persia | The point being to just override the nodeset for a job with purpose-defined nodesets, rather than introducing a new feature to use different nodesets. | 16:59 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint to delete RSVP Question Value https://review.openstack.org/554989 | 16:59 |
| fungi | purpose-named nodesets may also serve us well if what we want to be able to do is unconditionally swap out the backing node platform for one without needing to survive through another piecemeal migration like trusty->xenial ended up being | 16:59 |
| *** eernst has quit IRC | 16:59 | |
| *** eernst has joined #openstack-infra | 17:00 | |
| persia | And such a migration may be more complicated now, as a greater portion of the job definitions now live entirely in project repos. | 17:00 |
| fungi | we declare a flag day (like we did with precise->trusty) and if your jobs don't work on the new platform then you get to shift your development focus to fixing that before you can land other changes | 17:00 |
| *** eernst has quit IRC | 17:01 | |
| *** eernst has joined #openstack-infra | 17:01 | |
| *** bradjones has quit IRC | 17:03 | |
| *** eernst has quit IRC | 17:03 | |
| *** udesale has quit IRC | 17:03 | |
| *** eernst has joined #openstack-infra | 17:03 | |
| *** danpawlik has joined #openstack-infra | 17:07 | |
| corvus | yeah, i anticipated that just changing the nodeset for 'openstack-tox-py3X' would generally be sufficient, but that's only true for that job and its descendents. so if you're doing something like the requirements cross-tests, you'd also need to update those if we didn't do one of the things we're talking about here. | 17:09 |
| *** trown has joined #openstack-infra | 17:09 | |
| *** panda is now known as panda|off | 17:10 | |
| *** trown is now known as trown|lunch | 17:11 | |
| *** felipemonteiro__ has quit IRC | 17:11 | |
| *** danpawlik has quit IRC | 17:11 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:11 | |
| *** armaan has joined #openstack-infra | 17:11 | |
| persia | 'openstack-tox-py3X' makes me wonder if it is possible to define a job in such a way that it always runs on *both* 'openstack-tox-py27' and 'openstack-tox-py36', or whether that ends up always needing to be two jobs. | 17:15 |
| fungi | we mostly accomplish that by making two jobs and then grouping them in a project-template | 17:17 |
| fungi | so the project can just add that template instead of needing to add the jobs individually | 17:17 |
| *** zoli is now known as zoli|gone | 17:18 | |
| persia | Makes sense. | 17:18 |
| *** zoli|gone is now known as zoli | 17:18 | |
| fungi | a multinode job which just runs distinct tasks on each node with no communication between the nodes is almost (always?) better implemented as separate jobs since zuul can schedule them independently | 17:18 |
| fungi | er, (almost?) always | 17:19 |
| *** danpawlik has joined #openstack-infra | 17:20 | |
| clarkb | ok back to debugging bhs1 failures | 17:20 |
| clarkb | corvus: pabelanger dmsimard do you know if the ssh connection manager thing logs its state anywhere on the executors? I'm wondering if that might give me a clue | 17:21 |
| dmsimard | clarkb: missing context.. what ssh connection manager thing ? Ansible ? Paramiko ? | 17:22 |
| clarkb | dmsimard: the ssh -o controlmaster process ansible uses to ssh to remote hosts on our executors | 17:22 |
| dmsimard | clarkb: you get the ansible literal ssh commands with "ansible -vvv" I believe | 17:22 |
| clarkb | reading the ssh man page it should go to stderr | 17:23 |
| clarkb | but I'm not sure ansible is capturing that stderr anywhere | 17:23 |
| dmsimard | clarkb: do you see what you need in http://paste.openstack.org/show/707773/ ? | 17:24 |
| dmsimard | oh wait, that's not SSH, that's the special local connection thing, hang on | 17:24 |
| *** danpawlik has quit IRC | 17:25 | |
| clarkb | dmsimard: well in this case I'm hoping ansible/zuul are alrady logging it somewhere I'm not seeing so that I can review logs for jobs that have failed | 17:26 |
| *** ykarel is now known as ykarel|afk | 17:26 | |
| *** pbourke has quit IRC | 17:27 | |
| dmsimard | clarkb: with SSH: http://paste.openstack.org/raw/707777/ | 17:28 |
| dmsimard | clarkb: zuul executors don't run ansible with -vvv unless they have verbosity activated | 17:28 |
| dmsimard | they run with one -v by default iirc | 17:28 |
| clarkb | gotcha | 17:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enable autohold for RETRY_LIMIT https://review.openstack.org/554995 | 17:28 |
| *** agopi|lunch is now known as agopi| | 17:28 | |
| clarkb | corvus: thoughts on turning that on to help debug the bhs1 network problems? | 17:28 |
| *** agopi| is now known as agopi | 17:28 | |
| dmsimard | pabelanger: added a comment on that patch | 17:29 |
| *** florianf has quit IRC | 17:30 | |
| *** lucasagomes is now known as lucas-afk | 17:31 | |
| *** lpetrut_ has quit IRC | 17:32 | |
| *** jpich has quit IRC | 17:33 | |
| *** NobodyCam has quit IRC | 17:37 | |
| *** r-daneel has quit IRC | 17:37 | |
| *** dhajare has joined #openstack-infra | 17:37 | |
| *** r-daneel has joined #openstack-infra | 17:37 | |
| *** icey has quit IRC | 17:37 | |
| *** gus has quit IRC | 17:38 | |
| *** kuromagi has quit IRC | 17:38 | |
| *** kuromagi has joined #openstack-infra | 17:38 | |
| *** v1k0d3n has quit IRC | 17:38 | |
| *** gmann_ has quit IRC | 17:38 | |
| *** NobodyCam has joined #openstack-infra | 17:39 | |
| *** andreaf has quit IRC | 17:39 | |
| *** gus has joined #openstack-infra | 17:39 | |
| *** andreaf_ has joined #openstack-infra | 17:39 | |
| *** gmann_ has joined #openstack-infra | 17:40 | |
| *** icey has joined #openstack-infra | 17:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:40 | |
| *** v1k0d3n has joined #openstack-infra | 17:40 | |
| *** efoley has quit IRC | 17:40 | |
| *** danpawlik has joined #openstack-infra | 17:40 | |
| clarkb | as another datapoint my irc host is actually in bhs1 too | 17:40 |
| clarkb | and I've not noticed any networking trouble | 17:40 |
| openstackgerrit | Merged openstack-infra/zuul master: Add zuul-stream remote tests https://review.openstack.org/554714 | 17:41 |
| *** andreaf_ is now known as andreaf | 17:41 | |
| *** felipemonteiro__ has quit IRC | 17:41 | |
| *** myoung|food is now known as myoung | 17:45 | |
| *** danpawlik has quit IRC | 17:45 | |
| dmsimard | clarkb: I'm not up to date on BHS1.. can you summarize what's going on ? | 17:49 |
| clarkb | dmsimard: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22POST-RUN%20END%20RESULT_TIMED_OUT%5C%22 shows a rather large number of post run timeouts in bhs1. They all seem to be due to ssh timing out at the end of the job which times out the job then the rest of the taks end up working after that | 17:50 |
| clarkb | dmsimard: http://logs.openstack.org/03/529703/1/gate/nova-tox-functional/bd1f381/job-output.txt.gz#_2018-03-21_14_44_13_399218 is a specific example | 17:50 |
| clarkb | http://logs.openstack.org/97/554697/1/gate/openstack-tox-py35/7a5562b/job-output.txt#_2018-03-21_15_51_30_579759 is another | 17:51 |
| clarkb | interestingly they seem to maybe all be failing getting result data. Perhaps there is some correlation there | 17:52 |
| dmsimard | Where does "Copy files from /home/zuul/workspace/ on node" come from ? codesearch is turning up empty | 17:52 |
| clarkb | where do you see that? | 17:52 |
| dmsimard | ah, it's not failing at the same place for every job | 17:52 |
| clarkb | correct, these are distinct jobs with different post playbooks | 17:53 |
| dmsimard | I found it here http://logs.openstack.org/32/554832/2/check/networking-odl-rally-carbon/668995e/job-output.txt#_2018-03-21_17_07_30_450835 | 17:53 |
| clarkb | but they do seem to be doing similar tasks | 17:53 |
| clarkb | basically copying the data from test node to the executor | 17:53 |
| *** pickle has quit IRC | 17:53 | |
| *** pickle has joined #openstack-infra | 17:53 | |
| clarkb | ya doing a synchronize pill to executor work root | 17:55 |
| *** haleyb has quit IRC | 17:56 | |
| clarkb | thinking about it does that rsync go through the controlmaster ssh connection? | 17:57 |
| *** e0ne has joined #openstack-infra | 17:58 | |
| dmsimard | btw unrelated but I'm seeing repeated occurrences of puppet complaining: http://paste.openstack.org/show/707808/ | 17:58 |
| clarkb | if not it could explain why the rest of the job is happy if it continues on over the controlmaster ssh connection | 17:58 |
| dmsimard | mordred: ^ in case you know what this is | 17:58 |
| clarkb | while the synchronizes (rsync) in particular are unhappy | 17:58 |
| dmsimard | clarkb: rsync over ssh basically opens an ephemeral rsyncd server on the other side before pushing the data.. right ? | 17:59 |
| corvus | clarkb: i know of no reason it wouldn't use the same controlmaster | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| *** sambetts is now known as sambetts|afk | 18:00 | |
| clarkb | reading the docs and the code it appears to not use the same control master by default | 18:01 |
| clarkb | http://docs.ansible.com/ansible/latest/synchronize_module.html its an explicit flag you have to set: use_ssh_args | 18:01 |
| dmsimard | corvus: the synchronize module is ... very confusing to say the least. | 18:01 |
| clarkb | I think that explains at least the mode of failure and why the rest of the job is generally happy | 18:01 |
| clarkb | it doesn't explain why rsync/synchronize are failing | 18:01 |
| dmsimard | clarkb: I've also come across a suggestion that we set "ansible_ssh_common_args" in the inventory instead of under the [ssh] block | 18:02 |
| *** felipemonteiro__ has joined #openstack-infra | 18:02 | |
| dmsimard | I'm not exactly sure why | 18:03 |
| clarkb | on ze07 the zuul user has >17k files open which could potentially cause problems though unlike that would be so cloud region specific | 18:04 |
| clarkb | my normal login on that host has a 4k file limit | 18:05 |
| *** pblaho has quit IRC | 18:05 | |
| *** VW_ has joined #openstack-infra | 18:05 | |
| corvus | i switched ze01 to verbose | 18:05 |
| *** VW_ has quit IRC | 18:05 | |
| *** VW_ has joined #openstack-infra | 18:06 | |
| corvus | i want to see some of those command lines | 18:06 |
| *** felipemonteiro_ has quit IRC | 18:06 | |
| dmsimard | There's several upstream issues around ssh args and the synchronize module... tl;dr it seems complicated to make it use what we want it to use (in this case -o ControlMaster ?) i.e, https://github.com/ansible/ansible/issues/16767 | 18:08 |
| *** VW has quit IRC | 18:08 | |
| corvus | ssh -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/var/lib/zuul/builds/77b410b77f104d388a42304b7b0d9470/work/.ssh/known_hosts -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=zuul -o ConnectTimeout=30 -o | 18:09 |
| corvus | ControlPath=/var/lib/zuul/builds/77b410b77f104d388a42304b7b0d9470/.ansible/cp/.... | 18:09 |
| corvus | that's a typical ansible ssh invocation for reference | 18:09 |
| corvus | (not an rsync one) | 18:10 |
| clarkb | ConnectTimeout is probably one that would speed up this failure mode if it were set | 18:10 |
| clarkb | (on the rsync) | 18:10 |
| corvus | hrm, verbose doesn't appear to be outputting the rsync/ssh command. i just see the module args. | 18:12 |
| corvus | i don't suppose it makes it into ara or zuul_json? | 18:12 |
| *** danpawlik has joined #openstack-infra | 18:12 | |
| *** VW_ has quit IRC | 18:13 | |
| *** VW has joined #openstack-infra | 18:13 | |
| *** jpena is now known as jpena|off | 18:14 | |
| clarkb | 2018-03-21 18:14:06,585 WARNING kazoo.client: Connection dropped: socket connection error: Permission denied getting that trying to do a `sudo -H -u nodepool nodepool list` on nl01. I guess that is where frickler saw it and not in the logs (I do get the listing output though) | 18:14 |
| clarkb | ok theorying time | 18:15 |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| clarkb | our bhs1 nodes have ipv6 addrs according to neutron | 18:15 |
| clarkb | But they don't work because nothing on the node knows about them or how to configure them | 18:16 |
| clarkb | and occasionally rsync is going to use the ipv6 address instead of ipv4 | 18:16 |
| corvus | occasionally? | 18:16 |
| clarkb | corvus: well I don't think all the jobs in bhs1 are failing | 18:17 |
| clarkb | corvus: so I'm guessing there is some non determinism there? maybe order of ips returned by shade/nodepool? I dunno | 18:17 |
| *** danpawlik has quit IRC | 18:17 | |
| corvus | clarkb: i believe we only give ansible one ip address, so if v6 isn't showing up in the inventory file, it shouldn't be involved | 18:18 |
| clarkb | http://logs.openstack.org/03/529703/1/gate/nova-tox-functional/bd1f381/zuul-info/inventory.yaml ok and it is ipv4 there | 18:18 |
| clarkb | nodepool does list the public ipv6 addr under its listing though | 18:18 |
| corvus | yeah, but ansible_host is the important bit here | 18:18 |
| mordred | corvus, clarkb: reading scrollback | 18:19 |
| *** trown|lunch is now known as trown | 18:19 | |
| *** lpetrut has joined #openstack-infra | 18:20 | |
| clarkb | is it possible that the ssh key manipulation that runs during the job would be confused by the nodepool data ? | 18:20 |
| mordred | ok. it doesn't look like there's an immediate shade bug at least ... | 18:21 |
| corvus | /usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no --rsync-path=sudo rsync --safe-links --out-format=<<CHANGED>>%i %n%L zuul@158.69.64.111:/opt/stack/data/ca-bundle.pem /var/lib/zuul/builds/b425a5fddcf24242a85f5291aeb2b7a3/work/ca-bundle.pem | 18:21 |
| *** EmilienM is now known as mimi | 18:21 | |
| *** mimi is now known as EmilienM | 18:21 | |
| corvus | that looks like a typical rsync command according to zuul_json | 18:21 |
| corvus | switching ze01 to unverbose | 18:21 |
| *** gfidente is now known as gfidente|afk | 18:22 | |
| mordred | TIL synchronize uses ssh completely differently | 18:22 |
| corvus | yeah, i very much stand corrected on that | 18:22 |
| dmsimard | According to https://github.com/ansible/ansible/issues/16767#issuecomment-233898082 -- it seems a workaround is to tell the synchronize module to *really* use the SSH configuration we're running Ansible with | 18:23 |
| dmsimard | Which is sort of unfortunate | 18:23 |
| corvus | dmsimard: well even that bug suggests that use_ssh_args would work for us | 18:24 |
| dmsimard | It doesn't really explain why things are suddenly failing and (mostly) in bhs1 though | 18:24 |
| *** harlowja has joined #openstack-infra | 18:24 | |
| corvus | indeed -- we've found a difference, but not an explanation | 18:25 |
| clarkb | poking around the only places that seem to use ipv6 are multinode roles that setup host keys, /etc/hosts, and firewall rules | 18:25 |
| *** dhajare has quit IRC | 18:25 | |
| clarkb | it is possible the /etc/hosts stuff would braek on that but that would break in the job itself not post run | 18:26 |
| clarkb | also many of these jobs are single node | 18:26 |
| dmsimard | Does anyone know if we use custom ssh args in a synchronize module somewhere ? Looking at the upload-logs we don't do anything special. | 18:26 |
| dmsimard | I vaguely remember just falling back to a rsync command task due to this kind of nonsense before.. | 18:26 |
| corvus | dmsimard: i think zuul v2.5 used rsync directly mostly due to trying to achieve compat with jenkins. i think things got simpler with v3 and we can just use sync. | 18:28 |
| dmsimard | Ah, found it. It was for another issue related to delegation of the synchronize task https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/destroy-vm.sh#L77-L88 | 18:28 |
| corvus | (btw, there's a suggestion in that bug about defaulting use_ssh_args to true. and another about making it a config file option. either would be nice) | 18:28 |
| frickler | clarkb: ack on the kazoo.client warning, that's exactly what I saw, too. sorry for not having been more explicit | 18:29 |
| dmsimard | corvus: It sounds like setting use_ssh_args to true would be a good call but it needs to be done on a task basis unless we ship a custom synchronize module (like we do for other modules) | 18:30 |
| clarkb | 158.69.77.125 is a node that may hve this happenign to it | 18:30 |
| clarkb | there is no active zuul user session but we have a console log daemon thing floating around | 18:30 |
| clarkb | I can ssh into it just fine as root | 18:30 |
| corvus | what about from the executor? | 18:30 |
| *** haleyb has joined #openstack-infra | 18:30 | |
| corvus | (which executor is it) | 18:31 |
| clarkb | I don't know haven't gotten that far :) | 18:31 |
| clarkb | I went from nodepool up not zuul down | 18:31 |
| clarkb | course I only have about 4 more minutes before it gets auto deleted :/ | 18:31 |
| corvus | ze01 build 83f63965727e4159951d4f9b9d15de24 | 18:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 18:32 |
| dansmith | clarkb: here's another POST_FAILURE that actually ran to completion but failed after for some different reason: http://logs.openstack.org/02/545002/14/check/nova-multiattach/280795c/job-output.txt.gz | 18:32 |
| corvus | ze01 seems to be able to connect to that host over ssh | 18:33 |
| corvus | so if it's a network issue, it's a very transient one | 18:33 |
| *** dizquierdo has quit IRC | 18:33 | |
| clarkb | hrm that also isn't ending with a synchronize | 18:33 |
| clarkb | but on the host I don't see zuul /me looks harder | 18:34 |
| clarkb | netstat doesn't see an ssh either | 18:35 |
| clarkb | the ip for that build uuid doesn't seem to match tht may explain it | 18:35 |
| clarkb | oh maybe multinode | 18:35 |
| clarkb | so the task that is running is running on the other host? could mean this isn't exhibiting this problem in that case | 18:36 |
| dmsimard | in /tmp/tmp_hosts you have 158.69.77.125 and 158.69.77.136 | 18:36 |
| * dmsimard looks on 158.69.77.136 | 18:36 | |
| clarkb | I'm probably wrong about that host then | 18:36 |
| clarkb | if its multinode then it is likely just busy on the other node | 18:36 |
| dmsimard | it's odd that less is not installed on those machines | 18:38 |
| clarkb | dmsimard: they are based on the minimal elements from dib which is very minimal | 18:39 |
| dmsimard | yeah, it breaks journalctl and man pages (amongst probably other things) | 18:39 |
| clarkb | looking at the total number of jobs running on bhs1 this does seem to be fairly intermittent | 18:39 |
| dmsimard | even vi/vim isn't installed *gasp* | 18:39 |
| clarkb | I expect that if controlmaster fails during pre we just get a new node and try again and never notice. If it manages to connect things work because controlmaster | 18:40 |
| clarkb | then if you are lucky in post a new connection for rsync willfail | 18:40 |
| clarkb | corvus: is the controlmaster process shared across ansible processes? | 18:40 |
| dmsimard | I'm not sure what "auto" does | 18:40 |
| *** armaan has quit IRC | 18:42 | |
| corvus | clarkb: i think http://git.openstack.org/cgit/openstack-infra/zuul/commit/?id=a86aaf1158b2153e5aed5ae1fd550962330d01dc explains | 18:43 |
| dmsimard | hmm, we're not setting a controlpath ? Should we be doing that ? | 18:43 |
| *** rosmaita has quit IRC | 18:43 | |
| corvus | dmsimard: we do set a controlpath | 18:43 |
| *** yamamoto has quit IRC | 18:43 | |
| dmsimard | corvus: oh, I missed it in your paste, you're right | 18:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enable autohold for RETRY_LIMIT https://review.openstack.org/554995 | 18:43 |
| clarkb | ya so that fits into my current thinking | 18:44 |
| clarkb | it is intermittent enough that most jobs start fine and get through pre with a working connection then won't fail until the end when the rsync fails | 18:44 |
| *** andreas_s has joined #openstack-infra | 18:45 | |
| clarkb | corvus: we weren't getting any additioanl rsync logging from rsync itself were we when you turned on verbosity? | 18:45 |
| corvus | clarkb: that seemed to be the case | 18:46 |
| corvus | i only observed the module invocation dictionary as additional invocation | 18:46 |
| clarkb | I wonder if we could look for failed pres | 18:46 |
| clarkb | and catch ssh logging for that | 18:46 |
| dmsimard | corvus: I vaguely remember an issue where the controlpath path was too long... the one you pasted above seemed long enough, can you paste what the full control path actually looks like ? | 18:46 |
| clarkb | (this assumes that is happening at all which I don't have real evidence of yet) | 18:47 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/os-testr master: Use subunit and stestr API more https://review.openstack.org/509752 | 18:47 |
| corvus | dmsimard: it was only a little bit longer: /var/lib/zuul/builds/77b410b77f104d388a42304b7b0d9470/.ansible/cp/658a095346 | 18:47 |
| dmsimard | corvus: okay, so it's not that then. cool. | 18:47 |
| *** Swami has joined #openstack-infra | 18:48 | |
| dmsimard | clarkb: ssh logging where ? you mean on nodepool nodes ? or on the executor ? | 18:49 |
| *** ralonsoh has quit IRC | 18:49 | |
| clarkb | dmsimard: the executor | 18:49 |
| clarkb | dmsimard: to see what the failure condition is | 18:49 |
| *** andreas_s has quit IRC | 18:50 | |
| dmsimard | ok, fwiw the output of journalctl -u ssh on 158.69.77.136 (paste on fedoraproject due to paste.o.o truncating) https://paste.fedoraproject.org/paste/ppkb1pxXlhTiqs0bi6RobQ/raw | 18:51 |
| *** lpetrut has quit IRC | 18:51 | |
| *** danpawlik has joined #openstack-infra | 18:52 | |
| clarkb | dmsimard: ya I'm not longer convinced that pair of nodes was having problems | 18:52 |
| clarkb | I missed the fact that multinode could means similar | 18:52 |
| clarkb | *similar no zuul connection behavior | 18:52 |
| dmsimard | I don't remember seeing this kind of odd message before "Mar 21 17:37:36 ze03 sshd[3995]: Received disconnect from x.x.x.x port 42100:11: Normal Shutdown, Thank you for playing [preauth]" | 18:53 |
| bkero | http://lists.mindrot.org/pipermail/openssh-unix-dev/2014-January/031953.html | 18:54 |
| dmsimard | Some software have funny messages :) | 18:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix zuul-web port in zuul-from-scratch doc https://review.openstack.org/554829 | 18:56 |
| *** danpawlik has quit IRC | 18:56 | |
| *** jlabarre has joined #openstack-infra | 18:57 | |
| dmsimard | clarkb: have we isolated whether or not this is only occurring on synchronize tasks ? It's worth trying use_ssh_args if so -- it still won't explain the sudden ovh issues but if it works it's a worthwhile data point | 18:57 |
| dmsimard | (It seems to be only synchronize tasks) | 18:57 |
| clarkb | dmsimard: I've not examined each task no. But the ones I have looked at are synchronizes | 18:57 |
| dmsimard | I'll look at a couple | 18:58 |
| corvus | i also wonder, based on clarkb's multinode nodeset from earlier whether multinode jobs might be more likely to hit the controlpersist timeout and end up opening new connections on tasks mid-run | 18:59 |
| corvus | if we see multinode jobs hitting errors in bhs1 on non-synchronize tasks, that may be happening. if we aren't, then i wonder why it isn't happening. | 19:00 |
| *** lpetrut has joined #openstack-infra | 19:00 | |
| dmsimard | 9 out of 9 are synchronize tasks | 19:01 |
| dmsimard | oh, hey.. I know, we store these in graphite now, let's see when they started happening | 19:01 |
| * dmsimard looks | 19:01 | |
| dmsimard | my graphite-fu is rusty but the data points are in: stats_counts.zuul.executor.ze*_openstack_org.phase.*.RESULT_TIMED_OUT (or stats.zuul.executor.ze*_openstack_org.phase.*.RESULT_TIMED_OUT .. I'm not sure what's the difference between the two) | 19:05 |
| corvus | logstash says the rate has increased starting around 36 hours ago | 19:07 |
| corvus | maybe only 30 hours ago. hard to say. | 19:07 |
| clarkb | at a rate of 3-4 an hour? | 19:09 |
| clarkb | at least for the last 6 hours | 19:09 |
| *** jcoufal has quit IRC | 19:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enable autohold for RETRY_LIMIT / POST_FAILURE https://review.openstack.org/554995 | 19:13 |
| dmsimard | That seems to strangely correlate with an undergoing network maintenance in the BHS datacenter: http://status.ovh.com/?do=details&id=15328 | 19:13 |
| dmsimard | Which started yesterday | 19:13 |
| *** felipemonteiro__ has quit IRC | 19:13 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:13 | |
| dmsimard | (correlation != causation but just mentioning) | 19:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enable autohold for RETRY_LIMIT / POST_FAILURE https://review.openstack.org/554995 | 19:16 |
| clarkb | dmsimard: ya the more I dig into this the more I think it is likely a provider side issue. I can't find anywhere we'd use ipv6 that would affect this and break. Our images work most of the time and the jobs aren't consistent enough to point to a specific job | 19:17 |
| clarkb | the one thing consistent on our endappears to be synchronize but I think that is more innocent bystander not using the controlmaster than cause | 19:17 |
| corvus | yeah, i think our two next steps are: 1) give the ovh folks a heads up that we're seeing more connection timeouts than before. 2) start adopting use_ssh_args=true in our log copying tasks. | 19:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Return CORS headers on all requests https://review.openstack.org/555027 | 19:20 |
| *** myoung is now known as myoung|biab | 19:22 | |
| *** ykarel|afk is now known as ykarel|away | 19:22 | |
| *** dprince has quit IRC | 19:23 | |
| *** jaosorior has quit IRC | 19:23 | |
| *** savihou has joined #openstack-infra | 19:23 | |
| *** dprince has joined #openstack-infra | 19:23 | |
| *** savihou has quit IRC | 19:24 | |
| *** savihou has joined #openstack-infra | 19:25 | |
| *** eharney has quit IRC | 19:26 | |
| *** sree has joined #openstack-infra | 19:26 | |
| *** savihou has quit IRC | 19:28 | |
| *** savihou has joined #openstack-infra | 19:28 | |
| *** savihou has quit IRC | 19:28 | |
| *** eharney has joined #openstack-infra | 19:28 | |
| *** ykarel|away has quit IRC | 19:28 | |
| *** danpawlik has joined #openstack-infra | 19:28 | |
| prometheanfire | can I get an review (one more) glean, https://review.openstack.org/548604 | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Allow external zookeeper in tox py35 runs https://review.openstack.org/554810 | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Change test prints to log.info https://review.openstack.org/554058 | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix logging in tests to be quiet when expected https://review.openstack.org/554054 | 19:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add license and downgrade exception to alembic template https://review.openstack.org/554055 | 19:30 |
| *** sree has quit IRC | 19:31 | |
| *** danpawlik has quit IRC | 19:33 | |
| *** salv-orlando has quit IRC | 19:34 | |
| *** salv-orlando has joined #openstack-infra | 19:35 | |
| *** efried_rollin is now known as efried | 19:37 | |
| *** tesseract has quit IRC | 19:38 | |
| *** salv-orlando has quit IRC | 19:38 | |
| *** pickle is now known as dhill_ | 19:39 | |
| fungi | clarkb: not sure if you saw, but 552667 has a non-foundation-staff +2 now too | 19:40 |
| fungi | probably best if the ptl still approves that one, i suppose | 19:40 |
| clarkb | I'm running a while ssh clarkbsirchost 'echo foo' ; do sleep 5 ; done to see if I can catch ovh failing to my irc box | 19:40 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix zuul-web port in zuul-from-scratch doc https://review.openstack.org/554829 | 19:40 |
| clarkb | fungi: ok will look | 19:40 |
| fungi | clarkb: i'd give it even chances that the failures only impact certain instances at random (possibly those scheduled to certain hosts or something) rather than everything in their network | 19:42 |
| clarkb | fungi: ya likely | 19:43 |
| clarkb | also is _ valid in unix username? | 19:43 |
| corvus | clarkb, fungi: not sure if we want to just ping infra-root or something to lot folks know about https://review.openstack.org/552667 | 19:43 |
| *** dprince has quit IRC | 19:43 | |
| corvus | oh i just did | 19:43 |
| fungi | clarkb: that's a very good question, but hopefully one diablo_rojo has an answer to | 19:44 |
| *** jaosorior has joined #openstack-infra | 19:44 | |
| clarkb | corvus: ya I'm reviewing it now. as soon as I'm happy with the _ I will +2. | 19:44 |
| clarkb | (and approve once infra-root is done with it? | 19:44 |
| *** yamamoto has joined #openstack-infra | 19:44 | |
| pabelanger | +2 | 19:44 |
| *** salv-orlando has joined #openstack-infra | 19:44 | |
| clarkb | its valid as a filepath so should be fine for the homedir | 19:45 |
| corvus | it apparently matches the regex in debian's adduser | 19:46 |
| clarkb | ya internet seems to think anything that is a valid C identifier is fine | 19:47 |
| clarkb | so I think this should be fine | 19:48 |
| *** yamamoto has quit IRC | 19:49 | |
| clarkb | I've +2'd it will give it until after lunch for other roots to chime in and approve if there is no opposition | 19:50 |
| *** rfolco is now known as rfolco|ruck | 19:51 | |
| openstackgerrit | eldad marciano proposed openstack-infra/grafyaml master: Add datasource to template schema. https://review.openstack.org/548365 | 19:54 |
| *** VW has quit IRC | 19:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: add openstack-tox-lower-constraints https://review.openstack.org/555034 | 19:54 |
| *** VW has joined #openstack-infra | 19:54 | |
| *** eharney has quit IRC | 19:57 | |
| *** ekhugen has quit IRC | 19:59 | |
| *** danpawlik has joined #openstack-infra | 20:01 | |
| openstackgerrit | megan guiney proposed openstack-infra/project-config master: initial config for getting-started-with-openstack project https://review.openstack.org/554768 | 20:02 |
| *** ekhugen has joined #openstack-infra | 20:03 | |
| *** danpawlik has quit IRC | 20:06 | |
| pabelanger | nice, review-dev01.o.o is online | 20:12 |
| pabelanger | I'll now work on moving the volume from review-dev.o.o to review-dev01.o.o | 20:13 |
| *** eharney has joined #openstack-infra | 20:13 | |
| ianw | clarkb: https://review.openstack.org/#/c/554705/ ... hmmm tripleo is not looking happy | 20:14 |
| ianw | at this point i could force-merge a change to remove the test from dib, and then we could progress with fixing all that | 20:14 |
| *** Krenair has quit IRC | 20:14 | |
| clarkb | ianw: ya I was going to ask if we thought that was prudent. Considering there are other users of dib probably | 20:15 |
| clarkb | ianw: you may not need a force merge since it should be self testing config update? | 20:15 |
| pabelanger | ianw: clarkb: +1 | 20:15 |
| *** priteau has quit IRC | 20:15 | |
| ianw | oh, yeah, doh, it will drop the test | 20:16 |
| pabelanger | okay, powering off review-dev.o.o | 20:17 |
| *** Krenair has joined #openstack-infra | 20:18 | |
| clarkb | my ssh test to my bhs1 node never failed. I have stopped it. Likely only affecting certain subnets or l2 addresses over specific lacp links etc | 20:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove tripleo jobs https://review.openstack.org/555037 | 20:19 |
| *** Krenair has quit IRC | 20:22 | |
| *** gouthamr has quit IRC | 20:26 | |
| *** camunoz has quit IRC | 20:29 | |
| prometheanfire | pabelanger: mnaser https://review.openstack.org/548604 please? | 20:29 |
| *** Krenair has joined #openstack-infra | 20:30 | |
| *** armaan has joined #openstack-infra | 20:30 | |
| *** salv-orlando has quit IRC | 20:32 | |
| *** jaosorior_ has joined #openstack-infra | 20:33 | |
| *** danpawlik has joined #openstack-infra | 20:33 | |
| clarkb | anyone else want to ack https://review.openstack.org/#/c/555037/ ? | 20:34 |
| *** kgiusti has left #openstack-infra | 20:36 | |
| dhellmann | do any of you have tools you use for making automated edits to yaml files? I have something that preserves the order, but not whitespace or comments. | 20:36 |
| logan- | ruamel.yaml allows you preserve and manipulate comments | 20:37 |
| *** jaosorior has quit IRC | 20:37 | |
| dhellmann | thanks, logan-, I'll take a look at that | 20:38 |
| fungi | yeah, as much as i'm not a fan of ruamel.yaml due to its dependency tie-ins to the whole suite of ruamel libs, it's the only library i'm aware of which preserves yaml whitespace, comments and ordering | 20:38 |
| *** danpawlik has quit IRC | 20:38 | |
| dhellmann | this is for a one-off thing to add the lower-constraints job to a bunch of in-repo configs so I think I can accept the dependencies | 20:38 |
| fungi | i wouldn't personally choose to use ruamel.yaml in general-purpose software i intend to distribute, it's handy for hacky utility uses | 20:39 |
| fungi | so yeah, seems suited to your use case here | 20:39 |
| dhellmann | yeah | 20:39 |
| pabelanger | clarkb: fungi: how does https://etherpad.openstack.org/p/jgLaT4MRuC look so far with review-dev01.o.o | 20:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Properly deprecate stackforge https://review.openstack.org/554312 | 20:41 |
| clarkb | fungi: ^ thank you for the review, but I've realized that I forgot to update index things so got that done | 20:41 |
| corvus | dhellmann: we haven't finished making zuul safe for lots of simultaneous zuul.yaml changes yet, so when you do that, be careful. usually i put a 20 minute delay between each patchset upload. | 20:41 |
| *** Krenair has quit IRC | 20:41 | |
| pabelanger | clarkb: fungi: right now volumes have been moved to new server, and think I'm ready to enable puppet again to finish gerrit installation | 20:41 |
| corvus | dhellmann: (each such change uses too much memory, so we run out if there are lots. a fix is in progress, but probably won't be complete for a few weeks yet) | 20:42 |
| dhellmann | corvus : yeah, fungi and I talked about doing them in small batches | 20:42 |
| corvus | that works too | 20:42 |
| *** Krenair has joined #openstack-infra | 20:42 | |
| clarkb | pabelanger: make sure you chown the contents of the volume if necessary | 20:42 |
| dhellmann | we said ~10 at a time | 20:42 |
| clarkb | pabelanger: the uids don't necessarily line up | 20:42 |
| *** amoralej is now known as amoralej|off | 20:43 | |
| dhellmann | I can go smaller if I need to | 20:43 |
| pabelanger | clarkb: yah, it looks correct now. But good idea to call it out | 20:43 |
| *** camunoz has joined #openstack-infra | 20:43 | |
| *** ethfci has joined #openstack-infra | 20:45 | |
| dhellmann | well, ruamel.yaml supports comments but doesn't maintain whitespace | 20:45 |
| *** yamamoto has joined #openstack-infra | 20:45 | |
| fungi | clarkb: oh, hah, i missed that you had renamed the file in that change | 20:45 |
| dhellmann | awk it is, I guess | 20:45 |
| pabelanger | okay, rebooting review-dev01.o.o to confirm /etc/fstab | 20:46 |
| fungi | dhellmann: or round-trip through diff/patch using options to ignore whitespace changes | 20:46 |
| openstackgerrit | Merged openstack-infra/zuul master: Allow external zookeeper in tox py35 runs https://review.openstack.org/554810 | 20:46 |
| dhellmann | fungi : I'm not sure how that would work, can you elaborate? | 20:46 |
| tonyb | does someone have time to EOL OpenStackAnsible as described in http://lists.openstack.org/pipermail/openstack-dev/2018-March/128330.html (or add me to bootstrappers so I can do it) | 20:47 |
| *** myoung|biab is now known as myoung | 20:47 | |
| corvus | dhellmann: gimme a sec, i'll get you some code | 20:47 |
| dhellmann | corvus : thanks | 20:47 |
| fungi | dhellmann: make the edit, generate a diff using the option for ignoring whitespace changes, reset, apply the diff. also kinda hacky but may allow you to not alter whitespace that way | 20:47 |
| slaweq | hi guys, do You know how we can remove old "feature/xxx" branches from neutron repo? | 20:48 |
| dhellmann | fungi : fun, I've never done that. I'll give it a try. | 20:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Finish gerrit install for review-dev01.o.o https://review.openstack.org/555048 | 20:48 |
| fungi | dhellmann: diff -w to "ignore all white space" (may need -B for "ignore changes where lines are all blank" too, i can't remember if that counts as part of -w) | 20:49 |
| pabelanger | clarkb: fungi: if you are good with etherpad, I think we can land ^ and kick review-dev01.o.o | 20:51 |
| *** yamamoto has quit IRC | 20:51 | |
| clarkb | pabelanger: +2 | 20:52 |
| *** Krenair has quit IRC | 20:54 | |
| fungi | pabelanger: yeah, that looks entirely sane | 20:55 |
| *** camunoz has quit IRC | 20:55 | |
| clarkb | dmsimard: any luck with that limestone mirror today? | 20:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add batch project update script https://review.openstack.org/555053 | 20:56 |
| *** rosmaita has joined #openstack-infra | 20:56 | |
| corvus | dhellmann, fungi: sorry that's not more polished, but it should hopefully get you going ^ | 20:57 |
| fungi | corvus: oh, that's actually really slick | 20:58 |
| fungi | your definition of polished is a lot stricter than mine | 20:58 |
| pabelanger | great | 20:58 |
| corvus | it totally has the wrong number of newlines between methods. :) | 20:59 |
| fungi | the hobgoblins will be displeaseed | 20:59 |
| fungi | displeased too | 20:59 |
| clarkb | nibalizer pointed out this new project called black, its basically gofmt for python and the color they are painting the shed is black | 21:00 |
| clarkb | apparently the focus is on maintaining minimal diffs and readability for code review which seems like a good goal | 21:00 |
| *** esberglu has quit IRC | 21:01 | |
| fungi | so sorta like autopep8? | 21:02 |
| clarkb | kinda, they break a few pep8 rules by default | 21:02 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| fungi | i'm all for breaking pep8 rules | 21:02 |
| fungi | they should file it as pep888 | 21:02 |
| clarkb | the biggest drawback I think is that it requires python3.6 which isn't quite in all the places yet | 21:03 |
| clarkb | and you have to want the code style it produces | 21:03 |
| fungi | sure, but that could be said for pep8/autopep8 as well | 21:03 |
| clarkb | pep8 is a lot more flexible. I'm not sure how aggressive autopep8 is | 21:03 |
| pabelanger | prometheanfire: did we ever start on nodepool dsvm testing for gentoo? | 21:05 |
| fungi | it's configurable to not apply certain rules at least | 21:05 |
| clarkb | ianw: did you see tripleo has asked for an email about removing those tests from tripleo | 21:05 |
| clarkb | er removing those tripleo tests from dib | 21:06 |
| prometheanfire | pabelanger: no, we were going to switch to a systemd image | 21:06 |
| prometheanfire | pabelanger: which is waiting on glean to support gentoo systemd | 21:06 |
| clarkb | also any other infra root want to review that really quickly so that we can get a dib erlase out and unpause our image builds? | 21:06 |
| prometheanfire | that review has been up there for a while... | 21:06 |
| dhellmann | corvus : I also ran into issues with ruamel.yaml changing large multi-line strings into quoted strings; does that formatter handle that case? | 21:06 |
| *** danpawlik has joined #openstack-infra | 21:07 | |
| *** Krenair has joined #openstack-infra | 21:07 | |
| ianw | clarkb: ... ok | 21:08 |
| clarkb | ianw: I figure its worth a note to them. I don't think we need their approval to remove the cogating | 21:08 |
| clarkb | (dib becoming an infra project and moving out of tripleo gives us that freedom) | 21:09 |
| pabelanger | prometheanfire: k, it would be great to start work on bring that online. I can maybe see what would be needed, but we should just be able to add image and job into nodepool. We then would depends-on to glean for any needed changes | 21:09 |
| pabelanger | fungi: are you okay to proceed on https://review.openstack.org/555048/ ? | 21:09 |
| prometheanfire | pabelanger: you don't have workflow on glean? | 21:09 |
| * prometheanfire wonders who does so he can go bother them | 21:10 | |
| corvus | dhellmann: i'm not certain, but i think the deltas from what we typically have in zuul.yaml files is minimal, so i wouldn't expect it to change that. | 21:10 |
| pabelanger | prometheanfire: I do, but have no way to know if that is actually the fix | 21:10 |
| prometheanfire | I'm building a systemd image right now with it (redefined the git source for glean) | 21:10 |
| *** gfidente|afk has quit IRC | 21:10 | |
| fungi | pabelanger: yep! approved | 21:10 |
| pabelanger | fungi: danke | 21:10 |
| prometheanfire | I'll test boot it to be sure once it's built and let you know (if that works) | 21:10 |
| *** eharney has quit IRC | 21:10 | |
| dhellmann | corvus : ok, thanks, I'll give it a try | 21:10 |
| clarkb | prometheanfire: pabelanger keep in mind the mbr partition table is currently broken in dib (we are working to fix it) it will likely boot but growroot will have a sad | 21:11 |
| *** bnemec is now known as sin-master | 21:12 | |
| *** sin-master is now known as bnemec | 21:12 | |
| *** danpawlik has quit IRC | 21:12 | |
| pabelanger | ack | 21:13 |
| clarkb | oh mwhahaha acked the dib test change anyways so I think we are doubly good | 21:13 |
| prometheanfire | clarkb: oh, guess my images won't work then :| (is building off master+patches) | 21:13 |
| clarkb | prometheanfire: well it may work for a boot test | 21:13 |
| clarkb | prometheanfire: but just not have much disk to use after that :) | 21:13 |
| clarkb | mostly something to be aware of during your testing | 21:13 |
| prometheanfire | I am including the growroot element | 21:13 |
| *** VW has quit IRC | 21:13 | |
| pabelanger | prometheanfire: left +2 with comments, a few more eyes might be safer :) | 21:13 |
| prometheanfire | that's fine, I'll reboot and see | 21:13 |
| mwhahaha | yea go ahead and fix it, i'm working on the tripleo ci stuff | 21:13 |
| *** esberglu has joined #openstack-infra | 21:14 | |
| pabelanger | clarkb: https://review.openstack.org/548604 glean change we are talking about | 21:14 |
| prometheanfire | mwhahaha: D&D tonight :P | 21:14 |
| clarkb | pabelanger: also do you know if we are actually ssh'ing into nodes during the nodepool tests? nodepool doesn't do that anymore itself right? so we'd have to explicitly do it | 21:14 |
| openstackgerrit | Merged openstack-infra/zuul master: Enable autohold for RETRY_LIMIT / POST_FAILURE https://review.openstack.org/554995 | 21:15 |
| pabelanger | clarkb: yah, we SSH | 21:15 |
| clarkb | or wait its just the ready script ? basic connectivity is still checked iirc | 21:15 |
| pabelanger | clarkb: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/tools/check_devstack_plugin.sh#n27 | 21:15 |
| pabelanger | could be improved for more coverage, like growroot if we wanted | 21:16 |
| clarkb | pabelanger: ya | 21:17 |
| pabelanger | might be good to validate HDD size we expect | 21:17 |
| clarkb | ianw: I went ahead and approved the dib change to remove the jobs | 21:17 |
| clarkb | don't want to wait anylonger on that one | 21:17 |
| ianw | clarkb: thanks | 21:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/jeepyb master: Support cgit alias sites and short names https://review.openstack.org/555063 | 21:24 |
| pabelanger | mnaser: We seem to be in good shape with vexxhost, how does it look on your side? | 21:24 |
| *** boden has quit IRC | 21:24 | |
| pabelanger | mnaser: do we want to bump max-servers? | 21:24 |
| dhellmann | corvus : that seems to work great, thanks! | 21:26 |
| *** priteau has joined #openstack-infra | 21:27 | |
| corvus | dhellmann: \o/ | 21:30 |
| corvus | clarkb, fungi, mordred: can you see https://review.openstack.org/555063 and my comment when you have a moment. | 21:30 |
| pabelanger | clarkb: fungi: I' | 21:31 |
| pabelanger | err | 21:32 |
| pabelanger | clarkb: fungi: I'm going to kick review-dev01.o.o now | 21:32 |
| *** felipemonteiro_ has joined #openstack-infra | 21:32 | |
| *** salv-orlando has joined #openstack-infra | 21:32 | |
| *** felipemonteiro__ has quit IRC | 21:35 | |
| *** salv-orlando has quit IRC | 21:38 | |
| *** Krenair has quit IRC | 21:43 | |
| *** danpawlik has joined #openstack-infra | 21:45 | |
| *** agopi is now known as agopi|dinner | 21:46 | |
| *** yamamoto has joined #openstack-infra | 21:47 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added get ticket types endpoints https://review.openstack.org/555071 | 21:48 |
| *** danpawlik has quit IRC | 21:50 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added get ticket types endpoints https://review.openstack.org/555071 | 21:51 |
| *** danpawlik has joined #openstack-infra | 21:53 | |
| *** yamamoto has quit IRC | 21:53 | |
| *** Krenair_ has joined #openstack-infra | 21:53 | |
| *** agopi|dinner has quit IRC | 21:53 | |
| *** priteau has quit IRC | 21:54 | |
| *** pcaruana has quit IRC | 21:55 | |
| *** rfolco|ruck is now known as rfolco|off | 21:57 | |
| *** danpawlik has quit IRC | 21:57 | |
| *** Krenair_ has quit IRC | 21:57 | |
| *** salv-orlando has joined #openstack-infra | 21:59 | |
| *** Krenair_ has joined #openstack-infra | 22:00 | |
| clarkb | corvus: thinking about that I think I'm ok with only hosting zuul (and any potential other repos) via http(s) if we think that will make a less confusing user experience | 22:00 |
| clarkb | corvus: anymore git:// isn't really necessary with smart http being pretty ubiquitous | 22:01 |
| clarkb | I think it would be good to continue supporting git:// for openstack/ as those repos have had it set up that way for a long time | 22:01 |
| * clarkb will go transcribe that on the change | 22:03 | |
| *** eernst has quit IRC | 22:05 | |
| pabelanger | that seems reasonable, I've been using https a lot more over git:// | 22:07 |
| clarkb | I think centos6 was really the last place where it would make a real differencein our world | 22:08 |
| clarkb | because the git there was too old to smart http | 22:09 |
| clarkb | dib functests are not fast | 22:09 |
| *** eernst has joined #openstack-infra | 22:10 | |
| *** eernst has quit IRC | 22:10 | |
| openstackgerrit | Merged openstack-infra/system-config master: Finish gerrit install for review-dev01.o.o https://review.openstack.org/555048 | 22:11 |
| clarkb | arg dib change was actually hit by the bhs1 thing | 22:12 |
| ianw | clarkb: what's the bhs1 thing? | 22:15 |
| clarkb | ianw: flaky ansible synchronize in post-run playbooks. Apparently syncrhonize doesn't set up rsycn to use the controlmaster persistent connection thing we use for all the other ansible ssh connectivity so it apepars to be more susceptible to this | 22:16 |
| clarkb | ianw: my hunch is that in pre if connectivity fails from the get go we just delete the node and retry again until ssh works and then it works through pre and run because of the controlmaster process but then rsync is more susceptable to it | 22:17 |
| clarkb | corvus' proposed plan was to update our use of synchronize to use controlmaster and send ovh an email about it | 22:17 |
| clarkb | ianw: dmsimard also pointed out that ovh is in the process of upgrading the operating system on some of their networking gear in bhs1 which may be related | 22:17 |
| clarkb | ianw: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22POST-RUN%20END%20RESULT_TIMED_OUT%5C%22 logstash query for it | 22:18 |
| clarkb | rate appears to be 3-4 times per hour | 22:19 |
| pabelanger | okay, now kicking review-dev01.o.o since patch landed | 22:19 |
| pabelanger | and puppet ran okay | 22:20 |
| pabelanger | let me see if gerrit starts | 22:20 |
| *** lpetrut has quit IRC | 22:22 | |
| *** danpawlik has joined #openstack-infra | 22:23 | |
| *** yamahata has joined #openstack-infra | 22:24 | |
| *** e0ne has quit IRC | 22:24 | |
| *** rcernin has joined #openstack-infra | 22:25 | |
| pabelanger | doh, security email is me from review-dev01 | 22:26 |
| pabelanger | okay, gerrit looks to be running but an issue with apache config | 22:28 |
| *** danpawlik has quit IRC | 22:28 | |
| *** threestrands has joined #openstack-infra | 22:29 | |
| *** felipemonteiro_ has quit IRC | 22:29 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:29 | |
| *** threestrands has quit IRC | 22:30 | |
| pabelanger | woot | 22:30 |
| pabelanger | https://review-dev01.openstack.org | 22:30 |
| pabelanger | clarkb: fungi: ^ | 22:30 |
| *** threestrands has joined #openstack-infra | 22:30 | |
| *** bobh has quit IRC | 22:30 | |
| pabelanger | I had to modify apache2 manually, but will propose a fix in system-config | 22:30 |
| *** Krenair_ has quit IRC | 22:31 | |
| clarkb | pabelanger: login doesn't work because it wants to redirect to review-dev.o.o | 22:31 |
| clarkb | once dns is updated it should work | 22:32 |
| pabelanger | yah, there is some issues around numeric hostnames | 22:32 |
| pabelanger | let me add dns and revert apache change and see if it works | 22:32 |
| ianw | clarkb: thanks ... that sounds ... too much for me to deal with right now :) | 22:33 |
| *** Krenair has joined #openstack-infra | 22:34 | |
| *** d0ugal has quit IRC | 22:34 | |
| ianw | i'm just doing some manual boots to verify the dib fix too | 22:35 |
| *** d0ugal has joined #openstack-infra | 22:37 | |
| *** bmace has joined #openstack-infra | 22:40 | |
| ianw | clarkb/pabelanger: speaking of gerrit, any particular thoughts on https://review.openstack.org/#/c/552288/ which fixes some of our custom sql so it works with h2, which is used during hte git-review unit testing? | 22:41 |
| pabelanger | clarkb: once DNS updates, https://review-dev.openstack.org/107974 ready for review :D | 22:42 |
| ianw | i tested that via an online sql fiddle thing, so it's 100% to be absolutely fine :) | 22:43 |
| clarkb | ianw: it would be good to have mordred debug/review that one too | 22:43 |
| clarkb | ianw: mordred wrote the mysql ism updates to make our upgrade work in the first place | 22:43 |
| clarkb | ianw: and we can toss the resulting war onto review-dev once pabelanger gets it working | 22:43 |
| pabelanger | clarkb: so, I think tomorrow we can apply patches we used for review-dev01.o.o, merge, then launch the replacement review01.o.o server to obtain IP address. Then send out the email to ML and prepare for migrate next week? | 22:44 |
| clarkb | pabelanger: ya if review-dev ends up happy with the dns update I think that would be the next step. As for preparing to migrate next week may be hard for some because it is apparently easter | 22:44 |
| *** hashar has quit IRC | 22:45 | |
| clarkb | also tc discussions are related to connectivity issues we may want to consider more notice for the ip addr update | 22:45 |
| *** armaan has quit IRC | 22:45 | |
| pabelanger | sure, getting the replacement server online is first step, deciding when to move volumes can them be made. I'd say, 60min window (longer for buffer if we want) is all we'd need. review-dev01 went very well | 22:46 |
| clarkb | ya should go qucik since we aren't transforming any data | 22:46 |
| clarkb | just moving it | 22:46 |
| *** andreas_s has joined #openstack-infra | 22:47 | |
| pabelanger | yah, as long as we detach clean, should be fine | 22:47 |
| pabelanger | okay, going to get some food then poke around on new server | 22:48 |
| pabelanger | #status log review01-dev.o.o now online (ubuntu-xenial) and review-dev.o.o DNS redirected | 22:49 |
| openstackstatus | pabelanger: finished logging | 22:49 |
| *** iyamahat has joined #openstack-infra | 22:49 | |
| *** yamamoto has joined #openstack-infra | 22:49 | |
| *** hongbin has quit IRC | 22:50 | |
| clarkb | pabelanger: remember to check the ip against email blacklists | 22:51 |
| *** andreas_s has quit IRC | 22:51 | |
| *** esberglu has quit IRC | 22:51 | |
| *** yamamoto has quit IRC | 22:54 | |
| *** masber has joined #openstack-infra | 23:04 | |
| *** danpawlik has joined #openstack-infra | 23:04 | |
| pabelanger | clarkb: right, where I look for that? | 23:04 |
| njohnston_ | Quick question - how would I go about getting added to the -core group for a project that I created, but somehow the -core group was left without any members in it? | 23:04 |
| *** ldnunes has quit IRC | 23:05 | |
| clarkb | pabelanger: https://www.spamhaus.org/lookup/ you can check the IP (do v4 and v6) and request removals if necessary | 23:05 |
| clarkb | njohnston_: the initial group member is an explicit manual add | 23:05 |
| clarkb | njohnston_: can you point me to the change that created the group? and I can add the appropriate initial member based on that info | 23:05 |
| njohnston_ | clarkb: Thanks! https://review.openstack.org/#/c/546260/3 | 23:06 |
| *** Goneri has quit IRC | 23:06 | |
| njohnston_ | sorry, not sure why I have an out of date changeset bookmarked, should just be https://review.openstack.org/#/c/546260/ | 23:06 |
| clarkb | ya I found the current one :) | 23:07 |
| *** edmondsw has quit IRC | 23:07 | |
| *** jtomasek has quit IRC | 23:08 | |
| clarkb | njohnston_: I have added you to the group. The group is self owned so now that it has an initial member, that individual (you) can add whoever you like (and they too can add whoever they like) | 23:08 |
| *** danpawlik has quit IRC | 23:08 | |
| njohnston_ | Thanks very much clarkb! | 23:08 |
| *** tpsilva has quit IRC | 23:09 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix default partition type https://review.openstack.org/554771 | 23:11 |
| *** salv-orlando has quit IRC | 23:26 | |
| *** salv-orlando has joined #openstack-infra | 23:26 | |
| *** Krenair has quit IRC | 23:28 | |
| *** tosky has quit IRC | 23:28 | |
| *** r-daneel has quit IRC | 23:30 | |
| *** salv-orlando has quit IRC | 23:30 | |
| *** Krenair has joined #openstack-infra | 23:38 | |
| *** danpawlik has joined #openstack-infra | 23:39 | |
| *** Adri2000 has quit IRC | 23:40 | |
| *** Adri2000 has joined #openstack-infra | 23:41 | |
| *** felipemonteiro_ has quit IRC | 23:41 | |
| *** Krenair has quit IRC | 23:43 | |
| *** danpawlik has quit IRC | 23:44 | |
| *** gyee has quit IRC | 23:45 | |
| *** claudiub has quit IRC | 23:51 | |
| *** yamamoto has joined #openstack-infra | 23:51 | |
| *** Krenair has joined #openstack-infra | 23:52 | |
| pabelanger | clarkb: fungi: so far I don't see anything wrong on review-dev01.o.o. I haven't looked at storyboard-dev integration but can in the morning. anything else I should be looking at? Anything zuul related we should test? | 23:53 |
| clarkb | pabelanger: considering the biggest change is java 8 probably just normal functionality. Pushing code, reviewing changes, etc | 23:54 |
| pabelanger | Yah, I'll do more of that testing tomorrow morning for sure | 23:55 |
| *** yamamoto has quit IRC | 23:55 | |
| *** Krenair has quit IRC | 23:58 | |
| bmace | hey folks. i read through all the instructions and read through all the current values in project-config/gerrit/projects.yaml. it isn't clear if an upstream / imported code repository retains its branches / tags. can anyone tell me if it does or if it essentially just pulls master and the rest is lost? | 23:58 |
| clarkb | bmace: it should pull in all branches and tags as is | 23:59 |
| bmace | clarkb: thanks very much :) | 23:59 |
| clarkb | (it explicitly tries to do this at least and I don't recall anyone ever saying it failed at it) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!