| pabelanger | rolled back ubuntu-trusty dib | 00:00 |
|---|---|---|
| *** ijw has quit IRC | 00:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for operator_cloud functional tests https://review.openstack.org/460784 | 00:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for operator_cloud functional tests https://review.openstack.org/460784 | 00:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Force /bin/bash for jenkins user https://review.openstack.org/460781 | 00:02 |
| *** esikachev has quit IRC | 00:04 | |
| *** lamt has joined #openstack-infra | 00:08 | |
| *** petems has joined #openstack-infra | 00:10 | |
| *** esikachev has joined #openstack-infra | 00:11 | |
| *** ihrachys has joined #openstack-infra | 00:12 | |
| *** xgerman has joined #openstack-infra | 00:14 | |
| *** puiterwijk|cld has joined #openstack-infra | 00:15 | |
| pabelanger | ianw: seen a presentation about linuxkit last night, like DIB but driven from containers | 00:15 |
| pabelanger | interesting concept | 00:15 |
| fungi | clarkb: EmilienM: i think branch deletion is waiting for gerrit upgrade where there will be explicit permission for an api call to do that | 00:15 |
| *** esikachev has quit IRC | 00:16 | |
| fungi | so for now branch deletion is basically still manual as far as i'm aware | 00:16 |
| *** esikachev has joined #openstack-infra | 00:17 | |
| *** ijw has joined #openstack-infra | 00:20 | |
| ianw | pabelanger: hmm, probably the interesting bit of dib is the "d" part | 00:22 |
| *** esikachev has quit IRC | 00:22 | |
| Sukhdev_ | fungi clarkb : why do I get requirements mismatch on a new project - https://review.openstack.org/#/c/460767/ | 00:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 00:23 |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack master: Fix puppet lint errors https://review.openstack.org/460787 | 00:24 |
| pabelanger | ianw: disk? | 00:24 |
| *** ijw has quit IRC | 00:24 | |
| ianw | yeah, partitioning, bootloaders, lvm, encryption etc etc | 00:24 |
| *** ihrachys has quit IRC | 00:25 | |
| pabelanger | ianw: that's the thing, linuxkit does that. It creates a qcow2 image for you, but picked from container bits | 00:25 |
| pabelanger | it's pretty crazy | 00:25 |
| *** Julien-zte has joined #openstack-infra | 00:26 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Redirect stderr to stdout for some ansible calls https://review.openstack.org/458689 | 00:33 |
| *** xyang1 has quit IRC | 00:37 | |
| *** xyang1 has joined #openstack-infra | 00:38 | |
| clarkb | Sukhdev_: the job logs tell you | 00:38 |
| *** sflanigan has quit IRC | 00:38 | |
| clarkb | Sukhdev_: hacking requirement doesn't match | 00:38 |
| *** salv-orlando has joined #openstack-infra | 00:39 | |
| *** pvaneck has quit IRC | 00:40 | |
| *** markvoelker has joined #openstack-infra | 00:41 | |
| *** rfolco has joined #openstack-infra | 00:41 | |
| *** rfolco has quit IRC | 00:41 | |
| Sukhdev_ | clarkb: I saw that but, am confused as to what is that is needed though - the only requirement I see in the requirements.txt is pbr>-2.0 | 00:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add in a bunch of TODOs about interface=admin https://review.openstack.org/460790 | 00:42 |
| clarkb | Sukhdev_: https://review.openstack.org/#/c/460767/2/test-requirements.txt line 5 | 00:43 |
| *** salv-orlando has quit IRC | 00:44 | |
| Sukhdev_ | clarkb : got it - thanks | 00:46 |
| *** markvoelker has quit IRC | 00:46 | |
| pabelanger | fungi: sent out ML post about summit dinner, atleast get some numbers | 00:50 |
| clarkb | oh good someone else is goign to do it this time around :P | 00:51 |
| clarkb | pabelanger: as far as picking places I think it worked well where I essentially asserted a place and asked people to veto. Didn't get any veto | 00:51 |
| pabelanger | clarkb: cool, I'll take a look on some food sites, see what they recommend | 00:51 |
| clarkb | pabelanger: there is also a guide on the summit site. Thats what I used for ptg | 00:52 |
| *** esikachev has joined #openstack-infra | 00:52 | |
| pabelanger | k | 00:52 |
| fungi | pabelanger: thanks! replied | 00:54 |
| *** sflanigan has joined #openstack-infra | 00:55 | |
| *** sflanigan has joined #openstack-infra | 00:55 | |
| *** ihrachys has joined #openstack-infra | 00:55 | |
| *** kgiusti has joined #openstack-infra | 00:56 | |
| *** esikachev has quit IRC | 00:57 | |
| *** yee379 has quit IRC | 00:58 | |
| *** yee379 has joined #openstack-infra | 00:58 | |
| *** liusheng has quit IRC | 01:03 | |
| *** ihrachys has quit IRC | 01:05 | |
| *** SotK has quit IRC | 01:06 | |
| *** SotK has joined #openstack-infra | 01:06 | |
| *** jamesmca_ has joined #openstack-infra | 01:08 | |
| *** kgiusti has quit IRC | 01:09 | |
| *** afred312 has joined #openstack-infra | 01:09 | |
| *** Apoorva_ has joined #openstack-infra | 01:16 | |
| *** annegentle has joined #openstack-infra | 01:16 | |
| *** armax has quit IRC | 01:17 | |
| *** Gorian has quit IRC | 01:18 | |
| *** tphummel has quit IRC | 01:18 | |
| *** erlon has joined #openstack-infra | 01:19 | |
| *** Gorian has joined #openstack-infra | 01:19 | |
| *** Apoorva has quit IRC | 01:20 | |
| *** liusheng has joined #openstack-infra | 01:20 | |
| *** yolkfull has joined #openstack-infra | 01:20 | |
| *** Apoorva_ has quit IRC | 01:21 | |
| *** xarses_ has joined #openstack-infra | 01:21 | |
| *** xarses_ has quit IRC | 01:21 | |
| *** xarses_ has joined #openstack-infra | 01:21 | |
| *** spzala has quit IRC | 01:22 | |
| *** xarses_ has quit IRC | 01:22 | |
| *** xarses_ has joined #openstack-infra | 01:22 | |
| *** spzala has joined #openstack-infra | 01:23 | |
| *** spzala has quit IRC | 01:27 | |
| *** xarses_ has quit IRC | 01:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Fix warning from useradd https://review.openstack.org/460799 | 01:29 |
| pabelanger | clarkb: ianw: ^ interesting one, because /home does exist, useradd will complain and create a warning | 01:29 |
| *** zz_dimtruck is now known as dimtruck | 01:29 | |
| pabelanger | ianw: clarkb: maybe DIB should create /home out of the box? | 01:30 |
| ianw | hmmm, minimal doesn't maybe? devuser must? | 01:31 |
| *** tuanluong has joined #openstack-infra | 01:31 | |
| *** liujiong has joined #openstack-infra | 01:31 | |
| *** hongbin has joined #openstack-infra | 01:31 | |
| *** Julien-zte has quit IRC | 01:31 | |
| *** xarses_ has joined #openstack-infra | 01:32 | |
| ianw | surely it must be there from the base packages | 01:32 |
| pabelanger | actually, maybe that isn't the issue | 01:32 |
| pabelanger | let me google more | 01:32 |
| ianw | pabelanger: isn't that saying /home/jenkins already exists | 01:33 |
| ianw | so it's not going to copy anything over | 01:33 |
| pabelanger | maybe? | 01:33 |
| pabelanger | is the groupadd creating it? | 01:33 |
| ianw | i doubt it | 01:34 |
| pabelanger | ya, not sure what would be creating it now | 01:34 |
| pabelanger | let me test quick | 01:34 |
| ianw | mkdir -p ~jenkins/cache/files | 01:35 |
| ianw | maybe running prior? | 01:35 |
| *** yamahata_ has quit IRC | 01:35 | |
| pabelanger | don't think so | 01:35 |
| pabelanger | pretty sure it is after | 01:35 |
| ianw | both 20 in install.d ? | 01:36 |
| *** Julien-zte has joined #openstack-infra | 01:36 | |
| ianw | why do we even need the groupadd? | 01:37 |
| *** yanyanhu has joined #openstack-infra | 01:37 | |
| *** xarses_ has quit IRC | 01:38 | |
| *** gardlt has quit IRC | 01:38 | |
| *** markvoelker has joined #openstack-infra | 01:38 | |
| pabelanger | opensuse I think | 01:39 |
| pabelanger | I added it to be safe | 01:39 |
| pabelanger | was also what puppet-jenkins did | 01:39 |
| ianw | groupadd doesn't touch homedir on my test trusty | 01:40 |
| ianw | # useradd -g jenkins -m jenkins | 01:40 |
| ianw | useradd: group 'jenkins' does not exist | 01:40 |
| ianw | but, if you just leave the "-g jenkins" out, it creates the group | 01:41 |
| pabelanger | the behavior of useradd will depend on the USERGROUPS_ENAB variable in | 01:43 |
| pabelanger | /etc/login.defs | 01:43 |
| pabelanger | which, I think opensuse was disabled | 01:43 |
| clarkb | ya suse does users as default group | 01:43 |
| pabelanger | we can only do groupadd for opensuse if we want | 01:43 |
| pabelanger | but testing something now | 01:43 |
| *** spzala has joined #openstack-infra | 01:44 | |
| *** gouthamr has quit IRC | 01:44 | |
| ianw | pabelanger: 2017-04-27 07:00:48,402 in http://nb01.openstack.org/dib.ubuntu-trusty.log | 01:44 |
| ianw | puppet creating? | 01:44 |
| *** gouthamr has joined #openstack-infra | 01:45 | |
| pabelanger | http://nb02.openstack.org/dib.ubuntu-trusty.log is latest | 01:45 |
| *** salv-orlando has joined #openstack-infra | 01:45 | |
| EmilienM | fungi: if you can proceed to the branch deletion, when you can, that would be cool | 01:46 |
| EmilienM | good night folks | 01:46 |
| *** jamesmca_ has quit IRC | 01:47 | |
| pabelanger | confirmed /home is empty | 01:48 |
| pabelanger | testing again | 01:48 |
| *** rkukura has joined #openstack-infra | 01:48 | |
| *** spzala has quit IRC | 01:49 | |
| *** ramishra has joined #openstack-infra | 01:49 | |
| *** EricGonczer_ has joined #openstack-infra | 01:49 | |
| *** Julien-z_ has joined #openstack-infra | 01:49 | |
| *** salv-orlando has quit IRC | 01:50 | |
| *** spzala has joined #openstack-infra | 01:50 | |
| *** esikachev has joined #openstack-infra | 01:50 | |
| *** Julien-zte has quit IRC | 01:50 | |
| *** jamesmca_ has joined #openstack-infra | 01:51 | |
| *** Sukhdev_ has quit IRC | 01:52 | |
| *** EricGonc_ has joined #openstack-infra | 01:53 | |
| ianw | we just do "useradd -m zuul" later | 01:53 |
| *** EricGonczer_ has quit IRC | 01:53 | |
| *** gildub has joined #openstack-infra | 01:54 | |
| *** spzala has quit IRC | 01:55 | |
| *** esikachev has quit IRC | 01:55 | |
| pabelanger | I'll have to dig into it tomorrow | 01:55 |
| *** mat128 has joined #openstack-infra | 01:55 | |
| *** yolanda has quit IRC | 01:55 | |
| *** jamesmca_ has quit IRC | 01:56 | |
| *** Goneri has quit IRC | 01:58 | |
| *** gildub has quit IRC | 01:59 | |
| ianw | hmm, looking at the source, it's not like stat() /home/jenkins -> throw that message | 02:00 |
| ianw | it might be a red-herring | 02:00 |
| *** esikachev has joined #openstack-infra | 02:00 | |
| ianw | https://github.com/shadow-maint/shadow/blob/master/src/useradd.c#L1943 ... we don't see the mkdir failing. maybe it's the access() call that fails, maybe /home perms are wrong? | 02:02 |
| *** zhurong has joined #openstack-infra | 02:04 | |
| *** esikachev has quit IRC | 02:05 | |
| *** Nakato has quit IRC | 02:06 | |
| *** ramishra has quit IRC | 02:07 | |
| *** ramishra has joined #openstack-infra | 02:08 | |
| *** Nakato has joined #openstack-infra | 02:09 | |
| *** spzala has joined #openstack-infra | 02:11 | |
| *** rlandy|bbl is now known as rlandy | 02:12 | |
| *** esikachev has joined #openstack-infra | 02:13 | |
| *** spzala has quit IRC | 02:16 | |
| *** gildub has joined #openstack-infra | 02:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 02:16 |
| *** esikachev has quit IRC | 02:17 | |
| *** SotK has quit IRC | 02:22 | |
| *** SotK has joined #openstack-infra | 02:25 | |
| *** erikmwilson has quit IRC | 02:26 | |
| *** erikmwilson has joined #openstack-infra | 02:26 | |
| *** dave-mccowan has quit IRC | 02:27 | |
| *** yee379 has quit IRC | 02:31 | |
| *** yee37917 has joined #openstack-infra | 02:31 | |
| *** rkukura has quit IRC | 02:32 | |
| *** spzala has joined #openstack-infra | 02:32 | |
| *** Nakato has quit IRC | 02:32 | |
| *** Nakato has joined #openstack-infra | 02:33 | |
| *** rkukura has joined #openstack-infra | 02:35 | |
| *** spzala has quit IRC | 02:36 | |
| *** thorst has quit IRC | 02:36 | |
| *** tphummel has joined #openstack-infra | 02:37 | |
| *** yolkfull has quit IRC | 02:44 | |
| *** yolkfull has joined #openstack-infra | 02:44 | |
| *** armax has joined #openstack-infra | 02:45 | |
| *** salv-orlando has joined #openstack-infra | 02:46 | |
| ianw | yes of course adding a user has lead me down the path of converting things to python3 | 02:48 |
| *** spzala has joined #openstack-infra | 02:48 | |
| *** salv-orlando has quit IRC | 02:51 | |
| *** yolkfull_ has joined #openstack-infra | 02:51 | |
| *** spzala has quit IRC | 02:53 | |
| *** yolkfull__ has joined #openstack-infra | 02:53 | |
| *** yolkfull has quit IRC | 02:54 | |
| *** EricGonc_ has quit IRC | 02:55 | |
| *** yolkfull_ has quit IRC | 02:56 | |
| *** Julien-zte has joined #openstack-infra | 02:57 | |
| *** yolkfull has joined #openstack-infra | 02:57 | |
| *** yolkfull__ has quit IRC | 02:59 | |
| *** Julien-z_ has quit IRC | 03:00 | |
| *** spzala has joined #openstack-infra | 03:01 | |
| *** psachin has joined #openstack-infra | 03:06 | |
| *** thorst has joined #openstack-infra | 03:07 | |
| *** yolkfull_ has joined #openstack-infra | 03:09 | |
| *** yolkfull__ has joined #openstack-infra | 03:10 | |
| *** thorst has quit IRC | 03:11 | |
| *** yolkfull has quit IRC | 03:12 | |
| *** hongbin_ has joined #openstack-infra | 03:13 | |
| *** hongbin has quit IRC | 03:13 | |
| *** yolkfull_ has quit IRC | 03:14 | |
| *** mat128 has quit IRC | 03:15 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Don't run tempest tests in heat grenade job https://review.openstack.org/460810 | 03:16 |
| *** armax has quit IRC | 03:18 | |
| *** david-lyle is now known as dklyle | 03:19 | |
| *** dklyle is now known as david-lyle | 03:20 | |
| *** david-lyle is now known as dklyle | 03:21 | |
| *** nicolasbock has quit IRC | 03:22 | |
| *** dklyle is now known as david-lyle | 03:23 | |
| *** yolkfull__ is now known as yolkfull | 03:25 | |
| ianw | pabelanger: yep ... /home/jenkins/cache/files is what's pre-created the jenkins homedir | 03:26 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Add zun-ui server release job https://review.openstack.org/460815 | 03:29 |
| *** yamahata_ has joined #openstack-infra | 03:35 | |
| *** thorst has joined #openstack-infra | 03:38 | |
| *** zhurong has quit IRC | 03:38 | |
| *** Julien-zte has quit IRC | 03:40 | |
| *** john-davidge has joined #openstack-infra | 03:42 | |
| *** thorst has quit IRC | 03:45 | |
| *** rkukura has quit IRC | 03:47 | |
| *** mat128 has joined #openstack-infra | 03:47 | |
| *** john-davidge has quit IRC | 03:47 | |
| *** gouthamr has quit IRC | 03:50 | |
| *** gouthamr has joined #openstack-infra | 03:52 | |
| *** mriedem has quit IRC | 03:52 | |
| *** gouthamr has quit IRC | 03:56 | |
| *** Julien-zte has joined #openstack-infra | 04:05 | |
| *** ykarel has joined #openstack-infra | 04:06 | |
| *** yolkfull_ has joined #openstack-infra | 04:07 | |
| *** yolkfull has quit IRC | 04:10 | |
| *** yolkfull__ has joined #openstack-infra | 04:10 | |
| *** yolkfull_ has quit IRC | 04:12 | |
| *** jcoufal has joined #openstack-infra | 04:13 | |
| *** amotoki has joined #openstack-infra | 04:14 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 04:20 |
| *** jcoufal has quit IRC | 04:20 | |
| *** esikachev has joined #openstack-infra | 04:26 | |
| *** mat128 has quit IRC | 04:27 | |
| *** jaypipes has quit IRC | 04:29 | |
| *** hongbin_ has quit IRC | 04:29 | |
| *** Sukhdev has joined #openstack-infra | 04:30 | |
| *** esikachev has quit IRC | 04:31 | |
| *** aviau has quit IRC | 04:32 | |
| *** aviau has joined #openstack-infra | 04:32 | |
| *** thorst has joined #openstack-infra | 04:41 | |
| *** jaypipes has joined #openstack-infra | 04:41 | |
| *** thorst has quit IRC | 04:45 | |
| *** salv-orlando has joined #openstack-infra | 04:47 | |
| *** armax has joined #openstack-infra | 04:48 | |
| *** bhavik1 has joined #openstack-infra | 04:48 | |
| *** esikachev has joined #openstack-infra | 04:52 | |
| *** salv-orlando has quit IRC | 04:52 | |
| *** esikachev has quit IRC | 04:56 | |
| *** zhurong has joined #openstack-infra | 05:00 | |
| *** xyang1 has quit IRC | 05:02 | |
| *** dimtruck is now known as zz_dimtruck | 05:03 | |
| *** gyee has quit IRC | 05:03 | |
| *** zz_dimtruck is now known as dimtruck | 05:03 | |
| ianw | well this turns out to be a right pickle | 05:04 |
| *** beslemon has joined #openstack-infra | 05:08 | |
| *** dimtruck is now known as zz_dimtruck | 05:12 | |
| *** erlon has quit IRC | 05:13 | |
| *** Hal has joined #openstack-infra | 05:13 | |
| *** Sukhdev has quit IRC | 05:13 | |
| openstackgerrit | Merged openstack-dev/cookiecutter master: Remove testsenarios from test-requirements.txt https://review.openstack.org/450279 | 05:24 |
| *** prateek has joined #openstack-infra | 05:24 | |
| *** Hal has quit IRC | 05:24 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 05:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move jenkins cache to /opt/cache https://review.openstack.org/460835 | 05:28 |
| ianw | pabelanger: ^ i think that solves the case of the mystery /home/jenkins. http://paste.openstack.org/show/608278/ | 05:29 |
| ianw | in the mean time, i've discovered a bunch of python3 issues, and a bunch of dead code ... all of which i'll clean up another day. that just about fried my brain :/ | 05:29 |
| *** zul has quit IRC | 05:34 | |
| *** zul has joined #openstack-infra | 05:35 | |
| *** salv-orlando has joined #openstack-infra | 05:48 | |
| *** salv-orlando has quit IRC | 05:53 | |
| *** esikachev has joined #openstack-infra | 05:55 | |
| *** lamt has quit IRC | 05:56 | |
| *** yuanying_ has quit IRC | 05:57 | |
| *** sree has joined #openstack-infra | 06:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Move jenkins cache to /opt/cache https://review.openstack.org/460835 | 06:01 |
| *** zhurong has quit IRC | 06:04 | |
| *** adrian_otto has joined #openstack-infra | 06:05 | |
| *** beslemon has quit IRC | 06:08 | |
| *** andreas_s has joined #openstack-infra | 06:08 | |
| *** tjones has quit IRC | 06:09 | |
| *** zhurong has joined #openstack-infra | 06:12 | |
| *** adrian_otto has quit IRC | 06:15 | |
| *** kzaitsev_ws has joined #openstack-infra | 06:19 | |
| *** Julien-zte has quit IRC | 06:22 | |
| *** zhurong has quit IRC | 06:23 | |
| *** Julien-zte has joined #openstack-infra | 06:24 | |
| *** kzaitsev_ws has quit IRC | 06:28 | |
| *** yuanying has joined #openstack-infra | 06:36 | |
| *** matrohon has quit IRC | 06:36 | |
| *** pcaruana has joined #openstack-infra | 06:36 | |
| *** yolkfull_ has joined #openstack-infra | 06:37 | |
| *** ccamacho has joined #openstack-infra | 06:40 | |
| *** yolkfull__ has quit IRC | 06:40 | |
| *** yolkfull has joined #openstack-infra | 06:40 | |
| *** thorst has joined #openstack-infra | 06:42 | |
| *** yuanying_ has joined #openstack-infra | 06:42 | |
| *** yuanying has quit IRC | 06:42 | |
| *** yolkfull_ has quit IRC | 06:43 | |
| *** yanyanhu has quit IRC | 06:45 | |
| *** e0ne has joined #openstack-infra | 06:46 | |
| *** yuanying_ has quit IRC | 06:46 | |
| *** thorst has quit IRC | 06:46 | |
| *** yuanying has joined #openstack-infra | 06:47 | |
| *** jamesmca_ has joined #openstack-infra | 06:47 | |
| *** salv-orlando has joined #openstack-infra | 06:49 | |
| AJaeger | ianw: thanks, fixed zuul ^ | 06:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Allways add zuul group to zuul user https://review.openstack.org/460850 | 06:52 |
| AJaeger | dirk, cmurphy ^ | 06:52 |
| *** jamesmca_ has quit IRC | 06:52 | |
| *** tphummel has quit IRC | 06:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Always add zuul group to zuul user https://review.openstack.org/460850 | 06:53 |
| *** rkukura has joined #openstack-infra | 06:53 | |
| *** camunoz has quit IRC | 06:54 | |
| *** salv-orlando has quit IRC | 06:54 | |
| *** shardy has joined #openstack-infra | 06:56 | |
| *** salv-orlando has joined #openstack-infra | 06:57 | |
| *** matrohon has joined #openstack-infra | 06:58 | |
| *** flepied has joined #openstack-infra | 06:59 | |
| *** andreas-f has quit IRC | 06:59 | |
| *** tie has joined #openstack-infra | 07:03 | |
| *** tesseract has joined #openstack-infra | 07:05 | |
| *** pgadiya has joined #openstack-infra | 07:11 | |
| *** ggnel_t has joined #openstack-infra | 07:17 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** e0ne has quit IRC | 07:19 | |
| *** salv-orlando has quit IRC | 07:23 | |
| *** markus_z has joined #openstack-infra | 07:30 | |
| *** jpich has joined #openstack-infra | 07:32 | |
| *** kornica has joined #openstack-infra | 07:33 | |
| *** kornica is now known as kornicameister | 07:33 | |
| *** tuanluong has quit IRC | 07:34 | |
| *** gildub has quit IRC | 07:39 | |
| *** yamamoto has quit IRC | 07:40 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** thorst has joined #openstack-infra | 07:43 | |
| *** kornica has joined #openstack-infra | 07:44 | |
| *** kornicameister has quit IRC | 07:46 | |
| *** kornicameister has joined #openstack-infra | 07:48 | |
| *** kornica has quit IRC | 07:48 | |
| *** kornicameister has quit IRC | 07:49 | |
| *** kornica has joined #openstack-infra | 07:50 | |
| *** Julien-zte has quit IRC | 07:53 | |
| *** flepied has quit IRC | 07:53 | |
| *** makowals has joined #openstack-infra | 07:54 | |
| *** kornica is now known as kornicameister | 07:54 | |
| *** armax has quit IRC | 07:55 | |
| *** sree_ has joined #openstack-infra | 07:57 | |
| *** sree_ is now known as Guest90377 | 07:57 | |
| *** kornica has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** sree has quit IRC | 08:00 | |
| *** kornicameister has quit IRC | 08:02 | |
| *** andreaf has quit IRC | 08:02 | |
| *** thorst has quit IRC | 08:02 | |
| *** andreaf has joined #openstack-infra | 08:04 | |
| *** aarefiev_afk is now known as aarefiev | 08:07 | |
| *** asselin__ has joined #openstack-infra | 08:08 | |
| *** asselin has joined #openstack-infra | 08:10 | |
| *** asselin_ has quit IRC | 08:11 | |
| *** asselin__ has quit IRC | 08:13 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin https://review.openstack.org/460506 | 08:15 |
| *** stakeda has quit IRC | 08:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Always add zuul group to zuul user https://review.openstack.org/460850 | 08:16 |
| *** dtantsur|afk is now known as dtantsur | 08:17 | |
| *** florianf has joined #openstack-infra | 08:17 | |
| *** dizquierdo has joined #openstack-infra | 08:17 | |
| *** lucas-afk is now known as lucasagomes | 08:17 | |
| *** Julien-zte has joined #openstack-infra | 08:21 | |
| *** flepied has joined #openstack-infra | 08:23 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin https://review.openstack.org/460506 | 08:28 |
| *** rossella_s has joined #openstack-infra | 08:36 | |
| *** kornica is now known as kornicameister | 08:40 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** makowals has quit IRC | 08:42 | |
| *** tuanluong has joined #openstack-infra | 08:43 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin https://review.openstack.org/460506 | 08:44 |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config master: neutron: add ovs firewall tempest job to check queue https://review.openstack.org/460409 | 08:46 |
| *** electrofelix has joined #openstack-infra | 08:46 | |
| *** Julien-zte has quit IRC | 08:48 | |
| *** jaosorior_away is now known as jaosorior | 08:49 | |
| *** Julien-zte has joined #openstack-infra | 08:50 | |
| *** mhickey has joined #openstack-infra | 08:51 | |
| *** e0ne has joined #openstack-infra | 08:52 | |
| *** thorst has joined #openstack-infra | 08:59 | |
| *** deafult has quit IRC | 09:00 | |
| *** deafult has joined #openstack-infra | 09:01 | |
| *** waynr has quit IRC | 09:01 | |
| *** yolkfull_ has joined #openstack-infra | 09:03 | |
| *** hamzy has joined #openstack-infra | 09:04 | |
| *** chenying_ has joined #openstack-infra | 09:06 | |
| *** yolkfull has quit IRC | 09:06 | |
| *** yamahata_ has quit IRC | 09:08 | |
| *** sflanigan has quit IRC | 09:08 | |
| *** chenying__ has quit IRC | 09:09 | |
| *** liusheng has quit IRC | 09:09 | |
| *** liusheng has joined #openstack-infra | 09:09 | |
| *** thorst has quit IRC | 09:10 | |
| *** john-davidge has joined #openstack-infra | 09:10 | |
| *** kornicameister has quit IRC | 09:11 | |
| *** kornica has joined #openstack-infra | 09:12 | |
| *** MasterOfBugs has quit IRC | 09:14 | |
| *** markvoelker has quit IRC | 09:16 | |
| *** odyssey4me_ is now known as odyssey4me | 09:20 | |
| openstackgerrit | Arne Recknagel proposed openstack/diskimage-builder master: Add zipl support for s390 architecture with SCSI boot https://review.openstack.org/443548 | 09:20 |
| openstackgerrit | Evgeni Golov proposed openstack-infra/jenkins-job-builder master: properly support private builds in the registry https://review.openstack.org/460889 | 09:21 |
| *** adriant_ has quit IRC | 09:23 | |
| *** Julien-zte has quit IRC | 09:24 | |
| *** dhajare has quit IRC | 09:24 | |
| openstackgerrit | Arne Recknagel proposed openstack/diskimage-builder master: Add zipl support for s390 architecture with SCSI boot https://review.openstack.org/443548 | 09:24 |
| *** Guest90377 has quit IRC | 09:28 | |
| *** sree has joined #openstack-infra | 09:30 | |
| *** sambetts|afk is now known as sambetts | 09:31 | |
| *** jtomasek has quit IRC | 09:34 | |
| *** tosky has joined #openstack-infra | 09:37 | |
| *** jtomasek has joined #openstack-infra | 09:37 | |
| openstackgerrit | Evgeni Golov proposed openstack-infra/jenkins-job-builder master: conditional_publisher: add support for Matrix Aggregation https://review.openstack.org/460898 | 09:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/460899 | 09:41 |
| *** adarazs has left #openstack-infra | 09:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Output ZUUL variables to a file https://review.openstack.org/460907 | 09:55 |
| odyssey4me | project-config cores, as is evidenced by http://logs.openstack.org/39/458839/11/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/21b9266/console.html I broke our gate job with https://review.openstack.org/458890 - I've proposed https://review.openstack.org/460907 to fix it and would appreciate prompt review | 09:56 |
| odyssey4me | clarkb ^ that's the change we discussed previously | 09:57 |
| *** sdague has joined #openstack-infra | 10:00 | |
| *** jaosorior is now known as jaosorior_away | 10:00 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Make gate-ironic-inspector-tox-func3 voting https://review.openstack.org/460914 | 10:02 |
| *** lyarwood has quit IRC | 10:03 | |
| *** nicolasbock has joined #openstack-infra | 10:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/460899 | 10:04 |
| *** tesseract has quit IRC | 10:04 | |
| *** thorst has joined #openstack-infra | 10:07 | |
| *** tesseract has joined #openstack-infra | 10:08 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Make gate-ironic-inspector-tox-func3 voting https://review.openstack.org/460914 | 10:09 |
| *** thorst has quit IRC | 10:11 | |
| *** mkebich has joined #openstack-infra | 10:11 | |
| *** markvoelker has joined #openstack-infra | 10:17 | |
| *** sree has quit IRC | 10:17 | |
| *** bhavik1 has quit IRC | 10:17 | |
| *** markvoelker has quit IRC | 10:22 | |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config master: Add docs job for openstack-helm https://review.openstack.org/460925 | 10:28 |
| *** liujiong has quit IRC | 10:41 | |
| *** jkilpatr has quit IRC | 10:42 | |
| *** mat128 has joined #openstack-infra | 10:44 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files https://review.openstack.org/460947 | 10:44 |
| *** jamesmca_ has joined #openstack-infra | 10:49 | |
| *** tuanluong has quit IRC | 10:52 | |
| *** jamesmca_ has quit IRC | 10:53 | |
| *** BobBall_AWOL is now known as BobBall | 10:54 | |
| *** dhajare has joined #openstack-infra | 10:58 | |
| *** rossella_s has quit IRC | 11:04 | |
| *** rossella_s has joined #openstack-infra | 11:05 | |
| *** dhajare has quit IRC | 11:05 | |
| odyssey4me | AJaeger to answer your question in https://review.openstack.org/#/c/460907/1/jenkins/jobs/openstack-ansible-jobs.yaml the purpose of the if is to cover the case where the ZUUL_ vars may not be set... although I'm not sure that will ever be the case in the gate? | 11:05 |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| odyssey4me | are those vars always present? | 11:05 |
| *** yamamoto_ has joined #openstack-infra | 11:06 | |
| AJaeger | odyssey4me: ZUUL_ vars are always set. Depending on the queue you run in - post, periodic, check/gate, there might be a few different variables. | 11:07 |
| *** yolkfull_ has quit IRC | 11:08 | |
| AJaeger | odyssey4me: but your code would still work if ZUUL_ is not set, doesn't it? | 11:08 |
| odyssey4me | ok, if there is always at least one ZUUL_ var, then I don't need to fence off a possible failure there | 11:08 |
| odyssey4me | if no ZUUL_ var is set, then 'printenv | grep ^ZUUL' will results in an error | 11:09 |
| odyssey4me | the grep returns a fail :) | 11:09 |
| odyssey4me | I suppose I could use awk instead, which won't | 11:09 |
| AJaeger | odyssey4me: indeed, see it. | 11:09 |
| *** makowals has joined #openstack-infra | 11:09 | |
| AJaeger | you could work around that but I would just leave it for now. | 11:09 |
| *** yamamoto has quit IRC | 11:10 | |
| *** makowals has quit IRC | 11:10 | |
| odyssey4me | I was thinking of doing this instead: printenv | grep ^ZUUL > zuul.env || true | 11:10 |
| AJaeger | odyssey4me: but if you want to test, why not use "-n ${{SOME_VAR}}" ? | 11:11 |
| odyssey4me | but I think there might be a precedence issue there which may not do exactly what's intended | 11:11 |
| *** jkilpatr has joined #openstack-infra | 11:11 | |
| odyssey4me | yeah, that could work too | 11:12 |
| odyssey4me | but it's unnecessary to cage it, so I'll remove the test entirely | 11:13 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Output ZUUL variables to a file https://review.openstack.org/460907 | 11:14 |
| *** dizquierdo has quit IRC | 11:15 | |
| *** makowals has joined #openstack-infra | 11:17 | |
| *** yamamoto_ has quit IRC | 11:18 | |
| *** shardy is now known as shardy_lunch | 11:22 | |
| *** kornicameister has joined #openstack-infra | 11:26 | |
| *** Julien-zte has joined #openstack-infra | 11:26 | |
| *** dizquierdo has joined #openstack-infra | 11:27 | |
| *** kornica has quit IRC | 11:27 | |
| *** rhallisey has joined #openstack-infra | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** psachin has quit IRC | 11:37 | |
| *** csomerville has joined #openstack-infra | 11:40 | |
| *** cody-somerville has quit IRC | 11:40 | |
| *** gildub has joined #openstack-infra | 11:44 | |
| *** thorst has joined #openstack-infra | 11:44 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: CI test - never merge https://review.openstack.org/461001 | 11:45 |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze master: fix missing space in systemd logs https://review.openstack.org/461002 | 11:45 |
| sdague | clarkb: I'm out for the day starting soon, but https://review.openstack.org/461002 and https://review.openstack.org/460947 should get looked at. | 11:46 |
| *** sree has joined #openstack-infra | 11:48 | |
| *** gordc has joined #openstack-infra | 11:49 | |
| *** tesseract has quit IRC | 11:50 | |
| *** psachin has joined #openstack-infra | 11:51 | |
| *** sree has quit IRC | 11:52 | |
| *** shardy_lunch is now known as shardy | 11:55 | |
| *** dave-mccowan has joined #openstack-infra | 11:55 | |
| *** tinwood is now known as tinwood_lunch | 11:56 | |
| *** ccamacho is now known as ccamacho|lunch | 11:57 | |
| *** psachin has quit IRC | 11:59 | |
| *** d0ugal_ has joined #openstack-infra | 12:02 | |
| *** zhenguo has quit IRC | 12:02 | |
| *** d0ugal has quit IRC | 12:04 | |
| *** rodrigopaiva has joined #openstack-infra | 12:04 | |
| *** tesseract has joined #openstack-infra | 12:05 | |
| kornicameister | if anyone has time to review python-openstackclient | 12:06 |
| kornicameister | sorry | 12:06 |
| kornicameister | https://review.openstack.org/#/c/460506/ | 12:06 |
| kornicameister | that one | 12:06 |
| kornicameister | that'd be great | 12:06 |
| *** kornicameister has quit IRC | 12:06 | |
| *** EricGonczer_ has joined #openstack-infra | 12:06 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/project-config master: TripleO: experimental container upgrade job on more repos https://review.openstack.org/461007 | 12:07 |
| *** coreycb has quit IRC | 12:09 | |
| *** lucasagomes is now known as lucas-hungry | 12:11 | |
| *** dprince has joined #openstack-infra | 12:11 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** yamamoto has joined #openstack-infra | 12:18 | |
| *** coreycb has joined #openstack-infra | 12:19 | |
| *** coreycb has quit IRC | 12:19 | |
| *** coreycb has joined #openstack-infra | 12:20 | |
| *** chlong has quit IRC | 12:21 | |
| *** d0ugal_ is now known as d0ugal | 12:21 | |
| *** coreycb_ has joined #openstack-infra | 12:21 | |
| *** d0ugal is now known as Guest8049 | 12:21 | |
| *** coreycb_ has quit IRC | 12:21 | |
| *** Guest8049 has quit IRC | 12:21 | |
| *** d0ugal_ has joined #openstack-infra | 12:22 | |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config master: Add docs job for openstack-helm https://review.openstack.org/460925 | 12:23 |
| openstackgerrit | Merged openstack-infra/project-config master: OSA: Output ZUUL variables to a file https://review.openstack.org/460907 | 12:23 |
| *** pgadiya has quit IRC | 12:24 | |
| *** Kevin_Zheng has quit IRC | 12:27 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: [Zun] Add zun-ui server release job https://review.openstack.org/460815 | 12:29 |
| *** jpena is now known as jpena|lunch | 12:32 | |
| *** markvoelker has joined #openstack-infra | 12:38 | |
| *** gcb has quit IRC | 12:40 | |
| *** trown|outtypewww is now known as trown | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:41 | |
| weshay | ianw, looking for an infra review on https://review.openstack.org/#/c/458054/ | 12:41 |
| *** mriedem has joined #openstack-infra | 12:41 | |
| leifmadsen | zxiiro-away: ah no problem, if I haven't figured it out in two weeks, I'll probably have lost interest :) | 12:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Fix CleanupWorker exception messages https://review.openstack.org/461019 | 12:44 |
| *** yamamoto has quit IRC | 12:47 | |
| *** askb has quit IRC | 12:49 | |
| *** tinwood_lunch is now known as tinwood | 12:53 | |
| *** tinwood has quit IRC | 12:54 | |
| *** sree has joined #openstack-infra | 12:54 | |
| *** tinwood has joined #openstack-infra | 12:54 | |
| *** esikachev has quit IRC | 12:57 | |
| *** mat128 has quit IRC | 12:58 | |
| EmilienM | fungi: would it be possible to remove the stable/mitaka branches for tripleo projects please? | 12:58 |
| *** mat128 has joined #openstack-infra | 13:00 | |
| *** ihrachys has joined #openstack-infra | 13:01 | |
| *** jamesdenton has joined #openstack-infra | 13:03 | |
| *** Goneri has joined #openstack-infra | 13:04 | |
| *** lamt has joined #openstack-infra | 13:04 | |
| *** andreas_s has quit IRC | 13:04 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Fix repos for promotion jobs https://review.openstack.org/460558 | 13:04 |
| *** jcoufal has joined #openstack-infra | 13:05 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci master: Do not merge: Test Ceph Jewel 10.2.7 https://review.openstack.org/460121 | 13:06 |
| *** john-davidge has quit IRC | 13:07 | |
| *** john-dav_ has joined #openstack-infra | 13:07 | |
| *** ccamacho|lunch is now known as ccamacho | 13:07 | |
| *** lucas-hungry is now known as lucasagomes | 13:08 | |
| *** lamt has quit IRC | 13:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request https://review.openstack.org/461025 | 13:13 |
| *** lamt has joined #openstack-infra | 13:14 | |
| *** dizquierdo has quit IRC | 13:15 | |
| *** mdrabe has joined #openstack-infra | 13:16 | |
| *** jaypipes is now known as leakypipes | 13:16 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Use compat names for journal files https://review.openstack.org/460947 | 13:20 |
| rm_work | Is there some log file from a gate run that shows the configuration for the $PROJECTS var? | 13:22 |
| *** gildub has quit IRC | 13:22 | |
| rm_work | Ah nm just found it, reproduce.sh | 13:23 |
| *** kgiusti has joined #openstack-infra | 13:25 | |
| *** d0ugal_ has quit IRC | 13:26 | |
| *** d0ugal has joined #openstack-infra | 13:27 | |
| *** d0ugal has quit IRC | 13:27 | |
| *** d0ugal has joined #openstack-infra | 13:27 | |
| *** bauzas is now known as bauwser | 13:27 | |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config master: Add python-octaviaclient to neutron-lbaas jobs https://review.openstack.org/461030 | 13:27 |
| *** madhuvishy has quit IRC | 13:30 | |
| *** madhuvishy has joined #openstack-infra | 13:31 | |
| * AJaeger has two changes for infra-manual to update links https://review.openstack.org/#/c/459842/ and https://review.openstack.org/#/c/459844 - could anybody review these, please? | 13:32 | |
| dmellado | AJaeger: o/ we've been seeing a lot of failures in kuryr gates | 13:33 |
| dmellado | such as in here https://review.openstack.org/#/c/459292/6 | 13:33 |
| dmellado | POST_FAILURE | 13:33 |
| dmellado | I wanted to check that out but without logs, is there any way | 13:33 |
| dmellado | or would you recommend any | 13:33 |
| AJaeger | when did those start? | 13:33 |
| dmellado | to how to debug these | 13:33 |
| dmellado | I was tracking a 'suspect' commit | 13:33 |
| dmellado | which was the latest one that affected project.config some days ago | 13:34 |
| AJaeger | So, it's always gate-install-dsvm-default-kuryr-kubernetes - it sometimes fails, sometimes works. | 13:34 |
| dmellado | https://github.com/openstack-infra/project-config/commit/a69a9880b16de3ffc2ed6bc9ddda7e2316e9424d | 13:35 |
| dmellado | that one, and also /url 1 | 13:35 |
| AJaeger | dmellado: POST_FAILUREs often happen if your tests take down the machine, e.g. no network possible. | 13:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 13:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 13:36 |
| dmellado | AJaeger: also gate-install-dsvm-dragonflow-kuryr-kubernetes-nv | 13:36 |
| dmellado | have been failing a lot | 13:36 |
| dmellado | but my question is | 13:36 |
| dmellado | isn't is weird that it only happens 'sometimes' | 13:36 |
| dmellado | ? | 13:36 |
| dmellado | should this be happing all the time or none | 13:36 |
| AJaeger | dmellado: it could happen in some cloud providers and not in others | 13:36 |
| dmellado | AJaeger: I see, so it could depend on which cloud does the job end | 13:37 |
| *** Guest24728 is now known as zeus | 13:37 | |
| *** zeus has quit IRC | 13:37 | |
| *** zeus has joined #openstack-infra | 13:37 | |
| AJaeger | So, a recheck brought this live, you can now look at logs via http://status.openstack.org/zuul/ | 13:37 |
| AJaeger | The gate job links to telnet://198.72.124.178:19885 , so I run nc (instead of telnet 198.72.124.178 19885 - and wait ;) | 13:38 |
| dmellado | I'll check that, thanks for the tip AJaeger | 13:38 |
| AJaeger | dmellado: I do not have access to other logs myself, you need one of the infra-roots for that but nobody has spoken up here in the last few hours. Perhaps one of them has a better idea | 13:38 |
| dmellado | will follow-up with this | 13:39 |
| dmellado | (or get yolanda at some point) | 13:39 |
| AJaeger | dmellado: yolanda is currently not around | 13:39 |
| dmellado | AJaeger: I know ;) | 13:39 |
| dmellado | but I guess I'll be able to reach her xD | 13:39 |
| AJaeger | dmellado: so, this job runs on ubuntu-xenial-internap-mtl01-8644644 - that's internap. See second line of output | 13:39 |
| AJaeger | would be interesting to see where the jobs run that *succeeded* | 13:40 |
| dmellado | AJaeger: also, dummy question, is there any way to log those logs? | 13:40 |
| dmellado | totally | 13:40 |
| AJaeger | dmellado: nc 198.72.124.178 19885 |tee -a mylogfile | 13:40 |
| AJaeger | dmellado: in general those are logged but only safed if the job completed | 13:41 |
| rm_work | https://review.openstack.org/#/c/461030/ would be great :) pretty trivial | 13:41 |
| AJaeger | dmellado: for the nodes used for successful runs - see the second line of log file | 13:41 |
| AJaeger | rm_work: what kind of change is that? | 13:41 |
| rm_work | project-config, adds our client to the gates so we can merge the change that adds it to our devstack plugin without the gates barfing | 13:42 |
| dmellado | yes, I see | 13:42 |
| *** EricGonc_ has joined #openstack-infra | 13:42 | |
| dmellado | I'll need to check what's up with the underneath infra | 13:42 |
| rm_work | it's a copy of a patch I did yesterday that already merged, I had only done it for Octavia but also needs to be done for Neutron-LBaaS | 13:43 |
| *** srobert has joined #openstack-infra | 13:43 | |
| *** EricGonczer_ has quit IRC | 13:43 | |
| AJaeger | rm_work: LGTM | 13:43 |
| *** jpena|lunch is now known as jpena | 13:45 | |
| *** EricGonc_ has quit IRC | 13:45 | |
| *** mat128 has quit IRC | 13:45 | |
| *** Dinesh_Bhor has quit IRC | 13:46 | |
| rm_work | thanks :) | 13:47 |
| *** jcoufal_ has joined #openstack-infra | 13:48 | |
| *** jamesmca_ has joined #openstack-infra | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** lamt has quit IRC | 13:49 | |
| *** jcoufal has quit IRC | 13:50 | |
| *** eharney has joined #openstack-infra | 13:50 | |
| *** lamt has joined #openstack-infra | 13:50 | |
| AJaeger | dmellado: heisenbug - you tail the logs and it works ;) | 13:52 |
| *** jamesmca_ has quit IRC | 13:52 | |
| *** chlong has joined #openstack-infra | 13:53 | |
| *** sree has quit IRC | 13:55 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Make the TripleO job non-voting on Mistral for now https://review.openstack.org/461039 | 13:55 |
| dmellado | lol xD | 13:56 |
| pabelanger | morning | 13:57 |
| fungi | EmilienM: any of our infra-root gerrit admins can delete branches, but i'll see if i can script up something for a mass branch deletion for you here in a bit | 13:57 |
| EmilienM | fungi: thanks, i've asked multiple times but I guess folks are just busy. Really, it's just a very few projects | 13:57 |
| AJaeger | fungi, jhesketh has done the liberty retirement - perhaps he has a script? | 13:58 |
| *** mdrabe has quit IRC | 13:58 | |
| *** baoli has joined #openstack-infra | 13:59 | |
| fungi | well, it's more just that i need to make sure i have all their deliverables cloned, check for stable/mitaka branches in them, check that there are mitaka-eol tags on them, and then try to push an empty ref... can be done in a one-liner shell loop without too much trouble. though if it's only a handful of repos maybe using the webui is easier | 14:00 |
| *** makowals has quit IRC | 14:01 | |
| fungi | EmilienM: did you send a list of which ones you want deleted to one of the mailing lists so i have a quick reference, or should i just loop through all the tripleo deliverables in governance? | 14:01 |
| *** apuimedo has joined #openstack-infra | 14:01 | |
| pabelanger | dmellado: AJaeger: POST_FAILURE usually are issues with infrastructure, however in the case of your job, it was aborted for job timeout | 14:02 |
| EmilienM | fungi: stable/mitaka branches for instack-undercloud, tripleo-heat-templates, tripleo-puppet-elements, puppet-tripleo, python-tripleoclient and tripleo-common | 14:02 |
| dmellado | pabelanger: it was a job timeout? | 14:03 |
| dmellado | hmmm | 14:03 |
| dmellado | I'm trying to nc now here | 14:03 |
| dmellado | to check what's going on | 14:03 |
| dmellado | telnet://[2001:4800:1ae1:18:f816:3eff:fecd:1bcc]:19885 | 14:03 |
| * dmellado needs to check if his ipv6 stack is on | 14:03 | |
| pabelanger | dmellado: it ran for 2h 26m 10s | 14:03 |
| dmellado | but I'd say tihs is OSIC | 14:03 |
| fungi | dmellado: `nc 2001:4800:1ae1:18:f816:3eff:fecd:1bcc 19885` | 14:03 |
| EmilienM | fungi: also, https://github.com/openstack/instack has old branches like feature/V2 and mgt-kilo. You can also remove them safely. | 14:03 |
| dmellado | fungi: as I said | 14:04 |
| AJaeger | pabelanger, dmellado: Argh, confused it with RETRY_LimiT ;/ | 14:04 |
| dmellado | sadly, no route to host | 14:04 |
| dmellado | argh | 14:04 |
| dmellado | on Ipv6 | 14:04 |
| dmellado | pabelanger: AJaeger fungi should I assume IPv6==OSIC? | 14:04 |
| fungi | dmellado: no working ipv6 maybe? i can reach it | 14:04 |
| dmellado | fungi: most probably that | 14:04 |
| fungi | dmellado: or is your isp blocking egress to 19885/tcp? | 14:04 |
| dmellado | fungi: nope, that worked just before | 14:05 |
| dmellado | fungi: would you be so kind of logging that execution? | 14:05 |
| pabelanger | AJaeger: RETRY_LIMIT is also likely infra related, so far we lost SSH access to remote node, or zuul-launcher couldn't clean up a job propelry | 14:05 |
| pabelanger | dmellado: fungi: nc WFM | 14:05 |
| fungi | dmellado: sure, i can tail that to a file locally and then upload it somewhere. right now it's executing stack.sh as of ~3.5 minutes ago | 14:06 |
| *** eharney has quit IRC | 14:06 | |
| *** claudiub has quit IRC | 14:07 | |
| *** chlong has quit IRC | 14:07 | |
| *** felipemonteiro has joined #openstack-infra | 14:07 | |
| dmellado | migt be IPv6 issue on my end | 14:08 |
| dmellado | fungi: thanks! | 14:08 |
| dmellado | we've been hitting such errors for a while on a couple of patches | 14:08 |
| dmellado | and I'd like to dig this out | 14:08 |
| *** felipemonteiro_ has joined #openstack-infra | 14:09 | |
| fungi | EmilienM: i see an ARG-PRINT-TEMPLATES branch on os-apply-config, do you want that removed too? | 14:09 |
| EmilienM | fungi: please | 14:09 |
| EmilienM | fungi: old legacy things, I wasn't born at this time. | 14:10 |
| *** jaosorior_away has quit IRC | 14:10 | |
| fungi | yeah, probably an import artifact | 14:11 |
| fungi | some topic branch someone had on the origin before moving into gerrit | 14:11 |
| dmellado | EmilienM: maybe you could find BASIC code within that branch xD | 14:11 |
| fungi | dmellado: i was hoping for algol-60 | 14:12 |
| *** felipemonteiro has quit IRC | 14:12 | |
| EmilienM | dmellado: lol | 14:12 |
| dmellado | fungi: I had to search that out | 14:12 |
| dmellado | but maybe you'd prefer logos | 14:12 |
| dmellado | I do recall playing with that drawing turtle xD | 14:13 |
| fungi | yeah, logo was an amusing language | 14:13 |
| EmilienM | fungi: once mitaka branches are removed, we can review https://review.openstack.org/#/c/460347/ | 14:13 |
| *** dizquierdo has joined #openstack-infra | 14:13 | |
| *** rbrndt has joined #openstack-infra | 14:13 | |
| fungi | EmilienM: i see that tripleo-quickstart-extras has a branch called "develop" (last updated 5 months ago) | 14:14 |
| fungi | want that added to the deletion queue too? | 14:14 |
| AJaeger | EmilienM: I'll happily +2 it once fungi gives the green light ;9 | 14:14 |
| EmilienM | weshay: ^ can we go ahead? | 14:14 |
| *** prateek has quit IRC | 14:14 | |
| EmilienM | AJaeger: thanks ! | 14:14 |
| *** ltomasbo has joined #openstack-infra | 14:15 | |
| *** yamamoto has quit IRC | 14:16 | |
| weshay | ah.. remotes/gerrit/develop look at that.. EmilienM AJaeger yes | 14:16 |
| EmilienM | fungi: ^ shoot it | 14:17 |
| weshay | thanks | 14:17 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/project-config master: Make the TripleO job non-voting on Mistral for now https://review.openstack.org/461039 | 14:17 |
| AJaeger | pabelanger: could you +2A https://review.openstack.org/460850 as well to sync useradd commands, please? | 14:17 |
| *** esikachev has joined #openstack-infra | 14:18 | |
| AJaeger | Thanks! | 14:18 |
| *** aarefiev is now known as aarefiev_afk | 14:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Move jenkins cache to /opt/cache https://review.openstack.org/460835 | 14:19 |
| *** chlong has joined #openstack-infra | 14:20 | |
| fungi | EmilienM: so i _think_ these are all the branches you want deleted... can you confirm? http://paste.openstack.org/show/608337 | 14:21 |
| *** eharney has joined #openstack-infra | 14:21 | |
| EmilienM | fungi: that is perfect to me | 14:22 |
| *** tphummel has joined #openstack-infra | 14:22 | |
| EmilienM | fungi: I still see them on github but I guess it's not synced yet? | 14:22 |
| fungi | EmilienM: i haven't deleted anything yet. i'm still prepping | 14:23 |
| fungi | EmilienM: i see there are no "mataka-eol" tags pushed to the tips of those stable/mitaka branches. do you want those tags added too? | 14:23 |
| fungi | er, "mitaka-eol" tags i mean, of course | 14:23 |
| *** beslemon has joined #openstack-infra | 14:23 | |
| *** dingyichen has quit IRC | 14:23 | |
| *** gcb has joined #openstack-infra | 14:24 | |
| EmilienM | fungi: if you can do it, that would be awesome | 14:25 |
| fungi | okay, i'll push those too | 14:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Always add zuul group to zuul user https://review.openstack.org/460850 | 14:25 |
| *** yamahata_ has joined #openstack-infra | 14:25 | |
| EmilienM | thanks | 14:25 |
| *** hongbin has joined #openstack-infra | 14:26 | |
| AJaeger | pabelanger: https://review.openstack.org/460252 can be abandoned, and https://review.openstack.org/460728 used instead - don't you think? | 14:28 |
| *** wolverineav has joined #openstack-infra | 14:28 | |
| *** adrian_otto has joined #openstack-infra | 14:30 | |
| *** rvba has quit IRC | 14:30 | |
| *** armax has joined #openstack-infra | 14:33 | |
| *** ggillies has quit IRC | 14:34 | |
| pabelanger | AJaeger: ya, just reviewing that patch now | 14:34 |
| *** ggillies has joined #openstack-infra | 14:35 | |
| *** annegentle has quit IRC | 14:36 | |
| AJaeger | thanks, pabelanger | 14:37 |
| *** booleanfour has joined #openstack-infra | 14:40 | |
| *** booleanfour has quit IRC | 14:40 | |
| fungi | EmilienM: mitaka-eol tagging and branch deletion has now been completed for official tripleo deliverable repositories. please double-check that everything looks correct to you | 14:40 |
| EmilienM | fungi: ok, I'm looking now | 14:41 |
| AJaeger | fungi: do you want to +2A https://review.openstack.org/#/c/460347/ then? | 14:41 |
| AJaeger | (once EmilienM is happy;) | 14:41 |
| EmilienM | fungi, AJaeger: it looks perfect to me | 14:41 |
| *** adrian_otto has quit IRC | 14:44 | |
| *** dmsimard is now known as dmsimard|cavern | 14:44 | |
| *** gouthamr has joined #openstack-infra | 14:45 | |
| AJaeger | cmurphy: mkdir -p ~jenkins/cache/files is done in change Ibca6867f29b257a5110cb5522a5cca3a97fa9377 - isn't it? | 14:45 |
| *** ramishra has quit IRC | 14:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy overcloud with debug https://review.openstack.org/457037 | 14:46 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - deploy overcloud with debug https://review.openstack.org/457037 | 14:46 |
| *** esikachev has quit IRC | 14:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 14:48 |
| cmurphy | AJaeger: aha, looks like | 14:48 |
| *** annegentle has joined #openstack-infra | 14:48 | |
| *** gcb has quit IRC | 14:48 | |
| *** gcb has joined #openstack-infra | 14:49 | |
| AJaeger | cmurphy, pabelanger, so do we need to remove the mkdir to fix our images? | 14:49 |
| pabelanger | AJaeger: fix our images? | 14:50 |
| AJaeger | pabelanger: building our images | 14:50 |
| cmurphy | AJaeger: I thought 460835 fixed whatever the problem was? | 14:50 |
| pabelanger | AJaeger: sorry, not sure what the issue current is. Could you explain | 14:50 |
| *** dizquierdo has quit IRC | 14:50 | |
| AJaeger | cmurphy: I'm tired and confused right now, so assume I'M wrong ;/ | 14:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Reproduce non-puppet parts from install_puppet.sh https://review.openstack.org/460713 | 14:50 |
| * AJaeger grabs a link... | 14:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo: EOLing Mitaka https://review.openstack.org/460347 | 14:50 |
| cmurphy | i don't think running mkdir -p twice will hurt | 14:50 |
| AJaeger | I see http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/jenkins-slave/install.d/20-jenkins-slave#n19 | 14:51 |
| fungi | the -p makes it idempotent, yes | 14:52 |
| *** esikachev has joined #openstack-infra | 14:52 | |
| AJaeger | mkdir -p /home/jenkins/cache; ln -sf /opt/cache/files /home/jenkins/cache/files | 14:52 |
| AJaeger | and we have also "mkdir -p ~jenkins/cache/files" | 14:52 |
| AJaeger | so, a symbolic link vs a directory | 14:52 |
| *** dprince has quit IRC | 14:53 | |
| pabelanger | ya, we should drop mkdir -p ~jenkins/cache from prepare-node. I don't think we need / want that any more. We added it to jenkins-slave, to ensure we create the home directory first with adduser | 14:53 |
| pabelanger | other wise, we get a warning | 14:53 |
| pabelanger | useradd: warning: the home directory already exists. | 14:53 |
| fungi | you _can_ tell useradd not to create the homedir | 14:54 |
| fungi | but granted, you can also rely on it to do so as long as you move any steps relying on its existence to happen afterward | 14:54 |
| EmilienM | fungi, AJaeger: thank you again | 14:54 |
| *** jamesmca_ has joined #openstack-infra | 14:54 | |
| fungi | EmilienM: you're welcome! apologies for the delay | 14:54 |
| mtreinish | clarkb, fungi, pabelanger, mordred: are you around today to help get the last 2 things publishing on firehose? https://review.openstack.org/#/q/status:open++topic:more-firehose | 14:55 |
| EmilienM | fungi: no worries at all | 14:55 |
| pabelanger | fungi: yes, that is what we are doing now in elements | 14:55 |
| mtreinish | I promise these will be the last service adds until after the summit at least | 14:55 |
| fungi | mtreinish: yes, i am. it's rising in priority for me ;_ | 14:55 |
| fungi | ;) | 14:55 |
| mtreinish | fungi: heh, ok cool :) | 14:55 |
| fungi | (along with my other cnoference-related obligations) | 14:55 |
| fungi | c-no-ference | 14:56 |
| AJaeger | fungi, that bad? ;) | 14:56 |
| cmurphy | pabelanger: I was going to run some reproduce.sh's on https://review.openstack.org/#/c/460728/ but I don't think I'll have time tonight, if you have the time and desire to do that today feel free otherwise I'll get to it over the weekend | 14:56 |
| fungi | AJaeger: at least bad enough i can't type straight anymore | 14:56 |
| cmurphy | pabelanger: also if you want to remove the prepare-node in that patch go ahead | 14:56 |
| mtreinish | I'm looking forward to the ansible callback. Because I can hopefully use that to actually see when my puppet changes are applied | 14:56 |
| mtreinish | since puppetboard isn't a thing anymore | 14:56 |
| sbezverk | pabelanger: recent changes of infra, could impact connectivity between two VMs running on the same gate job node? | 14:56 |
| pabelanger | cmurphy: sure, I don't think we're merge today, we can aim for monday | 14:57 |
| pabelanger | I'll see about running a test too | 14:57 |
| sbezverk | pabelanger: until very recently we could ping between two VMs we spawn inside of a gate job | 14:57 |
| pabelanger | sbezverk: yes, it is possible ssh host key check might fail. Do you have a log file? | 14:58 |
| dmellado | fungi: hmm I bet that telnet://[2001:4800:1ae1:18:f816:3eff:fecd:1bcc]:19885 got stuck | 14:58 |
| *** andreww has joined #openstack-infra | 14:58 | |
| dmellado | is there something suspicious on the log? | 14:58 |
| fungi | sbezverk: pabelanger: maybe recent refactoring of how we set up iptables on job nodes has impacted some connectivity between nested instances? | 14:58 |
| sbezverk | pabelanger: ssh key part is fixed but what is failing is just a regular ping.. | 14:58 |
| *** bnemec is now known as beekneemech | 14:59 | |
| fungi | dmellado: this is the last line in the console log... | 14:59 |
| pabelanger | yes, iptables setup was changed last week too | 14:59 |
| fungi | 2017-04-28 14:02:50.100084 | + /opt/stack/new/devstack-gate/devstack-vm-gate.sh:main:L729: /tmp/ansible/bin/ansible primary -f 5 -i /home/jenkins/workspace/gate-install-dsvm-default-kuryr-kubernetes/inventory -m shell -a 'cd '\''/opt/stack/new/devstack'\'' && sudo -H -u stack DSTOOLS_VERSION=0.3.0 stdbuf -oL -eL ./stack.sh 2>&1 executable=/bin/bash' | 14:59 |
| *** aarefiev_afk has quit IRC | 14:59 | |
| fungi | dmellado: so nothing output on the console for nearly an hour | 14:59 |
| AJaeger | fungi, a new telnet to that line does not give any output | 14:59 |
| AJaeger | fungi, dmellado, so might be we killed the network? | 14:59 |
| fungi | i'll see when zuul lost contact with it | 15:00 |
| sbezverk | pabelanger: so what should I do then? | 15:00 |
| pabelanger | sbezverk: which job is this? Do you have a log? | 15:00 |
| sbezverk | pabelanger: if I change iptables inside a job would it be sufficient? | 15:00 |
| pabelanger | sbezverk: lets see if we can figure out the issue first | 15:01 |
| *** xarses_ has joined #openstack-infra | 15:02 | |
| dmellado | AJaeger: could be | 15:02 |
| *** andreww has quit IRC | 15:02 | |
| sbezverk | pabelanger: http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/console.html#_2017-04-28_14_20_41_843432 | 15:03 |
| dmellado | AJaeger: fungi but what I'm interested into is, in that case, why did that happen? | 15:03 |
| fungi | dmellado: yeah, i was hoping the timing might tell us something | 15:04 |
| pabelanger | sbezverk: http://logs.openstack.org/63/461063/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/de04fbb/console.html#_2017-04-28_14_54_31_484663 is an example of ssh host key failure, you likley need https://review.openstack.org/#/c/460783/ | 15:04 |
| pabelanger | sbezverk: looking at log you linked now | 15:04 |
| *** rossella_s has quit IRC | 15:04 | |
| AJaeger | dmellado: I can't help you further on this... | 15:04 |
| *** zz_dimtruck is now known as dimtruck | 15:04 | |
| dmellado | AJaeger: np! thanks for your help in any case | 15:04 |
| *** rossella_s has joined #openstack-infra | 15:05 | |
| dmellado | I'll try to reproduce something locally | 15:05 |
| dmellado | the only difference within the working/non working that comes to my mind | 15:05 |
| dmellado | is the cloud provider and IPv4/IPv6 | 15:05 |
| dmellado | I'll check what happens exactly on those commands | 15:05 |
| fungi | dmellado: i've held the node, but it seems i get no response on its ip address | 15:06 |
| fungi | so something is taking out its network connectivity (at least ipv6 connectivity since that's all we have ingress for that provider) | 15:06 |
| *** dprince has joined #openstack-infra | 15:06 | |
| clarkb | dmellado: fungi possibly another case of disabling RAs? | 15:06 |
| fungi | certainly possible | 15:07 |
| dmellado | clarkb: RAs? | 15:07 |
| fungi | router announcements | 15:07 |
| dmellado | oh, on IPv6 | 15:07 |
| clarkb | its how you know what your ipv6 address is in some cases | 15:07 |
| fungi | i'll hop into another running instance there and see if i can reach its linklocal address | 15:07 |
| clarkb | (and its what we use in our case) | 15:07 |
| dmellado | yeah | 15:07 |
| dmellado | clarkb: thanks! | 15:07 |
| fungi | s/announcements/advertisements/ | 15:07 |
| sbezverk | pabelanger: I added ssh key workaround in this ps and it helped | 15:07 |
| clarkb | we've seen neutron related testing break that in the past | 15:07 |
| clarkb | by disabling processing of them in the linux kernel | 15:08 |
| sbezverk | pabelanger: but ironic job is constantly failing so worried to merge it.. | 15:08 |
| clarkb | sbezverk: pabelanger that looks like the job is creating an overlay network for 172.21.0.0/24 and the instance is booting on the subnode then the overlay network can't carry the traffic to it for some reaosn | 15:08 |
| clarkb | sbezverk: pabelanger perhaps the overlay isn't set up properly or routes don't exist etc | 15:08 |
| fungi | yeah, ra support is generally disabled for devices configured as routers, so that it doesn't compete with your more advanced routing protocols | 15:09 |
| pabelanger | ya, that is what I am thinking too. I don't see any mention of it in iptables logs | 15:09 |
| dmellado | clarkb: fungi hm I see, that would explain why it worked just fine on the IPv4 node | 15:10 |
| dmellado | so it looked like a 'cloudroulette' xD | 15:10 |
| fungi | looks like it might actually be breaking ipv6 entirely | 15:12 |
| fungi | i can't reach its linklocal v6 address but _can_ reach its rfc-1918 address from another instance there | 15:12 |
| sbezverk | clarkb: that job was working 100% before the last week | 15:12 |
| pabelanger | sbezverk: do you know when it started failing? | 15:12 |
| clarkb | fungi: check sysctl ra values? | 15:13 |
| pabelanger | we can see which image it was using | 15:13 |
| *** rvba has joined #openstack-infra | 15:13 | |
| *** rvba has quit IRC | 15:13 | |
| *** rvba has joined #openstack-infra | 15:13 | |
| fungi | clarkb: well, need to work out how to get myself access to it | 15:13 |
| clarkb | fungi: ssh -A through mirror? | 15:13 |
| fungi | but it's definitely not participating in neighbor discovery on its linklocal | 15:13 |
| fungi | i'll see if i can get that working | 15:14 |
| AJaeger | clarkb, pabelanger, I have a small change to remove single node lists in projects.yaml - could you put that on your review queue, please? https://review.openstack.org/459049 | 15:15 |
| clarkb | sbezverk: pabelanger ovs.txt and ovs-init.txt are both empty files. Not sure if that is used for the overlay and expected to have content | 15:16 |
| clarkb | it also looks like the logs are primarily from the primary node so we can't really see what the subnode did? | 15:16 |
| pabelanger | Hmm, is it possible this could be related to switching to centos octca from rdo ? | 15:16 |
| pabelanger | it would be good to see when it started failing | 15:17 |
| sbezverk | clarkb: internal "net" is built using linux bridges | 15:17 |
| clarkb | sbezverk: is that where 172.21.0.0/24 is being used/ | 15:17 |
| *** openstackgerrit has quit IRC | 15:17 | |
| clarkb | sbezverk: and can you share where that is being configured? | 15:18 |
| sbezverk | clarkb: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_common.sh#L32 | 15:20 |
| dmellado | clarkb: fungi let me know if I can help you debugging that in any another way that doesn't require infra-core powers | 15:21 |
| dmellado | :D | 15:22 |
| fungi | dmellado: i've gotten into it and confirmed ipv6 seems entirely broken on the node. i'm getting some details for you now | 15:22 |
| clarkb | having logs from the subnode(s) would be very helpful in this case too. I see the primary node ip listing which show br-tenants and 172.21.0.10/24 which is where we are pinging 172.21.0.110/24 so the routes are configured properly on that side at least but not sure about remote | 15:22 |
| sbezverk | clarkb: check this out just run gatecheck http://logs.openstack.org/63/461063/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/b408799/console.html#_2017-04-28_15_16_44_155620 | 15:23 |
| clarkb | sbezverk: unrelated but that script could use indentation :P | 15:23 |
| sbezverk | clarkb: for sure :) | 15:24 |
| dmellado | fungi: thanks! | 15:24 |
| clarkb | sbezverk: net2 is 172.22.0.0/24 not 172.21.0.0/24. 172.21.0.0/24 is on br-tenants | 15:24 |
| fungi | dmellado: http://paste.openstack.org/show/608343 | 15:24 |
| fungi | dmellado: even linklocal addressing is gone from ens3 now | 15:25 |
| dmellado | :\ | 15:25 |
| fungi | dmellado: it's like something completely disabled ipv6 support on this machine | 15:25 |
| sbezverk | clarkb: it seems connectivity issue is intermittent | 15:25 |
| dmellado | I'll try to spawn an ipv6 dsvm and see if I can reproduce this | 15:25 |
| dmellado | but that's quite weird! | 15:26 |
| clarkb | sbezverk: do you know where br-tenants is configured? and how it is overlaying between test nodes? | 15:26 |
| fungi | sbezverk: net.ipv6.conf.all.disable_ipv6 and net.ipv6.conf.default.disable_ipv6 are set to 1, so that's why, the bigger question is what's setting that? | 15:27 |
| fungi | dmellado: ^ | 15:28 |
| fungi | (sorry sbezverk) | 15:28 |
| dmellado | Oo | 15:28 |
| clarkb | sbezverk: as for being intermittent my guess is that it works when your ping test is to an instance on the priamry node and it fails when it is on the subnode so depends on how the scheduling happens | 15:28 |
| dmellado | fungi: I'll need to check that | 15:28 |
| fungi | dmellado: so something seems to be explicitly turning off ipv6 support for you | 15:28 |
| *** pcaruana has quit IRC | 15:28 | |
| clarkb | can't have ipv6 | 15:29 |
| clarkb | too retro | 15:29 |
| fungi | welcome to the 20th century! | 15:29 |
| *** mdrabe has joined #openstack-infra | 15:29 | |
| sbezverk | clarkb: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/deploy_iscsi_common.sh#L246 | 15:29 |
| sbezverk | clarkb: ironic job is single node job, everything happens on the same gate VM | 15:30 |
| clarkb | sbezverk: http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/console.html#_2017-04-28_13_41_51_248665 no its a two node job | 15:30 |
| clarkb | sbezverk: so thats my best guess. Its two node job. You don't actually have an overlay on that l2, just a local bridge. If thing schedules to subnode it breaks | 15:31 |
| sbezverk | clarkb: could you please clarify where you see two nodes? If it is 2 node job it is mistake somewhere as we need a signle node for it.. | 15:32 |
| clarkb | sbezverk: on the link above, see the hostname | 15:32 |
| clarkb | centos-7 2-node rax-ord NNNNNNNN | 15:32 |
| sbezverk | clarkb: oh man I thought 7-2 means centos 7.2 | 15:33 |
| sbezverk | clarkb: regardless in the job there is no logic to deal with second node. it is always uses main node.. I guess I will have to push a change to switch it to a single node.. | 15:34 |
| *** chlong has quit IRC | 15:34 | |
| dmellado | fungi: clarkb I'll try to reproduce this and get back to you, thanks for your help! | 15:35 |
| fungi | dmellado: you're welcome. i'll keep this specific node held for a little longer in case there are particular logs you want checked | 15:35 |
| clarkb | sbezverk: yes I think thats step 0. If problem persists because second node was just unused and issue is in single node too then I'm going to guess the ironic container isn't being attached to that bridge | 15:35 |
| clarkb | sbezverk: so your next step would be to list all the things attached to the bridge | 15:36 |
| *** sdague has quit IRC | 15:36 | |
| clarkb | sbezverk: I think brctl show br-tenants is what you want for doing that | 15:39 |
| clarkb | maybe also brctl show | 15:40 |
| sbezverk | clarkb: could you point where I can specify that only single node is required for the job? | 15:42 |
| *** yolanda has joined #openstack-infra | 15:43 | |
| clarkb | sbezverk: change the node in your jjb config to centos-7 instead of centos-7-2-node | 15:43 |
| *** Apoorva has joined #openstack-infra | 15:43 | |
| *** dkehn has quit IRC | 15:44 | |
| sbezverk | clarkb: got it, thanks | 15:45 |
| *** openstackgerrit has joined #openstack-infra | 15:47 | |
| openstackgerrit | Khai Do proposed openstack/python-jenkins master: Should use fullname instead of name https://review.openstack.org/459087 | 15:47 |
| *** jpena is now known as jpena|off | 15:48 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack-infra/project-config master: Switch kolla-kubernetes ironic job for single node https://review.openstack.org/461085 | 15:49 |
| clarkb | actually wait its an ovs bridge? | 15:49 |
| clarkb | so brctl wont work | 15:49 |
| clarkb | and I always have to go re remember ovs things but same idea | 15:50 |
| sbezverk | clarkb: https://review.openstack.org/461085 | 15:50 |
| *** eranrom_ has quit IRC | 15:50 | |
| sbezverk | clarkb: we noticed as soon as ironic job lands on rax-ord there is a good chance it will fail | 15:50 |
| *** eranrom has joined #openstack-infra | 15:51 | |
| sbezverk | clarkb: so hopefully once it is switched to single node, it will not land there anymore :) | 15:51 |
| dmellado | fungi: clarkb I think I found the issue | 15:51 |
| dmellado | will work on it next week | 15:51 |
| clarkb | well rax-ord is valid for all the labels | 15:51 |
| dmellado | https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/plugin.sh#L113-L116 | 15:52 |
| *** gyee has joined #openstack-infra | 15:52 | |
| *** eranrom has quit IRC | 15:53 | |
| *** eranrom has joined #openstack-infra | 15:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Rename gate-dox-docker-save to gate-dox-buildimages-nv https://review.openstack.org/461087 | 15:55 |
| clarkb | sbezverk: is the ironic host a VM? | 15:56 |
| clarkb | sbezverk: this host http://logs.openstack.org/16/461016/1/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/383e702/logs/vm-1.log ? there is no log which seems supsicious | 15:57 |
| sbezverk | clarkb: inside on VM infra gives us I create a nested VM using virsh | 15:57 |
| sbezverk | and use it as a "baremetal" node | 15:57 |
| *** mhickey has quit IRC | 15:58 | |
| clarkb | gotcha so thats the other thing I would check | 15:58 |
| clarkb | why is there no log for the vm? | 15:58 |
| *** dkehn has joined #openstack-infra | 15:59 | |
| sbezverk | clarkb: well virsh and nova clearly indicates that VM sucessfully started | 15:59 |
| clarkb | sbezverk: yup so maybe you are getting the wrong log file? | 15:59 |
| clarkb | but I bet console logs from that VM will also be useful | 15:59 |
| clarkb | you can see if it bootstrapped networking properly and such | 16:00 |
| sbezverk | clarkb: right I should be able to get those logs from /var/lib/libvirt orsomething.. | 16:00 |
| clarkb | yup | 16:00 |
| *** david-lyle has quit IRC | 16:00 | |
| clarkb | possibly even the nova api? | 16:00 |
| sbezverk | clarkb: check this out, it landed on a different node and now success :) http://logs.openstack.org/16/461016/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/5dc77b2/console.html#_2017-04-28_16_00_32_157033 | 16:02 |
| * fungi wonders if the "breaks on rax-something" is tied more to running on xen. does the job maybe rely on working nested-virt acceleration support? | 16:02 | |
| clarkb | the xml says domain type = qemu | 16:03 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates https://review.openstack.org/461090 | 16:04 |
| EmilienM | fungi, pabelanger, AJaeger: please review this one if you can ^ I broke tripleo CI | 16:04 |
| EmilienM | with the mitaka EOL I forgot to remove these 2 jobs | 16:04 |
| *** tie has quit IRC | 16:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use the security context of the playbook when checking out roles https://review.openstack.org/461092 | 16:05 |
| clarkb | fungi: sbezverk one thing I notice is that the first 6 pings fail. maybe in the rax case you aren't waiting long enough for it to all be working properly? | 16:05 |
| clarkb | also there are significantly fewer logs on the successful job so hard to compare | 16:06 |
| *** spzala has quit IRC | 16:06 | |
| fungi | EmilienM: is there corresponding cleanup for that in jenkins/jobs/projects.yaml too? | 16:06 |
| *** spzala has joined #openstack-infra | 16:06 | |
| AJaeger | EmilienM: quick suggestion | 16:06 |
| clarkb | sbezverk: maybe try a longer ping and see if it comes up | 16:06 |
| sbezverk | clarkb: right, we do not collect logs on successful run | 16:06 |
| EmilienM | fungi: no, it's a template that is used by puppet ci | 16:07 |
| *** trown is now known as trown|lunch | 16:07 | |
| fungi | EmilienM: but do you instantiate that template for the project? | 16:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates https://review.openstack.org/461090 | 16:07 |
| EmilienM | AJaeger: done | 16:07 |
| * fungi checks | 16:07 | |
| *** ltomasbo is now known as ltomasbo|away | 16:08 | |
| *** mdrabe has quit IRC | 16:08 | |
| *** ykarel has quit IRC | 16:09 | |
| *** adrian_otto has joined #openstack-infra | 16:09 | |
| EmilienM | fungi: no I don't | 16:10 |
| fungi | EmilienM: oh, i see it's a job-group you're adding | 16:10 |
| EmilienM | yeah | 16:10 |
| fungi | so we'll still be generating those jobs for now unless we create a different job-group or explicitly specify the templates to instantiate for the project | 16:11 |
| fungi | good enough | 16:11 |
| *** mhickey has joined #openstack-infra | 16:11 | |
| *** spzala has quit IRC | 16:11 | |
| EmilienM | thanks for the quick review. It should be good otherwise | 16:11 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 16:12 |
| AJaeger | could somebody +2A this project-config change to unblock monasca, please? https://review.openstack.org/460506 | 16:14 |
| *** shardy has quit IRC | 16:14 | |
| *** derekh has quit IRC | 16:15 | |
| *** mdrabe has joined #openstack-infra | 16:16 | |
| *** chlong has joined #openstack-infra | 16:16 | |
| AJaeger | project-config cores, could you review these, please? They fix existing jobs: https://review.openstack.org/458392 https://review.openstack.org/458391 https://review.openstack.org/457484 https://review.openstack.org/439840 https://review.openstack.org/459827 https://review.openstack.org/460810 https://review.openstack.org/461030 | 16:17 |
| *** esikachev has quit IRC | 16:17 | |
| *** jcoufal has joined #openstack-infra | 16:20 | |
| *** esikachev has joined #openstack-infra | 16:21 | |
| *** matrohon has quit IRC | 16:21 | |
| *** jcoufal__ has joined #openstack-infra | 16:22 | |
| *** jcoufal_ has quit IRC | 16:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable voting stage of monasca-kibana-plugin https://review.openstack.org/460506 | 16:24 |
| *** jcoufal has quit IRC | 16:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove old jobs for Tripleo Heat Templates https://review.openstack.org/461090 | 16:25 |
| *** esikachev has quit IRC | 16:26 | |
| *** eharney has quit IRC | 16:28 | |
| *** annegentle has quit IRC | 16:28 | |
| *** annegentle has joined #openstack-infra | 16:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role https://review.openstack.org/459066 | 16:31 |
| *** beslemon has quit IRC | 16:32 | |
| sbezverk | clarkb: or pabelanger: could you please ack this one https://review.openstack.org/461085 | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: WIP: Create stdlib.prepare-workspace role https://review.openstack.org/459066 | 16:32 |
| *** jcoufal has joined #openstack-infra | 16:33 | |
| *** ykarel has joined #openstack-infra | 16:34 | |
| *** jcoufal__ has quit IRC | 16:34 | |
| *** jascott1 has joined #openstack-infra | 16:36 | |
| *** JayF has joined #openstack-infra | 16:36 | |
| *** dave-mccowan has quit IRC | 16:36 | |
| JayF | Is there an easy way to unsubscribe from all reviews in gerrit, or disabling email notifications of review? | 16:36 |
| *** jpich has quit IRC | 16:37 | |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Clean up driverfixes branch jobs https://review.openstack.org/461102 | 16:37 |
| clarkb | JayF: there are a subset of reviews that gerrit doesn't let you unsubscribe from iirc. Basically events you have pushed patchsets for? | 16:38 |
| *** rossella_s has quit IRC | 16:38 | |
| *** esikachev has joined #openstack-infra | 16:38 | |
| clarkb | JayF: otherwise look in your account settings under watched projects to remove those | 16:38 |
| JayF | clarkb: ah, that's probably most of mine then. So I should just filter them to /dev/null until every patch I've touched is abandoned for filtered through? | 16:38 |
| JayF | s/for/or/ | 16:38 |
| clarkb | JayF: ya | 16:38 |
| JayF | Aight, sensible enough. Just a periodic sad coming straight into my inbox, lol | 16:39 |
| *** erikmwil_ has joined #openstack-infra | 16:41 | |
| *** eharney has joined #openstack-infra | 16:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch kolla-kubernetes ironic job for single node https://review.openstack.org/461085 | 16:42 |
| clarkb | pabelanger: fungi: simple review on osloganalyze to make systemd things happier https://review.openstack.org/#/c/461002/ much appreciated if you can review it | 16:42 |
| *** erikmwilson has quit IRC | 16:42 | |
| *** esikachev has quit IRC | 16:42 | |
| *** esikachev has joined #openstack-infra | 16:43 | |
| *** eranrom_ has joined #openstack-infra | 16:43 | |
| pabelanger | +3 | 16:44 |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 16:45 |
| *** tesseract has quit IRC | 16:45 | |
| clarkb | thanks | 16:46 |
| *** markus_z has quit IRC | 16:46 | |
| *** eranrom has quit IRC | 16:46 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Handle Branches from Launchpad in Migration https://review.openstack.org/437469 | 16:47 |
| *** jcoufal_ has joined #openstack-infra | 16:47 | |
| clarkb | mtreinish: i opened your gerrit topic thinking it would just be a couple changes :P | 16:47 |
| *** flepied has quit IRC | 16:47 | |
| *** ildikov is now known as coffee_cat | 16:47 | |
| *** ldnunes has quit IRC | 16:47 | |
| openstackgerrit | Merged openstack-infra/os-loganalyze master: fix missing space in systemd logs https://review.openstack.org/461002 | 16:47 |
| *** esikachev has quit IRC | 16:48 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 16:48 |
| *** ldnunes has joined #openstack-infra | 16:48 | |
| rm_work | project-config change could use a +A :) https://review.openstack.org/#/c/461030/ | 16:49 |
| *** david-lyle has joined #openstack-infra | 16:50 | |
| *** jcoufal has quit IRC | 16:50 | |
| *** JayF has left #openstack-infra | 16:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Do not use single item lists for nodes https://review.openstack.org/459049 | 16:50 |
| *** esikachev has joined #openstack-infra | 16:51 | |
| *** tosky has quit IRC | 16:52 | |
| *** jamesmca_ has quit IRC | 16:52 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 16:53 |
| *** lucasagomes is now known as lucas-afk | 16:54 | |
| *** jascott1_ has joined #openstack-infra | 16:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Adding periodic job running tempest in tripleo ocata release https://review.openstack.org/458054 | 16:54 |
| *** ijw has joined #openstack-infra | 16:54 | |
| clarkb | rm_work: I went ahead and approved it, but keep in mind devstack default should continue to be installing that client from pypi so I think that change is a noop? | 16:55 |
| *** ijw has quit IRC | 16:56 | |
| *** esikachev has quit IRC | 16:56 | |
| rm_work | it allows us to run a git_clone as part of our devstack script | 16:56 |
| rm_work | because it'll already be there | 16:56 |
| clarkb | yes, but you shouldn't do that for libs/clients unless libs from git is set for that lib | 16:56 |
| rm_work | hmm... following the model of another project | 16:56 |
| *** ijw has joined #openstack-infra | 16:56 | |
| clarkb | (and if its set we automagically add the project to the projects list for you) | 16:56 |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 16:57 |
| rm_work | but outside of the gate, that doesn't work | 16:57 |
| rm_work | if you set LIBS_FROM_GIT in devstack outside of a gate job, it doesn't automatically install it, all it does is fail a check | 16:57 |
| *** jascott1 has quit IRC | 16:57 | |
| clarkb | what? | 16:57 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Updating co-chair of Neutron L3 meeting https://review.openstack.org/460660 | 16:57 |
| clarkb | LIBS_FROM_GIT is entirely about do I install from git source or do I use pypi | 16:58 |
| clarkb | and then error if I detect the wrong one was used | 16:58 |
| rm_work | err, wait i think i'm mixing up the two approaches i was taking | 16:58 |
| rm_work | because i tried that first, and then scrapped it and went to this | 16:58 |
| rm_work | but I think maybe a mixture would do what you're saying | 16:58 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Validate flavor specification in config https://review.openstack.org/451875 | 16:58 |
| clarkb | and devstack-gate now automatically adds you to PROJECTS list if in LIBS_FROM_GIT | 16:59 |
| clarkb | as a convenience | 16:59 |
| rm_work | though this is the model that Barbican is using currently, I just used their pattern | 16:59 |
| *** jamesmca_ has joined #openstack-infra | 16:59 | |
| rm_work | let me look at some other projects | 16:59 |
| *** dave-mccowan has joined #openstack-infra | 16:59 | |
| *** rbrndt has quit IRC | 17:00 | |
| *** rbrndt has joined #openstack-infra | 17:00 | |
| *** rbrndt has quit IRC | 17:00 | |
| rm_work | mmm yeah I see Magnum is doing it the LIBS_FROM_GIT way | 17:00 |
| rm_work | let me try switching to that | 17:01 |
| *** thorst has quit IRC | 17:01 | |
| *** ijw has quit IRC | 17:01 | |
| *** shadow-x has joined #openstack-infra | 17:01 | |
| *** trown|lunch is now known as trown | 17:02 | |
| *** shadow-x has quit IRC | 17:02 | |
| *** jamesmca_ has quit IRC | 17:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add python-octaviaclient to neutron-lbaas jobs https://review.openstack.org/461030 | 17:03 |
| rm_work | clarkb: do you know if there is an easy way to specify a patch for a client lib if you use LIBS_FROM_GIT? | 17:03 |
| clarkb | rm_work: I don't. In the gate case being in the PROJECTS list ensures you have the correct commit checked out | 17:04 |
| *** rbrndt has joined #openstack-infra | 17:04 | |
| clarkb | rm_work: but for local stuff you'd likely have to check it out yourself locally or have some other variable to use a specific ref | 17:04 |
| *** Sukhdev has joined #openstack-infra | 17:05 | |
| *** harlowja has quit IRC | 17:06 | |
| *** jcoufal has joined #openstack-infra | 17:07 | |
| *** e0ne has quit IRC | 17:08 | |
| *** mhickey has quit IRC | 17:09 | |
| *** asselin_ has joined #openstack-infra | 17:09 | |
| *** jcoufal_ has quit IRC | 17:09 | |
| mtreinish | clarkb: heh, I did say a couple services :) | 17:10 |
| mtreinish | the 2 big ones are the same that I linked to the other day | 17:11 |
| *** asselin has quit IRC | 17:11 | |
| mtreinish | for the ansible callback and the logstash worker | 17:11 |
| mtreinish | the logstash worker needed a few patches to get everything setup though | 17:11 |
| *** yamahata_ has quit IRC | 17:11 | |
| *** thorst has joined #openstack-infra | 17:11 | |
| rm_work | clarkb: ok, I switched to using that method, though it does actually make it more difficult to get the correct version if that's what you want | 17:12 |
| rm_work | but, it's cleaner from a devstack perspective I guess | 17:12 |
| *** annegentle has quit IRC | 17:15 | |
| odyssey4me | clarkb are the reverse proxies for RDO in effect yet and ready for broader use? | 17:16 |
| rm_work | ah actually, I think I see how the patch selection works, derp | 17:17 |
| rm_work | k hopefully this is good | 17:17 |
| clarkb | odyssey4me: they are there but are opt in | 17:18 |
| clarkb | pabelanger: EmilienM ^ do you have more info/ | 17:18 |
| odyssey4me | ok awesome, how do we opt in ? | 17:18 |
| * fungi sets out on a quest for a midday repast, but shall return forthwith | 17:18 | |
| rm_work | clarkb: thanks for the direction on that, I guess I picked the wrong project to emulate for clientlib stuff :P | 17:21 |
| clarkb | odyssey4me: I thoguht pabelanger sent email about it so searching for it (without much initial luck) | 17:22 |
| rm_work | so I guess neither of those project-config patches were necessary, but they also don't hurt, so it shouldn't really matter. in both cases we would have added it to PROJECTS anyway via LIBS_FOR_GIT, now it is just explicit | 17:22 |
| rm_work | so as you say, noop | 17:23 |
| clarkb | yup | 17:23 |
| clarkb | odyssey4me: but basically you use http:// $regionmirror.openstack.org/rdo as your rdo root url | 17:24 |
| *** gcb has quit IRC | 17:24 | |
| *** asselin__ has joined #openstack-infra | 17:25 | |
| *** amotoki has quit IRC | 17:25 | |
| odyssey4me | would that perhaps be http://lists.openstack.org/pipermail/openstack-dev/2017-April/115137.html ? | 17:26 |
| pabelanger | I thought I did one for RDO too | 17:26 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n54 | 17:26 |
| pabelanger | td;dr source /etc/ci/mirror_info.sh; echo $NODEPOOL_RDO_PROXY | 17:26 |
| clarkb | oh good | 17:27 |
| clarkb | and yes I remember that now because intially the var was called _MIRROR and I asked to chagne ti to be specfiic | 17:27 |
| *** asselin_ has quit IRC | 17:28 | |
| *** electrofelix has quit IRC | 17:28 | |
| pabelanger | odyssey4me: http://lists.openstack.org/pipermail/openstack-dev/2017-April/114908.html | 17:29 |
| odyssey4me | so to use it, all we do is set the URL for the repo to http://mirror.dfw.rax.openstack.org:8080/rdo ? | 17:30 |
| odyssey4me | obviously the dns name changes per region | 17:30 |
| *** persia has quit IRC | 17:30 | |
| pabelanger | odyssey4me: ya, baseurl for repo file | 17:31 |
| pabelanger | let me find patch to tripleo that set it up | 17:31 |
| clarkb | odyssey4me: ya you can source the mirror_info.sh script and use the $NODEPOOL_RDO_PROXY var to get the data | 17:31 |
| odyssey4me | ok, sweet thanks - we're looking to speed up our CentOS builds and this may be very helpful | 17:31 |
| pabelanger | kolla is doing it with: http://git.openstack.org/cgit/openstack/kolla/tree/tools/setup_gate.sh#n57 | 17:32 |
| *** jamesdenton has quit IRC | 17:32 | |
| pabelanger | same would apply, but in ansible :) | 17:32 |
| odyssey4me | cool, thank you very much sir | 17:33 |
| pabelanger | clarkb: just checked osic mirror, also see traffic on docker proxy. Maybe next week we can work on bumping our apache_cache a little more | 17:34 |
| clarkb | pabelanger: ya we should poll freespace on /opt across the nodes and go to something like 3/4 min remaining space | 17:35 |
| *** jascott1_ has quit IRC | 17:37 | |
| *** Swami has joined #openstack-infra | 17:37 | |
| *** ijw has joined #openstack-infra | 17:37 | |
| *** jascott1 has joined #openstack-infra | 17:39 | |
| *** thorst has quit IRC | 17:41 | |
| *** ijw has quit IRC | 17:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove useless quotes for nodes https://review.openstack.org/459065 | 17:43 |
| *** erikmwil_ has quit IRC | 17:44 | |
| *** rbrndt has quit IRC | 17:44 | |
| rm_work | clarkb: ah, it's NOT a noop for neutron-lbaas | 17:47 |
| *** sambetts is now known as sambetts|afk | 17:47 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix interactions with keystoneauth from newton https://review.openstack.org/461114 | 17:47 |
| rm_work | because it doesn't seem to add it to the PROJECTS list when it's from a sub-project | 17:47 |
| *** flepied has joined #openstack-infra | 17:48 | |
| rm_work | like.. | 17:48 |
| clarkb | rm_work: what is your LIBS_FROM_GIT? | 17:48 |
| rm_work | http://logs.openstack.org/99/460499/6/check/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/1b6006c/ | 17:48 |
| rm_work | looks empty | 17:49 |
| rm_work | because | 17:49 |
| *** neiljerram has quit IRC | 17:49 | |
| rm_work | we add it in octavia's devstack plugin, not neutron-lbaas | 17:49 |
| clarkb | you have to set it on the devstack-gate side | 17:49 |
| clarkb | so that PROJECTS is set up properly | 17:49 |
| rm_work | ah, so what magnum does isn't right either? | 17:49 |
| clarkb | maybe? I haven't looked | 17:49 |
| *** yamahata_ has joined #openstack-infra | 17:49 | |
| rm_work | https://github.com/openstack/magnum/blob/master/devstack/plugin.sh#L19 | 17:50 |
| rm_work | sorry was trying to pull that up | 17:50 |
| rm_work | they add it to LIBS_FROM_GIT in their plugin.py | 17:50 |
| rm_work | err | 17:50 |
| rm_work | plugin.sh | 17:50 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/nodepool.yaml#n111 is how we do it in nodepool | 17:51 |
| rm_work | so we need to do it in... our gate_hook? | 17:51 |
| rm_work | oh, so the same place I made the project-config change before | 17:51 |
| rm_work | just a different thing | 17:51 |
| clarkb | no you have to do it before anything else runs eg at top level so that PROJECTS is set properly and the code is checked out | 17:51 |
| clarkb | gate_hook is too late as that has already checked out code | 17:51 |
| rm_work | augh k | 17:51 |
| rm_work | so another project-config change | 17:51 |
| clarkb | also you likely don't want to ahrdcode libs from git in your devstack plugin because otherwise the only way to install is from source | 17:52 |
| rm_work | IMO If you're using our plugin ... that's fine | 17:52 |
| rm_work | but | 17:52 |
| * rm_work shrugs | 17:52 | |
| clarkb | rm_work: the reason its not that way by default is opeople running your software do it from releases | 17:52 |
| clarkb | so the idea was test that by default and make sure it works, then possibly depending on need have a separate test that does the source install to check that HEAD is fine | 17:53 |
| rm_work | hmm | 17:53 |
| rm_work | alright, well let me get that project-config change done then | 17:53 |
| rm_work | again >_> | 17:53 |
| *** esikachev has joined #openstack-infra | 17:53 | |
| *** jamesmca_ has joined #openstack-infra | 17:54 | |
| openstackgerrit | Ashwin Agate proposed openstack-infra/project-config master: Enable monasca-transform dvsm functional tests https://review.openstack.org/460822 | 17:54 |
| rm_work | clarkb: wait are you sure we can't do it in the gate_hook? | 17:54 |
| clarkb | rm_work: PROJECTS determines what code is checked out which includes your gate_hook | 17:55 |
| rm_work | we're editing devstack config and LIBS_FROM_GIT there already | 17:55 |
| clarkb | if you set it in gate_hook well your gate_hook won't even exist necessarily | 17:55 |
| clarkb | oh we might do it wrong with nodepool too let me find the canonical example which is oslo | 17:55 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/tests/contrib/gate_hook.sh#L18-L23 | 17:55 |
| rm_work | I know those work | 17:55 |
| clarkb | yes because they only affect devstack | 17:55 |
| rm_work | and LIBS_FROM_GIT below | 17:55 |
| clarkb | what I'm talking about is setting a thing that affects devstack and devstack-gate | 17:56 |
| rm_work | hmm | 17:56 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n173 bah of course its a different var name but there it is | 17:56 |
| *** atarakt has quit IRC | 17:57 | |
| clarkb | setting ^ translates into PROJECTS and LIBS_FROM_GIT being set appropriately | 17:57 |
| rm_work | ok so | 17:57 |
| *** e0ne has joined #openstack-infra | 17:57 | |
| rm_work | I should set DEVSTACK_PROJECT_FROM_GIT=python-octaviaclient | 17:57 |
| rm_work | in the project-config job for us | 17:57 |
| rm_work | can-do | 17:58 |
| rm_work | how about: | 17:58 |
| rm_work | export DEVSTACK_PROJECT_FROM_GIT="python-octaviaclient $DEVSTACK_PROJECT_FROM_GIT" | 17:58 |
| rm_work | is that safer? | 17:59 |
| *** rkukura has quit IRC | 17:59 | |
| *** flepied has quit IRC | 17:59 | |
| *** atarakt has joined #openstack-infra | 18:00 | |
| *** atarakt has quit IRC | 18:01 | |
| clarkb | ya either way should work | 18:01 |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config master: Use DEVSTACK_PROJECT_FROM_GIT for python-octaviaclient https://review.openstack.org/461116 | 18:01 |
| rm_work | clarkb: see if that is what you like ^^ | 18:01 |
| clarkb | I don't expect anything in a calling context to set it too | 18:01 |
| rm_work | k | 18:01 |
| *** atarakt has joined #openstack-infra | 18:01 | |
| clarkb | oh except the format is comma delimited? | 18:01 |
| clarkb | beacuse I think thats how LIBS_FROM_GIT is | 18:02 |
| rm_work | ah is it | 18:02 |
| rm_work | lol k | 18:02 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Clean up driverfixes branch jobs https://review.openstack.org/461102 | 18:02 |
| *** aspiers has quit IRC | 18:02 | |
| rm_work | whatever, fine | 18:02 |
| rm_work | won't try to re-use | 18:02 |
| rm_work | currently trying to parse what the branch override thing below it is doing | 18:02 |
| *** trown is now known as trown|brb | 18:02 | |
| rm_work | do you think that's *to allow* for us overriding the branch, or to prevent it? | 18:02 |
| *** thorst has joined #openstack-infra | 18:03 | |
| AJaeger | fungi, clarkb, pabelanger I have two small changes for infra-manual to update links https://review.openstack.org/#/c/459842/ and https://review.openstack.org/#/c/459844 - could anybody review these, please? | 18:03 |
| rm_work | or tldr, do I need to add that too | 18:03 |
| clarkb | rm_work: branch override is that you can test eg master $client against newton cloud | 18:04 |
| clarkb | rm_work: its a way of constructing tests to ensure your client doesn't regress against old clouds | 18:04 |
| rm_work | ok | 18:04 |
| rm_work | so | 18:04 |
| rm_work | what is THAT block doing | 18:04 |
| rm_work | un-doing the branch override? | 18:04 |
| rm_work | or actually fixing it to do it correctly | 18:05 |
| *** ijw has joined #openstack-infra | 18:05 | |
| clarkb | rm_work: can you link to the specific block? | 18:05 |
| rm_work | line 175-179 | 18:05 |
| rm_work | https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n175 | 18:05 |
| *** neiljerram has joined #openstack-infra | 18:06 | |
| clarkb | rm_work: you need it if you are doing the thing just above it with BRANCH_OVERRIDE | 18:06 |
| *** yolanda has quit IRC | 18:07 | |
| clarkb | rm_work: so BRANCH_OVERRIDE=stable/newton will apply to cloud but then you can get master of lbaasclient | 18:07 |
| rm_work | which we do | 18:07 |
| clarkb | otherwise it will newton both of them | 18:07 |
| rm_work | ok | 18:07 |
| rm_work | so probably we want that? | 18:07 |
| clarkb | ya I think so | 18:07 |
| rm_work | oh except i don't need all of that code | 18:08 |
| rm_work | because it's a specific project :P | 18:08 |
| rm_work | so i don't need to detect it | 18:08 |
| rm_work | just need to figure out what exactly it's doing to the formatting | 18:09 |
| rm_work | PYTHON_OCTAVIACLIENT | 18:09 |
| rm_work | cool | 18:09 |
| rm_work | export OVERRIDE_PYTHON_OCTAVIACLIENT_PROJECT_BRANCH=$ZUUL_BRANCH | 18:10 |
| rm_work | is actually what I need then | 18:10 |
| clarkb | ya | 18:10 |
| rm_work | what will $ZUUL_BRANCH be | 18:10 |
| rm_work | you know what | 18:10 |
| rm_work | I assume this will work | 18:10 |
| *** ijw has quit IRC | 18:11 | |
| rm_work | ok | 18:12 |
| openstackgerrit | Adam Harwell proposed openstack-infra/project-config master: Use DEVSTACK_PROJECT_FROM_GIT for python-octaviaclient https://review.openstack.org/461116 | 18:12 |
| rm_work | clarkb: ^^ would you be so kind as to take a gander? :) | 18:12 |
| rm_work | appreciate the direction | 18:12 |
| *** neiljerram has quit IRC | 18:13 | |
| rm_work | getting devstack+gates to play nicely always feels a bit like a carnival smoke and mirrors funhouse | 18:13 |
| clarkb | ZUUL_BRANCH is set by zuul to indicate to the test env what branch the change under test belongs to | 18:13 |
| *** BobBall has quit IRC | 18:14 | |
| clarkb | rm_work: I think thats because we haev a lot of wheel reinventing going on | 18:14 |
| clarkb | so any copy pasta inherits that wheels shape | 18:14 |
| rm_work | yeah... | 18:14 |
| clarkb | side effect of pushing more and more of the work out to the laeves... | 18:15 |
| *** yolanda has joined #openstack-infra | 18:16 | |
| rm_work | alright, guess I'll wait for that to merge now | 18:16 |
| *** rbrndt has joined #openstack-infra | 18:17 | |
| *** dprince has quit IRC | 18:19 | |
| *** jamesmca_ has quit IRC | 18:20 | |
| *** jamesmca_ has joined #openstack-infra | 18:21 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade master: Add in a bunch of TODOs about interface=admin https://review.openstack.org/460790 | 18:23 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 18:24 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 18:24 |
| *** david-lyle_ has joined #openstack-infra | 18:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request https://review.openstack.org/461025 | 18:25 |
| *** david-lyle has quit IRC | 18:26 | |
| rm_work | ok, could use +A then on this patch which should be "correct": https://review.openstack.org/#/c/461116/ | 18:27 |
| *** trown|brb is now known as trown | 18:29 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add support for specifying key-name per label https://review.openstack.org/455464 | 18:30 |
| *** gordc has quit IRC | 18:33 | |
| *** gordc has joined #openstack-infra | 18:33 | |
| *** gordc has quit IRC | 18:34 | |
| *** gordc has joined #openstack-infra | 18:36 | |
| *** ihrachys_ has joined #openstack-infra | 18:36 | |
| *** gordc has quit IRC | 18:36 | |
| *** gordc has joined #openstack-infra | 18:36 | |
| * fungi hath returned | 18:37 | |
| *** ihrachys has quit IRC | 18:38 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move makemergeritem into model https://review.openstack.org/460714 | 18:38 |
| *** harlowja has joined #openstack-infra | 18:38 | |
| *** andreww has joined #openstack-infra | 18:39 | |
| *** funzo has quit IRC | 18:40 | |
| *** john-dav_ has quit IRC | 18:41 | |
| *** funzo has joined #openstack-infra | 18:41 | |
| * mordred hands fungi a fatted calf | 18:42 | |
| Shrews | "and there was much rejoicing" | 18:43 |
| *** xarses has quit IRC | 18:44 | |
| *** odyssey4me has quit IRC | 18:44 | |
| *** dpaterson_ has joined #openstack-infra | 18:45 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update git-review link https://review.openstack.org/459842 | 18:45 |
| *** odyssey4me has joined #openstack-infra | 18:46 | |
| *** jamesmc__ has joined #openstack-infra | 18:48 | |
| *** jamesmca_ has quit IRC | 18:48 | |
| fungi | Shrews wins by conjuring images of mpfc in my brain | 18:48 |
| *** rbrndt has quit IRC | 18:49 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Use https in many more places https://review.openstack.org/459844 | 18:49 |
| *** ijw has joined #openstack-infra | 18:51 | |
| *** ykarel has quit IRC | 18:52 | |
| pabelanger | fungi: clarkb: where did we end up with maybe having AFS in vexxhost too? The email about sahara artifacts just me thinking about it again | 18:57 |
| *** _ari_ is now known as _ari_|gone | 18:57 | |
| clarkb | I think we should continue to work through hosting there too, but I have no idea how practical it will be to host in another country if we can't do a few states over with afs | 18:58 |
| clarkb | I guess that is mostly determined by volume size? | 18:58 |
| *** e0ne has quit IRC | 18:58 | |
| fungi | pabelanger: i think we were hoping to meet up with mnaser in boston to hash out details on moving the static.o.o volumes there, and the amout of space sahara's asking for seems somewhat reasonable for tarballs.o.o (though moving tarballs.o.o into afs also makes sense) | 18:59 |
| *** rodrigopaiva has quit IRC | 18:59 | |
| clarkb | oh right talking about straight up move not dual homing | 18:59 |
| mordred | yah | 18:59 |
| clarkb | (sorry distracted by cinder lvm things) | 18:59 |
| mordred | it's just the WAN sync that sucked | 18:59 |
| mnaser | i'm free to talk at boston, perhaps we could setup a little email thread to find a good time like we did at austin :) | 19:00 |
| pabelanger | it would also be good to discuss what a job artifact site would look like too, for temporary things too. Today we upload them to tarballs.o.o, but don't delete them when no longer needed | 19:01 |
| *** ihrachys_ is now known as ihrachys | 19:02 | |
| *** rodrigopaiva has joined #openstack-infra | 19:02 | |
| fungi | well, we mostly rely on jobs being designed in such a way that they continually replace artifacts that don't need to persist indefinitely | 19:02 |
| pabelanger | mordred: also, did you want to try upload vhd-utils again for xenial PPA? Might start work on nodepool-builder.o.o for xenial this weekend | 19:02 |
| pabelanger | fungi: this would be for the deb packaging pipeline, and even docker images | 19:02 |
| pabelanger | because we need to pass untrusted to trusted workers | 19:03 |
| *** Goneri has quit IRC | 19:03 | |
| openstackgerrit | Merged openstack-infra/shade master: Set interface=admin for keystonev2 keystone tests https://review.openstack.org/460784 | 19:04 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 19:05 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 19:05 |
| *** david-lyle_ is now known as david-lyle | 19:09 | |
| fungi | pabelanger: specifically for deb packages and docker images which don't need to persist indefinitely? | 19:10 |
| fungi | can't we do the same thing we do with branch tarballs in that case and just replace them? | 19:10 |
| fungi | rename to a common filename when stashing temporarily, rename back to whatever specific filename they need when the trusted worker job operates on them? | 19:12 |
| pabelanger | fungi: today we upload the artifacts by zuul_commit hash, so we know which one to download on trusted node. But maybe we need to optimize that better | 19:12 |
| fungi | if we're integrating an ephemeral file storage into zuul then we need to integrate an ephemeral file cleanup into zuul along with it | 19:12 |
| fungi | if we're not integrating ephemeral file storage into zuul, then we should either design a self-cleaning storage platform or replace files on each upload | 19:13 |
| fungi | (noting that file-replacing is also racy if you have any concurrent generators/consumers | 19:15 |
| fungi | ) | 19:15 |
| *** xyang1 has joined #openstack-infra | 19:15 | |
| *** EricGonczer_ has joined #openstack-infra | 19:20 | |
| *** florianf has quit IRC | 19:20 | |
| *** rodrigopaiva has quit IRC | 19:22 | |
| rm_work | plug for a project-config +A: https://review.openstack.org/#/c/461116/ | 19:23 |
| mtreinish | pabelanger: is this something related to the ruby/puppet removal from the test nodes?: http://logs.openstack.org/70/460570/1/gate/gate-openstackci-beaker-centos-7/2c3fa2d/console.html#_2017-04-28_19_18_37_687084 | 19:23 |
| *** mhickey has joined #openstack-infra | 19:25 | |
| *** fandi has joined #openstack-infra | 19:25 | |
| cmurphy | mtreinish: my first guess is that a gem changed its dependencies | 19:25 |
| *** EricGonczer_ has quit IRC | 19:27 | |
| *** erikmwilson has joined #openstack-infra | 19:27 | |
| dmsimard|cavern | project-config cores, still looking for easy jjb reviews on https://review.openstack.org/#/c/459913/ .. thanks | 19:28 |
| fungi | mtreinish: ooh, fun given that we don't do any centos-based stuff in those patches | 19:29 |
| mtreinish | cmurphy: hmm, I can check if there was a new release recently | 19:30 |
| fungi | yeah, centos 7 is probably running an older ruby release than that or something | 19:30 |
| mtreinish | cmurphy: the last release was 0.3.0 on Feb 16, 2017 | 19:31 |
| fungi | Package ruby-devel.x86_64 0:2.0.0.648-29.el7 will be installed | 19:31 |
| cmurphy | i don't think we ever had ruby 2.3 on centos 7, even in epel, so i don't think we could have accidentally taken it away | 19:31 |
| fungi | yeah, we're still installing ruby | 19:31 |
| *** florianf has joined #openstack-infra | 19:32 | |
| fungi | something probably upped its dependency on that gem to a version which needs newer ruby | 19:32 |
| fungi | many possible layers of indirection here | 19:32 |
| cmurphy | has it been broken since february and we didn't notice? | 19:32 |
| fungi | no, these changes just passed the same jobs in check | 19:33 |
| fungi | within the past 24 hours | 19:33 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Added query for bug 1687044 https://review.openstack.org/461121 | 19:33 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 19:33 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands https://review.openstack.org/460518 | 19:34 |
| mtreinish | :( all my firehose changes are failing | 19:34 |
| *** rbrndt has joined #openstack-infra | 19:36 | |
| *** adrian_otto has quit IRC | 19:37 | |
| cmurphy | something just added xmlrpc as a dependency | 19:37 |
| cmurphy | http://logs.openstack.org/70/460570/1/check/gate-openstackci-beaker-centos-7/9fc5806/console.html doesn't install it | 19:37 |
| mtreinish | this is where what-broke.py but for gems whould be useful :) | 19:38 |
| cmurphy | none of the gems listed in http://logs.openstack.org/70/460570/1/gate/gate-openstackci-beaker-centos-7/2c3fa2d/console.html have different versions so it's one of the gems that would have been installed after that broken point | 19:38 |
| clarkb | I love how its really what released but essentially ehat broke | 19:39 |
| *** ijw has quit IRC | 19:39 | |
| cmurphy | everything is terrible | 19:39 |
| mtreinish | heh | 19:39 |
| fungi | proof that the python package ecosystem is not the only one with these sorts of problems ;) | 19:40 |
| fungi | it's immediately after the message which says | 19:41 |
| fungi | "Installing docker-api 1.33.4" | 19:41 |
| fungi | hrm, release date on that is april 10 | 19:41 |
| fungi | so another dead end | 19:41 |
| *** ijw has joined #openstack-infra | 19:41 | |
| *** ijw has quit IRC | 19:47 | |
| cmurphy | found it https://github.com/fog/fog-core/commit/ffcce1066602b41b603675b751650d427bfec576 | 19:49 |
| *** flepied has joined #openstack-infra | 19:49 | |
| cmurphy | 3 hours ago | 19:49 |
| * fungi sounds the fog horn | 19:49 | |
| *** thorst has quit IRC | 19:49 | |
| *** thorst has joined #openstack-infra | 19:50 | |
| fungi | what did you do to track that down? i was going through the add_dependency lines in puppet-openstack_infra_spec_helper.gemspec, digging down through their runtime dep crosslinks on the rubygems site and looking for any with xmlrpc | 19:51 |
| fungi | hoping to hear there's a "faster" way ;) | 19:51 |
| *** thorst_ has joined #openstack-infra | 19:51 | |
| *** annegentle has joined #openstack-infra | 19:51 | |
| openstackgerrit | Merged openstack-infra/shade master: Add in a bunch of TODOs about interface=admin https://review.openstack.org/460790 | 19:52 |
| cmurphy | fungi: i have ruby2.3 on tumbleweed so i just ran bundle install and compared the list to the last successful beaker run | 19:52 |
| fungi | ahh | 19:52 |
| fungi | that makes much more sense ;) | 19:52 |
| cmurphy | it doesn't entirely make sense though, it shouldn't have triggered that dependency https://github.com/fog/fog-core/blob/master/fog-core.gemspec#L24 | 19:53 |
| fungi | cmurphy: am i reading this right? https://github.com/fog/fog-core/blob/v1.44.0/fog-core.gemspec#L24 | 19:53 |
| *** tphummel has quit IRC | 19:53 | |
| *** sree has joined #openstack-infra | 19:53 | |
| fungi | looks like it would only add xmlrpc if the installed ruby interpreter is 2.4 or later? | 19:53 |
| cmurphy | yeah that is confusing | 19:53 |
| fungi | looks like we both ended up there at the same time | 19:54 |
| *** thorst has quit IRC | 19:54 | |
| cmurphy | xmlrpc installed for me using ruby 2.3 | 19:54 |
| cmurphy | so i think maybe spec.add_dependency doesn't work like they think it does | 19:55 |
| fungi | also that line definitely appeared between 1.43.0 and 1.44.0 | 19:55 |
| fungi | so it's pretty certain to be the trigger | 19:55 |
| *** thorst_ has quit IRC | 19:56 | |
| fungi | do we need to add a skip for fog!=1.44.0 in puppet-openstack_infra_spec_helper? | 19:56 |
| *** sree has quit IRC | 19:57 | |
| fungi | cmurphy: https://github.com/fog/fog-core/issues/206 seems to confirm | 19:58 |
| fungi | safety in numbers! | 19:59 |
| cmurphy | excellent | 19:59 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core https://review.openstack.org/461141 | 19:59 |
| pabelanger | and back | 20:00 |
| pabelanger | had to run out for beer errand | 20:00 |
| fungi | pabelanger: just in time to help us polish rubies | 20:00 |
| fungi | cmurphy: i take it < is preferable to != in this scenario (or do specfiles even have skips?) | 20:01 |
| rcarrillocruz | dmsimard|cavern: heya, merged https://github.com/ansible/ansible/pull/24122 | 20:01 |
| rcarrillocruz | pabelanger: has the infra dinner been settled yet? | 20:01 |
| cmurphy | fungi: hmm i just copied the one above it, i'm not entirely sure | 20:01 |
| rcarrillocruz | i can do any day except thursday, flying back that day | 20:01 |
| * rcarrillocruz should probably just respond to that email | 20:02 | |
| pabelanger | rcarrillocruz: not yet, but Monday / Wednesday seems to be the front runners | 20:02 |
| pabelanger | I too leave on thursday | 20:02 |
| rcarrillocruz | weeeee | 20:02 |
| *** Apoorva has quit IRC | 20:02 | |
| fungi | hah, three reviewers just independently suggested linking the github issue on that change ;) | 20:02 |
| rcarrillocruz | i'm in then | 20:02 |
| *** jcoufal_ has joined #openstack-infra | 20:03 | |
| cmurphy | fungi: looks like != is possible, that's probably better https://www.devalot.com/articles/2012/04/gem-versions.html | 20:03 |
| pabelanger | rcarrillocruz: Yay, going to search this weekend for eats and reply back to thread | 20:03 |
| rcarrillocruz | \o/ thx sie | 20:03 |
| rcarrillocruz | sir too | 20:03 |
| fungi | cmurphy: our tests should tell. and given that the intent in their code seems to have been counter to this behavior and there's already an open issue odds are 1.44.1 won't have this problem | 20:04 |
| mtreinish | rcarrillocruz: ooh, you're the perfect person to review my ansible firehose patch: https://review.openstack.org/#/c/458248/ :) | 20:04 |
| * rcarrillocruz flees | 20:04 | |
| rcarrillocruz | :P | 20:04 |
| mtreinish | rcarrillocruz: I have the plugin there also submitted as a pr on ansible proper, but I need to do some work to fix dmsimard|cavern's comments on that PR | 20:04 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core https://review.openstack.org/461141 | 20:04 |
| cmurphy | fungi: agree ^ | 20:05 |
| * rcarrillocruz totally forgot all puppet he once knew | 20:05 | |
| fungi | cmurphy: thanks! | 20:05 |
| mtreinish | rcarrillocruz: heh, you can concentrate on the ansible/python bits :) | 20:05 |
| *** jcoufal has quit IRC | 20:05 | |
| fungi | rcarrillocruz: i'll happily approve take responsibility for reviewing the puppet bits on tat one, it's just the ansible-isms i wasn't so sure about | 20:06 |
| * fungi is typing sideways apparently. that made next to no sense | 20:06 | |
| mtreinish | fungi: to be fair like nothing in the callback plugin api was documented. I managed to get that working by copy and paste/trial and error | 20:06 |
| *** thorst has joined #openstack-infra | 20:07 | |
| dmsimard|cavern | mtreinish: ++ | 20:08 |
| dmsimard|cavern | tell me about reverse engineering ansible internals | 20:08 |
| *** beslemon has joined #openstack-infra | 20:09 | |
| *** jamesmc__ has quit IRC | 20:09 | |
| *** Apoorva has joined #openstack-infra | 20:09 | |
| *** jamesmca_ has joined #openstack-infra | 20:10 | |
| *** dave-mccowan has quit IRC | 20:10 | |
| fungi | cmurphy: latest patchset just passed the previously broken job, approving! | 20:12 |
| rcarrillocruz | yah | 20:12 |
| cmurphy | woohoo \o/ | 20:12 |
| rcarrillocruz | actually i think the docs say 'hey, check it out other plugins to see how it works to write your own' | 20:12 |
| rcarrillocruz | :D | 20:12 |
| mtreinish | rcarrillocruz: heh, yeah I think you're right | 20:12 |
| mtreinish | the fun thing was none of the packaged plugins did anything the same way | 20:13 |
| rcarrillocruz | i don't spot anything bad, so +2 | 20:13 |
| fungi | mtreinish: isn't that usually what happens with undocumented interfaces? | 20:13 |
| mtreinish | yep | 20:13 |
| * rcarrillocruz goes open an amber beer now | 20:13 | |
| mtreinish | rcarrillocruz: cool, thanks | 20:13 |
| fungi | rcarrillocruz: thanks, and happy beer! | 20:13 |
| rcarrillocruz | ++ | 20:13 |
| fungi | can't wait to hang out some in boston | 20:14 |
| rcarrillocruz | indeed | 20:14 |
| rcarrillocruz | sucks i missed PTG :/ | 20:14 |
| mtreinish | rcarrillocruz: oh, you're gonna be in boston, cool. | 20:14 |
| rcarrillocruz | yah | 20:14 |
| rcarrillocruz | thx to robyn :-) | 20:14 |
| rcarrillocruz | i'll give a talk on ansible day | 20:14 |
| fungi | should convince him to attend the firehose talk ;) | 20:14 |
| rcarrillocruz | talk i have not even started | 20:15 |
| rcarrillocruz | weeee | 20:15 |
| mtreinish | rcarrillocruz: heh, I've got 2 talks like that :) | 20:15 |
| fungi | same here (2 or 3) | 20:15 |
| rcarrillocruz | o robyn is here | 20:15 |
| mtreinish | rcarrillocruz: you just helped with one of them | 20:15 |
| rcarrillocruz | rbergeron: ^ | 20:15 |
| *** esikachev has quit IRC | 20:16 | |
| fungi | all good talks are written on the plane flight to the conference, right? ;) | 20:16 |
| mtreinish | fungi: btw, I think you'll appreciate this: https://github.com/mtreinish/handbrakecloud I wrote that for my budget cloud talk (to show what you could do with a cloud in your closet) | 20:16 |
| mtreinish | but after I finished it I realized I just wrote a really crappy nodepool | 20:16 |
| clarkb | fungi: I'm sort of working on mine. | 20:16 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper master: Pin fog-core https://review.openstack.org/461141 | 20:17 |
| fungi | mtreinish: hah! fun | 20:17 |
| *** morgabra has left #openstack-infra | 20:17 | |
| *** florianf has quit IRC | 20:20 | |
| *** esikachev has joined #openstack-infra | 20:20 | |
| *** jcoufal has joined #openstack-infra | 20:21 | |
| *** jcoufal_ has quit IRC | 20:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add optional error_message to adapter.request https://review.openstack.org/461025 | 20:26 |
| *** florianf has joined #openstack-infra | 20:26 | |
| *** EricGonczer_ has joined #openstack-infra | 20:28 | |
| *** jamesmc__ has joined #openstack-infra | 20:28 | |
| *** jamesmca_ has quit IRC | 20:29 | |
| openstackgerrit | Merged openstack-infra/puppet-logstash master: Remove logstash mqtt plugin install https://review.openstack.org/460729 | 20:31 |
| *** esikachev has quit IRC | 20:31 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Move paho-mqtt package install to worker node https://review.openstack.org/460570 | 20:32 |
| *** esikachev has joined #openstack-infra | 20:34 | |
| openstackgerrit | Merged openstack-infra/system-config master: Enable ansible callback events on firehose https://review.openstack.org/458248 | 20:34 |
| fungi | mtreinish: i'm going to start paying attention to our ansible periodic run just to make sure we haven't broken it with that ^ | 20:35 |
| *** rlandy is now known as rlandy|brb | 20:36 | |
| *** esikachev has quit IRC | 20:38 | |
| mtreinish | fungi: ok | 20:38 |
| mtreinish | fungi: I'm about to drive home, so hopefully nothing will break in the next 10-15min :) | 20:39 |
| mtreinish | but I assume something will break eventually | 20:39 |
| fungi | mtreinish: if it does, it's not a super-urgent issue as long as we're paying attention | 20:39 |
| mtreinish | it wouldn't feel like an infra patch from me if it didn't | 20:39 |
| fungi | we all break things all the time | 20:40 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Remove neutronclient mocks from ports tests https://review.openstack.org/461149 | 20:41 |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Adding CI jobs to monasca-kibana-plugin https://review.openstack.org/459511 | 20:42 |
| *** ijw has joined #openstack-infra | 20:43 | |
| *** gordc has quit IRC | 20:43 | |
| fungi | first puppet run since 458248 merged is about to kick off in 60 seconds. i'm tailing the log | 20:44 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Rewrite the query string when removing browse criteria https://review.openstack.org/455024 | 20:47 |
| *** asselin has joined #openstack-infra | 20:48 | |
| *** jcoufal_ has joined #openstack-infra | 20:48 | |
| *** vhosakot has joined #openstack-infra | 20:48 | |
| fungi | cmurphy: aha, some enlightenment has appeared in that github issue... apparently the conditional is evaluated at build time not install time, and the gem was built against newer ruby so baked in the xmlrpc dep unconditionally | 20:49 |
| *** asselin__ has quit IRC | 20:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook https://review.openstack.org/460779 | 20:50 |
| *** jcoufal has quit IRC | 20:50 | |
| cmurphy | that is fantastic | 20:50 |
| *** ijw has quit IRC | 20:51 | |
| fungi | certainly one of the better words for it | 20:51 |
| *** mhickey has quit IRC | 20:51 | |
| *** eharney has quit IRC | 20:52 | |
| clarkb | thats like when lxml wheel broke | 20:53 |
| fungi | pretty much exactly, yep | 20:54 |
| *** florianf has quit IRC | 20:54 | |
| fungi | sdists build on install, wheels are prebuilt. shifts assumptions about whether you can test for certain conditions at "installation" | 20:55 |
| *** tphummel has joined #openstack-infra | 20:55 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Remove neutronclient mocks from quotas tests https://review.openstack.org/461150 | 20:59 |
| *** Apoorva_ has joined #openstack-infra | 21:01 | |
| *** Sukhdev has quit IRC | 21:03 | |
| openstackgerrit | Merged openstack-infra/puppet-mqtt_statsd master: Update subscribed mosquitto metrics https://review.openstack.org/458953 | 21:03 |
| *** jcoufal has joined #openstack-infra | 21:03 | |
| *** Apoorva has quit IRC | 21:04 | |
| mtreinish | fungi: did anything eventful happen in the puppet run? | 21:05 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Add source_url to mqtt payload https://review.openstack.org/460313 | 21:05 |
| *** jamesmc__ has quit IRC | 21:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add subunit gearman worker mqtt info to firehose docs https://review.openstack.org/460323 | 21:05 |
| *** jcoufal_ has quit IRC | 21:05 | |
| fungi | mtreinish: it's still underway | 21:05 |
| fungi | (and i'm still watching) | 21:05 |
| fungi | we're only 20 minutes in and it can take >30 | 21:06 |
| *** dimtruck is now known as zz_dimtruck | 21:06 | |
| fungi | (started at 20:45 utc) | 21:06 |
| *** jamesmca_ has joined #openstack-infra | 21:06 | |
| mtreinish | ok, I just opened an mosquitto_sub on the ansible/# topic | 21:06 |
| mtreinish | hopefully something will pop up there | 21:06 |
| *** dpaterson_ has quit IRC | 21:07 | |
| mtreinish | it's supposed to publish something when a playbook starts, stops, and on each task finishing | 21:07 |
| *** beslemon has quit IRC | 21:07 | |
| fungi | good news is it doesn't look like we broke puppeting our servers | 21:07 |
| mtreinish | small victories :) | 21:07 |
| fungi | but there are tasks being run since you started gulping from the firehose, so if you're not seeing anything then we're likely still missing something | 21:08 |
| *** thorst_ has joined #openstack-infra | 21:08 | |
| fungi | okay, i think the full pass just completed. next will start at 21:15z | 21:08 |
| mtreinish | yeah, I'm not seeing anything on the other end of the firehose | 21:08 |
| *** ijw has joined #openstack-infra | 21:09 | |
| fungi | mtreinish: oh! it's possible we needed to wait through one pass | 21:09 |
| *** thorst has quit IRC | 21:09 | |
| mtreinish | fungi: heh, yeah I was just thinking that | 21:09 |
| mtreinish | because we rely on puppet to put the plugin in the repo | 21:09 |
| fungi | since the puppetmaster updates itself, the pass which updated it may have started before it put its files in place, yeah | 21:10 |
| *** trown is now known as trown|outtypewww | 21:10 | |
| fungi | i'm going to double-check stuff is there now | 21:10 |
| *** ijw has quit IRC | 21:10 | |
| *** ijw has joined #openstack-infra | 21:10 | |
| fungi | it has /etc/mqtt_ca_cert.pem.crt and /etc/mqtt_client.yaml with what look like correct contents | 21:11 |
| fungi | and /opt/ansible/lib/ansible/plugins/callback/mqtt.py too | 21:11 |
| mtreinish | fungi: cool, that's a good sign | 21:11 |
| fungi | so we should be all set for when the next pass begins in ~3 minutes | 21:12 |
| * mtreinish sits on the edge of his seat | 21:12 | |
| fungi | my standing workstation makes that harder than it should be | 21:12 |
| mtreinish | lol | 21:12 |
| *** thorst_ has quit IRC | 21:12 | |
| fungi | just realized the only sitting furniture in my lab are a kneeling chair and a barstool | 21:13 |
| *** Goneri has joined #openstack-infra | 21:13 | |
| *** rlandy|brb is now known as rlandy | 21:13 | |
| *** ldnunes has quit IRC | 21:14 | |
| fungi | and here it goes! | 21:15 |
| mtreinish | not seeing anything so far | 21:15 |
| fungi | i guess if i open up a packet sniffer looking for packets destined for the firehose, i should see something assuming it's publishing | 21:16 |
| *** esberglu has quit IRC | 21:17 | |
| *** harlowja has quit IRC | 21:17 | |
| *** harlowja has joined #openstack-infra | 21:17 | |
| mtreinish | yeah that would work. Wireshark has mqtt filters too | 21:17 |
| mtreinish | although we should be encrypting the packets, so that's not really useful :) | 21:17 |
| mtreinish | I'm still not seeing anything on my end | 21:18 |
| fungi | well, it at least tells me whether any communication is being attempted at all | 21:18 |
| mtreinish | yeah | 21:18 |
| fungi | and so far tcpdump is showing no packets from puppetmaster to the ipv4 nor ipv6 addresses of firehose01 | 21:18 |
| fungi | is ansible supposed to automatically pick up any plugins we drop into /opt/ansible/lib/ansible/plugins/callback/ or do they need to be registered with it somehow? | 21:19 |
| mtreinish | fungi: they should as long as 'CALLBACK_NEEDS_WHITELIST = True' isn't set in the module | 21:20 |
| *** ijw has quit IRC | 21:20 | |
| mtreinish | but one sec I can pull up the config docs on it | 21:20 |
| *** dmsimard|cavern is now known as dmsimard | 21:20 | |
| mtreinish | fungi: these 3 options: http://docs.ansible.com/ansible/intro_configuration.html#bin-ansible-callbacks | 21:21 |
| mtreinish | but it should set lib/ansible/plugins/callback to be the default path | 21:21 |
| mtreinish | otherwise it wouldn't pick up the preinstalled ones | 21:21 |
| fungi | there are definitely a ton of other *.py files in that directory | 21:22 |
| mtreinish | fungi: a simple way to test it would be to comment out the mqtt_hostname field in the config yaml and run a noop playbook locally | 21:24 |
| mtreinish | it should print a warning saying it can't find the hostname variable and disable itself | 21:24 |
| mtreinish | fungi: oh wait is paho-mqtt installed? | 21:25 |
| *** flepied has quit IRC | 21:26 | |
| fungi | would be a pypi package, right? | 21:26 |
| mtreinish | I was just looking at the patch and I don't think I put that in the puppetmaster.pp | 21:26 |
| mtreinish | yeah | 21:26 |
| fungi | not in the global environment at least | 21:26 |
| mtreinish | although if it is missing it should warn that it's not installed at the start of the play | 21:26 |
| mtreinish | do we run ansible from a venv? | 21:26 |
| *** askb has joined #openstack-infra | 21:27 | |
| fungi | that's what i'm checking now | 21:27 |
| * mtreinish isn't super familiar with the puppetmaster setup | 21:27 | |
| fungi | i don't believe we do | 21:27 |
| *** srobert has quit IRC | 21:27 | |
| mtreinish | heh, well I'll push a patch to add paho to the node | 21:27 |
| *** cody-somerville has joined #openstack-infra | 21:27 | |
| *** cody-somerville has joined #openstack-infra | 21:27 | |
| mtreinish | but I'm surprised we're not seeing the warning | 21:27 |
| *** thorst has joined #openstack-infra | 21:29 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Add paho-mqtt install if we turn mqtt on https://review.openstack.org/461156 | 21:29 |
| *** csomerville has quit IRC | 21:29 | |
| mtreinish | fungi: ^^^ | 21:29 |
| fungi | http://git.openstack.org/cgit/openstack-infra/puppet-ansible/tree/manifests/init.pp seems to globally pip install ansible (from a local source tree in our case) | 21:29 |
| mtreinish | ok cool, that makes it easier to get right | 21:30 |
| fungi | a little leery about pip installing things on puppetmaster.o.o, but we're doing that now for shade, ansible and openstackclient | 21:33 |
| *** thorst has quit IRC | 21:33 | |
| *** ijw has joined #openstack-infra | 21:34 | |
| *** asselin has quit IRC | 21:34 | |
| *** ijw has quit IRC | 21:34 | |
| *** ijw has joined #openstack-infra | 21:34 | |
| *** zz_dimtruck is now known as dimtruck | 21:35 | |
| mtreinish | fungi: if it helps paho has 0 deps: https://github.com/eclipse/paho.mqtt.python/blob/v1.2.3/setup.py#L13 | 21:35 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Fix interactions with keystoneauth from newton https://review.openstack.org/461114 | 21:35 |
| * mtreinish didn't realize that until just now | 21:35 | |
| fungi | maybe we'll interest someone in getting paho-mqtt into debian/ubuntu though | 21:35 |
| *** harlowja has quit IRC | 21:36 | |
| mtreinish | fungi: it might be in there already | 21:36 |
| fungi | not under any name including the string "mqtt" at least | 21:36 |
| fungi | http://packages.debian.org/mqtt http://packages.ubuntu.com/mqtt | 21:36 |
| *** Apoorva_ has quit IRC | 21:36 | |
| mtreinish | err yeah the only hit on ubuntu I see is: http://packages.ubuntu.com/yakkety/golang-eclipse-paho-dev | 21:37 |
| *** mriedem has quit IRC | 21:37 | |
| *** Apoorva has joined #openstack-infra | 21:37 | |
| mtreinish | fungi: ubuntu does package the old mosquitto python package: http://packages.ubuntu.com/precise-updates/python-mosquitto | 21:38 |
| mtreinish | but that was replaced by paho | 21:38 |
| mtreinish | https://mosquitto.org/documentation/python/ | 21:38 |
| *** chlong has quit IRC | 21:39 | |
| fungi | yeah, in debian up through jessie | 21:39 |
| fungi | dropped for stretch | 21:39 |
| mtreinish | fungi: I've never done debian packaging before, but if I find the time I'd be willing to push that (unless I can find someone else to do it, which would probably be easier) | 21:39 |
| mtreinish | how much harder can it be than maintaing arch packages :P | 21:40 |
| fungi | the debian python modules team (dpmt) has some fairly straightforward packaging guidelines and is usually willing to team-maintain stuff for you. also there's a tool for generating debs from sdists as a starting point | 21:40 |
| *** lin_yang has joined #openstack-infra | 21:41 | |
| clarkb | mtreinish: arch is much much much easier imo | 21:41 |
| clarkb | debian has rules | 21:42 |
| fungi | heh | 21:42 |
| fungi | (it even has a file _called_ "rules" in each package) | 21:42 |
| fungi | ((though it's really just a makefile)) | 21:43 |
| fungi | i'll consider trying to package it for debian once the stretch release freeze is over | 21:43 |
| mtreinish | clarkb: lol, yeah arch definitely doesn't. Especially for the AUR where all of my packages live | 21:43 |
| fungi | ubuntu will generally just suck packages like that straight in from debian/testing anyway | 21:44 |
| mtreinish | fungi: so was there no warning at the top of the playbook output? | 21:45 |
| jeblair | fungi: huh, my irc client cut you off mid-sentence there. weird. | 21:45 |
| mtreinish | fungi: because I wonder if we've got 2 issues, it's not being loaded and paho was missing | 21:46 |
| fungi | mtreinish: i'm looking for that tiny needle in this haystack of output now | 21:46 |
| *** ijw has quit IRC | 21:47 | |
| *** mdrabe has quit IRC | 21:47 | |
| mtreinish | fungi: I'll spin up my test vm to get an example | 21:47 |
| *** ijw has joined #openstack-infra | 21:48 | |
| fungi | mtreinish: not spotting it in the log (which i think is just redirected stdout from our wrapper script) | 21:48 |
| *** ijw has quit IRC | 21:48 | |
| *** ijw has joined #openstack-infra | 21:48 | |
| fungi | no matches for "warn" (case-insensitive) | 21:49 |
| *** baoli has quit IRC | 21:49 | |
| v1k0d3n | hello everyone | 21:50 |
| v1k0d3n | is there anyone who can help with this one? https://review.openstack.org/#/c/460925/ | 21:50 |
| *** jcoufal_ has joined #openstack-infra | 21:50 | |
| v1k0d3n | trying to get docs sorted before the summit | 21:50 |
| *** jcoufal has quit IRC | 21:50 | |
| mtreinish | fungi: it goes to stderr | 21:51 |
| mtreinish | I just tested it locally | 21:51 |
| mtreinish | fungi: http://paste.openstack.org/show/608371/ | 21:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 21:52 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 21:52 |
| mtreinish | I also just copied the plugin from system-config into my local ansible checkout which seemed to work without configuration | 21:52 |
| *** deafult has quit IRC | 21:52 | |
| fungi | okay | 21:52 |
| fungi | our cronjob does indeed >> logfile 2>&1 so we should be getting stderr in there too | 21:54 |
| *** felipemonteiro_ has quit IRC | 21:54 | |
| mtreinish | fungi: hmm, then I wonder if it's really not getting loaded | 21:55 |
| *** esikachev has joined #openstack-infra | 21:56 | |
| mtreinish | fungi: if your up for manually testing it, I was using this as my test playbook: http://paste.openstack.org/show/608372/ | 21:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 21:57 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 21:57 |
| fungi | mtreinish: i wonder if ansible is only loading plugins from /usr/local/lib/python2.7/dist-packages/ansible/plugins and not also from /opt/ansible/lib/ansible/plugins? | 21:57 |
| *** rlandy has quit IRC | 21:57 | |
| mtreinish | fungi: how do we install it? | 21:57 |
| mtreinish | fungi: I did pip install -e locally nad it seemed to work | 21:57 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Redirect stack-show output in tripleo.sh https://review.openstack.org/459257 | 21:57 |
| fungi | whatever the openstack_pip provider does, which is _probably_ not editable? | 21:58 |
| mtreinish | ok, that might do it | 21:58 |
| fungi | because the file we've added under the source tree isn't showing up under the installed location for the package | 21:58 |
| mtreinish | although if we pip install again it should move it | 21:58 |
| mtreinish | since it's in the package | 21:59 |
| mtreinish | ah, that would do it | 21:59 |
| mtreinish | do you want me to update the path in the puppet? | 21:59 |
| fungi | i think we only trigger pip install when the source git ref changes | 21:59 |
| *** cmurphy has quit IRC | 21:59 | |
| fungi | i'm wondering whether ansible provides an additional default path for local plugin additions (or an envvar you're supposed to set for a plugin search path or something) | 21:59 |
| mtreinish | fungi: you can specify a path in the config file (which I think also gets exposed as a env var) | 22:00 |
| jeblair | fungi: you can set the plugin path in the config file | 22:00 |
| jeblair | ya that | 22:00 |
| fungi | just got there in the ansible docs myselg | 22:00 |
| fungi | myself | 22:00 |
| jeblair | i don't know if it has automatic entry points stuff? (i don't think so) | 22:00 |
| mtreinish | fungi: http://docs.ansible.com/ansible/intro_configuration.html#callback-plugins | 22:00 |
| mtreinish | jeblair: there aren't any entrypoints that I could see when I was looking at the code before | 22:01 |
| fungi | yeah, that's the section i was looking at | 22:01 |
| mtreinish | just the search paths | 22:01 |
| jeblair | mtreinish: so do you have an ansible.cfg with your source or install directory for it? | 22:01 |
| jeblair | mtreinish: so do you have an ansible.cfg with your source or install directory added to the callback plugin path? | 22:01 |
| jeblair | (sorry about first nonsense question :) | 22:01 |
| mtreinish | jeblair: when I ran it I just did git clone ansible && pip install -e ansible && cp mqtt.py ansible/lib/ansible/plugin/callbacks/. | 22:02 |
| mtreinish | I didn't even bother creating an ansible.cfg | 22:02 |
| jeblair | ah yeah, so that's why it found it locally | 22:02 |
| fungi | there's also "You can activate a custom callback by either dropping it into a callback_plugins directory adjacent to your play or inside a role or by putting it in one of the callback directory sources configured in ansible.cfg." http://docs.ansible.com/ansible/dev_guide/developing_plugins.html#configuring-callback-plugins | 22:02 |
| mtreinish | yep | 22:02 |
| *** esikachev has quit IRC | 22:02 | |
| mtreinish | jeblair: I wrongly assumed it was also a editable install when I wrote the puppet to copy it into the source tree | 22:03 |
| *** jcoufal has joined #openstack-infra | 22:03 | |
| jeblair | so we could put this in system-config, adjacent to the playbook. but if we want to maintain it as a separate git repo (yes, i rather think), then we may just want to install it like we install puppet modules. 'git clone /opt/....', then add that to ansible.cfg. | 22:04 |
| *** cmurphy has joined #openstack-infra | 22:04 | |
| jeblair | oh there's puppet to copy it somewhere? | 22:04 |
| jeblair | mtreinish: link to that ^? | 22:04 |
| mtreinish | jeblair: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/puppetmaster.pp#n366 | 22:05 |
| *** Apoorva has quit IRC | 22:05 | |
| mtreinish | oh sry L383 more specifically | 22:05 |
| *** jcoufal_ has quit IRC | 22:05 | |
| fungi | yeah, i approved that without noticing it was installing it into the git source tree on disk rather than somewhere ansible knew to look | 22:05 |
| *** Apoorva has joined #openstack-infra | 22:05 | |
| mtreinish | jeblair: my longer term plan with this was to put it in ansible proper (I have an open PR for it now) | 22:05 |
| jeblair | ah. sorry, some of my earlier statements were under the misapprehension the mqtt plugin was a python package, but you meant 'pip install ansible' | 22:06 |
| fungi | just a bare plugin (not packaged) | 22:07 |
| mtreinish | yeah | 22:07 |
| jeblair | that actually makes doing system-config/playbooks/callback_plugins/mqtt.py a little more feasible. | 22:07 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Extend query for bug 1687044 https://review.openstack.org/461166 | 22:08 |
| openstack | bug 1687044 in Cinder "test_volume_extend_when_volume_has_snapshot fails with VolumeNotDeactivated" [Undecided,Confirmed] https://launchpad.net/bugs/1687044 | 22:08 |
| fungi | do the playbooks in system-config only ever run on the puppetmaster? | 22:08 |
| jeblair | (though, there may still be reasons to install it to neutral location and add it to ansible.cfg -- like if we think there are circumstances where people would be surprised that plugin is getting pulled in) | 22:08 |
| jeblair | fungi: ^ i think you may be getting at what i was saying there | 22:08 |
| fungi | yep | 22:08 |
| mtreinish | fungi: does it matter? woudln't that just more events for firehose :p | 22:09 |
| mtreinish | wow, I really can't write today... | 22:09 |
| jeblair | mtreinish: i probably don't have the right creds on my desktop. at least i hope i don't. | 22:09 |
| fungi | mtreinish: more likely it would be more warning messages from not having the dependency and config present | 22:09 |
| jeblair | fungi: i *think* so. but i kind of lean toward explicit ansible.cfg so we don't have to worry about it in the future. | 22:09 |
| fungi | that seems fine | 22:10 |
| jeblair | (i like levers for this sort of thing too) | 22:10 |
| fungi | should we stick it somewhere in /etc? /opt? /usr/local? bikeshed time | 22:10 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear master: Provide TextJob and TextWorker for convenience https://review.openstack.org/461172 | 22:11 |
| *** thorst has joined #openstack-infra | 22:11 | |
| jeblair | fungi: imma gonna let you pait this one | 22:11 |
| fungi | i don't know where the "more correct" location in our filesystem is since i'm not aware of any good prior art | 22:11 |
| fungi | but happy to just pick one | 22:11 |
| jeblair | wow. i typed 'pain' at first and made it halfway to 'paint'. | 22:11 |
| mtreinish | do we write out the ansible.cfg somewhere? or is that a manual step | 22:11 |
| *** jascott1 has quit IRC | 22:12 | |
| fungi | we have an /etc/ansible/ansible.cfg | 22:12 |
| mtreinish | fungi: I would probably just drop it in /opt/callback_plugins | 22:12 |
| fungi | _probably_ put there by openstack-infra/puppet-ansible | 22:12 |
| *** thorst has quit IRC | 22:12 | |
| jeblair | looks like it | 22:13 |
| fungi | we have roles_path=/etc/ansible/roles in it, so maybe we want /etc/ansible/plugins? | 22:13 |
| mtreinish | that works too | 22:13 |
| mtreinish | fungi: yep: https://github.com/openstack-infra/puppet-ansible/blob/master/templates/ansible.cfg.erb | 22:13 |
| *** jascott1 has joined #openstack-infra | 22:14 | |
| *** gouthamr has quit IRC | 22:14 | |
| fungi | i need to take a break from this soon to go do some yardwork before i lose the daylight, but can pick it back up after | 22:14 |
| *** jamesmca_ has quit IRC | 22:15 | |
| jeblair | mtreinish: if you want to write changes, i'm around for a little longer and can review | 22:15 |
| fungi | happy to babysit it some more this evening and see if it's any closer | 22:15 |
| mtreinish | ok, I'm working on writing up the puppet-ansible patch now | 22:16 |
| *** jamesmca_ has joined #openstack-infra | 22:16 | |
| fungi | awesome | 22:16 |
| mtreinish | and I'll push up the system-config depends on after that | 22:16 |
| *** xyang1 has quit IRC | 22:18 | |
| *** EricGonc_ has joined #openstack-infra | 22:19 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-ansible master: Set an ansible callback directory https://review.openstack.org/461174 | 22:19 |
| *** asselin has joined #openstack-infra | 22:19 | |
| mtreinish | fungi, jeblair: ^^^ | 22:19 |
| *** jamesmca_ has quit IRC | 22:20 | |
| *** EricGonczer_ has quit IRC | 22:20 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir https://review.openstack.org/461175 | 22:22 |
| mtreinish | fungi, jeblair: ^^^ | 22:22 |
| *** jcoufal has quit IRC | 22:22 | |
| *** EricGonc_ has quit IRC | 22:24 | |
| jeblair | mtreinish: +3 on first; tentative -1 on second | 22:26 |
| * jeblair imagines fungi mowing lawn with HUD | 22:26 | |
| *** vhosakot has quit IRC | 22:27 | |
| jeblair | what could possibly go wrong | 22:27 |
| fungi | weapons of yard destruction! | 22:27 |
| * fungi hasn't actually gone out to do yardwork yet | 22:27 | |
| mtreinish | err, I have lint on the first change | 22:28 |
| *** ijw has quit IRC | 22:28 | |
| fungi | apply ample quantities of lint remover | 22:28 |
| *** adrian_otto has joined #openstack-infra | 22:29 | |
| jeblair | wow, it sure looked like it had enough whitespace | 22:30 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-ansible master: Set an ansible callback directory https://review.openstack.org/461174 | 22:30 |
| mtreinish | jeblair: off by 1 | 22:30 |
| *** jkilpatr has quit IRC | 22:30 | |
| fungi | you anger the whitespace gods with your inadequate use of the spacebar | 22:30 |
| mtreinish | jeblair: for the system-config patch can I have a requires on the file if it's inside a different module? | 22:31 |
| mtreinish | jeblair: I wasn't sure about that which is why I left it out | 22:31 |
| jeblair | mtreinish: i believe so | 22:31 |
| mtreinish | ok, I'll add that then | 22:31 |
| mtreinish | I always screw up puppet-scoping so I'm never sure | 22:31 |
| fungi | should be able to, and then it'll wait to process that block until the other module has created the dependency | 22:32 |
| fungi | it merges them all into one ginormous manifest and then generates the graph | 22:32 |
| *** Apoorva_ has joined #openstack-infra | 22:32 | |
| fungi | as i understand it | 22:32 |
| fungi | which is also why we have so many silly-looking conditional guards around resources to prevent duplicate definitions | 22:33 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir https://review.openstack.org/461175 | 22:34 |
| mtreinish | fungi: ^^ I also added the absent to save you the rm | 22:34 |
| fungi | thanks | 22:34 |
| *** esikachev has joined #openstack-infra | 22:35 | |
| *** Apoorva has quit IRC | 22:36 | |
| jeblair | +6 | 22:36 |
| fungi | the boxcar | 22:37 |
| fungi | okay, i'm going to go do battle with the lawn while those churn through | 22:37 |
| fungi | back soon | 22:37 |
| mtreinish | fungi: enjoy | 22:37 |
| clarkb | tomorrow is my yard battle day | 22:39 |
| *** esikachev has quit IRC | 22:39 | |
| *** thorst has joined #openstack-infra | 22:43 | |
| openstackgerrit | Merged openstack-infra/puppet-ansible master: Set an ansible callback directory https://review.openstack.org/461174 | 22:43 |
| *** baoli has joined #openstack-infra | 22:45 | |
| *** baoli has quit IRC | 22:45 | |
| *** wolverineav has quit IRC | 22:46 | |
| *** adrian_otto has quit IRC | 22:46 | |
| *** baoli has joined #openstack-infra | 22:46 | |
| *** esikachev has joined #openstack-infra | 22:47 | |
| *** adrian_otto has joined #openstack-infra | 22:48 | |
| openstackgerrit | Merged openstack-infra/system-config master: Copy ansible mqtt callback plugin into the plugins dir https://review.openstack.org/461175 | 22:51 |
| *** ijw has joined #openstack-infra | 22:51 | |
| *** esikachev has quit IRC | 22:51 | |
| *** Sukhdev has joined #openstack-infra | 22:53 | |
| mtreinish | jeblair: oh there is one more patch we'll need before this can work: https://review.openstack.org/#/c/461156/ | 22:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger https://review.openstack.org/461176 | 22:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 22:55 |
| mtreinish | otherwise we'll be running the plugin without the mqtt libs installed | 22:55 |
| *** annegentle has quit IRC | 22:55 | |
| *** ijw has quit IRC | 22:56 | |
| *** asselin_ has joined #openstack-infra | 22:58 | |
| jeblair | mtreinish: +3 | 22:58 |
| mtreinish | thanks | 22:59 |
| jeblair | mtreinish: i'm afk for a bit now, will check in again soon | 22:59 |
| *** asselin__ has joined #openstack-infra | 22:59 | |
| mtreinish | ok | 22:59 |
| *** thorst has quit IRC | 22:59 | |
| *** lamt has quit IRC | 23:01 | |
| *** asselin has quit IRC | 23:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Store initial repo state in the merger https://review.openstack.org/461176 | 23:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use previously stored repo state on executor https://review.openstack.org/461177 | 23:02 |
| *** lamt has joined #openstack-infra | 23:02 | |
| *** rbrndt has quit IRC | 23:02 | |
| *** rhallisey has quit IRC | 23:02 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert test_role_assignments to requests mock https://review.openstack.org/448313 | 23:03 |
| *** asselin_ has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add paho-mqtt install if we turn mqtt on https://review.openstack.org/461156 | 23:04 |
| *** xarses_ has quit IRC | 23:05 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear master: Provide TextJob and TextWorker for convenience https://review.openstack.org/461172 | 23:05 |
| *** lamt has quit IRC | 23:08 | |
| *** ijw has joined #openstack-infra | 23:09 | |
| *** adrian_otto has quit IRC | 23:09 | |
| *** esikachev has joined #openstack-infra | 23:10 | |
| *** asselin has joined #openstack-infra | 23:10 | |
| *** adrian_otto has joined #openstack-infra | 23:11 | |
| *** asselin__ has quit IRC | 23:12 | |
| *** ijw has quit IRC | 23:13 | |
| *** asselin_ has joined #openstack-infra | 23:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook https://review.openstack.org/460779 | 23:15 |
| *** adrian_otto has quit IRC | 23:15 | |
| fungi | clarkb: tomorrow is _also_ my yard battle day ;) | 23:15 |
| fungi | just trying to not fall behind | 23:16 |
| *** esikachev has quit IRC | 23:17 | |
| *** asselin has quit IRC | 23:17 | |
| asselin_ | anyone know where the docs are to connect a new gerrit instance to authenticate with launchpad? how do you get the access token/secret/consumer key? https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n74 | 23:19 |
| clarkb | asselin_: I think thats for the hook scripts not for authentication with openid | 23:21 |
| clarkb | er not for gerrit's openid auth | 23:21 |
| asselin_ | oh...so maybe nothing needs to be done? just use the default setting? | 23:21 |
| mordred | fungi: every day is battle day | 23:21 |
| clarkb | if you don't need launchpad bugs to be updated by the hook scripts thats fine | 23:21 |
| fungi | mtreinish: we have paho-mqtt (1.2.3) on puppetmaster.o.o now, and callback_plugins=/etc/ansible/callback_plugins in /etc/ansible/ansible.cfg and the plugin is present in that dir and already has a corresponding mqtt.pyc file indicating ansible loaded it roughly 2 minutes after it was installed (so likely took effect partway into the run) | 23:21 |
| clarkb | otherwise you have to do the launchpadlib thing | 23:22 |
| clarkb | which is documented in launchpadlib iirc | 23:22 |
| asselin_ | clarkb, no...just looking for openid for users to create accounts in gerrit....maybe I don't need to do anything. Will give it a try. thanks! | 23:22 |
| fungi | asselin_: right, not necessary for that case | 23:23 |
| fungi | asselin_: just make sure to set the openid configuration in the gerrit.conf and it should just work | 23:23 |
| mtreinish | fungi: ok cool, I've got my mosquitto_sub still open in a window | 23:23 |
| mtreinish | hopefully it'll start emitting the next go around | 23:23 |
| asselin_ | fungi, yeah...seems that's already in puppet...so the defaults might 'just work' | 23:23 |
| *** esberglu has joined #openstack-infra | 23:24 | |
| fungi | asselin_: specifically the lp auth stuff is for having gerrit be able to comment on and reassign lp bugs and blueprints | 23:24 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Support GitHub PR webhooks https://review.openstack.org/439834 | 23:24 |
| fungi | as clarkb mentioned | 23:24 |
| asselin_ | ack...no need for that use case | 23:24 |
| fungi | mtreinish: it's already trying to use it, and i see warnings in the logs | 23:25 |
| fungi | mtreinish: i'll see if i can collect them all into a paste trimming out all the noise | 23:26 |
| mtreinish | fungi: ooh, ok | 23:26 |
| *** esikachev has joined #openstack-infra | 23:27 | |
| *** esberglu has quit IRC | 23:28 | |
| fungi | mtreinish: these are the unique errors i found http://paste.openstack.org/show/608373 | 23:29 |
| fungi | looks like they're probably all the same problem though? | 23:29 |
| *** sree has joined #openstack-infra | 23:29 | |
| mtreinish | fungi: yeah I think I missed a string cast somewhere | 23:29 |
| mtreinish | although why I didn't hit that locally I can't say | 23:29 |
| fungi | seems likely | 23:29 |
| fungi | python version? | 23:30 |
| mtreinish | fungi: if we ran with -vvv that should print the tracebacks | 23:30 |
| fungi | Python 2.7.6 on puppetmaster.o.o | 23:30 |
| mtreinish | I was running on ubuntu with 2.7 so I would expect the same | 23:30 |
| fungi | yeah | 23:30 |
| mtreinish | 2.7.12 | 23:30 |
| *** harlowja has joined #openstack-infra | 23:30 | |
| fungi | how were you invoking your hello world? | 23:31 |
| mtreinish | ansible-play -vvv hello_world.yml | 23:31 |
| mtreinish | s/ansible-play/ansible-playbook/ | 23:31 |
| mtreinish | fungi: I think I found it | 23:32 |
| *** esikachev has quit IRC | 23:32 | |
| fungi | yeah, it gets me the same warnings | 23:32 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Cast the playbook uuid as a string https://review.openstack.org/461180 | 23:32 |
| mtreinish | fungi: ^^^ | 23:32 |
| *** sree has quit IRC | 23:33 | |
| mtreinish | fungi: I'm thinking it might be my local ansible version, I'm running a random checkout on the current master | 23:34 |
| mtreinish | so > 2.3.0 which is the latest release | 23:34 |
| mtreinish | fungi: which ansible version is installed on puppetmaster.o.o? | 23:36 |
| fungi | mtreinish: extra verbosity still didn't show me any python tracebacks, but ansible 2.2.1.0 | 23:37 |
| fungi | (pinned a couple lines from where you edited the manifest) | 23:37 |
| fungi | mtreinish: anyway, that patch seems to do the trick | 23:40 |
| mtreinish | you tested it locally | 23:41 |
| mtreinish | ? | 23:41 |
| fungi | on the puppetmaster, with your hello world playbook | 23:41 |
| mtreinish | hmm, I didn't see any mqtt events | 23:41 |
| fungi | where i was able to reproduce previously | 23:42 |
| mtreinish | oh nope there they are | 23:42 |
| mtreinish | I had the wrong terminal open | 23:42 |
| fungi | i still get one failure from that playbook but it seems to be unrelated | 23:42 |
| fungi | the echo task at the end fails | 23:42 |
| *** jamesmca_ has joined #openstack-infra | 23:42 | |
| *** Swami has quit IRC | 23:42 | |
| mtreinish | http://paste.openstack.org/show/608375/ | 23:42 |
| mtreinish | yeah there is an exit 1 there | 23:42 |
| mtreinish | I did that on purpose to see the error code :) | 23:42 |
| fungi | aha, yep, i missed it before | 23:42 |
| fungi | okay, so working as intended | 23:43 |
| fungi | i've approved the one-liner fix to the plugin so we should see this working shortly | 23:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Let the jenkins user own the cache files https://review.openstack.org/461181 | 23:44 |
| mtreinish | \o/ that was easier than I was expecting | 23:44 |
| cmurphy | i think jobs are going to start failing tomorrow morning without that ^ | 23:44 |
| mtreinish | now I guess I'll need to write docs | 23:44 |
| mtreinish | ugh, describing the schema is gonna be annoying | 23:44 |
| mtreinish | ooh, I'm seeing lots of ansible events now | 23:45 |
| mtreinish | and there is a nice spike in grafana: http://grafana.openstack.org/render/dashboard-solo/db/mosquitto-status?panelId=6&fullscreen&from=1493419594595&to=1493423194595&width=1000&height=500 | 23:46 |
| *** jamesmca_ has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack-infra/system-config master: Cast the playbook uuid as a string https://review.openstack.org/461180 | 23:47 |
| mtreinish | lol, I really like that the facts end up in the output so we get a bunch of xeon model names in the json | 23:48 |
| *** Apoorva_ has quit IRC | 23:48 | |
| *** jamesmca_ has joined #openstack-infra | 23:48 | |
| fungi | yeah, i think we discussed getting rid of the fact reporting there since it's mostly useless noise to be generating on every single run | 23:49 |
| fungi | there have also been security bugs in the past where module authors didn't understand that custom facts are not a safe place to put things like service passwords | 23:51 |
| *** gildub has joined #openstack-infra | 23:53 | |
| *** jamesmca_ has quit IRC | 23:53 | |
| *** jamesmca_ has joined #openstack-infra | 23:54 | |
| mtreinish | heh, I was a bit worried about that actually | 23:54 |
| mtreinish | the blob was so big even on my hello world playbook I couldn't inspect it for secrets by eye | 23:54 |
| mtreinish | but I figured it was a well known enough space it was probably safe | 23:55 |
| mtreinish | (probably a foolish assumption to make) | 23:55 |
| *** harlowja has quit IRC | 23:55 | |
| *** thorst has joined #openstack-infra | 23:56 | |
| *** jkilpatr has joined #openstack-infra | 23:56 | |
| mtreinish | there sure are a lot of infracloud events | 23:56 |
| fungi | lots of machines in there | 23:57 |
| *** slagle has quit IRC | 23:57 | |
| *** esikachev has joined #openstack-infra | 23:58 | |
| mtreinish | ooh looks like the most recent run failed: {"status": "FAILED", "playbook_name": "compute*.ic.openstack.org:!disabled", "playbook_id": "2f16fa1a-d9fd-44ad-b717-cf83750fafa2"} | 23:58 |
| *** jamesmca_ has quit IRC | 23:59 | |
| *** harlowja has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!