| *** amotoki has quit IRC | 00:00 | |
| *** pvaneck has quit IRC | 00:01 | |
| *** amotoki has joined #openstack-infra | 00:01 | |
| *** lezbar has quit IRC | 00:03 | |
| *** spzala has quit IRC | 00:03 | |
| *** lezbar has joined #openstack-infra | 00:04 | |
| openstackgerrit | Dan Kolb proposed openstack-infra/jenkins-job-builder: corrected minor mispelling https://review.openstack.org/381345 | 00:04 |
|---|---|---|
| *** amotoki has quit IRC | 00:05 | |
| *** number80 has quit IRC | 00:08 | |
| *** ijw has joined #openstack-infra | 00:08 | |
| *** kaisers has quit IRC | 00:08 | |
| *** kaisers has joined #openstack-infra | 00:09 | |
| *** kaisers2 has joined #openstack-infra | 00:09 | |
| *** kaisers1 has quit IRC | 00:09 | |
| *** amotoki has joined #openstack-infra | 00:13 | |
| *** ijw has quit IRC | 00:13 | |
| *** verdurin_ has joined #openstack-infra | 00:14 | |
| *** sflanigan_afk has joined #openstack-infra | 00:16 | |
| *** verdurin has quit IRC | 00:17 | |
| *** thcipriani is now known as thcipriani|afk | 00:18 | |
| tonyb | Can I get an infra-core to have a quick look at: https://review.openstack.org/#/q/is:open+project:openstack-infra/irc-meetings+owner:tony%2540bakeyournoodle.com+status:open+label:Code-Review%253E%253D2 rather then self-approve ? | 00:18 |
| jhesketh | tonyb: I'll take al ook | 00:20 |
| tonyb | jhesketh: Thanks. | 00:21 |
| *** Apoorva has quit IRC | 00:23 | |
| jhesketh | tonyb: lgtm | 00:23 |
| tonyb | jhesketh: Thanks | 00:23 |
| openstackgerrit | Merged openstack-infra/irc-meetings: I18n: change time and chair information https://review.openstack.org/380998 | 00:24 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update meeting chair for Nova Bugs Team https://review.openstack.org/381310 | 00:24 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add translation jobs to neutron-lbaas-dashboard https://review.openstack.org/381347 | 00:24 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Expand the valid characters in chair names https://review.openstack.org/379912 | 00:26 |
| *** ijw has joined #openstack-infra | 00:26 | |
| *** ijw has quit IRC | 00:27 | |
| *** ijw_ has joined #openstack-infra | 00:28 | |
| *** tuannguyen has joined #openstack-infra | 00:29 | |
| *** ijw_ has quit IRC | 00:29 | |
| *** markvoelker has joined #openstack-infra | 00:29 | |
| *** adrian_otto has quit IRC | 00:29 | |
| *** thorst has quit IRC | 00:32 | |
| *** asselin__ has quit IRC | 00:33 | |
| *** thorst has joined #openstack-infra | 00:33 | |
| *** tuannguyen has quit IRC | 00:34 | |
| *** markvoelker has quit IRC | 00:34 | |
| *** gouthamr has joined #openstack-infra | 00:38 | |
| *** mtanino has joined #openstack-infra | 00:40 | |
| *** thorst has quit IRC | 00:42 | |
| *** Rockyg has quit IRC | 00:44 | |
| pabelanger | clarkb: More info, looks like this issue is limited to providers that are doing DHCP | 00:44 |
| armax | clarkb: ping | 00:44 |
| armax | sorry it took me so long to get back to you | 00:44 |
| pabelanger | clarkb: which makes sense, since I am only seeing failures in rackspace and ovh right now | 00:45 |
| armax | posted a comment on https://bugs.launchpad.net/devstack/+bug/1629133/comments/9 | 00:45 |
| openstack | Launchpad bug 1629133 in devstack "New neutron subnet pool support breaks multinode testing." [Critical,In progress] - Assigned to Dr. Jens Rosenboom (j-rosenboom-j) | 00:45 |
| clarkb | pabelanger: huh | 00:45 |
| clarkb | armax ok | 00:45 |
| pabelanger | clarkb: it looks like SSH is started before dhclient has finished being assigned an IP address, then /etc/network/if-up.d/openssh-server is restarting sshd | 00:45 |
| pabelanger | once the IP is assigned | 00:46 |
| *** tqtran has quit IRC | 00:46 | |
| pabelanger | however, I am able to establish a connection, because I get the banner failure | 00:46 |
| *** xarses has joined #openstack-infra | 00:46 | |
| pabelanger | http://paste.openstack.org/show/584098/ | 00:47 |
| pabelanger | showing the restart of SSHd | 00:47 |
| *** baoli has joined #openstack-infra | 00:47 | |
| *** amitgandhinz has joined #openstack-infra | 00:48 | |
| *** baoli_ has joined #openstack-infra | 00:49 | |
| pabelanger | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=502444 | 00:50 |
| openstack | Debian bug 502444 in openssh-server "sshd fails at boot-time following reload by /etc/network/if-up.d/openssh-server presumably due to race condition" [Grave,Fixed] | 00:50 |
| pabelanger | originally reloaded sshd | 00:50 |
| *** ijw has joined #openstack-infra | 00:50 | |
| pabelanger | but there was race conditions | 00:50 |
| pabelanger | so, they changed it to a restart | 00:50 |
| *** adrian_otto has joined #openstack-infra | 00:50 | |
| armax | clarkb: mind you that was the initial value I had set for the subnet pool :) | 00:51 |
| *** adrian_otto has quit IRC | 00:52 | |
| *** baoli has quit IRC | 00:52 | |
| clarkb | armax: the problem is while they may be different this is a backward inconpatible change with no docs and no signaling how to fix it | 00:52 |
| armax | you mean your change? | 00:53 |
| clarkb | in that situation you should use the existing method for avoiding subnet range conflicts | 00:53 |
| clarkb | no the change yo devstack | 00:53 |
| armax | oh, you mean https://review.openstack.org/#/c/282559/ | 00:53 |
| *** adrian_otto has joined #openstack-infra | 00:53 | |
| armax | ? | 00:53 |
| clarkb | my change us an attemot to pick the pieces up by using existing information available in drvstack to make a better choice at pool range | 00:53 |
| armax | or strictly speaking assaf's? | 00:54 |
| *** markvoelker has joined #openstack-infra | 00:54 | |
| armax | https://review.openstack.org/#/c/283289 | 00:54 |
| *** ijw has quit IRC | 00:54 | |
| clarkb | whatever change to devstack set that range with new variables | 00:54 |
| armax | fwiw I’d rather revert change https://review.openstack.org/#/c/283289 | 00:54 |
| clarkb | that is also fine by me | 00:55 |
| armax | I am not sure I understand why you think the change is bw incompat | 00:55 |
| *** adrian_otto has quit IRC | 00:56 | |
| armax | a user can still create his/her own networks without having subnet pools interfere | 00:56 |
| armax | how does the multinode breakage happen? | 00:56 |
| clarkb | its not backward compat because it is on by default and breaks anyone using 10/8 | 00:57 |
| clarkb | thr breakage happens because the change takes over 10/8 and applies a route for it on br-ex | 00:57 |
| clarkb | meaning you cannot use this change and 10/8 unless you set q bunch of new config | 00:57 |
| clarkb | thats not backward xompat | 00:57 |
| *** tiswanso has joined #openstack-infra | 00:57 | |
| *** adrian_otto has joined #openstack-infra | 00:58 | |
| *** rfolco has quit IRC | 00:58 | |
| *** zz_dimtruck is now known as dimtruck | 00:58 | |
| clarkb | another alternative is to not set a route for 10/8 | 00:59 |
| *** spzala has joined #openstack-infra | 00:59 | |
| *** adrian_otto has quit IRC | 00:59 | |
| *** amotoki has quit IRC | 00:59 | |
| armax | I suppose the route is installed during the tempest run? | 01:00 |
| armax | not at setup time, right? | 01:00 |
| *** amotoki has joined #openstack-infra | 01:00 | |
| *** purp has quit IRC | 01:00 | |
| armax | I mean, this happens as soon as the tests that make use of subnetpools get executed | 01:00 |
| clarkb | Its in devstack iirc | 01:00 |
| pabelanger | clarkb: I'm going to try a custom upstart script in the morning for openssh-server, I think we could do something like: start on runlevel [2345] and net-device-up IFACE!=lo for ssh.conf, which means SSH would only start once any other interface is only | 01:00 |
| clarkb | but I didnt dig into confirm | 01:00 |
| armax | because up until that point, I can’t see how that can happen | 01:01 |
| pabelanger | online* | 01:01 |
| armax | either way I agree that /8 is a poor choice | 01:01 |
| *** claudiub has quit IRC | 01:01 | |
| clarkb | armax: if it does happen as a side effect of tempest then neutron should stop doing that and only set routes for the actual subnets used not the entire pool | 01:01 |
| armax | I’ll have to check with carl_baldwin about that because he’s the mastermind behind subnetpools | 01:02 |
| armax | but as a stopgap we should do something right now | 01:02 |
| *** amotoki has quit IRC | 01:04 | |
| *** spzala has quit IRC | 01:04 | |
| mordred | I agree with all suggestoins that are correct | 01:04 |
| mordred | but let me go on record saying that I think all the incorrect suggestions for fixing this are wrong | 01:05 |
| armax | mordred: my suggestions are always sensible | 01:05 |
| armax | if that’s what you’re implying :) | 01:05 |
| armax | clarkb: I’ll talk to carl_baldwin tonight as funnily enough he’s in town | 01:07 |
| clarkb | ok | 01:07 |
| armax | clarkb: if you can point me out to how this fails in the gate, that would help me understand deeper what’s going on | 01:07 |
| *** amotoki has joined #openstack-infra | 01:08 | |
| armax | clarkb: as for https://review.openstack.org/#/c/379543/ I would personally hardcode new values (my original ones) rather than rely on the FIXED_RANGE_*, but I am not strongly opinionated on it | 01:09 |
| *** baoli_ has quit IRC | 01:09 | |
| *** baoli has joined #openstack-infra | 01:09 | |
| clarkb | armax: the overlsy for multinode doesnt work because it runs over 10/8 on eg eth1 not br-ex | 01:09 |
| clarkb | just look at any neutron multinode job on osic | 01:10 |
| armax | clarkb: how can I tell? | 01:10 |
| armax | I mean which one is a neutron multinode job on osic | 01:10 |
| clarkb | you can ask logstash, its also recorded at the beginning of the console log | 01:11 |
| armax | ok | 01:11 |
| clarkb | let me find anexample log | 01:12 |
| *** amitgandhinz has quit IRC | 01:12 | |
| armax | clarkb: don’t worry, I’ll dig one myself | 01:12 |
| armax | I need to dash now | 01:12 |
| *** tuanluong has joined #openstack-infra | 01:13 | |
| *** adrian_otto has joined #openstack-infra | 01:13 | |
| clarkb | http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/console.html#_2016-10-03_21_14_28_763192 is one way it fails | 01:13 |
| clarkb | basically ansible cant sshto subnodes because broken routes | 01:13 |
| clarkb | have alsoseen grenade and other jobs fail in other ways | 01:14 |
| armax | ack | 01:14 |
| * mordred apologizes for incoming patch bomb | 01:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Make sure we're matching image status properly https://review.openstack.org/377164 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Normalize images https://review.openstack.org/377163 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add helper properties to generate location info https://review.openstack.org/381339 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Start splitting normalize functions into a mixin https://review.openstack.org/381352 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add location to server record https://review.openstack.org/381353 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move image normalize calls to _normalize https://review.openstack.org/381354 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Move normalize_flavors to _normalize https://review.openstack.org/381355 | 01:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add location field to flavors https://review.openstack.org/381356 | 01:15 |
| armax | clarkb: ack | 01:15 |
| armax | I’ll follow up | 01:16 |
| *** adrian_otto has quit IRC | 01:16 | |
| *** adrian_otto has joined #openstack-infra | 01:16 | |
| armax | clarkb: as I said, because of the severity of the issue I am happy to see the default values change | 01:16 |
| armax | clarkb: I’ll talk to carl_baldwin to understand better how subnetpools fiddle with the host routes and get back to you | 01:17 |
| clarkb | thanks | 01:17 |
| cloudnull | armax: I'd be happy to kick a couple of tests nodes from within the osic if you need. | 01:17 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Add non-voting python3 jobs to requirements https://review.openstack.org/381357 | 01:18 |
| *** thorst has joined #openstack-infra | 01:21 | |
| *** thorst has quit IRC | 01:21 | |
| *** thorst has joined #openstack-infra | 01:22 | |
| *** adrian_otto has quit IRC | 01:22 | |
| *** amotoki has quit IRC | 01:24 | |
| *** amotoki has joined #openstack-infra | 01:24 | |
| *** adrian_otto has joined #openstack-infra | 01:26 | |
| *** adrian_otto has quit IRC | 01:27 | |
| *** number80 has joined #openstack-infra | 01:27 | |
| openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network https://review.openstack.org/343041 | 01:28 |
| *** amotoki has quit IRC | 01:29 | |
| *** adrian_otto has joined #openstack-infra | 01:29 | |
| *** thorst has quit IRC | 01:30 | |
| *** adrian_otto has quit IRC | 01:32 | |
| *** kaisers_ has joined #openstack-infra | 01:38 | |
| *** tuannguyen has joined #openstack-infra | 01:40 | |
| *** rfolco has joined #openstack-infra | 01:40 | |
| *** esikachev has joined #openstack-infra | 01:41 | |
| *** kaisers_ has quit IRC | 01:42 | |
| *** esikachev has quit IRC | 01:46 | |
| *** thorst has joined #openstack-infra | 01:47 | |
| *** adrian_otto has joined #openstack-infra | 01:48 | |
| *** thorst has quit IRC | 01:48 | |
| *** tphummel has quit IRC | 01:49 | |
| *** adrian_otto has quit IRC | 01:51 | |
| *** adrian_otto has joined #openstack-infra | 01:53 | |
| *** tuannguyen has quit IRC | 01:54 | |
| *** spzala has joined #openstack-infra | 01:54 | |
| *** adrian_otto has quit IRC | 01:55 | |
| *** tuannguyen has joined #openstack-infra | 01:56 | |
| *** spzala has quit IRC | 01:59 | |
| *** amotoki has joined #openstack-infra | 01:59 | |
| *** EricGonczer_ has joined #openstack-infra | 02:03 | |
| *** Swami__ has joined #openstack-infra | 02:04 | |
| *** Swami_ has quit IRC | 02:08 | |
| *** amitgandhinz has joined #openstack-infra | 02:09 | |
| *** timello has quit IRC | 02:09 | |
| *** spzala has joined #openstack-infra | 02:10 | |
| *** timello has joined #openstack-infra | 02:13 | |
| *** timello has quit IRC | 02:19 | |
| mordred | armax, cloudnull: is this related: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html#_2016-10-04_01_59_52_598273 ? | 02:23 |
| mordred | the order of the fixed ips returned seems to have changed, and it was stable before | 02:24 |
| mordred | it's also possible it was 'stable' accidentally and I should not just do an assertEqual there ... | 02:24 |
| *** david-lyle has joined #openstack-infra | 02:25 | |
| *** jamesdenton has joined #openstack-infra | 02:25 | |
| *** rfolco has quit IRC | 02:25 | |
| *** timello has joined #openstack-infra | 02:26 | |
| openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network https://review.openstack.org/343041 | 02:26 |
| cloudnull | mordred: i dont think so . | 02:31 |
| cloudnull | within the OSIC the subnet order should be consistent. | 02:31 |
| cloudnull | in the rax cloud they flip flow | 02:32 |
| cloudnull | *flop | 02:32 |
| *** sputnik13 has joined #openstack-infra | 02:32 | |
| mordred | cloudnull: hrm - those should be devstack subnets though, not underlying cloud subnets | 02:33 |
| cloudnull | in OSIC cloud I believe the ordering is 10/8, ::/64, fe80 | 02:33 |
| cloudnull | and I believe the assignment of the 10/8 within the devstack subnets is the culperit causing multi-node gate brokeness. | 02:34 |
| mordred | (I'm thinking I should just make that test more resilient) | 02:34 |
| clarkb | yes and its breaking a bunch if other random.stuff | 02:34 |
| *** rossella_s has quit IRC | 02:34 | |
| cloudnull | though I could just be tired and sick, and sick and tired and plain wrong. | 02:34 |
| *** rossella_s has joined #openstack-infra | 02:35 | |
| cloudnull | so I got that going for me, which is nice :) | 02:35 |
| mordred | cloudnull: at least you have something going for you | 02:35 |
| *** hongbin has joined #openstack-infra | 02:35 | |
| clarkb | you can depends on the workarou d change to see if that fixes it | 02:35 |
| *** esberglu has quit IRC | 02:35 | |
| mordred | I think I already am depends-on that change :) | 02:35 |
| cloudnull | mordred: this is true. | 02:36 |
| * cloudnull drinks more more of that half full glass. :) | 02:36 | |
| mordred | oh - wait - no, I'm not - that was a nodepool change I made depends-on the workaround | 02:36 |
| mordred | I think | 02:36 |
| *** yamahata has quit IRC | 02:36 | |
| cloudnull | s/more more/some more/ | 02:36 |
| EmilienM | clarkb: can you confirm that experimental jobs are non-voting anyway (without -nv thing) ? | 02:38 |
| *** david-lyle has quit IRC | 02:38 | |
| jroll | EmilienM: they are | 02:39 |
| mordred | EmilienM: yah. if they're in the experimental pipeline, they only report back results, they don't vote | 02:39 |
| EmilienM | thanks :-) | 02:39 |
| EmilienM | what do you think of this kind of change? https://review.openstack.org/#/c/381322/ | 02:40 |
| *** amitgandhinz has quit IRC | 02:43 | |
| *** tqtran has joined #openstack-infra | 02:43 | |
| *** tqtran has quit IRC | 02:48 | |
| *** thorst has joined #openstack-infra | 02:48 | |
| *** david-lyle has joined #openstack-infra | 02:49 | |
| *** dimtruck is now known as zz_dimtruck | 02:52 | |
| *** zz_dimtruck is now known as dimtruck | 02:52 | |
| *** david-lyle has quit IRC | 02:55 | |
| *** amotoki_ has joined #openstack-infra | 02:55 | |
| *** sputnik13 has quit IRC | 02:57 | |
| *** thorst has quit IRC | 02:57 | |
| *** sputnik13 has joined #openstack-infra | 02:57 | |
| openstackgerrit | Jonathan Como proposed openstack-infra/jenkins-job-builder: Implement a publisher for HockeyApp https://review.openstack.org/371178 | 02:58 |
| *** amotoki has quit IRC | 02:58 | |
| *** caowei has joined #openstack-infra | 03:00 | |
| *** caowei has quit IRC | 03:01 | |
| *** dimtruck is now known as zz_dimtruck | 03:02 | |
| *** tqtran has joined #openstack-infra | 03:02 | |
| openstackgerrit | Jonathan Como proposed openstack-infra/jenkins-job-builder: Fix reference repo configuration for Git SCM https://review.openstack.org/371181 | 03:03 |
| *** tuannguyen has quit IRC | 03:05 | |
| openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate: Support an IPv6 underlay network https://review.openstack.org/343041 | 03:06 |
| *** tqtran has quit IRC | 03:06 | |
| *** ijw has joined #openstack-infra | 03:10 | |
| *** links has joined #openstack-infra | 03:10 | |
| *** ijw_ has joined #openstack-infra | 03:10 | |
| *** 7F1ACHYUT has joined #openstack-infra | 03:12 | |
| *** ijw has quit IRC | 03:14 | |
| *** Swami__ has quit IRC | 03:14 | |
| *** sflanigan_afk is now known as sflanigan | 03:15 | |
| *** hongbin has quit IRC | 03:19 | |
| *** baoli has quit IRC | 03:20 | |
| *** kaisers_ has joined #openstack-infra | 03:27 | |
| *** sputnik13 has quit IRC | 03:27 | |
| *** sputnik13 has joined #openstack-infra | 03:27 | |
| openstackgerrit | Jonathan Como proposed openstack-infra/jenkins-job-builder: Implement a publisher for HockeyApp https://review.openstack.org/371178 | 03:27 |
| *** spzala has quit IRC | 03:27 | |
| *** kaisers_ has quit IRC | 03:32 | |
| *** kzaitsev_mb has quit IRC | 03:33 | |
| *** EricGonczer_ has quit IRC | 03:38 | |
| *** amitgandhinz has joined #openstack-infra | 03:39 | |
| *** jamesdenton has quit IRC | 03:42 | |
| *** tiswanso has quit IRC | 03:45 | |
| *** 7F1ACHYUT has quit IRC | 03:52 | |
| sdake | hey folks, in kolla we are splitting repos of kolla into a couple of repos | 03:53 |
| sdake | i am curiosu about the upstream: flag - wil lthat copy the branches intact? | 03:53 |
| sdake | (I dont want the branches copied intact in some cases in others do) | 03:53 |
| *** vikrant has joined #openstack-infra | 03:54 | |
| clarkb | it will so prep thr branches the way you want them to turn out | 03:54 |
| clarkb | tags too | 03:54 |
| *** thorst has joined #openstack-infra | 03:55 | |
| *** yuanying has quit IRC | 04:01 | |
| *** yuanying_ has joined #openstack-infra | 04:01 | |
| *** thorst has quit IRC | 04:02 | |
| *** mtanino has quit IRC | 04:03 | |
| *** yamamoto_ has joined #openstack-infra | 04:11 | |
| *** lezbar has quit IRC | 04:13 | |
| *** amitgandhinz has quit IRC | 04:14 | |
| *** lezbar has joined #openstack-infra | 04:14 | |
| *** amotoki has joined #openstack-infra | 04:18 | |
| *** amotoki_ has quit IRC | 04:21 | |
| *** salv-orlando has joined #openstack-infra | 04:29 | |
| *** salv-orlando has quit IRC | 04:34 | |
| *** adriant has quit IRC | 04:35 | |
| *** senk__ has joined #openstack-infra | 04:39 | |
| *** senk__ has quit IRC | 04:39 | |
| *** sputnik13 has quit IRC | 04:40 | |
| *** woodster_ has quit IRC | 04:40 | |
| *** Sukhdev has joined #openstack-infra | 04:41 | |
| *** pgadiya has joined #openstack-infra | 04:43 | |
| *** rbrndt has joined #openstack-infra | 04:44 | |
| *** sputnik13 has joined #openstack-infra | 04:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:49 | |
| *** lezbar has quit IRC | 04:50 | |
| *** lezbar has joined #openstack-infra | 04:50 | |
| *** narayrak has joined #openstack-infra | 04:52 | |
| *** tuannguyen has joined #openstack-infra | 04:52 | |
| *** tuannguyen has quit IRC | 04:58 | |
| *** rwsu has quit IRC | 04:59 | |
| *** sputnik13 has quit IRC | 05:01 | |
| *** thorst has joined #openstack-infra | 05:01 | |
| *** Ravikiran_K has joined #openstack-infra | 05:05 | |
| *** salv-orlando has joined #openstack-infra | 05:07 | |
| *** thorst has quit IRC | 05:08 | |
| *** amitgandhinz has joined #openstack-infra | 05:10 | |
| *** kaisers_ has joined #openstack-infra | 05:14 | |
| *** spzala has joined #openstack-infra | 05:28 | |
| *** rbrndt has quit IRC | 05:32 | |
| *** spzala has quit IRC | 05:32 | |
| *** Sukhdev has quit IRC | 05:35 | |
| *** sputnik13 has joined #openstack-infra | 05:38 | |
| *** sandanar has joined #openstack-infra | 05:40 | |
| *** nadya has joined #openstack-infra | 05:42 | |
| *** esikachev has joined #openstack-infra | 05:42 | |
| *** amitgandhinz has quit IRC | 05:45 | |
| *** jaosorior has joined #openstack-infra | 05:46 | |
| *** esikachev has quit IRC | 05:47 | |
| *** tnovacik has joined #openstack-infra | 05:48 | |
| *** flaper87 has joined #openstack-infra | 05:49 | |
| *** flaper87 has quit IRC | 05:49 | |
| *** flaper87 has joined #openstack-infra | 05:49 | |
| *** armax has quit IRC | 05:49 | |
| *** ArxCruz has joined #openstack-infra | 05:51 | |
| *** e0ne has joined #openstack-infra | 05:52 | |
| *** ArxCruz has quit IRC | 05:52 | |
| *** ArxCruz has joined #openstack-infra | 05:53 | |
| *** adrian_otto has joined #openstack-infra | 05:53 | |
| *** ArxCruz has quit IRC | 05:54 | |
| *** ArxCruz has joined #openstack-infra | 05:54 | |
| *** ArxCruz has quit IRC | 05:54 | |
| *** ArxCruz has joined #openstack-infra | 05:55 | |
| *** tuannguyen has joined #openstack-infra | 05:55 | |
| *** ArxCruz has quit IRC | 05:55 | |
| *** ArxCruz has joined #openstack-infra | 05:55 | |
| *** ArxCruz has quit IRC | 05:56 | |
| *** ArxCruz has joined #openstack-infra | 05:56 | |
| *** ArxCruz has quit IRC | 05:57 | |
| *** ArxCruz has joined #openstack-infra | 05:57 | |
| *** ArxCruz has quit IRC | 05:58 | |
| *** ArxCruz has joined #openstack-infra | 05:58 | |
| *** ArxCruz has quit IRC | 05:58 | |
| *** ArxCruz has joined #openstack-infra | 05:59 | |
| *** yamahata has joined #openstack-infra | 05:59 | |
| *** ArxCruz has quit IRC | 05:59 | |
| *** tuannguyen has quit IRC | 05:59 | |
| *** ArxCruz has joined #openstack-infra | 06:00 | |
| *** john-davidge has joined #openstack-infra | 06:00 | |
| *** ArxCruz has quit IRC | 06:00 | |
| *** ArxCruz has joined #openstack-infra | 06:00 | |
| *** ArxCruz has quit IRC | 06:01 | |
| *** ArxCruz has joined #openstack-infra | 06:01 | |
| *** ArxCruz has quit IRC | 06:02 | |
| *** ArxCruz has joined #openstack-infra | 06:02 | |
| *** ArxCruz has quit IRC | 06:02 | |
| *** ArxCruz has joined #openstack-infra | 06:03 | |
| *** ArxCruz has quit IRC | 06:03 | |
| *** drifterza has quit IRC | 06:03 | |
| *** ArxCruz has joined #openstack-infra | 06:04 | |
| *** ArxCruz has quit IRC | 06:04 | |
| *** ArxCruz has joined #openstack-infra | 06:04 | |
| *** thorst has joined #openstack-infra | 06:05 | |
| *** ArxCruz has quit IRC | 06:05 | |
| *** john-davidge has quit IRC | 06:05 | |
| *** ArxCruz has joined #openstack-infra | 06:05 | |
| *** tqtran has joined #openstack-infra | 06:05 | |
| *** ArxCruz has quit IRC | 06:06 | |
| *** watersoul has quit IRC | 06:06 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Add linters to the governance gate https://review.openstack.org/381425 | 06:06 |
| *** ArxCruz has joined #openstack-infra | 06:06 | |
| *** watersoul has joined #openstack-infra | 06:06 | |
| *** ArxCruz has quit IRC | 06:07 | |
| *** ArxCruz has joined #openstack-infra | 06:07 | |
| *** ArxCruz has quit IRC | 06:07 | |
| *** nadya has quit IRC | 06:08 | |
| *** ArxCruz has joined #openstack-infra | 06:08 | |
| *** ArxCruz has quit IRC | 06:08 | |
| *** ArxCruz has joined #openstack-infra | 06:08 | |
| *** armax has joined #openstack-infra | 06:09 | |
| *** ArxCruz has quit IRC | 06:09 | |
| *** tqtran has quit IRC | 06:10 | |
| *** thorst has quit IRC | 06:12 | |
| *** armax has quit IRC | 06:12 | |
| *** armax has joined #openstack-infra | 06:12 | |
| *** armax has quit IRC | 06:13 | |
| *** Jeffrey4l__ has quit IRC | 06:14 | |
| *** Jeffrey4l__ has joined #openstack-infra | 06:14 | |
| *** Jeffrey4l__ has quit IRC | 06:14 | |
| *** Jeffrey4l_ has joined #openstack-infra | 06:15 | |
| *** Jeffrey4l_ is now known as Jeffrey4l | 06:15 | |
| *** winggundamth has quit IRC | 06:17 | |
| *** sputnik13 has quit IRC | 06:20 | |
| *** salv-orlando has quit IRC | 06:21 | |
| *** esikachev has joined #openstack-infra | 06:22 | |
| *** AJaeger has quit IRC | 06:23 | |
| *** esikachev has quit IRC | 06:26 | |
| *** esikachev has joined #openstack-infra | 06:29 | |
| *** Ravikiran_K has quit IRC | 06:29 | |
| *** tqtran has joined #openstack-infra | 06:31 | |
| *** nadya has joined #openstack-infra | 06:33 | |
| *** esikachev has quit IRC | 06:34 | |
| *** tqtran has quit IRC | 06:35 | |
| sdake | clarkb have to hit rack | 06:36 |
| sdake | clarkb can we chat in morning | 06:36 |
| sdake | clarkb i didn't quite parse your msg - been up since 4am working with contractors | 06:36 |
| sdake | and its 11 | 06:36 |
| sdake | that could be part of the problem :) | 06:36 |
| openstackgerrit | Merged openstack-infra/project-config: Shorten OSA role job names https://review.openstack.org/381185 | 06:38 |
| *** tqtran has joined #openstack-infra | 06:38 | |
| *** AJaeger has joined #openstack-infra | 06:39 | |
| AJaeger | jeblair: we try already publishing of developer.openstack.org document to AFS - a side effect of adding the afs publisher to some macros. Please check http://logs.openstack.org/e6/e6e7d4a523fef49cb44efba6e1763b2dfcc44d2b/post/ironic-api-ref/a40b53f/_zuul_ansible/ansible_log.txt at bottom. | 06:40 |
| *** amitgandhinz has joined #openstack-infra | 06:41 | |
| AJaeger | jeblair: do you want to enable that directory for developer.o.o as well, ignore this for now - or should we fix the macro so that this only works with docs.o.o? | 06:41 |
| *** lezbar has quit IRC | 06:42 | |
| *** lezbar has joined #openstack-infra | 06:42 | |
| *** e0ne has quit IRC | 06:43 | |
| *** amotoki has quit IRC | 06:43 | |
| *** andreas_s has joined #openstack-infra | 06:43 | |
| *** rcernin has joined #openstack-infra | 06:43 | |
| *** e0ne has joined #openstack-infra | 06:44 | |
| *** winggundamth has joined #openstack-infra | 06:44 | |
| *** nadya has quit IRC | 06:45 | |
| *** adrian_otto has quit IRC | 06:45 | |
| *** ccamacho has joined #openstack-infra | 06:47 | |
| *** chem has joined #openstack-infra | 06:48 | |
| *** adrian_otto has joined #openstack-infra | 06:48 | |
| *** amotoki has joined #openstack-infra | 06:49 | |
| *** rlu has joined #openstack-infra | 06:49 | |
| *** pilgrimstack has quit IRC | 06:50 | |
| *** tesseract- has joined #openstack-infra | 06:52 | |
| *** pilgrimstack has joined #openstack-infra | 06:53 | |
| *** e0ne has quit IRC | 06:55 | |
| *** rlu has quit IRC | 06:56 | |
| *** yamahata has quit IRC | 06:56 | |
| *** nadya has joined #openstack-infra | 07:00 | |
| *** rlu has joined #openstack-infra | 07:02 | |
| *** senk_ has joined #openstack-infra | 07:04 | |
| *** kzaitsev_mb has quit IRC | 07:04 | |
| *** senk_ has quit IRC | 07:06 | |
| *** esikachev has joined #openstack-infra | 07:08 | |
| *** thorst has joined #openstack-infra | 07:10 | |
| *** salv-orlando has joined #openstack-infra | 07:11 | |
| *** amitgandhinz has quit IRC | 07:14 | |
| *** matrohon has joined #openstack-infra | 07:14 | |
| *** dragonmaster_ has joined #openstack-infra | 07:15 | |
| *** adrian_otto1 has joined #openstack-infra | 07:16 | |
| *** yaume has joined #openstack-infra | 07:16 | |
| *** thorst has quit IRC | 07:16 | |
| *** adrian_otto has quit IRC | 07:19 | |
| *** markvoelker has quit IRC | 07:24 | |
| *** amotoki_ has joined #openstack-infra | 07:24 | |
| *** dtantsur|afk is now known as dtantsur | 07:26 | |
| *** Naeil has joined #openstack-infra | 07:27 | |
| *** amotoki has quit IRC | 07:27 | |
| *** ihrachys has joined #openstack-infra | 07:28 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** Naeil has quit IRC | 07:33 | |
| *** pilgrimstack1 has joined #openstack-infra | 07:34 | |
| *** pilgrimstack has quit IRC | 07:34 | |
| *** adrian_otto1 has quit IRC | 07:37 | |
| *** hashar has joined #openstack-infra | 07:37 | |
| *** adrian_otto has joined #openstack-infra | 07:38 | |
| *** Adri2000 has quit IRC | 07:42 | |
| *** Adri2000 has joined #openstack-infra | 07:43 | |
| *** drifterza has joined #openstack-infra | 07:44 | |
| *** jpich has joined #openstack-infra | 07:44 | |
| *** vincentll has joined #openstack-infra | 07:45 | |
| *** salv-orlando has joined #openstack-infra | 07:46 | |
| *** adrian_otto has quit IRC | 07:46 | |
| vincentll | Is there an issue with Jenkins at the moment? Our patchset is stuck on "Needs verified" eventhough Jenkins gave a +1/verified (https://review.openstack.org/#/c/381092/) | 07:47 |
| *** adrian_otto has joined #openstack-infra | 07:49 | |
| *** jak-jak has joined #openstack-infra | 07:49 | |
| *** Naeil has joined #openstack-infra | 07:49 | |
| *** jak-jak is now known as Guest45220 | 07:49 | |
| *** adrian_otto has quit IRC | 07:50 | |
| *** jtomasek|afk is now known as jtomasek | 07:51 | |
| *** jpena|off is now known as jpena | 07:53 | |
| hashar | vincentll: from ssh -p 29418 review.openstack.org 'gerrit query 381092 --submit-records' | 07:55 |
| hashar | the change shows as status: NOT_READY | 07:55 |
| hashar | Verified: NEED | 07:55 |
| hashar | seems to be a chain of dependencies | 07:56 |
| hashar | ultimately landing on patchset 1 of https://review.openstack.org/#/c/380997/ | 07:56 |
| hashar | which now has a patchset 4. Maybe try to rebase? | 07:56 |
| *** yamahata has joined #openstack-infra | 07:56 | |
| *** tuannguyen has joined #openstack-infra | 07:56 | |
| *** pilgrimstack has joined #openstack-infra | 07:57 | |
| *** jistr has joined #openstack-infra | 07:57 | |
| *** pilgrimstack1 has quit IRC | 07:57 | |
| hashar | and https://review.openstack.org/#/c/380997/ got CR+2/W+1 on patchset 3, but a new patchset got sent which cleared the votes | 07:57 |
| *** yamahata has quit IRC | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** jed56 has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** tuannguyen has quit IRC | 08:01 | |
| *** rwsu has joined #openstack-infra | 08:03 | |
| *** amoralej|off is now known as amoralej | 08:03 | |
| *** jordanP has joined #openstack-infra | 08:06 | |
| *** yamahata has joined #openstack-infra | 08:07 | |
| *** Ravikiran_K has joined #openstack-infra | 08:09 | |
| *** tqtran has quit IRC | 08:09 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** amitgandhinz has joined #openstack-infra | 08:11 | |
| *** salv-orlando has quit IRC | 08:12 | |
| *** yamahata has quit IRC | 08:13 | |
| *** yamahata has joined #openstack-infra | 08:13 | |
| *** thorst has joined #openstack-infra | 08:15 | |
| *** amotoki_ has quit IRC | 08:18 | |
| *** salv-orlando has joined #openstack-infra | 08:19 | |
| vincentll | thanks hashar ! | 08:19 |
| *** amotoki has joined #openstack-infra | 08:21 | |
| *** salv-orlando has quit IRC | 08:21 | |
| *** salv-orlando has joined #openstack-infra | 08:21 | |
| *** thorst has quit IRC | 08:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:24 | |
| *** markvoelker has joined #openstack-infra | 08:24 | |
| *** markvoelker has quit IRC | 08:29 | |
| *** tosky has joined #openstack-infra | 08:31 | |
| *** kzaitsev_mb has quit IRC | 08:36 | |
| *** yamahata has quit IRC | 08:37 | |
| *** admcleod has joined #openstack-infra | 08:38 | |
| *** admcleod has joined #openstack-infra | 08:38 | |
| *** admcleod_ has quit IRC | 08:39 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:40 | |
| dragonmaster_ | morning all | 08:41 |
| *** amitgandhinz has quit IRC | 08:45 | |
| yolanda | good morning | 08:48 |
| *** mhickey has joined #openstack-infra | 08:48 | |
| *** ijw_ has quit IRC | 08:49 | |
| *** dtantsur is now known as dtantsur|bbl | 08:52 | |
| *** dtardivel has joined #openstack-infra | 08:52 | |
| *** kzaitsev_mb has quit IRC | 08:53 | |
| *** dizquierdo has joined #openstack-infra | 08:54 | |
| dragonmaster_ | anyone that could possibly point me to a zuul setup / deployment doc? we’re trying to implement the openstack-infra gating solution for our internal projects | 08:54 |
| *** john-davidge has joined #openstack-infra | 08:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/381559 | 08:55 |
| *** electrofelix has joined #openstack-infra | 09:00 | |
| AJaeger | dragonmaster_: http://docs.openstack.org/infra/system-config/third_party.html - and there's a third party meetign every week that you should attend | 09:02 |
| AJaeger | dragonmaster_: see http://eavesdrop.openstack.org/#Third_Party_Working_Group_Meeting | 09:03 |
| dragonmaster_ | cool, thanks AJaeger | 09:03 |
| *** derekh has joined #openstack-infra | 09:04 | |
| *** strigazi_AFK is now known as strigazi | 09:04 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Disable building ramdisk for ironic-multitest job. https://review.openstack.org/381570 | 09:05 |
| *** kzaitsev_mb has joined #openstack-infra | 09:06 | |
| *** verdurin_ is now known as verdurin | 09:07 | |
| AJaeger | fungi, looking at deb-glare, the upstream repo points master to " remotes/origin/HEAD -> origin/debian/newton" | 09:11 |
| AJaeger | fungi, zigo, this is now the 3rd or fourth repository that failed to import with our tools and had this setup. | 09:11 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Move the opensuse mkinitrd script to the zypper element https://review.openstack.org/381574 | 09:11 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element https://review.openstack.org/381575 | 09:11 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element https://review.openstack.org/381576 | 09:11 |
| AJaeger | fungi, is there a bug in our import - or should we require that HEAD points to master? | 09:12 |
| openstackgerrit | Pawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade https://review.openstack.org/364809 | 09:12 |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting python3 jobs to requirements https://review.openstack.org/381357 | 09:14 |
| openstackgerrit | Merged openstack-infra/project-config: Update the gate-tempest-neutron-dsvm-src-oslo.messaging test https://review.openstack.org/371502 | 09:20 |
| *** thorst has joined #openstack-infra | 09:20 | |
| openstackgerrit | Merged openstack-infra/project-config: layout: update tempest jobs coverage for puppet https://review.openstack.org/381285 | 09:20 |
| *** sambetts|afk is now known as sambetts | 09:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Create glare-specs project https://review.openstack.org/370749 | 09:26 |
| *** thorst has quit IRC | 09:27 | |
| vsaienk0 | AJaeger, yolanda: please help to merge https://review.openstack.org/#/c/381570/ 1 line patch that disables building ramdisk for ironic-multitest experimental job. | 09:29 |
| AJaeger | vsaienk0: I would appreciate dtantsur|bbl or jroll to +1 it. | 09:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add new project: devstack-plugin-libvirt-qemu https://review.openstack.org/377920 | 09:32 |
| openstackgerrit | Merged openstack-infra/project-config: Kolla Kubernetes Ansible test https://review.openstack.org/380848 | 09:32 |
| *** ijw has joined #openstack-infra | 09:35 | |
| *** lezbar has quit IRC | 09:36 | |
| *** lezbar has joined #openstack-infra | 09:36 | |
| *** bhavik1 has joined #openstack-infra | 09:39 | |
| *** ramishra has quit IRC | 09:39 | |
| *** ijw has quit IRC | 09:40 | |
| *** amitgandhinz has joined #openstack-infra | 09:41 | |
| *** ramishra has joined #openstack-infra | 09:42 | |
| *** shardy has joined #openstack-infra | 09:47 | |
| *** dtantsur|bbl is now known as dtantsur | 09:50 | |
| *** amotoki has quit IRC | 09:51 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element https://review.openstack.org/381575 | 09:57 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element https://review.openstack.org/381576 | 09:57 |
| vsaienk0 | AJaeger, just got +1 from Dmitry https://review.openstack.org/#/c/381570/ | 09:57 |
| *** tuannguyen has joined #openstack-infra | 09:58 | |
| AJaeger | vsaienk0: then you have my +2 ;) | 09:59 |
| *** jistr is now known as jistr|mtg | 09:59 | |
| vsaienk0 | AJaeger: thanks! | 09:59 |
| *** zigo has quit IRC | 10:01 | |
| *** tuannguyen has quit IRC | 10:02 | |
| *** zigo has joined #openstack-infra | 10:04 | |
| *** zigo is now known as Guest84780 | 10:05 | |
| *** amotoki has joined #openstack-infra | 10:07 | |
| *** dizquierdo has quit IRC | 10:11 | |
| *** senk_ has joined #openstack-infra | 10:11 | |
| *** senk_ has quit IRC | 10:12 | |
| *** Guest84780 has quit IRC | 10:14 | |
| *** senk_ has joined #openstack-infra | 10:15 | |
| *** amitgandhinz has quit IRC | 10:16 | |
| *** zigo_ has joined #openstack-infra | 10:16 | |
| *** senk_ has quit IRC | 10:21 | |
| *** _degorenko|afk is now known as degorenko | 10:22 | |
| *** thorst has joined #openstack-infra | 10:25 | |
| *** markvoelker has joined #openstack-infra | 10:26 | |
| *** yamamoto_ has quit IRC | 10:27 | |
| *** kzaitsev_mb has quit IRC | 10:27 | |
| *** vincentll has quit IRC | 10:27 | |
| *** number80 has quit IRC | 10:28 | |
| *** markvoelker has quit IRC | 10:31 | |
| *** hashar is now known as hasharFood | 10:31 | |
| *** thorst has quit IRC | 10:32 | |
| *** ijw has joined #openstack-infra | 10:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces https://review.openstack.org/380142 | 10:36 |
| *** amotoki has quit IRC | 10:39 | |
| *** amotoki has joined #openstack-infra | 10:40 | |
| *** ijw has quit IRC | 10:42 | |
| *** salv-orlando has quit IRC | 10:42 | |
| *** spzala has joined #openstack-infra | 10:44 | |
| *** kaisers has quit IRC | 10:46 | |
| *** kaisers2 has quit IRC | 10:46 | |
| *** EricGonczer_ has joined #openstack-infra | 10:48 | |
| *** spzala has quit IRC | 10:49 | |
| *** bhavik1 has quit IRC | 10:50 | |
| *** jkilpatr has quit IRC | 10:52 | |
| *** jkilpatr has joined #openstack-infra | 10:53 | |
| *** ralonsoh has joined #openstack-infra | 10:53 | |
| *** psilvad has joined #openstack-infra | 10:54 | |
| ralonsoh | Hi. I need to be added as member to devstack-plugin-libvirt-qemu-core and devstack-plugin-libvirt-qemu-release. My email is rodolfo.alonso.hernande@intel.com | 10:54 |
| ralonsoh | Sorry, my mail is rodolfo.alonso.hernandez@intel.com | 10:54 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Rework ironic-lib tempest jobs https://review.openstack.org/381652 | 10:55 |
| dtantsur | jroll, ^^^ | 10:55 |
| *** kzaitsev_mb has joined #openstack-infra | 10:57 | |
| *** rcernin has quit IRC | 10:57 | |
| *** rcernin has joined #openstack-infra | 10:58 | |
| *** psilvad has quit IRC | 11:00 | |
| *** jistr|mtg is now known as jistr | 11:00 | |
| *** dprince has joined #openstack-infra | 11:01 | |
| *** nadya has quit IRC | 11:01 | |
| *** coolsvap is now known as coolsvap_ | 11:01 | |
| *** jkilpatr has quit IRC | 11:03 | |
| *** Naeil has quit IRC | 11:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add option to disable default routes https://review.openstack.org/381660 | 11:08 |
| *** rtheis has joined #openstack-infra | 11:09 | |
| *** jpena is now known as jpena|lunch | 11:09 | |
| *** senk_ has joined #openstack-infra | 11:10 | |
| *** tpsilva has joined #openstack-infra | 11:11 | |
| *** amitgandhinz has joined #openstack-infra | 11:12 | |
| *** amoralej is now known as amoralej|lunch | 11:13 | |
| *** baoli has joined #openstack-infra | 11:14 | |
| *** ccamacho is now known as ccamacho|lunch | 11:14 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add zypper-minimal element https://review.openstack.org/381575 | 11:14 |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element https://review.openstack.org/381576 | 11:14 |
| *** baoli has quit IRC | 11:15 | |
| *** rhallisey has joined #openstack-infra | 11:15 | |
| *** sputnik13 has joined #openstack-infra | 11:18 | |
| *** Naeil has joined #openstack-infra | 11:18 | |
| *** jkilpatr has joined #openstack-infra | 11:19 | |
| *** kashyap has left #openstack-infra | 11:21 | |
| *** breton has joined #openstack-infra | 11:22 | |
| *** zigo_ is now known as zigo | 11:23 | |
| *** gildub has quit IRC | 11:24 | |
| *** sputnik13 has quit IRC | 11:25 | |
| *** EricGonczer_ has quit IRC | 11:26 | |
| *** sputnik13 has joined #openstack-infra | 11:27 | |
| *** Naeil has quit IRC | 11:27 | |
| *** markvoelker has joined #openstack-infra | 11:27 | |
| *** markvoelker has quit IRC | 11:32 | |
| *** mhickey has quit IRC | 11:32 | |
| *** ijw has joined #openstack-infra | 11:38 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton https://review.openstack.org/381309 | 11:38 |
| *** Naeil has joined #openstack-infra | 11:39 | |
| *** thorst has joined #openstack-infra | 11:40 | |
| *** asselin__ has joined #openstack-infra | 11:40 | |
| *** EricGonczer_ has joined #openstack-infra | 11:40 | |
| *** breton has left #openstack-infra | 11:40 | |
| *** salv-orlando has joined #openstack-infra | 11:41 | |
| *** Goneri has quit IRC | 11:43 | |
| *** ijw has quit IRC | 11:44 | |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| *** salv-orlando has quit IRC | 11:46 | |
| *** amitgandhinz has quit IRC | 11:46 | |
| *** rfolco has joined #openstack-infra | 11:47 | |
| *** esikachev has quit IRC | 11:48 | |
| *** ldnunes has quit IRC | 11:55 | |
| *** baoli has joined #openstack-infra | 11:55 | |
| *** nadya has joined #openstack-infra | 11:56 | |
| *** ldnunes has joined #openstack-infra | 11:57 | |
| *** amotoki has quit IRC | 11:57 | |
| *** tuannguyen has joined #openstack-infra | 11:59 | |
| *** lucas-afk is now known as lucasagomes | 11:59 | |
| *** krtaylor has quit IRC | 12:01 | |
| *** moravec has quit IRC | 12:01 | |
| *** flepied has quit IRC | 12:04 | |
| *** tuannguyen has quit IRC | 12:04 | |
| *** tuanluong has quit IRC | 12:05 | |
| *** hasharFood is now known as hashar | 12:06 | |
| *** mhickey has joined #openstack-infra | 12:09 | |
| *** raildo has joined #openstack-infra | 12:09 | |
| *** psilvad has joined #openstack-infra | 12:10 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces https://review.openstack.org/380142 | 12:12 |
| *** amotoki has joined #openstack-infra | 12:12 | |
| openstackgerrit | Erlon R. Cruz proposed openstack-infra/project-config: Adds new experimental Cinder migration job https://review.openstack.org/381737 | 12:14 |
| *** kevinbenton has quit IRC | 12:15 | |
| *** furlongm has quit IRC | 12:15 | |
| *** Ravikiran_K has quit IRC | 12:15 | |
| *** furlongm has joined #openstack-infra | 12:16 | |
| *** kevinbenton has joined #openstack-infra | 12:17 | |
| *** psilvad has quit IRC | 12:17 | |
| *** arxcruz has joined #openstack-infra | 12:17 | |
| *** esikachev has joined #openstack-infra | 12:17 | |
| *** kashyap has joined #openstack-infra | 12:18 | |
| *** edmondsw has joined #openstack-infra | 12:18 | |
| kashyap | Is Gerrit down, or is it just me? | 12:18 |
| AJaeger | kashyap: just checked - works for me. | 12:18 |
| *** akshai has joined #openstack-infra | 12:18 | |
| kashyap | AJaeger: Thanks, indeed. It's my VPN that's behaving | 12:19 |
| *** flepied has joined #openstack-infra | 12:20 | |
| *** esikachev has quit IRC | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:22 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Add opensuse-minimal element https://review.openstack.org/381576 | 12:22 |
| AJaeger | kashyap: that's sad to hear, hope you can heal it | 12:23 |
| dtantsur | jroll, are you ok with switching ironic-lib jobs to xenial completely? AJaeger demands it, and honestly I'm tired of arguing | 12:24 |
| jroll | dtantsur: why not? | 12:25 |
| dtantsur | jroll, we don't run many jobs on xenial so far, so this might cause a breakage on master and/or newton | 12:25 |
| jroll | dtantsur: if the iscsi jobs work on xenial, I doubt others would break too hard | 12:26 |
| dtantsur | jroll, we only run postgres job on xenial now.. and if we switch ironic-lib, maybe we should switch ironic and IPA too... though I've no clue if image buildings works out of box on Xenial.. | 12:27 |
| jroll | dtantsur: good point... ironic-lib is pretty low volume, it wouldn't be much trouble if it broke there first | 12:28 |
| AJaeger | dtantsur: I ask for it for new jobs on master. | 12:28 |
| dtantsur | AJaeger, well, for some definition of "new"... | 12:28 |
| jroll | dtantsur: I believe that IPA gate is still down due to the problem with osic cloud external networking | 12:28 |
| *** salv-orlando has joined #openstack-infra | 12:28 | |
| dtantsur | ugh, wonderful.. I guess switch to xenial won't hurt it any more | 12:29 |
| dtantsur | AJaeger, next time I'd love to see something in place to avoid duplicating all the jobs with -xenial and -whatever prefixes | 12:30 |
| *** mriedem has joined #openstack-infra | 12:30 | |
| *** ralonsoh_ has joined #openstack-infra | 12:30 | |
| kashyap | AJaeger: Hehe, healed (with a crude workaround, restarted OpenVPN :-( ) | 12:30 |
| AJaeger | dtantsur: we all hoped it would be easier ;( We had a long discussion on the pros and cons. In the past we did some magic and switched directly all jobs on stable/X to old Ubuntu version and everything newer to new Ubuntu version. But that surprised quite a few.... | 12:31 |
| pabelanger | hopefully zuulv3 will make this easier next time we need to add a new OS to master only | 12:32 |
| dtantsur | I'd prefer to be surprised in this case, rather than figuring it out in every patch I make while trying to fix our CI situation | 12:32 |
| *** markvoelker has joined #openstack-infra | 12:33 | |
| *** ralonsoh has quit IRC | 12:34 | |
| *** yamamoto has joined #openstack-infra | 12:34 | |
| AJaeger | dtantsur: understood. The other solution had different pitfalls. Let's see what options we have next time... | 12:34 |
| *** akshai has quit IRC | 12:35 | |
| *** number80 has joined #openstack-infra | 12:36 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces https://review.openstack.org/380142 | 12:36 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Add option to disable default routes https://review.openstack.org/381660 | 12:36 |
| *** fguillot has joined #openstack-infra | 12:37 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Move networking-ovn multinode job to check queue https://review.openstack.org/378032 | 12:37 |
| *** amoralej|lunch is now known as amoralej | 12:37 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Move networking-ovn multinode job to check queue https://review.openstack.org/378032 | 12:39 |
| *** ijw has joined #openstack-infra | 12:40 | |
| *** pgadiya has quit IRC | 12:41 | |
| *** burgerk has joined #openstack-infra | 12:41 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config: Add gate jobs for RHEL 7 STIG testing https://review.openstack.org/381775 | 12:41 |
| *** senk_ has quit IRC | 12:42 | |
| *** jpena|lunch is now known as jpena | 12:42 | |
| *** ccamacho|lunch is now known as ccamacho | 12:42 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix test_queries regression https://review.openstack.org/381776 | 12:42 |
| *** amitgandhinz has joined #openstack-infra | 12:43 | |
| *** yolanda has quit IRC | 12:44 | |
| *** baoli_ has joined #openstack-infra | 12:44 | |
| *** tuannguyen has joined #openstack-infra | 12:45 | |
| *** ijw has quit IRC | 12:45 | |
| *** EricGonczer_ has quit IRC | 12:46 | |
| *** yolanda has joined #openstack-infra | 12:46 | |
| *** drawsmcgraw has left #openstack-infra | 12:46 | |
| *** baoli has quit IRC | 12:47 | |
| dtantsur | jroll, do we want to switch to ipmitool jobs simultaneously? | 12:47 |
| *** amitgandhinz has quit IRC | 12:47 | |
| jroll | dtantsur: one part of me says one thing at a time, the other says we're overdue to switch | 12:47 |
| dtantsur | jroll, well, one thing at a time is good, but in this case I'll have to duplicate all jobs in this patch or ignore ipmitool for now | 12:48 |
| jroll | dtantsur: yeah let's just do it | 12:48 |
| *** amotoki has quit IRC | 12:49 | |
| *** jcoufal has joined #openstack-infra | 12:50 | |
| dtantsur | cool | 12:50 |
| mriedem | are jobs in the experimental queue automatically non-voting? | 12:51 |
| mriedem | i think they are | 12:51 |
| AJaeger | mriedem: implicitely ;) | 12:51 |
| mriedem | ok | 12:51 |
| AJaeger | they are voting jobs but the vote is ignored ;) | 12:52 |
| *** gordc has joined #openstack-infra | 12:52 | |
| AJaeger | So, "check experimental" does not change the Jenkins vote one way or the other. | 12:52 |
| *** yamamoto has quit IRC | 12:53 | |
| openstackgerrit | Cyril Roelandt proposed openstack-dev/hacking: Add a check to make sure the right assert* method is used https://review.openstack.org/354185 | 12:53 |
| *** amotoki has joined #openstack-infra | 12:54 | |
| *** ralonsoh_ is now known as ralonsoh | 12:54 | |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| ralonsoh | Hola yolanda. I need to be added as member to devstack-plugin-libvirt-qemu-core and devstack-plugin-libvirt-qemu-release. My email is rodolfo.alonso.hernandez@intel.com | 12:55 |
| ralonsoh | Thanks! | 12:55 |
| yolanda | hola ralonsoh :) ok, going to add you | 12:56 |
| ralonsoh | yolanda gracias | 12:56 |
| *** shardy is now known as shardy_mtg | 12:56 | |
| yolanda | ralonsoh, de nada, hecho | 12:58 |
| *** chandankumar has joined #openstack-infra | 12:59 | |
| *** moravec has joined #openstack-infra | 13:01 | |
| *** akshai has joined #openstack-infra | 13:01 | |
| *** Goneri has joined #openstack-infra | 13:02 | |
| *** yamamoto has quit IRC | 13:02 | |
| *** akshai has quit IRC | 13:02 | |
| *** yolanda has quit IRC | 13:02 | |
| *** moravec has quit IRC | 13:02 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config: Add gate jobs for RHEL 7 STIG testing https://review.openstack.org/381775 | 13:03 |
| *** amitgandhinz has joined #openstack-infra | 13:03 | |
| *** amitgandhinz has quit IRC | 13:04 | |
| *** akshai has joined #openstack-infra | 13:04 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add check-tripleo-undercloud-jobs template https://review.openstack.org/381790 | 13:04 |
| *** amitgandhinz has joined #openstack-infra | 13:05 | |
| pabelanger | mgagne: morning, we are seeing some issues with internap-mtl01 this morning, can you confirm? | 13:05 |
| pabelanger | outage maybe? | 13:05 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Bifurcate neutron-fwaas tempest gate for v2 https://review.openstack.org/371749 | 13:06 |
| *** yolanda has joined #openstack-infra | 13:07 | |
| *** spzala has joined #openstack-infra | 13:07 | |
| *** BobBall has quit IRC | 13:07 | |
| *** arxcruz has quit IRC | 13:07 | |
| *** fguillot has quit IRC | 13:08 | |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add check-tripleo-undercloud-jobs template https://review.openstack.org/381790 | 13:08 |
| *** mdrabe has joined #openstack-infra | 13:09 | |
| *** vikrant has quit IRC | 13:11 | |
| *** esikachev has joined #openstack-infra | 13:11 | |
| *** rbrndt has joined #openstack-infra | 13:15 | |
| *** akshai has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:17 | |
| *** mriedem has quit IRC | 13:18 | |
| AJaeger | pabelanger: we still have both http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rate and http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rates - could you delete the older one, please | 13:19 |
| *** arxcruz has joined #openstack-infra | 13:19 | |
| *** yamamoto has quit IRC | 13:20 | |
| *** akshai has joined #openstack-infra | 13:20 | |
| *** tiswanso has joined #openstack-infra | 13:21 | |
| *** lseki has joined #openstack-infra | 13:23 | |
| openstackgerrit | Hironori Shiina proposed openstack/diskimage-builder: Fix a command in Developer Documentation https://review.openstack.org/381833 | 13:24 |
| *** yamamoto has joined #openstack-infra | 13:24 | |
| *** yamamoto has quit IRC | 13:25 | |
| *** chandankumar has quit IRC | 13:25 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:28 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 13:29 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Fix test_queries regression https://review.openstack.org/381776 | 13:29 |
| *** ralonsoh has quit IRC | 13:30 | |
| *** ralonsoh_ has joined #openstack-infra | 13:30 | |
| *** daemontool has joined #openstack-infra | 13:33 | |
| *** jaosorior has quit IRC | 13:34 | |
| *** jaosorior has joined #openstack-infra | 13:34 | |
| *** esikachev has quit IRC | 13:35 | |
| *** matt-borland has joined #openstack-infra | 13:35 | |
| *** pilgrimstack has quit IRC | 13:36 | |
| EmilienM | Gerrit is super slow for me, maybe I'm wrong | 13:36 |
| dtantsur | for me too, waiting for my patch to be submitted | 13:36 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Rework ironic-lib tempest jobs https://review.openstack.org/381652 | 13:36 |
| dtantsur | jroll, AJaeger ^^^ | 13:36 |
| dtantsur | "502 proxy error in web UI" | 13:37 |
| *** yamamoto has quit IRC | 13:37 | |
| jroll | dtantsur: why would you tell me to click that when gerrit is so slow :P | 13:38 |
| dtantsur | hehe :) | 13:38 |
| jroll | got it now, will review | 13:38 |
| zigo | me too ... | 13:38 |
| *** amotoki has quit IRC | 13:38 | |
| *** rodrigods has quit IRC | 13:40 | |
| *** rodrigods has joined #openstack-infra | 13:40 | |
| *** ijw has joined #openstack-infra | 13:41 | |
| *** tuannguyen has quit IRC | 13:41 | |
| *** chandankumar has joined #openstack-infra | 13:41 | |
| *** tuannguyen has joined #openstack-infra | 13:42 | |
| ianychoi | me too.... | 13:42 |
| frickler | infra-core: gerrit looks like it might be flatlining again http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 13:45 |
| *** ijw has quit IRC | 13:46 | |
| *** burgerk has quit IRC | 13:46 | |
| *** chandankumar has quit IRC | 13:47 | |
| fungi | i'll take a peek at the javamelody graphs in a moment, and if it's the usual java gc death spiral i'll restart it | 13:48 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Missing location_id on Location images https://review.openstack.org/381851 | 13:48 |
| fungi | after the release this week, we're planning to upgrade to a newer jgit version that in theory has fixed this leak | 13:48 |
| *** links has quit IRC | 13:49 | |
| fungi | just worried to create unnecessary disruption leading into release day | 13:49 |
| *** jamesdenton has joined #openstack-infra | 13:49 | |
| *** pilgrimstack1 has joined #openstack-infra | 13:49 | |
| *** mariojv has joined #openstack-infra | 13:50 | |
| * zigo goes to upload 237 packages to Debian unstable ... :) | 13:51 | |
| zigo | (not jocking, this really is my todo...) | 13:51 |
| AJaeger | zigo, will that trigger 237 patches in our CI? ;) | 13:52 |
| zigo | AJaeger: That part is already done! :) | 13:52 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Missing location_id on Location images https://review.openstack.org/381851 | 13:52 |
| zigo | Mostly, editing debian/changelog to put "unstable" there ... | 13:52 |
| * AJaeger noticed - a new record of jobs in post queue at one day | 13:52 | |
| zigo | Took me the whole day ... | 13:52 |
| anteaya | AJaeger: what is the current record for post jobs? | 13:53 |
| *** hongbin has joined #openstack-infra | 13:53 | |
| AJaeger | anteaya: you were counting last time, weren't you? Wasn't it 370? | 13:53 |
| anteaya | was it that day last week? I think we were over 400 | 13:53 |
| AJaeger | seems ages ago ;( | 13:53 |
| anteaya | I was yeah, I will have to go back to the logs | 13:53 |
| anteaya | indeed | 13:53 |
| anteaya | but after i get back from errands | 13:53 |
| anteaya | I'm out for most of today, I am going to try to get back for the meeting | 13:54 |
| *** senk_ has joined #openstack-infra | 13:54 | |
| *** hamzy has quit IRC | 13:54 | |
| mariojv | hi - is there something wrong with gerrit? i'm getting intermittent slowness and 502s | 13:54 |
| anteaya | if I miss, the only thoughts I have about ianychoi's item is that we don't have bugs expected on both storyboard and launchpad | 13:54 |
| dasm | mariojv: it seems so. http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 13:55 |
| dasm | mariojv: already mentioned by frickler, EmilienM, dtantsur and others. | 13:55 |
| mariojv | thanks - wasn't in the channel but just wanted to add 1 more report | 13:55 |
| zigo | mariojv: Yeah, there's something wrong ... memory leak as always ! :) | 13:56 |
| *** moravec has joined #openstack-infra | 13:56 | |
| *** esikachev has joined #openstack-infra | 13:57 | |
| *** chandankumar has joined #openstack-infra | 13:57 | |
| *** yamamoto has joined #openstack-infra | 13:58 | |
| fungi | confirmed, gerrit's jvm started spending a lot of time on garbage collection right at 13:30. i'll go ahead and restart it | 13:58 |
| *** chandankumar1 has joined #openstack-infra | 13:58 | |
| dasm | fungi: thanks | 13:58 |
| *** yamamoto has quit IRC | 13:59 | |
| *** cardeois has joined #openstack-infra | 13:59 | |
| fungi | #status notice The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily | 13:59 |
| openstackstatus | fungi: sending notice | 13:59 |
| *** bin_ has joined #openstack-infra | 13:59 | |
| *** coolsvap_ is now known as coolsvap | 13:59 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address performance degradation and should return momentarily | 14:00 | |
| *** xyang1 has joined #openstack-infra | 14:01 | |
| ianychoi | anteaya, actually infra IRC meeting is 4 AM here so it is too difficult for me to participate in the meeting. I asked to ildikov to explain and discuss storyboard and LP for openstack-dev/sandbox. She will lead upstream training on Barcelona. | 14:01 |
| fungi | the webui has returned now | 14:01 |
| *** EricGonczer_ has joined #openstack-infra | 14:01 | |
| fungi | ianychoi: sounds good. she's already been asking me for some background on the current situation with sb | 14:02 |
| openstackstatus | fungi: finished sending notice | 14:02 |
| *** salv-orlando has quit IRC | 14:02 | |
| *** esberglu has joined #openstack-infra | 14:03 | |
| *** chandankumar has quit IRC | 14:03 | |
| ianychoi | fungi, oh I see I will read meeting notes when I will get up tomorrow! | 14:03 |
| *** senk_ has quit IRC | 14:03 | |
| *** kgiusti has joined #openstack-infra | 14:04 | |
| *** shardy_mtg is now known as shardy | 14:06 | |
| *** matrohon has quit IRC | 14:06 | |
| *** daemontool_ has joined #openstack-infra | 14:06 | |
| *** yamamoto has joined #openstack-infra | 14:08 | |
| *** yamamoto has quit IRC | 14:09 | |
| *** daemontool has quit IRC | 14:10 | |
| *** EricGonc_ has joined #openstack-infra | 14:10 | |
| *** xarses has quit IRC | 14:10 | |
| *** jheroux has joined #openstack-infra | 14:11 | |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| *** claudiub has joined #openstack-infra | 14:12 | |
| *** burgerk has joined #openstack-infra | 14:13 | |
| *** avladu has joined #openstack-infra | 14:13 | |
| *** arxcruz has quit IRC | 14:13 | |
| *** EricGonczer_ has quit IRC | 14:14 | |
| *** gouthamr has quit IRC | 14:14 | |
| *** fguillot has joined #openstack-infra | 14:14 | |
| avladu | hello, I get some failures during the docs generation on the cloudbase-init project. This is the test that fails: gate-cloudbase-init-docs-ubuntu-xenial | 14:15 |
| avladu | and this is an example of patch with that failure | 14:15 |
| avladu | https://review.openstack.org/#/c/333877 | 14:15 |
| *** chandankumar1 is now known as chandankumar | 14:15 | |
| *** yamamoto has quit IRC | 14:16 | |
| *** mgould has joined #openstack-infra | 14:17 | |
| *** amotoki has joined #openstack-infra | 14:17 | |
| mgould | hi everyone! Could I please have some reviews on https://review.openstack.org/#/c/346923 ? | 14:18 |
| mgould | ^^^ trivial patch, just adds a comment explaining something we found confusing in the Ironic team | 14:18 |
| *** tosky has quit IRC | 14:20 | |
| *** matrohon has joined #openstack-infra | 14:21 | |
| *** tosky has joined #openstack-infra | 14:21 | |
| *** moravec has quit IRC | 14:21 | |
| pabelanger | avladu: AJaeger: this looks to be an issue with our upcoming migration to afsdocs, we'll look into it shortly | 14:22 |
| *** rlu has quit IRC | 14:23 | |
| *** rlu has joined #openstack-infra | 14:24 | |
| *** rlu has quit IRC | 14:24 | |
| pabelanger | clarkb: fungi: jeblair: So, I am currently testing out 381883 Custom ssh.conf (upstart) for openssh-server to fix our sshd restart issue. I'm open to suggestions on how better to solve this, including making nodepool a little more forgiving if the SSH connection is reset when we first bring a node online | 14:25 |
| openstackgerrit | Daniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard https://review.openstack.org/381327 | 14:25 |
| *** yamamoto has joined #openstack-infra | 14:26 | |
| fungi | pabelanger: yeah, i'm starting to get the distinct impression that there are no real guarantees in boot-time that once sshd is up and listening, it won't be suddenly restarted by something else in the boot process | 14:27 |
| *** hamzy has joined #openstack-infra | 14:27 | |
| *** yamamoto has quit IRC | 14:27 | |
| *** Naeil has quit IRC | 14:27 | |
| pabelanger | yes, it is a battle for sure | 14:27 |
| *** yamamoto has joined #openstack-infra | 14:28 | |
| *** nadya has quit IRC | 14:28 | |
| *** drawsmcgraw has joined #openstack-infra | 14:28 | |
| pabelanger | so far I've found /etc/network/if-up.d/openssh-server | 14:28 |
| fungi | maybe we should have a very final initscript or whatever on all our nodes that touches something like /booted and then have nodepool keep checking for that and allow (some limited number of) ssh retries or whatever until that file exists | 14:29 |
| clarkb | mgould: that is a grenade change whoch is a qa project. You may have better luck getting reviews if you ask in #openstack-qa | 14:29 |
| clarkb | pabelanger: is it glean fighting the rest of startup? maybe we just need to remove that from glean? | 14:30 |
| pabelanger | clarkb: no, the patch I did yesterday was not the actually issue. infact, we can abandon it I think | 14:30 |
| clarkb | gotcha | 14:31 |
| pabelanger | its the same issue as always, sshd is restarted because of X | 14:31 |
| fungi | clarkb: it's interface configuration changes causing sshd restarts in that case | 14:31 |
| pabelanger | and with trusty SSH is started pretty early in the process | 14:31 |
| pabelanger | systemd tries to solved this | 14:31 |
| pabelanger | and explains why we see less failures now | 14:32 |
| pabelanger | fungi: a marker might help, but having nodepool retry a limited amount of times is likely the key to fixing this | 14:32 |
| clarkb | though the resets are due to sshd running without host keys not the restarts right? | 14:33 |
| pabelanger | not any more, we fixed that | 14:33 |
| clarkb | (it should generally be safe to restart sshd but not to run it without the required keys) | 14:33 |
| cloudnull | jroll: RE: "I believe that IPA gate is still down due to the problem with osic cloud external networking" -- this is still the devstack+neutron 10.0.0.0/8 network issue clarkb and a few others have mentioned before, right? There's not some new connectivity issue in the OSIC? | 14:33 |
| * cloudnull reading scrollback | 14:33 | |
| *** apetrich has quit IRC | 14:34 | |
| pabelanger | clarkb: the command is invoke-rc.d ssh restart >/dev/null 2>&1 || true, which kills our ssh connection | 14:34 |
| mgould | clarkb: d'oh, thanks! | 14:34 |
| *** apetrich has joined #openstack-infra | 14:34 | |
| *** rossella_s has quit IRC | 14:34 | |
| *** mgould has left #openstack-infra | 14:34 | |
| clarkb | pabelanger: I dont think that should kill connections. Are we sure host keys are generated at that point? | 14:34 |
| pabelanger | clarkb: yes, keys have been created | 14:35 |
| pabelanger | I can see ssh-keygen server start and stop successfully | 14:35 |
| pabelanger | then, sshd is started | 14:35 |
| *** rossella_s has joined #openstack-infra | 14:35 | |
| *** spzala has quit IRC | 14:35 | |
| clarkb | ok | 14:36 |
| *** gouthamr has joined #openstack-infra | 14:37 | |
| *** annegentle has joined #openstack-infra | 14:37 | |
| clarkb | cloudnull: I am not aware of any new issues with networking. I think that ironic using 10/8 to address their fale baremetal nodes in testing so the route update makes those hosts uncommunicative | 14:39 |
| cloudnull | makes sense . | 14:39 |
| *** rook_ is now known as rook | 14:39 | |
| cloudnull | just checking. | 14:39 |
| *** yamamoto has quit IRC | 14:40 | |
| * cloudnull making sure cloud1 isn't being a bad actor | 14:40 | |
| *** senk_ has joined #openstack-infra | 14:42 | |
| *** tuannguyen has quit IRC | 14:42 | |
| pabelanger | We could add an except handler for [Errno 104] Connection reset by peer http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodeutils.py#n47 like we did for cloud user created after sshd is up | 14:42 |
| *** ijw has joined #openstack-infra | 14:42 | |
| pabelanger | shrugs | 14:42 |
| *** tuannguyen has joined #openstack-infra | 14:42 | |
| *** john-davidge has quit IRC | 14:44 | |
| *** john-davidge has joined #openstack-infra | 14:44 | |
| *** burgerk_ has joined #openstack-infra | 14:46 | |
| *** yamamoto has joined #openstack-infra | 14:46 | |
| *** nadya has joined #openstack-infra | 14:47 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Switch coverage tox job to use pbr test command https://review.openstack.org/381898 | 14:47 |
| *** ijw has quit IRC | 14:47 | |
| *** mtanino has joined #openstack-infra | 14:48 | |
| *** john-davidge has quit IRC | 14:49 | |
| *** burgerk has quit IRC | 14:50 | |
| *** dragonmaster_ has quit IRC | 14:50 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** Naeil has joined #openstack-infra | 14:51 | |
| *** tuannguy_ has joined #openstack-infra | 14:51 | |
| *** salv-orlando has joined #openstack-infra | 14:51 | |
| *** avladu has quit IRC | 14:52 | |
| *** tuannguyen has quit IRC | 14:52 | |
| *** nadya has quit IRC | 14:52 | |
| clarkb | pabelanger: random brainstorming we could try to make it so firewall doesn't open port 22 until things are actually ready (this is hard because on boot the firewall is wide open then we restrict it from there) | 14:54 |
| *** salv-orlando has quit IRC | 14:56 | |
| AJaeger | avladu, pabelanger : cloudint setup is broken, they do not publish docs to doc/build/html. | 14:58 |
| *** drifterza is now known as notdrwahl | 14:59 | |
| jeblair | pabelanger, clarkb: i'm also curious why that restart affects existing ssh connections... does it perhaps only affect connections that are not fully established yet or something? | 14:59 |
| AJaeger | pabelanger: still, we should not abort in that case. Let me try fixing both sides | 15:00 |
| clarkb | jeblair: that was my theory before we ran into the no host keys behavior. Was thinking that maybe we are catching the connection before the subprocess has forked off and reparented | 15:00 |
| clarkb | jeblair: however i Have not been able to confirm that, and the no host keys behavior fit what we were seeing perfectly | 15:00 |
| *** rajinir has joined #openstack-infra | 15:01 | |
| rajinir | fatal: unable to access 'https://review.openstack.org/openstack-dev/devstack/': The requested URL returned error: 503 | 15:01 |
| rajinir | This was working last week | 15:02 |
| pabelanger | jeblair: clarkb: not much, but here is the traceback when the connection resets: http://paste.openstack.org/show/584234/ | 15:02 |
| jeblair | pabelanger, clarkb: i could probably get behind a retry for connection reset | 15:02 |
| jeblair | rajinir: what received that error? | 15:03 |
| AJaeger | rajinir: and when? | 15:03 |
| rajinir | This is in our CI pre_test_hook, we have to fetch an upstream patch | 15:03 |
| *** r-mibu has quit IRC | 15:04 | |
| *** r-mibu has joined #openstack-infra | 15:04 | |
| AJaeger | rajinir: we restarted gerrit an hour ago due to some issues, did it occur at that time? | 15:04 |
| *** HeOS has quit IRC | 15:04 | |
| *** HeOS has joined #openstack-infra | 15:05 | |
| *** spzala has joined #openstack-infra | 15:05 | |
| rajinir | AJaeger: seems intermittent | 15:05 |
| *** avladu has joined #openstack-infra | 15:05 | |
| rajinir | AJaeger: Saw some errors in the last hour | 15:06 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Cinder: don't skip changed tempest tests https://review.openstack.org/381910 | 15:06 |
| AJaeger | pabelanger, jeblair: there was a comment that https://review.openstack.org/#/c/333877 failed the docs job due to the "echo > doc/build/html/.root-marker". Looking at the repo, build_sphinx was misconfigured. Do we care about that - or should and thus should I make run-docs.sh robust to not fail in this case - or should we let it fail, so that people integrate nicely? | 15:07 |
| *** electrofelix has quit IRC | 15:07 | |
| fungi | jeblair: theory... nodepool attempts to connect right at the moment that the network interface configuration is updated and sshd restart has been triggered, so no listening socket | 15:07 |
| AJaeger | without that path, docs-draft publishing will not work. | 15:07 |
| *** electrofelix has joined #openstack-infra | 15:07 | |
| jeblair | fungi: well, it's a 'connection reset', so i expect it's at least started the tcp handshake... | 15:08 |
| *** notdrwahl is now known as drifterza | 15:09 | |
| fungi | hrm... maybe. as long as "connection reset" in this case doesn't mean tcp/rst in response to tcp/syn | 15:09 |
| jeblair | AJaeger: regarding developer-docs -- i think we just haven't made the afs volume for it yet. i think we can go ahead and do that now, yeah? | 15:09 |
| fungi | which would of course be the normal reaction when attempting to establish a connection with no listening socket | 15:10 |
| *** rcernin has quit IRC | 15:10 | |
| *** kmartin has quit IRC | 15:10 | |
| AJaeger | jeblair: I'm fine with creating it, all jobs should be ready. | 15:10 |
| jeblair | fungi: it's certainly different behavior from 'ssh isn't running right now' | 15:10 |
| AJaeger | jeblair: but I'll double check | 15:10 |
| fungi | got it | 15:10 |
| jroll | cloudnull: yes, the 10/8 thing | 15:11 |
| *** esikachev has quit IRC | 15:12 | |
| cloudnull | ok. just making sure. | 15:12 |
| jeblair | AJaeger: i'm in favor of standardization, so i'd leave run-docs.sh as-is. | 15:12 |
| AJaeger | jeblair: yes, you added all the afs jobs already to api-jobs, so we should be fine on that side. And checking the jobs, the root-markers are fine as well... | 15:13 |
| AJaeger | jeblair: that was my second thought as well ;) I only fixed the broken cloudbase-init ;). Let's see what they think... | 15:14 |
| avladu | AJaeger: so now it s all fine on that gate run? | 15:14 |
| *** moravec has joined #openstack-infra | 15:14 | |
| AJaeger | avladu: cloudbase-init is broken, I sent a patch for it.. https://review.openstack.org/381907 | 15:15 |
| *** yamahata has joined #openstack-infra | 15:16 | |
| jeblair | AJaeger: what kind of documentation will the 'developer-docs' volume contain? | 15:16 |
| AJaeger | avladu: you publish to a location that is different from what our infra scripts expect. | 15:16 |
| *** lucasagomes is now known as lucas-hungry | 15:16 | |
| jeblair | AJaeger: (it's *not* docs.o.o/developer -- i know that) | 15:16 |
| avladu | AJaeger: got it, thanks | 15:16 |
| *** mriedem has joined #openstack-infra | 15:16 | |
| AJaeger | jeblair: http://developer.openstack.org/ - so api-ref, api-quick-start (from api-site), firstapp application, api-guide documents | 15:16 |
| avladu | AJaeger: I will +2 asap | 15:16 |
| AJaeger | And python-openstacksdk | 15:16 |
| jeblair | AJaeger: got it, thx | 15:17 |
| *** daemontool_ has quit IRC | 15:19 | |
| *** armax has joined #openstack-infra | 15:22 | |
| AJaeger | pabelanger, jeblair , we added a vhost for docs.o.o in https://review.openstack.org/#/c/380483/ | 15:23 |
| AJaeger | What do you think of changing that to docs-beta (or similar) so that we can browse it properly? files.openstack.org has all the URLs wrong, they are relative to / and thus don't work... | 15:23 |
| *** akshai has quit IRC | 15:24 | |
| jeblair | AJaeger, pabelanger: maybe we can make docs-beta a serveralias | 15:24 |
| * AJaeger does not care about the name, just wants to check it properly ;) | 15:25 | |
| strigazi | hi clarkb, I tried your patch (379543) and the jobs in magnum succeed in osic. However the change is not merging. Any advice? | 15:25 |
| clarkb | strigazi: we need to wait for neutron/devstack to make a change globally that fixes the issue | 15:25 |
| clarkb | jeblair: fungi pabelanger https://review.openstack.org/#/c/370442/1 is a fix for nova net multinode testing | 15:25 |
| *** tuannguy_ has quit IRC | 15:26 | |
| strigazi | clarkb, so until then no patch merges for magnum | 15:26 |
| JayF | Where is the right place to go for TC Election feedback? I'm pretty sure I was just emailed a second ballot. | 15:26 |
| jroll | clarkb: is that "global change" you mention this or something else (is it up yet)? https://review.openstack.org/#/c/380023/1 | 15:26 |
| jroll | JayF: tonyb / njohnston / tristanC | 15:26 |
| *** yaume has quit IRC | 15:26 | |
| clarkb | strigazi: unless you reconfigure your job to stop using 10/8 or get lucky and jobs run on not osic | 15:26 |
| JayF | tonyb: njohnston: tristanC: I am pretty sure I just got mailed a second TC Election ballot. | 15:27 |
| clarkb | jroll: I have sort of somewhat intentionally tried to hand that off back to the responsible teams so I am not currently sure of what tactic they want to take | 15:27 |
| *** tuannguyen has joined #openstack-infra | 15:27 | |
| clarkb | jroll: revert has been mentioned though that requires some unwinding of a test or two that depends on the feature | 15:27 |
| *** daemontool has joined #openstack-infra | 15:27 | |
| clarkb | (I don't want to bandaid this in infra, this needs to be fixed where it was broken) | 15:27 |
| JayF | tonyb: njohnston: tristanC: oh; apparently just duplicate email w/same voter key. It won't let me revote. Disregard. | 15:27 |
| jroll | clarkb: how would one stop using 10/8? | 15:27 |
| strigazi | clarkb, ok I'll change out job. Is there a WIP in neutron/devstack to follow and revert my change later? | 15:27 |
| *** xarses has joined #openstack-infra | 15:28 | |
| *** Teresita-Warrior has joined #openstack-infra | 15:28 | |
| jeblair | JayF: (the election officials will have the authoritative answer, but if it was sent to the same email address, it's likely the same ballot just resent because of some rax delivery issues) | 15:28 |
| clarkb | strigazi: as I just said I don't know you will need to talk to them | 15:28 |
| strigazi | clarkb, thanks | 15:28 |
| clarkb | jroll: strigazi in magnums case they are connecting to rabbit over the 10/8 address they could just use localhost I think | 15:28 |
| JayF | jeblair: yeah, that's what it looks like. Just weird because I know better than to use my Rackspace email address for emails I want to recieve :P | 15:28 |
| jroll | clarkb: ah, I see. no hacks for external connectivity? | 15:28 |
| njohnston | JayF: We resent ballots to everyone because there were numerous reports of email issues. As jeblair said, it should have been sent to the same email address, which means it is the same ballot just reissued; you still only have one vote. :-) http://lists.openstack.org/pipermail/openstack-dev/2016-October/105002.html | 15:28 |
| fungi | JayF: yeah, see tonyb's announcement to the -dev ml yesterday/today (depending on what you call midnight). a lot of people didn't receive their ballots the first time around, so they resent copies of them all just tp be on the safe side | 15:29 |
| *** ilyashakhat_mobi has joined #openstack-infra | 15:29 | |
| clarkb | jroll: strigazi ironic's case is more complicated beacuse you are doing networking in and out of neutron. You would need to make your network subnets outside of neutron not overlap with 10/8 so you could use 192.168/16 or 172.whateverthatrangeisIforget | 15:29 |
| AJaeger | jeblair: I'm not a puppet expert, is this the right syntax ? https://review.openstack.org/381927 | 15:29 |
| jroll | clarkb: got it, thanks | 15:29 |
| jeblair | AJaeger: sorry, i'm digging into something, i'll pop back up to that in a few minutes | 15:29 |
| AJaeger | infra-root, we lost gerritbot - anybody has time to help, please? | 15:29 |
| AJaeger | jeblair: sure, not urgent for now. | 15:29 |
| fungi | njohnston: worth noting, i didn't receive a second copy. and my friend who didn't get a ballot the first time still hasn't seen one (i've confirmed i can contact the address he has in the electoral roll) | 15:29 |
| *** rohanion has joined #openstack-infra | 15:30 | |
| JayF | clarkb: 172.16/12 | 15:30 |
| clarkb | jroll: and I think the lesson here is we should be using reasonably sized ranges that meet our needs rather than the biggest possible range in hopes that we someday need it all | 15:30 |
| *** adrian_otto has joined #openstack-infra | 15:31 | |
| clarkb | so for ironic testing you probably only need a /24 | 15:31 |
| jroll | clarkb: indeed | 15:31 |
| clarkb | (not sure what you are currently using that may already be the case) | 15:31 |
| njohnston | fungi: Please encourage your friend to reach out to Tony, Tristan, or myself either directly or on #openstack-election, and we can work with your friend to ensure their ballot is received. | 15:31 |
| *** moravec has quit IRC | 15:32 | |
| avladu | AJaeger: thanks all for the help, change merged | 15:32 |
| *** akshai has joined #openstack-infra | 15:32 | |
| AJaeger | avladu: I rechecked the failing job, should be fine soon... | 15:32 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Bifurcate neutron-fwaas tempest gate for v2 https://review.openstack.org/371749 | 15:33 |
| AJaeger | oh, openstackgerrit is mirally back. Thanks! | 15:34 |
| fungi | njohnston: yep, it's hard enough to encourage people to vote, so getting them to e-mail someone so they can get a ballot is tough. seeing how convincing i can be | 15:34 |
| *** flepied1 has joined #openstack-infra | 15:35 | |
| njohnston | fungi: Thanks, I appreciate it. I'm passionate about elections - I'm an elections officer IRL as well - so if there is anything I can do to assist please let me know. | 15:35 |
| *** esikachev has joined #openstack-infra | 15:35 | |
| AJaeger | fungi, njohnston : I didn't get a second email either - but the first arrived. | 15:36 |
| *** flepied has quit IRC | 15:37 | |
| fungi | njohnston: appreciated! but yeah, as i feared the resend seems to have been about as spotty as the original batch of ballots. so if 80% of people received ballots each time then we have 10% of the electorate who didn't receive a ballot either time (numbers picked out of thin air for purposes of demonstration) | 15:38 |
| AJaeger | fungi, my math says 16 %... | 15:39 |
| fungi | AJaeger: your math is more precise than my estimates ;) | 15:39 |
| ajo_ | http://logs.openstack.org/77/378677/3/experimental/gate-octavia-v1-dsvm-scenario-multinode-nv/2183ce1/console.html#_2016-10-04_15_18_37_734959 | 15:39 |
| pabelanger | clarkb: fungi: jeblair: I'm also going to try having nodepool use host key checking (ssh-keyscan) to collect more data. Curious to see what a happens with sshd in that instance | 15:40 |
| ajo_ | any idea of why is this failing sometimes ? | 15:40 |
| pleia2 | good morning | 15:40 |
| ajo_ | (fetching logs from subnodes in a multinode) | 15:40 |
| clarkb | ajo_: yes, beacuse devstack and neutron have set an invalid route for 10.0.0.0/8 breaking that ipv4 connectivity between hosts in multinode testing | 15:40 |
| *** sandanar has quit IRC | 15:40 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Trifurcate neutron-fwaas tempest gate for v2 https://review.openstack.org/371749 | 15:40 |
| *** salv-orlando has joined #openstack-infra | 15:40 | |
| ajo_ | clarkb, is there any way around that ? | 15:41 |
| ajo_ | clarkb, where is that set? | 15:41 |
| clarkb | ajo_: not right now no | 15:41 |
| AJaeger | ajo_: clarkb is working on it AFAIU - feed him ;) | 15:41 |
| clarkb | ajo_: devstack is setting the range via the new subnetpool feature support and I am not sure if neutron or devstack is setting the route | 15:41 |
| pabelanger | Also testing connection failure retry logic | 15:41 |
| clarkb | No! I am trying my best to not work on it :) | 15:41 |
| AJaeger | clarkb: Sorry;( | 15:42 |
| clarkb | I have helped identify and diagnose the problem along with a couple others but would really really like devstack and neutron to fix it | 15:42 |
| ajo_ | clarkb, I'd be happy to help with that if I can/know how (I'm a neutron core) | 15:42 |
| AJaeger | clarkb: understood | 15:42 |
| clarkb | ajo_: let me get the bug | 15:42 |
| AJaeger | bbl | 15:42 |
| ajo_ | clarkb, awesome | 15:42 |
| clarkb | https://bugs.launchpad.net/bugs/1629133 | 15:42 |
| openstack | Launchpad bug 1629133 in devstack "New neutron subnet pool support breaks multinode testing." [Critical,In progress] - Assigned to Dr. Jens Rosenboom (j-rosenboom-j) | 15:42 |
| SpamapS | jeblair: did we ever figure out why geard doesn't work with the upstream gearman CLI worker? | 15:43 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/puppet-iptables: Change Cacti IP address for Iptables rules https://review.openstack.org/381936 | 15:43 |
| * SpamapS is poking at things left on the backburner for way too long | 15:43 | |
| *** ijw has joined #openstack-infra | 15:43 | |
| avladu | AJaeger: all good now, much appreciated | 15:43 |
| pabelanger | avladu: as a follow up, it is usually better just to ask your questions here, then PM. So, others can see and answer them | 15:44 |
| *** eharney has quit IRC | 15:44 | |
| SpamapS | looks like maybe geard doesn't understand GEARMAN_GRAB_JOB_ALL | 15:45 |
| *** eharney has joined #openstack-infra | 15:47 | |
| *** thcipriani|afk is now known as thcipriani | 15:47 | |
| *** matrohon has quit IRC | 15:47 | |
| *** ijw has quit IRC | 15:48 | |
| *** adrian_otto has quit IRC | 15:48 | |
| rcarrillocruz | haha, jesusaur i just pushed https://review.openstack.org/#/c/381936/1 while thinking 'oh my, why we have this hardcoded!' and just saw on the right column of the change you pushed something already to parameterize it | 15:49 |
| rcarrillocruz | so ++ | 15:49 |
| rcarrillocruz | happy to land your patch | 15:49 |
| *** akshai has quit IRC | 15:51 | |
| mtreinish | fungi: so it looks like my o-h change merged last night but health.o.o is still giving out 500s | 15:51 |
| mtreinish | fungi: is it still the same error message as before? | 15:51 |
| ajo_ | clarkb, I wonder if using a LLA range would work (169.254.0.0/16) | 15:53 |
| fungi | mtreinish: [Tue Oct 04 15:52:07.278906 2016] [:error] [pid 31032:tid 139914604582656] [remote 54.210.198.246:13423] TypeError: __init__() got an unexpected keyword argument 'es_url' | 15:53 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear: Add py35 to tox.ini default envs https://review.openstack.org/381944 | 15:53 |
| *** armax has quit IRC | 15:53 | |
| fungi | mtreinish: the timestamp there was just a few seconds ago, so i guess it is | 15:53 |
| clarkb | ajo_: maybe but that wouldn't be a valid use of that range right? We jsut need to stop using all encompassing ranges | 15:53 |
| clarkb | ajo_: we should use a /20 not a /8 | 15:53 |
| clarkb | then have a utility that will find you the next available /20 based on routes on the host | 15:54 |
| clarkb | but I think step one is stop breaking everyone | 15:54 |
| fungi | mtreinish: "Fix elasticsearch querying with recent e-r changes" is the current checkout in /opt/openstack-health so i'm seeing what's causing it to hold onto the old version | 15:54 |
| clarkb | ajo_: and also only write routes for subnets actually in use | 15:55 |
| mtreinish | fungi: ok, that's what I figured was happening. Maybe apache just needs a restart? | 15:55 |
| clarkb | (pretty sure I captured my argument in the bug already) | 15:55 |
| *** yamahata has quit IRC | 15:55 | |
| ajo_ | clarkb, that's a valid use of the range I think, if for testing we just need it to be accessible locally to the controller... | 15:55 |
| fungi | mtreinish: probably, but want to check where openstack-health is installed first | 15:55 |
| *** hashar is now known as hasharAway | 15:56 | |
| ajo_ | clarkb, the dynamic routes is more a deployment thing, | 15:56 |
| clarkb | ajo_: I thought that address range was for when you had no address managment | 15:56 |
| ajo_ | the routes must be there before neutron l3 agent kicks in | 15:56 |
| clarkb | ajo_: neutron is managing addresses and even runs a dhcp so that wouldn't be a valid use of the range | 15:56 |
| mtreinish | fungi: it's in a venv /opt/openstack-health/.venv iirc | 15:56 |
| *** tiswanso has quit IRC | 15:56 | |
| fungi | mtreinish: `/opt/openstack-health.venv/bin/pbr freeze` says openstack-health==0.0.1.dev680 # git sha c03378f but your change merged as fa9c804 | 15:57 |
| ajo_ | clarkb, let me investigate the thing a little more, I will even send a dummy patch to check if LLA works | 15:57 |
| fungi | mtreinish: checking puppet logs next to see why it failed to upgrade | 15:57 |
| *** mdrabe has quit IRC | 15:57 | |
| clarkb | ajo_: "are only used where stable, routable addresses are not available" from the rfc | 15:57 |
| clarkb | ajo_: so using that range would be a violation I think | 15:58 |
| fungi | mtreinish: yeah, c03378f is the merge commit prior to your change | 15:58 |
| mtreinish | fungi: oh, right the puppet does a pip install -U /opt/openstack-health exec (there was weirdness with the entrypoints and -e) | 15:58 |
| ajo_ | clarkb, we don't need them to be routed, right? | 15:58 |
| clarkb | ajo_: you do | 15:58 |
| *** adrian_otto has joined #openstack-infra | 15:58 | |
| clarkb | ajo_: at least I was told that was the whole reason we are in this mess in the first place :) | 15:58 |
| ajo_ | clarkb, ok, let me do my homework the, (checking how's this set'd up) | 15:59 |
| clarkb | ajo_: if this weren't routable we would not have problems and it would likely just work | 15:59 |
| ajo_ | clarkb, for routable, i mean, you need them to go out of the node itself for testing | 15:59 |
| *** tiswanso has joined #openstack-infra | 15:59 | |
| *** tiswanso has quit IRC | 15:59 | |
| clarkb | ajo_: yes, for multinode you do | 15:59 |
| ajo_ | like for floating ips, we just attack from tempest to local | 15:59 |
| ajo_ | yeah, in DVR may be | 15:59 |
| *** moravec has joined #openstack-infra | 15:59 | |
| clarkb | ajo_: not always. DVR and nova net need them to be routable | 15:59 |
| clarkb | DVR being the case that matters here | 16:00 |
| pleia2 | good morning | 16:00 |
| ajo_ | yeah | 16:00 |
| *** akshai has joined #openstack-infra | 16:00 | |
| ajo_ | undertood :/ | 16:00 |
| fungi | mtreinish: Oct 3 21:58:39 health puppet-user[302]: (/Stage[main]/Openstack_health::Api/Exec[requirements]/returns) ImportError: Building pandas requires cython | 16:00 |
| ajo_ | undrestood | 16:00 |
| ajo_ | whatever XD | 16:00 |
| SpamapS | jeblair: well I created this https://storyboard.openstack.org/#!/story/2000737 ... but I can't seem to find the description I submitted. :-P | 16:00 |
| ajo_ | I don't know how to write anymore | 16:00 |
| *** tiswanso has joined #openstack-infra | 16:00 | |
| mtreinish | fungi: huh, wasn't expecting that. Heh, I guess that's the downside with -U | 16:01 |
| mtreinish | I didn't even realize pandas added a cython buildep | 16:01 |
| SpamapS | Seriously.. what happens to the description when creating a new story? | 16:01 |
| SotK | SpamapS: oops, its there, just breaking the markdown renderer | 16:02 |
| *** jaosorior has quit IRC | 16:02 | |
| SpamapS | SotK: hah, doh | 16:02 |
| fungi | SotK: i wonder if it would make sense to wrap markdown or similar parsers in a try/except and fall back to displaying raw content | 16:02 |
| SpamapS | no fall back to plaintext eh | 16:03 |
| *** amotoki has quit IRC | 16:03 | |
| SotK | fungi: it probably would make sense to do that :) | 16:03 |
| *** mdrabe has joined #openstack-infra | 16:03 | |
| SpamapS | also should fail saving | 16:03 |
| fungi | might be a cheaper solution than alternatives anyway | 16:03 |
| SpamapS | "Markdown doesn't like this." | 16:03 |
| *** amitgandhinz has quit IRC | 16:03 | |
| SotK | SpamapS: try indenting any logs you've pasted in the description by 4 characters to make them code blocks, that might help | 16:04 |
| fungi | meh, if it fell back on rendering plaintext when the other format parsers break, then i'd be fine with saving still working | 16:04 |
| *** BobBall has joined #openstack-infra | 16:04 | |
| fungi | at least that way it's discoverable | 16:04 |
| SpamapS | SotK: Ok, however, that's.. not easy ;) | 16:04 |
| Zara | I think ``` should also work to make a codeblock | 16:05 |
| persia | SpamapS: `sed 's/^/ /' ? | 16:05 |
| SotK | wrapping in <pre> tags may work too | 16:05 |
| *** andreas_s has quit IRC | 16:06 | |
| fungi | oh, yep, i've done it in sb with <pre></pre> successfully before | 16:06 |
| SpamapS | persia: not easy. In my browser. | 16:06 |
| *** ilyashakhat_mobi has quit IRC | 16:06 | |
| SpamapS | pre's would be better yah | 16:06 |
| persia | Oh, ugh, I don't know the special JS trick for that. | 16:06 |
| Zara | (for backticks, there's an example here; I tend to use those most: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code-and-syntax-highlighting) | 16:06 |
| SpamapS | but I just vim'd things | 16:06 |
| ihrachys | AJaeger: that bindep.txt thingy, is it safe to land one in e.g. neutron? can't it break any other jobs in other gates? | 16:06 |
| SpamapS | and it's especially confused by the logs that are variable-left-space-filled | 16:07 |
| ihrachys | AJaeger: seems like devstack jobs are not affected? anything else to be cautious about? | 16:07 |
| *** drifterza has quit IRC | 16:07 | |
| SpamapS | ah no it was a missing blank line | 16:07 |
| jeblair | SpamapS: ah, yep. that'll be because GRAB_JOB_ALL does not appear in http://gearman.org/protocol/ so i didn't know to implement it :) | 16:08 |
| SpamapS | SotK: ok, so yeah if you want a test case.. take the leading spaces out of those logs. :) | 16:08 |
| SpamapS | jeblair: I think it does appear there now | 16:08 |
| SpamapS | or not | 16:08 |
| jeblair | SpamapS: i don't see it even after madly hitting shift-reload ;) | 16:08 |
| SpamapS | jeblair: well guess who maintains that website now ;-) | 16:08 |
| fungi | ihrachys: right now the only jobs that use bindep lists are project-specific jobs. probably the only thing to worry about is if you have file filters or skip-if for neutron jobs in zuul's layout.yaml causing some of your jobs not to run on the change adding bindep.txt | 16:08 |
| jeblair | SpamapS: is it me? ;) | 16:08 |
| SpamapS | jeblair: it can be! | 16:08 |
| SpamapS | --> https://github.com/gearman/gearmand | 16:09 |
| mtreinish | fungi: so I guess I should add cython to the puppet? | 16:09 |
| SpamapS | actually it's possible that /protocol just isn't up to date | 16:09 |
| ihrachys | fungi: I would be very surprised if we would have such a skip-if. | 16:09 |
| * SpamapS checks | 16:09 | |
| ihrachys | fungi: cool, thanks for the info | 16:09 |
| jeblair | SpamapS: cool -- though i think it might be best for not-me to write the docs for grab_job_all since i dunno what it is :) | 16:09 |
| jeblair | SpamapS: "Erm, it grabs all the jobs or something" | 16:09 |
| SpamapS | aha! yes I didn't realize I have to manually copy the new version into the website | 16:10 |
| fungi | mtreinish: that's probably the easiest solution yeah. cython-dev or whatever | 16:10 |
| SpamapS | https://github.com/gearman/gearmand/blob/master/PROTOCOL <-- up to date | 16:10 |
| SpamapS | and even in a release, 1.1.13 | 16:10 |
| jeblair | oh it's a mapreduce thing | 16:11 |
| *** dragonmaster has joined #openstack-infra | 16:11 | |
| *** tphummel has joined #openstack-infra | 16:11 | |
| jeblair | AJaeger: http://files.openstack.org/docs-developer/ exists now | 16:11 |
| *** tongli has joined #openstack-infra | 16:11 | |
| *** chandankumar has quit IRC | 16:11 | |
| *** jcoufal_ has joined #openstack-infra | 16:11 | |
| clarkb | jeblair: do we intend on moving docs-draft to afs as well? | 16:12 |
| *** amitgandhinz has joined #openstack-infra | 16:12 | |
| * clarkb should go pull up the spec | 16:12 | |
| *** avladu has quit IRC | 16:12 | |
| ihrachys | fungi: hm, one more thing. this is the bindep.txt file I have for neutron: https://review.openstack.org/#/c/381906/3/bindep.txt (I assume it misses some needed deps, but that's what I start with) | 16:12 |
| clarkb | was just thinking that that migration could give us a little headroom on the logs siguation if we can recoup that disk space for logs properl | 16:12 |
| ihrachys | fungi: it passes bindep test locally | 16:12 |
| *** jcoufal has quit IRC | 16:12 | |
| ihrachys | fungi: but in gate I catch this: http://logs.openstack.org/06/381906/3/check/gate-neutron-pep8-ubuntu-xenial/148dcd8/console.html#_2016-10-04_15_57_36_846699 | 16:13 |
| ihrachys | if I call 'bindep -b -f bindep.txt test' in the local venv it passes too | 16:13 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-openstack_health: Install cython with the openstack-health api server https://review.openstack.org/381960 | 16:13 |
| mtreinish | fungi: ^^^ | 16:13 |
| SpamapS | jeblair: _now_ refresh http://gearman.org/protocol/ | 16:13 |
| *** moravec has quit IRC | 16:13 | |
| jeblair | clarkb: out of scope for spec, but it certainly could be done | 16:13 |
| *** ArxCruz has joined #openstack-infra | 16:14 | |
| jeblair | clarkb: we may want to think about whether running a find on all of docs-draft is a good idea though -- we may need a variant of the afs publisher that doesn't do that | 16:14 |
| *** Ahharu has joined #openstack-infra | 16:15 | |
| jeblair | clarkb: (i am certain running a find on *logs* is a bad idea) | 16:15 |
| Ahharu | hey | 16:15 |
| Ahharu | are you aware of "2016-10-04 10:50:19.020907 | Failed to fetch ec2 uri http://169.254.169.254/latest/meta-data/ami-id: 500 Internal Server Error" ? | 16:15 |
| Ahharu | its bombing out the beaker tests | 16:15 |
| clarkb | jeblair: ya | 16:15 |
| *** weshay is now known as weshay_lunch | 16:15 | |
| SpamapS | jeblair: the reducer stuff needs some fleshing out, as I understand it finally. | 16:15 |
| clarkb | jeblair: well we do it weekly for deletes and compression but its infrequent and on weekends | 16:15 |
| *** Swami has joined #openstack-infra | 16:16 | |
| jeblair | clarkb: i think i'd probably go ahead and make the 'non-find' variant of the afs publisher before moving either docs-draft or logs | 16:16 |
| jeblair | AJaeger: 381927 lgtm. pabelanger want to aprv? | 16:17 |
| *** armax has joined #openstack-infra | 16:17 | |
| *** links has joined #openstack-infra | 16:17 | |
| *** amitgandhinz has quit IRC | 16:18 | |
| SpamapS | jeblair: but basically, it ties in with the coalescing. When the first client submits the job, they submit the 'reducer' argument along with the job data and unique, and workers can use that to submit the result of the work to that function. | 16:18 |
| *** amitgandhinz has joined #openstack-infra | 16:18 | |
| jeblair | SpamapS: that makes sense | 16:18 |
| *** links has quit IRC | 16:18 | |
| SpamapS | so geard doesn't have to do anything special | 16:18 |
| SpamapS | krow was working on some built-in thread-workers that would automatically do reducers though. | 16:19 |
| jeblair | sounds like the whole kit should be easy to add | 16:19 |
| SpamapS | jeblair: yeah, almost done. ;) | 16:19 |
| SpamapS | jeblair: also about 90% done on coalescing | 16:19 |
| jeblair | clarkb: looking at https://review.openstack.org/381944 should we make py35 voting? | 16:20 |
| SpamapS | though I think there's a missing coordination piece that's allowing coalesced jobs to stay in queue after completion. | 16:20 |
| *** Ahharu has quit IRC | 16:20 | |
| clarkb | jeblair: I think so esepcailyl since 34 is and there aren't a ton of differences in that direction of transition | 16:20 |
| SpamapS | Even though I know.. you're just about done removing gearman from the zuul/nodepool world. ;) | 16:20 |
| *** arxcruz_ has joined #openstack-infra | 16:21 | |
| *** claudiub has quit IRC | 16:21 | |
| jeblair | SpamapS: well, gearman is going to stick around for the initial version of v3 -- my plan to remove it completely is very long range :) | 16:21 |
| *** lucas-hungry is now known as lucasagomes | 16:21 | |
| *** david-lyle has joined #openstack-infra | 16:21 | |
| jeblair | SpamapS: (zk for all of nodepool and nodepool<->zuul interaction; but zuul->zuul-workers is gearman) | 16:21 |
| Shrews | v4 includes death star plans | 16:22 |
| Shrews | (oops, did i just let that slip?) | 16:22 |
| TheJulia | Shrews: you did | 16:22 |
| jeblair | Shrews: when v4 is fully operational you will see a demonstration of its power... | 16:23 |
| SpamapS | Shrews: that's no cloud | 16:23 |
| *** tiswanso has quit IRC | 16:23 | |
| SpamapS | jeblair: Ahh, forgot about the worker comms | 16:23 |
| *** nt has quit IRC | 16:23 | |
| *** dragonmaster has quit IRC | 16:24 | |
| *** nt has joined #openstack-infra | 16:24 | |
| ihrachys | fungi: is it even correct to call -b here https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/install-distro-packages.sh#L48 ? | 16:24 |
| ihrachys | seems to me like -b makes the tool to not write any stdout | 16:24 |
| strigazi | clarkb, is FIXED_RANGE=10.254.1.0/24 going to work? | 16:24 |
| *** dragonmaster has joined #openstack-infra | 16:25 | |
| strigazi | clarkb, or FIXED_RANGE=172.16.0.0/12 ? | 16:25 |
| clarkb | strigazi: we set FIXED_RANGE to 10.1.0.0/20 in the gate. Unfortunately this isn't sufficient because the subnetpools ignore the fixed range | 16:25 |
| *** tiswanso has joined #openstack-infra | 16:25 | |
| fungi | ihrachys: -b is brief/batch mode. it will output a list of all packages missing on the system. no output means nothing missing | 16:26 |
| jeblair | remote: https://review.openstack.org/381969 Gear: make python3.5 jobs voting | 16:26 |
| ihrachys | fungi: well, if I execute without -b it returns me a package | 16:26 |
| ihrachys | Bad versions of installed packages: | 16:26 |
| ihrachys | sqlite version 3.7.17-8.el7 does not match >=3.8 | 16:26 |
| ihrachys | but with -b, it's just return code = 1 | 16:26 |
| *** openstackgerrit has quit IRC | 16:26 | |
| ihrachys | but no output | 16:26 |
| pabelanger | jeblair: AJaeger: +3 | 16:27 |
| fungi | ihrachys: oh, that may be a bug in the version parsing, which is not used heavily yet | 16:27 |
| *** openstackgerrit has joined #openstack-infra | 16:27 | |
| ihrachys | fungi: yeah, I noticed it seems like not used anywhere ;) | 16:27 |
| ihrachys | fungi: was trying to find an example | 16:27 |
| *** baoli_ has quit IRC | 16:27 | |
| ajo_ | armax, clarkb, I sent a middle ground patch: https://review.openstack.org/#/c/381965/ | 16:28 |
| strigazi | clarkb, what if I set SUBNETPOOL_PREFIX_V4=172.0.0.0/8 | 16:28 |
| ajo_ | strigazi, 172.0.x.x is public | 16:28 |
| fungi | ihrachys: version specifications were roughed in originally but since most distros tend to not have multiple versions of the same package name in the same release there's not been a lot of call to improve it yet | 16:28 |
| clarkb | ajo_: the problem with that is we already tell devstack what range to use and devstack ignores it... | 16:28 |
| ajo_ | you would prevent any host on that range to be reached | 16:28 |
| *** openstackgerrit has quit IRC | 16:28 | |
| ajo_ | clarkb, what do you mean? | 16:28 |
| *** tesseract- has quit IRC | 16:29 | |
| ihrachys | fungi: well, we have a case where unit tests fail with old sqlite | 16:29 |
| clarkb | ajo_: I would like to see something that incorporates existing range selection signalling so that other people that have to set a range will continue to work | 16:29 |
| *** openstackgerrit has joined #openstack-infra | 16:29 | |
| ihrachys | fungi: I see a bug in code | 16:29 |
| clarkb | ajo_: every devstack job in the gate sets fixed range to 10.1.0.0/20 to avoid conflicts with clouds | 16:29 |
| clarkb | ajo_: this is the range that should be used. | 16:29 |
| armax | ajo_: I talkred to carl_baldwin yesterday, we’re gonna look into ways in which subnet_pools don’t interfere with host routes | 16:29 |
| clarkb | except it isn't anymore | 16:29 |
| ihrachys | fungi: the way we handle error[0] == 'badversion', it does not output if brief == True | 16:29 |
| *** ArxCruz has quit IRC | 16:29 | |
| ajo_ | armax, clarkb copy that | 16:29 |
| *** eharney has quit IRC | 16:29 | |
| *** openstackgerrit has quit IRC | 16:30 | |
| fungi | ihrachys: thanks. reviews in the usual place. since -b is designed to be inlined or piped to platform-specific package managers, i'm open to ideas on how that should behave for version mismatches | 16:30 |
| *** openstackgerrit has joined #openstack-infra | 16:30 | |
| ihrachys | fungi: I assume we should join name and version specifier. checking for yum. | 16:31 |
| strigazi | ajo_ clarkb would the combination FIXED_RANGE=12.0.0.0/24 and SUBNETPOOL_PREFIX_V4=12.0.0.0/8 work? | 16:31 |
| fungi | ihrachys: luckily in the case of our ci, at least, we always upgrade to the most recent versions of any installed or requested packages the system knows about so this shouldn't bite us there | 16:32 |
| *** amitgandhinz has quit IRC | 16:32 | |
| *** e0ne has quit IRC | 16:32 | |
| ihrachys | fungi: well, latest centos does not deliver needed sqlite | 16:32 |
| clarkb | strigazi: you shouldn't use 12/8 like that. ATT apparently own at least part of that range | 16:32 |
| ihrachys | fungi: so if we would have unit test job for the platform, it would fail with the proposed bindep | 16:32 |
| *** amitgandhinz has joined #openstack-infra | 16:32 | |
| fungi | ihrachys: right, and bindep won't fix that in our ci so you'd either need a newer centos release or a custom job to add those packages from somewhere | 16:33 |
| *** mdrabe has quit IRC | 16:33 | |
| *** mdrabe has joined #openstack-infra | 16:33 | |
| ihrachys | fungi: right. it's not going to fix anything, but it will at least give some clue to devs | 16:34 |
| jesusaur | rcarrillocruz: ah yeah, no reason to hardcode that in the iptables module | 16:34 |
| fungi | ihrachys: agreed | 16:34 |
| clarkb | strigazi: for your jobs specifically I think you just need to update how you talk to rabbit to use localhost | 16:35 |
| clarkb | strigazi: unless you have multinode jobs too | 16:35 |
| strigazi | clarkb, we don't have multinode jobs | 16:35 |
| *** daemontool has quit IRC | 16:35 | |
| jesusaur | rcarrillocruz: do you want to push a change that passes the new ip addresses from system-config and depends-on my iptables change? | 16:35 |
| fungi | ihrachys: anyway, you originally asked about use of -b in jenkins/scripts/install-distro-packages.sh so it's not really a concern there. if you have a convenience bindep env in your tox.ini i recommend not using -b so you have more useful and verbose output from the tool for local developer use anyway | 16:36 |
| strigazi | clarkb, and in conf we use 127.0.0.1 to talk to rabbit | 16:36 |
| ihrachys | fungi: well, the thing is, once I land the file, gate starts using it :) | 16:36 |
| ihrachys | fungi: and since there is no output, it fails the jobs | 16:36 |
| ihrachys | fungi: I will send a patch for bindep | 16:36 |
| strigazi | clarkb, see here: http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-k8s/c63884d/logs/etc/magnum/magnum.conf.txt.gz | 16:36 |
| clarkb | strigazi: are you sure that that is being used or that some other service isn't what is failing/ | 16:37 |
| rcarrillocruz | jesusaur: yeah, i rather land your patch and then i push a new patch that sets new cacti via site.pp | 16:37 |
| jesusaur | ++ | 16:37 |
| *** csomerville has joined #openstack-infra | 16:38 | |
| *** yamahata has joined #openstack-infra | 16:38 | |
| strigazi | clarkb, I think yes, I keep getting timeout when the is a request in magnum's api server. The reason is timeout while waiting for the message | 16:39 |
| fungi | ihrachys: can you point to the failure? if there's a job running bindep on centos 7 and you request for it to ensure sqlite>=3.8 is present, it's going to fail regardless (but i agree that a more verbose failure would be useful at any rate) | 16:39 |
| clarkb | strigazi: could it be the remote end failing to respond rather than magnum itself? Basically this does work in the normal single node neutron jobs so I don't think its a general issue when you avoid the routing problem | 16:40 |
| clarkb | which implies to me that somehow we are getting hit by the routing problem here | 16:40 |
| *** trown is now known as trown|lunch | 16:40 | |
| ihrachys | fungi: any job fails, because -b returns 1 while no output | 16:40 |
| ihrachys | fungi: http://logs.openstack.org/06/381906/3/check/gate-neutron-pep8-ubuntu-xenial/148dcd8/console.html#_2016-10-04_15_57_36_846699 | 16:40 |
| fungi | ihrachys: oh! i completely misunderstood where this was breaking. that traceback is useful | 16:41 |
| *** Ravikiran_K has joined #openstack-infra | 16:41 | |
| fungi | i thought you were saying it just silently reports a nonzero exit code in centos 7 jobs | 16:42 |
| fungi | ihrachys: i think version comparisons may not even be properly implemented across all platforms, so we're going to be breaking new ground by trying to include them in lists. may need to make it more clear that's an experimental feature of the tool that still needs a lot of work | 16:43 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/bindep: Fixed bindep -b mode for version mismatch errors https://review.openstack.org/381979 | 16:43 |
| ihrachys | fungi: the fix would be similar to ^ | 16:43 |
| ihrachys | fungi: checked locally for centos | 16:44 |
| fungi | for example, i think bindep currently uses rpm to query package lists on rh-specific platforms, so doesn't know whether there is even an available version to meet the requested requirement | 16:44 |
| ihrachys | seems to return what I want | 16:44 |
| *** ijw has joined #openstack-infra | 16:44 | |
| ihrachys | anyhow, need to run. please review if you have cycles. I will add tests later. | 16:44 |
| fungi | it can see what version is installed, but if for example sqlite isn't installed there yet then there is no version number known for it | 16:44 |
| ihrachys | fungi: ack I will need to check that | 16:45 |
| *** fguillot has quit IRC | 16:45 | |
| fungi | the feature was originally added back when bindep was debian-only, and how other distros track package versions wasn't taken into account at the time | 16:45 |
| *** amitgandhinz has quit IRC | 16:45 | |
| AJaeger | thanks, jeblair and pabelanger ! | 16:45 |
| fungi | ihrachys: so when we extended support to additional distros, version handling was left as something to add down the road when people needed it and figured out how to make it work | 16:46 |
| ihrachys | fungi: so here I am... I will need it :) | 16:46 |
| fungi | ihrachys: yep! i'm excited to see us get it working better | 16:47 |
| ihrachys | fungi: I will chew it in next days | 16:47 |
| fungi | thanks! | 16:47 |
| *** ihrachys has quit IRC | 16:48 | |
| *** yamahata has quit IRC | 16:49 | |
| *** ijw has quit IRC | 16:50 | |
| *** Apoorva has joined #openstack-infra | 16:50 | |
| *** tiswanso has quit IRC | 16:50 | |
| *** sputnik13 has quit IRC | 16:53 | |
| mtreinish | fungi, clarkb, rcarrillocruz: https://review.openstack.org/#/c/381960/ should be good to go I think | 16:54 |
| mtreinish | hopefully that's the last issue to unbreak health.o.o | 16:54 |
| *** weshay_lunch is now known as weshay | 16:54 | |
| openstackgerrit | Jordan Pittier proposed openstack-infra/shade: remove_router_interface: check subnet_id or port_id is provided https://review.openstack.org/381981 | 16:54 |
| *** tnovacik has quit IRC | 16:55 | |
| *** yamahata has joined #openstack-infra | 16:57 | |
| *** weshay is now known as weshay_mtg | 16:58 | |
| *** ccamacho has left #openstack-infra | 16:59 | |
| *** vsaienko has joined #openstack-infra | 16:59 | |
| *** amitgandhinz has joined #openstack-infra | 17:00 | |
| *** strigazi is now known as strigazi_AFK | 17:00 | |
| *** narayrak has quit IRC | 17:00 | |
| *** ralonsoh_ has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| clarkb | mtreinish: mriedem have you had a chance to look at why e-r is unhappy? | 17:00 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: track-upstream for deb-{glare,zaqar-ui} https://review.openstack.org/381984 | 17:00 |
| vsaienko | hello clarkb, could you please check my comment at L142 and let me know what do you think about it: https://review.openstack.org/#/c/352790/14/devstack-vm-gate.sh@142 | 17:00 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/project-config: Enable py35 voting on project shade https://review.openstack.org/381985 | 17:01 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/shade: List py35 in the default tox env list https://review.openstack.org/381987 | 17:03 |
| mtreinish | clarkb: I haven't yet. Do you have the paste handy again. I was afk for most of yesterday | 17:03 |
| *** vhosakot has joined #openstack-infra | 17:03 | |
| *** adrian_otto has quit IRC | 17:03 | |
| jeblair | pabelanger, AJaeger: http://docs-beta.openstack.org/ | 17:03 |
| fungi | mtreinish: did you confirm for 381960 that the cython package gets pandas sdists installing correctly? | 17:04 |
| AJaeger | jeblair: Woot! Thanks | 17:04 |
| fungi | mtreinish: if not, i'll take a stab at making sure it works | 17:04 |
| *** jordanP has quit IRC | 17:04 | |
| *** tuannguyen has quit IRC | 17:05 | |
| clarkb | mtreinish: mriedem http://paste.openstack.org/show/584064/ | 17:05 |
| fungi | jeblair: that's served out of afs? snappy! | 17:05 |
| mtreinish | fungi: I don't have a trusty env up and running right now to easily confirm that | 17:05 |
| clarkb | vsaienko: ya I think its fine if you want to have a generic way to signal roles to devstack and devstack plugins | 17:06 |
| pabelanger | jeblair: AJaeger: looks good, thanks | 17:06 |
| fungi | mtreinish: no worries, just making sure i'm not wasting my time by double-checking it | 17:06 |
| clarkb | vsaienko: re the ssh generic key thing, your comment doesn't really explain what it is used for | 17:06 |
| jeblair | fungi: yeah, the afs root is served from http://files.openstack.org/ | 17:06 |
| * AJaeger now fixes a few absolute references to docs.o.o ;) | 17:06 | |
| clarkb | vsaienko: I am still confused at why we need another ssh key there. Devstack-gate is configuring both sides of the bridge for you using the test system ssh key | 17:06 |
| clarkb | vsaienko: is that needed because neutron has to talk to ironic using some sort of ssh driver and it is the key for that? | 17:07 |
| jeblair | fungi: (docs-beta is rooted one level down at /docs so folks can verify the site with /urls) | 17:07 |
| fungi | neat | 17:07 |
| vsaienko | clarkb: yes, neutron should talk via ssh to subnode | 17:07 |
| clarkb | vsaienko: I see | 17:07 |
| *** krtaylor has joined #openstack-infra | 17:08 | |
| *** tiswanso has joined #openstack-infra | 17:08 | |
| vsaienko | I will update localrc variable names to be more generic | 17:08 |
| clarkb | vsaienko: ok so much of this setup is there to allow interaction between neutron on the controller and ironic on the subndoe? | 17:08 |
| *** vsaienko has quit IRC | 17:08 | |
| *** jpich has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack-infra/zuul: Combine shared queues explicitly https://review.openstack.org/358009 | 17:10 |
| *** strigazi_AFK is now known as strigazi | 17:11 | |
| *** vsaienko has joined #openstack-infra | 17:11 | |
| *** tuannguyen has joined #openstack-infra | 17:11 | |
| strigazi | clarkb, with FIXED_RANGE=10.254.1.0/24 and SUBNETPOOL_PREFIX_V4=$FIXED_RANGE all jobs succeeded in osic see here http://paste.openstack.org/show/584265/ Is it by accident? | 17:12 |
| clarkb | strigazi: no, that likely would make it happy. My point earlier about fixed range is that we already set it to 10.1.0.0/20 you shouldn't be overriding it to 10.254.1.0/24 | 17:13 |
| clarkb | strigazi: the key is getting devstack to continue to honor that data rather than ignoring it | 17:13 |
| vsaienko | clarkb: yes, we need ssh keys for two things. 1st ironic-conductor is talking to libvirt on subnode via ssh and manage VMs on subnode. 2nd neutron talks to subnode and change ovs settings | 17:14 |
| *** vhosakot has quit IRC | 17:15 | |
| *** vhosakot has joined #openstack-infra | 17:15 | |
| clarkb | vsaienko: is there ar eason the second one isn'g neutron talking to an agent over the existing message bus to manage that? I am guessing because those ovs settings are for the fake baremetal not for neutron itself? | 17:15 |
| strigazi | clarkb, is it an acceptable as a temporary solution until the problem is fixed by devstack/neutron ? I can't think of anything else :( | 17:16 |
| clarkb | strigazi: if you want to work around it taht way don't change the FIXED_RANGe just assign it to the subnet pool | 17:16 |
| clarkb | I have been avoiding bandaiding over it like that because thsi really needs to be fixed in devstack imo | 17:16 |
| strigazi | clarkb, I comletely agree with that ^^ | 17:17 |
| vsaienko | clarkb: neutron ml2 driver is designed to talk via ssh, normally it talks to real switches. But on the gates we emulate ironic nodes by VMs, and ml2 driver talks to subnode via ssh | 17:17 |
| strigazi | clarkb, but I would like to follow this path as a temporary solution | 17:18 |
| clarkb | strigazi: ok | 17:18 |
| *** dtantsur is now known as dtantsur|afk | 17:18 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 17:19 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement https://review.openstack.org/378014 | 17:19 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 17:19 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Fall back to previous failure list for older releases https://review.openstack.org/381998 | 17:19 |
| fungi | mtreinish: looks like pandas has an implicit dependency on cython, the python library. apt-get installing that won't help without system site packages enabled in the venv, so instead we may need cython added to the requirements.txt for openstack-health. i'm testing that next | 17:19 |
| vsaienko | clarkb if you are ok to setup ssh keys could you please vote here: https://review.openstack.org/#/c/364830 | 17:20 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton https://review.openstack.org/381309 | 17:20 |
| mtreinish | fungi: :/ sigh, it's never simple | 17:21 |
| openstackgerrit | Merged openstack-infra/shade: Add test for os_group Ansible module https://review.openstack.org/379854 | 17:21 |
| *** ArxCruz has joined #openstack-infra | 17:21 | |
| *** strigazi is now known as strigazi_AFK | 17:22 | |
| *** tiswanso has quit IRC | 17:22 | |
| fungi | mtreinish: oh, worse, since pandas expects to find cython at install time, simply declaring it as a runtime requirement is likely insufficient. we probably need it as a setup_requires in fact to avoid a race where pip might try to install cython after pandas | 17:22 |
| *** eharney has joined #openstack-infra | 17:24 | |
| mtreinish | fungi: it's probably just simple to break the venv and system install openstack-health | 17:24 |
| mtreinish | *break out of | 17:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref https://review.openstack.org/382002 | 17:25 |
| AJaeger | annegentle: ^ | 17:25 |
| *** vsaienko has quit IRC | 17:26 | |
| *** tiswanso has joined #openstack-infra | 17:26 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add provider upload framework https://review.openstack.org/379795 | 17:26 |
| *** vsaienko has joined #openstack-infra | 17:27 | |
| *** vhosakot_ has joined #openstack-infra | 17:27 | |
| *** baoli has joined #openstack-infra | 17:29 | |
| fungi | dstufft: stupid pip user question... we have a package with an install_requires on pandas. in turn, pandas expects cython to be available in the env to build its sdist but doesn't declare that. is it sufficient for our package to declare an install_requires on cython before pandas, or do we need to make cython a setup_requires in our package to ensure it's installed before pandas? | 17:29 |
| *** vsaienko has quit IRC | 17:30 | |
| *** sambetts is now known as sambetts|afk | 17:30 | |
| *** vhosakot has quit IRC | 17:30 | |
| *** vsaienko has joined #openstack-infra | 17:32 | |
| *** esikachev has quit IRC | 17:33 | |
| *** flepied has joined #openstack-infra | 17:34 | |
| *** vsaienko has quit IRC | 17:36 | |
| *** EricGonc_ has quit IRC | 17:36 | |
| fungi | mtreinish: i'm going to submit a pr to pandas to add this as a setup_requires, but in the meantime https://review.openstack.org/382010 | 17:37 |
| *** flepied1 has quit IRC | 17:37 | |
| *** tosky has quit IRC | 17:37 | |
| *** baoli has quit IRC | 17:37 | |
| *** baoli has joined #openstack-infra | 17:38 | |
| mtreinish | fungi: ok I'm +A on that, but I'm quickly gonna add a comment inline about it | 17:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref https://review.openstack.org/382002 | 17:38 |
| fungi | mtreinish: feel free to push another patchset if you want, or i can in a moment | 17:39 |
| fungi | hacking on pandas packaging now | 17:39 |
| mtreinish | fungi: actuall I don;t think that'll fix our issue | 17:39 |
| *** eranrom has joined #openstack-infra | 17:39 | |
| mtreinish | fungi: because we install requirements explicitly first | 17:39 |
| *** pilgrimstack1 has quit IRC | 17:39 | |
| fungi | ohhh | 17:39 |
| fungi | indeed we do | 17:39 |
| mtreinish | fungi: http://git.openstack.org/cgit/openstack-infra/puppet-openstack_health/tree/manifests/api.pp#n108 | 17:40 |
| fungi | i merely confirmed that a pip install cython followed by pip install . worked correctly | 17:40 |
| *** arxcruz_ has quit IRC | 17:40 | |
| *** jschwarz has quit IRC | 17:40 | |
| mtreinish | fungi: we can put cython in o-h requirements right before panda | 17:40 |
| mtreinish | s | 17:40 |
| *** piet_ has joined #openstack-infra | 17:40 | |
| mtreinish | that way we'll always install it first | 17:40 |
| mtreinish | or we can just cap pandas to a version before the implicit cython dep | 17:40 |
| *** hamzy has quit IRC | 17:41 | |
| *** weshay_mtg is now known as weshay | 17:41 | |
| *** ArxCruz has quit IRC | 17:41 | |
| *** jschwarz has joined #openstack-infra | 17:42 | |
| fungi | i'm unsure that it's possible to have them in the same requirements list. i think pip may run setup.py on pandas before cython is fully installed into the env. or at least there used to be some races around trying to treat ordering as a substitute for setup_requires | 17:42 |
| clarkb | cloudnull: on further inspection it seems that part of the issue here may be that the ipv4 subnet in use on osic is a /8 as well | 17:42 |
| clarkb | cloudnull: which means that a new route for 10/8 supercedes the old one rather than being a less specific route that doesn't affect a smaller range | 17:43 |
| mtreinish | fungi: ok, then I'm fine just capping pandas | 17:43 |
| fungi | that may be the easiest route, yes | 17:43 |
| mtreinish | fungi: I think it's only an issue with pandas 0.19.0 which was released 2 days ago | 17:44 |
| sc68cal | jeblair: ping - regarding raw IPv6 links via telnet - I swear there's an RFC - happy eyeballs? where we're supposed to give the ability to dual stack | 17:44 |
| fungi | in the meantime, the api seems to be working again with cython having been manually installed. we'll want to blow away .venv before we land a change with the pandas cap | 17:44 |
| sc68cal | jeblair: and that raw IPv6 addrs are not really good user experience. Think maybe we can design a dual stack proxy? | 17:44 |
| cloudnull | clarkb: so then the only way around that issue is to use a different subnet. yes? | 17:45 |
| clarkb | cloudnull: yes a different smaller subnet | 17:45 |
| clarkb | fungi: ^ may have opinions on that too | 17:45 |
| *** Goneri has quit IRC | 17:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add provider upload framework https://review.openstack.org/379795 | 17:47 |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| mtreinish | fungi: https://review.openstack.org/#/c/382019/ | 17:47 |
| *** xyang1 has quit IRC | 17:47 | |
| jeblair | sc68cal: there will be a websockets proxy in zuulv3 | 17:48 |
| sc68cal | jeblair: ack. if I can help in any way hit me up | 17:49 |
| *** drifterza has joined #openstack-infra | 17:49 | |
| *** fguillot has joined #openstack-infra | 17:49 | |
| jeblair | sc68cal: (but as an aside, the hosts in question are not dual stack hosts -- they are actually ipv6 only) | 17:50 |
| *** tnovacik has joined #openstack-infra | 17:51 | |
| *** claudiub has joined #openstack-infra | 17:51 | |
| sc68cal | jeblair: right - which is causing a UX issue since not everyone has v6 yet | 17:51 |
| *** EricGonczer_ has joined #openstack-infra | 17:51 | |
| *** yamamoto has quit IRC | 17:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref https://review.openstack.org/382002 | 17:52 |
| jeblair | sc68cal: sure, i just want to be clear, there is no rfc that says "you must run all services on both ipv4 and ipv6". the one you're thinking about discusses ways to mitigate delays on dual stack hosts with non-working v6 (which is not the situation we are in). that was to address the whole "my browser is hung" thing. | 17:54 |
| *** hamzy has joined #openstack-infra | 17:55 | |
| cloudnull | clarkb: couldn't devstack simply use the fixed range 10.1.0.0/20 or the 10.100.100.0/24 both of which have been carved out of the osic 10/8 for folks that need a 10.x.x.x network? | 17:55 |
| jeblair | sc68cal: there are (slowly) increasing numbers of sites on the internet that are only accessible over ipv6 -- after all, that's really the crux of ipv4 address exhaustion | 17:55 |
| clarkb | cloudnull: yup, I think we should absolutely do that too | 17:55 |
| cloudnull | I think those are the subnets. | 17:55 |
| * cloudnull goes to look | 17:55 | |
| clarkb | cloudnull: but generally you don't want to have an l2 on a /8 | 17:55 |
| clarkb | er | 17:55 |
| jeblair | sc68cal: having said *that* -- i'm not saying that we should start making our servers ipv6 only -- we try to dual-stack as many of them as possible | 17:55 |
| clarkb | s/l2/interface/ | 17:55 |
| *** tqtran has joined #openstack-infra | 17:55 | |
| sc68cal | jeblair: I'd have to remember the RFC - but basically having sites only accessible over v6 is sort of broken | 17:56 |
| jeblair | sc68cal: and so the zuul status page, gerrit, etc, are all ipv4 and ipv6 -- | 17:56 |
| jeblair | sc68cal: well, i mean, i agree that if you want people to use your sites, it's probably a bad idea at this stage | 17:56 |
| jeblair | sc68cal: but i don't think there's anything *broken* about it -- it's really the whole point of ipv6 is to replace v4 eventually, and it's happening now regardless of what we do here :) | 17:56 |
| sc68cal | right, and I think having raw IPv6 links to get event streams from the CI system causes UX frustration | 17:57 |
| clarkb | and from a pragmatic standpoint this gives us access to 1k more test instances than we would have otherwise | 17:57 |
| jeblair | sc68cal: https://tools.ietf.org/html/rfc6555 is about the user experience of clients on dual-stack hosts | 17:57 |
| jeblair | sc68cal: i agree it causes frustration | 17:57 |
| clarkb | *we wouldn't have | 17:57 |
| *** Sukhdev has joined #openstack-infra | 17:58 | |
| *** sdake has quit IRC | 17:58 | |
| sc68cal | clarkb: right, not arguing that - I just think having raw IPv6 links to the CI logs can be fixed with a proxy | 17:58 |
| jeblair | sc68cal: yep, we agree on this point | 17:58 |
| clarkb | ugh did I get my english right the first time then I broke it?> | 17:58 |
| clarkb | I should find some brunch | 17:58 |
| *** rcernin has joined #openstack-infra | 17:58 | |
| sc68cal | anyway I'd like to help if I am able | 17:58 |
| jeblair | sc68cal: but i also think that this is a great opportunity for folks to get an ipv6 connection -- tunnel brokers abound | 17:59 |
| *** shamail has joined #openstack-infra | 17:59 | |
| AJaeger | fungi, did you see my earlier comment about deb-glare? | 17:59 |
| *** ccamacho has joined #openstack-infra | 18:00 | |
| *** moravec has joined #openstack-infra | 18:00 | |
| jeblair | sc68cal: i think a lot of people see ipv6 as being very theoretical, and it's not -- as clarkb says, we have 1000 workers that we would not otherwise because of ipv6 | 18:00 |
| AJaeger | might have been the usual github issue as well - https://github.com/openstack/deb-glare is empty but git.openstack.org exists... | 18:00 |
| *** cardeois has quit IRC | 18:00 | |
| sc68cal | jeblair: ok, I think though getting people to tunnel for v6 just to see CI logs is .... not ideal | 18:00 |
| sc68cal | I mean hey, I'm all for v6 - remember who I am - but just want to avoid frustration | 18:00 |
| AJaeger | sc68cal: they can get CI logs from logs.openstack.org via gerrit once the run finishes | 18:00 |
| *** degorenko is now known as _degorenko|afk | 18:01 | |
| *** zz_ja is now known as zz_zz_ja | 18:01 | |
| AJaeger | sc68cal: the IPv6 addresses do not block anybody from getting work done. The streaming is nice and might help some to monitor what's going on - but we offer logs to everybody | 18:01 |
| jeblair | sc68cal: yep, and we would have put the websocket proxy in place already if the websocket situation in python2 wasn't so terrible | 18:01 |
| *** tphummel has quit IRC | 18:02 | |
| *** tphummel has joined #openstack-infra | 18:02 | |
| jeblair | sc68cal: but it's a big enough project that we deferred it to zuulv3, since the logs are accessible on v4 and v6 after the job completes | 18:02 |
| *** cardeois has joined #openstack-infra | 18:02 | |
| *** tiswanso has quit IRC | 18:02 | |
| AJaeger | infra-root, could somebody fix https://github.com/openstack/deb-glare setup, please? | 18:03 |
| *** harlowja has quit IRC | 18:03 | |
| jeblair | sc68cal: (the current system was put in place very quickly thanks to some surprises with the old one) | 18:03 |
| *** tiswanso has joined #openstack-infra | 18:03 | |
| AJaeger | mordred: https://review.openstack.org/#/c/381985 makes python35 voting on shade, hope that's ok for you... | 18:04 |
| *** tuannguyen has quit IRC | 18:05 | |
| *** electrofelix has quit IRC | 18:06 | |
| *** adrian_otto has joined #openstack-infra | 18:06 | |
| *** drifterza has quit IRC | 18:07 | |
| sc68cal | jeblair: ack - thanks for bringing me up to speed | 18:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref https://review.openstack.org/382002 | 18:07 |
| *** cardeois_ has joined #openstack-infra | 18:07 | |
| * AJaeger needs far too many iterations for such a change ;( ^ | 18:07 | |
| * AJaeger calls it a day now and waves good-night | 18:08 | |
| fungi | mtreinish: well, an alternative would be to upgrade to pip 8.1.0 or later since they provide manylinux1 wheels on pypi | 18:08 |
| jeblair | sc68cal: thanks for the feedback. and let me know how much time you have to work on zuulv3. :) | 18:08 |
| *** drifterza has joined #openstack-infra | 18:08 | |
| *** adrian_otto has quit IRC | 18:08 | |
| sc68cal | jeblair: ok let me pull down the code and start grepping :) | 18:08 |
| fungi | mtreinish: also, my discussion with the maintainer in https://github.com/pydata/pandas/issues/14204 suggests that they're openly hostile to setup_requires and would rather building their sdist simply break than rely on them | 18:09 |
| sc68cal | jeblair: is it just master branch of openstack-infra/zuul ? | 18:09 |
| *** cardeois has quit IRC | 18:09 | |
| jeblair | sc68cal: it's feature/zuulv3, and we're at the stage were there are no small jobs, only big refactorings happening. but mordred has actually done quite a bit of the work on the websocket thing, it's just not the highest priority right now. | 18:10 |
| jeblair | sc68cal: (i don't think his work is in tree yet, we really need to focus on things like "execute jobs" first) | 18:11 |
| *** pvaneck has joined #openstack-infra | 18:13 | |
| *** pilgrimstack has joined #openstack-infra | 18:13 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 18:13 |
| *** Goneri has joined #openstack-infra | 18:13 | |
| *** tuannguyen has joined #openstack-infra | 18:15 | |
| *** bin_ has quit IRC | 18:17 | |
| *** akshai has quit IRC | 18:18 | |
| mtreinish | fungi: hmm, we don't have pip 8.1.0 on health.o.o? | 18:19 |
| dragonmaster | asselin__: thanks for the help | 18:19 |
| fungi | mtreinish: nope, it's still on 7.1.0 | 18:19 |
| *** EricGonczer_ has quit IRC | 18:19 | |
| dragonmaster | i see that zuul has a state_dir setting, and this is a configurable item in the config | 18:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Always run openstackdocstheme-api-ref https://review.openstack.org/382002 | 18:20 |
| dragonmaster | i think it is what i was missing | 18:20 |
| fungi | mtreinish: and well, the version of virtualenv is what's important here as it bundles its own copy of pip, but it's bundling pip 7.1.0 | 18:20 |
| *** e0ne has joined #openstack-infra | 18:20 | |
| mtreinish | fungi: the more I think about this, the more I'm thinking we want to stop using a venv there | 18:20 |
| mtreinish | it just complicates things | 18:20 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add tool for aborting jobs in jenkins https://review.openstack.org/382029 | 18:20 |
| fungi | mtreinish: makes sense. i'm happy going either way with it | 18:20 |
| mtreinish | it was a low prio todo for me to stop that, maybe I'll just do that after I find some food | 18:21 |
| *** trown|lunch is now known as trown | 18:21 | |
| mtreinish | fungi: we'll still want to upgrade the pip version, but it makes doing upgrades simpler if we don't have to reconstruct the venv all the time | 18:21 |
| fungi | mtreinish: regardless, upgrading to a newer pip/virtualenv will allow us to get manylinux1 wheels pf pandas rather than needing to preinstall cython and numpy | 18:21 |
| jeblair | fungi: aren't manylinux1 things compiled lowest-common-denominator? | 18:22 |
| drifterza | heya dragonmaster :) | 18:22 |
| jeblair | (so perhaps preinstalling those and letting things be compiled is better?) | 18:22 |
| fungi | jeblair: yeah, i think the spec is loosely "rhel 5 with only glibc and a handful of very common libraries" | 18:23 |
| dragonmaster | hey drifterza | 18:23 |
| drifterza | you winning ? | 18:23 |
| *** kzaitsev_mb has quit IRC | 18:26 | |
| *** rohanion has quit IRC | 18:26 | |
| *** dtardivel has quit IRC | 18:27 | |
| *** shardy is now known as shardy_afk | 18:27 | |
| *** Naeil has quit IRC | 18:28 | |
| *** EricGonczer_ has joined #openstack-infra | 18:29 | |
| *** docaedo has quit IRC | 18:29 | |
| *** amoralej is now known as amoralej|off | 18:29 | |
| *** docaedo4 has joined #openstack-infra | 18:29 | |
| *** moravec has quit IRC | 18:29 | |
| *** jpena is now known as jpena|off | 18:30 | |
| dragonmaster | getting there slowly | 18:30 |
| *** tuannguyen has quit IRC | 18:30 | |
| *** Ravikiran_K has quit IRC | 18:31 | |
| *** adrian_otto has joined #openstack-infra | 18:31 | |
| *** hasharAway is now known as hashar | 18:32 | |
| fungi | AJaeger: i'm looking into the deb-glare failure now | 18:32 |
| *** senk_ has quit IRC | 18:33 | |
| *** bnemec has quit IRC | 18:34 | |
| clarkb | fungi: ya https://www.python.org/dev/peps/pep-0513/ the list is in there and relatively small and very old | 18:35 |
| *** tuannguyen has joined #openstack-infra | 18:35 | |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible https://review.openstack.org/371590 | 18:36 |
| clarkb | fungi: which is why the pyerasurecoode project shouldn't publish a wheel since liberasurecode is not part of that list | 18:39 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Switch coverage tox job to use pbr test command https://review.openstack.org/381898 | 18:39 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Remove discover from test-requirements https://review.openstack.org/345794 | 18:40 |
| fungi | mtreinish: seems i've talked a pandas maintainer into the need for a setup_requires on cython finally, fwiw, so will go ahead with a pr after meetings | 18:41 |
| clarkb | fungi: mtreinish catching up basically system install of cython won't help beacuse health runs in a venv and thats where we need cython? | 18:42 |
| *** caphrim007 has quit IRC | 18:42 | |
| *** Rockyg has joined #openstack-infra | 18:43 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add reno jobs for python-tripleoclient https://review.openstack.org/382041 | 18:45 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable the shared/independent queue test https://review.openstack.org/358010 | 18:45 |
| openstackgerrit | Merged openstack-infra/zuul: Handle a project-template with no jobs https://review.openstack.org/360783 | 18:45 |
| openstackgerrit | Merged openstack-infra/zuul: Add an OpenStack test https://review.openstack.org/360784 | 18:45 |
| openstackgerrit | Merged openstack-infra/zuul: Pass assigned nodes to the launcher https://review.openstack.org/360785 | 18:45 |
| openstackgerrit | Merged openstack-infra/zuul: Add a python27 job variant to the OpenStack test https://review.openstack.org/360786 | 18:46 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable test_failed_change_at_head https://review.openstack.org/363821 | 18:46 |
| openstackgerrit | Merged openstack-infra/zuul: Add nodeset as a top-level configuration item https://review.openstack.org/365089 | 18:46 |
| *** Teresita-Warrior has quit IRC | 18:47 | |
| *** ijw has joined #openstack-infra | 18:47 | |
| mriedem | clarkb: on that e-r failure i'm seeing this | 18:47 |
| mriedem | "[FIELDDATA] Data too large, data for [@timestamp] would be larger than limit of [22469214208/20.9gb]]" | 18:47 |
| openstackgerrit | Merged openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests https://review.openstack.org/365099 | 18:47 |
| *** salv-orlando has quit IRC | 18:48 | |
| mriedem | mtreinish: ^ | 18:48 |
| *** vhosakot_ has quit IRC | 18:48 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear: Support _ALL and _REDUCE_ jobs from protocol https://review.openstack.org/382042 | 18:48 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear: WIP: Handle new STATUS commands from protocol https://review.openstack.org/382043 | 18:48 |
| SpamapS | jeblair: ^^ 382042 talks to libgearman again. :-D | 18:49 |
| *** vhosakot has joined #openstack-infra | 18:49 | |
| clarkb | mriedem: oh ok maybe I need to make sure indexes are claened up again | 18:49 |
| SpamapS | morgan: ^ you're playing with gear too right? 382042 | 18:49 |
| mriedem | clarkb: i'll work on fixing that IndexError | 18:50 |
| clarkb | mriedem: I amseeing that in the ES logs too | 18:50 |
| *** akshai has joined #openstack-infra | 18:50 | |
| mriedem | clarkb: yeah i'm able to recreate locally | 18:50 |
| mriedem | tox -r -e venv -- elastic-recheck-uncategorized --verbose | 18:50 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Enable more reasonable logging https://review.openstack.org/360164 | 18:50 |
| clarkb | mriedem: I deleted our oldest index | 18:51 |
| *** spzala has quit IRC | 18:51 | |
| clarkb | it was really small and doesn't seem to have corrected the issue | 18:51 |
| clarkb | went ahead and did the next one too since it was much larger | 18:52 |
| *** ijw has quit IRC | 18:52 | |
| *** spzala has joined #openstack-infra | 18:52 | |
| *** thorst has quit IRC | 18:54 | |
| *** thorst has joined #openstack-infra | 18:55 | |
| anteaya | njohnston: the understanding that I was left with from the discussion about having a private channel for election questions was that queries would be made in public in the #openstack-dev channel with people emailing allo the officials directly for private matters | 18:55 |
| anteaya | tonyb: was going to remove the reference to the private channel from the documentation | 18:55 |
| *** adrian_otto has quit IRC | 18:55 | |
| clarkb | mriedem: I haven't seen anymore instances of that error in the logs. As for handling it e-r side I don't expect your query to end up being happy but thats per index I think. So one way to potentially handle it is log the error but then go on to the next index and just try to do what you can | 18:56 |
| clarkb | mriedem: basically if that error shows up on an index it will likely always persist on that index but moving to the next index may end up working | 18:56 |
| *** spzala has quit IRC | 18:57 | |
| njohnston | anteaya: Yes, that is my understanding as well. I had changes in to add logging to the elections channel, but those were abandoned. | 18:57 |
| njohnston | asa result of those discussions | 18:57 |
| anteaya | njohnston: then why are you instructing people to use the private channel to communicate with election officials regarding missing ballots? | 19:00 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| *** spzala has joined #openstack-infra | 19:01 | |
| *** rwsu_ has joined #openstack-infra | 19:01 | |
| anteaya | and for what its worth I also did not recieve my second ballot, by way of a data point, but I did recieve my first one so can still vote | 19:01 |
| *** docaedo has joined #openstack-infra | 19:02 | |
| *** eranrom has quit IRC | 19:02 | |
| *** vhosakot has quit IRC | 19:03 | |
| *** vhosakot has joined #openstack-infra | 19:03 | |
| *** rwsu has quit IRC | 19:04 | |
| *** spzala has quit IRC | 19:05 | |
| *** xyang1 has joined #openstack-infra | 19:05 | |
| njohnston | anteaya: Glad that you got a ballot! I contacted Andrew Myers, the person who runs CIVS, and he said that it seems the Cornell University SMTP servers are either delaying or discarding some outbound email. | 19:06 |
| *** esikachev has joined #openstack-infra | 19:07 | |
| anteaya | can you get any logs from him to find out what email is delayed or discarded? | 19:07 |
| njohnston | anteaya: And about #openstack-election, the notice that tonyb sent out referred people to http://governance.openstack.org/election/#election-officials which still indicates #openstack-election; I did not want to cause confusion by differing from those instructions. | 19:07 |
| anteaya | njohnston: well how about the election officials get together and discuss? | 19:08 |
| anteaya | since the understand I currently hold is that the private channel isn't to be used for queries | 19:08 |
| anteaya | understanding | 19:08 |
| njohnston | anteaya: Will do. | 19:09 |
| anteaya | so it would be nice to have some consistentcy amoungst the election officials | 19:09 |
| anteaya | thank you | 19:09 |
| anteaya | and thank you for emailing andrew myers, if he is willing to share logs of our email runs we can see if we can identify which email addresses never recieved a ballot | 19:10 |
| njohnston | anteaya: I've emailed him, hopefully he will be able to differentiate our emails from the other elections CIVS runs and be able to furnish us some logs without impinging on the privacy of someone else | 19:13 |
| njohnston | 's poll | 19:13 |
| openstackgerrit | Daniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard https://review.openstack.org/381327 | 19:13 |
| anteaya | njohnston: well if you give him all the details especially the timestamps the ballots were run, I am guessing that would help him narrow down our runs | 19:13 |
| anteaya | and thank you | 19:14 |
| njohnston | anteaya: My pleasure. | 19:14 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: Full quickstart gate run on OVB https://review.openstack.org/381094 | 19:14 |
| mriedem | clarkb: i'm working on a patch | 19:15 |
| *** tuannguyen has quit IRC | 19:20 | |
| *** david-lyle has quit IRC | 19:22 | |
| *** cardeois_ has quit IRC | 19:22 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Skip hits that don't have data https://review.openstack.org/382049 | 19:23 |
| mriedem | clarkb: ^ is hacky but i'm leaving soon | 19:23 |
| mriedem | and fixes the immediate breakage | 19:23 |
| mriedem | mtreinish: ^ | 19:23 |
| *** cardeois_ has joined #openstack-infra | 19:26 | |
| mtreinish | clarkb: yeah we need cython in the venv to build pandas | 19:27 |
| *** vhosakot has quit IRC | 19:27 | |
| mtreinish | mriedem: ok I'll take a look | 19:27 |
| *** spzala has joined #openstack-infra | 19:28 | |
| *** rkukura has quit IRC | 19:29 | |
| *** cardeois_ has quit IRC | 19:32 | |
| *** mriedem has quit IRC | 19:32 | |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible https://review.openstack.org/371590 | 19:33 |
| clarkb | mtreinish: mriedem e-r looks like it may still be failing after the index cleanup | 19:33 |
| clarkb | I still get an index out of range error | 19:33 |
| clarkb | but no new logs about the memory issue. Maybe those two things are unrelated? | 19:34 |
| mat128 | fguillot: lgtm :) | 19:34 |
| *** ijw has joined #openstack-infra | 19:35 | |
| openstackgerrit | Daniel Castellanos proposed openstack-infra/project-config: Update Jenkins jobs for craton-dashboard https://review.openstack.org/381327 | 19:36 |
| *** cardeois has joined #openstack-infra | 19:37 | |
| *** amitgandhinz has quit IRC | 19:39 | |
| *** ijw has quit IRC | 19:40 | |
| *** matrohon has joined #openstack-infra | 19:40 | |
| asselin__ | dragonmaster, cool, glad to hear you found the config setting | 19:40 |
| *** amitgandhinz has joined #openstack-infra | 19:40 | |
| mnaser | is tarballs.openstack.org having issues? it seems like the connectivity is being dropped mid-file download | 19:41 |
| *** shardy_afk is now known as shardy | 19:41 | |
| mnaser | "2016-10-04 19:39:41 (1.57 MB/s) - Connection closed at byte 150487655. Retrying." | 19:41 |
| *** jkilpatr has quit IRC | 19:42 | |
| *** cardeois_ has joined #openstack-infra | 19:42 | |
| *** cardeois has quit IRC | 19:44 | |
| *** woodster_ has joined #openstack-infra | 19:45 | |
| anteaya | mnaser: can you share more of your log? | 19:45 |
| mnaser | its just a simple wget that closed, happened when i wget'd the file on my local machine too | 19:45 |
| *** Apoorva has quit IRC | 19:47 | |
| *** Apoorva has joined #openstack-infra | 19:48 | |
| *** Apoorva has quit IRC | 19:48 | |
| *** Apoorva has joined #openstack-infra | 19:48 | |
| *** jkilpatr has joined #openstack-infra | 19:48 | |
| morgan | SpamapS: I was... Haven't been lately. | 19:49 |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible https://review.openstack.org/371590 | 19:50 |
| clarkb | mnaser: I see a small number of segfaults in the apache log | 19:50 |
| clarkb | thats interesting | 19:50 |
| *** moravec has joined #openstack-infra | 19:51 | |
| *** salv-orlando has joined #openstack-infra | 19:51 | |
| SpamapS | morgan: ah cool. It doesn't get many reviewers, so if you have a chance, take a gander. | 19:51 |
| *** rkukura has joined #openstack-infra | 19:51 | |
| mtreinish | clarkb: I just tried mriedem's change locally and it seems to workaround the issue | 19:51 |
| SpamapS | I'm looking at coalescing now | 19:51 |
| mtreinish | clarkb: my only concern with it is that it is silent about the bad indexes | 19:51 |
| *** Teresita-Warrior has joined #openstack-infra | 19:52 | |
| clarkb | mtreinish: ok, I am seeing the error continue on status.o.o but the bad indexes should be gone | 19:52 |
| clarkb | mtreinish: so maybe its some other issue too? | 19:52 |
| mtreinish | clarkb: hmm, yeah I'm not sure | 19:53 |
| *** nadya has joined #openstack-infra | 19:54 | |
| *** eharney has quit IRC | 19:55 | |
| *** salv-orlando has quit IRC | 19:55 | |
| *** moravec has quit IRC | 19:57 | |
| *** salv-orlando has joined #openstack-infra | 19:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:59 | |
| *** ArxCruz has joined #openstack-infra | 19:59 | |
| *** salv-orlando has quit IRC | 19:59 | |
| *** salv-orlando has joined #openstack-infra | 20:00 | |
| *** dprince has quit IRC | 20:01 | |
| *** csomerville has quit IRC | 20:02 | |
| *** shardy has quit IRC | 20:03 | |
| *** Teresita-Warrior has quit IRC | 20:03 | |
| *** ArxCruz has quit IRC | 20:04 | |
| *** jkilpatr has quit IRC | 20:06 | |
| *** karthikp has quit IRC | 20:07 | |
| *** david-lyle has joined #openstack-infra | 20:07 | |
| *** eharney has joined #openstack-infra | 20:08 | |
| *** ccamacho has left #openstack-infra | 20:08 | |
| *** cardeois has joined #openstack-infra | 20:08 | |
| *** Goneri has quit IRC | 20:09 | |
| *** e0ne has quit IRC | 20:09 | |
| rcarrillocruz | can i get more eyes on https://review.openstack.org/#/c/362490/1 ? if there are no objections i can babysit | 20:09 |
| *** cardeois_ has quit IRC | 20:09 | |
| *** adrian_otto has joined #openstack-infra | 20:09 | |
| *** karthikp has joined #openstack-infra | 20:09 | |
| *** shamail has quit IRC | 20:10 | |
| *** cardeois_ has joined #openstack-infra | 20:11 | |
| *** pilgrimstack has quit IRC | 20:11 | |
| *** cardeois has quit IRC | 20:13 | |
| pleia2 | rcarrillocruz: lgtm, feel free to +A at will | 20:14 |
| rcarrillocruz | thx | 20:15 |
| *** harlowja has joined #openstack-infra | 20:15 | |
| rcarrillocruz | jesusaur: ^ approving your parameterize snmp stuff for puppet-iptables | 20:15 |
| *** mixos has joined #openstack-infra | 20:16 | |
| *** cardeois has joined #openstack-infra | 20:17 | |
| *** cardeois_ has quit IRC | 20:18 | |
| *** david-lyle has quit IRC | 20:19 | |
| *** cardeois has quit IRC | 20:19 | |
| *** david-lyle has joined #openstack-infra | 20:19 | |
| *** jkilpatr has joined #openstack-infra | 20:20 | |
| *** cardeois has joined #openstack-infra | 20:20 | |
| *** tuannguyen has joined #openstack-infra | 20:21 | |
| openstackgerrit | Merged openstack-infra/puppet-iptables: Parameterize SNMP source hosts https://review.openstack.org/362490 | 20:21 |
| *** cardeois_ has joined #openstack-infra | 20:21 | |
| *** phschwartz has quit IRC | 20:22 | |
| *** mordred has quit IRC | 20:23 | |
| *** cardeois has quit IRC | 20:25 | |
| *** tuannguyen has quit IRC | 20:25 | |
| jesusaur | awesome, thanks | 20:26 |
| *** esikachev has quit IRC | 20:30 | |
| *** coolsvap has quit IRC | 20:32 | |
| *** dfflanders has joined #openstack-infra | 20:32 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata for compute057.chocolate https://review.openstack.org/382061 | 20:32 |
| dstufft | fungi: uhhh | 20:33 |
| dstufft | fungi: I think the answer is neither, at least if it works it only works on accident | 20:33 |
| *** Teresita-Warrior has joined #openstack-infra | 20:33 | |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add os_almanach to OpenStack-Ansible https://review.openstack.org/371590 | 20:33 |
| *** nadya has quit IRC | 20:34 | |
| *** Jeffrey4l_ has joined #openstack-infra | 20:34 | |
| *** cardeois has joined #openstack-infra | 20:35 | |
| *** john-davidge has joined #openstack-infra | 20:35 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata for compute055.chocolate https://review.openstack.org/382065 | 20:36 |
| *** cardeois_ has quit IRC | 20:37 | |
| fungi | dstufft: got it. after some back and forth i think i convinced a pandas maintainer that adding cython to their setup_requires rather than just aborting setup.py after attempting to import it was slightly better for their users | 20:37 |
| *** cardeois_ has joined #openstack-infra | 20:39 | |
| *** shardy has joined #openstack-infra | 20:39 | |
| *** john-davidge has quit IRC | 20:39 | |
| *** cardeois has quit IRC | 20:39 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update IPs and metadata of compute056.chocolate https://review.openstack.org/382066 | 20:39 |
| *** cardeois has joined #openstack-infra | 20:42 | |
| *** cardeois_ has quit IRC | 20:43 | |
| *** dragonmaster has quit IRC | 20:45 | |
| EmilienM | hi, any chance to have a last review on https://review.openstack.org/#/c/381322 please? | 20:45 |
| *** kgiusti has quit IRC | 20:47 | |
| clarkb | EmilienM: quick question about that. Does tripleo install nova from source? | 20:48 |
| clarkb | if not that job isn't going to help much | 20:48 |
| EmilienM | clarkb: yes | 20:48 |
| EmilienM | clarkb: we build an rpm by using a tool called delorean | 20:48 |
| pleia2 | EmilienM: lgtm | 20:48 |
| EmilienM | pleia2: merci! | 20:48 |
| *** cardeois has quit IRC | 20:49 | |
| clarkb | EmilienM: and that rpm is built using the nova change under test and not just eg master? | 20:49 |
| *** phschwartz has joined #openstack-infra | 20:49 | |
| EmilienM | clarkb: right | 20:49 |
| clarkb | cool | 20:49 |
| clarkb | looks like pleia2 got the approval so I have just responded to mriedems question about voting | 20:49 |
| EmilienM | thanks you :) | 20:49 |
| pleia2 | he mentioned in the comments that he'd talked to someone too | 20:50 |
| *** cardeois has joined #openstack-infra | 20:52 | |
| EmilienM | clarkb: if you're interested by how delorean works, here's the source code https://github.com/openstack-packages/DLRN | 20:52 |
| EmilienM | clarkb: we build packages at every commit in OpenStack https://trunk.rdoproject.org/centos7-master/report.html | 20:53 |
| clarkb | EmilienM: ya I have vague familiarity with what it does | 20:53 |
| *** mordred has joined #openstack-infra | 20:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Add TripleO multinode in Nova experimental jobs https://review.openstack.org/381322 | 20:53 |
| rcarrillocruz | bleh | 20:53 |
| rcarrillocruz | i have 6 nodes that textcons interface shows 'Monitor is in graphics mode' | 20:53 |
| *** cardeois has quit IRC | 20:53 | |
| clarkb | EmilienM: was curious if the jobs take into account the change to eg nova under test | 20:54 |
| rcarrillocruz | i need some jumphost to use the iLO graphical console to see what's going on | 20:54 |
| *** chem has quit IRC | 20:54 | |
| clarkb | rcarrillocruz: time to get a vga cable? | 20:54 |
| pleia2 | heh | 20:54 |
| rcarrillocruz | heh | 20:54 |
| *** cardeois has joined #openstack-infra | 20:54 | |
| EmilienM | clarkb: yes, even with Depends-On other projects | 20:54 |
| EmilienM | clarkb: like Cinder, etc | 20:54 |
| EmilienM | clarkb: so if you submit a patch in nova and use Depends-On python-novaclient, it will build both packages | 20:54 |
| *** gyee has joined #openstack-infra | 20:55 | |
| clarkb | EmilienM: perfect | 20:56 |
| clarkb | EmilienM: also someone pointed out to me that delorean is using review.o.o instaed of git.o.o for git clones? maybe we can change that? | 20:56 |
| clarkb | jeblair: mordred the stack starting at https://review.openstack.org/#/c/371679/6 could probably use some attention to get in finally | 20:56 |
| clarkb | rcarrillocruz: ^ may interest you as well | 20:57 |
| openstackgerrit | Merged openstack-infra/system-config: Update IPs and metadata for compute057.chocolate https://review.openstack.org/382061 | 20:57 |
| EmilienM | clarkb: I didn't know that. Maybe dmsimard knows? | 20:57 |
| rcarrillocruz | let me see | 20:57 |
| *** adriant has joined #openstack-infra | 20:57 | |
| * dmsimard reading | 20:57 | |
| clarkb | rcarrillocruz: there was a great big push on that right before we all got on planes to germany then that caused the work to stall | 20:58 |
| clarkb | or at least the reviews | 20:58 |
| rcarrillocruz | i miss all the fun! | 20:58 |
| * rcarrillocruz sobs | 20:58 | |
| dmsimard | clarkb, EmilienM: yeah I saw that too todya ? I have no clue where that comes from, our config definitely uses git.o.o. | 20:58 |
| dmsimard | s/todya/today/ | 20:58 |
| dmsimard | If it uses review.o.o it's a bug and we'll address it but we have to find where it's picking that up | 20:59 |
| *** pilgrimstack has joined #openstack-infra | 20:59 | |
| clarkb | fungi: pabelanger tyty for reviewing the change to fix nova net multinode testing | 20:59 |
| dmsimard | bkero: can you let me know where you saw the git clone from review.openstack.org ? | 20:59 |
| bkero | dmsimard: How's this? https://github.com/redhat-openstack/ansible-role-tripleo-gate/blob/master/tasks/dlrn-build.yml#L29 | 21:00 |
| dmsimard | well that's not delorean at all :) | 21:00 |
| rcarrillocruz | clarkb: i've done mental variable substitutions and lgtm | 21:00 |
| rcarrillocruz | +2 | 21:00 |
| bkero | dmsimard: Yeah, had to go read the code to determine what was actually doing the cloning. | 21:01 |
| dmsimard | bkero: adarazs is the guardian of that I think, I'll poke him | 21:01 |
| dmsimard | thanks | 21:01 |
| bkero | good luck | 21:01 |
| *** mhickey has quit IRC | 21:01 | |
| clarkb | rcarrillocruz: ya I kind of wanted jeblair to do a once over on it just because he was the one exercising it when it came up | 21:02 |
| *** krtaylor has quit IRC | 21:02 | |
| *** rfolco has quit IRC | 21:02 | |
| *** pilgrimstack has quit IRC | 21:04 | |
| *** rkukura_ has joined #openstack-infra | 21:04 | |
| *** david-lyle has quit IRC | 21:04 | |
| *** tuannguyen has joined #openstack-infra | 21:04 | |
| clarkb | rcarrillocruz: ansible 2.2 final realse should happen at the end of the week or early next week? | 21:05 |
| rcarrillocruz | hmm, i expect at least a couple more RCs, mordred Shrews ? | 21:05 |
| rcarrillocruz | superexcited about the include_role as a task thing | 21:05 |
| clarkb | (I am going off of their release process doc which says 5 days for RC1 and 2 days for subsequent RCs | 21:05 |
| dmsimard | rcarrillocruz: what? | 21:06 |
| clarkb | I just want the bug fix for the emtpy inventory group :) | 21:06 |
| dmsimard | rcarrillocruz: an include_role task ? | 21:06 |
| rcarrillocruz | that efectively allows to call a role from a role | 21:06 |
| dmsimard | rcarrillocruz: omg | 21:06 |
| dmsimard | rcarrillocruz: not just that | 21:06 |
| dmsimard | rcarrillocruz: it'd allow you to wrap roles inside a block | 21:06 |
| *** rkukura has quit IRC | 21:06 | |
| *** rkukura_ is now known as rkukura | 21:06 | |
| dmsimard | with rescues | 21:06 |
| clarkb | dmsimard: oh thats shiny | 21:06 |
| rcarrillocruz | dmsimard: https://docs.ansible.com/ansible/include_role_module.html :D | 21:06 |
| rcarrillocruz | and it allows to also change the entry point of a role | 21:06 |
| rcarrillocruz | e.g. go to main.yml or go to main2.yml | 21:07 |
| *** cody-somerville has joined #openstack-infra | 21:07 | |
| dmsimard | rcarrillocruz: https://github.com/ansible/ansible/issues/13587 | 21:07 |
| clarkb | basically I think we will be ready to turn on tls in the multinode jobs as soon as ansible 2.2 is available as it fixes a bunch of bugs necessary to do the setup cleany | 21:07 |
| rcarrillocruz | hehe yeah | 21:07 |
| rcarrillocruz | and it's also promising their claim of performance | 21:07 |
| dmsimard | rcarrillocruz: that is super awesome and could've been easy to miss, thanks so much for letting me know about that .. it'll allow me to clean up so much of my code | 21:08 |
| dmsimard | <3 | 21:08 |
| *** tuannguyen has quit IRC | 21:09 | |
| rcarrillocruz | ++ | 21:10 |
| rcarrillocruz | i'm going to make use of this for the launcher...will use it for my long standing pre/post create/delete tasks per resource action item | 21:10 |
| *** thorst has quit IRC | 21:10 | |
| clarkb | though if people are willing to do reviews today I have hacked around the 2.1 ansible issues | 21:10 |
| *** rtheis has quit IRC | 21:11 | |
| clarkb | https://review.openstack.org/#/c/373219/13 is the change needed to bootstrap tls for multinode and ansible 2.1, its child change uses the current 2.2 rc1 to test it simplified with ansible 2.2 | 21:11 |
| clarkb | and https://review.openstack.org/#/c/372374/ incorporates the whole stack with tls forced on to see that it actually does work | 21:12 |
| *** trown is now known as trown|outtypewww | 21:12 | |
| jeblair | clarkb, rcarrillocruz: i'm game for +3ing pansible if we're willing to risk a puppetmaster stoppage if something goes wrong | 21:13 |
| jeblair | i will be around for several more hours to deal with potential fallout | 21:13 |
| clarkb | jeblair: I don't know of any very urgent puppet related things right now | 21:13 |
| Shrews | clarkb: rcarrillocruz: depends on what issues are found. i would expect a few more RCs, personally | 21:13 |
| clarkb | and its a couple days before release when we likely should prvent that. All that to say I think its reasonably safe and we will know what to revert if anything pop ups | 21:14 |
| openstackgerrit | Merged openstack-infra/system-config: Update IPs and metadata for compute055.chocolate https://review.openstack.org/382065 | 21:14 |
| rcarrillocruz | i will be around at least one hour | 21:14 |
| clarkb | jeblair: I will be around too, if you want to go ahead and +A the stack | 21:14 |
| rcarrillocruz | so we can probably just go for it | 21:14 |
| jeblair | clarkb: all approved | 21:15 |
| *** shardy has quit IRC | 21:15 | |
| *** tiswanso has quit IRC | 21:15 | |
| *** matrohon has quit IRC | 21:15 | |
| *** raildo has quit IRC | 21:15 | |
| openstackgerrit | Merged openstack-infra/system-config: Update IPs and metadata of compute056.chocolate https://review.openstack.org/382066 | 21:15 |
| fungi | yeah, fine by me. we're not changing anything ci-related (worst case we might break job updates) | 21:15 |
| jeblair | aww, ansible-role-puppet doesn't share a queue with system-config | 21:15 |
| *** ldnunes has quit IRC | 21:16 | |
| jeblair | (i think we discussed that maybe we should throw an integration test into that repo though) | 21:16 |
| clarkb | Shrews: ok I will go ahead and get this working with 2.1 then | 21:16 |
| jeblair | so we'll need to poke the system-config changes after the role changes land | 21:16 |
| openstackgerrit | Nate Potter proposed openstack-infra/project-config: Rename RSC project to Valence https://review.openstack.org/382072 | 21:17 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Make sure environment winds up in basemodulepath https://review.openstack.org/371679 | 21:17 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Pass environment to puppet module https://review.openstack.org/371710 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Allow any logdest https://review.openstack.org/371764 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Add noop argument https://review.openstack.org/371774 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Simplify puppet cmd generation https://review.openstack.org/371778 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Add debug argument https://review.openstack.org/371811 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Don't set detailed-exitcodes when noop https://review.openstack.org/371827 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Sync puppet module changes from upstream ansible https://review.openstack.org/371835 | 21:18 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Remove basemodulepath from environment config https://review.openstack.org/371812 | 21:18 |
| *** vhosakot has joined #openstack-infra | 21:19 | |
| *** thorst has joined #openstack-infra | 21:19 | |
| *** r-daneel has joined #openstack-infra | 21:20 | |
| *** senk_ has joined #openstack-infra | 21:20 | |
| *** docaedo4 has quit IRC | 21:21 | |
| *** thorst has quit IRC | 21:24 | |
| *** dfflanders has quit IRC | 21:25 | |
| *** adam_g` is now known as adam_g | 21:25 | |
| *** jkilpatr has quit IRC | 21:26 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Do Not Merge - Test Undercloud upgrade mitaka -> newton https://review.openstack.org/381309 | 21:27 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Undercloud upgrade for mitaka and newton https://review.openstack.org/381286 | 21:27 |
| *** dfflanders has joined #openstack-infra | 21:28 | |
| *** burgerk__ has joined #openstack-infra | 21:29 | |
| *** jheroux has quit IRC | 21:30 | |
| *** akshai has quit IRC | 21:30 | |
| erlon | clarkb: Im trying to understand, one of those tests, despite of being checked out in the gate are not running | 21:30 |
| erlon | clarkb: does the tag: @test.attr(type='slow') affects in something? | 21:31 |
| erlon | clarkb: if not what does it implies? | 21:31 |
| *** mixos has quit IRC | 21:31 | |
| erlon | clarkb: sure, if you know | 21:31 |
| clarkb | erlon: I think that means its a "slow" test for the long periodic run | 21:32 |
| clarkb | which may be filtered out for the shorter gate runs | 21:32 |
| erlon | clarkb: long periodic run? | 21:32 |
| rcarrillocruz | k, three more deploys from servers i rescued from limbo, aka they were not inventoried yet on hiera | 21:32 |
| *** burgerk_ has quit IRC | 21:32 | |
| clarkb | there is a job or two tha run the long tests periodically (like once a day) that aren't run on every change | 21:33 |
| *** burgerk__ has quit IRC | 21:33 | |
| *** david-lyle has joined #openstack-infra | 21:33 | |
| *** vhosakot_ has joined #openstack-infra | 21:34 | |
| *** vhosakot has quit IRC | 21:35 | |
| erlon | clarkb: hmmm that makes sense if it is not running | 21:35 |
| *** moravec has joined #openstack-infra | 21:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add getProjectBranches to Source https://review.openstack.org/382076 | 21:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Implement branch-specific in-repo job config https://review.openstack.org/382077 | 21:36 |
| jeblair | clarkb: the system config patches failed | 21:37 |
| * clarkb digs up system config patches | 21:38 | |
| jeblair | clarkb: https://review.openstack.org/371762 | 21:38 |
| jeblair | clarkb: http://logs.openstack.org/62/371762/14/gate/gate-infra-puppet-apply-centos-7/a20fd49/console.html | 21:38 |
| *** amitgandhinz has quit IRC | 21:38 | |
| *** senk_ has quit IRC | 21:38 | |
| *** mixos has joined #openstack-infra | 21:39 | |
| clarkb | compile error? | 21:39 |
| clarkb | oh! | 21:39 |
| *** eharney has quit IRC | 21:40 | |
| clarkb | newer ansible relies on newer paramiko which needs cryptography | 21:40 |
| clarkb | and before we weren't using ansible in the apply test | 21:40 |
| clarkb | so I think we can just put a bindep update to include libffi-dev under that stack | 21:40 |
| jeblair | and that changed since 2 weeks ago? | 21:40 |
| *** cody-somerville has quit IRC | 21:40 | |
| clarkb | I don't think the ansible side changes that recently but our bindep side may have | 21:41 |
| *** annegentle has quit IRC | 21:41 | |
| jeblair | other jobs seem to be failing for different reasons? | 21:41 |
| *** matt-borland has quit IRC | 21:41 | |
| pabelanger | I can confirm the libffi-dev / libssl-dev issue with ansible | 21:42 |
| *** jamesdenton has quit IRC | 21:42 | |
| *** david-lyle has quit IRC | 21:42 | |
| clarkb | ah ya the ubuntu ones seem to be getting past that issue (I am guessing that a wheel or similar may be helping us work there) | 21:42 |
| *** vhosakot_ has quit IRC | 21:42 | |
| jeblair | centos7 and f24 failed on the compile error | 21:42 |
| jeblair | the others are failing on not finding the puppet manifest file (???) | 21:43 |
| clarkb | jeblair: http://logs.openstack.org/62/371762/14/gate/gate-infra-puppet-apply-ubuntu-xenial/d55b79d/console.html#_2016-10-04_21_20_01_054895 we seem to still hit it on ubuntu | 21:43 |
| clarkb | its just not fatal for some reason which is a little weird | 21:43 |
| jeblair | clarkb: weird... i think i still suspect a second problem | 21:45 |
| *** jcoufal_ has quit IRC | 21:45 | |
| jeblair | cause it doesn't seem like that error would cause the 'manifest file not found' error | 21:45 |
| jeblair | (but maybe? through some very convoluted process?) | 21:45 |
| *** david-lyle has joined #openstack-infra | 21:47 | |
| *** krtaylor has joined #openstack-infra | 21:47 | |
| clarkb | jeblair: all of the failed ones also have a .final file | 21:47 |
| *** david-lyle has quit IRC | 21:48 | |
| *** akshai has joined #openstack-infra | 21:48 | |
| *** david-lyle has joined #openstack-infra | 21:48 | |
| jeblair | clarkb: yeah, i think that's just part of the process of generating the input | 21:48 |
| jeblair | clarkb: (eg, ".final" is what we give to puppet after doing sed, etc) | 21:48 |
| *** docaedo4 has joined #openstack-infra | 21:48 | |
| *** akshai has quit IRC | 21:49 | |
| jeblair | clarkb: and the 1:2 between .final and .final.out.FAILED means "all puppet runs are failing" | 21:49 |
| jeblair | er 1:1 | 21:49 |
| clarkb | ya but only 3 of them are in that state (have a .final and also happen to have a .FAILED) | 21:49 |
| clarkb | oh I understand | 21:49 |
| clarkb | thsoe are the only ones that matched the distro release specifier | 21:49 |
| jeblair | clarkb: probably means only 3 node defs matched | 21:49 |
| jeblair | right | 21:49 |
| clarkb | ya I agree I think there is something else going on there | 21:50 |
| clarkb | it does look like its looking for a relative path to the manifest file, are we sure that when asnible does its thing the resulting pwd is in that dir? | 21:50 |
| *** gordc has quit IRC | 21:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-role-puppet: DNM: revert some commits https://review.openstack.org/382081 | 21:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 21:52 |
| jeblair | clarkb: two commits were added to the end of the ansible-role-puppet stack, but i don't think they were tested | 21:52 |
| jeblair | that reverts them to see if they were the cause | 21:52 |
| clarkb | jeblair: ah ok | 21:53 |
| clarkb | reading where we run ansible there is a sudo but that doesn't change pwd | 21:53 |
| clarkb | and the role doesn't seem to do anything magical to get the dir set | 21:54 |
| clarkb | its possible we want to set a full path for that file | 21:54 |
| jeblair | http://logs.openstack.org/62/371762/15/check/gate-infra-puppet-apply-debian-jessie/31a398a/console.html | 21:55 |
| *** Rockyg has quit IRC | 21:55 | |
| jeblair | okay, not the result of those commits | 21:55 |
| *** lezbar has quit IRC | 21:55 | |
| *** thorst has joined #openstack-infra | 21:55 | |
| jeblair | i'll work up a change to set the absolute path | 21:56 |
| *** lezbar has joined #openstack-infra | 21:56 | |
| clarkb | I think we want a change to bindep.txt to add in the proper libffi-dev deps and ^ | 21:56 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: enable undercloud/ssh on multinode jobs https://review.openstack.org/382082 | 21:56 |
| clarkb | I can work on bindep.txt if you want | 21:56 |
| jeblair | clarkb: ++ | 21:56 |
| *** fguillot has quit IRC | 21:58 | |
| *** piet_ has quit IRC | 21:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 21:58 |
| *** mixos has quit IRC | 21:58 | |
| clarkb | jeblair: where should I "place" the change? | 21:58 |
| *** spzala has quit IRC | 21:59 | |
| *** adrian_otto has quit IRC | 21:59 | |
| jeblair | clarkb: bottom of the stack? | 21:59 |
| clarkb | ok | 21:59 |
| jeblair | (i had to make the pwd change in 762) | 21:59 |
| jeblair | so let's rebase 762 on the bindep change | 21:59 |
| *** adrian_otto has joined #openstack-infra | 22:00 | |
| *** thorst has quit IRC | 22:00 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 22:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add libffi dev packages needed for ansible install https://review.openstack.org/382086 | 22:01 |
| clarkb | jeblair: ^ there reabsed | 22:01 |
| *** senk_ has joined #openstack-infra | 22:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Implement branch-specific in-repo job config https://review.openstack.org/382077 | 22:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add getProjectBranches to Source https://review.openstack.org/382076 | 22:02 |
| mnaser | fyi tarballs.openstack.org still dropping connections for $some_reason | 22:02 |
| mnaser | 4 failures during the download of the manila image (wget resume helps but yea) | 22:02 |
| clarkb | mnaser: does it always fail at the same spot? | 22:03 |
| mnaser | good question let me check logs clarkb | 22:03 |
| *** baoli has quit IRC | 22:03 | |
| *** EricGonczer_ has quit IRC | 22:03 | |
| clarkb | mnaser: and file is http://tarballs.openstack.org/manila-image-elements/images/manila-service-image-master.qcow2 ? | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet https://review.openstack.org/371714 | 22:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel https://review.openstack.org/371823 | 22:04 |
| mnaser | clarkb, http://paste.openstack.org/show/584321/ -- and yes | 22:04 |
| mnaser | thats 2 failures (out of 2 downloads). | 22:05 |
| clarkb | ok so its not consistent in location | 22:05 |
| clarkb | mnaser: and any idea if that is via ipv4 or ipv6? I am ipv4 locally and so far it hasn't broken for me | 22:05 |
| clarkb | mnaser: maybe you can force v4 or v6 if the other was used and failing? (we serve via apache mod auto index so its not a very complicated setup that should fail much) | 22:06 |
| *** adrian_otto has quit IRC | 22:06 | |
| mnaser | Connecting to tarballs.openstack.org (tarballs.openstack.org)|23.253.108.137|:80... connected. | 22:06 |
| mnaser | (for all failures) | 22:06 |
| *** hashar has quit IRC | 22:06 | |
| *** mdrabe has quit IRC | 22:07 | |
| mnaser | yeah its kinda confusing as to why its happening | 22:07 |
| clarkb | heh /me sees a slackbot in the logs for when I pasted the link above presumably | 22:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: DNM: squashed pansible https://review.openstack.org/382087 | 22:08 |
| mnaser | i'll see if this repeats, thats weird, dont have any issues with other servers | 22:09 |
| jeblair | clarkb: hooray for information leakage | 22:09 |
| jeblair | clarkb, mnaser: we saw apache segfaults on the mirrors a while back. i don't think anyone solved that. | 22:09 |
| mnaser | maybe that could have to do with it | 22:09 |
| mnaser | im troubleshooting another fun issue at the moment so ill give feedback a bit later | 22:10 |
| *** lseki has quit IRC | 22:10 | |
| *** edmondsw has quit IRC | 22:10 | |
| clarkb | thee are segfaults in the error log | 22:12 |
| clarkb | now trying to map that back to the requests in the access log | 22:13 |
| jeblair | clarkb: in 382087 i squashed all the changes together and depended on the revert -- that is making the non-redhat jobs pass | 22:13 |
| *** gouthamr has quit IRC | 22:13 | |
| jeblair | clarkb: so i think the "rc1" error may have been introduced by changes to ansible-role puppet. no idea how the 'pwd' error happened. and it's looking like maybe the bindep thing isn't helping? | 22:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: DNM: squashed pansible2 https://review.openstack.org/382090 | 22:15 |
| clarkb | jeblair: at least on debuntu libffi-dev fixes the error installing ansible http://logs.openstack.org/87/382087/1/check/gate-infra-puppet-apply-ubuntu-xenial/8dfc529/console.html#_2016-10-04_22_09_55_927867 | 22:15 |
| jeblair | (that's the same squashed change without the depends-on) | 22:15 |
| jeblair | clarkb: maybe also need to install something for openssl? | 22:16 |
| clarkb | ya looks like I need ssl dev headers on centos | 22:16 |
| clarkb | let me update the change with those | 22:16 |
| jeblair | "openssl-devel" ? | 22:17 |
| clarkb | and probably should include them on ubuntu for completeness (worried we are getting them transitvely without us noticing) | 22:17 |
| jeblair | ++ | 22:18 |
| clarkb | https://apps.fedoraproject.org/packages/openssl-devel and http://packages.ubuntu.com/xenial/libssl-dev | 22:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add libffi dev packages needed for ansible install https://review.openstack.org/382086 | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config" https://review.openstack.org/382092 | 22:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Sync puppet module changes from upstream ansible" https://review.openstack.org/382093 | 22:20 |
| clarkb | (unrelated and unscientific observation seems like the github PR closer bot has been very busy the last few days) | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet https://review.openstack.org/371714 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel https://review.openstack.org/371823 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: DNM: squashed pansible https://review.openstack.org/382087 | 22:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Turn on tls-proxy in nova net multinode job https://review.openstack.org/382094 | 22:22 |
| clarkb | woot http://logs.openstack.org/74/372374/22/check/gate-tempest-dsvm-multinode-full-ubuntu-xenial/3dd7f29/ passes which means that ^ is ready for review | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: DNM: squashed pansible2 https://review.openstack.org/382090 | 22:22 |
| JayF | clarkb: awesome, and ty for the patches to ironic to make us respect all the proper vars | 22:23 |
| JayF | TLS FTW | 22:24 |
| clarkb | JayF: still have to figure out a good way to deal with the cert on the ipa image (and other images) | 22:24 |
| clarkb | JayF: maybe a toggle that says "don't verify ssl because testing" though that makes the image less good for production | 22:24 |
| *** tpsilva has quit IRC | 22:24 | |
| JayF | If we added that toggle, we'd trigger it via kernel command line | 22:24 |
| JayF | and only configure it on during testing | 22:24 |
| JayF | IPA is a strange beast, because you're bootstrapping with an inherently untrusted protocol (PXE) we've been loathe to implement even some of the basic auth measures because it's all subject to simple attack if the network is compromised | 22:25 |
| clarkb | gotcha | 22:25 |
| JayF | let me see if I can whip up something for you real quick | 22:26 |
| clarkb | JayF: I will defer to ironic on how they want to handle that then, but I expect that ironic's jobs will work with tls too at this point once that hurdle is jumped | 22:26 |
| clarkb | JayF: you can use https://review.openstack.org/372374 as a depends on (in either direction) to show that it works | 22:26 |
| jeblair | clarkb: things are pointing toward https://review.openstack.org/382092 being the remaining culprit | 22:28 |
| jeblair | I'm going to go aheand and depends-on the stack to that | 22:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 22:29 |
| *** xarses has quit IRC | 22:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Stop managing puppet config in puppet https://review.openstack.org/371714 | 22:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Run ansible apply test in parallel https://review.openstack.org/371823 | 22:29 |
| *** pcrews has quit IRC | 22:29 | |
| clarkb | oh right that was the one we had trouble with before too but nibalizer confirmed we should be able to remove it so :/ | 22:33 |
| *** pcrews has joined #openstack-infra | 22:35 | |
| *** rossella_s has quit IRC | 22:35 | |
| *** rossella_s has joined #openstack-infra | 22:35 | |
| jeblair | yeah, i think we've got it -- some jobs are still running, but one of each has passed somewhere in the current stack | 22:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config" https://review.openstack.org/382092 | 22:36 |
| *** Teresita-Warrior has quit IRC | 22:36 | |
| clarkb | jeblair: +2 | 22:36 |
| jeblair | clarkb: everything in topic:pansible should be ready now (i've abandoned or updated commit msgs as appropriate) | 22:37 |
| clarkb | ok will review the remainder | 22:37 |
| jeblair | i think the abandon caused some -1s, i'm rechecking | 22:38 |
| *** baoli has joined #openstack-infra | 22:39 | |
| *** xarses has joined #openstack-infra | 22:40 | |
| jeblair | clarkb: i'm confused because https://review.openstack.org/371832 seems to have worked a while ago | 22:40 |
| clarkb | jeblair: one msall issue in https://review.openstack.org/#/c/371823/9 | 22:40 |
| *** senk_ has quit IRC | 22:40 | |
| clarkb | jeblair: at least for bindep I think ist relatively recent that we stopped using the fallback file in system-config | 22:40 |
| jeblair | clarkb: that changed moved the pwd for logdest out a bit on to line 18 | 22:41 |
| clarkb | as for the pathing things maybe new behavior in newer ansible? | 22:41 |
| clarkb | oh so it did | 22:41 |
| clarkb | vote updated | 22:41 |
| jeblair | (i may or may not have had a good reason for doing that originally... <shrug>) | 22:42 |
| *** cardeois has quit IRC | 22:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add puppet apply jobs to ansible-role-puppet https://review.openstack.org/382096 | 22:44 |
| jeblair | clarkb: also ^ | 22:44 |
| *** Teresita-Warrior has joined #openstack-infra | 22:44 | |
| clarkb | now with all that in place should we be upstream for that role again? I bet our CI is much more comprehensive for it :) | 22:45 |
| jeblair | mordred: https://review.openstack.org/382092 FYI -- i don't understand it yet. but it seems to be necessary. it will be easier to revisit after we add the test jobs to the repo | 22:46 |
| *** pradk has quit IRC | 22:48 | |
| jeblair | clarkb: https://review.openstack.org/371823 is green :) | 22:48 |
| openstackgerrit | Merged openstack-infra/ansible-role-puppet: Revert "Remove basemodulepath from environment config" https://review.openstack.org/382092 | 22:48 |
| *** ijw has joined #openstack-infra | 22:48 | |
| clarkb | woo | 22:49 |
| jeblair | i've reapproved the stack, so it should go in in about as long as it takes to run those test jobs serially :) | 22:49 |
| *** rcernin has quit IRC | 22:50 | |
| *** gildub has joined #openstack-infra | 22:50 | |
| mordred | jeblair: kk. I was mostly following spencer's lead there ... so ... nibalizer ^^ | 22:51 |
| *** r-daneel has quit IRC | 22:52 | |
| clarkb | mordred: jeblair with that mostly sorted out can I beg for reviews on https://review.openstack.org/382094 and its depends-on? | 22:52 |
| *** ijw has quit IRC | 22:53 | |
| openstackgerrit | Merged openstack-infra/system-config: Add libffi dev packages needed for ansible install https://review.openstack.org/382086 | 22:55 |
| *** rhallisey has quit IRC | 22:55 | |
| *** senk_ has joined #openstack-infra | 22:55 | |
| *** yamahata has quit IRC | 22:55 | |
| fungi | infra-root: (and anyone else) just fyi a mandatory evacuation is looking likely here. in an effort to still be able to go on our vacation we're planning on leaving friday morning leaving me only a couple days to get the house storm-ready. i'm still planning to be around through the release and hack on summit prep, but not expecting to get any other work done before i head out of town | 22:57 |
| *** yamamoto has joined #openstack-infra | 22:57 | |
| clarkb | fungi: gl | 22:57 |
| fungi | thx | 22:58 |
| pleia2 | fungi: stay safe | 22:58 |
| *** sdake has joined #openstack-infra | 22:58 | |
| pleia2 | fungi: feel free to brain dump anything you need to me, if such a thing is needed :) | 22:58 |
| fungi | pleia2: appreciated, and i definitely will! | 22:58 |
| jeblair | clarkb: lgtm | 23:00 |
| clarkb | jeblair: ty | 23:01 |
| JayF | clarkb: put up a quick POC patchset(s): https://review.openstack.org/382106 is the one that would pass devstack tests if it worked | 23:01 |
| *** salv-orlando has quit IRC | 23:01 | |
| JayF | clarkb: actually, come to think of it, IPA gate is busted if we hit osic cloud, so that might not pass anyway... but we'll see :D | 23:01 |
| clarkb | JayF: oh right that | 23:01 |
| mordred | fungi: yah. stay safe. last storm that hit somewhere named after a person I know turned into a super storm | 23:01 |
| pabelanger | fungi: good luck, will be on standby for releases | 23:02 |
| *** xyang1 has quit IRC | 23:02 | |
| *** john-davidge has joined #openstack-infra | 23:02 | |
| mordred | fungi: given olaph's real name, I have my concerns for this one | 23:02 |
| ianw | clarkb: that segfault ... | 23:02 |
| fungi | mordred: yeah, i heard noaa named this after olaph because they knew him so well | 23:02 |
| ianw | addr2line -e ./mod_mpm_event.so 0x65e0 | 23:02 |
| ianw | /build/apache2-Rau9Dr/apache2-2.4.7/server/mpm/event/event.c:1064 | 23:02 |
| clarkb | ianw: oh did you get a coredump to inspect? | 23:03 |
| ianw | after digging around, that puts it right in the middle of https://github.com/apache/httpd/commit/d62d4c3f5001d44cae2d5b023b36a9ebc0a8e6b5 | 23:03 |
| ianw | actually this line https://github.com/apache/httpd/commit/d62d4c3f5001d44cae2d5b023b36a9ebc0a8e6b5#diff-895d7e9f8add746606c82027dabc04d4L1177 | 23:03 |
| clarkb | ianw: so in theory fixable with a newer apache | 23:04 |
| *** hongbin has quit IRC | 23:04 | |
| anteaya | fungi: travel safely to all your destinations | 23:04 |
| *** mixos has joined #openstack-infra | 23:04 | |
| ianw | clarkb: no, that's from the offsets in the segfault log. there is probably a case for a trusty bug to include that patch | 23:05 |
| mordred | clarkb: +A | 23:05 |
| *** rajinir has quit IRC | 23:05 | |
| *** sdake_ has joined #openstack-infra | 23:05 | |
| *** Apoorva_ has joined #openstack-infra | 23:05 | |
| clarkb | mordred: woot thanks | 23:05 |
| clarkb | ianw: there are more verbose logs than [Tue Oct 04 23:05:36.185405 2016] [core:notice] [pid 3677:tid 140379199530880] AH00052: child pid 31920 exit signal Segmentation fault (11) ? | 23:06 |
| clarkb | ianw: wondering where you found 0x65e0 | 23:06 |
| *** john-davidge has quit IRC | 23:06 | |
| *** tuannguyen has joined #openstack-infra | 23:06 | |
| clarkb | in any case cool that it is tracked down | 23:07 |
| ianw | clarkb: /var/log/kern.log -- subtract the ip from the base address | 23:07 |
| *** Apoorva has quit IRC | 23:08 | |
| clarkb | ianw: aha thanks | 23:08 |
| *** Teresita-Warrior has quit IRC | 23:09 | |
| *** yamamoto has quit IRC | 23:09 | |
| *** tiswanso has joined #openstack-infra | 23:09 | |
| *** sdake has quit IRC | 23:09 | |
| *** amitgandhinz has joined #openstack-infra | 23:09 | |
| ianw | basically, it seems like good luck reproducing it, though. "possible crashes under load" i.e. good as calling random() | 23:09 |
| *** tuannguyen has quit IRC | 23:11 | |
| *** jamielennox is now known as jamielennox|away | 23:12 | |
| *** cody-somerville has joined #openstack-infra | 23:12 | |
| *** jamielennox|away is now known as jamielennox | 23:13 | |
| *** tiswanso has quit IRC | 23:13 | |
| *** amitgandhinz has quit IRC | 23:14 | |
| ianw | clarkb: unfortunately, but unsurprisingly, that patch doesn't apply cleanly. anyway, i'll report a bug since something is obviously going on | 23:16 |
| clarkb | ianw: sounds good, thanks | 23:17 |
| mordred | clarkb: so - this failure: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html ... is from two different neutron API calls for the same port | 23:17 |
| *** yamahata has joined #openstack-infra | 23:17 | |
| *** senk_ has quit IRC | 23:17 | |
| clarkb | mordred: either side of the assert is a different api call? | 23:18 |
| mordred | clarkb: the 'reference' dict is the one returned from create. the other dict is the one returned from get - getting via the id returned from create ... so the list of fixed_ips is just indeterminate order | 23:18 |
| clarkb | your fixed_ip order is different | 23:18 |
| *** gouthamr has joined #openstack-infra | 23:18 | |
| clarkb | mordred: yup | 23:18 |
| clarkb | mordred: you could return a set to express that more explicitly | 23:18 |
| openstackgerrit | Merged openstack-infra/system-config: Use ansible for puppet apply test https://review.openstack.org/371762 | 23:18 |
| clarkb | but that might not be backward compat | 23:18 |
| openstackgerrit | Merged openstack-infra/system-config: Stop managing puppet config in puppet https://review.openstack.org/371714 | 23:18 |
| openstackgerrit | Merged openstack-infra/system-config: Run ansible apply test in parallel https://review.openstack.org/371823 | 23:19 |
| clarkb | you could sort (ugh) or just make the assert smart enough to ignore order on that? | 23:19 |
| mordred | clarkb: yah - I think I'm going to have to just make a smart assert call | 23:19 |
| mordred | clarkb: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/tests/functional/test_port.py#n75 is the test | 23:19 |
| mordred | armax: if you're around ... ^^ is the lack of determinism in the fixed_ips list between the ipv4 and ipv6 address intentional/expected? | 23:20 |
| armax | mordred: I don’t believe determinism was ever intended | 23:20 |
| mordred | armax: mostly asking in case it's not and a bug filed would be useful - I can work around it on my side just fine | 23:21 |
| mordred | ok | 23:21 |
| mordred | so fixed_ips is a list, but the order of the ips in the list is not expected to be stable between api calls | 23:21 |
| mordred | necessarily | 23:21 |
| clarkb | mordred: also you should review the depends on for that one change you approved :) https://review.openstack.org/#/c/373219/ | 23:21 |
| mordred | clarkb: bah | 23:21 |
| clarkb | mordred: it would be neat to clean those things up and clearly specify them as sets vs lists if we ever get to make a new api | 23:21 |
| clarkb | (I don't think json can express that though) | 23:22 |
| armax | mordred: no, the ordering of the list of IPs is not something that’s embossed in the API contract | 23:22 |
| clarkb | armax: it is implicitly by returning a list | 23:22 |
| mordred | armax: ok. cool. thanks for clarifying! | 23:22 |
| armax | mordred: when have you started seeing lack of determinism? | 23:22 |
| clarkb | armax: lists are ordered... you should use unordred types if that is teh case | 23:22 |
| clarkb | but that ship may have sailed | 23:22 |
| mordred | armax: it just randomly happened recently - but it only happened on 2 out of 10 calls I made | 23:22 |
| mordred | 2 out of 10 test runs, that is | 23:23 |
| mordred | it had been stable before | 23:23 |
| mordred | or, had seemed that way | 23:23 |
| armax | clarkb: I am not sure I understand your claim | 23:23 |
| * olaph always wanted to be a major storm | 23:24 | |
| fungi | olaph: your wish is my command | 23:24 |
| armax | list are ordered how? | 23:24 |
| armax | lists | 23:24 |
| fungi | armax: python preserves order in its list datatype | 23:24 |
| fungi | armax: as opposed to, say, dict keys which are considered unordered | 23:25 |
| armax | right, but how data is packed in the list is where the lack of determinism comes from | 23:25 |
| clarkb | the definition of a list is that it is ordered | 23:25 |
| *** mixos has quit IRC | 23:25 | |
| clarkb | if you can't reliably order contents of a list you should use an unordered type | 23:25 |
| armax | right, I understand that | 23:26 |
| armax | one thing is saying once I create a list of [1, 2, 3] is always going to stay like that | 23:26 |
| clarkb | by definition it should | 23:26 |
| clarkb | because it is a list | 23:26 |
| armax | and that I agree | 23:26 |
| armax | the source of non determinsm is how that list is built | 23:27 |
| armax | once the list if built, it won’t change | 23:27 |
| mordred | armax: in this case, the first instance of the list is from the return of "POST /ports" ... the second instance of the list is from GET /ports/{port_id} | 23:27 |
| clarkb | right and I am saying that if you cannot garuntee that the list will be built deterministically that way, then that should be properly expressed by using an unordered type (I totally get that json makes this hard and python makes the other thing easy) | 23:28 |
| clarkb | beacuse from the end users perspective they don't care how you ahve implemented it | 23:28 |
| clarkb | to them that data structure is the same (or should be the same) in both places | 23:28 |
| mordred | I tihnk what I'm going to do shade-side is always sort the list of fixed_ips in a port by the port_id ... it's still arbitrary, but it'll be stable | 23:29 |
| armax | clarkb: as an API client I would not expect any order in the list of IPs that are returned to me | 23:29 |
| armax | but that’s a different point | 23:29 |
| armax | mordred: do you have an instance of the test failure? | 23:30 |
| mordred | armax: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html | 23:30 |
| mordred | armax: http://logs.openstack.org/63/377163/4/check/gate-shade-dsvm-functional-neutron/e621e3d/console.html#_2016-10-04_01_59_52_598273 more specifically | 23:30 |
| armax | yup I see | 23:30 |
| mordred | armax: here's teh test: http://git.openstack.org/cgit/openstack-infra/shade/tree/shade/tests/functional/test_port.py#n75 | 23:30 |
| mordred | anywho - I can deal with this in my world. if it's behaving right from your end, alls good | 23:32 |
| *** pahuang has joined #openstack-infra | 23:32 | |
| armax | mordred: I suppose we can address the lack of non-determinism | 23:32 |
| armax | on the neutron side | 23:32 |
| clarkb | armax: I guess from a client perspective if I get a datastructure from an API I would expect the same data structure back again if requested a second time with no changes made | 23:33 |
| armax | though, for the nature of the elements stored in the fixed_ips, I wonder what’s going to buy us | 23:33 |
| clarkb | armax: and changing the order of the list is not the same thing. However if you returned an unordered type that would be fine | 23:33 |
| armax | clarkb: yeah, it looks like mordred is saying that the change occurs between a POST and a GET | 23:34 |
| armax | I wonder what happens in susequent GETs | 23:34 |
| armax | I suspect the reason lies on how the create response is built vs the read one | 23:35 |
| armax | and I can think of which code change might have altered that | 23:35 |
| armax | clarkb, mordred: I bet that subsequent GETs will alway return the list in the same order | 23:37 |
| armax | I think that’s the culprit: https://review.openstack.org/#/c/373582 | 23:38 |
| armax | I can have kevinbenton look into restoring peace in the middle earth | 23:38 |
| anteaya | he is good at that | 23:39 |
| * clarkb waves https://review.openstack.org/#/c/373219/ in front of mordred one more time before going to attach shelves to walls | 23:39 | |
| anteaya | clarkb: you are offically out of the boxes? | 23:40 |
| clarkb | anteaya: not entirely | 23:40 |
| anteaya | close? | 23:41 |
| clarkb | ya pretty close | 23:42 |
| *** senk_ has joined #openstack-infra | 23:42 | |
| anteaya | yay! | 23:43 |
| *** mixos has joined #openstack-infra | 23:43 | |
| mordred | armax: you are like an encyclopedia of what's gone on in neutron | 23:44 |
| *** Swami has quit IRC | 23:44 | |
| armax | mordred: not sure about that, it’s more likely I was the one creating the damage :) | 23:44 |
| mordred | haha | 23:45 |
| armax | mordred: let me make it up to you: bug 1630410 | 23:49 |
| openstack | bug 1630410 in neutron "fixed_ips list out of order" [Undecided,New] https://launchpad.net/bugs/1630410 | 23:49 |
| *** thorst has joined #openstack-infra | 23:49 | |
| mordred | woot! armax is teh ossum | 23:50 |
| anteaya | kevinbenton shall fix this | 23:51 |
| anteaya | thank you kevinbenton | 23:51 |
| mordred | thank you kevinbenton | 23:51 |
| armax | mordred: having said that my suggestion is to try and avoid the implicit assumptions in relation to ordering | 23:56 |
| mordred | yup. also doing that | 23:57 |
| armax | mordred: we had other examples in the past were it was assumed that the IP allocation be deterministic and more specifically that IPs were handed out from the lowest IP to the highest | 23:57 |
| mordred | oh my. | 23:57 |
| armax | when in that there’s nothing in the API that says that’s the IP allocation policy | 23:57 |
| *** kaisers_ has quit IRC | 23:57 | |
| *** mixos has quit IRC | 23:57 | |
| armax | especially when pluggable IPAM backends are brought into the picture | 23:58 |
| armax | mordred: that’s not exactly what you experienced | 23:58 |
| armax | mordred: but you get the idea :) | 23:58 |
| mordred | armax: I'm actually going through the things we return at the moment and making sure we know what the contract is for all of the things (it's super fun, let me assure you) | 23:59 |
| mordred | yah | 23:59 |
| armax | mordred: I want to make sure I minimize your fun | 23:59 |
| armax | mordred: so do not hesitate to shout if we can make take some of that fun away | 23:59 |
| mordred | armax: yes, I appreciate that. as little fun as I can have the better | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!