| *** dingyichen has joined #openstack-infra | 00:00 | |
| *** sarob has joined #openstack-infra | 00:01 | |
| *** sarob has quit IRC | 00:05 | |
| *** tphummel has quit IRC | 00:06 | |
| ianw | pabelanger: firstly ... i can't seem to even get debug output out of the unit-tests | 00:06 |
|---|---|---|
| ianw | istr fixing that at some point | 00:06 |
| ianw | looks to be something in oslotest ... some debug logs for some scenarios come out, but not others | 00:19 |
| *** Julien-zte has joined #openstack-infra | 00:20 | |
| *** roxanaghe has joined #openstack-infra | 00:22 | |
| *** mriedem has joined #openstack-infra | 00:26 | |
| *** mriedem has left #openstack-infra | 00:26 | |
| *** roxanaghe has quit IRC | 00:27 | |
| *** fguillot has joined #openstack-infra | 00:28 | |
| ianw | http://paste.openstack.org/show/566929/ <- i'm not sure i want to go down the rabbit-hole of why that would capture debug logs from two tests, but none of the others | 00:31 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci: [DO NOT MERGE] Test a fake periodic job, newton 3 https://review.openstack.org/365802 | 00:36 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use OS_LOG_CAPTURE to get log output https://review.openstack.org/365807 | 00:40 |
| *** thorst has joined #openstack-infra | 00:40 | |
| *** thorst has quit IRC | 00:47 | |
| *** thorst has joined #openstack-infra | 00:47 | |
| *** thorst has quit IRC | 00:47 | |
| *** kaisers1 has joined #openstack-infra | 00:51 | |
| *** kaisers has quit IRC | 00:51 | |
| *** amitgandhinz has joined #openstack-infra | 00:52 | |
| *** kaisers has joined #openstack-infra | 00:52 | |
| *** kaisers2 has quit IRC | 00:52 | |
| *** markvoelker has joined #openstack-infra | 00:53 | |
| *** thorst has joined #openstack-infra | 00:54 | |
| *** amitgandhinz has quit IRC | 00:56 | |
| *** thorst has quit IRC | 01:05 | |
| pabelanger | ianw: feel free to push up any changes you need today to the patch. | 01:09 |
| ianw | pabelanger: i think the intersection of sets is what is actually wanted, just testing it out | 01:11 |
| *** vsaienko has quit IRC | 01:12 | |
| *** yamahata has joined #openstack-infra | 01:21 | |
| *** yanyanhu has joined #openstack-infra | 01:23 | |
| *** yuanying has joined #openstack-infra | 01:24 | |
| *** yuanying has quit IRC | 01:24 | |
| *** zhurong has joined #openstack-infra | 01:25 | |
| *** rbergeron has quit IRC | 01:38 | |
| *** rbergeron has joined #openstack-infra | 01:38 | |
| *** roxanaghe has joined #openstack-infra | 01:38 | |
| *** roxanaghe has quit IRC | 01:43 | |
| *** raddaoui has quit IRC | 01:43 | |
| *** raddaoui has joined #openstack-infra | 01:44 | |
| *** zshuo has joined #openstack-infra | 01:44 | |
| *** Genek has joined #openstack-infra | 01:46 | |
| *** baoli has joined #openstack-infra | 01:51 | |
| *** amitgandhinz has joined #openstack-infra | 01:52 | |
| *** baoli_ has joined #openstack-infra | 01:53 | |
| *** ijw has joined #openstack-infra | 01:55 | |
| *** akshai has joined #openstack-infra | 01:56 | |
| *** baoli has quit IRC | 01:56 | |
| *** amitgandhinz has quit IRC | 01:57 | |
| *** fguillot has quit IRC | 02:01 | |
| *** ijw has quit IRC | 02:01 | |
| *** fguillot has joined #openstack-infra | 02:02 | |
| *** fguillot has quit IRC | 02:02 | |
| *** baoli_ has quit IRC | 02:03 | |
| *** ijw has joined #openstack-infra | 02:03 | |
| *** ijw has quit IRC | 02:04 | |
| *** ijw has joined #openstack-infra | 02:04 | |
| openstackgerrit | guo yunxian proposed openstack/os-testr: Update home-page url https://review.openstack.org/356179 | 02:04 |
| *** thorst has joined #openstack-infra | 02:05 | |
| *** dkehn has joined #openstack-infra | 02:09 | |
| *** ijw has quit IRC | 02:10 | |
| *** thorst has quit IRC | 02:13 | |
| *** armax has quit IRC | 02:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Skip interfaces not in sys_interfaces https://review.openstack.org/365819 | 02:16 |
| ianw | pabelanger: ^ it's kind of different to yours, so I put it up separately to make comparing easier | 02:17 |
| *** rfolco has quit IRC | 02:18 | |
| tonyb | I assume the 'proposal' node is runnign trusty, but where is it managed so I can verify that? | 02:20 |
| *** akshai_ has joined #openstack-infra | 02:29 | |
| *** salv-orl_ has joined #openstack-infra | 02:29 | |
| *** caowei has joined #openstack-infra | 02:32 | |
| *** akshai has quit IRC | 02:32 | |
| *** salv-orlando has quit IRC | 02:32 | |
| *** rhallisey has quit IRC | 02:33 | |
| *** sflanigan is now known as sflanigan_afk | 02:36 | |
| *** ijw has joined #openstack-infra | 02:36 | |
| *** thorst has joined #openstack-infra | 02:40 | |
| *** ijw has quit IRC | 02:40 | |
| *** dkehn_ has joined #openstack-infra | 02:44 | |
| *** amotoki has joined #openstack-infra | 02:48 | |
| *** Genek has quit IRC | 02:48 | |
| *** Genek has joined #openstack-infra | 02:49 | |
| *** hongbin_ has quit IRC | 02:49 | |
| *** hongbin_ has joined #openstack-infra | 02:50 | |
| *** amotoki has quit IRC | 02:50 | |
| *** yuanying has joined #openstack-infra | 02:52 | |
| *** akshai_ has quit IRC | 02:52 | |
| *** markvoelker has quit IRC | 02:53 | |
| *** amitgandhinz has joined #openstack-infra | 02:53 | |
| *** yuanying has quit IRC | 02:55 | |
| *** yuanying has joined #openstack-infra | 02:57 | |
| *** amitgandhinz has quit IRC | 02:58 | |
| *** ijw has joined #openstack-infra | 03:00 | |
| *** yuanying has quit IRC | 03:00 | |
| *** yuanying has joined #openstack-infra | 03:00 | |
| *** sflanigan_afk is now known as sflanigan | 03:01 | |
| prometheanfire | tonyb: http://logs.openstack.org/65/365565/1/check/gate-requirements-integration-dsvm-ubuntu-xenial/3381ce2/console.html#_2016-09-05_10_55_44_920608 | 03:02 |
| prometheanfire | tonyb: looks like it's running xenial? | 03:03 |
| prometheanfire | at least by name | 03:03 |
| *** ijw has quit IRC | 03:05 | |
| *** amotoki has joined #openstack-infra | 03:07 | |
| *** roxanaghe has joined #openstack-infra | 03:09 | |
| *** senk has joined #openstack-infra | 03:10 | |
| tonyb | prometheanfire: right. I don't follow | 03:11 |
| tonyb | prometheanfire: We know the (requirements) gate is busted | 03:11 |
| prometheanfire | right | 03:12 |
| *** roxanaghe has quit IRC | 03:13 | |
| *** yuanying has quit IRC | 03:18 | |
| *** amotoki has quit IRC | 03:18 | |
| *** yuanying has joined #openstack-infra | 03:20 | |
| *** gildub has joined #openstack-infra | 03:33 | |
| *** gildub_ has joined #openstack-infra | 03:33 | |
| *** cody-somerville has joined #openstack-infra | 03:39 | |
| *** csomerville has joined #openstack-infra | 03:41 | |
| *** vikrant has joined #openstack-infra | 03:43 | |
| *** ijw has joined #openstack-infra | 03:44 | |
| *** cody-somerville has quit IRC | 03:45 | |
| *** ijw has quit IRC | 03:48 | |
| *** dkehn_ has quit IRC | 03:49 | |
| *** csomerville has quit IRC | 03:49 | |
| *** dkehn_ has joined #openstack-infra | 03:51 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm https://review.openstack.org/365840 | 03:51 |
| *** coolsvap_ has joined #openstack-infra | 03:52 | |
| *** amitgandhinz has joined #openstack-infra | 03:54 | |
| *** amotoki has joined #openstack-infra | 03:54 | |
| *** thorst has quit IRC | 03:58 | |
| *** thorst has joined #openstack-infra | 03:58 | |
| *** amitgandhinz has quit IRC | 03:59 | |
| *** ijw has joined #openstack-infra | 04:00 | |
| *** yuanying has quit IRC | 04:00 | |
| *** thorst has quit IRC | 04:00 | |
| *** hongbin_ has quit IRC | 04:01 | |
| *** Sukhdev has joined #openstack-infra | 04:01 | |
| *** yuanying has joined #openstack-infra | 04:02 | |
| *** thorst has joined #openstack-infra | 04:03 | |
| *** sdake has joined #openstack-infra | 04:05 | |
| *** watanabe_isao has joined #openstack-infra | 04:06 | |
| *** thorst has quit IRC | 04:08 | |
| *** ijw has quit IRC | 04:09 | |
| *** caowei has quit IRC | 04:10 | |
| *** thorst has joined #openstack-infra | 04:10 | |
| *** caowei has joined #openstack-infra | 04:13 | |
| *** thorst has quit IRC | 04:15 | |
| *** pgadiya has joined #openstack-infra | 04:17 | |
| *** roxanaghe has joined #openstack-infra | 04:25 | |
| *** sdake has quit IRC | 04:25 | |
| *** ijw has joined #openstack-infra | 04:30 | |
| *** roxanaghe has quit IRC | 04:30 | |
| *** abregman has joined #openstack-infra | 04:37 | |
| *** psachin has joined #openstack-infra | 04:39 | |
| *** links has joined #openstack-infra | 04:39 | |
| *** ijw has quit IRC | 04:40 | |
| *** ijw has joined #openstack-infra | 04:41 | |
| *** ijw has quit IRC | 04:42 | |
| *** harlowja_at_home has joined #openstack-infra | 04:42 | |
| *** harlowja_at_home has quit IRC | 04:47 | |
| *** sdake has joined #openstack-infra | 04:50 | |
| *** amitgandhinz has joined #openstack-infra | 04:55 | |
| AJaeger | jhesketh, ianw : Could you put this on your low-prio review queue, please? These are a few project-config changes of mine: https://review.openstack.org/365360 https://review.openstack.org/364782 https://review.openstack.org/365275 https://review.openstack.org/365074 https://review.openstack.org/364782 https://review.openstack.org/364896 | 04:55 |
| *** thorst has joined #openstack-infra | 04:55 | |
| AJaeger | tonyb, prometheanfire: we can revert clarkb's change for requirements if needed. We eventually need to make the change, Newton should be tested on Xenial. I prefer to give clarkb a chance to look at this later today - but if this blocks you, I'll approve a revert. | 04:57 |
| AJaeger | and then we'll do it again in a few days when we have all pieces together... | 04:57 |
| tonyb | AJaeger: we have a fix we can do in requirements that *shoudl* work | 04:57 |
| tonyb | AJaeger: I'd rather not revert because as you say we need to do this in the future anyway | 04:58 |
| tonyb | AJaeger: my change will let us work out what the next steps are | 04:58 |
| *** adriant has quit IRC | 04:59 | |
| *** thorst has quit IRC | 04:59 | |
| *** amitgandhinz has quit IRC | 04:59 | |
| ianw | tonyb: on proposal OS ... best way is probably too look at https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n945 | 05:00 |
| tonyb | ianw: Thanks. | 05:00 |
| *** sdake has quit IRC | 05:01 | |
| AJaeger | tonyb: I've seen the change, it looks fine. And I agree, I'd like us to move forward here as well. | 05:02 |
| tonyb | AJaeger: now to work out what I did wrong here: https://review.openstack.org/#/c/365840/1 | 05:04 |
| *** oanson has joined #openstack-infra | 05:05 | |
| AJaeger | tonyb: I just commented ;) | 05:05 |
| AJaeger | tonyb: the job is already there. You need to change zuul/layout to trigger that it runs on all branches. Right now we have a global branch regex that runs ubuntu-trusty only on liberty and mitaka, so add an override branches condition. | 05:06 |
| tonyb | AJaeger: Ahh okay that's what I was missing. Thanks. | 05:06 |
| AJaeger | ianw: thanks for reviewing! | 05:07 |
| *** thorst has joined #openstack-infra | 05:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix the Zun gate https://review.openstack.org/365803 | 05:08 |
| *** vikrant has quit IRC | 05:08 | |
| ianw | AJaeger: np :) i'm not sure if you've worked on log-copying in d-g before -- i wouldn't mind some eyes on https://review.openstack.org/#/c/364046/6 which is a refactor of the copy process. it's been very tangle for a while and it bugs me every time :) | 05:09 |
| *** vikrant has joined #openstack-infra | 05:10 | |
| *** sdake has joined #openstack-infra | 05:10 | |
| AJaeger | ianw: I haven't looked at d-g at all - but let me see what you've done... | 05:10 |
| *** itisha has quit IRC | 05:11 | |
| ianw | AJaeger: oh, no worries, don't worry too much | 05:11 |
| *** ilyashakhat_mobi has joined #openstack-infra | 05:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove some unused jobs from JJB https://review.openstack.org/365360 | 05:15 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job https://review.openstack.org/365862 | 05:15 |
| openstackgerrit | Merged openstack-infra/project-config: oslo.db: Add voting jobs to gate as well https://review.openstack.org/365275 | 05:15 |
| openstackgerrit | Merged openstack-infra/project-config: Remove octavia bandit job https://review.openstack.org/365074 | 05:15 |
| AJaeger | ianw: it's large ;( One thing I noticed: You remove archive_test_artifact - but don't seem to compress and chmod/chown these files anymore. Is that on purpose? | 05:16 |
| openstackgerrit | Merged openstack-infra/project-config: Remove experimental bandit job from anchor https://review.openstack.org/364896 | 05:17 |
| *** sdake has quit IRC | 05:17 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job https://review.openstack.org/365862 | 05:17 |
| *** rossella_s has quit IRC | 05:18 | |
| *** rossella_s has joined #openstack-infra | 05:19 | |
| ianw | AJaeger: good point, next review i'll compress it. the permissions is fixed up with recursive chown's at the end (so is a lot of zipping, but not for .subunit files) | 05:19 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: change post_test_hook location for osc functional tests https://review.openstack.org/365865 | 05:20 |
| AJaeger | ianw: ah... | 05:20 |
| *** jaosorior has joined #openstack-infra | 05:21 | |
| openstackgerrit | Zhenguo Niu proposed openstack-infra/project-config: Add #openstack-nimble to the IRC channels https://review.openstack.org/365866 | 05:22 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: change post_test_hook location for osc functional tests https://review.openstack.org/365865 | 05:24 |
| *** thorst has quit IRC | 05:25 | |
| *** ijw has joined #openstack-infra | 05:29 | |
| *** thorst has joined #openstack-infra | 05:29 | |
| *** florianf has joined #openstack-infra | 05:30 | |
| *** zzzeek has quit IRC | 05:30 | |
| *** ijw_ has joined #openstack-infra | 05:31 | |
| *** ijw has quit IRC | 05:32 | |
| *** thorst has quit IRC | 05:33 | |
| *** gildub has quit IRC | 05:34 | |
| *** senk has quit IRC | 05:34 | |
| *** gildub_ has quit IRC | 05:35 | |
| *** gildub has joined #openstack-infra | 05:35 | |
| *** ijw_ has quit IRC | 05:35 | |
| *** sarob has joined #openstack-infra | 05:37 | |
| *** sarob has quit IRC | 05:37 | |
| *** ijw has joined #openstack-infra | 05:38 | |
| *** ijw has quit IRC | 05:40 | |
| *** sandanar has joined #openstack-infra | 05:41 | |
| zhenguo | AJaeger: hi, regarding https://review.openstack.org/#/c/365866/ , you mean we can't add new channels now? | 05:41 |
| *** ilyashakhat_mobi has quit IRC | 05:43 | |
| *** ijw has joined #openstack-infra | 05:43 | |
| *** snarwade has quit IRC | 05:44 | |
| *** snarwade has joined #openstack-infra | 05:44 | |
| openstackgerrit | Zhenguo Niu proposed openstack-infra/project-config: Add #openstack-nimble to the IRC channels https://review.openstack.org/365866 | 05:45 |
| *** e0ne has joined #openstack-infra | 05:46 | |
| *** ijw has quit IRC | 05:46 | |
| *** e0ne has quit IRC | 05:47 | |
| *** ijw has joined #openstack-infra | 05:47 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 05:48 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm https://review.openstack.org/365840 | 05:52 |
| *** amitgandhinz has joined #openstack-infra | 05:56 | |
| *** abregman has quit IRC | 05:56 | |
| *** ijw has quit IRC | 05:57 | |
| *** abregman has joined #openstack-infra | 05:57 | |
| *** ijw has joined #openstack-infra | 05:58 | |
| *** amitgandhinz has quit IRC | 06:00 | |
| *** ijw has quit IRC | 06:01 | |
| *** rcarrillocruz has quit IRC | 06:02 | |
| *** ijw has joined #openstack-infra | 06:02 | |
| *** esikache1 has joined #openstack-infra | 06:02 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 06:02 | |
| *** ijw has quit IRC | 06:03 | |
| *** esikache1 has quit IRC | 06:05 | |
| *** esikachev has joined #openstack-infra | 06:05 | |
| *** martinkopec has joined #openstack-infra | 06:06 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Fix senlin integration test https://review.openstack.org/365875 | 06:07 |
| *** ijw has joined #openstack-infra | 06:10 | |
| *** binbincong_ has quit IRC | 06:12 | |
| *** roxanaghe has joined #openstack-infra | 06:13 | |
| *** senk has joined #openstack-infra | 06:16 | |
| frickler | seems gate-tempest-dsvm-neutron-identity-v3-only-full-nv has been broken since the morning of 2016-08-04, has anyone looked into this? if I run the reproduce.sh locally, I do not get the error, so maybe it is related to the trusty images | 06:18 |
| *** roxanaghe has quit IRC | 06:18 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ironic: Copy all deployment logs https://review.openstack.org/362867 | 06:18 |
| *** ilyashakhat_mobi has quit IRC | 06:19 | |
| clarkb | frickler: have an example log? how does it fail? | 06:22 |
| *** esikachev has quit IRC | 06:24 | |
| *** ijw has quit IRC | 06:25 | |
| *** dimtruck is now known as zz_dimtruck | 06:27 | |
| *** ihrachys has joined #openstack-infra | 06:29 | |
| *** claudiub|2 has joined #openstack-infra | 06:29 | |
| *** aeng_ has quit IRC | 06:29 | |
| *** _nadya_ has joined #openstack-infra | 06:30 | |
| *** claudiub has joined #openstack-infra | 06:31 | |
| *** claudiub|2 has quit IRC | 06:34 | |
| *** binbincong_ has joined #openstack-infra | 06:36 | |
| AJaeger | zhenguo: correct - see the failure you got. | 06:36 |
| zhenguo | Ajaeger: ok, I have removed the gerritbot changes:) | 06:38 |
| AJaeger | zhenguo: we wait for somebody to set up a second gerritbot in our infrastructure before we can accept more. | 06:39 |
| zhenguo | AJaeger: do we have a plan for that or is there somebody working on that? | 06:40 |
| *** berendt has joined #openstack-infra | 06:40 | |
| AJaeger | zhenguo: somebody volunteered but haven't seen followup on it ;( | 06:43 |
| AJaeger | but if anybody else wants to do it... | 06:43 |
| *** Hal1 has joined #openstack-infra | 06:44 | |
| *** andreas_s has joined #openstack-infra | 06:45 | |
| *** vsaienko1 has joined #openstack-infra | 06:45 | |
| zhenguo | AJaeger: maybe I can try, where can I get some instructions for gerritbot setup? | 06:45 |
| AJaeger | zhenguo: best talk with fungi later today, he might now what happened with the other volunteer | 06:46 |
| AJaeger | zhenguo: there's some part in system-config | 06:47 |
| AJaeger | and there'S a puppet-gerritbot module. | 06:47 |
| AJaeger | We need to run two instances of gerritbot. | 06:47 |
| AJaeger | zhenguo: there's already https://review.openstack.org/#/c/355588/ - reviewing that one might help... | 06:48 |
| *** zz_dimtruck is now known as dimtruck | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:48 | |
| *** gildub has quit IRC | 06:49 | |
| zhenguo | AJaeger: ok, will look into that, thanks! | 06:50 |
| *** gildub has joined #openstack-infra | 06:50 | |
| *** gildub has quit IRC | 06:51 | |
| *** gildub has joined #openstack-infra | 06:52 | |
| *** markvoelker has joined #openstack-infra | 06:54 | |
| vsaienko1 | AJaeger: could you please help to merge https://review.openstack.org/#/c/365706/ one line patch, to update ironic-multinode job timeout. | 06:54 |
| *** dkehn_ has quit IRC | 06:56 | |
| *** amitgandhinz has joined #openstack-infra | 06:56 | |
| AJaeger | vsaienko1: please wait for anteaya to reconsider her -1 first. | 06:56 |
| *** dimtruck is now known as zz_dimtruck | 06:58 | |
| frickler | clarkb: one of the last successful runs I found: http://logs.openstack.org/04/350804/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/96f3381/ | 06:58 |
| frickler | clarkb: failing: http://logs.openstack.org/88/350888/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/b92d026/ | 06:58 |
| *** markvoelker has quit IRC | 06:59 | |
| frickler | clarkb: the failure is in the glance-api getting an auth error while trying to find the swift endpoint | 06:59 |
| *** tesseract- has joined #openstack-infra | 07:00 | |
| *** ijw has joined #openstack-infra | 07:00 | |
| frickler | clarkb: like here http://logs.openstack.org/67/354367/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-nv/8cf4a21/logs/screen-g-api.txt.gz#_2016-08-11_21_45_47_229 | 07:00 |
| *** zz_dimtruck is now known as dimtruck | 07:00 | |
| *** ijw has quit IRC | 07:00 | |
| *** ijw has joined #openstack-infra | 07:00 | |
| *** dkehn_ has joined #openstack-infra | 07:01 | |
| *** amitgandhinz has quit IRC | 07:01 | |
| *** esikachev has joined #openstack-infra | 07:01 | |
| *** gildub has quit IRC | 07:02 | |
| *** gildub has joined #openstack-infra | 07:02 | |
| yolanda | good morning | 07:02 |
| openstackgerrit | Merged openstack-infra/glean: Add check to skip bridge interfaces https://review.openstack.org/359982 | 07:04 |
| *** Julien-zte has quit IRC | 07:04 | |
| vsaienko1 | AJaeger: sure, lets wait for anteaya decision, thanks! | 07:05 |
| *** Sukhdev has quit IRC | 07:06 | |
| *** Sukhdev has joined #openstack-infra | 07:06 | |
| AJaeger | morning, yolanda ! | 07:07 |
| AJaeger | clarkb: I added you also to https://review.openstack.org/#/c/365840 - another requirements xenial change. PLease review your ;) Tuesday morning | 07:09 |
| *** matrohon has joined #openstack-infra | 07:09 | |
| *** dimtruck is now known as zz_dimtruck | 07:10 | |
| *** Sukhdev has quit IRC | 07:11 | |
| *** esikachev has quit IRC | 07:11 | |
| *** ihrachys has quit IRC | 07:12 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output https://review.openstack.org/364045 | 07:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run https://review.openstack.org/364560 | 07:12 |
| *** ihrachys has joined #openstack-infra | 07:12 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Refactor devstack log copying https://review.openstack.org/364046 | 07:12 |
| *** rvba has joined #openstack-infra | 07:14 | |
| *** rvba has quit IRC | 07:14 | |
| *** rvba has joined #openstack-infra | 07:14 | |
| *** slaweq__ has joined #openstack-infra | 07:14 | |
| *** ifarkas_afk is now known as ifarkas | 07:16 | |
| *** slaweq_ has quit IRC | 07:17 | |
| *** ihrachys has quit IRC | 07:23 | |
| *** drifterza has joined #openstack-infra | 07:26 | |
| *** binbincong_ has quit IRC | 07:29 | |
| *** jpich has joined #openstack-infra | 07:30 | |
| *** Na3iL has joined #openstack-infra | 07:32 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm https://review.openstack.org/365840 | 07:33 |
| *** hashar has joined #openstack-infra | 07:33 | |
| *** ijw has quit IRC | 07:35 | |
| *** esikachev has joined #openstack-infra | 07:35 | |
| *** abregman has quit IRC | 07:35 | |
| *** ijw has joined #openstack-infra | 07:36 | |
| *** isaacb has joined #openstack-infra | 07:36 | |
| *** gildub has quit IRC | 07:38 | |
| *** rcarrillocruz has joined #openstack-infra | 07:39 | |
| *** matthewbodkin has joined #openstack-infra | 07:41 | |
| *** matthewbodkin has quit IRC | 07:41 | |
| *** matthewbodkin has joined #openstack-infra | 07:41 | |
| *** mptacekx has joined #openstack-infra | 07:43 | |
| *** yaume has joined #openstack-infra | 07:45 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 07:49 |
| *** Ravikiran_K has joined #openstack-infra | 07:50 | |
| *** abregman has joined #openstack-infra | 07:51 | |
| *** binbincong_ has joined #openstack-infra | 07:53 | |
| *** daemontool has joined #openstack-infra | 07:54 | |
| *** Julien-zte has joined #openstack-infra | 07:56 | |
| *** amitgandhinz has joined #openstack-infra | 07:57 | |
| *** mikelk has joined #openstack-infra | 07:59 | |
| HeOS | Hello, infra team! Have you read my letter in openstack-infra ML about adding members to the following group? https://review.openstack.org/#/admin/groups/1555,members | 07:59 |
| openstackgerrit | afazekas proposed openstack/os-testr: Make the listbuilder the default strategy https://review.openstack.org/349451 | 07:59 |
| HeOS | Could you please add them there? | 08:00 |
| *** roxanaghe has joined #openstack-infra | 08:01 | |
| openstackgerrit | afazekas proposed openstack/os-testr: Allow to specifiy black/white list at the same time https://review.openstack.org/349477 | 08:01 |
| *** amitgandhinz has quit IRC | 08:02 | |
| *** abregman has quit IRC | 08:04 | |
| openstackgerrit | afazekas proposed openstack/os-testr: Add --black-regex/-B option https://review.openstack.org/349505 | 08:05 |
| *** abregman has joined #openstack-infra | 08:05 | |
| *** roxanaghe has quit IRC | 08:05 | |
| openstackgerrit | afazekas proposed openstack/os-testr: Use dash instead of underscore in args https://review.openstack.org/349972 | 08:07 |
| *** jistr is now known as jistr|dentist | 08:07 | |
| *** asettle has joined #openstack-infra | 08:11 | |
| *** sflanigan has quit IRC | 08:11 | |
| openstackgerrit | shizhihui proposed openstack-infra/project-config: Make py35 voting for Horizon https://review.openstack.org/359123 | 08:12 |
| frickler | HeOS: yesterday was a holiday for most infra-people, I assume they should be able to handle this later today | 08:12 |
| HeOS | frickler, Ohhh, yes, I forgot about that. Thank you. :) | 08:12 |
| *** tsufiev_ is now known as tsufiev | 08:13 | |
| *** derekh has joined #openstack-infra | 08:15 | |
| *** lucas-dinner is now known as lucasagomes | 08:16 | |
| openstackgerrit | afazekas proposed openstack/os-testr: Make the listbuilder the default strategy https://review.openstack.org/349451 | 08:16 |
| *** ijw has quit IRC | 08:17 | |
| *** isaacb has quit IRC | 08:19 | |
| *** markvoelker has joined #openstack-infra | 08:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/365933 | 08:22 |
| *** dizquierdo has joined #openstack-infra | 08:23 | |
| *** ihrachys has joined #openstack-infra | 08:23 | |
| *** markvoelker has quit IRC | 08:24 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job https://review.openstack.org/365862 | 08:24 |
| *** gongysh has quit IRC | 08:25 | |
| *** gongysh has joined #openstack-infra | 08:26 | |
| *** Jeffrey4l has quit IRC | 08:28 | |
| *** salv-orlando has joined #openstack-infra | 08:29 | |
| *** salv-orl_ has quit IRC | 08:32 | |
| openstackgerrit | afazekas proposed openstack/os-testr: Allow to specifiy black/white list at the same time https://review.openstack.org/349477 | 08:33 |
| openstackgerrit | afazekas proposed openstack/os-testr: Add --black-regex/-B option https://review.openstack.org/349505 | 08:33 |
| openstackgerrit | afazekas proposed openstack/os-testr: Use dash instead of underscore in args https://review.openstack.org/349972 | 08:34 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job https://review.openstack.org/365862 | 08:35 |
| *** dingyichen has quit IRC | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add #openstack-nimble to the IRC channels https://review.openstack.org/365866 | 08:37 |
| *** watanabe_isao has quit IRC | 08:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix senlin integration test https://review.openstack.org/365875 | 08:39 |
| *** ijw has joined #openstack-infra | 08:39 | |
| openstackgerrit | Merged openstack-infra/project-config: npm-docs job is now voting https://review.openstack.org/365033 | 08:43 |
| openstackgerrit | Merged openstack-infra/project-config: murano-pkg-check: add coverage job https://review.openstack.org/364907 | 08:43 |
| openstackgerrit | Merged openstack-infra/project-config: Use python-db-jobs for Glare https://review.openstack.org/364370 | 08:43 |
| *** MmikeM is now known as Mmike | 08:45 | |
| *** abregman has quit IRC | 08:46 | |
| zynzel | fungi: can you take a look at http://lists.openstack.org/pipermail/openstack-infra/2016-September/004707.html ? :) | 08:47 |
| *** Mmike has quit IRC | 08:48 | |
| *** Mmike has joined #openstack-infra | 08:49 | |
| *** Mmike has joined #openstack-infra | 08:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish console logs for python-muranoclient job template https://review.openstack.org/365648 | 08:50 |
| openstackgerrit | Merged openstack-infra/project-config: Add python-storyboardclient https://review.openstack.org/365568 | 08:50 |
| *** isaacb has joined #openstack-infra | 08:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote OSA os_keystone nv jobs to voting https://review.openstack.org/365557 | 08:52 |
| *** m3m0 has left #openstack-infra | 08:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Change tripleo ha2 job types back too ha https://review.openstack.org/363409 | 08:58 |
| *** amitgandhinz has joined #openstack-infra | 08:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Add service-control charm interface https://review.openstack.org/363595 | 08:58 |
| *** abregman has joined #openstack-infra | 08:59 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove the ha2 JOBTYPE https://review.openstack.org/363411 | 09:00 |
| *** zz_dimtruck is now known as dimtruck | 09:01 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 09:01 |
| *** florianf has quit IRC | 09:02 | |
| *** amitgandhinz has quit IRC | 09:03 | |
| *** florianf has joined #openstack-infra | 09:04 | |
| *** ijw has quit IRC | 09:07 | |
| amrith | hiya infra, I'm trying to debug an issue with the trove gate. we've had no success on trove or trove-integration and can't reproduce this on our local machines (where things seem to pass). I've pushed up a commit https://review.openstack.org/#/c/365960/ and it is now running on 2001:4800:1ae1:18:f816:3eff:fe51:7092. Would you please add https://gist.github.com/amrith/ad6e47f805d7f83bdf630ad84ff0141a (my public key) to | 09:10 |
| amrith | allow access to that machine. | 09:10 |
| amrith | AJaeger yolanda ^^ g'morning | 09:10 |
| *** dimtruck is now known as zz_dimtruck | 09:10 | |
| *** nmagnezi has joined #openstack-infra | 09:13 | |
| *** ijw has joined #openstack-infra | 09:18 | |
| *** gongysh has quit IRC | 09:19 | |
| *** sambetts|afk is now known as sambetts | 09:21 | |
| *** gongysh has joined #openstack-infra | 09:21 | |
| *** r-mibu has quit IRC | 09:23 | |
| *** jistr|dentist is now known as jistr | 09:23 | |
| *** drybjed has quit IRC | 09:24 | |
| *** aderyugin has joined #openstack-infra | 09:26 | |
| *** gongysh has quit IRC | 09:26 | |
| *** acoles_ is now known as acoles | 09:28 | |
| *** daemontool has quit IRC | 09:29 | |
| *** drybjed has joined #openstack-infra | 09:30 | |
| *** r-mibu has joined #openstack-infra | 09:33 | |
| *** zshuo has quit IRC | 09:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Remove tracing since ansible seems to be working https://review.openstack.org/178398 | 09:40 |
| *** shardy_afk has joined #openstack-infra | 09:43 | |
| *** shardy has joined #openstack-infra | 09:44 | |
| *** sdake has joined #openstack-infra | 09:45 | |
| *** Genek has quit IRC | 09:46 | |
| amrith | hiya ... anybody home? | 09:49 |
| *** roxanaghe has joined #openstack-infra | 09:49 | |
| *** roxanaghe has quit IRC | 09:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/365933 | 09:55 |
| vsaienko1 | I'm wondered if anteaya around? | 09:58 |
| *** zhurong has quit IRC | 09:58 | |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| *** vsaienko has joined #openstack-infra | 09:59 | |
| *** amitgandhinz has joined #openstack-infra | 09:59 | |
| *** yanyanhu has quit IRC | 10:01 | |
| *** pgadiya has quit IRC | 10:02 | |
| *** amitgandhinz has quit IRC | 10:03 | |
| frickler | is there a way to see a list of results for -nv jobs? looks like http://status.openstack.org/openstack-health only has statistics for voting jobs | 10:03 |
| *** drybjed has quit IRC | 10:06 | |
| *** zz_dimtruck is now known as dimtruck | 10:06 | |
| *** Sukhdev has joined #openstack-infra | 10:07 | |
| *** vsaienko has quit IRC | 10:08 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Do not build any ramdisks in Ironic grenade jobs https://review.openstack.org/366028 | 10:08 |
| dtantsur | sambetts, jroll, aarefiev ^^^ | 10:08 |
| *** sshnaidm|afk is now known as sshnaidm | 10:10 | |
| dtantsur | folks, please take a look ^^^ this will make our grenade job feel a bit better hopefully | 10:10 |
| *** pgadiya has joined #openstack-infra | 10:11 | |
| dtantsur | AJaeger, yolanda, sorry for personal ping, trying to unbreak the gate before the RC ^^^ | 10:12 |
| *** Julien-zte has quit IRC | 10:13 | |
| *** Sukhdev has quit IRC | 10:13 | |
| *** pgadiya has quit IRC | 10:15 | |
| *** e0ne has joined #openstack-infra | 10:15 | |
| *** pgadiya has joined #openstack-infra | 10:17 | |
| *** drybjed has joined #openstack-infra | 10:20 | |
| *** ijw has quit IRC | 10:20 | |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module https://review.openstack.org/366033 | 10:22 |
| *** Julien-zte has joined #openstack-infra | 10:23 | |
| *** caowei has quit IRC | 10:23 | |
| vsaienko1 | infra core: recently new project networking-cumulus was added: https://review.openstack.org/#/c/354784/. Could you please add me to project-core group so I can grant access for other users. Thanks in advance. | 10:24 |
| *** asettle has quit IRC | 10:26 | |
| *** kaisers2 has joined #openstack-infra | 10:29 | |
| *** kaisers1 has quit IRC | 10:29 | |
| *** kaisers1 has joined #openstack-infra | 10:30 | |
| *** kaisers has quit IRC | 10:30 | |
| *** ramishra has quit IRC | 10:30 | |
| *** ramishra has joined #openstack-infra | 10:32 | |
| *** sdake has quit IRC | 10:33 | |
| *** dizquierdo is now known as dizquierdo_afk | 10:34 | |
| *** pgadiya has quit IRC | 10:35 | |
| *** askb has quit IRC | 10:40 | |
| *** abregman is now known as abregman|brb | 10:41 | |
| *** pblaho has joined #openstack-infra | 10:43 | |
| *** HenryG_ is now known as HenryG | 10:46 | |
| *** baoli has joined #openstack-infra | 10:46 | |
| *** Hal1 has quit IRC | 10:48 | |
| *** zzzeek has joined #openstack-infra | 10:51 | |
| *** pgadiya has joined #openstack-infra | 10:51 | |
| *** abregman|brb is now known as abregman | 10:52 | |
| *** dizquierdo_afk is now known as dizquierdo | 10:52 | |
| *** zzzeek has quit IRC | 10:56 | |
| *** zzzeek has joined #openstack-infra | 10:57 | |
| *** baoli has quit IRC | 10:57 | |
| *** kaisers2 has quit IRC | 10:58 | |
| *** kaisers1 has quit IRC | 10:59 | |
| *** kaisers has joined #openstack-infra | 10:59 | |
| *** kaisers1 has joined #openstack-infra | 10:59 | |
| *** baoli has joined #openstack-infra | 10:59 | |
| *** amitgandhinz has joined #openstack-infra | 10:59 | |
| *** dprince has joined #openstack-infra | 11:00 | |
| *** baoli has quit IRC | 11:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not build any ramdisks in Ironic grenade jobs https://review.openstack.org/366028 | 11:03 |
| *** amitgandhinz has quit IRC | 11:04 | |
| *** Jeffrey4l has joined #openstack-infra | 11:05 | |
| *** shardy_afk has quit IRC | 11:11 | |
| *** rhallisey has joined #openstack-infra | 11:13 | |
| *** Genek has joined #openstack-infra | 11:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Move the clouds_layouts.yaml file https://review.openstack.org/364871 | 11:16 |
| *** ldnunes has joined #openstack-infra | 11:16 | |
| *** lucasagomes is now known as lucas-hungry | 11:22 | |
| *** rtheis has joined #openstack-infra | 11:22 | |
| *** dkehn_ has quit IRC | 11:23 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH https://review.openstack.org/359909 | 11:24 |
| *** e0ne has quit IRC | 11:25 | |
| *** ijw has joined #openstack-infra | 11:27 | |
| *** Hal has joined #openstack-infra | 11:29 | |
| *** amotoki_ has joined #openstack-infra | 11:29 | |
| *** ijw has quit IRC | 11:29 | |
| *** ijw has joined #openstack-infra | 11:30 | |
| *** amotoki has quit IRC | 11:31 | |
| *** dizquierdo has quit IRC | 11:32 | |
| *** gildub has joined #openstack-infra | 11:35 | |
| *** dkehn_ has joined #openstack-infra | 11:35 | |
| *** asettle has joined #openstack-infra | 11:37 | |
| *** binbincong_ has quit IRC | 11:37 | |
| *** roxanaghe has joined #openstack-infra | 11:37 | |
| *** sdague has joined #openstack-infra | 11:39 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update infracloud documentation to remove old geo information https://review.openstack.org/365018 | 11:41 |
| *** jkilpatr has joined #openstack-infra | 11:41 | |
| *** kaisers has quit IRC | 11:41 | |
| *** roxanaghe has quit IRC | 11:42 | |
| *** kaisers has joined #openstack-infra | 11:42 | |
| *** kaisers1 has quit IRC | 11:43 | |
| *** kaisers1 has joined #openstack-infra | 11:43 | |
| *** dprince has quit IRC | 11:43 | |
| openstackgerrit | Mathieu Mitchell proposed openstack-infra/system-config: Introduce TinyCoreLinux mirror https://review.openstack.org/325981 | 11:43 |
| *** dprince has joined #openstack-infra | 11:43 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Retry stack-delete in case of failure https://review.openstack.org/350640 | 11:44 |
| *** kaisers2 has joined #openstack-infra | 11:46 | |
| *** kaisers3 has joined #openstack-infra | 11:47 | |
| *** kaisers1 has quit IRC | 11:48 | |
| *** kaisers has quit IRC | 11:49 | |
| *** salv-orl_ has joined #openstack-infra | 11:52 | |
| *** salv-orlando has quit IRC | 11:52 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 11:52 |
| *** asettle has quit IRC | 11:52 | |
| *** abregman has quit IRC | 11:53 | |
| *** abregman has joined #openstack-infra | 11:54 | |
| *** kaisers3 has quit IRC | 11:55 | |
| *** kaisers has joined #openstack-infra | 11:55 | |
| *** asettle has joined #openstack-infra | 11:58 | |
| *** amitgandhinz has joined #openstack-infra | 12:00 | |
| *** daemontool has joined #openstack-infra | 12:04 | |
| *** ijw has quit IRC | 12:05 | |
| *** amitgandhinz has quit IRC | 12:05 | |
| *** trown|outtypewww is now known as trown | 12:06 | |
| *** rfolco has joined #openstack-infra | 12:09 | |
| *** pgadiya has quit IRC | 12:10 | |
| *** daemontool_ has joined #openstack-infra | 12:10 | |
| *** zhurong has joined #openstack-infra | 12:10 | |
| AJaeger | amrith: Sorry, I can't help with holding notes - and seems indeed as not many are around yet | 12:11 |
| *** daemontool has quit IRC | 12:12 | |
| *** edmondsw has joined #openstack-infra | 12:13 | |
| *** gordc has joined #openstack-infra | 12:15 | |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:16 |
| *** gildub has quit IRC | 12:18 | |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:20 |
| *** oanson has quit IRC | 12:22 | |
| *** kgiusti has joined #openstack-infra | 12:23 | |
| *** Genek has quit IRC | 12:24 | |
| *** psilvad has joined #openstack-infra | 12:24 | |
| *** woodster_ has joined #openstack-infra | 12:26 | |
| *** sandanar_ has joined #openstack-infra | 12:26 | |
| *** clenimar has quit IRC | 12:26 | |
| *** lucas-hungry is now known as lucasagomes | 12:27 | |
| amrith | AJaeger, figured out what the problem was (blind squirrel, acorn ...). Fix is in review now. Thanks! | 12:28 |
| *** pgadiya has joined #openstack-infra | 12:28 | |
| AJaeger | fungi, https://review.openstack.org/#/c/364723/ is in "merge conflict" - but if I try rebasing, it tells me that it's already up-to-date. Any bug in gerrit? Anything that needs doing there - or should we wait for next translation push? | 12:29 |
| *** sandanar has quit IRC | 12:29 | |
| AJaeger | amrith: glad to hear | 12:29 |
| *** thorst has joined #openstack-infra | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** Julien-zte has quit IRC | 12:30 | |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:31 |
| *** clenimar has joined #openstack-infra | 12:31 | |
| *** sandanar_ has quit IRC | 12:31 | |
| openstackgerrit | Joe Hakim Rahme proposed openstack-infra/project-config: Adds a new project: nova-tempest-staging https://review.openstack.org/366091 | 12:36 |
| *** mdrabe has joined #openstack-infra | 12:37 | |
| *** tongli has joined #openstack-infra | 12:40 | |
| *** dmsimard|pto is now known as dmsimard | 12:43 | |
| openstackgerrit | Sean Dague proposed openstack-infra/os-loganalyze: add placement-api to list of understood logs https://review.openstack.org/366096 | 12:43 |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:44 |
| *** baoli has joined #openstack-infra | 12:44 | |
| *** baoli_ has joined #openstack-infra | 12:45 | |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:46 |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:47 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 12:47 |
| *** baoli has quit IRC | 12:48 | |
| *** furlongm_ is now known as furlongm | 12:48 | |
| *** jaosorior has quit IRC | 12:48 | |
| *** jaosorior has joined #openstack-infra | 12:49 | |
| openstackgerrit | Yaroslav Isakov proposed openstack-infra/project-config: Added networking-avaya project https://review.openstack.org/366087 | 12:49 |
| *** mmedvede has quit IRC | 12:49 | |
| *** jaosorior has quit IRC | 12:49 | |
| *** pcaruana has quit IRC | 12:50 | |
| *** jaosorior has joined #openstack-infra | 12:51 | |
| *** caowei has joined #openstack-infra | 12:53 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Save console logs from all vms to files https://review.openstack.org/365369 | 12:54 |
| *** dtardivel has joined #openstack-infra | 12:55 | |
| *** mhickey has joined #openstack-infra | 12:55 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 12:55 |
| *** psilvad has quit IRC | 12:58 | |
| *** amitgandhinz has joined #openstack-infra | 13:01 | |
| *** amitgandhinz has quit IRC | 13:05 | |
| *** Goneri has joined #openstack-infra | 13:07 | |
| *** mbacchi_ is now known as mbacchi | 13:09 | |
| *** mmedvede has joined #openstack-infra | 13:11 | |
| openstackgerrit | Waldemar Znoinski proposed openstack/diskimage-builder: fix systemd resource deadlock https://review.openstack.org/365797 | 13:11 |
| *** nmagnezi_ has joined #openstack-infra | 13:12 | |
| *** pgadiya has quit IRC | 13:13 | |
| *** moravec has quit IRC | 13:14 | |
| *** nmagnezi has quit IRC | 13:14 | |
| *** matt-borland has joined #openstack-infra | 13:17 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings: Fix count_slot_usage.py meeting links https://review.openstack.org/366123 | 13:17 |
| *** e0ne has joined #openstack-infra | 13:17 | |
| *** akshai has joined #openstack-infra | 13:18 | |
| *** rossella_s has quit IRC | 13:18 | |
| *** rossella_s has joined #openstack-infra | 13:19 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/irc-meetings: Create Architecture Working Group meeting https://review.openstack.org/366127 | 13:19 |
| openstackgerrit | Merged openstack-dev/hacking: Fix issues detected by pycodestyle https://review.openstack.org/359487 | 13:20 |
| openstackgerrit | Merged openstack-dev/hacking: Add documentation about off-by-default options https://review.openstack.org/359427 | 13:20 |
| *** moravec has joined #openstack-infra | 13:24 | |
| *** smarcet has joined #openstack-infra | 13:25 | |
| *** eharney has joined #openstack-infra | 13:25 | |
| smarcet | fungi: morning, i am trying to create a signed tag here https://github.com/openstack-infra/openstackid-resources.git | 13:25 |
| smarcet | but its said that i dont have the rites | 13:25 |
| *** roxanaghe has joined #openstack-infra | 13:25 | |
| smarcet | fungi: from what i am seeing here https://review.openstack.org/Documentation/error-prohibited-by-gerrit.html | 13:26 |
| *** zhurong has quit IRC | 13:26 | |
| *** ddieterly has joined #openstack-infra | 13:27 | |
| *** dizquierdo has joined #openstack-infra | 13:28 | |
| *** _degorenko|afk is now known as degorenko | 13:28 | |
| *** roxanaghe has quit IRC | 13:30 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 13:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/360445 | 13:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use MySQL create user statement to silence warning https://review.openstack.org/366133 | 13:31 |
| *** drawsmcgraw has left #openstack-infra | 13:31 | |
| *** vikrant has quit IRC | 13:32 | |
| AJaeger | smarcet: check the ACLs... | 13:33 |
| *** drawsmcgraw has joined #openstack-infra | 13:33 | |
| pabelanger | morning | 13:33 |
| AJaeger | morning, pabelanger | 13:33 |
| AJaeger | smarcet: https://review.openstack.org/#/admin/projects/openstack-infra/openstackid-resources,access | 13:33 |
| *** mriedem has joined #openstack-infra | 13:34 | |
| AJaeger | And release team is only infra - so fungi can do it for you. | 13:34 |
| *** amitgandhinz has joined #openstack-infra | 13:34 | |
| smarcet | AJaegar: well i have those rites on openstackid | 13:35 |
| *** _nadya_ has quit IRC | 13:35 | |
| smarcet | i could do the tags | 13:35 |
| smarcet | on openstackid | 13:35 |
| smarcet | and need the same on openstackid-resources | 13:35 |
| AJaeger | smarcet: then fungi needs to empower you - and add to that group | 13:35 |
| *** markvoelker has joined #openstack-infra | 13:36 | |
| smarcet | ok AJaeger thx for the info :) | 13:36 |
| *** andymaier has joined #openstack-infra | 13:37 | |
| *** vikrant has joined #openstack-infra | 13:37 | |
| *** dteselkin has quit IRC | 13:41 | |
| *** pblaho has quit IRC | 13:43 | |
| *** pblaho has joined #openstack-infra | 13:44 | |
| *** yamahata has quit IRC | 13:44 | |
| *** pblaho has quit IRC | 13:44 | |
| *** pblaho has joined #openstack-infra | 13:45 | |
| *** esikachev has quit IRC | 13:46 | |
| *** dteselkin has joined #openstack-infra | 13:46 | |
| *** dimtruck is now known as zz_dimtruck | 13:47 | |
| *** vikrant has quit IRC | 13:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Prevent Negative Caching https://review.openstack.org/363321 | 13:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Revert "Revert per-resource dogpile.cache work" https://review.openstack.org/366143 | 13:48 |
| *** xarses has quit IRC | 13:48 | |
| *** _nadya_ has joined #openstack-infra | 13:50 | |
| *** gongysh has joined #openstack-infra | 13:52 | |
| *** mhickey_ has joined #openstack-infra | 13:55 | |
| *** cardeois has joined #openstack-infra | 13:56 | |
| *** mhickey has quit IRC | 13:57 | |
| *** annegentle has joined #openstack-infra | 13:58 | |
| *** admcleod has joined #openstack-infra | 13:58 | |
| *** admcleod_ has quit IRC | 14:00 | |
| *** krtaylor has joined #openstack-infra | 14:01 | |
| *** Julien-zte has joined #openstack-infra | 14:03 | |
| *** smarcet has quit IRC | 14:03 | |
| *** bin_ has joined #openstack-infra | 14:04 | |
| *** thorst has quit IRC | 14:05 | |
| *** thorst has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Update ironic-multinode job timeout https://review.openstack.org/365706 | 14:07 |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| *** xarses has joined #openstack-infra | 14:09 | |
| *** smarcet has joined #openstack-infra | 14:09 | |
| fungi | too bad smarcet dropped off. anyway, if he comes back, i've made openstackid-resources-release have the same members as openstackid-release just now | 14:09 |
| *** amotoki_ has quit IRC | 14:09 | |
| *** jcoufal has joined #openstack-infra | 14:09 | |
| openstackgerrit | Alexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack https://review.openstack.org/366154 | 14:10 |
| smarcet | fungi: thx a lot :D | 14:10 |
| smarcet | had a glitch on connneciton | 14:10 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Enable a missing Magnum job https://review.openstack.org/366155 | 14:10 |
| fungi | oh, you came back! | 14:11 |
| fungi | cool, give it another try | 14:11 |
| lucasagomes | hi all, can I please have some eyes on https://review.openstack.org/#/c/364921/ ? It does add a query to elastic-search | 14:11 |
| jpmaxman | Fungi just pinged smarcet | 14:12 |
| *** esberglu has joined #openstack-infra | 14:12 | |
| fungi | AJaeger: it's possible 364723 was in merge conflict 3 days ago but is no longer (perhaps some other conflicting change got reverted?). try a recheck comment | 14:14 |
| *** thorst has quit IRC | 14:14 | |
| fungi | AJaeger: also possible that something happened to one of the zuul mergers to cause it to fail to generate that merge commit | 14:14 |
| cloudnull | mornings | 14:15 |
| *** inc0 has joined #openstack-infra | 14:15 | |
| *** nmagnezi_ has quit IRC | 14:15 | |
| *** amotoki has joined #openstack-infra | 14:17 | |
| pabelanger | fungi: Any objections on bringing cloud8 online with 10 nodes this morning? https://review.openstack.org/#/c/364408/ | 14:17 |
| pabelanger | cloudnull: day 3, 0 launch node errors in osic-cloud1 | 14:18 |
| AJaeger | fungi, recheck does not change anything | 14:19 |
| fungi | pabelanger: lgtm. looks like we've got images uploading successfully there | 14:20 |
| fungi | AJaeger: are you sure? i don't see another zuul comment there yet | 14:20 |
| *** xarses has quit IRC | 14:20 | |
| *** gongysh has quit IRC | 14:20 | |
| pabelanger | fungi: Ya, we've need uploading them for a few days now | 14:20 |
| *** hashar is now known as hasharAway | 14:20 | |
| *** links has quit IRC | 14:21 | |
| fungi | AJaeger: double-check zuul status and make sure it's not running jobs in the check pipeline right now | 14:21 |
| AJaeger | fungi, indeed it's building in zuul status | 14:21 |
| *** yaume_ has joined #openstack-infra | 14:21 | |
| fungi | in that case, the merge conflict message will disappear when we get new check results | 14:21 |
| *** yaume has quit IRC | 14:23 | |
| pabelanger | clarkb: https://review.openstack.org/#/c/365819/ ianw uploaded a new patch for glean, that does the same thing, just a little nicer I think. | 14:23 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: New Project timmy - ansible-like tool https://review.openstack.org/359831 | 14:23 |
| pabelanger | mordred: ^ might be interested too, since rackspace is currently broken with glean | 14:23 |
| mordred | pabelanger: blerg | 14:24 |
| *** senk has quit IRC | 14:25 | |
| mordred | pabelanger: so you like ianw's patch betterer? | 14:25 |
| *** zigo has quit IRC | 14:25 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs https://review.openstack.org/364479 | 14:25 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 14:25 |
| pabelanger | mordred: Ya, does same thing. Think his is cleaner | 14:25 |
| *** roxanaghe has joined #openstack-infra | 14:26 | |
| cloudnull | pabelanger: ++ that's awesomre | 14:27 |
| cloudnull | *awesome | 14:27 |
| *** akshai has quit IRC | 14:27 | |
| jkilpatr | are the jenkins check/gate jobs supposed to not have libffi anymore? I have builds failing today that didn't before this weekend with no changes. | 14:28 |
| *** jordanP has joined #openstack-infra | 14:28 | |
| mordred | pabelanger: looks good | 14:29 |
| *** zz_dimtruck is now known as dimtruck | 14:29 | |
| openstackgerrit | Alexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack https://review.openstack.org/366154 | 14:29 |
| anteaya | jkilpatr: sounds like one of your dependencies released a new version | 14:29 |
| *** salv-orlando has joined #openstack-infra | 14:29 | |
| *** yaume_ has quit IRC | 14:30 | |
| *** roxanaghe has quit IRC | 14:31 | |
| sdague | mordred / fungi - can I get a quick review of this os-log-analyzer filter update - https://review.openstack.org/#/c/366096/ ? | 14:31 |
| anteaya | jkilpatr: and your jobs have whatever they are told to have | 14:31 |
| openstackgerrit | Alexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack https://review.openstack.org/366154 | 14:31 |
| *** zigo has joined #openstack-infra | 14:32 | |
| *** salv-orl_ has quit IRC | 14:32 | |
| *** yar_ has joined #openstack-infra | 14:33 | |
| *** zigo is now known as Guest39147 | 14:33 | |
| openstackgerrit | Merged openstack-infra/glean: Skip interfaces not in sys_interfaces https://review.openstack.org/365819 | 14:33 |
| mordred | pabelanger: need we to cut a point-release of glean? | 14:33 |
| *** mhickey_ has quit IRC | 14:33 | |
| *** rook has joined #openstack-infra | 14:33 | |
| pabelanger | mordred: yes, if possible | 14:34 |
| fungi | jkilpatr: which job(s) specifically? | 14:34 |
| mordred | pabelanger: tag for 1.6.1 pushed | 14:34 |
| pabelanger | mordred: thanks | 14:35 |
| *** jamesdenton has joined #openstack-infra | 14:35 | |
| zaro | morning | 14:35 |
| *** xarses has joined #openstack-infra | 14:36 | |
| anteaya | morning zaro | 14:37 |
| *** akshai has joined #openstack-infra | 14:37 | |
| openstackgerrit | Merged openstack-infra/os-loganalyze: add placement-api to list of understood logs https://review.openstack.org/366096 | 14:37 |
| *** amotoki has quit IRC | 14:38 | |
| *** fguillot has joined #openstack-infra | 14:38 | |
| *** Guest39147 has quit IRC | 14:39 | |
| *** mhickey_ has joined #openstack-infra | 14:39 | |
| *** sarob has joined #openstack-infra | 14:40 | |
| jkilpatr | fungi, https://review.openstack.org/#/c/363688/9 | 14:40 |
| mtreinish | yolanda: thanks for the +A on the lpmqtt patches yesterday. If you get a sec could you populate the gerrit groups: https://review.openstack.org/#/admin/groups/1550,members https://review.openstack.org/#/admin/groups/1551,members https://review.openstack.org/#/admin/groups/1552,members and https://review.openstack.org/#/admin/groups/1553,members | 14:40 |
| *** amitgandhinz has quit IRC | 14:41 | |
| *** vinaypotluri has joined #openstack-infra | 14:42 | |
| *** zigo_ has joined #openstack-infra | 14:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add libffi-dev to bindep https://review.openstack.org/366168 | 14:42 |
| yolanda | hi mtreinish , sure | 14:42 |
| mtreinish | yolanda: cool, thanks | 14:43 |
| *** krtaylor has quit IRC | 14:43 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 14:43 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Move monasca-log tempest job to check queue https://review.openstack.org/365681 | 14:44 |
| fungi | jkilpatr: it looks like it may have previously been relying on a prebuilt wheel for cffi from our mirror, but cffi 1.8.2 was released today and we don't have a new wheel built for it yet so your job is falling back to attempting to build it from the sdist tarball | 14:44 |
| *** claudiub has quit IRC | 14:44 | |
| *** sdake has joined #openstack-infra | 14:44 | |
| *** sarob has quit IRC | 14:45 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 14:45 |
| fungi | jkilpatr: you should probably add cffi's build dependencies to http://git.openstack.org/cgit/openstack/browbeat/tree/bindep.txt, which would look something like the libffi-related lines at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt#n38 | 14:45 |
| jkilpatr | fungi, thanks | 14:45 |
| fungi | jkilpatr: i expect that would pass in the ci's current state, and would be more correct going forward | 14:46 |
| *** daemontool_ has quit IRC | 14:47 | |
| *** zigo_ has quit IRC | 14:47 | |
| *** daemontool has joined #openstack-infra | 14:48 | |
| *** zigo_ has joined #openstack-infra | 14:48 | |
| *** mikelk has quit IRC | 14:49 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config: Move monasca-log tempest job to check queue https://review.openstack.org/365681 | 14:49 |
| *** JerryOpenix has joined #openstack-infra | 14:52 | |
| *** _nadya_ has quit IRC | 14:52 | |
| *** annegentle has quit IRC | 14:53 | |
| mtreinish | corvus, clarkb, fungi: so now that lpmqtt is a thing, did we ever come to a conclusion on what to do for the imap for it? | 14:53 |
| fungi | mtreinish: i think it comes down to choosing between a dedicated alias address for our infra inbox (in rackspace apps where openstack.org e-mail is hosted), or configuring exim on firehose01 to accept e-mail from lp and then setting up courier or cyrus or something to listen on localhost | 14:56 |
| clarkb | pabelanger: cool looks like that fix for glean merged and we got a new release | 14:56 |
| fungi | mtreinish: i'm more in favor of the latter, personally, since we're not entirely in control of the former | 14:57 |
| *** ifarkas is now known as ifarkas_afk | 14:57 | |
| *** dkehn_ has quit IRC | 14:57 | |
| mtreinish | fungi: that's fine, although do we have puppet for exim and the imap server anywhere? | 14:57 |
| pabelanger | clarkb: Yup, and ubuntu-xenial build should pull it in today | 14:58 |
| *** amitgandhinz has joined #openstack-infra | 14:58 | |
| mtreinish | my personal mail infra is postfix and dovecot (although I used cyrus a long time ago) so I'm not sure how much input I can give on those modules | 14:59 |
| clarkb | AJaeger: catching up on requirements related changes/bugs | 15:00 |
| clarkb | AJaeger: will review though I have a call now | 15:00 |
| *** pradk has quit IRC | 15:00 | |
| *** piet has joined #openstack-infra | 15:01 | |
| *** rodrigods has quit IRC | 15:02 | |
| *** rodrigods has joined #openstack-infra | 15:02 | |
| *** aviau has quit IRC | 15:02 | |
| *** aviau has joined #openstack-infra | 15:03 | |
| *** yaume has joined #openstack-infra | 15:03 | |
| *** thcipriani|afk is now known as thcipriani | 15:03 | |
| *** AJaeger has quit IRC | 15:04 | |
| clarkb | pabelanger: nodepool.image.build.ubuntu-xenial: Successfully installed glean-1.6.0 might need to rebuild to get 1.6.1 | 15:06 |
| mordred | fungi, mtreinish: I'm also in favor of the second option | 15:06 |
| pabelanger | clarkb: nice | 15:07 |
| mordred | we have exim modules in puppet already. jeblair and I have been cyrus users for forever - although our existing cyrus is not puppeted ... but I'd imagine getting a cyrus up and going would not be terribly hard | 15:07 |
| pabelanger | clarkb: cool, back down to 1 hour builds | 15:08 |
| *** annegentle has joined #openstack-infra | 15:08 | |
| pabelanger | clarkb: oh, wait. that is from yesterday | 15:08 |
| clarkb | pabelanger: oh that would explain it | 15:08 |
| mtreinish | mordred: ok cool, iirc setting up cyrus wasn't very difficult when I did it before | 15:08 |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 15:09 |
| pabelanger | clarkb: we doing ubuntu-trusty now | 15:09 |
| mtreinish | mordred: sounds like you're volunteering to help :) | 15:09 |
| clarkb | kk so need a rebuild just hasn't happened yet :) | 15:09 |
| *** andreas_s has quit IRC | 15:09 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make tags reusable https://review.openstack.org/366188 | 15:09 |
| *** dkehn_ has joined #openstack-infra | 15:10 | |
| *** gouthamr has joined #openstack-infra | 15:10 | |
| *** amotoki has joined #openstack-infra | 15:11 | |
| *** thorst has joined #openstack-infra | 15:11 | |
| *** _bmjen is now known as bmjen | 15:11 | |
| mordred | mtreinish: I am theoretically volunteering ... but I will defer to jeblair for really thinking about it :) | 15:12 |
| *** gouthamr_ has joined #openstack-infra | 15:12 | |
| mtreinish | mordred: heh I only found 1 cyrus module on puppetforge and it doesn't look so complete: https://forge.puppet.com/buzzdeee/cyrus_imapd | 15:12 |
| mtreinish | and it might be openbsd specific | 15:13 |
| *** spzala has joined #openstack-infra | 15:13 | |
| *** yamahata has joined #openstack-infra | 15:14 | |
| *** gouthamr has quit IRC | 15:15 | |
| *** matt-borland has quit IRC | 15:17 | |
| clarkb | pabelanger: mgagne are we ready to increase the mtl01 max-servers today? | 15:17 |
| pabelanger | Think so, working to bring cloud8 online | 15:18 |
| clarkb | if mgagne acks I can approve | 15:18 |
| *** thorst has quit IRC | 15:19 | |
| *** yaume has quit IRC | 15:19 | |
| *** pradk has joined #openstack-infra | 15:20 | |
| fungi | mtreinish: it's entirely possible that the debian/ubuntu package for cyrus automatically does what we need without needing to be tweaked | 15:20 |
| fungi | so might not need a separate puppet module | 15:20 |
| *** zigo_ is now known as zigo | 15:20 | |
| *** AJaeger has joined #openstack-infra | 15:22 | |
| mtreinish | fungi: oh heh, yeah that's possible. My only experience running cyrus was with gentoo running on a ps3. (which I do not recommend to anyone) | 15:22 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 15:23 |
| rcarrillocruz | ps3 | 15:23 |
| rcarrillocruz | that's awesome mtreinish :D | 15:23 |
| *** spzala has quit IRC | 15:26 | |
| mtreinish | rcarrillocruz: heh, it makes a terrible mail server. The main cpu is too slow (and the spus were never used by anything) and interpreted languages were just terrible on that old powerpc cpu, it made using spamassassin fun. | 15:27 |
| *** piet has quit IRC | 15:27 | |
| mtreinish | my favorite part was that acpi didn't really work on it either so it ate >200w at idle | 15:27 |
| *** gokrokve has joined #openstack-infra | 15:27 | |
| *** mriedem1 has joined #openstack-infra | 15:27 | |
| *** Julien-zte has quit IRC | 15:28 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 15:29 |
| *** armax has joined #openstack-infra | 15:29 | |
| *** mriedem has quit IRC | 15:30 | |
| *** mptacekx has quit IRC | 15:30 | |
| *** thorst has joined #openstack-infra | 15:30 | |
| rcarrillocruz | well, yeah, the ps3 was an akward machine to code for... | 15:31 |
| rcarrillocruz | game devs complaining about it all the time | 15:31 |
| *** piet has joined #openstack-infra | 15:31 | |
| clarkb | mtreinish: I ran folding at home on mine during the winter because it kept my room warm | 15:31 |
| *** M-docaedo_vector has quit IRC | 15:32 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 15:32 |
| pabelanger | clarkb: rcarrillocruz: mind helping https://review.openstack.org/#/c/366168/ land? adds libffi-dev to bindep.txt | 15:32 |
| mtreinish | clarkb: heh, that probably stressed the cpu about the same as spamassassin. (it really didn't like running perl) | 15:32 |
| pabelanger | something is pulling it in for nodepool gate test | 15:32 |
| *** caowei has quit IRC | 15:33 | |
| clarkb | pabelanger: maybe we should just make sure nodepool has a bindep.txt? | 15:33 |
| clarkb | oh wait this is not the fallback its the bindep for project-config itself | 15:33 |
| clarkb | neverming | 15:33 |
| clarkb | *mind | 15:33 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918 | 15:34 |
| *** mhickey__ has joined #openstack-infra | 15:34 | |
| rcarrillocruz | sure | 15:34 |
| pabelanger | Also, working on mirror.regionone.osic-cloud8.openstack.org again now that networking issue has been sorted out | 15:35 |
| clarkb | https://review.openstack.org/#/c/363984/1 is the internap quota increase | 15:35 |
| *** spzala has joined #openstack-infra | 15:35 | |
| clarkb | looks like mgagne acked the change itself. Should I just approve then? | 15:35 |
| *** Jeffrey4l has quit IRC | 15:35 | |
| pabelanger | clarkb: ya, we need to land 366168 first, but works for me | 15:35 |
| clarkb | pabelanger: cool remember that you ened to update the existing dns records for that mirror as currently it points at cloud1 (rather than just adding a new set of records) | 15:36 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 15:36 |
| *** mhickey_ has quit IRC | 15:36 | |
| pabelanger | clarkb: ack | 15:36 |
| clarkb | pabelanger: looks like rcarrillocruz and I +2'd 366168 at the same time. I have since approved it so hopeflly that goes in soon | 15:36 |
| pabelanger | can't SSH into the server right now so need to fix that first | 15:37 |
| *** martinkopec has quit IRC | 15:37 | |
| pabelanger | While testing ssh access: [Errno None] Unable to connect to port 22 on 72.32.203.55 | 15:37 |
| clarkb | pabelanger: security groups? | 15:37 |
| *** tonytan4ever has joined #openstack-infra | 15:38 | |
| *** matbu is now known as matbu|bbl | 15:38 | |
| *** tonytan_brb has joined #openstack-infra | 15:39 | |
| odyssey4me | pabelanger hmm, have some sort of filtering things been implemented somewhere between Sep 1 and Sep 4... our non-master branches are now missing functional tests for trusty | 15:39 |
| odyssey4me | pabelanger an example: https://review.openstack.org/#/c/364356/ & https://review.openstack.org/#/c/366203/ | 15:39 |
| *** andymaier has quit IRC | 15:39 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 15:40 |
| *** pcaruana has joined #openstack-infra | 15:42 | |
| *** roxanaghe has joined #openstack-infra | 15:42 | |
| pabelanger | odyssey4me: could be a side affect of https://review.openstack.org/#/c/363639/ | 15:42 |
| *** tonytan4ever has quit IRC | 15:42 | |
| pabelanger | odyssey4me: yes, we need to include stable branches | 15:43 |
| pabelanger | odyssey4me: let me get a patch up | 15:43 |
| odyssey4me | pabelanger we already have a filter that allowed it - so that one has broken ours | 15:43 |
| odyssey4me | whatever we do - it needs to combine the functional result of both of those | 15:43 |
| pabelanger | odyssey4me: regex likely applied after yours, which removes them | 15:44 |
| *** sdake has quit IRC | 15:44 | |
| pabelanger | let me see | 15:44 |
| odyssey4me | https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L3084-L3087 | 15:44 |
| *** berendt has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Add libffi-dev to bindep https://review.openstack.org/366168 | 15:45 |
| odyssey4me | pabelanger a quick fix is probably to ensure that your additional rule doesn't act on any 'openstack-ansible' jobs | 15:46 |
| *** mhickey__ has quit IRC | 15:46 | |
| *** roxanaghe has quit IRC | 15:47 | |
| pabelanger | odyssey4me: ^gate-(?!openstack-ansible).*(-dsvm)?-ansible-func-ubuntu-trusty(-nv)?$ should work | 15:48 |
| *** psachin has quit IRC | 15:49 | |
| odyssey4me | pabelanger yep, or alternatively adjust https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L3084-L3087 to be something like | 15:49 |
| odyssey4me | urgh, my brain is unable to come up with something there | 15:50 |
| openstackgerrit | Sergei Chipiga proposed openstack-infra/project-config: Add non-voting job for new horizon ui tests https://review.openstack.org/366223 | 15:50 |
| *** tonytan_brb has quit IRC | 15:51 | |
| odyssey4me | so yeah, your suggestion seems fine | 15:51 |
| odyssey4me | pabelanger want me to patch it up? | 15:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Ignore openstack-ansible for ansible-func-ubuntu-trusty https://review.openstack.org/366225 | 15:52 |
| pabelanger | odyssey4me: ^\ | 15:52 |
| *** mriedem1 is now known as mriedem | 15:53 | |
| *** tonytan4ever has joined #openstack-infra | 15:53 | |
| *** dkehn__ has joined #openstack-infra | 15:54 | |
| mgagne | clarkb: go | 15:55 |
| *** matrohon has quit IRC | 15:55 | |
| *** gyee has joined #openstack-infra | 15:55 | |
| clarkb | mgagne: done :) | 15:55 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config: Correct the directory path for Zun https://review.openstack.org/366231 | 15:56 |
| odyssey4me | pabelanger yeah, that appears to correctly ignore the job names we're seeing being affected now | 15:56 |
| *** spzala has quit IRC | 15:57 | |
| pleia2 | good morning | 15:59 |
| *** isaacb has quit IRC | 15:59 | |
| *** claudiub has joined #openstack-infra | 16:00 | |
| *** adrian_otto has joined #openstack-infra | 16:00 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 16:03 |
| *** spzala has joined #openstack-infra | 16:03 | |
| *** gokrokve has quit IRC | 16:05 | |
| *** david-lyle_ has joined #openstack-infra | 16:05 | |
| *** david-lyle_ has quit IRC | 16:05 | |
| *** Swami has joined #openstack-infra | 16:06 | |
| *** abregman has quit IRC | 16:08 | |
| *** spzala has quit IRC | 16:08 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make tags reusable https://review.openstack.org/366188 | 16:09 |
| *** senk has joined #openstack-infra | 16:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Set max-servers value to 150 for internap-mtl01 https://review.openstack.org/363984 | 16:09 |
| *** dkehn__ has quit IRC | 16:09 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Make dev the default StoryBoard instance https://review.openstack.org/366238 | 16:10 |
| *** jascott1 has joined #openstack-infra | 16:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Bring osic-cloud8 online with 10 nodes https://review.openstack.org/364408 | 16:14 |
| *** derekh is now known as derekh_afk | 16:14 | |
| pabelanger | rcarrillocruz: seeing some scheduling errors on infracloud right now, looking to see why | 16:14 |
| clarkb | pabelanger: you should beable to take the insatnce uuid and track it from api to compute | 16:15 |
| *** M-docaedo_vector has joined #openstack-infra | 16:15 | |
| clarkb | at least thats what I do when debugging devstack I am sure its the same thing in a real cloud ;) | 16:15 |
| rcarrillocruz | pabelanger: ack | 16:16 |
| rcarrillocruz | i'm away from home now | 16:17 |
| rcarrillocruz | if it's too serious, disable it, i can look at it later after meeting or during , will be around | 16:17 |
| pabelanger | http://paste.openstack.org/show/567205/ | 16:17 |
| pabelanger | clarkb: rcarrillocruz: ^ looks like networking issues? | 16:17 |
| pabelanger | seeing a fair bit of ConnectTimeout exceptions | 16:18 |
| clarkb | pabelanger: thats specifically trying to do a networking allocation | 16:20 |
| *** e0ne has quit IRC | 16:20 | |
| *** smarcet has quit IRC | 16:20 | |
| *** gouthamr_ has quit IRC | 16:20 | |
| *** thorst has quit IRC | 16:21 | |
| *** daemontool has quit IRC | 16:21 | |
| clarkb | pabelanger: it may help to grep all the connection timeouts and see if they are all in the same code path. That would point at neutron whereas timeouts across the board might indicate general performacne issues or an underlying network problem fullfilling those requests | 16:21 |
| *** thorst has joined #openstack-infra | 16:21 | |
| *** rhallisey has quit IRC | 16:21 | |
| *** rhallisey has joined #openstack-infra | 16:22 | |
| pabelanger | clarkb: Ya, we've had 20 ConnectionTimeout today just for ports.json. Maybe we need to up our timeout value | 16:23 |
| *** tphummel has joined #openstack-infra | 16:23 | |
| openstackgerrit | Joe Hakim Rahme proposed openstack-infra/project-config: Adds a new project: nova-tempest-staging https://review.openstack.org/366091 | 16:23 |
| clarkb | pabelanger: ya if its just that one request having it then likely a tuning item related to that | 16:23 |
| *** yamahata has quit IRC | 16:23 | |
| clarkb | pabelanger: or maybe we need to run more neutron api servers or something | 16:23 |
| pabelanger | clarkb: Right, let me find the value, see what the default is and update. Also going to see how we configured neutron api | 16:24 |
| *** daemontool has joined #openstack-infra | 16:25 | |
| *** drifterza has quit IRC | 16:26 | |
| pabelanger | looks like timeout=30 by default in nova.conf | 16:26 |
| *** akshai has quit IRC | 16:26 | |
| clarkb | pabelanger: which seems like it should be plenty | 16:26 |
| clarkb | just off hand 30 seconds to get a port for a VM should be plenty imo | 16:26 |
| pabelanger | checking neutron now to see if there is an issue | 16:27 |
| *** dtantsur is now known as dtantsur|afk | 16:28 | |
| *** akshai has joined #openstack-infra | 16:29 | |
| *** tonytan_brb has joined #openstack-infra | 16:29 | |
| *** daemontool has quit IRC | 16:29 | |
| *** dkehn__ has joined #openstack-infra | 16:29 | |
| *** piet has quit IRC | 16:29 | |
| *** Apoorva has joined #openstack-infra | 16:29 | |
| *** SamYaple has quit IRC | 16:29 | |
| *** SamYaple has joined #openstack-infra | 16:30 | |
| *** Sukhdev has joined #openstack-infra | 16:30 | |
| *** thorst has quit IRC | 16:30 | |
| *** matthewbodkin has quit IRC | 16:30 | |
| pabelanger | neutron-server.log error http://paste.openstack.org/show/567209/ | 16:30 |
| *** piet has joined #openstack-infra | 16:31 | |
| *** piet has quit IRC | 16:31 | |
| *** stevemar has quit IRC | 16:31 | |
| openstackgerrit | Zara proposed openstack-infra/python-storyboardclient: Make dev the default StoryBoard instance https://review.openstack.org/366238 | 16:31 |
| *** tonytan4ever has quit IRC | 16:32 | |
| *** gouthamr has joined #openstack-infra | 16:33 | |
| pabelanger | clarkb: so, neutron is using sqlite right now. | 16:33 |
| pabelanger | connection = sqlite:////var/lib/neutron/neutron.sqlite | 16:33 |
| clarkb | pabelanger: that may explain it :) | 16:33 |
| pabelanger | ya | 16:33 |
| clarkb | since sqlite can't do concurrent connections iirc. Basically forcing everything to be serialized | 16:34 |
| mgagne | :O | 16:34 |
| pabelanger | incoming patch | 16:34 |
| clarkb | pabelanger: keep in mind that the migration from sqlite to mysql is likely not to be a clean one. Probably want to stop running instances there. And start over from scratch with neutron? | 16:34 |
| clarkb | its provider networking so that should mostly just work except we need to make sure we steady state to no instances first | 16:35 |
| *** jordanP has quit IRC | 16:35 | |
| *** derekh_afk is now known as derekh | 16:35 | |
| pabelanger | clarkb: Ya, we should likely disable infracloud for now | 16:35 |
| *** vhosakot has joined #openstack-infra | 16:35 | |
| *** isaacb has joined #openstack-infra | 16:36 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb-* git repo https://review.openstack.org/366246 | 16:36 |
| pabelanger | actuall | 16:37 |
| pabelanger | Oh | 16:37 |
| pabelanger | Hmm | 16:37 |
| *** tonytan_brb is now known as tonytan4ever | 16:37 | |
| pabelanger | my mistake | 16:37 |
| pabelanger | seems neutron is installed on the compute nodes | 16:38 |
| mgagne | pabelanger: you checked the agent and not neutron-server? | 16:38 |
| pabelanger | and I was looking at that | 16:38 |
| zigo | Since when Neutron supports SQLite migrations? :) | 16:38 |
| mgagne | pabelanger: yea, agent doesn't have access to database and usually Puppet or whatever won't put database access in there | 16:38 |
| clarkb | pabelanger: only the l2 agent should be on the compute hosts and maybe the metadata proxiy | 16:38 |
| *** smarcet has joined #openstack-infra | 16:38 | |
| pabelanger | mgagne: clarkb: Ya, that's what I was looking at | 16:38 |
| pabelanger | assuming l2 agent doesn't need database | 16:39 |
| clarkb | (not sure if we enabled metadat or not) | 16:39 |
| *** matt-borland has joined #openstack-infra | 16:39 | |
| mordred | we should not have | 16:39 |
| mordred | last we spoke about it it the plan was to not run it | 16:39 |
| clarkb | pabelanger: if the controller processes are all set up to use mysql the other thing to check is how many processes are we running and are we using pymysql or mysql-python | 16:41 |
| *** notmars-lounge has quit IRC | 16:41 | |
| *** docaedo3 has quit IRC | 16:41 | |
| clarkb | pabelanger: I want to say that with mysql-python you need many more processes as it serializes when it jumps into C. If we use pymysql we can get away with fewer processes (I would prefer pymysql as its what we test with) | 16:41 |
| *** xyang1 has joined #openstack-infra | 16:42 | |
| pabelanger | 51 neutron-server processes | 16:42 |
| pabelanger | 14 neutron-metadata-agent | 16:42 |
| pabelanger | 1 neutron-dhcp-agent | 16:42 |
| *** _nadya_ has joined #openstack-infra | 16:42 | |
| *** piet has joined #openstack-infra | 16:42 | |
| clarkb | ok with that number of neutron-servers I am gonna guess we are using mysql-python | 16:43 |
| *** roxanaghe has joined #openstack-infra | 16:43 | |
| nikhil | AJaeger: hey, so this being merged https://review.openstack.org/#/c/363088/ https://review.openstack.org/#/c/363066/ , if you'd like to consider them critical enough for backport to newton please feel free to create a review. we can consider then for a release for stable/newton if the gate is significantly affected. | 16:43 |
| pabelanger | clarkb: python-pymysql is installed | 16:44 |
| clarkb | thoguh since nodepool serializes its requests anyways maybe thats unrelated (to avoid api throttling ndoepool serializes its requests through the provider manager for each provider) | 16:44 |
| *** lucasagomes is now known as lucas-dinner | 16:44 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 16:46 |
| clarkb | pabelanger: maybe share that traceback with neutron devs and see if they have any useful info? | 16:47 |
| pabelanger | clarkb: https://bugs.launchpad.net/neutron/+bug/1615820 is the same traceback we are seeing | 16:47 |
| openstack | Launchpad bug 1612798 in neutron "duplicate for #1615820 Move db retry logic away from API layer" [Critical,In progress] - Assigned to Kevin Benton (kevinbenton) | 16:47 |
| clarkb | oh cool someone else has already reported it | 16:47 |
| pabelanger | ya, not sure the same | 16:47 |
| *** weshay is now known as weshay_bbiab | 16:47 | |
| *** pradk has quit IRC | 16:47 | |
| pabelanger | but recent | 16:47 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 16:47 |
| *** roxanaghe has quit IRC | 16:47 | |
| *** jamesdenton has quit IRC | 16:48 | |
| amrith | mtreinish, ping | 16:50 |
| mtreinish | amrith: pong | 16:50 |
| amrith | hiya matt | 16:50 |
| amrith | wanted to see if you had a chance to take a look at the message I'd sent you | 16:50 |
| amrith | I think I've gone over most of the things on your list from a couple of weeks ago | 16:51 |
| mtreinish | amrith: yeah it looks that way, most of the things on that etherpad were from the middle of mitaka | 16:53 |
| amrith | yes, and there may have been some from before as well | 16:53 |
| amrith | like trove-integration | 16:53 |
| amrith | dare I say redstack | 16:53 |
| *** Sukhdev has quit IRC | 16:53 | |
| amrith | ;) | 16:53 |
| amrith | that's not yet done, but on my list for O | 16:53 |
| mtreinish | I haven't really watched for the backwards incompat and api change stuff though. But I believe you if you say you're better about that now | 16:54 |
| amrith | we're taking a closer look at those now | 16:54 |
| mtreinish | and after the tox cleanup I think the cti stuff is fixed | 16:54 |
| *** ihrachys has quit IRC | 16:55 | |
| amrith | OK, I'm putting together a list, I'll send it by you. some of them may be related to the stuff you mentioned | 16:55 |
| amrith | related to CI | 16:55 |
| *** mminesh has quit IRC | 16:55 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments https://review.openstack.org/363313 | 16:56 |
| mtreinish | amrith: ok, I'll keep an eye out for it | 16:57 |
| *** sdake has joined #openstack-infra | 16:57 | |
| amrith | thx matt | 16:57 |
| *** asettle has quit IRC | 16:57 | |
| mtreinish | mordred, fungi: ^^^ I guess we'll need to add the install and setup imap in front of that patch | 16:58 |
| *** jcoufal_ has joined #openstack-infra | 16:58 | |
| *** hogepodge has quit IRC | 16:58 | |
| *** eranrom_ has joined #openstack-infra | 16:58 | |
| pabelanger | mrhillsman: osic-cloud8 now online | 16:59 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic | 16:59 |
| pabelanger | will show traffic once we start allocating nodes | 16:59 |
| *** jpich has quit IRC | 16:59 | |
| *** trown is now known as trown|lunch | 17:00 | |
| pabelanger | raddaoui: ^ | 17:00 |
| clarkb | pabelanger: thats still using cloud1 mirror though right? | 17:00 |
| *** eranrom has quit IRC | 17:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:00 | |
| pabelanger | clarkb: yes, still having some network issues launching the mirror | 17:00 |
| clarkb | ok I am gonna grab caffeine and breakfast but let me know if you want more eyes on that | 17:01 |
| *** jcoufal has quit IRC | 17:01 | |
| raddaoui | nice pabelanger was waiting for it | 17:02 |
| *** hogepodge has joined #openstack-infra | 17:03 | |
| *** florianf has quit IRC | 17:04 | |
| *** andymccr has quit IRC | 17:04 | |
| mrhillsman | pabelanger awesome to hear | 17:05 |
| *** bradjones has quit IRC | 17:06 | |
| mrhillsman | pabelanger what issues you having with networking? | 17:06 |
| *** AnarchyAo has joined #openstack-infra | 17:06 | |
| mrhillsman | are you in talks with raddaoui regarding? | 17:06 |
| pabelanger | not yet, just started an hour ago | 17:06 |
| *** Hal has quit IRC | 17:06 | |
| *** yamahata has joined #openstack-infra | 17:07 | |
| pabelanger | basically, want to see if we can launch the server you created using the external-v4 network and 72.32.203.55 address | 17:07 |
| *** andymccr has joined #openstack-infra | 17:07 | |
| mrhillsman | ok cool | 17:07 |
| *** reed has quit IRC | 17:07 | |
| *** reed has joined #openstack-infra | 17:07 | |
| pabelanger | right now, we fail to ssh for some reason. | 17:07 |
| pabelanger | Unable to connect to port 22 on 72.32.203.55 | 17:07 |
| mrhillsman | raddaoui ^ | 17:07 |
| pabelanger | but haven't debug yet | 17:08 |
| mrhillsman | he may be on lunch | 17:09 |
| pabelanger | ya, no rush | 17:09 |
| mrhillsman | good deal, i'll be around if needed :) | 17:09 |
| SamYaple | mrhillsman: youre always needed | 17:09 |
| SamYaple | <3 | 17:09 |
| mrhillsman | lol | 17:10 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 17:10 |
| mrhillsman | you are everywhere | 17:10 |
| *** vhosakot has quit IRC | 17:10 | |
| *** florianf has joined #openstack-infra | 17:10 | |
| irtermite | he kind of is mrhillsman | 17:10 |
| clarkb | 2016-09-06 16:22:07,522 INFO nodepool.image.build.ubuntu-xenial: Successfully installed glean-1.6.1 and that image build finished a few minutes ago | 17:11 |
| SamYaple | i cloned myself | 17:11 |
| *** armax has quit IRC | 17:11 | |
| mrhillsman | i'm b...^ | 17:11 |
| clarkb | pabelanger: ^ so if we can just get that into rax we should be good | 17:11 |
| irtermite | pretty sure I just watched SamYaple reply to 4 different chans and 2 different servers at once | 17:11 |
| SamYaple | irtermite: just the channels you are in | 17:11 |
| mrhillsman | "i'm beginning to think he has clones" | 17:11 |
| *** bradjones has joined #openstack-infra | 17:11 | |
| *** bradjones has quit IRC | 17:11 | |
| *** bradjones has joined #openstack-infra | 17:11 | |
| irtermite | apparently he only signs into chans i am in | 17:11 |
| mrhillsman | i could see him being the stalker type | 17:12 |
| irtermite | indeed | 17:12 |
| *** sdake_ has joined #openstack-infra | 17:12 | |
| SamYaple | im not arguing but i still feel i need to somehow defend myself | 17:12 |
| SamYaple | HEY! | 17:12 |
| SamYaple | all i got | 17:12 |
| irtermite | if he shows up in the pm between cloudnull and me, I'm closing my laptop and going home | 17:12 |
| *** isaacb has quit IRC | 17:12 | |
| SamYaple | i broke irc! | 17:13 |
| SamYaple | alright that enough off topic | 17:13 |
| pabelanger | clarkb: when you are back, maybe you can help me understand why clouds with less resource (10 max-server: infracloud, osic-cloud8) don't get allocated nodes, where we large capacity still get allocations | 17:13 |
| * irtermite throws up arms and runs away | 17:13 | |
| clarkb | pabelanger: nodepool allocates based on a ratio of quotas | 17:13 |
| clarkb | pabelanger: this way we fill up all of the providers at roughly the same time (in theory) | 17:14 |
| AJaeger | nikhil: thanks - I don't think we need to backport them. | 17:14 |
| pabelanger | clarkb: Ah, that explains it | 17:14 |
| clarkb | pabelanger: so when demand is low only those with high quotas get allocations | 17:14 |
| AJaeger | nikhil: it wouldn't harm to backport ;) | 17:15 |
| pabelanger | clarkb: Ya, now that internap-mtl01 is 150, it is getting allocated more nodes | 17:15 |
| *** sdake has quit IRC | 17:15 | |
| nikhil | AJaeger: ack... so may be see later :) | 17:15 |
| raddaoui | pabelanger: still unable to connect to 72.32.203.55 | 17:15 |
| anteaya | clarkb: pabelanger should that algorithm be tweaked? | 17:15 |
| *** pradk has joined #openstack-infra | 17:16 | |
| clarkb | yay updated openstackclient can talk to neutron now | 17:16 |
| clarkb | anteaya: no I think thats probably still a desireable behavior | 17:16 |
| pabelanger | raddaoui: let me try again | 17:16 |
| anteaya | I mean we _do_ value all our host providers, not just those with the largest numbers | 17:16 |
| mrhillsman | ha | 17:16 |
| anteaya | would be great if our algorithm supported that value | 17:16 |
| anteaya | mrhillsman: what do you mean ha? | 17:16 |
| clarkb | anteaya: pabelanger one reason for it is to avoid being our own noisy neighbors unless we have to | 17:16 |
| anteaya | clarkb: okay that is fine | 17:17 |
| anteaya | I just want to ensure _all_ our cloud providers feel valued | 17:17 |
| clarkb | but also it means that we don't completely throttle the small contributor 24/7 while we have plenty of extra capacity in larger cloud | 17:17 |
| clarkb | in theory the cost is spread evenly | 17:17 |
| anteaya | since we need them all and are grateful for their donations | 17:17 |
| * anteaya is grateful | 17:17 | |
| pabelanger | raddaoui: creating server now | 17:17 |
| *** vhosakot has joined #openstack-infra | 17:17 | |
| anteaya | clarkb: okay that is fine | 17:18 |
| anteaya | clarkb: thanks for taking the time to share your thoughts with me | 17:18 |
| *** rossella_s has quit IRC | 17:18 | |
| mrhillsman | don't mind me anteaya | 17:18 |
| anteaya | mrhillsman: as you wish | 17:18 |
| *** dizquierdo has quit IRC | 17:18 | |
| mrhillsman | :) | 17:18 |
| clarkb | pabelanger: rcarrillocruz one thing I have noticed about our security group rules now that openstackclient is working. We actually never want the group to group rules. They perform poorly in the database and at least one cloud has called me at 3am to delete the rules | 17:19 |
| * anteaya can ignore many folks and doubly so at their request | 17:19 | |
| pabelanger | clarkb: anteaya: Ya, it is only noticeable right now because we have 2 providers (infracloud-vanilla, osic-cloud8) with 10 max-servers. Once we bring them up to capacity, I think things will work as expected | 17:19 |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| clarkb | pabelanger: rcarrillocruz so we will want to add that to the clouds management playbook | 17:19 |
| *** ddieterly is now known as ddieterly[away] | 17:19 | |
| pabelanger | raddaoui: Oh, new expection | 17:20 |
| raddaoui | pabelanger: I think it is because of the image you're using, try with the other image it will work fine | 17:20 |
| *** asselin_ has joined #openstack-infra | 17:20 | |
| pabelanger | raddaoui: http://paste.openstack.org/show/567232/ | 17:20 |
| anteaya | pabelanger: oh, my interpretation from clarkb's explanation was that things already were working as expected | 17:21 |
| *** Benj_ has joined #openstack-infra | 17:21 | |
| clarkb | pabelanger: rcarrillocruz both osic-cloud8 and infracloud need to have their group to group security group rules deleted | 17:21 |
| pabelanger | anteaya: Yes, agreed | 17:21 |
| anteaya | pabelanger: okay thanks | 17:22 |
| pabelanger | raddaoui: sure, let me try with that image. But the image we are using now is just the ubuntu-cloud image | 17:22 |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module https://review.openstack.org/366033 | 17:23 |
| pabelanger | clarkb: okay, did we create them with cloud-launcher or was that done manually? | 17:24 |
| clarkb | pabelanger: its part of neutrons defaults which you "can't" change so as a user you have to edit them | 17:24 |
| clarkb | pabelanger: so the cloud launcher just needs to know how to remove them | 17:24 |
| pabelanger | clarkb: okay | 17:25 |
| clarkb | (you actually can change them and mgagne does for internap but its not supported as part of openstack proper aiui) | 17:25 |
| *** stevemar has joined #openstack-infra | 17:26 | |
| *** armax has joined #openstack-infra | 17:26 | |
| mgagne | clarkb: we don't provide group to group rules in our defaults for reasons mentioned above. | 17:26 |
| *** mriedem1 has joined #openstack-infra | 17:26 | |
| mgagne | I can double check and make sure they aren't present in mtl01 | 17:26 |
| *** thorst has joined #openstack-infra | 17:26 | |
| clarkb | mgagne: yup I think its teh right thing to do but we can't rely on every cloud deciding that neutrons defaults are wrong and should be hacked to fix them :) | 17:26 |
| clarkb | mgagne: I already checked at at least in our test account they aren't there | 17:27 |
| mgagne | sure, I think it's a good idea to enforce your rules and purge the unmanaged ones | 17:27 |
| clarkb | pabelanger: the problem is that any time you add a new instance or delete an instance in a security group you have to update all of the other instances in that group | 17:28 |
| *** mriedem has quit IRC | 17:28 | |
| *** e0ne has joined #openstack-infra | 17:28 | |
| *** sdake_ has quit IRC | 17:28 | |
| clarkb | pabelanger: and since nodepool add deletes in such a tight loop that can put an unnecessary strain on the control plane of the cloud. Its redundant too since we already allow all traffic regardless | 17:28 |
| *** sdake has joined #openstack-infra | 17:29 | |
| *** akshai has quit IRC | 17:30 | |
| *** tqtran has joined #openstack-infra | 17:31 | |
| clarkb | mrhillsman: fyi got "HttpException: Gateway Timeout, <html><body><h1>504 Gateway Time-out</h1>" running openstack security group rule show 2fe7489d-3eaf-4833-b6ca-214497188018 against osic-cloud8 a couple minutes ago | 17:31 |
| *** aspiers has quit IRC | 17:31 | |
| pabelanger | ya same | 17:31 |
| pabelanger | http://paste.openstack.org/show/567234/ | 17:31 |
| mrhillsman | looking | 17:32 |
| *** zz_ja is now known as zz_zz_ja | 17:32 | |
| *** tonytan4ever has quit IRC | 17:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Include both Trusty and Xenial gate-requirements-integration-dsvm https://review.openstack.org/365840 | 17:33 |
| *** jheroux has joined #openstack-infra | 17:34 | |
| clarkb | prometheanfire: tonyb ^ | 17:34 |
| prometheanfire | yep, saw that | 17:34 |
| prometheanfire | the oslo-db bump will probably be the first to test that | 17:34 |
| prometheanfire | if it's ever released... | 17:35 |
| *** thorst has quit IRC | 17:35 | |
| *** vhosakot has quit IRC | 17:36 | |
| *** jaosorior has quit IRC | 17:36 | |
| *** e0ne has quit IRC | 17:36 | |
| *** acoles is now known as acoles_ | 17:38 | |
| *** harlowja has joined #openstack-infra | 17:38 | |
| *** kzaitsev_mb has quit IRC | 17:38 | |
| *** amotoki has quit IRC | 17:39 | |
| *** sarob has joined #openstack-infra | 17:39 | |
| *** thiagolib has joined #openstack-infra | 17:40 | |
| *** Ravikiran_K has quit IRC | 17:40 | |
| fungi | okay, i'm back. my isp seems to be offline, but thankfully i have a backup connection through my mobile phone provider | 17:40 |
| *** florianf has quit IRC | 17:40 | |
| clarkb | fungi: did the hurricane turn around and come back to NC? | 17:41 |
| fungi | nah, but this _is_ probably related to post-storm utility work | 17:41 |
| fungi | we were lucky. some other neighborhoods out here are only just getting power back | 17:41 |
| fungi | and a number of roads are out, which makes it harder for the utility trucks to do stuff quickly | 17:42 |
| anteaya | fungi: I'm glad you fared well | 17:42 |
| anteaya | fungi: indeed | 17:42 |
| *** spzala has joined #openstack-infra | 17:43 | |
| *** tonytan4ever has joined #openstack-infra | 17:44 | |
| *** sarob has quit IRC | 17:44 | |
| fungi | corvus: i've put https://review.openstack.org/361463 on the agenda as a proposed spec superseding its counterpart from last week. let me know if i should not have | 17:44 |
| *** vhosakot has joined #openstack-infra | 17:44 | |
| fungi | proposed spec adjustment anyway | 17:44 |
| *** notnownikki has joined #openstack-infra | 17:47 | |
| *** oomichi- is now known as oomichi | 17:49 | |
| *** chem has quit IRC | 17:49 | |
| *** sdake has quit IRC | 17:49 | |
| *** chem has joined #openstack-infra | 17:49 | |
| anteaya | I was just reading that agenda, was wondering what context was for superseded, thank you | 17:51 |
| anteaya | well actually I was just outside watching a red squirrel bury cedar cones, and missing jog0 | 17:52 |
| fungi | yeah, that spec change has replaced the one we discussed in last week's meeting (if you look at the old change it's in an abandoned state referencing the new change linked in our agenda for this week) | 17:52 |
| anteaya | but before that I was looking at the agenda | 17:52 |
| anteaya | fungi: ah thank you | 17:52 |
| *** mriedem1 is now known as mriedem | 17:53 | |
| *** florianf has joined #openstack-infra | 17:54 | |
| *** jamesdenton has joined #openstack-infra | 17:55 | |
| *** sdake has joined #openstack-infra | 17:56 | |
| *** rderose_ is now known as rderose | 17:56 | |
| *** akshai has joined #openstack-infra | 17:57 | |
| *** spzala has quit IRC | 17:57 | |
| *** e0ne has joined #openstack-infra | 17:58 | |
| *** bknudson has joined #openstack-infra | 18:00 | |
| *** spzala has joined #openstack-infra | 18:00 | |
| *** thiagolib has quit IRC | 18:00 | |
| *** persia has quit IRC | 18:01 | |
| *** thiagolib has joined #openstack-infra | 18:01 | |
| *** mrjk has quit IRC | 18:01 | |
| *** maishsk has joined #openstack-infra | 18:02 | |
| *** dkehn__ has quit IRC | 18:03 | |
| *** dkehn_ has quit IRC | 18:03 | |
| *** maishsk has quit IRC | 18:03 | |
| *** dkehn has quit IRC | 18:03 | |
| *** piet has quit IRC | 18:04 | |
| *** spzala has quit IRC | 18:04 | |
| *** crinkle_ is now known as crinkle | 18:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells v1 snapshot race bug 1620761 https://review.openstack.org/366277 | 18:08 |
| openstack | bug 1620761 in OpenStack Compute (nova) "test_create_second_image_when_first_image_is_being_saved intermittently times out in teardown in cells v1 job" [Undecided,Confirmed] https://launchpad.net/bugs/1620761 | 18:08 |
| mrhillsman | clarkb check now | 18:08 |
| clarkb | mrhillsman: just ran the command 3 times and it seems happy each time | 18:08 |
| mrhillsman | awesome | 18:09 |
| *** dkehn has joined #openstack-infra | 18:09 | |
| *** oanson has joined #openstack-infra | 18:10 | |
| *** _nadya_ has quit IRC | 18:10 | |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config: Add Dragonflow fullstack test for ML2 module https://review.openstack.org/366033 | 18:11 |
| mordred | odyssey4me: I've made some progress on the role import topic | 18:11 |
| *** trown|lunch is now known as trown | 18:12 | |
| mordred | odyssey4me: turns out with two patches there is support for triggering an import that mostly works | 18:12 |
| *** mrjk has joined #openstack-infra | 18:13 | |
| raddaoui | pabelanger: try again and see if you can ssh | 18:13 |
| *** Na3iL has quit IRC | 18:13 | |
| clarkb | fungi: pabelanger I am going to do a little digging into the size of log dirs for various jobs to see if we can determine why we filled up our log server recently | 18:14 |
| *** dkehn__ has joined #openstack-infra | 18:15 | |
| *** dkehn_ has joined #openstack-infra | 18:16 | |
| anteaya | clarkb: is that /srv/static/log? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 18:17 |
| *** degorenko is now known as _degorenko|afk | 18:17 | |
| clarkb | anteaya: yes, we had to reduce our retention period in order to make space there again | 18:18 |
| *** pradk has quit IRC | 18:18 | |
| clarkb | and it seemed to happen relatively quickly like all of a sudden we had way more log data being stored | 18:18 |
| fungi | clarkb: thanks! in the short term, we've manually "expired" logs older than 2 months. which is unfortunate but was better than leaving things broken | 18:18 |
| * clarkb discovers the magic is wget -e robots=off | 18:19 | |
| anteaya | yay magic | 18:19 |
| *** ddieterly[away] has quit IRC | 18:19 | |
| anteaya | my read on the graph is the rate changed near the beginning of June | 18:20 |
| fungi | clarkb: i don't think it was super sudden. we began curving upward around may-june | 18:20 |
| anteaya | and then hit the ceiling in August | 18:20 |
| anteaya | fungi: I concur | 18:20 |
| openstackgerrit | Chris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available https://review.openstack.org/359471 | 18:20 |
| clarkb | fungi: in that graph we shot up 2TB in 2 weeks | 18:20 |
| fungi | and yeah, august was when things broke and i expired logs older than 2 months which only freed up about 10% | 18:21 |
| fungi | er, logs older than 3 months | 18:21 |
| fungi | last week is when we then had to knock that down to 2 months | 18:21 |
| clarkb | but yes looks like things generally started growing in an upward trajectory around june | 18:21 |
| clarkb | heh our help footer for the logs totally breaks recursive downloading of logs. So many 404s | 18:21 |
| openstackgerrit | Mike Fedosin proposed openstack-infra/project-config: Add py35 and pypi jobs to Glare and its client https://review.openstack.org/366281 | 18:22 |
| fungi | also my config management change to drop retention down to 3 months didn't work because we still have a dead copy of the log compression and expiration script in system-config but are now using the copy in openstackci | 18:22 |
| anteaya | clarkb: ha ha ha | 18:22 |
| fungi | it's on my list to remove the dead copy and update the live copy before we run out of space again | 18:23 |
| *** berendt has joined #openstack-infra | 18:23 | |
| anteaya | I also didn't know we have a repo in the infra namespace that is not an infra repo | 18:24 |
| anteaya | I learn something new everyday | 18:24 |
| anteaya | I think we as a group are just trying to test tonyb | 18:24 |
| anteaya | and see how many hats he can wear at the same time | 18:24 |
| anteaya | dhellmann: ^^ | 18:25 |
| *** spzala has joined #openstack-infra | 18:25 | |
| anteaya | fungi: can we get a kick in -meeting please? | 18:25 |
| clarkb | 88MB just for neutron logs might help explain it | 18:25 |
| anteaya | or any other channel op? | 18:26 |
| dolphm | abusive user in #openstack-meeting: antipsychiatry | 18:26 |
| clarkb | especially since we run more neutron now | 18:26 |
| lbragstad | anteaya ++ thanks | 18:26 |
| anteaya | can we ban that user please? | 18:26 |
| clarkb | also we have 46MB of libvirt logs each run so between neutron and libvirt thats 120 MB of logs | 18:26 |
| anteaya | I have seen them previously, same nick | 18:26 |
| fungi | done | 18:26 |
| knikolla | thanks | 18:26 |
| anteaya | thank you | 18:26 |
| *** oomichi has quit IRC | 18:26 | |
| *** oomichi has joined #openstack-infra | 18:27 | |
| *** senk has quit IRC | 18:27 | |
| *** shardy has quit IRC | 18:27 | |
| *** rhallisey has quit IRC | 18:28 | |
| *** spzala has quit IRC | 18:28 | |
| anteaya | clarkb: yes that may explain it | 18:28 |
| *** spzala has joined #openstack-infra | 18:28 | |
| *** oanson has quit IRC | 18:29 | |
| anteaya | the move to neutron as devstack default happened this release though I can't recall the timing | 18:29 |
| mordred | clarkb: did the neutrons start logging more things? | 18:29 |
| mordred | or is it just the neutron-by-default you think | 18:29 |
| mordred | anteaya: I cannot remember the timing either | 18:29 |
| mriedem | i'm seeing several failures with "error: command 'x86_64-linux-gnu-gcc' failed with exit status 1" on different nodes, is that known? | 18:29 |
| anteaya | armax: dougwig did the neutrons start logging more things? | 18:29 |
| armax | anteaya: what do you mean? | 18:29 |
| clarkb | mordred: I think maybe its a combo? also not sure why we need all of that libvirt logging seems excessive | 18:30 |
| anteaya | armax: our log storage filled up faster than expected | 18:30 |
| anteaya | armax: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 18:30 |
| mordred | clarkb: I think I remember someone coming in and chatting about saving the libvirt logging | 18:30 |
| mordred | clarkb: something something debugging something something | 18:30 |
| anteaya | armax: clarkb is looking into why that is | 18:30 |
| mriedem | the libvirt logging was needed for live migration debug | 18:30 |
| *** rhallisey has joined #openstack-infra | 18:30 | |
| anteaya | armax: we moved to running neutron as a devstack default | 18:30 |
| clarkb | mriedem: this isn't a live migration job though | 18:30 |
| clarkb | mriedem: I have no issues turning it on where its needed but it apepars to be on everywhere | 18:30 |
| anteaya | armax: clarkb notes taht neutron logs are large | 18:30 |
| mriedem | clarkb: yeah i know | 18:31 |
| anteaya | armax: we are wondering if neutron logs have always been large and we didn't notice | 18:31 |
| *** roxanagh_ has joined #openstack-infra | 18:31 | |
| anteaya | armax: or if they have increased of late | 18:31 |
| anteaya | armax: and hello, nice to see you :) | 18:31 |
| armax | anteaya: not that I am aware, we tend to keep an eye on it to make sure we don’t log trash | 18:31 |
| armax | but one thing to consider is that server side we one big log | 18:31 |
| armax | whereas other projects are smaller but multiple log files | 18:31 |
| anteaya | armax: okay thank you | 18:32 |
| armax | like nova has api, sched, ctrl, cpu | 18:32 |
| *** thorst has joined #openstack-infra | 18:32 | |
| clarkb | oh hrm my downloads may be inflating the logs | 18:32 |
| clarkb | in which case it would be slightly flawed numbers checking | 18:32 |
| *** pvaneck has joined #openstack-infra | 18:32 | |
| armax | anteaya: I’ll go over to see if there’s some particular bad offender | 18:32 |
| *** jamesdenton has quit IRC | 18:32 | |
| armax | and trim it down if we can | 18:32 |
| anteaya | armax: thanks we appreciate you taking a look | 18:33 |
| clarkb | ya its inflating them | 18:33 |
| clarkb | thats annoying. Let me rerun with better infoa | 18:33 |
| clarkb | the expanded numbers are still reatively interesting to each other | 18:33 |
| fungi | if it were me, i'd just ssh into static.o.o and check files as they exist on its filesystem, since you have shell access | 18:35 |
| *** roxanagh_ has quit IRC | 18:35 | |
| clarkb | ya but I like doing it the other way to show its possible and not unreasonably difficult | 18:35 |
| *** pradk has joined #openstack-infra | 18:35 | |
| fungi | fair enough. it's a worthwhile exercise | 18:35 |
| fungi | i just wouldn't want to try to mirror all ~12tb | 18:36 |
| anteaya | that would take a while | 18:36 |
| anteaya | expecially using your phone as your isp connection | 18:36 |
| fungi | indeed. this rather unfortunate outage is preventing me from easily building the last batch of contributor invites for barcelona | 18:37 |
| *** asettle has joined #openstack-infra | 18:37 | |
| anteaya | :( | 18:37 |
| *** piet has joined #openstack-infra | 18:37 | |
| anteaya | do you think you will have service tomorrow? | 18:37 |
| fungi | so that may have to wait for tomorrow depending on how soon they get things back up and running again | 18:37 |
| anteaya | hopefully by tomorrow then | 18:37 |
| lbragstad | anteaya fungi we seem to have the same use from the #openstack-meeting channel spamming #openstack-keystone now, too | 18:39 |
| *** thorst has quit IRC | 18:39 | |
| lbragstad | s/use/user/ | 18:39 |
| anteaya | lbragstad: sigh | 18:40 |
| fungi | lbragstad: yeah, i added them to the akick list for that channel now too, though they only /join'ed long enough to spam you and then /part'ed again immediately after | 18:41 |
| lbragstad | fungi ah - got it.. thank you! | 18:41 |
| anteaya | fungi: do we have a way of blacklisting a user like this one from all our channels? | 18:41 |
| *** oomichi has quit IRC | 18:42 | |
| *** asettle has quit IRC | 18:42 | |
| *** baoli has joined #openstack-infra | 18:42 | |
| fungi | anteaya: not easily. it's something we could probably add to accessbot but the delay involved in that may make it less attractive | 18:42 |
| *** baoli_ has quit IRC | 18:42 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Publish install guide for Ironic https://review.openstack.org/366290 | 18:44 |
| *** asettle has joined #openstack-infra | 18:45 | |
| anteaya | fungi: understood | 18:45 |
| *** asettle has quit IRC | 18:46 | |
| clarkb | fungi: oh ya definitely don't want to try and get all 12TB, but a small representative portion of the logs should be easy enough | 18:46 |
| *** priteau has joined #openstack-infra | 18:48 | |
| *** oomichi has joined #openstack-infra | 18:48 | |
| sshnaidm | yolanda, pabelanger could you please enable rh2? https://review.openstack.org/353163 | 18:49 |
| pabelanger | sshnaidm: I'd like to land https://review.openstack.org/#/c/357308/ first so we can remove the private infrastructure from nodepool logs | 18:50 |
| *** weshay_bbiab is now known as weshay | 18:51 | |
| pabelanger | sshnaidm: bascially, 3 projects in tripleo-test-cloud-rh2, openstackci (afs mirror), openstackzuul (nodepool), tripleo (for private infrastruture). | 18:52 |
| *** oomichi has quit IRC | 18:52 | |
| sshnaidm | pabelanger, you will not see the results though (or failures), ovb rh1 cloud is disabled | 18:52 |
| *** oomichi has joined #openstack-infra | 18:52 | |
| mordred | infra-root, Shrews: we're still running nodepool on a git checkout - I'm going to tag a shade release unless there is a rancorous outcry complete with wailing and gnashing of teeth | 18:52 |
| pabelanger | sshnaidm: there is no ovb for tripleo-test-cloud-rh2? | 18:52 |
| * fungi musters only a very limited quantity of gnashing | 18:53 | |
| *** mwhahaha has quit IRC | 18:53 | |
| sshnaidm | pabelanger, now it doesn't run, bnemec have this patch for enabling it, we're without any ovb now | 18:53 |
| sshnaidm | pabelanger, rh2 is out of configs and rh1 suffers from heat problem and is disabled | 18:54 |
| *** mwhahaha has joined #openstack-infra | 18:54 | |
| pabelanger | sshnaidm: even so, the issue is the private infrastructure in the nodepool project. If we can create openstackzuul for tripleo-test-cloud-rh2, I think that is all we need today to bring it back online | 18:54 |
| sshnaidm | pabelanger, I think it's safer to land your patch on working cloud so we can see any problems right after | 18:54 |
| Shrews | mordred: 2d70f8b? | 18:55 |
| mordred | Shrews: yah | 18:55 |
| mordred | Shrews: as 1.12.0 - yah? | 18:55 |
| sshnaidm | pabelanger, sorry, what do you mean by "create openstackzuul"? | 18:55 |
| Shrews | geez. how did we get so many changes in since the last tag? | 18:56 |
| Shrews | mordred: yeah | 18:56 |
| openstackgerrit | Merged openstack-infra/system-config: launch-node.py: restart interface https://review.openstack.org/359416 | 18:56 |
| openstackgerrit | Merged openstack-infra/system-config: launch-node.py: save key when failing early https://review.openstack.org/359417 | 18:57 |
| clarkb | fungi: and I have discovered the issue with doing it without root. If you set accept-encoding to gzip then it can't read the index.html files bceause they are gzipped :) | 18:57 |
| mordred | Shrews: because evil | 18:57 |
| mordred | Shrews: that's how | 18:57 |
| fungi | clarkb: ouch | 18:57 |
| clarkb | in any case it looks like things are actaully getting gzipped serverside which is good. Once inflated libvirt and neutron account for the majority of the disk use from what I can tell | 18:57 |
| pabelanger | sshnaidm: this is the current server list in the openstack-nodepool project: http://paste.openstack.org/show/567251/ | 18:57 |
| *** ddieterly has joined #openstack-infra | 18:58 | |
| clarkb | trying to see if the compression ratio preserves that on the compressed side of things | 18:58 |
| pabelanger | sshnaidm: Which is causing warnings to appear in nodepool.o.o. If you create a new project, openstackzuul, which shared network access to the openstack-nodepool project, we'd update nodepool.o.o to use that | 18:58 |
| clarkb | armax: if you inflate all of the screen-q-* services logs you get almost 100MB of logs for neutron on the normal neutron dsvm tempest full job fwiw. I am not sure if that is considered normal or not | 18:59 |
| rcarrillocruz | back | 18:59 |
| rcarrillocruz | pabelanger, clarkb : so, neutron sqlite | 18:59 |
| rcarrillocruz | is there anything in review? | 18:59 |
| pabelanger | rcarrillocruz: nope, my fail | 18:59 |
| rcarrillocruz | otherwise i can poke at it tomorrow | 18:59 |
| pabelanger | rcarrillocruz: got confused by compute nodes | 18:59 |
| rcarrillocruz | ah | 18:59 |
| armax | clarkb: what about other server projects? | 18:59 |
| rcarrillocruz | so you found the issue ? | 18:59 |
| armax | what’s the target you feel comfortable with? | 18:59 |
| rcarrillocruz | clarkb: yeah, i remember that from mid cycle | 19:00 |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| rcarrillocruz | we had a playbook for the default secgroup rules | 19:00 |
| rcarrillocruz | i think we can fix that by putting pertinent absent os_security_group_rule on the cloud launcher | 19:00 |
| mordred | rcarrillocruz: that should be a thing cloud-launcher should be able to express, yeah? | 19:00 |
| mordred | ++ | 19:00 |
| anteaya | meeting time | 19:00 |
| rcarrillocruz | mordred: yup | 19:01 |
| clarkb | armax: I am not sure I have a target, just pointing out that once uncompressed you have a lot more logs than the other services. This may mean that your logs are too chatty or maybe its the appropriate amount | 19:01 |
| clarkb | armax: mostly just sharing data at this point. | 19:01 |
| clarkb | armax: but if you consider that a cloud running for half an hour generates 100MB of logs just for neutron translate that to production running for months and you may not be the happiest ops person | 19:02 |
| mordred | Shrews: ok. shade 1.12.0 relased | 19:02 |
| mordred | Shrews: I, for one, am looking forward to whatever carnage we create with that | 19:02 |
| *** jascott1 has quit IRC | 19:04 | |
| *** jascott1 has joined #openstack-infra | 19:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Update shade depend to 1.12.0 https://review.openstack.org/366294 | 19:05 |
| armax | clarkb: understood, most clouds don’t run in debug mode though | 19:05 |
| clarkb | armax: I have heard the opposite :) | 19:05 |
| clarkb | armax: most ops people I talk to say they always run with debug on because its the only way to debug issues | 19:06 |
| armax | clarkb: let me look if there’s any fat that can be trimmed | 19:06 |
| armax | clarkb: knowing about the size of other projects’ logs would be good though, just to have a sense of how things compare to one another | 19:08 |
| *** e0ne has quit IRC | 19:08 | |
| *** akshai has quit IRC | 19:09 | |
| *** hasharAway is now known as hashar | 19:09 | |
| clarkb | armax: ya I am rerunning my crawl to get that comparative data | 19:09 |
| clarkb | armax: I can post a paste when I have it put together | 19:09 |
| *** Sukhdev has joined #openstack-infra | 19:09 | |
| *** pcaruana has quit IRC | 19:11 | |
| *** Guest78236 is now known as melwitt | 19:14 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets https://review.openstack.org/361463 | 19:14 |
| *** mriedem has quit IRC | 19:16 | |
| *** dahoo has joined #openstack-infra | 19:18 | |
| armax | clarkb: thanks | 19:20 |
| *** amitgandhinz has quit IRC | 19:20 | |
| *** pradk- has joined #openstack-infra | 19:20 | |
| *** amitgandhinz has joined #openstack-infra | 19:20 | |
| *** pradk has quit IRC | 19:21 | |
| dhellmann | mordred : did you see that the shade tarball job failed? it looks like a missing dev package for ffi -- http://logs.openstack.org/b1/b1f35b4bce91ae1da5070cd3fc0063e6632486db/release/shade-tarball/3c56972/console.html | 19:21 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs: Prioritize Docs Publishing via AFS https://review.openstack.org/366303 | 19:21 |
| *** pradk- is now known as pradk | 19:22 | |
| rcarrillocruz | dhellmann: i believe pabelanger pushed a change earlier to include libffi-dev | 19:22 |
| dhellmann | rcarrillocruz : ok, cool. I saw the failure email and wanted to make sure someone else had noticed it. | 19:23 |
| pabelanger | rcarrillocruz: dhellmann: Oh, that was for project-config. We likely need to update shade now too | 19:23 |
| mordred | pabelanger: oh goodie | 19:24 |
| mordred | dhellmann: thanks! | 19:24 |
| *** tesseract- has quit IRC | 19:25 | |
| pabelanger | guess we don't have wheels for cffi | 19:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add libffi-dev to bindep.txt https://review.openstack.org/366304 | 19:25 |
| mordred | pabelanger: yah? ^^ | 19:25 |
| *** akshai has joined #openstack-infra | 19:25 | |
| *** maishsk has joined #openstack-infra | 19:26 | |
| pabelanger | only see cff-1.7.0 right now | 19:26 |
| *** persia has joined #openstack-infra | 19:27 | |
| pabelanger | looks like 1.8.2 was release today | 19:27 |
| *** maishsk has quit IRC | 19:27 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 19:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:27 | |
| *** schang_ has joined #openstack-infra | 19:28 | |
| *** amitgandhinz has quit IRC | 19:30 | |
| *** schang_ has quit IRC | 19:30 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments https://review.openstack.org/363313 | 19:30 |
| clarkb | armax: http://paste.openstack.org/show/567259/ from one sample of the neutron tempest dsvm full job | 19:30 |
| *** amitgandhinz has joined #openstack-infra | 19:30 | |
| clarkb | armax: also that was from the gate not check to avoid broken change causing weirdness | 19:30 |
| *** roxanagh_ has joined #openstack-infra | 19:32 | |
| *** coolsvap_ has quit IRC | 19:32 | |
| mtreinish | armax, clarkb: I gave up trying to fix that right after I started with: https://review.openstack.org/#/c/330848/ (although I'm curious how many -1s I can get) | 19:33 |
| *** maishsk has joined #openstack-infra | 19:33 | |
| *** claudiub has quit IRC | 19:34 | |
| *** Apoorva has quit IRC | 19:34 | |
| *** roxanagh_ has quit IRC | 19:36 | |
| *** thorst has joined #openstack-infra | 19:36 | |
| *** maishsk_ has joined #openstack-infra | 19:37 | |
| *** maishsk_ has quit IRC | 19:37 | |
| *** maishsk has quit IRC | 19:38 | |
| *** maishsk has joined #openstack-infra | 19:39 | |
| *** ilyashakhat_mobi has quit IRC | 19:39 | |
| *** maishsk has quit IRC | 19:39 | |
| *** rcernin has joined #openstack-infra | 19:41 | |
| rcarrillocruz | so pabelanger | 19:41 |
| rcarrillocruz | red herring the sqlite | 19:41 |
| rcarrillocruz | it was other thing you were looking | 19:41 |
| rcarrillocruz | we don't know what those timeouts are coming from, yeah? | 19:41 |
| pabelanger | rcarrillocruz: Ya, I confused which server was running neutron-server | 19:41 |
| rcarrillocruz | or we have some suspicion already | 19:41 |
| rcarrillocruz | ? | 19:41 |
| pabelanger | not sure right now. I did see an error in neutron logs, but didn't match up the timestamps | 19:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add lpmqtt to firehose deployments https://review.openstack.org/363313 | 19:42 |
| rcarrillocruz | ok | 19:42 |
| pabelanger | I'll keep poking at the logs but clarkb suggested reaching out to neutron team, see if they have any suggestions | 19:43 |
| rcarrillocruz | i wip'd the bump change | 19:43 |
| rcarrillocruz | pabelanger: and sorry, catching up since i was out, the default secgroup thing you haven't poked at it or have you? | 19:44 |
| rcarrillocruz | is to start looking at it tomorrow in my morning | 19:44 |
| *** thorst has quit IRC | 19:45 | |
| pabelanger | rcarrillocruz: not yet. I wasn't sure the syntax we should be using | 19:45 |
| pabelanger | rcarrillocruz: Oh, should we remove infracloud mirror server from cloud-launcher right now? Since we manually provisioned it | 19:46 |
| *** ilyashakhat_mobi has joined #openstack-infra | 19:46 | |
| *** sdake has quit IRC | 19:46 | |
| *** smarcet has quit IRC | 19:46 | |
| rcarrillocruz | either way works, os_server is pseudo-idempotent, it won't recreate the server if there's already a server with that name | 19:47 |
| rcarrillocruz | i'll tackle os_server refactor this week | 19:47 |
| *** weshay is now known as weshay_mtg | 19:47 | |
| rcarrillocruz | i'm hoping to have the pre/post actions on the launcher draft based off it for the mid cycle | 19:47 |
| armax | mtreinish: problem is, if we remove it, what happens if you need it? | 19:48 |
| pabelanger | rcarrillocruz: okay, cool | 19:48 |
| *** rcernin has quit IRC | 19:48 | |
| *** pradk has quit IRC | 19:49 | |
| mtreinish | armax: so you log a prettified json blob (which can be quite big sometimes) ~600 times during a dsvm run in the off chance you need it once and it's useful for debug | 19:50 |
| *** ilyashakhat_mobi has quit IRC | 19:50 | |
| mtreinish | it's all about striking a balance, and that definitely feels like it's too far past the line | 19:50 |
| armax | mtreinish: what’s the proverb? it rains when you don’t have an umbrella? | 19:50 |
| *** andrey-mp has joined #openstack-infra | 19:54 | |
| andrey-mp | Hi, can anyone from core check this simple patch? https://review.openstack.org/#/c/357636/ | 19:54 |
| *** tqtran has quit IRC | 19:55 | |
| *** eharney has quit IRC | 19:56 | |
| *** devananda is now known as devananda|travel | 19:57 | |
| *** AnarchyAo has quit IRC | 19:58 | |
| *** asettle has joined #openstack-infra | 19:58 | |
| *** asettle has quit IRC | 19:58 | |
| *** davidlenwell has quit IRC | 19:59 | |
| *** florianf has quit IRC | 20:00 | |
| clarkb | fungi: I poked at the puppet/ansible/kolla logs to see if maybe their deployments had runaway with log generation and collection but they all do much less than devstack does | 20:00 |
| zaro | pleia2: browser load in itself isn't going to be a good test for gerrit. | 20:01 |
| clarkb | but they don't reliably compress all of their files like devstack does | 20:01 |
| *** amitgandhinz has quit IRC | 20:01 | |
| pleia2 | zaro: yeah, it would have to do certain actions | 20:01 |
| zaro | pleia2: the load has to be from the data | 20:01 |
| * pleia2 nods | 20:01 | |
| *** amitgandhinz has joined #openstack-infra | 20:01 | |
| pleia2 | well, I'll make some time to figure out what's out there and touch base with you to see if anything makes sense | 20:01 |
| *** gyee has quit IRC | 20:01 | |
| clarkb | probably worthwhile asking them to update their jobs to deal with compressed logs just because its a simple thing to do | 20:01 |
| zaro | pleia2: sounds good | 20:02 |
| *** ilyashakhat_mobi has joined #openstack-infra | 20:02 | |
| fungi | clarkb: i think you need to look at a sample of job logs from at least a week ago | 20:02 |
| fungi | to factor out the delayed compression | 20:02 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5 https://review.openstack.org/366318 | 20:03 |
| *** sdake has joined #openstack-infra | 20:03 | |
| *** AnarchyAo has joined #openstack-infra | 20:04 | |
| wznoinsk | hi infra | 20:04 |
| wznoinsk | ssh review.openstack.org -p 29418 | 20:04 |
| andrey-mp | clarkb: fungi: pleia2: please check this one-line changeset... https://review.openstack.org/#/c/357636/ | 20:04 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs https://review.openstack.org/365114 | 20:05 |
| pleia2 | andrey-mp: we can't merge that while meetings are going on | 20:05 |
| pleia2 | it restarts the bot, breaks in progress meetings :\ | 20:05 |
| wznoinsk | git review doesn't work over ssh to me and this is what I have when done manually, normally I saw 'non-interactive logons not permitted' AFAIR http://paste.openstack.org/show/567264/, | 20:05 |
| andrey-mp | pleia2: ok, I've got it | 20:06 |
| clarkb | fungi: ya I am saying we shouldnt have to wait for the delayed compression. they should compress upfront like devstack gate does | 20:06 |
| pleia2 | andrey-mp: I'll keep it on my radar for when we have a quite time meeting-wise | 20:06 |
| pleia2 | quiet too | 20:06 |
| andrey-mp | pleia2: thank you! | 20:06 |
| pleia2 | sure thing | 20:06 |
| fungi | clarkb: i agree on that, but it's unlikely to be the cause of our longer-term consumption which is the better thing to get a handle on | 20:06 |
| *** andrey-mp has left #openstack-infra | 20:06 | |
| mordred | wznoinsk: put the -p 29418 before the host | 20:07 |
| clarkb | fungi: yup agreed | 20:07 |
| mordred | wznoinsk: ssh -p 29418 review.openstack.org - the command in your paste is saying "please run thecommnad '-p 29418'" on review.openstack.org | 20:07 |
| *** eharney has joined #openstack-infra | 20:08 | |
| mordred | wznoinsk: nevermind. I lied | 20:08 |
| wznoinsk | mordred: didn't think it would make a diff, it didn't | 20:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Refactor side-effect functions https://review.openstack.org/320225 | 20:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Patch glean.cmd.open for unit testing https://review.openstack.org/320224 | 20:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use mock patching for most functions https://review.openstack.org/318464 | 20:08 |
| mordred | yah. I just suck today | 20:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Use OS_LOG_CAPTURE to get log output https://review.openstack.org/365807 | 20:08 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Add selinux context manager for writing files https://review.openstack.org/304357 | 20:08 |
| wznoinsk | mordred: yeah, but a good guess it would maybe try to run '-p 29418' as a command ;-) | 20:08 |
| mordred | :) | 20:08 |
| ianw | mordred / pabelanger: ^^ i feel like that series is in pretty good shape ... and i'd like to not have to rebase again :) | 20:09 |
| *** piet has quit IRC | 20:09 | |
| clarkb | wznoinsk: mordred that looks like your ssh executable is a bash script that isnt working | 20:10 |
| clarkb | look at what ssh actually is on that machine | 20:10 |
| zigo | fungi: Would you have time to check on this? https://review.openstack.org/365477 | 20:11 |
| zigo | We can't build oslo.cache without (ie: we need to update dogpile.cache, but can't because of missing tag...) | 20:11 |
| *** rockyg has joined #openstack-infra | 20:11 | |
| fungi | zigo: i commented in channel earlier, but you don't want to push upstream tags or upstream commits yourself. this is a use case for our track-upstream option | 20:11 |
| *** davidlenwell has joined #openstack-infra | 20:12 | |
| fungi | we do this with our fork of the gerrit repo, for example. upstream commits are replicated into separate branches and upstream tags are also replicated. then you can just propose a merge for review from those into your packaging branch | 20:13 |
| *** edmondsw has quit IRC | 20:13 | |
| zigo | fungi: In many cases, I need to re-tag upstream tags. For example, dogpile.cache has rel_0_6_2, and we need 0.6.2 instead. Many upstream have the silly habbit to prefix with a "v". Also, for the more openstack-ish case, we need to rename things like 4.0.0.0b2 as 4.0.0_b2, so that we then replace _ by ~. | 20:14 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/reviewday: Make Neutron script pull all Stadium projects https://review.openstack.org/361432 | 20:14 |
| wznoinsk | clarkb: mordred yeah, it looks like something around my NIS configuration on the machine | 20:14 |
| fungi | zigo: in those cases, you would sign the tag, right? i guess what you have in 365477 makes sense. earlier i was reacting more to the request to be able to re-push upstream's unsigned tags into the repo yourself | 20:15 |
| zigo | fungi: If Gerrit mandates signing tags, then I would, but in many cases, it's not necessary, because you can just add a new tag to an existing one, and then it keeps the original signature. | 20:16 |
| fungi | wznoinsk: i bet whatever wrapper you have around the ssh executable is looking for what it thinks is the end of command-line options, and is more strict about parameter ordering than openssh is | 20:16 |
| *** kzaitsev_mb has quit IRC | 20:16 | |
| *** dtardivel has quit IRC | 20:17 | |
| pabelanger | clarkb: have a moment to look at nodepool-builder logs? I'm not sure any images are getting uploaded ATM. Last timestamp: 2016-09-06 18:13:31,560 INFO nodepool.builder: Uploading dib image id: 1014 from /opt/nodepool_dib/1014.qcow2 in ovh-bhs1 | 20:17 |
| *** piet has joined #openstack-infra | 20:17 | |
| zigo | fungi: For example, in the deb-nova repo, look at the tag 13.0.0_b3, and you'll see Doug's pgp signature on that tag, even though he only signed 13.0.0.0b3 | 20:17 |
| clarkb | pabelanger: I think its just slow | 20:18 |
| zigo | fungi: So, will gerrit mandate signing? | 20:18 |
| clarkb | pabelanger: you can strace -fp builderpid to see if its writing data | 20:18 |
| fungi | zigo: we're generally strict about only allowing users to have pushSignedTag permission, which allows signed tags but not plain tags or lightweight tags | 20:19 |
| fungi | still cloning deb-nova to look at your example. i've had to switch to a mobile connection because my isp is offline today | 20:19 |
| pabelanger | clarkb: okay, I see write happening | 20:20 |
| fungi | (and since it's a clone of nova, it takes a while) | 20:20 |
| zigo | fungi: I can show you on a more lightweight repo ! :) | 20:20 |
| zigo | fungi: Any service with a beta release is fine ... | 20:20 |
| sdake | pabelanger isthere a way to override bindep.txt | 20:21 |
| sdake | I see in the script it checks for existence of this file | 20:21 |
| zigo | fungi: Tag 3.0.0_b2 in mistral for example. | 20:21 |
| sdake | (from the zuul output) | 20:21 |
| fungi | zigo: what you're demonstrating there is what's called a "lightweight" tag, that's basically a symlink. presumably you didn't use -a or -s when you ran `git tag` | 20:22 |
| clarkb | sdake: if you dont provide your own bindep.txt we use a fallback so just pr9vide your own file | 20:22 |
| sdake | clarkb just check into the repo then? | 20:22 |
| clarkb | yes | 20:22 |
| sdake | clarkb cool i'll give that a go | 20:22 |
| sdake | clarkb thanks :) | 20:22 |
| sdake | clarkb bindep installing all kinds of stuff we dont need | 20:23 |
| zigo | fungi: So what do you suggest I do then? | 20:23 |
| fungi | sdake: jobs using the "install-distro-packages" builder will use any bindep.txt (or other-requirments.txt) found in the repo for the change that triggered the job, and if there isn't one, it'll use our fallback package list | 20:23 |
| *** mlavalle has joined #openstack-infra | 20:23 | |
| *** kgiusti has left #openstack-infra | 20:23 | |
| *** srobert has joined #openstack-infra | 20:23 | |
| mlavalle | mtreinish: ping | 20:23 |
| fungi | zigo: we strongly recommend against lightweight (and annotated) tags, in favor of signed tags | 20:23 |
| clarkb | sdake: yes the idea is to give you control while installing what we have always installed for you if you dont have a bindep.txt | 20:24 |
| mtreinish | mlavalle: pong | 20:24 |
| fungi | zigo: lightweight tags are better thought of as "aliases" | 20:24 |
| zigo | fungi: Then I'm fine with signed tags. packaging core group will do. | 20:24 |
| fungi | zigo: they don't always refer to other tags. they can refer to any got object | 20:24 |
| sdake | clarkb sounds god i'll give it a go - since it appears our jobs ar einstall-distro-packages | 20:24 |
| fungi | er, git object | 20:24 |
| zigo | git it | 20:25 |
| zigo | er, got | 20:25 |
| zigo | :) | 20:25 |
| anteaya | good | 20:25 |
| mlavalle | mtreinish: hi.... I have a question about logstash / kibana..... Does it capture LOG.debug level messages? | 20:25 |
| *** ilyashakhat_mobi has quit IRC | 20:25 | |
| zigo | fungi: But I need https://review.openstack.org/#/c/365477/ to do signed tags, no? | 20:25 |
| zigo | Or is there a hidden way that I didn't see? | 20:26 |
| mtreinish | mlavalle: no | 20:26 |
| pabelanger | clarkb: looks like nodepool is trying to finish uploading an image to ovh-bhs1. But not sure why we haven't started the other clouds. I wonder if it has to do with my upload attempt this morning. Which is still running | 20:26 |
| mlavalle | mtreinish: yean that was what we were suspecting..... Thanks :-) | 20:26 |
| mlavalle | mtreinish: see you in Barcelona | 20:26 |
| mtreinish | mlavalle: it's too much data and of questionable value. If it's something we need to track it should be important enough for a higher log level | 20:26 |
| clarkb | pabelanger: we have 16 uploaf yhreads and many more uploads | 20:26 |
| mtreinish | mlavalle: heh, yep :) | 20:26 |
| clarkb | pabelanger: so all 16 are likely busy at the moment | 20:26 |
| clarkb | mlavalle: mtreinish ya that is an important thing to remwmber if it is important it shouldnt be debug | 20:27 |
| pabelanger | clarkb: Ya, I would expect that too. But logs and cacti.o.o show different | 20:28 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=982&rra_id=all | 20:28 |
| mlavalle | clarkb, mtreinish: agree, will work aroud it :-) Thanks again | 20:28 |
| pabelanger | no network traffic for 3 hours | 20:28 |
| fungi | zigo: yes, you do. i'm sorry, i'm trying really hard to follow the tc meeting right now and conversations with half a dozen other people are making that hard. can we discuss it later? or can you discuss it with someone who's available right now? | 20:28 |
| clarkb | pabelanger: it may be waitibg on a qcow2 to raw conversion in ovh | 20:28 |
| zigo | fungi: Sure, ping me when you can. | 20:28 |
| clarkb | pabelanger: or waiting for raxto import to glancefrom swift | 20:28 |
| clarkb | neither ofthose will need network traffic | 20:28 |
| pabelanger | we've only uploaded to osic-cloud1, infracloud-vanilla and ovh-bhs1 | 20:29 |
| pabelanger | and osic-cloud1 and infracloud-vanilla are online | 20:29 |
| pabelanger | I think my image-upload task is blocking the rest of the pipe | 20:30 |
| *** salv-orl_ has joined #openstack-infra | 20:30 | |
| clarkb | pabelanger: maybe? | 20:33 |
| *** salv-orlando has quit IRC | 20:33 | |
| clarkb | I see what you are saying thought we did cloud1 then vanilla then the first image for ovh | 20:33 |
| *** thorst has joined #openstack-infra | 20:33 | |
| clarkb | and cloud1 and vanilla are done | 20:33 |
| wznoinsk | fungi: mordred: clarkb: it was something as stupid as "/bin/bash" line on its own in my ~/.cshrc and it was clashing with ProxyCommand /usr/bin/nc ... in my ssh_config | 20:34 |
| clarkb | pabelanger: it also looks like we may have done those earlier uploads in a serialized fashion | 20:34 |
| *** tqtran has joined #openstack-infra | 20:34 | |
| clarkb | pabelanger: you might try running a manual upload command and see if that job gets picked up right away | 20:35 |
| clarkb | pabelanger: I am going to check the gear status | 20:35 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5 https://review.openstack.org/366318 | 20:36 |
| *** akscram has quit IRC | 20:36 | |
| *** ddieterly is now known as ddieterly[away] | 20:37 | |
| clarkb | pabelanger: according to gear status we have all 16 threads in use | 20:37 |
| pabelanger | really? | 20:37 |
| pabelanger | Hmm | 20:37 |
| clarkb | pabelanger: yup it looks like we are still uploading yesterdays images | 20:37 |
| *** akscram has joined #openstack-infra | 20:37 | |
| pabelanger | welp, that might explain it | 20:38 |
| clarkb | but I am trying to get more info to confirm that | 20:38 |
| pabelanger | and we likely didn't abort them before doing todays | 20:38 |
| clarkb | image-upload:1001 1 1 0 is one gear status | 20:39 |
| clarkb | thats a weird one becuse its saying we are uploading it with 1 worker but we don't haev any workers that can run it currently | 20:39 |
| *** thorst has quit IRC | 20:39 | |
| *** jascott1 has quit IRC | 20:40 | |
| clarkb | ya we are supposedly using wto workers to upload 1013 which is yesterday's xenial image | 20:40 |
| *** jascott1 has joined #openstack-infra | 20:40 | |
| * clarkb digs in logs to see if he can find which cloud that is to | 20:41 | |
| *** amitgandhinz has quit IRC | 20:41 | |
| *** Goneri has quit IRC | 20:41 | |
| *** amitgandhinz has joined #openstack-infra | 20:42 | |
| clarkb | looks like rax-dfw and ovh-bhs1 | 20:43 |
| pabelanger | yup | 20:43 |
| pabelanger | I see that now | 20:43 |
| pabelanger | nodepool-builder.log.2016-09-04_09:2016-09-04 16:58:40,155 INFO nodepool.builder: Uploading dib image id: 1001 from /opt/nodepool_dib/1001.qcow2 in ovh-bhs1 | 20:43 |
| pabelanger | 2 days ago | 20:43 |
| clarkb | we may want to consider switching ovh back to raw to see if that helps | 20:44 |
| *** pradk has joined #openstack-infra | 20:44 | |
| pabelanger | sure | 20:44 |
| clarkb | but yes this is our new scaling issue I think | 20:44 |
| clarkb | multiple builders may also help if its jsut a bandwidth issue but I doubt thats the only issue | 20:44 |
| pabelanger | clarkb: do you want to revert raw for just ovh-bhs1 or both? | 20:45 |
| clarkb | I would do both | 20:45 |
| pabelanger | ok | 20:45 |
| clarkb | then we can keep an eye on upload time over the next couple days to see if its still bad | 20:45 |
| *** weshay_mtg is now known as weshay | 20:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Force image-type qcow2 for OVH" https://review.openstack.org/366333 | 20:47 |
| pabelanger | works for me | 20:47 |
| *** roxanagh_ has joined #openstack-infra | 20:47 | |
| matt-borland | Hm, I had to fix something in freezer-web-ui, but I seem to have an invocation error (clearly not related to my change) in several of the gate jobs. | 20:47 |
| *** amitgandhinz has quit IRC | 20:47 | |
| pabelanger | once that lands, let restart nodepool-builder so rest of images get upload as raw | 20:47 |
| openstackgerrit | Merged openstack-infra/shade: Add libffi-dev to bindep.txt https://review.openstack.org/366304 | 20:48 |
| *** amitgandhinz has joined #openstack-infra | 20:48 | |
| clarkb | matt-borland: always link to example logs :) | 20:48 |
| matt-borland | clarkb, my pleasure! http://logs.openstack.org/58/366258/3/check/gate-freezer-web-ui-python27-ubuntu-xenial/c10042b/console.html | 20:48 |
| clarkb | pabelanger: hrm that doesn't need to be image-type: raw? | 20:48 |
| clarkb | pabelanger: I am assuming that the assumption is that we look it up in oscc but I don't knwo if the builder/dib will do that | 20:48 |
| pabelanger | clarkb: I believe that is the default from os-cloud-config, let me double check | 20:48 |
| clarkb | matt-borland: you are using py.test which isn't a thing we currently support so not really able to help | 20:49 |
| clarkb | matt-borland: I'd have to go digging and relearn a bunch of new tools | 20:49 |
| matt-borland | you're using 'you' a little liberally, but sure! | 20:49 |
| pabelanger | err, os-client-config | 20:49 |
| clarkb | matt-borland: that project's test infrastructure | 20:49 |
| matt-borland | clarkb, thank you | 20:50 |
| pabelanger | clarkb: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/vendors/ovh.json#n13 | 20:50 |
| clarkb | matt-borland: basically we have a setup for collecting ocverage and running unittests its not that | 20:50 |
| matt-borland | I'l take it up with whoever owns that project | 20:50 |
| clarkb | pabelanger: right but how does dib know to use that? | 20:50 |
| clarkb | pabelanger: I am going to guess nodepool looks it up and passes it along, just need to double check on that really quick | 20:50 |
| clarkb | yup found it. | 20:51 |
| pabelanger | clarkb: That is a good question, I assume nodepool does the right thing. But we can force it to raw | 20:51 |
| *** roxanagh_ has quit IRC | 20:52 | |
| clarkb | matt-borland: it looks like py.test decided to run no tests for some reason and that was correctly determined to be a failure situation | 20:52 |
| clarkb | pabelanger: nodepool does the right thing. Where it sets image-type in the config it defaults to what oscc says is the format | 20:52 |
| *** jascott1 has quit IRC | 20:52 | |
| matt-borland | clarkb, I'll take a look at it, just wondered if there was an 'aha!' moment | 20:52 |
| zaro | clarkb: was wondering whether you want to A+ this one for next Gerrit restart? | 20:52 |
| zaro | https://review.openstack.org/#/c/364424/ | 20:53 |
| *** jascott1 has joined #openstack-infra | 20:53 | |
| clarkb | zaro: maybe we can split it so that we can do review-dev now make sure that build works then do review.o.o? Since I think what is on review-dev isn't exactly the same as proposed there? | 20:54 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 20:54 |
| clarkb | mtreinish: the way i would do it is create a central devstack change with a bunch of depends on. Iterate on that until you can get that change to run with python3 enabled by default and no depends on. Then add a new job that is "run against python3" or just stop running with python2 | 20:58 |
| clarkb | (likely need to continue running with python2 also hence the new job) | 20:58 |
| *** Hal1 has joined #openstack-infra | 20:58 | |
| clarkb | mtreinish: but thats a really low cost from the CI/infrastructure side that allows devs to iterate quickly on making things work | 20:58 |
| mtreinish | clarkb: sure there is a simple path forward, my point was it's about someone owning doing that test patch and wrangling people to look at results and figure out failures | 20:59 |
| zaro | clarkb: actually you can't approve yet. there's a bug that will kill gerrit on javamelody updates. we'll need to do it at the time of a gerrit restart. | 20:59 |
| mtreinish | which if there isn't someone driving it just doesn't happen | 20:59 |
| clarkb | this is the same approach I would like to see with tls and devstack except we shouldn't make a new job for tls we should just run tls everywhere always | 20:59 |
| *** ddieterly[away] is now known as ddieterly | 20:59 | |
| clarkb | mtreinish: oh ok it definirely read more like "its too hard to juggle this in gerrit"? | 21:00 |
| fungi | clarkb: mtreinish: any idea if our py3k wheel mirror is in better shape now? | 21:00 |
| zaro | clarkb: btw, i did manually install on review-dev and it did work fine. i'll mention in commit message. | 21:00 |
| clarkb | fungi: I didn't realize there was something wrong with it | 21:00 |
| clarkb | zaro: ok just make sure you WIP the change for review.o.o and leave a note about that bug | 21:01 |
| clarkb | zaro: so that we update it correctly | 21:01 |
| mtreinish | clarkb: no it wasn't a tooling issue. I just brought up d-g and devstack as examples to show that there isn't a clear owner of the effort | 21:01 |
| *** mriedem has joined #openstack-infra | 21:01 | |
| *** raildo has quit IRC | 21:01 | |
| mtreinish | fungi: no idea | 21:01 |
| fungi | clarkb: well, we weren't building one for a while, then it was added but it made the wheel build job run too long and hit our timeout so it got disabled again. there was also some concern that we were unnecessarily re-building universal wheels in each pass | 21:02 |
| *** sdake has quit IRC | 21:02 | |
| mtreinish | fungi: I wasn't aware there was an issue with the py3 mirror before | 21:02 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| clarkb | fungi: huh I completely missed all that. FWIW the way our wheel mirroring works we have to at least copy all of the universal wheels | 21:02 |
| fungi | looks like we do independent jobs for python2 and python3 wheels now | 21:02 |
| clarkb | since we bucket by the unique tuple of distro,release,cpuarch | 21:03 |
| clarkb | another optimization may be to only make wheels for things not already wheeled in pypi | 21:03 |
| clarkb | though those are very cheap aiui since we just grab the wheel for pypi then copy into our mirror | 21:03 |
| fungi | right | 21:04 |
| fungi | i'm wondering if adding our wheel mirror to the index list for the python3 wheel build jobs would make sense | 21:04 |
| clarkb | er that tuple has a 4th section, python version | 21:04 |
| fungi | so that it would reuse the universal wheels we built with the python2 pass | 21:04 |
| *** derekh has quit IRC | 21:05 | |
| clarkb | fungi: yes, I think that would sovle that particular issue since pip should be smart enough to not use the python2 only wheels there | 21:05 |
| mordred | that souds non-crazy to me | 21:05 |
| *** jkilpatr has quit IRC | 21:05 | |
| clarkb | so the trickle down effect is pypi -> python2 wheels -> python3 wheels | 21:05 |
| *** sambetts is now known as sambetts|afk | 21:06 | |
| fungi | clarkb: challenge i guess would be to make it use the universal wheels already built under python 2 but not the previously build python 3 wheels | 21:06 |
| clarkb | mtreinish: also it might just take someone pushing that change getting it to break, then responding to the goal ml post and saying "heres the thing who is going to work on it now" | 21:07 |
| fungi | er, previously built | 21:07 |
| *** nt has quit IRC | 21:07 | |
| mtreinish | clarkb: that already was done | 21:07 |
| * mtreinish digs up the patch | 21:07 | |
| mtreinish | well for py34 | 21:07 |
| clarkb | mtreinish: was it advertised? I certainly never heard about it | 21:07 |
| mtreinish | that I don't know | 21:08 |
| clarkb | fungi: I think that also isn't an issue bceause using old python3 wheels is probably file as long as the package version didn't change and new version will get rebuilt | 21:08 |
| mtreinish | clarkb: https://review.openstack.org/#/c/331224/ | 21:08 |
| clarkb | mtreinish: ya I would advertise that directly and also don't do it the way its done there and use depends on | 21:08 |
| fungi | clarkb: one thing not reusing the previous wheels gets us is builds against newer dependency libraries | 21:09 |
| mtreinish | well it's dated at this point | 21:09 |
| fungi | anteaya: catching up on comments at the end of the tc meeting, but in gerrit a "change" has one or more "patch sets" (not change sets) | 21:09 |
| mtreinish | clarkb: I also have no idea who the submitter was | 21:09 |
| clarkb | fungi: ya but in theory those are relatively stable on our distros right? | 21:09 |
| clarkb | mtreinish: ya that patch is completely subverting how this should actually be done | 21:09 |
| clarkb | you want error on clone = true and use depends on | 21:09 |
| mtreinish | clarkb: sure, I didn't actually look at tbh. I just saw it linked in a tempest py3 fix at one point | 21:10 |
| fungi | clarkb: relatively. i guess you really only care about things like soversion bumps/symbol changes which _shouldn't_ be the case on lts distros at least | 21:10 |
| mtreinish | and I've seen patches doing the same thing in the past | 21:10 |
| clarkb | fungi: yup | 21:10 |
| fungi | though there are exceptions | 21:10 |
| clarkb | fungi: though it wouldn't surprise me if it happens every once in a while | 21:10 |
| fungi | indeed, and it does | 21:10 |
| mtreinish | clarkb: but that's my point is someone needs to volunteer to drive it. Otherwise it kinda just sits there | 21:10 |
| fungi | clarkb: just not often | 21:10 |
| mtreinish | and it's not like you or I have the bandwidth to deal with it | 21:11 |
| clarkb | mtreinish: yes but I think thats how you get volunteers | 21:11 |
| *** mmedvede has quit IRC | 21:11 | |
| fungi | so maybe infrequent enough for us to ignore until it bites us | 21:11 |
| clarkb | mtreinish: people don't know where to start but if you hand them a template for the work that needs to be done that gets them halfway there I bet you would have at least a couple volunteers | 21:11 |
| *** mmedvede has joined #openstack-infra | 21:11 | |
| clarkb | and it looks like 2 months ago that got really close | 21:11 |
| zigo | mordred: We need the PTL's +2: https://review.openstack.org/366318 | 21:12 |
| fungi | i need to run out and get something to eat, then come back and see if my isp has recovered so i can try to generate this last batch of summit invites | 21:12 |
| clarkb | grenade isn't going to work due to previous release and centos doesn't have python3 | 21:12 |
| fungi | bbiaw | 21:12 |
| mtreinish | clarkb: right but it assumes someone is going to push that template. I'm saying I don't want to be the one doing that, it needs someone to get things started | 21:12 |
| mtreinish | clarkb: it's step 0 vs step 1 | 21:12 |
| *** rhallisey has quit IRC | 21:12 | |
| clarkb | I mean its 5 minutes of work if you want I can update taht change and send an email | 21:12 |
| *** xarses has quit IRC | 21:14 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 21:14 |
| *** inc0 has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** dprince has quit IRC | 21:16 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 1/5 https://review.openstack.org/366318 | 21:16 |
| *** bin_ has quit IRC | 21:17 | |
| *** tonytan_brb has joined #openstack-infra | 21:17 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review-dev.o.o https://review.openstack.org/366348 | 21:17 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 21:18 |
| *** matt-borland has quit IRC | 21:18 | |
| *** rossella_s has quit IRC | 21:18 | |
| *** rossella_s has joined #openstack-infra | 21:19 | |
| *** tonytan4ever has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack/diskimage-builder: Revert "Revert "Pre-install pip/virtualenv packages"" https://review.openstack.org/341934 | 21:21 |
| *** Apoorva has joined #openstack-infra | 21:21 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review-dev.o.o https://review.openstack.org/366348 | 21:22 |
| *** dimtruck is now known as zz_dimtruck | 21:23 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin on review.o.o https://review.openstack.org/364424 | 21:24 |
| mordred | Shrews, infra-root: the earlier release of shade failed because of missing libffi so the tarball couldn't get built so it couldn't get released | 21:25 |
| mordred | that is now fixed | 21:25 |
| mordred | so I'm going ot tag/push 1.12.1 so that we have a release that exists | 21:25 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Force image-type qcow2 for OVH" https://review.openstack.org/366333 | 21:25 |
| clarkb | mordred: ok | 21:25 |
| zaro | clarkb: ^ done, split em up for you | 21:25 |
| clarkb | zaro: tyty don't forget to WIP the review.o.o one with a comment about merging it during planned outage | 21:26 |
| zaro | WIPed, warning is in the commit message | 21:27 |
| clarkb | zaro: perfect thanks | 21:27 |
| *** notnownikki has quit IRC | 21:28 | |
| openstackgerrit | Alexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack https://review.openstack.org/366154 | 21:28 |
| openstackgerrit | Alexey Khivin proposed openstack-infra/project-config: Add jenkins-plugins-for-murano project to Openstack https://review.openstack.org/366154 | 21:29 |
| *** jcoufal_ has quit IRC | 21:29 | |
| pabelanger | clarkb: restarted nodepool-builder to pickup qcow2 revert. Raw image uploads will start up tomorrow | 21:29 |
| pabelanger | already looking better | 21:30 |
| *** ddieterly is now known as ddieterly[away] | 21:30 | |
| clarkb | pabelanger: ok, probably want to upload rax xenial images manually now? | 21:30 |
| pabelanger | clarkb: they are still in the queue actually | 21:31 |
| pabelanger | still 51 images to upload | 21:31 |
| clarkb | oh cool for some reason I thought it dropped all of them when it restarted | 21:31 |
| pabelanger | I'll check back here in a bit, need to run some errands. | 21:31 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 3/5 https://review.openstack.org/366352 | 21:31 |
| *** rfolco has quit IRC | 21:32 | |
| clarkb | fungi: I have confirmed that the weekly compression run seems to be working for console logs and is compressing those. Now to check if its catching the puppet/kolla/ansible logs too | 21:32 |
| *** dizquierdo has joined #openstack-infra | 21:33 | |
| *** rtheis has quit IRC | 21:33 | |
| clarkb | fungi: it looks like it skips over all the *.log files http://logs.openstack.org/31/325631/8/check/gate-kolla-dsvm-deploy-centos-source/6953633/logs/kolla/nova/ | 21:34 |
| *** srobert has quit IRC | 21:34 | |
| clarkb | fungi: maybe we should add them to the find name patterns? | 21:34 |
| *** rhallisey has joined #openstack-infra | 21:34 | |
| zigo | mordred: I *really* need your PTL's approval: https://review.openstack.org/366318 | 21:35 |
| zigo | (sorry to insist...) | 21:36 |
| *** ddieterly[away] is now known as ddieterly | 21:36 | |
| zigo | I got 5 like that, probably they will merge conflict... | 21:37 |
| zigo | Then if I did right, all repo will be there. | 21:37 |
| *** thorst has joined #openstack-infra | 21:37 | |
| *** adrian_otto has quit IRC | 21:38 | |
| mordred | zigo: fine by me - next time, fwiw, it's easier if you put the same topic for each patch - that way it's easy to find the sibling patches | 21:38 |
| mordred | oh - you did. great | 21:38 |
| zigo | :) | 21:38 |
| zigo | mordred: As I wrote, I expect merge conflicts, since it's touching close enough lines... | 21:39 |
| mordred | yah | 21:39 |
| zigo | mordred: So let's process them one by one. | 21:39 |
| zigo | Thanks btw. | 21:39 |
| mordred | zigo: you've got a test failure on the governance patch | 21:39 |
| clarkb | also I have caved and am using du on the logs server | 21:39 |
| clarkb | since nothing really bad was showing up via web crawling | 21:39 |
| *** annegentle has quit IRC | 21:39 | |
| zigo | the Add deb-python-{keystoneauth1,requestsexceptions} ? | 21:40 |
| mordred | zigo: yah - I don't know if that's important or not - but I left a +1 there | 21:41 |
| clarkb | https://review.openstack.org/#/c/361500/ is using 1.8GB to store all of its logs for check and gate | 21:41 |
| zigo | Fixed. | 21:41 |
| mordred | clarkb: that's impressive | 21:41 |
| clarkb | there is 29MB of coverage data for every neutron coverage run | 21:43 |
| clarkb | we don't appear to compress any of it | 21:43 |
| clarkb | (probably because thats output of a tool and it won't host properly if not compressed | 21:43 |
| *** pradk has quit IRC | 21:43 | |
| *** adriant has joined #openstack-infra | 21:44 | |
| clarkb | devstack+tempset runs seem to need about 100MB per run on the high end for multinode and ~80MB on the low end | 21:45 |
| clarkb | with some variance in the middle | 21:45 |
| *** thorst has quit IRC | 21:45 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:47 | |
| clarkb | jackpot http://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/ used 1.1GB of disk just for that one job | 21:47 |
| clarkb | now run it a couple times per patchset then have more than one patchset | 21:47 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 21:48 |
| clarkb | odyssey4me: prometheanfire ^ is that you all? Can we cut back on the amount of logs being grabbed there. Thats about 10x what a devstack run needs | 21:48 |
| prometheanfire | clarkb: hi? | 21:49 |
| clarkb | its half a gig of uncompressed syslog and user.log | 21:49 |
| clarkb | prometheanfire: you hack on the openstack ansible things right? | 21:49 |
| prometheanfire | not too much :( | 21:49 |
| clarkb | sorry thats each half a gig for user.log and half a gig for syslog | 21:49 |
| clarkb | prometheanfire: ah ok | 21:49 |
| prometheanfire | that is a ton | 21:50 |
| clarkb | ya so I think we need to do two things here. 1) make sure people are precompressing before upload in their jobs like devstack does but also not trying to store all the data ever :) | 21:50 |
| clarkb | http://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/logs/host/ both files live there | 21:51 |
| *** mrjk has quit IRC | 21:51 | |
| * clarkb continues scanning the filesystem to find other places we can trim | 21:51 | |
| *** Sukhdev has quit IRC | 21:51 | |
| *** stevemar has quit IRC | 21:52 | |
| clarkb | ha a keystone change with 27 patchsets uses less disk than the openstack ansible change does with 2 tested patchsetse | 21:52 |
| mordred | clarkb: odyssey4me is the right human for openstack-ansible | 21:54 |
| clarkb | I am also only look at the 36XXXX series changes otherwise du would be too slow | 21:55 |
| *** Sukhdev has joined #openstack-infra | 21:55 | |
| clarkb | I figure better to look at recent things as far as how to improve for the future | 21:55 |
| *** mrjk has joined #openstack-infra | 21:56 | |
| clarkb | mordred: I will have to dig up their bug tracker and file a bug | 21:56 |
| clarkb | neutron changes are regularkly showing up in the higher end up disk usage though. Related to realtive size of neutron logs and number of jobs they run per patchset | 21:57 |
| clarkb | they are the distributed scale out disk users and openstack-ansible is the scale up disk user in 2 files | 21:57 |
| *** Sukhdev has quit IRC | 21:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 22:00 |
| *** askb has joined #openstack-infra | 22:00 | |
| *** rhallisey has quit IRC | 22:00 | |
| *** ddieterly is now known as ddieterly[away] | 22:00 | |
| mordred | clarkb: I just assume odyssey4me will see things in scrollback :) | 22:00 |
| anteaya | fungi: ah thank you, sorry I mis-communicated that, I was sure it was change sets | 22:01 |
| *** zz_dimtruck is now known as dimtruck | 22:02 | |
| mordred | ianw: in answer to your question from the mysql user creation patch - python shows mysql warnings by default, but the mysql command line does not - so I doubt we're seeing the warning there | 22:02 |
| *** annegentle has joined #openstack-infra | 22:02 | |
| ianw | mordred: ahh, that would explain why nobody ever noticed | 22:04 |
| *** cap_ has joined #openstack-infra | 22:05 | |
| *** asettle has joined #openstack-infra | 22:05 | |
| *** gyee has joined #openstack-infra | 22:05 | |
| *** sarob has joined #openstack-infra | 22:05 | |
| mordred | yah - I do think we can update it just for completeness | 22:05 |
| *** asettle has quit IRC | 22:06 | |
| *** asettle has joined #openstack-infra | 22:07 | |
| *** cap_ has quit IRC | 22:07 | |
| *** Sukhdev has joined #openstack-infra | 22:09 | |
| clarkb | according to logstash the last 7 days we ran those ansible jobs ~100 times | 22:09 |
| clarkb | so every 7 days of ansible jobs is worth about 100GB | 22:09 |
| mordred | ianw: oh bother. it's new syntax as of mysql 5.7.6 - so we really can't do the project config macro until we're on xenial | 22:09 |
| *** hashar has quit IRC | 22:10 | |
| *** Goneri has joined #openstack-infra | 22:10 | |
| clarkb | assuming that trend holds and we have actually run those jobs at that rate for the last 2 months that would account for about 5TB of our disk use? | 22:11 |
| clarkb | someone double check on my maths | 22:11 |
| mordred | ianw: I set -W on the nodepool change so that I can poke further | 22:11 |
| ianw | mordred: heh, ok. let the yak shaving begin :) | 22:11 |
| mordred | ianw: /me cries | 22:11 |
| *** ddieterly[away] is now known as ddieterly | 22:12 | |
| *** baoli has quit IRC | 22:12 | |
| mordred | clarkb: if one week is 100GB, then 2 months would be 800G not 5TB | 22:12 |
| mordred | clarkb: unless I'm missing a math | 22:12 |
| clarkb | mordred: ya I derped the maths :) | 22:13 |
| mordred | clarkb: if _one_ day is 100GB, then I would agree | 22:13 |
| *** baoli has joined #openstack-infra | 22:13 | |
| clarkb | ya thats how I derped it | 22:13 |
| clarkb | multiplied by 7 | 22:13 |
| clarkb | instead of divide | 22:13 |
| mordred | sometimes just multiplying by 7 is the right choice | 22:13 |
| clarkb | ok so thats still a nontrivial amount of our log usage | 22:13 |
| clarkb | we are at ~9TB now and almost 1TB is potentially just openstack ansible jobs | 22:13 |
| *** kzaitsev_mb has quit IRC | 22:14 | |
| clarkb | basically to even get close to per change disk usage of openstack ansible I need to find neutron changes with a bunch of patchsets that have each run the tests over and over | 22:14 |
| clarkb | https://review.openstack.org/#/c/361508/ is a tempest chagne using about half the disk of 363307 which is an osa change. osa has half as many patchsets as tempest | 22:15 |
| *** esberglu has quit IRC | 22:15 | |
| * clarkb continues to run du to see if there are any other outliers | 22:16 | |
| *** fguillot_ has joined #openstack-infra | 22:16 | |
| *** priteau has quit IRC | 22:16 | |
| mtreinish | clarkb: how many jobs does osa run? Tempest runs a lot | 22:16 |
| clarkb | mtreinish: 4 in check and 3 in gate. Its far fewer. But at 1.1GB per job... | 22:17 |
| clarkb | mtreinish: basically they add up very quickly in osa land whereas with tempest/neutron its a lot of jobs that slowly accumulate | 22:17 |
| *** xyang1 has quit IRC | 22:17 | |
| *** ddieterly is now known as ddieterly[away] | 22:17 | |
| clarkb | and the vast majority of other changes seem to come in under 100MB total | 22:18 |
| * clarkb looks at older changes just to make sure its not because these changes are so new that we see this behavior | 22:18 | |
| *** eharney has quit IRC | 22:18 | |
| mtreinish | clarkb: ah, ok. Yeah tempest has like ~24 check and ~13 gate | 22:21 |
| *** esberglu has joined #openstack-infra | 22:21 | |
| *** mdrabe has quit IRC | 22:22 | |
| clarkb | I think we can likely improve both. osa is easy because its basically 2 fiels that need to be dealt with but we should also get them (and kolla and puppet et al) to compress their logs | 22:22 |
| *** vhosakot has quit IRC | 22:22 | |
| *** priteau has joined #openstack-infra | 22:22 | |
| *** piet has quit IRC | 22:22 | |
| clarkb | then for the dsvm jobs I think its just a matter of keeping the log files useful and sane which has sort of been an ongoing project for the last few years | 22:22 |
| mtreinish | clarkb: well not really the last year or 2. That effort kinda died off unfortunately | 22:23 |
| mtreinish | it's the same fight over and over again | 22:23 |
| *** stevemar has joined #openstack-infra | 22:24 | |
| *** ddieterly[away] is now known as ddieterly | 22:24 | |
| *** esberglu has quit IRC | 22:26 | |
| *** stevemar has quit IRC | 22:26 | |
| ianw | clarkb: i'll have to look at ansible, i remember we went through getting good logs with puppet and came up with http://git.openstack.org/cgit/openstack/puppet-openstack-integration/tree/copy_logs.sh | 22:26 |
| *** priteau has quit IRC | 22:26 | |
| *** stevemar has joined #openstack-infra | 22:26 | |
| *** Swanson has joined #openstack-infra | 22:27 | |
| *** Goneri has quit IRC | 22:29 | |
| ianw | http://logs.openstack.org/01/365601/3/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial-nv/ea577a3/logs/host/syslog ... 506mb? | 22:29 |
| *** Hal1 has quit IRC | 22:29 | |
| clarkb | ianw: the puppet logs are also not coming through all compressed like they should. We should problably update that script to also gzip .logs | 22:29 |
| clarkb | ianw: yes and userl.log is about that big too | 22:30 |
| clarkb | ianw: so between two files we basically hit 1GB | 22:30 |
| *** sarob has quit IRC | 22:30 | |
| *** stevemar has quit IRC | 22:31 | |
| ianw | clarkb: audispd: ... it's like putting in about 3 lines for every syscall | 22:31 |
| clarkb | ianw: seems consistent across xenial and trusty too so not something release specific | 22:32 |
| clarkb | also syslog for dsvm jobs is only 14MB uncompressed | 22:32 |
| ianw | i wonder ... i had to fix installing auditd on fedora-minimal to stop log spew | 22:32 |
| *** krtaylor has joined #openstack-infra | 22:32 | |
| clarkb | https://review.openstack.org/#/c/352610/ a nova change with 4 patchsets hit 12GB my biggest change so far | 22:33 |
| * clarkb digs into that one | 22:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:34 | |
| *** dimtruck is now known as zz_dimtruck | 22:34 | |
| *** berendt has quit IRC | 22:35 | |
| clarkb | hrm that ran the neutron multinode test multiple times on ps4 to the tune of about 1GB each | 22:35 |
| clarkb | oh tahts the 1GB keystone logs | 22:35 |
| ianw | https://access.redhat.com/solutions/2380591 -> "Turn off audispd syslog plugin in the /etc/audisp/plugins.d/syslog.conf file:" | 22:35 |
| clarkb | fungi: ^ was keystone addressed for that? | 22:35 |
| *** sarob has joined #openstack-infra | 22:35 | |
| *** roxanagh_ has joined #openstack-infra | 22:35 | |
| mtreinish | clarkb: yeah that change was reverted | 22:36 |
| *** rajinir has joined #openstack-infra | 22:36 | |
| clarkb | mtreinish: ok so probably we are still hurting from that but the underlying issue is corrected | 22:36 |
| mtreinish | clarkb: https://review.openstack.org/#/c/356010/ | 22:36 |
| ianw | clarkb: and reading http://docs.openstack.org/developer/openstack-ansible-security/controls-cat1.html -> V-38471: The system must forward audit records to the syslog service. | 22:37 |
| *** vhosakot has joined #openstack-infra | 22:37 | |
| clarkb | mtreinish: and doesn't look like the original was backported so we don't need revert backports. Cool /me will ignore that for now | 22:38 |
| Swanson | Pushed up a patch https://review.openstack.org/361453 Trying to use it with dev stack via CINDER_BRANCH=refs/changes/53/361453/7 and getting a "couldn't find remote ref" fatal. Anyone seeing this? | 22:39 |
| Swanson | Previous ref still works. | 22:39 |
| *** yamahata has quit IRC | 22:40 | |
| *** roxanagh_ has quit IRC | 22:40 | |
| clarkb | Swanson: what remote are you using? | 22:40 |
| *** fguillot_ has quit IRC | 22:40 | |
| *** fguillot_ has joined #openstack-infra | 22:41 | |
| Swanson | clarkb, looks like git://git.openstack.org/openstack/cinder.git | 22:42 |
| *** ddieterly has quit IRC | 22:43 | |
| *** thorst has joined #openstack-infra | 22:43 | |
| *** rockyg has quit IRC | 22:44 | |
| clarkb | Swanson: looks like there may be a backlog syncing to the mirrors | 22:44 |
| Swanson | clarkb, I'll give it a while and try again. Thanks! | 22:45 |
| *** annegentle has quit IRC | 22:46 | |
| Swanson | clarkb, synced already. Thanks! | 22:46 |
| *** nt has joined #openstack-infra | 22:46 | |
| clarkb | Swanson: I am not sure what caused it but looks like its currently getting through it | 22:46 |
| clarkb | went from ~580 tasks to 27 in the last couple minutes | 22:47 |
| *** zz_dimtruck is now known as dimtruck | 22:47 | |
| Swanson | Glad it is working now. | 22:47 |
| *** thorst has quit IRC | 22:49 | |
| *** thorst has joined #openstack-infra | 22:49 | |
| *** yamahata has joined #openstack-infra | 22:49 | |
| *** mlavalle has left #openstack-infra | 22:52 | |
| *** rbrndt has quit IRC | 22:53 | |
| *** akshai has quit IRC | 22:53 | |
| *** asettle has quit IRC | 22:54 | |
| clarkb | ok from what I can tell digging around the log server the two issues we have as far as crazy disk utilization go are the osa jobs with their two half gig files and the leftovers from the apache for keystone logs being 1GB each | 22:54 |
| clarkb | when a normal dsvm run clocks in at most 100MB those that run at 1GB add up quickly | 22:55 |
| *** rlandy is now known as rlandy|bbl | 22:55 | |
| *** asettle has joined #openstack-infra | 22:57 | |
| *** asettle has quit IRC | 22:57 | |
| *** vhosakot has quit IRC | 22:57 | |
| EmilienM | I'm probably late in the party but anyone aware about pep8 job failures ? http://logs.openstack.org/87/366287/3/check/gate-python-tripleoclient-pep8-ubuntu-xenial/6bd1710/console.html#_2016-09-06_21_01_26_599382 | 22:58 |
| EmilienM | fatal error: ffi.h: No such file or directory | 22:58 |
| *** thiagolib has quit IRC | 22:58 | |
| clarkb | EmilienM: you need to add libffi-dev to your bindep.txt | 22:58 |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/project-config: Add py3 jobs for fuel-ccp-entrypoint repo https://review.openstack.org/366412 | 22:58 |
| *** baoli has quit IRC | 22:58 | |
| EmilienM | ok | 22:58 |
| clarkb | EmilienM: problem is related to some transient dep along the way somewhere that now depends on libffi-dev at build time | 22:58 |
| clarkb | EmilienM: I didn't bother to look up which dep it was but project-config and shade were both affected | 22:59 |
| EmilienM | clarkb: thx | 22:59 |
| clarkb | EmilienM: I think the package is libffi-dev for dpkg and libffi-devel for rpm | 22:59 |
| EmilienM | yeah | 22:59 |
| pabelanger | EmilienM: clarkb: or wait until we build a cffi wheel | 23:00 |
| pabelanger | but think we are capped right now | 23:00 |
| clarkb | pabelanger: you should still reflect what you need to build from scratch imo | 23:00 |
| pabelanger | clarkb: agreed | 23:00 |
| pabelanger | just explains how it worked until today | 23:00 |
| *** spzala has quit IRC | 23:05 | |
| clarkb | ianw: odyssey4me https://bugs.launchpad.net/openstack-ansible/+bug/1620849 there is a bug that can be used to fix this | 23:06 |
| openstack | Launchpad bug 1620849 in openstack-ansible "openstack ansible test jobs are logging excessive amounts of data" [Undecided,New] | 23:06 |
| *** krtaylor has quit IRC | 23:06 | |
| *** cardeois has quit IRC | 23:07 | |
| *** fguillot_ has quit IRC | 23:07 | |
| *** dizquierdo has quit IRC | 23:07 | |
| *** stevemar has joined #openstack-infra | 23:07 | |
| *** fguillot_ has joined #openstack-infra | 23:08 | |
| *** asettle has joined #openstack-infra | 23:09 | |
| *** asettle has quit IRC | 23:09 | |
| *** sdake has joined #openstack-infra | 23:10 | |
| fungi | clarkb: EmilienM: the libffi surprise is that cffi released a new version on pypi today so lots of jobs found no prebuilt wheel, fell back on trying to build from the sdist tarball, and then discovered that those projects hadn't put libffi-dev packages in their bindep.txt | 23:12 |
| EmilienM | fungi: right | 23:12 |
| clarkb | fungi: ah gotcha | 23:12 |
| fungi | basically, our wheel mirrors and out bindep rollout happened in parallel... the two hadn't taken one another into account | 23:13 |
| fungi | so bindep implementation assumed projects with missing build deps would fail self-testing bindep.txt additions, but then we added a wheel mirror while that was going on so we no longer actually test that jobs can build all their dependency sdists consistently | 23:13 |
| fungi | (because doing that takes lots of extra time, so we pre-build wheels elsewhere) | 23:14 |
| *** adrian_otto has joined #openstack-infra | 23:15 | |
| pabelanger | so, a test we could add is, if bindep.txt is modified, remove wheel mirrors from node, then try to build a wheel. If passes, great, if fails then you are missing a dependency | 23:16 |
| clarkb | or just always have a job that doesn't use the wheel mirror | 23:16 |
| fungi | i'm not a fan of the "if bindep.txt is modified" part | 23:16 |
| fungi | jobs dependent on which files have changed make me nervous | 23:17 |
| fungi | it's too easy to fail to run them when you need them | 23:17 |
| clarkb | especially when we are due for a bindep.txt filename change ina bout 3 months | 23:17 |
| pabelanger | Right, it could run every patch | 23:17 |
| clarkb | >_> | 23:17 |
| pabelanger | I said could | 23:17 |
| *** hongbin has quit IRC | 23:17 | |
| fungi | but yeah, a job that doesn't use wheel mirrors would be a sane addition. it's a question of whether there's an existing job that's a good candidate for it | 23:18 |
| pabelanger | hopefully once zuulv3 lands, we can rework out static wheel builders | 23:18 |
| fungi | given that our "pep8" macro already forces an sdist of the project, that might be a good one to overload | 23:18 |
| pabelanger | Ya, that could be a good fit | 23:19 |
| clarkb | ya I like that one because its a simple job to run locally and its relatively quick | 23:19 |
| *** markvoelker has quit IRC | 23:19 | |
| clarkb | that way its easier to reproduce the "why does thsi random job fail" fails locally | 23:19 |
| fungi | right, so the overhead from the job payload is small, making it likely to still complete when or before unit tests do | 23:19 |
| pabelanger | is there a way not to download wheels with pip? | 23:19 |
| pabelanger | while keeping our mirror? | 23:19 |
| clarkb | pabelanger: --no-use-wheel | 23:20 |
| pabelanger | nice | 23:20 |
| pabelanger | going to test it out | 23:20 |
| clarkb | oh thats deprecated should use --no-binary instead | 23:20 |
| clarkb | but looks like they do roughly the same thing | 23:20 |
| pabelanger | k | 23:20 |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/project-config: Add py3 jobs for fuel-ccp-entrypoint repo https://review.openstack.org/366412 | 23:20 |
| *** jamesdenton has joined #openstack-infra | 23:21 | |
| *** xarses has joined #openstack-infra | 23:22 | |
| clarkb | fungi: wiki-dev01.openstack.org is sending mail that pabelanger's MX is rejecting beacuse we lack DNS records for it? | 23:22 |
| *** akshai has joined #openstack-infra | 23:22 | |
| fungi | that's one i'm deleting and re-launching so wasn't planning to put in dns until it's working | 23:23 |
| clarkb | fungi: gotcha | 23:23 |
| fungi | right now fussing over puppet errors from vcsrepo | 23:23 |
| fungi | was hoping to continue on it today, but that hasn't happened | 23:23 |
| *** gildub has joined #openstack-infra | 23:24 | |
| clarkb | no problem, just wasn't sure if taht was an oversight | 23:24 |
| mordred | clarkb, pabelanger, fungi: I like the idea of the pep8 job skipping the wheel mirror - I think it's a good hack | 23:25 |
| *** jamesdenton has quit IRC | 23:26 | |
| *** mriedem has quit IRC | 23:30 | |
| *** mriedem has joined #openstack-infra | 23:32 | |
| *** sarob has quit IRC | 23:32 | |
| *** dimtruck is now known as zz_dimtruck | 23:32 | |
| openstackgerrit | Merged openstack/diskimage-builder: fix systemd resource deadlock https://review.openstack.org/365797 | 23:33 |
| *** akshai has quit IRC | 23:33 | |
| *** bapalm has quit IRC | 23:35 | |
| pabelanger | mordred: fungi: clarkb: Ya, just tested locally and appears to work. | 23:37 |
| mordred | pabelanger: woot | 23:38 |
| *** dingyichen has joined #openstack-infra | 23:38 | |
| *** Sukhdev has quit IRC | 23:40 | |
| *** bapalm has joined #openstack-infra | 23:41 | |
| *** fguillot_ has quit IRC | 23:42 | |
| *** sarob has joined #openstack-infra | 23:46 | |
| *** zz_dimtruck is now known as dimtruck | 23:46 | |
| clarkb | looks like tripleo jobs are copying /var/log twice... | 23:48 |
| clarkb | once in a tarball and again not in a tarball | 23:48 |
| clarkb | pabelanger: ^ maybe you can poke someon about that? gate-tripleo-ci-centos-7-ovb-ha-liberty is job where I noticed it not sure if it affects newer jobs too | 23:49 |
| *** Jeffrey4l has joined #openstack-infra | 23:49 | |
| pabelanger | clarkb: sure | 23:49 |
| dstufft | Does openstack support 2.6 | 23:50 |
| dstufft | Python 2.6 that is | 23:50 |
| clarkb | dstufft: no, its no longer a tested platform | 23:50 |
| dstufft | ok | 23:50 |
| fungi | dstufft: i almost chimed in on the distutils-sig thread about that | 23:50 |
| * clarkb wonders whats going on | 23:51 | |
| clarkb | pabelanger: tyty | 23:51 |
| dstufft | Unless a pip core speaks out I'm planning to kill pip's support for it in pip 10 (with a red warning in pip 9) | 23:51 |
| fungi | clarkb: you should subscribe to teh ml ;) | 23:51 |
| dstufft | wanted to warn y'all if you supported 2.6 still | 23:51 |
| clarkb | dstufft: ah | 23:51 |
| fungi | we dropped support for python 2.6 when we could successfully test things on rhel 7 | 23:51 |
| clarkb | I know lifeless wanted to make sure pbr was still tested with it but I don't think anyone did the work for that | 23:52 |
| dstufft | only ~3% of downloads from PyPI using a modern pip (6+) are via 2.6, so I don't feel like we should care about it anymore. | 23:52 |
| clarkb | and at this point if pip is dropping support I think its mostly moot | 23:52 |
| fungi | we supported python 2.6 while it was still the distro-provided default python in rhel (while 6 was current) | 23:52 |
| *** sarob has quit IRC | 23:52 | |
| dstufft | also I really want to stop thinking about 2.6 | 23:52 |
| dstufft | so there's that :D | 23:52 |
| *** sflanigan has joined #openstack-infra | 23:52 | |
| *** sflanigan has joined #openstack-infra | 23:52 | |
| fungi | once rhel 7 was out with python 2.7 default, and we had a release cycle to switch over, and then our supported stable branches that released back before we tested on rhel 7 finally reached eol, we stopped building rhel 6 servers to test changes on | 23:53 |
| fungi | and at that point, we had no existing means nor incentive to support python 2.6 | 23:54 |
| dstufft | awesome | 23:54 |
| fungi | and i felt we were on the conservative side of dropping 2.6 support | 23:54 |
| clarkb | fungi: I already have a hard enoug htime keeping up with all of our mailing lists | 23:54 |
| *** Jeffrey4l has quit IRC | 23:55 | |
| dstufft | I figured y'all were probably not to worried about it anymore, but also breaking openstack tends to mean I get yelled at on multiple fronts so :V | 23:55 |
| dstufft | "yelled at" | 23:55 |
| * fungi tties really hard not to yell at dstufft | 23:55 | |
| fungi | s/tties/tries/ | 23:55 |
| dstufft | (not actually yelled at, I just have an odd sense of humor) | 23:56 |
| *** ddieterly has joined #openstack-infra | 23:56 | |
| fungi | you do awesome work on pip/virtualenv/pypi/warehouse. i am always awed | 23:56 |
| dstufft | fungi: heh, thanks :) | 23:57 |
| *** markvoelker has joined #openstack-infra | 23:57 | |
| mordred | ++ | 23:58 |
| *** fguillot_ has joined #openstack-infra | 23:59 | |
| clarkb | I do have to say pypi/pip today compared to 2, 3 years ago is like night and day | 23:59 |
| clarkb | no longer do I fuss over setting up my mirrors. tl;dr yay things are much better today | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!