| mnaser | yeah my patch review failed because of it but ihingst was mentioned that patch would fix t | 00:00 |
|---|---|---|
| mnaser | accidentally touched my touchpad there, lol. | 00:00 |
| mnaser | it was mentioned that patch would fix it | 00:00 |
| clarkb | that moment when you click save on a gerrit comment just as backups happen | 00:01 |
| mnaser | http://logs.openstack.org/80/410380/3/check/gate-puppet-openstack-libraries-puppet-beaker-rspec-centos-7/7118db3/console.html#_2016-12-13_20_55_15_012112 | 00:01 |
| mnaser | i think that's what happened | 00:01 |
| *** annegentle has quit IRC | 00:01 | |
| mnaser | when its trying to manually install r10k, it fails | 00:01 |
| mnaser | it basically means gem install r10k on a centos 7 server is broken | 00:02 |
| mnaser | hm | 00:02 |
| mnaser | this is a r10k depedency, im not good with ruby's gemfile configs .. https://github.com/puppetlabs/gettext-setup-gem/commit/7c3b2fe7ee09136515dc84a5c9ac704e1ad2effc .. but i believe this is the cause | 00:03 |
| mnaser | https://github.com/puppetlabs/gettext-setup-gem/issues/16 | 00:04 |
| clarkb | ya I will admit that gems/rake/etc are all msotly a foreign language to me | 00:04 |
| clarkb | so its like trying to ride public transit in europe when I interact with them | 00:04 |
| mnaser | so it seems that the new gettext-setup release broke things | 00:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 00:04 |
| openstackgerrit | Merged openstack/diskimage-builder: Update sysctl-write-value to do conflict checking https://review.openstack.org/406413 | 00:04 |
| mnaser | i dont know if anyone want to comment/post on that issue :> | 00:05 |
| jeblair | well, i don't know the proper emoticon for a github pileon comment | 00:07 |
| *** baoli has quit IRC | 00:07 | |
| * fungi finds most european public transit quite navigable, but maybe that's just me | 00:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move pip-and-virtualenv source install to 10- https://review.openstack.org/408277 | 00:07 |
| *** tsg has quit IRC | 00:07 | |
| fungi | jeblair: "Me, too!" | 00:08 |
| fungi | or the shorter: "AOL!" | 00:08 |
| mnaser | everyone here to post a +1 in 5...4...3... | 00:09 |
| jeblair | :aol: | 00:09 |
| * fungi notes that "AOL!" as an interjection has an entry in the jargon file | 00:09 | |
| fungi | a see-also from "September that never ended" | 00:10 |
| *** mtanino has quit IRC | 00:10 | |
| *** asettle has joined #openstack-infra | 00:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Disable centos6 testing https://review.openstack.org/408936 | 00:11 |
| clarkb | fungi: ya its navigable you just have to figure out what a few of the words mean and how the diagrams go together | 00:11 |
| *** nadya has joined #openstack-infra | 00:11 | |
| clarkb | its different but parseable with effor | 00:12 |
| clarkb | t | 00:12 |
| *** tsg has joined #openstack-infra | 00:12 | |
| *** itisha has quit IRC | 00:12 | |
| *** esikachev has joined #openstack-infra | 00:12 | |
| mnaser | ok looks like it's already looked after from puppet-openstack folks :> | 00:13 |
| mnaser | https://review.openstack.org/#/c/410388/ | 00:13 |
| *** shashank_hegde has quit IRC | 00:13 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 00:14 |
| *** asettle has quit IRC | 00:14 | |
| *** katyafervent_awa has quit IRC | 00:14 | |
| *** katyafervent_awa has joined #openstack-infra | 00:14 | |
| *** dteselkin has quit IRC | 00:15 | |
| *** ativelkov has quit IRC | 00:15 | |
| *** igormarnat has quit IRC | 00:16 | |
| *** shashank_hegde has joined #openstack-infra | 00:16 | |
| *** nadya has quit IRC | 00:16 | |
| *** ativelkov has joined #openstack-infra | 00:17 | |
| *** esikachev has quit IRC | 00:17 | |
| *** dteselkin has joined #openstack-infra | 00:19 | |
| *** igormarnat has joined #openstack-infra | 00:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add install-types as pip-and-virtualenv dep https://review.openstack.org/408248 | 00:21 |
| *** rcernin has quit IRC | 00:22 | |
| *** hongbin has quit IRC | 00:22 | |
| *** spzala has quit IRC | 00:26 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:29 | |
| *** skelso has joined #openstack-infra | 00:29 | |
| *** rbuzatu has joined #openstack-infra | 00:30 | |
| *** harlowja has joined #openstack-infra | 00:30 | |
| *** fguillot has joined #openstack-infra | 00:31 | |
| *** Julien-zte has joined #openstack-infra | 00:31 | |
| *** gildub has quit IRC | 00:31 | |
| *** rbuzatu has quit IRC | 00:34 | |
| *** dimtruck is now known as zz_dimtruck | 00:36 | |
| openstackgerrit | Merged openstack/diskimage-builder: Move pip-and-virtualenv source install to 10- https://review.openstack.org/408277 | 00:36 |
| *** Julien-zte has quit IRC | 00:36 | |
| *** Julien-zte has joined #openstack-infra | 00:37 | |
| *** rbuzatu has joined #openstack-infra | 00:39 | |
| *** Julien-zte has quit IRC | 00:39 | |
| *** Julien-zte has joined #openstack-infra | 00:40 | |
| openstackgerrit | Ruijing proposed openstack-infra/project-config: Enable multinode connectivity for networking-odl reuse DEVSTACK_GATE_NEUTRON_DVR to set up internal vxlan for br-ex https://review.openstack.org/410477 | 00:40 |
| *** Julien-zte has quit IRC | 00:41 | |
| *** tsg has quit IRC | 00:41 | |
| *** Julien-zte has joined #openstack-infra | 00:42 | |
| *** Julien-zte has quit IRC | 00:42 | |
| *** sreeram_v has joined #openstack-infra | 00:42 | |
| *** Julien-zte has joined #openstack-infra | 00:42 | |
| *** Julien-zte has quit IRC | 00:43 | |
| *** rbuzatu has quit IRC | 00:43 | |
| jhesketh | fungi: I'm going to retire liberty branches as per tonyb's email this afternoon unless you have any objections | 00:44 |
| *** dteselkin has quit IRC | 00:44 | |
| *** yuanying has joined #openstack-infra | 00:44 | |
| *** Julien-zte has joined #openstack-infra | 00:44 | |
| *** Julien-zte has quit IRC | 00:45 | |
| *** sdake has quit IRC | 00:45 | |
| *** Julien-zte has joined #openstack-infra | 00:45 | |
| *** lindycoder has quit IRC | 00:45 | |
| fungi | jhesketh: thanks for taking care of that--no objections whatsoever! | 00:46 |
| jhesketh | of course :-) | 00:46 |
| fungi | and apparently in gerrit 2.13 or maybe 2.14, branch deletion becomes a separate permission we can delegate (finally!) | 00:46 |
| *** skelso has quit IRC | 00:46 | |
| *** sreeram_v has quit IRC | 00:47 | |
| *** devkulkarni has quit IRC | 00:48 | |
| *** dteselkin has joined #openstack-infra | 00:48 | |
| *** devkulkarni has joined #openstack-infra | 00:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin fast_gettext to work on CentOS7 ruby https://review.openstack.org/410478 | 00:49 |
| clarkb | ok if I have done that right then that should fix our fast_gettext issues | 00:49 |
| clarkb | jeblair: want me to go ahead and set a depends on for you on your nodepool back to master change/ | 00:49 |
| jeblair | clarkb: ++thx | 00:50 |
| * jeblair follows clarkb through the transit system | 00:50 | |
| clarkb | actually I don't think the openstackci puppet module gemfile will work with cross repo depends | 00:50 |
| clarkb | so we may just need to approve ^ if it passes then recheck jeblair's change and hope it fixes | 00:51 |
| clarkb | kind of like deciding to take a trasnfer in hopes that it gets you were you want to go | 00:51 |
| jeblair | clarkb: we can always go around to the other platform to turn around... but only if we get off at a station where you can do that without going through the turnstiles...? | 00:51 |
| *** yuanying has quit IRC | 00:52 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 00:52 |
| jeblair | clarkb: but actually, why wouldn't crd work? | 00:52 |
| *** yuanying has joined #openstack-infra | 00:52 | |
| jeblair | i thought the puppetopenstackci test was all about that | 00:52 |
| clarkb | jeblair: because the gemfile says install it from https://git.openstack.org/openstack-infra/puppet-openstack_infra_spec_helper | 00:53 |
| clarkb | in puppet-openstackci | 00:53 |
| jeblair | irony of ironies | 00:54 |
| *** zz_dimtruck is now known as dimtruck | 00:55 | |
| *** mmandel has joined #openstack-infra | 00:55 | |
| clarkb | so ya my suggestion would be to merge that, recheck, and revert if it doesn't work | 00:56 |
| jeblair | clarkb: test seemed to install gettext $oldversion. +3 | 00:56 |
| clarkb | otherwise I will spend more time trying to figure out how to setup a ruby env to test locally than it would take to do ^ | 00:56 |
| clarkb | jeblair: cool | 00:56 |
| *** yanyanhu has joined #openstack-infra | 00:57 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add install-types as pip-and-virtualenv dep https://review.openstack.org/408248 | 00:57 |
| *** mmandel has quit IRC | 01:00 | |
| *** jeckersb_gone is now known as jeckersb | 01:01 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Pin fast_gettext to work on CentOS7 ruby https://review.openstack.org/410478 | 01:02 |
| *** yanyanhu has quit IRC | 01:02 | |
| *** fguillot has quit IRC | 01:04 | |
| *** yanyanhu has joined #openstack-infra | 01:04 | |
| *** kzaitsev_mb has quit IRC | 01:05 | |
| *** pvaneck has quit IRC | 01:05 | |
| jeblair | clarkb: rechecked 410461 | 01:06 |
| jeblair | mnaser: ^ maybe you want to recheck now that 410478 has merged? | 01:07 |
| mnaser | jeblair thanks! | 01:08 |
| mnaser | done | 01:08 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 01:09 |
| EmilienM | mordred, clarkb ^ really wip but I'll make progress from here | 01:09 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 01:10 |
| *** dimtruck is now known as zz_dimtruck | 01:11 | |
| *** asettle has joined #openstack-infra | 01:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 01:13 |
| openstackgerrit | Merged openstack/diskimage-builder: Disable centos6 testing https://review.openstack.org/408936 | 01:14 |
| *** gildub has joined #openstack-infra | 01:15 | |
| *** yanyanhu has quit IRC | 01:16 | |
| *** yanyanhu has joined #openstack-infra | 01:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Added Gem Mirror to Infra https://review.openstack.org/253616 | 01:16 |
| clarkb | jeblair: jenkins is happy on 410461 now. Will let you approve when ready | 01:17 |
| jeblair | clarkb: i think maybe tomorrow morning at this point | 01:17 |
| jeblair | unless anyone else wants to babysit it | 01:18 |
| *** asettle has quit IRC | 01:18 | |
| clarkb | I am switching to dinner mode now soshouldnt | 01:19 |
| *** shu-mutou-AWAY is now known as shu-mutou | 01:19 | |
| *** zz_dimtruck is now known as dimtruck | 01:20 | |
| *** jkilpatr_ has quit IRC | 01:22 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: neutron-vpnaas: Remove gate-neutron-vpnaas-dsvm-functional-ubuntu-xenial https://review.openstack.org/410488 | 01:25 |
| *** tphummel has quit IRC | 01:25 | |
| clarkb | is^ the first undo xenialification change? | 01:26 |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Remove Horizon integrated test suite https://review.openstack.org/410489 | 01:28 |
| jeblair | do we have to move the spec out of implemented? | 01:28 |
| *** rbuzatu has joined #openstack-infra | 01:28 | |
| *** esberglu has joined #openstack-infra | 01:29 | |
| *** markvoelker has quit IRC | 01:29 | |
| *** piet has joined #openstack-infra | 01:30 | |
| *** mhickey has quit IRC | 01:30 | |
| *** esberglu has quit IRC | 01:30 | |
| *** john_Choi has joined #openstack-infra | 01:30 | |
| *** rbuzatu has quit IRC | 01:33 | |
| *** yamamoto has quit IRC | 01:33 | |
| *** gomarivera has joined #openstack-infra | 01:34 | |
| clarkb | jeblair: I hope not | 01:34 |
| *** claudiub has quit IRC | 01:39 | |
| *** ativelkov has quit IRC | 01:40 | |
| *** ativelkov has joined #openstack-infra | 01:40 | |
| *** dteselkin has quit IRC | 01:40 | |
| *** sreeram_v has joined #openstack-infra | 01:41 | |
| *** tuanluong has joined #openstack-infra | 01:42 | |
| *** dteselkin has joined #openstack-infra | 01:42 | |
| *** rbuzatu has joined #openstack-infra | 01:45 | |
| *** sreeram_v has quit IRC | 01:46 | |
| *** s-shiono has joined #openstack-infra | 01:48 | |
| *** rbuzatu has quit IRC | 01:48 | |
| *** rbuzatu has joined #openstack-infra | 01:49 | |
| *** yamamoto has joined #openstack-infra | 01:52 | |
| *** shashank_hegde has quit IRC | 01:52 | |
| *** s-shiono has quit IRC | 01:55 | |
| *** Julien-zte has quit IRC | 01:55 | |
| *** rkukura has quit IRC | 02:02 | |
| *** PramodJ has quit IRC | 02:03 | |
| *** pramodrj07 has quit IRC | 02:03 | |
| *** PramodJayathirth has quit IRC | 02:03 | |
| *** dimtruck is now known as zz_dimtruck | 02:04 | |
| *** john_Choi has quit IRC | 02:04 | |
| *** john_Choi has joined #openstack-infra | 02:05 | |
| *** Jeffrey4l has quit IRC | 02:05 | |
| *** rcarrillocruz has quit IRC | 02:06 | |
| *** tqtran has quit IRC | 02:07 | |
| *** rcarrillocruz has joined #openstack-infra | 02:07 | |
| *** Julien-zte has joined #openstack-infra | 02:07 | |
| *** dave-mccowan has quit IRC | 02:10 | |
| *** dave-mccowan has joined #openstack-infra | 02:10 | |
| *** lindycoder has joined #openstack-infra | 02:11 | |
| *** rbuzatu has quit IRC | 02:11 | |
| *** rbuzatu has joined #openstack-infra | 02:11 | |
| *** nadya has joined #openstack-infra | 02:12 | |
| *** Jeffrey4l has joined #openstack-infra | 02:15 | |
| *** zz_dimtruck is now known as dimtruck | 02:15 | |
| *** john_Choi has quit IRC | 02:16 | |
| *** nadya has quit IRC | 02:17 | |
| *** rbuzatu has quit IRC | 02:20 | |
| openstackgerrit | Merged openstack/diskimage-builder: Allow package-installs to parse DIB_PYTHON_VERSION https://review.openstack.org/408293 | 02:23 |
| *** lindycoder has quit IRC | 02:25 | |
| *** rbuzatu has joined #openstack-infra | 02:25 | |
| *** esikachev has joined #openstack-infra | 02:26 | |
| *** hongbin has joined #openstack-infra | 02:27 | |
| *** esikachev has quit IRC | 02:30 | |
| hongbin | ianw: ping | 02:31 |
| hongbin | ianw: if you have a chance, could you revisit this patch https://review.openstack.org/#/c/408330/ ? Your comment about dependency is addressed and it got +1 from Kuryr PTL | 02:32 |
| *** baoli has joined #openstack-infra | 02:32 | |
| tonyb | jhesketh: How far through the liberty EOL are you? | 02:35 |
| jhesketh | tonyb: haven't begun | 02:35 |
| tonyb | jhesketh: okay. | 02:35 |
| jhesketh | tonyb: any reason? | 02:36 |
| tonyb | jhesketh: I'm going to take zaqar from the list, so please re-fetch that list before you do start | 02:36 |
| jhesketh | ack | 02:36 |
| tonyb | jhesketh: I'm removing them as they have a tag final release chnage in the releases repo and I'd like to land that first | 02:36 |
| jhesketh | tonyb: looks like the gist you linked me to might be at a sha... perhaps re-link me once you've updated it to make sure I have the right version please | 02:36 |
| tonyb | jhesketh: do tou have one or tywo sha's in the URL? | 02:37 |
| jhesketh | tonyb: 2 | 02:37 |
| jhesketh | either way, a new link for confidence would be appreciated | 02:37 |
| tonyb | jhesketh: https://gist.githubusercontent.com/tbreeds/93cd346c37aa46269456f56649f0a4ac/raw/5e910d434808ae4ffb2ba7a6f857b4ba9613173b/liberty_eol_data.txt | 02:38 |
| tonyb | jhesketh: I'd expect that the first SHA is the same and the secon chnages | 02:38 |
| tonyb | jhesketh: but https://gist.githubusercontent.com/tbreeds/93cd346c37aa46269456f56649f0a4ac/raw/liberty_eol_data.txt is the link I meant to send the first time | 02:39 |
| jhesketh | tonyb: yep | 02:39 |
| tonyb | jhesketh: which shoudl always be the current HEAD of the master branch for the gist | 02:39 |
| *** thorst_ has joined #openstack-infra | 02:40 | |
| *** hieulq_ has joined #openstack-infra | 02:45 | |
| *** Swami_ has joined #openstack-infra | 02:46 | |
| *** hieulq_ has quit IRC | 02:46 | |
| *** thorst_ has quit IRC | 02:47 | |
| *** Julien-zte has quit IRC | 02:47 | |
| *** thorst_ has joined #openstack-infra | 02:47 | |
| *** Swami has quit IRC | 02:48 | |
| *** Swami__ has quit IRC | 02:48 | |
| *** mrunge has quit IRC | 02:49 | |
| *** Julien-zte has joined #openstack-infra | 02:49 | |
| *** Kevin_Zheng has joined #openstack-infra | 02:49 | |
| *** Swami has joined #openstack-infra | 02:49 | |
| *** gomarivera has quit IRC | 02:50 | |
| *** thorst_ has quit IRC | 02:52 | |
| *** yuanying has quit IRC | 02:53 | |
| *** mrunge has joined #openstack-infra | 02:54 | |
| *** yuanying has joined #openstack-infra | 02:54 | |
| *** sdake has joined #openstack-infra | 02:55 | |
| *** liusheng has joined #openstack-infra | 02:55 | |
| *** gomarivera has joined #openstack-infra | 02:57 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 02:58 |
| *** yuanying has quit IRC | 02:58 | |
| *** esikachev has joined #openstack-infra | 03:00 | |
| *** esikachev has quit IRC | 03:05 | |
| *** rbuzatu has quit IRC | 03:09 | |
| *** Julien-zte has quit IRC | 03:11 | |
| *** gongysh has joined #openstack-infra | 03:19 | |
| *** EmilienM has quit IRC | 03:19 | |
| *** EmilienM has joined #openstack-infra | 03:19 | |
| *** yamahata_ has quit IRC | 03:24 | |
| *** coolsvap has joined #openstack-infra | 03:27 | |
| *** esikachev has joined #openstack-infra | 03:30 | |
| *** markvoelker has joined #openstack-infra | 03:30 | |
| *** esikachev has quit IRC | 03:34 | |
| *** markvoelker has quit IRC | 03:35 | |
| *** piet has quit IRC | 03:35 | |
| *** john_Choi has joined #openstack-infra | 03:37 | |
| *** vikrant has joined #openstack-infra | 03:39 | |
| *** dave-mccowan has quit IRC | 03:39 | |
| *** jerryz has quit IRC | 03:41 | |
| *** sdake has quit IRC | 03:44 | |
| *** esikachev has joined #openstack-infra | 03:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 03:47 |
| *** john_Choi has quit IRC | 03:47 | |
| *** esikachev has quit IRC | 03:50 | |
| *** yuanying has joined #openstack-infra | 03:52 | |
| *** thorst_ has joined #openstack-infra | 03:54 | |
| *** Apoorva has joined #openstack-infra | 03:55 | |
| *** baoli has quit IRC | 03:57 | |
| *** Swami_ has quit IRC | 03:57 | |
| *** Swami_ has joined #openstack-infra | 03:57 | |
| *** thorst_ has quit IRC | 03:58 | |
| *** thorst_ has joined #openstack-infra | 03:59 | |
| *** ijw has quit IRC | 03:59 | |
| *** pgadiya has joined #openstack-infra | 04:00 | |
| *** rcarrillocruz has quit IRC | 04:01 | |
| *** sdake has joined #openstack-infra | 04:01 | |
| *** mmandel has joined #openstack-infra | 04:02 | |
| *** tqtran has joined #openstack-infra | 04:03 | |
| *** hongbin has quit IRC | 04:05 | |
| *** nicolasbock has quit IRC | 04:05 | |
| *** mmandel has quit IRC | 04:06 | |
| *** Apoorva has quit IRC | 04:06 | |
| *** thorst_ has quit IRC | 04:07 | |
| *** raissa has quit IRC | 04:07 | |
| *** raissa has joined #openstack-infra | 04:07 | |
| *** tqtran has quit IRC | 04:08 | |
| *** russellb has quit IRC | 04:08 | |
| *** shu-mutou has quit IRC | 04:09 | |
| *** russellb has joined #openstack-infra | 04:09 | |
| *** rbuzatu has joined #openstack-infra | 04:09 | |
| *** sdake_ has joined #openstack-infra | 04:10 | |
| *** sdake has quit IRC | 04:13 | |
| *** shu-mutou has joined #openstack-infra | 04:15 | |
| *** shashank_hegde has joined #openstack-infra | 04:17 | |
| *** psachin has joined #openstack-infra | 04:18 | |
| *** sreeram_v has joined #openstack-infra | 04:21 | |
| *** rbuzatu has quit IRC | 04:21 | |
| *** bhavik1 has joined #openstack-infra | 04:23 | |
| *** shashank_hegde has quit IRC | 04:24 | |
| *** shashank_hegde has joined #openstack-infra | 04:24 | |
| *** adrian_otto has joined #openstack-infra | 04:29 | |
| *** allanice001 has quit IRC | 04:33 | |
| *** thorst_ has joined #openstack-infra | 04:38 | |
| *** prometheanfire has left #openstack-infra | 04:38 | |
| *** sdake_ is now known as sdake | 04:40 | |
| *** rbuzatu has joined #openstack-infra | 04:41 | |
| *** adrian_otto has quit IRC | 04:42 | |
| *** sambetts_ has quit IRC | 04:43 | |
| *** sambetts_ has joined #openstack-infra | 04:44 | |
| *** chlong has joined #openstack-infra | 04:44 | |
| *** shashank_hegde has quit IRC | 04:44 | |
| *** sambetts_ is now known as Guest51566 | 04:44 | |
| *** shashank_hegde has joined #openstack-infra | 04:45 | |
| *** rbuzatu has quit IRC | 04:46 | |
| *** ijw has joined #openstack-infra | 04:46 | |
| *** Guest51566 has quit IRC | 04:55 | |
| *** sambetts has joined #openstack-infra | 04:57 | |
| *** esikachev has joined #openstack-infra | 04:57 | |
| *** links has joined #openstack-infra | 04:57 | |
| *** adrian_otto has joined #openstack-infra | 04:59 | |
| *** rcarrillocruz has joined #openstack-infra | 04:59 | |
| *** hieulq_ has joined #openstack-infra | 05:00 | |
| *** sdake has quit IRC | 05:00 | |
| *** esikachev has quit IRC | 05:01 | |
| *** rlandy has quit IRC | 05:02 | |
| *** thorst_ has quit IRC | 05:04 | |
| *** tqtran has joined #openstack-infra | 05:06 | |
| *** Swami_ has quit IRC | 05:06 | |
| *** Swami has quit IRC | 05:06 | |
| *** gomarivera has quit IRC | 05:07 | |
| *** tqtran has quit IRC | 05:10 | |
| *** adrian_otto has quit IRC | 05:10 | |
| *** hieulq_ has quit IRC | 05:11 | |
| *** pradk has quit IRC | 05:12 | |
| *** asettle has joined #openstack-infra | 05:14 | |
| *** wolverineav has joined #openstack-infra | 05:18 | |
| *** asettle has quit IRC | 05:21 | |
| *** wolverineav has quit IRC | 05:22 | |
| *** gomarivera has joined #openstack-infra | 05:28 | |
| *** yamahata_ has joined #openstack-infra | 05:29 | |
| *** kfox1111 has quit IRC | 05:30 | |
| *** kfox1111 has joined #openstack-infra | 05:30 | |
| *** snarwade has joined #openstack-infra | 05:31 | |
| *** markvoelker has joined #openstack-infra | 05:31 | |
| *** aeng has quit IRC | 05:31 | |
| *** thorst_ has joined #openstack-infra | 05:34 | |
| *** gcb has quit IRC | 05:35 | |
| *** markvoelker has quit IRC | 05:37 | |
| openstackgerrit | Mike Kolesnik proposed openstack-infra/project-config: Add non voting networking-odl job https://review.openstack.org/410240 | 05:38 |
| *** Ravikiran_K has joined #openstack-infra | 05:41 | |
| *** thorst_ has quit IRC | 05:41 | |
| *** rbuzatu has joined #openstack-infra | 05:42 | |
| *** rajinir has quit IRC | 05:46 | |
| *** jaosorior has joined #openstack-infra | 05:50 | |
| *** rbuzatu has quit IRC | 05:52 | |
| *** gcb has joined #openstack-infra | 05:52 | |
| *** adriant has quit IRC | 05:53 | |
| *** allanice001 has joined #openstack-infra | 05:53 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Make python35 voting https://review.openstack.org/410226 | 06:02 |
| *** hyakuhei has quit IRC | 06:03 | |
| *** gcb has quit IRC | 06:05 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update warnings plugin https://review.openstack.org/410011 | 06:07 |
| *** nadya has joined #openstack-infra | 06:08 | |
| *** nadya has quit IRC | 06:08 | |
| *** amotoki has joined #openstack-infra | 06:12 | |
| *** abregman has joined #openstack-infra | 06:13 | |
| *** amotoki has quit IRC | 06:14 | |
| *** nadya has joined #openstack-infra | 06:14 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix pip-and-virtualenv to work with python3 https://review.openstack.org/407723 | 06:14 |
| *** tuanluong has quit IRC | 06:16 | |
| *** amotoki has joined #openstack-infra | 06:17 | |
| *** nadya has quit IRC | 06:19 | |
| *** esikachev has joined #openstack-infra | 06:19 | |
| *** tuanluong has joined #openstack-infra | 06:21 | |
| *** gcb has joined #openstack-infra | 06:22 | |
| *** esikachev has quit IRC | 06:24 | |
| *** hurgleburgler has quit IRC | 06:27 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove files from undercloud to save space https://review.openstack.org/410131 | 06:28 |
| *** hurgleburgler has joined #openstack-infra | 06:28 | |
| jamielennox | jhesketh, ianw: any tips for how to kill a stalled job in zuul 2.5? | 06:28 |
| jamielennox | restarting the server is painful | 06:28 |
| *** thorst_ has joined #openstack-infra | 06:28 | |
| ianw | jamielennox: umm, log into the host and kill anything running under ssh (except yourself :) | 06:28 |
| ianw | ? | 06:28 |
| *** hurgleburgler has quit IRC | 06:28 | |
| ianw | that should at least just fail the job | 06:29 |
| *** eroux has joined #openstack-infra | 06:31 | |
| jamielennox | ianw: the launcher host? | 06:32 |
| ianw | jamielennox: the testing host. but maybe i'm misunderstanding how it's stalled | 06:32 |
| jamielennox | ianw: yea, that's kind of the thing i couldn't tell how, it was in zuul show running-jobs and it wasn't able to enqueue it again but it wasn't running anywhere | 06:33 |
| jamielennox | it seems to have timed out now, i just was hoping there was a forceful way to dequeue it from the pipeline | 06:34 |
| ianw | jamielennox: ahh, yeah not sure | 06:34 |
| *** thorst_ has quit IRC | 06:36 | |
| *** esikachev has joined #openstack-infra | 06:37 | |
| jamielennox | i end up with | 06:38 |
| jamielennox | zuul show running-jobs | 06:38 |
| jamielennox | +----------+---------+-------------+--------+ | 06:38 |
| jamielennox | | Job Name | Worker | Start Time | Result | | 06:38 |
| jamielennox | +----------+---------+-------------+--------+ | 06:38 |
| jamielennox | | tox_py27 | Unknown | Unknown ago | None | | 06:38 |
| jamielennox | +----------+---------+-------------+--------+ | 06:38 |
| jamielennox | and there's really nothing that says why, or any way to kill it | 06:39 |
| *** vladani has quit IRC | 06:40 | |
| *** afazekas has quit IRC | 06:40 | |
| *** gomarivera has quit IRC | 06:41 | |
| *** afazekas has joined #openstack-infra | 06:42 | |
| jhesketh | jamielennox: I'm not sure how zuul would go if you kill the runner itself.. I suspect the scheduler wouldn't notice and the job would be stuck (might be wrong though) | 06:42 |
| jhesketh | jamielennox: if you can find the ansible sub-process that'd be the one to kill | 06:42 |
| *** flepied has quit IRC | 06:44 | |
| jamielennox | jhesketh: so there's no ansible processes running | 06:44 |
| jamielennox | the only possible thing i've found is if i kill all available nodepool servers it can sometimes prompt it | 06:48 |
| jhesketh | jamielennox: so you have a stalled job with no ansible process.. is it a reoccuring problem? | 06:48 |
| *** chandankumar|off is now known as chandankumar | 06:48 | |
| *** rbuzatu has joined #openstack-infra | 06:48 | |
| jamielennox | jhesketh: yes, but honestly there's so many moving parts at the moment that i'm not going to say it's repeatable or anything serious | 06:49 |
| jhesketh | jamielennox: and you're running master? | 06:50 |
| jamielennox | yep | 06:50 |
| jhesketh | any hints in the launcher debug logs? | 06:50 |
| jamielennox | so yea, i killed all nodepool listeners | 06:50 |
| jamielennox | and it reported back that the job failed in 8 seconds | 06:51 |
| jamielennox | 20 minutes later | 06:51 |
| *** edisonxiang has quit IRC | 06:51 | |
| jhesketh | jamielennox: did the job have logs? | 06:51 |
| *** edisonxiang has joined #openstack-infra | 06:51 | |
| jamielennox | http://logs.bonnyci.com/logs/1/1/d4ff2139f57a78bd924fdfa33103739e8783ee1b/check_github/tox_py27/3e3afea/console.html | 06:52 |
| jamielennox | if you view-source the formatting is better | 06:52 |
| *** rbuzatu has quit IRC | 06:53 | |
| jhesketh | hmm it's not clear why killing the nodepool listeners unblocked it, but have you looked into the git errors there? | 06:55 |
| jamielennox | that's the current process i'm looking into | 06:55 |
| jamielennox | i'm not sure yet how exposing the zuul git repos over http works when they are not bare repos | 06:55 |
| openstackgerrit | Dobroslaw Zybort proposed openstack-infra/project-config: Add tox check for monasca-log-api https://review.openstack.org/410555 | 06:56 |
| clarkb | jamielennox: iirc nothing about git repo hosting requires they be bare | 06:58 |
| jamielennox | clarkb: so i was reading from: http://docs.openstack.org/infra/zuul/merger.html#serving-zuul-git-repos | 07:00 |
| *** markvoelker has joined #openstack-infra | 07:00 | |
| jamielennox | when using git-http-backend like that when i do a git clone against the URL with LogLevel debug i get 404s | 07:01 |
| jamielennox | and it looks like the git-http-backend is trying to find bare data | 07:01 |
| jamielennox | so maybe the zuul-cloner is doing something different, i haven't seen that yet | 07:01 |
| *** ijw has quit IRC | 07:02 | |
| clarkb | zuul cloner doesnt clone from the mergers it only fetches the required refs | 07:02 |
| clarkb | which works fine with not bare repos | 07:02 |
| jamielennox | ah, now that might be where we are having problems | 07:02 |
| clarkb | (and I think not bare would worl fine too if noy actively being modified) | 07:02 |
| jamielennox | (which may/may not be why the job is hanging) | 07:03 |
| jamielennox | so zuul-cloner takes a --zuul-url as well as a regular url | 07:03 |
| clarkb | the issues arise from races between cloning and changes to what HEAD is and the like | 07:03 |
| AJaeger | clarkb: regarding https://review.openstack.org/#/c/410488: This is stopping testing because of no openswan. There's another job that tests strongswan. So, that looks fine | 07:03 |
| jamielennox | that works so long as those are public and unchanging | 07:03 |
| *** nadya has joined #openstack-infra | 07:04 | |
| jamielennox | but we have multiple sources hooked up to zuul, so assuming that the base url is git.openstack.org or anything else doesn't really work | 07:04 |
| clarkb | yes I suggested to jlk that a mirror makes sense for other reasons in addition to solving this earlier today | 07:04 |
| AJaeger | ianw, jhesketh, yolanda: Could you put https://review.openstack.org/410073 https://review.openstack.org/410079 and https://review.openstack.org/410387 as trusty/py34 cleanups on your review queue, please? And also a change to get rid of custom py27: https://review.openstack.org/409015 | 07:05 |
| clarkb | other idea was to update z-c map to specify a specific source and ignore base url for those | 07:05 |
| clarkb | that would require a z-c change as thats not current behavior | 07:06 |
| *** markvoelker has quit IRC | 07:06 | |
| jhesketh | AJaeger: sure, but my queue is backed up so not sure when I'll get to them sorry | 07:06 |
| AJaeger | jhesketh: no worries - thanks. | 07:07 |
| jamielennox | clarkb: ah ok, so jlk's already discovered this - i'll have a further talk with him tomorrow | 07:07 |
| jamielennox | thanks | 07:07 |
| *** maeker has quit IRC | 07:09 | |
| openstackgerrit | Merged openstack/diskimage-builder: Catch errors in DIB_INIT_SYSTEM export https://review.openstack.org/400542 | 07:14 |
| *** netsin has quit IRC | 07:14 | |
| *** markus_z has joined #openstack-infra | 07:15 | |
| *** vi01et has quit IRC | 07:15 | |
| *** allanice001 has quit IRC | 07:16 | |
| *** nherciu has joined #openstack-infra | 07:16 | |
| *** Julien-zte has joined #openstack-infra | 07:16 | |
| *** asettle has joined #openstack-infra | 07:18 | |
| *** bapalm has quit IRC | 07:18 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix pip-and-virtualenv to work with python3 https://review.openstack.org/407723 | 07:18 |
| *** yamamoto has quit IRC | 07:18 | |
| Julien-zte | hi, can zuul monitor some type of dedicated file changes, such as *.rst and trigger some jobs. like file-paths of gerrit plugin in Jenkins | 07:18 |
| AJaeger | Julien-zte: yes, it does | 07:19 |
| *** Ravikiran_K has quit IRC | 07:19 | |
| Julien-zte | it's great | 07:19 |
| AJaeger | Julien-zte: check zuul/layout.yaml in project-config and search for "files:" | 07:19 |
| *** florianf has joined #openstack-infra | 07:19 | |
| Julien-zte | I don't find some examples | 07:19 |
| Julien-zte | ok, I will do it | 07:20 |
| *** Ravikiran_K has joined #openstack-infra | 07:20 | |
| AJaeger | Did you read the zuul docs? | 07:20 |
| Julien-zte | yes, from http://docs.openstack.org/infra/zuul/triggers.html , no usefule docs | 07:20 |
| *** thorst_ has joined #openstack-infra | 07:23 | |
| yolanda | hi, good morning | 07:24 |
| yolanda | AJaeger, i'll take a look | 07:24 |
| AJaeger | Julien-zte: http://docs.openstack.org/infra/zuul/zuul.html#pipelines | 07:24 |
| AJaeger | thanks, yolanda | 07:24 |
| *** andreas_s has joined #openstack-infra | 07:25 | |
| *** ramishra has quit IRC | 07:25 | |
| *** asettle has quit IRC | 07:25 | |
| toabctl | if I want todo a search on pipy while running a job in the CI, which pypi url should I use? | 07:25 |
| toabctl | I guess not https://pypi.python.org/pypi | 07:26 |
| *** edisonxiang has quit IRC | 07:26 | |
| Julien-zte | thanks Ajaeger | 07:26 |
| Julien-zte | I make a mistrake, find it in triggers not in jobs | 07:26 |
| *** ramishra has joined #openstack-infra | 07:26 | |
| AJaeger | toabctl: I'm not understanding what you want to do. Good morning to you! | 07:27 |
| toabctl | AJaeger, a script runs when calling tox -elint and the script has "xmlrpc_client.ServerProxy('https://pypi.python.org/pypi')" | 07:28 |
| toabctl | AJaeger, good morning :) | 07:28 |
| toabctl | AJaeger, is that ok ? or should I somehow use a local pypi mirror when running this in the openstack infrastructure? | 07:28 |
| AJaeger | toabctl: we have local mirrors on regions, let me find where we sat that... | 07:29 |
| AJaeger | toabctl: see project-config file nodepool/scripts/configure_mirror.sh - we write a pip.conf that contains the mirror to use. Can you parse that pip.conf? | 07:30 |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup py34 for nodepool, zuul https://review.openstack.org/410073 | 07:31 |
| AJaeger | toabctl: "grep '^index-url' /etc/pip.conf' - and take the first one ? | 07:31 |
| *** thorst_ has quit IRC | 07:31 | |
| toabctl | AJaeger, that should work. thanks! | 07:32 |
| openstackgerrit | Merged openstack-infra/project-config: Python34 cleanup: https://review.openstack.org/410079 | 07:32 |
| *** allanice001 has joined #openstack-infra | 07:33 | |
| AJaeger | yolanda: could review https://review.openstack.org/409015 as well, please? | 07:33 |
| openstackgerrit | Merged openstack-infra/project-config: Python34/trusty cleanup https://review.openstack.org/410387 | 07:33 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Remove TestScheduler.test_nonexistent_job https://review.openstack.org/410571 | 07:34 |
| *** drifterza has joined #openstack-infra | 07:35 | |
| *** yamahata_ has quit IRC | 07:37 | |
| *** bapalm has joined #openstack-infra | 07:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip security role gate for branches <= Mitaka https://review.openstack.org/410382 | 07:39 |
| openstackgerrit | Merged openstack-infra/project-config: Remove Horizon integrated test suite https://review.openstack.org/410489 | 07:39 |
| *** gcb has quit IRC | 07:44 | |
| *** gcb has joined #openstack-infra | 07:45 | |
| *** nherciu_ has joined #openstack-infra | 07:46 | |
| *** gongysh has quit IRC | 07:47 | |
| *** tphummel has joined #openstack-infra | 07:47 | |
| *** claudiub has joined #openstack-infra | 07:48 | |
| *** maeker has joined #openstack-infra | 07:50 | |
| *** nherciu has quit IRC | 07:50 | |
| *** rbuzatu has joined #openstack-infra | 07:50 | |
| *** ralonsoh has joined #openstack-infra | 07:51 | |
| *** claudiub has quit IRC | 07:53 | |
| *** maeker has quit IRC | 07:54 | |
| *** rbuzatu has quit IRC | 07:54 | |
| *** rcernin has joined #openstack-infra | 07:56 | |
| *** yamamoto has joined #openstack-infra | 07:56 | |
| *** savihou has joined #openstack-infra | 07:56 | |
| *** rcernin has quit IRC | 07:56 | |
| *** yuval has joined #openstack-infra | 07:57 | |
| *** rcernin has joined #openstack-infra | 07:57 | |
| *** yamahata_ has joined #openstack-infra | 07:59 | |
| *** bhavik1 has quit IRC | 07:59 | |
| *** pcaruana has joined #openstack-infra | 07:59 | |
| *** mmandel has joined #openstack-infra | 08:03 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** mmandel has quit IRC | 08:07 | |
| *** shardy has joined #openstack-infra | 08:07 | |
| *** tqtran has joined #openstack-infra | 08:08 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: snaps: Add OpenStack Snap package repos https://review.openstack.org/407459 | 08:09 |
| *** ccamacho has joined #openstack-infra | 08:10 | |
| openstackgerrit | Luka Peschke proposed openstack-infra/project-config: Moved cloudkitty-dsvm job from experimental to gate pipeline https://review.openstack.org/409849 | 08:10 |
| *** gcb has quit IRC | 08:10 | |
| *** gcb has joined #openstack-infra | 08:11 | |
| openstackgerrit | Luka Peschke proposed openstack-infra/project-config: Moved cloudkitty-dsvm job from experimental to gate pipeline https://review.openstack.org/409849 | 08:11 |
| *** knangia has joined #openstack-infra | 08:11 | |
| *** tqtran has quit IRC | 08:12 | |
| *** jaosorior has quit IRC | 08:16 | |
| *** jaosorior has joined #openstack-infra | 08:16 | |
| *** flepied has joined #openstack-infra | 08:17 | |
| *** ihrachys has joined #openstack-infra | 08:17 | |
| *** thorst_ has joined #openstack-infra | 08:19 | |
| *** shashank_hegde has quit IRC | 08:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Make python35 voting https://review.openstack.org/410226 | 08:22 |
| *** vladani has joined #openstack-infra | 08:23 | |
| *** john-davidge has joined #openstack-infra | 08:24 | |
| openstackgerrit | Merged openstack-infra/project-config: neutron-vpnaas: Remove gate-neutron-vpnaas-dsvm-functional-ubuntu-xenial https://review.openstack.org/410488 | 08:26 |
| openstackgerrit | Merged openstack-infra/project-config: Remove multinode grenade in Cinder's stable/mitaka https://review.openstack.org/407463 | 08:26 |
| openstackgerrit | Merged openstack-infra/project-config: Add tempest gate for Almanach https://review.openstack.org/406350 | 08:26 |
| *** jpich has joined #openstack-infra | 08:27 | |
| *** john-davidge has quit IRC | 08:27 | |
| *** matrohon has joined #openstack-infra | 08:27 | |
| *** thorst_ has quit IRC | 08:27 | |
| *** john-davidge has joined #openstack-infra | 08:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Make the pxe_ipmitool-virtualbmc-src job voting https://review.openstack.org/410236 | 08:30 |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Make grenade job voting https://review.openstack.org/410219 | 08:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add new charm CloudKitty https://review.openstack.org/407464 | 08:31 |
| openstackgerrit | Merged openstack-infra/project-config: [magnum][rally] Enable heat plugin https://review.openstack.org/410234 | 08:31 |
| *** jpena|off is now known as jpena | 08:37 | |
| *** john-dav_ has joined #openstack-infra | 08:37 | |
| *** john-davidge has quit IRC | 08:37 | |
| *** daemontool has joined #openstack-infra | 08:44 | |
| *** snarwade has quit IRC | 08:44 | |
| *** snarwade has joined #openstack-infra | 08:45 | |
| *** dimtruck is now known as zz_dimtruck | 08:46 | |
| *** hashar has joined #openstack-infra | 08:46 | |
| openstackgerrit | Sreeram Vancheeswaran proposed openstack-infra/project-config: Request to add new project os-brick-dpm https://review.openstack.org/410601 | 08:47 |
| *** netsin has joined #openstack-infra | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** jpena is now known as jpena|off | 08:49 | |
| *** snarwade has quit IRC | 08:51 | |
| *** savihou has quit IRC | 08:51 | |
| *** snarwade has joined #openstack-infra | 08:51 | |
| *** rbuzatu has joined #openstack-infra | 08:51 | |
| *** daemontool_ has joined #openstack-infra | 08:55 | |
| *** openstackgerrit has joined #openstack-infra | 08:55 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 08:55 |
| *** rbuzatu has quit IRC | 08:56 | |
| *** gildub has quit IRC | 08:56 | |
| *** amoralej|off is now known as amoralej | 08:56 | |
| *** dingyichen has quit IRC | 08:57 | |
| *** strigazi1AFK has joined #openstack-infra | 08:58 | |
| *** daemontool has quit IRC | 08:58 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:59 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** rossella_s has joined #openstack-infra | 09:00 | |
| *** zzzeek has joined #openstack-infra | 09:00 | |
| *** strigazi_AFK has quit IRC | 09:00 | |
| *** yolanda has quit IRC | 09:01 | |
| *** markvoelker has joined #openstack-infra | 09:02 | |
| *** csomerville has quit IRC | 09:02 | |
| *** tuanluong has quit IRC | 09:02 | |
| *** cody-somerville has joined #openstack-infra | 09:03 | |
| *** tuanluong has joined #openstack-infra | 09:03 | |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| *** strigazi_AFK has joined #openstack-infra | 09:04 | |
| *** ianychoi has joined #openstack-infra | 09:05 | |
| *** yolanda has joined #openstack-infra | 09:05 | |
| *** markvoelker has quit IRC | 09:07 | |
| *** strigazi1AFK has quit IRC | 09:07 | |
| *** jesusaur has quit IRC | 09:08 | |
| *** flepied has quit IRC | 09:15 | |
| openstackgerrit | Sreeram Vancheeswaran proposed openstack-infra/project-config: Request to add new project os-brick-dpm https://review.openstack.org/410601 | 09:15 |
| *** nadya has quit IRC | 09:16 | |
| *** savihou has joined #openstack-infra | 09:17 | |
| *** thorst_ has joined #openstack-infra | 09:18 | |
| *** jordanP has joined #openstack-infra | 09:18 | |
| savihou | Hello Everyone, hope I didn't miss anything, is there any chance applying tempest configuration using [[post-extra|\$TEMPEST_CONFIG]] in local.conf was deprecated ? | 09:19 |
| *** jesusaur has joined #openstack-infra | 09:21 | |
| *** gcb has quit IRC | 09:21 | |
| savihou | Please dismiss my question, found https://review.openstack.org/#/c/405446/ mail. | 09:22 |
| *** thorst_ has quit IRC | 09:24 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 09:25 |
| *** lucas-afk is now known as lucasagomes | 09:27 | |
| sshnaidm | yolanda, hi, did you have a chance to look at https://review.openstack.org/#/c/401282/ ? | 09:28 |
| *** snarwade has quit IRC | 09:28 | |
| yolanda | hi sshnaidm , sorry. I 'll take a look now | 09:29 |
| sshnaidm | yolanda, thanks | 09:29 |
| *** furlongm_ has joined #openstack-infra | 09:29 | |
| yolanda | looks fine to me | 09:29 |
| *** chem has quit IRC | 09:30 | |
| *** chem has joined #openstack-infra | 09:30 | |
| *** furlongm has quit IRC | 09:30 | |
| *** rossella_s has quit IRC | 09:31 | |
| *** rossella_s has joined #openstack-infra | 09:32 | |
| *** snarwade has joined #openstack-infra | 09:32 | |
| *** derekh has joined #openstack-infra | 09:32 | |
| *** tuanluong_ has joined #openstack-infra | 09:34 | |
| sshnaidm | yolanda, so extras will be always cloned together with quickstart, right? | 09:36 |
| yolanda | yes | 09:36 |
| *** tuanluong has quit IRC | 09:36 | |
| *** Julien-zte has quit IRC | 09:37 | |
| *** yamahata_ has quit IRC | 09:37 | |
| sshnaidm | yolanda, thanks! | 09:37 |
| yolanda | anytime | 09:37 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config: Always clone quickstart extras with quickstart https://review.openstack.org/401282 | 09:38 |
| *** gcb has joined #openstack-infra | 09:39 | |
| *** itisha has joined #openstack-infra | 09:39 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: elements: dib-python: Add python2 as the default version for openSUSE https://review.openstack.org/410639 | 09:40 |
| *** flepied has joined #openstack-infra | 09:40 | |
| *** asettle has joined #openstack-infra | 09:44 | |
| *** rbuzatu has joined #openstack-infra | 09:45 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Make ...ironic-pxe_ipmitool-pyghmi-src-ubuntu-xenial voting https://review.openstack.org/410178 | 09:47 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Enable python-jobs for pyghmi https://review.openstack.org/410177 | 09:47 |
| *** furlongm has joined #openstack-infra | 09:54 | |
| *** claudiub has joined #openstack-infra | 09:55 | |
| *** furlongm_ has quit IRC | 09:55 | |
| oanson | ttx, hi, are you in? | 09:56 |
| *** askb has quit IRC | 09:59 | |
| *** electrofelix has joined #openstack-infra | 10:02 | |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 10:02 |
| ttx | oanson: yes | 10:04 |
| oanson | ttx, Hi. I wanted to verify Dragonflow has a room in the PTG | 10:04 |
| oanson | I saw it was added to the list, but the mail I sent to faq@... was not replied (or I missed it) | 10:04 |
| ttx | yes, as mentioned on http://www.openstack.org/ptg | 10:04 |
| *** gongysh has joined #openstack-infra | 10:04 | |
| oanson | Great. Thanks! | 10:04 |
| ttx | oanson: np! | 10:04 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement major upgrade for Newton to Ocata https://review.openstack.org/404831 | 10:05 |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 10:05 |
| *** jpena|off is now known as jpena | 10:05 | |
| d0ugal | Is py35 replacing the py34 checks? | 10:05 |
| *** martinkopec has joined #openstack-infra | 10:05 | |
| d0ugal | I have noticed some projects have dropped py34, but py35 is still non-voting. | 10:05 |
| AJaeger | d0ugal: see http://lists.openstack.org/pipermail/openstack-dev/2016-December/108821.html | 10:06 |
| AJaeger | d0ugal: please make py35 voting | 10:06 |
| d0ugal | AJaeger: will do, thanks! | 10:07 |
| d0ugal | rakhmerov: ^ | 10:07 |
| d0ugal | jaosorior: ^ | 10:07 |
| rakhmerov | ok, thanks | 10:08 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: novajoin: Make py35 job voting https://review.openstack.org/410668 | 10:09 |
| *** jesusaur has quit IRC | 10:09 | |
| *** tqtran has joined #openstack-infra | 10:09 | |
| *** dtantsur is now known as dtantsur|brb | 10:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Use test-setup for monasca-analytics https://review.openstack.org/409015 | 10:10 |
| *** drifterza has quit IRC | 10:10 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Make the py35 jobs voting on Mistral https://review.openstack.org/410671 | 10:13 |
| *** tqtran has quit IRC | 10:13 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Make py35 voting on tripleoclient and tripleo-common https://review.openstack.org/410672 | 10:15 |
| dirk | does anyone understand what "POST_FAILURE" means? | 10:15 |
| dirk | https://review.openstack.org/#/c/408987/ | 10:15 |
| dirk | we get that pretty consistently with glance on this review, I begin to suspect its something with the review rather than infra.. | 10:16 |
| jaosorior | d0ugal: I'm doing it for the rest of the tripleo projects as well (os-collect-config, os-refresh-config...etc) | 10:17 |
| d0ugal | jaosorior: cool, thanks - I'll stop :) | 10:18 |
| d0ugal | jaosorior: I just done tripleo-common/tripleoclient | 10:18 |
| abregman | hi. question on nodepool - does it take into account the available resources when spawning up VMs? | 10:18 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: Make py35 jobs voting for tripleo jobs https://review.openstack.org/410674 | 10:18 |
| AJaeger | dirk: POST_FAILURE means that ansible failed after the job succeed | 10:19 |
| *** dfflanders has quit IRC | 10:19 | |
| AJaeger | Often a CI problem | 10:19 |
| AJaeger | mmh, it could happen that the node get killed somehow. | 10:20 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: novajoin: Make py35 job voting https://review.openstack.org/410668 | 10:20 |
| AJaeger | dirk: you could recheck and check the streaming output | 10:20 |
| *** knangia has quit IRC | 10:20 | |
| abregman | hi. question on nodepool - does it take into account the available resources when spawning up VMs? | 10:20 |
| AJaeger | d0ugal, jaosorior : Thanks! | 10:20 |
| *** thorst_ has joined #openstack-infra | 10:21 | |
| dirk | AJaeger: are there logs available after that happened? | 10:21 |
| AJaeger | abregman: AFAIK only limit is number of machines | 10:21 |
| *** jesusaur has joined #openstack-infra | 10:21 | |
| AJaeger | dirk: let's chat directly, will come to your office in a few... | 10:21 |
| *** panda|Zz is now known as panda | 10:21 | |
| *** nadya has joined #openstack-infra | 10:22 | |
| abregman | AJaeger: I mean, if you want to create a VM and you don't have enough resources atm, what would happen? will it fail or there is a use of some sort of queue? | 10:22 |
| abregman | does it inspect the provider's resources usage? | 10:23 |
| AJaeger | abregman: AFAIK you should define the limit (= number of VMs to use maximal) in such a way that this does not happen ;) | 10:23 |
| AJaeger | abregman: but I'm not an expert, better ask later again | 10:23 |
| abregman | AJaeger: oh :/ | 10:23 |
| abregman | AJaeger: thanks! | 10:23 |
| *** tuanluong_ has quit IRC | 10:24 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/project-config: novajoin: Make py35 job voting https://review.openstack.org/410668 | 10:25 |
| *** strigazi_AFK is now known as strigazi | 10:26 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add identity v3 test accounts job https://review.openstack.org/410677 | 10:27 |
| *** sdake has joined #openstack-infra | 10:29 | |
| *** thorst_ has quit IRC | 10:29 | |
| strigazi | Hi all, in magnum, in the post_hook, I want to get the dns_nameserver and I do it with networkmanager. For that I need the physical device. Is GUEST_INTERFACE_DEFAULT the correct variable to get it from? | 10:29 |
| *** jesusaur has quit IRC | 10:30 | |
| *** ldnunes has joined #openstack-infra | 10:33 | |
| *** yanyanhu has quit IRC | 10:33 | |
| *** esikachev has quit IRC | 10:37 | |
| *** links has quit IRC | 10:38 | |
| *** links has joined #openstack-infra | 10:38 | |
| *** saibarspeis has joined #openstack-infra | 10:40 | |
| *** sreeram_v has quit IRC | 10:43 | |
| *** AntonChevychalov has joined #openstack-infra | 10:43 | |
| *** gongysh has quit IRC | 10:43 | |
| *** AntonChevychalov is now known as achevychalov | 10:43 | |
| *** sreeram_v has joined #openstack-infra | 10:43 | |
| *** sreeram__ has joined #openstack-infra | 10:44 | |
| *** sreeram_v has quit IRC | 10:48 | |
| *** sreeram__ has quit IRC | 10:49 | |
| *** wolverineav has joined #openstack-infra | 10:50 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Test do not merge https://review.openstack.org/410685 | 10:50 |
| *** matrohon has quit IRC | 10:55 | |
| *** tosky has joined #openstack-infra | 10:55 | |
| *** tpsilva has joined #openstack-infra | 10:56 | |
| *** dtardivel has joined #openstack-infra | 10:58 | |
| *** mmandel has joined #openstack-infra | 11:02 | |
| *** markvoelker has joined #openstack-infra | 11:02 | |
| *** eroux has quit IRC | 11:05 | |
| *** mmandel has quit IRC | 11:07 | |
| *** tphummel has quit IRC | 11:07 | |
| *** markvoelker has quit IRC | 11:07 | |
| *** sdague has joined #openstack-infra | 11:08 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** thorst_ has joined #openstack-infra | 11:16 | |
| *** jesusaur has joined #openstack-infra | 11:18 | |
| *** jaosorior has quit IRC | 11:19 | |
| *** thorst_ has quit IRC | 11:23 | |
| *** gongysh has joined #openstack-infra | 11:24 | |
| *** yamamoto has joined #openstack-infra | 11:26 | |
| *** gmann_ has joined #openstack-infra | 11:27 | |
| *** dtantsur|brb is now known as dtantsur | 11:28 | |
| *** mkoderer has joined #openstack-infra | 11:28 | |
| *** nadya has quit IRC | 11:28 | |
| *** tojuvone has left #openstack-infra | 11:30 | |
| *** jaosorior has joined #openstack-infra | 11:30 | |
| *** hashar is now known as hasharAway | 11:31 | |
| *** nicolasbock has joined #openstack-infra | 11:32 | |
| *** mhickey has joined #openstack-infra | 11:37 | |
| *** nileshsawant has joined #openstack-infra | 11:39 | |
| nileshsawant | Hello All, can someone please help me for creating Third party CI | 11:40 |
| nileshsawant | am facing issue for ruuunig cmd : nodepool image-upload all dpc | 11:40 |
| nileshsawant | it is faling | 11:40 |
| nileshsawant | am not getting anything in /var/log/nodepool/image/image.log | 11:41 |
| *** furlongm_ has joined #openstack-infra | 11:41 | |
| nileshsawant | which logs to refer ? | 11:41 |
| *** furlongm has quit IRC | 11:41 | |
| AJaeger | nileshsawant: best ask on #openstack-third-party-ci | 11:42 |
| nileshsawant | okay thanks | 11:44 |
| *** rhallisey has joined #openstack-infra | 11:44 | |
| *** ldnunes has quit IRC | 11:45 | |
| *** ldnunes has joined #openstack-infra | 11:45 | |
| *** akscram has joined #openstack-infra | 11:49 | |
| *** ssbarnea has joined #openstack-infra | 12:00 | |
| *** e0ne has joined #openstack-infra | 12:03 | |
| *** dave-mccowan has joined #openstack-infra | 12:04 | |
| *** maeker has joined #openstack-infra | 12:06 | |
| *** jkilpatr has joined #openstack-infra | 12:08 | |
| *** maeker has quit IRC | 12:10 | |
| *** thorst_ has joined #openstack-infra | 12:10 | |
| *** tqtran has joined #openstack-infra | 12:11 | |
| *** rtheis has joined #openstack-infra | 12:12 | |
| *** tqtran has quit IRC | 12:15 | |
| *** jkilpatr has quit IRC | 12:15 | |
| *** lucasagomes is now known as lucas-hungry | 12:17 | |
| *** thorst_ has quit IRC | 12:18 | |
| *** gongysh has quit IRC | 12:19 | |
| *** gcb has quit IRC | 12:20 | |
| *** deadnull has joined #openstack-infra | 12:20 | |
| *** dtantsur is now known as dtantsur|brb | 12:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 12:23 |
| *** gcb has joined #openstack-infra | 12:25 | |
| *** gcb has quit IRC | 12:25 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Drop Java Implemention codebase https://review.openstack.org/410728 | 12:25 |
| *** florianf has quit IRC | 12:27 | |
| *** shardy is now known as shardy_lunch | 12:27 | |
| *** pgadiya has quit IRC | 12:28 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Drop Java Implemention codebase https://review.openstack.org/410728 | 12:28 |
| *** ociuhandu has joined #openstack-infra | 12:29 | |
| *** jkilpatr has joined #openstack-infra | 12:29 | |
| *** vikrant has quit IRC | 12:29 | |
| *** florianf has joined #openstack-infra | 12:31 | |
| *** sdake has quit IRC | 12:31 | |
| *** savihou has quit IRC | 12:34 | |
| *** jimbaker has quit IRC | 12:34 | |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Add a job to run federated identity tests https://review.openstack.org/410405 | 12:36 |
| *** jimbaker has joined #openstack-infra | 12:38 | |
| *** e0ne has quit IRC | 12:39 | |
| *** pgadiya has joined #openstack-infra | 12:40 | |
| openstackgerrit | Julia Varlamova proposed openstack-infra/project-config: Add Keystone multinode Grenade job https://review.openstack.org/407428 | 12:41 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Ansibly the start of fresh logging https://review.openstack.org/402208 | 12:43 |
| *** jamielennox is now known as jamielennox|away | 12:43 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 12:43 |
| kfox1111 | question. can you reboot a node in a gate job? | 12:43 |
| *** jimbaker has quit IRC | 12:44 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Ansibly the setup of stack user https://review.openstack.org/403664 | 12:46 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Ansibly the setup of the tempest user https://review.openstack.org/403727 | 12:47 |
| *** jimbaker has joined #openstack-infra | 12:48 | |
| *** _Cyclone_ has quit IRC | 12:49 | |
| *** jamielennox|away is now known as jamielennox | 12:50 | |
| *** trown|outtypewww is now known as trown | 12:51 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Drop Java Implemention codebase https://review.openstack.org/410728 | 12:52 |
| openstackgerrit | Oksana Voshchana proposed openstack-infra/devstack-gate: TEST for multinode grenade devstack https://review.openstack.org/396679 | 12:52 |
| *** amoralej is now known as amoralej|lunch | 12:55 | |
| *** ijw has joined #openstack-infra | 12:56 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Monasca Gate Setup for ORM+PostgreSQL https://review.openstack.org/366617 | 12:57 |
| *** ssbarnea has quit IRC | 12:57 | |
| *** sdake has joined #openstack-infra | 12:57 | |
| *** sdake_ has joined #openstack-infra | 12:58 | |
| *** dtantsur|brb is now known as dtantsur | 12:59 | |
| *** ijw has quit IRC | 13:00 | |
| openstackgerrit | Merged openstack-infra/release-tools: Handle non-python libraries in announce.sh https://review.openstack.org/398013 | 13:01 |
| yuval | Hey, does anyone knows a way to update a project directory (Karbor) cached on a gate? Our current cached repo is causing issues because of gitattributes update | 13:01 |
| *** sdake has quit IRC | 13:02 | |
| *** pgadiya has quit IRC | 13:02 | |
| *** pradk has joined #openstack-infra | 13:02 | |
| *** sdake_ is now known as sdake | 13:02 | |
| *** jheroux has joined #openstack-infra | 13:03 | |
| *** markvoelker has joined #openstack-infra | 13:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 13:04 |
| *** thorst_ has joined #openstack-infra | 13:05 | |
| *** Hal has joined #openstack-infra | 13:06 | |
| *** Hal is now known as Guest63086 | 13:06 | |
| *** jimbaker has quit IRC | 13:06 | |
| *** hasharAway is now known as hashar | 13:07 | |
| *** lucas-hungry is now known as lucasagomes | 13:07 | |
| *** markvoelker has quit IRC | 13:08 | |
| *** _Cyclone_ has joined #openstack-infra | 13:08 | |
| *** jimbaker has joined #openstack-infra | 13:10 | |
| *** thorst_ has quit IRC | 13:13 | |
| *** jeckersb is now known as jeckersb_gone | 13:14 | |
| *** e0ne has joined #openstack-infra | 13:15 | |
| *** shardy_lunch is now known as shardy | 13:16 | |
| *** abregman is now known as abregman|mtg | 13:18 | |
| *** sdake_ has joined #openstack-infra | 13:20 | |
| *** edmondsw_ has joined #openstack-infra | 13:20 | |
| *** edmondsw_ has quit IRC | 13:20 | |
| *** gordc has joined #openstack-infra | 13:21 | |
| *** mrtenio-afk is now known as mrtenio | 13:22 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/elastic-recheck: Add e-r query for bug/1646779 https://review.openstack.org/406278 | 13:23 |
| *** sdake has quit IRC | 13:23 | |
| *** markvoelker has joined #openstack-infra | 13:24 | |
| *** allanice001 has quit IRC | 13:24 | |
| AJaeger | yuval: please explain the problem some more | 13:26 |
| yuval | AJaeger: we made a change to gitattributes to make sure rst files use LF and not CRLF. Seems like one filed stayed CRLF, causing 'git checkout master' to be dirty since git automatically converts it to LF | 13:28 |
| yuval | AJaeger: now, it seems like that rogue state is being cached in gate | 13:29 |
| *** sdake_ is now known as sdake | 13:29 | |
| yuval | AJaeger: so fetching the latest commit and checkout fails | 13:29 |
| *** arxcruz is now known as arxcruz|brb | 13:30 | |
| *** sreeram_v has joined #openstack-infra | 13:30 | |
| *** daemontool_ has quit IRC | 13:30 | |
| andreaf | AJaeger: hi - thanks for your review on https://review.openstack.org/#/c/410677 - I've got a question | 13:31 |
| andreaf | AJaeger: why is the devstack-periodic-master-jobs job-group not needed? I thought that's what generates the periodic jobs | 13:31 |
| *** rossella_s has quit IRC | 13:31 | |
| *** chlong has quit IRC | 13:32 | |
| yuval | AJaeger: you can see that here: http://logs.openstack.org/48/410548/1/check/gate-karbor-dsvm-fullstack-nv/6e2a319/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-12-14_06_25_25_044 | 13:32 |
| sreeram_v | Request your review + approval on this one https://review.openstack.org/#/c/410601 | 13:32 |
| *** rossella_s has joined #openstack-infra | 13:32 | |
| *** snarwade has quit IRC | 13:33 | |
| dhellmann | if folks are seeing release note build failures, it's because now that the liberty stable branches are gone we need to remove the liberty pages for release notes | 13:33 |
| dhellmann | we should probably update the EOL procedures to include that step. | 13:33 |
| *** sshnaidm is now known as sshnaidm|afk | 13:33 | |
| *** gmann_ has quit IRC | 13:34 | |
| kfox1111 | can you safely reboot in a gate job? | 13:36 |
| *** lbeliveau has quit IRC | 13:36 | |
| *** lbeliveau has joined #openstack-infra | 13:38 | |
| *** lindycoder has joined #openstack-infra | 13:39 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Implement Experimental Infra jobs for Puppet 4 testing https://review.openstack.org/410483 | 13:42 |
| EmilienM | clarkb: I might need your help on https://review.openstack.org/410483 to add the logic that deploy puppet4 in the nodes, not sure how you want to do it | 13:43 |
| *** esikachev has joined #openstack-infra | 13:43 | |
| *** abregman|mtg is now known as abregman | 13:44 | |
| *** rlandy has joined #openstack-infra | 13:45 | |
| *** snarwade has joined #openstack-infra | 13:46 | |
| *** jpena is now known as jpena|lunch | 13:47 | |
| *** Goneri has joined #openstack-infra | 13:50 | |
| *** Julien-zte has joined #openstack-infra | 13:51 | |
| AJaeger | andreaf: are you using a periodic job? | 13:51 |
| AJaeger | yuval: where's the change with the gitattributes? Do you ahve a link? | 13:52 |
| *** tlian has joined #openstack-infra | 13:52 | |
| *** srobert has joined #openstack-infra | 13:53 | |
| AJaeger | andreaf: missed that you add it to periodic queue - that is worth to mention in commit message. I'll comment | 13:53 |
| *** baoli has joined #openstack-infra | 13:55 | |
| *** Goneri has quit IRC | 13:55 | |
| AJaeger | dhellmann: so, those liberty pages will be removed from our webserver completely now. Is that really desired? | 13:55 |
| *** Goneri has joined #openstack-infra | 13:55 | |
| dhellmann | AJaeger : no idea. probably not? | 13:55 |
| dhellmann | it is possible to generate a static version of the page, rather than deleting it entirely | 13:56 |
| *** tongli has joined #openstack-infra | 13:56 | |
| dhellmann | although that will be harder now that the branches are removed | 13:56 |
| AJaeger | dhellmann: soon we will remove stale pages with every update. So, the liberty.html file will be removed. | 13:56 |
| dhellmann | there's the "how do I fix every broken build" question and the "what do we want to do for long term publishing of release notes question" | 13:57 |
| AJaeger | ... removed automatically from docs.o.o | 13:57 |
| dhellmann | I was only focused on the first today | 13:57 |
| AJaeger | dhellmann: ah ;) I was focused on the second :) | 13:57 |
| dhellmann | if we have eol tags, we can manually recreate the missing pages one project at a time | 13:57 |
| dhellmann | or maybe I can teach reno to scan from tags, I don't know | 13:57 |
| pabelanger | morning | 13:58 |
| dhellmann | the data we need is still present in the repositories, it's just not there in a way reno can find it today | 13:58 |
| AJaeger | dhellmann: sorry, can't help further on this one right now. Thanks for the heads-up. | 13:58 |
| dhellmann | AJaeger : ack, I'll give it some thought | 13:58 |
| *** krtaylor has joined #openstack-infra | 13:59 | |
| dmsimard | Was there any changes to locales on the images recently ? Seeing some obnoxious amount of spam since this week afaict http://logs.openstack.org/48/410448/4/check/gate-puppet-openstack-integration-4-scenario002-tempest-centos-7/18d2062/console.html#_2016-12-14_08_04_35_567382 | 13:59 |
| *** ldnunes has quit IRC | 14:00 | |
| *** hrybacki is now known as hrybacki|mtg | 14:01 | |
| pabelanger | AJaeger: abregman: nodepool has no queue option, it simply launches VMs. So, if your cloud is lacking resources, nodepool will schedule the launch but the VM will fail to be created, the process repeats until the VM is able to boot correctly | 14:01 |
| *** abregman is now known as abregman|mtg | 14:02 | |
| pabelanger | dmsimard: diskimage-builder will remove non en_US locales at build time, looks like a new package install is looking for the originals | 14:04 |
| dmsimard | pabelanger: perhaps related to 7.3 release | 14:05 |
| dmsimard | pabelanger: is dib building off of 7.3 since monday ? | 14:05 |
| pabelanger | likely, currently centos-7 DIBs are broken | 14:05 |
| dmsimard | amazing | 14:05 |
| pabelanger | dmsimard: http://paste.openstack.org/show/592349/ | 14:06 |
| pabelanger | need to ping somebody for that | 14:06 |
| dmsimard | is that from the ceph storage sig ? | 14:06 |
| dmsimard | amoralej|lunch, EmilienM: ^ is http://paste.openstack.org/show/592349/ the hammer EOL/hammer on 7.3 issue ? | 14:07 |
| *** kgiusti has joined #openstack-infra | 14:07 | |
| *** amoralej|lunch is now known as amoralej | 14:07 | |
| pabelanger | dmsimard: what is ceph storage sig? | 14:07 |
| amoralej | that's from epel | 14:07 |
| dmsimard | looks like 0.80 is ceph firefly, not hammer.. so nevermind | 14:07 |
| dmsimard | yuck, epel ? | 14:08 |
| pabelanger | yes, needed for devstack | 14:08 |
| pabelanger | ovs for example | 14:08 |
| dmsimard | what for ? | 14:08 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: move unmaintained chef cookbooks to openstack-attic https://review.openstack.org/410769 | 14:08 |
| dmsimard | eh, carrying RDO repository might be safer but that's a topic for another time | 14:08 |
| amoralej | anyway, if you install ceph storage sig repo, it will be installed from there as it has higher nvr | 14:09 |
| pabelanger | dmsimard: which repo? | 14:09 |
| pabelanger | amoralej: which URL is that? | 14:09 |
| dmsimard | pabelanger: http://mirror.centos.org/centos/7/storage/x86_64/ceph-hammer/ | 14:09 |
| amoralej | there are repo rpms to install the repo files | 14:09 |
| dmsimard | or the centos-release-ceph-hammer rpm | 14:09 |
| amoralej | but if posible move to jewel instead of hammer | 14:10 |
| dmsimard | HOWEVER, hammer is slated for EOL shortly so probably use ceph-jewel instead.. | 14:10 |
| amoralej | yeah | 14:10 |
| dmsimard | why are ceph deps preinstalled, though ? | 14:10 |
| pabelanger | not installed, we cache it | 14:10 |
| pabelanger | --download-only | 14:10 |
| dmsimard | oh. | 14:10 |
| pabelanger | for devstack | 14:10 |
| dmsimard | TIL devstack tests a ceph implementation | 14:11 |
| amoralej | TIL epel ships packages for ceph | 14:11 |
| dmsimard | amoralej: outdated ones at that | 14:11 |
| amoralej | they should be removed from EPEL as there are newer ones in storage sig | 14:12 |
| *** tqtran has joined #openstack-infra | 14:12 | |
| dmsimard | pabelanger: so, two things: 1) RDO carries OVS for each shipped release (that would be openvswitch-2.5.0-2.el7.x86_64.rpm for newton, for example), that's in http://mirror.centos.org/centos/7/cloud/x86_64/openstack-newton/ (in common) | 14:13 |
| pabelanger | now I need to remember how we enable epel7 | 14:13 |
| *** liusheng has quit IRC | 14:13 | |
| abregman|mtg | pabelanger: so if I have two builds and for each build, nodepool fails to create VMs, how it handles it? will it simply try to recreate without any order? | 14:13 |
| dmsimard | pabelanger: 2) A better source for Ceph (compared to epel) would be the CentOS ceph storage SIG which carries both hammer (soon EOL) and jewel | 14:13 |
| *** ldnunes has joined #openstack-infra | 14:14 | |
| amoralej | remember that if you install hammer repo, you still may have issues with 7.3, so my advise is to go with jewel, which is what we test in newton and master | 14:14 |
| *** liusheng has joined #openstack-infra | 14:14 | |
| pabelanger | dmsimard: I'll have to wait for clarkb to comment, he knows the epel dependencies better | 14:14 |
| *** Diziet has joined #openstack-infra | 14:14 | |
| dmsimard | pabelanger: sure, should we document this somewhere better than irc ? | 14:15 |
| pabelanger | abregman|mtg: on the next loop, they both should be launched, which ever one is created first on the cloud wins | 14:15 |
| abregman|mtg | pabelanger: too bad :( | 14:15 |
| yuval | AJaeger: https://review.openstack.org/409530 | 14:15 |
| pabelanger | abregman|mtg: how so? | 14:16 |
| abregman|mtg | pabelanger: well I want the one that got in first to get VMs before the later change | 14:16 |
| abregman|mtg | pabelanger: so some sort of queue in nodepool I guess | 14:16 |
| kfox1111 | anyone know if its safe to reboot the node in a gate job? | 14:16 |
| *** tqtran has quit IRC | 14:17 | |
| clarkb | kfox1111: it is not, this is now documented at http://docs.openstack.org/infra/manual/testing.html | 14:19 |
| pabelanger | kfox1111: no, nodepool will assume the node went offline and restart the job | 14:19 |
| clarkb | pabelanger: we enable epel in order to cache packages for devstack | 14:19 |
| clarkb | rdo is where weget ovs | 14:19 |
| Diziet | I have a problem with the 2min timeout on git.openstack.org. When cloning openstack-nova, my client spends more than 2mins pratting about looking for nonexistent files etc., and then the connection gets dropped. | 14:20 |
| pabelanger | clarkb: right, was digging into a package list, looks like ceph is broken in epel. And Red Hat folks seem to recommend no using epel, something about zero testing also | 14:20 |
| clarkb | I wantto say westopped caching packages on debuntu due to mirroring. If we feel we have propermirroring of centos repos we can likely do the same thereto work around thia | 14:20 |
| abregman|mtg | pabelanger: perhaps I didn't describe it properly. I'm asking what happens if you run several builds/jobs and you don't have enough resources on your cloud/provider. so what will nodepool do in such case? does it make sure that the first build/change gets the resources before everyone else? | 14:21 |
| clarkb | pabelanger: that seems like a sadargument | 14:21 |
| Diziet | It actually seems to take it about 2m48s | 14:21 |
| clarkb | our package repo isbroken but not our fault you shouldnt use that repo for that particular package? | 14:21 |
| Diziet | Is it intented that it is impossible to clone openstack-nova on a slow computer ? | 14:21 |
| clarkb | Diziet: no I think this is the first we have heard ofit | 14:22 |
| pabelanger | clarkb: ya, maybe we stop caching devstack too. I'd be okay to try that for a few days, see the impact | 14:22 |
| Diziet | I guess you have to be using -q etc. or you may get some progress output. | 14:22 |
| pabelanger | however, I can't see how devstack is working today on centos-7 | 14:22 |
| pabelanger | since ceph fails to resolve | 14:22 |
| clarkb | it may not bethe centos devstack jobs have always been in a sad state because they get less care and feeding | 14:23 |
| clarkb | and I am not sure it votes anywhere as a result | 14:24 |
| clarkb | or that we hve devstack jobs on centos with ceph enabled | 14:24 |
| pabelanger | abregman|mtg: nodepool only launches the VMs in the cloud, it is up to zuul to decide which jobs gets to run them. So, if you have limited resources in the cloud, each node that comes online, zuul will first allocate it to jobs in the pipeline with the higher priority. In our case, that is gate | 14:24 |
| *** dprince has joined #openstack-infra | 14:24 | |
| abregman|mtg | pabelanger: but who inspects the resources in the tenant to make sure the job can run. is zuul? nodepool? no one? | 14:25 |
| abregman|mtg | is it* | 14:25 |
| clarkb | (fwiw it can launch whatever nova can launch which uncludes VMs containers and baremetal) | 14:25 |
| pabelanger | clarkb: maybe we disable caching for rpms only? | 14:25 |
| clarkb | pabelanger: I thought we already did it for debs | 14:25 |
| Diziet | Hrm, stracing it, my `git clone' process seems to be sitting on the cpu. | 14:25 |
| pabelanger | clarkb: looking at cache-devstack/extra-data.d/55-cache-devstack-repos I see debs, rpms and ebuilds | 14:26 |
| clarkb | Diziet: and this is git clone https://git.openstack.org/openstack/nova | 14:26 |
| clarkb | ? | 14:26 |
| *** thorst_ has joined #openstack-infra | 14:26 | |
| pabelanger | clarkb: I'd be okay for skipping rpm, to start | 14:26 |
| Diziet | No, via the git protocol. git clone -q --mirror git://git.openstack.org/openstack/nova.git nova </dev/null >log 2>&1 | 14:26 |
| abregman|mtg | pabelanger: so zuul decides to run job1 -> nodepool launches VMs -> fails. in the same time zuul runs job2 -> nodepool launches VMs -> also fails. now what? nodepool simply keeps trying to relaunch the VMs? every second? minute? for both jobs? | 14:27 |
| clarkb | pabelanger: I would needto check git logs but I seem to recall we changed our packge caching recently | 14:27 |
| pabelanger | abregman|mtg: nobody inspects, once the node is online in the cloud, is it then passed to zuul to be used | 14:27 |
| abregman|mtg | pabelanger: k. so zuul runs the job only after getting a node from nodepool? | 14:27 |
| *** jcoufal has joined #openstack-infra | 14:27 | |
| pabelanger | abregman|mtg: correct | 14:28 |
| abregman|mtg | pabelanger: awesome. thank you :) | 14:28 |
| abregman|mtg | pabelanger++ | 14:28 |
| Diziet | clarkb: I think the root cause is the enormous number of refs in that repo. | 14:28 |
| *** yamahata_ has joined #openstack-infra | 14:28 | |
| Diziet | On a system where I successfully cloned it: | 14:28 |
| Diziet | chiark:nova> git show-ref |wc -l | 14:28 |
| Diziet | 125614 | 14:28 |
| *** hongbin has joined #openstack-infra | 14:29 | |
| Diziet | The git client spends too long thinking about all those refs. | 14:29 |
| clarkb | yes its our biggest repo | 14:29 |
| *** jeckersb_gone is now known as jeckersb | 14:29 | |
| Diziet | Would it be possible to increase the timeout ? | 14:29 |
| clarkb | shouldbe possible, guessing its our haproxy in front of git daemondoing it | 14:29 |
| Diziet | I think (barring trying to get an improvement to the git client and then waiting for everyone to have the new client) that's probably a useful improvement. | 14:29 |
| Diziet | clarkb: Quite possibly. | 14:29 |
| Diziet | My tcpdump suggests the timeout is exactly 2 minutes. | 14:30 |
| Diziet | 14:10:54.655379 IP newcastle.e.test-lab.xenproject.org.46648 > git.openstack.org.git: Flags [.], ack 8767084, win 8530, options [nop,nop,TS val 3022124612 ecr 3602466320], length 0 | 14:30 |
| Diziet | 14:12:54.488119 IP git.openstack.org.git > newcastle.e.test-lab.xenproject.org.46648: Flags [F.], seq 8767084, ack 64, win 15, options [nop,nop,TS val 3602586164 ecr 3022124612], length 0 | 14:30 |
| pabelanger | clarkb: yes, that change was only for bindep, we didn't update devstack caching | 14:30 |
| *** fguillot has joined #openstack-infra | 14:30 | |
| clarkb | Diziet: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/git.pp#n96 is where we configure haproxy | 14:31 |
| clarkb | pabelanger: oh gotcha bcause devstack is its own package listing mechanism | 14:31 |
| *** asettle has quit IRC | 14:31 | |
| clarkb | in that case I think we can just stop doing devstack for debs and rpms amd watch it | 14:31 |
| Diziet | 'timeout' => [ ... 'client 2m', 'server 2m', ... | 14:32 |
| Diziet | clarkb: So err I should send a patch to that file ? | 14:32 |
| *** asettle has joined #openstack-infra | 14:32 | |
| clarkb | Diziet: yup | 14:32 |
| Diziet | I wouldn't be in a position to test it of course... | 14:33 |
| AJaeger | yuval: I see - never heard about this before and need to defer to others on that. | 14:33 |
| Diziet | But, sure. | 14:33 |
| *** srobert has quit IRC | 14:34 | |
| clarkb | Diziet: just make sure you mention in commit message the timeout is too low for cloning nova with less cpu | 14:34 |
| Diziet | Right, silldo. | 14:35 |
| Diziet | I mean, willdo. | 14:35 |
| Diziet | Where should I send the patch ? | 14:35 |
| clarkb | Diziet: you can do git clone git://git01.opemstack.org:29418/openstack/nova to get an idea for how much time you need | 14:35 |
| clarkb | filler: same as any other opemstack project, to gerrit (you can use git review to make this easy) | 14:36 |
| dmsimard | clarkb: so Ceph is the only thing retrieved from EPEL ? | 14:37 |
| Diziet | YM openstack not opemstack :-). | 14:37 |
| pabelanger | clarkb: ++, let me get a patch up | 14:37 |
| clarkb | dmsimard: no, everything in devstacks package lists are fetched, some willbe found in epel | 14:37 |
| clarkb | Diziet: ya sorry, early morning phone typing | 14:37 |
| * Diziet <- already had lunch | 14:38 | |
| dmsimard | clarkb: I'd be curious to disable EPEL and see what breaks, if anything | 14:38 |
| Diziet | I may still need more coffee though. | 14:38 |
| dmsimard | especially since RDO will be enabled (to get OVS amongst other things) | 14:38 |
| *** fguillot has quit IRC | 14:38 | |
| dmsimard | EPEL is wildly unreliable and can do stupid things like bump two major version of a package just like that | 14:38 |
| Diziet | clarkb: Was that instruction to use gerrit directed at me ? I haven't every submitted anything to any gerrit before, so I would appreciate a pointer or two. I read README.me and looked at http://docs.openstack.org/infra/system-config (which I don't think covers this). | 14:40 |
| *** fguillot has joined #openstack-infra | 14:41 | |
| *** marst has joined #openstack-infra | 14:41 | |
| clarkb | Diziet: it was, let me find our central doc | 14:42 |
| *** sdake_ has joined #openstack-infra | 14:42 | |
| clarkb | Diziet: http://docs.openstack.org/infra/manual/developers.html#account-setup note you dont need the clastuff for that repo iirc | 14:42 |
| *** gouthamr has joined #openstack-infra | 14:43 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: retire unmaintained cookbook projects and end project gating https://review.openstack.org/410769 | 14:44 |
| *** andymaier has joined #openstack-infra | 14:44 | |
| *** sdake has quit IRC | 14:45 | |
| clarkb | Diziet: I am also happy to push a patch when at a real computer if that is easier. Let me know how long it takes that machine to do the clone directly from backend so we know how big a bump we need. But also happy to help with gerrit if you eant to get it set up and push yourself. Either way | 14:45 |
| *** thorst_ has quit IRC | 14:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Stop caching devstack packages https://review.openstack.org/410788 | 14:46 |
| Diziet | I think, if you're willing, I would prefer you to submit it to gerrit. Should I email you the output of git-format-patch ? | 14:46 |
| Diziet | clarkb: ^ | 14:46 |
| *** jbernard has quit IRC | 14:46 | |
| yuval | AJaeger: thanks :) | 14:46 |
| Diziet | I'm just writing the commit message now. | 14:46 |
| *** jbernard has joined #openstack-infra | 14:46 | |
| AJaeger | yuval: best ask here in the channel for help. I hope clarkb or fungi can help... | 14:47 |
| clarkb | Diziet: or put it in a paste service and share here | 14:47 |
| *** sdake_ is now known as sdkae | 14:47 | |
| pabelanger | clarkb: fungi: ^ for your eyes, 410788. Currently un-tested, going to see if I can get a local build going again to validate it | 14:47 |
| clarkb | I believe git amwill accept from any old file | 14:47 |
| fungi | yuval: AJaeger: help with what? | 14:48 |
| *** sdkae is now known as sdake | 14:48 | |
| *** tongli has quit IRC | 14:49 | |
| yuval | Hey clarkb, fungi, maybe you can assist. Is there a way to update a project directory (Karbor) cached on a gate? Our current cached repo is causing issues because the last commit has gitattributes updated to ensure newlines are LF, while there is a rogue file with CRLF. As a consequence, karbor's dir cached on the gate is modified, and checking out any other branch is not possible | 14:49 |
| yuval | clarkb: fungi: you can see that here: http://logs.openstack.org/48/410548/1/check/gate-karbor-dsvm-fullstack-nv/6e2a319/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-12-14_06_25_25_044 | 14:50 |
| Diziet | clarkb: http://paste.debian.net/902177/ | 14:50 |
| yuval | I tried reverting gitattributes back, but it doesn't matter as the directory is already on master with modifications made automatically by git (because of the gitattributes change) | 14:50 |
| yuval | Maybe there is a way to update Karbor's dir cached on the gate, or removing it and letting devstack-gate reclone | 14:51 |
| dhellmann | AJaeger : https://review.openstack.org/410792 deals with EOL branches in reno | 14:52 |
| *** mtanino has joined #openstack-infra | 14:52 | |
| *** thorst_ has joined #openstack-infra | 14:52 | |
| openstackgerrit | Saverio Proto proposed openstack/diskimage-builder: Support sysv init system used by Debian Wheezy https://review.openstack.org/409866 | 14:53 |
| fungi | yuval: so when devstack-gate clones karbor right now, git is instructing it to modify files in the work tree and leave those modifications staged? | 14:54 |
| fungi | yuval: do you have a pointer to the change you merged which got it into this state? | 14:54 |
| yuval | fungi: https://review.openstack.org/409530 | 14:54 |
| *** jpena|lunch is now known as jpena | 14:54 | |
| *** sreeram_v has quit IRC | 14:55 | |
| clarkb | if there is no easy non admin fix we can force merge a revert of ^ | 14:55 |
| yuval | fungi: sorry, https://review.openstack.org/#/c/409757/ | 14:55 |
| fungi | yeah, that may be a faster solution for now | 14:55 |
| yuval | fungi: right now devstack-gate seems to find karbor dir on the gate, so it doesn't clone it | 14:56 |
| yuval | I tried removing the gitattributes rule causing the issue, but git refuses to checkout anything, because it is in a modified state over an older master with that issue | 14:57 |
| *** links has quit IRC | 14:57 | |
| fungi | devstack-gate clones from a cached copy baked into the filesystem of our test instance images at /opt/git/openstack/karbor into /opt/stack/new/karbor and then updates its git remote to git.openstack.org and pulls any outstanding changes from there | 14:57 |
| clarkb | dmsimard: regardless of how versions bump I would expect any rpm repo to properly suplort downloading a package | 14:58 |
| clarkb | dmsimard: and failure to do that is a bug no? | 14:58 |
| yuval | the cached copy is in the problematic state | 14:58 |
| fungi | after that it fetches a precomputer git ref from one of our zuul mergers and attempts to checkout that ref | 14:58 |
| fungi | er, precomputed | 14:58 |
| *** eroux has joined #openstack-infra | 14:58 | |
| dmsimard | clarkb: it's a good question and not one I have an answer for | 14:58 |
| dmsimard | --download-only shouldn't care about installed (or not) packages | 14:59 |
| fungi | yuval: if you can't update as a straight fastforward from an earlier version of the branch (such as the copy we've cached), then i'd say the problem is the got configuration not the cache | 14:59 |
| fungi | s/got/git/ | 14:59 |
| *** Swami has joined #openstack-infra | 14:59 | |
| dmsimard | clarkb: perhaps dnf (future yum with backwards compat) doesn't have that issue but I have no idea. | 14:59 |
| *** mmandel has joined #openstack-infra | 14:59 | |
| fungi | yuval: we can force merge a revert of that .gitattributes change to get things back into working order again | 14:59 |
| yuval | fungi: but the problematic git configuration is in the cached copy, making any checkout impossible | 15:00 |
| fungi | if devstack-gate is having trouble using existing clones of the repo, then anyone else who already cloned it before this is having similar issues | 15:00 |
| fungi | we should try to undo it | 15:00 |
| fungi | devstack-gate is merely emulating developer workflows for testing modifications to services under devstack, after all | 15:01 |
| *** jlk has quit IRC | 15:01 | |
| yuval | fungi: I agree that this is a problem for anyone who cloned or checked out that commit | 15:01 |
| *** hrybacki|mtg is now known as hrybacki | 15:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 15:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/360445 | 15:02 |
| yuval | fungi: if we could force merge or even delete the directory in the cache, which as far as I understand, will be updated when that flow will run again | 15:02 |
| *** jlk has joined #openstack-infra | 15:03 | |
| *** jlk has quit IRC | 15:03 | |
| *** jlk has joined #openstack-infra | 15:03 | |
| *** hurgleburgler has joined #openstack-infra | 15:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/410801 | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/410802 | 15:03 |
| fungi | we can't do anything directly to the cached git repos. they're baked into worker images that get refreshed and uploaded to glance in numerous service providers daily | 15:03 |
| yuval | fungi: are they updated by recloning or by pullin? (first will work, second will probably fail) | 15:05 |
| yuval | *pulling | 15:05 |
| fungi | pulling, like pretty much every developer would do | 15:05 |
| fungi | recloning is extremely wasteful, so we avoid it for the same reason anyone would | 15:06 |
| clarkb | we can force a reclone by removing the repo from the dib builder caches though | 15:06 |
| clarkb | but otherwise it will be anupdate like anyone else | 15:06 |
| fungi | right, but it still won't change on the test nodes until new images get created and uploaded | 15:07 |
| clarkb | yup | 15:07 |
| yuval | sounds to me like that pull will fail | 15:07 |
| *** lindycoder has quit IRC | 15:07 | |
| yuval | well, if we remove it, devstack-gate will just clone karbor until the new images will be created | 15:07 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: retire unmaintained cookbook projects and end project gating https://review.openstack.org/410769 | 15:08 |
| fungi | yuval: clarkb wasn't talking about removing it to get devstack-gate to reclone it | 15:08 |
| yuval | (not familiar with dib builder) | 15:08 |
| fungi | he was talking about removing it from the cache used by our image build process | 15:08 |
| yuval | I see | 15:08 |
| fungi | (a cache which is used by the process which creates the on-disk cache, so a second-order cache basically) | 15:08 |
| *** Rockyg has joined #openstack-infra | 15:08 | |
| fungi | to prevent teh same problem which is clogging devstack-gate from also clogging our image update process | 15:09 |
| jklare | AJaeger still around? | 15:09 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: retire unmaintained cookbook projects and end project gating https://review.openstack.org/410769 | 15:10 |
| *** psachin has quit IRC | 15:11 | |
| *** EricGonczer_ has joined #openstack-infra | 15:11 | |
| *** chem has quit IRC | 15:13 | |
| *** gouthamr has quit IRC | 15:13 | |
| *** spzala has joined #openstack-infra | 15:13 | |
| *** chem has joined #openstack-infra | 15:13 | |
| fungi | yuval: so if we merged a revert of https://review.openstack.org/409757 bypassing gate jobs, you don't think that would get things back on track? | 15:13 |
| *** sshnaidm|afk is now known as sshnaidm | 15:13 | |
| *** esberglu has joined #openstack-infra | 15:14 | |
| fungi | alternatively, maybe devstack-gate should do a reset --hard before checking out $ZUUL_REF to undo any staged changes created by application of git attributes | 15:15 |
| *** gouthamr has joined #openstack-infra | 15:15 | |
| yuval | fungi: I already merged a remove or rst from gitattribtues, and that didn't help, because the cached repo has unstaged changes | 15:15 |
| yuval | fungi: that solution sounds good | 15:15 |
| fungi | because that strikes me as the bigger issue, that updating .gitattributes can wedge it in the first place | 15:16 |
| *** EricGonczer_ has quit IRC | 15:16 | |
| AJaeger | jklare: yes | 15:16 |
| yuval | fungi: maybe git is making the crlf -> lf conversion after the reset as well | 15:17 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 15:17 |
| *** EricGonczer_ has joined #openstack-infra | 15:18 | |
| yuval | fungi: seems so :( | 15:19 |
| *** tsg has joined #openstack-infra | 15:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Git reset before checkout https://review.openstack.org/410810 | 15:20 |
| fungi | yuval: there ^ i think | 15:20 |
| AJaeger | dhellmann: thanks - but now you forced us to review the whole stack;) And I won't have time for that right now... | 15:20 |
| fungi | yuval: try to depends-on that with a karbor change and see if the dsvm jobs start passing | 15:21 |
| *** chlong has joined #openstack-infra | 15:21 | |
| *** tosky has quit IRC | 15:22 | |
| yuval | fungi: I already merged the revert and submitted a patch based on it, if I understand you correctly | 15:23 |
| yuval | fungi: that didn't help | 15:23 |
| fungi | yuval: i was talking about my "Git reset before checkout" change to devstack-gate there | 15:24 |
| fungi | https://review.openstack.org/410810 | 15:24 |
| AJaeger | yuval: "Depends-On: I..." of 410810 - see http://docs.openstack.org/infra/manual/developers.html#cross-repository-dependencies | 15:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 15:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/107597 | 15:24 |
| yuval | AJaeger: fungi: will do. By the way: 'git checkout -f ...' works | 15:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 15:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 15:25 |
| fungi | yuval: that could be a cleaner alternative to my proposed change (fewer git calls) | 15:26 |
| fungi | i didn't think about that option | 15:26 |
| yuval | fungi: trying to think about unwanted consequences of such change | 15:26 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: neutron: switch experimental centos job to single node https://review.openstack.org/410814 | 15:26 |
| dhellmann | AJaeger : yes, reno doesn't have branches yet so there was no where else to put the patch | 15:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Increase git daemon haproxy timeout https://review.openstack.org/410816 | 15:27 |
| AJaeger | dhellmann: hope dims and ttx can review as well... | 15:27 |
| clarkb | Diziet: ^ I had to amend it to add the gerrit change id, otherwise that is as provided | 15:27 |
| fungi | yuval: well, we already git reset --hard right after the checkout there anyway | 15:27 |
| dims | AJaeger : y, am trying :) | 15:28 |
| *** martinkopec has quit IRC | 15:28 | |
| AJaeger | dims: Sorry, no time today ;( | 15:29 |
| dims | ack AJaeger | 15:29 |
| *** gluytium_ has quit IRC | 15:29 | |
| *** baoli has quit IRC | 15:29 | |
| yuval | fungi: the checkout itself is failing, before the git reset --hard | 15:30 |
| yuval | fungi: btw, if the component responsible for updating the images (daily?) is using pull/checkout, it needs to handle that as well | 15:31 |
| fungi | yuval: right, i was merely saying that discarding before or during checkout is perfectly safe since we discard immediately after as well | 15:31 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Git checkout more forcefully https://review.openstack.org/410810 | 15:31 |
| fungi | updated to use -f ^ | 15:31 |
| fungi | yuval: and yes it's possible the source repos element in diskimage-builder already does but i'm digging in to confirm that now | 15:32 |
| *** rbrndt has joined #openstack-infra | 15:33 | |
| Diziet | clarkb: Thanks! I will keep an eye on that. Please ping me here or by email if more is needed from me. | 15:33 |
| openstackgerrit | Merged openstack/diskimage-builder: elements: dib-python: Add python2 as the default version for openSUSE https://review.openstack.org/410639 | 15:33 |
| *** eharney has joined #openstack-infra | 15:35 | |
| yuval | fungi: thanks, sorry for the inconvenience :( | 15:35 |
| *** annegentle has joined #openstack-infra | 15:35 | |
| fungi | yuval: clarkb: looks like we're safe in dib because it's just using fetch heads and tags, no pull or checkout http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/source-repositories/extra-data.d/98-source-repositories#n142 | 15:36 |
| *** Ravikiran_K has quit IRC | 15:37 | |
| fungi | also does a reset --hard FETCH_HEAD | 15:37 |
| *** zz_dimtruck is now known as dimtruck | 15:37 | |
| clarkb | fungi: cool | 15:38 |
| fungi | anyway, all the actions it takes look like they should ignore/discard local changes in the work tree if there are any | 15:38 |
| *** Guest63086 has quit IRC | 15:38 | |
| *** gomarivera has joined #openstack-infra | 15:38 | |
| fungi | yuval: no apologies needed. you helped us find a corner-case bug in devstack-gate | 15:39 |
| *** adrian_otto has joined #openstack-infra | 15:39 | |
| *** inc0 has joined #openstack-infra | 15:39 | |
| *** arxcruz|brb is now known as arxcruz | 15:39 | |
| *** annegentle has quit IRC | 15:40 | |
| *** eharney has quit IRC | 15:40 | |
| *** lindycoder has joined #openstack-infra | 15:40 | |
| *** annegentle has joined #openstack-infra | 15:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Convert glance parts of task test to requests_mock https://review.openstack.org/410425 | 15:42 |
| pabelanger | fungi: clarkb: just tested https://review.openstack.org/#/c/410788 locally, image built properly | 15:42 |
| *** gomarivera has quit IRC | 15:42 | |
| *** nadya has joined #openstack-infra | 15:44 | |
| *** thorst_ has quit IRC | 15:44 | |
| *** dizquierdo has joined #openstack-infra | 15:45 | |
| *** NobodyCam is now known as WeAreAllMiLan | 15:46 | |
| *** dave-mccowan has quit IRC | 15:46 | |
| *** nileshsawant has quit IRC | 15:47 | |
| *** WeAreAllMiLan is now known as NobodyCam | 15:48 | |
| pabelanger | clarkb: fungi: so, interesting fact, pretty sure download-pkgs script for ubuntu-xenial is broken today | 15:49 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add identity v3 test accounts job https://review.openstack.org/410677 | 15:49 |
| pabelanger | http://nodepool.openstack.org/dib.ubuntu-xenial.log | 15:49 |
| pabelanger | it is old, but you can see it fails to download | 15:49 |
| *** Sukhdev has joined #openstack-infra | 15:50 | |
| *** jkilpatr_ has joined #openstack-infra | 15:50 | |
| pabelanger | still fails on nb01.o.o | 15:50 |
| clarkb | pabelanger: ah ya so I think the problem there is we ended up with all packages on one line | 15:51 |
| clarkb | when we intended to do them one by one to allow errors to happen (hence the || true) | 15:51 |
| *** ralonsoh_ has joined #openstack-infra | 15:51 | |
| pabelanger | trusty is working | 15:51 |
| pabelanger | and only 1 branch in precise | 15:51 |
| clarkb | ya its because the packages listed are no longer in xenial | 15:51 |
| clarkb | like stud | 15:51 |
| pabelanger | yup | 15:51 |
| *** ralonsoh has quit IRC | 15:51 | |
| clarkb | so if we wanted to fix that we'd make the packages file put one package per line | 15:51 |
| *** jkilpatr has quit IRC | 15:52 | |
| *** trown is now known as trown|outtypewww | 15:52 | |
| clarkb | or somehow tell apt-get to ignore missing | 15:52 |
| *** annegentle has quit IRC | 15:52 | |
| clarkb | oh wait we already tell it to ignore missing | 15:52 |
| *** csomerville has joined #openstack-infra | 15:52 | |
| *** claudiub has quit IRC | 15:52 | |
| *** mmandel has quit IRC | 15:53 | |
| *** mmandel has joined #openstack-infra | 15:53 | |
| *** john-dav_ has quit IRC | 15:53 | |
| *** eharney has joined #openstack-infra | 15:53 | |
| clarkb | I think if we fixed openjdk-7-jre-headless to use the meta package for whatever java is current then we'd largely fix it | 15:54 |
| *** john-davidge has joined #openstack-infra | 15:54 | |
| * clarkb digs up devstack to push a patch for completeness even if we don't end up using it | 15:54 | |
| pabelanger | Ya, I keep meaning to poke at bindep support for devstack too | 15:54 |
| yuval | fungi: that seems to solve the issue, karbor's test seems to run (still did not complete) | 15:55 |
| yuval | fungi: thanks! | 15:55 |
| mordred | yuval: yay! | 15:55 |
| fungi | yuval: awesome, hopefully we'll get it merged in short order and prevent anyone else shooting themselves in teh foot that way | 15:55 |
| *** cody-somerville has quit IRC | 15:55 | |
| mordred | clarkb, pabelanger: think it would be worth jiggering that element to _actually_ do package per line? | 15:55 |
| yuval | fungi: just wondering if --force will not mask any other problems we might encounter | 15:56 |
| mordred | pkgs.write(" ".join(branch_data['debs']) + "\n") | 15:56 |
| fungi | yuval: well, we're using reset --hard right after, so all it's going to mask is issues which would prevent checkout after clone | 15:56 |
| mordred | we do not, in fact, do one package per line - but it would be trivial to make it do so | 15:56 |
| pabelanger | mordred: maybe, but we are also considering: https://review.openstack.org/#/c/410788/ | 15:57 |
| *** rkukura has joined #openstack-infra | 15:57 | |
| mordred | pabelanger: oh! I'd love to try that | 15:57 |
| pabelanger | same :) | 15:58 |
| mordred | fungi: ^^ | 15:58 |
| clarkb | pabelanger: https://review.openstack.org/#/c/410827/ | 15:59 |
| *** rkukura has quit IRC | 15:59 | |
| *** hieulq__ has joined #openstack-infra | 15:59 | |
| *** rkukura has joined #openstack-infra | 15:59 | |
| clarkb | also ^ will fix it if we go back to caching | 15:59 |
| clarkb | even without package per line | 15:59 |
| pabelanger | clarkb: +1 | 16:00 |
| *** tongli has joined #openstack-infra | 16:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Emit packages to cache one per line https://review.openstack.org/410830 | 16:00 |
| *** annegentle has joined #openstack-infra | 16:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Emit packages to cache one per line https://review.openstack.org/410830 | 16:01 |
| mordred | that's just for completeness | 16:01 |
| *** shamail has joined #openstack-infra | 16:01 | |
| *** gouthamr has quit IRC | 16:01 | |
| *** thorst_ has joined #openstack-infra | 16:02 | |
| *** adrian_otto has quit IRC | 16:02 | |
| pabelanger | ++ | 16:03 |
| yuval | fungi: clarkb: I have to go for today, thanks for the help :) | 16:03 |
| *** srobert has joined #openstack-infra | 16:03 | |
| pabelanger | mordred: jeblair: clarkb: fungi: anybody want to pull the trigger on https://review.openstack.org/#/c/410461/ switching back to master for nodepool? Able to support the effort | 16:03 |
| fungi | yuval: thanks for bringing it to our attention and helping us come up with a solution | 16:03 |
| pabelanger | also want to restart nodepool-builder to pick up latest fixes | 16:04 |
| *** yuval has quit IRC | 16:04 | |
| clarkb | pabelanger: I expect jeblair too as soon as he is around | 16:04 |
| clarkb | (which is why I haven't added it to my list) | 16:04 |
| *** gouthamr has joined #openstack-infra | 16:04 | |
| clarkb | did manage to fix the centos gettext ruby version issues though | 16:04 |
| fungi | pabelanger: i went ahead and approved the change at least | 16:04 |
| *** adrianofr_ has joined #openstack-infra | 16:04 | |
| fungi | pabelanger: so it should be in place as soon as you're ready to do the restart | 16:05 |
| *** gouthamr has quit IRC | 16:06 | |
| mordred | clarkb: wow. you fixed centos gettext ruby version issues. that seems like a morning | 16:07 |
| clarkb | mordred: it was yesterday evening. I wrote ruby | 16:07 |
| * mordred hands clarkb a morning beer | 16:07 | |
| clarkb | not very much ruby but I think it counts | 16:07 |
| mordred | clarkb: did you put vertical bars around a variable? | 16:08 |
| openstackgerrit | Merged openstack-infra/system-config: Switch nodepool and builder to master branch https://review.openstack.org/410461 | 16:12 |
| fungi | have to fence in those variables so they don't escape | 16:13 |
| *** eroux has quit IRC | 16:13 | |
| *** EricGonczer_ has quit IRC | 16:13 | |
| clarkb | mordred: I did not | 16:13 |
| *** rcernin has quit IRC | 16:14 | |
| kfox1111 | fungi: can you reboot a vm from witin a job? | 16:14 |
| clarkb | kfox1111: you cannot | 16:14 |
| *** dizquierdo has quit IRC | 16:14 | |
| clarkb | http://docs.openstack.org/infra/manual/testing.html documents this | 16:15 |
| kfox1111 | what about for the slave in a 2 node job? | 16:16 |
| fungi | clarkb: per your suggestion on 408284 i can implement that in a followup change if you still think it's a good idea, but right now it sort of protects puppet from crashing apache and taking down other unrelated vhosts if that module isn't loaded for some reason | 16:16 |
| clarkb | kfox1111: non primary nodes you should eb able to | 16:17 |
| *** EricGonczer_ has joined #openstack-infra | 16:17 | |
| kfox1111 | I'd like to simulate a power failure to see if things recover properly. | 16:17 |
| fungi | kfox1111: it may work for a subnode right now, but there's a good chance that's not a case we've accounted for in the zuul v3 design | 16:17 |
| clarkb | kfox1111: for that wouldn't it be easier to just kill processes? | 16:17 |
| *** dave-mccowan has joined #openstack-infra | 16:18 | |
| kfox1111 | clarkb: its a very complicated system. loop back volumes, docker containers, ip network namespaaces, etc. | 16:18 |
| kfox1111 | not sure I can adiquately always unwind to a pure power on state without a reboot. | 16:18 |
| *** Goneri has quit IRC | 16:18 | |
| fungi | oh, so you want to test the boot process, rather than merely loss of service? | 16:18 |
| kfox1111 | fungi: any chance that can be added to the v3 design? :) | 16:18 |
| kfox1111 | fungi: so... | 16:19 |
| sileht | AJaeger, yolanda, fungi https://review.openstack.org/#/c/405142/ have been merged yesterday and I still don't see the new jobs, I can't figure why, can you take a look ? | 16:19 |
| kfox1111 | I stand up a working openstack cloud in a vm with kolla-kubernetes. | 16:19 |
| kfox1111 | I want to then reboot the node, and see if the system comes back up and continues to work. | 16:19 |
| fungi | kfox1111: given how in v3 there are no multi-node constructs handled directly by nodepool and are instead assembled by zuul, so i'm not sure what the impact might be of one of them going offline | 16:19 |
| mordred | kfox1111, fungi: in v3 rebooting test should should be trivial | 16:19 |
| *** jaosorior has quit IRC | 16:20 | |
| fungi | mordred: oh, cool so we actually designed around the possibility of rebooting all nodes for a job in zuul v3? | 16:20 |
| mordred | the ansible driving the tests will be on the zuul launcher - so as long as you add a wait_for into the playbook, it should "just work" | 16:20 |
| *** jaosorior has joined #openstack-infra | 16:20 | |
| kfox1111 | hmm.. | 16:20 |
| mordred | fungi: I do not believe we explicitly designed for it - but the design does account for it | 16:20 |
| fungi | aha. i didn't consider that but you're right that makes it far cleaner to support | 16:20 |
| kfox1111 | when's v3 landing? | 16:21 |
| kfox1111 | (roughly) | 16:21 |
| mordred | kfox1111: as soon as it's ready | 16:21 |
| kfox1111 | (should I start with existing gate, or wait for v3?) | 16:21 |
| fungi | probably q1/q2 2017 but no clear timeline | 16:21 |
| mordred | kfox1111: (not very likely before march) | 16:21 |
| fungi | i would start with the subnode option for now | 16:21 |
| mordred | yup | 16:21 |
| kfox1111 | k. thx. | 16:21 |
| mordred | I agree | 16:21 |
| fungi | great strides to zuul v3 are being made daily, but it's a long road | 16:21 |
| mordred | and now a long and winding road is playing in my head | 16:22 |
| kfox1111 | :) | 16:22 |
| kfox1111 | will zuul v3 support 3+ node jobs? | 16:23 |
| pabelanger | 37 node jobs! | 16:23 |
| clarkb | kfox1111: it will support arbitrary numbers up to some limit likely determiend by size of clouds | 16:23 |
| kfox1111 | nice. | 16:23 |
| clarkb | kfox1111: that said current system does too just less flexibily | 16:23 |
| clarkb | you can have 3 node job today as long as you have working 2 node job | 16:23 |
| kfox1111 | looking forward to that. :) | 16:23 |
| *** filler has quit IRC | 16:24 | |
| kfox1111 | clarkb: oh, really? sweet. | 16:24 |
| kfox1111 | I've got a working 2 node job | 16:24 |
| clarkb | yes neutron and tripleo are doing it now iirc | 16:24 |
| kfox1111 | and would like to do 3. would make things way easier in some ways. | 16:24 |
| pabelanger | clarkb: it would be interesting experiment to do multi-node jobs across clouds. Even to see if work once | 16:24 |
| *** andymaier has quit IRC | 16:24 | |
| clarkb | pabelanger: I worry that the networking would get unhappy | 16:24 |
| kfox1111 | clarkb: thanks. I'll look at ttripleo/neutron for examples. :) | 16:24 |
| mmandel | Hi, my change set https://review.openstack.org/#/c/345626/ needs one more +2 and a W+1 to get merged, then a dependent change can get merged. Can someone please take a look at it? | 16:24 |
| pabelanger | clarkb: indeed | 16:24 |
| *** nicolasbock has quit IRC | 16:25 | |
| clarkb | pabelanger: overlays between san jose and gra1 could be :( | 16:25 |
| *** Goneri has joined #openstack-infra | 16:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Redirect to HTTPS for governance.openstack.org https://review.openstack.org/408284 | 16:25 |
| fungi | kfox1111: worth keeping in mind it still makes sense to optimize for the smallest number of nodes because every additional node means more chance of outside factors killing your jobs | 16:25 |
| fungi | random breakage in clouds is 3x as likely to tank a three-node job run as a single-node | 16:26 |
| AJaeger | sileht: argh, regex is wrong change "^(?!stable/(?:liberty|mitaka|newton))$" to "^(?!stable/(?:liberty|mitaka|newton)).*$" | 16:27 |
| *** burgerk has joined #openstack-infra | 16:27 | |
| clarkb | mmandel: does that work on mitaka and newton too? | 16:28 |
| sileht | AJaeger, thx ! I will fix that | 16:28 |
| bswartz | fungi: if single node job as 50% chance of failure, then 3 node job has 88% chance of failure -- not exactly 3x | 16:28 |
| clarkb | I guess the mitaka and newton jobs ran | 16:28 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: oslo.messaging: fix regex typo https://review.openstack.org/410846 | 16:28 |
| kfox1111 | fungi: +1. | 16:28 |
| *** hieulq__ has quit IRC | 16:28 | |
| kfox1111 | fungi: but I can do far more tests with far fewer resources with 3 nodes, then doing permutations on 2 nodes. | 16:29 |
| kfox1111 | so much less runtime means much less chance of random breakage affecting things. | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 16:29 |
| *** hieulq__ has joined #openstack-infra | 16:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/107597 | 16:29 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack-infra/project-config: oslo.messaging: fix regex typo https://review.openstack.org/410846 | 16:30 |
| fungi | kfox1111: yeah, i suppose that's a good point too. shorter runtime reduces some risk from random incident | 16:30 |
| sileht | AJaeger, it's ready: https://review.openstack.org/410846 (I wrote a better commitmsg) | 16:30 |
| *** nadya has quit IRC | 16:30 | |
| *** adrian_otto has joined #openstack-infra | 16:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 16:31 |
| fungi | bswartz: oh, right, 1-.5^3 obviously a 150% chance of failure makes no sense! ;) | 16:31 |
| clarkb | fungi: kfox1111 at least with other jobs our runtime stayed pretty consistent going from 1 to 2 nodes | 16:31 |
| clarkb | so not sure we have seen actual cases of reducing runtime simply by throwing nodes at the problem | 16:31 |
| clarkb | (a big part of that is the fixed overhead of doing setup) | 16:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 16:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Record nodeinfo as ansible facts and inventory https://review.openstack.org/410813 | 16:32 |
| AJaeger | fungi, clarkb : Could you +2A sileht'S 410846? Sorry, that regex slipped through review ;( | 16:33 |
| clarkb | AJaeger: looking | 16:33 |
| AJaeger | thanks, clarkb | 16:35 |
| *** nicolasbock has joined #openstack-infra | 16:35 | |
| *** andreas_s has quit IRC | 16:35 | |
| mmandel | clarkb: I don't think we need it there. it could go in newton, but doesn't need to | 16:37 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-diskimage_builder: Make vhd-util optional https://review.openstack.org/410849 | 16:38 |
| *** claudiub has joined #openstack-infra | 16:38 | |
| clarkb | mmandel: it is going into mitaka and newton in that change as is | 16:38 |
| clarkb | mmandel: which is why I asked, but the test logs for them seemed happy enough so I approved | 16:39 |
| *** vhosakot has joined #openstack-infra | 16:39 | |
| clarkb | mmandel: maybe keep an eye out for stable branch issues related to novnc and we can update if necessary | 16:39 |
| mmandel | clarkb: ok, thank you | 16:39 |
| clarkb | dhellmann: jhesketh fungi so is now the time where we can stop running all the liberty bitrot jobs? | 16:40 |
| clarkb | tempest, devstack-gate, periodic jobs, et al could all likely use the cleanup. Happy to work on that if we are ready for it | 16:40 |
| fungi | clarkb: yep, and can probably also clean up some lingering juno references as well | 16:40 |
| fungi | thought i spotted some recently | 16:41 |
| clarkb | juno, thats an older code not sure if it checks out | 16:41 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Use thread local data in ZK API https://review.openstack.org/410853 | 16:41 |
| *** deadnull has quit IRC | 16:41 | |
| clarkb | oh wow ya, /me pulls on yaml helmet and digs in | 16:42 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: List active jobs in postci https://review.openstack.org/410855 | 16:43 |
| clarkb | fungi: first one I dig into is networking-odl and they continue to have valid stable/icehouse and stable/juno https://git.openstack.org/cgit/openstack/networking-odl | 16:43 |
| clarkb | neither has been updated in 2 years | 16:43 |
| clarkb | armax: ^ who can we talk to about that? | 16:44 |
| *** inc0_ has joined #openstack-infra | 16:44 | |
| *** abregman|mtg is now known as abregman | 16:44 | |
| armax | yamahata_: ^ | 16:44 |
| pabelanger | restarting nodepool-builder on nb01 | 16:44 |
| armax | clarkb: he should be your go-to person | 16:44 |
| *** abregman has quit IRC | 16:44 | |
| *** inc0 has quit IRC | 16:44 | |
| clarkb | armax: thanks | 16:45 |
| *** chlong has quit IRC | 16:45 | |
| fungi | it might also be sane to come to some policy decision in the future that we can remove jobs for non-master branches which have seen no updates in at least a year | 16:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Use thread local data in ZK API https://review.openstack.org/410853 | 16:45 |
| *** jordanP has quit IRC | 16:46 | |
| *** rcernin has joined #openstack-infra | 16:46 | |
| clarkb | fungi: in this case its jobs for newer branches and tehy are excluding the old | 16:47 |
| fungi | aha | 16:47 |
| clarkb | but not having updates in 2 years makes me think it should just be tagged and the zuul config cleaned up | 16:47 |
| bswartz | why do change IDs always start with I? | 16:48 |
| *** tongli has quit IRC | 16:48 | |
| clarkb | bswartz: beacuse thats how gerrit defines them | 16:48 |
| *** jaosorior has quit IRC | 16:48 | |
| *** jaosorior has joined #openstack-infra | 16:48 | |
| bswartz | clarkb: so gerrit made the decision and we just inherit it? | 16:48 |
| clarkb | bswartz: print "Change-Id: I'"$id"'" from your commit message hook | 16:49 |
| fungi | well, it's a "Gerrit Change ID" | 16:49 |
| clarkb | bswartz: the I is literally hard coded there | 16:49 |
| fungi | so it makes sense that if we're using change ids with gerrit, we use the change id format gerrit expects | 16:50 |
| bswartz | k it just seemed like an arbitrary choice | 16:50 |
| fungi | it's not so much that we inherit them from anywhere, it's the format gerrit wants them in | 16:50 |
| mordred | I believe it's a choice to make it easy to distinguish them from git commit shas | 16:50 |
| *** knangia has joined #openstack-infra | 16:50 | |
| mordred | as git commit shas won't ever start with I | 16:50 |
| bswartz | indeed | 16:50 |
| pabelanger | and nb02.o.o builder restarted too | 16:50 |
| clarkb | mordred: they should've use G for Gerrit >_> | 16:50 |
| fungi | it's an arbitrary/nonstandard git commit header line gerrit has defined for use in its system | 16:50 |
| mordred | clarkb: well, it was reitveld before it was gerrit - so maybe it should have used R :) | 16:51 |
| *** EricGonczer_ has quit IRC | 16:51 | |
| fungi | mighth have made more sense to call the header Gerrit-Id: instead | 16:51 |
| mordred | if we're going to go down the road of choices gerrit could have made differently ... | 16:51 |
| * mordred dons hazmat suit | 16:51 | |
| bswartz | clarkb: G would cause problems if we switch from hexadecimal to septadecimal numbers | 16:52 |
| pabelanger | fungi: have time to comment on https://review.openstack.org/#/c/410788/. If we go that route, it will fix our centos-7 DIBs. Currently failing to build because of a package conflict in epel7 | 16:52 |
| fungi | bswartz: on the other hand, g could also get retconned to "google id" which i'm sure would make them plenty happy | 16:53 |
| mordred | ooh neat. I got a fun devstack fail: | 16:53 |
| mordred | 2016-12-14 15:58:26.866 | cp: cannot stat '/opt/stack/new/nova/etc/nova/policy.json': No such file or directory | 16:53 |
| openstackgerrit | Merged openstack-infra/system-config: governance.o.o: add redirect for badges https://review.openstack.org/408551 | 16:53 |
| mordred | sdague: ^^ know off the top of your head what I'm breaking? | 16:53 |
| sdague | well, that file isn't needed any more | 16:53 |
| dhellmann | clarkb : removing those jobs seems to make sense to me. What's our standard practice? | 16:53 |
| *** esikachev has quit IRC | 16:53 | |
| sdague | mordred: where is this fail? | 16:54 |
| *** EricGonczer_ has joined #openstack-infra | 16:54 | |
| mordred | sdague: neat - so, this is a job doing stable/liberty | 16:54 |
| mordred | sdague: gate-shade-dsvm-functional-keystone2 | 16:54 |
| clarkb | networking-cisco also has ancient juno branch | 16:54 |
| mordred | http://logs.openstack.org/25/410425/3/check/gate-shade-dsvm-functional-keystone2/a87fb0b/logs/devstacklog.txt.gz | 16:54 |
| fungi | pabelanger: would probably make sense to give some of the qa team a heads up on 410788 | 16:54 |
| sdague | mordred: hmmm... on stable/liberty that file should still be there | 16:54 |
| clarkb | dhellmann: I think we just go through the jjb and zuul config and remove the now redundant rules and jobs that won't run | 16:54 |
| mordred | sdague: I'm guessing I should maybe start working on reworking that job to not be stable/liberty though | 16:54 |
| *** tphummel has joined #openstack-infra | 16:54 | |
| clarkb | (or shouldn't run) | 16:54 |
| sdague | mordred: though is stable/liberty still a thing for devstack? | 16:54 |
| clarkb | sdague: mordred no liberty is gone now | 16:55 |
| clarkb | as of last night? | 16:55 |
| mordred | sdague: not sure - the reason that job is is so that I can have a devstack that has only keystone v2 and no keystone v3 | 16:55 |
| mordred | gotcha | 16:55 |
| * mordred goes to figure out new approach to that problem | 16:55 | |
| *** yamamoto has quit IRC | 16:55 | |
| *** shamail has quit IRC | 16:55 | |
| pabelanger | fungi: sure, I can do that now | 16:56 |
| *** bhunter71 has joined #openstack-infra | 16:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Juno Spring Cleaning https://review.openstack.org/410861 | 16:56 |
| clarkb | dhellmann: fungi ^ thats a step 0 to clean up some of the older branch stuff. Next kilo | 16:56 |
| dhellmann | clarkb : I meant in terms of timing. We still have some projects with liberty branches, don't we? | 16:56 |
| clarkb | dhellmann: we still have projects with icehouse branches... | 16:57 |
| *** esikachev has joined #openstack-infra | 16:57 | |
| dhellmann | whee | 16:57 |
| clarkb | dhellmann: I think for anything we call "openstack" we say too bad have a nice day | 16:57 |
| clarkb | dhellmann: for the others fungi and I were discussing informally what we might want to do | 16:57 |
| *** yamamoto has joined #openstack-infra | 16:57 | |
| mordred | stevemar: any chance you know if I can tell devstack to not install the keystone v3 routes? | 16:57 |
| *** mmandel has quit IRC | 16:58 | |
| *** chlong has joined #openstack-infra | 16:58 | |
| *** mmandel has joined #openstack-infra | 16:58 | |
| dhellmann | clarkb : I support cleaning up. It's probably a good idea to advertise it a bit to make sure projects aren't surprised. But yeah, I don't think we need to keep those old jobs. | 16:59 |
| stevemar | mordred: i think theres a flag for that | 16:59 |
| *** piet has joined #openstack-infra | 16:59 | |
| stevemar | mordred: line 57 here: https://review.openstack.org/#/c/410295/2/jenkins/jobs/keystone.yaml ? but that might still install v3 routes | 17:00 |
| pabelanger | #status log nb01.o.o and nb02.o.o builders restarted and running from master again. nodepool.o.o did not restart, but /opt/nodepool is pointing to master branch | 17:00 |
| openstackstatus | pabelanger: finished logging | 17:00 |
| *** bh526r has joined #openstack-infra | 17:00 | |
| *** rajinir has joined #openstack-infra | 17:00 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo/scenarios: exclude some stable branches https://review.openstack.org/410863 | 17:00 |
| mordred | stevemar: quick glance through current devstack leads me to believe that I may not havea choice about installing v3 routes- so I might just have to have both and force v2 codepaths through client config | 17:00 |
| mordred | stevemar: the glance code seems to hardcode the v3 endpoint | 17:00 |
| *** inc0_ is now known as inc0 | 17:00 | |
| *** maeker has joined #openstack-infra | 17:01 | |
| stevemar | mordred: wut | 17:01 |
| *** esikachev has quit IRC | 17:01 | |
| clarkb | dhellmann: one thing that helps is we know that devstack won't work with it anymore so anything trying to pair devstack with eol'd branches can just be removed | 17:01 |
| clarkb | dhellmann: mordred's thing is an example of this :) so we know it works that way | 17:02 |
| *** esikachev has joined #openstack-infra | 17:02 | |
| mordred | stevemar: lib/glance: iniset $GLANCE_CACHE_CONF DEFAULT auth_url $KEYSTONE_AUTH_URI/v3 | 17:02 |
| *** yamamoto has quit IRC | 17:02 | |
| clarkb | harlowja: anvil is your thing? you have a couple of jjb TODOs that say chagne this later for origin files with kilo in their names | 17:02 |
| clarkb | s/you have/anvil has/ | 17:02 |
| *** mmandel has quit IRC | 17:03 | |
| openstackgerrit | Merged openstack-infra/project-config: oslo.messaging: fix regex typo https://review.openstack.org/410846 | 17:04 |
| *** rbuzatu has quit IRC | 17:05 | |
| *** piet has quit IRC | 17:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Enable keystone-v2 for shade keystone-v2 job https://review.openstack.org/410864 | 17:05 |
| *** sdake has quit IRC | 17:05 | |
| *** rbuzatu has joined #openstack-infra | 17:05 | |
| *** maeker has quit IRC | 17:05 | |
| *** mmalik4 has joined #openstack-infra | 17:05 | |
| mordred | clarkb, stevemar: I think that ^^ should fix the shade gate from the liberty thing | 17:05 |
| *** hieulq__ has quit IRC | 17:05 | |
| *** maeker has joined #openstack-infra | 17:06 | |
| *** Kevin_Zheng has quit IRC | 17:07 | |
| stevemar | mordred: as long as it only affects the keystone2 job, lgtm | 17:08 |
| mordred | stevemar: yah - should only be for that job - we definitely want to test v3 in the other places | 17:08 |
| mordred | you know ... we _could_ combine that job and the nova-net job | 17:09 |
| mordred | clarkb: ^^ thoughts on whether combining keystone-v2 and nova-net devstack configs into a single job is good or bad? | 17:09 |
| stevemar | mordred: make a 'old stuff' job | 17:09 |
| fungi | "cobwebs" | 17:09 |
| *** hashar has quit IRC | 17:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Kilo spring cleaning https://review.openstack.org/410868 | 17:10 |
| clarkb | there is the kilo cleanup | 17:10 |
| clarkb | I think I need breakfast before I start worrying about liberty | 17:10 |
| fungi | i see you're using southern hemisphere spring | 17:10 |
| fungi | how global of you | 17:10 |
| clarkb | ~/src/openstack-infra/project-config (stable-cleanup)$ git grep liberty | wc -l says 286 | 17:11 |
| *** e0ne has quit IRC | 17:11 | |
| fungi | eep | 17:11 |
| *** shardy is now known as shardy_afk | 17:13 | |
| *** filler has joined #openstack-infra | 17:13 | |
| *** ihrachys has quit IRC | 17:14 | |
| *** inc0 has quit IRC | 17:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Collapse keystone2 and nova-net shade jobs https://review.openstack.org/410872 | 17:15 |
| mordred | clarkb, fungi, pabelanger: ^^ there is a follow up patch that collapses the two jobs into one | 17:15 |
| *** filler_ has joined #openstack-infra | 17:16 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:17 |
| *** rbuzatu has quit IRC | 17:17 | |
| *** tphummel has quit IRC | 17:17 | |
| *** jpich has quit IRC | 17:18 | |
| pabelanger | mordred: comment left, gate needs update | 17:19 |
| mordred | pabelanger: thanks! | 17:19 |
| *** filler has quit IRC | 17:19 | |
| *** filler_ is now known as filler | 17:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Collapse keystone2 and nova-net shade jobs https://review.openstack.org/410872 | 17:19 |
| *** mmandel has joined #openstack-infra | 17:20 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:21 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Default libvirt_cpu_mode to none when using qemu https://review.openstack.org/410359 | 17:21 |
| *** tqtran has joined #openstack-infra | 17:21 | |
| *** esikachev has quit IRC | 17:22 | |
| yamahata_ | clarkb: we can remove icehouse and juno support for networking-odl | 17:23 |
| *** jkilpatr_ has quit IRC | 17:24 | |
| *** tphummel has joined #openstack-infra | 17:24 | |
| *** hongbin has quit IRC | 17:25 | |
| *** sdake has joined #openstack-infra | 17:26 | |
| fungi | thanks yamahata_! | 17:26 |
| *** nadya has joined #openstack-infra | 17:27 | |
| *** rpodolyaka is now known as rpodolyaka_afk | 17:27 | |
| *** lucasagomes is now known as lucas-afk | 17:30 | |
| *** sdake has quit IRC | 17:30 | |
| *** nadya has quit IRC | 17:31 | |
| *** rossella_s has quit IRC | 17:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: WIP: Remove ZK lock variables https://review.openstack.org/410853 | 17:32 |
| *** rossella_s has joined #openstack-infra | 17:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Be explicit about not using ipv6 on rackspace https://review.openstack.org/410879 | 17:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Collapse keystone2 and nova-net shade jobs https://review.openstack.org/410872 | 17:35 |
| *** baoli has joined #openstack-infra | 17:36 | |
| *** rbuzatu has joined #openstack-infra | 17:36 | |
| *** mkoderer has quit IRC | 17:36 | |
| *** jkilpatr_ has joined #openstack-infra | 17:37 | |
| *** panda is now known as panda|bbl | 17:38 | |
| *** shashank_hegde has joined #openstack-infra | 17:38 | |
| *** yamahata_ has quit IRC | 17:42 | |
| *** EricGonczer_ has quit IRC | 17:44 | |
| *** piet has joined #openstack-infra | 17:45 | |
| JayF | I have found a weird thing. A job that passed, but is returning failure: http://logs.openstack.org/53/410253/1/gate/gate-python-ironicclient-pep8-ubuntu-xenial/8b5db84/ | 17:47 |
| rosmaita | JayF: that's literally ironic! :) | 17:47 |
| JayF | Last lines say "task exit code: 0" "Job complete result: FAILURE" | 17:47 |
| *** asettle has quit IRC | 17:48 | |
| JayF | and afict no failures in the job itself, console output for the pep8 looks clean | 17:48 |
| JayF | rosmaita: HA | 17:48 |
| clarkb | JayF: check the ansible log to see if any of the publishing tasks failed | 17:48 |
| *** thorst_ has quit IRC | 17:48 | |
| JayF | it looks almost like it died in the middle of ansible running http://logs.openstack.org/53/410253/1/gate/gate-python-ironicclient-pep8-ubuntu-xenial/8b5db84/_zuul_ansible/ansible_log.txt | 17:49 |
| *** thorst_ has joined #openstack-infra | 17:49 | |
| JayF | unless I'm misreading the logs, Idon't use a lot of ansible | 17:49 |
| mordred | rosmaita: :) | 17:50 |
| jeblair | JayF: well, that log gets truncated at 'copy console log' because that's where it gets copied itself | 17:50 |
| JayF | jeblair: oh, that makes sense. AFAIK no failures in that log then | 17:50 |
| jeblair | JayF: i'll look on the launcher to see if there are more details | 17:50 |
| *** ociuhandu has quit IRC | 17:50 | |
| *** nadya has joined #openstack-infra | 17:50 | |
| JayF | Yeah; I mean, I can always recheck | 17:51 |
| JayF | but I try to dig into this stuff and not just blindly recheck things, especially on more interesting failures :) | 17:51 |
| jeblair | but it's curious, isn't it :) | 17:51 |
| jeblair | JayF: i'm pretty sure that the failure showing up in the *console log* means it happened before then though | 17:52 |
| JayF | oh, that's a good point | 17:52 |
| JayF | the actual pep8 test passed cleanly, and I couldn't find anything angry in the console log | 17:52 |
| jeblair | 2016-12-14 17:05:57,317 WARNING zuul.NodeWorker.ubuntu-xenial-infracloud-chocolate-6172649: Ansible timeout exceeded | 17:53 |
| *** shashank_hegde has left #openstack-infra | 17:53 | |
| jeblair | JayF: there it is | 17:53 |
| *** jaosorior has quit IRC | 17:54 | |
| *** tosky has joined #openstack-infra | 17:54 | |
| jeblair | i guess it's not especially discoverable when that happens :( | 17:54 |
| JayF | jeblair: that's not a log I have access to, right? | 17:54 |
| jeblair | JayF: nope, that's from the launcher | 17:54 |
| mordred | jeblair: maybe we should inject an error log statement into the post playbook when that happens? | 17:54 |
| jeblair | mordred: yeah, set an extra flag like success for timeout? | 17:55 |
| mordred | yah | 17:55 |
| mordred | jeblair: I can make that patch if you have other things you're doing | 17:55 |
| jeblair | mordred: ++ thx | 17:55 |
| JayF | Does that mean you're going to make it succeed in this case instead? OR just print a log I'd have access to read? | 17:56 |
| jeblair | JayF: print a log | 17:56 |
| JayF | cool | 17:56 |
| jeblair | mordred: it looks like ansible finished, but did not exit -- the recap showed up | 17:56 |
| *** dtantsur is now known as dtantsur|afk | 17:56 | |
| *** yamamoto has joined #openstack-infra | 17:56 | |
| mordred | jeblair: oh lovely | 17:57 |
| jeblair | mordred: http://paste.openstack.org/show/592387/ | 17:57 |
| *** ihrachys has joined #openstack-infra | 17:57 | |
| *** mhickey has quit IRC | 17:57 | |
| *** Jeffrey4l has quit IRC | 17:58 | |
| *** thorst_ has quit IRC | 17:58 | |
| jeblair | it looks like we read stdout as long as it's open, then waitpid. so either stdout never closed, or wait never returned. weird. | 17:58 |
| *** Sukhdev has quit IRC | 17:59 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: test https://review.openstack.org/410470 | 17:59 |
| *** mvandijk has left #openstack-infra | 17:59 | |
| *** markus_z has quit IRC | 17:59 | |
| *** thorst_ has joined #openstack-infra | 18:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Be explicit about not using ipv6 on rackspace https://review.openstack.org/410879 | 18:00 |
| jeblair | JayF: thanks for pointing this out... we'll at least get a log line in there... and we know there's a weird ansible thing we should keep an eye out for | 18:01 |
| JayF | No problem, when I see the gate fail I always try to dig in | 18:01 |
| *** yamamoto has quit IRC | 18:01 | |
| JayF | That's the first thing I learned working in OpenStack as part of being a good citizen, and now that I actually know howtf to do it, I try to, lol | 18:01 |
| *** florianf has quit IRC | 18:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add a log message when ansible times out https://review.openstack.org/410888 | 18:03 |
| *** flepied has quit IRC | 18:03 | |
| mordred | jeblair: what about something like that ^^ (also, jlk if you're around, can you double-check my ansible when syntax?) | 18:03 |
| *** asettle has joined #openstack-infra | 18:04 | |
| *** ralonsoh_ has quit IRC | 18:04 | |
| *** thorst_ has quit IRC | 18:04 | |
| jlk | looking | 18:05 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for stable/liberty reno failures bug 1649982 https://review.openstack.org/410894 | 18:06 |
| openstack | bug 1649982 in reno "reno doesn't check for an -eol tag when a stable branch is deleted" [Wishlist,Fix committed] https://launchpad.net/bugs/1649982 - Assigned to Doug Hellmann (doug-hellmann) | 18:06 |
| *** thorst has joined #openstack-infra | 18:06 | |
| JayF | mriedem: would that be http://logs.openstack.org/92/398992/9/check/gate-ironic-python-agent-releasenotes/1842d6d/console.html#_2016-12-14_16_48_57_861974 this failure, perhaps/ | 18:06 |
| *** Rockyg has quit IRC | 18:07 | |
| mriedem | JayF: yes | 18:07 |
| JayF | mriedem: nope, but it makes me suspicious as to if my failure is related to eol | 18:07 |
| mriedem | it is | 18:07 |
| JayF | mriedem: aha, I'm not sure that e-r would catch this patch | 18:07 |
| mriedem | it will | 18:07 |
| mriedem | see the logstash link in the bug report | 18:07 |
| JayF | oh yeah, I see it now | 18:07 |
| mriedem | it hits a bunch of projects | 18:08 |
| JayF | Awesome, thanks for answering my question before I even asked it | 18:08 |
| JayF | I was about to head over here to ask about that failure | 18:08 |
| mriedem | doug is releasing the fix right now | 18:08 |
| *** rkukura has quit IRC | 18:08 | |
| mriedem | you may want to make the same workaround fix that nova did | 18:08 |
| *** derekh has quit IRC | 18:09 | |
| openstackgerrit | DavidPurcell proposed openstack-infra/project-config: Add patrole projects. https://review.openstack.org/409317 | 18:09 |
| harlowja | clarkb ya anvil sorta my thing, though i'm pretty sure i'm just gonna EOL it | 18:09 |
| harlowja | since yahoo isn't using it anymore, and i'm getting godaddy off of it (to kolla) | 18:10 |
| JayF | mriedem: not seeing any commits that look relevant in the nova repo, do you have a link? | 18:10 |
| harlowja | so i'll probably EOL it clarkb instead of doing much work on it | 18:10 |
| *** ihrachys has quit IRC | 18:10 | |
| harlowja | cray was using it to, but haven't heard from them in forever | 18:10 |
| clarkb | harlowja: ok can you clean that up please then? | 18:11 |
| harlowja | yup | 18:11 |
| *** thorst has quit IRC | 18:11 | |
| *** ihrachys has joined #openstack-infra | 18:11 | |
| clarkb | tyty | 18:12 |
| *** nadya has quit IRC | 18:12 | |
| *** thorst has joined #openstack-infra | 18:12 | |
| *** tobybot11 has joined #openstack-infra | 18:13 | |
| *** savihou has joined #openstack-infra | 18:14 | |
| *** stevelle has joined #openstack-infra | 18:14 | |
| *** e0ne has joined #openstack-infra | 18:15 | |
| mriedem | JayF: https://review.openstack.org/#/c/410851/ | 18:16 |
| *** thorst has quit IRC | 18:17 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for stable/liberty reno failures bug 1649982 https://review.openstack.org/410894 | 18:17 |
| openstack | bug 1649982 in reno "reno doesn't check for an -eol tag when a stable branch is deleted" [Wishlist,Fix committed] https://launchpad.net/bugs/1649982 - Assigned to Doug Hellmann (doug-hellmann) | 18:17 |
| *** asettle has quit IRC | 18:17 | |
| jeblair | mordred, pabelanger: did we restart all the launchers after 408194 or just zlstatic01? | 18:18 |
| *** riccardo_ has joined #openstack-infra | 18:18 | |
| mordred | jeblair: I do not know the answer to that ... pabelanger ^^ ? | 18:19 |
| mordred | jeblair: I _think_ just zlstatic01 | 18:19 |
| *** jkilpatr_ has quit IRC | 18:20 | |
| riccardo_ | hi community, where can I find the openstack-infra/project-config repository to add a project to the CI? http://docs.openstack.org/infra/manual/creators.html#adding-the-project-to-the-ci-system | 18:20 |
| JayF | mriedem: perfect; thanks! | 18:20 |
| jeblair | mordred: yeah, zl01 says nov 23 | 18:20 |
| jeblair | good | 18:20 |
| jeblair | let's not restart them | 18:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix watchdog timeout fix https://review.openstack.org/410902 | 18:21 |
| jeblair | mordred: ^ noticed that when reviewing your change | 18:21 |
| mordred | riccardo_: http://git.openstack.org/cgit/openstack-infra/project-config | 18:21 |
| mordred | jeblair: oh good | 18:21 |
| riccardo_ | mordred: thank you very much! :-) | 18:21 |
| *** david-lyle has quit IRC | 18:21 | |
| mordred | jeblair: purty | 18:22 |
| *** yamahata_ has joined #openstack-infra | 18:22 | |
| *** jkilpatr has joined #openstack-infra | 18:22 | |
| jeblair | mordred: i kind of want a watch-hog timer. | 18:23 |
| pabelanger | jeblair: I am unsure. I believe fungi did the restart of zlstatic01 | 18:23 |
| *** thorst has joined #openstack-infra | 18:23 | |
| *** riccardo_ has quit IRC | 18:23 | |
| jeblair | pabelanger, fungi: we should land https://review.openstack.org/410902 and https://review.openstack.org/410888 | 18:25 |
| mordred | jeblair: while I've got your attention ... https://review.openstack.org/#/c/410872 and https://review.openstack.org/#/c/410864 should fix a broken gate if you have a sec | 18:26 |
| *** yamamoto has joined #openstack-infra | 18:26 | |
| *** yamamoto has quit IRC | 18:27 | |
| *** yamamoto has joined #openstack-infra | 18:27 | |
| *** yamamoto has quit IRC | 18:27 | |
| pabelanger | jeblair: +2, so fungi can review | 18:27 |
| jeblair | mordred: i might -1 for s/legacy/cobwebs/ as stevemar suggests :) | 18:28 |
| mordred | jeblair: :) | 18:28 |
| *** markvoelker has quit IRC | 18:29 | |
| *** arif-ali has quit IRC | 18:30 | |
| *** thorst has quit IRC | 18:30 | |
| *** david-lyle has joined #openstack-infra | 18:30 | |
| *** EricGonczer_ has joined #openstack-infra | 18:30 | |
| *** esikachev has joined #openstack-infra | 18:30 | |
| *** electrofelix has quit IRC | 18:30 | |
| *** jpena is now known as jpena|off | 18:30 | |
| *** arif-ali has joined #openstack-infra | 18:31 | |
| *** piet has quit IRC | 18:31 | |
| *** EricGonczer_ has quit IRC | 18:32 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: ironic.yaml: Remove stable/liberty specific checks https://review.openstack.org/410910 | 18:34 |
| *** rbrndt has quit IRC | 18:34 | |
| *** markvoelker has joined #openstack-infra | 18:35 | |
| openstackgerrit | DavidPurcell proposed openstack-infra/project-config: Add patrole projects. https://review.openstack.org/409317 | 18:35 |
| *** rkukura has joined #openstack-infra | 18:36 | |
| *** tphummel has quit IRC | 18:36 | |
| *** thorst has joined #openstack-infra | 18:36 | |
| *** thorst has quit IRC | 18:36 | |
| *** thorst has joined #openstack-infra | 18:37 | |
| *** sdake has joined #openstack-infra | 18:38 | |
| *** gomarivera has joined #openstack-infra | 18:38 | |
| *** sdake_ has joined #openstack-infra | 18:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Wrap zuul_log in pre HTML tag https://review.openstack.org/410911 | 18:41 |
| *** sdake has quit IRC | 18:43 | |
| *** gomarivera has quit IRC | 18:43 | |
| *** e0ne has quit IRC | 18:44 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Drop redirects and pipes from postci call https://review.openstack.org/410915 | 18:45 |
| *** ijw has joined #openstack-infra | 18:45 | |
| *** zara_the_lemur__ has joined #openstack-infra | 18:46 | |
| *** e0ne has joined #openstack-infra | 18:47 | |
| pabelanger | and that is enough glean for me today | 18:47 |
| openstackgerrit | Merged openstack-infra/project-config: Enable keystone-v2 for shade keystone-v2 job https://review.openstack.org/410864 | 18:50 |
| *** ijw has quit IRC | 18:50 | |
| *** tsg has quit IRC | 18:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Collapse keystone2 and nova-net shade jobs https://review.openstack.org/410872 | 18:51 |
| *** sdake_ has quit IRC | 18:51 | |
| pabelanger | fungi: I left a note in openstack-qa about 410788, no objects currently | 18:52 |
| *** nadya has joined #openstack-infra | 18:55 | |
| *** wolverineav has quit IRC | 18:55 | |
| *** ijw has joined #openstack-infra | 18:56 | |
| *** ijw has quit IRC | 18:57 | |
| JayF | clarkb: https://review.openstack.org/#/c/398992/ I *think* this is the feature in IPA you need for your TLS-everything project. Once it lands it we'd just need to add ipa-insecure=1 to IRONIC_EXTRA_PXE_PARAMS in devstack. | 18:57 |
| *** ijw has joined #openstack-infra | 18:58 | |
| clarkb | JayF: cool, once thats done you can depends on the devstack change that just hard enables tls-proxy and if that passes you can add tls-proxy to your jobs. Happy to help with that | 18:59 |
| clarkb | JayF: also need to get it working on CentOS, I think EmilienM was going to look into that | 18:59 |
| clarkb | but once thats done I think we are down to a few functional jobs that need to be taught to use https and we are ready to just turn it on in ocata as default | 19:00 |
| rcarrillocruz | o/ | 19:00 |
| *** ihrachys has quit IRC | 19:00 | |
| JayF | Yeah, and we can probably add/remove that ipa-insecure bit based on the tls-proxy enabling setting (wherever it is) | 19:00 |
| *** knangia has quit IRC | 19:00 | |
| JayF | I'll look into it once that lands. | 19:00 |
| *** tosky has quit IRC | 19:00 | |
| openstackgerrit | Merged openstack-infra/storyboard: Specify python2 for tox venv https://review.openstack.org/404979 | 19:01 |
| *** wolverineav has joined #openstack-infra | 19:02 | |
| *** thorst has quit IRC | 19:03 | |
| *** knangia has joined #openstack-infra | 19:04 | |
| *** ijw has quit IRC | 19:04 | |
| *** hongbin has joined #openstack-infra | 19:05 | |
| *** ihrachys has joined #openstack-infra | 19:06 | |
| *** wolverin_ has joined #openstack-infra | 19:06 | |
| *** wolverineav has quit IRC | 19:06 | |
| *** tphummel has joined #openstack-infra | 19:07 | |
| *** ihrachys has quit IRC | 19:09 | |
| *** jgriffith is now known as jgriffith_AutoAw | 19:10 | |
| *** amoralej is now known as amoralej|off | 19:11 | |
| *** marst has quit IRC | 19:16 | |
| fungi | sorry, was afk to run a couple of lunch errands. catching back up now | 19:16 |
| *** marst has joined #openstack-infra | 19:16 | |
| *** thorst has joined #openstack-infra | 19:16 | |
| *** panda|bbl is now known as panda | 19:17 | |
| *** thorst has quit IRC | 19:17 | |
| fungi | JayF: jeblair: mordred: hah, the first thing i looked at in that job's console log was the last two timestamps. 16:35 task exit code 0, 17:44 job complete result failure. gap like that almost always means there was a timeout somewhere | 19:19 |
| pabelanger | been a while since I've seen a backlog in the gate pipeline | 19:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Kilo spring cleaning https://review.openstack.org/410868 | 19:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Juno Spring Cleaning https://review.openstack.org/410861 | 19:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Liberty spring cleaning https://review.openstack.org/410928 | 19:19 |
| *** coolsvap has quit IRC | 19:19 | |
| clarkb | dhellmann: fungi ok ^ is a first pass at cleaning up what we can clean up safely I think | 19:20 |
| fungi | jeblair: i only restarted zlstatic01 after 408194 merged because i got the impression that was irrelevant to single-use nodes anyway | 19:20 |
| *** Sukhdev has joined #openstack-infra | 19:20 | |
| *** rbrndt has joined #openstack-infra | 19:23 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Change 'stories' icon to a book https://review.openstack.org/400381 | 19:23 |
| *** thorst has joined #openstack-infra | 19:26 | |
| *** thorst has quit IRC | 19:26 | |
| *** yamamoto has joined #openstack-infra | 19:28 | |
| *** thorst has joined #openstack-infra | 19:28 | |
| *** wolverin_ has quit IRC | 19:29 | |
| openstackgerrit | Merged openstack-infra/zuul: Add a log message when ansible times out https://review.openstack.org/410888 | 19:30 |
| *** e0ne has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack-infra/zuul: Fix watchdog timeout fix https://review.openstack.org/410902 | 19:31 |
| *** thorst has quit IRC | 19:31 | |
| *** tobybot11 has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Stop caching devstack packages https://review.openstack.org/410788 | 19:32 |
| *** flepied has joined #openstack-infra | 19:33 | |
| *** Swami has quit IRC | 19:33 | |
| *** Swami has joined #openstack-infra | 19:34 | |
| *** EricGonczer_ has joined #openstack-infra | 19:34 | |
| openstackgerrit | Eric Harney proposed openstack-infra/bindep: get_pkg_version: Fix py3 bytes/str errors https://review.openstack.org/410933 | 19:34 |
| *** jkilpatr has quit IRC | 19:34 | |
| *** maeker has quit IRC | 19:35 | |
| *** yamamoto has quit IRC | 19:36 | |
| *** EricGonczer_ has quit IRC | 19:38 | |
| *** EricGonczer_ has joined #openstack-infra | 19:38 | |
| *** thorst has joined #openstack-infra | 19:38 | |
| *** sdake has joined #openstack-infra | 19:39 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Remove now-unused ZuulTestCase.resetGearmanServer https://review.openstack.org/410935 | 19:41 |
| clarkb | before we forget, should we plan a coordinated switch off of login.launchpad.net in favor of login.ubuntu.com for review.o.o? | 19:41 |
| clarkb | I should be around all week and next week for that | 19:41 |
| fungi | yeah, seems like a fine idea to me | 19:43 |
| *** sdague has quit IRC | 19:43 | |
| clarkb | tomorrow afternnon is the least good part of me (wife's birthday tomorrow) but otherwise open to whenever | 19:45 |
| fungi | oh, let's not sign you up for any scheduled work on her birthday | 19:46 |
| * fungi doesn't want to get on her bad side | 19:46 | |
| *** sdague has joined #openstack-infra | 19:46 | |
| *** pvaneck has joined #openstack-infra | 19:47 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Add tripleo-repos project https://review.openstack.org/410937 | 19:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder https://review.openstack.org/410939 | 19:49 |
| *** rbrndt has quit IRC | 19:53 | |
| *** jkilpatr has joined #openstack-infra | 19:53 | |
| *** EricGonczer_ has quit IRC | 19:53 | |
| *** sdake has quit IRC | 19:55 | |
| EmilienM | clarkb: /me back from lunch - so I talked with ianw and it's still unclear to us what is not working correctly, have you seen his comment on https://review.openstack.org/#/c/402940/ ? | 19:56 |
| *** bh526r has quit IRC | 19:56 | |
| *** rbrndt has joined #openstack-infra | 19:56 | |
| clarkb | EmilienM: I haven't, ianw's comment? | 19:56 |
| EmilienM | clarkb: yes | 19:57 |
| clarkb | which one? the one I am reading doesn't seem to address ssl | 19:58 |
| *** esikachev has quit IRC | 19:59 | |
| clarkb | I do think the earlier issue with epel does illustrate an issue that is completely ignored here | 19:59 |
| *** sdake has joined #openstack-infra | 19:59 | |
| clarkb | and that is its ok for upstreams to completely break the distro because we are just using teh distro wrong | 19:59 |
| *** Swami_ has joined #openstack-infra | 20:00 | |
| *** Swami has quit IRC | 20:00 | |
| clarkb | while at the same time not correcting that usage | 20:00 |
| clarkb | which maybe is what ian meant by people are not doign the proejct level work | 20:01 |
| clarkb | people that are familar with debuntu do do that work for debuntu | 20:01 |
| clarkb | we try to cache ceph because devstack tells us to. we cache it from epel because that is where the package lives. If that is wrong people that care about centos should fix it | 20:02 |
| fungi | EmilienM: the other thing that probably could still stand addressing is what mtreinish mentioned in last week's tc meeting... the reason the devstack centos7 job is passing somewhat regularly is that a lot of features and tests have been disabled to make that possible. ideally we should have some degree of parity there | 20:02 |
| *** jgriffith_AutoAw is now known as jgriffith | 20:02 | |
| EmilienM | fungi: do we have tracked the features/tests that are disabled, reported bug upstream somewhere? | 20:04 |
| *** EricGonczer_ has joined #openstack-infra | 20:04 | |
| *** spzala has quit IRC | 20:04 | |
| EmilienM | clarkb: on another topic: https://review.openstack.org/#/c/410483/ - I'll need your help when you have time to deploy puppet4 somewhere (using puppetlabs repo eventually) | 20:05 |
| clarkb | EmilienM: again this is the issue. Its impossible for the small handful of people here to keep up with it all. The individuals that are interested in making this work need to have more ownership/responsibility | 20:05 |
| *** EricGonczer_ has quit IRC | 20:05 | |
| clarkb | We can and do do a ton of work to make centos/fedora viable and do host some things on centos as well | 20:05 |
| clarkb | but when it comes to "why does apache not work like it does everywhere else" I am going to go do other tings | 20:05 |
| fungi | EmilienM: i don't know as far as what's been reported. odds are those features/tests were disabled by whoever was doing the work to get that job running and i have no idea who that is | 20:05 |
| *** EricGonczer_ has joined #openstack-infra | 20:06 | |
| EmilienM | clarkb: I'll check with ianw when he's around. I thought Ian was working on this topic but I might be wrong | 20:06 |
| clarkb | (and I did send amil asking for help on that particular case) | 20:06 |
| *** srobert has quit IRC | 20:07 | |
| EmilienM | I agree it's a lot of work and we're running with resources constraints... this one is not easy to solve | 20:08 |
| ianw | EmilienM: working on what? :) | 20:09 |
| EmilienM | ianw: hey good morning | 20:10 |
| EmilienM | ianw: making devstack jobs working on centos7 | 20:10 |
| EmilienM | ianw: it seems that 1) there are some issues when ssl is enabled in httpd 2) some projects have to ignore some features / tests | 20:10 |
| fungi | #2 shouldn't be surprising, but for the purposes of eventually making claims that we're testing/supporting it as well as ubuntu we likely would benefit from some sort of gap analysis there | 20:12 |
| fungi | some of the comments left on the proposed governance change seemed to make the assumption that having devstack "running" successfully on centos was a binary proposition, when it's really not at all | 20:14 |
| EmilienM | no, my idea was not to force projects to have centos7 projects | 20:14 |
| ianw | yeah, i'm not sure i said "everything works" though maybe it was taken that way. what i meant was "everything i test works, but there's not barriers to contribution for fixing it", which i still think is true | 20:15 |
| EmilienM | the idea was just about recognition that ubuntu and centos7 are the favorite plaftorms to test OpenStack | 20:15 |
| ianw | i've never really tried SSL, but if we need it, i can look into it | 20:15 |
| *** dfflanders has joined #openstack-infra | 20:15 | |
| clarkb | ianw: yes I would agree with that. The barriers are really low. Just need people that are willing and able to jump in | 20:15 |
| ianw | just nobody asked yet :) | 20:15 |
| *** ociuhandu has joined #openstack-infra | 20:15 | |
| clarkb | ianw: I asked on the mailing list , maybe I should've poked more direclty | 20:15 |
| *** adrian_otto has quit IRC | 20:16 | |
| *** Goneri has quit IRC | 20:16 | |
| clarkb | ianw: tl;dr is that devstack + tls-proxy does not work on centos 7 because apache is acting different htan ubuntu trusty, xenial, and fedora | 20:16 |
| openstackgerrit | Aleksey Zvyagintsev proposed openstack-infra/project-config: RefStack: add coverage job https://review.openstack.org/410283 | 20:16 |
| clarkb | ianw: seems to be related to stop/start/restart of apache service | 20:16 |
| clarkb | vs reload | 20:16 |
| *** ldnunes has quit IRC | 20:16 | |
| ianw | clarkb: ok, yeah i think actually we merged something to do with apache restart/reload not that long ago. it's probably sensible for me to turn on ssl for the devstack platform jobs ... i'll take that as an action item | 20:17 |
| *** asettle has joined #openstack-infra | 20:18 | |
| ianw | EmilienM: i'll do some investigation and update -> https://bugs.launchpad.net/devstack/+bug/1650006 | 20:20 |
| openstack | Launchpad bug 1650006 in devstack "Get SSL working on Centos7" [Undecided,New] | 20:20 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Kill dstat to prevent hanging of job https://review.openstack.org/410738 | 20:20 |
| EmilienM | ianw: thanks a lot :) | 20:21 |
| clarkb | ianw: mtreinish and I are hoping to maybe turn on tls-proxy by default in devstack this cycle. Ironic almost has their necessary bits to support it and a few functional test setups need to be more intelligent about using https too, but other than that I think centos7 support is the big hurdle | 20:21 |
| clarkb | hopefully we get there | 20:22 |
| *** asettle has quit IRC | 20:22 | |
| *** rhallisey has quit IRC | 20:24 | |
| *** Swami_ has quit IRC | 20:30 | |
| kfox1111 | are there any known gate issues right now? | 20:30 |
| clarkb | I think just the docs one with the badges | 20:31 |
| kfox1111 | hmm. k. | 20:31 |
| kfox1111 | thanks. | 20:32 |
| *** dangers_away has quit IRC | 20:32 | |
| *** rbrndt has quit IRC | 20:33 | |
| *** gouthamr has joined #openstack-infra | 20:33 | |
| *** pcaruana has quit IRC | 20:33 | |
| *** rm_work has quit IRC | 20:34 | |
| *** rm_work has joined #openstack-infra | 20:34 | |
| *** zara_the_lemur__ has left #openstack-infra | 20:34 | |
| *** ociuhandu has quit IRC | 20:35 | |
| *** sdake_ has joined #openstack-infra | 20:35 | |
| *** dangers has joined #openstack-infra | 20:35 | |
| *** rbrndt has joined #openstack-infra | 20:35 | |
| *** esikachev has joined #openstack-infra | 20:36 | |
| *** ociuhandu has joined #openstack-infra | 20:37 | |
| *** sdake has quit IRC | 20:37 | |
| *** StevenK has quit IRC | 20:38 | |
| fungi | anybody have any concerns about this three-character change to devstack-gate to solve for an unusual git behavior karbor managed to hang themselves with earlier? https://review.openstack.org/410810 | 20:38 |
| clarkb | fungi: thanks for reminding me to look at that | 20:38 |
| *** StevenK has joined #openstack-infra | 20:38 | |
| *** rbrndt_ has joined #openstack-infra | 20:39 | |
| *** asselin_ has joined #openstack-infra | 20:39 | |
| fungi | yuval reported that his depends-on that from karbor got devstack-gate jobs passing for them again | 20:39 |
| clarkb | and the jobs we test against still work | 20:39 |
| *** e0ne has joined #openstack-infra | 20:40 | |
| *** rbrndt has quit IRC | 20:40 | |
| fungi | the very next line in the script is a git reset --hard, so i don't think we possibly discard anything which could end up getting used anyway | 20:40 |
| *** ociuhandu has quit IRC | 20:41 | |
| fungi | jeblair: any guess as to whether we need to take similar precautions in zuul-cloner? | 20:42 |
| *** rtheis has quit IRC | 20:42 | |
| *** asselin has quit IRC | 20:42 | |
| *** john-davidge has quit IRC | 20:42 | |
| *** e0ne has quit IRC | 20:42 | |
| sdague | fungi: +A, that seems safe enough | 20:42 |
| Cibo | hello can anybody please tell what should variable NODEPOOL_CLOUD mean? in CI? | 20:43 |
| *** shamail has joined #openstack-infra | 20:43 | |
| fungi | thanks, sdague. apparently they wedged themselves good by merging a .gitattributes file which asserted conditions not present in their branches | 20:44 |
| *** esikachev has quit IRC | 20:44 | |
| pabelanger | Cibo: http://docs.openstack.org/infra/nodepool/scripts.html#ready-script it is the name of the cloud in nodepool | 20:44 |
| Cibo | but what should I set this variable to if I am using local domastick clod? | 20:45 |
| fungi | Cibo: we're lacking a little context. i see that envvar gets used by devstack-gate, kolla, puppet-openstack-integration and tripleo-ci | 20:46 |
| *** tlian has quit IRC | 20:46 | |
| fungi | Cibo: under normal circumstances you should never even need to know about the existence of that variable outside a nodepool-managed ci system | 20:46 |
| fungi | if any projects are baking assumptions about the existence of that variable into developer scripts without having sensible fallbacks when it's unset, that seems like a bug | 20:48 |
| *** shamail has quit IRC | 20:49 | |
| pabelanger | ya, I'd be surprised to see a project use them outside of the gate | 20:49 |
| openstackgerrit | Peter Stachowski proposed openstack/diskimage-builder: Pip install as 10- incompatible with 05-heat-cfntools https://review.openstack.org/410951 | 20:50 |
| *** thorst has quit IRC | 20:51 | |
| *** ociuhandu has joined #openstack-infra | 20:52 | |
| Cibo | the problem is that I have local CI | 20:52 |
| Cibo | and sometimes I get "Testing ICMP connectivity to mirror..defaults.openstack.org" | 20:53 |
| Cibo | that fails =( | 20:53 |
| *** jgriffith is now known as jgriffith_AutoAw | 20:53 | |
| *** savihou has quit IRC | 20:53 | |
| *** rbuzatu has quit IRC | 20:54 | |
| clarkb | that check should be updated to use google.com when not running on nodepool hosts | 20:54 |
| *** ociuhandu has quit IRC | 20:56 | |
| Cibo | can just do some workaround? | 20:57 |
| *** esikachev has joined #openstack-infra | 20:57 | |
| EmilienM | clarkb, fungi: we're dealing with a CI outage in puppet-ceph and need your review on https://review.openstack.org/#/c/410863/ - thanks a lot | 20:57 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Enable n-novnc service https://review.openstack.org/345626 | 20:57 |
| *** dprince has quit IRC | 20:57 | |
| kfox1111 | so, the dns resolution suddenly broke that we figured out the other day | 20:57 |
| kfox1111 | did something change? | 20:57 |
| kfox1111 | seeing it on at least two clouds. | 20:58 |
| fungi | Cibo: looking at http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n120 it seems to be checking that you can reach the pypi service configured in your /etc/pip.conf file | 20:58 |
| fungi | Cibo: odds are you have that set to one of our mirrors | 20:58 |
| mordred | and/or, that you're running the configure_mirror script but do not have mirrors nor have variables describing what they should be called - I'd remove the configure_mirror script if you're running | 20:59 |
| kfox1111 | did the ubuntu images get updated or something? | 20:59 |
| mordred | kfox1111: they get updated nightly | 20:59 |
| kfox1111 | hmm. k. | 20:59 |
| Cibo | like set hm I first time hear about that | 20:59 |
| mordred | kfox1111: but it's also possible when you figured out the fix the other day that it only worked on one of the clouds | 20:59 |
| kfox1111 | mordred: no, it landed on lots of clouds at the time. | 20:59 |
| mordred | kfox1111: and you just now happen to have hit one of the others? | 20:59 |
| mordred | ok. cool | 21:00 |
| kfox1111 | its possible these sets didn't though. | 21:00 |
| kfox1111 | usually it was rax or osc but now its more internap/infracloud | 21:00 |
| kfox1111 | trying to isolate now. | 21:00 |
| kfox1111 | just curious if anything was known changed with regurad to ubuntu config. | 21:00 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ansibly fix_etc_hosts https://review.openstack.org/401975 | 21:01 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ansibly BASE folder creation https://review.openstack.org/402107 | 21:01 |
| jeblair | fungi: wait -- 410810 is to fix a thing where git is *changing* the contents of a repo after checking it out? | 21:01 |
| clarkb | jeblair: yarp | 21:01 |
| jeblair | can we consider not doing *that* instead? :) | 21:02 |
| fungi | Cibo: looking at our network_sanity_check function, i think it's a bug that we directly check connectivity to $NODEPOOL_MIRROR_HOST down on lines 138/139 even if that variable isn't set (and without setting an appropriate fallback behavior either) | 21:02 |
| *** esikachev has quit IRC | 21:02 | |
| pabelanger | yay, after 2 days we built a centos-7 dib | 21:02 |
| clarkb | its a big footgun feature in git imo | 21:02 |
| jeblair | i kind of have this affinity for testing the things actually in the repo | 21:02 |
| fungi | jeblair: we could try to prevent projects from adding .gitattributes files | 21:02 |
| pabelanger | kfox1111: got a log of the failure? | 21:02 |
| Cibo | can I sat it by hand? | 21:02 |
| Cibo | can you please give a GOOD replacement? | 21:03 |
| kfox1111 | pabelanger: several. :) | 21:03 |
| kfox1111 | http://logs.openstack.org/23/410923/2/check/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/3ae2cd0/console.html | 21:03 |
| Cibo | fungi: ??? please? | 21:03 |
| jeblair | i definitely think openstack should adopt a 'no .gitattributes' policy, just because of the likely wide range of opinions on whether that's a good idea :) | 21:03 |
| fungi | Cibo: i'll write a patch to try to fix it in devstack-gate | 21:03 |
| jeblair | but separately, i think we should endeavor to avoid allowing it to alter tests | 21:03 |
| Cibo | the problem is that I do not have match time | 21:04 |
| *** thorst has joined #openstack-infra | 21:06 | |
| pabelanger | kfox1111: dns settings look good on an infracloud-vanilla node | 21:07 |
| ianw | pabelanger: hmm, i guess i need to update the build checker script | 21:07 |
| *** jkilpatr has quit IRC | 21:07 | |
| pabelanger | we are using local unbound server | 21:07 |
| Cibo | what should I asign during to $NODEPOOL_REGION $NODEPOOL_CLOUD during call of devstack gate wrap? | 21:07 |
| rm_work | i forget, is this also the devstack channel? | 21:07 |
| pabelanger | ianw: hoping to finally move that in to statsd soon | 21:07 |
| rm_work | running into an issue that JUST appeared like a couple hours ago: stacking on centos7, qemu-kvm package seems to have broken deps all the sudden | 21:08 |
| pabelanger | kfox1111: we do a good job checking DNS before we pass the node to zuul, so must be intermittent problem | 21:08 |
| ianw | rm_work: #openstack-qa , but i think that's a known issue | 21:08 |
| rm_work | k | 21:08 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Only check connectivity to mirrors when defined https://review.openstack.org/410957 | 21:08 |
| fungi | Cibo: https://review.openstack.org/410957 is what i think you need to fix that | 21:09 |
| fungi | Cibo: in short, don't assign anything to that variable, stop trying to run network tests against a nonexistent system | 21:09 |
| Cibo | ok | 21:10 |
| *** knangia has quit IRC | 21:10 | |
| jeblair | fungi: i think i would have review 410810 with a -1 and a suggestion that karbor simply fix the files in their repo | 21:11 |
| fungi | jeblair: well, fixing the files in their repo would require manual cleanup on our part | 21:11 |
| jeblair | fungi: shouldn't a change to fix those files be self-testing and correcting? | 21:11 |
| fungi | devstack-gate allowed them to merge the change adding that file, but once that file appeared in the on-disk cache of their repo it could no longer checkout a new ref in them | 21:12 |
| fungi | i suppose instead of using checkout -f we could try to do something to in-place apply changes implied by .gitattributes instead? | 21:12 |
| eglute | Hello Infra! I could use some assistance. I am trying to follow http://docs.openstack.org/infra/system-config/irc.html#renaming-an-irc-channel directions and it appears I do not have the power. We are moving openstack-defcore to openstack-interop | 21:13 |
| jeblair | fungi: how did https://review.openstack.org/#/c/410741/ merge then? | 21:15 |
| *** Swami has joined #openstack-infra | 21:15 | |
| jeblair | fungi: ah, it's non-voting | 21:16 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate: Add ARA callback and reporting https://review.openstack.org/405613 | 21:16 |
| fungi | oh, wait, do they not have any voting dsvm jobs? | 21:16 |
| jeblair | apparently not | 21:16 |
| jeblair | fungi: http://logs.openstack.org/57/409757/1/check/gate-karbor-dsvm-fullstack-nv/e90f0bc/logs/devstack-gate-setup-workspace-new.txt.gz | 21:18 |
| *** jgriffith_AutoAw is now known as jgriffith | 21:18 | |
| jeblair | fungi: that's the failing run they ignored for the change that added .gitattributes. | 21:18 |
| *** fguillot has quit IRC | 21:18 | |
| jeblair | i'm going to propose a revert in devstack-gate | 21:18 |
| fungi | please do, i'll approve it | 21:18 |
| jeblair | oh it hasn't landed | 21:18 |
| fungi | oh, i'll abandon then | 21:19 |
| jeblair | i -2d it | 21:19 |
| *** r-daneel has joined #openstack-infra | 21:20 | |
| jeblair | double not merged | 21:20 |
| *** asettle has joined #openstack-infra | 21:21 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ansibly the start of fresh logging https://review.openstack.org/402208 | 21:22 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ansibly the setup of stack user https://review.openstack.org/403664 | 21:22 |
| fungi | gotta run another errand, brb | 21:22 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Ansibly the setup of the tempest user https://review.openstack.org/403727 | 21:22 |
| *** aeng has joined #openstack-infra | 21:22 | |
| mordred | jeblair, fungi: should we insititute some sort of check at the zuul level to prevent landing changes to .gitattributes files? | 21:22 |
| *** jkilpatr has joined #openstack-infra | 21:22 | |
| eglute | mordred fungi who here handles IRC related issues? I could use some help! | 21:24 |
| clarkb | you can do it in gerrit iirc | 21:24 |
| jeblair | mordred: i don't know. this has given me a lot to think about. and what we might want to do for openstack is not necessarily what everyone using zuul might want.... | 21:24 |
| jeblair | eglute: i might be able to help | 21:24 |
| eglute | jeblair that would be great! | 21:24 |
| eglute | I tried doing http://docs.openstack.org/infra/system-config/irc.html#renaming-an-irc-channel | 21:25 |
| *** knangia has joined #openstack-infra | 21:25 | |
| eglute | i set the guard, but the next line didn't work | 21:25 |
| jeblair | eglute: let me catch up -- does the new channel (#openstack-interop exist and is it set up with accessbot in project-config?) | 21:25 |
| *** asettle has quit IRC | 21:25 | |
| jeblair | er, sorry, those parens were all wrong | 21:25 |
| jeblair | but you get the idea :) | 21:25 |
| eglute | yes,the channel exists | 21:26 |
| eglute | and i believe the bot was setup | 21:26 |
| mordred | kfox1111: have you considered using --dns= option to docker? https://docs.docker.com/engine/userguide/networking/default_network/configure-dns/ | 21:26 |
| jeblair | eglute: yeah, openstack is in there, so that should all be good | 21:27 |
| eglute | jeblair yes, the logging is on as well | 21:27 |
| jeblair | eglute: so basically i need to do the stuff starting with guard and on then, yeah? | 21:27 |
| eglute | correct | 21:27 |
| jeblair | eglute: cool, i will execute that in just a moment | 21:28 |
| *** ianychoi has quit IRC | 21:28 | |
| eglute | jeblair thank you, appreciate it!! | 21:28 |
| *** nadya has quit IRC | 21:28 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul: Re-enable test_nonvoting_job https://review.openstack.org/410961 | 21:29 |
| *** EricGonczer_ has quit IRC | 21:30 | |
| *** erlon has quit IRC | 21:30 | |
| openstackgerrit | Merged openstack-infra/shade: Replace swift capabilities call with REST https://review.openstack.org/408865 | 21:31 |
| *** rossella_s has quit IRC | 21:31 | |
| *** EricGonczer_ has joined #openstack-infra | 21:32 | |
| *** rossella_s has joined #openstack-infra | 21:32 | |
| *** nherciu_ has quit IRC | 21:33 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: extra testenv-client logging https://review.openstack.org/410965 | 21:36 |
| jeblair | eglute: okay, that's done -- how long ago did you set the topic and ask people to move? | 21:36 |
| eglute | we had a message on moving for several weeks now | 21:36 |
| eglute | this particular one today | 21:37 |
| eglute | thank you jeblair!!! | 21:37 |
| jeblair | eglute: then probably it's safe to kick everyone out of the defcore channel to make sure the people still there don't talk in it | 21:37 |
| *** palendae has joined #openstack-infra | 21:37 | |
| jeblair | (that seems especially bad since it's now a channel no one else can join) | 21:37 |
| jeblair | eglute: so i'll go ahead and do that now, with a message reminding people why i'm doing it | 21:38 |
| eglute | jeblair agreed! thank you for your help! | 21:38 |
| palendae | I have a change (https://review.openstack.org/#/c/410397/3) failing when reno tries to build off of origin/liberty; is there something to be done to the reno configuration to prevent this? | 21:38 |
| *** kgiusti has left #openstack-infra | 21:38 | |
| *** Goneri has joined #openstack-infra | 21:38 | |
| palendae | New clones locally don't have an origin/liberty ref anymore, presumably because it was EOLed today | 21:39 |
| clarkb | palendae: its a known reno issue dhellmann is working on a fix | 21:39 |
| palendae | clarkb: Thanks! | 21:39 |
| *** adriant has joined #openstack-infra | 21:39 | |
| clarkb | there is a bug for it somewhere if you dig around | 21:39 |
| clarkb | well somewhere being renos bug tracker but ya | 21:39 |
| palendae | Gptcja | 21:39 |
| palendae | Gotcha* | 21:39 |
| palendae | I'll look around to reference it, thanks! | 21:40 |
| *** jgriffith is now known as jgriffith_AutoAw | 21:40 | |
| openstackgerrit | Dirk Mueller proposed openstack-dev/pbr: Remove discover from test-requirements https://review.openstack.org/410966 | 21:41 |
| palendae | https://bugs.launchpad.net/reno/+bug/1649982 - says it's fixed in reno 1.9.0 | 21:41 |
| openstack | Launchpad bug 1649982 in reno "reno doesn't check for an -eol tag when a stable branch is deleted" [Wishlist,Fix committed] - Assigned to Doug Hellmann (doug-hellmann) | 21:41 |
| *** esikachev has joined #openstack-infra | 21:42 | |
| *** itisha has quit IRC | 21:42 | |
| openstackgerrit | Dirk Mueller proposed openstack-dev/pbr: Add Constraints support https://review.openstack.org/410968 | 21:45 |
| *** jgriffith_AutoAw is now known as jgriffith | 21:45 | |
| eglute | thanks again jeblair!! | 21:45 |
| *** sdake_ has quit IRC | 21:46 | |
| jeblair | eglute: no problem. all done now. you might want to set a topic in #openstack-interop | 21:46 |
| eglute | thank you, will do! | 21:46 |
| *** esikachev has quit IRC | 21:47 | |
| dhellmann | palendae, clarkb : we're waiting for the constraints update to allow reno 1.9.0 to be used https://review.openstack.org/#/c/410907/ | 21:47 |
| palendae | dhellmann: Cool, I figured as much based on that bug | 21:47 |
| eglute | jeblair looks like i am not authorized for that channel | 21:47 |
| eglute | jeblair tried setting topic, and failed. | 21:48 |
| EmilienM | hey infra, any chance to have a review on https://review.openstack.org/#/c/410863/ please? | 21:49 |
| clarkb | looks like it failed cross testing cinder in the gate, but things ahead of it may reset and resetart the testing | 21:49 |
| jeblair | eglute: oh i thought you were an op | 21:49 |
| openstackgerrit | Merged openstack-infra/shade: Honor image_api_version when doing version discovery https://review.openstack.org/409270 | 21:49 |
| eglute | jeblair it looks like i am but then i get denied | 21:49 |
| *** sshnaidm is now known as sshnaidm|afk | 21:49 | |
| openstackgerrit | Merged openstack-infra/shade: Change fixtures to use https https://review.openstack.org/409275 | 21:49 |
| eglute | jeblair odd, just tried and it worked again | 21:50 |
| jeblair | eglute: ah, i think you are an op, but not in chanserv's access list (which means you would lose operator status if you rejoined). i'll add you to that list. | 21:50 |
| eglute | jeblair thank you! | 21:50 |
| clarkb | EmilienM: I respodned to your question, will let you decide if y ouwant to update or not | 21:51 |
| EmilienM | clarkb: ok, looking thx | 21:51 |
| EmilienM | clarkb: ok, I prefer to be more specific per scenario, so the current patch is ok I think | 21:52 |
| jeblair | eglute: okay, you're set there too, so you can control chanserv on that channel | 21:54 |
| eglute | thank you jeblair! | 21:54 |
| kfox1111 | mordred: I don't control docker. | 21:54 |
| kfox1111 | k8s does. | 21:54 |
| mordred | ah | 21:54 |
| jeblair | infra-root: it looks like accessbot has not set acls as expected on at least some channels (#openstack-interop for one) | 21:54 |
| *** rbuzatu has joined #openstack-infra | 21:55 | |
| mordred | jeblair: boo | 21:55 |
| mordred | kfox1111: ok. I will learn some things about that | 21:55 |
| clarkb | jeblair: maybe it is suffering the channel join limit too? | 21:55 |
| kfox1111 | mordred: yeah. there's a few layers here. | 21:55 |
| jeblair | clarkb: hrm, i don't think it joins any channels... | 21:55 |
| kfox1111 | deploying k8s with kubeadm. | 21:55 |
| clarkb | jeblair: oh right it can just talk to chanserv | 21:56 |
| *** thorst has quit IRC | 21:56 | |
| kfox1111 | which sets up the nodes for us. but then hides even more details. :/ | 21:56 |
| jeblair | yeah. i mean, maybe chanserv got tired of talking to it | 21:56 |
| *** esikachev has joined #openstack-infra | 21:56 | |
| kfox1111 | hm.m. seeing it on osic now too. | 21:56 |
| kfox1111 | and it was working fine there before for sure. | 21:56 |
| pleia2 | also could have joined during a netsplit from services, or silently failed to connect or something | 21:56 |
| pleia2 | I don't think we have much (any?) fault tolerance for that | 21:57 |
| jeblair | clarkb, pleia2: it appears to be joining without a nick, which ircd is unhappy about. | 21:57 |
| jeblair | based on the log entries | 21:57 |
| pleia2 | doh | 21:57 |
| pleia2 | I don't even know how that can be a thing | 21:57 |
| jeblair | no, also, no timestamps so i don't have a lot of context | 21:57 |
| *** nherciu_ has joined #openstack-infra | 21:57 | |
| kfox1111 | hmm.. this looks interesting: | 21:58 |
| jeblair | looks like maybe the last 8 times it ran | 21:58 |
| jeblair | whenever those were | 21:58 |
| kfox1111 | http://logs.openstack.org/64/410464/3/check/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/ca19529/console.html#_2016-12-14_21_53_57_977772 | 21:58 |
| kfox1111 | its restarted, then I don't see it in a netstat... | 21:58 |
| kfox1111 | maybe its crashing now. | 21:58 |
| jeblair | nick= | 21:59 |
| *** thorst_ has joined #openstack-infra | 21:59 | |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config: neutron/tempest: make tempest dvr-multinode voting https://review.openstack.org/410973 | 21:59 |
| jeblair | that's from the config file | 21:59 |
| mordred | kfox1111: somethingsomething skydns something forwarding resolver ? | 22:00 |
| pabelanger | kfox1111: what is the contents of http://logs.openstack.org/64/410464/3/check/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/ca19529/console.html#_2016-12-14_21_53_57_955712 | 22:00 |
| *** rbuzatu has quit IRC | 22:00 | |
| jhesketh | Morning | 22:00 |
| *** esikachev has quit IRC | 22:00 | |
| mordred | morning jhesketh ! | 22:01 |
| jhesketh | o/ | 22:01 |
| jeblair | what is a "statusbot_accessbot_nick"? | 22:01 |
| *** e0ne has joined #openstack-infra | 22:02 | |
| pleia2 | a reminder that we're terrible at naming things? | 22:02 |
| jeblair | i believe it's an error introduced in the twitterification of the statusbot | 22:02 |
| pleia2 | that's entirely possible, when that went live there were a couple confusing changes | 22:03 |
| jeblair | fix in progress | 22:03 |
| mordred | kfox1111: "If you don’t want this, or if you want a different DNS config for pods, you can use the kubelet’s --resolv-conf flag. Setting it to “” means that pods will not inherit DNS. Setting it to a valid file path means that kubelet will use this file instead of /etc/resolv.conf for DNS inheritance. | 22:03 |
| mordred | kfox1111: but I'm guessing you're not controlling the kubelet yourself, but instead kubeadm does? | 22:04 |
| kfox1111 | mordred: I might be able to override kubelets value. | 22:04 |
| kfox1111 | but to what? | 22:04 |
| kfox1111 | all I have in /etc/resolv.conf is 127.0.0.1. which won't work. | 22:04 |
| *** esikachev has joined #openstack-infra | 22:04 | |
| kfox1111 | is there a place I can read out the real value? | 22:05 |
| *** jcoufal has quit IRC | 22:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Remove statusbot_accessbot_nick https://review.openstack.org/410975 | 22:05 |
| jeblair | pleia2, clarkb: ^ | 22:05 |
| mordred | jeblair: I have no idea how that worked before - the rocketships are not aligned! | 22:06 |
| pabelanger | kfox1111: why are you having unbound only use 172.19.0.1 ? http://git.openstack.org/cgit/openstack/kolla-kubernetes/tree/tools/setup_gate.sh#n18 | 22:06 |
| mordred | (also +2) | 22:06 |
| pabelanger | kfox1111: can you just drop that and use the default 0.0.0. | 22:07 |
| pabelanger | 0 | 22:07 |
| *** EricGonczer_ has quit IRC | 22:07 | |
| pabelanger | 127.0.0.1 in resolv.conf is likely why it is failing | 22:07 |
| mordred | pabelanger: iirc, the idea here is to set up a bridge and tell unbound to listen on the bridge interface | 22:08 |
| mordred | pabelanger: because listening on 0.0.0.0 is dangerous | 22:08 |
| mordred | pabelanger: and yes, the initial problem is that the container does not have access to the same 127.0.0,1 | 22:08 |
| jeblair | is there an ip visible in the container that will route to the host? | 22:09 |
| *** esikachev has quit IRC | 22:09 | |
| pabelanger | mordred: surprise it is dangerous, we just default to 0.0.0.0 on our test nodes | 22:09 |
| mordred | jeblair: that is what kfox1111 was setting up in http://git.openstack.org/cgit/openstack/kolla-kubernetes/tree/tools/setup_gate.sh#n18 | 22:09 |
| clarkb | pabelanger: the concern is reflection attacks iirc | 22:10 |
| clarkb | pabelanger: basically you don't want to become part of someones ddos | 22:10 |
| *** eggshell has joined #openstack-infra | 22:10 | |
| jeblair | iptables covers that for us | 22:10 |
| clarkb | jeblair: except that we allow jobs to edit those (and many do including this one) | 22:10 |
| pabelanger | Ya, I would think iptables would protect the kolla-kubernetes job too | 22:11 |
| mordred | we also allow jobs to edit the unbound config | 22:11 |
| *** rossella_s has quit IRC | 22:11 | |
| clarkb | this is true | 22:11 |
| jeblair | right, i'm just explaining why it's okay for us to listen on 0.0.0.0 (assuming we do, i didn't double check) | 22:11 |
| *** rossella_s has joined #openstack-infra | 22:12 | |
| pabelanger | will have to dig into the kubes some day, but that day is not today :) | 22:12 |
| clarkb | I think one of the complicated things with 0.0.0.0 is going to be setting up iptables such that "external" things can hit it but only if they are "external" and not externakl | 22:14 |
| clarkb | that may be easy or hard depending no how ips are laid out | 22:14 |
| jeblair | mordred, kfox1111: so does the bridge map into the container somehow? | 22:15 |
| *** jerryz has joined #openstack-infra | 22:16 | |
| kfox1111 | pabelanger: its bound only to 172.19.0.1 to ensure it doesn't make it outside the vm. | 22:17 |
| kfox1111 | but the acl allows anything that can talk to it to do so. | 22:17 |
| pabelanger | not sure, why you can't also bind to 127.0.0.1 | 22:17 |
| clarkb | ya I think you want to do both | 22:18 |
| *** palendae has left #openstack-infra | 22:18 | |
| kfox1111 | pabelanger: because pods run in their own network. | 22:18 |
| kfox1111 | 127.0.0.1 to the pod will be the pod's loopback, not the vm's loopback. | 22:18 |
| clarkb | kfox1111: right but you can listen on both is what pabelanger is saying I think | 22:18 |
| kfox1111 | this looks to be the issue: | 22:18 |
| kfox1111 | http://logs.openstack.org/74/410974/1/check/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/fac1471/console.html#_2016-12-14_22_07_19_599044 | 22:18 |
| clarkb | then that way all our jobs also have localhost resolver | 22:19 |
| clarkb | and some may optionally have the other thing | 22:19 |
| kfox1111 | unbound looks to not be taking the config anymore. | 22:19 |
| clarkb | (and we might consider baking in the other address and setting up firewalls appropriately if we want more consistency) | 22:19 |
| kfox1111 | consistency might be nice. yeah. | 22:19 |
| pabelanger | yes, bind to both bridge interface and loopback, then host and container have access to unbound | 22:19 |
| kfox1111 | does unbind actually allow dual binding? | 22:20 |
| pabelanger | yes | 22:20 |
| Zara | tdasilva: just noticed that storyboard came up in your meeting and you asked for a link; along with the production instance we also have a test instance up at https://storyboard-dev.openstack.org/#!/page/about if anyone wants to try things out. | 22:20 |
| kfox1111 | a lot of programs don't allow that. | 22:20 |
| pabelanger | IIRC: interface can be used multiple times | 22:20 |
| *** rbuzatu has joined #openstack-infra | 22:20 | |
| kfox1111 | k. | 22:20 |
| jeblair | yep: https://www.unbound.net/documentation/unbound.conf.html | 22:20 |
| kfox1111 | but, the problem does seem to be it not parsing the config enymore... | 22:20 |
| kfox1111 | hmm.. | 22:20 |
| jeblair | "can be given multiple times" | 22:20 |
| kfox1111 | so. its failing on "interface: 172.19.0.1"... | 22:21 |
| kfox1111 | ah. | 22:23 |
| kfox1111 | they have to be under a server section. | 22:23 |
| pabelanger | also, I don't think we manage unbound.conf in puppet, I know in nodepool-base we just append things to it (include: forwarding.conf). So it might not be consistent across clouds | 22:23 |
| kfox1111 | maybe tis more picky now. | 22:23 |
| *** jimbaker has quit IRC | 22:23 | |
| pabelanger | should be easy to update puppet-unbound to support that | 22:23 |
| *** jeckersb is now known as jeckersb_gone | 22:25 | |
| *** annegentle has quit IRC | 22:25 | |
| *** M-docaedo_vector has quit IRC | 22:26 | |
| *** annegentle has joined #openstack-infra | 22:26 | |
| *** jimbaker has joined #openstack-infra | 22:27 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Only check connectivity to mirrors when defined https://review.openstack.org/410957 | 22:27 |
| fungi | pabelanger: https://review.openstack.org/318714 introduced a hard dependency in devstack-gate on the /etc/nodepool/provider file and variables set within it. i'm thinking ^ is the fix but as the change introducing that problem was yours i'd appreciate your input | 22:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Remove the ability for nodepoold to launch a builder https://review.openstack.org/410939 | 22:28 |
| clarkb | I think we may want a connectivity check of some sort | 22:28 |
| fungi | we already perform one prior to that | 22:28 |
| clarkb | oh we do | 22:28 |
| clarkb | perfect | 22:29 |
| pabelanger | fungi: oh, right | 22:29 |
| *** annegentle has quit IRC | 22:29 | |
| pabelanger | doh | 22:29 |
| *** rkukura has quit IRC | 22:29 | |
| fungi | Cibo brought this up a little while ago, but i had to dash away to do some other stuff so am just now revisiting it | 22:29 |
| pabelanger | +3 | 22:29 |
| fungi | he's trying to run devstack-gate without nodepool, and understandably confused by what that should be doing | 22:29 |
| Cibo | fungi: cool! | 22:30 |
| fungi | presumably for him it was trying to ping "mirror...openstack.org" (literally) because those variables were all unset | 22:31 |
| *** e0ne has quit IRC | 22:31 | |
| *** adrian_otto has joined #openstack-infra | 22:32 | |
| fungi | Cibo: i'm still a little curious how it got that far. i assume you had to manually create a /etc/nodepool/provider file because before that it would have given you a no such file error? | 22:32 |
| Cibo | fungi: may be it is related to softwareFactory that I am using | 22:33 |
| pabelanger | Hmm | 22:33 |
| pabelanger | I think we have an issue with centos-7 | 22:34 |
| Cibo | and I am using centos | 22:34 |
| fungi | pabelanger: unrelated to this topic, presumably? | 22:34 |
| pabelanger | yes, sorry | 22:34 |
| fungi | cool, just making sure | 22:34 |
| *** panda is now known as panda|zZ | 22:35 | |
| fungi | Cibo: for issues with SoftwareFactory, you should probably talk to them. it's similar but it's a fork of what we support so not everything we expect will necessarily work there | 22:36 |
| *** rkukura has joined #openstack-infra | 22:37 | |
| fungi | tristanC knows a little about it and may be able to tell you who to get in touch with, if he's awake already (might still be a little early in his part of the world) | 22:37 |
| Cibo | it is strange because SF is just a manager that configs nodepool and other | 22:38 |
| Cibo | when I have fail in devstack-gate | 22:38 |
| *** eggshell has quit IRC | 22:38 | |
| *** jgriffith is now known as jgriffith_AutoAw | 22:39 | |
| fungi | Cibo: well, you had previously stated you weren't using nodepool | 22:39 |
| Cibo | I do not blame anybody | 22:39 |
| Cibo | realy? | 22:39 |
| *** esikachev has joined #openstack-infra | 22:39 | |
| fungi | or maybe i just got that impression because it's not setting up mirrors for you | 22:39 |
| fungi | at any rate, i strongly suspect https://review.openstack.org/410957 will solve it for you either way | 22:40 |
| Cibo | fungi: may be it is because I am not very good at nodepool | 22:40 |
| Cibo | cool | 22:40 |
| Cibo | my point is that all this stuf becomes quete complex | 22:40 |
| Cibo | tooooo complex | 22:40 |
| fungi | Cibo: well, not all nodepool deployments will necessarily introduce package mirror configuration, so the misunderstanding was mine regardless | 22:41 |
| cloudnull | were there image changes today ? | 22:41 |
| cloudnull | was looking at http://grafana.openstack.org/dashboard/db/nodepool-osic?panelId=11&fullscreen | 22:41 |
| fungi | cloudnull: we hope that there are image updates every day | 22:41 |
| pabelanger | yes, centos-7 DIB is failing to boot | 22:41 |
| pabelanger | looking at it now | 22:41 |
| cloudnull | and curious if that was a spike due to the thunderious hurd | 22:41 |
| pabelanger | wonder if this is a side affect of 7.3 release | 22:42 |
| cloudnull | or if I should go look into things | 22:42 |
| kfox1111 | fungi: pabelanger: adding an explicit Server: section fixed it. | 22:42 |
| cloudnull | I mean its not a super high failure rate | 22:42 |
| cloudnull | but was something that caught my eye | 22:42 |
| *** pradk has quit IRC | 22:42 | |
| Cibo | I build image localy | 22:42 |
| pabelanger | cloudnull: will let you know shortly, just manually booting an image | 22:43 |
| Cibo | oh not about me =) | 22:43 |
| cloudnull | ok. thanks pabelanger. let me know if there's something I can help with | 22:43 |
| *** esikachev has quit IRC | 22:44 | |
| *** baoli has quit IRC | 22:44 | |
| *** gouthamr has quit IRC | 22:45 | |
| *** marst has quit IRC | 22:45 | |
| mordred | pabelanger: oh goodie. centos released 7.3? | 22:46 |
| clarkb | mordred: ya yesterday iirc | 22:46 |
| mordred | \o/ | 22:46 |
| fungi | which would put it in our images starting with the updates we're just now uploading to providers i guess | 22:46 |
| pabelanger | [ 125.064186] dracut-initqueue[155]: Warning: dracut-initqueue timeout - starting timeout scripts | 22:46 |
| pabelanger | is spaming the console log | 22:46 |
| pabelanger | and fail to see glean or ssh being run | 22:47 |
| *** esberglu has quit IRC | 22:47 | |
| *** flepied has quit IRC | 22:49 | |
| *** Swami has quit IRC | 22:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Pause diskimage builds for centos-7 https://review.openstack.org/410988 | 22:50 |
| pabelanger | fungi: mordred: ^ so we can roll back to a working centos-7 | 22:50 |
| pabelanger | A start job is running for dev-disk...222.device (7min 14s / no limit) | 22:51 |
| *** jgriffith_AutoAw is now known as jgriffith | 22:51 | |
| pabelanger | is now spamming the log | 22:51 |
| *** flepied has joined #openstack-infra | 22:51 | |
| fungi | gotta say i like the new pause attribute | 22:52 |
| clarkb | reviewing | 22:52 |
| mordred | fungi: me too | 22:52 |
| mordred | pabelanger: yay! (+A'd) | 22:53 |
| clarkb | mordred wins the get diff to load race | 22:53 |
| *** esikachev has joined #openstack-infra | 22:53 | |
| *** john-davidge has joined #openstack-infra | 22:54 | |
| *** burgerk has quit IRC | 22:55 | |
| *** dave-mccowan has quit IRC | 22:56 | |
| *** gouthamr has joined #openstack-infra | 22:57 | |
| pabelanger | oh neat, config-validate failed | 22:58 |
| clarkb | pabelanger: does it need to be true not True? | 22:58 |
| pabelanger | True should be fine | 22:59 |
| *** aeng has quit IRC | 22:59 | |
| pabelanger | blarg | 22:59 |
| *** john-davidge has quit IRC | 22:59 | |
| pabelanger | guess we don't test config-validate today | 22:59 |
| clarkb | pabelanger: we do | 22:59 |
| clarkb | there are positive and negative tests for it | 22:59 |
| clarkb | but they require being updated when we add new things if ew want to cover the new things | 23:00 |
| *** eharney has quit IRC | 23:00 | |
| pabelanger | ya | 23:00 |
| pabelanger | see the issue | 23:00 |
| *** gordc has quit IRC | 23:00 | |
| pabelanger | okay, this is going to need an update to nodepool, but for now going to place nodepool.o.o into emergency so get centos-7 dibs back online | 23:00 |
| *** vhosakot has quit IRC | 23:01 | |
| pabelanger | sorry, nb01 and nb02 | 23:01 |
| *** sdake has joined #openstack-infra | 23:01 | |
| *** esberglu has joined #openstack-infra | 23:01 | |
| *** tpsilva has quit IRC | 23:01 | |
| *** jheroux has quit IRC | 23:02 | |
| *** mmandel has quit IRC | 23:02 | |
| *** mmandel has joined #openstack-infra | 23:03 | |
| *** esikachev has quit IRC | 23:03 | |
| clarkb | pabelanger: and edit their configs directly? | 23:03 |
| pabelanger | ya | 23:04 |
| *** thorst_ has quit IRC | 23:04 | |
| pabelanger | working on nodepool patch too | 23:04 |
| ianw | i'll take a look at these boot issues | 23:04 |
| ianw | did greghaynes release overnight? | 23:05 |
| *** rcernin has quit IRC | 23:05 | |
| pabelanger | #status log nb01.o.o and nb02.o.o added to emergency file on puppetmaster. To manually apply https://review.openstack.org/#/c/410988/ | 23:06 |
| openstackstatus | pabelanger: finished logging | 23:06 |
| pabelanger | ianw: I don't believe so | 23:06 |
| clarkb | ianw: git says no | 23:06 |
| clarkb | I think more likely realted to new centos release | 23:06 |
| ianw | yep, cool, well that's one less thing at least | 23:06 |
| *** esberglu has quit IRC | 23:06 | |
| *** mmandel has quit IRC | 23:07 | |
| *** sdake has quit IRC | 23:08 | |
| clarkb | snow is far too distracting | 23:08 |
| *** nadya has joined #openstack-infra | 23:08 | |
| *** rlandy has quit IRC | 23:09 | |
| clarkb | pabelanger: nodepool/tests/test_config_validator.py is where the config validator is tested | 23:10 |
| clarkb | I think you just edit the fixtures it reads to include new stuff to double check it | 23:10 |
| pabelanger | clarkb: Ack, writing test now | 23:10 |
| clarkb | not sure you need a new test | 23:11 |
| clarkb | just update existing fixture(s) but ya | 23:11 |
| pabelanger | ya, that | 23:11 |
| pabelanger | okay, we should have centos-7 nodes coming back online now | 23:12 |
| *** trown|outtypewww has quit IRC | 23:12 | |
| *** nadya has quit IRC | 23:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add missing pause fields to config-validate https://review.openstack.org/410992 | 23:13 |
| *** trown has joined #openstack-infra | 23:14 | |
| *** mtanino has quit IRC | 23:15 | |
| *** thorst_ has joined #openstack-infra | 23:15 | |
| *** mtanino has joined #openstack-infra | 23:15 | |
| *** dave-mccowan has joined #openstack-infra | 23:16 | |
| *** sdake has joined #openstack-infra | 23:16 | |
| *** jschwarz has quit IRC | 23:16 | |
| *** sdake has quit IRC | 23:16 | |
| pabelanger | ianw: some quick googles makes me think something with the partitions or mounts | 23:17 |
| *** migi has quit IRC | 23:17 | |
| ianw | pabelanger: yeah, we recently had some issues with labels | 23:17 |
| ianw | well, the full story is long and boring, and actually came down to systemd in a roundabout way :) as usual | 23:18 |
| *** migi has joined #openstack-infra | 23:18 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove statusbot_accessbot_nick https://review.openstack.org/410975 | 23:18 |
| fungi | all roads lead to systemd | 23:18 |
| greghaynes | ooo this sounds like fun | 23:19 |
| greghaynes | oh, ianw you got it figured out? | 23:19 |
| jeblair | clarkb, pabelanger: what we should do is have nodepool validate a config before using it. then we would have caught this when we added the tests for the underlying functionality. | 23:19 |
| clarkb | jeblair: thats a good idea, would be simple to add that to the function that applies the config | 23:20 |
| clarkb | iirc thats a common test thing | 23:20 |
| ianw | greghaynes: well not the current failure ; just spinning up a local build & see where that's at. we probably do want to build & boot centos in dib gate though too. currently we only try ubuntu with the nodepool job | 23:20 |
| *** yamahata_ has quit IRC | 23:21 | |
| ianw | of course that's a bit of a speed hit. maybe separate jobs so it happens in parallel | 23:21 |
| jeblair | yeah, or maybe it can be added to the nodepool class itself, i forget what the plumbing looks like in there right now | 23:21 |
| * clarkb will write that patch now while thinking bout it | 23:21 | |
| *** jschwarz has joined #openstack-infra | 23:21 | |
| clarkb | otherwise I will just stare out the window at the snow | 23:21 |
| greghaynes | ianw: yea I need to dust off my boot images in our CI patch | 23:21 |
| jeblair | clarkb: still with the snow? | 23:21 |
| *** asettle has joined #openstack-infra | 23:22 | |
| pabelanger | jeblair: agreed, this is more then the first time we've had this issue | 23:22 |
| clarkb | jeblair: round two | 23:22 |
| *** ijw has joined #openstack-infra | 23:22 | |
| pabelanger | clarkb: nice, only round one here. Hoping for a big dump in the next week | 23:23 |
| *** ianychoi has joined #openstack-infra | 23:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/410995 | 23:26 |
| *** tphummel has quit IRC | 23:26 | |
| clarkb | jeblair: pabelanger ^ I think its as simple as that | 23:26 |
| clarkb | pabelanger: I mean for us 1" is world ending | 23:26 |
| clarkb | supposedly due for 3-5" today/tonight | 23:26 |
| *** dave-mccowan has quit IRC | 23:27 | |
| clarkb | its just so distracting | 23:27 |
| *** asettle has quit IRC | 23:27 | |
| jeblair | clarkb: lgtm | 23:27 |
| clarkb | (part of that I am sure is I grew up on a tropical island, so snow is incredibly novel to me) | 23:27 |
| *** sdake has joined #openstack-infra | 23:28 | |
| pabelanger | clarkb: yar, I was out in Vancouver, BC in 1999. I want to say they got 6 inches in a shot. Shutdown the city for days | 23:30 |
| pabelanger | infact, I think snow blows came up from US to help | 23:30 |
| clarkb | we are also hampered by the fact we don't use the good salt on our roads | 23:30 |
| clarkb | because the good stuff is bad for salmon | 23:30 |
| *** esikachev has joined #openstack-infra | 23:31 | |
| *** sdague has quit IRC | 23:31 | |
| *** Julien-zte has quit IRC | 23:32 | |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config: Update octavia projects for octavia spinout https://review.openstack.org/410997 | 23:32 |
| pabelanger | neat | 23:33 |
| pabelanger | http://logs.openstack.org/92/410992/1/check/gate-nodepool-python27-ubuntu-xenial/3ccd075/console.html#_2016-12-14_23_26_49_464170 | 23:33 |
| pabelanger | first time seeing that | 23:33 |
| mordred | clarkb: salmon are tasty | 23:34 |
| clarkb | mordred: yes considering our track record I think thats the only reason we try so hard to save them | 23:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable net-iso in ha jobs https://review.openstack.org/410998 | 23:35 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable pacemaker in the updates job https://review.openstack.org/410999 | 23:35 |
| *** esikachev has quit IRC | 23:36 | |
| *** sdake_ has joined #openstack-infra | 23:36 | |
| *** thorst_ has quit IRC | 23:36 | |
| *** sdake has quit IRC | 23:37 | |
| *** chlong has quit IRC | 23:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/zuul: WIP Add per-repo public and private keys https://review.openstack.org/406382 | 23:41 |
| *** esikachev has joined #openstack-infra | 23:41 | |
| ianw | pabelanger: "No bootable disk" -- is that similar to what you saw in the logs? | 23:41 |
| openstackgerrit | Merged openstack/diskimage-builder: Pip install as 10- incompatible with 05-heat-cfntools https://review.openstack.org/410951 | 23:42 |
| ianw | i'm not sure if it's my image, or the cloud underneath! the fun of so many layers | 23:42 |
| pabelanger | ianw: don;t think so | 23:42 |
| pabelanger | but I do see | 23:42 |
| pabelanger | Welcome to [0;34mCentOS Linux 7 (Core) dracut-033-463.el7 (Initramfs)[0m! | 23:42 |
| pabelanger | ramdisk? | 23:43 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable pacemaker in the updates job https://review.openstack.org/410999 | 23:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: extra testenv-client logging https://review.openstack.org/410965 | 23:44 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Disable net-iso in ha jobs https://review.openstack.org/410998 | 23:44 |
| ianw | ahh, yeah, that might be my issue. trying again | 23:44 |
| *** dingyichen has joined #openstack-infra | 23:45 | |
| *** baoli has joined #openstack-infra | 23:46 | |
| clarkb | jeblair: pabelanger http://logs.openstack.org/95/410995/1/check/gate-nodepool-python27-ubuntu-xenial/214a080/console.html#_2016-12-14_23_29_33_304789 it worked :) and by worked I mean failed beacuse pause isn't valid | 23:48 |
| clarkb | will rebase on pabelanger's change | 23:49 |
| pabelanger | nice | 23:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/411002 | 23:49 |
| clarkb | and now to review the pause fix | 23:50 |
| clarkb | er wait | 23:50 |
| clarkb | did that make a new change? | 23:50 |
| mordred | clarkb: yes | 23:50 |
| clarkb | oh because pabelanger's fix is not on master | 23:50 |
| mordred | 0995 and 1002 | 23:50 |
| clarkb | pabelanger you need to make your change on master | 23:50 |
| jeblair | ++ | 23:50 |
| clarkb | since that is what we deploy now | 23:50 |
| clarkb | (and based on jeblair's breakdown of where patches go from before I think its the right spot) | 23:51 |
| pabelanger | Oh, ya. let me fix | 23:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add missing pause fields to config-validate https://review.openstack.org/411004 | 23:52 |
| pabelanger | that's my bad | 23:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Validate configs when used by tests https://review.openstack.org/410995 | 23:53 |
| clarkb | tehre we go | 23:53 |
| mordred | ++ | 23:55 |
| clarkb | I'll leave the zuulv3 change up if we want to also fix that branch | 23:55 |
| clarkb | without doing an immeditate merge | 23:55 |
| *** lindycoder has quit IRC | 23:56 | |
| *** Swami has joined #openstack-infra | 23:56 | |
| *** rbrndt_ has quit IRC | 23:57 | |
| *** rbrndt_ has joined #openstack-infra | 23:58 | |
| *** rbrndt_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!