| *** Swami has quit IRC | 00:00 | |
| *** nwkarsten has quit IRC | 00:00 | |
| *** nwkarsten has joined #openstack-infra | 00:01 | |
| *** dingyichen has joined #openstack-infra | 00:01 | |
| *** thorst has quit IRC | 00:01 | |
| *** roxanaghe has quit IRC | 00:02 | |
| *** kzaitsev_mb has quit IRC | 00:02 | |
| *** kaisers_ has joined #openstack-infra | 00:03 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip unit tests and linuxbridge job on Neutron doc changes https://review.openstack.org/360827 | 00:04 |
|---|---|---|
| *** zhurong has quit IRC | 00:04 | |
| *** xarses has quit IRC | 00:04 | |
| *** Julien-zte has quit IRC | 00:04 | |
| *** nwkarsten has quit IRC | 00:05 | |
| *** kaisers_ has quit IRC | 00:08 | |
| *** tqtran_ has quit IRC | 00:11 | |
| *** gongysh has joined #openstack-infra | 00:13 | |
| *** sflanigan has joined #openstack-infra | 00:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:14 | |
| *** ddieterly has joined #openstack-infra | 00:18 | |
| *** furlongm_ has quit IRC | 00:19 | |
| ianw | zaro: does any this mean anything to you? odd projects? -> http://paste.openstack.org/show/563698/ | 00:19 |
| *** kzaitsev_mb has quit IRC | 00:21 | |
| *** david-lyle has quit IRC | 00:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add internap-mtl01 to cacti.o.o https://review.openstack.org/360832 | 00:24 |
| *** tqtran has joined #openstack-infra | 00:24 | |
| pabelanger | ianw: sigh: http://logs.openstack.org/18/360818/1/gate/gate-openstackci-beaker-centos-7/abed281/console.html | 00:26 |
| pabelanger | ianw: I really need to get our gem mirror working | 00:26 |
| ianw | :( | 00:27 |
| *** jistr has quit IRC | 00:27 | |
| *** tonytan4ever has quit IRC | 00:28 | |
| *** gyee has quit IRC | 00:28 | |
| *** jistr has joined #openstack-infra | 00:30 | |
| *** thorst has joined #openstack-infra | 00:30 | |
| *** thorst has quit IRC | 00:37 | |
| *** fguillot has joined #openstack-infra | 00:37 | |
| *** thorst has joined #openstack-infra | 00:38 | |
| *** esberglu has joined #openstack-infra | 00:39 | |
| *** esberglu has quit IRC | 00:43 | |
| *** mtanino has quit IRC | 00:44 | |
| *** salv-orlando has joined #openstack-infra | 00:46 | |
| *** thorst has quit IRC | 00:46 | |
| *** gouthamr_ has quit IRC | 00:47 | |
| *** nwkarsten has joined #openstack-infra | 00:48 | |
| *** salv-orlando has quit IRC | 00:51 | |
| *** zz_dimtruck is now known as dimtruck | 00:51 | |
| *** nwkarsten has quit IRC | 00:53 | |
| *** Julien-zte has joined #openstack-infra | 00:53 | |
| *** rajinir has quit IRC | 00:55 | |
| *** tqtran has quit IRC | 00:57 | |
| *** Jeffrey4l__ has joined #openstack-infra | 01:04 | |
| *** tqtran has joined #openstack-infra | 01:08 | |
| *** dimtruck is now known as zz_dimtruck | 01:08 | |
| *** shashank_hegde has quit IRC | 01:08 | |
| ianw | zaro: so i think my conclusion is that because the new review-dev gets set up with a connection to the remote mysql host, it's not actually possible to re-deploy a new review-dev. the new host is trying to share the same, already initalised, database and that just doesn't work. | 01:09 |
| ianw | thus i'm not exactly sure what to do | 01:10 |
| *** jogo has quit IRC | 01:10 | |
| *** jogo has joined #openstack-infra | 01:12 | |
| *** yamahata has quit IRC | 01:13 | |
| ianw | i guess shut down the old one, reattach the gerrit cinder volume to the new one, update dns and cross fingers | 01:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Double rax-ord boot-timeout value" https://review.openstack.org/360837 | 01:15 |
| *** thcipriani is now known as thcipriani|afk | 01:15 | |
| pabelanger | ianw: so, we can revert ^. I wanted to see if it helped our launch failure rate in rax-ord, but it didn't. I'll have to open a ticket in the morning with rackspace to help debug the issue | 01:16 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-rackspace is an example of the issue | 01:16 |
| *** zhurong has joined #openstack-infra | 01:17 | |
| *** thorst has joined #openstack-infra | 01:18 | |
| *** thorst has quit IRC | 01:19 | |
| *** chlong has joined #openstack-infra | 01:19 | |
| *** thorst has joined #openstack-infra | 01:19 | |
| *** Apoorva_ has joined #openstack-infra | 01:19 | |
| ianw | pabelanger: hmm, yeah, doesn't seem like much you can do without logs | 01:23 |
| *** Apoorva has quit IRC | 01:23 | |
| *** yamahata has joined #openstack-infra | 01:27 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Disable rax-iad due to launch failure rate https://review.openstack.org/360842 | 01:27 |
| *** yanyanhu has joined #openstack-infra | 01:27 | |
| pabelanger | ianw: Ya, in fact, I propose we disable rax-iad^ until we can work with rackspace | 01:27 |
| *** raunak has quit IRC | 01:27 | |
| *** thorst has quit IRC | 01:28 | |
| pabelanger | we're basically flooding the debug logs for nodepool with ssh exceptions | 01:28 |
| ianw | yeah, i noticed that | 01:28 |
| pabelanger | fungi: if you are still around ^ | 01:28 |
| ianw | jhesketh: is this something anyone in a currently awake timezone can help with? | 01:28 |
| *** Julien-zte has quit IRC | 01:29 | |
| jhesketh | ianw: sure, I'm happy to look but get on a plane in a few hours | 01:30 |
| *** ddieterly has quit IRC | 01:31 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add -ubuntu-trusty to gate-grenade-dsvm https://review.openstack.org/360843 | 01:32 |
| *** gildub has joined #openstack-infra | 01:33 | |
| ianw | jhesketh: i guess it's a matter of correlating some of the failure to launch's seen in nodepool.log with some sort of internal log that gives more details? | 01:33 |
| jhesketh | pabelanger: I'm okay with disabling IAD, (just +2'd), but I don't think I'll be around to undo that if it recovers | 01:34 |
| *** yamamoto has quit IRC | 01:34 | |
| *** sdague has joined #openstack-infra | 01:34 | |
| jhesketh | ianw: do you mean internal in rackspace public cloud? | 01:34 |
| *** Julien-zte has joined #openstack-infra | 01:35 | |
| *** tonytan4ever has joined #openstack-infra | 01:35 | |
| ianw | jhesketh: well, yeah, in IAD | 01:35 |
| jhesketh | ianw: right, that's not something I have access to. We'd have to raise a ticket in public cloud | 01:36 |
| jhesketh | ianw, pabelanger: is all we have to go off now the ssh exceptions? | 01:37 |
| *** tqtran has quit IRC | 01:39 | |
| *** AnarchyAo has joined #openstack-infra | 01:40 | |
| *** tonytan4ever has quit IRC | 01:40 | |
| ianw | jhesketh: i believe so | 01:41 |
| openstackgerrit | Merged openstack-infra/system-config: Add internap-mtl01 to nodepool clouds.yaml https://review.openstack.org/360818 | 01:44 |
| *** roxanaghe has joined #openstack-infra | 01:46 | |
| *** pvaneck has quit IRC | 01:47 | |
| *** ddieterly has joined #openstack-infra | 01:48 | |
| *** jamielennox is now known as jamielennox|away | 01:50 | |
| *** roxanaghe has quit IRC | 01:50 | |
| *** kaisers_ has joined #openstack-infra | 01:53 | |
| *** salv-orlando has joined #openstack-infra | 01:53 | |
| *** thorst has joined #openstack-infra | 01:55 | |
| *** thorst has quit IRC | 01:55 | |
| *** thorst has joined #openstack-infra | 01:56 | |
| pabelanger | jhesketh: ianw: Yes, we've need SSH failures into rax-iad after we launch a node. Both ipv4 and ipv6 are non functional. I bumped out timeout to 240 seconds the other day, but that didn't help | 01:56 |
| *** kaisers_ has quit IRC | 01:57 | |
| pabelanger | I plan on raising the ticket in the morning too | 01:58 |
| *** jamielennox|away is now known as jamielennox | 02:00 | |
| pabelanger | ianw: we should be ready to land https://review.openstack.org/#/c/360820/ depends is live | 02:01 |
| jhesketh | pabelanger: okay, I'll leave it to you to raise as you've been tracking the issue closely and I'll be heading into the weekend | 02:01 |
| ianw | let's do it! | 02:02 |
| ianw | jhesketh: do you know about rackdns (the command ... and what's behind it i guess) | 02:02 |
| ianw | i believe i've cut review-dev.openstack.org over to the new server | 02:03 |
| ianw | but "rackdns record-list openstack.org | grep review-dev" returns nothing | 02:03 |
| *** salv-orlando has quit IRC | 02:04 | |
| *** ddieterly has quit IRC | 02:04 | |
| *** thorst has quit IRC | 02:04 | |
| jhesketh | ianw: I've not used it sorry, I've only ever worked with single records so I just used the webui | 02:05 |
| *** fguillot has quit IRC | 02:06 | |
| ianw | maybe i should poke at that and see if it's consistent | 02:06 |
| ianw | i think the 3600 ttl is telling me it's time to get some lunch and check it later; maybe it will sort itself out :) | 02:06 |
| *** jogo has quit IRC | 02:07 | |
| *** jogo has joined #openstack-infra | 02:08 | |
| *** sdake_ has joined #openstack-infra | 02:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o https://review.openstack.org/360820 | 02:08 |
| *** Sukhdev has quit IRC | 02:09 | |
| *** fguillot has joined #openstack-infra | 02:09 | |
| *** sdake has quit IRC | 02:12 | |
| *** _ari_ has joined #openstack-infra | 02:13 | |
| *** yamamoto has joined #openstack-infra | 02:13 | |
| *** raunak has joined #openstack-infra | 02:16 | |
| *** thorst has joined #openstack-infra | 02:16 | |
| *** sdake_ has quit IRC | 02:17 | |
| *** thorst has quit IRC | 02:17 | |
| *** sdague has quit IRC | 02:17 | |
| openstackgerrit | Merged openstack-infra/zuul: Always create foreign projects if needed https://review.openstack.org/360105 | 02:17 |
| *** ddieterly has joined #openstack-infra | 02:19 | |
| *** sdake has joined #openstack-infra | 02:19 | |
| *** tphummel has quit IRC | 02:19 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Fix projects entering pipeline without jobs https://review.openstack.org/334378 | 02:21 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Projects without jobs still enter pipeline https://review.openstack.org/302800 | 02:21 |
| *** sdake has quit IRC | 02:23 | |
| *** smarcet has joined #openstack-infra | 02:26 | |
| *** sdake has joined #openstack-infra | 02:29 | |
| *** ddieterly has quit IRC | 02:34 | |
| *** sflanigan is now known as sflanigan_afk | 02:34 | |
| *** raunak has quit IRC | 02:36 | |
| *** xarses has joined #openstack-infra | 02:39 | |
| *** Wei_Liu has quit IRC | 02:41 | |
| *** yamahata has quit IRC | 02:43 | |
| *** brad_behle_ has quit IRC | 02:44 | |
| *** smarcet has quit IRC | 02:45 | |
| *** ddieterly has joined #openstack-infra | 02:46 | |
| *** roxanaghe has joined #openstack-infra | 02:46 | |
| *** ddieterly has quit IRC | 02:46 | |
| *** Genek has joined #openstack-infra | 02:47 | |
| *** thorst has joined #openstack-infra | 02:47 | |
| *** mjb has quit IRC | 02:47 | |
| *** StevenK has quit IRC | 02:48 | |
| *** onovy has quit IRC | 02:48 | |
| *** vkramskikh has quit IRC | 02:48 | |
| *** rm_work has quit IRC | 02:48 | |
| *** onovy has joined #openstack-infra | 02:48 | |
| *** StevenK has joined #openstack-infra | 02:49 | |
| *** mjb has joined #openstack-infra | 02:49 | |
| *** vkramskikh has joined #openstack-infra | 02:49 | |
| *** roxanaghe has quit IRC | 02:50 | |
| *** rm_work has joined #openstack-infra | 02:52 | |
| *** thorst has quit IRC | 02:52 | |
| *** thorst has joined #openstack-infra | 02:53 | |
| *** fguillot has quit IRC | 02:53 | |
| *** AnarchyAo has quit IRC | 02:54 | |
| *** bexelbie has quit IRC | 02:54 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Fix gate_hook check https://review.openstack.org/360312 | 02:54 |
| *** dansmith has quit IRC | 02:54 | |
| *** _alastor_ has quit IRC | 02:55 | |
| *** katyafervent_awa has quit IRC | 02:55 | |
| *** rakhmerov has quit IRC | 02:55 | |
| *** bradm has quit IRC | 02:55 | |
| *** cj has quit IRC | 02:55 | |
| *** briancline has quit IRC | 02:55 | |
| *** Hunner has quit IRC | 02:55 | |
| *** bradm has joined #openstack-infra | 02:56 | |
| *** yanyanhu has quit IRC | 02:56 | |
| *** jamielennox is now known as jamielennox|away | 02:56 | |
| *** cj has joined #openstack-infra | 02:56 | |
| *** briancline has joined #openstack-infra | 02:57 | |
| *** Wei_Liu has joined #openstack-infra | 02:57 | |
| *** yanyanhu has joined #openstack-infra | 02:58 | |
| *** dansmith has joined #openstack-infra | 03:00 | |
| *** dansmith is now known as Guest71645 | 03:01 | |
| *** thorst has quit IRC | 03:01 | |
| *** Hunner has joined #openstack-infra | 03:01 | |
| *** Hunner has quit IRC | 03:01 | |
| *** Hunner has joined #openstack-infra | 03:01 | |
| *** bexelbie has joined #openstack-infra | 03:03 | |
| *** salv-orlando has joined #openstack-infra | 03:03 | |
| *** katyafervent_awa has joined #openstack-infra | 03:03 | |
| *** rakhmerov has joined #openstack-infra | 03:04 | |
| *** _alastor1 has joined #openstack-infra | 03:04 | |
| *** salv-orlando has quit IRC | 03:06 | |
| *** jamielennox|away is now known as jamielennox | 03:10 | |
| *** gongysh has quit IRC | 03:10 | |
| *** sflanigan_afk is now known as sflanigan | 03:11 | |
| *** sdake has quit IRC | 03:12 | |
| *** sdake has joined #openstack-infra | 03:13 | |
| *** baoli_ has quit IRC | 03:14 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenStackId 1.0.19 https://review.openstack.org/360862 | 03:15 |
| *** NomNKDJIvx has quit IRC | 03:16 | |
| *** Genek has quit IRC | 03:18 | |
| *** smarcet has joined #openstack-infra | 03:19 | |
| *** Genek has joined #openstack-infra | 03:19 | |
| smarcet | fungi: hey how are u doing ? i u see this , please review https://review.openstack.org/#/c/360862/1 need it asap on production , cheers! | 03:19 |
| ianw | jhesketh: well, "rackdns record-list openstack.org" shows way less records than what you seen if you log in and use the control panel dns. i wonder if it's paginating the api response correctly. a yak for another day | 03:20 |
| pabelanger | ianw: just restarted nodepool-buidler | 03:25 |
| pabelanger | builder* | 03:25 |
| ianw | ok, is it uploading? | 03:26 |
| pabelanger | looks like it | 03:26 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-internap | 03:27 |
| pabelanger | I need to add image-uploads to grafana | 03:27 |
| *** sarob has joined #openstack-infra | 03:27 | |
| pabelanger | ianw: I noticed you removed the custom kernel for f24, guess your patches landed upstream? | 03:27 |
| ianw | i got it into the fedora pkg | 03:28 |
| pabelanger | great | 03:28 |
| ianw | i've got one devstack patch, then i'm ready to cut over | 03:28 |
| ianw | we can kill f23 then | 03:28 |
| pabelanger | nice | 03:28 |
| *** smarcet has quit IRC | 03:32 | |
| *** sarob has quit IRC | 03:32 | |
| *** Genek has quit IRC | 03:34 | |
| *** chlong has quit IRC | 03:37 | |
| *** kaisers_ has joined #openstack-infra | 03:41 | |
| *** Sukhdev has joined #openstack-infra | 03:43 | |
| *** shashank_hegde has joined #openstack-infra | 03:44 | |
| *** kaisers_ has quit IRC | 03:46 | |
| *** roxanaghe has joined #openstack-infra | 03:47 | |
| *** roxanaghe has quit IRC | 03:51 | |
| craige | If you're about and available, https://review.openstack.org/#/c/360862 could do with an urgnet review, jhesketh | 03:52 |
| jhesketh | craige: +2 | 03:54 |
| jhesketh | I'm off to the airport now so I can't baby sit it sorry | 03:55 |
| craige | have fun, thanks jhesketh | 03:56 |
| *** Genek has joined #openstack-infra | 03:56 | |
| craige | approve all the papers you know I want to see, jhesketh ;-) | 03:56 |
| *** sdake has quit IRC | 03:56 | |
| *** gongysh has joined #openstack-infra | 03:57 | |
| *** thorst has joined #openstack-infra | 03:59 | |
| *** Genek has quit IRC | 03:59 | |
| *** yonglihe has quit IRC | 04:04 | |
| *** armax has quit IRC | 04:06 | |
| *** thorst has quit IRC | 04:07 | |
| *** mdrabe has quit IRC | 04:07 | |
| *** salv-orlando has joined #openstack-infra | 04:07 | |
| *** links has joined #openstack-infra | 04:07 | |
| *** rlandy|bbl is now known as rlandy | 04:07 | |
| *** chlong has joined #openstack-infra | 04:10 | |
| *** docaedo4 has quit IRC | 04:10 | |
| *** docaedo4 has joined #openstack-infra | 04:11 | |
| *** Julien-zte has quit IRC | 04:12 | |
| *** yamahata has joined #openstack-infra | 04:14 | |
| *** senk has joined #openstack-infra | 04:14 | |
| *** salv-orlando has quit IRC | 04:14 | |
| *** Genek has joined #openstack-infra | 04:16 | |
| *** Genek has quit IRC | 04:20 | |
| *** Genek has joined #openstack-infra | 04:21 | |
| *** sdake has joined #openstack-infra | 04:22 | |
| *** mdrabe has joined #openstack-infra | 04:23 | |
| *** apetrich has quit IRC | 04:25 | |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin https://review.openstack.org/317817 | 04:25 |
| *** apetrich has joined #openstack-infra | 04:25 | |
| *** mdrabe has quit IRC | 04:30 | |
| ianw | pabelanger: ha ha, famous last words. "msg": "ERROR! mkdir: cannot create directory ‘/home/jenkins/.ansible/tmp/ansible-tmp-1472167685.83-197144362572539’: No space left on device\n", <- ... this is a new one for f24 | 04:35 |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Don't ignore data-files https://review.openstack.org/345210 | 04:36 |
| *** Genek has quit IRC | 04:39 | |
| *** kushal has quit IRC | 04:43 | |
| clarkb | ianw: maybe growroot isnt working? | 04:44 |
| ianw | clarkb: yeah, i dunno ... it *was* working a day or so ago | 04:45 |
| *** jaosorior has joined #openstack-infra | 04:46 | |
| *** psachin has joined #openstack-infra | 04:49 | |
| *** _nadya_ has joined #openstack-infra | 04:52 | |
| *** tonytan4ever has joined #openstack-infra | 04:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:56 | |
| *** tonytan4ever has quit IRC | 04:57 | |
| *** yamahata has quit IRC | 05:01 | |
| *** reed_ has joined #openstack-infra | 05:03 | |
| *** thorst has joined #openstack-infra | 05:04 | |
| fungi | craige: i've approved 360862 so hopefully it'll update shortly | 05:06 |
| *** ilyashakhat has joined #openstack-infra | 05:06 | |
| *** senk has quit IRC | 05:11 | |
| *** thorst has quit IRC | 05:11 | |
| *** _nadya_ has quit IRC | 05:11 | |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackId 1.0.19 https://review.openstack.org/360862 | 05:12 |
| *** Sukhdev has quit IRC | 05:13 | |
| *** _nadya_ has joined #openstack-infra | 05:13 | |
| *** yamahata has joined #openstack-infra | 05:15 | |
| *** moravec1 has joined #openstack-infra | 05:15 | |
| *** moravec has quit IRC | 05:16 | |
| *** salv-orlando has joined #openstack-infra | 05:17 | |
| *** roxanaghe has joined #openstack-infra | 05:18 | |
| *** kzaitsev_mb has quit IRC | 05:19 | |
| *** salv-orlando has quit IRC | 05:20 | |
| *** chlong has quit IRC | 05:21 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:21 | |
| *** roxanaghe has quit IRC | 05:22 | |
| *** salv-orlando has joined #openstack-infra | 05:25 | |
| *** salv-orlando has quit IRC | 05:25 | |
| *** gildub has quit IRC | 05:25 | |
| *** salv-orlando has joined #openstack-infra | 05:26 | |
| *** sarob has joined #openstack-infra | 05:28 | |
| *** raunak has joined #openstack-infra | 05:28 | |
| *** kaisers_ has joined #openstack-infra | 05:30 | |
| *** reed_ has quit IRC | 05:31 | |
| *** sarob has quit IRC | 05:32 | |
| *** sandanar has joined #openstack-infra | 05:34 | |
| *** kaisers_ has quit IRC | 05:35 | |
| *** gildub has joined #openstack-infra | 05:41 | |
| *** ilyashakhat has quit IRC | 05:43 | |
| *** gouthamr has joined #openstack-infra | 05:44 | |
| *** senk has joined #openstack-infra | 05:49 | |
| *** kzaitsev_mb has quit IRC | 05:49 | |
| *** _nadya_ has quit IRC | 05:49 | |
| *** _nadya_ has joined #openstack-infra | 05:50 | |
| *** tphummel has joined #openstack-infra | 05:54 | |
| *** _nadya_ has quit IRC | 05:54 | |
| *** raunak has quit IRC | 05:59 | |
| *** raunak has joined #openstack-infra | 06:01 | |
| openstackgerrit | Merged openstack-infra/system-config: Increase httpd threads for review.o.o https://review.openstack.org/360744 | 06:01 |
| *** pgadiya has joined #openstack-infra | 06:04 | |
| *** _nadya_ has joined #openstack-infra | 06:06 | |
| *** matbu|afk is now known as matbu | 06:07 | |
| *** thorst has joined #openstack-infra | 06:09 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Grafana: Use smartSummarize where appropriate https://review.openstack.org/360912 | 06:09 |
| *** aeng has quit IRC | 06:13 | |
| *** Julien-zte has joined #openstack-infra | 06:14 | |
| *** raunak has quit IRC | 06:14 | |
| *** ilyashakhat has joined #openstack-infra | 06:16 | |
| *** thorst has quit IRC | 06:16 | |
| *** evgenyl has quit IRC | 06:20 | |
| *** evgenyl has joined #openstack-infra | 06:22 | |
| *** shardy has joined #openstack-infra | 06:23 | |
| *** rcernin has joined #openstack-infra | 06:25 | |
| *** salv-orlando has quit IRC | 06:31 | |
| *** caowei has joined #openstack-infra | 06:33 | |
| *** ccamacho has left #openstack-infra | 06:37 | |
| *** tesseract- has joined #openstack-infra | 06:44 | |
| *** salv-orlando has joined #openstack-infra | 06:45 | |
| *** _nadya_ has quit IRC | 06:49 | |
| *** _nadya_ has joined #openstack-infra | 06:49 | |
| *** adriant has quit IRC | 06:53 | |
| *** _nadya_ has quit IRC | 06:54 | |
| *** amotoki has joined #openstack-infra | 06:56 | |
| *** ilyashakhat has quit IRC | 06:59 | |
| *** andreas_s has joined #openstack-infra | 07:00 | |
| yolanda | good morning | 07:00 |
| *** martinkopec has joined #openstack-infra | 07:00 | |
| *** yaume has joined #openstack-infra | 07:02 | |
| *** DrifterZA has joined #openstack-infra | 07:02 | |
| *** dizquierdo has joined #openstack-infra | 07:04 | |
| *** tdasilva has quit IRC | 07:04 | |
| *** roxanaghe has joined #openstack-infra | 07:06 | |
| *** dizquierdo has quit IRC | 07:07 | |
| *** dtardivel has joined #openstack-infra | 07:07 | |
| *** dizquierdo has joined #openstack-infra | 07:07 | |
| *** roxanaghe has quit IRC | 07:10 | |
| *** thorst has joined #openstack-infra | 07:15 | |
| *** salv-orlando has quit IRC | 07:16 | |
| *** Julien-zte has quit IRC | 07:17 | |
| *** kaisers_ has joined #openstack-infra | 07:19 | |
| *** mikelk has joined #openstack-infra | 07:21 | |
| *** thorst has quit IRC | 07:21 | |
| *** jaosorior is now known as jaosorior_brb | 07:22 | |
| *** asettle has joined #openstack-infra | 07:23 | |
| *** ihrachys has joined #openstack-infra | 07:24 | |
| *** kaisers_ has quit IRC | 07:24 | |
| *** r-mibu has quit IRC | 07:27 | |
| *** r-mibu has joined #openstack-infra | 07:27 | |
| *** asettle has quit IRC | 07:28 | |
| *** Genek has joined #openstack-infra | 07:29 | |
| *** florianf has joined #openstack-infra | 07:30 | |
| *** kaisers_ has joined #openstack-infra | 07:30 | |
| *** flepied1 is now known as flepied | 07:31 | |
| *** ihrachys has quit IRC | 07:36 | |
| *** Genek has quit IRC | 07:38 | |
| *** anteaya has quit IRC | 07:38 | |
| *** gildub has quit IRC | 07:39 | |
| *** matrohon has joined #openstack-infra | 07:39 | |
| *** netsinn has joined #openstack-infra | 07:39 | |
| *** esikachev has joined #openstack-infra | 07:39 | |
| *** ifarkas_afk is now known as ifarkas | 07:41 | |
| *** tdasilva has joined #openstack-infra | 07:42 | |
| *** shashank_hegde has quit IRC | 07:42 | |
| *** netsin has quit IRC | 07:43 | |
| *** matthewbodkin has joined #openstack-infra | 07:44 | |
| *** caowei has quit IRC | 07:44 | |
| *** Genek has joined #openstack-infra | 07:45 | |
| *** yamahata has quit IRC | 07:45 | |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| *** jpich has joined #openstack-infra | 07:47 | |
| *** Julien-zte has joined #openstack-infra | 07:48 | |
| *** hashar has joined #openstack-infra | 07:53 | |
| *** hashar is now known as hasharAway | 07:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Add fuel-ccp-cinder repository https://review.openstack.org/357651 | 07:54 |
| *** hasharAway is now known as hashar | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka https://review.openstack.org/360396 | 07:54 |
| *** shashank_hegde has joined #openstack-infra | 07:54 | |
| *** salv-orlando has joined #openstack-infra | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip OSA Xenial integrated build jobs for Liberty & Mitaka https://review.openstack.org/360440 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add os_trove to OpenStack-Ansible https://review.openstack.org/360530 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config: Enable Neutron for the Tempest/Sahara tests https://review.openstack.org/360680 | 07:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add timestamp.txt to debian-openstack repo https://review.openstack.org/360750 | 07:55 |
| dtantsur | hi folks. what's " Submitted, Merge Pending" review status? it's been there for a some time already | 07:57 |
| openstackgerrit | Merged openstack-infra/system-config: Reduce job log retention to 90 days https://review.openstack.org/360551 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config: Switch all heat jobs to use devstack plugin https://review.openstack.org/317817 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add -ubuntu-trusty to gate-grenade-dsvm https://review.openstack.org/360843 | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Double rax-ord boot-timeout value" https://review.openstack.org/360837 | 07:57 |
| *** ihrachys has joined #openstack-infra | 07:58 | |
| *** ihrachys has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable rax-iad due to launch failure rate https://review.openstack.org/360842 | 08:00 |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** flepied has quit IRC | 08:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Update job skip condition for horizon doc-only changes https://review.openstack.org/360517 | 08:02 |
| *** sdake has quit IRC | 08:05 | |
| *** derekh has joined #openstack-infra | 08:05 | |
| *** sdake has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Almanach IRC channel https://review.openstack.org/360049 | 08:08 |
| AJaeger | dtantsur: have a look at your change, it failed to merge, didn't it? | 08:12 |
| dtantsur | AJaeger, true. I wonder why zuul hasn't left a comment | 08:12 |
| *** Genek has quit IRC | 08:13 | |
| AJaeger | Which change is it ? Let me check something... | 08:13 |
| *** Wei_Liu has quit IRC | 08:13 | |
| * AJaeger remembers this from yesterday | 08:13 | |
| *** ociuhandu has joined #openstack-infra | 08:13 | |
| AJaeger | dtantsur: I'm not seeing such a change in my queue - do you have a link, please? | 08:16 |
| dtantsur | AJaeger, I've rebased both already, these were https://review.openstack.org/357864 and https://review.openstack.org/360461 | 08:17 |
| *** Wei_Liu has joined #openstack-infra | 08:18 | |
| AJaeger | dtantsur: ah, ok. | 08:18 |
| *** berendt has joined #openstack-infra | 08:18 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** ociuhandu has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-infra | 08:18 | |
| *** thorst has joined #openstack-infra | 08:18 | |
| *** Na3iL has joined #openstack-infra | 08:19 | |
| AJaeger | looking at the first: We had problem yesterday around that time, might have been an infra problem that we fixed now - and tried to fix all fallout, seems this one was missed ;( | 08:19 |
| *** Ahharu has joined #openstack-infra | 08:19 | |
| AJaeger | dtantsur: so, thanks for telling - this is worth double checking. | 08:19 |
| *** dizquierdo has quit IRC | 08:20 | |
| dtantsur | np, thanks for looking into it | 08:20 |
| *** shashank_hegde has quit IRC | 08:21 | |
| *** coolsvap_ has quit IRC | 08:21 | |
| *** florianf has quit IRC | 08:25 | |
| *** thorst has quit IRC | 08:26 | |
| *** electrofelix has joined #openstack-infra | 08:27 | |
| *** lucas-dinner is now known as lucasagomes | 08:28 | |
| openstackgerrit | Alex Kavanagh proposed openstack-infra/project-config: Add charm-manila https://review.openstack.org/360762 | 08:29 |
| *** markmcd has quit IRC | 08:30 | |
| *** flepied has joined #openstack-infra | 08:31 | |
| *** flepied has quit IRC | 08:31 | |
| *** bexelbie has quit IRC | 08:32 | |
| *** bexelbie has joined #openstack-infra | 08:32 | |
| *** flepied has joined #openstack-infra | 08:33 | |
| *** sflanigan has quit IRC | 08:37 | |
| *** asettle has joined #openstack-infra | 08:38 | |
| *** asettle has quit IRC | 08:38 | |
| *** asettle has joined #openstack-infra | 08:39 | |
| *** tqtran has joined #openstack-infra | 08:39 | |
| *** florianf has joined #openstack-infra | 08:40 | |
| *** tqtran has quit IRC | 08:43 | |
| *** gongysh has quit IRC | 08:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/360985 | 08:47 |
| *** sdake has quit IRC | 08:48 | |
| *** amotoki has quit IRC | 08:54 | |
| *** roxanaghe has joined #openstack-infra | 08:54 | |
| *** tonytan4ever has joined #openstack-infra | 08:54 | |
| odyssey4me | hi everyone, now that https://review.openstack.org/360530 has merged please add me to https://review.openstack.org/#/admin/groups/1545,members | 08:56 |
| *** dizquierdo has joined #openstack-infra | 08:57 | |
| *** tonytan4ever has quit IRC | 08:58 | |
| *** roxanaghe has quit IRC | 08:59 | |
| *** jaosorior_brb is now known as jaosorior | 08:59 | |
| *** _degorenko|afk is now known as degorenko | 08:59 | |
| *** vinaypotluri has quit IRC | 09:01 | |
| *** gouthamr has quit IRC | 09:05 | |
| *** mcarden has quit IRC | 09:06 | |
| *** amitgandhinz has quit IRC | 09:06 | |
| *** jhesketh has quit IRC | 09:06 | |
| *** gus has quit IRC | 09:06 | |
| *** darrenc has quit IRC | 09:06 | |
| *** mattoliverau has quit IRC | 09:06 | |
| *** matt6434 has joined #openstack-infra | 09:06 | |
| *** darrenc has joined #openstack-infra | 09:06 | |
| *** mcarden has joined #openstack-infra | 09:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove release-notes from security-analysis https://review.openstack.org/361007 | 09:07 |
| *** gus has joined #openstack-infra | 09:08 | |
| *** jhesketh has joined #openstack-infra | 09:09 | |
| rcarrillocruz | clarkb: i just ran http://logs.openstack.org/82/349582/5/check/gate-tempest-dsvm-postgres-full-ubuntu-xenial/9409162/console.html on an xenial instance in the infracloud, took a little bit less than an hour | 09:11 |
| AJaeger | woot! | 09:16 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/360985 | 09:16 |
| *** salv-orl_ has joined #openstack-infra | 09:17 | |
| *** salv-orlando has quit IRC | 09:20 | |
| *** amitgandhinz has joined #openstack-infra | 09:20 | |
| *** pilgrimstack has quit IRC | 09:21 | |
| *** pilgrimstack has joined #openstack-infra | 09:22 | |
| *** ihrachys has joined #openstack-infra | 09:22 | |
| *** thorst has joined #openstack-infra | 09:24 | |
| *** abregman has joined #openstack-infra | 09:26 | |
| *** abregman has quit IRC | 09:27 | |
| odyssey4me | rcarrillocruz, AJaeger or yolanda: now that https://review.openstack.org/360530 has merged please add me to https://review.openstack.org/#/admin/groups/1545,members | 09:27 |
| openstackgerrit | Merged openstack-infra/storyboard: Hide timeline events and comments of private stories https://review.openstack.org/359895 | 09:27 |
| AJaeger | odyssey4me: I don't have those permissions and thus can't help | 09:28 |
| yolanda | odyssey4me, i can do it | 09:28 |
| odyssey4me | ah, AJaeger - I wasn't aware, apologies | 09:28 |
| odyssey4me | thanks yolanda | 09:28 |
| yolanda | done | 09:29 |
| *** thorst has quit IRC | 09:31 | |
| *** Julien-zte has quit IRC | 09:32 | |
| *** tkelsey has joined #openstack-infra | 09:32 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:33 | |
| *** tphummel has quit IRC | 09:35 | |
| *** matthewbodkin has quit IRC | 09:38 | |
| *** _nadya_ has joined #openstack-infra | 09:39 | |
| *** matthewbodkin has joined #openstack-infra | 09:42 | |
| *** alexpilotti has joined #openstack-infra | 09:46 | |
| *** alexpilotti has left #openstack-infra | 09:46 | |
| *** tosky has joined #openstack-infra | 09:51 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud" https://review.openstack.org/360579 | 09:51 |
| *** yamamoto has quit IRC | 09:52 | |
| *** kdas__ has joined #openstack-infra | 09:52 | |
| *** kdas__ is now known as kushal | 09:57 | |
| *** kushal has joined #openstack-infra | 09:58 | |
| *** zhurong has quit IRC | 10:01 | |
| rcarrillocruz | pabelanger: ^ put all images on infracloud | 10:02 |
| rcarrillocruz | we may want to land that to get DIB images upload rolling | 10:02 |
| *** yanyanhu has quit IRC | 10:02 | |
| *** yanyanhu has joined #openstack-infra | 10:03 | |
| rcarrillocruz | i ran a dsvm on xenial and took a little bit less than 1h, roughly the same time as OSIC per http://logs.openstack.org/82/349582/5/check/gate-tempest-dsvm-postgres-full-ubuntu-xenial/9409162/console.html | 10:03 |
| *** yanyanhu has quit IRC | 10:03 | |
| rcarrillocruz | also i believe we need a mirror on that, ping me when around since you are more familiar with afs and all than me | 10:04 |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| *** kzaitsev_mb has quit IRC | 10:19 | |
| *** matt6434 is now known as mattoliverau | 10:26 | |
| *** gongysh has joined #openstack-infra | 10:27 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Add __repr__ to a few classes https://review.openstack.org/361064 | 10:27 |
| *** hexlibris is now known as paulobanon | 10:36 | |
| paulobanon | hi all | 10:37 |
| *** Julien-zte has joined #openstack-infra | 10:39 | |
| *** roxanaghe has joined #openstack-infra | 10:42 | |
| *** rhallisey has joined #openstack-infra | 10:43 | |
| *** roxanaghe has quit IRC | 10:47 | |
| *** sdague has joined #openstack-infra | 10:49 | |
| *** rvasilets__ has joined #openstack-infra | 10:52 | |
| rvasilets__ | Hi, I've got http://logs.openstack.org/55/361055/1/check/gate-project-config-layout/dbfe0ad/console.html#_2016-08-26_10_26_18_872617 and an error msg doesn't tell me nothing( Could you help me? | 10:53 |
| rvasilets__ | Link to the patch https://review.openstack.org/#/c/361055/ | 10:53 |
| *** thorst has joined #openstack-infra | 10:56 | |
| sdague | fyi it looks like internap nodes are having cloning issues to git over ipv6, and failing a bunch of jobs | 10:57 |
| AJaeger | rvasilets__: it looks to me as if you missed to add lbaas_use_v1 at one place - double check the usage of the builder | 10:58 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Drop postgresql jobs in gate https://review.openstack.org/357444 | 10:59 |
| sdague | AJaeger: that should address your comments on that patch | 10:59 |
| AJaeger | rvasilets__: I commented - please double check | 10:59 |
| AJaeger | sdague: thanks, will review | 11:00 |
| *** yaume has quit IRC | 11:04 | |
| AJaeger | project-config cores, could you review https://review.openstack.org/361007, please? | 11:07 |
| *** gildub has joined #openstack-infra | 11:10 | |
| *** shardy is now known as shardy_lunch | 11:13 | |
| *** cody-somerville has quit IRC | 11:13 | |
| *** csomerville has joined #openstack-infra | 11:13 | |
| *** tosky has quit IRC | 11:13 | |
| *** tosky has joined #openstack-infra | 11:15 | |
| *** sdague has quit IRC | 11:17 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:18 | |
| openstackgerrit | Roman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1 https://review.openstack.org/361055 | 11:20 |
| rvasilets__ | AJaeger, yeah, Thx! | 11:20 |
| *** ramishra has quit IRC | 11:20 | |
| *** lucasagomes is now known as lucas-hungry | 11:21 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Add __repr__ to a few classes https://review.openstack.org/361064 | 11:21 |
| *** ramishra has joined #openstack-infra | 11:22 | |
| *** matbu is now known as matbu|lch | 11:22 | |
| *** DrifterZA has quit IRC | 11:25 | |
| *** DrifterZA has joined #openstack-infra | 11:26 | |
| *** DrifterZA has quit IRC | 11:26 | |
| *** salv-orl_ has quit IRC | 11:27 | |
| *** ldnunes has joined #openstack-infra | 11:27 | |
| *** kzaitsev_mb has quit IRC | 11:28 | |
| *** jkilpatr has joined #openstack-infra | 11:30 | |
| *** asettle has quit IRC | 11:32 | |
| *** yamamot__ has joined #openstack-infra | 11:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip https://review.openstack.org/360771 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 11:39 |
| *** matthewbodkin has quit IRC | 11:40 | |
| *** amotoki has joined #openstack-infra | 11:44 | |
| *** sdague has joined #openstack-infra | 11:45 | |
| openstackgerrit | Roman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1 https://review.openstack.org/361055 | 11:45 |
| *** Apoorva_ has quit IRC | 11:46 | |
| *** Apoorva has joined #openstack-infra | 11:47 | |
| *** senk has quit IRC | 11:47 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute003.vanilla to static Ansible inventory https://review.openstack.org/361113 | 11:49 |
| *** kzaitsev_mb has joined #openstack-infra | 11:50 | |
| sdague | mordred: you able to bring us home on this - https://review.openstack.org/#/c/357444/ ? | 11:51 |
| *** Apoorva has quit IRC | 11:51 | |
| sdague | rcarrillocruz: also, internap is acting up talking to git - http://logs.openstack.org/51/359951/2/gate/gate-nodepool-docs-ubuntu-xenial/1dc9f46/console.html#_2016-08-26_11_50_09_709037 | 11:52 |
| dtantsur | sigh, is it possible to fix gerrit to not log everyone out every day? I've just lost a huge comment because of the terrible way the whole UI is implemented... | 11:52 |
| * dtantsur remembers to write comments in a text editor first, then copy-paste them to gerrit | 11:53 | |
| mordred | sdague: damn. rcarrillocruz beat me to it | 11:53 |
| *** jaosorior has quit IRC | 11:53 | |
| openstackgerrit | Merged openstack-infra/nodepool: Remove unecessecary builder scheduler thread https://review.openstack.org/359949 | 11:54 |
| sdague | mordred: you can still pile on for posterity | 11:54 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove 'running' as a public method from builder https://review.openstack.org/359950 | 11:54 |
| *** pgadiya has quit IRC | 11:54 | |
| *** jaosorior has joined #openstack-infra | 11:54 | |
| rcarrillocruz | dtantsur: iirc, gerrit was tweaked some time ago to last sessions a week http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/templates/gerrit.config.erb | 11:55 |
| rcarrillocruz | however, gerrit was restarted yesterday, which is why your session might have died | 11:55 |
| *** tonytan4ever has joined #openstack-infra | 11:55 | |
| dtantsur | hmm, maybe? thanks, I'll keep watching it. | 11:56 |
| *** matbu|lch is now known as matbu | 11:56 | |
| *** gildub has quit IRC | 11:57 | |
| *** jamesdenton has quit IRC | 11:57 | |
| mordred | sdague: :) | 11:57 |
| *** tonytan4ever has quit IRC | 12:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Drop postgresql jobs in gate https://review.openstack.org/357444 | 12:01 |
| mordred | \o/ | 12:01 |
| * mordred hands sdague a pie | 12:01 | |
| AJaeger | mordred: should I send some review requests your way so that you can catch up? | 12:02 |
| AJaeger | https://review.openstack.org/#/dashboard/?foreach=%2528project%253Aopenstack%252Dinfra%252Fproject%252Dconfig%2529+status%253Aopen+NOT+label%253AWorkflow%253C%253D%252D1+label%253AVerified%253E%253D1%252Cjenkins+is%253Amergeable&title=Monty+Review+Inbox&Needs+final+%252B2=NOT+label%253ACode%252DReview%253C%253D%252D1%252Cproject%252Dconfig%252Dcore+label%253ACode%252DReview%253E%253D2 | 12:02 |
| *** sdague has quit IRC | 12:02 | |
| *** ansmith has joined #openstack-infra | 12:02 | |
| mordred | AJaeger: wow. you are very good at gerrit dashboards | 12:02 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Fix baremetal.json IP params for compute00(5|6) https://review.openstack.org/361126 | 12:03 |
| *** baoli has joined #openstack-infra | 12:03 | |
| AJaeger | mordred: just copied a link I had saved - and quickly edited the title ;) | 12:03 |
| *** Goneri has joined #openstack-infra | 12:04 | |
| *** baoli has quit IRC | 12:07 | |
| *** sdague has joined #openstack-infra | 12:08 | |
| *** tonytan4ever has joined #openstack-infra | 12:09 | |
| mordred | :) | 12:10 |
| *** asettle has joined #openstack-infra | 12:10 | |
| *** baoli has joined #openstack-infra | 12:11 | |
| *** kushal has quit IRC | 12:11 | |
| *** shardy_lunch is now known as shardy | 12:11 | |
| *** nwkarsten has joined #openstack-infra | 12:12 | |
| nibalizer | "monty review inbox" lololol | 12:12 |
| *** matthewbodkin has joined #openstack-infra | 12:15 | |
| *** senk has joined #openstack-infra | 12:16 | |
| *** xyang1 has joined #openstack-infra | 12:17 | |
| sdague | mordred et al (devstack-gate reviewers) - https://review.openstack.org/#/c/360565/ | 12:17 |
| openstackgerrit | Merged openstack-infra/project-config: Remove release-notes from security-analysis https://review.openstack.org/361007 | 12:18 |
| sdague | grenade no longer in horizon gate (it didn't do anything but install), so we can not install it in grenade jobs to save time | 12:18 |
| sdague | and any rando install errors that might exist | 12:18 |
| sdague | and there is a cleanup to remove some juno level testing after that | 12:19 |
| openstackgerrit | Merged openstack-infra/project-config: Add a tempest job for python-ironic-inspector-client https://review.openstack.org/358663 | 12:19 |
| openstackgerrit | Merged openstack-infra/project-config: Don't use devuser for zuul-worker https://review.openstack.org/333653 | 12:19 |
| *** baoli_ has joined #openstack-infra | 12:20 | |
| *** dprince has joined #openstack-infra | 12:23 | |
| *** snarwade is now known as snarwade-afk | 12:23 | |
| *** baoli has quit IRC | 12:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix baremetal.json IP params for compute00(5|6) https://review.openstack.org/361126 | 12:23 |
| *** kaisers1 has quit IRC | 12:23 | |
| *** kaisers has quit IRC | 12:23 | |
| *** chem|off` has quit IRC | 12:25 | |
| *** lucas-hungry is now known as lucasagomes | 12:25 | |
| *** chem|off` has joined #openstack-infra | 12:26 | |
| *** gordc has joined #openstack-infra | 12:26 | |
| *** ansmith has quit IRC | 12:27 | |
| *** pradk has joined #openstack-infra | 12:29 | |
| *** kaisers has joined #openstack-infra | 12:29 | |
| *** kaisers1 has joined #openstack-infra | 12:30 | |
| *** roxanaghe has joined #openstack-infra | 12:30 | |
| *** martinkopec has quit IRC | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** dizquierdo has quit IRC | 12:32 | |
| *** martinkopec has joined #openstack-infra | 12:33 | |
| *** woodster_ has joined #openstack-infra | 12:34 | |
| *** roxanaghe has quit IRC | 12:35 | |
| *** jed56 has joined #openstack-infra | 12:40 | |
| *** tqtran has joined #openstack-infra | 12:41 | |
| *** senk has quit IRC | 12:41 | |
| *** markusry has joined #openstack-infra | 12:43 | |
| sdague | is anyone looking at internap failing the world? | 12:43 |
| *** slashme has joined #openstack-infra | 12:44 | |
| sdague | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22errno%3DNetwork%20is%20unreachable%5C%22 | 12:44 |
| sdague | internap looks like it is failing everything on connectivity issue | 12:45 |
| *** tqtran has quit IRC | 12:45 | |
| rcarrillocruz | sdague: doesn't seem to be consistenly failing | 12:47 |
| rcarrillocruz | http://paste.openstack.org/show/563917/ | 12:47 |
| rcarrillocruz | at least i could clone and ping ipv4 fine | 12:47 |
| sdague | what is internap-mtl01 ? | 12:47 |
| sdague | all the fails are there | 12:47 |
| rcarrillocruz | sec, i kind of remember changes about it from yesterday or something | 12:48 |
| mat128 | sdague: can you show me a log? | 12:48 |
| sdague | rcarrillocruz: well, there are 249 fail events | 12:48 |
| sdague | http://logs.openstack.org/37/356137/9/gate/gate-nova-docs-ubuntu-xenial/d8e2b15/console.html | 12:48 |
| *** Apoorva has joined #openstack-infra | 12:48 | |
| sdague | mat128: also see the logstash query above | 12:48 |
| rcarrillocruz | yeah | 12:48 |
| sdague | I've also seen 3 gate patches fail on this today | 12:48 |
| rcarrillocruz | it's a new region | 12:48 |
| rcarrillocruz | 99526dee00ac7d6bc79413f11db1ad4acea174a3 from system-config shows | 12:48 |
| mat128 | yes | 12:49 |
| rcarrillocruz | Add mtl01 region to internap on puppetmaster | 12:49 |
| sdague | just by glancing at the zuul page | 12:49 |
| *** hashar has quit IRC | 12:49 | |
| mat128 | rcarrillocruz: I also work there | 12:49 |
| rcarrillocruz | ah good | 12:49 |
| mat128 | trying to get actual ops people involved | 12:49 |
| rcarrillocruz | thx! | 12:49 |
| sdague | my suggestion, disable that region until it can be debugged manually | 12:49 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Update job skip condition for neutron doc-only changes https://review.openstack.org/361151 | 12:49 |
| mat128 | sdague: I kind of agree... | 12:50 |
| *** amotoki has quit IRC | 12:50 | |
| rcarrillocruz | sdague: i'll push up a change | 12:50 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: This has caused 240+ failures in the last 24 hours https://review.openstack.org/361152 | 12:50 |
| sdague | rcarrillocruz: actually there it is | 12:50 |
| sdague | yeh, the nynj region seems fine, so it's isolated to one region at least, which is good | 12:51 |
| sdague | ok, got to drop off for ops summit. Will check in later | 12:51 |
| mat128 | sdague: you can always drop a word to mgagne about it | 12:52 |
| mordred | +A | 12:52 |
| mat128 | he's there | 12:52 |
| *** Apoorva has quit IRC | 12:53 | |
| *** rtheis has joined #openstack-infra | 12:53 | |
| rtheis | Is it possible to force a gate job to skip IPv6-only hosts? | 12:54 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Disable Internap mtl01 https://review.openstack.org/361156 | 12:56 |
| *** sdague has quit IRC | 12:56 | |
| rcarrillocruz | mordred: ^ | 12:56 |
| rcarrillocruz | sdague seems to have left | 12:56 |
| *** amotoki has joined #openstack-infra | 12:56 | |
| mat128 | rcarrillocruz: this is exactly what sdague submitted | 12:57 |
| mat128 | https://review.openstack.org/361152 | 12:57 |
| rcarrillocruz | oh | 12:57 |
| *** alaski is now known as lascii | 12:57 | |
| *** matt-borland has joined #openstack-infra | 12:57 | |
| rcarrillocruz | sorry, was in a console and hadn't checked the scrollback | 12:57 |
| rcarrillocruz | abandoning mine, thx | 12:57 |
| AJaeger | rcarrillocruz: thanks for submitting - no need to be sorry. | 12:58 |
| * rcarrillocruz goes back to baremetal provisioning and infracloud things | 12:58 | |
| mordred | rtheis: it is not | 12:59 |
| *** sdague has joined #openstack-infra | 12:59 | |
| rtheis | mordred: okay, thank you | 12:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Log request ids https://review.openstack.org/361157 | 12:59 |
| *** smarcet has joined #openstack-infra | 13:01 | |
| openstackgerrit | Yuriy Taraday proposed openstack-infra/project-config: Enable puppet checks for fuel-octane https://review.openstack.org/355265 | 13:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Log request ids https://review.openstack.org/361157 | 13:02 |
| *** amotoki has quit IRC | 13:02 | |
| *** hashar has joined #openstack-infra | 13:03 | |
| *** salv-orlando has joined #openstack-infra | 13:03 | |
| *** eharney has joined #openstack-infra | 13:04 | |
| *** zhurong has joined #openstack-infra | 13:04 | |
| *** gongysh has quit IRC | 13:04 | |
| pabelanger | morning | 13:05 |
| mordred | morning pabelanger ! | 13:06 |
| mordred | pabelanger: we just disabled mtl | 13:06 |
| rcarrillocruz | pabelanger: heya, yeah that, ^ | 13:06 |
| rcarrillocruz | since you pushed changes for it yesterday | 13:06 |
| mordred | pabelanger: because jobs running there are having connectivity problems talking to git.o.o | 13:06 |
| rcarrillocruz | mat128 is looking into it | 13:06 |
| rcarrillocruz | btw, i +2 your changes for the region name | 13:07 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add tracking artifact for network connectivity issues https://review.openstack.org/361162 | 13:07 |
| mat128 | trying to see how/why this is happening | 13:07 |
| rcarrillocruz | i can't remember if i made you core on the launcher project? | 13:07 |
| *** zul has joined #openstack-infra | 13:07 | |
| cloudnull | mornings | 13:07 |
| mat128 | I will provide an update before noon EDT | 13:07 |
| pabelanger | mordred: thanks, catching up on backscroll no | 13:08 |
| pabelanger | now* | 13:08 |
| openstackgerrit | Merged openstack-infra/project-config: This has caused 240+ failures in the last 24 hours https://review.openstack.org/361152 | 13:10 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /var/log from all nodes https://review.openstack.org/361164 | 13:11 |
| *** esberglu has joined #openstack-infra | 13:14 | |
| *** sdake has joined #openstack-infra | 13:14 | |
| *** sdake_ has joined #openstack-infra | 13:15 | |
| pabelanger | mat128: we are also seeing a lot these errors: | 13:15 |
| pabelanger | OpenStackCloudException: Error in creating instance (Inner Exception: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a2af3a7c-5a1f-4896-a766-becdc9333a21)) | 13:15 |
| pabelanger | a long with SSH Exceptions after we've launched a node and try to SSH into the assigned IP | 13:16 |
| *** dtantsur is now known as dtantsur|brb | 13:16 | |
| *** sdake has quit IRC | 13:19 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add tracking artifact for network connectivity issues https://review.openstack.org/361162 | 13:21 |
| openstackgerrit | Merged openstack-infra/nodepool: Simplify builder start/stop methods https://review.openstack.org/359951 | 13:23 |
| *** kushal has joined #openstack-infra | 13:24 | |
| *** abregman has joined #openstack-infra | 13:26 | |
| *** kgiusti has joined #openstack-infra | 13:27 | |
| *** anteaya has joined #openstack-infra | 13:29 | |
| *** amotoki has joined #openstack-infra | 13:31 | |
| *** roxanaghe has joined #openstack-infra | 13:31 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add baremetal00 to cacti hosts https://review.openstack.org/361181 | 13:33 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove hpuswest hosts from Cacti https://review.openstack.org/361183 | 13:34 |
| paulobanon | can anyone let me know if you guys are also responsible for python-jenkins or if its supposed to be discussed in #openstack-dev? i have a small doubt regarding a bug i'm seeing with jenkins job builder | 13:35 |
| pabelanger | rcarrillocruz: I need to step out to run an errand with family, but I can help with AFS mirror for omfracloud when I get back | 13:35 |
| pabelanger | rcarrillocruz: we'd use the openstackci project to launch it | 13:36 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 13:36 |
| *** roxanaghe has quit IRC | 13:36 | |
| rcarrillocruz | pabelanger: ++ | 13:36 |
| rcarrillocruz | so it seems RAX going private deal has gone thru | 13:37 |
| *** senk has joined #openstack-infra | 13:37 | |
| *** sdake_ has quit IRC | 13:37 | |
| openstackgerrit | Merged openstack-infra/system-config: Add compute003.vanilla to static Ansible inventory https://review.openstack.org/361113 | 13:37 |
| *** amotoki has quit IRC | 13:38 | |
| *** sdake has joined #openstack-infra | 13:40 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring https://review.openstack.org/361185 | 13:40 |
| *** salv-orlando has quit IRC | 13:40 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring https://review.openstack.org/361185 | 13:41 |
| *** berendt has quit IRC | 13:44 | |
| *** links has quit IRC | 13:45 | |
| openstackgerrit | Merged openstack-infra/system-config: Add internap-mtl01 to cacti.o.o https://review.openstack.org/360832 | 13:45 |
| *** pt_15 has joined #openstack-infra | 13:46 | |
| openstackgerrit | Roman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1 https://review.openstack.org/361055 | 13:46 |
| *** kushal has quit IRC | 13:46 | |
| yolanda | mordred, do you know if there are still issues with new python-openstackclient? with latest version i get Could not determine a suitable URL for the plugin error | 13:46 |
| mgagne | pabelanger: I'm online now, although a bit distracted by ops midcycle | 13:47 |
| mordred | yolanda: can you run with --debug and paste me the output? | 13:47 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add controller00.vanilla to Cacti https://review.openstack.org/361192 | 13:47 |
| anteaya | mgagne: how is that going? | 13:47 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 13:47 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Add c-vol,c-bak on subnode when c-api enabled https://review.openstack.org/352909 | 13:47 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic https://review.openstack.org/335981 | 13:47 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 13:47 |
| mgagne | anteaya: I just arrived, slept a bit more this morning. logging and monitoring about to start =) | 13:47 |
| *** gouthamr has joined #openstack-infra | 13:47 | |
| anteaya | mgagne: cool, I hope you have a good day | 13:48 |
| mordred | yolanda: (cause it works for me - I just installed into a new venv and everything is happy) | 13:48 |
| rcarrillocruz | clarkb, fungi : are we cool to land https://review.openstack.org/#/c/360579/ , re: reenabling infracloud, with max-servers to -1 ? | 13:48 |
| rcarrillocruz | so we can get the ball rolling for image builds | 13:48 |
| rcarrillocruz | (morning!) | 13:48 |
| *** Apoorva has joined #openstack-infra | 13:49 | |
| yolanda | mordred http://paste.openstack.org/show/563931/ | 13:50 |
| *** sdague has quit IRC | 13:51 | |
| yolanda | ah, have said nothing, 3.1.0 works | 13:51 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti https://review.openstack.org/361195 | 13:51 |
| mordred | yolanda: oh good | 13:51 |
| yolanda | i was not using latest latest... | 13:52 |
| *** ddieterly has joined #openstack-infra | 13:52 | |
| mordred | yah. there were a couple of rocky moments recently | 13:53 |
| *** sandanar has quit IRC | 13:53 | |
| *** amotoki has joined #openstack-infra | 13:53 | |
| *** Apoorva has quit IRC | 13:53 | |
| *** ddieterly has quit IRC | 13:54 | |
| *** ddieterly has joined #openstack-infra | 13:55 | |
| *** Marx314 has joined #openstack-infra | 13:56 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti https://review.openstack.org/361224 | 13:58 |
| *** xarses has quit IRC | 13:58 | |
| *** thiagop has joined #openstack-infra | 13:59 | |
| *** fguillot has joined #openstack-infra | 14:00 | |
| *** brucet has joined #openstack-infra | 14:02 | |
| mgagne | pabelanger: found those errors in nova-conductor logs: RescheduledException: Build of instance 1583a93f-1b2d-489d-9d28-8efc78f94455 was re-scheduled: Not authorized for image 2124d3ce-891b-412a-a951-309f79c008ec. | 14:03 |
| *** mdrabe has joined #openstack-infra | 14:05 | |
| *** bin_ has joined #openstack-infra | 14:06 | |
| *** zz_dimtruck is now known as dimtruck | 14:06 | |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| *** yamamot__ has quit IRC | 14:08 | |
| mordred | oh. that's a new one! | 14:08 |
| *** thiagolib has quit IRC | 14:08 | |
| *** senk has quit IRC | 14:09 | |
| *** Apoorva has joined #openstack-infra | 14:10 | |
| *** sdague has joined #openstack-infra | 14:11 | |
| openstackgerrit | Merged openstack-infra/system-config: Make Launchpad credentials reusable https://review.openstack.org/356665 | 14:13 |
| *** dtantsur|brb is now known as dtantsur | 14:15 | |
| *** dimtruck is now known as zz_dimtruck | 14:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Add LP creds and launchpadlib to signing job node https://review.openstack.org/356666 | 14:17 |
| *** mtanino has joined #openstack-infra | 14:17 | |
| mgagne | pabelanger: I see that some instances were created successfully, did the issue solved by itself? | 14:18 |
| *** csomerville has quit IRC | 14:20 | |
| *** csomerville has joined #openstack-infra | 14:21 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring https://review.openstack.org/361185 | 14:21 |
| *** ifarkas is now known as ifarkas_afk | 14:21 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Stop mirroring puppet modules https://review.openstack.org/361241 | 14:23 |
| *** _nadya_ has quit IRC | 14:23 | |
| *** zz_dimtruck is now known as dimtruck | 14:24 | |
| *** Guest71645 is now known as dansmith | 14:25 | |
| *** ogelbukh has quit IRC | 14:25 | |
| *** jaranovich has quit IRC | 14:25 | |
| *** Ng has quit IRC | 14:25 | |
| *** vdrok has quit IRC | 14:25 | |
| *** ashtokolov has quit IRC | 14:25 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Start using the image cache again https://review.openstack.org/361245 | 14:25 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Tweak 'about' page https://review.openstack.org/361246 | 14:25 |
| *** auggy has quit IRC | 14:26 | |
| *** monester_ has quit IRC | 14:26 | |
| *** iremizov has quit IRC | 14:26 | |
| *** ddieterly has quit IRC | 14:26 | |
| *** iremizov has joined #openstack-infra | 14:26 | |
| *** auggy has joined #openstack-infra | 14:26 | |
| *** inc0 has joined #openstack-infra | 14:26 | |
| *** monester_ has joined #openstack-infra | 14:27 | |
| *** szaher has quit IRC | 14:28 | |
| *** ogelbukh has joined #openstack-infra | 14:28 | |
| *** jaranovich has joined #openstack-infra | 14:28 | |
| *** Ng has joined #openstack-infra | 14:28 | |
| *** vdrok has joined #openstack-infra | 14:28 | |
| *** ashtokolov has joined #openstack-infra | 14:28 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Add initialize-urandom to fedora 24 https://review.openstack.org/356818 | 14:29 |
| fungi | rcarrillocruz: the clouds.yaml on nodepool.o.o still has the correct credentials for 360579 to work i guess? we didn't change any of that? | 14:29 |
| *** szaher has joined #openstack-infra | 14:29 | |
| rcarrillocruz | still work, i uploaded a trusty and xenial image manually from nodepool to infracloud | 14:30 |
| rcarrillocruz | so creds are good | 14:30 |
| *** dprince has quit IRC | 14:30 | |
| *** yamamoto has joined #openstack-infra | 14:30 | |
| zaro | morning | 14:30 |
| fungi | thanks rcarrillocruz! | 14:31 |
| *** dprince has joined #openstack-infra | 14:31 | |
| zaro | paulobanon: yes, python-jenkins responsibility lies here. | 14:31 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 14:32 |
| *** roxanaghe has joined #openstack-infra | 14:32 | |
| *** chlong has joined #openstack-infra | 14:32 | |
| *** kaisers_ has quit IRC | 14:32 | |
| fungi | clarkb: jeblair: mordred: zaro: looks like our http threads change worked... show-caches --show-threads indicates we have a total of 29 httpd threads in gerrit right now, which is more than the default maximum | 14:32 |
| rcarrillocruz | thx | 14:33 |
| YorikSar | AJaeger: Thanks for swift review! | 14:34 |
| zaro | fungi: cool, we still going ahead with those apache tweeks? or wait to see how gerrit responds first? | 14:34 |
| YorikSar | Anyone else around to merge https://review.openstack.org/355265 ? | 14:34 |
| fungi | zaro: i think we wait to see how gerrit and apache behave now | 14:34 |
| zaro | ianw: was wondering whether new review-dev was ready yet? | 14:34 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update undercloud flavor stats https://review.openstack.org/360756 | 14:35 |
| mordred | fungi: woot | 14:35 |
| fungi | zaro: ianw e-mailed the infra ml saying it's up now | 14:35 |
| zaro | ahh cool, must have missed it. Thanks ianw ! | 14:36 |
| fungi | zaro: lots of ram now! http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=7&rra_id=all | 14:36 |
| *** roxanaghe has quit IRC | 14:36 | |
| rcarrillocruz | \o/ | 14:36 |
| *** raunak has joined #openstack-infra | 14:38 | |
| *** ddieterly has joined #openstack-infra | 14:39 | |
| *** jheroux has joined #openstack-infra | 14:40 | |
| *** pblaho has quit IRC | 14:40 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Remove 'partial' support for Grenade jobs https://review.openstack.org/329532 | 14:41 |
| *** jianghuaw-m has joined #openstack-infra | 14:42 | |
| mat128 | sdague, pabelanger: mgagne will look at the Internap mtl01 issues (connection timed out and HTTP 500) | 14:42 |
| *** zul has quit IRC | 14:42 | |
| *** amitgandhinz has quit IRC | 14:43 | |
| *** yamahata has joined #openstack-infra | 14:43 | |
| *** amitgandhinz has joined #openstack-infra | 14:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Remove infracloud" https://review.openstack.org/360579 | 14:44 |
| openstackgerrit | Merged openstack-infra/project-config: Enable puppet checks for fuel-octane https://review.openstack.org/355265 | 14:44 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute011.vanilla to static ansible inventory https://review.openstack.org/361261 | 14:44 |
| *** sdague has quit IRC | 14:45 | |
| *** yamahata has quit IRC | 14:45 | |
| *** yamahata has joined #openstack-infra | 14:45 | |
| AJaeger | infra root, could you review these changes of mine, please? Documentation for bindep: https://review.openstack.org/#/c/358811/ and https://review.openstack.org/#/c/358998/ ; and a tiny requirements update for git-review: https://review.openstack.org/#/c/270410/ | 14:45 |
| *** vinaypotluri has joined #openstack-infra | 14:47 | |
| *** dizquierdo has joined #openstack-infra | 14:48 | |
| *** notnownikki has joined #openstack-infra | 14:48 | |
| *** raunak has quit IRC | 14:48 | |
| *** jianghuaw-m has quit IRC | 14:49 | |
| *** bauzas is now known as bauwser | 14:50 | |
| *** sdague has joined #openstack-infra | 14:50 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: add in missing gnocchi 2.2 job references https://review.openstack.org/361268 | 14:50 |
| *** csomerville has quit IRC | 14:51 | |
| *** sarob has joined #openstack-infra | 14:51 | |
| *** csomerville has joined #openstack-infra | 14:51 | |
| *** ddieterly is now known as ddieterly[away] | 14:51 | |
| *** xarses has joined #openstack-infra | 14:52 | |
| *** sskripnick has joined #openstack-infra | 14:53 | |
| *** coolsvap_ has joined #openstack-infra | 14:53 | |
| sskripnick | hi all. who can help with pbr? | 14:53 |
| sskripnick | fungi: AJaeger: clarkb: ^ | 14:54 |
| fungi | sskripnick: probably nobody unless you tell us what about pbr you need help with ;) | 14:55 |
| *** reed_ has joined #openstack-infra | 14:55 | |
| sskripnick | I just need to build a package which works >_< | 14:55 |
| *** niska has quit IRC | 14:55 | |
| *** martinkopec has quit IRC | 14:55 | |
| sskripnick | Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed? | 14:55 |
| *** sarob has quit IRC | 14:56 | |
| sskripnick | I've used PBR_VERSION=0.0.3, and still the same | 14:56 |
| *** HenryG has quit IRC | 14:56 | |
| sskripnick | How do I hardcode a version? | 14:56 |
| fungi | sskripnick: can you explain what source code you're working with? a non-sdist tarball from somewhere maybe? | 14:56 |
| *** kaisers1 has quit IRC | 14:56 | |
| *** andreas_s has quit IRC | 14:56 | |
| *** mjblack has quit IRC | 14:56 | |
| *** reed_ has quit IRC | 14:57 | |
| sskripnick | fungi: https://review.openstack.org/#/c/337633/ | 14:57 |
| openstackgerrit | Merged openstack-infra/nodepool: Rename BuilderScheduler to NodePoolBuilder https://review.openstack.org/359952 | 14:57 |
| sskripnick | fungi: I tried to use git tag, PBR_VERSION and hardcode version into setup.cfg | 14:58 |
| sskripnick | all the same | 14:58 |
| *** kaisers1 has joined #openstack-infra | 14:58 | |
| fungi | i'm pulling that source now to see if i can spot the issue | 14:58 |
| AJaeger | sskripnick: you don't have a setup.py file | 14:58 |
| mordred | rcarrillocruz, Shrews, clarkb: this one is ready to go: https://review.openstack.org/#/c/360771 | 14:59 |
| sskripnick | AJaeger: I have this patch in master already | 14:59 |
| sskripnick | this file* | 14:59 |
| mordred | fungi, sskripnick: also looking | 15:00 |
| *** ddieterly[away] is now known as ddieterly | 15:00 | |
| *** Apoorva has quit IRC | 15:00 | |
| *** tonytan4ever has quit IRC | 15:01 | |
| mordred | sskripnick: this repo (and that change) work fine for me - what command are you trying to run? | 15:01 |
| *** Apoorva has joined #openstack-infra | 15:01 | |
| *** jistr is now known as jistr|call | 15:01 | |
| *** tonytan4ever has joined #openstack-infra | 15:01 | |
| *** mjblack has joined #openstack-infra | 15:01 | |
| fungi | i'm trying building an sdist under tox with that change applied now | 15:01 |
| *** HenryG has joined #openstack-infra | 15:01 | |
| fungi | which seems to work | 15:02 |
| mordred | I have built an sdist outside of tox too, which also works | 15:02 |
| fungi | i get openstack_app_catalog-0.0.1.dev379.tar.gz | 15:02 |
| *** stew925 has quit IRC | 15:02 | |
| *** wcriswell has quit IRC | 15:02 | |
| *** skipp has quit IRC | 15:02 | |
| mordred | yup | 15:02 |
| mordred | me too | 15:02 |
| *** senk has joined #openstack-infra | 15:02 | |
| *** mikelk has quit IRC | 15:02 | |
| *** zhurong has quit IRC | 15:03 | |
| AJaeger | sskripnick: is it failing in OpenStack CI for you? | 15:03 |
| fungi | however that pep8 job isn't failing on sdist/packaging issues. it's failing because of bad code | 15:03 |
| *** Apoorva has quit IRC | 15:03 | |
| *** hockeynut has joined #openstack-infra | 15:03 | |
| mordred | yah | 15:03 |
| fungi | well, not "bad" code, but non-conformant code based on the rules it's configured to enforce for that project | 15:03 |
| mordred | btw - python setup.py sdist works just fine without that change too | 15:04 |
| *** amitgandhinz has quit IRC | 15:04 | |
| *** csomerville has quit IRC | 15:04 | |
| fungi | i'm also able to install openstack_app_catalog-0.0.1.dev379.tar.gz into a fresh virtualenv just fine | 15:04 |
| *** amitgandhinz has joined #openstack-infra | 15:04 | |
| *** rcernin has quit IRC | 15:04 | |
| mordred | fungi: oslo-incubator isn't a thing anymore, right? | 15:05 |
| fungi | mordred: correct | 15:05 |
| *** hongbin has joined #openstack-infra | 15:05 | |
| pleia2 | morning | 15:05 |
| AJaeger | morning, pleia2 | 15:05 |
| fungi | mordred: in fact, there's a tc resolution to have a cross-project goal ripping out the remaining old oslo-incubator code copies | 15:05 |
| *** ddieterly has quit IRC | 15:05 | |
| fungi | everywhere | 15:05 |
| *** rbrndt has quit IRC | 15:06 | |
| mordred | neat | 15:06 |
| sskripnick | fungi: AJaeger: I just trying to build and upload a package by hand. Like "python setup.py sdist upload" then I install this package on another VM "pip install openstack_app_catalog" and trying to run app-catalog-manage runserver | 15:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Give some examples https://review.openstack.org/358811 | 15:06 |
| AJaeger | fungi: fixed the typo, thanks for review ^ | 15:07 |
| *** dprince has quit IRC | 15:07 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.vanilla to static Ansible inventory https://review.openstack.org/361285 | 15:07 |
| *** dprince has joined #openstack-infra | 15:07 | |
| *** thcipriani|afk is now known as thcipriani | 15:08 | |
| rcarrillocruz | any objections if I self-approve myself additions of servers to ansible inventory and cacti ? | 15:08 |
| rcarrillocruz | i'm pushing changes as I provision them fine | 15:08 |
| *** dprince has quit IRC | 15:08 | |
| *** signed8bit has joined #openstack-infra | 15:08 | |
| mordred | rcarrillocruz: nope. do it | 15:08 |
| *** niska has joined #openstack-infra | 15:08 | |
| *** dprince has joined #openstack-infra | 15:08 | |
| jeblair | rcarrillocruz: wfm | 15:08 |
| rcarrillocruz | thx | 15:08 |
| mordred | rcarrillocruz: I have no way of reviewing their correctness anyway :) | 15:08 |
| rcarrillocruz | heh | 15:09 |
| jeblair | rcarrillocruz: are they the same in cacti, or do i need to clean anything up there? | 15:09 |
| fungi | sskripnick: probably best to start with a complete copy of the traceback pasted to paste.openstack.org | 15:09 |
| jeblair | (as in, same hostnames?) | 15:09 |
| rcarrillocruz | jeblair: i have https://review.openstack.org/#/c/361183/ that removes hpuswest naming | 15:09 |
| rcarrillocruz | then will approve the new vanilla/chocolate changes | 15:10 |
| rcarrillocruz | ah wait | 15:10 |
| rcarrillocruz | you mean cleaning as in the UI | 15:10 |
| jeblair | rcarrillocruz: cool -- are the historic data for those useful, or should i just delete them when that lands? | 15:10 |
| rcarrillocruz | ofc, i assume just by removing from hiera doesn't clean them from cacti DB | 15:10 |
| jeblair | rcarrillocruz: right, it does not | 15:10 |
| rcarrillocruz | i don't think that's useful no | 15:10 |
| mordred | sskripnick: I have reproduced your issue | 15:10 |
| rcarrillocruz | thx | 15:10 |
| fungi | sskripnick: like, is it the "import pbr.version" line in openstack_catalog/__init__.py which triggering this at runtime? you don't get any errors when you build the sdist or when you install it with pip? | 15:10 |
| jeblair | rcarrillocruz: (which is partly on purpose, because with some servers, we do want the historic data even after we delete) | 15:11 |
| jeblair | rcarrillocruz: cool, i will delete those after 183 lantds | 15:11 |
| rcarrillocruz | makes sense | 15:11 |
| *** sdake_ has joined #openstack-infra | 15:11 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Disable selinux on RHEL https://review.openstack.org/361290 | 15:11 |
| *** sarob has joined #openstack-infra | 15:11 | |
| mordred | fungi: yah | 15:11 |
| mordred | so - I know what the problem is | 15:11 |
| rcarrillocruz | i should find the t-shirt that had a message about 'i fixed it by disabling selinux' or something like that | 15:12 |
| rcarrillocruz | i believe pleia2 had it? | 15:12 |
| rcarrillocruz | re: yolanda change ^ | 15:12 |
| rcarrillocruz | :D | 15:12 |
| *** jistr|call is now known as jistr | 15:12 | |
| pleia2 | oh yes, the one from http://stopdisablingselinux.com/ :) | 15:12 |
| rcarrillocruz | haha | 15:12 |
| yolanda | always is selinux... | 15:12 |
| rcarrillocruz | that's awesome | 15:12 |
| mordred | sskripnick: mind if I just amend your commit? | 15:12 |
| mordred | or - how about I do this ... | 15:12 |
| jeblair | mordred: when you have a sec, can you explain how the code in 360771 achieves what is described by "cloud has a network from which floating ips come that is connected via a router to the network from which the private_v4 address came" ? (i'm assuming it has to do with _get_free_fixed_port, but that looks too simple to do all of that, so i must be missing some subtlety) | 15:13 |
| fungi | pleia2: wow, that's some selinux advocacy site for sure | 15:13 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add swap usage to dstat capture https://review.openstack.org/359515 | 15:13 |
| mordred | sskripnick: https://review.openstack.org/361292 | 15:13 |
| pleia2 | fungi: I'm a fan | 15:13 |
| mordred | sskripnick: feel free to squash that into your change and abandon my change | 15:13 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove white space between print and () https://review.openstack.org/341482 | 15:13 |
| mtreinish | fungi: heh, I like our cpu usage on firehose: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4917&rra_id=all | 15:13 |
| mordred | sskripnick: but I thought showing the solutoin would be useful | 15:13 |
| *** skipp has joined #openstack-infra | 15:13 | |
| *** rbrndt has joined #openstack-infra | 15:13 | |
| mordred | fungi: ^^ | 15:14 |
| jeblair | pleia2, fungi: oh the first set of t-shirts is so current and topical too! :) | 15:14 |
| * fungi always applies tee-shitrs topically | 15:14 | |
| yolanda | also... i got a working infra-cloud in CentOS for OPNFV | 15:14 |
| mordred | jeblair: I have a sec right now! you are right, _get_free_fixed_port does the trick | 15:14 |
| *** sdake has quit IRC | 15:15 | |
| sskripnick | mordred: it would be great | 15:15 |
| notnownikki | oh boy, i can't even describe how much i'd like to replace our jenkins with zuul right now | 15:15 |
| mordred | jeblair: _get_free_fixed_port has calls to find "nat_network" | 15:15 |
| sskripnick | fungi: there is no errors when building/uploading/installing. only error when trying to run | 15:15 |
| fungi | mtreinish: i like green | 15:15 |
| jeblair | mtreinish: did you get that thing you wanted added to zuul? | 15:15 |
| *** sarob has quit IRC | 15:16 | |
| fungi | sskripnick: looks like mordred has proposed a fix for you | 15:16 |
| *** mikelk has joined #openstack-infra | 15:16 | |
| pleia2 | jeblair: I just have the "setenforce 1" one ;) | 15:16 |
| rcarrillocruz | yolanda: that's awesome work! | 15:16 |
| rcarrillocruz | pabelanger: you might like that ^ :-) | 15:16 |
| *** kdas__ has joined #openstack-infra | 15:16 | |
| stevemar | silly question, all the ATC codes have already been dished out right? or is there another round coming up? | 15:16 |
| *** berendt has joined #openstack-infra | 15:16 | |
| fungi | stevemar: i'm going to send one final round the week after feature freeze (probably ~5th of september) | 15:17 |
| jeblair | mordred: what's nat_destination there? | 15:17 |
| fungi | stevemar: the previous batch went out on august 15 so anyone who contributed something that merged prior to that should have a registration discount code already | 15:18 |
| sskripnick | fungi: omg i should be looking at traceback before going here. openstack_catalog/__init__.py seems the issue | 15:18 |
| YorikSar | fungi: Thanks! | 15:18 |
| mordred | jeblair: a nat_destination is a network that has a subnet onto which a floating ip can be attached | 15:18 |
| mordred | jeblair: it is detected in _set_interesting_networks | 15:19 |
| rcarrillocruz | sigh... | 15:19 |
| stevemar | fungi: ah okay, thanks for the heads up | 15:19 |
| rcarrillocruz | i got somehow invited to the slack channel for the cloud appdev event in guadalajara, cos i gave some session via hangouts | 15:19 |
| rcarrillocruz | and now i get email notifications everytime someone puts '@all' | 15:19 |
| rcarrillocruz | like, WHY would that be a default setting | 15:19 |
| jeblair | mordred: so that's a filter that says "get me a port on a network who has a subnet which is the subnet that my private ip is on"? | 15:19 |
| rcarrillocruz | /facepalm | 15:19 |
| *** _nadya_ has joined #openstack-infra | 15:20 | |
| mtreinish | jeblair: oh, the extra metadata param for node type. I got sidetracked and haven't looked at it again | 15:20 |
| mordred | jeblair: no, it's saying "get me the port associated with an ip I have that does not have a FIP already on it and that is on a network that can have a FIP attached to it" | 15:20 |
| mordred | jeblair: (which might have been what you were saying and I just used more words) | 15:20 |
| fungi | stevemar: worth noting, registration prices increase next week (september 1) so anyone with a code up to now should use it before then or their code won't cover 100% of the cost. the batch i'm sending out after feature freeze cover the increased registration rate since those people won't have an opportunity to register before the rate increase | 15:21 |
| stevemar | fungi: oh thats interesting | 15:21 |
| jeblair | mtreinish: ok; i think i eventually context switched to that and dumped some more thoughts at you in channel while you were away. :) if you still want to do that, i can look them up again. | 15:21 |
| openstackgerrit | Merged openstack-infra/system-config: Remove hpuswest hosts from Cacti https://review.openstack.org/361183 | 15:21 |
| *** _nadya_ has quit IRC | 15:22 | |
| rcarrillocruz | jeblair: landed ^ | 15:22 |
| rcarrillocruz | thx for the cleanup offer | 15:22 |
| jeblair | mordred: no, that's not what i was saying, so i guess i'm confused. :( | 15:22 |
| mordred | jeblair: don't feel bad - it's massively confusing | 15:23 |
| mtreinish | jeblair: I don't really have the concentration for it right now, I'm at the ops midcycle | 15:23 |
| jeblair | mordred: why does "get_free_fixed_port" return the port for an existing ip? the word "free" in there is tripping me up. | 15:23 |
| jeblair | mtreinish: okay. let me know when you swing back around to it | 15:23 |
| *** esikachev has quit IRC | 15:24 | |
| mordred | jeblair: "free" as in "free destination" - like "a port that is attached to the server but is not attached to something else already" | 15:24 |
| *** amitgandhinz has quit IRC | 15:25 | |
| jeblair | mordred: but if it's "for an ip i have", surely it's attached already? | 15:25 |
| *** asselin_ has joined #openstack-infra | 15:25 | |
| fungi | i've learned to stop making those assumptions in openstack | 15:25 |
| *** amitgandhinz has joined #openstack-infra | 15:25 | |
| openstackgerrit | Dawid Deja proposed openstack-infra/project-config: Add mistral gate for testing kombu driver https://review.openstack.org/361307 | 15:25 |
| fungi | otherwise i just get mad when i eventually learn the truth | 15:26 |
| jeblair | mordred: oh wait, "get me the port associated with an ip I have that does not have a FIP already on it" is now confusing -- how can a free port have an ip but not have a fip? | 15:26 |
| *** kdas__ has quit IRC | 15:26 | |
| *** kdas__ has joined #openstack-infra | 15:26 | |
| *** asselin__ has quit IRC | 15:27 | |
| sskripnick | mordred: fungi: now it works. thanks a lot! | 15:27 |
| *** sarob has joined #openstack-infra | 15:27 | |
| jeblair | (i am reading that function now to try to grok it) | 15:27 |
| *** piet has joined #openstack-infra | 15:27 | |
| mordred | jeblair: I'm beginning to agree with you that "free" is a bad word here | 15:27 |
| *** armax has joined #openstack-infra | 15:27 | |
| fungi | mordred: sskripnick: however, i'll grant that error message (at least outside the context of the traceback) is mildly misleading in this particular case | 15:28 |
| jeblair | mordred: i'll drop that from my mental model and see if things make more sense :) | 15:28 |
| jeblair | mordred: also, i will see if breakfast helps | 15:28 |
| jeblair | biab | 15:28 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add controller00.vanilla to Cacti https://review.openstack.org/361192 | 15:29 |
| *** derekh has quit IRC | 15:30 | |
| *** sarob has quit IRC | 15:31 | |
| *** pradk has quit IRC | 15:32 | |
| *** roxanaghe has joined #openstack-infra | 15:33 | |
| *** wcriswell has joined #openstack-infra | 15:33 | |
| *** asettle has quit IRC | 15:33 | |
| *** pcaruana has quit IRC | 15:33 | |
| *** pradk has joined #openstack-infra | 15:33 | |
| *** pblaho has joined #openstack-infra | 15:34 | |
| *** esikachev has joined #openstack-infra | 15:34 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti https://review.openstack.org/361195 | 15:35 |
| *** Julien-zte has quit IRC | 15:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Rename _get_free_fixed_port to _nat_destination_port https://review.openstack.org/361316 | 15:36 |
| mordred | jeblair: ^^ follow up patch that renames the function and adds a better (hopefully) doc string | 15:36 |
| mordred | fungi, sskripnick: it is a bad error message ... although contextually I have not figured out a way to more effectively communicate the error | 15:37 |
| mordred | fungi: perhaps we should just add "It's also possible that the package name in setup.cfg does not match the argument to VersionInfo" | 15:37 |
| mordred | to the error message? | 15:37 |
| *** roxanaghe has quit IRC | 15:37 | |
| *** matthewbodkin has quit IRC | 15:38 | |
| *** tesseract- has quit IRC | 15:38 | |
| fungi | mordred: i wonder whether it's possible to distinguish that situation... i guess it's the chicken-and-egg problem of needing to know the package name to find the metadata | 15:38 |
| *** jamesdenton has joined #openstack-infra | 15:38 | |
| fungi | so there's probably no way to tell whether you are missing teh metadata or just got teh package name wrong | 15:38 |
| fungi | mordred: i wonder if it could be made more explicit that it looked for (and did not find) package metadata for <your_package_name_here> | 15:39 |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Add more words to a confusing error message https://review.openstack.org/361317 | 15:41 |
| *** gyee has joined #openstack-infra | 15:41 | |
| mordred | fungi: ^^ | 15:41 |
| mordred | fungi: lemme see if I can get your suggestoin in there too | 15:41 |
| *** zul has joined #openstack-infra | 15:42 | |
| *** eeiden has joined #openstack-infra | 15:42 | |
| *** mikelk has quit IRC | 15:43 | |
| *** mikelk has joined #openstack-infra | 15:43 | |
| *** stewie925 has joined #openstack-infra | 15:44 | |
| clarkb | rcarrillocruz: thats great news on the performance front (that test taking a little less than an hour is about where we want things to be I think) also you need to set max servers to 0 if you want images to be built and uploaded. -1 means do nothing but keep the config | 15:45 |
| rcarrillocruz | ah nice | 15:45 |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Add more words to a confusing error message https://review.openstack.org/361317 | 15:45 |
| rcarrillocruz | i was wondering if there was a difference, you just told me :-) | 15:45 |
| clarkb | fungi: re gerrit threads good to have that confirmed. Now I guess we wait and see if we encounter the issues we have been seeing more recently | 15:45 |
| rcarrillocruz | pushing change | 15:45 |
| clarkb | rcarrillocruz: ya 0 means make and upload images but don't boot insatnces and -1 means don't make images and don't boot instances | 15:46 |
| rcarrillocruz | ++ | 15:47 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Set max-servers on infracloud to 0 https://review.openstack.org/361323 | 15:49 |
| *** mikelk has quit IRC | 15:50 | |
| *** senk has quit IRC | 15:52 | |
| pabelanger | mgagne: sure, no problem. When you have time, I'm free to help | 15:53 |
| mgagne | pabelanger: I'm trying to investigate the issue on my side. So I booted 2 VMs, one ping, one doesn't. So we are investigating it. So yea, I suggest you don't enable it back. | 15:54 |
| mgagne | pabelanger: I tested earlier this week but looks like some compute nodes aren't working that well | 15:54 |
| pabelanger | mgagne: we only launched a handful of nodes over the few hours mtl01 was online, I haven't tried launching anything since we disabled mtl01 | 15:54 |
| clarkb | mgagne: pabelanger: jsut to double check we opened up the security groups like we do in the other regions? (allow all ipv4 and ipv6 in and out?) | 15:54 |
| mgagne | clarkb: AFAIK, default rules are: any<->any | 15:55 |
| mgagne | but better double check | 15:55 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti https://review.openstack.org/361224 | 15:55 |
| *** yamamoto has quit IRC | 15:55 | |
| pabelanger | clarkb: let me check with cloud_launcher | 15:55 |
| *** Marx314 has quit IRC | 15:56 | |
| *** zul has quit IRC | 15:56 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic https://review.openstack.org/335981 | 15:57 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 15:57 |
| *** jaosorior has quit IRC | 15:58 | |
| *** yamamoto has joined #openstack-infra | 15:58 | |
| AJaeger | I'd like another review for a bindep documentation change, please: https://review.openstack.org/#/c/358811/ | 15:59 |
| *** raunak has joined #openstack-infra | 15:59 | |
| pabelanger | clarkb: rcarrillocruz: mordred: Oh, we need to land https://review.openstack.org/#/c/360698/ Adds region support to ansible-role-cloud-launcher | 15:59 |
| pabelanger | then I can validate our security groups in internap-mtl01 | 16:00 |
| rcarrillocruz | pabelanger: are you not core on the launcher? | 16:00 |
| rcarrillocruz | can't remember if i did you core in there or infra-ansible | 16:00 |
| rcarrillocruz | pabelanger: you are core, feel free to approve the change | 16:01 |
| rcarrillocruz | i also did clarkb core on the project | 16:01 |
| *** flepied has quit IRC | 16:01 | |
| openstackgerrit | Dawid Deja proposed openstack-infra/project-config: Add mistral gate for testing kombu driver https://review.openstack.org/361307 | 16:03 |
| *** matrohon has quit IRC | 16:03 | |
| openstackgerrit | Merged openstack-infra/system-config: Add compute010.vanilla to static Ansible inventory https://review.openstack.org/361285 | 16:04 |
| *** Swami has joined #openstack-infra | 16:04 | |
| openstackgerrit | David Smith proposed openstack-infra/jenkins-job-builder: email-ext trigger script https://review.openstack.org/361331 | 16:05 |
| *** hashar has quit IRC | 16:05 | |
| *** e0ne has joined #openstack-infra | 16:05 | |
| *** tqtran has joined #openstack-infra | 16:07 | |
| *** chlong has quit IRC | 16:07 | |
| rcarrillocruz | weee, nodepool building images fine so far | 16:08 |
| *** edtubill has joined #openstack-infra | 16:08 | |
| clarkb | was the 1540UTC restart of nodepool builder to pick up the infra cloud? | 16:08 |
| clarkb | (it killed the ubuntu-trusty build, just making sure that was intentional and not some new bug that needs to be tracked down | 16:09 |
| fungi | dhellmann: sphinx question... if i have a document that needs to link to some static plain-text files, do those go in doc/source/_static/ or is there a better place? and what causes the contents of _static to get included in doc/build/html/ ? just sticking files in there doesn't seem to do it | 16:09 |
| fungi | dhellmann: scratch that, it did work. i think i need to adjust my urls | 16:10 |
| fungi | the log just doesn't enumerate them the way it does included images | 16:10 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add baremetal00 to cacti hosts https://review.openstack.org/361181 | 16:10 |
| *** pilgrimstack has quit IRC | 16:10 | |
| *** tqtran has quit IRC | 16:11 | |
| *** cody-somerville has joined #openstack-infra | 16:13 | |
| *** esikachev has quit IRC | 16:13 | |
| *** smarcet1981 has joined #openstack-infra | 16:13 | |
| *** swat30 has left #openstack-infra | 16:13 | |
| openstackgerrit | Merged openstack-infra/git-review: Remove argparse from requirements https://review.openstack.org/270410 | 16:15 |
| *** sdague has quit IRC | 16:15 | |
| *** smarcet has quit IRC | 16:16 | |
| openstackgerrit | David Smith proposed openstack-infra/jenkins-job-builder: email-ext trigger script https://review.openstack.org/361331 | 16:17 |
| *** senk has joined #openstack-infra | 16:17 | |
| *** itisha has joined #openstack-infra | 16:18 | |
| *** salv-orlando has joined #openstack-infra | 16:18 | |
| *** chlong has joined #openstack-infra | 16:19 | |
| *** shashank_hegde has joined #openstack-infra | 16:19 | |
| *** kaisers_ has joined #openstack-infra | 16:21 | |
| *** e0ne has quit IRC | 16:21 | |
| * clarkb notices that the ntpdate chagne fails on trusty where we have not updated our images as well as xenial | 16:22 | |
| clarkb | will try to get those images updated before approving the d-g ntpdate change | 16:22 |
| AJaeger | fungi, regarding the bindep change with openstack-ci-usage: Should I name that file "Adoption by projects" and have OpenStack as only section in it? | 16:23 |
| *** AnarchyAo has joined #openstack-infra | 16:24 | |
| fungi | AJaeger: my main concern is that it still makes it seem like something which is 1. primarily targeted at ci use cases, and 2. mostly only useful for openstack projects | 16:25 |
| *** psachin has quit IRC | 16:25 | |
| fungi | i'd very much prefer to move any documentation on how openstack's ci uses the tool into a documentation repository about openstack's ci | 16:26 |
| *** kaisers_ has quit IRC | 16:26 | |
| AJaeger | fungi, I made this section short on purpose ;) I'm fine just adding a link to our Infra Manual - but I want that link ;) | 16:26 |
| fungi | we can make that the primary url for people wanting to know how our ci leverages bindep, and then it can link to the general bindep documentation for details on things not directly related to our ci | 16:27 |
| *** kdas__ is now known as kushal | 16:27 | |
| AJaeger | Let me propose something. First patch is with your suggestion - and then I try something different... | 16:27 |
| *** kushal has quit IRC | 16:27 | |
| *** kushal has joined #openstack-infra | 16:27 | |
| fungi | okay | 16:27 |
| *** hockeynut has quit IRC | 16:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Document OpenStack usage https://review.openstack.org/358998 | 16:28 |
| AJaeger | fungi, ignore this review for now ^ | 16:29 |
| *** vhosakot has joined #openstack-infra | 16:29 | |
| * fungi has become remarkably adept at ignoring reviews | 16:30 | |
| *** roxanaghe has joined #openstack-infra | 16:30 | |
| *** bethwhite_ has quit IRC | 16:30 | |
| *** dtantsur is now known as dtantsur|afk | 16:30 | |
| *** sdague has joined #openstack-infra | 16:30 | |
| *** zul has joined #openstack-infra | 16:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Add link to Infra Manual https://review.openstack.org/361353 | 16:31 |
| AJaeger | fungi, could you persuade you to switch defaults and not ignore this review, please? ^ | 16:31 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 16:31 |
| pleia2 | AJaeger: bit of whitespace there | 16:31 |
| AJaeger | ;( | 16:32 |
| AJaeger | pleia2: thanks, fixed. | 16:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Add link to Infra Manual https://review.openstack.org/361353 | 16:32 |
| *** jerryz has joined #openstack-infra | 16:32 | |
| *** sandanar has joined #openstack-infra | 16:32 | |
| * pleia2 thumbs up | 16:33 | |
| AJaeger | pleia2: thanks. could you review https://review.openstack.org/#/c/358811/ as well ,please? | 16:33 |
| pleia2 | yep | 16:33 |
| pleia2 | oh yeah, I was looking at this earlier and it got lost in tabs | 16:33 |
| *** dizquierdo has quit IRC | 16:34 | |
| *** dimtruck is now known as zz_dimtruck | 16:34 | |
| AJaeger | ;) | 16:34 |
| *** sambetts is now known as sambetts|afk | 16:34 | |
| electrofelix | just wondering if the latest zuul codebase still supports jenkins + ansible? wondering if it's possible to upgrade to a release with both and then switch jobs across rather than doing a giant switch | 16:35 |
| *** cody-somerville has quit IRC | 16:35 | |
| *** psachin has joined #openstack-infra | 16:35 | |
| clarkb | electrofelix: zuul continues to support gearman. So any gearman worker follwing the zuul gearman protocol should work | 16:36 |
| *** david-lyle has joined #openstack-infra | 16:36 | |
| clarkb | electrofelix: this should include the jenkins gearman plugin and the zuul launcher code though we don't run the jenkins gearamn plugin anymore so I can't vouch for it | 16:36 |
| AJaeger | thanks, pleia2 ! | 16:36 |
| pleia2 | AJaeger: sure thing | 16:36 |
| electrofelix | clarkb: excellent, at least it's possible so there is an upgrade path | 16:36 |
| pleia2 | AJaeger: while I have your attention, I haven't talked to the i18n folks lately, given the time in the cycle, I think Zanata upgrade + Xenial will have to slip until next cycle | 16:37 |
| pleia2 | AJaeger: I'm really unhappy about this, but we're just not where we need to be with Xenial just yet (though I do think we're close) | 16:37 |
| clarkb | electrofelix: its only zuul version 3.0 that will not be backward compatbile with the gearman protocol. Anything onthe 2.x series should continue to do so | 16:38 |
| AJaeger | pleia2: haven't talked to them either ;( Yeah, it's rather late | 16:38 |
| pleia2 | I will spend some time next week seeing what we need to support, but major changes now are too late | 16:38 |
| openstackgerrit | Merged openstack-infra/project-config: Set max-servers on infracloud to 0 https://review.openstack.org/361323 | 16:38 |
| AJaeger | could we get translate-dev set up already? | 16:38 |
| AJaeger | thanks, pleia2 | 16:38 |
| pleia2 | AJaeger: we should, I'll work on that | 16:38 |
| *** lucasagomes is now known as lucas-afk | 16:39 | |
| *** tkelsey has quit IRC | 16:39 | |
| electrofelix | clarkb: thanks for the confirmation | 16:39 |
| *** electrofelix has quit IRC | 16:39 | |
| AJaeger | fungi, I have no great idea right now how to explain profiles in the README.rst of bindep better - if you have one, go ahead. | 16:40 |
| rcarrillocruz | pabelanger: it's kind of late for me now to look at the afs stuff on infracloud, got hooked on other inventory and provisioning work | 16:40 |
| AJaeger | fungi, if you're happy with https://review.openstack.org/361353 , then I'll abandon https://review.openstack.org/358998 | 16:40 |
| *** jpich has quit IRC | 16:40 | |
| pabelanger | rcarrillocruz: no problems, do you want me to stand it up or hold off until you are around? | 16:40 |
| pabelanger | rcarrillocruz: which need a node with 100GB disk or a volume we can attach | 16:41 |
| rcarrillocruz | if the afs thing is just about creating a node with mirror.blah.infracloud, add dns and put mirror class on site.pp, i think i'm ok you looking at it | 16:41 |
| rcarrillocruz | pabelanger: ok, given we have no cinder on the infracloud, then i guess we want to have a flavor with more disk | 16:41 |
| fungi | AJaeger: yeah, i think just having the bindep ci documentation in infra-manual link to the bindep documentation about profiles when it mentions them would be sufficient | 16:41 |
| rcarrillocruz | although we could use m1.large , iirc it has 160GB or something | 16:42 |
| pabelanger | rcarrillocruz: Right, so let me add the default into cloud_launcher | 16:42 |
| rcarrillocruz | ++ | 16:42 |
| pabelanger | or we can use that | 16:42 |
| pabelanger | rcarrillocruz: but ya, basically what you said to launch the AFS mirror | 16:42 |
| AJaeger | fungi: ah, that way around - ok. Let me look at infra manual... | 16:42 |
| rcarrillocruz | k | 16:43 |
| fungi | AJaeger: i'm not really keen on 361353 either. i think when people need to know about how bindep is used in our ci we should tell them to look in the infra manual. the bindep documentation should be about bindep, not about how we happen to use it in openstack ci | 16:43 |
| pleia2 | ah, that's fair | 16:43 |
| fungi | it's perfectly reasonable for the infra manual to link to the bindep documentation for more details though | 16:44 |
| *** pradk has quit IRC | 16:44 | |
| *** psachin has quit IRC | 16:44 | |
| *** rcernin has joined #openstack-infra | 16:44 | |
| * rcarrillocruz makes a mental note to open a jira for compute005, looks it has a bad hard drive | 16:44 | |
| fungi | consider if we weren't the bindep developers... for example we wouldn't expect the gerrit devs to include a link in their documentation about how openstack uses gerrit | 16:45 |
| *** links has joined #openstack-infra | 16:45 | |
| *** shashank_hegde has quit IRC | 16:45 | |
| fungi | but we have plenty of documentation on how we use gerrit, and link to gerrit's documentation | 16:45 |
| *** pradk has joined #openstack-infra | 16:45 | |
| AJaeger | fungi, http://docs.openstack.org/infra/manual/drivers.html#package-requirements already links to bindep - and has a link for "profile". | 16:46 |
| pabelanger | rcarrillocruz: Also, will check cloud-launcher again | 16:46 |
| rcarrillocruz | y, you have +2 powers | 16:47 |
| pabelanger | cool, thanks | 16:47 |
| fungi | AJaeger: maybe we should point people there when they ask about bindep in our ci? i forgot we'd added that section | 16:47 |
| openstackgerrit | Merged openstack-infra/bindep: Give some examples https://review.openstack.org/358811 | 16:47 |
| pabelanger | fungi: clarkb: if you have a moment: https://review.openstack.org/#/c/360698/ adds internap-mtl01 to cloud-launcher | 16:47 |
| AJaeger | yeah, we should point them to our drivers page. | 16:48 |
| fungi | AJaeger: and we can beef up the documentation in that section with your prose from the earlier change | 16:48 |
| fungi | which i thought was quite good | 16:48 |
| *** kien-ha has joined #openstack-infra | 16:48 | |
| *** esberglu has quit IRC | 16:48 | |
| EmilienM | nibalizer: how can we get a final approve on https://review.openstack.org/#/c/359539/ ? | 16:48 |
| EmilienM | fungi: if you have time, it's really a quick one ^ | 16:48 |
| openstackgerrit | Merged openstack-infra/system-config: Add controller00.vanilla to Cacti https://review.openstack.org/361192 | 16:49 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 16:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Improve wording on bindep profile https://review.openstack.org/361364 | 16:49 |
| AJaeger | fungi, ^ | 16:49 |
| pleia2 | EmilienM: lgtm | 16:49 |
| EmilienM | pleia2: thx a lot | 16:49 |
| *** tphummel has joined #openstack-infra | 16:50 | |
| clarkb | pabelanger: if we are using that to bootstrap new clouds now is the playbook for that somewhere? and can we document that in the system-config docs "under bringing up a new cloud" or something? | 16:50 |
| clarkb | also it is entirely possible this ahs been done I just missed it | 16:51 |
| openstackgerrit | Merged openstack-infra/project-config: Add initialize-urandom to fedora 24 https://review.openstack.org/356818 | 16:51 |
| *** flepied has joined #openstack-infra | 16:51 | |
| rcarrillocruz | clarkb: yeah, it's playbooks/run_cloud_launcher.yml | 16:51 |
| rcarrillocruz | and indeed, we should document that | 16:52 |
| rcarrillocruz | infracloud also needs a LOT of love in terms of documentation | 16:52 |
| rcarrillocruz | still references the HP1 HP2 regions :/ | 16:52 |
| pabelanger | clarkb: Right, I've been running it manually today. However, I think we are stable enough to add a playbook into our ansible wheel. I'll work on that now | 16:52 |
| clarkb | rcarrillocruz: but where are all the inputs ? | 16:52 |
| clarkb | rcarrillocruz: all taht says is "run this role" | 16:52 |
| clarkb | I am mostly interested in what the cloud is being told to do for example | 16:53 |
| rcarrillocruz | yeah, you need to feed to ansible-playbook -e /opt/system-config/production/cloud_launcher/clouds_layouts.yml | 16:53 |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/system-config/tree/cloud_launcher/clouds_layouts.yml | 16:53 |
| rcarrillocruz | clarkb: ^ | 16:53 |
| pabelanger | Yup, http://git.openstack.org/cgit/openstack-infra/system-config/tree/cloud_launcher/clouds_layouts.yml is what we have setup today | 16:54 |
| clarkb | why are they in different dirs? | 16:54 |
| rcarrillocruz | and i plan to put on the launcher a param to run the whole thing on a specific cloud | 16:54 |
| pabelanger | I am just cleaning up the last of the password issues with tripleo | 16:54 |
| rcarrillocruz | if you run it that way, it runs against all clousd on oscc file | 16:54 |
| AJaeger | pleia2: could you review https://review.openstack.org/361364 as well, please? | 16:54 |
| clarkb | rcarrillocruz: that sounds helpful | 16:54 |
| *** bin_ has quit IRC | 16:54 | |
| rcarrillocruz | so the idea is that you could just ansible-playbook -e @/opt/system-config/production/cloud_launcher/clouds_layouts.yml -e "cloud=infracloud' for example | 16:55 |
| *** notnownikki has quit IRC | 16:55 | |
| *** kushal has quit IRC | 16:55 | |
| *** sdague has quit IRC | 16:56 | |
| * AJaeger wishes everybody a great weekend... | 16:56 | |
| pleia2 | AJaeger: done, have a nice weekend :) | 16:56 |
| clarkb | I also semi wonder why it has to be separte files but I guess thats ansible's parser? | 16:56 |
| fungi | have a great weekend, AJaeger! | 16:56 |
| *** jamesdenton has quit IRC | 16:56 | |
| clarkb | basically one doesn't make sense without the other and they are fully dependent on each other (at least at this point in time) so colocating makes sense to me | 16:56 |
| AJaeger | thanks, pleia2 and fungi! | 16:56 |
| clarkb | pabelanger: is the issue in the new internap region security groups then? | 16:58 |
| rcarrillocruz | clarkb: btw, i'm ok moving clouds_layouts.yml to playbooks folder, providing we put a comment at the beginning "This is a variable file, meant to be passed to run_cloud_launcher blah" | 16:58 |
| rcarrillocruz | to avoid people thinking it's a playbook, just cos it's in playbooks folder and it has .yml extension | 16:58 |
| clarkb | rcarrillocruz: ya I think its easier to discover that way particularly since we don't have docs for it :) | 16:58 |
| rcarrillocruz | lol | 16:59 |
| rcarrillocruz | i'm not going to disagree with that | 16:59 |
| rcarrillocruz | :D | 16:59 |
| clarkb | or move the playbook into the cloud launcher dir | 16:59 |
| rcarrillocruz | nah, i think the primer may be better for consistency | 16:59 |
| rcarrillocruz | anyway, going away for my weekend | 16:59 |
| rcarrillocruz | have a good one folks | 16:59 |
| rcarrillocruz | cheers | 16:59 |
| clarkb | you too | 16:59 |
| pabelanger | clarkb: lets land https://review.openstack.org/#/c/360698 and see | 16:59 |
| clarkb | pabelanger: its been approved | 16:59 |
| clarkb | rcarrillocruz got it | 16:59 |
| pabelanger | okay cool | 17:00 |
| clarkb | but it should be easy to do a security group listing against internap and check | 17:00 |
| pabelanger | Ya, I did that last night and pretty sure it was open | 17:00 |
| pabelanger | however, I might have misread it | 17:00 |
| *** raildo has quit IRC | 17:01 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs-spec-helper" https://review.openstack.org/359539 | 17:01 |
| *** chlong has quit IRC | 17:02 | |
| openstackgerrit | Merged openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti https://review.openstack.org/361195 | 17:03 |
| clarkb | pabelanger: I only see 2 rules but I think you need 4. ingress + egress for ipv4 and ingress + egress for ipv6 | 17:03 |
| mgagne | pabelanger: our investigation is still ongoing. So far, we suspect some kind of misconfiguration in our TOR switches where some compute nodes reach for the passive switch instead of the active one. | 17:03 |
| *** smarcet has joined #openstack-infra | 17:04 | |
| clarkb | mgagne: ok, I am trying to double check the 2 security group rules we do have | 17:04 |
| clarkb | maybe you can make an ingress and egress rule somehow | 17:04 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Switch to import module instead of function in properties https://review.openstack.org/361373 | 17:05 |
| pabelanger | clarkb: sure, I let me add it | 17:05 |
| kien-ha | zaro: Hey, you around? | 17:05 |
| pabelanger | mgagne: thanks for the update | 17:06 |
| fungi | pleia2: were you interested in reviewing the rest of the open changes under topic:wiki-upgrade? i think they're basically complete now | 17:06 |
| clarkb | hrm openstackclient doesn't amke this easy... it only shows you the rule uuid and ip range on the group show output | 17:06 |
| *** smarcet1981 has quit IRC | 17:06 | |
| pleia2 | fungi: sure thing | 17:06 |
| clarkb | no protocol or direction data | 17:06 |
| pleia2 | fungi: oh hi, that's a lot of patches :D | 17:07 |
| * pleia2 gets to it! | 17:07 | |
| *** degorenko is now known as _degorenko|afk | 17:07 | |
| fungi | pleia2: yeah, just don't want them to get stale | 17:07 |
| fungi | pleia2: thanks! | 17:07 |
| *** sarob has joined #openstack-infra | 17:07 | |
| fungi | pleia2: feel free to approve any you're comfortable approving. i won't be reenabling puppet for production until i can correctly build a replacement wiki.o.o and a working wiki-dev.o.o with the current state of the puppet-mediawiki and system-config repos | 17:08 |
| pleia2 | fungi: noted, thanks | 17:08 |
| *** manjeets_ has joined #openstack-infra | 17:09 | |
| *** Apoorva has joined #openstack-infra | 17:10 | |
| clarkb | hrm I can't seem to use -c direction to get direction... | 17:10 |
| *** yamamoto has quit IRC | 17:10 | |
| clarkb | might have to use neutron client | 17:10 |
| kambiz | pabelanger: I think the rdo ci ntp issue is resolved. slagle is replying to the thread. | 17:10 |
| pabelanger | kambiz: great | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti https://review.openstack.org/361224 | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Add baremetal00 to cacti hosts https://review.openstack.org/361181 | 17:11 |
| *** Genek has joined #openstack-infra | 17:12 | |
| *** Genek has joined #openstack-infra | 17:12 | |
| pabelanger | clarkb: okay, cloud-launcher region patch merged, I can run it manually now if you want | 17:12 |
| *** sarob has quit IRC | 17:12 | |
| pabelanger | 360707 is in the gate now too | 17:12 |
| clarkb | pabelanger: one sec, trying to list these rules and osc is fighting me | 17:13 |
| *** _nadya_ has joined #openstack-infra | 17:13 | |
| pabelanger | ok | 17:13 |
| *** ilyashakhat has joined #openstack-infra | 17:13 | |
| *** njohnston has joined #openstack-infra | 17:13 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Improve wording on bindep profile https://review.openstack.org/361364 | 17:13 |
| clarkb | also noticing osc will occasionalyl take a long time to talk to that region potentially due to the switch issues mgagne was talking about? | 17:13 |
| clarkb | pabelanger: so not sure if playbook would even work right this moment | 17:14 |
| clarkb | of course could be pebkac on my part (but like half the commands do work so I dunno) | 17:14 |
| *** shashank_hegde has joined #openstack-infra | 17:15 | |
| *** _nadya_ has quit IRC | 17:15 | |
| *** asselin__ has joined #openstack-infra | 17:16 | |
| *** mwhahaha has quit IRC | 17:16 | |
| pabelanger | clarkb: I haven't notices the delay, but I haven't tried today | 17:16 |
| clarkb | pabelanger: currently `openstack --os-cloud openstackjenkins-internap --os-region-name mtl01 security group list` is failing | 17:17 |
| clarkb | or rather not returning data very quikcly | 17:17 |
| clarkb | (I set the env var for the all clouds clouds.yaml file too, if I get the cloudname or region name wrong it immedaitely returns and complains at me) | 17:17 |
| *** asselin_ has quit IRC | 17:17 | |
| clarkb | pabelanger: I would say go ahead wit hthe playbook, I just got "Connection failure that may be retried." | 17:19 |
| clarkb | pabelanger: curious to see if ansible + shade fair better | 17:19 |
| *** shashank_hegde has quit IRC | 17:20 | |
| zaro | kien-ha: what up? | 17:20 |
| clarkb | stevemar: dtroyer_zz is ^ a known issue with osc? | 17:20 |
| pabelanger | clarkb: sure, give me a sec | 17:20 |
| kien-ha | zaro: Just wanted you to do a really quick review for a patch. | 17:21 |
| kien-ha | zaro: https://review.openstack.org/#/c/361373/ | 17:22 |
| *** rcernin has quit IRC | 17:22 | |
| kien-ha | zaro: The patch can be found in the link I provided. You got a moment to spare for me? | 17:22 |
| *** esikachev has joined #openstack-infra | 17:22 | |
| stevemar | clarkb: definitely not a known issue | 17:23 |
| stevemar | clarkb: using version 2.6.0 or have you upgraded? | 17:24 |
| manjeets_ | anyone ever seen this issuse while stacking http://paste.openstack.org/show/564037 | 17:24 |
| clarkb | I am going to test against another cloud just to rule out switch issues | 17:24 |
| clarkb | stevemar: give me a minute to do that and get the version info | 17:24 |
| clarkb | stevemar: 3.1.0 here | 17:24 |
| stevemar | super :P | 17:24 |
| *** yamahata has quit IRC | 17:25 | |
| clarkb | ya just got the same thing against bluebox "Connection failure that may be retried." | 17:25 |
| clarkb | stevemar: running openstack --os-cloud openstackzuul-bluebox security group list | 17:25 |
| stevemar | clarkb: so no, not sure whats going on, and it's sad to hear that you're not getting enough info about a security group when trying to display a particular one | 17:26 |
| zaro | kien-ha: looking | 17:26 |
| *** DericHorn-HP has joined #openstack-infra | 17:26 | |
| *** kzaitsev_mb has quit IRC | 17:26 | |
| *** sputnik13 has joined #openstack-infra | 17:26 | |
| clarkb | stevemar: when it was semi working the info was very terse so I upgraded osc and now it just doesn't work at all | 17:26 |
| *** esikachev has quit IRC | 17:26 | |
| clarkb | (I thought maybe the info might be less terse with newer version :) ) | 17:26 |
| openstackgerrit | Merged openstack-infra/system-config: Add internap to cloud_launcher https://review.openstack.org/360707 | 17:27 |
| *** sdague has joined #openstack-infra | 17:27 | |
| *** piet has quit IRC | 17:27 | |
| stevemar | clarkb: whats the --debug output say? | 17:27 |
| mgagne | clarkb: are you talking about API or SSH access? | 17:28 |
| *** tosky has quit IRC | 17:28 | |
| clarkb | mgagne: api via osc | 17:29 |
| clarkb | stevemar: paste in a saec | 17:29 |
| clarkb | mgagne: its happening with osc against bluebox though so I doubt its internap at fault for this | 17:29 |
| mgagne | clarkb: api shouldn't have any issue since it's hosted in a different system/rack | 17:29 |
| clarkb | mgagne: kk | 17:30 |
| clarkb | stevemar: http://paste.openstack.org/show/564039/ | 17:30 |
| pabelanger | clarkb: cloud-launcher running now | 17:31 |
| zaro | kien-ha: lgtm | 17:31 |
| *** piet has joined #openstack-infra | 17:32 | |
| kien-ha | zaro: Thanks a lot! | 17:32 |
| *** aviau has quit IRC | 17:33 | |
| *** aviau has joined #openstack-infra | 17:33 | |
| pabelanger | clarkb: it does look like we missed an ingress rule for ipv4/ipv6 for one of the projects | 17:34 |
| pabelanger | http://paste.openstack.org/show/564041/ | 17:34 |
| clarkb | pabelanger: looks like one missed one rule and the other missed two which lines up with what I saw if the one missing two was the one I was looking at | 17:35 |
| dtroyer_zz | stevemar, clarkb: that connection retry seems like it might be the current known issue with the SDK: https://bugs.launchpad.net/python-openstacksdk/+bug/1616522 | 17:36 |
| openstack | Launchpad bug 1616522 in OpenStack SDK "openstacksdk 0.9.2 and higher does have one issue "SDKException: Connection failure that may be retried." when using python-openstackclient performing neutron part action" [Undecided,In progress] - Assigned to Zhao Jian (jianzj) | 17:36 |
| dtroyer_zz | https://review.openstack.org/#/c/360343/ is up to fix it | 17:36 |
| *** zz_dimtruck is now known as dimtruck | 17:37 | |
| clarkb | oh is it using http when it should be using https, interesting | 17:37 |
| clarkb | my log output does confirm the url is http | 17:38 |
| * clarkb checks catalog | 17:38 | |
| clarkb | catalog says we should use https for neutron. I agree that this is likeyl the same bug. Thank you for pointing that out | 17:38 |
| *** AnarchyAo has quit IRC | 17:39 | |
| dhellmann | fungi: sorry, I was offline when you had your sphinx question. is that all sorted now? | 17:39 |
| clarkb | pabelanger: before we put mtl01 back into circulation we should run a devstack-gate reproduce.sh on an instance there. I am happy to do that if you like | 17:39 |
| clarkb | pabelanger: and if that works and mgagne gives us the go ahead we can probably turn it back on | 17:40 |
| pabelanger | clarkb: WFM | 17:40 |
| mgagne | pabelanger: I wouldn't enable back mtl01 until we (I) tell you to =) | 17:40 |
| mgagne | yea | 17:40 |
| clarkb | mgagne: yup yup | 17:40 |
| clarkb | maybe I will wait for mgagne then I will test :) | 17:40 |
| clarkb | and in the meantime find some breakfast | 17:40 |
| *** shashank_hegde has joined #openstack-infra | 17:41 | |
| *** sarob has joined #openstack-infra | 17:42 | |
| fungi | dhellmann: yep, totally figured out | 17:42 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update TestNG to use valid_dict property of convert xml https://review.openstack.org/361396 | 17:42 |
| fungi | dhellmann: it was a mix of me failing at anonymous hyperlinks in rst and not realizing that static file copying is not logged as verbosely as image copying | 17:42 |
| dhellmann | fungi : oh, interesting. I wonder why that's different. | 17:43 |
| *** ericksonsantos has quit IRC | 17:43 | |
| dhellmann | maybe because they don't want to report all the css and js files | 17:43 |
| *** Apoorva has quit IRC | 17:45 | |
| *** sarob has quit IRC | 17:45 | |
| fungi | dhellmann: yeah, it could just be that it would be _too_ verbose | 17:45 |
| dhellmann | yeah | 17:47 |
| *** berendt has quit IRC | 17:47 | |
| fungi | i need to disappear for a couple hours to run errands, but will be back later | 17:48 |
| *** jheroux has quit IRC | 17:49 | |
| anteaya | fungi: happy errands | 17:50 |
| *** gyee has quit IRC | 17:50 | |
| *** rhallisey has quit IRC | 17:53 | |
| *** ihrachys has quit IRC | 17:55 | |
| *** jed56 has quit IRC | 17:55 | |
| *** tpsilva has joined #openstack-infra | 18:00 | |
| *** sarob has joined #openstack-infra | 18:00 | |
| *** rbrndt has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack-infra/system-config: Clean up tripleo-test-cloud-rh1 os-client-config https://review.openstack.org/360782 | 18:02 |
| *** sdake_ has quit IRC | 18:02 | |
| *** eharney has quit IRC | 18:03 | |
| *** yamahata has joined #openstack-infra | 18:03 | |
| *** tqtran has joined #openstack-infra | 18:04 | |
| *** sarob has quit IRC | 18:05 | |
| *** sandanar has quit IRC | 18:06 | |
| *** ociuhandu has quit IRC | 18:06 | |
| *** senk has quit IRC | 18:08 | |
| *** kaisers_ has joined #openstack-infra | 18:10 | |
| *** yamamoto has joined #openstack-infra | 18:10 | |
| *** Na3iL has quit IRC | 18:11 | |
| *** kien-ha has quit IRC | 18:11 | |
| *** sarob has joined #openstack-infra | 18:12 | |
| *** sarob has quit IRC | 18:13 | |
| *** chem|off` is now known as chem | 18:14 | |
| *** kaisers_ has quit IRC | 18:15 | |
| *** yamamoto has quit IRC | 18:15 | |
| *** matrohon has joined #openstack-infra | 18:16 | |
| *** bnemec is now known as beekneemech | 18:16 | |
| *** ilyashakhat has quit IRC | 18:17 | |
| *** _ari_ has quit IRC | 18:19 | |
| *** _nadya_ has joined #openstack-infra | 18:19 | |
| jlvillal | clarkb: Thanks for the multi-node info. good stuff :) | 18:20 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Switch to import module instead of function in properties https://review.openstack.org/361373 | 18:24 |
| pabelanger | rcarrillocruz: cloud_launcher now properly working. All password for defined clouds correct | 18:24 |
| rcarrillocruz | \o/ | 18:25 |
| mordred | pabelanger, rcarrillocruz: WOOT | 18:25 |
| clarkb | and it added 3 ingress rules for us in internap mtl01 | 18:25 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 18:25 |
| pabelanger | clarkb: all the same? | 18:25 |
| mgagne | clarkb: let me know what they are so we can see if we can add them to our default rule set | 18:26 |
| *** dprince has quit IRC | 18:26 | |
| clarkb | pabelanger: no one was one account and just ipv4 and the other two were in the other accuont and ipv4 + ipv6 | 18:26 |
| pabelanger | okay, cool | 18:26 |
| *** bnemec has joined #openstack-infra | 18:26 | |
| clarkb | mgagne: its just allow all traffic ingress over ipv4 and all traffic ingress over ipv6 according to the log pabelanger posted | 18:26 |
| *** beekneemech has quit IRC | 18:26 | |
| *** bnemec is now known as beekneemech | 18:27 | |
| mgagne | clarkb: I thought it was already allowed by default | 18:27 |
| *** Apoorva has joined #openstack-infra | 18:27 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/system-config/tree/cloud_launcher/clouds_layouts.yml | 18:27 |
| pabelanger | mgagne: ^ | 18:27 |
| clarkb | mgagne: neutron blocks everything by default iirc | 18:27 |
| pabelanger | that is our defaults now | 18:27 |
| mgagne | not with us =) | 18:27 |
| *** dtardivel has quit IRC | 18:27 | |
| clarkb | mordred: can you point me where I can be useful on the nodepool private ipv4 address stuff | 18:27 |
| pabelanger | localhost : ok=128 changed=0 unreachable=0 failed=0 | 18:27 |
| mgagne | clarkb: we have a custom extension/code which injects OUR default rules which are: any ipv4, any ipv6 | 18:27 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-log_processor: Switch to using the new log_processor project https://review.openstack.org/254992 | 18:28 |
| *** Apoorva has quit IRC | 18:28 | |
| mgagne | clarkb: rules that should be applied first time default SG is read/used: https://gist.github.com/mgagne/ae230b323ae32077f651f4ca9320419c | 18:28 |
| clarkb | mordred: fwiw I am looking on an osic instance and the private file does have its ipv4 addr in there | 18:28 |
| clarkb | mordred: so maybe things are working as expected right now? | 18:29 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes https://review.openstack.org/361164 | 18:29 |
| *** cody-somerville has joined #openstack-infra | 18:29 | |
| clarkb | mgagne: it looked like we had egress rules for that but not ingress? | 18:29 |
| *** sarob has joined #openstack-infra | 18:29 | |
| clarkb | mgagne: I am just going off of what pabelanger's ansible run did since osc is currently not functional for neutron things over https | 18:29 |
| mgagne | clarkb: I know egress isn't filtered in Neutron. I will check with my own account. maybe our tests didn't triggered that code path for some reasons. will check | 18:30 |
| *** _nadya_ has quit IRC | 18:30 | |
| clarkb | mordred: or did the restart happen and I missed it? | 18:30 |
| mgagne | strange, my account is fine :-/ | 18:31 |
| *** Shrews has quit IRC | 18:31 | |
| *** phschwartz has quit IRC | 18:31 | |
| *** mordred has quit IRC | 18:31 | |
| pabelanger | clarkb: mgagne: I actually think it might have been openstackci-internap-nyj01 that changed | 18:31 |
| clarkb | pabelanger: oh huh | 18:31 |
| *** csomerville has joined #openstack-infra | 18:31 | |
| clarkb | in any case its hard to independently verify without getting a different client | 18:31 |
| pabelanger | 996689aad9074f8d8bf8bbb36f48084f was the tenant_id | 18:32 |
| pabelanger | I'll look it up shortly | 18:32 |
| *** shardy has quit IRC | 18:33 | |
| *** cody-somerville has quit IRC | 18:34 | |
| rtheis | Is there a known issue with gate jobs resetting? We've been seeing that on several networking-ovn patch sets and I'm not sure how to track that down. | 18:34 |
| pabelanger | rtheis: which review? | 18:34 |
| rtheis | https://review.openstack.org/#/c/361310/ | 18:34 |
| rtheis | gate-tempest-dsvm-networking-ovn-native-services has ran to near completion several times | 18:34 |
| clarkb | rtheis: I am guessing its related to the ipv6 only cloud as I noticed a coupe ovn related jobs with ansible rc 3s | 18:35 |
| pabelanger | Ya, I think we still have some jobs that need patches | 18:35 |
| clarkb | rtheis: basically make sure that you can run on an instance that gets its networking via ipv6 RAs | 18:35 |
| clarkb | rtheis: in devstack we had to set the public interface's accept_ra sysctl value to 2 because neutron sets forwarding to 1 | 18:35 |
| *** asettle has joined #openstack-infra | 18:36 | |
| rtheis | clarkb: ah, and the networking-ovn native services job doesn't use neutron L3 agent so it may not be setting that | 18:36 |
| *** tkelsey has joined #openstack-infra | 18:36 | |
| *** csomerville has quit IRC | 18:37 | |
| clarkb | hrm mordred just ping timed out :/ | 18:37 |
| rtheis | clarkb: can I tell if accept_ra is being set by looking in the logs for the job? | 18:37 |
| *** rvasilets___ has joined #openstack-infra | 18:37 | |
| clarkb | nodepool has not restarted since the 12th so I think the current nodepool code will just work with osic and multinoded | 18:37 |
| pabelanger | telnet://2001:4800:1ae1:18:f816:3eff:fe79:bea2:19885 should fail | 18:37 |
| clarkb | rtheis: maybe? it would be in the devstack log if so | 18:37 |
| rtheis | ack | 18:38 |
| clarkb | rtheis: in any case it should be pretty easy to reproduce. Get an instance with ipv6 route provided via RA and run the test on it | 18:38 |
| clarkb | the result should be the route goes away if you set forwarding = 1 | 18:38 |
| haleyb | rtheis: see my email to the dev list on wednesday, or find the patch in devstack for more info | 18:38 |
| clarkb | unless you set accept_ra = 2 on the interface | 18:38 |
| *** ilyashakhat has joined #openstack-infra | 18:39 | |
| rtheis | clarkb, haleyb: thank you | 18:39 |
| * kambiz shakes fist at pabelanger | 18:39 | |
| *** amotoki has quit IRC | 18:40 | |
| *** piet has quit IRC | 18:40 | |
| kambiz | pabelanger: that's for asking a very sensible question of asking for a network diagram. If you only knew how messy it is. | 18:40 |
| *** tkelsey has quit IRC | 18:41 | |
| *** piet has joined #openstack-infra | 18:41 | |
| *** mordred has joined #openstack-infra | 18:41 | |
| clarkb | I am going to work on cleaning up leaked iamges in our clouds while waiting for mordred to come back | 18:41 |
| pabelanger | kambiz: Ya, I've asked that question about 15 times :) | 18:42 |
| mordred | heya | 18:42 |
| kambiz | pabelanger: lol I guess I might have only seen that for the first time in that thread from hell. (sorry) | 18:42 |
| mordred | pabelanger: the answer is "green" | 18:42 |
| clarkb | mordred: did you see my message about how the private IP looks good on current osic instances? nodepoold hasn't restarted since teh 12th so no fixes are in that I know of | 18:42 |
| *** yamamoto has joined #openstack-infra | 18:43 | |
| clarkb | mordred: so I guess the question is if nodepool restarted right now would it continue to work like that or is it new behavior that is going to break us until your fixes happen | 18:43 |
| pabelanger | kambiz: Ya, not no progblems. I'm still trying to navigate how that whole system works | 18:43 |
| mordred | clarkb: that is a great question. let me run some tests against osic | 18:43 |
| clarkb | mordred: beacuse i would like to run multinode tests there considering how much quota it represents :) | 18:43 |
| clarkb | mordred: great thanks. I will continue to clean up leaked images in the mean time | 18:44 |
| *** smarcet1981 has joined #openstack-infra | 18:44 | |
| *** asettle has quit IRC | 18:45 | |
| mordred | clarkb: we run nodepool on releases, right? | 18:45 |
| mordred | (of shadE) | 18:45 |
| *** smarcet has quit IRC | 18:46 | |
| clarkb | mordred: let me double check but pretty sure we do | 18:46 |
| clarkb | mordred: shade==1.11.1 # git sha 273ea8c is what we have installed now | 18:46 |
| clarkb | we have leaked 26 images on osic compared to the 290 in ovh-bhs1 fwiw | 18:47 |
| clarkb | so definitely something going on to make it so much worse in ovh | 18:47 |
| mordred | cool | 18:47 |
| *** Sukhdev has joined #openstack-infra | 18:47 | |
| *** csomerville has joined #openstack-infra | 18:48 | |
| mordred | clarkb: with that same ref locally, I do not show private_v4 addresses being appropriately returned by shade | 18:48 |
| mordred | clarkb: where are you looking to see that they are working as expected? | 18:48 |
| *** Apoorva has joined #openstack-infra | 18:48 | |
| clarkb | mordred: I logged into the instances and checked their files in /etc/nodepool (which is what the multinode tests depend on) | 18:49 |
| *** yamamoto has quit IRC | 18:49 | |
| *** eharney has joined #openstack-infra | 18:49 | |
| *** ociuhandu has joined #openstack-infra | 18:50 | |
| mordred | clarkb: oh - well, it's working accidentally | 18:50 |
| mordred | clarkb: self.subnode.ip_private = server.get('private_v4') | 18:50 |
| mordred | if not self.subnode.ip_private: | 18:50 |
| mordred | self.subnode.ip_private = server.get('public_v4') | 18:50 |
| clarkb | thats actually intentional | 18:50 |
| YorikSar | Hello, Infra masters. Is there any way to make jobs in Zuul start not in the root of the repo's work dir? | 18:51 |
| clarkb | but I guess an accident that it handles this case too | 18:51 |
| mordred | yah | 18:51 |
| mordred | the server record says: | 18:51 |
| mordred | 'private_v4': '', | 18:51 |
| mordred | 'public_v4': u'10.199.169.53', | 18:51 |
| clarkb | mordred: so am I ok to reenable multinode there? | 18:51 |
| mordred | 'public_v6': u'2001:4800:1ae1:18:f816:3eff:fec7:88b5', | 18:51 |
| mordred | clarkb: I think so | 18:51 |
| *** zul has quit IRC | 18:51 | |
| clarkb | mordred: given the accident should be deterministic? ok | 18:51 |
| mordred | clarkb: I thought originally that we had determined this was the case but you still thought there was a problem for reasons I forget | 18:51 |
| YorikSar | We have puppet pieces stuck in a subdir in the repo and want to lint it: https://git.openstack.org/cgit/openstack/fuel-octane/tree/deployment/puppet/octane_tasks | 18:52 |
| clarkb | YorikSar: yes, you can edit your zuul cloner settings for the job | 18:52 |
| *** kzaitsev_mb has joined #openstack-infra | 18:52 | |
| clarkb | YorikSar: or just cd :) | 18:52 |
| mordred | clarkb: but it took you a few minutes to come to the conclusion that the settings going into the files wouldn't work and that we needed to fix them to be correct | 18:52 |
| clarkb | mordred: oh I thought you said the public v6 addr was going into the file | 18:52 |
| mordred | ah. nope | 18:52 |
| clarkb | mordred: maybe you meant public v4 and I misparsed | 18:52 |
| *** Apoorva has quit IRC | 18:52 | |
| mordred | the actual private address, which becomes server['public_v4'] will go into the file as both public and private | 18:53 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes https://review.openstack.org/361164 | 18:53 |
| YorikSar | clarkb: Well, I hope to reuse existing macros and job templates as much as possible, so sticking cd in there is not an easy task. | 18:53 |
| mordred | so I agree - I think it's fine for now, and we'll know pretty immediately if we're wrong | 18:53 |
| YorikSar | clarkb: I'll loog into cloner cfg then. | 18:53 |
| mtreinish | mordred, fungi, nibalizer, clarkb: if you get a sec can you check health.o.o it's being quite sluggish and the ram usage is quite high: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2860&rra_id=all | 18:54 |
| mgagne | pabelanger: I think mtl01 will have to wait until Monday (at least), leaving ops meetup soon so won't have connectivity with our ops team until ~ Monday. | 18:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Revert "Remove OSIC from 2-node label types" https://review.openstack.org/361421 | 18:56 |
| mgagne | pabelanger: will poke back if I get more updates | 18:56 |
| *** tqtran has quit IRC | 18:56 | |
| clarkb | mordred: pabelanger ^ that should be ready to go if you can review quickly | 18:56 |
| *** ociuhandu has quit IRC | 18:56 | |
| clarkb | mgagne: roger | 18:56 |
| clarkb | mtreinish: ya give me a sec to find a good stopping place on this leaked image cleanup | 18:57 |
| *** e0ne has joined #openstack-infra | 18:57 | |
| *** markvoelker has quit IRC | 18:57 | |
| *** tqtran has joined #openstack-infra | 18:57 | |
| pabelanger | mgagne: great, thanks again | 18:57 |
| pabelanger | clarkb: +3 | 18:58 |
| *** phschwartz has joined #openstack-infra | 18:58 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: doc: add Storyboard integration info https://review.openstack.org/361423 | 18:58 |
| *** sdague has quit IRC | 18:59 | |
| clarkb | pabelanger: danke | 18:59 |
| clarkb | mtreinish: top says an apache is using all the cpu and much of the memory | 18:59 |
| *** markvoelker has joined #openstack-infra | 19:00 | |
| * clarkb tries to figure out what is making it unhappy | 19:00 | |
| *** rhallisey has joined #openstack-infra | 19:00 | |
| *** piet has quit IRC | 19:00 | |
| *** piet has joined #openstack-infra | 19:01 | |
| clarkb | mtreinish: there is a lot of "[Fri Aug 26 17:29:17.078766 2016] [:error] [pid 17723:tid 140502063863552] OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')" in the error logs | 19:01 |
| YorikSar | clarkb: Looks like Cloner is not a way to go: it seems to operate with repos as whole only. | 19:01 |
| clarkb | YorikSar: our defautl setup for zuul cloner is to clone into . though | 19:02 |
| clarkb | YorikSar: so if you want to chagne it to clone somewhere else you will have to udpate the config for it | 19:02 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: doc: add Storyboard integration info https://review.openstack.org/361423 | 19:02 |
| YorikSar | clarkb: I basically need to have "cd deployment/puppet/octane_tasks" after cloner finishes, but before the job is started. | 19:03 |
| clarkb | oh you want to be further into the repo not out of the repo entirely | 19:03 |
| *** rbrndt has joined #openstack-infra | 19:03 | |
| YorikSar | Yeah | 19:03 |
| clarkb | yes for that I think you need to use cd or tell your linter where to lint from. pep8 can do this iirc not sure about puppet lint | 19:03 |
| *** Shrews has joined #openstack-infra | 19:04 | |
| mtreinish | clarkb: hmm, so it's dropping connection to the trove node? | 19:05 |
| mtreinish | clarkb: is it getting OOM'ed or something | 19:05 |
| YorikSar | I guess I need to add some parameter to the puppet-syntax/puppet-lint builder and add 'if [ {start_from_dir}]; then cd {start_from_dir}; fi' to the beginning of their scripts. | 19:05 |
| clarkb | mtreinish: I don't see any recent OOMs on health | 19:06 |
| *** sarob has quit IRC | 19:06 | |
| clarkb | mtreinish: I haven't checked teh db side yet as I am still looking for the config file for healt so I can mysql client to the db | 19:06 |
| *** ociuhandu has joined #openstack-infra | 19:06 | |
| clarkb | YorikSar: or see if you can just pass the dir to lint as a parameter? | 19:07 |
| *** vhosakot has quit IRC | 19:07 | |
| YorikSar | clarkb: Is there any way to pass down parameters to builders from project.yaml level? I mean, to pass them down to all builders in all jobs in the job-group. | 19:07 |
| mtreinish | clarkb: I don't know the root password or user but you can get at it via the logstash.o.o proxy | 19:07 |
| mtreinish | clarkb: it should be /etc/openstack-health* something | 19:07 |
| mtreinish | maybe openstack-health-api.conf | 19:08 |
| clarkb | no there isn't any /etc/openstack-health. The readme says to pass the path but our apache wsgi stuff doesn't so must be whatever is default | 19:08 |
| *** csomerville has quit IRC | 19:08 | |
| YorikSar | clarkb: Oh, it hickups on 'bundle install' (no Gemfile in the root of the repo) or on 'rake line' (no Rakefile in the root of the repo) | 19:08 |
| mtreinish | clarkb: it should be '/etc/openstack-health.conf' | 19:09 |
| YorikSar | *line=lint | 19:09 |
| mtreinish | that's the default path in code | 19:09 |
| clarkb | ah its the file not the dir found it | 19:09 |
| clarkb | YorikSar: you could add those to the root of the repo? | 19:09 |
| pabelanger | YorikSar: clarkb: I would write a builder specific to fuel-octane then work to migrate the job to use the default puppet-lint jobs. EmilienM has a good job making those jobs pretty generic today, and trying to shoehorn other project layouts doesn't seem like the best approach | 19:09 |
| *** salv-orlando has quit IRC | 19:10 | |
| EmilienM | pabelanger, YorikSar: where is the patch? | 19:10 |
| *** piet has quit IRC | 19:11 | |
| YorikSar | clarkb: I don't think that will help: Gemfile should be in the root of the gem afaiu... | 19:11 |
| pabelanger | EmilienM: https://git.openstack.org/cgit/openstack/fuel-octane/tree/deployment/puppet/octane_tasks was the module in question which fails to puppet-lint because of the location of the directory | 19:11 |
| *** piet has joined #openstack-infra | 19:11 | |
| EmilienM | oh dear | 19:11 |
| pabelanger | yes | 19:12 |
| EmilienM | a puppet module in a repo | 19:12 |
| YorikSar | pabelanger, EmilienM: Yeah, both puppet-lint and puppet-syntax jobs hickup. | 19:12 |
| EmilienM | this day started well | 19:12 |
| EmilienM | YorikSar: adding a puppet module in a directory within a repo is a terrible idea | 19:12 |
| YorikSar | EmilienM: It's complicated :) It's actually not a module for octane, it's a part of octane :) | 19:12 |
| clarkb | mtreinish: ok I am connected to the db so its up now at least. Let me see if I can ask mysql for more running infos | 19:12 |
| YorikSar | EmilienM: Like a part of the tool is written in Puppet. | 19:13 |
| mtreinish | clarkb: I logged into the mysql server too and it's also really slow | 19:13 |
| YorikSar | EmilienM: (is going to, we're experimenting with that) | 19:13 |
| mtreinish | err, maybe only somewhat slow | 19:13 |
| EmilienM | YorikSar: you should have create a module called openstack/puppet-octane | 19:13 |
| EmilienM | YorikSar: that is how OpenStack is doing Puppet modules, we have a forge with nearly 40 puppet modules | 19:14 |
| mtreinish | nm, it's working fine it's probably just my network here | 19:14 |
| clarkb | mtreinish: | Connections | 2037348 | not sure if that number is cummulative or point int time | 19:14 |
| EmilienM | YorikSar: I don't see why Fuel would make exceptions here | 19:14 |
| YorikSar | EmilienM: afaiu 'puppet-SOMENAME' is a puppet module to setup software SOMENAME. We don't have the same relation between octane and octane_tasks. | 19:15 |
| clarkb | mtreinish: I dont' really see anything else in show status that looks odd | 19:15 |
| mtreinish | clarkb: iirc connections is the total connection attempts | 19:15 |
| YorikSar | EmilienM: octane_tasks contains tasks with Puppet modules that are called from within octane. | 19:16 |
| mtreinish | threads_connected is the active one | 19:16 |
| clarkb | | Aborted_clients | 39362 | | 19:16 |
| EmilienM | YorikSar: you lost me | 19:16 |
| mtreinish | but it's been a while since I looked that up in the docs | 19:16 |
| EmilienM | but as a good news, I just understans the fuel/octane wording :) really fun | 19:16 |
| clarkb | we have 10 threads connected | 19:16 |
| mtreinish | clarkb: I'm fine just restarting the api server on health.o.o and hoping this doesn't happen again | 19:16 |
| openstackgerrit | James Slagle proposed openstack-infra/project-config: Add new promote jobs for TripleO https://review.openstack.org/361427 | 19:17 |
| *** ociuhandu has quit IRC | 19:17 | |
| clarkb | mtreinish: ok let me start with that | 19:17 |
| YorikSar | EmilienM: Octane does some deployment stuff that touches Fuel itself and OpenStack deployed by Fuel. Everything is currently written in Python, but we would like to move away from that by calling out from Python to Fuel that would run some Puppet code on OpenStack nodes (or Fuel node). | 19:18 |
| clarkb | mtreinish: thats done | 19:18 |
| mtreinish | clarkb: ok thanks, let me query it | 19:18 |
| YorikSar | EmilienM: So basically we have pieces of Python code within Octane tool being replaced with Puppet code. | 19:18 |
| clarkb | #status log restarted apache2 on health.o.o to remove a runaway apache process using all the cpu and memory. Looked like it may be related to mysql connections issues. DB currently looks happy. | 19:19 |
| openstackstatus | clarkb: finished logging | 19:19 |
| mtreinish | clarkb: yep it's responding now: http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern renders things again | 19:20 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Support promoting based on $TOCI_JOBTYPE https://review.openstack.org/361429 | 19:20 |
| *** rodrigods has quit IRC | 19:21 | |
| *** rodrigods has joined #openstack-infra | 19:21 | |
| *** ociuhandu has joined #openstack-infra | 19:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Remove OSIC from 2-node label types" https://review.openstack.org/361421 | 19:21 |
| *** senk has joined #openstack-infra | 19:22 | |
| YorikSar | EmilienM: BTW, running tests from a subdir might be a good feature for generic test running: we shouldn't force projects to be split by the test boundaries. | 19:22 |
| *** sridhar_ram is now known as sridhar_ram_afk | 19:22 | |
| *** ilyashakhat has quit IRC | 19:23 | |
| *** Na3iL has joined #openstack-infra | 19:24 | |
| *** raunak has quit IRC | 19:25 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/reviewday: Make neutron script pull all Stadium projects https://review.openstack.org/361432 | 19:25 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo: align CI scenarios with Puppet CI https://review.openstack.org/361433 | 19:25 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/reviewday: Make neutron script pull all Stadium projects https://review.openstack.org/361432 | 19:25 |
| EmilienM | YorikSar: I don't think we need this feature | 19:25 |
| clarkb | the way it works in python land is you just edit your tox.ini | 19:26 |
| EmilienM | YorikSar: it's encouraging people to add puppet modules in their repo while it's extremely bad practise | 19:26 |
| clarkb | is there no equivalent to that in your rakefile? | 19:26 |
| EmilienM | YorikSar: I don't support this idea. | 19:26 |
| clarkb | then all the jobs have to do is run rake and not care about specifics | 19:26 |
| pabelanger | YorikSar: EmilienM: Agreed, I still breaking that puppet task to an example repo is the correct thing to do. You don't need to call it puppet-octane, it could be puppet-octane_tasks. And there is not hard rule to say how you are using puppet can't be done in an external module | 19:26 |
| pabelanger | YorikSar: EmilienM: then close the repo locally, or puppet install (cannot remember syntax) and use the --modules_path flag | 19:27 |
| EmilienM | pabelanger: agree | 19:30 |
| *** e0ne has quit IRC | 19:30 | |
| EmilienM | YorikSar: it's like asking running tox inside a project directory | 19:32 |
| EmilienM | does osinfra allows it? | 19:32 |
| YorikSar | EmilienM: I think it should :) | 19:33 |
| YorikSar | EmilienM: Google has one huge repo for everything they have after all, so putting more than one things in one repo shouldn't be the end of the world. | 19:33 |
| *** links has quit IRC | 19:33 | |
| EmilienM | I didn't know we were Google | 19:34 |
| EmilienM | anyway, you're free to propose an infra patch to be able to run lint tests using our builder | 19:34 |
| YorikSar | EmilienM: btw, do you have smth for me to read on about why is it a bad practise? | 19:35 |
| YorikSar | EmilienM: you mean puppet-lint? | 19:35 |
| EmilienM | YorikSar: I've been working on Puppet for several years, have been leading Puppet work in OpenStack and also core contributor in Puppetlabs modules, I haven't seen this practise before. | 19:36 |
| EmilienM | YorikSar: yes lint | 19:36 |
| *** senk has quit IRC | 19:38 | |
| clarkb | no we expect the top level to drive whatever is under it | 19:40 |
| clarkb | not for the jobs to magically know how to tox somewhere lese | 19:40 |
| pabelanger | ++ | 19:40 |
| YorikSar | EmilienM: While I'm still not convinced why having Puppet code in Python repo, we can't do that split right now. So we'd like to have an option to run linters in subdir at least as an interim solution. | 19:41 |
| *** gordc has quit IRC | 19:41 | |
| *** tqtran has quit IRC | 19:42 | |
| clarkb | like I said, could you put the rakefiel thing at the top level dir just like the tox.ini? | 19:42 |
| YorikSar | clarkb, pabelanger: so if I propose a change that adds parametrised 'cd' at the top of puppet-lint builders, you all will -1 it? | 19:42 |
| clarkb | and have it execute only on the inner dir? | 19:42 |
| YorikSar | clarkb: It will work for rake, it won't work for Gemfile afaiu. | 19:42 |
| *** tqtran has joined #openstack-infra | 19:42 | |
| *** dprince has joined #openstack-infra | 19:43 | |
| clarkb | but rake is what the job runs right? | 19:43 |
| clarkb | rake lint or something iirc | 19:43 |
| YorikSar | clarkb: and puppet-syntax builder does 'bundle install' in the root of the repo prior to everything. | 19:43 |
| *** esikachev has joined #openstack-infra | 19:43 | |
| YorikSar | clarkb: Yes, puppet-lint builder does that. | 19:44 |
| pabelanger | YorikSar: I'd rather see you write your own builder for fuel-octane then force the existing puppet-lint job to handle it. When work on migrating when you are able too | 19:44 |
| *** gyee has joined #openstack-infra | 19:44 | |
| pabelanger | But, I feel you could setup a Rakefile to hand the subdirectory | 19:45 |
| pabelanger | https://github.com/rodjek/puppet-lint | 19:45 |
| clarkb | osic and ovh have had their images cleaned. Internap is running now | 19:46 |
| pabelanger | clarkb: thanks! | 19:46 |
| YorikSar | pabelanger: It won't work for puppet-syntax builder. | 19:46 |
| clarkb | mordred: I think some of these leaks may happen when we restart the build when it is uploading | 19:46 |
| pabelanger | clarkb: I noticed ovh-bhs1 is still running an old ubuntu-xenial image | 19:46 |
| clarkb | mordred: FYI for when you want to debug it more | 19:46 |
| clarkb | pabelanger: ya 10 days or something | 19:46 |
| clarkb | pabelanger: it is having the most trouble uploading of all the regions. Hoping today's uplaods go better for it | 19:47 |
| pabelanger | YorikSar: maybe us gate-puppet-openstack-libraries-puppet-syntax | 19:47 |
| pabelanger | or gate-puppet-openstack-libraries-puppet-lint | 19:47 |
| YorikSar | pabelanger: Thanks, I'll take a look at them. | 19:48 |
| pabelanger | gate-{name}-puppet-syntax-{puppet_version}-{node} should even work with the right Rakefile | 19:48 |
| *** HeOS has joined #openstack-infra | 19:48 | |
| pabelanger | clarkb: cool, that seems to be the only failures with configure_mirror.sh recently | 19:49 |
| *** sridhar_ram_afk is now known as sridhar_ram | 19:49 | |
| pabelanger | clarkb: just seen an osic-cloud1 failure | 19:49 |
| *** raildo has joined #openstack-infra | 19:49 | |
| cloudnull | oh, noes | 19:49 |
| pabelanger | http://paste.openstack.org/show/564050/ | 19:50 |
| pabelanger | was a subnode | 19:50 |
| *** berendt has joined #openstack-infra | 19:50 | |
| pabelanger | should have worked | 19:50 |
| pabelanger | going to test | 19:50 |
| cloudnull | ah not specifically an issue with the cloud itself. | 19:51 |
| pabelanger | cloudnull: :) | 19:51 |
| * cloudnull was thinking things were on fire | 19:51 | |
| clarkb | pabelanger: weird, I think maybe its possible that that ran prior to iptables ruels being applied out of the persistent thing? | 19:51 |
| pabelanger | clarkb: ya, maybe | 19:51 |
| clarkb | pabelanger: or possibly related to firewalld and maybe we have that breaking stuff on centos7 for us again | 19:51 |
| pabelanger | let me see if we launch a ubuntu one | 19:52 |
| pabelanger | 2001:4800:1ae1:18:f816:3eff:fe46:193c should be ubuntu-xenial-2-node | 19:53 |
| pabelanger | http://paste.openstack.org/show/564052/ | 19:53 |
| clarkb | you can boot the centos7 image and check the same thing | 19:56 |
| clarkb | I think it could either be a race or it is possibly firewalld coming to visit again | 19:57 |
| *** esikache1 has joined #openstack-infra | 19:57 | |
| pabelanger | ya, have a system online now | 19:58 |
| *** asettle has joined #openstack-infra | 19:59 | |
| *** asettle has quit IRC | 19:59 | |
| *** kaisers_ has joined #openstack-infra | 19:59 | |
| *** esikachev has quit IRC | 19:59 | |
| *** flaviof has quit IRC | 20:01 | |
| *** fdegir has quit IRC | 20:01 | |
| *** tdurakov has quit IRC | 20:01 | |
| *** gouthamr has quit IRC | 20:01 | |
| *** ashtokolov has quit IRC | 20:01 | |
| *** scottda has quit IRC | 20:01 | |
| *** simonmcc has quit IRC | 20:01 | |
| *** vdrok has quit IRC | 20:01 | |
| pabelanger | clarkb: looks like ip6tables is its own service for centos-7 | 20:02 |
| *** jraim has quit IRC | 20:02 | |
| pabelanger | which isn't running | 20:02 |
| *** esikachev has joined #openstack-infra | 20:02 | |
| *** betherly has quit IRC | 20:02 | |
| *** maximov has quit IRC | 20:02 | |
| *** ameade has quit IRC | 20:02 | |
| fungi | that's wacky | 20:02 |
| pabelanger | yup | 20:02 |
| *** tpsilva has quit IRC | 20:02 | |
| *** skath has quit IRC | 20:02 | |
| *** zhiyan has quit IRC | 20:03 | |
| pabelanger | once enabled are rebooted, ip6tables -L works | 20:03 |
| clarkb | huh | 20:03 |
| pabelanger | well, our opentack-INPUT label is added | 20:03 |
| *** esikache1 has quit IRC | 20:03 | |
| *** aimeeu has quit IRC | 20:04 | |
| *** kaisers_ has quit IRC | 20:04 | |
| clarkb | bluebox leaked images are clearing out now | 20:04 |
| *** rvasilets___ has quit IRC | 20:04 | |
| *** serverascode has quit IRC | 20:04 | |
| *** NobodyCam has quit IRC | 20:04 | |
| *** rpodolyaka has quit IRC | 20:04 | |
| *** sridhar_ram has quit IRC | 20:05 | |
| *** TheJulia has quit IRC | 20:07 | |
| clarkb | jeblair: is it ok to delete things like jeblair-devstack-precise-1373496311.template.openstack.org from rax ord? | 20:07 |
| marcusvrn_ | asselin: hi! It worked (about clean caches and env vars and re-build an image), now nodepool can create and access instances in OpenStack. Now I think the last problem that I'm facing is that safe-devstack-vm-gate-wrap.sh is failing...I'm trying to debug it, but maybe you or someone else had faced or know what is going on. The script stop trying to setup | 20:08 |
| marcusvrn_ | the workspace: http://paste.openstack.org/show/564054/ and console log: http://paste.openstack.org/show/564056/ | 20:08 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: doc: add Storyboard integration info https://review.openstack.org/361423 | 20:09 |
| jeblair | clarkb: i'm going to say "yes" and "please don't look at the date" | 20:10 |
| clarkb | jeblair: :) ok | 20:10 |
| clarkb | jeblair: there are a few of them I assume realted to nodepool testing at one time | 20:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-iptables: Add ip6tables service support for centos-7 https://review.openstack.org/361449 | 20:11 |
| jeblair | even though the time is right there in the name. it's hard to miss. | 20:11 |
| pabelanger | clarkb: fungi: ^ should be the magic we need | 20:11 |
| * clarkb looks before taking on rax image leaks | 20:11 | |
| jeblair | i'll just give a hint -- the *current* time starts with 14..., not 13... | 20:12 |
| *** tonytan_brb has joined #openstack-infra | 20:13 | |
| fungi | clarkb: `date -d @1373496311` says "Wed Jul 10 22:45:11 UTC 2013" | 20:13 |
| pabelanger | woah | 20:13 |
| fungi | so, um... really, really old | 20:13 |
| fungi | aldo, date -d @ is super useful for turning epoch time into something human brainable | 20:14 |
| pabelanger | might been our longest lived server | 20:14 |
| fungi | s/aldo/also/ | 20:14 |
| jeblair | pabelanger: just an image | 20:14 |
| pabelanger | ah | 20:14 |
| fungi | a ghostly, ghostly image of a server that once was... | 20:14 |
| clarkb | pabelanger: commented just one thing about how puppet is silly that will likely make that fail as is :) | 20:15 |
| *** roxanaghe has quit IRC | 20:15 | |
| clarkb | jeblair: its also a gnu extension iirc. We had to use perl on the old solaris machines in order to manipulate unix time | 20:15 |
| clarkb | jeblair: there are ~6 images like that. I am guessing youdon't need any of them | 20:16 |
| jeblair | clarkb: probably that was addressed to fungi. but i think gnu extensions are the best extensions. | 20:16 |
| jeblair | clarkb: correct | 20:16 |
| *** tonytan4ever has quit IRC | 20:16 | |
| clarkb | oh hrm they have a queued state. I wonder if that has made them undeletable | 20:17 |
| fungi | gnu's not unix, and sometimes it's not even posix, but it's still awesome | 20:17 |
| clarkb | I will probably find out soon :) | 20:17 |
| jeblair | clarkb: oh good, there's an excuse for why i left them laying around | 20:17 |
| fungi | clarkb: openbsd's date function has a similar option, i just forget what it is off the top of my head | 20:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Also use the virtualenv for checking https://review.openstack.org/361452 | 20:18 |
| clarkb | fungi: in particular we wanted to compare deltas between file mode times and the time now and it just got easer to do the whole thing in perl without gnu date | 20:18 |
| mordred | jeblair, clarkb: ^^ sigh. one more patch related to the nodepool-in-venv thing we did the other day | 20:18 |
| fungi | clarkb: on openbsd it's `date -r 1373496311` | 20:18 |
| clarkb | fungi: the worst part of it was we had gnu date compiled for sparc but not for x86 and needed ti on both :/ | 20:18 |
| *** Goneri has quit IRC | 20:19 | |
| *** markusry has quit IRC | 20:20 | |
| fungi | hrm... i need to disappear again for a bit but will bbiaw | 20:21 |
| *** hashar has joined #openstack-infra | 20:21 | |
| *** cody-somerville has joined #openstack-infra | 20:23 | |
| *** cody-somerville has joined #openstack-infra | 20:23 | |
| *** kgiusti has left #openstack-infra | 20:23 | |
| armax | hi folks any idea why I don’t see the test_results file on a failure like this one? | 20:23 |
| armax | http://logs.openstack.org/04/353704/9/gate/gate-tempest-dsvm-neutron-linuxbridge/e6ca9f3/console.html | 20:23 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes https://review.openstack.org/361164 | 20:23 |
| *** roxanaghe has joined #openstack-infra | 20:23 | |
| *** gouthamr has joined #openstack-infra | 20:23 | |
| clarkb | armax: what is the test_results file? | 20:25 |
| clarkb | I am not familiar with that one for d-g runs | 20:25 |
| *** e0ne has joined #openstack-infra | 20:26 | |
| armax | clarkb: sorry I meant this testr_results.html.gz | 20:26 |
| *** raunak has joined #openstack-infra | 20:26 | |
| *** _ari_ has joined #openstack-infra | 20:27 | |
| *** eranrom has joined #openstack-infra | 20:29 | |
| mordred | clarkb: SOOOOOOO | 20:29 |
| *** cody-somerville has quit IRC | 20:29 | |
| mordred | clarkb: we are somehow returning None from create_image sometimes | 20:31 |
| *** abregman has quit IRC | 20:31 | |
| clarkb | armax: http://logs.openstack.org/04/353704/9/gate/gate-tempest-dsvm-neutron-linuxbridge/e6ca9f3/logs/devstack-gate-cleanup-host.txt it tries to there at least | 20:32 |
| clarkb | mordred: byt cloud side iy actually made an image? | 20:33 |
| mordred | clarkb: well... that I don't know. but that method is not supposed to return None | 20:33 |
| mordred | so I'm tracking down how that's possible first | 20:34 |
| pabelanger | clarkb: looking | 20:34 |
| *** Jeffrey4l_ has joined #openstack-infra | 20:34 | |
| mordred | however, it could, just by itself, explain the leak | 20:34 |
| *** dkehn_ has quit IRC | 20:34 | |
| mordred | bcause it then throws an exception in nodepool on trying to access image.id (no attribute 'id' on NoneType) | 20:34 |
| mordred | so from nodepool's pov it's a failed call to uploadImage | 20:35 |
| clarkb | mordred: ya | 20:35 |
| *** Jeffrey4l__ has quit IRC | 20:35 | |
| *** jraim has joined #openstack-infra | 20:35 | |
| clarkb | armax: I can look more after lunch | 20:35 |
| clarkb | pabelanger: at which thing? | 20:35 |
| mordred | this is also awesome: | 20:35 |
| mordred | 2016-08-25 20:29:14,655 ERROR shade.exc: Image creation failed: 404 Not Found: Failed to find image af05cf54-af45-4889-a043-96675f014b20 to delete (HTTP 404) | 20:35 |
| armax | clarkb: no rush | 20:35 |
| armax | thanks | 20:35 |
| armax | I am not quite sure what’s going on | 20:36 |
| mordred | "Image creation failed, couldn't find image to delete" | 20:36 |
| clarkb | mordred: its probably trying to cleanuo after itself due ti the two step image creat process | 20:36 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/elastic-recheck: Add E-R query for bug 1617447 https://review.openstack.org/361458 | 20:37 |
| openstack | bug 1617447 in OpenStack Compute (nova) "InterfaceAttachFailed: Failed to attach network adapter device" [High,New] https://launchpad.net/bugs/1617447 | 20:37 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add networking-cumulus project https://review.openstack.org/354784 | 20:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-iptables: Add ip6tables service support for centos-7 https://review.openstack.org/361449 | 20:37 |
| pabelanger | clarkb: comments on ^ | 20:37 |
| *** simonmcc has joined #openstack-infra | 20:39 | |
| *** fguillot has quit IRC | 20:39 | |
| *** _ari_ has quit IRC | 20:40 | |
| *** _ari_ has joined #openstack-infra | 20:40 | |
| *** serverascode has joined #openstack-infra | 20:42 | |
| *** ameade has joined #openstack-infra | 20:43 | |
| *** zhiyan has joined #openstack-infra | 20:45 | |
| *** skath has joined #openstack-infra | 20:45 | |
| *** roxanaghe has quit IRC | 20:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets https://review.openstack.org/361463 | 20:47 |
| *** sridhar_ram has joined #openstack-infra | 20:48 | |
| *** rvasilets___ has joined #openstack-infra | 20:50 | |
| *** edtubill has quit IRC | 20:50 | |
| *** smarcet1981 has quit IRC | 20:50 | |
| *** ilyashakhat has joined #openstack-infra | 20:51 | |
| *** fdegir has joined #openstack-infra | 20:52 | |
| *** sc` has quit IRC | 20:52 | |
| *** sc` has joined #openstack-infra | 20:53 | |
| *** scottda has joined #openstack-infra | 20:54 | |
| *** jkilpatr has quit IRC | 20:54 | |
| mordred | clarkb: I'm going to need to upload an image by hand ... you ok with me doing that now? | 20:54 |
| pabelanger | https://review.openstack.org/#/c/318714/ might be interesting to folks following a long with AFS. Updates devstack-gate to use our AFS mirror for network_sanity_check, and not just rackspace | 20:55 |
| openstackgerrit | Merged openstack-infra/shade: Detect the need for FIPs better in auto_ip https://review.openstack.org/360771 | 20:56 |
| openstackgerrit | Merged openstack-infra/shade: Log request ids https://review.openstack.org/361157 | 20:56 |
| asselin__ | marcusvrn_, not sure....you'll have to trace it back. Here's where it's failing: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n511 | 20:56 |
| jeblair | pabelanger: +2 | 20:57 |
| *** dprince has quit IRC | 20:57 | |
| clarkb | mordred: ya its fine, would be good if you can upload to not rax though | 20:58 |
| clarkb | I havent been able to clean rax yet got distracted by kids and lunch | 20:58 |
| mordred | clarkb: just gonna upload an image to ovh-bhs with name mttest | 20:58 |
| mordred | or testmt | 20:58 |
| mordred | or something | 20:58 |
| clarkb | mordred: thats fine | 20:58 |
| *** maximov has joined #openstack-infra | 20:59 | |
| *** florianf has quit IRC | 21:00 | |
| *** ashtokolov has joined #openstack-infra | 21:00 | |
| *** betherly has joined #openstack-infra | 21:00 | |
| *** clenimar has quit IRC | 21:01 | |
| *** cody-somerville has joined #openstack-infra | 21:01 | |
| *** sdake has joined #openstack-infra | 21:02 | |
| openstackgerrit | Merged openstack-infra/shade: Rename _get_free_fixed_port to _nat_destination_port https://review.openstack.org/361316 | 21:02 |
| *** salv-orlando has joined #openstack-infra | 21:03 | |
| *** tpsilva has joined #openstack-infra | 21:03 | |
| *** flaviof has joined #openstack-infra | 21:04 | |
| *** raildo has quit IRC | 21:06 | |
| *** sridhar_ram is now known as sridhar_ram_afk | 21:06 | |
| *** inc0 has quit IRC | 21:06 | |
| *** tdurakov has joined #openstack-infra | 21:06 | |
| *** rpodolyaka has joined #openstack-infra | 21:07 | |
| *** vdrok has joined #openstack-infra | 21:07 | |
| *** iurygregory has quit IRC | 21:07 | |
| *** thorst has quit IRC | 21:07 | |
| *** eharney has quit IRC | 21:08 | |
| *** NobodyCam has joined #openstack-infra | 21:08 | |
| *** AnarchyAo has joined #openstack-infra | 21:08 | |
| *** Goneri has joined #openstack-infra | 21:08 | |
| *** aimeeu has joined #openstack-infra | 21:08 | |
| mordred | I would like to announce that uploading giant image as part of a debug cycle is FUN | 21:09 |
| anteaya | thank you for announcing | 21:09 |
| *** TheJulia has joined #openstack-infra | 21:10 | |
| *** thiagop has quit IRC | 21:10 | |
| *** slagle has quit IRC | 21:12 | |
| *** matrohon has quit IRC | 21:12 | |
| clarkb | mordred: yes I concur and I think we should investigate ro cinder volumes once nova supports that (in maybe 3 years across all our clouds) | 21:14 |
| *** tqtran has quit IRC | 21:14 | |
| *** ilyashakhat has quit IRC | 21:15 | |
| *** salv-orl_ has joined #openstack-infra | 21:18 | |
| *** jkilpatr has joined #openstack-infra | 21:18 | |
| pleia2 | off to the dentist (woo), bbiab | 21:18 |
| *** e0ne has quit IRC | 21:18 | |
| mordred | clarkb: we could just have our "image upload" process be "boot a server, attach a volume, dd image to volume, detach volume" - then our boot process be "clone volume, boot from volume" | 21:20 |
| *** salv-orlando has quit IRC | 21:20 | |
| *** ldnunes has quit IRC | 21:20 | |
| jeblair | mordred: i think you'd need to do: "attach volume to nodepool builder, dd image to volume, detach volume" for the upload process | 21:22 |
| jeblair | mordred: that requires running a builder in each cloud | 21:23 |
| *** rtheis has quit IRC | 21:23 | |
| jeblair | mordred: but it avoids the network. if you do the thing you suggest, you're still transiting the network with the image | 21:23 |
| *** cody-somerville has quit IRC | 21:23 | |
| irtermite | Folks... I understand that was some misunderstanding about the comparison of a split finger from an electrical burn to a burnt hot dog yesterday. As I understand it, that was taken in a sexual manner and I am not quite sure how that was construed that way. But, there was no way that was my intension. Regardless, I apologize for the misunderstanding. Have a great weekend! | 21:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 21:24 |
| jeblair | mordred: unless, you're suggesting that glance is slow enough that just sftping an image onto a server would be faster? | 21:24 |
| irtermite | Never want to upset folks. You folks rock. Carry on! | 21:24 |
| mordred | jeblair: well, it might be more reliable | 21:25 |
| *** tqtran has joined #openstack-infra | 21:26 | |
| *** roxanaghe has joined #openstack-infra | 21:26 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds an api-ref publishing job for cinder - Block Storage API https://review.openstack.org/361475 | 21:27 |
| *** roxanaghe has quit IRC | 21:27 | |
| *** baoli_ has quit IRC | 21:28 | |
| *** roxanaghe has joined #openstack-infra | 21:29 | |
| *** piet has quit IRC | 21:29 | |
| *** abregman has joined #openstack-infra | 21:31 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds an api-ref publishing job for cinder - Block Storage API https://review.openstack.org/361475 | 21:31 |
| *** gouthamr_ has joined #openstack-infra | 21:33 | |
| *** gouthamr has quit IRC | 21:36 | |
| *** _ari_ has quit IRC | 21:37 | |
| clarkb | mordred: ya that would work but would require volume quota. I was thinking more of maimtain the job caches in a volume then use the new multiattach stuff going on in nova (which us not ready yet) to mount that fs on the volume many times read only | 21:46 |
| clarkb | and that volume could be a terabyte large without much issue | 21:46 |
| clarkb | (in theory) | 21:46 |
| mordred | ah | 21:47 |
| mordred | I grok what you are saying now | 21:47 |
| mordred | and yeah - if we're all still working on tech when that's in production at all of our clouds I'll be amazed | 21:47 |
| *** kaisers_ has joined #openstack-infra | 21:48 | |
| *** rhallisey has quit IRC | 21:48 | |
| *** Sukhdev has quit IRC | 21:50 | |
| *** Hal1 has joined #openstack-infra | 21:52 | |
| *** kaisers_ has quit IRC | 21:52 | |
| *** Swami has quit IRC | 21:53 | |
| *** gouthamr_ is now known as gouthamr | 21:57 | |
| *** xyang1 has quit IRC | 21:57 | |
| *** lascii is now known as alaski | 22:00 | |
| *** sdake_ has joined #openstack-infra | 22:01 | |
| SpamapS | hahaha | 22:01 |
| SpamapS | mordred: 2024 , the year of volume multi-attach | 22:01 |
| SpamapS | mordred: regarding your "dd to volume" .. why volume? Just dd to your root disk, and snapshot. | 22:01 |
| SpamapS | also known as "user mode glance" | 22:02 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Extract /var/log from all nodes https://review.openstack.org/361164 | 22:03 |
| *** sdake has quit IRC | 22:03 | |
| *** rlandy has quit IRC | 22:04 | |
| *** abregman has quit IRC | 22:04 | |
| fungi | mordred: jeblair: i don't know that writing from a virtual machine to an attached cinder volume _actually_ avoids "the network" since (in at least most implementations) cinder is using a network storage protocol (fc san, iscsi, et cetera) | 22:05 |
| fungi | fwiw, most of our cinder-related outages have actually been network outages | 22:06 |
| *** thorst has joined #openstack-infra | 22:07 | |
| clarkb | that would actually be the biggest drawback of my idea but we already fail when the network doesn't work so shouldn't make things fail more | 22:08 |
| *** thorst has quit IRC | 22:11 | |
| clarkb | I have started deleting the leaked images in rax-iad | 22:11 |
| clarkb | there are 252 | 22:12 |
| *** Hal1 has quit IRC | 22:12 | |
| *** dmsimard is now known as dmsimard|pto | 22:12 | |
| clarkb | and the first one hasn't finished yet... this may take a while | 22:14 |
| fungi | from what i've seen, they service their api calls (especially deletions) on a best-effort basis | 22:15 |
| fungi | you could just spam them, then come back in a bit and see how many are still showing up | 22:15 |
| clarkb | I am just running for entry in file do openstack image delete entry | 22:16 |
| clarkb | Failed to delete image with name or ID 'b727fa80-1ea6-4a89-aa56-c3bee86b2340': HTTPServiceUnavailable (HTTP 503) | 22:16 |
| clarkb | well then | 22:16 |
| fungi | all that needs is a & (and a sleep $n if you're feeling generous) | 22:16 |
| *** adrian_otto has joined #openstack-infra | 22:17 | |
| clarkb | ya that loosk consistent | 22:18 |
| clarkb | so maybe I don't get to delete iamges in rax-iad today | 22:18 |
| clarkb | mordred: ^ remember your tattoo idea? | 22:18 |
| *** fguillot has joined #openstack-infra | 22:18 | |
| *** hashar has quit IRC | 22:22 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring https://review.openstack.org/361185 | 22:23 |
| clarkb | armax: the reason it didn't create the testr html file is that there is no testrepository.subunit file which is raelly the subunit from devstack | 22:23 |
| armax | clarkb: ok | 22:23 |
| armax | I wonder why the subunit file is missing then | 22:24 |
| *** cody-somerville has joined #openstack-infra | 22:24 | |
| openstackgerrit | Kunal Mehta proposed openstack-infra/zuul: Don't merge queues if the common job is 'noop' https://review.openstack.org/361505 | 22:25 |
| clarkb | armax: I am not sure how that works in devstack btu will take a look | 22:25 |
| armax | clarkb: this feels like a regression of some sort | 22:26 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Correct path to Launchpad creds on signing node https://review.openstack.org/361506 | 22:26 |
| fungi | there's a simple fix for anyone looking to pad their review stats! ;) | 22:26 |
| *** dimtruck is now known as zz_dimtruck | 22:27 | |
| *** matt-borland has quit IRC | 22:28 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Correct path to Launchpad creds on signing node https://review.openstack.org/361506 | 22:29 |
| fungi | now with minor improvement | 22:29 |
| *** berendt has quit IRC | 22:30 | |
| *** tqtran has quit IRC | 22:30 | |
| *** cody-somerville has quit IRC | 22:31 | |
| clarkb | armax: looks like it should be handled by the exit trap btu I don't see that running | 22:31 |
| clarkb | maybe something is clearing it out? | 22:31 |
| mordred | SpamapS: snapshots are hard - there is a reason we stopped doing them :) | 22:32 |
| mordred | clarkb: btw - still uploading test image | 22:32 |
| armax | clarkb: I can’t think of anything, I wonder if this is something affecting the linuxbridge job only, but if I am not mistaken I have seen this elsewhere | 22:32 |
| clarkb | armax: ya I think this is likely a devstack bug | 22:33 |
| clarkb | fungi: I stopped my deletes and switched to shows and the first 4 iamges (which had attempted deletes) were no longer showable | 22:34 |
| clarkb | fungi: making me think its actually deleting them but also returning 500 errors | 22:34 |
| *** tqtran has joined #openstack-infra | 22:34 | |
| clarkb | \o/ | 22:34 |
| fungi | clarkb: ECLOUD | 22:34 |
| clarkb | so I uh restarted it and have my fingers crossed :) | 22:34 |
| *** tphummel has quit IRC | 22:35 | |
| SpamapS | mordred: I just thought that you stopped doing them because you were _CHICKEN_. That's right. YELLOW. | 22:35 |
| * SpamapS bocks | 22:35 | |
| * mordred hands SpamapS a bucket of corn | 22:35 | |
| jeblair | mmm. chicken. | 22:35 |
| *** berendt has joined #openstack-infra | 22:36 | |
| clarkb | now I know what I am having for dinner | 22:36 |
| fungi | a simple hyperchicken lawyer from a backwoods asteroid? | 22:36 |
| clarkb | armax: if you like I can file the bug against devstack | 22:36 |
| clarkb | armax: not sure if you are investigating more | 22:37 |
| armax | clarkb: I am not today, but I can follow up on Monday | 22:37 |
| armax | clarkb: I thought you perhaps knew what was happening | 22:37 |
| clarkb | fungi: confirmed. 71fa4b18-8a2c-4a83-aaa8-762dd85d5f5b got a 503 on deletion but now I can't show it. I was able to show it prior | 22:38 |
| fungi | son, as your lawyer, i declare y'all are in a 12-piece bucket o' trouble | 22:40 |
| *** itisha has quit IRC | 22:40 | |
| bkero | fungi++ | 22:42 |
| *** tphummel has joined #openstack-infra | 22:42 | |
| clarkb | armax: https://bugs.launchpad.net/devstack/+bug/1617476 | 22:42 |
| openstack | Launchpad bug 1617476 in devstack "devstack.subunit is not generated" [Undecided,New] | 22:42 |
| armax | clarkb: ack | 22:42 |
| armax | thanks | 22:42 |
| *** edmondsw has quit IRC | 22:44 | |
| *** Swami has joined #openstack-infra | 22:45 | |
| openstackgerrit | Ryan Brandt proposed openstack-infra/project-config: Enable java test as voting on monasca-api https://review.openstack.org/361511 | 22:45 |
| *** pt_15 has quit IRC | 22:46 | |
| clarkb | we still haev 45 image uploads running today | 22:47 |
| clarkb | the images all built by 11am though which is good | 22:47 |
| clarkb | looks like all the images that have completed for internap nyj so far have failed. trying to see where else things are failing | 22:49 |
| *** rbrndt has quit IRC | 22:49 | |
| *** kien-ha has joined #openstack-infra | 22:52 | |
| *** signed8bit is now known as signed8bit_Zzz | 22:54 | |
| clarkb | mgagne: http://paste.openstack.org/show/564092/ appears to be the underlying cause for the internap image upload failures | 22:54 |
| *** signed8bit_Zzz is now known as signed8bit | 22:56 | |
| *** simonmcc has quit IRC | 22:56 | |
| jlvillal | tonyb: On openstack/requirements and the "normalize" that is done. Shouldn't the items in global-requirements be run through that? So that when someone copies from global-requirements.txt the proposal bot doesn't end up proposing a normalized version a few days later. | 22:57 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/log_processor: Add a simple test for the worker https://review.openstack.org/361515 | 22:57 |
| jlvillal | tonyb: At least that seems to be what happened with me and changing the 'hacking' requirement in some of the ironic projects | 22:58 |
| jesusaur | clarkb: I finally got around to writing a test case for the log_processor worker ^^ | 22:58 |
| *** skath has quit IRC | 22:58 | |
| clarkb | jesusaur: neat | 22:58 |
| clarkb | jesusaur: is that self testing yet? I am pretty sure I reviewed the changes to add the test but not sure if it got merged | 22:58 |
| openstackgerrit | Merged openstack-infra/system-config: Correct path to Launchpad creds on signing node https://review.openstack.org/361506 | 22:58 |
| jesusaur | clarkb: you're probably thinking of the test case for the client that was added a while back, and yeah it's self-testing | 22:59 |
| jesusaur | clarkb: but we still haven't actually switched to that project yet | 22:59 |
| *** vdrok has quit IRC | 23:00 | |
| *** salv-orl_ has quit IRC | 23:00 | |
| clarkb | oh right this is the worker, we could probably turn that test into an integration test if we spun up a client, gearman, worker, and logstash | 23:00 |
| *** betherly has quit IRC | 23:00 | |
| *** TheJulia has quit IRC | 23:01 | |
| jesusaur | ya | 23:01 |
| *** serverascode has quit IRC | 23:01 | |
| *** Julien-zte has joined #openstack-infra | 23:01 | |
| *** aimeeu has quit IRC | 23:01 | |
| *** NobodyCam has quit IRC | 23:01 | |
| *** tdurakov has quit IRC | 23:01 | |
| *** flaviof has quit IRC | 23:02 | |
| *** zhiyan has quit IRC | 23:02 | |
| *** ashtokolov has quit IRC | 23:02 | |
| *** scottda has quit IRC | 23:02 | |
| fungi | jlvillal: probably, not only should changes to the global reqs list be run through that, they should be run through that on every proposed change and failed if the pre-normalized for doesn't match | 23:02 |
| *** rpodolyaka has quit IRC | 23:02 | |
| fungi | s/for/form/ | 23:03 |
| *** ameade has quit IRC | 23:03 | |
| *** tpsilva has quit IRC | 23:03 | |
| *** sridhar_ram_afk has quit IRC | 23:03 | |
| *** rvasilets___ has quit IRC | 23:04 | |
| *** maximov has quit IRC | 23:04 | |
| *** fdegir has quit IRC | 23:04 | |
| *** cody-somerville has joined #openstack-infra | 23:05 | |
| *** cody-somerville has joined #openstack-infra | 23:05 | |
| *** esikachev has quit IRC | 23:05 | |
| *** bapalm_ has quit IRC | 23:06 | |
| jlvillal | fungi: I proposed this: https://review.openstack.org/361517 Normalize the requirement files | 23:06 |
| jlvillal | fungi: But not the 2nd part you suggested | 23:06 |
| *** hongbin has quit IRC | 23:06 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for Groovy Label Assignment plugin https://review.openstack.org/355247 | 23:08 |
| *** jraim has quit IRC | 23:09 | |
| *** bapalm has joined #openstack-infra | 23:11 | |
| *** rpodolyaka has joined #openstack-infra | 23:15 | |
| *** TheJulia has joined #openstack-infra | 23:15 | |
| *** NobodyCam has joined #openstack-infra | 23:15 | |
| *** ashtokolov has joined #openstack-infra | 23:15 | |
| *** betherly has joined #openstack-infra | 23:15 | |
| *** vhosakot has joined #openstack-infra | 23:16 | |
| *** DericHorn-HP has quit IRC | 23:17 | |
| *** vdrok has joined #openstack-infra | 23:19 | |
| *** vhosakot has quit IRC | 23:19 | |
| *** serverascode has joined #openstack-infra | 23:20 | |
| *** sridhar_ram_afk has joined #openstack-infra | 23:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Zuul has a channel, let's tell people about it https://review.openstack.org/359976 | 23:23 |
| *** tdurakov has joined #openstack-infra | 23:23 | |
| *** tphummel has quit IRC | 23:24 | |
| *** DericHorn-HP has joined #openstack-infra | 23:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Configure jessie-security for our debian mirror https://review.openstack.org/347058 | 23:26 |
| *** cody-somerville has quit IRC | 23:26 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/log_processor: Add a simple test for the worker https://review.openstack.org/361515 | 23:27 |
| *** zhiyan has joined #openstack-infra | 23:28 | |
| *** rvasilets___ has joined #openstack-infra | 23:28 | |
| *** fguillot is now known as fguillot_afk | 23:29 | |
| *** maximov has joined #openstack-infra | 23:30 | |
| *** fguillot_afk has quit IRC | 23:30 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update properties.slack to use convert xml https://review.openstack.org/351841 | 23:30 |
| tonyb | jlvillal: That's a little questionable. A meta question would be where would you expect the documentation about how to do that thing to be? | 23:31 |
| *** jerryz has quit IRC | 23:32 | |
| *** aimeeu has joined #openstack-infra | 23:32 | |
| *** fdegir has joined #openstack-infra | 23:33 | |
| *** raunak has quit IRC | 23:35 | |
| *** kaisers_ has joined #openstack-infra | 23:37 | |
| *** amit213 has quit IRC | 23:37 | |
| *** xarses has quit IRC | 23:37 | |
| *** amit213 has joined #openstack-infra | 23:40 | |
| *** kaisers_ has quit IRC | 23:41 | |
| *** simonmcc has joined #openstack-infra | 23:41 | |
| *** skath has joined #openstack-infra | 23:41 | |
| pabelanger | clarkb: if you want to review https://review.openstack.org/#/c/361449/ I can kick off a build later tonight | 23:41 |
| pabelanger | that should be our ip6tables fix for centos-7 | 23:42 |
| *** fguillot has joined #openstack-infra | 23:43 | |
| *** yamahata has quit IRC | 23:43 | |
| *** esikachev has joined #openstack-infra | 23:44 | |
| *** NobodyCam has quit IRC | 23:46 | |
| *** zhiyan has quit IRC | 23:46 | |
| *** rpodolyaka has quit IRC | 23:46 | |
| *** TheJulia has quit IRC | 23:46 | |
| *** maximov has quit IRC | 23:46 | |
| *** betherly has quit IRC | 23:46 | |
| *** rvasilets___ has quit IRC | 23:46 | |
| *** sridhar_ram_afk has quit IRC | 23:46 | |
| *** fdegir has quit IRC | 23:46 | |
| *** aimeeu has quit IRC | 23:46 | |
| *** simonmcc has quit IRC | 23:46 | |
| *** vdrok has quit IRC | 23:46 | |
| *** serverascode has quit IRC | 23:46 | |
| *** ashtokolov has quit IRC | 23:46 | |
| *** tdurakov has quit IRC | 23:46 | |
| *** skath has quit IRC | 23:46 | |
| *** mdrabe has quit IRC | 23:47 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add display-name option to github plugin https://review.openstack.org/348726 | 23:48 |
| *** AnarchyAo has quit IRC | 23:50 | |
| clarkb | pabelanger: have hyou tested it anywhere? | 23:52 |
| clarkb | pabelanger: I always get a little scared of iptables upadtes particularly at the end of day on friday | 23:52 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update rebuild plugin to use convert xml https://review.openstack.org/346999 | 23:52 |
| *** xarses has joined #openstack-infra | 23:53 | |
| clarkb | pabelanger: I went ahead and +2'd it and if you are comforatable approving feel free to | 23:53 |
| *** xarses has quit IRC | 23:53 | |
| pabelanger | clarkb: I tested start ip6tables service, not puppet | 23:53 |
| pabelanger | clarkb: I am also okay for waiting until next week | 23:54 |
| *** xarses has joined #openstack-infra | 23:54 | |
| pabelanger | lets do that | 23:54 |
| clarkb | ok | 23:54 |
| clarkb | also we have 44 images uploading nwo so its not like it will get uploaded soon | 23:54 |
| *** xarses has quit IRC | 23:54 | |
| *** yamahata has joined #openstack-infra | 23:54 | |
| *** xarses has joined #openstack-infra | 23:54 | |
| *** xarses has quit IRC | 23:55 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update build-discarder to use convert xml https://review.openstack.org/339228 | 23:56 |
| *** xarses has joined #openstack-infra | 23:57 | |
| clarkb | I am also pausing the image deletes in rax since they are taking forever and I don't awnt them to continue while I am off weekending (because I won't be able to watch them) | 23:57 |
| *** DericHorn-HP has quit IRC | 23:58 | |
| jlvillal | tonyb, I'm confused what you mean by documentation? | 23:58 |
| *** baoli has joined #openstack-infra | 23:58 | |
| tonyb | jlvillal: ? | 23:58 |
| clarkb | jlvillal: when the test fails because your line isn't normalized where do you read about how to normalize it | 23:58 |
| jlvillal | I didnt' propose doing that | 23:59 |
| jlvillal | I just that the global-requirements.txt file should be normalized | 23:59 |
| jlvillal | s/just that/just proposed that/ | 23:59 |
| clarkb | jlvillal: sure but how do you keep it normalized if you don't have a test running that fails on non normalized lines? | 23:59 |
| tonyb | jlvillal: my point is you only get in the situation you're in because you didn't use the tools available to update the files. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!