Friday, 2016-08-26

*** Swami has quit IRC00:00
*** nwkarsten has quit IRC00:00
*** nwkarsten has joined #openstack-infra00:01
*** dingyichen has joined #openstack-infra00:01
*** thorst has quit IRC00:01
*** roxanaghe has quit IRC00:02
*** kzaitsev_mb has quit IRC00:02
*** kaisers_ has joined #openstack-infra00:03
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Skip unit tests and linuxbridge job on Neutron doc changes
*** zhurong has quit IRC00:04
*** xarses has quit IRC00:04
*** Julien-zte has quit IRC00:04
*** nwkarsten has quit IRC00:05
*** kaisers_ has quit IRC00:08
*** tqtran_ has quit IRC00:11
*** gongysh has joined #openstack-infra00:13
*** sflanigan has joined #openstack-infra00:14
*** kzaitsev_mb has joined #openstack-infra00:14
*** ddieterly has joined #openstack-infra00:18
*** furlongm_ has quit IRC00:19
ianwzaro: does any this mean anything to you?  odd projects?  ->
*** kzaitsev_mb has quit IRC00:21
*** david-lyle has quit IRC00:21
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add internap-mtl01 to cacti.o.o
*** tqtran has joined #openstack-infra00:24
pabelangerianw: sigh:
pabelangerianw: I really need to get our gem mirror working00:26
*** jistr has quit IRC00:27
*** tonytan4ever has quit IRC00:28
*** gyee has quit IRC00:28
*** jistr has joined #openstack-infra00:30
*** thorst has joined #openstack-infra00:30
*** thorst has quit IRC00:37
*** fguillot has joined #openstack-infra00:37
*** thorst has joined #openstack-infra00:38
*** esberglu has joined #openstack-infra00:39
*** esberglu has quit IRC00:43
*** mtanino has quit IRC00:44
*** salv-orlando has joined #openstack-infra00:46
*** thorst has quit IRC00:46
*** gouthamr_ has quit IRC00:47
*** nwkarsten has joined #openstack-infra00:48
*** salv-orlando has quit IRC00:51
*** zz_dimtruck is now known as dimtruck00:51
*** nwkarsten has quit IRC00:53
*** Julien-zte has joined #openstack-infra00:53
*** rajinir has quit IRC00:55
*** tqtran has quit IRC00:57
*** Jeffrey4l__ has joined #openstack-infra01:04
*** tqtran has joined #openstack-infra01:08
*** dimtruck is now known as zz_dimtruck01:08
*** shashank_hegde has quit IRC01:08
ianwzaro: so i think my conclusion is that because the new review-dev gets set up with a connection to the remote mysql host, it's not actually possible to re-deploy a new review-dev.  the new host is trying to share the same, already initalised, database and that just doesn't work.01:09
ianwthus i'm not exactly sure what to do01:10
*** jogo has quit IRC01:10
*** jogo has joined #openstack-infra01:12
*** yamahata has quit IRC01:13
ianwi guess shut down the old one, reattach the gerrit cinder volume to the new one, update dns and cross fingers01:13
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Double rax-ord boot-timeout value"
*** thcipriani is now known as thcipriani|afk01:15
pabelangerianw: so, we can revert ^. I wanted to see if it helped our launch failure rate in rax-ord, but it didn't. I'll have to open a ticket in the morning with rackspace to help debug the issue01:16
pabelanger is an example of the issue01:16
*** zhurong has joined #openstack-infra01:17
*** thorst has joined #openstack-infra01:18
*** thorst has quit IRC01:19
*** chlong has joined #openstack-infra01:19
*** thorst has joined #openstack-infra01:19
*** Apoorva_ has joined #openstack-infra01:19
ianwpabelanger: hmm, yeah, doesn't seem like much you can do without logs01:23
*** Apoorva has quit IRC01:23
*** yamahata has joined #openstack-infra01:27
openstackgerritPaul Belanger proposed openstack-infra/project-config: Disable rax-iad due to launch failure rate
*** yanyanhu has joined #openstack-infra01:27
pabelangerianw: Ya, in fact, I propose we disable rax-iad^ until we can work with rackspace01:27
*** raunak has quit IRC01:27
*** thorst has quit IRC01:28
pabelangerwe're basically flooding the debug logs for nodepool with ssh exceptions01:28
ianwyeah, i noticed that01:28
pabelangerfungi: if you are still around ^01:28
ianwjhesketh: is this something anyone in a currently awake timezone can help with?01:28
*** Julien-zte has quit IRC01:29
jheskethianw: sure, I'm happy to look but get on a plane in a few hours01:30
*** ddieterly has quit IRC01:31
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add -ubuntu-trusty to gate-grenade-dsvm
*** gildub has joined #openstack-infra01:33
ianwjhesketh: i guess it's a matter of correlating some of the failure to launch's seen in nodepool.log with some sort of internal log that gives more details?01:33
jheskethpabelanger: I'm okay with disabling IAD, (just +2'd), but I don't think I'll be around to undo that if it recovers01:34
*** yamamoto has quit IRC01:34
*** sdague has joined #openstack-infra01:34
jheskethianw: do you mean internal in rackspace public cloud?01:34
*** Julien-zte has joined #openstack-infra01:35
*** tonytan4ever has joined #openstack-infra01:35
ianwjhesketh: well, yeah, in IAD01:35
jheskethianw: right, that's not something I have access to. We'd have to raise a ticket in public cloud01:36
jheskethianw, pabelanger: is all we have to go off now the ssh exceptions?01:37
*** tqtran has quit IRC01:39
*** AnarchyAo has joined #openstack-infra01:40
*** tonytan4ever has quit IRC01:40
ianwjhesketh: i believe so01:41
openstackgerritMerged openstack-infra/system-config: Add internap-mtl01 to nodepool clouds.yaml
*** roxanaghe has joined #openstack-infra01:46
*** pvaneck has quit IRC01:47
*** ddieterly has joined #openstack-infra01:48
*** jamielennox is now known as jamielennox|away01:50
*** roxanaghe has quit IRC01:50
*** kaisers_ has joined #openstack-infra01:53
*** salv-orlando has joined #openstack-infra01:53
*** thorst has joined #openstack-infra01:55
*** thorst has quit IRC01:55
*** thorst has joined #openstack-infra01:56
pabelangerjhesketh: ianw: Yes, we've need SSH failures into rax-iad after we launch a node. Both ipv4 and ipv6 are non functional.  I bumped out timeout to 240 seconds the other day, but that didn't help01:56
*** kaisers_ has quit IRC01:57
pabelangerI plan on raising the ticket in the morning too01:58
*** jamielennox|away is now known as jamielennox02:00
pabelangerianw: we should be ready to land depends is live02:01
jheskethpabelanger: okay, I'll leave it to you to raise as you've been tracking the issue closely and I'll be heading into the weekend02:01
ianwlet's do it!02:02
ianwjhesketh: do you know about rackdns (the command ... and what's behind it i guess)02:02
ianwi believe i've cut over to the new server02:03
ianwbut "rackdns record-list | grep review-dev" returns nothing02:03
*** salv-orlando has quit IRC02:04
*** ddieterly has quit IRC02:04
*** thorst has quit IRC02:04
jheskethianw: I've not used it sorry, I've only ever worked with single records so I just used the webui02:05
*** fguillot has quit IRC02:06
ianwmaybe i should poke at that and see if it's consistent02:06
ianwi think the 3600 ttl is telling me it's time to get some lunch and check it later; maybe it will sort itself out :)02:06
*** jogo has quit IRC02:07
*** jogo has joined #openstack-infra02:08
*** sdake_ has joined #openstack-infra02:08
openstackgerritMerged openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o
*** Sukhdev has quit IRC02:09
*** fguillot has joined #openstack-infra02:09
*** sdake has quit IRC02:12
*** _ari_ has joined #openstack-infra02:13
*** yamamoto has joined #openstack-infra02:13
*** raunak has joined #openstack-infra02:16
*** thorst has joined #openstack-infra02:16
*** sdake_ has quit IRC02:17
*** thorst has quit IRC02:17
*** sdague has quit IRC02:17
openstackgerritMerged openstack-infra/zuul: Always create foreign projects if needed
*** ddieterly has joined #openstack-infra02:19
*** sdake has joined #openstack-infra02:19
*** tphummel has quit IRC02:19
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Fix projects entering pipeline without jobs
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Projects without jobs still enter pipeline
*** sdake has quit IRC02:23
*** smarcet has joined #openstack-infra02:26
*** sdake has joined #openstack-infra02:29
*** ddieterly has quit IRC02:34
*** sflanigan is now known as sflanigan_afk02:34
*** raunak has quit IRC02:36
*** xarses has joined #openstack-infra02:39
*** Wei_Liu has quit IRC02:41
*** yamahata has quit IRC02:43
*** brad_behle_ has quit IRC02:44
*** smarcet has quit IRC02:45
*** ddieterly has joined #openstack-infra02:46
*** roxanaghe has joined #openstack-infra02:46
*** ddieterly has quit IRC02:46
*** Genek has joined #openstack-infra02:47
*** thorst has joined #openstack-infra02:47
*** mjb has quit IRC02:47
*** StevenK has quit IRC02:48
*** onovy has quit IRC02:48
*** vkramskikh has quit IRC02:48
*** rm_work has quit IRC02:48
*** onovy has joined #openstack-infra02:48
*** StevenK has joined #openstack-infra02:49
*** mjb has joined #openstack-infra02:49
*** vkramskikh has joined #openstack-infra02:49
*** roxanaghe has quit IRC02:50
*** rm_work has joined #openstack-infra02:52
*** thorst has quit IRC02:52
*** thorst has joined #openstack-infra02:53
*** fguillot has quit IRC02:53
*** AnarchyAo has quit IRC02:54
*** bexelbie has quit IRC02:54
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Fix gate_hook check
*** dansmith has quit IRC02:54
*** _alastor_ has quit IRC02:55
*** katyafervent_awa has quit IRC02:55
*** rakhmerov has quit IRC02:55
*** bradm has quit IRC02:55
*** cj has quit IRC02:55
*** briancline has quit IRC02:55
*** Hunner has quit IRC02:55
*** bradm has joined #openstack-infra02:56
*** yanyanhu has quit IRC02:56
*** jamielennox is now known as jamielennox|away02:56
*** cj has joined #openstack-infra02:56
*** briancline has joined #openstack-infra02:57
*** Wei_Liu has joined #openstack-infra02:57
*** yanyanhu has joined #openstack-infra02:58
*** dansmith has joined #openstack-infra03:00
*** dansmith is now known as Guest7164503:01
*** thorst has quit IRC03:01
*** Hunner has joined #openstack-infra03:01
*** Hunner has quit IRC03:01
*** Hunner has joined #openstack-infra03:01
*** bexelbie has joined #openstack-infra03:03
*** salv-orlando has joined #openstack-infra03:03
*** katyafervent_awa has joined #openstack-infra03:03
*** rakhmerov has joined #openstack-infra03:04
*** _alastor1 has joined #openstack-infra03:04
*** salv-orlando has quit IRC03:06
*** jamielennox|away is now known as jamielennox03:10
*** gongysh has quit IRC03:10
*** sflanigan_afk is now known as sflanigan03:11
*** sdake has quit IRC03:12
*** sdake has joined #openstack-infra03:13
*** baoli_ has quit IRC03:14
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenStackId 1.0.19
*** NomNKDJIvx has quit IRC03:16
*** Genek has quit IRC03:18
*** smarcet has joined #openstack-infra03:19
*** Genek has joined #openstack-infra03:19
smarcetfungi: hey how are u doing ? i u see this , please review need it asap on production , cheers!03:19
ianwjhesketh: well, "rackdns record-list" shows way less records than what you seen if you log in and use the control panel dns.  i wonder if it's paginating the api response correctly.  a yak for another day03:20
pabelangerianw: just restarted nodepool-buidler03:25
ianwok, is it uploading?03:26
pabelangerlooks like it03:26
pabelangerI need to add image-uploads to grafana03:27
*** sarob has joined #openstack-infra03:27
pabelangerianw: I noticed you removed the custom kernel for f24, guess your patches landed upstream?03:27
ianwi got it into the fedora pkg03:28
ianwi've got one devstack patch, then i'm ready to cut over03:28
ianwwe can kill f23 then03:28
*** smarcet has quit IRC03:32
*** sarob has quit IRC03:32
*** Genek has quit IRC03:34
*** chlong has quit IRC03:37
*** kaisers_ has joined #openstack-infra03:41
*** Sukhdev has joined #openstack-infra03:43
*** shashank_hegde has joined #openstack-infra03:44
*** kaisers_ has quit IRC03:46
*** roxanaghe has joined #openstack-infra03:47
*** roxanaghe has quit IRC03:51
craigeIf you're about and available, could do with an urgnet review, jhesketh03:52
jheskethcraige: +203:54
jheskethI'm off to the airport now so I can't baby sit it sorry03:55
craigehave fun, thanks jhesketh03:56
*** Genek has joined #openstack-infra03:56
craigeapprove all the papers you know I want to see, jhesketh ;-)03:56
*** sdake has quit IRC03:56
*** gongysh has joined #openstack-infra03:57
*** thorst has joined #openstack-infra03:59
*** Genek has quit IRC03:59
*** yonglihe has quit IRC04:04
*** armax has quit IRC04:06
*** thorst has quit IRC04:07
*** mdrabe has quit IRC04:07
*** salv-orlando has joined #openstack-infra04:07
*** links has joined #openstack-infra04:07
*** rlandy|bbl is now known as rlandy04:07
*** chlong has joined #openstack-infra04:10
*** docaedo4 has quit IRC04:10
*** docaedo4 has joined #openstack-infra04:11
*** Julien-zte has quit IRC04:12
*** yamahata has joined #openstack-infra04:14
*** senk has joined #openstack-infra04:14
*** salv-orlando has quit IRC04:14
*** Genek has joined #openstack-infra04:16
*** Genek has quit IRC04:20
*** Genek has joined #openstack-infra04:21
*** sdake has joined #openstack-infra04:22
*** mdrabe has joined #openstack-infra04:23
*** apetrich has quit IRC04:25
openstackgerritRabi Mishra proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin
*** apetrich has joined #openstack-infra04:25
*** mdrabe has quit IRC04:30
ianwpabelanger: ha ha, famous last words.  "msg": "ERROR! mkdir: cannot create directory ‘/home/jenkins/.ansible/tmp/ansible-tmp-1472167685.83-197144362572539’: No space left on device\n", <- ... this is a new one for f2404:35
openstackgerritSachi King proposed openstack-dev/pbr: Don't ignore data-files
*** Genek has quit IRC04:39
*** kushal has quit IRC04:43
clarkbianw: maybe growroot isnt working?04:44
ianwclarkb: yeah, i dunno ... it *was* working a day or so ago04:45
*** jaosorior has joined #openstack-infra04:46
*** psachin has joined #openstack-infra04:49
*** _nadya_ has joined #openstack-infra04:52
*** tonytan4ever has joined #openstack-infra04:52
*** kzaitsev_mb has joined #openstack-infra04:56
*** tonytan4ever has quit IRC04:57
*** yamahata has quit IRC05:01
*** reed_ has joined #openstack-infra05:03
*** thorst has joined #openstack-infra05:04
fungicraige: i've approved 360862 so hopefully it'll update shortly05:06
*** ilyashakhat has joined #openstack-infra05:06
*** senk has quit IRC05:11
*** thorst has quit IRC05:11
*** _nadya_ has quit IRC05:11
openstackgerritMerged openstack-infra/system-config: OpenStackId 1.0.19
*** Sukhdev has quit IRC05:13
*** _nadya_ has joined #openstack-infra05:13
*** yamahata has joined #openstack-infra05:15
*** moravec1 has joined #openstack-infra05:15
*** moravec has quit IRC05:16
*** salv-orlando has joined #openstack-infra05:17
*** roxanaghe has joined #openstack-infra05:18
*** kzaitsev_mb has quit IRC05:19
*** salv-orlando has quit IRC05:20
*** chlong has quit IRC05:21
*** kzaitsev_mb has joined #openstack-infra05:21
*** roxanaghe has quit IRC05:22
*** salv-orlando has joined #openstack-infra05:25
*** salv-orlando has quit IRC05:25
*** gildub has quit IRC05:25
*** salv-orlando has joined #openstack-infra05:26
*** sarob has joined #openstack-infra05:28
*** raunak has joined #openstack-infra05:28
*** kaisers_ has joined #openstack-infra05:30
*** reed_ has quit IRC05:31
*** sarob has quit IRC05:32
*** sandanar has joined #openstack-infra05:34
*** kaisers_ has quit IRC05:35
*** gildub has joined #openstack-infra05:41
*** ilyashakhat has quit IRC05:43
*** gouthamr has joined #openstack-infra05:44
*** senk has joined #openstack-infra05:49
*** kzaitsev_mb has quit IRC05:49
*** _nadya_ has quit IRC05:49
*** _nadya_ has joined #openstack-infra05:50
*** tphummel has joined #openstack-infra05:54
*** _nadya_ has quit IRC05:54
*** raunak has quit IRC05:59
*** raunak has joined #openstack-infra06:01
openstackgerritMerged openstack-infra/system-config: Increase httpd threads for review.o.o
*** pgadiya has joined #openstack-infra06:04
*** _nadya_ has joined #openstack-infra06:06
*** matbu|afk is now known as matbu06:07
*** thorst has joined #openstack-infra06:09
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Use smartSummarize where appropriate
*** aeng has quit IRC06:13
*** Julien-zte has joined #openstack-infra06:14
*** raunak has quit IRC06:14
*** ilyashakhat has joined #openstack-infra06:16
*** thorst has quit IRC06:16
*** evgenyl has quit IRC06:20
*** evgenyl has joined #openstack-infra06:22
*** shardy has joined #openstack-infra06:23
*** rcernin has joined #openstack-infra06:25
*** salv-orlando has quit IRC06:31
*** caowei has joined #openstack-infra06:33
*** ccamacho has left #openstack-infra06:37
*** tesseract- has joined #openstack-infra06:44
*** salv-orlando has joined #openstack-infra06:45
*** _nadya_ has quit IRC06:49
*** _nadya_ has joined #openstack-infra06:49
*** adriant has quit IRC06:53
*** _nadya_ has quit IRC06:54
*** amotoki has joined #openstack-infra06:56
*** ilyashakhat has quit IRC06:59
*** andreas_s has joined #openstack-infra07:00
yolandagood morning07:00
*** martinkopec has joined #openstack-infra07:00
*** yaume has joined #openstack-infra07:02
*** DrifterZA has joined #openstack-infra07:02
*** dizquierdo has joined #openstack-infra07:04
*** tdasilva has quit IRC07:04
*** roxanaghe has joined #openstack-infra07:06
*** dizquierdo has quit IRC07:07
*** dtardivel has joined #openstack-infra07:07
*** dizquierdo has joined #openstack-infra07:07
*** roxanaghe has quit IRC07:10
*** thorst has joined #openstack-infra07:15
*** salv-orlando has quit IRC07:16
*** Julien-zte has quit IRC07:17
*** kaisers_ has joined #openstack-infra07:19
*** mikelk has joined #openstack-infra07:21
*** thorst has quit IRC07:21
*** jaosorior is now known as jaosorior_brb07:22
*** asettle has joined #openstack-infra07:23
*** ihrachys has joined #openstack-infra07:24
*** kaisers_ has quit IRC07:24
*** r-mibu has quit IRC07:27
*** r-mibu has joined #openstack-infra07:27
*** asettle has quit IRC07:28
*** Genek has joined #openstack-infra07:29
*** florianf has joined #openstack-infra07:30
*** kaisers_ has joined #openstack-infra07:30
*** flepied1 is now known as flepied07:31
*** ihrachys has quit IRC07:36
*** Genek has quit IRC07:38
*** anteaya has quit IRC07:38
*** gildub has quit IRC07:39
*** matrohon has joined #openstack-infra07:39
*** netsinn has joined #openstack-infra07:39
*** esikachev has joined #openstack-infra07:39
*** ifarkas_afk is now known as ifarkas07:41
*** tdasilva has joined #openstack-infra07:42
*** shashank_hegde has quit IRC07:42
*** netsin has quit IRC07:43
*** matthewbodkin has joined #openstack-infra07:44
*** caowei has quit IRC07:44
*** Genek has joined #openstack-infra07:45
*** yamahata has quit IRC07:45
*** dtantsur|afk is now known as dtantsur07:46
*** jpich has joined #openstack-infra07:47
*** Julien-zte has joined #openstack-infra07:48
*** hashar has joined #openstack-infra07:53
*** hashar is now known as hasharAway07:53
openstackgerritMerged openstack-infra/project-config: Add fuel-ccp-cinder repository
*** hasharAway is now known as hashar07:54
openstackgerritMerged openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka
*** shashank_hegde has joined #openstack-infra07:54
*** salv-orlando has joined #openstack-infra07:54
openstackgerritMerged openstack-infra/project-config: Skip OSA Xenial integrated build jobs for Liberty & Mitaka
openstackgerritMerged openstack-infra/project-config: Add os_trove to OpenStack-Ansible
openstackgerritMerged openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5
openstackgerritMerged openstack-infra/project-config: Enable Neutron for the Tempest/Sahara tests
openstackgerritMerged openstack-infra/project-config: Add timestamp.txt to debian-openstack repo
dtantsurhi folks. what's " Submitted, Merge Pending" review status? it's been there for a some time already07:57
openstackgerritMerged openstack-infra/system-config: Reduce job log retention to 90 days
openstackgerritMerged openstack-infra/project-config: Switch all heat jobs to use devstack plugin
openstackgerritMerged openstack-infra/project-config: Add -ubuntu-trusty to gate-grenade-dsvm
openstackgerritMerged openstack-infra/project-config: Revert "Double rax-ord boot-timeout value"
*** ihrachys has joined #openstack-infra07:58
*** ihrachys has quit IRC07:59
*** zzzeek has quit IRC08:00
openstackgerritMerged openstack-infra/project-config: Disable rax-iad due to launch failure rate
*** zzzeek has joined #openstack-infra08:00
*** flepied has quit IRC08:01
openstackgerritMerged openstack-infra/project-config: Update job skip condition for horizon doc-only changes
*** sdake has quit IRC08:05
*** derekh has joined #openstack-infra08:05
*** sdake has joined #openstack-infra08:05
openstackgerritMerged openstack-infra/project-config: Add Almanach IRC channel
AJaegerdtantsur: have a look at your change, it failed to merge, didn't it?08:12
dtantsurAJaeger, true. I wonder why zuul hasn't left a comment08:12
*** Genek has quit IRC08:13
AJaegerWhich change is it ? Let me check something...08:13
*** Wei_Liu has quit IRC08:13
* AJaeger remembers this from yesterday08:13
*** ociuhandu has joined #openstack-infra08:13
AJaegerdtantsur: I'm not seeing such a change in my queue - do you have a link, please?08:16
dtantsurAJaeger, I've rebased both already, these were and
*** Wei_Liu has joined #openstack-infra08:18
AJaegerdtantsur: ah, ok.08:18
*** berendt has joined #openstack-infra08:18
*** openstackgerrit has quit IRC08:18
*** ociuhandu has quit IRC08:18
*** openstackgerrit has joined #openstack-infra08:18
*** thorst has joined #openstack-infra08:18
*** Na3iL has joined #openstack-infra08:19
AJaegerlooking at the first: We had  problem yesterday around that time, might have been an infra problem that we fixed now - and tried to fix all fallout, seems this one was missed ;(08:19
*** Ahharu has joined #openstack-infra08:19
AJaegerdtantsur: so, thanks for telling - this is worth double checking.08:19
*** dizquierdo has quit IRC08:20
dtantsurnp, thanks for looking into it08:20
*** shashank_hegde has quit IRC08:21
*** coolsvap_ has quit IRC08:21
*** florianf has quit IRC08:25
*** thorst has quit IRC08:26
*** electrofelix has joined #openstack-infra08:27
*** lucas-dinner is now known as lucasagomes08:28
openstackgerritAlex Kavanagh proposed openstack-infra/project-config: Add charm-manila
*** markmcd has quit IRC08:30
*** flepied has joined #openstack-infra08:31
*** flepied has quit IRC08:31
*** bexelbie has quit IRC08:32
*** bexelbie has joined #openstack-infra08:32
*** flepied has joined #openstack-infra08:33
*** sflanigan has quit IRC08:37
*** asettle has joined #openstack-infra08:38
*** asettle has quit IRC08:38
*** asettle has joined #openstack-infra08:39
*** tqtran has joined #openstack-infra08:39
*** florianf has joined #openstack-infra08:40
*** tqtran has quit IRC08:43
*** gongysh has quit IRC08:44
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** sdake has quit IRC08:48
*** amotoki has quit IRC08:54
*** roxanaghe has joined #openstack-infra08:54
*** tonytan4ever has joined #openstack-infra08:54
odyssey4mehi everyone, now that has merged please add me to,members08:56
*** dizquierdo has joined #openstack-infra08:57
*** tonytan4ever has quit IRC08:58
*** roxanaghe has quit IRC08:59
*** jaosorior_brb is now known as jaosorior08:59
*** _degorenko|afk is now known as degorenko08:59
*** vinaypotluri has quit IRC09:01
*** gouthamr has quit IRC09:05
*** mcarden has quit IRC09:06
*** amitgandhinz has quit IRC09:06
*** jhesketh has quit IRC09:06
*** gus has quit IRC09:06
*** darrenc has quit IRC09:06
*** mattoliverau has quit IRC09:06
*** matt6434 has joined #openstack-infra09:06
*** darrenc has joined #openstack-infra09:06
*** mcarden has joined #openstack-infra09:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove release-notes from security-analysis
*** gus has joined #openstack-infra09:08
*** jhesketh has joined #openstack-infra09:09
rcarrillocruzclarkb: i just ran on an xenial instance in the infracloud, took a little bit less than an hour09:11
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** salv-orl_ has joined #openstack-infra09:17
*** salv-orlando has quit IRC09:20
*** amitgandhinz has joined #openstack-infra09:20
*** pilgrimstack has quit IRC09:21
*** pilgrimstack has joined #openstack-infra09:22
*** ihrachys has joined #openstack-infra09:22
*** thorst has joined #openstack-infra09:24
*** abregman has joined #openstack-infra09:26
*** abregman has quit IRC09:27
odyssey4mercarrillocruz, AJaeger or yolanda: now that has merged please add me to,members09:27
openstackgerritMerged openstack-infra/storyboard: Hide timeline events and comments of private stories
AJaegerodyssey4me: I don't have those permissions and thus can't help09:28
yolandaodyssey4me, i can do it09:28
odyssey4meah, AJaeger - I wasn't aware, apologies09:28
odyssey4methanks yolanda09:28
*** thorst has quit IRC09:31
*** Julien-zte has quit IRC09:32
*** tkelsey has joined #openstack-infra09:32
*** kzaitsev_mb has joined #openstack-infra09:33
*** tphummel has quit IRC09:35
*** matthewbodkin has quit IRC09:38
*** _nadya_ has joined #openstack-infra09:39
*** matthewbodkin has joined #openstack-infra09:42
*** alexpilotti has joined #openstack-infra09:46
*** alexpilotti has left #openstack-infra09:46
*** tosky has joined #openstack-infra09:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud"
*** yamamoto has quit IRC09:52
*** kdas__ has joined #openstack-infra09:52
*** kdas__ is now known as kushal09:57
*** kushal has joined #openstack-infra09:58
*** zhurong has quit IRC10:01
rcarrillocruzpabelanger: ^ put all images on infracloud10:02
rcarrillocruzwe may want to land that to get DIB images upload rolling10:02
*** yanyanhu has quit IRC10:02
*** yanyanhu has joined #openstack-infra10:03
rcarrillocruzi ran a dsvm on xenial and took a little bit less than 1h, roughly the same time as OSIC per
*** yanyanhu has quit IRC10:03
rcarrillocruzalso i believe we need a mirror on that, ping me when around since you are more familiar with afs and all than me10:04
*** ociuhandu has joined #openstack-infra10:08
*** kzaitsev_mb has quit IRC10:19
*** matt6434 is now known as mattoliverau10:26
*** gongysh has joined #openstack-infra10:27
openstackgerritAntoine Musso proposed openstack-infra/zuul: Add __repr__ to a few classes
*** hexlibris is now known as paulobanon10:36
paulobanonhi all10:37
*** Julien-zte has joined #openstack-infra10:39
*** roxanaghe has joined #openstack-infra10:42
*** rhallisey has joined #openstack-infra10:43
*** roxanaghe has quit IRC10:47
*** sdague has joined #openstack-infra10:49
*** rvasilets__ has joined #openstack-infra10:52
rvasilets__Hi, I've got and an error msg doesn't tell me nothing( Could you help me?10:53
rvasilets__Link to the patch
*** thorst has joined #openstack-infra10:56
sdaguefyi it looks like internap nodes are having cloning issues to git over ipv6, and failing a bunch of jobs10:57
AJaegerrvasilets__: it looks to me as if you missed to add  lbaas_use_v1 at one place - double check the usage of the builder10:58
openstackgerritSean Dague proposed openstack-infra/project-config: Drop postgresql jobs in gate
sdagueAJaeger: that should address your comments on that patch10:59
AJaegerrvasilets__: I commented - please double check10:59
AJaegersdague: thanks, will review11:00
*** yaume has quit IRC11:04
AJaegerproject-config cores, could you review, please?11:07
*** gildub has joined #openstack-infra11:10
*** shardy is now known as shardy_lunch11:13
*** cody-somerville has quit IRC11:13
*** csomerville has joined #openstack-infra11:13
*** tosky has quit IRC11:13
*** tosky has joined #openstack-infra11:15
*** sdague has quit IRC11:17
*** kzaitsev_mb has joined #openstack-infra11:18
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1
rvasilets__AJaeger, yeah, Thx!11:20
*** ramishra has quit IRC11:20
*** lucasagomes is now known as lucas-hungry11:21
openstackgerritAntoine Musso proposed openstack-infra/zuul: Add __repr__ to a few classes
*** ramishra has joined #openstack-infra11:22
*** matbu is now known as matbu|lch11:22
*** DrifterZA has quit IRC11:25
*** DrifterZA has joined #openstack-infra11:26
*** DrifterZA has quit IRC11:26
*** salv-orl_ has quit IRC11:27
*** ldnunes has joined #openstack-infra11:27
*** kzaitsev_mb has quit IRC11:28
*** jkilpatr has joined #openstack-infra11:30
*** asettle has quit IRC11:32
*** yamamot__ has joined #openstack-infra11:35
openstackgerritMonty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
*** matthewbodkin has quit IRC11:40
*** amotoki has joined #openstack-infra11:44
*** sdague has joined #openstack-infra11:45
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1
*** Apoorva_ has quit IRC11:46
*** Apoorva has joined #openstack-infra11:47
*** senk has quit IRC11:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute003.vanilla to static Ansible inventory
*** kzaitsev_mb has joined #openstack-infra11:50
sdaguemordred: you able to bring us home on this - ?11:51
*** Apoorva has quit IRC11:51
sdaguercarrillocruz: also, internap is acting up talking to git -
dtantsursigh, is it possible to fix gerrit to not log everyone out every day? I've just lost a huge comment because of the terrible way the whole UI is implemented...11:52
* dtantsur remembers to write comments in a text editor first, then copy-paste them to gerrit11:53
mordredsdague: damn. rcarrillocruz beat me to it11:53
*** jaosorior has quit IRC11:53
openstackgerritMerged openstack-infra/nodepool: Remove unecessecary builder scheduler thread
sdaguemordred: you can still pile on for posterity11:54
openstackgerritMerged openstack-infra/nodepool: Remove 'running' as a public method from builder
*** pgadiya has quit IRC11:54
*** jaosorior has joined #openstack-infra11:54
rcarrillocruzdtantsur: iirc, gerrit was tweaked some time ago to last sessions a week
rcarrillocruzhowever, gerrit was restarted yesterday, which is why your session might have died11:55
*** tonytan4ever has joined #openstack-infra11:55
dtantsurhmm, maybe? thanks, I'll keep watching it.11:56
*** matbu|lch is now known as matbu11:56
*** gildub has quit IRC11:57
*** jamesdenton has quit IRC11:57
mordredsdague: :)11:57
*** tonytan4ever has quit IRC12:00
openstackgerritMerged openstack-infra/project-config: Drop postgresql jobs in gate
* mordred hands sdague a pie12:01
AJaegermordred: should I send some review requests your way so that you can catch up?12:02
*** sdague has quit IRC12:02
*** ansmith has joined #openstack-infra12:02
mordredAJaeger: wow. you are very good at gerrit dashboards12:02
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Fix baremetal.json IP params for compute00(5|6)
*** baoli has joined #openstack-infra12:03
AJaegermordred: just copied a link I had saved - and quickly edited the title ;)12:03
*** Goneri has joined #openstack-infra12:04
*** baoli has quit IRC12:07
*** sdague has joined #openstack-infra12:08
*** tonytan4ever has joined #openstack-infra12:09
*** asettle has joined #openstack-infra12:10
*** baoli has joined #openstack-infra12:11
*** kushal has quit IRC12:11
*** shardy_lunch is now known as shardy12:11
*** nwkarsten has joined #openstack-infra12:12
nibalizer"monty review inbox" lololol12:12
*** matthewbodkin has joined #openstack-infra12:15
*** senk has joined #openstack-infra12:16
*** xyang1 has joined #openstack-infra12:17
sdaguemordred et al (devstack-gate reviewers) -
openstackgerritMerged openstack-infra/project-config: Remove release-notes from security-analysis
sdaguegrenade no longer in horizon gate (it didn't do anything but install), so we can not install it in grenade jobs to save time12:18
sdagueand any rando install errors that might exist12:18
sdagueand there is a cleanup to remove some juno level testing after that12:19
openstackgerritMerged openstack-infra/project-config: Add a tempest job for python-ironic-inspector-client
openstackgerritMerged openstack-infra/project-config: Don't use devuser for zuul-worker
*** baoli_ has joined #openstack-infra12:20
*** dprince has joined #openstack-infra12:23
*** snarwade is now known as snarwade-afk12:23
*** baoli has quit IRC12:23
openstackgerritMerged openstack-infra/system-config: Fix baremetal.json IP params for compute00(5|6)
*** kaisers1 has quit IRC12:23
*** kaisers has quit IRC12:23
*** chem|off` has quit IRC12:25
*** lucas-hungry is now known as lucasagomes12:25
*** chem|off` has joined #openstack-infra12:26
*** gordc has joined #openstack-infra12:26
*** ansmith has quit IRC12:27
*** pradk has joined #openstack-infra12:29
*** kaisers has joined #openstack-infra12:29
*** kaisers1 has joined #openstack-infra12:30
*** roxanaghe has joined #openstack-infra12:30
*** martinkopec has quit IRC12:30
*** rlandy has joined #openstack-infra12:31
*** dizquierdo has quit IRC12:32
*** martinkopec has joined #openstack-infra12:33
*** woodster_ has joined #openstack-infra12:34
*** roxanaghe has quit IRC12:35
*** jed56 has joined #openstack-infra12:40
*** tqtran has joined #openstack-infra12:41
*** senk has quit IRC12:41
*** markusry has joined #openstack-infra12:43
sdagueis anyone looking at internap failing the world?12:43
*** slashme has joined #openstack-infra12:44
sdagueinternap looks like it is failing everything on connectivity issue12:45
*** tqtran has quit IRC12:45
rcarrillocruzsdague: doesn't seem to be consistenly failing12:47
rcarrillocruzat least i could clone and ping ipv4 fine12:47
sdaguewhat is internap-mtl01 ?12:47
sdagueall the fails are there12:47
rcarrillocruzsec, i kind of remember changes about it from yesterday or something12:48
mat128sdague: can you show me a log?12:48
sdaguercarrillocruz: well, there are 249 fail events12:48
*** Apoorva has joined #openstack-infra12:48
sdaguemat128: also see the logstash query above12:48
sdagueI've also seen 3 gate patches fail on this today12:48
rcarrillocruzit's a new region12:48
rcarrillocruz99526dee00ac7d6bc79413f11db1ad4acea174a3 from system-config shows12:48
rcarrillocruzAdd mtl01 region to internap on puppetmaster12:49
sdaguejust by glancing at the zuul page12:49
*** hashar has quit IRC12:49
mat128rcarrillocruz: I also work there12:49
rcarrillocruzah good12:49
mat128trying to get actual ops people involved12:49
sdaguemy suggestion, disable that region until it can be debugged manually12:49
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Update job skip condition for neutron doc-only changes
mat128sdague: I kind of agree...12:50
*** amotoki has quit IRC12:50
rcarrillocruzsdague: i'll push up a change12:50
openstackgerritSean Dague proposed openstack-infra/project-config: This has caused 240+ failures in the last 24 hours
sdaguercarrillocruz: actually there it is12:50
sdagueyeh, the nynj region seems fine, so it's isolated to one region at least, which is good12:51
sdagueok, got to drop off for ops summit. Will check in later12:51
mat128sdague: you can always drop a word to mgagne about it12:52
mat128he's there12:52
*** Apoorva has quit IRC12:53
*** rtheis has joined #openstack-infra12:53
rtheisIs it possible to force a gate job to skip IPv6-only hosts?12:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Disable Internap mtl01
*** sdague has quit IRC12:56
rcarrillocruzmordred: ^12:56
rcarrillocruzsdague seems to have left12:56
*** amotoki has joined #openstack-infra12:56
mat128rcarrillocruz: this is exactly what sdague submitted12:57
*** alaski is now known as lascii12:57
*** matt-borland has joined #openstack-infra12:57
rcarrillocruzsorry, was in a console and hadn't checked the scrollback12:57
rcarrillocruzabandoning mine, thx12:57
AJaegerrcarrillocruz: thanks for submitting - no need to be sorry.12:58
* rcarrillocruz goes back to baremetal provisioning and infracloud things12:58
mordredrtheis: it is not12:59
*** sdague has joined #openstack-infra12:59
rtheismordred: okay, thank you12:59
openstackgerritMonty Taylor proposed openstack-infra/shade: Log request ids
*** smarcet has joined #openstack-infra13:01
openstackgerritYuriy Taraday proposed openstack-infra/project-config: Enable puppet checks for fuel-octane
openstackgerritMonty Taylor proposed openstack-infra/shade: Log request ids
*** amotoki has quit IRC13:02
*** hashar has joined #openstack-infra13:03
*** salv-orlando has joined #openstack-infra13:03
*** eharney has joined #openstack-infra13:04
*** zhurong has joined #openstack-infra13:04
*** gongysh has quit IRC13:04
mordredmorning pabelanger !13:06
mordredpabelanger: we just disabled mtl13:06
rcarrillocruzpabelanger: heya, yeah that, ^13:06
rcarrillocruzsince you pushed changes for it yesterday13:06
mordredpabelanger: because jobs running there are having connectivity problems talking to git.o.o13:06
rcarrillocruzmat128 is looking into it13:06
rcarrillocruzbtw, i +2 your changes for the region name13:07
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: Add tracking artifact for network connectivity issues
mat128trying to see how/why this is happening13:07
rcarrillocruzi can't remember if i made you core on the launcher project?13:07
*** zul has joined #openstack-infra13:07
mat128I will provide an update before noon EDT13:07
pabelangermordred: thanks, catching up on backscroll no13:08
openstackgerritMerged openstack-infra/project-config: This has caused 240+ failures in the last 24 hours
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Extract /var/log from all nodes
*** esberglu has joined #openstack-infra13:14
*** sdake has joined #openstack-infra13:14
*** sdake_ has joined #openstack-infra13:15
pabelangermat128: we are also seeing a lot these errors:13:15
pabelangerOpenStackCloudException: Error in creating instance (Inner Exception: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a2af3a7c-5a1f-4896-a766-becdc9333a21))13:15
pabelangera long with SSH Exceptions after we've launched a node and try to SSH into the assigned IP13:16
*** dtantsur is now known as dtantsur|brb13:16
*** sdake has quit IRC13:19
openstackgerritMerged openstack-infra/elastic-recheck: Add tracking artifact for network connectivity issues
openstackgerritMerged openstack-infra/nodepool: Simplify builder start/stop methods
*** kushal has joined #openstack-infra13:24
*** abregman has joined #openstack-infra13:26
*** kgiusti has joined #openstack-infra13:27
*** anteaya has joined #openstack-infra13:29
*** amotoki has joined #openstack-infra13:31
*** roxanaghe has joined #openstack-infra13:31
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add baremetal00 to cacti hosts
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Remove hpuswest hosts from Cacti
paulobanoncan anyone let me know if you guys are also responsible for python-jenkins or if its supposed to be discussed in #openstack-dev? i have a small doubt regarding a bug i'm seeing with jenkins job builder13:35
pabelangerrcarrillocruz: I need to step out to run an errand with family, but I can help with AFS mirror for omfracloud when I get back13:35
pabelangerrcarrillocruz: we'd use the openstackci project to launch it13:36
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
*** roxanaghe has quit IRC13:36
rcarrillocruzpabelanger: ++13:36
rcarrillocruzso it seems RAX going private deal has gone thru13:37
*** senk has joined #openstack-infra13:37
*** sdake_ has quit IRC13:37
openstackgerritMerged openstack-infra/system-config: Add compute003.vanilla to static Ansible inventory
*** amotoki has quit IRC13:38
*** sdake has joined #openstack-infra13:40
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring
*** salv-orlando has quit IRC13:40
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring
*** berendt has quit IRC13:44
*** links has quit IRC13:45
openstackgerritMerged openstack-infra/system-config: Add internap-mtl01 to cacti.o.o
*** pt_15 has joined #openstack-infra13:46
openstackgerritRoman Vasilets proposed openstack-infra/project-config: Switch our neutron related jobs into LBaaS v1
*** kushal has quit IRC13:46
yolandamordred, do you know if there are still issues with new python-openstackclient? with latest version i get Could not determine a suitable URL for the plugin error13:46
mgagnepabelanger: I'm online now, although a bit distracted by ops midcycle13:47
mordredyolanda: can you run with --debug and paste me the output?13:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add controller00.vanilla to Cacti
anteayamgagne: how is that going?13:47
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Add c-vol,c-bak on subnode when c-api enabled
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
mgagneanteaya: I just arrived, slept a bit more this morning. logging and monitoring about to start =)13:47
*** gouthamr has joined #openstack-infra13:47
anteayamgagne: cool, I hope you have a good day13:48
mordredyolanda: (cause it works for me - I just installed into a new venv and everything is happy)13:48
rcarrillocruzclarkb, fungi : are we cool to land , re: reenabling infracloud, with max-servers to -1 ?13:48
rcarrillocruzso we can get the ball rolling for image builds13:48
*** Apoorva has joined #openstack-infra13:49
*** sdague has quit IRC13:51
yolandaah, have said nothing, 3.1.0 works13:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti
mordredyolanda: oh good13:51
yolandai was not using latest latest...13:52
*** ddieterly has joined #openstack-infra13:52
mordredyah. there were a couple of rocky moments recently13:53
*** sandanar has quit IRC13:53
*** amotoki has joined #openstack-infra13:53
*** Apoorva has quit IRC13:53
*** ddieterly has quit IRC13:54
*** ddieterly has joined #openstack-infra13:55
*** Marx314 has joined #openstack-infra13:56
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti
*** xarses has quit IRC13:58
*** thiagop has joined #openstack-infra13:59
*** fguillot has joined #openstack-infra14:00
*** brucet has joined #openstack-infra14:02
mgagnepabelanger: found those errors in nova-conductor logs: RescheduledException: Build of instance 1583a93f-1b2d-489d-9d28-8efc78f94455 was re-scheduled: Not authorized for image 2124d3ce-891b-412a-a951-309f79c008ec.14:03
*** mdrabe has joined #openstack-infra14:05
*** bin_ has joined #openstack-infra14:06
*** zz_dimtruck is now known as dimtruck14:06
*** rbrndt has joined #openstack-infra14:07
*** yamamot__ has quit IRC14:08
mordredoh. that's a new one!14:08
*** thiagolib has quit IRC14:08
*** senk has quit IRC14:09
*** Apoorva has joined #openstack-infra14:10
*** sdague has joined #openstack-infra14:11
openstackgerritMerged openstack-infra/system-config: Make Launchpad credentials reusable
*** dtantsur|brb is now known as dtantsur14:15
*** dimtruck is now known as zz_dimtruck14:16
openstackgerritMerged openstack-infra/system-config: Add LP creds and launchpadlib to signing job node
*** mtanino has joined #openstack-infra14:17
mgagnepabelanger: I see that some instances were created successfully, did the issue solved by itself?14:18
*** csomerville has quit IRC14:20
*** csomerville has joined #openstack-infra14:21
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring
*** ifarkas is now known as ifarkas_afk14:21
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Stop mirroring puppet modules
*** _nadya_ has quit IRC14:23
*** zz_dimtruck is now known as dimtruck14:24
*** Guest71645 is now known as dansmith14:25
*** ogelbukh has quit IRC14:25
*** jaranovich has quit IRC14:25
*** Ng has quit IRC14:25
*** vdrok has quit IRC14:25
*** ashtokolov has quit IRC14:25
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Start using the image cache again
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Tweak 'about' page
*** auggy has quit IRC14:26
*** monester_ has quit IRC14:26
*** iremizov has quit IRC14:26
*** ddieterly has quit IRC14:26
*** iremizov has joined #openstack-infra14:26
*** auggy has joined #openstack-infra14:26
*** inc0 has joined #openstack-infra14:26
*** monester_ has joined #openstack-infra14:27
*** szaher has quit IRC14:28
*** ogelbukh has joined #openstack-infra14:28
*** jaranovich has joined #openstack-infra14:28
*** Ng has joined #openstack-infra14:28
*** vdrok has joined #openstack-infra14:28
*** ashtokolov has joined #openstack-infra14:28
openstackgerritKevin Carter (cloudnull) proposed openstack-infra/project-config: Add initialize-urandom to fedora 24
fungircarrillocruz: the clouds.yaml on nodepool.o.o still has the correct credentials for 360579 to work i guess? we didn't change any of that?14:29
*** szaher has joined #openstack-infra14:29
rcarrillocruzstill work, i uploaded a trusty and xenial image manually from nodepool to infracloud14:30
rcarrillocruzso creds are good14:30
*** dprince has quit IRC14:30
*** yamamoto has joined #openstack-infra14:30
fungithanks rcarrillocruz!14:31
*** dprince has joined #openstack-infra14:31
zaropaulobanon: yes, python-jenkins responsibility lies here.14:31
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar
*** roxanaghe has joined #openstack-infra14:32
*** chlong has joined #openstack-infra14:32
*** kaisers_ has quit IRC14:32
fungiclarkb: jeblair: mordred: zaro: looks like our http threads change worked... show-caches --show-threads indicates we have a total of 29 httpd threads in gerrit right now, which is more than the default maximum14:32
YorikSarAJaeger: Thanks for swift review!14:34
zarofungi: cool, we still going ahead with those apache tweeks? or wait to see how gerrit responds first?14:34
YorikSarAnyone else around to merge ?14:34
fungizaro: i think we wait to see how gerrit and apache behave now14:34
zaroianw: was wondering whether new review-dev was ready yet?14:34
openstackgerritMerged openstack-infra/tripleo-ci: Update undercloud flavor stats
mordredfungi: woot14:35
fungizaro: ianw e-mailed the infra ml saying it's up now14:35
zaroahh cool, must have missed it.  Thanks ianw !14:36
fungizaro: lots of ram now!
*** roxanaghe has quit IRC14:36
*** raunak has joined #openstack-infra14:38
*** ddieterly has joined #openstack-infra14:39
*** jheroux has joined #openstack-infra14:40
*** pblaho has quit IRC14:40
openstackgerritMerged openstack-infra/devstack-gate: Remove 'partial' support for Grenade jobs
*** jianghuaw-m has joined #openstack-infra14:42
mat128sdague, pabelanger: mgagne will look at the Internap mtl01 issues (connection timed out and HTTP 500)14:42
*** zul has quit IRC14:42
*** amitgandhinz has quit IRC14:43
*** yamahata has joined #openstack-infra14:43
*** amitgandhinz has joined #openstack-infra14:44
openstackgerritMerged openstack-infra/project-config: Revert "Remove infracloud"
openstackgerritMerged openstack-infra/project-config: Enable puppet checks for fuel-octane
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute011.vanilla to static ansible inventory
*** sdague has quit IRC14:45
*** yamahata has quit IRC14:45
*** yamahata has joined #openstack-infra14:45
AJaegerinfra root, could you review these changes of mine, please? Documentation for bindep: and ; and a tiny requirements update for git-review:
*** vinaypotluri has joined #openstack-infra14:47
*** dizquierdo has joined #openstack-infra14:48
*** notnownikki has joined #openstack-infra14:48
*** raunak has quit IRC14:48
*** jianghuaw-m has quit IRC14:49
*** bauzas is now known as bauwser14:50
*** sdague has joined #openstack-infra14:50
openstackgerritgordon chung proposed openstack-infra/project-config: add in missing gnocchi 2.2 job references
*** csomerville has quit IRC14:51
*** sarob has joined #openstack-infra14:51
*** csomerville has joined #openstack-infra14:51
*** ddieterly is now known as ddieterly[away]14:51
*** xarses has joined #openstack-infra14:52
*** sskripnick has joined #openstack-infra14:53
*** coolsvap_ has joined #openstack-infra14:53
sskripnickhi all. who can help with pbr?14:53
sskripnickfungi: AJaeger: clarkb: ^14:54
fungisskripnick: probably nobody unless you tell us what about pbr you need help with ;)14:55
*** reed_ has joined #openstack-infra14:55
sskripnickI just need to build a package which works >_<14:55
*** niska has quit IRC14:55
*** martinkopec has quit IRC14:55
sskripnickException: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?14:55
*** sarob has quit IRC14:56
sskripnickI've used PBR_VERSION=0.0.3, and still the same14:56
*** HenryG has quit IRC14:56
sskripnickHow do I hardcode a version?14:56
fungisskripnick: can you explain what source code you're working with? a non-sdist tarball from somewhere maybe?14:56
*** kaisers1 has quit IRC14:56
*** andreas_s has quit IRC14:56
*** mjblack has quit IRC14:56
*** reed_ has quit IRC14:57
openstackgerritMerged openstack-infra/nodepool: Rename BuilderScheduler to NodePoolBuilder
sskripnickfungi: I tried to use git tag, PBR_VERSION and hardcode version into setup.cfg14:58
sskripnickall the same14:58
*** kaisers1 has joined #openstack-infra14:58
fungii'm pulling that source now to see if i can spot the issue14:58
AJaegersskripnick: you don't have a file14:58
mordredrcarrillocruz, Shrews, clarkb: this one is ready to go:
sskripnickAJaeger: I have this patch in master already14:59
sskripnickthis file*14:59
mordredfungi, sskripnick: also looking15:00
*** ddieterly[away] is now known as ddieterly15:00
*** Apoorva has quit IRC15:00
*** tonytan4ever has quit IRC15:01
mordredsskripnick: this repo (and that change) work fine for me - what command are you trying to run?15:01
*** Apoorva has joined #openstack-infra15:01
*** jistr is now known as jistr|call15:01
*** tonytan4ever has joined #openstack-infra15:01
*** mjblack has joined #openstack-infra15:01
fungii'm trying building an sdist under tox with that change applied now15:01
*** HenryG has joined #openstack-infra15:01
fungiwhich seems to work15:02
mordredI have built an sdist outside of tox too, which also works15:02
fungii get openstack_app_catalog-0.0.1.dev379.tar.gz15:02
*** stew925 has quit IRC15:02
*** wcriswell has quit IRC15:02
*** skipp has quit IRC15:02
mordredme too15:02
*** senk has joined #openstack-infra15:02
*** mikelk has quit IRC15:02
*** zhurong has quit IRC15:03
AJaegersskripnick: is it failing in OpenStack CI for you?15:03
fungihowever that pep8 job isn't failing on sdist/packaging issues. it's failing because of bad code15:03
*** Apoorva has quit IRC15:03
*** hockeynut has joined #openstack-infra15:03
fungiwell, not "bad" code, but non-conformant code based on the rules it's configured to enforce for that project15:03
mordredbtw - python sdist works just fine without that change too15:04
*** amitgandhinz has quit IRC15:04
*** csomerville has quit IRC15:04
fungii'm also able to install openstack_app_catalog-0.0.1.dev379.tar.gz into a fresh virtualenv just fine15:04
*** amitgandhinz has joined #openstack-infra15:04
*** rcernin has quit IRC15:04
mordredfungi: oslo-incubator isn't a thing anymore, right?15:05
fungimordred: correct15:05
*** hongbin has joined #openstack-infra15:05
AJaegermorning, pleia215:05
fungimordred: in fact, there's a tc resolution to have a cross-project goal ripping out the remaining old oslo-incubator code copies15:05
*** ddieterly has quit IRC15:05
*** rbrndt has quit IRC15:06
sskripnickfungi: AJaeger: I just trying to build and upload a package by hand. Like "python sdist upload" then I install this package on another VM "pip install openstack_app_catalog" and trying to run app-catalog-manage runserver15:06
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Give some examples
AJaegerfungi: fixed the typo, thanks for review ^15:07
*** dprince has quit IRC15:07
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.vanilla to static Ansible inventory
*** dprince has joined #openstack-infra15:07
*** thcipriani|afk is now known as thcipriani15:08
rcarrillocruzany objections if I self-approve myself additions of servers to ansible inventory and cacti ?15:08
rcarrillocruzi'm pushing changes as I provision them fine15:08
*** dprince has quit IRC15:08
*** signed8bit has joined #openstack-infra15:08
mordredrcarrillocruz: nope. do it15:08
*** niska has joined #openstack-infra15:08
*** dprince has joined #openstack-infra15:08
jeblairrcarrillocruz: wfm15:08
mordredrcarrillocruz: I have no way of reviewing their correctness anyway :)15:08
jeblairrcarrillocruz: are they the same in cacti, or do i need to clean anything up there?15:09
fungisskripnick: probably best to start with a complete copy of the traceback pasted to paste.openstack.org15:09
jeblair(as in, same hostnames?)15:09
rcarrillocruzjeblair: i have that removes hpuswest naming15:09
rcarrillocruzthen will approve the new vanilla/chocolate changes15:10
rcarrillocruzah wait15:10
rcarrillocruzyou mean cleaning as in the UI15:10
jeblairrcarrillocruz: cool -- are the historic data for those useful, or should i just delete them when that lands?15:10
rcarrillocruzofc, i assume just by removing from hiera doesn't clean them from cacti DB15:10
jeblairrcarrillocruz: right, it does not15:10
rcarrillocruzi don't think that's useful no15:10
mordredsskripnick: I have reproduced your issue15:10
fungisskripnick: like, is it the "import pbr.version" line in openstack_catalog/ which triggering this at runtime? you don't get any errors when you build the sdist or when you install it with pip?15:10
jeblairrcarrillocruz: (which is partly on purpose, because with some servers, we do want the historic data even after we delete)15:11
jeblairrcarrillocruz: cool, i will delete those after 183 lantds15:11
rcarrillocruzmakes sense15:11
*** sdake_ has joined #openstack-infra15:11
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Disable selinux on RHEL
*** sarob has joined #openstack-infra15:11
mordredfungi: yah15:11
mordredso - I know what the problem is15:11
rcarrillocruzi should find the t-shirt that had a message about 'i fixed it by disabling selinux' or something like that15:12
rcarrillocruzi believe pleia2 had it?15:12
rcarrillocruzre: yolanda change ^15:12
*** jistr|call is now known as jistr15:12
pleia2oh yes, the one from :)15:12
yolandaalways is selinux...15:12
rcarrillocruzthat's awesome15:12
mordredsskripnick: mind if I just amend your commit?15:12
mordredor - how about I do this ...15:12
jeblairmordred: when you have a sec, can you explain how the code in 360771 achieves what is described by "cloud has a network from which floating ips come that is connected via a router to the network from which the private_v4 address came" ?  (i'm assuming it has to do with _get_free_fixed_port, but that looks too simple to do all of that, so i must be missing some subtlety)15:13
fungipleia2: wow, that's some selinux advocacy site for sure15:13
openstackgerritMerged openstack-infra/tripleo-ci: Add swap usage to dstat capture
pleia2fungi: I'm a fan15:13
mordredsskripnick: feel free to squash that into your change and abandon my change15:13
openstackgerritMerged openstack-infra/tripleo-ci: Remove white space between print and ()
mtreinishfungi: heh, I like our cpu usage on firehose:
mordredsskripnick: but I thought showing the solutoin would be useful15:13
*** skipp has joined #openstack-infra15:13
*** rbrndt has joined #openstack-infra15:13
mordredfungi: ^^15:14
jeblairpleia2, fungi: oh the first set of t-shirts is so current and topical too!  :)15:14
* fungi always applies tee-shitrs topically15:14
yolandaalso... i got a working infra-cloud in CentOS for OPNFV15:14
mordredjeblair: I have a sec right now! you are right, _get_free_fixed_port does the trick15:14
*** sdake has quit IRC15:15
sskripnickmordred: it would be great15:15
notnownikkioh boy, i can't even describe how much i'd like to replace our jenkins with zuul right now15:15
mordredjeblair: _get_free_fixed_port has calls to find "nat_network"15:15
sskripnickfungi: there is no errors when building/uploading/installing. only error when trying to run15:15
fungimtreinish: i like green15:15
jeblairmtreinish: did you get that thing you wanted added to zuul?15:15
*** sarob has quit IRC15:16
fungisskripnick: looks like mordred has proposed a fix for you15:16
*** mikelk has joined #openstack-infra15:16
pleia2jeblair: I just have the "setenforce 1" one ;)15:16
rcarrillocruzyolanda: that's awesome work!15:16
rcarrillocruzpabelanger: you might like that ^ :-)15:16
*** kdas__ has joined #openstack-infra15:16
stevemarsilly question, all the ATC codes have already been dished out right? or is there another round coming up?15:16
*** berendt has joined #openstack-infra15:16
fungistevemar: i'm going to send one final round the week after feature freeze (probably ~5th of september)15:17
jeblairmordred: what's nat_destination there?15:17
fungistevemar: the previous batch went out on august 15 so anyone who contributed something that merged prior to that should have a registration discount code already15:18
sskripnickfungi: omg i should be looking at traceback before going here. openstack_catalog/ seems the issue15:18
YorikSarfungi: Thanks!15:18
mordredjeblair: a nat_destination is a network that has a subnet onto which a floating ip can be attached15:18
mordredjeblair: it is detected in _set_interesting_networks15:19
stevemarfungi: ah okay, thanks for the heads up15:19
rcarrillocruzi got somehow invited to the slack channel for the cloud appdev event in guadalajara, cos i gave some session via hangouts15:19
rcarrillocruzand now i get email notifications everytime someone puts '@all'15:19
rcarrillocruzlike, WHY would that be a default setting15:19
jeblairmordred: so that's a filter that says "get me a port on a network who has a subnet which is the subnet that my private ip is on"?15:19
rcarrillocruz /facepalm15:19
*** _nadya_ has joined #openstack-infra15:20
mtreinishjeblair: oh, the extra metadata param for node type. I got sidetracked and haven't looked at it again15:20
mordredjeblair: no, it's saying "get me the port associated with an ip I have that does not have a FIP already on it and that is on a network that can have a FIP attached to it"15:20
mordredjeblair: (which might have been what you were saying and I just used more words)15:20
fungistevemar: worth noting, registration prices increase next week (september 1) so anyone with a code up to now should use it before then or their code won't cover 100% of the cost. the batch i'm sending out after feature freeze cover the increased registration rate since those people won't have an opportunity to register before the rate increase15:21
stevemarfungi: oh thats interesting15:21
jeblairmtreinish: ok; i think i eventually context switched to that and dumped some more thoughts at you in channel while you were away.  :)  if you still want to do that, i can look them up again.15:21
openstackgerritMerged openstack-infra/system-config: Remove hpuswest hosts from Cacti
*** _nadya_ has quit IRC15:22
rcarrillocruzjeblair: landed ^15:22
rcarrillocruzthx for the cleanup offer15:22
jeblairmordred: no, that's not what i was saying, so i guess i'm confused. :(15:22
mordredjeblair: don't feel bad - it's massively confusing15:23
mtreinishjeblair: I don't really have the concentration for it right now, I'm at the ops midcycle15:23
jeblairmordred: why does "get_free_fixed_port" return the port for an existing ip?  the word "free" in there is tripping me up.15:23
jeblairmtreinish: okay.  let me know when you swing back around to it15:23
*** esikachev has quit IRC15:24
mordredjeblair: "free" as in "free destination" - like "a port that is attached to the server but is not attached to something else already"15:24
*** amitgandhinz has quit IRC15:25
jeblairmordred: but if it's "for an ip i have", surely it's attached already?15:25
*** asselin_ has joined #openstack-infra15:25
fungii've learned to stop making those assumptions in openstack15:25
*** amitgandhinz has joined #openstack-infra15:25
openstackgerritDawid Deja proposed openstack-infra/project-config: Add mistral gate for testing kombu driver
fungiotherwise i just get mad when i eventually learn the truth15:26
jeblairmordred: oh wait, "get me the port associated with an ip I have that does not have a FIP already on it" is now confusing -- how can a free port have an ip but not have a fip?15:26
*** kdas__ has quit IRC15:26
*** kdas__ has joined #openstack-infra15:26
*** asselin__ has quit IRC15:27
sskripnickmordred: fungi: now it works. thanks a lot!15:27
*** sarob has joined #openstack-infra15:27
jeblair(i am reading that function now to try to grok it)15:27
*** piet has joined #openstack-infra15:27
mordredjeblair: I'm beginning to agree with you that "free" is a bad word here15:27
*** armax has joined #openstack-infra15:27
fungimordred: sskripnick: however, i'll grant that error message (at least outside the context of the traceback) is mildly misleading in this particular case15:28
jeblairmordred: i'll drop that from my mental model and see if things make more sense :)15:28
jeblairmordred: also, i will see if breakfast helps15:28
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add controller00.vanilla to Cacti
*** derekh has quit IRC15:30
*** sarob has quit IRC15:31
*** pradk has quit IRC15:32
*** roxanaghe has joined #openstack-infra15:33
*** wcriswell has joined #openstack-infra15:33
*** asettle has quit IRC15:33
*** pcaruana has quit IRC15:33
*** pradk has joined #openstack-infra15:33
*** pblaho has joined #openstack-infra15:34
*** esikachev has joined #openstack-infra15:34
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti
*** Julien-zte has quit IRC15:36
openstackgerritMonty Taylor proposed openstack-infra/shade: Rename _get_free_fixed_port to _nat_destination_port
mordredjeblair: ^^ follow up patch that renames the function and adds a better (hopefully) doc string15:36
mordredfungi, sskripnick: it is a bad error message ... although contextually I have not figured out a way to more effectively communicate the error15:37
mordredfungi: perhaps we should just add "It's also possible that the package name in setup.cfg does not match the argument to VersionInfo"15:37
mordredto the error message?15:37
*** roxanaghe has quit IRC15:37
*** matthewbodkin has quit IRC15:38
*** tesseract- has quit IRC15:38
fungimordred: i wonder whether it's possible to distinguish that situation... i guess it's the chicken-and-egg problem of needing to know the package name to find the metadata15:38
*** jamesdenton has joined #openstack-infra15:38
fungiso there's probably no way to tell whether you are missing teh metadata or just got teh package name wrong15:38
fungimordred: i wonder if it could be made more explicit that it looked for (and did not find) package metadata for <your_package_name_here>15:39
openstackgerritMonty Taylor proposed openstack-dev/pbr: Add more words to a confusing error message
*** gyee has joined #openstack-infra15:41
mordredfungi: ^^15:41
mordredfungi: lemme see if I can get your suggestoin in there too15:41
*** zul has joined #openstack-infra15:42
*** eeiden has joined #openstack-infra15:42
*** mikelk has quit IRC15:43
*** mikelk has joined #openstack-infra15:43
*** stewie925 has joined #openstack-infra15:44
clarkbrcarrillocruz: thats great news on the performance front (that test taking a little less than an hour is about where we want things to be I think) also you need to set max servers to 0 if you want images to be built and uploaded. -1 means do nothing but keep the config15:45
rcarrillocruzah nice15:45
openstackgerritMonty Taylor proposed openstack-dev/pbr: Add more words to a confusing error message
rcarrillocruzi was wondering if there was a difference, you just told me :-)15:45
clarkbfungi: re gerrit threads good to have that confirmed. Now I guess we wait and see if we encounter the issues we have been seeing more recently15:45
rcarrillocruzpushing change15:45
clarkbrcarrillocruz: ya 0 means make and upload images but don't boot insatnces and -1 means don't make images and don't boot instances15:46
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Set max-servers on infracloud to 0
*** mikelk has quit IRC15:50
*** senk has quit IRC15:52
pabelangermgagne: sure, no problem. When you have time, I'm free to help15:53
mgagnepabelanger: I'm trying to investigate the issue on my side. So I booted 2 VMs, one ping, one doesn't. So we are investigating it. So yea, I suggest you don't enable it back.15:54
mgagnepabelanger: I tested earlier this week but looks like some compute nodes aren't working that well15:54
pabelangermgagne: we only launched a handful of nodes over the few hours mtl01 was online, I haven't tried launching anything since we disabled mtl0115:54
clarkbmgagne: pabelanger: jsut to double check we opened up the security groups like we do in the other regions? (allow all ipv4 and ipv6 in and out?)15:54
mgagneclarkb: AFAIK, default rules are: any<->any15:55
mgagnebut better double check15:55
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti
*** yamamoto has quit IRC15:55
pabelangerclarkb: let me check with cloud_launcher15:55
*** Marx314 has quit IRC15:56
*** zul has quit IRC15:56
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** jaosorior has quit IRC15:58
*** yamamoto has joined #openstack-infra15:58
AJaegerI'd like another review for a bindep documentation change, please:
*** raunak has joined #openstack-infra15:59
pabelangerclarkb: rcarrillocruz: mordred: Oh, we need to land Adds region support to ansible-role-cloud-launcher15:59
pabelangerthen I can validate our security groups in internap-mtl0116:00
rcarrillocruzpabelanger: are you not core on the launcher?16:00
rcarrillocruzcan't remember if i did you core in there or infra-ansible16:00
rcarrillocruzpabelanger: you are core, feel free to approve the change16:01
rcarrillocruzi also did clarkb core on the project16:01
*** flepied has quit IRC16:01
openstackgerritDawid Deja proposed openstack-infra/project-config: Add mistral gate for testing kombu driver
*** matrohon has quit IRC16:03
openstackgerritMerged openstack-infra/system-config: Add compute010.vanilla to static Ansible inventory
*** Swami has joined #openstack-infra16:04
openstackgerritDavid Smith proposed openstack-infra/jenkins-job-builder: email-ext trigger script
*** hashar has quit IRC16:05
*** e0ne has joined #openstack-infra16:05
*** tqtran has joined #openstack-infra16:07
*** chlong has quit IRC16:07
rcarrillocruzweee, nodepool building images fine so far16:08
*** edtubill has joined #openstack-infra16:08
clarkbwas the 1540UTC restart of nodepool builder to pick up the infra cloud?16:08
clarkb(it killed the ubuntu-trusty build, just making sure that was intentional and not some new bug that needs to be tracked down16:09
fungidhellmann: sphinx question... if i have a document that needs to link to some static plain-text files, do those go in doc/source/_static/ or is there a better place? and what causes the contents of _static to get included in doc/build/html/ ? just sticking files in there doesn't seem to do it16:09
fungidhellmann: scratch that, it did work. i think i need to adjust my urls16:10
fungithe log just doesn't enumerate them the way it does included images16:10
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add baremetal00 to cacti hosts
*** pilgrimstack has quit IRC16:10
*** tqtran has quit IRC16:11
*** cody-somerville has joined #openstack-infra16:13
*** esikachev has quit IRC16:13
*** smarcet1981 has joined #openstack-infra16:13
*** swat30 has left #openstack-infra16:13
openstackgerritMerged openstack-infra/git-review: Remove argparse from requirements
*** sdague has quit IRC16:15
*** smarcet has quit IRC16:16
openstackgerritDavid Smith proposed openstack-infra/jenkins-job-builder: email-ext trigger script
*** senk has joined #openstack-infra16:17
*** itisha has joined #openstack-infra16:18
*** salv-orlando has joined #openstack-infra16:18
*** chlong has joined #openstack-infra16:19
*** shashank_hegde has joined #openstack-infra16:19
*** kaisers_ has joined #openstack-infra16:21
*** e0ne has quit IRC16:21
* clarkb notices that the ntpdate chagne fails on trusty where we have not updated our images as well as xenial16:22
clarkbwill try to get those images updated before approving the d-g ntpdate change16:22
AJaegerfungi, regarding the bindep change with openstack-ci-usage: Should I name that file "Adoption by projects" and have OpenStack as only section in it?16:23
*** AnarchyAo has joined #openstack-infra16:24
fungiAJaeger: my main concern is that it still makes it seem like something which is 1. primarily targeted at ci use cases, and 2. mostly only useful for openstack projects16:25
*** psachin has quit IRC16:25
fungii'd very much prefer to move any documentation on how openstack's ci uses the tool into a documentation repository about openstack's ci16:26
*** kaisers_ has quit IRC16:26
AJaegerfungi, I made this section short on purpose ;) I'm fine just adding a link to our Infra Manual - but I want that link ;)16:26
fungiwe can make that the primary url for people wanting to know how our ci leverages bindep, and then it can link to the general bindep documentation for details on things not directly related to our ci16:27
*** kdas__ is now known as kushal16:27
AJaegerLet me propose something. First patch is with your suggestion - and then I try something different...16:27
*** kushal has quit IRC16:27
*** kushal has joined #openstack-infra16:27
*** hockeynut has quit IRC16:28
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Document OpenStack usage
AJaegerfungi, ignore this review for now ^16:29
*** vhosakot has joined #openstack-infra16:29
* fungi has become remarkably adept at ignoring reviews16:30
*** roxanaghe has joined #openstack-infra16:30
*** bethwhite_ has quit IRC16:30
*** dtantsur is now known as dtantsur|afk16:30
*** sdague has joined #openstack-infra16:30
*** zul has joined #openstack-infra16:30
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add link to Infra Manual
AJaegerfungi, could you persuade you to switch defaults and not ignore this review, please? ^16:31
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
pleia2AJaeger: bit of whitespace there16:31
AJaegerpleia2: thanks, fixed.16:32
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add link to Infra Manual
*** jerryz has joined #openstack-infra16:32
*** sandanar has joined #openstack-infra16:32
* pleia2 thumbs up16:33
AJaegerpleia2: thanks. could you review as well ,please?16:33
pleia2oh yeah, I was looking at this earlier and it got lost in tabs16:33
*** dizquierdo has quit IRC16:34
*** dimtruck is now known as zz_dimtruck16:34
*** sambetts is now known as sambetts|afk16:34
electrofelixjust wondering if the latest zuul codebase still supports jenkins + ansible? wondering if it's possible to upgrade to a release with both and then switch jobs across rather than doing a giant switch16:35
*** cody-somerville has quit IRC16:35
*** psachin has joined #openstack-infra16:35
clarkbelectrofelix: zuul continues to support gearman. So any gearman worker follwing the zuul gearman protocol should work16:36
*** david-lyle has joined #openstack-infra16:36
clarkbelectrofelix: this should include the jenkins gearman plugin and the zuul launcher code though we don't run the jenkins gearamn plugin anymore so I can't vouch for it16:36
AJaegerthanks, pleia2 !16:36
pleia2AJaeger: sure thing16:36
electrofelixclarkb: excellent, at least it's possible so there is an upgrade path16:36
pleia2AJaeger: while I have your attention, I haven't talked to the i18n folks lately, given the time in the cycle, I think Zanata upgrade + Xenial will have to slip until next cycle16:37
pleia2AJaeger: I'm really unhappy about this, but we're just not where we need to be with Xenial just yet (though I do think we're close)16:37
clarkbelectrofelix: its only zuul version 3.0 that will not be backward compatbile with the gearman protocol. Anything onthe 2.x series should continue to do so16:38
AJaegerpleia2: haven't talked to them either ;( Yeah, it's rather late16:38
pleia2I will spend some time next week seeing what we need to support, but major changes now are too late16:38
openstackgerritMerged openstack-infra/project-config: Set max-servers on infracloud to 0
AJaegercould we get translate-dev set up already?16:38
AJaegerthanks, pleia216:38
pleia2AJaeger: we should, I'll work on that16:38
*** lucasagomes is now known as lucas-afk16:39
*** tkelsey has quit IRC16:39
electrofelixclarkb: thanks for the confirmation16:39
*** electrofelix has quit IRC16:39
AJaegerfungi, I have no great idea right now how to explain profiles in the README.rst of bindep better - if you have one, go ahead.16:40
rcarrillocruzpabelanger: it's kind of late for me now to look at the afs stuff on infracloud, got hooked on other inventory and provisioning work16:40
AJaegerfungi, if you're happy with , then I'll abandon
*** jpich has quit IRC16:40
pabelangerrcarrillocruz: no problems, do you want me to stand it up or hold off until you are around?16:40
pabelangerrcarrillocruz: which need a node with 100GB disk or a volume we can attach16:41
rcarrillocruzif the afs thing is just about creating a node with mirror.blah.infracloud, add dns and put mirror class on site.pp, i think i'm ok you looking at it16:41
rcarrillocruzpabelanger: ok, given we have no cinder on the infracloud, then i guess we want to have a flavor with more disk16:41
fungiAJaeger: yeah, i think just having the bindep ci documentation in infra-manual link to the bindep documentation about profiles when it mentions them would be sufficient16:41
rcarrillocruzalthough we could use m1.large , iirc it has 160GB or something16:42
pabelangerrcarrillocruz: Right, so let me add the default into cloud_launcher16:42
pabelangeror we can use that16:42
pabelangerrcarrillocruz: but ya, basically what you said to launch the AFS mirror16:42
AJaegerfungi: ah, that way around - ok. Let me look at infra manual...16:42
fungiAJaeger: i'm not really keen on 361353 either. i think when people need to know about how bindep is used in our ci we should tell them to look in the infra manual. the bindep documentation should be about bindep, not about how we happen to use it in openstack ci16:43
pleia2ah, that's fair16:43
fungiit's perfectly reasonable for the infra manual to link to the bindep documentation for more details though16:44
*** pradk has quit IRC16:44
*** psachin has quit IRC16:44
*** rcernin has joined #openstack-infra16:44
* rcarrillocruz makes a mental note to open a jira for compute005, looks it has a bad hard drive16:44
fungiconsider if we weren't the bindep developers... for example we wouldn't expect the gerrit devs to include a link in their documentation about how openstack uses gerrit16:45
*** links has joined #openstack-infra16:45
*** shashank_hegde has quit IRC16:45
fungibut we have plenty of documentation on how we use gerrit, and link to gerrit's documentation16:45
*** pradk has joined #openstack-infra16:45
AJaegerfungi, already links to bindep - and has a link for "profile".16:46
pabelangerrcarrillocruz: Also, will check cloud-launcher again16:46
rcarrillocruzy, you have +2 powers16:47
pabelangercool, thanks16:47
fungiAJaeger: maybe we should point people there when they ask about bindep in our ci? i forgot we'd added that section16:47
openstackgerritMerged openstack-infra/bindep: Give some examples
pabelangerfungi: clarkb: if you have a moment: adds internap-mtl01 to cloud-launcher16:47
AJaegeryeah, we should point them to our drivers page.16:48
fungiAJaeger: and we can beef up the documentation in that section with your prose from the earlier change16:48
fungiwhich i thought was quite good16:48
*** kien-ha has joined #openstack-infra16:48
*** esberglu has quit IRC16:48
EmilienMnibalizer: how can we get a final approve on ?16:48
EmilienMfungi: if you have time, it's really a quick one ^16:48
openstackgerritMerged openstack-infra/system-config: Add controller00.vanilla to Cacti
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Improve wording on bindep profile
AJaegerfungi, ^16:49
pleia2EmilienM: lgtm16:49
EmilienMpleia2: thx a lot16:49
*** tphummel has joined #openstack-infra16:50
clarkbpabelanger: if we are using that to bootstrap new clouds now is the playbook for that somewhere? and can we document that in the system-config docs "under bringing up a new cloud" or something?16:50
clarkbalso it is entirely possible this ahs been done I just missed it16:51
openstackgerritMerged openstack-infra/project-config: Add initialize-urandom to fedora 24
*** flepied has joined #openstack-infra16:51
rcarrillocruzclarkb: yeah, it's playbooks/run_cloud_launcher.yml16:51
rcarrillocruzand indeed, we should document that16:52
rcarrillocruzinfracloud also needs a LOT of love in terms of documentation16:52
rcarrillocruzstill references the HP1 HP2 regions :/16:52
pabelangerclarkb: Right, I've been running it manually today. However, I think we are stable enough to add a playbook into our ansible wheel.  I'll work on that now16:52
clarkbrcarrillocruz: but where are all the inputs ?16:52
clarkbrcarrillocruz: all taht says is "run this role"16:52
clarkbI am mostly interested in what the cloud is being told to do for example16:53
rcarrillocruzyeah, you need to feed to ansible-playbook -e /opt/system-config/production/cloud_launcher/clouds_layouts.yml16:53
rcarrillocruzclarkb: ^16:53
pabelangerYup, is what we have setup today16:54
clarkbwhy are they in different dirs?16:54
rcarrillocruzand i plan to put on the launcher a param to run the whole thing on a specific cloud16:54
pabelangerI am just cleaning up the last of the password issues with tripleo16:54
rcarrillocruzif you run it that way, it runs against all clousd on oscc file16:54
AJaegerpleia2: could you review as well, please?16:54
clarkbrcarrillocruz: that sounds helpful16:54
*** bin_ has quit IRC16:54
rcarrillocruzso the idea is that you could just ansible-playbook -e @/opt/system-config/production/cloud_launcher/clouds_layouts.yml -e "cloud=infracloud' for example16:55
*** notnownikki has quit IRC16:55
*** kushal has quit IRC16:55
*** sdague has quit IRC16:56
* AJaeger wishes everybody a great weekend...16:56
pleia2AJaeger: done, have a nice weekend :)16:56
clarkbI also semi wonder why it has to be separte files but I guess thats ansible's parser?16:56
fungihave a great weekend, AJaeger!16:56
*** jamesdenton has quit IRC16:56
clarkbbasically one doesn't make sense without the other and they are fully dependent on each other (at least at this point in time) so colocating makes sense to me16:56
AJaegerthanks, pleia2 and fungi!16:56
clarkbpabelanger: is the issue in the new internap region security groups then?16:58
rcarrillocruzclarkb: btw, i'm ok moving clouds_layouts.yml to playbooks folder, providing we put a comment at the beginning "This is a variable file, meant to be passed to run_cloud_launcher blah"16:58
rcarrillocruzto avoid people thinking it's a playbook, just cos it's in playbooks folder and it has .yml extension16:58
clarkbrcarrillocruz: ya I think its easier to discover that way particularly since we don't have docs for it :)16:58
rcarrillocruzi'm not going to disagree with that16:59
clarkbor move the playbook into the cloud launcher dir16:59
rcarrillocruznah, i think the primer may be better for consistency16:59
rcarrillocruzanyway, going away for my weekend16:59
rcarrillocruzhave a good one folks16:59
clarkbyou too16:59
pabelangerclarkb: lets land and see16:59
clarkbpabelanger: its been approved16:59
clarkbrcarrillocruz got it16:59
pabelangerokay cool17:00
clarkbbut it should be easy to do a security group listing against internap and check17:00
pabelangerYa, I did that last night and pretty sure it was open17:00
pabelangerhowever, I might have misread it17:00
*** raildo has quit IRC17:01
openstackgerritMerged openstack-infra/puppet-openstack_infra_spec_helper: Revert "Pin puppetlabs-spec-helper"
*** chlong has quit IRC17:02
openstackgerritMerged openstack-infra/system-config: Add compute00[0-4].vanilla to Cacti
clarkbpabelanger: I only see 2 rules but I think you need 4. ingress + egress for ipv4 and ingress + egress for ipv617:03
mgagnepabelanger: our investigation is still ongoing. So far, we suspect some kind of misconfiguration in our TOR switches where some compute nodes reach for the passive switch instead of the active one.17:03
*** smarcet has joined #openstack-infra17:04
clarkbmgagne: ok, I am trying to double check the 2 security group rules we do have17:04
clarkbmaybe you can make an ingress and egress rule somehow17:04
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Switch to import module instead of function in properties
pabelangerclarkb: sure, I let me add it17:05
kien-hazaro: Hey, you around?17:05
pabelangermgagne: thanks for the update17:06
fungipleia2: were you interested in reviewing the rest of the open changes under topic:wiki-upgrade? i think they're basically complete now17:06
clarkbhrm openstackclient doesn't amke this easy... it only shows you the rule uuid and ip range on the group show output17:06
*** smarcet1981 has quit IRC17:06
pleia2fungi: sure thing17:06
clarkbno protocol or direction data17:06
pleia2fungi: oh hi, that's a lot of patches :D17:07
* pleia2 gets to it!17:07
*** degorenko is now known as _degorenko|afk17:07
fungipleia2: yeah, just don't want them to get stale17:07
fungipleia2: thanks!17:07
*** sarob has joined #openstack-infra17:07
fungipleia2: feel free to approve any you're comfortable approving. i won't be reenabling puppet for production until i can correctly build a replacement wiki.o.o and a working wiki-dev.o.o with the current state of the puppet-mediawiki and system-config repos17:08
pleia2fungi: noted, thanks17:08
*** manjeets_ has joined #openstack-infra17:09
*** Apoorva has joined #openstack-infra17:10
clarkbhrm I can't seem to use -c direction to get direction...17:10
*** yamamoto has quit IRC17:10
clarkbmight have to use neutron client17:10
kambizpabelanger: I think the rdo ci ntp issue is resolved.  slagle is replying to the thread.17:10
pabelangerkambiz: great17:11
openstackgerritMerged openstack-infra/system-config: Add compute006 and compute007 vanilla to cacti
openstackgerritMerged openstack-infra/system-config: Add baremetal00 to cacti hosts
*** Genek has joined #openstack-infra17:12
*** Genek has joined #openstack-infra17:12
pabelangerclarkb: okay, cloud-launcher region patch merged, I can run it manually now if you want17:12
*** sarob has quit IRC17:12
pabelanger360707 is in the gate now too17:12
clarkbpabelanger: one sec, trying to list these rules and osc is fighting me17:13
*** _nadya_ has joined #openstack-infra17:13
*** ilyashakhat has joined #openstack-infra17:13
*** njohnston has joined #openstack-infra17:13
openstackgerritMerged openstack-infra/infra-manual: Improve wording on bindep profile
clarkbalso noticing osc will occasionalyl take a long time to talk to that region potentially due to the switch issues mgagne was talking about?17:13
clarkbpabelanger: so not sure if playbook would even work right this moment17:14
clarkbof course could be pebkac on my part (but like half the commands do work so I dunno)17:14
*** shashank_hegde has joined #openstack-infra17:15
*** _nadya_ has quit IRC17:15
*** asselin__ has joined #openstack-infra17:16
*** mwhahaha has quit IRC17:16
pabelangerclarkb: I haven't notices the delay, but I haven't tried today17:16
clarkbpabelanger: currently `openstack --os-cloud openstackjenkins-internap --os-region-name mtl01 security group list` is failing17:17
clarkbor rather not returning data very quikcly17:17
clarkb(I set the env var for the all clouds clouds.yaml file too, if I get the cloudname or region name wrong it immedaitely returns and complains at me)17:17
*** asselin_ has quit IRC17:17
clarkbpabelanger: I would say go ahead wit hthe playbook, I just got "Connection failure that may be retried."17:19
clarkbpabelanger: curious to see if ansible + shade fair better17:19
*** shashank_hegde has quit IRC17:20
zarokien-ha: what up?17:20
clarkbstevemar: dtroyer_zz is ^ a known issue with osc?17:20
pabelangerclarkb: sure, give me a sec17:20
kien-hazaro: Just wanted you to do a really quick review for a patch.17:21
*** rcernin has quit IRC17:22
kien-hazaro: The patch can be found in the link I provided. You got a moment to spare for me?17:22
*** esikachev has joined #openstack-infra17:22
stevemarclarkb: definitely not a known issue17:23
stevemarclarkb: using version 2.6.0 or have you upgraded?17:24
manjeets_anyone ever seen this issuse while stacking
clarkbI am going to test against another cloud just to rule out switch issues17:24
clarkbstevemar: give me a minute to do that and get the version info17:24
clarkbstevemar: 3.1.0 here17:24
stevemarsuper :P17:24
*** yamahata has quit IRC17:25
clarkbya just got the same thing against bluebox "Connection failure that may be retried."17:25
clarkbstevemar: running openstack --os-cloud openstackzuul-bluebox security group list17:25
stevemarclarkb: so no, not sure whats going on, and it's sad to hear that you're not getting enough info about a security group when trying to display a particular one17:26
zarokien-ha: looking17:26
*** DericHorn-HP has joined #openstack-infra17:26
*** kzaitsev_mb has quit IRC17:26
*** sputnik13 has joined #openstack-infra17:26
clarkbstevemar: when it was semi working the info was very terse so I upgraded osc and now it just doesn't work at all17:26
*** esikachev has quit IRC17:26
clarkb(I thought maybe the info might be less terse with newer version :) )17:26
openstackgerritMerged openstack-infra/system-config: Add internap to cloud_launcher
*** sdague has joined #openstack-infra17:27
*** piet has quit IRC17:27
stevemarclarkb: whats the --debug output say?17:27
mgagneclarkb: are you talking about API or SSH access?17:28
*** tosky has quit IRC17:28
clarkbmgagne: api via osc17:29
clarkbstevemar: paste in a saec17:29
clarkbmgagne: its happening with osc against bluebox though so I doubt its internap at fault for this17:29
mgagneclarkb: api shouldn't have any issue since it's hosted in a different system/rack17:29
clarkbmgagne: kk17:30
pabelangerclarkb: cloud-launcher running now17:31
zarokien-ha: lgtm17:31
*** piet has joined #openstack-infra17:32
kien-hazaro: Thanks a lot!17:32
*** aviau has quit IRC17:33
*** aviau has joined #openstack-infra17:33
pabelangerclarkb: it does look like we missed an ingress rule for ipv4/ipv6 for one of the projects17:34
clarkbpabelanger: looks like one missed one rule and the other missed two which lines up with what I saw if the one missing two was the one I was looking at17:35
dtroyer_zzstevemar, clarkb: that connection retry seems like it might be the current known issue with the SDK:
openstackLaunchpad bug 1616522 in OpenStack SDK "openstacksdk 0.9.2 and higher does have one issue "SDKException: Connection failure that may be retried." when using python-openstackclient performing neutron part action" [Undecided,In progress] - Assigned to Zhao Jian (jianzj)17:36
dtroyer_zz is up to fix it17:36
*** zz_dimtruck is now known as dimtruck17:37
clarkboh is it using http when it should be using https, interesting17:37
clarkbmy log output does confirm the url is http17:38
* clarkb checks catalog17:38
clarkbcatalog says we should use https for neutron. I agree that this is likeyl the same bug. Thank you for pointing that out17:38
*** AnarchyAo has quit IRC17:39
dhellmannfungi: sorry, I was offline when you had your sphinx question. is that all sorted now?17:39
clarkbpabelanger: before we put mtl01 back into circulation we should run a devstack-gate on an instance there. I am happy to do that if you like17:39
clarkbpabelanger: and if that works and mgagne gives us the go ahead we can probably turn it back on17:40
pabelangerclarkb: WFM17:40
mgagnepabelanger: I wouldn't enable back mtl01 until we (I) tell you to =)17:40
clarkbmgagne: yup yup17:40
clarkbmaybe I will wait for mgagne then I will test :)17:40
clarkband in the meantime find some breakfast17:40
*** shashank_hegde has joined #openstack-infra17:41
*** sarob has joined #openstack-infra17:42
fungidhellmann: yep, totally figured out17:42
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update TestNG to use valid_dict property of convert xml
fungidhellmann: it was a mix of me failing at anonymous hyperlinks in rst and not realizing that static file copying is not logged as verbosely as image copying17:42
dhellmannfungi : oh, interesting. I wonder why that's different.17:43
*** ericksonsantos has quit IRC17:43
dhellmannmaybe because they don't want to report all the css and js files17:43
*** Apoorva has quit IRC17:45
*** sarob has quit IRC17:45
fungidhellmann: yeah, it could just be that it would be _too_ verbose17:45
*** berendt has quit IRC17:47
fungii need to disappear for a couple hours to run errands, but will be back later17:48
*** jheroux has quit IRC17:49
anteayafungi: happy errands17:50
*** gyee has quit IRC17:50
*** rhallisey has quit IRC17:53
*** ihrachys has quit IRC17:55
*** jed56 has quit IRC17:55
*** tpsilva has joined #openstack-infra18:00
*** sarob has joined #openstack-infra18:00
*** rbrndt has quit IRC18:01
openstackgerritMerged openstack-infra/system-config: Clean up tripleo-test-cloud-rh1 os-client-config
*** sdake_ has quit IRC18:02
*** eharney has quit IRC18:03
*** yamahata has joined #openstack-infra18:03
*** tqtran has joined #openstack-infra18:04
*** sarob has quit IRC18:05
*** sandanar has quit IRC18:06
*** ociuhandu has quit IRC18:06
*** senk has quit IRC18:08
*** kaisers_ has joined #openstack-infra18:10
*** yamamoto has joined #openstack-infra18:10
*** Na3iL has quit IRC18:11
*** kien-ha has quit IRC18:11
*** sarob has joined #openstack-infra18:12
*** sarob has quit IRC18:13
*** chem|off` is now known as chem18:14
*** kaisers_ has quit IRC18:15
*** yamamoto has quit IRC18:15
*** matrohon has joined #openstack-infra18:16
*** bnemec is now known as beekneemech18:16
*** ilyashakhat has quit IRC18:17
*** _ari_ has quit IRC18:19
*** _nadya_ has joined #openstack-infra18:19
jlvillalclarkb: Thanks for the multi-node info. good stuff :)18:20
openstackgerritMerged openstack-infra/jenkins-job-builder: Switch to import module instead of function in properties
pabelangerrcarrillocruz: cloud_launcher now properly working. All password for defined clouds correct18:24
mordredpabelanger, rcarrillocruz: WOOT18:25
clarkband it added 3 ingress rules for us in internap mtl0118:25
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
pabelangerclarkb: all the same?18:25
mgagneclarkb: let me know what they are so we can see if we can add them to our default rule set18:26
*** dprince has quit IRC18:26
clarkbpabelanger: no one was one account and just ipv4 and the other two were in the other accuont and ipv4 + ipv618:26
pabelangerokay, cool18:26
*** bnemec has joined #openstack-infra18:26
clarkbmgagne: its just allow all traffic ingress over ipv4 and all traffic ingress over ipv6 according to the log pabelanger posted18:26
*** beekneemech has quit IRC18:26
*** bnemec is now known as beekneemech18:27
mgagneclarkb: I thought it was already allowed by default18:27
*** Apoorva has joined #openstack-infra18:27
pabelangermgagne: ^18:27
clarkbmgagne: neutron blocks everything by default iirc18:27
pabelangerthat is our defaults now18:27
mgagnenot with us =)18:27
*** dtardivel has quit IRC18:27
clarkbmordred: can you point me where I can be useful on the nodepool private ipv4 address stuff18:27
pabelangerlocalhost                  : ok=128  changed=0    unreachable=0    failed=018:27
mgagneclarkb: we have a custom extension/code which injects OUR default rules which are: any ipv4, any ipv618:27
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-log_processor: Switch to using the new log_processor project
*** Apoorva has quit IRC18:28
mgagneclarkb: rules that should be applied first time default SG is read/used:
clarkbmordred: fwiw I am looking on an osic instance and the private file does have its ipv4 addr in there18:28
clarkbmordred: so maybe things are working as expected right now?18:29
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes
*** cody-somerville has joined #openstack-infra18:29
clarkbmgagne: it looked like we had egress rules for that but not ingress?18:29
*** sarob has joined #openstack-infra18:29
clarkbmgagne: I am just going off of what pabelanger's ansible run did since osc is currently not functional for neutron things over https18:29
mgagneclarkb: I know egress isn't filtered in Neutron. I will check with my own account. maybe our tests didn't triggered that code path for some reasons. will check18:30
*** _nadya_ has quit IRC18:30
clarkbmordred: or did the restart happen and I missed it?18:30
mgagnestrange, my account is fine :-/18:31
*** Shrews has quit IRC18:31
*** phschwartz has quit IRC18:31
*** mordred has quit IRC18:31
pabelangerclarkb: mgagne: I actually think it might have been openstackci-internap-nyj01 that changed18:31
clarkbpabelanger: oh huh18:31
*** csomerville has joined #openstack-infra18:31
clarkbin any case its hard to independently verify without getting a different client18:31
pabelanger996689aad9074f8d8bf8bbb36f48084f was the tenant_id18:32
pabelangerI'll look it up shortly18:32
*** shardy has quit IRC18:33
*** cody-somerville has quit IRC18:34
rtheisIs there a known issue with gate jobs resetting?   We've been seeing that on several networking-ovn patch sets and I'm not sure how to track that down.18:34
pabelangerrtheis: which review?18:34
rtheisgate-tempest-dsvm-networking-ovn-native-services has ran to near completion several times18:34
clarkbrtheis: I am guessing its related to the ipv6 only cloud as I noticed a coupe ovn related jobs with ansible rc 3s18:35
pabelangerYa, I think we still have some jobs that need patches18:35
clarkbrtheis: basically make sure that you can run on an instance that gets its networking via ipv6 RAs18:35
clarkbrtheis: in devstack we had to set the public interface's accept_ra sysctl value to 2 because neutron sets forwarding to 118:35
*** asettle has joined #openstack-infra18:36
rtheisclarkb: ah, and the networking-ovn native services job doesn't use neutron L3 agent so it may not be setting that18:36
*** tkelsey has joined #openstack-infra18:36
*** csomerville has quit IRC18:37
clarkbhrm mordred just ping timed out :/18:37
rtheisclarkb: can I tell if accept_ra is being set by looking in the logs for the job?18:37
*** rvasilets___ has joined #openstack-infra18:37
clarkbnodepool has not restarted since the 12th so I think the current nodepool code will just work with osic and multinoded18:37
pabelangertelnet://2001:4800:1ae1:18:f816:3eff:fe79:bea2:19885 should fail18:37
clarkbrtheis: maybe? it would be in the devstack log if so18:37
clarkbrtheis: in any case it should be pretty easy to reproduce. Get an instance with ipv6 route provided via RA and run the test on it18:38
clarkbthe result should be the route goes away if you set forwarding = 118:38
haleybrtheis: see my email to the dev list on wednesday, or find the patch in devstack for more info18:38
clarkbunless you set accept_ra = 2 on the interface18:38
*** ilyashakhat has joined #openstack-infra18:39
rtheisclarkb, haleyb: thank you18:39
* kambiz shakes fist at pabelanger18:39
*** amotoki has quit IRC18:40
*** piet has quit IRC18:40
kambizpabelanger: that's for asking a very sensible question of asking for a network diagram.  If you only knew how messy it is.18:40
*** tkelsey has quit IRC18:41
*** piet has joined #openstack-infra18:41
*** mordred has joined #openstack-infra18:41
clarkbI am going to work on cleaning up leaked iamges in our clouds while waiting for mordred to come back18:41
pabelangerkambiz: Ya, I've asked that question about 15 times :)18:42
kambizpabelanger: lol I guess I might have only seen that for the first time in that thread from hell. (sorry)18:42
mordredpabelanger: the answer is "green"18:42
clarkbmordred: did you see my message about how the private IP looks good on current osic instances? nodepoold hasn't restarted since teh 12th so no fixes are in that I know of18:42
*** yamamoto has joined #openstack-infra18:43
clarkbmordred: so I guess the question is if nodepool restarted right now would it continue to work like that or is it new behavior that is going to break us until your fixes happen18:43
pabelangerkambiz: Ya, not no progblems. I'm still trying to navigate how that whole system works18:43
mordredclarkb: that is a great question. let me run some tests against osic18:43
clarkbmordred: beacuse i would like to run multinode tests there considering how much quota it represents :)18:43
clarkbmordred: great thanks. I will continue to clean up leaked images in the mean time18:44
*** smarcet1981 has joined #openstack-infra18:44
*** asettle has quit IRC18:45
mordredclarkb:  we run nodepool on releases, right?18:45
mordred(of shadE)18:45
*** smarcet has quit IRC18:46
clarkbmordred: let me double check but pretty sure we do18:46
clarkbmordred: shade==1.11.1  # git sha 273ea8c is what we have installed now18:46
clarkbwe have leaked 26 images on osic compared to the 290 in ovh-bhs1 fwiw18:47
clarkbso definitely something going on to make it so much worse in ovh18:47
*** Sukhdev has joined #openstack-infra18:47
*** csomerville has joined #openstack-infra18:48
mordredclarkb: with that same ref locally, I do not show private_v4 addresses being appropriately returned by shade18:48
mordredclarkb: where are you looking to see that they are working as expected?18:48
*** Apoorva has joined #openstack-infra18:48
clarkbmordred: I logged into the instances and checked their files in /etc/nodepool (which is what the multinode tests depend on)18:49
*** yamamoto has quit IRC18:49
*** eharney has joined #openstack-infra18:49
*** ociuhandu has joined #openstack-infra18:50
mordredclarkb: oh - well, it's working accidentally18:50
mordredclarkb:         self.subnode.ip_private = server.get('private_v4')18:50
mordred        if not self.subnode.ip_private:18:50
mordred            self.subnode.ip_private = server.get('public_v4')18:50
clarkbthats actually intentional18:50
YorikSarHello, Infra masters. Is there any way to make jobs in Zuul start not in the root of the repo's work dir?18:51
clarkbbut I guess an accident that it handles this case too18:51
mordredthe server record says:18:51
mordred 'private_v4': '',18:51
mordred 'public_v4': u'',18:51
clarkbmordred: so am I ok to reenable multinode there?18:51
mordred 'public_v6': u'2001:4800:1ae1:18:f816:3eff:fec7:88b5',18:51
mordredclarkb: I think so18:51
*** zul has quit IRC18:51
clarkbmordred: given the accident should be deterministic? ok18:51
mordredclarkb: I thought originally that we had determined this was the case but you still thought there was a problem for reasons I forget18:51
YorikSarWe have puppet pieces stuck in a subdir in the repo and want to lint it:
clarkbYorikSar: yes, you can edit your zuul cloner settings for the job18:52
*** kzaitsev_mb has joined #openstack-infra18:52
clarkbYorikSar: or just cd :)18:52
mordredclarkb: but it took you a few minutes to come to the conclusion that the settings going into the files wouldn't work and that we needed to fix them to be correct18:52
clarkbmordred: oh I thought you said the public v6 addr was going into the file18:52
mordredah. nope18:52
clarkbmordred: maybe you meant public v4 and I misparsed18:52
*** Apoorva has quit IRC18:52
mordredthe actual private address, which becomes server['public_v4'] will go into the file as both public and private18:53
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes
YorikSarclarkb: Well, I hope to reuse existing macros and job templates as much as possible, so sticking cd in there is not an easy task.18:53
mordredso I agree - I think it's fine for now, and we'll know pretty immediately if we're wrong18:53
YorikSarclarkb: I'll loog into cloner cfg then.18:53
mtreinishmordred, fungi, nibalizer, clarkb: if you get a sec can you check health.o.o it's being quite sluggish and the ram usage is quite high:
mgagnepabelanger: I think mtl01 will have to wait until Monday (at least), leaving ops meetup soon so won't have connectivity with our ops team until ~ Monday.18:56
openstackgerritClark Boylan proposed openstack-infra/project-config: Revert "Remove OSIC from 2-node label types"
mgagnepabelanger: will poke back if I get more updates18:56
*** tqtran has quit IRC18:56
clarkbmordred: pabelanger ^ that should be ready to go if you can review quickly18:56
*** ociuhandu has quit IRC18:56
clarkbmgagne: roger18:56
clarkbmtreinish: ya give me a sec to find a good stopping place on this leaked image cleanup18:57
*** e0ne has joined #openstack-infra18:57
*** markvoelker has quit IRC18:57
*** tqtran has joined #openstack-infra18:57
pabelangermgagne: great, thanks again18:57
pabelangerclarkb: +318:58
*** phschwartz has joined #openstack-infra18:58
openstackgerritKhai Do proposed openstack-infra/system-config: doc: add Storyboard integration info
*** sdague has quit IRC18:59
clarkbpabelanger: danke18:59
clarkbmtreinish: top says an apache is using all the cpu and much of the memory18:59
*** markvoelker has joined #openstack-infra19:00
* clarkb tries to figure out what is making it unhappy19:00
*** rhallisey has joined #openstack-infra19:00
*** piet has quit IRC19:00
*** piet has joined #openstack-infra19:01
clarkbmtreinish: there is a lot of "[Fri Aug 26 17:29:17.078766 2016] [:error] [pid 17723:tid 140502063863552] OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')" in the error logs19:01
YorikSarclarkb: Looks like Cloner is not a way to go: it seems to operate with repos as whole only.19:01
clarkbYorikSar: our defautl setup for zuul cloner is to clone into . though19:02
clarkbYorikSar: so if you want to chagne it to clone somewhere else you will have to udpate the config for it19:02
openstackgerritKhai Do proposed openstack-infra/system-config: doc: add Storyboard integration info
YorikSarclarkb: I basically need to have "cd deployment/puppet/octane_tasks" after cloner finishes, but before the job is started.19:03
clarkboh you want to be further into the repo not out of the repo entirely19:03
*** rbrndt has joined #openstack-infra19:03
clarkbyes for that I think you need to use cd or tell your linter where to lint from. pep8 can do this iirc not sure about puppet lint19:03
*** Shrews has joined #openstack-infra19:04
mtreinishclarkb: hmm, so it's dropping connection to the trove node?19:05
mtreinishclarkb: is it getting OOM'ed or something19:05
YorikSarI guess I need to add some parameter to the puppet-syntax/puppet-lint builder and add 'if [ {start_from_dir}]; then cd {start_from_dir}; fi' to the beginning of their scripts.19:05
clarkbmtreinish: I don't see any recent OOMs on health19:06
*** sarob has quit IRC19:06
clarkbmtreinish: I haven't checked teh db side yet as I am still looking for the config file for healt so I can mysql client to the db19:06
*** ociuhandu has joined #openstack-infra19:06
clarkbYorikSar: or see if you can just pass the dir to lint as a parameter?19:07
*** vhosakot has quit IRC19:07
YorikSarclarkb: Is there any way to pass down parameters to builders from project.yaml level? I mean, to pass them down to all builders in all jobs in the job-group.19:07
mtreinishclarkb: I don't know the root password or user but you can get at it via the logstash.o.o proxy19:07
mtreinishclarkb: it should be /etc/openstack-health* something19:07
mtreinishmaybe openstack-health-api.conf19:08
clarkbno there isn't any /etc/openstack-health. The readme says to pass the path but our apache wsgi stuff doesn't so must be whatever is default19:08
*** csomerville has quit IRC19:08
YorikSarclarkb: Oh, it hickups on 'bundle install' (no Gemfile in the root of the repo) or on 'rake line' (no Rakefile in the root of the repo)19:08
mtreinishclarkb: it should be '/etc/openstack-health.conf'19:09
mtreinishthat's the default path in code19:09
clarkbah its the file not the dir found it19:09
clarkbYorikSar: you could add those to the root of the repo?19:09
pabelangerYorikSar: clarkb: I would write a builder specific to fuel-octane then work to migrate the job to use the default puppet-lint jobs.  EmilienM has a good job making those jobs pretty generic today, and trying to shoehorn other project layouts doesn't seem like the best approach19:09
*** salv-orlando has quit IRC19:10
EmilienMpabelanger, YorikSar: where is the patch?19:10
*** piet has quit IRC19:11
YorikSarclarkb: I don't think that will help: Gemfile should be in the root of the gem afaiu...19:11
pabelangerEmilienM: was the module in question which fails to puppet-lint because of the location of the directory19:11
*** piet has joined #openstack-infra19:11
EmilienMoh dear19:11
EmilienMa puppet module in a repo19:12
YorikSarpabelanger, EmilienM: Yeah, both puppet-lint and puppet-syntax jobs hickup.19:12
EmilienMthis day started well19:12
EmilienMYorikSar: adding a puppet module in a directory within a repo is a terrible idea19:12
YorikSarEmilienM: It's complicated :) It's actually not a module for octane, it's a part of octane :)19:12
clarkbmtreinish: ok I am connected to the db so its up now at least. Let me see if I can ask mysql for more running infos19:12
YorikSarEmilienM: Like a part of the tool is written in Puppet.19:13
mtreinishclarkb: I logged into the mysql server too and it's also really slow19:13
YorikSarEmilienM: (is going to, we're experimenting with that)19:13
mtreinisherr, maybe only somewhat slow19:13
EmilienMYorikSar: you should have create a module called openstack/puppet-octane19:13
EmilienMYorikSar: that is how OpenStack is doing Puppet modules, we have a forge with nearly 40 puppet modules19:14
mtreinishnm, it's working fine it's probably just my network here19:14
clarkbmtreinish: | Connections                                   | 2037348        | not sure if that number is cummulative or point int time19:14
EmilienMYorikSar: I don't see why Fuel would make exceptions here19:14
YorikSarEmilienM: afaiu 'puppet-SOMENAME' is a puppet module to setup software SOMENAME. We don't have the same relation between octane and octane_tasks.19:15
clarkbmtreinish: I dont' really see anything else in show status that looks odd19:15
mtreinishclarkb: iirc connections is the total connection attempts19:15
YorikSarEmilienM: octane_tasks contains tasks with Puppet modules that are called from within octane.19:16
mtreinishthreads_connected is the active one19:16
clarkb| Aborted_clients                               | 39362          |19:16
EmilienMYorikSar: you lost me19:16
mtreinishbut it's been a while since I looked that up in the docs19:16
EmilienMbut as a good news, I just understans the fuel/octane wording :) really fun19:16
clarkbwe have 10 threads connected19:16
mtreinishclarkb: I'm fine just restarting the api server on health.o.o and hoping this doesn't happen again19:16
openstackgerritJames Slagle proposed openstack-infra/project-config: Add new promote jobs for TripleO
*** ociuhandu has quit IRC19:17
clarkbmtreinish: ok let me start with that19:17
YorikSarEmilienM: Octane does some deployment stuff that touches Fuel itself and OpenStack deployed by Fuel. Everything is currently written in Python, but we would like to move away from that by calling out from Python to Fuel that would run some Puppet code on OpenStack nodes (or Fuel node).19:18
clarkbmtreinish: thats done19:18
mtreinishclarkb: ok thanks, let me query it19:18
YorikSarEmilienM: So basically we have pieces of Python code within Octane tool being replaced with Puppet code.19:18
clarkb#status log restarted apache2 on health.o.o to remove a runaway apache process using all the cpu and memory. Looked like it may be related to mysql connections issues. DB currently looks happy.19:19
openstackstatusclarkb: finished logging19:19
mtreinishclarkb: yep it's responding now: renders things again19:20
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Support promoting based on $TOCI_JOBTYPE
*** rodrigods has quit IRC19:21
*** rodrigods has joined #openstack-infra19:21
*** ociuhandu has joined #openstack-infra19:21
openstackgerritMerged openstack-infra/project-config: Revert "Remove OSIC from 2-node label types"
*** senk has joined #openstack-infra19:22
YorikSarEmilienM: BTW, running tests from a subdir might be a good feature for generic test running: we shouldn't force projects to be split by the test boundaries.19:22
*** sridhar_ram is now known as sridhar_ram_afk19:22
*** ilyashakhat has quit IRC19:23
*** Na3iL has joined #openstack-infra19:24
*** raunak has quit IRC19:25
openstackgerritArmando Migliaccio proposed openstack-infra/reviewday: Make neutron script pull all Stadium projects
openstackgerritEmilien Macchi proposed openstack-infra/project-config: tripleo: align CI scenarios with Puppet CI
openstackgerritArmando Migliaccio proposed openstack-infra/reviewday: Make neutron script pull all Stadium projects
EmilienMYorikSar: I don't think we need this feature19:25
clarkbthe way it works in python land is you just edit your tox.ini19:26
EmilienMYorikSar: it's encouraging people to add puppet modules in their repo while it's extremely bad practise19:26
clarkbis there no equivalent to that in your rakefile?19:26
EmilienMYorikSar: I don't support this idea.19:26
clarkbthen all the jobs have to do is run rake and not care about specifics19:26
pabelangerYorikSar: EmilienM: Agreed, I still breaking that puppet task to an example repo is the correct thing to do. You don't need to call it puppet-octane, it could be puppet-octane_tasks. And there is not hard rule to say how you are using puppet can't be done in an external module19:26
pabelangerYorikSar: EmilienM: then close the repo locally, or puppet install (cannot remember syntax) and use the --modules_path flag19:27
EmilienMpabelanger: agree19:30
*** e0ne has quit IRC19:30
EmilienMYorikSar: it's like asking running tox inside a project directory19:32
EmilienMdoes osinfra allows it?19:32
YorikSarEmilienM: I think it should :)19:33
YorikSarEmilienM: Google has one huge repo for everything they have after all, so putting more than one things in one repo shouldn't be the end of the world.19:33
*** links has quit IRC19:33
EmilienMI didn't know we were Google19:34
EmilienManyway, you're free to propose an infra patch to be able to run lint tests using our builder19:34
YorikSarEmilienM: btw, do you have smth for me to read on about why is it a bad practise?19:35
YorikSarEmilienM: you mean puppet-lint?19:35
EmilienMYorikSar: I've been working on Puppet for several years, have been leading Puppet work in OpenStack and also core contributor in Puppetlabs modules, I haven't seen this practise before.19:36
EmilienMYorikSar: yes lint19:36
*** senk has quit IRC19:38
clarkbno we expect the top level to drive whatever is under it19:40
clarkbnot for the jobs to magically know how to tox somewhere lese19:40
YorikSarEmilienM: While I'm still not convinced why having Puppet code in Python repo, we can't do that split right now. So we'd like to have an option to run linters in subdir at least as an interim solution.19:41
*** gordc has quit IRC19:41
*** tqtran has quit IRC19:42
clarkblike I said, could you put the rakefiel thing at the top level dir just like the tox.ini?19:42
YorikSarclarkb, pabelanger: so if I propose a change that adds parametrised 'cd' at the top of puppet-lint builders, you all will -1 it?19:42
clarkband have it execute only on the inner dir?19:42
YorikSarclarkb: It will work for rake, it won't work for Gemfile afaiu.19:42
*** tqtran has joined #openstack-infra19:42
*** dprince has joined #openstack-infra19:43
clarkbbut rake is what the job runs right?19:43
clarkbrake lint or something iirc19:43
YorikSarclarkb: and puppet-syntax builder does 'bundle install' in the root of the repo prior to everything.19:43
*** esikachev has joined #openstack-infra19:43
YorikSarclarkb: Yes, puppet-lint builder does that.19:44
pabelangerYorikSar: I'd rather see you write your own builder for fuel-octane then force the existing puppet-lint job to handle it.  When work on migrating when you are able too19:44
*** gyee has joined #openstack-infra19:44
pabelangerBut, I feel you could setup a Rakefile to hand the subdirectory19:45
clarkbosic and ovh have had their images cleaned. Internap is running now19:46
pabelangerclarkb: thanks!19:46
YorikSarpabelanger: It won't work for puppet-syntax builder.19:46
clarkbmordred: I think some of these leaks may happen when we restart the build when it is uploading19:46
pabelangerclarkb: I noticed ovh-bhs1 is still running an old ubuntu-xenial image19:46
clarkbmordred: FYI for when you want to debug it more19:46
clarkbpabelanger: ya 10 days or something19:46
clarkbpabelanger: it is having the most trouble uploading of all the regions. Hoping today's uplaods go better for it19:47
pabelangerYorikSar: maybe us gate-puppet-openstack-libraries-puppet-syntax19:47
pabelangeror gate-puppet-openstack-libraries-puppet-lint19:47
YorikSarpabelanger: Thanks, I'll take a look at them.19:48
pabelangergate-{name}-puppet-syntax-{puppet_version}-{node} should even work with the right Rakefile19:48
*** HeOS has joined #openstack-infra19:48
pabelangerclarkb: cool, that seems to be the only failures with recently19:49
*** sridhar_ram_afk is now known as sridhar_ram19:49
pabelangerclarkb: just seen an osic-cloud1 failure19:49
*** raildo has joined #openstack-infra19:49
cloudnulloh, noes19:49
pabelangerwas a subnode19:50
*** berendt has joined #openstack-infra19:50
pabelangershould have worked19:50
pabelangergoing to test19:50
cloudnullah not specifically an issue with the cloud itself.19:51
pabelangercloudnull: :)19:51
* cloudnull was thinking things were on fire19:51
clarkbpabelanger: weird, I think maybe its possible that that ran prior to iptables ruels being applied out of the persistent thing?19:51
pabelangerclarkb: ya, maybe19:51
clarkbpabelanger: or possibly related to firewalld and maybe we have that breaking stuff on centos7 for us again19:51
pabelangerlet me see if we launch a ubuntu one19:52
pabelanger2001:4800:1ae1:18:f816:3eff:fe46:193c should be ubuntu-xenial-2-node19:53
clarkbyou can boot the centos7 image and check the same thing19:56
clarkbI think it could either be a race or it is possibly firewalld coming to visit again19:57
*** esikache1 has joined #openstack-infra19:57
pabelangerya, have a system online now19:58
*** asettle has joined #openstack-infra19:59
*** asettle has quit IRC19:59
*** kaisers_ has joined #openstack-infra19:59
*** esikachev has quit IRC19:59
*** flaviof has quit IRC20:01
*** fdegir has quit IRC20:01
*** tdurakov has quit IRC20:01
*** gouthamr has quit IRC20:01
*** ashtokolov has quit IRC20:01
*** scottda has quit IRC20:01
*** simonmcc has quit IRC20:01
*** vdrok has quit IRC20:01
pabelangerclarkb: looks like ip6tables is its own service for centos-720:02
*** jraim has quit IRC20:02
pabelangerwhich isn't running20:02
*** esikachev has joined #openstack-infra20:02
*** betherly has quit IRC20:02
*** maximov has quit IRC20:02
*** ameade has quit IRC20:02
fungithat's wacky20:02
*** tpsilva has quit IRC20:02
*** skath has quit IRC20:02
*** zhiyan has quit IRC20:03
pabelangeronce enabled are rebooted, ip6tables -L works20:03
pabelangerwell, our opentack-INPUT label is added20:03
*** esikache1 has quit IRC20:03
*** aimeeu has quit IRC20:04
*** kaisers_ has quit IRC20:04
clarkbbluebox leaked images are clearing out now20:04
*** rvasilets___ has quit IRC20:04
*** serverascode has quit IRC20:04
*** NobodyCam has quit IRC20:04
*** rpodolyaka has quit IRC20:04
*** sridhar_ram has quit IRC20:05
*** TheJulia has quit IRC20:07
clarkbjeblair: is it ok to delete things like from rax ord?20:07
marcusvrn_asselin:  hi! It worked (about clean caches and env vars and re-build an image), now nodepool can create and access instances in OpenStack. Now I think the last problem that I'm facing is that is failing...I'm trying to debug it, but maybe you or someone else had faced or know what is going on. The script stop trying to setup20:08
marcusvrn_the workspace:     and console log:
openstackgerritKhai Do proposed openstack-infra/system-config: doc: add Storyboard integration info
jeblairclarkb: i'm going to say "yes" and "please don't look at the date"20:10
clarkbjeblair: :) ok20:10
clarkbjeblair: there are a few of them I assume realted to nodepool testing at one time20:11
openstackgerritPaul Belanger proposed openstack-infra/puppet-iptables: Add ip6tables service support for centos-7
jeblaireven though the time is right there in the name.  it's hard to miss.20:11
pabelangerclarkb: fungi: ^ should be the magic we need20:11
* clarkb looks before taking on rax image leaks20:11
jeblairi'll just give a hint -- the *current* time starts with 14..., not 13...20:12
*** tonytan_brb has joined #openstack-infra20:13
fungiclarkb: `date -d @1373496311` says "Wed Jul 10 22:45:11 UTC 2013"20:13
fungiso, um... really, really old20:13
fungialdo, date -d @ is super useful for turning epoch time into something human brainable20:14
pabelangermight been our longest lived server20:14
jeblairpabelanger: just an image20:14
fungia ghostly, ghostly image of a server that once was...20:14
clarkbpabelanger: commented just one thing about how puppet is silly that will likely make that fail as is :)20:15
*** roxanaghe has quit IRC20:15
clarkbjeblair: its also a gnu extension iirc. We had to use perl on the old solaris machines in order to manipulate unix time20:15
clarkbjeblair: there are ~6 images like that. I am guessing youdon't need any of them20:16
jeblairclarkb: probably that was addressed to fungi.  but i think gnu extensions are the best extensions.20:16
jeblairclarkb: correct20:16
*** tonytan4ever has quit IRC20:16
clarkboh hrm they have a queued state. I wonder if that has made them undeletable20:17
fungignu's not unix, and sometimes it's not even posix, but it's still awesome20:17
clarkbI will probably find out soon :)20:17
jeblairclarkb: oh good, there's an excuse for why i left them laying around20:17
fungiclarkb: openbsd's date function has a similar option, i just forget what it is off the top of my head20:17
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Also use the virtualenv for checking
clarkbfungi: in particular we wanted to compare deltas between file mode times and the time now and it just got easer to do the whole thing in perl without gnu date20:18
mordredjeblair, clarkb: ^^ sigh. one more patch related to the nodepool-in-venv thing we did the other day20:18
fungiclarkb: on openbsd it's `date -r 1373496311`20:18
clarkbfungi: the worst part of it was we had gnu date compiled for sparc but not for x86 and needed ti on both :/20:18
*** Goneri has quit IRC20:19
*** markusry has quit IRC20:20
fungihrm... i need to disappear again for a bit but will bbiaw20:21
*** hashar has joined #openstack-infra20:21
*** cody-somerville has joined #openstack-infra20:23
*** cody-somerville has joined #openstack-infra20:23
*** kgiusti has left #openstack-infra20:23
armaxhi folks any idea why I don’t see the test_results file on a failure like this one?20:23
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Extract /var/log and /etc from all nodes
*** roxanaghe has joined #openstack-infra20:23
*** gouthamr has joined #openstack-infra20:23
clarkbarmax: what is the test_results file?20:25
clarkbI am not familiar with that one for d-g runs20:25
*** e0ne has joined #openstack-infra20:26
armaxclarkb: sorry I meant this testr_results.html.gz20:26
*** raunak has joined #openstack-infra20:26
*** _ari_ has joined #openstack-infra20:27
*** eranrom has joined #openstack-infra20:29
mordredclarkb: SOOOOOOO20:29
*** cody-somerville has quit IRC20:29
mordredclarkb: we are somehow returning None from create_image sometimes20:31
*** abregman has quit IRC20:31
clarkbarmax: it tries to there at least20:32
clarkbmordred: byt cloud side iy actually made an image?20:33
mordredclarkb: well... that I don't know. but that method is not supposed to return None20:33
mordredso I'm tracking down how that's possible first20:34
pabelangerclarkb: looking20:34
*** Jeffrey4l_ has joined #openstack-infra20:34
mordredhowever, it could, just by itself, explain the leak20:34
*** dkehn_ has quit IRC20:34
mordredbcause it then throws an exception in nodepool on trying to access (no attribute 'id' on NoneType)20:34
mordredso from nodepool's pov it's a failed call to uploadImage20:35
clarkbmordred: ya20:35
*** Jeffrey4l__ has quit IRC20:35
*** jraim has joined #openstack-infra20:35
clarkbarmax: I can look more after lunch20:35
clarkbpabelanger: at which thing?20:35
mordredthis is also awesome:20:35
mordred2016-08-25 20:29:14,655 ERROR shade.exc: Image creation failed: 404 Not Found: Failed to find image af05cf54-af45-4889-a043-96675f014b20 to delete (HTTP 404)20:35
armaxclarkb: no rush20:35
armaxI am not quite sure what’s going on20:36
mordred"Image creation failed, couldn't find image to delete"20:36
clarkbmordred: its probably trying to cleanuo after itself due ti the two step image creat process20:36
openstackgerritArmando Migliaccio proposed openstack-infra/elastic-recheck: Add E-R query for bug 1617447
openstackbug 1617447 in OpenStack Compute (nova) "InterfaceAttachFailed: Failed to attach network adapter device" [High,New]
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add networking-cumulus project
openstackgerritPaul Belanger proposed openstack-infra/puppet-iptables: Add ip6tables service support for centos-7
pabelangerclarkb: comments on ^20:37
*** simonmcc has joined #openstack-infra20:39
*** fguillot has quit IRC20:39
*** _ari_ has quit IRC20:40
*** _ari_ has joined #openstack-infra20:40
*** serverascode has joined #openstack-infra20:42
*** ameade has joined #openstack-infra20:43
*** zhiyan has joined #openstack-infra20:45
*** skath has joined #openstack-infra20:45
*** roxanaghe has quit IRC20:46
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets
*** sridhar_ram has joined #openstack-infra20:48
*** rvasilets___ has joined #openstack-infra20:50
*** edtubill has quit IRC20:50
*** smarcet1981 has quit IRC20:50
*** ilyashakhat has joined #openstack-infra20:51
*** fdegir has joined #openstack-infra20:52
*** sc` has quit IRC20:52
*** sc` has joined #openstack-infra20:53
*** scottda has joined #openstack-infra20:54
*** jkilpatr has quit IRC20:54
mordredclarkb: I'm going to need to upload an image by hand ... you ok with me doing that now?20:54
pabelanger might be interesting to folks following a long with AFS. Updates devstack-gate to use our AFS mirror for network_sanity_check, and not just rackspace20:55
openstackgerritMerged openstack-infra/shade: Detect the need for FIPs better in auto_ip
openstackgerritMerged openstack-infra/shade: Log request ids
asselin__marcusvrn_, not'll have to trace it back. Here's where it's failing:
jeblairpabelanger: +220:57
*** dprince has quit IRC20:57
clarkbmordred: ya its fine, would be good if you can upload to not rax though20:58
clarkbI havent been able to clean rax yet got distracted by kids and lunch20:58
mordredclarkb: just gonna upload an image to ovh-bhs with name mttest20:58
mordredor testmt20:58
mordredor something20:58
clarkbmordred: thats fine20:58
*** maximov has joined #openstack-infra20:59
*** florianf has quit IRC21:00
*** ashtokolov has joined #openstack-infra21:00
*** betherly has joined #openstack-infra21:00
*** clenimar has quit IRC21:01
*** cody-somerville has joined #openstack-infra21:01
*** sdake has joined #openstack-infra21:02
openstackgerritMerged openstack-infra/shade: Rename _get_free_fixed_port to _nat_destination_port
*** salv-orlando has joined #openstack-infra21:03
*** tpsilva has joined #openstack-infra21:03
*** flaviof has joined #openstack-infra21:04
*** raildo has quit IRC21:06
*** sridhar_ram is now known as sridhar_ram_afk21:06
*** inc0 has quit IRC21:06
*** tdurakov has joined #openstack-infra21:06
*** rpodolyaka has joined #openstack-infra21:07
*** vdrok has joined #openstack-infra21:07
*** iurygregory has quit IRC21:07
*** thorst has quit IRC21:07
*** eharney has quit IRC21:08
*** NobodyCam has joined #openstack-infra21:08
*** AnarchyAo has joined #openstack-infra21:08
*** Goneri has joined #openstack-infra21:08
*** aimeeu has joined #openstack-infra21:08
mordredI would like to announce that uploading giant image as part of a debug cycle is FUN21:09
anteayathank you for announcing21:09
*** TheJulia has joined #openstack-infra21:10
*** thiagop has quit IRC21:10
*** slagle has quit IRC21:12
*** matrohon has quit IRC21:12
clarkbmordred: yes I concur and I think we should investigate ro cinder volumes once nova supports that (in maybe 3 years across all our clouds)21:14
*** tqtran has quit IRC21:14
*** ilyashakhat has quit IRC21:15
*** salv-orl_ has joined #openstack-infra21:18
*** jkilpatr has joined #openstack-infra21:18
pleia2off to the dentist (woo), bbiab21:18
*** e0ne has quit IRC21:18
mordredclarkb: we could just have our "image upload" process be "boot a server, attach a volume, dd image to volume, detach volume" - then our boot process be "clone volume, boot from volume"21:20
*** salv-orlando has quit IRC21:20
*** ldnunes has quit IRC21:20
jeblairmordred: i think you'd need to do: "attach volume to nodepool builder, dd image to volume, detach volume" for the upload process21:22
jeblairmordred: that requires running a builder in each cloud21:23
*** rtheis has quit IRC21:23
jeblairmordred: but it avoids the network.  if you do the thing you suggest, you're still transiting the network with the image21:23
*** cody-somerville has quit IRC21:23
irtermiteFolks... I understand that was some misunderstanding about the comparison of a split finger from an electrical burn to a burnt hot dog yesterday. As I understand it, that was taken in a sexual manner and I am not quite sure how that was construed that way. But, there was no way that was my intension. Regardless, I apologize for the misunderstanding. Have a great weekend!21:24
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
jeblairmordred: unless, you're suggesting that glance is slow enough that just sftping an image onto a server would be faster?21:24
irtermiteNever want to upset folks. You folks rock. Carry on!21:24
mordredjeblair: well, it might be more reliable21:25
*** tqtran has joined #openstack-infra21:26
*** roxanaghe has joined #openstack-infra21:26
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds an api-ref publishing job for cinder - Block Storage API
*** roxanaghe has quit IRC21:27
*** baoli_ has quit IRC21:28
*** roxanaghe has joined #openstack-infra21:29
*** piet has quit IRC21:29
*** abregman has joined #openstack-infra21:31
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds an api-ref publishing job for cinder - Block Storage API
*** gouthamr_ has joined #openstack-infra21:33
*** gouthamr has quit IRC21:36
*** _ari_ has quit IRC21:37
clarkbmordred: ya that would work but would require volume quota. I was thinking more of maimtain the job caches in a volume then use the new multiattach stuff going on in nova (which us not ready yet) to mount that fs on the volume many times read only21:46
clarkband that volume could be a terabyte large without much issue21:46
clarkb(in theory)21:46
mordredI grok what you are saying now21:47
mordredand yeah - if we're all still working on tech when that's in production at all of our clouds I'll be amazed21:47
*** kaisers_ has joined #openstack-infra21:48
*** rhallisey has quit IRC21:48
*** Sukhdev has quit IRC21:50
*** Hal1 has joined #openstack-infra21:52
*** kaisers_ has quit IRC21:52
*** Swami has quit IRC21:53
*** gouthamr_ is now known as gouthamr21:57
*** xyang1 has quit IRC21:57
*** lascii is now known as alaski22:00
*** sdake_ has joined #openstack-infra22:01
SpamapSmordred: 2024 , the year of volume multi-attach22:01
SpamapSmordred: regarding your "dd to volume" .. why volume? Just dd to your root disk, and snapshot.22:01
SpamapSalso known as "user mode glance"22:02
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Extract /var/log from all nodes
*** sdake has quit IRC22:03
*** rlandy has quit IRC22:04
*** abregman has quit IRC22:04
fungimordred: jeblair: i don't know that writing from a virtual machine to an attached cinder volume _actually_ avoids "the network" since (in at least most implementations) cinder is using a network storage protocol (fc san, iscsi, et cetera)22:05
fungifwiw, most of our cinder-related outages have actually been network outages22:06
*** thorst has joined #openstack-infra22:07
clarkbthat would actually be the biggest drawback of my idea but we already fail when the network doesn't work so shouldn't make things fail more22:08
*** thorst has quit IRC22:11
clarkbI have started deleting the leaked images in rax-iad22:11
clarkbthere are 25222:12
*** Hal1 has quit IRC22:12
*** dmsimard is now known as dmsimard|pto22:12
clarkband the first one hasn't finished yet... this may take a while22:14
fungifrom what i've seen, they service their api calls (especially deletions) on a best-effort basis22:15
fungiyou could just spam them, then come back in a bit and see how many are still showing up22:15
clarkbI am just running for entry in file do openstack image delete entry22:16
clarkbFailed to delete image with name or ID 'b727fa80-1ea6-4a89-aa56-c3bee86b2340': HTTPServiceUnavailable (HTTP 503)22:16
clarkbwell then22:16
fungiall that needs is a & (and a sleep $n if you're feeling generous)22:16
*** adrian_otto has joined #openstack-infra22:17
clarkbya that loosk consistent22:18
clarkbso maybe I don't get to delete iamges in rax-iad today22:18
clarkbmordred: ^ remember your tattoo idea?22:18
*** fguillot has joined #openstack-infra22:18
*** hashar has quit IRC22:22
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] scenario jobs refactoring
clarkbarmax: the reason it didn't create the testr html file is that there is no testrepository.subunit file which is raelly the subunit from devstack22:23
armaxclarkb: ok22:23
armaxI wonder why the subunit file is missing then22:24
*** cody-somerville has joined #openstack-infra22:24
openstackgerritKunal Mehta proposed openstack-infra/zuul: Don't merge queues if the common job is 'noop'
clarkbarmax: I am not sure how that works in devstack btu will take a look22:25
armaxclarkb: this feels like a regression of some sort22:26
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Correct path to Launchpad creds on signing node
fungithere's a simple fix for anyone looking to pad their review stats! ;)22:26
*** dimtruck is now known as zz_dimtruck22:27
*** matt-borland has quit IRC22:28
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Correct path to Launchpad creds on signing node
funginow with minor improvement22:29
*** berendt has quit IRC22:30
*** tqtran has quit IRC22:30
*** cody-somerville has quit IRC22:31
clarkbarmax: looks like it should be handled by the exit trap btu I don't see that running22:31
clarkbmaybe something is clearing it out?22:31
mordredSpamapS: snapshots are hard - there is a reason we stopped doing them :)22:32
mordredclarkb: btw - still uploading test image22:32
armaxclarkb: I can’t think of anything, I wonder if this is something affecting the linuxbridge job only, but if I am not mistaken I have seen this elsewhere22:32
clarkbarmax: ya I think this is likely a devstack bug22:33
clarkbfungi: I stopped my deletes and switched to shows and the first 4 iamges (which had attempted deletes) were no longer showable22:34
clarkbfungi: making me think its actually deleting them but also returning 500 errors22:34
*** tqtran has joined #openstack-infra22:34
fungiclarkb: ECLOUD22:34
clarkbso I uh restarted it and have my fingers crossed :)22:34
*** tphummel has quit IRC22:35
SpamapSmordred: I just thought that you stopped doing them because you were _CHICKEN_. That's right. YELLOW.22:35
* SpamapS bocks22:35
* mordred hands SpamapS a bucket of corn22:35
jeblairmmm.  chicken.22:35
*** berendt has joined #openstack-infra22:36
clarkbnow I know what I am having for dinner22:36
fungia simple hyperchicken lawyer from a backwoods asteroid?22:36
clarkbarmax: if you like I can file the bug against devstack22:36
clarkbarmax: not sure if you are investigating more22:37
armaxclarkb: I am not today, but I can follow up on Monday22:37
armaxclarkb: I thought you perhaps knew what was happening22:37
clarkbfungi: confirmed. 71fa4b18-8a2c-4a83-aaa8-762dd85d5f5b got a 503 on deletion but now I can't show it. I was able to show it prior22:38
fungison, as your lawyer, i declare y'all are in a 12-piece bucket o' trouble22:40
*** itisha has quit IRC22:40
*** tphummel has joined #openstack-infra22:42
openstackLaunchpad bug 1617476 in devstack "devstack.subunit is not generated" [Undecided,New]22:42
armaxclarkb: ack22:42
*** edmondsw has quit IRC22:44
*** Swami has joined #openstack-infra22:45
openstackgerritRyan Brandt proposed openstack-infra/project-config: Enable java test as voting on monasca-api
*** pt_15 has quit IRC22:46
clarkbwe still haev 45 image uploads running today22:47
clarkbthe images all built by 11am though which is good22:47
clarkblooks like all the images that have completed for internap nyj so far have failed. trying to see where else things are failing22:49
*** rbrndt has quit IRC22:49
*** kien-ha has joined #openstack-infra22:52
*** signed8bit is now known as signed8bit_Zzz22:54
clarkbmgagne: appears to be the underlying cause for the internap image upload failures22:54
*** signed8bit_Zzz is now known as signed8bit22:56
*** simonmcc has quit IRC22:56
jlvillaltonyb: On openstack/requirements and the "normalize" that is done. Shouldn't the items in global-requirements be run through that? So that when someone copies from global-requirements.txt the proposal bot doesn't end up proposing a normalized version a few days later.22:57
openstackgerritK Jonathan Harker proposed openstack-infra/log_processor: Add a simple test for the worker
jlvillaltonyb: At least that seems to be what happened with me and changing the 'hacking' requirement in some of the ironic projects22:58
jesusaurclarkb: I finally got around to writing a test case for the log_processor worker ^^22:58
*** skath has quit IRC22:58
clarkbjesusaur: neat22:58
clarkbjesusaur: is that self testing yet? I am pretty sure I reviewed the changes to add the test but not sure if it got merged22:58
openstackgerritMerged openstack-infra/system-config: Correct path to Launchpad creds on signing node
jesusaurclarkb: you're probably thinking of the test case for the client that was added a while back, and yeah it's self-testing22:59
jesusaurclarkb: but we still haven't actually switched to that project yet22:59
*** vdrok has quit IRC23:00
*** salv-orl_ has quit IRC23:00
clarkboh right this is the worker, we could probably turn that test into an integration test if we spun up a client, gearman, worker, and logstash23:00
*** betherly has quit IRC23:00
*** TheJulia has quit IRC23:01
*** serverascode has quit IRC23:01
*** Julien-zte has joined #openstack-infra23:01
*** aimeeu has quit IRC23:01
*** NobodyCam has quit IRC23:01
*** tdurakov has quit IRC23:01
*** flaviof has quit IRC23:02
*** zhiyan has quit IRC23:02
*** ashtokolov has quit IRC23:02
*** scottda has quit IRC23:02
fungijlvillal: probably, not only should changes to the global reqs list be run through that, they should be run through that on every proposed change and failed if the pre-normalized for doesn't match23:02
*** rpodolyaka has quit IRC23:02
*** ameade has quit IRC23:03
*** tpsilva has quit IRC23:03
*** sridhar_ram_afk has quit IRC23:03
*** rvasilets___ has quit IRC23:04
*** maximov has quit IRC23:04
*** fdegir has quit IRC23:04
*** cody-somerville has joined #openstack-infra23:05
*** cody-somerville has joined #openstack-infra23:05
*** esikachev has quit IRC23:05
*** bapalm_ has quit IRC23:06
jlvillalfungi: I proposed this: Normalize the requirement files23:06
jlvillalfungi: But not the 2nd part you suggested23:06
*** hongbin has quit IRC23:06
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Groovy Label Assignment plugin
*** jraim has quit IRC23:09
*** bapalm has joined #openstack-infra23:11
*** rpodolyaka has joined #openstack-infra23:15
*** TheJulia has joined #openstack-infra23:15
*** NobodyCam has joined #openstack-infra23:15
*** ashtokolov has joined #openstack-infra23:15
*** betherly has joined #openstack-infra23:15
*** vhosakot has joined #openstack-infra23:16
*** DericHorn-HP has quit IRC23:17
*** vdrok has joined #openstack-infra23:19
*** vhosakot has quit IRC23:19
*** serverascode has joined #openstack-infra23:20
*** sridhar_ram_afk has joined #openstack-infra23:22
openstackgerritMerged openstack-infra/system-config: Zuul has a channel, let's tell people about it
*** tdurakov has joined #openstack-infra23:23
*** tphummel has quit IRC23:24
*** DericHorn-HP has joined #openstack-infra23:25
openstackgerritMerged openstack-infra/system-config: Configure jessie-security for our debian mirror
*** cody-somerville has quit IRC23:26
openstackgerritK Jonathan Harker proposed openstack-infra/log_processor: Add a simple test for the worker
*** zhiyan has joined #openstack-infra23:28
*** rvasilets___ has joined #openstack-infra23:28
*** fguillot is now known as fguillot_afk23:29
*** maximov has joined #openstack-infra23:30
*** fguillot_afk has quit IRC23:30
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update properties.slack to use convert xml
tonybjlvillal: That's a little questionable.  A meta question would be where would you expect the documentation about how to do that thing to be?23:31
*** jerryz has quit IRC23:32
*** aimeeu has joined #openstack-infra23:32
*** fdegir has joined #openstack-infra23:33
*** raunak has quit IRC23:35
*** kaisers_ has joined #openstack-infra23:37
*** amit213 has quit IRC23:37
*** xarses has quit IRC23:37
*** amit213 has joined #openstack-infra23:40
*** kaisers_ has quit IRC23:41
*** simonmcc has joined #openstack-infra23:41
*** skath has joined #openstack-infra23:41
pabelangerclarkb: if you want to review I can kick off a build later tonight23:41
pabelangerthat should be our ip6tables fix for centos-723:42
*** fguillot has joined #openstack-infra23:43
*** yamahata has quit IRC23:43
*** esikachev has joined #openstack-infra23:44
*** NobodyCam has quit IRC23:46
*** zhiyan has quit IRC23:46
*** rpodolyaka has quit IRC23:46
*** TheJulia has quit IRC23:46
*** maximov has quit IRC23:46
*** betherly has quit IRC23:46
*** rvasilets___ has quit IRC23:46
*** sridhar_ram_afk has quit IRC23:46
*** fdegir has quit IRC23:46
*** aimeeu has quit IRC23:46
*** simonmcc has quit IRC23:46
*** vdrok has quit IRC23:46
*** serverascode has quit IRC23:46
*** ashtokolov has quit IRC23:46
*** tdurakov has quit IRC23:46
*** skath has quit IRC23:46
*** mdrabe has quit IRC23:47
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add display-name option to github plugin
*** AnarchyAo has quit IRC23:50
clarkbpabelanger: have hyou tested it anywhere?23:52
clarkbpabelanger: I always get a little scared of iptables upadtes particularly at the end of day on friday23:52
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update rebuild plugin to use convert xml
*** xarses has joined #openstack-infra23:53
clarkbpabelanger: I went ahead and +2'd it and if you are comforatable approving feel free to23:53
*** xarses has quit IRC23:53
pabelangerclarkb: I tested start ip6tables service, not puppet23:53
pabelangerclarkb: I am also okay for waiting until next week23:54
*** xarses has joined #openstack-infra23:54
pabelangerlets do that23:54
clarkbalso we have 44 images uploading nwo so its not like it will get uploaded soon23:54
*** xarses has quit IRC23:54
*** yamahata has joined #openstack-infra23:54
*** xarses has joined #openstack-infra23:54
*** xarses has quit IRC23:55
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update build-discarder to use convert xml
*** xarses has joined #openstack-infra23:57
clarkbI am also pausing the image deletes in rax since they are taking forever and I don't awnt them to continue while I am off weekending (because I won't be able to watch them)23:57
*** DericHorn-HP has quit IRC23:58
jlvillaltonyb, I'm confused what you mean by documentation?23:58
*** baoli has joined #openstack-infra23:58
tonybjlvillal: ?23:58
clarkbjlvillal: when the test fails because your line isn't normalized where do you read about how to normalize it23:58
jlvillalI didnt' propose doing that23:59
jlvillalI just that the global-requirements.txt file should be normalized23:59
jlvillals/just that/just proposed that/23:59
clarkbjlvillal: sure but how do you keep it normalized if you don't have a test running that fails on non normalized lines?23:59
tonybjlvillal: my point is you only get in the situation you're in because you didn't use the tools available to update the files.23:59

Generated by 2.14.0 by Marius Gedminas - find it at!