Thursday, 2016-08-25

*** mtanino has quit IRC00:00
*** sarob has joined #openstack-infra00:01
*** Julien-zte has quit IRC00:02
*** zhurong has quit IRC00:02
*** ddieterly has joined #openstack-infra00:08
*** david-lyle has quit IRC00:08
*** sarob has quit IRC00:11
openstackgerritMerged openstack-infra/shade: Ensure per-resource caches work without global cache
openstackgerritMerged openstack-infra/shade: Move list_server cache to dogpile
*** sarob has joined #openstack-infra00:11
openstackgerritMerged openstack-infra/shade: Allow image and flavor by name for create_server
*** dcwangmit01_ has quit IRC00:12
*** dcwangmit01 has joined #openstack-infra00:12
*** chlong has quit IRC00:13
*** yolanda has joined #openstack-infra00:13
*** gouthamr_ has joined #openstack-infra00:13
*** sarob has quit IRC00:16
*** gouthamr has quit IRC00:16
*** roxanagh_ has joined #openstack-infra00:19
*** sdague has joined #openstack-infra00:22
sdaguehey, so the generated at time for still seems off, I think that's about 4 hours ago00:24
sdagueI wonder if the cron job is still failing for other reasons00:26
*** chlong has joined #openstack-infra00:26
*** sputnik13 has quit IRC00:30
pabelangersdague: I think pleia2 and mtreinish might have been working on it today. Not sure the outcome00:30
sdaguepabelanger: they were, and we got at least one manual run00:30
pleia2I'll have a look again00:31
sdaguepleia2: awesome, thank you00:31
sdagueI wasn't sure who was still around00:31
clarkbI have seen older generators get stuck holding the log file00:32
sdague is still on the 18th00:34
sdaguewhich means at least something is not updating that bit correctly as well00:34
pleia2Generated at: 2016-08-24T20:5900:34
pleia2where do you see 18th?00:34
sdaguepleia2: the other pge00:35
sdagueOther Projects Uncategorized00:35
pleia2yeah, that's what I'm looking at00:35
sdaguewhich, honestly, I don't really care about that much, it just seems to indicate some issues in running behind00:35
sdagueoh... gah, this caches really weirdly00:36
sdagueI had to force flush my cache00:36
sdagueok, 20:59, yep00:36
pleia2still, that's old00:36
pleia2running it manually again to see if there are any errors..00:36
sdaguethere is also definitely a template change we landed, which is not showing up00:36
sdaguethat, ironicly, would highlight the date when it was more than an hour behind :)00:37
openstackgerritMerged openstack-infra/devstack-gate: Remove q-lbaas from tempest pre-installed stuff.
pleia2oh yeah, big traceback00:38
sdagueok, what now?00:38
pleia2the last line goes on and on for pages, just included a sample00:39
sdagueoh... that's interesting00:41
sdaguethe query killed it00:41
*** Julien-zte has joined #openstack-infra00:42
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: drop parens on query, they kill ES
sdaguepleia2: ok, so I think that will fix the stack trace00:43
pleia2I'll give it a try00:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_monasca repository to OpenStack-Ansible
*** raunak has quit IRC00:44
sdagueorthoginal to that, I'm not sure why puppet isn't applying the template changes -
sdaguewhich merged a number of hours ago00:44
pleia2sdague: that patch didn't seem to go far enough, still dying00:48
*** markvoelker has joined #openstack-infra00:55
*** Julien-zte has quit IRC00:59
*** gyee has quit IRC00:59
sdaguepleia2: I wonder if I need to quote the cinder part, I'm poking locally to see if I can get a clean run00:59
sdagueunrelated, can you see if the git tree for CDing updated after hit?01:00
*** yanyanhu has joined #openstack-infra01:00
openstackgerritMerged openstack-infra/elastic-recheck: drop parens on query, they kill ES
pleia2yep, let's see01:01
pleia2sdague: it's definitely on disk, but the timestamp is 21:01 and since the integrated_gate.html template hasn't been applied since then it hasn't gone live01:02
pleia2darn 2 minutes01:03
sdagueah... ok01:03
sdaguewell, that is at least reasuring01:03
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: projects must be quoted
sdaguethat has not *yet* failed for me locally01:04
*** pvaneck has quit IRC01:04
sdaguebut... it doesn't really give much of a progress report01:04
pleia2heh, yeah01:04
sdagueI'll have to yell at mtreinish about that :)01:04
*** Julien-zte has joined #openstack-infra01:04
*** shashank_hegde has quit IRC01:05
* pleia2 patpat ES servers01:05
*** sdake has quit IRC01:06
sdagueI honestly think that patch might merge before I get a local return here01:07
pleia2it is quite a query01:08
*** thorst has quit IRC01:09
*** thorst has joined #openstack-infra01:10
anteayathere is currently nothing in the check queue older than 3 hours01:10
*** Sukhdev has quit IRC01:10
anteayaso yay working ipv601:10
*** jamielennox is now known as jamielennox|away01:11
*** jamielennox|away is now known as jamielennox01:11
asselin__anyone know why 'openstack server list' would be very very slow recently?
sdagueyeh, the gate is burning down nicely now01:13
*** raunak has joined #openstack-infra01:13
*** caowei has joined #openstack-infra01:14
asselin__I guess it's something to do between python-openstackclient==2.6.0 and python-openstackclient==3.0.101:16
*** zhurong has joined #openstack-infra01:16
*** yolanda has quit IRC01:18
*** ddieterly has quit IRC01:18
*** thorst has quit IRC01:19
asselin__so much for backwards compatibility: Unable to establish connection to
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
anteayaasselin__: mordred might know more about what is happening with openstackclient01:21
asselin__anteaya, thanks...I find it odd that there's a deprecation notice coupled with it failing to work altogether on my cloud. :(01:23
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: WIP: enable more reasonable logging
sdaguefirst pass on some logging tweaks that might make things better later, I'll sync with mtreinish in the morning01:24
sdaguehave a good night folks01:24
pleia2night sdague01:24
sdaguepleia2: thanks again for looking into the fails01:24
pleia2sure thing01:24
anteayaasselin__: :(01:24
anteayasdague: night01:25
asselin__mordred, if you care, here's the error output with pip freeze on a box that doesn't work: and another box setup a week or 2 ago with it working:
*** Apoorva has quit IRC01:29
*** yolanda has joined #openstack-infra01:31
*** tphummel has quit IRC01:32
*** baoli has joined #openstack-infra01:32
*** fguillot has quit IRC01:33
*** roxanagh_ has quit IRC01:34
*** raunak has quit IRC01:35
*** markvoelker has quit IRC01:36
*** sdake has joined #openstack-infra01:36
*** yolanda has quit IRC01:37
*** sdague has quit IRC01:38
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Split multinode.yaml template
*** sdake_ has joined #openstack-infra01:39
*** sdake has quit IRC01:42
*** kgiusti is now known as kgiusti_afk01:44
*** fguillot has joined #openstack-infra01:45
amrithgood evening folks, is anyone else reporting issues logging in to gerrit; launchpad's auth system has been timing out for the past hour or so01:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Split multinode.yaml template
*** yolanda has joined #openstack-infra01:47
* amrith cracks open a bottle of wine and waits for ... anyone want to come join me?01:48
*** kgiusti_afk has left #openstack-infra01:48
*** ddieterly has joined #openstack-infra01:49
*** shashank_hegde has joined #openstack-infra01:49
*** sdake_ has quit IRC01:54
*** mtanino has joined #openstack-infra01:58
fungiit's beer over here, but yes01:58
*** thorst has joined #openstack-infra01:59
fungiat casa del fungi we've reached the beer 'n' video games portion of this evening's program, already in progress01:59
* fungi gets back to it01:59
*** thorst has quit IRC01:59
*** thorst has joined #openstack-infra02:00
clarkbI have a wrird ginger beer02:00
clarkbits not a proper ginger beer. It is real beer with ginger in it02:00
*** ddieterly has quit IRC02:02
*** yamahata has quit IRC02:03
*** sdake has joined #openstack-infra02:03
openstackgerritXiang Chen proposed openstack/diskimage-builder: Explain difference between two envvar
*** ddieterly has joined #openstack-infra02:04
*** gouthamr_ is now known as gouthamr02:05
fungiclarkb: still sounds marvellous02:05
openstackgerritXiang Chen proposed openstack/diskimage-builder: Explain difference between two envvars
*** thorst has quit IRC02:08
*** zhurong_ has joined #openstack-infra02:09
*** zhurong has quit IRC02:11
*** vinaypotluri has quit IRC02:11
amrithHey clarkb, looks like it works again02:14
amrithhi fungi02:14
amrithmanaged to push a review
amrithso all is good for now02:15
amrithtime to call it a night02:15
*** shashank_hegde has quit IRC02:15
*** yolanda has quit IRC02:17
clarkbamrith: fwiw puahing code shouldnt require openid02:18
clarkbits either your ssh key or your http password key thing that is sued depending on protocol02:19
*** raunak has joined #openstack-infra02:20
amrithclarkb, sorry. i needed to look at reviews (open reviews) before I could push a patch set.02:20
*** yolanda has joined #openstack-infra02:21
*** jcook has quit IRC02:22
*** jcook has joined #openstack-infra02:22
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Tidy up absent rule
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Tidy up zuul-env-reqs
*** thorst has joined #openstack-infra02:25
*** rackertom has quit IRC02:25
amrithwanted to make sure that a couple of patches had landed before I marked this interim release for the client.02:25
amrithOK, now, really ... g'night02:26
*** simonmcc has quit IRC02:26
jheskethcraige: I was just about to update the infra agenda to discuss vagrant files next meeting.. but perhaps we should just kick another thread off on openstack-infra?02:26
craigeSounds plan jhesketh02:27
*** rackertom has joined #openstack-infra02:27
*** thiagolib has quit IRC02:27
jheskethcraige: want me to kick that off?02:27
craigeWhy not goo sir.02:27
*** simonmcc has joined #openstack-infra02:28
craigei'm about to knock off for lunch (which really just means working on the farm instead of the computer)02:28
*** thiagolib has joined #openstack-infra02:28
*** thorst has quit IRC02:28
jheskethcraige: no worries, have fun :-002:28
*** serverascode has quit IRC02:28
craigeThanks jhesketh02:29
*** thcipriani is now known as thcipriani|afk02:29
*** mtanino has quit IRC02:31
*** serverascode has joined #openstack-infra02:31
*** ddieterly has quit IRC02:31
*** sdake has quit IRC02:34
*** sdake has joined #openstack-infra02:35
*** julim has quit IRC02:35
openstackgerritIan Wienand proposed openstack-infra/puppet-gerrit: Create index on fresh install
*** ddieterly has joined #openstack-infra02:37
ianwzaro: ^ as discussed02:37
*** ddieterly has quit IRC02:37
*** bswartz has quit IRC02:38
*** Wei_Liu has quit IRC02:43
*** fguillot has quit IRC02:43
*** Wei_Liu has joined #openstack-infra02:43
*** jamesdenton has joined #openstack-infra02:44
*** sdake has quit IRC02:45
*** sdake has joined #openstack-infra02:47
*** dimtruck is now known as zz_dimtruck02:48
*** yuanying has quit IRC02:49
mordredasselin__: I do care, and thanks! today kind of fell apart, I'll look tomorrow02:50
*** raunak has quit IRC02:51
*** yuanying has joined #openstack-infra02:53
*** changzhi has joined #openstack-infra02:55
*** gildub has quit IRC02:59
*** sarob has joined #openstack-infra02:59
*** gildub has joined #openstack-infra03:01
*** sarob has quit IRC03:02
*** shashank_hegde has joined #openstack-infra03:02
*** sarob has joined #openstack-infra03:02
*** sflanigan has quit IRC03:04
*** raunak has joined #openstack-infra03:09
*** jamielennox is now known as jamielennox|away03:09
*** woodster_ has quit IRC03:09
*** sflanigan has joined #openstack-infra03:10
*** sflanigan has joined #openstack-infra03:10
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add osic-cloud8 to grafana
*** sdake has quit IRC03:12
*** tonytan4ever has quit IRC03:13
*** hughhalf has quit IRC03:16
*** hughhalf has joined #openstack-infra03:16
*** salv-orlando has joined #openstack-infra03:17
asselin__mordred, sorry...after more investigation, I can't even ping that IP....which is strange since its the same cloud & tenant hosting the VM and the previous box. I'll look at that tomorrow. Just want to let you know there's likely some env issue on my env. I will try launching a new node and see if I can reproduce.03:20
*** salv-orl_ has quit IRC03:20
*** chlong has quit IRC03:23
*** jamielennox|away is now known as jamielennox03:23
*** nwkarsten has joined #openstack-infra03:24
*** senk has joined #openstack-infra03:25
*** xcGlQHdZGT has quit IRC03:28
*** thorst has joined #openstack-infra03:29
*** shashank_hegde has quit IRC03:29
*** dkehn_ has quit IRC03:31
*** dkehn has quit IRC03:31
*** Julien-zte has quit IRC03:32
*** chlong has joined #openstack-infra03:35
*** thorst has quit IRC03:37
*** vikrant has joined #openstack-infra03:38
*** dkehn has joined #openstack-infra03:38
*** bhunter71 has quit IRC03:38
*** chlong has quit IRC03:38
*** chlong has joined #openstack-infra03:38
*** rlandy|bbl is now known as rlandy03:39
*** rlandy has quit IRC03:40
*** Julien-zte has joined #openstack-infra03:41
*** changzhi has quit IRC03:41
*** vinaypotluri has joined #openstack-infra03:42
*** raunak has quit IRC03:42
*** dkehn_ has joined #openstack-infra03:43
*** gongysh has joined #openstack-infra03:43
*** senk has quit IRC03:43
*** baoli has quit IRC03:44
*** yuanying has quit IRC03:45
*** raunak has joined #openstack-infra03:46
*** sarob has quit IRC03:47
mordredasselin__: cool. well, I certainly don't want new code to break you03:47
*** armax has quit IRC03:47
*** yuanying has joined #openstack-infra03:48
*** coolsvap_ has joined #openstack-infra03:50
*** bhunter71 has joined #openstack-infra03:51
*** raunak has quit IRC03:57
*** raunak has joined #openstack-infra03:58
*** hqkLChMUKc has joined #openstack-infra04:02
*** jogo has quit IRC04:06
*** shashank_hegde has joined #openstack-infra04:06
*** amotoki has joined #openstack-infra04:06
*** mtanino has joined #openstack-infra04:08
*** raunak has quit IRC04:12
*** bswartz has joined #openstack-infra04:12
*** tonytan4ever has joined #openstack-infra04:13
stevemarclarkb: not sure what i should do about this bug:
openstackLaunchpad bug 1602407 in OpenStack Identity (keystone) "MySQL tests significantly slower than PostgreSQL tests" [Undecided,New]04:18
*** tonytan4ever has quit IRC04:18
stevemarclarkb: apparently mysql tests are always slower than postgres tests :P04:18
*** senk has joined #openstack-infra04:18
asselin__mordred, ok, I setup a new env using the same setup script. Here's the error AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult' Sanitized debug output for openstack, ansible & pip output
*** gongysh has quit IRC04:25
asselin__ping works this time. for the ansible openstack script I'm using commit acc8a13eff73d27977525e04454fe7e904484e6f04:26
asselin__. Basically following, but using ansible==
*** raunak has joined #openstack-infra04:29
*** Ravikiran_K has joined #openstack-infra04:29
*** thorst has joined #openstack-infra04:35
*** links has joined #openstack-infra04:37
*** nwkarsten has quit IRC04:38
*** gouthamr has quit IRC04:39
*** caowei has quit IRC04:39
*** thorst has quit IRC04:42
openstackgerritPaul Belanger proposed openstack-infra/zuul: Revert "Store ssh_host_key of remote node"
pabelangerianw: jhesketh: do you mind a quick +A^04:43
pabelangerreverts zuul patch does is broken04:43
jheskethpabelanger: looking04:43
pabelangerjust tested it on zlstatic01.o.o04:43
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: congress: remove new_arch jobs
pabelanger is the failure04:44
pabelangerlooks like and issue with the args passed to ssh-keygen04:45
jheskethpabelanger: +w04:45
*** psachin has joined #openstack-infra04:48
AJaegergood morning!04:48
AJaegerpabelanger: you're still here? What miracles are you doing? ;)04:49
*** psachin has quit IRC04:49
AJaegerjhesketh: good you review , please?04:49
jheskethAJaeger: sure, looking04:49
*** yamahata has joined #openstack-infra04:50
pabelangerAJaeger: Ya, about to call it a night. Have sick kids tonight, offered to say up a little later in case they wake up with issues.04:50
pabelangerokay, figure out ssh-keygen issue. Will upload a patch in the morning.04:52
*** mtanino has quit IRC04:53
*** senk has quit IRC04:54
AJaegerpabelanger: hope your kids get well soon!04:58
AJaegerthanks, jhesketh04:58
AJaegerpabelanger: thanks for looking at hte ssh-keygen04:59
*** raunak has quit IRC04:59
*** raunak has joined #openstack-infra05:00
openstackgerritMerged openstack-infra/zuul: Revert "Store ssh_host_key of remote node"
*** scottda has quit IRC05:06
*** sarob has joined #openstack-infra05:09
*** jaosorior has joined #openstack-infra05:10
AJaegerproject-config cores, could you review , please? This removes an unneeded post job05:13
*** sarob has quit IRC05:14
*** psachin has joined #openstack-infra05:14
AJaegertherve: is ready now?05:14
*** raunak has quit IRC05:17
*** Sukhdev has joined #openstack-infra05:20
*** baoli has joined #openstack-infra05:21
AJaeger#success Devstack runs fine on IPv6 only nodes05:22
openstackstatusAJaeger: Added success to Success page05:22
*** senk has joined #openstack-infra05:23
openstackgerritTomasz Trębski proposed openstack-infra/project-config: Adding check-requirements for log-api
*** cody-somerville has joined #openstack-infra05:25
*** harlowja_at_home has joined #openstack-infra05:25
*** ilyashakhat has joined #openstack-infra05:26
*** baoli has quit IRC05:27
ianwi don't know why, but fedora-24 hasn't updated in rax-iad for quite a few days05:27
ianwuploading image 921 now ... see what happens ...05:30
openstackgerritAleksandr Dobdin proposed openstack-infra/project-config: add: ansible-like tool
*** _nadya_ has joined #openstack-infra05:31
*** kaisers_ has joined #openstack-infra05:32
*** ilyashakhat has quit IRC05:33
*** _nadya_ has quit IRC05:36
*** cody-somerville has quit IRC05:38
*** thorst has joined #openstack-infra05:40
*** david-lyle has joined #openstack-infra05:43
*** harlowja_at_home has quit IRC05:45
*** thorst has quit IRC05:48
*** ilyashakhat has joined #openstack-infra05:48
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Make Undercloud OpenStack service workers configurable
*** simonmcc has quit IRC05:51
*** claudiub has joined #openstack-infra05:51
*** jogo has joined #openstack-infra05:52
*** jogo has quit IRC05:52
*** jogo has joined #openstack-infra05:52
*** simonmcc has joined #openstack-infra05:53
openstackgerritIan Wienand proposed openstack-infra/shade: Delete objname in image_delete
*** aeng has quit IRC05:55
ramishraAJaeger: hi, therve is on leave this week, yeah, the experimental job is now green
ianwmordred: ^^ this may be wrong.  i'm more interested in why it doesn't fail CI.  if you have any thoughts to drop on that in the review, i can look at it05:57
*** e0ne has joined #openstack-infra05:57
*** sandanar has joined #openstack-infra05:58
*** Sukhdev has quit IRC05:59
*** edtubill has quit IRC06:03
*** adriant has quit IRC06:03
openstackgerritIan Wienand proposed openstack-infra/project-config: Remove Fedora 24 work-around kernel
*** ccamacho has joined #openstack-infra06:04
*** rajinir has quit IRC06:05
*** david-lyle has quit IRC06:09
*** caowei has joined #openstack-infra06:10
*** Genek has joined #openstack-infra06:14
GenekHi, I'm now working as a intern in Foundation. I'm currently working on improving . Now we have 4800+ tags on it where most of than have less than 1 question associate to it.06:16
GenekWe're currently working on cleaning those tags. However it'll take a lot of time deleting them using the admin panel.06:17
GenekIs it possible to delete those tags directly on the server? Thanks a lot :)06:17
AJaegerramishra: thanks for confirming and answering! Would appreciate one or more heat cores to +1 the change.06:19
ramishraAJaeger: I'm one of them and have +1ed it:)06:22
*** baoli has joined #openstack-infra06:23
AJaegerWelcome, Genek! I doubt anybody is around right now that can help you, hope others read it later and answer - but you might need to reach out to some via email.06:23
AJaegerramishra: understood, just saying I'd like some broader support for the change - so if you could ask one ore more to speak up, I'll +2 if it's fine.06:24
AJaegerramishra: I remember some discussion about that change, so like to be careful06:25
ramishraAJaeger: np. Have asked a few to have a look.06:25
clarkbGenek I think there is a postgresql database behind Itay be possibly to directly modify that but it depends on the application and db schema06:26
clarkbGenek: I dont know that anyone here knows if it is possible without reading docs and testing it first06:26
*** baoli has quit IRC06:28
*** rcernin has joined #openstack-infra06:28
*** VnrycepuxO has quit IRC06:31
*** r-mibu has quit IRC06:32
*** claudiub has quit IRC06:33
Genekclarkb I think there's a command called "delete_tags", haven't tried it yet. I think I'll set up a test environment for testing.06:34
Genekclarkb: I think there's a command called "delete_tags", haven't tried it yet. I think I'll set up a test environment for testing.06:34
*** nikhil has quit IRC06:36
*** nikhil has joined #openstack-infra06:38
*** kushal has quit IRC06:39
*** hqkLChMUKc has quit IRC06:39
*** andreas_s has joined #openstack-infra06:40
*** watersoul has quit IRC06:42
*** watersoul has joined #openstack-infra06:42
*** thorst has joined #openstack-infra06:44
*** watersoul has joined #openstack-infra06:47
*** ihrachys has joined #openstack-infra06:50
*** salv-orlando has quit IRC06:51
*** thorst has quit IRC06:52
*** kzaitsev_mb has joined #openstack-infra06:52
*** Na3iL has joined #openstack-infra06:53
yolandagood morning06:56
*** ihrachys has quit IRC06:56
*** ilyashakhat has quit IRC06:59
*** senk has quit IRC06:59
*** yaume has joined #openstack-infra07:00
AJaegergood morning, yolanda !07:00
Genekgood morning!07:01
*** florianf has joined #openstack-infra07:04
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud
*** senk has joined #openstack-infra07:08
*** tesseract- has joined #openstack-infra07:08
*** ifarkas_afk is now known as ifarkas07:09
*** DrifterZA has joined #openstack-infra07:11
*** mikelk has joined #openstack-infra07:14
*** sarob has joined #openstack-infra07:14
*** Ravikiran_K has quit IRC07:14
*** tonytan4ever has joined #openstack-infra07:15
*** kzaitsev_mb has quit IRC07:16
*** sarob has quit IRC07:19
*** tonytan4ever has quit IRC07:20
*** dtardivel has joined #openstack-infra07:20
*** vinaypotluri has quit IRC07:21
*** Genek has quit IRC07:23
*** baoli has joined #openstack-infra07:23
*** Genek has joined #openstack-infra07:24
*** Genek has joined #openstack-infra07:25
*** martinkopec has joined #openstack-infra07:25
*** r-mibu has joined #openstack-infra07:27
*** baoli has quit IRC07:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix translation setup for networking-odl
AJaegerproject-config cores, the change above should go in soonish. Please review ^.07:30
*** e0ne has quit IRC07:32
*** e0ne has joined #openstack-infra07:33
*** matthewbodkin has joined #openstack-infra07:36
*** yarkot1 has quit IRC07:40
*** jpich has joined #openstack-infra07:41
*** yaume has quit IRC07:43
*** yaume has joined #openstack-infra07:44
*** abregman has joined #openstack-infra07:44
*** shardy has joined #openstack-infra07:47
*** thorst has joined #openstack-infra07:50
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Add docker-suite-app-murano project to openstack
*** hashar has joined #openstack-infra07:52
*** sflanigan has quit IRC07:57
*** sflanigan has joined #openstack-infra07:58
*** sflanigan has joined #openstack-infra07:58
*** thorst has quit IRC07:58
*** shashank_hegde has quit IRC07:58
*** salv-orl_ has joined #openstack-infra07:59
*** salv-orl_ has quit IRC07:59
*** zzzeek has quit IRC08:00
*** salv-orl_ has joined #openstack-infra08:00
*** zzzeek has joined #openstack-infra08:00
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud
*** berendt has joined #openstack-infra08:00
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Revert "networking-midonet: Consume gate_hook if exists"
*** dizquierdo has joined #openstack-infra08:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add launcher profiles to infracloud openstackci/openstackzuul oscc clouds
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Revert "networking-midonet: Consume gate_hook if exists"
*** yamamoto has quit IRC08:11
*** flepied1 is now known as flepied08:11
openstackgerritMerged openstack-infra/project-config: congress: remove new_arch jobs
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml
*** sflanigan has quit IRC08:17
*** skath has quit IRC08:18
*** skath has joined #openstack-infra08:19
*** esikachev has joined #openstack-infra08:20
*** asettle has joined #openstack-infra08:20
*** dingyichen has quit IRC08:22
*** Genek has quit IRC08:22
*** Genek has joined #openstack-infra08:22
*** chem has joined #openstack-infra08:22
*** pgadiya has joined #openstack-infra08:23
*** newcomer25 has joined #openstack-infra08:24
openstackgerritMerged openstack-infra/project-config: Update api-site zuul config
*** claudiub has joined #openstack-infra08:25
openstackgerritMerged openstack-infra/system-config: Remove rcarrillocruz from infracloud users
*** newcomer25 has left #openstack-infra08:26
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Add k8s-docker-suite-app-murano project to openstack
*** lucas-dinner is now known as lucasagomes08:28
*** senk has quit IRC08:29
openstackgerritMerged openstack-infra/project-config: Adding check-requirements for log-api
*** _nadya_ has joined #openstack-infra08:30
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud oscc to nodepool clouds
*** _nadya_ has quit IRC08:30
*** _nadya_ has joined #openstack-infra08:31
*** jordanP has joined #openstack-infra08:32
*** dtantsur|afk is now known as dtantsur08:32
openstackgerritMerged openstack-infra/project-config: Add os_monasca repository to OpenStack-Ansible
openstackgerritMerged openstack-infra/project-config: Remove whitespaces and add parenthesis to print statements
openstackgerritMerged openstack-infra/project-config: Fix legend for 'Time to Ready' for nodepool dashboard
openstackgerritMerged openstack-infra/project-config: Fix ready node launch attemps panel for nodepool
openstackgerritMerged openstack-infra/project-config: Switch ironic-multinode job to wholedisk agent_ssh
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml
openstackgerritMerged openstack-infra/project-config: Add osic-cloud8 to grafana
openstackgerritMerged openstack-infra/project-config: remove vitrage-release tag rights for vitrage repos
openstackgerritMerged openstack-infra/project-config: sahara/tempest: run also client tests and a pre script
openstackgerritMerged openstack-infra/project-config: Move cinder multinode grenade job to check
openstackgerritMerged openstack-infra/project-config: kuryr-libnetwork-core addition
openstackgerritMerged openstack-infra/project-config: kuryr-kubernetes-core addition
openstackgerritMerged openstack-infra/project-config: kuryr: Add fuxi subproject to gerritbot
openstackgerritMerged openstack-infra/project-config: Few changes to the kolla-kubernetes job
*** andymaier has joined #openstack-infra08:34
*** Genek has quit IRC08:35
*** psachin has quit IRC08:35
*** ccamacho is now known as ccamacho|afk08:37
*** senk has joined #openstack-infra08:38
*** Julien-zte has quit IRC08:41
*** aarefiev_ has joined #openstack-infra08:43
*** azvyagintsev has joined #openstack-infra08:44
*** zul has joined #openstack-infra08:46
*** dizquierdo has quit IRC08:47
*** alexz__ has quit IRC08:47
*** aarefiev has quit IRC08:47
*** yamamoto has joined #openstack-infra08:48
*** openstackgerrit has quit IRC08:48
*** openstackgerrit has joined #openstack-infra08:48
*** aarefiev_ is now known as aarefiev08:50
*** bethwhite_ has joined #openstack-infra08:51
*** ihrachys has joined #openstack-infra08:52
*** rkukura has quit IRC08:53
openstackgerritMerged openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml
*** Na3iL has quit IRC08:55
*** jordanP has quit IRC08:56
*** thorst has joined #openstack-infra08:56
*** jordanP has joined #openstack-infra08:57
*** yamamoto has quit IRC08:57
*** andreas_s has quit IRC08:57
*** yamamoto has joined #openstack-infra08:57
*** andreas_s has joined #openstack-infra08:57
*** yamamoto has quit IRC08:58
*** yamamoto has joined #openstack-infra08:58
openstackgerritMerged openstack-infra/system-config: Add launcher profiles to infracloud openstackci/openstackzuul oscc clouds
openstackgerritMerged openstack-infra/system-config: Add infracloud oscc to nodepool clouds
openstackgerritMerged openstack-infra/project-config: Fix translation setup for networking-odl
*** yaume has quit IRC09:00
*** yaume_ has joined #openstack-infra09:00
*** derekh has joined #openstack-infra09:00
*** sambetts|afk is now known as sambetts09:01
*** thorst has quit IRC09:03
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
openstackgerritRabi Mishra proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin
*** ccamacho|afk is now known as ccamacho09:14
ramishraAJaeger, yolanda , ^^^ rebased the patch as if failed to merge.09:15
*** sarob has joined #openstack-infra09:15
*** tonytan4ever has joined #openstack-infra09:16
*** salv-orlando has joined #openstack-infra09:18
*** e0ne has quit IRC09:18
*** sarob has quit IRC09:20
*** salv-orl_ has quit IRC09:20
*** tonytan4ever has quit IRC09:20
*** gildub has quit IRC09:21
*** dkehn has quit IRC09:22
*** dkehn_ has quit IRC09:22
*** dizquierdo has joined #openstack-infra09:22
*** kaisers_ has quit IRC09:23
*** kzaitsev_mb has joined #openstack-infra09:26
*** yamamoto has quit IRC09:29
*** dkehn has joined #openstack-infra09:29
*** e0ne has joined #openstack-infra09:31
*** yamamoto has joined #openstack-infra09:32
*** zul has quit IRC09:33
AJaegerfungi, thanks for your zuul-env change. Today's translation import looks fine, no zuul-cloner failures! Still waiting for some post jobs to check that the problem is really fixed09:34
AJaegerhashar: could you review , please?09:36
*** kaisers_ has joined #openstack-infra09:37
hasharAJaeger: don't you need to raise bindep requirement to >=2.1.0 ?09:37
hasharif one runs an older version, it would expect other-requirements.txt which is no more around09:38
hasharbut I am nitpicking really09:38
*** zul has joined #openstack-infra09:38
hasharAJaeger: if my comment does not make sense, I will happily CR+2 / W+1 the change09:39
*** dkehn_ has joined #openstack-infra09:39
AJaegertox will install the latest version. Using >= 2.1.0 wouldn't harm but only few will have it.09:40
AJaegerhashar: good thinking.09:40
*** baoli has joined #openstack-infra09:40
AJaegeryolanda: is not imported, could you do the usual magic, please?09:41
hasharAJaeger: I am probably overthinking  anyway09:41
hasharfor openstack CI it is going to be just fine09:41
hasharsince the tox env is build fresh on each run09:41
hasharthe edge case is when running using system packages (unlikely)09:41
AJaegerhashar: the tox env is never used09:41
AJaegerthe tox env is for developers only09:41
hasharor for folks having outdated tox venv in this case they would just have to recreate them with tox -r09:42
*** tosky has joined #openstack-infra09:42
AJaegerthe tox env is a one time think when you setup your system. So, nobody has done this already ;)09:42
AJaegerhashar: so, yes, let's merge it, please09:43
hasharI am sometime abusing:   apt-get install <list of python modules> ;  http_proxy=. https_proxy=. tox --sitepackages09:43
hasharAJaeger: W+1 ed it :}09:44
*** baoli has quit IRC09:44
AJaegerthanks, hashar09:45
*** claudiub|2 has joined #openstack-infra09:45
openstackgerritMerged openstack/python-jenkins: Move other-requirements.txt to bindep.txt
*** claudiub has quit IRC09:47
*** mkoderer__ has joined #openstack-infra09:48
yolandaAJaeger, let me take a look09:49
hasharAJaeger: also I poked you about some quota issue  you had with  OVH a month or so ago. pabelanger got me the answer (OVH fixed the tenant quota manually)09:49
yolandaAJaeger, fixed09:51
*** yamamoto has quit IRC09:58
*** yamamoto has joined #openstack-infra09:58
*** yamamoto has quit IRC09:58
*** claudiub|2 has quit IRC10:00
*** thorst has joined #openstack-infra10:00
*** zhurong_ has quit IRC10:01
*** sshnaidm|afk is now known as sshnaidm10:01
*** yanyanhu has quit IRC10:02
*** xarses_ has joined #openstack-infra10:02
*** yamamoto has joined #openstack-infra10:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job
*** xarses has quit IRC10:05
*** thorst has quit IRC10:07
*** yamamoto has quit IRC10:07
*** yamamoto has joined #openstack-infra10:08
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka
*** e0ne has quit IRC10:09
*** Na3iL has joined #openstack-infra10:15
*** electrofelix has joined #openstack-infra10:17
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka
*** psachin has joined #openstack-infra10:19
*** chem has quit IRC10:22
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH
*** chem has joined #openstack-infra10:22
*** hockeynut has quit IRC10:23
*** lucasagomes is now known as lucas-relocate10:25
*** Genek has joined #openstack-infra10:27
*** Genek_ has joined #openstack-infra10:29
*** Genek has quit IRC10:29
mordredianw: I think that patch is correct.10:29
mordredianw: as for CI - it's possible we have poor test coverage for that I'm guessing (I know we don't have any functional tests for that code path, becaue it's not possible to get a devstack installed that actually works in that configuration, so we have to rely on unit tests for that)10:30
openstackgerritDominic Cleal proposed openstack-infra/jenkins-job-builder: Add cancel-builds-on-update to GHPRB trigger
*** jed56 has quit IRC10:35
*** gildub has joined #openstack-infra10:42
yolandahi mordred, care reviewing a pair of glean patches?,
*** sdague has joined #openstack-infra10:51
*** ccamacho is now known as ccamacho|afk10:53
rcarrillocruzpabelanger: when you get around, i was getting failurs on the cloud launcher runs, by using /etc/openstack/all-clouds.yaml10:53
rcarrillocruzthere was no creds for openstackzuul-bluebox10:53
rcarrillocruzadded creds in hiera10:53
rcarrillocruzand now getting failures due to openstackci-tripleo-test-cloud-rh1 not existing, it appears you call it openstack-nodepool-tripleo10:54
rcarrillocruzshould I change to that name on clouds_launcher/clouds_layouts.yaml  or you want to change it the other way around?10:54
odyssey4meHi everyone - now that has merged, please add me to,members10:55
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: Add query for broken apt mirrors
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: Add query for broken apt mirrors
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
rcarrillocruzodyssey4me: done10:56
odyssey4methanks rcarrillocruz10:56
Genek_hi rcarrillocruz, I'm currently testing on shade. I've heard from David Flanders that you've implemented the Magnum support. Do you know any public clouds that supports Magnum? Thanks10:58
rcarrillocruzoh yeah, sorry, totally forgot to reply10:58
rcarrillocruzi don't know of any clouds with magnum i'm afraid10:58
rcarrillocruzmordred may know?10:58
rcarrillocruzother than that, you better ask in the openstack magnum channel10:58
Genek_Thanks for the suggestion10:59
*** YorikSar has quit IRC11:00
*** asettle has quit IRC11:01
*** asettle has joined #openstack-infra11:02
*** YorikSar has joined #openstack-infra11:02
*** thorst has joined #openstack-infra11:05
*** ccamacho|afk is now known as ccamacho11:05
AJaegerthanks, yolanda.11:07
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Skip OSA Xenial integrated build jobs for Liberty & Mitaka
*** jkilpatr has quit IRC11:07
*** sandanar_ has joined #openstack-infra11:07
AJaegerhashar: I saw that - thanks.11:07
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Fix rabbitmq install for RHEL/CentOS
* rcarrillocruz is uploading ubuntu-trusty dib image from nodepool to infracloud glance11:09
*** sandanar_ has quit IRC11:09
*** sandanar__ has joined #openstack-infra11:09
*** sdague has quit IRC11:10
yolandarcarrillocruz, woot!11:10
yolandareally looking forward to see all that job we've done applied finally11:10
*** sandanar has quit IRC11:10
rcarrillocruzlong trip11:12
*** thorst has quit IRC11:12
*** lucas-relocate is now known as lucas-hungry11:13
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic
mordredrcarrillocruz, Genek_: I am not aware of any public clouds with magnum11:15
*** tonytan4ever has joined #openstack-infra11:17
mordredyolanda: done. I want clarkb to look at
yolandaok no rush11:18
mordredyolanda: I mean, I'm pretty sure it's fine - but he's mr. bridge man :)11:19
yolandamordred, so we were not hitting the problem with glean dealing with bridges, because we had it configured under the same vlan config file. But for CentOS, it needs to be on an indepdendent file, and along with Ricardo we decided that should be cleaner to have it on an independent file as well for Debian11:19
yolandabut then glean comes and tries to deal with it :)11:20
*** Genek_ has quit IRC11:20
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** ramishra has quit IRC11:20
*** rodrigods has quit IRC11:21
*** rodrigods has joined #openstack-infra11:21
openstackgerritMerged openstack-infra/glean: Improve the support for checking vlan interfaces
*** tonytan4ever has quit IRC11:22
*** ramishra has joined #openstack-infra11:22
*** Na3iL has quit IRC11:22
*** kencjohnston_ has quit IRC11:23
*** sdague has joined #openstack-infra11:24
*** kencjohnston has joined #openstack-infra11:24
*** _degorenko|afk is now known as degorenko11:24
openstackgerritMerged openstack-infra/elastic-recheck: Add query for broken apt mirrors
rcarrillocruzregardless of CentOS needing one interface per file, i really prefer having one interface per file myself11:26
rcarrillocruzcos doing ls /etc/network/interfaces.d you can get with a glance what interfaces are configured on that machine11:27
*** YorikSar has quit IRC11:27
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories
yolandarcarrillocruz, also for opnfv, i moved bridge to hiera. I'm thinking on that, or either giving the bridge a static name11:28
yolandabecause on debian/centos, the interface naming is different11:28
yolandado you see value of having br-vlanxxx name or br-ethxxx name? or we could just go with br-infracloud?11:28
rcarrillocruzthere are many things we should move off puppet-infracloud to either system-config or hiera11:29
rcarrillocruzcos any hit of br-vlan2551 on the puppet-infracloud repo is a bug11:29
yolandaip ranges for example...11:29
*** YorikSar has joined #openstack-infra11:29
rcarrillocruzbr-vlan2551 is env spcific, config11:29
rcarrillocruzthus , it should be out of the puppet module11:29
*** matbu is now known as matbu|lch11:29
yolandarcarrillocruz, my point is... we could even name it with a fixed, non meaningful name, such as br-infracloud and that will work everywhere11:30
*** sdague has quit IRC11:31
*** jkilpatr has joined #openstack-infra11:35
rcarrillocruzso cute11:35
rcarrillocruznodepool@nodepool:~$ OS_CLOUD=infracloud openstack server list11:35
rcarrillocruzWARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils11:35
*** sandanar__ has quit IRC11:35
rcarrillocruz| ID                                   | Name        | Status | Networks           | Image Name    |11:35
rcarrillocruz| 1e50978e-1f15-4203-887c-e507afdd0933 | test-trusty | ACTIVE | public= | ubuntu-trusty |11:35
*** sandanar__ has joined #openstack-infra11:36
*** jcoufal has joined #openstack-infra11:37
*** deadnull has joined #openstack-infra11:39
*** jcoufal has quit IRC11:39
*** gildub has quit IRC11:39
*** thorst has joined #openstack-infra11:44
asselin__good morning. Is it just me or is very slow?11:44
*** rfolco has joined #openstack-infra11:47
*** dansmith has quit IRC11:49
asselin__nevermind...working now...11:49
*** dansmith has joined #openstack-infra11:49
*** rfolco_ has joined #openstack-infra11:50
*** pt_15 has joined #openstack-infra11:50
*** rfolco has quit IRC11:51
*** jaosorior has quit IRC11:52
*** jaosorior has joined #openstack-infra11:52
*** dprince has joined #openstack-infra11:55
*** ldnunes has joined #openstack-infra11:55
*** asettle has quit IRC11:55
*** asettle has joined #openstack-infra11:56
*** esberglu has joined #openstack-infra11:58
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Stop using NodePoolBuilder class
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for registering a watch.
*** pradk has joined #openstack-infra12:09
*** tonytan4ever has joined #openstack-infra12:10
*** gouthamr has joined #openstack-infra12:11
*** degorenko has quit IRC12:12
*** degorenko has joined #openstack-infra12:13
*** degorenko has quit IRC12:14
*** degorenko has joined #openstack-infra12:14
asselin__mordred, fyi I submitted a bug for the issue I'm running into:
openstackLaunchpad bug 1616891 in python-openstackclient "openstack server list fails with AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'" [Undecided,New]12:14
*** sandanar_ has joined #openstack-infra12:15
*** zul has quit IRC12:15
*** lucas-hungry is now known as lucasagomes12:15
*** salv-orlando has quit IRC12:15
*** kgiusti has joined #openstack-infra12:16
mordredasselin__: oh wow. lovely12:16
*** tonytan_brb has joined #openstack-infra12:17
*** kgiusti has left #openstack-infra12:17
mordredasselin__: if you only downgrade python-novaclient and oslo-utils, does it work?12:18
*** tpsilva has joined #openstack-infra12:18
asselin__I will try12:18
rcarrillocruzhmm, in the new zuul v2.5/v3 , is there  a recipe to quickly reproduce the test on a vm? i see when we run a job, we keep on folder _zuul_ansible the playbooks, inventory and scripts used by the job execution. However, that's run in the context of a zuul launcher. I assume I could make my VM a zuul launcher itself12:18
*** csatari has quit IRC12:18
*** sandanar__ has quit IRC12:18
rcarrillocruzwondering if there's some doc or someone has a gist for ^12:18
*** jcook has quit IRC12:18
*** baoli has joined #openstack-infra12:18
*** tonytan4ever has quit IRC12:19
*** baoli has quit IRC12:19
*** tchaypo_ has joined #openstack-infra12:19
*** tchaypo has quit IRC12:19
*** prometheanfire has quit IRC12:19
*** tchaypo_ is now known as tchaypo12:19
*** wendar_ has joined #openstack-infra12:19
*** rpodolyaka has quit IRC12:19
*** sc68cal has quit IRC12:20
*** jswarren has quit IRC12:20
*** arif-ali has quit IRC12:20
*** wendar has quit IRC12:20
*** matbu|lch is now known as matbu12:20
mordredasselin__: also you havea  really old six installed12:20
mordredasselin__: I  have 1.10.0 on my system12:20
mordredand oslo.utils says: six>=1.9.0 # MIT12:20
*** shardy has quit IRC12:21
*** sc68cal has joined #openstack-infra12:21
asselin__mordred, I noticed that too...but why? This is what gets installed on a fresh system12:21
mordredin its requirements.txt12:21
*** prometheanfire has joined #openstack-infra12:21
*** sandanar_ has quit IRC12:21
*** csatari has joined #openstack-infra12:21
mordredasselin__: do you happen to know exactly what you are pip installing to get that?12:21
*** bradjones has quit IRC12:21
*** eeiden has quit IRC12:21
mordredasselin__: I just installed python-openstackclient with pip into a new venv and I got six 1.10.012:22
asselin__not's all through puppet. I can try to trace it12:22
mordredasselin__: is it possible you have the python-six distro package installed?12:22
*** shardy has joined #openstack-infra12:22
*** mwhahaha_ has joined #openstack-infra12:23
asselin__python-six                                      install12:23
*** timsim_ has joined #openstack-infra12:23
*** timsim has quit IRC12:23
*** electrical has quit IRC12:23
*** timsim_ is now known as timsim12:23
*** arif-ali has joined #openstack-infra12:23
*** edwarnicke has quit IRC12:23
*** briancurtin has quit IRC12:23
*** PalTale has quit IRC12:23
*** vern has quit IRC12:23
*** mwhahaha has quit IRC12:23
mordredis that maybe at version 1.5 ?12:24
*** mwhahaha_ is now known as mwhahaha12:24
*** asaprykin has quit IRC12:24
asselin__python-six/trusty-updates,now 1.5.2-1ubuntu1 all [installed]12:24
mordredthere's your problem12:24
*** sigmavirus has quit IRC12:24
mordredso, now, the question is - how to fix it12:24
*** rpodolyaka has joined #openstack-infra12:24
*** vern has joined #openstack-infra12:24
*** ansmith has joined #openstack-infra12:24
*** baoli has joined #openstack-infra12:24
asselin__right...this is starting on clean ubuntu 14.04 image12:24
*** baoli has quit IRC12:24
*** PalTale has joined #openstack-infra12:25
*** asaprykin has joined #openstack-infra12:25
mordredyah - so my curiosity is ... is python-six on the clean ubuntu image and it needs to be uninstalled before the pip stuff gets installed?12:25
mordredor is something in our puppet installing python-six explcitly12:25
*** jaosorior is now known as jaosorior_away12:25
*** _sigmavirus24 has joined #openstack-infra12:25
asselin__i will check that12:25
*** electrical has joined #openstack-infra12:25
*** briancurtin has joined #openstack-infra12:25
*** jcook has joined #openstack-infra12:26
mordredI don't see any references to python-six in our puppet on codesearch.o.o12:26
*** edwarnicke has joined #openstack-infra12:26
*** bradjones has joined #openstack-infra12:27
*** bradjones has quit IRC12:27
*** bradjones has joined #openstack-infra12:27
*** mtreinish has quit IRC12:27
mordredso I'm thinking that perhaps putting in (somewhere) an apt-get remove of python-six12:27
*** rkukura has joined #openstack-infra12:27
mordredor, maybe an explicit pip install -U six12:27
mordrednot sure whta the 'best' way is to make this resilient ... this, btw, is the reason we ditched cloud-init  :)12:27
mordredwell, one of them12:27
*** uberjay has quit IRC12:28
mordredbecause cloud-init brings in additional distro python packages, but we have to install some stuff via pip, and the two do not play well together12:28
asselin__git grep python-six12:28
asselin__ doesn't return anything, but could be in a puppet-module or hidden somewhere else. clean image is booting now....12:28
*** _sigmavirus24 is now known as sigmavirus12:30
*** sigmavirus has joined #openstack-infra12:30
asselin__installed in the image: python-six/trusty-updates,now 1.5.2-1ubuntu1 all [installed]12:30
*** mtreinish has joined #openstack-infra12:30
*** uberjay has joined #openstack-infra12:30
*** nwkarsten has joined #openstack-infra12:30
mordredso, try apt-get remove python-six before you install other things12:30
*** weshay_bbiab is now known as weshay12:30
mordredand see if that fixes it12:30
*** gordc has joined #openstack-infra12:31
*** ccamacho is now known as ccamacho|lunch12:33
*** sarob has joined #openstack-infra12:34
*** edmondsw has joined #openstack-infra12:34
AJaegerteam, today's problem might be cinder-python27-db-ubuntu-xenial jobs OOMing - I got a couple of those.12:35
*** kgiusti has joined #openstack-infra12:36
*** scottda has joined #openstack-infra12:36
*** martinkopec_ has joined #openstack-infra12:36
*** rlandy has joined #openstack-infra12:37
AJaegerah, already classified as
openstackLaunchpad bug 1616561 in Cinder "Unit tests periodically oom " [Medium,Confirmed]12:37
*** sbezverk_ has joined #openstack-infra12:37
*** sarob has quit IRC12:38
*** sbezverk has quit IRC12:39
*** mdrabe has joined #openstack-infra12:39
*** martinkopec has quit IRC12:39
*** jcoufal has joined #openstack-infra12:40 is unreachable for me12:40
mordredEmilienM: I can get to it12:41
EmilienMmhh now it is, nevermind.12:41
*** degorenko has quit IRC12:41
*** rhallisey has joined #openstack-infra12:41
openstackgerritMonty Taylor proposed openstack-infra/shade: Refactor TaskManager to be more generic
openstackgerritMonty Taylor proposed openstack-infra/shade: Add submit_function method to TaskManager
openstackgerritMonty Taylor proposed openstack-infra/shade: Change naming style of submitTask
anteayaummmm, gerrit memory usage is quite high:
*** derekh has quit IRC12:44
*** degorenko has joined #openstack-infra12:44
*** degorenko_ has joined #openstack-infra12:44
anteayawe seem to have a pattern of restarting it on a weekly basis12:45
anteayaand appears that according to that cadence we are due anytime now12:45
*** ddieterly has joined #openstack-infra12:45
*** xyang1 has joined #openstack-infra12:45
*** jcoufal has quit IRC12:47
*** derekh has joined #openstack-infra12:47
mordredoh lovely12:47
*** apetrich has quit IRC12:48
*** markvoelker has joined #openstack-infra12:48
anteayamordred: are you in a position to be able to restart it?12:48
anteayaor are you in nyc at a live event and on a mobile device?12:48
* anteaya is guessing the later12:48
mordrednope. I'm at home ...12:49
anteayaoh okay12:49
mordredI have not been involved in the previous restarts - so I don't know if I have all the context and would love it if someone else who did was also around12:49
anteayafair enough12:49
openstackgerritYuval Brik proposed openstack-infra/zuul: webapp: Support get change status by hash id
anteayayolanda: you have performed some gerrit restarts recently, have you not?12:49
*** markusry has joined #openstack-infra12:50
*** thiagop has joined #openstack-infra12:51
mgagne_pabelanger: re credentials: they stay the same12:52
*** apetrich has joined #openstack-infra12:52
anteayamorning cloudnull12:52
cloudnullo/ anteaya12:52
anteayacloudnull: how does your cloud look today?12:52
*** Na3iL has joined #openstack-infra12:52
anteayaall our queues look much happier today12:52
sshnaidmAJaeger, yolanda, pabelanger hi, do you know where can I download template-centos-7 built image from web? if it's possible at all12:52
cloudnulli've not looked at it yet.12:53
anteayacloudnull: fair enough12:53
mgagne_pabelanger: keystone is global at internap. We pre-provisioned the networks for both accounts (one for the account with the mirror, and the other for test nodes)12:53
*** eharney has quit IRC12:53
AJaegersshnaidm: template-centos-7? What is that?12:53
AJaegersshnaidm: if you mean the images we use in our clouds: Those are not uploaded but you can build them yourself12:54
cloudnullbut if the queues in infra are happier I'm thrilled.12:54
cloudnullso i just wont look at it, because no news is good news.12:54
AJaegersshnaidm: there'S tools/ in project-config to built it yourself12:54
*** mgagne_ is now known as mgagne12:54
*** ddieterly has quit IRC12:55
sshnaidmAJaeger, yeah, I mean the images from nodepool12:56
sshnaidmAJaeger, but can't I get it ready from somewhere..?12:56
asselin__mordred, ok, passed the six error. Now ssl cert errors even though verify: false is in clouds.yaml.12:57
openstackLaunchpad bug 1616891 in python-openstackclient "openstack server list fails with AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'" [Undecided,New]12:57
*** woodster_ has joined #openstack-infra12:57
AJaegersshnaidm: no, you cannot - those have keys build in and are really large (5GB+) for sharing12:58
*** zhurong has joined #openstack-infra12:58
sshnaidmAJaeger, ok, thanks12:58
AJaegersshnaidm: what problem are you trying to solve?12:58
*** edtubill has joined #openstack-infra12:58
sshnaidmAJaeger, running job on it, just wanted to save a time12:58
*** salv-orlando has joined #openstack-infra12:58
toskysshnaidm: related:
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Update job skip condition for horizon doc-only changes
mordredasselin__: hrm. paste?12:59
toskyoh, AJaeger already answered12:59
anteayacloudnull: well there is that approach as well12:59
mordredasselin__: oh - I don't thnk openstackclient honors verify: False?12:59
mordred(I'd have to go check)12:59
*** ccamacho|lunch is now known as ccamacho12:59
dtroyer_zzmordred: it should12:59
sshnaidmtosky, right, it is, thanks13:00
asselin__hmmm...this worked fine before...and still does on my other machine.13:00
* dtroyer_zz says in a drive-by as he closes his laptop to go to the airport…13:00
anteayadtroyer_zz: safe travels13:00
mordreddtroyer_zz: cool13:00
mordredasselin__: can you try openstack --insecure server list13:00
mordredasselin__: and just see if that works?13:00
mordred(will help track down the issue)13:00
asselin__yes that works13:01
mordredasselin__: k13:01
mordredasselin__: can you paste the output of "env | grep OS_"13:01
anteaya<-- breakfast13:01
*** psilvad has joined #openstack-infra13:01
mordredasselin__: just out of curiosity, can you try putting insecure: True in your clouds.yaml ?13:02
asselin__yup, that works13:03
*** vikrant has quit IRC13:03
*** fguillot has joined #openstack-infra13:03
*** yarkot1 has joined #openstack-infra13:03
mordredok. thanks. I think there is a bug somewhere in the occ/osc handoff that dtroyer_zz and I will need to go look in to - but maybe for now just doing insecure: True will keep you moving13:03
asselin__mordred, yes that works for me. Let me know if you need a new bug for that and how to update the current bug status13:04
*** yamamoto has quit IRC13:05
*** yamamoto has joined #openstack-infra13:06
mordredasselin__: nah - I think the existing one is fine13:06
*** martinkopec_ has quit IRC13:06
*** baoli has joined #openstack-infra13:07
*** martinkopec has joined #openstack-infra13:07
*** baoli_ has joined #openstack-infra13:07
*** dizquierdo has quit IRC13:08
*** notmyname has joined #openstack-infra13:09
*** julim_ has joined #openstack-infra13:09
*** fguillot_ has joined #openstack-infra13:11
*** psilvad has quit IRC13:11
*** baoli has quit IRC13:11
*** fguillot_ has quit IRC13:12
*** fguillot_ has joined #openstack-infra13:12
*** Wei_Liu has quit IRC13:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_trove to OpenStack-Ansible
*** julim_ has quit IRC13:15
*** dizquierdo has joined #openstack-infra13:16
*** yamamoto has quit IRC13:17
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
*** yamamoto has joined #openstack-infra13:17
*** julim_ has joined #openstack-infra13:17
sc68calgate looks a lot healthier today13:17
AJaegersc68cal: indeed - with exception of cinder python27 tests - lots of OOMs13:18
anteayathanks for your help yesterday, sc68cal13:19
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting
*** jordanP has quit IRC13:20
anteayasc68cal: there was someone asking for help setting up devstack with linuxbridge in -dev yesterday, I told them today might be better13:20
anteayasc68cal: if you feel like taking a peek13:20
sc68calok let me check scrollback13:21
anteayathank you13:21
anteayait was in the -dev channel13:22
*** links has quit IRC13:22
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Upgrade to LV 5.2
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework
*** Wei_Liu has joined #openstack-infra13:25
AJaegerIs there any infra-core to review for openstack-infra/js-afs-blob-store, please? It's adding a bindep.txt file. krotscheck already gave a +2.13:27
*** eranrom has quit IRC13:27
*** fguillot has quit IRC13:28
*** claudiub|2 has joined #openstack-infra13:29
AJaegerthanks, rcarrillocruz !13:29
*** zul has joined #openstack-infra13:31
*** mat128 is now known as mat128|away13:31
*** eharney has joined #openstack-infra13:31
*** roxanagh_ has joined #openstack-infra13:32
anteayaI'm out for most of the morning, I'm still looking for a gerrit restart due to memory usage13:32
anteayathank you13:32
*** matt-borland has joined #openstack-infra13:32
openstackgerritMathieu Mitchell proposed openstack/diskimage-builder: Generate and use upper-constraints for ironic-agent
*** asettle has quit IRC13:35
*** asettle has joined #openstack-infra13:35
*** salv-orlando has quit IRC13:35
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Reduce job log retention to 90 days
*** roxanagh_ has quit IRC13:36
openstackgerritMerged openstack-infra/nodepool: Add new ZK method for registering a watch.
*** edtubill has quit IRC13:36
openstackgerritMerged openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat
openstackgerritMerged openstack-infra/nodepool: Stop using NodePoolBuilder class
openstackgerritFrédéric Guillot proposed openstack-infra/project-config: Add Almanach IRC channel
*** jlvillal has left #openstack-infra13:38
*** senk has quit IRC13:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_trove to OpenStack-Ansible
*** sdague has joined #openstack-infra13:41
*** DrifterZA has quit IRC13:43
sc68calanteaya: lol, I read scrollback13:46
sc68calanteaya: oh god he uses slack somewhere and knows how to use @all to ping everyone at once. :-\13:46
sc68calthe worst idea ever13:46
*** eharney_ has joined #openstack-infra13:46
*** eharney has quit IRC13:47
*** jaosorior_away is now known as jaosorior13:48
*** haaj has joined #openstack-infra13:49
*** haaj has quit IRC13:50
dimspabelanger : fungi : is there a sunset date for switching off the mirantis stackalytics in favor of ?13:50
*** yamamoto has quit IRC13:50
*** eharney_ is now known as eharney13:51
*** yamamoto has joined #openstack-infra13:51
*** melwitt_ has joined #openstack-infra13:51
*** kaisers_ has quit IRC13:52
fungidims: pabelanger was having trouble getting up with the stackalytics maintainers to help us solve a couple of lingering manageability issues. for example right now the only configuration update possible results in a multi-hour outage for the service, but much of its correlation/override data is configuration-driven so it needs frequent reloads13:52
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add iLO, private and public IP addresses for compute003.vanilla
*** hongbin has joined #openstack-infra13:53
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate: Revert "Add q-vpn back to base services but exclude for liberty runs"
openstackgerritSean Dague proposed openstack-infra/project-config: remove grenade jobs from horizon
fungidims: also we need to ascertain whether the dead gerrit ssh connection slash-and-burn has been experiencing is a possible problem for stackalytics.o.o as well13:54
*** ddieterly has joined #openstack-infra13:54
*** rhallisey has quit IRC13:54
dimsfungi : ack thanks!13:55
*** jheroux has joined #openstack-infra13:56
fungifor the reconfiguration issue, we boiled it down to the fact that the service is mostly stateless and seems to reprocess all its data to regenerate stats from scratch, then stores them all in an in-memory database. so any restart means hours of it redoing all that statistical analysis13:56
openstackgerritSean Dague proposed openstack-infra/devstack-gate: don't start horizon in grenade
*** ddieterly has quit IRC13:56
*** bauzas has joined #openstack-infra13:56
dimsfungi : right13:56
*** ddieterly has joined #openstack-infra13:57
*** yamamoto has quit IRC13:57
sdaguefungi et al - would be really nice as it would decouple horizon from the rest of the integrated gate13:57
*** rhallisey has joined #openstack-infra13:57
sdagueespecially as that job doesn't do anything any more13:57
sdagueat least from a horizon perspective13:57
fungidims: so solving that probably means some significant rework of its backend data handling design13:57
*** ociuhandu has joined #openstack-infra13:57
* fungi is unsure whether he should be flattered or frightened that he's the only one called out before "et al"13:58
anteayasc68cal: indeed13:58
dimsfungi : ack. some folks were asking for better kubernetes stats in that tool and i am going to push back on that. hence checking13:58
sdaguefungi: well, you are an active nic that I saw once I connected :)13:58
anteayasc68cal: it amazes me that people enter and ask for advice and then clearly demonstrate they are unwilling to take advice13:58
anteayasc68cal: and then still expect people to spend time on thm13:59
fungisdague: heh13:59
anteayasc68cal: I gave four options which the person then proceeded to reject, demonstrating their cleverness13:59
fungirobcresswell: you seem to be subscribed as a requested reviewer on
sdaguefungi: thank you14:00
sdagueanteaya would you also like to look at - ?14:00
*** e0ne has joined #openstack-infra14:00
sc68calanteaya: yeah. I mean we do have a section in the devstack docs, in the neutron guide about linuxbridge14:00
*** devananda|OSE is now known as devananda|OPS14:01
sc68calnot sure about that create_ovs_taps call he's talking about14:02
anteayasdague: I tried14:02
*** aviau has quit IRC14:02
anteayasc68cal: I had no idea14:02
*** aviau has joined #openstack-infra14:02
anteayafungi: what do you think about a gerrit restart?14:02
robcresswellfungi: It appears so14:02
EmilienMcan someone approve please?14:03
fungianteaya: i need to switch computers to one where i can view the javamelody graphs for it more readily14:03
EmilienMfungi: if you have time, could you add do your review list please? my group is waiting for this new repo, thanks a lot14:04
*** zul has quit IRC14:04
* fungi failed at his policy of not breaking radio silence while still trying to catch up on e-mail14:05
*** zul has joined #openstack-infra14:05
sdaguercarrillocruz: thank you for the review14:05
*** xarses_ has quit IRC14:06
anteayafungi: fair enough14:06
anteayafungi: ah14:06
anteayayeah you gotta keep your own rules14:07
*** Julien-zte has joined #openstack-infra14:08
*** jaosorior has quit IRC14:08
fungiyeah, once people realize that i'm around, the rest of my day is driven by random requests14:08
*** sarob has joined #openstack-infra14:10
*** hockeynut has joined #openstack-infra14:11
*** _ari_ has joined #openstack-infra14:11
fungianteaya: gerrit's not really spending much/any time on java garbage collection yet, which is usually what drives us to restart it14:12
EmilienMAJaeger: can you also look please?14:12
*** zz_dimtruck is now known as dimtruck14:12
*** rbrndt has joined #openstack-infra14:12
AJaegerEmilienM: is that project-config?14:12
EmilienMAJaeger: no14:12
EmilienMAJaeger: aren't you core on puppet infra modules?14:13
AJaegerEmilienM: then I don't have +2 powers - and little knowledge...14:13
fungianteaya: were you getting timeouts from the webui or something?14:13
AJaegerEmilienM: no, I'm not14:13
asselin__mordred, thank you very much for all your help getting passed those issues ^^. I'm not back to where I was and able to launch a node! cc/ rcarrillocruz is now working again (with some local fixes that I'll push up)14:13
EmilienMAJaeger: dang !14:13
EmilienMwho can approve this puppet infra change?
AJaegerEmilienM: project-config is enough for me ;)14:13
robcresswellfungi: Its just because it affects Horizon I assume (I dont know who tagged me) but the change is fine.14:13
rcarrillocruz\o/ too14:13
*** esker has joined #openstack-infra14:13
fungirobcresswell: yeah, i haven't looked but assumed you were added as the horizon infra liaison or something14:13
AJaegerrobcresswell: I tagged you14:14
fungirobcresswell: er, ptl! i forgot you're the horizon ptl now14:14
fungiso de-facto infra liaison for your project ;)14:14
anteayafungi: some folks were reporting they couldn't reach gerrit, so I looked at memory usage in cacti14:14
AJaegerfungi: yeah, wanted to make the PTL aware ;)14:14
*** deadnull is now known as _deadnull14:15
anteayafungi: I personally had no gerrit issues14:15
robcresswellAJaeger: Ah, cool, yeah its fine with me.14:15
anteayafungi: also I'm out for the rest of the morning, thanks for looking at it, I leave the decision to you14:15
*** fguillot_ has quit IRC14:16
fungianteaya: yeah, system memory utilization on review.o.o is misleading. about the only thing that server runs is a java jvm, which is configured to be able to use up to the majority of the memory available14:16
*** kushal has joined #openstack-infra14:16
*** mtanino has joined #openstack-infra14:16
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud-west"
fungiit in turn performs its own memory management within that jvm14:16
*** pgadiya has quit IRC14:17
rcarrillocruzi will wip ^ btw14:17
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud"
*** dtardivel has quit IRC14:17
fungiso while system memory utilization does drop when we restart the jvm (because its cache and uncollected garbage are suddenly dumped into the bitbucket), that's not the reason we restart it14:17
*** fguillot has joined #openstack-infra14:19
fungianteaya: as far as i can see the server is behaving in a healthy manner. if people were complaining about access from the conference, it could have been a local network issue there14:19
openstackgerritMarc Aubry proposed openstack-infra/gerritbot: Pin several dependencies
*** sarob has quit IRC14:20
*** sarob has joined #openstack-infra14:21
*** claudiub|2 has quit IRC14:26
pleia2fungi: prepping the gerrit rename email, we're loosely confirmed for 18:00-22:00 UTC, yeah? (that's what we decided in the meeting)14:28
*** berendt has quit IRC14:28
slaglehi, i had to recreate the flavor used by nodepool to launch instnaces in the tripleo test cloud and the id changed14:28
slaglepabelanger: ^14:28
slaglei'm seeing some instances fail to launch now. i'm thinking nodepool might be using the old id14:29
slagleactually i can recreate the new flavor with the old id...14:29
*** e0ne has quit IRC14:29
fungipleia2: confirmation is slightly firmer now that we've heard back from teh release team14:29
fungipleia2: basically, they expect to have all milestone 3 release activities out of the way well in advance of our start time, but they'll let us know if there's any work running over (and we should try to remember to check in with them regardless, just to be safe)14:30
pleia2ok yeah, I saw dhellmann had talked to ttx14:30
* pleia2 nods14:30
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute003.vanilla metadata for better tracking
fungipleia2: so let's treat it as a firm start time, and if we need to delay the start a little or postpone and reschedule then we'll do that14:31
pleia2and we don't expect to use the whole window, but just in case we need to do offline reindex (I won't mention that, let's just make everyone happy with early gerrit return ;))14:31
pleia2sounds good14:31
*** esker has quit IRC14:32
*** rajinir has joined #openstack-infra14:32
openstackgerritMerged openstack-infra/project-config: Create puppet-panko repository
*** e0ne has joined #openstack-infra14:34
*** edtubill has joined #openstack-infra14:35
openstackgerritMerged openstack-infra/system-config: Document use of its-storyboard in Gerrit config
*** esker has joined #openstack-infra14:35
*** dtardivel has joined #openstack-infra14:35
*** esker has quit IRC14:36
fungiouch, just saw a gate-cinder-python34-db job oom in the gate pipeline...
AJaegerfungi, there are several ;(14:36
AJaegerfungi already classified as
openstackLaunchpad bug 1616561 in Cinder "Unit tests periodically oom " [Medium,Confirmed]14:36
fungiwell that's not good :/14:37
AJaegerindeed - currently no #1 problem14:37
*** ociuhandu has quit IRC14:38
*** esker has joined #openstack-infra14:38
*** esker has quit IRC14:38
fungii wonder if py35 is any better?14:38
AJaegerpy35 is non-voting for cinder14:39
dtroyer_zzasselin: should fix your verify problem in OSC.14:39
*** melwitt_ has quit IRC14:40
AJaegerfungi, didn't hear anything there for py34/py35 - but didn't dig deeper.14:40
fungiyeah, looks like qa/cinder are on it, so good enough for me14:40
*** rkukura has quit IRC14:40
fungiif it becomes too painful i guess they can stop gating on it temporarily14:41
*** sdake has joined #openstack-infra14:42
*** zul has quit IRC14:42
*** DrifterZA has joined #openstack-infra14:43
sshnaidmAJaeger, can you please help with build image? I'm trying to build it using "DISTRO=centos7 IMAGE_NAME=template_centos ./project-config/tools/" but I have an error:  Element 'openstack-repos' not found in 'nodepool/elements:/usr/bin/../share/diskimage-builder/elements' How can I get all elements I need?14:43
AJaegersshnaidm: no, I cannot any further - never used the script. Hope somebody else from the team can...14:44
sshnaidmpabelanger, maybe you know? ^^14:44
*** flepied has quit IRC14:44
AJaegersshnaidm: I would call it from project-config directory, so try "cd project-config" first14:44
fungisshnaidm: i think the most common cause of that error is that your working directory is not the root of the project-config repo. you need to do `DISTRO=centos7 IMAGE_NAME=template_centos ./tools/`14:45
sshnaidmAJaeger, I checked, I really don't have these elements14:45
*** e0ne has quit IRC14:45
fungisshnaidm: you don't have ?14:45
AJaegerThey are in ./nodepool/elements/openstack-repos14:46
*** senk has joined #openstack-infra14:46
sshnaidmAJaeger, fungi aah, I was looking in /usr/share/..14:46
AJaegersshnaidm: see the path, I really think you should cd project-config14:46
fungisshnaidm: the script is looking for elements in ./nodepool/elements which is why your cwd needs to be the root of the project-config repo14:46
fungithey are in that repo14:46
sshnaidmAJaeger, fungi thanks, it works!14:47
*** openstackgerrit has quit IRC14:48
*** openstackgerrit has joined #openstack-infra14:49
*** hockeynut has quit IRC14:51
*** david-lyle has joined #openstack-infra14:53
*** yamamoto has joined #openstack-infra14:54
*** caowei has quit IRC14:56
*** xarses_ has joined #openstack-infra14:56
*** david-lyle has quit IRC14:59
*** yamamoto has quit IRC15:00
*** zul has joined #openstack-infra15:01
*** thcipriani|afk is now known as thcipriani15:02
openstackgerritMerged openstack-infra/puppet-infracloud: Parameterize certificates in infracloud
openstackgerritMerged openstack-infra/system-config: Add iLO, private and public IP addresses for compute003.vanilla
*** tonytan4ever has joined #openstack-infra15:03
*** jpich has quit IRC15:04
*** flepied has joined #openstack-infra15:04
*** tonytan_brb has quit IRC15:06
clarkbWe have a really odd shared queue in the gate pipeline. It includes puppet-swift, puppet-nova, puppet-trove but also python-neutronclient, python-ironic-inspector-client, designate-tempest-plugin, and tripleo-heat-templates15:07
*** bnemec has quit IRC15:08
*** dprince has quit IRC15:09
*** jamesden_ has joined #openstack-infra15:10
clarkbthe common test thread putting them in the same queue is not immediately obvious to me. The tripleo things have a shared job with tripleo things and the puppet things have a shared job with puppet things but why the overlap and inclusion of neutronclient and ironic-inspector-client and the designate tempest plugin15:10
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton
*** zul has quit IRC15:11
*** vinaypotluri has joined #openstack-infra15:11
*** jamesdenton is now known as Guest150615:11
mordredclarkb: sooooooo15:11
mordredclarkb: the dual-network stack detection support uncovered that devstack is providing a dual network stack :)15:12
*** dizquierdo has quit IRC15:12
*** salv-orlando has joined #openstack-infra15:12
mordredShrews: ^^ (you'll love that just as much as clarkb I think)15:12
*** ociuhandu has joined #openstack-infra15:12
*** bnemec has joined #openstack-infra15:13
mordredI'm on a call I'll chat with sc68cal about his review on the related shade patch when I get off the phone15:13
clarkband the shade force ipv4 means give me a public ip thats ipv4 and do floating ips etc even if ipv6 is an option?15:13
mordredbasically "I know you detected that I can do ipv6, but srrsly, I wanna do ipv4"15:14
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove EPEL usage
mordredthe nodepool thing makes me wnat to bang on the shade patch a little more15:15
*** WJMUqqJZRD has joined #openstack-infra15:15
mordredto make sure there isn't another niggly that got missed15:15
clarkbcouldn't shade just always do its best to do both by default?15:15
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster
pabelangermgagne: thanks, creds appear to be working15:15
clarkbrather than needing to explicitly flag things?15:15
*** mat128|away is now known as mat12815:16
pabelangermordred: clarkb: fungi: ^ first changes to add internap mtl01 to puppetmaster15:16
pabelangeron to cloud_launcher next15:16
mordredmgagne: should we add mtl01 to os-client-config vendors?15:16
mordredclarkb: it could ... although we'd need a way to turn that off for osic in that case15:17
mgagnemordred: the region is not publicly available yet and won't be for a couple of months.15:17
mordredmgagne: ok. cool15:17
mgagnepabelanger: each region has a set of networks too. Will it be considered in the config?15:17
mordredclarkb: because with osic we could make a floating ip but we don't want to15:17
mordredclarkb: but with devstack we do want to make a floating ip15:17
*** salv-orl_ has joined #openstack-infra15:18
mordredclarkb: (this is why I think it's a good nudge that the nodepool test gave us and I think it's worth digging in to further when I'm not on the phone... I agree with you on the ick factor of the config here)15:18
pabelangermgagne: Ya, I was wondering about that.  We might need to make changes to our layout15:18
clarkbmordred: but osic should only make a floating ip if you are using GATEWAY_NET?15:18
clarkbI guess I don't fully understand the path to "can/should I floating IP"15:18
mordredclarkb: right. it's possible that I've overtought that part15:18
*** vhosakot has joined #openstack-infra15:19
mordredclarkb: and we're safe on osic because of lack of router connected to GATEWAY_NET_IPV615:19
*** furlongm_ has joined #openstack-infra15:19
mordredanywho - tl;dr - I think there is more work to be done here before we're happy15:19
clarkbat least that is what I was expecting in my "do as much as I can by default" idea15:19
*** furlongm has quit IRC15:20
*** salv-orlando has quit IRC15:20
anteayafungi: thanks for explaining the gerrit memory utilization situation, it was EmilienM who mentioned not being able to reach gerrit this morning, I don't know if he is at a conference or not15:20
clarkbmordred: and shows it to be preferring ipv4 with devstack15:21
mgagnepabelanger: and AFAIK, you don't use the LAN network right?15:21
*** links has joined #openstack-infra15:21
*** weshay is now known as weshay_afk15:21
*** sarob has quit IRC15:21
EmilienManteaya: no, I'm in old Québec :)15:22
pabelangermgagne: let me check, unless clarkb know of the top of his head15:22
*** sarob has joined #openstack-infra15:22
mgagnepabelanger: ok, just want to make sure I made the right assumption. Because we didn't provision a LAN network in mtl01.15:22
*** sdague has quit IRC15:22
clarkbpabelanger: mgagne meaning do we rely on l2 between the instances? no because enough clouds don't provide us that so we end up just making our own l2 overlays if they are necessary15:22
openstackgerritMerged openstack-infra/system-config: Add compute003.vanilla metadata for better tracking
mgagneclarkb: just the L2/L3 LAN networks. (whatever connectivity type you need)15:23
pabelangermordred: So, with clouds.yaml for internap. Should we remove the current network fields (since nyj01 and mtl01 are different) or create a new entry per region?  I feel removing them is right way, but wanted to confirm15:23
fungiEmilienM: ahs it continued to be a problem? are you stuck having to reach gerrit through the oqc municipal aqueducts?15:23
pabelangerclarkb: Ya, we have 2 networks defined in clouds.yaml today (WAN / LAN).  Why is that?15:23
openstackgerritMaksim Malchuk proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5
mgagnepabelanger: I think the issue was: How can nodepool detect the right network with public connectivity.15:24
pabelangermgagne: maybe15:24
pabelangerchecking git history15:24
EmilienMfungi: no, it was during ~5 min but now everything is fine I can continue to reach gerrit through canadian cables :)15:24
clarkbmordred: ok reviewed the nodepool chnage15:24
rajinir Hi, is there automatic way or script of some sort to recheck all the failed builds in the past week?15:25
mordredclarkb: thx15:25
clarkbpabelanger: that I do not know. I think it was jeblair who set up the first internap cloud so may know15:25
EmilienMfungi: my only hope is that cable doesn't disturb the mooses15:25
fungiEmilienM: okay, good. if the problem comes back and persists, we can try to troubleshoot15:25
mordredpabelanger: because we cannot detect their qualities15:25
mordredpabelanger: you need to put them into clouds.yaml to tell shade which network is public and which is private15:25
clarkbrajinir: I think our preference would be that there is a little critical thought into whether or not a failure is due to the change or needs a recheck15:25
mordredpabelanger: it's easy for a human - the public network has "WAN" in the name15:26
*** zul has joined #openstack-infra15:26
mordredpabelanger: those settings can be set on a per-region basis15:26
rajinirclarb: ok, I take that as a no. In our case we had hardware issues which is solved now15:27
pabelangermordred: okay, cool. let me read the docs15:27
fungirajinir: oh, you mean for your third-party ci results?15:27
*** hockeynut has joined #openstack-infra15:27
fungirajinir: you can write a shell script that loops over the `zuul enqueue` subcomannd15:27
*** rhallisey_ has joined #openstack-infra15:28
fungirajinir: that's one of the primary reasons that command-line tool exists15:28
pabelangerthere is an internap example in the os-client-config readme15:28
anteayaEmilienM: figured you were in old quebec15:28
*** sarob has quit IRC15:28
mordredpabelanger: up15:28
mordredpabelanger: we wrote the feature because of internap :)15:28
anteayayeah don't disturb the mooses15:28
*** nmagnezi has joined #openstack-infra15:28
pabelangermordred: excellent! Using that now15:28
* anteaya has a story of being chased by a moose15:28
rajinirfungi: exactly. that's cool. will give it a try15:29
fungirajinir: you'd likely need to perform a gerrit query first to look up the change,patchset numbers for every change where your ci account has a verify -1 vote15:29
fungirajinir: and then turn that into a series of zuul enqueue commands15:29
*** jlvillal has joined #openstack-infra15:29
*** _deadnull is now known as deadnull15:29
*** deadnull has quit IRC15:30
*** sdague has joined #openstack-infra15:30
fungirajinir: gerrit has a very flexible and well-documented rest api:
*** derekh has quit IRC15:31
rajinirfungi: This is very helpful. Thank you15:31
*** fguillot has quit IRC15:32
*** rhallisey has quit IRC15:32
*** sandanar has joined #openstack-infra15:32
fungiyou're welcome15:32
fungirajinir: and zuul enqueue is documented at
openstackgerritMaksim Malchuk proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5
clarkbpabelanger: mordred mgagne then in nodepool we can specify that we only need the WAN network?15:34
*** derekh has joined #openstack-infra15:34
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster
mordredclarkb: whatever we did for the other internap in nodepool15:35
* mordred can't remember15:35
*** andreas_s has quit IRC15:35
*** psachin has quit IRC15:36
*** xarses_ is now known as xarses15:37
*** hockeynu_ has joined #openstack-infra15:38
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster
*** gouthamr is now known as gouthamr|afk15:38
*** sarob has joined #openstack-infra15:39
*** Swami has joined #openstack-infra15:39
*** martinkopec has quit IRC15:40
*** kaisers_ has joined #openstack-infra15:40
*** armax has joined #openstack-infra15:41
*** hockeynut has quit IRC15:41
*** yaume_ has quit IRC15:41
*** roxanagh_ has joined #openstack-infra15:42
*** kaisers_ has quit IRC15:44
mgagnepabelanger: openstackci-internap vs openstackjenkins-internap ?15:44
*** rhallisey_ is now known as rhallisey15:45
pabelangermgagne: 1 for mirror, other is nodepool15:45
mgagnepabelanger: ok, both don't have the same network name fyi15:46
pabelangermgagne: also, just did a test: any thoughts on the error?15:46
*** andymaier has quit IRC15:46
clarkbhrm now we have an integrated gate and a horizon/tempest-horizon queue except the integrated gate has nothingin it and the horizon one has not horizon in it15:46
pabelangermgagne: okay, will confirm that now15:46
clarkbI wonder if this is just a transitive state while zuul shuffles things around. It looks really funny on the status page15:46
*** roxanagh_ has quit IRC15:46
clarkbcan someone thati s not me confirm they see the same thing ^15:47
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar
mordredclarkb: I see horizon and not-horizon in the horizon pipeline15:47
openstackgerritMerged openstack-infra/system-config: Add compute004 to compute007 vanilla machines to static infracloud inventory
mgagnepabelanger: will be looking into it shortly15:48
*** _nadya_ has quit IRC15:48
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Limit Heat engine workers to 4 in rh1
mgagnepabelanger: make sure you use the right tenant.15:48
mgagnepabelanger: the one you used is for nodepool and don't have capacity for B series, only A series15:48
pabelangermgagne: okay, that could be it15:49
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
clarkbjeblair: the current state of the zuul status page may be of interest to you15:49
clarkbjeblair: it seems to have confused the integreated gate and horizon15:50
pabelangermgagne: thanks, that was the issue15:50
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
sdagueclarkb: we just split that15:52
clarkbhrm and there is a straggler neutron change that could potentially merge without cross testing with the rest of the integreated gate15:52
clarkbsdague: I think we may have broken split it out then. Look at hte zuul status page15:52
sdagueok, well all that done was to stop 1 job on horizon15:53
*** shashank_hegde has joined #openstack-infra15:53
sdagueare you sure it's not just a visualization bug15:53
fungiclarkb: i want to say in the past we've tracked that down to a status rendering issue15:54
fungibut i could certainly be wrong15:54
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
clarkbsdague: no I am not. I guess I could try to wrap my head around all the json15:54
* clarkb tries15:54
*** abashmak has joined #openstack-infra15:54
fungior just double-check by grabbing zuul refs from the merger15:54
clarkbfungi: well they are all still tested together15:55
sdagueclarkb: for reference is the change that probably triggered this15:55
clarkbjust in the horizon queue not the integrated gate queue15:55
fungiyeah, it was purely a job removal15:55
jeblairmy guess based on the timestamps is that the previous contents of the integrated queue ended up in the horizon queue, but new changes are going into the correct ones15:56
sdagueit should make horizon have an independent gate15:56
sdaguebut there are enough changes in the pipeline that I can imagine it being confused until everything is fresh in there15:56
abashmakGood morning folks. have a question about using OpenStack with mysql 5.6+ (MariaDB 10.0+). I'm seeing an "foregin key too large" error when accessing DB on Ubuntu 16.04, which uses MariaDB 10.015:57
jeblairwe should probably do a full restart, otherwise we're going to have 2 integrated gates fighting it out for the next several hours15:57
clarkbjeblair: ok15:57
*** kushal has quit IRC15:57
clarkbanother thing I have noticed is that puppet openstack things, tripleo, and a bunch of the python clients all share a queue15:57
fungioh, so it's not that the horizon change there is actually still running jobs in the integrated queue but zuul thinks it should display it as being in its own queue?15:57
clarkbnot sure if that is also related but it means that python-*clients will take up to 3 ish hours to get tested if a puppet/tripleo change is ahead of them15:57
*** matbu is now known as matbu|afk15:57
clarkbwhich is :(15:58
abashmakWas wondering if someone's run into that. I think I heard OS just moved the gate jobs to 16.04. Wondering how that issue was solved?15:58
pabelangerclarkb: I haven't checked, but this could be a result of tripleo starting to run more jobs in public clouds and other projects15:58
pabelangerclarkb: we likely need to optimize it15:58
*** kushal has joined #openstack-infra15:58
fungiabashmak: openstack deployed from master branch tip of all projects?15:58
clarkbpabelanger: it definitely need to be cleaned up. I don't think we want tripleo bogging down the python clients. And not sure if the puppet folks want that either but I will let them decide15:58
pabelangerclarkb: yup15:59
fungiabashmak: if you're running mitaka, you're probably better off checking with the ubuntu package maintainers since they shipped mitaka packages with ubuntu 16.04 lts15:59
fungiupstream, we didn't test mitaka on 16.0415:59
abashmakfungi: I'm just deploying individual services one at a time, i.e. keystone, I also built Glance from source15:59
*** matthewbodkin has quit IRC16:00
clarkbalso designate is in that queue too16:00
jeblairit looks liek all of the "real" release jobs have happened, it's just the wheel build stuff that's stuck in the queue16:00
jeblairso i think we're safe to restart now16:00
*** links has quit IRC16:00
fungiabashmak: and we're still in the process of getting master branch jobs working on 16.04 actually. i think a lot of our jobs for newton are still running on 14.0416:00
jeblairi'm thinking: save queue; stop zuul; hard-stop all launchers; start zuul; start all launchers; restore queue16:01
jeblairthat should do a full nodepool recycle as well16:01
SpamapSjeblair: how does one save/restore the queue btw? We've been looking at doing that.16:02
fungiabashmak: the qa team in #openstack-qa is likely to know more about whether there are lingering issues with mariadb on ubuntu 16.04 and/or whether they did anything to work around that16:02
* SpamapS closes eyes and crosses fingers and mumbles "There's no place like documentation..."16:02
abashmakfungi: thanks much, I'll ping them16:02
jeblairSpamapS: wish granted :)16:02
*** derekh has quit IRC16:02
pabelangerclarkb: is it possible it is the releasenote jobs?16:03
pabelangerclarkb: that's causing the shared queue16:03
jeblairi didn't see any objection with my plan, so i'm going to proceed...16:03
fungijeblair: yeah, seems like a reasonable precaution16:03
clarkbpabelanger: the release notes jobs are project specific not a single shared job so I don't think so16:03
clarkbjeblair: yup go for it16:03
jeblairbombs away16:04
*** jianghuaw-m has joined #openstack-infra16:04
* SpamapS opens eyes, and hugs the python16:04
sdaguejeblair / clarkb - I think as soon as 349997,1 merges, the visualization probably fixes itself16:04
anteayamorning zaro16:05
*** zul has quit IRC16:07
sdagueclarkb: yeh, why is check-osc-clients joining the world?16:07
clarkbsdague: oh is that the job thats doing it?16:08
sdagueI think so16:08
clarkbsdague: any shared (transitive) job will do that16:08
clarkbthe idea being that its an integration test that should not be broken by not cross testing16:08
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
*** sarob has quit IRC16:09
sdagueright, but we know doing that arbitrarily just means the giant pile of mud16:10
fungidoing it selectively is even more accident-prone though, yeah?16:10
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
fungifor a lot of these integration test jobs, trying to keep track of all the transitively related projects which should be serialized to confirm the integration test actually integrates changes in the pipeline would be madness16:11
mugsiei seem to remember there was a place to add filters to make sure files from devstack jobs got sent to logstash - but I cannot find it now ... does anyone have any pointers?16:12
jeblairclarkb: re tripleo+puppet -- since the big slow tripleo jobs happen in their own pipeline, if they keep the regulare check/gate jobs to reasonable lengths, it shouldn't necessarily adversely impact other projects sharing their queue.16:12
sc68calmordred: I'm back16:13
EmilienMfungi: gerrit is super slow for me now16:13
fricklerEmilienM: +1, was about to write the same16:13
jlvillalNot responding for me16:13
mugsiefungi: EmilienM: met too - getting 502 proxy Errors16:13
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
clarkbjeblair: there was a 2.5 hour long job in the shared pipelinr so I dont think they have limitrd the long tests to their pipeline16:14
sdaguethat I think is the link16:14
clarkbjeblair: it was running against puppet swift and holding up everything behind it including python-neutronclient16:15
*** tonytan_brb has joined #openstack-infra16:15
sdaguewhich was fine when it wasn't in the gate16:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
fungii'm seeing 1019 sockets for 443/tcp on review.openstack.org16:15
fungiit's falling now16:15
fungiEmilienM: frickler: mugsie: clarkb surmised yesterday we're maxing out the open sockets on review.o.o16:15
sdagueclarkb: removing that lynch pin would probably let the python clients not sit behind giant puppet testing16:16
sdagueclarkb: anyway, dropping again here. Lunch at ops summit, but if you want to decouple, that's the thing I'd remove16:16
clarkbsdague: ok16:16
sdaguethe osc team I think got way to agressive on gate the world against them to prevent their breaks, which caused this16:17
pabelangersdague: clarkb: Ya, I think once check-tripleo-jobs landed, the queues merged16:17
jeblairmaybe we could look into it a little more before we just remove it -- python clients testing with osc does not sound necessarily like a bad thing16:17
pabelangersince those jobs now run on public clouds16:17
fungiEmilienM: frickler: mugsie: well, file descriptors ultimately i guess? so apache is proxying each connection and when we get to ~1024 that's 2048 file descriptors apache has open16:17
sdaguejeblair: it's fine, but when you classifiy the tripleo-client as one of those clients16:17
sdaguethen it gets weird16:17
*** tonytan4ever has quit IRC16:17
sdaguebecause that cross checks with puppet + tripleo16:17
jeblairsdague: oh, you're saying specifically remove it from tripleoclient16:17
jeblairsdague: and yeah, that does sound like the place to focus :)16:18
jeblairsorry i misparsed16:18
sdagueyep, no worries16:18
*** dprince has joined #openstack-infra16:19
sdagueok, dropping, have fun folks :)16:19
redrobotHi Infra friends!16:20
redrobotI was wondering where I can report an account that is vandalizing the wiki16:20
*** abashmak has left #openstack-infra16:20
*** zhurong has quit IRC16:20
redrobotThe user
redrobotjust deleted the blueprint page to my project to redirect it to a spam page about Dell OpenStack support16:20
jeblairit's probably worth considering the proximity of the gerrit slowness to the zuul restart (where zuul will have been issueing many queries) it won't have been using http sockets, but it may have slowed gerrit down enough to generate a backlog16:21
fungiredrobot: i check for them several times a day and clean them up16:21
fungilooking again now16:21
redrobotfungi thanks!16:21
fungiredrobot: i've blocked and deleted new pages created by the two spam users who just showed up in the last hour. one of them moves some existing pages, so i'll unroll that16:22
fungier, moved16:22
redrobotfungi good deal, thanks!16:22
*** sdague has quit IRC16:23
*** raunak has joined #openstack-infra16:24
*** zul has joined #openstack-infra16:25
*** shashank_hegde has quit IRC16:25
rcarrillocruzpabelanger, clarkb : i created an instance on the infracloud, based off Nodepoo ubuntu trusty dib image16:25
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
rcarrillocruznow i'm not sure how to test drive a given job on it. In zuul v2.5/3, there's _zuul_ansible folder containing the playbook, inventory and bash scripts for a given test run, however that is executed in a Zuul runner context16:26
rcarrillocruzhow you folks tackle that?16:26
rcarrillocruzdo you make your test vms a zuul runner and run everything from it?16:26
clarkbrcarrillocruz: I wpuod use xenial and run the script for a devstack gate run16:27
*** fguillot has joined #openstack-infra16:27
rcarrillocruzso a dsvm16:27
clarkbno I just run and or tox16:27
clarkbclone nova and tox -epy2716:27
rcarrillocruzand fwiw16:27
clarkbwget and run it16:27
rcarrillocruzi wip'd it16:27
*** gyee has joined #openstack-infra16:27
fungiredrobot: okay, all unrolled and cleaned up now. a third spammer showed up while i was doing that, but i've now blocked and cleaned up after that one as well16:28
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster
clarkbsince these are our puvlished run it lcally steps I like to use them16:28
*** hockeynu_ has quit IRC16:28
rcarrillocruzi find potentially the new zuul stuff could be easier for folks to reproduce test runs locally, but didnt' want to go the rabbit hole of figuring zuul runner puppet deps and all16:28
pabelangermgagne: thanks! Update network information^16:28
rcarrillocruzjust in case anyone had a paste/gist to do it16:28
*** tphummel has joined #openstack-infra16:28
rcarrillocruz is the revert of the infracloud nodepool.yaml, but tweaked a bit for what we have now16:29
*** hockeynut has joined #openstack-infra16:29
clarkbrcarrillocruz: you would need to edit the playbooks too to remove the file copies and probably other items16:29
rcarrillocruzyeah, and the inventory to put the relevant IP16:29
rcarrillocruzbut i think it may be doable-ish16:29
rcarrillocruzi'll get back to it when i have sorted all the infracloud bring up16:30
openstackgerritZara proposed openstack-infra/infra-manual: Document commit message syntax for StoryBoard
fungiclarkb: so are you thinking we should be doing something similar to in puppet-gerrit?16:34
jeblairrcarrillocruz: we won't be doing the same thing in zuulv3, so i would rather avoid recommending people use the zuul ansible folder to reproduce builds16:35
jeblairrcarrillocruz: i don't want to introduce a feature that we're going to remove16:35
pabelangerclarkb: mordred: we haven't restarted nodepool today right?16:36
*** dprince has quit IRC16:37
zigoWeird, I'm not having any ready nodes, they're not building either, and there's 4 jobs waiting for a VM. Has zuul gone mad?16:37
zigoOr are we out of capacity?16:37
clarkbfungi: ya I think so16:37
clarkbpabelanger: not yet16:37
zigoAh, building now! :)16:38
pabelangerclarkb: okay, tripleo-test-cloud-rh1 is basically down because is cannot get FIPs16:38
jeblairrcarrillocruz: also, it's important that our tests are simple -- as clarkb says, all you really need to do is run 'tox' or ''16:38
clarkbjeblair: ++ thats why I try to stick to currently published methods. and tox work well16:38
fungiclarkb: fwiw, that last burst of problem reports seems to coincide with another spike of similar magnitude to yesterday
pabelangerclarkb: which I think mordred has fixed16:38
*** e0ne has joined #openstack-infra16:39
clarkbpabelanger: we are waiting on a few more fixes aiui including one to nodepool16:39
clarkbpabelanger: to handle the dual ip stack of osic16:39
fungiclarkb: my guess is it's topping out at ~1024 open connections but not for a full 5-minute sample so it's only averaging out to ~80016:39
clarkbfungi: gotcha16:39
*** Apoorva has joined #openstack-infra16:40
fungithat would basically be a 4-minute spike to 102416:40
pabelangerclarkb: no problems, I'm going to decrease capacity in tripleo-test-cloud-rh1 to make it lest bursty. We only started having problems when we bumped it16:40
*** sshnaidm is now known as sshnaidm|afk16:40
*** Apoorva has quit IRC16:40
* zigo has built 97 packages so far using OpenStack infra16:40
*** jpich has joined #openstack-infra16:40
zigopabelanger: It's working very well now! :P16:40
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories
openstackgerritKiall Mac Innes proposed openstack-infra/system-config: Add Designate logs to Logstash
*** hashar has quit IRC16:41
*** Apoorva has joined #openstack-infra16:42
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Increase rh1 max-servers to 75"
*** ifarkas is now known as ifarkas_afk16:42
pabelangerclarkb: ^16:42
pabelangerzigo: excellent16:42
clarkbmordred: maybe you can tldr the current backlog for nodepool restart?16:42
*** dimtruck is now known as zz_dimtruck16:42
*** e0ne has quit IRC16:43
*** roxanagh_ has joined #openstack-infra16:43
pabelangerslagle: ^ revert tripleo-test-cloud-rh1 to 50nodes, until we restart nodepool16:43
slaglepabelanger: ok, worth trying i guess16:44
slaglei still don't see any erorrs with requesting or associating fip's in the cloud16:44
slagleit happens very quickly16:44
pabelangeranother 5mins and nodes will start deleting again16:45
*** rcernin has quit IRC16:45
slagleyes, and last time i saw a bunch of requests come in to create fip's, all were successful16:46
slaglebut nothing assigned them to the instances16:46
*** gouthamr|afk is now known as gouthamr16:46
jeblairpabelanger: my process to get fingerprints from all ansible runs on the log server is still running.  it's collected 950k fingerprints so far, but has not found the one we're looking for.16:46
jeblairi'll check again tomorrow :)16:47
*** roxanagh_ has quit IRC16:47
*** jamesdenton has joined #openstack-infra16:48
pabelangerjeblair: Oh, nice.  I had to revert our ssh-keygen patch in zuul, because ssh-keygen is different between fedora and trusty. I have a fix, will push it up now16:48
openstackgerritChris Krelle proposed openstack/diskimage-builder: WIP: A hardware burn-in element.
*** jamesden_ has quit IRC16:48
*** florianf has quit IRC16:50
*** david-lyle has joined #openstack-infra16:50
*** Kiall has quit IRC16:51
openstackgerritPaul Belanger proposed openstack-infra/zuul: Revert "Revert "Store ssh_host_key of remote node""
pabelangerjeblair: ^ correct syntax now16:51
anteayafungi: are the current wiki spammers experiencing all our latest captchas and still able to spam?16:51
* anteaya boggles at the fantastic condition of the check, gate and post queues16:52
pabelangerindeed, looks to be in good shape now16:53
*** jamesdenton has quit IRC16:53
pabelangerminus cinder's OOMs16:53
*** dprince has joined #openstack-infra16:53
anteayabest I've ever seen at this point in the release16:53
*** _nadya_ has joined #openstack-infra16:53
anteayaincluding cinder's OOMs16:53
*** esikachev has quit IRC16:54
*** jamesdenton has joined #openstack-infra16:54
pabelangerslagle: should have see an attempt that time16:55
*** DrifterZA has quit IRC16:55
*** _nadya_ has quit IRC16:55
*** tesseract- has quit IRC16:56
stevemarAJaeger: regarding the translation sync job break ... do we need to revert the osc change?16:57
*** kushal has quit IRC16:57
*** mikelk has quit IRC16:58
*** fguillot_ has joined #openstack-infra16:58
*** tkelsey has joined #openstack-infra16:58
*** fguillot_ has quit IRC16:58
*** fguillot has quit IRC16:58
slaglepabelanger: yea, 6 new instnaces got spawned and they got fip's16:59
*** edtubill has quit IRC17:01
pabelangerslagle: should see a few more happening now17:01
*** edtubill has joined #openstack-infra17:01
pabelangerslagle: like I said, problem happens a lot more when we burst a lot of nodes a once17:02
openstackgerritLuigi Toscano proposed openstack-infra/project-config: Enable Neutron for the Tempest/Sahara tests
*** weshay_afk is now known as weshay17:03
slaglepabelanger: maybe we should enable debug logs for nova/neutron in the cloud to see what might be happening17:03
*** zul has quit IRC17:04
AJaegerstevemar: no need to revert the osc change - it should be fine as is with current zuul-cloner. I checked logs and it is fine now.17:04
AJaegerstevemar: you can if you want - should work either way.17:04
pabelangerslagle: if we still have the issue after the latest release of shade and nodepool retart we could do that17:04
*** jerryz has joined #openstack-infra17:05
*** hockeynut has quit IRC17:05
*** jkilpatr has quit IRC17:05
slaglepabelanger: ok. b/c we just assigned 40 fip's almost all at once, and it went fine17:05
dougwigare the gerrit issues a known problem?17:05
*** e0ne has joined #openstack-infra17:07
anteayadougwig: what are you experiencing?17:07
*** sarob has joined #openstack-infra17:07
dougwiganteaya: overall slow, 502 errors.  whole room here at midcycle, across several networks.17:07
* AJaeger experiences right now a timeout on every second change in gerrit ;(17:07
*** zz_dimtruck is now known as dimtruck17:07
mordredpabelanger, clarkb: tl;dr - I'm finally off the phone and now I need to go do some poking at the issue exposed by the nodepool test17:07
*** hockeynut has joined #openstack-infra17:08
*** asettle has quit IRC17:08
mordredI'll give a better tl;dr once I've context switched from the call into the troubleshooting17:08
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to grafana dashboard
*** asettle has joined #openstack-infra17:09
anteayadougwig: thank you, yes we are having a problem with how apache is configured regarding its max connections17:09
dougwiganteaya: k, thanks.17:09
anteayadougwig: it can take more connecitons but the current configuration is telling it it is at its max17:09
anteayadougwig: thanks for sharing your experience17:10
*** kushal has joined #openstack-infra17:10
anteayadougwig: I don't think we had hit it before but we had a spike yesterday, then again today17:10
stevemarAJaeger yeah, gerrit needs a restart :P17:11
dhellmannI have a patch for which jenkins claims to have started the gate jobs but that isn't showing up in the zuul status page:
dhellmannjenkins posted ~15 minutes ago17:11
anteayadhellmann: queue lengths just reached zero17:11
anteayais it there now?17:12
dhellmannanteaya : I don't see it; I thought I saw the event queue lengths at 0 a while back but maybe I had stale data in my browser17:12
*** lucasagomes is now known as lucas-dinner17:12
anteayahmmm, no I also don't see it17:13
*** chem is now known as chem|off17:13
anteayaand I confirm that starting gate jobs was posted on the patch almost 20 minutes ago17:13
dhellmannanteaya : it seemed weird that jenkins was reporting already if zuul wasn't showing it, but I may have a bad mental model of how those things are related17:13
*** asettle has quit IRC17:13
AJaegerdid zuul get a timeout accessing gerrit - like us humans?17:13
anteayaAJaeger: maybe?17:14
*** sarob has quit IRC17:14
dhellmanndoesn't zuul tell jenkins when to start the jobs?17:14
AJaegerjeblair: do you have any ideas? ^ or should dhellmann toggle the +W (remove/add it again) ?17:14
fungianteaya: yes, some spammers are bypassing the newer captcha implementation. i think it's preventing more than the old one did17:16
*** flepied1 has joined #openstack-infra17:16
fungianteaya: also i reenabled image uploads, which i've confirmed is still not covered by the confirmedit captchas, so even with that the volume has been relatively manageable17:16
EmilienMdhellmann: there is no jenkins anymore17:17
anteayafungi: okay thank you17:17
dhellmannEmilienM : that makes it even more mysterious :-)17:17
*** amotoki has quit IRC17:17
fungistevemar: i don't think gerrit needs a restart from what i can see. instead it's getting an increased number of connections lately. we need to tune apache better to handle that volume17:17
EmilienMdhellmann: zuul now runs the jobs with ansible (zuul-launcher)17:17
anteayadougwig: ^^17:18
anteayadougwig: fungi's answer to stevemar17:18
*** abregman has quit IRC17:18
stevemarfungi: i just assume gerrit needs a restart whenever it's slow :P17:18
dhellmannEmilienM : I assume it's still just using the old jenkins user in gerrit to announce when the jobs start. Since it did that, what happened to the info about those jobs in the status page?17:18
fungistevemar: unfortunately, computer software is more subtly complex than that :/17:19
dhellmannfungi : bah17:19
fungii'm putting together an apache tuning change now17:19
*** hockeynut has quit IRC17:19
stevemarfungi: just teasing :)17:19
*** flepied has quit IRC17:19
fungistevemar: great show though ;)17:19
EmilienMdhellmann: jobs in status page ( ) are still updated17:19
jeblairdhellmann: looking17:19
dhellmannEmilienM : do you see 360612 on your page?17:20
*** jkilpatr has joined #openstack-infra17:20
*** dizquierdo has joined #openstack-infra17:20
dhellmannjeblair : thanks17:20
EmilienMdhellmann: no, but I think it was squeezed during a service interruption, maybe networking or zuul was restarted?17:21
*** fguillot has joined #openstack-infra17:21
EmilienMdhellmann: best thing to do is to do is to reapprove it17:21
dhellmannEmilienM : ok, I'll wait to see what jeblair says first since he's looking at it17:21
EmilienMdhellmann: remove -A and +A againt17:22
EmilienMit will trigger jobs in gate17:22
EmilienMyou can also run "recheck" but it will also run check pipeline jobs again, but you don't want that.17:22
AJaegerEmilienM: like I suggested - but let's first jeblair check quickly whether there is a bug, he might have a better idea. I'm sure dhellmann can wait a few minutes...17:22
*** senk has quit IRC17:23
dhellmannyeah, I always like to make sure it's not a symptom of a broader problem. if it's just a hiccup, I can re-approve, but if there's an issue that might affect other folks and this is just an early sign keeping the state of the system as it is may help with debugging.17:23
anteayadhellmann: thank you17:23
*** zul has joined #openstack-infra17:24
pabelangerrcarrillocruz: okay, have region_name working with ansible-role-cloud-launcher.  Tested with creating a security group17:24
pabelangerrcarrillocruz: incoming patch17:24
*** ilyashakhat has joined #openstack-infra17:25
*** _nadya_ has joined #openstack-infra17:26
*** hashar has joined #openstack-infra17:26
*** hashar is now known as hasharAway17:26
openstackgerritGraham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs
*** AnarchyAo has joined #openstack-infra17:26
openstackgerritKen'ichi Ohmichi proposed openstack-infra/bugdaystats: Add "daily" argument to update_stats()
oomichipleia2: hi, I drop on ^^^17:28
slaglepabelanger: so with this latest batch, we ended up with just 8 stuck in building17:29
slaglepabelanger: i was going to try deleting those17:29
slaglepabelanger: will that cause nodepool to try again?17:29
*** kaisers_ has joined #openstack-infra17:29
openstackgerritGraham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs
pabelangerslagle: should be able just to leave it, need to wait 30mins for nodepool to time out17:30
*** sarob has joined #openstack-infra17:30
pabelanger30min more*17:30
*** sarob has quit IRC17:31
slaglepabelanger: yea, was just wondering if i could short circuit that  by deleting them17:31
*** sarob has joined #openstack-infra17:31
jeblairdhellmann, fungi, AJaeger: zuul did not log sending 'Staring gate jobs' to gerrit.  this is mysterious.17:31
pabelangerrcarrillocruz: let me know when you have cycles to talk about
dhellmannjeblair : you're saying that the message that appears on the review isn't in the log file for zuul?17:32
dhellmannor at least a record of having left that message17:32
pabelangerrcarrillocruz: maybe it is better just do use a region_name setting, and drop the with_items17:32
*** zul has quit IRC17:32
jeblairdhellmann: exactly.  though zuul separately recorded having received a comment-added event around that time.17:32
dhellmannit sounds like there's a leak in a tube somewhere17:33
dhellmannjeblair : it sounds like there's not much info to debug with, and that this is probably not a wide-spread problem. is it ok to re-approve that patch?17:33
jeblairdhellmann: yep17:34
*** dizquierdo has quit IRC17:34
*** kaisers_ has quit IRC17:34
jeblairdhellmann: oh, let me enqueue that for you17:34
jeblairit will need a recheck otherwise17:34
dhellmannoh, so just removing the approval and reapplying it isn't enough?17:35
jeblairdhellmann: (zuul did see the workflow +1 but did not act on it because there was no verified+1 from jenkins -- because mystery zuul removed it)17:35
fungi[2016-08-25 16:55:49,093 +0000] 521da478 jenkins a/3 gate jobs.17:36
jeblairdhellmann: should be in now17:36
fungithat's from the gerrit sshd_log17:36
jeblairfungi: i was just about to go look for that log line17:36
*** Kiall has joined #openstack-infra17:36
*** dtantsur is now known as dtantsur|afk17:36
jeblairfungi: i don't suppose we can trace that back to an originating ip address?17:36
*** sambetts is now known as sambetts|afk17:36
fungijeblair: checking17:36
dhellmannjeblair : thank you17:37
*** sputnik13 has joined #openstack-infra17:37
*** degorenko is now known as _degorenko|afk17:37
*** ddieterly is now known as ddieterly[away]17:37
jeblairfungi: [2016-08-25 16:06:35,011 +0000] 521da478 jenkins a/3 LOGIN FROM 2001:4800:7815:101:3bc3:d7f6:ff04:e07f17:38
fungijeblair: according to show-connections... 521da478 16:06:34 00:00:00   a/3             2001:4800:7815:101:3bc3:d7f6:ff04:e07f17:38
fungijeblair: so yes, confirmed two ways17:38
jeblairreverse dns calls that zuul.openstack.org17:38
jeblairforward dns does not17:38
jeblairnor is that thi ip address on the zuul server.17:38
fungii wonder if someone rebooted our old zuul server we shut down and never deleted17:39
jeblairso i think we have a rogue wandering around here17:39
anteayadhellmann: so thanks for reporting17:39
fungiprecise server, up 135 days17:39
dhellmannoh, wow17:39
fungii just now did a `sudo service zuul stop` on it17:40
anteayadid puppet start it up?17:40
fungii'm checking syslog to find out17:40
* dhellmann almost refrains from making ghost in the machine references17:40
fungipuppet is definitely still active on that server17:40
anteayadhellmann: thanks for your almost restraint17:40
openstackgerritGraham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs
dhellmannanteaya :-)17:40
fungicourtesy of our ansible inventory considering anything in our openstack tenants a viable target17:40
jeblairso i "started" it17:41
*** ilyashakhat has quit IRC17:41
anteayaha ha ha17:41
jeblairwhen i told ansible to start zuul17:41
anteayait did what you asked17:41
anteayayou just didn't specify how many17:41
*** Jeffrey4l__ has quit IRC17:41
fungianyone mind if i poweroff this instance, take a snapshot and then delete it?17:41
*** StevenK has quit IRC17:41
jeblairwhich, i did to avoid having two gates fighting each other.17:41
anteayaI do not mind17:41
jeblairfungi: how about just deleting it?17:41
fungijeblair: i'm fine with that if we don't think anybody needs any old files from it17:42
*** StevenK has joined #openstack-infra17:42
*** tosky has quit IRC17:42
jeblairi think it's been quite long enough17:42
fungiinfra-root: heads up, i'm deleting the old from before we upgraded to trusty17:42
mordredfungi: ++17:43
*** ilyashakhat has joined #openstack-infra17:43
fungialso the old eavesdrop and the hacked wiki17:44
anteayafungi: ack17:44
jeblairpresumably, this is something that would be addressed by having unique hostnames -- however, i think we would have a nearly symmetrical problem in that case -- we will need to remember to update playbooks to, eg, reference  otherwise we will restart the wrong hosts because of a lingering reference to zuul01.17:44
dasmhmm... is it me, or gerrit has problems?17:44
anteayadasm: we need to tune the apache config, fungi is working on it17:44
mordredjeblair: yah.17:44
anteayaafter he finishes removing one too many zuuls17:45
dasmanteaya: thanks. will pass down the info.17:45
anteayadasm: thank you17:45
mordredjeblair: I think the current thing is clearly deficient. I do not yet have a good idea as to the way to make it less deficient17:45
jeblairturns out you *can* have too many zuuls.17:45
mordredjeblair: sometimes you have all the zuuls17:45
jeblairmordred: yeah, this is maybe where another layer would need to come in (like terraform or etcd or something) that has metadata like "this is the current zuul"?17:46
mordredjeblair: yah17:46
*** eharney_ has joined #openstack-infra17:47
pabelangerrcarrillocruz: mordred: lets try for region_names in cloud-launcher17:47
*** sdague has joined #openstack-infra17:48
fungiwe've still got duplicate ci-backup-rs-ord servers, puppetdb servers (3 if you include puppetdb01), review-dev servers (i think ianw is working on that though), and static servers17:48
jeblairanteaya: do you have a few mins to help me clean this up?17:48
fungii'm pretty comfortable deleting the old static.o.o now, but i don't know the state for the backup and puppetdb servers17:49
anteayajeblair: how can I be of assistance?17:49
jeblairanteaya: these are the changes that may have similarly been affected:
anteayajeblair: and yes, I have time if you tell me what I can do that you would consider helpful17:49
anteayaonly 19 of them17:49
*** eharney has quit IRC17:49
jeblairanteaya: we need to go through them manually and look for the same circumstance -- the last comment from 'jenkins' being 'starting gate jobs'17:49
*** eharney_ is now known as eharney17:50
jeblairanteaya: then i can enqueue those17:50
fungiokay, old static.o.o at has been deleted now too17:50
jeblairanteaya: can you work down from the top, and i'll work up from the bottom?17:50
anteayaso from that list find the ones that have a starting gate jobs comment on them17:50
*** hockeynut has joined #openstack-infra17:50
*** hockeynut has quit IRC17:51
* anteaya notes gerrit is slow :)17:51
*** ilyashakhat has quit IRC17:51
* clarkb returns from proper breakfast17:51
clarkbmordred: what issue was exposed? is this the thing your nodepool change is intended to fix?17:52
jeblairanteaya: let's make the ones that need re-enquing bold, and the ones that don't strikthrough17:52
mordredclarkb: yes. ish.17:52
mordredclarkb: can you look at this:
mordredclarkb: and tell me if you agree with the english?17:52
anteayajeblair: very good17:53
clarkbmordred: I agree with your english17:53
anteayajeblair: to make sure I'm doing this right, this patch:
mordredclarkb: ok. good17:53
*** sandanar has quit IRC17:53
anteayajeblair: does not have the required comment, correct?17:53
mordredclarkb: I need to make one more change to make that english true17:53
mordredand then we'll be good17:53
*** hockeynut has joined #openstack-infra17:53
* anteaya strikes through
*** rcernin has joined #openstack-infra17:54
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
jeblairanteaya: i will tell you in just a few moments once it loads :)17:54
jeblairanteaya: correct17:54
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add internap to cloud_launcher
jeblairanteaya: (and be sure you have toggle ci set to show you the ci comments)17:55
pabelangermordred: rcarrillocruz: example how regions work in cloud-launcher ^17:55
*** tkelsey has quit IRC17:55
*** pt_15 has quit IRC17:56
anteayajeblair: thank you, and yes toggle ci is set17:56
jeblairanteaya: oh i just remembered one more thing i'll need, sorry17:57
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections
fungiclarkb: ^ a first stab17:57
jeblairanteaya: can you add the project name for each change to the etherpad?17:57
jeblairanteaya: well, at least the bold ones17:57
clarkbthe cinder python jobs are the ones OOMing right? I am skimming the ansible ssh connection errors and cinder python jobs show up17:57
clarkbgonna guess related to OOM if that is the test that was mentioned way back ago in scrollback17:58
*** nwkarsten has quit IRC17:58
fungiclarkb: gate-cinder-python34-db17:58
fungion trusty, obviously17:58
funginot sure if others are as well17:58
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
fungibut i saw that one pop this morning in the gate17:59
clarkbI am seeing the python35 job on xenial error with the ssh errors17:59
fungiahh, py35 is nonvoting for cinder still17:59
clarkbI am going to attribute that to the OOMing17:59
AJaegerand also gate-cross-cinder-python27-db-ubuntu-xenial17:59
*** jcoufal has joined #openstack-infra17:59
AJaegerthat is run on requirements changes17:59
clarkbpabelanger: so maybe we do want working cgroups17:59
fungiokay, so 2.7 (in some variant) is seeing it as well17:59
clarkbpabelanger: to constrain the tests just enough that ssh can continue working17:59
*** david-lyle has quit IRC18:00
* clarkb skims the list to see if there are any other jobs causing problems18:00
pabelangerclarkb: just readying backscroll18:00
AJaegerfungi, I only remember cinder-python27-db-ubuntu-xenial - not seeing 34 myself.18:01
AJaegermight happen less often on 3.418:01
rcarrillocruzSorry, away from home now18:01
*** dstufft has quit IRC18:01
pabelangerclarkb: however, we should be using cgroups in xenial. Last patch was to add systemd support18:01
*** dstufft has joined #openstack-infra18:01
anteayajeblair: project name, ack, will start now and will go back once finished18:01
pabelangerslagle: should be online now18:02
*** senk has joined #openstack-infra18:02
fungiAJaeger: i linked the logs for one in here earlier. finding it again18:02
*** ihrachys has quit IRC18:02
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: added first block of code, tox still works
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: added ironic_build_deploy_ramdisk if statements
*** rhallisey has quit IRC18:03
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: works without %% agent
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: agent condition working
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: added code for BUILD_TIMEOUT=${DEVSTACK_GATE_TEMPEST_BAREMETAL_BUILD_TIMEOUT:-600}
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: ironic deploy driver set
openstackgerritVarun Gadiraju proposed openstack-infra/project-config: fixed weird equals sign inclusion
*** sdake has quit IRC18:03
clarkbpabelanger: ok maybe its not applying the memory limits to the test processes in a way that protects the ssh process18:03
clarkbprometheanfire: that could happen because they all share the same user maybe18:03
anteayajeblair: I'll go back and add project names now18:03
*** gyee has quit IRC18:03
clarkb this is the list off of zl01's current launcher-debug log18:03
clarkbmost of it is cinder18:04
*** david-lyle has joined #openstack-infra18:04
anteayajeblair: on line two the patch hasn't been seen by zuul at all, but it hadn't gotten a comment about gate jobs, so i struckthrough it18:04
anteayamany of the patches I saw had merged18:04
clarkbwe have ovn and neutron and fuel jobs too. Would not surprise me if those three just happen to break networkign on the host for some reason18:04
pabelangerclarkb: yes, that is possible. I didn't actually test cgroups on xenial18:04
*** sdake has joined #openstack-infra18:04
prometheanfireclarkb: mistab?18:04
AJaegerfungi, I believe you - and glad for the info. I was concentrating more on the 27 ones but looking at clarkb's list, it seems spread evenly...18:05
clarkbprometheanfire: yes meant to get pabelanger18:05
*** tqtran has joined #openstack-infra18:05
jeblairanteaya: i think i won't dig into that one.18:05
anteayajeblair: very good18:06
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Remove 'running' as a public method from builder
pabelangerproject-config-core: when you have cycles, if you could help land 360673. We are decreasing capacity in tripleo-test-cloud-rh1 until we schedule a nodepool restart18:06
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Simplify builder start/stop methods
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Remove unecessecary builder scheduler thread
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Rename BuilderScheduler to NodePoolBuilder
*** jkilpatr has quit IRC18:06
clarkbfungi: did you forget to git add the tuning file?18:06
*** yamahata has quit IRC18:06
*** tqtran_ has joined #openstack-infra18:07
*** vgadiraj has joined #openstack-infra18:07
mordredclarkb: if you have to ask to see the tuning file, you're not ready for the tuning file18:07
pabelangerclarkb: still trying to catch up to the issue you are debugging. Can you summarize please?18:07
vgadirajHiwas multitasking, messed up, and a bunch of local commits on my machine got pushed to gerrit as individual commit18:07
vgadirajsorry all18:07
clarkbpabelanger: which one? the continuation of the ansible rc 3? that appears to be related in part to cinder python tests OOMing which breaks ssh18:07
fungiclarkb: indeed i did. just a sec18:08
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections
fungiclarkb: ^ now with more git add18:08
*** ociuhandu has quit IRC18:09
anteayajeblair: I think I am done18:09
jeblairanteaya: thanks!18:09
anteayajeblair: is there anything else I can do to help18:09
anteayajeblair: thanks for asking me18:09
fungiclarkb: though i just spotted a typo. another patchset incoming18:10
*** tqtran has quit IRC18:10
jeblairanteaya: i think that's it, i just need to transform these into commands now and run them18:10
anteayajeblair: very good18:10
anteayathank you18:10
clarkbfungi: ok I was really confused and checking the current master wondering if the file existed already :)18:10
*** shashank_hegde has joined #openstack-infra18:10
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections
fungiclarkb: ^ there we go18:10
*** david-lyle has quit IRC18:11
clarkbpabelanger: I don't think fixing that is hugely urgent. The scope and frequency of it is small and localized to cinder mostly18:11
clarkbpabelanger: just something to keep in mind if we want to make ssh reliability better18:12
pabelangerclarkb: okay, let me finish getting internap working, and I then see how to debug OOMs18:12
fungiclarkb: we'll need to manually reload apache with that version... i wasn't sure how to subscribe an apache reload to a file being added in the calling class there. we could get fancier with puppeting it through into the gerrit module as an optional/templated thing if you think that's warranted18:12
*** david-lyle has joined #openstack-infra18:12
clarkbfungi: I think those turning parameters require a full restart anyways18:12
fungiclarkb: oh, yep you're right18:13
fungiwe likely also want to consider adding the stats module like we're using on etherpad (or have we done that already? i didn't even look)18:13
pabelangerrcarrillocruz: crinkle: looks like infracloud is on the alternate list for Barcelona18:13
*** hockeynut has quit IRC18:13
clarkbI don't think we have added the stats module anywhere else18:13
*** zul has joined #openstack-infra18:13
fungiyeah, without adding it in, we're sort of blind to the results of adjusting these values18:14
clarkbwant to add that in a parent change and we can do both together?18:14
* clarkb reviews the tuning params now18:14
*** david-lyle_ has joined #openstack-infra18:15
*** julim_ has quit IRC18:15
fungii wonder if we should also tune the jvm memory allocation back down a bit to avoid additional apache threads increasing memory pressure?18:15
*** david-lyle_ has quit IRC18:15
fungiclarkb: i need to step away for about 20 minutes to help chris pick up her car from the mechanic, and then i can work on the server-status module change18:16
clarkbtop says apache is using so little memory18:16
*** WJMUqqJZRD has quit IRC18:16
fungiyeah, we likely don't need any tweaks to the jvm for this then18:16
jeblairfungi, clarkb: remember that gerrit has tuning parameters too regarding how many requests it services at once18:16
clarkband we have half a gig of free memory apache can grow into with about 10GB of cache/buffers18:16
clarkbjeblair: I see 100 threads for sshd and a maxqueued value of 200 for gerrit18:17
clarkbsshd doesn't go through apache so I think we can mostly ignore that18:18
* clarkb reads docs on maxqueued for httpd18:18
jeblairthere's something about the fact that ssh+http access methods actually sharing workers though -- i put comments in the gerrit module about it18:18
clarkbok will look there too18:19
*** jkilpatr has joined #openstack-infra18:19
openstackgerritMerged openstack-infra/devstack-gate: Change default for FORCE_CONFIG_DRIVE to False
jeblairis anyone aware of problems with the deb-* repos on github?18:20
openstackgerritChris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available
jeblairit looks like many (all?) of them lack team acls18:20
jeblairso gerrit is unable to push there18:20
jeblairwhich is why i'm seeing 0f0ac405 waiting .... 18:11:24.780      (retry 2819) [681463c0] push
jeblairin the gerrit queue18:21
*** tphummel has quit IRC18:21
*** NomNKDJIvx has joined #openstack-infra18:21
*** senk has quit IRC18:21
clarkblooks like there is an httpd.minthreads and httpd.maxthreads. Defaults for both are 5 and 25 respectively and we don't set either18:21
*** tphummel has joined #openstack-infra18:21
jeblairanyone who cares about github want to fix that, or should we just turn it off? :)18:21
clarkbjeblair: I want to say thats a problem with the github api throttling18:22
*** dizquierdo has joined #openstack-infra18:22
clarkbjeblair: users only get like 5k requests per hour and we easily cross that threshold when creating that many projects at once18:22
jeblairclarkb: yeah, likely we were throttled when they were all created, so now they are broken18:22
AJaegerwhen I review the "Normalize projects.yaml" changes, I always check that github is uptodate18:22
*** esikachev has joined #openstack-infra18:22
AJaegerSeems that this check was not done with ;(18:23
jeblairAJaeger: i'm not sure that would have helped18:23
jeblairAJaeger: i think the issue is that the projects were created in github, but adding acls failed18:23
jeblairAJaeger: and i don't think jeepyb is smart enough to detect that and correct it in a subsequent run18:24
AJaegerjeblair: What I mean: I only approve those changes when there's content - and if not, I speak up here.18:24
clarkbjeblair: looks like the relationship between httpd thraeds and sshd threads is that we need to have more db threads than the max of sshd and httpd18:24
*** e0ne has quit IRC18:24
AJaegerboth yesterday and today I had to ping yolanda to fix new repos...18:24
clarkbfungi: I think we should also consider increasing the gerrit parameters, maybe even start there18:25
AJaegerjeblair: 279 repos were created at that day ;(18:25
clarkbfungi: increase httpd.maxthreads to ~100, minthreads to ~20, then increase db threads to whatever the math comes out to18:25
clarkbthen keep a really close eye on memory use18:26
clarkbjeblair: does gertty use a persistent http connection? I wonder if as the number of gertty users goes up the free slots fall drastically18:26
apuimedoAJaeger: I'd need to have myself added to,members kuryr-libnetwork-core and also to kuryr-kubernetes-core18:27
apuimedootherwise I can't manage to add other people18:27
apuimedonor to add kuryr-core as a subgroup of the other two groups18:27
pabelangerjeblair: clarkb: fungi: When you have time to spare, 360622 adds internap-mtl01 to puppetmaster.o.o.18:27
apuimedothese two groups and the acls are very recent18:27
*** tqtran_ has quit IRC18:28
AJaegerapuimedo: I cannot help you with those. Please give the link to the review as reference and somebody else with admin nights will most likely pick it up.18:28
*** tqtran has joined #openstack-infra18:28
apuimedoAJaeger: do you know who?18:28
AJaegerIt's nearly night time - I meant admin rights ;918:28
*** esikachev has quit IRC18:28
clarkbya I think httpd.maxthreads to 100 to match sshd threads amkes sense. Then we can support roughly the same number of incoming requests over either protocol18:28
apuimedothis is the other group18:28
AJaegerapuimedo: just ask here in the round, the admins are around. And please link to the reviews!18:29
anteayaAJaeger: admin nights, I don't want any of them18:29
*** nwkarsten has joined #openstack-infra18:29
*** pvaneck has joined #openstack-infra18:29
jeblairclarkb: there is a suggestion in that using the session object gets you a persistent connection.18:29
clarkbjeblair: zaro I think you all have way more experience tuning gerrits though. What do you think?18:29
anteayaAJaeger: but admin knights might be useful18:29
jeblairclarkb: so maybeso.18:29
apuimedothese are the reviews18:29
jeblairclarkb: though those connections will still timeout quickly18:29
AJaegeranteaya: yeah, we should ask for admin knights instead of infra admins ;)18:29
anteayawe could see what kind of response we get18:30
*** electrofelix has quit IRC18:30
*** roxanaghe has joined #openstack-infra18:31
anteayanew high for me: Queue lengths: 6783 events, 2790 results.  it is a gate reset, just high numbers18:31
zaroclarkb: yeah, that's a good idea.18:32
clarkbjeblair: and the trade off there is when gertty needs to make a bunch of requests it uses a single connection rather than many so probably better overall18:32
jeblairi can not believe we broke a url across lines in puppet just becaues the line was too long18:32
AJaegeranteaya: I've seen over 11k some time ago ;(18:32
jeblairplease never ever ever ever never ever do that18:32
*** rhallisey has joined #openstack-infra18:33
anteayaAJaeger: ah18:33
*** Sukhdev has joined #openstack-infra18:33
AJaegeranteaya: could you review to help with tripleo cloud, please?18:33
openstackgerritJames E. Blair proposed openstack-infra/system-config: Fix URL in gerrit tuning comments
AJaegeranteaya: but 6k is already high and not a normal value18:33
anteayaAJaeger: once gerrit lets me18:34
anteayalots moved on in the gate, so yay!18:34
openstackgerritJames E. Blair proposed openstack-infra/system-config: Fix URL in gerrit tuning comments
*** zul has quit IRC18:34
zaroclarkb: and maybe set the others to match, acceptorThreads and maxQueued?18:34
clarkbzaro: maxqueued is already 200 I think we can leave it that way18:34
jeblairdue to the ssh slowness, i'm more inclined to say the issue is closer to gerrit than apache.18:34
*** tqtran has quit IRC18:35
clarkbjeblair: ya I think you may be right that we start seeing the apache issues because gerrit is not keeping up forcing apache to queue18:35
jeblairclarkb: did you read!topic/repo-discuss/uRSRwYUpWnE18:35
clarkbjeblair: not yet just the comments in the puppet18:35
jeblairsshd.threads - number of Git fetch or push threads for the entire18:35
jeblairserver. These threads are shared between the SSHD and HTTPD.18:35
*** roxanaghe has quit IRC18:35
jeblairthat's what i was remembering18:35
clarkbjeblair: reading current docs I don't knwo if thats true anymore18:35
openstackgerritSean Dague proposed openstack-infra/project-config: Revert "set check-osc-job to voting"
clarkbjeblair: the httpd section has its own min and max threads. 5 and 25 by default18:35
jeblairwe should probably update the comments then :)18:36
jeblaironce we figure out how things work again18:36
sdaguejeblair / clarkb ok, I looked into the client pipeline issue, and proposed the revert
AJaegerstevemar: please see ^18:37
sdaguebecause I think that test architecture for collision detection is going to grow forever18:37
*** salv-orl_ has quit IRC18:37
sdagueAJaeger: yeh, I've been talking with stevemar, I did ask him to comment on it18:37
AJaegersdague: thanks18:37
clarkbjeblair: specifically see compared to
clarkbjeblair: granted this assumes the docs are correct :)18:38
*** ddieterly[away] has quit IRC18:38
jeblairsdague: okay, well, that comment i said earlier now applies :)18:38
sdaguejeblair: it does18:38
clarkbbut it sounds like our intent was to have 100 httpd threads in the past so upping to that value now makes even more sense to me18:38
*** asettle has joined #openstack-infra18:38
*** asettle has quit IRC18:38
*** asettle has joined #openstack-infra18:39
stevemarsdague: ah this only causes an issue cause the gates queues things together18:39
jeblairsdague: what was wrong with just removing it from tripleo?18:39
*** ddieterly has joined #openstack-infra18:39
sdaguejeblair: because the intent of the job is to do that with everything with osc plugin18:39
sdagueso we can start saying "you can do this, unless you also do X"18:39
sdaguebut that gets pretty complicated18:39
sdagueand doesn't actually give them the coverage they are looking for18:40
sdaguestevemar: right, that's the point of the gate model :)18:40
zarosuggestions on those settings18:40
jeblairsdague: oh, so this is not testing that these things work together, they're just testing that they don't have colliding commands?18:41
sdaguejeblair: yes18:41
*** marcusvrn_ has joined #openstack-infra18:41
*** dizquierdo has quit IRC18:41
stevemarsdague: i think i'd prefer de-coupling (and risking the chance of a wedge), rather than making it check and non-voting (no one was reading why non-voting jobs fail)18:41
jeblairi agree that throwing a shared queue at that problem may be heavyweight18:41
fungiokay, i'm back now and catching up18:42
stevemarjeblair: i think they only share a queue because the jobs has the same name, i didn't know that :\18:42
sdaguestevemar: but that's the point18:42
jeblairstevemar: well, the job hase the same name because it's the same job :)18:43
*** kaisers_ has joined #openstack-infra18:43
sdagueI think there is enough confusion here about what is wanted we should revert18:43
jeblairstevemar: so it is correct that they share a queue18:43
sdagueand approach again next cycle18:43
*** yamahata has joined #openstack-infra18:44
jeblairstevemar: (the idea is that if a job uses both projects A and B, then those projects both influence the outcome of the job, so they should always be tested together and should share a queue)18:44
jeblairthis is quite likely exactly the case here18:44
sdaguejeblair: it's definitely the case here, but as you said, the mechanism being used is super heavy weight for this18:45
jeblairi think sdague is just arguing that doesn't scale to all projects that might possibly have an osc plugin, so some other approach might be warranted18:45
stevemarjeblair: so it sounds like its correct :P18:45
sdaguejeblair: right, exactly18:45
AJaegeryeah, having a shared queue avoids the race where two projects both define the same command and merge at the same time...18:45
sdaguethis is solving a social problem with a massive CI solution18:45
sdaguethat has a lot of negative fallout18:45
AJaegersdague: yeah, the cost is high.18:45
*** tqtran has joined #openstack-infra18:45
openstackgerritChris Krelle proposed openstack/diskimage-builder: WIP: A hardware burn-in element.
*** hasharAway is now known as hashar18:46
sdagueso, lets roll it back now. Make it one of the osc discussion points in barcelona, try to get a better solution there18:46
stevemarand i guess we can't make the check job voting if there is no gate job?18:46
krotscheckinfra-core: I'm going to cease my activities supporting the various javascript efforts here in infra, in preparation of joining a new non-openstack employer. As it's become pretty clear that after pushing this effort for two years, there's little-to-no community support for advanced UI development, I'm saving myself the continued frustration and am throwing in the towel. I wish all of you the best of luck, it's been a blast.18:46
AJaegerstevemar: you could - but that would allow two cchanges to merge with same commands in it.18:47
sdaguestevemar: and wedge18:47
sdagueand block everyone until you could chase down fixes in one of those projects18:47
AJaegerstevemar: imagine both test in the check and then both go through gate - and both go in, and then nobody will be able to merge change until this is reverted18:47
AJaegerwhat sdague says ;)18:47
sdagueanyway, the week before freeze is probably not a time to figure out a novel solution for osc here18:47
marcusvrn_asselin: ping18:48
jeblairkrotscheck: thank you for the excellent work you've done, and for letting us know.  i particularly enjoyed working with you on the mirror project.  :)18:48
sdagueand the world was ok before 7 weeks ago18:48
sdagueso reverting to that state seems not terrible18:48
sdagueand make this part of barcelona discussions18:48
* AJaeger hugs krotscheck and wishes him all the best. Sad to see you move on18:48
stevemarit gave me piece of mind knowing they couldn't stomp18:48
sdaguestevemar: at a huge cost18:48
stevemaryeah, alright18:49
stevemari will write to the ML and let the plugins know18:49
*** YqwHkAMaYX has joined #openstack-infra18:49
stevemarhopefully they read why things are failing and actually check18:49
jeblairis the problem workable if we drop tripleo from that?18:49
pabelangerkrotscheck: yes, thanks for the work you've done. Like jeblair, the mirror projects was a fantastic base18:49
sdaguejeblair: it slices it down a bit, it's still gluing quite a bit together18:49
fungiclarkb: jeblair: okay, caught up on the scrollback now and i agree we should look at gerrit performance tuning before jumping to apache tuning18:50
sdaguebut, I really think that someone is going to readd tripleoclient later because they see it missing18:50
sdagueand add some other new client18:50
fungii'm going to see if i can figure out anything interesting from javamelody too18:50
melwitthi, gerrit is extra slow right now and returning 502s sometimes. just reporting it in case it hasn't been reported yet18:50
sdaguewhich joins in 6 other projects18:50
*** krotscheck has left #openstack-infra18:50
jeblairmelwitt: it has but we keep getting distracted18:50
anteayamelwitt: thanks, we are working on a fix18:51
pabelangerAJaeger: So, I'm think maybe we should revert back to 20 min-ready for both ubuntu-trusty and ubuntu-xenail? Now that we have the capacity again18:51
*** salv-orlando has joined #openstack-infra18:51
anteayamelwitt: right now we are looking at options to tune apache and gerrit18:51
fungimelwitt: usage seems to have increased drastically this week18:51
*** jcoufal has quit IRC18:51
melwittokay, thanks all18:51
anteayamelwitt: it is being told it can only handle so many connections, and we are trying to figure out how to tell it it can handle more than that18:52
anteayamelwitt: thanks for reporting18:52
*** ilyashakhat has joined #openstack-infra18:52
AJaegerpabelanger: looking at the constant demand we have for these, it seems reasanable.18:52
jeblairsdague: i'm also wondering how much of an issue this is in practice -- i mean, the report was basically clarkb looking at the status page and saying 'huh that's weird'18:52
*** sarob has quit IRC18:52
melwittanteaya: a-ha, cool18:52
* dtroyer_zz just caught up on the gate question18:52
sdaguejeblair: my experience is that these things continue to grow until the whole universe is joined together18:53
dtroyer_zzI'm good with dropping that job for now, we do need a better solution to the command problem18:53
anteayadtroyer_zz: hello, welcome, I take it your travel was successful18:53
jeblairsdague: so i'm wondering whether a scaled-down approach would be okay till bcn18:53
fungijeblair: clarkb: worth noting, javamelody reports the gc activity picked up mostly out of nowhere just around 3 hours ago, so another gerrit restart will likely relieve some of this now too18:53
dtroyer_zzanteaya: layover, one more flight18:53
anteayadtroyer_zz: ah, happy layover to you18:53
jeblairfungi: around when i restarted zuuls18:53
sdaguedtroyer_zz: you want to +1 the review?18:54
fungibut if we want to implement some tuning changes, we need a restart anyway, so that seems like it might be well-timed18:54
fungijeblair: right at 16:00 utc18:54
fungiso maybe we creamed it18:54
dtroyer_zzsdague: trying, gerrit not appy as already noted18:54
dtroyer_zzwhy I wanted to mention it here ;)18:55
stevemarsdague: go ahead and revert it, dtroyer_zz and i both know about it18:56
jeblairdtroyer_zz, sdague, stevemar: if we want to get creative, we could consider a new pipeline+gerrit category18:56
stevemaryou're clear in our books18:56
openstackgerritPaul Belanger proposed openstack-infra/project-config: Bump ubuntu-(trusty|xenial) to 20 min-ready
*** ilyashakhat has quit IRC18:57
pabelangerAJaeger: anteaya: If you have a moment to review:^18:57
stevemarjeblair: i don't have the know how to come up with a more clever solution, i thought my osc-check-job was already pretty clever! :P18:57
fungiclarkb: jeblair: so should i hand-patch httpd.maxThreads=100 into the gerrit.config immediately before restarting?18:57
dtroyer_zzjeblair: I would actually prefer a way for plugins to do this outside the gate so if anything does happen again inside it is just a backstop18:57
fungiclarkb: jeblair: and then write up the puppet change for that?18:57
dtroyer_zzthis isn't the sort of thing you should be finding out that late in the development18:57
jeblairdtroyer_zz, stevemar: yeah, i think the problem is that this *is* the correct solution18:57
anteayapabelanger: I'm on board, waiting for gerrit to let me say so18:58
jeblair(lacking a registry like sdague was suggesting)18:58
*** sdague has quit IRC18:58
dtroyer_zzI think the registry is where we are headed.  we already track plugins existance and the list of resources, that's just another thing ans what will drive us to formalize it rather than just keeping a doc18:59
jeblairanyway, i guess just pay attention to it in check for now.  :)18:59
AJaegerpabelanger: +2A18:59
fungijavamelody has total thread counts but doesn't seem to break out httpd and sshd threads into their own counts19:00
anteayafungi: I'm fine with the hand patch and restart followed by the puppet change, I have no comment on the question about whether that is the correct value though19:00
jeblairssh review gerrit show-caches -w 120 --show-threads19:00
jeblairfungi: that may help19:00
fungijeblair: aha, i can grep that, yes19:00
jeblairfungi: well you may want to skip the "-w 120" :)19:00
dtroyer_zzboarding time!19:00
fungijeblair: yep, getting a count now19:01
* dtroyer_zz puts mic gentrly back into the mic stand19:01
fungior at least i will when that command finishes19:01
anteayadtroyer_zz: safe flight19:01
*** Ahharu_ has joined #openstack-infra19:01
Ahharu_Hello, is the mitaka.puppet-openstack.fuel-library.pkgs.ubuntu.neutron_vlan_ha gate working properly?19:01
fungigerrit: show-threads: not found19:01
* fungi reads documentation19:01
jeblairfungi, clarkb: can we move this to incident?19:02
fungijeblair: yep19:02
*** bhunter71 has quit IRC19:02
Ahharu_It is failing all the time I recheck even on a cherry-picked commit19:02
*** ilyashakhat has joined #openstack-infra19:02
AJaegerAhharu_: that's a job executed by the Fuel CI - a third party CI.19:03
pabelangerAJaeger: anteaya: great, thanks19:03
openstackgerritMerged openstack-infra/project-config: Revert "Increase rh1 max-servers to 75"
AJaegerAhharu_: nothing we can help you with - best talk with the fuel folks on their channel19:03
Ahharu_ok, will do :)19:06
*** sarob has joined #openstack-infra19:08
*** david-lyle has quit IRC19:10
*** zul has joined #openstack-infra19:11
*** sarob has quit IRC19:12
*** guilherme has joined #openstack-infra19:12
pabelangerclarkb: okay, added an auto-hold for gate-cross-cinder-python27-db-ubuntu-xenial. So we can check cgroups19:15
pabelangeralso going to clean up our held nodes for ipv6 failures19:15
clarkbpabelanger: ok ty for doing that. Was on my list to do today just haven't gotten to it19:16
clarkbI have rechecked 358105 since our images are mostly up to date in all cloud regions now19:16
*** raunak has quit IRC19:18
*** markusry has quit IRC19:19
*** YqwHkAMaYX has quit IRC19:19
*** markusry has joined #openstack-infra19:19
*** Apoorva has quit IRC19:20
clarkbif ntp looks happy on those I am going to self approve (it already ahs the +2s) this should make our d-g jobs run a little quicker19:20
*** e0ne has joined #openstack-infra19:21
*** JasonF has joined #openstack-infra19:21
*** salv-orlando has quit IRC19:21
openstackgerritMerged openstack-infra/project-config: Bump ubuntu-(trusty|xenial) to 20 min-ready
*** aeng has joined #openstack-infra19:23
*** e0ne has quit IRC19:23
*** sarob has joined #openstack-infra19:24
*** rhallisey has quit IRC19:25
*** rhallisey has joined #openstack-infra19:26
*** sdague has joined #openstack-infra19:26
clarkbone thing I have noticed is that we are not uploading images to ovh bhs1 reliably and an image listing there shows we have leaked a bunch of images19:27
clarkbI am going to clear the leaked images out so we can see if that is related to image upload reliability19:27
clarkbmordred: ^ that may be something shade/glance need to sort out though19:28
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Increase httpd threads for review.o.o
*** tphummel has quit IRC19:33
*** Na3iL has quit IRC19:34
openstackgerritMonty Taylor proposed openstack/os-client-config: Clean up vendor support list
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add timestamp.txt for rsync mirrors
*** edtubill has quit IRC19:35
pabelangercloudnull: odyssey4me: adds timestamp.txt to rsync repos first ^19:35
mordredclarkb: oh goodie!19:35
pabelangerclarkb: maybe issues with qcow2?19:35
clarkbpabelanger: maybe? like if they have to do a conversion on the backend that could be less reliable19:36
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Increase httpd threads for review.o.o
clarkbalso its really hard to filter this list out and be careful not to delete any images we are using at the same time19:36
pabelangerclarkb: Right, my thinking too19:36
* clarkb tries to be extra careful19:37
clarkb(the "building images don't have uuids in nodepool db yet so trying to make sure I don't accidentally delete any of those19:38
*** sdague has quit IRC19:39
*** sdague has joined #openstack-infra19:39
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add timestamp.txt for rsync mirrors
sdagueclarkb: ok, given that you were the first to notice - - dtroyer_zz is +119:40
*** zul has quit IRC19:41
*** cody-somerville has joined #openstack-infra19:41
*** cody-somerville has quit IRC19:41
*** cody-somerville has joined #openstack-infra19:41
clarkbI estimate we have leaked 290 images in ovh-bhs119:41
*** njohnston has joined #openstack-infra19:45
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add timestamp.txt to debian-openstack repo
*** njohnston has left #openstack-infra19:46
pabelangerclarkb: Wow19:46
-openstackstatus- NOTICE: The Gerrit service on is restarting to implement some performance tuning adjustments, and should return to working order momentarily.19:46
pabelangerzigo: ^ packaging job updates to add timestamp.txt to the repo19:46
pabelangerzigo: some extra info to know when we last changed it19:46
openstackgerritSean Dague proposed openstack-infra/devstack-gate: don't start horizon in grenade
zigoWell, we last changed it *today*, and this will be turth every day! :)19:47
clarkbpabelanger: mordred I have started the deletes (doing them serially to avoid api limits so not very quick)19:47
zigoLooking at the CR.19:47
pabelangerzigo: right, mostly helpful if we have have a problem and it becomes stale for some reason19:47
clarkbwe should probably do a check of the other clouds for similar situations19:47
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
clarkb(I can do that but likely not until much later as I have to run a bunch of errands in about an hour19:47
zigopabelanger: What I also plan on implementing once everything is built, is a tempest validation job which we can use to validate the whole repo.19:48
*** Ahharu_ has quit IRC19:48
cloudnullpabelanger: awesome!19:48
mordredclarkb: I'm almost done with the FIP follow up patch19:48
clarkbmordred: cool19:48
*** jheroux has quit IRC19:48
mordredclarkb: I'll start looking at images after that19:48
pabelangerzigo: Ya, that would be good19:49
zigopabelanger: BASE=/afs/ <--- Wasn't it debian-openstack ?19:49
clarkbtoday is remove cast from kids leg day so its a "fun" errand19:49
zigopabelanger: FYI, I have that already on my Jenkins.19:49
pabelangerzigo: it is deb-openstack in AFS, we have char limts19:49
*** tphummel has joined #openstack-infra19:50
mordredzigo: fwiw, if you "apt-get install openafs-client" and then "ls /afs/" you can verify it yourself locally19:50
fungizigo: also, the openafs-client package in debian works great. i use it to check stuff from home, no authentication needed for the read-only volumes
mordredfungi: jinx19:51
* fungi goes back to breaking things19:51
*** jheroux has joined #openstack-infra19:51
clarkbmordred: is a list of the image uuids I am deleting19:51
pabelangerzigo: so, wouldn't it be neat is in the check / gate we somehow merged the package being build with the packages already built, then ran tempest.19:51
clarkbif you need them to compare to logs19:52
pabelangerzigo: so we validate everything before pushing in the post pipeline19:52
zigopabelanger: Unfortunately, the full tempest test takes currently more than 1 hour, and we can't afford to wait that long.19:53
clarkbsdague: stevemar will run that job non voting against mitaka. The old code was excluding mitaka. Do we need to edit the revert to continue excluding mitaka?19:53
clarkbsdague: stevemar in general though I Think we should do this19:53
zigopabelanger: I'd have to think about ways to make it faster.19:53
pabelangerclarkb: do we have the ability to resize puppetmaster.o.o? I haven't check in rackspace19:54
*** raunak has joined #openstack-infra19:54
clarkbpabelanger: I don't think so19:54
*** zul has joined #openstack-infra19:54
clarkbpabelanger: not on the performance flavors iirc19:54
pabelangerzigo: Ya, some sort of validation before doing vos release would be neat.19:54
sdagueclarkb: I suspect excluding mitaka was because they couldn't vote with it19:55
clarkbsdague: gotcha19:55
pabelangerzigo: I'm sure we'll talk more about it in Barcelona19:55
sdaguewe can always clean up that separately19:55
zigopabelanger: I already plan on adding piuparts tests.19:55
clarkband since this is non voting its fine19:55
clarkbwill approve19:55
sdagueclarkb: coolio19:55
*** sarob has quit IRC19:55
*** Apoorva has joined #openstack-infra19:56
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Update undercloud flavor stats
clarkbmordred: in the past I thought that leakage was happening beacuse the glance image upload is a two step process of make image, then upload and if the upload fails we aren't deleting the image. But I thought shade addressed that as I recall reviewing changes to properly delete the newly created image if an upload fails19:58
mordredyah. it's supposed to do that for sure19:58
mordredhowever ...19:58
mordredI betcha if a proxy throws a 500 timeout like HP used to ... and the image is still uploading19:59
apuimedoanybody with gerrit adming rights?19:59
mordredand/or converting19:59
*** jheroux has quit IRC19:59
mordredand then you try to delete and the cloud says "nope, image isn't active"19:59
mordredI mean ... it's a pattern we've seen before19:59
apuimedowe need to populate kuryr-libnetwork-core and kuryr-kubernetes-core20:00
*** JayF has quit IRC20:00
mordredclarkb: we should start naming these patterns20:00
*** jpich has quit IRC20:00
*** JasonF is now known as JayF20:00
openstackgerritSean Dague proposed openstack-infra/devstack-gate: remove old tests
clarkbmordred: we can call that one "cloud is too busy for you user no deletions"20:01
*** sambetts|afk has quit IRC20:02
mordredclarkb: yah. I feel like I should get that tattooed20:02
*** tonytan_brb is now known as tonytan4ever20:02
*** csomerville has joined #openstack-infra20:02
*** sdake has quit IRC20:03
openstackgerritAlex Kavanagh proposed openstack-infra/project-config: Add charm-manila
*** jheroux has joined #openstack-infra20:03
*** sdake has joined #openstack-infra20:04
marcusvrn_Hi folks, I'm configuring a nodepool CI and I got stuck in the following step: I added a keypair into openstack, configured common.yaml, adding "keypair" label, same name as openstack keypair....but noodepool service fails when tries to create a server ( Any ideas?20:04
clarkbapuimedo: have a link to the change that created those groups?20:05
clarkbapuimedo: I can go ahead and get things sorted out but it helps to have the change20:05
*** sambetts has joined #openstack-infra20:05
marcusvrn_detail: Metadata Key name is NONE20:05
apuimedogetting it to you right now20:05
clarkbalso looks like gal is ptl? we will likely add gal to the groups20:05
*** ilyashakhat has quit IRC20:06
*** cody-somerville has quit IRC20:06
* clarkb double checks the governance stuff20:06
anteayamarcusvrn_: so it created the server, and then it deleted the server20:06
apuimedoclarkb: well, it would be very helpful if you add gal and kuryr-core as a subgroup20:06
apuimedoto both of them20:06
anteayamarcusvrn_: do you have a configuration file for nodepool yet?20:06
clarkbapuimedo: the process we use is the add the ptl to the groups then the ptl can elect to manage it as they see fit20:07
*** dprince has quit IRC20:07
anteayamarcusvrn_: if you have a maximum server quota set to 0 that might be what is telling nodepool to delete the server20:07
*** sarob has joined #openstack-infra20:07
*** markusry has quit IRC20:07
apuimedoclarkb: the problem with that is that Friday is non-working in israel, so tomorrow the other cores won't be able to +2 anything20:07
*** markusry has joined #openstack-infra20:07
*** markusry has quit IRC20:08
anteayamarcusvrn_: timestamps are 19 seconds apart so the server wouldn't even have been available yet20:08
clarkbapuimedo: I am not sure I feel comfortable going behind a PTL's back to delegate for them20:08
clarkbapuimedo: this system is set up this way so that infra does not have to be involved beyond handing over control to the ptl20:08
marcusvrn_anteaya: nodepool.yaml ->
marcusvrn_it's set to 220:09
clarkband both those projects live under kuryr according to the governance repo20:09
apuimedoclarkb: I understand. I'll try to nag him on his weekend20:09
openstackgerritMerged openstack-infra/project-config: Revert "set check-osc-job to voting"
*** rhallisey has quit IRC20:09
apuimedoclarkb: right, that's why kuryr-core should be a sub group to both :P20:09
pabelangerclarkb: so we don't setup cgroups for single_use_slaves20:09
clarkbapuimedo: that is for the ptl to decide20:09
clarkbapuimedo: not for infra to decide20:09
clarkbpabelanger: aha! ok we probably want to add it there then20:09
apuimedoclarkb: as I said, I understand20:09
anteayamarcusvrn_: yes, good, is that the first server your nodepool is trying to create?20:09
apuimedodid you add him already?20:10
marcusvrn_anteaya: hmmm...but the keypair need to be added to the server, right? the metadata "key name" should be "jenkins2"20:10
clarkbapuimedo: yes its done20:10
apuimedothank you clarkb! Have a good rest of day20:10
anteayain 19 seconds I doubt nodepool got as far as worrying about keys at all20:10
anteayaI'm not sure what happens in the first 19 seconds of server creation that would tell nodepool to delete that server20:11
anteayabut I don't think that is time for the server to go live at all20:11
*** tqtran has quit IRC20:12
anteayaand yes, keys should be added to the server, but I dont' think keys came into play here20:12
*** sarob has quit IRC20:12
clarkbmarcusvrn_: nodepool doesn't add keys to insatnces like that. It expects them to exist already20:12
clarkbmarcusvrn_: eg your image build process should take care of that not nova20:12
anteayaah thank you, clarkb20:12
openstackgerritAlex Kavanagh proposed openstack-infra/project-config: Add charm-manila
marcusvrn_clarkb: ah, ok. I'll try to generate again the image, maybe keys are not matching20:14
clarkbmarcusvrn_: you can always boot one manually and fiddle with it to see20:14
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add jenkins::cgroups to single_use_slaves
pabelangerclarkb: ^20:14
clarkbmarcusvrn_: running a nova show on the instances then booting one of my own is my typical process to debug20:14
clarkbpabelanger: I am good with that but with the release coming up maybe we should WIP that until after feature freeze? Slight worry that it will start causing jobs to fail that were right up against the memory limit20:15
rajinirRestarted zuul with a new job in jenkins. I see an error job is not registered with Gearman. Have restarted zuul-merger and nodepool as well. Any solution?20:15
clarkbpabelanger: since I know some tests for projects like magnum run deep into swap20:15
pabelangerclarkb: sure, I am fine with that20:15
*** kzaitsev_mb has quit IRC20:15
clarkbrajinir: the jobs are registered by the gearman works you need to make sure whatever is acting as your gearman worker is conencted to gearman and registering jobs20:15
clarkbrajinir: if using jenkins you can double check it is connected and if so make sure it has instances capable of running that job20:16
rajinirclarkb: its running the old jobs, only the new jobs have this problem. Will restarting jenkins help?20:17
marcusvrn_clarkb: nice, I'll do that20:17
*** dtardivel has quit IRC20:17
marcusvrn_clarkb: anteaya thanks!20:17
clarkbrajinir: no don't restart jenkins... just make sure jenkins is connected to gearman and thati t has slaves capable of running the new job. Maybe your new job isn't in jenkins yet? or none of your slave labels match?20:17
rajinirclarkb: good point, will check20:17
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
openstackgerritMonty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip
mordredclarkb: ok ^^ that's the gist of it - I need one more patch to add unittests for the first patch20:19
*** roxanaghe has joined #openstack-infra20:19
mordredclarkb: but the second patch changes the way we get networks, so I thought it would be harder to read combined20:19
clarkbmordred: ok I am going to disappear in a little bit so don't wait around for me unless you think you need to20:19
mordredclarkb: I won't - I wanna see those run in the gate and I want a patch with unittests20:19
* clarkb hopes that they don't have to do xrays again20:20
clarkbconvincing a 14 month old to sit still for xrays is unfun20:20
mordredclarkb: I currently have in my house a 2 month old and a 4 year old (and their parents)20:21
mordredclarkb: if the 4 year old is any indication ... you have some fun awaiting you in 2 years and 10 months20:21
clarkbmordred: yes... I had to figure out health benefits again last week and it basically came down to lets just pay for the better one because they are constantly hurting themselves20:21
*** nmagnezi has quit IRC20:22
apuimedomine already got stitches when he was two20:22
*** kgiusti has left #openstack-infra20:22
apuimedo(3.5yo now)20:22
*** roxanaghe has quit IRC20:24
clarkbapuimedo: my 14 month old has been in a cast for the last 4 weeks after breaking her leg falling off a chair20:24
apuimedoclarkb: I'm a very lucky dad20:24
apuimedomy little kamikaze even rolled down the tiled stairs (16 of them)20:24
apuimedoand not even a scratch20:24
apuimedoI was astounded20:24
openstackgerritKen Giusti proposed openstack-infra/project-config: Run the AMQP 1.0 oslo.messaging functional tests on Python 3.4
fungiclarkb: falling off chairs already, and not even old enough to drink? that's not setting a good precedent20:26
apuimedomore of less like
apuimedobut backwards20:26
*** kzaitsev_mb has joined #openstack-infra20:27
*** tqtran has joined #openstack-infra20:28
*** rhallisey has joined #openstack-infra20:28
*** tphummel has quit IRC20:28
mordredfungi: I did not hear anything about not being old enough to drink20:28
anteayamordred: does the 4 year old know how to swim?20:28
mordredfungi: my assumption is that all liquid at clarkb's house is beer20:28
mordredanteaya: very well20:28
anteayaoh good20:29
bkeromordred: you're forgetting about the gin20:29
mordredbkero: that's an excellent point you make20:29
anteayaI assume the 2 month old can't make it to the pool yet20:29
*** sdake has quit IRC20:29
mordredanteaya: well, the 2 month old can make it near the pool if a larger human carries it there20:30
clarkbapuimedo: ya normally they bounce no idea why this one time it was a problem20:31
anteayabut hopefully the larger human sticks around20:31
clarkbbkero: the botanist is pretty good20:31
apuimedoclarkb: yes. It's usually the smallest hits that break20:31
apuimedoanteaya: the trick is to have a very very cold swimming pool. They usually don't get in it20:32
*** asettle has quit IRC20:32
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: update release dashboard output for newton
anteayaapuimedo: good call20:34
anteayaapuimedo: you also must have been a life guard20:34
*** sdake has joined #openstack-infra20:35
apuimedoanteaya: the best way to bay watch is handing a couple of euros to little kids and have them shout "jellyfish!" every half an hour20:35
apuimedonobody ever gets deep into the water like that20:36
*** asettle has joined #openstack-infra20:36
*** kzaitsev_mb has quit IRC20:36
anteayaI can't even share the stories of what they used to do at the pool to accomplish the same goal20:37
*** zul has quit IRC20:38
*** tphummel has joined #openstack-infra20:39
*** sarob has joined #openstack-infra20:39
*** csomerville has quit IRC20:39
fungii am totally unable to figure out why the trusty apply job failed on
fungievery test seems to have exited 020:40
pabelangerfungi: timeout? took 33mins to run20:40
fungihrm... maybe20:40
fungipabelanger: oh, yep!20:40
fungithere's a 25 minute lag before the job complete line at the end of the log20:41
* fungi is blind20:41
pabelangerYa, we need to restart zuul-launcher still, to pick up the timeout fix in ansible20:41
*** ansmith has quit IRC20:42
*** gyee has joined #openstack-infra20:42
*** jkilpatr has quit IRC20:42
*** sarob has quit IRC20:44
fungiokay, i'm going to disappear to grab some dinner, then i'll come back and continue trying to make sure gerrit still looks sane20:45
*** asettle has quit IRC20:45
anteayaenjoy dinner20:45
*** salv-orlando has joined #openstack-infra20:46
*** sarob has joined #openstack-infra20:47
*** senk has joined #openstack-infra20:47
*** eharney has quit IRC20:49
asselin__marcusvrn_, hi20:50
oomichihi, are there any relationship between nodes with/without some suffix like "-181073"?20:52
oomichieg:  ubuntu-xenial-2-node-rax-ord-387498720:52
oomichiand ubuntu-xenial-2-node-rax-ord-3874987-18107320:53
*** tqtran has quit IRC20:53
openstackgerritDoug Hellmann proposed openstack-infra/irc-meetings: add link to the requirements team agenda
oomichiI saw a little bit for live-migration test failure, and the host was different from expected host.20:54
openstackgerritPaul Belanger proposed openstack-infra/system-config: Clean up tripleo-test-cloud-rh1 os-client-config
*** sarob has quit IRC20:56
*** tqtran has joined #openstack-infra20:57
*** eharney has joined #openstack-infra20:57
openstackgerritJames E. Blair proposed openstack-infra/zuul: Handle a project-template with no jobs
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add an OpenStack test
openstackgerritJames E. Blair proposed openstack-infra/zuul: Pass assigned nodes to the launcher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add a python27 job variant to the OpenStack test
*** tphummel has quit IRC20:57
ianwfungi: yes, working on the review-dev one (from 4 hours ago :)20:59
*** senk has quit IRC21:00
*** csomerville has joined #openstack-infra21:01
*** rfolco_ has quit IRC21:02
*** rfolco_ has joined #openstack-infra21:03
*** rfolco_ has quit IRC21:03
openstackgerritMerged openstack-infra/tripleo-ci: Use delorean's exit code to decide on retry
*** sdake_ has joined #openstack-infra21:05
*** sarob has joined #openstack-infra21:07
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: update release dashboard output for newton
*** sdake has quit IRC21:08
jrollclarkb | convincing a 14 month old to sit still for xrays is unfun <- fiancee is a rad tech, she is thankful for people like you that actually help with that instead of just watching her struggle21:09
*** thorst has quit IRC21:09
*** tphummel has joined #openstack-infra21:10
asselin__dtroyer_zz, any idea when new osc-lib will get released after merges?21:10
jeblairjroll: you're assuming clarkb was relating his own experience and not saying "wow it looks like the rad tech didn't have fun doing that!"  :)21:10
*** thorst has joined #openstack-infra21:12
*** sarob has quit IRC21:12
*** ldnunes has quit IRC21:15
*** thorst has quit IRC21:16
*** salv-orl_ has joined #openstack-infra21:17
*** sdague has quit IRC21:18
*** thiagop has quit IRC21:18
*** hongbin_ has joined #openstack-infra21:18
*** salv-orlando has quit IRC21:19
*** sdake_ has quit IRC21:20
clarkboh no I was on the tabke in a lead syit trying ti hold her down21:20
*** hongbin has quit IRC21:21
pabelangerclarkb: ianw: any chance we can land today? Adding mtl01 region for internap on puppetmaster.o.o21:22
*** sdake has joined #openstack-infra21:22
*** guilherme has quit IRC21:22
*** hongbin_ has quit IRC21:26
anteayait is moments like this I think of
*** david-lyle has joined #openstack-infra21:26
*** hongbin has joined #openstack-infra21:26
ianwwhen i thought my son at a similar age had swallowed loose change (he hadn't) ISTR they stuck him in a jig.  nobody enjoyed that experience.  The dr's helpful advice was "don't let children play with small coins" ... thanks for that21:27
anteayathe dr obviously has no children21:27
*** _ari_ has quit IRC21:27
*** kzaitsev_mb has joined #openstack-infra21:29
*** matt-borland has quit IRC21:30
pleia2my little sister stuck a penny in an outlet once21:31
pleia2zap, emergency room21:31
pleia2kids :)21:31
*** david-lyle has quit IRC21:31
*** eharney has quit IRC21:32
irtermitenephew, fingers, lamp socket... ever overcooked a hot dog on the grill and had it split open?21:32
anteayacan't do this21:32
jeblairyeah, i think irtermite "wins" and the game is over now :)21:33
irtermitewe were playing a game? what did I win?21:35
anteayathe opportunity to select the next topic of conversation21:37
*** jamesdenton has quit IRC21:40
*** dizquierdo has joined #openstack-infra21:40
*** kzaitsev_mb has quit IRC21:41
*** pradk has quit IRC21:41
openstackgerritMerged openstack-infra/system-config: Add mtl01 region to internap on puppetmaster
irtermiteanteaya: ummm puppy dogs and unicorns?21:42
*** rlandy is now known as rlandy|bbl21:43
marcusvrn_asselin__: hi I was talking to clarkb  and anteaya  about nodepool CI. I created again the image used by nodepool, uploaded it to openstack and when nodepool try to setup a server, it fails... here is the console log of the VM that nodepool tried to setup:
*** dimtruck is now known as zz_dimtruck21:44
*** Hal1 has joined #openstack-infra21:44
asselin__marcusvrn_, I have this documented somewhere....21:44
asselin__marcusvrn_, did you follow these instructions?
asselin__marcusvrn_, this part in particular: Ensure the NODEPOOL_SSH_KEY variable is in the environment21:46
marcusvrn_asselin__: yes, that's what I did21:48
asselin__marcusvrn_, I don't think you'll find anything in the VM console log. Instead, if you did this correctly and need to debug, you need to mount your DIB created image and inspect the authorized keys21:48
marcusvrn_yes, I just create an instance with a keypair generated by me to go into the VM and check the authorized keys21:49
asselin__marcusvrn_, you can mount the image has the correct keys by mounting it. Instructions are here:
*** jerryz has quit IRC21:52
*** kzaitsev_mb has joined #openstack-infra21:53
openstackgerritMerged openstack-infra/project-config: Remove Fedora 24 work-around kernel
*** kushal has quit IRC21:54
*** _nadya_ has quit IRC21:54
marcusvrn_asselin__: hmm it shows a different key into authorized_keys...21:55
*** kushal has joined #openstack-infra21:56
asselin__marcusvrn_, I bet it's -infra's might find it here:
*** shardy has quit IRC21:57
*** gouthamr has quit IRC22:00
marcusvrn_asselin__: no, it was my old key...but I grep the whole system to check if I forgot some place to change, before generate the new image...and that key does not exist anymore22:00
*** gouthamr has joined #openstack-infra22:00
*** gouthamr has quit IRC22:00
asselin__marcusvrn_, perhaps saved in an env variable in some shell? try stopping and restarting services22:01
marcusvrn_ok, restarting all services22:02
marcusvrn_asselin__: just a question, is it normal to take an hour to generate a image?22:02
asselin__marcusvrn_, first time yes, subsequent times should be much faster since there's a lot cached22:02
asselin__marcusvrn_,  you can make it faster for debug purposes22:03
openstackgerritMonty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip
openstackgerritMonty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks
*** esberglu has quit IRC22:04
*** gordc has quit IRC22:05
*** ddieterly is now known as ddieterly[away]22:05
*** roxanaghe has joined #openstack-infra22:07
marcusvrn_asselin__: hmm nice!22:08
*** dizquierdo has quit IRC22:08
*** fguillot has quit IRC22:10
*** ddieterly[away] is now known as ddieterly22:10
*** csomerville has quit IRC22:10
openstackgerritMerged openstack-infra/puppet-gerrit: Create index on fresh install
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job
*** roxanagh_ has joined #openstack-infra22:12
*** roxanaghe has quit IRC22:12
*** hashar has quit IRC22:13
*** dizquierdo has joined #openstack-infra22:14
*** kaisers_ has quit IRC22:15
*** chem|off has quit IRC22:16
*** csomerville has joined #openstack-infra22:16
*** roxanagh_ has quit IRC22:16
*** chem|off has joined #openstack-infra22:18
*** dizquierdo has quit IRC22:18
*** tpsilva has quit IRC22:18
*** Goneri has joined #openstack-infra22:22
*** vhosakot has quit IRC22:23
zigoHow much RAM does the jessie flavor have?22:24
*** sdake has quit IRC22:24
*** dizquierdo has joined #openstack-infra22:24
*** gouthamr has joined #openstack-infra22:25
openstackgerritMerged openstack/os-client-config: Clean up vendor support list
*** jheroux has quit IRC22:26
cloudnullzigo: i dont work on infra specifically, but last i recall all of the flavors are 8GB ram, 4cpus, ~80gb disk22:28
cloudnullzigo: this is the flavor used within the osic
zigocloudnull: Ok, then now I understand why one of my packages failed.22:30
zigoI mean, failed to build.22:30
zigo8 GB is not enough RAM to build webkit2gtk ... :(22:31
zigoAt least, not enough if no swap.22:31
zigoFYI, webkit2gtk is needed to build sphinx.22:31
zigo(yes, that's weird, but such are dependencies...)22:32
*** dizquierdo has quit IRC22:32
cloudnullour flavor is 2GB of swap, however other providers may not have that set.22:32
zigoThe package maintainer told me that webkit2gtk needs at least 4500 MB of RAM + same of swap.22:32
cloudnullmake a swap file?22:34
*** salv-orl_ has quit IRC22:34
pabelangerYa, we can add that to the job22:35
zigopabelanger: Then the package will use more time than the timeout, I guess.22:35
pabelangerzigo: have a log?22:36
zigopabelanger: So, I believe just downloading the package from official jessie-backports was the right thing to do...22:36
zigopabelanger: Not anymore, since I cheated and did a download from jessie-backports.22:36
zigoIt was the correct thing to do IMO.22:36
zigoNo need to bother so much for just an indirect dependency for sphinx ...22:37
*** sdake has joined #openstack-infra22:37
zigopabelanger: FYI, I also tried rebuilding on a machine with 4GB of swap, and it failed at the exact same point: when linking hundreds of objects together.22:37
zigoOne day, hopefully, we'll have a Build-Memory field in debian/control ...22:38
zigoAnother beast is pypy.22:38
*** Goneri has quit IRC22:39
zigoIt wouldn't even START building if you have less than 4GB of RAM.22:39
zigoAnyway, moving on ... ;P22:39
*** thorst has joined #openstack-infra22:42
*** Guest1506 is now known as jamesdenton22:45
*** chem|off` has joined #openstack-infra22:46
*** thorst has quit IRC22:47
*** thorst has joined #openstack-infra22:47
*** chem|off has quit IRC22:48
pabelangerclarkb: jeblair: So, I've been going back and forth in my mind about using nodepool.o.o to create images for our control plan servers.  Since today we are just using stock ubuntu images from cloud-images.u.c, I'd love to switch them out to a basic ubuntu-minimal image22:48
*** yamamoto has joined #openstack-infra22:49
pabelangeranother option is creating a periodic job to do it, but we still have to upload them22:49
*** Apoorva has quit IRC22:50
*** kzaitsev_mb has quit IRC22:54
persiaAre you sure you want -minimal?  -standard is usually small enough for most things, and -minimal lacks many things that are usually taken for granted.22:54
*** tqtran has quit IRC22:55
persiaFor Xenial, is standard, whereas minimal is only (and dependencies, in both cases)22:55
*** tqtran has joined #openstack-infra22:55
*** adriant has joined #openstack-infra22:55
pabelangerYa, we use -minimal today for test jobs plus an infra specific element22:55
*** ddieterly has quit IRC22:56
*** thorst has quit IRC22:56
pabelangerI wonder if that minimal seed matches the DIB element -minimal22:56
persiaWow.  I'm surprised (but pleased) that is useful.22:57
cloudnullpabelanger: what would it take to build lxc images using DIB?22:57
*** eset has quit IRC22:57
cloudnulli see there is a docker option22:57
pabelangercloudnull: not sure, I haven't tried before22:57
*** rhallisey has quit IRC22:57
cloudnulli guess i could try and build off of that?22:57
*** chlong has quit IRC22:58
pabelangercloudnull: greghaynes or ianw might be better to answer that22:58
*** Apoorva has joined #openstack-infra22:59
*** hockeynut has joined #openstack-infra22:59
*** eset has joined #openstack-infra22:59
*** tqtran has quit IRC22:59
cloudnull-^- ping greghaynes ianw22:59
*** gouthamr_ has joined #openstack-infra22:59
cloudnulli'm +/- curious .23:00
*** tqtran has joined #openstack-infra23:00
jeblairpabelanger: you may want to include mordred in that conversation as he has been inching us toward being ready for control-plane images for a while, and greghaynes because he's interested in nodepool as a sort of dib scheduler like you are suggesting.23:00
*** hongbin has quit IRC23:01
jeblairpabelanger: if you used nodepool.o.o, we'd have to stick an openstackci cred on a system that only knows openstackjenkins.  i've always been worried about what happens when something there goes wrong (ie, automatically delete entire control plane).23:02
jeblairpabelanger: maybe worth considering running a copy of nodepool on another server just for this purpose.23:02
*** gouthamr has quit IRC23:03
pabelangerjeblair: Ya, I was thinking about that too. Adding a second nodepool server might be in the cards23:03
pabelangerDoesn't have to happen today, but I think we've nailed the DIB process down pretty good now. Would be nice to start expanding them23:04
pabelangerI'll add a topic for our next meeting23:04
cloudnullmy interest in DIB is related to OpenStack-Ansible, we use LXC base images which we get from and while they work great I'd love to be able to pull the image build pipeline into the our OpenStack build pipeline using DIB (if it at all possible)23:04
*** tqtran has quit IRC23:04
cloudnullfrom a million miles away, which is my perspective of DIB only having hacked on it once, it looks like DIB could/should be able to do the needful.23:06
jeblaircloudnull: one thing you might be able to do soonish is create a periodic job to use dib to build an image and stash it on tarballs.o.o, then fetch it from there in your jobs23:06
persiapabelanger: As far as I can tell with a quick browse of the code, DIB will use apt to install the ubuntu-minimal task, which is generated from the minimal seed, so yes.23:07
cloudnulljeblair: ++ i'll look into that .23:07
pabelangercloudnull: Ya, I did that recently with python-tripleoclient. It is experimental right now, but will give you an idea of everything works together.23:08
cloudnulldo you have a review handy i can go steal from ?23:08
cloudnulli can look it up.23:09
cloudnulltoo so no worries.23:09
*** tqtran has joined #openstack-infra23:09
*** hockeynut has quit IRC23:10
pabelangercloudnull: so, it doesn't call disk-image-create directly, python-tripleclient does that, but it shows a good example of setting up zuul-cloner for external elements and preparing the virtualenv23:10
pabelangercloudnull: there are a few other projects doing it today, grep buildimage in job names23:11
cloudnullok. that's great. gives me a starting point .23:11
cloudnullmuch appreciated.23:11
*** chlong has joined #openstack-infra23:12
*** tqtran_ has joined #openstack-infra23:13
*** tqtran has quit IRC23:13
*** cody-somerville has joined #openstack-infra23:14
*** csomerville has quit IRC23:17
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Zuul v3: update branches job keyword
pabelanger#status log now online23:20
openstackstatuspabelanger: finished logging23:20
pabelangertime to update nodepool :)23:21
ianwzaro: relaunching ....23:21
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add internap-mtl01 to nodepool clouds.yaml
*** yamahata has quit IRC23:25
*** chem|off` has quit IRC23:25
*** chem|off` has joined #openstack-infra23:26
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Zuul v3: don't implicitly define job variables
*** xarses has quit IRC23:29
*** rbrndt has quit IRC23:30
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o
*** Hal1 has quit IRC23:31
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Update nodepool zookeeper spec with branch
pabelangermgagne: ^ I believe you said there is 120 nodes in mtl01, right?23:31
pabelangerwe'll need to restart nodepool-builder too23:31
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o
*** sputnik13 has quit IRC23:32
*** chlong has quit IRC23:35
jeblairpabelanger: are we still running with 4 upload workers instead of 16?23:37
*** yamahata has joined #openstack-infra23:37
pabelangerjeblair: no, we changed to 16 a while back.23:38
jeblairpabelanger: oh ok23:39
jeblairsheesh it's slow23:39
jeblairpabelanger: we don't have any critical updates needed in today's image do we?  maybe we should land the nodepool changes now and restart the builder?23:40
*** jianghuaw-m has quit IRC23:41
*** AnarchyAo has quit IRC23:41
pabelangerjeblair: ya, I think we can do the restart tonight. I only know of the ntp change clarkb did, and pretty sure we've updated everywhere now23:41
jeblairpabelanger: +2s on both from me23:42
ianwhappy to babysit if it helps23:42
jeblairianw: thanks, i'm very close to eod23:43
pabelangerianw: sure, trying to launch internap-mtl01, have a few reviews up23:43
jeblair and
jeblairany others?23:44
pabelangerthat should be all for now23:44
jeblairoh, the less urgent
pabelangerYa, a nice to have23:45
ianwpabelanger: what's up with ?  is that a complete log?23:45
pabelangercould not find job23:45
pabelangerthat's what ansible said23:45
jeblairi assumed it was a zuul-ansible snafu and rechecked23:45
pabelangerI don't think I've see that one before23:46
jeblairpabelanger: any idea what that means?23:46
pabelangermaybe async_runner on node died?23:46
pabelangerwould need to dig more into it23:46
jeblairi added topic:internap to those 323:47
pabelangerlooks like ~/.ansible_async was 40423:48
pabelangerfor the job id23:49
pabelangerwill keep an eye on it23:49
openstackgerritMerged openstack-infra/irc-meetings: add link to the requirements team agenda
*** david-lyle has joined #openstack-infra23:52
*** xyang1 has quit IRC23:52
*** rcernin has quit IRC23:53
*** thorst has joined #openstack-infra23:54
*** kzaitsev_mb has joined #openstack-infra23:54
*** zhurong has joined #openstack-infra23:54
*** xarses has joined #openstack-infra23:56
ianwgreat ... now puppet just times out building review-dev.  it's probably lucky we're not trying to rebuild in a crisis situation ....23:57
*** roxanaghe has joined #openstack-infra23:57
*** nwkarsten has quit IRC23:58
openstackgerritMerged openstack-infra/project-config: Add internap-mtl01 to grafana dashboard
*** nwkarsten has joined #openstack-infra23:58
openstackgerritPaul Belanger proposed openstack-infra/devstack-gate: Use local AFS mirror for network_sanity_check
*** woodster_ has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!