Wednesday, 2016-01-20

*** weshay_xchat has quit IRC00:00
*** morazi has quit IRC00:02
*** panda_ has quit IRC00:06
*** panda_ has joined #tripleo00:06
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc  https://review.openstack.org/26202800:21
*** morazi has joined #tripleo00:38
*** eil397 has quit IRC00:52
*** jcoufal has joined #tripleo00:57
*** shivrao has quit IRC01:01
*** dmacpher-afk has quit IRC01:15
*** trozet has joined #tripleo01:29
*** david-lyle has quit IRC01:31
*** alop has quit IRC01:57
dsneddonI'm getting MongoDB errors in CI on a change that didn't touch MongoDB: http://logs.openstack.org/72/269872/2/check-tripleo/gate-tripleo-ci-f22-ceph/4de7701/console.html02:02
*** dmacpher has joined #tripleo02:09
*** sthillma has quit IRC02:09
*** tiswanso has quit IRC02:17
*** tiswanso has joined #tripleo02:17
*** jcoufal has quit IRC02:28
*** penick has quit IRC02:39
*** trozet has quit IRC02:54
*** shivrao has joined #tripleo03:02
*** rlandy has quit IRC03:14
*** yuanying_ has joined #tripleo03:18
*** dmacpher has quit IRC03:18
*** yuanying has quit IRC03:19
*** coolsvap|away is now known as coolsvap03:20
*** yamahata has quit IRC03:20
*** shivrao_ has joined #tripleo03:20
*** shivrao has quit IRC03:23
*** shivrao_ is now known as shivrao03:23
*** dmacpher has joined #tripleo03:31
*** shivrao has quit IRC03:41
*** dmacpher has quit IRC03:43
*** Marga_ has quit IRC03:44
*** thrash is now known as thrash|g0ne03:46
*** Marga_ has joined #tripleo03:47
*** andreaf has quit IRC03:50
*** Marga_ has quit IRC03:51
*** rhallisey has quit IRC03:52
*** andreaf has joined #tripleo03:52
*** dmacpher has joined #tripleo03:55
*** shivrao has joined #tripleo04:01
*** yuanying_ has quit IRC04:05
*** links has joined #tripleo04:06
*** panda_ has quit IRC04:06
*** panda_ has joined #tripleo04:06
*** yuanying_ has joined #tripleo04:07
*** yuanying has joined #tripleo04:20
*** yuanying_ has quit IRC04:22
*** Marga_ has joined #tripleo04:32
*** stendulker has joined #tripleo04:36
*** Marga_ has quit IRC04:36
*** yuanying has quit IRC04:37
*** yuanying has joined #tripleo04:39
*** yuanying has quit IRC04:40
*** mandre has quit IRC04:42
*** mandre has joined #tripleo04:44
*** yuanying has joined #tripleo04:44
*** david-lyle has joined #tripleo04:44
*** dmacpher has quit IRC04:44
*** Marga_ has joined #tripleo04:46
*** yuanying_ has joined #tripleo04:49
*** yuanying has quit IRC04:50
*** tiswanso has quit IRC04:50
*** tzumainn has quit IRC04:51
*** masco has joined #tripleo05:00
*** masco has quit IRC05:00
*** jaosorior has joined #tripleo05:06
openstackgerritgreghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d  https://review.openstack.org/25973405:11
*** dmacpher has joined #tripleo05:21
*** morazi has quit IRC05:21
*** masco has joined #tripleo05:29
*** teju has joined #tripleo05:31
openstackgerritgreghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic  https://review.openstack.org/27001005:39
openstackgerritgreghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic  https://review.openstack.org/27001005:39
*** yamahata has joined #tripleo05:53
*** xinwu has quit IRC05:54
*** xinwu has joined #tripleo05:54
*** xinwu has quit IRC05:55
*** stendulker_ has joined #tripleo05:55
*** anande has joined #tripleo05:56
*** stendulker_ has quit IRC05:56
*** stendulker has quit IRC05:59
*** stendulker has joined #tripleo06:02
*** jtomasek has joined #tripleo06:14
*** regebro has joined #tripleo06:14
*** absubram has joined #tripleo06:25
*** shivrao_ has joined #tripleo06:31
*** shivrao has quit IRC06:33
*** shivrao_ is now known as shivrao06:33
*** stendulker has quit IRC06:42
*** stendulker has joined #tripleo06:44
*** liverpooler has quit IRC06:45
*** aufi has joined #tripleo06:48
*** dmacpher has quit IRC06:54
*** jaosorior has quit IRC06:54
*** shardy has joined #tripleo06:58
*** shivrao has quit IRC06:59
*** rcernin has joined #tripleo07:02
*** spredzy has quit IRC07:03
*** spredzy has joined #tripleo07:03
*** paramite has joined #tripleo07:05
*** xinwu has joined #tripleo07:18
*** aufi has quit IRC07:20
*** aufi has joined #tripleo07:23
*** dsneddon has quit IRC07:42
*** dsneddon has joined #tripleo07:44
*** cwolferh has quit IRC07:47
*** dsneddon has quit IRC07:49
*** 18VAAPK0U has joined #tripleo07:50
*** dsneddon has joined #tripleo07:51
*** dsneddon has quit IRC07:59
*** dsneddon has joined #tripleo07:59
*** dshulyak_ has joined #tripleo08:00
*** jprovazn has joined #tripleo08:02
*** panda_ has quit IRC08:06
*** fgimenez has joined #tripleo08:06
*** fgimenez has quit IRC08:06
*** fgimenez has joined #tripleo08:06
*** panda_ has joined #tripleo08:06
*** dtantsur|afk is now known as dtantsur08:27
*** jcoufal has joined #tripleo08:27
*** liverpooler has joined #tripleo08:30
*** liverpooler has quit IRC08:34
*** liverpooler has joined #tripleo08:35
*** devvesa has joined #tripleo08:39
*** mcornea has joined #tripleo08:42
*** coolsvap is now known as coolsvap|away08:43
*** derekh has joined #tripleo08:47
*** devvesa has quit IRC08:48
*** bvandenh has joined #tripleo08:56
derekhCI is throwing this up this morning  "/home/stack/overcloudrc: No such file or directory"08:57
derekhanybody know if its moved?08:57
derekhnevermind the error is up further, 2016-01-20 04:11:41.163 | 2016-01-20 04:07:12 [0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 608:58
* greghaynes bets $20 on pip 8 release09:00
*** yamahata has quit IRC09:01
mariosgreghaynes: hey man, saw your ping. sorry about that. there was no-one else about and dtanstur had been waiting for >1 day and it was starting to block them09:03
greghaynesmarios: Its no problem :)_09:03
mariosgreghaynes: in future i'll refrain without asking about the v (I have tried to before but hellabusy last few days and just quickly did the release for them yesterday)09:04
mariosgreghaynes: thx09:04
*** devvesa has joined #tripleo09:04
greghaynesmarios: eh, feel free to release whenever, we might want to document how we release better...09:05
*** nico_auv has joined #tripleo09:11
*** dtantsur is now known as dtantsur|brb09:14
*** jistr has joined #tripleo09:16
*** mbound has joined #tripleo09:24
openstackgerritSteven Hardy proposed openstack/instack-undercloud: Fix puppet-lint failure  https://review.openstack.org/27008009:31
*** gfidente has joined #tripleo09:31
*** gfidente has quit IRC09:31
*** gfidente has joined #tripleo09:31
openstackgerritMerged openstack/instack-undercloud: Enable loadbalancer with ssl on undercloud  https://review.openstack.org/26537609:35
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with []  https://review.openstack.org/26752309:36
*** akrivoka has joined #tripleo09:37
*** devvesa has quit IRC09:37
*** xinwu has quit IRC09:46
*** xinwu has joined #tripleo09:48
*** devvesa has joined #tripleo09:52
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings)  https://review.openstack.org/26810309:52
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds v6 capability to the deploy validation test (pings)  https://review.openstack.org/26810309:54
*** mkovacik has quit IRC09:54
openstackgerritDerek Higgins proposed openstack/tripleo-common: Pin the puppetlabs mongodb puppet module  https://review.openstack.org/27009509:55
*** xinwu has quit IRC09:56
*** shivrao has joined #tripleo10:02
*** jaosorior has joined #tripleo10:02
*** jaosorior has quit IRC10:04
*** jaosorior has joined #tripleo10:04
*** xinwu has joined #tripleo10:05
shardyHi all, could someone approve https://review.openstack.org/#/c/265267/ (2 line stable backport) please?10:05
mariosdone10:06
shardythanks marios! :)10:07
mariosnp, i clicked a button ;)10:07
*** shivrao has quit IRC10:07
*** xinwu has quit IRC10:07
*** paramite is now known as paramite|afk10:07
*** jaosorior_ has joined #tripleo10:09
openstackgerritMerged openstack/tripleo-puppet-elements: Use a stable version of MidoNet Neutron Plugin  https://review.openstack.org/26526710:10
openstackgerritCyril Lopez proposed openstack/tripleo-heat-templates: Allow to allow_resize_to_same_host to true in nova  https://review.openstack.org/26956210:10
*** trellooobot has joined #tripleo10:10
trellooobotURGENT TRIPLEO TASKS NEED ATTENTION10:10
trellooobot+-------------------------------------+-------------------------------+-------------------+-------------+10:10
trellooobot| Title                               |              URL              | Members           | Last Active |10:10
trellooobot+-------------------------------------+-------------------------------+-------------------+-------------+10:10
trellooobot| Could not evalute MongoDB printjson | https://trello.com/c/GrdUk6RM | **NEEDS MEMBERS** |    17 min   |10:10
trellooobot+-------------------------------------+-------------------------------+-------------------+-------------+10:10
*** trellooobot has quit IRC10:10
openstackgerritMerged openstack/tripleo-heat-templates: Wait for cluster to settle in yum_update.sh  https://review.openstack.org/26367310:11
*** jaosorior has quit IRC10:11
openstackgerritMerged openstack/python-tripleoclient: Allow adding new overcloud passwords  https://review.openstack.org/26812710:12
*** mgould has joined #tripleo10:15
*** dtantsur|brb is now known as dtantsur10:15
*** electrofelix has joined #tripleo10:16
gfidentemarios, https://review.openstack.org/#/c/235423/10:17
gfidentedsneddon, will be happy too :)10:17
gfidenteand good morning10:17
mariosgfidente: yeah o/ hey will look in a bit on sthing else right now10:18
*** mkovacik has joined #tripleo10:28
*** links has quit IRC10:28
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6  https://review.openstack.org/27011010:36
jaosorior_Hey guys, this patch has already been merged in master https://review.openstack.org/#/c/267945/ any workflows for it? :D10:38
jaosorior_(it's a backport ^^)10:39
*** jaosorior_ is now known as jaosorior10:39
jaosoriorjistr: You had checked this patch before, now it's simpler than before since we figured out some parts were not needed, can you check it out again? https://review.openstack.org/#/c/268647/10:40
jaosoriormarios: thanks for the +A :D10:45
*** tosky has joined #tripleo10:46
openstackgerritMerged openstack/puppet-tripleo: Enable X-Forwarded-Proto header for cinder  https://review.openstack.org/26794510:47
*** links has joined #tripleo10:51
*** stendulker_ has joined #tripleo11:01
*** stendulker has quit IRC11:04
*** rcernin has quit IRC11:05
*** links has quit IRC11:05
*** paramite|afk is now known as paramite11:05
jistrjaosorior: looks alright :) +211:06
shardyderekh: Hey, how much memory do we have now for undercloud VMs?11:07
*** rcernin has joined #tripleo11:07
jaosoriorjistr: Thanks dude :D11:07
shardyI suspect it's not enough, as I've seen heat-engine processes getting OOM killed here:11:07
shardyhttps://review.openstack.org/#/c/260413/711:07
jaosoriorjistr: Is the bottle still at your team area?11:07
jaosorior'11:08
*** mgould has quit IRC11:08
*** stendulker_ has quit IRC11:08
jistrjaosorior: lol yes, still almost untouched :)11:09
jistrwe have it as a decoration at this point11:09
jaosoriorjistr: uhm... was thinking of bringing another one, but that doesn't sound promising11:09
jaosoriorjistr: Offer hewbrocca some if he's around :P11:10
derekhshardy: it was upped from 4G to 5G on sunday (actually looking at the logs free is reporting it as 4.8G)11:10
hewbroccajaosorior: WAT I had no idea what that bottle was11:11
hewbroccaconsider it comsumed11:11
jistr:D11:11
shardyderekh: ack - I suspect that's not enough, as the increase in vCPU count is spawning more processes11:11
hewbroccajaosorior: if you want something drunk at this office, you have to engage the managers11:12
* shardy looks around for stuff to turn off11:12
derekhfeck it anyways ;-)11:12
jaosoriorhewbrocca: haha now I know :P Anyway, that better be done before DevConf, cause I'm bringing more bottles11:13
*** rcernin is now known as rcernin|lunch11:13
jaosoriorright now the count of bottles is: 2 for two guys from the ID team, one for the tripleo midcycle and another one for DevConf11:14
derekhshardy: ack, if you can't find something to turn off, we can either up it a little more or add a little swap somehow.11:14
shardyderekh: Yeah, I think having the option to enable swap would be good, then we can figure out a way to scrape the logs and ensure it's not routinely getting used to excess11:15
shardyI guess perhaps we can add it as an option to the undercloud install11:15
shardyor at least to tripleo.sh11:16
gfidenteshardy, derekh I while ago I did some investigation on which processes were scaling with the number of cores11:17
gfidentehttps://bugzilla.redhat.com/show_bug.cgi?id=1247924#c311:17
openstackbugzilla.redhat.com bug 1247924 in rhel-osp-director "MariaDB max_connections must be based on controller count * core count" [High,New] - Assigned to jstransk11:17
jistrshardy, derekh: yeah that's the other thing we did. We upped the vCPUs too, so i reckon the mem usage went up as well.11:18
derekhgfidente: not as many as I thought there would be, but obviously to many....11:18
gfidenteshardy, derekh also I have a running undercloud with 2 vcores11:18
shardygfidente: AFAIK nearly all projects use oslo processutils, which defaults to a worker process per core11:19
gfidente$ free -m11:19
gfidente              total        used        free      shared  buff/cache   available11:19
gfidenteMem:           7823        4072         128         374        3622        313411:19
*** Marga_ has quit IRC11:19
jistrmine is11:20
jistr[stack@instack ~]$ free -m11:20
jistr              total        used        free      shared  buff/cache   available11:20
jistrMem:           5826        5342         132           0         351         24511:20
gfidentejistr, interestingly the difference in caches is a lot11:20
*** fgimenez has quit IRC11:20
*** mgould has joined #tripleo11:20
jistrgfidente: i think the cache is so small in my case because i'm almost OOM11:20
gfidenteisn't cache part of used?11:21
derekhthe difference in cache could be explained by how long each undercloud has been running and how much data been accessed11:21
derekhgfidente: the cache number if the OS cahcing file access, it gets released as memory is required11:22
derekhthen file reads get slower because less data is cached11:22
gfidentederekh, yeah so I was noticing on mine seems to be using 3g on caches and a total of 4g11:23
derekhso its not part of used11:23
gfidentederekh, and on jistr the opposite11:23
gfidentederekh, I think it is included in used11:24
*** fgimenez has joined #tripleo11:24
*** fgimenez has quit IRC11:24
*** fgimenez has joined #tripleo11:24
derekhgfidente: yup, you OS is caching 3.6G of data as that whats available, jistr doesn't have spare ram to cache as much11:24
jistrthis is the most hungry processes on my system http://fpaste.org/312720/28895214/11:24
derekhnope, don't think it is11:24
derekhmy laptop at the moment11:25
shardyI raised bug #1536136 where we can decide the best solution11:25
openstackbug 1536136 in tripleo "Undercloud sometimes OOM kills things in CI" [Undecided,New] https://launchpad.net/bugs/153613611:25
derekh              total        used        free      shared  buff/cache   available11:25
derekhMem:          15737        3038        6592         649        6106       1172711:25
derekhcached is begger then used11:25
derekh*bigger11:25
shardyjistr: interesting, is that during a deploy?11:26
shardywe can perhaps force the number of workers to 2 for many things - in particular heat-engine workers now default to 4, but 2 may be sufficient for simple CI tests11:26
jistrshardy: nope, but it's been running for about 2 days11:26
jistrshardy: yeah perhaps... and also i don't think we need 5 nova-api processes probably11:28
shardyjistr: Hmm, in that case the heat-engine memory usage seems high11:28
shardyyeah, probably some tuning we can do specific to CI testing11:28
jistri'm probably using newer heat engine then others11:29
jistr[stack@instack ~]$ rpm -q openstack-heat-engine11:29
jistropenstack-heat-engine-6.0.0.0b2-dev183.el7.centos.noarch11:29
jistri needed to bump heat manually to overcome the UPDATE_IN_PROGRESS getting stuck11:29
shardyYeah so you've picked up https://review.openstack.org/#/c/259172/11:30
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add output of top to hostinfo  https://review.openstack.org/27013911:32
derekhthat should give a little info about hich processes are using most memory ^^11:32
*** links has joined #tripleo11:32
gfidentejistr, shardy I have openstack-heat-engine-2015.1.2-7.el7ost.noarch11:32
gfidentejistr, shardy running from a couple of days as well11:33
derekhHi all, master is broken because of a commit to the mongodb puppet module, the pin fixes it https://review.openstack.org/#/c/270095/111:34
gfidentejistr, shardy but virtual memory was very similar, I see in your paste 79045611:34
gfidentemine is more or less same size11:34
gfidente 8308 heat      20   0  676116 285624   2988 S   0.0  3.6   4:19.41 heat-engine11:34
gfidente 8309 heat      20   0  663648 272676   2988 S   0.0  3.4   3:24.60 heat-engine11:34
derekhI'm looking at the puppet module at the moment to see if I can fix it, wanna merge the pin in the meantime11:34
shardygfidente, jistr: I'm seeing much lower heat-engine memory usage on a freshly started undercloud11:35
shardydoes the usage go down a lot if you restart heat-engine via systemctl?11:36
jistryes11:36
jistr70 - 140 MB now11:36
gfidentefor me too11:36
jistrper process11:36
shardyOk, so we probably have a memory leak then :(11:36
gfidentebut it would't explain why in ci 5 are not enough though11:37
openstackgerritMerged openstack/tripleo-common: Pin the puppetlabs mongodb puppet module  https://review.openstack.org/27009511:37
jistrone thing i don't understand in my env is where does 5 nova-api proceses come from when i have osapi_compute_workers=211:38
shardygfidente: the peak memory usage may be higher than the idle usage, even with the leak11:38
*** links has quit IRC11:38
jistri changed the numper of nova api workers to 1 and now i have 4 processes... i guess nova-api might fork for some other reason than just serving web requests then11:41
hewbroccageneral principles11:41
gfidenteshardy, indeed but I mean in my env it seems stable doing deploy/delete with less than 4G11:42
gfidenteshardy, or about 4G11:42
gfidentefrom a couple of days11:42
gfidentethough I run RHEL fwiw11:42
*** openstackgerrit has quit IRC11:43
gfidentenot sure if kernel settings and python deps are identical to centos11:43
*** openstackgerrit has joined #tripleo11:44
jistr(ahh i guess i got the nova-api thing, the other ones are metadata workers, and 1 main process. makes sense now)11:45
-openstackstatus- NOTICE: review.openstack.org is being restarted to apply patches11:45
*** ChanServ changes topic to "review.openstack.org is being restarted to apply patches"11:45
*** ChanServ changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs http://docs.openstack.org/developer/tripleo-docs/"11:56
-openstackstatus- NOTICE: Restart done, review.openstack.org is available11:56
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6  https://review.openstack.org/27015412:02
gfidentederekh, caches accounts indeed in the available (and non-free) memory12:04
*** panda_ has quit IRC12:06
derekhgfidente: is this related to todays breakage ? https://review.openstack.org/#/c/270154/112:06
*** panda_ has joined #tripleo12:06
gfidentederekh, nah it's for the ipv6 deployments12:07
derekhgfidente: ack, just checking12:07
gfidentewhat is today issue?12:07
derekhgfidente: https://bugs.launchpad.net/tripleo/+bug/153608312:08
openstackLaunchpad bug 1536083 in tripleo "Could not evalute MongoDB shell command: printjson(rs.conf())" [Critical,Triaged] - Assigned to Derek Higgins (derekh)12:08
derekhgfidente: pinning puppetlbs-mondb unblocked things, looks like this commit is when it started happening https://github.com/puppetlabs/puppetlabs-mongodb/commit/a77d1a191419a3f1f468707aefa69abd689c3e4b12:09
derekhEmilienM: ^12:09
derekhEmilienM: any ideas?12:09
*** thrash|g0ne is now known as thrash12:09
EmilienMderekh: yes12:10
EmilienMderekh: I recently patched mongo module to support IPv612:10
EmilienMderekh: is it affecting master CI?12:11
EmilienMor stable?12:11
derekhEmilienM: master12:11
EmilienMargh12:11
derekhEmilienM: looks to me like the output of this command can't be parsed as json12:12
EmilienMderekh: can you pin ? I need time to debug, lot of issues in parallel12:12
derekhEmilienM: http://paste.openstack.org/show/484392/12:12
derekhEmilienM: I hvae already pinned12:12
EmilienMderekh: I'm not sure that's my patch12:13
EmilienMI need time12:13
derekhthe various date types in the json seem to be the problem to me12:13
derekhEmilienM: no problem, no hurry the pin keeps us moving12:13
shardyDoes anyone know how we can make the various puppet check jobs voting?12:14
shardythey're all working modulo https://review.openstack.org/#/c/270080/ getting backported12:14
shardyit'd be good to stop them constantly getting broken, as it's clear folks are sometimes ignoring failures12:15
* shardy can't see where to change them in project-config atm12:15
derekhshardy: sec, I'll find it12:15
*** thrash has quit IRC12:17
derekhshardy: probably here, but it might be matching more then just the lint jobs http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n140012:19
*** thrash has joined #tripleo12:19
*** thrash has quit IRC12:19
*** thrash has joined #tripleo12:19
shardyderekh: ah, thanks, yeah that looks like those two will do it12:20
shardyderekh: I sent https://review.openstack.org/270165, thanks - I'll also drop a list to the ML so folks are aware12:25
derekhshardy: +112:27
*** rhallisey has joined #tripleo12:35
*** dmacpher has joined #tripleo12:39
openstackgerritRyan Hallisey proposed openstack/tripleo-heat-templates: Allow container template to recognize an update  https://review.openstack.org/26536812:39
*** olap_ has joined #tripleo12:40
*** olap_ has quit IRC12:40
*** olap_ has joined #tripleo12:41
*** weshay_xchat has joined #tripleo12:42
derekhEmilienM: found the problem, https://bugs.launchpad.net/tripleo/+bug/1536083/comments/312:43
openstackLaunchpad bug 1536083 in tripleo "Could not evalute MongoDB shell command: printjson(rs.conf())" [Critical,Triaged] - Assigned to Derek Higgins (derekh)12:43
derekhEmilienM: I'll see if I can muster up enough ruby Foo to submit a fix12:43
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Config fixes for proper nova/neutron notifications  https://review.openstack.org/26421712:51
*** rlandy has joined #tripleo12:55
gfidenteshardy, what is best way to have newlines in the str_replace/template thing?12:55
*** athomas has quit IRC12:57
*** athomas has joined #tripleo13:01
gfidenteshardy, going for : |13:01
shardygfidente: Yeah, I'd do : |13:01
gfidentetks13:02
*** shivrao has joined #tripleo13:04
*** morazi has joined #tripleo13:05
*** rcernin|lunch is now known as rcernin13:07
*** shivrao has quit IRC13:08
*** anande has quit IRC13:09
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6  https://review.openstack.org/27015413:10
EmilienMderekh: looking13:13
EmilienMthat's weird, we tested it againsts both ipv4 & ipv613:14
*** jayg|g0n3 is now known as jayg13:17
*** trozet has joined #tripleo13:19
*** trown|outttypeww is now known as trown13:24
*** ayoung has quit IRC13:24
*** dprince has joined #tripleo13:26
gfidenteshardy, another question, can I concat as value {get_attr}+string ?13:29
shardygfidente: Yes, if it doesn't work directly you can use either str_replace or list_join13:31
gfidenteok I'll settle for list_join13:32
shardygfidente: Hey, can you remind me, what was going wrong with NetworkDeployment when it runs on update?13:32
shardyIf you set NetworkDeploymentActions to include update, e.g to apply a new nic config, will it break the update due to pacemaker issues?13:33
gfidenteshardy, doing ifdown/ifup caused troubles to corosync13:33
gfidenteshardy, yeah exactly13:33
gfidenteshardy, os-net-config does the right thing when the nic config does not change though13:33
gfidenteshardy, it's the upgrades to the nic configs which triggered ifdown/ifup13:34
shardygfidente: thanks13:35
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: WIP: Emits a different hostname for each network the node uses  https://review.openstack.org/27018913:38
*** mbound has quit IRC13:39
jistranybody recalls what's the issue with "CREATE_FAILED  Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 6499acd4-80c5-4fae-866f-db9877fa3b75." ?13:39
*** trozet has quit IRC13:39
jistri seem to be able to re-introspect the nodes just fine, so ironic can still control them13:39
* jistr digs for more info13:39
*** Goneri has joined #tripleo13:43
hewbroccajistr: hang on, that's the damn scheduler bug13:45
*** jcoufal_ has joined #tripleo13:47
*** hewbrocc` has joined #tripleo13:48
EmilienMderekh: I'm working on a fix right now13:49
*** crinkle_ has joined #tripleo13:49
jistrhewbrocca: thanks, unfortunately it seems to be something different. There's no mention of NodeAssociated in ironic-api.log. But it's most probably some problem between nova and ironic too.13:49
hewbrocc`hmm OK13:49
*** hewbrocc` is now known as hewbrocca`13:50
*** teju has left #tripleo13:50
hewbrocca`INteresting13:50
hewbrocca`dtantsur: ^^^13:50
dtantsurmoar bugs \o/13:50
dtantsurjistr, look for errors in both ironic-conductor and nova-compute logs13:51
dtantsurnova tends to clean up everything after failure (so nice of it, damn)13:51
derekhEmilienM: ack, here is my attempt, don't have ipv6 to check both cases though13:51
derekhEmilienM: http://paste.openstack.org/show/484406/13:51
*** liverpoo1er has joined #tripleo13:51
*** lifeless_ has joined #tripleo13:51
*** bandini_ has joined #tripleo13:51
*** Slower_ has joined #tripleo13:51
*** gothicmi1dfood has joined #tripleo13:51
jistrdtantsur: ironic-api.log is clean of error messages, in nova-compute there's this http://fpaste.org/312783/53297910/13:52
dtantsurjistr, ironic-conductor, not ironic-api13:52
jistrdtantsur: ah thanks13:53
jistryea got it13:53
dtantsurawesome error message, thanks nova :D13:53
jistr2016-01-20 13:02:58.975 23433 ERROR ironic.drivers.modules.iscsi_deploy [-] Deploy failed for instance 8a1f6072-acad-4c5a-8acd-38b2a8e67fde. Error: [Errno 12] Cannot allocate memory13:53
dtantsurwell, that looks like your problem13:53
trownnot enough RAM on the undercloud?13:53
* dtantsur puts this nova behavior on his troubleshooting talk for the rdo day13:53
jistryeah most probably, where not enough == 6 GB13:54
*** mandre_ has joined #tripleo13:55
*** leifmadsen_ has joined #tripleo13:55
jistrheat-engine grew to 1 GB after a few stack-updates13:55
*** cinerama` has joined #tripleo13:55
jistr(all workers summed together)13:55
*** athomas has quit IRC13:56
*** nico_auv has quit IRC13:56
*** 18VAAPK0U has quit IRC13:56
*** paramite has quit IRC13:56
*** regebro has quit IRC13:56
*** hewbrocca has quit IRC13:56
*** Slower has quit IRC13:56
*** d0ugal has quit IRC13:56
*** tserong has quit IRC13:56
*** shadower has quit IRC13:56
*** gothicmindfood has quit IRC13:56
*** mcornea has quit IRC13:56
*** liverpooler has quit IRC13:56
*** jcoufal has quit IRC13:56
*** mandre has quit IRC13:56
*** bandini has quit IRC13:56
*** crinkle has quit IRC13:56
*** cinerama has quit IRC13:56
*** lifeless has quit IRC13:56
*** leifmadsen has quit IRC13:56
*** leifmadsen_ is now known as leifmadsen13:56
*** absubram has quit IRC13:56
*** mandre_ is now known as mandre13:58
*** nico_auv has joined #tripleo14:02
*** paramite has joined #tripleo14:02
*** tserong has joined #tripleo14:02
*** mcornea has joined #tripleo14:03
*** d0ugal has joined #tripleo14:03
jistrhmm nope still having issues getting the machines to spawn. this in nova-compute log, nothing in ironic-conductor or ironic-api logs http://fpaste.org/312786/14532985/14:04
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc  https://review.openstack.org/26202814:04
*** regebro has joined #tripleo14:04
*** egafford has joined #tripleo14:05
*** ukalifon has joined #tripleo14:05
*** athomas has joined #tripleo14:05
dtantsurjistr, do you see ironicclient output a bit above your paste? might contain the actual error14:06
jistrdtantsur: http://fpaste.org/312789/32988211/ so a problem between ironic api and ironic-conductor?14:07
dtantsurjistr, it looks like conductor failed to start, can you check it?14:07
*** tosky has quit IRC14:08
jistrdtantsur: looks ok http://fpaste.org/312791/98954145/raw/14:09
*** olap_ has quit IRC14:09
* jistr tries to find where the connection between api and conductor is configured14:09
dtantsurjistr, it's amqp14:10
hewbrocca`the rabbit done died14:10
dtantsurjistr, what does 'ironic driver-list' output for you?14:10
hewbrocca`uh-yes it did14:10
*** hewbrocca` is now known as hewbrocca14:10
*** mbound has joined #tripleo14:10
* hewbrocca walk this way14:10
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: Create linux bridge vlans environments  https://review.openstack.org/27020514:11
jistrdtantsur: driver-list is empty. Rabbit is running according to systemctl. Looking for more suspicious stuff.14:11
*** olap has joined #tripleo14:11
dtantsurjistr, you may try to restart the conductor then.. unless you find some failures14:11
jistrdtantsur: that has probably done it, after restarting conductor i see driver-list populated with entries. Will retry the stack create. Thanks!14:13
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: Fix MidoNet errors  https://review.openstack.org/27020714:14
*** tosky has joined #tripleo14:14
*** mbound has quit IRC14:15
EmilienMderekh: https://github.com/puppetlabs/puppetlabs-mongodb/pull/25714:15
EmilienMderekh: it will land today14:15
EmilienMI'll ping you when done14:15
EmilienMderekh: and sorry for that14:15
*** lblanchard has joined #tripleo14:18
derekhEmilienM: no prob and thanks for looking at it14:19
*** tzumainn has joined #tripleo14:20
jistrok now i'm getting "just" the PortBindingFailed error, will try to debug neutron for a change :) i wonder if the OOM error prompted the operating system to do something not-so-nice to the undercloud in general14:22
dtantsurit's pretty possible, I would restart everything just in case14:22
jistryeah ovs agent is down14:23
*** tiswanso has joined #tripleo14:23
jistrdtantsur: alright thanks a lot for the help :)14:23
*** athomas has quit IRC14:24
dtantsuryvw14:26
*** athomas has joined #tripleo14:28
*** rpothier has joined #tripleo14:29
*** tiswanso has quit IRC14:30
*** tiswanso has joined #tripleo14:31
*** absubram has joined #tripleo14:33
*** liverpoo1er has quit IRC14:36
*** paramite has quit IRC14:39
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6  https://review.openstack.org/27015414:50
*** eggmaster has joined #tripleo14:51
*** jprovazn has quit IRC14:52
*** masco has quit IRC14:53
rbradydprince: I'm available from now until 3pm EST whenever you want to meet up14:54
dprincerbrady: how about we shoot for 1PM EST?14:58
* dprince is home with a sick kid today and will have more help by then14:58
rbradydprince: that's good for me.  Can we record it for the folks in Europe that cannot make it that late?14:58
dprinceanyone else interested in an ad-hoc Mistral demo... rbrady and I will be discussing it later today on BlueJeans14:58
*** electrofelix has quit IRC14:59
dprincerbrady: but if we record it that means I won't be able to tell you what I really think!!!14:59
dprincerbrady: what I mean is... I might just prefer to make a separate (polished) demo recording without my rants14:59
rbradydprince: I thought everyone has already had the chance to experience dprince unfiltered at least once already15:00
rbradydprince: ack15:00
*** jaosorior has quit IRC15:06
*** shivrao has joined #tripleo15:06
*** shivrao has quit IRC15:11
*** electrofelix has joined #tripleo15:13
dtantsurd0ugal or thrash: could you review https://review.openstack.org/#/c/250405/ please? profile matching: long, but important :)15:14
thrashdtantsur: I'll take a look shortly.15:14
dtantsurthanks15:14
thrashanybody that can give some review love to https://review.openstack.org/#/c/266404/ would be awesome. :)15:14
*** electrofelix has quit IRC15:16
*** electrofelix has joined #tripleo15:17
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6  https://review.openstack.org/27015415:17
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: SoftwareDeployment interface for OS::TripleO::Tasks::PackageUpdate  https://review.openstack.org/27024215:19
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Move controller upgrade to the yum_update hook  https://review.openstack.org/27024315:19
*** trozet has joined #tripleo15:21
thrashdprince: d0ugal thanks15:23
d0ugaldtantsur: Yup, I'll take a look - but it might be tomorrow morining with a strong coffee :)15:26
*** mbound has joined #tripleo15:26
dtantsurack15:27
*** yamahata has joined #tripleo15:30
*** bvandenh has quit IRC15:31
*** jprovazn has joined #tripleo15:36
*** fabbione has quit IRC15:36
*** fabbione has joined #tripleo15:37
*** tiswanso has quit IRC15:39
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Wire Neutron ML2 plugin and OVS agent settings as arrays  https://review.openstack.org/26989815:39
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Revert "Revert "neutron: enable nova-event-callback by default""  https://review.openstack.org/26982715:39
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Fix tunnel_types hieradata on compute nodes  https://review.openstack.org/26982915:39
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Fix neutron-nova notifications  https://review.openstack.org/26982815:39
*** tiswanso has joined #tripleo15:41
*** electrofelix has left #tripleo15:42
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file.  https://review.openstack.org/26791715:43
dtantsurdevvesa, hi! do you still plan on working on https://review.openstack.org/#/c/265827/ ? I need it for clean rebase of another patch15:46
devvesadtantsur: I should do it15:46
devvesadtantsur: What about if I do it tomorrow? Does it work for you?15:46
dtantsuryeah, sure. thanks!15:47
devvesaOk. Tomorrow you'll have it :)15:47
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder: Mark ironic-discoverd-ramdisk as deprecated in favor of ironic-agent  https://review.openstack.org/27026715:54
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud  https://review.openstack.org/27027415:58
derekhpign test passed for nonha and ceilometer jobs, need to fix HA now15:59
derekh*ping15:59
*** fabbione has quit IRC16:02
*** fabbione has joined #tripleo16:03
trownderekh: is the HA issue just a resource thing, or does it actually fail?16:03
trownderekh: I have had issues getting HA to pass some of the tempest smoke tests, but I am almost positive it is just resource constraints16:04
*** shivrao has joined #tripleo16:05
*** panda_ has quit IRC16:06
*** panda_ has joined #tripleo16:07
derekhtrown: 2016-01-20 15:37:50.143 | An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-77948921-a027-4e6c-8ac5-ca62072a4e92)16:07
derekhtrown: an error returned by one of the nova or neutron api, not sure of anything more yet16:08
*** jaosorior has joined #tripleo16:08
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Surround MongoDB IPs with braces in the connection string if IPv6  https://review.openstack.org/27015416:08
trownderekh: hmm... it seems like either HA is broken, or we do not have enough resources in CI to deploy HA... I have a hard time seeing how else we would have nonha ping test pass but ha fail16:10
openstackgerritJaume Devesa proposed openstack/tripleo-docs: Extending the image build information  https://review.openstack.org/27029016:10
trownderekh: ie, there should not be anything specific on the test side for ha vs nonha16:10
derekhtrown: yup16:10
*** jaosorior has quit IRC16:11
trownderekh: personally, I think we should merge the ping test then, at least it will prevent merging things that break the nonha overcloud16:12
trownderekh: would it be possible to only run the ping test on nonha and ceph, like introspection only runs on nonha?16:13
derekhtrown: fair enough, there is a couple of comments on the test in tripleo-common , we should figure out if we want to sort them out now or post merge, then do it16:13
derekhtrown: yes that would be possible also16:15
trowndeploying HA realisically in a resource constrained env might be impossible16:15
derekhYup, Ive wrechecked to see if it was a once off, then we can decide if we want to go without it or not16:17
*** shivrao has quit IRC16:19
bnemecderekh: I opened a bug for the ha failure.  It's another wsrep error, which I've been seeing a lot lately: https://bugs.launchpad.net/tripleo/+bug/153625016:22
openstackLaunchpad bug 1536250 in tripleo "Intermittent database problems in the HA job" [High,Triaged]16:22
bnemecI don't think it actually has anything to do with the ping test because I've seen it happen a lot on the existing job too.16:22
bnemecAlthough it's possible the ping test will trigger it more often.16:23
*** Marga_ has joined #tripleo16:24
*** rcernin has quit IRC16:25
*** aufi has quit IRC16:30
*** tiswanso has quit IRC16:34
*** cinerama` is now known as cinerama16:37
*** cwolferh has joined #tripleo16:39
*** electrofelix has joined #tripleo16:39
*** mbound has quit IRC16:53
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Move controller upgrade to the yum_update hook  https://review.openstack.org/27024316:55
openstackgerritJaume Devesa proposed openstack/tripleo-docs: Extending the image build information  https://review.openstack.org/27029016:55
*** sthillma has joined #tripleo16:58
*** sthillma_ has joined #tripleo16:59
openstackgerritJaume Devesa proposed openstack/tripleo-docs: Add MidoNet documentation in advanced deployment  https://review.openstack.org/27032017:00
*** mbound has joined #tripleo17:02
*** sthillma has quit IRC17:02
*** sthillma_ is now known as sthillma17:02
*** trown is now known as trown|lunch17:03
*** devvesa has quit IRC17:03
*** jistr has quit IRC17:04
*** tiswanso has joined #tripleo17:05
*** akrivoka is now known as akrivoka|afk17:06
*** mkovacik has quit IRC17:06
*** tiswanso has quit IRC17:06
*** tiswanso has joined #tripleo17:07
afazekaswhat can be used in tripleo-common repo for changing  a *.ini file which is at least as fast as the devstack iniset ?17:09
*** yamahata has quit IRC17:10
*** dtantsur is now known as dtantsur|afk17:11
*** bandini_ is now known as bandini17:12
*** fgimenez has quit IRC17:18
*** rasca has quit IRC17:21
afazekasI would like to set the overcloud credentials in the tempest.conf17:22
afazekasderekh, ^17:23
afazekasopenstack-config has a similar interface but it is slower (python load time)17:24
*** rasca has joined #tripleo17:25
*** Marga_ has quit IRC17:25
shardyafazekas: personally I think copying the iniset function into tripleo-common would be OK - it's only like 30 lines of code17:26
*** Marga_ has joined #tripleo17:26
shardyunless there's some existing alternative (I'm not aware of one)17:26
derekhafazekas: shardy yum, I'd be fine with that, was about to say the same thing, shardy was quicker looking it up ;-)17:27
derekhif its a fairly simple case even a subset of iniset might do17:27
derekh:q17:28
bnemecafazekas: Would it be possible to do it in here: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L11417:28
bnemecThat's where we are already setting tempest things that are specific to the deployment.17:28
afazekasI do not see 'tempest-deployer-input.conf' realy gets created17:29
afazekasWhere should I find it ?17:30
afazekasIs the config supposed to contain everything ? or just the list of deployed components and available features ?17:31
* afazekas I also do not have overcloudrc , maybe something is wrong with my deployment ..17:32
* afazekas yes it was broken ..17:34
afazekasopenstack-config can merge ini files for that it can be faster than a shell script when the arg number is high17:35
*** yamahata has joined #tripleo17:37
*** yamahata has quit IRC17:37
*** yamahata has joined #tripleo17:37
*** shivrao has joined #tripleo17:38
*** shivrao has quit IRC17:39
*** olap has quit IRC17:40
*** jdob has quit IRC17:43
afazekaswhen tempest used from rpm this https://github.com/redhat-openstack/khaleesi/blob/master/playbooks/tester/templates/configure-tempest.j2  is used by khaleesi17:44
afazekasshould I follow the same configuration pattern ?17:44
afazekastools/config_tempest.py is not in the upstream tempest repo17:45
*** ayoung has joined #tripleo17:47
dprincegfidente: again https://review.openstack.org/#/c/26906817:48
gfidentedprince, hey not to steal a vote but does this look valid or is there a reason for not doing it that you can think https://review.openstack.org/270342 ?17:52
afazekasderekh: Do you recommend to use the rpm specific scripts ?17:52
*** jcoufal_ has quit IRC17:54
*** mbound has quit IRC17:54
*** shivrao has joined #tripleo17:58
*** weshay_xchat has quit IRC17:58
*** shardy has quit IRC17:59
afazekasderekh, is the tempest usage in the tripleo ci should depend on the /usr/share/openstack-tempest*/tools/config_tempest.py  which is part of the rpm ?17:59
derekhIs that what you mean by the rpm specif scripts?18:00
*** sthillma has quit IRC18:00
derekhafazekas: I'm not very familiar with tempest, its been about 2 years since I used it, what is the alternative?18:00
afazekasderekh: yes18:00
derekhafazekas: if you mean files from /usr/share/openstack-tempest* or fomr a git checkout of tempest18:01
derekhafazekas: then I would say we should use the rpm ones18:02
*** weshay_xchat has joined #tripleo18:02
bnemecThis is our currently documented method of running tempest: http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html#validate-the-overcloud18:02
bnemecIf the deployer-input.conf file is not being created then it's a bug.18:02
openstackgerritgreghaynes proposed openstack/diskimage-builder: Use pip 7 for ironic  https://review.openstack.org/27001018:02
derekhYup, ci should be doing what we are documenting18:03
*** weshay_xchat is now known as weshay18:03
afazekasderekh: these scripts are not in the official upstream tempest repo, so I do not really  like to use them, the alternative is implementing something similar ..18:03
derekhor if that can't be done perhapes something in the docs is wrong18:04
*** crinkle_ is now known as crinkle18:04
derekhafazekas:  the problem is, if you don't use them then why do we bother package them at all18:04
derekhI mean they are in the RDO repository that we are trying to validate, if its not a package we intent to test then we should remove the package18:05
afazekasderekh: This was not request from QE,  AFAIK customers wanted it  in rpm18:05
afazekasIt is also related somehow to certification processes18:06
afazekasbnemec: Ok, I'll will try to follow the doc18:07
derekhafazekas: sorry, I'm not saying you wanted it there, I'm saying it is there and if we don't use it to test then its probably broken and we shoudl remove it18:07
derekhafazekas: I think, if we as a community think its important enough a package to keep in the repository then we should be using it, thats my opinion anyways18:08
*** paramite has joined #tripleo18:08
derekhafazekas: sorry I got to run for the evening, others might have other opinions on here about the packaged tempest vs. tempest from git18:10
derekhfor anybody interested afazekas is looking into adding some tepest tests to our periodic jobs18:10
derekhttyl18:10
*** derekh has quit IRC18:10
*** lifeless_ is now known as lifeless18:12
*** trown|lunch is now known as trown18:12
*** jtomasek has quit IRC18:15
*** ukalifon has quit IRC18:15
*** dhellmann_ has joined #tripleo18:20
*** isq_ has joined #tripleo18:20
*** trozet_ has joined #tripleo18:20
*** larstobi_ has joined #tripleo18:21
*** mandre has quit IRC18:21
*** trozet has quit IRC18:22
*** rpothier has quit IRC18:22
*** athomas has quit IRC18:22
*** akrivoka|afk has quit IRC18:22
*** cody-somerville- has quit IRC18:22
*** dhellmann has quit IRC18:22
*** mburned has quit IRC18:22
*** isq has quit IRC18:22
*** larstobi has quit IRC18:22
*** larstobi_ is now known as larstobi18:22
*** dhellmann_ is now known as dhellmann18:22
*** cody-somerville- has joined #tripleo18:23
*** akrivoka|afk has joined #tripleo18:24
*** athomas has joined #tripleo18:24
*** trozet_ is now known as trozet18:25
*** mandre has joined #tripleo18:25
*** penick has joined #tripleo18:27
*** paramite is now known as paramite|afk18:29
*** gfidente is now known as gfidente|afk18:30
*** paramite|afk is now known as paramite18:33
*** akrivoka|afk has quit IRC18:35
*** athomas has quit IRC18:35
*** ayoung has quit IRC18:37
*** ayoung has joined #tripleo18:37
*** akrivoka|afk has joined #tripleo18:37
*** athomas has joined #tripleo18:37
openstackgerritEthan Gafford proposed openstack/python-tripleoclient: Sahara integration  https://review.openstack.org/22141818:40
*** jdob_lt has joined #tripleo18:42
*** mgould has quit IRC18:43
*** electrofelix has quit IRC18:47
*** paramite has quit IRC18:47
*** akrivoka|afk has quit IRC18:48
*** athomas has quit IRC18:48
*** akrivoka|afk has joined #tripleo18:48
*** athomas has joined #tripleo18:48
*** gothicmi1dfood has quit IRC18:53
*** cmyster has quit IRC18:53
*** cmyster has joined #tripleo18:53
*** gothicmindfood has joined #tripleo18:56
*** hewbrocca has quit IRC18:56
*** hewbrocca has joined #tripleo18:56
*** Marga_ has quit IRC18:56
*** bandini has quit IRC18:56
*** bandini has joined #tripleo18:56
*** tosky has quit IRC18:57
*** tosky has joined #tripleo18:57
*** lblanchard has quit IRC18:57
*** lblanchard has joined #tripleo18:57
*** weshay has quit IRC18:58
*** weshay has joined #tripleo18:58
*** akrivoka|afk has quit IRC18:59
*** athomas has quit IRC18:59
openstackgerritafazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node  https://review.openstack.org/27042318:59
*** trozet has quit IRC18:59
*** trozet has joined #tripleo18:59
*** penick has quit IRC18:59
*** penick has joined #tripleo18:59
*** ayoung has quit IRC18:59
*** ayoung has joined #tripleo18:59
*** rwsu has quit IRC19:03
*** liverpooler has joined #tripleo19:05
*** xinwu has joined #tripleo19:07
*** sthillma has joined #tripleo19:08
*** Marga_ has joined #tripleo19:09
*** mburned has joined #tripleo19:11
*** tosky has quit IRC19:15
*** bnemec has quit IRC19:17
*** Goneri has quit IRC19:17
*** dshulyak_ has quit IRC19:22
*** Marga_ has quit IRC19:25
*** bnemec has joined #tripleo19:25
*** rwsu has joined #tripleo19:30
EmilienMok mongodb issue is fixed upstream19:47
EmilienMI'm going to unping ooo ci19:47
EmilienMunpin*19:48
openstackgerritEmilien Macchi proposed openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module"  https://review.openstack.org/27045119:50
openstackgerritEmilien Macchi proposed openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module"  https://review.openstack.org/27045119:50
*** trown is now known as trown|mtg19:51
*** Marga_ has joined #tripleo19:55
openstackgerritJames Slagle proposed openstack/puppet-tripleo: packages: secure upgrade workflow from dependency cycles  https://review.openstack.org/26838819:57
*** jdob_lt has quit IRC19:59
*** Goneri has joined #tripleo20:02
*** mcornea has quit IRC20:06
*** panda_ has quit IRC20:06
*** weshay has quit IRC20:06
*** dshulyak_ has joined #tripleo20:18
*** panda_ has joined #tripleo20:18
*** akrivoka|afk has joined #tripleo20:18
*** athomas has joined #tripleo20:18
*** weshay has joined #tripleo20:18
*** davidlenwell has quit IRC20:18
*** davidlenwell has joined #tripleo20:20
*** ChanServ sets mode: +v davidlenwell20:20
*** jprovazn has quit IRC20:23
*** dshulyak_ has quit IRC20:28
*** panda_ has quit IRC20:28
*** akrivoka|afk has quit IRC20:28
*** athomas has quit IRC20:28
*** dprince has quit IRC20:36
*** jdob_lt has joined #tripleo20:37
ayoungEmilienM, running instack-install-undercloud'  and the tripleo.sh equivlaent I get Error: Could not find class ::ironic::keystone::auth_inspector for instack on node instack20:40
ayoungknown issue?20:40
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc  https://review.openstack.org/26202820:40
EmilienMayoung: that's definitly an OPM issue20:40
EmilienMayoung: on RDO manager?20:40
ayoungEmilienM, Tripleo upstream20:41
EmilienMoh20:41
ayoungCenteos 720:41
EmilienMeven worse20:41
EmilienMayoung: let me check, asec20:41
EmilienMayoung: master?20:41
EmilienMI see the class in ironic::keystone::auth_inspector (puppet-ironic)20:41
ayoungEmilienM, um...the only git checkout was of tripleo.sh.  THe rest was following the docs, so I don't know what branch20:41
EmilienMdo we pin puppet-ironic?20:42
EmilienMlet me check that20:42
ayounghttp://docs.openstack.org/developer/tripleo-docs/installation/installing.html20:42
ayoungdelorean...that is master, no?20:42
EmilienMyeah20:42
EmilienMderek & dprince are not around :(20:42
EmilienMthis is weird!20:42
*** dshulyak_ has joined #tripleo20:42
*** panda_ has joined #tripleo20:42
*** akrivoka|afk has joined #tripleo20:42
*** athomas has joined #tripleo20:42
EmilienMayoung: the puppet class is in master and tripleo CI is usng master modules20:42
*** weshay has quit IRC20:44
egaffordayoung, EmilienM: Possible that ayoung needs to use export DIB_INSTALLTYPE=source to pull in changes in master that haven't propagated to delorean?20:46
ayoungEmilienM, say I want to try and run this with Keystone in HTTPD.  can I do that now?20:47
egaffordOr rather export DIB_INSTALLTYPE_puppet_modules=source20:47
EmilienMayoung: you've seen my patch in progress20:47
ayoungegafford, export that and then rerun tripleo-common/scripts/tripleo.sh --undercloud20:47
EmilienMbut I'm not making any progress on it right now20:47
ayoungEmilienM, I have ,but I thought that was overcloud only20:47
EmilienMayoung: undercloud, right I can send a patch20:48
EmilienMayoung: let me 2 min20:48
ayoungsince undercloud is already failing on a Keystone issue, hey, why not?20:48
ayoungexcellent20:48
EmilienMayoung: let me 10 min20:49
EmilienMI need to write the patch20:49
EmilienMayoung: it's actually not easy20:50
EmilienMayoung: keystone is not managed by Puppet on the undercloud20:50
EmilienM:(20:50
ayoungEmilienM, OK,  for now, let me try to complete the Undercloud as is with install from source20:50
egaffordayoung: I think, tragically, that you might have to respin your instack-virt-setup to get the puppet module in there properly; the puppet modules are packed into the undercloud node by DIB and then spawned with virsh. You may have better luck just grabbing master of the aodh module and shoving it into your undercloud node directly.20:51
EmilienMslagle: where is init-keystone ?20:52
EmilienMoh nevermind20:53
ayoungegafford, what do you mean by "grabbing master of the aodh module and shoving it into your undercloud node directly."20:53
*** dshulyak_ has quit IRC20:53
*** panda_ has quit IRC20:53
*** akrivoka|afk has quit IRC20:53
*** athomas has quit IRC20:53
egaffordayoung: Fun story: I am not totally sure. Work on the undercloud is newish to me.20:53
egaffordLet me investigate a bit.20:53
ayoungegafford, also, I suspect that I want to destroy the VMs and recreate.  Is that just a matter of using vish on the baremetal to delete the VMs and then re-run instack-virt-setup?20:54
egaffordYup, it's that.20:54
egafforddestroy and undefine the lot.20:54
egaffordHm; maybe you don't have to.20:55
EmilienMok I have a patch20:56
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: run keystone in a wsgi process  https://review.openstack.org/27047720:57
egaffordYou may wish to try export `DIB_INSTALLTYPE_puppet_modules=source` and running `tripleo.sh --undercloud` within your extant virt envs. You may also wish to spin all the way back down.20:57
EmilienMayoung: ^ I'll need to validate orchestration is good though20:57
egaffordI can't tell you without testing which stage is necessary.20:57
ayoungegafford, I realize I had never updated the base OS.  Doing that now20:57
egaffordBut from the looks of it, you *should* be able to make it work within your extant virt envs.20:57
egaffordAh, yeah, that's importnat.20:58
egaffords/that/spelling/20:58
*** weshay has joined #tripleo21:00
ayoungrerunning  instack-virt-setup21:00
ayoungegafford, OK, I've respun the instack-virt-setup.  trying the export ^^21:04
bnemecegafford: ayoung: As it happens, there is a way to deploy an undercloud with a proposed patch: http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#installing-the-undercloud21:11
bnemecSee the second source admonition under step 5.21:11
ayoungbnemec, that is exactly what I am right now doing.  Thanks21:12
bnemectripleo.sh already sets the puppet installtype to source because that's what we test in CI.21:12
ayoungbnemec, egafford ok,  undercloud completed without reported error21:16
bnemecAn auspicious start. :-)21:17
egaffordSuper auspicious.21:23
*** ayoung has quit IRC21:24
*** jayg is now known as jayg|g0n321:33
egaffordbnemec: https://review.openstack.org/#/c/221418/ has now passed each gate on at least one occasion. I can keep pressing the shiny, red, candy-like recheck button, but your CI lab has other things to do, and could instead just note that set-math-wise, the patch has passed its gates. Thoughts?21:33
bnemecegafford: Yeah, I'm fine with merging it, but we had agreed to hold off merging new patches until the ping test merges: https://review.openstack.org/#/c/262028/21:36
bnemecIn the interest of not breaking the overcloud again before we can get the testing enabled.21:36
egaffordbnemec: I respect that completely. :)21:37
egaffordI just wanted to make sure I shouldn't keep hitting the button.21:37
bnemecegafford: Unfortunately CI seems to be really flaky right now.  Very few patches are passing all three tests at once. :-(21:38
openstackgerritAndrew Bramley proposed openstack/tripleo-image-elements: Update location of get-pip.py  https://review.openstack.org/27049421:38
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Rename validate tox env to linters  https://review.openstack.org/25817321:40
egaffordbnemec: Yeah, I know. We have set math CI problems in Sahara sometimes too. Lots of moving parts.21:40
*** gfidente|afk has quit IRC21:40
bnemecEngine went down during stack CREATE21:42
bnemecBleh, I think we're going to have to bump our undercloud size again.21:42
bnemecWe must have undone the benefits of the extra memory with the vcpu bump.21:43
*** jdob_lt has quit IRC21:45
*** jdob_lt has joined #tripleo21:45
*** penick has quit IRC21:46
*** panda_ has joined #tripleo21:48
*** davidlenwell has quit IRC21:52
*** davidlenwell has joined #tripleo21:53
*** ChanServ sets mode: +v davidlenwell21:53
openstackgerritMerged openstack/instack-undercloud: Allow setting the hostname during undercloud install  https://review.openstack.org/24945721:59
*** davidlenwell has quit IRC22:00
*** athomas has joined #tripleo22:01
*** akrivoka|afk has joined #tripleo22:01
*** davidlenwell has joined #tripleo22:02
*** ChanServ sets mode: +v davidlenwell22:02
*** penick has joined #tripleo22:05
*** jdob_lt has quit IRC22:10
*** athomas has quit IRC22:11
*** akrivoka|afk has quit IRC22:11
*** trozet has quit IRC22:11
*** trown|mtg is now known as trown|outttypeww22:11
*** athomas has joined #tripleo22:11
*** akrivoka|afk has joined #tripleo22:11
*** akrivoka|afk has quit IRC22:12
*** egafford has quit IRC22:14
*** egafford has joined #tripleo22:15
*** lblanchard has quit IRC22:17
*** ayoung has joined #tripleo22:20
*** athomas has quit IRC22:22
*** thrash is now known as thrash|g0ne22:22
*** tiswanso has quit IRC22:25
*** tiswanso has joined #tripleo22:26
*** tiswanso has quit IRC22:26
openstackgerritMerged openstack/diskimage-builder: Move speedup section to image building guide  https://review.openstack.org/26876922:27
*** weshay has quit IRC22:47
*** alop has joined #tripleo22:54
alopCan I get some eyes on https://review.openstack.org/#/c/266016/ please?22:55
*** cwolferh has quit IRC22:59
bnemecThe CI test for overcloud ping test has passed CI.  We should merge it ASAP: https://review.openstack.org/#/c/262028/623:01
*** weshay has joined #tripleo23:01
*** regebro has quit IRC23:04
*** mordred has quit IRC23:04
bnemecalop: Does that actually work in RHEL 6?  It's not an accident that the repo name was hard-coded - rhel 6 has its own registration script: https://github.com/openstack/diskimage-builder/blob/master/elements/rhel-common/pre-install.d/00-rhsm23:05
*** regebro has joined #tripleo23:05
*** rasca has quit IRC23:05
bnemecI have no idea why they're both in rhel-common.23:05
alopbnemec: yes, if using Satellite for registration, you have to have the repos named appropriately23:06
*** mordred has joined #tripleo23:06
*** rasca has joined #tripleo23:06
alopotherwise, subscription-manager tries to enable rhel-7-server-rpms AND rhel-6-server-rpms, which causes it to fail23:07
bnemecalop: Right, I guess my concern is that the script was never supposed to work on rhel-6.  That distro is supposed to use https://github.com/openstack/diskimage-builder/blob/master/elements/rhel-common/pre-install.d/00-rhsm23:08
bnemecAnd since that script happens to be updated to support rhel7 too, if we then add support to the new one for rhel6 there will essentially be two completely separate methods of registering both distro version.23:09
bnemec*versions23:09
alopI don't think the rhel element actually uses 00-rhsm, or at least, it doesn't NOT use 00-rhel-registration23:12
alopat the minimum, introducing the RHEL_MAJ_VER in the environment.d phase would allow us to further clean up all the 6's and 7's23:14
*** trozet has joined #tripleo23:23
openstackgerritAndrew Bramley proposed openstack/tripleo-image-elements: Pin pip to 7.1.2 in heat-cfntools elements  https://review.openstack.org/27052623:25
*** eil397 has joined #tripleo23:26
*** nico_auv has quit IRC23:38
*** athomas has joined #tripleo23:41
*** trozet has quit IRC23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!