| sshnaidm | bnemec, but I see there: -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml so it should be ok.. | 00:00 |
|---|---|---|
| sshnaidm | bnemec, but I don't get something, look, in the same job: | 00:01 |
| bnemec | sshnaidm: Yeah, but it broke the pre-newton configs because they don't have that env. | 00:01 |
| sshnaidm | bnemec, http://logs.openstack.org/72/397972/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-newton/56af1c8/console.html#_2016-11-16_20_10_44_067436 and http://logs.openstack.org/72/397972/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-newton/56af1c8/console.html#_2016-11-16_20_10_44_346537 | 00:01 |
| bnemec | So the only service getting configured properly was heat-engine | 00:01 |
| bnemec | sshnaidm: Yes, we set them correctly in deploy.sh, then they get overwritten in tripleo.sh. | 00:02 |
| bnemec | See http://logs.openstack.org/72/397972/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-newton/56af1c8/console.html#_2016-11-16_20_10_44_346537 | 00:02 |
| *** absubram has quit IRC | 00:02 | |
| *** owalsh has joined #tripleo | 00:02 | |
| *** dmarlin has quit IRC | 00:03 | |
| sshnaidm | bnemec, hmm, I mentioned this a some time ago, thought it's not an issue.. | 00:04 |
| sshnaidm | panda|Zz, made a genius patch to print these parameters in triple.sh itself.. | 00:05 |
| sshnaidm | bnemec, FYI https://bugs.launchpad.net/tripleo/+bug/1642429 | 00:08 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Undecided,New] | 00:08 |
| sshnaidm | bnemec, there's a good news, btw | 00:09 |
| *** mhenkel has quit IRC | 00:09 | |
| sshnaidm | bnemec, all this time we didn't use low-memory settings are were good! | 00:09 |
| bnemec | sshnaidm: That's because our nodes only had one vcpu. I actually wonder why we ever needed a special config for ci now. | 00:09 |
| bnemec | Then we bumped them to 8 cpus today and things went sideways. | 00:10 |
| bnemec | sshnaidm: Do you have a patch in progress for this? | 00:10 |
| *** achadha_ has joined #tripleo | 00:10 | |
| sshnaidm | bnemec, will try now to change deploy.env path | 00:11 |
| *** achadha has quit IRC | 00:14 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Source deploy.env on multinode only https://review.openstack.org/398650 | 00:15 |
| sshnaidm | bnemec, ^^ | 00:15 |
| *** achadha_ has quit IRC | 00:15 | |
| sshnaidm | bnemec, but I'll check it only tomorrow.. | 00:15 |
| bnemec | sshnaidm: lgtm, can you add a closes-bug? | 00:16 |
| sshnaidm | bnemec, yeah, although it doesn't link it properly.. | 00:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Source deploy.env on multinode only https://review.openstack.org/398650 | 00:17 |
| sshnaidm | bnemec, thanks for help, see you tomorrow! | 00:18 |
| bnemec | sshnaidm: Yep, np | 00:18 |
| *** sshnaidm is now known as sshnaidm|away | 00:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Restore worker configs for mitaka and below https://review.openstack.org/398652 | 00:24 |
| *** owalsh_ has joined #tripleo | 00:31 | |
| *** owalsh has quit IRC | 00:31 | |
| *** achadha has joined #tripleo | 00:32 | |
| *** david-lyle_ is now known as david-lyle | 00:35 | |
| *** rajinir has quit IRC | 00:36 | |
| *** limao has joined #tripleo | 00:37 | |
| *** bfournie has quit IRC | 00:50 | |
| *** chem has quit IRC | 00:51 | |
| *** lblanchard has joined #tripleo | 01:03 | |
| *** florianf|afk has quit IRC | 01:09 | |
| achadha | For an all-in-one (virtualized) TripleO deployment, is there any environment-variable that we can use to create multiple NICs on the overcloud VMs? (export NODE_NIC=2 ?) | 01:16 |
| *** michapma_alt has joined #tripleo | 01:18 | |
| *** dsneddon_ has quit IRC | 01:19 | |
| *** bfournie has joined #tripleo | 01:21 | |
| *** florianf|afk has joined #tripleo | 01:24 | |
| *** dmacpher-afk has quit IRC | 01:24 | |
| *** jeckersb_gone is now known as jeckersb | 01:45 | |
| *** tiswanso has joined #tripleo | 01:48 | |
| openstackgerrit | Merged openstack/instack-undercloud: Add option to not update packages during undercloud install https://review.openstack.org/391473 | 01:50 |
| *** tiswanso has quit IRC | 01:53 | |
| *** tiswanso has joined #tripleo | 01:53 | |
| *** ctayal has joined #tripleo | 01:54 | |
| openstackgerrit | zhangyanxian proposed openstack/tripleo-image-elements: Fix typos in rootwrap.conf https://review.openstack.org/373922 | 02:04 |
| openstackgerrit | zhangyanxian proposed openstack/tripleo-image-elements: Fix typos in README.md & rootwrap.conf https://review.openstack.org/373922 | 02:11 |
| *** dmacpher has joined #tripleo | 02:11 | |
| *** cwolferh has quit IRC | 02:20 | |
| *** ctayal has quit IRC | 02:23 | |
| *** tzumainn has quit IRC | 02:30 | |
| *** achadha_ has joined #tripleo | 02:31 | |
| *** achadha has quit IRC | 02:33 | |
| *** achadha_ has quit IRC | 02:35 | |
| *** achadha has joined #tripleo | 02:46 | |
| *** achadha has quit IRC | 02:47 | |
| *** fragatin_ has joined #tripleo | 02:47 | |
| *** achadha has joined #tripleo | 02:47 | |
| *** achadha has quit IRC | 02:48 | |
| *** achadha has joined #tripleo | 02:48 | |
| *** achadha has quit IRC | 02:50 | |
| *** fragatina has quit IRC | 02:50 | |
| *** fragatin_ has quit IRC | 02:51 | |
| *** newmember has quit IRC | 02:54 | |
| *** fragatina has joined #tripleo | 02:57 | |
| *** fragatina has quit IRC | 02:58 | |
| openstackgerrit | chenyingnan proposed openstack/instack: Fix "wrap functions with 2 blank lines" pep8 check https://review.openstack.org/398712 | 03:02 |
| *** cwolferh has joined #tripleo | 03:02 | |
| *** bkopilov has quit IRC | 03:14 | |
| *** coolsvap has joined #tripleo | 03:15 | |
| *** lblanchard has quit IRC | 03:21 | |
| *** fragatina has joined #tripleo | 03:24 | |
| *** achadha has joined #tripleo | 03:24 | |
| *** achadha has quit IRC | 03:28 | |
| *** fragatina has quit IRC | 03:29 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore https://review.openstack.org/365449 | 03:30 |
| *** achadha has joined #tripleo | 03:30 | |
| *** achadha has quit IRC | 03:35 | |
| *** rlandy has quit IRC | 03:38 | |
| *** udesale has joined #tripleo | 03:39 | |
| *** achadha has joined #tripleo | 03:45 | |
| *** achadha has quit IRC | 03:49 | |
| *** kjw3 has joined #tripleo | 03:56 | |
| *** yamahata has quit IRC | 04:01 | |
| *** bana_k has joined #tripleo | 04:12 | |
| *** achadha has joined #tripleo | 04:13 | |
| *** cwolferh has quit IRC | 04:13 | |
| *** bkopilov has joined #tripleo | 04:33 | |
| *** thrash|g0ne has quit IRC | 04:51 | |
| *** thrash has joined #tripleo | 04:51 | |
| *** thrash has quit IRC | 04:51 | |
| *** thrash has joined #tripleo | 04:51 | |
| *** I has joined #tripleo | 04:55 | |
| *** I is now known as Guest28272 | 04:55 | |
| *** Guest28272 has quit IRC | 05:01 | |
| *** bana_k has quit IRC | 05:02 | |
| *** masco has joined #tripleo | 05:05 | |
| *** owalsh_ has quit IRC | 05:10 | |
| *** owalsh_ has joined #tripleo | 05:11 | |
| *** bana_k has joined #tripleo | 05:17 | |
| *** charliejllewelly has joined #tripleo | 05:32 | |
| *** chandankumar has joined #tripleo | 05:34 | |
| *** prateek has joined #tripleo | 05:35 | |
| *** bana_k has quit IRC | 05:40 | |
| *** numans has joined #tripleo | 05:42 | |
| *** yamahata has joined #tripleo | 05:47 | |
| *** fragatina has joined #tripleo | 05:47 | |
| *** charliejllewelly has quit IRC | 05:49 | |
| *** kjw3 has quit IRC | 05:51 | |
| *** fragatina has quit IRC | 05:52 | |
| *** jbadiapa has quit IRC | 05:59 | |
| *** owalsh_ has quit IRC | 06:00 | |
| *** owalsh_ has joined #tripleo | 06:00 | |
| *** limao has quit IRC | 06:06 | |
| *** limao has joined #tripleo | 06:08 | |
| *** jaosorior has joined #tripleo | 06:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use encrypted volume in scenario004 https://review.openstack.org/398221 | 06:21 |
| *** abregman has joined #tripleo | 06:23 | |
| *** cwolferh has joined #tripleo | 06:29 | |
| jaosorior | clear | 06:33 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-specs: Real-time compute nodes https://review.openstack.org/388162 | 06:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Configure Keystone Fernet Keys https://review.openstack.org/397350 | 06:38 |
| *** dsariel has joined #tripleo | 06:38 | |
| *** anshul has joined #tripleo | 06:38 | |
| *** jaosorior has quit IRC | 06:41 | |
| *** jaosorior has joined #tripleo | 06:42 | |
| *** lmiccini has joined #tripleo | 06:44 | |
| *** fragatina has joined #tripleo | 06:49 | |
| *** achadha_ has joined #tripleo | 06:50 | |
| openstackgerrit | Ade Lee proposed openstack/tripleo-quickstart: Add support to set DNS server on the undercloud https://review.openstack.org/398771 | 06:52 |
| openstackgerrit | Ade Lee proposed openstack/tripleo-quickstart: changes to add novajoin to undercloud https://review.openstack.org/398772 | 06:52 |
| *** achadha has quit IRC | 06:53 | |
| *** pgadiya has joined #tripleo | 06:54 | |
| *** fragatina has quit IRC | 06:54 | |
| *** oshvartz has joined #tripleo | 06:56 | |
| *** owalsh_ has quit IRC | 07:02 | |
| *** iranzo has joined #tripleo | 07:03 | |
| *** iranzo has joined #tripleo | 07:03 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 07:04 |
| *** florianf|afk is now known as florianf | 07:13 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 07:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Fix unbound variable in bootstrap-overcloud-full.sh https://review.openstack.org/398782 | 07:15 |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 07:16 |
| *** oshvartz has quit IRC | 07:18 | |
| *** pcaruana has joined #tripleo | 07:18 | |
| *** owalsh_ has joined #tripleo | 07:23 | |
| *** rasca has joined #tripleo | 07:24 | |
| *** achadha_ has quit IRC | 07:25 | |
| *** owalsh_ has quit IRC | 07:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Configure Keystone Fernet Keys https://review.openstack.org/397350 | 07:29 |
| *** kjw3 has joined #tripleo | 07:30 | |
| openstackgerrit | Babu Shanmugam proposed openstack/tripleo-heat-templates: Split OVN northd and ml2 plugin https://review.openstack.org/387940 | 07:31 |
| *** jbadiapa has joined #tripleo | 07:34 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add necessary parameters for encrypted volumes support https://review.openstack.org/398177 | 07:34 |
| *** b00tcat has joined #tripleo | 07:35 | |
| openstackgerrit | Babu Shanmugam proposed openstack/tripleo-heat-templates: OVN plugin configuration fixes https://review.openstack.org/397674 | 07:36 |
| *** leanderthal|afk is now known as leanderthal | 07:39 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 07:39 |
| *** tobias_fiberdata has joined #tripleo | 07:47 | |
| *** ebarrera has joined #tripleo | 07:47 | |
| *** athomas has joined #tripleo | 07:57 | |
| *** dmacpher has quit IRC | 07:59 | |
| *** ealcaniz has joined #tripleo | 07:59 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Deploy heat APIs over httpd https://review.openstack.org/394837 | 08:00 |
| jaosorior | sshnaidm|away: ping | 08:02 |
| *** sshnaidm|away is now known as sshnaidm | 08:02 | |
| sshnaidm | jaosorior, hi | 08:02 |
| *** jprovazn has joined #tripleo | 08:02 | |
| jaosorior | sshnaidm: seems that the multinode job is brokenf or stable branches, trying to fix it here https://review.openstack.org/#/c/398782/ | 08:02 |
| jaosorior | sshnaidm: for instance, the CR that you rechecked failed with this http://logs.openstack.org/52/398652/1/check/gate-tripleo-ci-centos-7-nonha-multinode/50ae95d/console.html#_2016-11-17_06_52_49_055350 | 08:03 |
| *** chem has joined #tripleo | 08:03 | |
| sshnaidm | jaosorior, I see.. | 08:06 |
| sshnaidm | jaosorior, if we anyway source deploy.env in each start of tripleo.sh, why stable_release is unbound then | 08:06 |
| openstackgerrit | Merged openstack/puppet-tripleo: Sort parameters in keystone profile alphabetically https://review.openstack.org/398323 | 08:07 |
| jaosorior | sshnaidm: cause I messed up and added it in the wrong place | 08:09 |
| jaosorior | wait up | 08:09 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Fix unbound variable in bootstrap-overcloud-full.sh https://review.openstack.org/398782 | 08:10 |
| jaosorior | sshnaidm: there ^^ | 08:10 |
| *** chem has quit IRC | 08:11 | |
| jaosorior | sshnaidm: so the deal is that, even if we source that file in tripleo.sh, we don't in bootstrap-overcloud-full.sh | 08:12 |
| sshnaidm | jaosorior, omg, I see | 08:13 |
| jaosorior | anyway, that should do it | 08:13 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:14 |
| sshnaidm | jaosorior, yeah, we need now another core to push it | 08:14 |
| *** shardy_afk is now known as shardy | 08:15 | |
| jaosorior | sshnaidm: meanwhile you could rebase your patch on top of that one | 08:15 |
| sshnaidm | jaosorior, sure | 08:16 |
| *** chem has joined #tripleo | 08:16 | |
| ccamacho | morning folks! | 08:18 |
| jaosorior | ccamacho: sup dude, how's it going? | 08:18 |
| jaosorior | shardy: no bug reference, just saw the bug happening in the morning and went on and tried to fix it up | 08:19 |
| ccamacho | jaosorior :) awesome! | 08:19 |
| ccamacho | jaosorior, you? | 08:19 |
| jaosorior | ccamacho: pretty good, a bit tired, but coffee will make it aaaall better | 08:20 |
| ccamacho | coffee++ | 08:20 |
| jaosorior | shardy, ccamacho any idea on how can I get more info about what's causing this? http://logs.openstack.org/50/397350/3/check/gate-tripleo-ci-centos-7-nonha-multinode/5ed8872/console.html#_2016-11-17_08_13_35_515351 the patch is this one https://review.openstack.org/#/c/397350/3 | 08:22 |
| shardy | jaosorior: s/Keys/Key | 08:24 |
| shardy | commented on the patch | 08:24 |
| shardy | the wrong parameter isn't surfaced due to a bug in heat outputs validation | 08:24 |
| shardy | which has fixes posted but not yet landed | 08:24 |
| shardy | https://bugs.launchpad.net/heat/+bug/1599114 | 08:25 |
| openstack | Launchpad bug 1599114 in heat "Outputs aren't correctly validated" [Medium,In progress] - Assigned to Oleksii Chuprykov (ochuprykov) | 08:25 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Configure Keystone Fernet Keys https://review.openstack.org/397350 | 08:25 |
| jaosorior | shardy: thanks dude! | 08:26 |
| *** oshvartz has joined #tripleo | 08:26 | |
| *** jroll has quit IRC | 08:27 | |
| *** abregman has quit IRC | 08:28 | |
| *** abregman has joined #tripleo | 08:30 | |
| jaosorior | what's up with the ovb gate? O_o | 08:32 |
| openstackgerrit | Adriano Petrich proposed openstack/python-tripleoclient: Use stack name or id for backwards compatibility https://review.openstack.org/398289 | 08:33 |
| sshnaidm | jaosorior, looking.. | 08:34 |
| openstackgerrit | Adriano Petrich proposed openstack/python-tripleoclient: Give better output on scale failures https://review.openstack.org/398226 | 08:36 |
| *** cylopez has joined #tripleo | 08:36 | |
| *** abregman has quit IRC | 08:40 | |
| *** jroll has joined #tripleo | 08:41 | |
| *** amoralej|off is now known as amoralej | 08:44 | |
| *** liverpooler has joined #tripleo | 08:46 | |
| *** anshul has quit IRC | 08:46 | |
| *** jpich has joined #tripleo | 08:49 | |
| *** abregman has joined #tripleo | 08:53 | |
| *** hewbrocca_afk is now known as hewbrocca | 08:53 | |
| yolanda | hi, good morning. I had to rebase tripleo changes, so i lost +2. Can i get reviews again on https://review.openstack.org/394426 ? | 08:56 |
| openstackgerrit | Merged openstack/instack-undercloud: Use same logging format for file and stream https://review.openstack.org/377813 | 08:56 |
| openstackgerrit | Merged openstack/instack-undercloud: Clean up validation error message https://review.openstack.org/377814 | 08:56 |
| *** mhenkel has joined #tripleo | 08:56 | |
| shardy | yolanda: good morning, done! | 08:57 |
| yolanda | hi shardy, can you take a look at this as well? https://review.openstack.org/397075 | 08:58 |
| yolanda | it's related, including documentation | 08:58 |
| *** lucas-afk is now known as lucasagomes | 08:59 | |
| panda|Zz | sshnaidm: I think increasing the quota led to collateral effect yesterday | 09:01 |
| *** panda|Zz is now known as panda | 09:01 | |
| hewbrocca | bandini: hey regarding eck's oslo.messaging fix | 09:02 |
| hewbrocca | apevec seems to think we need to get oslo.messaging to do a stable release | 09:03 |
| hewbrocca | a stable/newton release | 09:03 |
| bandini | hewbrocca: yeah. I spoke to eck about it. He will do the magic today | 09:03 |
| hewbrocca | sweet | 09:03 |
| hewbrocca | I really wonder how many of our crazy performance problems this time around are caused by this one bug | 09:03 |
| bandini | I wonder how much of that is actually causing our CI issues | 09:03 |
| bandini | yeah exactly | 09:03 |
| hewbrocca | we should not ship without it, I'm sure of that | 09:04 |
| jaosorior | which oslo.messaging issue? | 09:04 |
| bandini | yeah we have it downstream | 09:04 |
| bandini | jaosorior: https://review.openstack.org/#/c/394963/ | 09:04 |
| hewbrocca | Oh, it's in the product already? | 09:04 |
| bandini | hewbrocca: downstream yes | 09:04 |
| *** anshul has joined #tripleo | 09:04 | |
| hewbrocca | I didn't think we were doing that any more :) | 09:04 |
| bandini | hewbrocca: I am as confused as you tbh | 09:04 |
| jaosorior | daaaamn | 09:05 |
| bandini | jaosorior: yep yep and yep | 09:05 |
| hewbrocca | it's a really nasty bug | 09:06 |
| bandini | yeah CPU will spike up even when services are idle | 09:07 |
| hewbrocca | causes your machines to be ridiculously busy even though there's no load showing up on e.g. top | 09:07 |
| *** gfidente has joined #tripleo | 09:07 | |
| *** gfidente has quit IRC | 09:07 | |
| *** gfidente has joined #tripleo | 09:07 | |
| shardy | Oh wow yeah that's a nasty one | 09:07 |
| *** jaosorior is now known as jaosorior_lunch | 09:08 | |
| *** hjensas has joined #tripleo | 09:08 | |
| *** hjensas has quit IRC | 09:08 | |
| *** hjensas has joined #tripleo | 09:08 | |
| bandini | shardy: I spoke to eck and today he will try and get a new oslo messaging release for newton so we can pick it up in rdo asap | 09:09 |
| shardy | bandini: we also need a current-tripleo promotion I think, as it (along with RDO) hasn't promoted in 13 days :( | 09:10 |
| shardy | https://dashboards.rdoproject.org/rdo-dev | 09:10 |
| shardy | we could consider temporarily pulling oslo.messaging from master/current if it'd not adding a bunch of deps | 09:10 |
| bandini | shardy: I see. Where can I find more infos about the 7issues mentioned in the tripleo pin packages? | 09:11 |
| *** gfidente has quit IRC | 09:11 | |
| bandini | shardy: yeah that's also an option | 09:11 |
| shardy | If you click on the "7 issues" it takes you to https://etherpad.openstack.org/p/tripleo-ci-status | 09:11 |
| shardy | sshnaidm may also have more issues re the periodic job problems | 09:12 |
| shardy | s/issues/info | 09:12 |
| *** fragatina has joined #tripleo | 09:12 | |
| * bandini looks | 09:13 | |
| b00tcat | Hi, I'm having issues when creating a *-base service due to duplicated parameters | 09:16 |
| *** gfidente has joined #tripleo | 09:16 | |
| *** gfidente has quit IRC | 09:16 | |
| *** gfidente has joined #tripleo | 09:16 | |
| b00tcat | I have these two files: http://paste.fedoraproject.org/483646/93740751/ | 09:16 |
| b00tcat | but when I include the 2nd yaml in my roles-data.yaml the overcloud process finishes with a rather cryptic message | 09:16 |
| panda | sshnaidm: you have access to the ovb infrastructure ? | 09:17 |
| sshnaidm | panda, yes | 09:17 |
| *** yamahata has quit IRC | 09:17 | |
| *** fragatina has quit IRC | 09:18 | |
| gfidente | b00tcat these two | 09:18 |
| gfidente | midonet::cluster::keystone_admin_token: '"%{hiera(''keystone::admin_token'')}"' | 09:18 |
| gfidente | midonet::cluster::keystone_host: {get_param: [EndpointMap, KeystoneInternal, uri]} | 09:18 |
| gfidente | go below the 'config_settings:' section | 09:18 |
| gfidente | not at the same level | 09:18 |
| sshnaidm | jaosorior_lunch, panda Forbidden: Quota exceeded for ram: Requested 8192, but already used 2999296 of 3000000 ram (HTTP 403) (Request-ID: req-1595ae32-966f-4f9b-88b8-270965d9622b) | 09:18 |
| shardy | b00tcat: yeah and you need to use a map_merge to combine those with the get_attr: [MidonetBase, role_data, config_settings] | 09:18 |
| yolanda | nice, my change got 2 +2s... is something else needed to be merged? | 09:19 |
| b00tcat | I took the example from https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/aodh-listener.yaml as putting them underneath config settings with a map_merge didn't do the trick hm | 09:19 |
| sshnaidm | we're outta memory | 09:19 |
| gfidente | b00tcat also you can get the admin token as param | 09:19 |
| gfidente | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/ceph-rgw.yaml#L56 | 09:19 |
| shardy | b00tcat: that example isn't the same, as it's not merging anything into config_settings | 09:19 |
| gfidente | b00tcat instead of using a hiera call, which would only work if keystone is deployed on the same node where the midonet plugin goes | 09:20 |
| gfidente | you have no guarantee it will be there instead | 09:20 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/heat-api.yaml#L60 | 09:20 |
| shardy | b00tcat: ^^ you need to do the map_merge like that | 09:20 |
| shardy | gfidente: Yeah I already commented about that on the review - tbh I don't think we should use admin token at all | 09:20 |
| shardy | it's deprecated by keystone, and pretty soon we're going to remove it completely | 09:21 |
| gfidente | shardy yeah though rgw needs that until it supports v3 | 09:21 |
| panda | sshnaidm: no, yesterdat bnemec increased RAM and CPU for all the nodes, but he had problems with the quotas. He set them in two different places and thought he solved it | 09:21 |
| shardy | but yeah, if we have to, use get_param: AdminToken | 09:21 |
| panda | sshnaidm: he didnt' :( | 09:21 |
| b00tcat | gfidente: isn't *all* hieradata shared amongst all nodes? | 09:21 |
| gfidente | shardy I think for ocata we can do rgw with keystone v3 and it won't need the token anymore, regarding the midonet plugin less sure | 09:21 |
| sshnaidm | panda, it's memory of the tenant, not of the node | 09:21 |
| gfidente | b00tcat oh not anymore not | 09:21 |
| b00tcat | ow this is new | 09:22 |
| gfidente | that is the supercool refactoring to have composable services done in newton | 09:22 |
| b00tcat | I'll check all these links then thanks! | 09:22 |
| sshnaidm | panda, were these problems with quotas for tenant? | 09:22 |
| panda | sshnaidm: which tenant. I meant he increased the requested RAM for all the node created in OVS stack creation, so he ha d to increase the quota too | 09:23 |
| gfidente | shardy I actually tried to switch rgw to not use admin token https://review.openstack.org/#/c/389373/ https://review.openstack.org/#/c/389372/ https://review.openstack.org/#/c/389355/ | 09:24 |
| panda | sshnaidm: effectively bumping the requirements for all the deployments | 09:24 |
| gfidente | shardy but things were not ready in rgw | 09:24 |
| sshnaidm | panda, I see.. | 09:25 |
| sshnaidm | panda, do you know what is the quota he tried to set? | 09:26 |
| *** jpena|off is now known as jpena | 09:27 | |
| sshnaidm | panda, as I see the quotas are set for 3000000 memory, but acc. to logs it's not enough | 09:29 |
| sshnaidm | maybe the calculation was wrong | 09:29 |
| shardy | gfidente: ack - can you raise a bug please with details of what needs to be fixed? | 09:31 |
| shardy | gfidente: it'd be cool if we could figure out the steps to move to keystone-manage bootstrap this cycle, even if we're not quite ready to do it yet | 09:31 |
| panda | sshnaidm: 3000000 | 09:31 |
| shardy | definitely not adding more things that depend on it would be a good idea if possible :) | 09:32 |
| gfidente | shardy right :) | 09:33 |
| *** kjw3 has quit IRC | 09:35 | |
| *** dbecker has joined #tripleo | 09:35 | |
| sshnaidm | panda, yeah, that's the quota on ovb nwo | 09:35 |
| *** chandankumar has quit IRC | 09:36 | |
| sshnaidm | panda, but Requested 8192, but already used 2999296 of 3000000 ram | 09:36 |
| panda | sshnaidm: we the quota really need to be bumped more | 09:36 |
| panda | s/we/so | 09:36 |
| *** chandankumar has joined #tripleo | 09:37 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 09:37 |
| sshnaidm | either node memory decreased | 09:37 |
| panda | sshnaidm: this was done to try to solve all the memory and timeout problems we're experiencing lately | 09:40 |
| sshnaidm | panda, I can set quotas, but I need to know which value to.. | 09:41 |
| *** jaosorior_lunch is now known as jaosorior | 09:43 | |
| *** dsariel has quit IRC | 09:44 | |
| *** anshul has quit IRC | 09:45 | |
| panda | sshnaidm: 3000000 is 3T right ? bumped from 2.5T yesterday. How many jobs are failing ? | 09:45 |
| jaosorior | panda: I haven't seen many pass in the last couple of hours | 09:46 |
| panda | how's it possible ? who's eating all the ram if none of the jobs are passing ? | 09:47 |
| *** openstackgerrit has quit IRC | 09:48 | |
| *** openstackgerrit has joined #tripleo | 09:49 | |
| sshnaidm | I see about 50 stacks failed to create today | 09:49 |
| panda | sshnaidm: out of ? | 09:49 |
| sshnaidm | running 40 | 09:53 |
| *** tremble has joined #tripleo | 09:57 | |
| *** derekh has joined #tripleo | 09:59 | |
| *** jbadiapa has quit IRC | 09:59 | |
| panda | sshnaidm: ha is 4 + undercloud = 8Gx4 + 8G = 40G x 40 = 1.6T .. well below quota... maybe some cleanup issues ? | 10:01 |
| *** anshul has joined #tripleo | 10:01 | |
| *** fzdarsky has joined #tripleo | 10:03 | |
| sshnaidm | panda, not that I'm aware of.. | 10:03 |
| fzdarsky | pong | 10:03 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Configure Keystone Fernet Keys https://review.openstack.org/397350 | 10:03 |
| panda | derekh: you remember what bnemec tried to do yesterday ? OVB is failing to create stacks on most of the jobs today. looks like we are overquota but based on the number of running jobs we should be ok | 10:05 |
| derekh | panda: looking | 10:06 |
| *** hewbrocca is now known as hewbrocca_afk | 10:09 | |
| *** paramite has joined #tripleo | 10:12 | |
| *** jbadiapa has joined #tripleo | 10:12 | |
| *** apevec has joined #tripleo | 10:12 | |
| apevec | shardy, hi, https://review.openstack.org/397959 is ready to merge (CI blocker) | 10:13 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: TEST FERNET TOKENS https://review.openstack.org/398897 | 10:13 |
| gfidente | apevec +w | 10:15 |
| apevec | thanks! | 10:15 |
| gfidente | apevec sure only I wonder if we're tracking we need to add it back somewhere? | 10:15 |
| gfidente | maybe we should put up a submission to add it back and wait for it to pass CI? | 10:16 |
| *** shardy is now known as shardy_mtg | 10:16 | |
| apevec | gfidente, good idea | 10:16 |
| apevec | bnemec suggested different approach in the review | 10:16 |
| apevec | but yeah, first step would be to keep simple revert around | 10:16 |
| gfidente | yeah it's not cool to include it in every image | 10:17 |
| derekh | panda: nova-api.log:2016-11-17 09:07:07.602 6193 INFO nova.api.openstack.wsgi [req-1b449ace-be79-40d7-8318-6b05d693f21f ba119eef29ce49f5b8697f4d63948e3c b79291658f384b7ebbc9019b6349e5c9 - - -] HTTP exception thrown: Quota exceeded for ram: Requested 8192, but already used 2999808 of 3000000 ram | 10:17 |
| gfidente | it can't work like that for all vendor plugins | 10:17 |
| derekh | panda: log entry about quota was over an hour ago | 10:17 |
| derekh | panda: *last log entry, did ye change something in the last hour? maybe the system just got less loaded | 10:18 |
| jpich | honza: Morning Honza! Do you think you could post a WIP patch with the developer docs you've been working on? Even if incomplete I think it'd be useful (to me anyway!!), d0ugal did that for the Mistral docs yonks ago ( https://review.openstack.org/#/c/358685/ ) and it's been incredibly valuable | 10:18 |
| *** nyechiel has joined #tripleo | 10:19 | |
| openstackgerrit | Merged openstack/os-net-config: Add support for name replacement in OVS_EXTRA https://review.openstack.org/398242 | 10:22 |
| openstackgerrit | Merged openstack/os-net-config: Add ovs_fail_mode option for OVS bridges https://review.openstack.org/398245 | 10:22 |
| *** limao has quit IRC | 10:24 | |
| panda | mh, we don't have statistics from the last hour ... is anyone else having problems with gates in the last hour ? | 10:24 |
| d0ugal | panda: I think jpich seen a number of failures this morning | 10:27 |
| d0ugal | not sure about the last hour, specifically | 10:27 |
| jpich | d0ugal, panda: Those sad results were from last night, not sure about the last hour specifically | 10:28 |
| sshnaidm | derekh, it's just failed jobs and freed some memory | 10:30 |
| sshnaidm | derekh, but I don't understand where is all memory, there were about 50-60 stacks, each one 40GB, still can not reach 3T | 10:31 |
| panda | do we have the current memory utilization ? | 10:31 |
| derekh | sshnaidm: nodepool is configured to max out at 60 jobs | 10:31 |
| derekh | 60 * 6 * 8192 = 2949120 | 10:32 |
| sshnaidm | derekh, why 6? | 10:32 |
| *** pblaho has joined #tripleo | 10:32 | |
| sshnaidm | derekh, it's 5 + bmc, isn't it? | 10:32 |
| derekh | + te-server + mirror + a couple of others | 10:32 |
| derekh | sshnaidm: because of the +1 here - https://review.openstack.org/#/c/111011/81/toci_gate_test.sh | 10:32 |
| sshnaidm | derekh, yeah, I counted it | 10:33 |
| sshnaidm | it's 4+1 | 10:33 |
| derekh | sshnaidm: undercloud + 3xcontroller + compute + spare | 10:33 |
| *** dsariel has joined #tripleo | 10:33 | |
| derekh | + bmc | 10:33 |
| openstackgerrit | Merged openstack/python-tripleoclient: Format the nodes list in openstack overcloud delete node https://review.openstack.org/395083 | 10:34 |
| sshnaidm | derekh, ouch, I forgot we spend also on undercloud, sorry | 10:34 |
| derekh | (6 * 812 ) + .5 | 10:34 |
| *** akrivoka has joined #tripleo | 10:34 | |
| sshnaidm | derekh, right, so we need to increase quota | 10:34 |
| derekh | sshnaidm: Ya, probably a little | 10:35 |
| derekh | sshnaidm: I'm a bit surprised we hit the limit though, | 10:36 |
| derekh | sshnaidm: I'll bump it a bit more and see what happens | 10:37 |
| jaosorior | derekh: well, the tripleo-ci jobs do use a lot of ovb nodes | 10:37 |
| panda | well, we added 2G to 3 nodes per build.. it's 6G more per build, 6Gx50= 300G more. quota was bumper 500G. did we increase undecloud RAM too ? | 10:40 |
| panda | bumped | 10:40 |
| *** fragatina has joined #tripleo | 10:42 | |
| jaosorior | gfidente: could you check this out? https://review.openstack.org/#/c/387432/ | 10:43 |
| sshnaidm | panda, we request 5 nodes from ovb each time | 10:44 |
| derekh | panda: sshnaidm bnemec I'v increased the quota for nodepool too 3145728 | 10:44 |
| jaosorior | or marios ^^ | 10:44 |
| sshnaidm | derekh, thanks | 10:44 |
| sshnaidm | let's go wild then | 10:44 |
| panda | recheck everything! | 10:45 |
| marios | jaosorior: in a bit ack | 10:45 |
| panda | derekh: how did you get that number ? | 10:45 |
| derekh | I think that should cover us if we are maxed out in CI with 162 G for the utility VM's | 10:45 |
| derekh | panda: 1024 * 1024 * 3 | 10:45 |
| *** fragatina has quit IRC | 10:47 | |
| *** jtomasek has quit IRC | 10:47 | |
| *** leanderthal has quit IRC | 10:48 | |
| d0ugal | apetrich: this is conflicting now: https://review.openstack.org/#/c/398289/ | 10:59 |
| d0ugal | apevec: sorry :) | 10:59 |
| *** shardy_mtg is now known as shardy | 11:00 | |
| * shardy reads backscroll | 11:00 | |
| apevec | should I get out of this channel? :) | 11:00 |
| shardy | apevec: thanks for the heads-up re https://review.openstack.org/397959, looks like it's approved now :) | 11:01 |
| *** charliejllewelly has joined #tripleo | 11:03 | |
| apevec | shardy, yep, gfidente suggested immediate revert to keep it on radar while trying to find a proper solution for vendor plugins | 11:03 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Move calculation of neutron l3_ha into puppet profile https://review.openstack.org/398926 | 11:05 |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates: Enable enforce_password_check https://review.openstack.org/397755 | 11:06 |
| *** fragatina has joined #tripleo | 11:07 | |
| yolanda | hi, i'm looking at tripleo-ci jobs for diskimage-builder, i have some questions. Who is best person to ask? | 11:08 |
| *** arxcruz has joined #tripleo | 11:08 | |
| openstackgerrit | Merged openstack/tripleo-common: Remove python-networking-cisco from overcloud-full image https://review.openstack.org/397959 | 11:11 |
| *** fragatina has quit IRC | 11:12 | |
| *** kjw3 has joined #tripleo | 11:12 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove conditional for neutron l3_ha https://review.openstack.org/398934 | 11:13 |
| shardy | beagles: ^^ Hey interested in your feedback on that when you're around | 11:13 |
| shardy | Aiming to fix https://bugs.launchpad.net/tripleo/+bug/1629187 | 11:14 |
| openstack | Launchpad bug 1629187 in tripleo "auto_enable_l3_ha should not be derived from ControllerCount" [High,In progress] - Assigned to Steven Hardy (shardy) | 11:14 |
| shardy | bandini: ^^ also your view on that solution would be good | 11:14 |
| yolanda | shardy, who can i better ask for tripleo-ci issues? i'm looking specifically at dib jobs. I'm seeing the logs, and i'd say it's installing DIB from master, instead of picking the DIB ref for the change is going to be tested | 11:17 |
| shardy | yolanda: do you have a link to the patch? | 11:18 |
| yolanda | shardy, look at http://logs.openstack.org/61/375261/22/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/b607fc3/console.html#_2016-11-13_05_53_58_396774 | 11:18 |
| yolanda | i'm looking at the "Building diskimage-builder" entries, and shows clones from master | 11:19 |
| yolanda | i was very surprised when i saw that patch passing tripleo-ci ... | 11:19 |
| bandini | shardy: looking | 11:22 |
| yolanda | shardy, also image build logs do not resemble to the expected output with that new patch, as it changes block device totally | 11:22 |
| *** hewbrocca_afk is now known as hewbrocca | 11:23 | |
| *** chandankumar has quit IRC | 11:24 | |
| hewbrocca | shardy: do we have a plan to address scaling/performance issues in Ocata at all? | 11:25 |
| hewbrocca | Not that we don't already have enough to do, but... | 11:25 |
| *** andreaf has quit IRC | 11:26 | |
| *** andreaf has joined #tripleo | 11:26 | |
| shardy | hewbrocca: the first I'd heard of them was yesterday, so not yet | 11:26 |
| shardy | we need to get someone access to a large environment to figure out why it's performing so much slower | 11:27 |
| hewbrocca | Yes -- or a large OVB cloud | 11:27 |
| shardy | yup, basically need more data | 11:27 |
| shardy | I'm not currently clear if the results were from newton-3 or final newton GA tho | 11:27 |
| hewbrocca | I have the sense that our performance is suffering overall for reasons we're not quite sure of | 11:27 |
| shardy | a bunch of performance related fixes happened very late in the cycle | 11:28 |
| hewbrocca | Right | 11:28 |
| hewbrocca | plus there's this oslo.messaging one that hasn't even been released on stable yet | 11:28 |
| derekh | hewbrocca: shardy how large a deployment do ye need ? /me is wondering if we could use rh1 or rh2 over a weekend, we could probably do a 60 node overcloud on rh1 right now (I kinda pulled that number out of the air but think its doable) | 11:31 |
| derekh | hewbrocca: shardy rh2 I meant rh2 | 11:31 |
| *** athomas has quit IRC | 11:32 | |
| d0ugal | florianf: Do you know who is interested in using the overcloudrc generation from Mistral? | 11:32 |
| apevec | hewbrocca, it's not released even on master | 11:34 |
| *** jbadiapa has quit IRC | 11:34 | |
| *** chandankumar has joined #tripleo | 11:34 | |
| shardy | derekh: anything with about 40 computes or more would be perfect | 11:35 |
| apevec | release requests are open now, I'll push them where I can | 11:35 |
| shardy | derekh: can you see why the patch mentioned by yolanda isn't delorean building from the ZUUL_CHANGES? | 11:36 |
| shardy | http://logs.openstack.org/61/375261/22/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/b607fc3/console.html#_2016-11-13_05_53_58_396774 | 11:36 |
| *** andreaf has quit IRC | 11:36 | |
| shardy | Processing diskimage-builder 587d14feed3cfdc6df9bf2d31898d5c81a377710 | 11:36 |
| yolanda | shardy, derekh, so the issue is.. tripleo-ci grabs project from zuul_changes, like: ZUUL_CHANGES=openstack/diskimage-builder:feature/v2:refs/changes/61/375261/22 . Then it just filters for project name (openstack/diskimage-builder). And clones ignoring the ref | 11:36 |
| shardy | which is https://github.com/openstack/diskimage-builder/commit/587d14feed3cfdc6df9bf2d31898d5c81a377710 | 11:36 |
| *** andreaf has joined #tripleo | 11:36 | |
| derekh | shardy: I'm trying to figure it out at the moment | 11:36 |
| shardy | derekh: ack, thanks | 11:37 |
| derekh | yolanda: it ignores the ref because it relies on zuul-cloner to have the repostory left at the correct HEAD (the one that should be tested) | 11:38 |
| derekh | yolanda: which I guess doesn't work here because the branch is feature/v2 | 11:38 |
| *** andreaf has quit IRC | 11:39 | |
| derekh | shardy: hewbrocca I'll see if I can get a 40 node overcloud up on rh2 | 11:40 |
| *** jkilpatr has quit IRC | 11:41 | |
| derekh | shardy: newton/rdo ? | 11:41 |
| hewbrocca | derekh: perfect | 11:41 |
| hewbrocca | apevec: thanks for that | 11:41 |
| yolanda | derekh, is not using a git clone? looking at scripts/tripleo.sh | 11:42 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Move the overcloudrc generation from tripleoclient to a Mistral action https://review.openstack.org/397211 | 11:42 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Move the overcloudrc generation from tripleoclient to a Mistral action https://review.openstack.org/397211 | 11:42 |
| *** andreaf has joined #tripleo | 11:43 | |
| yolanda | derekh, may we add diskimage-builder to $PROJECTS var, so zuul-cloner installs it? | 11:43 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Only start the deploy if the Heat stack isn't already in progress https://review.openstack.org/398959 | 11:44 |
| derekh | yolanda: yes, it should be there, | 11:44 |
| yolanda | let me submit a change | 11:44 |
| derekh | yolanda: it mush have been removed at some stage | 11:44 |
| derekh | *must | 11:44 |
| derekh | yolanda: we rely on it being there | 11:45 |
| derekh | yolanda: ack | 11:45 |
| yolanda | i don't see that on on tripleo-ci jobs, and looking at logs, zuul-cloner is not doing the job for it | 11:45 |
| derekh | yolanda: its in the job definition http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/tripleo.yaml#n159 | 11:46 |
| derekh | or should be | 11:46 |
| yolanda | is not there | 11:46 |
| yolanda | just adding it | 11:46 |
| derekh | which begs the question, what happened to it | 11:47 |
| openstackgerrit | Merged openstack/tripleo-docs: Add documentation on how to use an external Ceph cluster https://review.openstack.org/397825 | 11:48 |
| *** jpena is now known as jpena|lunch | 11:49 | |
| yolanda | yep, i'm looking at project-config history, but is so long, this repo is very active | 11:49 |
| yolanda | derekh https://review.openstack.org/398961 | 11:51 |
| openstackgerrit | Merged openstack/instack-undercloud: Stop pinning Glance API https://review.openstack.org/387432 | 11:52 |
| derekh | yolanda: I think this may have been it https://review.openstack.org/#/c/345818/ | 11:53 |
| derekh | shardy: ^ | 11:53 |
| yolanda | ah, so it was on default projects to clone | 11:55 |
| derekh | yolanda: yup, and I guess we have the same problem with dib-utils, want to update your patch? | 11:56 |
| yolanda | ++ | 11:56 |
| yolanda | updated | 11:58 |
| *** jkilpatr has joined #tripleo | 12:00 | |
| shardy | derekh: 40 node overcloud would be great - thanks! The complaints are about deployment time, so if you can loop me in while you're doing it it'd be good to see if we can figure out where the time is going, perhaps we can add some basic instrumentation | 12:04 |
| shardy | derekh: the complaint is that large overclouds take longer on newton than on mitaka | 12:05 |
| shardy | so I guess we start with newton trunk - if there's any easy way to automate deploying mitaka, then newton that would be even better | 12:05 |
| shardy | so we can compare | 12:05 |
| *** jbadiapa has joined #tripleo | 12:05 | |
| *** jkilpatr has quit IRC | 12:06 | |
| shardy | in theory we can load images for both versions into glance, then deploy with the appropriate tht on the same newton undercloud | 12:06 |
| shardy | although the problem may well be with the newton undercloud | 12:06 |
| shardy | that would still be a useful comparison | 12:06 |
| shardy | nice work spotting the dib project-config issue :) | 12:07 |
| *** fragatina has joined #tripleo | 12:08 | |
| *** panda is now known as panda|lunch | 12:10 | |
| *** charliejllewelly has quit IRC | 12:10 | |
| *** jkilpatr has joined #tripleo | 12:11 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Pass the plan name when tagging nodes https://review.openstack.org/398967 | 12:12 |
| *** fragatina has quit IRC | 12:13 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove conditional for neutron l3_ha https://review.openstack.org/398934 | 12:14 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Move calculation of neutron l3_ha into puppet profile https://review.openstack.org/398926 | 12:15 |
| derekh | shardy: ack, yup ye were talking about the same thread I thought, I'll let you know when I have something up | 12:15 |
| openstackgerrit | Carlos Camacho proposed openstack/instack-undercloud: Check if nested KVM is enabled on host. https://review.openstack.org/398969 | 12:15 |
| shardy | derekh: great, thanks! | 12:16 |
| *** karthiks has quit IRC | 12:18 | |
| openstackgerrit | Adriano Petrich proposed openstack/python-tripleoclient: Use stack name or id for backwards compatibility https://review.openstack.org/398289 | 12:18 |
| *** udesale has quit IRC | 12:18 | |
| *** jkilpatr has quit IRC | 12:19 | |
| openstackgerrit | Adriano Petrich proposed openstack/python-tripleoclient: Give better output on scale failures https://review.openstack.org/398226 | 12:19 |
| *** kaslcrof has joined #tripleo | 12:21 | |
| *** bkopilov has quit IRC | 12:22 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 12:25 |
| honza | jpich: I never actually got around to writing anything because of research into where these docs should go | 12:26 |
| jpich | honza: Oh :( | 12:26 |
| honza | jpich: what sort of thing are you urgently in need of? | 12:26 |
| florianf | d0ugal: interested as in "would the UI like to call that action"? | 12:26 |
| jpich | honza: tripleo-common has their docs in-tree so I think in-tree is fine | 12:27 |
| honza | jpich: i thought it should go with the tripleo-ui repo, or do you think we should shove it in tripleo-docs? | 12:27 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 12:28 |
| jpich | honza: I would have liked to familiarise myself with the conventions / tips and tricks / anything I'm not thinking of before attempting a patch, but now it's too late - I have a patch that works but prints ugly warnings in the console :-) So if I can humbly request your advice on it once I post it in a bit... | 12:28 |
| honza | jpich: that's why we have code review :) | 12:29 |
| jpich | :) | 12:29 |
| honza | jpich: i guess documentation was low priority, sorry :( | 12:29 |
| jpich | honza: Ah that's fine, I thought you had had a chance to start something at the time | 12:30 |
| jpich | honza: My bad! | 12:30 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Include the plan name on node assignment https://review.openstack.org/398979 | 12:31 |
| jpich | honza: Here we go ^ 100% done via fumbling | 12:31 |
| honza | jpich: looking! | 12:31 |
| *** jkilpatr has joined #tripleo | 12:32 | |
| jpich | honza: Thank you! | 12:32 |
| *** karthiks has joined #tripleo | 12:33 | |
| weshay | panda|lunch, sshnaidm what's the status on a tripleo periodic promotion? | 12:33 |
| honza | jpich: what's the warning that you're getting? | 12:34 |
| jpich | honza: http://paste.openstack.org/show/589577/ | 12:34 |
| honza | jpich: thanks | 12:35 |
| honza | jpich: somehow i don't think that's related to your code | 12:35 |
| honza | jpich: looks great | 12:35 |
| jpich | honza: huh actually I see it without applying my patch too so | 12:35 |
| jpich | honza: yay \o/ | 12:36 |
| honza | jpich: good work on the bug, i was a bit baffled | 12:36 |
| jpich | honza: Thanks for the review! And I think most of the credit goes to jtomasek really | 12:37 |
| jpich | I was a lot baffled | 12:37 |
| *** masco has quit IRC | 12:37 | |
| honza | haha | 12:37 |
| sshnaidm | weshay, the fix was merged today, so I'll trigger them now | 12:40 |
| *** rodrigods has quit IRC | 12:40 | |
| *** rodrigods has joined #tripleo | 12:40 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 12:40 |
| *** prateek has quit IRC | 12:41 | |
| *** lucasagomes is now known as lucas-hungry | 12:42 | |
| *** lblanchard has joined #tripleo | 12:42 | |
| weshay | sshnaidm, thanks | 12:44 |
| d0ugal | florianf: Yeah, exactly | 12:44 |
| d0ugal | florianf: I just want to make sure what I am doing works for the UI too | 12:44 |
| d0ugal | florianf: http://paste.openstack.org/show/589578/ | 12:45 |
| apevec | weshay, sshnaidm what was the fix? | 12:45 |
| d0ugal | florianf: basically a json object with two keys, one for each rc file. | 12:45 |
| weshay | I think he's talking about https://review.openstack.org/#/c/397959/ | 12:47 |
| weshay | causes db-sync issues | 12:47 |
| openstackgerrit | gyani pillala proposed openstack/puppet-tripleo: New Class for VSA in the puppet to support HPE VSA Storevirtual Cinder Backend https://review.openstack.org/387191 | 12:48 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Source deploy.env on multinode only https://review.openstack.org/398650 | 12:48 |
| apevec | weshay, ack - we also have pymysql update now | 12:49 |
| *** karthiks has quit IRC | 12:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Restore worker configs for mitaka and below https://review.openstack.org/398652 | 12:49 |
| apevec | so only one left is heat BadStatusLine issue, hope is that oslo.msg update will help | 12:50 |
| *** amoralej is now known as amoralej|lunch | 12:51 | |
| *** panda|lunch is now known as panda | 12:55 | |
| jaosorior | slagle: these two were failing on that https://review.openstack.org/#/c/398650/3 https://review.openstack.org/#/c/398652/2 | 12:59 |
| *** chlong has joined #tripleo | 13:00 | |
| slagle | jaosorior: yes, but the problem is with those patches | 13:00 |
| slagle | jaosorior: https://review.openstack.org/#/c/398650 changes the name of deploy.env on the subnodes so the bootstrap script finds nothing to source | 13:01 |
| slagle | i dont think we should then "fix" that by passing in $STABLE_RELEASE over ssh | 13:02 |
| *** jpena|lunch is now known as jpena | 13:02 | |
| jaosorior | slagle: ok, should I abandon the patch then? | 13:02 |
| *** ccamacho is now known as ccamacho|lunch | 13:03 | |
| *** ealcaniz has quit IRC | 13:03 | |
| sshnaidm | slagle, I'll fix it also in bootstrap.sh | 13:06 |
| *** karthiks has joined #tripleo | 13:06 | |
| slagle | i really dont like having to rename deploy.env | 13:06 |
| slagle | i think we can fix this without having to do that | 13:06 |
| sshnaidm | slagle, I'd prefer to not cancel all variables changes in running tripleo.sh, it already brought a few problems for ci | 13:06 |
| *** thrash is now known as thrash|g0ne | 13:06 | |
| sshnaidm | slagle, a few people including me didn't understand why tripleo.sh doesn't include changes we did before, until bnemec showed me this commit, it's really non obvious and hidden | 13:07 |
| slagle | deploy.env should influence tripleo.sh | 13:08 |
| slagle | sshnaidm: yes, there's a bug, but the bug is that we change $OVERCLOUD_DEPLOY_ARGS after we've written it to deploy.env | 13:08 |
| *** pradk has joined #tripleo | 13:09 | |
| sshnaidm | slagle, yeah, there quite a lot of code before sourcing deploy env and running tripleo.sh, and it can influence not only overcloud deploy args | 13:09 |
| sshnaidm | slagle, either to dump all variables each time we gonna run tripleo.sh? | 13:10 |
| *** jayg|g0n3 is now known as jayg | 13:12 | |
| sshnaidm | slagle, actually we source deploy.env in the first lines of deploy.sh, any change to variable in deploy.sh will not be applied in tripleo.sh | 13:12 |
| *** tiswanso has quit IRC | 13:14 | |
| *** apevec has left #tripleo | 13:14 | |
| slagle | yes, that's right | 13:15 |
| *** kjw3 has quit IRC | 13:16 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Source deploy.env on multinode only https://review.openstack.org/398650 | 13:17 |
| *** fultonj has joined #tripleo | 13:20 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-specs: Add a tag specific to documentation issues https://review.openstack.org/399005 | 13:20 |
| *** athomas has joined #tripleo | 13:23 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Remove Combination alarms support https://review.openstack.org/398579 | 13:24 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-docs: Mistral API Documentation https://review.openstack.org/358685 | 13:25 |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud: Add gnocchi support on undercloud https://review.openstack.org/392992 | 13:25 |
| florianf | d0ugal: so the response of the action/api call would be the json, right? will the two rc files also be created on the undercloud? | 13:26 |
| d0ugal | florianf: no, they will only be created if the user deploys with the CLI | 13:27 |
| *** sshnaidm has quit IRC | 13:27 | |
| d0ugal | florianf: the UI would have to give them to the user to save/download | 13:27 |
| shardy | therve: Hey, a few weeks ago you did some tests with a mocked out server and a script to signal deployments, do you have any notes? | 13:27 |
| shardy | therve: we've got reports of performance regressions between mitaka/newton, so it'd be interesting to run your dummy server stress test on each | 13:27 |
| d0ugal | florianf: I'll create a new CLI command to generate the files, so users can also do that | 13:28 |
| *** lucas-hungry is now known as lucasagomes | 13:28 | |
| *** sshnaidm has joined #tripleo | 13:28 | |
| florianf | d0ugal: excellent! I can imagine we want both, even for UI users. | 13:28 |
| *** athomas has quit IRC | 13:28 | |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates: Enable enforce_password_check https://review.openstack.org/397755 | 13:28 |
| *** rlandy has joined #tripleo | 13:29 | |
| *** kjw3 has joined #tripleo | 13:29 | |
| therve | shardy, Yeah let me find it out | 13:31 |
| d0ugal | therve: hey | 13:31 |
| d0ugal | oops | 13:31 |
| d0ugal | sorry | 13:31 |
| d0ugal | thrash|g0ne: hey | 13:31 |
| therve | shardy, http://paste.openstack.org/show/589586/ is the small script I used. I tweaked server_update_allowed to not have nova. | 13:33 |
| therve | shardy, What kind of regressions are we talking about? | 13:35 |
| *** florianf has quit IRC | 13:36 | |
| *** florianf has joined #tripleo | 13:36 | |
| *** rhallisey has joined #tripleo | 13:37 | |
| shardy | therve: https://www.redhat.com/archives/rdo-list/2016-November/msg00016.html | 13:38 |
| shardy | therve: reports of a much much longer deployment time for large stacks | 13:39 |
| shardy | there could be a variety of reasons, but some basic benchmarking of things would be useful I think | 13:40 |
| *** ctayal has joined #tripleo | 13:40 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 13:42 |
| *** abregman has quit IRC | 13:43 | |
| *** pgadiya has quit IRC | 13:45 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 13:46 |
| *** trown|outtypewww is now known as trown | 13:48 | |
| *** panda is now known as panda|bbl | 13:49 | |
| thrash|g0ne | d0ugal: yo | 13:49 |
| d0ugal | thrash|g0ne: I put my question here :) https://bugs.launchpad.net/tripleo/+bug/1615720/comments/3 | 13:50 |
| openstack | Launchpad bug 1615720 in tripleo "overcloudrc should not be managed by the CLI" [Medium,In progress] - Assigned to Dougal Matthews (d0ugal) | 13:50 |
| *** prateek has joined #tripleo | 13:55 | |
| *** bkopilov has joined #tripleo | 13:55 | |
| *** limao has joined #tripleo | 13:56 | |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| *** ccamacho|lunch is now known as ccamacho | 14:03 | |
| *** Goneri has joined #tripleo | 14:04 | |
| *** dsavineau has joined #tripleo | 14:05 | |
| *** tiswanso has joined #tripleo | 14:06 | |
| *** Disova_ has joined #tripleo | 14:08 | |
| *** paramite has quit IRC | 14:08 | |
| *** morazi has joined #tripleo | 14:08 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Move the overcloudrc generation from tripleoclient to a Mistral action https://review.openstack.org/397211 | 14:09 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 14:09 |
| *** fragatina has joined #tripleo | 14:10 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-specs: Add a tag specific to documentation issues https://review.openstack.org/399005 | 14:10 |
| *** kjw3 has quit IRC | 14:10 | |
| *** dsavineau has left #tripleo | 14:13 | |
| jaosorior | ayoung: hey, your commit wasn't meant to actually start using fernet tokens, right? | 14:14 |
| ayoung | right | 14:14 |
| *** morazi has quit IRC | 14:14 | |
| ayoung | it just configures them | 14:14 |
| jaosorior | ok | 14:14 |
| jaosorior | ayoung: fixed it a bit | 14:14 |
| ayoung | jaosorior, thanks. | 14:14 |
| ayoung | jaosorior, you talking about the common one? | 14:15 |
| jaosorior | ayoung: this attempts to use them https://review.openstack.org/#/c/398897/1 just for testing, seemed fine | 14:15 |
| *** fragatina has quit IRC | 14:15 | |
| ayoung | jaosorior, can you look at https://review.openstack.org/#/c/397381/ which is a pre-req? | 14:15 |
| jaosorior | aaah yeah | 14:16 |
| jaosorior | forgot to score that | 14:16 |
| ayoung | sorry I didn't add you to the review, but there is not tripleo-core group in Gerrit. I picked the wrong set of people | 14:16 |
| jaosorior | d0ugal: hey dude, could you check out https://review.openstack.org/#/c/397381/ as well? | 14:16 |
| ayoung | jaosorior, if you make any more changes to the THT one, add yourself as a co-author. You've already extended the functionality beyond what I did, which justifies authorship | 14:17 |
| ayoung | going to leave it for now, since it has you as committer. | 14:18 |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-quickstart: Add blockstorage to default node flavor https://review.openstack.org/396577 | 14:19 |
| *** mnaser has joined #tripleo | 14:20 | |
| jaosorior | ayoung: I'll update the commit message so it's clear to other people that this only sets of the necessary stuff to deploy with fernet as a provider, BUT does not intend to use it as a default. I saw that someone -1ed your previous attempt because of that. | 14:21 |
| mnaser | is "adopting" an existing cloud within tripleo something that's still really far away down the line? | 14:21 |
| ayoung | jaosorior, ++ | 14:21 |
| mnaser | i understand it's quite complex but we really want to take advantage of tripleo to continue scaling things up | 14:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Configure Keystone Fernet Keys https://review.openstack.org/397350 | 14:22 |
| jaosorior | ayoung: at what point should we be switching the default? | 14:23 |
| beagles | shardy: thanks for getting that l3ha controllercount thing .. beat me to it! | 14:24 |
| beagles | shardy: been too heads down on this ovs bridge fail_mode mess the last couple of weeks :( | 14:24 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 14:25 |
| *** jcoufal has joined #tripleo | 14:26 | |
| shardy | beagles: np - hopefully the patches look reasonable | 14:27 |
| *** morazi has joined #tripleo | 14:28 | |
| beagles | shardy: they do.. made a comment on the calculation about api count vs agent count though | 14:28 |
| shardy | beagles: ack, just seen it, thanks, fixing | 14:28 |
| *** Disova_ has quit IRC | 14:28 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: TEST FERNET TOKENS https://review.openstack.org/398897 | 14:28 |
| sshnaidm | jaosorior, so what we'll do with https://review.openstack.org/#/c/398782 ? I've already included sourcing in boostrap.sh file here: https://review.openstack.org/#/c/398650 | 14:29 |
| shardy | beagles: one question was that given I needed to add l3_ha_override, do we still need to remove the deprecated NeutronL2HA parameter? | 14:30 |
| shardy | I was thinking we could just un-deprecate it and leave it as an optional override | 14:30 |
| shardy | but we could also remove it after this fix gets backported I guess | 14:30 |
| *** tzumainn has joined #tripleo | 14:31 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Do not manage overcloud repositories when using external Ceph https://review.openstack.org/398475 | 14:31 |
| jaosorior | sshnaidm: I can abandon it if your change works without i | 14:31 |
| jaosorior | *without it | 14:31 |
| sshnaidm | jaosorior, I think I need to remove rebase to be sure.. | 14:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Source deploy.env on multinode only https://review.openstack.org/398650 | 14:33 |
| shardy | beagles: so, to clarify, we need to count how many times the puppet/services/neutron-l3.yaml service is deployed, right? | 14:33 |
| jaosorior | sshnaidm: done | 14:33 |
| beagles | shardy: yup | 14:33 |
| shardy | we can do that via hiera neutron_l3_short_node_names | 14:33 |
| shardy | beagles: Ok, updating, thanks! | 14:33 |
| beagles | shardy: cool | 14:33 |
| sshnaidm | jaosorior, you abandoned your patch and then did rebase in upper one, right? | 14:34 |
| jaosorior | right | 14:34 |
| sshnaidm | jaosorior, great | 14:35 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates: Add ec2-api service https://review.openstack.org/398634 | 14:36 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Move calculation of neutron l3_ha into puppet profile https://review.openstack.org/398926 | 14:37 |
| ayoung | jaosorior, once we QE running with Fernet tokens and saying there are no new regressions | 14:37 |
| ayoung | jaosorior, lets get the mechanism implemented, tested, and people confident in it, then make it the default. | 14:37 |
| jaosorior | sure | 14:39 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates: Closes-Bug: #1642551 https://review.openstack.org/399040 | 14:39 |
| openstack | bug 1642551 in tripleo "os-net-config-mappings.yaml - Example does not work on most hardware due to "Consistent Network Device Naming"" [Undecided,New] https://launchpad.net/bugs/1642551 | 14:39 |
| weshay | can we get someone to triage this bug please https://bugs.launchpad.net/ironic/+bug/1639013 | 14:39 |
| openstack | Launchpad bug 1639013 in Ironic "fail: openstack baremetal import --json instackenv.json, Exception registering nodes: No valid host was found. Reason: No conductor service registered which supports driver..." [Undecided,New] | 14:39 |
| weshay | lucasagomes, would you mind taking a peak at that ^ | 14:40 |
| *** jaosorior has quit IRC | 14:41 | |
| weshay | sshnaidm, panda|bbl any udpates in https://etherpad.openstack.org/p/tripleo-ci-status for the periodic failures? | 14:42 |
| lucasagomes | weshay, hmm is the ir-conductor running ? | 14:42 |
| shardy | weshay: there looks to be an error in the conductor log related to access of the DB | 14:43 |
| shardy | added a comment, not sure if it's related | 14:43 |
| sshnaidm | weshay, not yet | 14:43 |
| weshay | lucasagomes, I'm just going through the tripleo-ci status page looking at older issues | 14:43 |
| weshay | thanks shardy | 14:43 |
| lucasagomes | weshay, right... cause that bug seems to be closed in bugzilla | 14:44 |
| lucasagomes | weshay, apparently it was a problem with mariadb | 14:44 |
| lucasagomes | weshay, https://bugzilla.redhat.com/show_bug.cgi?id=1391602 | 14:44 |
| openstack | bugzilla.redhat.com bug 1391602 in openstack-ironic "fail: openstack baremetal import --json instackenv.json, Exception registering nodes: No valid host was found. Reason: No conductor service registered which supports driver..." [High,Closed: worksforme] - Assigned to lmartins | 14:44 |
| lucasagomes | so I guess we could close the launchpad ticket as well | 14:44 |
| weshay | k.. myoung hrybacki ^ | 14:45 |
| * hrybacki reads up | 14:45 | |
| weshay | hrybacki, let me know if we can remove the ironic-conductor restart workaround | 14:45 |
| *** limao has quit IRC | 14:47 | |
| hrybacki | weshay: reviewing notes from rlandy and myoung I think we should be fine merging it | 14:48 |
| hrybacki | myoung: and the bug is closed (no longer relevant? need specifics from myoung there) | 14:49 |
| weshay | hrybacki, I think I can cross out issue #14 now too right? | 14:49 |
| myoung | ack...i did a sample CI run removing the workaround and it worked last week | 14:50 |
| *** kjw3 has joined #tripleo | 14:50 | |
| myoung | weshay, hrybacki: also have a patch up with gates passing to remove the workaround. It has not landed yet --> https://review.gerrithub.io/#/c/300979/ | 14:50 |
| weshay | great | 14:51 |
| weshay | thanks | 14:51 |
| hrybacki | weshay: yep, 14 can be closed (saw your merge) | 14:51 |
| myoung | weshay, hrybacki, can we get eyes from whomever can +2 that patch? I would like to cross this off my lists as well (internal, trello) | 14:53 |
| ansiwen | shardy (and everyone else who wants to help me with ec2api): I created a THT change for ec2api now and made a couple of inline comments on my change. Could you help me ansering them? I still didn't finish reading tht-walkthrough, so please ignore all questions, that are answered by it. | 14:53 |
| hrybacki | myoung: just got a +2 from trown | 14:53 |
| hrybacki | +1 from rlandy and I | 14:54 |
| ansiwen | shardy: https://review.openstack.org/#/c/398634 | 14:54 |
| *** jbadiapa has quit IRC | 14:59 | |
| bnemec | derekh: sshnaidm: panda|bbl: Weird, I used 4.5 nodes per job in my calculations (5 for ha, 4 for nonha) which came out to around 2.7 TB, and that's right where we were sitting yesterday with the full 60 jobs running. | 15:01 |
| bnemec | I wonder why it started using more later? | 15:02 |
| bnemec | derekh: sshnaidm: panda|bbl: Also, we're still hitting the new quota: Quota exceeded for ram: Requested 8192, but already used 3139584 of 3145728 ram | 15:03 |
| derekh | bnemec: I'm surprised we hit the quota too, am kind of wondering if the usage is off somewhere (maybe some usage leaked at some stage) | 15:03 |
| bnemec | derekh: Hmm, I show 72 stacks right now. | 15:05 |
| bnemec | Some are creating or deleting, but that still seems unusually high since our max concurrent jobs is 60. | 15:05 |
| sshnaidm | http://paste.openstack.org/show/589600/ | 15:05 |
| sshnaidm | 438 instances | 15:05 |
| derekh | strange | 15:06 |
| sshnaidm | bnemec, how do you limit jobs count? | 15:08 |
| bnemec | sshnaidm: Infra does. We're supposed to have a max of 60 jobs running at any one time on rh1. | 15:08 |
| bnemec | I see 65 create_complete heat stacks though. | 15:08 |
| bnemec | That's definitely not right. | 15:09 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Move the overcloudrc generation from tripleoclient to a Mistral action https://review.openstack.org/397211 | 15:10 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use the overcloudrc generated in a Mistral action https://review.openstack.org/398975 | 15:10 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 15:10 |
| ansiwen | ccamacho: you would probably a great candidate to help me as well ;-) | 15:10 |
| *** jbadiapa has joined #tripleo | 15:11 | |
| ccamacho | hey ansiwen | 15:11 |
| ccamacho | sure | 15:11 |
| *** oshvartz has quit IRC | 15:11 | |
| derekh | bnemec: sshnaidm ok, I think I know whats going on | 15:12 |
| derekh | we have 2 limits (because why not) | 15:12 |
| derekh | infra should be limiting the number of slaves to 60 (which I assume it is) | 15:12 |
| derekh | but | 15:13 |
| *** panda|bbl is now known as panda | 15:13 | |
| derekh | some time ago I noticed that if zuul kills a CI job before testenv-client releases it (a hard kill from zuul), the testenv isn't released until 20 minutes later | 15:14 |
| derekh | because it was never released it stays around until the 20 minute timeout is hit | 15:14 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-specs: Add a tag specific to documentation issues https://review.openstack.org/399005 | 15:14 |
| sshnaidm | derekh, so we have overlapping? | 15:15 |
| derekh | to allow for envs that arn't being used but still exist the te-broker limit is set higher then the nodepool limit | 15:15 |
| derekh | here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/rh1.env#n12 | 15:15 |
| bnemec | derekh: Hmm, so this could happen any time someone (for example) pushes a new patch set to a change that already has tests running on it? | 15:16 |
| derekh | sshnaidm: kindoff at any one time the number of testenvs is equal to the number of currently running tests + the number of abandoned tests in the last 20 minutes | 15:17 |
| sshnaidm | bnemec, I think it happens when zuul kiils job by timeout | 15:17 |
| derekh | bnemec: yup | 15:17 |
| derekh | and sometimes I think it does it for other reasons | 15:18 |
| panda | ah, cleanup problems then | 15:18 |
| sshnaidm | derekh, bnemec so to increase quota acc. to 80 stacks? | 15:19 |
| *** tzumainn has quit IRC | 15:19 | |
| *** prateek has quit IRC | 15:19 | |
| panda | or cleanup sync with zuul ? | 15:19 |
| bnemec | We don't have a lot more space. I think we have 3.6 TB active right now. | 15:19 |
| panda | I don't even know how to get this .. | 15:19 |
| derekh | sshnaidm: I think so, or figure out a way to release a Test env when zuul abandons a job | 15:20 |
| bnemec | And we can't use all of that for nodepool because there are other vms running. | 15:20 |
| sshnaidm | derekh, why do we use 20 minutes? | 15:21 |
| sshnaidm | derekh, is it time for creating env and start running? | 15:21 |
| derekh | sshnaidm: no reason for that specific number | 15:22 |
| *** dsavineau has joined #tripleo | 15:23 | |
| sshnaidm | derekh, I mean what if it'll be 5 mins, than fewer stacks will live | 15:23 |
| derekh | sshnaidm: actually now that I think about it, I might be thinking about the wrong timeout, it could be longer | 15:23 |
| panda | maybe it's the entire testenv timeout | 15:23 |
| derekh | sshnaidm: we should test it, it could be.....ya what panda said | 15:23 |
| yolanda | hi derekh , so the dib test was fixed, but getting another failure now: http://logs.openstack.org/61/375261/22/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/5b6813a/console.html#_2016-11-17_15_12_39_227660 | 15:24 |
| *** ctayal has quit IRC | 15:24 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-specs: Add a tag specific to documentation issues https://review.openstack.org/399005 | 15:24 |
| yolanda | also looking at http://logs.openstack.org/61/375261/22/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/5b6813a/logs/postci.txt.gz , i see errors there, but not sure if that's related | 15:25 |
| sshnaidm | derekh, are we talking about this 1200 https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L234 ? | 15:26 |
| panda | sshnaidm: that one is the timeout to wait for a stack | 15:26 |
| panda | sshnaidm: I think we are talking about TIMEOUT_SECS | 15:27 |
| panda | sshnaidm: two lines below | 15:27 |
| sshnaidm | panda, aha | 15:27 |
| bnemec | I don't think we want to shorten that one. If a lot of jobs come in at once we can easily hit 20 minutes to create all the testenvs. | 15:27 |
| derekh | yolanda: looks like a problem building the package | 15:27 |
| derekh | yolanda: DEBUG: install: cannot stat 'lib/*': No such file or directory | 15:27 |
| derekh | panda: ya, I think your right, TIMEOUT_SECS is the one, which is a lot worse | 15:28 |
| yolanda | derekh, and that's related to our change? | 15:28 |
| sshnaidm | derekh, panda this timeout is about 3 hours | 15:29 |
| panda | bnemec: No, we're not touching that, I think the best cours would be to tell zuul to tell testenv to destroy the env, before killing the job | 15:29 |
| derekh | we should spin up a VM request a test-env and then destroy the VM and see how long it take to release the env, this will confirm the theory | 15:29 |
| derekh | panda: ya, that would be ideal | 15:30 |
| panda | sshnaidm: I don't think that's correct | 15:31 |
| derekh | yolanda: possible or maybe just related to the branch, does the feature/v2 branch differ from master much? | 15:31 |
| panda | sshnaidm: TIMEOUT_SEC is DEVSTACK TIMEOUT * 60 = 80 * 60 = 4800 sec | 15:31 |
| panda | sshnaidm: I think it's 80 minutes | 15:31 |
| * bnemec needs to look at how te-worker functions again. | 15:31 | |
| yolanda | derekh, yes, so much | 15:32 |
| yolanda | https://git.openstack.org/cgit/openstack/diskimage-builder/log/?h=feature/v2 | 15:32 |
| bnemec | panda: That's calculated internally though by subtracting the time for undercloud install and image build from the overall timeout. | 15:32 |
| sshnaidm | panda, why DEVSTACK_GATE_TIMEOUT is 80? | 15:32 |
| sshnaidm | panda, isn't it full job time? | 15:33 |
| panda | bnemec: yep, roughly base on the entire devstack timeout set in jjb definition | 15:33 |
| derekh | yolanda: the problem could be related to any(or many) of those | 15:33 |
| shardy | marios: Hey upgrades question - do you take steps in the current implementation to ensure haproxy is stopped before all the services? | 15:33 |
| yolanda | derekh, bad thing | 15:33 |
| *** bana_k has joined #tripleo | 15:33 | |
| shardy | I couldn't see where, and it seems like stopping it before the services would be a good idea | 15:33 |
| yolanda | i'll share with dib cores | 15:34 |
| panda | sshnaidm: oh, no you're right, too many variable called similarly | 15:34 |
| *** absubram has joined #tripleo | 15:34 | |
| panda | sshnaidm: it's 170 minutes then, the entire timeout set in jjb definition for tripleo jobs | 15:34 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Only start the deploy if the Heat stack isn't already in progress https://review.openstack.org/398959 | 15:34 |
| sshnaidm | panda, exactly | 15:34 |
| *** absubram_ has joined #tripleo | 15:35 | |
| yolanda | derekh, where did you see that lib/* failure? i'm not able to see on logs | 15:35 |
| *** michapma_alt has quit IRC | 15:36 | |
| *** fragatina has joined #tripleo | 15:37 | |
| derekh | yolanda: I'd imagine this is the patch when patching stopped working https://review.openstack.org/#/c/367156/6/setup.cfg | 15:37 |
| derekh | yolanda: but can't tell for sure because we were testing master .... | 15:38 |
| *** absubram has quit IRC | 15:38 | |
| *** absubram_ is now known as absubram | 15:38 | |
| derekh | yolanda: see ./opt/stack/new/delorean/data/repos/99/fd/99fd82f0dc2a40ab7e314f7cee2ebc2283e40bf3_65436912/rpmbuild.log | 15:38 |
| derekh | yolanda: in delorean_repos.tar.xz | 15:38 |
| yolanda | derekh, so well, at least it was detected before landing | 15:38 |
| derekh | yolanda: for that error message | 15:38 |
| yolanda | ah, thanks for the pointer | 15:40 |
| weshay | sshnaidm, do you still have the link on the page for the top issues? | 15:41 |
| weshay | see it | 15:41 |
| *** fragatina has quit IRC | 15:41 | |
| *** fragatina has joined #tripleo | 15:42 | |
| weshay | sshnaidm, panda, amoralej FYI.. we should add the signature for this issue asap to the regex https://bugs.launchpad.net/tripleo-quickstart/+bug/1638908 | 15:42 |
| openstack | Launchpad bug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] - Assigned to Alfredo Moralejo (amoralej) | 15:42 |
| panda | does zuul support post-script ? | 15:42 |
| weshay | in https://github.com/sshnaidm/sova | 15:42 |
| sshnaidm | weshay, ok | 15:42 |
| sshnaidm | weshay, wasn't it solved? | 15:43 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 15:43 |
| weshay | sshnaidm, not quite yet.. still seeing it | 15:43 |
| sshnaidm | weshay, can you drop a link? | 15:44 |
| *** jbadiapa has quit IRC | 15:44 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Add tuned check to remote provision role https://review.openstack.org/396362 | 15:44 |
| openstackgerrit | Ade Lee proposed openstack/tripleo-quickstart: changes to add novajoin to undercloud https://review.openstack.org/398772 | 15:44 |
| openstackgerrit | Ade Lee proposed openstack/tripleo-quickstart: Add support to set DNS server on the undercloud https://review.openstack.org/398771 | 15:44 |
| weshay | I know amoralej has a box setup w/ it right now /me looks for a job | 15:45 |
| amoralej | yeah, weshay | 15:46 |
| sshnaidm | weshay, because last times I saw it was just a symptom, but root cause was different | 15:46 |
| *** chandankumar has quit IRC | 15:46 | |
| weshay | myoung, amoralej, we want to see how often this is happening by adding it to https://github.com/sshnaidm/sova/blob/master/tripleoci/patterns.py | 15:46 |
| weshay | sshnaidm, we can probably say that about a number of these errors | 15:47 |
| ansiwen | ccamacho: thanks :-) | 15:47 |
| weshay | amoralej, myoung can we get a link to some logs | 15:48 |
| weshay | don't see them in the etherpad | 15:48 |
| panda | maybe the patters should be a yaml file ... | 15:48 |
| sshnaidm | weshay, I think it happens often with various root causes, that's why I don't add it, it doesn't say anything about the problem actually | 15:48 |
| *** tzumainn has joined #tripleo | 15:49 | |
| greghaynes | derekh: hey - so I think you got poked that you alls CI is not so happy with the dib v2 branch | 15:50 |
| amoralej | weshay, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-minimal-132/undercloud/home/stack/overcloud_deploy.log.gz | 15:50 |
| amoralej | ('Connection aborted.', BadStatusLine("''",)) | 15:50 |
| trown | sshnaidm: what do you mean... it says the overcloud deploy failed due to haproxy timeouts... it doesnt say "why" sure, but it seems like a worthy category to track given how annoying and hard to reproduce the issue is | 15:51 |
| derekh | greghaynes: yup, package building is failing, with the following error --- DEBUG: install: cannot stat 'lib/*': No such file or directory | 15:51 |
| derekh | greghaynes: which I think probbaly started here with this commit https://review.openstack.org/#/c/367156/6/setup.cfg | 15:52 |
| greghaynes | derekh: gotcha, so we moved the code outside of the diskimage_builder dir to be under it (this makes our python packaging a ton better because its all part of the python module then) | 15:52 |
| greghaynes | derekh: not sure what that means from an rpm packaging standpoint, but that dir is now diskimage_builder/lib | 15:53 |
| greghaynes | derekh: elements were moved similarly: elements -> diskimage_builder/elements | 15:53 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Add tuned check to remote provision role https://review.openstack.org/396362 | 15:53 |
| sshnaidm | trown, I think I saw it also where pingtest failed to create because of oom killers | 15:53 |
| amoralej | shardy, i have reproduced the badlinestatus issue locally trying to deploy a overcloud in a pretty fast server | 15:54 |
| greghaynes | derekh: also, realistically I think itll be a couple weeks before we will potentially cut an RC of that branch, so its not super pressing but im betting itll be nicer to fix before then :) | 15:55 |
| derekh | greghaynes: I'd guess it means a reletivly small tweak to the packaging, but the real question is does the RDO packaging support that branch | 15:55 |
| sshnaidm | trown, weshay no problem to insert it, but connection can not be established because of very various reasons, not only because of timeouts | 15:55 |
| derekh | greghaynes: so that plan is to merge that branch into master? | 15:56 |
| greghaynes | derekh: yes | 15:56 |
| marios | shardy: sorry on scrum right now will look in a minute | 15:56 |
| sshnaidm | trown, could you look please at last "periodic" log? I think neutron still has problems there: http://logs.openstack.org/15/359215/20/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/83c5c50/logs/undercloud/var/log/undercloud_install.txt.gz#_2016-11-17_13_59_29_000 | 15:56 |
| panda | sshnaidm: trown weshay, I too remember seeing it on various occasions, it's not a 1:1 correlation with a single cause. | 15:56 |
| shardy | amoralej: ack, do you have https://review.openstack.org/#/c/394963/ applied? | 15:57 |
| shardy | would be interesting to know if that helps if so | 15:57 |
| amoralej | yes, it's applied | 15:57 |
| derekh | greghaynes: ok, I guess RDO should be ready for it so, I'll ping them in #rdo to see who should be aware of it | 15:57 |
| amoralej | and didn't improve response times, although this system is fast and are lower that in CI | 15:57 |
| greghaynes | derekh: awesome, thanks | 15:57 |
| shardy | amoralej: Ok, so are you still seeing heat GET reponses exceeding the haproxy timeout? | 15:57 |
| amoralej | not really | 15:58 |
| amoralej | so | 15:58 |
| amoralej | situation in my test system is | 15:58 |
| amoralej | withouth the messaging patch -> response times around 10 seconds | 15:58 |
| weshay | panda, that's fine.. I want to see how often we hit it | 15:58 |
| amoralej | with messaging patch -> response times similar (to be honest a bit higher, 14 secs) and i got badstatusline | 15:58 |
| amoralej | so it could be different issues | 15:59 |
| myoung | weshay: most current failures are on RHEL, so not on thirdparty logs. digging up more recent on centos...moment | 15:59 |
| amoralej | badstatusline could not be caused by heat long response times | 15:59 |
| shardy | Ok so way under the haproxy timeout then | 15:59 |
| shardy | amoralej: is the undercloud running in a VM on this system? | 16:00 |
| yolanda | derekh, thanks. So my concern is that this branching can be possible on the packaging | 16:00 |
| amoralej | yes shardy | 16:01 |
| amoralej | always oooq | 16:01 |
| shardy | amoralej: OK how many vcpus does the VM have? | 16:01 |
| amoralej | 8 | 16:01 |
| amoralej | i can give you access if you want to log in | 16:01 |
| myoung | weshay, amoralej, sshnaidm: last fail i've got a log for on centos based (public logs). I can send internal links on another channel for the recent fails on rhel --> https://thirdparty-logs.rdoproject.org/jenkins-tripleo-quickstart-periodic-newton-delorean-ha_192gb-16/undercloud/home/stack/overcloud_deploy.log.gz#_2016-11-07_04_27_51 | 16:01 |
| derekh | yolanda: I'm not sure what you mean | 16:02 |
| *** penick has joined #tripleo | 16:02 | |
| yolanda | i mean, if the packaging is going to be different for feature/v2 and master branches, will the packaging system support it? | 16:02 |
| *** absubram has quit IRC | 16:03 | |
| derekh | yolanda: as it right now it doesn't and the tests will continue to fail, so the way I see it there is 3 options | 16:04 |
| derekh | yolanda: 1. add changes to the master packing of DIB in rdo too support both branches | 16:05 |
| *** tremble has quit IRC | 16:05 | |
| derekh | 2. or maybe add a v2 branch to RDO packaging | 16:06 |
| *** abregman has joined #tripleo | 16:06 | |
| derekh | but you should talk with #rdo about whats possible there | 16:06 |
| *** abregman has quit IRC | 16:06 | |
| derekh | or 3. leave it broken and fix it once it merges but the obvious down side is the lack of tripleo ci) | 16:06 |
| * greghaynes really doesnt like v3 | 16:07 | |
| greghaynes | although I bet you dont either | 16:07 |
| *** achadha has joined #tripleo | 16:07 | |
| yolanda | i don't like v3 :) | 16:08 |
| *** achadha has quit IRC | 16:09 | |
| *** achadha has joined #tripleo | 16:09 | |
| *** pcaruana has quit IRC | 16:10 | |
| *** ayoung has quit IRC | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 16:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| *** ebarrera has quit IRC | 16:10 | |
| *** tiswanso has quit IRC | 16:11 | |
| *** tiswanso has joined #tripleo | 16:14 | |
| *** fragatina has quit IRC | 16:15 | |
| yolanda | i need to go now , but tomorrow i'll poke on #rdo channel for that | 16:17 |
| *** nyechiel has quit IRC | 16:19 | |
| *** ramishra has quit IRC | 16:19 | |
| *** ramishra has joined #tripleo | 16:21 | |
| openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Changes default MidoNet API port on HAProxy https://review.openstack.org/399125 | 16:24 |
| *** liverpooler has quit IRC | 16:27 | |
| *** b00tcat has quit IRC | 16:29 | |
| *** achadha_ has joined #tripleo | 16:29 | |
| *** anshul has quit IRC | 16:29 | |
| *** fragatina has joined #tripleo | 16:30 | |
| *** penick has quit IRC | 16:30 | |
| *** dsariel has quit IRC | 16:32 | |
| *** achadha has quit IRC | 16:32 | |
| *** penick has joined #tripleo | 16:32 | |
| flaper87 | is there a way to have tripleo-quickstart not setting up localhost ? | 16:34 |
| flaper87 | It connects to the local node to add the host to the inventory and gather facts | 16:34 |
| flaper87 | local node being the node I'm running quickstart from | 16:35 |
| *** yamahata has joined #tripleo | 16:35 | |
| *** cylopez has left #tripleo | 16:36 | |
| hewbrocca | hmm bogdando was trying to sort this a couple weeks ago IIRC | 16:36 |
| *** tiswanso has quit IRC | 16:39 | |
| bogdando | flaper87, hi. I'd only tried the localhost case with the GCE host node as a VMs carrier and failed miserably | 16:39 |
| *** tiswanso has joined #tripleo | 16:39 | |
| bogdando | not sure this related to the topic | 16:39 |
| panda | flaper87: why you don't want your facts gathered ? | 16:40 |
| panda | flaper87: something to hide ? :) | 16:40 |
| flaper87 | panda: because they don't support py3 :P | 16:40 |
| flaper87 | it's a long story | 16:40 |
| panda | flaper87: you have an error to show ? | 16:41 |
| flaper87 | http://paste.openstack.org/show/589610/ | 16:41 |
| *** fragatina has quit IRC | 16:41 | |
| flaper87 | ok, I'm passed that problem. I can explicitly pass the python interpreter I want | 16:41 |
| *** anshul has joined #tripleo | 16:42 | |
| panda | flaper87: when it's so easy it usually breaks things later in the run. | 16:43 |
| *** ctayal has joined #tripleo | 16:43 | |
| flaper87 | panda: yeah, that's why I'm not planning to run this command anymore | 16:43 |
| flaper87 | I'd rather keep myself happy | 16:43 |
| ansiwen | shardy: thank you a lot for your comments! | 16:45 |
| shardy | ansiwen: np | 16:45 |
| *** aufi has joined #tripleo | 16:46 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 16:50 |
| *** arxcruz has quit IRC | 16:50 | |
| *** migi has joined #tripleo | 16:51 | |
| *** penick has quit IRC | 16:52 | |
| *** arxcruz has joined #tripleo | 16:53 | |
| *** derekh has quit IRC | 16:55 | |
| *** achadha_ has quit IRC | 16:56 | |
| *** tiswanso has quit IRC | 16:59 | |
| *** penick has joined #tripleo | 17:00 | |
| ansiwen | shardy: if there is no wsgi folder, I can assume it's not running as an apache plugin, right? | 17:01 |
| *** tiswanso has joined #tripleo | 17:05 | |
| *** ctayal has quit IRC | 17:06 | |
| *** pblaho has quit IRC | 17:08 | |
| *** ebarrera has joined #tripleo | 17:08 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 17:10 |
| *** ayoung has joined #tripleo | 17:10 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-quickstart: Pass the libvirt_uri to the pool-define command https://review.openstack.org/399141 | 17:11 |
| *** anshul has quit IRC | 17:11 | |
| *** numans has quit IRC | 17:16 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles https://review.openstack.org/330659 | 17:17 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Update image building CI to do full deploy https://review.openstack.org/398556 | 17:18 |
| *** fragatina has joined #tripleo | 17:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add worker config envs in toci_gate_test https://review.openstack.org/399146 | 17:23 |
| bnemec | slagle: sshnaidm: An alternate fix for the worker config problem ^ | 17:23 |
| *** migi is now known as migi_afk | 17:25 | |
| trown | bnemec: I like that, as it is easier to factor that out as part of my patch to set all the env variables in a common place | 17:25 |
| *** migi_afk is now known as migi | 17:25 | |
| sshnaidm | bnemec, slagle I still don't think overriding everything in tripleo.sh it's a good idea | 17:25 |
| *** panda is now known as panda|bbl | 17:27 | |
| slagle | we only override it if we source deploy.env, which is a CI thing, or for those trying to run CI locally | 17:28 |
| slagle | generally, I like the idea of deploy.env being accurate, and we should avoid changing the vars after we write to it | 17:29 |
| slagle | that being said, i'm not blocking any fix at this point | 17:29 |
| shardy | ansiwen: AFAICS from the git repo it's a stanalone eventlet based API not running under httpd | 17:29 |
| *** hjensas has quit IRC | 17:29 | |
| *** fragatina has quit IRC | 17:29 | |
| shardy | ansiwen: that means you definitely need to add an endpoint as commented | 17:30 |
| *** fragatina has joined #tripleo | 17:30 | |
| *** aufi has quit IRC | 17:32 | |
| *** chandankumar has joined #tripleo | 17:32 | |
| bnemec | sshnaidm: I don't particularly either, but I want to get a fix in somewhere. | 17:32 |
| bnemec | I'm not up for arguing this to death, so meh. | 17:32 |
| *** hewbrocca is now known as hewbrocca_afk | 17:33 | |
| sshnaidm | bnemec, I don't see my patch contradicts your actually | 17:34 |
| sshnaidm | bnemec, both could be merged | 17:34 |
| *** lucasagomes is now known as lucas-afk | 17:35 | |
| *** chandankumar has quit IRC | 17:36 | |
| *** achadha has joined #tripleo | 17:37 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Replace hard-coded haproxy/keepalived coupling https://review.openstack.org/399152 | 17:39 |
| *** achadha_ has joined #tripleo | 17:40 | |
| *** achadha has quit IRC | 17:40 | |
| *** achadha_ has quit IRC | 17:41 | |
| *** achadha has joined #tripleo | 17:42 | |
| *** yamahata has quit IRC | 17:43 | |
| *** ebarrera has quit IRC | 17:43 | |
| *** achadha has quit IRC | 17:45 | |
| *** achadha has joined #tripleo | 17:45 | |
| *** achadha has quit IRC | 17:45 | |
| *** achadha has joined #tripleo | 17:46 | |
| *** tbonds has joined #tripleo | 17:47 | |
| ansiwen | shardy: ok, and the ec2api::api::service_name I leave unset? | 17:48 |
| *** lmiccini has quit IRC | 17:48 | |
| *** bana_k has quit IRC | 17:49 | |
| *** fzdarsky is now known as fzdarsky|afk | 17:49 | |
| shardy | https://github.com/openstack/puppet-ec2api/blob/master/manifests/api.pp#L219 | 17:50 |
| * shardy shrugs | 17:50 | |
| shardy | ansiwen: I've never run this service, it appears to have a default, but I've got no idea if ec2api::params::api_service_name will be set | 17:50 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates: Add ec2-api service https://review.openstack.org/398634 | 17:50 |
| shardy | probably best to try it and see if it works ;) | 17:50 |
| ansiwen | shardy: yes, it will | 17:51 |
| ansiwen | shardy: my question was more in the line of: it just can be the default then | 17:51 |
| shardy | ansiwen: ah, AFAICS the answer is yes :) | 17:51 |
| ansiwen | shardy: thank you :-) | 17:53 |
| *** ayoung has quit IRC | 17:55 | |
| *** jbadiapa has joined #tripleo | 17:58 | |
| *** derekjhyang has joined #tripleo | 17:58 | |
| *** jpena is now known as jpena|off | 18:01 | |
| *** abehl has joined #tripleo | 18:02 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: [WIP] config for containerized-compute https://review.openstack.org/393348 | 18:03 |
| *** fragatin_ has joined #tripleo | 18:04 | |
| ansiwen | pradk: yes, it should be optional | 18:07 |
| ansiwen | pradk: so it still must be added to the registry, no? | 18:07 |
| *** fragatina has quit IRC | 18:07 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 18:10 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Missing comma in sample could be confusing. https://review.openstack.org/399168 | 18:11 |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Missing comma in sample could be confusing https://review.openstack.org/399168 | 18:12 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove conditional for neutron l3_ha https://review.openstack.org/398934 | 18:14 |
| ansiwen | pradk: do you have an example of an env file for an optional service? I have to add it to the Controller role, but I don't find an example for that in the environment folder | 18:19 |
| *** yamahata has joined #tripleo | 18:19 | |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/cinder-backup.yaml | 18:20 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L113 | 18:21 |
| shardy | ansiwen: ^^ CinderBackup is an example | 18:21 |
| *** ayoung has joined #tripleo | 18:21 | |
| ansiwen | shardy: but this just adds it to the registry? how do I add it to the Controller node? | 18:21 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/roles_data.yaml#L29 | 18:22 |
| ansiwen | shardy: oh, now I understand, so I can add it to the role_data.yaml, and if it's not registered, its not an error but is just ignored? | 18:23 |
| pradk | ansiwen, yea just set the heat param to None | 18:23 |
| shardy | ansiwen: no, you add it to roles_data.yaml and by default it uses the mapping to OS::Heat::None in the overcloud-resource-registry | 18:23 |
| pradk | ansiwen, yea that just ignores it unless you pass in the env file | 18:23 |
| shardy | then environment/enabled_foo.yaml switches the mapping to a non-none template | 18:23 |
| pradk | ansiwen, see my panko service https://review.openstack.org/#/c/396439/ | 18:24 |
| shardy | ansiwen: in future we'll instead manipulate the list of services using heat environment merging, but for now that's how it works | 18:24 |
| openstackgerrit | d.marlin proposed openstack/diskimage-builder: Change path for dnf arch override so basearch is not overwritten. https://review.openstack.org/399175 | 18:24 |
| ansiwen | shardy: last time I understood it exactly in the other way, I though you proposed to only add it to overcloud-resource-registry-puppet.j2.yaml and NOT to roles_data.yaml | 18:24 |
| *** trown is now known as trown|lunch | 18:25 | |
| shardy | ansiwen: well, you could do that, then folks would have to make a copy of the services list and pass it as an environment file to enable it for the controller | 18:25 |
| shardy | until we integrate with the heat environment merging I mentioned | 18:25 |
| pradk | can i get some reviews on https://review.openstack.org/#/q/topic:remove-comb-alarms please .. quite simple | 18:26 |
| ansiwen | shardy: ok, they would have to put the whole list to the environment then, you can't just add it until the merging... got it | 18:26 |
| shardy | ansiwen: yup, exactly | 18:27 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/hyperconverged-ceph.yaml | 18:27 |
| shardy | ansiwen: that's an example of the merging approach | 18:27 |
| shardy | then you would not need to add anything to roles_data.yaml | 18:28 |
| shardy | but last time I checked that doesn't work because of problems in tripleoclient | 18:28 |
| shardy | so we need to fix that before using it elsewhere | 18:28 |
| *** jeckersb is now known as jeckersb_gone | 18:29 | |
| *** jeckersb_gone is now known as jeckersb | 18:29 | |
| weshay | anyone around to help push on https://review.openstack.org/#/c/397239/ | 18:34 |
| openstackgerrit | Sven Anderson proposed openstack/tripleo-heat-templates: Add ec2-api service https://review.openstack.org/398634 | 18:34 |
| ansiwen | shardy, pradk: thanks a bunch guys... new version is up | 18:36 |
| *** coolsvap has quit IRC | 18:37 | |
| ansiwen | shardy, pradk: another question: maybe discussed here before. I'm working on the bug which requires deployment of ssh keypairs for the nova user on all nodes. puppet-nova supports the deployment of these. could we dynamically create an ssh keypair with heat and then feed it into the puppet config of the compute nodes? | 18:39 |
| *** newmember has joined #tripleo | 18:39 | |
| shardy | ansiwen: yes, except that we can't do it directly via the nova heat resource, because we already pass an operator KeyName: | 18:40 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/compute-role.yaml#L130 | 18:41 |
| shardy | ansiwen: so you may need to install the additional key via cloud-init like this: | 18:41 |
| *** rasca has quit IRC | 18:41 | |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/firstboot/userdata_heat_admin.yaml#L30 | 18:41 |
| shardy | ansiwen: does it have to be an additional keypair? | 18:42 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-docs: Uses Ceph/Hammer repos for Liberty and Mitaka only https://review.openstack.org/392987 | 18:42 |
| shardy | there is a heat nova keypair resource you can use to generate a new key | 18:42 |
| ansiwen | shardy: no, there must be just ANY keypair, the same on all compute nodes, so that live migration works. that is, any nova user must be able to log into another nova user on another node without password. (authorized_keys) | 18:43 |
| shardy | ansiwen: Ok, so that's more complex because you want to ssh between compute nodes, right? | 18:44 |
| ansiwen | shardy: right | 18:44 |
| ansiwen | shardy: I'm talking about ssh keypairs | 18:45 |
| *** achadha_ has joined #tripleo | 18:45 | |
| shardy | yeah, so just putting a public key on the nodes is not enough | 18:45 |
| ansiwen | shardy: and puppet-nova supports to deplay the private key and the authorized_keys content. | 18:45 |
| ansiwen | shardy: no | 18:45 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 18:46 |
| ansiwen | shardy: so, it would be easy to add a parameter to the THT nova service, where you can add a precreated key pair to the deployment. | 18:46 |
| shardy | ansiwen: Ok so you can just add the parameters and wire them in via config_settings to puppet-nova | 18:47 |
| ansiwen | shardy: but it would be nice, if this can be a dynamically created key-pair by default, because live-migration is something you would like to have by default I guess. but a fixed keypair would be a security problem of course | 18:47 |
| shardy | if you want to create a keypair you can use this http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::KeyPair-prop-save_private_key | 18:47 |
| *** jpich has quit IRC | 18:47 | |
| ansiwen | shardy: exactly, I have to get it into the config_settings, then the problem is solved | 18:48 |
| *** achadha has quit IRC | 18:48 | |
| shardy | ansiwen: yeah conditionally creating a heat keypair resource based on an EnableLiveMigration parameter or something would work | 18:48 |
| shardy | or you could create the keypair in a mistral workflow and pass it to heat | 18:48 |
| shardy | I'd probably try the heat based approach initially | 18:49 |
| ansiwen | shardy: that is OS::Nova::KeyPair ? | 18:49 |
| shardy | yup | 18:49 |
| *** achadha_ has quit IRC | 18:49 | |
| ansiwen | ok, cool, I'll look into that | 18:49 |
| ansiwen | thanks! | 18:49 |
| *** amoralej is now known as amoralej|off | 18:55 | |
| *** shardy has quit IRC | 18:57 | |
| *** gfidente has quit IRC | 19:00 | |
| *** fragatin_ has quit IRC | 19:01 | |
| *** fragatina has joined #tripleo | 19:02 | |
| *** panda|bbl is now known as panda | 19:06 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 19:10 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add worker config envs in toci_gate_test https://review.openstack.org/399146 | 19:15 |
| *** penick has quit IRC | 19:17 | |
| *** achadha has joined #tripleo | 19:18 | |
| *** arxcruz has quit IRC | 19:20 | |
| *** ctayal has joined #tripleo | 19:20 | |
| *** ctayal has quit IRC | 19:20 | |
| *** ctayal has joined #tripleo | 19:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: Test https://review.openstack.org/393415 | 19:22 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add zipl support for s390 architecture with SCSI boot https://review.openstack.org/386031 | 19:23 |
| *** rbrady is now known as rbrady-afk | 19:28 | |
| *** penick has joined #tripleo | 19:28 | |
| *** penick has quit IRC | 19:31 | |
| *** abregman has joined #tripleo | 19:34 | |
| *** oshvartz has joined #tripleo | 19:36 | |
| *** tbonds has quit IRC | 19:38 | |
| *** trown|lunch is now known as trown | 19:46 | |
| *** openstackgerrit has quit IRC | 19:48 | |
| *** openstackgerrit has joined #tripleo | 19:48 | |
| openstackgerrit | Merged openstack/diskimage-builder: Don't use ssh-keygen -A for init scripts https://review.openstack.org/378985 | 19:54 |
| openstackgerrit | Merged openstack/diskimage-builder: In disk-image-create, append to INSTALL_PACKAGES instead of clobbering. https://review.openstack.org/396702 | 20:07 |
| *** abregman is now known as abregman|afk | 20:08 | |
| openstackgerrit | Merged openstack/instack: Don't include openstack/common in flake8 exclude list https://review.openstack.org/398216 | 20:09 |
| openstackgerrit | Ade Lee proposed openstack/instack-undercloud: Add code to support novajoin in the undercloud https://review.openstack.org/399220 | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 20:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| *** jkilpatr has quit IRC | 20:12 | |
| *** rbrady-afk is now known as rbrady | 20:12 | |
| *** jkilpatr has joined #tripleo | 20:13 | |
| *** newmember has quit IRC | 20:22 | |
| *** oshvartz has quit IRC | 20:23 | |
| *** florianf_ has joined #tripleo | 20:23 | |
| *** florianf has quit IRC | 20:24 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: redhat-common add option to select networking service NetworkManager/network https://review.openstack.org/392170 | 20:40 |
| *** tbonds has joined #tripleo | 20:42 | |
| *** tiswanso has quit IRC | 20:46 | |
| *** kbyrne has quit IRC | 20:49 | |
| *** kbyrne has joined #tripleo | 20:52 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 20:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement https://review.openstack.org/378014 | 20:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 20:56 |
| *** ayoung has quit IRC | 20:59 | |
| *** penick has joined #tripleo | 21:03 | |
| *** nyechiel has joined #tripleo | 21:05 | |
| *** iranzo has quit IRC | 21:05 | |
| *** ctayal has quit IRC | 21:06 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 21:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| *** penick has quit IRC | 21:12 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Devmode: reposition convert and update https://review.openstack.org/397999 | 21:12 |
| *** jayg is now known as jayg|g0n3 | 21:17 | |
| *** Goneri has quit IRC | 21:18 | |
| *** penick has joined #tripleo | 21:18 | |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-puppet-elements: Add puppet-qdr module https://review.openstack.org/373488 | 21:23 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 21:28 |
| *** jkilpatr has quit IRC | 21:29 | |
| *** penick has quit IRC | 21:40 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Transition to quickstart: make toci_gate_test.sh a symlink to toci_gate_orig.sh. https://review.openstack.org/396451 | 21:42 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Transition to quickstart: add toci_gate_oooq.sh and nonha job configuration https://review.openstack.org/399256 | 21:42 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: DO NOT MERGE Transition to quickstart: simulate transition https://review.openstack.org/399257 | 21:42 |
| *** dsavineau has quit IRC | 21:47 | |
| *** michapma_alt has joined #tripleo | 21:51 | |
| *** yamahata has quit IRC | 21:52 | |
| *** jkilpatr has joined #tripleo | 21:53 | |
| *** yamahata has joined #tripleo | 21:53 | |
| *** penick has joined #tripleo | 21:56 | |
| *** tbonds has quit IRC | 21:57 | |
| *** sshnaidm_ has joined #tripleo | 21:58 | |
| *** sshnaidm_ has quit IRC | 21:58 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** akrivoka has quit IRC | 22:00 | |
| *** abehl has quit IRC | 22:02 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Stop running liberty jobs https://review.openstack.org/399266 | 22:03 |
| bnemec | ^If you're tired of waiting for testenvs because patches against tripleo-ci use so damn many^ | 22:03 |
| * bnemec is, in case you hadn't guessed | 22:05 | |
| sshnaidm | slagle, EmilienM can you please re-review the quickstart job? I addressed your comments, thanks | 22:07 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add feature testing matrix to readme https://review.openstack.org/399269 | 22:09 |
| *** rhallisey has quit IRC | 22:10 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 22:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| *** rhallisey has joined #tripleo | 22:10 | |
| *** myoung is now known as myoung|bbl | 22:11 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Add panko api support to service templates https://review.openstack.org/396439 | 22:13 |
| slagle | sshnaidm: i will | 22:14 |
| *** rhallisey has quit IRC | 22:17 | |
| *** mhenkel has quit IRC | 22:17 | |
| *** jcoufal has quit IRC | 22:22 | |
| *** penick has quit IRC | 22:23 | |
| *** florianf_ has quit IRC | 22:30 | |
| *** jprovazn has quit IRC | 22:31 | |
| *** hjensas has joined #tripleo | 22:32 | |
| *** hjensas has quit IRC | 22:32 | |
| *** hjensas has joined #tripleo | 22:32 | |
| *** Goneri has joined #tripleo | 22:38 | |
| panda | can I have another +2 here ? https://review.openstack.org/391197 | 22:45 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates: No longer hard coding to a specifc network interface name. https://review.openstack.org/399040 | 22:56 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 23:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** ayoung has joined #tripleo | 23:10 | |
| *** pradk has quit IRC | 23:17 | |
| *** Goneri has quit IRC | 23:20 | |
| *** tiswanso has joined #tripleo | 23:21 | |
| *** tiswanso has quit IRC | 23:25 | |
| dtrainor | anyone mind helping me with a failed deployment? I had an issue once back in the day with galera-ready but this does not appear to be it, but something different http://paste.openstack.org/show/589651/ | 23:31 |
| dtrainor | i'm using an older osp10 build (a day old) | 23:32 |
| dtrainor | it looks like pacemaker has issues setting up galera | 23:33 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 23:33 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add feature testing matrix to readme https://review.openstack.org/399269 | 23:33 |
| dtrainor | it really looks like https://review.openstack.org/#/c/382883/ but I've confirmed that this is applied in my puppet-tripleo package... | 23:39 |
| *** bnemec has quit IRC | 23:42 | |
| openstackgerrit | Kevin Jones proposed openstack/tripleo-validations: Adds validation to check stack delete policy on undercloud https://review.openstack.org/399297 | 23:51 |
| *** rhallisey has joined #tripleo | 23:51 | |
| dtrainor | kjw3++ | 23:52 |
| *** nyechiel has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!