| *** mburned is now known as mburned_out | 00:18 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-apply-config package driven https://review.openstack.org/366404 | 00:22 |
|---|---|---|
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-refresh-config element package driven https://review.openstack.org/366405 | 00:22 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-collect-config element package driven https://review.openstack.org/366403 | 00:22 |
| *** thrash is now known as thrash|g0ne | 00:40 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Manage Redis VIP when deploying with keepalived https://review.openstack.org/364916 | 00:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Set Redis VIP on all nodes https://review.openstack.org/366128 | 00:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use Redis VIP when deploying with keepalived https://review.openstack.org/364917 | 00:50 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix https://review.openstack.org/365117 | 00:50 |
| *** limao has joined #tripleo | 00:53 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Test scenario001, 002, 003 https://review.openstack.org/366427 | 00:54 |
| *** xuao has joined #tripleo | 01:13 | |
| *** bfournie has joined #tripleo | 01:16 | |
| *** bana_k has quit IRC | 01:21 | |
| *** fultonj has quit IRC | 01:25 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Test scenario001, 002, 003 https://review.openstack.org/366427 | 01:26 |
| openstackgerrit | Merged openstack/python-tripleoclient: Add libffi-dev to bindep.txt https://review.openstack.org/366413 | 01:31 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-apply-config package driven https://review.openstack.org/366404 | 01:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-refresh-config element package driven https://review.openstack.org/366405 | 01:40 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-collect-config element package driven https://review.openstack.org/366403 | 01:40 |
| *** lblanchard has quit IRC | 01:40 | |
| *** yamahata has quit IRC | 01:44 | |
| *** xuao has quit IRC | 02:07 | |
| *** Ryjedo_ has joined #tripleo | 02:13 | |
| *** Ryjedo has quit IRC | 02:14 | |
| *** Ryjedo_ is now known as Ryjedo | 02:14 | |
| *** akshai has joined #tripleo | 02:23 | |
| *** maeca2 has quit IRC | 02:25 | |
| *** akshai has quit IRC | 02:30 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-apply-config package driven https://review.openstack.org/366404 | 02:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-refresh-config element package driven https://review.openstack.org/366405 | 02:35 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-image-elements: Make os-collect-config element package driven https://review.openstack.org/366403 | 02:35 |
| *** cwolferh has quit IRC | 02:37 | |
| *** akshai has joined #tripleo | 02:42 | |
| *** kjw3 has quit IRC | 02:46 | |
| *** bana_k has joined #tripleo | 02:52 | |
| *** cwolferh has joined #tripleo | 03:02 | |
| *** tzumainn has quit IRC | 03:04 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format https://review.openstack.org/364748 | 03:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 03:04 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Get template contents from plan, not local path https://review.openstack.org/365735 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add bootstrap_node and vip_data to hierarchy for all roles https://review.openstack.org/366049 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add VIP names to allNodesConfig https://review.openstack.org/365895 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts https://review.openstack.org/357765 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add parameters for internal TLS https://review.openstack.org/365942 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service https://review.openstack.org/356430 | 03:06 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services https://review.openstack.org/366075 | 03:07 |
| *** bana_k has quit IRC | 03:14 | |
| *** fragatin_ has joined #tripleo | 03:44 | |
| *** links has joined #tripleo | 03:45 | |
| *** fragatin_ has quit IRC | 03:46 | |
| *** fragatina has quit IRC | 03:46 | |
| *** coolsvap_ has joined #tripleo | 03:50 | |
| *** fragatina has joined #tripleo | 03:53 | |
| *** fragatina has quit IRC | 03:57 | |
| *** padkrish has joined #tripleo | 04:00 | |
| *** akshai has quit IRC | 04:27 | |
| *** bvandenh has joined #tripleo | 04:30 | |
| *** oshvartz has quit IRC | 04:31 | |
| *** bana_k has joined #tripleo | 04:38 | |
| sshnaidm | morning | 04:42 |
| *** akshai has joined #tripleo | 04:48 | |
| *** bkopilov_ has joined #tripleo | 04:50 | |
| *** abregman has joined #tripleo | 04:57 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: create release files that link to master and master-testing for promotion jobs https://review.openstack.org/366488 | 05:03 |
| *** masco has joined #tripleo | 05:03 | |
| *** bvandenh has quit IRC | 05:04 | |
| *** honza has quit IRC | 05:08 | |
| *** yamahata has joined #tripleo | 05:11 | |
| *** jaosorior has joined #tripleo | 05:12 | |
| *** openstackgerrit has quit IRC | 05:18 | |
| *** openstackgerrit has joined #tripleo | 05:18 | |
| *** akshai has quit IRC | 05:24 | |
| *** akshai has joined #tripleo | 05:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 05:35 |
| *** akshai has quit IRC | 05:39 | |
| jaosorior | Any +A for this? https://review.openstack.org/#/c/365895/4 :D | 05:41 |
| *** matbu|bbl is now known as matbu | 05:42 | |
| sshnaidm | jaosorior, can you please take a look at https://review.openstack.org/#/c/365369/ ? I don't understand why it doesn't work | 05:54 |
| sshnaidm | jaosorior, seems like these hiera settings are not applied - no changes in configuration | 05:55 |
| *** tremble has quit IRC | 05:55 | |
| *** masco_ has joined #tripleo | 06:03 | |
| *** dsariel has joined #tripleo | 06:04 | |
| *** rajinir has quit IRC | 06:05 | |
| *** masco has quit IRC | 06:07 | |
| *** pgadiya has joined #tripleo | 06:13 | |
| *** leanderthal|afk is now known as leanderthal | 06:17 | |
| *** oshvartz has joined #tripleo | 06:17 | |
| *** shardy has joined #tripleo | 06:19 | |
| jaosorior | sshnaidm: that will only work on master, by the way | 06:19 |
| jaosorior | but lets see | 06:19 |
| shardy | morning all | 06:20 |
| jaosorior | shardy: supd due | 06:20 |
| jaosorior | * sup dude | 06:20 |
| sshnaidm | jaosorior, yeah, these changes were introduced by dtantsur|afk recently | 06:20 |
| shardy | https://review.openstack.org/#/c/364748 and https://review.openstack.org/#/c/365763 ready for reviews if anyone has time to look | 06:20 |
| jaosorior | sshnaidm: are you sure those changes were included already in the image? | 06:22 |
| jaosorior | sshnaidm: you are right, they don't seem to have been applied | 06:22 |
| sshnaidm | jaosorior, I think so | 06:22 |
| *** jprovazn has joined #tripleo | 06:22 | |
| jaosorior | shardy: hey man | 06:23 |
| *** dbecker has quit IRC | 06:23 | |
| jaosorior | shardy: so for 365763 | 06:23 |
| jaosorior | shardy: to enable then a new node type, we will need to create a file puppet/<node type>-post.yaml? | 06:24 |
| *** radeks has joined #tripleo | 06:24 | |
| jaosorior | sorry | 06:24 |
| jaosorior | <node type>-config.yaml | 06:24 |
| jaosorior | shardy: they all seem quite similar | 06:25 |
| shardy | jaosorior: yes, although as mentioned we might be able to avoid that in future | 06:25 |
| jaosorior | shardy: how? | 06:25 |
| shardy | its already a huge patch so i didnt want to mess with the puppet stuff there | 06:25 |
| jaosorior | shardy: fair enough, just wanted to understand the direction of that series | 06:26 |
| shardy | jaosorior: not quite sure or i'd have done it already ;) | 06:26 |
| jaosorior | shardy: seems to me that the main issue is that get_file | 06:26 |
| shardy | yeah | 06:27 |
| shardy | we need to completely remove those almost empty manifests | 06:27 |
| shardy | I think we can set the node type in hiera and do the hiera_include etc in puppet-tripleo | 06:28 |
| shardy | just not tried it yet | 06:28 |
| *** padkrish has quit IRC | 06:29 | |
| jaosorior | shardy: I think that's the only way to go around that get_file. At least the only way available at the moment. | 06:30 |
| shardy | yeah, but we've been aiming to remove the per-role manifests anyway | 06:30 |
| jaosorior | sshnaidm: can you pass a link to the puppet-tripleo change that added those? | 06:30 |
| shardy | so we just need to finish that | 06:30 |
| shardy | IMO it's not super high priority, and custom-roles can work fine with this small duplication for now | 06:30 |
| sshnaidm | jaosorior, it's puppet-ironic: https://github.com/openstack/puppet-ironic/commit/d14c611c4bbc979911e6ced0cbdc55e44a3a7fa9 | 06:30 |
| jaosorior | shardy: would still be nice to have a cleaner solution. | 06:31 |
| shardy | jaosorior: yeah, I agree, will try to look into it as a followup | 06:32 |
| jaosorior | shardy: Hey man, by the way, can you check this commit out? https://review.openstack.org/#/c/365475/ | 06:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add bootstrap_node and vip_data to hierarchy for all roles https://review.openstack.org/366049 | 06:33 |
| *** pcaruana has joined #tripleo | 06:34 | |
| jaosorior | sshnaidm: it seems to me that it's not being included | 06:35 |
| jaosorior | sshnaidm: I didn't find any trace of it here http://logs.openstack.org/69/365369/9/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/95d4db1/logs/undercloud/var/log/undercloud_install.txt.gz | 06:35 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 06:36 |
| jaosorior | sshnaidm: I think we still don't get the version of puppet-ironic in order to use that | 06:36 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/365625 | 06:36 |
| shardy | jaosorior: Yeah, looks OK, as previously mentioned I suspect it can be simplified but I can help with that later | 06:36 |
| shardy | jaosorior: my only other comment is the same data could be derived in puppet by combining the $service_vip with $network_virtual_ip lookups | 06:37 |
| shardy | but this solution is a little more convenient I guess | 06:37 |
| sshnaidm | jaosorior, it used puppet-ironic-9.2.0-0.20160905145838.d14c611.el7.centos.noarch with hash d14c611 and it's exactly the hash of this change | 06:37 |
| *** mcornea has joined #tripleo | 06:37 | |
| sshnaidm | jaosorior, it's last change of puppet-ironic actually | 06:38 |
| jaosorior | sshnaidm: well, the logs show that ironic::conductor is being included, but the ironic::drivers::agent is nowhere to be seen | 06:38 |
| sshnaidm | jaosorior, where do you see it? | 06:38 |
| jaosorior | sshnaidm: this log http://logs.openstack.org/69/365369/9/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/95d4db1/logs/undercloud/var/log/undercloud_install.txt.gz | 06:38 |
| jaosorior | sshnaidm: grep for Ironic::Conductor | 06:39 |
| sshnaidm | jaosorior, yeah, I see now | 06:39 |
| jaosorior | sshnaidm: if you grep for "agent" (ignoring caps) only thing that shows up is neutron related stuff | 06:40 |
| sshnaidm | jaosorior, I would say it may be a problem | 06:40 |
| sshnaidm | jaosorior, maybe something in my override file is wrong?? | 06:40 |
| sshnaidm | jaosorior, syntax and all this | 06:40 |
| jaosorior | sshnaidm: the point with my comment is that, if nothing from ironic::drivers::agent is showing up in the logs. It means that the include for ironic::drivers::agent is not being done. Which will then ignore your overrides | 06:41 |
| sshnaidm | jaosorior, oh, I see | 06:42 |
| sshnaidm | that's odd | 06:42 |
| *** aufi has joined #tripleo | 06:42 | |
| jaosorior | sshnaidm: | 06:42 |
| jaosorior | ah | 06:42 |
| jaosorior | also | 06:42 |
| jaosorior | your syntax IS actually wrong | 06:42 |
| jaosorior | I'll fix it up quickly | 06:42 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 06:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Save console logs from all vms to files https://review.openstack.org/365369 | 06:43 |
| sshnaidm | jaosorior, thanks, I suspected it is.. | 06:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Save console logs from all vms to files https://review.openstack.org/365369 | 06:43 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 06:43 |
| jaosorior | sshnaidm: I fixed it up and rebased it on top of tripleo-ci. Just in case | 06:43 |
| sshnaidm | jaosorior, oh, two colons, thanks! | 06:43 |
| jaosorior | sshnaidm: this is what I changed https://review.openstack.org/#/c/365369/9..11/scripts/deploy.sh | 06:43 |
| jaosorior | sshnaidm: I'm still puzzled why ironic::drivers::agent didn't show up in the logs at all. As it was explicitly included in the commit by dtantsur|afk you showed me | 06:44 |
| jaosorior | buuut lets see what the next run says | 06:44 |
| sshnaidm | jaosorior, let's see.. | 06:44 |
| jaosorior | shardy: what do you mean you can derive the same info? | 06:45 |
| jaosorior | shardy: Ok, I'll show you in a bit what I want to do with that networkname. That I don't think I can derive from the VIP parts you mentioned | 06:45 |
| *** anshul has joined #tripleo | 06:46 | |
| jaosorior | just gotta brew coffee first :P | 06:46 |
| *** pgadiya has quit IRC | 06:47 | |
| *** bana_k has quit IRC | 06:48 | |
| *** anshul has quit IRC | 06:49 | |
| *** anshul has joined #tripleo | 06:49 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 06:52 |
| shardy | d0ugal: Hey, FYI I got the j2 templated deploys working via tripleoclient w/mistral yesterday | 06:53 |
| d0ugal | shardy: Nice one | 06:53 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated action names. https://review.openstack.org/366529 | 06:53 |
| shardy | d0ugal: https://review.openstack.org/#/c/365735 is the tripleoclient part (needs tests) feedback appreciated | 06:53 |
| d0ugal | shardy: I'll take a look now. | 06:53 |
| shardy | d0ugal: I think there's scope for simplifying the client side stuff, but for now I've just switched from pointing heatclient at the local tht_root, instead pointing it at the swift bucket | 06:54 |
| shardy | which heatclient supports, just in a slightly non-obvious way | 06:54 |
| d0ugal | shardy: right, it looks like it was fairly easy to fit in which is nice! | 06:55 |
| shardy | d0ugal: Yeah, it took a while to figure out but it's not too bad | 06:56 |
| shardy | d0ugal: also see the Depends-On patch, I added the process_templates action to the create_plan* workflows | 06:56 |
| shardy | so we do the j2 render when the plan is created | 06:56 |
| shardy | we'll also need to figure out how that step happens when an existing plan gets updated | 06:56 |
| shardy | but that looks like a problem in general that's not super well solved atm? | 06:56 |
| * shardy didn't see any update plan workflows | 06:57 | |
| d0ugal | shardy: hmm, good point, I don't think it is very well tested at least. | 06:58 |
| cmyster | did someone say test? | 06:58 |
| shardy | d0ugal: we need to fix that asap, like what happens if you run deploy --templates tht/old then deploy --templates tht/new ? | 06:58 |
| cmyster | morning | 06:58 |
| shardy | it looks like we skip creating the plan and just use the old one? | 06:59 |
| d0ugal | shardy: due to backwards compat, `openstack overcloud deploy --templates` pretty much ignores there is a plan if it exists and overwrites it. | 06:59 |
| d0ugal | shardy: because we couldn't have it change behaviour on a second deploy now we suddenly have a plan behind it | 06:59 |
| shardy | d0ugal: Ok, it wasn't clear if it just ignored it and used the old one, will test it out later | 06:59 |
| d0ugal | Really? I need to check that... that isn't what I had wanted :/ | 06:59 |
| d0ugal | I'll look at that now | 06:59 |
| shardy | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L418 | 07:00 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names. https://review.openstack.org/366529 | 07:00 |
| shardy | d0ugal: to be honest I was looking at it very late last night, so I could be wrong | 07:00 |
| d0ugal | shardy: damn, I think you are correct. | 07:00 |
| shardy | but I don't see where we update the container with tht/new when an existing deploy is updated | 07:00 |
| d0ugal | shardy: this probably got lost in all the rebasing | 07:01 |
| shardy | d0ugal: can you please raise a critical bug and target it at rc1 ? | 07:01 |
| d0ugal | shardy: Sure. | 07:01 |
| shardy | I don't think we can release without fixing it | 07:01 |
| shardy | thanks! | 07:01 |
| cmyster | shardy: are you talking about the new (simpler) tht templates that were getting ? | 07:02 |
| d0ugal | shardy: sure, I'll have a fix soon. | 07:02 |
| *** mhenkel has quit IRC | 07:02 | |
| *** tesseract- has joined #tripleo | 07:02 | |
| shardy | cmyster: No I'm talking about our new method of deploying which is tripleoclient talks to mistral and swift, vs directly to heat | 07:02 |
| shardy | cmyster: there's a few remaining issues we're working out | 07:02 |
| *** pgadiya has joined #tripleo | 07:03 | |
| *** jpena|off is now known as jpena | 07:03 | |
| d0ugal | shardy: https://bugs.launchpad.net/tripleo/+bug/1620932 | 07:03 |
| openstack | Launchpad bug 1620932 in tripleo "openstack overcloud deploy will use the old plan and wont update on a second deploy" [Critical,Confirmed] | 07:03 |
| shardy | d0ugal: thanks - if you've got bandwidth to help with a fix that would be great | 07:04 |
| d0ugal | shardy: doing it now | 07:04 |
| shardy | I can help with testing/reviews when it's ready | 07:04 |
| *** masco__ has joined #tripleo | 07:04 | |
| *** tremble has joined #tripleo | 07:04 | |
| d0ugal | shardy: The easy option is just to delete the plan if it exists, but that is a bit hacky - since we don't have any history anyway it is almost the same as updating the plan but easier and cleaner. what do you think? | 07:05 |
| *** mhenkel has joined #tripleo | 07:06 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:07 | |
| shardy | d0ugal: sounds OK as a first step, but I wonder if we'll need something more robust if we offer a create plan, modify plan, deploy scheme | 07:07 |
| shardy | vs just create plan, deploy | 07:07 |
| shardy | I guess it fixes the immediate regression tho | 07:07 |
| d0ugal | shardy: Yeah, agreed. I'll get that review up and then start looking at it properly. | 07:08 |
| shardy | dentist, bbiab | 07:08 |
| *** shardy is now known as shardy_afk | 07:08 | |
| *** masco_ has quit IRC | 07:08 | |
| *** ohamada has joined #tripleo | 07:10 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Cleanup the existing plan before deploying if one already exists https://review.openstack.org/366541 | 07:14 |
| *** zoliXXL is now known as zoli_gone-proxy | 07:17 | |
| *** masco__ is now known as masco | 07:23 | |
| *** zoli_gone-proxy is now known as zoliXXL | 07:24 | |
| *** jpich has joined #tripleo | 07:24 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Have docker start script honor configuration https://review.openstack.org/366138 | 07:27 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Add steps to containerized compute deployment https://review.openstack.org/346927 | 07:27 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles https://review.openstack.org/330659 | 07:27 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host https://review.openstack.org/347218 | 07:27 |
| *** abregman_ has joined #tripleo | 07:28 | |
| *** jlinkes has joined #tripleo | 07:29 | |
| *** ebarrera has joined #tripleo | 07:29 | |
| *** akuznetsov has joined #tripleo | 07:29 | |
| *** abregman has quit IRC | 07:31 | |
| *** rlandy|bbl is now known as rlandy | 07:31 | |
| *** rlandy has quit IRC | 07:31 | |
| *** florianf has joined #tripleo | 07:31 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 07:33 |
| *** ifarkas_afk is now known as ifarkas | 07:42 | |
| openstackgerrit | Tomas Sedovic proposed openstack/tripleo-common: Allow the validations to run openstack commands https://review.openstack.org/366175 | 07:44 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services https://review.openstack.org/366075 | 07:49 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service https://review.openstack.org/356430 | 07:49 |
| *** mbound has joined #tripleo | 07:51 | |
| *** abregman_ has quit IRC | 07:52 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Expand help message on config sample file https://review.openstack.org/365707 | 07:56 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 07:56 |
| *** shardy_afk is now known as shardy | 07:57 | |
| jaosorior | shardy: hey man, this is what I was planning to use the network for https://review.openstack.org/#/c/366548/1/manifests/haproxy/endpoint.pp | 08:00 |
| *** nyechiel_ has joined #tripleo | 08:03 | |
| *** jlinkes has quit IRC | 08:03 | |
| jaosorior | shardy: the other way to do it is to pass everything via t-h-t. But that we need to set up a big chunk of yaml with all the services and a reference to their specific certificate... So I figured this was the best way to do it in a composable service type of way. | 08:04 |
| *** masco_ has joined #tripleo | 08:04 | |
| shardy | jaosorior: ack, yeah that makes sense | 08:06 |
| shardy | thanks for the clarification | 08:06 |
| shardy | fits pretty well with the other $service_* stuff we have in hiera now | 08:06 |
| d0ugal | shardy: https://review.openstack.org/#/c/366541/ | 08:06 |
| *** abregman_ has joined #tripleo | 08:06 | |
| jaosorior | shardy: so what do you think? Did it make sense to you? Or do you think we could do it another way? | 08:06 |
| d0ugal | shardy: Just waiting for a deploy to finish to test it | 08:07 |
| *** akuznetsov has quit IRC | 08:08 | |
| *** tobias_fiberdata has joined #tripleo | 08:08 | |
| *** masco has quit IRC | 08:08 | |
| shardy | jaosorior: I think it's fine given the cert-per-network requirement | 08:09 |
| d0ugal | jtomasek: ping | 08:09 |
| jaosorior | shardy: alright. Will continue this path and test it out. Thanks for checking it out | 08:09 |
| jtomasek | d0ugal: pong | 08:09 |
| d0ugal | jtomasek: Can you point me to the UI code that creates a plan? | 08:10 |
| d0ugal | jtomasek: or do you only offer the default plan now? | 08:10 |
| jtomasek | d0ugal: sec | 08:10 |
| *** tobias-fiberdata has quit IRC | 08:11 | |
| shardy | d0ugal: ack - we also should follow-up on the update/upgrade testing CI patches, as we should ensure this is tested in future (including making a change to the plan/templates) | 08:11 |
| d0ugal | shardy: +1, shall I open another bug for that? | 08:11 |
| jtomasek | d0ugal: https://github.com/openstack/tripleo-ui/blob/master/src/js/actions/PlansActions.js#L198 | 08:11 |
| jtomasek | d0ugal: it is a series of api calls | 08:11 |
| d0ugal | jtomasek: You create the container with swift directly? | 08:12 |
| jtomasek | d0ugal: yes, as that is what the workflow expects as an input | 08:12 |
| jtomasek | florianf: ^ | 08:12 |
| d0ugal | jtomasek: k, you should sitch to the create_container action | 08:12 |
| d0ugal | otherwise the container will be missing the tripleo metadata | 08:13 |
| jtomasek | d0ugal: ook, what metadata is it? | 08:13 |
| d0ugal | jtomasek: It's just a key to mark it as a tripleo-managed container | 08:13 |
| d0ugal | jtomasek: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/plan.py#L29-L31 | 08:14 |
| jtomasek | d0ugal: won't that happen during plan creation? why does that need to be run before it? | 08:14 |
| florianf | d0ugal: https://github.com/openstack/tripleo-ui/blob/master/src/js/services/SwiftApiService.js#L28 | 08:14 |
| d0ugal | jtomasek: because that needs to happen when you create the container AFAIK | 08:15 |
| florianf | d0ugal: we add a bit of metadata when creating the container. | 08:15 |
| d0ugal | florianf: oh :( | 08:15 |
| *** akuznetsov has joined #tripleo | 08:15 | |
| d0ugal | it would be better to use the action - I'll open a bug. | 08:16 |
| jtomasek | d0ugal, florianf: I think we can do that | 08:16 |
| florianf | d0ugal: the metadata was exactly my concern about creating the plan via swift. because we basically need to keep the business logic of tripleo-common and tripleo-ui in sync | 08:16 |
| jtomasek | florianf: what metadata do we add? is it still relevant to mistral driven plan management? | 08:16 |
| d0ugal | florianf: unless you use the action and then it is all the same :) | 08:16 |
| florianf | d0ugal: Of course, it makes things slightly more stable to use the action | 08:16 |
| florianf | d0ugal: But what if tripleo-common decides to add meta data to each object? | 08:17 |
| jtomasek | yeah, not possible:) | 08:17 |
| d0ugal | florianf: well, we would only do that in an action or a workflow? | 08:17 |
| d0ugal | florianf: but "what if" could be said about many things ;) | 08:17 |
| florianf | d0ugal: Sure, but that's why we advocated to create a plan using *only* the tripleo-common api and ignore swift | 08:18 |
| d0ugal | florianf: I didn't disagree, I wanted it too. | 08:18 |
| florianf | d0ugal: Let the API take care of storage | 08:18 |
| shardy | d0ugal: sure, please do, and tag it with ci | 08:18 |
| d0ugal | florianf: I also don't want anyone to use actions directly | 08:18 |
| *** masco_ is now known as masco | 08:18 | |
| jtomasek | florianf, d0ugal: I think we're going to get there, eventually | 08:18 |
| * jpich side-glance to the JSON workflow patch | 08:19 | |
| florianf | d0ugal: But yeah, it still totally make sense to use the action for container creation. | 08:19 |
| openstackgerrit | Martin André proposed openstack/puppet-tripleo: Manage tripleo-ui configuration files with puppet https://review.openstack.org/363167 | 08:20 |
| d0ugal | florianf: +1, I'll open a bug, it's non-urgent :) | 08:20 |
| florianf | d0ugal: thanks! | 08:20 |
| florianf | jpich: :-) | 08:20 |
| *** tobias_fiberdata has quit IRC | 08:21 | |
| jbadiapa | I tried to test tripleo-quickstart with newton and apparently the links are broken. I meant there is no undercloud.qcow2 at http://images.rdoproject.org/mitaka/delorean/consistent/stable/undercloud.qcow2 | 08:22 |
| *** lucas-dinner is now known as lucasagomes | 08:22 | |
| jbadiapa | can anyone help to solve this? | 08:23 |
| jaosorior | jbadiapa: you might want to ask in #oooq | 08:23 |
| jaosorior | jbadiapa: there's most of the people that could help with that | 08:24 |
| *** hjensas has joined #tripleo | 08:24 | |
| jbadiapa | jaosorior, thanks | 08:24 |
| * shardy feels sad that has a different channel | 08:24 | |
| *** jlinkes has joined #tripleo | 08:24 | |
| jaosorior | shardy: I actually don't know why it's a different channel either :/ | 08:24 |
| jaosorior | it's not like theres a lot of traffic in this one | 08:25 |
| *** aqkhan_ has joined #tripleo | 08:25 | |
| shardy | It's part of a broader problem which is that oooq was never fully integrated upstream, e.g in our CI/docs etc | 08:25 |
| jaosorior | :( | 08:26 |
| shardy | hopefully we can discuss the way forward with that for Ocata | 08:26 |
| jaosorior | I think it would be great to include tripleo-quickstart in the tripleo-docs | 08:26 |
| jaosorior | but yeah, probably something to talk about in Barcelona | 08:26 |
| shardy | I would support that, but not until we're testing it in upstream CI | 08:26 |
| *** aqkhan__ has joined #tripleo | 08:26 | |
| *** aqkhan has quit IRC | 08:26 | |
| jaosorior | shardy: I think sshnaidm had a commit for that | 08:26 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-ui: Update the Mistral action names to use the new version https://review.openstack.org/366579 | 08:27 |
| *** tobias_fiberdata has joined #tripleo | 08:27 | |
| * shardy rebases patches due to merge-confict | 08:27 | |
| shardy | sigh | 08:27 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names. https://review.openstack.org/366529 | 08:27 |
| jaosorior | shardy: haha I know the feel. This chain is gonna be a real pain https://review.openstack.org/#/c/366075/ | 08:27 |
| shardy | Yeah, just frustrating, I expected those two patches to land this morning | 08:28 |
| shardy | oh well, same for all of us :) | 08:28 |
| *** liverpooler has joined #tripleo | 08:29 | |
| jaosorior | shardy: they already had a +A, right? | 08:29 |
| shardy | Only one of them | 08:29 |
| *** paramite has joined #tripleo | 08:29 | |
| jaosorior | shardy: I'll just +A it again. | 08:30 |
| sshnaidm | jaosorior, shardy oooq patch, fyi: https://review.openstack.org/#/c/358919/ | 08:30 |
| jaosorior | shardy: the other one was the one moving the swift stuff, right? | 08:30 |
| shardy | jaosorior: Yeah, the swift one is the top of the branch | 08:30 |
| shardy | we'll need to wait for CI anyway | 08:30 |
| *** aqkhan_ has quit IRC | 08:30 | |
| jaosorior | sshnaidm: nice! | 08:31 |
| *** flepied has quit IRC | 08:32 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 08:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format https://review.openstack.org/364748 | 08:33 |
| *** akuznetsov has quit IRC | 08:33 | |
| *** openstackgerrit has quit IRC | 08:34 | |
| *** openstackgerrit has joined #tripleo | 08:34 | |
| *** akuznetsov has joined #tripleo | 08:35 | |
| *** akuznetsov has quit IRC | 08:36 | |
| *** derekh has joined #tripleo | 08:40 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml https://review.openstack.org/365792 | 08:40 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 08:40 |
| *** athomas has joined #tripleo | 08:41 | |
| sshnaidm | OMG, delorean setup takes 45(!) minutes in jobs | 08:43 |
| sshnaidm | and they tell me there is no AFS mirror issues | 08:43 |
| *** masco is now known as masco|lunch | 08:43 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Expand help message on config sample file https://review.openstack.org/365707 | 08:45 |
| *** dtantsur|afk is now known as dtantsur | 08:46 | |
| *** fzdarsky has joined #tripleo | 08:47 | |
| shardy | sshnaidm: that can't be right, surely? | 08:50 |
| shardy | it takes a few seconds locally | 08:50 |
| sshnaidm | shardy, either it gets socket timeout and fails or takes about 45 mins | 08:51 |
| *** karthiks has quit IRC | 08:53 | |
| *** links has quit IRC | 08:53 | |
| *** anshul has quit IRC | 08:53 | |
| shardy | real1m18.457s | 08:54 |
| *** skramaja has quit IRC | 08:54 | |
| *** skramaja_ has joined #tripleo | 08:54 | |
| shardy | sshnaidm: Hope we can fix that. I guess it also impacts image building too? | 08:54 |
| *** pgadiya has quit IRC | 08:54 | |
| shardy | I know we have that cached now, but some jobs will still need to build images | 08:54 |
| *** anshul has joined #tripleo | 08:54 | |
| *** links has joined #tripleo | 08:55 | |
| *** pgadiya has joined #tripleo | 08:55 | |
| *** skramaja_ is now known as skramaja | 08:55 | |
| *** karthiks has joined #tripleo | 08:55 | |
| sshnaidm | shardy, it affects every job, we install delorean anyway, looking into.. | 08:56 |
| *** mbound has quit IRC | 08:56 | |
| sshnaidm | who changed scripts/te-broker/create-env on te-broker machine? | 08:57 |
| sshnaidm | derekh, bnemec ^^ | 08:57 |
| *** saneax-_-|AFK is now known as saneax | 08:57 | |
| derekh | sshnaidm: I think bnemec did, he mentioned it in his email last night | 08:58 |
| sshnaidm | derekh, ok, seems like there is na error in it, but it affects only failed cases, so it's ok | 08:59 |
| derekh | sshnaidm: where is the setup taking 45 minutes | 08:59 |
| derekh | sshnaidm: ok | 08:59 |
| sshnaidm | derekh, pip install, using afs mirrors | 08:59 |
| sshnaidm | derekh, pip installs delorean for a 45 minutes, or just disconnects | 09:00 |
| sshnaidm | derekh, seems like we have the same problem as yesterday in addition | 09:01 |
| derekh | sshnaidm: hmm, I don't see it on any jobs I've looked at so far, can you point me at an example? | 09:02 |
| sshnaidm | derekh, http://logs.openstack.org/15/359215/7/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/2946607/console.html#_2016-09-07_06_25_57_316738 | 09:03 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 09:03 |
| jtomasek | d0ugal: the new action names are now merged in tripleo-common, right? | 09:04 |
| d0ugal | jtomasek: yup | 09:04 |
| derekh | sshnaidm: thats a job failing to get an env, I was looking for an example of a 45 minute delorean setup | 09:04 |
| *** masco_ has joined #tripleo | 09:05 | |
| *** limao has quit IRC | 09:05 | |
| sshnaidm | derekh, oh, just a sec | 09:05 |
| *** limao has joined #tripleo | 09:06 | |
| *** limao has quit IRC | 09:07 | |
| *** masco|lunch has quit IRC | 09:09 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove "type: direct" from workflows as it is the default https://review.openstack.org/341617 | 09:09 |
| sshnaidm | derekh, telnet://66.187.229.108:19885 | 09:10 |
| sshnaidm | still running | 09:10 |
| *** akuznetsov has joined #tripleo | 09:10 | |
| shardy | dtantsur: Hey, if we land this docs patch, is that the ironic-integration BP completed? | 09:12 |
| shardy | https://review.openstack.org/#/c/354016/ | 09:12 |
| shardy | nvm, sorry just refreshed and saw your reply ;) | 09:12 |
| shardy | sorry for the noise | 09:12 |
| shardy | can someone please review ^^ | 09:13 |
| dtantsur | :) | 09:13 |
| *** flepied has joined #tripleo | 09:13 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Implement Validation Detail modal https://review.openstack.org/365921 | 09:14 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Run Validations automatically https://review.openstack.org/366068 | 09:14 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: ModalPanel component https://review.openstack.org/366615 | 09:14 |
| dtantsur | shardy, so yeah, feature-wise it's complete. I might eventually extend these docs to cover two NICs, but I should figure it out myself first. this works and is safe to land IMO | 09:14 |
| shardy | dtantsur: ack, thanks | 09:14 |
| shardy | just trying to complete all-the-things for rc1 | 09:15 |
| dtantsur | fair call :) | 09:15 |
| dtantsur | also good morning/afternoon everyone | 09:15 |
| *** limao has joined #tripleo | 09:16 | |
| *** flepied has quit IRC | 09:16 | |
| derekh | sshnaidm: wow, that did take a long time | 09:16 |
| dtantsur | shardy, btw, does the host aggregates thing make sense to you? it's the first time I'm doing something like that. works for me locally, but dunno if it can be made simpler/saner... | 09:17 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Fixed nic numbering issue of DPDK nics after the nic has bound https://review.openstack.org/364354 | 09:18 |
| shardy | dtantsur: Yeah it does, but I suspect we'll be looking for ways to simplify the docs overall during Ocata | 09:19 |
| shardy | seems fine to me for a first pass tho, thanks for including so much detail | 09:19 |
| derekh | sshnaidm: the AFS mirror seems to have gotten fast again now, I wonder was the repository updated or something | 09:19 |
| dtantsur | shardy, yeah, I was essentially dumping everything I was doing while testing it :) | 09:19 |
| shardy | dtantsur: Yeah, I kinda guessed that :) | 09:20 |
| shardy | seems like a good starting point | 09:20 |
| derekh | sshnaidm: testing it like this, the first download of each file is slow and then it gets faster http://paste.openstack.org/show/567381/ | 09:20 |
| shardy | karthiks: Hey I spotted an issue with https://review.openstack.org/#/c/361367 | 09:20 |
| derekh | sshnaidm: although 40 minutes still seems too long | 09:21 |
| shardy | please see if you agree, I think the yaml list syntax is wrong | 09:21 |
| *** yamahata has quit IRC | 09:21 | |
| dtantsur | on a bright side, I've learned so many new things while working on it :) | 09:21 |
| *** flepied has joined #tripleo | 09:21 | |
| shardy | would be good to get some confirmation this has been tested, as I don't think that will work as posted | 09:21 |
| karthiks | shardy, Will look in to | 09:21 |
| skramaja | shardy: i guess beagles just followed as like https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/neutron-api.yaml#L78 | 09:23 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 09:23 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: create release files for consistent and current-tripleo https://review.openstack.org/366488 | 09:23 |
| skramaja | shardy: i will test it and revert back. | 09:26 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Make sure major upgrade script fails. https://review.openstack.org/366623 | 09:28 |
| shadower | are the OVB jobs busted again or just generally flakey? | 09:31 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 09:31 |
| sshnaidm | derekh, first download problem I know from my dev envs runs, it's usual thing there, didn't have a chance to handle this | 09:31 |
| shardy | skramaja: weird, I don't see how that can work | 09:32 |
| shardy | it's not valid yaml | 09:32 |
| skramaja | shardy: even for dpdk, i have did it in the same way - https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/neutron-ovs-dpdk-agent.yaml#L68 | 09:32 |
| shardy | skramaja: I'll test it too, I suspect the evaluation order of the functions means we're getting away with it | 09:32 |
| skramaja | shardy: and i have tested and validated it. i will cross check again and will revert back. | 09:32 |
| *** r-mibu has quit IRC | 09:33 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add VIP names to allNodesConfig https://review.openstack.org/365895 | 09:33 |
| *** r-mibu has joined #tripleo | 09:33 | |
| dtantsur | how to screw up your tripleo deployment: 1. expect that overcloud-full contains the latest puppet modules, 2. done! >_< | 09:34 |
| shardy | upload-puppet-modules ftw | 09:35 |
| *** masco_ is now known as masco | 09:35 | |
| shardy | make sure you have the latest tripleo-common though as I landed a fix to it yesterday | 09:35 |
| jaosorior | shardy: derekh seems that the ovb gate is broken again (seen 4 commits fail very early on the process just now) | 09:35 |
| jaosorior | derekh: http://logs.openstack.org/17/341617/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/7f1d40d/console.html#_2016-09-07_09_21_52_651427 | 09:36 |
| dtantsur | shardy, yeah... I wonder how I ended up with overcloud-full not matching undercloud, provided that I've built the image myself... | 09:36 |
| shardy | karthiks: http://paste.openstack.org/show/567388/ shows the error I'm referring to | 09:37 |
| skramaja | shardy: i am looking in to.. | 09:38 |
| shardy | skramaja: actually, my example was wrong, it works! | 09:39 |
| shardy | http://paste.fedoraproject.org/423282/41179147 | 09:39 |
| shardy | the fact that it works is a heat bug IMO | 09:40 |
| shardy | so we should change it anyway | 09:40 |
| *** akuznetsov has quit IRC | 09:40 | |
| *** akuznetsov has joined #tripleo | 09:41 | |
| skramaja | shardy: ok. got it. we will rework both sriov and dpdk templates. | 09:41 |
| derekh | jaosorior: all kinds of random things are still happening on the controller and we havn't nailed down the problem, for one you linked we got | 09:41 |
| derekh | 2016-09-07 09:20:43.892 19121 ERROR heat.engine.resource MessagingTimeout: Timed out waiting for a reply to message ID 4c6911b7a50d40c18438f37e95d09c30 | 09:41 |
| *** zoliXXL is now known as zoli|lunch | 09:41 | |
| jaosorior | derekh: ah, same stuff as yesterday then? | 09:41 |
| jaosorior | derekh: sorry for the noise. Thought this was a different issue. | 09:42 |
| derekh | jaosorior: looks like it | 09:42 |
| *** zoli|lunch is now known as zoli_gone-proxy | 09:42 | |
| shardy | skramaja: actually, it doesn't work - the error is sliently ignored, but the output format is wrong | 09:42 |
| shardy | http://paste.openstack.org/show/567390/ | 09:42 |
| shardy | I'll raise a heat bug | 09:42 |
| skramaja | ok.. thanks shardy .. i will modify accordingly. | 09:43 |
| derekh | sshnaidm: I'm thinking we just try the reboot to make bnemec's cpu change, any one of those should make the situation better, do you know how to do the CPU thing? | 09:43 |
| *** apetrich has quit IRC | 09:44 | |
| derekh | shardy: btw, sshnaidm got heat-manage to work yesterday, something to do with a rc file that needed to be sourced | 09:45 |
| shardy | derekh: Ok, that sounds unexpected, a bug report would be good as it should at least fail with an error if it's missing variables from the env | 09:46 |
| derekh | shardy: ack, sshnaidm ^ | 09:46 |
| shardy | https://bugs.launchpad.net/heat/+bug/1620985 | 09:46 |
| openstack | Launchpad bug 1620985 in heat "map_merge accepts invalid input, produces unexpected output" [Undecided,New] | 09:46 |
| shardy | skramaja: ^^ FYI | 09:46 |
| jaosorior | sshnaidm: what did heat-manage need to work? | 09:47 |
| sshnaidm | derekh, he wrote it here: http://etherpad.corp.redhat.com/rh1-profile-switch | 09:48 |
| sshnaidm | jaosorior, I forgot to source rc file | 09:48 |
| sshnaidm | derekh, we talked about enabling rh2 yesterday | 09:50 |
| jaosorior | sshnaidm: why would you need to source the rc file? | 09:51 |
| sshnaidm | derekh, the problem was that pabelanger wanted to create openstackzuul tenant there and share networks between the current one | 09:51 |
| sshnaidm | jaosorior, I don't know, but it stared to work then | 09:51 |
| jaosorior | sshnaidm: heat-manage should need admin permissions from the system it's ran at, but shouldn't be using keystone :/ | 09:51 |
| jaosorior | damn, that's weird dude | 09:51 |
| shardy | it definitely shouldn't need the rc file to purge DB data, and it shouldn't fail silently | 09:53 |
| shardy | so at least one bug there | 09:53 |
| shardy | sshnaidm: can you please raise a bug report against heat showing what you found? | 09:53 |
| derekh | sshnaidm: I'm not sure rh2 would be enough, we've renabled a bunch of jobs and would end up with queues days long | 09:54 |
| sshnaidm | shardy, sorry, which exactly problem to report? | 09:54 |
| shardy | sshnaidm: that heat-manage silently did nothing until you sourced an rc file | 09:55 |
| sshnaidm | shardy, oh, heat-manage, ok | 09:55 |
| shardy | it probably shouldn't need the RC file, and if it does then it should really fail loudly when it's missing | 09:55 |
| shardy | some weird stuff got added to heat-manage the last couple of cycles, but I don't see why we'd need the rc file to purge the DB | 09:55 |
| *** radeks has quit IRC | 09:55 | |
| jaosorior | shardy: WTF... heat-manage requires admin context from keystone | 09:57 |
| jaosorior | that makes no sense... | 09:57 |
| jaosorior | shardy: I see this was the thing https://review.openstack.org/#/c/146044/ | 09:57 |
| shardy | jaosorior: ack, makes sense - still don't see why it needs it for purge, probably a bug | 09:59 |
| derekh | sshnaidm: bnemec I can't get a console to change the biod setting on the controller because "This feature requires an iDRAC Enterprise license. For more details on how to obtain a license, visit License Page." | 09:59 |
| jaosorior | shardy: seems like it. It should only need it for the heat services status | 09:59 |
| derekh | zoli_gone-proxy: ^^ any ideas ? https://10.1.8.22 | 09:59 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates: Add base neutron service configuration https://review.openstack.org/361367 | 10:00 |
| *** limao has quit IRC | 10:01 | |
| *** limao has joined #tripleo | 10:01 | |
| sshnaidm | derekh, could you please check before this: /opt/stack/tripleo-ci/scripts/te-broker/create-env on centos@te-broker machine | 10:02 |
| sshnaidm | derekh, last lines there | 10:03 |
| *** limao has quit IRC | 10:03 | |
| sshnaidm | derekh, I see in te logs a lot of repeating creating env, but it seems to succeed each time.. or I don't understand something | 10:04 |
| derekh | sshnaidm: is that what bnemec put in the file? | 10:05 |
| sshnaidm | derekh, yes | 10:05 |
| *** masco_ has joined #tripleo | 10:05 | |
| derekh | sshnaidm: it looks ok to me, this isn't the part where we create a env, this is where we get details about the env | 10:06 |
| derekh | sshnaidm: last night we noticed that the envs sometimes got created correctly but then | 10:06 |
| derekh | sshnaidm: build-nodes-json hit errors when getting details about the env | 10:06 |
| derekh | sshnaidm: so bnemec added a retry to get the env details | 10:07 |
| derekh | I wonder should thse be cleared http://paste.openstack.org/show/567398/ | 10:08 |
| *** fultonj has joined #tripleo | 10:08 | |
| *** masco has quit IRC | 10:09 | |
| sshnaidm | derekh, and /var/log/testenv-worker.log looks ok? | 10:10 |
| sshnaidm | derekh, yeah, rabbit is the monster in controller | 10:12 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks. https://review.openstack.org/357750 | 10:12 |
| *** athomas has quit IRC | 10:14 | |
| derekh | sshnaidm: actually your correct, something is wrong with create-env, its retrying to call build-nodes-json when it doesn't need too and eventually gives up | 10:17 |
| derekh | sshnaidm: I think the break isn'#t working because it in a subshell, tesating something | 10:18 |
| sshnaidm | derekh, yeah, looks like this | 10:18 |
| sshnaidm | and hav_json.. is always 0 | 10:19 |
| *** ohamada has quit IRC | 10:20 | |
| *** ohamada has joined #tripleo | 10:20 | |
| derekh | sshnaidm: updated the file, can you see what you think | 10:21 |
| jaosorior | shadower: ovb jobs are currently broken, so if you tried to recheck because of that it's not gonna work yet :/ | 10:22 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: Get template contents from plan, not local path https://review.openstack.org/365735 | 10:23 |
| *** athomas has joined #tripleo | 10:23 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks. https://review.openstack.org/357750 | 10:25 |
| shadower | jaosorior: oh ok :-( | 10:26 |
| sshnaidm | derekh, yep, should work, let's see | 10:26 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING https://review.openstack.org/316436 | 10:27 |
| sshnaidm | derekh, did you see this? http://en.community.dell.com/techcenter/b/techcenter/archive/2012/05/24/idrac7-virtual-console-enhanced-security-checks-in-the-integrated-dell-remote-access-controller | 10:28 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks. https://review.openstack.org/357750 | 10:29 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:29 |
| sshnaidm | derekh, is there described a problem with your login to drac? | 10:29 |
| derekh | sshnaidm: I'm able to login to drac, and poke around a things, but the place where the console should be it just says "This feature requires an iDRAC Enterprise license. For more details on how to obtain a license, visit License Page." | 10:31 |
| *** ramishra has quit IRC | 10:31 | |
| sshnaidm | ok, didnt' see this before.. | 10:32 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks. https://review.openstack.org/357750 | 10:32 |
| *** ramishra has joined #tripleo | 10:32 | |
| b00tcat | I can't get an overcloud deployed successfully (following the docs) because the resource ControllerNodesPostDeployment can't be created correctly... | 10:33 |
| b00tcat | I can see this on the logs https://paste.fedoraproject.org/423299/47324435/ does anybody have a clue? | 10:33 |
| derekh | cielometer should be consuming these correct? http://paste.openstack.org/show/567398/ so I should be ok purging those queues? | 10:33 |
| jaosorior | b00tcat: can you check if the heat logs say anything? | 10:34 |
| shardy | b00tcat: what is the output from "openstack stack failures list overcloud" | 10:34 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:35 |
| shardy | b00tcat: that resource applies puppet to configure all the services, we need to see the actual puppet error to help | 10:35 |
| *** akuznetsov has quit IRC | 10:35 | |
| b00tcat | jaosorior, shardy : can that be found elsewhere than /var/log/heat/heat-engine.log ? | 10:36 |
| b00tcat | I only see python stacktraces there | 10:36 |
| jaosorior | shardy: the log he sent seemed like a heat crash | 10:36 |
| jaosorior | b00tcat: well, what stacktraces are there? | 10:37 |
| shardy | jaosorior: No it looks like the deployment failed | 10:39 |
| shardy | Deployment exited with non-zero status code: 6 | 10:39 |
| shardy | puppet failed | 10:39 |
| shardy | AFAICS | 10:39 |
| * b00tcat creating a paste | 10:39 | |
| jaosorior | shardy: ah I had misunderstood. Thought that came up from the overcloud deploy | 10:40 |
| shardy | jaosorior: Yeah, sometimes heat logs a traceback when a resource failure happens | 10:40 |
| jaosorior | b00tcat: the logs won't tell much. I do recommend you follow shardy's advice and run openstack stack failures list | 10:40 |
| shardy | perhaps we should fix that, but it's sometimes useful | 10:40 |
| b00tcat | right | 10:40 |
| b00tcat | you're right the logs don't tell much | 10:40 |
| b00tcat | hm it says that `stack failures` is not a valid command | 10:41 |
| jaosorior | b00tcat: alright, seems you don't have that command available yet. Then lets do this | 10:42 |
| jaosorior | b00tcat: heat resource-list -n 5 overcloud | grep FAIL | 10:42 |
| jaosorior | b00tcat: that will tell you which specific heat resources failed | 10:43 |
| jaosorior | b00tcat: then you need to look for something of the type OS::Heat::SoftwareDeployment or OS::Heat::StructuredDeployment | 10:43 |
| jaosorior | you can then use the UUID of that to inspect the failure | 10:43 |
| jaosorior | b00tcat: heat deployment-show <failed deployment UUID> | 10:44 |
| b00tcat | right I see two of these | 10:44 |
| b00tcat | is `StructuredDeployments` also relevant? | 10:44 |
| jaosorior | b00tcat: no, that is just the envelope that groups several deployments | 10:44 |
| jaosorior | b00tcat: you can also do heat deployment-output-show <failed deployment UUID> deploy_stderr | 10:45 |
| *** dsariel has quit IRC | 10:45 | |
| b00tcat | nice, Puppet output - now this I can understand :-) | 10:46 |
| b00tcat | I see memory allocation problems | 10:46 |
| jaosorior | b00tcat: well, that's that. Seems that your nodes have too little memory for the deployment :/ | 10:47 |
| jaosorior | b00tcat: what are the specs you gave? | 10:47 |
| b00tcat | I followed everything as is in the docs:/ | 10:47 |
| b00tcat | let me check | 10:47 |
| b00tcat | 5120M | 10:48 |
| jaosorior | b00tcat: so these are the values I've been using | 10:49 |
| jaosorior | control_memory: 6144 | 10:49 |
| jaosorior | compute_memory: 6144 | 10:49 |
| jaosorior | undercloud_memory: 8192 | 10:49 |
| b00tcat | so I set these values as env variables before executing `instack-undercloud` - now that I have my undercloud created (and I'm inside it) how can I change that 5120M memory value? | 10:50 |
| b00tcat | just re-export the var? | 10:50 |
| jaosorior | b00tcat: that I don't know :/ I use quickstart myself | 10:50 |
| b00tcat | I'll investigate | 10:50 |
| b00tcat | this was very helpful thanks! | 10:50 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates: Fix service config files having wrong map_merge format https://review.openstack.org/366674 | 10:52 |
| *** zoli_gone-proxy is now known as zoliXXL | 10:52 | |
| shardy | b00tcat: it's possible you need to increase the memory available - usage has crept up recently as we've added a bunch of stuff | 10:53 |
| shardy | it also depends what services you have enabled, you could disable stuff by passing a modified ControllerServices list | 10:54 |
| shardy | http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html shows how | 10:54 |
| b00tcat | thanks shardy, going to read it | 10:55 |
| shardy | b00tcat: FYI this is how I configure things http://paste.openstack.org/show/567410/ | 10:55 |
| shardy | 8G for undercloud and overcloud nodes | 10:55 |
| shardy | I can launch three overcloud nodes and the undercloud on a box with 32G ram | 10:56 |
| skramaja | shardy: while verifying the map_merge issue, i found another 2 files with the same problem in the whole of puppet/services. i have fixed all.. | 10:56 |
| skramaja | shardy: raised https://bugs.launchpad.net/tripleo/+bug/1621008 | 10:57 |
| openstack | Launchpad bug 1621008 in tripleo "Fix service config files having wrong map_merge format" [Undecided,In progress] - Assigned to Saravanan KR (skramaja) | 10:57 |
| b00tcat | ok I'll give them more resources then - this server that I'm using has 128G of ram and I just followed the docs values ^^" | 10:57 |
| shardy | skramaja: nice, thanks! | 10:57 |
| shardy | b00tcat: sounds good - we should probably update the docs and/or show how to turn off services there | 10:57 |
| dtantsur | I even use 12 Gi for undercloud, 8 Gi is not too much | 11:00 |
| *** mhenkel has quit IRC | 11:00 | |
| dtantsur | control_memory: 8192 | 11:00 |
| dtantsur | compute_memory: 6144 | 11:00 |
| dtantsur | undercloud_memory: 12288 | 11:00 |
| dtantsur | undercloud_vcpu: 4 | 11:00 |
| dtantsur | (ironic does put some additional load on controllers, hence 8 Gi for them) | 11:01 |
| dtantsur | b00tcat, ^^^ | 11:01 |
| dtantsur | I launch 6 vms on my 64 Gi machine, so your can have more :) | 11:02 |
| dtantsur | * 7 vms counting undercloud | 11:02 |
| b00tcat | good to know ;) | 11:03 |
| *** slagle has joined #tripleo | 11:06 | |
| *** masco__ has joined #tripleo | 11:06 | |
| *** masco_ has quit IRC | 11:09 | |
| *** mburned_out is now known as mburned | 11:13 | |
| shardy | dtantsur: Yeah, I do enable 1G swap on my undercloud, although it doesn't get used much | 11:13 |
| * shardy looks around for more ram | 11:13 | |
| *** lucasagomes is now known as lucas-hungry | 11:20 | |
| sshnaidm | derekh, it seems not bad at all now | 11:21 |
| derekh | sshnaidm: whoop ;-) | 11:21 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-ui: Extract Mistral action and workflow names into constants https://review.openstack.org/366685 | 11:22 |
| sshnaidm | derekh, I think to workaround this afs issue with first pip failure.. | 11:22 |
| *** akrivoka has quit IRC | 11:23 | |
| *** akrivoka has joined #tripleo | 11:23 | |
| derekh | sshnaidm: not sure what that means | 11:24 |
| *** thrash|g0ne is now known as thrash | 11:25 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Try again to install modules from AFS PyPi mirror https://review.openstack.org/366687 | 11:26 |
| sshnaidm | derekh, ^^ | 11:26 |
| sshnaidm | derekh, it's what we talked about, with long and broke pip installations | 11:26 |
| jaosorior | jistr: could you check again this https://review.openstack.org/#/c/357765/32 ? I was told to change the path for the puppet manifest | 11:27 |
| derekh | sshnaidm: ok | 11:28 |
| sshnaidm | derekh, but I still see delete_failed and create_failed stacks in the controller :( | 11:28 |
| sshnaidm | seems like restart is inevitable | 11:29 |
| derekh | sshnaidm: yup, I just don't think we can make the cpu change without console access or somebody at the actual host doing it | 11:30 |
| sshnaidm | derekh, and where are hosts physically? | 11:32 |
| sshnaidm | or geographically | 11:33 |
| *** zoliXXL is now known as zoli|brb | 11:36 | |
| derekh | shardy: phoenix , we may need to talk to zoli|brb about having somebody at the host to change the bios | 11:41 |
| *** pkovar has joined #tripleo | 11:42 | |
| jaosorior | thanks jistr | 11:44 |
| jistr | hehe np :) | 11:45 |
| derekh | shardy: sorry wrong person, sshnaidm ^ | 11:45 |
| jaosorior | marios, shardy: You guys think we can merge this? https://review.openstack.org/#/c/357765 | 11:45 |
| shardy | jaosorior: have we moved creating hosts files into puppet now? | 11:48 |
| jaosorior | shardy: only the overcloud VIP-related hosts for now. haven't gotten to move the other hosts yet | 11:48 |
| jaosorior | shardy: currently there is nothing creating /etc/hosts entries for the VIP endpoints | 11:48 |
| shardy | I was wondering how we deal with the overlap with an element and puppet configuring the same file | 11:49 |
| *** jpena is now known as jpena|lunch | 11:49 | |
| shardy | the ordering means it'll probably work | 11:49 |
| shardy | but it may end up being fragile | 11:49 |
| jaosorior | shardy: we already went through that actually. The ordering works, and when there's an update, the element only changes the stuff that is between some special flags it sets | 11:50 |
| dtantsur | shardy, speaking of overlaps, we need you opinion on https://review.openstack.org/345980 - overlap between PXE in puppet and the iPXE element in i-u | 11:50 |
| jaosorior | shardy: the element doesn't nuke the /etc/hosts file | 11:50 |
| shardy | jaosorior: ack, that's what I was wondering | 11:50 |
| shardy | thanks | 11:50 |
| jaosorior | does anyone have some extra cycles to help me debug some heat templates? | 11:51 |
| *** flepied has quit IRC | 11:51 | |
| EmilienM | hello | 11:51 |
| *** flepied has joined #tripleo | 11:51 | |
| *** flepied has quit IRC | 11:51 | |
| jaosorior | EmilienM: hey dude, what's up? Saw your commits regarding the fernet tokens work. Good stuff! | 11:51 |
| *** flepied has joined #tripleo | 11:51 | |
| jaosorior | EmilienM: I just have one doubt about those. But need to wait for ayoung to login to ask him about it | 11:51 |
| EmilienM | jaosorior: it's not fernet, it's only credentials now | 11:51 |
| jaosorior | EmilienM: ah, I see. thought we were also covering fernet with that. I misunderstood then | 11:52 |
| marios | jaosorior: i haven't looked at that one before | 11:52 |
| EmilienM | jaosorior: I'll do fernet right after this | 11:52 |
| marios | jaosorior: added to queue but for tomorrow if still around | 11:52 |
| shardy | dtantsur: approved | 11:52 |
| dtantsur | thnx | 11:53 |
| *** shardy is now known as shardy_lunch | 11:53 | |
| *** bfournie has quit IRC | 11:54 | |
| marios | jaosorior: thanks I see is related to the " Add VIP names to allNodesConfig" https://review.openstack.org/#/c/365895 i looked at earlier - anyway will have a acloser look | 11:54 |
| sshnaidm | derekh, http://paste.openstack.org/show/567425/ | 11:55 |
| sshnaidm | derekh, connection to afs mirror is really bad | 11:56 |
| dtantsur | folks, do we still need https://github.com/openstack/instack-undercloud/blob/master/elements/overcloud-full/package-installs.yaml ? | 11:56 |
| sshnaidm | derekh, but no idea whom to blame, rh1 or afs.. | 11:56 |
| derekh | sshnaidm: looks like a DNS problem, "Could not resolve host: mirror.regionone.tripleo-test-cloud-rh1.openstack.org; Unknown error" | 11:57 |
| EmilienM | fyi, CI scenarios are now all working, we're going to enable voting on them https://review.openstack.org/#/c/366428/ | 11:57 |
| jaosorior | EmilienM: duuuude nice! | 11:57 |
| jaosorior | EmilienM: was zaqar added already to a scenario? | 11:58 |
| dtantsur | also do we really need https://github.com/openstack/instack-undercloud/tree/master/elements/centos-cr ? | 11:58 |
| jaosorior | EmilienM: don't remember exactly which was it that it was being added on | 11:58 |
| EmilienM | jaosorior: it's in progress | 11:58 |
| EmilienM | jaosorior: https://github.com/openstack-infra/tripleo-ci#service-testing-matrix | 11:58 |
| dtantsur | EmilienM, do you plan on adding ironic there now? ;) sorry for being annoying, would be cool to get it there before RC, even if it only covers the actual installation only | 11:59 |
| *** maeca1 has joined #tripleo | 11:59 | |
| EmilienM | dtantsur: yes, ironic too | 11:59 |
| dtantsur | cool! | 12:00 |
| EmilienM | dtantsur: I can describe the process to do it | 12:00 |
| dtantsur | EmilienM, yes please. | 12:00 |
| *** coolsvap_ is now known as coolsvap | 12:01 | |
| jaosorior | EmilienM: can you do that via documentation or a blog post? I would like to start working on barbican (for Ocata) | 12:01 |
| * dtantsur will be back in ~ 30 minutes, sorry | 12:01 | |
| *** trown|outtypewww is now known as trown | 12:02 | |
| EmilienM | dtantsur: you first need to submit a patch into tripleo-ci to patch scenario002 heat template and add Ironic services. Also patch the scenario002 pingtest to create an Ironic resource (goal here is to test the API). Send the patch and -1 it. Now send a patch into THT and modify puppet/services/ironic-base.yaml for example with a dumb modification. Ue depends-on in the commit message and jobs will run | 12:02 |
| EmilienM | I'm going to blog post about it | 12:02 |
| EmilienM | today | 12:02 |
| *** links has quit IRC | 12:04 | |
| *** chlong has joined #tripleo | 12:05 | |
| *** masco_ has joined #tripleo | 12:06 | |
| *** jayg|g0n3 is now known as jayg | 12:07 | |
| *** zoli|brb is now known as zoli | 12:08 | |
| *** zoli is now known as zoliXXL | 12:08 | |
| *** NikoHermannsEric has joined #tripleo | 12:09 | |
| NikoHermannsEric | hey, I am using tripleo in redhat OpenStackPlatfrom | 12:09 |
| NikoHermannsEric | I see the problem that the authentication token is somehow wrong of nova | 12:10 |
| *** masco__ has quit IRC | 12:10 | |
| NikoHermannsEric | nova-compute is flipping from up to down constantly | 12:10 |
| zoliXXL | derekh, shardy_lunch - which hosts need BIOS update/cahneg/whatever? | 12:11 |
| NikoHermannsEric | seeing this errors in nova-api: | 12:11 |
| NikoHermannsEric | 2016-09-07 08:03:50.960 44603 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: c5b36be6b9224de282612bd6af88fdff", "code": 404, "title": "Not Found"}}2016-09-07 08:03:50.960 44603 WARNING keystonemiddleware.auth_token [-] Authorization failed for token | 12:11 |
| NikoHermannsEric | Timeout, server cic1 not responding. | 12:11 |
| jaosorior | NikoHermannsEric: which version (liberty, mitaka, newton) and which nova? the one from the overcloud or undercloud? | 12:12 |
| NikoHermannsEric | mitaka | 12:12 |
| NikoHermannsEric | ohh no sorry osp 8 that is liberty i think | 12:13 |
| jaosorior | jistr: hadn't there been a similar issue in mitaka that was backported and fixed? ^^ | 12:13 |
| ansiwen | chem, mwhahaha: what user and project can I use in the init.pp of puppet-tempest. Are there certain users/project predefined, that I can use? | 12:13 |
| derekh | zoliXXL: we were talking about chaing the CPu setting on the overcloud controller, it has to be done in the BIOS, but I can't get a console , see https://10.1.8.22 | 12:14 |
| * zoliXXL looks | 12:15 | |
| zoliXXL | hmmmm | 12:15 |
| zoliXXL | same here | 12:15 |
| zoliXXL | iDRAC issue ? | 12:16 |
| *** links has joined #tripleo | 12:16 | |
| jistr | NikoHermannsEric, jaosorior: hmm it doesn't ring a bell to me at least... We've seen general problems with keystone tokens at times, if the overcloud nodes didn't have a properly configured NTP and their clocks were offset from each other. Is there a chance this could be the problem? | 12:17 |
| *** pradk has joined #tripleo | 12:18 | |
| NikoHermannsEric | mhhh | 12:18 |
| ansiwen | chem, mwhahaha: ok, I guess I found the answer by myself, $project_name and $username should be a fair guess | 12:18 |
| NikoHermannsEric | at least that makes it clear why I don't see issues in other deployments | 12:18 |
| NikoHermannsEric | ok I will check that | 12:19 |
| NikoHermannsEric | thanks for the help | 12:19 |
| jistr | sure thing :) | 12:19 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 12:20 |
| derekh | zoliXXL: don't know, do this things need a license installed or something? | 12:21 |
| beagles | shardy: is someone going through the other yaml files and making sure that they dont' have that syntax-after-map-merge issue? | 12:22 |
| *** Hazelesque_ is now known as Hazelesque | 12:22 | |
| zoliXXL | derekh, it seems that the time for another round of "iDRAC housekeeping" is coming :) | 12:22 |
| *** lucas-hungry is now known as lucasagomes | 12:23 | |
| beagles | shardy ne'er mind | 12:23 |
| skramaja | beagles: i have gone through it and and found fiew including dpdk. added https://review.openstack.org/#/c/366674/ | 12:23 |
| *** sshnaidm is now known as sshnaidm|afk | 12:24 | |
| beagles | skramaja, yup looking at patch now | 12:24 |
| skramaja | thanks. | 12:25 |
| beagles | skramaja, considering it is neutron api and ovs agent - pretty weird things haven't been blowing up all along | 12:25 |
| skramaja | beagles: since neutron base config is included in multiple locations, those hiera will be set by other means.. and as per the bug, whatever set on the yaml will work.. what evenr included is not working. | 12:26 |
| beagles | skramaja, ah good point | 12:26 |
| *** bfournie has joined #tripleo | 12:27 | |
| *** dprince has joined #tripleo | 12:27 | |
| *** liverpooler has quit IRC | 12:28 | |
| dtantsur | EmilienM, hmm, I think we don't have ironic resources in Heat.. | 12:28 |
| *** rlandy has joined #tripleo | 12:28 | |
| EmilienM | dtantsur: sad | 12:28 |
| dtantsur | yeah... it was blocked by devananda some time ago.. now he no longer blocks it, but somebody needs to finish them | 12:29 |
| dtantsur | still, I would like to test that the installation succeeds | 12:29 |
| EmilienM | right | 12:29 |
| EmilienM | we can find a workaround | 12:29 |
| *** karthiks has quit IRC | 12:31 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: create release files for consistent and current-tripleo https://review.openstack.org/366488 | 12:31 |
| dtantsur | EmilienM, thanks anyway, I will propose something after lunch | 12:32 |
| EmilienM | dtantsur: i'll also think about it later today | 12:34 |
| ayoung | jaosorior, yeah, I saw those, too. | 12:36 |
| *** apetrich has joined #tripleo | 12:40 | |
| *** jprovazn has quit IRC | 12:40 | |
| *** rbrady has joined #tripleo | 12:42 | |
| dtantsur | EmilienM, hmm, I guess I also need to update project-config to trigger the job on *ironic*.yaml, right? | 12:43 |
| EmilienM | yes | 12:43 |
| EmilienM | dtantsur: I'm writting a blog post atm | 12:44 |
| mandre | my overcloud deployment fails with "Could not find class ::tripleo::profile::base::database::mysql for overcloud-controller-0.localdomain", does it ring a bell? | 12:44 |
| *** rhallisey has joined #tripleo | 12:45 | |
| *** karthiks has joined #tripleo | 12:45 | |
| tbarron | marios: I just reported a new overcloud deploy result on https://review.openstack.org/#/c/354019 | 12:46 |
| mandre | I do have the ::tripleo::profile::base::database::mysql profile on my undercloud | 12:46 |
| dtantsur | EmilienM, mm, I think a see a problem. do we really configure nova-compute on a controller in these scenarios? | 12:47 |
| *** pradk has quit IRC | 12:47 | |
| *** jpena|lunch is now known as jpena | 12:48 | |
| *** jaosorior has quit IRC | 12:49 | |
| *** jaosorior has joined #tripleo | 12:50 | |
| *** Goneri has joined #tripleo | 12:51 | |
| dtantsur | mandre, you should have it in your overcloud-full, not on undercloud | 12:51 |
| marios | tbarron: thanks will have a closer look in a bit... could be an issue with the manila-ceph backend since that has already landed into puppet-tripleo | 12:52 |
| dtantsur | mandre, see example #2 in https://hardysteven.blogspot.cz/2016/08/tripleo-deploy-artifacts-and-puppet.html or try rebuilding the image | 12:52 |
| mandre | dtantsur: ohhh thanks... I'm using atomic host so rebuiling overcloud-full is not really an option | 12:54 |
| dtantsur | too many moving bits still, such cases should become more rare after RC, I hope :) | 12:55 |
| dtantsur | I got hit by an even nastier problem with the same root cause today | 12:55 |
| mandre | dtantsur: haha, this morning it took me way too long to realize I wasn't dealing with a bash variable in https://review.openstack.org/#/c/366138/ | 12:58 |
| dtantsur | heh | 12:58 |
| *** tzumainn has joined #tripleo | 12:58 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:59 | |
| *** tzumainn has quit IRC | 13:00 | |
| *** tzumainn has joined #tripleo | 13:01 | |
| openstackgerrit | Merged openstack/tripleo-ui: Update the Mistral action names to use the new version https://review.openstack.org/366579 | 13:03 |
| thrash | dtantsur: fyi... https://review.openstack.org/#/c/363095/ | 13:04 |
| dtantsur | awesome, thanks | 13:05 |
| *** chem has quit IRC | 13:07 | |
| tbarron | marios: that is my suspicion as well, i'm going to attempt to patch that locally and re-deploy | 13:07 |
| *** masco__ has joined #tripleo | 13:07 | |
| *** dsariel has joined #tripleo | 13:07 | |
| *** chem has joined #tripleo | 13:07 | |
| dtantsur | do we support (out of box at least) advances network configuration NOT related to network isolation? | 13:07 |
| dtantsur | everything I've seen so far was more or less part of this feature... | 13:08 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Fixed nic numbering issue of DPDK nics after the nic has bound https://review.openstack.org/364354 | 13:08 |
| shardy_lunch | dtantsur: you can configure the host networking however you like, within the constraints of what is supported by os-net-config, if that's what you mean? | 13:08 |
| *** shardy_lunch is now known as shardy | 13:08 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Update validations after Mistral executions listing is updated https://review.openstack.org/365623 | 13:08 |
| dtantsur | shardy, probably :) please bear with me, I don't know much about OS networking. I want to explore 2 NICs configuration for Ironic in overcloud, where a separate NIC (and a separate flat networking on it) is using for bare metal instances. | 13:09 |
| dtantsur | social_, is something wrong with https://review.openstack.org/#/c/345980/ ? | 13:09 |
| social_ | dtantsur: nevermind taking it away, it's RHEL issue | 13:10 |
| EmilienM | dtantsur, jaosorior: http://my1.fr/blog/scaling-up-tripleo-ci-coverage-with-scenarios/ | 13:10 |
| *** florianf has quit IRC | 13:10 | |
| openstackgerrit | Merged openstack/instack-undercloud: Use ironic::pxe class to setup PXE https://review.openstack.org/345980 | 13:10 |
| *** masco_ has quit IRC | 13:10 | |
| social_ | dtantsur: it will break anyway just later, I'll try to get that fixed before we hit it :) | 13:10 |
| dtantsur | EmilienM, please see my ping above.. if I get it right, we configure nova-compute on a controller with libvirt... this is not compatible with how we cook ironic now. | 13:11 |
| shardy | dtantsur: I'd look at these as a starting point: | 13:11 |
| shardy | https://github.com/openstack/tripleo-heat-templates/tree/master/network/config/multiple-nics | 13:11 |
| EmilienM | dtantsur: I know Sir | 13:11 |
| shardy | you'd need to adapt them to have two nics, and pass in the configuration parameters for the baremetal dedicated interface | 13:11 |
| *** links has quit IRC | 13:12 | |
| jrist | jtomasek, flfuchs - did you guys get my email about https://review.openstack.org/365580 ? | 13:12 |
| EmilienM | shardy: can I have a review on https://review.openstack.org/#/c/366287/ and https://review.openstack.org/#/c/366400/ please? | 13:12 |
| shardy | dtantsur: see the list in the network_config: key from line 92 | 13:12 |
| dtantsur | shardy, thanks. so it's not quite out of box, we need to build such configuration ourselves | 13:12 |
| dtantsur | shardy, do you think it would be useful to have it in tree? or is it too specific? | 13:13 |
| EmilienM | dprince: how many time is required before tripleo planet shows a new blog post? | 13:13 |
| shardy | dtantsur: Yeah, we provide a few sample configs, but it's pretty hard to ancipate all the special snowflakes around nic configs | 13:13 |
| jtomasek | jrist: yeah, make sure you have latest tripleo-common installed | 13:13 |
| jrist | ah | 13:13 |
| jtomasek | jrist: pull latest master from tripleo-common and do steps in README.md | 13:13 |
| shardy | dtantsur: not sure, push the review when you have something working & we can discuss if it fits in t-h-t | 13:13 |
| jrist | thanks jtomasek | 13:14 |
| jrist | jtomasek: thought I did have latest but there are a few changes | 13:14 |
| dtantsur | shardy, fair enough | 13:14 |
| jrist | jtomasek: bleeding edge! | 13:14 |
| jtomasek | jrist: it is especially bleeding lately heh | 13:14 |
| jrist | LOL | 13:15 |
| shardy | EmilienM: sure, I looked at those this morning but it looked like we were blocked on the puppet-keystone patch | 13:15 |
| *** zoliXXL is now known as zoli|brb | 13:16 | |
| openstackgerrit | Martin André proposed openstack/instack-undercloud: Setup CORS settings for tripleo-ui https://review.openstack.org/360593 | 13:16 |
| openstackgerrit | Martin André proposed openstack/instack-undercloud: Introduce 'enable_ui' option https://review.openstack.org/344140 | 13:16 |
| dtantsur | shardy, do I get it right that I'll only have to override network configuration for controllers in my (ironic) case? | 13:16 |
| jaosorior | shardy: do you have any idea what could be wrong with this? https://review.openstack.org/#/c/356430/17/puppet/services/haproxy-internal-tls-certmonger.yaml seems that something in the certificates_specs section is messed up | 13:17 |
| shardy | dtantsur: sure, if you only need the extra nic configured there | 13:17 |
| *** trown is now known as trown|brb | 13:17 | |
| dtantsur | I *suspect* so, as I guess virtual traffic will get routed via controllers anyway | 13:17 |
| EmilienM | shardy: ok good, just making sure you're ok with all of this | 13:17 |
| jaosorior | shardy: I tested it locally in a separate stack, with sample data (that I got from my deployment) and it works there. But when I run in the template. It seems to break the output of that stack. having roles_data as None | 13:17 |
| shardy | EmilienM: I think the tripleoclient stuff may need to go into a mistral action (like the password generation) | 13:18 |
| shardy | otherwise this change will break tripleo-ui | 13:18 |
| * shardy looks for where it should go | 13:18 | |
| *** jlinkes_ has joined #tripleo | 13:18 | |
| *** jprovazn has joined #tripleo | 13:19 | |
| EmilienM | shardy: I would like to move forward with my work, it is currently blocking the promotion of keystone package | 13:19 |
| EmilienM | shardy: we can have a mistral workflow in the next iteration if you don't mind. | 13:19 |
| *** jlinkes has quit IRC | 13:19 | |
| EmilienM | shardy: I don't see how it would block UI, since I'm using the same mechanism as Ceph FSID. | 13:19 |
| shardy | d0ugal, jtomasek: did password generation move to mistral? I don't see it | 13:19 |
| EmilienM | not afik | 13:19 |
| shardy | EmilienM: we're moving all this stuff into mistral actions, where it can be consumed by the UI | 13:20 |
| shardy | the UI doesn't use tripleoclient | 13:20 |
| shardy | so any fixes which only land in tripleoclient now, break the UI | 13:20 |
| shardy | we don't want to release with a broken UI, hence raising this | 13:20 |
| EmilienM | I'm happy to move is in mistral after that | 13:20 |
| shardy | EmilienM: sure | 13:20 |
| EmilienM | shardy: well, right now we're even not deploying latest keystone | 13:21 |
| EmilienM | imho that's more critical than everything else | 13:21 |
| EmilienM | if you're missing context, Keystone merged a patch that is not backward compatible because it requires to setup Keystone credentials | 13:21 |
| EmilienM | hence my patches to do it in tripleo | 13:21 |
| shardy | EmilienM: I understand the problem, I'm just trying to ensure the fixes go in the right place | 13:21 |
| EmilienM | I would use some help to do it in mistral workflows | 13:22 |
| EmilienM | afik the password generation is still in tripleoclient | 13:22 |
| dprince | EmilienM: 5 minutes | 13:22 |
| shardy | EmilienM: that's why I'm asking d0ugal and jtomasek how the UI handles this | 13:22 |
| EmilienM | dprince: I see it now :) | 13:22 |
| * shardy sighs | 13:22 | |
| EmilienM | dprince: my first post | 13:22 |
| jistr | shardy: hey sorry -1'd this https://review.openstack.org/#/c/365763 -- if you are swamped i can do the change and rebase the things on top | 13:22 |
| EmilienM | shardy: no problem | 13:23 |
| *** pgadiya has quit IRC | 13:23 | |
| shardy | EmilienM: to clarify, I'm not blocking this, just trying to get someone to commit to the follow-up fix on the mistral/UI side | 13:23 |
| EmilienM | shardy: ok, good. | 13:23 |
| jtomasek | shardy: I don't know tbh, I'd expect it to happen in deploy workflow | 13:23 |
| dprince | EmilienM: nice, thanks for posting it | 13:23 |
| jaosorior | jistr: how's your yaql foo? | 13:24 |
| shardy | jtomasek: I expected that too, but I don't see the list of passwords referenced there | 13:24 |
| * shardy looks more closely | 13:24 | |
| *** florianf has joined #tripleo | 13:24 | |
| jistr | jaosorior: heh mediocre i'd say, but had some success with things earlier. Do you want help with sth? | 13:25 |
| shardy | jistr: Ok, thanks, I'll update it now | 13:25 |
| shardy | thanks for the review | 13:25 |
| jistr | shardy: ok thanks | 13:25 |
| jaosorior | jistr: yeah dude, can't figure this out. Got this https://review.openstack.org/#/c/356430/17/puppet/services/haproxy-internal-tls-certmonger.yaml | 13:25 |
| jaosorior | jistr: and somewhere in the certificates_specs section things get messed up | 13:25 |
| jaosorior | jistr: I tested it locally in a separate template and it worked there will sample that that I got from my deployment | 13:26 |
| jaosorior | jistr: but when I actually try it in the overcloud I just can't figure out what's wrong there :/ | 13:26 |
| jaosorior | jistr: what happens is that it results in the whole role_data being Null | 13:26 |
| *** ohamada has quit IRC | 13:27 | |
| jaosorior | jistr: what I try to do there is iterate through the networks that are being used by the services, and generate the map that I'll use for the certificates from there. | 13:27 |
| *** trown|brb is now known as trown | 13:28 | |
| jistr | jaosorior: ah i think i spotted it -- it happened to me too earlier. the heat_template_version needs a bump to 2016-10-xx otherwise the yaql function isn't available | 13:28 |
| *** jcoufal has joined #tripleo | 13:28 | |
| jistr | heat_template_version: 2016-10-14 | 13:28 |
| jistr | jaosorior: ^ | 13:28 |
| *** karthiks has quit IRC | 13:28 | |
| jaosorior | DUDE nice!!! | 13:28 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 13:29 |
| jaosorior | jistr: will try it out, thanks a lot dude! | 13:29 |
| jistr | hehe np :) | 13:29 |
| *** lblanchard has joined #tripleo | 13:30 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services https://review.openstack.org/366075 | 13:31 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service https://review.openstack.org/356430 | 13:31 |
| *** DrBacchus is now known as rbowen | 13:32 | |
| *** weshay is now known as weshay_mtg | 13:32 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Wire in jinja templating for custom roles https://review.openstack.org/362465 | 13:33 |
| EmilienM | shardy: the tripleoclient, https://review.openstack.org/#/c/366287/ can be merged without dependencies, if you ok | 13:33 |
| shardy | EmilienM: Ok, can you please follow up with d0ugal and rbrady about where/how that should be wired in when deployments happen via the UI | 13:34 |
| EmilienM | shardy: I will for sure | 13:35 |
| EmilienM | matbu: can you follow-up https://review.openstack.org/#/c/346995/ regarding comments please? this patch has been standing for long, I want to merge it asap. | 13:35 |
| EmilienM | matbu: everything is WIP for too long time, let's make progress and merge the patches when they pass CI. | 13:36 |
| jistr | EmilienM, matbu: yea i'd be fine with that approach. Anyway this shouldn't be handled in the CI eventually, we should rather land the tripleoclient and instack-undercloud patches we referenced. | 13:37 |
| EmilienM | jistr: instack is already merged | 13:39 |
| EmilienM | jistr: we miss tht now | 13:39 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Fix the ansible-lint gate job https://review.openstack.org/366758 | 13:40 |
| matbu | EmilienM: +1 yep i want to merge it today if it's possible | 13:40 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 13:40 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml https://review.openstack.org/365792 | 13:40 |
| EmilienM | matbu: hum. Have you seen comments first? | 13:40 |
| matbu | EmilienM: not the one from jistr | 13:41 |
| matbu | EmilienM: the +1 from marios was fine too me :D | 13:41 |
| marios | tbarron: you're right, gimme few mins trying to untangle, reviews incoming | 13:41 |
| derekh | sshnaidm: can you run the heat-manage purge again with 2 days, its not working for me and I'm not sure what file you sourced? I wanna see if it makes heat stack-list faster | 13:41 |
| *** zoli|brb is now known as zoli | 13:42 | |
| *** zoli is now known as zoliXXL | 13:42 | |
| rbrady | EmilienM, shardy: I think https://review.openstack.org/#/c/366287/ really belongs in tripleo-common. I can get an equivalent patch up in a couple of minutes | 13:42 |
| marios | tbarron: but its a bit of a mess.. basicallypuppet-tripleo side for manila-cephfs landed wihtout default for class params, so when you only include generic for example, that file is still expecting those values, cos no defaults. but you didn't set them , cos no cephfs native | 13:42 |
| shardy | rbrady: nice, thanks - I grepped and failed to find where it needs to go | 13:43 |
| d0ugal | shardy: Sorry, was on lunch - yeah, the password stuff hasn't been moved over yet | 13:44 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-heat-templates: Update capabilities-map.yaml https://review.openstack.org/364842 | 13:44 |
| d0ugal | shardy, rbrady: we might just want to do it all in one step rather than adding that one bit to Mistral. | 13:44 |
| shardy | d0ugal: Ok, should we go with the tripleocient patch now, then port over to mistral asap? | 13:44 |
| matbu | jistr: EmilienM so iiuc , we can try to just stop the service and make a yum update instead ? | 13:44 |
| d0ugal | shardy: Yup, sounds good. I am writing the status email now, I'll include this. | 13:44 |
| shardy | d0ugal: relatedly - when I delete a stack, why isn't the overcloud plan deleted from swift? | 13:45 |
| shardy | I guess I can use your delete plan on deploy patch to work around it | 13:45 |
| *** karthiks has joined #tripleo | 13:45 | |
| d0ugal | shardy: How did you delete the stack? overcloud stack delete? | 13:45 |
| shardy | lol | 13:45 |
| shardy | d0ugal: No. I need more coffee :) | 13:45 |
| rbrady | d0ugal: guess I should wait on the port then? | 13:45 |
| * rbrady needs to sync up on current status after PTO | 13:46 | |
| d0ugal | rbrady: I'm suggesting you move all the password generation etc. to the Mistral workflows, not just one or two new parameters | 13:46 |
| shardy | rbrady: sounds like we need all this stuff in mistral actions/workflow anyway so tripleoclient can switch over to it? | 13:46 |
| EmilienM | honestly, I would not block the patches I did | 13:46 |
| d0ugal | rbrady: because without all the others I doubt it is useful. | 13:46 |
| d0ugal | shardy: Are you asking why when you delete the environment it isn't deleted from swift? | 13:46 |
| * shardy still wants to know how the UI is generating these | 13:47 | |
| EmilienM | we can move it to tripleo-common but I spent all my day from yesterday to make it working on overcloud | 13:47 |
| d0ugal | shardy: I don't think the UI is generating them yet. | 13:47 |
| shardy | EmilienM: we're just having a discussion here | 13:47 |
| d0ugal | EmilienM: This? https://review.openstack.org/#/c/366287/ | 13:47 |
| shardy | nobody is blocking anything | 13:47 |
| d0ugal | EmilienM: I think it should go into tripleoclient first, because as it stands the workflows are not ready yet. | 13:47 |
| EmilienM | d0ugal: the puppet-keystone patch, THT and the tripleoclient | 13:47 |
| shardy | Ok, lets do that | 13:47 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic https://review.openstack.org/354014 | 13:48 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 13:48 |
| ansiwen | mwhahaha: thanks for you review! btw. Is there a simple way to do some local consistency checks before pushing? to find these kinds of mistakes? | 13:48 |
| shardy | rbrady: Are you OK if we land https://review.openstack.org/#/c/366287/ then rework to pull all the password stuff into mistral? | 13:48 |
| rbrady | shardy: I removed my -1 | 13:49 |
| shardy | rbrady: Ok, thanks | 13:49 |
| EmilienM | thanks | 13:49 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 13:49 |
| EmilienM | marios: I don't understand your +1 on https://review.openstack.org/#/c/346995/ , why not +2 if you're core reviewer? | 13:50 |
| jrist | jtomasek: https://blueprints.launchpad.net/tripleo-ui/+spec/mistral-parameter-config is rc1 right? | 13:50 |
| EmilienM | marios: does it need more work? | 13:50 |
| jtomasek | jrist: that one is finished | 13:51 |
| jrist | jtomasek: thanks | 13:51 |
| jistr | matbu: re just stopping, i think we can do that yea, but if the current state passes CI then i'd be ok to land and iterate | 13:52 |
| marios | EmilienM: i wanted to get some thoughts on the inline comment about the service stop and the package remove/re-install. still not clear to me if this is a packaging issue or what | 13:52 |
| openstackgerrit | Merged openstack/tripleo-ui: Implement Validation Detail modal https://review.openstack.org/365921 | 13:52 |
| EmilienM | marios: i'll let matbu comment and unblock the situation. We want to test upgrades asap | 13:52 |
| matbu | jistr: k, yep the current state passed the CI (i mean the upgrade) | 13:52 |
| EmilienM | and I see nothing in our gate | 13:52 |
| jistr | EmilienM, matbu: regarding the unlanded patches that would be solving these issues, i meant https://review.openstack.org/#/c/331804 and https://review.openstack.org/#/c/350657 | 13:53 |
| *** pradk has joined #tripleo | 13:53 | |
| matbu | jistr: marios EmilienM but i think we can itterate | 13:54 |
| jistr | matbu: yea i agree | 13:54 |
| matbu | jistr: marios EmilienM merge this review as is and then iterate with a new one when ^ would be merged | 13:54 |
| b00tcat | if I see a bug on the docs, what should I do with it? is on a repo or something so I can create a patch? | 13:54 |
| matbu | i'm pretty close to deliver the full upgrade jobs | 13:55 |
| marios | EmilienM: also, imo, being core does not make the +1 button unclickable... just saying, for me +1 can mean 'see comment but not blocking' or 'im not confident to +2 this cos big review and will revisit, but not blocking' etc | 13:55 |
| openstackgerrit | Merged openstack/tripleo-ui: Run Validations automatically https://review.openstack.org/366068 | 13:55 |
| openstackgerrit | Merged openstack/tripleo-ui: Expand help message on config sample file https://review.openstack.org/365707 | 13:55 |
| EmilienM | marios: I agree, but in the case you use +1, please explain why not +2 | 13:56 |
| marios | matbu: ack thanks so i can revote if we are happy ... _right now_ those reviews for the service stop and package remove/reinstall are in review and apparently very controversial so we just do it in the script as is | 13:56 |
| EmilienM | marios, jistr: anyway, let's make progress on upgrade testing. I'm really scared by all upgrade work that you guys are doing and this isn't tested by our CI at all | 13:56 |
| EmilienM | it's blind reviewing | 13:57 |
| chem | EmilienM: like it was before :) | 13:57 |
| EmilienM | marios: I've +2ed a lot of upgrades patches I could not test myself | 13:57 |
| EmilienM | chem: well, it has to change | 13:57 |
| *** rcernin has joined #tripleo | 13:57 | |
| marios | EmilienM: ok, i thought that was apparent in the comment i made my apologies i will try and be clearer on my intentions in future | 13:57 |
| marios | EmilienM: yes so have I | 13:57 |
| EmilienM | marios: I initiated the work to have upgrade job testing | 13:57 |
| matbu | marios: cool | 13:57 |
| EmilienM | please help me now | 13:57 |
| marios | EmilienM: and i will now revote but let the records show that EmilienM threatened my family in order for this to happen | 13:57 |
| chem | EmilienM: anyway I think the removal of the package is not needed | 13:57 |
| chem | EmilienM: stoping services is enought | 13:58 |
| EmilienM | marios: ? | 13:58 |
| marios | EmilienM: i am joking man, trying to make light of the discussion... like "he was threatened and changed his vote to +2" | 13:59 |
| EmilienM | oh ok | 13:59 |
| marios | EmilienM: i am going to revote on the review . | 13:59 |
| EmilienM | marios: once we have undercloud upgrade tested (non voting), matbu will make progress on overcloud | 13:59 |
| jistr | marios, EmilienM, matbu: i gave in under the threats and +2'd the patch | 13:59 |
| EmilienM | thanks guys | 13:59 |
| jistr | no torturing pls | 13:59 |
| matbu | lol | 14:00 |
| EmilienM | jistr: i'll make you write Puppet unit tests if you don't +2 lol | 14:00 |
| EmilienM | (joke) | 14:00 |
| * matbu is now afraid of working on tripleo-ci | 14:00 | |
| chem | EmilienM: well you should remove the deinstall/reinstall | 14:00 |
| jistr | not sure if we need the OVB jobs passig too though, before we merge? | 14:00 |
| sshnaidm | derekh, sorry, was in the meeting | 14:00 |
| *** abregman_ is now known as abregman | 14:00 | |
| EmilienM | chem: right. matbu wdyt? | 14:01 |
| EmilienM | jistr: yeah, we don't want to break CI, I think another iteration is necessary | 14:01 |
| matbu | EmilienM: chem ack | 14:01 |
| sshnaidm | derekh, I ran it before, in the morning as well, but not sure what it does, it's very quiet tool | 14:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 14:03 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 14:03 |
| *** pkovar has quit IRC | 14:03 | |
| *** ayoung has quit IRC | 14:03 | |
| *** liverpooler has joined #tripleo | 14:04 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services https://review.openstack.org/366075 | 14:04 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service https://review.openstack.org/356430 | 14:04 |
| *** oshvartz has quit IRC | 14:06 | |
| *** rodrigods has quit IRC | 14:06 | |
| *** cdearborn has joined #tripleo | 14:06 | |
| *** rodrigods has joined #tripleo | 14:06 | |
| *** masco_ has joined #tripleo | 14:07 | |
| *** pkovar has joined #tripleo | 14:07 | |
| *** dtrainor has quit IRC | 14:07 | |
| bnemec | derekh: sshnaidm: Damn, so I broke create-env? | 14:07 |
| *** dtrainor has joined #tripleo | 14:08 | |
| sshnaidm | bnemec, it's already over :) | 14:08 |
| marios | jistr: yeah everyone has their breaking point man, don't be hard onyrself | 14:08 |
| bnemec | sshnaidm: Sorry about that. What's the current status? | 14:08 |
| jistr | lol | 14:09 |
| bnemec | I saw the thing about the controller DRAC not having the proper license. | 14:09 |
| tobias_fiberdata | Hiya | 14:09 |
| bnemec | Or possibly the proper hardware. I thought iDRAC enterprise was actually a separate card from express. | 14:09 |
| tobias_fiberdata | i've got a question about the different networks you can configure. in the openstack-plattform we have today i have 3 different networks for API, one public API, one adminAPI and one internalAPI, is the adminAPI gone? | 14:10 |
| tobias_fiberdata | i cannot see that network in the heat-templates | 14:10 |
| derekh | bnemec: dunno, but I can't see how to get a console(was going to try it earlier), so to change the CPU profile we need somebody at the host to do it | 14:10 |
| tobias_fiberdata | in the stable mitaka one | 14:11 |
| openstackgerrit | Merged openstack/tripleo-common: Fix error when identity file is missing https://review.openstack.org/365906 | 14:11 |
| sshnaidm | bnemec, we still suffer from similar issues | 14:11 |
| *** masco__ has quit IRC | 14:11 | |
| bnemec | derekh: Yeah, I see the same thing. | 14:11 |
| derekh | bnemec: current status is that lots of stacks are still failing to be crated | 14:11 |
| bnemec | I noticed we've mixed single and dual rank DIMMs on that box too for some reason. | 14:11 |
| bnemec | Which may be exacerbating the problem. | 14:11 |
| sshnaidm | bnemec, it's not good to do, right? | 14:11 |
| derekh | bnemec: just noticed this in the scheduler logs | 14:11 |
| derekh | 2016-09-07 14:05:19.299 18985 INFO nova.filters [req-8d618f13-a11b-4915-a922-73f32b956626 ba119eef29ce49f5b8697f4d63948e3c b79291658f384b7ebbc9019b6349e5c9 - - -] Filtering removed all hosts for the request with instance ID '20f43fe5-5d69-41ad-bb04-a29ccd244cdc'. Filter results: ['RetryFilter: (start: 33, end: 32)', 'AvailabilityZoneFilter: (start: 32, end: 32)', 'RamFilter: (start: 32, end: 30)', 'DiskFilter: (start: 30, end: 4)', 'ComputeFilter: (start: | 14:11 |
| derekh | 4, end: 0)'] | 14:11 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 14:12 |
| derekh | bnemec: disk filter is removing all the nodes | 14:12 |
| bnemec | derekh: Yeah, I have to think that's because of all the leftover resources still in use. | 14:12 |
| bnemec | We've got a ton of dead stacks that haven't been removed. | 14:12 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-ui: Extract Mistral action and workflow names into constants https://review.openstack.org/366685 | 14:12 |
| derekh | bnemec: yum, need another cleanup | 14:13 |
| EmilienM | rbrady: do you want me to file a launchpad bug and assign to you about porting my oooclient patch? | 14:14 |
| derekh | bnemec: and I turned on mysql logging of slow queries too /var/lib/mysql/overcloud-controller-0-slow.log | 14:15 |
| derekh | bnemec: most of the queries taking over a second are listing nova instances | 14:15 |
| bnemec | derekh: Yeah, not surprising. | 14:16 |
| derekh | bnemec: so I tries to purge them but guess what .... it doesn't seem to be doing anything | 14:17 |
| derekh | > nova-manage db archive_deleted_rows --max_rows 100 | 14:18 |
| *** dmsimard is now known as dmsimard|afk | 14:19 | |
| marios | tbarron: posted a comment hope that helps for now. i guess i promised erno an update tht side for that cephfs-manila since he is away, and this has forced my hand :) but i will do that tht side later or tomorr. shouldn't impact the generic/netapp if you include the cephfs puppet-tripleo side | 14:20 |
| *** pradk has quit IRC | 14:20 | |
| *** bkopilov has quit IRC | 14:20 | |
| *** cinerama has quit IRC | 14:20 | |
| *** wfoster has quit IRC | 14:20 | |
| *** rbowen has quit IRC | 14:20 | |
| *** stevebaker has quit IRC | 14:20 | |
| *** jidar has quit IRC | 14:20 | |
| rbrady | EmilienM: d0ugal created a bug | 14:21 |
| EmilienM | rbrady: ok, good | 14:21 |
| *** dciabrin has quit IRC | 14:24 | |
| openstackgerrit | Merged openstack/tripleo-validations: Cleanup tox.ini, enable constraints https://review.openstack.org/360994 | 14:25 |
| *** pradk has joined #tripleo | 14:25 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 14:26 |
| *** yamahata has joined #tripleo | 14:26 | |
| *** lucasagomes is now known as lucas-afk | 14:29 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/365625 | 14:32 |
| *** weshay_mtg is now known as weshay | 14:33 | |
| bnemec | derekh: Thinking about trying something a little crazy on the controller since we can't change the performance profile. I think if I start a process that uses 100% cpu and is niced it will force the cpu to stay at full speed, but shouldn't steal cpu from the openstack processes. | 14:36 |
| *** dciabrin has joined #tripleo | 14:37 | |
| derekh | bnemec: thinking outside the box, I like it | 14:37 |
| derekh | bnemec: should we stop the te_broker again and clean up? | 14:37 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 14:39 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 14:40 |
| shadower | mandre: do you know whether the tripleo-validations docs live anywhere public? | 14:41 |
| shadower | I tried to look for them on docs.openstack.org but couldn't find anything | 14:41 |
| bnemec | derekh: We definitely need to clean up. I don't know whether stop the broker helps or not. | 14:42 |
| *** bkopilov has joined #tripleo | 14:42 | |
| *** cinerama has joined #tripleo | 14:42 | |
| *** wfoster has joined #tripleo | 14:42 | |
| *** rbowen has joined #tripleo | 14:42 | |
| *** stevebaker has joined #tripleo | 14:42 | |
| *** jidar has joined #tripleo | 14:42 | |
| shadower | (maybe we need to enable something somewhere to get them published) | 14:42 |
| openstackgerrit | Merged openstack/tripleo-common: Allow the validations to run openstack commands https://review.openstack.org/366175 | 14:43 |
| *** mwhahaha has quit IRC | 14:44 | |
| derekh | sshnaidm: wanna try and delete the old envs again? I'm gonna see if I can find out how to remove the old instances for the nova db, | 14:44 |
| *** rbrady has quit IRC | 14:44 | |
| mandre | shadower: they don't atm, you need to add a doc-publish job at https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L13571 | 14:45 |
| sshnaidm | derekh, you mean to run that script? | 14:45 |
| derekh | sshnaidm: which script? | 14:45 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy https://review.openstack.org/360329 | 14:45 |
| thrash | jaosorior: ^^^ just rebased it | 14:45 |
| shadower | mandre: ah, awesome, thanks! /me creates a LP so we don't forget | 14:45 |
| sshnaidm | derekh, that deletes ports, networks and then stacks | 14:45 |
| mandre | shadower: i'm unsure if the validations doc should be in at http://docs.openstack.org/developer/tripleo-validations/ or http://docs.openstack.org/developer/tripleo/ | 14:46 |
| derekh | sshnaidm: yup, and if its not clearing them fast enough we may want to stop the broker service again so it can catch up | 14:46 |
| mandre | or wherever the tripleo docs live | 14:46 |
| sshnaidm | derekh, yeah, actually I'm already running it for 20 minutes.. | 14:46 |
| derekh | sshnaidm: ok | 14:46 |
| *** mwhahaha has joined #tripleo | 14:47 | |
| shadower | mandre: yeah. I'd prefer them to be at tripleo.org but barring that, anything public is fine | 14:47 |
| sshnaidm | derekh, maybe really to purge all dbs - nova, neutron, heat like dprince suggested? | 14:47 |
| jaosorior | thrash: alright. You should probably use this one too https://review.openstack.org/#/c/360350/ that's how you'll know it's working (or not) | 14:47 |
| thrash | jaosorior: ack. thanks. | 14:47 |
| shardy | shadower, mandre: can the docs not just go into tripleo-docs? | 14:47 |
| *** ayoung has joined #tripleo | 14:48 | |
| *** jlinkes_ has quit IRC | 14:48 | |
| derekh | sshnaidm: iirc dprince suggested dropping the heat db, the others wouldn't be some simple as they have things we want to keep | 14:48 |
| mandre | shardy: part of the documentation is autogenerated from the content of the ansible files | 14:48 |
| shardy | mandre: ah, OK | 14:48 |
| shadower | yeah what mandre said | 14:49 |
| sshnaidm | derekh, ok | 14:49 |
| mandre | shadower, shardy: probably there's a way to integrate the doc nicely in tripleo-docs, I haven't really looked at it yet | 14:50 |
| dprince | derekh: yep, if you drop more than heat you'd get into rebuilding territory pretty quick | 14:52 |
| *** toure has quit IRC | 14:54 | |
| *** mbound has joined #tripleo | 14:54 | |
| bnemec | I'm not entirely convinced heat is the problem here. It's neutron that's throwing the errors about subnets in use and not deleting ports and such. | 14:54 |
| *** karthiks has quit IRC | 14:54 | |
| derekh | bnemec: there is lost of weird crap | 14:56 |
| derekh | 2016-09-07 14:52:48.993 19028 INFO nova.api.openstack.wsgi [req-7a709547-cdfc-4fc9-b898-972c476dd7d2 ba119eef29ce49f5b8697f4d63948e3c b79291658f384b7ebbc9019b6349e5c9 - - -] HTTP exception thrown: Flavor baremet | 14:56 |
| derekh | al could not be found. | 14:56 |
| derekh | bnemec: the baremetal flavor dissapears and reappears again ? | 14:56 |
| bnemec | derekh: Those were actually there before all the trouble started. My theory is that Heat uses EAFP when finding flavors, so it first looks for a flavor with the id "baremetal", which doesn't exist, then looks for the name "baremetal" after. | 14:57 |
| derekh | bnemec: ok, I buy that | 14:57 |
| *** saneax is now known as saneax-_-|AFK | 14:57 | |
| bnemec | derekh: But that notwithstanding, there is indeed a lot of weird crap going on here. :-) | 14:58 |
| dprince | bnemec: perhaps a bug that heat even logs it as an exception if that is the logic | 14:58 |
| bnemec | dprince: Heat isn't logging it, Nova is. | 14:59 |
| dprince | bnemec: same :) | 14:59 |
| bnemec | I think it's only logged as info too, which seems valid. | 14:59 |
| *** rbrady has joined #tripleo | 15:00 | |
| *** tremble has quit IRC | 15:01 | |
| *** abregman has quit IRC | 15:03 | |
| *** abregman has joined #tripleo | 15:03 | |
| bnemec | derekh: Note that nova-manage claims it is doing things when you run it verbose: http://paste.openstack.org/show/567496/ | 15:05 |
| *** masco__ has joined #tripleo | 15:08 | |
| *** jlinkes has joined #tripleo | 15:09 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Generate Keystone credentials for overcloud https://review.openstack.org/366287 | 15:09 |
| *** kjw3 has joined #tripleo | 15:10 | |
| *** thrash is now known as thrash|biab | 15:10 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 15:10 |
| *** masco_ has quit IRC | 15:11 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: scenario001: test when ceph is installed https://review.openstack.org/366812 | 15:12 |
| *** hjensas has quit IRC | 15:13 | |
| *** masco__ has quit IRC | 15:15 | |
| derekh | bnemec: so it thinks it doing something but yet shadow_instances contains no entries | 15:15 |
| derekh | MariaDB [nova]> select count(*) from shadow_instances; | 15:15 |
| derekh | | 0 | | 15:15 |
| derekh | How this file got to be 1.3G baffles me | 15:16 |
| derekh | -rw-rw----. 1 mysql mysql 1.3G Sep 7 15:01 /var/lib/mysql/nova/shadow_instances.ibd | 15:16 |
| bnemec | derekh: Well, it doesn't actually claim that it archived any instances. | 15:16 |
| bnemec | Wow. | 15:16 |
| *** ebarrera has quit IRC | 15:17 | |
| bnemec | I do seem to recall reading that those ibd files don't shrink though, so there must have been a lot of data in it at some point. | 15:18 |
| tobias_fiberdata | a question, why is the network part of tripleO so damn complicated? Wouldnt it be easier to put every configuration needed in one file instead of having 15 different files that depends on 15 other files?I'm thinking on the heat-templates. | 15:19 |
| derekh | bnemec: which suggests the archive_deleted_rows command worked at some point | 15:19 |
| sshnaidm | pabelanger, ping me please when you're around | 15:19 |
| openstackgerrit | Alejandro Andreu proposed openstack/tripleo-docs: Fixes path for configure-tempest-directory https://review.openstack.org/366816 | 15:20 |
| b00tcat | ^ just made this small patch, as I had to locate that file manually - this might come in handy for someone else | 15:21 |
| derekh | bnemec: I'm running archive_deleted_rows several times until its stops claiming its archiving things | 15:23 |
| bnemec | derekh: Sounds reasonable. | 15:24 |
| shardy | b00tcat: thanks! | 15:24 |
| *** beagles is now known as beagles_brb | 15:24 | |
| *** lucas-afk is now known as lucasagomes | 15:24 | |
| *** akrivoka has quit IRC | 15:26 | |
| *** ifarkas is now known as ifarkas_afk | 15:27 | |
| derekh | now we're suckin diesel | 15:28 |
| *** akrivoka has joined #tripleo | 15:29 | |
| EmilienM | tripleo CI jobs are broken because RDO has expired SSL certificate | 15:30 |
| EmilienM | http://logs.openstack.org/12/366812/1/check/gate-tripleo-ci-centos-7-undercloud/4f0ab5b/console.html#_2016-09-07_15_28_13_159600 | 15:30 |
| derekh | bnemec: so apparently when using --max_rows X it looks like you need to have at least X instances to archive https://bugs.launchpad.net/nova/+bug/1305892 | 15:31 |
| openstack | Launchpad bug 1305892 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows fails with pgsql on low row count" [Undecided,Expired] | 15:31 |
| nijaba | 669979 | 15:31 |
| EmilienM | jpena: I don't want to disturb during RDO meeting but TripleO CI is broken | 15:31 |
| EmilienM | flepied: ^ | 15:31 |
| jpena | EmilienM: trying to get the SSL cert back, we should be good soon (I hope!= | 15:31 |
| *** kjw3 has quit IRC | 15:31 | |
| EmilienM | thanks | 15:32 |
| EmilienM | @all: please do not 'recheck' | 15:32 |
| *** jpeeler has joined #tripleo | 15:35 | |
| *** jpeeler has joined #tripleo | 15:35 | |
| jpena | EmilienM: review.rdoproject.org should be back in business | 15:36 |
| *** thrash|biab is now known as thrash | 15:37 | |
| EmilienM | jpena: you rocks. | 15:38 |
| *** aufi has quit IRC | 15:39 | |
| sshnaidm | derekh, should we stop te.worker? I see AFS mirrors are slow like a hell again, anyway it won't pass.. | 15:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 15:39 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: scenario001: test when ceph is installed https://review.openstack.org/366812 | 15:39 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Fix the ansible-lint gate job https://review.openstack.org/366836 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop https://review.openstack.org/337587 | 15:39 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: WIP: Fix ansible-lint errors in all the repo https://review.openstack.org/366837 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop https://review.openstack.org/337267 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 15:39 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop https://review.openstack.org/364749 | 15:39 |
| derekh | sshnaidm: go for it, looks like just are failing anyways due to the ssl cert problems | 15:41 |
| bnemec | derekh: I did optimize table on nova.shadow_instances and it dropped to 304 MB. | 15:41 |
| *** chlong has quit IRC | 15:41 | |
| *** anshul has quit IRC | 15:41 | |
| derekh | bnemec: and dropping the number of row you archive at once amd it archive some of them, its not doing nay more for some reason | 15:41 |
| bnemec | derekh: Yeah, I see there are 13000 some rows in the shadow table now. | 15:43 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: Get template contents from plan, not local path https://review.openstack.org/365735 | 15:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 15:44 |
| sshnaidm | derekh, stopped | 15:44 |
| derekh | sshnaidm: ack | 15:45 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 15:46 |
| pabelanger | sshnaidm: hello | 15:46 |
| sshnaidm | pabelanger, hi | 15:46 |
| sshnaidm | pabelanger, maybe you can help | 15:47 |
| sshnaidm | pabelanger, we suffer from weird slowness while using afs pypi mirrors in rh1 | 15:47 |
| bnemec | derekh: Woot, optimize table on heat.event and heat.raw_template took our disk usage down 90% for the Heat db. | 15:47 |
| sshnaidm | pabelanger, sometimes pip install takes about 40 mins, usually it should take about 2 mins | 15:47 |
| pabelanger | sshnaidm: networking issue? | 15:48 |
| sshnaidm | pabelanger, sometimes it just disconnects | 15:48 |
| sshnaidm | pabelanger, seems like it, it passed away during a day but is back now | 15:48 |
| sshnaidm | pabelanger, although I saw first pip connection dropped during last weeks when used rh1 manually.. | 15:49 |
| sshnaidm | pabelanger, afaiu yolanda saw dropped connections in afs logs, maybe it'll help | 15:50 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 15:51 |
| *** rajinir has joined #tripleo | 15:51 | |
| sshnaidm | rh1 is being affected from all kinds of issues that you can imagine.. | 15:51 |
| derekh | bnemec: now we're seeing something usfull in the slow query log (I think we had to wait for new sql session to be restarted) | 15:52 |
| derekh | bnemec: a bunch of 3 second updates on heat tables | 15:52 |
| derekh | bnemec: e.g. | 15:53 |
| derekh | UPDATE resource SET nova_instance='27c8f7eb-833d-459a-8290-a518bf16d7f8' WHERE resource.id = 632236; | 15:53 |
| pabelanger | sshnaidm: I'd say this is likely a networking issue limited to tripleo-test-cloud-rh1. I don't think we are having issues with pip in other cloud regions | 15:53 |
| derekh | # Query_time: 3.707108 Lock_time: 0.000104 Rows_sent: 0 Rows_examined: 1 | 15:53 |
| shardy | ouch :( | 15:53 |
| sshnaidm | pabelanger, I see | 15:53 |
| *** yolanda has quit IRC | 15:53 | |
| bnemec | derekh: Interesting. The resource table isn't actually all that large. | 15:54 |
| pabelanger | sshnaidm: you're going to have to talk with NOC and see what is going on with UDP traffic from the AFS mirror to upstream. Since the network is behind a firewall and port filtering enabled, it could be anything | 15:54 |
| *** yolanda has joined #tripleo | 15:55 | |
| derekh | sshnaidm: in the mean time we could just switch back to the centos mirrors maybe ? | 15:55 |
| bnemec | derekh: Note that I did start a process to keep the cpu pegged. It's niced, but maybe it's still impacting mysql? | 15:56 |
| pabelanger | derekh: not sure that will be easy, we setup mirrors in nodepool | 15:56 |
| derekh | pabelanger: the mirror would still be there, we just wouldn't be using it | 15:57 |
| *** dtantsur is now known as dtantsur|afk | 15:57 | |
| pabelanger | I would not be in favor of that | 15:57 |
| pabelanger | we shouldn't be by passing openstack-infra when things stop working | 15:58 |
| derekh | pabelanger: ya I didn't think you would be, it was just an option | 15:58 |
| sshnaidm | pabelanger, I'm not sure it's not controller machine issue, so talking to NOC..I don't even know who are them :) | 15:58 |
| sshnaidm | pabelanger, why do you think UDP? | 15:58 |
| pabelanger | sshnaidm: derekh: I'm trying to bring rax-iad back online, but happy to help once I finish that | 15:58 |
| pabelanger | when did the problem start? | 15:58 |
| pabelanger | sshnaidm: AFS only supports udp | 15:58 |
| sshnaidm | pabelanger, about 8-10 hours ago | 15:59 |
| pabelanger | So, we should try to isolate the problem | 15:59 |
| pabelanger | is the issue between the node / afs mirror or afs mirror / afs01.dfw.openstack.org | 16:00 |
| derekh | bnemec: I'm not sure if it would be impacting mysql or not | 16:00 |
| *** apevec has joined #tripleo | 16:01 | |
| *** pcaruana has quit IRC | 16:01 | |
| *** dbecker has joined #tripleo | 16:01 | |
| derekh | pabelanger: I don't think we can answer that as we can't get onto the mirror | 16:01 |
| pabelanger | sshnaidm: and since there has been a lot of issue around tripleo-test-cloud-rh1 the while, I'm leaning towards a local issue | 16:02 |
| EmilienM | @all you can do recheck if you see a rdoproject SSL issue. Problem is fixed and tested | 16:02 |
| pabelanger | derekh: once I'm finished with rax-iad, I can look | 16:02 |
| derekh | pabelanger: ack | 16:02 |
| pabelanger | otherwise, #openstack-infra might be able to help | 16:02 |
| *** fultonj_ has joined #tripleo | 16:02 | |
| sshnaidm | pabelanger, derekh maybe it's better not to deal with this until we have rh1 running well, just not to waste time on something that could be a consequence of current problems | 16:04 |
| thrash | jaosorior: so, what part wasn't working? | 16:04 |
| *** beagles_brb is now known as beagles | 16:05 | |
| jaosorior | thrash: the actual haproxy configuration. When you enable SSL websockets aren't working | 16:05 |
| sshnaidm | any neutron problem on rh1 could be a reason | 16:05 |
| pabelanger | right | 16:05 |
| *** abregman has quit IRC | 16:06 | |
| jaosorior | thrash: websockets start with http(s) and then switches down to TCP. For some reason the handshake (I think the one switching down to TCP) fails | 16:06 |
| thrash | jaosorior: tips for doing that? :) | 16:07 |
| thrash | Got a curl command or something? | 16:07 |
| jaosorior | thrash: well, I was just trying to do introspection, since it uses websockets for that | 16:07 |
| *** bana_k has joined #tripleo | 16:07 | |
| thrash | jaosorior: ah... good man. :) | 16:07 |
| weshay | trown, is there a bug for the issue you're hitting locally.. is it gnocci-statsd still? | 16:08 |
| apevec | weshay, trown, EmilienM, derekh - so ovb jobs failing in https://review.openstack.org/359215 is infra issue or something else? | 16:08 |
| sshnaidm | pabelanger, derekh but this is better to merge, I've been seeing this for a weeks on rh1, it's not recently: https://review.openstack.org/#/c/366687/ | 16:08 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop https://review.openstack.org/337587 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop https://review.openstack.org/337267 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml https://review.openstack.org/365792 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop https://review.openstack.org/364749 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format https://review.openstack.org/364748 | 16:09 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop https://review.openstack.org/365796 | 16:09 |
| trown | weshay: I have not filed a bug yet... but yes it is gnocchi-statsd flapping and consuming all resources on the overcloud | 16:09 |
| *** tesseract- has quit IRC | 16:09 | |
| sshnaidm | apevec, ovb is not functional now | 16:09 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic https://review.openstack.org/354019 | 16:09 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add integration with Manila CephFS Native driver https://review.openstack.org/358525 | 16:09 |
| *** zoliXXL is now known as zoli|gone | 16:09 | |
| trown | can be seen pretty clearly in https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-113/overcloud-controller-0/var/log/messages.gz | 16:09 |
| weshay | mburned, ^ | 16:09 |
| trown | filing a bug now | 16:10 |
| *** zoli|gone is now known as zoli_gone-proxy | 16:10 | |
| apevec | weshay, patches that EmilienM listed in master_current_issues had Related-Bug: #1618510 | 16:10 |
| openstack | bug 1618510 in tripleo "unable to reach redis service in non-ha deployments" [Critical,Fix released] https://launchpad.net/bugs/1618510 - Assigned to Jiřà Stránský (jistr) | 16:10 |
| *** akshai has joined #tripleo | 16:10 | |
| trown | actually https://bugs.launchpad.net/tripleo/+bug/1619243 looks like a possible match | 16:10 |
| openstack | Launchpad bug 1619243 in tripleo "CI: periodic jobs fail because of exceeded timeout" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 16:10 |
| apevec | that was for issue 60. in https://etherpad.openstack.org/p/delorean_master_current_issues | 16:10 |
| derekh | apevec: We're definitely having infra issues, our cloud is having problems and we're trying to track it down | 16:11 |
| trown | oh nvm | 16:11 |
| weshay | trown, just confirming.. Sep 7 15:58:34 localhost systemd: Unit openstack-gnocchi-statsd.service entered failed state. | 16:11 |
| weshay | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-113/overcloud-controller-0/var/log/messages.gz | 16:11 |
| panda | bnemec: hey, in https://review.openstack.org/364479, when you set resource_registry in test-environments/ipv6-network-templates/network-isolation.yaml, you use relative paths, and deploy does not find some of the files. | 16:11 |
| EmilienM | I'm not sure why gnocchi tries to use swift backend | 16:11 |
| apevec | trown, ah so that's 59. | 16:11 |
| EmilienM | oh, that's the default backend in Gnocchi composable service. | 16:11 |
| trown | can we change that default? | 16:12 |
| EmilienM | maybe should just we disable gnocchi ? | 16:12 |
| panda | bnemec: should I use that or stick to the one in /usr/share ? | 16:12 |
| EmilienM | we already test it with scenario001 | 16:12 |
| trown | ya, or disable gnocchi by default would also work | 16:12 |
| pabelanger | sshnaidm: not sure I like the idea of hiding networking issue. We should be working to fix them over adding work-arounds to the installation process | 16:12 |
| derekh | sshnaidm: lgtm, but we still gotta sort out the cloud errors | 16:12 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network https://review.openstack.org/366548 | 16:12 |
| EmilienM | trown: I'll let you going ahead | 16:12 |
| EmilienM | or i can do it | 16:13 |
| trown | EmilienM: k, filing a bug first, then will put up a patch | 16:13 |
| bnemec | panda: Yeah, that's intentional. That environment file is supposed to be copied into the tht tree so it can find the templates (this is covered in the new shiny README file :-) | 16:13 |
| EmilienM | trown: put the service optional like we do with sahara, etc | 16:13 |
| apevec | trown, so it's not https://bugs.launchpad.net/tripleo/+bug/1619243 ? | 16:13 |
| openstack | Launchpad bug 1619243 in tripleo "CI: periodic jobs fail because of exceeded timeout" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 16:13 |
| bnemec | panda: I would just use the one from tht itself though: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/network-isolation-v6.yaml | 16:13 |
| pabelanger | sshnaidm: I don't like the idea of adding fixes specific to tripleo-test-cloud-rh1, and we should avoid doing that if possible | 16:13 |
| b00tcat | hi, is there any extensive doc on tripleo heat templates? as in, what's in each folder and so on? | 16:13 |
| trown | apevec: the description is similar, but looks like a different root cause | 16:13 |
| bnemec | panda: That should be doing the same thing in this case since we're enabling all the networks. | 16:13 |
| bnemec | The generated one is more for if you're only using some networks and need a custom network-isolation.yaml. | 16:14 |
| derekh | sshnaidm: bnemec: I only got another hour, wanna just restart the controller anyways and see if it helps, I can't think of anything else worth trying | 16:15 |
| weshay | trown, EmilienM so we're going to give it one more go w/ disabling gnocci? | 16:15 |
| apevec | trown, yeah, that looks like networking issue from rh1 to trunk.rdo | 16:15 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 16:15 |
| apevec | weshay, that means new hash | 16:15 |
| weshay | aye | 16:15 |
| apevec | change would be in THT right? | 16:15 |
| shardy | b00tcat: http://docs.openstack.org/developer/tripleo-docs/developer/tht_walkthrough/tht_walkthrough.html is a good place to start for per-service templates | 16:15 |
| derekh | sshnaidm: bnemec I mean the outage is about 2 days now, so it can't get much worse ;-)....we'll >2 is worse but you know what I mean | 16:16 |
| panda | bnemec: ok, thanks, rerunning with stock net-iso-v6 from tht then. | 16:16 |
| shardy | b00tcat: I have some blog posts which may help at http://hardysteven.blogspot.co.uk/ | 16:16 |
| b00tcat | thanks shardy , I just stumbled across your blog too in google :D | 16:16 |
| shardy | b00tcat: also we recorded some deep-dives which are on youtube: | 16:16 |
| shardy | https://etherpad.openstack.org/p/tripleo-deep-dive-topics | 16:16 |
| marios | tbarron: so, ftr, the issue you saw was indeed cephfs related and not to do with generic or netapp | 16:17 |
| bnemec | derekh: Yeah, I have no objections to trying a reboot. Like you say, things are pretty well hosed anyway. | 16:17 |
| shardy | b00tcat: https://www.youtube.com/watch?v=gX5AKSqRCiU in particular may help give you an overview | 16:17 |
| b00tcat | thanks! | 16:17 |
| tbarron | marios: ack, and thans for the patches, i'll try again when I get out of meeting hell | 16:17 |
| *** florianf has quit IRC | 16:17 | |
| derekh | sshnaidm: bnemec: ok, anything ye need to finish up on the controller before I reboot ?? | 16:18 |
| marios | tbarron: i have updated the tht for cephfs https://review.openstack.org/#/c/358525/ and made a new (fixup) puppet-tripleo depends on which it now points at which is https://review.openstack.org/#/c/366760/ | 16:18 |
| tbarron | marios: ack | 16:18 |
| marios | tbarron: both of those are rebased onto their netapp counterpart... i.e. tht side if you see shortlog https://review.openstack.org/gitweb?p=openstack/tripleo-heat-templates.git;a=shortlog;h=80b658f6135fbc7863c83abfc95d705477bf3c68 | 16:18 |
| derekh | pabelanger: FYI: rebooting the rh1 controller , for lack of a better thing to try | 16:18 |
| marios | tbarron: same for the puppet-tripleo reviews... made sense since cephfs side was waiting to see the pattern established by netapp/generic | 16:19 |
| bnemec | derekh: No, the cleanup I'm running can be restarted once the controller is back up. | 16:19 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 16:19 |
| marios | tbarron: but for your testing, because of the manila puppet-tripleo class error (cos of cephfs) you will need to include the puppet-tripleo side cephfs fixup too. | 16:19 |
| tbarron | marios: ok | 16:20 |
| trown | EmilienM: thinking about it, wouldnt making our user-facing default match the multinode test be better than disabling gnocchi? | 16:20 |
| trown | filed https://bugs.launchpad.net/tripleo/+bug/1621164 btw | 16:21 |
| openstack | Launchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [Critical,Triaged] | 16:21 |
| marios | tbarron: will catchup tomorrow. so in any case, the netapp should land first, but really they'll all land together (since you won't be able to deploy wi/out the cephfs fixup) | 16:21 |
| derekh | sshnaidm: bnemec ok, going for it | 16:21 |
| *** akshai_ has joined #tripleo | 16:21 | |
| *** akshai has quit IRC | 16:21 | |
| tbarron | marios: have a good night, see you tomorrow :) | 16:24 |
| EmilienM | trown: how would you do that? | 16:25 |
| EmilienM | pradk: in case you're missing the conversation about gnocchi problem ^ | 16:25 |
| *** pkovar has quit IRC | 16:25 | |
| trown | EmilienM: hmm not sure actually... there is no default environment file that is passed in for the base case | 16:27 |
| trown | EmilienM: we could change the default in puppet | 16:27 |
| EmilienM | trown: I would suggest to switch the default backend to "file" in tripleo | 16:28 |
| *** dmsimard|afk is now known as dmsimard | 16:28 | |
| EmilienM | 'file' backend just work out of the box | 16:28 |
| EmilienM | and we are currently gating on it | 16:28 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 16:28 |
| EmilienM | I'm afk for lunch but will follow-up later | 16:28 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Cleanup the existing plan before deploying if one already exists https://review.openstack.org/366541 | 16:29 |
| trown | EmilienM: where can we set that default though in tripleo that it would be picked up by simply `openstack overcloud deploy` with no args | 16:29 |
| trown | k | 16:29 |
| sshnaidm | derekh, yeah, I think it's the best thing to do now.. | 16:29 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 16:29 |
| sshnaidm | derekh, gotta run for a while, I'll be back later | 16:30 |
| *** pkovar has joined #tripleo | 16:30 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:30 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names. https://review.openstack.org/366529 | 16:30 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove "type: direct" from workflows as it is the default https://review.openstack.org/341617 | 16:30 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages https://review.openstack.org/364252 | 16:30 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use a released version of tripleo-common https://review.openstack.org/364425 | 16:30 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration https://review.openstack.org/326036 | 16:31 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Fix an autoclass reference and add missing pages to the toctree https://review.openstack.org/342747 | 16:31 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Ignore the .eggs directory https://review.openstack.org/351126 | 16:31 |
| *** jpich has quit IRC | 16:31 | |
| *** mhenkel has joined #tripleo | 16:31 | |
| *** jaosorior has quit IRC | 16:33 | |
| *** ayoung has quit IRC | 16:35 | |
| *** yamahata has quit IRC | 16:35 | |
| *** ayoung has joined #tripleo | 16:36 | |
| pradk | EmilienM, hmm so we dont run swift out of the box either? | 16:36 |
| pradk | EmilienM, thought we did? | 16:36 |
| pradk | trown, trying to understand why you want to change the backend.. i would prefer we test it with swift as backend by default | 16:37 |
| trown | pradk: because with swift backend statsd blows up the overcloud | 16:38 |
| trown | it loops trying to start and consumes all CPU | 16:38 |
| trown | https://bugs.launchpad.net/tripleo/+bug/1621164 | 16:38 |
| openstack | Launchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [Critical,Triaged] | 16:38 |
| pradk | trown, the same auth error? | 16:38 |
| trown | pradk: ya https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-113/overcloud-controller-0/var/log/gnocchi/statsd.log.gz | 16:39 |
| *** mbound has quit IRC | 16:39 | |
| derekh | bnemec: controller back up, FIP's not working | 16:39 |
| pradk | trown, looking at the conf.. one sec | 16:39 |
| bnemec | derekh: Awesome. :-( | 16:40 |
| bnemec | derekh: Wonder if we need to bounce ovs-agent on the compute nodes. | 16:41 |
| EmilienM | pradk, trown: AFIK swift is deployed by default so it might we something we would need to fix in Gnocchi? | 16:42 |
| pradk | yea i'm checking the conf .. | 16:42 |
| openstackgerrit | Merged openstack/tripleo-common: Fix an autoclass reference and add missing pages to the toctree https://review.openstack.org/342747 | 16:42 |
| pradk | no point switching to file by default, instead lets fix the auth issue.. lemme chase it down | 16:42 |
| pradk | trown, EmilienM, if you want to override to file just for ci.. thats probably ok | 16:43 |
| derekh | bnemec: nope, I thinks the controller might be the problem http://paste.openstack.org/show/567578/ | 16:43 |
| pradk | i'll figure out the auth issue in a sec | 16:43 |
| *** akshai_ has quit IRC | 16:43 | |
| derekh | bnemec: this also used to happen on the 3 year old deployment of RH1... | 16:43 |
| trown | pradk: well the other option is to disable gnocchi by default... | 16:43 |
| EmilienM | pradk: I agree | 16:43 |
| *** myoung is now known as myoung|bbl | 16:44 | |
| EmilienM | pradk: i'm working on testing rbd backend instead of file for tripleo/scenario001 | 16:44 |
| trown | it doesnt work as configured...and our CI is just masking it | 16:44 |
| pradk | trown, if nothing is using it sure | 16:44 |
| derekh | bnemec: the hostname now has .localdomain appended to it, we need to remove that and restart the neutron services | 16:44 |
| bnemec | derekh: Oh ****, that bug. | 16:44 |
| derekh | bnemec: so that they have the origional hostname | 16:44 |
| bnemec | derekh: Yeah, I think we may need to do nova too. | 16:45 |
| derekh | bnemec: doing it now | 16:45 |
| derekh | bnemec: ok | 16:45 |
| *** lucasagomes is now known as lucas-dinner | 16:45 | |
| *** fultonj_ has quit IRC | 16:48 | |
| *** trown is now known as trown|lunch | 16:50 | |
| pradk | EmilienM, shouldnt keystone create the service info at the beginning? or is it still in post deploy step via tripleoclient? | 16:54 |
| pradk | the keystone init step i mean | 16:55 |
| derekh | bnemec: ok, FIPS working again | 16:55 |
| EmilienM | pradk: i'm not sure tbh | 16:55 |
| EmilienM | pradk: do we deploy swift in the CI job? | 16:56 |
| bnemec | derekh: Cool | 16:56 |
| derekh | bnemec: I believe the cleanup script may have deleted the mirror server | 16:56 |
| EmilienM | yes I see https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-113/overcloud-controller-0/var/log/swift/swift.log.gz | 16:56 |
| bnemec | derekh: !!! | 16:56 |
| derekh | bnemec: I'm going to deploy a new one then I gotta vanish of a couple of hours, will come back to see what else I can help with | 16:56 |
| EmilienM | pradk: i'm afk for lunch, brb | 16:56 |
| pradk | EmilienM, the issue seems to be that swift client cannot get a token so its credential are invalid | 16:57 |
| *** bana_k has quit IRC | 16:57 | |
| *** jbadiapa has quit IRC | 16:57 | |
| bnemec | derekh: Cool, thanks | 16:59 |
| derekh | bnemec: I think in ~heat-admin/cleanup-stack nova list for -${num} I think may have got it (mathced the instance uuid) | 16:59 |
| derekh | bnemec: anyways its gone now | 16:59 |
| bnemec | derekh: Ah, dammit. I forgot the uuid also has -'s in it. | 17:00 |
| * bnemec headdesks | 17:00 | |
| EmilienM | pradk: swift_tenant_name is missing in gnocchi.conf, no? | 17:00 |
| EmilienM | is it critical? | 17:00 |
| pradk | EmilienM, the error seems to indicate not.. it thinks the credentials are not valid. which means they are not in keystone | 17:01 |
| *** jlinkes has quit IRC | 17:01 | |
| EmilienM | why would we fail now? | 17:02 |
| EmilienM | what did we change recently? | 17:02 |
| pradk | EmilienM, not sure if this could be relevant but the url and uri are different | 17:03 |
| pradk | [keystone_authtoken] | 17:03 |
| pradk | auth_uri=http://172.16.2.4:5000/v2.0 | 17:03 |
| pradk | auth_url=http://192.0.2.8:35357 | 17:03 |
| pradk | should they both be the same 172. | 17:03 |
| pradk | me checks other services | 17:03 |
| EmilienM | it's network isolation, admin network vs public network, should be fine | 17:03 |
| pradk | yea ceilo has the same | 17:04 |
| pradk | i'm wondering if swift is even up before gnocchi tries to connect | 17:04 |
| pradk | are swift credentials in keystone when statsd requests it | 17:04 |
| EmilienM | ok I'm afk for real, back in 20 min | 17:05 |
| pradk | this was an issue in mitaka as keystone init was done by tripleoclient | 17:05 |
| derekh | ok, its there now, will be back later | 17:05 |
| pradk | i thought we moved aways from that to puppet managing it | 17:05 |
| derekh | bnemec: ^ | 17:05 |
| pradk | away | 17:05 |
| pradk | but not sure | 17:05 |
| *** derekh is now known as derekh_ark | 17:05 | |
| *** derekh_ark is now known as derekh_afk | 17:05 | |
| EmilienM | afik puppet manage endpoints etc | 17:05 |
| EmilienM | maybe gnocchi starts before that | 17:06 |
| EmilienM | we need to check in puppet tripleo, the steps | 17:06 |
| bnemec | derekh_afk: Thanks. Will fix the cleanup script so it doesn't happen again. | 17:06 |
| *** fragatina has joined #tripleo | 17:08 | |
| *** jlinkes has joined #tripleo | 17:08 | |
| *** sshnaidm|afk is now known as sshnaidm | 17:11 | |
| *** dbecker has quit IRC | 17:12 | |
| *** flepied has quit IRC | 17:12 | |
| *** jcoufal has quit IRC | 17:13 | |
| jpena | we've had the latest python-tripleoclient fail to build from source in RDO: https://review.rdoproject.org/r/2104 | 17:13 |
| jpena | from the logs and some tests I've done, it only happens with more recent python-heatclient than what's in the gate | 17:13 |
| *** links has joined #tripleo | 17:14 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common: Wire in jinja templating for custom roles https://review.openstack.org/362465 | 17:14 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common: Enable j2 rendering of any file, not just overcloud.yaml https://review.openstack.org/366877 | 17:14 |
| sshnaidm | bnemec, derekh_afk so what is the plan now? continue to clean up? | 17:15 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common: Enable j2 rendering of any file, not just overcloud.yaml https://review.openstack.org/366877 | 17:16 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 17:18 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 17:18 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop https://review.openstack.org/365796 | 17:18 |
| *** jpena is now known as jpena|away | 17:19 | |
| *** shardy has quit IRC | 17:19 | |
| sshnaidm | slagle, hi | 17:20 |
| EmilienM | pradk: back | 17:25 |
| *** pkovar has quit IRC | 17:25 | |
| EmilienM | gnocchi statsd is deployed at step 4 while keystone create gnocchi credentials at step 5 | 17:26 |
| EmilienM | pradk: i'm proposing a patch. weshay, trown|lunch: is there a quick way to test my patch? (RDO CI?) | 17:26 |
| *** ayoung has quit IRC | 17:27 | |
| bnemec | sshnaidm: Yeah, basically. | 17:27 |
| weshay | EmilienM, there is.. it stopped working last night after a ci.centos jenkins upgrade.. I have an email on it. apetrich you still around? | 17:27 |
| pradk | EmilienM, that matches the error.. i see.. $step >= 4 | 17:28 |
| *** links has quit IRC | 17:29 | |
| EmilienM | pradk: I'm working on a patch now. | 17:29 |
| pradk | ok | 17:29 |
| EmilienM | pradk: it will be the first time scenario001 will be actually useful, as the patch will be test against a real gnocchi deployment (with file backend now though) | 17:31 |
| pradk | EmilienM, cool | 17:31 |
| pradk | EmilienM, hmm but since statsd is loopong around shouldnt it fix itself after step 5 is called? unless its not getting called | 17:34 |
| pradk | gnocchi_api_enabled fallsback to false.. lemme check if thats true in deploy | 17:34 |
| slagle | sshnaidm: hello | 17:36 |
| *** yamahata has joined #tripleo | 17:38 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: gnocchi: move statsd and metricd at step5 https://review.openstack.org/366887 | 17:40 |
| EmilienM | pradk: ^ | 17:40 |
| EmilienM | weshay: ^ feel free to test it | 17:40 |
| sshnaidm | slagle, I have a question about multinode job, how does it known where exactly to deploy overcloud? How is all subnodes info is got into overcloud deployment? | 17:41 |
| slagle | sshnaidm: there are various files that are populated under /etc/nodepool, by nodepool itself | 17:41 |
| pradk | EmilienM, cool, I'll do a new deploy in a bit .. will pull this in | 17:41 |
| sshnaidm | slagle, yeah | 17:42 |
| weshay | EmilienM, k | 17:42 |
| EmilienM | gate-tripleo-ci-centos-7-scenario001-multinode-nv is in queue, let's see how it goes | 17:42 |
| *** nyechiel_ has quit IRC | 17:43 | |
| slagle | sshnaidm: so then all we need to do in tripleo-ci is look at those files to get the IP addresses of the other nodes | 17:43 |
| *** saneax-_-|AFK is now known as saneax | 17:46 | |
| sshnaidm | slagle, ok, and then? | 17:46 |
| slagle | sshnaidm: then what? can you elaborate on what you're asking | 17:46 |
| apetrich | weshay, if you open the config and save again it starts to work again | 17:46 |
| sshnaidm | slagle, do you create instackenv.json from this? | 17:47 |
| weshay | apetrich, yup.. just took the time to read it.. apetrich I'm adding puppet-tripleo to the trigger.. ok w/ you? will send a review shortly | 17:47 |
| slagle | sshnaidm: no. we don't need that b/c the multinode jobs do not use ironic | 17:47 |
| apetrich | sure am | 17:47 |
| apetrich | weshay, ^ | 17:47 |
| sshnaidm | slagle, I mean how overcloud deploy knows exactly where to deploy, I think I miss the connection here | 17:47 |
| slagle | sshnaidm: once the overcloud deployment starts, we ssh into the subnodes and configure os-collect-config to poll for metadata from Heat | 17:48 |
| slagle | sshnaidm: we configure os-collect-config on the subnodes to poll Heat for metadata on the undercloud | 17:48 |
| slagle | usually that step is done via injected user-data from nova | 17:48 |
| slagle | since we're not using nova to deploy the nodes, we do it manually via ssh | 17:49 |
| *** bana_k has joined #tripleo | 17:49 | |
| slagle | sshnaidm: look at where we call get-occ-config.sh in tripleo-ci | 17:49 |
| slagle | that is the script that will ssh to each subnode and configure os-collect-config to poll the undercloud | 17:50 |
| weshay | apetrich, https://review.gerrithub.io/290481 | 17:50 |
| apetrich | weshay, neat +1ed | 17:52 |
| *** mbound has joined #tripleo | 17:52 | |
| *** fragatina has quit IRC | 17:52 | |
| *** flepied has joined #tripleo | 17:52 | |
| *** fragatina has joined #tripleo | 17:53 | |
| EmilienM | I'm currently testing scenario001 with Ceph and RBD backend for Glance, Nova and Gnocchi. The pingtest is failing at uploading a second image on glance (first seems to work) with this error: 504 Gateway Time-out: The server didn't respond in time | 17:53 |
| EmilienM | does it ring a bell? | 17:53 |
| EmilienM | logs are available here http://logs.openstack.org/12/366812/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/40f9846/logs/subnode-2 | 17:53 |
| EmilienM | I haven't found 504 in glance logs | 17:53 |
| *** trown|lunch is now known as trown | 17:56 | |
| EmilienM | it sounds like a timeout | 17:57 |
| *** mandre has quit IRC | 17:57 | |
| EmilienM | trown: FYI https://review.openstack.org/#/c/366887/ | 17:57 |
| *** mcornea has quit IRC | 17:58 | |
| trown | EmilienM: thanks giving it a spin locally | 17:58 |
| openstackgerrit | Paul Belanger proposed openstack/python-tripleoclient: DNM Testing experimental https://review.openstack.org/366892 | 18:02 |
| *** saneax is now known as saneax-_-|AFK | 18:02 | |
| *** saneax-_-|AFK is now known as saneax | 18:02 | |
| *** fragatina has quit IRC | 18:03 | |
| *** fragatina has joined #tripleo | 18:04 | |
| *** mandre has joined #tripleo | 18:05 | |
| *** dsariel has quit IRC | 18:07 | |
| *** ebarrera has joined #tripleo | 18:09 | |
| *** panda is now known as panda|dinner | 18:09 | |
| *** jcoufal has joined #tripleo | 18:10 | |
| *** akshai has joined #tripleo | 18:13 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenarios: set Debug to True https://review.openstack.org/366896 | 18:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 18:21 |
| trown | EmilienM: in order to test puppet-tripleo changes does the change need to be on the overcloud image? | 18:23 |
| *** jlinkes has quit IRC | 18:23 | |
| EmilienM | trown: yes | 18:23 |
| trown | ah k that would explain my test run still doing gnochhi in step 4 | 18:24 |
| *** jlinkes has joined #tripleo | 18:36 | |
| EmilienM | bnemec: I missed all your convo with derekh_afk, what is rh1 status now? still down? | 18:37 |
| EmilienM | still see lot of red on http://tripleo.org/cistatus.html | 18:37 |
| bnemec | EmilienM: Yes, still down. :-( | 18:38 |
| bnemec | Close to being able to turn it back on though. | 18:38 |
| EmilienM | ok great, thanks | 18:39 |
| bnemec | One stack that doesn't want to go away for some reason. | 18:39 |
| EmilienM | please let us know, maybe on ML or something | 18:39 |
| *** ayoung has joined #tripleo | 18:39 | |
| *** goneri_ has joined #tripleo | 18:39 | |
| *** goneri_ has quit IRC | 18:41 | |
| *** abregman has joined #tripleo | 18:41 | |
| *** jcoufal_ has joined #tripleo | 18:44 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 18:44 |
| *** jcoufal has quit IRC | 18:45 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Expose parameter to enable combination alarms https://review.openstack.org/363748 | 18:45 |
| *** abregman is now known as abregman|nb | 18:46 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 18:47 |
| *** myoung|bbl is now known as myoung | 18:54 | |
| *** athomas has quit IRC | 18:56 | |
| *** akshai has quit IRC | 18:59 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: gnocchi: move statsd and metricd at step5 https://review.openstack.org/366887 | 19:01 |
| trown | anyone can do a quick review on https://review.openstack.org/366887 it would be blocking the OVB jobs if the OVB jobs were not already blocked and it blocks promotion in RDO | 19:03 |
| *** saneax is now known as saneax-_-|AFK | 19:03 | |
| EmilienM | trown: thx for the review | 19:03 |
| EmilienM | trown: i'm afk a bit, bbl | 19:03 |
| trown | EmilienM: thanks for the quick patch | 19:03 |
| EmilienM | hope it will help | 19:03 |
| trown | still seems like a bit of a gnocchi bug that it can go so haywire if it is unable to start | 19:04 |
| trown | but that patch does fix it | 19:04 |
| trown | thanks slagle | 19:06 |
| slagle | np | 19:10 |
| *** dsariel has joined #tripleo | 19:11 | |
| *** panda|dinner is now known as panda | 19:11 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient: Add heat-config-apply-config element to images https://review.openstack.org/366912 | 19:12 |
| *** saneax-_-|AFK is now known as saneax | 19:13 | |
| derekh_afk | bnemec: back | 19:13 |
| *** derekh_afk is now known as derekh | 19:13 | |
| *** lucas-dinner has quit IRC | 19:13 | |
| *** jbadiapa has joined #tripleo | 19:14 | |
| bnemec | derekh: o/ | 19:14 |
| derekh | bnemec: does anything seem better at all? | 19:14 |
| bnemec | The stacks and instances are all gone. | 19:14 |
| bnemec | Finishing cleanup on the network bits. | 19:14 |
| *** lucasagomes has joined #tripleo | 19:15 | |
| *** jbadiapa has quit IRC | 19:16 | |
| *** fragatin_ has joined #tripleo | 19:17 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient: Add heat-config-apply-config element to images https://review.openstack.org/366912 | 19:19 |
| *** fragatina has quit IRC | 19:21 | |
| *** rcernin has quit IRC | 19:21 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Use new 'apply-config' to apply oac data https://review.openstack.org/366918 | 19:28 |
| beagles | dprince, will these patches effectively address https://bugs.launchpad.net/tripleo/+bug/1596373 ? | 19:31 |
| openstack | Launchpad bug 1596373 in tripleo "40-hiera-datafiles takes over 20 seconds to run" [High,Triaged] | 19:31 |
| *** saneax is now known as saneax-_-|AFK | 19:34 | |
| bnemec | derekh: I think we're just about ready to try te-broker again. I assume we want to restart geard again too? | 19:35 |
| derekh | bnemec: yup | 19:36 |
| derekh | bnemec: are you doing it or will I? | 19:36 |
| bnemec | derekh: I'll do it. | 19:36 |
| derekh | ok | 19:37 |
| bnemec | te-workers are starting | 19:37 |
| derekh | bnemec: ok | 19:38 |
| bnemec | derekh: I also tried halving the number of api workers for nova and neutron in hopes that maybe less concurrency will help with all these issues. | 19:39 |
| derekh | bnemec: makes sense, there is a lot of processes running that are possibly redundant | 19:39 |
| openstackgerrit | Merged openstack/python-tripleoclient: Ignore the .eggs directory https://review.openstack.org/351126 | 19:40 |
| *** abregman|nb has quit IRC | 19:41 | |
| *** ebarrera has quit IRC | 19:51 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Re-add undercloud.yaml https://review.openstack.org/352037 | 19:52 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Use new 'apply-config' to apply oac data https://review.openstack.org/366918 | 19:52 |
| *** fragatin_ has quit IRC | 19:53 | |
| *** fragatina has joined #tripleo | 19:54 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient: Deploy the undercloud with Heat https://review.openstack.org/351351 | 19:55 |
| *** sshnaidm is now known as sshnaidm|afk | 20:01 | |
| *** dprince has quit IRC | 20:01 | |
| derekh | bnemec: everythings looking ok so far that I've seen, you see any problems? | 20:02 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/365625 | 20:03 |
| bnemec | derekh: Nope, not so far. Crossing my fingers. | 20:04 |
| *** openstackgerrit has quit IRC | 20:04 | |
| *** openstackgerrit has joined #tripleo | 20:04 | |
| *** mbound has quit IRC | 20:04 | |
| weshay | EmilienM, your patch passed my local test | 20:07 |
| weshay | trown, ^ | 20:07 |
| * weshay checks it to see if everything we expect is there | 20:08 | |
| *** jlinkes has quit IRC | 20:11 | |
| *** jayg is now known as jayg|g0n3 | 20:17 | |
| apetrich | weshay, I don't understand I've just reran the jjb and it created correctly | 20:20 |
| weshay | apetrich, yes sir | 20:20 |
| weshay | apetrich, I think we're hitting a jenkins bug | 20:21 |
| apetrich | weshay, seems likely | 20:21 |
| weshay | will explain later | 20:21 |
| *** akrivoka has quit IRC | 20:22 | |
| *** pblaho has quit IRC | 20:22 | |
| apetrich | weshay, no worries about that. what I'm more worried about is the testing job that failed because of the new --release ${CI_ENV:+$CI_ENV/}$RELEASE${REL_TYPE:+-$REL_TYPE} \ | 20:25 |
| apetrich | that is trying to get the release cicentos/master-testing.yml that is not there | 20:25 |
| derekh | bnemec: sshnaidm|afk things appear to be doing reasonably, I've seen 2 failures getting testenvs, hopefully isolated problems, I'm gonna call it a night | 20:25 |
| *** jlinkes has joined #tripleo | 20:25 | |
| *** derekh has quit IRC | 20:25 | |
| apetrich | weshay, there's only https://github.com/openstack/tripleo-quickstart/tree/master/config/release/centosci | 20:26 |
| apetrich | weshay, should we rename REL_TYPE from testing so something else? like consistent ? | 20:27 |
| weshay | apetrich, I just opened a bug on that in launchpad | 20:27 |
| apetrich | oh cool | 20:27 |
| weshay | apetrich, I think adarazs needs to have a look | 20:28 |
| apetrich | aye | 20:28 |
| *** Goneri has quit IRC | 20:30 | |
| *** kjw3 has joined #tripleo | 20:34 | |
| *** jeckersb_gone is now known as jeckersb | 20:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: OVN heat templates https://review.openstack.org/307734 | 20:46 |
| *** jcoufal_ has quit IRC | 20:48 | |
| *** dsariel has quit IRC | 20:53 | |
| *** jcoufal has joined #tripleo | 20:54 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 20:57 |
| *** trown is now known as trown|outtypewww | 20:57 | |
| *** lblanchard has quit IRC | 20:58 | |
| *** rcrit has joined #tripleo | 21:00 | |
| rcrit | quickstart is trying to fetch a non-existant undercloud image, http://artifacts.ci.centos.org/artifacts/rdo/images/master/delorean/stable/undercloud.qcow2.md5, and of course failing | 21:01 |
| rcrit | this worked a couple of days ago. | 21:01 |
| rcrit | I updated my pull and same | 21:02 |
| rcrit | starting with ./quickstart.sh --config config/general_config/ha.yml -R master --no-clone $HOST | 21:02 |
| *** NikoHermannsEri1 has joined #tripleo | 21:02 | |
| *** NikoHermannsEric has quit IRC | 21:03 | |
| *** mbound has joined #tripleo | 21:05 | |
| *** jlinkes has quit IRC | 21:05 | |
| *** fzdarsky has quit IRC | 21:06 | |
| *** mburned is now known as mburned_out | 21:09 | |
| *** mbound has quit IRC | 21:10 | |
| *** jlinkes has joined #tripleo | 21:12 | |
| *** bana_k has quit IRC | 21:14 | |
| *** bank_ has joined #tripleo | 21:14 | |
| *** rhallisey has quit IRC | 21:15 | |
| *** colonwq is now known as colonwq_afk | 21:18 | |
| *** fzdarsky has joined #tripleo | 21:23 | |
| rcrit | dropping the --config value seems to have helped | 21:26 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 21:39 |
| apevec | weshay, EmilienM, so issue is that gnocchi goes mad w/o keystone!? re. https://review.openstack.org/366887 | 21:47 |
| apevec | if so, I don't think closes-bug is quite right, it should be added as a bug in gnocchi | 21:47 |
| apevec | pradk, https://bugs.launchpad.net/tripleo/+bug/1621164 | 21:48 |
| openstack | Launchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:48 |
| * apevec adds it in LP | 21:49 | |
| pradk | apevec, the service is just retrying in a loop as it cant get auth token, i dont know if its a bug | 21:50 |
| apevec | pradk, bug is that it tries in such a tight loop | 21:51 |
| apevec | this is esp bad w/ new HA plan, where all services must reconnect reliably | 21:51 |
| apevec | anyway, I've added https://bugs.launchpad.net/gnocchi/+bug/1621164 please comment there | 21:52 |
| openstack | Launchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:52 |
| pradk | apevec, k please add necessary info so we can look at it upstream | 21:52 |
| pradk | cool | 21:52 |
| apevec | yeah, we'll need simpler reproducer that "run tripleo CI job" | 21:52 |
| apevec | but it's a serious issue imho | 21:53 |
| *** jprovazn has quit IRC | 21:53 | |
| EmilienM | I agree, service should loop until it works | 21:55 |
| apevec | EmilienM, yes but not all-cpu-belongs-to-me kind of loop :) | 21:56 |
| EmilienM | Right | 21:56 |
| EmilienM | apevec: today promotion in puppet CI failed for transients issues | 21:57 |
| EmilienM | It should land tonight | 21:57 |
| apevec | cool, many stars aligned tonight | 21:57 |
| EmilienM | Yeah | 21:57 |
| EmilienM | Now I'm afk ttyk | 21:58 |
| EmilienM | Ttyl even | 21:58 |
| *** jlinkes_ has joined #tripleo | 22:00 | |
| *** jlinkes has quit IRC | 22:03 | |
| *** Goneri has joined #tripleo | 22:05 | |
| *** bfournie has quit IRC | 22:08 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 22:09 |
| openstackgerrit | Merged openstack/puppet-tripleo: gnocchi: move statsd and metricd at step5 https://review.openstack.org/366887 | 22:11 |
| *** chlong has joined #tripleo | 22:12 | |
| *** cdearborn has quit IRC | 22:16 | |
| *** akshai has joined #tripleo | 22:30 | |
| *** dbecker has joined #tripleo | 22:31 | |
| *** fragatin_ has joined #tripleo | 22:34 | |
| *** apevec_ has joined #tripleo | 22:34 | |
| *** rlandy has quit IRC | 22:36 | |
| *** apevec has quit IRC | 22:37 | |
| *** fragatina has quit IRC | 22:37 | |
| *** fzdarsky has quit IRC | 22:41 | |
| *** jlinkes_ has quit IRC | 22:55 | |
| *** saneax-_-|AFK is now known as saneax | 22:55 | |
| *** pradk has quit IRC | 22:56 | |
| *** dbecker has quit IRC | 22:56 | |
| *** fragatin_ has quit IRC | 22:57 | |
| *** apevec_ has quit IRC | 23:03 | |
| *** Goneri has quit IRC | 23:09 | |
| *** jlinkes has joined #tripleo | 23:09 | |
| *** yamahata has quit IRC | 23:11 | |
| *** NikoHermannsEri1 has quit IRC | 23:11 | |
| *** rajinir has quit IRC | 23:15 | |
| *** maeca2 has joined #tripleo | 23:16 | |
| *** maeca1 has quit IRC | 23:16 | |
| *** dhill_ has quit IRC | 23:18 | |
| *** beagles has left #tripleo | 23:18 | |
| *** akshai has quit IRC | 23:23 | |
| *** akshai has joined #tripleo | 23:24 | |
| *** dhill_ has joined #tripleo | 23:24 | |
| *** akshai has quit IRC | 23:25 | |
| *** yamahata has joined #tripleo | 23:38 | |
| *** colonwq_afk is now known as colonwq | 23:43 | |
| *** akshai has joined #tripleo | 23:49 | |
| *** fultonj has quit IRC | 23:54 | |
| *** akshai has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!