| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 00:00 |
|---|---|---|
| *** jkilpatr has quit IRC | 00:01 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: WIP: restrict nova migration ssh tunnel https://review.openstack.org/458077 | 00:03 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684297 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1684297 in tripleo "Augeas[docker-daemon.json]: Could not evaluate. docker package removed /etc/docker/daemon.json" [Critical,In progress] - Assigned to Dan Prince (dan-prince) | 00:10 |
| *** jkilpatr has joined #tripleo | 00:14 | |
| *** chkumar|sleeping is now known as chandankumar | 00:20 | |
| *** brault has quit IRC | 00:22 | |
| *** brault has joined #tripleo | 00:28 | |
| *** mhenkel_ has joined #tripleo | 00:33 | |
| *** mhenkel_ has quit IRC | 00:38 | |
| *** karimb has quit IRC | 00:43 | |
| *** mhenkel_ has joined #tripleo | 00:49 | |
| *** jerrygb has joined #tripleo | 00:50 | |
| *** limao has joined #tripleo | 00:52 | |
| *** mhenkel_ has quit IRC | 00:54 | |
| *** jerrygb has quit IRC | 00:56 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 01:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684297 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1684297 in tripleo "Augeas[docker-daemon.json]: Could not evaluate. docker package removed /etc/docker/daemon.json" [Critical,In progress] - Assigned to Dan Prince (dan-prince) | 01:10 |
| *** mhenkel_ has joined #tripleo | 01:13 | |
| *** mhenkel_ has quit IRC | 01:18 | |
| *** cwolferh has quit IRC | 01:18 | |
| *** fzdarsky_ has joined #tripleo | 01:22 | |
| *** fzdarsky has quit IRC | 01:25 | |
| *** dmacpher-afk has quit IRC | 01:26 | |
| *** jerrygb has joined #tripleo | 01:43 | |
| *** dsariel has quit IRC | 02:03 | |
| *** kjw3 has joined #tripleo | 02:04 | |
| *** mhenkel_ has joined #tripleo | 02:07 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684297 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1684297 in tripleo "Augeas[docker-daemon.json]: Could not evaluate. docker package removed /etc/docker/daemon.json" [Critical,In progress] - Assigned to Dan Prince (dan-prince) | 02:10 |
| *** yamahata has quit IRC | 02:11 | |
| *** mhenkel_ has quit IRC | 02:12 | |
| *** bkopilov has quit IRC | 02:13 | |
| *** michapma_dsk has joined #tripleo | 02:25 | |
| *** rlandy|bbl is now known as rlandy | 02:31 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 02:42 |
| *** gkadam has joined #tripleo | 02:53 | |
| *** mhenkel_ has joined #tripleo | 02:54 | |
| *** jerrygb has quit IRC | 02:55 | |
| *** mhenkel_ has quit IRC | 02:58 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Ensure /etc/docker/daemon.json https://review.openstack.org/458253 | 03:03 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 03:10 |
| *** dmacpher has joined #tripleo | 03:14 | |
| *** psahoo has joined #tripleo | 03:31 | |
| *** bkopilov has joined #tripleo | 03:44 | |
| *** mhenkel_ has joined #tripleo | 03:49 | |
| *** atheurer has quit IRC | 03:50 | |
| *** ykarel has joined #tripleo | 03:51 | |
| *** mhenkel_ has quit IRC | 03:53 | |
| *** mhenkel_ has joined #tripleo | 04:05 | |
| *** limao has quit IRC | 04:07 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 04:10 |
| *** mhenkel_ has quit IRC | 04:10 | |
| *** ratailor has joined #tripleo | 04:11 | |
| *** jerrygb has joined #tripleo | 04:15 | |
| *** fragatin_ has joined #tripleo | 04:15 | |
| *** limao has joined #tripleo | 04:16 | |
| *** tzumainn has quit IRC | 04:16 | |
| *** fragatina has quit IRC | 04:19 | |
| *** fragatin_ has quit IRC | 04:20 | |
| *** limao has quit IRC | 04:20 | |
| *** jerrygb has quit IRC | 04:32 | |
| *** jerrygb has joined #tripleo | 04:32 | |
| *** mhenkel_ has joined #tripleo | 04:33 | |
| *** fragatina has joined #tripleo | 04:35 | |
| *** mhenkel_ has quit IRC | 04:38 | |
| *** fragatina has quit IRC | 04:39 | |
| *** yamahata has joined #tripleo | 04:46 | |
| *** jerrygb has quit IRC | 04:47 | |
| *** fragatina has joined #tripleo | 04:55 | |
| *** mdnadeem has joined #tripleo | 04:57 | |
| *** fragatina has quit IRC | 04:57 | |
| *** fragatina has joined #tripleo | 04:58 | |
| *** pmannidi has quit IRC | 05:02 | |
| *** pmannidi has joined #tripleo | 05:04 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates stable/ocata: TEST - DO NOT MERGE https://review.openstack.org/458329 | 05:06 |
| *** udesale has joined #tripleo | 05:06 | |
| *** hjensas has joined #tripleo | 05:10 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 05:10 |
| *** limao has joined #tripleo | 05:15 | |
| *** fragatina has quit IRC | 05:17 | |
| *** iranzo has joined #tripleo | 05:18 | |
| *** skramaja has joined #tripleo | 05:24 | |
| *** janki has joined #tripleo | 05:26 | |
| *** prateek has joined #tripleo | 05:26 | |
| *** jbadiapa has quit IRC | 05:34 | |
| *** dsariel has joined #tripleo | 05:34 | |
| *** akuznetsov has joined #tripleo | 05:36 | |
| *** lmiccini has joined #tripleo | 05:36 | |
| *** saibarspeis has joined #tripleo | 05:38 | |
| *** akuznetsov has quit IRC | 05:38 | |
| *** yprokule has joined #tripleo | 05:39 | |
| *** stendulker has joined #tripleo | 05:42 | |
| *** masco has joined #tripleo | 05:43 | |
| *** jkilpatr has quit IRC | 05:50 | |
| *** jkilpatr has joined #tripleo | 05:51 | |
| *** mhenkel_ has joined #tripleo | 05:53 | |
| *** mhenkel_ has quit IRC | 05:58 | |
| *** anshul has joined #tripleo | 06:00 | |
| *** jaganathan has joined #tripleo | 06:04 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 06:10 |
| *** jprovazn has joined #tripleo | 06:18 | |
| *** florianf has joined #tripleo | 06:26 | |
| *** rcernin has joined #tripleo | 06:28 | |
| *** mhenkel_ has joined #tripleo | 06:29 | |
| *** pcaruana has joined #tripleo | 06:40 | |
| *** saibarspeis has quit IRC | 06:42 | |
| *** chem_gon` has joined #tripleo | 06:43 | |
| *** chem_gon` has quit IRC | 06:44 | |
| *** chem_gon` has joined #tripleo | 06:44 | |
| *** chem_gon` is now known as chem | 06:44 | |
| *** jtomasek has quit IRC | 06:45 | |
| *** chem has quit IRC | 06:45 | |
| *** chem has joined #tripleo | 06:45 | |
| *** chem_gone has quit IRC | 06:46 | |
| *** nyechiel has joined #tripleo | 06:47 | |
| *** bogdando has joined #tripleo | 06:48 | |
| *** mhenkel__ has joined #tripleo | 06:49 | |
| *** mhenkel_ has quit IRC | 06:50 | |
| *** mhenke___ has joined #tripleo | 06:50 | |
| *** jbadiapa has joined #tripleo | 06:52 | |
| *** dparkes has joined #tripleo | 06:53 | |
| *** mhenkel__ has quit IRC | 06:53 | |
| *** mhenke___ has quit IRC | 06:54 | |
| *** pmannidi has quit IRC | 06:54 | |
| *** dmacpher has quit IRC | 06:55 | |
| *** tesseract has joined #tripleo | 06:56 | |
| *** d0ugal has quit IRC | 06:56 | |
| *** pmannidi has joined #tripleo | 06:57 | |
| *** jpich has joined #tripleo | 07:00 | |
| *** leanderthal|afk is now known as leanderthal | 07:02 | |
| *** jchhatbar has joined #tripleo | 07:07 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Mount individual hostpath logs on /var/log https://review.openstack.org/442603 | 07:07 |
| *** saibarspeis has joined #tripleo | 07:09 | |
| *** janki has quit IRC | 07:09 | |
| *** jlinkes has joined #tripleo | 07:10 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 07:10 |
| *** jaosorior_away is now known as jaosorior | 07:11 | |
| jaosorior | marios, bandini hey guys, can you check this when you have some time https://review.openstack.org/#/c/457582/ ? | 07:11 |
| *** karimb has joined #tripleo | 07:12 | |
| *** mhenkel_ has joined #tripleo | 07:13 | |
| *** jtomasek has joined #tripleo | 07:14 | |
| *** shardy has joined #tripleo | 07:14 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: glance: deploy services with Keystone v3 endpoints https://review.openstack.org/442798 | 07:15 |
| *** jlinkes_ has joined #tripleo | 07:15 | |
| *** jlinkes has quit IRC | 07:16 | |
| *** aufi has joined #tripleo | 07:16 | |
| *** anshul has quit IRC | 07:18 | |
| *** anshul has joined #tripleo | 07:18 | |
| *** Vijayendra has joined #tripleo | 07:19 | |
| marios | ack jaosorior | 07:21 |
| jaosorior | marios: thanks dude | 07:23 |
| jaosorior | marios: need any reviews? starting the review round here | 07:23 |
| marios | jaosorior: thanks i think we're good there might be some cherrypicks later will ping you | 07:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 07:23 |
| jaosorior | marios: sure | 07:23 |
| jaosorior | bandini: thanks dude | 07:24 |
| bandini | jaosorior: np! | 07:24 |
| bandini | hohum I am still getting 403 on some trunk.rdoproject.org rpm downloads. am I the only one? | 07:25 |
| jaosorior | haven't tried | 07:25 |
| jaosorior | bandini: do you think it would be problematic so start usinb nbproc in haproxy? | 07:25 |
| *** cylopez has joined #tripleo | 07:25 | |
| bandini | jaosorior: well it is discouraged upstream. what would be the benefit? | 07:27 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Mount hostpath logs on /var/log https://review.openstack.org/442603 | 07:28 |
| jaosorior | bandini: well it's discouraged upstream cause it's harder to debug, but we might gain some performance from making haproxy mutli-process, as long as we keep it below the number of processors | 07:28 |
| bandini | jaosorior: I'd be very surprised if haproxy was our bottleneck. But I am not against adding a tunable option that defaults to today's config | 07:29 |
| *** gkadam has quit IRC | 07:29 | |
| *** gkadam has joined #tripleo | 07:29 | |
| *** ebarrera has joined #tripleo | 07:31 | |
| *** Vijayendra has quit IRC | 07:34 | |
| *** jchhatbar_ has joined #tripleo | 07:35 | |
| *** jchhatbar has quit IRC | 07:38 | |
| *** hjensas has quit IRC | 07:39 | |
| *** ffiore has joined #tripleo | 07:39 | |
| bogdando | folks, please review and merge https://review.openstack.org/#/q/topic:rfe1676373 | 07:39 |
| *** jchhatbar_ has quit IRC | 07:42 | |
| *** jchhatbar_ has joined #tripleo | 07:42 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** karimb has quit IRC | 07:45 | |
| jprovazn | hi, anyone knows if it's possible to deploy overcloud with network isolation with *multiple nics* in in virtual env (libvirt)? it seems that tripleo-quickstart doesn't support this option (at least I don't see it would be possible to create virt hsot with multiple nics) | 07:47 |
| *** ratailor is now known as ratailor|Lunch | 07:49 | |
| shardy | jprovazn: I've done this in the past by modifying the VM after it was created via virt-manager to add more nics | 07:51 |
| shardy | jprovazn: or you can probably hack ./roles/libvirt/setup/overcloud/templates/baremetalvm.xml.j2 in quickstart to add the nics (or even better add a j2 loop and a new variable via a patch :) | 07:51 |
| shardy | jprovazn: FWIW this was possible via the old instack-virt-setup tool so we should enable it via quickstart IMO | 07:52 |
| jprovazn | shardy: aha, thanks - so for now I will update existing VMs manullay to save 2 hours for re-deployment | 07:53 |
| *** pgadiya has joined #tripleo | 07:53 | |
| shardy | jprovazn: 2 hours? That sounds long to me, but yeah sounds good | 07:53 |
| jprovazn | shardy: slow machine | 07:54 |
| jprovazn | maybe 90 mins | 07:54 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-docs master: Fix the last known good rdo trunk delorean repo https://review.openstack.org/458376 | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #tripleo | 08:00 | |
| bogdando | jistr, shardy: hi! I have a Q, does host_prep_tasks run before docker_puppet_tasks ? | 08:05 |
| bogdando | I expect that the latter runs at the primary role only, but after the host has been prepared as well, right? | 08:06 |
| bogdando | (working on a fix for https://bugs.launchpad.net/tripleo/+bug/1684075 and https://bugs.launchpad.net/tripleo/+bug/1677652) | 08:06 |
| openstack | Launchpad bug 1684075 in tripleo "Data races for init containers (docker_puppet_tasks) vs docker_config steps" [High,Incomplete] | 08:06 |
| openstack | Launchpad bug 1677652 in tripleo "Make all db sync tasks idempotent and not racy with running services" [Medium,Triaged] | 08:06 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Run Zaqar with httpd in puppet service https://review.openstack.org/447963 | 08:07 |
| *** michapma_dsk has quit IRC | 08:07 | |
| *** karimb has joined #tripleo | 08:08 | |
| openstackgerrit | Thomas Herve proposed openstack/puppet-tripleo master: Include zaqar apache module https://review.openstack.org/447957 | 08:10 |
| *** zoli|gone is now known as zoli | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 08:10 |
| shardy | bogdando: yes https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L182 | 08:10 |
| *** hjensas has joined #tripleo | 08:10 | |
| *** hjensas has quit IRC | 08:10 | |
| *** hjensas has joined #tripleo | 08:10 | |
| shardy | bogdando: you can follow the depends_on through to see that we first do the HostPrepDeployment, then GenerateConfigDeployment, then the deployment steps | 08:11 |
| shardy | where the deployment steps deploy on baremetal, containers, then the docker_puppet_tasks | 08:11 |
| shardy | bogdando: it's a little confusing though because we run docker-puppet twice, once on all nodes to generate the config, and again to configure things only on the bootstrap node | 08:12 |
| shardy | bogdando: FWIW the ansible patches I'm working on should simplify this sequence somewhat | 08:12 |
| bogdando | shardy: it is ocnfusing, and I can't see depends-on for https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L41 | 08:12 |
| bogdando | so this is rather "no" an answer | 08:13 |
| shardy | bogdando: the thing that applies the config is the SoftwareDeployment* resources | 08:13 |
| chandankumar | arxcruz: hello | 08:13 |
| shardy | bogdando: so we can create the OS::Heat::Value and OS::Heat::SoftwareConfig resources in any order | 08:13 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Run Zaqar with httpd in puppet service https://review.openstack.org/447963 | 08:13 |
| openstackgerrit | Thomas Herve proposed openstack/puppet-tripleo master: Include zaqar apache module https://review.openstack.org/447957 | 08:14 |
| shardy | bogdando: note however that heat also creates implicit dependencies via get_resource | 08:14 |
| chandankumar | arxcruz: can we get a temprory job to test tempest-16.0.0 in tripleo-ci? | 08:14 |
| shardy | so if a Deployment references a Config, the Config is always created first | 08:14 |
| shardy | regardless of any depends_on | 08:14 |
| arxcruz | chandankumar: hey | 08:14 |
| arxcruz | chandankumar: you mean in ci.centos.org ? | 08:14 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L85 | 08:14 |
| shardy | bogdando: that shows where the tasks are made to run after the ContainersDeployment_Step | 08:15 |
| chandankumar | arxcruz: yes | 08:15 |
| *** paramite has joined #tripleo | 08:15 | |
| arxcruz | chandankumar: I guess so | 08:15 |
| shardy | which always runs after the host prep tasks due to the dependency I linked above | 08:15 |
| shardy | bogdando: basically it's easiest to focus only on the Deployment resources, then the sequence becomes clearer | 08:16 |
| *** gbarros has joined #tripleo | 08:21 | |
| bogdando | shardy: I can see that host_prep_task has a name for dependency "HostPrepDeployment", and only GenerateConfigDeployment depends on it directly. While docker_puppet_tasks has a name for deps "DockerPuppetJsonDeployment" and there is no depends-on for that item https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L63 . So how could we be sure that host_prep_tasks run before docker_puppet_tasks?.. | 08:23 |
| bogdando | perhaps we should be using direct names mappings | 08:24 |
| bogdando | its hard to decrypt 1:1 relations encoded | 08:24 |
| shardy | bogdando: host_prep_tasks runs before any of the deployment steps | 08:24 |
| shardy | it's probably helpful to write out the sequence on paper, or perhaps use the dot visualization we discussed last week | 08:25 |
| bogdando | that's multi-node, but does that applies for singletons running at the primary role only? (docker_puppet_tasks). that was my Q and I can't figure that out from the code | 08:25 |
| shardy | bogdando: yes the sequence is the same, you just run some extra things only on the primary node | 08:26 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L182 | 08:26 |
| shardy | GenerateConfigDeployment depends on HostPrepDeployment | 08:26 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L305 | 08:26 |
| shardy | ContainersDeployment_StepN depends on GenerateConfigDeployment | 08:27 |
| bogdando | shardy: yeah, perhaps we should create a template having all of those steps named as their real names, like host_prep_tasks having an item named host_prep_tasks and so on, for all steps. Then use that dots magic | 08:27 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L85 | 08:27 |
| *** florianf has quit IRC | 08:27 | |
| shardy | DockerPuppetTasksDeployment depends on ContainersDeploymentStepN | 08:27 |
| shardy | bogdando: well, host_prep_tasks has a deployment called HostPrepDeployment | 08:28 |
| bogdando | shardy: oh, thanks! I've missed that last mapping https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-steps.j2#L89 | 08:28 |
| shardy | so it'd fairly clear to me, but I'm sure we can improve it | 08:28 |
| shardy | s/it'd/it's | 08:29 |
| shardy | bogdando: I do agree this is a little hard to follow, which is why I'm refactoring all the config steps into a single ansible playbeook | 08:29 |
| *** florianf has joined #tripleo | 08:29 | |
| shardy | then we can just have a Deployment per step and the sequence will be clearer | 08:30 |
| shardy | (and probably faster) | 08:30 |
| bogdando | shardy: or perhaps then that template for building graphs should have exactly those *Deployment names, as we want only them to be presented in the graph, i.e. host_prep_tasks: HostPrepDeployment | 08:30 |
| bogdando | and the same for all possible steps | 08:30 |
| shardy | bogdando: we could, but like I said, I'm planning to remove almost all of these deployments to support minor updates | 08:31 |
| shardy | and IMO the current naming is fairly clear | 08:31 |
| bogdando | shardy: I'm ok with naming, just thinking of a way to auto generate a graph | 08:31 |
| *** lucas-afk is now known as lucasagomes | 08:31 | |
| shardy | bogdando: Heat already generates it, so you can e.g dot graph the dependencies from the heat output? | 08:32 |
| shardy | like the dotstack thing from larsks we discussed | 08:32 |
| shardy | I think there are a few tools that do that already | 08:32 |
| bogdando | it generates it for end tasks, not for the metadata describing the steps those tasks belong to | 08:32 |
| shardy | but again, if we refactor this to be just one deployment per step, this all gets much easier | 08:33 |
| shardy | bogdando: Not sure I really get it, but happy to see a patch and discuss further :) | 08:33 |
| shardy | https://review.openstack.org/#/q/status:open+project:openstack/tripleo-heat-templates+branch:master+topic:docker_ansible4 | 08:33 |
| bogdando | shardy: that I'd like to get is a graph containing HostPrepDeployment et al, not the things it contains | 08:34 |
| shardy | just be aware of that series, which will completely rework this | 08:34 |
| bogdando | I called that metadata steps, not real steps | 08:34 |
| bogdando | shardy: and that's why I want to automate the graph, cuz it's changing fast | 08:34 |
| shardy | bogdando: Heat exposes the graph it generates, e.g the relationships between all *Deployment resources, so it should be fairly easy to consume that data | 08:35 |
| bogdando | shardy: yeah, but I need to have HostPrepDeployment depicted, not the "Deployment resources" it contains | 08:35 |
| openstackgerrit | Luke Hinds proposed openstack/tripleo-heat-templates master: Implements management of `/etc/login.defs` https://review.openstack.org/457985 | 08:36 |
| shardy | bogdando: HostPrepDeployment is one resource | 08:36 |
| bogdando | and I hoped to make this working with a meta-template for those steps :) and dotstack. Ok, I'll try to make something... | 08:36 |
| shardy | it doesn't contain anything | 08:36 |
| bogdando | shardy: but there are deployment resources bund to that step, right? | 08:37 |
| bogdando | shardy: I mean heat templates have many entries for host_prep_tasks: | 08:37 |
| shardy | bogdando: well, under the hood heat creates a nested stack with one resource per server | 08:37 |
| bogdando | and I want to see not those entries, but the thing itself | 08:37 |
| shardy | but that doesn't really matter here, it just applies HostPrepConfig to all nodes | 08:37 |
| bogdando | its placement in the execution workflow | 08:37 |
| bogdando | I can't explain it better, sorry | 08:38 |
| shardy | bogdando: Ok, then you need to look at the "config" attribute of the HostPrepConfig that is associated with the HostPrepDeployment | 08:38 |
| *** gfidente has joined #tripleo | 08:38 | |
| shardy | *Deployment resources only apply a config, they make the connection between a *Config resource and a server (or in this case multiple servers) | 08:38 |
| bogdando | shardy: the plan is to create docker/services/deployment_steps.yaml meta resource | 08:39 |
| bogdando | and put all possible steps there, containing only a single entry named as its corresponding *Deployment item | 08:39 |
| shardy | bogdando: Ok, but like I said, I'm aiming to remove almost all of this | 08:39 |
| bogdando | :( | 08:40 |
| *** hewbrocca_afk is now known as hewbrocca | 08:40 | |
| shardy | so it's cool to try somethings and we can discuss, but I don't want to overlap too much | 08:40 |
| bogdando | but we have to understand things now, while writing patches :) | 08:40 |
| *** ratailor|Lunch is now known as ratailor | 08:40 | |
| shardy | bogdando: why :( ? | 08:40 |
| bogdando | it's hard to move things around not seeing real execution flow impact | 08:40 |
| shardy | bogdando: your complaint is this is hard to understand, I'm proposing we replace it with just one deployment per step | 08:40 |
| shardy | which should make things much easier to understand/follow | 08:40 |
| shardy | and also enable minor updates | 08:41 |
| bogdando | shardy: let me show be example patch | 08:41 |
| shardy | Hopefully I can make progress on my patch series this week then we can discuss further | 08:41 |
| *** suuuper has joined #tripleo | 08:41 | |
| *** pgadiya has quit IRC | 08:42 | |
| *** jchhatbar has joined #tripleo | 08:44 | |
| bogdando | shardy: an example, I moved this db sync task from docker_config, to docker_puppet_tasks https://github.com/bogdando/tripleo-heat-templates/commit/c578d042bf63cc4709c3f94596be46d96c84d7d0#diff-3fbcdcafa53eccd6e1c3dc23a29771acR100 Now I need to know, would host_prep_tasks ensure persistent logs dir by that new point of the graph? | 08:44 |
| bogdando | before the move, I knew it would | 08:45 |
| bogdando | that's why the graph for steps is required, not for deployment resoutces | 08:45 |
| bogdando | so before the change it was: host_prep_steps(create /var/log/aodh) -> docker_config(db_sync) | 08:46 |
| *** jchhatbar_ has quit IRC | 08:46 | |
| bogdando | shardy: and after the change it becomes ???host_prep_steps(create /var/log/aodh) -> docker_puppet_tasks(db_dync for primary role only) -> docker_config(no more db sync for other nodes) | 08:47 |
| shardy | bogdando: No, we always run deployment step N (puppet), deployment step N (containers), docker_puppet_tasks | 08:47 |
| bogdando | so my complaint is not related to hard/easy understanding, rather to automated understanding of the graph for the current code | 08:48 |
| shardy | host_prep_tasks happens before any steps happen | 08:48 |
| shardy | so it will always happen first | 08:48 |
| bogdando | shardy: yeah, I know that now, after I took a half of a hour of yoyr time :) I hoped to make things automatically illustrate the flow... | 08:48 |
| bogdando | either the things are about to change or stay still | 08:49 |
| shardy | bogdando: alright, I'm going to make some coffee and get back to work then ;) | 08:49 |
| bogdando | shardy: thanks for help! | 08:49 |
| shardy | np | 08:50 |
| bogdando | shardy: I'll postpone the patch then, until your changes done or I figured out some graph automation for steps... I can be asking for ever otherwise, like do we have now the steps interleaved or not: "deployment step 1 (puppet), deployment step 1 (containers), docker_puppet_tasks step 1" vs "deployment step 1-6 (puppet), deployment step 1-6 (containers), docker_puppet_tasks 1-6" etc... :) | 08:59 |
| shardy | bogdando: sure, yes they are interleaved, it's fairly clear if you spend a while looking at the template | 09:00 |
| bogdando | shardy: those YAMLs with jinja2 sauce are write only :/ | 09:01 |
| bogdando | we need a robot to read :) | 09:01 |
| bogdando | or an author perhaps! :) | 09:01 |
| *** karimb has quit IRC | 09:02 | |
| shardy | bogdando: it gets easier over time | 09:02 |
| *** karimb has joined #tripleo | 09:06 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Show all roles in inventory https://review.openstack.org/450233 | 09:06 |
| *** dbecker has quit IRC | 09:07 | |
| *** dbecker has joined #tripleo | 09:08 | |
| *** salmankhan has joined #tripleo | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 09:10 |
| *** mcornea has joined #tripleo | 09:11 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 09:11 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add host list by service to inventory https://review.openstack.org/457972 | 09:11 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Retreive AdminPassword from heat instead of mistral https://review.openstack.org/458408 | 09:13 |
| *** milan has joined #tripleo | 09:17 | |
| *** amoralej|off is now known as amoralej | 09:21 | |
| *** limao has quit IRC | 09:24 | |
| *** gkadam is now known as gkadam-afk | 09:26 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Haproxy: When using TLS everywhere, use verifyhost for the balancermembers https://review.openstack.org/457582 | 09:26 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Pluggable server type per Role https://review.openstack.org/456739 | 09:27 |
| *** limao has joined #tripleo | 09:28 | |
| *** yamahata has quit IRC | 09:29 | |
| *** zoli is now known as zoli|lunch | 09:31 | |
| *** milan has quit IRC | 09:32 | |
| jaosorior | owalsh: Hey dude, I'm seeing errors like this in a failed overcloud deployment (timed out deplying the compute) any idea if this migth be related http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-fakeha-caserver/e69db84/logs/undercloud/var/log/nova/nova-scheduler.txt.gz#_2017-04-20_06_50_49_980 ? | 09:32 |
| *** derekh has joined #tripleo | 09:33 | |
| *** ckyriakidou has joined #tripleo | 09:34 | |
| openstackgerrit | Merged openstack/tripleo-common master: Rename contrib to container-images for packaging https://review.openstack.org/453428 | 09:34 |
| openstackgerrit | Merged openstack/tripleo-ui master: Add support for Indonesian language https://review.openstack.org/455636 | 09:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: N->O upgrade, fix wrong parameters to nova placement. https://review.openstack.org/457965 | 09:34 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure libvirt when not needed https://review.openstack.org/458414 | 09:37 |
| jistr | bogdando, mandre: i hope i'm focusing on the right thing now :D i'm yet about to test it https://review.openstack.org/#/c/458414 | 09:38 |
| jistr | though my yesterday's upgrade passed even without this, which is very strange... | 09:40 |
| *** karimb has quit IRC | 09:41 | |
| * jistr will test it via CI too | 09:41 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/ocata: N->O upgrade, fix wrong parameters to nova placement. https://review.openstack.org/458416 | 09:41 |
| *** tosky has joined #tripleo | 09:45 | |
| jaosorior | shardy: any idea why the compute creation might have timed out but not the controller http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-fakeha-caserver/e69db84/console.html#_2017-04-20_08_16_46_725815 ? | 09:50 |
| *** karimb has joined #tripleo | 09:56 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: move mistral base action dependency to mistral_lib https://review.openstack.org/454632 | 09:59 |
| jaosorior | shardy: it seems to be timing out on the NetworkDeployment, any idea how I can debug that? | 10:01 |
| jprovazn | any ironic expert around? ironic-inspector is complaining about "[Errno 32] Broken pipe" - http://paste.openstack.org/show/607303/ | 10:03 |
| *** b00tcat has joined #tripleo | 10:05 | |
| hewbrocca | lucasagomes: ^^^ | 10:06 |
| jprovazn | it seems that a client (conductor?) closes connection sooner than inspector replies? | 10:07 |
| *** karimb has quit IRC | 10:08 | |
| *** karimb has joined #tripleo | 10:09 | |
| shardy | jaosorior: probably need to see /var/log/messages on the compute node to debug it, but it seems we failed to get those logs :( | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 10:10 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI https://review.openstack.org/450784 | 10:11 |
| lucasagomes | jprovazn, hmm seems like two errors there, conductor not being able to change the node's power state and this inspector api's broken pipe... Is there any other errors in ir-conductor log ? | 10:12 |
| jaosorior | shardy: I thought we had a timeout for the overcloud deploy | 10:14 |
| *** abishop has joined #tripleo | 10:16 | |
| openstackgerrit | Radomir Dopieralski proposed openstack/tripleo-heat-templates master: WIP: Containerize Horizon https://review.openstack.org/450303 | 10:17 |
| jprovazn | lucasagomes: I don't see anything else (though I might miss something of course, it's not the smallest one) | 10:18 |
| *** limao has quit IRC | 10:18 | |
| lucasagomes | jprovazn, heh yeah, I was just wondering what was the failure for the ipmitool to not work | 10:19 |
| shardy | http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-fakeha-caserver/e69db84/console.html#_2017-04-20_06_05_28_169055 | 10:19 |
| jtomasek | shardy, dtantsur|afk: in canse when I have 8 nodes and 3 of them are tagged to different roles, how is it decided which node is going to be used for deployment? | 10:19 |
| shardy | jaosorior: we do, it's set to 80mins there | 10:19 |
| lucasagomes | jprovazn, if you try to change the power state of the node via ironic's api it works ? ironic node-set-power-state <uuid> on/off | 10:19 |
| shardy | jaosorior: there was a bug saying it wasn't working, but I've been unable to reproduce | 10:19 |
| *** ukalifon has joined #tripleo | 10:20 | |
| jprovazn | lucasagomes: before this try, I manually added extra network interfaces to the VMs - which may cause issues, but I would expect to find a reason about it in logs if it's the case | 10:20 |
| openstackgerrit | Thomas Herve proposed openstack/puppet-tripleo master: Include zaqar apache module https://review.openstack.org/447957 | 10:20 |
| jprovazn | trying, sec | 10:20 |
| jaosorior | shardy: so this works in my deployment :/ I'm trying to rebuild my environment to see if I can reprodce it. | 10:20 |
| shardy | jtomasek: it depends on the flavor "baremetal" will pick any node regardless of tags, or you can select one where the capabilities match the tag to control placement | 10:20 |
| jaosorior | shardy: this started failing in CI yesterday | 10:20 |
| jtomasek | shardy: yes, but lets say I want to deploy 3 of 8 nodes 1 controller, 1 compute, 1 ceph. so I tag 3 nodes appropriately and set the flavor at roles correctly | 10:21 |
| jprovazn | lucasagomes: no - manula "on" doesn't work | 10:21 |
| lucasagomes | jprovazn, hmm right yeah, the only thing I can think off is the wrong interface being pick for pxe boot... but it seems unrelated to the problems in the logs | 10:22 |
| jtomasek | shardy: ukalifon is hitting a problem when he does just that, untagged nodes are used instead of prefering the tagged ones | 10:22 |
| lucasagomes | jprovazn, right, does the ir-cond log says why ? | 10:22 |
| shardy | jtomasek: Ok, that sounds wrong then | 10:22 |
| jtomasek | shardy: because iiuc, in that case nova or whatever decides about picking nodes, does not prefer the tagged ones | 10:23 |
| jprovazn | IPMI Error while attempting "ipmitool -I lanplus -H 127.0.0.1 -L ADMINISTRATOR -p 6230 -U admin -R 3 -N 5 -f /tmp/tmpU3Czxm power on" for node e3307de7-945f-4227-9964-68fc811913ce. Error: Unexpected error while running command. | 10:23 |
| jprovazn | lucasagomes: ^ - http://paste.openstack.org/show/607310/ | 10:23 |
| shardy | jtomasek: we have a filter which does an exact match based on the capabilites, so this should work and has been previously tested | 10:23 |
| jtomasek | shardy: iiuc, with that setup 'controller' node can get selected from 6 nodes in total (1 tagged and remaining untagged ones) | 10:24 |
| lucasagomes | jprovazn, "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n'" | 10:24 |
| lucasagomes | jprovazn, seems to be BMC related, can you check if it's working ? You can try to reset it as well | 10:24 |
| lucasagomes | BMC's sucks in general | 10:24 |
| jtomasek | shardy: ok, so it should prefer tagged nodes first, right? | 10:24 |
| lucasagomes | (or is it a virt env ? in that case I haven't seem similar errors in that kinda of env0 | 10:25 |
| lucasagomes | )* | 10:25 |
| jprovazn | lucasagomes: virtualenv | 10:25 |
| shardy | jtomasek: AFAIK yes | 10:25 |
| shardy | jtomasek: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/filters/capabilities_filter.py#L35 | 10:25 |
| lucasagomes | jprovazn, is it running with virtualbmc ? | 10:25 |
| lucasagomes | jprovazn, can you check "vbmc list" | 10:25 |
| lucasagomes | see if the services are running | 10:25 |
| jprovazn | lucasagomes: vbmc list is empty | 10:25 |
| lucasagomes | jprovazn, try with sudo | 10:26 |
| jprovazn | +-------------+---------+---------+------+ | 10:26 |
| jprovazn | | Domain name | Status | Address | Port | | 10:26 |
| jprovazn | +-------------+---------+---------+------+ | 10:26 |
| jprovazn | | ceph_0 | running | :: | 6232 | | 10:26 |
| jprovazn | | compute_0 | running | :: | 6231 | | 10:26 |
| jprovazn | | control_0 | running | :: | 6230 | | 10:26 |
| jprovazn | +-------------+---------+---------+------+ | 10:26 |
| jprovazn | lucasagomes: ^ | 10:26 |
| lucasagomes | yeah seems it's all running, hmm strange | 10:26 |
| lucasagomes | jprovazn, it was working before ? No updates ? | 10:27 |
| jtomasek | shardy: ok thanks | 10:27 |
| lucasagomes | maybe after you added the nics for some reason you can't turn the nodes on ? | 10:27 |
| jaosorior | shardy: actually. I think the timeout won't help. it hits the overcloud deployment's timeout, not actual job's timeout. So it finishes cleanly and everything. I don't understand though, why the compute logs are empty | 10:27 |
| lucasagomes | jprovazn, can you just try to do a "virsh start <domain>" | 10:27 |
| shardy | jtomasek: this may be a bug, where we respect exact placement for capabilities node:foo, but not profile:foo | 10:27 |
| lucasagomes | jprovazn, see if doesn't complain ? | 10:27 |
| jprovazn | lucasagomes: I'm sure that first OC deployment passed, since then I'm trying to deploy with multiple nics -> I had to add more nics to VMs manually | 10:27 |
| shardy | e.g where we use SchedulerHints vs flavors | 10:27 |
| lucasagomes | jprovazn, right, try to manually turn those vms on/off with virsh because VBMC just uses the libvirt library to do it anyway | 10:28 |
| jtomasek | shardy: aha, I was confused on what capabilities:node means, never saw that, is that docummented somewhere? | 10:28 |
| shardy | jtomasek: it'd be interesting to see what happens if all the remaining nodes are tagged e.g notcontroller | 10:28 |
| shardy | https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/node_placement.html | 10:28 |
| shardy | jtomasek: ^^ | 10:28 |
| jtomasek | ukalifon: can you test that?^^ | 10:28 |
| jprovazn | lucasagomes: I'll try it, thanks, I have to relocate now, will be back in 20 mins | 10:28 |
| lucasagomes | jprovazn, ok | 10:28 |
| jtomasek | shardy: thanks! | 10:28 |
| *** jprovazn has quit IRC | 10:28 | |
| ukalifon | jtomasek: shardy: I'll check that | 10:28 |
| jtomasek | ukalifon: UI enables you to add custom tag on all nodes, try to tag other 5 nodes to 'randomtag' and redeploy | 10:29 |
| shardy | Yeah, if that works then we have a bug but at least it's an easy workaround | 10:29 |
| *** fzdarsky_ is now known as fzdarsky|lunch | 10:31 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458439 | 10:34 |
| chem | marios: ^ | 10:34 |
| chem | mcornea: ^ | 10:34 |
| marios | thanks checking | 10:36 |
| *** apetrich has quit IRC | 10:36 | |
| arxcruz | adarazs: trown|outtypewww hey, can it be merged https://review.openstack.org/#/c/455219/ | 10:36 |
| *** jkilpatr has quit IRC | 10:41 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458439 | 10:41 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 10:44 |
| *** fragatina has joined #tripleo | 10:47 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458439 | 10:48 |
| chem | marios: thanks for the update in the description ... M,N,O ... silly me. | 10:50 |
| marios | :/ np chem was no huge burden ! | 10:51 |
| *** zoli|lunch is now known as zoli | 10:51 | |
| matbu | chem: marios did you already see this issue: https://paste.fedoraproject.org/paste/cVHnU5qgHrvOQ8tVVLkyuF5M1UNdIGYhyRLivL9gydE= | 10:52 |
| matbu | chem: marios i don't understand it tbh, every thing looks good on the env | 10:52 |
| marios | matbu: i have not | 10:52 |
| matbu | the systemd LimitNOFILE exist in the rb code | 10:53 |
| matbu | chem: you are a puppet expert ^ | 10:53 |
| chem | matbu: looking ... what is the context here ? | 10:54 |
| matbu | chem: puppet is failing during the controller upgrade | 10:54 |
| matbu | chem: on this error (limitNOFILE) on tripleo / mysql.pp | 10:54 |
| chem | matbu: ack | 10:54 |
| matbu | it's upstream | 10:55 |
| matbu | i didn't check if the downstream package have this setting | 10:55 |
| *** karimb has quit IRC | 10:57 | |
| chem | matbu: do you have puppet-systemd installed ? | 10:58 |
| chem | matbu: but it shouldn't be the issue, still looking | 11:00 |
| matbu | looking | 11:01 |
| *** jprovazn has joined #tripleo | 11:01 | |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-heat-templates master: Run Zaqar with httpd in puppet service https://review.openstack.org/447963 | 11:01 |
| matbu | chem: puppet-systemd-0.4.0-0.20170210184545.a032136.el7.centos.noarch | 11:01 |
| matbu | yep | 11:01 |
| *** jkilpatr has joined #tripleo | 11:02 | |
| *** arxcruz has quit IRC | 11:03 | |
| *** fragatina has quit IRC | 11:03 | |
| *** fragatina has joined #tripleo | 11:03 | |
| *** karimb has joined #tripleo | 11:03 | |
| *** arxcruz has joined #tripleo | 11:05 | |
| *** bkopilov has quit IRC | 11:07 | |
| marios | chem: check https://review.openstack.org/#/c/458439/3/puppet/services/nova-api.yaml ? | 11:08 |
| *** pkovar has joined #tripleo | 11:08 | |
| fultonj | shardy: you had reviewed https://review.openstack.org/#/c/423304/15 in the past. would you mind looking at it one last time? | 11:09 |
| *** jaosorior has quit IRC | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,Triaged] | 11:10 |
| *** milan has joined #tripleo | 11:12 | |
| *** karimb has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Fix dlrn link to point to trunk.rdoproject.org https://review.openstack.org/458065 | 11:16 |
| shardy | fultonj: done! | 11:17 |
| fultonj | shardy thank you! | 11:17 |
| EmilienM | hello | 11:18 |
| *** ratailor has quit IRC | 11:19 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Touch /etc/httpd/conf.d/ssl.conf https://review.openstack.org/457688 | 11:20 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458439 | 11:20 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: TLS-everywhere: Enable for TLS libvirt live migration https://review.openstack.org/450726 | 11:20 |
| *** jaosorior has joined #tripleo | 11:20 | |
| *** stendulker has quit IRC | 11:22 | |
| marios | jaosorior && anyone else : can you check https://review.openstack.org/#/c/458416/1 when you get a chance please is a cherrypick | 11:23 |
| jaosorior | marios: will approve once it passes the gate | 11:24 |
| marios | thank you jaosorior | 11:24 |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** lblanchard has joined #tripleo | 11:26 | |
| *** lblanchard has quit IRC | 11:26 | |
| EmilienM | cschwede: fyi, I'm now working on https://bugs.launchpad.net/tripleo/+bug/1684272 | 11:28 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:28 |
| *** dparkes has quit IRC | 11:29 | |
| *** dparkes has joined #tripleo | 11:29 | |
| chem | matbu: he, sorry got distracted, could you share a tmate on your env ? | 11:32 |
| *** openstackgerrit has quit IRC | 11:32 | |
| chem | matbu: cannot reproduce it easily | 11:32 |
| *** openstackgerrit has joined #tripleo | 11:35 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: [DO_NOT_MERGE] Deploy Ceph using ceph-ansible via external workflow https://review.openstack.org/458058 | 11:35 |
| openstackgerrit | Merged openstack/tripleo-validations master: Create the neutron-sanity-check validations https://review.openstack.org/381118 | 11:39 |
| *** fzdarsky|lunch is now known as fzdarsky | 11:40 | |
| jaosorior | EmilienM: is that only affecting the update gates? or is it a performance regression for all jobs? | 11:42 |
| *** shardy is now known as shardy_lunch | 11:44 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Upgrade to keystone v3 https://review.openstack.org/451431 | 11:44 |
| *** arxcruz has quit IRC | 11:45 | |
| jtomasek | honza: I wanted to do a normal review, but I ended up coding updates as I tested things, so I sent it as a patchset https://review.openstack.org/#/c/451431/ | 11:46 |
| jtomasek | honza: could you check it please?^ | 11:46 |
| honza | jtomasek: sure thing | 11:46 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart-extras master: Upgrade to containerized overcloud https://review.openstack.org/448576 | 11:46 |
| jtomasek | honza: thanks! | 11:46 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Add ovb-fakeha-caserver (TLS everywhere) job to cistatus https://review.openstack.org/458468 | 11:47 |
| *** jzimnowoda has quit IRC | 11:48 | |
| *** dsariel has quit IRC | 11:49 | |
| *** jerrygb has joined #tripleo | 11:49 | |
| *** arxcruz has joined #tripleo | 11:50 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add language switcher to login page https://review.openstack.org/441435 | 11:51 |
| ukalifon | jtomasek: shardy_lunch: All the nodes tagged with "randomtag" were provisioned. The nodes tagged with control, compute and ceph-storage remained undeplyed. | 11:52 |
| ukalifon | jtomasek: shardy_lunch: There is still some randomness here, because I noticed this bug a few days ago already and when I tried to recreate it - it didn't happen any more. But now it happens every time. | 11:53 |
| ukalifon | jtomasek: shardy_lunch: Jiri, could it be something in the environment from a previous deployment ? | 11:53 |
| ukalifon | (although I think I didn't ruin the default plan_) | 11:53 |
| honza | jtomasek: How broken was the keystone code when you tried to run it? It seems like your changes aren't trivial, and yet I didn't see anything broken when I uploaded the patch... | 11:54 |
| *** jerrygb has quit IRC | 11:54 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI https://review.openstack.org/450784 | 11:55 |
| matbu | chem: yep , sorry i missed your ping | 11:55 |
| matbu | chem: pinging you on internal chan | 11:56 |
| jtomasek | honza: if you don't use endpoints from keystone response (you probably override the endpoints in tripleo_ui_config_file), then you don't see the error | 11:56 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 11:56 |
| jtomasek | honza: I've fixed that selector and removed having unnnecessary token>token nesting in state | 11:56 |
| jtomasek | honza: thats all | 11:56 |
| openstackgerrit | Merged openstack/tripleo-validations master: Add lookup plugin for tripleo heat templates https://review.openstack.org/441164 | 11:56 |
| honza | jtomasek: yep, looks good to me, just surprised that i didn't catch it :( | 11:56 |
| honza | jtomasek: reviews ftw :) | 11:57 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing increased overcloud deploy timeout tls everywhere https://review.openstack.org/458473 | 11:57 |
| jtomasek | honza: this line let serviceUrl = appConfig[serviceName] || getFromServiceCatalog(serviceName, urlType); | 11:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 11:57 |
| honza | jtomasek: can you paste your config.js somewhere? Is it mostly empty then? | 11:58 |
| jtomasek | honza: yours is probably getting url from appConfig, so getFromServiceCatalog is not called for you | 11:58 |
| jtomasek | honza: yes, no endpoints there at all | 11:58 |
| jtomasek | honza: but it may not work on your setup | 11:59 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 11:59 |
| honza | jtomasek: my undercloud is busted at the moment, but i'll check it out later, thanks | 12:00 |
| honza | interesting | 12:00 |
| matbu | chem: creds sent in pv | 12:01 |
| chem | matbu: ack, thanks | 12:01 |
| *** jpena|lunch is now known as jpena | 12:01 | |
| *** zoli is now known as zoli|mtg | 12:03 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 12:03 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Move ceilometer upgrade re-run out of collector https://review.openstack.org/458036 | 12:03 |
| *** pradk has joined #tripleo | 12:04 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:08 |
| *** karimb has joined #tripleo | 12:09 | |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates master: Restrict nova migration ssh tunnel https://review.openstack.org/458082 | 12:09 |
| chem | matbu: so I think I got the errorr | 12:09 |
| *** jerrygb has joined #tripleo | 12:09 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| *** dmacpher has joined #tripleo | 12:10 | |
| *** apetrich has joined #tripleo | 12:10 | |
| matbu | chem: \o/ | 12:10 |
| matbu | chem: i know you were the guy, what is the error ? | 12:10 |
| thrash | EmilienM: https://review.openstack.org/#/c/455425/ | 12:11 |
| thrash | looks good | 12:11 |
| thrash | EmilienM: still don't totally understand how it was broken. I was never able to break it on my end. | 12:12 |
| EmilienM | thrash: all approved | 12:12 |
| thrash | EmilienM: awesome! | 12:12 |
| EmilienM | thrash: next step, enable wsgi in tripleo for mistral | 12:12 |
| thrash | EmilienM: was just about to say that. :) | 12:12 |
| chem | matbu: if you go to the controller and sudo diff test-ko.pp with test-ok.pp | 12:13 |
| chem | matbu: and the problem is that https://github.com/openstack/puppet-tripleo/commit/c9acf8a687ea64686c1ecceeff45add014752121 | 12:14 |
| *** jerrygb has quit IRC | 12:14 | |
| chem | matbu: we got the left version packaged | 12:14 |
| chem | matbu: we need the right verison packaged asap and the current one shouldn't hit the puddle | 12:15 |
| chem | matbu: which bug tracker should I hit ? | 12:16 |
| matbu | chem: erf ok, i tried to put the int value with quote, not the variable in quote | 12:16 |
| matbu | chem: i didn't open a LP | 12:16 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:16 |
| matbu | chem: i will open a LP and take care of it | 12:18 |
| matbu | chem: thank you for the help | 12:18 |
| chem | matbu: so puppet-tripleo-6.3.0-0.20170308072655.6d204f4.el7.centos.noarch is broken | 12:19 |
| chem | matbu: maybe a new version has come up since then, I mean 03/08 seems kinda old ? | 12:20 |
| *** fragatina has quit IRC | 12:22 | |
| *** fragatina has joined #tripleo | 12:23 | |
| matbu | chem: yep , really old... im checking the upstream repo | 12:24 |
| openstackgerrit | Brad P. Crochet proposed openstack/instack-undercloud master: Enable mistral mod_wsgi in undercloud https://review.openstack.org/458482 | 12:24 |
| *** shardy_lunch is now known as shardy | 12:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:27 |
| *** trozet has quit IRC | 12:28 | |
| *** dsariel has joined #tripleo | 12:32 | |
| *** abishop is now known as abishop|bbl | 12:33 | |
| *** jlinkes_ is now known as jlinkes | 12:34 | |
| *** dsariel has quit IRC | 12:35 | |
| *** psahoo has quit IRC | 12:35 | |
| *** dsariel has joined #tripleo | 12:37 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize clustercheck galera monitor for HA deployments https://review.openstack.org/458489 | 12:38 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: WIP - Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:38 |
| *** rlandy has joined #tripleo | 12:39 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud https://review.openstack.org/437544 | 12:39 |
| *** thrash is now known as thrash|biab | 12:39 | |
| *** eck`gone is now known as eck` | 12:40 | |
| *** nyechiel has quit IRC | 12:40 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Automatically enable all available languages https://review.openstack.org/456646 | 12:42 |
| bogdando | larsks, shardy, jistr: I think I've found that I wanted ! https://github.com/lxsli/heat-viz/pull/1/files?short_path=04c6e90#diff-04c6e90faac2675aa89e2176d2eec7d8 | 12:43 |
| bogdando | I'd appreciate if we could fix nodes omitted from the graph... | 12:44 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Containerize clustercheck galera monitor for HA deployments https://review.openstack.org/457800 | 12:45 |
| *** liverpooler has joined #tripleo | 12:46 | |
| *** masco has quit IRC | 12:46 | |
| *** b00tcat has quit IRC | 12:47 | |
| *** b00tcat has joined #tripleo | 12:47 | |
| *** bkopilov has joined #tripleo | 12:50 | |
| *** gkadam-afk has quit IRC | 12:51 | |
| weshay | rasca, https://review.openstack.org/#/c/391209/ | 12:51 |
| *** mhenkel_ has quit IRC | 12:53 | |
| *** snecklifter has joined #tripleo | 12:55 | |
| *** jerrygb has joined #tripleo | 12:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:56 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 12:57 |
| *** mburned is now known as mburned_out | 12:58 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 12:59 |
| *** trown|outtypewww is now known as trown | 13:08 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 13:10 |
| *** tzumainn has joined #tripleo | 13:11 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Move OVB settings to tripleo-quickstart-extras https://review.openstack.org/448805 | 13:11 |
| *** hewbrocca is now known as hewbrocca_afk | 13:14 | |
| *** tobias_fiberdata has joined #tripleo | 13:14 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Upgrade to keystone v3 https://review.openstack.org/451431 | 13:15 |
| *** Goneri has joined #tripleo | 13:15 | |
| *** abishop|bbl is now known as abishop | 13:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 13:23 |
| *** hewbrocca_afk is now known as hewbrocca | 13:24 | |
| zzzeek | is it normal that the tripleo CI jobs just fail like 50% of the time? | 13:24 |
| EmilienM | zzzeek: which jobs? | 13:25 |
| zzzeek | EmilienM: jobs like gate-tripleo-ci-centos-7-nonha-multinode-oooq | 13:25 |
| zzzeek | this change does nothing and like undercloud installs are sporadically failing per the logs https://review.openstack.org/#/c/457805/ | 13:25 |
| EmilienM | zzzeek: not normal | 13:26 |
| EmilienM | I'll look asap | 13:26 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: Rename misleading undercloud/overcloud hieradata overrides https://review.openstack.org/458501 | 13:26 |
| weshay | zzzeek, no 50% is not normal, over the course of the week.. that job has passed 80.6% of the time | 13:26 |
| zzzeek | EmilienM: my change there is add a new config option to haproxy.pp. if it was an invalid option, all the jobs would fail every time | 13:26 |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo master: Clustercheck, monitor service for galera containers https://review.openstack.org/457797 | 13:27 |
| weshay | w/ 18.2% of the time it's failing on previously identified bugs | 13:27 |
| weshay | s/bugs/errors | 13:27 |
| zzzeek | im like doing oooq w/ devmode to reproduce and over there, I keep getting undercloud ssh timeouts :( (that's my env likely) | 13:27 |
| * weshay looking at http://status-tripleoci.rhcloud.com/ and then clicking on the job name | 13:28 | |
| *** dsariel has quit IRC | 13:29 | |
| *** anshul has quit IRC | 13:29 | |
| EmilienM | I know why | 13:31 |
| EmilienM | http://logs.openstack.org/05/457805/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/62a4f0d/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz | 13:31 |
| EmilienM | 2017-04-19 23:03:08 | 2017-04-19 23:03:08,584 INFO: [1;31mError: /Stage[main]/Tripleo::Profile::Base::Docker/Augeas[docker-daemon.json]: Could not evaluate: Saving failed, see debug[0m | 13:31 |
| EmilienM | dprince fixed it last night | 13:31 |
| EmilienM | zzzeek: just do recheck, the problem was affecting all CI and is fixed now. | 13:31 |
| zzzeek | EmilienM: ok thanks! | 13:32 |
| weshay | here's the link http://logs.openstack.org/05/457805/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/94e9f40/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-04-19_22_53_15 | 13:32 |
| EmilienM | zzzeek: problem was fixed by https://github.com/openstack/puppet-tripleo/commit/be27b5cb0429e0370ddb4a83a8b710bc81ec1fd2 | 13:33 |
| *** limao has joined #tripleo | 13:33 | |
| zzzeek | EmilienM: ah ok, and sporadic b.c. file would suddenly exist due to previos job or something? | 13:34 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo master: Enable mistral to run under mod_wsgi https://review.openstack.org/458503 | 13:34 |
| *** ykarel has quit IRC | 13:34 | |
| *** jmelvin has joined #tripleo | 13:34 | |
| EmilienM | zzzeek: it's something in docker packaging, that we didn't control | 13:35 |
| openstackgerrit | Merged openstack/tripleo-common master: add caching the GetParametersAction https://review.openstack.org/444220 | 13:35 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates master: Enable mistral to run under mod_wsgi https://review.openstack.org/458504 | 13:35 |
| *** kjw3 has quit IRC | 13:38 | |
| *** limao has quit IRC | 13:38 | |
| *** hjensas has quit IRC | 13:38 | |
| *** arxcruz has quit IRC | 13:38 | |
| *** arxcruz has joined #tripleo | 13:39 | |
| *** limao has joined #tripleo | 13:39 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: Rename misleading undercloud/overcloud hieradata overrides https://review.openstack.org/458501 | 13:41 |
| bandini | mandre: in a puppet manifest I need to add an if {} stub that executes code only when the service itself is running via docker. are there any hiera keys I can check for that? | 13:44 |
| bandini | jistr: ^ | 13:46 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 13:47 |
| jistr | bandini: i don't think there are at the moment. You can however put in custom hiera keys or overrides in any of the manifest, e.g. see how we override apache::default_vhost here | 13:48 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/b5675f3b7f68e0e2b8d65c8b864897477c06fd58/docker/services/nova-api.yaml#L51-L54 | 13:48 |
| jistr | bandini: we could perhaps come up with something better if this ^ doesn't seem like a good solution for the use case you have in mind | 13:49 |
| bandini | jistr: ack, I'll inject something custom for the time being | 13:49 |
| bandini | jistr: it would probably be good to have some global list of dockerized services in hiera. I expect this to be useful in general | 13:50 |
| *** trozet has joined #tripleo | 13:50 | |
| bandini | jistr: thanks ;) | 13:50 |
| jistr | bandini: sure thing :) | 13:51 |
| *** chlong has joined #tripleo | 13:51 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-specs master: Deriving TripleO Parameters https://review.openstack.org/423304 | 13:53 |
| fultonj | skramaja: abishop shardy ^ is jut for spelling fixes (lost update I promise). if you don't mind scoring one last time. thanks. | 13:53 |
| fultonj | (just* for spelling fixes) | 13:54 |
| *** zoli|mtg is now known as zoli | 13:55 | |
| *** jpich has quit IRC | 13:57 | |
| abishop | fultonj: Added my +1 | 13:57 |
| fultonj | abishop: thanks | 13:57 |
| *** atheurer has joined #tripleo | 13:58 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 14:02 |
| *** thrash|biab is now known as thrash | 14:02 | |
| *** colonwq has joined #tripleo | 14:03 | |
| *** jchhatbar has quit IRC | 14:04 | |
| thrash | EmilienM: http://logs.openstack.org/82/458482/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/e1cbaec/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-04-20_12_41_47 | 14:06 |
| *** skramaja has quit IRC | 14:06 | |
| thrash | EmilienM: not sure how to solve that one | 14:06 |
| *** jkilpatr has quit IRC | 14:07 | |
| fultonj | thanks shardy | 14:07 |
| trozet | flaper87: hi | 14:07 |
| flaper87 | trozet: hey :) | 14:07 |
| *** b00tcat has quit IRC | 14:07 | |
| EmilienM | thrash: Class['::ironic::inspector'] ~> Exec['mistral-db-populate'] | 14:07 |
| trozet | flaper87: hey wanted to let you know the ODL congtainer patch was just accepted into kolla | 14:07 |
| flaper87 | fultonj: hey there, quick question. This is the repo y'all are working on for the ceph stuff, right? https://github.com/fultonj/tripleo-ceph-ansible | 14:07 |
| trozet | flaper87: so will be working on getting ODL conatiner into OOO soon | 14:08 |
| *** b00tcat has joined #tripleo | 14:08 | |
| fultonj | flaper87: yes, for now | 14:08 |
| thrash | EmilienM: that's already there. | 14:08 |
| EmilienM | thrash: problem is that ironic inspector runs in httpd but mistral-db-populate needs to be run after ironic inspector | 14:08 |
| flaper87 | fultonj: ok, you might get some patches from me and questions :D | 14:08 |
| thrash | EmilienM: you mean that's the problem? | 14:08 |
| EmilienM | thrash: yes sir | 14:08 |
| fultonj | flaper87: awesome, thanks | 14:08 |
| trozet | flaper87: i think when i talked to Slower_ at PTG, he mentioned some repo where you were holding the built containers. Is the first step to upload it there after I build it? Or do you have a build system now? | 14:08 |
| flaper87 | trozet: fantastic! Thanks for the heads up | 14:08 |
| EmilienM | mistral-db-populate needs to be run after httpd then | 14:08 |
| EmilienM | we need to check if mistral-db-populate can be run after mistral api (httpd) starts | 14:09 |
| EmilienM | if yes, just remove this line | 14:09 |
| flaper87 | trozet: we need to upload it manually for now | 14:09 |
| thrash | EmilienM: Don't think that can happen. Let me check though | 14:09 |
| flaper87 | flaper87: how are you generating this? https://github.com/fultonj/tripleo-ceph-ansible/blob/master/mistral-ceph-ansible.yaml | 14:09 |
| trozet | flaper87: sorry if this is already been answered in the container guide, but can you link me instructions on that or the repo itself if the instructions dont exist? | 14:09 |
| fultonj | flaper87: i'm just improving the POC there at the moment i'm looking to merge it into tripleo-common once the heat resource is done and the ceph-ansible package in the undercloud | 14:09 |
| *** jpich has joined #tripleo | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| thrash | EmilienM: populate needs to happen prior to api starting | 14:10 |
| flaper87 | fultonj: https://github.com/fultonj/tripleo-ceph-ansible/blob/master/heat/workflow_execution.py this one? | 14:10 |
| *** dsariel has joined #tripleo | 14:10 | |
| flaper87 | fultonj: that's a heat-hook, right ? | 14:10 |
| thrash | EmilienM: or we need to bounce it. | 14:10 |
| *** cwolferh has joined #tripleo | 14:10 | |
| *** mburned_out is now known as mburned | 14:10 | |
| thrash | EmilienM: But I'm pretty sure it will fail. I can check for sure though. | 14:11 |
| fultonj | flaper87: that's the right idea, but it's now this one https://github.com/fultonj/tripleo-ceph-ansible/blob/master/heat/external_resource.py | 14:11 |
| flaper87 | trozet: it's not, we actually need to do it ourselves :( if the image is already in kolla then one of us can build it and upload it | 14:11 |
| flaper87 | fultonj: cool, I'll read through it | 14:11 |
| flaper87 | trozet: I'm a bit stuck right now, maybe mandre can do this quickly for you | 14:11 |
| trozet | flaper87: ok thanks | 14:11 |
| trozet | flaper87: also I had a question about the networking with containers. How is that being done? | 14:12 |
| fultonj | flaper87: it came from https://review.openstack.org/#/c/420664/ | 14:12 |
| trozet | flaper87: and also thinking ahead to the future with k8s, what network plugin do you plan to use? | 14:12 |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Add maxconn parameter to MySQL / HAProxy https://review.openstack.org/457805 | 14:13 |
| flaper87 | trozet: the k8s question we'll answer later. we're not there yet | 14:13 |
| *** morazi has joined #tripleo | 14:13 | |
| flaper87 | trozet: the networking for the docker-cmd based deployment uses the host network | 14:13 |
| EmilienM | thrash: we have a problem then. /me brb 10 min from now | 14:13 |
| flaper87 | fultonj: is that patch going to be abandoned in favor of the hook ? | 14:14 |
| thrash | EmilienM: ack | 14:14 |
| fultonj | external_resource.py will replace workflow_execution.py | 14:15 |
| fultonj | gfidente: ^ | 14:15 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: move mistral base action dependency to mistral_lib https://review.openstack.org/454632 | 14:15 |
| fultonj | flaper87: sorry, let me find the correct heat hook :) | 14:15 |
| fultonj | flaper87: 420664 is correct (https://review.openstack.org/#/c/420664) OS::Mistral::ExternalResource would be used for Heat to trigger Mistral | 14:16 |
| shardy | flaper87: it's not a heat hook, it's a new heat resource plugin that can run an mistral workflow | 14:17 |
| flaper87 | fultonj: fantastic, glad there's a patch already. I'll give this a try shortly | 14:17 |
| shardy | so a little different to how e.g the docker hook etc works | 14:17 |
| flaper87 | shardy: I just realized that, awesome! | 14:17 |
| fultonj | flaper87: it's cool because different states of the resource and trigger workflows differently | 14:18 |
| *** jpich has quit IRC | 14:18 | |
| fultonj | e.g. we can have it clean up after itself | 14:18 |
| *** jpich has joined #tripleo | 14:19 | |
| fultonj | like a propper heat resource | 14:19 |
| flaper87 | nice, even better | 14:19 |
| flaper87 | my mom liked it when I used to clean after myself, now I understand why | 14:19 |
| trozet | flaper87: sorry i'm a container noob, so you create a docker bridged network and connect to the host interface for internal_api network in OOO? | 14:19 |
| flaper87 | trozet: no worries. We don't create a bridged network but use the host network directly | 14:20 |
| flaper87 | trozet: which means we bypass docker networking entirely | 14:20 |
| flaper87 | this allows us for keeping the current network architecture | 14:20 |
| flaper87 | for now | 14:20 |
| flaper87 | that's not going to be the case for k8s | 14:20 |
| openstackgerrit | Brad P. Crochet proposed openstack/instack-undercloud master: Enable mistral mod_wsgi in undercloud https://review.openstack.org/458482 | 14:21 |
| *** jkilpatr has joined #tripleo | 14:22 | |
| EmilienM | thrash: ok back. So yeah, if both ironic & mistral run in wsgi, we need to 1) check if mistral-db-populate can be run before httpd start 2) if not, restart apache after mistral-db-populate again (it sucks) | 14:23 |
| *** fragatina has quit IRC | 14:23 | |
| fultonj | flaper87: i'm compensating for missing pieces with https://github.com/fultonj/tripleo-ceph-ansible/blob/master/init.sh . the new resrouce is starting the workflow but i'm debugging my mistral workflow itself. so if you try it see_last_task.sh will expose the current issue i hope to fix today | 14:24 |
| thrash | EmilienM: Or we need to figure out why ironic-inspector has to be running in order for the mistral actions to be created properly. | 14:24 |
| thrash | But 2) is probably what's going to be necessary. | 14:24 |
| fultonj | then I thought it would be nice to get a better ansible inventory | 14:24 |
| thrash | EmilienM: I'm trying a run with the line removed to see what happens. :) | 14:24 |
| flaper87 | fultonj: how are you generating the inventory ? | 14:25 |
| thrash | EmilienM: also playing around with things locally. | 14:25 |
| thrash | EmilienM: will let you know what I find. | 14:25 |
| EmilienM | thrash: nice. you're about to break CI :P | 14:25 |
| fultonj | flaper87: a script gets it from nova | 14:25 |
| thrash | EmilienM: haha only if it lands. :D | 14:25 |
| flaper87 | fultonj: gotcha | 14:26 |
| flaper87 | fultonj: brb, call | 14:26 |
| fultonj | ack | 14:26 |
| *** saibarspeis has quit IRC | 14:28 | |
| jtomasek | jpich: so this one is most important now https://blueprints.launchpad.net/tripleo/+spec/stop-using-mistral-env | 14:28 |
| jtomasek | jpich: specifically: https://review.openstack.org/#/c/452291/6 | 14:28 |
| *** ykarel has joined #tripleo | 14:28 | |
| jtomasek | jpich: I am going to try to test it and provide a review | 14:29 |
| chem | EmilienM: he, if you could have a quick look at https://review.openstack.org/#/c/458439/ ? | 14:29 |
| EmilienM | chem: ok | 14:29 |
| EmilienM | weshay: tripleo ci meeting this week? | 14:29 |
| *** jaganathan has quit IRC | 14:29 | |
| chem | EmilienM: just to make sure you're ok with the way it's done :) | 14:29 |
| EmilienM | chem: lgtm | 14:30 |
| chem | EmilienM: ack, thanks. | 14:30 |
| *** jbadiapa has quit IRC | 14:30 | |
| *** shardy has quit IRC | 14:30 | |
| *** liupe__ has joined #tripleo | 14:31 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/ocata: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458530 | 14:31 |
| jpich | jtomasek: Ok, so the patch is already there. I'm looking at these today, planning to review and test as well | 14:31 |
| jpich | jtomasek: The only other patch I was planning on doing related to these was a follow-up patch for the TODO, as in removing the last traces of the mistral env in the CLI once everything else has merged | 14:32 |
| *** shardy has joined #tripleo | 14:32 | |
| weshay | EmilienM, I think we have too many on pto | 14:32 |
| EmilienM | k | 14:32 |
| EmilienM | I left and arxcruz too | 14:32 |
| jtomasek | jpich: ack, another nice one would be https://blueprints.launchpad.net/tripleo/+spec/get-roles-action | 14:33 |
| *** kjw3 has joined #tripleo | 14:33 | |
| jpich | jtomasek: If the important work is stuff akrivoka already provided patches for then we're on the same page :) I was wondering about the other sub-blueprints that have actions related to networks and roles | 14:33 |
| jtomasek | jpich: we'll need roles crud eventually, but having at least updated listing would be nice | 14:34 |
| jpich | jtomasek: Just getting for now, or the update/validate as well? | 14:34 |
| jtomasek | jpich: just getting is fine | 14:34 |
| jtomasek | jpich: currently we're parsing overcloud.yaml template to get a list of roles. this should be changed to getting them from roles_data.yaml in swift | 14:35 |
| jpich | jtomasek: That seems somewhat manageable (says the person who knows next to nothing about this ;)) | 14:36 |
| jtomasek | jpich: haha | 14:36 |
| jpich | jtomasek: Thank you for the pointers. I'll add that one to my to-dos and chat again if I get in over my head. *Theoretically* if we wanted to look into the other sub-blueprints, order of importance would be update/validate roles, then the networking actions -- or get-networks, and then update/validate for both...? | 14:37 |
| *** thrash is now known as thrash|biab | 14:37 | |
| jtomasek | jpich: get-networks first I guess, its up to you though | 14:38 |
| jpich | jtomasek: Ok, fine. Just wanted to make sure there aren't other blueprints/work depending on these ones | 14:39 |
| jpich | jtomasek: I'll aim my best with GetRoles but won't make any other promises as to the rest for now | 14:39 |
| jtomasek | jpich: ok, sounds great | 14:40 |
| *** hjensas has joined #tripleo | 14:41 | |
| *** hjensas has quit IRC | 14:41 | |
| *** hjensas has joined #tripleo | 14:41 | |
| *** b00tcat has quit IRC | 14:43 | |
| *** dprince has joined #tripleo | 14:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 14:46 |
| trozet | flaper87: thanks | 14:46 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo master: Switch to overlay driver for storage https://review.openstack.org/451916 | 14:51 |
| *** jcoufal has joined #tripleo | 14:52 | |
| *** bkopilov has quit IRC | 14:54 | |
| *** prateek has quit IRC | 14:54 | |
| jaosorior | trown: so, for some reason, the nova flavor_list mistral action is not returning the profile :/ | 14:55 |
| jaosorior | trown: got an environment I could access? | 14:58 |
| *** itzikb has joined #tripleo | 14:58 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Allow complex upgrade deployment for N to O https://review.openstack.org/439598 | 14:58 |
| itzikb | EmilienM: hi, regarding https://review.openstack.org/#/c/457541/. what can I do? | 14:58 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart-extras master: Download rpm tht package for mixed upgrade https://review.openstack.org/449350 | 14:59 |
| *** rcernin has quit IRC | 15:01 | |
| EmilienM | itzikb: I'm working on CI now but got lot of interruptions to finish the work | 15:01 |
| itzikb | EmilienM: ok, sorry | 15:02 |
| jaosorior | EmilienM: it seems it's not the same issue. The TLS everywhere gate just stalls when creating the compute. regardless of the timeout :/ | 15:03 |
| trown | jaosorior: my env is currently deployed with master, instead of master-tripleo-ci | 15:03 |
| trown | jaosorior: I can redeploy though with your exact args | 15:03 |
| jaosorior | trown: that would be great | 15:03 |
| jaosorior | hrybacki: hey dude, I need help figuring out why the TLS everywhere gate is broken :/ | 15:04 |
| *** ebarrera has quit IRC | 15:04 | |
| trown | jaosorior: what is in deploytls.yaml? | 15:04 |
| jaosorior | hrybacki: submitted this https://bugs.launchpad.net/tripleo/+bug/1684630 | 15:04 |
| openstack | Launchpad bug 1684630 in tripleo "ovb-fake-caserver job failing, timing out creating the Compute resource" [Undecided,New] | 15:04 |
| jaosorior | trown: http://jaormx.github.io/2017/deploying-a-tls-everywhere-environment-with-oooq-and-an-existing-freeipa-server/ | 15:04 |
| jaosorior | trown: it's the parameters to deploy a TLS everywhere environment | 15:05 |
| jaosorior | trown: well, for the overcloud. not undercloud | 15:05 |
| jaosorior | trown: you need a FreeIPA server somewhere to get that working though. I can help with that if you want | 15:06 |
| trown | jaosorior: hmm if it is only on the overcloud it is probably not related to your issue | 15:06 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Move OVB settings to tripleo-quickstart-extras https://review.openstack.org/448805 | 15:06 |
| trown | jaosorior: I am quite confused by what you are hitting though | 15:06 |
| jaosorior | trown: yeah, I'm pretty sure it's not related | 15:06 |
| jaosorior | trown: well, from what I can tell the nova.flavors_list action is not returning the profile info | 15:06 |
| jaosorior | it returns a bunch of stuff but not that | 15:07 |
| trown | jaosorior: I am trying your command with everything but the tls stuff | 15:07 |
| hrybacki | looking | 15:07 |
| trown | jaosorior: I wonder if you have a stale image in the cache and for some reason quickstart is not pulling a new one | 15:07 |
| jaosorior | trown: alright | 15:07 |
| trown | jaosorior: could you post your delorean.repo? | 15:07 |
| jaosorior | trown: let me try removing that | 15:07 |
| jaosorior | trown: sure | 15:07 |
| jaosorior | [delorean] | 15:08 |
| jaosorior | name=delorean-openstack-neutron-b49764cdc7eb0057677efea224603ddf6d4b42c0 | 15:08 |
| jaosorior | baseurl=https://trunk.rdoproject.org/centos7/b4/97/b49764cdc7eb0057677efea224603ddf6d4b42c0_ad1f2ce1 | 15:08 |
| jaosorior | enabled=1 | 15:08 |
| jaosorior | gpgcheck=0 | 15:08 |
| jaosorior | priority=20 | 15:08 |
| jaosorior | trown: ^^ | 15:08 |
| trown | jaosorior: hmm that is the correct one | 15:09 |
| trown | so you have the latest image | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 15:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 15:10 |
| *** aufi has quit IRC | 15:10 | |
| jaosorior | trown: is there a way that I can give parameters to the flavors_list? | 15:11 |
| trown | jaosorior: I unfortunately do not know that mistral code very well | 15:11 |
| dprince | EmilienM, mandre: looking at this again https://review.openstack.org/#/c/457252/. I know how to fix it. But question is do we want netiso enabled for the containers job. I think we do.... but for parity with nonha that does not use network isolation | 15:11 |
| *** udesale has quit IRC | 15:11 | |
| jaosorior | trown: here https://github.com/openstack/tripleo-common/blob/2ecb9fc6fa2fc2506c736b48bf495039efc5f273/workbooks/validations.yaml#L336 | 15:11 |
| jaosorior | thrash|biab: ^^ | 15:11 |
| *** jcoufal_ has joined #tripleo | 15:12 | |
| jaosorior | dprince: would be goot to have the docker jobs deploying with netiso. Seems like a more realistic scenario. | 15:12 |
| jaosorior | trown: who can I ask about this? | 15:12 |
| *** liupe__ has quit IRC | 15:13 | |
| *** dparkes has quit IRC | 15:13 | |
| jaosorior | trown: oh, I think I can just use input. Where do these yaml files get stored? | 15:13 |
| trown | jaosorior: thrash|biab or rbrady would know | 15:14 |
| *** pcaruana has quit IRC | 15:14 | |
| *** apetrich has quit IRC | 15:15 | |
| *** jcoufal has quit IRC | 15:15 | |
| *** apetrich has joined #tripleo | 15:16 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Allow permissive access to non_root_user files https://review.openstack.org/418198 | 15:16 |
| dprince | jaosorior: agree, but we still want one job using the non-netiso case right? | 15:17 |
| jaosorior | dprince: maybe.... or do we want to make netiso the standard? | 15:18 |
| jaosorior | bnemec: any idea why the compute logs where not copied here http://logs.openstack.org/26/458026/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/967ec0f/logs/postci.txt.gz ? | 15:19 |
| *** atheurer has quit IRC | 15:20 | |
| bnemec | jaosorior: The compute node wasn't created. There's nothing to collect from it. | 15:21 |
| *** atheurer has joined #tripleo | 15:21 | |
| openstackgerrit | Michael Bayer proposed openstack/puppet-tripleo master: Add maxconn parameter to MySQL / HAProxy https://review.openstack.org/457805 | 15:21 |
| jaosorior | bnemec: oh, I thought it was | 15:21 |
| jaosorior | bnemec: any idea why it wasn't? I saw no errors in the nova logs :/ | 15:21 |
| *** ykarel has quit IRC | 15:21 | |
| jaosorior | or actaully, I've been reading logs for hours and am pretty stumped as to why the compute doesn't get created there | 15:22 |
| *** dprince has quit IRC | 15:22 | |
| bnemec | jaosorior: Not off the top of my head. It's actually very strange that the compute node shows in nova list but the heat resource timed out: http://logs.openstack.org/26/458026/1/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/967ec0f/console.html#_2017-04-20_09_31_44_305580 | 15:23 |
| *** mdnadeem has quit IRC | 15:23 | |
| *** itzikb has quit IRC | 15:23 | |
| jaosorior | bnemec: but isn't that the base resource? the compute role thingy. And that one contains the nova compute node | 15:23 |
| jaosorior | bnemec: as far as I could tell, what actually timed out is the networkdeployment | 15:24 |
| *** arxcruz has quit IRC | 15:24 | |
| *** arxcruz has joined #tripleo | 15:24 | |
| bnemec | jaosorior: Why do you think that? According to what I see NetworkDeployment completed. | 15:25 |
| *** atheurer has quit IRC | 15:26 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 15:26 |
| bnemec | Although that would also explain why no logs were gathered. | 15:26 |
| jaosorior | bnemec: the one from the controller did. not the one from the compute | 15:26 |
| jaosorior | oh, why would it explain that? I've been pretty confused about that | 15:26 |
| *** jtomasek_ has joined #tripleo | 15:27 | |
| *** jtomasek has quit IRC | 15:27 | |
| *** jlinkes has quit IRC | 15:27 | |
| shardy | jaosorior: if the network is broken, scping the logs won't work? | 15:28 |
| *** milan has quit IRC | 15:28 | |
| jaosorior | shardy: thought the ctlplane plane would be fine either way. | 15:28 |
| jaosorior | shardy: since that's not set up by that networkdeployment | 15:28 |
| bnemec | jaosorior: Nope, the fallback doesn't work. | 15:28 |
| EmilienM | bnemec: do you have metrics about your theory of stack update longer? | 15:28 |
| bnemec | And os-net-config wipes all the config when it starts running. | 15:28 |
| jaosorior | aw shit | 15:28 |
| bnemec | EmilienM: They're in the bug. | 15:29 |
| bnemec | I've been meaning to fix the fact that our safe default failsafe doesn't actually work for about a year now. | 15:29 |
| EmilienM | bnemec: indeed | 15:30 |
| EmilienM | bnemec: i'm investigating what could have caused this increase | 15:30 |
| jaosorior | bnemec, shardy: What other options do we have for debugging that? | 15:31 |
| bnemec | EmilienM: My guess is the promotion: https://bugs.launchpad.net/tripleo/+bug/1684272/comments/5 | 15:31 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud update time" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:31 |
| *** liupe__ has joined #tripleo | 15:31 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Add defaults for docker puppet tasks https://review.openstack.org/455718 | 15:31 |
| EmilienM | bnemec: a promotion in ocata, right? | 15:31 |
| *** artom has quit IRC | 15:32 | |
| EmilienM | bnemec: it is affecting stable/ocata jobs | 15:32 |
| bnemec | jaosorior: If you're running locally you could log into the console, but in ci there isn't much. | 15:32 |
| bnemec | EmilienM: No, it's not affecting stable branches. Only master in this case. | 15:32 |
| jaosorior | bnemec: I can't reproduce it locally :/ | 15:32 |
| *** artom has joined #tripleo | 15:32 | |
| bnemec | EmilienM: See the graph in the description: https://66.187.229.172/S/I | 15:32 |
| EmilienM | bnemec: I have seen a lot of timeouts on stable/ocata for upgrade jobs | 15:32 |
| bnemec | EmilienM: Yes, that's a different bug that has been going on longer than this one. | 15:33 |
| EmilienM | bnemec: ok | 15:33 |
| EmilienM | bnemec: indeed, we hav 2 problems | 15:33 |
| bnemec | That's why I opened a second bug. :-) | 15:33 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Fix depends-on for tipleo-quickstart https://review.openstack.org/452225 | 15:34 |
| EmilienM | bnemec: remember the issue we had with ssl with heat | 15:34 |
| EmilienM | bnemec: we had to increase timeout | 15:34 |
| *** artom has quit IRC | 15:34 | |
| *** thrash|biab is now known as thrash | 15:34 | |
| EmilienM | bnemec: it sounds related | 15:34 |
| jaosorior | EmilienM: which ssl issue with heat? | 15:34 |
| *** artom has joined #tripleo | 15:34 | |
| EmilienM | bnemec: I asked zaneb and he has no clue what in heat could have regressed | 15:34 |
| bnemec | EmilienM: Yeah, I was planning to troll the heat git log to see if anything looks suspicious. | 15:34 |
| bnemec | But...time. | 15:34 |
| EmilienM | jaosorior: https://github.com/openstack/puppet-tripleo/commit/6cb95e6a69677755a070cc73062b70c84452824e | 15:35 |
| EmilienM | launchpad 1666072 | 15:35 |
| openstack | Launchpad bug 1666072 in tripleo "CI / SSL: 504 Gateway Time-out" [Critical,Fix released] https://launchpad.net/bugs/1666072 - Assigned to Emilien Macchi (emilienm) | 15:35 |
| thrash | jaosorior: Sure. | 15:35 |
| *** ukalifon has quit IRC | 15:35 | |
| thrash | jaosorior: you would just add them as input: on that action | 15:35 |
| bnemec | I wonder if that resource chain takes as long on the initial deployment. | 15:35 |
| shardy | bnemec: For the master updates job, are we sure it's not triggering yum update with the new pike-1 packages that aren't in the cached image or something? | 15:35 |
| thrash | jaosorior: in that particular workflow? Or in a different workflow? | 15:35 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 15:35 |
| *** dparkes has joined #tripleo | 15:36 | |
| shardy | oh nvm I see you've isolated it to servicechain | 15:36 |
| bnemec | shardy: Shouldn't be. This is a master job that should already have newer packages than pike-1 | 15:36 |
| bnemec | Yeah | 15:36 |
| jaosorior | thrash: the file I need to modify? I want to add that here https://github.com/openstack/tripleo-common/blob/master/workbooks/validations.yaml#L336 | 15:36 |
| shardy | I'm seeing that locally with the undercloud installer I think | 15:36 |
| bnemec | Oh, interesting. This isn't just updates. Initial deployments are having a much longer controller service chain time too. | 15:37 |
| shardy | Yeah that's what I'm seeing for undercloud deploy | 15:38 |
| shardy | I thought it was due to the single process heat, but sounds like it's not | 15:38 |
| bnemec | Before: ControllerServiceChain 36.0 | 15:38 |
| bnemec | After: ControllerServiceChain 624.0 | 15:38 |
| shardy | ouch | 15:38 |
| * shardy builds 3c06d0a heat | 15:40 | |
| *** dprince has joined #tripleo | 15:40 | |
| bnemec | It's a miracle any of the updates jobs are completing under the timeout. That's something like 20 minutes added per job. | 15:40 |
| *** dsariel has quit IRC | 15:40 | |
| *** jcoufal_ has quit IRC | 15:41 | |
| *** jcoufal has joined #tripleo | 15:42 | |
| EmilienM | bnemec: tbh I spent some time this weekend to investigate and haven't found anything. I've pinged heat folks but not much help here | 15:42 |
| *** liupe__ has quit IRC | 15:44 | |
| *** atheurer has joined #tripleo | 15:44 | |
| EmilienM | bnemec: and zaneb told me to increase the timeout, which is what I did | 15:44 |
| *** jkilpatr has quit IRC | 15:44 | |
| EmilienM | but really increasing timeouts is not really helpful... | 15:44 |
| bnemec | EmilienM: Yeah, looks like the timeout increase just masked a legitimate bug. | 15:45 |
| thrash | jaosorior: there's no reason you can't add anything there. | 15:45 |
| zaneb | bnemec: I sort-of agree, but I can't see anything that could cause a regression in the timeframe EmilienM mentioned | 15:46 |
| thrash | jaosorior: what are you looking to add anyway? | 15:46 |
| jaosorior | thrash: right, but... what I don't know is where in my installation do I edit that file and how do I reflect the changes. | 15:46 |
| *** ebarrera has joined #tripleo | 15:46 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458439 | 15:46 |
| jaosorior | thrash: nova flavors list has a "detailed" option, which I want to set to true | 15:46 |
| thrash | jaosorior: ahh. I see. Just make a copy of the file from /usr/share/openstack-tripleo-common | 15:46 |
| thrash | then do a 'openstack workbook update validations.yaml' | 15:47 |
| jaosorior | awesomeness | 15:47 |
| jaosorior | thanks dude | 15:47 |
| thrash | with validations.yaml in your $(cwd) | 15:47 |
| jaosorior | gotcha | 15:47 |
| jaosorior | thrash: thanks dude, I'll try it out | 15:47 |
| *** apetrich has quit IRC | 15:47 | |
| thrash | jaosorior: np. lemme know if you need anything else. :) | 15:47 |
| EmilienM | bnemec: on the bug I was working on (upgrade jobs timeouting) I think the mirror thing can help | 15:47 |
| EmilienM | bnemec: because the upgrade takes time to pull packages | 15:48 |
| EmilienM | if we can save little time here, it's always a bonus | 15:48 |
| bnemec | EmilienM: Sure, that makes sense. | 15:48 |
| EmilienM | bnemec: sorry I was confused | 15:48 |
| EmilienM | for the ovb-updates thing, we need to restore the CI escalation | 15:48 |
| zaneb | bnemec: did you see my comment https://bugs.launchpad.net/tripleo/+bug/1666072/comments/31 ? | 15:49 |
| openstack | Launchpad bug 1666072 in tripleo "CI / SSL: 504 Gateway Time-out" [Critical,Fix released] - Assigned to Emilien Macchi (emilienm) | 15:49 |
| zaneb | bnemec: slow validation of the ResourceChain was definitely the cause in the case I debugged | 15:50 |
| zaneb | bnemec: that could presumably be due to a change in t-h-t or in Heat. if we have some data that can narrow it down then that would be helpful | 15:51 |
| *** morazi has quit IRC | 15:52 | |
| bnemec | zaneb: Yeah, I suppose it's also possible that an earlier change in tht triggered this behavior, although apparently it wasn't present in the older version of Heat because we didn't see this until the promotion happened. | 15:53 |
| *** morazi has joined #tripleo | 15:53 | |
| bnemec | I was only looking at tht patches that merged around the 18th, but it's possible there was something earlier that has an impact here. | 15:53 |
| zaneb | bnemec: http://git.openstack.org/cgit/openstack/heat/commit?id=f94d76cb322f14a001e7990c0918a67f1a09ea16 is the most suspicious commit | 15:53 |
| bnemec | I'm not sure how long it had been since we last had a promotion before the 18th. | 15:53 |
| zaneb | but it shouldn't have had any measurable effect on create, when self.action == self.INIT | 15:54 |
| bnemec | zaneb: We can try a temprevert of that pretty easily. | 15:54 |
| *** morazi has quit IRC | 15:54 | |
| zaneb | worth a try | 15:55 |
| flaper87 | /query dsneddon | 15:56 |
| flaper87 | ops, fail | 15:56 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Temprevert f94d76cb322f14a001e7990c0918a67f1a09ea16 in heat https://review.openstack.org/458570 | 15:56 |
| *** iranzo has quit IRC | 15:56 | |
| *** morazi has joined #tripleo | 15:57 | |
| *** jmelvin has quit IRC | 15:58 | |
| *** afazekas_ is now known as afazekas | 16:00 | |
| *** liverpooler has quit IRC | 16:01 | |
| *** salmankhan has quit IRC | 16:02 | |
| *** newmember has joined #tripleo | 16:03 | |
| zaneb | bnemec: the good news is if that is the issue (which is plausible for stack *updates*) then cwolferh's patch https://review.openstack.org/#/c/422983/ should resolve it or get us very close | 16:03 |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Make docker_registry_namespace a variable https://review.openstack.org/458574 | 16:03 |
| *** liverpooler has joined #tripleo | 16:03 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Generate the list of required containers from t-h-t https://review.openstack.org/458575 | 16:03 |
| *** lmiccini has quit IRC | 16:07 | |
| *** zoli is now known as zoli|gone | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 16:10 |
| jistr | ouch | 16:11 |
| jistr | the upgrades job has regressed to the error which looks like i'm not passing the multinode environment files in | 16:12 |
| jistr | http://logs.openstack.org/84/450784/17/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/65e56b2/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-04-20_15_34_44 | 16:12 |
| jistr | but i *am* passing them in | 16:12 |
| * jistr searches for what merged lately that could affect this | 16:12 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 16:13 |
| *** leanderthal is now known as leanderthal|afk | 16:13 | |
| *** trown is now known as trown|lunch | 16:16 | |
| *** trozet has quit IRC | 16:18 | |
| *** jpich has quit IRC | 16:20 | |
| *** suuuper has quit IRC | 16:20 | |
| *** limao has quit IRC | 16:21 | |
| *** newmember has quit IRC | 16:22 | |
| *** jaganathan has joined #tripleo | 16:23 | |
| *** newmember has joined #tripleo | 16:23 | |
| pabelanger | EmilienM: left comments on ^. You shouldn't need to manipulate variables any more, just source the file and use NODEPOOL_CENTOS_MIRROR directly | 16:23 |
| pabelanger | same with NODEPOOL_RDO_PROXY | 16:23 |
| EmilienM | pabelanger: yeah I saw | 16:24 |
| EmilienM | i'm working on it no | 16:24 |
| EmilienM | now | 16:24 |
| *** yprokule has quit IRC | 16:24 | |
| *** hewbrocca is now known as hewbrocca_afk | 16:24 | |
| *** paramite has quit IRC | 16:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/456030 | 16:28 |
| *** japestinho has quit IRC | 16:30 | |
| *** lucasagomes is now known as lucas-afk | 16:30 | |
| *** arxcruz has quit IRC | 16:33 | |
| *** arxcruz has joined #tripleo | 16:33 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 16:33 |
| EmilienM | pabelanger: can you review it again? ^ | 16:33 |
| *** karimb has quit IRC | 16:34 | |
| *** atheurer has quit IRC | 16:34 | |
| *** tesseract has quit IRC | 16:34 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 16:34 |
| pabelanger | EmilienM: feedback on NODEPOOL_CENTOS_MIRROR. | 16:38 |
| EmilienM | pabelanger: merci | 16:40 |
| *** arxcruz has quit IRC | 16:40 | |
| *** arxcruz has joined #tripleo | 16:41 | |
| *** salmankhan has joined #tripleo | 16:41 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 16:42 |
| *** cylopez has quit IRC | 16:43 | |
| *** jaganathan has quit IRC | 16:43 | |
| *** fragatina has joined #tripleo | 16:45 | |
| *** jmelvin has joined #tripleo | 16:45 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/ocata: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458530 | 16:45 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 16:45 |
| *** jcoufal has quit IRC | 16:46 | |
| shardy | bogdando: Hey see https://review.openstack.org/#/c/458407 which we discussed yesterday, it updates the tripleo-undercloud-passwords.yaml so I think re-running undercloud deploy should now work modulo other bugs | 16:46 |
| bogdando | shardy: thanks, looking. What should be the next step to make https://review.openstack.org/#/c/457984/ deploying rabbit with non empty cookie? Shall I restore the tripleo-common patch with depends on set on your patch?.. | 16:48 |
| *** derekh has quit IRC | 16:48 | |
| bogdando | shardy: I mean the abandoned one here https://review.openstack.org/#/q/topic:bug/1684044 | 16:49 |
| *** jaosorior has quit IRC | 16:49 | |
| *** ckyriakidou has quit IRC | 16:51 | |
| shardy | bogdando: I'm still not sure why you're seeing it empty, with my patch we should save the RabbitCookie to tripleo-undercloud-passwords.yaml so it will always be set to the same string as the first undercloud deploy | 16:51 |
| bogdando | shardy, please also take a look the graph builder, if you have time. It works well but omits some nodes (unknown from/to)... f.e. UndercloudDockerPuppetTasksDeployment4 in the rendered post.yaml (undercloud) | 16:51 |
| shardy | bogdando: Yeah, IMHO we don't need the tripleo-common part, but I'm still testing this, feedback welcome | 16:51 |
| bogdando | so I 'm about to resurrect https://openstack.nimeyo.com/10145/openstack-dev-heat-dependency-visualisation | 16:51 |
| bogdando | and fix a little bit, with some help needed though :) | 16:52 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Automatically enable all available languages https://review.openstack.org/456646 | 16:52 |
| shardy | bogdando: Hmm, actually there's a bug, we need to not update existing keys after the first deploy | 16:52 |
| shardy | (a bug in my patch) | 16:52 |
| *** atheurer has joined #tripleo | 16:56 | |
| *** newmember has quit IRC | 16:56 | |
| bnemec | Hmm, Juan left. | 16:58 |
| bnemec | It looks like that compute node timeout problem he was looking into is a common one: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22%5Bovercloud.Compute.0%5D%3A%20CREATE_FAILED%20%20CREATE%20aborted%5C%22 | 16:59 |
| bnemec | 31 hits in the past 24 hours. :-/ | 16:59 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 16:59 |
| *** ffiore has quit IRC | 17:02 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 17:02 |
| *** liverpooler has quit IRC | 17:03 | |
| EmilienM | pabelanger: it failed because the / in the variable, in the sed. Should I use alternate regex ? | 17:06 |
| *** liverpooler has joined #tripleo | 17:06 | |
| pabelanger | EmilienM: escape it? | 17:07 |
| EmilienM | pabelanger: I use the variable that you provide in infra | 17:07 |
| pabelanger | EmilienM: got a log file? | 17:07 |
| EmilienM | pabelanger: http://logs.openstack.org/74/458474/16/check/gate-tripleo-ci-centos-7-multinode-upgrades-nv/a8614e0/console.html#_2017-04-20_16_54_16_686816 | 17:08 |
| EmilienM | pabelanger: but I could reproduce it locally | 17:08 |
| EmilienM | it's simply because the variable contains / so the sed fails | 17:08 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 17:09 |
| mwhahaha | EmilienM, use # maybe? | 17:09 |
| mwhahaha | i think that can be used instead | 17:10 |
| mwhahaha | or maybe i'm confusing ruby | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 17:10 |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| *** gfidente is now known as gfidente|afk | 17:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| *** dprince has quit IRC | 17:11 | |
| *** shardy has quit IRC | 17:12 | |
| *** salmankhan has quit IRC | 17:12 | |
| EmilienM | pabelanger, mwhahaha found it | 17:13 |
| EmilienM | TIL sed | 17:13 |
| *** saibarspeis has joined #tripleo | 17:13 | |
| mwhahaha | no one actually learns sed | 17:14 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Update fakeha net-iso configuration https://review.openstack.org/458596 | 17:14 |
| pabelanger | EmilienM: sed -e "s|^baseurl=http://mirror.centos.org/centos|baseurl=${NODEPOOL_CENTOS_MIRROR}|;" -i /foo.bar | 17:15 |
| EmilienM | yeah | 17:15 |
| EmilienM | pabelanger: I have another syntax but i think it will work too | 17:15 |
| *** ebarrera has quit IRC | 17:16 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 17:17 |
| EmilienM | pabelanger, mwhahaha : https://review.openstack.org/#/c/458474/16..17/scripts/tripleo.sh | 17:17 |
| *** jtomasek_ has quit IRC | 17:17 | |
| mwhahaha | EmilienM: i like # better but i guess ~ works :D | 17:18 |
| *** yamahata has joined #tripleo | 17:18 | |
| EmilienM | mwhahaha: teach me please the difference, i'm happy to change it | 17:18 |
| mwhahaha | i don't think there is a difference | 17:18 |
| mwhahaha | other than visibility | 17:18 |
| mwhahaha | or readability | 17:19 |
| *** saibarsp_ has joined #tripleo | 17:19 | |
| *** bogdando has quit IRC | 17:21 | |
| *** jtomasek has joined #tripleo | 17:22 | |
| *** saibarspeis has quit IRC | 17:22 | |
| *** dsneddon has quit IRC | 17:30 | |
| *** jkilpatr has joined #tripleo | 17:30 | |
| *** gbarros has quit IRC | 17:32 | |
| *** dsneddon has joined #tripleo | 17:32 | |
| bandini | anyone else seeing this on the undercloud: 2017-04-20 18:31:14.703 28461 ERROR ironic.conductor.manager [req-8c2c2cd9-2fd8-4aff-92f8-ad22c58e3765 9fb8c1bad7ca46218c67b2c2894f9cb8 a800cadc87bb4fcca0be37ff0e92e6f1 - - -] Error while preparing to deploy to node c9d8afe3-917b-4dca-9f92-fa04778c4414: Swift temporary URLs require a shared secret to be created. You must provide "swift_temp_url_key" as a config | 17:34 |
| bandini | option. | 17:35 |
| *** saibarsp_ has quit IRC | 17:35 | |
| *** dspano has joined #tripleo | 17:36 | |
| *** dprince has joined #tripleo | 17:44 | |
| *** tosky has quit IRC | 17:45 | |
| *** atoth has joined #tripleo | 17:47 | |
| *** trown|lunch is now known as trown | 17:48 | |
| *** dtrainor has quit IRC | 17:48 | |
| *** dtrainor has joined #tripleo | 17:49 | |
| *** cylopez has joined #tripleo | 17:57 | |
| *** chem is now known as chem_gone | 17:57 | |
| *** mhenkel_ has joined #tripleo | 17:58 | |
| *** fragatina has quit IRC | 17:58 | |
| *** dprince has quit IRC | 17:59 | |
| *** mhenkel_ has quit IRC | 18:02 | |
| *** cylopez has quit IRC | 18:05 | |
| *** karimb has joined #tripleo | 18:05 | |
| *** karimb has quit IRC | 18:08 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 18:10 |
| *** jcoufal has joined #tripleo | 18:13 | |
| slagle | owalsh: are you going to backport all of https://review.openstack.org/#/q/topic:bp/tripleo-cold-migration to ocata? | 18:14 |
| *** pkovar has quit IRC | 18:17 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 18:18 |
| *** rbowen has quit IRC | 18:24 | |
| *** rbowen has joined #tripleo | 18:24 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 18:26 |
| *** saibarspeis has joined #tripleo | 18:30 | |
| *** mcornea has quit IRC | 18:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike https://review.openstack.org/457603 | 18:30 |
| mwhahaha | i'm seeing dlrn failures occasionally http://logs.openstack.org/55/458155/1/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/eaf81d6/console.html#_2017-04-20_17_27_18_331982 | 18:33 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: WIP: Add OVB support to devmode.sh https://review.openstack.org/452249 | 18:33 |
| *** artom has quit IRC | 18:37 | |
| EmilienM | mwhahaha: echo openstack/puppet-openstack-integration ? | 18:37 |
| *** artom has joined #tripleo | 18:37 | |
| mwhahaha | i know, i am confused about this | 18:37 |
| mwhahaha | http://logs.openstack.org/85/457585/6/gate/gate-tripleo-ci-centos-7-undercloud-oooq/9f19e0e/console.html#_2017-04-20_17_40_28_071226 | 18:38 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo master: Switch to overlay driver for storage https://review.openstack.org/451916 | 18:38 |
| mwhahaha | unfortunately i have no idea what it's doing (have i said how much i hate ansible output) | 18:38 |
| EmilienM | I don't see errors on http://logs.openstack.org/55/458155/1/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/eaf81d6/logs/undercloud/home/jenkins/repo_setup.log.txt.gz | 18:38 |
| *** jerrygb has quit IRC | 18:39 | |
| mwhahaha | is it because p-o-i isn't packaged? | 18:39 |
| mwhahaha | or is it packaged | 18:39 |
| EmilienM | dmacpher: any idea ^ ? | 18:39 |
| EmilienM | err | 18:39 |
| EmilienM | dmsimard: ^ | 18:39 |
| *** rcrit has quit IRC | 18:40 | |
| mwhahaha | but where is that even coming from | 18:40 |
| mwhahaha | because rdpokg findpkg puppet-openstack-integration does fail | 18:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 18:40 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike https://review.openstack.org/457603 | 18:40 |
| EmilienM | mwhahaha: do you see it randomly or consistently? | 18:43 |
| mwhahaha | randomly | 18:43 |
| mwhahaha | because these are in the gate | 18:43 |
| mwhahaha | but the check passed | 18:43 |
| mwhahaha | i'm rechecking to see if it's consistent or something but it's weird | 18:44 |
| *** jerrygb has joined #tripleo | 18:51 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Switch the ovb-containers-oooq to use network iso https://review.openstack.org/457252 | 18:53 |
| jkilpatr | bnemec, is there anywhere I should open a patchest for a scale CI job? | 19:00 |
| *** jcoufal_ has joined #tripleo | 19:01 | |
| bnemec | jkilpatr: You could try a tripleo-ci patch that hijacks an existing job. | 19:03 |
| bnemec | For example, http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test-orig.sh#n220 | 19:03 |
| bnemec | Change the NODECOUNT and add a --compute-scale argument to the deploy args. | 19:03 |
| bnemec | You'll probably have issues due to the small undercloud though. | 19:04 |
| jkilpatr | how much ram do these underclouds get? | 19:04 |
| bnemec | Only 8 GB. | 19:04 |
| *** jcoufal has quit IRC | 19:04 | |
| jkilpatr | uh how does that even deploy? I've seen a trivial 3 node cloud push more than that. | 19:05 |
| bnemec | It's pretty much the bare minimum. | 19:06 |
| bnemec | But it matches what is used elsewhere in openstack-infra so we're somewhat limited there. | 19:06 |
| bnemec | It might be an interesting experiment to bump the ovb undercloud flavor though. I could see it significantly improving performance in those jobs. | 19:07 |
| bnemec | And we'd still have the 8 GB multinode jobs to make sure our memory usage didn't get out of control. | 19:07 |
| *** jerrygb has quit IRC | 19:07 | |
| jkilpatr | do they push in to swap? do you have enough metric gathering to even know if they do? | 19:07 |
| *** mcornea has joined #tripleo | 19:08 | |
| bnemec | We had swap at one point. I don't recall whether we do at the moment. | 19:08 |
| bnemec | Oh, we have 8 GB of swap. | 19:09 |
| jkilpatr | you're probably pushing into that | 19:09 |
| bnemec | Only about 100 MB used at the end of the job though. | 19:09 |
| bnemec | Swap: 8.0G 100M 7.9G | 19:09 |
| jkilpatr | yeah but if it's being used during the highest load point of the deploy it would still slow things down | 19:10 |
| jkilpatr | need a graph over time | 19:10 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 19:10 |
| bnemec | Yeah, I thought we had dstat output somewhere. I'm not immediately seeing it though. | 19:10 |
| * bnemec keeps looking | 19:10 | |
| jkilpatr | considering a 10 minute delay was marked as critical not using swap sounds high priority :) | 19:11 |
| jkilpatr | for Zuul how do I view logs and stuff? the xdg open with telnet doesn't work in chromium or firefox or if I just try and telnet from my terminal | 19:11 |
| bnemec | Hmm, the telnet console generally works for me. Except for ipv6 nodes, where I have to telnet in from an ipv6 vm that I have. | 19:15 |
| openstackgerrit | Brad P. Crochet proposed openstack/instack-undercloud master: Enable mistral mod_wsgi in undercloud https://review.openstack.org/458482 | 19:15 |
| bnemec | jkilpatr: Swap actually doesn't look too bad in http://logs.openstack.org/06/453806/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/6bd6718/logs/dstat.txt.gz | 19:15 |
| bnemec | There's some paging that happens for a few minutes, but other than that it's pretty minimal. | 19:15 |
| bnemec | Nearly 2 GB of buffers/cache too. | 19:16 |
| *** fragatina has joined #tripleo | 19:16 | |
| rook | um, please no swapping, even if we aren't talking about performance | 19:16 |
| rook | jkilpatr yes! | 19:17 |
| jkilpatr | rook, these people need grafana | 19:18 |
| rook | jkilpatr bro. I told weshay to get our shit a long time ago | 19:18 |
| rook | he is too busy :-P | 19:18 |
| * mwhahaha puts rook and jkilpatr into swap and will check back next week | 19:19 | |
| rook | mwhahaha happens with most performance people... | 19:19 |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Generate the list of required containers from t-h-t https://review.openstack.org/458575 | 19:20 |
| bnemec | Performance? That's someone else's problem. | 19:20 |
| bnemec | - Every developer ever who doesn't have "performance" in their job title | 19:20 |
| *** cylopez has joined #tripleo | 19:20 | |
| *** pkovar has joined #tripleo | 19:20 | |
| jkilpatr | bnemec, then there are only like what 8 people with that title... | 19:20 |
| jkilpatr | bnemec, anyways what would it take to get a larger undercloud flavor and some smaller overcloud node flavors? | 19:20 |
| jkilpatr | compute node specifically | 19:20 |
| rook | bnemec haha | 19:21 |
| mwhahaha | act of god | 19:21 |
| *** cylopez has quit IRC | 19:21 | |
| rook | jkilpatr in the OVB cloud, are you disabling telemetry on the UC | 19:21 |
| rook | I hope to god you are. | 19:21 |
| bnemec | We do that automatically for everything except the nonha job. | 19:21 |
| jkilpatr | rook, you mean the host/real/what do we call this? cloud | 19:22 |
| rook | in any CI case | 19:22 |
| bnemec | jkilpatr: Well, we can't make the overcloud nodes smaller. They will also oom if we do. | 19:22 |
| rook | why have that on, unless it is to test telem | 19:22 |
| *** karimb has joined #tripleo | 19:22 | |
| bnemec | "unless it is to test telem" <- That is why | 19:22 |
| bnemec | We have users which need telemetry on the undercloud. | 19:23 |
| jkilpatr | rook, I was under the impression that quickstart disabled it but I never actually checked. | 19:23 |
| jkilpatr | bnemec, any of these jobs on quickstart yet? | 19:23 |
| *** newmember has joined #tripleo | 19:23 | |
| bnemec | Maybe some? I don't know, I don't do quickstart. | 19:23 |
| bnemec | There exist some quickstart ovb jobs but I don't know what they test off the top of my head. | 19:24 |
| rook | bnemec "i don't do quickstart" | 19:25 |
| rook | what do you do? | 19:25 |
| bnemec | Follow the docs. Like our users have to. | 19:25 |
| *** atoth has quit IRC | 19:25 | |
| * mwhahaha seconds that | 19:25 | |
| jkilpatr | the docs don't work if you have a cloud larger than like 30 nodes (was 20 fixed that) | 19:26 |
| jkilpatr | which is why we want a scale job... | 19:26 |
| bnemec | The persistence of developer "easy mode" tools is why the TripleO user experience is still not where it should be. | 19:26 |
| weshay | +1 | 19:26 |
| bnemec | "User experience? That's someone else's problem" | 19:26 |
| bnemec | - Every developer ever who doesn't have "user experience" in their job title | 19:26 |
| * bnemec has an idea for a new irc bot... | 19:26 | |
| jkilpatr | bnemec, uh the perf/scale team is on the other side | 19:27 |
| weshay | jkilpatr, I think we need to chat w/ arxcruz | 19:27 |
| trown | a bit of a chicken egg problem though | 19:27 |
| bnemec | jkilpatr: I'm being somewhat facetious. | 19:27 |
| bnemec | But it is a problem. | 19:27 |
| trown | if the developer experience is not good, then who is going to work on the thing to make the user experience good | 19:27 |
| *** newmember has quit IRC | 19:27 | |
| arxcruz | weshay: yes? | 19:27 |
| mwhahaha | developer experiance shouldn't matter as much as end user experiance. because the goal should be to have 10x the users than developers | 19:28 |
| mwhahaha | where as we currently have 10x developers to users | 19:28 |
| *** newmember has joined #tripleo | 19:28 | |
| weshay | jkilpatr, is looking to get a big scale job running.. since there is no capacity to do that kind of thing in a check gate.. I thought your publish results app would come in handy | 19:28 |
| trown | I dont think that is true: "10x developers to users" | 19:28 |
| bnemec | jkilpatr: So, back off the tangent I wandered onto, we _could_ potentially increase the undercloud size in ovb jobs, and I can write a testenv patch that will give you smaller compute nodes for the overcloud. | 19:28 |
| weshay | for instance if jkilpatr were to run a 20-30 node scale out test.. w/ internal hardware.. we have the public logs | 19:29 |
| weshay | but nobody would see the results w/o a tool like yours | 19:29 |
| bnemec | We'd need to discuss the undercloud size change to see if anyone has concerns. | 19:29 |
| weshay | jkilpatr, do you want to chat about that for a minute w/ arx? | 19:29 |
| jkilpatr | weshay, sure | 19:29 |
| weshay | arxcruz, you have time? | 19:29 |
| arxcruz | weshay: sure | 19:30 |
| arxcruz | what can i do for you boss? | 19:30 |
| *** hjensas has quit IRC | 19:30 | |
| jkilpatr | weshay, I do have the entire perf monitoring/visualization stack sitting outside the firewall for public publishing | 19:30 |
| weshay | jkilpatr, aye | 19:30 |
| mwhahaha | trown: i would argue that many of the users today would be closer to 'developers' than end users. yes 10x is an exageration but it's definately more devs than users | 19:33 |
| *** newmember has quit IRC | 19:34 | |
| *** newmember has joined #tripleo | 19:35 | |
| jkilpatr | I'll take a moment to point out that most real users will have on the order of more 10-15 nodes and that's the usecase I'm trying to improve here | 19:35 |
| jkilpatr | because it's not tested well, the workflow may not be great but at least it works on smaller scale. | 19:35 |
| *** trozet has joined #tripleo | 19:35 | |
| mwhahaha | well we need to be able to expose the tunables in order to make that happen in a user friendly fashion. Ideally some should ship out of the box but they may not be the defaults we can do in our CI | 19:36 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates master: Enable mistral to run under mod_wsgi https://review.openstack.org/458504 | 19:36 |
| *** jerrygb has joined #tripleo | 19:36 | |
| mwhahaha | and those should be in the docs which we shoiuld be following... | 19:37 |
| bnemec | zaneb: Hmm, the temprevert knocked about half off the time off the service chain create and update. | 19:38 |
| bnemec | ControllerServiceChain 333.0 | 19:38 |
| bnemec | http://logs.openstack.org/70/458570/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/40cbd60/logs/undercloud/var/log/heat-deploy-times.txt.gz | 19:38 |
| *** saibarspeis has quit IRC | 19:39 | |
| zaneb | bnemec: so... that's one of but not the only culprit? | 19:41 |
| bnemec | zaneb: That would be my conclusion. | 19:41 |
| zaneb | ugh | 19:41 |
| bnemec | Yep | 19:42 |
| bnemec | Always fun when problems stack. | 19:42 |
| stevebaker | morning | 19:43 |
| *** eck` is now known as eck`gone | 19:43 | |
| EmilienM | mwhahaha, pabelanger: I saw https://review.openstack.org/#/c/458474/ working in CI. The patch is ready for review, when you have time of courser | 19:43 |
| bnemec | o/ | 19:43 |
| EmilienM | stevebaker: hey | 19:45 |
| EmilienM | bnemec: nice one | 19:45 |
| *** jprovazn has quit IRC | 19:46 | |
| bnemec | EmilienM: Well, zaneb suggested the patch. I just did the typey typey in tripleo-ci. :-) | 19:46 |
| EmilienM | bnemec: ah ok, so revert the "nice one" :P | 19:46 |
| zaneb | If it's important then I'm ok with revert now and resubmit once cwolferh's patch has made this efficient again | 19:48 |
| EmilienM | bnemec: could we monitor ControllerServiceChain and make job failing if over a certain value | 19:49 |
| bnemec | EmilienM: We could potentially set upper bounds on certain resources. Although this one only gets us about half way back to normal. | 19:51 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Add yum-utils as EPEL dependency https://review.openstack.org/458638 | 19:51 |
| bnemec | We could also just set an upper bound on deployment time as a whole. | 19:51 |
| bnemec | Although that would vary significantly by job type, so it might be a little trickier to get right. We have the metrics on what is normal though. | 19:53 |
| dmsimard | EmilienM: sorry just saw your ping, I don't understand the question ? | 19:53 |
| EmilienM | dmsimard: mwhahaha found some weird things when delorean runs in gate, he posted logs if you can look when you have time | 19:53 |
| dmsimard | When it rebuilds a package inside a job ? Not familiar with the implementation but I can look | 19:54 |
| *** fragatina has quit IRC | 19:55 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add os-collect-config data as an output https://review.openstack.org/458641 | 19:55 |
| *** snecklifter has quit IRC | 19:56 | |
| *** snecklifter has joined #tripleo | 19:57 | |
| dmsimard | EmilienM, mwhahaha: so it's coming from here.. https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/build-test-packages | 20:07 |
| mwhahaha | yea i looked at that but have no idea how that works | 20:07 |
| dmsimard | But why is it trying to build something for a puppet review ? | 20:07 |
| mwhahaha | i wasn't sure how the list of things to build gets generated | 20:08 |
| mwhahaha | because there are no p-o-i packages as far as i know | 20:08 |
| bnemec | We filtered that out in tripleo-ci | 20:08 |
| dmsimard | Of course there isn't | 20:08 |
| mwhahaha | and it shouldn't be building that for puppet-tripleo or instack-undercloud | 20:08 |
| * bnemec finds the code | 20:08 | |
| mwhahaha | did the filter get lost in oooq | 20:08 |
| bnemec | http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n298 | 20:09 |
| dmsimard | mwhahaha: it looks like Sagi might be familiar with that role but he doesn't seem to be around | 20:09 |
| *** gbarros has joined #tripleo | 20:10 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 20:10 |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 20:10 |
| dmsimard | bnemec: that doesn't look very pretty but I guess it works ? | 20:12 |
| dmsimard | Maybe that filter should live in the role instead so that it doesn't try to build nonsense | 20:13 |
| *** salmankhan has joined #tripleo | 20:13 | |
| *** jcoufal_ has quit IRC | 20:14 | |
| *** jcoufal has joined #tripleo | 20:15 | |
| *** mcornea has quit IRC | 20:15 | |
| *** liverpooler has quit IRC | 20:16 | |
| *** gbarros has quit IRC | 20:19 | |
| *** jcoufal has quit IRC | 20:20 | |
| *** florianf has quit IRC | 20:21 | |
| EmilienM | patch to use AFS mirrors in multinode jobs not deployed by quickstart: https://review.openstack.org/#/c/458474/ - please review | 20:33 |
| *** jprovazn has joined #tripleo | 20:34 | |
| *** yolanda has quit IRC | 20:35 | |
| *** gbarros has joined #tripleo | 20:37 | |
| *** Goneri has quit IRC | 20:37 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: Fix path to overcloud_containers.yaml https://review.openstack.org/458650 | 20:40 |
| *** mburned is now known as mburned_out | 20:41 | |
| *** jkilpatr has quit IRC | 20:45 | |
| *** mburned_out is now known as mburned | 20:48 | |
| EmilienM | not sure why but gate-tripleo-ci-centos-7-multinode-upgrades-nv is green on master | 20:48 |
| EmilienM | and I confirm it upgraded from ocata to pike \o/ | 20:50 |
| * EmilienM out | 20:50 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Support heterogeneous OVB environments https://review.openstack.org/458651 | 20:50 |
| bnemec | Hmm, jkilpatr left. :-/ | 20:51 |
| bnemec | rook: ^would allow testenvs with a few controller nodes and a bunch of smaller computes. | 20:51 |
| *** trown is now known as trown|outtypewww | 20:52 | |
| jprovazn | bnemec, hi, do you have a minute? I'm struggling with multiple-nics virtual env | 20:54 |
| bnemec | jprovazn: Sure | 21:00 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Add yum-utils as EPEL dependency https://review.openstack.org/458638 | 21:01 |
| *** fragatina has joined #tripleo | 21:01 | |
| jprovazn | bnemec, when I deploy OC with multiple nics, it fails probably because external network is not set properly: http://paste.openstack.org/show/607403/ | 21:01 |
| jprovazn | bnemec, UC and OC's controller interfaces look like: http://paste.openstack.org/show/607404/ | 21:02 |
| jprovazn | bnemec, I suspect that the vlan on UC is not correct | 21:02 |
| bnemec | jprovazn: If you're using multi-nic, you shouldn't use a vlan on the undercloud. Just configure the appropriate undercloud nic directly with 10.0.0.1. | 21:03 |
| jprovazn | bnemec, yes, I thought that vlan is inappropriate - seems oooq set it because I omit some variable | 21:04 |
| jprovazn | bnemec, dump question then... which interface is appropriate? | 21:06 |
| bnemec | jprovazn: I'm not sure what the appropriate interface in quickstart is. I would expect it's whichever one has vlan10 associated with it though. | 21:07 |
| jprovazn | bnemec, thanks | 21:09 |
| bnemec | np | 21:09 |
| * bnemec goes for afternoon tea | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 21:10 |
| *** ramishra has quit IRC | 21:10 | |
| *** gfidente|afk has quit IRC | 21:12 | |
| *** jprovazn has quit IRC | 21:15 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-collect-config stable/ocata: Updated from global requirements https://review.openstack.org/443854 | 21:16 |
| *** jlinkes has joined #tripleo | 21:16 | |
| *** jlinkes has quit IRC | 21:16 | |
| *** ramishra has joined #tripleo | 21:17 | |
| *** jkilpatr has joined #tripleo | 21:23 | |
| *** mburned is now known as mburned_out | 21:26 | |
| *** salmankhan has quit IRC | 21:26 | |
| *** mburned_out is now known as mburned | 21:28 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Use comma_delimited_list for token flush cron time settings https://review.openstack.org/458003 | 21:30 |
| *** pkovar has quit IRC | 21:34 | |
| *** karimb has quit IRC | 21:34 | |
| *** ramishra has quit IRC | 21:39 | |
| *** ramishra has joined #tripleo | 21:41 | |
| *** Goneri has joined #tripleo | 21:44 | |
| *** rbrady has quit IRC | 21:49 | |
| *** lblanchard has joined #tripleo | 21:52 | |
| *** gbarros has quit IRC | 21:56 | |
| *** rbowen has quit IRC | 21:57 | |
| *** tobias_fiberdata has quit IRC | 22:01 | |
| *** mburned is now known as mburned_out | 22:02 | |
| *** Goneri has quit IRC | 22:03 | |
| *** limao has joined #tripleo | 22:03 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-docs master: Fix references to files in tripleo-common/contrib https://review.openstack.org/458669 | 22:06 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 22:10 |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 22:10 |
| *** dspano has quit IRC | 22:15 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/ocata: Add MigrationSshKey to generated passwords https://review.openstack.org/458671 | 22:16 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/ocata: SSH known_hosts config https://review.openstack.org/458672 | 22:16 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates stable/ocata: Add migration SSH tunneling support https://review.openstack.org/458673 | 22:17 |
| *** lblanchard1 has joined #tripleo | 22:18 | |
| *** morazi has quit IRC | 22:18 | |
| *** lblanchard has quit IRC | 22:21 | |
| openstackgerrit | James Slagle proposed openstack/puppet-tripleo stable/ocata: Configure migration SSH tunnel https://review.openstack.org/458674 | 22:25 |
| *** rbowen has joined #tripleo | 22:25 | |
| *** lblanchard1 has quit IRC | 22:25 | |
| *** rbowen has quit IRC | 22:31 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add yum-utils as EPEL dependency https://review.openstack.org/458638 | 22:31 |
| *** jmelvin has quit IRC | 22:38 | |
| *** thrash is now known as thrash|g0ne | 22:39 | |
| *** kjw3 has quit IRC | 23:00 | |
| *** colonwq has quit IRC | 23:09 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 23:10 |
| mnaser | is it safe to make some small changes and rerun the deploy cli command? i'm trying to figure out the best way to test changes to create an ideal environment file | 23:20 |
| mnaser | if i have to reinstall the overcloud everytime that might end up taking quite sometime. | 23:20 |
| mwhahaha | yea rerunning it should result in an update | 23:21 |
| mwhahaha | so depends on what your small change is | 23:21 |
| *** limao has quit IRC | 23:23 | |
| *** dsneddon is now known as dsneddon_afk | 23:29 | |
| *** jerrygb has quit IRC | 23:45 | |
| *** mburned_out is now known as mburned | 23:49 | |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: Add favicon icons https://review.openstack.org/420111 | 23:52 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add all hosts to HostsEntry output https://review.openstack.org/457381 | 23:55 |
| mnaser | mwhahaha cool, i'll be experimenting! | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!