| openstackgerrit | Merged openstack/puppet-tripleo master: Update UI language list https://review.openstack.org/455656 | 00:06 |
|---|---|---|
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add NeutronDnsDomain heat option, undercloud fix https://review.openstack.org/451517 | 00:08 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Automatically read both voting and non-voting forms of job https://review.openstack.org/453806 | 00:08 |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 00:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 00:10 |
| *** artom has quit IRC | 00:10 | |
| *** artom has joined #tripleo | 00:11 | |
| *** artom has quit IRC | 00:12 | |
| *** artom has joined #tripleo | 00:12 | |
| *** rlandy has quit IRC | 00:17 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Add multinode-upgrades scenario jobs to cistatus https://review.openstack.org/453807 | 00:40 |
| openstackgerrit | Merged openstack/instack-undercloud master: Disable Telemetry services on undercloud by default https://review.openstack.org/458155 | 00:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add environment to preselect only VIP IP addresses https://review.openstack.org/336749 | 00:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add network_data.yaml to encapsulate list of networks for j2 https://review.openstack.org/409921 | 00:41 |
| *** jerrygb has joined #tripleo | 00:45 | |
| *** limao has joined #tripleo | 00:50 | |
| *** jerrygb has quit IRC | 00:52 | |
| *** tobias_fiberdata has joined #tripleo | 00:56 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/ocata: Configure migration SSH tunnel https://review.openstack.org/458674 | 01:02 |
| *** jerrygb has joined #tripleo | 01:04 | |
| *** fragatina has quit IRC | 01:08 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 01:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 01:10 |
| *** newmember has quit IRC | 01:17 | |
| *** eck`gone is now known as eck` | 01:18 | |
| *** fzdarsky_ has joined #tripleo | 01:21 | |
| *** fzdarsky has quit IRC | 01:24 | |
| *** japestinho has joined #tripleo | 01:58 | |
| *** yamahata has quit IRC | 02:03 | |
| *** newmember has joined #tripleo | 02:07 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 02:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 02:10 |
| *** rbowen has joined #tripleo | 02:13 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata https://review.openstack.org/458714 | 02:25 |
| *** rbowen has quit IRC | 02:40 | |
| *** Erming_ has joined #tripleo | 03:03 | |
| *** Erming__ has quit IRC | 03:03 | |
| *** chem_gone has quit IRC | 03:03 | |
| *** chem_gone has joined #tripleo | 03:04 | |
| *** bugzy has quit IRC | 03:04 | |
| *** bugzy has joined #tripleo | 03:06 | |
| *** dmacpher has quit IRC | 03:06 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 03:09 |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 03:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 03:10 |
| *** portdirect has quit IRC | 03:16 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Use conditionals for neutron and glance worker defaults https://review.openstack.org/425423 | 03:16 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 03:18 |
| *** fragatina has joined #tripleo | 03:19 | |
| *** dmacpher has joined #tripleo | 03:19 | |
| *** portdirect has joined #tripleo | 03:23 | |
| *** pmannidi has quit IRC | 03:25 | |
| *** tzumainn has quit IRC | 03:26 | |
| *** psahoo has joined #tripleo | 03:31 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: N->O upgrade, fix wrong parameters to nova placement. https://review.openstack.org/458416 | 03:39 |
| *** gkadam has joined #tripleo | 03:42 | |
| *** ykarel has joined #tripleo | 04:00 | |
| *** pmannidi has joined #tripleo | 04:04 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 04:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 04:10 |
| *** atheurer has quit IRC | 04:22 | |
| *** yamahata has joined #tripleo | 04:24 | |
| *** atheurer has joined #tripleo | 04:33 | |
| *** pgadiya has joined #tripleo | 04:34 | |
| *** udesale has joined #tripleo | 04:42 | |
| *** jaganathan has joined #tripleo | 04:43 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Refactor SSHD config to allow both SSHD options and banner/motd to be set https://review.openstack.org/457585 | 05:00 |
| *** atheurer has quit IRC | 05:01 | |
| *** ratailor has joined #tripleo | 05:05 | |
| *** skramaja has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 05:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 05:10 |
| *** apetrich has joined #tripleo | 05:12 | |
| *** jaganathan has quit IRC | 05:25 | |
| *** apetrich has quit IRC | 05:30 | |
| *** apetrich has joined #tripleo | 05:32 | |
| *** prateek has joined #tripleo | 05:35 | |
| *** apetrich has quit IRC | 05:38 | |
| *** janki has joined #tripleo | 05:48 | |
| *** apetrich has joined #tripleo | 05:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: BlockDevice - pass command line arguments directly https://review.openstack.org/458727 | 05:52 |
| *** liupe__ has joined #tripleo | 05:53 | |
| *** dsneddon_afk is now known as dsneddon | 05:54 | |
| *** lmiccini has joined #tripleo | 05:55 | |
| *** yolanda has joined #tripleo | 05:58 | |
| *** jbadiapa has joined #tripleo | 05:59 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Move ceilometer wsgi to step 3 https://review.openstack.org/457259 | 06:00 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs https://review.openstack.org/458474 | 06:01 |
| *** brault has quit IRC | 06:01 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Replace references to the 192.0.2 network https://review.openstack.org/455775 | 06:01 |
| *** brault has joined #tripleo | 06:01 | |
| *** brault has quit IRC | 06:05 | |
| *** jaosorior has joined #tripleo | 06:06 | |
| jaosorior | marios: Hey dude, could you +A this commit? https://review.openstack.org/#/c/458596/ it makes the fakeha-caserver gate to pass again | 06:07 |
| jaosorior | bandini: ^^ | 06:07 |
| *** iranzo has joined #tripleo | 06:07 | |
| bandini | lgtm | 06:09 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 06:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 06:10 |
| jaosorior | bandini: thanks dude | 06:10 |
| jaosorior | bandini: hey dude, any idea what would cause this http://logs.openstack.org/03/457603/3/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/94626b6/logs/subnode-2/var/log/barbican/main.txt.gz#_2017-04-20_21_25_41_665 | 06:11 |
| *** liupe__ has quit IRC | 06:14 | |
| *** liupe__ has joined #tripleo | 06:14 | |
| *** rcernin has joined #tripleo | 06:15 | |
| marios | o/ jaosorior bandini beat me to it | 06:15 |
| *** pcaruana has joined #tripleo | 06:18 | |
| *** liupe__ has quit IRC | 06:19 | |
| *** liupe__ has joined #tripleo | 06:19 | |
| *** dparkes has quit IRC | 06:20 | |
| *** aufi has joined #tripleo | 06:25 | |
| bandini | jaosorior: looking | 06:25 |
| jaosorior | marios, bandini thanks dudes | 06:28 |
| *** liupe__ has quit IRC | 06:29 | |
| *** liupe__ has joined #tripleo | 06:29 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add role specific information to the service template https://review.openstack.org/437956 | 06:29 |
| *** yprokule has joined #tripleo | 06:32 | |
| *** jaganathan has joined #tripleo | 06:34 | |
| bandini | jaosorior: so something is shutting down mysql http://logs.openstack.org/03/457603/3/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/94626b6/logs/subnode-2/var/log/mysqld.txt.gz#_2017-04-20_20_24_24 at the time of failure | 06:35 |
| * bandini still looking | 06:35 | |
| bandini | jaosorior: nevermind I might be blind with the time | 06:36 |
| jaosorior | that was one minute before | 06:36 |
| *** chem_gone is now known as chem | 06:36 | |
| *** tobias_fiberdata has quit IRC | 06:36 | |
| *** chem has quit IRC | 06:36 | |
| *** chem has joined #tripleo | 06:36 | |
| bandini | well but it takes time for galera to come up | 06:36 |
| bandini | but it's more like an hour before | 06:37 |
| bandini | unless one log has some timezone skew in it and the other one does not | 06:37 |
| *** udesale__ has joined #tripleo | 06:37 | |
| *** tobias_fiberdata has joined #tripleo | 06:38 | |
| jaosorior | oh shit, that's write | 06:38 |
| jaosorior | it's like an hour | 06:38 |
| jaosorior | well, damn | 06:38 |
| jaosorior | maybe barbican just sucks at reconnecting | 06:38 |
| *** udesale has quit IRC | 06:40 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Add composable role support for NetApp Cinder back end https://review.openstack.org/456264 | 06:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: SSHD Service extensions https://review.openstack.org/444622 | 06:42 |
| *** janki has quit IRC | 06:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Add ovb-fakeha-caserver (TLS everywhere) job to cistatus https://review.openstack.org/458468 | 06:50 |
| *** janki has joined #tripleo | 06:50 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/newton: Add special case upgrade from openvswitch 2.5.0-14 https://review.openstack.org/458737 | 06:50 |
| jaosorior | any reviews on this backport https://review.openstack.org/#/c/458005/ ? | 06:51 |
| marios | chem: stable/newton https://review.openstack.org/458737 :/ had an email about this on a bz | 06:51 |
| *** gbarros has joined #tripleo | 06:53 | |
| *** chlong has quit IRC | 06:56 | |
| bandini | jaosorior: looking | 06:57 |
| *** dparkes has joined #tripleo | 06:59 | |
| *** liupe__ has quit IRC | 06:59 | |
| jaosorior | bandini: hey, thanks dude | 06:59 |
| *** liupe__ has joined #tripleo | 06:59 | |
| bandini | has anyone hit "Error: Swift temporary URLs require a shared secret to be created. You must provide "swift_temp_url_key" as a config option." on the undercloud when deploying? | 06:59 |
| bandini | I keep getting this no matter which repos I use | 06:59 |
| bandini | marios: do you need reviews on that backport? | 07:00 |
| marios | bandini: yes please | 07:01 |
| marios | bandini: haven't come across that swift thing fwiw | 07:01 |
| bandini | "the ghastly openvswitch update saga" | 07:02 |
| marios | bandini: honestly, it completely ruined my mojo of a friday morning to have to do that | 07:02 |
| bandini | marios: nothing can break us down on a Friday! ;) | 07:02 |
| marios | relevant context bandini https://bugzilla.redhat.com/show_bug.cgi?id=1416088#c5 fwiW | 07:03 |
| openstack | bugzilla.redhat.com bug 1416088 in openvswitch "Rebase openvswitch to 2.5.0.23 in OSP 10" [Medium,New] - Assigned to mburns | 07:03 |
| * bandini reads | 07:03 | |
| *** jlinkes has joined #tripleo | 07:03 | |
| chem | marios: what about special case upgrade from openvswitch 2.5.0-14 ? | 07:04 |
| chem | marios: hi by the way :) | 07:04 |
| bandini | it's needed for osp10/newton I believe | 07:04 |
| bandini | chem: morning sir ;) | 07:04 |
| marios | chem: o/ did you hear, ovs hates us | 07:04 |
| *** ckyriakidou has joined #tripleo | 07:05 | |
| marios | chem: nothing major actually, just looks like they are gearing up for ovs 2.6 in 10+ so i went looking. we landed the *removal* of the workaround on newton, but didn't add the 'new and improved' on with --notriggerun | 07:05 |
| marios | chem: assaf emailed me about it and just posted the backport | 07:05 |
| marios | chem: would appreciate eyes at https://review.openstack.org/#/c/458737/ as you know it is a beautiful review ful of bash and heredoc | 07:06 |
| chem | marios: looking ... | 07:08 |
| *** liupe__ has quit IRC | 07:09 | |
| *** mhenkel_ has joined #tripleo | 07:09 | |
| *** liupe__ has joined #tripleo | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 07:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 07:10 |
| *** ffiore has joined #tripleo | 07:10 | |
| *** tesseract has joined #tripleo | 07:11 | |
| bandini | marios: what's "declare -f special_case_ovs_upgrade_if_needed > $UPGRADE_SCRIPT" for in the ceph upgrade script? | 07:11 |
| bandini | it's probably just my lack of bash-fu I guess | 07:11 |
| *** florianf has joined #tripleo | 07:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Update Dell EMC Cinder back end services https://review.openstack.org/456382 | 07:14 |
| bandini | marios: ah nm. got it | 07:14 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Make DockerImageUploader work with missing tags https://review.openstack.org/458743 | 07:14 |
| *** saibarspeis has joined #tripleo | 07:15 | |
| openstackgerrit | Martin André proposed openstack/tripleo-quickstart-extras master: Generate the list of required containers from t-h-t https://review.openstack.org/458575 | 07:15 |
| *** masco has joined #tripleo | 07:17 | |
| bandini | marios, chem: I +Aed that backport. if you did not want that yet for some reason, speak up :) | 07:17 |
| marios | bandini: thanks yeah we need that | 07:18 |
| marios | chem: bandini right now there is *no* workaround for stable/newton ovs (we were told we didn't need one) | 07:18 |
| *** liupe__ has quit IRC | 07:18 | |
| *** ckyriakidou has quit IRC | 07:18 | |
| bandini | got it let | 07:19 |
| bandini | s let the jenkins gods do their job then | 07:19 |
| marios | thanks bandini | 07:19 |
| *** jprovazn has joined #tripleo | 07:21 | |
| *** udesale has joined #tripleo | 07:24 | |
| *** shardy has joined #tripleo | 07:24 | |
| *** udesale__ has quit IRC | 07:26 | |
| marios | chem: lets try a recheck @ https://review.openstack.org/#/c/458530/2 ovb-ha failed on 'not enough hosts' | 07:29 |
| *** cylopez has joined #tripleo | 07:30 | |
| *** hjensas has joined #tripleo | 07:34 | |
| *** hjensas has quit IRC | 07:34 | |
| *** hjensas has joined #tripleo | 07:34 | |
| chem | marios: ack, but I would say that it's unrelated :) | 07:36 |
| *** paramite has joined #tripleo | 07:37 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** bogdando has joined #tripleo | 07:41 | |
| *** jpich has joined #tripleo | 07:43 | |
| *** leanderthal|afk is now known as leanderthal | 07:47 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: glance: deploy services with Keystone v3 endpoints https://review.openstack.org/442798 | 07:48 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Update fakeha net-iso configuration https://review.openstack.org/458596 | 07:49 |
| *** janki has quit IRC | 07:53 | |
| jaosorior | mandre: could you check this out https://review.openstack.org/#/c/458026/1 ? | 07:53 |
| mandre | jaosorior: looking | 07:54 |
| mandre | jaosorior: I wonder if we really need to make a conditional for internal_tls_enabled | 07:56 |
| mandre | jaosorior: wouldn't bind mounting the directories all the time work? | 07:56 |
| jaosorior | mandre: cause those directories don't exist if internal_tls_enabled is false | 07:57 |
| *** pmannidi has quit IRC | 07:57 | |
| mandre | jaosorior: I don't think it's a problem for docker, AFAIK it will create them | 07:57 |
| jaosorior | uhm | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #tripleo | 08:00 | |
| jaosorior | mandre: just not too keen on bind mounting empty directories | 08:00 |
| *** apetrich has quit IRC | 08:01 | |
| *** apetrich has joined #tripleo | 08:02 | |
| mandre | jaosorior: I understand... in my mind that it would simplify the templates quite a bit and wouldn't necessitate modification for each new service that runs on top of apache | 08:04 |
| mandre | I don't think mounting empty dirs is necessarily bad if it simplifies the code | 08:05 |
| mandre | anyways, I'm fine merging your patch as it | 08:06 |
| jaosorior | mandre: I think it can merge. Lets bring it up with jistr and other folks as well to get more input, and if necessary, I'll submit another patch cleaning those bits up | 08:06 |
| mandre | jaosorior: sounds good | 08:07 |
| *** lucas-afk is now known as lucasagomes | 08:07 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 08:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 08:10 |
| *** jerrygb_ has joined #tripleo | 08:11 | |
| *** janki has joined #tripleo | 08:11 | |
| *** jerrygb has quit IRC | 08:11 | |
| *** derekh has joined #tripleo | 08:12 | |
| *** rcernin has quit IRC | 08:13 | |
| *** assassin is now known as ubijtsa | 08:15 | |
| *** mcornea has joined #tripleo | 08:16 | |
| *** jerrygb_ has quit IRC | 08:21 | |
| *** jerrygb has joined #tripleo | 08:22 | |
| *** karimb has joined #tripleo | 08:23 | |
| shardy | chem: Hey FYI see https://review.openstack.org/#/c/458510/ | 08:24 |
| shardy | chem: looks like jaosorior beat you to it adding a str_concat function to heat ;) | 08:24 |
| *** suuuper has joined #tripleo | 08:24 | |
| *** zoli|gone is now known as zoli | 08:24 | |
| chem | shardy: well ... as long as it's there | 08:25 |
| *** cmyster has quit IRC | 08:26 | |
| *** cmyster has joined #tripleo | 08:26 | |
| *** brault has joined #tripleo | 08:27 | |
| shardy | chem: cool, I just wanted to make you aware so we don't duplicate effort | 08:27 |
| chem | shardy: thanks | 08:27 |
| shardy | chem: there are probably a few more functions we could add where we're using yaql atm, if you're interested we could start an etherpad with the yaql examples and try to show native heat alternatives | 08:27 |
| chem | shardy: anyway it was nowhere near the top of my todo list unfortunatly :) | 08:28 |
| *** rcernin has joined #tripleo | 08:28 | |
| shardy | chem: ack, Ok no worries | 08:28 |
| chem | shardy: but yes, it would bu super cool to start something like this, like some fresh air | 08:28 |
| *** jpich has quit IRC | 08:29 | |
| chem | s/bu/be/ | 08:29 |
| *** jpich has joined #tripleo | 08:30 | |
| *** shardy has quit IRC | 08:34 | |
| *** shardy has joined #tripleo | 08:35 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 08:36 |
| *** social has joined #tripleo | 08:42 | |
| jistr | i'm getting this error when trying to do an `openstack overcloud deploy` to update existing overcloud | 08:46 |
| jistr | 2017-04-20 18:34:51 | Unable to create plan. A container with the name overcloud already exists. | 08:46 |
| jistr | it rings a bell vaguely, but i thought it was long fixed | 08:46 |
| jistr | if anyone remembers any additional info around this i'd be grateful for any hints | 08:47 |
| jistr | fwiw here's the log | 08:47 |
| jistr | http://logs.openstack.org/84/450784/17/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4c05d45/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-04-20_18_34_51 | 08:47 |
| shardy | jistr: is the unable to create plan line just a warning? | 08:49 |
| shardy | jistr: it looks to me like the actual error is tripleo.validations.v1.check_pre_deployment_validations? | 08:49 |
| shardy | and that it seems doesn't allow for multinode as it expects the nodes to exist | 08:50 |
| jistr | shardy: that's what i thought too at first, but see a job where this passed through that step fine: | 08:50 |
| jistr | http://logs.openstack.org/84/450784/15/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/823dfd7/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-04-19_10_26_13 | 08:50 |
| *** afazekas has quit IRC | 08:50 | |
| jistr | it seems it's the contrary: the validations errors are non-fatal, but the plan update issue is | 08:50 |
| shardy | https://github.com/openstack/tripleo-common/commit/b69b950a0723cda6a812ecef3d12e7b83f450864 | 08:51 |
| shardy | jistr: Hmm, yeah but the way we do the validations has changed recently I think | 08:51 |
| jistr | we run the deploy with --validation-errors-nonfatal | 08:51 |
| * jistr looks at the link | 08:51 | |
| shardy | https://review.openstack.org/#/c/445914/ is the patch, but it did pass some multinode jobs | 08:52 |
| shardy | Hmm | 08:52 |
| jistr | yea... another point of info -- the upgrade job started failing on this just yesterday. On Wednesday 19th the job got further than this, it actually started the overcloud stack update. | 08:53 |
| *** fzdarsky_ is now known as fzdarsky|afk | 08:55 | |
| jistr | i'm even wondering where does the error message come from | 08:55 |
| jistr | i found https://github.com/openstack/tripleo-common/blob/6dfd35f95375c671af41558e0a74c07cdbb7dc4a/workbooks/plan_management.yaml#L82 | 08:55 |
| jistr | but that's not the same | 08:55 |
| *** jrist has quit IRC | 08:57 | |
| jistr | ah here https://github.com/openstack/tripleo-common/blob/6dfd35f95375c671af41558e0a74c07cdbb7dc4a/tripleo_common/actions/plan.py#L119 | 08:59 |
| *** salmankhan has joined #tripleo | 08:59 | |
| *** stendulker has joined #tripleo | 08:59 | |
| *** tobias-fiberdata has joined #tripleo | 08:59 | |
| *** pgadiya has quit IRC | 08:59 | |
| *** hewbrocca_afk is now known as hewbrocca | 08:59 | |
| *** tobias_fiberdata has quit IRC | 09:01 | |
| *** salmankhan has quit IRC | 09:01 | |
| *** salmankhan has joined #tripleo | 09:02 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 09:03 |
| *** pgadiya has joined #tripleo | 09:04 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Run token flush cron job hourly by default https://review.openstack.org/458005 | 09:06 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 09:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 09:10 |
| *** tosky has joined #tripleo | 09:12 | |
| *** brault has quit IRC | 09:14 | |
| *** brault has joined #tripleo | 09:14 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure libvirt when not needed https://review.openstack.org/458414 | 09:16 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: containers: TLS in the internal network for telemetry services https://review.openstack.org/458026 | 09:18 |
| *** brault has quit IRC | 09:19 | |
| *** flepied has quit IRC | 09:22 | |
| *** limao has quit IRC | 09:23 | |
| *** limao has joined #tripleo | 09:24 | |
| *** flepied has joined #tripleo | 09:24 | |
| *** pgadiya has quit IRC | 09:28 | |
| *** limao has quit IRC | 09:28 | |
| jistr | shardy: hmm this looks like a very likely candidate https://review.openstack.org/#/c/456889 | 09:31 |
| jistr | shardy: it may be that none of our jobs caught it because they don't upgrade undercloud and overcloud at the same time | 09:31 |
| jaosorior | jistr: ?? | 09:31 |
| jistr | jaosorior: what ?? :) | 09:32 |
| *** anshul has joined #tripleo | 09:32 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/newton: Add special case upgrade from openvswitch 2.5.0-14 https://review.openstack.org/458737 | 09:32 |
| *** anshul has quit IRC | 09:33 | |
| *** flepied has quit IRC | 09:34 | |
| *** anshul has joined #tripleo | 09:35 | |
| jaosorior | jistr: nevermind, read up now | 09:35 |
| jaosorior | jistr: hadn't noticed that you were asking about a bug you're seeing | 09:36 |
| jistr | hehe ok :) | 09:36 |
| jistr | yea i'm still chasing it | 09:36 |
| *** brault has joined #tripleo | 09:37 | |
| *** ramishra has quit IRC | 09:37 | |
| openstackgerrit | Jiri Stransky proposed openstack/instack-undercloud master: Revert "Migrate plans from Mistral to Swift" https://review.openstack.org/458792 | 09:38 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI https://review.openstack.org/450784 | 09:40 |
| *** pgadiya has joined #tripleo | 09:41 | |
| jistr | damn | 09:45 |
| jistr | i just realized that the revert cannot be tested by CI anyway | 09:45 |
| jistr | as the undercloud upgrade will pull the latest delorean, which might be already broken | 09:45 |
| *** ramishra has joined #tripleo | 09:45 | |
| jistr | i'll also disable undercloud upgrade in the container upgrade CI job then | 09:47 |
| jistr | /o\ | 09:47 |
| *** dixiaoli has joined #tripleo | 09:50 | |
| *** afazekas has joined #tripleo | 09:50 | |
| *** zoli is now known as zoli|lunch | 09:52 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-quickstart master: Add featureset011 - upgrade from BM to containerized https://review.openstack.org/450295 | 09:54 |
| shardy | jistr: Is it failing on the initial deploy or the upgrade? | 09:55 |
| jistr | shardy: just the upgrade | 09:55 |
| shardy | jistr: Depends-On to a revert should work then, as we overlay the tripleo-ci repo? | 09:55 |
| jistr | i'm suspecting that the migration to swift does something that makes it impossible to do a stack update | 09:55 |
| shardy | the problem is if you want to test a revert to the stuff we use before upgrade IME | 09:55 |
| shardy | currently there's no way to do that, because cross-branch Depends-On doesn't work | 09:56 |
| jistr | shardy: there's no cross branch, it's just master->master still | 09:56 |
| jistr | shardy: but i'm running both undercloud and overcloud upgrade | 09:56 |
| shardy | jistr: Ok, then Depends-On should work I think | 09:57 |
| jistr | so what i thought would be an issue is that zuul sets up the correct instack-undercloud with depends-on | 09:57 |
| jistr | and then i re-setup the repos and run undercloud upgrade | 09:57 |
| jistr | so it will pull latest instack-undercloud regardless of what zuul set up earlier | 09:57 |
| shardy | jistr: for the non-container upgrade job, we enable the tripleo-ci repo before doing the upgrade, along with the other repos we want to upgrade to | 09:58 |
| shardy | and ensure tripleo-ci takes priority | 09:58 |
| shardy | jistr: also, for non-container we don't upgrade the undercloud, we just deploy master undercloud, ocata overcloud, then upgrade the overcloud | 09:58 |
| shardy | which saves some time | 09:58 |
| *** dixiaoli has quit IRC | 09:59 | |
| jistr | shardy: but is that for OOOQ jobs too? re CI taking priority? | 10:00 |
| openstackgerrit | Joe Talerico proposed openstack/tripleo-heat-templates master: Adding Tuned Service https://review.openstack.org/454123 | 10:00 |
| jistr | here's how we exchange the repos currently | 10:00 |
| jistr | https://review.openstack.org/#/c/448576/20/roles/overcloud-upgrade/templates/upgrade-repo-tripleo-ci.sh.j2 | 10:00 |
| shardy | jistr: I don't know re oooq, but that is how it should work IMO | 10:00 |
| jistr | right yea | 10:00 |
| *** zoli|lunch is now known as zoli | 10:00 | |
| jistr | if there's something not named delorean* with higher prio, then yea it would persist and take priority i think | 10:01 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-common master: WIP kolla template overrides for pacemaker HA containers https://review.openstack.org/456174 | 10:01 |
| jistr | here are the undercloud upgrade repos | 10:02 |
| jistr | http://logs.openstack.org/84/450784/17/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4c05d45/logs/undercloud/etc/yum.repos.d/ | 10:02 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Unpin ansible-lint https://review.openstack.org/456564 | 10:02 |
| jistr | so the gating.repo could be it perhaps | 10:02 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart master: Implement fact caching https://review.openstack.org/448478 | 10:03 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix ansible-lint.sh to check playbooks https://review.openstack.org/446525 | 10:03 |
| shardy | jistr: Hmm, no idea what that is | 10:03 |
| jistr | shardy: anyway, i can disable the undercloud upgrade in the containers job anyway, to save time as well | 10:03 |
| shardy | it's always been called delorean-ci, so quickstart should just call it that IMO | 10:03 |
| *** pgadiya has quit IRC | 10:04 | |
| jistr | but the fact that we generally don't test uc + oc upgrade together means that we miss these bugs | 10:04 |
| shardy | jistr: maybe, but this isn't because of the undercloud upgrade is it? | 10:04 |
| jistr | shardy: i think it likely is | 10:04 |
| shardy | jistr: Hmm, OK - well until now we've viewed separating the two upgrades as a reasonable compromise given the infra timeout constraints | 10:05 |
| jistr | shardy: the topic and merge time of this commit both match in terms of what could potentially be the cause https://review.openstack.org/#/c/456889/ | 10:05 |
| *** ccamacho has joined #tripleo | 10:07 | |
| shardy | jistr: aha, yeah I see | 10:07 |
| shardy | jistr: FWIW I didn't expect us to do that, I thought we were going to just put new overcloud data in to swift, and rewrite plan-environment when the plan next gets updated | 10:08 |
| shardy | I guess a forcible undercloud migration is OK, but reverting it should be low impact if we think it's broken things | 10:08 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 10:10 |
| *** brault has quit IRC | 10:10 | |
| jistr | shardy: not sure if we can reliably prove or disprove that via CI right now though (probably depending on how the repository setup and upgrade works there). I'll see how the upgrade goes now that i've disabled undercloud upgrade, and then maybe try re-enabling it. | 10:11 |
| *** gfidente has joined #tripleo | 10:11 | |
| *** gfidente has quit IRC | 10:11 | |
| *** gfidente has joined #tripleo | 10:11 | |
| jistr | i mean i'm not sure if we can prove/disprove what caused the issue | 10:11 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add role specific information to the service template https://review.openstack.org/437956 | 10:14 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Add undercloud-containers job to cistatus https://review.openstack.org/446957 | 10:15 |
| jpich | jistr: I've reviewed a few of Ana's patches around this work and am taking over them while she's away for a few weeks, looking at the logs you pasted earlier now to see if something rings a bell | 10:15 |
| jistr | jpich: awesome, thanks :) | 10:15 |
| jpich | jistr: I recall a conversation between akrivoka and d0ugal around having the migration happen during the undercloud upgrade to avoid... $some_kind_of_problem... but I don't remember which one so I'm gonna do some IRC log digging too :) | 10:16 |
| *** zoli is now known as zoli|lunch | 10:16 | |
| *** cylopez has quit IRC | 10:18 | |
| jistr | jpich: thanks | 10:18 |
| *** pgadiya has joined #tripleo | 10:19 | |
| jistr | jpich: just for tl;dr context -- currently we don't have much of a lead as to why the job started failing this way aside from the time (somewhere during 19th and 20th). And due to what's the error message, the cause *might* be https://review.openstack.org/#/c/456889/ but we don't know that for sure. | 10:19 |
| jpich | jistr: Thank you | 10:21 |
| shardy | jistr: ack, well FWIW I'd try to prove it with a Depends-On to the instack-undercloud revert, hopefully we can get that to work via oooq | 10:23 |
| jistr | yea, for now i've also disabled the UC upgrade completely, just to see if that helps in general, and then we can try to re-enabling it again | 10:26 |
| jistr | also linked good/bad job logs at https://bugs.launchpad.net/tripleo/+bug/1685171 | 10:26 |
| openstack | Launchpad bug 1685171 in tripleo "updating overcloud after undercloud upgrade: Unable to create plan. A container with the name overcloud already exists." [High,In progress] - Assigned to Jiří Stránský (jistr) | 10:26 |
| * jistr lunch, brb | 10:26 | |
| bandini | jistr, shardy: since you mentioned some validations being borked, I just filed https://bugs.launchpad.net/tripleo/+bug/1685177 because I see them non working in my deploy | 10:29 |
| openstack | Launchpad bug 1685177 in tripleo "Since moving some validations to mistral deploy warnings seem to be broken" [Undecided,New] | 10:29 |
| bandini | thrash|g0ne: can you take a look at https://bugs.launchpad.net/tripleo/+bug/1685177 if you have some time? | 10:30 |
| *** brault has joined #tripleo | 10:34 | |
| *** stendulker has quit IRC | 10:34 | |
| *** brault has quit IRC | 10:39 | |
| *** jtomasek_ has joined #tripleo | 10:43 | |
| yolanda | hi, hitting this on an ocata deploy using quickstart | 10:44 |
| yolanda | oslo_db.exception.DBError: (pymysql.err.InternalError) (1061, u"Duplicate key name 'uniq_ports0standard_attr_id'") [SQL: u'ALTER TABLE ports ADD CONSTRAINT uniq_ports0standard_attr_id UNIQUE (standard_attr_id)'] | 10:44 |
| yolanda | is that a known issue? | 10:44 |
| *** salmankhan has quit IRC | 10:52 | |
| *** salmankhan has joined #tripleo | 10:52 | |
| *** udesale has quit IRC | 10:53 | |
| *** pgadiya has quit IRC | 10:54 | |
| *** atoth has joined #tripleo | 10:54 | |
| *** dbecker has quit IRC | 10:55 | |
| *** yamahata has quit IRC | 11:00 | |
| *** abishop has quit IRC | 11:01 | |
| *** pgadiya has joined #tripleo | 11:07 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 11:10 |
| *** zoli|lunch is now known as zoli | 11:15 | |
| *** tvignaud has quit IRC | 11:15 | |
| EmilienM | hello | 11:15 |
| EmilienM | cschwede: can you please look https://review.openstack.org/#/c/458714/ and compare the runtime of upgrade jobs with the ones we had yesterday? | 11:17 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Replace mistral command with OpenStack CLI https://review.openstack.org/458808 | 11:23 |
| *** gkadam has quit IRC | 11:24 | |
| *** lucasagomes is now known as lucas-afk | 11:25 | |
| jaosorior | EmilienM: the barbican issue is pretty funky, I'll need to reproduce it but it's gonna take a bit. | 11:27 |
| EmilienM | jaosorior: cool | 11:30 |
| *** masco has quit IRC | 11:32 | |
| *** pkovar has joined #tripleo | 11:32 | |
| *** shardy is now known as shardy_lunch | 11:32 | |
| yolanda | EmilienM, do you know about that failures deploying undercloud on ocata? | 11:33 |
| yolanda | or jaosorior | 11:33 |
| jaosorior | I don't :( | 11:33 |
| EmilienM | yolanda: what repos do you use.? | 11:34 |
| yolanda | just a quickstart default rdo deploy | 11:34 |
| yolanda | in that case, nothing customized | 11:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 11:36 |
| owalsh | jistr: hola | 11:36 |
| jistr | owalsh: heya i'm just reading your comment ;) | 11:36 |
| owalsh | jistr: I'm probably missing something crucial :-) | 11:37 |
| cschwede | EmilienM: i had a look; gate-tripleo-ci-centos-7-scenario002-multinode-upgrades was much faster, but gate-tripleo-ci-centos-7-multinode-upgrades and gate-tripleo-ci-centos-7-scenario001-multinode-upgrades are slow again | 11:37 |
| cschwede | EmilienM: also, gate-tripleo-ci-centos-7-scenario001-multinode-upgrades would have been cancelled again without the timeout increase | 11:37 |
| owalsh | jistr: but I don't understand why tripleo::profile::base::nova::libvirt_enabled would be true in e.g he nova-api service, unless it's picking up the nova-libvirt hiera | 11:38 |
| jistr | owalsh: indeed we discussed this with mandre earlier today | 11:38 |
| jistr | owalsh: we have an issue with the container puppet things | 11:38 |
| jistr | owalsh: the hiera is global on the host, not per-container | 11:39 |
| *** abishop has joined #tripleo | 11:39 | |
| jistr | so hiera settings from across all services get mangled together | 11:39 |
| owalsh | ahhhh, that's what I was missing ^^^ | 11:39 |
| jistr | further interesting might be that this is actually beneficial in some cases | 11:40 |
| jistr | i think jaosorior had some use case for this with certmonger, IIRC | 11:40 |
| jistr | but it breaks us in other cases, like this libvirt issue | 11:41 |
| owalsh | probably should move the migration bits out of the base nova profile, it's not common to all nova services | 11:41 |
| EmilienM | cschwede: I think it depends on which clouds it runs | 11:41 |
| EmilienM | cschwede: see my last comment on the bug report https://bugs.launchpad.net/tripleo/+bug/1680259 | 11:42 |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 11:42 |
| *** abishop_ has joined #tripleo | 11:42 | |
| jistr | owalsh: hmm i'm not sure about this, but doesn't e.g. nova-api need to have some of the migration settings as well? It's also configuring nova.conf, not just libvirt. | 11:42 |
| *** morazi has joined #tripleo | 11:43 | |
| cschwede | EmilienM: good catch! wondering if that is related to slower network connectivity or slower cpus? | 11:43 |
| *** tobias-fiberdata has quit IRC | 11:43 | |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo master: Enable mistral to run under mod_wsgi https://review.openstack.org/458503 | 11:43 |
| *** abishop has quit IRC | 11:43 | |
| *** thrash|g0ne is now known as thrash | 11:43 | |
| thrash | bandini: sure | 11:44 |
| jistr | owalsh: either way for the short term immediate fix we could probably just set the libvirt_enabled option to false in docker nova-compute, effectively disabling the migration. | 11:45 |
| *** ratailor has quit IRC | 11:45 | |
| *** tvignaud has joined #tripleo | 11:45 | |
| jistr | jpich, shardy_lunch: FWIW https://bugs.launchpad.net/tripleo/+bug/1685171/comments/3 | 11:47 |
| openstack | Launchpad bug 1685171 in tripleo "updating overcloud after undercloud upgrade: Unable to create plan. A container with the name overcloud already exists." [High,In progress] - Assigned to Jiří Stránský (jistr) | 11:47 |
| owalsh | jistr: probably nova-libvirt ^^ but yea... I think disable it. Docker support is the next big thing on my list once the other migration patches land | 11:47 |
| jistr | owalsh: ok great, thanks for the involvement on that patch! i wasn't aware of the things you brought up | 11:48 |
| *** fzdarsky|afk is now known as fzdarsky | 11:48 | |
| owalsh | jistr: and nova-api doens't need any migration settings, it's just the nova-compute nova.conf and the libvirtd.conf | 11:49 |
| EmilienM | cschwede: probably a little mix of both | 11:49 |
| EmilienM | cschwede: I would continue the discussion on infra channel, when pabelanger is up | 11:49 |
| jistr | owalsh: ack, thanks | 11:50 |
| *** dougbtv_ has joined #tripleo | 11:50 | |
| *** dougbtv_ is now known as dougbtv|laptop | 11:50 | |
| owalsh | np | 11:51 |
| *** tobias-fiberdata has joined #tripleo | 11:51 | |
| *** prateek has quit IRC | 11:52 | |
| jistr | owalsh: any opinion whether i should set to false manage_migration, or just libvirt_enabled? | 11:55 |
| *** jpena is now known as jpena|lunch | 11:55 | |
| arxcruz | EmilienM: Thanks for review my patch. If you don't mind can you take a look at https://review.openstack.org/#/c/455210/ too? | 11:56 |
| EmilienM | arxcruz: done | 11:59 |
| arxcruz | EmilienM: thanks! | 11:59 |
| yolanda | maybe it was a glitch, second try went fine | 11:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata https://review.openstack.org/458714 | 11:59 |
| EmilienM | yolanda: good | 12:00 |
| *** salmankhan has quit IRC | 12:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata https://review.openstack.org/458714 | 12:00 |
| EmilienM | cschwede: trying again to see if it's consistent ^ | 12:00 |
| *** brault has joined #tripleo | 12:01 | |
| jistr | owalsh: i'll try just libvirt_enabled. I see that manage_migration is set in nova-compute.yaml as well, and if that comes after nova-libvirt.yaml, it would override its setting. We might need to implement hieradata isolation between puppet containers to get rid of these issues... | 12:01 |
| jistr | (and create new ones... :)) ) | 12:01 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/456030 | 12:02 |
| *** trown|outtypewww is now known as trown | 12:03 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure live migration https://review.openstack.org/458414 | 12:05 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/ocata: Add MigrationSshKey to generated passwords https://review.openstack.org/458671 | 12:06 |
| *** rbowen has joined #tripleo | 12:09 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 12:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 12:10 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel https://review.openstack.org/458077 | 12:11 |
| *** jaosorior is now known as jaosorior_away | 12:12 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: N->O Manual puppet commands have the right modulepath. https://review.openstack.org/458530 | 12:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add defaults for docker puppet tasks https://review.openstack.org/455718 | 12:20 |
| *** lmiccini has quit IRC | 12:22 | |
| jpich | jistr: Thank you for the update, been combing the mistral logs but not finding anything obvious | 12:22 |
| *** hewbrocca has quit IRC | 12:23 | |
| *** markmc` has joined #tripleo | 12:23 | |
| *** shardy_lunch is now known as shardy | 12:25 | |
| *** markmc has quit IRC | 12:25 | |
| *** amoralej has quit IRC | 12:25 | |
| *** jpena|lunch has quit IRC | 12:25 | |
| *** dmanchad has quit IRC | 12:26 | |
| *** lucas-afk is now known as lucasagomes | 12:26 | |
| bandini | thrash: thanks ;) | 12:33 |
| *** lmiccini has joined #tripleo | 12:33 | |
| *** jtomasek_ has quit IRC | 12:34 | |
| *** jchhatbar has joined #tripleo | 12:34 | |
| owalsh | jistr: just commented - I think we will need to set manage_migration=false | 12:35 |
| *** hjensas has quit IRC | 12:37 | |
| *** brault has quit IRC | 12:38 | |
| *** dmanchad has joined #tripleo | 12:38 | |
| *** janki has quit IRC | 12:38 | |
| *** jpena has joined #tripleo | 12:38 | |
| *** markmc` has quit IRC | 12:41 | |
| *** amoralej has joined #tripleo | 12:43 | |
| EmilienM | zaneb: could we try https://review.openstack.org/#/c/422983/ in triploe-ci? (depends-on or check experimental) so we can see if it actually helps | 12:44 |
| *** markmc has joined #tripleo | 12:46 | |
| *** yprokule has quit IRC | 12:48 | |
| *** Goneri has joined #tripleo | 12:49 | |
| *** rlandy has joined #tripleo | 12:50 | |
| *** tvignaud has quit IRC | 12:50 | |
| dmsimard | EmilienM: I don't know what to do about https://review.openstack.org/#/c/448108/, I've rechecked plenty of times and it keeps failing on unrelated things | 12:53 |
| EmilienM | dmsimard: looking | 12:54 |
| EmilienM | dmsimard: +2 | 12:55 |
| *** hewbrocca_afk has joined #tripleo | 12:55 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add ML2 configuration for Bagpipe BGPVPN extension https://review.openstack.org/456153 | 12:55 |
| *** hewbrocca_afk is now known as hewbrocca | 12:55 | |
| *** lblanchard has joined #tripleo | 12:55 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Use -net=host for docker-puppet.py config gen https://review.openstack.org/455794 | 12:56 |
| *** jmelvin has joined #tripleo | 12:56 | |
| *** jrist has joined #tripleo | 12:59 | |
| *** saibarspeis has quit IRC | 12:59 | |
| *** pgadiya has quit IRC | 13:00 | |
| *** lblanchard1 has joined #tripleo | 13:01 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Add resource profile for vmware nsx_v3 https://review.openstack.org/452047 | 13:01 |
| *** dprince has joined #tripleo | 13:02 | |
| *** tvignaud has joined #tripleo | 13:03 | |
| *** salmankhan has joined #tripleo | 13:04 | |
| *** lblanchard has quit IRC | 13:05 | |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/ocata: SSHD Service extensions https://review.openstack.org/458827 | 13:06 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/ocata: Stop SSHD profile clobbering SSH client config https://review.openstack.org/458828 | 13:07 |
| openstackgerrit | Oliver Walsh proposed openstack/puppet-tripleo stable/ocata: Refactor SSHD config to allow both SSHD options and banner/motd to be set https://review.openstack.org/458830 | 13:08 |
| openstackgerrit | Damien Ciabrini proposed openstack/puppet-tripleo master: WIP containerized pacemaker haproxy https://review.openstack.org/458832 | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 13:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 13:10 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Cover gnocchi api step 4 and 5 https://review.openstack.org/456376 | 13:10 |
| openstackgerrit | Oliver Walsh proposed openstack/tripleo-heat-templates stable/ocata: SSHD Service extensions https://review.openstack.org/458836 | 13:14 |
| *** thrash is now known as thrash|biab | 13:14 | |
| *** rbrady has joined #tripleo | 13:17 | |
| *** Goneri has quit IRC | 13:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/456030 | 13:17 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 13:18 |
| jistr | owalsh: ok thanks, i'll see how far we get with the current CI run in progress and then i'll update the patch | 13:19 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart master: Add supplemental node provisioner https://review.openstack.org/451523 | 13:22 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart master: Update quickstart-extras to deploy FreeIPA https://review.openstack.org/453223 | 13:22 |
| *** dsariel has joined #tripleo | 13:23 | |
| *** dhill__ has joined #tripleo | 13:23 | |
| *** tzumainn has joined #tripleo | 13:24 | |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart-extras master: Add role to deploy FreeIPA https://review.openstack.org/436198 | 13:24 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure live migration https://review.openstack.org/458414 | 13:25 |
| owalsh | jistr: should pass CI for now I would think | 13:25 |
| owalsh | jistr: but I'm pretty sure I'll break it soon :-) | 13:26 |
| jistr | owalsh: i realized i mentioned libvirt_enabled in commit message but set manage_migration in the code, but only on nova-libvirt so i should set it also in nova-compute (last one wins, and we don't want to take chances which one it will be). So i updated the patch right now ^^ and i'll retrigger CI. | 13:26 |
| skramaja | jtomasek: hey, as per the comments, looks like you are ok with the derived params spec as such, barring the implementation details we disucssion on the patch, we can continue discussing on the code review, if agree, can you accept the spec? | 13:27 |
| EmilienM | flaper87, shardy, bnemec: I hope you can join https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management | 13:28 |
| owalsh | jistr: cool :-) LGTM, maybe open an LP for migration support in docker? | 13:28 |
| rook | trozet: hey - did you want to review the Tuned work? | 13:29 |
| rook | s/docker/moby | 13:29 |
| *** amoralej is now known as amoralej|lunch | 13:29 | |
| *** liverpooler has joined #tripleo | 13:31 | |
| jistr | owalsh: yup https://bugs.launchpad.net/tripleo/+bug/1685219 | 13:32 |
| openstack | Launchpad bug 1685219 in tripleo "Nova migration support in dockerized deployments" [Medium,Triaged] | 13:32 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Combine writing docker-puppet json with host_prep_tasks https://review.openstack.org/456307 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Use ansible to write the docker json configs https://review.openstack.org/454816 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Move DockerConfigJsonStartupData to HostPrepConfig https://review.openstack.org/456310 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Move KollaJsonConfig to HostPrepConfig https://review.openstack.org/456311 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Move create-config-dir steps to HostPrepConfig ansible tasks https://review.openstack.org/456308 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Move GenerateConfig to HostPrepConfig https://review.openstack.org/456309 | 13:33 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Convert DockerPuppetTasks to run via ansible https://review.openstack.org/456254 | 13:33 |
| *** newmember has quit IRC | 13:33 | |
| *** flepied has joined #tripleo | 13:34 | |
| honza | jtomasek: florianf: would you be really sad if I disabled colors in redux-logger? | 13:36 |
| *** psahoo has quit IRC | 13:36 | |
| rnoriega | shardy, ping | 13:37 |
| shardy | rnoriega: Hi | 13:38 |
| therve | dtrainor, You around? I have a question about the undercloud SSL certificate, if you know about it | 13:38 |
| rnoriega | shardy, hi! reading your blogpost about custom roleS: http://hardysteven.blogspot.com.es/2016/10/tripleo-composablecustom-roles.html | 13:38 |
| rnoriega | shardy, I have one small question. Do you have 1 min? | 13:39 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Allow to configure haproxy daemon's status https://review.openstack.org/458025 | 13:39 |
| rnoriega | shardy, awesome blog btw, very educative. | 13:39 |
| shardy | rnoriega: sure, and thanks! | 13:39 |
| rnoriega | shardy, how do you assign, for example, the NtpRole to a introspected node?? | 13:40 |
| florianf | honza: Yes, very sad. Can you add animations or emoji instead? | 13:40 |
| honza | <3 | 13:41 |
| shardy | rnoriega: you tag the nodes with a particular role, which is then selected by the flavor, e.g "controller" "compute" or whatever | 13:41 |
| *** brault has joined #tripleo | 13:41 | |
| rnoriega | shardy, yes, but for instance, the role name is Controller...but the profile (flavor) is called Control | 13:41 |
| rnoriega | shardy, how do you map flavor-role? | 13:41 |
| shardy | rnoriega: we don't, all roles default to the baremetal flavor | 13:42 |
| shardy | rnoriega: but you can pass an environment file with MyRoleFlavor: myroleflavor | 13:42 |
| *** kjw3 has joined #tripleo | 13:42 | |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/role.role.j2.yaml#L6 | 13:43 |
| shardy | rnoriega: actually OvercloudMyRoleFlavor: myroleflavor | 13:43 |
| shardy | the ^Overcloud thing is a historical inconsistency we couldn't remove | 13:43 |
| *** chlong has joined #tripleo | 13:43 | |
| *** shardy is now known as shardy_afk | 13:44 | |
| rnoriega | shardy, so you say I'd have to add a OvercloudNtpRoleFlavor section and the default would be a new flavor I create in the undercloud. | 13:45 |
| rnoriega | shardy_afk, ok...I think I get it | 13:45 |
| *** thrash|biab is now known as thrash | 13:49 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 13:50 |
| *** jchhatbar has quit IRC | 13:50 | |
| *** gbarros has quit IRC | 13:54 | |
| rnoriega | shardy_afk, nope, I don't get it... coz Overcloud{{role}}Flavor: ends up in a OvercloudControlFlavor in the compute-role.yaml | 13:54 |
| rnoriega | shardy_afk, and not a OvercloudControllerFlavor which is the name of the role... | 13:55 |
| *** ratailor has joined #tripleo | 13:56 | |
| *** prateek has joined #tripleo | 13:57 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Configure migration SSH tunnel https://review.openstack.org/458674 | 13:58 |
| *** prateek has quit IRC | 13:58 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Adding subunit files to artcl_collect_list https://review.openstack.org/455219 | 13:58 |
| openstackgerrit | Merged openstack/tripleo-ui master: Add Toolbar component and filters state https://review.openstack.org/452175 | 13:58 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Adding testrepository.subunit to artcl_collect_list https://review.openstack.org/455210 | 13:58 |
| dtrainor | therve, sure, what's up? | 14:00 |
| jtomasek | honza: ouch colors! | 14:01 |
| *** ykarel has quit IRC | 14:01 | |
| jtomasek | honza: what is the problem you're hitting? is the output mangled? | 14:01 |
| honza | jtomasek: color control characters look out of place in a server-side log file | 14:02 |
| jtomasek | skramaja: ok, will do | 14:02 |
| skramaja | thanks jtomasek | 14:02 |
| florianf | honza: that sounds like a good enough reason to drop colors IMO. | 14:03 |
| honza | jtomasek: florianf: e.g. "%c prev state,color: #9E9E9E; font-weight: bold,[object Object]" | 14:03 |
| jpich | jistr: I think I understand what is happening, and believe you got the right patch for the revert... However, it looks like it should be solvable by commenting out one line as opposed to reverting the whole thing | 14:04 |
| therve | dtrainor, I'd like to modify the cert, to change subject alt name. Do you have an idea how I can do it? | 14:04 |
| *** lblanchard1 has quit IRC | 14:04 | |
| *** colonwq has joined #tripleo | 14:05 | |
| jaosorior_away | therve: which cert? | 14:05 |
| *** jaosorior_away is now known as jaosorior_mtg | 14:05 | |
| therve | jaosorior_mtg, The undercloud pem file | 14:05 |
| jaosorior_mtg | you would need this commit https://review.openstack.org/#/c/457619/ | 14:06 |
| jistr | jpich: sounds great, i'd very much prefer a fix than a revert. I think we're not in an emergency as it looks like for the time being, given that the upgrade in the job i'm implementing is master->master anyway, i should be able to just temporarily disable it (since it shouldn't effectively do anything anyway :) ). | 14:06 |
| jtomasek | honza: I am fine with disabling colors, it is not a big deal | 14:06 |
| jaosorior_mtg | therve: and then support would be needed to be added for the undercloud | 14:06 |
| therve | jaosorior_mtg, Yeah I mostly just want to make a local hack for now | 14:06 |
| jistr | jpich: the undercloud upgrade would become important when we start testing actual ocata->master upgrade, if we're testing both uc + oc upgrades in one job | 14:06 |
| *** dsariel has quit IRC | 14:07 | |
| jistr | jpich: but we're not at that point yet :) So we don't have to hurry with any reverting there. | 14:07 |
| jaosorior_mtg | you could try to resubmit the cert via certmonger | 14:07 |
| *** amoralej|lunch is now known as amoralej | 14:08 | |
| dmsimard | any tripleo-ci core want to look at https://review.openstack.org/#/c/448108/ ? It's a trivial outstanding change for a month now. | 14:09 |
| therve | jaosorior_mtg, Yeah I think I'm looking for the right getcert call, but I can't find it for now | 14:09 |
| jpich | jistr: Cool, thank you! Yes I would prefer to fix it soon if we can too. Would you prefer I submit a separate patch with that small change, or modify the patch you have up? | 14:09 |
| jaosorior_mtg | therve: getcert list -> will give you the nickname | 14:10 |
| jaosorior_mtg | therve: and you're looking for getcert resubmit | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 14:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 14:10 |
| jistr | jpich: i think a separate one would be better perhaps?. I'll -2-A the revert for now. | 14:10 |
| jpich | jistr: Considering the difficulties around testing the revert this morning, I guess there's no simple way to test and confirm it does work before the patch gets merged...? | 14:10 |
| therve | jaosorior_mtg, Yeah, I get "State NEED_GUIDANCE, stuck: yes." | 14:10 |
| jaosorior_mtg | ah shit | 14:10 |
| jpich | jistr: Ok, will prepare that now | 14:10 |
| jaosorior_mtg | therve: how did you call it? | 14:11 |
| therve | jaosorior_mtg, getcert resubmit -i "undercloud-haproxy-public-cert" -w -v -U id-kp-clientAuth -A 192.168.24.2 -U id-kp-serverAuth | 14:11 |
| skramaja | shardy_afk: EmilienM - all open ends are clarified for the spec - https://review.openstack.org/#/c/423304/, can we merge it? - fyi: fultonj | 14:11 |
| EmilienM | skramaja: done | 14:12 |
| skramaja | thanks EmilienM | 14:12 |
| jaosorior_mtg | therve: funky, that looks about right | 14:13 |
| jistr | jpich: re testing before merging i'm not 100% sure -- it depends on how the zuul patch-in-question repos work with OOOQ, which i haven't dug through yet. We might try testing it then and hope that the repos will be correct. Or we could just merge it too, if it passes all the standard jobs. | 14:13 |
| jaosorior_mtg | therve: maybe you gotta specify the key and cert file again? | 14:13 |
| jistr | jpich: we can sync about that further. Either way thanks a lot for looking into the bug :) | 14:14 |
| *** dixiaoli has joined #tripleo | 14:15 | |
| therve | jaosorior_mtg, OK I'll try | 14:15 |
| openstackgerrit | Merged openstack/tripleo-specs master: Deriving TripleO Parameters https://review.openstack.org/423304 | 14:16 |
| *** jcoufal has joined #tripleo | 14:16 | |
| *** shardy_afk is now known as shardy | 14:17 | |
| numans | it would be really nice if some one could review the OVN patches - https://review.openstack.org/#/c/372274/ and https://review.openstack.org/#/c/436767/ | 14:17 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Test I71f8aa431a60457326167b8c82adc03ca750eda6 https://review.openstack.org/458858 | 14:18 |
| shardy | rnoriega: yeah we had some inconsistent naming for historical reasons, and we couldn't just change the parameters to match the role name | 14:18 |
| jpich | jistr: No problems! :) I think the change will be minor enough to just "merge & test" but we'll talk then, I could just be reckless | 14:19 |
| fultonj | Thanks EmilienM | 14:19 |
| shardy | rnoriega: for all custom roles we use the role.role.j2.yaml though, which is Overcloud$RoleFlavor | 14:19 |
| EmilienM | fultonj: thanks for pushing it :) | 14:19 |
| *** dixiaoli has quit IRC | 14:19 | |
| rnoriega | shardy, so If I create a flavor named MyRole...it'll work | 14:19 |
| shardy | rnoriega: yes create a flavor called myrole, then add OvercloudMyRoleFlavor: myrole to an environment file | 14:20 |
| shardy | rnoriega: then set the capabilities in the myrole flavor to match whatever you tag the nodes with | 14:20 |
| *** bnemec is now known as beekneemech | 14:20 | |
| shardy | e.g myrole or whatever tag/profile you want | 14:20 |
| *** yamahata has joined #tripleo | 14:21 | |
| rnoriega | shardy, good! got it! thanks!! | 14:21 |
| shardy | rnoriega: np! | 14:21 |
| zaneb | EmilienM: ControllerServiceChain 101.0 | 14:24 |
| zaneb | http://logs.openstack.org/83/422983/20/experimental/gate-tripleo-ci-centos-7-nonha-multinode-oooq/577f861/logs/postci.txt.gz | 14:24 |
| *** rcernin has quit IRC | 14:24 | |
| *** brault has quit IRC | 14:25 | |
| EmilienM | zaneb: sounds better, no? | 14:25 |
| zaneb | EmilienM: yeah, down from like 500 | 14:25 |
| EmilienM | :) | 14:26 |
| EmilienM | nice | 14:26 |
| zaneb | ControllerServiceChain 547.0 in https://bugs.launchpad.net/tripleo/+bug/1684272/comments/2 | 14:26 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 14:26 |
| EmilienM | zaneb: should we cancel the revert? | 14:26 |
| zaneb | I think it's ok to revert | 14:26 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP THT-part of galera ha container https://review.openstack.org/455348 | 14:27 |
| *** brault has joined #tripleo | 14:28 | |
| therve | jaosorior_mtg, http://paste.openstack.org/show/607472/ does that look correct? | 14:29 |
| *** atheurer has joined #tripleo | 14:29 | |
| jistr | jpich: read your comment with analysis of the problem -- good catch! So the hope is that we'd just be able to stop deleting the mistral environment for now? | 14:29 |
| jistr | so that the original code paths in tripleo-common work as expected? | 14:29 |
| *** paramite has quit IRC | 14:29 | |
| jpich | jistr: Yes! | 14:29 |
| jistr | sounds good | 14:29 |
| *** trozet has quit IRC | 14:29 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP THT-part of rabbitmq HA docker container https://review.openstack.org/452889 | 14:30 |
| openstackgerrit | Julie Pichon proposed openstack/instack-undercloud master: Plan format migration: keep the Mistral env around https://review.openstack.org/458859 | 14:30 |
| *** skramaja has quit IRC | 14:30 | |
| jaosorior_mtg | therve: well, that looks well, although that's a request and not a resubmit | 14:30 |
| therve | jaosorior_mtg, Yeah I did a stop-tracking beforehand | 14:30 |
| *** michapma_dsk has joined #tripleo | 14:30 | |
| jaosorior_mtg | therve: oh, alright | 14:30 |
| therve | jaosorior_mtg, What's that supposed to generate though? :) | 14:31 |
| jaosorior_mtg | therve: you might wanna clean up the cert file as well. | 14:32 |
| jaosorior_mtg | therve: it generates the cert file that you give with the -f flag | 14:32 |
| therve | jaosorior_mtg, So remove it beforehand? | 14:32 |
| jaosorior_mtg | therve: yeah | 14:32 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP rabbitmq HA docker container https://review.openstack.org/455189 | 14:32 |
| *** jaganathan has quit IRC | 14:34 | |
| *** hjensas has joined #tripleo | 14:36 | |
| *** hjensas has quit IRC | 14:36 | |
| *** hjensas has joined #tripleo | 14:36 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP galera HA docker container https://review.openstack.org/455345 | 14:36 |
| chandankumar | trown: ggillies https://review.openstack.org/#/c/458013/ please have a look, thanks! | 14:41 |
| *** pliu has quit IRC | 14:41 | |
| *** adarazs has quit IRC | 14:42 | |
| *** slagle has quit IRC | 14:42 | |
| *** marios has quit IRC | 14:43 | |
| *** lhinds has quit IRC | 14:43 | |
| *** gchamoul has quit IRC | 14:43 | |
| *** cschwede has quit IRC | 14:43 | |
| *** jschlueter has quit IRC | 14:43 | |
| *** myoung has quit IRC | 14:43 | |
| openstackgerrit | Numan Siddique proposed openstack/puppet-tripleo master: Pacemaker support for OVN DB servers https://review.openstack.org/372274 | 14:43 |
| *** stevebaker has quit IRC | 14:44 | |
| *** rnoriega has quit IRC | 14:44 | |
| *** fultonj has quit IRC | 14:44 | |
| *** zoli has quit IRC | 14:44 | |
| *** trozet has joined #tripleo | 14:44 | |
| *** dparkes has quit IRC | 14:47 | |
| pradk | can i get some reviews on https://review.openstack.org/#/c/456385/ | 14:50 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Move ceilometer upgrade re-run out of collector https://review.openstack.org/458036 | 14:50 |
| *** atheurer has quit IRC | 14:50 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Remove no longer used environment files - older upgrade workflows https://review.openstack.org/458864 | 14:51 |
| dtrainor | therve, sorry i got stuck in a call. was jaosorior_mtg's answer sufficient? | 14:51 |
| *** atheurer has joined #tripleo | 14:51 | |
| therve | dtrainor, I'm progressing :) | 14:51 |
| dtrainor | you would need to interface with certmonger, looks like https://review.openstack.org/#/c/457619/ is a good start | 14:51 |
| *** marios has joined #tripleo | 14:52 | |
| therve | http://paste.openstack.org/show/607473/ stuck with NEED_GUIDANCE again | 14:52 |
| dtrainor | ugh i exploded my environment | 14:53 |
| dtrainor | give me a few minutes here | 14:53 |
| *** chlong_ has joined #tripleo | 14:54 | |
| *** chlong_ has quit IRC | 14:54 | |
| dtrainor | off the top of my head it looks good - is it not working? | 14:54 |
| *** atheurer has quit IRC | 14:55 | |
| *** ratailor has quit IRC | 14:56 | |
| *** lmiccini has quit IRC | 14:56 | |
| *** lmiccini has joined #tripleo | 14:57 | |
| jaosorior_mtg | therve: if all else fails, look for ayoung, he's pretty versed with this issues. | 14:58 |
| *** markmc has quit IRC | 14:58 | |
| *** jaosorior_mtg is now known as jaosorior | 14:58 | |
| jaosorior | alright, I'm off. Have a good weekend everyone! | 14:58 |
| therve | jaosorior, Cheers :) | 14:59 |
| *** jaosorior has quit IRC | 14:59 | |
| *** florianf has quit IRC | 14:59 | |
| therve | dtrainor, Yeah it doesn't work | 14:59 |
| EmilienM | ouch, a lof of Red Hat servers are down (or networking issue) at this time | 15:00 |
| *** michapma_dsk has quit IRC | 15:00 | |
| EmilienM | it might affect our CI, I would be careful before rechecking | 15:01 |
| therve | EmilienM, What did you do again | 15:01 |
| EmilienM | therve: bugzilla is down, but also a bunch of other things | 15:01 |
| beekneemech | Oh good, so it's not just me. | 15:01 |
| dtrainor | vpn appears to be affected too | 15:01 |
| EmilienM | dtrainor: try rdu2 | 15:01 |
| *** markmc has joined #tripleo | 15:01 | |
| dtrainor | ya a friend said that worked | 15:01 |
| jrist | EmilienM: phx | 15:02 |
| jrist | is down | 15:02 |
| * dtrainor peeks out the window | 15:02 | |
| jrist | probably the whole data center | 15:02 |
| jrist | :) | 15:02 |
| dtrainor | well, the dc isn't on fire | 15:02 |
| *** flepied has quit IRC | 15:02 | |
| jrist | lol | 15:02 |
| dtrainor | i can confirm that much at least | 15:02 |
| jrist | don't let the smoke escape! | 15:02 |
| dmsimard | FYI PHX2 is confirmed down right now, bunch of things affected -- rdoproject.org, bugzilla, etc | 15:03 |
| dtrainor | no smoke til brooklyn | 15:03 |
| *** yamahata has quit IRC | 15:03 | |
| *** yamahata_ has joined #tripleo | 15:03 | |
| EmilienM | hopefully we now use AFS mirrors :D | 15:04 |
| *** ykarel has joined #tripleo | 15:04 | |
| dmsimard | trunk.rdo is okay actually | 15:04 |
| EmilienM | https://trunk.rdoproject.org/centos7-master/consistent/ is not down | 15:04 |
| *** aufi has quit IRC | 15:04 | |
| EmilienM | yeah | 15:04 |
| dmsimard | it's not hosted out of phx2 | 15:04 |
| EmilienM | I think tripleo CI shouldn't be affected | 15:04 |
| EmilienM | but just in case | 15:04 |
| dmsimard | and buildlogs is out of RDU | 15:04 |
| EmilienM | if you see an outage in our CI, please let us know here | 15:04 |
| dmsimard | basically OS1 and RDO Cloud are down | 15:05 |
| dmsimard | amongst other things | 15:05 |
| *** leanderthal is now known as leanderthal|afk | 15:05 | |
| therve | dtrainor, Hum no it does GENERATING_KEY_PAIR in a loop :/ | 15:05 |
| *** brault has quit IRC | 15:05 | |
| *** atheurer has joined #tripleo | 15:06 | |
| dtrainor | therve, as soon as i can get to these systems again i'll give it a stab | 15:06 |
| dtrainor | but that is bizarre | 15:06 |
| *** brault has joined #tripleo | 15:07 | |
| *** slagle has joined #tripleo | 15:07 | |
| *** newmember has joined #tripleo | 15:08 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 15:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 15:10 |
| *** brault has quit IRC | 15:12 | |
| *** markmc has quit IRC | 15:12 | |
| dtrainor | sorry therve i don't have an environment in which i can risk breaking certmonger on at the moment | 15:13 |
| *** migi has quit IRC | 15:13 | |
| *** chlong has quit IRC | 15:13 | |
| therve | dtrainor, No worries | 15:13 |
| *** milan has joined #tripleo | 15:13 | |
| *** anshul has quit IRC | 15:13 | |
| *** migi has joined #tripleo | 15:14 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Fix bogus parameters in get_param https://review.openstack.org/457541 | 15:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: scenario001/pingtest: enable Gnocchi resource again https://review.openstack.org/442795 | 15:14 |
| EmilienM | pradk: ^ | 15:15 |
| pradk | nice | 15:16 |
| *** florianf has joined #tripleo | 15:16 | |
| *** markmc has joined #tripleo | 15:17 | |
| *** paramite has joined #tripleo | 15:18 | |
| *** asbishop has joined #tripleo | 15:19 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-common master: Add ceilometer agent images https://review.openstack.org/454308 | 15:20 |
| *** paramite has quit IRC | 15:20 | |
| jistr | mandre: we're past the config generation trouble, into DB sync trouble http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz | 15:21 |
| *** myoung has joined #tripleo | 15:21 | |
| *** abishop_ has quit IRC | 15:21 | |
| flaper87 | EmilienM: I'll be there | 15:23 |
| mandre | jistr: yay! progress \o/ | 15:23 |
| *** colonwq has quit IRC | 15:23 | |
| flaper87 | jistr: yaaaaaaaaaaaaaaaaaaaaaaaaahoooo | 15:25 |
| jistr | :D :D | 15:25 |
| flaper87 | or rather, yeeeeehaaaaaa 'cause texas | 15:25 |
| *** kjw3 has quit IRC | 15:26 | |
| mandre | jistr: so you have a error with nova and neutron db, is that right? | 15:28 |
| owalsh | jistr: http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/subnode-2/var/log/messages.txt.gz#_Apr_21_15_04_04 | 15:28 |
| owalsh | nova db sync can't access the api db | 15:29 |
| jistr | mandre, owalsh: indeed nova and neutron db sync | 15:30 |
| *** zoliXXL has joined #tripleo | 15:31 | |
| dtrainor | oh and now my internet is throwing fits | 15:33 |
| *** migi has quit IRC | 15:33 | |
| owalsh | jistr: is start_order being ignored in nova-api.yaml? | 15:33 |
| *** migi has joined #tripleo | 15:33 | |
| owalsh | jistr: or are the tasks running concurrently? | 15:34 |
| *** atheurer has quit IRC | 15:35 | |
| mandre | jistr: I'm also past the libvirt error myself, onto a ceilometer-upgrade issue :) | 15:35 |
| jistr | hahaha | 15:35 |
| *** myoung has quit IRC | 15:35 | |
| jistr | but yea it's good, at least we're going forward again :) | 15:36 |
| *** zoliXXL has quit IRC | 15:36 | |
| owalsh | jrist: ah, hang on. It's an upgrade yea? | 15:36 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add '_comment' as a valid field in instackenv.json https://review.openstack.org/458887 | 15:36 |
| thrash | rasca: ^^^ | 15:36 |
| jistr | owalsh: yea upgrade | 15:36 |
| *** hewbrocca is now known as hewbrocca_afk | 15:36 | |
| mandre | jistr: ceilometer on the host is missing the gnocchiclient module | 15:36 |
| mandre | http://logs.openstack.org/52/454152/10/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/933db18/logs/subnode-2/var/log/ceilometer/ceilometer-upgrade.log.txt.gz | 15:36 |
| mandre | pradk: ^^ | 15:36 |
| mandre | this one looks like a packaging issue | 15:37 |
| *** morazi has quit IRC | 15:37 | |
| EmilienM | beekneemech: do you know out of your fresh mind where can we change permissions of log files before collecting them in tripleo ci (non quickstart) jobs? | 15:38 |
| EmilienM | beekneemech: /etc/sensu has some files owned by root and it makes CI failing | 15:38 |
| rasca | thrash++ | 15:38 |
| *** thrash is now known as thrash|biab | 15:39 | |
| beekneemech | EmilienM: We do some of that in postci today: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n281 | 15:39 |
| *** colonwq has joined #tripleo | 15:39 | |
| EmilienM | pabelanger: do you have the logs details of the sensu problem please? | 15:40 |
| jistr | owalsh: nova_api_db_sync seems to have run fine though | 15:41 |
| *** dprince has quit IRC | 15:41 | |
| pabelanger | EmilienM: it is a permission issue, /etc/sensu on subnodes should be 0755, but is 0444 I think | 15:41 |
| pabelanger | no logs | 15:41 |
| EmilienM | pabelanger: weird, we chown the workspace, https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n281 | 15:42 |
| owalsh | jistr: don't think upgrades have been implemented for nova-api. It's err, complex: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L194 | 15:42 |
| owalsh | ^^ have not been implemented for nova-api docker service | 15:43 |
| pabelanger | EmilienM: chmod | 15:43 |
| EmilienM | pabelanger: can you tell where in the script it fails? | 15:44 |
| EmilienM | pabelanger: in zuul logs? | 15:44 |
| pabelanger | EmilienM: it doesn't fail, that is the issue. We rsync things to zuul-launcher, and because we preserve the 0440 directory permission, we cannot delete it, then job aborts. | 15:44 |
| owalsh | jistr: and for upgrades the order is different, api db sync run as the very end https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L268 | 15:44 |
| pabelanger | we have fixed this today on zuul side, but need to schedule a restart | 15:44 |
| pabelanger | and zuulv3 will make this easy to dealwith | 15:45 |
| EmilienM | pabelanger: ok I see. I'm trying to fix it in tripleo-ci, give me a minut | 15:45 |
| pabelanger | but today, you could ensure your logs folder is 0755 for all directories | 15:45 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart master: Add supplemental node provisioner https://review.openstack.org/451523 | 15:45 |
| EmilienM | pabelanger: it fails during the tarball creation or extraction? | 15:46 |
| pabelanger | EmilienM: find logs -type d -exec chmod 755 {} \; | 15:46 |
| EmilienM | yeah I can do that,I'm just trying where | 15:46 |
| pabelanger | EmilienM: no, when we try to delete the tmp directories | 15:46 |
| jistr | owalsh: alright :D i thought the general nova cell funkiness was specific to N->O upgrade (this upgrade is master->master, so P->P). But yea if the order of db syncs should be reversed on upgrade vs. initial deployment, then that could be the problem. | 15:46 |
| pabelanger | on zuul | 15:46 |
| EmilienM | ah... gotcha | 15:46 |
| *** jpena has quit IRC | 15:46 | |
| mandre | pradk: when you have time, https://review.rdoproject.org/r/#/c/6337/ | 15:47 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy https://review.openstack.org/458407 | 15:47 |
| *** mburned is now known as mburned_out | 15:47 | |
| jistr | owalsh: also we currently don't have a way to express this order difference in templates easily, but maybe i could use heat conditionals for that | 15:47 |
| owalsh | jistr: although this is an ocata -> pike upgrade so we already have cells_v2 setup in ocata. So there should be nothing to do | 15:48 |
| owalsh | jistr: yea, like you just said :-) | 15:48 |
| owalsh | the order only matter when upgrading from non-cell_v2 to cell_v2 | 15:49 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: logs: chmod 755 all workspace directories https://review.openstack.org/458895 | 15:49 |
| EmilienM | pabelanger: ^^^ | 15:49 |
| EmilienM | larsks: fyi ^ | 15:49 |
| jistr | owalsh: ah ok so we should be good hopefully w/r/t order? | 15:49 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: diskimage-builder: Use correct path for element-info https://review.openstack.org/458896 | 15:50 |
| *** jcoufal_ has joined #tripleo | 15:51 | |
| *** trozet has quit IRC | 15:51 | |
| *** jpena has joined #tripleo | 15:52 | |
| EmilienM | pabelanger: anyway we can test this one? | 15:53 |
| owalsh | jistr: yea, I'd hazard a guess that /etc/my.cnf.d/tripleo.cnf is the culprit | 15:54 |
| owalsh | jistr: probably has the wrong IP address | 15:54 |
| owalsh | although that should break all DB operations | 15:55 |
| *** jcoufal has quit IRC | 15:55 | |
| jistr | we're using host mode networking | 15:55 |
| pabelanger | EmilienM: gate-tripleo-ci-centos-7-scenario001-multinode-upgrades-nv is the job that fails | 15:55 |
| jistr | so 192.168.24.9 doesn't look too suspicious | 15:56 |
| pabelanger | EmilienM: so depends on maybe from openstack/tripleo-heat-templates | 15:56 |
| EmilienM | pabelanger: indeed | 15:56 |
| *** lyarwood is now known as me | 15:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 15:56 |
| *** me is now known as Guest41383 | 15:57 | |
| owalsh | jistr: yea, don't think it's that. We wouldn't connect to the DB at all | 15:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike https://review.openstack.org/457603 | 15:57 |
| *** Guest41383 is now known as lyarwood | 15:57 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Make 'reauthentication_auth_method=trusts' the default https://review.openstack.org/457837 | 15:58 |
| owalsh | jistr: just looks like the api_db sync hasn't gotten to the expected schema version | 15:59 |
| jistr | owalsh: yea... maybe just bad RPM or wrong version? We had a very similar issue with Glance db sync not finding Pike target in alembic, was fixed by rebuilding the containers with newer glance RPM. | 16:01 |
| jistr | it could simply be that the Nova in container is older in terms of available DB versions than the one on bare metal that we're upgrading from | 16:02 |
| jistr | that would be quite likely as the RPMs on BM come from delorean known-good snapshots, but the containers images aren't being rebuilt that often (manual process currently AFAIK) | 16:03 |
| *** bugzy_ has joined #tripleo | 16:04 | |
| *** bugzy has quit IRC | 16:06 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 16:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 16:10 |
| *** jpich has quit IRC | 16:10 | |
| *** mcornea has quit IRC | 16:13 | |
| zaneb | EmilienM: apparently I was looking at the wrong job :( it's still 574s in the ovb-nonha job. revert has landed though, so you should have some breathing room while I figure it out | 16:13 |
| EmilienM | zaneb: yeah I'm now trying to promote CI | 16:14 |
| *** lmiccini has quit IRC | 16:14 | |
| *** jlinkes has quit IRC | 16:22 | |
| mnaser | hey folks, i was stuck with tripleo deploys failing pxe boots for the longest time ever till i found out that stp was blocking the port for too long (and setting the port to portfast was a common thing to work around this) | 16:22 |
| mnaser | would a note about this in the tripleo-docs be in the right place or not? | 16:23 |
| *** ramishra has quit IRC | 16:24 | |
| *** ramishra has joined #tripleo | 16:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/456030 | 16:27 |
| *** jkilpatr has quit IRC | 16:28 | |
| *** jkilpatr_ has joined #tripleo | 16:28 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Test I71f8aa431a60457326167b8c82adc03ca750eda6 https://review.openstack.org/458858 | 16:29 |
| *** dprince has joined #tripleo | 16:29 | |
| openstackgerrit | Merged openstack/tripleo-docs master: Add export to deployment document for image builds. https://review.openstack.org/457780 | 16:29 |
| *** dsneddon has quit IRC | 16:30 | |
| shardy | mnaser: yes, perhaps a note in the troubleshooting section? | 16:30 |
| *** trown is now known as trown|lunch | 16:30 | |
| *** beekneemech has quit IRC | 16:30 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode https://review.openstack.org/454152 | 16:31 |
| *** colonwq has quit IRC | 16:33 | |
| *** lucasagomes is now known as lucas-afk | 16:36 | |
| *** jayg is now known as jayg|g0n3 | 16:38 | |
| *** lblanchard has joined #tripleo | 16:39 | |
| *** lblanchard has quit IRC | 16:39 | |
| *** jprovazn has quit IRC | 16:41 | |
| *** dsneddon has joined #tripleo | 16:41 | |
| *** gfidente is now known as gfidente|afk | 16:42 | |
| *** adarazs has joined #tripleo | 16:43 | |
| *** bnemec has joined #tripleo | 16:44 | |
| *** salmankhan has quit IRC | 16:45 | |
| *** bnemec is now known as beekneemech | 16:45 | |
| *** adarazs has quit IRC | 16:47 | |
| *** shardy has quit IRC | 16:50 | |
| *** rasca has quit IRC | 16:52 | |
| *** mburned_out has quit IRC | 16:52 | |
| *** markmc has quit IRC | 16:53 | |
| *** hewbrocca_afk has quit IRC | 16:53 | |
| *** amoralej has quit IRC | 16:53 | |
| *** akrzos has quit IRC | 16:53 | |
| *** asbishop has quit IRC | 16:53 | |
| *** pabelanger has quit IRC | 16:54 | |
| *** dmanchad has quit IRC | 16:54 | |
| *** jpena has quit IRC | 16:54 | |
| *** weshay has quit IRC | 16:54 | |
| *** migi has quit IRC | 16:54 | |
| *** atheurer has joined #tripleo | 16:55 | |
| *** florianf has quit IRC | 16:56 | |
| *** derekh has quit IRC | 17:00 | |
| *** rasca has joined #tripleo | 17:00 | |
| *** abishop has joined #tripleo | 17:00 | |
| *** hewbrocca_afk has joined #tripleo | 17:00 | |
| *** jpena|off has joined #tripleo | 17:00 | |
| *** migi has joined #tripleo | 17:00 | |
| *** pabelanger has joined #tripleo | 17:00 | |
| *** dmanchad has joined #tripleo | 17:01 | |
| *** akrzos has joined #tripleo | 17:01 | |
| *** markmc has joined #tripleo | 17:01 | |
| *** weshay has joined #tripleo | 17:01 | |
| *** mburned_out has joined #tripleo | 17:02 | |
| *** amoralej has joined #tripleo | 17:04 | |
| *** jpena|off is now known as jpena | 17:04 | |
| *** jpena is now known as jpena|off | 17:07 | |
| EmilienM | FYI RH1 is unreachable, don't expect OVB jobs working today | 17:07 |
| *** tosky has quit IRC | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 17:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 17:10 |
| *** brault has joined #tripleo | 17:10 | |
| *** trown|lunch is now known as trown | 17:14 | |
| *** ffiore has quit IRC | 17:16 | |
| *** brault has quit IRC | 17:18 | |
| *** tesseract has quit IRC | 17:19 | |
| beekneemech | EmilienM: I opened https://bugs.launchpad.net/tripleo/+bug/1685318 to track it. | 17:21 |
| openstack | Launchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged] | 17:21 |
| * beekneemech re-prioritizes the ability to run OVB on an arbitrary public cloud | 17:21 | |
| beekneemech | It would be nice if we could spin up a new ci environment on a random public cloud when this sort of thing happens. | 17:22 |
| *** marios has quit IRC | 17:22 | |
| pabelanger | 1000++ | 17:24 |
| pabelanger | it would be great to have tripleo not depend on a single cloud | 17:24 |
| *** bugzy_ is now known as bugzy | 17:24 | |
| pabelanger | Was there any progress on getting OVB upstream for nova? | 17:24 |
| *** jcoufal_ has quit IRC | 17:25 | |
| *** pkovar has quit IRC | 17:25 | |
| beekneemech | pabelanger: I think https://blueprints.launchpad.net/nova/+spec/virt-rescue-stable-disk-devices would allow us to do full OVB in vanilla nova. | 17:25 |
| *** jtomasek has quit IRC | 17:26 | |
| *** jcoufal has joined #tripleo | 17:26 | |
| beekneemech | But for ci we can actually use an ipxe image too, so it's possible today. | 17:26 |
| beekneemech | Integrated pxe boot support in nova is more of an interactive developer environment issue. | 17:26 |
| *** thrash|biab is now known as thrash | 17:27 | |
| beekneemech | So theoretically it's possible to do this today, but I haven't had a chance to investigate whether any public clouds actually expose all of the features we need. | 17:27 |
| *** jcoufal has quit IRC | 17:28 | |
| pabelanger | sounds like something to add into devstack, if possible. Then we'd know for sure if all our clouds support it | 17:28 |
| *** ykarel has quit IRC | 17:31 | |
| *** dspano has joined #tripleo | 17:36 | |
| *** jaganathan has joined #tripleo | 17:36 | |
| openstackgerrit | Merged openstack/tripleo-common master: Workflow to select nodes matching a profile/role https://review.openstack.org/441054 | 17:37 |
| dspano | Good afternoon. I have a question regarding post-deploy overcloud config. Are there any folks using Ansible to configure their clouds after deploying? | 17:38 |
| dspano | Not asking for support, more of a am I stupid for thinking of it kind of thing. | 17:39 |
| *** dprince has quit IRC | 17:41 | |
| *** lblanchard has joined #tripleo | 17:41 | |
| beekneemech | dspano: There are people doing that, but you do need to be careful. TripleO uses puppet to configure things, and if puppet and ansible end up trying to manage the same thing there can be issues. | 17:41 |
| *** ramishra has quit IRC | 17:42 | |
| *** newmember has quit IRC | 17:42 | |
| dspano | @beekneemech, thanks for the feedback. I totally understand. | 17:43 |
| *** ramishra has joined #tripleo | 17:44 | |
| *** anshul has joined #tripleo | 17:46 | |
| *** ramishra has quit IRC | 17:49 | |
| *** rwsu has quit IRC | 17:51 | |
| *** ramishra has joined #tripleo | 17:51 | |
| *** dprince has joined #tripleo | 17:53 | |
| *** dprince has quit IRC | 17:53 | |
| *** jaganathan has quit IRC | 17:53 | |
| *** dprince has joined #tripleo | 17:53 | |
| *** gfidente|afk has quit IRC | 17:54 | |
| *** dparkes has joined #tripleo | 18:03 | |
| *** yamahata_ has quit IRC | 18:04 | |
| *** owalsh_ has joined #tripleo | 18:06 | |
| *** jmelvin has quit IRC | 18:09 | |
| *** ykarel has joined #tripleo | 18:09 | |
| *** owalsh has quit IRC | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1685318 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 18:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 18:10 |
| openstack | Launchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged] | 18:10 |
| *** ccamacho has quit IRC | 18:12 | |
| *** ykarel has quit IRC | 18:14 | |
| *** yamahata_ has joined #tripleo | 18:21 | |
| *** amoralej is now known as amoralej|off | 18:23 | |
| *** gchamoul has joined #tripleo | 18:24 | |
| *** atheurer has quit IRC | 18:24 | |
| *** cschwede has joined #tripleo | 18:24 | |
| *** rnoriega has joined #tripleo | 18:25 | |
| *** atheurer has joined #tripleo | 18:25 | |
| *** pliu has joined #tripleo | 18:27 | |
| *** dparkes has quit IRC | 18:27 | |
| *** zoli has joined #tripleo | 18:27 | |
| *** stevebaker has joined #tripleo | 18:27 | |
| *** lhinds has joined #tripleo | 18:27 | |
| *** dparkes has joined #tripleo | 18:28 | |
| *** marios has joined #tripleo | 18:29 | |
| *** slagle_ has joined #tripleo | 18:29 | |
| *** tvignaud has quit IRC | 18:29 | |
| *** pkovar has joined #tripleo | 18:30 | |
| *** bogdando has quit IRC | 18:32 | |
| *** paramite has joined #tripleo | 18:35 | |
| *** rwsu has joined #tripleo | 18:36 | |
| *** suuuper has quit IRC | 18:38 | |
| *** ihrachys has joined #tripleo | 18:40 | |
| *** jerrygb has quit IRC | 18:40 | |
| ihrachys | pradk, ping re telemetry on overcloud taking all resources | 18:40 |
| ihrachys | https://bugs.launchpad.net/tripleo/+bug/1621164 | 18:40 |
| openstack | Launchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [High,In progress] | 18:40 |
| ihrachys | just hit something similar, it was ceilo/gnocchi + swift taking all CPU | 18:41 |
| ihrachys | I wonder why we have it on by default in the first place | 18:41 |
| pradk | ihrachys, whats the error in statsd log? | 18:41 |
| ihrachys | I would think those interested would opt in | 18:41 |
| pradk | ihrachys, there were some optimizations made in swift in ocata i believe.. what version are you using? | 18:42 |
| ihrachys | pradk, no error, just wasted cycles | 18:42 |
| *** myoung has joined #tripleo | 18:42 | |
| ihrachys | it's ocata | 18:43 |
| pradk | ihrachys, as jd said in the comment, we saw this before, but the issue turned out to be on swift side | 18:43 |
| ihrachys | for now I disabled ceilo and gnocchi, bumped swift replication intervals. | 18:43 |
| ihrachys | seems a bit better | 18:43 |
| pradk | ok | 18:43 |
| ihrachys | so my question is mostly about why we have it on by default. | 18:44 |
| ihrachys | especially since we tend to struggle with getting resources to deploy the thing in e.g. dev env | 18:44 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI https://review.openstack.org/407733 | 18:44 |
| pradk | ihrachys, we want telemetry enabled by default for events and metric processing .. but if you dont care about it, you can disbale by removing from the roles yaml | 18:45 |
| *** zoli has quit IRC | 18:45 | |
| pradk | ihrachys, we saw perf issues in newton but most of those have been fixed afaik in ocata | 18:45 |
| *** rwsu has quit IRC | 18:45 | |
| ihrachys | pradk, I deploy with infrared so it's same libvirt handling all nodes | 18:45 |
| ihrachys | pradk, would need to have a look how to use a custom yaml there. | 18:46 |
| ihrachys | it seems to me the least there should be some ready profiles for minimal env | 18:46 |
| ihrachys | if not default | 18:46 |
| ihrachys | something that environments like infrared and generally devs could use to get work done with reasonable resources | 18:47 |
| ihrachys | not e.g. 64gb+ | 18:47 |
| *** tvignaud has joined #tripleo | 18:47 | |
| *** myoung|remote has joined #tripleo | 18:47 | |
| *** myoung|remote is now known as myoung|alt | 18:48 | |
| pradk | ihrachys, in pike, we're removing ceilo api and collector so should help with load we have 2 less services on by default. But yea good idea to have some pre-canned yamls for minimal deploys | 18:49 |
| *** jerrygb has joined #tripleo | 18:49 | |
| ihrachys | pradk, I have 32gb workstation. imagine the struggle. ;) | 18:50 |
| *** rwsu has joined #tripleo | 18:50 | |
| pradk | ihrachys, i can feel your pain :/ | 18:50 |
| *** zoli has joined #tripleo | 18:50 | |
| *** jmelvin has joined #tripleo | 18:50 | |
| *** atheurer has quit IRC | 18:52 | |
| *** atheurer has joined #tripleo | 18:52 | |
| *** gbarros has joined #tripleo | 18:52 | |
| *** gbarros_ has joined #tripleo | 18:54 | |
| *** dparkes_ has joined #tripleo | 18:54 | |
| *** zoli has quit IRC | 18:55 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Deprecate ceilometer collector https://review.openstack.org/450885 | 18:56 |
| *** dparkes has quit IRC | 18:57 | |
| *** gbarros has quit IRC | 18:58 | |
| *** ansmith has quit IRC | 18:59 | |
| *** rwsu has quit IRC | 19:00 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud https://review.openstack.org/437544 | 19:01 |
| *** fragatina has quit IRC | 19:01 | |
| *** zoli has joined #tripleo | 19:01 | |
| *** pcaruana has quit IRC | 19:02 | |
| *** dparkes_ has quit IRC | 19:03 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI https://review.openstack.org/407733 | 19:03 |
| *** fragatina has joined #tripleo | 19:03 | |
| *** kbyrne has quit IRC | 19:04 | |
| *** dparkes has joined #tripleo | 19:05 | |
| *** tvignaud has quit IRC | 19:06 | |
| *** rwsu has joined #tripleo | 19:07 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1685318 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 19:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 19:10 |
| openstack | Launchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged] | 19:10 |
| *** Slower_ is now known as Slower | 19:13 | |
| *** brault has joined #tripleo | 19:14 | |
| *** newmember has joined #tripleo | 19:17 | |
| *** dparkes has quit IRC | 19:17 | |
| *** brault has quit IRC | 19:18 | |
| *** iranzo has quit IRC | 19:19 | |
| *** fultonj has joined #tripleo | 19:19 | |
| *** zoli has quit IRC | 19:20 | |
| *** tvignaud has joined #tripleo | 19:20 | |
| *** gbarros_ has quit IRC | 19:20 | |
| *** zoli has joined #tripleo | 19:21 | |
| *** gbarros has joined #tripleo | 19:21 | |
| *** fragatina has quit IRC | 19:22 | |
| *** dparkes has joined #tripleo | 19:24 | |
| EmilienM | beekneemech: sounds like it's back | 19:26 |
| EmilienM | I see ovb jobs running | 19:27 |
| *** dparkes has quit IRC | 19:28 | |
| *** dparkes has joined #tripleo | 19:30 | |
| *** zoli has quit IRC | 19:30 | |
| *** tvignaud has quit IRC | 19:34 | |
| *** dparkes has quit IRC | 19:34 | |
| beekneemech | EmilienM: Yep, looks better now. | 19:35 |
| -openstackstatus- NOTICE: Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 19:44 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI https://review.openstack.org/407733 | 19:46 |
| *** tvignaud has joined #tripleo | 19:47 | |
| *** stevebaker has quit IRC | 19:48 | |
| *** gchamoul has quit IRC | 19:48 | |
| *** pliu has quit IRC | 19:48 | |
| *** marios has quit IRC | 19:48 | |
| *** cschwede has quit IRC | 19:48 | |
| *** rnoriega has quit IRC | 19:49 | |
| *** lhinds has quit IRC | 19:49 | |
| *** myoung has quit IRC | 19:49 | |
| *** fragatina has joined #tripleo | 19:50 | |
| paramite | EmilienM, Hey Emilien, are you here? | 19:51 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Move ceilometer upgrade re-run out of collector https://review.openstack.org/458036 | 19:51 |
| *** anshul has quit IRC | 19:52 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add service config settings to agent services https://review.openstack.org/456385 | 19:52 |
| *** dougbtv|laptop has quit IRC | 19:57 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Redis service https://review.openstack.org/442151 | 20:01 |
| EmilienM | paramite: yes | 20:05 |
| paramite | EmilienM, I've sent you a mail re sensu. Are you sure correct opstools repo is used in CI jobs? | 20:05 |
| *** gchamoul has joined #tripleo | 20:05 | |
| paramite | because it seems that old sensu is installed instead of latest | 20:06 |
| *** cschwede has joined #tripleo | 20:06 | |
| -openstackstatus- NOTICE: Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 20:06 | |
| *** ChanServ changes topic to "Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html" | 20:06 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 20:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 20:10 |
| *** jerrygb has quit IRC | 20:10 | |
| EmilienM | paramite: i'm sure no nothing | 20:11 |
| EmilienM | paramite: http://logs.openstack.org/85/450885/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/785654d/logs/subnode-2/etc/yum.repos.d/centos-opstools.repo.txt.gz | 20:11 |
| EmilienM | tell us if it's good | 20:11 |
| EmilienM | it's a mirror from centos | 20:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike https://review.openstack.org/457603 | 20:11 |
| *** gchamoul has quit IRC | 20:12 | |
| *** slagle_ has quit IRC | 20:12 | |
| *** cschwede has quit IRC | 20:13 | |
| paramite | EmilienM, uhm ... 0.27.0 is there ... let me check which sensu version has been installed | 20:13 |
| *** atheurer has quit IRC | 20:15 | |
| EmilienM | sensu-0.27.0-1.el7.noarch | 20:18 |
| EmilienM | http://logs.openstack.org/85/450885/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/785654d/logs/subnode-2/rpm-qa.txt.gz | 20:18 |
| EmilienM | paramite: ^ | 20:18 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Expose metric delay processing metric https://review.openstack.org/458959 | 20:18 |
| *** atheurer has joined #tripleo | 20:18 | |
| *** ramishra has quit IRC | 20:20 | |
| paramite | EmilienM, ok, now again I don't have an idea what's going on ... is there a possibility that 0.23.2 has been installed before on image and the README.md just did not get cleaned? | 20:20 |
| EmilienM | paramite: we found the issue on upgrade jobs but AFIK opstools are not versioned in RDO, since they are hosted on centos | 20:21 |
| *** jmelvin has quit IRC | 20:21 | |
| EmilienM | so I don't see how 0.23.2 can be installed | 20:21 |
| EmilienM | we can check in logs btw... | 20:21 |
| EmilienM | but no, 0.23.2 can't be installed, since it's not the version provided in the mirror | 20:22 |
| *** jmelvin has joined #tripleo | 20:22 | |
| paramite | EmilienM, there is 0.23.2 in mirror repo: http://mirror.regionone.osic-cloud1.openstack.org/centos/7/opstools/x86_64/sensu/sensu-0.23.2-3.el7.noarch.rpm | 20:22 |
| EmilienM | paramite: unless we pin a version somewhere in tripleo, I think we pick the latest version... | 20:23 |
| paramite | ok | 20:23 |
| *** jerrygb has joined #tripleo | 20:25 | |
| *** anshul has joined #tripleo | 20:26 | |
| paramite | EmilienM, well, /etc/sensu/conf.d/README.md is located only in 0.23.2 package ... I will try to install it and then update to see if the file is left in the directory | 20:27 |
| *** atheurer has quit IRC | 20:27 | |
| *** dprince has quit IRC | 20:29 | |
| paramite | EmilienM, well nope, the file is correctly cleaned after update | 20:29 |
| EmilienM | paramite: where do you see that? | 20:30 |
| paramite | EmilienM, I just tried that in centos VM with the mirror repo | 20:30 |
| *** anshul has quit IRC | 20:31 | |
| EmilienM | paramite: maybe the package didn't get updated and zuul failed to collect logs. Weird, why would it work on stable/ocata | 20:31 |
| paramite | EmilienM, no idea, but even in 0.23.2 the file is not owned by root: http://paste.openstack.org/show/607523/ | 20:33 |
| EmilienM | paramite: weird. Puppet manage the dir or files somehow? | 20:35 |
| paramite | EmilienM, yes but AFAIR default owner is sensu:sensu ... let me check | 20:35 |
| *** gbarros has quit IRC | 20:36 | |
| paramite | EmilienM, yup https://github.com/sensu/sensu-puppet/blob/f1ed6eace5ec0a2e9b77ee70516bf89f35b651df/manifests/init.pp#L547 | 20:36 |
| EmilienM | ah found it | 20:37 |
| EmilienM | https://github.com/sensu/sensu-puppet/blob/f1ed6eace5ec0a2e9b77ee70516bf89f35b651df/manifests/init.pp#L553 | 20:37 |
| EmilienM | this is not enough for zuul to get logs | 20:37 |
| EmilienM | that's why I think | 20:37 |
| openstackgerrit | Alan Bishop proposed openstack/tripleo-common master: WIP: Derive Params (part 1): Initial workflow to start https://review.openstack.org/456469 | 20:37 |
| paramite | EmilienM, ah-hah, so we need o+r? | 20:37 |
| *** rbowen has quit IRC | 20:38 | |
| *** ramishra has joined #tripleo | 20:38 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-docs master: WIP: Post deployment installation instructions for Octavia https://review.openstack.org/447496 | 20:38 |
| EmilienM | paramite: 755 prob | 20:38 |
| *** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 20:38 | |
| -openstackstatus- NOTICE: Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes | 20:38 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-docs master: WIP: Post deployment installation instructions for Octavia https://review.openstack.org/447496 | 20:38 |
| paramite | EmilienM, uhm, on plain files? | 20:38 |
| *** lblanchard has quit IRC | 20:40 | |
| EmilienM | paramite: that's what pabelanger suggested yeah | 20:40 |
| beekneemech | Hmm, ovb jobs died again. | 20:42 |
| EmilienM | beekneemech: network again? | 20:43 |
| paramite | EmilienM, hmm, ok that's gonna need sensu-puppet patch, because as you can see those variables are hardcoded | 20:43 |
| beekneemech | EmilienM: Not sure. I still have access through the bastion. | 20:43 |
| pabelanger | we dumped zuul queues | 20:47 |
| EmilienM | paramite: yeah, sad | 20:47 |
| pabelanger | so jobs would have been killed | 20:47 |
| beekneemech | Ahh, that would do it. | 20:47 |
| *** dparkes has joined #tripleo | 20:47 | |
| pabelanger | project renames per notice^ | 20:47 |
| * beekneemech will worry about it if they're still broken on Monday at this point | 20:48 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** ramishra has quit IRC | 20:49 | |
| *** trown is now known as trown|outtypewww | 20:50 | |
| *** atheurer has joined #tripleo | 20:52 | |
| *** dsneddon has quit IRC | 20:53 | |
| *** abishop has quit IRC | 20:53 | |
| paramite | EmilienM, so gate-tripleo-ci-centos-7-scenario001-multinode-upgrades-nv is failing on sensu error? | 20:54 |
| EmilienM | paramite: Zuul fails to collect logs | 20:55 |
| EmilienM | because of sensu README in etc | 20:55 |
| leifmadsen | I feel like I've seen this error before, and it was just a mismatch in profile names or something, but I get this on a Newton deploy.... | 20:56 |
| leifmadsen | Error: Could not find class ::tripleo::profile::base::time::ntp for overcloud-novacompute-0.localdomain on node overcloud-novacompute-0.localdomain | 20:56 |
| *** tvignaud has quit IRC | 20:58 | |
| *** zaneb has quit IRC | 20:59 | |
| mwhahaha | leifmadsen: outdated packages | 21:03 |
| leifmadsen | ahhh | 21:04 |
| mwhahaha | the puppet-tripleo is too old | 21:04 |
| leifmadsen | oic | 21:04 |
| *** tvignaud has joined #tripleo | 21:04 | |
| mwhahaha | leifmadsen: or the tripleo-heat-templates package is too n ew | 21:04 |
| mwhahaha | which ever :D | 21:04 |
| leifmadsen | :D | 21:04 |
| leifmadsen | ah guess whatever I get in centos7-newton/current-passed-ci/ is too old :\ | 21:05 |
| mwhahaha | let me see | 21:05 |
| leifmadsen | 5.5.1 | 21:05 |
| leifmadsen | hrmmm.... April 10th | 21:05 |
| leifmadsen | seems like it should be new enough | 21:05 |
| mwhahaha | leifmadsen: yea it should be, https://review.openstack.org/#/c/443333/ was the patch that added that class | 21:06 |
| mwhahaha | which was from march | 21:06 |
| leifmadsen | oh wait, I bet it's because of something I'm doing... | 21:06 |
| mwhahaha | though did you use pre-built images maybe? | 21:06 |
| leifmadsen | (I'm trying to do a Contrail deploy with their instructions, and I bet some of the things they had me do reverted this) | 21:06 |
| leifmadsen | not pre-built, but I bet I'm overriding the modules with something too old | 21:07 |
| leifmadsen | actually huh... well whatever I updated for modules has nothing to do with ntp | 21:07 |
| mwhahaha | check what package is actually installed on the node | 21:07 |
| leifmadsen | ah ha I think | 21:09 |
| leifmadsen | 5.5.0 | 21:09 |
| leifmadsen | march 07? | 21:09 |
| leifmadsen | let me check the modules | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 21:10 |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 21:10 |
| mwhahaha | leifmadsen: you can workaround it in your THT by reverting to https://review.openstack.org/#/c/443334/ | 21:10 |
| leifmadsen | +1 | 21:10 |
| leifmadsen | mwhahaha: thanks for the tips, I knew I'd seen this before, but I couldn't remember how I'd resolved it before | 21:10 |
| mwhahaha | leifmadsen: switch back to include ::ntp, but you'll end up with ntp/chronyd fighting on reboots | 21:10 |
| leifmadsen | yea, all good | 21:11 |
| leifmadsen | I know what I can do to work around it / resolve it then | 21:11 |
| leifmadsen | basically need to make sure I get a newer overcloud image | 21:11 |
| leifmadsen | have a mismatch it seems | 21:11 |
| leifmadsen | mwhahaha: thanks again! this will move me forward next week | 21:11 |
| mwhahaha | leifmadsen: np | 21:11 |
| *** zaneb has joined #tripleo | 21:12 | |
| *** brault has joined #tripleo | 21:15 | |
| *** brault has quit IRC | 21:20 | |
| *** fzdarsky has quit IRC | 21:20 | |
| *** karimb has quit IRC | 21:21 | |
| *** dparkes has quit IRC | 21:22 | |
| *** openstackgerrit has joined #tripleo | 21:22 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI https://review.openstack.org/407733 | 21:22 |
| *** mhenkel_ has quit IRC | 21:26 | |
| *** ramishra has joined #tripleo | 21:34 | |
| *** kbyrne has joined #tripleo | 21:39 | |
| *** ramishra has quit IRC | 21:40 | |
| *** fragatin_ has joined #tripleo | 21:42 | |
| *** ramishra has joined #tripleo | 21:42 | |
| *** fragatina has quit IRC | 21:45 | |
| *** dspano has quit IRC | 21:46 | |
| *** dsavineau has quit IRC | 21:47 | |
| *** ramishra has quit IRC | 21:48 | |
| pabelanger | paramite: EmilienM: we should be protected on zuul side now, haven't checked the job yet | 21:50 |
| EmilienM | ok | 21:50 |
| paramite | so what does it mean? | 21:51 |
| *** myoung|alt has quit IRC | 21:51 | |
| *** jcoufal has joined #tripleo | 21:52 | |
| pabelanger | you should still fix your directory permissions, but zuul will no longer abort the job on failure | 21:55 |
| pabelanger | in zuulv3, we'll have better control over this with ansible playbooks | 21:55 |
| *** rlandy has quit IRC | 21:57 | |
| paramite | ack | 22:01 |
| *** ramishra has joined #tripleo | 22:05 | |
| *** jmelvin has quit IRC | 22:06 | |
| *** newmember has quit IRC | 22:08 | |
| *** fragatin_ has quit IRC | 22:08 | |
| *** fragatina has joined #tripleo | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] | 22:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 22:10 |
| *** dsneddon has joined #tripleo | 22:11 | |
| *** ramishra has quit IRC | 22:13 | |
| honza | I'm having trouble enqueuing new messages in zaqar on the undercloud, I'm getting this strange 500 error from swift | 22:33 |
| honza | "Object GET failed: http://192.168.24.1:8080/v1/AUTH_8eabab8673e7431280817d6d8fe59696/zaqar_subscription%3Atripleo%3Ab57034af85334fbea5d650487c2db9bf/42e69b1e-a380-4250-951a-9ce9f523800d 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<" | 22:33 |
| honza | It was working for a while but suddenly stopped :( | 22:34 |
| honza | Where could one start looking? | 22:34 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for TripleO-UI https://review.openstack.org/407733 | 22:35 |
| mwhahaha | honza: i wonder if that's an old auth url? did you deploy then delete the stack without clearing out the old plan? | 22:37 |
| mwhahaha | honza: https://bugs.launchpad.net/tripleo/+bug/1644875 | 22:38 |
| openstack | Launchpad bug 1644875 in tripleo "Plan isn't purged of roles_data on a new create" [High,Triaged] | 22:38 |
| honza | mwhahaha: I haven't touched the plan at all. All I'm doing is logging into the GUI and sending a bunch of messages via websockets. | 22:38 |
| * mwhahaha knows nothing about the ui | 22:38 | |
| honza | the issue is between zaqar and swift it seems | 22:39 |
| *** ramishra has joined #tripleo | 22:41 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for TripleO-UI https://review.openstack.org/407733 | 22:42 |
| honza | message_post works fine via a Python script (websockets), but not via js :( | 22:43 |
| *** ramishra has quit IRC | 22:46 | |
| *** ramishra has joined #tripleo | 22:48 | |
| *** ramishra has quit IRC | 22:53 | |
| *** jcoufal has quit IRC | 22:55 | |
| *** ramishra has joined #tripleo | 23:00 | |
| *** ramishra has quit IRC | 23:05 | |
| *** ramishra has joined #tripleo | 23:06 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1680259 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1684272 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] | 23:10 |
| openstack | Launchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress] | 23:10 |
| *** mmethot has joined #tripleo | 23:13 | |
| *** owalsh_ is now known as owalsh | 23:13 | |
| *** brault has joined #tripleo | 23:16 | |
| *** ramishra has quit IRC | 23:17 | |
| *** owalsh_ has joined #tripleo | 23:19 | |
| *** brault has quit IRC | 23:20 | |
| *** owalsh has quit IRC | 23:22 | |
| *** ramishra has joined #tripleo | 23:24 | |
| *** pkovar has quit IRC | 23:28 | |
| *** ramishra has quit IRC | 23:29 | |
| *** dixiaoli has joined #tripleo | 23:30 | |
| *** dixiaoli has quit IRC | 23:35 | |
| *** ramishra has joined #tripleo | 23:37 | |
| *** yamahata_ has quit IRC | 23:40 | |
| *** owalsh has joined #tripleo | 23:44 | |
| *** owalsh_ has quit IRC | 23:44 | |
| *** ramishra has quit IRC | 23:44 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!