| *** slagle has joined #tripleo | 00:03 | |
| *** rook has quit IRC | 00:04 | |
| *** rook has joined #tripleo | 00:06 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Revert "Convert allNodesConfig properties to composable jinja2" https://review.openstack.org/370486 | 00:06 |
|---|---|---|
| *** hjensas has quit IRC | 00:06 | |
| *** rook is now known as Guest8952 | 00:06 | |
| *** sai has quit IRC | 00:10 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 00:10 |
| *** lucas-dinner has quit IRC | 00:10 | |
| *** Guest8952 has quit IRC | 00:12 | |
| *** lucasagomes has joined #tripleo | 00:16 | |
| *** rook_ has joined #tripleo | 00:16 | |
| *** sai has joined #tripleo | 00:16 | |
| EmilienM | thrash: I'm afk | 00:29 |
| EmilienM | But back in 1h30 | 00:29 |
| EmilienM | What's up with puppet? | 00:30 |
| EmilienM | I think it's OK | 00:30 |
| EmilienM | slagle: another revert? | 00:30 |
| *** phpcodemonkey has quit IRC | 00:33 | |
| EmilienM | Bbl | 00:33 |
| slagle | EmilienM: was just experimenting. it didnt work anyway | 00:38 |
| thrash | EmilienM: I'll ping you then. | 00:39 |
| *** akshai has joined #tripleo | 00:51 | |
| *** akshai has quit IRC | 00:53 | |
| *** sarath has joined #tripleo | 01:00 | |
| *** sarath has quit IRC | 01:10 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 01:10 |
| *** bana_k has quit IRC | 01:14 | |
| *** cwolferh has quit IRC | 01:18 | |
| *** saneax is now known as saneax-_-|AFK | 01:27 | |
| *** dmacpher has quit IRC | 01:34 | |
| *** dmacpher has joined #tripleo | 01:34 | |
| *** bana_k has joined #tripleo | 01:35 | |
| EmilienM | slagle, thrash: back | 01:56 |
| EmilienM | thrash: looking the zaqar error | 01:57 |
| thrash | EmilienM: hey... so the require => User['zaqar'] is cool? | 01:57 |
| EmilienM | it should be but now I have a doubt | 01:58 |
| EmilienM | thrash: ok this is not related to our CI downtime, right? | 01:59 |
| thrash | EmilienM: I don't think so. So if you have more pressing matters, by all means, deal with them. | 02:00 |
| thrash | EmilienM: like Ci downtime. :) | 02:00 |
| EmilienM | thrash: I'm sending a quick patch | 02:00 |
| EmilienM | thrash: https://review.openstack.org/370513 | 02:02 |
| thrash | EmilienM: awesome. Thanks! | 02:03 |
| EmilienM | thrash: you can blame dprince :P | 02:05 |
| EmilienM | thrash: joking | 02:05 |
| EmilienM | he wrote most of this module :) | 02:05 |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 02:10 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| *** bkopilov has quit IRC | 02:16 | |
| EmilienM | I'm wondering if http://logs.openstack.org/50/370250/2/check/gate-tripleo-ci-centos-7-nonha-multinode/9e8c125/console.html#_2016-09-14_16_57_06_622563 is also critical | 02:20 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Revert "Migrate to using osc-lib" https://review.openstack.org/370516 | 02:23 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/370517 | 02:24 |
| EmilienM | thrash, slagle: I found out that we missed this backport in oooclient ^ | 02:25 |
| EmilienM | it might be the reason | 02:25 |
| EmilienM | yeah | 02:25 |
| EmilienM | I did that patch : https://review.rdoproject.org/r/#/c/2224/ to come back on what RDO wants, to use latest branches for clients | 02:26 |
| *** michchap has quit IRC | 02:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Add `overcloud parameters set` to set Heat params in a plan https://review.openstack.org/370518 | 02:26 |
| *** michchap has joined #tripleo | 02:26 | |
| EmilienM | i'll continue tomorrow morning. /me afk | 02:29 |
| *** absubram has quit IRC | 02:31 | |
| thrash | EmilienM: that'll do it | 02:35 |
| *** thrash is now known as thrash|g0ne | 02:39 | |
| *** rlandy has quit IRC | 02:44 | |
| *** masco has joined #tripleo | 03:06 | |
| *** noslzzp has quit IRC | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 03:10 |
| *** yamahata has quit IRC | 03:11 | |
| *** fzdarsky has joined #tripleo | 03:29 | |
| *** bkopilov has joined #tripleo | 03:33 | |
| *** cwolferh has joined #tripleo | 04:08 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 04:10 |
| *** kjw3 has quit IRC | 04:11 | |
| *** fzdarsky has quit IRC | 04:13 | |
| *** kjw3 has joined #tripleo | 04:27 | |
| *** fragatin_ has joined #tripleo | 04:29 | |
| *** fragatina has quit IRC | 04:32 | |
| *** fragatin_ has quit IRC | 04:33 | |
| *** coolsvap has joined #tripleo | 04:34 | |
| *** abregman has joined #tripleo | 04:38 | |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Have dev server listen everywhere instead of just local https://review.openstack.org/370537 | 04:39 |
| *** jaosorior has joined #tripleo | 04:47 | |
| *** saneax-_-|AFK is now known as saneax | 04:58 | |
| *** jaosorior has quit IRC | 05:01 | |
| *** jaosorior has joined #tripleo | 05:02 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add VNC Proxy common manifest https://review.openstack.org/370541 | 05:09 |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 05:10 |
| *** jprovazn has joined #tripleo | 05:13 | |
| *** florianf has joined #tripleo | 05:15 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Set VNC common data for compute nodes also https://review.openstack.org/370331 | 05:16 |
| *** pgadiya has joined #tripleo | 05:16 | |
| *** mcornea has joined #tripleo | 05:16 | |
| *** rcernin has joined #tripleo | 05:17 | |
| *** skramaja has joined #tripleo | 05:18 | |
| openstackgerrit | Merged openstack/tripleo-ui: Treat eslint warnings as jenkins failures https://review.openstack.org/368152 | 05:23 |
| skramaja | op stack list | 05:27 |
| jaosorior | WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils | 05:27 |
| jaosorior | +--------------------------------------+------------+--------------------+----------------------+----------------------+ | 05:27 |
| jaosorior | | ID | Stack Name | Stack Status | Creation Time | Updated Time | | 05:27 |
| jaosorior | +--------------------------------------+------------+--------------------+----------------------+----------------------+ | 05:27 |
| jaosorior | | 9175bb2c-60c2-4b06-ba64-0e8e782ba688 | overcloud | UPDATE_IN_PROGRESS | 2016-09-14T14:09:10Z | 2016-09-15T05:19:46Z | | 05:27 |
| jaosorior | +--------------------------------------+------------+--------------------+----------------------+----------------------+ | 05:28 |
| jaosorior | skramaja: ^^ | 05:28 |
| jaosorior | :P | 05:28 |
| skramaja | jaosorior: :) | 05:28 |
| *** fragatina has joined #tripleo | 05:32 | |
| *** fragatina has quit IRC | 05:37 | |
| *** tzumainn has quit IRC | 05:44 | |
| *** oshvartz has joined #tripleo | 05:49 | |
| jaosorior | mcornea: hey dude | 05:59 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add VNC Proxy common manifest https://review.openstack.org/370541 | 06:00 |
| mcornea | jaosorior: heya | 06:00 |
| jaosorior | mcornea: Do you remember a bug where the VNC URL is set to http in SSL deployments? | 06:00 |
| mcornea | jaosorior: this one? https://bugzilla.redhat.com/show_bug.cgi?id=1372678 | 06:01 |
| openstack | bugzilla.redhat.com bug 1372678 in rhel-osp-director "On SSL enabled overcloud the novnc URL gets configured with http protocol instead of https" [Urgent,Post] - Assigned to josorior | 06:01 |
| jaosorior | mcornea: yep!\ got a fix https://review.openstack.org/#/c/370331/ https://review.openstack.org/#/c/370541/ :D | 06:01 |
| jaosorior | if you have time can you try it out? | 06:01 |
| jaosorior | mcornea: I did it in my deployment and it seems to work | 06:01 |
| mcornea | jaosorior: sure, I'll try them | 06:01 |
| jaosorior | mcornea: how's it going over there dude? | 06:01 |
| mcornea | jaosorior: pretty good, grabbing some coffee to start the day :D | 06:03 |
| jaosorior | mcornea: by the way, where are you based at? | 06:04 |
| mcornea | jaosorior: i'm usually working from Bucharest, Romania and sometimes from Brno | 06:04 |
| *** pcaruana has joined #tripleo | 06:04 | |
| jaosorior | mcornea: ah, that explains why you're online so early | 06:05 |
| jaosorior | mcornea: I think we're in the same timezone | 06:05 |
| *** karthiks has joined #tripleo | 06:06 | |
| mcornea | jaosorior: well, it's 9am here | 06:06 |
| jaosorior | same | 06:06 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 06:10 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** karthiks has quit IRC | 06:10 | |
| *** rasca has joined #tripleo | 06:12 | |
| *** jtomasek has joined #tripleo | 06:12 | |
| *** pcaruana is now known as pcaruana|afk| | 06:13 | |
| jaosorior | mcornea: crap, came up with another bug | 06:14 |
| mcornea | jaosorior: which one? | 06:14 |
| jaosorior | mcornea: https://bugs.launchpad.net/tripleo/+bug/1623796 | 06:16 |
| openstack | Launchpad bug 1623796 in tripleo "VNC Proxy is not being enabled in haproxy" [Undecided,New] | 06:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix wrong flag name for VNC Proxy in HAProxy https://review.openstack.org/370552 | 06:17 |
| jaosorior | mcornea: gonna try that as a fix ^^ | 06:18 |
| mcornea | jaosorior: ok, adding it as well | 06:18 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix wrong flag name for VNC Proxy in HAProxy https://review.openstack.org/370552 | 06:19 |
| * d0ugal drinks coffee and attempts to understand where we are with https://bugs.launchpad.net/tripleo/+bug/1623606 | 06:23 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 06:23 |
| jaosorior | d0ugal: I haven't checked that part out | 06:23 |
| jaosorior | seems like quite a bummer. Have people beena able to reproduce that one locally? | 06:23 |
| d0ugal | jaosorior: not that I am aware of | 06:24 |
| d0ugal | jaosorior: I couldn't yesterday | 06:24 |
| jaosorior | d0ugal: maybe there was a mistral update? | 06:24 |
| d0ugal | jaosorior: Yeah, maybe, that is one possible option but I tried with the same version CI had | 06:25 |
| jaosorior | whaaa | 06:25 |
| jaosorior | ok | 06:25 |
| jaosorior | funky | 06:25 |
| d0ugal | and it only happened on specific jobs | 06:26 |
| d0ugal | the multinode for example | 06:26 |
| d0ugal | but not the other | 06:26 |
| d0ugal | I think | 06:26 |
| jaosorior | ah oik | 06:26 |
| * d0ugal may be getting confused which passed and which failed | 06:26 | |
| jaosorior | yeah, the ovb jobs are broken because of something else | 06:26 |
| d0ugal | lol | 06:26 |
| *** absubram has joined #tripleo | 06:26 | |
| jaosorior | d0ugal: http://lists.openstack.org/pipermail/openstack-dev/2016-September/103710.html | 06:27 |
| d0ugal | jaosorior: Thanks | 06:27 |
| d0ugal | jaosorior: I hadn't dared open my email yet :) | 06:27 |
| *** absubram_ has joined #tripleo | 06:28 | |
| * d0ugal rebuilds both his underclouds | 06:29 | |
| *** absubram has quit IRC | 06:31 | |
| openstackgerrit | Merged openstack/tripleo-ui: Update Nodes listing https://review.openstack.org/365580 | 06:31 |
| bkero | howdy | 06:33 |
| *** absubram_ has quit IRC | 06:34 | |
| openstackgerrit | Merged openstack/tripleo-ui: ModalPanel component https://review.openstack.org/366615 | 06:38 |
| *** absubram has joined #tripleo | 06:38 | |
| *** sshnaidm|afk is now known as sshnaidm | 06:40 | |
| *** pcaruana|afk| is now known as pcaruana | 06:40 | |
| jaosorior | mcornea: works for me. With those three commits | 06:41 |
| *** nyechiel has joined #tripleo | 06:41 | |
| *** rwsu has joined #tripleo | 06:44 | |
| mcornea | jaosorior: ok, I'll get back with my results once my deployment is finished | 06:45 |
| *** akuznetsov has joined #tripleo | 06:47 | |
| *** bana_k has quit IRC | 06:49 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Stop using deprecated network range https://review.openstack.org/343443 | 06:51 |
| *** jlinkes has joined #tripleo | 06:53 | |
| *** dciabrin has joined #tripleo | 06:55 | |
| *** absubram has quit IRC | 06:58 | |
| *** dsariel has joined #tripleo | 07:00 | |
| *** liverpooler has joined #tripleo | 07:01 | |
| *** thegodfather is now known as fabbione | 07:02 | |
| *** paramite has joined #tripleo | 07:03 | |
| *** florianf has quit IRC | 07:05 | |
| *** jlinkes has quit IRC | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 07:10 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| *** mcornea has quit IRC | 07:11 | |
| *** jlinkes has joined #tripleo | 07:11 | |
| *** florianf has joined #tripleo | 07:14 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move keystone::auth into global settings https://review.openstack.org/370573 | 07:20 |
| *** aufi has joined #tripleo | 07:20 | |
| *** shardy has joined #tripleo | 07:20 | |
| *** jpena|off is now known as jpena | 07:24 | |
| *** mcornea has joined #tripleo | 07:24 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE: Test oooq with osinfra https://review.openstack.org/370437 | 07:27 |
| *** abregman_ has joined #tripleo | 07:28 | |
| *** abregman has quit IRC | 07:30 | |
| *** abregman_ has quit IRC | 07:30 | |
| *** abregman has joined #tripleo | 07:32 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix dependencies for HAProxy when certmonger is used https://review.openstack.org/370577 | 07:33 |
| *** mcornea has quit IRC | 07:34 | |
| d0ugal | Is anyone else looking into https://bugs.launchpad.net/tripleo/+bug/1623606 ? | 07:35 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 07:35 |
| *** jpich has joined #tripleo | 07:36 | |
| *** chem has joined #tripleo | 07:36 | |
| *** mcornea has joined #tripleo | 07:36 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix dependencies for HAProxy when certmonger is used https://review.openstack.org/370577 | 07:36 |
| *** zoli_gone-proxy is now known as zoliXXL | 07:38 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing https://review.openstack.org/370580 | 07:39 |
| *** abregman is now known as abregman|mtg | 07:39 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix dependencies for HAProxy when certmonger is used https://review.openstack.org/370577 | 07:43 |
| *** milan has joined #tripleo | 07:44 | |
| therve | d0ugal, Having a look. I wonder about https://review.openstack.org/#/c/356343/ | 07:48 |
| d0ugal | therve: Any particular reason? | 07:49 |
| d0ugal | therve: It was merged 7 days ago - would we not have seen this earlier? | 07:49 |
| therve | d0ugal, https://review.openstack.org/#/c/356343/12/mistral/engine/rpc_backend/oslo/oslo_server.py Mostly | 07:49 |
| therve | Possibly. It seems to be in the previous build indeed | 07:49 |
| d0ugal | therve: ooh | 07:49 |
| d0ugal | That certainly does look suspicious :) | 07:50 |
| *** zoliXXL is now known as zoli_gone-proxy | 07:50 | |
| d0ugal | I have no idea how we would revert something like that and have TripleO CI check it. | 07:50 |
| jaosorior | d0ugal: well, we've pinned packages from other services before. I believe you gotta so it in the tripleo-ci repo.... not sure though | 07:51 |
| jaosorior | shardy: how do we pin mistral? It might be the culprit of the cI breakages | 07:51 |
| d0ugal | Everything might be the culprit :-D | 07:51 |
| jaosorior | daaaamn | 07:52 |
| *** jbcraig has quit IRC | 07:53 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-quickstart: WIP gate upgrade https://review.openstack.org/342161 | 07:54 |
| *** dtantsur|afk is now known as dtantsur | 07:55 | |
| shardy | jaosorior: git log --oneline | grep pin in tripleo-ci will show you examples | 07:56 |
| *** zoli_gone-proxy is now known as zoliXXL | 07:56 | |
| jaosorior | d0ugal: ^^ | 07:57 |
| *** zoliXXL is now known as zoli|trng | 07:57 | |
| shardy | d0ugal, therve: we use mistral (and other services) from the current-tripleo pin in CI | 07:58 |
| shardy | that was promoted one day ago, so it's entirely possible we only then got a commit from last week | 07:58 |
| shardy | https://dashboards.rdoproject.org/rdo-dev | 07:58 |
| shardy | in theory we should never promote the current-tripleo pin to a broken version, as we promote based on a periodic CI job passing | 07:59 |
| shardy | sounds like that didn't catch an issue tho | 07:59 |
| d0ugal | shardy: It isn't happening 100% of the time. | 07:59 |
| *** dbecker_ has joined #tripleo | 08:00 | |
| *** dbecker_ has quit IRC | 08:00 | |
| *** dbecker has quit IRC | 08:00 | |
| shardy | the other way to test this is to post a WIP revert to mistral, then do a cosmetic patch to any TripleO repo with a Depends-On | 08:00 |
| therve | Yeah it's intermittent, so it's highly possible it passed promotion | 08:00 |
| d0ugal | shardy: Is there a way I can tell CI to try the older version? | 08:00 |
| shardy | d0ugal: Yeah ^^ or a WIP patch to tripleo-ci with a pinned version | 08:00 |
| d0ugal | shardy: huh, cool - I did not know that worked. | 08:00 |
| shardy | yeah, we'll just build a delorean package based on the WIP review | 08:00 |
| d0ugal | therve: I'll give it a go, unless you want to do it | 08:00 |
| therve | d0ugal, Please do so! | 08:01 |
| *** dbecker has joined #tripleo | 08:01 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing reverted Mistral https://review.openstack.org/370596 | 08:03 |
| *** eggmaster has quit IRC | 08:03 | |
| *** eggmaster has joined #tripleo | 08:03 | |
| *** akuznetsov has quit IRC | 08:05 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Move rabbit's clustering port away from the ephemeral port range https://review.openstack.org/345851 | 08:07 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 08:10 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing reverted Mistral https://review.openstack.org/370596 | 08:10 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Test a delay to check it isn't a race condition between calls https://review.openstack.org/370600 | 08:12 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Test that plan updating isn't the issue https://review.openstack.org/370580 | 08:13 |
| * d0ugal starts a smoke test by putting up a bunch of tests to try and elimiate different things | 08:13 | |
| *** yamahata has joined #tripleo | 08:13 | |
| *** ohamada has joined #tripleo | 08:14 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:14 |
| *** ccamacho has joined #tripleo | 08:17 | |
| *** panda|Zz is now known as panda | 08:18 | |
| *** mbound has joined #tripleo | 08:20 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Test removing template processing at creation time https://review.openstack.org/370612 | 08:31 |
| jaosorior | shardy: do you know if we set any proxy values for the undercloud in CI? | 08:34 |
| *** jlinkes has quit IRC | 08:35 | |
| shardy | jaosorior: I believe we do, grep proxy in tripleo-ci shows where/how | 08:38 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing to find the exact location of the error in tripleoclient https://review.openstack.org/370618 | 08:40 |
| *** electrofelix has joined #tripleo | 08:43 | |
| jaosorior | shardy: thanks | 08:43 |
| d0ugal | This is weird. Why didn't CI bomb out at this point? http://logs.openstack.org/80/370580/1/check/gate-tripleo-ci-centos-7-nonha-multinode/a8489ce/console.html#_2016-09-15_08_03_49_972644 | 08:43 |
| *** jlinkes has joined #tripleo | 08:44 | |
| *** akrivoka has joined #tripleo | 08:45 | |
| *** jbcraig has joined #tripleo | 08:45 | |
| *** derekh has joined #tripleo | 08:46 | |
| panda | d0ugal: probably because it's "likely" not "surely" and the only way to be sure is deploy anyway | 08:48 |
| *** abehl has joined #tripleo | 08:50 | |
| d0ugal | panda: ah, but when I run it on my machine it does fail then - how can I see what CI passes to overcloud deploy? | 08:50 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Also set Haproxy addresses in no_proxy https://review.openstack.org/370623 | 08:57 |
| panda | d0ugal: checking, not sure the command is logged, the alternative is to look at the builder invocation in the job. | 08:57 |
| d0ugal | panda: Thanks, I was surprised it wasn't logged. | 08:58 |
| *** abregman_ has joined #tripleo | 09:00 | |
| *** abregman|mtg has quit IRC | 09:02 | |
| *** phpcodemonkey has joined #tripleo | 09:03 | |
| panda | d0ugal: nope, not logged, I'll propose a patch for this. Checking the invocation then. | 09:04 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing to find the exact location of the error in tripleoclient https://review.openstack.org/370618 | 09:04 |
| *** phpcodemonkey has quit IRC | 09:07 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing reverted Mistral https://review.openstack.org/370596 | 09:07 |
| therve | d0ugal, I don't think just changing the execution engine is enough | 09:08 |
| d0ugal | therve: The full revert had the same error, so I thought I'd try | 09:08 |
| *** jtomasek has quit IRC | 09:08 | |
| therve | d0ugal, Did it? I couldn't see any result | 09:08 |
| *** jtomasek has joined #tripleo | 09:09 | |
| d0ugal | therve: http://logs.openstack.org/96/370596/2/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/7cf5240/logs/var/log/mistral/mistral-server.txt.gz#_2016-09-15_08_41_56_960 | 09:09 |
| d0ugal | I didn't wait for jenkins to post the result | 09:09 |
| therve | Ah, okay | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 09:10 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Provide more information when 'node provide' fails https://review.openstack.org/367553 | 09:11 |
| *** mbound has quit IRC | 09:12 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: tripleo.sh: log overcloud deploy command arguments before deploy https://review.openstack.org/370632 | 09:15 |
| *** fzdarsky has joined #tripleo | 09:16 | |
| *** pmannidi has quit IRC | 09:19 | |
| d0ugal | huh, has the multinode job gone? | 09:21 |
| d0ugal | I didn't get it here: https://review.openstack.org/#/c/370596/ | 09:21 |
| therve | d0ugal, I just noticed that https://review.openstack.org/#/c/370457/1/elements/puppet-stack-config/puppet-stack-config.pp didn't work AFAICT | 09:23 |
| therve | http://logs.openstack.org/57/370457/1/check/gate-tripleo-ci-centos-7-nonha-multinode/11f23a3/logs/etc/mistral/mistral.conf.txt.gz doesn't contain the change | 09:23 |
| *** tremble has joined #tripleo | 09:24 | |
| akrivoka | d0ugal: could you point me to the implementation of mistral nova actions? (the ones I get with mistral action-list | grep nova) | 09:25 |
| shardy | akrivoka: They're automatically generated from python-novaclient I think | 09:27 |
| shardy | https://github.com/openstack/mistral/blob/master/mistral/actions/generator_factory.py | 09:27 |
| shardy | so if you look at the novalcient Client object interfaces, I think that shows how to drive the nova actions | 09:27 |
| akrivoka | it seems that mistral flavor_get action does not return all the information about the flavor | 09:28 |
| akrivoka | you can see here that the capabilities:profile is not returned by the mistral action, while it is returned by nova flavor-show https://paste.fedoraproject.org/428034/87165614/ | 09:28 |
| akrivoka | the 'extra_specs' is not contained in the response from the mistral action | 09:30 |
| jaosorior | akrivoka: is it available if you try to fetch that info through novaclient? | 09:30 |
| panda | d0ugal: I'm getting these arguments for the nonha-multinode job from the builder script (assembling the pieces by hand) -e $TRIPLEO_ROOT/tripleo-ci/test-environments/enable-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml -e $TRIPLEO_ROOT/tripleo-ci/test-environments/inject-trust-anchor-hiera.yaml --ceph-storage-scale 1 -e | 09:31 |
| panda | /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml --libvirt-type=qemu -t $TIMEOUT -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml TRIPLEO_ROOT/tripleo-ci/test-environments/multinode.yaml --compute-scale 0 --overcloud-ssh-user jenkins --validation-errors-nonfatal | 09:31 |
| akrivoka | jaosorior: I'm having problems getting novaclient to authenticate which is why I am now trying to do it via mistral action instead of nova client... | 09:31 |
| panda | d0ugal: but yeah, the patch to log the command is proposed. | 09:31 |
| akrivoka | jaosorior: I wanted to add novaclient here https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/base.py but it seems it only accepts username/password auth and not auth_token like the rest of the clients in that file... | 09:33 |
| jaosorior | akrivoka: well, mistral autogenerates a lot of actions with a json mapping, so here's flavors_get for isntance https://github.com/openstack/mistral/blob/master/mistral/actions/openstack/mapping.json#L63 | 09:33 |
| jaosorior | akrivoka: does it take a session? | 09:34 |
| jaosorior | akrivoka: you can build a keystone session from the auth_token, and pass that instead | 09:35 |
| akrivoka | jaosorior: is there example somewhere? | 09:35 |
| openstackgerrit | Jiri Stransky proposed openstack/instack-undercloud: [NO MERGE] test CI https://review.openstack.org/370647 | 09:36 |
| jaosorior | akrivoka: right, so novaclient can take both a session and a keystone auth plugin https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/client.py#L64 they are the session and auth parameters | 09:37 |
| jaosorior | akrivoka: but... what do you mean it doesn't take an auth_token? Surely seems to be there in the list of parameters | 09:38 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: TESTING https://review.openstack.org/370648 | 09:38 |
| d0ugal | akrivoka, shardy: https://github.com/openstack/mistral/blob/master/mistral/actions/openstack/mapping.json#L3 | 09:39 |
| d0ugal | That file is useful to see what calls what. | 09:39 |
| d0ugal | therve: oh, weird. Maybe I did something wrong. | 09:39 |
| therve | d0ugal, How so? | 09:40 |
| d0ugal | therve: not sure, I don't really understand :-D | 09:40 |
| d0ugal | panda: Thanks - I'll test with them | 09:40 |
| therve | Ah jistr made the exact same comment :) | 09:41 |
| d0ugal | lol | 09:41 |
| jistr | heh | 09:41 |
| d0ugal | It's only 10:30am and my brain is already fried :( | 09:41 |
| panda | BRunch | 09:42 |
| *** stendulker has joined #tripleo | 09:42 | |
| *** tosky has joined #tripleo | 09:44 | |
| therve | jistr, It feels like it could be a bug in puppet-mistral. Is there a way to change that config without puppet? | 09:48 |
| *** bandini has quit IRC | 09:48 | |
| *** yamahata has quit IRC | 09:49 | |
| jistr | i think we could hack it with sed perhaps | 09:49 |
| jistr | but the puppet-mistral implementation of mistral_config looks fairly standard https://github.com/openstack/puppet-mistral/blob/master/lib/puppet/provider/mistral_config/ini_setting.rb | 09:49 |
| jistr | i wonder if CI tested the submitted change at all | 09:50 |
| jistr | i saw mentions of delorean running on correct commit hash in the console.txt log | 09:50 |
| jistr | but still wondering if the code changes really took effect in the end | 09:51 |
| jistr | that's why i submitted the test patch that deletes the .pp file altogether -- if it doesn't fail on undercloud install that the file doesn't exist, we have a problem in the CI | 09:52 |
| * jistr gonna grab something to eat, biab | 09:53 | |
| *** gfidente has joined #tripleo | 09:57 | |
| akrivoka | jaosorior: it does seem to take the auth_token, but I can figure out the right combination of the parameters https://paste.fedoraproject.org/428291/47393347/ | 09:58 |
| akrivoka | jaosorior: also I couldn't find any other example using auth_token with novaclient, and this bug suggests it is not possible https://bugs.launchpad.net/python-novaclient/+bug/1197746 | 09:59 |
| openstack | Launchpad bug 1197746 in python-cinderclient "token + service_url based authentication" [Undecided,Confirmed] | 09:59 |
| *** gfidente has quit IRC | 10:00 | |
| shardy | jaosorior: Hey, see my comment on https://review.openstack.org/#/c/370573/ | 10:05 |
| shardy | jaosorior: I do agree with your comment, but the patch actually doesn't make things any less secure than what we have now | 10:05 |
| shardy | I'd like to see it improve, but that will be an ocata thing I think | 10:05 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 10:10 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| therve | d0ugal, http://logs.openstack.org/96/370596/2/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/7cf5240/logs/var/log/mistral/mistral-server.txt.gz#_2016-09-15_08_38_27_437 | 10:11 |
| therve | The mistral change wasn't taken into account | 10:11 |
| therve | (From the log you paste to me earlier) | 10:11 |
| *** gfidente has joined #tripleo | 10:12 | |
| *** gfidente has quit IRC | 10:12 | |
| *** gfidente has joined #tripleo | 10:12 | |
| d0ugal | therve: Dang, any idea why? | 10:12 |
| akrivoka | jaosorior: tried with keystone session now (as described here http://docs.openstack.org/developer/python-keystoneclient/using-api-v3.html) but now I'm getting NotFound when I try to list flavors https://paste.fedoraproject.org/428299/47393422/ | 10:12 |
| therve | d0ugal, Well maybe shardy lied to you :) | 10:12 |
| d0ugal | lol | 10:12 |
| akrivoka | lol I'm probably missing something very obvious here... | 10:12 |
| therve | Depends-On doesn't work always as we expect to | 10:13 |
| d0ugal | therve: I didn't think it done anything as clever, it never seemed to work between python-tripleoclient and tripleo-common | 10:13 |
| *** snecklifter has quit IRC | 10:14 | |
| shardy | hehe, it should work, otherwise our CI is broken | 10:15 |
| therve | Well, now that you mention it :D | 10:16 |
| d0ugal | https://media.giphy.com/media/3o6UBpHgaXFDNAuttm/giphy.gif | 10:16 |
| shardy | Where does gate-tripleo-ci-centos-7-nonha-multinode-updates-nv come from? I don't see the job linked from the comments on the review? | 10:18 |
| shardy | And ZUUL_REFS look wrong in the logs | 10:18 |
| shardy | https://review.openstack.org/#/c/370596/ | 10:18 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing to find the exact location of the error in tripleoclient https://review.openstack.org/370618 | 10:19 |
| shardy | http://logs.openstack.org/96/370596/2/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/7cf5240/console.html#_2016-09-15_08_14_33_992865 | 10:19 |
| therve | shardy, https://review.openstack.org/#/c/370647/ is super frightening | 10:20 |
| therve | It should be super duper red | 10:21 |
| d0ugal | therve: lol what. | 10:21 |
| *** fzdarsky has quit IRC | 10:24 | |
| jistr | hmm so it's an issue with the CI toolchain indeed | 10:27 |
| *** cwolferh has quit IRC | 10:27 | |
| *** ramishra has quit IRC | 10:32 | |
| *** hjensas has joined #tripleo | 10:32 | |
| jaosorior | shardy: it still doesn't mean that we will dump passwords everywhere in the cloud | 10:32 |
| jaosorior | I think a solution can still come | 10:32 |
| *** ramishra has joined #tripleo | 10:33 | |
| shardy | jaosorior: We dump all kinds of sensitive information on the nodes already, but I see your point | 10:35 |
| *** fzdarsky has joined #tripleo | 10:35 | |
| shardy | I'll give it some thought and see how we might filter the data to enable mapping it onto the roles that need it | 10:36 |
| jaosorior | shardy: did you see my last comment on that CR? | 10:36 |
| shardy | perhaps we can have a mapping of service_settings where we look for e.g "keystone" and wire that in only to the role running keystone | 10:36 |
| shardy | jaosorior: ack, yeah my idea is similar to that but without the hard-coded keys | 10:37 |
| shardy | service_settingsL | 10:37 |
| shardy | keystone: {map of keystone stuff} | 10:37 |
| jaosorior | shardy: which hard-coded keys? | 10:37 |
| shardy | and as you say mangle it with yaql to get it onto the right role | 10:37 |
| shardy | keystone_config_settings: foo | 10:37 |
| jaosorior | aaah I see the difference now | 10:38 |
| jaosorior | well, I like it | 10:38 |
| shardy | I'd prefer we didn't special-case keystone, as I'm sure it won't be the only one | 10:38 |
| shardy | jaosorior: Ok, I'll comment on the review | 10:38 |
| jaosorior | shardy: actually haproxy would need something similar | 10:38 |
| jaosorior | so yeah | 10:38 |
| shardy | thanks for the feedback | 10:38 |
| jaosorior | I like that solution | 10:38 |
| shardy | if we had this, do we even need global_config_settings? | 10:38 |
| *** dtantsur is now known as dtantsur|bbl | 10:39 | |
| *** snecklifter has joined #tripleo | 10:39 | |
| jaosorior | shardy: I don't see a use for it now | 10:39 |
| jaosorior | shardy: do we have something like deep merge? | 10:39 |
| jaosorior | we would need it for haproxy https://review.openstack.org/#/c/355366/9/puppet/services/haproxy.yaml | 10:39 |
| shardy | jaosorior: Yes, it is possible with yaql, but pretty ugly: | 10:41 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.j2.yaml#L293 | 10:41 |
| shardy | jaosorior: also see the thread I started on openstack-dev about it where a few other ideas were discussed | 10:42 |
| shardy | jaosorior: during ocata I expect we'll add a map_deep_merge option to heat | 10:42 |
| jaosorior | shardy: alright, so an "ugly" version in the meantime is no big deal | 10:43 |
| shardy | yeah, I expect over time we'll gradually rework things to remove the scary yaql, but it's an OK interim solution | 10:43 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Deployment Plan page wizard styling https://review.openstack.org/369974 | 10:43 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Add Role Detail https://review.openstack.org/367993 | 10:43 |
| jpich | akrivoka: Are you still trying to resolve that issue? I think I got the auth sorted, starting from the example in your first paste | 10:45 |
| akrivoka | jpich: yep still trying... can you share the code that's working for you? | 10:45 |
| jaosorior | shardy: could you spare a minute for a review? Seems I missed a spot when fixing the vnc-poxy naming https://review.openstack.org/#/c/370552/ I tried it out in my deployment and mcornea was also gonna try it out | 10:45 |
| shardy | therve, jistr: Yeah something is very wrong - look in the undercloud install log - we're not installing the instack-undercloud package we built with delorean in CI | 10:48 |
| shardy | I don't have a good answer for why atm | 10:48 |
| jpich | akrivoka: Sure! Obviously it needs to be cleaned up a tad :-) http://paste.openstack.org/show/577367/ | 10:48 |
| jistr | i wonder if this is normal when building just one package http://logs.openstack.org/47/370647/1/check/gate-tripleo-ci-centos-7-undercloud/da9f360/console.html#_2016-09-15_09_51_22_978960 | 10:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add CephRgw to roles_data.yaml https://review.openstack.org/370687 | 10:48 |
| jpich | akrivoka: aka I don't know why this is needed but obviously other projects hit the same issue | 10:49 |
| *** chem` has joined #tripleo | 10:49 | |
| jistr | also we're building a diskimage-builder RPM too, even though the patches don't depend on a change in diskimage-builder http://logs.openstack.org/47/370647/1/check/gate-tripleo-ci-centos-7-undercloud/da9f360/console.html#_2016-09-15_09_51_25_758632 | 10:49 |
| jistr | shardy, therve ^ | 10:49 |
| Jokke_ | marios: ping | 10:49 |
| jistr | it might be that it is normal/correct too though | 10:49 |
| *** chem has quit IRC | 10:50 | |
| shardy | jaosorior: Done - please can you self-triage bugs and target them to a milestone when reporting? | 10:50 |
| shardy | jistr: Ah! Are we running tripleo.sh --build-images twice to get a newer DIB? | 10:52 |
| shardy | the second run will delete all packages built by the first one | 10:52 |
| akrivoka | jpich: amazing | 10:53 |
| akrivoka | thank you jpich | 10:53 |
| jistr | shardy: we're running tripleo.sh --delorean-build twice | 10:53 |
| shardy | https://github.com/openstack-infra/tripleo-ci/commit/c96a620fe861eea43c4989cc42aaf91ce67581d9 | 10:53 |
| shardy | this is the problem | 10:53 |
| shardy | we need to copy ~/tripleo/delorean/data/repos/current/* somewhere before running it again | 10:53 |
| jpich | akrivoka: Happy if it helped! :) | 10:53 |
| shardy | then copy the packages back to where they need to be for the tripleo-ci repo | 10:54 |
| akrivoka | jpich: I owe you one | 10:54 |
| shardy | I need to go get a flight soon, can someone copy this info into a bug, mark it critical and look at fixing that script? | 10:54 |
| jistr | shardy: yea i'll do that | 10:55 |
| shardy | jistr: awesome, thanks! | 10:55 |
| shardy | Unfortunately, that means we've not been testing anything in CI for two days :( | 10:55 |
| *** apetrich has quit IRC | 10:56 | |
| jaosorior | d0ugal: ^^ | 10:56 |
| *** rbrady has joined #tripleo | 10:56 | |
| *** rbrady has quit IRC | 10:56 | |
| *** rbrady has joined #tripleo | 10:56 | |
| * shardy heads off for flight, bbl | 10:57 | |
| *** shardy has quit IRC | 10:57 | |
| *** electrofelix has quit IRC | 11:01 | |
| jistr | fyi here's the bug, looking into it, i'm hoping i can submit a patch for it shortly https://bugs.launchpad.net/tripleo/+bug/1623891 | 11:04 |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 11:04 |
| *** stendulker has quit IRC | 11:04 | |
| derekh | jistr: https://review.openstack.org/#/c/370281/ | 11:05 |
| derekh | jistr: we figured that out yesterday, but we can't merge the fix because the multinode job keeps failing the gate, here https://review.openstack.org/#/c/370250/ | 11:05 |
| *** lucasagomes is now known as lucas-hungry | 11:06 | |
| d0ugal | jaosorior: d'oh | 11:07 |
| gfidente | d0ugal any luck with the file:// connector yesterday? | 11:07 |
| jaosorior | derekh: what about that fix to tripleo-ci + pinning os-collect-config in tripleo-ci? | 11:07 |
| d0ugal | gfidente: lol | 11:07 |
| marios | Jokke_: o/ | 11:07 |
| jistr | derekh: is it safe to remove that build now? (i was going to just merge the two calls to --delorean-build) | 11:07 |
| d0ugal | gfidente: everything else has been on fire, so I've not been able to even look at that. | 11:07 |
| gfidente | d0ugal fine, I have commented on the LP bug where the problem seems to come from, will continue investigation later today | 11:08 |
| jistr | derekh: i mean make the two calls a single one | 11:08 |
| *** bkopilov has quit IRC | 11:08 | |
| d0ugal | gfidente: Thanks! | 11:09 |
| derekh | jistr: we can just remove the second call to DIB , its no longer needed as DIB has been released | 11:09 |
| d0ugal | gfidente: I have been looking into CI issues, seems our CI was totally bust | 11:09 |
| derekh | jistr: the problem is we can't merge anything because of a multinode bug | 11:09 |
| derekh | jistr: see the email from EmilienM on the list "[openstack-dev] [tripleo] CI is currently down: 2 blockers" | 11:09 |
| jistr | yea i've read that | 11:10 |
| d0ugal | derekh: but we can't figure out why the multinode gate is failing. | 11:10 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 11:10 |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 11:10 |
| d0ugal | derekh: so should it just be removed temporarily? | 11:10 |
| *** apetrich has joined #tripleo | 11:11 | |
| Jokke_ | marios: I'm trying to figure out what's going on with that Manila change | 11:11 |
| derekh | d0ugal: we could, if that what people wanted to do. slagle thoughts ? (when your in) | 11:12 |
| Jokke_ | marios: spoke with gfidente just that there was reason those things were in the puppet-tripleo | 11:12 |
| marios | Jokke_: thanks yeah i updated both reviews with a comment (had the same question for the netapp puppet-tripleo). | 11:13 |
| d0ugal | derekh: I'm not sure, but I am about to give up looking into the multinode bug, because I have no idea what else to try :( | 11:13 |
| *** paramite has quit IRC | 11:14 | |
| slagle | d0ugal: i'm not in favor of disabling the job | 11:15 |
| slagle | it's failing for a reason | 11:15 |
| jistr | derekh: is there a way to force-merge something even though gate is red on it? E.g. give a depends-on on slagle's fix for the multinode Mistral timeout, so that it pulls in the DIB building revert too, and if it passes, we'd force-merge the DIB build revert, because it was tested by the patch on top of it. | 11:16 |
| slagle | why does it matter that the DIB fix goes in when the gate is failing anyway? | 11:16 |
| jistr | slagle: because without it the gate will always fail. Anything that we submit is never tested, the tests run on master --> always red. | 11:17 |
| derekh | jistr: I'm not sure, having an actual gate is new to me | 11:17 |
| jistr | i.e. without the DIB fix there's no way we can ever fix CI, because whatever we submit isn't tested | 11:18 |
| slagle | jistr: the gate is failing due to a different bug, the mistral messaging timeout | 11:18 |
| slagle | when that is fixed, the dib fix will merge | 11:18 |
| derekh | slagle: do you know if the mistral messaging timeout bug is a problem in a tripleo project or $other? | 11:19 |
| slagle | derekh: i don't know | 11:19 |
| jistr | slagle: i think we're still not on the same page :) the fix for the mistral messaging timeout wasn't tested by the CI though. I think the bump you did to 600 could fix it, but CI still ran on 60 | 11:19 |
| jistr | slagle: and the reason it didn't use the 600 value is the DIB problem | 11:20 |
| jistr | slagle: see this https://review.openstack.org/#/c/370647/ | 11:20 |
| slagle | jistr: i don't think that's going to fix it tbh | 11:20 |
| slagle | jistr: when it passes, the rpc call takes < 1 second | 11:20 |
| jistr | ahh ok | 11:20 |
| slagle | jistr: but if we do want to try the increase, we can workaround it | 11:20 |
| slagle | by just sed'ing mistral.conf or something | 11:20 |
| derekh | slagle: if its it a tripleo project that needed to be fixed, then we fist need to remove the second call to DIB in order to successfully test the fix for the mistral timeout | 11:21 |
| slagle | jistr: after packages are installed | 11:21 |
| therve | slagle, It seems the DIB problem prevents from testing mistral changes, too | 11:21 |
| jistr | ok | 11:21 |
| therve | Or at least we weren't able to depend properly | 11:21 |
| mcornea | jaosorior: tested the vnc patches and it looks good | 11:21 |
| jaosorior | mcornea: thanks for checking them out dude | 11:21 |
| jaosorior | mcornea: if at some point you do a deployment where the undercloud doesn't have haproxy pre-installed. I also have a commit for that failure https://review.openstack.org/#/c/370577/ | 11:22 |
| jaosorior | mcornea: tested it in my environment. But if you have time that's there :D | 11:23 |
| mcornea | jaosorior: yep, I saw it, added to my list | 11:23 |
| jaosorior | mcornea: thanks dude, appreciate it | 11:23 |
| slagle | therve: we can fix the DIB problem. if the mistral issues is just raising the timeout, we can do it temporarily outside of instack-undercloud packaging | 11:24 |
| therve | slagle, Yeah I agree that the timeout is unlikely to be the issue | 11:25 |
| Jokke_ | marios: so is that hiera data there when we get the tht change in to shape and merged or are we in chicken-egg situation where we broke composibility by making it all more composible-like? ;) | 11:25 |
| therve | slagle, That's why we'd like to be able to test mistral code changes | 11:25 |
| slagle | therve: or the tripleo-ci patch that fixes the DIB issue, can depends-on a potential mistral patch that fixes it | 11:25 |
| therve | slagle, Ah, that's a good one | 11:25 |
| marios | Jokke_: the tht is fine fornetapp at least .. the cephfs needed a pass i left a comment. | 11:25 |
| marios | Jokke_: it is puppet-tripleo side where we need to update | 11:26 |
| marios | Jokke_: but we are already passing the things from tht | 11:26 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Testing to find the exact location of the error in tripleoclient https://review.openstack.org/370618 | 11:27 |
| *** skramaja has quit IRC | 11:30 | |
| *** fragatina has joined #tripleo | 11:30 | |
| *** skramaja has joined #tripleo | 11:30 | |
| *** thrash|g0ne is now known as thrash | 11:32 | |
| *** fragatina has quit IRC | 11:34 | |
| *** skramaja_ has joined #tripleo | 11:34 | |
| *** skramaja has quit IRC | 11:35 | |
| *** pgadiya has quit IRC | 11:35 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-common: Add node tagging workflow https://review.openstack.org/332132 | 11:36 |
| *** skramaja_ is now known as skramaja | 11:36 | |
| *** jbcraig has quit IRC | 11:37 | |
| *** jbcraig has joined #tripleo | 11:38 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-common: Add node tagging workflow https://review.openstack.org/332132 | 11:40 |
| *** fzdarsky has quit IRC | 11:41 | |
| *** skramaja_ has joined #tripleo | 11:42 | |
| *** skramaja has quit IRC | 11:42 | |
| *** jbcraig has quit IRC | 11:42 | |
| *** jlinkes has quit IRC | 11:42 | |
| *** pkovar has joined #tripleo | 11:43 | |
| *** panda is now known as panda|lunch | 11:44 | |
| akrivoka | d0ugal: I updated the node tagging workflow ^, just letting as you offered to test yesterday :) | 11:46 |
| thrash | d0ugal: are we missing more backports for stable/newton on oooclient? | 11:46 |
| thrash | d0ugal: http://paste.openstack.org/show/577385/ diff of stable/newton..master | 11:48 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Remove forced delorean dib build https://review.openstack.org/370724 | 11:49 |
| *** skramaja has joined #tripleo | 11:49 | |
| *** skramaja_ has quit IRC | 11:49 | |
| Jokke_ | marios: that's kind of what I mean. So if we had the THT change merged, hiera would provide the data and we did not have problem with puppet-tripleo, or do we need the defaults specified in the puppet-tripleo still, because we might not be providing the data always if the env is not provided? | 11:51 |
| *** bfournie has quit IRC | 11:51 | |
| d0ugal | akrivoka: k, I'll try and get to it at some point | 11:52 |
| akrivoka | d0ugal: thanks! | 11:52 |
| Jokke_ | marios: sorry if I'm being stupid with these questions, I just think I'm missing something in my understanding here. | 11:52 |
| d0ugal | thrash: Not sure. EmilienM squashed and backported stuff. | 11:52 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Migrate Deploy action to Mistral https://review.openstack.org/357125 | 11:52 |
| *** trown|outtypewww is now known as trown | 11:52 | |
| *** jlinkes has joined #tripleo | 11:52 | |
| thrash | d0ugal: right... forgot about that. gotta check that too. | 11:52 |
| d0ugal | thrash: CI is totally bust, so I'm not sure if any of that is still in flight | 11:53 |
| *** cwolferh has joined #tripleo | 11:53 | |
| marios | Jokke_: sorry doing couple things at same time... np with questions... so. the thing me and gfidente were discussing is that we will need to explicitly pass the config into the backend definitions in the puppet-tripleo side, like comments at https://review.openstack.org/#/c/354014/8/manifests/profile/pacemaker/manila.pp | 11:53 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Migrate Deploy action to Mistral https://review.openstack.org/357125 | 11:54 |
| marios | Jokke_: we already set them in hieradata from the tht, for example https://review.openstack.org/#/c/354019/17/puppet/services/manila-backend-generic.yaml or https://review.openstack.org/#/c/354019/17/puppet/services/manila-backend-netapp.yaml | 11:54 |
| Jokke_ | marios: yes, correct, but IIUC you tried to fix issue of not having those values we try to provide explicitely when you proposed that cleanup patch to remove them | 11:54 |
| marios | Jokke_: well, we don't need them to be class parameters anymore for sure | 11:55 |
| marios | Jokke_: we just need to pick them from hiera, like i do for the driver_handles_share_servers => hiera('manila::backend::generic::driver_handles_share_servers', true) | 11:55 |
| Jokke_ | marios: and the ceph tht part is not merged so we don't provide those values, which contributes to the original issues that part breaking? | 11:55 |
| marios | Jokke_: i am not really clear what the questino is sorry... | 11:56 |
| marios | Jokke_: maybe questino on the review and pointing at things would be clearer? | 11:57 |
| Jokke_ | marios: so iiuc the ps to remove those from the puppet-tripleo was to fix issue deploying manila without ceph (and those parameters not being set per default), right? So if the manila ceph backend tht change would be merged, that original issue would not have surfaced, as the hiera would have the defaults? | 11:58 |
| *** pgadiya has joined #tripleo | 11:59 | |
| marios | Jokke_: well it would surface if you didn't deploy cephfs still | 12:00 |
| marios | Jokke_: i mean the way the puppet-tripleo is now, if you try and deploy manila, any backend, but not cephfs you get the error afaik | 12:00 |
| marios | Jokke_: cos only when you include the cephfs do those values get passed | 12:01 |
| Jokke_ | marios: Cool, that was the part I was confused about | 12:01 |
| Jokke_ | marios: because I thought that would be the case based on how the last iteration of those patches were formatted | 12:01 |
| *** cwolferh has quit IRC | 12:02 | |
| *** jpena is now known as jpena|lunch | 12:02 | |
| Jokke_ | marios: so we need to pull all those things we have there, but instead of having them as class variables, we just pull them from hiera within that 'if ...' on line 67 of manifests/profile/pacemaker/manila.pp | 12:03 |
| Jokke_ | like you did with the "drive_handles_share_servers" on line 70 | 12:03 |
| *** jkraj has joined #tripleo | 12:04 | |
| *** cwolferh has joined #tripleo | 12:06 | |
| sshnaidm | derekh, hi, can I help with something in CI? | 12:06 |
| openstackgerrit | Lars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 12:06 |
| marios | jistr: can you checkout https://review.openstack.org/#/c/321027/23/extraconfig/tasks/major_upgrade_controller_pacemaker_1.sh when you get a sec please looking for bandini too but he isnt about atm | 12:06 |
| marios | jistr: i am sanity checking them finally | 12:07 |
| EmilienM | hello | 12:07 |
| jistr | marios: yea bandini os on his way home from Brno atm | 12:07 |
| jistr | s/os/is/ | 12:07 |
| *** rbrady has quit IRC | 12:08 | |
| EmilienM | guys we need https://review.openstack.org/#/c/370518/ | 12:08 |
| EmilienM | thrash: this is fixing our CI I'm pretty sure now | 12:08 |
| EmilienM | slagle: ^ | 12:08 |
| thrash | EmilienM: that's what it looks like to me. | 12:09 |
| EmilienM | yes | 12:10 |
| EmilienM | jaosorior: thanks dude | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 12:10 |
| EmilienM | let's bring back this CI | 12:10 |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 12:10 |
| jistr | marios: replied on the patch, i think it's ok (you'd run bandini's migration to NG HA prior to that i think) | 12:10 |
| marios | jistr: ah thanks yes... i am trying it manually | 12:10 |
| d0ugal | EmilienM: parameters set isn't used in CI | 12:10 |
| marios | jistr: makes sense now :) | 12:10 |
| marios | jistr: i also have to export mariadb_do_major_upgrade="no" | 12:11 |
| EmilienM | jistr, bnemec: https://review.openstack.org/#/c/369792/ | 12:11 |
| EmilienM | I already did that patch but no review :( | 12:11 |
| marios | jistr: so in that case, at least syntax etc is fine looks like (i have obviously run them in the past but been struggling to get to it lately) | 12:11 |
| *** rbrady has joined #tripleo | 12:11 | |
| marios | jistr: i'll update in a bit with some output | 12:11 |
| jistr | EmilienM: ah... we should have bugs for these things with alert tag, so that well we don't start over debugging the same thing from scratch :) | 12:12 |
| d0ugal | EmilienM, thrash - I don't understand how this fixes CI? https://review.openstack.org/#/c/370518/ | 12:12 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Throw an error/exception when interactive overcloud update fails https://review.openstack.org/370742 | 12:12 |
| EmilienM | jistr: what this thing is fixing? | 12:12 |
| *** pradk has joined #tripleo | 12:13 | |
| EmilienM | jistr: I'm confused why it breaks other packages | 12:13 |
| marios | jistr: but then that means we can't land them until that migration lands | 12:13 |
| jistr | EmilienM: i submitted mine just because there's uncertainty whether the rpc timeout bump solves the MessagingTimeout issue or not. Unfortunately the DIB fix is blocked by the MessagingTimeout problem and vice versa. It's a chicken and egg problem, so i'm trying to hack around it in the patch i posted. | 12:13 |
| marios | jistr: i mean if we remove openstack-core | 12:13 |
| jistr | just to see if we can fix it this way or not | 12:14 |
| EmilienM | jistr: what patch? | 12:14 |
| EmilienM | https://review.openstack.org/#/c/370724/ ok | 12:14 |
| jistr | yea that one | 12:14 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Removing locking hash on undercloud post role https://review.openstack.org/369870 | 12:14 |
| EmilienM | we have seen that https://review.openstack.org/#/c/370457/ has no effect | 12:15 |
| marios | jistr: well no, i mean mitaka already has its own upgrade scripts which are OK.. this would be for newton upgrade and it doesn't exist yet. so perhaps we could land them ... will add note anyway on review | 12:15 |
| EmilienM | mhh | 12:15 |
| jistr | EmilienM: just in CI or manually too? | 12:15 |
| jistr | EmilienM: it was never effectively tested in CI, because of the DIB problem | 12:15 |
| EmilienM | jistr: can you describre DIB problem? | 12:15 |
| EmilienM | describe even | 12:16 |
| EmilienM | again, I'm very confused about this packaging issue | 12:16 |
| jistr | EmilienM: yea it's in the bug description here https://bugs.launchpad.net/tripleo/+bug/1623891 | 12:16 |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 12:16 |
| jistr | EmilienM: https://github.com/openstack-infra/tripleo-ci/blob/4e6154027133f20dce4d300fc6701848000e9c3b/scripts/common_functions.sh#L266-L273 | 12:16 |
| jistr | EmilienM: ^^ the second --delorean-build "destroys" the results of the first one apparently | 12:17 |
| EmilienM | ok | 12:17 |
| EmilienM | that explains it now :) | 12:17 |
| EmilienM | damn | 12:17 |
| *** lucas-hungry is now known as lucasagomes | 12:17 | |
| EmilienM | d0ugal: to reply to your question, I don't know but we still need to backport all the things in tripleoclient | 12:17 |
| EmilienM | d0ugal: until final release is done | 12:17 |
| *** snecklifter has quit IRC | 12:18 | |
| d0ugal | EmilienM: Sure, I get that. I doubt it will fix CI tho' | 12:18 |
| jistr | marios: yea it should be all fine as long as we don't backport that :)) that's the approach we've been using for upgrades/migrations anyway | 12:18 |
| jistr | marios: by backport i mean backport to mitaka | 12:19 |
| *** pgadiya has quit IRC | 12:19 | |
| EmilienM | d0ugal: like I said, we're exploring everything at this stage :) | 12:19 |
| EmilienM | but I think jistr's patch might help | 12:19 |
| d0ugal | EmilienM: k, fair enough. I have been too. | 12:19 |
| d0ugal | hitting my head against all the walls :) | 12:20 |
| marios | jistr: yeah see what you are saying... OOK... i mean yeah... the 'newton upgrade' isnt a thing yet so if you *are* running it T&C apply I guess :) | 12:20 |
| jistr | marios: yea exactly... regardless which one we land first, it will not work in the interim step | 12:22 |
| jrist | jtomasek: I do have capabilities-map; I am using tripleo-heat-templates master | 12:25 |
| therve | jistr, sed: can't read /etc/mistral/mistral.conf: Permission denied :/ | 12:25 |
| jrist | jtomasek: I realized that my validations weren't supposed to be clapper based but I still have no validations with mistral | 12:25 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/os-cloud-config: Replace ucs_hostname with ucs_address https://review.openstack.org/370756 | 12:25 |
| jrist | jtomasek: this is holding up lots of my reviews of your work | 12:26 |
| jistr | therve: thanks. Amendment coming right now. | 12:26 |
| *** bfournie has joined #tripleo | 12:26 | |
| beagles | jrist: sorry to interrupt, but is there a pointer to how the capabilities-map.yaml file "works", i.e. what pulls it in, what mechanisms use that information? | 12:26 |
| jrist | beagles: only the code at the moment afaik | 12:27 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Remove forced delorean dib build https://review.openstack.org/370724 | 12:27 |
| beagles | jrist: k | 12:27 |
| *** panda|lunch is now known as panda | 12:27 | |
| jtomasek | jrist: the error message you sent says that the plan does not include capabilities-map.yaml | 12:27 |
| *** paramite has joined #tripleo | 12:27 | |
| jrist | jtomasek: but it is there | 12:28 |
| jrist | perhaps I also need your capabilities map patch | 12:28 |
| jtomasek | jrist: I don't think that is it | 12:28 |
| *** jayg|g0n3 is now known as jayg | 12:28 | |
| jtomasek | jrist: can you try to create new plan again? how did you create the tar file? | 12:29 |
| jtomasek | jrist: it needs to get created from inside the tripleo-heat-templates directory | 12:29 |
| EmilienM | jistr, slagle: about mistral timeout, i'll have to patch puppet-mistral first, because doing that in instack-undercloud will block me to do it in puppet module later (duplicated resource) | 12:30 |
| jrist | jtomasek: I created it "tar cvf plan.tar *" | 12:31 |
| jrist | jtomasek: within the dir | 12:31 |
| jrist | jtomasek: and then I gzipped it | 12:31 |
| jrist | just like the instructions say | 12:31 |
| therve | EmilienM, I'd wait for the test results | 12:32 |
| *** bkopilov has joined #tripleo | 12:32 | |
| EmilienM | therve: yes, let's see | 12:32 |
| jtomasek | jrist: I am going to try to reproduce it after demo | 12:33 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: When deploy finishes, show overcloud info https://review.openstack.org/370765 | 12:33 |
| *** ccamacho is now known as ccamacho|lunch | 12:35 | |
| *** snecklifter has joined #tripleo | 12:35 | |
| *** rhallisey has joined #tripleo | 12:36 | |
| *** jlinkes has quit IRC | 12:36 | |
| jrist | jtomasek: I'm concerned that I don't have akrivoka's default plan too | 12:37 |
| jrist | jtomasek: it just says | 12:37 |
| jrist | No Deployment Plans Available | 12:37 |
| jrist | There are no Deployment Plans available. Please create one first. | 12:37 |
| jrist | like it always has | 12:37 |
| d0ugal | jrist: Can you create one from the CLI? | 12:38 |
| d0ugal | jrist: openstack overcloud plan create overcloud | 12:38 |
| jrist | great question :) | 12:38 |
| d0ugal | jrist: openstack overcloud plan create overcloud --templates path/to/tht | 12:38 |
| d0ugal | jrist: use the second if you want custom templates, the first will use what is packaged on the undercloud | 12:39 |
| akrivoka | jrist: what does swift list say in the cli? | 12:40 |
| jrist | oh yeah | 12:40 |
| jrist | d0ugal, akrivoka - your default worked | 12:40 |
| jrist | via CLI | 12:40 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/os-cloud-config: Replace ucs_hostname with ucs_address https://review.openstack.org/370773 | 12:40 |
| d0ugal | jrist: okay, so things are not totally bust then. | 12:41 |
| jrist | akrivoka: now it says overcloud | 12:41 |
| jrist | ;) | 12:41 |
| d0ugal | jrist: and if it worked without an error, your previous attempts clear didn't :) | 12:41 |
| d0ugal | lol | 12:41 |
| jrist | this is progress | 12:41 |
| jrist | now to get validations to show up | 12:41 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Update tripleo-ui-deps RPM https://review.openstack.org/370775 | 12:41 |
| jrist | right now it says No Validations | 12:41 |
| jrist | There are no validations at this time. | 12:41 |
| jtomasek | jrist: I believe not all required patches for enable_ui landed yet? mandre? | 12:41 |
| jtomasek | jrist: oh, cool | 12:42 |
| jtomasek | jrist: validations require you to install tripleo-validations afaik | 12:43 |
| jtomasek | shadower: ^ | 12:43 |
| jrist | jtomasek: ok thanks I thought it was just via tripleo-common | 12:43 |
| shadower | jrist, jtomasek: it's this: https://github.com/openstack/tripleo-common/#validations | 12:44 |
| *** snecklifter has quit IRC | 12:44 | |
| EmilienM | jistr: don't change it now, but see my comment. | 12:45 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: When deploy finishes, show overcloud info https://review.openstack.org/370765 | 12:45 |
| shadower | jrist: we need 2 more patches merged to make it automatic: https://review.openstack.org/322893 and https://review.openstack.org/362194 | 12:45 |
| shadower | jrist: the first one was +A yesterday but the gate's failing so it's not been merged yet | 12:45 |
| jistr | EmilienM: ok thanks, will update the patch if it works in the current CI run | 12:46 |
| *** jlinkes has joined #tripleo | 12:46 | |
| jrist | yeah I'm aware thanks shadower | 12:49 |
| *** shardy has joined #tripleo | 12:50 | |
| shadower | np | 12:50 |
| *** rlandy has joined #tripleo | 12:50 | |
| *** morazi has joined #tripleo | 12:51 | |
| *** dtantsur|bbl is now known as dtantsur | 12:51 | |
| *** Goneri has joined #tripleo | 12:53 | |
| *** electrofelix has joined #tripleo | 12:53 | |
| jrist | so jtomasek, looks like angular 2 landed last night https://angular.io/ | 12:54 |
| jrist | :-P | 12:54 |
| jtomasek | jrist: we switch?:D | 12:54 |
| jrist | yeah why not | 12:54 |
| jrist | :) | 12:54 |
| *** jprovazn has quit IRC | 12:58 | |
| *** jprovazn has joined #tripleo | 12:58 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Remove external requirements https://review.openstack.org/370264 | 12:59 |
| *** rbrady has quit IRC | 12:59 | |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-common: Add node tagging workflow https://review.openstack.org/332132 | 12:59 |
| akrivoka | d0ugal: thanks for the review! | 12:59 |
| *** rbrady has joined #tripleo | 12:59 | |
| *** rbrady has left #tripleo | 12:59 | |
| *** rbrady has joined #tripleo | 13:00 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Rework the pacemaker_common_functions for M..N upgrades https://review.openstack.org/321027 | 13:00 |
| *** jlinkes has quit IRC | 13:01 | |
| *** myoung|gone is now known as myoung | 13:01 | |
| *** jpena|lunch is now known as jpena | 13:02 | |
| *** jaosorior has quit IRC | 13:02 | |
| *** noslzzp has joined #tripleo | 13:02 | |
| *** dsariel has quit IRC | 13:02 | |
| *** jaosorior has joined #tripleo | 13:03 | |
| *** liverpooler has quit IRC | 13:03 | |
| *** jlinkes has joined #tripleo | 13:06 | |
| EmilienM | jistr, slagle: FYI https://review.openstack.org/370798 | 13:06 |
| *** fzdarsky has joined #tripleo | 13:06 | |
| *** paramite has quit IRC | 13:06 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Remove external requirements https://review.openstack.org/370264 | 13:07 |
| marios | jistr: jaosorior EmilienM please consider re-adding your +2 .. the diff is very small from v23 https://review.openstack.org/#/c/321027/23..24/extraconfig/tasks/pacemaker_common_functions.sh and I added a comment with some output | 13:07 |
| derekh | shardy: you statalytics link, didn't take the tripleo-ci repository into account http://stackalytics.com/report/contribution/tripleo-ci/90 | 13:07 |
| EmilienM | d0ugal: have you seen http://paste.openstack.org/show/577433/ ? | 13:07 |
| derekh | shardy:: I was feeling a little hard done by :-) | 13:07 |
| EmilienM | the traceback looks like related to your patch that switched to tripleoclient | 13:07 |
| EmilienM | marios: I will | 13:08 |
| shardy | derekh: Oh, I didn't spot that - I guess we need a patch to stackalytics to fix that ;) | 13:08 |
| EmilienM | marios: +2 but not +A, CI is broken so useless to approve it | 13:08 |
| derekh | shardy: I think its driven by the governance repository, the tripleo-ci repo is included under the infrastructure group | 13:09 |
| *** fultonj has quit IRC | 13:09 | |
| *** fultonj has joined #tripleo | 13:09 | |
| derekh | shardy: although it would probably make more sense to have it under tripleo | 13:10 |
| marios | EmilienM: ack thanks | 13:10 |
| d0ugal | EmilienM: looking | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 13:10 |
| *** akshai has joined #tripleo | 13:10 | |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 13:10 |
| EmilienM | d0ugal: it sounds like a non critical trace but still horrible in logs | 13:10 |
| *** snecklifter has joined #tripleo | 13:11 | |
| d0ugal | EmilienM: Yeah, I don't know why we are getting those image errors. | 13:12 |
| EmilienM | jistr, d0ugal: though I'm watching in zuul jistr's patch now, and it's running mistral, let's see | 13:12 |
| EmilienM | it sounds like it doesn't work | 13:13 |
| EmilienM | it's still waiting for the rpc message | 13:13 |
| d0ugal | EmilienM: dang | 13:13 |
| EmilienM | yes | 13:13 |
| d0ugal | EmilienM: I don't understand how anything works if the images can't be found btw | 13:13 |
| EmilienM | go in zuul and watch | 13:13 |
| jistr | hmm yea probably doesn't work :/ | 13:16 |
| EmilienM | message timeout | 13:16 |
| EmilienM | it's failing now again | 13:16 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Zaqar services https://review.openstack.org/331682 | 13:17 |
| d0ugal | EmilienM: I don't know how to find it in zuul tbh | 13:17 |
| *** paramite has joined #tripleo | 13:17 | |
| EmilienM | d0ugal: status.openstack.org/zuul/ | 13:18 |
| d0ugal | I know what bit :) | 13:18 |
| d0ugal | that* | 13:18 |
| jistr | then you can find the job by gerrit patch number | 13:19 |
| *** lblanchard has joined #tripleo | 13:19 | |
| jistr | and the links are telnet links, so either you need something that handles them, or copy/paste them to terminal while changing the format to `telnet <IP> <port>` | 13:20 |
| d0ugal | jistr: oh, wow - I did not know that was there. | 13:20 |
| thrash | EmilienM: I seem to recall you posting something about handling the telnet:// url from chrome? I can't find it... | 13:21 |
| thrash | EmilienM: was that you? | 13:21 |
| EmilienM | yes | 13:21 |
| EmilienM | https://etherpad.openstack.org/p/chrome-telnet | 13:21 |
| *** pblaho has quit IRC | 13:21 | |
| thrash | EmilienM: You rock! | 13:21 |
| thrash | d0ugal: fyi... http://status.openstack.org/zuul and then type in the review number | 13:21 |
| thrash | d0ugal: and use what EmilienM just pasted. :) | 13:22 |
| *** ccamacho|lunch is now known as ccamacho | 13:22 | |
| thrash | EmilienM: doh! Looks like it's gone. | 13:22 |
| *** shardy has quit IRC | 13:22 | |
| EmilienM | so it doesn't fail on ovb jobs right? | 13:22 |
| openstackgerrit | Martin André proposed openstack/tripleo-specs: Spec for TripleO validations https://review.openstack.org/255792 | 13:22 |
| *** yamahata has joined #tripleo | 13:23 | |
| *** dsariel has joined #tripleo | 13:23 | |
| EmilienM | from what I can see in http://tripleo.org/cistatus.html | 13:23 |
| thrash | man, I totally missed y'all telling d0ugal about status. :P | 13:23 |
| d0ugal | lol | 13:23 |
| larsks | If an "overcloud deploy" gets stuck at "uploading new plan files" forever, what should I be looking at? | 13:23 |
| thrash | d0ugal: you can also use nc (netcat) and a pipe to grep the output | 13:24 |
| d0ugal | larsks: That suggests the mistral workflow has an error | 13:25 |
| larsks | d0ugal, this seems to happen reliably if an initial deploy fails, I fix things, and then re-run overcloud deploy. I get "removing current plan files", then "uploading", and there it stays forever. | 13:25 |
| slagle | chatting about the CI issues in https://redhat.bluejeans.com/7754237859/ if anyone wants to join | 13:26 |
| *** pkovar has quit IRC | 13:26 | |
| larsks | There don't appear to be any ERROR message in the mistral log (other than the template syntax error which is what caused the initial failure) | 13:26 |
| d0ugal | larsks: mistral execution-list might have an error. | 13:28 |
| larsks | d0ugal, indeed, there are some errors there. Thanks. I will poke about there and see if I can figure out what's going on. | 13:30 |
| d0ugal | larsks: cool, I can probably try and help in a bit. In the bluejeans call at the moment ^ | 13:31 |
| *** mgarciam has joined #tripleo | 13:33 | |
| *** trown is now known as trown|afk | 13:40 | |
| larsks | d0ugal, when you're back, I see two InvalidActionExceptions. (1) InvalidActionException: Failed to find action [action_name=tripleo.plan.update], and (2) InvalidActionException: Failed to find action [action_name=tripleo.baremetal.configure_boot] | 13:42 |
| larsks | There are some tripleo.* actions registered. | 13:42 |
| d0ugal | larsks: huh, that is a strange one. | 13:42 |
| *** pkovar has joined #tripleo | 13:42 | |
| rbrady | larsks: what is the output when you run "mistral-db-manage populate" ? | 13:43 |
| larsks | rbrady, let me try. | 13:43 |
| *** akshai has quit IRC | 13:44 | |
| larsks | rbrady, lots of warnings, which make me wonder if I am suffering from version skew: http://chunk.io/f/c9f34c81e7dd423ab158f1b1fd6dbe48 | 13:45 |
| *** limao has joined #tripleo | 13:47 | |
| *** mgarciam has quit IRC | 13:47 | |
| jaosorior | larsks: well, those look pretty normal (it's bad like that) | 13:47 |
| jaosorior | as long as tripleo.* actions are not failing it should be alright | 13:48 |
| *** akshai has joined #tripleo | 13:48 | |
| jaosorior | larsks: does mistral action-list | grep tripleo show anything? | 13:49 |
| larsks | jaosorior, Yes. http://chunk.io/f/db714cc723ea47ba9f38b58786f5134c | 13:49 |
| larsks | Just not the ones that are missing... | 13:50 |
| *** akshai has quit IRC | 13:50 | |
| jaosorior | larsks: those should come in tripleo-common | 13:50 |
| larsks | jaosorior, let me update that and retry. | 13:51 |
| jaosorior | larsks: so they should be here https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml | 13:51 |
| *** tzumainn has joined #tripleo | 13:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Try Mistral API with 4 workers https://review.openstack.org/370847 | 13:51 |
| jaosorior | larsks: what I'm not sure about is if you need to update your undercloud after updating that | 13:51 |
| larsks | "update your undercloud" means "re-run undercloud install"? | 13:52 |
| EmilienM | slagle: trying 1 and 4 workers | 13:52 |
| jaosorior | larsks: that is the case, you need to update the undercloud for it to load those | 13:52 |
| therve | actions are stevedore plugins, so you need to rerun tripleo-common install | 13:53 |
| larsks | jaosorior, so, I do have a recent tripelo-common (commit dated Monday, 9/12) | 13:53 |
| larsks | Let me see if the workbooks look sane. | 13:54 |
| jaosorior | therve: https://github.com/openstack/instack-undercloud/blob/master/elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup#L99 | 13:54 |
| ccamacho | hey mcornea, dtantsur re https://bugs.launchpad.net/tripleo/+bug/1619205 I think the issue might be related to a memory leak affecting apache, recycling the apache seems like a workarround, In my case the RAM was bumping up to the 6GB I have, and got the issue, do you know if you were out of RAM? | 13:54 |
| openstack | Launchpad bug 1619205 in tripleo "Overcloud API services go down after some time due to keystonemiddleware failure" [High,Confirmed] | 13:54 |
| dtantsur | ccamacho, I had 8 GiB | 13:54 |
| *** akshai has joined #tripleo | 13:55 | |
| EmilienM | http://paste.openstack.org/show/577433/ | 13:55 |
| *** mgarciam has joined #tripleo | 13:56 | |
| therve | jaosorior, That's not how actions are registered AFAIK | 13:56 |
| jpich | jaosorior, larsks: There's a bunch of interesting commands at https://github.com/openstack/tripleo-common/#action-development to get new actions to get picked up. Updating workbooks should probably be done before restarting the mistral services though | 13:56 |
| larsks | jaosorior, I don't see a "plan.update" in worksbooks/baremetal.yaml, even in tripleo-common master. I do see the configure_boot action, in the installed files in /usr/share/openstack-tripleo-common/... | 13:56 |
| larsks | So not sure why that is missing. | 13:57 |
| jaosorior | now that's funky | 13:57 |
| therve | larsks, it ought to be defined in setup.cfg | 13:57 |
| *** akshai has quit IRC | 13:57 | |
| jpich | larsks: Actions are registered in https://github.com/openstack/tripleo-common/blob/master/setup.cfg#L61 | 13:57 |
| ccamacho | we might reduce the apache recycling time, i'm doing some tests to see if it works, then we should check which service is taking the RAM, Ill let you know for further testing | 13:57 |
| larsks | jpich, let me look. | 13:57 |
| mcornea | ccamacho: nope, I don't think so. I remember I checked the services on the controllers and they were running fine, just httpd was stuck | 13:57 |
| jpich | larsks: That other place is for workflows, which usually call to these actions as they go | 13:58 |
| *** akshai has joined #tripleo | 13:58 | |
| larsks | jpich, well, they seem to be there. I've run a 'setup.py install' followed by another 'undercloud install'. | 13:59 |
| larsks | I see then entrypoints listed in the installed /usr/lib/python2.7/site-packages/tripleo_common-5.0.1.dev39-py2.7.egg-info | 13:59 |
| jpich | larsks: and yet they don't show up in your mistral action-list output? (/me trying to catch up on the earlier bits of the conversation) | 14:00 |
| larsks | jpich, that's correct. The output of action-list | grep tripleo is http://chunk.io/f/db714cc723ea47ba9f38b58786f5134c | 14:01 |
| *** akshai has quit IRC | 14:01 | |
| *** ramishra has quit IRC | 14:01 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: TESTING https://review.openstack.org/370855 | 14:01 |
| EmilienM | http://logs.openstack.org/16/370516/1/check/gate-tripleo-ci-centos-7-nonha-multinode/b9e374c/logs/var/log/messages | 14:02 |
| EmilienM | look at the very bottom | 14:02 |
| EmilienM | journal stopped | 14:02 |
| *** mcornea has quit IRC | 14:02 | |
| *** mcornea has joined #tripleo | 14:03 | |
| *** akshai has joined #tripleo | 14:03 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Delete default plan before cli deployment https://review.openstack.org/370857 | 14:03 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Delete default plan before cli deployment https://review.openstack.org/370857 | 14:04 |
| jpich | larsks: Huh, this is weird. Usually if there's some kind of weird failure none of the actions in the file get picked up, but here there's a bunch of other plan-related actions that are just fine... Is it possible there's some kind of older version tripleo-common hanging around somewhere, somehow? | 14:05 |
| larsks | jpich, it's totally possible. There was an older version on the system initially; I ran 'setup.py install' from a more recent version to update. | 14:06 |
| larsks | I'll see about clearing out the old version completely and seeing if that helps. | 14:07 |
| jrist | jtomasek: did you see my patch for listening on 0.0.0.0 in webpack? can we get some merges? it is bugging me :) | 14:07 |
| jrist | because I set it up to listen from several areas | 14:07 |
| jrist | I have to edit webpack every time | 14:07 |
| EmilienM | so the /var/log/messages is flushing when file limit is reached but never restarted | 14:07 |
| jtomasek | jrist: sure | 14:08 |
| *** ramishra has joined #tripleo | 14:08 | |
| jtomasek | jrist: oh, I've commented on it before, could you please update it? | 14:09 |
| marios | thanks jistr EmilienM jaosorior | 14:09 |
| jaosorior | marios: no biggie | 14:09 |
| jpich | larsks: I think you might need to run the populate (and maybe restart?) commands in https://github.com/openstack/tripleo-common/#action-development too if you updated tripleo-common manually, for Mistral to register the new commands | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 14:10 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 14:10 |
| marios | jaosorior: well its a pretty awful thing to review lots of bash :) but it runs fine fwiw as commented | 14:10 |
| larsks | jpich, yeah, now I see the actions. | 14:10 |
| * larsks tries another deploy! | 14:10 | |
| jpich | larsks: Yay! | 14:11 |
| *** oshvartz has quit IRC | 14:11 | |
| mwhahaha | ccamacho: can you join #puppet-openstack, i'd like to chat about the 140 lint thing | 14:13 |
| ccamacho | mwhahaha sure man | 14:14 |
| mwhahaha | ccamacho: thanks | 14:14 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: TEST - DO NOT MERGE https://review.openstack.org/370868 | 14:14 |
| *** limao has quit IRC | 14:14 | |
| *** osp has joined #tripleo | 14:15 | |
| jrist | jtomasek: oh soryr I didn't see the comment | 14:15 |
| * jrist merges rbrady's 370868 | 14:15 | |
| openstackgerrit | Jason E. Rist proposed openstack/tripleo-ui: Have dev server listen everywhere instead of just local https://review.openstack.org/370537 | 14:16 |
| jrist | jtomasek: ^ | 14:16 |
| rbrady | jrist: noooooooo | 14:16 |
| larsks | jpich, next failure: Object GET failed: https://192.0.2.2:13808/v1/AUTH_9abf4fc45df944b695df8b1930b8b5d8/overcloud/overcloud-without-mergepy.yaml 404 Not Found | 14:16 |
| rbrady | larsks: congrats, you've caught up to us | 14:16 |
| larsks | rbrady, i think you just made me sad :( | 14:16 |
| *** hjensas has quit IRC | 14:16 | |
| d0ugal | larsks: That one is safe to ignore AFAICT | 14:16 |
| larsks | d0ugal, but the deploy fails after that point. | 14:17 |
| jrist | jtomasek: it was late, sorry about the double quotes ;) | 14:17 |
| d0ugal | larsks: Any other errors? | 14:17 |
| jpich | larsks: Probably related to https://bugs.launchpad.net/tripleo/+bug/1622720 and duplicates? | 14:17 |
| openstack | Launchpad bug 1622683 in tripleo "duplicate for #1622720 Updating plans breaks deployment" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 14:17 |
| jtomasek | jrist: np, the linter does not reach that file:) | 14:17 |
| larsks | d0ugal, there is a giant dictionary dump that includes: "Failed to run action [action_ex_id=801b62b4-9f34-48f4-916c-254b6206e416, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'container': u'overcloud', u'timeout': 240}']\n 'Result' object has no attribute 'copy'", | 14:17 |
| *** akuznetsov has joined #tripleo | 14:18 | |
| d0ugal | larsks: Right, I know that error! | 14:18 |
| larsks | d0ugal, it is as familiar as an old friend, or you know a solution? :) | 14:18 |
| shadower | haha | 14:19 |
| d0ugal | larsks: https://review.openstack.org/#/c/368081/ | 14:20 |
| d0ugal | rbrady: ^ | 14:20 |
| d0ugal | larsks: so, you actually have another error - but it is being hidden. That fix reveals it for you :/ | 14:20 |
| d0ugal | larsks: otherwise it might be in the mistral logs | 14:21 |
| larsks | d0ugal, yay? I will apply that patch and redeploy and also poke at the logs. | 14:21 |
| d0ugal | larsks: Okay, thanks - sorry! | 14:21 |
| jpich | d0ugal: That error was reported at https://bugs.launchpad.net/tripleo/+bug/1623086 , do we want to reference that LP in the patch or is it kinda only coincidentally related? | 14:22 |
| openstack | Launchpad bug 1623086 in tripleo "'Result' object has no attribute 'copy' - overcloud deployment fails" [Undecided,New] | 14:22 |
| *** akuznetsov has quit IRC | 14:22 | |
| d0ugal | jpich: it's the same, they should be referenced. | 14:22 |
| d0ugal | s/they/it/ | 14:22 |
| *** rajinir has joined #tripleo | 14:23 | |
| jpich | d0ugal: Ok, thanks! I can add the ref if you're in the middle of something else | 14:24 |
| *** mlupton has joined #tripleo | 14:24 | |
| larsks | d0ugal, does something require a restart after updating tripleo-common? | 14:24 |
| d0ugal | jpich: if you don't mind, otherwise I can do it in a bit | 14:24 |
| jpich | d0ugal: Will do it now | 14:24 |
| d0ugal | larsks: I'd restart mistral to be safe. | 14:25 |
| d0ugal | larsks: sudo systemctl restart openstack-mistral-*; | 14:25 |
| d0ugal | jpich: Thanks! | 14:25 |
| *** flepied1 has joined #tripleo | 14:25 | |
| *** flepied has quit IRC | 14:26 | |
| larsks | d0ugal, I think I have an actual error: u'message': u"No connection adapters were found for 'file:///home/stack/tripleo-heat-templates/puppet/services/logging/fluentd-base.yaml' | 14:28 |
| larsks | Not sure what that is telling me, unless it is "I don't know how to handle file:// urls". | 14:28 |
| d0ugal | larsks: ah, yes, this is a known issue that doesn't have a fix. Progress on resolving it has been stalled by CI issues | 14:28 |
| d0ugal | it was found yesterday by gfidente | 14:28 |
| d0ugal | (AFAIK) | 14:28 |
| *** larsks has left #tripleo | 14:29 | |
| *** larsks has joined #tripleo | 14:29 | |
| d0ugal | larsks: https://bugs.launchpad.net/tripleo/+bug/1623552 | 14:29 |
| openstack | Launchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Confirmed] | 14:29 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Check the result of the parent action when subclassing https://review.openstack.org/368081 | 14:29 |
| larsks | d0ugal, thanks, was just about to ask for that... | 14:29 |
| d0ugal | larsks: That is exactly what it is telling you, whihc I know isn't useful :/ | 14:29 |
| larsks | d0ugal, wouldn't that make pretty much every deploy fail? | 14:29 |
| osp | hi, I am currently deploying openstack (mitaka) and running some pre puppet config on all nodes, and additional manifests on controller and compute nodes... | 14:30 |
| d0ugal | larsks: I think it probably is impacting a large number of them | 14:30 |
| *** trown|afk is now known as trown | 14:30 | |
| d0ugal | larsks: but it slipped through CI somehow, and now CI is failing before that. | 14:30 |
| osp | This runs a manifest locally on each server but without the advantages of templates etc. Is there a way to pre-puppet config which would use an entire puppet module i have created and be able to use templates/files etc | 14:31 |
| * larsks cries on his desk. | 14:31 | |
| *** dhill_ has quit IRC | 14:34 | |
| *** dhill_ has joined #tripleo | 14:35 | |
| *** flepied1 is now known as flepied | 14:35 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common: Revert "Add template processing to the update plan workflow." https://review.openstack.org/370434 | 14:35 |
| social | -M hacluster' returned 6: useradd: group 'haclient' does not exist | 14:39 |
| *** noslzzp has quit IRC | 14:42 | |
| *** radeks has joined #tripleo | 14:42 | |
| *** noslzzp has joined #tripleo | 14:42 | |
| *** pradk has quit IRC | 14:42 | |
| bnemec | slagle: EmilienM: https://review.openstack.org/#/c/370724/ still failed even with a 10 minute timeout. :-( | 14:43 |
| EmilienM | bnemec: yeah | 14:44 |
| EmilienM | bnemec: I'm exploring the revert ^ | 14:44 |
| EmilienM | https://review.openstack.org/370434 | 14:44 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: [TESTING] Force debug on tripleo.sh https://review.openstack.org/370900 | 14:45 |
| *** pradk has joined #tripleo | 14:45 | |
| bnemec | EmilienM: Wait, http://logs.openstack.org/24/370724/3/check/gate-tripleo-ci-centos-7-nonha-multinode/347b7f9/logs/var/log/mistral/mistral-server.txt.gz#_2016-09-15_14_36_10_290 | 14:45 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Delete default plan before cli deployment https://review.openstack.org/370857 | 14:45 |
| bnemec | The message came in less than a second after the timeout exception? | 14:45 |
| bnemec | I don't believe that's a coincidence. | 14:45 |
| bnemec | EmilienM: I wonder if we should squash your 4 worker patch into that. | 14:46 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic https://review.openstack.org/354014 | 14:47 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Migrate Deploy action to Mistral https://review.openstack.org/357125 | 14:48 |
| slagle | bnemec: we are on https://redhat.bluejeans.com/7754237859/ | 14:48 |
| slagle | discussing CI | 14:48 |
| bnemec | slagle: omw | 14:48 |
| therve | bnemec, Yeah, I believe that has something to do with the blocking executor | 14:49 |
| therve | I don't why that works sometimes, though | 14:49 |
| *** liverpooler has joined #tripleo | 14:49 | |
| *** akshai has quit IRC | 14:49 | |
| bnemec | slagle: Nobody is talking. :-) | 14:49 |
| slagle | bnemec: we're waiting for you to say something profound | 14:50 |
| social | chem`: puppet-pacemaker seems to lack dependancy on package providing the haclient group | 14:50 |
| *** mbound has joined #tripleo | 14:50 | |
| *** chem` is now known as chem | 14:50 | |
| *** akshai has joined #tripleo | 14:51 | |
| marios | gfidente: https://review.openstack.org/#/c/354014/ updated if you get a chance thanks faidentee... its... its almost good to have you back o_O | 14:51 |
| *** akshai has quit IRC | 14:51 | |
| chem | social: hum ... not sure I get, why there would be the haclient group in puppet-pacemaker ? | 14:52 |
| *** absubram has joined #tripleo | 14:52 | |
| *** akshai has joined #tripleo | 14:53 | |
| chem | social: actually I'm not sure what you mean by "haclient group" | 14:53 |
| *** absubram_ has joined #tripleo | 14:53 | |
| *** saneax is now known as saneax-_-|AFK | 14:56 | |
| *** absubram has quit IRC | 14:57 | |
| *** absubram_ is now known as absubram | 14:57 | |
| slagle | EmilienM: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name:*tripleo-ci* AND build_status:FAILURE AND message:\"Timed out waiting for a reply to message ID\" | 14:58 |
| slagle | http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A*tripleo-ci*%20AND%20build_status%3AFAILURE%20AND%20message%3A%5C%22Timed%20out%20waiting%20for%20a%20reply%20to%20message%20ID%5C%22 | 14:58 |
| *** bvandenh_ has joined #tripleo | 15:00 | |
| *** radeks has quit IRC | 15:00 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Use the proper private keys for ssh config file https://review.openstack.org/370919 | 15:02 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 15:03 |
| *** akuznetsov has joined #tripleo | 15:05 | |
| *** yamahata has quit IRC | 15:06 | |
| chem | social: ? | 15:07 |
| EmilienM | slagle: project-config patch to make multinode jobs non voting https://review.openstack.org/#/c/370922/ | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 15:10 |
| *** bvandenh_ has quit IRC | 15:10 | |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 15:10 |
| social | chem: group 'haclient' does not exist\u001b[0m\n\u001b[1;31mError: /Stage[main]/Pacemaker::Corosync/User[hacluster]/ensure: change from absent to present failed | 15:10 |
| *** tremble has quit IRC | 15:15 | |
| EmilienM | slagle: https://review.openstack.org/#/c/370069/ | 15:15 |
| EmilienM | slagle: http://logs.openstack.org/69/370069/2/check/gate-tripleo-ci-centos-7-nonha-multinode/40d2ea1/console.html#_2016-09-14_12_28_08_902337 | 15:17 |
| *** jaosorior has quit IRC | 15:18 | |
| *** pcaruana has quit IRC | 15:19 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 15:19 |
| *** paramite has quit IRC | 15:22 | |
| *** aufi has quit IRC | 15:25 | |
| d0ugal | https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L74-L78 | 15:26 |
| EmilienM | marios, jistr, bnemec, slagle, thrash, shadower: just a reminder in case you missed my email on ML: please do not approve any patch today, until our CI gets stable again. Thanks a lot. | 15:27 |
| bnemec | EmilienM: I'm on the call where you said that. :-P | 15:27 |
| marios | EmilienM: ack thanks | 15:28 |
| jistr | me too :) | 15:28 |
| EmilienM | bnemec: I know | 15:28 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic https://review.openstack.org/354014 | 15:28 |
| EmilienM | I'm just making sure it's written for everybody :-) | 15:28 |
| shadower | thanks, noted | 15:28 |
| marios | gfidente: updated thanks https://review.openstack.org/354014 | 15:28 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Delete default plan before cli deployment https://review.openstack.org/370857 | 15:30 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults https://review.openstack.org/366760 | 15:31 |
| *** pkovar has quit IRC | 15:32 | |
| marios | gfidente: | 15:33 |
| *** TicToc has joined #tripleo | 15:34 | |
| *** pkovar has joined #tripleo | 15:37 | |
| gfidente | matbu will you update https://bugzilla.redhat.com/show_bug.cgi?id=1374076 ? | 15:38 |
| openstack | bugzilla.redhat.com bug 1374076 in openstack-tripleo-heat-templates "OSP9/10 Ceph osd node upgrade fails." [Unspecified,On_dev] - Assigned to gfidente | 15:38 |
| gfidente | daaahh wrong link sorry, I meant https://review.openstack.org/#/c/370127/ | 15:38 |
| *** nyechiel has quit IRC | 15:41 | |
| openstackgerrit | Dimitri Savineau proposed openstack/puppet-tripleo: Cinder: Add Nexenta Support https://review.openstack.org/370956 | 15:42 |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful https://review.openstack.org/369247 | 15:47 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Add all the gating functionality to full-deploy.sh https://review.openstack.org/370194 | 15:47 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 15:47 |
| EmilienM | revert is passing, https://review.openstack.org/#/c/370434/ | 15:48 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test dib build removal and temprevert together https://review.openstack.org/370961 | 15:49 |
| EmilienM | hopefully we are not in the 20% of success that we see without the revert | 15:49 |
| d0ugal | EmilienM: Worth rechecking to have it pass twice? | 15:49 |
| bnemec | EmilienM: I think you're being very generous saying 20%. :-) | 15:49 |
| d0ugal | lol | 15:49 |
| matbu | gfidente: did you see my comment ? | 15:50 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: TEST - DO NOT MERGE https://review.openstack.org/370868 | 15:50 |
| EmilienM | bnemec: right, it's less | 15:50 |
| matbu | gfidente: the last one | 15:51 |
| EmilienM | d0ugal: yes I'll do the recheck when my project-config is merged so we can land it | 15:51 |
| EmilienM | slagle: https://etherpad.openstack.org/p/tripleo-ci-status | 15:53 |
| gfidente | matbu yeah it's in SoftwareDeploymentGroup | 15:54 |
| gfidente | which hinerits from ResourceGroup | 15:54 |
| gfidente | did you update the template version and it still didn't pass> | 15:55 |
| gfidente | ? | 15:55 |
| matbu | gfidente: yep | 15:55 |
| gfidente | with the same error? | 15:56 |
| gfidente | matbu and is this with the new tripleoclient which pushes stuff in swift, or with the old version which is deploying from files | 15:56 |
| gfidente | ? | 15:56 |
| *** jlinkes has quit IRC | 15:57 | |
| matbu | gfidente: yes the same error and the tripleocli is python-tripleoclient-5.0.0-0.20160907170033 | 15:57 |
| *** jlinkes has joined #tripleo | 15:57 | |
| matbu | gfidente: but i'll look more closely later, you're right, i didn't see that SoftwareDeploymentGroup inherit from ResourceGroup | 15:58 |
| *** mgarciam has quit IRC | 15:58 | |
| *** jkraj has quit IRC | 15:58 | |
| gfidente | matbu so with the new client it is possible that changes to the template files aren't updated in swift | 15:58 |
| *** Guest25001 has quit IRC | 15:58 | |
| *** ubijtsa has joined #tripleo | 15:59 | |
| matbu | gfidente: ack | 15:59 |
| *** matbu is now known as matbu|brb | 15:59 | |
| *** ubijtsa is now known as Guest98103 | 15:59 | |
| *** abregman_ has quit IRC | 16:00 | |
| *** abregman__ has joined #tripleo | 16:00 | |
| *** abregman__ has quit IRC | 16:01 | |
| *** abregman has joined #tripleo | 16:01 | |
| *** noslzzp has quit IRC | 16:06 | |
| *** abregman has quit IRC | 16:06 | |
| *** abregman has joined #tripleo | 16:06 | |
| *** rasca has quit IRC | 16:09 | |
| *** ooolpbot has joined #tripleo | 16:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 16:11 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 16:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 16:11 |
| *** ooolpbot has quit IRC | 16:11 | |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 16:11 |
| *** mlupton has quit IRC | 16:11 | |
| *** zoli|trng is now known as zoli_gone-proxy | 16:11 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Add all the gating functionality to full-deploy.sh https://review.openstack.org/370194 | 16:11 |
| *** absubram has quit IRC | 16:14 | |
| EmilienM | for the record, James wrote a summary of our Mistral problem and why CI broke: https://bugs.launchpad.net/tripleo/+bug/1623606/comments/10 | 16:15 |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 16:15 |
| *** yamahata has joined #tripleo | 16:15 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 16:16 |
| *** mgarciam has joined #tripleo | 16:16 | |
| openstackgerrit | Sudipta Biswas proposed openstack/diskimage-builder: ppc64el: fix for grub2 missing package https://review.openstack.org/370980 | 16:17 |
| jpich | Great summary, thanks | 16:18 |
| *** mcornea has quit IRC | 16:18 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test dib build removal and temprevert together https://review.openstack.org/370961 | 16:18 |
| *** zoli_gone-proxy is now known as zoliXXL | 16:19 | |
| *** fultonj_ has joined #tripleo | 16:20 | |
| *** mlupton has joined #tripleo | 16:20 | |
| derekh | slagle: bnemec we got nothing sensitive in the nova database on the RH1 overcloud controller have we? thinking of attaching it to a bug | 16:21 |
| bnemec | derekh: The only thing would be service passwords, but I assume they're encrypted. | 16:22 |
| derekh | bnemec: they'd be in the keystone db wouldn't they? | 16:22 |
| EmilienM | slagle: https://review.openstack.org/#/c/369792/ ? | 16:23 |
| bnemec | derekh: Oh, yeah. Just the nova db is probably fine. | 16:23 |
| derekh | bnemec: ya, I think so | 16:23 |
| *** lucasagomes is now known as lucas-dinner | 16:24 | |
| *** dtantsur is now known as dtantsur|afk | 16:25 | |
| *** florianf has quit IRC | 16:25 | |
| *** rcernin has quit IRC | 16:26 | |
| *** fzdarsky has quit IRC | 16:26 | |
| *** zoliXXL is now known as zoli|gone | 16:29 | |
| *** zoli|gone is now known as zoli_gone-proxy | 16:30 | |
| *** bkopilov has quit IRC | 16:30 | |
| *** florianf has joined #tripleo | 16:30 | |
| *** absubram has joined #tripleo | 16:31 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test dib build removal and temprevert together https://review.openstack.org/370961 | 16:31 |
| *** jpich has quit IRC | 16:32 | |
| *** mgarciam has quit IRC | 16:34 | |
| *** abehl has quit IRC | 16:35 | |
| d0ugal | slagle, EmilienM - it passed! https://review.openstack.org/#/c/369247/ | 16:44 |
| *** derekh has quit IRC | 16:49 | |
| *** akuznetsov has quit IRC | 16:54 | |
| *** tosky has quit IRC | 16:56 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Add mongo config settings in collector service templates https://review.openstack.org/370426 | 16:56 |
| *** yamahata has quit IRC | 16:58 | |
| *** yamahata has joined #tripleo | 16:58 | |
| *** fragatina has joined #tripleo | 16:59 | |
| *** fragatina has quit IRC | 17:00 | |
| *** fragatina has joined #tripleo | 17:01 | |
| *** itlinux has joined #tripleo | 17:03 | |
| *** mlupton has quit IRC | 17:06 | |
| *** fultonj_ has quit IRC | 17:07 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623891 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 17:10 |
| openstack | Launchpad bug 1623891 in tripleo "CI not testing changes made in submitted patches" [Critical,Triaged] - Assigned to Jiřà Stránský (jistr) | 17:10 |
| *** jpena is now known as jpena|off | 17:10 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: remove DIB workaround https://review.openstack.org/369792 | 17:14 |
| itlinux | do you guys have the right location for the new features on the roadmap for OOO | 17:15 |
| *** thrash is now known as thrash|biab | 17:17 | |
| *** milan has quit IRC | 17:18 | |
| *** mlupton_ has joined #tripleo | 17:21 | |
| *** ohamada has quit IRC | 17:22 | |
| *** abregman has quit IRC | 17:23 | |
| *** snecklifter has quit IRC | 17:24 | |
| *** pkovar has quit IRC | 17:25 | |
| d0ugal | slagle, EmilienM - | 17:25 |
| d0ugal | slagle, EmilienM - I just dropped out the call. Late in the day here. Let me know if there is anythng I can continue with tomorrow. | 17:25 |
| EmilienM | d0ugal: thanks a ton for your help | 17:25 |
| EmilienM | d0ugal: good evening | 17:26 |
| *** trown is now known as trown|lunch | 17:28 | |
| *** chem has quit IRC | 17:29 | |
| *** chem has joined #tripleo | 17:30 | |
| *** chem has quit IRC | 17:30 | |
| *** chem has joined #tripleo | 17:30 | |
| slagle | d0ugal: ttyl! | 17:30 |
| *** electrofelix has quit IRC | 17:31 | |
| *** dtrainor has quit IRC | 17:32 | |
| *** dtrainor has joined #tripleo | 17:32 | |
| *** fragatina has quit IRC | 17:36 | |
| *** abehl has joined #tripleo | 17:37 | |
| openstackgerrit | Merged openstack/tripleo-common: Revert "Add template processing to the update plan workflow." https://review.openstack.org/370434 | 17:37 |
| EmilienM | ok multinode job should be green again | 17:38 |
| *** abehl has quit IRC | 17:39 | |
| *** tosky has joined #tripleo | 17:39 | |
| *** anshul has joined #tripleo | 17:39 | |
| EmilienM | https://review.openstack.org/#/c/370250/ is in gate, once it's merged we'll make sure everything is green again, re-enable voting on multinode job and then approve patches again. | 17:39 |
| slagle | EmilienM: once dlrn builds the new tripleo-common package, we should recheck something to make sure multinode is passing | 17:40 |
| slagle | EmilienM: actually this one: https://review.openstack.org/#/c/369247/ | 17:40 |
| slagle | we should land that next | 17:40 |
| slagle | in fact, we could go ahead and approve that one now | 17:41 |
| *** jbadiapa has quit IRC | 17:41 | |
| EmilienM | slagle: I wanted to recheck it before we land it to see if ovb is green | 17:41 |
| EmilienM | oh, ok | 17:41 |
| EmilienM | slagle: approved | 17:41 |
| bnemec | Is the dib patch in the gate? | 17:41 |
| bnemec | Err | 17:41 |
| slagle | bnemec: already merged | 17:41 |
| bnemec | The "don't build dib" patch. | 17:41 |
| EmilienM | yeah it's merged | 17:41 |
| EmilienM | https://review.openstack.org/369792 | 17:42 |
| bnemec | Ah, that's got the necessary depends-on anyway. | 17:42 |
| bnemec | Too many patches to keep track of. :-/ | 17:42 |
| *** itlinux has quit IRC | 17:43 | |
| bnemec | It's looking like the temprevert patch is going to work too: https://review.openstack.org/#/c/370961/ | 17:43 |
| slagle | we fixed everything at the same time | 17:43 |
| bnemec | Assuming it passes, I'll drop the dib part of it and re-propose individually. | 17:44 |
| *** gfidente has quit IRC | 17:44 | |
| *** bana_k has joined #tripleo | 17:50 | |
| *** mlupton_ has quit IRC | 17:51 | |
| *** thrash|biab is now known as thrash | 17:51 | |
| EmilienM | I'm also working on a fix to bring back HA job on liberty/mitaka, without removing EPEL | 17:52 |
| EmilienM | so we'll avoid breaking CI again | 17:52 |
| *** itlinux has joined #tripleo | 17:53 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Re-enable temprevert/cherry-pick/pin functionality https://review.openstack.org/370961 | 17:54 |
| *** mlupton has joined #tripleo | 17:54 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow. https://review.openstack.org/371027 | 17:54 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: [mitaka-only] mysql: never add brackets to mysql_bind_host https://review.openstack.org/371029 | 17:56 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: [liberty-only] mysql: never add brackets to mysql_bind_host https://review.openstack.org/371031 | 17:58 |
| EmilienM | dciabrin: ^ | 17:58 |
| *** dhill_ has quit IRC | 18:00 | |
| *** TicToc has quit IRC | 18:01 | |
| *** dhill__ has joined #tripleo | 18:01 | |
| dciabrin | EmilienM, thx | 18:01 |
| *** TicToc has joined #tripleo | 18:02 | |
| *** dhill__ has quit IRC | 18:02 | |
| *** dhill_ has joined #tripleo | 18:02 | |
| slagle | EmilienM: bnemec : we might want to stop https://review.openstack.org/#/c/369247/ | 18:04 |
| slagle | the ovb ha job failed with an error | 18:04 |
| slagle | thrash: ^ | 18:04 |
| EmilienM | ok | 18:04 |
| slagle | "Timed out creating the default Deployment Plan" | 18:04 |
| EmilienM | slagle: done | 18:04 |
| *** fragatina has joined #tripleo | 18:04 | |
| EmilienM | let's wait for https://review.openstack.org/#/c/370250/ to be merged | 18:05 |
| slagle | EmilienM: yea and after dlrn builds that, we can recheck 369247 | 18:05 |
| *** hjensas has joined #tripleo | 18:05 | |
| EmilienM | +1 | 18:05 |
| *** rook_ is now known as rook | 18:07 | |
| *** bana_k has quit IRC | 18:08 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Add all the gating functionality to full-deploy.sh https://review.openstack.org/370194 | 18:09 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 18:09 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 18:10 |
| *** mlupton has quit IRC | 18:10 | |
| *** yamahata has quit IRC | 18:13 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-common: Add template processing to the update plan workflow. https://review.openstack.org/371027 | 18:13 |
| *** liverpooler has quit IRC | 18:14 | |
| *** mlupton has joined #tripleo | 18:14 | |
| openstackgerrit | Merged openstack/os-collect-config: Revert "Treat ec2 collector data as immutable" https://review.openstack.org/370250 | 18:20 |
| bnemec | \o/ | 18:20 |
| *** bana_k has joined #tripleo | 18:21 | |
| *** rodrigods has quit IRC | 18:22 | |
| *** rodrigods has joined #tripleo | 18:22 | |
| *** pcaruana has joined #tripleo | 18:22 | |
| EmilienM | boom | 18:24 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/371034 | 18:25 |
| EmilienM | let's see how it goes now | 18:25 |
| EmilienM | it's funny, even if a patch passed the gate and merged, jobs are still running in tripleo cloud | 18:26 |
| EmilienM | which is cool, so you can still monitor them | 18:26 |
| slagle | EmilienM: new tripleo-common and os-collect-config packages arent built yet | 18:26 |
| slagle | https://trunk.rdoproject.org/centos7/report.html | 18:26 |
| EmilienM | I'm too fast, isn't what you just said? :P | 18:27 |
| slagle | yes :) | 18:27 |
| EmilienM | I'm killing the CI jobs | 18:27 |
| EmilienM | (with abandon button) | 18:27 |
| openstackgerrit | Paul Belanger proposed openstack/python-tripleoclient: [WIP] Switch to centos-minimal element https://review.openstack.org/371046 | 18:30 |
| slagle | EmilienM: tripleo-common is built | 18:33 |
| EmilienM | yep | 18:33 |
| slagle | i'm wondering if os-collect-config will even get built? | 18:33 |
| EmilienM | of course | 18:33 |
| slagle | trown|lunch: is os-collect-config still pinned to uc in delorean trunk? | 18:33 |
| EmilienM | grep os-collect-config in the UI | 18:33 |
| EmilienM | ahh the pin you mean | 18:33 |
| EmilienM | let me check that | 18:33 |
| slagle | well i guess the original breaking patch got built yesterday | 18:33 |
| slagle | so i suspect the fix will too :) | 18:34 |
| EmilienM | so no we don't ping it | 18:34 |
| EmilienM | git clone https://review.rdoproject.org/r/rdoinfo | 18:34 |
| EmilienM | you can see the project in rdo.yaml | 18:34 |
| *** florianf has quit IRC | 18:40 | |
| *** fragatina has quit IRC | 18:43 | |
| bnemec | Ouch. 26 stacks creating at the same time on rh1. | 18:43 |
| *** trown|lunch is now known as trown | 18:43 | |
| bnemec | It will be a miracle if this doesn't end in tears. | 18:43 |
| *** fragatina has joined #tripleo | 18:43 | |
| openstackgerrit | Paul Belanger proposed openstack/python-tripleoclient: [WIP] Switch to centos-minimal element https://review.openstack.org/371046 | 18:44 |
| *** yamahata has joined #tripleo | 18:45 | |
| *** florianf has joined #tripleo | 18:46 | |
| bnemec | Hmm, lot of jenkins instances without fips. | 18:49 |
| *** Guest98103 has quit IRC | 18:50 | |
| *** Guest98103 has joined #tripleo | 18:50 | |
| *** Guest98103 is now known as assassin | 18:50 | |
| *** florianf has quit IRC | 18:50 | |
| *** fzdarsky has joined #tripleo | 18:51 | |
| EmilienM | jrist: https://blueprints.launchpad.net/tripleo/+spec/assign-nodes-workflow | 18:54 |
| EmilienM | are we hoping to land everything for newton? | 18:54 |
| EmilienM | or will it slip to ocata? | 18:55 |
| jrist | I believe so | 18:55 |
| jrist | are you trying to cut today? | 18:55 |
| EmilienM | no | 18:55 |
| EmilienM | I'm trying to update launchpad to make sure everything is clear | 18:55 |
| jrist | EmilienM: I just migrated that over yesterday from our launchpad | 18:55 |
| EmilienM | ok | 18:55 |
| jrist | to make it more clear/visible ;) | 18:55 |
| EmilienM | ++ | 18:55 |
| jrist | it sort of slipped as it was in tripleo-ui | 18:55 |
| jrist | EmilienM: we talked about moving everything to tripleo the week of summit | 18:55 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add node tagging workflow https://review.openstack.org/332132 | 18:55 |
| jrist | EmilienM: everything for our launchpad | 18:56 |
| jrist | just to fully unify but also to expose stuff that needs backend work | 18:56 |
| jrist | and tie it all together | 18:56 |
| d0ugal | jrist: A fun task for you! | 18:56 |
| jrist | d0ugal: LET ME TELL YOU | 18:56 |
| jrist | d0ugal: do you want to do it? | 18:56 |
| jrist | ;) | 18:56 |
| d0ugal | jrist: Sure | 18:56 |
| jrist | I am happy to share | 18:56 |
| d0ugal | jrist: oh wait | 18:56 |
| jrist | lol | 18:56 |
| d0ugal | no | 18:56 |
| jrist | you said sure | 18:56 |
| * d0ugal runs | 18:56 | |
| jrist | you said sure!!! | 18:56 |
| EmilienM | I have to be afk for 1h, bbl | 18:56 |
| * jrist runs after | 18:56 | |
| * jrist too | 18:56 | |
| EmilienM | don't merge anything yet ! | 18:57 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/371034 | 18:57 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Update newton release config for new image location https://review.openstack.org/370267 | 18:58 |
| *** fzdarsky has quit IRC | 18:59 | |
| *** absubram has quit IRC | 19:01 | |
| *** mburned is now known as mburned_out | 19:02 | |
| *** mburned_out is now known as mburned | 19:04 | |
| *** hjensas has quit IRC | 19:04 | |
| *** abregman has joined #tripleo | 19:08 | |
| *** akshai has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 19:10 |
| *** akshai has joined #tripleo | 19:11 | |
| *** fzdarsky has joined #tripleo | 19:11 | |
| openstackgerrit | Paul Belanger proposed openstack/python-tripleoclient: [WIP] Switch to centos-minimal element https://review.openstack.org/371046 | 19:15 |
| *** r-mibu has quit IRC | 19:16 | |
| *** r-mibu has joined #tripleo | 19:17 | |
| *** fragatina has quit IRC | 19:17 | |
| *** fragatina has joined #tripleo | 19:18 | |
| *** fragatina has quit IRC | 19:19 | |
| *** fragatina has joined #tripleo | 19:19 | |
| *** abregman has quit IRC | 19:21 | |
| *** fzdarsky has quit IRC | 19:22 | |
| *** bfournie has quit IRC | 19:26 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Update newton release config for new image location https://review.openstack.org/370267 | 19:28 |
| *** fragatina has quit IRC | 19:32 | |
| *** pcaruana has quit IRC | 19:32 | |
| *** davidlenwell has quit IRC | 19:34 | |
| *** fzdarsky has joined #tripleo | 19:34 | |
| pabelanger | okay, that was too easy. python-tripleoclient was able to build an overcloud-full image using the centos-minimal element | 19:37 |
| pabelanger | http://logs.openstack.org/46/371046/3/check/gate-tripleo-buildimage-overcloud-full-centos-7-nv/5d0cf61/logs/dib-overcloud-full.log | 19:38 |
| *** fragatina has joined #tripleo | 19:38 | |
| *** jlinkes has quit IRC | 19:39 | |
| *** itlinux has quit IRC | 19:39 | |
| *** dciabrin is now known as dciabrin|away | 19:40 | |
| *** fragatin_ has joined #tripleo | 19:40 | |
| *** fragatina has quit IRC | 19:40 | |
| *** jprovazn has quit IRC | 19:40 | |
| *** david-lyle has quit IRC | 19:40 | |
| *** david-lyle has joined #tripleo | 19:40 | |
| *** fragatin_ has quit IRC | 19:44 | |
| *** jlinkes has joined #tripleo | 19:46 | |
| *** TicToc has quit IRC | 19:47 | |
| *** dciabrin|away has quit IRC | 19:47 | |
| *** davidlenwell has joined #tripleo | 19:47 | |
| *** ChanServ sets mode: +v davidlenwell | 19:47 | |
| *** itlinux has joined #tripleo | 19:48 | |
| *** fzdarsky_ has joined #tripleo | 19:48 | |
| *** fzdarsky has quit IRC | 19:49 | |
| *** TicToc has joined #tripleo | 19:51 | |
| *** rcernin has joined #tripleo | 19:53 | |
| *** akrivoka has quit IRC | 19:53 | |
| *** lblanchard has quit IRC | 19:53 | |
| *** dciabrin|away has joined #tripleo | 20:01 | |
| *** fragatina has joined #tripleo | 20:01 | |
| *** mburned is now known as mburned_out | 20:03 | |
| *** mburned_out is now known as mburned | 20:05 | |
| *** fragatina has quit IRC | 20:05 | |
| pabelanger | EmilienM: So, check-tripleo jobs for python-tripleoclient seem to be using cached images when deploying ovb? | 20:06 |
| openstackgerrit | Merged openstack/python-tripleoclient: Add `overcloud parameters set` to set Heat params in a plan https://review.openstack.org/370518 | 20:06 |
| pabelanger | does that mean, anything that is landing in python-tripleoclient are by passing testing for tripleo? | 20:07 |
| EmilienM | pabelanger: yes except if it changed recently | 20:07 |
| pabelanger | EmilienM: little confused, then why bother running python-tripleoclient jobs in check-tripleo? I don't see how it is actually getting tested for image builds | 20:09 |
| EmilienM | delorean still build a repo and update tripleoclient in the job | 20:09 |
| *** absubram has joined #tripleo | 20:09 | |
| *** fragatina has joined #tripleo | 20:10 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1623606 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1623606 in tripleo "CI: jobs failing with mistral MessagingTimeout" [Critical,In progress] - Assigned to James Slagle (james-slagle) | 20:10 |
| pabelanger | EmilienM: right, but that is testing that a package can be built if I understand | 20:11 |
| EmilienM | in my opinion, we should use zuul cloner | 20:11 |
| EmilienM | for all of this | 20:11 |
| EmilienM | for all TripleO projects we should use zuul-cloner | 20:11 |
| pabelanger | if so, dmsimard now has a job (3rd party CI) to make sure python-tripleoclient is built properly | 20:11 |
| pabelanger | So, here is my issue | 20:12 |
| pabelanger | https://review.openstack.org/#/c/371046/ | 20:12 |
| EmilienM | bnemec: mhh, ovb still looks unstable | 20:12 |
| pabelanger | changes python-tripleoclient to use centos-minimal element | 20:12 |
| pabelanger | how does CI actually validate the overcloud-full image works before we merge it | 20:12 |
| EmilienM | bnemec: actually, all ovb jobs are failing | 20:12 |
| *** itlinux has quit IRC | 20:13 | |
| pabelanger | I cannot see how it is tested before merge | 20:13 |
| bnemec | EmilienM: Yeah, rh1 got very unhappy when a huge number of jobs came in all at once. | 20:13 |
| pabelanger | then, the periodic job picks it up I think | 20:13 |
| bnemec | pabelanger: tripleo-ci uses the image | 20:13 |
| EmilienM | http://logs.openstack.org/34/371034/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/7c0d6ae/console.html#_2016-09-15_19_08_32_623219 | 20:13 |
| pabelanger | bnemec: I don't see that right now nc 66.187.229.169 19885 | 20:14 |
| EmilienM | bnemec: is there anything we can do? | 20:14 |
| pabelanger | bnemec: it downloaded overcloud-full.tar from the private server | 20:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: CI test - never merge https://review.openstack.org/371034 | 20:14 |
| *** fragatina has quit IRC | 20:15 | |
| bnemec | EmilienM: I don't know. I've been watching it and things seem to have calmed down somewhat, but there are a bunch of testenvs without all of the ports attached. :-( | 20:15 |
| bnemec | My best guess is Neutron got overloaded and just failed to attach ports to instances. | 20:16 |
| *** rhallisey has quit IRC | 20:16 | |
| bnemec | pabelanger: On a tripleoclient patch, or a different project? tripleoclient shouldn't be using the cached image. | 20:17 |
| pabelanger | bnemec: Ya, https://review.openstack.org/#/c/371046/ | 20:18 |
| pabelanger | bnemec: I changed the DIB element from centos7 to centos-minimal, and pretty sure all of CI is going to pass | 20:18 |
| pabelanger | which means, I don't think anything actually deployed the new overcloud image | 20:18 |
| *** snecklifter_ has left #tripleo | 20:20 | |
| bnemec | pabelanger: Sigh. The caching logic is wrong for that project. One second. | 20:21 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/370517 | 20:22 |
| *** jayg is now known as jayg|g0n3 | 20:22 | |
| bnemec | EmilienM: We'll probably also need https://review.openstack.org/370401 | 20:22 |
| EmilienM | bnemec: yes ! | 20:23 |
| EmilienM | bnemec: I'll backport it | 20:23 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Remove settings from release configs https://review.openstack.org/371114 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't use cached images in tripleoclient changes https://review.openstack.org/371115 | 20:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements https://review.openstack.org/350905 | 20:24 |
| bnemec | pabelanger: ^ | 20:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/361875 | 20:26 |
| pabelanger | bnemec: will try shortly | 20:28 |
| *** abregman has joined #tripleo | 20:28 | |
| *** fragatina has joined #tripleo | 20:29 | |
| EmilienM | thrash: do we need https://review.openstack.org/#/c/357195/ ? | 20:30 |
| EmilienM | same for https://review.openstack.org/#/c/357194/ | 20:30 |
| EmilienM | and https://review.openstack.org/#/c/357192/ | 20:30 |
| bnemec | It's taking 20 minutes to create an OVB stack in rh1. | 20:31 |
| bnemec | That's...not good. | 20:32 |
| bnemec | I wonder if our heat db has exploded again... | 20:32 |
| *** abregman has quit IRC | 20:33 | |
| thrash | EmilienM: need as in for newton? | 20:34 |
| thrash | EmilienM: or in general? | 20:34 |
| EmilienM | thrash: in newton | 20:34 |
| thrash | EmilienM: would be nice | 20:35 |
| EmilienM | bnemec: let me know if there is anything I can do for helping there | 20:35 |
| thrash | EmilienM: It's ripping code out. | 20:35 |
| EmilienM | thrash++ | 20:35 |
| EmilienM | thrash: we need to make sure to backport it in stable | 20:35 |
| thrash | EmilienM: stagnant code that shouldn't be used. | 20:35 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Add settings to general config https://review.openstack.org/371144 | 20:35 |
| thrash | EmilienM: don't really care about the tripleo-common ones. Those can wait for ocata. | 20:35 |
| thrash | EmilienM: but getting rid of the commands is definitely ideal. | 20:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: [mitaka-only] mysql: never add brackets to mysql_bind_host https://review.openstack.org/371029 | 20:36 |
| EmilienM | thrash: ok | 20:36 |
| *** bfournie has joined #tripleo | 20:37 | |
| *** fzdarsky_ has quit IRC | 20:37 | |
| bnemec | I wonder if there would be benefit to tuning mysql in rh1. | 20:38 |
| bnemec | We've got a ton of memory on the controller just not being used. | 20:38 |
| *** anshul has quit IRC | 20:41 | |
| *** morazi has quit IRC | 20:42 | |
| *** morazi has joined #tripleo | 20:42 | |
| EmilienM | bnemec: increasing file limit? | 20:42 |
| *** morazi has quit IRC | 20:43 | |
| bnemec | EmilienM: No, that should be good. I'm just wondering if there are some parameters we could tweak to speed things up. | 20:44 |
| bnemec | I feel like our controller should not be struggling to keep up like it is. | 20:44 |
| EmilienM | what is consumming? how is neutron server and ovs? | 20:45 |
| *** Goneri has quit IRC | 20:45 | |
| slagle | EmilienM: multinode jobs look like they are passing | 20:45 |
| EmilienM | slagle: yes | 20:45 |
| EmilienM | I proposed the revert on infra | 20:45 |
| slagle | cool :) | 20:45 |
| bnemec | EmilienM: neutron-server is probably our biggest cpu user. | 20:46 |
| EmilienM | slagle: but I'm a bit perplex and prefer to wait 1h | 20:46 |
| EmilienM | slagle: ftr https://review.openstack.org/#/c/371133/ | 20:46 |
| bnemec | But I was seeing heat stacks where the resources were complete but it just hadn't updated the status yet. | 20:46 |
| bnemec | Which makes me wonder if there was a contention issue with heat on the database or something. | 20:46 |
| slagle | EmilienM: ok, np. perplexed about what though? | 20:46 |
| *** myoung is now known as myoung|bbl | 20:47 | |
| EmilienM | slagle: nothing, just making sure job is stable again | 20:47 |
| EmilienM | and no race is happenning | 20:48 |
| slagle | ok, got it | 20:48 |
| *** trown is now known as trown|outtypewww | 20:48 | |
| EmilienM | bnemec: I would be glad to help but I haven't access. Maybe can I request access sometimes | 20:50 |
| *** akshai has quit IRC | 20:54 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Remove external requirements https://review.openstack.org/370264 | 20:55 |
| *** dsneddon has joined #tripleo | 20:55 | |
| *** dsneddon has quit IRC | 20:55 | |
| *** TicToc has quit IRC | 20:56 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Remove the rest of a removed path https://review.openstack.org/369431 | 20:57 |
| *** TicToc has joined #tripleo | 20:58 | |
| *** anshul has joined #tripleo | 21:00 | |
| bnemec | EmilienM: I think to get access you just submit a review like this: https://review.openstack.org/#/c/353103 | 21:04 |
| EmilienM | ok i'm doing it | 21:05 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-incubator: Allow Emilien Macchi to be root on TripleO Cloud https://review.openstack.org/371157 | 21:08 |
| *** akshai has joined #tripleo | 21:10 | |
| EmilienM | we have 54 bugs WIP for RC1 | 21:11 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.milestone%3Alist=79340&field.tag=&field.tags_combinator=ANY&f | 21:11 |
| EmilienM | ield.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search | 21:11 |
| EmilienM | or https://goo.gl/l93sNA even | 21:11 |
| bkero | wow | 21:12 |
| *** mburned is now known as mburned_out | 21:15 | |
| *** ccamacho has quit IRC | 21:17 | |
| *** absubram has quit IRC | 21:19 | |
| *** mburned_out is now known as mburned | 21:21 | |
| *** absubram has joined #tripleo | 21:24 | |
| EmilienM | slagle: can I close https://bugs.launchpad.net/tripleo/+bug/1623574 ? | 21:25 |
| openstack | Launchpad bug 1623574 in tripleo "CI: all jobs timing out during NetworkDeployment" [Critical,Confirmed] - Assigned to James Slagle (james-slagle) | 21:25 |
| EmilienM | since all OVB jobs are red, i would wait a bit | 21:25 |
| EmilienM | bnemec: have you tried something? all jobs are still red | 21:26 |
| bnemec | EmilienM: I've tried a few things, but not sure it's helping at this point. | 21:30 |
| slagle | have we tried turning it off and on again? | 21:31 |
| slagle | i'm halfway serious | 21:31 |
| *** panda is now known as panda|Zz | 21:32 | |
| bnemec | slagle: I've considered it, but if we resort to that then it's just going to happen again. | 21:36 |
| openstackgerrit | Merged openstack/python-tripleoclient: Remove one last reference to openstackclient.tests https://review.openstack.org/370401 | 21:36 |
| *** adam_g has quit IRC | 21:45 | |
| EmilienM | slagle, bnemec: I just saw the first green ovb job https://review.openstack.org/#/c/371069/ | 21:46 |
| *** saneax-_-|AFK has quit IRC | 21:46 | |
| EmilienM | gate-tripleo-ci-centos-7-ovb-nonha - 2h11 | 21:46 |
| *** vkmc has quit IRC | 21:47 | |
| *** tbarron has quit IRC | 21:47 | |
| bnemec | EmilienM: Yeah, some of the testenvs got created successfully. There were just a lot of failures too. | 21:47 |
| *** jlinkes has quit IRC | 21:47 | |
| *** jlinkes has joined #tripleo | 21:47 | |
| EmilienM | bnemec: I really want to see an ha job passin | 21:48 |
| bnemec | I just rechecked a ci patch to see how it handles 8 jobs coming in at once. | 21:48 |
| EmilienM | and then i'm gone | 21:48 |
| *** pradk has quit IRC | 21:48 | |
| bnemec | EmilienM: You better hope 370401 passes then. Otherwise the next running ha job is at least an hour out. :-) | 21:50 |
| EmilienM | I'll go running a bit | 21:51 |
| *** adam_g has joined #tripleo | 21:51 | |
| *** adam_g has quit IRC | 21:51 | |
| *** adam_g has joined #tripleo | 21:51 | |
| *** tbarron has joined #tripleo | 21:52 | |
| *** saneax-_-|AFK has joined #tripleo | 21:56 | |
| *** vkmc has joined #tripleo | 21:57 | |
| bnemec | That's a good idea. I haven't gotten a lot of exercise lately. | 21:57 |
| *** absubram has quit IRC | 22:02 | |
| *** akshai has quit IRC | 22:02 | |
| *** absubram has joined #tripleo | 22:08 | |
| slagle | EmilienM: bnemec : let's not forget to send an update to the ML before we all sign off | 22:17 |
| slagle | we can wait a bit to see if ovb clears up | 22:17 |
| EmilienM | slagle: yes, I'm online this evening | 22:18 |
| EmilienM | i'll send an email if situation is still that bad | 22:19 |
| EmilienM | I'll update anyway wrt my email of last night | 22:19 |
| bnemec | slagle: It looks happier now. | 22:19 |
| bnemec | It's possible we just need to cron job those db cleanup commands. | 22:19 |
| *** jkraj has joined #tripleo | 22:19 | |
| slagle | bnemec: i think i missed which ones those were | 22:21 |
| slagle | was is the normal keystone token purge and heat raw_template purge? | 22:22 |
| bnemec | slagle: heat-manage purge_deleted 3 | 22:22 |
| bnemec | nova-manage db archive_deleted_rows --verbose --max_rows 1000000 | 22:22 |
| bnemec | I also ran sudo mysqlcheck -o -A to compact the dbs. | 22:22 |
| slagle | ah, k | 22:22 |
| *** absubram has quit IRC | 22:23 | |
| slagle | in logstash, can i get it to return just 1 result in the table per job? | 22:23 |
| slagle | something like a "group by build_id" | 22:23 |
| *** absubram has joined #tripleo | 22:23 | |
| bnemec | slagle: I've wondered the same thing. I'm not aware of anything, but I haven't looked all that hard either. | 22:24 |
| bnemec | Stacks are creating in a mere 11 minutes now. :-/ | 22:25 |
| bnemec | That feels kind of absurd for a half dozen vms and some virtual networks. | 22:26 |
| *** anshul has quit IRC | 22:35 | |
| slagle | bnemec: i guess it's the nova instances | 22:37 |
| slagle | in the stack i'm looking at, the nested stack openstack_baremetal_servers took 6 minutes | 22:37 |
| EmilienM | should we decrease quotas? | 22:39 |
| bnemec | http://paste.openstack.org/show/577739/ | 22:40 |
| *** rlandy is now known as rlandy|bbl | 22:41 | |
| bnemec | We're not exactly taxing the computes. | 22:41 |
| bnemec | | memory_mb | 3608889 | | 22:41 |
| bnemec | | memory_mb_used | 1396224 | | 22:41 |
| bnemec | | running_vms | 246 | | 22:41 |
| bnemec | | vcpus | 696 | | 22:41 |
| bnemec | | vcpus_used | 330 | | 22:41 |
| slagle | 3 minutes for a nova instance | 22:42 |
| slagle | and 3 minutes for all the ports | 22:42 |
| slagle | we might want to enable nova debug logs so we can see more in the scheduler log particularly | 22:43 |
| slagle | maybe we should raise max_concurrent_builds in nova.conf? | 22:44 |
| slagle | it defaults to 10 | 22:44 |
| slagle | just one tripleo-ci patch check is going to go over that, and keeps things waiting | 22:44 |
| bnemec | Yeah, I was wondering about that. Except I'm not sure whether raising or lowering it would help. | 22:44 |
| bnemec | If it's a resource contention issue of some sort then raising it will only make it worse. | 22:45 |
| slagle | yea, definitely | 22:45 |
| bnemec | Although that in itself would be a useful data point. | 22:45 |
| *** chem` has joined #tripleo | 22:49 | |
| *** yamahata has quit IRC | 22:49 | |
| *** chem has quit IRC | 22:51 | |
| *** dsariel has quit IRC | 22:53 | |
| *** absubram has quit IRC | 22:59 | |
| *** absubram has joined #tripleo | 23:03 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 23:09 |
| *** jkraj has quit IRC | 23:13 | |
| *** pmannidi has joined #tripleo | 23:17 | |
| *** chem` has quit IRC | 23:49 | |
| *** mlupton has quit IRC | 23:55 | |
| *** mlupton has joined #tripleo | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
