| *** kbyrne has joined #tripleo | 00:01 | |
| *** TSCHAK has quit IRC | 00:02 | |
| *** akshai has quit IRC | 00:07 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 00:10 |
|---|---|---|
| *** sanjayu has quit IRC | 00:11 | |
| *** dmacpher has joined #tripleo | 00:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 00:17 |
| *** jefrite has joined #tripleo | 00:27 | |
| *** julim has quit IRC | 00:27 | |
| *** tzumainn has quit IRC | 00:29 | |
| *** rajinir has quit IRC | 00:34 | |
| *** mbound has joined #tripleo | 00:44 | |
| *** limao has joined #tripleo | 00:48 | |
| *** mbound has quit IRC | 00:49 | |
| *** kbyrne has quit IRC | 00:57 | |
| *** TSCHAK has joined #tripleo | 01:03 | |
| *** sanjayu has joined #tripleo | 01:16 | |
| *** rhallisey has joined #tripleo | 01:16 | |
| *** liliars has quit IRC | 01:18 | |
| *** limao_ has joined #tripleo | 01:22 | |
| *** yamahata has joined #tripleo | 01:22 | |
| *** openstack has joined #tripleo | 01:23 | |
| *** liliars has joined #tripleo | 01:33 | |
| *** dmacpher has quit IRC | 01:59 | |
| *** yamahata has quit IRC | 02:02 | |
| *** thrash is now known as thrash|g0ne | 02:05 | |
| *** rhallisey has quit IRC | 02:17 | |
| *** myoung has quit IRC | 02:26 | |
| *** myoung has joined #tripleo | 02:27 | |
| *** morazi has quit IRC | 02:30 | |
| *** akshai has joined #tripleo | 02:49 | |
| *** yamahata has joined #tripleo | 02:49 | |
| *** numans has quit IRC | 02:49 | |
| *** fragatina has quit IRC | 02:55 | |
| *** fragatina has joined #tripleo | 02:55 | |
| *** akshai_ has joined #tripleo | 03:01 | |
| *** akshai has quit IRC | 03:05 | |
| *** akuznetsov has joined #tripleo | 03:07 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 03:11 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 03:13 |
| *** sanjayu has quit IRC | 03:22 | |
| *** dmacpher has joined #tripleo | 03:24 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying RGW on controllers. (Composable) https://review.openstack.org/289027 | 03:27 |
| *** numans has joined #tripleo | 03:30 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 03:33 |
| *** akshai_ has quit IRC | 03:37 | |
| *** akshai has joined #tripleo | 03:38 | |
| *** akuznetsov has quit IRC | 03:39 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 03:52 |
| *** MaxPC has joined #tripleo | 04:03 | |
| *** ramishra_ has quit IRC | 04:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/335169 | 04:21 |
| *** links has joined #tripleo | 04:22 | |
| *** ramishra has joined #tripleo | 04:24 | |
| *** MaxPC has quit IRC | 04:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/332383 | 04:28 |
| *** liverpooler has quit IRC | 04:30 | |
| *** akshai has quit IRC | 04:32 | |
| *** sanjayu has joined #tripleo | 04:34 | |
| sanjayu | GM folks | 04:37 |
| * sanjayu learns oooq is tripleo quickstart | 04:37 | |
| *** myoung has quit IRC | 04:42 | |
| *** myoung has joined #tripleo | 04:52 | |
| *** myoung has quit IRC | 04:52 | |
| *** myoung has joined #tripleo | 04:53 | |
| *** ETP has joined #tripleo | 04:58 | |
| *** pcaruana has quit IRC | 04:58 | |
| *** masco has joined #tripleo | 05:02 | |
| *** dmacpher has quit IRC | 05:02 | |
| *** mgunjan has joined #tripleo | 05:03 | |
| mgunjan | hello .. i am trying to do rdo undercloud install but install is failing | 05:04 |
| mgunjan | need help | 05:05 |
| *** Goneri has joined #tripleo | 05:05 | |
| mgunjan | has anyone recently used rdo manager for liberty baremetal install | 05:05 |
| *** numans has quit IRC | 05:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308 | 05:26 |
| *** numans has joined #tripleo | 05:27 | |
| *** dmacpher has joined #tripleo | 05:35 | |
| *** egafford has quit IRC | 05:40 | |
| *** dtantsur|afk is now known as dtantsur | 05:40 | |
| *** numans has quit IRC | 05:48 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop) https://review.openstack.org/319591 | 05:52 |
| *** Goneri has quit IRC | 05:57 | |
| *** numans has joined #tripleo | 05:59 | |
| *** liverpooler has joined #tripleo | 06:00 | |
| *** cmyster has quit IRC | 06:04 | |
| *** dmacpher has quit IRC | 06:08 | |
| *** cmyster has joined #tripleo | 06:09 | |
| *** cmyster has quit IRC | 06:09 | |
| *** cmyster has joined #tripleo | 06:09 | |
| *** eggmaster has quit IRC | 06:10 | |
| *** eggmaster has joined #tripleo | 06:11 | |
| *** pcaruana has joined #tripleo | 06:16 | |
| *** jtomasek_ has joined #tripleo | 06:18 | |
| *** pcaruana is now known as pcaruana|afk| | 06:19 | |
| sanjayu | mgould, what is the problem | 06:28 |
| sanjayu | sorry mgunjan ^ | 06:28 |
| *** spredzy has quit IRC | 06:28 | |
| *** spredzy has joined #tripleo | 06:28 | |
| *** fragatina has quit IRC | 06:29 | |
| *** rasca has joined #tripleo | 06:34 | |
| d0ugal | Anyone got time to give me a second review here? https://review.openstack.org/#/c/322611/ | 06:36 |
| d0ugal | ^ is all green | 06:36 |
| d0ugal | Another small one that is important to the Mistral work: https://review.openstack.org/#/c/331985/ | 06:37 |
| d0ugal | Also all green | 06:37 |
| *** mcornea has joined #tripleo | 06:37 | |
| *** bootsha has joined #tripleo | 06:38 | |
| *** egafford has joined #tripleo | 06:39 | |
| *** ccamacho has quit IRC | 06:41 | |
| *** ramishra has quit IRC | 06:41 | |
| *** aufi has joined #tripleo | 06:41 | |
| *** masco has left #tripleo | 06:42 | |
| *** ramishra has joined #tripleo | 06:42 | |
| *** tremble has joined #tripleo | 06:43 | |
| *** tremble has joined #tripleo | 06:43 | |
| *** ccamacho has joined #tripleo | 06:46 | |
| *** pcaruana|afk| is now known as pcaruana | 06:49 | |
| *** itamarl has joined #tripleo | 06:55 | |
| *** zoli_gone-proxy is now known as zoliXXL | 06:59 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Avoid repetition in endpoint map https://review.openstack.org/308182 | 07:01 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add placeholder for Internal TLS template https://review.openstack.org/328214 | 07:01 |
| *** jprovazn has joined #tripleo | 07:03 | |
| *** jpena|off is now known as jpena | 07:04 | |
| openstackgerrit | Carlos Camacho proposed openstack/instack-undercloud: Use Openstack CLI for setting undercloud quota https://review.openstack.org/333150 | 07:04 |
| *** zoliXXL is now known as zoli|brb | 07:05 | |
| *** tesseract- has joined #tripleo | 07:09 | |
| *** amoralej|off is now known as amoralej | 07:10 | |
| *** paramite has joined #tripleo | 07:11 | |
| *** jpich has joined #tripleo | 07:15 | |
| *** ebalduf has quit IRC | 07:19 | |
| *** hjensas has joined #tripleo | 07:20 | |
| *** hjensas has quit IRC | 07:20 | |
| *** hjensas has joined #tripleo | 07:20 | |
| *** abehl has joined #tripleo | 07:25 | |
| *** bootsha has quit IRC | 07:28 | |
| *** ifarkas has joined #tripleo | 07:30 | |
| *** bootsha has joined #tripleo | 07:31 | |
| *** panda has quit IRC | 07:34 | |
| *** panda has joined #tripleo | 07:35 | |
| *** andreas-f has quit IRC | 07:38 | |
| *** paramite is now known as paramite|afk | 07:38 | |
| *** gfidente has joined #tripleo | 07:39 | |
| *** chem|off` has joined #tripleo | 07:40 | |
| *** chem|off` is now known as chem | 07:40 | |
| *** chem|off has quit IRC | 07:41 | |
| *** ETP has quit IRC | 07:42 | |
| *** ebarrera has joined #tripleo | 07:43 | |
| *** florianf has joined #tripleo | 07:44 | |
| *** jtomasek has joined #tripleo | 07:52 | |
| *** zoli|brb is now known as zoli | 07:55 | |
| *** zoli is now known as zoliXXL | 07:55 | |
| *** jtomasek_ has quit IRC | 07:55 | |
| *** fzdarsky has joined #tripleo | 07:56 | |
| *** jpich has quit IRC | 08:04 | |
| *** ohamada has joined #tripleo | 08:11 | |
| *** bootsha has quit IRC | 08:12 | |
| *** ohamada has quit IRC | 08:14 | |
| *** ohamada has joined #tripleo | 08:15 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 08:16 |
| dtantsur | folks, please-please-please approve https://review.openstack.org/#/c/319297/ and https://review.openstack.org/#/c/316128/. both passed CI, both have 1x +2 already, both tested locally | 08:16 |
| *** jpich has joined #tripleo | 08:17 | |
| *** mgunjan has quit IRC | 08:18 | |
| chem | matbu: hi, are you aware of any doc for upgrading liberty to mitaka ? | 08:18 |
| *** myoung|remote has joined #tripleo | 08:20 | |
| *** myoung has quit IRC | 08:20 | |
| matbu | chem: yep | 08:22 |
| matbu | chem: give me few secs | 08:22 |
| matbu | chem: first this one : https://etherpad.openstack.org/p/tripleo-liberty-mitaka-upgrades | 08:23 |
| matbu | chem: probably easier to follow the jinja template: https://github.com/redhat-openstack/ansible-role-tripleo-overcloud-upgrade/blob/master/templates/major-upgrade-overcloud.sh.j2 | 08:24 |
| chem | matbu: thanks :) | 08:24 |
| marios | check matbu o/ also https://review.openstack.org/#/c/308985/7/doc/source/post_deployment/upgrade.rst should be pretty up to date wrt the migrations i think (there are version specific admonitions) | 08:26 |
| marios | err chem sry ^^ | 08:26 |
| chem | marios: thanks | 08:26 |
| *** paramite|afk is now known as paramite | 08:27 | |
| dtantsur | marios, morning, could you please review a couple of patches (see above)? | 08:29 |
| *** hjensas has quit IRC | 08:30 | |
| *** electrofelix|afk is now known as electrofelix | 08:31 | |
| *** bootsha has joined #tripleo | 08:31 | |
| gfidente | dtantsur, I think ::ironic | 08:33 |
| gfidente | should only be included if sync_db ? | 08:33 |
| gfidente | (in step 3) | 08:33 |
| gfidente | the purpose is to run dbsync only from one node in step 3 | 08:33 |
| dtantsur | gfidente, ::ironic does a lot of things in addition to syncing db | 08:34 |
| gfidente | yeah so the problem is that we don't want any of those (especially not sync db) to happen on all nodes | 08:34 |
| gfidente | in step 3 | 08:35 |
| dtantsur | gfidente, wait, why? we need ironic.conf on all nodes, no? | 08:35 |
| gfidente | yes but not in step 3 | 08:35 |
| gfidente | basically we don't want to be creating the database from multiple nodes at the same time | 08:35 |
| gfidente | because db_sync is idempotent yes, but it might be spawned on multiple nodes at the same time, in which case it fails | 08:36 |
| *** oshvartz has joined #tripleo | 08:36 | |
| dtantsur | gfidente, sync_db is not done by ::ironic, it's done later on step 38 | 08:36 |
| dtantsur | anyway, my previous attempts to do it differently all failed | 08:36 |
| *** sshnaidm has joined #tripleo | 08:36 | |
| *** stendulker has joined #tripleo | 08:36 | |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Correct the path to the tripleo-common workbooks https://review.openstack.org/335376 | 08:37 |
| dtantsur | gfidente, so sync_db is only run if sync_db. I'm really confused.. | 08:37 |
| gfidente | yes | 08:37 |
| gfidente | but in init.pp there is https://github.com/openstack/puppet-ironic/blob/master/manifests/init.pp#L419 | 08:37 |
| gfidente | so include ::ironic | 08:37 |
| gfidente | will run sync from all nodes | 08:37 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal registration https://review.openstack.org/322612 | 08:38 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use Mistral for baremetal introspection https://review.openstack.org/327780 | 08:38 |
| dtantsur | gfidente, interesting; I think we should set sync_db to false then in ::ironic, but we still need it included | 08:38 |
| *** dmk0202 has joined #tripleo | 08:39 | |
| d0ugal | hrm, where can I find the rpm spec tripleo-common? | 08:39 |
| d0ugal | I forget where these live | 08:39 |
| dtantsur | gfidente, anyway, mind if I fix it in a follow-up? it does not break anything (no HA support anyway) | 08:40 |
| gfidente | dtantsur, looking at other submissions | 08:41 |
| gfidente | I think we can set sync_db in puppet-tripleo | 08:41 |
| gfidente | basing on hiera('bootstrap_nodeid') | 08:41 |
| * d0ugal found it: https://github.com/rdo-packages/tripleo-common-distgit/blob/rpm-master/openstack-tripleo-common.spec | 08:41 | |
| gfidente | if $::hostname == downcase($bootstrap_node) | 08:41 |
| dtantsur | gfidente, yeah, but lets do it when it actually supports > 1 controllers | 08:41 |
| gfidente | but we're deploying 3 controllers in ha job anyway no? | 08:42 |
| shardy | gfidente: this isn't actually tested in CI AFAICS because it's not added to the ControllerServices parameter | 08:43 |
| *** zoliXXL is now known as zoli|lunch | 08:43 | |
| gfidente | ah I didn't get to the tht change yet | 08:43 |
| shardy | We'll probably want to make CI add Ironic to the deployed services when the HA part is fixed (even if we don't deploy it by default in tht_ | 08:44 |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Correct the path to the tripleo-common workbooks https://review.openstack.org/335376 | 08:44 |
| dtantsur | gfidente, ironic is not in CI, I have a separate patch to temporary enable and test it; but I didn't implement HA support at all for now | 08:45 |
| dtantsur | ironic HA is a bit interesting, so I'm leaving it for later :) | 08:45 |
| shardy | dtantsur: it'd be good to reference the WIP tripleo-ci patch in the comments to the tht patch, so we can see the results | 08:46 |
| dtantsur | I thought I did it, lemme check | 08:46 |
| shardy | maybe I missed it :) | 08:46 |
| dtantsur | shardy, gfidente: https://review.openstack.org/#/c/329872/ | 08:46 |
| shardy | dtantsur: we will need to remove those unset passwords, even if it's in a followup - we've already had security vulnerabilities due to those bad defaults | 08:48 |
| dtantsur | shardy, I'll follow up with it, just lemme know what exactly to change | 08:48 |
| shardy | dtantsur: just don't ever do "default: unset" in a heat template where a password is passed ;) | 08:48 |
| dtantsur | okie | 08:48 |
| d0ugal | A really quick fix for the broken Mistral workbook loading if anyone has a moment: https://review.openstack.org/#/c/335376/ | 08:49 |
| shardy | we need to add the random password generation to tripleoclient, so we can be sure of a non-predictable value | 08:49 |
| dtantsur | shardy, I think we generate ironic password already | 08:50 |
| shardy | dtantsur: ack, then it's just a case of deleting the default from the heat templates then | 08:50 |
| dtantsur | https://review.openstack.org/#/c/316146/ | 08:50 |
| shardy | Ok, so we'll get a good (random) value then :) | 08:51 |
| d0ugal | shardy: Thanks | 08:52 |
| *** derekh has joined #tripleo | 08:52 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 08:59 |
| social | why everyone has to deliver hacks to tripleoclient? | 09:00 |
| *** athomas has joined #tripleo | 09:01 | |
| dtantsur | lets rename it to hackclient \o/ | 09:01 |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud: Use the print function to fix the tests on Python 3 https://review.openstack.org/335383 | 09:02 |
| d0ugal | dtantsur: lol | 09:03 |
| *** bootsha has quit IRC | 09:03 | |
| d0ugal | social: it is the easiest place to add hacks :( | 09:04 |
| social | just fix the packages and the services | 09:04 |
| *** chlong has quit IRC | 09:04 | |
| social | and if you need hack add it into instack-undercloud ideally the puppet part with reasonable comment | 09:04 |
| *** nijaba has quit IRC | 09:06 | |
| derekh | shardy: are you finished with the last env I had set up? I was going to delete it | 09:07 |
| shardy | derekh: Sure if you need to delete then go ahead | 09:07 |
| *** nijaba has joined #tripleo | 09:07 | |
| *** nijaba has quit IRC | 09:07 | |
| *** nijaba has joined #tripleo | 09:07 | |
| shardy | derekh: I did some good testing of a patch from stevebaker btw, which speeds up nested event listing by a factor of 12 :) | 09:07 |
| derekh | shardy: if your making use of it, it can stick around for another day or so | 09:07 |
| derekh | shardy: aftert that infra will need the capacity (hopefully) | 09:08 |
| shardy | derekh: There's probably some other stuff I could test, but nothing super urgent | 09:08 |
| derekh | shardy: nice speedup | 09:08 |
| shardy | stevebaker has some other DB optimizations I was planning to try out, but I can do that locally instead | 09:08 |
| derekh | shardy: ok, I'll leave it around until tomorrow, if I need to remove it befor then, I'll let you know | 09:09 |
| shardy | derekh: ack, thanks! | 09:09 |
| dtantsur | gfidente, shardy, can we please get to some agreement if we need this sync_db issue fixed right now? I'd like to move on with these patches either way | 09:10 |
| dtantsur | I can fix it when I introduce the actual HA support, before that we can't even test it | 09:12 |
| gfidente | dtantsur, commented on the tht part too, is ::my_ip the bind address? | 09:12 |
| gfidente | dtantsur I mean, we could +2 and get it merged but I don't see much benefit in landing quickly stuff which is known not to work yet | 09:13 |
| *** fzdarsky has quit IRC | 09:13 | |
| dtantsur | gfidente, it works for non-ha and landing it makes my life easier by not requiring to do the whole thing right at one pass :( | 09:13 |
| dtantsur | in this case we risk not having even non-ha ironic by newton | 09:14 |
| dtantsur | gfidente, my_ip is used in many places, but not as bind address for API :) the biggest use case iirc is TFTP address | 09:14 |
| * dtantsur would love to get rid of this my_ip, but we have what we have | 09:14 | |
| gfidente | so it should be the public virtual ip? | 09:14 |
| shardy | dtantsur: Hey, I'm sure we can land this soon, but I commented on https://review.openstack.org/#/c/316128/ re a few minor issues where it doesn't align correctly with the composable services model | 09:15 |
| *** bootsha has joined #tripleo | 09:15 | |
| shardy | I'd prefer to see it fixed in that patch, but would be OK with seeing a follow-up posted | 09:16 |
| dtantsur | shardy, I'm fine with landing only puppet part because it complicates local testing way too much | 09:16 |
| dtantsur | I can fix the THT patch right away | 09:16 |
| shardy | dtantsur: thanks - with CI hopefully working again now it shouldn't be too much delay | 09:16 |
| *** chlong has joined #tripleo | 09:17 | |
| shardy | it'd be good to correct that patch, as it will provide a valuable reference for other folks integrating new services into the new composable services architecture | 09:17 |
| dtantsur | shardy, that = THT? sure. what about puppet-ooo one? | 09:17 |
| shardy | gfidente: can you pls check my comment on https://review.openstack.org/#/c/316128/24/puppet/hieradata/database.yaml? | 09:17 |
| shardy | it looks like since https://review.openstack.org/#/c/305751 landed we should remove all the per-service stuff from hieradata/database.yaml? | 09:18 |
| dtantsur | shardy, looks like other services still have it there, no? | 09:19 |
| gfidente | shardy, yeah we're only wiring database.yaml in controller | 09:19 |
| gfidente | :( | 09:19 |
| shardy | dtantsur: Yeah, hence my question to gfidente, I think it's wrong | 09:19 |
| gfidente | dtantsur, yeah in general it should go with the service template | 09:20 |
| gfidente | but I thought we were wiring database hieradata to all roles | 09:20 |
| gfidente | but we're not so it's definitely better to move it within the service template from the start | 09:21 |
| gfidente | like shardy said | 09:21 |
| shardy | gfidente: We set heat::database_connection in services/heat-engine.yaml, but heat::db::mysql::host in hieradata/database.yaml | 09:21 |
| shardy | seems like that connection IP should be derived from the EndpointMap in heat-engine.yaml in both cases? | 09:21 |
| * dtantsur makes a smart face pretending he understands what's going on | 09:21 | |
| shardy | dtantsur: basically everything associated with service configuration should now do in services/foo.yaml | 09:22 |
| shardy | but we've got a bunch of legacy stuff around from the old monolithic controller implementation | 09:22 |
| dtantsur | the problem here is that my knowledge of THT is limited to copy-pasting things around with s/service_name_1/service_name_2/g :) | 09:23 |
| shardy | dtantsur: if you just remove the VIP addition to controller.yaml and the unset passwords, we can fix the other stuff via a followup if you like | 09:23 |
| shardy | dtantsur: ack, I appreciate it's difficult, sorry about that | 09:23 |
| dtantsur | shardy, I'll fix everything I understand. No worries, I know I'm a bit annoying, but I never touched THT before | 09:23 |
| shardy | dtantsur: one reason I'm nit-picking the review somewhat is I want a nice example to point folks to | 09:23 |
| shardy | "go do it like *this*" ;) | 09:23 |
| shardy | dtantsur: thanks, let me know if you need some help :) | 09:24 |
| dtantsur | shardy, as I said, I'm totally fine with fixing the THT patch; it's the puppet-tripleo patch which is giving me a big pain | 09:24 |
| *** coolsvap has joined #tripleo | 09:24 | |
| shardy | dtantsur: personally I'm OK with landing the initial puppet-tripleo patch and iterating on it | 09:24 |
| dtantsur | thanks; that's why I'm asking how principal gfidente's objections about sync_db are, provided that HA will be in the next patch | 09:25 |
| gfidente | dtantsur, ok let's go ahead given that fixing the logic today | 09:29 |
| sshnaidm | derekh, hi | 09:29 |
| gfidente | doesn't address the problem until we gather sync_db from bootstrap_nodeid | 09:29 |
| dtantsur | thanks, much appreciated; and sorry again for being annoying :) | 09:29 |
| sshnaidm | derekh, I see CI is quite stable right now, do you have any issues that I can handle? | 09:29 |
| gfidente | dtantsur, it's good to have the conversation | 09:30 |
| gfidente | not annoying! | 09:30 |
| shardy | +1 feel free to keep discussing - it's actually been very helpful and has highlighted a couple of things we've missed in the composable services refactor | 09:31 |
| *** sanjayu has quit IRC | 09:32 | |
| derekh | sshnaidm: I think when ever CI jobs arn't blocked a good thing to do is to try and reproduce some of the intermittent errors and see if they are fixable | 09:34 |
| derekh | sshnaidm: for example why did the Ha job fail here https://review.openstack.org/#/c/334305/3 | 09:34 |
| gfidente | dtantsur, if you update tht remember to add the enable-tls.yaml part as I think ironic will be on ssl port | 09:35 |
| gfidente | 13386 | 09:35 |
| gfidente | doh, 13385 | 09:35 |
| derekh | sshnaidm: or if not fixable, what more information can be added to a bug for the problem | 09:35 |
| derekh | sshnaidm: another problem I've been meaning to look into is swift proxy and why its using soo much ram in some cases | 09:36 |
| dtantsur | gfidente, doing it right now | 09:36 |
| dtantsur | shardy, how do I detect the current IP address without passing IronicApiVirtualIP? | 09:40 |
| gfidente | dtantsur, which ip address is the one you need to gather? | 09:41 |
| shardy | dtantsur: that same data is included in the EndpointMap, which is passed to all service/foo templates | 09:41 |
| dtantsur | gfidente, of the controller | 09:41 |
| gfidente | I think this is a problem bigger than it seems | 09:42 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/heat-api.yaml#L37 | 09:42 |
| gfidente | because in servicenetmap we have internalapinetwork | 09:42 |
| dtantsur | shardy, does it contain IPs? | 09:42 |
| gfidente | but for tftp you probably don't want to use that same network? | 09:42 |
| dtantsur | shardy, is it IP, not a URL? | 09:42 |
| gfidente | dtantsur, also if you have 3 controllers, should that be the IP of a different node on each node? | 09:43 |
| gfidente | or rather the internal vip of all the controller nodes | 09:43 |
| dtantsur | gfidente, of the current controller yeah; not the vip | 09:43 |
| shardy | dtantsur: you can just specify "host" like here https://review.openstack.org/#/c/305751/33/puppet/services/heat-engine.yaml | 09:43 |
| dtantsur | well... it might be even worse than that | 09:43 |
| shardy | then you get a hostname, not a URL | 09:44 |
| dtantsur | shardy, hostname or IP? I'm not sure how well TFTP resolves host names tbh | 09:44 |
| gfidente | shardy, also that is a vip, not a local ip | 09:44 |
| *** bootsha has quit IRC | 09:45 | |
| *** sanjayu has joined #tripleo | 09:45 | |
| dtantsur | shardy, gfidente, thinking more about that, ironic is abusing this my_ip thing heavily, so maybe it's better to use a vip | 09:45 |
| *** hjensas has joined #tripleo | 09:45 | |
| gfidente | dtantsur, so if you need a local ip you probably want something like: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/controller.yaml#L881\ | 09:45 |
| dtantsur | cause this value is used as a default e.g. for neutron URL and for TFTP host | 09:45 |
| dtantsur | which is O_o | 09:45 |
| sshnaidm | derekh, you mean "Resource CREATE failed: NotFound: resources.server1_floating_ip" error, right? | 09:45 |
| gfidente | if you need a vip for the service what shardy said instead applies | 09:46 |
| gfidente | EndpointMap | 09:46 |
| sshnaidm | derekh, it happens in some ha jobs, I tried to reproduce it but it didn't, I need to look more. Do you have an issue/bug opened about this? | 09:47 |
| d0ugal | Am I looking in the correct place for the latest tripleo-common rpm? There are more recent commits but it hasn't been updated in 19 days here. http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/ | 09:47 |
| *** tosky has joined #tripleo | 09:48 | |
| derekh | sshnaidm: yes, that looks like it, but its only an example, whats I think is most important is to identify the intermittent bug that occurs most then find fin as much info for a bug as you can, and fix it if its something your familiar with or escalated it to somebody who can fix it | 09:49 |
| dtantsur | gfidente, ok thanks | 09:49 |
| derekh | sshnaidm: nope, I havn't opened a bug | 09:49 |
| gfidente | marios, I am not sure what happened but I think the ceilometer part was reverted | 09:49 |
| gfidente | marios, so it pulled it up in a rebase but now seems newly added | 09:49 |
| gfidente | thanks for pointing it out :) | 09:50 |
| marios | gfidente: ack thanks for looking | 09:50 |
| dtantsur | please review the puppet-tripleo part in the meanwhile | 09:50 |
| sshnaidm | derekh, ok, will look at this | 09:51 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 09:52 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Convert swift storage to composable services https://review.openstack.org/320580 | 09:54 |
| *** fzdarsky has joined #tripleo | 09:55 | |
| derekh | sshnaidm: I'm setting up and dedicated CI env now for reproducing some of this stuff will add details to the ether pad in a bit | 09:55 |
| sshnaidm | derekh, cool! great news | 09:55 |
| derekh | sshnaidm: although rh1 is been taken down next week so it wont be around long | 09:55 |
| ccamacho | Last night I dreamt about TripleO, that the minimum RAM requisites was bumped up to 54GB of RAM... :$ A really crazy dream.. | 09:56 |
| derekh | sshnaidm: once that happen jobs will be on rh2, we can create an account on that with some quota to reporduce things | 09:56 |
| sshnaidm | ccamacho, think about pto :)) | 09:56 |
| ccamacho | hehe, still with energy :) | 09:57 |
| sshnaidm | derekh, great, rh2 is new cloud, right? | 09:57 |
| derekh | sshnaidm: yes | 09:57 |
| sshnaidm | derekh, it will be cool to have some diagram describing the new environment | 09:57 |
| sshnaidm | derekh, when it'll be ready, it may help to newbies to understand the whole picture | 09:58 |
| derekh | sshnaidm: yup, I'll be putting some info together, thankfully its not as complicated as rh1 | 09:59 |
| derekh | sshnaidm: its basically just one openstack cloud | 09:59 |
| derekh | sshnaidm: configured so OVB works | 09:59 |
| derekh | sshnaidm: https://github.com/cybertron/openstack-virtual-baremetal | 09:59 |
| sshnaidm | derekh, oh, great, just wanted to ask you for some resources about ovb :) | 10:00 |
| *** akrivoka has joined #tripleo | 10:00 | |
| derekh | sshnaidm: yup, once the rh2 job is running, I'll create you an env there | 10:01 |
| *** hjensas has quit IRC | 10:02 | |
| sshnaidm | derekh, sorry for stupid question - is this ovb idea to manage bm machines as they were vms? | 10:02 |
| derekh | sshnaidm: yes, it allows ironic to control nova VM's as if they were actual baremetal | 10:04 |
| *** coolsvap has quit IRC | 10:07 | |
| *** coolsvap has joined #tripleo | 10:08 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Add base ironic profiles https://review.openstack.org/319297 | 10:08 |
| dtantsur | \o/ | 10:09 |
| *** limao_ has quit IRC | 10:09 | |
| *** athomas has quit IRC | 10:11 | |
| *** abehl has quit IRC | 10:11 | |
| *** abehl has joined #tripleo | 10:12 | |
| *** coolsvap has quit IRC | 10:13 | |
| *** athomas has joined #tripleo | 10:17 | |
| sshnaidm | derekh, FYI https://bugs.launchpad.net/tripleo/+bug/1597265 I succeeded to reproduce, so will handle it | 10:18 |
| openstack | Launchpad bug 1597265 in tripleo "CI: pingtest fails often in HA jobs because "Resource CREATE failed: NotFound: resources.server1_floating_ip"" [Undecided,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:18 |
| derekh | sshnaidm: great ;-) | 10:18 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Nova API and Neutron server should run database creation in step 3 https://review.openstack.org/335419 | 10:20 |
| *** pblaho has quit IRC | 10:24 | |
| derekh | sshnaidm: added a new host to https://etherpad.openstack.org/p/tripleo-ci-devenvs | 10:25 |
| derekh | sshnaidm: I've taken a testenv away from ci for it details in /home/jenkins/testenv.json | 10:26 |
| *** jtomasek has quit IRC | 10:27 | |
| *** pblaho has joined #tripleo | 10:27 | |
| *** zoli|lunch is now known as zoli | 10:27 | |
| *** zoli is now known as zoliXXL | 10:27 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Basic support for deploying Ironic in overcloud https://review.openstack.org/316128 | 10:30 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: DO NOT MERGE: testing ironic https://review.openstack.org/329872 | 10:30 |
| dtantsur | gfidente, shardy, ^^^ | 10:30 |
| sshnaidm | derekh, ok, thanks | 10:31 |
| *** thrash|g0ne is now known as thrash | 10:32 | |
| *** rasca has quit IRC | 10:35 | |
| *** stendulker has quit IRC | 10:35 | |
| *** bootsha has joined #tripleo | 10:35 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Replace NodeConfigIdentifiers with DeployIdentifier https://review.openstack.org/315616 | 10:36 |
| *** sambetts|afk is now known as sambetts|deadlin | 10:36 | |
| *** rasca has joined #tripleo | 10:36 | |
| *** sambetts|deadlin is now known as sambetts|halfher | 10:38 | |
| *** sambetts|halfher is now known as sambetts|HAK | 10:38 | |
| *** sambetts|HAK is now known as sambetts|HFK | 10:38 | |
| *** bootsha has quit IRC | 10:38 | |
| *** bootsha has joined #tripleo | 10:39 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add Management net to ips-from-pool-all example https://review.openstack.org/318699 | 10:40 |
| *** sambetts|HFK is now known as sam|schrodinger | 10:41 | |
| *** sam|schrodinger is now known as sambetts|cat | 10:42 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 10:45 |
| *** jefrite has quit IRC | 10:46 | |
| *** rasca has quit IRC | 10:48 | |
| *** rasca has joined #tripleo | 10:49 | |
| *** coolsvap has joined #tripleo | 10:49 | |
| *** bootsha has quit IRC | 10:51 | |
| *** bootsha has joined #tripleo | 10:53 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move keystone::db data within service template https://review.openstack.org/335429 | 10:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move glance::db data within service template https://review.openstack.org/335430 | 10:53 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Make deploy validation errors/warnings fatal for CI https://review.openstack.org/320883 | 10:54 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move cinder::db data within service template https://review.openstack.org/335431 | 10:55 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move heat::db data within service template https://review.openstack.org/335434 | 10:58 |
| *** ccamacho is now known as ccamacho|lunch | 10:58 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move sahara::db data within service template https://review.openstack.org/335436 | 11:00 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move neutorn::db data within service template https://review.openstack.org/335438 | 11:02 |
| gfidente | shardy, ^^ I'm doing them one by one this time | 11:03 |
| *** jtomasek has joined #tripleo | 11:08 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Validations new workflow https://review.openstack.org/334770 | 11:10 |
| shardy | gfidente: nice, thanks! :) | 11:12 |
| gfidente | I was also looking at the tree here: https://review.openstack.org/#/c/332740/7 | 11:12 |
| gfidente | I think the submissions combined passed all jobs in the most recent attempts | 11:12 |
| gfidente | but none of them passed all jobs in a single run | 11:13 |
| shardy | Hmm, yeah | 11:13 |
| *** lucasagomes is now known as lucas-hungry | 11:13 | |
| openstackgerrit | vinay kumar muddu proposed openstack/diskimage-builder: Build wholedisk images capable of UEFI secure boot https://review.openstack.org/333888 | 11:13 |
| shardy | I'd like to see that series land, particularly https://review.openstack.org/#/c/322858/ which I need to unblock the custom-roles work | 11:14 |
| gfidente | yeah | 11:14 |
| gfidente | also I need to reprise the bind within service template stuff | 11:14 |
| gfidente | as I think we need to dump ips per network in flat keys rather than hash | 11:14 |
| *** md123 has joined #tripleo | 11:14 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 11:14 |
| shardy | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/logs/postci.txt.gz | 11:17 |
| shardy | looks like we're still timing out trying to connect to keystone :( | 11:17 |
| shardy | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/logs/postci.txt.gz#_2016-06-29_10_01_23_000 | 11:17 |
| gfidente | I need the blockstorage role for ceph | 11:19 |
| gfidente | https://review.openstack.org/#/c/304692/ | 11:19 |
| gfidente | it passed all jobs in the past and we landed the puppet part but tht is still on hold :( | 11:19 |
| *** jcoufal has quit IRC | 11:19 | |
| shardy | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/logs/overcloud-controller-0/var/log/keystone/keystone.txt.gz | 11:20 |
| shardy | there's a bunch of keystone errors there, not sure it's related | 11:21 |
| shardy | rabbitmq_start_0 on overcloud-controller-2 'unknown error' (1): call=79, status=complete, exitreason='none | 11:26 |
| shardy | some pcs status errors also in host_info.txt.gz | 11:26 |
| *** rodrigods has quit IRC | 11:28 | |
| *** rodrigods has joined #tripleo | 11:28 | |
| *** bootsha has quit IRC | 11:29 | |
| *** limao has joined #tripleo | 11:30 | |
| *** bootsha has joined #tripleo | 11:30 | |
| *** panda has quit IRC | 11:34 | |
| *** bootsha has quit IRC | 11:35 | |
| *** panda has joined #tripleo | 11:35 | |
| *** fzdarsky has quit IRC | 11:37 | |
| *** limao has quit IRC | 11:37 | |
| *** lblanchard has joined #tripleo | 11:40 | |
| *** ayoung has quit IRC | 11:43 | |
| *** morazi has joined #tripleo | 11:43 | |
| gfidente | anyone +a https://review.openstack.org/#/c/333326/ ? | 11:44 |
| *** weshay has joined #tripleo | 11:46 | |
| slagle | gfidente: we do still set selinux permissive in the image build though | 11:47 |
| slagle | gfidente: are we sure that works? | 11:47 |
| gfidente | slagle, we also still set it to true in hieradata but first change needs to go in the .pp | 11:48 |
| *** numans has quit IRC | 11:48 | |
| gfidente | slagle, judging from the BZ comments I think we might need updates to the selinux policy to make it pass in enforcing mode | 11:49 |
| gfidente | but the good thing is that we're testing it in CI so we can progress with smaller steps | 11:49 |
| gfidente | *small | 11:49 |
| slagle | gfidente: ok, so this patch doesnt actually change anything since "ceph_osd_selinux_permissive: true" is in the hieradata | 11:50 |
| slagle | might be good to add that to the commit msg | 11:51 |
| gfidente | right | 11:51 |
| slagle | but if we do switch to enforcing, it won't be covered by CI | 11:51 |
| gfidente | okay, in 8 and 9 we had hieradata and .pp in a single repo so a single change could have tested it | 11:51 |
| slagle | since in the image build we still configure selinux to permissive | 11:51 |
| gfidente | sorry I meant liberty/mitaka :( | 11:51 |
| gfidente | but now we need puppet-tripleo and tht so it has to be two separate patches | 11:52 |
| *** noslzzp has joined #tripleo | 11:53 | |
| *** noslzzp has quit IRC | 11:53 | |
| *** noslzzp has joined #tripleo | 11:54 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Don't set selinux in 'permissive' mode on CephStorage nodes https://review.openstack.org/335456 | 11:55 |
| slagle | gfidente: what i'm saying is that these patches don't actually close the bug...selinux is still permissive | 11:57 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Default ceph_osd_selinux_permissive to false https://review.openstack.org/333326 | 11:57 |
| *** akshai has joined #tripleo | 11:57 | |
| *** sanjayu has quit IRC | 11:57 | |
| gfidente | slagle, wait one step at a time, so I've updated the COMMIT for puppet-tripleo and pushed the matching part for THT | 11:57 |
| gfidente | I should say partial-bug there too | 11:59 |
| *** rhallisey has joined #tripleo | 11:59 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Don't set selinux in 'permissive' mode on CephStorage nodes https://review.openstack.org/335456 | 11:59 |
| gfidente | and now the last bit missing would be the change at image building | 12:00 |
| gfidente | right? | 12:00 |
| EmilienM | hello | 12:01 |
| *** jayg|g0n3 is now known as jayg | 12:01 | |
| gfidente | EmilienM, hey! :) | 12:02 |
| EmilienM | gfidente: did we test https://review.openstack.org/#/c/319297/ in THT before merging it? | 12:02 |
| gfidente | EmilienM, yes and it is known to work only on single node | 12:02 |
| gfidente | dtantsur, has both the tht change and the testing change | 12:02 |
| EmilienM | ok | 12:03 |
| gfidente | we had a conversation about it with shardy and dtantsur earlier today | 12:03 |
| gfidente | EmilienM, hey I wanted to move the nova database_connection stuff from controller.yaml within the service templates so we can continue with the tree here: https://review.openstack.org/#/c/335438/ | 12:04 |
| gfidente | sounds good? | 12:04 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 12:06 |
| EmilienM | gfidente: why didn't you move the password? | 12:08 |
| gfidente | EmilienM, which password? | 12:10 |
| *** ccamacho|lunch is now known as ccamacho | 12:10 | |
| EmilienM | gfidente: neutron::db::mysql::password | 12:10 |
| EmilienM | but yeah, it sounds good | 12:10 |
| *** bfournie has quit IRC | 12:10 | |
| gfidente | password is in the correct place | 12:11 |
| *** bfournie has joined #tripleo | 12:11 | |
| gfidente | you mean order? | 12:11 |
| EmilienM | gfidente: oh right | 12:11 |
| EmilienM | gfidente: I'm blind | 12:11 |
| EmilienM | I have noticed that our CI is still not super stable | 12:14 |
| shardy | EmilienM: Yeah, we're still getting timeouts connecting to keystone | 12:14 |
| EmilienM | we havehttp://logs.openstack.org/97/320597/17/check-tripleo/gate-tripleo-ci-centos-7-ha/b11ca3f/console.html#_2016-06-28_23_43_38_350029 | 12:14 |
| shardy | e.g http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/logs/overcloud-controller-0/var/log/keystone/keystone.txt.gz | 12:14 |
| *** bootsha has joined #tripleo | 12:14 | |
| EmilienM | during Overcloud pingtest, uploading demo tenant image to glance | 12:14 |
| EmilienM | shardy: your issue is during deployment or after? | 12:15 |
| EmilienM | shardy: the one I randomly took is after | 12:15 |
| EmilienM | during pingtest | 12:15 |
| shardy | EmilienM: during deployment when puppet tries to configure keystone users etc | 12:15 |
| EmilienM | again? | 12:15 |
| EmilienM | chem: do we have any progress on tunning the timeout in the manifest? | 12:16 |
| EmilienM | it sounds really odd | 12:16 |
| shardy | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/logs/postci.txt.gz#_2016-06-29_10_01_23_000 | 12:16 |
| EmilienM | we increased it to 180s | 12:16 |
| EmilienM | when you see " has been running for more then 20 seconds" | 12:16 |
| EmilienM | it's the issue | 12:16 |
| EmilienM | we can logstash it | 12:16 |
| shardy | Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Command: 'openstack [\"user\", \"show\", \"--format\", \"shell\", [\"admin\", \"--domain\", \"default\"]]' has been running for more then 20 seconds | 12:16 |
| EmilienM | yes | 12:17 |
| EmilienM | that's this one again | 12:17 |
| shardy | EmilienM: when did the 180s change land? | 12:17 |
| EmilienM | again, I'm not sure increasing the timeout is the right path | 12:17 |
| shardy | this failed ~1.5hrs ago | 12:17 |
| shardy | https://review.openstack.org/#/c/322858 | 12:17 |
| EmilienM | https://github.com/openstack/puppet-openstacklib/commit/489099632446b1903986549e5feb80b509f9b77d | 12:18 |
| shardy | Looking at host_info there are also some pcs failures, but I'm not sure if they are related or not | 12:18 |
| EmilienM | ~ 18h ago | 12:18 |
| EmilienM | really, we have 3 min of timeout | 12:18 |
| EmilienM | it's already a lot imho | 12:18 |
| EmilienM | there is something else | 12:18 |
| chem | EmilienM: shardy it can't be pure timeout again. I'm looking at the log | 12:19 |
| shardy | EmilienM: I'm confused, because that error says 20s, are we somehow pulling an old version? | 12:19 |
| *** md123 has quit IRC | 12:19 | |
| shardy | Package puppet-openstacklib.noarch 0:9.0.0-0.20160610212643.7a70e28.el7.centos will be installed | 12:20 |
| shardy | We are using an old version | 12:20 |
| chem | EmilienM: shardy: no it's the new I've added this part "(tried 0, for a total of 0 seconds)" which doesn't match what it should ... | 12:21 |
| chem | EmilienM: shardy: hasty patch, .... so there is still hope for timeout. I'm reviewing it again. | 12:21 |
| EmilienM | for the upgrade job I think we still use the old version | 12:21 |
| EmilienM | because we didn't promote rdo packages | 12:22 |
| gfidente | I'm worried that something might be trying to use Keystone_user before we expect it to | 12:22 |
| EmilienM | but for ha job, we should not deploy puppet modules with packages, right? | 12:22 |
| shardy | can we add that to the list of puppet things we build from source? | 12:22 |
| EmilienM | wait, it's not in the list? | 12:22 |
| *** lucas-hungry is now known as lucasagomes | 12:22 | |
| EmilienM | it's in t-p-e right? | 12:23 |
| *** oshvartz has quit IRC | 12:23 | |
| shardy | https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/puppet-modules/source-repository-puppet-modules#L54 | 12:23 |
| EmilienM | right, so we're good | 12:24 |
| shardy | yes, it is, but then why did we build 7a70e28 not d25bb91 for the undercloud? | 12:24 |
| *** akshai has quit IRC | 12:24 | |
| shardy | I'm assuming that means we use the same old version for the overcloud images (or perhaps even a cached image?) | 12:24 |
| EmilienM | and http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/console.html#_2016-06-29_08_58_34_156908 | 12:25 |
| EmilienM | d25bb915f42bd45fc6d1deb836818bec84b9edab is the latest commit | 12:26 |
| *** akshai has joined #tripleo | 12:27 | |
| *** pradk has joined #tripleo | 12:28 | |
| *** akshai_ has joined #tripleo | 12:28 | |
| *** trown|outtypewww is now known as trown | 12:28 | |
| shardy | EmilienM: but the package we actually install doesn't match that SHA: | 12:29 |
| shardy | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/console.html#_2016-06-29_08_34_09_784495 | 12:29 |
| EmilienM | right | 12:29 |
| EmilienM | but I thought we git checkouted it | 12:29 |
| shardy | the delorean package should still match the SHA AFAIK | 12:30 |
| EmilienM | why do we install puppet modules using packages instead of git? | 12:30 |
| shardy | we build packages from source with delorean, then install those packages | 12:30 |
| EmilienM | shardy: maybe it match the latest tripleo promoted repo? | 12:30 |
| EmilienM | for example, python-tripleoclient is not installed by packaging | 12:30 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Fixes for the update image script in dev mode https://review.openstack.org/335140 | 12:31 |
| *** mgould is now known as mgould|lunch | 12:31 | |
| EmilienM | http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-ha/7e5b0a6/console.html#_2016-06-29_08_34_51_387901 | 12:31 |
| shardy | EmilienM: yes, it matches what's in current-tripleo | 12:31 |
| EmilienM | I'm really confused now | 12:31 |
| EmilienM | I just hope we test master of all modules like before | 12:32 |
| EmilienM | otherwise we have a big problem | 12:32 |
| *** akshai has quit IRC | 12:32 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo: Add RGW to the Ceph rgw profile. https://review.openstack.org/334081 | 12:32 |
| marios | EmilienM: call? | 12:32 |
| EmilienM | marios: yes | 12:33 |
| chem | EmilienM: for the 3 min timeout I'm not sure as the added debug output indicate not retry (tried 0, for a total of 0 seconds). It should be (tried 3, for a total of 180s) . So I'm debugging this part | 12:33 |
| shardy | EmilienM: I think it's an ordering thing, we install something that depends on puppet-openstacklib, then afterwards we build the package | 12:34 |
| shardy | we do need to check if we're still using the old version in a cached overcloud image though | 12:34 |
| shardy | probably it doesn't matter if it's the old version on the undercloud | 12:35 |
| EmilienM | I'm in a call, brb | 12:35 |
| openstackgerrit | Keith Schincke proposed openstack/python-tripleoclient: Add support for CephRgwKey generation https://review.openstack.org/296002 | 12:36 |
| yolanda | hi, question about tripleo-quickstart again... i'm looking to have one job that just launches overcloud-validate, independent of previous steps. So once i get the overcloud deployed, i can launch so many overcloud-validate as I want. But i'm hitting a problem with teardown, it always want to remove my nodes or virthost. Is there some config to don't teardown previous environment? | 12:38 |
| trown | yolanda: ya, just recently panda implemented explicit teardown | 12:39 |
| panda | yolanda: currently we have three levels of teardown: everything, net & nodes, just nodes | 12:41 |
| trown | hmm panda, just looking at the explicit teardown patch[1] and I don't see any option for "teardown none", [1] https://github.com/openstack/tripleo-quickstart/commit/0d30b47966065ceabca54664cfdc1ad35f6d6944 | 12:41 |
| *** amoralej is now known as amoralej|lunch | 12:41 | |
| panda | yolanda: trown yeahm never occured to me that someone would want to retain everything from a previous deployment | 12:42 |
| trown | yolanda: actually for tearing down nothing... wouldnt you just want a playbook that has only the overcloud validate role? | 12:42 |
| *** rlandy has joined #tripleo | 12:43 | |
| social | matbu: so openstack-puppet-modules issue in newton is package issue. the stuck nova-compute in liberty seems to be too new commit of instack-undercloud for opm puppet-nova | 12:43 |
| social | matbu: nova-compute gets stuck because it does not have rabbitmq config :/ | 12:44 |
| trown | yolanda: or you could use tags to only run the overcloud validate when using the default playbook `quickstart.sh -t overcloud-validate -R <release> $VIRTHOST` | 12:44 |
| yolanda | trown, that's what i do but is calling teardown anyway | 12:45 |
| trown | yolanda: I think that is the best solution actually ^, maybe you could make a doc patch for it :) | 12:45 |
| *** egafford has quit IRC | 12:45 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Add extra-vars for enable pacamaker to upgrade ci script https://review.openstack.org/334371 | 12:45 |
| social | what creates delorean-current repo? | 12:46 |
| yolanda | trown, so teardown by default is teardown-nodes, that is called unconditionally | 12:47 |
| trown | yolanda: hmm, ya I think that might be a slight bug in explicit teardown actually | 12:47 |
| trown | yolanda: we either need a "teardown none" or a way to get that behavior | 12:47 |
| yolanda | i was thinking in adding teardown-nodes to skip-tags | 12:48 |
| yolanda | not sure if that will work | 12:48 |
| shardy | derekh: Hey, where does logs/rpm-qa.txt.gz come from? Only the undercloud? | 12:48 |
| trown | yolanda: mind filing a quick bug, and I will put up a patch | 12:48 |
| yolanda | sure | 12:48 |
| shardy | I'm looking for a way to confirm what version of puppet-openstacklib we're actually using for overcloud nodes | 12:48 |
| matbu | social: k, thanks for the update | 12:49 |
| *** bfournie has joined #tripleo | 12:49 | |
| EmilienM | shardy: I should rpm-qa.txt was for the node which run the scripts | 12:50 |
| EmilienM | I thought* | 12:50 |
| trown | panda: wdyt instead of "teardown none" or maybe in addition to, not doing teardown if tags are explicitly passed in. presumably if someone is using tags they have some specific flow they want | 12:50 |
| derekh | shardy: its not one of our files, its probably whats on the jenkins slave, generated from infra scripts | 12:52 |
| *** tzumainn has joined #tripleo | 12:52 | |
| derekh | shardy: look in host_info.txt in the overcloud tarball | 12:52 |
| panda | trown: it's not a bad idea, but make teardown (well, "no teardown") less explicit | 12:52 |
| chem | EmilienM: shardy I think I know why we got this one. When checking if an user exists we override the timeout and set it to 0 (command_timeout = 20 but request_timeout set to 0) to run it only once. Because otherwise it would take forever to check all user when then don't exist (n_users * request_tiemount sec). I'm going to look for an alternate way to check for user existance (getting the listing of all users is not an option because | 12:53 |
| chem | of possible LDAP backend) | 12:53 |
| yolanda | trown, https://bugs.launchpad.net/tripleo-quickstart/+bug/1597331 | 12:54 |
| openstack | Launchpad bug 1597331 in tripleo-quickstart "As a developer, I want to run validation playbooks without tearing down" [Undecided,New] | 12:54 |
| shardy | derekh: ack, thanks! | 12:54 |
| *** jprovazn has quit IRC | 12:54 | |
| trown | panda: ya, I am jus thinking that the UX of needing an extra option for the case yolanda is solving seems unneeded | 12:54 |
| trown | thanks yolanda! | 12:55 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 12:56 |
| trown | yolanda: on the skip-tags idea, I am not sure how ansible handles an explicit tags and matching skip-tags, if last one wins, it should work | 12:56 |
| *** jcoufal has joined #tripleo | 12:59 | |
| *** abehl has quit IRC | 12:59 | |
| d0ugal | thrash: Thanks! | 13:00 |
| *** rook_ has joined #tripleo | 13:02 | |
| *** liverpooler has quit IRC | 13:03 | |
| yolanda | trown, going to try | 13:04 |
| yolanda | --skip-tags teardown-all,teardown-nodes,teardown-inventory | 13:06 |
| yolanda | seemed to work | 13:06 |
| yolanda | but it would be better to have proper teardown none flag.... | 13:07 |
| trown | yolanda: indeed | 13:07 |
| trown | good there is some workaround at least, but I will put up a patch shortly for a better fix | 13:07 |
| yolanda | thanks | 13:08 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Move undercloud post install set facts to pre deploy overcloud https://review.openstack.org/335491 | 13:08 |
| *** cdearborn has joined #tripleo | 13:08 | |
| *** rook_ has quit IRC | 13:11 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Add a method to wait for messages on the websocket https://review.openstack.org/322611 | 13:11 |
| *** abehl has joined #tripleo | 13:11 | |
| EmilienM | chem: http://logs.openstack.org/97/320597/17/check-tripleo/gate-tripleo-ci-centos-7-upgrades/260332e/logs/overcloud-cephstorage-0/var/log/host_info.txt.gz | 13:12 |
| EmilienM | + sudo pcs stonith show --full | 13:12 |
| thrash | d0ugal: np | 13:12 |
| *** rook_ has joined #tripleo | 13:12 | |
| chem | EmilienM: http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-upgrades/a4e20fa/logs/overcloud-controller-0/var/log/cluster/corosync.txt.gz | 13:12 |
| dciabrin | marios, ping do you have a minute to discuss https://review.openstack.org/#/c/325205/24/extraconfig/tasks/major_upgrade_controller_pacemaker_1.sh@5 ? | 13:13 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Add ability to explicitly NOT teardown https://review.openstack.org/335493 | 13:14 |
| EmilienM | crmd: notice: throttle_handle_load: High CPU load detected: 8.670000 | 13:14 |
| EmilienM | in http://logs.openstack.org/58/322858/11/check-tripleo/gate-tripleo-ci-centos-7-upgrades/a4e20fa/logs/overcloud-controller-0/var/log/cluster/corosync.txt.gz | 13:14 |
| EmilienM | is it something we should consider as dangerous? ^ | 13:14 |
| trown | yolanda: ^ https://review.openstack.org/335493 | 13:14 |
| trown | panda: mind checking out https://review.openstack.org/335493 when you get a chance as well? | 13:15 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Disable stonith temporarily during upgrades https://review.openstack.org/334049 | 13:17 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Disable stonith temporarily during upgrades https://review.openstack.org/335494 | 13:17 |
| yolanda | trown, can you extend that flag to collect-logs or other scripts? i run that after overcloud deploy and it tears down nodes as well | 13:18 |
| panda | trown: looking | 13:19 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-ui: Add a reusable component for disable-able Link https://review.openstack.org/335497 | 13:20 |
| trown | yolanda: hmm, collect-logs is broken too? | 13:21 |
| * trown goes to look for a successful job | 13:21 | |
| yolanda | trown, np, i'm hitting another issue with inventory in collect-logs, not the same destroy | 13:21 |
| trown | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-periodic-master-delorean-full-deploy-minimal-50/ looks like it collected all the logs | 13:22 |
| yolanda | i was passing wrong working path on that, it's another issue, sorry | 13:22 |
| trown | np | 13:23 |
| yolanda | i'm redeploying the cloud now (nodes were wiped on previous run), and will test with that patch, to see if i can run the overcloud-deploy in an independent step | 13:23 |
| trown | awesome thanks! | 13:24 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - Controller https://review.openstack.org/335499 | 13:26 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Move undercloud post install set facts to pre deploy overcloud https://review.openstack.org/335491 | 13:27 |
| *** rook has quit IRC | 13:27 | |
| *** jprovazn has joined #tripleo | 13:27 | |
| *** mbound has joined #tripleo | 13:28 | |
| *** fzdarsky has joined #tripleo | 13:29 | |
| *** rook_ is now known as rook | 13:29 | |
| chem | gfidente: manifests/profile/base/keystone.pp in puppet-tripleo is the keystone::role::admin | 13:29 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-quickstart: Move undercloud post install set facts to pre deploy overcloud https://review.openstack.org/335491 | 13:33 |
| EmilienM | it's weird, I still see some https on ha job: http://logs.openstack.org/97/332897/3/check-tripleo/gate-tripleo-ci-centos-7-ha/70d39e4/console.html#_2016-06-28_21_46_34_870913 | 13:33 |
| *** myoung has joined #tripleo | 13:33 | |
| EmilienM | it was last night... but after that we merged python-tripleoclient | 13:34 |
| *** nico_auv has joined #tripleo | 13:34 | |
| EmilienM | I don't see python-tripleoclient installed from source | 13:35 |
| *** links has quit IRC | 13:36 | |
| *** mgould|lunch is now known as mgould | 13:36 | |
| EmilienM | mhh, I think it was not in delorean yet | 13:36 |
| hewbrocca | folks, how are things rolling along with our lovely green CI | 13:37 |
| *** amoralej|lunch is now known as amoralej | 13:37 | |
| *** jeckersb has joined #tripleo | 13:38 | |
| EmilienM | chem: see https://review.openstack.org/#/c/334049/ | 13:38 |
| EmilienM | maybe that's why we had failing cluster on upgrade job | 13:38 |
| *** myoung has quit IRC | 13:39 | |
| EmilienM | chem, shardy: the pingtest issue i told you should be fixed now, it was a delorean thing, last night when doing recheck we didn't download latest tripleoclient | 13:39 |
| EmilienM | now we do, so we're good i think | 13:39 |
| EmilienM | hewbrocca: we still have an random case where ha job fails to create admin user, chem is on it | 13:40 |
| marios | dciabrin: o/ damien sure | 13:41 |
| gfidente | chem, ack, ::roles depends on ::service::end so yes puppet will try to create the user immediately after but it will wait for the service to come up | 13:41 |
| dciabrin | marios, i'll well in favour of creating a dedicated review for the timeout increase, except i'm not sure of why exactely it's needed? | 13:42 |
| EmilienM | we also have this one: https://bugs.launchpad.net/tripleo/+bug/1597265 | 13:42 |
| openstack | Launchpad bug 1597265 in tripleo "CI: pingtest fails often in HA jobs because "Resource CREATE failed: NotFound: resources.server1_floating_ip"" [Critical,Confirmed] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 13:42 |
| *** ayoung has joined #tripleo | 13:42 | |
| EmilienM | that is random too | 13:42 |
| florianf | jtomasek: do you know an quick and easy way to listen to a zaqar queue locally (on the undercloud) | 13:42 |
| *** ayoung has quit IRC | 13:42 | |
| *** ayoung has joined #tripleo | 13:42 | |
| gfidente | shardy, is the keystone Could not evaluate: happening on bootstrap_node ? | 13:43 |
| dciabrin | marios, i.e. is that N-specific, or does it apply for M as well? etc. | 13:43 |
| marios | dciabrin: so fwiw i was hitting it too, as mentioned with the pointer to the relevant review in my comment. compared to last time we were running this (upgrades to Liberty) we have more services now so the older timeout was expiring. we spoke with bandini and at one point he poked at the cib | 13:43 |
| gfidente | cause I think I spotted something which could be making the non-bootstrap nodes trying that | 13:43 |
| openstackgerrit | Ana Krivokapic proposed openstack/tripleo-ui: Add a reusable component for disable-able Link https://review.openstack.org/335497 | 13:43 |
| gfidente | when they aren't supposed to | 13:43 |
| marios | dciabrin: i mean i captured it from my setup and he said the 1800 was roughly a good ballpark figure (I think he may have said 1200 and we agreen 1800 was sensible) | 13:44 |
| shardy | gfidente: Hmm, I only looked at the controller0 logs | 13:44 |
| chem | gfidente: aggreed, the services are definitively up (openstack client connect), but responce is very slow (more than 20s). Anyway I'm slashing the last place where we don't have retry in the puppet code, so this kind of issue shoud be behind us soon. | 13:44 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Remove base/neutron/ml2.pp https://review.openstack.org/332870 | 13:44 |
| dciabrin | marios, ok so shall I open a new lp and report it against M? | 13:44 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: neutron plugins: include neutron base https://review.openstack.org/332897 | 13:44 |
| marios | dciabrin: https://gist.github.com/anonymous/6106df256b2c94aa52e67e165214b8ae from the comments at https://review.openstack.org/#/c/321027/ fwiw | 13:44 |
| *** [1]cdearborn has joined #tripleo | 13:45 | |
| marios | dciabrin: well i was hitting this for mitaka to newton so i guess it is for both but whatever getting a bug and review for that would be good | 13:45 |
| marios | dciabrin: thanks man | 13:45 |
| dciabrin | marios, ok i'll do that later in the afternoon, plz bear with /me getting up to speed :) | 13:46 |
| marios | dciabrin: np thanks for following up, i'll revisit tomorrow morning in reviews time | 13:46 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Set manage_roles for all steps in Keystone/Pacemaker https://review.openstack.org/335505 | 13:48 |
| *** r-mibu has quit IRC | 13:48 | |
| gfidente | EmilienM, shardy, chem ^^ | 13:48 |
| gfidente | fwiw | 13:48 |
| *** r-mibu has joined #tripleo | 13:48 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable Horizon service - puppet-tripleo https://review.openstack.org/335506 | 13:49 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 13:50 |
| *** zoliXXL is now known as zoli|brb | 13:51 | |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Set manage_roles for all steps in Keystone/Pacemaker https://review.openstack.org/335505 | 13:52 |
| gfidente | I think that before this change we were delaying Keystone_user to Step5 | 13:52 |
| chem | gfidente: it means we were triggering the ::keystone::roles::admin in step 4 or step 3 when db_sync was true | 13:53 |
| chem | gfidente: humm ... that would definitively help, good catch :) | 13:53 |
| dtantsur | shardy, gfidente, now the patch fails with Unknown Property IronicApiVirtualIP :( | 13:54 |
| dtantsur | (now that I've removed it from manifests) | 13:54 |
| dtantsur | it's not used in my manifests, and ironic support is not enabled, but it still fails | 13:55 |
| gfidente | chem, it was delayed to step 5 though because in the pacemaker role it was always set to false until step 5 | 13:55 |
| gfidente | and in step 5 pacemaker is creating the apache service | 13:55 |
| shardy | dtantsur: you're still passing it in overcloud.yaml AFAICS | 13:55 |
| gfidente | chem, let's see how it goes | 13:56 |
| dtantsur | shardy, yes, I have to | 13:56 |
| dtantsur | shardy, or rather: I'm sending it to EndpointMap. is it wrong? | 13:56 |
| * dtantsur is too confused by this whole thing... | 13:56 | |
| gfidente | shardy, marios, https://review.openstack.org/#/c/320580/ | 13:56 |
| shardy | dtantsur: you can't pass it on line 902 to the Controller ResourceGroup | 13:56 |
| shardy | dtantsur: OS::TripleO::Controller maps to puppet/controller.yaml, which no longer contains that paramter | 13:57 |
| chem | gfidente: ack, anyway, it's cleaner. | 13:57 |
| gfidente | EmilienM, https://review.openstack.org/#/c/320580/ | 13:57 |
| dtantsur | shardy, so what should I do about it? add somewhere/remove somewhere? | 13:57 |
| shardy | dtantsur: delete line 902 in overcloud.yaml | 13:57 |
| dtantsur | ok, I'll try. thanks | 13:58 |
| *** egafford has joined #tripleo | 13:58 | |
| *** cdearborn has quit IRC | 13:58 | |
| dtantsur | shardy, are the remaining places fine? | 13:58 |
| shardy | dtantsur: yes, I think so | 13:58 |
| shardy | dtantsur: the reason for the error is that list of properties around line 902 are converted into parameters that are input to controller.yaml | 13:59 |
| marios | gfidente: ack tomorrow | 13:59 |
| shardy | since you've now removed IronicApiVirtualIP from controller.yaml, that is not a valid thing to pass into OS::TripleO::Controller anymore | 13:59 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Basic support for deploying Ironic in overcloud https://review.openstack.org/316128 | 13:59 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: DO NOT MERGE: testing ironic https://review.openstack.org/329872 | 13:59 |
| shardy | hence "unknown property" | 13:59 |
| dtantsur | shardy, aha got it. the next attempt ^^^ | 14:00 |
| EmilienM | gfidente: ok | 14:00 |
| *** rook- has joined #tripleo | 14:00 | |
| *** ohamada has quit IRC | 14:02 | |
| *** bootsha has quit IRC | 14:03 | |
| *** cdearborn has joined #tripleo | 14:06 | |
| EmilienM | gfidente: +2 | 14:09 |
| EmilienM | gfidente: not on the keystone thing, I'll wait a bit | 14:09 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Default ceph_osd_selinux_permissive to false https://review.openstack.org/333326 | 14:09 |
| gfidente | EmilienM, yeah the keystone thing has deep implications | 14:09 |
| EmilienM | yeah | 14:10 |
| gfidente | needs to be reviewed carefully | 14:10 |
| EmilienM | gfidente: can we land the swift stuff? it's passing ci | 14:11 |
| *** mbound has quit IRC | 14:13 | |
| chem | EmilienM: the bug for last timeout problem on puppet side https://bugs.launchpad.net/keystone/+bug/1597357 | 14:15 |
| openstack | Launchpad bug 1597357 in OpenStack Identity (keystone) "When kestone is slow to respond getting user fails" [Undecided,New] | 14:15 |
| EmilienM | chem: ok, I'm changing it to be puppet-keystone, not keystone | 14:16 |
| social | matbu: but something's wrong with my RDO env :( | 14:17 |
| chem | EmilienM: oups ... sorry and thanks | 14:17 |
| EmilienM | trown: fyi ^ this one too | 14:17 |
| matbu | social: liberty ? | 14:18 |
| *** [1]cdearborn has quit IRC | 14:18 | |
| *** ebalduf has joined #tripleo | 14:18 | |
| matbu | social: /me too, it's not a good day today | 14:18 |
| social | I have partial liberty partial trunk and I have no idea how delorean trunk got there | 14:19 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 14:19 |
| gfidente | EmilienM, in HA job glance is backed by swift so we've tested it in CI and pingtest passed | 14:19 |
| gfidente | so yes I think it's good to merge | 14:20 |
| EmilienM | gfidente: I +2 | 14:20 |
| EmilienM | gfidente: ask marios maybe | 14:20 |
| *** zoli|brb is now known as zoli | 14:20 | |
| trown | EmilienM: k, I think RDO HA jobs are now failing to db sync, again because of resource contention when trying to fit everything in 32G host with slow CPU... I am checking to make sure it doesn't reproduce locally, but for RDO/tripleo-quickstart case we may have to do single node pacemaker to test pacemaker path, at least until we can selectively choose a smaller set of services for the overcloud | 14:21 |
| *** dtantsur is now known as dtantsur|afk | 14:21 | |
| gfidente | EmilienM++ :) | 14:21 |
| *** ohamada has joined #tripleo | 14:22 | |
| gfidente | trown, is it neutron failing dbsync? | 14:22 |
| trown | gfidente: no, it is nova and keystone on node and glance on another, which is what leads me to think it is just resource contention | 14:23 |
| derekh | the port we use to serve out /httpboot on the undercloud "8088", is that a standard port iPXE or something we(or ironic) selected? | 14:23 |
| trown | gfidente: will know shortly if I can't get it to happen locally | 14:24 |
| EmilienM | chem: also tagging tripleo so we can add ci tag and easily track it | 14:25 |
| gfidente | trown, it's probably unlrelated but https://review.openstack.org/#/c/335419 | 14:25 |
| jtomasek | florianf: with GUI installed? | 14:25 |
| *** tosky has quit IRC | 14:25 | |
| gfidente | trown, previously the galera-ready should have ensured appropriate ordering anyway | 14:25 |
| gfidente | with roles we can't assume galera-ready is on the same node so those moved to another step | 14:26 |
| florianf | jtomasek: without. I'm trying to debug the messaging flow. | 14:26 |
| EmilienM | bnemec: can we close https://bugs.launchpad.net/tripleo/+bug/1459348 ? | 14:26 |
| openstack | Launchpad bug 1459348 in tripleo "puppet ha job timeout too close to Jenkins timeout" [Medium,Triaged] | 14:26 |
| EmilienM | sshnaidm: can we close https://bugs.launchpad.net/tripleo/+bug/1572976 or you still see it? | 14:27 |
| openstack | Launchpad bug 1572976 in tripleo "Periodically Apache httpd restart fails" [Medium,Triaged] | 14:27 |
| EmilienM | jistr: what is status on https://bugs.launchpad.net/tripleo/+bug/1458520 ? can we close it? | 14:27 |
| openstack | Launchpad bug 1458520 in tripleo "puppet jobs failing to start memcached" [Low,Triaged] - Assigned to Jiří Stránský (jistr) | 14:27 |
| *** noslzzp has quit IRC | 14:28 | |
| EmilienM | trown, shardy: ok this is the list of our current bugs in CI: | 14:29 |
| *** myoung|remote has quit IRC | 14:29 | |
| EmilienM | https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=ci+&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_d | 14:29 |
| EmilienM | upes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search | 14:29 |
| EmilienM | oops | 14:29 |
| jtomasek | florianf: I used this to list the messages http://paste.openstack.org/show/524064/ | 14:29 |
| jtomasek | florianf: not sure if that helps | 14:29 |
| *** myoung|remote has joined #tripleo | 14:29 | |
| EmilienM | shardy, trown: https://goo.gl/oNcBKP | 14:29 |
| jistr | EmilienM: yup, closed it | 14:30 |
| gfidente | thrash, good to see it worked :P | 14:30 |
| gfidente | thrash, have more https://review.openstack.org/#/c/320580/ | 14:30 |
| gfidente | and https://review.openstack.org/#/c/320580/ | 14:30 |
| gfidente | <3 | 14:31 |
| thrash | gfidente: ack. :) | 14:31 |
| gfidente | thanks! | 14:31 |
| gfidente | oh sorry that was same link twice | 14:32 |
| florianf | jtomasek: thanks! does this show all messages or just the ones that haven't been delivered? | 14:32 |
| gfidente | the other submission was https://review.openstack.org/335419 | 14:32 |
| thrash | gfidente: the puppet already merged for that? | 14:32 |
| gfidente | thrash, no | 14:33 |
| gfidente | it's in the depends-on | 14:33 |
| thrash | gfidente: ack | 14:33 |
| florianf | jtomasek: btw, piping the output to python -m json.tool formats the json: http://paste.openstack.org/show/524067/ | 14:33 |
| jtomasek | florianf: all of them. I don't think there is a way to determine if message is delivered. | 14:34 |
| jtomasek | florianf: thanks | 14:34 |
| florianf | jtomasek: ok, so at least I know my introspection workflow actually creates a zaqar message. better than nothing :-) | 14:34 |
| jtomasek | florianf: note that it shows 20 messages at max | 14:34 |
| florianf | jtomasek: thx | 14:34 |
| jtomasek | florianf: then you need to update the marker if there are more of them | 14:35 |
| florianf | jtomasek: how do I do that? | 14:35 |
| openstackgerrit | Merged openstack/python-tripleoclient: Add support for CephRgwKey generation https://review.openstack.org/296002 | 14:35 |
| *** itamarl has quit IRC | 14:35 | |
| jtomasek | florianf: ...es?marker=0&limit=20" in the url | 14:36 |
| *** fultonj has joined #tripleo | 14:36 | |
| jtomasek | florianf: so this brings first 20 messages, es?marker=20&limit=20" brings 20-40 etc. | 14:36 |
| florianf | jtomasek: oh, nice, thanks | 14:37 |
| *** hewbrocca is now known as hewbrocca-afk | 14:39 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix JOB_NAME parameter https://review.openstack.org/334305 | 14:43 |
| *** sanjayu has joined #tripleo | 14:44 | |
| EmilienM | slagle: nice spot on https://review.openstack.org/#/c/329686/ (nova libvirt profile), I need to re-do the profile, I missed that thing | 14:44 |
| *** ebarrera has quit IRC | 14:44 | |
| shardy | EmilienM: https://review.openstack.org/#/c/320580 looks ready to land, are you happy to approve it? | 14:45 |
| derekh | lucasagomes: the port we use to serve out /httpboot on the undercloud "8088", is that a standard port iPXE or something we(or ironic) selected? | 14:45 |
| shardy | I +2'd although I was a co-author of that patch | 14:45 |
| EmilienM | shardy: we need https://review.openstack.org/#/c/320574/ | 14:46 |
| EmilienM | shardy: you can +A it first and then we'll approve the THT patch | 14:46 |
| shardy | EmilienM: there is a question from marios there, gfidente can you check that please? | 14:48 |
| *** dsariel has joined #tripleo | 14:49 | |
| sshnaidm | EmilienM, the bug 1572976 still appears sometimes, so we can't close it | 14:49 |
| openstack | bug 1572976 in tripleo "Periodically Apache httpd restart fails" [Medium,Triaged] https://launchpad.net/bugs/1572976 | 14:49 |
| gfidente | looking | 14:49 |
| shardy | I think it's fine because we set manage/enabled in the ConfigSettings hiera, so it will be correct for all steps | 14:50 |
| EmilienM | sshnaidm: ack | 14:50 |
| gfidente | yeah | 14:50 |
| gfidente | from template | 14:50 |
| *** abehl has quit IRC | 14:50 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 14:52 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Fix openstack-tripleo-common package name https://review.openstack.org/334467 | 14:52 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 14:52 |
| derekh | slagle: how far are we from a multinode job? rh1 will be going away tuesday or wednedsay of next week | 14:55 |
| shardy | gfidente: Hmm, actually, maybe marios is right, and we need a pacemaker specific profile, that includes steps 4 and 5, only setting manage_service/enabled in step 5? | 14:57 |
| *** tosky has joined #tripleo | 14:57 | |
| gfidente | shardy, nah swift is not under pcmk | 14:58 |
| *** hewbrocca-afk is now known as hewbrocca | 14:58 | |
| shardy | gfidente: Ok, so the logic in https://review.openstack.org/#/c/320580/12/puppet/manifests/overcloud_controller_pacemaker.pp doesn't really do anything then | 14:58 |
| shardy | and we can safely start the services in step4 | 14:59 |
| shardy | thanks | 14:59 |
| slagle | derekh: my local testing is going fine | 14:59 |
| * shardy may have figured that out when writing the patch, then forgotten ;) | 14:59 | |
| slagle | derekh: we need to get the project-config patch merged and the tripleo-ci patches merged | 14:59 |
| openstackgerrit | Merged openstack/python-tripleoclient: Run post deploy config on force https://review.openstack.org/334486 | 14:59 |
| gfidente | I think it was delaying the start of swit-storage to step 5 which is when pcmk was starting the services | 15:00 |
| slagle | derekh: there was some preference from infra for us to start with a 2 node job | 15:00 |
| slagle | derekh: which would mean an aio overcloud, but i haven't been able to get that working yet | 15:01 |
| shardy | gfidente: ack, thanks - perhaps you could comment on the bug, then we'll approve it, and the tht patch | 15:01 |
| *** trozet has quit IRC | 15:01 | |
| shardy | s/bug/patch | 15:01 |
| gfidente | commented | 15:04 |
| *** yamahata has quit IRC | 15:04 | |
| *** dsariel has quit IRC | 15:05 | |
| shardy | gfidente: thanks! EmilienM I think you can re-approve https://review.openstack.org/#/c/320574 now :) | 15:05 |
| lucasagomes | derekh, we selected it | 15:08 |
| lucasagomes | derekh, you can add whatever port to the http server, by default ironic looks for 8088, but can be changed in the [deploy] http_url configuration option | 15:09 |
| EmilienM | shardy: done | 15:10 |
| EmilienM | I'll +A the THT once puppet is merged | 15:10 |
| shardy | EmilienM: thanks! | 15:11 |
| *** tesseract- has quit IRC | 15:12 | |
| *** rajinir has joined #tripleo | 15:12 | |
| *** lblanchard1 has joined #tripleo | 15:13 | |
| *** trozet has joined #tripleo | 15:16 | |
| *** lblanchard has quit IRC | 15:16 | |
| *** pcaruana has quit IRC | 15:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: nova: manage migration bits in a composable way https://review.openstack.org/335557 | 15:18 |
| pradk | EmilienM, when you get a chance can you review https://review.openstack.org/#/c/333954/ https://review.openstack.org/#/c/333475/ https://review.openstack.org/#/c/329692/ | 15:19 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: First iteration of libvirt and nova-compute as a composable services https://review.openstack.org/329686 | 15:19 |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: Don't pass None via UpdateIdentifier https://review.openstack.org/334598 | 15:20 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add puppet profile for swift storage https://review.openstack.org/320574 | 15:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: First iteration of libvirt and nova-compute as a composable services https://review.openstack.org/329686 | 15:22 |
| EmilienM | slagle: see https://review.openstack.org/335557 and https://review.openstack.org/329686 -- I think it should be good now, I made it composable | 15:22 |
| EmilienM | shardy: +a'ing swift thing | 15:22 |
| EmilienM | pradk: ok | 15:23 |
| Ng | is there a summary/blueprint/wiki/whatever which covers the composable role stuff and what it's all about? | 15:23 |
| slagle | EmilienM: k will take a look | 15:23 |
| Ng | I have been tasked with figuring out the current state of that work, because we want to do something that depends on it | 15:23 |
| EmilienM | thx | 15:23 |
| EmilienM | pradk: I did recheck, not passing CI | 15:23 |
| slagle | Ng: bp: https://blueprints.launchpad.net/tripleo/+spec/composable-services-within-roles | 15:24 |
| slagle | and there's a linked etherpad | 15:24 |
| Ng | slagle: awesome, thanks | 15:24 |
| slagle | Ng: ccamacho also wrote up docs on the interface, https://review.openstack.org/#/c/311512/ | 15:24 |
| Ng | how much is there left to convert? | 15:24 |
| slagle | Ng: see etherpad :) | 15:25 |
| Ng | :) | 15:25 |
| pradk | EmilienM, thx if the code looks good if we can get some +2's we can land when ci passes | 15:25 |
| slagle | i think that's everything we know about | 15:25 |
| ccamacho | Ng, about what is left, check https://etherpad.openstack.org/p/tripleo-composable-services | 15:25 |
| pradk | EmilienM, should be pretty straight forward changes | 15:25 |
| ccamacho | And a review is welcomed for the tutorial | 15:25 |
| EmilienM | pradk: | 15:25 |
| EmilienM | pradk: I'm still curious | 15:26 |
| EmilienM | https://review.openstack.org/#/c/329692/3/puppet/hieradata/controller.yaml | 15:26 |
| Ng | I was also wondering if I could persuade someone to pop the second +2 on https://review.openstack.org/#/c/332127/2 :D | 15:26 |
| pradk | EmilienM, one of those will conflict with ceilo composable .. I'll rebase and include the change in composable patch once merged .. we want these backported to mitaka | 15:26 |
| EmilienM | why don't we patch puppet-gnocchi default? | 15:26 |
| derekh | slagle: ack, unless there is some sort of disaster the job on rh2 should be ok, once I got it running I'll see if I can help out with the mutilnode stuff | 15:27 |
| pradk | EmilienM, because its an optional flag we want puppet-gnocchi to be flexible? | 15:27 |
| ccamacho | Ng, what I actually did yesterday to see what is left was to open https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp for the controller case and compare it with the etherpad status which assume is updated | 15:27 |
| pradk | EmilienM, if someone wants to only use metrics and dont want to bother with events they dont need this | 15:27 |
| EmilienM | mhh ok | 15:28 |
| Ng | ccamacho: ok, thanks. I have some reading to do :) | 15:28 |
| slagle | derekh: i noticed the multinode patch is failing the regular ci jobs now, so i'll look into that as well | 15:28 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Convert swift storage to composable services https://review.openstack.org/320580 | 15:29 |
| EmilienM | pradk: +2 | 15:29 |
| derekh | slagle: ok, I rebased it about 40 minutes ago, so you should have a recent ci result soon | 15:29 |
| derekh | if not already | 15:29 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: keystone: deploy composable firewall rules https://review.openstack.org/330785 | 15:30 |
| EmilienM | gfidente: if you have time, please look ^ | 15:30 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles https://review.openstack.org/333556 | 15:30 |
| pradk | EmilienM, thx | 15:30 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move nova::db data within service template https://review.openstack.org/335561 | 15:30 |
| EmilienM | ccamacho: sorry... can you rebase your timezone patches? :) we landed swift | 15:31 |
| ccamacho | Hey! Sure :) | 15:31 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Drop unused VIP params to controller.yaml https://review.openstack.org/333511 | 15:31 |
| ccamacho | EmilienM, nice we are landing patches again :) | 15:31 |
| EmilienM | thanks! | 15:31 |
| rbrady | rwsu: do you still have time to continue working on https://review.openstack.org/#/c/330283/? If not, do you mind if I update it for you? | 15:32 |
| *** pcaruana has joined #tripleo | 15:32 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Configure BlockStorage services via resource chains https://review.openstack.org/322858 | 15:34 |
| *** panda has quit IRC | 15:34 | |
| *** panda has joined #tripleo | 15:35 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Timezone service - Controller https://review.openstack.org/331528 | 15:35 |
| d0ugal | Trivial review anyone? https://review.openstack.org/#/c/335376/ | 15:37 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: neutron/plugins/ml2/bigswitch: do not require agent https://review.openstack.org/333398 | 15:37 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Timezone service - Compute https://review.openstack.org/331530 | 15:38 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Timezone service - Controller https://review.openstack.org/331528 | 15:38 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Timezone service - CephStorage https://review.openstack.org/331534 | 15:38 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Timezone service - ObjectStorage https://review.openstack.org/331536 | 15:38 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Wire NTP and Timezone services into BlockStorage nodes https://review.openstack.org/332740 | 15:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: deploy composable neutron service https://review.openstack.org/331234 | 15:38 |
| *** jreypo has joined #tripleo | 15:38 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Switch Ceph Monitor/OSD/Client/External to composable roles https://review.openstack.org/304692 | 15:38 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: profile/nova/compute: add iscsi bits https://review.openstack.org/331246 | 15:38 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: moving out iscsi bits, now in puppet-tripleo https://review.openstack.org/331247 | 15:39 |
| *** aufi has quit IRC | 15:39 | |
| EmilienM | shardy, gfidente: can you guys look https://review.openstack.org/#/c/332954/ please? | 15:39 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Remove NFS bits from overcloud_compute manifest https://review.openstack.org/332955 | 15:40 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 15:40 |
| shardy | EmilienM: lgtm | 15:41 |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable Horizon service - puppet-tripleo https://review.openstack.org/335506 | 15:41 |
| *** jpich has quit IRC | 15:42 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 15:42 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Move undercloud post install set facts to pre deploy overcloud https://review.openstack.org/335491 | 15:42 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 15:43 |
| *** Nakato has quit IRC | 15:43 | |
| gfidente | EmilienM, commented though looks different from overcloud_compute.pp | 15:44 |
| *** Nakato has joined #tripleo | 15:45 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Re-enable Ceilometer composable roles for controller https://review.openstack.org/333498 | 15:45 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Ceilometer Compute Agent composable service https://review.openstack.org/331802 | 15:45 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Implement Ceilometer Polling Agent service https://review.openstack.org/332483 | 15:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: services/ceilometer: switch to polling agent https://review.openstack.org/332504 | 15:48 |
| EmilienM | gfidente: ok looking now | 15:48 |
| openstackgerrit | Merged openstack/instack-undercloud: Correct the path to the tripleo-common workbooks https://review.openstack.org/335376 | 15:48 |
| EmilienM | gfidente: why do you want a parameter? | 15:49 |
| EmilienM | what does it change? | 15:49 |
| gfidente | it doesn't change the behaviour | 15:50 |
| gfidente | but it binds the selinux setting where it belongs | 15:50 |
| gfidente | even the parameter name seems misleading to me | 15:51 |
| *** yamahata has joined #tripleo | 15:51 | |
| EmilienM | gfidente: see my comments | 15:51 |
| gfidente | but the reality is that we need that to happen on the compute nodes when cinder uses nfs | 15:51 |
| gfidente | so it's a relation in between the two services | 15:51 |
| EmilienM | right but I don't think a parameter is useful but I can add it if you like | 15:52 |
| EmilienM | gfidente: see my review in gerrit, and reply. I'll adjust | 15:52 |
| *** jpich has joined #tripleo | 15:53 | |
| *** trozet has quit IRC | 15:54 | |
| *** fzdarsky has quit IRC | 15:56 | |
| *** zoli is now known as zoli|gone | 15:57 | |
| *** zoli|gone is now known as zoli_gone-proxy | 15:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: profile/base/nova: declare nova class properly https://review.openstack.org/328347 | 15:59 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Remove usage of ::nova class in THT https://review.openstack.org/325983 | 15:59 |
| slagle | derekh: i'm not sure why the CI jobs are failing on the multinode patch: http://logs.openstack.org/77/324777/15/check-tripleo/gate-tripleo-ci-centos-7-upgrades/a91763d/console.html#_2016-06-28_07_15_13_024288 | 15:59 |
| slagle | derekh: is not everything under /opt/stack/new a git repo? | 15:59 |
| derekh | slagle: its a problem with the patch, one sec | 16:00 |
| slagle | derekh: i did make a change in create_dib_vars_for_puppet yesterday | 16:00 |
| slagle | to do $(ls $TRIPLEO_ROOT/puppet-*) | 16:00 |
| slagle | insetad of just $TRIPLEO_ROOT/puppet-* | 16:00 |
| slagle | the problem with the latter is that if no puppet-* dirs exist, it assumes you want the puppet-* literal, which doesnt exist | 16:01 |
| gfidente | EmilienM, on nova-compute restart, it probably isn't dangerous but I also think it's safer to not force it | 16:01 |
| *** trozet has joined #tripleo | 16:01 | |
| derekh | slagle: https://review.openstack.org/#/c/334787/32/scripts/common_functions.sh | 16:01 |
| *** jreypo has quit IRC | 16:01 | |
| derekh | slagle: line 279 | 16:01 |
| derekh | slagle: you need the -d | 16:01 |
| EmilienM | sudo find /etc/neutron/plugins/ml2/ml2_conf.ini $WORKSPACE -type f | sudo xargs chmod 644 | 16:01 |
| EmilienM | what is that? | 16:01 |
| derekh | slagle: otherwise you'll list the contents of the puppet folders | 16:01 |
| EmilienM | derekh: ^ | 16:02 |
| derekh | slagle: I that one of the things I was talking about when I said there was a few things we might move out of my patch and into your | 16:02 |
| slagle | derekh: ah, ok, yea we'll need that in 324777 | 16:02 |
| derekh | slagle: I'll push a new version of your patch up in a bit with everything I've had to tweak | 16:03 |
| slagle | k | 16:03 |
| EmilienM | derekh: is the neutron thing a workaround? | 16:03 |
| derekh | EmilienM: the end of the CI jobs include a ZUUL, hook thing to get logs, its trying to read /etc/neutron/plugins/ml2/ml2_conf.ini, which it doesn't have permission to do | 16:04 |
| derekh | EmilienM: without giving it permission, you get 0 logs from a CI job, yes its a workaround | 16:05 |
| EmilienM | oh, makes sense | 16:05 |
| *** tremble has quit IRC | 16:05 | |
| EmilienM | I thought we had an issue on puppet-neutron | 16:06 |
| *** ifarkas has quit IRC | 16:06 | |
| derekh | EmilienM: nope, alls good | 16:08 |
| *** Goneri has joined #tripleo | 16:08 | |
| *** coolsvap_ has joined #tripleo | 16:08 | |
| *** athomas has quit IRC | 16:10 | |
| EmilienM | gfidente: ok, I'm addressing your comments | 16:10 |
| *** jmiu__ has joined #tripleo | 16:11 | |
| EmilienM | gfidente: thx man | 16:11 |
| gfidente | EmilienM, !?!? | 16:11 |
| gfidente | you're doing the work | 16:11 |
| *** coolsvap has quit IRC | 16:11 | |
| *** sanjayu has quit IRC | 16:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: nova/compute: add NFS bits https://review.openstack.org/332954 | 16:13 |
| EmilienM | gfidente: is it better? ^ | 16:13 |
| *** HenryG has quit IRC | 16:16 | |
| *** HenryG has joined #tripleo | 16:16 | |
| *** coolsvap_ is now known as coolsvap | 16:17 | |
| gfidente | EmilienM, see my last comment, so we set both parameters from same CinderEnableNfsBackend | 16:18 |
| gfidente | making effectively both Cinder and Nova know about NFS even though deployed on different nodes | 16:19 |
| EmilienM | gfidente: right, I'll update the tht patch | 16:19 |
| gfidente | it's not great but at least we know what is driving them | 16:19 |
| gfidente | a bad workaround would be a role deployed on all roles with the only purpose of sharing data | 16:21 |
| gfidente | but that would go against service-isolation | 16:21 |
| gfidente | as we'd have service specific params in non-service template :( | 16:21 |
| EmilienM | right, we won't do that I think | 16:22 |
| gfidente | shardy, I think we could create an output in services.yaml and merge that output from all *ServiceChain in overcloud.yaml ? | 16:26 |
| gfidente | ^^ this would be to share settings from service to all nodes | 16:26 |
| shardy | gfidente: Yes, that's the sort of change I was thinking we'd need | 16:26 |
| gfidente | ah nice | 16:27 |
| gfidente | I'll try that tomorrow if you don't do it first | 16:27 |
| shardy | we probably don't want to share the step_config, but there could be some other key which we merge for all roles | 16:27 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-docs: Group advanced topics by type https://review.openstack.org/335232 | 16:29 |
| gfidente | shardy, create_$service_endpoint | 16:29 |
| gfidente | emitted by $service | 16:29 |
| gfidente | so that we can selectively create the endpoints from keystone role | 16:29 |
| gfidente | and I don't remember why we don't create the endpoint from $service anyway? | 16:30 |
| gfidente | ah cause it needs an additional step | 16:30 |
| rwsu | rbrady: sure, go for it | 16:30 |
| rbrady | rwsu: ack, thanks | 16:30 |
| gfidente | so we *could* do it but it increases time to deplo | 16:30 |
| *** dmk0202 has quit IRC | 16:31 | |
| *** pcaruana has quit IRC | 16:36 | |
| *** fpan has joined #tripleo | 16:37 | |
| *** florianf has quit IRC | 16:38 | |
| *** mcornea has quit IRC | 16:39 | |
| *** jpich has quit IRC | 16:41 | |
| *** fpan has quit IRC | 16:42 | |
| shardy | Can anyone please paste the output of sudo rabbitmqctl list_users from a working undercloud please? | 16:46 |
| shardy | After re-running openstack undercloud install I'm seeing nova-compute unable to connect to rabbit & there's not rabbit connection details in nova.conf | 16:46 |
| thrash | Listing users ... | 16:50 |
| thrash | 6ef3e06da12054901e6730142e10786a8e96e9d7 [administrator] | 16:50 |
| thrash | shardy: ^^^ | 16:50 |
| *** jpena is now known as jpena|off | 16:50 | |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo: Adds OpenDaylight https://review.openstack.org/335173 | 16:50 |
| shardy | thrash: thanks, so it looks like it's because rabbit is listening on br-ctlplane and nova is trying to connect on localhost | 16:51 |
| *** gfidente is now known as gfidente|afk | 17:00 | |
| shardy | raised bug #1597438 - possibly a puppet regression I guess | 17:00 |
| openstack | bug 1597438 in tripleo "re-running undercloud install fails with hanging nova-compute" [Undecided,Triaged] https://launchpad.net/bugs/1597438 | 17:00 |
| *** coolsvap is now known as coolsvap|dinner | 17:01 | |
| *** krotscheck is now known as krotscheck_vaca | 17:01 | |
| *** krotscheck_vaca is now known as krot_vaca_jul19 | 17:01 | |
| *** Goneri has quit IRC | 17:05 | |
| *** trown is now known as trown|lunch | 17:08 | |
| *** abehl has joined #tripleo | 17:09 | |
| *** fultonj has quit IRC | 17:10 | |
| *** Goneri has joined #tripleo | 17:12 | |
| *** andreas-f has joined #tripleo | 17:12 | |
| *** ccamacho is now known as ccamacho|away | 17:14 | |
| bnemec | Sigh. And my undercloud idempotency test merge conflicted last night. | 17:15 |
| *** dmk0202 has joined #tripleo | 17:16 | |
| *** lucasagomes is now known as lucas-afk | 17:17 | |
| shardy | I think it may be because we don't specify nova::rabbit* in the undercloud hieradata for puppet-nova's init.pp | 17:19 |
| shardy | but I'm confused how it ever worked if that's the case | 17:19 |
| *** coolsvap|dinner has quit IRC | 17:20 | |
| *** shardy has quit IRC | 17:20 | |
| social | shardy: nova::default_transport_url | 17:20 |
| yolanda | hi, can i get another +2 for https://review.openstack.org/329556 ? the tripleo-quickstart change to customize the t-h-t path | 17:24 |
| *** ohamada has quit IRC | 17:33 | |
| *** abehl has quit IRC | 17:33 | |
| *** julim has joined #tripleo | 17:38 | |
| trown|lunch | yolanda: we actually have a single +2 policy for tripleo-quickstart at the moment becauase there are only 2 dedicated core reviewers, when we add another we will change it to 2x +2. | 17:40 |
| trown|lunch | I will merge it though :) | 17:40 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Allow to specify templates path on overcloud deployment https://review.openstack.org/329556 | 17:40 |
| *** electrofelix has quit IRC | 17:41 | |
| *** [1]cdearborn has joined #tripleo | 17:41 | |
| *** cdearborn has quit IRC | 17:41 | |
| *** amoralej is now known as amoralej|off | 17:42 | |
| *** mgould is now known as mgould|afk | 17:42 | |
| *** tosky has quit IRC | 17:46 | |
| *** ayoung has quit IRC | 17:47 | |
| *** ayoung has joined #tripleo | 17:47 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 17:48 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 17:48 |
| *** ebalduf has quit IRC | 17:48 | |
| derekh | slagle: ^^ updates your patch with some things I needed | 17:48 |
| derekh | bnemec: last try, instances failed to deploy http://logs.openstack.org/87/334787/32/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/e9a05f1/console.html | 17:49 |
| derekh | bnemec: I'll poke at it a bit more later but in the mean time if you think of anything feel free to push a new version of ^^, no need to wait for that CI to finish it was just a shuffle of things from my patch to slagles | 17:50 |
| derekh | bnemec: with a timout, so we might get some logs | 17:50 |
| *** derekh is now known as derekh_afk | 17:50 | |
| bnemec | derekh: Okay, I'll take a look. | 17:50 |
| *** thrash is now known as thrash|biab | 17:51 | |
| *** chlong has quit IRC | 17:52 | |
| *** fultonj has joined #tripleo | 17:56 | |
| *** Goneri has quit IRC | 18:04 | |
| pradk | can i get a +A on https://review.openstack.org/#/c/333954/ please | 18:05 |
| *** chem is now known as chem|off | 18:07 | |
| slagle | yes :) | 18:07 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/332383 | 18:09 |
| pradk | slagle, thx! :) | 18:09 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix ceilometer auth url https://review.openstack.org/333954 | 18:13 |
| *** akshai_ has quit IRC | 18:13 | |
| *** julim has quit IRC | 18:16 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Fix ceilometer auth url https://review.openstack.org/335612 | 18:19 |
| *** abehl has joined #tripleo | 18:20 | |
| *** [1]cdearborn has quit IRC | 18:20 | |
| *** [1]cdearborn has joined #tripleo | 18:20 | |
| *** xinwu has quit IRC | 18:22 | |
| slagle | derekh_afk: i'm going to split up the multinode tripleo-ci patch into 2, one for just the refactor, and one for the multinode support | 18:24 |
| slagle | derekh_afk: that way you can base the ovb patch on just the refactor patch | 18:24 |
| slagle | i think it will be easier to get it landed that way | 18:24 |
| *** akshai has joined #tripleo | 18:29 | |
| *** dmk0202 has quit IRC | 18:32 | |
| *** sambetts|cat is now known as sambetts|afk | 18:33 | |
| *** dmk0202 has joined #tripleo | 18:34 | |
| *** trown|lunch is now known as trown | 18:36 | |
| *** dmk0202 has quit IRC | 18:37 | |
| yolanda | ah, trown ... even better :) thanks for landing that | 18:40 |
| derekh_afk | bnemec: the problem looks like its mtu on eth1 | 18:40 |
| derekh_afk | bnemec 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000 | 18:41 |
| derekh_afk | bnemec: but I'm sure I'm setting it, one sec | 18:41 |
| derekh_afk | bnemec: hmm http://logs.openstack.org/87/334787/32/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/e9a05f1/console.html#_2016-06-29_15_27_24_847889 | 18:42 |
| derekh_afk | bnemec: ok, I'm gonna change the patch to set it after the undercloud install, incase that switches it back | 18:43 |
| *** ayoung has quit IRC | 18:44 | |
| *** ayoung has joined #tripleo | 18:44 | |
| *** thrash|biab is now known as thrash | 18:44 | |
| *** fragatina has joined #tripleo | 18:44 | |
| *** fragatina has quit IRC | 18:44 | |
| *** fragatina has joined #tripleo | 18:45 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 18:45 |
| derekh_afk | slagle: ack, sounds good | 18:45 |
| *** jayg is now known as jayg|g0n3 | 18:45 | |
| *** jeckersb is now known as jeckersb_gone | 18:45 | |
| derekh_afk | slagle: there is some other duplicated stuff I think we can remove, I'll have a go a clean some of it up after | 18:46 |
| slagle | derekh_afk: k, i'm about to push the split | 18:46 |
| derekh_afk | slagle: ok | 18:46 |
| *** julim has joined #tripleo | 18:47 | |
| *** derekh_afk is now known as derekh_afk_ish | 18:47 | |
| *** egafford has quit IRC | 18:50 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 18:52 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Refactor common code for tripleo-ci https://review.openstack.org/335622 | 18:52 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 18:54 |
| slagle | derekh_afk_ish: k, all done ^ | 18:55 |
| derekh_afk_ish | slagle: cool, thanks | 18:55 |
| *** fragatina has quit IRC | 18:56 | |
| *** julim has quit IRC | 18:58 | |
| *** jeckersb_gone is now known as jeckersb | 18:59 | |
| *** sambetts|afk has quit IRC | 19:02 | |
| *** therve has quit IRC | 19:02 | |
| *** jokke_ has quit IRC | 19:03 | |
| *** egafford has joined #tripleo | 19:03 | |
| *** sambetts has joined #tripleo | 19:04 | |
| *** jokke_ has joined #tripleo | 19:05 | |
| *** myoung|remote is now known as myoung | 19:06 | |
| *** therve has joined #tripleo | 19:07 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 19:09 |
| yolanda | trown, do collect_logs script need to be called with sudo? i'm getting Timeout (12s) waiting for privilege escalation prompt on controller and computes | 19:10 |
| trown | yolanda: shouldn't... this is how we call it in RDO CI https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/tripleo-quickstart-cleanup.sh | 19:12 |
| trown | just `bash collect-logs.sh` | 19:12 |
| yolanda | trown, i'll need to debug but times out waiting for a prompt, in the collect list of installed rpms, kernel, etc | 19:14 |
| *** derekh_afk_ish has quit IRC | 19:16 | |
| trown | yolanda: hmm... I am not as familiar with the internals of that role, the main author is on PTO until next week as well | 19:17 |
| yolanda | i increased verbosity to see more details | 19:18 |
| yolanda | btw, collect-logs doesn't support --ansible-debug flag, i can send a patch for it | 19:18 |
| trown | sure | 19:19 |
| Ng | slagle: thanks for the links about composable roles, some good info in there. I'm left with one question, which is, do we feel like the implementation is at a point where feature work can begin on top of them? or are they not ready for that yet? | 19:23 |
| Ng | looking at the list of work done so far, I'm thinking the answer is they are ready to be built on, if only because so many have already landed :) | 19:24 |
| slagle | Ng: yea, i think the interface and pattern is pretty well established at this point | 19:25 |
| slagle | Ng: what feature did you have in mind? | 19:25 |
| Ng | slagle: we have some HA stuff we're tasked with doing for an imminent downstream release :) | 19:26 |
| Ng | and aiui, it assumes that composable roles is a thing | 19:26 |
| slagle | Ng: ok; just fyi, what we are working on now is just the services side | 19:27 |
| slagle | so the services are composable | 19:28 |
| slagle | but the roles themselves (controller, compute, etc) are still static | 19:28 |
| Ng | ok | 19:28 |
| slagle | so you can say what services you want to make up the 4 roles we offer | 19:28 |
| slagle | true composable roles is coming next, and the implementation for that is much less defined right now | 19:28 |
| Ng | slagle: is that more of an N-or-later thing? | 19:29 |
| slagle | Ng: but shardy has got a prototype: https://review.openstack.org/#/c/315679/ | 19:29 |
| Ng | err, I mean O-or-later | 19:29 |
| slagle | i think there's a lot of people hoping we get it in newton :) | 19:30 |
| Ng | heh | 19:30 |
| slagle | but there is only so much sand left in this hourglass | 19:30 |
| *** xinwu has joined #tripleo | 19:30 | |
| EmilienM | slagle: /b morazi | 19:31 |
| EmilienM | oops | 19:31 |
| EmilienM | I need to learn weechat one day | 19:31 |
| yolanda | trown, it seems to be failing on BECOME-SUCCESS | 19:31 |
| *** akshai has quit IRC | 19:31 | |
| Ng | slagle: thanks very much for the info, I can now go and sound like I know what I'm talking about ;) | 19:31 |
| slagle | Ng: np, just dont tell anyone you heard it from me | 19:31 |
| Ng | haha | 19:32 |
| *** akshai has joined #tripleo | 19:32 | |
| EmilienM | slagle: err, I wanted to ping you about snmp patch to know if we could land it: https://review.openstack.org/#/c/320597/ | 19:32 |
| EmilienM | but I realize it needs a rebase | 19:33 |
| EmilienM | it's passing CI now though | 19:33 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] - Add support for OVB based CI https://review.openstack.org/334787 | 19:34 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Refactor common code for tripleo-ci https://review.openstack.org/335622 | 19:34 |
| *** panda has quit IRC | 19:34 | |
| *** panda has joined #tripleo | 19:35 | |
| openstackgerrit | Bob Fournier proposed openstack/tripleo-heat-templates: Remove NeutronEnableTunnelling from THT and puppet manifests https://review.openstack.org/335235 | 19:36 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Convert SNMP configuration to composable services format https://review.openstack.org/320597 | 19:38 |
| EmilienM | ok it's rebased ^ and ready to land imo | 19:38 |
| *** jmiu__ has quit IRC | 19:39 | |
| pradk | EmilienM, you missed a spot in the conflict | 19:42 |
| EmilienM | pradk: damn, nice catch | 19:43 |
| EmilienM | thx | 19:43 |
| pradk | np | 19:43 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Convert SNMP configuration to composable services format https://review.openstack.org/320597 | 19:44 |
| *** fragatina has joined #tripleo | 19:47 | |
| *** akshai has quit IRC | 19:50 | |
| *** fragatina has quit IRC | 19:51 | |
| Slower | 2016-06-29 19:51:56.793 28494 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 9348b2da-c98f-42bb-be9e-5ccd209ec4 | 19:52 |
| Slower | 5a, please check neutron logs for more information. | 19:52 |
| Slower | 2016-06-29 19:52:03.135 22170 ERROR neutron.callbacks.manager AttributeError: 'NoneType' object has no attribute 'notify_routers_updat | 19:53 |
| Slower | ed' | 19:53 |
| *** akshai has joined #tripleo | 19:53 | |
| Slower | what the heck is going wrong here.. been stuck ont his for a while now | 19:53 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Drop unused VIP params to controller.yaml https://review.openstack.org/333511 | 19:55 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Re-enable Ceilometer composable roles for controller https://review.openstack.org/333498 | 19:55 |
| *** fragatina has joined #tripleo | 19:55 | |
| *** Guest20454 is now known as mgagne | 19:58 | |
| *** mgagne has joined #tripleo | 19:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Ceilometer Compute Agent composable service https://review.openstack.org/331802 | 19:58 |
| EmilienM | pradk: thx for your help on the ceilo profiles | 19:58 |
| *** jayg|g0n3 is now known as jayg | 19:59 | |
| pradk | EmilienM, np, i'm fixing few things as i run into them | 19:59 |
| *** fultonj has quit IRC | 20:00 | |
| EmilienM | I'm rebasing polling | 20:00 |
| pradk | cool | 20:00 |
| openstackgerrit | Merged openstack/python-tripleoclient: Print success/failure message after image build https://review.openstack.org/331289 | 20:00 |
| openstackgerrit | Merged openstack/python-tripleoclient: fix the output message about 'driver_info/deploy_kernel' https://review.openstack.org/319123 | 20:01 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: services/ceilometer: switch to polling agent https://review.openstack.org/332504 | 20:01 |
| EmilienM | pradk: you can review it when you have time ^ | 20:01 |
| pradk | will do | 20:01 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: nova: manage migration bits in a composable way https://review.openstack.org/335557 | 20:02 |
| Slower | hmm, openvswitch isn't running | 20:02 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: First iteration of libvirt and nova-compute as a composable services https://review.openstack.org/329686 | 20:02 |
| *** stevebaker has quit IRC | 20:02 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: deploy composable neutron service https://review.openstack.org/331234 | 20:03 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: compute: moving out iscsi bits, now in puppet-tripleo https://review.openstack.org/331247 | 20:03 |
| *** jmiu_ has quit IRC | 20:03 | |
| *** marios has quit IRC | 20:03 | |
| *** cschwede has quit IRC | 20:03 | |
| *** adarazs has quit IRC | 20:04 | |
| pradk | EmilienM, missed a spot on conflict again :) | 20:04 |
| EmilienM | damn | 20:04 |
| *** jschlueter has quit IRC | 20:04 | |
| *** slagle has quit IRC | 20:04 | |
| *** psanchez has quit IRC | 20:04 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: services/ceilometer: switch to polling agent https://review.openstack.org/332504 | 20:05 |
| pradk | EmilienM, dont we support ipmi for polling? | 20:06 |
| EmilienM | pradk: do we deploy ipmi agent today? | 20:08 |
| *** andreas-f has quit IRC | 20:09 | |
| pradk | EmilienM, its supported in ceilo.. but its probably not in tripleo ? we could add that separate if we want to achieve feature parity with the above patch first | 20:10 |
| *** jayg is now known as jayg|g0n3 | 20:10 | |
| pradk | EmilienM, polling agent supports all 3 name sapces compute, central and ipmi | 20:11 |
| pradk | namespaces* | 20:11 |
| *** jmiu_ has joined #tripleo | 20:11 | |
| EmilienM | pradk: right, in a next iteration, ++ for adding it | 20:12 |
| EmilienM | right now we're doing feature parit | 20:12 |
| EmilienM | parity | 20:12 |
| EmilienM | feel free to add it to our todo | 20:12 |
| pradk | EmilienM, sure puppet ceilo already supports it so should be easy.. i'll add it once these merge | 20:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: don't show error if no images needs to be cleaned-up https://review.openstack.org/335655 | 20:13 |
| EmilienM | pradk++ | 20:13 |
| *** egafford has quit IRC | 20:13 | |
| EmilienM | I see a lot of this recently: | 20:14 |
| EmilienM | NotFound: resources.server1_floating_ip: External network 131d9cde-bee2-40e1-8da6-bb2b6fd1668b is not reachable from subnet fa2af21d-2d38-47b2-b434-dc033cc29b9b. Therefore, cannot associate Port 4eeb5a17-4cf5-446c-b3d0-e9ec4013fc9a with a Floating IP. | CREATE_FAILED | 20:14 |
| EmilienM | during pingtest | 20:14 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bug/1597265 | 20:14 |
| openstack | Launchpad bug 1597265 in tripleo "CI: pingtest fails often in HA jobs because "Resource CREATE failed: NotFound: resources.server1_floating_ip"" [Critical,Confirmed] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 20:14 |
| EmilienM | I'm wondering if anyone already started to investigate it ^ | 20:15 |
| EmilienM | bnemec ^ fyi | 20:15 |
| Slower | I think I've seen that | 20:17 |
| bnemec | I have not been looking into it, no. | 20:18 |
| EmilienM | ok I'm looking | 20:18 |
| *** julim has joined #tripleo | 20:19 | |
| Slower | I've seen all the errors apparently | 20:20 |
| EmilienM | it seems like private net can't connect to public network when maanging the router | 20:22 |
| EmilienM | I found something weird | 20:25 |
| EmilienM | with FLOATING_IP_CIDR /me investigating it | 20:26 |
| EmilienM | ok there is a race | 20:27 |
| *** egafford has joined #tripleo | 20:27 | |
| EmilienM | sometimes FLOATING_IP_CIDR=10.0.0.0/24 | 20:27 |
| EmilienM | instead of 192.168.2.0 | 20:27 |
| EmilienM | see: WORKING job: http://logs.openstack.org/97/332897/4/check-tripleo/gate-tripleo-ci-centos-7-nonha/48c8504/console.html#_2016-06-29_17_32_10_377327 | 20:27 |
| bnemec | EmilienM: That's probably okay, depending on the job. | 20:27 |
| EmilienM | NOT WORKING: http://logs.openstack.org/97/332897/4/check-tripleo/gate-tripleo-ci-centos-7-ha/342dcdf/console.html#_2016-06-29_17_33_50_257564 | 20:27 |
| bnemec | Some of them are net-iso, in which case you get 10. and others are not net-iso, which would be 192. | 20:28 |
| EmilienM | bnemec: look http://logs.openstack.org/97/332897/4/check-tripleo/gate-tripleo-ci-centos-7-ha/342dcdf/console.html#_2016-06-29_17_35_19_534176 | 20:28 |
| EmilienM | that's not ok for the stack | 20:28 |
| EmilienM | External network 131d9cde-bee2-40e1-8da6-bb2b6fd1668b is not reachable from subnet fa2af21d-2d38-47b2-b434-dc033cc29b9b | 20:28 |
| EmilienM | which makes sense | 20:28 |
| sshnaidm | EmilienM, I've started investigating it but still don't know what is the problem exactly, unfortunately. | 20:29 |
| sshnaidm | EmilienM, need a help from somebody that knows more heat and tripleo.. | 20:29 |
| EmilienM | bnemec is right, the ip addressing looks good | 20:30 |
| EmilienM | something happens in heat though, I'm looking | 20:30 |
| sshnaidm | EmilienM, seems like router or one of its interfaces isn't created in time | 20:30 |
| EmilienM | failing job: http://logs.openstack.org/97/332897/4/check-tripleo/gate-tripleo-ci-centos-7-ha/342dcdf/console.html#_2016-06-29_17_33_50_257229 | 20:30 |
| bnemec | Yeah, the networking _looks_ right there, but obviously something is going sideways. | 20:30 |
| EmilienM | working job : http://logs.openstack.org/37/299937/8/check-tripleo/gate-tripleo-ci-centos-7-ha/d8b2599/console.html#_2016-06-29_19_37_19_325698 | 20:30 |
| EmilienM | 2016-06-29 17:35:27.397945 | | router_interface | 3cc41f5b-9dc9-446c-9ce3-03f8c2173e46 | state changed | CREATE_IN_PROGRESS | 2016-06-29T17:31:17 | | 20:32 |
| bnemec | Heh, was just about to post that. | 20:32 |
| EmilienM | ok so heat is still trying to connect the router to ext net | 20:32 |
| EmilienM | but in the meantime it assign the VM a floating ip | 20:33 |
| EmilienM | it can't work... | 20:33 |
| bnemec | Maybe we need a depends_on in the tenant template? | 20:33 |
| EmilienM | shouldn't it be automatic in heat? | 20:33 |
| EmilienM | we can add depends_on: <resource ID or list of ID> on the resource | 20:33 |
| EmilienM | ok I'm trying a patch | 20:34 |
| bnemec | Maybe Heat only knows that the network is created, but not whether the underlying wiring has completed? | 20:34 |
| bnemec | I don't actually know what the template router configuration looks like off the top of my head. | 20:34 |
| bnemec | Shocking, I know. :-) | 20:34 |
| bnemec | EmilienM: Yeah, the floating ip needs a depends_on for the router_interface. | 20:35 |
| EmilienM | ok | 20:35 |
| EmilienM | I have a patch coming | 20:35 |
| *** jprovazn has quit IRC | 20:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest/tenantvm_floatingip.yaml: connect vrouter before creating fip https://review.openstack.org/335663 | 20:38 |
| EmilienM | et voilà | 20:38 |
| Slower | cool | 20:38 |
| Slower | here's what I'm getting now | 20:38 |
| Slower | Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerLoadBalancerDeployment_Step1.resources[0]: Deployment to server failed: status code: 1 | 20:39 |
| Slower | checking controller.. | 20:39 |
| EmilienM | Slower: postci.log and grep "Error:" | 20:40 |
| EmilienM | show puppet logs? | 20:40 |
| *** fragatina has quit IRC | 20:40 | |
| *** [1]cdearborn has quit IRC | 20:40 | |
| *** chem|off has quit IRC | 20:41 | |
| *** [1]cdearborn has joined #tripleo | 20:41 | |
| Slower | hmm | 20:41 |
| Slower | Jun 29 20:26:50 localhost os-collect-config: Could not retrieve fact='mongodb_is_master', resolution='<anonymous>': 757: unexpected to | 20:41 |
| Slower | ken at '2016-06-29T20:26:46.872+0000 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused | 20:41 |
| Slower | EmilienM: ok | 20:41 |
| *** chem|off has joined #tripleo | 20:41 | |
| *** akrivoka has quit IRC | 20:41 | |
| thrash | Slower: that happens a lot on errors. Doesn't always mean anything. | 20:41 |
| Slower | ok | 20:42 |
| Slower | where's postci.log? | 20:42 |
| Slower | I just ran deploy by hand.. openstack overcloud deploy --templates=tripleo-heat-templates | 20:42 |
| Slower | with master heat templates | 20:42 |
| Slower | thrash: EmilienM: can you guys help me debug this? | 20:44 |
| EmilienM | Slower: do you still go running? | 20:45 |
| Slower | https://paste.fedoraproject.org/386478/33107146/ | 20:45 |
| Slower | EmilienM: it's still up.. create failed | 20:45 |
| EmilienM | messages:Jun 29 20:26:50 localhost os-collect-config: #033[1;31mError: Could not find class ::tripleo::profile::base::swift::storage for overcloud-controller-0.localdomain on node overcloud-controller-0.localdomain#033[0m | 20:45 |
| Slower | yeah | 20:45 |
| EmilienM | it's related to our merges today | 20:45 |
| EmilienM | we landed swift profiles | 20:45 |
| EmilienM | just run recheck | 20:45 |
| Slower | I'm trying to get it going locally | 20:46 |
| Slower | so I need new images probably | 20:46 |
| EmilienM | yes | 20:47 |
| EmilienM | make sure you run trunk of tht & puppet tripleo | 20:47 |
| Slower | k | 20:47 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates: Increase cluster sync timeout for M->N major upgrades https://review.openstack.org/335666 | 20:49 |
| Slower | EmilienM: and I'm going to need your patch aren't I/? | 20:50 |
| *** akshai has quit IRC | 20:52 | |
| *** shivrao has joined #tripleo | 20:52 | |
| thrash | Slower: or you can use the swift upload trick. | 20:52 |
| EmilienM | Slower: maybe :) | 20:52 |
| EmilienM | thrash: is it documented? | 20:53 |
| Slower | I'm just trying to get a working stack so I can work on container stuff | 20:53 |
| *** dmk0202 has joined #tripleo | 20:54 | |
| EmilienM | bnemec: can you land https://review.openstack.org/#/c/335494/ please? | 20:54 |
| thrash | EmilienM: sort of? :P | 20:54 |
| bnemec | EmilienM: Yes, although it's a backport proposed by a core, so we generally count that as the first +2. | 20:57 |
| thrash | EmilienM: Slower docs here... it's what I used: https://review.openstack.org/#/c/311512/30/doc/source/tht_walkthrough/tips.rst | 20:58 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Disable stonith temporarily during upgrades https://review.openstack.org/335494 | 21:02 |
| *** rhallisey has quit IRC | 21:03 | |
| EmilienM | bnemec: you mean I could have +A myself? | 21:05 |
| bnemec | EmilienM: Yes | 21:05 |
| EmilienM | ah ok | 21:05 |
| EmilienM | cool! | 21:05 |
| *** trown is now known as trown|outtypewww | 21:11 | |
| *** jcoufal has quit IRC | 21:12 | |
| Slower | thrash: thanks! | 21:13 |
| *** jmiu_ has quit IRC | 21:17 | |
| *** ayoung has quit IRC | 21:20 | |
| openstackgerrit | Merged openstack/diskimage-builder: Install docker for tests https://review.openstack.org/278021 | 21:23 |
| *** abehl has quit IRC | 21:26 | |
| *** derekh has joined #tripleo | 21:27 | |
| derekh | bnemec: your off the hook, I'll clean the patch up tomorrow and take it out of WIP http://logs.openstack.org/87/334787/37/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha/f24d8a4/console.html#_2016-06-29_21_24_25_567498 | 21:27 |
| bnemec | derekh: Nice! | 21:28 |
| bnemec | Just as well, I'm still stuck in the controller node replacement rat hole. :-/ | 21:29 |
| derekh | bnemec: hope your not stuck in there for much longer, I'm gonna call it a night | 21:31 |
| derekh | ttyl | 21:31 |
| bnemec | o/ | 21:31 |
| *** derekh has quit IRC | 21:31 | |
| openstackgerrit | Merged openstack/instack-undercloud: Use Openstack CLI for setting undercloud quota https://review.openstack.org/333150 | 21:31 |
| *** fpan has joined #tripleo | 21:34 | |
| sshnaidm | EmilienM, tested your solution, it works. https://paste.fedoraproject.org/386492/14672361/ | 21:36 |
| EmilienM | sshnaidm: cool! | 21:36 |
| EmilienM | thx | 21:36 |
| sshnaidm | EmilienM, I think you can submit | 21:36 |
| EmilienM | we'll wait for CI | 21:36 |
| EmilienM | and I'll let someone approving it. | 21:37 |
| sshnaidm | it's weird how did it work till now | 21:37 |
| sshnaidm | I thought there were some default dependencies in heat.. | 21:38 |
| openstackgerrit | Merged openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 21:38 |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/335169 | 21:38 |
| *** shivrao has quit IRC | 21:43 | |
| *** shivrao has joined #tripleo | 21:44 | |
| openstackgerrit | Merged openstack/puppet-pacemaker: CI: specinfra, xenial, lint, puppet bug. https://review.openstack.org/332836 | 21:46 |
| *** mbound has joined #tripleo | 21:51 | |
| openstackgerrit | Mark Hamzy proposed openstack/diskimage-builder: dmidecode does not exist for ppc64/ppc64el https://review.openstack.org/333932 | 21:52 |
| *** mbound_ has joined #tripleo | 21:53 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates: Dump and restore galera db during major upgrades https://review.openstack.org/325205 | 21:54 |
| *** [2]cdearborn has joined #tripleo | 21:54 | |
| *** mbound has quit IRC | 21:57 | |
| *** mbound_ has quit IRC | 22:06 | |
| *** mbound has joined #tripleo | 22:06 | |
| *** [1]cdearborn has quit IRC | 22:06 | |
| *** dmk0202 has quit IRC | 22:08 | |
| *** mbound has quit IRC | 22:08 | |
| *** mbound has joined #tripleo | 22:08 | |
| *** hewbrocca is now known as hewbrocca-afk | 22:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/332383 | 22:13 |
| *** egafford has quit IRC | 22:13 | |
| *** chlong has joined #tripleo | 22:17 | |
| *** bfournie has quit IRC | 22:23 | |
| *** ayoung has joined #tripleo | 22:27 | |
| *** [2]cdearborn has quit IRC | 22:29 | |
| *** rlandy has quit IRC | 22:30 | |
| *** stevebak` has joined #tripleo | 22:31 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 22:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 22:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML https://review.openstack.org/334786 | 22:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Making element overriding explicit https://review.openstack.org/334785 | 22:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308 | 22:35 |
| *** limao has joined #tripleo | 22:40 | |
| *** thrash is now known as thrash|g0ne | 22:43 | |
| *** sanjayu has joined #tripleo | 22:54 | |
| *** weshay has quit IRC | 22:58 | |
| *** limao has quit IRC | 23:05 | |
| *** sanjayu has quit IRC | 23:05 | |
| *** stevebak` is now known as stevebaker | 23:19 | |
| *** slagle has joined #tripleo | 23:33 | |
| *** panda has quit IRC | 23:34 | |
| *** panda has joined #tripleo | 23:35 | |
| *** mbound has quit IRC | 23:48 | |
| *** fpan has quit IRC | 23:52 | |
| *** adarazs has joined #tripleo | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!