| *** jaimguer_ has joined #tripleo | 00:02 | |
| *** jaimguer_ has quit IRC | 00:03 | |
| thrash | EmilienM: Ok. | 00:03 |
|---|---|---|
| thrash | EmilienM: tomorrow. :) | 00:04 |
| *** thrash is now known as thrash|g0ne | 00:04 | |
| *** limao has joined #tripleo | 00:12 | |
| openstackgerrit | Merged openstack/os-apply-config: Updated from global requirements https://review.openstack.org/332332 | 00:32 |
| *** shivrao has quit IRC | 00:38 | |
| *** Goneri has joined #tripleo | 00:53 | |
| EmilienM | thrash|g0ne: sure! no hurry at all, just a remark | 01:01 |
| *** dmacpher has joined #tripleo | 01:04 | |
| *** shivrao has joined #tripleo | 01:04 | |
| *** Goneri has quit IRC | 01:04 | |
| *** saneax is now known as saneax_AFK | 01:06 | |
| *** Goneri has joined #tripleo | 01:09 | |
| *** shivrao has quit IRC | 01:20 | |
| *** limao has quit IRC | 01:24 | |
| *** limao has joined #tripleo | 01:24 | |
| *** rajinir has quit IRC | 01:24 | |
| *** fzdarsky_ has joined #tripleo | 01:31 | |
| *** fzdarsky has quit IRC | 01:35 | |
| *** Goneri has quit IRC | 01:37 | |
| *** weshay has quit IRC | 01:38 | |
| *** chlong has quit IRC | 01:40 | |
| *** shivrao has joined #tripleo | 01:52 | |
| *** chlong has joined #tripleo | 01:53 | |
| *** toure has quit IRC | 02:05 | |
| *** rhallisey has quit IRC | 02:21 | |
| *** amoralej|pto has quit IRC | 02:33 | |
| *** amoralej has joined #tripleo | 02:33 | |
| *** mburned is now known as mburned_out | 02:37 | |
| *** hanchao has joined #tripleo | 02:45 | |
| *** julim has quit IRC | 03:21 | |
| *** ayoung has quit IRC | 03:30 | |
| *** ramishra has quit IRC | 04:01 | |
| *** ramishra has joined #tripleo | 04:02 | |
| *** shivrao has quit IRC | 04:10 | |
| *** mburned_out has quit IRC | 04:27 | |
| *** shivrao has joined #tripleo | 04:28 | |
| *** liverpooler has quit IRC | 04:34 | |
| *** links has joined #tripleo | 04:48 | |
| *** limao has quit IRC | 05:08 | |
| *** limao has joined #tripleo | 05:09 | |
| *** dtrainor has quit IRC | 05:23 | |
| *** dtrainor has joined #tripleo | 05:28 | |
| *** saneax_AFK is now known as saneax | 05:45 | |
| *** oshvartz has joined #tripleo | 05:47 | |
| *** numans has quit IRC | 05:48 | |
| *** numans has joined #tripleo | 05:49 | |
| *** tbonds has quit IRC | 05:53 | |
| *** apetrich has quit IRC | 05:54 | |
| *** apetrich has joined #tripleo | 05:54 | |
| hewbrocca | First day with only OVB CI.... what will it bring?? | 05:55 |
| *** dtrainor has quit IRC | 05:56 | |
| *** pcaruana has joined #tripleo | 06:05 | |
| *** fragatina has joined #tripleo | 06:06 | |
| *** fragatina has quit IRC | 06:06 | |
| *** fragatina has joined #tripleo | 06:07 | |
| *** limao has quit IRC | 06:08 | |
| *** limao_ has joined #tripleo | 06:08 | |
| *** dtrainor has joined #tripleo | 06:08 | |
| *** rcernin has joined #tripleo | 06:08 | |
| *** liverpooler has joined #tripleo | 06:21 | |
| *** limao has joined #tripleo | 06:23 | |
| *** limao_ has quit IRC | 06:26 | |
| -openstackstatus- NOTICE: All python 3.5 jobs are failing today, we need to build new xenial images first. | 06:29 | |
| *** liverpooler has quit IRC | 06:39 | |
| *** liverpooler has joined #tripleo | 06:39 | |
| *** tbonds has joined #tripleo | 06:41 | |
| *** tremble has joined #tripleo | 06:45 | |
| *** tremble has joined #tripleo | 06:45 | |
| *** dmacpher has quit IRC | 06:54 | |
| *** ccamacho has joined #tripleo | 07:03 | |
| *** athomas has joined #tripleo | 07:05 | |
| *** florianf has joined #tripleo | 07:11 | |
| *** shivrao has quit IRC | 07:12 | |
| *** jpena|off is now known as jpena | 07:13 | |
| ccamacho | Happy Wednesday all! Good morning! | 07:17 |
| d0ugal | Morning | 07:20 |
| *** devvesa has joined #tripleo | 07:20 | |
| *** gfidente has joined #tripleo | 07:21 | |
| *** gfidente has quit IRC | 07:21 | |
| *** gfidente has joined #tripleo | 07:21 | |
| *** tesseract- has joined #tripleo | 07:21 | |
| *** shardy has joined #tripleo | 07:26 | |
| openstackgerrit | Xiang Chen proposed openstack/diskimage-builder: Give a more clear definition abount vm element in README https://review.openstack.org/338060 | 07:29 |
| *** yolanda has joined #tripleo | 07:30 | |
| *** rain has joined #tripleo | 07:31 | |
| *** rain is now known as leanderthal | 07:31 | |
| *** akuznetsov has joined #tripleo | 07:31 | |
| *** yolanda has quit IRC | 07:38 | |
| *** yolanda has joined #tripleo | 07:38 | |
| *** ifarkas has joined #tripleo | 07:39 | |
| hewbrocca | How's the CI looking folks? | 07:41 |
| *** jpich has joined #tripleo | 07:41 | |
| hewbrocca | I'm curious how our 15-machine OVB cloud is holding up to the load | 07:41 |
| *** yolanda has quit IRC | 07:43 | |
| shardy | http://tripleo.org/cistatus.html doesn't look too bad | 07:44 |
| shardy | So it looks like it's holding up pretty well :) | 07:44 |
| hewbrocca | Excellent | 07:45 |
| hewbrocca | Green jobs are passing, I guess? | 07:45 |
| shardy | yup | 07:45 |
| hewbrocca | Of course we've lost our upgrade job and a couple of others until we get the rack back | 07:45 |
| hewbrocca | but still this is good | 07:45 |
| shardy | I've not looked at the failing jobs yet to see if they're real or issues with the job/infra | 07:45 |
| shardy | Yeah. we'll have to be careful of merging some patches, anything that touches upgrades or network-isolation in particular | 07:46 |
| hewbrocca | Oh, right, no net-iso job either :( | 07:46 |
| hewbrocca | Is it possible to test net-iso on OVB at all? | 07:47 |
| hewbrocca | I guess it is with no bonding | 07:47 |
| shardy | I think it will be, but not right now due to the network/nic setup | 07:47 |
| *** yolanda has joined #tripleo | 07:49 | |
| gfidente | we could add ceph though | 07:57 |
| *** numans has quit IRC | 07:57 | |
| gfidente | useful to test the rgw submissions | 07:58 |
| *** yolanda has quit IRC | 07:59 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add Ceph to the OVB job https://review.openstack.org/338088 | 08:05 |
| *** dsariel has joined #tripleo | 08:06 | |
| *** akuznetsov has quit IRC | 08:07 | |
| hewbrocca | gfidente cool, yes indeed | 08:09 |
| gfidente | it takes one more node though | 08:10 |
| hewbrocca | with OVB I don't think that matters | 08:10 |
| hewbrocca | Derek had an 80-node test job on there at one point | 08:11 |
| hewbrocca | on 15 physical hosts | 08:11 |
| hewbrocca | Cloud, how it works? | 08:11 |
| gfidente | :) | 08:11 |
| gfidente | yeah though I think it might lower the number of jobs we can run in parallel | 08:12 |
| hewbrocca | true | 08:12 |
| gfidente | because it'll just be one more node for all submissions | 08:12 |
| gfidente | shardy, ^^ about the above, I figured we can filter which jobs are executed matching the files changed in a submission | 08:13 |
| *** chem has joined #tripleo | 08:13 | |
| gfidente | but it wasn't easy for me to find any set of files which would be sufficient to test on a subset of the jobs | 08:14 |
| gfidente | oh actually, we don't need one more node | 08:15 |
| gfidente | we have roles | 08:15 |
| gfidente | we can deploy osd on the compute node! | 08:15 |
| * gfidente looks around him suspiciously | 08:16 | |
| gfidente | leaving a comment on the review, to see what other people think | 08:17 |
| hewbrocca | LOL | 08:20 |
| *** derekh has joined #tripleo | 08:24 | |
| *** lucas|afk is now known as lucasagomes | 08:25 | |
| shardy | gfidente: actually, there are some folks asking for co-located compute and OSD, so testing that (even locally) would be a very good thing! :) | 08:34 |
| gfidente | shardy, yeah | 08:34 |
| gfidente | so should be simple, I add an environment and call it from toci | 08:35 |
| gfidente | trying | 08:35 |
| shardy | I actually got access to the 80 node OVB environment, it was very useful from a scale testing standpoint | 08:35 |
| shardy | https://etherpad.openstack.org/p/tripleo-ci-performance-notes | 08:35 |
| shardy | Note the event-list took over 6 *minutes* | 08:35 |
| shardy | I then applied a heat patch from stevebaker and it went down by a factor of 12 :)) | 08:36 |
| gfidente | yeah | 08:36 |
| gfidente | so curiosity, what was cpu/memory on the undercloud? | 08:36 |
| shardy | Hmm, I don't actually recall, sorry, derekh probably knows | 08:37 |
| * shardy should have run sar or something to collect more data | 08:38 | |
| gfidente | oh I meant how many cores and how much memory | 08:38 |
| gfidente | not the load | 08:38 |
| shardy | Ideally I'd like to do regular testing at that scale, then we can more clearly see the impact of fixes over time | 08:38 |
| shardy | gfidente: Yeah, sorry, I didn't record that, probably should have | 08:39 |
| derekh | shardy: gfidente 16G RAM, 8vCPU | 08:39 |
| gfidente | ah ok I thought you mentioned sar to collect load | 08:39 |
| gfidente | derekh, ack | 08:39 |
| derekh | I tried it first with 2vCPU, that got nowhere | 08:39 |
| gfidente | derekh, weren't we fixing workers on undercloud? | 08:40 |
| gfidente | number of workers | 08:40 |
| gfidente | not for the scale test I suppose | 08:40 |
| shardy | derekh: do you still run your home lab with OVB on packstack? | 08:40 |
| derekh | gfidente: for ci we are, this isn't ci | 08:40 |
| derekh | shardy: it still exists, but I havn't turned it on in some time since I've been testing rh2, so I've just been using that | 08:41 |
| shardy | cool - I've got three local boxes and would like to make a small OVB cloud - I tried a while ago using TripleO but found one of the boxes was UEFI and I couldn't get ironic to boot it (that may be fixed now) | 08:42 |
| derekh | gfidente: actually maybe we're just pinning the number of workers on the overcloud | 08:43 |
| *** akrivoka has joined #tripleo | 08:43 | |
| derekh | shardy: iirc, I went through a normal packstack install and then edited the various config options listed in the OVB readme | 08:44 |
| openstackgerrit | Merged openstack/puppet-tripleo: nova: do not manage nova-compute with pacemaker https://review.openstack.org/336753 | 08:45 |
| derekh | shardy: you'll have a choice of using an unpatched nova-compute with a PXE image | 08:45 |
| derekh | shardy: or patch nova-compute | 08:45 |
| derekh | shardy: I'd personaly go with the patch option, I'm thinking of switching to this on rh2 also, will talk to bnemec about it later | 08:46 |
| derekh | shardy: if you go with the pxe image option, you have to redeploy nodes on your cloud after each time you used them, as the pxe image has been overwritten | 08:47 |
| shardy | derekh: cool, thanks for the info :) | 08:49 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Modify ComputeServices to include CephOSD in puppet-ceph-devel env https://review.openstack.org/338113 | 08:52 |
| *** ramishra has quit IRC | 08:52 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add Ceph to the OVB job https://review.openstack.org/338088 | 08:53 |
| *** ramishra has joined #tripleo | 08:54 | |
| *** oneswig has joined #tripleo | 08:56 | |
| *** electrofelix has joined #tripleo | 09:00 | |
| *** yolanda has joined #tripleo | 09:04 | |
| ccamacho | Hey guys, anyone available for some love review? https://review.openstack.org/#/c/318413/ already +2 and passing all CI gates :) | 09:10 |
| *** ebarrera has joined #tripleo | 09:14 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:15 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: Minor updates might fail with missing MysqlClustercheckPassword property https://review.openstack.org/337304 | 09:16 |
| gfidente | shardy, derekh I posted the changes to test osd on compute, let's see how it goes | 09:23 |
| gfidente | I also wanted to ask if we can use the 80nodes environment to gather some data for https://bugzilla.redhat.com/show_bug.cgi?id=1313479 ? | 09:23 |
| openstack | bugzilla.redhat.com bug 1313479 in rhel-osp-director "[Heat] NodeUserData cannot scale beyond 3 nodes" [Urgent,New] - Assigned to athomas | 09:23 |
| gfidente | I don't think we have a launchpad for this, but seems valid | 09:24 |
| *** ebarrera has quit IRC | 09:28 | |
| *** numans has joined #tripleo | 09:30 | |
| *** sambetts|afk is now known as sambetts | 09:30 | |
| *** mgould|afk is now known as mgould | 09:38 | |
| sshnaidm | shardy, FYI, you have now successful periodic jobs listed here: http://status-tripleoci.rhcloud.com/ | 09:47 |
| *** athomas has quit IRC | 09:48 | |
| shardy | gfidente: To make OSD on compute work with net-iso, do we need to wire in the StorageMgmtPort, or is just the StoragePort enough? | 09:52 |
| gfidente | shardy, ah good point | 09:52 |
| shardy | gfidente: re the 80node environment, it's been deleted to make space for our CI I believe | 09:52 |
| shardy | hopefully we can get access to a similar setup again in future tho | 09:53 |
| shardy | sshnaidm: great, thanks! | 09:53 |
| gfidente | shardy, so regarding the networks, we want storage and storagemgmt on computes yes | 09:54 |
| gfidente | the ports are wired into the templates, I think we're just nooping them in registry | 09:54 |
| shardy | gfidente: yeah, we noop the StorageMgmt one in network-isolation.yaml | 09:54 |
| shardy | I guess we can change that in a ceph specific template | 09:55 |
| gfidente | but the cluster works with single network too | 09:55 |
| gfidente | ack we can change it in env file | 09:55 |
| shardy | cool, will be interesting to see how this works :) | 09:55 |
| *** athomas has joined #tripleo | 09:55 | |
| gfidente | well let me update puppet-ceph-devel to add storagemgmt port first ok? | 09:55 |
| gfidente | storagemgmt addresses default to ctlplane when nooped | 09:56 |
| gfidente | and it uses storage network for clients | 09:56 |
| gfidente | s/how/if/ :) | 09:56 |
| *** numans has quit IRC | 09:57 | |
| *** links has quit IRC | 10:00 | |
| *** oneswig has quit IRC | 10:06 | |
| *** oneswig has joined #tripleo | 10:08 | |
| *** limao has quit IRC | 10:09 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:18 | |
| *** thrash|g0ne is now known as thrash | 10:18 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Modify ComputeServices to include CephOSD in puppet-ceph-devel env https://review.openstack.org/338113 | 10:18 |
| jpich | Is http://status-tripleoci.rhcloud.com using bug signatures to identify failures, kinda like elastic-recheck? Where are they stored? | 10:18 |
| *** panda|Zz is now known as panda | 10:25 | |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Split Sahara pacemaker roles into separate services https://review.openstack.org/327721 | 10:26 |
| shardy | sshnaidm|afk: ^^ perhaps you can answer jpich's question, I'm also interested in the answer :) | 10:27 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Split Sahara pacemaker roles into separate services https://review.openstack.org/327721 | 10:27 |
| sshnaidm|afk | jpich, shardy yes, they're now WIP in repo https://github.com/sshnaidm/sova , patterns themselves are in https://github.com/sshnaidm/sova/blob/master/tripleoci/patterns.py | 10:28 |
| sshnaidm|afk | jpich, shardy when I'll feel it's ready maybe I'll fill a blueprint to include it in TripleO CI officially, but for now it's completely useful | 10:29 |
| shardy | sshnaidm|afk: it is useful - I think figuring out how to wire this in to tripleo-ci (and the status report on tripleo.org) is a good idea | 10:30 |
| jpich | sshnaidm|afk: Thanks! | 10:30 |
| shardy | it'd be nice to consider the potential overlap with tools like elastic-recheck tho | 10:31 |
| thrash | d0ugal: can you look at https://review.openstack.org/#/c/336642/? | 10:32 |
| jpich | Since our logs are stored in the same place maybe there's bits of the e-r infrastructure we can reuse | 10:32 |
| sshnaidm|afk | shardy, in best case I'd use my own elastic-recheck setup, the openstack's one can only print a hardcoded bug attention. I worked a little on it (https://review.openstack.org/#/c/312985/), but haven't found too much practical usage | 10:34 |
| d0ugal | thrash: sure | 10:34 |
| sshnaidm|afk | shardy, but involving elastic-recheck is definitely in roadmap of this effort.. | 10:35 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Adds parameters actions https://review.openstack.org/298682 | 10:36 |
| d0ugal | thrash: LGTM | 10:36 |
| shardy | sshnaidm|afk: ack, sounds good, thanks :) | 10:36 |
| thrash | d0ugal: awesome. Now go brow-beat another core to +2 it. :P | 10:36 |
| derekh | sshnaidm|afk: are you going afk? or coming back? | 10:36 |
| thrash | derekh: j/k | 10:36 |
| thrash | :D | 10:36 |
| openstackgerrit | Merged openstack/tripleo-docs: Composable services within roles Tutorial https://review.openstack.org/311512 | 10:37 |
| d0ugal | thrash: lol | 10:37 |
| sshnaidm|afk | derekh, already running for a lunch :) | 10:37 |
| sshnaidm|afk | derekh, brb | 10:37 |
| derekh | sshnaidm|afk: ok, ping me when your back, I'm ready for you to be the guinne pig to try out rh2 | 10:37 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Split Sahara pacemaker roles into separate services https://review.openstack.org/327722 | 10:41 |
| *** dmacpher has joined #tripleo | 10:50 | |
| *** dsariel has quit IRC | 10:55 | |
| *** coolsvap has joined #tripleo | 11:05 | |
| jpich | sshnaidm|afk: Thank you for the links, they were helpful | 11:14 |
| *** weshay has joined #tripleo | 11:14 | |
| weshay | sshnaidm|afk, let me know if this looks the same.. | 11:15 |
| weshay | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-ha-351/undercloud/home/stack/undercloud_install.log.gz | 11:15 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1582651 | 11:15 |
| openstack | Launchpad bug 1582651 in tripleo "Mistral db-sync failure in CI jobs" [High,Fix released] - Assigned to Steven Hardy (shardy) | 11:15 |
| *** lucasagomes is now known as lucas|hungry | 11:17 | |
| thrash | shardy: it's likely that the sahara templates need more work. I may end up rebasing your change on top of another fix. | 11:18 |
| *** sshnaidm|afk is now known as sshnaidm | 11:19 | |
| *** links has joined #tripleo | 11:24 | |
| sshnaidm | weshay, the effect is the same, but reason is different, anyway it should be solved in new versions of mistral as I see in the code | 11:25 |
| sshnaidm | derekh, hi, I'm here now | 11:25 |
| weshay | sshnaidm, which new code are you looking at? | 11:26 |
| sshnaidm | weshay, https://github.com/openstack/mistral/blob/master/mistral/actions/openstack/actions.py | 11:26 |
| EmilienM | hello | 11:27 |
| sshnaidm | weshay, afaiu the import errors should be solved by importutils.try_import | 11:27 |
| ansiwen | EmilienM: hi, can you explain me more or give me pointers regarding the beaker tests you were talking about in my ec2api change? | 11:28 |
| weshay | sshnaidm, k.. I see the diff.. thanks! | 11:28 |
| EmilienM | sshnaidm: fyihttps://review.openstack.org/#/c/337967/1 | 11:28 |
| *** dtrainor has quit IRC | 11:28 | |
| EmilienM | sshnaidm: tempest broke puppet CI (we fixed it by promoting to latest OpenStack trunk) | 11:28 |
| EmilienM | sshnaidm: but tripleo CI won't pass tempest tests until next promotion to trunk | 11:29 |
| EmilienM | sshnaidm: see context in the patch | 11:29 |
| sshnaidm | EmilienM, hi, thanks, will look at it | 11:29 |
| EmilienM | ansiwen: puppet-ec2api has 0 functional test that deploys ec2 api service | 11:29 |
| EmilienM | sshnaidm: there is nothing to do | 11:29 |
| EmilienM | ansiwen: look nova https://github.com/openstack/puppet-nova/blob/master/spec/acceptance/nova_wsgi_apache_spec.rb | 11:30 |
| sshnaidm | EmilienM, although right now tempest is broken in tripleoci anyway, I've replied to your comment there | 11:30 |
| thrash | EmilienM: so, just no pacemaker profile in puppet-tripleo, and point to the same template for both in tht? | 11:30 |
| EmilienM | ansiwen: look ec2api https://github.com/openstack/puppet-ec2api/blob/master/spec/acceptance/basic_ec2api_spec.rb | 11:30 |
| EmilienM | sshnaidm: ok | 11:30 |
| EmilienM | thrash: yes? | 11:30 |
| thrash | EmilienM: re: removal of pacemaker from zaqar and mistral | 11:31 |
| ansiwen | EmilienM: ok, I will read. But how is it releated to beaker? | 11:31 |
| thrash | EmilienM: sorry for the lack of context. :) | 11:31 |
| EmilienM | thrash: yes I know | 11:31 |
| derekh | sshnaidm: ok, wanna try the instructions I put on the etherpad ? | 11:31 |
| EmilienM | thrash: I'm just wondering if it follows pacemaker lite approach but I think so | 11:31 |
| sshnaidm | derekh, yeah, let's do it | 11:31 |
| EmilienM | ansiwen: https://github.com/puppetlabs/beaker | 11:31 |
| EmilienM | ansiwen: not beaker redhat | 11:31 |
| EmilienM | thrash: go for it | 11:32 |
| sshnaidm | derekh, where should I connect to first? | 11:32 |
| ansiwen | EmilienM: oh, thanks... that was actually my misunderstanding :-) | 11:32 |
| derekh | sshnaidm: ok, I've email you some credentials, the commands on the etherpad with local> can be run on your local machine (assuming it has novaclient installed) | 11:33 |
| *** dsariel has joined #tripleo | 11:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix tempest configuration https://review.openstack.org/331997 | 11:33 |
| sshnaidm | derekh, ok, got it | 11:33 |
| thrash | EmilienM: sounds good | 11:34 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add zaqar profiles https://review.openstack.org/331681 | 11:35 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Adds action for template processing https://review.openstack.org/337615 | 11:36 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Zaqar services https://review.openstack.org/331682 | 11:37 |
| sshnaidm | derekh, the new keypair should be combined from existing one (yours) and my keys, right? | 11:38 |
| derekh | sshnaidm: correct, whatever you add there will just be put into authorized_keys | 11:39 |
| *** rodrigods has quit IRC | 11:41 | |
| *** rodrigods has joined #tripleo | 11:41 | |
| *** bfournie has quit IRC | 11:44 | |
| *** saneax is now known as saneax_AFK | 11:46 | |
| *** rook-lappy has joined #tripleo | 11:52 | |
| thrash | shardy: I'm just going to put my fixes in your review. | 11:53 |
| thrash | otherwise, I think we'd be in a chicken-egg situation. | 11:54 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Update overcloud log retrieval of rh2 https://review.openstack.org/338209 | 11:55 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Split Sahara pacemaker roles into separate services https://review.openstack.org/327722 | 11:57 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Sahara services to ControllerServices list https://review.openstack.org/336119 | 11:57 |
| *** hrybacki|afk is now known as hrybacki | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 12:01 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove Fedora support from CI scripts https://review.openstack.org/338210 | 12:03 |
| *** weshay is now known as weshay_mtg | 12:04 | |
| derekh | sshnaidm: going for lunch, back in a bit | 12:04 |
| sshnaidm | derekh, ok, I'm in middle of script.. | 12:04 |
| *** oneswig has quit IRC | 12:04 | |
| derekh | sshnaidm: ok | 12:05 |
| *** oneswig has joined #tripleo | 12:06 | |
| *** dprince has joined #tripleo | 12:08 | |
| *** oneswig has quit IRC | 12:10 | |
| *** oneswig has joined #tripleo | 12:10 | |
| *** leifmadsen_ is now known as leifmadsen | 12:12 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Update overcloud passwords on deploy command https://review.openstack.org/338213 | 12:13 |
| *** fragatina has quit IRC | 12:13 | |
| gfidente | therve, d0ugal, ^^ assuming it'll need unit tests fixing, would you check if the above looks valid? | 12:13 |
| gfidente | ah sorry I meant to ping thrash | 12:14 |
| thrash | gfidente: ack | 12:14 |
| thrash | will look in a sec | 12:14 |
| *** fragatina has joined #tripleo | 12:14 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: WIP: Update overcloud passwords on update command https://review.openstack.org/338213 | 12:14 |
| gfidente | thanks | 12:14 |
| *** dprince has quit IRC | 12:16 | |
| *** dprince has joined #tripleo | 12:16 | |
| shardy | thrash: sorry been at lunch, that's fine, pls feel free to push whatever is needed to my review | 12:16 |
| *** rhallisey has joined #tripleo | 12:16 | |
| *** _xou_ has joined #tripleo | 12:17 | |
| thrash | shardy: already done. :) | 12:18 |
| _xou_ | hi | 12:19 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Adds parameters actions https://review.openstack.org/298682 | 12:20 |
| *** lucas|hungry is now known as lucasagomes | 12:21 | |
| dprince | rbrady, jtomasek: good morning. Just wanted to point out this one from yesterday https://review.openstack.org/#/c/337837/ | 12:21 |
| dprince | rbrady, jtomasek: basically I hope that is the last time we see instack break due to a new python client dependency getting added to Mistral | 12:21 |
| rbrady | dprince: ack | 12:22 |
| *** jpena is now known as jpena|lunch | 12:23 | |
| *** jayg|g0n3 is now known as jayg | 12:23 | |
| *** ccamacho has quit IRC | 12:24 | |
| *** bfournie has joined #tripleo | 12:24 | |
| *** ccamacho has joined #tripleo | 12:25 | |
| snecklifter | Morning folks, would it be possible for someone to review https://review.openstack.org/#/c/331680/ please? | 12:27 |
| gfidente | shardy, hey the ceph osd role works! | 12:28 |
| gfidente | updating the submissions | 12:28 |
| snecklifter | Its a Mitaka backport and the upstream patch has got stuck on the big composable roles changes | 12:28 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Modify ComputeServices to include CephOSD in puppet-ceph-devel env https://review.openstack.org/338113 | 12:28 |
| yolanda | hi trown, i'm testing oooq deployment in another server and works fine for me. But i have a problem with swift failing as i told you yesterday, on a single server. I actually can see errors with http://192.0.2.1:8080 , 503 service unavailable | 12:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/puppet-tripleo: Add Mistral profiles https://review.openstack.org/323431 | 12:34 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services https://review.openstack.org/323436 | 12:34 |
| hewbrocca | gfidente woop woop! | 12:34 |
| shardy | EmilienM: Hey, when you get a moment can you pls check https://review.openstack.org/#/c/260226/ | 12:35 |
| shardy | it's the master patch referred to by snecklifter, and IMO we'd be better to just land it (it's passing CI) so the backport to mitaka will be clean | 12:36 |
| *** rook-lappy has quit IRC | 12:36 | |
| snecklifter | shardy: yes, thanks | 12:36 |
| shardy | If it had just been posted, I'd be more inclined to say block it until composable services lands, but it's been there since december last year :( | 12:37 |
| shardy | snecklifter: thanks for highlighting it | 12:37 |
| snecklifter | shardy: no prob, wasn't sure how much of an interest folks take in backports | 12:37 |
| snecklifter | but as it took me a while to track down the fix I guess it will be affecting others | 12:38 |
| shardy | gfidente: nice! | 12:38 |
| *** ramishra has quit IRC | 12:43 | |
| *** rlandy has joined #tripleo | 12:43 | |
| shardy | gfidente: Hey, FYI I was thinking more about ControllerEnableCephStorage/ControllerEnableSwiftStorage | 12:44 |
| *** ramishra has joined #tripleo | 12:44 | |
| shardy | gfidente: despite me arguing we should include them, I actually don't think they will work, because we've no longer got any way to pass a Controller specific value to that parameter | 12:44 |
| shardy | e.g parameter_defaults will affect both the Controller and the *Storage nodes, because they use the same service template | 12:45 |
| shardy | I'm thinking of ways we can work around that | 12:45 |
| shardy | possibly we can process the ControllerServices list with yaql, so we exclude certain services if e.g CephStorageCount is non-zero | 12:46 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Split Sahara pacemaker roles into separate services https://review.openstack.org/327722 | 12:48 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Sahara services to ControllerServices list https://review.openstack.org/336119 | 12:48 |
| *** fultonj has joined #tripleo | 12:48 | |
| *** pradk has joined #tripleo | 12:49 | |
| *** myoung has quit IRC | 12:50 | |
| *** myoung has joined #tripleo | 12:51 | |
| *** lblanchard has joined #tripleo | 12:59 | |
| *** Goneri has joined #tripleo | 13:01 | |
| *** weshay_mtg is now known as weshay | 13:04 | |
| ccamacho | Guys quick question is anyone getting this message when deploying master? http://paste.openstack.org/show/526552/ heat stack-list shows a CREATE_COMPLETE but that last message is, I think wrong.. | 13:04 |
| _xou_ | nop ccamacho | 13:07 |
| derekh | sshnaidm: hows it going? | 13:07 |
| *** trown|outtypewww is now known as trown | 13:07 | |
| *** jtomasek_ has joined #tripleo | 13:07 | |
| ccamacho | _xou_ ack, Ill double check it. | 13:08 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 13:08 |
| slagle | shardy: hi, do you have any ideas about how I might be able to seed the credentials on the deployed servers for https://review.openstack.org/#/c/222772/ ? | 13:09 |
| slagle | shardy: right now, my script just copies over the admin credentials, which is obviously not ideal | 13:09 |
| sshnaidm | derekh, fine, installed undercloud, registering nodes | 13:09 |
| derekh | sshnaidm: cool | 13:09 |
| derekh | slagle: dprince bnemec so I'm trying to set up instructions on rh2 so that people can try a replicate CI https://etherpad.openstack.org/p/tripleo-ci-devenvs | 13:10 |
| pradk | EmilienM, i kept talking guess you guys couldnt hear me in the mtg | 13:11 |
| pradk | :) | 13:11 |
| EmilienM | nope :) | 13:11 |
| derekh | slagle: dprince bnemec I've added a user "tripleo-user", with enough quota for 3 HA ci tests | 13:11 |
| shardy | slagle: Hi, have you tried deploying the configuration stack first, then looking at heat resource-metadata for the resources that own the deployed-server.yaml nested stacks? | 13:11 |
| pradk | EmilienM, so yea gnocchi i fixed the db sync issues you pointed out.. should be in good shape | 13:11 |
| *** jcoufal has joined #tripleo | 13:11 | |
| EmilienM | pradk: ok, i'll test & review today | 13:11 |
| pradk | EmilienM, aodh the profiles are looking good and passing ci | 13:11 |
| shardy | slagle: there should be credentials for each server (or, actually, a swift tempurl in recent tripleo) there, which your script can grab? | 13:11 |
| derekh | slagle: dprince bnemec , sshnaidm is going through the instructions at the moment, and I'll send ye the credentials now | 13:11 |
| pradk | EmilienM, the tht changes are in merge conflict i'll resolve that in a bit | 13:12 |
| slagle | shardy: ok, let me check that. i see the users created in the heat_stack domain in keystone | 13:12 |
| slagle | i didnt realize i could query heat for them as well | 13:12 |
| shardy | slagle: or, rather, it's the metadata pushed to deployed-server inside deployed-server.yaml | 13:12 |
| shardy | slagle: Yeah you should be able to, see line 113 in https://etherpad.openstack.org/p/noop-softwareconfig | 13:14 |
| shardy | you can get the exact same data normally read via o-c-c on the heat CLI | 13:14 |
| *** amoralej|lunch is now known as amoralej | 13:14 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add hyperconverged-ceph environment to include CephOSD on computes https://review.openstack.org/338113 | 13:14 |
| derekh | bnemec: About https://bugs.launchpad.net/tripleo/+bug/1599299 , I've updated the bmc image we use to include that patch | 13:17 |
| openstack | Launchpad bug 1599299 in tripleo "openstackbmc still failing with concatenation errors" [High,Triaged] | 13:17 |
| derekh | bnemec: but I'm also wondering if we should just apply the nova-compute network boot patch to rh2, | 13:18 |
| shardy | slagle: Hmm, actually when using the swift transport that may only give you the URL for signalling back to heat, not collecting the metadata | 13:18 |
| shardy | slagle: so you may need to use SoftwareConfigTransport POLL_SERVER_HEAT | 13:18 |
| derekh | bnemec: mainly to make it easier to use for non CI | 13:18 |
| shardy | that will give you a username and random password for each server, which can be used to both poll and signal heat | 13:18 |
| shardy | slagle: probably we could add the poll tempurl to the metadata too inside heat, I think it's currently only added to the server user_data | 13:19 |
| derekh | bnemec: but also, I'm worried, maybe the instances get partialy deployed and then the retry's (after a timout) don't work because the PXE boot image is gone | 13:19 |
| slagle | shardy: yea all i see is the signal url | 13:20 |
| slagle | let me try setting SoftwareConfigTransport | 13:20 |
| slagle | shardy: related question, would there be any way to predefine the stack uuid of the deployed-server nested stack? | 13:21 |
| _xou_ | anyone can drive me to the right path to have CinderVolume BlockStorage to be installed on all my compute nodes ? | 13:21 |
| slagle | shardy: so i don't have to do a lot of fuzzy querying for it | 13:21 |
| shardy | slagle: Hmm, sorry SoftwareConfigTransport won't work, because that's still only the polling configuration (which you don't get in the current server metadata) | 13:22 |
| shardy | you need to change one of any SoftwareDeployment signal_transport to HEAT_SIGNAL | 13:22 |
| EmilienM | pradk: ack | 13:22 |
| slagle | ok | 13:22 |
| shardy | slagle: or you can set default_deployment_signal_transport = HEAT_SIGNAL in heat.conf | 13:23 |
| shardy | that may be easier | 13:23 |
| EmilienM | gfidente, shardy, dprince: can you guys review https://review.openstack.org/#/c/337358/ and https://review.openstack.org/#/c/337359/ please? | 13:24 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: Fix retrieval of hostname fact based on network. https://review.openstack.org/332736 | 13:26 |
| shardy | slagle: Hmm, re the ID, you might be able to make the nested stack ID predictable via OS::stack_id, but I'm not sure if the metadata polling will work if you do that | 13:26 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates: Move nova constraints to tripleo-puppet. https://review.openstack.org/332071 | 13:27 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-tripleo: Move nova constraint, and refactor its declaration https://review.openstack.org/332069 | 13:27 |
| shardy | because the metadata is really being pushed to the deployed-server-config resource | 13:27 |
| shardy | for which you can't control the UUID | 13:27 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: WIP: integrate PCS provider in the merge. https://review.openstack.org/310713 | 13:27 |
| shardy | slagle: we might be able to refine that so the metadata is pushed directly to the nested stack, but IIRC that didn't work when I last tried it | 13:28 |
| slagle | shardy: ok, i already gave that a try, didnt work :) at least what i was doing | 13:28 |
| slagle | shardy: i tried adding OS::stack_id as an output | 13:28 |
| slagle | didnt seem to do anything | 13:28 |
| shardy | It'll just redefine the get_resource output in the owning stack | 13:28 |
| *** rhallisey has quit IRC | 13:29 | |
| shardy | which will probably break the pushing of metadata to the "server" | 13:29 |
| shardy | slagle: if you need that I can take a closer look and see how we might enable it | 13:29 |
| shardy | for now a heat resource-list -n5 overcloud | grep deployed-server-config or something is probably best | 13:29 |
| slagle | shardy: i dont need it per se, i can make do with the script i have for now | 13:29 |
| shardy | slagle: cool, when I get some time I'll see if/how we might do it | 13:30 |
| slagle | ideally, i could preconfigure occ on each node before we have to start creating the stack | 13:30 |
| slagle | that's my perfect world anyway | 13:30 |
| shardy | yeah, it's a bit chicken/egg | 13:30 |
| *** rhallisey has joined #tripleo | 13:30 | |
| slagle | for now, i just start a backgroud script that starts resource-list'ing heat until the nested stacks get created | 13:30 |
| shardy | it's probably possible, will give it some thought | 13:30 |
| shardy | slagle: even if we fixed the predictable ID part, I think you'd still need to create the stack to get the credentials | 13:32 |
| shardy | heat internally creates the users that poll in a special domain/project, so that has to exist before any user/password details can be defined | 13:33 |
| shardy | (unless you just use admin or something else in the default domain with access) | 13:33 |
| shardy | even more so with swift, as we've got no way to predict the tempurls used for polling/signalling | 13:34 |
| shardy | so what you're doing sounds fine to me | 13:34 |
| slagle | ok | 13:34 |
| *** jpena|lunch is now known as jpena | 13:35 | |
| slagle | trown: any thoughts on https://bugs.launchpad.net/tripleo-quickstart/+bug/1599509 ? | 13:36 |
| openstack | Launchpad bug 1599509 in tripleo-quickstart " [Errno 13] Permission denied: '/var/cache/tripleo-quickstart'" [Undecided,New] | 13:36 |
| *** oneswig has quit IRC | 13:37 | |
| *** ayoung has joined #tripleo | 13:38 | |
| *** jeckersb_gone is now known as jeckersb | 13:39 | |
| trown | slagle: will look, localhost probably broke again... a workaround to allow running locally is to make sure you can `ssh root@127.0.0.2`, then use 127.0.0.2 instead of localhost to trick ansible | 13:39 |
| slagle | trown: ok, yea, using 127.0.0.2 looks like it's working | 13:41 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Add new nuage agent profile. https://review.openstack.org/338263 | 13:43 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Split Sahara pacemaker roles into separate services https://review.openstack.org/327722 | 13:43 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Add Sahara services to ControllerServices list https://review.openstack.org/336119 | 13:43 |
| *** oneswig has joined #tripleo | 13:43 | |
| trown | slagle: ya by doing that you are not using the ansible local connection, which is a different path than what is CI'd (though our usbkey job is supposed to approximate it) | 13:43 |
| trown | clearly it is lacking though, as that is pretty constantly breaking | 13:44 |
| *** r-mibu has quit IRC | 13:49 | |
| *** r-mibu has joined #tripleo | 13:49 | |
| sshnaidm | derekh, overcloud failed: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]: Failed to call refresh: /usr/bin/nova-manage db sync returned 1 instead of one of [0] | 13:52 |
| sshnaidm | derekh, I'll try again | 13:52 |
| derekh | sshnaidm: wait a sec, and I'll take a quick look | 13:52 |
| sshnaidm | derekh, ok | 13:52 |
| *** myoung is now known as myoung|bbiab | 13:53 | |
| *** toure has joined #tripleo | 13:53 | |
| derekh | sshnaidm: ok, its not what I thought it might be, was just checking your mtu settings | 13:54 |
| derekh | sshnaidm: all 4 nodes went active, which show the deployment/provision part is working | 13:54 |
| shardy | How does puppet resolve e.g Package[openstack-swift] ? | 13:54 |
| shardy | I'm refactoring the ringbuilder stuff and puppet says Could not find dependency Package[openstack-swift] | 13:55 |
| derekh | sshnaidm: so here is the down side, if you want to try again you have to either | 13:55 |
| EmilienM | you're missing ::swift | 13:55 |
| shardy | the package is installed, so I guess I'm missing a puppet dependency? | 13:55 |
| shardy | EmilienM: ah, thanks, will try that | 13:55 |
| EmilienM | https://github.com/openstack/puppet-swift/blob/fbc530a9d3912ccaaa00df463237d1339e88be52/manifests/init.pp#L69 | 13:55 |
| shardy | awesome, that fixed it, thanks EmilienM! :) | 13:56 |
| EmilienM | cool | 13:56 |
| derekh | sshnaidm: 1. delete the overcloud, and then to this command on all of the baremetal_sshnaidm_* nodes "nova rebuild baremetal_sshnaidm_x ipxe-boot" | 13:56 |
| derekh | sshnaidm: 2 or start again, you can't delete the overcloud and then deploy it again | 13:57 |
| *** links has quit IRC | 13:57 | |
| derekh | sshnaidm: this is because those nodes are using a PXE booting image, and once you use them that image is overwriten and replace with the overcloud | 13:57 |
| sshnaidm | derekh, ok | 13:58 |
| derekh | I'm thinking of changing that by patching the comute node on rh2 but want to run it by bnemec first | 13:59 |
| derekh | sshnaidm: ^ | 13:59 |
| sshnaidm | derekh, ok, and how could it be patched? | 14:00 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Restore the NtpServer parameter name https://review.openstack.org/338268 | 14:00 |
| dprince | ccamacho, EmilienM:: ^^^ | 14:00 |
| dprince | ccamacho: just noticed my dev environment was broken by the recent NTP composable service. We need to be careful not to rename parameters... in this case I don't think CI covers NTP | 14:01 |
| derekh | sshnaidm: its a patch on the underlying cloud your using | 14:01 |
| ccamacho | drpince, sorry for that, ack | 14:01 |
| derekh | sshnaidm: this would be it https://github.com/cybertron/openstack-virtual-baremetal/blob/master/patches/nova/nova-pxe-boot.patch | 14:02 |
| derekh | sshnaidm: if we do that it makes the baremetal nodes reusable | 14:02 |
| ccamacho | dprince, sorry for that, ack | 14:02 |
| EmilienM | dprince: nice catch | 14:02 |
| dprince | ccamacho: we were getting by without NTP in our CI before because all testenv's lived on the same host. With OVB I'd guess we might see intermittent results without NTP enabled... unless the base cloud guarantees the VMs are also in sync | 14:02 |
| sshnaidm | derekh, I see | 14:02 |
| *** julim has joined #tripleo | 14:03 | |
| derekh | dprince: clocks are all in sync on the base cloud | 14:03 |
| dprince | derekh: cool. If they get out of sync we'll know | 14:04 |
| derekh | dprince: it should be using ntp, checking now | 14:05 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 14:08 |
| *** electrofelix has quit IRC | 14:11 | |
| *** jtomasek_ has quit IRC | 14:12 | |
| derekh | dprince: I can't see any logs of it running but all the servers are < .001 seconds out, so it must be keeping in sync ... where does it log too.... | 14:13 |
| dprince | derekh: not sure actually | 14:13 |
| dprince | derekh: I think .001 is probably close enough for us :) | 14:14 |
| derekh | dprince: id imagine so | 14:14 |
| trown | yolanda: mind checking out https://review.openstack.org/337824 it moves your patch around a bit and adds the ability to use a refspec from an open review | 14:14 |
| *** fzdarsky_ is now known as fzdarsky|afk | 14:14 | |
| yolanda | trown, i reviewed, looks good to me | 14:14 |
| trown | yolanda: whoops... sorry | 14:14 |
| yolanda | np :) | 14:15 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Move THT clone to undercloud post install https://review.openstack.org/337824 | 14:16 |
| yolanda | trown, have you ever hit an issue with swift proxy giving 503 errors? same test with same oooq settings, works in one server and fails in another | 14:16 |
| yolanda | so far the error i can see is a 503 error on port 8080, but when i login to undercloud, and i do a swift stat or swift list, it works | 14:17 |
| trown | odd... | 14:17 |
| yolanda | the server that fails has plenty of free disk, and has even more memory than the one that succeeds | 14:17 |
| trown | maybe swift was not up when the 503 happened, but came up later? | 14:17 |
| yolanda | mm, i can reproduce with a swift put | 14:18 |
| yolanda | swift upload test overcloud-full.qcow2 | 14:18 |
| yolanda | Object PUT failed: http://192.0.2.1:8080/v1/AUTH_f6b06a891bf04a0790760fb6db9a8094/test/overcloud-full.qcow2 503 Service Unavailable [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently | 14:18 |
| trown | so maybe it is glance that is failing, because I think swift is using glance backend | 14:19 |
| trown | yolanda: is it causing undercloud post install to fail? | 14:19 |
| yolanda | trown, no, i did a swift upload test overcloud-full.qcow2 to valiadte | 14:19 |
| yolanda | that's command that fails, i can reproduce | 14:19 |
| yolanda | and yep, causes undercloud post install to fail | 14:19 |
| trown | k, so it is swift upload of introspection data then... someone was asking about that yesterday | 14:20 |
| chem | he, are you aware of any error with Exec[neutron-db-sync], I have a big stack strace with "Table 'agents' already exists" ? | 14:20 |
| trown | workaround is to skip introspection, but I am curious why that is happening | 14:20 |
| yolanda | trown, seems that any swift upload commands on my server fail | 14:20 |
| *** tzumainn has joined #tripleo | 14:20 | |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Plumgrid compute helper https://review.openstack.org/338284 | 14:21 |
| trown | adding "-e step_introspect=false" to quickstart invocation will workaround it | 14:21 |
| yolanda | trown, i did yesterday and doesn't work | 14:22 |
| trown | yolanda: but mind filing a tripleo-quickstart bug for it? not sure if it is tripleo-quickstart specific or a larger tripleo issue | 14:22 |
| yolanda | it's the step of openstack upload overcloud image, basically | 14:22 |
| trown | oh... that shouldnt hit swift though | 14:22 |
| yolanda | i see swift configured as backend | 14:23 |
| trown | oh | 14:23 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Opencontrail helper profile https://review.openstack.org/338287 | 14:25 |
| yolanda | trown, | 14:25 |
| yolanda | /etc/glance/glance-api.conf:stores = glance.store.filesystem.Store,glance.store.swift.Store | 14:25 |
| yolanda | /etc/glance/glance-api.conf:default_store = swift | 14:26 |
| *** oneswig has quit IRC | 14:26 | |
| d0ugal | shardy: Hey, did you get a chance to try the CLI commands with Mistral? | 14:27 |
| shardy | d0ugal: sorry, not yet, it's the next thing on my todo list today | 14:28 |
| d0ugal | shardy: no worries, just thought I'd check in :) | 14:29 |
| *** myoung|bbiab is now known as myoung | 14:30 | |
| yolanda | trown, so that's not expected oooq behaviour? swift as default backend? | 14:30 |
| yolanda | i see that swift-proxy is down on my server, and with swift-init main restart, i can see now Exception: Could not bind to 192.0.2.1:6001 after trying for 30 seconds | 14:30 |
| trown | yolanda: well oooq does not do anything different there :), it is just my ignorance on tripleo expected behavior | 14:31 |
| yolanda | trown, we are learning together then :) | 14:31 |
| trown | hmm that is something... is there something else on 6001? or is SELinux somehow in enforcing? | 14:31 |
| yolanda | selinux is disabled, i already tried | 14:32 |
| shardy | yolanda: swift is configured as the backend for glance, but we recently had to temporarily switch it back to file because of memory issues with swift-proxy | 14:32 |
| shardy | bug #1595916 | 14:32 |
| openstack | bug 1595916 in tripleo "Swift memory usage grows until it is killed" [High,In progress] https://launchpad.net/bugs/1595916 - Assigned to Derek Higgins (derekh) | 14:32 |
| shardy | https://github.com/openstack/instack-undercloud/commit/b8c5ac736733e28315364a0c9e70465b6f41166d | 14:32 |
| trown | hmm that commit should be in the latest master stable image though | 14:33 |
| trown | yolanda: could you cat your /etc/yum.repos.d/delorean.repo on the undercloud? | 14:33 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Remove superfluous 'fi' from feature-scale-deploy.sh https://review.openstack.org/338290 | 14:33 |
| yolanda | trown ok | 14:34 |
| trown | yolanda: at least it seems like known/fixed issue :) | 14:34 |
| trown | s/fixed/worked around/ | 14:35 |
| EmilienM | dprince: I did some reviews on your neutron patches... not really blockers but still | 14:35 |
| yolanda | trown http://paste.openstack.org/ | 14:35 |
| yolanda | oops | 14:35 |
| yolanda | http://paste.openstack.org/show/526566/ | 14:35 |
| trown | yolanda: ah mitaka | 14:36 |
| trown | yolanda: maybe we need to backport that instack-undercloud patch | 14:36 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 14:36 |
| *** oneswig has joined #tripleo | 14:36 | |
| yolanda | trown, so seems that there are old processes for proxy, object-server, etc... that are stale. When i start again, there is that error "cannot bind". If i kill the processes and start again, seem to work | 14:37 |
| yolanda | so memory... | 14:37 |
| trown | yolanda: you could try that server on master with "--release master" | 14:37 |
| yolanda | ok | 14:37 |
| dprince | EmilienM: I will get them, trying to finish the idea first :) | 14:37 |
| yolanda | trown, mitaka is the default release supplied in oooq right? i did not specify any setting for it | 14:38 |
| trown | yolanda: default is set to mitaka, because the aim of defaults is to provide the highest liklihood of success at any given time... though I guess that is not happening at the moment | 14:38 |
| *** yamahata has joined #tripleo | 14:39 | |
| yolanda | trown, even worse. I use the usbkey script, and that doesn't seem to support the --release param | 14:39 |
| trown | yolanda: you are using usbkey script so that running against localhost works? | 14:40 |
| yolanda | trown yes, i had issues with root script, was not working in localhost | 14:40 |
| *** oneswig has quit IRC | 14:41 | |
| yolanda | mm, it supports, but with a different command line | 14:41 |
| yolanda | usbkey/quickstart.sh virthost release | 14:42 |
| *** Goneri has quit IRC | 14:42 | |
| trown | oh interesting, so something is breaking localhost specifically in quickstart.sh, that is really helpful :) | 14:42 |
| yolanda | trown, actually i was hitting some of the reported bugs.. i guess one about permissions | 14:42 |
| yolanda | and i've been using the usbkey for a pair of weeks now | 14:42 |
| trown | yolanda: ya, slagle put up a bug this morning... I will try to look into that later... another workaround is to use 127.0.0.2 to trick ansible into not using a local connection | 14:43 |
| *** weshay is now known as weshay_afk | 14:43 | |
| EmilienM | we still don't have logs on ovb jobs | 14:43 |
| trown | yolanda: that does require being able to `ssh root@127.0.0.2` | 14:43 |
| EmilienM | is someone on it already? | 14:43 |
| EmilienM | see http://logs.openstack.org/39/337839/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/cd29526/logs/ | 14:43 |
| bnemec | derekh: If we're going to have developers on rh2 then definitely +1 on patching nova-compute | 14:43 |
| yolanda | trown yes, i did that config for 127.0.0.2 as well | 14:44 |
| EmilienM | bnemec: your patch didn't help to have logs | 14:44 |
| yolanda | usbkey was having issues with 127.0.0.1 | 14:44 |
| *** oneswig has joined #tripleo | 14:44 | |
| derekh | bnemec: ok, I'll patch it this afternoon | 14:44 |
| EmilienM | derekh: do you know why we don't have overcloud logs on OVB jobs? | 14:45 |
| derekh | bnemec: btw, I noticed some problems getting testenvs today again, looking at the rh2 heat logs, I see these | 14:46 |
| derekh | bnemec: 2016-07-06 12:38:03.434 24257 ERROR heat.engine.resource ResourceFailure: ConnectionError: resources.openstack_baremetal_servers.resources[2].resources.baremetal_server: HTTPSConnectionPool(host='ci-overcloud.rh2.tripleo.org', port=13774): Max retries exceeded with url: /v2.1/462104cced0f442d91f9a25f9b80a29a/extensions (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7a6a090>: Failed to estab | 14:46 |
| derekh | lish a new connection: [Errno -3] Temporary failure in name resolution',)) | 14:46 |
| derekh | bnemec: I've added the dns entry to /etc/hosts on the overcloud | 14:46 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Opencontrail vrouter profile https://review.openstack.org/338287 | 14:46 |
| derekh | bnemec: slagle dprince I'm tracking any tweaks I do to rh2 here while we find out all the teething problems https://etherpad.openstack.org/p/tripleo-rh2-tweaks | 14:47 |
| derekh | EmilienM: yes I do | 14:47 |
| derekh | EmilienM: https://review.openstack.org/#/c/338209/ | 14:47 |
| bnemec | derekh: Cool, hopefully that will help. | 14:47 |
| EmilienM | bnemec, derekh: can we approve that please? ^ https://review.openstack.org/#/c/338209/ | 14:49 |
| EmilienM | mitaka & liberty failure are not related | 14:49 |
| EmilienM | they worry me though | 14:49 |
| derekh | EmilienM: +1 to approving | 14:50 |
| derekh | EmilienM: the liberty error is fixed by https://review.openstack.org/#/c/336470/ | 14:50 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 14:51 |
| derekh | EmilienM: I've never seen the mitaka error though | 14:51 |
| *** eggmaster has joined #tripleo | 14:51 | |
| dprince | EmilienM: are we merging sysctl_settings parameters? | 14:52 |
| dprince | EmilienM: I don't want to set this unless plumgrid is enabled... so I think we'd need to do it in the manifest | 14:52 |
| dprince | EmilienM: anyways, I think this can be a cleanup that comes later. I will rename now though | 14:53 |
| EmilienM | dprince: in a next iteration I'll add the param in https://review.openstack.org/#/c/337359/2/puppet/services/kernel.yaml | 14:53 |
| EmilienM | dprince: that's why I wanted you to review it | 14:53 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Plumgrid helper https://review.openstack.org/338284 | 14:53 |
| EmilienM | so I can move forward with kernel params but the role would be in place | 14:53 |
| dprince | EmilienM: How are you going to "compose" the sysctl parameters? | 14:54 |
| dprince | EmilienM: is it merging hiera? | 14:54 |
| EmilienM | dprince: don't know yet :) but probably some merging stuff yes | 14:54 |
| EmilienM | like we do for firewall | 14:54 |
| EmilienM | it should not be hard | 14:54 |
| derekh | bnemec: ok, I think I know what happened on friday, now that we have stderr, its a bit more obvious | 14:54 |
| dprince | EmilienM: okay, I'd like to look into it. But my goal here is parity | 14:55 |
| EmilienM | dprince: my first step is parity now | 14:55 |
| openstackgerrit | Julie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud node introspect' command https://review.openstack.org/336595 | 14:55 |
| EmilienM | dprince: if you look my patch, it's parity | 14:55 |
| dprince | EmilienM: If kernel lands first I and supports merging I'm happy to use it | 14:55 |
| dprince | EmilienM: are you suggesting we block the compute work on this!? | 14:55 |
| EmilienM | derekh: why do we have a lot of ovb failures due to "no valid host found"? | 14:56 |
| dprince | EmilienM: cause... I can finish compute today... but with this extra merging I think it will take some more time | 14:56 |
| *** trown is now known as trown|brb | 14:56 | |
| EmilienM | dprince: not at all | 14:56 |
| EmilienM | dprince: I just want you to +A my 2 kernel roles patches asap | 14:56 |
| dprince | EmilienM: I will look at them | 14:56 |
| EmilienM | dprince: also, please open https://review.openstack.org/#/c/330785/ in a tab and when you have time give me some feedback | 14:57 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 14:57 |
| dprince | EmilienM: already got it :) | 14:57 |
| bnemec | derekh: Sounds good. I was planning to poke through the logs today just for reference anyway. | 14:57 |
| derekh | bnemec: sorry got distracted and didn't finish my explanation | 14:58 |
| bnemec | derekh: That _never_ happens to me. :-) | 14:58 |
| openstackgerrit | Dan Prince proposed openstack/puppet-tripleo: Add new nuage agent profile. https://review.openstack.org/338263 | 14:58 |
| derekh | bnemec: so we have 15 workers max and any one time creating envs for CI | 14:58 |
| derekh | bnemec: sometimes they hit this error, one sec looking for it again | 14:59 |
| derekh | + nova interface-attach --net-id 7f859f59-6f60-4234-8fc2-3031e0d25253 01448a30-9521-4b6f-93cf-712a12b0c7d6 | 14:59 |
| derekh | ERROR (CommandError): No server with a name or ID of '01448a30-9521-4b6f-93cf-712a12b0c7d6' exists. | 14:59 |
| *** trown|brb is now known as trown | 15:00 | |
| derekh | bnemec: I reckon zuul started a job and then delete the node before we got a chance to allocate it a obv env | 15:00 |
| *** rook-lappy has joined #tripleo | 15:00 | |
| *** jdob1 is now known as jdob | 15:00 | |
| derekh | bnemec: this would be ok, only I thought I was being smart when I added line 6 here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/te-broker/create-env#n6 | 15:00 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: deploy composable firewall rules for Keystone & HAproxy https://review.openstack.org/330785 | 15:01 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: deploy composable firewall rules for Keystone & HAproxy https://review.openstack.org/330785 | 15:01 |
| *** Guest84 has joined #tripleo | 15:01 | |
| derekh | bnemec: that sleep infinity was intended to stop a worker if it failed to creat an env, my assumption was that, heat stacks would fail with we hit a resource limit of some kind | 15:02 |
| derekh | bnemec: se we would want it to be taken out of the pool until somebody intervened | 15:02 |
| derekh | bnemec: I think we should just remove the trap and deal with any problems instead if they arise | 15:03 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove execution_id from the workflows https://review.openstack.org/332003 | 15:06 |
| *** pcaruana has quit IRC | 15:06 | |
| *** rcernin has quit IRC | 15:10 | |
| *** fzdarsky|afk is now known as fzdarsky | 15:11 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: make sure we start nova-compute after nova-conductor https://review.openstack.org/337839 | 15:11 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update overcloud log retrieval of rh2 https://review.openstack.org/338209 | 15:14 |
| derekh | <EmilienM> derekh: why do we have a lot of ovb failures due to "no valid host found"? | 15:15 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable midonet for neutron https://review.openstack.org/333387 | 15:15 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable neutron core compute plugin https://review.openstack.org/338315 | 15:15 |
| derekh | EmilienM: I don't know yet, I made a change to the bmc image we use to see if it help | 15:15 |
| *** liverpooler has quit IRC | 15:15 | |
| derekh | EmilienM: and I'm also about to patch the compute nodes with something else that might help | 15:16 |
| EmilienM | ok cool | 15:16 |
| EmilienM | thanks for the info | 15:16 |
| *** rajinir has joined #tripleo | 15:18 | |
| *** dprince has quit IRC | 15:18 | |
| sshnaidm | derekh, Stack overcloud CREATE_COMPLETE \o/ | 15:21 |
| gfidente | bnemec, regarding the jobs where to enable ceph | 15:21 |
| gfidente | bnemec, we actually had ceph in nonha and upgrades so adding to ha means they will all use ceph | 15:22 |
| gfidente | is there work in progress for more ovbjobs? | 15:22 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles https://review.openstack.org/333556 | 15:24 |
| *** weshay_afk is now known as weshay | 15:26 | |
| *** rhallisey has quit IRC | 15:27 | |
| derekh | sshnaidm: nice one, so that us using the exact same env that CI uses but its not yet exactly the same as CI, two things are needed for that | 15:28 |
| bnemec | gfidente: Not sure yet. I think eventually we'll be all OVB, but there's still discussion going on about whether to do that when rh1 comes back up or wait and bring rh1 back up the same as it was before. | 15:28 |
| derekh | sshnaidm: 1. the same undercloud image, I'm working on this now | 15:28 |
| derekh | sshnaidm: and 2. instead of using ./tripleo.sh directly we should be using toci_gate_test.sh which drives the whole thing | 15:29 |
| derekh | sshnaidm: for 2. it should mainly involve making sure the correct env variables are set before calling it | 15:30 |
| derekh | sshnaidm: do you want to see if you can try that out? | 15:30 |
| EmilienM | pradk: where did you fix the dbsync thing in gnocchi? https://review.openstack.org/#/c/315527/ | 15:30 |
| sshnaidm | derekh, ok | 15:31 |
| sshnaidm | derekh, I think I ran some time "env" on tripleo-ci job, I can see if I find the results and just copy most of them, maybe to save them for future anywhere | 15:32 |
| derekh | sshnaidm: sounds good, before you run it set TE_DATAFILE=~/instackenv.json so it uses the env you created and doesn't create a new one | 15:33 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the HA job https://review.openstack.org/338088 | 15:33 |
| sshnaidm | derekh, where should I run toci_gate_test.sh - on undercloud? | 15:34 |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Update overcloud passwords on deploy command https://review.openstack.org/338213 | 15:35 |
| derekh | sshnaidm: yup, on the undercloud | 15:35 |
| EmilienM | trown: hey, sorry for asking it again but it's still unclear to me. WHat is the automatic / manual process that pin TripleO CI to a RDO repo? | 15:35 |
| pradk | EmilienM, https://review.openstack.org/#/c/315527/45..46/manifests/profile/base/gnocchi/api.pp | 15:35 |
| derekh | sshnaidm: see the bottom of http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test.sh | 15:35 |
| *** oshvartz has quit IRC | 15:36 | |
| derekh | sshnaidm: if TE_DATAFILE isn't set it will get a new one and it will be deleted when its done | 15:36 |
| pradk | EmilienM, i fixed that in PS#46 | 15:36 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 15:36 |
| derekh | sshnaidm: so you want to have it set to use the one you've created | 15:36 |
| EmilienM | pradk: right sorry | 15:36 |
| EmilienM | pradk: +A | 15:36 |
| sshnaidm | derekh, ok | 15:36 |
| pradk | EmilienM, yea ccamacho might have replied by mistake | 15:37 |
| pradk | EmilienM, cool np | 15:37 |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: WIP: Update overcloud passwords on update command https://review.openstack.org/338213 | 15:37 |
| EmilienM | pradk: +2 THT also | 15:37 |
| trown | EmilienM: may need derekh to explain/confirm the tripleo-ci side, but there is a job that gets triggered in RDO when tripleo-ci succeeds | 15:37 |
| pradk | EmilienM, yay no lets hope it stays that way ;) | 15:38 |
| pradk | now* | 15:38 |
| EmilienM | pradk: if ccamacho addressed all shardy's comments, maybe shardy can +A | 15:38 |
| trown | EmilienM: prior to the server move, there was a cron job that collected results of periodic jobs and if all passed would trigger the job in RDO which does the promote | 15:38 |
| openstackgerrit | Merged openstack/instack-undercloud: Maintain quotes if present at end of secure_path https://review.openstack.org/336470 | 15:38 |
| trown | EmilienM: not sure if that changed with server move though | 15:39 |
| pradk | cool, i'll let shardy decide .. i think its in good shape | 15:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow neutron_options customization for dashboard https://review.openstack.org/260226 | 15:39 |
| EmilienM | stable/liberty CI is now fixed | 15:39 |
| EmilienM | mgould: fyi ^ | 15:39 |
| derekh | trown: EmilienM what do ye want confirmed? | 15:39 |
| derekh | EmilienM: I don't think it is fixed, yet | 15:39 |
| trown | derekh: just how current-tripleo is promoted | 15:39 |
| EmilienM | trown: do you know why we haven't promoted in 5 days? | 15:40 |
| EmilienM | trown: I see 0 blocker on puppet side | 15:40 |
| trown | EmilienM: there was an issue with mistral that just got resolved | 15:40 |
| derekh | EmilienM: trown: its isn't promoted at the moment, its in my mail about rh2 | 15:40 |
| trown | EmilienM: it was failing to import tackerclient, but dprince++ fixed it | 15:40 |
| derekh | EmilienM: trown I've yet to put the promotion bits in place | 15:40 |
| EmilienM | coolio | 15:40 |
| *** Goneri has joined #tripleo | 15:41 | |
| derekh | EmilienM: trown we need to answer the question about an instack image first, if we promote we wont have one | 15:41 |
| trown | EmilienM: I expect to get a promote in RDO today | 15:41 |
| openstackgerrit | Merged openstack/puppet-tripleo: neutron/plugins/ml2/bigswitch: do not require agent https://review.openstack.org/333398 | 15:41 |
| EmilienM | derekh: can we try to get a promotion by Friday? neutron milestone 2 is next week, we might need to promote before it | 15:41 |
| *** rhallisey has joined #tripleo | 15:41 | |
| trown | derekh: ya, I think we can use the centos image from infra as long as the overcloud images are still published | 15:42 |
| derekh | EmilienM: so we would promote and have no instack.qcow image? | 15:42 |
| trown | derekh: it seems like there is not much done on the instack.qcow2 anyways | 15:42 |
| derekh | trown: EmilienM ok, I'll take a look at it tonight | 15:42 |
| derekh | trown: ya, we mainly just preinstall the packages | 15:42 |
| trown | derekh: ya and only some | 15:43 |
| EmilienM | sounds like we have a plan | 15:43 |
| trown | so quickstart will be slightly less quick, but not such a big deal | 15:43 |
| derekh | trown: it was all packages at one stage, we just didn't keep it uptodate, probably a good reason not to do it anyways | 15:43 |
| *** rook-lappy has quit IRC | 15:43 | |
| mgould | EmilienM: great! | 15:44 |
| *** leanderthal is now known as leanderthal|afk | 15:44 | |
| ccamacho | EmilienM pradk, yeahp sorry before patchset 46 I added that comment, about changing it to step 3. About the THT submission, indeed there was an unused parameter (removed already) But I always used the map_merge (In those cases using parameters from the base file) | 15:44 |
| trown | derekh: ya, actually... I might rework the code that consumed tripleo-ci image to instead consume the overcloud-full image... that is actually pretty close to an undercloud image, and we already have the conversion code in the image building role | 15:45 |
| EmilienM | trown: I'll proceed to puppet n2 release monday morning and then work on ooo release | 15:45 |
| *** panda is now known as panda|afk | 15:45 | |
| EmilienM | ccamacho: cool | 15:45 |
| derekh | trown: sounds good | 15:45 |
| trown | EmilienM: cool, I think it is pretty safe to take ooo hashes from RDO current-passed-ci if we havent worked out promote on RH2 | 15:46 |
| EmilienM | trown: ok, milestone don't really block us | 15:46 |
| sshnaidm | derekh, should I run destroy-env before running toci_gate_test.sh ? | 15:46 |
| ccamacho | EmilienM In any case, please let's wait for CI :) | 15:46 |
| EmilienM | trown: it's more an upstream visibility | 15:46 |
| trown | EmilienM: ya exactly | 15:46 |
| openstackgerrit | Merged openstack/python-tripleoclient: Add Mistral password to deployment https://review.openstack.org/329987 | 15:46 |
| EmilienM | ccamacho: CI ran https://review.openstack.org/#/c/318413/ | 15:47 |
| derekh | sshnaidm: if your creating a new env you should run destroy and and then createenv again | 15:47 |
| sshnaidm | derekh, I have it installed now, but it's better to start from scratch, no? | 15:48 |
| *** rcernin has joined #tripleo | 15:48 | |
| ccamacho | EmilienM, neat! shardy, when having some time, can you check the changes and my comments for https://review.openstack.org/#/c/318413/ ?? | 15:48 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add gnocchi profiles https://review.openstack.org/315527 | 15:48 |
| *** sambetts is now known as sambetts|afk | 15:49 | |
| derekh | sshnaidm: so what have you got now, a fresh undercloud with a new unused ovb env attached ? | 15:50 |
| sshnaidm | derekh, not yet, right now I have a overcloud deployed | 15:51 |
| derekh | sshnaidm: ah ok, I'd start with a brand new undercloud and testenv, so yesy call destroy-env to delete the ovb env you have | 15:52 |
| derekh | sshnaidm: then start on a branch new undercloud and create and env for it | 15:52 |
| sshnaidm | derekh, ok, doing so | 15:53 |
| EmilienM | derekh: will we get upgrade job tomorrow? | 15:53 |
| EmilienM | I hope we didn't inject regression over the last days, we haven't tested upgrades | 15:53 |
| derekh | EmilienM: no, why tomorrow? we only have the jobs we have until rh1 is redeployed | 15:54 |
| EmilienM | derekh: I remember you mentionned Thursday but I might be wrong | 15:54 |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Implement aodh profiles https://review.openstack.org/332854 | 15:54 |
| derekh | EmilienM: ahh, I said the HW would be back in the new datacenter tomorrow, but don't expect it to be back running for a week after that minimum | 15:55 |
| EmilienM | ok | 15:55 |
| EmilienM | derekh: don't we have an upgrade ovb job? | 15:55 |
| *** trown is now known as trown|lunch | 15:56 | |
| sshnaidm | EmilienM, derekh we have a periodic one, do we? | 15:56 |
| EmilienM | ok, let me rephrase, why don't we have an upgrade ovb job? | 15:57 |
| derekh | EmilienM: nope we don't, we didn't have time to get it working and wouldn't have the capacity on rh2 for it | 15:57 |
| EmilienM | ok, makes sense. | 15:57 |
| *** penick has joined #tripleo | 15:57 | |
| yolanda | mm, trown, when i deploy with release master, i hit a different issue: /home/stack/stackrc is not present, i found it on /root/stackrc | 15:57 |
| derekh | sshnaidm: yes, I added a periodic one, but it wont work unless we add multinic support to the OVB envs we are creating | 15:57 |
| EmilienM | derekh: is there something I can help for rh1 re-deployment? do we have an etherpad etc? who is working on it? | 15:58 |
| EmilienM | blame me if I didn't read an email I might have missed | 15:59 |
| derekh | EmilienM: RE rh1 redeployment , nothing to be done yet but once its ready to go we can figure out what work needs to be done | 16:00 |
| *** ayoung has quit IRC | 16:00 | |
| EmilienM | derekh: please let me know | 16:00 |
| derekh | EmilienM: the main thing we have to fist figure out is if we are going to be redeploying a OVB cloud or going back to the old deployment | 16:00 |
| derekh | *first | 16:00 |
| EmilienM | derekh: re upgrade job: is it only a resource problem? or also because the job is actually not working | 16:01 |
| EmilienM | ? | 16:01 |
| EmilienM | derekh: I see | 16:01 |
| derekh | EmilienM: both | 16:01 |
| EmilienM | derekh: ok | 16:01 |
| derekh | EmilienM: we don't have the resource right now and also | 16:01 |
| EmilienM | derekh: we need to carefuly patch tripleo-ci then (ie logs) if we switch back into old system | 16:01 |
| derekh | EmilienM: need to add multiple nics to the testenvs that we create for the overcloud | 16:01 |
| EmilienM | ok it's more clear now | 16:02 |
| *** yamahata has quit IRC | 16:02 | |
| EmilienM | I really hope we won't break the upgrade job too much | 16:02 |
| EmilienM | and fix it asap | 16:02 |
| derekh | EmilienM: yup, manual testing is what we have to rely on at the moment | 16:02 |
| *** _xou_ has quit IRC | 16:03 | |
| *** tremble has quit IRC | 16:03 | |
| derekh | bnemec: compute nodes all patched, and nova-compute restarted on each node | 16:04 |
| *** athomas has quit IRC | 16:04 | |
| EmilienM | ok so from now, we should potentially have the "No hosts" thing fixed | 16:04 |
| bnemec | derekh: Cool. | 16:04 |
| derekh | EmilienM: maybe, that two things I've updated todate to try and help it | 16:05 |
| derekh | EmilienM: but now 100% sure | 16:05 |
| bnemec | derekh: I'll try to take a look at your quintupleo patch today. That would get us multinic support basically for free. | 16:06 |
| *** oneswig has quit IRC | 16:06 | |
| derekh | EmilienM: if neither of those work, the next thing to try would be changing lin 196 to have $(NODECOUNT+1) here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test.sh#n196 | 16:06 |
| derekh | EmilienM: that would make another overcloud node available to the deployment if one failed | 16:07 |
| derekh | EmilienM: but I'd like to avoid that if we can because it would also take more ram from the ci quota | 16:07 |
| derekh | bnemec: soulds good | 16:07 |
| EmilienM | derekh: right | 16:09 |
| lucasagomes | hi all, maybe it's a dummy question but... Does tripleo have any spec/patches (something more tangible) on the split-stack work? | 16:10 |
| *** matbu is now known as matbu|brb | 16:10 | |
| lucasagomes | I can see blog posts and ML threads about splitting the heat stack in two smaller pieces (one for baremetal configuration and another one for configuring the OS services) | 16:10 |
| derekh | sshnaidm: I've applied this patch to rh2, so baremetal nodes should now be reusable https://github.com/cybertron/openstack-virtual-baremetal/blob/master/patches/nova/nova-pxe-boot.patch | 16:11 |
| lucasagomes | but I can't find whether it's being worked on or not | 16:11 |
| lucasagomes | slagle, ^ maybe you know? | 16:11 |
| ccamacho | gfidente, can you check my last comment on https://review.openstack.org/#/c/318413/ just to see if I got it right. | 16:12 |
| *** ifarkas has quit IRC | 16:13 | |
| *** tzumainn has quit IRC | 16:13 | |
| gfidente | ccamacho, sure, thanks | 16:13 |
| gfidente | so the parameters should be distributed to the services which need them | 16:13 |
| gfidente | -base should be fine | 16:14 |
| gfidente | but yes it's mainly about removing those from ceph-cluster.yaml yes | 16:14 |
| gfidente | ccamacho, and this guy as well https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/ceph-cluster-config.yaml#L34 | 16:15 |
| *** Guest84 has quit IRC | 16:15 | |
| *** ayoung has joined #tripleo | 16:15 | |
| ccamacho | gfidente, didn't noticed the last one, /me applying the changes! Thanks! | 16:16 |
| gfidente | ty! | 16:16 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-specs: Add initial next generation HA architecture spec https://review.openstack.org/299628 | 16:21 |
| slagle | lucasagomes: the most concrete thing we have so far is: https://review.openstack.org/#/c/222772/ | 16:22 |
| slagle | not a true split stack though | 16:22 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Gnocchi composable roles https://review.openstack.org/318413 | 16:23 |
| lucasagomes | slagle, thanks | 16:23 |
| *** yamahata has joined #tripleo | 16:25 | |
| *** jaimguer_ has joined #tripleo | 16:26 | |
| *** jaimguer_ has quit IRC | 16:27 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo: WIP Next generation HA architecture work https://review.openstack.org/338387 | 16:28 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates: WIP Next generation HA architecture work https://review.openstack.org/314208 | 16:28 |
| derekh | can somebody else give this a look, https://review.openstack.org/#/c/337775/2 | 16:28 |
| *** snecklifter has quit IRC | 16:28 | |
| *** jaimguer_ has joined #tripleo | 16:29 | |
| openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: Composable Horizon service - puppet-tripleo https://review.openstack.org/335506 | 16:30 |
| *** dprince has joined #tripleo | 16:32 | |
| *** jaimguer_ has quit IRC | 16:32 | |
| openstackgerrit | Merged openstack/instack-undercloud: Use the print function to fix the tests on Python 3 https://review.openstack.org/335383 | 16:33 |
| EmilienM | derekh: +A | 16:35 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Log stderr when creating and destroying envs https://review.openstack.org/337775 | 16:35 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Composable Horizon service - tripleo-heat-templates https://review.openstack.org/335499 | 16:35 |
| *** jaimguer_ has joined #tripleo | 16:35 | |
| *** jaimguer_ has quit IRC | 16:35 | |
| *** florianf has quit IRC | 16:36 | |
| derekh | EmilienM: thanks, bnemec I'm going to reinable the cronjob on the te_broker node so it again gets updates to tripleo-ci scripts automatically | 16:36 |
| bnemec | derekh: Finally getting back to your comment on http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/te-broker/create-env#n6 | 16:36 |
| *** jaimguer_ has joined #tripleo | 16:36 | |
| bnemec | Removing that for OVB probably makes sense. | 16:36 |
| *** jaimguer_ has quit IRC | 16:36 | |
| bnemec | In the old CI env if something went wrong it was likely we couldn't recover automatically, but now we're in THE CLOUD!!1 | 16:37 |
| bnemec | Spurious failures have to be expected. :-) | 16:37 |
| derekh | bnemec: ok, will push a patch up in a minute | 16:37 |
| *** jaimguer_ has joined #tripleo | 16:37 | |
| *** jaimguer_ has quit IRC | 16:38 | |
| *** jaimguer_ has joined #tripleo | 16:38 | |
| *** jaimguer_ has quit IRC | 16:39 | |
| *** jaimguer_ has joined #tripleo | 16:39 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove sleeps if testenv create/destroy fails https://review.openstack.org/338394 | 16:42 |
| derekh | bnemec: ^ | 16:42 |
| derekh | bnemec: I've re-enabled auto update for the broker, so any changes should be picked up automatically | 16:43 |
| *** trown|lunch is now known as trown | 16:44 | |
| bnemec | derekh: Cool, +2. | 16:46 |
| bnemec | derekh: To recover from this, did you just kill the stuck te-workers? | 16:46 |
| *** rook-lappy has joined #tripleo | 16:46 | |
| derekh | bnemec: to recover from a worker going to sleep? | 16:47 |
| derekh | bnemec: I had never killed the worker, just the sleep command, that way the destroy script would be called | 16:47 |
| bnemec | derekh: Yeah, just wondering in general what the recovery steps from a worker going horribly wrong are. | 16:47 |
| bnemec | Ah, okay. Makes sense. | 16:48 |
| * bnemec makes a note to not kill testenv-workers | 16:48 | |
| derekh | bnemec: but once thats merged any worker will no longer sleep, so they shouldn't get "stuck" | 16:49 |
| bnemec | It saddens me how little we are actually using our faster compute nodes in rh2. | 16:49 |
| bnemec | derekh: Right. | 16:49 |
| *** amoralej is now known as amoralej|off | 16:49 | |
| *** tesseract- has quit IRC | 16:49 | |
| *** yamahata has quit IRC | 16:49 | |
| derekh | bnemec: ya, we should fix that also | 16:50 |
| *** devvesa has quit IRC | 16:51 | |
| *** penick has quit IRC | 16:51 | |
| *** yamahata has joined #tripleo | 16:51 | |
| *** matbu|brb is now known as matbu | 16:51 | |
| bnemec | derekh: I've looked into it, but I haven't found a simple way. From a pure numbers standpoint the slower boxes just look better (more cpus, more ram, more disk). | 16:52 |
| derekh | bnemec: pitty ;-( | 16:53 |
| *** shivrao has joined #tripleo | 16:54 | |
| bnemec | Although we might be able to force certain node types onto the faster boxes using host aggregates. | 16:54 |
| *** lucasagomes is now known as lucas|afk | 16:55 | |
| *** jaimguer_ has quit IRC | 16:56 | |
| derekh | bnemec: I'm kindof under the impression (maybe illusion) that the extra RAM on the boxes with the slowers disks compensates | 16:56 |
| bnemec | Or we could just disable a couple of the slower boxes. With 15 testenvs we're not even close to maxing out the available hardware from what I can see. | 16:56 |
| derekh | bnemec: because we have the VM running in an unsafe disk caching mode, so nothing is waiting for disk syncs as longs as there is enaough RAM to allow it | 16:56 |
| *** shivrao_ has joined #tripleo | 16:57 | |
| bnemec | derekh: Good point. I guess the jobs really aren't that slow right now either. They're mostly finishing in 1:30-1:40. | 16:58 |
| derekh | bnemec: worth thinking about, but then we might end up CPU bound, | 16:58 |
| *** jaimguer_ has joined #tripleo | 16:58 | |
| bnemec | Maybe a benefit of spreading the load of multiple hosts instead of putting all the HA overcloud nodes on one box at the same time. | 16:58 |
| *** shivrao has quit IRC | 16:58 | |
| *** shivrao_ is now known as shivrao | 16:58 | |
| derekh | bnemec: yup, I thinkk once we improve the failure rate a bit, we can try tweaking a couple of things until we find whats optimum | 16:58 |
| derekh | bnemec: anyways I gotta run, will check in on the cluod later to see if anything is on fire | 16:59 |
| *** jaimguer_ has quit IRC | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** jaimguer_ has joined #tripleo | 17:00 | |
| *** padkrish has joined #tripleo | 17:00 | |
| *** chem has quit IRC | 17:00 | |
| *** cdearborn has joined #tripleo | 17:01 | |
| *** jaimguer_ has quit IRC | 17:02 | |
| *** jaimguer_ has joined #tripleo | 17:02 | |
| *** jaimguer_ has quit IRC | 17:03 | |
| *** jaimguer_ has joined #tripleo | 17:03 | |
| padkrish | Hello all, i am running into issues trying to install TripleO in a VM environment. Can i ask in this forum or is there a different IRC or mailing list? | 17:04 |
| *** jaimguer_ has joined #tripleo | 17:04 | |
| *** shivrao has quit IRC | 17:04 | |
| *** ayoung has quit IRC | 17:06 | |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: WIP: Update overcloud passwords on update command https://review.openstack.org/338213 | 17:07 |
| *** fzdarsky is now known as fzdarsky|afk | 17:07 | |
| *** jaimguer_ has quit IRC | 17:08 | |
| *** jpena is now known as jpena|off | 17:10 | |
| shardy | padkrish: Hi, you can ask your question here :) | 17:11 |
| padkrish | shardy# thanks :) | 17:12 |
| padkrish | I was following the instructions in http://docs.openstack.org/developer/tripleo-docs/installation/installation.html (stable/liberty) | 17:13 |
| padkrish | When i do a "“openstack undercloud install”", it eventually fails with: | 17:13 |
| padkrish | + puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp | 17:14 |
| padkrish | Error: Resource type oslo::cache doesn't exist on node instack.localdomain | 17:14 |
| padkrish | Error: Resource type oslo::cache doesn't exist on node instack.localdomain | 17:14 |
| padkrish | I had to make couple of changes manually, as per https://bugzilla.redhat.com/show_bug.cgi?id=1304395. | 17:15 |
| openstack | bugzilla.redhat.com bug 1304395 in openstack-tripleo "openstack overcloud image upload fails with "Required file "./ironic-python-agent.initramfs" does not exist."" [Unspecified,Closed: worksforme] - Assigned to jslagle | 17:15 |
| *** jaimguer_ has joined #tripleo | 17:16 | |
| shardy | padkrish: Hmm, sounds like a puppet module is missing or the wrong version, it's possible there's a docs bug if you've followed the docs for stable branch | 17:16 |
| *** jaimguer_ has quit IRC | 17:16 | |
| shardy | padkrish: you might try cloning the tripleo-ci repo and re-configuring the repos with tripleo.sh | 17:16 |
| shardy | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh | 17:16 |
| shardy | you can do export STABLE_RELEASE=liberty | 17:17 |
| shardy | then tripleo.sh --repo-setup | 17:17 |
| shardy | maybe copy/tar your /etc/yum.repos.d before so you can compare before/after and figure out what in the docs is wrong | 17:17 |
| shardy | that script is what we use in CI, so it's possible it's more current than the docs, especially for old stable branches | 17:18 |
| shardy | is there a reason you're using liberty instead of mitaka? | 17:18 |
| padkrish | shardy# no specific reason as of now, i was just following the docs for the stable version :) | 17:19 |
| *** ayoung has joined #tripleo | 17:19 | |
| *** jaimguer_ has joined #tripleo | 17:19 | |
| *** jpich has quit IRC | 17:20 | |
| padkrish | shardy# it's my first install, so thought would follow the stable branch, nothing else | 17:20 |
| *** dsariel has quit IRC | 17:22 | |
| *** chem has joined #tripleo | 17:23 | |
| *** chem has quit IRC | 17:23 | |
| padkrish | shardy# i am assuming i should be cloning the tripleo-ci repo in the undercloud VM? | 17:23 |
| shardy | padkrish: ack, I can see the docs still point to liberty, so we should probably update to the current latest stable/mitaka | 17:23 |
| shardy | padkrish: yes, if you clone that repo, you can then run tripleo.sh which will automate a few steps from the docs | 17:24 |
| shardy | it's possible (even likely) that the stable branch docs and that script have diverged | 17:24 |
| *** jaimguer_ has quit IRC | 17:24 | |
| padkrish | looks like the script is also deploying the overcloud, should i pass any arguments for it to deploy it as VM...like --libvirt-type qemu as given in the docs? | 17:28 |
| *** jaimguer_ has joined #tripleo | 17:28 | |
| shardy | padkrish: did you just run tripleo.sh --repo-setup? | 17:28 |
| padkrish | shardy# i will do that first...sorry, i started looking into the script.. | 17:29 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable neutron core compute plugin https://review.openstack.org/338315 | 17:29 |
| ayoung | shardy, is it possible to run just a nested stack that makes up part of tripleo? Like, can I run Tripleo one Heat (sub) stack at a time? | 17:29 |
| shardy | ayoung: Yes, but in some (many) cases there are dependencies on data from the parent | 17:30 |
| shardy | so it can get tricky | 17:30 |
| *** jaimguer_ has joined #tripleo | 17:30 | |
| *** Goneri has quit IRC | 17:30 | |
| ayoung | shardy, is there someway to snapshot that data? | 17:30 |
| shardy | it is possible to deploy any heat stack seperately though, provided you can pass in some valid data | 17:30 |
| shardy | ayoung: you could write a script that dumps the stack-show output for all stacks returned from heat stack-list -n I guess | 17:31 |
| shardy | it'd be kinda hard work unravelling all the data though | 17:32 |
| shardy | and you'd need to figure out the DAG so you could create things in the correct order | 17:32 |
| ayoung | shardy, but otherwise there is no way to say "run just the hardward setup stage, but use all the data from the full overcloud deploy" | 17:32 |
| ayoung | shardy, we need a Heat debugger | 17:32 |
| shardy | personally I only sometimes create nested stacks directly, mostly when debugging an issue with some nested stack template | 17:32 |
| *** shivrao has joined #tripleo | 17:32 | |
| *** jaimguer_ has quit IRC | 17:32 | |
| shardy | ayoung: well, it depends what you're trying to do, it is possible to noop the software configuration, run the deployment, then enable it | 17:33 |
| *** jaimguer_ has joined #tripleo | 17:33 | |
| shardy | and you could enable it incrementally | 17:33 |
| shardy | ayoung: there is also an interface for heat hooks (aka breakpoints) which enables stopping at a pre-defined point in the graph | 17:33 |
| ayoung | shardy, because in a redeploy heat is smart enough to skip what is already done | 17:33 |
| shardy | ayoung: yup | 17:33 |
| ayoung | shardy, that is key knowledge... | 17:34 |
| shardy | even from a FAILED state, it won't replace COMPLETE resources, it'll just walk the graph and carry on | 17:34 |
| EmilienM | pradk: https://review.openstack.org/311762 is green! | 17:34 |
| shardy | ayoung: yeah, you can just keep running openstack overcloud deploy until it works | 17:35 |
| shardy | with no delete inbetween | 17:35 |
| *** jaimguer_ has joined #tripleo | 17:35 | |
| ayoung | shardy, I can only find this as docs for breakpoints https://specs.openstack.org/openstack/heat-specs/specs/juno/stack-breakpoint.html | 17:37 |
| *** jaimguer_ has quit IRC | 17:37 | |
| ayoung | is there better? | 17:37 |
| *** jaimguer_ has joined #tripleo | 17:38 | |
| shardy | http://docs.openstack.org/developer/heat/template_guide/environment.html#pause-stack-creation-update-or-deletion-on-a-given-resource | 17:38 |
| shardy | ayoung: ^^ | 17:38 |
| shardy | the interface is a bit awkward because the status of the hook is output via events | 17:38 |
| ayoung | shardy, so, one thing about OOOqs is that it does a shell script, and keeps adding more -e options. Am I right in thinking that a deploy should have a singe record of environmental state, and we should have one file that you include with all of the customizations for the deploy? | 17:41 |
| ayoung | something like | 17:41 |
| ayoung | openstack overcloud deploy --templates $TEMPLATE_DIR -e my_setup.yml | 17:42 |
| *** Goneri has joined #tripleo | 17:42 | |
| ayoung | so if I do a redeploy, I am sure to get the same state each time, maybe with just the delta for what I am working on? | 17:43 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: [WIP] Update to use quintupleo https://review.openstack.org/336309 | 17:43 |
| shardy | ayoung: You can do that, but it requires manually merging the various -e enable_foo.yaml templates | 17:44 |
| shardy | ayoung: anoter alternative is to point to a directory of environment files: | 17:44 |
| shardy | https://github.com/openstack/python-tripleoclient/commit/0e10a7935b8fda00ff7ca01d25aeec92ae9ed80d | 17:44 |
| shardy | mostly folks do want to have several -e enable_foo.yaml -e enable_bar.yaml environments | 17:44 |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: WIP: Update overcloud passwords on update command https://review.openstack.org/338213 | 17:44 |
| shardy | as it's a bit easier to manage than maintaining one mega-environment | 17:45 |
| ayoung | shardy, right, but what happens if you execute opentack overcloud deploy a second time and leave on off? Didn't you possibly undo work someonelse did? | 17:45 |
| *** jaimguer_ has quit IRC | 17:45 | |
| ayoung | "leave One off" | 17:45 |
| shardy | ayoung: Possibly, but we do a PATCH update, so in most cases existing data shouldn't be changed | 17:46 |
| *** jaimguer_ has joined #tripleo | 17:46 | |
| shardy | we just reuse the current e.g parameter_defaults if you don't pass a new value | 17:46 |
| shardy | same for all the templates | 17:46 |
| padkrish | shardy# i ran tripleo.sh --repo-setup? after setting the STABLE_RELEASE to liberty | 17:46 |
| *** jaimguer_ has quit IRC | 17:47 | |
| ayoung | shardy, I see. So if I do a stack show, I see the current state of the env vars. When I do a new redeploy, it updates those before continuing | 17:47 |
| *** jaimguer_ has joined #tripleo | 17:47 | |
| *** jaimguer_ has joined #tripleo | 17:47 | |
| shardy | ayoung: yes, any new data you pass in to the next overcloud deploy will take effect when the update happens | 17:48 |
| *** jaimguer_ has quit IRC | 17:49 | |
| padkrish | shardy# after running it, i do not see the delorean-liberty.repo in /etc/yum.repos.d, i see delorean-current.repo | 17:49 |
| shardy | padkrish: yup, but if you look inside it, does it point to the same place as the old delorean-liberty.repo? | 17:50 |
| *** jaimguer_ has joined #tripleo | 17:50 | |
| padkrish | shardy# nope, it's empty, the file is not populated at all...i did not get any error when i ran tripleo.sh | 17:51 |
| padkrish | shardy# delorean-deps.repo seem to be the same as the old delorean-deps-liberty.repo except for priority | 17:52 |
| *** jaimguer_ has quit IRC | 17:52 | |
| pradk | EmilienM, nice, thx! | 17:53 |
| *** jaimguer_ has joined #tripleo | 17:53 | |
| shardy | padkrish: ack, I think that's OK http://paste.openstack.org/show/526598/ | 17:54 |
| shardy | possibly the priority is what we have wrong in the docs | 17:54 |
| shardy | does yum check-update show you changes to the puppet modules? | 17:55 |
| openstackgerrit | Matt Young proposed openstack/tripleo-common: Add support to image build yaml input to handle env vars https://review.openstack.org/318087 | 17:55 |
| thrash | myoung: where ya been? :) | 17:57 |
| trown | lol | 17:57 |
| *** shardy has quit IRC | 17:58 | |
| *** bootsha has joined #tripleo | 17:59 | |
| *** jaimguer_ has quit IRC | 17:59 | |
| myoung | thrash: see, there was this bus...on fire...full of orphans...heading for a cliff. someone had to step up. I'm back now. | 18:00 |
| padkrish | shardy# http://paste.openstack.org/show/526600/ looks similar | 18:00 |
| *** jaimguer_ has joined #tripleo | 18:00 | |
| *** jaimguer_ has joined #tripleo | 18:01 | |
| gfidente | myoung, I suppose the orphans are safe now? | 18:01 |
| gfidente | I mean, I hope so | 18:02 |
| thrash | myoung: lol | 18:02 |
| padkrish | shardy# yum check-update does not show changes to puppet | 18:02 |
| *** jaimguer_ has quit IRC | 18:02 | |
| *** jaimguer_ has joined #tripleo | 18:03 | |
| padkrish | shardy# http://paste.openstack.org/show/526603/ | 18:04 |
| *** rook-lappy has quit IRC | 18:05 | |
| *** rook-lappy has joined #tripleo | 18:05 | |
| *** jaimguer_ has quit IRC | 18:05 | |
| EmilienM | Went to status ERROR due to "Message: No valid host was found | 18:05 |
| EmilienM | I still see the error FYI | 18:05 |
| EmilienM | bnemec: ^ | 18:05 |
| gfidente | I hit that as well it sees | 18:06 |
| gfidente | EmilienM, ^^ | 18:06 |
| EmilienM | gfidente: hey can you review https://review.openstack.org/#/c/337358/ and https://review.openstack.org/#/c/337359/ please? | 18:07 |
| *** jaimguer_ has joined #tripleo | 18:07 | |
| *** jaimguer_ has joined #tripleo | 18:07 | |
| bnemec | Hmm, okay. I haven't seen any errors in the bmcs all day, so I suspect this is a different thing. | 18:08 |
| yolanda | trown, found problem with swift... ERROR with Object server 192.0.2.1:6000/1 re: Trying to write to /v1/AUTH_ea990020037e482c9f7eb01762e325fb/glance/5564afbd-03a5-4ef4-8a1c-cd7917f875ae: ChunkWriteTimeout (10.0s) | 18:08 |
| yolanda | is there any way to manipulate that timeout from oooq? | 18:08 |
| *** jaimguer_ has joined #tripleo | 18:08 | |
| *** jaimguer_ has joined #tripleo | 18:10 | |
| trown | yolanda: not directly, but if it is exposed via puppet, we could change it in instack-undercloud | 18:10 |
| openstackgerrit | Richard Su proposed openstack/tripleo-image-elements: WIP Remove unneeded SELinux custom policies https://review.openstack.org/338440 | 18:12 |
| trown | yolanda: it is happening even on master where we switched to file backend? | 18:12 |
| bnemec | trown: yolanda: https://review.openstack.org/#/c/332014/ | 18:12 |
| yolanda | trown, master had another problem. stackrc was not present under /home/stack, but on /root/stack | 18:12 |
| bnemec | yolanda: That means the undercloud install failed. | 18:13 |
| yolanda | oh bnemec , that's the change i needed! | 18:13 |
| *** jaimguer_ has quit IRC | 18:13 | |
| bnemec | We should probably backport it. | 18:13 |
| yolanda | please :) | 18:13 |
| *** jaimguer_ has joined #tripleo | 18:13 | |
| yolanda | confirmed, that timeout increase solves the problem | 18:15 |
| yolanda | as a workaround i can just add a sed on proxy-server.conf on undercloud post deploy script, but it will be super helpful to have it on mitaka | 18:16 |
| *** jaimguer_ has joined #tripleo | 18:16 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Increase swift-proxy node_timeout https://review.openstack.org/338453 | 18:18 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Increase swift-proxy node_timeout https://review.openstack.org/338456 | 18:19 |
| bnemec | <yolanda> as a workaround i can just add a sed on proxy-server.conf on undercloud post deploy script | 18:20 |
| bnemec | ^Please don't do that. Anyone can propose backports, which is the right way to handle something like this. | 18:20 |
| openstackgerrit | Merged openstack/puppet-tripleo: Create kernel profile https://review.openstack.org/337358 | 18:20 |
| EmilienM | yolanda: right, please keep in mind config management is done via Puppet, nothing else for now. | 18:21 |
| EmilienM | don't hack things with bash please :-) | 18:21 |
| EmilienM | bnemec: +2 on backports | 18:21 |
| *** padkrish has quit IRC | 18:22 | |
| EmilienM | gfidente: thx man | 18:22 |
| trown | +1 to not adding it into quickstart tree, but hacking around broken stuff to get work done is totally fine | 18:23 |
| dmsimard | +1 to not forget to eventually remove the hacking | 18:23 |
| dmsimard | :) | 18:23 |
| trown | dmsimard: right that is why it would not go in tree :P | 18:23 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add kernel service https://review.openstack.org/337359 | 18:24 |
| *** jaimguer_ has joined #tripleo | 18:25 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Composable midonet for neutron https://review.openstack.org/333387 | 18:25 |
| *** jaimguer_ has joined #tripleo | 18:25 | |
| EmilienM | dprince: rebasing our tht work | 18:27 |
| *** padkrish has joined #tripleo | 18:27 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Composable neutron core compute plugin https://review.openstack.org/338315 | 18:27 |
| dprince | EmilienM: no please! | 18:28 |
| dprince | EmilienM: I have 4 more patches to post now... :/ | 18:28 |
| openstackgerrit | Giulio Fidente proposed openstack/python-tripleoclient: Update overcloud passwords on update command https://review.openstack.org/338213 | 18:28 |
| dprince | EmilienM: okay, besides the rebase there were no changes? | 18:28 |
| *** Goneri has quit IRC | 18:30 | |
| *** bootsha has quit IRC | 18:30 | |
| jidar | I'm trying to deploy using the DeployIdentifier stuff from suggested by shardy and running into an issue, I'm on a fairly old patchset from Kilo and I keep getting this error: Resource CREATE failed: resources.ExtraConfig: Property error: resources.LDAPConfig.properties: Property DeployIdentifier not assigned | 18:30 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable Neutron Core Compute Plugin https://review.openstack.org/338315 | 18:31 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable Midonet for Neutron https://review.openstack.org/333387 | 18:31 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable Nuage Compute Plugin https://review.openstack.org/338477 | 18:31 |
| jidar | anybody familiar with how that param gets passed in and how it needs to be setup in a template? | 18:31 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable Midonet compute plugin https://review.openstack.org/338478 | 18:31 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable Plumgrid compute plugin https://review.openstack.org/338479 | 18:31 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Composable OpenContrail compute plugin https://review.openstack.org/338480 | 18:31 |
| *** penick has joined #tripleo | 18:31 | |
| EmilienM | dprince: no change | 18:32 |
| dprince | EmilienM: there are 4 more patches now | 18:32 |
| dprince | EmilienM: ^^^ | 18:32 |
| dprince | EmilienM: neutron is now gone from compute | 18:32 |
| *** jaimguer_ has joined #tripleo | 18:32 | |
| dprince | EmilienM: still testing but I think this should work out | 18:32 |
| *** jaimguer_ has quit IRC | 18:32 | |
| EmilienM | k | 18:33 |
| EmilienM | I'll test it too | 18:33 |
| EmilienM | I can't deploy quickstart now, having troubles | 18:33 |
| EmilienM | is master supposed to work? | 18:33 |
| dprince | EmilienM: that is next up for me as well. Something seems to have broken in my dev environment | 18:34 |
| dprince | EmilienM: using master | 18:34 |
| dprince | EmilienM: just looked at your kernel stuff. Looks like it landed | 18:35 |
| *** ayoung has quit IRC | 18:35 | |
| dprince | EmilienM: I might have split out sysctl... but if others are fine I guess it is fine | 18:36 |
| dprince | EmilienM: Honestly we always want sysctl there anyways. | 18:36 |
| EmilienM | dprince: split kmod & sysctl? | 18:37 |
| dprince | EmilienM: yes, they are separate things | 18:37 |
| dprince | EmilienM: but we likely always want them on so whatever | 18:37 |
| EmilienM | what? | 18:38 |
| EmilienM | they are not really separate things | 18:38 |
| EmilienM | one is loading the module and another is configuring the module | 18:38 |
| EmilienM | afik | 18:38 |
| dprince | EmilienM: it is entire possible to use sysctl without kmod | 18:38 |
| dprince | EmilienM: perhaps risky, but possible | 18:38 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Remove superfluous 'fi' from feature-scale-deploy.sh https://review.openstack.org/338290 | 18:38 |
| EmilienM | yeah except if your module is not loaded | 18:38 |
| dprince | EmilienM: it would be if a used a static kernel | 18:38 |
| EmilienM | yeah, right now I did a baby step by moving the code. | 18:38 |
| EmilienM | we can change it later | 18:39 |
| EmilienM | sysctl is clearly not my priority | 18:39 |
| dprince | EmilienM: you asked me to review. This is my feedback :) | 18:39 |
| EmilienM | dprince: my goal is to evacuate overcloud_*.pp | 18:40 |
| openstackgerrit | Merged openstack/puppet-tripleo: Add non-pcmk Trove API/Conductor/Taskmanager profiles https://review.openstack.org/310795 | 18:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Trove Integration https://review.openstack.org/233240 | 18:40 |
| EmilienM | dprince: thx for feedback | 18:40 |
| dprince | EmilienM: yep, those last 4 neutron patches removed a huge chunk of overcloud_compute.pp. https://review.openstack.org/#/c/338480/1/puppet/manifests/overcloud_compute.pp | 18:41 |
| EmilienM | dprince: excellent, this is our priority now :) | 18:41 |
| gfidente | those kmod/sysctl are probably a bit like role-specific implementations of OVS | 18:42 |
| gfidente | we might want to distribute role-specific keys from the templates | 18:42 |
| *** sshnaidm is now known as sshnaidm|afk | 18:42 | |
| EmilienM | dprince: there is a puppet syntax error: https://review.openstack.org/#/c/333387/ | 18:42 |
| gfidente | say load a certain controller module only on storage nodes, or sysctl settings only on the nodes running neutron | 18:42 |
| EmilienM | gfidente: right, also swift | 18:42 |
| EmilienM | gfidente: I could make it like firewall rules | 18:43 |
| *** Goneri has joined #tripleo | 18:43 | |
| *** jaimguer_ has joined #tripleo | 18:43 | |
| EmilienM | gfidente: what do you think about https://review.openstack.org/#/c/330785/ ? | 18:43 |
| *** jaimguer_ has quit IRC | 18:43 | |
| EmilienM | can you look at it? | 18:43 |
| gfidente | naaaah I was about to leave man :) | 18:43 |
| *** anshul has joined #tripleo | 18:44 | |
| gfidente | though the various *ExtraConfig interfaces should be good for the purpose of having role-specific module/keys | 18:44 |
| EmilienM | gfidente: yeah also | 18:45 |
| EmilienM | dprince: if you got 2 min, can you look https://review.openstack.org/#/c/330785/ before I continue this work? | 18:45 |
| *** jaimguer_ has joined #tripleo | 18:45 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 18:45 |
| *** jaimguer_ has joined #tripleo | 18:46 | |
| *** jaimguer_ has joined #tripleo | 18:47 | |
| gfidente | fwiw this is a recent 'no valid host' | 18:48 |
| gfidente | http://logs.openstack.org/88/338088/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/5c31117/console.html | 18:48 |
| *** padkrish has quit IRC | 18:48 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Add environment file to enable DVR https://review.openstack.org/332147 | 18:48 |
| dprince | EmilienM: seems fine. But per our conversation earlier about sysctl settings... as I understood you your preference was to configure things via Hiera in t-h-t. | 18:48 |
| dprince | EmilienM: with these firewall rules you are going to make an exception though? | 18:49 |
| *** trown is now known as trown|brb | 18:49 | |
| dprince | EmilienM: could we figure out a way to merge the hiera (only for firewall settings) and then apply it via one function? | 18:49 |
| *** jaimguer_ has quit IRC | 18:49 | |
| dprince | EmilienM: because I would prefer managing it that way, as opposed to hard coding the ports into the manifest | 18:49 |
| colonwq | gfidente, can you review 324774, 334081 and 289027? They are 'all green' but I need some good eyes to help finish polishing these patches. | 18:49 |
| *** jaimguer_ has joined #tripleo | 18:50 | |
| EmilienM | dprince: I'm still thinking about it. What do you prefer on your side? | 18:50 |
| EmilienM | ah hiera | 18:50 |
| dprince | EmilienM: I like the idea of the t-h-t mechanism | 18:50 |
| EmilienM | ok I can investigate a hiera thing for iptables & sysctl, let me think about it :) | 18:50 |
| EmilienM | I'll WIP my patch in the meantime | 18:50 |
| dprince | EmilienM: If it were just a single patch I would say lets land it. But I wouldn't spend time moving this around until we investigate the hiera mechanism myself. | 18:51 |
| EmilienM | dprince: right let me some time to think about it | 18:51 |
| bnemec | gfidente: Hmm, the ironic logs for that job look pretty ugly. NodeAssociateds everywhere, then a couple of ipmi errors. | 18:52 |
| *** bootsha has joined #tripleo | 18:53 | |
| *** jaimguer_ has joined #tripleo | 18:53 | |
| *** jaimguer_ has joined #tripleo | 18:55 | |
| EmilienM | dprince: we'll need github.com/rnelson0/hiera_resources | 18:55 |
| EmilienM | this thing is AWESOME | 18:55 |
| *** penick has quit IRC | 18:56 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles https://review.openstack.org/333556 | 18:56 |
| EmilienM | dprince: but I have to think about a merge mechanism | 18:56 |
| dprince | EmilienM: cool | 18:58 |
| *** jaimguer_ has quit IRC | 18:58 | |
| gfidente | coolsvap, done, we're almost there I think :) | 18:58 |
| *** jaimguer_ has joined #tripleo | 18:59 | |
| *** penick has joined #tripleo | 18:59 | |
| *** jaimguer_ has quit IRC | 19:00 | |
| *** jaimguer_ has joined #tripleo | 19:01 | |
| *** jaimguer_ has quit IRC | 19:03 | |
| *** gfidente has quit IRC | 19:03 | |
| *** jaimguer_ has joined #tripleo | 19:03 | |
| *** padkrish has joined #tripleo | 19:07 | |
| *** jaimguer_ has quit IRC | 19:09 | |
| *** jaimguer_ has joined #tripleo | 19:12 | |
| EmilienM | bnemec: nice catch on https://review.openstack.org/337736 | 19:12 |
| EmilienM | bnemec: a comment though | 19:13 |
| yolanda | EmilienM, ok.. i meant temporarily to make my tests run | 19:14 |
| yolanda | so backports are managed via a normal change? | 19:15 |
| *** jaimguer_ has quit IRC | 19:16 | |
| *** trown|brb is now known as trown | 19:16 | |
| *** jaimguer_ has joined #tripleo | 19:17 | |
| *** fzdarsky|afk has quit IRC | 19:17 | |
| *** akrivoka has quit IRC | 19:18 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo: Fix Ceilometer profiles https://review.openstack.org/336699 | 19:18 |
| EmilienM | pradk: if you do that ^ can you recheck ceilo patch? | 19:21 |
| *** fragatina has quit IRC | 19:21 | |
| pradk | ok | 19:21 |
| *** fragatina has joined #tripleo | 19:22 | |
| *** rook-lappy has quit IRC | 19:23 | |
| *** jaimguer_ has joined #tripleo | 19:24 | |
| *** jaimguer_ has joined #tripleo | 19:26 | |
| EmilienM | dprince: sorry wrong window, but here: https://review.openstack.org/#/c/330785/10/manifests/haproxy/endpoint.pp | 19:26 |
| EmilienM | we can't use hiera for firewall rules | 19:26 |
| EmilienM | because we have some (good) logic that enable binding options in our manifests | 19:26 |
| EmilienM | maybe for sysctl we can do it with Hiera, I'm pretty sure | 19:26 |
| EmilienM | and even for some firewall rules | 19:26 |
| EmilienM | but not all | 19:26 |
| EmilienM | like https://review.openstack.org/#/c/330785/10/manifests/profile/base/keystone.pp | 19:27 |
| *** penick has quit IRC | 19:27 | |
| EmilienM | we can do it with hiera | 19:27 |
| EmilienM | we'll need to add https://github.com/rnelson0/puppet-hiera_resources though | 19:27 |
| EmilienM | I can prepare a demo and PoC | 19:27 |
| *** bootsha has quit IRC | 19:27 | |
| *** Jabadia has joined #tripleo | 19:29 | |
| *** jaimguer_ has joined #tripleo | 19:30 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: deploy composable firewall rules for HAproxy https://review.openstack.org/330785 | 19:30 |
| *** jaimguer_ has quit IRC | 19:30 | |
| EmilienM | dprince: I updated ^ for HAproxy and now I'm working on the PoC for keystone with hiera | 19:30 |
| Jabadia | last night I was able to use disk image build. now i cant. | 19:30 |
| Jabadia | I'm getting "failure: repodata/repomd.xml from openstack-kilo: [Errno 256] No more mirrors to try." | 19:31 |
| Jabadia | and indeed 'http://mirror.centos.org/centos/7Server/cloud/x86_64/openstack-kilo/repodata/repomd.xml' does not exists | 19:31 |
| Jabadia | it need to be s/7Server/7/g | 19:31 |
| *** padkrish has quit IRC | 19:32 | |
| Jabadia | is there a channel for DiB ? or this is the right place ? | 19:32 |
| *** jaimguer_ has joined #tripleo | 19:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-puppet-elements: Add puppet-hiera_resources to TripleO https://review.openstack.org/338512 | 19:35 |
| *** penick has joined #tripleo | 19:36 | |
| *** Jabadia has quit IRC | 19:41 | |
| *** rook-lappy has joined #tripleo | 19:43 | |
| *** padkrish has joined #tripleo | 19:44 | |
| *** Lokesh_Jain has quit IRC | 19:47 | |
| *** ramishra has quit IRC | 19:51 | |
| *** jaimguer_ has joined #tripleo | 19:52 | |
| *** ramishra has joined #tripleo | 19:52 | |
| *** ayoung has joined #tripleo | 19:56 | |
| *** jaimguer_ has quit IRC | 19:58 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Create firewall_rules from an Hiera Hash https://review.openstack.org/338526 | 20:00 |
| *** julim has quit IRC | 20:01 | |
| *** jaimguer_ has joined #tripleo | 20:02 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move Keystone firewall rules into the service https://review.openstack.org/338527 | 20:02 |
| EmilienM | dprince: boom ^ last 2 patches | 20:02 |
| EmilienM | dprince: pure hiera | 20:02 |
| *** julim has joined #tripleo | 20:05 | |
| *** jaimguer_ has quit IRC | 20:05 | |
| dprince | EmilienM: could you just do them all at once? It wouldn't be too large to do that way... especially since it is currently one big blog anyways | 20:05 |
| *** rcernin_ has joined #tripleo | 20:06 | |
| dprince | EmilienM: I hate to see us spend time cranking 20+ patches through CI when it could be a simple medium size one... | 20:06 |
| openstackgerrit | Merged openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 20:06 |
| *** jaimguer_ has joined #tripleo | 20:07 | |
| EmilienM | dprince: it's a PoC | 20:09 |
| *** jaimguer_ has quit IRC | 20:09 | |
| EmilienM | dprince: if you like it I can do them all in one man | 20:09 |
| jidar | man I'd kill for a validater that actually told me what was wrong with my templates | 20:09 |
| * jidar groans | 20:09 | |
| *** padkrish has quit IRC | 20:09 | |
| dprince | EmilienM: cool, lets see if it passes | 20:10 |
| dprince | EmilienM: +2 on the puppet-tripleo patch | 20:10 |
| EmilienM | dprince: ok I'll check logs & make sure it does what we want | 20:10 |
| *** rcernin_ has quit IRC | 20:10 | |
| EmilienM | dprince: let me submit another patchset with another service to see if merge workd | 20:11 |
| *** jaimguer_ has joined #tripleo | 20:11 | |
| *** jaimguer_ has joined #tripleo | 20:11 | |
| ayoung | openstack overcloud deploy is the same as heat stack-create <what>? | 20:11 |
| dprince | EmilienM: same patch would work too I think | 20:12 |
| dprince | EmilienM: just need to put it into another nested service stack | 20:12 |
| thrash | ayoung: there's a lot more to it than that | 20:12 |
| jidar | ayoung: you can run a --debug and get the http posts to heat | 20:12 |
| ayoung | thrash, maybe now, but I read somewhere that it was originally a heat-stack create | 20:12 |
| jidar | but yea, what thrash said | 20:12 |
| *** jaimguer_ has quit IRC | 20:12 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move Keystone and Neutron Server firewall rules into the service https://review.openstack.org/338527 | 20:13 |
| EmilienM | dprince: done ^ | 20:13 |
| thrash | ayoung: you *can* recreate it by hand. And as jidar said, you can run overcloud deploy with --debug, and see the api call that is made to heat | 20:13 |
| EmilienM | dprince: if ^ pass, please tell me it's good and I'll do all services in this same patch so we can easily switch | 20:13 |
| ayoung | jidar, thrash I have a fairly unstable system, and am unwilling to poke at it too hard. I'd like to be able to look at the existing stack and see how it was put together, first | 20:13 |
| ayoung | I'm running quickstart, and it adds a bunch of additional parameters | 20:14 |
| *** jaimguer_ has joined #tripleo | 20:14 | |
| *** jaimguer_ has quit IRC | 20:14 | |
| ayoung | is there any way to track back from running stack to template? | 20:14 |
| *** rcernin_ has joined #tripleo | 20:15 | |
| trown | ayoung: the deploy script in quickstart accepts additional args now... so you can run `overcloud-deploy.sh --debug` and "--debug" will be passed to the deploy command | 20:15 |
| EmilienM | dprince: btw, we'll still need https://review.openstack.org/#/c/330785/ because of the logic we have in manifests, but not a big deal | 20:15 |
| ayoung | trown, cool | 20:15 |
| trown | ayoung: assuming a relatively recent checkout of quickstart anyways | 20:15 |
| trown | otherwise you could always `vim overcloud-deploy.sh` and just add it | 20:16 |
| dprince | EmilienM: see my comment on https://review.openstack.org/#/c/330785/ | 20:16 |
| dprince | EmilienM: just put the check for 'manage_firewall' there | 20:16 |
| dprince | EmilienM: and I think we'd be good... | 20:16 |
| ayoung | trown, seems to be working, but the questions still stands: how do I work backwards, or do I? If I have a failiuing deploy, I want to figure out which template failed | 20:16 |
| EmilienM | dprince: ack thw | 20:17 |
| EmilienM | thx* | 20:17 |
| jidar | ayoung: there's a few commands you can run to figure out what's going wrong | 20:17 |
| jidar | ayoung: http://hardysteven.blogspot.com/2015/05/tripleo-heat-templates-part-3-cluster.html | 20:17 |
| trown | ayoung: if a software deployment failed, quickstart will even create that for you https://github.com/openstack/tripleo-quickstart/blob/master/roles/tripleo/overcloud/templates/overcloud-deploy.sh.j2#L61-L65 | 20:18 |
| jidar | then depending on what method you're deploying with you can either set a root password on the overcloud image and log in and check the logs, etc. | 20:18 |
| thrash | ayoung: and http://hardysteven.blogspot.com/2015/04/debugging-tripleo-heat-templates.html | 20:18 |
| trown | ayoung: so you could just check for a "failed_deployment_<UUID>.log" in /home/stack/ | 20:19 |
| *** rcernin_ has quit IRC | 20:19 | |
| *** rcernin_ has joined #tripleo | 20:19 | |
| ayoung | thrash, I know that page well :) | 20:19 |
| jidar | speaking of, is there a way to debug what the validation failed on? | 20:19 |
| trown | if something other than a StructuredDeployment failed, then it requires more sleuthing | 20:19 |
| jidar | ie: a deploy isn't actually run | 20:19 |
| *** rcernin_ has quit IRC | 20:20 | |
| jidar | all I've got is: >{"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "Failed to validate: Failed to validate: Invalid type (String)", | 20:20 |
| thrash | hmm | 20:20 |
| EmilienM | dprince: commented https://review.openstack.org/#/c/338526/1/manifests/firewall.pp | 20:20 |
| EmilienM | I have an improvement to do in the patch for performance | 20:21 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Create firewall_rules from an Hiera Hash https://review.openstack.org/338526 | 20:21 |
| EmilienM | dprince: ^ see | 20:21 |
| jmiu | aside from sshing into the node itself and looking at the logs, what else can you do? | 20:21 |
| *** jayg is now known as jayg|g0n3 | 20:22 | |
| ayoung | jidar, where do you see that message? | 20:22 |
| *** jaimguer_ has joined #tripleo | 20:22 | |
| dprince | EmilienM: cool. Then we are good. I don't think we'd need the one-off HAproxy patch then right? | 20:22 |
| jidar | ayoung: inside the --debug of a deploy | 20:22 |
| thrash | jidar: and a stack isn't even created in that instance, correct? | 20:22 |
| jidar | thrash: I'm re-running the deploy on an excisting cloud | 20:22 |
| EmilienM | dprince: yes we need | 20:23 |
| jidar | existing rather | 20:23 |
| thrash | jidar: any failed resources at that point? | 20:23 |
| EmilienM | dprince: because we can't compose the HAproxy rules in Hiera | 20:23 |
| jidar | thrash: it never kicks off | 20:23 |
| thrash | jidar: gotcha | 20:23 |
| jidar | ie: the validate failed, not the deploy | 20:23 |
| ayoung | jidar, that looks like an API call that failed. Any logging before it? | 20:23 |
| EmilienM | because we have some logic in haproxy.pp about what services are enabled | 20:23 |
| *** jaimguer_ has quit IRC | 20:23 | |
| jidar | ayoung: nothing helpful | 20:23 |
| thrash | jidar: or, better yet, what did you change? :D | 20:23 |
| dprince | EmilienM: okay, so I see. it is using variables | 20:23 |
| EmilienM | yes | 20:24 |
| jidar | hah! now that's the real question - I'm trying to get DeployIdentifer to work | 20:24 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Move Keystone and Neutron Server firewall rules into the service https://review.openstack.org/338527 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist https://review.openstack.org/319337 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Stop piping yes to Heat https://review.openstack.org/317730 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job https://review.openstack.org/296765 | 20:24 |
| jidar | like so: https://review.openstack.org/#/c/296488/1/extraconfig/post_deploy/example_run_on_update.yaml | 20:24 |
| *** rcernin has quit IRC | 20:24 | |
| *** jaimguer_ has joined #tripleo | 20:24 | |
| ayoung | jidar, ok, so am I right in understanding that most API calls are done on the controller, from the os-update-config [name?] daemon and that should have a log? | 20:24 |
| *** jaimguer_ has quit IRC | 20:24 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-puppet-elements: Add puppet-ec2api module https://review.openstack.org/336538 | 20:24 |
| jidar | the undercloud is going to be making calls to its own heat engine | 20:25 |
| *** rcernin has joined #tripleo | 20:25 | |
| ayoung | jidar, so that call was made from Heat itself? | 20:25 |
| *** rook-lappy has quit IRC | 20:25 | |
| jidar | that call was made TO heat | 20:25 |
| *** jaimguer_ has joined #tripleo | 20:26 | |
| ayoung | jidar, can you paste the log, with maybe the 50 lines before it, so I can see? I need to learn how to do this | 20:26 |
| jidar | it's obviously an issue with my templates, but figuring out which one or where is not really well explained | 20:26 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Run undercloud idempotency test on all jobs https://review.openstack.org/336112 | 20:26 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test https://review.openstack.org/279218 | 20:26 |
| jidar | I can not :( customer info | 20:26 |
| jmiu | what does doing a dry run on the template give? | 20:27 |
| jidar | this is an older version of kilo, on rhel - no --dry-run/--dryrun option I can pass in | 20:28 |
| *** jaimguer_ has joined #tripleo | 20:28 | |
| jmiu | oh :( | 20:29 |
| jidar | yea... tell me about it | 20:29 |
| jidar | you should see what I did to rebuild the lab to the same patch set as prod :P | 20:29 |
| jidar | I'll give you a hint, it involves 1000 packages on a single line installed by yum and unlinked the yum install script used by undercloud install | 20:30 |
| jmiu | :| ... | 20:30 |
| *** jaimguer_ has joined #tripleo | 20:32 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Record Heat resource deployment times to Graphite https://review.openstack.org/314308 | 20:33 |
| ayoung | OK, so I followed jidar and thrash 's advice and now I have UPDATE_FAILED | 20:33 |
| *** shivrao_ has joined #tripleo | 20:33 | |
| ayoung | so I've got that going for me | 20:33 |
| *** shivrao has quit IRC | 20:34 | |
| *** shivrao_ is now known as shivrao | 20:34 | |
| ayoung | http://paste.openstack.org/show/526619/ | 20:34 |
| *** jaimguer_ has joined #tripleo | 20:35 | |
| *** jaimguer_ has quit IRC | 20:35 | |
| *** jaimguer_ has joined #tripleo | 20:37 | |
| *** bootsha has joined #tripleo | 20:37 | |
| openstackgerrit | Ben Nemec proposed openstack/python-tripleoclient: Re-enable keystone init deprecation message https://review.openstack.org/290571 | 20:38 |
| *** rcernin has quit IRC | 20:38 | |
| *** rcernin has joined #tripleo | 20:39 | |
| *** jaimguer_ has joined #tripleo | 20:39 | |
| *** jaimguer_ has quit IRC | 20:41 | |
| *** jaimguer_ has joined #tripleo | 20:44 | |
| *** jaimguer_ has joined #tripleo | 20:45 | |
| *** jaimguer_ has joined #tripleo | 20:46 | |
| *** jeckersb is now known as jeckersb_gone | 20:47 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 20:48 |
| *** jaimguer_ has joined #tripleo | 20:49 | |
| *** jaimguer_ has quit IRC | 20:49 | |
| *** dprince has quit IRC | 20:49 | |
| *** jaimguer_ has joined #tripleo | 20:50 | |
| *** jaimguer_ has quit IRC | 20:50 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Add swift ringbuilder profile https://review.openstack.org/337803 | 20:51 |
| *** padkrish has joined #tripleo | 20:51 | |
| *** jaimguer_ has joined #tripleo | 20:51 | |
| *** shivrao_ has joined #tripleo | 20:51 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert Swift ringbuilder to composable services format https://review.openstack.org/338551 | 20:52 |
| *** jaimguer_ has joined #tripleo | 20:52 | |
| *** shivrao has quit IRC | 20:53 | |
| *** shivrao_ is now known as shivrao | 20:53 | |
| *** jaimguer_ has quit IRC | 20:53 | |
| *** lblanchard has quit IRC | 20:53 | |
| *** padkrish has quit IRC | 20:55 | |
| *** padkrish has joined #tripleo | 20:56 | |
| *** jaimguer_ has joined #tripleo | 20:57 | |
| *** Goneri has quit IRC | 20:57 | |
| *** anshul has quit IRC | 20:58 | |
| *** jaimguer_ has joined #tripleo | 20:59 | |
| *** jaimguer_ has joined #tripleo | 21:00 | |
| *** jaimguer_ has joined #tripleo | 21:02 | |
| *** jaimguer_ has joined #tripleo | 21:03 | |
| *** rcernin has quit IRC | 21:03 | |
| *** jaimguer_ has joined #tripleo | 21:04 | |
| *** rcernin has joined #tripleo | 21:04 | |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** jaimguer_ has quit IRC | 21:05 | |
| *** lucas|afk has quit IRC | 21:05 | |
| *** jcoufal has quit IRC | 21:07 | |
| *** julim has quit IRC | 21:07 | |
| *** rcernin has quit IRC | 21:07 | |
| *** rcernin has joined #tripleo | 21:07 | |
| *** jaimguer_ has joined #tripleo | 21:08 | |
| *** derekh has joined #tripleo | 21:08 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 21:08 |
| derekh | bnemec: something occured to me, see ^^ | 21:08 |
| derekh | bnemec: when testing rh2, I became paranoid that ARP traffic was leaking between tenant nodes | 21:09 |
| derekh | bnemec: it didn't always happen, but when it did, the deploy image failed to get an IP | 21:09 |
| *** bootsha has quit IRC | 21:10 | |
| bnemec | derekh: Hmm, interesting. | 21:10 |
| *** jaimguer_ has quit IRC | 21:10 | |
| bnemec | I'm pretty sure that won't work though. | 21:10 |
| *** jaimguer_ has joined #tripleo | 21:10 | |
| derekh | bnemec: the first thing that dhclient does is sent out an ARP request for the IP it was given over dhcp, it then stopped and just sat there until a timeout | 21:10 |
| bnemec | You can't randomly select the ip ranges that way. | 21:10 |
| derekh | bnemec: no? | 21:11 |
| *** bootsha has joined #tripleo | 21:11 | |
| * derekh thought it might work because we got arp spoofing turned off | 21:11 | |
| *** jaimguer_ has quit IRC | 21:11 | |
| bnemec | derekh: Oh wait, I'm reading that wrong. | 21:12 |
| derekh | bnemec: this was the main reason that I didn't select the default range for rh2 | 21:12 |
| *** jaimguer_ has joined #tripleo | 21:12 | |
| *** jaimguer_ has quit IRC | 21:12 | |
| bnemec | That will work fine. | 21:12 |
| *** lucasagomes has joined #tripleo | 21:12 | |
| bnemec | I was thinking the change randomized the last octet, but it doesn't. | 21:12 |
| derekh | bnemec: I never tracked down exactly what combination of a setup made it happen but I thought using a different range for rh2 would prevent it | 21:13 |
| derekh | bnemec: anyways, I'm going to recheck that a bunch of time to see if it reproduces or not | 21:13 |
| derekh | bnemec: just a hunch, but once based on something I observed a few weeks ago | 21:13 |
| bnemec | derekh: Yeah, sounds good. I feel like it _shouldn't_ make a difference, but bugs happen. | 21:13 |
| derekh | bnemec: ya, it shouldn't make a difference, it if it does its probably a neutron bug allowing traffic between tenant networks in some cases | 21:15 |
| *** bfournie has quit IRC | 21:15 | |
| derekh | bnemec: but we do have arp spoofing allowed and have turned off the firewall...who does that.. | 21:15 |
| bnemec | Yeah, OVB seems to be pretty good at finding OpenStack bugs. :-) | 21:15 |
| *** bootsha has quit IRC | 21:16 | |
| bnemec | derekh: NFV people, who are kind of important from what I hear. ;-) | 21:16 |
| *** dmsimard is now known as dmsimard|afk | 21:16 | |
| *** rhallisey has quit IRC | 21:16 | |
| *** jaimguer_ has joined #tripleo | 21:16 | |
| derekh | bnemec: they might have something to say about it | 21:16 |
| derekh | bnemec: if a few recheck work I'll make it a proper patch and we can see if it helps | 21:17 |
| *** jaimguer_ has quit IRC | 21:17 | |
| bnemec | derekh: Yeah, honestly I wouldn't mind doing it anyway. It would be nice to catch bad assumptions on our part about what CIDRs are being used simply because they're the default. | 21:18 |
| jidar | (╯°□°)╯︵ ┻━┻ | 21:18 |
| jidar | arrggggggg | 21:18 |
| derekh | bnemec: yup, thats also a thing | 21:19 |
| *** jaimguer_ has joined #tripleo | 21:20 | |
| *** jaimguer_ has quit IRC | 21:24 | |
| *** bfournie has joined #tripleo | 21:26 | |
| *** padkrish has quit IRC | 21:27 | |
| *** jaimguer_ has joined #tripleo | 21:28 | |
| *** jaimguer_ has joined #tripleo | 21:31 | |
| *** padkrish has joined #tripleo | 21:31 | |
| *** jaimguer_ has joined #tripleo | 21:34 | |
| *** jaimguer_ has quit IRC | 21:37 | |
| *** rcernin has quit IRC | 21:37 | |
| *** jaimguer_ has joined #tripleo | 21:38 | |
| *** jaimguer_ has quit IRC | 21:39 | |
| openstackgerrit | Merged openstack/instack-undercloud: Increase swift-proxy node_timeout https://review.openstack.org/338453 | 21:39 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove the rewrite of TOCI_JOBTYPE https://review.openstack.org/338566 | 21:42 |
| *** derekh has quit IRC | 21:43 | |
| *** yamahata has quit IRC | 21:44 | |
| *** myoung has quit IRC | 21:45 | |
| *** yamahata has joined #tripleo | 21:45 | |
| *** bootsha has joined #tripleo | 21:46 | |
| *** jaimguer_ has joined #tripleo | 21:47 | |
| *** jaimguer_ has joined #tripleo | 21:51 | |
| *** jaimguer_ has quit IRC | 21:54 | |
| *** jaimguer_ has joined #tripleo | 21:55 | |
| *** jaimguer_ has quit IRC | 21:57 | |
| *** jaimguer_ has joined #tripleo | 22:02 | |
| *** maeca1 has joined #tripleo | 22:06 | |
| *** jaimguer_ has joined #tripleo | 22:06 | |
| *** jaimguer_ has joined #tripleo | 22:09 | |
| *** jaimguer_ has quit IRC | 22:14 | |
| *** padkrish has quit IRC | 22:14 | |
| *** Goneri has joined #tripleo | 22:15 | |
| *** jaimguer_ has joined #tripleo | 22:16 | |
| *** padkrish has joined #tripleo | 22:17 | |
| *** jaimguer_ has quit IRC | 22:17 | |
| *** jaimguer_ has joined #tripleo | 22:20 | |
| *** david-lyle_ has joined #tripleo | 22:21 | |
| *** jaimguer_ has joined #tripleo | 22:22 | |
| *** jaimguer_ has joined #tripleo | 22:25 | |
| *** jaimguer_ has quit IRC | 22:27 | |
| *** jaimguer_ has joined #tripleo | 22:32 | |
| *** jaimguer_ has joined #tripleo | 22:32 | |
| *** jaimguer_ has joined #tripleo | 22:33 | |
| *** david-lyle_ is now known as david-lyle | 22:35 | |
| *** lblanchard has joined #tripleo | 22:35 | |
| *** jaimguer_ has joined #tripleo | 22:36 | |
| *** padkrish has quit IRC | 22:36 | |
| *** jaimguer_ has joined #tripleo | 22:37 | |
| *** jaimguer_ has joined #tripleo | 22:37 | |
| *** jaimguer_ has quit IRC | 22:37 | |
| *** penick has quit IRC | 22:38 | |
| *** jaimguer_ has joined #tripleo | 22:39 | |
| *** bfournie has quit IRC | 22:39 | |
| *** jaimguer_ has joined #tripleo | 22:40 | |
| *** fragatina has quit IRC | 22:40 | |
| *** jaimguer_ has quit IRC | 22:40 | |
| *** jaimguer_ has joined #tripleo | 22:41 | |
| *** jaimguer_ has quit IRC | 22:43 | |
| *** rook-lappy has joined #tripleo | 22:43 | |
| *** jaimguer_ has joined #tripleo | 22:44 | |
| *** cdearborn has quit IRC | 22:48 | |
| *** padkrish has joined #tripleo | 22:48 | |
| *** fragatina has joined #tripleo | 22:52 | |
| *** padkrish has quit IRC | 22:53 | |
| *** padkrish has joined #tripleo | 22:54 | |
| *** jaimguer_ has joined #tripleo | 22:54 | |
| *** jaimguer_ has quit IRC | 22:58 | |
| *** padkrish has quit IRC | 23:03 | |
| *** jaimguer_ has joined #tripleo | 23:05 | |
| *** bootsha has quit IRC | 23:06 | |
| *** jaimguer_ has joined #tripleo | 23:11 | |
| *** padkrish has joined #tripleo | 23:11 | |
| *** jaimguer_ has quit IRC | 23:11 | |
| *** rook has joined #tripleo | 23:13 | |
| *** jaimguer_ has joined #tripleo | 23:13 | |
| *** rook-lappy has quit IRC | 23:14 | |
| openstackgerrit | Keith Schincke proposed openstack/puppet-tripleo: Add RGW to the Ceph rgw profile. https://review.openstack.org/334081 | 23:14 |
| *** padkrish has quit IRC | 23:19 | |
| *** saneax_AFK is now known as saneax | 23:19 | |
| *** jaimguer_ has joined #tripleo | 23:20 | |
| *** rlandy has quit IRC | 23:21 | |
| *** jaimguer_ has joined #tripleo | 23:25 | |
| *** jaimguer_ has quit IRC | 23:28 | |
| *** jaimguer_ has joined #tripleo | 23:31 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 23:31 |
| *** jaimguer_ has quit IRC | 23:31 | |
| *** jaimguer_ has joined #tripleo | 23:32 | |
| *** rook has quit IRC | 23:38 | |
| *** padkrish has joined #tripleo | 23:41 | |
| *** padkrish has quit IRC | 23:42 | |
| *** rook has joined #tripleo | 23:43 | |
| *** jaimguer_ has joined #tripleo | 23:43 | |
| *** padkrish has joined #tripleo | 23:46 | |
| openstackgerrit | Merged openstack/diskimage-builder: Make ubuntu-core support releases https://review.openstack.org/294149 | 23:49 |
| *** padkrish has quit IRC | 23:50 | |
| *** padkrish has joined #tripleo | 23:53 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 23:55 |
| *** padkrish has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!