| *** tiswanso has joined #tripleo | 00:52 | |
| *** Marga_ has quit IRC | 00:53 | |
| *** shivrao has joined #tripleo | 01:02 | |
| *** tiswanso has quit IRC | 01:03 | |
| *** shivrao has quit IRC | 01:13 | |
| *** shivrao has joined #tripleo | 01:44 | |
| *** shivrao has joined #tripleo | 01:45 | |
| *** shivrao has quit IRC | 01:50 | |
| *** tiswanso has joined #tripleo | 02:00 | |
| openstackgerrit | Merged openstack/tripleo-common: Expose TENANT_STACK_DEPLOY_ARGS https://review.openstack.org/288161 | 02:13 |
|---|---|---|
| *** tiswanso has quit IRC | 02:15 | |
| *** dmacpher has joined #tripleo | 02:38 | |
| *** chlong has joined #tripleo | 02:45 | |
| *** tiswanso has joined #tripleo | 03:11 | |
| *** yuanying has quit IRC | 03:22 | |
| *** tiswanso has quit IRC | 03:24 | |
| *** yuanying has joined #tripleo | 03:40 | |
| *** yuanying has quit IRC | 03:44 | |
| *** shivrao has joined #tripleo | 03:48 | |
| *** shivrao has quit IRC | 03:53 | |
| *** ayoung has quit IRC | 03:54 | |
| *** yuanying has joined #tripleo | 04:08 | |
| *** links has joined #tripleo | 04:15 | |
| *** Marga_ has joined #tripleo | 04:17 | |
| *** tiswanso has joined #tripleo | 04:20 | |
| *** cwolferh has quit IRC | 04:20 | |
| *** tiswanso has quit IRC | 04:27 | |
| *** masco has joined #tripleo | 04:39 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Fix spurious = in dib-python readme https://review.openstack.org/286991 | 04:47 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix cloud-init-disable-resizefs README title https://review.openstack.org/286999 | 04:49 |
| openstackgerrit | Merged openstack/diskimage-builder: Fix spurious = in dib-python readme https://review.openstack.org/286991 | 04:59 |
| *** chlong has quit IRC | 05:12 | |
| *** oshvartz has quit IRC | 05:23 | |
| *** tiswanso has joined #tripleo | 05:24 | |
| *** saneax_AFK is now known as saneax | 05:25 | |
| *** chlong has joined #tripleo | 05:29 | |
| *** tiswanso has quit IRC | 05:29 | |
| *** trozet has quit IRC | 05:31 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make OpenStack service ports configurable in HAProxy https://review.openstack.org/287199 | 05:40 |
| *** jaosorior has joined #tripleo | 05:43 | |
| *** rcernin has joined #tripleo | 05:47 | |
| *** rcernin has quit IRC | 05:57 | |
| *** cmyster has joined #tripleo | 06:03 | |
| *** cmyster has quit IRC | 06:03 | |
| *** cmyster has joined #tripleo | 06:03 | |
| *** tiswanso has joined #tripleo | 06:25 | |
| *** tiswanso has quit IRC | 06:32 | |
| *** olap has joined #tripleo | 06:48 | |
| *** akuznetsov has joined #tripleo | 06:52 | |
| *** aufi has joined #tripleo | 06:56 | |
| *** oshvartz has joined #tripleo | 07:07 | |
| *** aufi has quit IRC | 07:07 | |
| *** aufi has joined #tripleo | 07:09 | |
| *** Marga_ has quit IRC | 07:17 | |
| marios | :( gerrit... | 07:20 |
| jaosorior | marios: What's up? | 07:21 |
| *** liverpooler has joined #tripleo | 07:21 | |
| *** dmacpher has quit IRC | 07:21 | |
| marios | jaosorior: slow/getting 503 proxy errors | 07:23 |
| *** ccamacho has joined #tripleo | 07:23 | |
| -openstackstatus- NOTICE: gerrit is going to be restarted due to bad performance | 07:25 | |
| *** ChanServ changes topic to "gerrit is going to be restarted due to bad performance" | 07:25 | |
| *** ChanServ changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 07:28 | |
| *** tiswanso has joined #tripleo | 07:29 | |
| *** chlong has quit IRC | 07:30 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/267523 | 07:32 |
| *** cmyster has quit IRC | 07:33 | |
| *** tiswanso has quit IRC | 07:41 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/267523 | 07:42 |
| *** leanderthal|afk is now known as leanderthal | 07:43 | |
| *** akuznetsov has quit IRC | 07:43 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6 https://review.openstack.org/270110 | 07:46 |
| *** olap has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/287318 | 07:55 |
| *** jprovazn has joined #tripleo | 07:57 | |
| *** ohamada has joined #tripleo | 07:58 | |
| *** akuznetsov has joined #tripleo | 08:00 | |
| *** ohamada has quit IRC | 08:01 | |
| *** xinwu has quit IRC | 08:01 | |
| *** akuznetsov has quit IRC | 08:01 | |
| *** akuznetsov has joined #tripleo | 08:01 | |
| *** akuznetsov has quit IRC | 08:02 | |
| *** ohamada has joined #tripleo | 08:02 | |
| *** xinwu has joined #tripleo | 08:02 | |
| *** fgimenez has joined #tripleo | 08:03 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Introduce a UpgradeScriptDeliveryWorfklow as part of tripleo upgrades https://review.openstack.org/289212 | 08:04 |
| *** dshulyak has joined #tripleo | 08:05 | |
| *** rdopiera has joined #tripleo | 08:07 | |
| *** mikelk has joined #tripleo | 08:07 | |
| *** xinwu has quit IRC | 08:07 | |
| *** ccamacho has quit IRC | 08:09 | |
| *** athomas has joined #tripleo | 08:16 | |
| *** ishant has joined #tripleo | 08:23 | |
| *** pcaruana has joined #tripleo | 08:25 | |
| *** hjensas has joined #tripleo | 08:27 | |
| openstackgerrit | Ishant Tyagi proposed openstack/os-collect-config: Add insecure option to the cfn collector https://review.openstack.org/284725 | 08:29 |
| *** ccamacho has joined #tripleo | 08:33 | |
| *** tiswanso has joined #tripleo | 08:37 | |
| *** ccamacho has quit IRC | 08:37 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Fix default IP addressed violating rfc5737 https://review.openstack.org/289221 | 08:42 |
| *** ccamacho has joined #tripleo | 08:42 | |
| *** ccamacho_ has joined #tripleo | 08:44 | |
| *** ccamacho has quit IRC | 08:47 | |
| *** ccamacho_ is now known as ccamacho | 08:47 | |
| *** bandini has quit IRC | 08:47 | |
| *** bandini has joined #tripleo | 08:48 | |
| *** tiswanso has quit IRC | 08:48 | |
| *** olap has joined #tripleo | 08:53 | |
| *** akrivoka has joined #tripleo | 08:56 | |
| *** jaosorior has quit IRC | 09:00 | |
| *** jaosorior has joined #tripleo | 09:00 | |
| *** gfidente has joined #tripleo | 09:01 | |
| *** paramite has joined #tripleo | 09:01 | |
| *** mbound has joined #tripleo | 09:05 | |
| *** akuznetsov has joined #tripleo | 09:06 | |
| *** akuznetsov has quit IRC | 09:06 | |
| gfidente | anyone wants to merge netiso in ci? https://review.openstack.org/#/c/288163/ | 09:10 |
| *** ccamacho has quit IRC | 09:10 | |
| *** ccamacho has joined #tripleo | 09:11 | |
| *** shardy has joined #tripleo | 09:11 | |
| *** ifarkas has joined #tripleo | 09:12 | |
| *** palexster has joined #tripleo | 09:15 | |
| jprovazn | gfidente: good morning | 09:16 |
| jprovazn | gfidente: do you have a sec/minute/hour for a couple of questions about IPv6? | 09:17 |
| jaosorior | gfidente: Was there a reason why the 192.0.2.0/24 network was used for the defaults of network isolation? | 09:20 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fixup swift device string to delimit the ipv6 address with [] https://review.openstack.org/267523 | 09:20 |
| gfidente | jprovazn, sure | 09:21 |
| gfidente | jaosorior, not sure I got your question right, which network you mean? | 09:21 |
| shardy | Morning all | 09:22 |
| shardy | Hey, I have a question re the ipv6 patches - we're just landing them to master right, not backporting to liberty? | 09:22 |
| jprovazn | gfidente: nova db sync is failing with http://paste.openstack.org/show/489508/ - shadower hit the same too, have you already seen this? | 09:22 |
| gfidente | shardy, we will backport to liberty | 09:23 |
| jaosorior | well, addresses such as the ControlPlaneDefaultRoute, EC2MetadataIp which are in the netiso templates use 192.0.2.X addresses. And I was just wondering if there's a reason for that | 09:23 |
| jaosorior | gfidente ^^ | 09:23 |
| gfidente | jprovazn, yep there is two more patches I added to the list in trello | 09:23 |
| gfidente | jprovazn, this is for the db_sync issues https://review.openstack.org/#/c/288813/ | 09:23 |
| gfidente | jprovazn, this you will also hit https://review.openstack.org/#/c/288826/ | 09:23 |
| jprovazn | gfidente: thanks, I will try with these | 09:24 |
| gfidente | jaosorior, oh the ctlplane is the network where the undercloud does baremetal provisioning | 09:24 |
| shardy | gfidente: That seems pretty risky from a regressions perspective - we'll have to ensure trown|outtypewww is aware | 09:24 |
| jprovazn | shadower: ^ | 09:24 |
| jaosorior | gfidente: ah, thanks, the thing is, I'm changings the network the undercloud does the provisioning on, cause of this bug https://bugs.launchpad.net/tripleo/+bug/1553222 . So yeah, seems I'll have to change that too. Thanks | 09:25 |
| openstack | Launchpad bug 1553222 in tripleo "Default undercloud control plane network violates rfc5737" [Undecided,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 09:25 |
| jprovazn | gfidente: and second one - on Fri I was hitting keepalived segfault issue because I didn't use pacemaker env file. Is keepalived expected to work, do we support non-pacemaker deployments? | 09:25 |
| gfidente | jprovazn, I'd say not initially no | 09:26 |
| gfidente | jaosorior, I see, unfortunate decision that was :( | 09:26 |
| gfidente | jaosorior, we probably have references to that subnet in the docs around too :() | 09:26 |
| jprovazn | gfidente: ack, should we update https://etherpad.openstack.org/p/tripleo-ipv6-support instructions - I don't see pacemaker env file included there | 09:27 |
| jprovazn | ? | 09:27 |
| gfidente | jprovazn, yes! | 09:27 |
| gfidente | thanks! | 09:27 |
| jprovazn | np | 09:27 |
| gfidente | shardy, regressions you mean breaking ipv4 deployments? | 09:28 |
| openstackgerrit | Moshe Levi proposed openstack/diskimage-builder: Add lshw package to ironic-agent https://review.openstack.org/289233 | 09:32 |
| *** jistr has joined #tripleo | 09:33 | |
| *** mkovacik has joined #tripleo | 09:33 | |
| shadower | jprovazn, gfidente: awesome, thanks! | 09:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make defautl addresses comply with rfc5737 https://review.openstack.org/289234 | 09:35 |
| *** panda has quit IRC | 09:40 | |
| *** olap has quit IRC | 09:40 | |
| *** panda has joined #tripleo | 09:40 | |
| *** olap has joined #tripleo | 09:42 | |
| *** mgould has joined #tripleo | 09:42 | |
| *** electrofelix has joined #tripleo | 09:44 | |
| *** derekh has joined #tripleo | 09:44 | |
| *** tiswanso has joined #tripleo | 09:45 | |
| *** dtantsur|afk is now known as dtantsur | 09:50 | |
| marios | gfidente: is there stable/liberty for the main v6 review? /me looks | 09:53 |
| gfidente | marios, no not yes | 09:53 |
| gfidente | *yet | 09:53 |
| gfidente | you can cherry-pick that if it's clean! | 09:53 |
| marios | gfidente: ah k... i want to make the stable/liberty for that swift-device one | 09:53 |
| gfidente | yeah I think we need to figure what stable/liberty misses to make the port of the main ipv6 patch clean | 09:54 |
| marios | gfidente: kk thanks man i'll revisit later today see if is done/if i have time to try the cherrypick myself | 09:54 |
| *** tiswanso has quit IRC | 09:57 | |
| dtantsur | Morning TripleOwl, happy monday: https://dl.dropboxusercontent.com/u/1730743/owls/owl-d16023ad6f67d3ca66722c195d18878e.jpg | 09:59 |
| *** jtomasek_ has joined #tripleo | 10:02 | |
| *** xinwu has joined #tripleo | 10:03 | |
| openstackgerrit | Merged openstack/tripleo-common: Adds override for the overcloud node user in upgrade-non-controller https://review.openstack.org/287953 | 10:05 |
| *** xinwu has quit IRC | 10:09 | |
| shadower | my Controller & Compute resource groups are stuck in DELETE_IN_PROGRESS. Any ideas on how to get them unstuck? | 10:12 |
| shadower | e.g. could restarting heat engine help? | 10:12 |
| dtantsur | shadower, sometimes repeating stack-delete several times helps :) | 10:12 |
| shadower | dtantsur: I've tried a few times, but I'll try some more :-) | 10:13 |
| dtantsur | lol | 10:13 |
| dtantsur | eventual consistency, y'know.. | 10:13 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove unsafe "unset" defaults https://review.openstack.org/288057 | 10:14 |
| shadower | shardy: any suggestions for a stuck stack-delete command? I'm not too comfortable with the "edit the database" hammer | 10:17 |
| shardy | shadower: Any clues as to why it's stuck? | 10:18 |
| dtantsur | shadower, btw, do you have problems deleting nova instances or what? | 10:18 |
| * dtantsur is not familiar with heat concepts | 10:19 | |
| *** ccamacho has quit IRC | 10:19 | |
| shadower | shardy, dtantsur: yeah it's the nova instances. I deleted them manually after that kept failing, but Heat's still stuck on deleting them (even though they're gone now) | 10:19 |
| shardy | dtantsur: that's a good question actually - if nova can't delete the instances, doing the delete via heat won't help | 10:19 |
| shadower | shardy: nova deleted them fine when I asked it to directly | 10:20 |
| shardy | shadower: heat should ignore any resources which 404 on delete | 10:20 |
| shardy | so I'd check the heat-engine log for other errors | 10:20 |
| shadower | nothing suspicious, it just keeps running the destroy step | 10:23 |
| shardy | https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L1425 | 10:24 |
| shardy | you should just hit that ignore_not_found path and return | 10:24 |
| shadower | yeah | 10:24 |
| shadower | I'll just edit the db | 10:25 |
| *** ccamacho has joined #tripleo | 10:27 | |
| *** tzumainn has joined #tripleo | 10:28 | |
| *** olap has quit IRC | 10:29 | |
| *** olap has joined #tripleo | 10:30 | |
| shardy | shadower: OK - it's be good to actually understand the bug tho | 10:31 |
| shadower | shardy: yeah I know, but I'm feeling the pressure to test & review the ipv6 stuff, too :-( | 10:32 |
| shadower | if it happens again, I'll debug it properly | 10:32 |
| shardy | kk, fair enough :) | 10:32 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Support network isolation without external nets https://review.openstack.org/287599 | 10:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use IP addresses compliant with rfc5737 https://review.openstack.org/289250 | 10:34 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Fix default IP addressed violating rfc5737 https://review.openstack.org/289221 | 10:34 |
| *** tzumainn has quit IRC | 10:34 | |
| *** jtomasek_ has quit IRC | 10:36 | |
| jaosorior | shardy, derekh: the above commits depends on each other, and I set the Depends-On flag for both... do you guys know if that will work for the CI to test those commits together? | 10:39 |
| dtantsur | gfidente, hi! could you please take a look at the liberty backport https://review.openstack.org/#/c/287713/ ? 1x +2, gate is fine | 10:39 |
| *** ishant has quit IRC | 10:40 | |
| shadower | shardy: okay so it may have been a neutron deletion issue instead: Heat tries to delete subnets that have ports with IP addresses in them | 10:41 |
| derekh | jaosorior: https://review.openstack.org/#/c/289221/ seems weird, when I click on the depends on it brings me back to itself | 10:43 |
| derekh | jaosorior: you mean (a needs b) and also (b needs a) ? | 10:43 |
| jaosorior | vittu | 10:44 |
| shardy | shadower: that's odd - it should delete all the ports first | 10:44 |
| jaosorior | copy paste problem | 10:44 |
| jaosorior | yeah, that should be the case | 10:44 |
| jaosorior | derekh: I missed up the SHAs | 10:45 |
| jaosorior | let me fix that | 10:45 |
| jaosorior | but yeah, they both depend on each other | 10:45 |
| shadower | shardy: I've seen that happen a few times throughout the years but it's quite inconsistent | 10:45 |
| derekh | jaosorior: never tried it that way, somebody mentioned one time using merge-with | 10:45 |
| derekh | jaosorior: or something like that instead of depends on | 10:45 |
| *** chlong has joined #tripleo | 10:45 | |
| shardy | shadower: you can look at the events output from tripleoclient and figure out if it's trying to do things in the wrong order | 10:46 |
| derekh | jaosorior: but haven't tried that either, worth looking for to see if it works, but I'm not sure exacactly of the wording | 10:46 |
| shardy | shadower: FWIW I did add overcloud deletes to our CI, but had to remove it again because it took too much time :( | 10:46 |
| *** athomas has quit IRC | 10:46 | |
| shadower | shardy: what's the command for that? | 10:46 |
| shadower | I know heat event-list, but dunno abotu tripleoclient | 10:47 |
| shardy | shadower: it outputs all the events by default to stdout | 10:47 |
| shardy | it's roughly the same as heat event-list -n5 overcloud | 10:47 |
| shadower | shadower: ah, I've been doing "heat stack-delete overcloud" | 10:47 |
| shadower | lol | 10:47 |
| shadower | shardy: ^ | 10:47 |
| shardy | shadower: No, that's fine | 10:47 |
| shardy | duh | 10:48 |
| shardy | I was thinking about the deployment, where we output of the events from tripleoclient | 10:48 |
| shardy | -enocoffee ;) | 10:48 |
| shadower | right | 10:48 |
| shadower | although I've deleted the ports manually and the stack is gone now. I'll look at the events next time this happens | 10:48 |
| shadower | but yeah, the error Heat reported was identical to me deleting the subnet manually -- before removing the ports | 10:49 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Fix default IP addressed violating rfc5737 https://review.openstack.org/289221 | 10:49 |
| *** tosky has joined #tripleo | 10:50 | |
| *** ccamacho has quit IRC | 10:51 | |
| *** ccamacho has joined #tripleo | 10:52 | |
| *** tiswanso has joined #tripleo | 10:53 | |
| *** dsneddon has joined #tripleo | 10:57 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: updating enable_ceph conditions for controller https://review.openstack.org/288060 | 10:59 |
| openstackgerrit | Merged openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector https://review.openstack.org/287713 | 11:00 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Add IP addresses compliant with rfc5737 https://review.openstack.org/289250 | 11:00 |
| *** tiswanso has quit IRC | 11:00 | |
| jaosorior | derekh: Well, at least in the documentation they highly discourage the usage of circular depends-on... And I didn't find that merge-with flag (or similar). If you find it or remember it at some point let me know. I have the feeling I will need it in subsequent changes. | 11:01 |
| *** ccamacho has quit IRC | 11:01 | |
| jaosorior | still looking for it though | 11:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix password issue with mysql address for ceilometer https://review.openstack.org/287970 | 11:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add a sample network-environment.yaml file to environments https://review.openstack.org/286144 | 11:08 |
| jaosorior | marios, jistr, gfidente: If there is a value in the puppet manifests being set. And that same value is also being set in the hieradata, what gets prioritized? | 11:11 |
| gfidente | jaosorior, it depends on which hiera | 11:12 |
| jaosorior | gfidente: puppet/hieradata/compute.yaml | 11:12 |
| gfidente | jaosorior, the static controller.yaml stuff is *after* tht | 11:12 |
| gfidente | oh sorry you're asking manifests and hiera or templates and hiera? | 11:12 |
| jaosorior | in a commit, there is a value in puppet/manifests/overcloud_compute.pp that is being set, and is also being set in puppet/hieradata/compute.yaml | 11:13 |
| gfidente | the one in pp preveals | 11:13 |
| jaosorior | I see | 11:13 |
| jaosorior | alright, thanks dude | 11:14 |
| gfidente | I was confused by tht vs hiera because we had similar conversation with jistr the other day | 11:14 |
| gfidente | then I figured we were talking about manifest vs hiera , not tht | 11:14 |
| jaosorior | gfidente: Ah, now I see you co-authored this https://review.openstack.org/#/c/270831/8 I +1ed it since it won't be a problem. But yeah, there is that same value being set in the hierdata | 11:15 |
| gfidente | oh man no problem, -1 that | 11:15 |
| gfidente | thanks for pointing it out | 11:15 |
| *** jcoufal has joined #tripleo | 11:15 | |
| jaosorior | dtanstur, I've actually tried it | 11:16 |
| jaosorior | your comment on this CR https://review.openstack.org/#/c/289221/ | 11:16 |
| jaosorior | and it gets correctly updated by the puppet manifests with the br-ctlplane updated | 11:17 |
| jaosorior | * dtantsur ^^ | 11:17 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: IPv4 network isolation testing for HA and Ceph https://review.openstack.org/288163 | 11:19 |
| gfidente | derekh, that was bold :) | 11:19 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Neutron settings https://review.openstack.org/289270 | 11:21 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Cinder settings https://review.openstack.org/289271 | 11:21 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Heat settings https://review.openstack.org/289272 | 11:21 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: puppet: allow config of ad-hoc Glance settings https://review.openstack.org/289273 | 11:21 |
| dtantsur | jaosorior, ok good, but please mention it in the commit message | 11:21 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Set swift replicas = min(device_count, replicas) https://review.openstack.org/289274 | 11:22 |
| jaosorior | dtantsur: I usually don't add test results in the commit message.. Do you mind if I just mention it in a comment in that CR? | 11:23 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Update VNI and TunnelID ranges. https://review.openstack.org/289276 | 11:23 |
| dtantsur | jaosorior, in case of upgrade implications, the note should be in the commit message IMO | 11:23 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: [WIP] Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 11:26 |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates: [WIP] Enable IPv4/IPv6 dual-stack Public API endpoints https://review.openstack.org/289279 | 11:27 |
| *** chlong has quit IRC | 11:29 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: DO NOT MERGE: Add network templates for multiple NIC configuration https://review.openstack.org/289286 | 11:38 |
| *** ccamacho has joined #tripleo | 11:41 | |
| *** chlong has joined #tripleo | 11:42 | |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Allow the containerized compute node to spawn larger VMs https://review.openstack.org/288822 | 11:46 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Remove unused Neutron Agents container https://review.openstack.org/287918 | 11:46 |
| *** masco has quit IRC | 11:47 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Change default bond-mode https://review.openstack.org/287603 | 11:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Adding ManagementIpSubnet to linux bridge net conf https://review.openstack.org/287602 | 11:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add Management Network For System Administration. https://review.openstack.org/264963 | 11:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add all isolated networks to all nodes. https://review.openstack.org/268833 | 11:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add network templates for multiple NIC configuration https://review.openstack.org/287600 | 11:48 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow for usage of pre-allocated IPs for the management network https://review.openstack.org/289289 | 11:48 |
| *** tiswanso has joined #tripleo | 11:56 | |
| *** jtomasek_ has joined #tripleo | 11:57 | |
| *** saneax is now known as saneax_AFK | 11:58 | |
| gfidente | guys what is going to happen with aodh in liberty? | 11:58 |
| gfidente | should we merge all tht/puppet changes for that or rather not? | 11:59 |
| gfidente | shardy, ^^ | 11:59 |
| gfidente | (backport) | 11:59 |
| *** lucasagomes is now known as lucas-hungry | 12:03 | |
| jprovazn | gfidente: it seems my compute has unset rabbit_hosts in /etc/nova/nova.conf which causes that nova-compute fails to start, I wonder if you hit this? | 12:05 |
| *** xinwu has joined #tripleo | 12:05 | |
| *** tiswanso has quit IRC | 12:06 | |
| jprovazn | shadower: ^ how is your rabbit_hosts doing on compute node? | 12:07 |
| gfidente | jprovazn, I got a working guest with the patches | 12:08 |
| gfidente | (in the overcloud) | 12:08 |
| * jprovazn wonders what's different :( | 12:08 | |
| gfidente | so you have it set on the controllers but not on the computes? | 12:09 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 12:09 |
| *** xinwu has quit IRC | 12:10 | |
| jprovazn | gfidente: good point. controller is unset too | 12:10 |
| gfidente | shit ? | 12:10 |
| jprovazn | I checked compute first because heat stack-create is stuck on it ATM | 12:10 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Enable IPA debug logging during introspection when undercloud_debug is True https://review.openstack.org/289297 | 12:11 |
| jprovazn | gfidente: just to confirm, I used the exacty same network environment yaml file and OC deploy command as is here: https://etherpad.openstack.org/p/tripleo-ipv6-support | 12:11 |
| *** trown|outtypewww is now known as trown | 12:12 | |
| gfidente | jprovazn, nah, let me update the pad | 12:13 |
| shardy | gfidente: I'd chat with EmilienM - my assumption was that we wouldn't backport all those patches to liberty though | 12:13 |
| jprovazn | shadower: ^ you will hit this too I guess | 12:14 |
| gfidente | shardy, yeah I'd be inclined to trop aodh in liberty too | 12:15 |
| gfidente | *drop | 12:15 |
| *** mburned_out is now known as mburned | 12:15 | |
| *** jtomasek_ has quit IRC | 12:16 | |
| gfidente | jprovazn, though I am not sure if/how that would change the rabbit_hosts thing | 12:16 |
| jprovazn | gfidente: what did yo uadd? -e $THT/environments/net-single-nic-with-vlans-v6.yaml? | 12:17 |
| gfidente | jprovazn, yep | 12:17 |
| jprovazn | gfidente: this is 14 patches I have applied on Friday afternoon, since then at least the biggest one was merged. There wasn't any fix/change in code itself though, was it? should I re-apply? | 12:18 |
| gfidente | jprovazn, nothing, it doesn't change anything | 12:18 |
| jprovazn | http://paste.openstack.org/show/489527/ | 12:18 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 12:19 |
| *** cmyster has joined #tripleo | 12:21 | |
| shadower | jprovazn, gfidente: my overcloud actually deployed, but the post-deployment steps couldn't reach keystone | 12:31 |
| jprovazn | shadower: nice | 12:31 |
| shadower | that's with gfidente's sdn and rabbit patches on top | 12:31 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Update typos https://review.openstack.org/289305 | 12:32 |
| gfidente | shadower, ok that's because the undercloud can't reach the keystone public address | 12:32 |
| shadower | ah ok | 12:32 |
| gfidente | shadower, let me paste something | 12:32 |
| gfidente | shadower, try this on the undercloud https://gist.github.com/gfidente/729fa398c368640c739a | 12:32 |
| gfidente | you should be able to ping the ping the public_virtual_ip from the undercloud then | 12:33 |
| shadower | gfidente: I can! | 12:33 |
| jaosorior | Hey guys, do you know if there are already any defined talks for tripleo on the openstack summit? | 12:34 |
| gfidente | jaosorior, I think dprince and shardy have one? | 12:34 |
| gfidente | shadower, so in theory if you re-deploy now ... | 12:35 |
| gfidente | :) | 12:35 |
| shadower | I'll try :-) | 12:35 |
| shadower | also time to show some love to the patches :-) | 12:36 |
| gfidente | shadower, try using the overcloud, things might just fail on stuff which puppet doesn't figure itself during configuration | 12:38 |
| shadower | yeah | 12:39 |
| shadower | gfidente: what's the difference (in puppet) between "include" and "class"? | 12:39 |
| shadower | I've seen you replace one with the another in the rabbitmq patch | 12:39 |
| gfidente | include sets the class param values to its default or hiera bindings | 12:41 |
| gfidente | class allows you to override some of those within the manifest | 12:41 |
| *** dprince has joined #tripleo | 12:42 | |
| *** weshay has joined #tripleo | 12:43 | |
| gfidente | marios, that looks like communication across the pacemaker nodes is unstable? this just changed in ci yes, it's using multiple nics on each vm to test netiso | 12:44 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make injected CA file readable by others https://review.openstack.org/289307 | 12:48 |
| jprovazn | gfidente: shadower: it's failing like a boss today :/ - http://paste.openstack.org/show/489529/ | 12:48 |
| jprovazn | I haven't hit this one yet though | 12:48 |
| *** rhallisey has joined #tripleo | 12:51 | |
| shadower | :-( | 12:53 |
| shadower | jprovazn: I'm having issues with Heat deleting stuff today | 12:54 |
| gfidente | jprovazn, shadower I remember something with the selinux context in which httpd was running? | 12:54 |
| *** ccamacho has quit IRC | 12:55 | |
| shadower | gfidente: yea me too but that was over a year ago... | 12:56 |
| *** jayg|g0n3 is now known as jayg | 12:56 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs https://review.openstack.org/269872 | 12:56 |
| shadower | though it's possible there's more than on instances of that | 12:56 |
| gfidente | I think jistr might help us here | 12:57 |
| * jistr scans the conversation | 12:57 | |
| gfidente | jistr, do you have any pointer to the systemd/selinux issue which was causing httpd start to fail? | 12:57 |
| *** cmyster has quit IRC | 12:59 | |
| *** colonwq has joined #tripleo | 12:59 | |
| *** cmyster has joined #tripleo | 13:01 | |
| *** cmyster has quit IRC | 13:01 | |
| *** cmyster has joined #tripleo | 13:01 | |
| jistr | that was because something in horizon/httpd wanted to use syslog, but systemd-journald was down due to some selinux-mislabeled file, so thing went bad. The paste doesn't look too familiar to me though. jprovazn is systemd-journald running on that system? | 13:01 |
| *** cmyster has quit IRC | 13:02 | |
| *** cmyster has joined #tripleo | 13:03 | |
| *** tiswanso has joined #tripleo | 13:03 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Remove Ceilometer Alarm from the overcloud https://review.openstack.org/288120 | 13:04 |
| jprovazn | jistr: I deleted the stack meantime :(, I'll tell when I hit it again | 13:05 |
| *** nico_auv has joined #tripleo | 13:09 | |
| *** lucas-hungry is now known as lucasagomes | 13:10 | |
| *** rbrady has joined #tripleo | 13:14 | |
| *** cmyster has quit IRC | 13:16 | |
| *** cmyster has joined #tripleo | 13:16 | |
| *** cmyster has quit IRC | 13:16 | |
| *** cmyster has joined #tripleo | 13:16 | |
| *** tiswanso has quit IRC | 13:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: loadbalancer: add Aodh API support https://review.openstack.org/288122 | 13:18 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/288130 | 13:21 |
| shadower | shardy: I hit the stack-delete issue again: http://paste.fedoraproject.org/335112/45735700/raw/ | 13:24 |
| gfidente | jistr, is the keystone/httpd thing going to liberty? | 13:24 |
| shadower | shardy: I misunderstood it before -- looks like it tries to delete the nested stacks under Compute & Controller RGs but then doesn't actually try deleting the resources under them | 13:25 |
| shardy | shadower: Are you running an older heat? | 13:25 |
| shadower | shardy: could be -- it's on a centos-based instack | 13:26 |
| shardy | shadower: Ah, if it's upstream instack based that should be fairly recent | 13:26 |
| shadower | shardy: $ heat-engine --version | 13:26 |
| shadower | 6.0.0 | 13:26 |
| shardy | shadower: there was a bug fixed a month or so ago where hooks weren't cleared properly and operations got stuck as a result | 13:27 |
| * shardy finds the patch | 13:27 | |
| *** links has quit IRC | 13:28 | |
| shardy | https://review.openstack.org/#/q/Ie894f416a898edeca3b6a123392853213893ff74,n,z | 13:28 |
| jprovazn | jistr: gfidente: after re-deployment I hit httpd issue again, to answer your prev question - yes systemd-journald is running | 13:28 |
| jistr | gfidente: no it's not AFAIK. It would need upgrades support. And i have the same concern about Ceilometer / AODH. | 13:28 |
| gfidente | jistr, ah good point about aodh | 13:28 |
| gfidente | EmilienM, ^^ | 13:29 |
| jistr | jprovazn, gfidente: hmm so it's probably something different than the selinux issue we had before | 13:29 |
| EmilienM | jistr: what concerns? | 13:29 |
| EmilienM | jistr: have you ping pradk about that? | 13:29 |
| shardy | shadower: it might be worth double checking there's no hooks around - I guess you could query the resource data in the DB just to be sure | 13:29 |
| *** ccamacho has joined #tripleo | 13:30 | |
| *** ohamada has quit IRC | 13:30 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add meta notify=true to rabbitmq resource https://review.openstack.org/285198 | 13:30 |
| *** ohamada has joined #tripleo | 13:30 | |
| jistr | EmilienM: missing migration logic for upgrades. Do we absolutely need Ceilo/AODH change in Liberty? At this point we need to cut off even the things we need, only the things that we *absolutely* need can stay :D | 13:30 |
| shadower | shardy: thanks, I do seem to have that in though (checked the source) | 13:30 |
| jistr | EmilienM: didn't ping him. Doesn't seem like he's around atm. | 13:31 |
| gfidente | jistr just set another channel topic to me | 13:32 |
| jistr | gfidente: :)) | 13:32 |
| jistr | gfidente: btw if we land this, we'll have a place where we can start putting the upgrade migration logic (removing 'delay' resource, eventlet->wsgi migration, ceilometer/aodh migration) https://review.openstack.org/#/c/285416/ | 13:33 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common: Change IPs used for the pingtest to comply with rfc5737 https://review.openstack.org/289331 | 13:33 |
| jistr | gfidente: thx! | 13:34 |
| *** dmacpher has joined #tripleo | 13:35 | |
| *** pradk has joined #tripleo | 13:35 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Function library for major upgrades https://review.openstack.org/285416 | 13:38 |
| *** xinwu has joined #tripleo | 13:39 | |
| *** panda has quit IRC | 13:39 | |
| jistr | pradk: hi, i wanted to verify with you, is it absolutely necessary to have the AODH change backported to Liberty? At this point we're trying to keep potentially invasive changes away from Liberty. If we don't have the AODH patch, will something break? | 13:40 |
| *** panda has joined #tripleo | 13:40 | |
| *** ishant has joined #tripleo | 13:41 | |
| pradk | jistr, Hi, So Aodh replaces ceilometer alarm in liberty which is deprecated. So if aodh doesnt make it into liberty we'll be shipping pretty much dead code and will incur a large backport cost in liberty. | 13:41 |
| shadower | shardy: good idea, thanks | 13:41 |
| pradk | jistr, so imho it is necessary | 13:42 |
| * shadower has been reading the scheduler code (where it seems stuck) in teh meantime but nothing useful yet | 13:42 | |
| jprovazn | gfidente: jistr: httpd is failing to start because haproxy is sitting on the port already - http://paste.openstack.org/show/489540/ | 13:42 |
| jprovazn | I would expect then that httpd has to listen on localhost port only | 13:43 |
| jprovazn | but *:8042 is set: | 13:43 |
| jprovazn | [root@overcloud-controller-0 systemd]# grep -r 8042 /etc/httpd/ | 13:43 |
| jprovazn | /etc/httpd/conf/ports.conf:Listen 8042 | 13:43 |
| jprovazn | /etc/httpd/conf.d/10-aodh_wsgi.conf:<VirtualHost *:8042> | 13:43 |
| shadower | shardy: do you happen to remember which tables we store hooks in? | 13:43 |
| shadower | oh you actually said resource data, sorry | 13:44 |
| jistr | pradk: ok. Just to give more context, the main concern here is about migrating existing deployments to the new model. E.g. despite a change that moves Keystone from eventlet to WSGI is done in Mitaka, we're not backporting it to Liberty because we miss the upgrade migration logic for this change. | 13:44 |
| *** jdob has joined #tripleo | 13:44 | |
| *** xinwu has quit IRC | 13:44 | |
| jistr | jprovazn: hmm ok that looks related to the AODH change we're just talking about /cc pradk | 13:45 |
| gfidente | jprovazn, so I think haproxy is meant to bind there, httpd to use a local ip | 13:45 |
| gfidente | and yes that's from the aodh changes we're trying to figure if should be ported to liberty | 13:45 |
| shadower | shardy: so, all I see in resource_data are credentials and keys, no hooks | 13:46 |
| pradk | jistr, i see, so we cannot start aodh with wsgi in liberty | 13:46 |
| pradk | ? | 13:46 |
| gfidente | jprovazn, it's interesting it didn't happen to me though or shadower ? | 13:46 |
| jprovazn | gfidente: yes, weird | 13:46 |
| jprovazn | might be reace issue maybe | 13:46 |
| jprovazn | gfidente: can you please try "grep -r 8042 /etc/httpd/" on your OC? | 13:47 |
| jprovazn | for shadower it pased because he doesn't have any aodh config under httpd at all | 13:47 |
| jprovazn | I wonder if I should pass an additional param when deploying stack? | 13:47 |
| gfidente | jprovazn, I was doing just that and I don't have it indeed | 13:48 |
| *** ishant has quit IRC | 13:48 | |
| gfidente | jprovazn, we're also using the same overcloud image right? | 13:49 |
| jprovazn | gfidente: only difference I know about is that shadower uses hiw own OC images, I use yours copied on Friday | 13:49 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test. https://review.openstack.org/281376 | 13:49 |
| *** tiswanso has joined #tripleo | 13:50 | |
| jistr | pradk: we could use wsgi, but for every change we do, we need to think about how to get an existing deployment into the new state. E.g. here we remove the pacemaker resources for ceilometer alarm from the manifest, which means they won't be created in a new deployment, but it doesn't mean that they will be removed on an already existing deployment. | 13:52 |
| jistr | pradk: https://review.openstack.org/#/c/288120/3 | 13:52 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Add a service provider. https://review.openstack.org/286124 | 13:55 |
| *** morazi has joined #tripleo | 13:55 | |
| *** tiswanso has quit IRC | 13:55 | |
| pradk | jistr, i understand now, thanks .. i can try to help with upgrade logic for ceilo to aodh if thats something we need for this to get into liberty | 13:55 |
| pradk | jistr, can you tell me what and where it needs to be done? | 13:56 |
| jistr | pradk: we just merged this change https://review.openstack.org/285416 where the functions for upgrade logic can go, and they'd then be called from | 13:58 |
| jistr | https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/major_upgrade_controller_pacemaker_1.sh | 13:58 |
| jistr | or https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/major_upgrade_controller_pacemaker_2.sh | 13:58 |
| jistr | as applicable | 13:58 |
| *** rlandy has joined #tripleo | 13:59 | |
| *** dustins has joined #tripleo | 14:00 | |
| jprovazn | EmilienM: it might be https://review.openstack.org/#/c/241408/ introduces a bug with httpd failing to start: http://paste.openstack.org/show/489540/ | 14:01 |
| gfidente | jistr, pradk, EmilienM I think the issue jprovazn just found out tells we should even revert it in master | 14:01 |
| jprovazn | thanks gfidente for pointing this out | 14:01 |
| thrash | trown: you ever see this or have an idea on how to fix? /tmp/in_target.d/install.d/60-ironic-agent-install: line 13: virtualenv: command not found | 14:02 |
| gfidente | jprovazn, you did all the things :) | 14:02 |
| jprovazn | gfidente: hah, sure - I made it fail :) | 14:02 |
| trown | thrash: you are hitting that building the IPA ramdisk? | 14:02 |
| thrash | trown: yes | 14:03 |
| gfidente | jprovazn, exactly it's always been like that | 14:03 |
| trown | thrash: seems like python-virtualenv is not installed | 14:03 |
| gfidente | I remember it | 14:03 |
| jprovazn | hehe | 14:03 |
| trown | thrash: but... why is IPA element needing a virtualenv? | 14:03 |
| thrash | trown: not really sure. | 14:04 |
| pino|work | (dib world, never known what nonsense you might find) | 14:04 |
| trown | lol pino|work :) | 14:04 |
| pino|work | (and yet the biggest nonsense is that people keep using it... "but that's none of my business") | 14:05 |
| pradk | gfidente, i think there is already a revert in progress in master, so apache is not binding to that port is the issue? | 14:05 |
| pradk | gfidente, interesting i dont see that locally | 14:06 |
| gfidente | pradk, apache is binding on the port but it's in use by haproxy | 14:06 |
| gfidente | pradk, I think we need to pass via hieraconfig the virtualhost ip | 14:06 |
| gfidente | pradk, like we do already for horizon | 14:07 |
| pradk | gfidente, thought we already did that.. lemme check | 14:07 |
| *** cmyster has quit IRC | 14:07 | |
| gfidente | pradk, but even then it's not working as expected because jprovazn gets *:8042 | 14:07 |
| thrash | trown: I looks like it's doing the source install of ironic agent and not package. | 14:07 |
| trown | thrash: ya that would not be what we want :) | 14:07 |
| thrash | trown: let me figure out why that is happening. | 14:08 |
| *** cmyster has joined #tripleo | 14:09 | |
| *** cmyster has quit IRC | 14:09 | |
| *** cmyster has joined #tripleo | 14:09 | |
| *** lblanchard has joined #tripleo | 14:09 | |
| pradk | gfidente, the *:8042 is what puppet configures the apache vhost config to be | 14:10 |
| thrash | trown: maybe this? Missing package name for distro/element: centos7/ironic-agent | 14:10 |
| pradk | gfidente, so if we want that to be localhost bind only, then we'll have to update puppet-aodh to use the right template | 14:11 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Remove unused Neutron Agents container https://review.openstack.org/287918 | 14:11 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:13 |
| trown | thrash: ya, I guess it is falling back to source because that package is missing? All of this could be avoided by starting with base images that have repos already setup | 14:14 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 14:14 |
| *** tzumainn has joined #tripleo | 14:16 | |
| *** Marga_ has joined #tripleo | 14:17 | |
| gfidente | pradk, not localhost only, we need to be able to configure the binding address ... it depends on a few factors when doing network isolation | 14:19 |
| *** absubram has joined #tripleo | 14:21 | |
| *** absubram_ has joined #tripleo | 14:23 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 versions of the Controller NIC configs https://review.openstack.org/269883 | 14:24 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add IPv6 Support to Isolated Networks https://review.openstack.org/289355 | 14:24 |
| *** jpeeler has joined #tripleo | 14:27 | |
| *** absubram has quit IRC | 14:27 | |
| *** absubram_ is now known as absubram | 14:27 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Ensure access to Redis is password protected https://review.openstack.org/210405 | 14:27 |
| *** apetrich has quit IRC | 14:28 | |
| *** tiswanso has joined #tripleo | 14:29 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Revert "Deploy Aodh services, replacing Ceilometer Alarm" https://review.openstack.org/288714 | 14:31 |
| *** absubram has quit IRC | 14:31 | |
| *** jaosorior is now known as jaosorior_climbi | 14:32 | |
| gfidente | pradk, ^^ so it was reverted, can you pick it up again so we can test it more? | 14:38 |
| openstackgerrit | Nisha Agarwal proposed openstack/diskimage-builder: Add psmisc to the packages for ironic-agent https://review.openstack.org/289364 | 14:40 |
| *** apetrich has joined #tripleo | 14:41 | |
| *** tiswanso has quit IRC | 14:43 | |
| jprovazn | gfidente: so revert of the patch solved httpd issue, unfortunately I'm hitting again the issue from this morning - rabbit_hosts is unset both on controller and compute nodes | 14:44 |
| gfidente | jprovazn, which sounds like glance-api thing? | 14:45 |
| jprovazn | gfidente: I'm hitting it with nova, but I also have the 2 patches you linked today, applied | 14:46 |
| gfidente | no I mean sounds like could be due to that patch | 14:46 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make the Neutron subnet ipv6_{ra,address}_mode configurable https://review.openstack.org/271208 | 14:52 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Allow the containerized compute node to spawn larger VMs https://review.openstack.org/288822 | 14:54 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-heat-templates: Remove unused Neutron Agents container https://review.openstack.org/287918 | 14:54 |
| *** trozet has joined #tripleo | 14:55 | |
| *** pradk_ has joined #tripleo | 14:55 | |
| EmilienM | hello, would it be possible to have a review on puppet-tripleo for Ipv6 dual stack ? https://review.openstack.org/#/c/286344/ | 14:55 |
| *** rpothier has joined #tripleo | 14:55 | |
| *** pradk has quit IRC | 14:56 | |
| *** ohamada has quit IRC | 14:56 | |
| *** pradk_ is now known as pradk | 14:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge https://review.openstack.org/286732 | 14:56 |
| *** dustins_ has joined #tripleo | 14:59 | |
| *** dustins_ is now known as dustins|away | 14:59 | |
| *** jprovazn has quit IRC | 15:00 | |
| EmilienM | slagle or dprince : do we have a matrix / document that explain what our CI tests? AFIK we did lot of changes lately | 15:00 |
| *** dustins has quit IRC | 15:01 | |
| slagle | we don't have a document | 15:02 |
| dprince | EmilienM: I'm not sure we maintain an official matrix document. We could add one into tripleo-ci if it helps | 15:02 |
| EmilienM | no problem, I was just asking | 15:03 |
| EmilienM | dprince: so now we have netiso for ceph and ha jobs: what plans we have for ipv6? | 15:03 |
| *** ayoung has joined #tripleo | 15:03 | |
| EmilienM | I like slagle's proposal on the ML about reducing the jobs in check queue | 15:04 |
| dprince | EmilienM: I would like to work on that w/ gfidente | 15:04 |
| dprince | EmilienM: I replied to your thread. I'm not a fan of dropping nonha at this time | 15:04 |
| EmilienM | dprince: sure, it was just an idea but sounds like a bad one | 15:04 |
| dprince | EmilienM: IMO there are good reasons to have it. And also it has proven to be more stable than the HA job | 15:04 |
| dprince | EmilienM: IPV6 will come shortly. We've got the architecture sorted out for that so that once IPv6 works adding it into CI should be easy | 15:05 |
| EmilienM | dprince: slagle also kind of agreed with me on non-ha job | 15:05 |
| dprince | EmilienM: The #1 way to help at this point with CI is getting CI on upgrades | 15:06 |
| dprince | EmilienM: that will unblock many features that will allow us to improve other various areas in CI | 15:07 |
| EmilienM | I'm afraid upgrade testing will consume lot of resources | 15:07 |
| EmilienM | but I agree it's very important | 15:07 |
| dprince | EmilienM: which is why the periodic image build caching work derekh is pursuing is of interest | 15:08 |
| EmilienM | dprince: do we have plans to increase overcloud RAM? 4GB sounds like causing problems | 15:09 |
| trown | dprince: maybe it would change if we did not have a job for it, but I have never seen the nonHA job fail legitimately when the HA job passed | 15:10 |
| EmilienM | shardy: if I plan to patch split stack, should I use split_stack2 Gerrit topic? | 15:10 |
| dprince | EmilienM: we can look into it. We would need to reallocate things for sure to accodidate it | 15:10 |
| shardy | EmilienM: sure, that was the series which I was planning to do the t-h-t refactoring on | 15:11 |
| dprince | trown: I'd still want a separate HA job to ensure single node HA works anways | 15:12 |
| *** cmyster has quit IRC | 15:14 | |
| trown | dprince: right, all I am pointing out is that if the failure rate module HA passing is near zero, we could do that in a periodic job | 15:14 |
| trown | dprince: I think that is what slagle was getting at | 15:14 |
| dprince | EmilienM, trown, slagle: I get the interest in consolidating our CI. But I gotta say... as one who helps deploy our baremetal resources I'm not in favor of complicating the TripleO CI rack w/ pacemaker | 15:14 |
| trown | s/module/modulo/ | 15:14 |
| *** cmyster has joined #tripleo | 15:14 | |
| *** cmyster has quit IRC | 15:14 | |
| *** cmyster has joined #tripleo | 15:14 | |
| trown | dprince: isnt the TripleO CI rack using Icehouse? | 15:15 |
| trown | or maybe Juno? | 15:15 |
| slagle | dprince: i didnt know the tripleoci rack was using updated templates to be honest | 15:15 |
| shardy | dprince: Hey, when you get a sec I wanted to clarify your plan ref https://review.openstack.org/#/c/288626/1/scripts/tripleo.sh | 15:15 |
| slagle | if it is, that's certainly a good reason to not drop CI coverage of single node | 15:16 |
| dprince | trown, slagle: it is old. I'm talking about when we rebuild it | 15:16 |
| slagle | still, we'd have a periodic job | 15:16 |
| trown | ya, I think periodic jobs can tell us if we need check jobs | 15:16 |
| dprince | trown: I'd still like more than one job running that accurately represents the developer case. Most developers can't use 3 controller nodes all the time. | 15:17 |
| trown | ie, single HA check job, if periodic jobs start failing based on bad merges that passed HA check, we can revisit | 15:17 |
| dprince | trown: single node HA (a separate job for that) would be a required case too I think regardless | 15:17 |
| dprince | I don't think just periodic is enough for that | 15:17 |
| shardy | dprince: I was initially looking at changing the puppet-modules element, so it'd support setting a global reporef, e.g to stable/liberty or whatever | 15:17 |
| shardy | https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/puppet-modules/environment.d/01-puppet-modules-install-types.sh | 15:17 |
| shardy | but that doesn't work because we mix up puppetlabs and openestack modules there | 15:18 |
| shardy | are you proposing I break that element into two? | 15:18 |
| shardy | Or can we just tag another element on which overrides the branches pulled by the main element? | 15:19 |
| dprince | shardy: we could do that if it makes sense. I'm really open to whatever here and I'm not actually meaning to block your tripleo-ci change. Just asking the question: what if we did this via elements instead? | 15:19 |
| dprince | shardy: I suppose I was thinking of adding another element which overrides the specific values | 15:20 |
| dprince | shardy: that was my initial thought anyways (your last response) | 15:20 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-docs: Document using node capabilities to control placement https://review.openstack.org/274217 | 15:20 |
| shardy | dprince: Ok, I can definitely give that a try | 15:20 |
| rhallisey | shardy, ^ updated your doc and added to it | 15:20 |
| shardy | dprince: for CI, all we actually need is to force building from source, because all the repos are there and reset to stable/liberty | 15:20 |
| shardy | so this is only for developer convenience really | 15:21 |
| *** olap has quit IRC | 15:21 | |
| shardy | rhallisey: thanks! | 15:21 |
| dprince | shardy: cool. For developers I think using this element would work well | 15:22 |
| dprince | shardy: also, for developers we might go a totally different approach and have a Puppetfile (or shell script, etc.) to help stage all the liberty modules. And then use the artifact deployment to dynamically deploy them into the overcloud | 15:24 |
| dprince | shardy: now that we have puppet modules via swift, I'd like to consider the option to move away from using DIB as a means to deploy the puppet modules | 15:24 |
| shardy | dprince: Yup, we could definitely look at that too, this was really just a first step to move us away from opm for liberty - I'm trying the element approach now, thanks! | 15:30 |
| trown | shardy: dprince, doesnt the whole path of puppet modules from source go away once we have a package per puppet module? | 15:34 |
| trown | dprince: shardy at that point arent the puppet packages just another package? | 15:35 |
| dprince | trown: no, I don't think it does. Turns out packages aren't how most developers (and some opperators) like to consume puppet | 15:36 |
| dprince | trown: we can support both I think | 15:36 |
| openstackgerrit | Javier Peña proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6 https://review.openstack.org/287068 | 15:37 |
| trown | dprince: if the delorean puppet packages are just packaged source with no patches... does that change? | 15:37 |
| trown | dprince: I get not wanting to consume a monolithic package of puppet modules with no clear origin story for the individual modules, but it seems like we want to get rid of the special case of puppet modules | 15:38 |
| dprince | trown: not for me :) | 15:38 |
| trown | dprince: using them from source leaves it as a special case | 15:39 |
| shardy | trown: that will certainly help, but I think ref the deploy artefacts thing, some folks would still prefer a local tree vs packages | 15:39 |
| *** dustins|away is now known as dustins | 15:39 | |
| dprince | trown: forcing packages for all puppet users in TripleO IMO is a step sideways. Not everyone wants it | 15:39 |
| dprince | trown: we support both. Lets just go with that | 15:40 |
| shardy | dprince: not everyone wants to deploy from source either tho, e.g in the case where folks consume tripleo and the puppet modules from a distributor | 15:40 |
| shardy | +1 :) | 15:40 |
| trown | dprince: ok, I am thinking alot about unifying the CI between RDO and TripleO, and that is where I am coming from with this | 15:40 |
| shardy | FWIW I think anything which reduces our reliance on elements is a good thing | 15:41 |
| trown | shardy: +1 | 15:41 |
| dprince | shardy: right, for CI I'd rather see us use swift deployment to do that | 15:41 |
| derekh | dprince: wouldn't it be nice though it we could be more consistent with what we are deploying with, i.e. have the puppetmodule and core package come from the same pinned repository? | 15:41 |
| trown | elements are very opaque | 15:41 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 15:41 |
| dprince | derekh: yes, we can do that with swift modules deployment though right? | 15:42 |
| shardy | dprince: if everything comes from a yum repo, why do we need the swift transport, vs building the packages into the image, or updating them via yum pointing at the repo? | 15:42 |
| shardy | I get it for the from-source case, but I'm not clear on the advantage in the packaged case | 15:43 |
| openstackgerrit | Miles Gould proposed openstack/python-tripleoclient: Use Ironic API v1.11 to support ENROLL state https://review.openstack.org/272206 | 15:43 |
| dprince | shardy, derekh: okay, sure. I'm fine with that angle too | 15:43 |
| dprince | shardy, derekh: I don't think that is very helpful for people who will actually develop t-h-t -> puppet | 15:43 |
| dprince | it is a lot of work to build packages for each little change | 15:43 |
| dprince | again, I would like CI to represent both cases I think | 15:44 |
| derekh | dprince: yes, we could I suppose (havn't tried it), what I mean is i'd like to see us approach the situation where where everything is being deployed from a trunk repository and the only think we use from git is the projects we are actually testing | 15:44 |
| shardy | dprince: sure, for developers the swift from-source workflow makes a lot of sense | 15:44 |
| shardy | although running a local delorean repo with source isn't that hard, it is added complexity I guess | 15:44 |
| trown | dprince: the nice thing with that approach, is that the 'current-tripleo' or hopefully just the 'current-passed-ci' repo would be totally fixed. currently puppet changes could break that repo since they are external | 15:44 |
| dprince | derekh: I had envisioned installing puppet modules (from packages) on the undercloud. And then deploying those via t-h-t via Swift to the overcloud. | 15:44 |
| shardy | dprince: for non-developer usage, that has the disadvantage that you lose versioning of the deployed modules on the overcloud nodes | 15:45 |
| shardy | which is probably difficult from a support standpoint when something breaks ;) | 15:45 |
| dprince | shardy: we can test both I think. Again, I think most puppet users would use their own Puppetfile (or equivalent) rather than packages. | 15:46 |
| shardy | yup, supporting both sounds like a good plan | 15:46 |
| derekh | dprince: I'd be fine with that, I just want us to stop looking at git repositories for our default deployment, so long as they originate from a yum repository that would be fine | 15:46 |
| trown | I think we could create tooling (in tripleo-quickstart or elsewhere) that makes delorean usage really simple | 15:46 |
| dprince | shardy: that is my take anyways. I think the distro angle is fine, I just think it is perhaps too much of a RedHat'ism to state it is the most common/useful case. There are advanced operators who won't like puppet being packaged at all | 15:47 |
| trown | tripleo.sh already does IMHO | 15:47 |
| *** dprince has quit IRC | 15:49 | |
| *** jprovazn has joined #tripleo | 15:49 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use MysqlVirtualIPUri for nova_api and sahara database https://review.openstack.org/288813 | 15:51 |
| *** ohamada has joined #tripleo | 15:54 | |
| openstackgerrit | Javier Peña proposed openstack/tripleo-heat-templates: Fix vncproxy_host for IPv6 https://review.openstack.org/287068 | 15:54 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list https://review.openstack.org/286062 | 15:58 |
| *** oshvartz has quit IRC | 16:00 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test. https://review.openstack.org/281376 | 16:00 |
| *** lazy_prince has joined #tripleo | 16:01 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Add a service provider. https://review.openstack.org/286124 | 16:01 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Make the Neutron subnet ipv6_{ra,address}_mode configurable https://review.openstack.org/289417 | 16:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync https://review.openstack.org/267073 | 16:02 |
| *** liverpooler has quit IRC | 16:03 | |
| *** Goneri has quit IRC | 16:06 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Allow to enable IPv6 on Corosync https://review.openstack.org/289422 | 16:08 |
| jprovazn | gfidente, what is your heat version? | 16:11 |
| dtantsur | lucasagomes, could you please take one more look at https://review.openstack.org/288417 ? | 16:12 |
| lucasagomes | dtantsur, hi there, sure | 16:12 |
| * lucasagomes looks | 16:12 | |
| jprovazn | shardy, do you know if openstack-heat-engine-5.0.1-3.el7ost.noarch is sufficient version for running current master tripleo-heat-templates? | 16:14 |
| shardy | jprovazn: not sure tbh, but you will require this fix: | 16:15 |
| shardy | https://review.openstack.org/#/q/Ib934f443a8b8e4f75335a9d8b992e7f86791aa45,n,z | 16:15 |
| jprovazn | shardy, I have that one applied, I wonder how it could be that rabbit_hosts is empty both on my compute and ctontroller nodes | 16:17 |
| jprovazn | and I noticed a new syntax in tht - nova::rabbit_hosts: *rabbit_nodes_array | 16:17 |
| jprovazn | at least I was not faimilar with the asterisk reference | 16:17 |
| shardy | jprovazn: that's a yaml alias, not something specific to heat | 16:18 |
| jprovazn | ah | 16:18 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet https://review.openstack.org/289427 | 16:18 |
| shadower | shardy: So during stack-delete a controller has a call to {get_attr: [TenantPort, ip_subnet]} which ends up going through: https://github.com/openstack/tripleo-heat-templates/blob/master/network/ports/tenant.yaml#L56 and that results to a string split on a None which in turn results in this exception: http://paste.openstack.org/show/489561/ | 16:18 |
| *** Goneri has joined #tripleo | 16:19 | |
| shadower | shardy: regarding my overcloud delete voes | 16:19 |
| shardy | shadower: Hmm, why is it splitting on None at delete time? | 16:20 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet https://review.openstack.org/276127 | 16:20 |
| shadower | shardy: yeah, that I don't know | 16:20 |
| shadower | could it be it's running some sort of validation? | 16:20 |
| shardy | shadower: it may still be a heat bug tho, if we created it, we should delete it | 16:21 |
| shadower | yeah | 16:21 |
| jprovazn | gfidente, ha, so the reason is that rabbit_node_ips parameter passed to the heat nested stack puppet/all-nodes-config.yaml is already empty | 16:21 |
| shardy | shadower: can you raise a heat bug please - I'm not sure why we're doing th enforce_stack thing on delete | 16:22 |
| *** dprince has joined #tripleo | 16:22 | |
| shadower | shardy: will do | 16:22 |
| shardy | need to dig deeper, but it seems like we shouldn't be doing that preview_resources on delete | 16:22 |
| shadower | shardy: yeah I was surprised to see it there but it's been a while I hacked on heat | 16:23 |
| lucasagomes | dtantsur, left a comment re overwrite | 16:23 |
| lucasagomes | lemme know what you think | 16:23 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix rabbit_hosts list for glance-api for IPv6 https://review.openstack.org/288826 | 16:24 |
| dtantsur | lucasagomes, actually I was planning this to be a normal situation (see the comment) | 16:26 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Fix rabbit_hosts list for glance-api for IPv6 https://review.openstack.org/289432 | 16:26 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Deploy Logging on the undercloud with Puppet https://review.openstack.org/258653 | 16:29 |
| lucasagomes | dtantsur, will take a look | 16:29 |
| *** jaosorior_climbi is now known as jaosorior | 16:29 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/289435 | 16:30 |
| openstackgerrit | Merged openstack/instack-undercloud: Enable notifications on undercloud https://review.openstack.org/241341 | 16:31 |
| jaosorior | bnemec: well, this did work with the upgrade https://review.openstack.org/#/c/289221/ only thing is that when upgrading, this needs to be taken into consideration to set some parameters for the overcloud (if needed) | 16:31 |
| gfidente | bnemec, ack on https://review.openstack.org/210405 | 16:32 |
| jaosorior | bnemec: Or at least it did work when going from a deployment with the old range, to do an $ openstack undercloud isntall that has the new range | 16:32 |
| gfidente | marios, jistr ^^ do you guys have any way to check https://review.openstack.org/210405 ? | 16:32 |
| *** absubram has joined #tripleo | 16:33 | |
| gfidente | I think now that we do pcmk restart it should be okay but ... | 16:33 |
| *** aufi has quit IRC | 16:33 | |
| lucasagomes | dtantsur, re-commented... I was just thinking about someone reconfiguring the node | 16:34 |
| bnemec | jaosorior: It can't work with upgrades when you have a deployed overcloud. When you re-ran the undercloud install it would try to re-create the subnet and fail because ports are still allocated. I think anyway. | 16:34 |
| lucasagomes | dtantsur, in case a new disk is added so he wants to refresh the root device hints or situations like that | 16:34 |
| *** mkovacik has quit IRC | 16:35 | |
| bnemec | gfidente: That's fine, I just want someone to test that it doesn't break upgrades before we merge it. | 16:35 |
| dtantsur | lucasagomes, the last thing we want to do is to silently change the root device if a new disk is added, to be honest | 16:35 |
| gfidente | bnemec, agreed | 16:35 |
| jaosorior | bnemec: well, crap... isn't there a way to get the undercloud to re-allocate those addresses? | 16:35 |
| marios | gfidente: i just tried to apply it to my 7.1 env but has merge conflict and really can't handle vimdiff right now | 16:36 |
| bnemec | jaosorior: Not that I know of. | 16:36 |
| lucasagomes | dtantsur, it will only change if it's requested right? | 16:36 |
| lucasagomes | if one do not pass the --root-device-hints we won't touch it | 16:36 |
| jaosorior | bnemec: Do you have any other ideas? The choosing of those addresses was quite unfortunate :/ | 16:37 |
| lucasagomes | or will we? | 16:37 |
| gfidente | marios, to 7.1? | 16:37 |
| gfidente | marios, shouldn't it go into 8 only? | 16:37 |
| marios | gfidente: in which case i can't help you my envs are all 7.x | 16:38 |
| bnemec | jaosorior: I don't disagree. I also hate that the provisioning network is called ctlplane, but it's not trivial to change that either. :-/ | 16:38 |
| jaosorior | bummer :/ | 16:38 |
| dtantsur | lucasagomes, we won't. changing the root device on rebuild without cleaning is a receipt for problems | 16:38 |
| jaosorior | well, lets talk about the ip addresses tomorrow in the meeting as you suggested | 16:38 |
| marios | gfidente: (well 7.3 which is being upgraded to 8 but can't use that /is in progress) | 16:38 |
| *** xinwu has joined #tripleo | 16:39 | |
| bnemec | jaosorior: Yeah, somebody else may know something we don't. :-) | 16:39 |
| *** yamahata has quit IRC | 16:39 | |
| gfidente | marios, so I think the problem is understanding if landing that change in liberty makes an update from kilo to succeed | 16:40 |
| thrash | bnemec: I want to say that was fixed... | 16:40 |
| lucasagomes | dtantsur, right, maybe I'm out of context where that function will be called | 16:41 |
| lucasagomes | cause I thought it was the operator manually calling it to update the nodes | 16:41 |
| dtantsur | lucasagomes, unfortunately I can't show you the bug, it's super-private.. well, yes. but still it should not overwrite anything without a warning | 16:42 |
| dtantsur | otherwise we won't be able to fine-tune it AND it will change the root device in cases like adding a new disk | 16:42 |
| bnemec | thrash: It's fixed in the case where the cidrs match. If we change the default they wouldn't. You can't change your provisioning IP range with an overcloud deployed (AFAIK). | 16:42 |
| dtantsur | (which IMO we should not do) | 16:42 |
| lucasagomes | dtantsur, yeah, exactly. I'm not saying we should overwrite it, quite the oposite we should fail saying "the root device hints is already set, use --overwrite if you mean to overwrite it" | 16:43 |
| dtantsur | lucasagomes, if we fail, how do we fine-tune it? | 16:43 |
| dtantsur | i.e. I want another root device only for one nodes | 16:43 |
| lucasagomes | because if the user request B but A is set and we return success | 16:43 |
| dtantsur | * node | 16:43 |
| lucasagomes | that's the bad form | 16:43 |
| dtantsur | no. that's how it should work | 16:43 |
| lucasagomes | why? | 16:44 |
| dtantsur | lucasagomes, otherwise how do you fine-tune it? or how do you rerun this command btw? | 16:44 |
| lucasagomes | I'm requesting something, this return sucess saying it was applied but it wasn't | 16:44 |
| dtantsur | lucasagomes, well, we document it does not override the existing things.. | 16:44 |
| dtantsur | which is the same e.g. for "overcloud image upload" | 16:44 |
| *** leanderthal is now known as leanderthal|afk | 16:45 | |
| thrash | bnemec: i haven't looked at it in a couple of weeks, but I think you're correct now that you put it like that. :) | 16:45 |
| *** david_lyle has quit IRC | 16:46 | |
| dtantsur | lucasagomes, in other words: we're changing *the default* behavior. it can still be overwritten | 16:46 |
| lucasagomes | dtantsur, sounds a bit odd to me... I'm not familiar with "overcloud image upload" but if I built a new image and call this command again opinting to it I would expect it to upload the new image for me | 16:46 |
| dtantsur | lucasagomes, it doesn't :) | 16:46 |
| lucasagomes | not just say "success" and the old image is uploaded | 16:46 |
| lucasagomes | so that sucks | 16:46 |
| lucasagomes | it's very misleading | 16:46 |
| dtantsur | but ok, nevermind about image upload | 16:46 |
| *** david_lyle has joined #tripleo | 16:46 | |
| dtantsur | lucasagomes, another example: IPA does not fail when root device hints are already set, right? :) it just uses them. the same here: we define the default, but users can explicitly override them by setting root device hints | 16:47 |
| lucasagomes | dtantsur, right... maybe I don't have the full picture here | 16:47 |
| lucasagomes | dtantsur, I get it, but disks fails | 16:47 |
| lucasagomes | say the disk you were using died... now you replaced it | 16:48 |
| lucasagomes | so you run the command again and it says "success", but it still pointing to the old disk what happens? | 16:48 |
| lucasagomes | I think it should fail at least saying "look device hints is already set, do you mean to overwrite it? if so, use --overwrite" | 16:49 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 16:49 |
| dtantsur | lucasagomes, then how do we use it with existing root device hints? | 16:49 |
| *** adarazs has quit IRC | 16:50 | |
| lucasagomes | dtantsur, maybe I'm missing how this funciton will be called. Do we expect it to be called over and over without failing ? | 16:51 |
| *** mikelk has quit IRC | 16:51 | |
| *** dmacpher is now known as dmacpher-afk | 16:51 | |
| dtantsur | lucasagomes, well, yes. I think so. we may issue a warning on existing hints, this is a sane idea. but it should be callable. e.g. if you add 2 nodes, you should be able to call it. | 16:52 |
| *** Marga_ has quit IRC | 16:53 | |
| lucasagomes | dtantsur, so it's not a per node thing? when you call it it goes and fill out all nodes? | 16:53 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: add IPv6 gate job https://review.openstack.org/289445 | 16:53 |
| dtantsur | lucasagomes, yep (like all commands in tripleo) (and it's not something I really like) | 16:54 |
| lucasagomes | dtantsur, gotcha | 16:55 |
| lucasagomes | x.x | 16:55 |
| lucasagomes | dtantsur, ok... well yeah warning then to keep it consistent | 16:55 |
| dtantsur | yeah, not so obvious :( | 16:55 |
| dtantsur | yep. so that people clearly know when nodes are NOT updated | 16:55 |
| lucasagomes | but urgh, it's very ugly | 16:55 |
| lucasagomes | these "bulk" things are not ideal | 16:55 |
| lucasagomes | IMHO | 16:55 |
| dtantsur | lucasagomes, ideally we should allow changing the default logic in IPA.. nice feature to consider for newton | 16:56 |
| lucasagomes | dtantsur, yeah, I which we had operators for root device hints too (lesser than and so on) | 16:56 |
| dtantsur | lucasagomes, or extend our root device hints to support things like "largest"... actually we already have "name", but it's mitaka only and does not allow several options | 16:56 |
| lucasagomes | for size that would great | 16:56 |
| dtantsur | yep | 16:56 |
| lucasagomes | dtantsur, yeah indeed | 16:57 |
| dtantsur | so yeah "in" operator would be of great value | 16:57 |
| lucasagomes | I will try to add a quick RFE for it | 16:57 |
| lucasagomes | see if we can do w/o the need of a spec | 16:57 |
| lucasagomes | shouldn't be hard | 16:57 |
| lucasagomes | dtantsur, yeah | 16:57 |
| dtantsur | like "root_device": {"name": ["sda", "hda"]} | 16:57 |
| lucasagomes | yeah, it should be easy done since it's all python now | 16:59 |
| * lucasagomes start a small RFE about it | 16:59 | |
| *** mgrohar has joined #tripleo | 16:59 | |
| dtantsur | lucasagomes, and then "root_device": {"strategy": "largest"} | 16:59 |
| dtantsur | that would be much better | 16:59 |
| *** dcain has joined #tripleo | 16:59 | |
| dtantsur | but for liberty + mitaka we have to have something...... | 16:59 |
| lucasagomes | ++ | 17:00 |
| *** jaosorior has quit IRC | 17:00 | |
| Erming__ | xxxx has no server available! | 17:00 |
| dtantsur | lucasagomes, so yeah, lets try to land it at least for newton | 17:00 |
| *** jaosorior has joined #tripleo | 17:01 | |
| lucasagomes | dtantsur, yeah, I will try to come up with something about it after the meeting (or tomorrow morning) | 17:01 |
| dtantsur | awesome, thnx | 17:02 |
| shadower | shardy: https://bugs.launchpad.net/heat/+bug/1554124 | 17:03 |
| openstack | Launchpad bug 1554124 in heat "Deleting a stack fails during preview_resources" [Undecided,New] | 17:03 |
| *** ifarkas has quit IRC | 17:03 | |
| shadower | it's one of the worse bug reports I've filed, but I didn't get to dig deep enough yet :-( | 17:03 |
| *** mkovacik has joined #tripleo | 17:04 | |
| Erming__ | trown: After manually started mysqld, I tried to restart pacemaker to restart all the services, however, it reports: proxy xxxx has no server available. | 17:04 |
| Erming__ | trown: for example: haproxy[1896]: proxy keystone_admin has no server available! | 17:04 |
| *** yamahata has joined #tripleo | 17:05 | |
| Erming__ | trown: same to all services except swift ones. what should I do to start them normally. | 17:05 |
| *** trown is now known as trown|lunch | 17:05 | |
| *** absubram has quit IRC | 17:06 | |
| *** mbound has quit IRC | 17:09 | |
| *** cwolferh has joined #tripleo | 17:10 | |
| bnemec | shadower: I believe that's already fixed on Heat master. It's just a problem on the pinned tripleo repo. | 17:13 |
| bnemec | At least moving to master Heat got me past the problem. | 17:13 |
| *** dustins has quit IRC | 17:14 | |
| derekh | bnemec: shadower FYI if it helps, we've moved the pin on saturday | 17:14 |
| *** lazy_prince has quit IRC | 17:15 | |
| shadower | bnemec, derekh: ah, okay. My undercloud comes from Friday | 17:16 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Add glance api & registry classes https://review.openstack.org/289459 | 17:16 |
| bnemec | Yeah, it was definitely still broken Friday. | 17:16 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Set /64 cidr_netmask for pcmk VIPs when IPv6 https://review.openstack.org/289461 | 17:17 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Set /64 cidr_netmask for pcmk VIPs when IPv6 https://review.openstack.org/267647 | 17:19 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: deploy glance using split roles https://review.openstack.org/289466 | 17:22 |
| EmilienM | dprince, shardy: this is a first attemp to split Glance stack https://review.openstack.org/#/q/topic:glance_splitstack | 17:23 |
| *** Marga_ has joined #tripleo | 17:24 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Add glance api & registry classes https://review.openstack.org/289459 | 17:24 |
| *** dustins has joined #tripleo | 17:25 | |
| *** absubram has joined #tripleo | 17:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: deploy glance using split roles https://review.openstack.org/289466 | 17:30 |
| *** ohamada has quit IRC | 17:32 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: deploy glance using split roles https://review.openstack.org/289466 | 17:32 |
| *** trown|lunch is now known as trown | 17:33 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud: Add an option to enable cleaning https://review.openstack.org/289473 | 17:34 |
| *** Marga_ has quit IRC | 17:35 | |
| *** chem has quit IRC | 17:36 | |
| *** mgrohar has quit IRC | 17:39 | |
| *** panda has quit IRC | 17:40 | |
| *** panda has joined #tripleo | 17:40 | |
| *** jaosorior has quit IRC | 17:41 | |
| *** jaosorior has joined #tripleo | 17:42 | |
| *** cmyster has quit IRC | 17:45 | |
| *** jaosorior has quit IRC | 17:46 | |
| *** fgimenez has quit IRC | 17:48 | |
| *** Marga_ has joined #tripleo | 17:48 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 17:50 |
| *** Marga_ has quit IRC | 17:51 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml https://review.openstack.org/235569 | 17:51 |
| *** Marga_ has joined #tripleo | 17:51 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/289435 | 17:54 |
| *** lucasagomes is now known as lucas-dinner | 17:56 | |
| *** adarazs has joined #tripleo | 18:01 | |
| slagle | dprince: fyi, this https://review.openstack.org/#/c/289085/ and the 2 depends-on should fix the ValidationDeployment problems | 18:03 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common: Change the private subnet of the overcloud tenant network https://review.openstack.org/289489 | 18:04 |
| gfidente | slagle, ^^ I think I nailed the subnet collision | 18:04 |
| slagle | gfidente: ok. any idea how it's been passing on master though, and not stable/liberty? | 18:05 |
| *** ayoung has quit IRC | 18:06 | |
| gfidente | slagle, exactly what I was thinking and no, I don't | 18:06 |
| slagle | maybe neutron in master allows subnet overlaps | 18:06 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 18:08 |
| *** tremble has quit IRC | 18:09 | |
| *** dtantsur is now known as dtantsur|afk | 18:11 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 18:11 |
| *** yamahata has quit IRC | 18:11 | |
| gfidente | slagle, though ^^ was supposed to be customizing that already! | 18:11 |
| openstackgerrit | Dmitry Tantsur proposed openstack/python-tripleoclient: Allow 'openstack baremetal configure boot' to guess the root device https://review.openstack.org/288417 | 18:12 |
| openstackgerrit | Brad P. Crochet proposed openstack/python-tripleoclient: Add 'undercloud upgrade' command https://review.openstack.org/289498 | 18:12 |
| gfidente | slagle, oh tripleo-common has stable/liberty ... | 18:15 |
| gfidente | slagle, so given in stable/liberty we were creating 10.0.0.0/8 I think the heat client in liberty just wasn't doing well with -P parameters | 18:18 |
| gfidente | slagle, they weren't being effectively overridden causing the overlap to happen only in liberty | 18:18 |
| gfidente | slagle, makes sense? | 18:18 |
| gfidente | slagle, in that case it's https://review.openstack.org/#/c/289495/ which together with the depends-on should fix liberty | 18:19 |
| *** dustins has quit IRC | 18:19 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test. https://review.openstack.org/281376 | 18:19 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 18:20 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/puppet-pacemaker: Add a service provider. https://review.openstack.org/286124 | 18:22 |
| *** dustins has joined #tripleo | 18:22 | |
| *** mgould has quit IRC | 18:22 | |
| *** Marga_ has quit IRC | 18:23 | |
| *** xinwu has quit IRC | 18:24 | |
| slagle | gfidente: k, makes sense | 18:24 |
| *** shivrao has joined #tripleo | 18:27 | |
| *** akrivoka has quit IRC | 18:31 | |
| *** slagle changes topic to "TripleO | stable/liberty CI failing: https://bugs.launchpad.net/tripleo/+bug/1554169 | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 18:33 | |
| Erming__ | larsks: I noticed that you article on deploying ha openstack, and you also use the undercloud.qcow2. Is the the same as Trown's image? Assuming you know Trown already. :-) | 18:37 |
| trown | Erming__: ya larsks has been making a ton of contributions to tripleo-quickstart | 18:38 |
| larsks | trown: beat me to the reply... :) | 18:38 |
| Erming__ | trown: thanks | 18:38 |
| larsks | I was just using tripleo-quickstart, which pulls images from centosci infra. | 18:38 |
| Erming__ | larsks: what's the difference between your two solutions? | 18:39 |
| trown | Erming__: just got to your question above... no clue, probably need someone with a better clue about pacemaker to answer | 18:39 |
| larsks | Erming__: there is no difference. | 18:39 |
| Erming__ | larsks: thanks | 18:39 |
| larsks | Erming__: I was using trown's solution... | 18:39 |
| Erming__ | larsks: I got proxy xxxx has no server available! issue | 18:39 |
| trown | Erming__: ya the blog details different (better) usage instructions for tripleo-quickstart, but it is the same ansible bits being used in both cases | 18:40 |
| larsks | That generally means that one of the services fronted by haproxy is not running. | 18:40 |
| larsks | Figure out which one, check the logs... | 18:40 |
| larsks | ...lather, rinse, repeat. | 18:40 |
| Erming__ | larsks: after deployed overcloud by trown's way. All services like this. so I guess it's haproxy issue? | 18:40 |
| larsks | Probably *not* an haproxy issue, actually. | 18:41 |
| Erming__ | larsks: haproxy[1896]: proxy keystone_admin has no server available! | 18:41 |
| *** jistr has quit IRC | 18:41 | |
| *** dustins has quit IRC | 18:41 | |
| larsks | Erming__: Okay. So you need to (a) verify that keystone is not, in fact, running, and (b) look at the keystone server logs to see if there is a clue there as to the problem. | 18:42 |
| Erming__ | Yes. I did. I found this: Lost connection to MySQL server during query' | 18:42 |
| trown | Erming__: are you using the recent (from Friday) mitaka undercloud.qcow2? | 18:42 |
| Erming__ | and then I restarted mysqld, it 's running now | 18:42 |
| trown | Erming__: ah... if mysqld was stopped, you probably have OOM issues | 18:43 |
| Erming__ | trown: no I deployed a few weeks ago | 18:43 |
| Erming__ | trown: really? good point. | 18:43 |
| Erming__ | trown: but i have 128GB on the host | 18:43 |
| Erming__ | memory | 18:43 |
| trown | Erming__: oh, as in this environment has been running for 2 weeks then stopped working? | 18:43 |
| Erming__ | trown: yes | 18:44 |
| trown | Erming__: cool, that is the longest running tripleo-quickstart environment I have encountered :) | 18:44 |
| *** rdopiera has quit IRC | 18:44 | |
| trown | Erming__: I would check /var/log/messages on overcloud nodes for OOM kill messages | 18:45 |
| Erming__ | trown: sorry. a bit confusion. I meant I deployed a few weeks ago, but didn't get time to check it carefuly. later found mysqld and almost all cloud services not running | 18:45 |
| Erming__ | on the host? | 18:45 |
| Erming__ | sorry | 18:46 |
| Erming__ | overcloud | 18:46 |
| trown | Erming__: ya | 18:46 |
| trown | Erming__: how much RAM did you give the overcloud nodes? by default it is pretty low, but on a 128GB virthost you could give them alot | 18:47 |
| Erming__ | should I check instackenv.json? | 18:47 |
| trown | that would work | 18:47 |
| Erming__ | 4096, I keep your instackenv.json no change | 18:47 |
| trown | ok that is the problem then | 18:48 |
| Erming__ | why? | 18:48 |
| trown | the defaults are really meant for CI/dev env use case | 18:48 |
| trown | which is 32G hosts mostly | 18:48 |
| Erming__ | 32G? | 18:48 |
| trown | and the envs dont need to stay up for 2 weeks :) | 18:48 |
| trown | in your case, you would want to give the overcloud nodes more RAM | 18:49 |
| trown | 32G RAM | 18:49 |
| Erming__ | how much for each ? | 18:49 |
| Erming__ | 32GB for each controller? | 18:49 |
| trown | do you use that host for anything else? | 18:49 |
| Erming__ | basically no | 18:49 |
| *** Marga_ has joined #tripleo | 18:50 | |
| trown | I would do 16GB undercloud, 8GB controllers | 18:50 |
| *** jprovazn is now known as jprovazn_bbl | 18:50 | |
| trown | that leaves plenty for compute or ceph nodes | 18:51 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Create new partitioning-sfdisk element. https://review.openstack.org/259881 | 18:51 |
| Erming__ | do I have a place to set RAM for undercloud? | 18:52 |
| Erming__ | undercloud.conf? | 18:52 |
| trown | Erming__: it would be on deploy of tripleo-quickstart... you could manually change up libvirt, but I would recommend redeploy | 18:52 |
| Erming__ | trown: in redeploy, how can I set that in your script | 18:53 |
| trown | I plan to make more "scenario" example settings files, but https://github.com/redhat-openstack/tripleo-quickstart/blob/master/playbooks/centosci/centosci_minimal_nodes.yml shows everything you would want to change | 18:53 |
| trown | Erming__: as far as how to use it, I would follow larsks blog post to use ansible-playbook directly instead of quickstart.sh | 18:54 |
| *** dustins has joined #tripleo | 18:54 | |
| trown | Erming__: quickstart.sh is really a quick try it out script, I always just use ansible-playbook directly | 18:55 |
| Erming__ | trown: I will try to find the place to change that. thanks. what else you think I need change to your instackenv file? | 18:55 |
| trown | Erming__: you would not need to change the instackenv file if you redeploy, it will get created from the settings | 18:55 |
| Erming__ | trown: makes sense. | 18:55 |
| Erming__ | trown: for baremetal, I don't see a place in lars's instruction, how to set up mac address for IPMI. | 18:56 |
| *** tosky has quit IRC | 18:56 | |
| Erming__ | I just took a glance | 18:56 |
| trown | Erming__: you have baremetal too? | 18:56 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Enable notifications on undercloud https://review.openstack.org/289518 | 18:56 |
| Erming__ | trown: I am going to have. | 18:57 |
| Erming__ | trown: that's the target | 18:57 |
| trown | Erming__: ah ok, that is bleeding edge for tripleo-quickstart... as in I have not even personally tried it :) | 18:57 |
| trown | you will want to manually create the instackenv.json for that | 18:57 |
| Erming__ | trown: why? in the rdo-manager, baremetal is the other option | 18:57 |
| trown | Erming__: well, I dont see why it wouldnt work, but I have not pursued that use case | 18:59 |
| trown | mostly due to lack of hardware | 18:59 |
| Erming__ | trown: I will create my own instackenv.json for sure. So all in all, you think the virtual machine case is basically OOM issue? In default it should work as I don't have anything customized. | 19:00 |
| Erming__ | trown: I have hardware for pilot deployment, maybe we could work together if you want :-) | 19:01 |
| trown | Erming__: ya, 4GB overcloud nodes is barely enough to pass CI, I am sure they would have memory problems if left running for 2 weeks | 19:01 |
| trown | Erming__: I am up for collaborating on that | 19:01 |
| *** pradk_ has joined #tripleo | 19:01 | |
| Erming__ | trown: I can create account(s) for you to check with me. I have 3 head nodes and a bunch of compute nodes. | 19:01 |
| Erming__ | each head node has 128GB mem. Just a bit old. (IBM x3650) | 19:02 |
| trown | cool | 19:03 |
| *** pradk has quit IRC | 19:03 | |
| Erming__ | trown: Awesome. I will show you more details about accessing the nodes later afternoon. (going to play badminton now in lunch time) Thanks!!!! | 19:03 |
| trown | nice, have fun! | 19:04 |
| *** tzumainn has quit IRC | 19:07 | |
| *** pradk has joined #tripleo | 19:10 | |
| *** pradk_ has quit IRC | 19:11 | |
| *** cwolferh has quit IRC | 19:16 | |
| *** cwolferh has joined #tripleo | 19:17 | |
| *** shardy_ has joined #tripleo | 19:18 | |
| *** yamahata has joined #tripleo | 19:18 | |
| *** shardy has quit IRC | 19:18 | |
| *** tzumainn has joined #tripleo | 19:19 | |
| dprince | slagle: thanks for these validation deployment fixes | 19:19 |
| *** jprovazn_bbl has quit IRC | 19:24 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Make AllNodesExtraConfig depend on the validation deployments https://review.openstack.org/288747 | 19:27 |
| *** gfidente has quit IRC | 19:29 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Set host in nova.conf for compute nodes https://review.openstack.org/288866 | 19:32 |
| *** paramite has quit IRC | 19:44 | |
| *** jtomasek_ has joined #tripleo | 19:51 | |
| *** dcain has quit IRC | 20:01 | |
| *** dcain has joined #tripleo | 20:03 | |
| *** Marga_ has quit IRC | 20:04 | |
| *** akuznetsov has joined #tripleo | 20:08 | |
| *** david-lyle_ has joined #tripleo | 20:12 | |
| *** david_lyle has quit IRC | 20:13 | |
| *** Marga_ has joined #tripleo | 20:15 | |
| *** akuznetsov has quit IRC | 20:20 | |
| *** akuznetsov has joined #tripleo | 20:25 | |
| *** david-lyle_ is now known as david-lyle | 20:26 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add element to force loading of the 3ware module https://review.openstack.org/289552 | 20:28 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add element to force loading of the 3ware module https://review.openstack.org/289552 | 20:29 |
| *** Goneri has quit IRC | 20:30 | |
| *** Goneri has joined #tripleo | 20:30 | |
| *** admin0 has joined #tripleo | 20:36 | |
| *** zeroshft has joined #tripleo | 20:38 | |
| *** nico_auv has quit IRC | 20:39 | |
| EmilienM | dprince: when you have some time, I have a patch in puppet-tripleo that I would like some feedback so I can continue with other services: https://review.openstack.org/#/c/286344 | 20:42 |
| EmilienM | dsneddon: do you have a PoC handy in THT? | 20:42 |
| EmilienM | dsneddon: about dual stack | 20:42 |
| *** jtomasek_ has quit IRC | 20:43 | |
| *** admin0 has quit IRC | 20:44 | |
| *** xinwu has joined #tripleo | 20:44 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 20:50 |
| bnemec | dprince: derekh: ^ We're spending a bunch of time and bandwidth downloading the Fedora image every run now. | 20:51 |
| *** yamahata has quit IRC | 20:55 | |
| *** yamahata has joined #tripleo | 20:55 | |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-common: Use Fedora 23 atomic in container gate https://review.openstack.org/289563 | 21:04 |
| dprince | bnemec: ack. looking | 21:05 |
| openstackgerrit | Ryan Hallisey proposed openstack/tripleo-common: Use Fedora 23 atomic in container gate https://review.openstack.org/289565 | 21:06 |
| dprince | bnemec: when it passes CI feel free to +A | 21:07 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 21:07 |
| bnemec | dprince: Thanks, will do. | 21:07 |
| dprince | bnemec: I would really like CI metrics so we notice stuff like this front and central | 21:11 |
| *** akuznetsov has quit IRC | 21:11 | |
| bnemec | dprince: Yeah, we should see if we could do something like http://status.openstack.org/openstack-health/#/ for toci. | 21:12 |
| bnemec | Although I thought that had test time metrics too, which I'm not seeing now. | 21:14 |
| *** dcain has quit IRC | 21:15 | |
| dprince | bnemec: yeah. THat is a nice report but I want metrics on each part (fine grained) of our CI setup | 21:15 |
| bnemec | dprince: I thought it had something kind of like that, but now I don't see it so I must have been thinking of something else. | 21:16 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Make AllNodesExtraConfig depend on the validation deployments https://review.openstack.org/289568 | 21:16 |
| dprince | bnemec: I used to have custom metrics in SmokeStack. I might could leverage some of my old work and we could raise it from there | 21:17 |
| EmilienM | bnemec: I think you need tempest runs for openstack-health | 21:17 |
| *** jcoufal has quit IRC | 21:17 | |
| dprince | bnemec: I want DIB build times, image download times, RPM build times, etc. | 21:17 |
| EmilienM | in puppet CI, we use dstat a lot to grab metrics and store them in /var/log/dstat.log | 21:17 |
| bnemec | dprince: Yeah, that would definitely be good. | 21:17 |
| EmilienM | it helped us to find out timeouts | 21:17 |
| bnemec | EmilienM: We actually have dstat now too. I haven't looked at the output yet though. | 21:18 |
| EmilienM | cool! | 21:18 |
| dprince | EmilienM: dstat would be cool too | 21:19 |
| dprince | EmilienM: I really just think having wall time for all of our tasks is what I'm after | 21:19 |
| EmilienM | yeah, I can submit a patch for that | 21:20 |
| *** ayoung has joined #tripleo | 21:20 | |
| EmilienM | I was looking at code to see how it works | 21:20 |
| *** dcain has joined #tripleo | 21:20 | |
| dprince | EmilienM: as a start could you just link me how puppet CI does this? | 21:20 |
| derekh | We got dstat on the jenkins node and undercloud, iirc not the overcloud nodes | 21:21 |
| *** jayg is now known as jayg|g0n3 | 21:22 | |
| dprince | derekh: I'm interested in undercloud and jenkins for starters anyways | 21:22 |
| EmilienM | dprince: about dstat? sure. Code is here: https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L81 and example of file: http://logs.openstack.org/46/289446/1/check/gate-puppet-openstack-integration-scenario002-tempest-dsvm-centos7/35e11b8/logs/dstat.txt.gz | 21:22 |
| derekh | what I'd like to see is metrics for timings between specific checkpoints that we can litter around the ci test | 21:22 |
| dprince | derekh: yes, exactly. I was thinking just a shell script function. (this is what I had in smokestack FWIW) | 21:23 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 21:23 |
| EmilienM | dprince, derekh: I like this idea | 21:23 |
| dprince | derekh: See my start_metric/stop_metric functions here https://github.com/dprince/smokestack/blob/master/app/templates/puppet_runner.sh.erb#L78 | 21:24 |
| dprince | EmilienM: this is how I kept my CI runtime < 25 minutes | 21:24 |
| EmilienM | dprince: multi node? | 21:25 |
| dprince | EmilienM: yes | 21:25 |
| EmilienM | o_O | 21:25 |
| EmilienM | that's what we have, on a single node (in puppet CI) | 21:25 |
| dprince | EmilienM: before there was a devstack man | 21:25 |
| dprince | EmilienM: there didn't used to be so many services I think | 21:26 |
| dprince | EmilienM: things have exploaded a bit | 21:26 |
| EmilienM | "a bit" :) | 21:26 |
| *** shardy_ has quit IRC | 21:27 | |
| derekh | dprince: where does start_metric set the data to? | 21:27 |
| dprince | EmilienM: anyways, a combination of dstat and some custom shell fuctions we can put in our scripts would help | 21:27 |
| dprince | derekh: let me check and see | 21:27 |
| *** weshay has quit IRC | 21:28 | |
| derekh | dprince: doesn't matter too much, was just curious, what I had in my imaginary world was some kind of graphite server | 21:28 |
| dprince | derekh: https://github.com/dprince/smokestack/blob/master/app/templates/common.sh.erb#L304 | 21:28 |
| dprince | derekh: THis was built to log to a file, and then I had a POST script that either created HTML, or could offload it to a production service somewhere (graphite, etc.) | 21:29 |
| derekh | dprince: cool | 21:29 |
| *** admin0 has joined #tripleo | 21:29 | |
| dprince | derekh: I think perhaps having both is useful. The central service really makes changes pop out. But having each run self-contain something that is readable can be useful too | 21:30 |
| derekh | dprince: ack, makes sense | 21:30 |
| dprince | derekh: whatever we use. I want the report on tripleo.org. Along side the owl :) | 21:31 |
| derekh | ;-) | 21:31 |
| dprince | jrist: Hey, BTW how is the new owl logo | 21:31 |
| dprince | jrist: want to mention your prototype in the tripleo IRC meeting tomorrow? | 21:32 |
| jrist | sure, was going to ask around today about writing a spec - a jtomasek and a few others mentioned a 'style guide' for usage | 21:33 |
| dprince | jrist: if people like it the code it plugs into lives here for now: https://github.com/dprince/tripleosphinx | 21:33 |
| jrist | I was going to throw it into the header and login screen | 21:33 |
| jrist | dprince: ok | 21:33 |
| jrist | yeah lets add it to the agenda if you don't mind | 21:33 |
| jrist | I wanted to get it going today but I'm staring blankly at keystone at the moment | 21:33 |
| dprince | jrist: fine, you mentioned it last week when I was really busy w/ some sysadmin tasks, it looked coolish to me | 21:34 |
| jrist | dprince: no worries | 21:34 |
| dprince | jrist: only the feet... owl's gotta have some talons I think | 21:35 |
| jrist | haha | 21:35 |
| jrist | should we hold a vote about that? | 21:35 |
| jrist | perhaps he's looking up at you and his feet are hidden below his body | 21:35 |
| dprince | jrist: meetings have been running long, but perhaps on the list we can | 21:36 |
| *** derekh has quit IRC | 21:36 | |
| jrist | 10-4 | 21:37 |
| *** admin0 has quit IRC | 21:46 | |
| *** r-mibu has quit IRC | 21:47 | |
| *** r-mibu has joined #tripleo | 21:47 | |
| *** dshulyak has quit IRC | 21:47 | |
| *** ccamacho has quit IRC | 21:55 | |
| *** rpothier has quit IRC | 21:55 | |
| *** trown is now known as trown|outtypewww | 21:57 | |
| *** david-lyle has quit IRC | 22:02 | |
| *** david-lyle has joined #tripleo | 22:02 | |
| *** dshulyak has joined #tripleo | 22:03 | |
| *** rhallisey has quit IRC | 22:06 | |
| *** lblanchard has quit IRC | 22:06 | |
| *** yamahata has quit IRC | 22:07 | |
| *** dshulyak has quit IRC | 22:08 | |
| *** dprince has quit IRC | 22:09 | |
| *** admin0 has joined #tripleo | 22:19 | |
| *** psanchez has quit IRC | 22:23 | |
| *** shadower has quit IRC | 22:23 | |
| *** slagle has quit IRC | 22:24 | |
| *** adarazs has quit IRC | 22:25 | |
| *** stevebaker has quit IRC | 22:25 | |
| *** adarazs has joined #tripleo | 22:27 | |
| *** jtomasek_ has joined #tripleo | 22:33 | |
| *** dcain has quit IRC | 22:36 | |
| *** dustins has quit IRC | 22:41 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: Add an environment to use a swap partition https://review.openstack.org/289610 | 22:59 |
| *** jtomasek_ has quit IRC | 23:01 | |
| *** jtomasek has quit IRC | 23:02 | |
| *** absubram has quit IRC | 23:06 | |
| *** admin0 has quit IRC | 23:18 | |
| *** palexster has quit IRC | 23:19 | |
| *** zeroshft has quit IRC | 23:25 | |
| *** palexster has joined #tripleo | 23:32 | |
| *** saneax_AFK is now known as saneax | 23:35 | |
| *** palexster has quit IRC | 23:37 | |
| *** palexster has joined #tripleo | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
