| *** panda is now known as panda|Zz | 00:01 | |
| *** ayoung has quit IRC | 00:02 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Support whole disk images in TripleO https://review.openstack.org/394426 | 00:04 |
|---|---|---|
| *** asalkeld has joined #tripleo | 00:07 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 00:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** asalkeld has quit IRC | 00:12 | |
| *** penick has joined #tripleo | 00:15 | |
| *** penick_ has joined #tripleo | 00:18 | |
| *** penick has quit IRC | 00:19 | |
| *** penick_ is now known as penick | 00:19 | |
| openstackgerrit | Merged openstack/diskimage-builder: Change path for dnf arch override so basearch is not overwritten. https://review.openstack.org/399175 | 00:20 |
| openstackgerrit | Merged openstack/diskimage-builder: debian: install dialog package https://review.openstack.org/397218 | 00:21 |
| *** kjw3 has quit IRC | 00:21 | |
| openstackgerrit | Merged openstack/diskimage-builder: lib: common-functions: Fix tmpfs umounting https://review.openstack.org/392002 | 00:22 |
| openstackgerrit | Merged openstack/diskimage-builder: Cleanup yumdownloader repos https://review.openstack.org/395921 | 00:23 |
| openstackgerrit | Merged openstack/diskimage-builder: Disable all repos in os-refresh-config too https://review.openstack.org/398630 | 00:23 |
| *** asalkeld has joined #tripleo | 00:27 | |
| openstackgerrit | Andreas Karis proposed openstack/tripleo-heat-templates: Disable Options Indexes in horizon https://review.openstack.org/391550 | 00:31 |
| *** asalkeld has quit IRC | 00:32 | |
| *** hjensas has quit IRC | 00:35 | |
| *** limao has joined #tripleo | 00:39 | |
| *** ayoung has joined #tripleo | 00:49 | |
| *** morazi has quit IRC | 00:53 | |
| *** dhill_ has quit IRC | 01:05 | |
| *** dhill__ has joined #tripleo | 01:06 | |
| *** ayoung has quit IRC | 01:06 | |
| *** achadha has quit IRC | 01:10 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 01:10 |
| *** achadha has joined #tripleo | 01:10 | |
| *** chlong has quit IRC | 01:11 | |
| *** dhill__ has quit IRC | 01:12 | |
| *** hjensas has joined #tripleo | 01:12 | |
| *** hjensas has joined #tripleo | 01:12 | |
| *** achadha has quit IRC | 01:13 | |
| *** achadha_ has joined #tripleo | 01:13 | |
| *** achadha_ has quit IRC | 01:15 | |
| *** achadha has joined #tripleo | 01:15 | |
| *** penick has quit IRC | 01:17 | |
| *** achadha_ has joined #tripleo | 01:19 | |
| *** achadha has quit IRC | 01:19 | |
| *** achadha_ has quit IRC | 01:24 | |
| *** rhallisey has quit IRC | 01:24 | |
| *** myoung|bbl is now known as myoung | 01:41 | |
| *** myoung is now known as myoung|biab | 01:42 | |
| *** dmacpher has joined #tripleo | 01:43 | |
| *** newmember has joined #tripleo | 01:48 | |
| *** achadha has joined #tripleo | 01:58 | |
| *** tiswanso has joined #tripleo | 02:02 | |
| *** tiswanso has quit IRC | 02:03 | |
| *** achadha has quit IRC | 02:03 | |
| *** tiswanso has joined #tripleo | 02:03 | |
| *** achadha has joined #tripleo | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 02:10 |
| openstackgerrit | gecong proposed openstack/diskimage-builder: Fix a typo https://review.openstack.org/399341 | 02:31 |
| *** nyechiel has joined #tripleo | 02:33 | |
| *** lblanchard has quit IRC | 02:43 | |
| *** fzdarsky_ has joined #tripleo | 02:56 | |
| *** fzdarsky|afk has quit IRC | 02:59 | |
| *** fragatina has quit IRC | 03:03 | |
| *** nyechiel has quit IRC | 03:04 | |
| *** fragatina has joined #tripleo | 03:05 | |
| *** nyechiel has joined #tripleo | 03:08 | |
| *** fragatin_ has joined #tripleo | 03:09 | |
| *** fragatina has quit IRC | 03:09 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 03:10 |
| *** fragatin_ has quit IRC | 03:14 | |
| *** rlandy has quit IRC | 03:14 | |
| *** Goneri has joined #tripleo | 03:17 | |
| *** numans has joined #tripleo | 03:25 | |
| *** fultonj has quit IRC | 03:32 | |
| *** yamahata has quit IRC | 03:36 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn of tracing around pid/chroot check https://review.openstack.org/399365 | 03:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn off tracing around pid/chroot check https://review.openstack.org/399365 | 03:51 |
| openstackgerrit | Cao Xuan Hoang proposed openstack/instack-undercloud: Changed author and author-email https://review.openstack.org/399367 | 04:02 |
| openstackgerrit | Aparna proposed openstack/diskimage-builder: element: proliant-tools: Update hpssacli to ssacli https://review.openstack.org/396504 | 04:05 |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 04:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| *** ctayal has joined #tripleo | 04:14 | |
| *** ctayal has quit IRC | 04:20 | |
| *** bana_k has joined #tripleo | 04:26 | |
| *** Goneri has quit IRC | 04:27 | |
| *** tiswanso has quit IRC | 04:35 | |
| *** coolsvap has joined #tripleo | 04:36 | |
| *** ayoung has joined #tripleo | 04:43 | |
| *** tzumainn has quit IRC | 04:55 | |
| *** nyechiel has quit IRC | 04:58 | |
| *** yamahata has joined #tripleo | 04:58 | |
| *** pgadiya has joined #tripleo | 05:03 | |
| *** masco has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 05:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| *** udesale has joined #tripleo | 05:18 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Add puppet-qdr module https://review.openstack.org/373488 | 05:21 |
| *** prateek has joined #tripleo | 05:24 | |
| *** I has joined #tripleo | 05:24 | |
| *** I is now known as Guest66254 | 05:24 | |
| openstackgerrit | Noam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication https://review.openstack.org/391765 | 05:25 |
| *** bana_k has quit IRC | 05:26 | |
| *** bana_k has joined #tripleo | 05:27 | |
| *** Guest66254 has quit IRC | 05:29 | |
| *** achadha has quit IRC | 05:57 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 06:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| *** achadha has joined #tripleo | 06:10 | |
| *** mgagne has quit IRC | 06:18 | |
| *** limao has quit IRC | 06:20 | |
| *** limao has joined #tripleo | 06:21 | |
| *** mgagne has joined #tripleo | 06:21 | |
| *** mgagne is now known as Guest52285 | 06:21 | |
| *** dmacpher has quit IRC | 06:24 | |
| *** abehl has joined #tripleo | 06:40 | |
| *** lmiccini has joined #tripleo | 06:40 | |
| *** iranzo has joined #tripleo | 06:51 | |
| *** iranzo has joined #tripleo | 06:51 | |
| *** dsariel has joined #tripleo | 06:55 | |
| *** achadha has quit IRC | 06:57 | |
| openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Changes default MidoNet API port on HAProxy https://review.openstack.org/399125 | 06:59 |
| *** b00tcat has joined #tripleo | 07:00 | |
| *** myoung|biab is now known as myoung|pto | 07:08 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 07:10 |
| *** tesseract has joined #tripleo | 07:18 | |
| *** tesseract is now known as Guest90313 | 07:18 | |
| *** rasca has joined #tripleo | 07:19 | |
| *** achadha has joined #tripleo | 07:29 | |
| *** chandankumar has joined #tripleo | 07:34 | |
| dciabrin | dtrainor if the galera issue is still there, can you paste "pcs status" somewhere? | 07:39 |
| *** jaosorior has joined #tripleo | 07:39 | |
| *** mhenkel has joined #tripleo | 07:43 | |
| jaosorior | d0ugal: hey dude, could you take a look at this when you have time? https://review.openstack.org/#/c/397381/ | 07:44 |
| jaosorior | d0ugal: this would be how it's used in t-h-t https://review.openstack.org/#/c/397350/ | 07:45 |
| *** pcaruana has joined #tripleo | 07:45 | |
| *** ebarrera has joined #tripleo | 07:47 | |
| *** hjensas has quit IRC | 07:48 | |
| *** cylopez has joined #tripleo | 07:56 | |
| *** ealcaniz has joined #tripleo | 07:59 | |
| *** _milan_ has joined #tripleo | 07:59 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:01 |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #tripleo | 08:03 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:07 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 08:10 |
| *** amoralej|off is now known as amoralej | 08:12 | |
| openstackgerrit | Flavio Percoco proposed openstack/tripleo-quickstart: Pass the libvirt_uri to the pool-define command https://review.openstack.org/399141 | 08:14 |
| *** hogepodge has quit IRC | 08:15 | |
| *** florianf has joined #tripleo | 08:20 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates: Add system-uuid based hostname entries https://review.openstack.org/358643 | 08:21 |
| *** mcornea has joined #tripleo | 08:25 | |
| openstackgerrit | Alejandro Andreu proposed openstack/puppet-tripleo: Changes default MidoNet API port on HAProxy https://review.openstack.org/399125 | 08:26 |
| *** jprovazn has joined #tripleo | 08:28 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Correct AllNodesDeploySteps depends_on https://review.openstack.org/397983 | 08:31 |
| *** panda|Zz is now known as panda | 08:32 | |
| *** hjensas has joined #tripleo | 08:33 | |
| *** jpena|off is now known as jpena | 08:35 | |
| *** arxcruz has joined #tripleo | 08:37 | |
| *** achadha has quit IRC | 08:42 | |
| *** lifeless has quit IRC | 08:45 | |
| *** pmannidi has quit IRC | 08:45 | |
| *** lifeless has joined #tripleo | 08:47 | |
| *** lazy_prince has quit IRC | 08:47 | |
| *** fzdarsky_ is now known as fzdarsky | 08:48 | |
| *** bana_k has quit IRC | 08:50 | |
| *** lazy_prince has joined #tripleo | 08:52 | |
| *** shardy has joined #tripleo | 08:53 | |
| *** jpich has joined #tripleo | 08:55 | |
| *** pmannidi has joined #tripleo | 08:58 | |
| ccamacho | good morning guys! | 08:59 |
| shardy | Morning all! | 09:01 |
| panda | ccamacho: and girls. | 09:01 |
| shardy | We released ocata-1 yesterday, 2 blueprints and 118 bugs fixed! | 09:01 |
| shardy | Great work everybody :) | 09:01 |
| shardy | https://launchpad.net/tripleo/+milestone/ocata-1 | 09:01 |
| panda | and how many bugs created ? :) | 09:02 |
| shardy | panda: sssh! ;) | 09:02 |
| shardy | we don't talk about those :) | 09:02 |
| *** gfidente has joined #tripleo | 09:03 | |
| *** gfidente has quit IRC | 09:03 | |
| *** gfidente has joined #tripleo | 09:03 | |
| shardy | https://www.explainxkcd.com/wiki/index.php/1739:_Fixing_Problems | 09:03 |
| *** paramite has joined #tripleo | 09:07 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 09:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| matbu | shardy: hey :) i commented the LP : https://bugs.launchpad.net/tripleo/+bug/1583125 | 09:13 |
| openstack | Launchpad bug 1583125 in tripleo "There is no 'major version' upgrades job for ci " [High,In progress] - Assigned to mbu (mat-bultel) | 09:13 |
| matbu | shardy: the full upgrade job works | 09:13 |
| matbu | shardy: but the one from this night reach the 290minutes timeout :/ | 09:13 |
| *** derekh has joined #tripleo | 09:14 | |
| *** tremble has joined #tripleo | 09:14 | |
| matbu | shardy: so i removed the ceilometer migration (specific to M->N) and re-kick the experimental job to what happen | 09:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Deploy heat APIs over httpd https://review.openstack.org/394837 | 09:14 |
| shardy | matbu: Ok - what are our options to reduce the time? I saw a patch from bnemec to always use cached images | 09:14 |
| shardy | matbu: is your test using multinode or ovb ? | 09:15 |
| matbu | shardy: ovb | 09:15 |
| matbu | shardy: i didn't see the patch from ben, but could be a solution | 09:15 |
| derekh | shardy: so I got the env setup to deploy a overcloud with 40 compute nodes, but http://paste.openstack.org/show/589683/ | 09:15 |
| matbu | shardy: i think it missed 20 minutes, because it was during the converge | 09:15 |
| shardy | I was also thinking we could have a more minimal test based on multinode, as I've been testing upgrades locally with an all-in-one setup | 09:16 |
| shardy | the coverage isn't as good but it's another option to at least exercise some of the upgrade of the controlplane | 09:16 |
| shardy | matbu: ack, OK I'll check out the patch and lets see if we can reduce the time somehow | 09:17 |
| matbu | shardy: yes, btw i need to figure out why the undercloud-upgrades failed | 09:17 |
| shardy | matbu: one option would be to deploy with less services - e.g start by just proving we can upgrade some subset of the services normally enabled | 09:17 |
| matbu | shardy: AFAIK this jobs wasn't really do a major UC upgrade | 09:17 |
| matbu | shardy: yep, i was wondering if we could deploy only the controller (1nodes) ? (looks stupid but ...) | 09:18 |
| shardy | matbu: yeah, we could do that via the multinode job, which is already faster than ovb | 09:19 |
| shardy | slagle has that working with more than one node now too | 09:19 |
| shardy | derekh: nice, thanks, looking! | 09:19 |
| matbu | shardy: we can switch to multinode | 09:19 |
| jaosorior | derekh: any idea where that error is coming from? Is it from heat or from here? | 09:19 |
| jaosorior | *from where | 09:19 |
| shardy | derekh: you're missing a patch for the undercloud, sec | 09:20 |
| shardy | https://review.openstack.org/#/c/398396/ | 09:20 |
| shardy | derekh: we need that patch landed to stable/newton, it fixes the broken increase of the yaql iterators limit | 09:20 |
| shardy | in heat.conf | 09:21 |
| shardy | you can alternatively manually fix it and restart heat-engine | 09:21 |
| shardy | we bumped it from 200 to 1000 due to this issue | 09:21 |
| derekh | shardy: ok, I'll bump it up and retry | 09:21 |
| shardy | jaosorior: it's from heat | 09:21 |
| shardy | derekh: nice, thanks! | 09:21 |
| derekh | jaosorior: the error is in the heat-engine log, looks like I got a fix | 09:22 |
| jaosorior | nice :D | 09:22 |
| openstackgerrit | Frederic Lepied proposed openstack/tripleo-specs: Step by step validation Spec https://review.openstack.org/372336 | 09:26 |
| shardy | flepied: Hi, good morning! | 09:29 |
| shardy | flepied: thanks for the update on the step by step validation spec | 09:29 |
| flepied | morning shardy | 09:30 |
| flepied | shardy: catching up after my pto ;-) | 09:30 |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** shardy has quit IRC | 09:34 | |
| *** shardy has joined #tripleo | 09:34 | |
| * shardy had connection drop | 09:36 | |
| shardy | flepied: I've been thinking, we could do it via ansible, with the exact same approach I'm proposing for upgrades here: | 09:36 |
| shardy | https://review.openstack.org/#/c/393448/10/puppet/services/heat-engine.yaml | 09:36 |
| shardy | we're already using ansible for pre-flight validations so at least we'd be sticking to one tool | 09:36 |
| shardy | interested in your views on that - I'm happy to write a patch with a prototype if you like the idea | 09:36 |
| jaosorior | gfidente: out of curiosity, is there a reason why ceph rgw uses civetweb as a fronend instead of the apache fastcgi frontend? | 09:40 |
| gfidente | jaosorior I think performances | 09:42 |
| *** achadha has joined #tripleo | 09:42 | |
| jaosorior | gfidente: when ceph is running over civetweb, is it running under a user specific for civetweb (like is the case for apache, that the user and group are httpd) or is it under a ceph user? | 09:44 |
| gfidente | so the user can be configured and we default we ceph | 09:46 |
| jaosorior | alright, thanks dude | 09:47 |
| jaosorior | gfidente: I'm checking what's up with getting SSL for ceph (starting with rgw) | 09:47 |
| gfidente | we could do ssl termination in civetweb | 09:47 |
| gfidente | it's just an argument for the binding option | 09:47 |
| *** achadha has quit IRC | 09:48 | |
| jaosorior | gfidente: yeah, it doesn't seem to hard | 09:48 |
| jaosorior | the format is the same as haproxy | 09:48 |
| jaosorior | (key and cert in the same pem file) | 09:48 |
| jaosorior | and we just gotta add "s" to the end of the port to specify that it should be using TLS | 09:48 |
| gfidente | well, we always have haproxy on front so "public ssl" can be managed there without changes to civetweb no? | 09:48 |
| jaosorior | right | 09:48 |
| jaosorior | that's not an issue | 09:48 |
| jaosorior | for internal TLS we need the traffic between haproxy and civetweb to be encrypted as well | 09:49 |
| gfidente | ok so yes civetweb I think expects the s appended to the port and the path to the .pem file | 09:49 |
| jaosorior | yeah, that's what I could discern from the docco | 09:49 |
| jaosorior | I need to try to do a ceph deployment though | 09:49 |
| gfidente | is key and cert in same .pem complicated to get? | 09:49 |
| jaosorior | nah, we do exactly the same for haproxy | 09:50 |
| gfidente | so if you want I have an environment | 09:50 |
| *** derekjhyang has quit IRC | 09:51 | |
| gfidente | where you can mess with rgw | 09:51 |
| gfidente | or I could try the submission myself | 09:51 |
| jaosorior | awesome | 09:51 |
| jaosorior | thanks dude | 09:51 |
| jaosorior | you'll need a FreeIPA deployment though | 09:51 |
| jaosorior | I'll ping you when I have something usable | 09:51 |
| gfidente | ack we can test it there | 09:52 |
| jaosorior | gfidente: at some point I gotta think about the traffic between monitors and OSDs though | 09:52 |
| jaosorior | and I guess that's gonna be a bit problematic, since the nodes are configured without the domain, right? And if we would change the hosts to use the domains, that would end up with issues in the CRUSH map, or not? | 09:53 |
| *** masco has quit IRC | 09:54 | |
| gfidente | wait I think you're thinking ahead of me | 09:55 |
| gfidente | I don't think we can encrypt traffic in between the nodes natively | 09:55 |
| jaosorior | gfidente: what would we need to do if we want to encrypt it? | 09:56 |
| gfidente | so you're supposing to add terminations to do encryption transparently in front of every node? | 09:56 |
| jaosorior | gfidente: for every service, yeah, the point is to end up with TLS everywhere. | 09:56 |
| *** dtantsur|afk is now known as dtantsur | 09:56 | |
| *** jtomasek has joined #tripleo | 09:57 | |
| gfidente | though the clients would not support encryption either | 09:59 |
| gfidente | and they need to get to both the monitors and the osds | 09:59 |
| gfidente | so it looks like we'd need something like a vpn overlay | 10:00 |
| jaosorior | gfidente: yeah... swift has the same issue apparently | 10:00 |
| jaosorior | alright, well, that's good to know | 10:00 |
| gfidente | amongst the replicas you mean? | 10:01 |
| jaosorior | yeah | 10:01 |
| jaosorior | I was given the same recommendation from the swift folks: "get the nodes on a vpn instead" | 10:02 |
| jaosorior | ok, meanwhile I'll get TLS for the civetweb front-end | 10:04 |
| jaosorior | since that seems doable | 10:05 |
| shardy | VPN seems like a large overhead for high traffic storage networks which are already isolated | 10:05 |
| jaosorior | gfidente: thanks for the info dude | 10:05 |
| jaosorior | shardy: any other suggestions? | 10:05 |
| *** rbowen has joined #tripleo | 10:05 | |
| jaosorior | shardy: not using TLS is not a solution for some use-cases, specially when they have to meet government regulations | 10:05 |
| *** ealcaniz has quit IRC | 10:06 | |
| shardy | jaosorior: I'm questioning the requirement I guess - we've already isolated that traffic, it could be on a physically separate network where hardware provides a secure transport between disparate locations | 10:06 |
| shardy | I don't get why you'd demand potentially horribly slow software encryption between nodes in the same rack | 10:07 |
| shardy | if folks have access to the rack, over the wire encryption doesn't really help | 10:07 |
| gfidente | on the other hand, crazy but if we managed to get a vpn overlay on any given network, I wonder if that wouldn't be the "less intrusive" solution to encrypt all internal communications | 10:07 |
| * gfidente hides | 10:08 | |
| jaosorior | shardy: regulations... | 10:08 |
| jaosorior | gfidente: I wouldn't mind deleting all the TLS parts if we end up doing that, to be honest. | 10:09 |
| shardy | jaosorior: Every bank I ever worked with terminated https in a hardware load balancer | 10:09 |
| shardy | tunneling storage traffic via hardware is no different | 10:09 |
| gfidente | jaosorior I don't know how hard it would be to get os-net-config to create a vpn | 10:09 |
| jaosorior | neither do I | 10:09 |
| gfidente | jaosorior it isn't necessary cleaner or easier | 10:09 |
| gfidente | it's probably less intrusive on the services configuration | 10:09 |
| gfidente | but not easier to manage I mean | 10:10 |
| jaosorior | for sure | 10:10 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 10:10 |
| shardy | http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Neutron::VPNService | 10:10 |
| panda | another promotion blocker: https://bugs.launchpad.net/os-client-config/+bug/1642897 | 10:10 |
| openstack | Launchpad bug 1642897 in os-client-config "osc commands fail when using os-client-config >= 1.23.0" [Undecided,New] | 10:10 |
| gfidente | shardy those would be fake networks in the UC though | 10:10 |
| gfidente | we need onc to actually do the thing in ifcfg too | 10:10 |
| jaosorior | shardy: honestly, not up to me, apparently some clients really want TLS everywhere support | 10:10 |
| shardy | jaosorior: yeah, I'm just saying lets fully understand the requirement | 10:11 |
| shardy | folks often say they want one thing, but don't actually know what they really need ;) | 10:11 |
| *** katkapilatova has joined #tripleo | 10:12 | |
| *** nyechiel has joined #tripleo | 10:13 | |
| *** hewbrocca_afk is now known as hewbrocca | 10:14 | |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates: Add system-uuid based hostname entries https://review.openstack.org/358643 | 10:18 |
| *** limao has quit IRC | 10:22 | |
| *** akrivoka has joined #tripleo | 10:32 | |
| *** fragatina has joined #tripleo | 10:35 | |
| *** fragatina has quit IRC | 10:36 | |
| *** fragatina has joined #tripleo | 10:37 | |
| *** yamahata has quit IRC | 10:41 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Add pxe_drac to the node registration driver list https://review.openstack.org/399512 | 10:43 |
| *** udesale has quit IRC | 10:50 | |
| *** nyechiel has quit IRC | 10:51 | |
| shardy | jaosorior: Hey do you happen to know what the default cache configuration is for keystone when deployed via TripleO? | 10:57 |
| rasca | guys have we got somewhere a schema resuming the relations between services in newton? I'm trying to debug a problem with glance (timeout) but this service is fine, so I'm looking to swift, but would like to have an idea of the entire relations picture | 10:57 |
| shardy | jaosorior: I don't see anything explicitly configured in the profile, and I'm trying to deploy a minimal keystone only setup | 10:58 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart: WIP: Update OOOQ to ansible 2.2 https://review.openstack.org/398194 | 10:58 |
| shardy | http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html | 10:58 |
| shardy | I did that previously as described there, but it doesn't work anymore, trying to figure out what changed | 10:58 |
| jaosorior | I don't know what the default cache is | 10:59 |
| shardy | jaosorior: Ok, I'll dig into it | 10:59 |
| jaosorior | shardy: stable/newton or master? | 10:59 |
| shardy | we deploy memcache and redis, and AFAICS we don't configure keystone to use either, although puppet-keystone supports both | 11:00 |
| shardy | jaosorior: on master | 11:00 |
| shardy | the blog was written before we cut newton | 11:00 |
| *** rbowen has quit IRC | 11:00 | |
| shardy | Now haproxy won't deploy without keepalived, and keystone is running but unresponsive | 11:01 |
| shardy | https://review.openstack.org/#/c/399152/ aims to fix the keepalived coupling | 11:01 |
| *** jaosorior is now known as jaosorior_lunch | 11:04 | |
| dtantsur | hi folks! how is stable/newton CI feeling today? | 11:04 |
| *** arxcruz has quit IRC | 11:06 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use j2 loops in post.j2.yaml https://review.openstack.org/396230 | 11:08 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles https://review.openstack.org/330659 | 11:09 |
| *** arxcruz has joined #tripleo | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 11:10 |
| *** jkilpatr has quit IRC | 11:11 | |
| *** jtomasek has quit IRC | 11:13 | |
| gfidente | shardy https://review.openstack.org/#/c/399152 any clue how come ha was still passing? | 11:18 |
| gfidente | I mean before the patch | 11:18 |
| gfidente | (ha job) | 11:18 |
| *** nyechiel has joined #tripleo | 11:21 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-specs: Composable Service Upgrades https://review.openstack.org/392116 | 11:32 |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Add pxe_drac to the node registration driver list https://review.openstack.org/399512 | 11:32 |
| shardy | gfidente: it was hard-coded to true, and it's still true because hiera keepalived_enabled is true | 11:33 |
| shardy | gfidente: it only fixes the case where you want to do a local deployment without keepalived | 11:33 |
| gfidente | shardy no in HA job it should be false | 11:33 |
| gfidente | HA job does not use keepalived | 11:34 |
| shardy | gfidente: where do we disable keepalived? | 11:34 |
| gfidente | huh but we are not disabling it indeed | 11:34 |
| shardy | afaics it's always on | 11:34 |
| gfidente | though this seems a mistake to me | 11:34 |
| shardy | bandini: ^^ ? | 11:35 |
| gfidente | bandini ^^ | 11:35 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart: WIP: Update OOOQ to ansible 2.2 https://review.openstack.org/398194 | 11:35 |
| *** katkapilatova has left #tripleo | 11:35 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart: Configuring vcpus for undercloud, controller and compute nodes https://review.openstack.org/396593 | 11:35 |
| gfidente | shardy indeed it does not start | 11:36 |
| gfidente | it's configured but not enabled on boot | 11:37 |
| shardy | ugh | 11:38 |
| gfidente | yeah | 11:38 |
| shardy | we set a boolean in ./puppet/services/pacemaker/haproxy.yaml: enable_keepalived: false | 11:38 |
| gfidente | the VIPs are in charge to pcmk in HA job | 11:38 |
| gfidente | ah cook | 11:38 |
| gfidente | *cool | 11:38 |
| shardy | Ok I'll have to fix that, as it will break with composable upgrades | 11:38 |
| gfidente | so we could just set the resource to None | 11:38 |
| shardy | we cannot have any services disabled except via OS::Heat::None or removing from the *Services lists | 11:38 |
| gfidente | agreed | 11:39 |
| gfidente | can I take it? | 11:39 |
| shardy | gfidente: ack, thanks - I'll post a patch which does that now | 11:39 |
| gfidente | oh okay I won't do it then | 11:39 |
| gfidente | add me on review | 11:39 |
| *** jtomasek has joined #tripleo | 11:41 | |
| *** anton has quit IRC | 11:41 | |
| akrivoka | jpich: hey I just spotted a copy-paste error I made when refactoring the driver fields UI components | 11:42 |
| *** prateek has quit IRC | 11:42 | |
| akrivoka | jpich: https://github.com/openstack/tripleo-ui/blob/master/src/js/components/nodes/driver_fields/PXEAndIPMIToolDriverFields.js#L5 | 11:42 |
| akrivoka | jpich: the class should be called PXEAndIPMIToolDriverFields, of course | 11:43 |
| akrivoka | jpich: do you want to include the fix with your patch https://review.openstack.org/#/c/399512 since it touches the same files? | 11:43 |
| *** jaosorior_lunch is now known as jaosorior | 11:44 | |
| *** anton has joined #tripleo | 11:44 | |
| jpich | akrivoka: I don't actually touch that one, I think it'd fit better on its own? How come the import didn't fail since the class doesn't exist? | 11:44 |
| akrivoka | jpich: because javascript :) | 11:45 |
| akrivoka | jpich: I guess when you export default, and then import from another file, it does not matter what you name the thing you imported | 11:45 |
| akrivoka | jpich: no probs then, I'll submit a fix | 11:46 |
| jpich | akrivoka: I see I still have a long way to go. Great refactoring btw, I accidentally looked at an older version of these files first and was full of sad | 11:46 |
| akrivoka | haha | 11:46 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Disable keepalived for HA deployments via t-h-t https://review.openstack.org/399554 | 11:48 |
| shardy | gfidente: ^^ | 11:48 |
| jpich | akrivoka: Actually thinking further it might be easier to sneakily backport the typo fix if I include it in my fix, if that refactoring is in Newton as well? | 11:49 |
| akrivoka | jpich: it is in Newton, yeah go for it :) | 11:50 |
| jpich | akrivoka: Theoretically no backport is allowed without associated bug AFAIU | 11:50 |
| jpich | akrivoka: Alright | 11:50 |
| *** nyechiel has quit IRC | 11:53 | |
| jaosorior | shardy: do you happen to know if the heat-templates that has the fix I did for the content-type is available in the promoted images? | 11:54 |
| openstackgerrit | Frederic Lepied proposed openstack/tripleo-specs: Step by step validation Spec https://review.openstack.org/372336 | 11:54 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Remove conditional in keepalived profile https://review.openstack.org/399556 | 11:54 |
| shardy | jaosorior: current-tripleo hasn't promoted in two weeks, so I'd guess not | 11:54 |
| shardy | you can check by looking at the current-tripleo repo | 11:54 |
| gfidente | there is bandini leaving +2s :) | 11:54 |
| shardy | http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/ | 11:55 |
| bandini | gfidente: yeah saw the backlog now ;) when I disabled keepalive that way I was still obviously not entirely clear on the composable stuff :) | 11:55 |
| gfidente | or we didn't have it finished yet maybe | 11:56 |
| gfidente | but I was more emphasizing on the +2 | 11:56 |
| *** pkovar has joined #tripleo | 11:56 | |
| bandini | :) | 11:57 |
| shardy | bandini: I have an upgrade related question - does it make sense to stop haproxy and (pacemaker|keepalived) as a first step of upgrades, before stopping any other services? | 11:58 |
| shardy | I asked the same question to marios yesterday but didn't see a reply | 11:59 |
| shardy | I'm initially testing my WIP upgrades patch with a simple nonha environment, and haproxy spews errors if you don't stop it first, then all the services as a second step | 11:59 |
| shardy | I assume we handle this in the pcmk case by taking down the pacemaker cluster, which has dependencies such that haproxy is stopped first? | 12:00 |
| bandini | shardy: correct we basically do "bring down the cluster, yum update, bring up the cluster" (I am skipping over a bunch of details but yeah) | 12:00 |
| bandini | actually right after the yum upgrade we actually start a minimal subset of services (the ones needed for the schema upgrade tools to work) and then start the rest | 12:01 |
| bandini | shardy: does that answer your question? | 12:02 |
| panda | what's the upstrea for "import gear" we are using in tripleo-ci ? | 12:02 |
| shardy | bandini: Ok, I'm interested in making this work for the non pacemaker case, so I assume it will be 1. stop haproxy + keepalived, 2. stop all services, 3. yum update, 4. start db etc, 5. db sync, 6. start services | 12:02 |
| shardy | bandini: yep I think so, thanks | 12:02 |
| bandini | shardy: that sounds about right | 12:02 |
| *** dsariel has quit IRC | 12:05 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 12:10 |
| openstackgerrit | yolanda.robla proposed openstack-infra/tripleo-ci: Enable consuming packages for a feature branch https://review.openstack.org/399562 | 12:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Pass hostname to ceph-rgw https://review.openstack.org/399563 | 12:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make Ceph RGW bind to hostname instead of IP https://review.openstack.org/399564 | 12:13 |
| jaosorior | gfidente: what do you think? ^^ | 12:13 |
| panda | derekh: what's the upstrea for "import gear" we are using in tripleo-ci ? | 12:14 |
| *** abregman|afk has quit IRC | 12:16 | |
| derekh | panda: http://git.openstack.org/cgit/openstack-infra/gear/ | 12:17 |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add retries to ipxe rom installation https://review.openstack.org/389818 | 12:20 |
| openstackgerrit | yolanda.robla proposed openstack-infra/tripleo-ci: Enable consuming packages for a feature branch https://review.openstack.org/399562 | 12:21 |
| *** derekjhyang has joined #tripleo | 12:21 | |
| flaper87 | jrist: hey, could you take a look here ? https://review.openstack.org/#/c/330659/ ? :D | 12:21 |
| *** jbadiapa has quit IRC | 12:22 | |
| *** jbadiapa has joined #tripleo | 12:22 | |
| gfidente | jaosorior nah it can't | 12:22 |
| gfidente | it only works with IPs | 12:22 |
| gfidente | I tried this before :( | 12:22 |
| *** jkilpatr has joined #tripleo | 12:22 | |
| panda | derekh: thanks | 12:23 |
| jaosorior | gfidente: :( | 12:24 |
| gfidente | jaosorior yeah commented in https://review.openstack.org/#/c/399563/1 | 12:24 |
| *** ccamacho is now known as ccamacho|lunch | 12:25 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-ui: Add pxe_drac to the node registration driver list https://review.openstack.org/399512 | 12:26 |
| *** pkovar has quit IRC | 12:28 | |
| *** tobias-fiberdata has joined #tripleo | 12:29 | |
| hewbrocca | flaper87: did you mean jistr | 12:30 |
| hewbrocca | or jstir | 12:30 |
| *** hogepodge has joined #tripleo | 12:31 | |
| hewbrocca | Yes, I think you meant jistr | 12:31 |
| hewbrocca | He's off today and (I think) all next week, moving | 12:32 |
| *** tobias_fiberdata has quit IRC | 12:32 | |
| flaper87 | hewbrocca: yeah, I meant jistr | 12:32 |
| flaper87 | :( | 12:32 |
| flaper87 | ok, will ping him on monday | 12:32 |
| hewbrocca | We have a bot somewhere named jstir | 12:33 |
| hewbrocca | just to mix it up :D | 12:33 |
| *** rhallisey has joined #tripleo | 12:33 | |
| openstackgerrit | Javier Peña proposed openstack-infra/tripleo-ci: Properly set distro branch in DLRN when STABLE_RELEASE=newton https://review.openstack.org/399578 | 12:36 |
| hewbrocca | flaper87: maybe shardy could +A in his absence, or dprince? | 12:36 |
| *** lucasagomes is now known as lucas-hungry | 12:37 | |
| flaper87 | hewbrocca: that'd be awesome, although dprince hacked on that patch | 12:38 |
| flaper87 | I think he can +2 but not +A | 12:38 |
| flaper87 | shardy: if you have time https://review.openstack.org/#/c/330659/ | 12:39 |
| *** dtantsur is now known as dtantsur|brb | 12:39 | |
| weshay | panda, any open issues on the undercloud neutron-db-manage failing that you know of? | 12:42 |
| *** abregman has joined #tripleo | 12:42 | |
| weshay | panda, oh ya.. https://bugs.launchpad.net/tripleo/+bug/1641571 | 12:43 |
| openstack | Launchpad bug 1641571 in tripleo "CI: master jobs fail on neutron-db-manage" [High,Confirmed] | 12:43 |
| panda | weshay: yes | 12:43 |
| weshay | https://bugs.launchpad.net/networking-cisco/+bug/1641311 | 12:44 |
| openstack | Launchpad bug 1641311 in networking-cisco "neutron-db-manage fails after https://review.openstack.org/#/c/394201/" [High,Confirmed] | 12:44 |
| panda | weshay: it has a link to the already opened | 12:44 |
| panda | yes | 12:44 |
| weshay | panda, was that just the overcloud? | 12:44 |
| weshay | I'm seeing the issue on the undercloud | 12:44 |
| weshay | on master | 12:44 |
| panda | weshay: link ? todays there's another promotion blocker that gives similar errors on the undercloud | 12:45 |
| weshay | sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'neutron'@'192.168.24.1' (using password: YES)")[0m | 12:45 |
| gfidente | shardy I don't think we can *include* environment files from another right? | 12:45 |
| weshay | panda, I'm working on the etherpad | 12:45 |
| weshay | links are ther | 12:45 |
| weshay | e | 12:45 |
| panda | weshay: yeah saw that, I think this happens because all the previous keystone tasks fail, and no user neutron is created | 12:47 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job https://review.openstack.org/338088 | 12:48 |
| weshay | panda, ah I see /me scrolling up | 12:48 |
| *** fultonj has joined #tripleo | 12:48 | |
| weshay | panda, ya.. from 2016-11-18 03:27:20,039 INFO: [1;31mError: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: __init__() got an unexpected keyword argument 'project_domain_id' (tried 24, for a total of 170 seconds)[0m | 12:48 |
| panda | weshay: yes, caused by https://bugs.launchpad.net/os-client-config/+bug/1642897, taht too is in the etherpad | 12:49 |
| openstack | Launchpad bug 1642897 in os-client-config "osc commands fail when using admin_token plugin in keystoneauth1" [Undecided,New] | 12:49 |
| panda | amoralej looks at the weirdo results that finish at 4am, and when I by the time I discover at the same failures at 10:30 when periodic jobs finish, he's usually already done the root cause analysis :) | 12:51 |
| marios | shardy: sorry, i was afk and only just remembered the ping from you last night, getting setup and will read back | 12:51 |
| amoralej | we've been hitting it since yesterday in rdo-ci panda | 12:52 |
| panda | amoralej: ah, ok. I think this is good, rdo ci catching things a lot earlier. | 12:55 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-puppet-elements: Add qpid-dispatch-router to overcloud-controller element https://review.openstack.org/373489 | 12:55 |
| weshay | panda, amoralej ya.. that's why we have it run every 4 hours | 12:55 |
| marios | shardy: so did it answer your question (ie. yes in the current workflow the controlplane comes down, pcs cluster etc) | 12:55 |
| weshay | I don't have access to mark this as critical | 12:55 |
| amoralej | yeap it does its work | 12:55 |
| weshay | for tripleo | 12:56 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-heat-templates: Nothing to see here https://review.openstack.org/399583 | 12:56 |
| amoralej | i'm waiting for dtroyer to be online | 12:56 |
| weshay | panda, amoralej we need this escalated via the process in mojo | 12:56 |
| amoralej | but we may need to pin os-client-config back | 12:56 |
| amoralej | i was waiting to see if upstream awareness makes it work | 12:56 |
| *** abregman is now known as abregman|afk | 12:57 | |
| amoralej | and people seems not to be online yet | 12:57 |
| marios | 14:02 < shardy> bandini: Ok, I'm interested in making this work for the non pacemaker case, so I assume it will be 1. stop haproxy + keepalived, 2. stop all services, 3. yum update, 4. start db etc, 5. db sync, 6. start services | 12:57 |
| marios | shardy: sounds like the current workflow ^ | 12:57 |
| amoralej | do you think we should escalate it already weshay? | 12:57 |
| weshay | amoralej, yes.. because master hasn't passed in 2 weeks now | 12:57 |
| weshay | amoralej, ocata1 is now ish | 12:57 |
| amoralej | yea, we are chaining issues... | 12:58 |
| amoralej | i will send the mail | 12:58 |
| weshay | amoralej, https://dashboards.rdoproject.org/rdo-dev | 12:58 |
| amoralej | i know, i know, don'm make me look at those reds... | 12:58 |
| weshay | amoralej, panda if that said 3-5 days vs. 14d it wouldn't be as critical | 12:58 |
| panda | weshay: ocata-1 was released yesterday | 12:58 |
| weshay | but we also have to get a passing job for an upcoming test day | 12:58 |
| weshay | panda, ya.. but rdo hasn't released ocata-1 | 12:59 |
| weshay | because we don't have a build | 12:59 |
| *** pkovar has joined #tripleo | 13:02 | |
| *** pgadiya has quit IRC | 13:02 | |
| panda | amoralej: mind if I send the escalation ? | 13:02 |
| amoralej | ok, no problem, i was writing it but if you have it, send it | 13:03 |
| *** jkilpatr has quit IRC | 13:03 | |
| *** dsneddon is now known as dsneddon_afk | 13:04 | |
| jpich | The backport at https://review.openstack.org/#/c/396523/ is green and has several +2s, if someone would like to give it the missing +A | 13:06 |
| *** morazi has joined #tripleo | 13:09 | |
| *** jeckersb is now known as jeckersb_gone | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:11 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:11 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 13:11 |
| *** ooolpbot has quit IRC | 13:11 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 13:11 |
| panda | weshay: amoralej ehre's already a proposed patch: https://review.openstack.org/#/c/398917 | 13:13 |
| *** jayg|g0n3 is now known as jayg | 13:13 | |
| amoralej | yeah, i've seen it | 13:14 |
| *** jkilpatr has joined #tripleo | 13:14 | |
| *** trown|outtypewww is now known as trown | 13:15 | |
| amoralej | we need someone from the osc team to look at it | 13:16 |
| mhenkel | hi All, quick question on enabling neutron dhcp for internal_api and mangement networks: | 13:17 |
| *** prateek has joined #tripleo | 13:17 | |
| mhenkel | is it sufficient to set InternalApiNetEnableDHCP and ManagementNetEnableDHCP to true? | 13:17 |
| mhenkel | I did that in my env file but my neutron networks are created without dhcp | 13:18 |
| mhenkel | is there anything else I need to do? | 13:18 |
| jkilpatr | so is overcloud prep config now required for overcloud deployments but has not been added to the default extras file in quickstart? | 13:21 |
| jkilpatr | fun | 13:21 |
| panda | jkilpatr: ? | 13:23 |
| *** pradk has joined #tripleo | 13:24 | |
| jkilpatr | panda, just chasing down an error it looks like the network-environmental templating task has been moved to a new role that's not landed yet in some places. | 13:24 |
| panda | jkilpatr: I see it in quickstart-extras-requirements.txt | 13:24 |
| panda | jkilpatr: you have a link to the error ? | 13:24 |
| *** flepied has quit IRC | 13:25 | |
| jkilpatr | panda, it just says the role isn't there maybe i should check more carefully then. | 13:25 |
| panda | weshay: amoralej escalation sent | 13:26 |
| jkilpatr | panda, ok I think I see the issue, I was trying to add it to my playbook but the example on the github as the full role name instead of the shortened name | 13:27 |
| panda | jkilpatr: ok, beware that from monday (probably) all roles will lose the ansible-role-tripleo- prefix | 13:28 |
| *** rain has joined #tripleo | 13:30 | |
| *** rain is now known as leanderthal | 13:30 | |
| *** lucas-hungry is now known as lucasagomes | 13:31 | |
| *** jkilpatr has quit IRC | 13:31 | |
| *** rlandy has joined #tripleo | 13:33 | |
| *** jbadiapa has quit IRC | 13:34 | |
| shardy | jaosorior: was there a bug raised ref https://review.openstack.org/#/c/398127/ and https://review.openstack.org/#/c/398128 ? | 13:35 |
| shardy | I thought there was one but it's not linked from the patches | 13:35 |
| *** ccamacho|lunch is now known as ccamacho | 13:36 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add verify required and CA bundle to haproxy https://review.openstack.org/399591 | 13:40 |
| *** fultonj has quit IRC | 13:40 | |
| *** flepied has joined #tripleo | 13:41 | |
| *** fultonj has joined #tripleo | 13:42 | |
| *** jaosorior has quit IRC | 13:42 | |
| *** amoralej is now known as amoralej|lunch | 13:42 | |
| *** jbadiapa has joined #tripleo | 13:43 | |
| *** Vijayendra_ has joined #tripleo | 13:46 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/399595 | 13:47 |
| *** jpena is now known as jpena|lunch | 13:47 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/399595 | 13:47 |
| *** dtantsur|brb is now known as dtantsur | 13:48 | |
| *** Vijayendra has quit IRC | 13:48 | |
| *** abregman|afk is now known as abregman | 13:49 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Use keystone profile parameter to pass heat password https://review.openstack.org/399599 | 13:51 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Remove explicit hiera calls for heat in keystone profile https://review.openstack.org/399595 | 13:51 |
| shardy | slagle, jaosorior: ^^ | 13:51 |
| matbu | shardy: ho i wonder, i figured out the badstatusline thing on my env. I discovered during the upgrade of the undercloud to newton release, puppet was setting all the workers of the UC services to 2 | 13:53 |
| matbu | shardy: which was really low sized, so i override this value with a bigger one | 13:53 |
| hewbrocca | matbu: wait, is that the bug that's been blocking CI? | 13:54 |
| shardy | matbu: interesting - it should be the number of cores in most cases I think | 13:54 |
| shardy | amoralej|lunch: ^^ | 13:54 |
| hewbrocca | I thought we pushed it down to fix undercloud memory issues | 13:55 |
| *** lblanchard has joined #tripleo | 13:55 | |
| trown | ya increasing workers will increase memory pressure for sure | 13:55 |
| *** nyechiel has joined #tripleo | 13:55 | |
| matbu | hewbrocca: yep, but the "badstatusline" is really generic python error | 13:56 |
| hewbrocca | Right | 13:56 |
| matbu | hewbrocca: some maybe there was differents/severals errors behind | 13:56 |
| shardy | hewbrocca: we did tune a few things down a bit, but for the undercloud heat, two workers is not enough given the hundreds of stacks we're throwing at it | 13:56 |
| hewbrocca | no clearly not | 13:56 |
| hewbrocca | no wonder the API response is so slow... | 13:56 |
| matbu | shardy: ack, i wasn't sure of the expecting behavior, the number of cores should have been 8 .. i set i to 6 and it worked fine | 13:56 |
| hewbrocca | rook: you around? | 13:57 |
| shardy | yeah, we shouldn't have gone that low, especially for heat - we put a minimum of 4 in the heat codebase a while back because of these sorts of issues | 13:57 |
| shardy | but if you pass an explicit 2 we'll respect it | 13:57 |
| shardy | so yeah, my local box only has two heat-engine workers | 13:57 |
| shardy | num_engine_workers = 2 | 13:58 |
| shardy | that'll do it | 13:58 |
| matbu | shardy: but i think it depend also of the box itself. i have a local box with a ssd and it's pretty fast , even with a low size of workers | 13:58 |
| trown | hmm so we need to patch instack-undercloud hiera for heat workers? | 13:58 |
| shardy | well it appears someone already has, sec | 13:58 |
| shardy | the default should be the number of cores, or 4, whichever is more | 13:58 |
| trown | ya, I have never been able to reproduce this issue on my dev env with ssd | 13:58 |
| hewbrocca | At some point we thought it was OK to make it 2 | 13:59 |
| trown | well, we made it 2 or (total CPU)/2 for all services in puppet | 14:00 |
| hewbrocca | oh dear | 14:00 |
| trown | I think that is still fine, but for heat on the undercloud seems we need to override that | 14:00 |
| shardy | https://review.openstack.org/#/q/I9ed855648e23b0a7e452e6a840a92779fa3f6d48 | 14:00 |
| shardy | Ok so we'll need to revisit that I think | 14:00 |
| hewbrocca | Sounds that way | 14:00 |
| matbu | hewbrocca: when i hit memory issues on the undercloud in upgrade, i used to set "number of core / 2 == heat_engine_workers" | 14:00 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-heat-templates: Make Ceilometer notifications non-blocking https://review.openstack.org/391985 | 14:00 |
| hewbrocca | I'm all about saving memory on the overcloud nodes | 14:01 |
| hewbrocca | but frankly | 14:01 |
| hewbrocca | I think we should test with as big an undercloud as we need | 14:01 |
| hewbrocca | and no bigger | 14:01 |
| hewbrocca | :) | 14:01 |
| trown | ya that is not a static quantifiable thing though :P | 14:01 |
| hewbrocca | If that means we need an undercloud with 32GB RAM to get a proper test | 14:01 |
| hewbrocca | fine | 14:01 |
| matbu | "The os_workers fact will be 2 for unless the cpu count is greater than 8 with an incremental increase of 1 worker for every 4 processors until 32 processors." | 14:02 |
| *** abregman has quit IRC | 14:03 | |
| trown | I wonder if just giving undercloud 8 vcpus in CI would work | 14:03 |
| trown | the os_workers default feels sane for production | 14:03 |
| trown | so long as we say 8 cores is minimum for undercloud | 14:04 |
| shardy | trown: amoralej|lunch said he'd also hit this with an 8vcpu env, so I think the main issue is the bottleneck of workers for some services we hit hard | 14:04 |
| *** artom has quit IRC | 14:04 | |
| shardy | IME 4vcpus is fine provided you also have 4 heat-engine workers | 14:04 |
| shardy | mwhahaha: Hey, we're discussing os_workers | 14:05 |
| mwhahaha | ? | 14:06 |
| shardy | mwhahaha: turns out vcpus/2 doesn't work for some services, particularly heat where we probably want number of CPUs up to some limit, perhaps 8 | 14:06 |
| trown | hmm, ya I guess uniform workers maybe doesnt work for undercloud | 14:06 |
| matbu | shardy: trown maybe we should just documented this by saying : if you have 8 cores or less on this UC, you should override the workers to 4 or 6 | 14:06 |
| mwhahaha | It's tunable | 14:06 |
| *** ctayal has joined #tripleo | 14:06 | |
| shardy | mwhahaha: in the past we hit issues in CI due to the bottlneck of throwing huge stacks at heat with small number of workers, and https://review.openstack.org/#/c/387523 took us from a minimum of 4 to 2 | 14:06 |
| shardy | mwhahaha: cool, that was my question :) | 14:07 |
| *** ctayal has quit IRC | 14:07 | |
| shardy | e.g can we tune this, or should we just go back to the heat calculated default | 14:07 |
| mwhahaha | Tune | 14:07 |
| mwhahaha | The defaults are bad | 14:07 |
| *** ctayal has joined #tripleo | 14:07 | |
| mwhahaha | The number of CPUs is not a good default | 14:07 |
| *** tiswanso has joined #tripleo | 14:07 | |
| mwhahaha | You could osworkers *2 for heat | 14:08 |
| trown | that is a good idea | 14:08 |
| shardy | mwhahaha: for heat, we want the number of CPUs up to a limit of 8, but with a minimum of 4 - what's the cleanest way to do that? | 14:08 |
| shardy | yeah, we can't do that in the hiera interpolation tho, can we? | 14:08 |
| mwhahaha | Not sure I'd have to test it | 14:08 |
| mwhahaha | I'm on pto today so I'm not at a computer at the moment | 14:09 |
| shardy | mwhahaha: ack, OK we'll figure it out | 14:09 |
| shardy | thanks | 14:09 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: [WIP] first pass at composable roles, this is just the config https://review.openstack.org/399609 | 14:09 |
| mwhahaha | Os workers is num CPUs / 4 | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 14:10 |
| mwhahaha | With a cap of 8 so if you * 2 you should get some decent defaults. The issue is for CPU GPU t of <4 | 14:10 |
| shardy | Ok for now I'm going to stop us setting heat::engine::num_engine_workers: "%{::os_workers}" | 14:10 |
| shardy | that's a quick fix for CI | 14:10 |
| shardy | then we can work out the capping afterwards | 14:10 |
| hewbrocca | shardy: +1 | 14:10 |
| mwhahaha | Why not just override it in the CI environment | 14:11 |
| matbu | shardy: we can just override the value an extra hieradata file | 14:11 |
| mwhahaha | Yea that | 14:11 |
| shardy | mwhahaha: because it's also hitting folks in the development environments | 14:11 |
| matbu | with hieradata_override = | 14:11 |
| matbu | in undercloud.conf | 14:11 |
| shardy | (and we've got three different CI systems all breaking with this) | 14:12 |
| mwhahaha | So if you increase it your going to hit Mem usage issues | 14:12 |
| mwhahaha | So you need to figure out which is worse | 14:12 |
| mwhahaha | Especially cause heat is the #1 Mem user | 14:13 |
| rook | hewbrocca: sup? | 14:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Only start the deploy if the Heat stack isn't already in progress https://review.openstack.org/398959 | 14:13 |
| shardy | mwhahaha: Yeah, we've got to find a balance for sure | 14:13 |
| mwhahaha | alternatively drop the workers logic into puppet-tripleo | 14:14 |
| shardy | but we know 4 is a good minimum for heat in this environment based on previous issues | 14:14 |
| hewbrocca | rook: just curious -- on your OSP10 scale testing | 14:14 |
| hewbrocca | What do you set Heat workers to on the undercloud? | 14:14 |
| shardy | mwhahaha: yeah, the problem is the undercloud tho, and we're not yet using puppet-tripleo there | 14:15 |
| *** abregman has joined #tripleo | 14:15 | |
| trown | I kind of like the hiera override as the temp solution... we only need to do that in tripleo-ci and tripleo-quickstart, both of which already have a default hiera override file | 14:15 |
| shardy | I can wire it in via puppet-stack-config.pp tho | 14:15 |
| trown | and are not branchless | 14:15 |
| trown | otherwise we have to make an instack-undercloud change and backport it... then if we fix in some better way later we would have to again fix and backport? | 14:16 |
| trown | s/and are not branchless/and ARE branchless/ | 14:16 |
| gfidente | rook :) | 14:16 |
| mwhahaha | %{scope('::os_workers') * 2} | 14:17 |
| *** hjensas has quit IRC | 14:17 | |
| mwhahaha | Should work by the way | 14:17 |
| openstackgerrit | yolanda.robla proposed openstack-infra/tripleo-ci: Enable consuming packages for a feature branch https://review.openstack.org/399562 | 14:17 |
| trown | mwhahaha: thanks. shardy, I can submit that ^ | 14:18 |
| mwhahaha | You would get a min of 4 all the time with some sliding scale up to 16 | 14:18 |
| *** Goneri has joined #tripleo | 14:18 | |
| shardy | mwhahaha, trown: OK sounds good, go for it :) | 14:19 |
| shardy | thanks for the help mwhahaha | 14:19 |
| trown | shardy: we only need to do that for engine workers and not api right? | 14:19 |
| trown | ya thanks mwhahaha... now go back to PTO :) | 14:19 |
| shardy | trown: Yeah I'd start with the engine workers | 14:19 |
| *** cdearborn has joined #tripleo | 14:20 | |
| *** dsariel has joined #tripleo | 14:20 | |
| gfidente | shardy kind of related note | 14:22 |
| gfidente | if we batch create of ::Server, then ::ResourceGroup will still wait for all members to come up before starting, right? | 14:23 |
| openstackgerrit | Merged openstack/instack-undercloud: correctly spell yaql_limit_iterators https://review.openstack.org/398396 | 14:23 |
| openstackgerrit | Merged openstack/os-net-config: Stop dhclient in os-net-config if interface not set for DHCP https://review.openstack.org/398498 | 14:23 |
| beagles | \o/ ^^ | 14:23 |
| rook | hewbrocca default 1:1 cpu:worker | 14:23 |
| rook | gfidente: ? :) | 14:24 |
| jrist | flaper87: of course I can look but I think you mean jistr :) | 14:24 |
| gfidente | rook not sure if you saw the email from Ben but sounds like we're going to reprise that, probably together | 14:24 |
| gfidente | or at least /me would like to | 14:24 |
| jrist | matbu: I was playing destiny last night and there was someone named matbu so I thought it was you | 14:25 |
| rook | reprise what gfidente ? (i haven't seen it, still on leave, but lurking) | 14:25 |
| gfidente | rook but I think if you could reply adding some details that would be cool | 14:25 |
| gfidente | rook ah didn't know you were out, sorry | 14:25 |
| rook | hewbrocca shardy is the discussion to reduce heat workers? | 14:26 |
| matbu | jrist: hehe nop, it wasn't me :) | 14:26 |
| hewbrocca | rook: the other way | 14:26 |
| derekh | shardy: the 40 node deployment appears to be stalled, I'm seeing plenty of errors in signals being sent too heat-api-cfn and I think 2 compute nodes think they have nothing to do but heat is waiting for a signal | 14:26 |
| derekh | shardy: http://chunk.io/f/81a87c10941745de8a7df8c9ce3a397b | 14:27 |
| hewbrocca | rook: looks like we have been setting Heat workers too low in CI which is causing performance problems | 14:27 |
| shardy | derekh: ack, ref discussion above, can you please check num_engine_workers on the undercloud? | 14:27 |
| shardy | derekh: and how many cpus does the undercloud VM have? | 14:27 |
| derekh | shardy: num_engine_workers = 2 | 14:28 |
| *** jeckersb_gone is now known as jeckersb | 14:28 | |
| derekh | shardy: 8xvCPU | 14:28 |
| rook | #num_engine_workers = <None> | 14:28 |
| *** nyechiel has quit IRC | 14:29 | |
| *** ccamacho has quit IRC | 14:29 | |
| rook | oh, this must be quickstart? | 14:29 |
| shardy | derekh: ack - OK that may be part of the problem - I'd suggest increasing that to num_engine_workers = 4 at least (perhaps even 6 or 8) | 14:29 |
| rook | mine isn't quickstart. | 14:29 |
| derekh | shardy: http://paste.openstack.org/show/589718/ | 14:29 |
| shardy | derekh: https://review.openstack.org/#/c/387523/ reduced the number of workers accross the board, which did reduce memory usage, but I think is the cause of some of our performance problems | 14:29 |
| derekh | rook: I'm not trying this with quickstart either | 14:29 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Increase the default number of workers for heat engine https://review.openstack.org/399619 | 14:30 |
| rook | hm, must be newer code then what I have | 14:30 |
| rook | ah, i see the change shardy references... looking | 14:30 |
| derekh | shardy: ok, I'm gonna quick this attempt and start a new deploy | 14:30 |
| *** ccamacho has joined #tripleo | 14:30 | |
| derekh | shardy: nothing has happened in 40 minutes | 14:30 |
| *** derekjhyang has quit IRC | 14:31 | |
| derekh | *quit | 14:31 |
| trown | derekh: https://review.openstack.org/399619 is the patch to change default to #CPU for heat engine rather than #CPU/2 | 14:31 |
| trown | though... with 8vCPU undercloud... shouldnt engine workers be set to 4 with os_workers? | 14:31 |
| shardy | derekh: Ok, there may be other issues, but I'm sure that engine count won't work for such a large deployment | 14:31 |
| shardy | trown: no, it's cpus/4 | 14:32 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 14:32 |
| shardy | "(number of cpus/4) or 2 but is capped at 8" | 14:32 |
| derekh | ya, mine is set to cpu/4 | 14:32 |
| rook | how did we determine 8 was a sweet spot? | 14:32 |
| trown | ah ok... my commit message is slightly wrong then | 14:33 |
| *** numans has quit IRC | 14:33 | |
| * rook wonders if we consulted with each of the teams (storage,networking,etc) to come up with 8 | 14:33 | |
| shardy | rook: I don't think we did | 14:33 |
| * mwhahaha points to a mailing list message with no feedback | 14:33 | |
| shardy | rook: a while back we hit RPC timeouts using 2 heat workers, and at that time increasing the minimum to 4 in heat fixed it | 14:33 |
| rook | ok... I know for a fact that < workers for neutron == less performance. | 14:34 |
| mwhahaha | so this was tested with other services and this is just the default, it's tunable. but the goal was not to use $::processorcount anymore | 14:34 |
| rook | glance might not be a huge hit... Swift -- my guess would be a hit. | 14:34 |
| mwhahaha | due to the fact that on baremetal that's terible | 14:34 |
| shardy | mwhahaha: yup, understood | 14:34 |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Increase the default number of workers for heat engine https://review.openstack.org/399619 | 14:34 |
| rook | mwhahaha so, not just with the UC, but OC this is changing? | 14:34 |
| gfidente | rook shardy though from mulitple parties I hear that batching is good practice | 14:34 |
| mwhahaha | so it's the default in the puppet modules | 14:34 |
| shardy | mwhahaha: it's just that we've been burned by this before - we know that certain services on the undercloud are hit very hard, so we can't tune them down too far | 14:35 |
| rook | correct mwhahaha | 14:35 |
| gfidente | and was experimenting with different places where it could be used | 14:35 |
| mwhahaha | which THT controls | 14:35 |
| rook | it isn't just processes, but the # of open DB connections. | 14:35 |
| mwhahaha | on the UC we lowered it because of memory issues since we have 17 different services runing | 14:35 |
| rook | sure | 14:35 |
| mwhahaha | so using 17 * $::processorcount = bad time | 14:35 |
| rook | i mean, look at the IO we see on the UC | 14:36 |
| rook | what are you going to do about that? | 14:36 |
| trown | I think the only services that are hit hard on UC are swift and heat though | 14:36 |
| trown | and swift is only in a couple bursts | 14:36 |
| rook | well the burts can be quite large. | 14:36 |
| rook | ie 40 node deployment at once. | 14:37 |
| mwhahaha | the only thing we tune for swift is proxy | 14:37 |
| shardy | Yeah, I think just special-casing the new defaults for heat and perhaps swift should work OK | 14:37 |
| trown | for neutron it is really just an IP manager on UC, so the lower default is good | 14:37 |
| rook | trown I agree -- we only spawn 10 at a time, so interface creation should be low. | 14:38 |
| mwhahaha | so for ref, https://review.openstack.org/#/c/386696/ that's what we tuned down | 14:38 |
| mwhahaha | we don't tun swift server, only proxy | 14:38 |
| rook | mwhahaha: i _think_ the proxy is the biggest hitter though -- i could be wrong though, I have only played a little bit with object-stores | 14:39 |
| * derekh kicks off a new 40 node deployment | 14:39 | |
| rook | derekh 3 controllres, 40 compute? | 14:39 |
| mwhahaha | i would assume swift server is the IO | 14:40 |
| rook | mwhahaha yup, that & provisioning | 14:40 |
| derekh | rook: yup | 14:40 |
| rook | derekh so, I had to bump timeouts with previous newton releases. | 14:40 |
| mwhahaha | so we could tune swift server, but we dont and the upstream module has 1 as the default and i think that's for $reasons | 14:40 |
| rook | ah derekh initial deployment is good, nm | 14:41 |
| rook | derekh: https://gist.github.com/jtaleric/4f422ccbf89d7c413d68e0d3cdbaabbf | 14:41 |
| mwhahaha | https://github.com/openstack/puppet-swift/blob/master/manifests/storage/server.pp#L54 | 14:41 |
| derekh | rook: I'm deploying with "-t 480 " , were there others you had to increase? | 14:41 |
| rook | derekh: solid. | 14:42 |
| *** amoralej|lunch is now known as amoralej | 14:42 | |
| *** bnemec has joined #tripleo | 14:42 | |
| *** abehl has quit IRC | 14:42 | |
| derekh | rook: btw this is all on virt (using OVB) so not exactly replicating a production deployment | 14:44 |
| derekh | but I think worth looking at at least | 14:44 |
| rook | derekh have the steps to do OVB work? | 14:47 |
| rook | derekh: I would like to compare : real deployments to ovb. | 14:47 |
| trown | bnemec: was just about to +A https://review.openstack.org/#/c/399146/ for the same reason :) | 14:47 |
| derekh | rook: not sure what your question meant | 14:48 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Adds basic internationalization support https://review.openstack.org/399626 | 14:48 |
| rook | derekh I think there is more overhead with real deployments (ie, IPMI flakeyness) -- so using OVB might be good for consistency. | 14:48 |
| *** bnemec is now known as beekneemech-semi | 14:48 | |
| *** beekneemech-semi is now known as bnemec-semi-here | 14:49 | |
| rook | derekh so you are using OVB for your deployment, do you have a document/etherpad/napkin-with-nodes on how to set that up. | 14:49 |
| * bnemec-semi-here needs to document --quintupleo for ovb | 14:50 | |
| derekh | rook: possibly , were still doing ipmi but not against hardware with potential problems, and no POST time so thats a plus | 14:50 |
| rook | oh trust me.. IPMI on Dell vs Supermicro is night and day | 14:50 |
| derekh | rook: you first need access to a OVB cloud, then follor the steps in the OVB repo https://github.com/cybertron/openstack-virtual-baremetal | 14:51 |
| rook | well, how do i create said ovb cloud | 14:51 |
| rook | derekh lets say I wnt to create one in the scale lab. | 14:51 |
| *** jpena|lunch is now known as jpena | 14:51 | |
| *** rook is now known as rook-baby | 14:51 | |
| *** charliejllewelly has joined #tripleo | 14:51 | |
| derekh | rook: I've also put together instructions of using RH2 here for some developers that use it https://etherpad.openstack.org/p/tripleo-devenvs | 14:52 |
| *** abregman is now known as abregman|afk | 14:52 | |
| derekh | rook-baby: the README in that repo give details on how to set it up | 14:52 |
| *** chlong has joined #tripleo | 14:52 | |
| jrist | florianf: : could I pull down your patch and have some strings i18n? | 14:53 |
| derekh | rook-baby: I'll send you the etherpad with the notes on how we setup RH1 and RH2 | 14:53 |
| florianf | jrist: It's still WIP. But if you want to review/test, type localStorage.setItem('language', 'fr') in your console and reload the page. some strings will be marked with 'FR' | 14:55 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Add tuned check to remote provision role https://review.openstack.org/396362 | 14:55 |
| jrist | !!! | 14:55 |
| openstack | jrist: Error: "!!" is not a valid command. | 14:55 |
| dtantsur | folks, do we always use swift as glance backend in undercloud (similar question for overcloud)? | 14:55 |
| jrist | florianf: cool | 14:55 |
| jrist | florianf: gonna check it out shortly | 14:55 |
| trown | dtantsur: ya, except in the overcloud on some multinode scenario jobs where we dont deploy swift | 14:56 |
| jpich | florianf: How goes getting the dependency rpm updated? Or will it work with the older version of the library for now? | 14:56 |
| dtantsur | trown, ack, thanks. some ironic drivers imply glance backed by swift, hence my question. | 14:56 |
| florianf | jpich: It does work with the older version. I thought it might be a good idea to update the dependency as part of a more general review of all our current dependency versions. | 14:57 |
| *** panda is now known as panda|bbl | 14:58 | |
| jpich | florianf: Cool! Review sounds good to me, though isn't that a massive task?? | 14:58 |
| amoralej | shardy, trown, so tunning heat workers may help in the badlinestatus issue | 14:58 |
| trown | amoralej: did you try it out? | 14:59 |
| hewbrocca | amoralej: so we think | 14:59 |
| amoralej | yeah, make sense | 14:59 |
| amoralej | after i hit the error yesterday in my test machine | 14:59 |
| florianf | jpich: probably. but can we get around it? we have our deps pretty tightly managed, so at some point we need to check for useful updates, I guess... | 14:59 |
| amoralej | i've been running it in a loop | 14:59 |
| amoralej | about 8-10 times | 15:00 |
| amoralej | with haproxy and openstack overcloud deploy running in debug and didn't hit again | 15:00 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add tuned check to remote provision role https://review.openstack.org/396362 | 15:00 |
| amoralej | with the same server and undercloud image | 15:00 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Pass the libvirt_uri to the pool-define command https://review.openstack.org/399141 | 15:00 |
| trown | amoralej: I put up https://review.openstack.org/399619 to increase heat engine workers | 15:01 |
| jpich | florianf: Among other things, yeah... | 15:01 |
| jpich | florianf: Let's open a bug to track this, if it can/should be done outside of the blueprint? | 15:01 |
| amoralej | so in a 8vcpus system we'd move from 2 to 4 workers? | 15:02 |
| trown | amoralej: also, I think https://review.openstack.org/#/c/396362/ will make it so we get similar performance in all CI nodes rather than 3/4 being much slower | 15:02 |
| trown | amoralej: yep | 15:02 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Add qpid-dispatch-router to overcloud-controller element https://review.openstack.org/373489 | 15:02 |
| openstackgerrit | Merged openstack/instack-undercloud: Increase Mistral Task Size limit https://review.openstack.org/396523 | 15:02 |
| shardy | amoralej: yep, minimum of 4, possibly more for large deployments and/or if you have a lot of ram | 15:02 |
| florianf | jpich: good idea. yeah, outside the blueprint sounds fine to me. It's really not needed to implement what we need. | 15:02 |
| jpich | florianf: Cool :) | 15:04 |
| amoralej | trown, i think the tuned profile will help also, yes | 15:04 |
| flaper87 | my quickstart run is failing to boot the undercloud because the pool disk is owned by root: http://paste.openstack.org/show/589722/ has this happened to other folks? | 15:08 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add blockstorage to default node flavor https://review.openstack.org/396577 | 15:08 |
| flaper87 | I'm trying to get quickstart to install tripleo on f24 | 15:08 |
| flaper87 | I know it's not supported but I'm working on that as I try this | 15:08 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 15:10 |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| shardy | flaper87: https://review.openstack.org/#/c/384892/ looks related? | 15:10 |
| flaper87 | yeah, might be | 15:11 |
| trown | flaper87: hmm, must be something different with fedora... my env has the pool owned by stack user | 15:12 |
| trown | flaper87: did you run quickstart.sh as root user? | 15:13 |
| *** ctayal_ has joined #tripleo | 15:13 | |
| flaper87 | trown: no, it seems to be related to https://review.openstack.org/#/c/384892/ | 15:14 |
| trown | hmm... actually that should not matter for this issue I think | 15:14 |
| flaper87 | trown: :D | 15:14 |
| dtrainor | dciabrin, sure, i'll be doing another deployment shortly | 15:15 |
| *** ctayal has quit IRC | 15:15 | |
| trown | flaper87: I wonder if virsh is behaving differently on fedora vs centos when volumes are created | 15:18 |
| trown | flaper87: this task in particular seems like it would create the volume as non-root user if run as non-root user | 15:19 |
| trown | https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/main.yml#L70-L78 | 15:19 |
| flaper87 | trown: trying something out but that might actually be the case | 15:19 |
| flaper87 | trown: just added become/become_user to these two https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/undercloud/tasks/main.yml#L146-L164 | 15:20 |
| flaper87 | let's see if that has any effect | 15:20 |
| flaper87 | otherwise, I'd say it's virsh's fault and need to figure out what it's doing differently | 15:20 |
| *** abregman|afk has quit IRC | 15:22 | |
| trown | flaper87: ya checking on a fedora box if the the pool gets created differently with the same xml | 15:23 |
| *** prateek has quit IRC | 15:23 | |
| flaper87 | trown: adding become did nothing | 15:25 |
| *** prateek has joined #tripleo | 15:26 | |
| *** jlinkes has joined #tripleo | 15:27 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-common: Add python-memcached to agent container. https://review.openstack.org/398582 | 15:29 |
| *** abregman has joined #tripleo | 15:29 | |
| trown | flaper87: hmm... just manually run pool creation steps as a non-root user on F23 resulted in pool directory and volumes being owned by that non-root user | 15:29 |
| *** prateek has quit IRC | 15:30 | |
| flaper87 | T_T | 15:32 |
| *** andrey-mp has joined #tripleo | 15:34 | |
| *** jcoufal has joined #tripleo | 15:34 | |
| andrey-mp | Hi! I'm trying to deploy Newton with tripleo to CentOS7. But my deployment stucks on os-collect-config. It tries to run something that uses 'lsb_release' tool. But this tool is absent. When I install 'redhat-lsb-core' then os-collect-config ends and deploy goes further. | 15:37 |
| andrey-mp | What is my mistake here? | 15:37 |
| *** rook-baby is now known as rook | 15:37 | |
| pradk | can i get some reviews on https://review.openstack.org/#/c/396439/ and https://review.openstack.org/#/c/396435/ | 15:38 |
| andrey-mp | How I can build an overcloud image with this tool? | 15:38 |
| rook | derekh++ | 15:38 |
| pradk | please* | 15:38 |
| shardy | andrey-mp: how did you build your image? My local overcloud image contains that package | 15:39 |
| andrey-mp | shardy: with this instruction - http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html | 15:40 |
| andrey-mp | shardy: code is here - https://github.com/cloudscaling/redhat-kvm/blob/master/__undercloud-install-2-as-stack-user.sh#L37 | 15:40 |
| *** markmc` is now known as markmc | 15:43 | |
| rook | shardy: mwhahaha hewbrocca so the thought is to increase # of workers because of failed deployments? -- sorrytrying to catch up on the convo. | 15:43 |
| rook | shardy mwhahaha hewbrocca I thought we were hitting mem issues. | 15:43 |
| rook | with too many workers | 15:43 |
| *** panda|bbl is now known as panda | 15:44 | |
| shardy | rook: yeah, but now we're hitting response issues because of too few workers ;) | 15:44 |
| *** abregman is now known as abregman|afk | 15:44 | |
| shardy | can't satisfy both constraints unfortunately | 15:44 |
| *** hoobaman has quit IRC | 15:44 | |
| openstackgerrit | Merged openstack/tripleo-common: Fernet Key management https://review.openstack.org/397381 | 15:44 |
| shardy | rook: the compromise is probably to increase at least the heat-engine workers, despite the increase in memory usage | 15:45 |
| shardy | rook: FWIW the heat memory usage issues which IIRC prompted this were largely resolved late in newton | 15:45 |
| shardy | http://people.redhat.com/~shardy/heat/plots/heat_before_after_end_newton.png | 15:46 |
| panda | flaper87: if you add --teardown all, does it change anything ? It's not the first time I hear of this problem, and this usually work around it, until we find some peace to deal wit it properly. | 15:46 |
| rook | shardy: nice -- have those fixes landed in RDO? | 15:47 |
| shardy | rook: should have, they're all in stable/newton AFAIK | 15:47 |
| rook | shardy we just need to keep a eye on this. | 15:47 |
| mwhahaha | rook: it might be ok if we just up heat only. the total num procs was an issue | 15:47 |
| *** pradk has quit IRC | 15:47 | |
| rook | mwhahaha: it is the # of available workers to service requests. | 15:48 |
| rook | since we are hitting timeouts | 15:48 |
| rook | the other possibility is to increase the RPC timeout? | 15:48 |
| *** [1]cdearborn has joined #tripleo | 15:48 | |
| shardy | I think we already increased it to a very high value | 15:49 |
| shardy | the problem this time was RPC didn't time out, so haproxy did | 15:49 |
| rook | ah ha | 15:49 |
| rook | ok | 15:49 |
| rook | wtf | 15:49 |
| rook | HAProxy on the UC | 15:49 |
| shardy | increasing timeouts isn't a good solution IMO | 15:50 |
| rook | i agree | 15:50 |
| mwhahaha | yea i was reffering to the history behind the original worker decrease | 15:50 |
| rook | but trying to tune workers without data across the board is abd. | 15:50 |
| rook | bad | 15:50 |
| *** pcaruana has quit IRC | 15:50 | |
| *** HenryG has quit IRC | 15:50 | |
| mwhahaha | we had data | 15:50 |
| rook | for more than just heat mwhahaha ? | 15:50 |
| mwhahaha | yea | 15:50 |
| rook | because you are tuning across the board. | 15:50 |
| rook | alrighty | 15:50 |
| mwhahaha | from puppet and fuel | 15:51 |
| rook | i came into the convo late | 15:51 |
| mwhahaha | this is a many release thing | 15:51 |
| *** HenryG has joined #tripleo | 15:51 | |
| mwhahaha | the openstack defaults of proc count only works for small vms, aka devstack | 15:51 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-quickstart: Create directories with root https://review.openstack.org/384892 | 15:52 |
| mwhahaha | weve also done rounds of perf tests | 15:52 |
| rook | sure mwhahaha, this has bit us in the ass plenty of times. | 15:52 |
| rook | mwhahaha: yup, we have. | 15:52 |
| rook | mwhahaha we show that # of workers does increase response times with some services. | 15:52 |
| rook | but if you have 96 cores.... you don't really need 96 workers. | 15:52 |
| mwhahaha | anyway, afk. if you want to chat further feel free to hit me up next week | 15:52 |
| rook | but to limit to 8 might be too low. | 15:52 |
| *** kjw3 has joined #tripleo | 15:53 | |
| *** pradk has joined #tripleo | 15:54 | |
| *** ctayal_ has quit IRC | 15:54 | |
| *** chandankumar has quit IRC | 15:55 | |
| *** ctayal has joined #tripleo | 15:55 | |
| rook | shardy: so the change of # of workers is something pretty recent | 15:57 |
| rook | hm, 5 weeks ago | 15:57 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: [WIP] config for containerized-compute https://review.openstack.org/393348 | 15:57 |
| openstackgerrit | Merged openstack/puppet-tripleo: Replace hard-coded haproxy/keepalived coupling https://review.openstack.org/399152 | 15:59 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add worker config envs in toci_gate_test https://review.openstack.org/399146 | 15:59 |
| rbrady | honza: I looked into supporting logging a bit | 15:59 |
| bnemec-semi-here | andrey-mp: There was a bug in dib for newton that means lsb_release will be missing if you don't include ceph. The workaround is to include the ceph repo. | 16:00 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Restore worker configs for mitaka and below https://review.openstack.org/398652 | 16:00 |
| *** jprovazn has quit IRC | 16:01 | |
| rbrady | honza: trying to use what's already in zaqar (e.g. custom pipeline stage or custom syslog notifier) were seen somewhat as abusing zaqar | 16:01 |
| rbrady | honza: also, we'd run into the same problem with zaqar that we do for using mistral WRT to permissions or logging.conf | 16:01 |
| rbrady | honza: two other options left. a logging service in tripleo or some sort of logging service in openstack itself we could use for tripleo | 16:02 |
| *** cdearborn has quit IRC | 16:03 | |
| *** dsavineau has joined #tripleo | 16:05 | |
| *** ebarrera has quit IRC | 16:06 | |
| flaper87 | trown: what libvirt_uri did you use in your test? | 16:09 |
| *** Guest90313 has quit IRC | 16:09 | |
| trown | flaper87: qemu:///session | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 16:10 |
| flaper87 | trown: a-ha, could you try with qemu:///system ? | 16:10 |
| *** rajinir has joined #tripleo | 16:10 | |
| derekh | shardy: rook, keystone has been running along at 110%+ CPU for a while now and using its fair share of RAM (1.35G although it doesn't seem to be increasing any more) | 16:11 |
| trown | flaper87: hmm, then I would need to use sudo so I think it would be expected to have it owned by root | 16:11 |
| derekh | 18436 keystone 20 0 2409372 1.353g 6988 S 115.2 8.7 240:46.82 keystone-admin -DFOREGROUND | 16:11 |
| trown | flaper87: you are explicitly setting system in your quickstart config? | 16:12 |
| flaper87 | trown: I did because of a different bug, ok, think I found my issue | 16:12 |
| shardy | derekh: ouch, that doesn't sound too good | 16:12 |
| flaper87 | trown: the fix for that other bug landed already | 16:12 |
| trown | flaper87: ah that is maybe the patch I merged this morning :) | 16:12 |
| shardy | derekh: I noticed today that we're not enabling any cache backend for keystone | 16:12 |
| shardy | not sure if that could be related | 16:13 |
| *** bana_k has joined #tripleo | 16:13 | |
| derekh | shardy: maybe, logs suggest its currently getting about (or getting through about) 3 token requests a second | 16:14 |
| shardy | wow | 16:14 |
| shardy | well, at least you found something, hopefully we can figure out ways to make that a lot faster :) | 16:14 |
| andrey-mp | bnemec-semi-here: thanks! | 16:14 |
| derekh | shardy: and if I'm right these are connections to keystone queueing up? | 16:16 |
| derekh | [root@undercloud-scale httpd]# netstat -pn | grep -i 35357 | grep ESTABLISHED | grep http | wc | 16:16 |
| derekh | 46 322 4646 | 16:16 |
| *** bana_k has quit IRC | 16:18 | |
| *** penick has joined #tripleo | 16:18 | |
| shardy | derekh: looks like it | 16:19 |
| *** jlinkes has quit IRC | 16:19 | |
| yolanda | hi derekh , can i get your review on https://review.openstack.org/399562 ? | 16:19 |
| derekh | shardy: or maybe ignore that netstat, I could be counting thing twice | 16:19 |
| shardy | derekh: We pass the same token around to all the services, so they're all going to hit the DB to have the token authenticated | 16:19 |
| *** ramishra has quit IRC | 16:19 | |
| shardy | which is going to be particularly bad if we're letting keystone hit the db every time without caching I guess | 16:20 |
| *** bana_k has joined #tripleo | 16:20 | |
| shardy | actually | 16:20 |
| shardy | the caching settings I'm referring to are for the overcloud keystone, but I guess the same applies | 16:20 |
| *** ramishra has joined #tripleo | 16:21 | |
| derekh | yolanda: looking | 16:24 |
| *** achadha has joined #tripleo | 16:25 | |
| *** jpena is now known as jpena|brb | 16:27 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart: Remove OVB stack cleanup dependance on network isolation type https://review.openstack.org/399678 | 16:30 |
| *** tremble has quit IRC | 16:30 | |
| dtrainor | The guide on deploying an SSL Overcloud go in to pretty fine detail but omit some steps on finding some needed information. One thing that I'm running in to is needing a predictable IP for the IP-based SSL configuration. Where is the pool of Public VIPs located? http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/ssl.html | 16:33 |
| dtrainor | Public is not External, correct? | 16:33 |
| dtrainor | Specifically, I'm looking for the value - or list of available values - I can use for PublicVirtualFixedIPs | 16:34 |
| panda | derekh: in trieplo-ci, what's the part that actually creates the ovb stack ? testenv-client is launchin a gearman worker, but I can't find the function the worker is using to create the stack | 16:34 |
| openstackgerrit | Dimitri Savineau proposed openstack/puppet-tripleo: Allow neutron_options customization for dashboard https://review.openstack.org/397927 | 16:34 |
| derekh | panda: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/te-broker/start_workers.sh | 16:35 |
| panda | derekh: ack, thanks | 16:35 |
| derekh | panda: that starts X workers (but doesn't create the env), then when they are connected too this script creates the envs http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/te-broker/create-env | 16:36 |
| yolanda | hi derekh, we still cannot test because we do not have the package available for that | 16:37 |
| yolanda | a new feature/v2 branch needs to be added to packaging | 16:38 |
| *** larstobi has quit IRC | 16:40 | |
| hewbrocca | wild re keystone | 16:40 |
| hewbrocca | shardy: it's almost like we need a cloud operator to look at our undercloud settings and tell us what to improve... | 16:40 |
| hewbrocca | I wonder where we could find one of those | 16:41 |
| *** dsariel has quit IRC | 16:41 | |
| *** larstobi has joined #tripleo | 16:42 | |
| shardy | hewbrocca: heh, +1, although there's no one-size answer I'm sure we can do better | 16:42 |
| derekh | yolanda: ok, should we wait for it? or is this needed first? | 16:43 |
| yolanda | yep, we shall wait for it | 16:44 |
| *** achadha has quit IRC | 16:44 | |
| hewbrocca | Might be worth a mail to Graeme | 16:44 |
| *** achadha has joined #tripleo | 16:44 | |
| shardy | hewbrocca: yup - derekh do you want to drop him a mail with your findings and/or raise a bug with details? | 16:45 |
| openstackgerrit | Martin André proposed openstack/tripleo-common: Update container images to point to newton https://review.openstack.org/399687 | 16:46 |
| shardy | I was planning to raise one re the cache thing but wanted to understand the options better first | 16:46 |
| derekh | shardy: will open a bug | 16:47 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Only start the deploy if the Heat stack isn't already in progress https://review.openstack.org/398959 | 16:47 |
| derekh | shardy: looks like my deployment has stalled again, 12 minutes since anything output from the deploy command + keystone is no longer being hit as hard | 16:49 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping composable upgrades with Heat+Ansible https://review.openstack.org/393448 | 16:49 |
| derekh | shardy: and plenty of these http://paste.openstack.org/show/589737/ | 16:50 |
| shardy | derekh: Hmm, is that with the incresed heat engine workers? | 16:51 |
| derekh | shardy: yes | 16:51 |
| derekh | shardy: its set too 4, confirmed 4 are running and the parent | 16:52 |
| shardy | hrm, looks like the heat requests are still timing out tho, you'll probably see an RPC timeout in the heat logs associated with that | 16:52 |
| shardy | derekh: actually, we're polling swift | 16:52 |
| shardy | so it may be more workers are needed there too | 16:53 |
| shardy | but I'd expect a different error in that case | 16:53 |
| derekh | shardy: want access to this box, /me is supposed to be doing other things, I can try the swift thing first though | 16:53 |
| shardy | derekh: sure, if you can pm me details I'll take a look | 16:53 |
| derekh | 2 swift proxies running | 16:53 |
| shardy | Ok, I'd guess that's not enough, because all 40 boxes will be polling swift tempurls | 16:54 |
| hewbrocca | LOL | 16:55 |
| *** ccamacho has quit IRC | 16:58 | |
| *** rasca has quit IRC | 16:59 | |
| derekh | shardy: shoudl I bump up the swift proxy workers and try again? | 17:02 |
| d0ugal | This is a silly question, but after using tripleo.sh --delorean-build, how are others installing the rpm? | 17:02 |
| shardy | derekh: Yeah, best I can tell it's an error hitting the tempurl from the request collector in os-collect-config | 17:03 |
| derekh | shardy: ok, will try it out | 17:03 |
| shardy | Not sure what a reasonable number is, I'd probably try 8 if there's enough ram | 17:04 |
| *** ctayal has quit IRC | 17:05 | |
| *** jpena|brb is now known as jpena | 17:05 | |
| openstackgerrit | Dimitri Savineau proposed openstack/puppet-tripleo: Allow neutron_options customization for dashboard https://review.openstack.org/397927 | 17:06 |
| *** rickflare has quit IRC | 17:07 | |
| shardy | d0ugal: there are a few ways, you can either virt-customize it into the image, use deploy artifacts to install it, put it in a local yum repo and have the nodes update from it | 17:07 |
| derekh | shardy: rook https://bugs.launchpad.net/tripleo/+bug/1643006 | 17:08 |
| openstack | Launchpad bug 1643006 in tripleo "keystone maxed out during overcloud deploy" [Undecided,New] | 17:08 |
| shardy | d0ugal: or you can add a local repo to the undercloud yum.repos.d, and add that to OVERCLOUD_IMAGES_DIB_YUM_REPO_CONF when calling tripleo.sh --overcloud-images | 17:08 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 17:10 |
| derekh | shardy: bumping it too 8 | 17:10 |
| d0ugal | shardy: I want to install it on the undercloud - so I guess a local yum repo is what I want | 17:11 |
| *** charliejllewelly has quit IRC | 17:11 | |
| * d0ugal finally goes and learns something about yum | 17:12 | |
| *** panda is now known as panda|bbl | 17:12 | |
| amoralej | panda, we've reverted os-client-config pin to 1.22.0 for the osc issue | 17:15 |
| amoralej | panda|bbl ^ | 17:15 |
| amoralej | wehay ^ | 17:16 |
| derekh | shardy: the new deploy is running, feel free to check in on it later, I'll probably poke at it a bit over the weekend to see if I can find anything out | 17:16 |
| amoralej | i've launch promotion pipeline in rdo, let's see | 17:16 |
| panda|bbl | weshay: ^ | 17:16 |
| openstackgerrit | Christian Schwede proposed openstack/tripleo-quickstart: Rename objectstorage flavor to swift-storage https://review.openstack.org/399703 | 17:16 |
| shardy | derekh: ack will do, thanks! | 17:16 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: WIP: VirtualBMC support for tripleo-quickstart https://review.openstack.org/399704 | 17:17 |
| *** paramite has quit IRC | 17:18 | |
| *** hewbrocca is now known as hewbrocca_afk | 17:22 | |
| *** derekh has quit IRC | 17:24 | |
| *** yamahata has joined #tripleo | 17:25 | |
| *** dtantsur is now known as dtantsur|afk | 17:26 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Make the openvswitch 2.4->2.5 upgrade more robust https://review.openstack.org/399708 | 17:28 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping composable upgrades with Heat+Ansible https://review.openstack.org/393448 | 17:31 |
| *** lucasagomes is now known as lucas-afk | 17:32 | |
| *** Guest52285 is now known as mgagne | 17:33 | |
| *** mgagne has quit IRC | 17:33 | |
| *** mgagne has joined #tripleo | 17:33 | |
| *** mcornea has quit IRC | 17:37 | |
| *** trown is now known as trown|lunch | 17:44 | |
| *** jpich has quit IRC | 17:53 | |
| *** achadha has quit IRC | 17:54 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Properly set distro branch in DLRN when STABLE_RELEASE=newton https://review.openstack.org/399578 | 17:56 |
| *** cylopez has quit IRC | 17:57 | |
| *** chandankumar has joined #tripleo | 18:02 | |
| *** fzdarsky is now known as fzdarsky|afk | 18:02 | |
| *** dbecker has quit IRC | 18:04 | |
| *** lmiccini has quit IRC | 18:06 | |
| *** jpena is now known as jpena|off | 18:06 | |
| openstackgerrit | Brent Eagles proposed openstack/os-net-config: Add support for enabling hotplug on interfaces https://review.openstack.org/394660 | 18:09 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 18:10 |
| *** achadha has joined #tripleo | 18:11 | |
| *** ccamacho has joined #tripleo | 18:11 | |
| *** ccamacho has quit IRC | 18:11 | |
| weshay | lucas-afk++ | 18:13 |
| *** achadha_ has joined #tripleo | 18:13 | |
| *** achadha_ has quit IRC | 18:14 | |
| *** achadha_ has joined #tripleo | 18:14 | |
| *** achadha_ has quit IRC | 18:15 | |
| *** achadha has quit IRC | 18:15 | |
| *** achadha has joined #tripleo | 18:15 | |
| *** yamahata has quit IRC | 18:22 | |
| dtrainor | Hi. I'm getting a failed deployment when trying to set PublicVirtualFixedIPs for an ip-based SSL Overcloud deployment. The error I'm getting is: Resource CREATE failed: InvalidIpForNetworkClient: resources.PublicVirtualIP.resources.ExternalPort: IP address 10.12.148.193 is not a valid IP for any of the subnets on the specified network. Neutron server returns request_ids: ['req-1fde4302-a5d4-483b-9109-19f2e8a5a6e7'] | 18:41 |
| dtrainor | Then IP that I'm using (10.12.148.193) is in fact in my ExternalAllocationPools range | 18:42 |
| dtrainor | hmm I don't need environments/ips-from-pool.yaml do I? | 18:44 |
| *** _milan_ has quit IRC | 18:50 | |
| *** milan has joined #tripleo | 18:55 | |
| *** pkovar has quit IRC | 18:56 | |
| *** yamahata has joined #tripleo | 18:56 | |
| *** oshvartz has joined #tripleo | 18:59 | |
| *** jkilpatr has joined #tripleo | 19:02 | |
| *** cwolferh has quit IRC | 19:03 | |
| *** kjw3 has quit IRC | 19:03 | |
| *** chandankumar has quit IRC | 19:04 | |
| *** milan has quit IRC | 19:05 | |
| *** milan has joined #tripleo | 19:05 | |
| *** trown|lunch is now known as trown | 19:07 | |
| *** cwolferh has joined #tripleo | 19:07 | |
| jkilpatr | rlandy, any idea why my browbeat job is trying to use a network environmental yaml even when I have network seperation disabled. | 19:07 |
| jkilpatr | ? | 19:07 |
| jkilpatr | s/seperation/isolation | 19:08 |
| jkilpatr | nevermind I think I see a fix landed a few hours ago | 19:09 |
| jkilpatr | https://github.com/redhat-openstack/ansible-role-tripleo-overcloud/commit/51d96d1a23064002d0c7d4f4a8c357677f3f8c78 | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 19:10 |
| *** penick has quit IRC | 19:11 | |
| rlandy | jkilpatr: hello!! | 19:13 |
| rlandy | jkilpatr: pls see internal | 19:13 |
| *** penick has joined #tripleo | 19:14 | |
| gfidente | have good weekend tripleo | 19:16 |
| *** gfidente has quit IRC | 19:16 | |
| *** milan has quit IRC | 19:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: Test timeout https://review.openstack.org/393415 | 19:21 |
| *** milan has joined #tripleo | 19:21 | |
| *** dsariel has joined #tripleo | 19:24 | |
| *** jbadiapa has quit IRC | 19:25 | |
| *** ctayal has joined #tripleo | 19:29 | |
| *** milan has quit IRC | 19:36 | |
| *** dsneddon_afk is now known as dsneddon | 19:37 | |
| *** leanderthal is now known as leanderthal|afk | 19:39 | |
| *** abregman|afk has quit IRC | 19:39 | |
| *** bnemec-semi-here has quit IRC | 19:39 | |
| *** milan has joined #tripleo | 19:41 | |
| *** milan has quit IRC | 19:50 | |
| *** amoralej is now known as amoralej|off | 19:51 | |
| *** iranzo has quit IRC | 19:52 | |
| *** milan has joined #tripleo | 20:01 | |
| *** tzumainn has joined #tripleo | 20:02 | |
| *** milan has quit IRC | 20:05 | |
| *** hjensas has joined #tripleo | 20:07 | |
| *** panda|bbl is now known as panda | 20:08 | |
| *** penick has quit IRC | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1642429 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1642429 in tripleo "CI: low-memory template doesn't apply and jobs are killed by oom" [Critical,Triaged] | 20:10 |
| *** abehl has joined #tripleo | 20:10 | |
| *** tzumainn has quit IRC | 20:14 | |
| *** ctayal has quit IRC | 20:15 | |
| *** ctayal has joined #tripleo | 20:16 | |
| *** dsariel has quit IRC | 20:16 | |
| *** akrivoka has quit IRC | 20:20 | |
| *** milan has joined #tripleo | 20:21 | |
| *** fultonj has quit IRC | 20:23 | |
| *** cwolferh has quit IRC | 20:27 | |
| *** cwolferh has joined #tripleo | 20:29 | |
| slagle | is it just my patches, or is anyone else seeing this error in CI | 20:30 |
| slagle | Nov 18 19:37:25 localhost os-collect-config: #033[1;31mError: /Stage[main]/Rabbitmq::Config/Rabbitmq_erlang_cookie[/var/lib/rabbitmq/.erlang.cookie]/content: change from NFNKSOGWBJTZCZDNLQEN to weDcaWyG7Obatc48K8T6 failed: Execution of '/usr/bin/puppet resource service rabbitmq-server ensure=stopped' returned 1: /usr/share/gems/gems/json-1.7.7/lib/json/common.rb:155:in `encode': "\xC3" on US-ASCII (Encoding::InvalidByteSequenceError) | 20:30 |
| slagle | hmm, I think it's across the board | 20:31 |
| *** milan has quit IRC | 20:35 | |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo: Adds auto-detection for VIP interfaces https://review.openstack.org/390400 | 20:38 |
| *** bana_k has quit IRC | 20:39 | |
| panda | slagle: that sounds new. | 20:39 |
| *** milan has joined #tripleo | 20:41 | |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo: Adds auto-detection for VIP interfaces https://review.openstack.org/390400 | 20:41 |
| slagle | yea | 20:43 |
| *** penick has joined #tripleo | 20:44 | |
| *** ipsecguy has joined #tripleo | 20:45 | |
| slagle | panda: filed a bug: https://bugs.launchpad.net/tripleo/+bug/1643059 | 20:49 |
| openstack | Launchpad bug 1643059 in tripleo "CI: jobs failing with Error: /Stage[main]/Rabbitmq::Config/Rabbitmq_erlang_cookie[/var/lib/rabbitmq/.erlang.cookie]/content: <snip> `encode': "\xC3" on US-ASCII (Encoding::InvalidByteSequenceError)" [Critical,Triaged] | 20:49 |
| slagle | i think it might be the new puppet-remote package | 20:49 |
| *** ipsecguy_ has quit IRC | 20:49 | |
| panda | \xC3 is the utf-8 core for é | 20:50 |
| panda | code* | 20:50 |
| panda | at least part of it | 20:51 |
| *** milan has quit IRC | 20:51 | |
| *** chlong has quit IRC | 20:53 | |
| *** shardy has quit IRC | 20:59 | |
| *** milan has joined #tripleo | 21:01 | |
| *** arxcruz has quit IRC | 21:02 | |
| dsneddon | Hey everyone, is there a way to test-build the overcloud stack without deploying? I just want to see what the value of certain resources will be when I deploy, to help iterating while developing. | 21:02 |
| *** jayg is now known as jayg|g0n3 | 21:02 | |
| openstackgerrit | John Trowbridge proposed openstack/instack-undercloud: Increase the default number of workers for heat engine https://review.openstack.org/399619 | 21:03 |
| dsneddon | zaneb, ^^^? | 21:03 |
| zaneb | dsneddon: there's a stack-preview command | 21:04 |
| dsneddon | zaneb, Ah, thanks, I'll look that up. | 21:04 |
| zaneb | dsneddon: not sure if it will do what you want or not, but worth a look | 21:04 |
| dsneddon | zaneb, Yes, it looks like exactly what I want. | 21:05 |
| *** milan has quit IRC | 21:05 | |
| *** mhenkel has quit IRC | 21:08 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1643059 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1643059 in tripleo "CI: jobs failing with Error: /Stage[main]/Rabbitmq::Config/Rabbitmq_erlang_cookie[/var/lib/rabbitmq/.erlang.cookie]/content: <snip> `encode': "\xC3" on US-ASCII (Encoding::InvalidByteSequenceError)" [Critical,Triaged] | 21:10 |
| *** fragatina has quit IRC | 21:11 | |
| *** florianf has quit IRC | 21:16 | |
| *** milan has joined #tripleo | 21:17 | |
| *** jkilpatr has quit IRC | 21:17 | |
| *** rlandy has quit IRC | 21:17 | |
| *** mhenkel has joined #tripleo | 21:19 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Use default Sensu redact https://review.openstack.org/398281 | 21:25 |
| panda | slagle: https://github.com/paramite/puppet-remote/issues/2 it wasn't new | 21:33 |
| *** jeckersb is now known as jeckersb_gone | 21:34 | |
| *** andrey-mp has left #tripleo | 21:35 | |
| *** dmarlin_ has left #tripleo | 21:35 | |
| panda | slagle: the á in Mágr encoding is \xC3\xA1 in iso-8859-1. Weird because JSON.parse is reading that correctly in local irb. but anyway | 21:35 |
| *** fragatina has joined #tripleo | 21:41 | |
| slagle | it might be the á, or it could have been the invalid json | 21:47 |
| slagle | he merged my PR that fixed the invalid json, and a new package is available, we can see if it's fixed | 21:48 |
| *** ctayal has quit IRC | 21:53 | |
| *** trown is now known as trown|outtypewww | 21:54 | |
| *** lblanchard has quit IRC | 22:01 | |
| *** penick has quit IRC | 22:01 | |
| *** penick has joined #tripleo | 22:03 | |
| *** lblanchard has joined #tripleo | 22:04 | |
| *** lblanchard has quit IRC | 22:08 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1643059 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1643059 in tripleo "CI: jobs failing with Error: /Stage[main]/Rabbitmq::Config/Rabbitmq_erlang_cookie[/var/lib/rabbitmq/.erlang.cookie]/content: <snip> `encode': "\xC3" on US-ASCII (Encoding::InvalidByteSequenceError)" [Critical,Triaged] | 22:10 |
| *** tiswanso has quit IRC | 22:13 | |
| *** dsavineau has quit IRC | 22:19 | |
| *** jcoufal has quit IRC | 22:23 | |
| *** milan has quit IRC | 22:27 | |
| *** tiswanso has joined #tripleo | 22:36 | |
| *** tiswanso has quit IRC | 22:40 | |
| *** dsneddon is now known as dsneddon_afk | 22:43 | |
| *** penick has quit IRC | 22:52 | |
| panda | slagle: how did you trigger new package creation ? | 22:53 |
| *** cl has joined #tripleo | 22:53 | |
| *** cl has quit IRC | 22:54 | |
| *** [1]cdearborn has quit IRC | 22:59 | |
| *** panda is now known as panda|Zz | 23:08 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1643059 | 23:10 |
| openstack | Launchpad bug 1643059 in tripleo "CI: jobs failing with Error: /Stage[main]/Rabbitmq::Config/Rabbitmq_erlang_cookie[/var/lib/rabbitmq/.erlang.cookie]/content: <snip> `encode': "\xC3" on US-ASCII (Encoding::InvalidByteSequenceError)" [Critical,Triaged] | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** bana_k has joined #tripleo | 23:14 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT REVIEW: Test timeout https://review.openstack.org/393415 | 23:18 |
| *** b00tcat has quit IRC | 23:25 | |
| *** b00tcat has joined #tripleo | 23:25 | |
| *** b00tcat has joined #tripleo | 23:25 | |
| *** abehl has quit IRC | 23:25 | |
| *** bfournie has quit IRC | 23:28 | |
| *** achadha_ has joined #tripleo | 23:30 | |
| *** achadha_ has quit IRC | 23:31 | |
| *** achadha has quit IRC | 23:31 | |
| *** achadha has joined #tripleo | 23:31 | |
| *** achadha_ has joined #tripleo | 23:35 | |
| *** achadha has quit IRC | 23:36 | |
| *** mhenkel has quit IRC | 23:39 | |
| *** achadha_ has quit IRC | 23:39 | |
| *** fragatin_ has joined #tripleo | 23:50 | |
| *** fragatina has quit IRC | 23:53 | |
| *** bana_k has quit IRC | 23:55 | |
| *** bana_k has joined #tripleo | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
