| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role https://review.openstack.org/289027 | 00:04 |
|---|---|---|
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: [NO MERGE] Test Ceph RadosGW as replacement for Swift https://review.openstack.org/357182 | 00:04 |
| *** myoung has quit IRC | 00:09 | |
| *** jlinkes has joined #tripleo | 00:10 | |
| gfidente | we got it working :) | 00:13 |
| gfidente | leaving for today :) | 00:13 |
| *** gfidente has quit IRC | 00:17 | |
| *** maeca1 has quit IRC | 00:17 | |
| *** myoung has joined #tripleo | 00:19 | |
| *** thrash is now known as thrash|g0ne | 00:26 | |
| *** limao has joined #tripleo | 00:36 | |
| *** limao has quit IRC | 00:37 | |
| *** limao has joined #tripleo | 00:37 | |
| *** adarazs has quit IRC | 00:40 | |
| *** dmacpher has quit IRC | 00:42 | |
| *** adarazs has joined #tripleo | 00:42 | |
| *** kjw3 has quit IRC | 00:47 | |
| *** jlinkes_ has joined #tripleo | 00:51 | |
| *** jlinkes has quit IRC | 00:51 | |
| *** Goneri has joined #tripleo | 00:54 | |
| *** bana_k has quit IRC | 01:01 | |
| *** chlong has joined #tripleo | 01:18 | |
| *** kjw3 has joined #tripleo | 01:20 | |
| *** pmannidi has quit IRC | 01:20 | |
| *** pmannidi has joined #tripleo | 01:21 | |
| *** pmannidi has left #tripleo | 01:22 | |
| *** pmannidi has joined #tripleo | 01:22 | |
| *** trozet_ has joined #tripleo | 01:23 | |
| *** trozet has quit IRC | 01:27 | |
| *** dsariel has joined #tripleo | 01:42 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Last round of modern authtoken update https://review.openstack.org/363182 | 01:42 |
| *** ayoung has joined #tripleo | 01:43 | |
| openstackgerrit | Steve Baker proposed openstack/os-collect-config: Treat ec2 collector data as immutable https://review.openstack.org/364021 | 01:55 |
| *** fpan has quit IRC | 01:57 | |
| *** apetrich has quit IRC | 02:05 | |
| *** apetrich has joined #tripleo | 02:07 | |
| *** dsariel has quit IRC | 02:22 | |
| *** Ryjedo_ has joined #tripleo | 02:31 | |
| *** Ryjedo has quit IRC | 02:32 | |
| *** Ryjedo_ is now known as Ryjedo | 02:32 | |
| *** Goneri has quit IRC | 02:35 | |
| *** jlinkes_ has quit IRC | 02:39 | |
| *** jlinkes_ has joined #tripleo | 02:39 | |
| *** yamahata has quit IRC | 03:17 | |
| *** limao has quit IRC | 03:20 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Ensure iscsi_ip_address is in brackets if IPv6 https://review.openstack.org/363961 | 03:22 |
| *** chem|off has quit IRC | 03:34 | |
| *** anbu has quit IRC | 03:36 | |
| *** anbu has joined #tripleo | 03:39 | |
| *** ayoung has quit IRC | 03:47 | |
| *** akshai_ has quit IRC | 03:48 | |
| *** nyechiel has joined #tripleo | 03:52 | |
| *** rlandy has quit IRC | 03:59 | |
| *** nyechiel has quit IRC | 04:00 | |
| *** kjw3 has quit IRC | 04:06 | |
| colonwq | EmilienM, Patch 339106 passed all tests including experimental. Please take a look. | 04:08 |
| *** limao has joined #tripleo | 04:09 | |
| EmilienM | colonwq: ok looking | 04:09 |
| colonwq | crud... It just updated | 04:10 |
| EmilienM | colonwq: it's failing in xenial | 04:11 |
| *** yamahata has joined #tripleo | 04:11 | |
| EmilienM | but yeah it seems to work on centos \o/ | 04:11 |
| EmilienM | fwiw I don't mind having the job working only on centos | 04:12 |
| colonwq | OK. I can make it a lower priority and fix for xenial later. | 04:12 |
| EmilienM | colonwq: +2 on https://review.openstack.org/#/c/347956/ | 04:13 |
| *** limao has quit IRC | 04:13 | |
| EmilienM | colonwq: http://logs.openstack.org/06/339106/39/experimental/gate-puppet-openstack-integration-3-scenario004-tempest-ubuntu-xenial-nv/bcab303/logs/ceph/radosgw.txt.gz | 04:14 |
| *** limao has joined #tripleo | 04:14 | |
| EmilienM | that is the error on xenial | 04:14 |
| EmilienM | maybe you can find out what's wrong | 04:14 |
| *** fragatina has quit IRC | 04:14 | |
| EmilienM | it's weird it doesn't work. | 04:14 |
| EmilienM | I'm out now | 04:14 |
| colonwq | EmilienM Ill look tomorrow | 04:14 |
| colonwq | thanks for the +2 | 04:15 |
| jrist | EmilienM: how do you know all of the things? | 04:16 |
| jrist | where does it fit? | 04:16 |
| jrist | how do you have room for knowing how to fly | 04:16 |
| *** masco has joined #tripleo | 04:17 | |
| EmilienM | jrist: I've been working on openstack deployment for almost 4 years | 04:19 |
| jrist | heh | 04:19 |
| EmilienM | so I know some things | 04:19 |
| jrist | I've been doing it for 2 and have the loosest grasp | 04:19 |
| EmilienM | for flying, I'm a rookie though :) | 04:19 |
| EmilienM | next week should be my first solo, let's see how it goes :P | 04:19 |
| jrist | haha nice | 04:19 |
| jrist | good luck! | 04:19 |
| EmilienM | now sleep | 04:19 |
| jrist | how long for flying? | 04:19 |
| jrist | 'night | 04:19 |
| EmilienM | jrist: how long for flying solo? | 04:20 |
| EmilienM | ~15 hours minimum | 04:20 |
| jrist | nice | 04:20 |
| EmilienM | but you have some exams to pass first | 04:20 |
| jrist | understandable | 04:20 |
| EmilienM | ok good night now :) | 04:20 |
| jrist | 'night! | 04:21 |
| *** links has joined #tripleo | 04:32 | |
| *** fragatina has joined #tripleo | 04:34 | |
| *** rajinir has quit IRC | 04:35 | |
| *** Ng has quit IRC | 04:38 | |
| *** NobodyCam has quit IRC | 04:38 | |
| openstackgerrit | Merged openstack/diskimage-builder: Explain difference between two envvars https://review.openstack.org/345935 | 04:38 |
| *** NobodyCam has joined #tripleo | 04:41 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Disabled IPv6 privacy extensions https://review.openstack.org/356456 | 04:41 |
| *** Ng has joined #tripleo | 04:42 | |
| *** ChanServ sets mode: +v Ng | 04:42 | |
| *** yamahata has quit IRC | 04:42 | |
| *** bana_k has joined #tripleo | 04:46 | |
| *** yamahata has joined #tripleo | 04:54 | |
| *** pcaruana has quit IRC | 04:57 | |
| *** pgadiya has joined #tripleo | 05:02 | |
| *** dhill_ has quit IRC | 05:02 | |
| *** hrybacki has quit IRC | 05:03 | |
| *** kjw3 has joined #tripleo | 05:04 | |
| *** hrybacki has joined #tripleo | 05:04 | |
| *** pgadiya has quit IRC | 05:09 | |
| *** CaptTofu has quit IRC | 05:13 | |
| openstackgerrit | Merged openstack/diskimage-builder: add no_proxy when debootstrap trying to use proxy https://review.openstack.org/345830 | 05:14 |
| openstackgerrit | Merged openstack/diskimage-builder: Update GRUB_MKCONFIG for detecting what's installed https://review.openstack.org/358588 | 05:14 |
| *** CaptTofu has joined #tripleo | 05:15 | |
| *** pgadiya has joined #tripleo | 05:22 | |
| *** fragatin_ has joined #tripleo | 05:22 | |
| *** jlinkes_ has quit IRC | 05:24 | |
| *** jlinkes__ has joined #tripleo | 05:24 | |
| *** fragatina has quit IRC | 05:25 | |
| *** jprovazn has joined #tripleo | 05:31 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML https://review.openstack.org/335265 | 05:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Making element overriding explicit https://review.openstack.org/334785 | 05:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308 | 05:36 |
| *** tzumainn has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add tests for building *-minimal images https://review.openstack.org/181162 | 05:44 |
| *** yamahata has quit IRC | 05:52 | |
| *** florianf has joined #tripleo | 05:58 | |
| *** ebalduf has quit IRC | 06:00 | |
| *** coolsvap has joined #tripleo | 06:05 | |
| *** abregman has quit IRC | 06:07 | |
| *** rcernin has quit IRC | 06:08 | |
| openstackgerrit | Merged openstack/diskimage-builder: Disabled IPv6 privacy extensions https://review.openstack.org/356456 | 06:11 |
| *** jlinkes__ has quit IRC | 06:16 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Modify sriov_numvfs config from puppet-neutron to tripleo https://review.openstack.org/361430 | 06:21 |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates: Added environment file for SR-IOV deployment https://review.openstack.org/363617 | 06:22 |
| *** jlinkes has joined #tripleo | 06:22 | |
| *** saneax-_-|AFK is now known as saneax | 06:23 | |
| *** bvandenh has joined #tripleo | 06:25 | |
| *** aqkhan__ has joined #tripleo | 06:25 | |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates: Add base neutron service configuration https://review.openstack.org/361367 | 06:28 |
| *** aqkhan_ has quit IRC | 06:28 | |
| openstackgerrit | Merged openstack/diskimage-builder: Generate and use upper-constraints for ironic-agent https://review.openstack.org/360547 | 06:29 |
| openstackgerrit | Merged openstack/diskimage-builder: Update portage only if needed https://review.openstack.org/358592 | 06:29 |
| openstackgerrit | Merged openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs https://review.openstack.org/355607 | 06:29 |
| *** jbadiapa has joined #tripleo | 06:29 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add blurb about communication to docs landing page https://review.openstack.org/355533 | 06:29 |
| *** gfidente has joined #tripleo | 06:30 | |
| *** pcaruana has joined #tripleo | 06:30 | |
| openstackgerrit | Karthik S proposed openstack/tripleo-heat-templates: Add base neutron service configuration https://review.openstack.org/361367 | 06:32 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: bind on hostnames by default https://review.openstack.org/364089 | 06:32 |
| *** oshvartz has joined #tripleo | 06:33 | |
| *** aqkhan_ has joined #tripleo | 06:38 | |
| *** aqkhan__ has quit IRC | 06:38 | |
| *** aqkhan__ has joined #tripleo | 06:39 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: bind on hostnames by default https://review.openstack.org/364089 | 06:39 |
| *** rcernin has joined #tripleo | 06:42 | |
| *** mhenkel has joined #tripleo | 06:42 | |
| *** aqkhan_ has quit IRC | 06:43 | |
| *** rcernin has quit IRC | 06:45 | |
| *** rcernin has joined #tripleo | 06:45 | |
| *** bana_k has quit IRC | 06:46 | |
| *** leanderthal|afk is now known as leanderthal | 06:49 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Refactor DeploymentPlan Component https://review.openstack.org/358814 | 06:50 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Mistral Actions error handling https://review.openstack.org/359764 | 06:50 |
| *** Guest44588 has quit IRC | 06:56 | |
| *** Guest44588 has joined #tripleo | 06:56 | |
| *** Guest44588 is now known as assassin | 06:56 | |
| *** aqkhan_ has joined #tripleo | 06:59 | |
| *** aqkhan__ has quit IRC | 06:59 | |
| *** aqkhan__ has joined #tripleo | 07:00 | |
| *** mlupton has quit IRC | 07:00 | |
| *** tesseract- has joined #tripleo | 07:03 | |
| *** rwsu has joined #tripleo | 07:03 | |
| *** aqkhan_ has quit IRC | 07:04 | |
| *** aufi has joined #tripleo | 07:05 | |
| *** yolanda has joined #tripleo | 07:06 | |
| *** abregman has joined #tripleo | 07:06 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: bind on hostnames by default https://review.openstack.org/364089 | 07:07 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: WIP: bind on hostnames by default https://review.openstack.org/364089 | 07:09 |
| *** aqkhan_ has joined #tripleo | 07:10 | |
| *** ebarrera has joined #tripleo | 07:13 | |
| *** aqkhan__ has quit IRC | 07:14 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:14 | |
| *** jpena|off is now known as jpena | 07:18 | |
| *** matbu|afk is now known as matbu | 07:20 | |
| *** jpich has joined #tripleo | 07:25 | |
| *** ifarkas_afk is now known as ifarkas | 07:28 | |
| *** nyechiel has joined #tripleo | 07:30 | |
| *** bkopilov has quit IRC | 07:30 | |
| *** apetrich has quit IRC | 07:31 | |
| *** abregman_ has joined #tripleo | 07:32 | |
| *** abregman has quit IRC | 07:32 | |
| *** apetrich has joined #tripleo | 07:33 | |
| *** qasims has joined #tripleo | 07:36 | |
| *** hjensas has joined #tripleo | 07:36 | |
| *** jlinkes has quit IRC | 07:36 | |
| *** jlinkes has joined #tripleo | 07:36 | |
| *** bkopilov has joined #tripleo | 07:40 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: [WIP] Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 07:41 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW profile https://review.openstack.org/334081 | 07:41 |
| *** dsariel has joined #tripleo | 07:42 | |
| *** dbecker has joined #tripleo | 07:50 | |
| *** liverpooler has joined #tripleo | 07:54 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-heat-templates: WIP Add a function to upgrade from full HA to NG HA https://review.openstack.org/358626 | 07:56 |
| *** ebarrera_ has joined #tripleo | 07:57 | |
| *** shardy has joined #tripleo | 07:57 | |
| *** dtantsur|afk is now known as dtantsur | 07:58 | |
| openstackgerrit | Merged openstack/python-tripleoclient: NTP arg should be optional when specified in an environment https://review.openstack.org/332694 | 07:58 |
| *** ohamada has joined #tripleo | 07:59 | |
| *** akuznetsov has joined #tripleo | 08:00 | |
| *** mcornea has joined #tripleo | 08:01 | |
| *** anbu_ has joined #tripleo | 08:03 | |
| *** jaosorior has joined #tripleo | 08:04 | |
| *** athomas has joined #tripleo | 08:04 | |
| jaosorior | gfidente: hey dude, really like your idea for the FQDNs | 08:04 |
| jaosorior | commented on a detail about it though, cause I've been trying to do that | 08:05 |
| *** anbu has quit IRC | 08:07 | |
| dtantsur | morning folks! is this a known breakage: http://logs.openstack.org/76/363576/1/check/gate-tripleo-ci-centos-7-nonha-multinode/0c67d4f/console.html#_2016-09-01_07_53_28_236130 ? | 08:07 |
| dtantsur | is it worth rechecking now? | 08:07 |
| marios | dtantsur: fwiw i saw a similar issue http://logs.openstack.org/66/362866/2/check/gate-tripleo-ci-centos-7-nonha-multinode/c582772/console.html#_2016-08-31_16_54_48_160892 from https://review.openstack.org/#/c/362866 | 08:09 |
| marios | dtantsur: i didn't poke much further but was planning on recheck... but might be the same. | 08:09 |
| dtantsur | hmmm | 08:09 |
| dtantsur | I'm afraid it's something infra-related.. I've seen it several times already | 08:09 |
| *** milan has joined #tripleo | 08:10 | |
| dtantsur | will try one more recheck while anyway most of the folks are still sleeping | 08:10 |
| marios | dtantsur: i guess if no-one else knows for sure yet youd could probably get away with a recheck... ah right... well that is good to know at least | 08:10 |
| marios | dtantsur: ack :) | 08:10 |
| *** ohamada has quit IRC | 08:10 | |
| *** ohamada has joined #tripleo | 08:11 | |
| Jokke_ | hey all | 08:13 |
| Jokke_ | I noticed that marios' changes on https://review.openstack.org/#/c/354019/ started to pass tests (does it actually work as well or just not break anything?). Should I refactor https://review.openstack.org/#/c/358525/ based on that approach or what are we doing with that? | 08:14 |
| Jokke_ | Just wanted to make sure that I'm on same page before I fully hit on my PTO | 08:15 |
| marios | o/ hey Jokke_ | 08:15 |
| marios | sorry was a bit busy yesterday especially so only fleeting passes to update the review | 08:15 |
| marios | Jokke_: so, not sure yet, you tell me | 08:15 |
| marios | Jokke_: i mean, i am waiting for feedback. in latest version we have no more need for 'base' env file for example | 08:16 |
| marios | Jokke_: i'll try get shardy to look at that today and get his thoughts | 08:16 |
| marios | Jokke_: the tests at https://review.openstack.org/#/c/354019/ are not exercising manila at all | 08:16 |
| marios | Jokke_: for that i have made https://review.openstack.org/#/c/362866/ which can test manila-generic at least (no netapp backend in ci) | 08:17 |
| marios | Jokke_: that is failing, but i don't think related. it was what dtantsur and i were just discussing, seems infra related | 08:17 |
| Jokke_ | marios: ok, cool ... someone give me a ping when you know what you want to do with those, please. I've been busy past few days as well and didn't want to rush on it before seeing how that work you stater plans out. | 08:17 |
| Jokke_ | my pto started today, but I wanted to follow up on that one patch I have pending | 08:18 |
| Jokke_ | I'll do the same at the evening and if so decided align my change to that one either tonight or tomorrow | 08:19 |
| marios | Jokke_: oh ic. well ok man, worst case i can have a go at fixing the ceph review, assuming the netapp stuff lands | 08:19 |
| Jokke_ | <3 | 08:20 |
| marios | Jokke_: and i will gladly take your pto if you'd much rather just hang with us | 08:20 |
| Jokke_ | LOL | 08:20 |
| Jokke_ | marios: I'd rather see the work finished and enjoy my pto without worrying it ;) | 08:20 |
| marios | i hear ya Jokke_ | 08:23 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 08:28 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 08:29 |
| mandre | hey folks, can I get a +A on https://review.openstack.org/#/c/363545/? this fixes the gate when it's using IPv6 | 08:33 |
| *** r-mibu has quit IRC | 08:33 | |
| d0ugal | Are the obv jobs all failing? | 08:38 |
| jaosorior | d0ugal: seems so | 08:38 |
| *** r-mibu has joined #tripleo | 08:38 | |
| *** lucas-dinner is now known as lucasagomes | 08:38 | |
| d0ugal | jaosorior: k, thanks | 08:42 |
| * d0ugal looks to see if he can make any sense of it | 08:42 | |
| *** bkopilov has quit IRC | 08:45 | |
| Jokke_ | bbl o/~ | 08:45 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 08:50 |
| *** dobson` has joined #tripleo | 08:53 | |
| *** derekh has joined #tripleo | 08:54 | |
| *** HenryG_ has joined #tripleo | 08:54 | |
| *** shardy has quit IRC | 08:58 | |
| *** mhenkel has quit IRC | 08:58 | |
| *** adarazs has quit IRC | 08:58 | |
| *** wfoster has quit IRC | 08:58 | |
| *** rbowen has quit IRC | 08:58 | |
| *** benoit_ has quit IRC | 08:58 | |
| *** mat128 has quit IRC | 08:58 | |
| *** dobson has quit IRC | 08:58 | |
| *** HenryG has quit IRC | 08:58 | |
| *** stevebaker has quit IRC | 08:58 | |
| *** shadower has quit IRC | 08:58 | |
| *** jidar has quit IRC | 08:58 | |
| *** adarazs has joined #tripleo | 08:59 | |
| *** mat128 has joined #tripleo | 08:59 | |
| *** benoit_ has joined #tripleo | 09:00 | |
| *** shardy has joined #tripleo | 09:03 | |
| *** mhenkel has joined #tripleo | 09:03 | |
| *** wfoster has joined #tripleo | 09:03 | |
| *** rbowen has joined #tripleo | 09:03 | |
| *** stevebaker has joined #tripleo | 09:03 | |
| *** jidar has joined #tripleo | 09:03 | |
| *** akuznetsov has quit IRC | 09:04 | |
| jaosorior | shardy: hey dude, do we do any package build in any of the jobs at the moment? | 09:06 |
| *** abregman_ has quit IRC | 09:08 | |
| shardy | jaosorior: Yes we build a delorean package for the patch under review and any Depends-On | 09:10 |
| jaosorior | shardy: sorry (I'm a bit flu-ish) I meant image build | 09:13 |
| jaosorior | not package build | 09:13 |
| shardy | jaosorior: Yes we were caching images but then switched back to building images every commit AFAIK | 09:13 |
| shardy | that's one reason our CI jobs keep timing out | 09:13 |
| jaosorior | shardy: is that done on the ovb jobs only? | 09:13 |
| shardy | jaosorior: Not sure, would have to check the scripts, I guess looking at the runtime multinode must be using cached images | 09:14 |
| jaosorior | yeah,t hat would make sense | 09:14 |
| derekh | jaosorior: shardy we don't need images in the multinode jobs, we're installing stuff on a already running VM | 09:15 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Fix pingtest in tripleo/validate to use template https://review.openstack.org/364178 | 09:15 |
| *** akrivoka has joined #tripleo | 09:16 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:16 |
| *** HenryG_ is now known as HenryG | 09:17 | |
| jaosorior | I see | 09:18 |
| jaosorior | thanks | 09:18 |
| *** tremble has joined #tripleo | 09:19 | |
| *** tremble has joined #tripleo | 09:19 | |
| shardy | derekh: ah, I wasn't sure what the base image for those nodes was | 09:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/361875 | 09:22 |
| *** abregman_ has joined #tripleo | 09:23 | |
| *** anbu__ has joined #tripleo | 09:25 | |
| *** mrunge has quit IRC | 09:25 | |
| *** numans has joined #tripleo | 09:27 | |
| *** anbu_ has quit IRC | 09:27 | |
| *** HenryG has quit IRC | 09:29 | |
| *** HenryG has joined #tripleo | 09:29 | |
| dtantsur | folks, I'm hitting a pretty severe bug: https://bugs.launchpad.net/tripleo/+bug/1619205 | 09:31 |
| openstack | Launchpad bug 1619205 in tripleo "Overcloud API services go down after some time due to keystonemiddleware failure" [High,New] | 09:31 |
| dtantsur | can someone confirm it? | 09:31 |
| dtantsur | tl;dr I have to restart httpd once a day or overcloud API services (at least some of them) or inaccessible | 09:32 |
| shardy | Hey all, I've been getting a few questions about the scary nested intrinsic functions in the custom-roles patches, so I blogged about how it works: | 09:32 |
| shardy | http://hardysteven.blogspot.co.uk/2016/09/complex-data-transformations-with.html | 09:33 |
| shardy | hopefully that will help demystify it somewhat ;) | 09:33 |
| shardy | dtantsur: that does indeed sound pretty bad, are there any related keystone bugs about it? | 09:34 |
| shardy | sounds like keystone either crashed or got stuck? | 09:34 |
| shardy | was keystone running before you restarted httpd? | 09:34 |
| dtantsur | shardy, it looked like it's running fine, no errors, etc | 09:35 |
| *** mrunge has joined #tripleo | 09:36 | |
| shardy | Yeah but was the API responsive? Sounds like it probably wasn't | 09:36 |
| *** flepied has quit IRC | 09:36 | |
| *** flepied1 has joined #tripleo | 09:36 | |
| *** dmacpher has joined #tripleo | 09:37 | |
| dtantsur | shardy, no, it was not. see the update on the bug, I've found something suspicious in httpd logs. | 09:37 |
| shardy | dtantsur: ack, thanks | 09:37 |
| shardy | perhaps we can loop ayoung in later when he arrives | 09:38 |
| dtantsur | though I doubt it's the root cause.. it happened some time ago | 09:38 |
| *** chem|off has joined #tripleo | 09:43 | |
| *** chem|off is now known as chem | 09:43 | |
| *** limao has quit IRC | 09:45 | |
| *** limao has joined #tripleo | 09:45 | |
| *** chem has quit IRC | 09:49 | |
| *** chem has joined #tripleo | 09:49 | |
| *** panda|afk is now known as panda | 09:52 | |
| *** panda is now known as Guest58717 | 09:52 | |
| *** Guest58717 is now known as panda | 09:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses https://review.openstack.org/363545 | 09:55 |
| *** flepied1 has quit IRC | 10:00 | |
| *** flepied has joined #tripleo | 10:00 | |
| *** limao has quit IRC | 10:02 | |
| *** akrivoka has quit IRC | 10:09 | |
| *** dsariel has quit IRC | 10:14 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: service catalog entries should not have a suffix https://review.openstack.org/364213 | 10:16 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: service catalog entries should not have a suffix https://review.openstack.org/364213 | 10:19 |
| *** thrash|g0ne is now known as thrash | 10:19 | |
| *** akrivoka has joined #tripleo | 10:20 | |
| *** hjensas has quit IRC | 10:28 | |
| marios | shardy: nice looking writeup noted for deskercize later :) esp the yaql stuff | 10:28 |
| marios | shardy: i also mentioned another of your blog posts yesterday on a call where we were talking about debugging when it explodes http://hardysteven.blogspot.com.cy/2015/04/debugging-tripleo-heat-templates.html | 10:29 |
| *** yolanda has quit IRC | 10:32 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Install tripleo-admin ssh keys on CI nodes https://review.openstack.org/363630 | 10:33 |
| marios | shardy: can you please add to your review queue https://review.openstack.org/#/c/354019/ i hope addresses your comments wrt composability of the backends. we have a test review too at https://review.openstack.org/#/c/362866 to exercise the tidy up of manila-generic | 10:36 |
| shardy | marios: nice, thanks will do | 10:36 |
| marios | appreciate | 10:36 |
| *** paramite has joined #tripleo | 10:36 | |
| flepied | how could I get info about the current investigation of the tripleo CI periodic job? | 10:39 |
| shardy | sshnaidm: ^^ | 10:40 |
| shardy | flepied: the main issue is timeouts, because we've got some performance issues with rh1 which are under investigation | 10:40 |
| shardy | until that is resolved it's hard to debug any functional problems | 10:40 |
| shardy | that said, sshnaidm may have more info | 10:40 |
| *** jlinkes has quit IRC | 10:41 | |
| flepied | shardy: where do you track these issues? | 10:41 |
| shardy | https://bugs.launchpad.net/tripleo/+bugs?field.tag=ci | 10:41 |
| shardy | flepied: ^^ | 10:41 |
| *** hjensas has joined #tripleo | 10:41 | |
| *** hjensas has quit IRC | 10:41 | |
| *** hjensas has joined #tripleo | 10:41 | |
| flepied | shardy: cool thx | 10:42 |
| shardy | I'm not actually sure if there is a bug for the performance investigation, we should probably create one | 10:42 |
| sshnaidm | flepied, what issues are we talking? | 10:43 |
| flepied | shardy: I'm going to put this link in the RDO dashboard. it's much useful than the http://tripleo.org/cistatus-periodic.html that is not updated anymore | 10:43 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Install ipxe-bootimgs and tftp-server on controller https://review.openstack.org/363576 | 10:43 |
| flepied | sshnaidm: having the periodic job green | 10:43 |
| shardy | marios: Looks much better, but I think I spotted an issue, commented | 10:44 |
| sshnaidm | flepied, it's continuous process :) | 10:44 |
| shardy | marios: have those been tested? | 10:44 |
| *** paramite has quit IRC | 10:44 | |
| flepied | sshnaidm: yes but that's the start of our pipeline and nothing is coming through :-( | 10:44 |
| *** stendulker has joined #tripleo | 10:45 | |
| marios | shardy: thanks looking, not by me wrt testing, i've just been responding to comments. i think tbarron bswartz et al are testing manually, there were some comments on earlier version of puppet-tripleo side | 10:45 |
| shardy | marios: ack - I could be wrong but AFAICS they actually won't work | 10:45 |
| sshnaidm | flepied, sorry, I don't follow.. | 10:45 |
| shardy | we just need an empty step_config in all the service templates AFAICS | 10:45 |
| sshnaidm | flepied, you mean we haven't promoted for a week? | 10:46 |
| flepied | sshnaidm: yes | 10:46 |
| sshnaidm | flepied, it's more or less different issues each time | 10:46 |
| shardy | flepied: Yeah, we're trying hard to fix it - the first step is fixing rh1 performance, and solving that is proving tricky | 10:46 |
| shardy | we may have to consider reducing coverage to get things inside the timeout if we can't solve that soon | 10:47 |
| *** jlinkes has joined #tripleo | 10:47 | |
| *** dsariel has joined #tripleo | 10:47 | |
| shardy | I'd really rather not do that though, so close to a release | 10:47 |
| sshnaidm | flepied, for example today it's a problem with packaging in consistent repo, which was lack of some packages and jobs failed | 10:47 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: [NO MERGE] Test Ceph RadosGW as replacement for Swift https://review.openstack.org/364227 | 10:48 |
| sshnaidm | flepied, after jpena solved this I retriggered the jobs, we'll see it in a few hours | 10:48 |
| sshnaidm | flepied, and as shardy said, we have also performance issues on the cloud, it creates problems too | 10:48 |
| marios | shardy: thanks, missed it indeed for the backends... (generic/netapp) but i think i should skip it on manila-base, will make it so incoming thanks | 10:48 |
| *** dtantsur is now known as dtantsur|lunch | 10:49 | |
| sshnaidm | flepied, before this was bug in one of python modules, which blocked all CI.. there are very different problems in various places, that we can't even control sometimes | 10:49 |
| assassin | marios: am here too :) | 10:49 |
| shardy | marios: ack, yup you're right, not needed in manila-base | 10:50 |
| shardy | only the two new ones | 10:50 |
| openstackgerrit | Merged openstack/puppet-tripleo: Default haproxy listeners activation on hiera service _enabled https://review.openstack.org/362798 | 10:51 |
| marios | assassin: hi o/. so i'm going to fixup a nit in latest rev thanks to shardy. but to be clear, my involvement with this is meant to be 'asssist' so for example, i don't have spare cycles (and most importantly, environments) to test this on, so waiting for feedback on the review | 10:51 |
| flepied | sshnaidm: are we making progress to augment the cadence of the periodic job? | 10:51 |
| assassin | marios: Yeah, understand that. what I am being asked is what the danger is that this may miss the release | 10:51 |
| assassin | shardy: ^^ | 10:51 |
| *** limao has joined #tripleo | 10:52 | |
| sshnaidm | flepied, I think there is work in this direction, weshay should know more, but I'm very skeptical about changing it in current zull/nodepool environment | 10:53 |
| flepied | sshnaidm: pabelanger was working on this iirc | 10:54 |
| sshnaidm | flepied, asaiu either we wait for ZUULv3 and its flexible pipelines, or we change the way how we promote | 10:54 |
| flepied | sshnaidm: what do you suggest as a change? | 10:54 |
| shardy | assassin: well, we can merge it before the final release provided folks can test it and confirm it actually works | 10:55 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 10:55 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add profile to write overcloud VIPs into /etc/hosts https://review.openstack.org/357762 | 10:55 |
| shardy | assassin: I've given a FFE, but the current patch clearly hasn't been tested, so as marios says we need help validating it works, or there's not much point merging it | 10:55 |
| sshnaidm | flepied, pabelanger mentioned that we can use rdoproject zuul pipelines for periodic jobs, but I'm not familiar with this, but it seems to be one of possible solutions | 10:56 |
| assassin | shardy: what I guessed was that based on risk, you might not want to wait until 23:59 on Sep 11th to yank it. | 10:56 |
| shardy | assassin: Yeah, but the patch looks pretty good, I see no reason we can't land it soon if folks can help with testing | 10:56 |
| sshnaidm | flepied, either to promote repo in different gate job, but it will increase load on our system, which is already suffer from performance problems | 10:56 |
| assassin | shardy: and testing kind of hinges on having the right hardware, right ? | 10:57 |
| shardy | my current -1 is a very easy fix, but it proves pretty clearly no testing at all has been done | 10:57 |
| assassin | shardy: *nod* I'll take that back as feedback | 10:57 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Remove Node registration username validation https://review.openstack.org/364238 | 10:57 |
| flepied | sshnaidm: in which bug is described the performance issue? | 10:58 |
| tbarron | marios: shardy assassin we've been trying to overcloud deploy with these patchers and have been rep;orting (at least from time to time) deploy failures | 11:00 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic https://review.openstack.org/354019 | 11:01 |
| sshnaidm | flepied, I don't think we have it tracked anywhere, bug I can create it now | 11:01 |
| tbarron | marios: shardy assassin the folkds trying to deploy are pretty much OOO n00bs so it hasn't been entirely clear to us whether to blame code or pilot error | 11:01 |
| sshnaidm | shardy, do we have performance issues tracked anywhere in lp? | 11:01 |
| flepied | sshnaidm: yes please create it if you can | 11:02 |
| marios | tbarron: ack, i also missed email/pings had to take some time off sorry about that. i think we are close | 11:02 |
| tbarron | marios: shardy assassin it would help to have a literal (not schematic -e A -e B etc) example of the exact overcloud command that we should use to test | 11:02 |
| assassin | tbarron: I am afraid I am also a total noob on this, so I would be of little use trying to help test this. :-/ | 11:02 |
| marios | tbarron: sure, sec (essentially just include the relevate env file) | 11:03 |
| shardy | sshnaidm: I don't think so, but bnemec is investigating, lets create one and ask if he can provide a summary of status | 11:03 |
| tbarron | marios: yeah, but include it on what, we are n00bs :) | 11:03 |
| tbarron | marios: lemme look up the last overcloud deploy cmd attempt | 11:03 |
| * tbarron gets his first coffee, so it will be a minute | 11:04 | |
| marios | tbarron: openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/ | 11:04 |
| sshnaidm | shardy, flepied https://bugs.launchpad.net/tripleo/+bug/1619243 | 11:04 |
| openstack | Launchpad bug 1619243 in tripleo "CI: periodic jobs fail because of exceeded timeout" [Critical,Triaged] - Assigned to Ben Nemec (bnemec) | 11:04 |
| marios | share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server '0.fedora.pool.ntp.org' -e /usr/share/openstack-tripleo-heat-templates/environments/manila-generic-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/manila-netapp-config.yaml | 11:04 |
| flepied | thx sshnaidm | 11:04 |
| marios | hah... lemme get an fpaste perhaps, or i'll note on the review instead tbarron | 11:04 |
| *** rhallisey has joined #tripleo | 11:05 | |
| flepied | sshnaidm: how do you follow the progress of the job? is there an url I can take a look? | 11:06 |
| sshnaidm | flepied, progress of current now running job? | 11:06 |
| flepied | sshnaidm: yes | 11:06 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Composable per-service VIP hieradata https://review.openstack.org/361778 | 11:06 |
| sshnaidm | flepied, http://status.openstack.org/zuul/ | 11:06 |
| sshnaidm | flepied, there is regular "periodic" pipeline | 11:07 |
| tbarron | marios++ | 11:07 |
| sshnaidm | flepied, if I trigger them manually (like I did now) look for a patch "359215" | 11:07 |
| sshnaidm | flepied, just Ctrl+F on the page and search for 359215 | 11:08 |
| tbarron | marios that's exactly what we neede, we'd been leaving out some of the earlier -e stuff | 11:08 |
| sshnaidm | flepied, click on the patch label there and you'll see running job, click on one of them for opening it in terminal | 11:08 |
| flepied | sshnaidm: ok thx | 11:08 |
| marios | tbarron: so you could probably leave out the network isolation and single-nic-vlans stuff but yeah at minimum i'd deploy with the puppet-pacemaker and 3/1compute for testing | 11:08 |
| *** limao has quit IRC | 11:09 | |
| tbarron | assassin: netapp has shipped redhat QE a box to use for testing this but it isn't installed yet | 11:09 |
| jaosorior | shardy: would it be possible for you to add the specific vip fqdn or at least which network should a service be at as part of this? https://review.openstack.org/#/c/361778/4 | 11:09 |
| *** limao has joined #tripleo | 11:10 | |
| assassin | tbarron: ah, excellent. | 11:10 |
| tbarron | assassin: i've been trying to light a fire on this, but I'm not QE and things are moving (from my perspective) slowly | 11:10 |
| tbarron | assassin: i'll forward you an email, in case you have any influence | 11:10 |
| tbarron | marios: so i am currently deploying without the actual backend and the manila service should start and fail | 11:11 |
| tbarron | marios: but in the mean time i'm hoping to see the overcloud deploy anywasys and be able to get into controllers, see proper enabled backends, etc., and go to /var/log/manila and see that the services attempted to start and fail b/c the backend isn't actually there yet | 11:12 |
| tbarron | marios: does that seem sane? | 11:12 |
| marios | tbarron: yeah i guess it would since there is no config set if you aren't including the backend | 11:12 |
| shardy | jaosorior: probably, do we have any per-vip hostnames in the hieradata already? | 11:13 |
| jaosorior | shardy: not in the hieradata, no | 11:13 |
| marios | tbarron: yeah if it fails on the bringup of manila it will stop there so whatever has already been done will stay so/should be able to get to the overcloud nodes for debug etc. | 11:13 |
| shardy | I just need to know where to look them up | 11:13 |
| tbarron | marios: assassin shardy also, once I reach that point i'll get bswartz to reproduce what I did, in-house at netapp, pointing at a real netapp | 11:13 |
| shardy | it should be pretty simple to expand my vip-config changes is we can pass in a mapping containing the names needed | 11:14 |
| jaosorior | shardy: https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L410 | 11:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 11:14 |
| marios | tbarron: oh so hold on... actually | 11:14 |
| marios | tbarron: if you don't include either of the backends, nothing happends at all for manila | 11:14 |
| marios | tbarron: because in the resource registry manila-* OS::Heat::None | 11:15 |
| marios | tbarron: and we only make it point to actual service files in either of the backends. does it make sense? https://review.openstack.org/gitweb?p=openstack/tripleo-heat-templates.git;a=commitdiff;h=43651f0f720ed0d03a0f678fd8b627cb3f887f62 | 11:15 |
| tbarron | marios: i'll include the -e for netapp, what i'm trying to say is that at first there won't actually be a netapp there | 11:16 |
| tbarron | marios: so it should write /etc/manila/manila.conf **as if** there's a netapp box out there to connect to | 11:16 |
| shardy | jaosorior: can you not use the "host" output from the EndpointMap? | 11:16 |
| marios | tbarron: oh yeah, i expect the config to at least be written if it just fails on service startup | 11:17 |
| jaosorior | shardy: no. that doesn't link it at all to what network a service in and then I still have to parse and remove the URI bits | 11:17 |
| jaosorior | such as the port, protocol and version stuff | 11:17 |
| tbarron | marios: and systemd shoud attempt to start api and scheduler services, and pacemaker should attempt to start share service, leaving traces of their attempts under /var/log/manila | 11:17 |
| jaosorior | ah wait | 11:18 |
| jaosorior | the host part | 11:18 |
| jaosorior | uhm... | 11:18 |
| jaosorior | maybe I could | 11:18 |
| tbarron | marios: so far we haven't been able to deploy to that point, so that would be progress. | 11:18 |
| jaosorior | shardy: yeah, could be. | 11:18 |
| jaosorior | let me look into that | 11:18 |
| shardy | jaosorior: ack, Ok lets try that first then | 11:18 |
| tbarron | marios: and if we get there, i will get bswartz to confirm that what we have in /etc/manila/manila.conf on the overcloud controllers is sane, | 11:19 |
| marios | tbarron: you need to make sure you have the puppet-tripleo side to this too though | 11:20 |
| *** ramishra_ has joined #tripleo | 11:20 | |
| *** ramishra has quit IRC | 11:21 | |
| *** gfidente has quit IRC | 11:21 | |
| tbarron | marios: i'm buiding from latest deloriean but using local git for your changes and copyying the tht stuff intio /user/share/... before running overcloud deploy cmd | 11:22 |
| marios | tbarron: just checking (is the depends on) ... right so you will need the puppet-tripleo change for the overcloud image too like http://docs.openstack.org/developer/tripleo-docs/developer/in_progress_review.html | 11:22 |
| marios | tbarron: otherwise it is using what is currently there, which is probably fine for generic but it has no idea about netapp | 11:22 |
| *** gfidente has joined #tripleo | 11:23 | |
| *** gfidente has quit IRC | 11:23 | |
| *** gfidente has joined #tripleo | 11:23 | |
| marios | tbarron: 'there' being puppet-tripleo side | 11:23 |
| tbarron | marios: check | 11:23 |
| *** mburned_out is now known as mburned | 11:23 | |
| tbarron | marios: i'll log what I do and point to the log in review comment. Then you can look at the log and confirm that I'm picking up the changes from your reviews correctly. | 11:24 |
| marios | tbarron: sure, i have tried to make a daily pass on these patches this week will do again tomorrow or possibly later today | 11:26 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 11:26 |
| jaosorior | gfidente: could I get a review for this? https://review.openstack.org/#/c/357762/ | 11:29 |
| tbarron | marios: cool, i'll try your latest today | 11:29 |
| *** limao has quit IRC | 11:30 | |
| *** limao has joined #tripleo | 11:31 | |
| *** lucasagomes is now known as lucas-hungry | 11:33 | |
| *** jlinkes has quit IRC | 11:33 | |
| *** jlinkes has joined #tripleo | 11:34 | |
| *** dprince has joined #tripleo | 11:36 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: IGNORE - testing manila-generic deploy after tidyup https://review.openstack.org/362866 | 11:37 |
| assassin | marios++ shardy++ tbarron++ | 11:38 |
| assassin | need a karma bot in here | 11:38 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Mv pacemaker and firewall out of controller https://review.openstack.org/361398 | 11:39 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move snmp settings into composable services https://review.openstack.org/360062 | 11:39 |
| *** hjensas has quit IRC | 11:43 | |
| *** anbu__ has quit IRC | 11:46 | |
| dprince | shardy: I'm keen to move the rest of the extra config interfaces over to composable services as well http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/controller.yaml#n390 | 11:48 |
| dprince | shardy: would you support doing this for newton? | 11:48 |
| dprince | shardy: looks like there are 8 of them or so left... | 11:48 |
| dprince | should be fairly quick to do and review though | 11:48 |
| sshnaidm | did anyone see such error in pingtest? InternalServerError: resources.router: Not enough l3 agents available to ensure HA. Minimum required 2, available 1 | 11:49 |
| d0ugal | thrash: I think https://review.openstack.org/#/c/335460/ might actually be broken | 11:51 |
| d0ugal | thrash: It has failed 100% of the time for a while | 11:51 |
| thrash | d0ugal: ack | 11:51 |
| d0ugal | thrash: I've not had a chance to manually test it in a while and it has had a few rebases... I'll try today. | 11:51 |
| thrash | Will look at it again once it finishes. | 11:52 |
| d0ugal | Thanks | 11:52 |
| *** limao has quit IRC | 11:52 | |
| d0ugal | I wonder if osc_lib could be missing in packages. | 11:52 |
| *** limao has joined #tripleo | 11:52 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: [WIP] Add an optional timeout when waiting for websocket messages https://review.openstack.org/364252 | 11:54 |
| *** jlinkes has quit IRC | 11:54 | |
| *** maeca1 has joined #tripleo | 11:55 | |
| *** jlinkes has joined #tripleo | 11:55 | |
| thrash | d0ugal: I don't think so. | 11:55 |
| thrash | d0ugal: I thought I had checked for that. | 11:55 |
| *** jpena is now known as jpena|lunch | 11:56 | |
| thrash | d0ugal: it's there | 11:56 |
| thrash | I think it's a dependency for openstackclient anyway | 11:56 |
| d0ugal | thrash: ah, makes sense. | 11:57 |
| shardy | dprince: If we can do it in a backwards compatible way then yes | 11:58 |
| shardy | dprince: would we have e.g OS::TripleO::Services::ExtraConfig, which is included by default on all roles? | 11:59 |
| shardy | I'm just wondering how we map the per-role *ExtraConfig parameter names | 11:59 |
| shardy | we can't remove them without deprecation | 11:59 |
| dprince | shardy: if there are environments for the features I think I can just swap in the service w/ OS::Heat::None | 12:00 |
| shardy | dprince: Yeah, but we can't break any out of tree users of e.g ControllerExtraConfigPre | 12:00 |
| dprince | shardy: similar to what you are proposing, yep | 12:00 |
| shardy | or the controllerExtraConfig/ControllerExtraConfig parameters | 12:00 |
| dprince | shardy: I'll make sure we can do it backwards compat | 12:00 |
| shardy | dprince: ack, sounds good then :) | 12:00 |
| *** fzdarsky has joined #tripleo | 12:01 | |
| EmilienM | hello | 12:01 |
| shardy | 'morning EmilienM | 12:01 |
| *** zoli_gone-proxy is now known as zoliXXL | 12:02 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 12:04 |
| *** dtantsur|lunch is now known as dtantsur | 12:04 | |
| sshnaidm | derekh, hi, do you know why there is no eth1 on multinode jobs? do we create it anywhere..? | 12:04 |
| EmilienM | sshnaidm: because they are virtual instances on multiple cloud providers, ie RAX, etc | 12:05 |
| EmilienM | and AFIK the flavors deploy only one vNIC | 12:06 |
| *** jaosorior has quit IRC | 12:06 | |
| *** jaosorior has joined #tripleo | 12:07 | |
| sshnaidm | EmilienM, so we don't need eth1 for multinode job, right?? | 12:07 |
| EmilienM | sshnaidm: afk we fake it | 12:08 |
| EmilienM | sshnaidm: I'm looking at latest promotion job, what happpenned? | 12:08 |
| *** pradk has joined #tripleo | 12:09 | |
| openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Mistral workflow for creating a deployment plan from JSON input https://review.openstack.org/347830 | 12:09 |
| sshnaidm | EmilienM, there was https://bugs.launchpad.net/tripleo/+bug/1619147 | 12:09 |
| openstack | Launchpad bug 1619147 in tripleo "CI: periodic jobs fail on undercloud install because no package python-nova available in consistent repo" [Critical,Triaged] | 12:09 |
| sshnaidm | EmilienM, but retriggered them again today | 12:09 |
| EmilienM | I see puppet apply failed on undercloud | 12:10 |
| EmilienM | right python noca | 12:10 |
| EmilienM | sshnaidm: where do you retrigger? | 12:11 |
| thrash | d0ugal: I wonder if tirpleoclient package needs an explicit depends? | 12:11 |
| sshnaidm | EmilienM, there was a problem with parallel building of consistent repo, it wasn't built properly, jpena|lunch can give additional details about it | 12:11 |
| sshnaidm | EmilienM, https://review.openstack.org/#/c/359215/ | 12:12 |
| EmilienM | sshnaidm: thx | 12:12 |
| EmilienM | sshnaidm: it just failed | 12:12 |
| EmilienM | timeout again | 12:13 |
| sshnaidm | yes, right now | 12:13 |
| sshnaidm | arrgh | 12:13 |
| *** limao has quit IRC | 12:13 | |
| EmilienM | this is a nightmare | 12:13 |
| EmilienM | why suddenly our CI is really down | 12:13 |
| *** limao has joined #tripleo | 12:13 | |
| *** alee has quit IRC | 12:16 | |
| sshnaidm | it was waiting 15 min for the environment, 25 undercloud, 45 image building, and overcloud haven't finished within 1 hour 10 minutes | 12:17 |
| sshnaidm | 70 min for overcloud seems too much, isn't it? | 12:17 |
| d0ugal | thrash: for osc_lib? Yeah, it does. | 12:17 |
| thrash | d0ugal: i mean in the package.. | 12:18 |
| d0ugal | thrash: I agree, the rpm package does need it | 12:18 |
| thrash | d0ugal: ah. :) | 12:18 |
| *** skramaja_ has joined #tripleo | 12:19 | |
| *** skramaja has quit IRC | 12:19 | |
| *** lucas-hungry is now known as lucasagomes | 12:21 | |
| *** skramaja_ is now known as skramaja | 12:22 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:22 |
| *** fultonj has joined #tripleo | 12:23 | |
| *** yolanda has joined #tripleo | 12:25 | |
| thrash | d0ugal: the requires: is already there. | 12:25 |
| d0ugal | thrash: oh, good. Thanks for checking. | 12:25 |
| EmilienM | sshnaidm: is it something related to our infra itself? | 12:26 |
| EmilienM | sshnaidm: I'm really lost on what we can do | 12:26 |
| sshnaidm | EmilienM, the times are not good, everything is too much slow | 12:26 |
| thrash | d0ugal: https://review.rdoproject.org/r/gitweb?p=openstack/tripleoclient-distgit.git;a=blob;f=python-tripleoclient.spec;h=80f890d62d259c586681e0cde63ca4067eca9c3d;hb=HEAD#l40 | 12:27 |
| *** jayg|g0n3 is now known as jayg | 12:27 | |
| EmilienM | dprince, derekh: sorry to ping you guys, are you aware about the situation? | 12:27 |
| d0ugal | thrash: nice, everyone is way ahead of me :) | 12:29 |
| dprince | EmilienM: whats up? | 12:32 |
| * dprince reads back | 12:33 | |
| EmilienM | dprince: all OVB are down for 2 days | 12:33 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Configure Undercloud VM to start at virthost boot. https://review.openstack.org/351665 | 12:33 |
| EmilienM | dprince: some jobs are passing sometimes | 12:34 |
| EmilienM | dprince: but the rate is not enough for our periodic jobs to pass | 12:34 |
| EmilienM | dprince: and we haven't had a promotion for a while | 12:34 |
| EmilienM | https://dashboards.rdoproject.org/rdo-dev | 12:34 |
| EmilienM | 8 days | 12:34 |
| EmilienM | that is really catastrophic | 12:34 |
| *** limao has quit IRC | 12:34 | |
| dprince | EmilienM: should we manually promote? | 12:34 |
| *** limao has joined #tripleo | 12:35 | |
| EmilienM | dprince: I'm not sure, we should work on making the job pass | 12:35 |
| EmilienM | the rate of failure is really high and at this stage of the cycle we don't need that | 12:35 |
| dprince | EmilienM: that too | 12:35 |
| openstackgerrit | Merged openstack/tripleo-quickstart: Replace the native quickstart inventory with ansible-role-tripleo-inventory https://review.openstack.org/355962 | 12:35 |
| EmilienM | dprince: sshnaidm reported that our cloud is overloaded | 12:35 |
| EmilienM | jobs are timeouting | 12:36 |
| dprince | EmilienM: it is still rather new. OVB | 12:36 |
| d0ugal | thrash: Yeah, so I am getting weird errors with it :( | 12:36 |
| dprince | EmilienM: the rebuilt cloud that is | 12:36 |
| *** milan has quit IRC | 12:37 | |
| dprince | EmilienM: I need more info | 12:37 |
| sshnaidm | EmilienM, dprince I'll try to check network performance there today.. | 12:37 |
| *** jlinkes has quit IRC | 12:37 | |
| EmilienM | I have an idea but I guess people won't like it | 12:37 |
| *** jlinkes has joined #tripleo | 12:38 | |
| EmilienM | why not running promotion on multinode jobs | 12:38 |
| EmilienM | instead of OVB | 12:38 |
| EmilienM | multinode jobs are much more reliable | 12:38 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Use undercloud-virt-setup tag for Ironic pxe_ssh configuration https://review.openstack.org/348545 | 12:38 |
| EmilienM | ok they don't test ironic/nova baremetal but still are very useful | 12:38 |
| dprince | EmilienM: the multinode jobs don't use Ironic so we'd be at risk for some pretty crucial breakage there I think | 12:39 |
| EmilienM | yes I know | 12:39 |
| EmilienM | but right now we are at risk of releasing Newton without testing latest Newton | 12:39 |
| EmilienM | what I'm proposing is temporary | 12:40 |
| EmilienM | and if promotion broke OVB jobs totally (ie: ironic), we'll see it immediatly | 12:40 |
| EmilienM | because overcloud won't get started | 12:40 |
| EmilienM | right now, they timeout to finish | 12:41 |
| dprince | EmilienM: I'd rather just do a manual promition once or twice rather than implement a temporary multinode promotion mechanism | 12:41 |
| sshnaidm | EmilienM, dprince actually it may be network issue problem with consistent repo, just now checked access to current-tripleo (used in gate jobs) and consistent (used in periodic ones) | 12:41 |
| sshnaidm | EmilienM, dprince for current-tripleo it's 7.22MB/s for consistent: 1.58MB/s | 12:41 |
| sshnaidm | in 5 times slower | 12:41 |
| EmilienM | weird i thought it was the same server | 12:41 |
| dprince | sshnaidm: so the RDO repo is slower? | 12:41 |
| sshnaidm | dprince, yep and very slower | 12:42 |
| sshnaidm | it's main difference in configuration between periodic and gate jobs | 12:42 |
| sshnaidm | in total it can cause big timeouts | 12:42 |
| d0ugal | thrash: so, when I tried it, I hit an issue in Mistral client. I don't know why I don't hit it otherwise. Fixed with this: https://review.openstack.org/#/c/364275/ | 12:43 |
| EmilienM | sshnaidm: it's weird Puppet CI doesn't have that | 12:45 |
| EmilienM | sshnaidm: but I guess it's because we don't build images? | 12:45 |
| sshnaidm | EmilienM, it affects any package installation from repo | 12:46 |
| EmilienM | sshnaidm: puppet CI doesn't have the problem afik | 12:46 |
| EmilienM | we deploy all from RDO repo | 12:46 |
| sshnaidm | EmilienM, from which one? | 12:46 |
| sshnaidm | EmilienM, can you show an example? | 12:46 |
| *** stendulker has quit IRC | 12:47 | |
| sshnaidm | EmilienM, puppet Ci doesn't run on rh1 cloud, right? | 12:47 |
| thrash | d0ugal: ack | 12:48 |
| EmilienM | sshnaidm: nope | 12:48 |
| EmilienM | sshnaidm: on infra nodes | 12:48 |
| EmilienM | sshnaidm: using multiple cloud providers | 12:48 |
| EmilienM | sshnaidm: so it might be a networking infra issue on rh1 ? | 12:48 |
| sshnaidm | EmilienM, it may be a problem between rh1 and rdo.trunk | 12:48 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Rework the pacemaker_common_functions for M..N upgrades https://review.openstack.org/321027 | 12:48 |
| sshnaidm | EmilienM, which repo do you use? | 12:48 |
| sshnaidm | weshay, ^^ | 12:49 |
| EmilienM | sshnaidm: https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L33-L44 | 12:49 |
| dprince | sshnaidm: how many periodic jobs run concurrently? | 12:49 |
| sshnaidm | EmilienM, these repos are ok on rh1 too | 12:49 |
| sshnaidm | dprince, 4 | 12:50 |
| *** rlandy has joined #tripleo | 12:50 | |
| EmilienM | sshnaidm: what repo cause problem? | 12:50 |
| sshnaidm | dprince, sorry, 5 | 12:50 |
| EmilienM | i'm going to try from home | 12:50 |
| EmilienM | give me an URL | 12:50 |
| EmilienM | (please) | 12:50 |
| *** links has quit IRC | 12:51 | |
| sshnaidm | EmilienM, https://trunk.rdoproject.org/centos7/consistent | 12:51 |
| EmilienM | thanks | 12:51 |
| sshnaidm | EmilienM, try to download these links and look at the speed: | 12:51 |
| EmilienM | sshnaidm: wait, it's not the same server as centos7-master? | 12:51 |
| sshnaidm | wget http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/openstack-dashboard-10.0.0-0.20160821123117.0e0b01f.el7.centos.noarch.rpm -O - > /dev/null | 12:51 |
| sshnaidm | wget https://trunk.rdoproject.org/centos7/consistent/openstack-dashboard-10.0.0-0.20160831115927.152d387.el7.centos.noarch.rpm -O - > /dev/null | 12:52 |
| sshnaidm | EmilienM, no, it's different | 12:52 |
| EmilienM | sshnaidm: I have 3MB/s | 12:52 |
| EmilienM | well it goes down to less | 12:53 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup https://review.openstack.org/358089 | 12:53 |
| EmilienM | yeah much less | 12:53 |
| *** Goneri has joined #tripleo | 12:54 | |
| *** abregman_ has quit IRC | 12:54 | |
| sshnaidm | EmilienM, maybe it's worth to check a few times, it's not stable, but http://buildlogs.centos.org/centos/7 give much more faster results, in 10-20 times | 12:54 |
| EmilienM | the problem is that there is no big file | 12:55 |
| EmilienM | we need jpena to access on server | 12:55 |
| *** jpena|lunch is now known as jpena | 12:55 | |
| sshnaidm | EmilienM, to put a big file there? | 12:55 |
| EmilienM | jpena: hey | 12:55 |
| EmilienM | jpena: we need you rhelp | 12:55 |
| *** limao has quit IRC | 12:56 | |
| jpena | EmilienM: yes? | 12:56 |
| *** limao has joined #tripleo | 12:56 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup https://review.openstack.org/358089 | 12:56 |
| EmilienM | jpena: tripleo periodic jobs timeout for some reasons, and we think the networking between rh1 and https://trunk.rdoproject.org/centos7/consistent is too slow | 12:57 |
| EmilienM | can you put a big file with dd in https://trunk.rdoproject.org/centos7/consistent please? | 12:57 |
| EmilienM | jpena: it doesn't seem we have the problem with other repos ie https://trunk.rdoproject.org/centos7-master/ | 12:57 |
| jpena | EmilienM: sure, would 1G be enough? | 12:57 |
| EmilienM | jpena: yes please | 12:57 |
| sshnaidm | EmilienM, jpena then also on http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/ | 12:57 |
| jpena | sshnaidm: that's a bit more complicated, the centos CDN is synced via a periodic job (we don't have direct access there), and once a file is uploaded, it's complicated to remove | 12:58 |
| *** lblanchard has joined #tripleo | 12:59 | |
| sshnaidm | jpena, ok, let's start from consistent.. | 12:59 |
| EmilienM | sshnaidm: do you have ssh access on rh1? | 13:01 |
| sshnaidm | EmilienM, yep | 13:01 |
| EmilienM | sshnaidm: i'll do the test from home but I'll let you test from rh1 | 13:01 |
| sshnaidm | EmilienM, sure | 13:01 |
| sshnaidm | EmilienM, from my home it seems to be similar | 13:01 |
| *** sshnaidm is now known as sshnaidm|mtg | 13:02 | |
| EmilienM | jpena: when you're done with consistent, can you also create a 1g file on https://trunk.rdoproject.org/centos7-master/ ? | 13:02 |
| EmilienM | so we can compare | 13:02 |
| jpena | EmilienM: https://trunk.rdoproject.org/centos7/consistent/testfile | 13:02 |
| EmilienM | thanks | 13:02 |
| *** numans has quit IRC | 13:02 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone https://review.openstack.org/327029 | 13:02 |
| jpena | EmilienM centos7-master and centos7 are the same, or do you mean outside the "consistent" dir | 13:03 |
| EmilienM | from home, I'm at 3.4MB/s on average | 13:03 |
| EmilienM | jpena: aren't they on same server? | 13:03 |
| EmilienM | sshnaidm|mtg: I have good results :( | 13:03 |
| jpena | EmilienM; yes | 13:04 |
| sshnaidm|mtg | EmilienM, from rh1 it's from 4 to 6 MB/s | 13:04 |
| EmilienM | ok I don't think it's on rdo server side for consistent | 13:04 |
| bswartz | tbarron: pong | 13:04 |
| *** akrivoka has quit IRC | 13:05 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient: WIP wire in jinja templating for custom roles https://review.openstack.org/336625 | 13:05 |
| EmilienM | jpena: thank you | 13:05 |
| EmilienM | jpena: you can delete the file on my side | 13:05 |
| jpena | EmilienM: ok, removing | 13:06 |
| sshnaidm|mtg | 5.47 MB/s for rh1 | 13:06 |
| EmilienM | sshnaidm|mtg: it's good | 13:06 |
| EmilienM | let's look at the logs again | 13:06 |
| *** cdearborn has joined #tripleo | 13:08 | |
| EmilienM | http://logs.openstack.org/15/359215/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/5544bc7/console.html#_2016-09-01_10_10_45_238961 | 13:08 |
| sshnaidm|mtg | EmilienM, I don't know.. from http://buildlogs.centos.org/centos/ I get about 50 MB/s , how could it be? | 13:08 |
| EmilienM | so yeah the image build is taking 1h ! | 13:08 |
| sshnaidm|mtg | EmilienM, yes, it take about 45-50 minutes | 13:09 |
| EmilienM | dprince: what is the URL again for tripleo CI metrics? | 13:09 |
| sshnaidm|mtg | EmilienM, https://23.253.94.71/dashboard/#temporary-3 | 13:10 |
| *** abregman_ has joined #tripleo | 13:10 | |
| EmilienM | sshnaidm|mtg: thx | 13:10 |
| *** cdearborn has quit IRC | 13:10 | |
| tbarron | bswartz: hi | 13:11 |
| sshnaidm|mtg | jpena, is trunk.rdoproject.org/centos7/consistent and http://buildlogs.centos.org/centos on one real server indeed? | 13:11 |
| tbarron | bswartz: took your name in vain earlier, marios has given us an explicit overcloud deploy command which I will try out today, see backlog for our plan | 13:12 |
| jpena | sshnaidm|mtg: nope. trunk.rdo is a single server, synced from the build server at ci.centos.org. buildlogs is the CDN, also synced from the build server | 13:12 |
| jpena | CDN == CentOS CDN | 13:12 |
| sshnaidm|mtg | EmilienM, ^^ | 13:12 |
| sshnaidm|mtg | jpena, that's why I get files from cdn in one moment | 13:13 |
| *** cdearborn has joined #tripleo | 13:13 | |
| derekh | EmilienM: sshnaidm|mtg was at lunch, what can I hepl with? | 13:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add `overcloud parameters set` to set Heat params in a plan https://review.openstack.org/360391 | 13:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 13:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Print the execution ID when starting Mistral workflows https://review.openstack.org/340711 | 13:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add `openstack overcloud plan deploy` https://review.openstack.org/360305 | 13:13 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral https://review.openstack.org/342261 | 13:13 |
| *** abregman_ is now known as abregman|mtg | 13:13 | |
| sshnaidm|mtg | derekh, I just asked about rh1 interface, sent you a mail, not urgent | 13:13 |
| dprince | EmilienM: https://23.253.94.71/ | 13:13 |
| *** akshai has joined #tripleo | 13:14 | |
| EmilienM | derekh: we're investigating why ovb jobs are timeouting as hell | 13:14 |
| sshnaidm|mtg | jpena, and can we mirror consistent on cdn? | 13:15 |
| *** [1]cdearborn has joined #tripleo | 13:15 | |
| derekh | sshnaidm|mtg: ok, looking | 13:16 |
| *** tzumainn has joined #tripleo | 13:16 | |
| EmilienM | the image built logs: http://logs.openstack.org/15/359215/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/fbd5228/logs/undercloud/var/log/image_build.txt.gz | 13:16 |
| jpena | sshnaidm|mtg it is mirrored, http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master/ | 13:16 |
| EmilienM | is it normal to take more than 30 min for image build? | 13:16 |
| jpena | note that due to the way mirroring is done, it's not *just* what is in consistent, but also the previous versions are there | 13:17 |
| *** limao has quit IRC | 13:17 | |
| derekh | EmilienM: bnemec has been looking into that for the last few days, the machines we host this stuff on appear to be under preforming but not sure why yet | 13:17 |
| jpena | so previous RPMs from "consistent" can be found (although the latest ones should win) | 13:17 |
| *** limao has joined #tripleo | 13:17 | |
| derekh | EmilienM: sshnaidm|mtg also I've just noticed we don't seem to be using the proxy server | 13:17 |
| *** [1]cdearborn has quit IRC | 13:18 | |
| EmilienM | on August 26th we also used 30 min or so | 13:18 |
| sshnaidm|mtg | jpena, hmm.. as I see they have different delorean hash.. | 13:18 |
| EmilienM | I'm trying to find out if image building was faster | 13:18 |
| openstackgerrit | Martin André proposed openstack/puppet-tripleo: Manage tripleo-ui configuration files with puppet https://review.openstack.org/363167 | 13:19 |
| *** pgadiya has quit IRC | 13:19 | |
| *** akrivoka has joined #tripleo | 13:19 | |
| *** yamahata has joined #tripleo | 13:19 | |
| jpena | sshnaidm|mtg, I'm comparing them right now: name=delorean-python-networking-cisco-8e9986665cd960b47914b969c34e4ea9240c10ba in both | 13:20 |
| sshnaidm|mtg | jpena, you're right, sorry | 13:20 |
| EmilienM | on mitaka CI jobs, image build was around 25 min | 13:20 |
| sshnaidm|mtg | I think I can try this repo instead of "consistent" | 13:20 |
| EmilienM | and now it's around 30 | 13:20 |
| EmilienM | so not a really big deal | 13:20 |
| EmilienM | (well it's still 5 min but not timeout worthing) | 13:20 |
| *** cdearborn has quit IRC | 13:21 | |
| jaosorior | EmilienM: hey dude, do you mind if I merge this one? https://review.openstack.org/#/c/363578/ it just adds one package and it already passed nonha. | 13:21 |
| jpena | sshnaidm|mtg be aware that if you just download the delorean.repo file, it will point you back at trunk.rdo (the single server), so you need to build the repo config yourself | 13:21 |
| EmilienM | jaosorior: done | 13:21 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: configure disk erasing during cleaning https://review.openstack.org/364304 | 13:21 |
| jaosorior | EmilienM: alright, thanks man | 13:21 |
| sshnaidm|mtg | jpena, oh, that's sad.. | 13:21 |
| shardy | gfidente: Hey, did you look into refactoring puppet/swift-devices-and-proxy-config.yaml into the swift composable services template and/or puppet-tripleo? | 13:21 |
| shardy | I seem to recall discussing it, but could be mistaken | 13:22 |
| shardy | I think we need to move that out of overcloud.yaml to enable custom-roles | 13:22 |
| EmilienM | I have some doubts about if we're looking at the right place, image building is not so terrible except in one CI job I saw where it timeouted | 13:22 |
| sshnaidm|mtg | jpena, so I need just to replace a hostname and url path there? | 13:22 |
| jpena | sshnaidm|mtg, yes, that should be enough | 13:23 |
| sshnaidm|mtg | EmilienM, I'll try to use CDN after a meeting and retrigger the jobs, let's see if it helps | 13:23 |
| panda | EmilienM: still no ipv6 job in experimental queue it seems, how many times do they reload zuul configuration ? | 13:24 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Decouple deployment plans from Heat stacks https://review.openstack.org/363950 | 13:25 |
| *** cdearborn has joined #tripleo | 13:25 | |
| *** alee has joined #tripleo | 13:29 | |
| *** masco has quit IRC | 13:29 | |
| *** bfournie has joined #tripleo | 13:30 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Fix pingtest in tripleo/validate to use template https://review.openstack.org/364178 | 13:32 |
| EmilienM | panda: well it should be here. | 13:32 |
| EmilienM | panda: let me check project config | 13:32 |
| EmilienM | sshnaidm|mtg: ok | 13:32 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: configure disk erasing during cleaning https://review.openstack.org/364304 | 13:34 |
| *** skramaja has quit IRC | 13:35 | |
| *** skramaja has joined #tripleo | 13:36 | |
| *** limao has quit IRC | 13:38 | |
| dprince | sshnaidm|mtg: so, should we attempt to enable caching tomorrow evening perhaps? | 13:38 |
| *** limao has joined #tripleo | 13:39 | |
| dprince | sshnaidm|mtg: that won't help the periodic jobs but it will help other ones right | 13:39 |
| sshnaidm|mtg | dprince, for periodic jobs it will not help, anyway we need to build images there | 13:39 |
| dprince | sshnaidm|mtg: correct | 13:39 |
| sshnaidm|mtg | dprince, yeah, for others it will be easier | 13:39 |
| EmilienM | panda: I might have found out | 13:40 |
| openstackgerrit | Imre Farkas proposed openstack/tripleo-common: Default to Ironic API v1.15 https://review.openstack.org/364319 | 13:41 |
| panda | EmilienM: *drumroll* | 13:42 |
| *** rajinir has joined #tripleo | 13:43 | |
| *** morazi has joined #tripleo | 13:43 | |
| *** ayoung has joined #tripleo | 13:43 | |
| *** sshnaidm|mtg is now known as sshnaidm | 13:44 | |
| *** egafford has joined #tripleo | 13:46 | |
| ramishra_ | :q! | 13:47 |
| ramishra_ | oops! | 13:47 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 13:48 |
| * sshnaidm crosssed fingers | 13:48 | |
| derekh | So jobs are currently passing, it it just the lentch of time it take the periodic job to run that ye are trying to figure out ? | 13:49 |
| *** openstackgerrit has quit IRC | 13:49 | |
| sshnaidm | derekh, right | 13:49 |
| *** openstackgerrit has joined #tripleo | 13:49 | |
| EmilienM | panda: https://review.openstack.org/364324 | 13:49 |
| *** fultonj_ has joined #tripleo | 13:51 | |
| EmilienM | dprince, sshnaidm, derekh: maybe can we manually promote, and see how CI works. Worst case we roll back | 13:51 |
| EmilienM | FWIW, Puppet CI is all green with latest consistent, I think tripleo CI would also be ok | 13:52 |
| dprince | EmilienM: ++ | 13:52 |
| EmilienM | I suggest to wait for sshnaidm's last test on periodic | 13:52 |
| EmilienM | and if result is negative, we go ahead and promote manually | 13:52 |
| dprince | EmilienM: I'd like to promote more often. But 8 days isn't that old | 13:52 |
| EmilienM | dprince: 8 days at this stage of cycle is critical | 13:53 |
| dprince | EmilienM: but still, I'd be willing to try it | 13:53 |
| EmilienM | dprince: other projets are landing stuff before end of cycle | 13:53 |
| dprince | EmilienM: agree, it isn't idea | 13:53 |
| dprince | l | 13:53 |
| EmilienM | and things could break for us | 13:53 |
| sshnaidm | dprince, afaiu rdo is waiting for us now | 13:53 |
| EmilienM | dprince: what is the mechanism to promote manually? | 13:53 |
| derekh | EmilienM: If you manually promote, then all jobs will just start failing, how could it not? | 13:54 |
| sshnaidm | dprince, EmilienM we need good built images to be uploaded too, if we care about them | 13:54 |
| EmilienM | sshnaidm: right | 13:55 |
| *** rodrigods has quit IRC | 13:55 | |
| sshnaidm | and it's usually stops before images are uploaded | 13:55 |
| *** rodrigods has joined #tripleo | 13:55 | |
| EmilienM | derekh: what do you suggest? | 13:55 |
| sshnaidm | derekh, could it be possible to organize a dedicated environment for promotion jobs? | 13:56 |
| sshnaidm | let's say with lot of cpu, memory.. | 13:56 |
| sshnaidm | baremetal maybe | 13:57 |
| derekh | EmilienM: My suggestion would be to continue to try a repository that we think will pass, similar to what sshnaidm is doing, if its timeing out because something is taking too long then figure it out | 13:58 |
| derekh | sshnaidm: we can't do that, we'd risk a promotion of something that wont pass in the normal ci env's | 13:58 |
| EmilienM | derekh: okay | 13:58 |
| *** limao has quit IRC | 14:00 | |
| *** limao has joined #tripleo | 14:00 | |
| sshnaidm | derekh, and hacking zuul maximum time is not an option also, right? | 14:00 |
| EmilienM | sshnaidm: timeout limit you mean? | 14:01 |
| EmilienM | sshnaidm: IMHO 2h30 is already too much | 14:01 |
| EmilienM | I remember some jobs taking less than 1h30 a few time ago | 14:02 |
| sshnaidm | derekh, yes | 14:02 |
| derekh | sshnaidm: technically it would work, but EmilienM is right its already long enough and I'd expect pushback from infra | 14:02 |
| EmilienM | they won't accept it | 14:02 |
| trown | is the whole reason for increased time that we arent using the proxy? | 14:02 |
| EmilienM | and increasing the timeout is just moving the problem | 14:03 |
| EmilienM | the idea of using multinode job to promote might be considered here. | 14:03 |
| EmilienM | even if ironic is not tested in this job, though all the rest is tested. | 14:03 |
| EmilienM | and the job rarely fails/timeout. | 14:03 |
| sshnaidm | trown, I see there export http_proxy=${http_proxy-"http://192.168.1.100:3128/"} | 14:03 |
| sshnaidm | trown, isn't it | 14:03 |
| sshnaidm | ? | 14:04 |
| *** jaosorior has quit IRC | 14:04 | |
| derekh | trown: possibly, we should go back to using it again | 14:04 |
| trown | EmilienM: doesnt that have the same problem of potentially promoting something that wont work for gates? | 14:04 |
| EmilienM | trown: why? | 14:04 |
| EmilienM | I fail to understand why | 14:04 |
| derekh | sshnaidm: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_instack_ovb.sh#n67 | 14:04 |
| trown | derekh: seems like package download is taking a long time is why I thought that... that would not be the case with proxy | 14:05 |
| derekh | trown: yup, it only got removed from being used by accident I think | 14:05 |
| trown | EmilienM: promote job mimic what is done in gates... so if we promote something that does not pass the way it is currently set up, we risk it breaking the gates | 14:06 |
| sshnaidm | derekh, is this squid? | 14:06 |
| derekh | sshnaidm: yes | 14:06 |
| sshnaidm | derekh, so to remove this line and give a try? | 14:06 |
| sshnaidm | derekh, is it on undercloud machine on rh1? | 14:06 |
| EmilienM | trown: what do you mean by "gates"? My proposal is about to run periodic job on multinode env and promote usin git | 14:07 |
| EmilienM | trown: multinode vs ovb | 14:07 |
| trown | EmilienM: right the ovb gates are what I am talking about | 14:07 |
| derekh | sshnaidm: yup, give it a go, there might be a place we need to selectively not use the proxy but try and find out, just push up a patch that removes the line and see what happens | 14:07 |
| EmilienM | trown: ok because they don't actually "gate", as they are non voting. | 14:08 |
| EmilienM | trown: though the multinode job is voting, and in gate. | 14:08 |
| derekh | sshnaidm: do a nova list as openstack-nodepool and you'll see a proxy-server, thats squid | 14:08 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use proxy on OVB environemnt https://review.openstack.org/364335 | 14:09 |
| *** assassin has quit IRC | 14:10 | |
| *** ubijtsa has joined #tripleo | 14:10 | |
| dtantsur | EmilienM, trown, so it boils down to what we favor: breaking periodic jobs more often or breaking OVB jobs on actual patches more often. A question of priorities. | 14:10 |
| *** ubijtsa is now known as Guest70368 | 14:10 | |
| dtantsur | EmilienM, trown, my first reaction was to favor breaking periodic jobs, but on a second thought, OVB jobs not passing on patches will be a more explicit signal for people to fix the gate first | 14:11 |
| dtantsur | sooo... really good question to raise, EmilienM | 14:11 |
| EmilienM | in my opinion, baremetal (now OVB) jobs have reached a very unstable stage in the last cycle. We should stop relying on them for promotions | 14:12 |
| beagles | what? no deep dive? | 14:12 |
| * beagles ducks | 14:12 | |
| dtantsur | EmilienM, maybe we're putting too much stuff in OVB promotion jobs? maybe we should only leave the simplest non-HA job there (even without SSL)? | 14:13 |
| dtantsur | maybe it could be a compromise between not testing ironic and nova at all and being always broken? | 14:13 |
| dtantsur | (and plus multinode job as well) | 14:14 |
| EmilienM | dtantsur: that is an excellent idea IMHO. | 14:14 |
| *** ebalduf has joined #tripleo | 14:16 | |
| shardy | Is there a convenient way to add a suffix to all list items in puppet, like the opposite of the stdlib "prefix" function? | 14:17 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 14:18 |
| shardy | https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/lib/puppet/parser/functions/suffix.rb | 14:19 |
| shardy | hmm, looks pretty new | 14:19 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Sync the config of the ha JOBTYPE with ha2 https://review.openstack.org/362974 | 14:20 |
| mwhahaha | they added suffix in the 4.x series | 14:20 |
| shardy | mwhahaha: ack, Ok thanks, I guess I'll do it longhand then | 14:20 |
| *** coolsvap has quit IRC | 14:22 | |
| derekh | sshnaidm: you patch failed the multinode jobs and is probably going to fail OVB because we can no longer read the repo hash from delorean.repo | 14:22 |
| skramaja | shardy: EmilienM we have created an environment file for SR-IOV for, could you please take a look - https://review.openstack.org/#/c/363617/ - CI passed. | 14:22 |
| EmilienM | skramaja: currently in the middle of releasing tripleo newton-3 | 14:23 |
| EmilienM | bad timing | 14:23 |
| skramaja | oops. sorry. | 14:23 |
| *** liverpooler has quit IRC | 14:23 | |
| skramaja | will ping later then.. | 14:23 |
| EmilienM | skramaja: I'll look later | 14:23 |
| skramaja | thanks. | 14:23 |
| *** limao has quit IRC | 14:24 | |
| *** limao has joined #tripleo | 14:25 | |
| sshnaidm | derekh, where do we use this TRUNKREPOUSED? | 14:27 |
| sshnaidm | derekh, oh, I see, it's in uploading.. ok | 14:27 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Add mod_ssl to overcloud-controller element https://review.openstack.org/363578 | 14:28 |
| *** Guest70368 has quit IRC | 14:28 | |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Allow net config to proceed if interface is already bound to DPDK driver https://review.openstack.org/364354 | 14:29 |
| derekh | sshnaidm: yup, that how if know what to repo to promote | 14:29 |
| *** saneax is now known as saneax-_-|AFK | 14:30 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 14:30 |
| *** ubijtsa_ has joined #tripleo | 14:30 | |
| *** masco has joined #tripleo | 14:32 | |
| EmilienM | skramaja: +2 | 14:36 |
| *** gfidente has quit IRC | 14:36 | |
| skramaja | thanks EmilienM | 14:36 |
| * EmilienM wonders what shardy is preparing with suffix :-) | 14:37 | |
| shardy | EmilienM: I want to create this list in the puppet profile | 14:38 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/swift-devices-and-proxy-config.yaml#L32 | 14:38 |
| shardy | It looks like ["192.0.2.18:11211"] | 14:38 |
| shardy | I have a list which looks like ["192.0.2.18"] that is hiera memcached_node_ips | 14:38 |
| shardy | I need to append the port | 14:39 |
| EmilienM | ah I did that one time | 14:39 |
| EmilienM | a sec | 14:39 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 14:39 |
| * shardy expected this to be easy :) | 14:39 | |
| EmilienM | https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy/endpoint.pp#L91 | 14:39 |
| shardy | EmilienM: perfect, thanks! | 14:40 |
| *** dhill_ has joined #tripleo | 14:40 | |
| EmilienM | shardy: see my unit tests to see what are results https://github.com/openstack/puppet-tripleo/blob/master/spec/defines/tripleo_haproxy_endpoint_spec.rb | 14:40 |
| shardy | EmilienM: so the suffix function isn't puppet4 only then? | 14:41 |
| *** abregman|mtg has quit IRC | 14:41 | |
| EmilienM | no you can use it in puppet3, we do use it in tripleo (pupppet3) | 14:41 |
| openstackgerrit | Martin André proposed openstack/puppet-tripleo: Manage tripleo-ui configuration files with puppet https://review.openstack.org/363167 | 14:41 |
| shardy | EmilienM: ack, from the docs it looked like I couldn't use it, thanks | 14:41 |
| EmilienM | I think what Alex says is that the function is now native in puppet 4 | 14:42 |
| *** limao has quit IRC | 14:42 | |
| EmilienM | shardy: which doc? | 14:42 |
| *** limao has joined #tripleo | 14:43 | |
| shardy | EmilienM: actually the git tag | 14:44 |
| shardy | https://github.com/puppetlabs/puppetlabs-stdlib/commit/88a93ac6cdf38045e1cf29325a70e5e4143016b3 | 14:44 |
| shardy | 4.0.0 | 14:44 |
| shardy | EmilienM: thanks for the help | 14:44 |
| EmilienM | ah no | 14:44 |
| EmilienM | 4.0.0 is the tag of stdlib :) not puppet | 14:44 |
| EmilienM | so no worries | 14:44 |
| EmilienM | AFIK stlib can be used on both puppet 3 and 4 | 14:44 |
| shardy | Aha, my mistake then, thanks! | 14:44 |
| EmilienM | this information can be confirmed here: https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/metadata.json#L108 | 14:45 |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Add mod_auth_mellon to overcloud-controller element https://review.openstack.org/363579 | 14:45 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 14:48 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Create memcache_servers list in swift proxy profile https://review.openstack.org/364367 | 14:50 |
| *** jcoufal has joined #tripleo | 14:50 | |
| dtantsur | folks, do we recheck things that are optional, if e.g. the pass one of the OVB gates? | 14:50 |
| dtantsur | a specific example: https://review.openstack.org/#/c/364213/ | 14:50 |
| dtantsur | I don't want to put unneeded load on the CI | 14:51 |
| EmilienM | shardy: +2 it will work I think | 14:51 |
| shardy | EmilienM: ack, thanks, testing locally now | 14:52 |
| bnemec | dtantsur: That actually passed the job, but timed out during postci. Personally I wouldn't require a recheck on it. | 14:52 |
| openstackgerrit | Saravanan KR proposed openstack/os-net-config: Allow net config to proceed if interface is already bound to DPDK driver https://review.openstack.org/364354 | 14:53 |
| dtantsur | that's what I assumed. Hopefully folks won't just walk away after seing this POST_FAILURE :) | 14:53 |
| *** abregman|mtg has joined #tripleo | 14:53 | |
| *** sshnaidm has quit IRC | 14:55 | |
| bnemec | dtantsur: Ping me if someone else +2's but doesn't approve. I'll be happy to do it. | 14:57 |
| dtantsur | thanks! | 14:57 |
| EmilienM | dtantsur: +A | 14:58 |
| dtantsur | \o/ | 14:58 |
| *** sshnaidm has joined #tripleo | 15:00 | |
| *** pcaruana has quit IRC | 15:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 15:03 |
| *** limao_ has joined #tripleo | 15:04 | |
| *** fzdarsky has quit IRC | 15:04 | |
| *** trozet_ is now known as trozet | 15:04 | |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Create memcache_servers list in swift proxy profile https://review.openstack.org/364367 | 15:06 |
| *** rcernin has quit IRC | 15:06 | |
| *** limao has quit IRC | 15:07 | |
| *** aufi has quit IRC | 15:07 | |
| jpich | jtomasek: Is plan editing still a WIP in the UI at the moment, or am I perhaps missing some things in my environment? I get an error when I try to look at the parameters list for the default/any plan ( http://paste.openstack.org/show/565729/ ) | 15:09 |
| *** limao_ has quit IRC | 15:09 | |
| *** ubijtsa_ is now known as ubijtsa | 15:09 | |
| *** ubijtsa is now known as Guest4855 | 15:09 | |
| jtomasek | jpich: I am seeing this too. A workaround is to set that parameter in parameter_defaults section of overcloud-resource-registry-pupet.yaml and recreate the plan | 15:12 |
| *** fzdarsky has joined #tripleo | 15:12 | |
| jtomasek | jpich: It seems that parameter does not have any default value and so the validation won't pass | 15:12 |
| jtomasek | shardy: is this known thing?^ | 15:12 |
| jpich | jtomasek: I see, thank you for the workaround! Should I open a bug about this? | 15:13 |
| jtomasek | jpich: that would be great, I am not exactly sure what the error message about the parameter means | 15:13 |
| shardy | jtomasek: Yes, I thought we'd discussed it before, we'll still need the workaround for Newton because the heat bug in nested validation didn't get fixed | 15:14 |
| *** pblaho has quit IRC | 15:14 | |
| shardy | I sent a message to openstack-dev with this, among other, issues listed | 15:14 |
| * shardy finds the link | 15:14 | |
| shardy | http://lists.openstack.org/pipermail/openstack-dev/2016-August/102297.html | 15:15 |
| shardy | jtomasek: it's a side-effect of not knowing if a value was passed from the parent | 15:15 |
| shardy | it means at validation time, required parameters in nested templates see None | 15:16 |
| shardy | so we'll need to work around it with some parameter_defaults for now | 15:16 |
| shardy | (it should only be the Passwords I think) | 15:16 |
| dtantsur | calling it a day, see you tomorrow | 15:17 |
| *** dtantsur is now known as dtantsur|afk | 15:17 | |
| jtomasek | shardy: I see. I've seen this happening only with this single parameter (DeployIdentifier) I am able to succesfully run heat validation even when Passwords are not set | 15:17 |
| derekh | sshnaidm: btw, I'm also manually running a promote job, will see what I can do to help it through, will check back on it tonight and try and figure out any problems | 15:17 |
| sshnaidm | derekh, ok, thanks | 15:18 |
| *** oneswig has joined #tripleo | 15:18 | |
| sshnaidm | derekh, just ran it again, a stupid typo there.. | 15:18 |
| derekh | sshnaidm: k | 15:18 |
| shardy | jtomasek: Hmm, that's strange, because we default DeployIdentifer in overcloud.yaml | 15:18 |
| shardy | it's possible we don't in some other template tho | 15:19 |
| shardy | jtomasek: are you sure DeployIdentifier isn't getting passed as None during validation? | 15:19 |
| shardy | we had a bug in tripleoclient related to that | 15:19 |
| jtomasek | shardy: how could I test it? | 15:19 |
| shardy | jtomasek: look at the debug logs from mistral and see what the DeployIdentifier in the call to heat is set to | 15:20 |
| *** tremble has quit IRC | 15:20 | |
| d0ugal | shardy, jtomasek - it sounds like you want: https://review.openstack.org/#/c/361440/ | 15:20 |
| * d0ugal is guessing a bit | 15:21 | |
| *** dsariel has quit IRC | 15:22 | |
| shardy | d0ugal: Yeah that looks good, but I don't think that's related to validation? | 15:22 |
| shardy | I approved it anyway, lgtm | 15:22 |
| jtomasek | d0ugal: that could solve it, but problem is that untill we set the DeployIdentifier, we can't display parameters, so doing it on deploy is late | 15:22 |
| *** leanderthal is now known as leanderthal|afk | 15:22 | |
| *** jcoufal_ has joined #tripleo | 15:23 | |
| jtomasek | (in terms of making heat-validate pass) | 15:23 |
| shardy | during validation we could just set it to "" in the parameters passed to heat | 15:23 |
| d0ugal | aha, well, at least deploying is a bit easier now :) | 15:23 |
| jtomasek | shardy: right | 15:23 |
| d0ugal | dprince, rbrady: FYI https://bugs.launchpad.net/tripleo/+bug/1619350 | 15:24 |
| openstack | Launchpad bug 1619350 in tripleo "Refactor tests in tripleoclient" [Medium,Confirmed] | 15:24 |
| shardy | Or, we can assign a default value in puppet-*post.yaml and puppet/all-nodes-config.yaml in tht | 15:24 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup https://review.openstack.org/358089 | 15:24 |
| jpich | jtomasek, shardy, d0ugal: Thanks! I opened https://bugs.launchpad.net/tripleo/+bug/1619352 if we want to add information about how/where best to fix this | 15:25 |
| openstack | Launchpad bug 1619352 in tripleo "Cannot edit plan: "The Parameter (DeployIdentifier) was not provided."" [Undecided,New] | 15:25 |
| shardy | jtomasek: I'm still confused why you don't need to default the passwords, I'm guessing that is already handled in the mistral action? | 15:25 |
| openstackgerrit | Emilien Macchi proposed openstack/instack-undercloud: Work around deletion of _member_ role assignments on upgrade https://review.openstack.org/307352 | 15:25 |
| *** jcoufal has quit IRC | 15:25 | |
| d0ugal | shardy: There is nothing in the Mistral work to handle passwords (yet?) | 15:26 |
| shardy | Ok, then we may as well default DeployIdentifier there as I'm pretty sure we'll need to add dummy passwords for the nested validation to pass | 15:27 |
| jtomasek | shardy: no idea tbh, it is confusing me too. I was thiking that jdob resolved the issue with validation failing when parameter has no default | 15:27 |
| shardy | jtomasek: have you tested, and confirmed that only setting DeployIdentier works for you? | 15:27 |
| jtomasek | shardy: yes | 15:28 |
| jtomasek | shardy: when I set that parameter in parameter_defaults of overcloud-resource-registry-puppet.yaml the validation passes | 15:28 |
| *** ifarkas is now known as ifarkas_afk | 15:28 | |
| jtomasek | shardy: note that I used only that single environment though (so not all templates are used) | 15:29 |
| EmilienM | can someone approve this easy patch? https://review.openstack.org/#/c/363178/ | 15:30 |
| jtomasek | shardy: when I run getParameters mistral action I am not seeing any other errors, although when I tried to deploy. I was hitting SNMPReadonlyPassword is not set error | 15:30 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add default for DeployIdentifier in nested templates https://review.openstack.org/364388 | 15:30 |
| *** rcernin has joined #tripleo | 15:31 | |
| shardy | jtomasek: ack, please can you see if that solves the getParameters issue? | 15:31 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE https://review.openstack.org/359796 | 15:31 |
| jtomasek | shardy: on it | 15:31 |
| shardy | On deployment we'll have to generate random passwords, just like tripleoclient currently does | 15:32 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Do not merge - test Telemetry services (scenario001) https://review.openstack.org/364390 | 15:32 |
| EmilienM | pradk: FYI ^ I'm checking telemetry actually works again | 15:32 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/instack-undercloud: Add option to set a custom dnsmasq configuration file https://review.openstack.org/364391 | 15:33 |
| pradk | EmilienM, cool, is there a way i can execute scenario1 locally ? | 15:33 |
| jtomasek | shardy: ack, works | 15:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Do not merge - test Cinder services (scenario002) https://review.openstack.org/364392 | 15:34 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Do not merge - test Sahara services (scenario003) https://review.openstack.org/364393 | 15:34 |
| EmilienM | pradk: yes | 15:34 |
| openstackgerrit | Harry Rybacki proposed openstack/tripleo-quickstart: Add config parameter to collect-logs ci-script https://review.openstack.org/364394 | 15:35 |
| EmilienM | pradk: https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/scenario001-multinode.yaml | 15:35 |
| *** fzdarsky has quit IRC | 15:35 | |
| pradk | EmilienM, so just heat stack create the yaml? or is there anything else i need to do? | 15:35 |
| EmilienM | pradk: and pingtest is here: https://github.com/openstack-infra/tripleo-ci/blob/master/templates/scenario001-multinode-pingtest.yaml | 15:35 |
| jtomasek | shardy: I think you wanted to reference the other bug then one you mention in commit message https://bugs.launchpad.net/tripleo/+bug/1619352 | 15:36 |
| openstack | Launchpad bug 1619352 in tripleo "Cannot edit plan: "The Parameter (DeployIdentifier) was not provided."" [High,Confirmed] | 15:36 |
| jtomasek | shardy: or maybe not, I am not sure:) | 15:36 |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Add default for DeployIdentifier in nested templates https://review.openstack.org/364388 | 15:37 |
| shardy | jtomasek: Yeah I guess I copied the wrong one out of IRC ;) | 15:37 |
| *** rhallisey has quit IRC | 15:39 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 15:39 |
| rbrady | jrist, jtomasek: I've been told this bug is blocking the UI. Do either of you have a moment to discuss it? https://bugs.launchpad.net/mistral/+bug/1617351 | 15:45 |
| openstack | Launchpad bug 1617351 in Mistral "No output in executions listing" [Undecided,Confirmed] - Assigned to Dougal Matthews (d0ugal) | 15:45 |
| *** skramaja has quit IRC | 15:46 | |
| jtomasek | rbrady: yes | 15:47 |
| jrist | rbrady: yes | 15:47 |
| * d0ugal grabs popcorn | 15:47 | |
| jtomasek | lol | 15:47 |
| * jrist gets a 'murica size coke | 15:48 | |
| openstackgerrit | Merged openstack/tripleo-quickstart: Add a big HA deployment configuration https://review.openstack.org/359173 | 15:48 |
| * jtomasek eats banana | 15:48 | |
| *** zoliXXL is now known as zoli|gone | 15:48 | |
| jrist | jtomasek: gotta get that potassium | 15:49 |
| EmilienM | sshnaidm: any results so far? | 15:49 |
| rbrady | jrist, jtomasek: I was going to update the bug to set the importance value, but wanted to know if this was truly a blocker. the bug description seems to mention a workaround in paragraph 2 that makes this seem like it's important, but not blocking | 15:49 |
| jtomasek | rbrady: the workaround is quite complicated in GUI and not very performant because on page load, it requires the same number of GET execution requests as the number of validations | 15:50 |
| jtomasek | which could be ~15-20 or even more | 15:51 |
| openstackgerrit | Steven Hardy proposed openstack/puppet-tripleo: Create memcache_servers list in swift proxy profile https://review.openstack.org/364367 | 15:51 |
| rbrady | jtomasek: I get the performance, but how complicated is the code to handle? | 15:51 |
| rbrady | jtomasek: Is the code already in place for the multiple calls? | 15:52 |
| jtomasek | rbrady: it is not | 15:52 |
| *** oshvartz has quit IRC | 15:53 | |
| jtomasek | rbrady: currently in GUI we GET executions and store them. then we filter that selection and figure out the status of each validation. To fix the current bug, we'd have to split the logic of getting the validation status to 2 places and make additional api calls when each specific validation is rendered | 15:54 |
| jtomasek | rbrady: and tbh, there is already so much work on the GUI side that needs to get done by rc1 deadline :/ | 15:56 |
| jrist | yeah | 15:56 |
| shardy | IMO we should be 100% focussed on getting basic deployment working before caring at all about validations | 15:57 |
| shardy | followed by updates/upgrades | 15:57 |
| shardy | not saying validations aren't important, but we need the basic deployment flow working as a first step | 15:57 |
| shardy | then we can build out the additional features | 15:58 |
| jrist | obviously deployment is incredibly important :) since it's the meat of the UI | 15:58 |
| d0ugal | jtomasek, rbrady, jrist - we all have deadline issues :) we should really be talking about this in #openstack-mistral btw. We don't even know if this change will be accepted. | 15:58 |
| d0ugal | AFAIK, no Mistral cores have looked at the bug | 15:58 |
| d0ugal | I think it should be fine, but from my brief investigation it isn't as trivial as I thought it might be | 15:59 |
| *** fzdarsky has joined #tripleo | 16:00 | |
| jrist | it isn't trivial on UI side either | 16:00 |
| jrist | :( | 16:00 |
| d0ugal | jrist: I don't mean to suggest it is | 16:00 |
| pradk | EmilienM, hey can you check this https://review.openstack.org/#/c/363748/ | 16:01 |
| EmilienM | pradk: +2 | 16:02 |
| *** ebarrera_ has quit IRC | 16:02 | |
| *** ebarrera has quit IRC | 16:02 | |
| jrist | d0ugal, rbrady, jtomasek - I propose that you guys present the bug to Mistral team, we wait a little bit | 16:03 |
| jrist | see what Mistral team thinks | 16:03 |
| jrist | if it's not feasible or not allowed (FFE?) | 16:03 |
| jrist | then we will work around it in UI | 16:03 |
| jrist | does this seem reasonable? | 16:03 |
| d0ugal | jrist: Sure, I'll try pinging them. Might be a bit late in the day for most, so if I don't get a reply soon I'll try tomorrow morning | 16:04 |
| jrist | d0ugal: honestly I think that's fine | 16:04 |
| jrist | jtomasek: ^? | 16:04 |
| rbrady | d0ugal: it's midnight now in renat's timezone, I can try pinging him in about 8 hours | 16:05 |
| d0ugal | rbrady: heh, sure | 16:05 |
| jrist | yeah I understand that | 16:05 |
| EmilienM | tripleo newton-3 is out :-) | 16:06 |
| jrist | you did say Mistral team :) is he Mistral team? :) | 16:06 |
| jrist | EmilienM: ! | 16:06 |
| jrist | EmilienM: thanks for all your work on that | 16:06 |
| d0ugal | EmilienM: \o/ | 16:06 |
| d0ugal | jrist: he is 90% OF IT | 16:06 |
| d0ugal | oops .lowercase() | 16:06 |
| shardy | EmilienM: nice, thanks for proposing the release | 16:06 |
| EmilienM | jrist: we had to release tripleo-ui as 1.0.0 | 16:07 |
| rbrady | jrist: he is often consulted for decision | 16:07 |
| EmilienM | because it was the first release | 16:07 |
| shardy | congratulations everyone, another milestone towards our final Newton release :) | 16:07 |
| jrist | EmilienM: yeah that's fine | 16:07 |
| jtomasek | jrist, d0ugal, rbrady +1 | 16:08 |
| jrist | cool | 16:09 |
| *** masco has quit IRC | 16:09 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use a released version of tripleo-common https://review.openstack.org/364425 | 16:10 |
| *** mwhahaha has quit IRC | 16:10 | |
| shardy | Folks, is anyone else sometimes seeing puppet trying to run openstackclient to create keystone projects/roles, but failing because keystone isn't fully configured yet? | 16:17 |
| shardy | https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/keystone.pp#L54 | 16:17 |
| openstackgerrit | Merged openstack/tripleo-common: Disable retry files for ansible validations https://review.openstack.org/329039 | 16:17 |
| shardy | during step3 I'm not sure how we make the keystone configuration happen before creating roles etc? | 16:18 |
| openstackgerrit | Merged openstack/puppet-tripleo: Write restart flags to restart services only when necessary https://review.openstack.org/358510 | 16:18 |
| shardy | in my environment it's failing because keystone isn't yet configured to talk to rabbit | 16:18 |
| *** lucasagomes is now known as lucas-dinner | 16:21 | |
| *** ramishra_ has quit IRC | 16:22 | |
| sshnaidm | EmilienM, no results yet, proxy jobs started right now, and periodic are in queue.. need to wait a little | 16:22 |
| *** ramishra has joined #tripleo | 16:23 | |
| EmilienM | sshnaidm: ok | 16:24 |
| *** coolsvap has joined #tripleo | 16:24 | |
| *** prometheanfire has joined #tripleo | 16:26 | |
| prometheanfire | can someone give more info on the update to tripleo-common in requirements? https://review.openstack.org/364423 | 16:26 |
| *** alee is now known as alee_lunch | 16:28 | |
| *** jpich has quit IRC | 16:29 | |
| *** bana_k has joined #tripleo | 16:29 | |
| *** akuznetsov has joined #tripleo | 16:30 | |
| *** akuznetsov has quit IRC | 16:35 | |
| *** weshay is now known as weshay_lunch | 16:35 | |
| *** ohamada has quit IRC | 16:36 | |
| openstackgerrit | Merged openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/361440 | 16:37 |
| *** yamahata has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack/puppet-tripleo: midonet: update neutron parameter https://review.openstack.org/363178 | 16:42 |
| *** oneswig has quit IRC | 16:44 | |
| *** akshai has quit IRC | 16:45 | |
| *** trown is now known as trown|lunch | 16:45 | |
| *** akshai has joined #tripleo | 16:47 | |
| *** rhallisey has joined #tripleo | 16:50 | |
| shardy | prometheanfire: done | 16:50 |
| prometheanfire | thanks | 16:51 |
| *** pradk has quit IRC | 16:51 | |
| jrist | jtomasek: still around? | 16:52 |
| *** derekh has quit IRC | 16:56 | |
| ansiwen | chem: around re: ec2 credentials | 16:56 |
| chem | ansiwen: hi | 16:58 |
| ansiwen | chem: hey, I'm just going through your advices | 16:58 |
| ansiwen | chem: with zero ruby knowledge it's a bit... well... funny | 16:59 |
| chem | ansiwen: hehe | 16:59 |
| ansiwen | chem: what are the @ ? | 16:59 |
| chem | ansiwen: oula ! | 16:59 |
| chem | ansiwen: instance variable | 17:00 |
| ansiwen | chem: ok, so, members | 17:00 |
| *** prometheanfire has left #tripleo | 17:00 | |
| chem | ansiwen: in python parlance yeap | 17:00 |
| *** yamahata has joined #tripleo | 17:00 | |
| *** tesseract- has quit IRC | 17:01 | |
| ansiwen | chem: I have c++ background... so I would call it member fields, or something like that :-) | 17:01 |
| chem | ansiwen: ha, good to know that not everybody is python or ruby :) | 17:02 |
| ansiwen | chem: ok, so @credentials is declared in the base class, which I assume is Puppet::Provider::Tempest | 17:02 |
| jtomasek | jrist: now | 17:02 |
| jrist | jtomasek: just emailed you | 17:02 |
| chem | ansiwen: you just don't have to worry that much about it, just copy/paste this part it will work. | 17:02 |
| jtomasek | jrist: yep, I've looked at it. I'll test it tomorrow morning | 17:03 |
| jrist | sweet | 17:03 |
| ansiwen | chem: well, the problem is, without understanding the code, I can't understand which parts I have to change. | 17:03 |
| chem | ansiwen: ack, hold on a minute | 17:04 |
| ansiwen | chem: I copied the whole tempest_neutron_net_id_setter directory and called it tempest_ec2_credentials_setter | 17:04 |
| *** abregman has joined #tripleo | 17:04 | |
| *** snecklifter has joined #tripleo | 17:05 | |
| *** athomas has quit IRC | 17:05 | |
| *** abregman|mtg has quit IRC | 17:06 | |
| *** nyechiel has quit IRC | 17:06 | |
| *** bana_k has quit IRC | 17:07 | |
| *** mwhahaha has joined #tripleo | 17:07 | |
| *** weshay_lunch is now known as weshay | 17:08 | |
| ansiwen | chem: since network_id is a instance variable, I have to add @ec2_secret and @ec2_access? | 17:09 |
| ansiwen | chem: the problem is, I don't even know what the functionality of this provider is. It gets the network_id from somewhere and then puts it into the tempest.conf? | 17:10 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Release puppet-tripleo 2.0.0 https://review.openstack.org/364470 | 17:10 |
| chem | ansiwen: you have to use it as a template | 17:11 |
| chem | ansiwen: http://paste.openstack.org/show/565748/ | 17:11 |
| chem | ansiwen: I've added some comment to get you started | 17:11 |
| chem | ansiwen: the file should be like provider/tempest_ec2_setter/openstack.rb | 17:12 |
| chem | ansiwen: and type/tempest_ec2_setter.rb | 17:12 |
| ansiwen | oh, I have to create type/... as well, good to know | 17:13 |
| ansiwen | chem: thanks for the comments | 17:13 |
| chem | ansiwen: so the get_network_id becomes get_ec2_cred | 17:14 |
| ansiwen | chem: what does "request" mean in line 41? | 17:14 |
| chem | ansiwen: this is where the openstack client with the right permission is trigger | 17:14 |
| chem | ansiwen: all wrapped in a nice lib | 17:14 |
| ansiwen | chem: oh, ok, the the request replaces that cli execution, I see | 17:14 |
| chem | ansiwen: for you it would resp = request('ec2 credential', 'create', ...) | 17:15 |
| chem | ansiwen: in resp you will have the parameters back | 17:15 |
| chem | ansiwen: parsed in a nice hash structure | 17:15 |
| ansiwen | chem: but why is the request done with file_path, which is the tempest.conf? how is that related? | 17:15 |
| chem | ansiwen: (it depends on the output actualy, we will see ) | 17:16 |
| chem | ansiwen: it's not done with file_path | 17:16 |
| dsneddon | shardy, Just FYI, I have been testing tripleo master, and although IPv4 deployments are workign 100% so far I have had 5 IPv6 failures in a row. I'm investigating. | 17:16 |
| ansiwen | chem: in line 42 there is file_path | 17:16 |
| dsneddon | shardy, Have the latest composable services networking changes been tested against IPv6 successfully anywhere? | 17:16 |
| EmilienM | dsneddon: did you sync with bnemec ? he's doing similar testing atm | 17:17 |
| *** bana_k has joined #tripleo | 17:17 | |
| chem | ansiwen: .... it may be a bug, silently ignore by the api :) | 17:18 |
| chem | ansiwen: have to verify but it look ... weird :) | 17:18 |
| chem | ansiwen: like I said it's not related ... | 17:18 |
| chem | ansiwen: just do your request it will work | 17:19 |
| ansiwen | chem: ok, I assumed the code makes sense, that's why I ask :-) | 17:19 |
| chem | ansiwen: well ... it should :) | 17:20 |
| bnemec | dsneddon: Yeah, there are some issues with ipv6 right now, but I think we've got patches up to fix them. | 17:20 |
| chem | ansiwen: anyway. Do you have enough to get you started ? | 17:20 |
| dsneddon | bnemec, EmilienM: Great news, thanks. | 17:21 |
| *** mcornea has quit IRC | 17:21 | |
| *** dmacpher is now known as dmacpher-afk | 17:21 | |
| *** pradk has joined #tripleo | 17:22 | |
| bnemec | dsneddon: Oh, I haven't tested HA yet though. And I see we have https://bugs.launchpad.net/tripleo/+bug/1605363 so it may not work. | 17:22 |
| openstack | Launchpad bug 1605363 in tripleo "[Newton] ipv6 HA deployments are currently broken" [Critical,Triaged] | 17:22 |
| bnemec | I'll have to try that today. | 17:22 |
| ansiwen | chem: so, I make two requests, first the "create", ignoring the return value, and then the "list", getting the data? | 17:22 |
| chem | ansiwen: don't know how this part of the openstack api work | 17:23 |
| ansiwen | chem: https://github.com/openstack/ec2-api/blob/master/devstack/create_config#L51-L53 | 17:23 |
| chem | ansiwen: I don't have this kind of time available :) | 17:23 |
| ansiwen | chem: first "openstack ec2 credentials create" and then "openstack ec2 credentials list" to get the result | 17:24 |
| ansiwen | chem: ok, sorry, thank you a lot | 17:24 |
| shardy | dsneddon: Unfortunately that doesn't surprise me because we've not had ipv6 coverage in CI for a few weeks | 17:24 |
| chem | ansiwen: behind the scene it's just the openstack cli which is trigger. | 17:24 |
| shardy | dsneddon: we're working to reinstate it, but any help with bugfixes would be good | 17:24 |
| chem | ansiwen: triggered | 17:24 |
| ansiwen | chem: got it | 17:24 |
| chem | ansiwen: experiment with it on test env and you'll figure it out | 17:24 |
| shardy | dsneddon: the new composable stuff has not been tested with ipv6, and it is possible there | 17:24 |
| shardy | dsneddon: FYI we're trying to move away from mangling formats for ipv6 in t-h-t and instead doing conversions from a simple list of IPs in puppet-triploe | 17:25 |
| chem | ansiwen: when you have code (even non-working) just make a review and add me to the reviewer | 17:25 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo: Release puppet-tripleo 5.0.0 https://review.openstack.org/364470 | 17:25 |
| dsneddon | shardy, Yeah, that's why I'm putting it through its paces with IPv6. I wasn't expecting things to be 100% without some testing and iteration | 17:25 |
| chem | ansiwen: will be easier to help this way | 17:25 |
| ansiwen | chem: sure, ok | 17:25 |
| shardy | dsneddon: e.g https://github.com/openstack/puppet-tripleo/commit/96f22c629de5343a63442d7c3c811d3e31aca00b | 17:25 |
| dsneddon | shardy, Agreed that the less string mangling we are doing with brackets in THT the better. | 17:25 |
| shardy | dsneddon: ack, well thanks for kicking the tyres on it, let me know if I can provide any help/fixes | 17:26 |
| ansiwen | chem: I don't know how to run all this in a test env, but I will find out :-) | 17:26 |
| *** dobson` has quit IRC | 17:26 | |
| chem | ansiwen: just find the right openstack cli sequence, try to adapt as much as you can to the code and then upload it to gerrit. | 17:27 |
| *** alee_lunch is now known as alee | 17:28 | |
| *** ramishra has quit IRC | 17:30 | |
| *** ramishra has joined #tripleo | 17:30 | |
| *** snecklifter is now known as snecklifter|brb | 17:31 | |
| *** akshai has quit IRC | 17:33 | |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates: De-bracket vncproxy_host in compute profile https://review.openstack.org/363204 | 17:33 |
| *** abregman is now known as abregman|mtg | 17:38 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs https://review.openstack.org/364479 | 17:38 |
| bnemec | EmilienM: panda: ^will be needed for ipv6 jobs in CI | 17:38 |
| ansiwen | chem: I have two values, ec2_secret and ec2_access... I will handle both in the same provider, right? | 17:39 |
| EmilienM | bnemec: ack, thanks. panda: you'll need to rebase it on top of bnemec's patch eventually. And Depends-On all the patch in progress to fix IPv6. gfidente and bnemec have some. | 17:40 |
| chem | ansiwen: yes | 17:40 |
| EmilienM | bnemec: thx for your help on this thing | 17:40 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: [WIP] add newton as a release https://review.openstack.org/364480 | 17:42 |
| ayoung | I need to test something out that is supposed to happen when the undercloud launches an overcloud node. Launching the whole overcloud is too much for debugability. I really just want to do openstack server create manually. Ironic node. What should I be using for the params? | 17:42 |
| ayoung | I assume that if the flavor is control then it should launch on one of the baremetal nodes | 17:43 |
| shardy | ayoung: you should probably use the generic baremetal flavor, unless you've tagged your nodes for specific flavors in ironic | 17:44 |
| shardy | and the standard overcloud-full image | 17:44 |
| ayoung | shardy, right | 17:44 |
| shardy | I guess you can start a node just via nova boot, but I've not got the exact command to hand | 17:44 |
| shardy | you could also copy e.g puppet/controller.yaml from triploe-heat-templates, and remove everything except the OS::Nova::Server::Resource, then create a small heat stack with one node in it | 17:45 |
| *** jtomasek is now known as jtomasek|afk | 17:45 | |
| shardy | basically that will just call the same API as nova boot tho | 17:45 |
| ayoung | shardy, I think I'll start wuith openstack server create. I can piece together the options. | 17:46 |
| *** jpena is now known as jpena|away | 17:46 | |
| *** jbadiapa has quit IRC | 17:50 | |
| panda | EmilienM: ack | 17:51 |
| beagles | hey EmilienM could https://bugs.launchpad.net/tripleo/+bug/1619383 be contributing OVB blowing up, etc? | 17:51 |
| openstack | Launchpad bug 1619383 in tripleo "Poor default for API workers in Neutron " [High,Confirmed] - Assigned to Brent Eagles (beagles) | 17:51 |
| panda | EmilienM: bnemec how can I get a list of IPv6 fixes ? | 17:52 |
| beagles | EmilienM, afaict the change in neutron occurred the end of July .. so related issues would've started occurring between now and then | 17:52 |
| *** zoli|gone is now known as zoli_gone-proxy | 17:54 | |
| *** dobson has joined #tripleo | 17:56 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Change NeutronWorkerCount default to result in previous behavior https://review.openstack.org/364483 | 17:57 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: add newton as a release https://review.openstack.org/364480 | 17:58 |
| *** rasca has quit IRC | 17:58 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Restart only services that need it https://review.openstack.org/358511 | 18:00 |
| EmilienM | beagles: looking | 18:01 |
| EmilienM | beagles: I'm curious to see the result :) | 18:02 |
| EmilienM | beagles: have you notices some OOMs or other symptoms to think about it? or is it purely random? | 18:03 |
| beagles | EmilienM, most of the time I'm seeing timeouts. If it were OOM issues, fixing this would probably make it worse. | 18:04 |
| beagles | EmilienM, which... now that I say that, I'd better keep an eye out for that | 18:04 |
| *** nyechiel has joined #tripleo | 18:05 | |
| EmilienM | beagles: cool I'll also monitor it | 18:05 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 18:06 |
| *** trown|lunch is now known as trown | 18:06 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:08 | |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Change NeutronWorkerCount default to result in previous behavior https://review.openstack.org/364483 | 18:10 |
| EmilienM | panda: only https://review.openstack.org/#/c/363204/ afik | 18:14 |
| EmilienM | panda: other patches from gfidente are merged. | 18:15 |
| *** kjw3 has quit IRC | 18:15 | |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** openstackgerrit has joined #tripleo | 18:18 | |
| bnemec | Sorry, yeah, that's the only one I'm aware of too. | 18:21 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure https://review.openstack.org/363918 | 18:23 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Do not merge - test Telemetry services (scenario001) https://review.openstack.org/364390 | 18:23 |
| *** abregman|mtg is now known as abregman | 18:28 | |
| stevebaker | shardy: hey, are you still about? | 18:28 |
| shardy | stevebaker: Hey! Yeah, just about.. | 18:29 |
| shardy | what's up? | 18:30 |
| *** akrivoka has quit IRC | 18:30 | |
| shardy | stevebaker: actually, I was meaning to chat to you about the heatclient event formatter | 18:31 |
| stevebaker | shardy: I wondered if you could give me your thoughts on these bugs I raised https://bugs.launchpad.net/tripleo/+bug/1619072 https://bugs.launchpad.net/tripleo/+bug/1619074 | 18:31 |
| openstack | Launchpad bug 1619072 in tripleo "os-collect-config ec2 collector should be treated as immutable" [Undecided,New] - Assigned to Steve Baker (steve-stevebaker) | 18:31 |
| openstack | Launchpad bug 1619074 in tripleo "os-collect-config ec2 collector should attempt config-drive before nova metadata server" [Undecided,New] - Assigned to Steve Baker (steve-stevebaker) | 18:31 |
| openstackgerrit | Michael Henkel proposed openstack/tripleo-heat-templates: add composable services for Contrail https://review.openstack.org/346799 | 18:31 |
| shardy | stevebaker: ack, definitely - is it cool if I go grab dinner then either comment on the bugs or ping you to discuss further? | 18:31 |
| stevebaker | specifically with release schedule timing | 18:31 |
| stevebaker | yeah, no hurry | 18:32 |
| shardy | stevebaker: Ok, will do | 18:32 |
| stevebaker | ta | 18:32 |
| shardy | stevebaker: re the event formatting, see http://paste.openstack.org/show/565767/ | 18:32 |
| shardy | when you create a ResourceGroup or ResourceChain we get non-useful names out | 18:32 |
| shardy | was wondering if maybe you'd looked into it | 18:32 |
| shardy | I guess we need to add the parent stack name with a verbose flag or something | 18:32 |
| shardy | which tripleoclient can pass by default | 18:33 |
| shardy | so it'll be like overcloud.ControllerSericeChain.53 | 18:33 |
| stevebaker | for the stack failures command I did this dot-delimited list of resource names from the root stack | 18:33 |
| shardy | stevebaker: aha, maybe we can use that | 18:33 |
| stevebaker | its expensive to compute though, maybe [parent_resource.56] would be better | 18:34 |
| stevebaker | i mean would give enough context | 18:34 |
| shardy | it's particularly a problem now we're using ResourceChains, as we spew a bunch more uninformative events as a result | 18:34 |
| shardy | 2016-09-01 17:47:42 [56]: CREATE_COMPLETE state changed | 18:34 |
| shardy | stevebaker: yeah, just parent_resource would possibly be enough | 18:34 |
| shardy | or, at least better | 18:34 |
| stevebaker | yeah, I'll take a look. Is there a bug for it yet> | 18:35 |
| stevebaker | ? | 18:35 |
| shardy | maybe there's some other way | 18:35 |
| shardy | stevebaker: No, I'll raise one now | 18:35 |
| shardy | been meaning to look into it for a few weeks tho | 18:35 |
| stevebaker | tripleoclient will be switching over to the heatclient formatter with this change, so I'll just look at heatclient for the fix https://review.openstack.org/#/c/360141/ | 18:36 |
| *** akshai has joined #tripleo | 18:37 | |
| shardy | https://bugs.launchpad.net/tripleo/+bug/1619415 | 18:37 |
| openstack | Launchpad bug 1619415 in tripleo "tripleoclient events aren't always informative" [High,Triaged] | 18:37 |
| shardy | stevebaker: ^^ | 18:37 |
| shardy | stevebaker: looking at the resources, I don't think the parent resource will be enough unfortunately | 18:37 |
| shardy | stevebaker: or we'll just have about 300 events with ServiceChain.NN | 18:38 |
| stevebaker | this might need a solution on the heat side | 18:38 |
| mhenkel | shardy: I added the proposed changes | 18:38 |
| shardy | mhenkel: thanks! | 18:38 |
| *** mcornea has joined #tripleo | 18:38 | |
| shardy | stevebaker: Ok, well any help would be hugely appreciated if you get a few moments to dig into it | 18:38 |
| mhenkel | regarding testing: I’ll try to figure out something by next week | 18:38 |
| shardy | mhenkel: Ok, sounds good - we can chat again if you'd like some ideas on how we might test it without custom-roles | 18:39 |
| *** shardy is now known as shardy_afk | 18:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/361501 | 18:41 |
| *** maeca1 has quit IRC | 18:42 | |
| beagles | pradk, has anybody pointed these out to you already: Sep 1 18:10:20 localhost gnocchi-metricd: File "/usr/lib64/python2.7/multiprocessing/process.py", line 143, in join | 18:42 |
| beagles | Sep 1 18:10:20 localhost gnocchi-metricd: assert self._parent_pid == os.getpid(), 'can only join a child process' | 18:42 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use a released version of tripleo-common https://review.openstack.org/364425 | 18:42 |
| pradk | beagles, yes | 18:43 |
| beagles | pradk, k.. figured :) | 18:43 |
| pradk | beagles, there is a bug on this we need newer cotyledon | 18:43 |
| *** mhenkel has quit IRC | 18:46 | |
| *** nyechiel has quit IRC | 18:47 | |
| *** fragatin_ has quit IRC | 18:48 | |
| *** david-lyle has quit IRC | 18:49 | |
| *** jprovazn has quit IRC | 18:49 | |
| *** david-lyle has joined #tripleo | 18:50 | |
| *** abregman has quit IRC | 18:56 | |
| *** akshai has quit IRC | 18:57 | |
| dprince | EmilienM: I think a lot of our issues are just simply that we are getting slammed | 19:03 |
| dprince | EmilienM: the load average on the controller in the OVB cloud is really high | 19:03 |
| dprince | load average: 20.00, 18.98, 19.12 | 19:03 |
| openstackgerrit | ayoung proposed openstack/puppet-tripleo: Optionally include mod auth mellon for apache https://review.openstack.org/359942 | 19:03 |
| EmilienM | dprince: yes, that is what we thought earlier when we mentionned the rh1 overload | 19:04 |
| EmilienM | dprince: what can we do ? | 19:05 |
| EmilienM | decrease the number of nodes? | 19:05 |
| dprince | EmilienM: perhaps, nova and neutron are both very high. We need those | 19:06 |
| dprince | EmilienM: we could turn off some things, ceilometer for example | 19:07 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup https://review.openstack.org/358089 | 19:07 |
| *** fragatina has joined #tripleo | 19:10 | |
| bnemec | We already shut off ceilometer, gnocchi, and sahara on rh1. | 19:12 |
| dprince | bnemec: ceilometer was still running | 19:12 |
| EmilienM | yeah shut it down | 19:12 |
| EmilienM | we have scenario001 now | 19:12 |
| EmilienM | scenario001 cover all telemetry | 19:12 |
| bnemec | Oh wait, what are we talking about? | 19:13 |
| EmilienM | ovb-ha? | 19:13 |
| bnemec | The rh1 controller or the test job controller? | 19:13 |
| dprince | bnemec: we are talking about *the actual* controller in our OVB cloud | 19:13 |
| EmilienM | oh good question :P | 19:13 |
| dprince | we have 1 | 19:13 |
| EmilienM | dprince: remove ceilo | 19:13 |
| bnemec | dprince: EmilienM wasn't. :-) | 19:13 |
| dprince | ack, I gathered that | 19:13 |
| bnemec | But yeah, go ahead and kill ceilometer. | 19:14 |
| dprince | bnemec: it won't help much, neutron-server is the top offender | 19:14 |
| dprince | bnemec: I'm wondering if nodepool is more aggressive or something | 19:14 |
| EmilienM | dprince: how many workers? | 19:14 |
| bnemec | Yeah, I'm half hoping that our CPU load will go down once we put the testenvs in a different tenant from the Jenkins slaves. | 19:14 |
| dprince | api_workers=0 | 19:15 |
| dprince | EmilienM: so it is dynamic base on CPU | 19:15 |
| EmilienM | ok | 19:15 |
| bnemec | As I understand it, nodepool is looking at all of our resources, including the bmcs and baremetal vms. | 19:15 |
| bnemec | Which have a lot of Neutron ports assigned to them. | 19:15 |
| dprince | Queries per second avg: 320.219 | 19:26 |
| dprince | bnemec: ^^ seems a bit high for us | 19:27 |
| dprince | bnemec: for reference an idle overcloud seems to have < 20 | 19:27 |
| dprince | bnemec: we could just be slow also due to the memory issue you've highlighted. Have you checked if it affects all the nodes? | 19:29 |
| EmilienM | dprince: I have to be afk for a while, I'm back a bit later | 19:29 |
| dprince | EmilienM: ack, thanks | 19:30 |
| beagles | EmilienM, if a service took a long time to restart on a node, it would slow down any puppet apply's that modified a configuration that it had a relationship with, wouldn't it? | 19:30 |
| EmilienM | beagles: right | 19:30 |
| EmilienM | puppet has a timeout to restart a service | 19:30 |
| EmilienM | 120s iirc | 19:30 |
| EmilienM | but I'm not sure | 19:30 |
| EmilienM | anyway if it takes long, puppe timeout and catalog fails | 19:30 |
| * EmilienM brb in a few hours | 19:30 | |
| beagles | dprince: it is not | 19:31 |
| beagles | api_workers = 0 means 1 process now | 19:31 |
| beagles | see https://review.openstack.org/#/c/364483/ | 19:32 |
| beagles | and related bug | 19:32 |
| beagles | dprince, noticed it today | 19:32 |
| dprince | beagles: okay, but our overcloud is much older than latest Newton. We seem to have multiple neutron-server's running with this... | 19:33 |
| *** jlinkes has quit IRC | 19:34 | |
| bnemec | dprince: It definitely affects the controller. I just sent another email maybe an hour or two ago with details on what I've found recently. | 19:34 |
| beagles | dprince, might be, but there are two worker counts, rpc and api, the rpc one was untouched so it would still be processor count | 19:34 |
| bnemec | My current theory is that most of the performance hit we're taking in rh1 is due to the profile set on the nodes themselves. | 19:34 |
| beagles | dprince, the change was merged to neutron in late july so if we are running older neutron, it would be an issue, but if more recent.... | 19:34 |
| bnemec | It's RDO Mitaka and it's definitely got more than one neutron-server running. | 19:35 |
| dprince | bnemec: profile, meaning in the BIOS? | 19:35 |
| beagles | bnemec, ah okay... that change was *not* backported fwict, so we should be good | 19:36 |
| dprince | bnemec: rabbitmq's beam.smp is also a top offender | 19:36 |
| bnemec | dprince: Yeah, it was forcing CPU clock scaling in the BIOS so the OS had no control. | 19:36 |
| bnemec | When I gave the OS control, it knocked five minutes off the undercloud install. | 19:36 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Change NeutronWorkers default to result in previous behavior https://review.openstack.org/364483 | 19:37 |
| bnemec | Of course, now the cpus are running at a higher clock speed in general, but that's about a 20% performance increase, so I think it's worth it. | 19:38 |
| dprince | bnemec: that setting could make things run hotter too | 19:38 |
| *** mcornea has quit IRC | 19:38 | |
| dprince | bnemec: so we might alert DCops or whoever monitors temps and the like | 19:39 |
| dprince | bnemec: but sure I'd agree with doing it | 19:39 |
| *** qasims has quit IRC | 19:39 | |
| dprince | bnemec: does that change require a restart? | 19:39 |
| dprince | bnemec: probably does... | 19:39 |
| bnemec | dprince: Unfortunately. | 19:39 |
| dprince | bnemec: yeah, it would be absolute evil if it didn't | 19:40 |
| bnemec | But we can do a few at a time without affecting our CI capacity. | 19:40 |
| dprince | bnemec: right, but it is the controller I'm concerned with here. There is just the 1 | 19:40 |
| bnemec | There are a few we can't take down because they're hosting the mirror, proxy, and te-broker too. | 19:40 |
| *** jlinkes has joined #tripleo | 19:40 | |
| bnemec | dprince: I don't think the controller is the source of our problems though. The CPU isn't pegged there, even though it is high. | 19:41 |
| dprince | bnemec: I think we aren't getting ports assigned sometimes because the controller is just slow | 19:41 |
| dprince | bnemec: we don't need cloudwatch do we/ | 19:42 |
| dprince | bnemec: it is running on the controller too so I could kill it as well | 19:42 |
| bnemec | dprince: Okay, that's possible. I didn't realize we were seeing a lot of that. | 19:42 |
| bnemec | dprince: I don't think so. | 19:42 |
| dprince | bnemec: well, I saw more today. That is why I'm looking | 19:42 |
| dprince | bnemec: stopping cloudwatch | 19:42 |
| dprince | bnemec: FWIW, I've stopped ceilometer-* and now heat-cloudwatch | 19:43 |
| *** snecklifter|brb is now known as snecklifter | 19:43 | |
| bnemec | Sounds good. | 19:43 |
| dprince | bnemec: we don't use cinder either do we? | 19:43 |
| bnemec | dprince: Nope | 19:44 |
| openstackgerrit | Merged openstack/instack-undercloud: Stop using deprecated ironic::drivers::deploy https://review.openstack.org/361291 | 19:45 |
| dprince | bnemec: nodepool does though | 19:45 |
| dprince | bnemec: I best leave that one | 19:46 |
| bnemec | I don't think it's actually using much CPU anyway. | 19:47 |
| dprince | bnemec: we may want to scheduler a reboot to apply your BIOS fix to the controller too then | 19:51 |
| beagles | pradk, on what interface is the redis virtual ip supposed to be configured? | 19:51 |
| dprince | bnemec: step at a time though | 19:51 |
| bnemec | dprince: Yeah, I definitely think we do. I think it wouldn't hurt to fix the memory setup in all of the nodes either, even though it didn't entirely solve the problem by itself. | 19:52 |
| bnemec | It's definitely going to hurt performance. | 19:52 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Mv pacemaker and firewall out of controller https://review.openstack.org/361398 | 19:52 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Move snmp settings into composable services https://review.openstack.org/360062 | 19:52 |
| pradk | beagles, 6379 i believe | 19:52 |
| beagles | pradk, interface, not port ;) | 19:52 |
| beagles | pradk, I've got ports on the brain myself :) | 19:53 |
| pradk | oh hehe | 19:53 |
| pradk | beagles, hmm not sure what interface, lemme check | 19:53 |
| *** lblanchard has quit IRC | 19:53 | |
| *** Goneri has quit IRC | 19:54 | |
| openstackgerrit | Merged openstack/puppet-tripleo: Release puppet-tripleo 5.0.0 https://review.openstack.org/364470 | 19:55 |
| *** jcoufal_ has quit IRC | 19:56 | |
| sshnaidm|afk | EmilienM, bnemec, dprince, the repo was promoted. So the trick with changing repository to CDN worked and I'd suggest to make it constant. The jobs took 2h 30min, it's the best result we had I think, better than with proxy. | 19:57 |
| *** abregman has joined #tripleo | 19:57 | |
| sshnaidm|afk | s/was promoted/will be soon/ - jobs passed, waiting for the script | 19:58 |
| dprince | sshnaidm|afk: ack, thanks for the update and work on this | 20:01 |
| *** kjw3 has joined #tripleo | 20:01 | |
| dprince | sshnaidm|afk: if we need to keep using the CDN for now to speed it up I would support it | 20:01 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use centos CDN repository with periodic jobs https://review.openstack.org/364534 | 20:02 |
| sshnaidm|afk | dprince, it's the same repo, just much faster | 20:02 |
| sshnaidm|afk | dprince, I don't know why didn't we use it before | 20:02 |
| dprince | sshnaidm|afk: we didn't know :) | 20:03 |
| *** florianf has quit IRC | 20:03 | |
| sshnaidm|afk | dprince, thanks to jpena|away that revealed it for us :) | 20:04 |
| pradk | beagles, i'm not sure i dont see any interface specific config, may be Emilien knows.. public maybe? | 20:04 |
| pradk | beagles, why do you ask? anything broken? | 20:04 |
| bnemec | I could swear we had already switched to using the buildlogs address. | 20:05 |
| bnemec | Now I'm really confused where I saw that. | 20:05 |
| *** derekh has joined #tripleo | 20:06 | |
| bnemec | Oh, we did it in the docs. | 20:06 |
| openstackgerrit | Russell Bryant proposed openstack/tripleo-heat-templates: OVN heat templates https://review.openstack.org/307734 | 20:06 |
| beagles | pradk, not sure | 20:06 |
| *** maeca1 has joined #tripleo | 20:06 | |
| derekh | sshnaidm|afk: I see you're patch passed, we should have a promote in about 40 minutes | 20:06 |
| derekh | EmilienM: ^ | 20:07 |
| *** coolsvap has quit IRC | 20:12 | |
| *** flepied has quit IRC | 20:12 | |
| sshnaidm|afk | bnemec, for regular repos, not for "consistent" | 20:12 |
| *** dsariel has joined #tripleo | 20:17 | |
| beagles | pradk, oh maybe I don't understand something here.. so the redis server on the controller is binding to RedisNetwork, which I think is set to the internal api. I noticed the gnocchi client is configured to connect to the VIP, but fwict, the VIP isn't actually bound anywhere on my system | 20:18 |
| beagles | pradk, fwiw, if I configure gnocchi-metricd to reference the port redis is actually on, those stack traces go away | 20:19 |
| beagles | pradk, or seem to anyways | 20:19 |
| beagles | pradk, so piece of configuration is either missing or not making it to where it needs to go | 20:20 |
| *** flepied has joined #tripleo | 20:21 | |
| pradk | beagles, so are you referring to this issue: https://bugs.launchpad.net/tripleo/+bug/1618510 | 20:22 |
| openstack | Launchpad bug 1618510 in tripleo "unable to reach redis service" [Critical,Fix committed] | 20:22 |
| *** fzdarsky has quit IRC | 20:23 | |
| pradk | beagles, can you check if redis_password is set in puppet/services/haproxy.yaml | 20:23 |
| pradk | beagles, tripleo::haproxy::redis_password should be set other wise AUTH tcp check will be skipped when haproxy config is written causing the vip to not respond | 20:24 |
| pradk | i donno if you have the same issue.. but we ran into this couple of days ago and Dan's patch fixes it | 20:25 |
| beagles | pradk, redis password is set | 20:25 |
| *** dprince has quit IRC | 20:26 | |
| derekh | sshnaidm|afk: btw, this one also passed without the CDN https://review.openstack.org/#/c/111011/ | 20:27 |
| sshnaidm|afk | derekh, so it may be not related? | 20:28 |
| sshnaidm|afk | derekh, maybe it's good time to run periodics now :) | 20:28 |
| derekh | sshnaidm|afk: there is a lot of variation in job times, the CDN may have helped but I wouldn't say its definitive | 20:29 |
| derekh | sshnaidm|afk: run them now? | 20:29 |
| sshnaidm|afk | derekh, I mean the hours | 20:29 |
| sshnaidm|afk | derekh, maybe in these hours there's no such load? | 20:29 |
| *** maeca1 has quit IRC | 20:30 | |
| derekh | sshnaidm|afk: ah ok, we con't control the time, everything in the periodic queue is triggered at a time set by infra | 20:30 |
| bnemec | It probably depends on how many people are pulling from the rdo servers. | 20:30 |
| sshnaidm|afk | bnemec, the delorean.repo file still contains trunk.rdo.* in baseurl which will point you to old servers, I need to change the hostname in the file itself to builds.* | 20:34 |
| *** jpena|away is now known as jpena|off | 20:35 | |
| sshnaidm|afk | bnemec, so just downloading the same file from buildlogs.. server will not do the difference | 20:35 |
| bnemec | sshnaidm|afk: Really? That seems like a problem with the repo file then. :-/ | 20:35 |
| sshnaidm|afk | bnemec, yeah, that is | 20:35 |
| beagles | pradk, going to try redeploying. I had DVR configured, so now I'm wondering if there was something | 20:38 |
| beagles | interfering | 20:38 |
| *** fultonj has quit IRC | 20:41 | |
| derekh | I'm pretty sure we talked about that being changed months ago | 20:42 |
| derekh | in rdo | 20:42 |
| *** jayg is now known as jayg|g0n3 | 20:44 | |
| pradk | beagles, hmm possible, we did see some issues recently, but afaik they should be resolved so long as you have latest master | 20:46 |
| *** ayoung has quit IRC | 20:46 | |
| beagles | pradk, my tht is pretty up to date, but my images are a day or so old | 20:46 |
| *** derekh has quit IRC | 20:47 | |
| *** kjw3 has quit IRC | 20:49 | |
| beagles | pradk, I'll let you know if there is any difference | 20:50 |
| pradk | ok | 20:51 |
| beagles | pradk, maybe something is getting messed because of the non ha deployment | 20:51 |
| pradk | yea quite possible.. i only have ha in my env | 20:53 |
| pradk | i'll kick off a non ha and confirm | 20:54 |
| openstackgerrit | Merged openstack/python-tripleoclient: Add 'openstack overcloud plan create' command https://review.openstack.org/359784 | 20:56 |
| *** rbrady is now known as rbrady-afk | 20:56 | |
| *** trown is now known as trown|outtypewww | 21:07 | |
| *** kjw3 has joined #tripleo | 21:22 | |
| *** rcernin has quit IRC | 21:24 | |
| *** Goneri has joined #tripleo | 21:26 | |
| *** abregman has quit IRC | 21:31 | |
| *** abregman has joined #tripleo | 21:31 | |
| *** abregman has quit IRC | 21:33 | |
| beagles | pradk: actually.. why is OS::TripleO::Network::Ports::RedisVipPort the control plane vip and the others are all noop by default | 21:37 |
| beagles | pradk, see overcloud-resource-registry-puppet ... | 21:37 |
| beagles | pradk, I'm not saying it shouldn't be, just asking :) | 21:38 |
| *** shardy_afk has quit IRC | 21:38 | |
| *** shardy_afk has joined #tripleo | 21:40 | |
| *** cdearborn has quit IRC | 21:42 | |
| pradk | beagles, donno , but seems to be the same in mitaka too, so i assume that isnt the issue | 21:45 |
| beagles | pradk, yeah | 21:45 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init https://review.openstack.org/364568 | 21:46 |
| *** yamahata has quit IRC | 21:50 | |
| *** morazi has quit IRC | 21:53 | |
| *** Goneri has quit IRC | 21:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements https://review.openstack.org/361875 | 22:00 |
| beagles | pradk, fwiw, I just checked a couple of u/s patches logs and on the same patch, the ha controller did not have those stack traces in /var/log/gnocchi/metricd.log whereas the non-ha one did | 22:01 |
| beagles | pradk, so ... yeah | 22:01 |
| *** yamahata has joined #tripleo | 22:02 | |
| pradk | yea i definitely dint see it in ha locally .. my non ha is coming up lets see | 22:02 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init https://review.openstack.org/364568 | 22:03 |
| *** rlandy is now known as rlandy|bbl | 22:07 | |
| *** myoung is now known as myoung|gone | 22:10 | |
| *** rlandy|bbl is now known as rlandy | 22:11 | |
| *** rlandy is now known as rlandy|bbl | 22:12 | |
| jrist | EmilienM: still here? | 22:15 |
| jrist | EmilienM: I'm trying to understand why this is FAILURE http://logs.openstack.org/98/355598/5/check/gate-tripleo-ci-centos-7-nonha-multinode/8a17297/console.html | 22:15 |
| bnemec | jrist: Looks like it failed to contact a mirror when setting up the host: http://logs.openstack.org/98/355598/5/check/gate-tripleo-ci-centos-7-nonha-multinode/8a17297/logs/devstack-gate-setup-host.txt | 22:18 |
| *** thrash is now known as thrash|g0ne | 22:19 | |
| jrist | bnemec: sorry for my ignorance, but what does that mean re: nonha? | 22:21 |
| jrist | also where in that log is there failure? | 22:21 |
| bnemec | jrist: It means it was an infra problem. recheck away. | 22:21 |
| jrist | bnemec: ah :( | 22:21 |
| jrist | bnemec: that's exactly why I asked, so I don't just recheck for no reason | 22:22 |
| bnemec | jrist: It tries to download that release file 10 times, and presumably never succeeds. | 22:22 |
| jrist | ah | 22:22 |
| jrist | but doesn't actually *say* failure ;) | 22:22 |
| bnemec | True. :-) | 22:23 |
| bnemec | In any case, the failure pretty clearly happened before it even got to our stuff, so it wasn't an issue with the patch. | 22:24 |
| jrist | bnemec: appreciated | 22:24 |
| jrist | thanks for taking the time | 22:24 |
| bnemec | np | 22:24 |
| bnemec | I very much appreciate it when people actually investigate CI failures instead of just blindly rechecking. :-) | 22:24 |
| jrist | as do I | 22:24 |
| *** yolanda has quit IRC | 22:26 | |
| mwhahaha | jrist: i think the failure was in http://logs.openstack.org/98/355598/5/check/gate-tripleo-ci-centos-7-nonha-multinode/8a17297/_zuul_ansible/ansible_log.txt | 22:27 |
| *** ramishra has quit IRC | 22:27 | |
| jrist | ah yes | 22:28 |
| mwhahaha | fatal: [node]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."} | 22:28 |
| jrist | lots of failed | 22:28 |
| jrist | thanks | 22:28 |
| *** ramishra has joined #tripleo | 22:29 | |
| *** ayoung has joined #tripleo | 22:30 | |
| *** alee has quit IRC | 22:35 | |
| *** shardy_afk has quit IRC | 22:40 | |
| *** jlinkes_ has joined #tripleo | 23:00 | |
| *** jlinkes has quit IRC | 23:00 | |
| *** ayoung has quit IRC | 23:05 | |
| *** kjw3 has quit IRC | 23:11 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Remove support for legacy rh1 https://review.openstack.org/347918 | 23:19 |
| *** saneax-_-|AFK is now known as saneax | 23:26 | |
| *** chlong has quit IRC | 23:33 | |
| pabelanger | 101787 | 23:47 |
| *** jlinkes_ has quit IRC | 23:53 | |
| *** jlinkes has joined #tripleo | 23:59 | |
| *** maeca1 has joined #tripleo | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!