| *** lblanchard1 has joined #tripleo | 00:06 | |
| *** chlong has quit IRC | 00:11 | |
| *** dmacpher-afk has quit IRC | 00:16 | |
| *** jcoufal has joined #tripleo | 00:27 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Increment step count to include ringbuilder https://review.openstack.org/236243 | 00:30 |
|---|---|---|
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: composable keystone services https://review.openstack.org/295588 | 00:30 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Configure ControllerServices via resource chains https://review.openstack.org/259568 | 00:30 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: composable glance services https://review.openstack.org/237370 | 00:30 |
| *** akshai_ has quit IRC | 00:56 | |
| *** dmacpher has joined #tripleo | 01:05 | |
| *** yamahata has quit IRC | 01:11 | |
| *** myoung has joined #tripleo | 01:15 | |
| *** penick has quit IRC | 01:30 | |
| *** lblanchard1 has quit IRC | 01:33 | |
| *** chlong has joined #tripleo | 01:36 | |
| *** akshai has joined #tripleo | 01:41 | |
| *** ebalduf has quit IRC | 01:52 | |
| *** akshai_ has joined #tripleo | 01:57 | |
| *** yamahata has joined #tripleo | 01:57 | |
| *** akshai has quit IRC | 02:00 | |
| *** akshai_ has quit IRC | 02:04 | |
| *** panda has quit IRC | 02:05 | |
| *** ebalduf has joined #tripleo | 02:05 | |
| *** panda has joined #tripleo | 02:06 | |
| *** akshai has joined #tripleo | 02:07 | |
| *** myoung has quit IRC | 02:22 | |
| *** rlandy has quit IRC | 02:44 | |
| prometheanfire | any dib person here able to give the other +2? https://review.openstack.org/#/c/294973/ | 02:46 |
| *** jcoufal has quit IRC | 02:47 | |
| *** rwsu has quit IRC | 02:51 | |
| *** rasca has quit IRC | 03:15 | |
| *** rasca has joined #tripleo | 03:18 | |
| *** mannidi_ has joined #tripleo | 03:24 | |
| *** mannidi__ has joined #tripleo | 03:26 | |
| *** links has joined #tripleo | 03:27 | |
| *** mannidi_ has quit IRC | 03:30 | |
| *** akshai_ has joined #tripleo | 03:40 | |
| *** akshai has quit IRC | 03:42 | |
| *** bswartz has quit IRC | 03:55 | |
| *** bswartz has joined #tripleo | 03:55 | |
| *** Marga_ has quit IRC | 03:58 | |
| *** Marga_ has joined #tripleo | 03:59 | |
| *** Marga_ has quit IRC | 03:59 | |
| *** Marga_ has joined #tripleo | 03:59 | |
| *** absubram has joined #tripleo | 04:07 | |
| *** masco has joined #tripleo | 04:07 | |
| *** absubram_ has joined #tripleo | 04:08 | |
| *** akshai has joined #tripleo | 04:11 | |
| *** absubram has quit IRC | 04:12 | |
| *** absubram_ is now known as absubram | 04:12 | |
| *** akshai_ has quit IRC | 04:14 | |
| *** penick has joined #tripleo | 04:33 | |
| *** absubram has quit IRC | 04:36 | |
| openstackgerrit | Michael Chapman proposed openstack/tripleo-heat-templates: Use keystone and db sync profiles https://review.openstack.org/292891 | 04:49 |
| *** chlong has quit IRC | 04:56 | |
| *** saneax is now known as saneax_AFK | 05:04 | |
| *** saneax_AFK is now known as saneax | 05:06 | |
| *** rbrady has quit IRC | 05:07 | |
| openstackgerrit | Merged openstack/diskimage-builder: add pkg-map to pip-and-virtualenv element https://review.openstack.org/294973 | 05:12 |
| *** chlong has joined #tripleo | 05:12 | |
| *** penick_ has joined #tripleo | 05:16 | |
| *** penick has quit IRC | 05:17 | |
| *** penick_ is now known as penick | 05:17 | |
| *** rbrady has joined #tripleo | 05:22 | |
| *** akshai has quit IRC | 05:22 | |
| *** penick has quit IRC | 05:35 | |
| *** pcaruana has quit IRC | 05:48 | |
| *** panda has quit IRC | 06:05 | |
| *** liverpooler has quit IRC | 06:05 | |
| *** panda has joined #tripleo | 06:06 | |
| *** TSCHAK has joined #tripleo | 06:07 | |
| *** chlong has quit IRC | 06:08 | |
| *** jaosorior has joined #tripleo | 06:15 | |
| *** jaosorior has quit IRC | 06:17 | |
| *** jprovazn has joined #tripleo | 06:17 | |
| *** jaosorior has joined #tripleo | 06:18 | |
| *** chlong has joined #tripleo | 06:21 | |
| *** rcernin has joined #tripleo | 06:23 | |
| *** florianf has joined #tripleo | 06:31 | |
| *** florianf has quit IRC | 06:32 | |
| *** florianf has joined #tripleo | 06:33 | |
| *** liverpooler has joined #tripleo | 06:56 | |
| *** lifeless has quit IRC | 07:06 | |
| *** dmacpher has quit IRC | 07:08 | |
| *** mikelk has joined #tripleo | 07:11 | |
| *** oshvartz has joined #tripleo | 07:12 | |
| *** rwsu has joined #tripleo | 07:15 | |
| *** yamahata has quit IRC | 07:21 | |
| *** rwsu has quit IRC | 07:21 | |
| *** rwsu has joined #tripleo | 07:22 | |
| *** lmiccini_ is now known as lmiccini | 07:26 | |
| *** Marga_ has quit IRC | 07:27 | |
| *** pcaruana has joined #tripleo | 07:32 | |
| *** qasims has joined #tripleo | 07:33 | |
| qasims | Folks, http://logs.openstack.org/16/257916/8/check-tripleo/gate-tripleo-ci-f22-ha/a4bb94b/ | 07:35 |
| qasims | Logs for failing CI job? Can any help me with this? | 07:39 |
| *** Marga_ has joined #tripleo | 07:40 | |
| *** tesseract has joined #tripleo | 07:41 | |
| *** tesseract is now known as Guest27982 | 07:42 | |
| *** Marga_ has quit IRC | 07:45 | |
| *** aufi has joined #tripleo | 07:56 | |
| *** ifarkas has joined #tripleo | 07:59 | |
| *** mcornea has joined #tripleo | 08:02 | |
| *** ebalduf has quit IRC | 08:05 | |
| *** olap has joined #tripleo | 08:08 | |
| *** athomas has joined #tripleo | 08:09 | |
| *** rdopiera has joined #tripleo | 08:12 | |
| *** hewbrocc` has quit IRC | 08:13 | |
| *** florianf has quit IRC | 08:15 | |
| openstackgerrit | PanFengyun proposed openstack/tripleo-common: fix do_interactive_update output resource event. https://review.openstack.org/296935 | 08:16 |
| *** dsneddon has quit IRC | 08:20 | |
| *** akrivoka has joined #tripleo | 08:21 | |
| *** hewbrocca has joined #tripleo | 08:26 | |
| *** mbound has joined #tripleo | 08:26 | |
| *** shardy has joined #tripleo | 08:27 | |
| *** chlong has quit IRC | 08:38 | |
| *** mannidi__ has quit IRC | 08:48 | |
| *** dobson has quit IRC | 08:51 | |
| *** dsneddon has joined #tripleo | 08:56 | |
| *** snecklifter has quit IRC | 09:03 | |
| *** ramishra has quit IRC | 09:05 | |
| *** chem has joined #tripleo | 09:07 | |
| *** ramishra has joined #tripleo | 09:07 | |
| *** lucas-dinner is now known as lucasagomes | 09:11 | |
| marios | new ci issue with cinder-scheduler it seems (seen on a couple reviews today) | 09:19 |
| marios | cinder-scheduler log like http://paste.openstack.org/show/491671/ | 09:19 |
| *** ccamacho has joined #tripleo | 09:20 | |
| *** social has joined #tripleo | 09:20 | |
| jaosorior | marios: Funky... any idea what's up there? | 09:28 |
| marios | jaosorior: no was wondering if it was already known but don't see anything in tripleo bugs yet | 09:29 |
| jaosorior | I guess it wasn't known | 09:29 |
| shardy | I was wondering if we have a handle on the ha job failures - we seem to have a really high failure rate on otherwise good patches the last few days | 09:30 |
| jaosorior | shardy: I guess we don't have a good handle :/ | 09:31 |
| hewbrocca | are they timing out? | 09:31 |
| jaosorior | and this failure is damn strange... I haven't noticed anything puppet-cinder, or something that might configure cinder, or even mysql that has changed lately | 09:31 |
| shardy | Not timing out, I've seen a couple of issues, failure to restart heat-engine, mysql connection broken so we can't db sync, keystone 500 errors trying to create the heat domain | 09:32 |
| hewbrocca | hmmph | 09:33 |
| shardy | Not had time to sift through them yet, just wondered if anyone else had been | 09:33 |
| *** jcoufal has joined #tripleo | 09:33 | |
| shardy | the other two jobs seem to be working pretty well tho | 09:33 |
| hewbrocca | Undercloud running out of memory? | 09:33 |
| shardy | Don't think so, it's overcloud failures I'm seeing | 09:33 |
| shardy | could be controller memory again I guess, need to trawl some logs | 09:34 |
| hewbrocca | that would be strange given how much we increased it, but OTOH it's the most obvious suspect | 09:34 |
| jaosorior | marios: mariadb seems to shut down for some reason | 09:35 |
| jaosorior | or so say the mariadb logs | 09:36 |
| marios | jaosorior: hmm... well that could be the root.. i mean from the cinder log it says 2016-03-23 22:56:31.978 21085 ERROR cinder.service "Lost connection to MySQL server during query (%s)" % (e,)) | 09:37 |
| marios | jaosorior: so might not be a problem with cinder-scheduler per say | 09:38 |
| jaosorior | I don't think it is | 09:38 |
| marios | 2016-03-23 22:56:31.978 21085 ERROR cinder.service DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query ([Errno 110] Connection timed out)') | 09:38 |
| jaosorior | There's also a bunch of these InternalError: (1205, u'Lock wait timeout exceeded; try restarting transaction') | 09:40 |
| jaosorior | and this | 09:40 |
| jaosorior | DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)') [SQL: u'UPDATE services SET updated_at=%s, report_count=%s WHERE services.id = %s'] [parameters: (datetime.datetime(2016, 3, 24, 8, 7, 54, 616328), 405, 3)] | 09:40 |
| jaosorior | And there is this funky thing in the mysql logs, but I'm not sure if it's related | 09:41 |
| jaosorior | Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (74c99623-f191-11e5-9785-278e2b2903a7): 1 (Operation not permitted) | 09:41 |
| jaosorior | at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable. | 09:41 |
| hewbrocca | wut | 09:42 |
| jaosorior | hewbrocca: wut indeed | 09:43 |
| hewbrocca | Any chance these failures are happening on the same set of testenvs | 09:43 |
| hewbrocca | I'm wondering if there is actually bad network config on some of the virtual clusters | 09:43 |
| jaosorior | all the stuff that I put is from the same run | 09:44 |
| *** florianf has joined #tripleo | 09:45 | |
| shardy | Aha that looks similar to the DB related errors I saw yesterday | 09:45 |
| *** athomas has quit IRC | 09:45 | |
| shardy | only in that case it blew up during a db sync | 09:45 |
| *** florianf1 has joined #tripleo | 09:46 | |
| jaosorior | shardy: Might be the same thing, but here it's just blowing up later in the deployment | 09:46 |
| shardy | yup | 09:46 |
| hewbrocca | Flaky overcloud database would definitely cause a heap of trouble | 09:47 |
| shardy | Here's another different one: | 09:48 |
| shardy | http://logs.openstack.org/16/257916/8/check-tripleo/gate-tripleo-ci-f22-ha/a4bb94b/console.html | 09:48 |
| shardy | pingtest failed with a neutron connection aborted error | 09:49 |
| marios | shardy: actually i think is the same there | 09:49 |
| marios | shardy:that is the first on e i saw today (plumgrid right?) | 09:49 |
| shardy | Ok, cool - is that the one you're already looking at? | 09:49 |
| marios | shardy: i mean, it is where i first saw the cinder/scheduler.log | 09:49 |
| marios | (like http://paste.openstack.org/show/491671/) | 09:49 |
| shardy | I've pulled a bunch of logs trying to figure out a common failure | 09:49 |
| *** athomas has joined #tripleo | 09:50 | |
| shardy | marios: kk, I won't trawl those logs again then :) | 09:50 |
| *** florianf has quit IRC | 09:50 | |
| marios | shardy: though, i went straight for the controller logs, and hadn't noticed the pingtest issue (but since controller services are having issues we can't expect the pingtest to run anyway) | 09:51 |
| shardy | Yeah, that's one failure of the console.log - it doesn't do a great job of surfacing the actual error in a lot of cases | 09:52 |
| shardy | What would be cool is some tool that could scrape all the logs and correlate errors in different CI runs | 09:52 |
| marios | shardy: so wtf.. i mean we have this error on controller, but the stack went create complete???! http://logs.openstack.org/16/257916/8/check-tripleo/gate-tripleo-ci-f22-ha/a4bb94b/console.html#_2016-03-23_22_55_47_380 | 09:52 |
| shardy | I guess like elastic-recheck, but it'd be good to generate queries based on the data instead of manually | 09:53 |
| *** dobson has joined #tripleo | 09:53 | |
| shardy | marios: the stack will complete if puppet configures the services and they get started | 09:54 |
| shardy | so possibly the controller errors happened after that? | 09:54 |
| marios | shardy: i c, i assumed the error (applying any of the puppet for example...) would cause heat to signal back - oh right if the error happened after successful puppet run... | 09:55 |
| shardy | This is where we really need the sensu stuff monitoring services after deploy | 09:55 |
| jaosorior | shardy: Isn't that what elastic search is supposed to do? scrapping logs and such | 09:58 |
| jaosorior | oh yeah, elastic-recheck; just read you mentioned it | 09:58 |
| *** ohamada has joined #tripleo | 09:59 | |
| *** nico_auv has joined #tripleo | 10:11 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/python-tripleoclient: Allow 'openstack baremetal configure boot' to guess the root device https://review.openstack.org/288417 | 10:11 |
| *** mkovacik__ has joined #tripleo | 10:11 | |
| *** florianf1 is now known as florianf | 10:14 | |
| *** dobson has quit IRC | 10:20 | |
| *** electrofelix has joined #tripleo | 10:22 | |
| *** saneax is now known as saneax_AFK | 10:30 | |
| *** dobson has joined #tripleo | 10:31 | |
| *** mgould has joined #tripleo | 10:33 | |
| *** jaosorior has quit IRC | 10:34 | |
| *** ohamada has quit IRC | 10:42 | |
| *** jaosorior has joined #tripleo | 10:43 | |
| *** saneax_AFK is now known as saneax | 10:45 | |
| *** tosky has joined #tripleo | 10:46 | |
| jaosorior | shardy, marios: Currently the compute post config (the puppet stuff) will run in parallel as the controllers, right? | 10:49 |
| jaosorior | is there a way to make the compute configuration depend on a certain step of the controller configuration? | 10:51 |
| marios | jaosorior: yeah i think that's right. well you'd have to rework the ComputeNodesPostDeployment to depend on the COntrollerNodesPostDeployment first | 10:52 |
| jaosorior | marios, shardy: would you guys be against that? | 10:53 |
| marios | jaosorior: i seem to recall we *had* it that way and changed it but not 100% on that | 10:53 |
| jaosorior | currently I'm working on stopping the usage of the default vhost for rabbitmq, and it will end up as having a vhost for notifications and another one for RPCs (each with it's own user and it's own permissions); And currently the fact that the postconfigs are run in parallel is complicating it :/ | 10:54 |
| slagle | easy review if anyone can look, https://review.openstack.org/#/c/296451/ | 11:02 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove the glance/rbd user name from static hieradata https://review.openstack.org/296433 | 11:02 |
| *** ohamada has joined #tripleo | 11:02 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Remove the glance/rbd user name from static hieradata https://review.openstack.org/297024 | 11:05 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Comment out the ManagementPort in external-loadbalancer-vip.yaml https://review.openstack.org/296138 | 11:06 |
| *** lifeless has joined #tripleo | 11:07 | |
| openstackgerrit | Merged openstack/python-tripleoclient: Fix path to rhel registration environment https://review.openstack.org/296451 | 11:08 |
| *** jcoufal has quit IRC | 11:13 | |
| openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Fix path to rhel registration environment https://review.openstack.org/297031 | 11:18 |
| *** electrofelix has quit IRC | 11:26 | |
| *** rdopiera has quit IRC | 11:32 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038 | 11:33 |
| *** weshay has joined #tripleo | 11:34 | |
| *** pradk_ is now known as pradk | 11:35 | |
| pradk | shardy, hi | 11:35 |
| pradk | shardy, see my comment in aodh patch, the trace you see is due to redis issue we discussed yesterday.. once redis patches merge this should go away, its not specific to aodh so unrelated to this patch | 11:36 |
| *** rdopiera has joined #tripleo | 11:39 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: configure horizon with memcached ipv6 when needed https://review.openstack.org/296834 | 11:44 |
| *** TSCHAK has quit IRC | 11:45 | |
| *** akrivoka has quit IRC | 11:52 | |
| *** rhallisey has joined #tripleo | 11:52 | |
| *** ohamada has quit IRC | 11:54 | |
| *** akrivoka has joined #tripleo | 11:54 | |
| *** lblanchard has joined #tripleo | 11:55 | |
| *** mannidi__ has joined #tripleo | 11:56 | |
| *** trown|outtypewww is now known as trown | 11:58 | |
| *** morazi has joined #tripleo | 11:59 | |
| *** jcoufal has joined #tripleo | 12:03 | |
| *** electrofelix has joined #tripleo | 12:04 | |
| *** tedezed has joined #tripleo | 12:05 | |
| tedezed | Update bug https://bugs.launchpad.net/tripleo/+bug/1558467 | 12:06 |
| openstack | Launchpad bug 1558467 in tripleo "Error 'ascii' codec can't decode in introspection nodes overcloud" [Undecided,New] | 12:06 |
| EmilienM | o/ | 12:08 |
| EmilienM | adarazs: what is failing now on ipv6 jobs? | 12:08 |
| EmilienM | another question, do you need stable/mitaka passing ipv6 also? I'll need to backport a patch otherwise | 12:09 |
| EmilienM | jaosorior: can you look https://review.openstack.org/#/c/296458/ please ? | 12:10 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: nonha: fix memcached servers string in nova.conf for ipv6 https://review.openstack.org/297066 | 12:11 |
| EmilienM | adarazs: nevermind for stable, I'm backporting it anyway ^ | 12:11 |
| *** tedezed has left #tripleo | 12:13 | |
| *** jayg|g0n3 is now known as jayg | 12:13 | |
| *** mbound has quit IRC | 12:16 | |
| adarazs | EmilienM: I updated the ticket with the error | 12:16 |
| adarazs | dunno yet what causes it. | 12:17 |
| EmilienM | adarazs: where is but report? | 12:17 |
| *** mbound has joined #tripleo | 12:18 | |
| adarazs | EmilienM: I'll make one, I was just looking at it. | 12:18 |
| EmilienM | adarazs: which ticket did you update I mean | 12:19 |
| adarazs | EmilienM: https://trello.com/c/wiJrOgZN/40-ipv6-add-upstream-ipv6-gate-jobs | 12:19 |
| adarazs | EmilienM: it's this bug: https://bugs.launchpad.net/tripleo/+bug/1560503 | 12:23 |
| openstack | Launchpad bug 1560503 in tripleo "MongoDB Connection refused on overcloud/controllers" [Undecided,New] - Assigned to Emilien Macchi (emilienm) | 12:23 |
| EmilienM | this is not a bug | 12:23 |
| EmilienM | I mean | 12:23 |
| adarazs | EmilienM: not sure why this cropped up in all jobs now. | 12:23 |
| EmilienM | this is not a blocker | 12:23 |
| adarazs | EmilienM: okay, what's the workaround? | 12:23 |
| EmilienM | adarazs: please show me logs link? | 12:23 |
| EmilienM | there is no workaround yet, I haven't worked on a fix yet | 12:24 |
| adarazs | EmilienM: http://logs.openstack.org/45/289445/25/check-tripleo/gate-tripleo-ci-f22-ha/5753bfc/console.html | 12:24 |
| adarazs | EmilienM: it killed all three of the IPv6 jobs and not the IPv4 one. | 12:24 |
| adarazs | and it happened on all three rechecks | 12:24 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Pass iSCSI bind address in brackets to Cinder when IPv6 https://review.openstack.org/296458 | 12:25 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Pass iSCSI bind address in brackets to Cinder when IPv6 https://review.openstack.org/297078 | 12:26 |
| *** lblanchard has quit IRC | 12:26 | |
| EmilienM | adarazs: that ? http://logs.openstack.org/45/289445/25/check-tripleo/gate-tripleo-ci-f22-ha/5753bfc/console.html#_2016-03-24_10_14_25_208 | 12:28 |
| adarazs | EmilienM: yes. | 12:28 |
| EmilienM | adarazs: lot of issues in there, not only mongo | 12:29 |
| EmilienM | adarazs: I'm looking now. | 12:29 |
| *** masco has quit IRC | 12:30 | |
| adarazs | EmilienM: it seems like there's a part of the mongodb puppet module that checks if it's master or not and that's failing to connect to the localhost. | 12:30 |
| EmilienM | right, I'm checking if it's a false negative | 12:30 |
| EmilienM | it sounds a real problem, I was wrong in the bug report. | 12:31 |
| *** pkovar has joined #tripleo | 12:35 | |
| *** bvandenh has joined #tripleo | 12:38 | |
| *** jdob has joined #tripleo | 12:39 | |
| *** electrofelix has quit IRC | 12:39 | |
| *** akrivoka has quit IRC | 12:45 | |
| *** akrivoka has joined #tripleo | 12:45 | |
| *** bvandenh has quit IRC | 12:52 | |
| *** akshai has joined #tripleo | 12:55 | |
| EmilienM | adarazs: I think I found out ;-) | 12:56 |
| EmilienM | adarazs: I sent a fix in January and never reviewed | 12:57 |
| EmilienM | https://review.openstack.org/#/c/269871/ | 12:57 |
| EmilienM | I'm rebasing it and run CI again | 12:57 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: puppet: specify server lists for mongodb_conn_validator provider https://review.openstack.org/269871 | 12:57 |
| EmilienM | adarazs: same error on non-ha? | 12:57 |
| *** mannidi__ has quit IRC | 12:58 | |
| *** mbound has quit IRC | 12:58 | |
| *** mbound has joined #tripleo | 12:58 | |
| adarazs | EmilienM: woot. yes, all of them had the same error. | 12:58 |
| EmilienM | ok that's bad | 12:58 |
| EmilienM | it might be something else | 12:58 |
| EmilienM | adarazs: well, I need to update the patch rebase is not enough | 13:00 |
| *** Goneri has joined #tripleo | 13:00 | |
| *** MaxPC has joined #tripleo | 13:00 | |
| adarazs | EmilienM: all right, let me know, I'll depend on it from the CI change and try if it helps | 13:01 |
| EmilienM | adarazs: ok, I'll update your CI patch once I'm ready | 13:02 |
| adarazs | EmilienM: you can delete the other depends-on, it was merged. | 13:03 |
| openstackgerrit | Dimitri Savineau proposed openstack/tripleo-heat-templates: Allow neutron_options customization for dashboard https://review.openstack.org/260226 | 13:04 |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common: Add Heat Capabilities Actions https://review.openstack.org/284460 | 13:04 |
| *** rodrigods has quit IRC | 13:07 | |
| *** rodrigods has joined #tripleo | 13:08 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: puppet: specify server lists for mongodb_conn_validator provider https://review.openstack.org/269871 | 13:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 13:12 |
| *** gfidente has joined #tripleo | 13:13 | |
| EmilienM | adarazs: ok the mongodb is not a problem in fact, I see it in other Ci jobs and it does not make it fail. Though the hiera error we see is the problem | 13:16 |
| EmilienM | http://logs.openstack.org/45/289445/25/check-tripleo/gate-tripleo-ci-f22-nonha/e55d26d/console.html#_2016-03-24_00_30_01_480 | 13:16 |
| EmilienM | found unexpected ':' | 13:16 |
| adarazs | EmilienM: yeah, I didn't know if that's a cause or effect. :) | 13:16 |
| EmilienM | it might be a problem in the template | 13:18 |
| adarazs | EmilienM: also keep in mind that we had the same error in the nonha job, which doesn't use pacemaker. so if you fix something in overcloud_controller_pacemaker.pp should be also fixed in overcloud_controller.pp | 13:19 |
| openstackgerrit | Miles Gould proposed openstack/python-tripleoclient: Use Ironic API v1.11 to support ENROLL state https://review.openstack.org/272206 | 13:20 |
| slagle | EmilienM: can you review https://review.openstack.org/#/c/295937/? | 13:22 |
| EmilienM | sure | 13:22 |
| EmilienM | slagle: should we set a passwd by default? | 13:23 |
| gfidente | bandini hello | 13:24 |
| gfidente | hey why the w-1 on https://review.openstack.org/#/c/285207/1 ? | 13:24 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: composable glance services https://review.openstack.org/237370 | 13:24 |
| EmilienM | slagle: I guess that's fine for now, +A | 13:24 |
| trown | slagle, shardy, we are starting to create all of the mitaka packages in RDO, how close is tripleo to branching? should we create packages from the git tags for current-passed-ci (manual), or wait? | 13:26 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use the mirror server for the centos image and git repos https://review.openstack.org/285257 | 13:26 |
| trown | by current-passed-ci I mean: https://trunk.rdoproject.org/centos7-mitaka/current-passed-ci/versions.csv | 13:27 |
| bandini | gfidente: heya, I guess I can remove the w-1 now | 13:27 |
| *** dmacpher has joined #tripleo | 13:28 | |
| *** links has quit IRC | 13:28 | |
| *** absubram has joined #tripleo | 13:33 | |
| *** absubram_ has joined #tripleo | 13:34 | |
| *** lblanchard has joined #tripleo | 13:37 | |
| *** absubram has quit IRC | 13:38 | |
| *** absubram_ is now known as absubram | 13:38 | |
| shardy | trown: Hi, we're pretty close but need to land the aodh patches before branching | 13:39 |
| shardy | unfortunately been suffering from a bunch of false negatives in CI | 13:39 |
| trown | ya, the ha job seems particularly bad | 13:39 |
| shardy | yeah, we may have to land the aodh stuff and just branch | 13:40 |
| shardy | slagle: what are your thoughts? | 13:40 |
| openstackgerrit | Giulio Fidente proposed openstack/puppet-tripleo: Allow the Redis specific monitor to use authentication https://review.openstack.org/297115 | 13:41 |
| gfidente | shardy I think the aodh issues are fixed now with the redis auth fixes for haproxy | 13:42 |
| slagle | gfidente: won't merge now since you proposed that to liberty ^ | 13:42 |
| slagle | https://review.openstack.org/#/c/295958/ | 13:42 |
| gfidente | oh the depeds-on | 13:43 |
| slagle | due to the depends on | 13:43 |
| gfidente | dah | 13:43 |
| gfidente | :( | 13:43 |
| openstackgerrit | Merged openstack/puppet-tripleo: Allow the Redis specific monitor to use authentication https://review.openstack.org/295937 | 13:44 |
| shardy | So, is it just https://review.openstack.org/#/c/289435/ we need to land, or is there anything else we're blocking on? | 13:44 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Wire redis_password to the tripleo module https://review.openstack.org/295958 | 13:44 |
| slagle | shardy: i'm unchanged since the meeting really | 13:44 |
| slagle | shardy: we can branch now, but if we do, we must allow aodh to be backported | 13:44 |
| slagle | or we wait for aodh to land | 13:44 |
| slagle | or we land it now with failing ci | 13:44 |
| shardy | slagle: I'm wondering if we just approve the aodh patch - it's been failing the HA job, but then nearly all patches have the last couple of days :( | 13:44 |
| gfidente | shardy https://review.openstack.org/#/c/289435/ won't pass ci until https://review.openstack.org/#/c/295958/ is merged | 13:45 |
| gfidente | and https://review.openstack.org/#/c/295958/ won't get merged because the depends-on now points to two submissions (one from master, which was merged and the other I just pushed for liberty which isn't merged yet) | 13:45 |
| EmilienM | has anyone already seen this error? Error: Error from DataBinding 'hiera' while looking up 'tripleo::packages::enable_install': (<unknown>): found unexpected ':' while scanning a plain scalar at line 303 column 27 on node overcloud-controller | 13:46 |
| shardy | gfidente: Ok, can you temporarily abandon the stable/liberty patch please? | 13:46 |
| shardy | then we can land the master patches and you can restore it | 13:46 |
| gfidente | okay yes | 13:46 |
| slagle | gfidente: what did you do there? cherry pick the same patch to master? | 13:46 |
| shardy | slagle: ah, good catch I didn't spot that | 13:47 |
| slagle | gfidente: just remove the depends-on, and we can merge it | 13:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Wire redis_password to the tripleo module https://review.openstack.org/295958 | 13:48 |
| slagle | ok, +A'd | 13:48 |
| gfidente | sorry for the mess :( | 13:49 |
| slagle | np, it's not your fault, it's b/c depends-on isn't branch aware :) | 13:49 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Wire redis_password to the tripleo module https://review.openstack.org/297135 | 13:49 |
| gfidente | yeah well I should have known | 13:49 |
| gfidente | anyway | 13:49 |
| EmilienM | slagle: what? | 13:49 |
| EmilienM | slagle: depends-on is branch aware if you use zuul-cloner to clone repos | 13:49 |
| gfidente | zuul-cloner? | 13:50 |
| EmilienM | gfidente: http://docs.openstack.org/infra/zuul/cloner.html | 13:50 |
| shardy | Well evidently we aren't using that in CI, because patches won't gate and merge if there's a Depends-On which includes an unmerged stable/liberty patch | 13:51 |
| openstackgerrit | Mike Burns proposed openstack/python-tripleoclient: Pass parameter names from parameter_defaults to clear_parameters https://review.openstack.org/297163 | 13:51 |
| EmilienM | ok I was confused, if your patch depends-on a patch in another branch, AFIK the depends-on won't be effective - but - if you use depends-on on patchs on the same branch, that's fine | 13:54 |
| slagle | pradk: is https://review.openstack.org/#/c/289435/ the only patch that's needed for mitaka? | 13:54 |
| shardy | EmilienM: the problem is when you Depends-On a patch which is proposed to more than one branch | 13:54 |
| slagle | pradk: i thought i saw another one related to gnoicchi | 13:54 |
| slagle | gnochi | 13:54 |
| shardy | the ChangeId is the same for master and stable patches, and the gate can't figure that out apparently so it just stops | 13:55 |
| EmilienM | shardy: it will only take effect on the patch from same branch, afik | 13:55 |
| EmilienM | but I might be wrong | 13:55 |
| shardy | EmilienM: That's not my experience with TripleO or Heat patches | 13:55 |
| pradk | slagle, for aodh that all thats remaining.. there are gnocchi patches in line as well https://review.openstack.org/#/c/252032/ and few that this depend on | 13:55 |
| shardy | maybe we have something configured wrong | 13:55 |
| EmilienM | can anyone help with the hiera error I just pasted 10 min ago? | 13:55 |
| slagle | pradk: and all that is needed for mitaka? | 13:55 |
| pradk | slagle, it is highly desirable yes and gnocchi patch should be low impact as its a new service and no upgrades needed | 13:56 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things https://review.openstack.org/294071 | 13:56 |
| gfidente | jaosorior ^^ the TLS columns | 13:56 |
| EmilienM | when enabling ipv6 templates, we have a databinding issue, and log files don't provide templates so we can't know what is wrong at L303 | 13:56 |
| shardy | pradk: what happens to all the metrics in the old storage backend? | 13:56 |
| pradk | slagle, aodh is made a blocker for mitaka, gnocchi is bestest effort, gnocchi is pretty much ready to go as well | 13:57 |
| shardy | on upgrade | 13:57 |
| gfidente | EmilienM ack looking into it | 13:57 |
| pradk | shardy, they would remain in old db.. we're not committing to automatic upgrades .. gnocchi is a new backend service that people can use from mitaka on | 13:58 |
| shardy | I guess I mean, if you have an alarm based on an average of some collected metric, what happens on upgrade - do we end up with both backends simultaneously configured? | 13:59 |
| shardy | Or does everything cut over to the new single backend | 13:59 |
| jaosorior | gfidente: thanks | 14:00 |
| pradk | shardy, we still have mongo for events, and metric will flow into gnocchi.. in upgrade case, probably the best thing to do is keep mongo as default on upgrade and let the manually switch to gnocchi.. enable gnocchi on fresh install only | 14:00 |
| pradk | probably the least impacted way i can think of | 14:01 |
| pradk | we're not supporting or committing to data migration | 14:01 |
| pradk | so thats for sure | 14:01 |
| shardy | pradk: Ok, but this patch doesn't handle any of that, it just switches everything over to gnocchi, right? | 14:02 |
| pradk | shardy, isnt upgrades a separate script anyway? | 14:02 |
| slagle | shardy: pradk : i dont think the gnocchi patch is ready to land | 14:03 |
| slagle | shardy: pradk : it only passed HA 1 time on 27 PS's since december | 14:04 |
| shardy | pradk: Not exactly, we have to consider the heat workflow associate with the stack update, it's not just the pacemaker stuff in this script | 14:04 |
| pradk | shardy, this patch by default lets ceilo use gnocchi as the dispatcher.. only difference is what we set the meter_dispatcher to | 14:04 |
| pradk | slagle, i tested gnocchi in ha case and its in good shape i think | 14:04 |
| *** ebalduf has joined #tripleo | 14:04 | |
| *** panda has quit IRC | 14:05 | |
| shardy | if you imagine someone chasing trunk and consuming t-h-t, they do an update and this switches out their metric backend without warning | 14:05 |
| slagle | shardy: pradk : how about this, we land aodh, and consider a backport exception for gnocchi at a later date. we could vote on it or something | 14:05 |
| shardy | so at least it's something we'll have to add to the todo list wrt upgrades | 14:05 |
| *** panda has joined #tripleo | 14:05 | |
| shardy | as it seems to be a potentially backwards incompatible upgrade | 14:05 |
| pradk | slagle, i'm cool with that.. i can get any suggestion into gnocchi asap if there is opinion on how we want to handle upgrades | 14:05 |
| pradk | shardy, so in upgrade case if we want to default to database instead of gnocchi we just have to flip CeilometerMeterDispatcher flag | 14:06 |
| pradk | gnocchi is still installed in upgrades | 14:06 |
| *** myoung has joined #tripleo | 14:06 | |
| pradk | just not configured in ceilometer | 14:06 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates: Set UpdateIdentifier for upgrade converge, to prevent services down https://review.openstack.org/297175 | 14:07 |
| pradk | shardy, https://review.openstack.org/#/c/252032/28/overcloud.yaml in here | 14:07 |
| pradk | shardy, that would be done via upgrade script right? or somewhere else? | 14:07 |
| pradk | alternately for mitaka, we can default to mongo everywhere and make gnocchi an optional backend | 14:08 |
| slagle | trown: where does current-passed-ci come from? it's quite old, march 4th | 14:08 |
| slagle | trown: i thought we've had a successful promote since then | 14:08 |
| shardy | pradk: I'm wondering if it'd be safer to switch the default to be the current backend | 14:08 |
| pradk | that way we can get the code in? but not enabled by default | 14:08 |
| shardy | Or, we'll have to document it carefully so folks know to enable the old one if they don't want to deal with the change immediately | 14:09 |
| pradk | shardy, we prefer to enable gnocchi by default, but it means its the only way to get it into mitaka, i can try to convince pms | 14:09 |
| trown | slagle: we are not running CI on master right now, so we would need to look at current-passed-ci on mitaka | 14:09 |
| shardy | pradk: Sure, I'm really just trying to get the conversation started, so we can start to understand what's involved in an upgrade where there's real data in the current backend | 14:09 |
| trown | slagle: https://trunk.rdoproject.org/centos7-mitaka/current-passed-ci/versions.csv | 14:10 |
| slagle | trown: ok, great, that's better | 14:10 |
| pradk | shardy, in upgrade case since we're not providing a way to migrate data, its best to keep using database backend and let them switch manually | 14:10 |
| shardy | pradk: I've got no issue with it being the default for existing deployments, it's just the upgrade case we've got to figure out | 14:11 |
| shardy | pradk: Ok, we'll need to document the steps to do that I guess, and have someone test it | 14:11 |
| pradk | shardy, understand, so as i said above, for upgrades https://review.openstack.org/#/c/252032/28/overcloud.yaml ceilometer_meter_dispatcher should be database | 14:11 |
| *** myoung has quit IRC | 14:11 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying RGW on controllers. https://review.openstack.org/289027 | 14:11 |
| shardy | pradk: ack, thanks - that's basically what I wanted to know :) | 14:12 |
| pradk | shardy, that would be done via upgrade script or in the above patch i'm not sure | 14:12 |
| shardy | pradk: the upgrade script doesn't influence heat parameters | 14:12 |
| pradk | shardy, so in upgrade case where would the swicth in params happen? | 14:12 |
| *** hjensas has quit IRC | 14:13 | |
| shardy | pradk: It'll just accept your new default and switch everything over, hence all my questions :) | 14:13 |
| shardy | https://github.com/openstack/tripleo-heat-templates/tree/master/environments/updates | 14:13 |
| shardy | we might end up having an environment file in there which shows sensible defaults for upgrading | 14:13 |
| beagles | is there known issue with the swift puppet modules when doing an undercloud install? I'm getting this for an error: "Could not find dependency File[/etc/swift/account-server.conf]". | 14:13 |
| pradk | shardy, yea we probably dont want that .. is there a way to flip that flag in upgrade case? i think this will be the case regardless of release | 14:14 |
| pradk | shardy, yea env file for upgrades might be the way to go | 14:14 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Wire redis_password to the tripleo module https://review.openstack.org/295958 | 14:14 |
| beagles | it's vaguely familiar, but didn't have this problem when I did the same thing last week... | 14:15 |
| pradk | shardy, i assume such an env file doesnt exist yet? | 14:16 |
| pradk | shardy, would that block gnocchi from getting into mitaka? alternately as a last resort we just default everthing to database backend and make gnocchi non default option and flip to gnocchi default in Newton | 14:17 |
| pradk | if that eases upgrade concerns | 14:18 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-ui: Ports parameters to redux https://review.openstack.org/297180 | 14:18 |
| chem | hi guys, would one core review this, it's a very small patch, about the removal of an unused resource, which prevents me from pushing puppet-pacemaker changes : https://review.openstack.org/#/c/294517/ | 14:19 |
| *** dustins has joined #tripleo | 14:21 | |
| shardy | pradk: I'm not blocking on anything, just trying to figure out the safest approach | 14:22 |
| pradk | slagle, shardy, yea i understand, i'm willing to work with you guys on the safest path .. If upgrades are a concern and we dont have a way to override that param separately during upgrades to use database then we should not default to gnocchi | 14:23 |
| *** mcornea has quit IRC | 14:24 | |
| pradk | once the env file for upgrades you mentioned comes into play then i think its safe to use gnocchi for fresh installs and existing db for upgrades | 14:25 |
| *** mcornea has joined #tripleo | 14:25 | |
| pradk | if we have a legit reason to not default to gnocchi, i can try to convince Nick and others .. my goal is to get gnocchi into mitaka even if it mean it might not be the default, at least it in the installer and we can document how to enable it prtty easily as its just a single flag to flip | 14:27 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Remove unused user resource. https://review.openstack.org/294517 | 14:27 |
| *** Ryjedo has joined #tripleo | 14:29 | |
| chem | ^ thanks | 14:31 |
| shardy | pradk: There's some hardcoded resource/user/project id's for statsd, is that intentional? | 14:34 |
| shardy | https://review.openstack.org/#/c/252032/28/puppet/hieradata/controller.yaml | 14:34 |
| shardy | it looks like something that should be parameterized? | 14:34 |
| pradk | shardy, yes those are seed ids for statsd | 14:34 |
| pradk | shardy, not openstack ids hence i put them in hiera | 14:35 |
| shardy | Ok, and they don't need to be unique per deployment | 14:35 |
| *** jaosorior has quit IRC | 14:37 | |
| adarazs | EmilienM: FYI the mongodb patch we depended on didn't help. the jobs are just failing with the same error now. | 14:39 |
| EmilienM | adarazs: ack | 14:39 |
| EmilienM | adarazs: I have no idea about the hiera thing | 14:40 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 14:40 |
| EmilienM | gfidente: did you look at it yet? | 14:40 |
| adarazs | EmilienM: I don't understand the real cause of this error either. | 14:40 |
| gfidente | EmilienM I am worried about the patch for the iscsi_ip_address in brackets as hiera might be interpreting that as an array | 14:44 |
| gfidente | so we might have to add quotes around it :( | 14:44 |
| *** absubram has quit IRC | 14:45 | |
| EmilienM | adarazs: did we have the hiera bug before iscsi patch? let me check logs | 14:46 |
| adarazs | EmilienM: no. it might be caused by that, but I don't understand why that could be the source. but it's possible. | 14:48 |
| EmilienM | gfidente: you're maybe right, I don't see it | 14:48 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: composable glance services https://review.openstack.org/237370 | 14:49 |
| *** electrofelix has joined #tripleo | 14:49 | |
| *** electrofelix has quit IRC | 14:50 | |
| *** bvandenh has joined #tripleo | 14:50 | |
| openstackgerrit | Imre Farkas proposed openstack/python-tripleoclient: [WIP] Update baremetal ready state command https://review.openstack.org/285977 | 14:54 |
| openstackgerrit | Dmitry Ilyin proposed openstack/puppet-pacemaker: [WIP] Merge with fuel-infra/puppet-pacemaker https://review.openstack.org/296440 | 14:57 |
| *** yamahata has joined #tripleo | 14:58 | |
| gfidente | EmilienM that's the problem yes, I just reproduced it! :( | 15:00 |
| EmilienM | ok, cool | 15:01 |
| gfidente | so I am trying to figure how to put it in quotes now :( | 15:01 |
| *** mbound has quit IRC | 15:01 | |
| *** yamahata has quit IRC | 15:02 | |
| *** links has joined #tripleo | 15:02 | |
| *** yamahata has joined #tripleo | 15:02 | |
| *** bvandenh has quit IRC | 15:04 | |
| *** mcornea has quit IRC | 15:04 | |
| *** mcornea has joined #tripleo | 15:05 | |
| pradk | slagle, shardy aodh is passing all green https://review.openstack.org/#/c/289435/ can we merge before soemthing happens again :) | 15:05 |
| slagle | pradk: shardy : it passed https://review.openstack.org/#/c/289435/ :) | 15:05 |
| slagle | i did | 15:05 |
| pradk | oh ty slagle | 15:06 |
| *** bvandenh has joined #tripleo | 15:06 | |
| *** tosky_ has joined #tripleo | 15:06 | |
| shardy | yay :) | 15:08 |
| adarazs | gfidente: let me know when you have some solution, I can try it with the CI jobs. | 15:11 |
| *** lucasagomes_ has joined #tripleo | 15:11 | |
| *** Ryjedo has quit IRC | 15:13 | |
| *** MaxPC has quit IRC | 15:13 | |
| *** lifeless has quit IRC | 15:13 | |
| *** tosky has quit IRC | 15:13 | |
| *** rwsu has quit IRC | 15:13 | |
| *** david-lyle has quit IRC | 15:13 | |
| *** therve has quit IRC | 15:13 | |
| *** yuanying has quit IRC | 15:13 | |
| *** lucasagomes has quit IRC | 15:13 | |
| *** bhunter has quit IRC | 15:13 | |
| *** mburned has quit IRC | 15:13 | |
| *** dmsimard has quit IRC | 15:13 | |
| gfidente | adarazs marios I am sorry about that but I tested it manually | 15:13 |
| gfidente | and while the brackets fixed the cinder issue, they introduced this parsing problem :) | 15:13 |
| gfidente | gfidente-- | 15:13 |
| *** shardy has quit IRC | 15:14 | |
| adarazs | gfidente: well, I would not have thought of that... granted I don't really have a clue how all this works and the possible interaction. but it looks like very few people can really follow it :) | 15:14 |
| adarazs | gfidente: and the non-ipv6 gates didn't catch it, so that's okay. we'll have an ipv6 gate soon enough :) | 15:15 |
| *** hewbrocc` has joined #tripleo | 15:15 | |
| *** mcornea has quit IRC | 15:17 | |
| *** Guest27982 has quit IRC | 15:17 | |
| *** rcernin has quit IRC | 15:17 | |
| *** david-lyle has joined #tripleo | 15:17 | |
| *** hewbrocca has quit IRC | 15:18 | |
| *** links has quit IRC | 15:18 | |
| openstackgerrit | Dmitry Ilyin proposed openstack/puppet-pacemaker: [WIP] Merge with fuel-infra/puppet-pacemaker https://review.openstack.org/296440 | 15:20 |
| *** liverpooler has quit IRC | 15:22 | |
| *** dmsimard has joined #tripleo | 15:25 | |
| *** yamahata has quit IRC | 15:26 | |
| *** Guest27982 has joined #tripleo | 15:29 | |
| *** Ryjedo has joined #tripleo | 15:29 | |
| *** MaxPC has joined #tripleo | 15:29 | |
| *** lifeless has joined #tripleo | 15:29 | |
| *** rwsu has joined #tripleo | 15:29 | |
| *** therve has joined #tripleo | 15:29 | |
| *** yuanying has joined #tripleo | 15:29 | |
| *** bhunter has joined #tripleo | 15:29 | |
| *** mburned has joined #tripleo | 15:29 | |
| *** rcernin has joined #tripleo | 15:29 | |
| *** mcornea has joined #tripleo | 15:29 | |
| *** lifeless has quit IRC | 15:30 | |
| *** lifeless has joined #tripleo | 15:30 | |
| *** lucasagomes_ is now known as lucasagomes | 15:36 | |
| *** yamahata has joined #tripleo | 15:37 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 15:37 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add quotes around the cinder_iscsi_ip_address value https://review.openstack.org/297222 | 15:37 |
| *** absubram has joined #tripleo | 15:39 | |
| gfidente | adarazs https://review.openstack.org/297222 | 15:39 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 15:41 |
| adarazs | gfidente: thanks! I kicked off the CI jobs :) | 15:41 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add ability to auto-generate self-signed certificates https://review.openstack.org/273233 | 15:41 |
| *** shardy has joined #tripleo | 15:42 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Add quotes around the cinder_iscsi_ip_address value https://review.openstack.org/297222 | 15:44 |
| *** shardy has quit IRC | 15:46 | |
| *** florianf has quit IRC | 15:46 | |
| *** shardy has joined #tripleo | 15:47 | |
| *** shardy has quit IRC | 15:48 | |
| *** shardy has joined #tripleo | 15:49 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 15:52 |
| pradk | slagle, its like it just doesn't want to go in :) | 15:54 |
| slagle | all the puppet syntax jobs are down for some reason | 15:54 |
| *** mbound has joined #tripleo | 15:55 | |
| bnemec | Yeah, looks like EmilienM has notified infra about it. | 15:59 |
| EmilienM | slagle: yes all CI is broken | 15:59 |
| EmilienM | slagle: https://github.com/jwt/ruby-jwt/issues/133 | 16:00 |
| *** mikelk has quit IRC | 16:00 | |
| EmilienM | and also OpenStack Infra is having issues with Jenkins | 16:00 |
| *** pblaho has quit IRC | 16:00 | |
| EmilienM | so you might see NOT_REGISTERED | 16:00 |
| EmilienM | they are working on it | 16:00 |
| *** ifarkas has quit IRC | 16:00 | |
| *** aufi has quit IRC | 16:02 | |
| *** oshvartz has quit IRC | 16:03 | |
| gfidente | pradk so ceph is tested in CI by the nonha job | 16:07 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Delete test stack without confirmation https://review.openstack.org/295047 | 16:07 |
| gfidente | pradk but if you think it'll help we could, maybe, add support for ceph in a followup patch | 16:07 |
| gfidente | and drop it from the initial submission | 16:07 |
| EmilienM | ok just talked on -infra, they are updating JJB right now | 16:08 |
| *** florianf has joined #tripleo | 16:08 | |
| EmilienM | don't run recheck if you see NOT_REGISTERED, it's useless | 16:08 |
| *** bnemec changes topic to "TripleO | Puppet jobs failing for multiple reasons. No point rechecking at the moment. | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 16:08 | |
| *** trown is now known as trown|lunch | 16:09 | |
| pradk | gfidente, i addressed your comments, will push once the anticipated merge conflict occurs | 16:09 |
| EmilienM | bnemec: thx | 16:09 |
| bnemec | EmilienM: np, the old topic was way out of date anyway :-) | 16:10 |
| EmilienM | once JJB is updated, I'll come here and say you can recheck | 16:10 |
| *** ifarkas has joined #tripleo | 16:10 | |
| *** mbound has quit IRC | 16:17 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 16:18 |
| *** Marga_ has joined #tripleo | 16:21 | |
| *** Marga_ has quit IRC | 16:22 | |
| *** Marga_ has joined #tripleo | 16:23 | |
| *** saneax is now known as saneax_AFK | 16:37 | |
| *** mbound has joined #tripleo | 16:46 | |
| *** mbound has quit IRC | 16:46 | |
| *** mcornea has quit IRC | 16:51 | |
| *** mbound has joined #tripleo | 16:55 | |
| *** openstackgerrit has quit IRC | 17:01 | |
| *** openstackgerrit has joined #tripleo | 17:01 | |
| *** mbound has quit IRC | 17:02 | |
| *** shanky has joined #tripleo | 17:07 | |
| *** mkovacik__ has quit IRC | 17:09 | |
| jpeeler | shardy: if you're still around - https://review.openstack.org/#/c/247724/ | 17:10 |
| *** pradk_ has joined #tripleo | 17:14 | |
| *** ifarkas has quit IRC | 17:16 | |
| *** pradk has quit IRC | 17:16 | |
| *** pradk_ is now known as pradk | 17:16 | |
| EmilienM | bnemec: puppet CI jobs are back | 17:20 |
| bnemec | EmilienM: Thanks | 17:20 |
| *** bnemec changes topic to "TripleO | Puppet jobs should be back. Recheck away! | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/" | 17:20 | |
| EmilienM | i'll give an update if something is still broken but for tripleo it should be good now | 17:20 |
| *** rdopiera has quit IRC | 17:23 | |
| *** pradk_ has joined #tripleo | 17:25 | |
| *** lucasagomes is now known as lucas-afk | 17:28 | |
| *** olap has quit IRC | 17:28 | |
| *** pradk has quit IRC | 17:30 | |
| *** pradk has joined #tripleo | 17:31 | |
| *** pradk_ has quit IRC | 17:31 | |
| *** trown|lunch is now known as trown | 17:32 | |
| *** rcernin has quit IRC | 17:36 | |
| *** tosky_ has quit IRC | 17:37 | |
| *** pcaruana has quit IRC | 17:40 | |
| *** ccamacho has quit IRC | 17:43 | |
| *** ccamacho has joined #tripleo | 17:44 | |
| *** eil397 has joined #tripleo | 17:44 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm https://review.openstack.org/289435 | 17:51 |
| Goneri | https://review.openstack.org/#/c/288041/ can someone take a look at this, it adds the ability to adjust the MTU of the br-ctlplane | 17:54 |
| slagle | trown: given aodh needs to be in mitaka, and it just merged, it doesnt make sense to branch from current-passed-ci *right* now | 17:55 |
| slagle | trown: should we wait until monday? and see if we get a successful promote? | 17:55 |
| trown | slagle: ya, I think that is a good idea | 17:58 |
| shardy | +1 sounds good | 17:58 |
| trown | although we should probably be careful what else we merge between now and then | 17:58 |
| shardy | I guess the alternative is to branch with the sha's from today and deal with any fixes afterwards | 17:58 |
| slagle | shardy: yea, was thinking the same | 17:58 |
| shardy | waiting for promote feedback sounds reasonable tho | 17:58 |
| slagle | but...we might as well wait | 17:59 |
| shardy | I won't be here on Monday btw, although I may check in on things over the weekend | 17:59 |
| shardy | if one of you guys is happy to cut the branches Monday that'd be good :) | 17:59 |
| trown | ya I will be out Monday too | 17:59 |
| slagle | ok, if things look alright, i will cut the branches on monday | 17:59 |
| shardy | slagle: sounds good, thanks! | 17:59 |
| trown | or at least mostly afk, daycare is closed | 18:00 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add systemctl reload haproxy to the pacemaker_resource_restart.sh https://review.openstack.org/296491 | 18:00 |
| slagle | trown: i assume when i do that, rdo packaging will start building from mitaka automagically? | 18:00 |
| trown | indeed | 18:00 |
| shardy | Yeah it's a public holiday here so I'll mostly be trying to not work ;) | 18:00 |
| slagle | k | 18:00 |
| shardy | jpeeler: looks good but I'll give it a proper review over the weekend | 18:00 |
| shardy | sorry it's been around for so long | 18:01 |
| pradk | shardy, so for gnocchi i changed it as we discussed and dint make it default, so database is default backend for ceilo and gnocchi is deployed but not configured | 18:02 |
| pradk | though it would end up in merge conflict now with aodh merged | 18:02 |
| pradk | so if gnocchi passes with this, are we cool with getting it into mitaka? | 18:02 |
| pradk | shardy, slagle, ^^ | 18:02 |
| pradk | it will be deployed but not used by ceilometer by default, so upgrades shouldnt be an issue any more | 18:03 |
| *** panda has quit IRC | 18:04 | |
| *** panda has joined #tripleo | 18:05 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Fix satellite registration for http or https https://review.openstack.org/296071 | 18:06 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Fix satellite registration for http or https https://review.openstack.org/297284 | 18:06 |
| shardy | pradk: Definitely rebase the patch now aodh has landed and lets see what CI says | 18:08 |
| shardy | pradk: can you also please push a patch removing the unset default for AodhPassword? | 18:08 |
| shardy | ideally we don't want to release with that | 18:08 |
| slagle | marios: given the plan to set UpdateIdentifier on updates now, is this still needed: https://review.openstack.org/#/c/296491/ | 18:08 |
| slagle | marios: it merged on master, but thought i'd double check before backporting | 18:08 |
| pradk | shardy, yep will do | 18:09 |
| slagle | "on upgrades" i should say | 18:09 |
| *** shardy has quit IRC | 18:10 | |
| marios | slagle: right we probably don't need that, but only on assumption of https://review.openstack.org/#/c/297175 landing into stable instead | 18:10 |
| slagle | marios: ok, i won't backport it then | 18:11 |
| *** jdob has quit IRC | 18:12 | |
| marios | slagle: well... idk. i mean i'd rather we *do* land it, since is ready... and then | 18:12 |
| marios | slagle: we can add its removal to https://review.openstack.org/#/c/297175 | 18:12 |
| *** jdob has joined #tripleo | 18:12 | |
| marios | slagle: as even in the present form (what is in master) it resolves an issue with the post upgrade overcloud glance | 18:12 |
| marios | slagle: brb, other chat about borked upgrades | 18:13 |
| *** athomas has quit IRC | 18:13 | |
| marios | slagle: so wdyt? | 18:13 |
| slagle | marios: either way is fine with me. propose the backport if you'd like :) | 18:13 |
| marios | slagle: oh sry, eetoomanythings, i thought it was already there,.... sec | 18:14 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Add systemctl reload haproxy to the pacemaker_resource_restart.sh https://review.openstack.org/297288 | 18:14 |
| marios | slagle: also added note at https://review.openstack.org/#/c/297175 | 18:15 |
| *** qasims has quit IRC | 18:18 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 18:21 |
| slagle | marios: cool thanks | 18:22 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Remove password default for AodhPassword https://review.openstack.org/297295 | 18:22 |
| *** jdob has quit IRC | 18:22 | |
| *** jdob has joined #tripleo | 18:23 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates: Set UpdateIdentifier for upgrade converge, to prevent services down https://review.openstack.org/297175 | 18:26 |
| *** jobewan has joined #tripleo | 18:27 | |
| *** matbu has joined #tripleo | 18:28 | |
| *** mgould has quit IRC | 18:44 | |
| *** Guest27982 has quit IRC | 18:46 | |
| *** Marga_ has quit IRC | 18:46 | |
| EmilienM | first time tripleo CI goes until pingtest with IPv6 https://jenkins05.openstack.org/job/gate-tripleo-ci-f22-nonha/1370/console | 18:58 |
| EmilienM | adarazs: sounds like we have good progress here :-) | 18:59 |
| EmilienM | 2016-03-24 19:04:38.157 | Finished: SUCCESS | 19:04 |
| EmilienM | ipv6 is working now :-) | 19:04 |
| *** qasims has joined #tripleo | 19:06 | |
| matbu | EmilienM: /me curious | 19:07 |
| matbu | EmilienM: which tools does this job used ? | 19:08 |
| *** gfidente has quit IRC | 19:09 | |
| * weshay reads | 19:09 | |
| EmilienM | matbu: https://github.com/openstack-infra/tripleo-ci | 19:09 |
| EmilienM | it's devtest | 19:09 |
| EmilienM | and this is what adarazs is doing https://review.openstack.org/#/c/289445/ | 19:10 |
| matbu | EmilienM: ha k | 19:10 |
| * matbu looks | 19:10 | |
| weshay | nice | 19:10 |
| *** pkovar has quit IRC | 19:11 | |
| *** Marga_ has joined #tripleo | 19:12 | |
| *** nico_auv has quit IRC | 19:13 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Remove default values in 02-puppet-stack-config https://review.openstack.org/297319 | 19:14 |
| bnemec | FTR, devtest is no longer maintained and almost certainly doesn't work anymore. | 19:16 |
| weshay | this ring any bells? | 19:16 |
| weshay | | resources.Controller: RemoteError: resources[1].resources.NetworkDeployment: Remote error: OperationalError (pymysql.err.OperationalError) (1040, u'Too many connections') | 19:16 |
| EmilienM | weshay: where are other logs? it can be in many places | 19:16 |
| weshay | EmilienM, I have a live system w/ that error | 19:17 |
| *** rhallisey has quit IRC | 19:22 | |
| *** jprovazn has quit IRC | 19:24 | |
| *** rhallisey has joined #tripleo | 19:25 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 19:27 |
| *** yamahata has quit IRC | 19:28 | |
| *** qasims has quit IRC | 19:31 | |
| *** davidlenwell has quit IRC | 19:35 | |
| *** tremble has quit IRC | 19:37 | |
| *** davidlenwell has joined #tripleo | 19:38 | |
| *** ChanServ sets mode: +v davidlenwell | 19:38 | |
| bnemec | weshay: We found a bug recently where mariadb wasn't able to set its file descriptor limit high enough. If you're sitting right around 1024 connections to mysql that might be the problem. | 19:43 |
| weshay | k.. thank you.. going to see if it's repeatable | 19:44 |
| *** Marga_ has quit IRC | 19:46 | |
| *** Marga_ has joined #tripleo | 19:46 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Remove default values in 02-puppet-stack-config https://review.openstack.org/297319 | 19:58 |
| *** davidlenwell has quit IRC | 20:02 | |
| *** akrivoka has quit IRC | 20:03 | |
| *** davidlenwell has joined #tripleo | 20:12 | |
| *** ChanServ sets mode: +v davidlenwell | 20:12 | |
| *** hewbrocc` has quit IRC | 20:15 | |
| *** ccamacho has quit IRC | 20:15 | |
| *** ccamacho has joined #tripleo | 20:21 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 20:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 20:23 |
| bnemec | Heh, variable naming collision. ^ was trying to delete the overcloud the first time we ran tripleo.sh. | 20:23 |
| bnemec | Have I mentioned how much I hate input env vars? | 20:23 |
| rhallisey | anyone wana checkout the container job :) https://review.openstack.org/#/c/288915/ | 20:27 |
| rhallisey | just needs another review | 20:27 |
| openstackgerrit | Merged openstack/puppet-tripleo: Allow the Redis specific monitor to use authentication https://review.openstack.org/297115 | 20:32 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Remove default values in 02-puppet-stack-config https://review.openstack.org/297319 | 20:33 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add zaqar password option https://review.openstack.org/297353 | 20:33 |
| * bnemec will not rest until every 'unset' password in TripleO is eradicated | 20:34 | |
| *** trown is now known as trown|outtypewww | 20:41 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Add systemctl reload haproxy to the pacemaker_resource_restart.sh https://review.openstack.org/297288 | 20:42 |
| *** Goneri has quit IRC | 20:44 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #tripleo | 20:48 | |
| *** Marga_ has quit IRC | 20:53 | |
| *** florianf has quit IRC | 20:53 | |
| *** Marga_ has joined #tripleo | 20:53 | |
| *** jdob has quit IRC | 20:55 | |
| *** rhallisey has quit IRC | 20:56 | |
| openstackgerrit | Keith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying RGW on controllers. https://review.openstack.org/289027 | 20:59 |
| *** lblanchard has quit IRC | 21:01 | |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Add missing zaqar password and horizon key https://review.openstack.org/297353 | 21:12 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud: Remove default values in 02-puppet-stack-config https://review.openstack.org/297319 | 21:12 |
| bnemec | Two bugs. Ah, ah, ah. | 21:14 |
| bnemec | https://c1.staticflickr.com/9/8233/8539517000_9a44748db4_b.jpg | 21:14 |
| *** panda has quit IRC | 21:20 | |
| *** jayg is now known as jayg|g0n3 | 21:27 | |
| *** Ng has quit IRC | 21:33 | |
| *** Ng has joined #tripleo | 21:34 | |
| *** ChanServ sets mode: +v Ng | 21:34 | |
| *** patrickeast has quit IRC | 21:34 | |
| *** ayoung has quit IRC | 21:34 | |
| *** weshay has quit IRC | 21:35 | |
| *** patrickeast has joined #tripleo | 21:37 | |
| *** openstack has joined #tripleo | 21:48 | |
| *** openstackgerrit has joined #tripleo | 21:49 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend https://review.openstack.org/252032 | 21:56 |
| *** dustins has quit IRC | 21:57 | |
| *** pradk has quit IRC | 21:58 | |
| *** Ryjedo has joined #tripleo | 22:00 | |
| *** MaxPC has joined #tripleo | 22:00 | |
| *** rwsu has joined #tripleo | 22:00 | |
| *** therve has joined #tripleo | 22:00 | |
| *** yuanying has joined #tripleo | 22:00 | |
| *** bhunter has joined #tripleo | 22:00 | |
| *** mburned has joined #tripleo | 22:00 | |
| *** penick has joined #tripleo | 22:01 | |
| *** Ng has quit IRC | 22:03 | |
| *** Ng has joined #tripleo | 22:05 | |
| *** ChanServ sets mode: +v Ng | 22:05 | |
| dmsimard | colonwq: good job being very persistent with the puppet-ceph and tripleo work :) | 22:05 |
| *** r-mibu has quit IRC | 22:31 | |
| *** r-mibu has joined #tripleo | 22:31 | |
| *** penick has quit IRC | 22:44 | |
| *** jobewan has quit IRC | 22:53 | |
| *** chlong has joined #tripleo | 22:55 | |
| *** ccamacho has quit IRC | 22:56 | |
| *** ccamacho has joined #tripleo | 22:57 | |
| *** saneax_AFK is now known as saneax | 23:00 | |
| *** akshai has quit IRC | 23:00 | |
| *** ccamacho has quit IRC | 23:01 | |
| *** ccamacho has joined #tripleo | 23:01 | |
| *** jcoufal has quit IRC | 23:06 | |
| *** davidlenwell has quit IRC | 23:07 | |
| colonwq | dmsimard, thanks. Trying the round off the last bits of my patch. | 23:11 |
| *** davidlenwell has joined #tripleo | 23:14 | |
| *** ChanServ sets mode: +v davidlenwell | 23:14 | |
| *** mburned is now known as mburned_out | 23:15 | |
| *** ccamacho has quit IRC | 23:28 | |
| *** Ryjedo has quit IRC | 23:42 | |
| *** Ryjedo has joined #tripleo | 23:46 | |
| *** Ryjedo has quit IRC | 23:47 | |
| *** lblanchard has joined #tripleo | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
