Monday, 2015-08-17

« Sunday, 2015-08-16 Index Tuesday, 2015-08-18 »
openstackgerritPeter Stachowski proposed openstack/trove: Add support for configuration groups in int-tests  https://review.openstack.org/21338800:09
openstackgerritPeter Stachowski proposed openstack/trove: Add generic int-test classes  https://review.openstack.org/20327400:09
*** imandhan has joined #openstack-trove00:10
*** imandhan has quit IRC00:12
*** nexusz99 has joined #openstack-trove01:03
*** erkules_ has joined #openstack-trove01:26
*** topshare has joined #openstack-trove01:28
*** erkules has quit IRC01:29
openstackgerritMasaki Matsushita proposed openstack/trove-integration: test REDSTACK_SCRIPTS instead of REQUIREMENTS_FILE  https://review.openstack.org/21357602:00
*** 6JTABTZX7 has joined #openstack-trove02:02
*** macjack has joined #openstack-trove02:07
*** logan2 has quit IRC02:18
openstackgerritMasaki Matsushita proposed openstack/trove-integration: add HOST_USERNAME existence check  https://review.openstack.org/21357802:18
openstackgerritMasaki Matsushita proposed openstack/trove-integration: use HOST_USERNAME as default of HOST_SCP_USERNAME  https://review.openstack.org/21358002:31
*** sabeen has joined #openstack-trove02:54
*** logan2 has joined #openstack-trove02:57
*** 6JTABTZX7 has quit IRC03:01
*** haomaiwang has joined #openstack-trove03:02
*** imandhan has joined #openstack-trove03:12
*** imandhan has quit IRC03:13
*** sabeen has quit IRC03:14
*** sabeen has joined #openstack-trove03:15
*** sabeen has quit IRC03:44
*** haomaiwang has quit IRC04:01
*** haomaiwang has joined #openstack-trove04:01
*** openstack has joined #openstack-trove04:17
*** haomaiwa_ has joined #openstack-trove04:49
*** haomaiwang has quit IRC04:49
*** haomaiwa_ has quit IRC05:01
*** haomaiwang has joined #openstack-trove05:02
*** kmARC__ has joined #openstack-trove05:25
*** kmARC_ has joined #openstack-trove05:25
*** coolsvap|away is now known as coolsvap05:32
*** macjack has quit IRC05:38
*** macjack has joined #openstack-trove05:42
*** exploreshaifali has joined #openstack-trove05:52
*** coolsvap is now known as coolsvap|away05:55
*** haomaiwang has quit IRC06:01
*** haomaiwa_ has joined #openstack-trove06:02
*** macjack has quit IRC06:30
*** macjack has joined #openstack-trove06:36
*** macjack has quit IRC06:53
*** macjack has joined #openstack-trove06:55
*** kmARC_ has quit IRC06:55
*** kmARC__ has quit IRC06:55
*** haomaiwa_ has quit IRC07:01
*** haomaiwang has joined #openstack-trove07:02
*** macjack has quit IRC07:06
-openstackstatus- NOTICE: Gerrit is currently under very high load and may be unresponsive. infra are looking into the issue.07:07
*** macjack has joined #openstack-trove07:07
*** k4n0 has joined #openstack-trove07:17
*** macjack has quit IRC07:19
*** macjack has joined #openstack-trove07:22
*** haomaiwa_ has joined #openstack-trove07:40
*** haomaiwang has quit IRC07:40
*** IanGovett has joined #openstack-trove07:50
*** haomaiwa_ has quit IRC08:01
*** haomaiwang has joined #openstack-trove08:02
*** exploreshaifali has quit IRC08:02
*** sgotliv has joined #openstack-trove08:09
*** kmARC__ has joined #openstack-trove08:30
*** kmARC_ has joined #openstack-trove08:30
*** sgotliv has quit IRC08:38
*** haomaiwang has quit IRC08:43
*** haomaiwang has joined #openstack-trove08:44
*** exploreshaifali has joined #openstack-trove08:45
*** exploreshaifali has quit IRC08:51
*** sgotliv has joined #openstack-trove08:52
*** haomaiwang has quit IRC09:01
*** haomaiwang has joined #openstack-trove09:02
*** sgotliv has quit IRC09:02
*** kbyrne has quit IRC09:12
*** sgotliv has joined #openstack-trove09:14
*** kbyrne has joined #openstack-trove09:17
*** tosky has joined #openstack-trove09:21
*** topshare has quit IRC09:22
*** macjack has quit IRC09:31
*** exploreshaifali has joined #openstack-trove09:32
*** haomaiwang has quit IRC10:01
*** haomaiwang has joined #openstack-trove10:02
*** macjack has joined #openstack-trove10:14
-openstackstatus- NOTICE: review.openstack.org (aka gerrit) is going down for an emergency restart10:23
*** ChanServ changes topic to "review.openstack.org (aka gerrit) is going down for an emergency restart"10:23
*** exploreshaifali has quit IRC10:27
*** macjack has quit IRC10:29
*** exploreshaifali has joined #openstack-trove10:43
*** ChanServ changes topic to "Trove Openstack Database as a Service | Docs https://github.com/openstack/trove | Channel Logs http://eavesdrop.openstack.org/irclogs/ | IRC must read http://sackheads.org/~bnaylor/spew/away_msgs.html"10:49
-openstackstatus- NOTICE: Gerrit restart has resolved the issue and systems are back up and functioning10:49
*** exploreshaifali has quit IRC10:51
*** haomaiwang has quit IRC11:01
*** sgotliv has quit IRC11:01
*** haomaiwang has joined #openstack-trove11:02
*** kmARC_ has quit IRC11:06
*** kmARC__ has quit IRC11:07
*** kmARC__ has joined #openstack-trove11:21
*** kmARC_ has joined #openstack-trove11:21
*** sgotliv has joined #openstack-trove11:23
*** haomaiwang has quit IRC11:36
*** sgotliv has quit IRC11:42
*** exploreshaifali has joined #openstack-trove11:46
*** sgotliv has joined #openstack-trove11:57
*** chlong has quit IRC11:59
*** sgotliv has quit IRC11:59
*** sgotliv has joined #openstack-trove12:00
*** k4n0 has quit IRC12:18
*** kmARC_ has quit IRC12:19
*** kmARC__ has quit IRC12:20
*** k4n0 has joined #openstack-trove12:21
*** pmackinn has joined #openstack-trove12:22
*** k4n0 has quit IRC12:23
*** kmARC__ has joined #openstack-trove12:40
*** kmARC_ has joined #openstack-trove12:40
*** lifeless has quit IRC13:05
*** tomblank has joined #openstack-trove13:08
*** haomaiwa_ has joined #openstack-trove13:13
*** lifeless has joined #openstack-trove13:28
*** exploreshaifali has quit IRC13:48
*** exploreshaifali has joined #openstack-trove13:50
*** haomaiwa_ has quit IRC14:01
*** haomaiwang has joined #openstack-trove14:02
*** chlong has joined #openstack-trove14:03
*** _crobertsrh is now known as crobertsrh14:17
*** georgelorch has quit IRC14:21
*** georgelorch has joined #openstack-trove14:29
*** pmackinn has quit IRC14:40
*** exploreshaifali has quit IRC14:41
*** ashleighfarnham has quit IRC14:56
*** haomaiwang has quit IRC15:01
*** haomaiwang has joined #openstack-trove15:02
*** csoukup has joined #openstack-trove15:06
*** morgan_503 is now known as morgan_254915:06
*** haomaiwang has quit IRC15:12
*** chlong has quit IRC15:18
openstackgerritPetr Malik proposed openstack/trove: Add backup & restore for Cassandra  https://review.openstack.org/20675115:19
*** topshare has joined #openstack-trove15:30
*** chlong has joined #openstack-trove15:31
*** topshare has quit IRC15:33
*** chlong has quit IRC15:38
*** pmackinn has joined #openstack-trove15:40
*** chlong has joined #openstack-trove15:40
*** pmackinn_ has joined #openstack-trove15:43
*** pmackinn has quit IRC15:45
*** pmackinn_ is now known as pmackinn15:45
*** imandhan has joined #openstack-trove15:57
*** imandhan has quit IRC16:11
*** david-ly_ is now known as david-lyle16:15
*** exploreshaifali has joined #openstack-trove16:16
openstackgerritSaurabh Surana proposed openstack/trove: Add floating ip to vertica master node  https://review.openstack.org/20757816:17
*** ashleighfarnham has joined #openstack-trove16:18
*** kmARC_ has quit IRC16:19
*** kmARC__ has quit IRC16:20
*** tomblank has quit IRC16:20
*** tomblank has joined #openstack-trove16:30
*** tomblank has quit IRC16:44
*** imandhan has joined #openstack-trove16:44
*** c_soukup has joined #openstack-trove16:49
*** csoukup_ has joined #openstack-trove16:50
*** sgotliv has quit IRC16:51
*** c_soukup has quit IRC16:53
*** csoukup has quit IRC16:54
*** rjaiswal has joined #openstack-trove17:05
*** tomblank has joined #openstack-trove17:26
*** tosky has quit IRC17:54
*** crobertsrh has left #openstack-trove17:56
imandhanIs there a way to force delete trove instances that are stuck in the build stage?18:00
*** crobertsrh has joined #openstack-trove18:02
*** tomblank_ has joined #openstack-trove18:04
*** tomblank has quit IRC18:06
*** tomblank_ is now known as tomblank18:06
*** IanGovett1 has joined #openstack-trove18:06
*** IanGovett has quit IRC18:08
*** IanGovett has joined #openstack-trove18:16
pmackinnimandhan, i delete the instance in nova then trove-manage db_recreate...someone else might have a less heavy-handed approach18:16
*** IanGovett1 has quit IRC18:17
imandhanThat worked perfectly! Thanks pmackinn18:18
pmackinnnot sure how "perfect" it is :-)18:22
*** sharikapongubala has joined #openstack-trove18:23
*** saurabhs has joined #openstack-trove18:30
imandhanhaha, works for now!18:41
sharikapongubalavkmc , amrith , peterstac can i please have reviews, on this patch https://review.openstack.org/#/c/197360/ ? :) it should be easy to test with https://review.openstack.org/#/c/196913/ . Together they pass all integration and unittests. I really appreciate it ! Thank you!18:55
*** jwiles_ has joined #openstack-trove19:01
*** imandhan has quit IRC19:06
*** imandhan has joined #openstack-trove19:22
pmackinnSlickNik, ping19:23
pmackinnredstack question (for anyone): does the redstack default requirements file somehow get merged with the one from /opt/stack/trove?19:26
peterstacsharikapongubala, I should be able to test it today19:31
sharikapongubalapeterstac cool thank you! :)19:32
peterstacpmackinn, the requirements files in scripts/files/requirements ?19:32
peterstacI believe those are used to set up the guest image, and are stand-alone19:33
pmackinnpeterstac, yes, more specifically where does the image get oslo.log from?19:33
peterstacI was just looking into that recently - I think it's brought in by oslo.service19:33
pmackinnhttps://github.com/openstack/trove-integration/blob/master/scripts/files/requirements/ubuntu-requirements-default.txt19:34
peterstacI'm not sure why we don't have an entry for it explicitly19:34
pmackinn^^ doesn't have it either19:34
pmackinnahhh so maybe getting away with it? :-)19:34
pmackinntransitive dep19:34
peterstacThat's my guess - but I think we should probably require it explicitly too19:35
*** sharikapongubala has quit IRC19:35
peterstacI believe that's what other projects do19:35
peterstacMaybe SlickNik can chime in if he's around (there may be a reason it isn't declared outright)19:35
pmackinnbut it must be coming in the redstack gate right? somehow19:36
pmackinnmagic19:36
peterstacin the gate we run the equivalent of kick-start, so it gets it from scripts/files/*-requirements-*19:37
pmackinnright but if oslo.log wasn't getting installed the gate would fail19:38
pmackinnand it isn't19:38
pmackinnfailing that is19:38
pmackinnhttps://github.com/openstack/trove/blob/master/trove/cmd/guest.py#L2819:39
*** pmackinn is now known as pmackinn-afk19:47
*** _amrith_ is now known as amrith19:55
amrithvkmc ... yt?19:55
vkmcamrith, here19:55
SlickNikpmackinn-afk: I suspect — like peterstac mentioned — it's getting pulled in transitively by another oslo.* requirement.20:06
cp16neti've manually set the instance to active in the db so it can be deleted.20:16
*** sharikapongubala has joined #openstack-trove20:16
cp16netknow the instance id and "update service_statuses set status_id=1, status_description='running' where instance_id='<instance id>'"20:17
cp16neti think there is a cmd to do this20:18
*** sharikapongubala has quit IRC20:21
*** sharikapongubala has joined #openstack-trove20:33
*** pmackinn-afk is now known as pmackinn20:36
*** kfox1111 has joined #openstack-trove20:36
kfox11112015-08-17 20:36:02.210 3744 INFO trove.guestagent.datastore.service [-] DB server is not installed or is in restart mode, so for now we'll skip determining the status of DB on this instance.20:36
kfox1111but it keeps spewing once a second:20:36
kfox1111trove.guestagent.datastore.mysql.service [-] MySQL Service Status is RUNNING.20:36
kfox1111know why that would be?20:36
cp16netif its saying that the DB server is not installed there must have been some error with install20:41
cp16netor something in the prepare method that failed20:41
cp16netthe periodic task checks the service about every second20:41
cp16netso it sounds like the service is running but probably failed..  like install or secure or configuration20:42
cp16netkfox1111: check earlier in the log for something20:43
kfox1111k...20:43
kfox1111http://pastebin.com/uBLGxiqZ20:45
kfox1111not really...20:45
kfox1111its like part of trove thinks its up, and part down.20:47
cp16netCaught SIGTERM, exiting20:49
cp16netlooks like the the guest stopped and was restarted20:50
kfox1111probably at the very begining during the provisioning of the node.20:50
cp16nethmm20:50
cp16netStopping MySQL.20:50
cp16netlooks like it was trying to stop mysql and it couldnt complete that task20:51
cp16netis this with devstack?20:51
kfox1111no. kilo rdo cloud. ubuntu 15.04 vm.20:51
cp16netok20:52
kfox1111in the setup, mysql isn't installed until trove-guestagent installs it.20:52
cp16netso it looks like its trying to stop mysql to setup the volume20:52
SlickNikkfox1111 / cp16net: I think you can safely ignore that line — I think what's going on is that while running "prepare" the main python task thread puts the service in "restart-mode" and then restarts mysql (after securing it).20:52
kfox1111so it does seem to install, and start it.20:53
kfox1111then part of it looses track of if its running or not?20:53
SlickNikSo if the heartbeat thread kicks in when this is happening, it realizes it is in "restart mode" — i.e. it's a known restart, so it doesn't send an unhealthy heartbeat message.20:53
SlickNikWhen mysql comes back up, it should start sending the healthy heartbeats again — I think these correspond to your "MySQL Service Status is RUNNING." messages.20:55
cp16netyeah then it attempts to stop mysql after its installed and running to setup the volume20:55
SlickNikHaving DEBUG logging turned on would help confirm that this is what's going on, but I suspect that this is what's happening and it's expected.20:56
kfox1111ok.20:56
SlickNik(since a prepare of the mysql instance involves stopping mysql mounting the volume, securing mysql, and then restarting it.)20:57
SlickNik^^You should see all of this happening if you turn on debug logging20:57
kfox1111k. I'll turn it back on and launch another.20:57
kfox1111is there a command to cancel a backup?20:58
cp16netif it didnt get passed that in a minute or 2 then i'm guessing there was a time out or something trying to stop mysql20:58
kfox1111let me look in the logs...20:59
kfox1111yup: 2015-08-17 20:32:13.424 3744 ERROR oslo_messaging.rpc.dispatcher [req-57ebeda9-c1c3-46dc-82fa-010ac1e7396f f5a3891411f34614b499f53fae5674ad 94c1fbbc041f477f9b56275c887ac724 - - -] Exception during message handling: Could not stop MySQL!21:00
kfox1111darn. meeting time. bbiab.21:00
cp16nethad a feeling it was something like that21:02
cp16netnow we need to see why that happened. a timeout? service call couldnt find pid? something else?21:04
*** tomblank has quit IRC21:14
*** rjaiswal has quit IRC21:20
*** crobertsrh is now known as _crobertsrh21:33
kfox1111back.21:41
kfox1111hmm... if I go to the users panel in horzon, I get this:21:43
kfox1111OperationalError: (OperationalError) (1045, "Access denied for user 'os_admin'@'localhost' (using password: NO)") None None21:43
kfox1111is there an equiv of nova reset-state in trove?21:45
*** amrith is now known as _amrith_21:45
*** csoukup_ has quit IRC21:49
*** exploreshaifali has quit IRC21:56
*** jwiles_ has quit IRC22:05
kfox1111is there a way to set the default security group rules for trove vm's?22:10
kfox1111(ssh for example)22:10
SlickNikkfox1111: Yes, there's a per-datastore config I think — one sec, let me check.22:11
kfox1111k. thanks. :)22:12
kfox1111still seeing the cant stop mysql thing on a fresh vm.22:13
kfox1111if I user list, I get: OperationalError: (OperationalError) (1045, "Access denied for user 'os_admin'@'localhost' (using password: NO)") None None, so I'm guessing its related to that?22:14
kfox1111is ubuntu 15.04 mysql tighter permissions?22:14
SlickNikkfox: https://github.com/openstack/trove/blob/master/etc/trove/trove.conf.sample#L21822:15
openstackgerritPetr Malik proposed openstack/trove: Add backup & restore for Cassandra  https://review.openstack.org/20675122:15
kfox1111ah. thanks. :)22:16
SlickNikkfox: that may be happening because the initial prepare call failed — it's the initial prepare call that creates the local os_admin user which gets used for later tasks (like user/db creation via the API).22:16
SlickNikkfox: can you paste the guest log you are seeing (with debug logging enabled)? Probably need to look at why that first prepare call is failing.22:17
kfox1111k. let me launch again with debug on.22:17
kfox1111nice. the ssh thing worked....22:19
kfox1111its installing mysql...22:20
kfox1111wait a sec...22:21
kfox1111has anyone tried trove without upstart on an ubuntu?22:21
kfox1111CMD "sudo stop mysql || true" returned: 022:22
kfox1111root@kfox-testdb:~# sudo stop mysql22:22
kfox1111sudo: stop: command not found22:22
kfox1111root@kfox-testdb:~# stop22:22
kfox1111The program 'stop' is currently not installed. You can install it by typing:22:22
kfox1111apt-get install upstart22:22
*** IanGovett has quit IRC22:23
kfox1111darn.. thats it.22:24
kfox1111the code's checking for /etc/init, which exists on 15.04, but upstart's not installed.22:24
kfox1111is this fixed in liberty?22:25
SlickNiknope, I don't think so.22:26
SlickNikWe're building 15.04 images (I think) in the gate.22:27
SlickNikIs there a reason upstart wouldn't be installed on the images you're trying to install on?22:27
kfox1111no clue. they are stock right from ubutu's cloud page.22:27
kfox1111ubuntu22:27
kfox1111seems like they went all in with systemd on those images.22:28
SlickNikThe guest does support using systemd, but currently it _always_ chooses the upstart codepath for ubuntu22:28
*** rjaiswal has joined #openstack-trove22:29
SlickNikAh, I think we're building 14.04 images in the gate22:29
kfox1111yeah. and that seems backwards for the 15.04 case. :/22:29
SlickNikI bet those still have upstart22:29
kfox1111yeah.22:29
kfox1111I'm filing a bug.22:29
SlickNikSo this seems like an issue with 15.04 — and is a bug that we should fix so we work with 15.0422:29
SlickNikkfox1111: thanks!22:30
SlickNikyup, let me know once you have the bug filed — will get a fix in for Liberty.22:30
kfox1111https://bugs.launchpad.net/trove/+bug/148578922:31
openstackLaunchpad bug 1485789 in Trove "ubuntu systemd" [Undecided,New]22:31
kfox1111hm... doesn't look like upstart and systemd can co exist easily?22:33
*** chlong has quit IRC22:34
kfox1111arg... debian's code to attempt to support all init systems forces upstart, then init, and only lastly systemd to be checked. :/22:36
kfox1111lets see if this works: sed -i 's@/etc/init/%s.conf@/etc/init/%s.conf.disabled@;s@/etc/init.d/%s@/etc/init.d/%s.disabled@' /usr/lib/python2.7/dist-packages/trove/guestagent/common/operating_system.py22:45
kfox1111:)22:45
kfox1111there we go. :)22:49
kfox1111when you build a replica, can you specify a server group?23:09
kfox1111or can trove ensure they are all on different hosts?23:10
openstackgerritSaurabh Surana proposed openstack/trove-specs: Floating ip on vertica master node  https://review.openstack.org/21335423:17
*** tiny-hands has joined #openstack-trove23:20
openstackgerritSharika Pongubala proposed openstack/trove: Modifying Vertica clusters to have a master node  https://review.openstack.org/20427123:22
openstackgerritSharika Pongubala proposed openstack/trove: Modifying Vertica clusters to have a master node  https://review.openstack.org/20427123:24
*** saurabhs has quit IRC23:44
« Sunday, 2015-08-16 Index Tuesday, 2015-08-18 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!