| *** sgotliv_ has joined #openstack-trove | 00:11 | |
| *** sgotliv has quit IRC | 00:14 | |
| *** openstackgerrit has quit IRC | 00:16 | |
| *** openstackgerrit has joined #openstack-trove | 00:16 | |
| *** sgotliv_ has quit IRC | 00:22 | |
| *** PeterSchwaller has joined #openstack-trove | 00:38 | |
| *** PeterS has quit IRC | 00:40 | |
| *** ViswaV has joined #openstack-trove | 01:03 | |
| *** tellesnobrega_af is now known as tellesnobrega | 01:03 | |
| *** ViswaV_ has joined #openstack-trove | 01:07 | |
| *** ViswaV has quit IRC | 01:07 | |
| *** ianbrown has joined #openstack-trove | 01:10 | |
| *** PeterSchwaller has quit IRC | 01:20 | |
| *** ashleighfarnham has quit IRC | 01:21 | |
| imandhan | is there a way to force delete backups? | 01:26 |
|---|---|---|
| imandhan | peterstac, atomic77, ^^ | 01:27 |
| *** mingdang1 has joined #openstack-trove | 01:57 | |
| *** PeterS has joined #openstack-trove | 01:59 | |
| *** 17SAD4YI2 has joined #openstack-trove | 02:00 | |
| *** 17SAD4YI2 has quit IRC | 02:01 | |
| *** haomaiwa_ has joined #openstack-trove | 02:01 | |
| *** mingdang1 has quit IRC | 02:09 | |
| *** caishan has joined #openstack-trove | 02:11 | |
| *** ianbrown has quit IRC | 02:16 | |
| *** imandhan has quit IRC | 02:16 | |
| dougshelley66 | imandhan what state is the backup in? | 02:25 |
| *** david-ly_ has quit IRC | 02:36 | |
| *** tellesnobrega is now known as tellesnobrega_af | 02:36 | |
| *** haomaiwa_ has quit IRC | 03:01 | |
| *** haomaiwang has joined #openstack-trove | 03:01 | |
| *** pcaruana has quit IRC | 03:27 | |
| *** fabian1 has joined #openstack-trove | 03:36 | |
| *** haomaiwang has quit IRC | 04:01 | |
| *** haomaiwa_ has joined #openstack-trove | 04:01 | |
| *** david-lyle has joined #openstack-trove | 04:08 | |
| caishan | why i get get the error message `SSH Keys were not staged by host` even if /tmp/in_target.d/ssh-authorized-keys exists ?! when i run:disk-image-create | 04:08 |
| caishan | anybody knows how to solve it? | 04:14 |
| caishan | :D | 04:14 |
| *** mingdang1 has joined #openstack-trove | 04:34 | |
| *** haomaiwa_ has quit IRC | 05:01 | |
| *** haomaiwang has joined #openstack-trove | 05:01 | |
| *** fabian1 has left #openstack-trove | 05:07 | |
| *** rcernin has joined #openstack-trove | 05:15 | |
| *** haomaiwang has quit IRC | 05:19 | |
| *** haomaiwa_ has joined #openstack-trove | 05:27 | |
| *** rcernin has quit IRC | 05:34 | |
| *** sabeen has joined #openstack-trove | 05:38 | |
| *** pcaruana has joined #openstack-trove | 05:48 | |
| *** imandhan has joined #openstack-trove | 05:53 | |
| *** imandhan has quit IRC | 05:53 | |
| *** imandhan has joined #openstack-trove | 05:53 | |
| *** haomaiwa_ has quit IRC | 06:01 | |
| *** khushbu_ has joined #openstack-trove | 06:01 | |
| *** haomaiwa_ has joined #openstack-trove | 06:02 | |
| *** rcernin has joined #openstack-trove | 06:18 | |
| *** sgotliv has joined #openstack-trove | 06:19 | |
| *** khushbu_ has quit IRC | 06:24 | |
| *** pcaruana has quit IRC | 06:28 | |
| *** khushbu_ has joined #openstack-trove | 06:34 | |
| imandhan | dougshelley66, the backups are stuck in building or new mode. the instances that created those backups were deleted. trying to delete the backups though since they are stuck | 06:34 |
| *** khushbu_ has quit IRC | 06:48 | |
| *** khushbu_ has joined #openstack-trove | 06:49 | |
| *** mingdang1 has quit IRC | 06:50 | |
| *** mingdang1 has joined #openstack-trove | 06:51 | |
| *** amrith is now known as _amrith_ | 06:54 | |
| *** khushbu_ has quit IRC | 07:00 | |
| *** haomaiwa_ has quit IRC | 07:01 | |
| *** imandhan has quit IRC | 07:01 | |
| *** haomaiwang has joined #openstack-trove | 07:01 | |
| *** sabeen has quit IRC | 07:33 | |
| *** imandhan has joined #openstack-trove | 07:35 | |
| openstackgerrit | zouyee proposed openstack/trove-integration: mysql version in datastore change updates https://review.openstack.org/180864 | 07:40 |
| *** haomaiwang has quit IRC | 08:01 | |
| *** haomaiwa_ has joined #openstack-trove | 08:02 | |
| *** imandhan has quit IRC | 08:02 | |
| *** haomaiwa_ has quit IRC | 09:01 | |
| *** haomaiwa_ has joined #openstack-trove | 09:01 | |
| *** openstackgerrit has quit IRC | 09:16 | |
| *** openstackgerrit has joined #openstack-trove | 09:16 | |
| *** haomaiwa_ has quit IRC | 10:01 | |
| *** haomaiwa_ has joined #openstack-trove | 10:02 | |
| *** tosky has joined #openstack-trove | 10:22 | |
| *** caishan has quit IRC | 10:28 | |
| *** tellesnobrega_af is now known as tellesnobrega | 10:34 | |
| *** haomaiwa_ has quit IRC | 11:01 | |
| *** haomaiwang has joined #openstack-trove | 11:01 | |
| *** ViswaV_ has quit IRC | 11:05 | |
| *** _amrith_ is now known as amrith | 11:06 | |
| *** ViswaV has joined #openstack-trove | 11:08 | |
| *** rcernin is now known as rcernin|lunch | 11:13 | |
| amrith | pmalik, re: https://review.openstack.org/#/c/248421/ I don't quite understand the change. One question that I'm interested in is the one that cp16net asked. will catch you later this AM. I want to know what the race in fake mode is first ... | 11:17 |
| *** amrith is now known as _amrith_ | 11:37 | |
| *** haomaiwang has quit IRC | 12:01 | |
| *** haomaiwa_ has joined #openstack-trove | 12:02 | |
| *** tellesnobrega is now known as tellesnobrega_af | 12:19 | |
| *** tellesnobrega_af is now known as tellesnobrega | 12:25 | |
| *** rcernin|lunch is now known as rcernin | 12:36 | |
| *** haomaiwa_ has quit IRC | 13:01 | |
| *** haomaiwang has joined #openstack-trove | 13:01 | |
| *** pmackinn has joined #openstack-trove | 13:04 | |
| *** pmackinn has quit IRC | 13:07 | |
| *** pmackinn has joined #openstack-trove | 13:08 | |
| *** haomaiwang has quit IRC | 13:11 | |
| *** PeterSchwaller has joined #openstack-trove | 13:14 | |
| *** PeterS has quit IRC | 13:18 | |
| *** khushbu_ has joined #openstack-trove | 13:20 | |
| *** khushbu_ has quit IRC | 13:43 | |
| *** _crobertsrh is now known as crobertsrh | 13:48 | |
| *** haomaiwang has joined #openstack-trove | 13:49 | |
| *** PeterS has joined #openstack-trove | 13:58 | |
| *** haomaiwang has quit IRC | 14:01 | |
| *** tosky_ has joined #openstack-trove | 14:01 | |
| *** PeterSchwaller has quit IRC | 14:01 | |
| *** haomaiwa_ has joined #openstack-trove | 14:01 | |
| *** tosky has quit IRC | 14:03 | |
| *** rcernin has quit IRC | 14:03 | |
| *** caishan has joined #openstack-trove | 14:18 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove: Updated from global requirements https://review.openstack.org/245732 | 14:47 |
| *** tosky_ is now known as tosky | 14:48 | |
| *** mingdang1 has quit IRC | 14:55 | |
| *** haomaiwa_ has quit IRC | 15:01 | |
| *** haomaiwa_ has joined #openstack-trove | 15:01 | |
| *** PeterSchwaller has joined #openstack-trove | 15:23 | |
| *** PeterS has quit IRC | 15:24 | |
| *** haomaiwa_ has quit IRC | 15:49 | |
| openstackgerrit | Peter Stachowski proposed openstack/trove: Fix race condition in replication backup delete https://review.openstack.org/248421 | 16:10 |
| *** PeterS has joined #openstack-trove | 16:12 | |
| *** PeterSchwaller_ has joined #openstack-trove | 16:13 | |
| *** PeterSchwaller has quit IRC | 16:14 | |
| *** rcernin has joined #openstack-trove | 16:16 | |
| *** PeterS has quit IRC | 16:16 | |
| *** caishan has quit IRC | 16:19 | |
| *** _amrith_ is now known as amrith | 16:26 | |
| *** sabeen1 has joined #openstack-trove | 16:49 | |
| *** tosky has quit IRC | 16:53 | |
| *** kbyrne has quit IRC | 16:53 | |
| *** tosky has joined #openstack-trove | 16:54 | |
| *** david-ly_ has joined #openstack-trove | 17:19 | |
| *** david-ly_ has quit IRC | 17:23 | |
| *** david-l__ has joined #openstack-trove | 17:23 | |
| *** david-lyle has quit IRC | 17:23 | |
| *** david-l__ has quit IRC | 17:30 | |
| *** david-lyle has joined #openstack-trove | 17:44 | |
| *** ashleighfarnham has joined #openstack-trove | 17:46 | |
| openstackgerrit | Petr Malik proposed openstack/trove: Experimental datastores use new service commands https://review.openstack.org/245399 | 17:48 |
| *** imandhan has joined #openstack-trove | 18:01 | |
| *** david-lyle has quit IRC | 18:03 | |
| mvandijk_ | cp16net: yt? | 18:03 |
| *** david-lyle has joined #openstack-trove | 18:11 | |
| *** ViswaV has quit IRC | 18:12 | |
| *** ViswaV has joined #openstack-trove | 18:13 | |
| cp16net | mvandijk_: yes | 18:13 |
| cp16net | whats up | 18:13 |
| cp16net | pmackinn: take a peek at this one. https://review.openstack.org/#/c/243749/ | 18:14 |
| imandhan | hey guys i'm trying to force delete backups stuck in build stage without having to unstack and reinstall redstack. Anyone know how to? | 18:16 |
| cp16net | imandhan: update the record in the db to error | 18:16 |
| imandhan | can you do that manually? | 18:16 |
| cp16net | yup | 18:17 |
| cp16net | its just reading the db | 18:17 |
| cp16net | so updating it will be ok | 18:17 |
| vkmc | cp16net, we need to create some sort of garbage collector for those cases | 18:17 |
| vkmc | someday | 18:17 |
| cp16net | and i always suggest setting backups that are "stuck" to error and not complete | 18:17 |
| pmackinn | cp16net, lgtm +1 | 18:18 |
| cp16net | pmackinn: thanks :) | 18:18 |
| imandhan | I've never manually changed status, can you point me to some resource for that? | 18:18 |
| cp16net | vkmc: we need to know why that happens and not let it happen | 18:18 |
| vkmc | yeah | 18:19 |
| cp16net | mysql; select * from backups; | 18:19 |
| cp16net | update backups set status='error' where id='uuid'; | 18:19 |
| cp16net | something like that imandhan | 18:19 |
| imandhan | but that's just for mysql ones, isn't it? | 18:20 |
| vkmc | imandhan, no, he is talking about the management database | 18:20 |
| cp16net | all the backups are in there | 18:20 |
| vkmc | in the control plane | 18:20 |
| imandhan | ahhh okay | 18:20 |
| cp16net | yea in the mysql db where everything lives for trove | 18:20 |
| cp16net | i guess doing mysql trove; is better :-P | 18:20 |
| imandhan | cool cool, will give it a shot | 18:20 |
| cp16net | i think its "update backups set state="FALIED" where id='backup_uuid';" | 18:22 |
| cp16net | that should get your instance out of a backup state as well | 18:23 |
| imandhan | okay, perfect! | 18:23 |
| cp16net | :) | 18:23 |
| imandhan | I didn't have any problems deleting instances even though they were in backup state... | 18:23 |
| imandhan | which I initially had expected to be an issue | 18:23 |
| cp16net | i think you would if it was in horizon | 18:24 |
| imandhan | ohk I was using cl | 18:24 |
| *** PeterSchwaller_ has quit IRC | 18:26 | |
| *** PeterS has joined #openstack-trove | 18:26 | |
| *** tosky has quit IRC | 18:26 | |
| openstackgerrit | Ishita Mandhan proposed openstack/trove: Addition of DB2 backup & restore functionality https://review.openstack.org/246709 | 18:26 |
| amrith | cp16net, yt? | 18:41 |
| cp16net | amrith: yup | 18:44 |
| mvandijk_ | cp16net: I replied to your concerns on https://review.openstack.org/#/c/247744/3 | 18:56 |
| mvandijk_ | the gist being that those exceptions are saying "we shouldnt be trying to do this... api/tm should know better" | 18:57 |
| cp16net | right but my thought it thought since those calls are async there is nothing to handle the exception other than logging it in the guestagent | 18:59 |
| *** tellesnobrega is now known as tellesnobrega_af | 19:03 | |
| *** PeterSchwaller has joined #openstack-trove | 19:03 | |
| *** PeterS has quit IRC | 19:03 | |
| *** arist has quit IRC | 19:15 | |
| *** arist has joined #openstack-trove | 19:16 | |
| *** sabeen1 has quit IRC | 19:41 | |
| mvandijk_ | so no different to just logging it | 19:57 |
| mvandijk_ | but if we one day change things like create user to a call (no idea why it isn't already...) then it will be useful | 19:57 |
| mvandijk_ | or if we add some capabilities to show messages from the guest like nova does | 19:57 |
| *** amrith is now known as _amrith_ | 20:16 | |
| openstackgerrit | Merged openstack/trove: Update internal status when waiting for a change https://review.openstack.org/243247 | 20:36 |
| *** georgelorch has quit IRC | 20:38 | |
| *** georgelorch has joined #openstack-trove | 20:39 | |
| *** crobertsrh1 has joined #openstack-trove | 20:53 | |
| cp16net | mvandijk_: yt? | 21:33 |
| *** sabeen1 has joined #openstack-trove | 21:39 | |
| mvandijk_ | yes | 21:40 |
| mvandijk_ | cp16net: am now | 21:40 |
| cp16net | ok have a final comment on the exceptions on that review | 21:42 |
| *** sabeen3 has joined #openstack-trove | 21:43 | |
| *** sabeen1 has quit IRC | 21:43 | |
| cp16net | mvandijk_: so raising an exception thats just logged in the guest with a traceback leads to believe something broke | 21:44 |
| mvandijk_ | thats the point. if you get there, something in the taskmanager or api is wrong | 21:45 |
| mvandijk_ | it shouldn't make that request to the guest | 21:45 |
| cp16net | but this didnt help the validation in taskmanager | 21:46 |
| mvandijk_ | eg https://github.com/openstack/trove/blob/master/trove/guestagent/datastore/mysql_common/service.py#L365-L368 | 21:46 |
| mvandijk_ | we use this pattern for mysql too | 21:47 |
| cp16net | i'm not strongly against the change how you did it just asking if there is a cleaner way | 21:47 |
| mvandijk_ | the taskmanager has that validation. | 21:47 |
| mvandijk_ | the bug was smaller in scope than the change, it prompted me to add more protection against changing the reserved users | 21:48 |
| mvandijk_ | the exceptions are there to tell us we did something wrong in trove | 21:48 |
| cp16net | yeah thats true | 21:49 |
| cp16net | ok i think it makes sense to me now | 21:50 |
| cp16net | thanks | 21:50 |
| mvandijk_ | excellent :) | 21:51 |
| mvandijk_ | saves me pushing up a new patchset | 21:51 |
| *** tornopil has joined #openstack-trove | 21:58 | |
| *** sgotliv has quit IRC | 22:02 | |
| *** _amrith_ is now known as amrith | 22:03 | |
| *** ashleighfarnham has quit IRC | 22:07 | |
| *** tornopil has quit IRC | 22:08 | |
| *** tornopil has joined #openstack-trove | 22:18 | |
| tornopil | hi, I get this error from trove-guestagent.log https://gist.github.com/anonymous/0c92b0e9f04d271af956 and then mysql stops, I use kilo ubuntu | 22:20 |
| *** crobertsrh1 has quit IRC | 22:22 | |
| tornopil | does anyone knows about the error I get? | 22:25 |
| cp16net | tornopil: so the volume trove attaches to the guest is causing a timeout on unmount | 22:25 |
| tornopil | cp16net, but the volume is already unmounted when I ssh into the guest machine | 22:26 |
| tornopil | could you give me hint about where should I look to solve it? | 22:27 |
| cp16net | tornopil: does the user that the guestagent run as have permissions to call sudo unmount? | 22:28 |
| tornopil | cp16net, yes it does | 22:28 |
| *** tornopil has quit IRC | 22:30 | |
| cp16net | hmm strange | 22:30 |
| *** tornopil has joined #openstack-trove | 22:31 | |
| cp16net | so trove syncs the data from /var/lib/mysql to /mnt/volume (where this is just a temp mount place to sync the files) | 22:31 |
| cp16net | then unmounts it from there and mounts to /var/lib/mysql | 22:32 |
| cp16net | so that then mysql can start up | 22:32 |
| cp16net | if you manually remount the volume do you see the files sync'd to it? | 22:33 |
| tornopil | cp16net, I can see the files insde /mnt/volume after mounting is identical to /var/lib/mysql | 22:34 |
| cp16net | sounds like the umount cmd is not sending a EOF | 22:34 |
| cp16net | i've not heard of this issue before | 22:35 |
| cp16net | if you manually then "sudo umount /mnt/volume" after changed to the trove-user does it hang or complete like normal? | 22:36 |
| cp16net | it sounds like the issue is something to do with calling umount from the sound of it | 22:36 |
| cp16net | because everything else sounds like it worked. | 22:37 |
| tornopil | cp16net, it does unmount the volume seccessfully | 22:37 |
| cp16net | the mount call even waits for an EOF and that sounds like it worked | 22:37 |
| cp16net | but umount hung | 22:38 |
| cp16net | does this happen if you try again? | 22:38 |
| tornopil | actually I've done it many times, same every time | 22:39 |
| cp16net | ummm i running out of ideas | 22:41 |
| cp16net | wonder if running umount to quickly and causing a volume busy error? | 22:42 |
| tornopil | if you plz tell me how to check it | 22:43 |
| cp16net | is there anything in /var/log/syslog? | 22:45 |
| cp16net | that might help why umount failed? | 22:45 |
| tornopil | I don't see anything related to umount last are these | 22:49 |
| tornopil | kernel: [ 4446.918863] EXT4-fs (vdb): mounting ext3 file system using the ext4 subsystem | 22:49 |
| tornopil | [ 4446.986107] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null) | 22:49 |
| cp16net | Nov 24 17:29:45 test-pxc-member-1 kernel: [ 12.763575] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) | 22:52 |
| cp16net | ok looks similar to mine | 22:52 |
| cp16net | is it using the right mount point? | 22:52 |
| cp16net | err device | 22:53 |
| tornopil | yes the device is /dev/vdb | 22:53 |
| cp16net | ok | 22:53 |
| cp16net | tornopil: i might suggest trying to build a new image then | 22:56 |
| cp16net | for mysql | 22:57 |
| cp16net | and see if that helps | 22:57 |
| tornopil | cp16net, ok thanks, I've created one with ubuntu trusty cloud img and mariadb and trove-guestagent install in it with chrooting, | 22:58 |
| tornopil | is it correct way to go with? | 22:58 |
| tornopil | do you suggest any other approch | 22:59 |
| cp16net | that should work but we usually use the image elements to build an image | 23:00 |
| tornopil | cp16net, I've had some problems with image elemnts, with images created with them the guest_info would be empty | 23:01 |
| cp16net | tornopil: amrith wrote an article on building a guest image http://www.dbta.com/BigDataQuarterly/Articles/Building-a-database-guest-image-for-OpenStack-Trove-107368.aspx | 23:02 |
| tornopil | cp16net, thank you for the link | 23:02 |
| cp16net | well the guest_info should be injected at boot time of the instance | 23:03 |
| tornopil | cp16net, I know it does sound strange, but it won't with them, I'll give another try, maybe with centos image | 23:04 |
| tornopil | thanks | 23:04 |
| cp16net | tornopil: let us know if you get it working | 23:05 |
| cp16net | i'm sure many of us trove-sters will be on vacay this week tho for the holidays | 23:05 |
| cp16net | might be quiet around here for the rest of this week | 23:06 |
| cp16net | (i'm off the rest of the week but i might check in here and there on irc.) | 23:06 |
| dougshelley66 | cp16net us Canadians will still be working... | 23:06 |
| cp16net | :-P | 23:07 |
| cp16net | oh thats true. there will be a few people around :-P | 23:07 |
| *** tornopil has quit IRC | 23:09 | |
| cp16net | dougshelley66: have you heard of a volume not unmounting in the prepare call? | 23:09 |
| dougshelley66 | one sec | 23:09 |
| dougshelley66 | was just reading the scroll back - i haven't seen that either | 23:11 |
| cp16net | yeah i didnt know what else to suggest for debugging | 23:12 |
| dougshelley66 | i assume something was using the volume - lsof would help but only if you could run it before the umount timed out | 23:12 |
| *** johnma has quit IRC | 23:12 | |
| cp16net | yeah | 23:13 |
| cp16net | and using the ubuntu kilo install it might not be a good idea to change things :-P | 23:13 |
| *** tornopil has joined #openstack-trove | 23:16 | |
| *** sabeen3 has quit IRC | 23:23 | |
| *** johnma has joined #openstack-trove | 23:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove: Updated from global requirements https://review.openstack.org/245732 | 23:32 |
| *** johnma has quit IRC | 23:32 | |
| openstackgerrit | Merged openstack/trove: Do not use api-paste.ini osprofiler options https://review.openstack.org/245081 | 23:38 |
| *** rcernin has quit IRC | 23:42 | |
| *** tornopil has quit IRC | 23:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!