| *** imandhan has quit IRC | 00:00 | |
| *** imandhan has joined #openstack-trove | 00:01 | |
| *** imandhan has quit IRC | 00:06 | |
| *** tqtran has quit IRC | 00:25 | |
| *** chlong has quit IRC | 00:32 | |
| *** gouthamr_ has joined #openstack-trove | 00:44 | |
| mmasaki | I tried to fix it. https://bugs.launchpad.net/trove/+bug/1485789 | 01:48 |
|---|---|---|
| openstack | Launchpad bug 1485789 in OpenStack DBaaS (Trove) "ubuntu systemd" [High,Triaged] - Assigned to Nikhil Manchanda (slicknik) | 01:48 |
| mmasaki | However, I found trove-integration doesn't work in Ubuntu 16.04. | 01:48 |
| mmasaki | There are several problems not only service scripts. | 01:49 |
| mmasaki | How can we fix them? Now we have trove-image-builder. trove-integration will be maintained after Newton? | 01:51 |
| mmasaki | Let me know I should fix trove-integration to work on Ubuntu 16.04, or I should fix the problem in another way. | 01:54 |
| *** songjian has joined #openstack-trove | 01:59 | |
| songjian | amrith,I have a question, it/trove/run_tests py line203 import_tests () the function, can I delete? Because I see the back of the import are commented out | 02:07 |
| openstackgerrit | jian.song proposed openstack/trove: Delete #noqa https://review.openstack.org/346720 | 02:13 |
| *** gouthamr_ has quit IRC | 02:29 | |
| *** itisha has quit IRC | 02:40 | |
| *** pmackinn has quit IRC | 02:52 | |
| *** songjian has quit IRC | 03:06 | |
| *** songjian has joined #openstack-trove | 03:10 | |
| *** chlong has joined #openstack-trove | 03:49 | |
| *** imandhan has joined #openstack-trove | 03:51 | |
| *** rcernin has joined #openstack-trove | 05:31 | |
| *** links has joined #openstack-trove | 05:36 | |
| *** songjian has quit IRC | 05:42 | |
| *** ubuntu_ has joined #openstack-trove | 05:42 | |
| *** Ravikiran_K has joined #openstack-trove | 06:06 | |
| *** chlong has quit IRC | 06:08 | |
| *** ubuntu__ has joined #openstack-trove | 06:13 | |
| *** ubuntu_ has quit IRC | 06:16 | |
| *** diga has joined #openstack-trove | 06:34 | |
| *** chlong has joined #openstack-trove | 06:40 | |
| *** chlong has quit IRC | 06:52 | |
| *** flaper87 has joined #openstack-trove | 07:36 | |
| *** flaper87 is now known as Guest73548 | 07:36 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-trove | 08:03 | |
| *** tesseract- has joined #openstack-trove | 08:16 | |
| *** ubuntu_ has joined #openstack-trove | 08:29 | |
| *** ubuntu__ has quit IRC | 08:33 | |
| *** Guest73548 is now known as flaper87 | 08:39 | |
| *** flaper87 is now known as Guest78641 | 08:39 | |
| *** Guest78641 has quit IRC | 08:44 | |
| *** imandhan has quit IRC | 08:47 | |
| *** imandhan has joined #openstack-trove | 08:48 | |
| *** imandhan has quit IRC | 08:52 | |
| *** ubuntu__ has joined #openstack-trove | 08:55 | |
| *** ubuntu_ has quit IRC | 08:59 | |
| *** itisha has joined #openstack-trove | 09:01 | |
| *** flaper87 has joined #openstack-trove | 09:40 | |
| *** flaper87 has quit IRC | 09:40 | |
| *** flaper87 has joined #openstack-trove | 09:40 | |
| *** tosky has joined #openstack-trove | 10:43 | |
| *** dasTor has quit IRC | 10:43 | |
| *** Ravikiran_K has quit IRC | 10:58 | |
| *** gouthamr has joined #openstack-trove | 11:43 | |
| *** catintheroof has quit IRC | 12:53 | |
| *** ubuntu__ has quit IRC | 12:56 | |
| *** catintheroof has joined #openstack-trove | 13:05 | |
| *** diga has quit IRC | 13:07 | |
| *** cdelatte has joined #openstack-trove | 13:30 | |
| *** itisha has quit IRC | 13:30 | |
| *** Ravikiran_K has joined #openstack-trove | 13:33 | |
| *** catintheroof has quit IRC | 14:00 | |
| *** openstackgerrit has quit IRC | 14:03 | |
| *** openstackgerrit has joined #openstack-trove | 14:03 | |
| *** catintheroof has joined #openstack-trove | 14:07 | |
| *** cdelatte has quit IRC | 14:16 | |
| *** pmackinn has joined #openstack-trove | 14:19 | |
| openstackgerrit | Merged openstack/trove: Fix for tcp_ports configuration in devstack. https://review.openstack.org/350228 | 14:22 |
| *** hoonetorg has quit IRC | 14:57 | |
| *** hoonetorg has joined #openstack-trove | 15:09 | |
| openstackgerrit | Ali Asgar Adil proposed openstack/trove-dashboard: Add support for instance datastore-flavors https://review.openstack.org/286159 | 15:16 |
| *** tesseract- has quit IRC | 15:21 | |
| *** trevormc has joined #openstack-trove | 15:24 | |
| *** rcernin has quit IRC | 15:26 | |
| *** itisha has joined #openstack-trove | 15:26 | |
| *** links has quit IRC | 15:33 | |
| *** gouthamr has quit IRC | 15:34 | |
| *** gouthamr has joined #openstack-trove | 15:34 | |
| *** davideagnello has joined #openstack-trove | 16:04 | |
| *** pcaruana has quit IRC | 16:05 | |
| *** davideagnello has quit IRC | 16:12 | |
| *** gregragenet has joined #openstack-trove | 16:12 | |
| *** davideagnello has joined #openstack-trove | 16:13 | |
| openstackgerrit | Matthew Van Dijk proposed openstack/trove: Improve guestagent datastore models https://review.openstack.org/236082 | 17:04 |
| *** itlinux has quit IRC | 17:15 | |
| *** davideagnello has quit IRC | 17:20 | |
| gregragenet | If any devs are online which are familiar w/ the file injection mechanism - I would like to ask some questions rather than hash it out in comments in a bug. I've read the code, and from what I can tell file injection wouldn't work if using the default values for Trove and Nova in Mitaka. | 17:23 |
| *** tosky has quit IRC | 17:27 | |
| *** davideagnello has joined #openstack-trove | 17:27 | |
| amrith | ask away | 17:32 |
| gregragenet | re: my comment on https://bugs.launchpad.net/trove/+bug/1609915/comments/3 - as best I can tell Trove depends on nova to inject the files through libvirt, but the default libvirt setting is to disable injection. Is there some other mechanism in the mix that I missed when reading through the trove and nova code? | 17:34 |
| openstack | Launchpad bug 1609915 in OpenStack DBaaS (Trove) "guest_id injection in guest is undocumented" [Undecided,Invalid] | 17:34 |
| amrith | trove uses config drive | 17:35 |
| amrith | and when it runs, the two injected files end up in /etc/trove/conf.d | 17:36 |
| gregragenet | The default setting in trove to enable config_drive is false. https://github.com/openstack/trove/blob/97a2fb89ff38c61e2f45cf7b4e4eb4657621fa87/trove/common/cfg.py#L194 | 17:36 |
| amrith | you are correct. We pass the files parameter to nova servers.create() and they get to the guest instance | 17:39 |
| amrith | It is my assumption that nova does whatever it needs to to get the files there | 17:41 |
| amrith | and I can see that they do in fact show up there | 17:41 |
| amrith | furthermore, in debugging something recently I did in fact find that it was using config-drive under the covers | 17:42 |
| amrith | so, you've got me on that; I don't know how nova does things | 17:42 |
| amrith | we just get our files there ... | 17:42 |
| gregragenet | Does your nova.conf have an 'inject_partition' setting that's something other than '-2'? -2 is the default, which disables file injection. | 17:43 |
| amrith | grep inject nova.conf | 17:44 |
| amrith | inject_partition = -2 | 17:44 |
| *** arist has quit IRC | 17:46 | |
| amrith | on your guest instance; are you able to ssh into it? | 17:46 |
| amrith | if you are, d | 17:47 |
| amrith | do this | 17:47 |
| amrith | cd /var/lib/cloud/data/openstack | 17:47 |
| *** arist has joined #openstack-trove | 17:47 | |
| gregragenet | I'll hop on it and check. | 17:47 |
| gregragenet | I have a /var/lib/cloud/data directory - but not a /var/lib/cloud/data/openstack directory. The image is based on a Trusty cloud image. | 17:50 |
| amrith | ok, one second, my image is xenial based but I doubt that's the issue; let me check | 17:53 |
| amrith | gregragenet, go to /var/log/ and look for cloud-init.log | 17:56 |
| amrith | you should have one | 17:56 |
| amrith | wait, is this a guest with your customized cloud-init file or something else? | 17:56 |
| gregragenet | I can look in cloud-init.log - what should I look for? - this is the instance using the cloudinit file that's basically the same as the one in https://review.openstack.org/#/c/351364/1 | 17:58 |
| gregragenet | a few small changes to it - putting in my appropriate openstack settings and adding a debug account so I can login to it. | 17:59 |
| amrith | in that file you should see a line like this | 18:00 |
| amrith | ubuntu [CLOUDINIT] util.py[DEBUG]: Writing to /etc/trove/conf.d/guest_info.conf | 18:00 |
| gregragenet | also, for completeness, here's my settings for trove re: server_volume and use_heat: | 18:01 |
| amrith | you are using heat? | 18:01 |
| gregragenet | egrep -r "^use_heat|^use_nova_server_volume" /etc/trove/* | 18:01 |
| gregragenet | /etc/trove/trove-taskmanager.conf:use_nova_server_volume = False | 18:01 |
| amrith | look for a line in cloud-init.log that reads: ubuntu [CLOUDINIT] util.py[DEBUG]: Running command ['mount', '-o', 'ro,sync', '-t', 'auto', | 18:01 |
| amrith | the next parameter is a config drive devie | 18:02 |
| amrith | device | 18:02 |
| amrith | what is that device? | 18:02 |
| vkmc | 236541 | 18:02 |
| amrith | should be /dev/sr0 | 18:02 |
| gregragenet | I will look.. Is your trove*conf set up w/ use_nova_server_volume = True ? | 18:03 |
| amrith | no setting, whatever the default is | 18:03 |
| *** ShaikApsar has joined #openstack-trove | 18:04 | |
| gregragenet | I just grepped through cloud-init.log for 'mount' - no mount calls. | 18:06 |
| gregragenet | and just tried mounting /dev/sr0 manually, and the device does not exist. | 18:07 |
| gregragenet | here's the config_drive settings in my nova - which I believe are the defaults: grep config_drive /etc/nova/* | 18:10 |
| gregragenet | /etc/nova/nova.conf:#config_drive_skip_versions=1.0 2007-01-19 2007-03-01 2007-08-29 2007-10-10 2007-12-15 2008-02-01 2008-09-01 | 18:10 |
| gregragenet | /etc/nova/nova.conf:#config_drive_format=iso9660 | 18:10 |
| gregragenet | /etc/nova/nova.conf:#force_config_drive=false | 18:10 |
| gregragenet | /etc/nova/nova.conf:#config_drive_inject_password=false | 18:10 |
| gregragenet | /etc/nova/nova.conf:#config_drive_cdrom=false | 18:10 |
| gregragenet | /etc/nova/policy.json: "compute_extension:config_drive": "rule:admin_or_owner", | 18:10 |
| *** gouthamr has quit IRC | 18:10 | |
| amrith | how did you setup your nova and trove? are they devstack based or other configuration tools based? | 18:14 |
| amrith | on my setup, nova does have that setting you pointed to (inject_partitions) set to -2. yet I do see a dev/sr0 | 18:16 |
| gregragenet | RPM packages for Mitaka on CentOS7, then Ansible customizations starting with the install-guide guidance at http://docs.openstack.org/mitaka/install-guide-rdo/ - and then any changes which were required to get services working. | 18:16 |
| gregragenet | I believe that /dev/sr0 would be from configdrive. | 18:17 |
| amrith | so, I have (in nova.conf) | 18:17 |
| amrith | nova.conf:force_config_drive = True | 18:17 |
| gregragenet | the default for that is False. https://github.com/openstack/nova/blob/d0905df10a48212950c0854597a2df923e6ddd0c/nova/conf/configdrive.py#L49 | 18:18 |
| *** gouthamr has joined #openstack-trove | 18:18 | |
| amrith | yes, so mine uses config drive. | 18:19 |
| amrith | so, with your proposed change, how does your cloud-init file make its way over to the guest? | 18:20 |
| gregragenet | btw, random factoid, looks like you can't do libvirt live migrations w/ the default config_drive_format (iso9660)... but you can if switched to 'vfat'. | 18:20 |
| *** davideagnello has quit IRC | 18:21 | |
| ShaikApsar | https://github.com/openstack/trove/blob/master/etc/trove/trove-taskmanager.conf.sample#L64 | 18:24 |
| gregragenet | the cloudinit is passed into the nova create call as 'userdata'. That userdata makes its way into the metadata service (haven't looked at the plumbing here). And then when the guest boots and starts cloud-init, which makes a http req to the metadata server using the special ip 169.254.169.254 - it pulls down that cloudconfig file parses it and takes the appropriate actions. | 18:25 |
| amrith | so you are passing your file on the same conduit that is being used for the injected_files taht Trove is trying to send anyway | 18:26 |
| amrith | on your guest, do you have an /etc/trove/conf.d directory? | 18:27 |
| amrith | since you are getting it from the special metadata IP, you should also see on that server, the meta_data.json file | 18:28 |
| amrith | which should include the two injected files | 18:28 |
| gregragenet | Lemme look for the meta_data.json, but I feel like I checked for that and didn't see anything trove related. I'll double-check. | 18:29 |
| ShaikApsar | amrith: Could you please review the change https://review.openstack.org/#/c/340022/ | 18:29 |
| amrith | well, config-drive or metadata-ip are merely conduits; eventually the stuff is processed in the same way in cloud-init | 18:29 |
| ShaikApsar | amrith: This change is for task Manager.publish_quota_notifications failure | 18:30 |
| *** davideagnello has joined #openstack-trove | 18:30 | |
| amrith | yes, let me check. | 18:31 |
| gregragenet | in the nova_client.servers.create files and userdata are passed in via different arguments (files and userdata). checking into my VM for meta_data.json | 18:31 |
| amrith | ShaikApsar, I was confused by the novaclient changes that I see in remote.py (https://review.openstack.org/#/c/340022/30/trove/common/remote.py) | 18:32 |
| ShaikApsar | amrith: http://logs.openstack.org/22/340022/30/check/gate-trove-scenario-functional-dsvm-mysql/f50e3a8/logs/screen-tr-tmgr.txt.gz | 18:33 |
| ShaikApsar | amrith: 2016-08-04 21:15:09.057 24042 DEBUG oslo_service.periodic_task [-] Running periodic task Manager.publish_quota_notifications run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215 | 18:33 |
| ShaikApsar | amrith: Initially i thought issue is with nova client and later i came to know that we were trying to get tenant list using nova client | 18:34 |
| ShaikApsar | amrith: we should use keystone client to get tenant list and i removed all the novaclient changes from my previous patches | 18:34 |
| ShaikApsar | amrith: nova_client = remote.create_nova_client(self.admin_context) is the wrong | 18:35 |
| ShaikApsar | amrith: https://review.openstack.org/#/c/340022/30/trove/taskmanager/manager.py | 18:36 |
| gregragenet | this is what's in meta_data.json: | 18:38 |
| * amrith waits eagerly | 18:38 | |
| gregragenet | root@webui-trove:~# curl http://169.254.169.254/openstack/latest/meta_data.json 2>/dev/null |python -c 'import json,sys;obj=json.load(sys.stdin);print obj.keys();' | 18:38 |
| gregragenet | [u'random_seed', u'uuid', u'availability_zone', u'hostname', u'launch_index', u'project_id', u'name'] | 18:38 |
| ShaikApsar | amrith: 'for tenant in nova_client.tenants.list():' this line is wrong | 18:39 |
| gregragenet | the uuid is the uuid for the instance.. | 18:39 |
| amrith | that's all correct | 18:39 |
| amrith | looks like the stuff that trove is providing isn't making it over | 18:40 |
| amrith | mind changing the nova flag (force_config_drive = True) and restarting the nova services and retrying? | 18:40 |
| amrith | without your cloud-init change | 18:40 |
| amrith | I'm doing the opposite; removing the force_config_drive = True in nova.conf and retrying | 18:41 |
| gregragenet | I would suspect it would work if I make that change.. ATM the environment I have that I can do that on is a cluster that some other team members are using. so it'll take a little bit for me to make the change. | 18:48 |
| *** catintheroof has quit IRC | 19:00 | |
| *** fifieldt has quit IRC | 19:02 | |
| *** ShaikApsar has quit IRC | 19:09 | |
| *** fifieldt has joined #openstack-trove | 19:18 | |
| *** davideagnello has quit IRC | 19:33 | |
| *** ShaikApsar has joined #openstack-trove | 19:34 | |
| *** davideagnello has joined #openstack-trove | 19:34 | |
| amrith | and I roached my environment in trying to kill nova; trying to rebuild | 19:43 |
| gregragenet | I'm trying use_nova_server_volume = True in my trove conf, similar (not exact) operation path, but without me having to coordinate restart of nova across my cluster. | 19:45 |
| amrith | ok | 19:46 |
| amrith | does force_config_drive require more of a restart? | 19:46 |
| gregragenet | I wasn't sure if that required restart of all nova-compute instances, or of just the controller side ones. either way, there's other users of nova on this cluster, I'm the only one using trove, so I don't have to coordinate w/ anyone to restart trove services. | 19:49 |
| amrith | :) | 19:50 |
| amrith | my usual explanation is 'testing high availability' | 19:50 |
| *** trevormc has quit IRC | 20:01 | |
| gregragenet | oops wrong param. I need to change use_nova_server_config_drive in trove. trying again. | 20:02 |
| gregragenet | use_nova_server_config_drive=True injects /etc/trove/conf.d/guest_info.conf into the VM correctly. | 20:07 |
| openstackgerrit | Merged openstack/trove-dashboard: Imported Translations from Zanata https://review.openstack.org/345856 | 20:08 |
| gregragenet | So perhaps that should default to True in https://github.com/openstack/trove/blob/97a2fb89ff38c61e2f45cf7b4e4eb4657621fa87/trove/common/cfg.py#L194 ? - given that nova defaults to not automatically creating a config drive unless the nova client (trove) requests it. | 20:13 |
| amrith | gregragenet, that seems to make sense | 20:14 |
| amrith | with that, does the code on master work for you? | 20:15 |
| *** davideagnello has quit IRC | 20:22 | |
| gregragenet | It works w/ Mitaka, so I'd assume it also works w/ master as the default config settings for Nova and Trove related to config drive have not changed since Mitaka. | 20:23 |
| gregragenet | I have to drop off.. meeting w/ the Hadoop guys to deliver the bad news that I don't have Sahara up yet. Thank you for all the time and help debugging this. | 20:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/trove: Updated from global requirements https://review.openstack.org/350933 | 20:29 |
| *** davideagnello has joined #openstack-trove | 20:31 | |
| *** gouthamr has quit IRC | 20:41 | |
| openstackgerrit | Merged openstack/python-troveclient: Add vCPUs to flavor-list https://review.openstack.org/337533 | 20:56 |
| *** gouthamr has joined #openstack-trove | 21:05 | |
| *** Ravikiran_K has quit IRC | 21:21 | |
| openstackgerrit | Merged openstack/trove: Introduce "icmp" option for security group rule https://review.openstack.org/214056 | 21:28 |
| openstackgerrit | Merged openstack/trove: Fix config registration for Trove tempest plugin https://review.openstack.org/336710 | 21:37 |
| *** davideagnello has quit IRC | 21:40 | |
| amrith | gregragenet, thanks for your help getting to the bottom of this. | 21:45 |
| amrith | gregragenet, you are correct; my question re: works with master was to say that it works without your proposed change to cloud-init etc., | 21:46 |
| *** davideagnello has joined #openstack-trove | 21:52 | |
| gregragenet | yes. we can drop the cloud init change - it is unnecessary assuming the docs make the injection mechanisms incl. the config_drive setting, and having that default to True. I'd really like there to be an explicit error message when trove-guestagent starts up and guest_id is Null - that cost me a lot of time troubleshooting, and I think it may affect a lot of users if their guestimages are misconfigured such that they don't read in the correct | 21:56 |
| gregragenet | injected files. | 21:57 |
| amrith | good points, I'll update the bug and re-open it | 21:58 |
| *** ShaikApsar has quit IRC | 22:03 | |
| amrith | gregragenet, would you please resubmit your change with the change to the default; use_nova_server_config_drive = True and let's get the CI to run through it. Also a similar change in trove-integration to establish it for the future. thx | 22:06 |
| amrith | also any additional error message changes in guestagent; I think all of those should be fine. | 22:07 |
| gregragenet | I just abandoned my cloudinit pr... but i'll start a new one.. it may take me a bit to find the correct spot to detect and error out that the guest_id is missing. | 22:08 |
| amrith | I'll post some suggestions in the bug; I'm thinking trove/cmd/guest.py would be a good place to start | 22:10 |
| gregragenet | yes. that looks like an obvious place. RuntimeError, same as if datastore_manager is missing? | 22:12 |
| amrith | yes, I'm trying it out | 22:12 |
| amrith | I'm surprised that your system didn't fail with that error | 22:13 |
| amrith | I'd assume that you would have tripped on that | 22:13 |
| amrith | since datastore_manager only comes from the injected files | 22:13 |
| amrith | yes, as best as I can tell, you should have hit that error | 22:18 |
| amrith | see: https://gist.github.com/amrith/0d993964b12ac2c266b20239feefcbbb | 22:19 |
| amrith | sorry, fat fingered it | 22:20 |
| amrith | https://gist.github.com/amrith/79eebccf6520892236446760ea3f1624 | 22:20 |
| gregragenet | I probably did get the error for datastore_manager, but then that param will always be the same value across all guests for a given datastore, so the value can be baked into the image, or injected via cloud-init. | 22:27 |
| *** gouthamr has quit IRC | 22:49 | |
| *** davideagnello has quit IRC | 22:58 | |
| *** itisha has quit IRC | 23:00 | |
| *** davideagnello has joined #openstack-trove | 23:06 | |
| *** davideagnello has quit IRC | 23:20 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!