| *** thetrav has joined #openstack-trove | 00:45 | |
| thetrav | ok, so I got a guestagent sort of up and going | 00:46 |
|---|---|---|
| thetrav | but the Mysql agent _get_actual_status appears to be raise NotImplementedError | 00:46 |
| thetrav | am I supposed to implement my own guest agents? | 00:47 |
| thetrav | oh ok | 00:59 |
| thetrav | I'm looking at a base class | 00:59 |
| thetrav | where then is the concrete? | 00:59 |
| thetrav | mysql_common | 01:00 |
| *** Administrator_ has joined #openstack-trove | 01:15 | |
| *** zhugaoxiao has quit IRC | 01:18 | |
| *** zhugaoxiao has joined #openstack-trove | 01:19 | |
| *** Administrator_ has quit IRC | 01:19 | |
| *** pmackinn has quit IRC | 01:25 | |
| thetrav | ok so it seems the guest agent is looking for a status on mysql but nothing ever calls the update function | 01:50 |
| thetrav | so my trove instance is booting up, but it's still not transitioning from the BUILD state into the next state (except ERROR when it times out) | 02:04 |
| thetrav | is there anywhere I can read up on what is supposed to trigger these state transitions? | 02:04 |
| thetrav | there's definitely a nova instance, and it's booted up and has MQ connectivity | 02:04 |
| *** itisha has quit IRC | 02:10 | |
| thetrav | ok, maybe getting closer, I've got a feeling the trove-conductor install is busted | 02:23 |
| thetrav | docs indicate it's supposed to be doing some heartbeat stuff | 02:24 |
| thetrav | its log is silent | 02:24 |
| thetrav | does the initial oslo config dump, but then nothing | 02:25 |
| thetrav | is task manager supposed to send it some message? or guests? | 02:25 |
| *** imandhan has joined #openstack-trove | 03:06 | |
| *** imandhan has quit IRC | 03:11 | |
| *** zhugx has joined #openstack-trove | 04:58 | |
| *** pcaruana has quit IRC | 05:26 | |
| *** dkehn has quit IRC | 06:19 | |
| *** dkehn_ has quit IRC | 06:19 | |
| *** dkehn has joined #openstack-trove | 06:26 | |
| *** dkehn_ has joined #openstack-trove | 06:32 | |
| thetrav | so /etc/trove/conf.d/guest_info.conf is supposed to have guest_id in it | 06:40 |
| thetrav | I assume that's either the VM ID or the Trove ID? | 06:40 |
| thetrav | either way my copy has nothing in it | 06:40 |
| thetrav | gregragenet: in case you read logs, https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py:353 | 06:43 |
| thetrav | that is where the guest_id is passed from trove-taskmanager to the guest vm, into the guest_info.conf file which I assume the agent loads | 06:44 |
| thetrav | however in my case that file is never injected into the VM | 06:44 |
| thetrav | also if any devs here can explain to me what is going on line 361? Seems a little odd | 06:47 |
| thetrav | ok, so the guest_info.conf file is passed to novaclient as the "files" param, which is a list of files to overwrite on instance boot | 07:00 |
| thetrav | so... | 07:00 |
| thetrav | that implies that the task manager creates a guest_info file somewhere while creating the new instance? | 07:00 |
| thetrav | ahh no | 07:02 |
| thetrav | it's a path => content mapping | 07:02 |
| thetrav | and I assume nova implements it with the cloud_config stuff | 07:03 |
| thetrav | which is probably blown away by my custom cloud config stuff | 07:03 |
| thetrav | hmm | 07:03 |
| *** pcaruana has joined #openstack-trove | 07:20 | |
| thetrav | ok. no, not cloud config | 08:03 |
| thetrav | personality list | 08:03 |
| thetrav | not working | 08:03 |
| thetrav | being passed to nova but not applied on the instance | 08:04 |
| amrith | thetrav, yt? | 08:08 |
| amrith | you should see the change that gregragenet proposed in https://review.openstack.org/#/c/351958/, it is likely the simplest explanation of the problem you are having. | 08:13 |
| thetrav | ta | 08:17 |
| thetrav | ok, so it looks to me like config_drive is just another option for getting guest_id in there | 08:20 |
| thetrav | I'm all up for using it if it works | 08:20 |
| thetrav | however it's a little odd that the personality list doesn't work | 08:20 |
| thetrav | ok, so config-drive gets /etc/trove/conf.d loaded | 08:29 |
| thetrav | erm | 08:29 |
| thetrav | injected | 08:29 |
| thetrav | not loaded | 08:29 |
| thetrav | which is the problem | 08:29 |
| thetrav | do I need to edit /etc/init/trove-guestagent.conf ? or is there some sort of dynamic loading supposed to be going on | 08:29 |
| *** khushbu has joined #openstack-trove | 08:34 | |
| amrith | what is /etc/init/trove-guestagent.conf? | 08:35 |
| amrith | I don't know of /etc/init/ | 08:35 |
| thetrav | service | 08:36 |
| amrith | sorry, what? | 08:36 |
| thetrav | ubuntu packaging | 08:36 |
| thetrav | it's the upstart script | 08:36 |
| amrith | you don't have to update it as best as I know of. | 08:36 |
| amrith | as far as I know, ubuntu or not, you should get a directory /etc/trove/conf.d | 08:36 |
| amrith | with two files in it | 08:36 |
| amrith | those are the two files that get injected | 08:37 |
| thetrav | yep, that has happened | 08:37 |
| amrith | and you should launch your guest agent not with the files option but the dirs option | 08:37 |
| thetrav | dirs option? | 08:37 |
| amrith | trove-guestagent --config-dir=/etc/trove/conf.d | 08:38 |
| amrith | rather than trove-guestagent --config-file=/etc/trove/conf.d/guest_info.conf | 08:38 |
| amrith | rather than trove-guestagent --config-file=/etc/trove/conf.d/guest_info | 08:38 |
| thetrav | ok | 08:38 |
| thetrav | I'll give that a shot | 08:38 |
| amrith | and to your earlier question, if the guest knows to connect to rabbit, it will have a message waiting for it. | 08:38 |
| amrith | and it'll process it and do it's thing | 08:39 |
| thetrav | ok so taskmanager / conductor send an initial message to the mq | 08:39 |
| thetrav | right so maybe config-dir works, however it's missing details like, where to put the logs | 08:39 |
| amrith | that should be in the conf file that was sent over | 08:40 |
| amrith | the task manager sends a message to the guestagent | 08:40 |
| thetrav | not according to the openstack docs | 08:40 |
| thetrav | http://docs.openstack.org/mitaka/install-guide-ubuntu/trove-install.html | 08:41 |
| amrith | ok | 08:41 |
| thetrav | I'll add it though | 08:41 |
| amrith | well, I'm going based on this: https://www.amazon.com/OpenStack-Trove-Amrith-Kumar/dp/1484212223 | 08:41 |
| thetrav | thanks | 08:42 |
| amrith | what does http://docs.openstack.org/mitaka/install-guide-ubuntu/trove-install.html not say? | 08:42 |
| amrith | that the task manager sends a message to the guest? | 08:42 |
| thetrav | the docs have a very limited set of details to put in trove-guestagent.conf | 08:43 |
| amrith | the log dir and stuff should be setup in the controller based file, /etc/trove/trove-guestagent.conf | 08:44 |
| thetrav | in fact | 08:44 |
| thetrav | by default | 08:44 |
| thetrav | that file does not exist | 08:44 |
| amrith | speak with the canonical packagers about that ... here's what I put in mine | 08:44 |
| thetrav | ... | 08:45 |
| amrith | https://gist.github.com/amrith/dd918bb1e72c261a2ab5d1d41fb40c3a | 08:45 |
| thetrav | there's a lack of documentation is my point | 08:45 |
| thetrav | it's not great to have an open source project relying on a paid book to be deployed | 08:46 |
| thetrav | thanks for the link and the help though | 08:46 |
| amrith | I'm not relying on a paid book; I just wrote it. | 08:46 |
| thetrav | I really appreciate it | 08:46 |
| thetrav | you're the author? | 08:46 |
| thetrav | oh yeah | 08:46 |
| amrith | I don't think it tells you how to configure these files either | 08:47 |
| amrith | what it does tell you is that the task manager sends a message to the guestagent | 08:47 |
| amrith | which you seemed to say was not the case. | 08:47 |
| amrith | anyway, enter a bug about the specific things that aren't clear on the documentation in http://docs.openstack.org/mitaka/install-guide-ubuntu/trove-install.html | 08:47 |
| amrith | there's a little bug icon at the bottom right and we'll update the document | 08:48 |
| amrith | but as developers, we are too close to the project to know what an end user finds to be missing | 08:48 |
| amrith | and we need the help and the input. | 08:48 |
| amrith | and that comes in the form of either bugs, or even better, fixes. | 08:48 |
| amrith | and in your case, the bug is that the settings in /etc/trove/trove-guestagent.conf aren't sufficiently clear | 08:49 |
| amrith | and not knowing what the canonical packaging looks like, I have no way of knowing. | 08:49 |
| amrith | what I do know is that in the source tree, there is a file that is 166 lines long, with sample configurations for trove-guestagent | 08:50 |
| thetrav | what is control_exchange? | 08:50 |
| amrith | trove/etc/trove/trove-guestagent.conf.sample | 08:50 |
| thetrav | also is the nova_proxy stuff used? | 08:50 |
| thetrav | and the trove_auth_url | 08:52 |
| amrith | in which section of the config file is that; i suspect it is an oslo/rabbit thng | 08:52 |
| thetrav | I thought the guest-agent only spoke to mq? | 08:52 |
| thetrav | https://gist.github.com/amrith/dd918bb1e72c261a2ab5d1d41fb40c3a | 08:52 |
| thetrav | DEFAULT | 08:52 |
| amrith | control_exchange is used to initialize rpc | 08:53 |
| amrith | should check if it is still required | 08:53 |
| amrith | may be obsolete | 08:53 |
| amrith | no, still required | 08:53 |
| amrith | used by oslo.messaging | 08:53 |
| thetrav | ok | 08:54 |
| thetrav | so the trove_auth_url and nova_proxy stuff? | 08:54 |
| thetrav | the main reason I ask is because I don't have those guests with access to the managment net | 08:54 |
| thetrav | they can't hit trove auth url | 08:54 |
| amrith | sorry, what? | 08:55 |
| amrith | your guest can't hit what network? | 08:55 |
| thetrav | trove guest instances cannot hit any of the APIs | 08:55 |
| thetrav | they can only access the mq | 08:55 |
| amrith | this will be a problem | 08:55 |
| amrith | the guest does in fact need to hit some API end points | 08:55 |
| thetrav | bummer | 08:55 |
| amrith | such as when it wants to take a backup | 08:55 |
| amrith | and so on | 08:55 |
| thetrav | and it needs admin access for these functions? | 08:57 |
| amrith | 1s | 08:59 |
| amrith | i need to clean up that config file | 08:59 |
| amrith | and make it more generally meaningful | 09:01 |
| amrith | not unless you are using single_tenant_remote configurations | 09:02 |
| thetrav | I don't know what that means | 09:02 |
| amrith | sorry, got distracted | 09:26 |
| amrith | that's a specific kind of configuration | 09:26 |
| amrith | and I doubt that you are using it | 09:26 |
| *** khushbu has quit IRC | 09:46 | |
| *** khushbu_ has joined #openstack-trove | 09:49 | |
| *** khushbu_ has quit IRC | 10:03 | |
| *** khushbu__ has joined #openstack-trove | 10:04 | |
| amrith | thetrav, where are you located (timezone) | 10:17 |
| *** khushbu__ has quit IRC | 10:36 | |
| *** thetrav has quit IRC | 10:50 | |
| *** 6A4ABHQ1G has joined #openstack-trove | 11:35 | |
| *** 6A4ABHQ1G has quit IRC | 12:10 | |
| *** imandhan has joined #openstack-trove | 12:19 | |
| *** imandhan has quit IRC | 12:35 | |
| *** zhugaoxiao has quit IRC | 12:37 | |
| *** zhugaoxiao has joined #openstack-trove | 12:38 | |
| *** imandhan has joined #openstack-trove | 12:47 | |
| *** imandhan has quit IRC | 12:52 | |
| *** khushbu_ has joined #openstack-trove | 12:58 | |
| *** khushbu_ has quit IRC | 13:15 | |
| *** itisha has joined #openstack-trove | 13:32 | |
| *** imandhan has joined #openstack-trove | 14:08 | |
| *** imandhan has quit IRC | 14:10 | |
| *** zhugx has quit IRC | 14:34 | |
| *** imandhan has joined #openstack-trove | 15:03 | |
| *** imandhan has quit IRC | 15:06 | |
| *** zhugaoxiao has quit IRC | 15:21 | |
| *** zhugaoxiao has joined #openstack-trove | 15:22 | |
| *** imandhan has joined #openstack-trove | 15:31 | |
| *** imandhan has quit IRC | 16:38 | |
| *** imandhan has joined #openstack-trove | 17:25 | |
| *** arist has quit IRC | 17:47 | |
| openstackgerrit | Shaik Apsar proposed openstack/trove: Fix for 'Client' object has no attribute 'tenants' https://review.openstack.org/340022 | 17:48 |
| *** arist has joined #openstack-trove | 17:48 | |
| *** itlinux has joined #openstack-trove | 18:01 | |
| *** itlinux has quit IRC | 18:02 | |
| *** imandhan has quit IRC | 18:47 | |
| *** fifieldt has quit IRC | 19:07 | |
| *** fifieldt has joined #openstack-trove | 19:18 | |
| *** itlinux has joined #openstack-trove | 19:34 | |
| *** itlinux has quit IRC | 19:59 | |
| *** fifieldt has quit IRC | 20:25 | |
| *** thetrav has joined #openstack-trove | 20:27 | |
| thetrav | amrith Melbourne Australia AEST (GMT+10) | 20:27 |
| amrith | thetrav, thanks. have been trying to catch you for some days now | 20:29 |
| amrith | and wasn't quite sure what time zone you were in | 20:29 |
| amrith | and by the time I woke up you were gone! | 20:29 |
| thetrav | I see | 20:29 |
| thetrav | what time zone are you? | 20:29 |
| amrith | good to connect today, andhopefully get your issue(s) resolved. UTC-4 now (Boston) | 20:29 |
| thetrav | I have 6 month old twins, so my hours are pretty irregular lately | 20:30 |
| amrith | congratulations! | 20:30 |
| thetrav | thanks | 20:30 |
| thetrav | although for the moment my reward is a head full of snot and a house full of screams :P | 20:30 |
| thetrav | cute little munchkins though | 20:31 |
| amrith | so let me look over the scrollback; did we get your system up and running this morning? | 20:31 |
| thetrav | no I've only just started looking at it | 20:31 |
| amrith | :) | 20:31 |
| thetrav | I'm going to re-create my guest image with something that uses that config-dirs | 20:31 |
| thetrav | for some reason the service was not working, but when I ran the command manually it picked up the message and started doing stuff | 20:31 |
| amrith | ok | 20:31 |
| thetrav | it had already timed out on task maanger though | 20:31 |
| amrith | does your image attempt to install software on first boot? | 20:32 |
| amrith | that takes time sometimes | 20:32 |
| amrith | and causes the timeout | 20:32 |
| amrith | in practice you have to align the timeout to the time it takes for your guest to process that prepare message successfully | 20:32 |
| amrith | i.e. get to the point where it receives the message and then process it. | 20:32 |
| amrith | which requires an installed and functioning database. | 20:32 |
| amrith | a lot of things have to happen for that | 20:32 |
| amrith | get the db stopped | 20:33 |
| amrith | mount the cinder volume | 20:33 |
| amrith | migrate the empty db | 20:33 |
| amrith | relaunch the db | 20:33 |
| amrith | get it to 'online' | 20:33 |
| amrith | reply on the conductor channel | 20:33 |
| amrith | and have the TM process the reply | 20:33 |
| amrith | all within whatever your timeout happens to be | 20:33 |
| thetrav | so I've created a snapshot with both trove-guestagent and mysql pre-installed | 20:34 |
| thetrav | that's my guest image | 20:34 |
| amrith | hmm | 20:34 |
| amrith | so firstboot is out | 20:35 |
| thetrav | now do I want to have --config-file as well as --config-dirs ? | 20:35 |
| amrith | ok | 20:35 |
| amrith | no, just config-dirs | 20:35 |
| thetrav | ok | 20:35 |
| thetrav | and it's plural right? | 20:35 |
| amrith | --config-dir DIR Path to a config directory to pull *.conf files from. | 20:35 |
| amrith | This file set is sorted, so as to provide a | 20:35 |
| amrith | predictable parse order if individual options are | 20:35 |
| amrith | over-ridden. The set is parsed after the file(s) | 20:35 |
| amrith | specified via previous --config-file, arguments hence | 20:35 |
| amrith | over-ridden options in the directory take precedence. | 20:35 |
| amrith | from: trove-guestagent --help | 20:35 |
| *** fifieldt has joined #openstack-trove | 20:35 | |
| thetrav | oh, so that could be ok then? | 20:36 |
| thetrav | like, I can set some defaults in /etc/trove/trove.conf and overwrite them with injected values? | 20:36 |
| thetrav | I'll go with just config-dir through | 20:36 |
| amrith | yes, that is absolutely the idea. except that you set them in /etc/trove/trove-guestagent.conf | 20:37 |
| amrith | let me find you a link, one second | 20:37 |
| amrith | https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L358-L361 | 20:38 |
| amrith | https://github.com/openstack/trove/blob/master/trove/common/cfg.py#L356-L359 | 20:38 |
| thetrav | weird log file | 20:42 |
| thetrav | ^@^@^@^@^ repeated | 20:42 |
| amrith | ? | 20:42 |
| amrith | maybe you have formatting characters in the log file | 20:42 |
| thetrav | looking at trove-guestagent.log on the guest I just started | 20:42 |
| *** vipuls has quit IRC | 20:42 | |
| amrith | to colorify it | 20:42 |
| amrith | what's in your '/etc/trove/trove-guestagent.conf' on the controller? | 20:43 |
| thetrav | http://pastebin.com/q9x1nF35 | 20:44 |
| *** vipuls has joined #openstack-trove | 20:44 | |
| amrith | that's odd | 20:45 |
| amrith | you shouldn't see weird things like that then :) | 20:45 |
| thetrav | also: http://pastebin.com/4wC2FnH2 | 20:45 |
| amrith | that looks right | 20:46 |
| thetrav | ahh | 20:47 |
| thetrav | so it's run as the trove user | 20:47 |
| thetrav | the files are owned by root | 20:48 |
| thetrav | and have what... 660 permissions? | 20:48 |
| thetrav | +rw user and group | 20:48 |
| thetrav | nothing for other | 20:48 |
| thetrav | then: oslo_config.cfg.ConfigFileParseError: Failed to parse /etc/trove/conf.d/trove-guestagent.conf: at /etc/trove/conf.d/trove-guestagent.conf:2, Section must be started before assignment: None | 20:50 |
| thetrav | so I guess it needs a [Default] block | 20:50 |
| amrith | [DEFAULT] | 20:50 |
| amrith | take a look at the sample | 20:50 |
| thetrav | right, caps | 20:50 |
| amrith | let me find you a link | 20:50 |
| thetrav | yeah, I remember the sample | 20:50 |
| amrith | https://github.com/openstack/trove/blob/master/etc/trove/trove-guestagent.conf.sample | 20:51 |
| thetrav | ta | 20:51 |
| thetrav | ok, so the permissions thing is a pain | 20:54 |
| amrith | the default elements perform the installation and stuff with that in mind | 20:54 |
| thetrav | ubuntu's packaging of trove-guestagent runs as an upstart service as the trove user | 20:54 |
| amrith | ok | 20:54 |
| thetrav | for the config files to be root read only is problematic | 20:55 |
| thetrav | additionally I think python was trying to open the files with rw? | 20:55 |
| thetrav | unless I fat fingered the permissions | 20:55 |
| amrith | i think oslo.config does try to read the files rw | 20:56 |
| thetrav | seems like an odd choice | 20:57 |
| thetrav | maybe that's a default that can be changed? | 20:57 |
| amrith | haven't seen that before, let me look | 20:58 |
| amrith | can't see one | 21:04 |
| thetrav | ok, sort of progress, churning through various permissions errors in setting up a guest image | 21:15 |
| thetrav | WOO! | 21:20 |
| thetrav | ACTIVE! | 21:20 |
| thetrav | no idea if it actually works... but that's the first I've seen conductor logging so it feels like a major step | 21:20 |
| thetrav | aright, this seems to work to some degree | 21:27 |
| thetrav | now to automate and document | 21:27 |
| thetrav | thanks for all your help amrith | 21:27 |
| thetrav | I sent a bug report to the openstack docs | 21:28 |
| thetrav | I might also do a bit of a write up on how to create a trove guest-image for people who don't know trove | 21:28 |
| amrith | why not just update the one in the repo already? | 21:29 |
| amrith | there is a write-up and greg just added some useful stuff that he found today | 21:29 |
| amrith | see his change https://review.openstack.org/#/c/351958/ | 21:30 |
| amrith | one of the things it does is updates the doc | 21:30 |
| amrith | which you probably read as well | 21:30 |
| amrith | if you could add your findings to that, it would help the next person who comes along. | 21:30 |
| thetrav | the docs seem heavily focussed on devstack | 21:39 |
| thetrav | I could write a completely new guide in there if you like | 21:40 |
| thetrav | however it'd be "how to build a mysql guest image using the default ubuntu cloudimage" | 21:41 |
| *** gouthamr has joined #openstack-trove | 21:42 | |
| thetrav | do you think it'd be valuable to have a "walkthrough for openstack noobs" style process? | 21:42 |
| amrith | absolutely | 21:44 |
| amrith | go for it | 21:45 |
| amrith | and please do contribute a change set for it | 21:45 |
| amrith | would be much appreciated | 21:45 |
| amrith | I've got to run now, will be back in a couple of hours | 21:45 |
| *** chlong has joined #openstack-trove | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!