| sorrison | lxkong: thanks for the local.conf. I used that (except didn't set HOST_IP and use master trove) and still get the same issue. my local.conf is http://paste.openstack.org/show/800526/ | 00:03 |
|---|---|---|
| sorrison | I just rebuilt it and the trove guest image doesn't have ping anymore which I'm trying to figure out | 00:03 |
| lxkong | you are using a vm for devstack host? | 00:04 |
| sorrison | yeah | 00:08 |
| sorrison | my trove instances only have the trove-mgmt interface | 00:08 |
| sorrison | maybe it's assuming it has a user interface too and that is how it's getting out to the controller? | 00:09 |
| lxkong | have you checked the console log of the vm and the trove task manager log? | 00:09 |
| sorrison | yeah once I change the controller IP in the trove.conf etc. I can get it upto trying to talk to docker registry | 00:09 |
| sorrison | so I'm guessing it needs the users network attached to actually work and the trove-mgmt port isn't enough? | 00:10 |
| sorrison | as it doesn't even have a default route | 00:10 |
| lxkong | so you are not specifying a user network? | 00:11 |
| lxkong | what's your command to create the instance? | 00:11 |
| sorrison | no I didn't specify one | 00:11 |
| sorrison | openstack database instance create --flavor d2 --size 1 sam1 | 00:11 |
| sorrison | I assumed the trove-mgmt interface was for talking to the controller etc.? | 00:12 |
| sorrison | ok with using the `private` network it gets further but I thinks registry is localhost `404 Client Error: Not Found for url: http+docker://localhost/v1.40/containers/database/json` | 00:22 |
| *** congnt96 has quit IRC | 00:24 | |
| sorrison | I'm guessing https://review.opendev.org/c/openstack/trove/+/764260 removed ping and other handy tools from the guest images too, be good to get those back, they take up no room and can he very handy for operators when debugging issues | 00:25 |
| sorrison | I can't see any config option to be able to set the container registry to something other than localhost too, maybe I'm missing something | 00:28 |
| *** tosky has quit IRC | 00:39 | |
| lxkong | back | 00:41 |
| lxkong | by default, the user network is supposed to be the first nic of the vm and its gateway is served as the default network gateway, so the docker pull is going through that interface (could be modified by cloud admin). Mgmt network should be used to talk to the controller. | 00:41 |
| lxkong | the set up in devstack is special and is just for testing | 00:41 |
| sorrison | yeah ok, so looks like the trove-mgmt network is not used by devstack at all | 00:42 |
| lxkong | the cloud operator has full control over the networking by e.g. providing cloud-init config or config in the infra layer | 00:42 |
| lxkong | in our case, we did something in our infra router for the mgmt traffic | 00:43 |
| * lxkong goes to a meeting | 00:44 | |
| sorrison | yeah makes sense. So do you know why it's using localhost for container registry? | 00:44 |
| sorrison | ah ok, chat later | 00:44 |
| openstackgerrit | Merged openstack/trove master: Use nested virt node for trove functional test https://review.opendev.org/c/openstack/trove/+/763630 | 01:48 |
| *** congnt95 has quit IRC | 03:52 | |
| *** congnt95 has joined #openstack-trove | 03:54 | |
| *** congnt95 has quit IRC | 03:57 | |
| *** congnt95 has joined #openstack-trove | 03:59 | |
| sorrison | lxkong: Just wondering if you're around. I can't seem to figure out why it's using the URL `http+docker://localhost/v1.40/containers/database/json` to get a container | 04:41 |
| sorrison | it them looks for `404 Client Error for http+docker://localhost/v1.40/images/mysql:5.7.29/json` | 04:41 |
| openstackgerrit | Sam Morrison proposed openstack/trove-tempest-plugin master: [WIP] reinstate removed api tests https://review.opendev.org/c/openstack/trove-tempest-plugin/+/761472 | 05:25 |
| openstackgerrit | Sam Morrison proposed openstack/trove-tempest-plugin master: [WIP] reinstate removed api tests https://review.opendev.org/c/openstack/trove-tempest-plugin/+/761472 | 05:31 |
| *** tosky has joined #openstack-trove | 08:37 | |
| *** damien_r has quit IRC | 08:48 | |
| *** damien_r has joined #openstack-trove | 09:01 | |
| openstackgerrit | Cong Nguyen Thanh proposed openstack/trove stable/victoria: Get slave_pos to choose latest replica https://review.opendev.org/c/openstack/trove/+/764507 | 09:02 |
| lxkong | sorrison: are you able to access the dockerhub inside the guest instance? Anything suspicious in `/etc/hosts`? I've several devstack installations in different VMs but never seen this before. | 09:03 |
| openstackgerrit | Merged openstack/trove master: Get slave_pos to choose latest replica https://review.opendev.org/c/openstack/trove/+/763498 | 09:39 |
| openstackgerrit | Lingxian Kong proposed openstack/trove stable/victoria: Get slave_pos to choose latest replica https://review.opendev.org/c/openstack/trove/+/764507 | 09:57 |
| *** e0ne has joined #openstack-trove | 11:01 | |
| *** tosky has quit IRC | 12:24 | |
| *** tosky has joined #openstack-trove | 12:25 | |
| *** e0ne has quit IRC | 13:00 | |
| *** e0ne has joined #openstack-trove | 13:56 | |
| *** e0ne has quit IRC | 14:21 | |
| *** e0ne has joined #openstack-trove | 14:59 | |
| *** e0ne has quit IRC | 15:38 | |
| *** dkehn has quit IRC | 15:44 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to troubleshoot high load and poor query caching performance, downtime should be less than 5 minutes | 16:22 | |
| *** e0ne has joined #openstack-trove | 16:45 | |
| *** e0ne has quit IRC | 16:51 | |
| *** e0ne has joined #openstack-trove | 17:21 | |
| *** hoonetorg has joined #openstack-trove | 17:23 | |
| *** e0ne has quit IRC | 17:25 | |
| *** e0ne has joined #openstack-trove | 17:29 | |
| *** e0ne has quit IRC | 17:42 | |
| *** hoonetorg has quit IRC | 19:25 | |
| *** hoonetorg has joined #openstack-trove | 19:26 | |
| *** tosky has quit IRC | 19:45 | |
| *** tosky has joined #openstack-trove | 20:48 | |
| sorrison | kxkong: Yeah I can do a `curl https://hub.docker.com/` ok | 21:52 |
| sorrison | nothing of interest in /etc/hosts | 21:52 |
| sorrison | also see a `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.40/images/create?tag=5.7.29&fromImage=mysql` | 22:04 |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to make further query caching and Git garbage collection adjustments, downtime should be less than 5 minutes | 22:37 | |
| openstackgerrit | Merged openstack/trove stable/victoria: Get slave_pos to choose latest replica https://review.opendev.org/c/openstack/trove/+/764507 | 22:40 |
| lxkong | sorrison: the error happened when you do 'docker pull'? | 22:43 |
| sorrison | no it's in the logs | 22:45 |
| lxkong | sorrison: can you paste me the trove-guestagent log? | 22:45 |
| sorrison | http://paste.openstack.org/show/800576/ | 22:46 |
| sorrison | I'm thinking localhost in the log errors is wrong as it's getting a 404 and a 500 and there is no webservice running on localhost? | 22:46 |
| lxkong | the same error if you try 'docker pull mysql:5.7.29'? | 22:48 |
| sorrison | that works | 22:48 |
| sorrison | as root, should it be done as a different user | 22:49 |
| lxkong | the guest image was built by devstack or manually? | 22:49 |
| sorrison | devstack built it | 22:49 |
| lxkong | yeah, you can try to use the ubuntu user | 22:49 |
| sorrison | yeah works as the ubuntu users too | 22:50 |
| lxkong | inside the instance, can you try https://dpaste.com/AKP9YJF8H? | 22:54 |
| *** rcernin has joined #openstack-trove | 23:03 | |
| lxkong | sorrison: did you mention that you don't have user port attached to the guest instance? Is that going to be the problem? | 23:22 |
| sorrison | No this is with a user port | 23:22 |
| sorrison | that snippet you sent worked | 23:23 |
| sorrison | ok I just tied again and it worked. I'm guessing there were some docker hub issues | 23:30 |
| sorrison | going by those 500 errors that docker hub was giving me. the logs say localhost but it must be dockerhub returning the 500s | 23:31 |
| sorrison | another question (sorry I have so many :-) ) with the tempest plugin I see it creates it's own network/subnet/router etc. Just wondering why it does this as opposed to using the network that tempest creates for each job? | 23:32 |
| sorrison | tried again and it failed. seems dockerhub is very flakey | 23:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!