| *** jamesmcarthur has quit IRC | 00:04 | |
| *** jamesmcarthur has joined #openstack-trove | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** jamesmcarthur has joined #openstack-trove | 00:27 | |
| *** spatel has joined #openstack-trove | 00:50 | |
| *** spatel has quit IRC | 00:54 | |
| *** spatel has joined #openstack-trove | 00:59 | |
| *** jamesmcarthur has quit IRC | 01:05 | |
| *** sapd1 has joined #openstack-trove | 01:15 | |
| *** sapd1 has quit IRC | 01:23 | |
| *** jamesmcarthur has joined #openstack-trove | 01:32 | |
| *** sapd1 has joined #openstack-trove | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** jamesmcarthur has joined #openstack-trove | 01:49 | |
| *** jamesmcarthur has quit IRC | 01:54 | |
| *** congnt95 has joined #openstack-trove | 02:06 | |
| *** jamesmcarthur has joined #openstack-trove | 02:24 | |
| *** sapd1 has quit IRC | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:34 | |
| *** sapd1 has joined #openstack-trove | 02:47 | |
| *** congnt95 has quit IRC | 02:50 | |
| *** congnt95 has joined #openstack-trove | 02:50 | |
| *** sapd1 has quit IRC | 02:53 | |
| *** sapd1 has joined #openstack-trove | 02:53 | |
| *** jmlowe has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-trove | 03:08 | |
| openstackgerrit | Zihao Wang proposed openstack/python-troveclient master: Remove six https://review.opendev.org/757538 | 03:09 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 03:11 | |
| *** jamesmcarthur has joined #openstack-trove | 03:11 | |
| *** jmlowe has joined #openstack-trove | 03:11 | |
| *** jamesmcarthur has quit IRC | 03:20 | |
| *** sapd1 has quit IRC | 03:28 | |
| *** sapd1 has joined #openstack-trove | 03:53 | |
| *** tonyb has quit IRC | 04:10 | |
| sorrison | Hi lxkong are you around? | 04:18 |
| lxkong | sorrison: yes, how can i help | 04:18 |
| sorrison | I'm debugging a race condition I'm seeing. | 04:18 |
| lxkong | yep, any detailed info? | 04:19 |
| sorrison | postgres is set on our images to start on boot but sometimes trove fails as it init starts postgres just as trove is going to stop postgres | 04:19 |
| sorrison | do you have your images to have postgres/mysql to be disabled in systemd on boot? | 04:19 |
| sorrison | would that work with trove? (ie. it would see they aren't running and start it) | 04:20 |
| sorrison | eg. I see `Oct 13 00:46:14 bgno3maogld sudo[681]: trove : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/service postgresql stop` | 04:21 |
| sorrison | then next line is `Oct 13 00:46:14 bgno3maogld systemd[1]: Starting PostgreSQL RDBMS...` | 04:21 |
| lxkong | sorrison: Ussuri? | 04:21 |
| sorrison | ussuri taskmanager/api/conductor but rocky guest agent :-) | 04:22 |
| sorrison | maybe this is handled in ussuri guest agent? | 04:22 |
| sorrison | it only happens sometimes in my tempest tests and just trying to figure out best way to handle it | 04:23 |
| lxkong | when creating the instance, trove guest agent just does some stops/starts during the initialization | 04:23 |
| sorrison | and it would be ok if mysql/postgres was not running to begin with I assume? Eg. init doesn't start it | 04:24 |
| lxkong | you can't simply disable the service in systemd as it affects the vm restart or something | 04:24 |
| lxkong | that logic was totally changed after ussuri when using containers | 04:24 |
| sorrison | yeah that was what I was thinking. does trove-guest agent ensure mysql/postgres is running when it starts up after a reboot | 04:24 |
| *** jamesmcarthur has joined #openstack-trove | 04:25 | |
| lxkong | no, when the vm is restarting, it relies on systemd to start the db service | 04:25 |
| lxkong | trove-guestagent won't operate db service when starts up | 04:26 |
| sorrison | yeah ok, thanks that's what I thought | 04:26 |
| sorrison | Might just need to ensure the trove-guestagent systemd goes after mysql/postgres then I think | 04:26 |
| openstackgerrit | Merged openstack/python-troveclient master: Remove six https://review.opendev.org/757538 | 04:27 |
| lxkong | sorrison: but in your case, i don't think it's race condition, trove-guestagent should guarantee the db service started | 04:27 |
| sorrison | I'm pretty sure it's a race | 04:28 |
| sorrison | as trove goes to stop postgres just as systemd is starting it | 04:28 |
| sorrison | so it fails as postgres never stops | 04:28 |
| lxkong | in the guest, the operations are mostly happening in sequence. | 04:28 |
| sorrison | so trove issues the stop command | 04:28 |
| sorrison | systemd then issues the start | 04:28 |
| sorrison | and trove errors as it never stops | 04:29 |
| *** jamesmcarthur has quit IRC | 04:29 | |
| sorrison | I have logs of a working and a broken one and can confirm it's a timing thing as we are only seeing it about 30% of the time in our tests | 04:29 |
| lxkong | sorrison: have you log into the guest and do something like systemd-analyze | 04:30 |
| sorrison | yeah I can see by looking at the journal | 04:31 |
| lxkong | if that's related to systemd service order, another work around is to change the trove-guest.systemd config in DIB element | 04:34 |
| lxkong | then re-build the image... | 04:34 |
| sorrison | yeah that's what I'm thinking | 04:34 |
| sorrison | could ensure trove-guest agent starts after postgres | 04:35 |
| sorrison | once we are fully on ussuri we will be more vanila, currently we have lots of patches on our rocky branch | 04:37 |
| sorrison | the upgrade planning is going well though. We have managed to get our existing images working with rocky and ussuri taskmanager | 04:37 |
| sorrison | we have mysql5.7, mysql8.0, postgresql 9.6 and 11 (and soon 13) | 04:38 |
| *** spatel has quit IRC | 05:07 | |
| *** e0ne has joined #openstack-trove | 07:10 | |
| *** tosky has joined #openstack-trove | 07:26 | |
| *** rcernin has quit IRC | 07:52 | |
| *** rcernin has joined #openstack-trove | 08:45 | |
| openstackgerrit | Zihao Wang proposed openstack/trove master: Remove six usage [1] and basestring check https://review.opendev.org/757771 | 08:53 |
| openstackgerrit | Zihao Wang proposed openstack/trove master: Remove six usage [2] https://review.opendev.org/757783 | 09:09 |
| openstackgerrit | Zihao Wang proposed openstack/trove master: Remove six usage [3] https://review.opendev.org/757792 | 09:19 |
| openstackgerrit | Lingxian Kong proposed openstack/trove master: [Doc] datastore guide https://review.opendev.org/757794 | 09:22 |
| *** rcernin has quit IRC | 09:33 | |
| *** rcernin has joined #openstack-trove | 09:35 | |
| *** jamesmcarthur has joined #openstack-trove | 09:38 | |
| *** jamesmcarthur has quit IRC | 09:42 | |
| *** rcernin has quit IRC | 09:58 | |
| *** rcernin has joined #openstack-trove | 09:59 | |
| *** rcernin has quit IRC | 10:14 | |
| *** rcernin has joined #openstack-trove | 10:25 | |
| openstackgerrit | Lingxian Kong proposed openstack/trove master: [Doc] datastore guide https://review.opendev.org/757794 | 10:27 |
| *** congnt96 has joined #openstack-trove | 11:09 | |
| *** congnt96 has quit IRC | 11:13 | |
| *** sapd1_x has joined #openstack-trove | 11:14 | |
| *** sapd1 has quit IRC | 11:18 | |
| *** sapd1_x has quit IRC | 11:44 | |
| *** rcernin has quit IRC | 12:11 | |
| *** openstackgerrit has quit IRC | 13:17 | |
| *** jamesmcarthur has joined #openstack-trove | 13:56 | |
| *** jamesmcarthur has quit IRC | 14:38 | |
| *** jamesmcarthur has joined #openstack-trove | 14:40 | |
| *** jamesmcarthur has quit IRC | 14:51 | |
| *** jamesmcarthur has joined #openstack-trove | 14:58 | |
| *** sapd1_x has joined #openstack-trove | 15:13 | |
| *** jamesmcarthur has quit IRC | 15:33 | |
| *** jamesmcarthur has joined #openstack-trove | 15:49 | |
| *** congnt96 has joined #openstack-trove | 16:36 | |
| *** sapd1_x has quit IRC | 16:41 | |
| *** jamesmcarthur has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-trove | 17:41 | |
| *** e0ne has quit IRC | 19:07 | |
| *** e0ne has joined #openstack-trove | 19:17 | |
| *** e0ne has quit IRC | 19:22 | |
| *** jamesmcarthur has quit IRC | 20:51 | |
| *** jamesmcarthur has joined #openstack-trove | 20:56 | |
| *** jamesmcarthur has quit IRC | 21:19 | |
| *** jamesmcarthur has joined #openstack-trove | 21:26 | |
| *** rcernin has joined #openstack-trove | 22:17 | |
| *** rcernin has quit IRC | 22:45 | |
| *** rcernin has joined #openstack-trove | 22:51 | |
| *** rcernin has quit IRC | 22:51 | |
| *** rcernin has joined #openstack-trove | 22:51 | |
| *** hoonetorg has quit IRC | 23:22 | |
| *** zigo has quit IRC | 23:23 | |
| *** zigo has joined #openstack-trove | 23:24 | |
| *** hoonetorg has joined #openstack-trove | 23:34 | |
| *** tosky has quit IRC | 23:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!