Tuesday, 2017-05-02

johnsomharlowja I'm using the patchset one behind the one you just posted.  I'm having problems where the octavia-working isn't stopping after a devstack clean.sh00:11
harlowjajohnsom hi ho00:11
harlowjahi ho silver00:11
harlowjasooo whats hanging00:12
johnsomI've also given it the 60 seconds just to let the graceful timeout fire, still lives on00:12
harlowjaya, that makes me think cotyledon then00:13
johnsomI'm all for "takes a lickin' but keeps on ticking" but....00:13
johnsomYeah, this is new after that change00:13
johnsomStrange thing is if I kill -15 the master process it does eventually shutdown00:14
harlowjaany idea if u can get where it is locked00:14
johnsomI would assume that devstack is kill -15 the processes...00:15
johnsomAll that does is sit and wait for oslo messages00:15
johnsomThe worst part is devstack nukes the log directory so not a lot of output to look at00:16
harlowjahttps://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L283 would be nice to know if it comes out00:17
harlowjacan u try to get the logs :-P00:18
johnsomI am not oslo logging ninja enough to know if I can temp move that log to /tmp or something00:18
harlowjago for it00:19
harlowjai believe in u00:19
harlowjau are the ninja u want to be00:19
johnsomI guess I can hack clean.sh to just not nuke the directory00:19
harlowjaya, need more info about what is waiting to stop00:22
johnsomclean.sh hack in place, waiting for stack to finish to repo00:23
*** armax has quit IRC00:30
openstackgerritJude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client  https://review.openstack.org/46123100:32
rm_workI mean, I think the signal it's listening for might be specifically -9?00:32
*** sanfern has quit IRC00:32
johnsomNo, -9 doesn't get to the process, the kernel traps that one and kills without prejudice00:33
harlowjaonly thing i can think is the fast_exit of cotleddon00:34
rm_workthen yeah dunno00:34
harlowjau should see that though00:34
johnsomHmm, I don't see any logs from cotyledon00:34
johnsomexcept the config and the run service line00:34
harlowjalogging configured for that?00:34
harlowjathough u should see something :-/00:35
* harlowja doesn't see any special cotledyon stuff in https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L19-L30 00:36
harlowjaso something should pop out i guess00:36
johnsomThat is all I get, dump the config, say I'm running and nothing00:38
johnsomEven when I send it kill -1500:39
harlowjaall i see is config_glue, lol00:39
johnsomThough that does seem to start the 60 second graceful shutdown clock00:39
harlowjadoes http://paste.openstack.org/raw/608559/ havfe more :-P00:41
harlowjaall i see is a bunch of config flue, lol00:41
johnsomOh, crumb, that is odd.00:41
johnsomPaste fail.00:41
johnsomJust a second00:41
johnsomThough, 90% of it is config glue00:41
johnsomgreat, paste is truncating...  gist on the way...00:43
harlowjalooks ok to me :_P00:44
harlowjaexcept ' is unreachable' stuff00:44
harlowjahow are u stopping that, ctrl-c?00:45
johnsomThat's from the ./clean.sh killing rabbit00:45
openstackgerritJude Cross proposed openstack/python-octaviaclient master: Add loadbalancer commands to client  https://review.openstack.org/45451600:46
JudeCsorry about the spam just doing some cleanup :)00:46
JudeCload balancer commands look pretty good right now.00:47
johnsomJudeC I have zero problems with progress spam...  Grin00:48
harlowjajohnsom do u have the kill logs? or was that them ?00:50
harlowjawondering if u are even seeing https://github.com/openstack/octavia/blob/master/octavia/controller/queue/consumer.py#L4900:50
johnsomThat was it...00:51
johnsomI do have that in the code I'm running, but yeah, doesn't seem to get there or the logs go into the ehter00:52
*** bread has joined #openstack-lbaas00:52
*** bread is now known as aamerine00:52
johnsomYeah, just the start, not the stop00:53
harlowjaanyway u can add some kind print/output or something to https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L31300:57
harlowjathat is supposed to log a reason, though perhaps its not?00:57
johnsom^C2017-05-01 17:30:08.024 37227 INFO cotyledon._service_manager [-] Caught SIGIN00:59
johnsomT signal, instantaneous exiting00:59
johnsomI get that log message when I control-c it from the console00:59
johnsomJust not kill -1500:59
johnsomnor clean.sh00:59
harlowjaright SIGTERM is trying to graceful shutdown01:00
harlowjanot sure what signal clean.sh is using01:01
johnsomIt should use sigterm01:02
harlowjaya, the thing that seems odd is that u aren't seeing https://github.com/sileht/cotyledon/blob/master/cotyledon/_service_manager.py#L25501:02
*** blogan_ has joined #openstack-lbaas01:07
*** blogan has quit IRC01:09
harlowjai mean lookin at clean01:11
openstackgerritMichael Johnson proposed openstack/octavia master: Fix an failure if the vip network has no gateway  https://review.openstack.org/46158201:11
harlowjait seems to just call into unstack01:11
harlowjalike if https://github.com/openstack-dev/devstack/blob/master/unstack.sh#L172-L176 is happening01:12
harlowjathen i wonder if that's just shutting things down incorrectly01:12
harlowjabut ya, not sure, my devstack foo is weak01:12
johnsomHmm, wonder if oslo service sees the terminal go away and dies but cotyl doesn'01:14
* johnsom grasps at straws01:16
johnsomI think I need a break and will apply head to wall again tomorrow01:16
rm_workyeah would be nice to get the gate working and some of this pending stuff merged in the meantime too01:27
openstackgerritJude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client  https://review.openstack.org/46123101:27
openstackgerritJude Cross proposed openstack/python-octaviaclient master: Add loadbalancer commands to client  https://review.openstack.org/45451601:27
*** JudeC has quit IRC01:32
*** chlong has quit IRC01:50
*** fnaval has joined #openstack-lbaas02:33
*** fnaval has quit IRC03:31
*** gcheresh has joined #openstack-lbaas04:00
*** links has joined #openstack-lbaas04:05
*** fnaval has joined #openstack-lbaas04:06
*** csomerville has joined #openstack-lbaas04:11
*** cody-somerville has quit IRC04:14
*** aojea has joined #openstack-lbaas04:19
*** aojea has quit IRC04:23
*** gcheresh has quit IRC04:50
*** fnaval has quit IRC05:20
*** blogan_ has quit IRC05:25
*** cpuga has quit IRC05:33
*** cpuga has joined #openstack-lbaas05:33
*** cpuga has quit IRC05:33
*** cpuga has joined #openstack-lbaas05:34
*** JudeC has joined #openstack-lbaas05:37
*** cpuga has quit IRC05:39
openstackgerritcheng proposed openstack/octavia master: Add monitor address and port to pool member  https://review.openstack.org/46164906:19
*** aojea has joined #openstack-lbaas06:50
openstackgerritZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id  https://review.openstack.org/45830806:51
*** voelzmo has joined #openstack-lbaas06:52
*** pcaruana has joined #openstack-lbaas07:06
*** gcheresh has joined #openstack-lbaas07:23
openstackgerritZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id  https://review.openstack.org/45830807:30
*** tesseract-RH has joined #openstack-lbaas07:30
*** JudeC has quit IRC07:36
*** gcheresh has quit IRC07:59
openstackgerritMerged openstack/octavia master: Fix a bug where ports may not be deleted  https://review.openstack.org/43132408:40
*** aojea has quit IRC08:43
*** links has quit IRC09:07
openstackgerritAdam Harwell proposed openstack/octavia master: Add support for single-create for APIv2  https://review.openstack.org/45903209:08
openstackgerritAdam Harwell proposed openstack/octavia master: Implement sorting and pagination for octavia  https://review.openstack.org/38214709:10
openstackgerritAdam Harwell proposed openstack/octavia master: Update devstack to properly configure nova-lxd img  https://review.openstack.org/45323709:13
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561209:16
rm_work^^ doing some rebasing of older patches that I think are still possibly relevant09:16
rm_workpartly just so they'll float to the top of the change list and not get lost, because we need to take them over09:17
openstackgerritAdam Harwell proposed openstack/octavia master: LB/Listener Stats  https://review.openstack.org/44924309:18
*** links has joined #openstack-lbaas09:25
*** mdavidson has joined #openstack-lbaas09:25
*** aojea has joined #openstack-lbaas09:31
*** amit213 has quit IRC09:43
*** amitry has quit IRC09:43
*** sindhu has quit IRC09:43
*** dasanind has quit IRC09:43
*** amitry has joined #openstack-lbaas09:44
*** yamamoto has quit IRC09:52
rm_workwhere are you openstackgerrit09:56
rm_workjust submitted 4 patchsets09:56
rm_workah it must not announce for stable branches09:56
openstackgerritcheng proposed openstack/octavia master: Add monitor address and port to pool member  https://review.openstack.org/46164909:59
*** links has quit IRC10:01
*** amit213 has joined #openstack-lbaas10:16
*** dasanind has joined #openstack-lbaas10:17
*** sindhu has joined #openstack-lbaas10:17
*** links has joined #openstack-lbaas10:19
*** kbyrne has quit IRC11:08
*** kbyrne has joined #openstack-lbaas11:12
*** kbyrne has quit IRC11:16
*** kbyrne has joined #openstack-lbaas11:21
*** b3nt_pin is now known as beagles11:24
*** beagles is now known as Guest9361811:24
*** Guest93618 is now known as b3nt_pin11:25
*** yamamoto has joined #openstack-lbaas11:32
*** yamamoto_ has joined #openstack-lbaas11:33
*** yamamoto has quit IRC11:36
*** yamamoto_ has quit IRC11:58
*** yamamoto has joined #openstack-lbaas11:59
*** yamamoto has quit IRC12:01
*** gcheresh has joined #openstack-lbaas12:03
*** lunarlamp has joined #openstack-lbaas12:06
*** krypto has joined #openstack-lbaas12:42
*** krypto has joined #openstack-lbaas12:42
*** cpuga has joined #openstack-lbaas12:43
*** b3nt_pin is now known as beagles12:46
*** cpuga has quit IRC12:48
*** catintheroof has joined #openstack-lbaas12:55
*** cpuga has joined #openstack-lbaas12:56
*** yamamoto has joined #openstack-lbaas13:02
*** chlong has joined #openstack-lbaas13:13
*** links has quit IRC13:20
*** yamamoto has quit IRC13:25
*** yamamoto has joined #openstack-lbaas13:25
*** openstackgerrit has quit IRC13:48
*** openstackgerrit has joined #openstack-lbaas13:54
openstackgerritGerman Eichberger proposed openstack/octavia master: Adds status tree to V2 API  https://review.openstack.org/45988013:54
*** fnaval has joined #openstack-lbaas14:06
*** krypto has quit IRC14:08
*** voelzmo has quit IRC14:16
*** armax has joined #openstack-lbaas14:18
*** voelzmo has joined #openstack-lbaas14:24
*** voelzmo has quit IRC14:29
*** voelzmo has joined #openstack-lbaas14:29
*** krypto has joined #openstack-lbaas14:32
*** fnaval has quit IRC14:32
*** krypto has quit IRC14:37
*** basilAB has quit IRC14:42
*** vaishali has quit IRC14:43
*** robcresswell has quit IRC14:43
*** dougwig has quit IRC14:44
*** ankur-gupta-f4 has quit IRC14:44
*** redrobot has quit IRC14:44
*** m-greene has quit IRC14:45
*** ctracey has quit IRC14:45
*** jmccrory has quit IRC14:45
*** pcaruana has quit IRC14:45
*** fnaval has joined #openstack-lbaas14:46
*** ptoohill has quit IRC14:46
*** sshank has quit IRC14:47
*** m-greene has joined #openstack-lbaas14:47
*** greghaynes has quit IRC14:47
*** ptoohill has joined #openstack-lbaas14:48
*** ianychoi has quit IRC14:49
*** ianychoi has joined #openstack-lbaas14:50
*** jmccrory has joined #openstack-lbaas14:51
*** pcaruana has joined #openstack-lbaas14:51
*** basilAB has joined #openstack-lbaas14:51
*** sshank has joined #openstack-lbaas14:52
*** redrobot has joined #openstack-lbaas14:52
*** redrobot is now known as Guest9210214:52
*** krypto has joined #openstack-lbaas14:53
*** krypto has quit IRC14:53
*** Guest12904 has joined #openstack-lbaas14:54
*** dougwig has joined #openstack-lbaas14:54
*** basilAB has quit IRC14:56
*** vaishali has joined #openstack-lbaas14:56
*** basilAB has joined #openstack-lbaas14:57
*** blogan has joined #openstack-lbaas14:57
*** greghaynes has joined #openstack-lbaas14:57
*** gcheresh has quit IRC15:01
*** Guest12904 has quit IRC15:02
*** ctracey has joined #openstack-lbaas15:04
johnsomrm_work Are you around?15:05
*** chlong has quit IRC15:13
rm_workjohnsom: yeah15:14
rm_worklook at the results of the rechecks on the other stable branch patches15:14
rm_workthe tempest ones15:14
rm_worksomething that isn't using stable branches (gate maybe?) is failing on LIBS_FROM_GIT for the client15:14
rm_workmaybe the correct fix is to hack a bit on the gate side15:15
rm_workbut i figured i'd try this and see if it was easier just to add the client bits15:15
rm_workotherwise our stable gates are blocked ATM15:15
johnsomI see15:15
rm_workinteresting, this is a new one15:16
rm_workat least for the ocata gate15:16
johnsomYeah, we need to fix the project-config15:17
rm_worksame for newton15:17
johnsomInstalling python-octaviaclient isn't consistent with the stable policy...15:17
rm_workjust prevent from adding the devstack project line for the client, on stables?15:18
rm_work... some stables15:18
rm_workuntil we get to stable/pike15:18
rm_workthen we will need it15:18
rm_workso can't block it on ALL stable branches15:18
johnsomI still don't understand why we can't just do that in the gate hook15:20
rm_workdo you have any ideas? I can ask infra, they might have some guidance15:20
rm_workbecause the gate hook is too late to affect PROJECTS15:20
rm_workbecause by then it's already checked out everything15:20
johnsomOk, so remind me why we need this at all in the gate?  None of the tests will use the client15:22
rm_workbecause in order to have OUR devstack plugin install the client15:23
rm_workit has to be installed via gate15:23
rm_workthis is a VERY common pattern15:23
rm_workevery other project does it15:23
rm_workI will admit it seems like a really weird confluence of crazy15:24
rm_workwith regard to the little quirks in how all of this stuff works lining up in a very peculiar way15:24
johnsomYeah, especially since I haven't had my coffee yet.15:24
rm_workbut go look at any other project15:24
rm_workseriously, pick one15:24
rm_workpretty sure all other projects do it15:25
rm_workunless their plugin is in OSC by default maybe15:25
rm_workbut even glance does it IIRC15:25
rm_workand magnum and barbican15:25
rm_worklet me look at nova really quick15:25
rm_workaah they have a bash-completion thing too15:27
rm_worki think that's what we need to set u15:27
rm_work*set up15:27
johnsomSo meet me halfway.  Let's do patches that just fix the lines in gate_hook and not install the client.15:28
rm_workright, just not sure how15:28
rm_workjust check for the two stable branches we know about15:28
rm_workand override?15:28
*** chlong has joined #openstack-lbaas15:30
johnsomNevermind, I know that wont work15:30
rm_workit won't?15:30
rm_workit might15:30
rm_worklooking for how to see if the job that's running is for stable15:30
*** sanfern has joined #openstack-lbaas15:31
johnsomYou mean conditional in the jenkins job?  That would work.  I was thinking about the stable branch patches15:31
rm_workyeah i meant in project-config15:31
rm_worki'm there now15:31
rm_workjust looking for the right var to check15:31
johnsomYou can just check for the BRANCH_OVERRIDE15:31
rm_worki thought that was for ZUUL15:32
rm_worknot for us15:32
rm_workso what will that var look like15:32
johnsomCould be.15:32
rm_workI can ask infra15:32
johnsomI'm going to go make coffee real quick15:32
*** amotoki has joined #openstack-lbaas15:40
rm_workjohnsom: https://review.openstack.org/#/c/461830/15:49
rm_worklol already +A'd15:57
johnsomSigh, see above, coffee15:58
rm_workyeah so speaking of15:58
rm_workit's been ... a while since i slept15:58
rm_workso gonna take a 4h or so nap <_<15:58
rm_workfeel free to poke any patches I have up15:59
rm_workmake changes, do whatever, go nuts ^_^15:59
xgermango nuts? Is this an inviation…15:59
*** krypto has joined #openstack-lbaas16:01
*** tesseract-RH has quit IRC16:01
*** pcaruana has quit IRC16:02
*** chlong has quit IRC16:04
*** voelzmo has quit IRC16:05
johnsomAlright!  REvert cotyledon....16:05
xgermanI was thinking rewrite everyhting in Clojure16:07
johnsomYeah, that will help....16:08
*** csomerville has quit IRC16:11
*** cody-somerville has joined #openstack-lbaas16:11
*** amotoki has quit IRC16:21
*** csomerville has joined #openstack-lbaas16:23
*** cody-somerville has quit IRC16:26
*** blogan has quit IRC16:30
*** aojea has quit IRC16:31
*** robcresswell has joined #openstack-lbaas16:43
*** ankur-gupta-f4 has joined #openstack-lbaas16:45
*** blogan has joined #openstack-lbaas16:45
*** harlowja has quit IRC17:04
*** JudeC has joined #openstack-lbaas17:11
*** harlowja has joined #openstack-lbaas18:00
harlowjadid u guys figure out the cotyledon or whatever stuff?18:01
harlowjajohnsom ?18:01
johnsomHi, no still broken18:01
harlowjai'm sure medhi would like to know whatever u find out, especially if its a behavior difference18:01
harlowjajust need to get some more logs, lol18:02
harlowjasomebody took yer logs18:02
*** chlong has joined #openstack-lbaas18:03
*** gcheresh has joined #openstack-lbaas18:05
johnsomYeah, I need to play with it more.18:06
johnsomWorking on summit slides at the moment18:06
johnsomAnd working with Adam to un-break the stable branches....  grin18:07
*** voelzmo has joined #openstack-lbaas18:09
*** aojea has joined #openstack-lbaas18:11
*** csomerville has quit IRC18:13
*** aojea has quit IRC18:15
*** voelzmo has quit IRC18:15
*** voelzmo has joined #openstack-lbaas18:16
*** krypto has quit IRC18:28
*** csomerville has joined #openstack-lbaas18:43
*** csomerville has quit IRC18:43
*** csomerville has joined #openstack-lbaas18:44
*** csomerville has quit IRC18:54
*** cody-somerville has joined #openstack-lbaas18:58
*** voelzmo has quit IRC19:01
*** ianychoi_ has joined #openstack-lbaas19:10
*** ianychoi has quit IRC19:13
*** ianychoi_ is now known as ianychoi19:19
*** armax has quit IRC19:30
*** voelzmo has joined #openstack-lbaas19:36
*** aojea has joined #openstack-lbaas19:50
*** cody-somerville has quit IRC19:52
*** voelzmo_ has joined #openstack-lbaas19:56
*** voelzmo has quit IRC19:58
*** voelzmo_ has quit IRC20:01
rm_worklolwut: http://logs.openstack.org/03/445003/1/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/aaeb962/console.html#_2017-05-02_09_49_06_51588220:10
rm_workjohnsom: want me to look at the cotyledon thing?20:10
*** voelzmo has joined #openstack-lbaas20:12
*** voelzmo has quit IRC20:13
*** cody-somerville has joined #openstack-lbaas20:16
*** cpuga has quit IRC20:27
*** blogan_ has joined #openstack-lbaas20:27
*** cody-somerville has quit IRC20:28
*** JudeC has quit IRC20:29
*** blogan has quit IRC20:30
*** Guest75 has joined #openstack-lbaas20:30
*** cody-somerville has joined #openstack-lbaas20:37
*** JudeC has joined #openstack-lbaas20:47
*** armax has joined #openstack-lbaas20:49
*** cody-somerville has quit IRC20:55
*** cody-somerville has joined #openstack-lbaas20:57
*** cody-somerville has quit IRC20:57
*** cody-somerville has joined #openstack-lbaas20:57
*** cody-somerville has quit IRC20:58
*** cody-somerville has joined #openstack-lbaas20:58
*** chlong has quit IRC21:01
*** catintheroof has quit IRC21:01
rm_workjohnsom: ok this one is passing: https://review.openstack.org/#/c/442246/21:02
*** blogan_ has quit IRC21:02
xgermanlet’s see if it progresses further21:08
*** chlong has joined #openstack-lbaas21:16
*** gcheresh has quit IRC21:18
rm_workfixed the newton one i think21:21
rm_workhopefully should pass now21:21
johnsomsudo not found, that is nice21:23
*** dougwig has quit IRC21:25
*** cody-somerville has quit IRC21:25
rm_workyeah IDKWTF21:25
rm_workmight just recheck21:25
johnsomOur luck, someone posted a new cirros image21:26
johnsomOr devstack has played with the glance images more and it's picking yet a different image now21:27
openstackgerritAdam Harwell proposed openstack/octavia master: Switch to use stable remote_client  https://review.openstack.org/44500321:28
johnsomOh, right, we hard coded the image to work around that devstack issue21:28
rm_workgonna rebase it anyway cause whatever21:28
rm_workjohnsom: how do i repro this cotyledon thing21:31
rm_workgot a devstack up21:31
rm_workctrl-c seems to shit it down perfectly21:31
johnsomI do ./clean.sh21:31
rm_workyes, I see the typo, and I refuse to correct it21:31
rm_worklol well, what does clean.sh do?21:31
rm_worki don't want to clean.sh my devstack21:31
johnsom /opt/stack/devstack/clean.sh burns down the devstack21:32
rm_workthat's what "revert to snapshot" or "openstack server delete" is for21:32
johnsomafter it is done you should be able to ps -ef | grep stack and see no processes still running under the stack user21:32
rm_workok so if this is ONLY a problem with clean.sh21:32
rm_workwhy are we even caring21:32
rm_worki thought this was something that could actually happen in a deployment21:33
johnsomWell, I use it, users use it.  If you do this and then startup stack again, your lbaas requests go into the ether.21:33
johnsomWhich becomes a support headache for said PTL21:33
rm_workif you use clean.sh and startup stack again, you're 99% sure to get some other bug too21:34
johnsomWell, I think this will crop up with various init script setups too21:34
johnsomNo, I use it all the time21:34
rm_workthe whole idea of re-doing a stack on the same VM is bad21:34
*** aojea has quit IRC21:35
johnsomSo, a test case that makes you slightly less crazy/rage-y at me.21:35
rm_workhmm, kill -15 works21:36
johnsomIf you kill -15 the controller worker master process, why does it not stop it until after the 60 second "graceful shutdown timeout" expires?21:36
johnsomDid kill -15 work right way or did it have to wait 60 seconds?21:36
rm_workI think that's how cotyledon works21:36
rm_workit has a graceful thing21:37
johnsomYeah, but it's not gracefully stopping, it's timing out and killing it21:37
rm_work2017-05-03 06:35:40.195 25974 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process21:37
rm_work2017-05-03 06:35:40.195 25984 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [25984]21:37
rm_work2017-05-03 06:35:40.195 25986 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(1) [25986]21:37
rm_work2017-05-03 06:35:40.196 25984 INFO octavia.controller.queue.consumer [-] Stopping consumer...21:37
rm_work2017-05-03 06:35:40.196 25986 INFO octavia.controller.queue.consumer [-] Stopping consumer...21:37
johnsomBoth of these behaviors started with this swtich.  They both worked with oslo server21:37
rm_workthat's with kill -1521:37
rm_worklooks like it all shut down nicely21:37
rm_workdidn't take long either21:38
johnsomMine does not do that21:38
rm_workhow long are you waiting after starting it to kill it21:38
rm_worki wonder if there's an initial period or something21:38
johnsomHow are you starting o-cw?21:39
rm_workdevstack default21:39
johnsomYeah, it's up a bit21:39
*** cpuga has joined #openstack-lbaas21:39
rm_worki just went over to the screen it was on21:39
johnsomYeah, well, devstack default just changed....21:39
rm_worki just redid my stack21:39
rm_workand i seem to still be on screen?21:40
johnsomYeah, I could control-c no problem, but if I kill -15 it doesn't do the graceful21:40
rm_workif I ctrl-c it does the SIGINT instant exit21:40
rm_workwhich actually does look like it has an error?21:40
rm_work2017-05-03 06:40:20.855 26953 INFO cotyledon._service_manager [-] Caught SIGINT signal, instantaneous exiting21:40
rm_worko-cw failed to start. Exit code: 121:40
rm_workoh is that second thing a different line21:40
rm_workoutput by the other bit on the console21:40
rm_workk nm works fine21:41
rm_workah hold on that time it took like 10 seconds21:41
rm_workafter the "Stopping consumer..." lines21:41
johnsomI don't get  "Caught SIGTERM signal, graceful exiting of service ConsumerService" nor "Stopping consumer..."21:42
rm_workbut at that point we're inside OUR code21:42
rm_worki wonder if clean.sh is busted21:43
johnsomrestacking to get it back21:43
rm_workwhat signal is it sending21:43
xgermandon’t go that far21:43
johnsomI don't get it by just kill -15, no clean.sh21:43
rm_workugh not like that'll tell you anything, restacks are inherently broken21:43
rm_workso tired of debugging shit because a restack did something dumb21:43
johnsomrm_work no, I restack (via clean.sh/stack.sh) all of the time21:43
rm_workto find out it's not a real problem21:44
rm_workjohnsom: maybe that explains why you are having this weird problem? :P21:44
rm_workI mean21:44
rm_workjust sayin'21:44
johnsomGrrrr, it worked fine with oslo service....21:44
rm_worki stopped having dumb problems once I stopped using restack and just use a fresh VM every time21:45
rm_worki don't even want to think about all the time I wasted before I switched to that workflo21:45
johnsomI don't use restack.sh or rejoin.sh  (both of which I don't think are a thing anymore21:45
xgermanyeah, rejoin is gone21:45
rm_workI always used unstack.sh and stack.sh21:46
johnsomYeah, unstack.sh doesn't work reliabily21:46
rm_workmaybe clean.sh does a better job?21:46
rm_workbut i still don't really trust it21:46
rm_workyeah i dunno, kill -15 seems to be working super reliably here21:50
johnsomYeah, I will test again in about 10 minutes21:50
rm_workif yours is systemd, maybe it's a bug with the systemd stuff?21:50
johnsomNo, I'm on the old screen21:50
johnsomI am 99.9% sure it's that oslo.service dies when the tty goes away and cylon doesn't.  Frankly, we probably need to fix this:21:53
johnsomWhich makes me wonder, did we disable the service in apache with the wsgi patch......21:53
rm_workhmm maybe not?21:54
johnsomOh, yes we did..  good21:54
rm_workexcellent ;P21:54
johnsomOk, stacked.  Let me try something...21:55
* rm_work lets johnsom try something21:56
rm_workbut just this once!21:56
johnsomYeah, ok22:01
johnsomSo, if you kill -15 while it is happy in screen it stops ok22:01
johnsomHowever, while in screen it's parent is bash22:01
johnsomWhen clean runs, it does a "screen -X -S $SESSION quit"22:01
johnsomWhich all of the other octavia process then exit22:02
johnsomoctavia-worker does not and it's parent process is now 122:02
johnsomAt this point kill -15 doesn't work right22:02
johnsomI think we just need to fix the stop function22:03
*** cpuga has quit IRC22:06
openstackgerritJude Cross proposed openstack/python-octaviaclient master: Add Listener commands to client  https://review.openstack.org/46123122:09
*** cpuga has joined #openstack-lbaas22:13
openstackgerritMerged openstack/octavia master: Fix an failure if the vip network has no gateway  https://review.openstack.org/46158222:18
*** cody-somerville has joined #openstack-lbaas22:20
*** cody-somerville has quit IRC22:20
*** cody-somerville has joined #openstack-lbaas22:20
*** chlong has quit IRC22:22
rm_workhere's the next one! http://logs.openstack.org/47/442247/9/check/gate-octavia-v1-dsvm-scenario-ubuntu-xenial/e1c9a46/console.html#_2017-05-02_22_18_31_82676622:37
johnsomThis one we have merged a fix for, it must just not be backported22:38
*** fnaval has quit IRC22:38
johnsomOk, we are idiots sometimes....  Wait until you see the fix for this cylon issue22:47
johnsomYes, I'm going to call it cylon22:47
*** fnaval has joined #openstack-lbaas22:48
*** fnaval has quit IRC22:49
*** fnaval has joined #openstack-lbaas22:50
rm_worklooking for the list_networks fix22:57
*** cpuga has quit IRC22:57
rm_workwas it in... the other stable branch?23:01
rm_workfound it23:06
johnsomYep, that one23:06
openstackgerritMichael Johnson proposed openstack/octavia master: Fix devstack plugin to stop octavia processes  https://review.openstack.org/46192723:10
johnsomRead and weep ^^^^23:10
rm_workgot it i think23:13
rm_workcherry-pick: https://review.openstack.org/#/c/461926/23:13
rm_workand rebased the other on top23:13
rm_workI wonder if they need to be merged together to actually pass >_<23:13
rm_workwe'll see23:13
rm_workjohnsom: wait what about the stop_process for all the other stuff23:14
johnsomWhat other stuff that doesn't have it's own devstack plugin?23:14
rm_worksorry i read that backwards23:15
rm_workyou added it, not removed it23:15
rm_worklooks right23:15
rm_workdid we NEVER have those?23:15
johnsomYeah, it actually works too23:15
rm_workbecause no one cares about cleanup23:16
johnsomYou really don't want to look at the history for that function either....  At one point it was killing haproxy processes....23:16
* rm_work doesn't care about cleanup23:17
rm_worki care so little, I'm +2ing that without testing it, because it looks right...23:17
rm_workand I trust you, the one person ever who tried to use the stop function :P23:17
*** catintheroof has joined #openstack-lbaas23:23
*** catintheroof has quit IRC23:23
*** catintheroof has joined #openstack-lbaas23:24
*** Guest75 has quit IRC23:26
*** catintheroof has quit IRC23:29
*** catintheroof has joined #openstack-lbaas23:31
johnsomrm_work FYI, I am going fix the stats patch23:34
rm_workflying tomorrow, so i'll be out the whole day23:41
rm_workunless i get bored briefly in LAX23:41
johnsomHey, on the project ID thing23:48
johnsomI swear I commented on it, but we can discuss now if you want23:48
rm_workit didn't occur to me before that it's still super useful on responses23:52
rm_workuntil i thought about getting a response without it when querying like23:52
rm_worklike sure, you should know who you are23:53
rm_workit's pretty standard to RETURN on every object23:53
rm_workacross all of openstack23:53
johnsomYeah, I have no problem with the return.  However, I would prefer we leave the project_id in the tables23:54
johnsomIt will make queries much easier/faster23:55
rm_worki guess if you want23:55
johnsomThat or we do foreign key columns23:55
rm_worki don't honestly care23:55
rm_worki just thought removing it was the plan23:55
*** yamamoto has quit IRC23:56
johnsomI'm just thinking of the "get_all" case23:56
johnsomActually any of the gets we are going to filter by project id23:57
rm_workhmm k23:58
*** catintheroof has quit IRC23:59
rm_worki can undo the part that pulls the project_id from the highest level object23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!