| *** vaishali has quit IRC | 00:05 | |
| *** basilAB has quit IRC | 00:06 | |
| rm_work | \o/ | 00:12 |
|---|---|---|
| *** cpuga has joined #openstack-lbaas | 00:12 | |
| *** sanfern has joined #openstack-lbaas | 00:13 | |
| xgerman | ok, still hosed | 00:13 |
| xgerman | (my devstack) | 00:13 |
| johnsom | xgerman what issue? | 00:16 |
| xgerman | the wrong ubuntu packages I assume | 00:16 |
| xgerman | now need to figure out how to restart nova in systemd | 00:17 |
| johnsom | qemu processes hung at 100% cpu? | 00:17 |
| johnsom | http://paste.openstack.org/show/608915/ | 00:17 |
| *** cpuga has quit IRC | 00:17 | |
| xgerman | yep | 00:17 |
| johnsom | systemctl restart devstack@n* | 00:17 |
| rm_work | xgerman: just install devstack with USE_SCREEN=True | 00:18 |
| rm_work | in your localrc | 00:18 |
| johnsom | That won't fix his issue | 00:18 |
| xgerman | nah, I gotta get with the program | 00:19 |
| rm_work | also I recommend : export OCTAVIA_USE_MOD_WSGI=False | 00:19 |
| rm_work | johnsom: it'll fix his "how do I restart nova in systemd" issue :P | 00:19 |
| rm_work | I'm not thoroughly onboard with this "program" in order to "get with it" | 00:19 |
| rm_work | I kinda think I despise the "program" | 00:20 |
| xgerman | well, I am learning | 00:20 |
| rm_work | systemd is a dumpsterfire | 00:20 |
| xgerman | systemd is the future of comouting | 00:20 |
| xgerman | I have seen it in Cloud Fiundry | 00:20 |
| rm_work | the future of computing is a dumpsterfire | 00:20 |
| rm_work | thanks xgerman | 00:20 |
| rm_work | you've ruined my evening | 00:20 |
| xgerman | no more tail -f but commands | 00:20 |
| rm_work | I *like* tail -f | 00:20 |
| xgerman | that was one of my problems with log stash - why no tail -f | grep syntax? | 00:21 |
| rm_work | but besides that, the monolithic unified systemd architecture is just the exact opposite of the philosophy I adhere to | 00:21 |
| johnsom | Yeah, me too. The "journalctl -f -u devstack@o-cw" is umm, yeah, leaves a lot to desired | 00:21 |
| rm_work | which is, keep things small and independent | 00:21 |
| xgerman | at least you can peipe into grep | 00:21 |
| johnsom | You could have files with logstash.... Not so sure with systemd stuff | 00:22 |
| rm_work | devstack does dump logs properly with systemd in the gates, right? | 00:23 |
| johnsom | They do extracts | 00:24 |
| johnsom | It's a bit ugly with the full hostname pre-pended and the color highlighting removed | 00:24 |
| rm_work | anywho I'm looking at the uwsgi stuff, there's a good example here: https://github.com/openstack-dev/devstack/blob/master/lib/keystone#L61 | 00:25 |
| xgerman | anyhow, my single create resuted in an ERROR LB and the logs are - no good | 00:25 |
| xgerman | will try again with debug settings | 00:25 |
| rm_work | thinking about trying to clone the keystone setup really quickly | 00:25 |
| johnsom | rm_work I looked at the uWSGI stuff too. It's seems like there are a lot of opportunities for problems. I don' t think we should merge that until after the lab.... | 00:25 |
| rm_work | yeah | 00:26 |
| rm_work | for sure | 00:26 |
| rm_work | for the lab i'm just disabling mod_wsgi | 00:26 |
| johnsom | Well, c-rap, I don't know why the second node in stable/newton isn't getting all of the services installed. That's devstack stuff and the main node gets it fine | 00:28 |
| xgerman | ok, something is not right with single create | 00:29 |
| johnsom | It worked for me this morning | 00:30 |
| xgerman | ok, there is no ERROR in the logs but the lb comes back with operating_status ERROR | 00:32 |
| xgerman | https://www.irccloud.com/pastebin/aVkJHYAk/ | 00:32 |
| rm_work | what did you pass in? | 00:33 |
| rm_work | i'll try it in my devstack, i think it's running right | 00:33 |
| johnsom | No working members? | 00:33 |
| johnsom | Though I don't think that means lb in error | 00:33 |
| johnsom | hmmm | 00:34 |
| rm_work | xgerman: can you pastebin the ACTUAL return? | 00:34 |
| rm_work | xgerman: since the initial return is the full populated tree | 00:34 |
| rm_work | it'd be nice to see that | 00:34 |
| rm_work | uhh yeah that's weird though | 00:34 |
| rm_work | prov status is ACTIVE O_o | 00:34 |
| rm_work | wtf sets operating_status to ERROR | 00:35 |
| johnsom | HAProxy frontend down | 00:35 |
| rm_work | hmm | 00:35 |
| xgerman | ok, so it’s not single create related | 00:35 |
| rm_work | your devstack is just borked? :/ | 00:36 |
| rm_work | oh err | 00:36 |
| rm_work | so if qemu is broken | 00:36 |
| xgerman | probably | 00:36 |
| rm_work | does the image create properly? | 00:36 |
| rm_work | I think you really have to do the qemu pin BEFORE devstack runs | 00:36 |
| xgerman | it comes up | 00:37 |
| xgerman | I have a ton of haproxies: | 00:37 |
| rm_work | hmm | 00:38 |
| xgerman | https://www.irccloud.com/pastebin/34OI3nEN/ | 00:38 |
| xgerman | which worries me | 00:38 |
| rm_work | uhh wut | 00:38 |
| xgerman | yeah, somethings off | 00:38 |
| xgerman | but if it works for johnsom and rm_work - I am ok to +2 | 00:39 |
| rm_work | hmmmm | 00:39 |
| rm_work | i'd really rather get your testing working tho | 00:39 |
| rm_work | can you pastebin your input json again? | 00:39 |
| xgerman | but I don;t have any of the backends so that’s why I get ERROR | 00:40 |
| xgerman | ok, I think we are good | 00:40 |
| xgerman | +A | 00:41 |
| *** catinthe_ has quit IRC | 00:43 | |
| *** catintheroof has joined #openstack-lbaas | 00:43 | |
| *** catintheroof has quit IRC | 00:44 | |
| rm_work | weird | 00:44 |
| rm_work | aight | 00:44 |
| rm_work | can you pastebin your input again tho | 00:44 |
| rm_work | i want to test it | 00:44 |
| *** tongl has quit IRC | 00:45 | |
| rm_work | xgerman: ^^ | 00:46 |
| xgerman | https://www.irccloud.com/pastebin/nl89bGZU/ | 00:46 |
| rm_work | or is it the same you posted in a comment a while back | 00:46 |
| xgerman | this is what I send to the LB | 00:46 |
| xgerman | yep | 00:48 |
| xgerman | same | 00:48 |
| rm_work | ok here's what my initial return is: http://paste.openstack.org/show/609040/ | 00:50 |
| rm_work | let's see now if it went active | 00:50 |
| xgerman | yep, same for me | 00:50 |
| rm_work | hmmm | 00:51 |
| rm_work | no, i actually got the same as you this time | 00:51 |
| rm_work | prov goes active, but operating is error | 00:51 |
| rm_work | i wonder if a rebase broke something | 00:51 |
| rm_work | or if something in devstack broke | 00:51 |
| rm_work | cause yeah, i also see no errors | 00:51 |
| rm_work | I do see two Mark ACTIVE for the same LB | 00:52 |
| rm_work | which is odd | 00:52 |
| rm_work | curious where those are coming from | 00:52 |
| xgerman | i think expected since members are all down | 00:52 |
| rm_work | yeah ok but what is up | 00:53 |
| xgerman | I think when all members are down we go ERROR instead of DEGRADED but memory is fuzzy | 00:54 |
| xgerman | so it would end up as expected | 00:55 |
| xgerman | afk for a while | 00:55 |
| rm_work | ah yeah let me delete the HM | 00:56 |
| rm_work | I have a ton of haproxy too | 00:57 |
| rm_work | but | 00:57 |
| rm_work | does it spawn one process per listener? | 00:57 |
| johnsom | Yes, sadly | 00:57 |
| rm_work | lol | 00:57 |
| johnsom | My big argument with Stephen... | 00:57 |
| rm_work | k | 00:57 |
| rm_work | you know | 00:57 |
| rm_work | we can undo that | 00:57 |
| rm_work | it'll just be a bit of work | 00:57 |
| johnsom | Oh I know. It's just priorities | 00:57 |
| johnsom | It would actually simplify a lot of the amp agent code | 00:58 |
| rm_work | uhhh | 00:59 |
| rm_work | hmm | 00:59 |
| rm_work | looking closely at this | 00:59 |
| johnsom | Ok folks. I'm going to step away for a bit and make dinner. Later I guess I will be learning way more than I want about devstack multinode | 01:00 |
| rm_work | the pools and HMs seem to be stuck in PENDING_CREATE for some reason | 01:00 |
| rm_work | wtf happened to this | 01:00 |
| rm_work | i guess rebasing like 10 times is what happened | 01:00 |
| rm_work | Listeners went ACTIVE tho | 01:00 |
| rm_work | but on the controller side this is the same flow as the v1 create... | 01:01 |
| rm_work | did the v1 create break too? | 01:01 |
| rm_work | *v1 single-create | 01:09 |
| rm_work | aaaand it failed in the gate anyway | 01:10 |
| rm_work | with the same provides issue xgerman saw | 01:10 |
| rm_work | so IDKWTF | 01:11 |
| rm_work | I'm gonna grab dinner and then hack at this a bit | 01:11 |
| rm_work | something is actually borked | 01:11 |
| rm_work | but it's weird because i didn't touch anything on the flow side | 01:12 |
| rm_work | it should be the same as v1 | 01:12 |
| rm_work | yeah ok so without HMs it does go ONLINE | 01:17 |
| rm_work | but | 01:17 |
| rm_work | the pool prov statuses is wrong | 01:17 |
| rm_work | almost done with fix | 01:40 |
| *** sanfern has quit IRC | 01:55 | |
| *** armax has quit IRC | 01:57 | |
| *** gongysh has quit IRC | 01:57 | |
| rm_work | OK | 02:05 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 02:05 |
| rm_work | I don't know why I had to do a bunch of work on the backend | 02:05 |
| rm_work | but there it is | 02:05 |
| rm_work | maybe we never fully tested single-create for v1? lol | 02:05 |
| rm_work | re-spinning my stack with that | 02:05 |
| rm_work | but i tested by changing manually and it seemed good | 02:05 |
| *** fnaval has joined #openstack-lbaas | 02:09 | |
| *** yamamoto has joined #openstack-lbaas | 02:20 | |
| *** basilAB has joined #openstack-lbaas | 02:27 | |
| *** vaishali has joined #openstack-lbaas | 02:28 | |
| *** yamamoto has quit IRC | 02:33 | |
| rm_work | oops | 02:49 |
| rm_work | forgot members | 02:49 |
| *** cody-somerville has quit IRC | 03:08 | |
| *** cody-somerville has joined #openstack-lbaas | 03:09 | |
| *** cody-somerville has quit IRC | 03:09 | |
| *** cody-somerville has joined #openstack-lbaas | 03:09 | |
| *** fnaval has quit IRC | 03:15 | |
| *** jerrygb has joined #openstack-lbaas | 03:21 | |
| *** yamamoto has joined #openstack-lbaas | 03:34 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 03:40 |
| *** yamamoto has quit IRC | 03:41 | |
| rm_work | ugh not having cascade delete sux | 03:43 |
| rm_work | https://review.openstack.org/#/c/461182/ | 03:43 |
| rm_work | johnsom: WTB | 03:43 |
| rm_work | oops and HMs and members didn't have project_id set right | 03:56 |
| rm_work | womp womp | 03:56 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add support for single-create for APIv2 https://review.openstack.org/459032 | 03:58 |
| rm_work | ok should be good to go now | 03:58 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Implement sorting and pagination for octavia https://review.openstack.org/382147 | 04:11 |
| rm_work | k | 04:11 |
| rm_work | rebased | 04:11 |
| *** cpuga has joined #openstack-lbaas | 04:14 | |
| *** jerrygb has quit IRC | 04:14 | |
| *** cpuga has quit IRC | 04:19 | |
| *** catintheroof has joined #openstack-lbaas | 04:21 | |
| *** catintheroof has quit IRC | 04:32 | |
| rm_work | tempted to shove cascade-delete on the end of that chain | 04:36 |
| rm_work | preemptively | 04:36 |
| johnsom | Curious to see what my test didn't catch on cascade. It built fine and passed traffic for me. | 05:10 |
| *** fnaval has joined #openstack-lbaas | 05:10 | |
| *** gcheresh has joined #openstack-lbaas | 05:12 | |
| rm_work | cascade? | 05:17 |
| rm_work | yeah it should be good | 05:17 |
| rm_work | johnsom: if we merge it now, that's fine | 05:18 |
| rm_work | i was thinking you were testing single create first | 05:18 |
| johnsom | I am talking about single | 05:18 |
| rm_work | oh | 05:18 |
| rm_work | statuses of subobjects | 05:19 |
| rm_work | it wasn't ever setting them to ACTIVE | 05:19 |
| rm_work | so the LB was | 05:19 |
| rm_work | but nothing else was | 05:19 |
| *** fnaval has quit IRC | 05:19 | |
| rm_work | in v1 it didn't matter | 05:19 |
| rm_work | because all the statuses are new in v2 :P | 05:19 |
| rm_work | i forgot that part | 05:19 |
| rm_work | diff the last.... 2 or 3 | 05:19 |
| rm_work | from 20 i think | 05:20 |
| rm_work | also wasn't setting project_id on HM or members | 05:26 |
| rm_work | members it doesn't matter so much but | 05:27 |
| rm_work | HM you get nothing back from the /healthmonitors endpoint GET | 05:27 |
| rm_work | fixed now | 05:28 |
| rm_work | also it was setting the LB ACTIVE before it was done | 05:28 |
| rm_work | (and twice) | 05:28 |
| rm_work | johnsom: i'll hit you up tomorrow maybe, if you're on | 06:08 |
| rm_work | last full day for me to work before i'm on a plane to boston <_< | 06:09 |
| johnsom | Ok | 06:09 |
| rm_work | ah you're still around? | 06:09 |
| rm_work | this single-create seems good now | 06:10 |
| rm_work | i just forgot to look at sub-objects until xgerman's example | 06:10 |
| johnsom | I am awake, but on mobile | 06:10 |
| rm_work | ah lol then yeah, catch you tomorrow? | 06:11 |
| rm_work | one of these days I'm going to have a weekend off | 06:11 |
| rm_work | after boston, and after the next week I think <_< | 06:11 |
| johnsom | I hope so! | 06:11 |
| johnsom | Hey, thanks for all your work and code! | 06:12 |
| johnsom | You have been rocking it | 06:12 |
| johnsom | I hope you feel better too | 06:12 |
| rm_work | yeah... i'll be super happy when this thing is *working* | 06:12 |
| rm_work | completely on v2 | 06:12 |
| rm_work | just filters and stats and status-tree to go | 06:12 |
| rm_work | after the ones i have in-flight | 06:13 |
| *** gcheresh has quit IRC | 06:17 | |
| *** voelzmo has joined #openstack-lbaas | 06:18 | |
| *** voelzmo has quit IRC | 06:19 | |
| *** voelzmo has joined #openstack-lbaas | 06:20 | |
| *** voelzmo has quit IRC | 06:25 | |
| *** krypto has joined #openstack-lbaas | 06:36 | |
| *** krypto has quit IRC | 06:36 | |
| *** krypto has joined #openstack-lbaas | 06:36 | |
| *** voelzmo has joined #openstack-lbaas | 06:36 | |
| *** voelzmo has quit IRC | 06:41 | |
| *** pcaruana has joined #openstack-lbaas | 06:50 | |
| *** gcheresh has joined #openstack-lbaas | 06:52 | |
| *** gcheresh has quit IRC | 07:46 | |
| *** cpuga has joined #openstack-lbaas | 08:15 | |
| *** cpuga has quit IRC | 08:21 | |
| *** gcheresh has joined #openstack-lbaas | 08:32 | |
| *** krypto has quit IRC | 08:35 | |
| *** krypto has joined #openstack-lbaas | 08:37 | |
| *** gcheresh has quit IRC | 08:45 | |
| *** krypto has quit IRC | 09:53 | |
| *** gcheresh has joined #openstack-lbaas | 09:54 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add cascade delete for APIv2 https://review.openstack.org/461182 | 10:27 |
| *** yamamoto has joined #openstack-lbaas | 11:20 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** yamamoto has joined #openstack-lbaas | 11:58 | |
| *** yamamoto has quit IRC | 12:02 | |
| *** cpuga has joined #openstack-lbaas | 12:18 | |
| *** cpuga has quit IRC | 12:22 | |
| *** yamamoto has joined #openstack-lbaas | 13:02 | |
| *** gcheresh has quit IRC | 13:06 | |
| *** yamamoto has quit IRC | 13:06 | |
| xgerman | FYI: https://t.co/UX8ixpdwA2 - | 13:07 |
| *** yamamoto has joined #openstack-lbaas | 13:19 | |
| *** krypto has joined #openstack-lbaas | 13:24 | |
| *** chlong has quit IRC | 13:34 | |
| *** krypto has quit IRC | 13:45 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** gcheresh has joined #openstack-lbaas | 14:36 | |
| *** armax has joined #openstack-lbaas | 14:45 | |
| *** yamamoto has joined #openstack-lbaas | 14:59 | |
| *** yamamoto has quit IRC | 15:09 | |
| *** gcheresh has quit IRC | 15:15 | |
| *** gcheresh has joined #openstack-lbaas | 15:22 | |
| *** armax has quit IRC | 15:27 | |
| *** sanfern has joined #openstack-lbaas | 15:34 | |
| *** jerrygb has joined #openstack-lbaas | 16:18 | |
| *** jerrygb has quit IRC | 17:06 | |
| *** jerrygb has joined #openstack-lbaas | 17:09 | |
| *** fnaval has joined #openstack-lbaas | 17:47 | |
| *** JudeC has joined #openstack-lbaas | 17:52 | |
| *** fnaval_ has joined #openstack-lbaas | 18:02 | |
| *** jerrygb has quit IRC | 18:04 | |
| *** fnaval has quit IRC | 18:05 | |
| *** aojea has joined #openstack-lbaas | 18:09 | |
| *** jerrygb has joined #openstack-lbaas | 19:10 | |
| *** jerrygb has quit IRC | 19:15 | |
| *** pcaruana has quit IRC | 19:16 | |
| *** fnaval has joined #openstack-lbaas | 19:42 | |
| *** fnaval_ has quit IRC | 19:45 | |
| *** jerrygb has joined #openstack-lbaas | 20:23 | |
| *** cpuga has joined #openstack-lbaas | 20:47 | |
| *** cpuga has quit IRC | 20:51 | |
| *** jerrygb has quit IRC | 20:52 | |
| *** gcheresh has quit IRC | 21:01 | |
| *** jerrygb has joined #openstack-lbaas | 21:37 | |
| *** jerrygb has quit IRC | 21:39 | |
| *** jerrygb has joined #openstack-lbaas | 21:40 | |
| *** jerrygb has quit IRC | 21:44 | |
| *** catintheroof has joined #openstack-lbaas | 22:35 | |
| *** fnaval_ has joined #openstack-lbaas | 22:37 | |
| *** fnaval has quit IRC | 22:40 | |
| *** aojea has quit IRC | 22:47 | |
| *** catintheroof has quit IRC | 23:05 | |
| *** catintheroof has joined #openstack-lbaas | 23:06 | |
| rm_work | so i'm kinda around | 23:14 |
| rm_work | ping me if anyone is actually working | 23:14 |
| *** catintheroof has quit IRC | 23:32 | |
| *** armax has joined #openstack-lbaas | 23:38 | |
| *** fnaval_ has quit IRC | 23:53 | |
| *** fnaval has joined #openstack-lbaas | 23:54 | |
| *** armax has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!