| *** puranamr has quit IRC | 00:27 | |
| *** sarob has quit IRC | 00:35 | |
| *** openstack has joined #akanda | 01:22 | |
| -cameron.freenode.net- [freenode-info] if you're at a conference and other people are having trouble connecting, please mention it to staff: http://freenode.net/faq.shtml#gettinghelp | 01:22 | |
| *** jordantardif has quit IRC | 01:25 | |
| *** openstack has joined #akanda | 01:37 | |
| *** openstack has quit IRC | 01:52 | |
| *** openstack has joined #akanda | 01:54 | |
| *** rods has joined #akanda | 10:45 | |
| *** puranamr has joined #akanda | 13:33 | |
| *** puranamr has quit IRC | 13:40 | |
| *** puranamr has joined #akanda | 13:41 | |
| *** rods has quit IRC | 13:45 | |
| *** puranamr has quit IRC | 13:50 | |
| *** puranamr has joined #akanda | 13:50 | |
| *** rods has joined #akanda | 13:54 | |
| *** rods1 has joined #akanda | 13:56 | |
| *** rods1 has quit IRC | 13:56 | |
| *** rods1 has joined #akanda | 13:56 | |
| *** rods1 has quit IRC | 13:57 | |
| *** rods1 has joined #akanda | 13:57 | |
| *** rods has quit IRC | 13:59 | |
| *** puranamr has quit IRC | 14:09 | |
| openstackgerrit | Merged stackforge/akanda-appliance: Restart netfilter-persistent instead of iptables-persistent https://review.openstack.org/188191 | 14:15 |
|---|---|---|
| *** rods has joined #akanda | 15:08 | |
| *** rods2 has joined #akanda | 15:09 | |
| *** rods has quit IRC | 15:09 | |
| *** rods1 has quit IRC | 15:12 | |
| *** puranamr has joined #akanda | 15:43 | |
| *** puranamr has quit IRC | 16:23 | |
| *** puranamr has joined #akanda | 16:28 | |
| *** rods has joined #akanda | 16:36 | |
| *** rods2 has quit IRC | 16:37 | |
| *** sarob has joined #akanda | 16:37 | |
| openstackgerrit | Adam Gandelman proposed stackforge/akanda-rug: Silence exception when provider rules not found https://review.openstack.org/188520 | 17:19 |
| markmcclain | adam_g: was talking with ryanpetrello at lunch today.. with the recent changes you've made to our devstack plugin | 17:33 |
| markmcclain | seems like we might be able to avoid stopping and starting neutron | 17:34 |
| markmcclain | might just need to tweak when we set a few vars so that when neutron does start things will work well | 17:34 |
| adam_g | markmcclain, maybe, im not sure how to hook into the neutron plugin stuff in devstack but its probably doable | 17:35 |
| adam_g | so that when neutron sets itself up before akanda, its doing so with akanda | 17:35 |
| *** puranamr has quit IRC | 17:35 | |
| markmcclain | yeah really it's just making sure the Q_PLUGIN is set right | 17:35 |
| adam_g | oh we can proabbly just set that in akanda-rug/devstack/settings | 17:36 |
| adam_g | markmcclain, this look familiar to you at all? http://logs.openstack.org/94/188194/1/experimental/check-functional-dsvm-akanda-appliance-src/e1e0c28/logs/screen-ak-rug.txt.gz#_2015-06-03_23_40_18 | 17:37 |
| markmcclain | yeah... that's a sign that the config update failed on the appliance | 17:40 |
| *** puranamr has joined #akanda | 17:41 | |
| adam_g | markmcclain, right. wondering if that error points to something specific | 17:41 |
| adam_g | we should get some kinda logging happening inside that we can copy out | 17:42 |
| markmcclain | yeah... so there's some odd thing with gunicorn because originally the appliance would send the traceback | 17:44 |
| markmcclain | traceback with the 500 error | 17:45 |
| markmcclain | sometimes the request processing fails and requests doesn't get a valid response | 17:45 |
| adam_g | yeah, i see tracebacks for other things | 17:46 |
| adam_g | oh, cool. reproduced locally | 17:52 |
| markmcclain | awesome | 18:02 |
| adam_g | is there a way to reapply config without rebuilding? | 18:05 |
| markmcclain | yeah.. the config command should push a new copy of the config | 18:05 |
| markmcclain | rug_ctl router update <uuid> | 18:06 |
| markmcclain | which pushes an update event onto the rug's event queue for the router | 18:06 |
| adam_g | k | 18:10 |
| *** puranamr has quit IRC | 19:05 | |
| *** puranamr has joined #akanda | 19:14 | |
| *** puranamr has quit IRC | 19:28 | |
| *** puranamr has joined #akanda | 19:37 | |
| *** rods1 has joined #akanda | 19:38 | |
| *** puranamr has quit IRC | 19:38 | |
| davidlenwell | markmcclain: ryanpetrello same results on a "hyperspeed" instance .. which is 16 gigs of memory and 8 vcpus | 19:39 |
| davidlenwell | oh weird .. just as I typed that the instance switched to "ACTIVE" | 19:39 |
| davidlenwell | so maybe things are okay and it was just slow to come up this time.. no nova errors | 19:40 |
| markmcclain | yeah.. seems like some strange error in scheduling | 19:40 |
| davidlenwell | anyways .. its active now | 19:40 |
| *** rods has quit IRC | 19:41 | |
| adam_g | interesting | 19:48 |
| adam_g | with some guincorn logging enabled in the appliance, those BadStatusLine errors correspond to a gunicorn [CRITICAL] WORKER TIMEOUT (pid:426) | 19:49 |
| *** sarob has quit IRC | 19:50 | |
| adam_g | the default timeout is 30s, wonder whats going on thats taking so long | 19:51 |
| *** puranamr has joined #akanda | 19:52 | |
| *** puranamr has quit IRC | 19:54 | |
| *** puranamr has joined #akanda | 19:56 | |
| *** sarob has joined #akanda | 20:00 | |
| adam_g | also strange: arping'ing timesout on jessie but works on wheezie | 20:03 |
| ryanpetrello | odd | 20:13 |
| ryanpetrello | I've been seeing an arping issue on our end, too | 20:13 |
| ryanpetrello | been trying to track down what's causing it | 20:13 |
| ryanpetrello | adam_g: I think it's some sort of race with the interface add | 20:21 |
| adam_g | ah | 20:22 |
| adam_g | update_metadata sometimes takes a while | 20:22 |
| adam_g | (re worker timeout) | 20:22 |
| *** sarob has quit IRC | 20:22 | |
| adam_g | ryanpetrello, the arping fails initially on wheezy but later works (afaics), but on jessie it just doesn't ever work | 20:25 |
| *** sarob has joined #akanda | 20:33 | |
| ryanpetrello | yea, I'm seeing the same | 20:35 |
| ryanpetrello | I'm actually working on debugging it currently in our DHC sprint | 20:35 |
| adam_g | so the metadata restart takes >30s and times out the gunicorn worker | 20:36 |
| adam_g | (at least the initial restart) | 20:36 |
| adam_g | that is, in an unaccelearted qemu guest | 20:37 |
| adam_g | i think, in the gate slaves, it times out consistently because theyre slower | 20:37 |
| adam_g | considering raising the gunicorn worker timeout but dont know if there other implications there | 20:37 |
| adam_g | (dont know much about gunicorn at all atm tbh) | 20:38 |
| adam_g | "Generally set to thirty seconds. Only set this noticeably higher if you’re sure of the repercussions for sync workers" | 20:39 |
| adam_g | i have NFI about the repercussions for sync works :) | 20:40 |
| ryanpetrello | heh | 20:40 |
| ryanpetrello | we used to raise this when we tested awhile back on virt-in-virt-less hardware | 20:40 |
| ryanpetrello | the issue, I guess is that you're tying up a sync worker until the HTTP call completes | 20:41 |
| ryanpetrello | (one of the rug workers) | 20:41 |
| ryanpetrello | for your testing, it's probably fine to configure it higher | 20:41 |
| ryanpetrello | since you're really just testing devsack | 20:41 |
| ryanpetrello | *devstack | 20:41 |
| ryanpetrello | and there's really no way of knowing how long it'll take on different hardware | 20:41 |
| *** sarob has quit IRC | 20:41 | |
| adam_g | yea | 20:41 |
| adam_g | maybe expose it as a DIB config option that defaults to 30s? | 20:42 |
| *** sarob has joined #akanda | 20:42 | |
| adam_g | so we can potentially tune later if needed | 20:42 |
| ryanpetrello | yea? | 20:42 |
| ryanpetrello | or maybe just set a saner default, like a minute or so | 20:42 |
| adam_g | or that, i just dont know enough about gunicorn to know how that might affect other htings | 20:44 |
| markmcclain | ok.. so the slaves I guess can't do virt-in-virt that well | 20:58 |
| markmcclain | taking longer than 30s is usually a sign that the instance is backed by qemu or out cpu starved | 20:58 |
| adam_g | markmcclain, probably, they're usually slow (especially on HP Cloud) but even on a non-accelerated qemu here on my system takes 15s-20s to restart the metadata service, and it seems to timeout on the first attempt (probably due to other startup load0 | 21:07 |
| openstackgerrit | Adam Gandelman proposed stackforge/akanda-appliance: Allow for building non-wheezy releases https://review.openstack.org/188194 | 21:16 |
| openstackgerrit | Adam Gandelman proposed stackforge/akanda-appliance: Bump gunicorn worker timeout to 60 https://review.openstack.org/188596 | 21:16 |
| *** sarob has quit IRC | 21:35 | |
| *** puranamr has quit IRC | 22:04 | |
| davidlenwell | is there a reason why our default service image size is 2gb? | 22:06 |
| adam_g | davidlenwell, what do you mean? | 22:09 |
| adam_g | image i just bulit is 472M | 22:09 |
| davidlenwell | installed fresh devstack in an isntance and the service vm uses 2gb of memory.. sorry I should have said service vm flavor uses a lot of ram for some reason | 22:09 |
| adam_g | davidlenwell, oh yea, i tried booting a /w m1.micro but the vm image is too big. we should create our own flavor to use that specifies 512M with enough disk to satisfy the image | 22:10 |
| davidlenwell | would make running it in a vm that has 4gb of ram possible | 22:11 |
| *** puranamr has joined #akanda | 22:13 | |
| *** sarob has joined #akanda | 22:23 | |
| adam_g | sarob, not sure why this is marked as completed, but it is and not showing up in the main blueprint list @ https://blueprints.launchpad.net/akanda/ | 22:26 |
| sarob | adam_g: which what? | 22:26 |
| adam_g | oops | 22:26 |
| adam_g | https://blueprints.launchpad.net/akanda/+spec/ci-updates | 22:26 |
| adam_g | do you have access to change the status? i dont | 22:26 |
| openstackgerrit | Adam Gandelman proposed stackforge/akanda-appliance: Setup logging in the appliance https://review.openstack.org/188617 | 22:44 |
| * adam_g goes to learn more about the miracle of life at baby school | 22:45 | |
| adam_g | back later | 22:45 |
| *** sarob has quit IRC | 22:59 | |
| *** sarob has joined #akanda | 23:09 | |
| *** sarob has quit IRC | 23:21 | |
| *** puranamr has quit IRC | 23:25 | |
| *** puranamr has joined #akanda | 23:28 | |
| *** puranamr has quit IRC | 23:40 | |
| *** puranamr has joined #akanda | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!