| clarkb | but I too do not grok the ruby | 00:00 |
|---|---|---|
| mordred | kk | 00:00 |
| *** peristeri has quit IRC | 00:00 | |
| mordred | also - what's a defaultfor declaration? | 00:00 |
| fungi | clarkb: should be safe to undo the fw rule and reenable puppet agent on the frontends, yeah | 00:01 |
| fungi | i've been sniffing for that source address and see nothing coming from it now | 00:01 |
| clarkb | mordred: Ithink that goes in your puppet manifest so that puppet knows what a default for a resource is? | 00:01 |
| clarkb | fungi: doing that now | 00:01 |
| * nibalizer punts to crinkle | 00:02 | |
| clarkb | and done | 00:02 |
| * fungi too is sorely lacking in rupies | 00:02 | |
| fungi | or rupees | 00:02 |
| *** zz_dimtruck is now known as dimtruck | 00:03 | |
| clarkb | mordred: I was wrong https://docs.puppetlabs.com/guides/provider_development.html#default-providers | 00:03 |
| *** melwitt has quit IRC | 00:03 | |
| nibalizer | hrm that docs should probably use osfamily instaed of operatingsystem | 00:04 |
| clarkb | mordred: looks like that goes in your provider and you would basically say defaultfor :somefact => value | 00:04 |
| clarkb | mordred: in this case I think defaultfor :useexim => true | 00:04 |
| clarkb | where I just made up that mapping | 00:04 |
| mordred | is useexim a fact? | 00:05 |
| clarkb | no | 00:05 |
| mordred | yeah - I need a fact to match on | 00:05 |
| *** rmcall has quit IRC | 00:05 | |
| *** camunoz has quit IRC | 00:05 | |
| *** camunoz has joined #openstack-infra | 00:05 | |
| clarkb | but I need to afk now, if I don't do grocery shopping I won't have any food | 00:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-mailman: Add non-alias-modifying mailman provider https://review.openstack.org/160343 | 00:06 |
| mordred | ok. there is the remove-the-method thing | 00:06 |
| mordred | I don't see a way to do defaultfor that makes sense | 00:06 |
| mordred | although Iagree with teh theory | 00:06 |
| mordred | I'd suggest that "what mail system is in use" should be a fact | 00:06 |
| clarkb | mordred: ya I think that would be the correct way to defaultfor but would need the fact to be implemented | 00:07 |
| mordred | because honestly - exim and sendmail and postfix and qmail are different | 00:07 |
| mordred | :) | 00:07 |
| clarkb | so I am good setting the provider for now and getting that in to | 00:07 |
| mordred | ++ | 00:07 |
| *** rmcall has joined #openstack-infra | 00:09 | |
| mordred | SpamapS: so - I have a patch up you might be interested in | 00:09 |
| mordred | SpamapS: 177599 | 00:09 |
| *** otter768 has joined #openstack-infra | 00:10 | |
| mordred | SpamapS: if you apply that, it's possible to source a bunch of OS_ env vars that correspond to a real cloud account on a real cloud | 00:10 |
| mordred | and then run the functional tests in shade | 00:10 |
| mordred | SpamapS: I have tested the current functinoal tests against an account at rax | 00:10 |
| *** rmcall has quit IRC | 00:10 | |
| mordred | SpamapS: so once you get an account, I suggest trying that | 00:10 |
| ianw | now the crisis is over, can someone look at https://review.openstack.org/#/c/177006/ (Fix swapiness sysctl set) ... centos7 is unstable now it's doing more than smoke testing and i want to make sure i'm getting consistent tests | 00:11 |
| mordred | SpamapS: you'll want these set: http://paste.openstack.org/show/211029 | 00:11 |
| mordred | ianw: looking | 00:11 |
| mordred | ianw: +A | 00:12 |
| openstackgerrit | Merged openstack-infra/project-config: Correct output path for infra-site index https://review.openstack.org/177839 | 00:13 |
| ianw | mordred: thanks, i'll work on getting the +e checking back around all the ansible stuff | 00:13 |
| ianw | we shouldn't be missing typos like that | 00:13 |
| *** sputnik13 has quit IRC | 00:13 | |
| *** otter768 has quit IRC | 00:14 | |
| *** claudiub has quit IRC | 00:15 | |
| *** salv-orlando has quit IRC | 00:16 | |
| * crinkle wonders what was punted to her | 00:20 | |
| *** rmcall has joined #openstack-infra | 00:21 | |
| nibalizer | clarkb: mordred would like one of us to just push a new patchset to https://review.openstack.org/#/c/160343/ although maybe monty did it | 00:21 |
| *** mtanino_ has quit IRC | 00:21 | |
| *** achanda has joined #openstack-infra | 00:23 | |
| mordred | I did the main part | 00:23 |
| mordred | but there was a comment about defaultfor that I could not figure out a tangible way to implement | 00:23 |
| mordred | since there are no relevant facts to attach it to | 00:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Put input variables into ansible inventory https://review.openstack.org/177943 | 00:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: Move all the ansible calls into playbooks https://review.openstack.org/177944 | 00:24 |
| mordred | clarkb: fixed your comments ^^ also left a response where I mostly agree with you about whether or not eh patches are a good idea | 00:26 |
| *** armax has joined #openstack-infra | 00:29 | |
| *** bknudson has quit IRC | 00:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 00:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Fix swapiness sysctl set https://review.openstack.org/177006 | 00:31 |
| *** david-lyle has quit IRC | 00:32 | |
| ianw | mordred: doh, i think i rebased that, appreciate if you could look | 00:32 |
| ianw | ^ | 00:32 |
| *** Somay has quit IRC | 00:34 | |
| nibalizer | mordred: :kernel => 'Linux' /mike drop | 00:34 |
| greghaynes | poor mik | 00:35 |
| greghaynes | mike | 00:35 |
| mordred | nibalizer: wow. yeah? | 00:35 |
| nibalizer | mordred: why not its a site specific configuration | 00:35 |
| crinkle | that was sort of what i had in mind | 00:35 |
| crinkle | otherwise you'd have to give provider => noaliasmailman to every resource that used that type | 00:35 |
| crinkle | i'm not sure if there's a more graceful way to do it | 00:36 |
| * crinkle can ponder it later | 00:36 | |
| nibalizer | clarkb: well could set a default provider in the manifest | 00:36 |
| *** Somay has joined #openstack-infra | 00:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-mailman: Add non-alias-modifying mailman provider https://review.openstack.org/160343 | 00:36 |
| mordred | like that? | 00:36 |
| crinkle | mordred: nibalizer's suggestion is probably less hacky | 00:37 |
| *** Somay has quit IRC | 00:37 | |
| * nibalizer will keep providing terrible suggestions | 00:37 | |
| *** jtriley has joined #openstack-infra | 00:37 | |
| nibalizer | also is it :linux or "Linux"? I can't tell from the docs | 00:38 |
| crinkle | :linux i think | 00:39 |
| * crinkle not sure if it matters | 00:39 | |
| *** tiswanso has joined #openstack-infra | 00:41 | |
| *** jtriley has quit IRC | 00:42 | |
| *** tiswanso_ has joined #openstack-infra | 00:45 | |
| *** tiswanso has quit IRC | 00:46 | |
| *** asettle has joined #openstack-infra | 00:49 | |
| *** Somay has joined #openstack-infra | 00:49 | |
| *** bswartz has quit IRC | 00:53 | |
| *** bswartz has joined #openstack-infra | 00:54 | |
| *** asettle has quit IRC | 00:54 | |
| *** asettle has joined #openstack-infra | 00:56 | |
| *** asettle has quit IRC | 00:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 00:57 |
| *** signed8bit has joined #openstack-infra | 00:58 | |
| *** asettle has joined #openstack-infra | 00:59 | |
| *** tjones1 has joined #openstack-infra | 01:01 | |
| *** zhiwei has joined #openstack-infra | 01:01 | |
| *** xyang1 has quit IRC | 01:05 | |
| *** emagana has joined #openstack-infra | 01:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Fix functional tests to run against live clouds https://review.openstack.org/177599 | 01:09 |
| mordred | jhesketh: if you're bored ^^ | 01:09 |
| *** woodster_ has quit IRC | 01:10 | |
| *** emagana has quit IRC | 01:10 | |
| *** emagana has joined #openstack-infra | 01:11 | |
| jhesketh | mordred: I'm never bored, but always happy to take a look | 01:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 01:16 |
| *** salv-orlando has joined #openstack-infra | 01:17 | |
| *** spzala has joined #openstack-infra | 01:20 | |
| *** david-lyle has joined #openstack-infra | 01:20 | |
| *** tjones1 has quit IRC | 01:21 | |
| *** salv-orlando has quit IRC | 01:22 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** jamesmcarthur has joined #openstack-infra | 01:24 | |
| *** yamamoto has joined #openstack-infra | 01:24 | |
| *** erikmwilson has quit IRC | 01:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 01:25 |
| ianw | bash really needs goto | 01:25 |
| openstackgerrit | Angus Salkeld proposed openstack-infra/project-config: Add missing ceilometer services to heat job https://review.openstack.org/178472 | 01:27 |
| mordred | ianw: that's the sickest thing I've ever heard | 01:30 |
| greghaynes | ianw: ive seen hacks that do it | 01:31 |
| greghaynes | IIRC they basically reeval'd the script at that line | 01:31 |
| greghaynes | not that im encouraging this | 01:31 |
| ianw | mordred: hey, you wrote all this ansible stuff into the scripts :) "goto cleanup" is the best way to deal with failures | 01:31 |
| SpamapS | any argument that includes bash needs more language features should also acknowledge that kittens _WILL DIE_ as a result of said argument. | 01:32 |
| mordred | ianw: heh | 01:32 |
| ianw | http://logs.openstack.org/17/177017/21/check/check-tempest-dsvm-full/0087919//logs/ <- this is how we should handle errors in the scripts, i think | 01:32 |
| ianw | i'm not 100% how those logs are picked up when the cleanup fn didn't run, but anyway... | 01:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb: Add a command to generate configs for hound https://review.openstack.org/178474 | 01:34 |
| mordred | ok. so, I'm on a plane - which means it's ok for me to waste time ^^ | 01:34 |
| SpamapS | ianw: technically traps are goto's in bash btw | 01:36 |
| *** markvoelker has quit IRC | 01:36 | |
| SpamapS | closest thing anyway | 01:36 |
| SpamapS | not technically.. arguably I suppose | 01:37 |
| ianw | SpamapS: i kind of want to keep away from them, to avoid nesting problems when running arbitrary gate functions others might provide | 01:37 |
| *** marun has quit IRC | 01:38 | |
| SpamapS | ianw: wait, what exactly are we bashing here? | 01:38 |
| *** rmcall has quit IRC | 01:38 | |
| ianw | SpamapS: i'm beating in https://review.openstack.org/#/c/177017/ into some sort of workability | 01:39 |
| *** otter768 has joined #openstack-infra | 01:41 | |
| SpamapS | ianw: yeah, if you could nest traps easily that would be useful in this case. | 01:41 |
| SpamapS | I once made it work sort of generically | 01:41 |
| SpamapS | then recoiled in horror and threw the implementation into a boiling vat of sulfuric acid | 01:42 |
| *** jtriley has joined #openstack-infra | 01:43 | |
| *** jtriley has quit IRC | 01:48 | |
| *** yamahata has quit IRC | 01:48 | |
| *** signed8bit has quit IRC | 01:48 | |
| *** mwagner_lap has quit IRC | 01:49 | |
| *** baoli has joined #openstack-infra | 01:49 | |
| *** dims_ is now known as dims | 01:52 | |
| *** Somay has quit IRC | 01:52 | |
| *** baoli has quit IRC | 01:53 | |
| *** baoli has joined #openstack-infra | 01:53 | |
| *** camunoz has quit IRC | 01:54 | |
| *** Somay has joined #openstack-infra | 01:55 | |
| *** yamamoto has quit IRC | 01:56 | |
| *** bnemec has quit IRC | 01:56 | |
| *** asettle has quit IRC | 01:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 01:58 |
| greghaynes | mordred: just released all the dib goodness | 01:58 |
| *** bnemec has joined #openstack-infra | 01:59 | |
| SpamapS | ooooo | 01:59 |
| SpamapS | test 'sploded nice first try | 01:59 |
| SpamapS | http://paste.ubuntu.com/10933318/ | 01:59 |
| SpamapS | greghaynes: may god have mercy on dib's soul :) | 02:00 |
| greghaynes | and its users | 02:00 |
| *** dannywilson has joined #openstack-infra | 02:02 | |
| SpamapS | mordred: wait.. is there no v2 CLI in glanceclient? | 02:03 |
| *** harlowja is now known as harlowja_away | 02:03 | |
| *** sputnik13 has joined #openstack-infra | 02:04 | |
| greghaynes | muahahha | 02:04 |
| greghaynes | so, AIUI, v2 means upload to swift, and task-create to import | 02:04 |
| SpamapS | right but task-create where? | 02:04 |
| SpamapS | glanceclient has no task-anything | 02:04 |
| greghaynes | glance task-create | 02:04 |
| greghaynes | what | 02:04 |
| greghaynes | mine does | 02:04 |
| SpamapS | ubuntu@devstack-trusty:~/shade$ glance --version | 02:05 |
| SpamapS | 0.16.1 | 02:05 |
| *** dimtruck is now known as zz_dimtruck | 02:05 | |
| greghaynes | SpamapS: You may need to set export OS_IMAGE_API_VERSION=2 | 02:06 |
| SpamapS | Oh thats just insane | 02:06 |
| *** kiseok7_ has quit IRC | 02:06 | |
| SpamapS | to hide it from the HELP? | 02:06 |
| greghaynes | :) | 02:06 |
| greghaynes | welcome to the jungle | 02:06 |
| *** mwagner_lap has joined #openstack-infra | 02:06 | |
| SpamapS | "its a secret!!!" | 02:06 |
| SpamapS | I'm upgrading to trunk first | 02:06 |
| SpamapS | just to see | 02:06 |
| *** camunoz has joined #openstack-infra | 02:07 | |
| greghaynes | im on 0.17.0 | 02:07 |
| SpamapS | of course now I see my system cryptography upgrading and I'm eh-scared | 02:07 |
| greghaynes | haha | 02:07 |
| * greghaynes needs to afk and dinner for a couple | 02:07 | |
| *** HeOS has quit IRC | 02:09 | |
| *** HeOS has joined #openstack-infra | 02:10 | |
| *** baoli has quit IRC | 02:11 | |
| dims | anyone poked at this error? | 02:11 |
| dims | "fatal: unable to access 'https://git.openstack.org/openstack/nova/': GnuTLS recv error (-9): A TLS packet with unexpected length was received." | 02:11 |
| SpamapS | whhhatttt... | 02:11 |
| SpamapS | ImportError: No module named glanceclient | 02:11 |
| SpamapS | ubuntu@devstack-trusty:/opt/stack/python-glanceclient$ ls -l /usr/local/lib/python2.7/dist-packages/glanceclient | 02:11 |
| SpamapS | so.. confused.. | 02:11 |
| lifeless | SpamapS: is /usr/local//.... on your PYTHONPATH ? | 02:12 |
| lifeless | python -c 'import sys; print(sys.path)' | 02:12 |
| *** baoli has joined #openstack-infra | 02:12 | |
| SpamapS | lifeless: yes | 02:13 |
| *** woodster_ has joined #openstack-infra | 02:13 | |
| *** sputnik13 has quit IRC | 02:13 | |
| mordred | greghaynes: woot! | 02:13 |
| mordred | SpamapS: also - yes re: v2 being hidden. wtf | 02:13 |
| SpamapS | http://paste.ubuntu.com/10933477/ | 02:14 |
| SpamapS | I notice there's no __init__.py in glanceclient/ | 02:14 |
| mordred | SpamapS: that seems broken | 02:14 |
| SpamapS | so confused | 02:15 |
| SpamapS | they're all deleted in my git tree | 02:15 |
| SpamapS | no idea why | 02:15 |
| *** sputnik13 has joined #openstack-infra | 02:15 | |
| mordred | SpamapS: because you were a naughty bow | 02:15 |
| mordred | or boy | 02:15 |
| *** sputnik13 has quit IRC | 02:15 | |
| SpamapS | devstack punished me? | 02:15 |
| mordred | ya | 02:15 |
| SpamapS | $ glance help | 02:16 |
| SpamapS | 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult' | 02:16 |
| SpamapS | the fun continues | 02:16 |
| mordred | SpamapS: what did you do??? | 02:16 |
| *** dannywilson has quit IRC | 02:16 | |
| SpamapS | I think I summoned a Balrog | 02:16 |
| SpamapS | or Gozer | 02:16 |
| SpamapS | either way I'm pretty sure I have to duel it for eternity now | 02:16 |
| *** dannywilson has joined #openstack-infra | 02:17 | |
| *** Somay has quit IRC | 02:18 | |
| SpamapS | picking up system six | 02:19 |
| *** jamesmcarthur has quit IRC | 02:19 | |
| nibalizer | SpamapS: are you a god? | 02:19 |
| *** zz_dimtruck is now known as dimtruck | 02:20 | |
| SpamapS | nibalizer: eleventeen | 02:20 |
| *** ddieterly has quit IRC | 02:20 | |
| SpamapS | wouldn't it make sense for the default path to have /usr/local/lib before /usr/lib ? | 02:21 |
| *** tjones1 has joined #openstack-infra | 02:21 | |
| lifeless | SpamapS: yes | 02:21 |
| *** dannywilson has quit IRC | 02:21 | |
| lifeless | SpamapS: also its a lot better to use a venv if you're replacing system things | 02:22 |
| SpamapS | lifeless: have you fallen to the bottom of this rabbit hole before? | 02:22 |
| lifeless | SpamapS: avoids any confusion about versions etc | 02:22 |
| SpamapS | perhaps devstack is using venvs and I missed that | 02:22 |
| *** tjones1 has left #openstack-infra | 02:22 | |
| lifeless | suspect not | 02:22 |
| lifeless | /u/l/l is not a venv path | 02:22 |
| SpamapS | well I was trying to install newer glanceclient so I pip installed it | 02:22 |
| SpamapS | but I didn't bother to pay attention to the one I replaced | 02:23 |
| lifeless | SpamapS: oh | 02:23 |
| * SpamapS checks the others | 02:23 | |
| lifeless | SpamapS: are you running vivid? | 02:23 |
| SpamapS | no they're not venv installed | 02:23 |
| SpamapS | lifeless: trusty vm | 02:23 |
| lifeless | SpamapS: ok phew, eliminates that problem | 02:23 |
| SpamapS | vivid have some new fun? | 02:23 |
| *** unicell1 has quit IRC | 02:24 | |
| *** Swami_ has quit IRC | 02:24 | |
| lifeless | yah | 02:24 |
| SpamapS | wow I have no idea how I got from having a working 0.16.x to non-working thing | 02:30 |
| SpamapS | but at this point, just venv'd and moving on with life | 02:30 |
| * SpamapS will burn this vm later :-P | 02:30 | |
| *** david-lyle has quit IRC | 02:30 | |
| *** david-lyle has joined #openstack-infra | 02:31 | |
| *** patrickeast has quit IRC | 02:32 | |
| SpamapS | yeah so thats amazing, glance's help adapts to the api version | 02:32 |
| SpamapS | instead of showing you all commands, and saying something like "v2" "v1" | 02:32 |
| SpamapS | oy | 02:32 |
| lifeless | SpamapS: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1419695 | 02:33 |
| openstack | Launchpad bug 1419695 in pip "Should default to --user to not fail default pip install usage" [Undecided,New] | 02:33 |
| SpamapS | lifeless: oh... my | 02:34 |
| lifeless | upstream -> ubuntu 'please don't do this' | 02:34 |
| lifeless | ubuntu -> upstream 'you're too slow, deal with it' | 02:34 |
| SpamapS | thats just amazing | 02:35 |
| *** sputnik13 has joined #openstack-infra | 02:35 | |
| SpamapS | As is the fact that the glance task API doesn't allow you to cancel things. | 02:35 |
| lifeless | I strongly suspect that sudo python -m pip install -U pip done twice will confuse the hell outta folk | 02:35 |
| lifeless | first one will install --user; second will run the new one and install dist wide | 02:35 |
| lifeless | next run after that will no longer do --user by default. | 02:35 |
| lifeless | HILARIOUS | 02:36 |
| SpamapS | well | 02:36 |
| *** ayoung has joined #openstack-infra | 02:36 | |
| *** david-lyle has quit IRC | 02:36 | |
| SpamapS | at least if you hit it enough times, it behaves | 02:36 |
| lifeless | so | 02:36 |
| lifeless | also | 02:36 |
| mordred | wow | 02:36 |
| lifeless | first time will no longer upgrade it for other users | 02:36 |
| mordred | have I mentioned that I avoid python-pip like the plague? | 02:37 |
| SpamapS | yeah seriously, blacklist that thing | 02:37 |
| SpamapS | we should mirror get-pip.py btw | 02:37 |
| lifeless | I now wonder if they patch up cpython's ensurepip too | 02:37 |
| SpamapS | in every swift installation, just because | 02:37 |
| mordred | I think there is absolutely no reason to install python-pip | 02:38 |
| mordred | pip should always be installed from get-pip | 02:39 |
| mordred | mostly because of insanity like this | 02:39 |
| mordred | there has literallly not been a time when distro pacakges of python-pip have not screwed us in some way | 02:39 |
| *** emagana has quit IRC | 02:40 | |
| SpamapS | maybe we need a pip-downloader package | 02:40 |
| *** sputnik13 has quit IRC | 02:40 | |
| lifeless | SpamapS: what, with ensure-pip in it? | 02:40 |
| SpamapS | lifeless: I actually don't know what ensure-pip is | 02:40 |
| SpamapS | have only ever used get-pip.py | 02:40 |
| lifeless | SpamapS: sorry, I meant get-pip | 02:41 |
| lifeless | ensure-pip is get-pip inside the std lib | 02:41 |
| mordred | wget https://bootstrap.pypa.io/get-pip.py ; sudo python get-pip.py == pretty easy | 02:41 |
| SpamapS | oh ok. I thought maybe there was some magic thing that I wasn't aware of :) | 02:41 |
| lifeless | it is :) | 02:41 |
| SpamapS | ah nice | 02:41 |
| lifeless | get-pip isn't infallible though | 02:41 |
| lifeless | https://github.com/pypa/pip/issues/2726 | 02:41 |
| SpamapS | and yes I suspect we can't trust distro ensure pip either | 02:41 |
| mordred | sure | 02:41 |
| mordred | but ^^ | 02:41 |
| lifeless | you'll eyeballs after you read that bug | 02:41 |
| *** markvoelker has joined #openstack-infra | 02:41 | |
| mordred | that's awesome | 02:42 |
| SpamapS | mordred: how well tested is shade's v2 upload? | 02:43 |
| *** ashleighfarnham has joined #openstack-infra | 02:43 | |
| SpamapS | mordred: cause it's 'sploding for me. | 02:43 |
| greghaynes | SpamapS: did the OS_IMAGE_API thing work? | 02:44 |
| SpamapS | greghaynes: yes | 02:45 |
| SpamapS | annoyingly well :-P | 02:45 |
| greghaynes | SpamapS: awesome, that should probably be a bug | 02:45 |
| greghaynes | of "at least document the damn thing" | 02:45 |
| SpamapS | greghaynes: i dunno | 02:46 |
| SpamapS | the bug is v2 at all | 02:46 |
| SpamapS | honestly | 02:46 |
| SpamapS | this thing.. | 02:46 |
| greghaynes | yes, I think theres a spec for that already though | 02:46 |
| SpamapS | :) | 02:46 |
| *** ashleighfarnham has quit IRC | 02:47 | |
| openstackgerrit | Merged openstack-dev/pbr: Improve integration.sh. https://review.openstack.org/177500 | 02:49 |
| SpamapS | is there an extra service that has to be running to run tasks? | 02:49 |
| *** Somay has joined #openstack-infra | 02:50 | |
| *** asettle has joined #openstack-infra | 02:51 | |
| *** ildikov has quit IRC | 02:51 | |
| *** gyee has quit IRC | 02:53 | |
| mordred | SpamapS: I've used it many many times | 02:55 |
| lifeless | mordred: I'm guessing that SpamapS context is devstack | 02:55 |
| mordred | SpamapS: I've even got an ansible playbook that uses it to upload images to all of the clouds | 02:56 |
| *** nfedotov has joined #openstack-infra | 02:56 | |
| mordred | lifeless: yah | 02:56 |
| SpamapS | mordred: so what I'm seeing here is that the task never creates the image, so it's never seen in the list | 02:56 |
| mordred | I have not run it against devstack | 02:56 |
| mordred | SpamapS:the poll/wait timesout? | 02:56 |
| SpamapS | mordred: it seems to fall through immediately.. still debugging | 02:57 |
| mordred | SpamapS: because if youre not using the shade wait function - you'll need to do glance task-show over and over again | 02:58 |
| SpamapS | but I now have like 9 'processing' tasks | 02:58 |
| SpamapS | I suspect there's a worker that doesn't run | 02:58 |
| *** Somay has quit IRC | 02:58 | |
| mordred | SpamapS: did you add "wait=True" to the image_create call in shade? (I'm just saying things outloud, btw) | 02:58 |
| SpamapS | mordred: yeah, using wait=True | 02:58 |
| mordred | ossum | 02:58 |
| SpamapS | mordred: I'm more concerned that the tasks are just sitting there | 02:59 |
| mordred | yay for tseting! | 02:59 |
| mordred | yah | 02:59 |
| SpamapS | and there are so many lovely layers of indirection in glance | 02:59 |
| SpamapS | no idea who actually reads and updates tasks | 02:59 |
| mordred | fairies | 02:59 |
| mordred | magical fairies | 02:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add a hound source code indexer https://review.openstack.org/178488 | 03:00 |
| mordred | nibalizer: ^^ | 03:00 |
| mordred | nibalizer: I did not split it into a puppet-hound module because I coyuld't figure out a sane split point, since things need to know about each other | 03:01 |
| *** Somay has joined #openstack-infra | 03:01 | |
| mordred | nibalizer: also - don't actually hack on that - I'm just doing stupid things while I'm on the plane with laggy internet | 03:02 |
| mordred | but, it turns out, hound is insanely easy to set up | 03:02 |
| SpamapS | ok so.. | 03:03 |
| SpamapS | there's this thing in setup.cfg that I don't undersatnd | 03:03 |
| SpamapS | entry_points has 'glance.flows' | 03:03 |
| SpamapS | and 'glance.flows.import' | 03:03 |
| SpamapS | I only understand entry_points for console_scripts | 03:03 |
| mordred | so - that's defining a plugin point called glance.flows | 03:04 |
| mordred | and is also saying that glance is going to install a plugin into that namespace | 03:04 |
| *** ildikov has joined #openstack-infra | 03:04 | |
| mordred | SpamapS: so : glance/async/taskflow_executor.py | 03:05 |
| SpamapS | that is so helping me | 03:05 |
| SpamapS | it realy is | 03:05 |
| mordred | glance/async/taskflow_executor.py: return driver.DriverManager('glance.flows', task.type, | 03:05 |
| SpamapS | mordred: yeah I'm trying to figure out if and where and how that gets installed/run | 03:05 |
| mordred | is where it looks for drivers ... and glance/async/flows/base_import.py: extensions = named.NamedExtensionManager('glance.flows.import', | 03:05 |
| mordred | is where it loads the import flow | 03:05 |
| *** miqui has joined #openstack-infra | 03:05 | |
| SpamapS | harlowja_away: if you happen to be paying attention, this is _maddeningly complicated_ | 03:05 |
| SpamapS | mordred: do we even run v2 in the gate anywhere? | 03:07 |
| mordred | well, I would hope that we always run it | 03:08 |
| mordred | however | 03:08 |
| mordred | AIUI, none of the other services use it yet | 03:08 |
| *** camunoz has quit IRC | 03:09 | |
| SpamapS | reading taskflow's manual now | 03:10 |
| SpamapS | maybe it can explain how those things happen | 03:10 |
| lifeless | I didn't realise v2 was using taskflow | 03:12 |
| lifeless | perhaps it predates it | 03:12 |
| SpamapS | lifeless: I'm not even sure it does use taskflow. | 03:13 |
| *** ilyashakhat has quit IRC | 03:14 | |
| SpamapS | I've been staring at the guts of glance for 45 minutes and I honestly can't tell what acts on the records in the tasks table. | 03:14 |
| *** salv-orlando has joined #openstack-infra | 03:14 | |
| *** camunoz has joined #openstack-infra | 03:17 | |
| *** Somay has quit IRC | 03:18 | |
| * SpamapS gets more brute force | 03:18 | |
| *** Somay has joined #openstack-infra | 03:20 | |
| *** dimtruck is now known as zz_dimtruck | 03:21 | |
| *** ddieterly has joined #openstack-infra | 03:21 | |
| *** Somay has quit IRC | 03:24 | |
| SpamapS | it almost looks like glance-api might do it | 03:25 |
| *** salv-orlando has quit IRC | 03:25 | |
| SpamapS | but damn if I can read through the path to that | 03:25 |
| *** tiswanso_ has quit IRC | 03:26 | |
| *** erikmwilson has joined #openstack-infra | 03:26 | |
| *** ddieterly has quit IRC | 03:26 | |
| *** coolsvap|afk is now known as coolsvap | 03:27 | |
| *** yamahata has joined #openstack-infra | 03:28 | |
| *** jklare has quit IRC | 03:28 | |
| *** erikmwilson has quit IRC | 03:31 | |
| *** jklare has joined #openstack-infra | 03:31 | |
| *** david-lyle has joined #openstack-infra | 03:32 | |
| *** Somay has joined #openstack-infra | 03:32 | |
| *** Swami has joined #openstack-infra | 03:33 | |
| *** changbl has joined #openstack-infra | 03:33 | |
| *** camunoz has quit IRC | 03:34 | |
| *** ivar-laz_ has joined #openstack-infra | 03:34 | |
| *** baoli has quit IRC | 03:34 | |
| *** jklare has quit IRC | 03:35 | |
| *** ivar-lazzaro has quit IRC | 03:38 | |
| *** ivar-laz_ has quit IRC | 03:38 | |
| *** jklare_ has joined #openstack-infra | 03:39 | |
| SpamapS | ok, so glance-api runs the things | 03:41 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add -e to gate wrap script https://review.openstack.org/177017 | 03:41 |
| SpamapS | which is like zomg o_O | 03:41 |
| SpamapS | Invalid: Input does not contain 'import_from_format' field | 03:41 |
| SpamapS | and was throwing an exception | 03:41 |
| *** bswartz has quit IRC | 03:43 | |
| *** bswartz has joined #openstack-infra | 03:45 | |
| *** kiseok7_ has joined #openstack-infra | 03:46 | |
| SpamapS | mordred: so I have a suspicion that whatever they use in RAX is nothing like what is in trunk. | 03:52 |
| SpamapS | mordred: the devstack v2 is super picky about things, and reports 0 errors ever. :-( | 03:53 |
| SpamapS | it's.. almost not worth pursuing. | 03:53 |
| *** Swami has quit IRC | 03:53 | |
| SpamapS | anyway I'm spent | 03:54 |
| *** camunoz has joined #openstack-infra | 03:54 | |
| * SpamapS quits on v2 | 03:54 | |
| *** Swami has joined #openstack-infra | 03:56 | |
| *** rm_work is now known as rm_work|away | 04:00 | |
| *** ivar-lazzaro has joined #openstack-infra | 04:01 | |
| *** sdake_ has joined #openstack-infra | 04:04 | |
| *** yfried|afk is now known as yfried | 04:05 | |
| *** sdake has quit IRC | 04:08 | |
| *** armax has quit IRC | 04:09 | |
| *** Somay has quit IRC | 04:10 | |
| *** sdake_ has quit IRC | 04:11 | |
| *** btully has quit IRC | 04:11 | |
| *** rm_work|away is now known as rm_work | 04:13 | |
| *** Swami_ has joined #openstack-infra | 04:15 | |
| *** Somay has joined #openstack-infra | 04:16 | |
| *** Swami has quit IRC | 04:18 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Fix swapiness sysctl set https://review.openstack.org/177006 | 04:19 |
| *** stevemar has joined #openstack-infra | 04:19 | |
| *** cody-somerville has joined #openstack-infra | 04:20 | |
| *** nfedotov has quit IRC | 04:22 | |
| *** jyuso1 has quit IRC | 04:23 | |
| *** yfried is now known as yfried|afk | 04:26 | |
| *** asettle has quit IRC | 04:30 | |
| *** yfried|afk is now known as yfried | 04:31 | |
| *** asettle has joined #openstack-infra | 04:34 | |
| *** mwagner_lap has quit IRC | 04:36 | |
| *** miqui has quit IRC | 04:38 | |
| *** btully has joined #openstack-infra | 04:39 | |
| *** yfried has quit IRC | 04:41 | |
| *** beisner has joined #openstack-infra | 04:43 | |
| *** ildikov has quit IRC | 04:44 | |
| *** asettle has quit IRC | 04:47 | |
| *** miqui has joined #openstack-infra | 04:47 | |
| *** miqui__ has joined #openstack-infra | 04:48 | |
| beisner | hi. ping clarkb / fungi | 04:49 |
| *** mwagner_lap has joined #openstack-infra | 04:52 | |
| *** miqui has quit IRC | 04:52 | |
| *** unicell has joined #openstack-infra | 04:53 | |
| blahdeblah | beisner: I'm not sure if clarkb & fungi are in .us TZs; lifeless? | 04:53 |
| lifeless | blahdeblah: they are | 04:54 |
| lifeless | blahdeblah: clark is west coast, fungi east. | 04:54 |
| beisner | fyi, i'm in malta atm | 04:54 |
| lifeless | blahdeblah: 'sup ? | 04:54 |
| beisner | hi lifeless | 04:54 |
| lifeless | oh, beisner pinging. Hai :). | 04:54 |
| lifeless | there's a europe admin nowadays, IIRC. | 04:54 |
| lifeless | beisner: hi; whats up ? | 04:55 |
| beisner | lifeless, fyi the traffic is related to deploy from git source testing | 04:55 |
| lifeless | beisner: yeah, we did figure that much :) | 04:56 |
| beisner | ;-) | 04:56 |
| lifeless | something about the requests was particularly expensive AIUI | 04:57 |
| lifeless | I don't think we identified what particular git request was so expensive | 04:57 |
| *** ildikov has joined #openstack-infra | 04:57 | |
| *** camunoz has quit IRC | 04:57 | |
| beisner | i can only imagine the demand rising, so i think we'll need to set up a local daily sync | 04:59 |
| beisner | instead of a zillion jobs pulling from you | 04:59 |
| *** unicell1 has joined #openstack-infra | 05:00 | |
| lifeless | yeah | 05:00 |
| lifeless | so FYI | 05:00 |
| lifeless | we map src IP to just one git backend | 05:00 |
| lifeless | you're behind NAT, so *all* your load hits just one server | 05:01 |
| *** unicell has quit IRC | 05:01 | |
| blahdeblah | lifeless: What's the best way to get an initial sync happening so that beisner and co. can get going? | 05:01 |
| blahdeblah | lifeless: All the load is coming from one box anyway. :-) | 05:01 |
| lifeless | blahdeblah: pull from github | 05:01 |
| lifeless | blahdeblah: everything is mirrored there | 05:01 |
| blahdeblah | lifeless: Any idea on the lag? | 05:01 |
| lifeless | blahdeblah: seconds | 05:01 |
| blahdeblah | cool | 05:02 |
| lifeless | blahdeblah: its pushed to synchronously with the pushes out to our own mirror farm | 05:02 |
| lifeless | blahdeblah: if something fails it could be out of date but thats rare | 05:02 |
| bradm | surely it'd be less lag than being blocked :) | 05:02 |
| lifeless | blahdeblah: and certainly for seeding a local mirror to get going with, it will be straight forward | 05:02 |
| blahdeblah | lifeless, beisner: So it sounds like even just an initial sync from github would be enough to get us going without messing with your load too much. | 05:03 |
| lifeless | I imagine if the source tests are disabled atm that we could remove the block anyway | 05:03 |
| lifeless | (But I can't - I don't have ops access) | 05:03 |
| lifeless | blahdeblah: yeah | 05:03 |
| lifeless | all our CI jobs that work from source work from local mirrors on the test nodes | 05:03 |
| lifeless | we do an incremental pull to bring them up to date at the start of the CI job | 05:04 |
| beisner | that's the right thing to do, we just need to set it up. but we do have a time-sensitive need to deploy one more today, if that is manageable? | 05:04 |
| lifeless | but our full clone load is very low | 05:05 |
| *** mrmartin has joined #openstack-infra | 05:05 | |
| *** camunoz has joined #openstack-infra | 05:05 | |
| lifeless | beisner: There's noone active in the channel at the moment with ops access | 05:05 |
| lifeless | beisner: you'd be a lot better off doing s/git.openstack.org/github.com/ if you've got something time sensitive to do | 05:05 |
| lifeless | beisner: I'm just looking up the european sysop | 05:06 |
| lifeless | oh there might be two | 05:06 |
| lifeless | I don't remember | 05:06 |
| lifeless | jhesketh: do you have root? | 05:07 |
| bradm | do we know they didn't remove the block? fungi said they'd do it soon, so it might be done? | 05:07 |
| lifeless | worth trying | 05:07 |
| jhesketh | lifeless: no sorry | 05:07 |
| lifeless | I'm fairly sure SergeyLukjanov does but might be too early for him | 05:07 |
| *** dhritishikhar has joined #openstack-infra | 05:07 | |
| *** ajmiller__ has joined #openstack-infra | 05:09 | |
| beisner | thanks lifeless - appreciate the help! | 05:09 |
| lifeless | sounds like the rule is removed: | 05:10 |
| lifeless | "12:01 < clarkb> fungi: doing that now | 05:10 |
| lifeless | " | 05:10 |
| lifeless | so if blahdeblah unblocks at your end, we can find out | 05:10 |
| *** Somay has quit IRC | 05:11 | |
| blahdeblah | lifeless: We'll probably end up with exactly the same behaviour if we just crank it up again... | 05:12 |
| lifeless | blahdeblah: beisner is saying 'one job' | 05:12 |
| lifeless | beisner: how many jobs were running when we hit the wall? | 05:12 |
| *** ajmiller_ has quit IRC | 05:12 | |
| *** miqui__ has quit IRC | 05:13 | |
| beisner | what's the utc time-ish? | 05:13 |
| blahdeblah | 20:50 was when it started | 05:14 |
| lifeless | now? 0513 | 05:14 |
| blahdeblah | beisner: See graphs linked in the ticket | 05:14 |
| *** abregman has joined #openstack-infra | 05:14 | |
| beisner | ok so that was actually not automation. that was one guy prepping for an internal demo. | 05:14 |
| beisner | but i've paused our -from-git automation for the moment. | 05:15 |
| *** salv-orlando has joined #openstack-infra | 05:15 | |
| lifeless | beisner: ok so don't do that thing again | 05:16 |
| lifeless | beisner: how much load would we expect from what you want to do now vs what was happening then ? | 05:16 |
| *** hyakuhei has joined #openstack-infra | 05:17 | |
| *** yamamoto has joined #openstack-infra | 05:18 | |
| beisner | actually i'm quite surprised there wasn't an issue earlier. we've had concurrency of deploy from git tests at 6 for a couple of weeks. it's interesting that those haven't run in 2 days as there haven't been any internal triggers. but one guy prepping a demo ... ha. | 05:19 |
| *** asettle has joined #openstack-infra | 05:19 | |
| beisner | those tests have run dozens of times in that time frame. | 05:20 |
| *** asettle has quit IRC | 05:20 | |
| beisner | at any rate, we'll make adjustments to place nicely | 05:20 |
| lifeless | it may warrant closer investigation into why it was expensive | 05:21 |
| lifeless | different protocol perhaps? git vs https? | 05:21 |
| jamespage | lifeless, the code is using git | 05:21 |
| jamespage | morning all | 05:22 |
| lifeless | jamespage: both the demo and the automated tests? | 05:22 |
| lifeless | git:// that is ? | 05:22 |
| blahdeblah | \o jamespage | 05:22 |
| *** dannywilson has joined #openstack-infra | 05:22 | |
| blahdeblah | lifeless: Yep - that's the only port open in the firewall | 05:22 |
| jamespage | lifeless, no difference | 05:22 |
| jamespage | lifeless, we can switch pretty easily | 05:22 |
| jamespage | beisner, lets just hold the ci tests for now - we can fixup and then re-release | 05:23 |
| jamespage | release rather | 05:23 |
| *** ddieterly has joined #openstack-infra | 05:23 | |
| jamespage | hey blahdeblah | 05:23 |
| *** hdd has joined #openstack-infra | 05:24 | |
| beisner | jamespage, ack, automation on that is on hold | 05:24 |
| greghaynes | lifeless: clarkb: was pretty convinced there are certain conditions where a git fetch will baloon in memory size, and then causes everything else to backlog once the box goes into swap | 05:24 |
| lifeless | greghaynes: yes, the question is why | 05:25 |
| greghaynes | yerp, there was some log digging, nothing was found :( | 05:25 |
| lifeless | greghaynes: I know :) I was here... | 05:26 |
| blahdeblah | greghaynes: I've seen that quite a lot on git gc of small-ish repos, but never on a fetch. | 05:26 |
| *** salv-orlando has quit IRC | 05:26 | |
| *** dannywilson has quit IRC | 05:26 | |
| *** erikmwilson has joined #openstack-infra | 05:27 | |
| *** ildikov has quit IRC | 05:28 | |
| beisner | thx again lifeless. let us know if you'd like to have us 'load test' again. ;-) | 05:28 |
| *** ddieterly has quit IRC | 05:28 | |
| * beisner will be offline for a bit | 05:28 | |
| lifeless | beisner: I think it would be ok to try as long as you keep your eyes on the cactii graph and if it starts to spike shut down your thing | 05:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add errexit to setup & cleanup functions https://review.openstack.org/177017 | 05:30 |
| *** erikmwilson has quit IRC | 05:31 | |
| *** yfried has joined #openstack-infra | 05:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] Add errexit to setup & cleanup functions https://review.openstack.org/177017 | 05:36 |
| *** yamamoto has quit IRC | 05:37 | |
| ianw | ^ 25 revisions that took ... hopefully it's in good shape now | 05:37 |
| *** dhritishikhar has quit IRC | 05:39 | |
| *** yamamoto has joined #openstack-infra | 05:40 | |
| *** HeOS has quit IRC | 05:43 | |
| *** yfried is now known as yfried|afk | 05:44 | |
| *** rlandy has joined #openstack-infra | 05:44 | |
| *** Krinkle is now known as Krinkle|detached | 05:45 | |
| *** mwagner_lap has quit IRC | 05:46 | |
| *** Somay has joined #openstack-infra | 05:46 | |
| *** yfried|afk is now known as yfried | 05:49 | |
| openstackgerrit | Merged openstack-dev/pbr: Build all the wheels in one pass rather than many. https://review.openstack.org/177503 | 05:52 |
| *** stevemar has quit IRC | 05:54 | |
| *** yamamoto has quit IRC | 05:54 | |
| *** mrunge has joined #openstack-infra | 05:54 | |
| *** yamamoto has joined #openstack-infra | 05:54 | |
| *** Somay has quit IRC | 05:55 | |
| *** lucap has joined #openstack-infra | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/178051 | 06:01 |
| *** Longgeek has joined #openstack-infra | 06:03 | |
| *** mwagner_lap has joined #openstack-infra | 06:03 | |
| clarkb | couple things, I did remove the block, anyone can trivially test with `telnet git.openstack.org 9418` from that source | 06:03 |
| clarkb | it would probably be helpful to know what was going on at the time as git daemon doesnt really log this well | 06:04 |
| StevenK | nc -vz! | 06:04 |
| *** otter768 has quit IRC | 06:04 | |
| clarkb | and yes, I dont think its a matter of scale, Instead I think its a particular (set) request that causes things to fall over | 06:06 |
| clarkb | we do happen to run a pretty giant test farm against this mirror just fine for the most part | 06:06 |
| clarkb | StevenK or nc :) | 06:07 |
| *** doude has quit IRC | 06:07 | |
| StevenK | clarkb: telnet's escape sequence is annoying, I like: | 06:08 |
| StevenK | steven@undermined:~% nc -vz git.openstack.org 9418 | 06:08 |
| StevenK | Connection to git.openstack.org 9418 port [tcp/git] succeeded! | 06:08 |
| *** Longgeek has quit IRC | 06:08 | |
| *** yamamoto has quit IRC | 06:08 | |
| clarkb | I usually suggest telnet because unlike nc it seems to exist everywhere and with the same options | 06:09 |
| clarkb | so annoying when you get nc and its bsd nc | 06:09 |
| clarkb | and nothing works | 06:09 |
| *** pcaruana has joined #openstack-infra | 06:09 | |
| clarkb | (also I removed the block hours ago) | 06:09 |
| *** isviridov_away is now known as isviridov | 06:10 | |
| StevenK | clarkb: I forcibly install the openbsd one everywhere I can. I think -vz works with either | 06:10 |
| clarkb | I will read scrollback in the morning if people want to leave more info, but now its time to return to my book then sleep | 06:12 |
| *** Swami has joined #openstack-infra | 06:13 | |
| *** bboese has quit IRC | 06:14 | |
| *** jlvillal has quit IRC | 06:14 | |
| *** Swami_ has quit IRC | 06:14 | |
| *** vlaza has joined #openstack-infra | 06:14 | |
| *** jlvillal has joined #openstack-infra | 06:16 | |
| *** zul has joined #openstack-infra | 06:17 | |
| *** Somay has joined #openstack-infra | 06:17 | |
| *** yfried is now known as yfried|afk | 06:18 | |
| *** Somay has quit IRC | 06:19 | |
| *** jamespage_ has joined #openstack-infra | 06:19 | |
| *** bboese has joined #openstack-infra | 06:20 | |
| *** jyuso1 has joined #openstack-infra | 06:22 | |
| lifeless | can I get 2x+2 on https://review.openstack.org/#/c/177504/ ? | 06:22 |
| lifeless | clarkb: jhesketh: ^ - its super shallow | 06:23 |
| jhesketh | lifeless: looking | 06:23 |
| *** ddieterly has joined #openstack-infra | 06:24 | |
| *** zul has quit IRC | 06:24 | |
| *** zul has joined #openstack-infra | 06:25 | |
| *** sushilkm has left #openstack-infra | 06:27 | |
| jhesketh | lifeless: lgtm, however you're overwriting the existing pip venv whereas all the other tests create a new one | 06:27 |
| jhesketh | should you create a new one for consistency, or was this deliberate | 06:27 |
| lifeless | jhesketh: they all overwrite old venvs | 06:27 |
| lifeless | jhesketh: it looks like they are unique, but they aren't: its in a loop, and mkvenv rm -r's the old tree as its first thing | 06:28 |
| *** tnovacik has joined #openstack-infra | 06:28 | |
| lifeless | jhesketh: the uniqueness of the last component is merely a debugging aid, I think its clear enough from logs where this would get to | 06:28 |
| jhesketh | right, but in that loop they are all unique | 06:28 |
| lifeless | jhesketh: I am happy enough to change it, or to not. | 06:28 |
| jhesketh | yeah I'm not blocking on it, more highlighting that it's inconsistent | 06:29 |
| lifeless | jhesketh: they're distinct, not unique ;) | 06:29 |
| lifeless | jhesketh: if you see what I mean | 06:29 |
| lifeless | yeah | 06:29 |
| *** ddieterly has quit IRC | 06:29 | |
| jhesketh | lifeless: heh, fair enough | 06:29 |
| *** jamespage_ has quit IRC | 06:29 | |
| lifeless | its more oversight from me | 06:29 |
| lifeless | TBH I'd be inclined to change this to use just one venv path | 06:30 |
| lifeless | to reduce page cache impact | 06:30 |
| lifeless | log a **** Testing SDIST **** | 06:30 |
| lifeless | or whatever before each one | 06:30 |
| jhesketh | yep | 06:30 |
| lifeless | future work | 06:30 |
| lifeless | so -> with that, inclined to ignore the inconsistency here | 06:30 |
| lifeless | so sigh, oslo-incubator is one of the things that will wedge when pbr releases | 06:31 |
| lifeless | http://logs.openstack.org/29/177629/4/check/check-pbr-installation-dsvm/def4362/console.html#_2015-04-28_02_21_25_668 | 06:31 |
| *** hashar has joined #openstack-infra | 06:32 | |
| *** salv-orlando has joined #openstack-infra | 06:34 | |
| *** yfried|afk is now known as yfried | 06:38 | |
| *** Swami has quit IRC | 06:38 | |
| *** Somay has joined #openstack-infra | 06:39 | |
| *** ildikov has joined #openstack-infra | 06:40 | |
| *** salv-orlando has quit IRC | 06:42 | |
| *** dhritishikhar has joined #openstack-infra | 06:42 | |
| *** nfedotov has joined #openstack-infra | 06:44 | |
| *** Ala has joined #openstack-infra | 06:45 | |
| *** ssam2 has joined #openstack-infra | 06:47 | |
| *** yfried is now known as yfried|afk | 06:48 | |
| *** jcoufal has joined #openstack-infra | 06:49 | |
| ttx | lifeless: the trick is "release" is not an atomic event, it takes time to tag and make Launchpad release pages for everything | 06:49 |
| ttx | takes about 5 hours to target all fixed bugs to relevant milestones | 06:50 |
| lifeless | ttx: mainly API performance? | 06:50 |
| ttx | I plan to start release when I get up Thursday (around 8 UTC) and finish around 13 UTC | 06:50 |
| lifeless | ttx: yeah, I'll be fast asleep - which is fine | 06:50 |
| ttx | lifeless: well to be fair it's a lot of bugs :) | 06:51 |
| lifeless | ttx: I wasn't in any way complaining about the timing of the release | 06:51 |
| ttx | but yeah, takes about 2 seconds per bug | 06:51 |
| lifeless | ttx: can you ping dims when its done? | 06:51 |
| ttx | lifeless: sure | 06:51 |
| lifeless | ttx: 2s per - there's something fundamentally wrong there | 06:51 |
| lifeless | ttx: either excessive use of multi-project bugs (which are a big source of issues) | 06:51 |
| ttx | maybe there is an optimization to be made somewhere | 06:52 |
| lifeless | ttx: or possibly not enough clobbering of the ORM anti-features in launchpadlib | 06:52 |
| lifeless | ttx: one of the things to watch out for is that any use of an attribute that is a relation triggers an API call | 06:52 |
| lifeless | ttx: using attribute_id and maintaining an in-process cache can avoid this | 06:52 |
| ttx | See consolidate_release_page.py on openstack-infra/release-tools | 06:53 |
| lifeless | ttx: and make a huge perf difference - but we may be doing that already | 06:53 |
| *** Somay has quit IRC | 06:53 | |
| ttx | in retrospect we should probably not do that (consolidate all milestone stuff onto a single release page per project) but since we historically did that, I'll continue as long as we use Lp | 06:54 |
| lifeless | https://git.openstack.org/cgit/openstack-infra/release-tools/tree/consolidate_release_page.py#n95 | 06:54 |
| lifeless | that should use the _id trick | 06:54 |
| lifeless | bp.milestone will be doing an API call | 06:54 |
| lifeless | (if its not None) | 06:54 |
| ttx | lifeless: care to propose a patch there ? I'll gladly used a sped-up version | 06:55 |
| *** BharatK_ has joined #openstack-infra | 06:55 | |
| *** BharatK has quit IRC | 06:55 | |
| ttx | lifeless: moving BPs is not very long. Bugs is where the hit is though | 06:55 |
| lifeless | the assignee etc looksups likewise down on line 122 | 06:55 |
| *** Longgeek has joined #openstack-infra | 06:56 | |
| lifeless | (since bt.assignee is a ref) | 06:56 |
| ttx | ack. If you have example code (or better, can propose a quick patch), I look forward cutting down on that time | 06:57 |
| ttx | although it was not painful, just long :) | 06:57 |
| lifeless | I can't do a patch just now | 06:58 |
| lifeless | I've too much in-head to do the context switch sanely | 06:58 |
| ttx | ok, I'll look in launchpadlib litterature | 06:58 |
| ttx | I can relate to that | 06:58 |
| *** camunoz is now known as camunoz_gone | 06:58 | |
| lifeless | I may be able to find you a ref | 06:59 |
| lifeless | I don't think it will be a huge win in this case but its worth doing | 06:59 |
| lifeless | its a shame we need to materialise bug | 06:59 |
| lifeless | now | 07:00 |
| ttx | well in that code we usually don't use the copytask feature, so we geenrally don't use L121-126 | 07:00 |
| lifeless | ok | 07:00 |
| ttx | when I tried to optimize, realized what took time was bt.lp_save() | 07:00 |
| lifeless | so that means that we're mainly hitting bad perf | 07:00 |
| lifeless | yes | 07:01 |
| lifeless | could you for this release do this? | 07:01 |
| ttx | which is why I start early, perf also drops when the US gets up | 07:01 |
| lifeless | put a start/end time.time call around the lp_save | 07:01 |
| *** ajo has quit IRC | 07:01 | |
| lifeless | and log to a trace log | 07:01 |
| ttx | lifeless: sure | 07:01 |
| lifeless | the start time, duration, bug id | 07:02 |
| lifeless | then we can give that to wgrant with puppy-dog eyes | 07:02 |
| *** Longgeek has quit IRC | 07:02 | |
| lifeless | setting a milestone is a rare thing in the API | 07:02 |
| lifeless | most of the optimisation work gets guided by data | 07:02 |
| lifeless | e.g. rare things have to be REALLLY bad to be spotted | 07:02 |
| lifeless | and setting a milestone is -so- rare I think you'd need to be looking at just the right time | 07:03 |
| lifeless | but we're going to do several hundred/ thousand | 07:03 |
| lifeless | so lots of data captured | 07:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: Add errexit to setup & cleanup functions https://review.openstack.org/177017 | 07:03 |
| *** devvesa has joined #openstack-infra | 07:04 | |
| *** devvesa has quit IRC | 07:04 | |
| lifeless | oh, I think its not _id, its _link | 07:04 |
| lifeless | to bt.bug_link | 07:04 |
| lifeless | vs bt.bug, which is an ORM like lazy loading object | 07:04 |
| lifeless | (inferred from http://blog.launchpad.net/general/this-week-in-launchpads-web-api-2008-08-15) | 07:04 |
| *** devvesa has joined #openstack-infra | 07:04 | |
| *** Longgeek has joined #openstack-infra | 07:06 | |
| *** jamespage_ has joined #openstack-infra | 07:07 | |
| *** jtriley has joined #openstack-infra | 07:09 | |
| *** yfried|afk is now known as yfried | 07:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/178051 | 07:10 |
| lifeless | failing that, look at the descriptor on type(BugTask).bug | 07:11 |
| *** jamespage__ has joined #openstack-infra | 07:11 | |
| *** MaxV has joined #openstack-infra | 07:13 | |
| *** jamespage_ has quit IRC | 07:13 | |
| *** niedbalski_ has joined #openstack-infra | 07:13 | |
| *** jtriley has quit IRC | 07:14 | |
| *** luqas has joined #openstack-infra | 07:15 | |
| lifeless | anyone here happen to know/remember where the updating from zuul of the stuff being tested, and of /opt/git to trunk appens? | 07:16 |
| AJaeger | lifeless, do you mean this script in project-config: jenkins/scripts/gerrit-git-prep.sh | 07:18 |
| AJaeger | lifeless, it'S called in many jobs as builder gerrit-git-prep | 07:18 |
| lifeless | that looks like it | 07:19 |
| lifeless | so I have this patch | 07:19 |
| lifeless | https://review.openstack.org/#/c/177629/ | 07:19 |
| lifeless | in which I was hoping to avoid a chunk of IO | 07:19 |
| *** yfried is now known as yfried|afk | 07:19 | |
| mordred | lifeless: chunk | 07:20 |
| mordred | morning! | 07:20 |
| *** mpavone has joined #openstack-infra | 07:20 | |
| lifeless | mordred: 1.mumble GB right | 07:20 |
| lifeless | but | 07:20 |
| lifeless | /opt/stack has all the stale refs | 07:20 |
| *** nmagnezi has joined #openstack-infra | 07:20 | |
| lifeless | so I need a variant of that prep script that works in-place I guess | 07:20 |
| AJaeger | lifeless, check the zuul-cloner line in jenkins/jobs/infra.yaml | 07:21 |
| openstackgerrit | Ilia Meerovich proposed openstack-infra/jenkins-job-builder: Adding unittest to ownership property https://review.openstack.org/178533 | 07:21 |
| AJaeger | lifeless, but better talk with clarkb, mordred or fungi about this. I'll disappear now again and they know this far better than I do. | 07:22 |
| lifeless | AJaeger: yeah, I don't know enough either | 07:22 |
| lifeless | mordred: morning, where are you? | 07:22 |
| *** yfried|afk is now known as yfried | 07:22 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Use /opt/git directly https://review.openstack.org/177629 | 07:23 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Stop testing setup.py easy_install behaviour https://review.openstack.org/177505 | 07:23 |
| lifeless | jhesketh: https://review.openstack.org/#/c/177505/9 if youplease | 07:23 |
| lifeless | mordred: https://review.openstack.org/#/c/177505/9 and https://review.openstack.org/#/c/177504/6 if you please | 07:23 |
| mordred | lifeless: I have just arrived in Paris | 07:24 |
| mordred | lifeless: and I'm waiting on jeblair to clear customs whilst having a coffee | 07:24 |
| mordred | lifeless: looking at the patches | 07:24 |
| lifeless | Paris heh heh... for? | 07:24 |
| *** aarefiev has quit IRC | 07:24 | |
| *** ddieterly has joined #openstack-infra | 07:25 | |
| mordred | lifeless: we're meeting with markmc and the folks who used to be enovance about the work they did on software-factory and making infra reconsumable | 07:25 |
| mordred | lifeless: hopefully to outline a path forward to working more closely together and stuff | 07:26 |
| *** btully has quit IRC | 07:26 | |
| lifeless | cool | 07:27 |
| *** arxcruz has joined #openstack-infra | 07:27 | |
| *** erikmwilson has joined #openstack-infra | 07:28 | |
| mordred | lifeless: both lgtm | 07:28 |
| lifeless | mordred: have you applied your +2 to them :) | 07:28 |
| lifeless | mordred: thats all I really care aboot | 07:28 |
| mordred | lifeless: yup | 07:28 |
| lifeless | yaaay | 07:28 |
| *** mwagner_lap has quit IRC | 07:28 | |
| mordred | lifeless: all +2d and whatnot | 07:29 |
| *** ddieterly has quit IRC | 07:29 | |
| lifeless | mordred: so why can't I see the +2 on https://review.openstack.org/#/c/177504/6 ? | 07:29 |
| lifeless | mordred: yeah, you went up the stack, to the /opt/git patch which is subtley broken and I need some -infra braintime to fix :) | 07:30 |
| StevenK | I'm going to guess "Because gertty" | 07:30 |
| *** zul has quit IRC | 07:32 | |
| *** zul has joined #openstack-infra | 07:32 | |
| *** erikmwilson has quit IRC | 07:33 | |
| *** Hal has joined #openstack-infra | 07:38 | |
| *** Hal is now known as Guest60666 | 07:38 | |
| *** nfedotov has quit IRC | 07:38 | |
| *** kaisers has quit IRC | 07:39 | |
| *** scheuran has joined #openstack-infra | 07:39 | |
| *** dizquierdo has joined #openstack-infra | 07:39 | |
| *** aarefiev has joined #openstack-infra | 07:40 | |
| *** yfried has quit IRC | 07:41 | |
| *** kaisers has joined #openstack-infra | 07:41 | |
| *** Guest60666 has quit IRC | 07:42 | |
| hashar | mordred: jeblair: oh do you guys come to Paris and don't ping me beforehand :D | 07:42 |
| hashar | enjoy your stay! | 07:42 |
| *** jlanoux has joined #openstack-infra | 07:43 | |
| *** ildikov has quit IRC | 07:43 | |
| lifeless | mordred: ping?, sadface - https://review.openstack.org/#/c/177504/ | 07:43 |
| *** yfried has joined #openstack-infra | 07:44 | |
| *** dguerri is now known as _dguerri | 07:44 | |
| *** ildikov has joined #openstack-infra | 07:44 | |
| *** kaisers has quit IRC | 07:45 | |
| *** mwagner_lap has joined #openstack-infra | 07:46 | |
| hashar | lifeless: hello! I started using diskimage-builder at Wikimedia , is there any specific irc channel I could join for that utility ? | 07:46 |
| lifeless | hashar: #tripleo or here | 07:47 |
| lifeless | hashar: or #openstack-dev | 07:47 |
| hashar | ah tripleo is probably a good one | 07:47 |
| *** kaisers has joined #openstack-infra | 07:48 | |
| *** achanda has quit IRC | 07:48 | |
| hashar | I am going to send some patches against a bunch of documentation related patches since I am actually reading/using the doc :D | 07:49 |
| *** ajo has joined #openstack-infra | 07:51 | |
| *** _dguerri is now known as dguerri | 07:52 | |
| *** sergsh has joined #openstack-infra | 07:53 | |
| *** achanda has joined #openstack-infra | 07:56 | |
| *** jeblair has quit IRC | 07:56 | |
| *** jeblair has joined #openstack-infra | 07:56 | |
| *** sergsh has quit IRC | 07:58 | |
| *** jistr has joined #openstack-infra | 07:58 | |
| *** imcsk8|afk is now known as imcsk8 | 07:59 | |
| *** Somay has joined #openstack-infra | 08:00 | |
| *** jamespage__ has quit IRC | 08:00 | |
| *** derekh has joined #openstack-infra | 08:01 | |
| *** dizquierdo has quit IRC | 08:02 | |
| *** _nadya_ has joined #openstack-infra | 08:04 | |
| *** hyakuhei has quit IRC | 08:04 | |
| *** otter768 has joined #openstack-infra | 08:05 | |
| *** Longgeek has quit IRC | 08:05 | |
| *** imcsk8 is now known as imcsk8|zZz | 08:06 | |
| *** salv-orlando has joined #openstack-infra | 08:08 | |
| *** HeOS has joined #openstack-infra | 08:08 | |
| *** zul has quit IRC | 08:09 | |
| *** otter768 has quit IRC | 08:10 | |
| *** dims has quit IRC | 08:10 | |
| *** notnownikki has joined #openstack-infra | 08:12 | |
| *** sergsh has joined #openstack-infra | 08:12 | |
| *** Somay_ has joined #openstack-infra | 08:24 | |
| *** Hal has joined #openstack-infra | 08:24 | |
| *** Somay has quit IRC | 08:24 | |
| *** Hal is now known as Guest92304 | 08:24 | |
| *** zul has joined #openstack-infra | 08:25 | |
| *** yamahata has quit IRC | 08:25 | |
| *** ddieterly has joined #openstack-infra | 08:26 | |
| *** ddieterly has quit IRC | 08:30 | |
| mescanef | hello, please add me (mmatuszkowiak@mirantis.com) to the gerrit groups of a project regarding the https://review.openstack.org/#/c/174763/ :-) | 08:32 |
| *** achanda has quit IRC | 08:33 | |
| *** jeblair has quit IRC | 08:33 | |
| *** achanda has joined #openstack-infra | 08:33 | |
| mescanef | fungi: ^^ :-) | 08:35 |
| *** achanda has quit IRC | 08:35 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:36 | |
| DuncanT | Hi. Who are the review team for the requirements repo? Any chance of getting some eyes on https://review.openstack.org/#/c/177372/ please? | 08:40 |
| *** woodster_ has quit IRC | 08:40 | |
| *** dmellado has quit IRC | 08:41 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 08:44 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:45 | |
| *** Mithrandir has quit IRC | 08:46 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstackci: Add developer guidelines https://review.openstack.org/178559 | 08:46 |
| AJaeger | DuncanT, you can check this yourself, got to your change, click on "openstack/requirements" and then "Access" | 08:50 |
| *** dmellado has joined #openstack-infra | 08:51 | |
| DuncanT | AJaeger: That's a handy tip, thanks! | 08:51 |
| *** jeblair has joined #openstack-infra | 08:51 | |
| *** dizquierdo has joined #openstack-infra | 08:52 | |
| *** mwagner_lap has quit IRC | 08:55 | |
| *** Mithrandir has joined #openstack-infra | 08:55 | |
| *** dhritishikhar has quit IRC | 08:56 | |
| *** jtriley has joined #openstack-infra | 08:58 | |
| *** deepakcs has joined #openstack-infra | 08:59 | |
| *** dhritishikhar has joined #openstack-infra | 09:01 | |
| *** jtriley has quit IRC | 09:02 | |
| *** dhritishikhar has quit IRC | 09:03 | |
| *** dhritishikhar has joined #openstack-infra | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:06 | |
| *** Longgeek has joined #openstack-infra | 09:06 | |
| *** e0ne is now known as e0ne_ | 09:06 | |
| *** dtantsur is now known as dtantsur|brb | 09:08 | |
| *** Somay_ has quit IRC | 09:08 | |
| *** fhubik has joined #openstack-infra | 09:10 | |
| *** mwagner_lap has joined #openstack-infra | 09:11 | |
| *** Longgeek has quit IRC | 09:12 | |
| *** Longgeek has joined #openstack-infra | 09:13 | |
| *** MaxV has quit IRC | 09:13 | |
| *** jyuso1 has quit IRC | 09:14 | |
| *** jyuso1 has joined #openstack-infra | 09:14 | |
| *** e0ne_ is now known as e0ne | 09:14 | |
| SergeyLukjanov | lifeless, hey, what's up? | 09:14 |
| lifeless | SergeyLukjanov: nothing :) | 09:15 |
| lifeless | SergeyLukjanov: we were just talking about whether some stuff had or had not happened and who could do it if it hadn't | 09:15 |
| SergeyLukjanov | lifeless, ok :) | 09:15 |
| *** MaxV has joined #openstack-infra | 09:16 | |
| *** jyuso has joined #openstack-infra | 09:17 | |
| *** jyuso1 has quit IRC | 09:17 | |
| notnownikki | lifeless, hi, did i get the right hp email for you? I sent some stuff about pip dependency handling | 09:22 |
| *** jyuso has quit IRC | 09:24 | |
| *** ddieterly has joined #openstack-infra | 09:26 | |
| *** zul has quit IRC | 09:27 | |
| *** erikmwilson has joined #openstack-infra | 09:28 | |
| *** ddieterly has quit IRC | 09:31 | |
| *** tnovacik has quit IRC | 09:31 | |
| *** afazekas has quit IRC | 09:32 | |
| *** afazekas has joined #openstack-infra | 09:32 | |
| lifeless | notnownikki: I don't know. What email address did you use? BTW,for open source stuff, my @robertcollins.net address is best | 09:33 |
| *** erikmwilson has quit IRC | 09:33 | |
| notnownikki | lifeless, rbtcollins@, but I'll fwd to your other email | 09:34 |
| *** jamespage_ has joined #openstack-infra | 09:34 | |
| *** fhubik is now known as fhubik_afk | 09:34 | |
| *** fhubik_afk is now known as fhubik | 09:35 | |
| lifeless | notnownikki: ah, found it | 09:36 |
| *** jamespage_ has quit IRC | 09:39 | |
| *** aswadr has joined #openstack-infra | 09:44 | |
| lifeless | notnownikki: I'll reply to you shorlty | 09:45 |
| lifeless | bah | 09:45 |
| lifeless | shortly | 09:45 |
| notnownikki | :) | 09:49 |
| *** markmcclain has quit IRC | 09:49 | |
| *** jamespage_ has joined #openstack-infra | 09:50 | |
| *** zul has joined #openstack-infra | 09:51 | |
| *** dims has joined #openstack-infra | 09:51 | |
| *** reed has quit IRC | 09:56 | |
| *** yfried is now known as yfried|afk | 09:58 | |
| *** markmcclain has joined #openstack-infra | 09:59 | |
| *** sergsh has quit IRC | 10:02 | |
| *** jeblair has quit IRC | 10:03 | |
| *** e0ne is now known as e0ne_ | 10:03 | |
| *** jeblair has joined #openstack-infra | 10:03 | |
| *** sergsh has joined #openstack-infra | 10:03 | |
| *** e0ne_ is now known as e0ne | 10:05 | |
| *** jamespage_ has quit IRC | 10:05 | |
| *** jamespage_ has joined #openstack-infra | 10:05 | |
| *** otter768 has joined #openstack-infra | 10:06 | |
| *** mtreinish has quit IRC | 10:07 | |
| *** jamespage_ has quit IRC | 10:08 | |
| *** doude has joined #openstack-infra | 10:09 | |
| *** tnovacik has joined #openstack-infra | 10:10 | |
| *** otter768 has quit IRC | 10:10 | |
| *** shardy_ has joined #openstack-infra | 10:13 | |
| *** shardy has quit IRC | 10:15 | |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| lifeless | notnownikki: replied | 10:17 |
| lifeless | notnownikki: you did have the right HP address btw | 10:17 |
| *** dtantsur|brb is now known as dtantsur | 10:18 | |
| notnownikki | lifeless, thank you! looks like I have a lot of reading to do... :) | 10:18 |
| *** shardy_ has quit IRC | 10:19 | |
| *** yfried|afk is now known as yfried | 10:19 | |
| *** doude has quit IRC | 10:19 | |
| *** zul has quit IRC | 10:19 | |
| *** shardy has joined #openstack-infra | 10:19 | |
| lifeless | right, waaaay past EOD for me, gl. | 10:21 |
| *** hashar is now known as hasharLunch | 10:26 | |
| *** ddieterly has joined #openstack-infra | 10:27 | |
| *** wznoinsk has quit IRC | 10:27 | |
| *** doude has joined #openstack-infra | 10:27 | |
| *** reed has joined #openstack-infra | 10:28 | |
| *** yfried is now known as yfried|afk | 10:31 | |
| *** ociuhandu has quit IRC | 10:31 | |
| ekarlso | what's wrong with something when: py27 develop-inst just hangs ? | 10:31 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Bump stankevich-python version https://review.openstack.org/174820 | 10:31 |
| *** ddieterly has quit IRC | 10:31 | |
| *** ociuhandu has joined #openstack-infra | 10:34 | |
| *** yfried|afk is now known as yfried | 10:34 | |
| *** sputnik13 has joined #openstack-infra | 10:35 | |
| *** mwagner_lap has quit IRC | 10:41 | |
| *** dhritishikhar has quit IRC | 10:42 | |
| *** dhritishikhar has joined #openstack-infra | 10:43 | |
| *** dhritishikhar has quit IRC | 10:44 | |
| *** mmedvede has quit IRC | 10:45 | |
| *** dhritishikhar has joined #openstack-infra | 10:45 | |
| *** jtriley has joined #openstack-infra | 10:47 | |
| *** jtriley has quit IRC | 10:51 | |
| *** mmedvede has joined #openstack-infra | 10:53 | |
| *** _nadya_ has quit IRC | 10:54 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 10:58 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: fix git_timed retry logic https://review.openstack.org/178608 | 10:58 |
| *** mwagner_lap has joined #openstack-infra | 10:59 | |
| sdague | fungi / mordred / jeblair ^^^^^ | 10:59 |
| *** dhritishikhar has quit IRC | 10:59 | |
| sdague | I noticed another git clone fail this morning, then realized we had the code wrong yesterday | 10:59 |
| *** wznoinsk has joined #openstack-infra | 10:59 | |
| sdague | http://logs.openstack.org/48/176748/6/gate/gate-grenade-dsvm/82726bf/console.html#_2015-04-28_22_00_20_712 | 10:59 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: fix git_timed retry logic https://review.openstack.org/178608 | 11:01 |
| AJaeger | sdague, so, we have to handle it differently than https://github.com/openstack-dev/devstack/blob/master/functions-common#L489-L518 ? | 11:01 |
| AJaeger | Or should we change that one as well? | 11:02 |
| sdague | so, honestly, that function does a very specific thing that ianw needed, and not the thing we care about | 11:02 |
| AJaeger | Ah, wrong copy & paste ;) | 11:02 |
| AJaeger | Just a nit - I would write "exited" but I'm not giving a comment for that. Just in case you want to update again... | 11:03 |
| sdague | commit message? | 11:03 |
| *** zhiwei has quit IRC | 11:03 | |
| AJaeger | https://review.openstack.org/#/c/178608/2/jenkins/scripts/gerrit-git-prep.sh | 11:03 |
| sdague | ok, sure will fix | 11:04 |
| AJaeger | Also, we now have two echos there using the git arguments, what about compressing the two lines | 11:04 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: fix git_timed retry logic https://review.openstack.org/178608 | 11:04 |
| AJaeger | Just nits... | 11:04 |
| sdague | yeh, so it was an attempt to be a tactical patch, I have a whole function rewrite patch as well | 11:04 |
| *** yfried is now known as yfried|afk | 11:05 | |
| sdague | I think double output lines are fine | 11:05 |
| sdague | for now | 11:05 |
| AJaeger | ;) | 11:05 |
| *** yfried|afk is now known as yfried | 11:06 | |
| *** maishsk has joined #openstack-infra | 11:10 | |
| maishsk | Good afternoon. | 11:10 |
| maishsk | Who can help me out with a mailman problem? | 11:12 |
| *** ociuhandu has quit IRC | 11:13 | |
| *** dizquierdo has quit IRC | 11:14 | |
| *** kashyap has joined #openstack-infra | 11:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Add keystoneauth library and testing infrastructure https://review.openstack.org/175596 | 11:18 |
| openstackgerrit | Merged openstack-infra/project-config: Use zaqar's devstack plugin https://review.openstack.org/178076 | 11:18 |
| *** dims has quit IRC | 11:18 | |
| *** dhritishikhar has joined #openstack-infra | 11:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Let kolla-release create branches https://review.openstack.org/177980 | 11:19 |
| *** fhubik is now known as fhubik_afk | 11:19 | |
| *** maishsk has quit IRC | 11:20 | |
| *** maishsk has joined #openstack-infra | 11:21 | |
| openstackgerrit | Merged openstack-infra/project-config: add project fuel-plugin-glance-nfs https://review.openstack.org/178109 | 11:21 |
| *** doug-fish has joined #openstack-infra | 11:22 | |
| *** dhritishikhar_ has joined #openstack-infra | 11:23 | |
| AJaeger | sdague, please put https://review.openstack.org/176947 in your review queue as well... | 11:25 |
| *** dhritishikhar has quit IRC | 11:26 | |
| *** ddieterly has joined #openstack-infra | 11:28 | |
| *** erikmwilson has joined #openstack-infra | 11:29 | |
| *** jamespage_ has joined #openstack-infra | 11:30 | |
| sergmelikyan | Hi folks, we are trying to release stackforge/yaql following new library release process but we don't have rights to create stable branches :( | 11:30 |
| sergmelikyan | https://review.openstack.org/177917 - this patch corrects ACLs for stackforge/yaql | 11:30 |
| *** e0ne is now known as e0ne_ | 11:30 | |
| *** sputnik13 has quit IRC | 11:30 | |
| sergmelikyan | Please, take a look | 11:30 |
| *** HeOS has quit IRC | 11:30 | |
| *** dhritishikhar has joined #openstack-infra | 11:31 | |
| *** dhritishikhar_ has quit IRC | 11:31 | |
| *** sputnik13 has joined #openstack-infra | 11:31 | |
| *** HeOS has joined #openstack-infra | 11:31 | |
| *** sputnik13 has quit IRC | 11:32 | |
| *** ddieterly has quit IRC | 11:33 | |
| *** coolsvap is now known as coolsvap|afk | 11:33 | |
| *** jamespage_ has quit IRC | 11:33 | |
| *** erikmwilson has quit IRC | 11:34 | |
| kashyap | I find it strange - 'check-grenade-dsvm' job seems to succeed on _some_ CI machines, but not others (failing to invoke a certain `pretty_tox.sh` command. | 11:35 |
| kashyap | E.g - http://logs.openstack.org/61/178061/1/check/check-grenade-dsvm/92458a9/logs/grenade.sh.txt.gz | 11:35 |
| kashyap | Looking at logstash, I see just about 11 hits in the last 48 hours. | 11:35 |
| *** HeOS has quit IRC | 11:36 | |
| *** dizquierdo has joined #openstack-infra | 11:37 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Merge State Management and Oslo Meetings https://review.openstack.org/176215 | 11:37 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add OpenStackClient Team Meeting https://review.openstack.org/176203 | 11:37 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Cue team meeting https://review.openstack.org/176204 | 11:38 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Chef Cookbook meeting. https://review.openstack.org/176205 | 11:38 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add magnetoDB Team Meeting https://review.openstack.org/176206 | 11:38 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add DefCore Progress Update Meeting. https://review.openstack.org/176207 | 11:39 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Community Office hours. https://review.openstack.org/176208 | 11:39 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Alternate Sahara meeting https://review.openstack.org/176209 | 11:39 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Neutron VPNaaS Meeting https://review.openstack.org/176210 | 11:40 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Glace meeting no longer alternates https://review.openstack.org/176211 | 11:40 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Move Puppet Meeting https://review.openstack.org/176212 | 11:40 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Place novanet to Neutron migration meeting on hold https://review.openstack.org/176213 | 11:40 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Telco WG Meeting https://review.openstack.org/176214 | 11:41 |
| *** cdent has joined #openstack-infra | 11:41 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Rename OpenStack Security Group (OSSG) Meeting https://review.openstack.org/176216 | 11:41 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update nova API Meeting Chair https://review.openstack.org/176217 | 11:41 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update the Meeting chairs. https://review.openstack.org/176218 | 11:41 |
| *** nmagnezi has quit IRC | 11:42 | |
| *** blahdeblah has left #openstack-infra | 11:44 | |
| *** marcusvrn1 has joined #openstack-infra | 11:45 | |
| *** mmedvede has quit IRC | 11:46 | |
| sdague | kashyap: is this an existing Elastic Recheck tracked bug? | 11:46 |
| kashyap | sdague: I don't find one, | 11:46 |
| kashyap | I'm trying to add a signature. | 11:46 |
| sdague | what query are you using | 11:46 |
| kashyap | sdague: For logstash? If so - | 11:47 |
| sdague | yes for logstash | 11:47 |
| kashyap | sdague: There we go -- http://ur1.ca/k9lwe | 11:47 |
| kashyap | sdague: Incidentally, I was reading your thread on -dev list | 11:47 |
| *** marcusvrn has quit IRC | 11:47 | |
| kashyap | about "trimming down Tempest smoke tag" | 11:48 |
| kashyap | when I noticed the failure in question is also tagged w/ 'smoke' | 11:48 |
| sdague | oh, so we removed the generic SSHTimeout signature because it's really not useful enough to track anything | 11:49 |
| kashyap | I see, I recall seeing that signature | 11:49 |
| kashyap | In this specific context, | 11:49 |
| sdague | you also need to remove the identifying ip address | 11:49 |
| kashyap | InvocationError: '/bin/bash tools/pretty_tox.sh (?!.*\\[.*\\bslow\\b.*\\])((smoke)|(^tempest\\.scenario)) --concurrency=4' | 11:49 |
| sdague | kashyap: right, that just means a test failed | 11:49 |
| *** _nadya_ has joined #openstack-infra | 11:50 | |
| *** prad is now known as prad-afk | 11:50 | |
| kashyap | I think I can use regex in logstash queries? | 11:50 |
| *** baoli has joined #openstack-infra | 11:50 | |
| sdague | no | 11:51 |
| sdague | just segment it into 2 message matches | 11:51 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwidGVtcGVzdC5leGNlcHRpb25zLlNTSFRpbWVvdXQ6IENvbm5lY3Rpb24gdG8gdGhlXCIgQU5EIG1lc3NhZ2U6XCJ2aWEgU1NIIHRpbWVkIG91dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDMwMzA4MTYxNjYyLCJtb2RlIjoic2NvcmUiLCJhbmFseXplX2ZpZWxkIjoiYnVpbGRfbmFtZSJ9 | 11:51 |
| sdague | that's the break down by job view | 11:51 |
| sdague | only 4 grenade jobs hit it | 11:51 |
| sdague | sorry 5 | 11:52 |
| kashyap | Ah, I see | 11:52 |
| openstackgerrit | Merged openstack-infra/project-config: Install kpartx for DIB tests https://review.openstack.org/178283 | 11:52 |
| kashyap | sdague: Any action I can take to fix this? Because, this doesn't seem deterministic | 11:52 |
| sdague | dig into one of the failures and try to figure out why the networking did not come up for that guest | 11:53 |
| *** baoli has quit IRC | 11:53 | |
| kashyap | Thanks, I'll check. | 11:53 |
| *** zul has joined #openstack-infra | 11:53 | |
| *** baoli has joined #openstack-infra | 11:54 | |
| sdague | these are never fun to debug, it requires a lot of tracing through the logs to figure out what was all going on when these failed | 11:54 |
| openstackgerrit | Merged openstack-infra/project-config: Make oslo.messaging py3 job voting https://review.openstack.org/177534 | 11:54 |
| *** mmedvede has joined #openstack-infra | 11:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Create networking-zvm project on StackForge https://review.openstack.org/171030 | 11:54 |
| openstackgerrit | Merged openstack-infra/project-config: fix ec2-api rally job https://review.openstack.org/177947 | 11:55 |
| openstackgerrit | Merged openstack-infra/project-config: Enable gate jobs for remaining StackTach libraries https://review.openstack.org/177909 | 11:55 |
| *** HeOS has joined #openstack-infra | 11:55 | |
| *** woodster_ has joined #openstack-infra | 11:56 | |
| *** fhubik_afk is now known as fhubik | 11:56 | |
| kashyap | sdague: Yeah, can imagine. I was wondering if I can do all of it via some CLI, as browsing through plenty of logs in a browser just slows the workflow | 11:56 |
| *** dims has joined #openstack-infra | 11:56 | |
| * kashyap fires up gertty to see if can find anything there | 11:57 | |
| sdague | you'd have to download all the logs, there are no tools for that at this point | 11:57 |
| sdague | gertty? | 11:57 |
| openstackgerrit | Merged openstack-infra/project-config: project-config: Merge lint jobs (2/2) https://review.openstack.org/164520 | 11:57 |
| kashyap | sdague: This one from James Blair and co -- https://git.openstack.org/cgit/stackforge/gertty/ | 11:59 |
| sdague | right, but I'm confused why you think that will help here | 11:59 |
| *** scheuran has quit IRC | 12:04 | |
| *** dprince has joined #openstack-infra | 12:04 | |
| *** bauzas is now known as bauzas_notme | 12:05 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Enhance error message in update_patch https://review.openstack.org/177985 | 12:06 |
| *** bauzas_notme is now known as bauzas | 12:06 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update recent Ironic exceptions https://review.openstack.org/177986 | 12:06 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Convert node_set_provision_state to task https://review.openstack.org/177987 | 12:06 |
| *** openstackgerrit has quit IRC | 12:07 | |
| *** otter768 has joined #openstack-infra | 12:07 | |
| *** openstackgerrit has joined #openstack-infra | 12:07 | |
| *** mrmartin has quit IRC | 12:08 | |
| *** hasharLunch is now known as hashar | 12:08 | |
| *** _nadya_ has quit IRC | 12:10 | |
| *** otter768 has quit IRC | 12:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix mistyped requrieChangeId in Gerrit ACLs https://review.openstack.org/178307 | 12:12 |
| *** maishsk has quit IRC | 12:16 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 12:16 |
| *** maishsk_ has joined #openstack-infra | 12:17 | |
| *** dhritishikhar has quit IRC | 12:17 | |
| *** dhritishikhar has joined #openstack-infra | 12:17 | |
| *** samueldmq has joined #openstack-infra | 12:20 | |
| *** bswartz has quit IRC | 12:20 | |
| *** abregman_ has joined #openstack-infra | 12:21 | |
| *** rfolco has joined #openstack-infra | 12:22 | |
| *** kgiusti has joined #openstack-infra | 12:23 | |
| *** abregman_ has quit IRC | 12:23 | |
| *** abregman_ has joined #openstack-infra | 12:24 | |
| *** abregman_ has quit IRC | 12:24 | |
| *** abregman_ has joined #openstack-infra | 12:24 | |
| *** ivar-lazzaro has quit IRC | 12:25 | |
| *** e0ne_ is now known as e0ne | 12:26 | |
| *** ivar-lazzaro has joined #openstack-infra | 12:28 | |
| *** ddieterly has joined #openstack-infra | 12:29 | |
| *** abregman_ has quit IRC | 12:29 | |
| *** dkranz has joined #openstack-infra | 12:32 | |
| *** ddieterly has quit IRC | 12:33 | |
| *** gordc has joined #openstack-infra | 12:33 | |
| *** erikmwilson has joined #openstack-infra | 12:34 | |
| *** erikmwilson has quit IRC | 12:35 | |
| *** erikmwilson_ is now known as erikmwilson | 12:35 | |
| *** jtriley has joined #openstack-infra | 12:35 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Explicitly say when job times out https://review.openstack.org/178330 | 12:36 |
| *** claudiub has joined #openstack-infra | 12:36 | |
| *** kgiusti has left #openstack-infra | 12:39 | |
| *** jtriley has quit IRC | 12:40 | |
| *** EmilienM is now known as EmilienM|afk | 12:41 | |
| fungi | well, you _can_ override the browser callout for gertty's url handling to run an arbitrary program (i have it fire up elinks, but you could make it use wget to mirror the subtree from the url to a local directory or something pretty easily i'm betting) | 12:41 |
| *** lifeless has quit IRC | 12:43 | |
| *** chlong has joined #openstack-infra | 12:44 | |
| *** MaxV has quit IRC | 12:44 | |
| *** fhubik has quit IRC | 12:44 | |
| *** kgiusti has joined #openstack-infra | 12:45 | |
| kashyap | sdague: Was afk for lunch, no I didn't mean it (gertty) has specific options to do that (logs, etc). Was only trying to see if it has anything related to logs. | 12:45 |
| AJaeger | fungi, http://docs.openstack.org/infra/ is available now - and zuul, openstackid, storyboard are missing content. What's the best way to push content there? | 12:46 |
| openstackgerrit | Merged openstack-infra/project-config: fix git_timed retry logic https://review.openstack.org/178608 | 12:46 |
| kashyap | fungi: Yeah, that sounds interesting, maybe I'll file a bug or something in storyboard for it :) | 12:46 |
| sdague | kashyap: or just build the tool you want as a cli | 12:47 |
| fungi | AJaeger: in theory, assuming their publish jobs are working, i'll just approve some trivial changes to those and they should pop right in | 12:48 |
| AJaeger | fungi, let's test the theory when you're not in fire fighting ;) | 12:49 |
| AJaeger | fungi, speaking about approving patches - the translation "compression" works fine for manuals, I think we can take the next step, a review is appreciated: https://review.openstack.org/176947 | 12:50 |
| sdague | fungi: so https://review.openstack.org/#/c/178608/ will get picked up in the image builds today? | 12:50 |
| fungi | kashyap: i invoke it like `BROWSER=w3m gertty` but you could substitute w3m with the name of whatever script does the downloading you want | 12:50 |
| *** MaxV has joined #openstack-infra | 12:50 | |
| fungi | sdague: in theory, yes | 12:50 |
| sdague | coolio | 12:50 |
| kashyap | sdague: Yes, if only I can priortize like a ninja | 12:50 |
| *** mpaolino has joined #openstack-infra | 12:51 | |
| kashyap | fungi: Oh, cool. I already use `elinks` for mutt. | 12:51 |
| sdague | kashyap: so, honestly, when tracing bugs like this you are going to be jumping back and forth between probably 4 - 6 log files. So I find tabs in browsers works fairly well | 12:51 |
| kashyap | fungi: Any reason you prefer w3m over elinks? | 12:51 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 12:52 |
| kashyap | sdague: Yep, I see what you're saying - it's not that I don't appreciate the ability to easily switch back n forth in the browser :-) | 12:52 |
| maishsk_ | fungi: can you guys help me find out why I am not receiving all the emails from openstack-dev? | 12:52 |
| *** maishsk_ is now known as maishsk | 12:53 | |
| fungi | kashyap: it was an example. actually playing around right now with lynx vs w3m vs links vs elinks, lynx seems to be the best and handling our job logs and making them readable on an 80x24 character terminal | 12:54 |
| *** amitgandhinz has joined #openstack-infra | 12:54 | |
| *** dhritishikhar has quit IRC | 12:54 | |
| *** dhritishikhar has joined #openstack-infra | 12:54 | |
| fungi | maishsk: do you have an example of a message you didn't receive? | 12:54 |
| fungi | maishsk: or is it all of them? | 12:54 |
| *** e0ne is now known as e0ne_ | 12:54 | |
| kashyap | fungi: Oh, really, thanks for the test w/ lynx | 12:55 |
| maishsk | Anything in this thread http://lists.openstack.org/pipermail/openstack-dev/2015-April/062914.html | 12:55 |
| maishsk | absolutely not a single mail from this thread | 12:55 |
| *** ibiris is now known as ibiris_away | 12:55 | |
| fungi | kashyap: also, minor bug (i'll probably get around to submitting a patch soon if nobody else beats me to it), after exiting the browser you need to hit ctrl-l to refresh the gertty display | 12:56 |
| *** asahlin_afk is now known as asahlin | 12:56 | |
| kashyap | fungi: From time to time the sqlite databse Gertty uses goes corrupt or something? If I start gertty after a few weeks, | 12:57 |
| kashyap | It almost always errors out trying to sync. I haven't dug into it deeply though | 12:57 |
| fungi | kashyap: it doesn't handle git repos being renamed in gerrit. my post-maintenance notices have included example sqlite commands for updating the gertty database | 12:58 |
| fungi | kashyap: generally that only affects it if you've subscribed to a repo that then gets renamed | 12:58 |
| kashyap | fungi: Oh, notices to the -dev list? | 12:59 |
| fungi | or possibly managed to review a change in one of those repos even if not subscribed | 12:59 |
| fungi | kashyap: yeah, i can find you an example after i'm done looking into listserv logs for maishsk | 12:59 |
| *** shardy_ has joined #openstack-infra | 12:59 | |
| *** yfried has quit IRC | 12:59 | |
| kashyap | fungi: Sure, no rush. Thanks for looking. | 13:00 |
| *** sandywalsh has joined #openstack-infra | 13:00 | |
| *** shardy has quit IRC | 13:01 | |
| *** ir2ivps has quit IRC | 13:01 | |
| fungi | maishsk: is it possible you've set your openstack-dev subscription to not include messages with the [all] (cross-project discussions) tag? | 13:01 |
| *** Alexandra_ has joined #openstack-infra | 13:02 | |
| maishsk | fungi: How would I check that? | 13:02 |
| fungi | maishsk: log in at http://lists.openstack.org/cgi-bin/mailman/options/openstack-dev (do a password reminder first if you need) | 13:03 |
| *** annegentle has joined #openstack-infra | 13:03 | |
| maishsk | fungi: checking... | 13:04 |
| fungi | maishsk: then for the "Which topic categories would you like to subscribe to?" option, make sure "Cross-project coordination" is included | 13:04 |
| fungi | that subscribes you to messages with [all] in the subject line | 13:04 |
| maishsk | ah... Crap - you are right!! | 13:04 |
| maishsk | fungi: Thank you. | 13:05 |
| fungi | maishsk: any time! | 13:05 |
| *** shardy_ has quit IRC | 13:05 | |
| *** julim has joined #openstack-infra | 13:05 | |
| *** spzala has quit IRC | 13:05 | |
| jistr | hi folks, could someone from infra-core please take a look at this change when you have some time? it has +1 from both PTLs https://review.openstack.org/#/c/176226 | 13:05 |
| *** shardy has joined #openstack-infra | 13:05 | |
| *** e0ne_ is now known as e0ne | 13:06 | |
| maishsk | fungi: but wouldn't selecting 'yes' on Do you want to receive messages that do not match any topic filter? cover that? | 13:06 |
| *** julim has quit IRC | 13:06 | |
| *** dhritishikhar_ has joined #openstack-infra | 13:06 | |
| *** bknudson has joined #openstack-infra | 13:06 | |
| *** radez_g0n3 is now known as radez | 13:08 | |
| fungi | maishsk: no, that's for messages which aren't covered by any of the matching rules | 13:08 |
| maishsk | fungi: OK - that explains it! | 13:08 |
| *** ZZelle has quit IRC | 13:08 | |
| *** julim has joined #openstack-infra | 13:08 | |
| fungi | maishsk: so for example if i send a message to the list with no [tag] in the subject line at all, or an unrecognized tag like [xyzzy] | 13:08 |
| *** ZZelle has joined #openstack-infra | 13:09 | |
| fungi | kashyap: see the notes/example for renaming congress in the gertty db at the end of http://lists.openstack.org/pipermail/openstack-dev/2015-April/061878.html | 13:09 |
| *** ibiris_away is now known as ibiris | 13:09 | |
| * kashyap clicks | 13:09 | |
| fungi | kashyap: also i think if you look in gertty's error log you should see what database keys it's having trouble with | 13:10 |
| kashyap | fungi: Yeah, I was indeed looking at a database error | 13:10 |
| *** dhritishikhar has quit IRC | 13:10 | |
| kashyap | fungi: I'm afraid, guess now I'm too late to catch the exact error as I deleted the db and started a resync again :( | 13:11 |
| *** ociuhandu has joined #openstack-infra | 13:12 | |
| *** bhunter71 has joined #openstack-infra | 13:13 | |
| *** deepakcs has quit IRC | 13:14 | |
| *** julim has quit IRC | 13:18 | |
| *** abregman_ has joined #openstack-infra | 13:18 | |
| *** abregman has quit IRC | 13:20 | |
| *** tiswanso has joined #openstack-infra | 13:22 | |
| *** bhunter71 has quit IRC | 13:22 | |
| *** mattfarina has joined #openstack-infra | 13:23 | |
| kashyap | fungi: And, btw - thanks to the infra team for Gertty - it really made using Gerrit palatable in my workflow :) | 13:23 |
| *** mattfarina has quit IRC | 13:24 | |
| fungi | kashyap: i can't take responsibility for that. it's a personal project of jeblair's (to whom i too am thankful for this awesome piece of software) | 13:24 |
| *** amitgandhinz has quit IRC | 13:25 | |
| kashyap | fungi: Yep, that's what I meant, thanks to jeblair too! | 13:25 |
| fungi | it's technically not an infra project, just happens to be written by our illustrious ptl | 13:25 |
| fungi | all praise be unto him | 13:25 |
| *** peristeri has joined #openstack-infra | 13:25 | |
| *** vhoward has joined #openstack-infra | 13:26 | |
| *** abregman_ has quit IRC | 13:29 | |
| kashyap | fungi: Any plans to make it an "official" (maybe it'll help test/catch more bugs) infra project | 13:30 |
| kashyap | ? | 13:30 |
| fungi | kashyap: i'm not opposed to the idea, but it's up to jeblair since it's his project (and i think he enjoys not being encumbered by our typical bureaucracy when he's hacking on something for fun like that) | 13:31 |
| kashyap | fungi: Sure. And, I totally understand his hesitation for fear of beaurocracy. | 13:32 |
| kashyap | s/fear/burden? :-) | 13:32 |
| fungi | clarkb: when you're around, what's a good way to troubleshoot boot issues with dib nodepool images in hpcloud? i think we're dib'ing unbootable devstack-precise images (they fail nodepool's ssh test and get deleted immediately) | 13:32 |
| *** ir2ivps has joined #openstack-infra | 13:33 | |
| fungi | clarkb: does the novaclient console support work for cases like that? | 13:33 |
| fungi | if so, i'll try to dig into it later today | 13:34 |
| *** jtriley has joined #openstack-infra | 13:34 | |
| *** EmilienM|afk is now known as EmilienM | 13:37 | |
| *** dustins has joined #openstack-infra | 13:37 | |
| *** e0ne is now known as e0ne_ | 13:38 | |
| *** lifeless has joined #openstack-infra | 13:38 | |
| claudiub | fungi: hi, I would require your assistance. I'm having trouble pushing a new version of networking-hyperv to gerrit | 13:39 |
| *** e0ne_ is now known as e0ne | 13:39 | |
| *** ddieterly has joined #openstack-infra | 13:39 | |
| *** MaxV has quit IRC | 13:39 | |
| fungi | claudiub: you mean pushing a networking-hyperv change to review? | 13:39 |
| fungi | claudiub: what's the error? | 13:40 |
| claudiub | fungi: no, pushing a tag | 13:40 |
| claudiub | fungi: for a new release, 1.0.3 | 13:40 |
| fungi | oh! yes. following http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release hopefully? | 13:40 |
| claudiub | ubuntu@ubuntu:/opt/stack/hyperv/networking-hyperv$ git push gerrit 1.0.3 | 13:40 |
| claudiub | ! [remote rejected] 1.0.3 -> 1.0.3 (prohibited by Gerrit) | 13:40 |
| claudiub | yes | 13:40 |
| fungi | you used git tag -s right (so that it's signed)? | 13:41 |
| claudiub | did a checkout master, git pull, git tag -s 1.0.3 | 13:41 |
| claudiub | yeah | 13:41 |
| *** esker has quit IRC | 13:41 | |
| claudiub | and i did a git tag --verify | 13:41 |
| claudiub | ubuntu@ubuntu:/opt/stack/hyperv/networking-hyperv$ git tag --verify 1.0.3 object e625ea751f6eadd65c88d1a0eb2eb25eeea3d651 type commit tag 1.0.3 tagger Claudiu Belu <cbelu@cloudbasesolutions.com> 1430314358 -0700 | 13:41 |
| *** pabelanger has joined #openstack-infra | 13:42 | |
| fungi | then chances are either the acl doesn't have a section to permit tagging, or you're not a member of the group currently authorized by that acl. checking now... | 13:42 |
| claudiub | networking-hyperv Release 1.0.3 gpg: Signature made Wed 29 Apr 2015 06:32:38 AM PDT using RSA key ID 8450D996 gpg: Good signature from "Claudiu Belu <cbelu@cloudbasesolutions.com>" | 13:42 |
| fungi | the acl itself looks fine | 13:43 |
| claudiub | then I wonder what could be wrong.. | 13:43 |
| fungi | claudiub: get the current member of https://review.openstack.org/#/admin/groups/604,members to add you to that group | 13:44 |
| mescanef | hello, please add me (mmatuszkowiak@mirantis.com) to the gerrit groups of a project regarding the https://review.openstack.org/#/c/174763/ :-) | 13:44 |
| *** abregman_ has joined #openstack-infra | 13:44 | |
| mescanef | fungi: ^^ pls :-) | 13:44 |
| claudiub | fungi: ah, i see... I talk to alexpilotti then. thanks. :D | 13:44 |
| fungi | mescanef: i think i did already--have you checked? | 13:44 |
| fungi | saw your request in the channel history when i woke up and did it then | 13:45 |
| mescanef | fungi: oh yeah, thanks :-) | 13:45 |
| fungi | no problem--have fun! | 13:45 |
| *** Alexandra_ has quit IRC | 13:46 | |
| *** pabelanger has quit IRC | 13:46 | |
| *** flaviosr has joined #openstack-infra | 13:47 | |
| claudiub | fungi: added to the list you've linked, pushing the new version worked, thanks! :D | 13:47 |
| *** prad has joined #openstack-infra | 13:48 | |
| fungi | claudiub: you're welcome! | 13:49 |
| *** mriedem_away has quit IRC | 13:51 | |
| *** marcusvrn1 has quit IRC | 13:54 | |
| *** mtreinish has joined #openstack-infra | 13:54 | |
| *** hdd has quit IRC | 13:54 | |
| *** marcusvrn has joined #openstack-infra | 13:55 | |
| *** asselin has joined #openstack-infra | 13:56 | |
| *** liam_ has joined #openstack-infra | 13:58 | |
| sdague | fungi: did we lose gerrit bot? | 13:58 |
| *** BobH1 has joined #openstack-infra | 13:58 | |
| fungi | we may have lost gerrit | 13:58 |
| fungi | checking | 13:58 |
| *** liam_ is now known as Guest38949 | 13:58 | |
| fungi | yeah, probably have a dead stream-events queue for the last 30 minutes judging from the most recent change age in the check pipeline | 13:59 |
| fungi | confirming and collecting some more debug details now before restarting gerrit _again_ | 13:59 |
| *** bswartz has joined #openstack-infra | 14:00 | |
| *** dustins_ has joined #openstack-infra | 14:00 | |
| *** yfried has joined #openstack-infra | 14:00 | |
| fungi | most recent event zuul saw was 13:29:24 | 14:01 |
| *** BobH has quit IRC | 14:01 | |
| *** BobH1 has quit IRC | 14:01 | |
| *** zz_jgrimm is now known as jgrimm | 14:01 | |
| *** sslypushenko has quit IRC | 14:02 | |
| *** BobH has joined #openstack-infra | 14:02 | |
| *** sslypushenko has joined #openstack-infra | 14:02 | |
| *** dalgaaf has joined #openstack-infra | 14:02 | |
| *** mriedem has joined #openstack-infra | 14:03 | |
| *** dustins has quit IRC | 14:03 | |
| *** dkranz has quit IRC | 14:04 | |
| *** maishsk has quit IRC | 14:04 | |
| *** marun has joined #openstack-infra | 14:04 | |
| fungi | #status notice gerrit has been restarted to clear a stuck events queue. any change events between 13:29-14:05 utc should be rechecked or have their approval votes reapplied to trigger jobs | 14:04 |
| openstackstatus | fungi: sending notice | 14:04 |
| *** alexpilotti has joined #openstack-infra | 14:05 | |
| *** mtreinish has quit IRC | 14:05 | |
| -openstackstatus- NOTICE: gerrit has been restarted to clear a stuck events queue. any change events between 13:29-14:05 utc should be rechecked or have their approval votes reapplied to trigger jobs | 14:05 | |
| fungi | so... in GOOD news, i have corresponding jstack dumps, connection list and queue list (including jeblair's debug details showing corresponding gerrit accounts for each task now) | 14:06 |
| fungi | i'll see if i can make heads or tails of any of it | 14:06 |
| openstackstatus | fungi: finished sending notice | 14:06 |
| openstackgerrit | Russell Bryant proposed openstack/requirements: global-requirements: Add ovs https://review.openstack.org/177891 | 14:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Decrease PO files for python projects https://review.openstack.org/176947 | 14:07 |
| fungi | looks like gerritbot is happy again | 14:08 |
| *** otter768 has joined #openstack-infra | 14:08 | |
| *** btully has joined #openstack-infra | 14:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Decrease PO files for imports (horizon+friends) https://review.openstack.org/176943 | 14:09 |
| fungi | 15 hung stream events tasks for netapp-ci | 14:09 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:09 | |
| fungi | 8 for neutronryu | 14:10 |
| *** nfedotov has joined #openstack-infra | 14:10 | |
| fungi | the others are 5 or fewer | 14:10 |
| *** mase_x200 has joined #openstack-infra | 14:11 | |
| *** pabelanger has joined #openstack-infra | 14:11 | |
| fungi | 17 connections from netapp-ci, 12 from neutronryu, 11 from our ci system (zuul and all its mergers) | 14:12 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process https://review.openstack.org/170830 | 14:12 |
| *** otter768 has quit IRC | 14:12 | |
| *** annegentle has quit IRC | 14:12 | |
| *** amitgandhinz has joined #openstack-infra | 14:14 | |
| *** yamahata has joined #openstack-infra | 14:14 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Allow Magnum to conform to global requirements process https://review.openstack.org/170830 | 14:15 |
| sdague | so, that seems excessive | 14:16 |
| sdague | any idea why so many connections? | 14:16 |
| fungi | well, looking at the connections list, netapp-ci connects from at least half a dozen different addresses in widely varying netblocks | 14:16 |
| fungi | so my guess is that one account is shared by a lot of different ci systems | 14:16 |
| *** abregman_ has quit IRC | 14:17 | |
| *** annegentle has joined #openstack-infra | 14:17 | |
| sdague | ok, are they really reporting that many different ways? maybe thingee or someone could help illuminate there | 14:17 |
| fungi | there's not necessarily any correlation between the accounts connecting and listening to the event stream and the hung state it gets into | 14:17 |
| sdague | that's true | 14:17 |
| frickler | fungi: where does the statusbot get its channel list from? I've seen the gerrit notice on some openstack channels, but not on all of them | 14:17 |
| *** isviridov is now known as isviridov_away | 14:18 | |
| fungi | frickler: inconveniently, https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n270 | 14:18 |
| pabelanger | Ohai | 14:18 |
| *** flaviosr has quit IRC | 14:19 | |
| frickler | fungi: o.k., will do a patch for that, thx | 14:19 |
| *** erikmwil_ has joined #openstack-infra | 14:20 | |
| *** erikmwilson is now known as Guest84636 | 14:20 | |
| *** erikmwil_ is now known as erikmwilson | 14:20 | |
| thingee | sdague: they have about 6 drivers. Not sure how they have things configured, but maybe those have their own CI/stream? Also they may have internal ones. | 14:20 |
| *** rossella_s has quit IRC | 14:20 | |
| *** erikmwilson_ has joined #openstack-infra | 14:21 | |
| *** rossella_s has joined #openstack-infra | 14:21 | |
| thingee | fungi: ^ | 14:22 |
| clarkb | fungi: usually I would debug by booting a test node then checking the `nova console-log` output for it, you can get an interactive console too if that helps but unsure if we can actually login unless it has dropped to root shell | 14:22 |
| thingee | sdague: I'm still getting ready to head into the office, but I can talk to folks listed here: https://wiki.openstack.org/wiki/ThirdPartySystems/NetApp_CI | 14:22 |
| pabelanger | So, wanted to officially offer my support to -infra now that I'm full time employed. For those unware, I took a position with Red Hat last week with the mandate to help support upstream infrastructure when ever possible. Plan to be at the summit in Vancouver also, so I can finally put a face to handles. | 14:22 |
| clarkb | fungi: then if that doesn't shed any light you can grab the image used and boot locally against kvm, I used greghaynes, d3.sh to make that pretty easy | 14:22 |
| pabelanger | either way, just tossing that out there. | 14:23 |
| *** jtriley has quit IRC | 14:23 | |
| AJaeger | pabelanger, great! | 14:23 |
| sdague | clarkb: hey, you are awake. Want to quick review - https://review.openstack.org/#/c/178678/ - which should make it easier to share ER links | 14:24 |
| pabelanger | But, did have a thought / question last night. I wanted to see what people thought about maybe adding grafana into -infra, so people have the ability to generate dashboards based on the nodepool / zuul / jenkins. Tried searching around for info, but didn't see too much chatter about it | 14:24 |
| *** ajmiller__ has quit IRC | 14:25 | |
| pabelanger | It wouldn't replace statsd / granite but sit a top of it | 14:25 |
| anteaya | pabelanger: welcome and yay!! | 14:25 |
| AJaeger | clarkb, fungi : Is there an easy way to get the post jobs for documentation run with higher priority? | 14:25 |
| clarkb | and now that I have caught up I see gerrit was restarted | 14:25 |
| pabelanger | I used it a while ago for some call center stats, and found it was super flexible for rendering data | 14:26 |
| clarkb | AJaeger: we can change the post job priority back to being on the same level as check | 14:26 |
| pabelanger | AJaeger, anteaya: Thanks | 14:26 |
| *** asrangne has joined #openstack-infra | 14:26 | |
| AJaeger | WE currently have to wait 3 hours for publishing content - and that blocks us, since we need to prepare for Kilo and test some things | 14:26 |
| sdague | AJaeger: I don't understand why it's waiting | 14:26 |
| anteaya | pabelanger: as for graphana I think someone may have mentioned it recently but I have forgotten who and the context | 14:26 |
| sdague | we have the nodes | 14:26 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: allow hash urls to work before page loads https://review.openstack.org/178678 | 14:27 |
| clarkb | sdague: beacuse check wins, so we have to make check completely satisfied before any jobs go to post | 14:27 |
| sdague | clarkb: sure | 14:27 |
| sdague | but we've got an extra 400 nodes | 14:27 |
| clarkb | sdague: I think its a case of where we don't predict enough extra nodes to build so as new patchsets come in they "steal" from the nodes allocated to post | 14:27 |
| anteaya | pabelanger: my best suggestion is stick around in channel and learn the answers to common questions so you too can hand them out | 14:27 |
| *** esker has joined #openstack-infra | 14:27 | |
| clarkb | we build what we need then what we need gets larger | 14:27 |
| *** zz_dimtruck is now known as dimtruck | 14:27 | |
| sdague | clarkb: so the fix there is just to up the extra nodes? | 14:28 |
| pabelanger | anteaya, Ya, plan is for me to do local test and see what I can consume from statsd upstream. Not sure any authenticate is required, but could be wrong | 14:28 |
| *** aswadr has quit IRC | 14:28 | |
| sdague | post competing with check puts us into other weird hangs | 14:28 |
| *** tiswanso has quit IRC | 14:28 | |
| pabelanger | anteaya, Agreed, I know some people I talked to, trying to support RHT efforts upstream were interested in the ability to generate dashboard to easily view data | 14:28 |
| anteaya | pabelanger: okay, for any new features I defer to jeblair who is currently traveling i believe | 14:28 |
| *** wenlock has joined #openstack-infra | 14:28 | |
| clarkb | I think the real fix is to change post's priority then the numbers we calculate will be more accurate for post | 14:28 |
| pabelanger | /me nods | 14:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Give post queue normal priority https://review.openstack.org/178696 | 14:29 |
| AJaeger | clarkb, sdague: Thanks for the info, here's a patch ^ | 14:29 |
| anteaya | pabelanger: if you want to know what is important to infra our meeting logs are helpful: http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-04-28-19.04.txt | 14:29 |
| anteaya | pabelanger: any priority effort is something that we have identified as important | 14:29 |
| *** tiswanso has joined #openstack-infra | 14:29 | |
| sdague | clarkb: that seems wrong to me, because we do specifically want post to be handled after check | 14:29 |
| sdague | and check win when there is contention | 14:30 |
| anteaya | pabelanger: my suggestion would be help with any priority effort, and the propose what you want to the queue | 14:30 |
| clarkb | sdague: but nodepool has no concept of queue priority | 14:30 |
| AJaeger | clarkb, for the other suggestion: I don't know how to change that | 14:30 |
| jaypipes | pcrews: looking at https://github.com/pcrews/rannsaka. Very cool. I've been wanting to look at locust.io for a while now. rannsaka is a good way to do that :) BTW, are you familiar with Rally's spawning internals? /me wonders if locust.io might be a good replacement spawn/collect engine for it. | 14:30 |
| *** abregman_ has joined #openstack-infra | 14:30 | |
| anteaya | pabelanger: then when a priority effort is finished, there are space for others which are drawn from the queue | 14:30 |
| anteaya | :) | 14:30 |
| pabelanger | indeed | 14:30 |
| clarkb | sdague: so while we could say always build 5 extra nodes (or some arbitrary number) that isn't going to directly address the issue, it may help make it better but only if we don't exceed that number with inoming check patchsets | 14:30 |
| sdague | clarkb: so if we're actually slamming that hard, I think that's fine | 14:31 |
| *** julim has joined #openstack-infra | 14:31 | |
| sdague | but looking at the ready pool count, I don't understand how things could be hanging out for 3 hours to get docs nodes | 14:31 |
| *** wenlock_ has joined #openstack-infra | 14:31 | |
| sdague | we've *always* had available nodes | 14:31 |
| *** dkranz has joined #openstack-infra | 14:32 | |
| fungi | i'm guessing we have too many of the "wrong" node types in a ready state right now because of an earlier poor prediction | 14:32 |
| fungi | checking now | 14:32 |
| clarkb | ya I am looking too | 14:32 |
| clarkb | a good chunk are tripleo nodes | 14:32 |
| fungi | pabelanger: also thanks and welcome! | 14:32 |
| *** tiswanso has quit IRC | 14:32 | |
| clarkb | the others look like potentially "leaked" nodes from hpcloud | 14:33 |
| clarkb | oh there are some from rax too so likely not a cloud specific issue | 14:33 |
| clarkb | https://jenkins06.openstack.org/computer/bare-trusty-hpcloud-b1-2328080/ is alive and ready | 14:34 |
| clarkb | has been for almost 2 days | 14:34 |
| sdague | AJaeger: it might also be optimizing to not have 4 distinct post jobs for the manuals, so that you need 4 nodes per change | 14:34 |
| fungi | we have 66 bare-trusty nodes in a ready state | 14:34 |
| *** Krinkle|detached is now known as Krinkle | 14:34 | |
| fungi | definitely abnormal | 14:35 |
| AJaeger | sdague, let me see what I can do there... | 14:35 |
| clarkb | https://jenkins02.openstack.org/computer/bare-trusty-hpcloud-b1-2352977/ is another case of the above | 14:35 |
| clarkb | fungi: so they are in jenkins just fine, I wonder if they do not have gearman connection? | 14:35 |
| fungi | if a lot of those aren't picking up jobs for some reason, then that would explain why changes are sitting waiting for a longer than usual time (because nodepool doesn't realize it needs to make more) | 14:35 |
| clarkb | fungi: I am going to try taking 2328080 offline in jenkins and bring it back to see if it gets used | 14:35 |
| fungi | clarkb: that would be my guess | 14:36 |
| sdague | fungi: so... could the gerrit event stream hangs contribute to the orphans? | 14:36 |
| fungi | sdague: i don't believe so, no | 14:36 |
| fungi | clarkb: though it's just one gearman connection to zuul from each jenkins master right? | 14:36 |
| *** tiswanso_ has joined #openstack-infra | 14:37 | |
| clarkb | fungi: no its a connection for each slave | 14:37 |
| fungi | ahh, okay | 14:37 |
| AJaeger | sdague, publishlang and publishdocs are both "long" running jobs. Merging them is possible but then it's sequential... | 14:37 |
| AJaeger | (long means around 20-30 mins) | 14:37 |
| sdague | AJaeger: what kind of node do they run on ? | 14:37 |
| sdague | those seem to be the ones not getting allocated nodes | 14:38 |
| clarkb | fungi: I have found the gearman worker thread for 2328080 and it seems to be alive and working (running epoll) | 14:38 |
| sdague | fungi / clarkb: what's node build time now adays? | 14:38 |
| *** wznoinsk has quit IRC | 14:39 | |
| AJaeger | sdague, should be "bare-precise || bare-trusty" | 14:39 |
| fungi | sdague: depends a lot on local conditions within each provider/region | 14:39 |
| sdague | AJaeger: under load you are also going to pay node build time cost, so it's only cheaper to make them parallel if the whole system is idel | 14:39 |
| AJaeger | sdague, so, worth to create a job that runs for an hour? | 14:39 |
| fungi | AJaeger: i believe we're starved on bare-trusty jobs right now. nodepool thinks there are a lot ready but most of those aren't actually picking up tasks | 14:40 |
| clarkb | but so far everything says they should pick up tasks just fine | 14:40 |
| clarkb | I don't know that I will be able to map 2328080 to a specific tcp connection but I wonder if the remote end disagrees that there is something to epoll on | 14:41 |
| AJaeger | this situation is there for some time already, this is not something suddenly changing today. It's just hurting us more today ;) | 14:41 |
| sdague | AJaeger: it might be, also your other jobs that run in about 50s that run in post would be good to consolidate | 14:41 |
| *** dims has quit IRC | 14:42 | |
| AJaeger | sdague, those are "proposal" nodes | 14:42 |
| *** dims has joined #openstack-infra | 14:42 | |
| AJaeger | (please double check that I'm right with the node types) | 14:42 |
| clarkb | netstat connection counts look sane at least | 14:43 |
| *** wznoinsk has joined #openstack-infra | 14:43 | |
| AJaeger | the upstream-translation job is used by several repos, the propose one is unique to openstack-manuals. | 14:43 |
| AJaeger | Let me check the descriptiosn... | 14:43 |
| *** jamesmcarthur has joined #openstack-infra | 14:43 | |
| clarkb | though jenkins06 claims 89 established connectinos and zuul just 87, but thats not a big enough discepancy to account for these nodes | 14:44 |
| *** mtreinish has joined #openstack-infra | 14:45 | |
| clarkb | fungi: I have confirmed via jenkins logs that 2328080 did do a gearman reconnect as part of my disable enabling it | 14:45 |
| AJaeger | So, consolidation of these jobs is possible - fungi, clarkb, sdague: I wait for your advise whether it's worth it | 14:46 |
| sdague | AJaeger: well, I think it will on average be faster | 14:46 |
| *** mriedem has quit IRC | 14:46 | |
| clarkb | fungi: so far everything looks good other than this node not running jobs | 14:46 |
| fungi | also those ready bare-precise nodes are pretty evenly spread across all jenkins masters (except the one which is disabled) so it seems to be not a localized problem with one jenkins master | 14:47 |
| clarkb | (we should however consider logging the connection details in the gearman plugin since right now there is no way to figure that out from what I can tell) | 14:47 |
| *** bhunter71 has joined #openstack-infra | 14:47 | |
| clarkb | I know I have a connection but don't know which is which | 14:47 |
| clarkb | I want my next step to be tcpdump, maybe I can grep on the snarfed output and see if that shows me what the connection is | 14:48 |
| fungi | there's a possibility, yeah | 14:48 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move server outside of openstack_project::git https://review.openstack.org/178701 | 14:49 |
| *** alexpilotti has quit IRC | 14:49 | |
| *** marcusvrn_ has joined #openstack-infra | 14:49 | |
| *** tonytan4ever has joined #openstack-infra | 14:49 | |
| *** mtreinish has quit IRC | 14:49 | |
| fungi | you can just dump all gearman traffic for the master in question to a pcap file and then use tcpdump to read that file and pare it down to the right connection with an additional pcap filter expression after you've identified the right one by matching on some content in the packets | 14:50 |
| fungi | though you'll probably have to toggle the slave status again to trigger another registration | 14:50 |
| *** katyafervent is now known as katyafervent_awa | 14:52 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/project-config: Add tripleo-core to puppet-tripleo https://review.openstack.org/176226 | 14:52 |
| *** BharatK_ has quit IRC | 14:52 | |
| *** mriedem has joined #openstack-infra | 14:52 | |
| *** claudiub has quit IRC | 14:52 | |
| clarkb | `sudo tcpdump -i any -A port 4730 | grep 2328080` on jenkins06 has not found anything even after toggling that node's state | 14:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Merge some openstack-manuals post jobs https://review.openstack.org/178705 | 14:52 |
| AJaeger | sdague, fungi, clarkb: here's what sdague proposed ^ | 14:53 |
| openstackgerrit | Gary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack https://review.openstack.org/175063 | 14:53 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Merge some openstack-manuals post jobs https://review.openstack.org/178705 | 14:53 |
| fungi | AJaeger: sdague: that probably doesn't make any difference at the moment. those run on the proposal.slave.openstack.org worker, which is static and doesn't deplete our worker pool | 14:54 |
| sdague | AJaeger: ok, it's still probably worth letting clarkb and fungi figure out why we aren't making bare nodes | 14:54 |
| *** tteggel_ has quit IRC | 14:54 | |
| sdague | fungi: ok, I didn't know about proposal node being static | 14:54 |
| AJaeger | fungi, ah - ok, I abandon.... | 14:55 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/requirements: Add python-manilaclient to global requirements. https://review.openstack.org/135007 | 14:55 |
| AJaeger | sdague, I didn't remember either... | 14:55 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/requirements: Bump ddt to min version 0.7.0 https://review.openstack.org/166162 | 14:55 |
| *** katyafervent_awa is now known as katyafervent | 14:55 | |
| fungi | also it doesn't look like we had any trouble running those. they've all completed. it's stuff running on bare-trusty nodepool workers which are currently blocked | 14:55 |
| *** emagana has joined #openstack-infra | 14:55 | |
| *** tteggel has joined #openstack-infra | 14:56 | |
| sdague | fungi: yeh, I just didn't realize they weren't sucking up nodes | 14:56 |
| clarkb | oh right nodes don't register with any more specific info that their label iirc, so the plan to grep off of the contents in the gearman packets is probably a bad plan | 14:57 |
| *** unicell has joined #openstack-infra | 14:57 | |
| *** mase_x200 has quit IRC | 14:57 | |
| *** unicell1 has quit IRC | 14:57 | |
| *** imcsk8|zZz is now known as imcsk8 | 14:57 | |
| fungi | other than possibly timing of the registration datagram? | 14:58 |
| AJaeger | fungi, clarkb, sdague: Thanks for looking into this! I'm cycling home soon and will log back in later this evening | 14:58 |
| *** lucap has quit IRC | 14:58 | |
| fungi | clarkb: if you can narrow down the registration message to the timeframe where you toggled the status back to active | 14:58 |
| *** cody-somerville has quit IRC | 14:58 | |
| *** jtriley has joined #openstack-infra | 14:58 | |
| fungi | chances are nodepool's not adding lots of bare-trusty workers right this moment | 14:59 |
| openstackgerrit | Gal Sagie proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow https://review.openstack.org/175152 | 14:59 |
| *** patrickeast has joined #openstack-infra | 15:00 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/system-config: Add two missing channels to statusbot https://review.openstack.org/178709 | 15:00 |
| clarkb | ya pulling up gearman protocol doc now to see what I should be looking for | 15:00 |
| *** gsagie has joined #openstack-infra | 15:00 | |
| gsagie | Hello all, any review for https://review.openstack.org/#/c/175152/ will be appreciated | 15:00 |
| openstackgerrit | Brant Knudson proposed openstack/requirements: Document discover used for python26 https://review.openstack.org/178710 | 15:01 |
| *** packet has joined #openstack-infra | 15:02 | |
| *** nelsnelson has joined #openstack-infra | 15:02 | |
| *** gary-smith has joined #openstack-infra | 15:04 | |
| *** BharatK_ has joined #openstack-infra | 15:04 | |
| *** gsagie has left #openstack-infra | 15:04 | |
| *** peristeri has quit IRC | 15:04 | |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/project-config: Add grenade jobs to Heat https://review.openstack.org/178352 | 15:05 |
| openstackgerrit | Merged openstack-infra/storyboard: Refactored event worker plugin structure into plugin directory. https://review.openstack.org/176530 | 15:05 |
| openstackgerrit | Merged openstack-infra/storyboard: Every worker now gets its own session. https://review.openstack.org/176545 | 15:06 |
| openstackgerrit | Merged openstack-infra/storyboard: Pre-resolve author record for events. https://review.openstack.org/176546 | 15:06 |
| *** alexpilotti has joined #openstack-infra | 15:11 | |
| pabelanger | what flavor of OS is -infra defaulting to now? I still see 12.04 in launch-node script, but also know 14.04 has been used | 15:13 |
| ryanpetrello_ | It's pretty well decided that OpenStack will *not* be supporting Python 3.2, yes? | 15:13 |
| *** claudiub has joined #openstack-infra | 15:13 | |
| ryanpetrello_ | considering removing the support from pecan because I doubt anybody is actually using it | 15:13 |
| *** wznoinsk_ has joined #openstack-infra | 15:14 | |
| fungi | ryanpetrello_: yep. or 3.3 for that matter | 15:15 |
| fungi | ryanpetrello_: our current py3k testing upstream is all with 3.4 | 15:15 |
| ryanpetrello_ | okay, good to know | 15:15 |
| ryanpetrello_ | thanks :) | 15:15 |
| fungi | pabelanger: we're starting to put new things on trusty and slowly move services from precise to trusty as opportunity allows | 15:15 |
| *** dprince has quit IRC | 15:16 | |
| pabelanger | fungi, thanks | 15:16 |
| *** miqui has joined #openstack-infra | 15:17 | |
| fungi | most recent thing to move was gerrit | 15:17 |
| *** rlandy has quit IRC | 15:17 | |
| *** wznoinsk__ has joined #openstack-infra | 15:18 | |
| fungi | and people are contributing patches for, e.g., apache 2.4 configuration support so that more things will hopefully work on trusty when we get around to rebuilding a particular machine | 15:18 |
| *** vlaza has left #openstack-infra | 15:20 | |
| *** heyongli has quit IRC | 15:21 | |
| pabelanger | fungi, okay, sold. Going to trying building new server using trusty and see how it rolls | 15:22 |
| *** heyongli has joined #openstack-infra | 15:22 | |
| *** wznoinsk has quit IRC | 15:23 | |
| *** eharney has quit IRC | 15:23 | |
| *** wznoinsk__ has quit IRC | 15:24 | |
| *** wznoinsk_ is now known as wznoinsk | 15:24 | |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient https://review.openstack.org/178285 | 15:24 |
| openstackgerrit | Alexander Tivelkov proposed openstack-infra/project-config: Added branch creation to yaql ACL https://review.openstack.org/177917 | 15:25 |
| fungi | pabelanger: lots of people are running zuul on trusty, for example. we just haven't gotten around to rebuilding ours yet | 15:28 |
| fungi | so it's assumed it probably works, but i don't have first hand experience making ti work | 15:28 |
| pabelanger | fungi, cool. Ya, last zuul stand-up was still precise. | 15:28 |
| pabelanger | Some how I suspect next one will be centos 7 :) | 15:28 |
| fungi | that could be interesting | 15:29 |
| fungi | i hear the latest opensuse release is really nifty too | 15:29 |
| pabelanger | Ya, plan to start working on zuul RPM shortly. Get it up into zuul-packaging | 15:29 |
| *** cody-somerville has joined #openstack-infra | 15:31 | |
| *** cody-somerville has quit IRC | 15:31 | |
| *** cody-somerville has joined #openstack-infra | 15:31 | |
| *** hdd has joined #openstack-infra | 15:31 | |
| *** dangers_away is now known as dangers | 15:32 | |
| fungi | pabelanger: awesome--have you seen the packaging discussion on the infra ml? StevenK and you should definitely collaborate on cross-distro ideas there | 15:32 |
| *** peristeri has joined #openstack-infra | 15:33 | |
| *** Guest38949 has quit IRC | 15:33 | |
| *** liam_ has joined #openstack-infra | 15:33 | |
| *** liam_ has quit IRC | 15:33 | |
| pabelanger | fungi, Yes, exciting times. Looking forward to it and eager to help | 15:34 |
| *** sdake has joined #openstack-infra | 15:34 | |
| *** frobware_ has joined #openstack-infra | 15:37 | |
| *** mtanino has joined #openstack-infra | 15:38 | |
| *** bswartz has quit IRC | 15:39 | |
| *** sabeen has joined #openstack-infra | 15:40 | |
| clarkb | fungi: I think I have narrows it down, port 55192 on jenkins06 appears to the source side of the gearman connection for 2328080 | 15:40 |
| *** ildikov has quit IRC | 15:41 | |
| *** stevemar has joined #openstack-infra | 15:42 | |
| *** jamesmcarthur has quit IRC | 15:43 | |
| openstackgerrit | Terry Howe proposed openstack-infra/project-config: Add OpenStack Python SDK gate https://review.openstack.org/177554 | 15:43 |
| *** julim has quit IRC | 15:43 | |
| clarkb | fungi: I am running a tcpdump against that port only and not seeing the typical work request traffic that I would expect | 15:44 |
| clarkb | fungi: however the can do function registration appears to have succeeded properly | 15:44 |
| *** mtreinish has joined #openstack-infra | 15:44 | |
| *** dizquierdo has quit IRC | 15:45 | |
| clarkb | iirc gearman workers are supposed to be chatty about requesting work | 15:45 |
| *** armax has joined #openstack-infra | 15:45 | |
| clarkb | also wireshark totally groks gearman which is awesome | 15:45 |
| *** jcoufal has quit IRC | 15:45 | |
| clarkb | yup reading the protocol doc the worker should be sending GRAB_JOB requests | 15:46 |
| *** sdake_ has joined #openstack-infra | 15:47 | |
| openstackgerrit | Terry Howe proposed openstack-infra/project-config: Add OpenStack Python SDK gate https://review.openstack.org/177554 | 15:48 |
| *** hashar has quit IRC | 15:48 | |
| stevemar | bumping keystoneclient for requirements... https://review.openstack.org/#/c/168187/ would be awesome >.> | 15:49 |
| *** dtantsur is now known as dtantsur|afk | 15:50 | |
| *** jamesmcarthur has joined #openstack-infra | 15:51 | |
| *** sdake has quit IRC | 15:51 | |
| clarkb | I am beginning to suspect a gearman plugin bug, possibly we don't change our state on reconnection if we had entered PRE_SLEEP? there are probably a bunch of ways the state machine could break here | 15:52 |
| *** eglynn has joined #openstack-infra | 15:52 | |
| *** rm_work is now known as rm_work|away | 15:53 | |
| *** annegentle has quit IRC | 15:53 | |
| *** asrangne has quit IRC | 15:53 | |
| claudiub | fungi: quick question: if I pushed a new version / tag to gerrit (it appeared on the stackforge repo), shouldn't it be automatically pushed to pypi as well? Or does it take a while? | 15:54 |
| *** dannywilson has joined #openstack-infra | 15:54 | |
| *** dannywilson has quit IRC | 15:54 | |
| *** bswartz has joined #openstack-infra | 15:54 | |
| clarkb | claudiub: what project was the tag made on? | 15:55 |
| *** dannywilson has joined #openstack-infra | 15:55 | |
| claudiub | networking-hyperv | 15:55 |
| *** Sukhdev has joined #openstack-infra | 15:55 | |
| claudiub | clarkb: openstackci is added as an admin user on the networking-hyperv pypi project | 15:56 |
| *** ssam2 has quit IRC | 15:56 | |
| clarkb | claudiub: networking-hyperv does not have the pypi publishing jobs so will not publish to pypi on tags | 15:56 |
| claudiub | clarkb: I see. So, that should have been added to the project-config commit, right? | 15:58 |
| *** sergsh has quit IRC | 15:58 | |
| clarkb | yes | 15:58 |
| claudiub | clarkb: I see, thanks for the info! :D I will create a commit to add those jobs. | 15:58 |
| clarkb | fungi: still no grab_job packets | 15:59 |
| clarkb | fungi: so I don't think this is merely my impatience, it appears to be "dead" | 15:59 |
| fungi | huh | 16:01 |
| fungi | that's weird | 16:01 |
| clarkb | "or any packets at all" | 16:01 |
| fungi | receiving you mean? | 16:02 |
| fungi | at the jenkins end or the zuul end (or are you sniffing at both endpoints)? | 16:02 |
| fungi | well, when you're done exploring available avenues, i can clear all the stale nodes which have been in a ready state for more than an hour | 16:02 |
| clarkb | fungi: I am only sniffing jenkins side | 16:02 |
| clarkb | fungi: because we should see jenkins send GRAB_JOB packets indicating it wants to run a job | 16:02 |
| clarkb | fungi: so I think the state machine in jenkins is borked | 16:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file https://review.openstack.org/178749 | 16:03 |
| *** dprince has joined #openstack-infra | 16:03 | |
| fungi | but just for those nodes? | 16:03 |
| clarkb | fungi: ya, possibly because there is a race with some server response aginst the current state of the jenkins gearman state machines | 16:03 |
| *** zz_ja is now known as zz_zz_ja | 16:03 | |
| *** abregman_ has quit IRC | 16:04 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| clarkb | fungi: so most of the time it just works, but at other times it does not. My hunch is something like if jenkins slave has to reconnect to gearman when it is in a sleep state then it doesn't undo the sleep state and the server never sends a noop to the slave to wake it because after reconnecting it shouldn't be in a sleep state any longer | 16:04 |
| clarkb | reading the code we try to handle this, but I am not sure that we actually do | 16:05 |
| *** jlanoux has quit IRC | 16:05 | |
| pleia2 | morning | 16:06 |
| *** sarob has joined #openstack-infra | 16:06 | |
| *** hemnafk is now known as hemna | 16:06 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 16:07 |
| *** ryanpetrello_ has quit IRC | 16:08 | |
| *** ildikov has joined #openstack-infra | 16:08 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cinder bug 1450110 https://review.openstack.org/178752 | 16:08 |
| openstack | bug 1450110 in tempest "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [Undecided,New] https://launchpad.net/bugs/1450110 | 16:08 |
| *** ryanpetrello_ has joined #openstack-infra | 16:08 | |
| *** devvesa has quit IRC | 16:09 | |
| *** otter768 has joined #openstack-infra | 16:09 | |
| *** mriedem is now known as mriedem_lunch | 16:09 | |
| *** eharney has joined #openstack-infra | 16:09 | |
| *** akshai has joined #openstack-infra | 16:09 | |
| *** Ala has quit IRC | 16:09 | |
| clarkb | pleia2: morning | 16:09 |
| clarkb | fungi: I am going to chew through more of this gearman-plugin state machine after breakfast, but need brain food first | 16:10 |
| fungi | i am similarly lunching | 16:10 |
| *** jistr has quit IRC | 16:11 | |
| gary-smith | clarkb, fungi: if you need a diversion while you eat, could you review https://review.openstack.org/175063 ? :-) | 16:11 |
| *** jamesmcarthur has joined #openstack-infra | 16:12 | |
| bswartz | gary-smith: +1 | 16:13 |
| bswartz | the sooner that change happens the sooner we can proceed with setting up tarballs and packaging | 16:13 |
| gary-smith | clarkb: thanks for the review! | 16:13 |
| *** otter768 has quit IRC | 16:13 | |
| *** ryanpetrello_ has quit IRC | 16:14 | |
| *** zul has quit IRC | 16:15 | |
| *** whoops has joined #openstack-infra | 16:15 | |
| *** ryanpetrello_ has joined #openstack-infra | 16:15 | |
| *** mpaolino has quit IRC | 16:15 | |
| *** julim has joined #openstack-infra | 16:17 | |
| *** Swami has joined #openstack-infra | 16:18 | |
| *** unicell has quit IRC | 16:19 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cinder bug 1450110 https://review.openstack.org/178752 | 16:19 |
| openstack | bug 1450110 in Cinder "boto tests randomly fail in resource_cleanup with "Failed to delete volume"" [Undecided,New] https://launchpad.net/bugs/1450110 | 16:19 |
| *** chlong has quit IRC | 16:19 | |
| *** ociuhandu has quit IRC | 16:20 | |
| *** alexpilotti has quit IRC | 16:20 | |
| *** pcaruana has quit IRC | 16:22 | |
| *** _nadya_ has joined #openstack-infra | 16:22 | |
| *** _nadya_ has quit IRC | 16:23 | |
| *** nfedotov has left #openstack-infra | 16:24 | |
| *** rmcall has joined #openstack-infra | 16:25 | |
| *** gyee has joined #openstack-infra | 16:25 | |
| zaro | morning | 16:25 |
| zaro | yolanda: parallel update is looking good. | 16:26 |
| yolanda | great | 16:26 |
| yolanda | i finally was able to setup my jenkins master and test live there | 16:26 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 16:27 |
| zaro | yolanda: a few more tests but i still may -1 for 'workers' param being in 'test' command | 16:28 |
| openstackgerrit | Gary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack https://review.openstack.org/175063 | 16:28 |
| *** btully has quit IRC | 16:28 | |
| clarkb | zaro please see scrollback I think we have a bug in gearman plugin where slaves never send a grab job request after getting into a bad state | 16:28 |
| *** mpaolino has joined #openstack-infra | 16:28 | |
| yolanda | zaro, ok , initially i just wanted to do a clean rebase, but i'm fine taking ownership of the change and amending things | 16:28 |
| yolanda | i can check with David as well | 16:29 |
| clarkb | zaro my current theory is that if we are sleeping then some event happens that we should stop sleeping but we dont | 16:29 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins https://review.openstack.org/178760 | 16:30 |
| * zaro looing thru scrollbak | 16:30 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Enable ceilometer functional tests https://review.openstack.org/170359 | 16:31 |
| *** arxcruz has quit IRC | 16:31 | |
| *** ivar-lazzaro has quit IRC | 16:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file https://review.openstack.org/178749 | 16:33 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 16:33 |
| *** HeOS has quit IRC | 16:34 | |
| krtaylor | patrickeast, asselin, when you get a chance, please weigh in on https://review.openstack.org/#/c/176519/ | 16:34 |
| *** mattfarina has joined #openstack-infra | 16:34 | |
| *** mattfarina has quit IRC | 16:34 | |
| patrickeast | krtaylor: done | 16:35 |
| clarkb | zaro reading the worker implementation we send ourselves a noop but only in the registerfunctions() method but we seem to use setFunctions() more | 16:36 |
| clarkb | we can problablu simplify that code so it is less confusing | 16:36 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: provide a more targetted location setting https://review.openstack.org/178763 | 16:37 |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 16:40 |
| yolanda | zaro, amended prior to your -1 :) | 16:40 |
| clarkb | fungi did you catch sb from last night? the canonical thibg was apparebtly a demo? | 16:40 |
| clarkb | but I think it supports the theory that its not a ton of stuff | 16:40 |
| yolanda | clarkb, fungi, i was starting with openstack_project::git usage, and i see that there is no upstream module to define git and haproxy there, and it involves some functionality. What do you think of some puppet-git-ha for example ? i see a puppet-cgit, but nothing for ha | 16:41 |
| clarkb | just very persistent | 16:41 |
| *** _nadya_ has joined #openstack-infra | 16:42 | |
| clarkb | yolanda its really just an haproxy config does that need a new module? | 16:42 |
| yolanda | clarkb, ideally i'd liked a puppet-git, with ::cgit and ::haproxy classes , everything together | 16:43 |
| *** jtriley has quit IRC | 16:43 | |
| yolanda | currently that openstack_project::git installs packages | 16:43 |
| yolanda | adds config files | 16:43 |
| *** achanda has joined #openstack-infra | 16:43 | |
| fungi | clarkb: no, i missed that. a demo huh? | 16:44 |
| yolanda | sets up haproxy | 16:44 |
| yolanda | so it's embedded functionality there, not just config | 16:44 |
| *** mriedem_lunch is now known as mriedem | 16:44 | |
| *** dhritishikhar_ has quit IRC | 16:44 | |
| *** dhritishikhar_ has joined #openstack-infra | 16:45 | |
| yolanda | clarkb, i was tempted to add a class to puppet-cgit, but the cgit name seems to be isolating that module just for backend | 16:45 |
| fungi | yolanda: clarkb: i'm not opposed to adding a similar haproxy load balancing class to our puppet-cgit module | 16:45 |
| yolanda | fungi, that module is yours | 16:45 |
| *** armax has quit IRC | 16:46 | |
| fungi | what module is mine? | 16:46 |
| yolanda | fungi, the naming was throwing me back , as cgit refers to that backend, but if you are ok with that, i'm happy to propose a change | 16:46 |
| yolanda | puppet-cgit | 16:46 |
| yolanda | has your name there | 16:46 |
| fungi | i'm not following. where is my name? | 16:47 |
| *** asselin has quit IRC | 16:47 | |
| yolanda | ah, just saw that in some commit, i may be wrong | 16:47 |
| fungi | i have a few commits in it, but am far from the most prominent contributor to the repo | 16:48 |
| clarkb | ya if cgit wants an haproxy frontend manifest that makes sense | 16:48 |
| *** armax has joined #openstack-infra | 16:48 | |
| yolanda | ok, happy to propose the change then | 16:48 |
| fungi | anyway, i can see an argument for having a cgit::lb class or something | 16:48 |
| fungi | which happens to implement load balancing via haproxy as an opinionated implementation | 16:49 |
| yolanda | fungi, but that won't be part of the manifest, but a class | 16:49 |
| yolanda | you can use or not | 16:49 |
| fungi | right | 16:49 |
| fungi | hence a separate class | 16:49 |
| fungi | entirely optional | 16:50 |
| yolanda | yep, that's the idea | 16:50 |
| *** Guest92304 has quit IRC | 16:50 | |
| *** unicell has joined #openstack-infra | 16:50 | |
| *** unicell1 has joined #openstack-infra | 16:51 | |
| *** unicell has quit IRC | 16:51 | |
| pleia2 | ok, confirmed translations tooling session will be in i18n rather than our track (& noted in etherpad) | 16:51 |
| openstackgerrit | Gary W. Smith proposed openstack-infra/project-config: Add manila-ui to OpenStack https://review.openstack.org/175063 | 16:52 |
| fungi | that's excellent news. implies they're taking the effort seriously | 16:53 |
| *** annegentle has joined #openstack-infra | 16:54 | |
| fungi | which i assumed, but it's great to have it confirmed like that | 16:54 |
| *** marcusvrn has quit IRC | 16:55 | |
| *** marcusvrn_ is now known as marcusvrn | 16:55 | |
| zaro | clarkb: do slaves pick up new jobs after enable/re-enable gearman plugin? | 16:56 |
| fungi | zaro: the effected slaves do not, no | 16:56 |
| clarkb | zaro I havent tried that, only offline/online the slave | 16:56 |
| *** marcusvrn1 has joined #openstack-infra | 16:56 | |
| fungi | oh, of the plugin | 16:57 |
| *** Shrews has quit IRC | 16:57 | |
| fungi | i misread, thought you were asking about toggling the slave status, yeah | 16:57 |
| *** gilliard has quit IRC | 16:57 | |
| fungi | i wonder if briefly disabling and enabling the plugin is particularly disruptive | 16:57 |
| *** gilliard has joined #openstack-infra | 16:58 | |
| fungi | but also worth noting we're currently seeing this behavior across all of our jenkins masters | 16:58 |
| fungi | so it doesn't seem to be a particularly rare situation | 16:58 |
| clarkb | but not all slaves | 16:58 |
| zaro | when online/offline the fucntions are correctly registered? or do they not get registered at all with the slave? | 16:58 |
| fungi | right | 16:58 |
| clarkb | zaro they are registered | 16:58 |
| *** derekh has quit IRC | 16:59 | |
| clarkb | zaro but the slave never sends a grab job which makes me thibk they are sleeping | 16:59 |
| *** sdake_ has quit IRC | 16:59 | |
| *** akshai has quit IRC | 16:59 | |
| fungi | did you confirm that the registration message is arriving at zuul's end of the socket? | 16:59 |
| *** btully has joined #openstack-infra | 16:59 | |
| *** sdake has joined #openstack-infra | 16:59 | |
| clarkb | fungi zuul sends acks | 17:00 |
| *** Shrews has joined #openstack-infra | 17:00 | |
| clarkb | I did not tcpdump from that end though | 17:00 |
| fungi | ahh, perfect. so definitely not a dead socket | 17:00 |
| zaro | might be miscommunication between workers and server? | 17:00 |
| fungi | doesn't sound like miscommunication unless it's at layer 7 | 17:00 |
| *** patrickeast has quit IRC | 17:00 | |
| zaro | i mean gearman server. we only have 1 gearman server correct? | 17:01 |
| *** ashleighfarnham has joined #openstack-infra | 17:01 | |
| fungi | or, well, above layer 4 at any rate (the rest of the osi layers are sort of irrelevant distinctions in my opinion) | 17:01 |
| *** annegentle has quit IRC | 17:01 | |
| fungi | we only have one gearman server, the one running as a subprocess of the zuul scheduler | 17:01 |
| clarkb | zaro correct just the one zuul starts | 17:01 |
| clarkb | I think its a state issue on the plugin side | 17:01 |
| clarkb | I am failry sure we do not clear the sleep state properly when we offline online | 17:02 |
| zaro | odd, why only happens with certain slaves? | 17:02 |
| *** bswartz has quit IRC | 17:02 | |
| clarkb | zaro probably a race in the state transitions | 17:02 |
| clarkb | eg if you sleep, the reconnect before getting a noop | 17:02 |
| *** sdake has quit IRC | 17:02 | |
| clarkb | or if server sends noop but you dont process it quickly enough | 17:03 |
| *** sdake has joined #openstack-infra | 17:03 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:03 | |
| *** signed8bit has joined #openstack-infra | 17:03 | |
| *** annegentle has joined #openstack-infra | 17:03 | |
| zaro | if that's the case wouldn't it occur randomly? not happen to the same slave all the time? | 17:04 |
| *** dkranz has quit IRC | 17:04 | |
| clarkb | it is random | 17:04 |
| clarkb | but once a slave gets in that state it cant get out | 17:04 |
| *** mpavone has quit IRC | 17:04 | |
| fungi | well, there is _one_ way out... sweet, sweet death | 17:04 |
| *** ashleighfarnham has quit IRC | 17:04 | |
| fungi | via deletion | 17:04 |
| clarkb | (this is my theory at least especially after seeing that registerfunctions and setfunctions are different) | 17:05 |
| *** ashleighfarnham has joined #openstack-infra | 17:05 | |
| clarkb | fungi we should make sure they find manny calavera on their way to a better place | 17:05 |
| *** bswartz has joined #openstack-infra | 17:06 | |
| *** unicell1 has quit IRC | 17:06 | |
| *** tonytan4ever has quit IRC | 17:06 | |
| *** unicell has joined #openstack-infra | 17:06 | |
| fungi | heh | 17:06 |
| *** sdake has quit IRC | 17:07 | |
| zaro | a jenkins slave disconnect/delete should kill the worker thread i believe so not sure why workers would get stuck in that state. | 17:07 |
| SpamapS | am I crazy or did mordred mention there's a spec to kill glance v2 already? | 17:07 |
| clarkb | zaro: I did not do that, I only offlined it then onlined it | 17:07 |
| *** ldnunes has joined #openstack-infra | 17:07 | |
| fungi | disconnect/reconnect might be worth trying | 17:07 |
| clarkb | zaro: I am sure that if I delete the slave then add it back again that would fix it | 17:07 |
| fungi | just as a confirmation | 17:08 |
| zaro | clarkb: mind giving that a try? | 17:08 |
| clarkb | sure I can do that with 2328080 | 17:08 |
| fungi | i'm guessing that offline/online doesn't recycle the thread | 17:08 |
| clarkb | actually 2328080 is gone | 17:08 |
| clarkb | and my tcpdump shows it actually did something | 17:08 |
| fungi | oh, did it finally wake back up? | 17:09 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: provide a more targetted location setting https://review.openstack.org/178763 | 17:09 |
| clarkb | fungi: well it was deleted at least, not sure if it ran a job, looking in nodepool logs now | 17:09 |
| greghaynes | SpamapS: Theres a spec to make glance support image-upload again IIRC | 17:09 |
| fungi | the bare-trusty ready node distribution on the whole still looks the same as it did | 17:09 |
| fungi | roughly | 17:09 |
| SpamapS | greghaynes: ah | 17:09 |
| fungi | but yeah, could have finally hit the ready node delete time | 17:09 |
| clarkb | ERROR nodepool.NodePool: SSH Check failed for node id: 2328080 | 17:09 |
| fungi | huh | 17:10 |
| clarkb | so gearman packets we saw there were it going away, it did not run a job | 17:10 |
| fungi | fun times | 17:10 |
| * clarkb finds a different guinea pig node | 17:10 | |
| clarkb | 2353037 | 17:10 |
| *** yamahata has quit IRC | 17:11 | |
| clarkb | I am going to offline online it, then see if it grabs some jobs, then do the disconnect connect | 17:11 |
| *** dhritishikhar_ has quit IRC | 17:12 | |
| clarkb | disconnect, connect fixed it | 17:14 |
| clarkb | offline online did not | 17:14 |
| *** ivar-lazzaro has quit IRC | 17:14 | |
| *** akshai has joined #openstack-infra | 17:15 | |
| clarkb | I think that confirms its something funny with the slave state somewhere | 17:15 |
| *** dustins_ has quit IRC | 17:15 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:15 | |
| *** eglynn has quit IRC | 17:16 | |
| *** ivar-lazzaro has quit IRC | 17:16 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: WiP: Add keystone services/endpoints methods https://review.openstack.org/177621 | 17:16 |
| zaro | clarkb: i concur. | 17:16 |
| *** ivar-lazzaro has joined #openstack-infra | 17:17 | |
| clarkb | OH! | 17:17 |
| clarkb | I think I may have found something interesting in the logs | 17:17 |
| *** dguerri is now known as _dguerri | 17:18 | |
| * clarkb is putting a paste together | 17:18 | |
| *** _dguerri is now known as dguerri | 17:18 | |
| fungi | zaro: also, when you have a moment, i have a jstack dump and connections/queue output (including the corresponding account names for the tasks) from this morning's stream-events hang | 17:19 |
| fungi | some assistance analyzing it for any interesting details would be appreciated | 17:19 |
| *** marcusvrn has quit IRC | 17:19 | |
| *** melwitt has joined #openstack-infra | 17:19 | |
| fungi | as far as i can tell, the thread dump doesn't have any sensitive keys/passwords/tokens in it, so should be safe for me to just stick in a paste | 17:20 |
| *** andrey-mp has joined #openstack-infra | 17:20 | |
| *** sputnik13 has joined #openstack-infra | 17:21 | |
| zaro | fungi: ok, can take a look. | 17:22 |
| zaro | fungi: did you restart gerrit with 2.8.4.19? | 17:22 |
| zaro | ohh i see it's there | 17:22 |
| zaro | i assume no problem with the update? | 17:23 |
| fungi | zaro: well, clarkb's change you reviewed yesterday was the upshot of us debugging the problem | 17:23 |
| *** Apsu has quit IRC | 17:23 | |
| fungi | as you guessed, we had more than one bouncy castle lib when unpacking the .10 war | 17:23 |
| clarkb | fungi: zaro http://paste.openstack.org/show/212418/ | 17:24 |
| clarkb | fungi: zaro that shows the node come up, do a grab job, then nothing. I need to read the code but I think that confirms the node got a directive to go to sleep since a job was not available | 17:24 |
| *** sdake has joined #openstack-infra | 17:24 | |
| andrey-mp | does anybody know can Ubuntu be used in functional tests? e.g. - can I run vm with ubuntu in gating? | 17:24 |
| clarkb | andrey-mp: are you asking if you can run a nested ubuntu VM on our test infrastructure? | 17:25 |
| fungi | andrey-mp: by default we test on ubuntu 14.04 lts | 17:25 |
| *** patrickeast has joined #openstack-infra | 17:25 | |
| andrey-mp | fungi: i mean - can i run ubuntu instead cirros? | 17:25 |
| fungi | ahh, nesting. that would get crazy slow trying to boot ubuntu under qemy inside a cloud instance | 17:25 |
| fungi | er, qemu | 17:25 |
| pleia2 | openstack gets weird when you run it under qemu, too slow | 17:26 |
| *** tonytan4ever has joined #openstack-infra | 17:26 | |
| clarkb | andrey-mp: you can, but it will be terribly slow | 17:26 |
| *** pelix has quit IRC | 17:26 | |
| SpamapS | andrey-mp: It will take many many minutes to boot | 17:26 |
| fungi | there were some heat functional tests i think that booted fedora under qemu and that was pretty painful. no idea if they still run that | 17:27 |
| SpamapS | and your SSH key will take many many more minutes to generate. ;) | 17:27 |
| andrey-mp | fungi: yeah, I understand it :) but is it possible? if i should load this image into cloud - can i download it from download.ubuntu.com in gating? | 17:27 |
| fungi | downloading random things in a job is fragile | 17:27 |
| andrey-mp | ok, sounds good :) | 17:27 |
| fungi | speaking of which... http://logs.openstack.org/44/174844/4/gate/gate-neutron-lbaas-dsvm-api/a87332b//console.html#_2015-04-29_16_57_19_222 | 17:28 |
| fungi | telling tox to clone tempest seems like a bad idea | 17:28 |
| clarkb | zaro: fungi hrm reading the code I think that the slave really did receive a job assignment, it just doesn't appear to have done anything with it | 17:28 |
| *** dkranz has joined #openstack-infra | 17:28 | |
| andrey-mp | maybe it can be downloaded from some inside resource? | 17:28 |
| fungi | clarkb: that might explain some of the changes we see spending way longer than they should in some pipelines. i bet zuul gives up on the assignment and puts it back to queued after a while | 17:29 |
| andrey-mp | i understand about fedora - i will see to heat | 17:29 |
| fungi | andrey-mp: devstack has a list of files it can use, and nodepool reads that and downloads them to a local cache when generating our worker images. it would probably need to be a fairly small ubuntu image and you would need to convince devstack reviewers that there's a benefit to adding it | 17:30 |
| *** dprince has quit IRC | 17:30 | |
| *** wenlock_ has quit IRC | 17:30 | |
| *** dprince has joined #openstack-infra | 17:30 | |
| *** wenlock_ has joined #openstack-infra | 17:31 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Test v1 and v2 https://review.openstack.org/178774 | 17:31 |
| clarkb | fungi: ya I think if the worker doesn't acknowledge back that it is taking the job the server offers it to another worker | 17:31 |
| fungi | andrey-mp: i also recommend starting an 8gb nova instance with no hardware acceleration for nested virtualization and seeing how well booting that image under qemu does or doesn't work | 17:31 |
| andrey-mp | fungi: thank you! we will discuss it inside our team first | 17:31 |
| *** yamahata has joined #openstack-infra | 17:32 | |
| fungi | andrey-mp: that should give you some indication of whether it will be usable at all for what you're considering | 17:32 |
| krtaylor | timrc, ping, re: your question and neg on https://review.openstack.org/#/c/135170/ was your question answered? we'd really like to move that along | 17:33 |
| clarkb | zaro: so, I definitely think it is something worker side, but I am not convinced its related to sleep anymore | 17:33 |
| *** emagana has quit IRC | 17:33 | |
| timrc | krtaylor, Ah sorry. I was super distracted last week. Let me go look. | 17:34 |
| clarkb | the logs are revealing much more info, we may actually have this solved shortly :) | 17:34 |
| *** Apsu has joined #openstack-infra | 17:34 | |
| zaro | clarkb: i believe if the worker receives the job assignment it will execute the job regardless of whether the sever acks or not | 17:34 |
| krtaylor | timrc, thanks! | 17:35 |
| zaro | clarkb: but you are saying the job doesn't get executed? | 17:35 |
| clarkb | zaro: ya there is nothing in the logs after that point about that executor | 17:35 |
| *** AJaeger__ has joined #openstack-infra | 17:36 | |
| clarkb | zaro: if you compare it to the job that did end up running after I did a disconnect/connect the job that did run goes on to log its executing a function and what the params are and all that | 17:36 |
| clarkb | zaro: the job assign uniq from before the slave livelocked did none of that | 17:36 |
| *** mpaolino has quit IRC | 17:38 | |
| clarkb | zaro: we never hit https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/MyGearmanWorkerImpl.java#n372 | 17:39 |
| *** andrey-mp has quit IRC | 17:39 | |
| clarkb | zaro: but don't appear to otherwise register any failure | 17:39 |
| clarkb | zaro: which makes me think the work() loop is blocked on something | 17:39 |
| clarkb | there is also a 2 hour gap between it getting the job assignment and it going into the periodic register functions behavior | 17:42 |
| *** shardy_ has joined #openstack-infra | 17:44 | |
| fungi | clarkb: interesting example, there's a neutron change at the top of the gate right now. all its jobs but one have finished, and that one's only started a few minutes ago | 17:45 |
| *** shardy has quit IRC | 17:46 | |
| zaro | clarkb: does it do anything in work()? | 17:46 |
| fungi | oddly, the nova change immediately below it is in the same situation, and even over the same job | 17:46 |
| *** jamesmcarthur has quit IRC | 17:47 | |
| fungi | i wonder if there's some separate problem with the neutron tempest full job | 17:47 |
| fungi | maybe tanking the worker under certain situations and resulting in restarts | 17:47 |
| fungi | hrm, no further down the pipeline there's a swift change in that situation over the non-neutron tempest full job | 17:48 |
| clarkb | zaro: its doing enough to send the grab job request | 17:49 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Expand fingerprint for git fetch error https://review.openstack.org/178338 | 17:49 |
| clarkb | zaro: beause we get the job assignment back, I think I see where the potential race is in work() | 17:49 |
| clarkb | zaro: will have a bunch of links with some execution order that may result in us hanging perpetually | 17:50 |
| *** shardy_ has quit IRC | 17:50 | |
| *** shardy has joined #openstack-infra | 17:50 | |
| *** notnownikki has quit IRC | 17:51 | |
| *** dkranz has quit IRC | 17:51 | |
| clarkb | first we send the grab job https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/MyGearmanWorkerImpl.java#n310 then we check if the function list is empty https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/MyGearmanWorkerImpl.java#n331 at this point if our handleSessionEvent is called and we hit | 17:52 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/MyGearmanWorkerImpl.java#n412 we will put a job on the funclist and drain the input buffer then we hit https://git.openstack.org/cgit/openstack-infra/gearman-plugin/tree/src/main/java/hudson/plugins/gearman/MyGearmanWorkerImpl.java#n340 and block until IO is received that is never sent to us | 17:52 |
| fungi | it's possible we're separately having some connectivity issues to workers. i'm seeing if i can stumble across some additional examples | 17:52 |
| clarkb | fungi: connectivity issues in general may be tickling these bugs since it appears to be a race, so possible its related | 17:53 |
| *** jamesmcarthur has joined #openstack-infra | 17:53 | |
| clarkb | fungi: but not the direct cause | 17:53 |
| fungi | right, that's what i'm suspecting. mostly searching for the answer to why we don't see this more often | 17:53 |
| clarkb | zaro: I think we can avoid this case by having stronger locking around those critical sections or we can go lock free witha timeout on the select, check the funclist, reselect if still empty | 17:54 |
| clarkb | I think I prefer going lock free | 17:54 |
| fungi | also, the neutron plugin-specific job i linked to earlier is one place where our git retries aren't going to help us | 17:54 |
| clarkb | fungi: because of the direct clone in tox.ini ya? | 17:55 |
| fungi | yep | 17:55 |
| *** Sukhdev has quit IRC | 17:55 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:56 | |
| *** hdd has quit IRC | 17:57 | |
| *** e0ne has quit IRC | 17:57 | |
| fungi | and not just trivially. it's got pip git cloning neutron and tempest both | 17:57 |
| *** hdd has joined #openstack-infra | 17:58 | |
| AJaeger__ | fungi, clarkb: Is this really still debugging our docs problem? Wow - and THANKS! | 17:58 |
| clarkb | AJaeger__: I think I have found the underlying issue :) at the very least it is a race that could cause this problem. Don't have enough logging to confirm it was the exact cuase here | 17:58 |
| AJaeger__ | clarkb: Good, those are tricky. | 17:59 |
| *** hyakuhei has joined #openstack-infra | 17:59 | |
| *** harlowja_away is now known as harlowja | 18:00 | |
| *** hyakuhei has left #openstack-infra | 18:00 | |
| fungi | in the jenkins plugin which communicates with zuul to handle job assignment to workers | 18:00 |
| *** woodster_ has quit IRC | 18:00 | |
| *** emagana has joined #openstack-infra | 18:01 | |
| *** marcusvrn has joined #openstack-infra | 18:01 | |
| *** dhellman_ has joined #openstack-infra | 18:03 | |
| harlowja | SpamapS whats up, something with glance taskflow executor stuff? i didn't make that part of glance, so maybe the glance folks know? | 18:03 |
| harlowja | hopefully u figured it out | 18:04 |
| SpamapS | harlowja: I finally figured it out, but it was _maddening_ to find the entry point for the executor. | 18:04 |
| *** marcusvrn1 has quit IRC | 18:04 | |
| harlowja | sounds like a glance issue? | 18:04 |
| SpamapS | harlowja: also I didnt want to believe it was just glance-api | 18:04 |
| SpamapS | like, all that trouble, to have the app servers do all the work themselves? | 18:04 |
| *** dkranz has joined #openstack-infra | 18:04 | |
| harlowja | gotta start somewhere afaik | 18:05 |
| *** davidlenwell has quit IRC | 18:05 | |
| asilenkov | hello! I can't signup to review.openstack.org. who could help? | 18:05 |
| *** dannywilson has quit IRC | 18:05 | |
| *** davidlenwell has joined #openstack-infra | 18:05 | |
| *** dhellmann has quit IRC | 18:05 | |
| *** _nadya_ has quit IRC | 18:05 | |
| harlowja | SpamapS i assume u are talking about how the workflow runs on glance-api ? | 18:05 |
| harlowja | *runs on the glance-api 'server' | 18:05 |
| fungi | asilenkov: did you follow the steps at http://docs.openstack.org/infra/manual/developers.html#account-setup in the sequence documented there? | 18:05 |
| *** ubuntu2 has quit IRC | 18:06 | |
| AJaeger__ | clarkb, fungi: IS this only hitting the post queue or is this just the one where we noticed it? | 18:06 |
| fungi | AJaeger__: it's more obvious to changes in the post pipeline because it only gets the dregs after release, gate, check et cetera have their pick of workers | 18:06 |
| harlowja | SpamapS afaik the idea was that when the glance folks wanted they can move it to run elsewhere, but if u have an inflexible framework that can't do that in the first place, moving to one that can start to make that possible (while maintaining the status quo) is easier to integrate than moving to one that can't do that (therefore making a new glance release that much more work for operators...) | 18:07 |
| fungi | so teh starvation becomes more obvious there | 18:07 |
| asilenkov | fungi: yes | 18:07 |
| *** rm_work|away is now known as rm_work | 18:07 | |
| harlowja | SpamapS but i'm not sure of the current status of all that stuff, the glance folks know best | 18:07 |
| asilenkov | fatal: ICLA contributor agreement requires current contact information. | 18:07 |
| asilenkov | Please review your contact information: | 18:07 |
| SpamapS | harlowja: So, I don't blame taskflow as much as I blame people landing code with 0 documentation. | 18:07 |
| asilenkov | I can't store it - server error | 18:07 |
| AJaeger__ | ah, understood. thanks - will continue listening in. If there's anything I can help, please tell me. | 18:07 |
| harlowja | SpamapS undersetandable | 18:07 |
| fungi | asilenkov: see the step there in the documentation about signing up as a foundation individual member | 18:07 |
| fungi | asilenkov: did you do that step? | 18:07 |
| harlowja | SpamapS just saying what i know (which isn't all that much with regards to glance + taskflow integration) | 18:07 |
| harlowja | tis all :-P | 18:08 |
| *** ubuntu2 has joined #openstack-infra | 18:08 | |
| SpamapS | harlowja: and it's something to look out for too. | 18:08 |
| SpamapS | harlowja: the entry points are not as obvious because of the nice disconnected nature of flows. | 18:08 |
| fungi | asilenkov: second paragraph in that section where it says "join The OpenStack Foundation" | 18:08 |
| harlowja | SpamapS understood | 18:08 |
| clarkb | zaro: I am working on a non locking patch | 18:08 |
| *** imcsk8 has quit IRC | 18:08 | |
| SpamapS | harlowja: the place where executors are entered should be really clear about it. | 18:09 |
| SpamapS | Also, the words are all double-meaning in glance | 18:09 |
| harlowja | thats a glance thing though :-p | 18:09 |
| SpamapS | import and python import, task and task flow... | 18:09 |
| * harlowja didn't write any of that glance + taskflow code (others afaik did) lol | 18:09 | |
| *** otter768 has joined #openstack-infra | 18:09 | |
| *** claudiub has quit IRC | 18:10 | |
| zaro | clarkb: i was still trying to grok but i do agree that non-blocking would be better way to go | 18:10 |
| SpamapS | harlowja: my point is, the way they're using your library is confusing and that _does_ end up with frustration, even if it's not your fault. | 18:11 |
| SpamapS | harlowja: It's a PR problem. :-/ | 18:12 |
| harlowja | SpamapS ha, sounds like u should go talk to the glance folks :-P | 18:12 |
| SpamapS | harlowja: I've been trying to all morning. :) | 18:12 |
| *** rossella_s has quit IRC | 18:12 | |
| SpamapS | harlowja: appears none of the task stuff is tested in the gate | 18:12 |
| harlowja | that su | 18:13 |
| harlowja | *sux | 18:13 |
| SpamapS | harlowja: which probably explains why it doesn't work at all for me :) | 18:13 |
| harlowja | hopefully works for someone :-/ | 18:13 |
| clarkb | zaro: this is more complicated than I expected | 18:13 |
| fungi | harlowja: unlikely. if it's not tested, it's broken ;) | 18:13 |
| fungi | might have worked for someone at one time | 18:14 |
| harlowja | fungi sure, but hopefully someone somewhere has ran it, lol | 18:14 |
| harlowja | but i know what u mean | 18:14 |
| clarkb | zaro: since I think there are cases where we will want to not block there when the funclist is empty | 18:14 |
| *** otter768 has quit IRC | 18:14 | |
| *** dustins has joined #openstack-infra | 18:15 | |
| zaro | anybody seeing network issues from hpcloud? | 18:15 |
| *** harlowja has quit IRC | 18:16 | |
| *** harlowja has joined #openstack-infra | 18:16 | |
| zaro | getting an error resovling git.openstack.org from my hpcloud V< | 18:16 |
| clarkb | zaro: hrm maybe its not complicated since the handleSessionEvent function will send presleep directly when it gets a NO_JOB and will send a grab job when it receives a NOOP | 18:16 |
| asilenkov | fungi: well ty it works now, sorry ) | 18:16 |
| fungi | asilenkov: no worries, glad you got it figured out | 18:17 |
| *** harlowja has quit IRC | 18:17 | |
| asilenkov | fungi: we should update our internal mirantis documentation about this stuff, I'll fill a bug. | 18:17 |
| fungi | asilenkov: sure, you can also feel free to link to our infra manual--that's official documentation | 18:18 |
| fungi | we do try very hard to keep it current and correct | 18:18 |
| *** harlowja has joined #openstack-infra | 18:20 | |
| *** sabeen3 has joined #openstack-infra | 18:20 | |
| fungi | clarkb: here we go... see end of https://jenkins02.openstack.org/job/check-tempest-dsvm-nova-v21-neutron-full/121/console | 18:22 |
| fungi | looks like we're getting more than a few of those | 18:22 |
| *** sabeen has quit IRC | 18:22 | |
| clarkb | and during tempest testing too, so tempest is likely at fault | 18:23 |
| fungi | first one i've caught in the act so maybe it's just the nova v21 job | 18:23 |
| fungi | though evidence suggests it's other tempest jobs too | 18:23 |
| *** cdent has quit IRC | 18:25 | |
| *** maishsk has joined #openstack-infra | 18:25 | |
| *** rossella_s has joined #openstack-infra | 18:25 | |
| *** bswartz has quit IRC | 18:26 | |
| * zaro afk, bb in a hrs. | 18:27 | |
| *** bswartz has joined #openstack-infra | 18:28 | |
| *** markvoelker has quit IRC | 18:28 | |
| *** harlowja has quit IRC | 18:29 | |
| *** sdake has quit IRC | 18:29 | |
| *** harlowja has joined #openstack-infra | 18:29 | |
| *** dguerri is now known as _dguerri | 18:30 | |
| clarkb | hrm untangling this is difficult, I am very afraid of adding new (dead|live)locks | 18:30 |
| *** e0ne has joined #openstack-infra | 18:31 | |
| *** e0ne is now known as e0ne_ | 18:31 | |
| *** abregman has joined #openstack-infra | 18:32 | |
| *** e0ne_ is now known as e0ne | 18:32 | |
| *** e0ne is now known as e0ne_ | 18:32 | |
| *** e0ne_ is now known as e0ne | 18:32 | |
| *** spredzy is now known as spredzy|afk | 18:32 | |
| *** e0ne has quit IRC | 18:32 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 18:33 |
| *** _dguerri is now known as dguerri | 18:34 | |
| *** abregman has quit IRC | 18:34 | |
| *** arxcruz has joined #openstack-infra | 18:35 | |
| ctlaugh_ | krtaylor: In the meeting earlier today, you mentioned sending out some e-mails that no one had commented on. I don't believe I am subscribed to that mailing list -- could you point me to which one that is? | 18:36 |
| *** annegentle has quit IRC | 18:36 | |
| *** emagana has quit IRC | 18:36 | |
| *** sdake has joined #openstack-infra | 18:37 | |
| *** dhp_ has quit IRC | 18:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow https://review.openstack.org/175152 | 18:39 |
| *** marun has quit IRC | 18:39 | |
| clarkb | fungi: did you delete the nodes that had been ready and idle? I don't need them but can tell nodepool to delete them if you haven't | 18:39 |
| *** arxcruz has quit IRC | 18:39 | |
| fungi | clarkb: i hadn't. didn't want to yank any out from under you, but will do it now | 18:40 |
| clarkb | ya I think I grok the race(s) here, basically we can "driveIO" from multiple places so can't assume that blocking waiting for new IO before driving further is going to return | 18:41 |
| fungi | all cleaned up now, though i wouldn't be surprised to see us accumulate more (perhaps fairly rapidly if the network problems keep up) | 18:41 |
| clarkb | it also explains why we would register out of band but not actually run a job, because the loop that handles the work isn't reset in that case | 18:41 |
| clarkb | fungi: ya we should keep an eye on it | 18:41 |
| *** dguerri has quit IRC | 18:42 | |
| lifeless | fungi: https://review.openstack.org/#/c/177504/ - could you re-add your +2 ? | 18:42 |
| krtaylor | ctlaugh_, that was the openstack-dev mail list | 18:43 |
| *** emagana has joined #openstack-infra | 18:43 | |
| clarkb | its kind of a mess because the java gearman lib doesn't really give us a good async model. Instead we are sharing a Selector to select on which is a bit :( imo | 18:46 |
| *** maishsk has quit IRC | 18:47 | |
| clarkb | if we weren't doing that it would be as simple as matching up Selector.wakeup() and Selector.select() calls | 18:47 |
| *** e0ne has joined #openstack-infra | 18:48 | |
| *** rmcall has quit IRC | 18:48 | |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add functional test job for python-glanceclient https://review.openstack.org/178285 | 18:49 |
| *** rmcall has joined #openstack-infra | 18:49 | |
| *** annegentle has joined #openstack-infra | 18:49 | |
| *** annegentle has quit IRC | 18:52 | |
| *** annegentle has joined #openstack-infra | 18:53 | |
| clarkb | hrm handleSessionEvent is synchronous with driveSessionIO | 18:58 |
| *** erlon has joined #openstack-infra | 18:58 | |
| *** tsg_ has joined #openstack-infra | 18:59 | |
| clarkb | except maybe not because we have two different handleSessionEvents and one does not appear to override the other | 19:00 |
| *** emagana has quit IRC | 19:01 | |
| *** mrmartin has joined #openstack-infra | 19:02 | |
| *** ivar-laz_ has joined #openstack-infra | 19:02 | |
| *** ivar-laz_ has quit IRC | 19:02 | |
| *** mrunge has quit IRC | 19:03 | |
| *** dustins has quit IRC | 19:03 | |
| *** ivar-laz_ has joined #openstack-infra | 19:03 | |
| fungi | the post pipeline backlog dropped like a stone after i cleaned up those stale workers gumming up the ready pool | 19:03 |
| *** emagana has joined #openstack-infra | 19:04 | |
| fungi | also my fishing expedition has turned up no new restarted jobs, so i think whatever the cause was has subsided | 19:04 |
| *** ivar-lazzaro has quit IRC | 19:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update secgroup API for new get/list/search API https://review.openstack.org/178807 | 19:04 |
| *** dhellmann has joined #openstack-infra | 19:06 | |
| *** jtriley has joined #openstack-infra | 19:06 | |
| *** jaypipes has quit IRC | 19:06 | |
| openstackgerrit | Merged openstack/requirements: Update keystoneclient requirement https://review.openstack.org/168187 | 19:07 |
| *** rmcall has quit IRC | 19:07 | |
| *** rmcall has joined #openstack-infra | 19:07 | |
| clarkb | the more I read the less I am convinced this is the issue, if select blocked indefinitely it would certinly explain this problem I just can't figure out why it would block like that | 19:08 |
| *** cody-somerville has quit IRC | 19:09 | |
| *** dhellman_ has quit IRC | 19:10 | |
| clarkb | aha! | 19:12 |
| *** teran has joined #openstack-infra | 19:12 | |
| clarkb | I think its the function registration | 19:14 |
| clarkb | it happens in the log and does a select wakeup | 19:14 |
| clarkb | which means if there is >1 select happening only that one will wake | 19:14 |
| *** marcusvrn has quit IRC | 19:15 | |
| clarkb | ya ioAvailable is shared with the session object which could select on it independently | 19:16 |
| *** ivar-laz_ has quit IRC | 19:16 | |
| teran | Hi folks, I have a user Denis Meltsaykin with two accounts in review.openstack.org, could you please help? | 19:17 |
| *** ivar-lazzaro has joined #openstack-infra | 19:17 | |
| clarkb | yup the gearman session runs selectNow() on the selector which clears and wakeups | 19:17 |
| *** salv-orl_ has joined #openstack-infra | 19:18 | |
| fungi | teran: sure, just a sec and i'll see what i can do | 19:18 |
| *** marcusvrn has joined #openstack-infra | 19:18 | |
| *** unicell has quit IRC | 19:18 | |
| *** salv-orlando has quit IRC | 19:22 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 19:22 |
| fungi | teran: there shouldn't be any duplicate as of 8 days ago. i think i remember cleaning that one up myself when someone mentioned it in here | 19:22 |
| fungi | teran: is it still causing an issue? | 19:22 |
| *** jaypipes has joined #openstack-infra | 19:23 | |
| teran | fungi: I see one of accounts just without email, but there are two still. I'll poke him check but now I asking since there was no reply from previous mentioning him(or I simply missed it): | 19:24 |
| *** AJaeger__ has quit IRC | 19:25 | |
| teran | fungi: anyway thanks :) | 19:25 |
| *** dustins has joined #openstack-infra | 19:25 | |
| fungi | teran: i can also blank out his full name in the database, but that will cause any comments left by that account to appear as "anonymous" | 19:26 |
| fungi | so i usually don't do that | 19:26 |
| teran | fungi: no necessary for now, I think, let's give Denis a chance to try authentication :) | 19:26 |
| *** cody-somerville has joined #openstack-infra | 19:28 | |
| *** akshai has quit IRC | 19:28 | |
| *** miqui has quit IRC | 19:29 | |
| *** rlandy has joined #openstack-infra | 19:29 | |
| *** woodster_ has joined #openstack-infra | 19:31 | |
| *** dizquierdo has joined #openstack-infra | 19:35 | |
| *** unicell has joined #openstack-infra | 19:43 | |
| lifeless | fungi: thanks | 19:44 |
| *** david-lyle has quit IRC | 19:44 | |
| *** spzala has joined #openstack-infra | 19:44 | |
| lifeless | so I need some brainpan time from someone familiar with the zuul application stuff | 19:44 |
| lifeless | for https://review.openstack.org/#/c/177629/ | 19:44 |
| *** sarob has quit IRC | 19:44 | |
| *** david-lyle has joined #openstack-infra | 19:44 | |
| *** sarob has joined #openstack-infra | 19:45 | |
| openstackgerrit | lifeless proposed openstack-infra/project-config: Fixup the PBR_PIP_VERSION more. https://review.openstack.org/177513 | 19:46 |
| openstackgerrit | lifeless proposed openstack-infra/project-config: Don't setup devstack for pbr integration tests. https://review.openstack.org/177514 | 19:46 |
| fungi | lifeless: the goal is to update the git clones in /opt without copying them to another directory tree first? | 19:46 |
| lifeless | fungi: yes | 19:47 |
| lifeless | fungi: well there are two goals | 19:47 |
| lifeless | fungi: they're tangled together | 19:47 |
| lifeless | fungi: one is to stop running devstack | 19:47 |
| lifeless | fungi: since its entirely wasted in this job | 19:47 |
| lifeless | fungi: that seems to require switching to using /opt/git. | 19:47 |
| fungi | lifeless: /opt/stack and /opt/git are on the same filesystem... what was the i/o involved in copying a directory tree there? | 19:48 |
| lifeless | fungi: the second goal is avoiding the copy of a couple GB of data around to no point | 19:48 |
| lifeless | fungi: I'm fairly sure its not hardlinking them | 19:48 |
| bknudson | is it possible for us to do https://wheel.readthedocs.org/en/latest/#defining-conditional-dependencies in our libs? e.g., for argparse since it's only needed for 2.6 | 19:48 |
| lifeless | bknudson: no, for two reasons | 19:48 |
| fungi | lifeless: yeah, if it clones from file:// rather than just the pathname then it won't hardlink | 19:49 |
| *** rmcall has quit IRC | 19:49 | |
| lifeless | bknudson: firsly, its needed in 2.7+ when things Require('argparse'), because the one in the stdlib isn't registered as a package | 19:49 |
| openstackgerrit | Andrew Smith proposed openstack-infra/jenkins-job-builder: Add support for git sparse checkout https://review.openstack.org/178822 | 19:49 |
| *** HeOS has joined #openstack-infra | 19:49 | |
| lifeless | bknudson: secondly, we don't have proper support for *either* extras, or [better] environment markers in pbr yet. We're working up the stack to do that. | 19:49 |
| bknudson | lifeless: I think that's exactly what we're running into. | 19:49 |
| lifeless | bknudson: but excluding argparse would be wrong anyway | 19:50 |
| lifeless | bknudson: (because of 1)) | 19:50 |
| lifeless | fungi: however I don't know how to best sensibly reuse the 'apply the zuul refs to the trees' in situ on /opt/git | 19:50 |
| lifeless | bknudson: what are you running into ? | 19:51 |
| *** dannywilson has joined #openstack-infra | 19:51 | |
| bknudson | lifeless: so we're building rpms... the rpms say they require python-argparse, and python-2.7 package says it provides python-argparse | 19:51 |
| bknudson | but then pkg_resources says there's no argparse | 19:52 |
| *** dannywilson has quit IRC | 19:52 | |
| lifeless | bknudson: sounds like you need a real python-argparse package with just the egg metadata in it | 19:52 |
| clarkb | lifeless: you want to use zuul-cloner, look in openstack-infra/system-config/tools/apply-test.sh iirc | 19:52 |
| *** dannywilson has joined #openstack-infra | 19:52 | |
| *** dprince has quit IRC | 19:52 | |
| clarkb | also zuul-cloner should hardlink iirc | 19:52 |
| fungi | lifeless: i'm looking now to see whether zuul-cloner will let you update and operate on an existing git repository in your workspace without helpfully blowing away and re-cloning it for you | 19:53 |
| bknudson | lifeless: yes, and there is a real python-argparse rpm, but yum isn't installing it automatically probably because python says it provides it. | 19:53 |
| clarkb | fungi: I don't think it will, but it will hardlink | 19:53 |
| clarkb | fungi: so effectively the same thing | 19:53 |
| lifeless | clarkb: jenkins/scripts/gerrit-git-prep.sh ? | 19:53 |
| fungi | hardlinks might be "good enough" in this case | 19:53 |
| lifeless | it has git clone file:/// | 19:53 |
| clarkb | lifeless: no, openstack-infra/system-config/tools/apply-test.sh uses zuul-cloner, this has nothing to do with g-g-p | 19:53 |
| fungi | lifeless: talking about changing up the job to call the zuul-cloner utility from openstack-infra/zuul | 19:53 |
| *** ivar-lazzaro has quit IRC | 19:54 | |
| fungi | lifeless: for example infra's puppet-apply integration tests use it | 19:54 |
| *** jamesmcarthur has quit IRC | 19:54 | |
| fungi | it's basically a rewrite of devstack-gate's git cloning logic in python | 19:54 |
| *** ivar-lazzaro has joined #openstack-infra | 19:54 | |
| lifeless | bknudson: so, yum's python 2.7 advertises something falsely. How would changing the install-requires on openstack projects fix the issue? | 19:55 |
| lifeless | bknudson: whatever is calling pkg_resources.requires('argparse') would still do so, no? | 19:55 |
| bknudson | lifeless: yes, that python 2.7 advertisement does seem wrong. | 19:55 |
| fungi | this goes back to the "we don't test specific versions of python, we test distributions of python" discussion | 19:57 |
| *** ajmiller has joined #openstack-infra | 19:58 | |
| clarkb | fungi: the more I stare at this code the less convinced I become of my fix and the bug | 19:59 |
| clarkb | I am going to take a break and find lunch | 19:59 |
| *** ajmiller_ has joined #openstack-infra | 19:59 | |
| *** doude has quit IRC | 20:01 | |
| fungi | k | 20:01 |
| *** doude has joined #openstack-infra | 20:01 | |
| clarkb | it does definitely appear to be an interaction between sending a grab job requests and having setFunctions() be called at that same time due to a configuration update | 20:01 |
| clarkb | both affect the selector, one by running selectNow() and the other by doing a wakeup() then later we do a vanilla blocking select() that seems to select forever | 20:02 |
| lifeless | fungi: so the second angle- stopping bringing up devstack | 20:03 |
| lifeless | fungi: would be a win in and of itself | 20:03 |
| lifeless | fungi: thats about 10m of runtime in the job that we entirely fail to care about | 20:03 |
| clarkb | its actually almost twice that so even more win | 20:03 |
| clarkb | lifeless: you can either just run d-g with an overridden gate_hook, or have zuul-cloner do the cloning for you | 20:04 |
| fungi | yeah, the comment saying 10 minutes has... uh... bitrotted a little | 20:04 |
| fungi | clarkb: devstack is started before the gate_hook though, right? | 20:04 |
| lifeless | ok so - I have a patch to projcet-config and one to pbr both aiming at that | 20:05 |
| * fungi refreshes recollections | 20:05 | |
| lifeless | but I'm out of my depth :0 | 20:05 |
| fungi | ahh, no clarkb's right | 20:05 |
| fungi | https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n402 | 20:05 |
| lifeless | so - if either of you could take mercy on me and be very very specific, or perhaps push up something, that would rock. There's no super urgency on it, though cutting fat out would be very nice | 20:06 |
| fungi | devstack-vm-gate.sh won't be run if you supply a different gate_hook | 20:06 |
| *** kgiusti has quit IRC | 20:06 | |
| fungi | lifeless: yeah, just a sec. i'll see what can be done there | 20:06 |
| lifeless | so | 20:07 |
| clarkb | which we actually already seem to do | 20:07 |
| lifeless | there's a 5m gap here: http://logs.openstack.org/05/177505/9/check/check-pbr-installation-dsvm/69c4aae/console.html#_2015-04-29_07_24_59_910 | 20:07 |
| clarkb | see https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/requirements.yaml | 20:07 |
| *** eharney has quit IRC | 20:07 | |
| lifeless | and then a 10m gap here | 20:07 |
| lifeless | http://logs.openstack.org/05/177505/9/check/check-pbr-installation-dsvm/69c4aae/console.html#_2015-04-29_07_30_10_042 | 20:07 |
| clarkb | so devstack is not run | 20:07 |
| *** megm has joined #openstack-infra | 20:08 | |
| clarkb | lifeless: the first gap is the clone and fetch things gap | 20:08 |
| *** esker has quit IRC | 20:08 | |
| lifeless | I don't have whats going on in those internalised in any way | 20:08 |
| lifeless | clarkb: yeah, I thought so :) | 20:08 |
| *** Rockyg has joined #openstack-infra | 20:08 | |
| fungi | lifeless: the first gap is http://logs.openstack.org/05/177505/9/check/check-pbr-installation-dsvm/69c4aae/logs/devstack-gate-setup-workspace-new.txt.gz | 20:09 |
| clarkb | lifeless: the setup_workspace log file is the log for that ya ^ | 20:09 |
| clarkb | and it appears it does run devstack? thats weird | 20:10 |
| fungi | the second gap seems to be installing things http://logs.openstack.org/05/177505/9/check/check-pbr-installation-dsvm/69c4aae/logs/devstacklog.txt.gz | 20:10 |
| *** ivar-lazzaro has quit IRC | 20:10 | |
| *** otter768 has joined #openstack-infra | 20:10 | |
| lifeless | so about 30s to update just nova | 20:11 |
| *** ivar-lazzaro has joined #openstack-infra | 20:11 | |
| fungi | oh, yeah, and running devstack services | 20:11 |
| clarkb | OH! this https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/requirements.yaml#n61 runs devstack | 20:11 |
| clarkb | I think you can just delete that line | 20:11 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Set root disk == node disk for puppet jobs https://review.openstack.org/172210 | 20:11 |
| clarkb | and only run the integration script | 20:11 |
| *** emagana has quit IRC | 20:11 | |
| lifeless | https://review.openstack.org/177514 ? | 20:11 |
| lifeless | clarkb: ^ | 20:11 |
| clarkb | yup +2 | 20:11 |
| lifeless | clarkb: it has a dependency too | 20:11 |
| fungi | i have to admit i don't know the difference between #egg=pipexport and #egg=pip | 20:13 |
| fungi | oh! #egg=pipexport was a typo | 20:13 |
| clarkb | ya, it was appended | 20:13 |
| fungi | from one of the many other export lines | 20:13 |
| fungi | got it | 20:13 |
| clarkb | ok really lunch time now, I think that solves a large chunk of the time waste here, using zuul-cloner is going to be a bit more involved | 20:13 |
| *** emagana has joined #openstack-infra | 20:14 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/devstack-gate: Follow up with some comments to clarify OVS commands https://review.openstack.org/173621 | 20:14 |
| *** otter768 has quit IRC | 20:15 | |
| *** jtriley has quit IRC | 20:15 | |
| *** annegentle has quit IRC | 20:15 | |
| megm | After lunch, I'm having trouble getting set up with gerrit if someone can help. | 20:16 |
| fungi | megm: did you follow the steps at http://docs.openstack.org/infra/manual/developers.html#account-setup in the sequence documented there? | 20:17 |
| megm | Yes -- Anne has been helping me but I'm getting errors she hasn't seen after I run ssh-keygen -f ~/.ssh/known_hosts -R [review.openstack.org]:29418 | 20:18 |
| *** dguerri has joined #openstack-infra | 20:18 | |
| megm | # Host [review.openstack.org]:29418 found: line 1 type RSA | 20:19 |
| megm | line 3 invalid key: ssh-rsa... | 20:19 |
| megm | /home/megm/.ssh/known_hosts is not a valid known_hosts file. | 20:19 |
| megm | Not replacing existing known_hosts file because of errors | 20:19 |
| *** jtriley has joined #openstack-infra | 20:19 | |
| *** tonytan4ever has quit IRC | 20:21 | |
| fungi | megm: that's odd. sounds like your .ssh/known_hosts file is corrupt somehow. can you try renaming it? mv .ssh/known_hosts .ssh/known_hosts.old | 20:22 |
| openstackgerrit | Merged openstack-infra/project-config: Fixup the PBR_PIP_VERSION more. https://review.openstack.org/177513 | 20:22 |
| fungi | megm: then see if trying again prompts you properly to accept the host key for review.openstack.org | 20:22 |
| megm | Okay | 20:22 |
| openstackgerrit | Merged openstack-infra/project-config: Don't setup devstack for pbr integration tests. https://review.openstack.org/177514 | 20:22 |
| fungi | megm: oh! wait | 20:22 |
| megm | Yes? | 20:23 |
| *** marcusvrn has quit IRC | 20:23 | |
| fungi | let me test what you have there just to be sure | 20:23 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add Ironic machine power state pass-through https://review.openstack.org/172284 | 20:23 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Change ironic maintenance method to align with power method https://review.openstack.org/178832 | 20:23 |
| *** annegentle has joined #openstack-infra | 20:23 | |
| *** _nadya_ has joined #openstack-infra | 20:23 | |
| megm | hi, annegentle... | 20:24 |
| annegentle | megm: hi1 | 20:24 |
| annegentle | er that should be ! | 20:24 |
| megm | I'm pretty good at interpreting typos ;-) | 20:24 |
| annegentle | megm: heh | 20:24 |
| fungi | megm: yeah, looks like that should delete the host key for review.openstack.org if it finds it in the known_hosts file, but for some reason it's having a hard time reading your known_hosts file | 20:24 |
| annegentle | fungi: thanks for the help, I wasn't sure how to troubleshoot further without risking the system | 20:25 |
| fungi | megm: i've also confirmed that (at least for me) the error message is different if the known_hosts file doesn't exist at all | 20:25 |
| annegentle | megm: what does ls -la ~/.ssh/ look like | 20:25 |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 20:25 |
| *** e0ne has quit IRC | 20:26 | |
| *** marun has joined #openstack-infra | 20:26 | |
| fungi | megm: so, yeah, chances are just moving that known_hosts file out of the way (since it seems to be corrupt) will let you proceed to testing ssh | 20:27 |
| megm | Okay, let me try | 20:28 |
| megm | ssh-keygen -f ~/.ssh/known_hosts -R [review.openstack.org]:29418 -- do I need to manually recreate the known_hosts file? | 20:30 |
| fungi | megm: no, you should just be able to skip that step now | 20:30 |
| *** jamesmcarthur has joined #openstack-infra | 20:31 | |
| fungi | megm: that command is for removing an existing entry for review.openstack.org from your known_hosts file if it exists | 20:31 |
| megm | here is ls -lai of .ssh: 13109276 drwx------ 2 megm megm 4096 Apr 29 13:28 . | 20:31 |
| megm | 11403265 drwxr-xr-x 50 megm megm 4096 Apr 29 10:42 .. | 20:31 |
| megm | 13109261 -rw------- 1 megm megm 1766 Jan 20 2014 id_rsa | 20:31 |
| megm | 13109264 -rw-r--r-- 1 megm megm 404 Jan 20 2014 id_rsa.pub | 20:31 |
| megm | 13109265 -rw-r--r-- 1 megm megm 1288 Mar 3 18:54 know_hosts.old | 20:31 |
| megm | 13109280 -rw-rw-r-- 1 megm megm 2200 Jan 20 2014 meg.notes | 20:31 |
| megm | 13109285 -rw-rw-r-- 1 megm megm 10240 Feb 7 2014 ssh.tar | 20:31 |
| fungi | megm: but the next time you use ssh it will make a new one | 20:31 |
| megm | Ah, so on to ssh -p etc | 20:31 |
| fungi | yep | 20:31 |
| *** ivar-lazzaro has quit IRC | 20:32 | |
| *** ivar-lazzaro has joined #openstack-infra | 20:32 | |
| *** ajmiller_ has quit IRC | 20:33 | |
| *** ajmiller has quit IRC | 20:34 | |
| megm | Okay, progress... ssh -p seemed to work -- it spit out a long list of stackforge/* filenames. And git remote -v gives gerrit ssh://SstackScribe@review.openstack.org:29418/openstack/doc-specs.git (fetch) | 20:34 |
| megm | gerrit ssh://SstackScribe@review.openstack.org:29418/openstack/doc-specs.git (push) | 20:34 |
| megm | origin http://github.com/openstack/docs-specs.git (fetch) | 20:34 |
| megm | origin http://github.com/openstack/docs-specs.git (push) | 20:34 |
| megm | But git review -s fails: Problem running 'git remote update gerrit' | 20:34 |
| megm | Fetching gerrit | 20:34 |
| megm | fatal: Project not found: openstack/doc-specs | 20:34 |
| megm | fatal: Could not read from remote repository. | 20:34 |
| megm | Please make sure you have the correct access rights | 20:34 |
| megm | and the repository exists. | 20:34 |
| megm | error: Could not fetch gerrit | 20:34 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions https://review.openstack.org/178833 | 20:36 |
| *** e0ne has joined #openstack-infra | 20:36 | |
| fungi | megm: looks like that repository doesn't exist | 20:37 |
| fungi | aha, i bet it's a typo in the .gitreview file in docs-specs | 20:37 |
| fungi | checking | 20:37 |
| fungi | nope, it's correct there | 20:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add test_server.py for testing web content https://review.openstack.org/178834 | 20:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: actually actually make window location work https://review.openstack.org/178835 | 20:38 |
| megm | Maybe because my user name isn't in there? I screwed up a long time ago -- I'm StackScribe in git but SstackScribe in gerrit... | 20:38 |
| *** mrmartin has quit IRC | 20:39 | |
| fungi | megm: did it really say "Project not found: openstack/doc-specs" or did it say "Project not found: openstack/docs-specs" instead? | 20:39 |
| fungi | oh, your remote list seems to confirm | 20:40 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: Adding publisher support to join-trigger plugin https://review.openstack.org/178836 | 20:40 |
| fungi | somehow you've got an origin of docs-specs at github but doc-specs in gerrit | 20:40 |
| megm | I just copied the text from my tty window... | 20:41 |
| *** eharney has joined #openstack-infra | 20:41 | |
| fungi | megm: try 'git remote remove gerrit' followed by 'git review -s' | 20:41 |
| *** teran has quit IRC | 20:42 | |
| fungi | it looks like maybe your gerrit remote was added manually or something, and mistyped | 20:42 |
| megm | That's odd -- I have no idea how I did that. | 20:42 |
| *** teran has joined #openstack-infra | 20:42 | |
| fungi | but git review -s should add the correct remote based on the contents of the .gitreview file in the repository | 20:42 |
| megm | I think that worked! Horray! Let me try a git review -d to confirm | 20:42 |
| megm | I AM IN!!!!! Thank you so very much! | 20:43 |
| *** radez is now known as radez_g0n3 | 20:44 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 20:44 |
| annegentle | megm: woohoo! | 20:44 |
| *** e0ne has quit IRC | 20:44 | |
| fungi | megm: you're welcome--have fun! | 20:45 |
| megm | Words cannot express how happy this makes me! Now I can actually dig in and do some writing ;-) | 20:45 |
| *** jcoufal has joined #openstack-infra | 20:45 | |
| *** mtreinish_ has joined #openstack-infra | 20:45 | |
| fungi | you're in like kevin flynn | 20:45 |
| annegentle | heh | 20:46 |
| *** teran has quit IRC | 20:46 | |
| *** dustins has quit IRC | 20:48 | |
| openstackgerrit | Merged openstack-dev/pbr: Test pip install -e of projects. https://review.openstack.org/177504 | 20:49 |
| openstackgerrit | Merged openstack-dev/pbr: Stop testing setup.py easy_install behaviour https://review.openstack.org/177505 | 20:49 |
| *** dizquierdo_ has joined #openstack-infra | 20:51 | |
| *** mtreinish has quit IRC | 20:51 | |
| *** mtreinish_ is now known as mtreinish | 20:51 | |
| *** ajo has quit IRC | 20:51 | |
| *** dizquierdo has quit IRC | 20:51 | |
| *** bswartz has quit IRC | 20:54 | |
| fungi | lifeless: here's a testing conundrum for you... is it safe/sane to monkeypatch sys.exit? | 20:54 |
| *** belmoreira has joined #openstack-infra | 20:55 | |
| fungi | i'm trying to add a version to optparse in bindep and write a test to exercise it, but optparse seems to directly call sys.exit(0) on its own, unsolicited, when passed --version | 20:55 |
| *** amitgandhinz has quit IRC | 20:56 | |
| fungi | doesn't actually seem to be configurable | 20:56 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: actually actually make window location work https://review.openstack.org/178835 | 20:57 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add test_server.py for testing web content https://review.openstack.org/178834 | 20:57 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: Adding publisher support to join-trigger plugin https://review.openstack.org/178836 | 20:57 |
| *** dannywilson has quit IRC | 20:57 | |
| *** dannywilson has joined #openstack-infra | 20:58 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 20:59 |
| fungi | oh, maybe i can assertRaises on SystemExit as a solution | 20:59 |
| sdague | fungi: you can catch SystemExit | 20:59 |
| sdague | yeh | 20:59 |
| *** amitgandhinz has joined #openstack-infra | 20:59 | |
| *** tiswanso_ has quit IRC | 21:00 | |
| fungi | in this case i have to make lambda: main() too i think | 21:00 |
| *** tonytan4ever has joined #openstack-infra | 21:01 | |
| fungi | closer, but it's still not getting the output into the fixture logger | 21:02 |
| *** pabelanger has quit IRC | 21:02 | |
| lifeless | fungi: hi | 21:02 |
| * fungi is embarrassingly a total n00b at testing | 21:02 | |
| lifeless | fungi: there's a way to override the beaviour of the parser | 21:02 |
| lifeless | fungi: its usually recommended here. | 21:03 |
| fungi | for optparse? | 21:03 |
| lifeless | yup | 21:03 |
| openstackgerrit | Gary W. Smith proposed openstack/requirements: Add python-manilaclient to global requirements. https://review.openstack.org/135007 | 21:03 |
| *** jcoufal has quit IRC | 21:03 | |
| fungi | patching optparse.OptionParser or something else? | 21:03 |
| lifeless | well | 21:03 |
| lifeless | so for testing the general pattern is to inject your dependency | 21:04 |
| *** _nadya_ has quit IRC | 21:04 | |
| lifeless | so you'd make main accept a parser factory | 21:04 |
| *** esker has joined #openstack-infra | 21:04 | |
| lifeless | or decouple things in some way - there's a few things you could do | 21:04 |
| lifeless | but the least instrusive would be to monkey patch optparse.OptionParser.exit / .error | 21:04 |
| lifeless | https://docs.python.org/2/library/optparse.html#how-optparse-handles-errors | 21:05 |
| *** cody-somerville has quit IRC | 21:05 | |
| fungi | got it. so far the other tests here are patching sys.argv and then calling main() which engages optparse. i'll study up--thanks! | 21:05 |
| *** marun has quit IRC | 21:08 | |
| *** peristeri has quit IRC | 21:09 | |
| *** jtriley has quit IRC | 21:12 | |
| *** Sukhdev has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:14 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions https://review.openstack.org/178833 | 21:15 |
| *** mriedem has quit IRC | 21:17 | |
| *** samueldmq has quit IRC | 21:21 | |
| *** dkranz has quit IRC | 21:22 | |
| gary-smith | fungi, clarkb: so for pestering, but could you re-review https://review.openstack.org/175063/ ? Just trying to get this done | 21:22 |
| gary-smith | s/so/sorry/ | 21:22 |
| *** achanda has quit IRC | 21:22 | |
| fungi | gary-smith: my +2 is still there from ~4 hours ago | 21:24 |
| fungi | AJaeger: ^ | 21:24 |
| fungi | he seems to have addressed your comment | 21:24 |
| gary-smith | fungi: apologies, read that wrong | 21:24 |
| fungi | well, my irc nick isn't immediately obvious from my gerrit full name | 21:25 |
| gary-smith | fungi: thanks for re-re-reviewing it then :-) | 21:25 |
| fungi | no apologies needed | 21:25 |
| *** ddieterly has quit IRC | 21:25 | |
| gary-smith | yeah, I noticed that infra team nicks are a bit, well, colorful :-) | 21:25 |
| *** shardy has quit IRC | 21:26 | |
| *** ddieterly has joined #openstack-infra | 21:26 | |
| fungi | it's a funhouse 'round here | 21:26 |
| gary-smith | so is 'fungi' a reference to being a fun-guy, or is it like ringworm? | 21:28 |
| fungi | whatever you prefer | 21:28 |
| *** Longgeek has quit IRC | 21:29 | |
| fungi | truth be known, it's a reference to http://en.wikipedia.org/wiki/Fungi_from_Yuggoth | 21:29 |
| *** marun has joined #openstack-infra | 21:29 | |
| fungi | but don't tell anyone | 21:29 |
| gary-smith | will do. er, won't do | 21:30 |
| *** ddieterly has quit IRC | 21:30 | |
| *** Rockyg has quit IRC | 21:30 | |
| *** julim has quit IRC | 21:30 | |
| *** cody-somerville has joined #openstack-infra | 21:31 | |
| *** marun has quit IRC | 21:35 | |
| *** stevemar has quit IRC | 21:36 | |
| *** achanda has joined #openstack-infra | 21:36 | |
| *** harlowja is now known as harlowja_away | 21:36 | |
| *** teran has joined #openstack-infra | 21:36 | |
| *** harlowja_away is now known as harlowja | 21:37 | |
| *** esker has quit IRC | 21:39 | |
| *** mattfarina has joined #openstack-infra | 21:40 | |
| *** unicell has quit IRC | 21:40 | |
| *** unicell has joined #openstack-infra | 21:40 | |
| *** esker has joined #openstack-infra | 21:40 | |
| *** mattfarina has quit IRC | 21:40 | |
| jesusaurus | how do logs make it from the nodepool node to elasticsearch? im vaguely aware of a python script that reads files and writes events over tcp to logstash but I don't have a firm grasp on how that system works | 21:41 |
| fungi | jesusaurus: http://docs.openstack.org/infra/system-config/logstash.html | 21:42 |
| fungi | jesusaurus: if that doesn't answer your questions, then we should expand it | 21:42 |
| redrobot | hello infra folk! | 21:42 |
| redrobot | does anyone have pointers as to why this CR is not merging? https://review.openstack.org/#/c/178242/ | 21:43 |
| jesusaurus | fungi: oh, perfect, a magic link, i like those | 21:43 |
| *** SumitNaiksatam has quit IRC | 21:43 | |
| redrobot | it appears the gate job is not triggering? :( | 21:43 |
| *** marun has joined #openstack-infra | 21:44 | |
| fungi | redrobot: top of https://wiki.openstack.org/wiki/Infrastructure_Status | 21:44 |
| redrobot | fungi ack, thanks! | 21:44 |
| fungi | looks like it was approved between 13:29-14:05 | 21:45 |
| fungi | so someone needs to (if necessary remove and then re-)add a workflow +1 to it | 21:45 |
| fungi | redrobot: ^ | 21:45 |
| redrobot | fungi I see... I thought "reverify" would take care of it? | 21:46 |
| fungi | nope. reverify is like recheck. they're synonymous | 21:46 |
| *** jtriley has joined #openstack-infra | 21:47 | |
| fungi | they cause the check pipeline jobs to be rerun, however because of the way gerrit works when zuul tries to leave a new +1 over top of the old +1, gerrit streams that event out as not updating the vote, so gating doesn't get triggered | 21:48 |
| fungi | it's something we'd like to fix | 21:48 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling https://review.openstack.org/178852 | 21:48 |
| redrobot | fungi interesting. that's good to know. thanks again! | 21:48 |
| lifeless | fungi: ^ thats an rc bug vis-a-vis releasing 0.11 | 21:49 |
| fungi | generally only becomes an issue when zuul misses an event from the stream in the first place though, which we'd like to fix more (the current bugs we're experiencing with gerrit causing us to have to restart it to get the event stream working again) | 21:49 |
| fungi | lifeless: good catch | 21:50 |
| lifeless | fungi: been tracking down oddness with alan pevec for a bit | 21:51 |
| lifeless | fungi: finally got a case that was happening with master | 21:51 |
| *** annegentle has quit IRC | 21:51 | |
| *** bnemec has quit IRC | 21:52 | |
| lifeless | clarkb: ^ if you're back | 21:54 |
| *** tonytan4ever has quit IRC | 21:54 | |
| *** bhunter71 has quit IRC | 21:54 | |
| lifeless | mordred: if you're up? https://review.openstack.org/178852 | 21:54 |
| *** shivharis has joined #openstack-infra | 21:55 | |
| *** esker has quit IRC | 21:55 | |
| clarkb | looking | 21:56 |
| shivharis | anteaya: hi | 21:56 |
| zaro | clarkb: so have you decided what you want to do with gearman plugin? | 21:58 |
| zaro | are we monitoring it? | 21:58 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: OpenstackId resource server https://review.openstack.org/178853 | 21:58 |
| fungi | +6671, -0 | 21:59 |
| clarkb | zaro: not actively, fungi cleared out the bad nodes. Its easy to check if we get in that state again. I have mostly been staring at the code all day unraveling what it is doing | 21:59 |
| clarkb | zaro: I think there may be layers of bugs here, but basically we have to driveIO and depend on the select() for some of that but then will submit tasks and jobs behind work()s back | 22:00 |
| *** tsg_ has quit IRC | 22:00 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Migration of resource server https://review.openstack.org/178854 | 22:00 |
| *** tsg_ has joined #openstack-infra | 22:01 | |
| *** bnemec has joined #openstack-infra | 22:02 | |
| *** melwitt has quit IRC | 22:02 | |
| *** melwitt has joined #openstack-infra | 22:02 | |
| clarkb | zaro: I think the intent of the taskmap in the worker implementation was to keep track of this but then work() never uses it? | 22:03 |
| *** hdd has quit IRC | 22:03 | |
| clarkb | zaro: we basically add and remove tasks from that but we never check it | 22:03 |
| *** Swami has quit IRC | 22:04 | |
| *** patrickeast_ has joined #openstack-infra | 22:04 | |
| *** prad has quit IRC | 22:04 | |
| *** patrickeast has quit IRC | 22:05 | |
| *** patrickeast_ is now known as patrickeast | 22:05 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add keystone services/endpoints methods https://review.openstack.org/177621 | 22:07 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Rename get_endpoint() to get_session_endpoint() https://review.openstack.org/178859 | 22:07 |
| *** rlandy has quit IRC | 22:09 | |
| clarkb | zaro: I am not sure we properly sleep | 22:10 |
| clarkb | zaro: instead we go into a sleep state then immediately send a grab job then will go into a sleep state again | 22:10 |
| clarkb | hrm no thats not quite right either | 22:11 |
| *** otter768 has joined #openstack-infra | 22:11 | |
| *** shivharis has quit IRC | 22:12 | |
| *** shivharis has joined #openstack-infra | 22:12 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add option to skip certification check https://review.openstack.org/178794 | 22:12 |
| clarkb | zaro: handleSessionEvent definitely doesn't have a way to communicate to work() that it sent grab job on receiving a NOOP from what I can tell | 22:12 |
| zaro | clarkb: looking at it i'm not sure why there's a taskmap. it's definately not getting used. | 22:14 |
| clarkb | zaro: I think its meant to keep track of this | 22:14 |
| clarkb | its just not doing that | 22:14 |
| clarkb | so basically we don't need the grab job sent boolean, instead we should check the taskmap | 22:15 |
| lifeless | clarkb: thanks, wht did you think? | 22:15 |
| clarkb | lifeless: sorry got completely distracted by java and I don't know why, opened the change in my browser then squirrel! | 22:15 |
| clarkb | will actually look now | 22:15 |
| *** otter768 has quit IRC | 22:16 | |
| clarkb | zaro: also we should only have a single session right? not sure why we need a map, i think we just need a reference to the current task | 22:17 |
| zaro | clarkb: you mean taskmap object is meant for communicating with work()? | 22:17 |
| clarkb | zaro: ya, its tracking what the outstanding task is | 22:17 |
| clarkb | zaro: so that work() knows if it needs to drive IO or send another grab job etc | 22:17 |
| lifeless | clarkb: java the drink or java the language? | 22:18 |
| *** belmoreira has quit IRC | 22:18 | |
| *** dguerri is now known as _dguerri | 22:18 | |
| *** tsg_ has quit IRC | 22:18 | |
| *** nelsnelson has quit IRC | 22:18 | |
| clarkb | lifeless: java the language | 22:18 |
| lifeless | squirrel! | 22:19 |
| *** salv-orlando has joined #openstack-infra | 22:19 | |
| clarkb | lifeless: the bug says 2015.1.0.0rc3.dev1 is expected but isn't that an invalid version under pep440? | 22:20 |
| lifeless | nope | 22:20 |
| lifeless | its entirely valid | 22:20 |
| lifeless | fixing pbr to generate those was the thing master was blocked on for 6 months | 22:20 |
| zaro | clarkb: yeah, i think multiple tasks per session, but a single session. | 22:20 |
| clarkb | zaro: but its using the same session always as the key | 22:21 |
| clarkb | lifeless: ah yup, I am brushing up on 440 now | 22:21 |
| lifeless | clarkb: http://legacy.python.org/dev/peps/pep-0440/#version-scheme | 22:21 |
| lifeless | clarkb: note that the version generation logic isn't involved in this patch | 22:21 |
| lifeless | clarkb: its simply logic to not emit a lower version than we've already seen | 22:21 |
| fungi | clarkb: also remember that even if you brushed up on it while it was being drafted, it changed in subtle ways several times (including after it was finalized) so refreshing is highly recommended | 22:22 |
| openstackgerrit | Alexander Yip proposed openstack-infra/project-config: Add congress.yaml file https://review.openstack.org/178862 | 22:22 |
| *** salv-orl_ has quit IRC | 22:23 | |
| *** amitgandhinz has quit IRC | 22:25 | |
| *** bswartz has joined #openstack-infra | 22:27 | |
| *** esker has joined #openstack-infra | 22:27 | |
| clarkb | lifeless: so basically distance was calculated from 2015.1.0.0rc2 as 1, but then we use target_version anyways | 22:27 |
| clarkb | lifeless: wouldit be simpler to just use the tag returned, make a version of tag + distance? | 22:27 |
| clarkb | assuming tag is not none | 22:28 |
| lifeless | clarkb: no | 22:28 |
| lifeless | clarkb: if you have done a commit | 22:28 |
| lifeless | clarkb: you don't want to say that the version is the version of the tag which is a commit back | 22:28 |
| lifeless | clarkb: distance is 'number of commits since a tag' | 22:28 |
| clarkb | you want to sa its the tag + distance | 22:28 |
| clarkb | right | 22:28 |
| *** dimtruck is now known as zz_dimtruck | 22:29 | |
| lifeless | clarkb: so there are two cases. preversioning and postversioning | 22:29 |
| clarkb | yes | 22:29 |
| lifeless | postversioning is easy: increment the version number and use distance for the dev count. | 22:29 |
| lifeless | preversioning is complicated by two things | 22:29 |
| clarkb | but in either case its tag + distance thenflip the string from dev to post depending | 22:29 |
| lifeless | clarkb: no, we never use post | 22:29 |
| clarkb | lifeless: I think we have to with new pip | 22:30 |
| lifeless | clarkb: no, we don't | 22:30 |
| clarkb | 0.10 pbr definitely does use post because of this | 22:30 |
| fungi | er, for postversioning you don't even increment the version number do you? just the dev count | 22:30 |
| clarkb | fungi: you use postN | 22:30 |
| lifeless | clarkb: it does because folk around at the time chose to hack up 0.10 rather than fix master. | 22:30 |
| fungi | er, right post not dev | 22:30 |
| clarkb | lifeless: yes because master was in a bad place | 22:30 |
| lifeless | clarkb: which was a mistake, and a bug, and not aligned with pep-440. | 22:30 |
| clarkb | lifeless: and 0.10 worked | 22:30 |
| clarkb | it is aligned with 440... | 22:30 |
| fungi | for postversioning, don't increment the version number, just increment the post counter | 22:30 |
| lifeless | clarkb: it really wasn't, it was very fast to change for me. But I don't want to fight over that. Its pointless | 22:31 |
| lifeless | clarkb: no, its not. read the definition of postN. They are public versions. | 22:31 |
| fungi | as opposed to preversioning where you need to increment the version number and then the dev count from the most recent tag | 22:31 |
| clarkb | lifeless: both dev and post are public versions | 22:31 |
| mordred | morning | 22:31 |
| fungi | well, post is "public" in the sense that it denotes minimal changes to metadata/packaging | 22:32 |
| lifeless | clarkb: yes | 22:32 |
| lifeless | clarkb: public and defined semantics | 22:32 |
| lifeless | clarkb: "Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes). | 22:32 |
| fungi | whereas dev is something the pep440 authors didn't envision being uploaded to pypi | 22:32 |
| lifeless | " | 22:32 |
| lifeless | clarkb: that is *not* what pbr 0.10 uses post or. | 22:32 |
| clarkb | on a number line we have 1.1.dev1 < 1.1 < 1.1.post1 | 22:32 |
| lifeless | *for* | 22:32 |
| openstackgerrit | Alexander Yip proposed openstack-infra/project-config: Add congress.yaml file https://review.openstack.org/178862 | 22:33 |
| clarkb | this is important because for post versioning we need to sort after 1.1 | 22:33 |
| lifeless | clarkb: Can we step back. | 22:33 |
| clarkb | sure, we need to ensure consistent sorting | 22:33 |
| clarkb | lets start from there | 22:33 |
| lifeless | clarkb: I'm confused. We have a spec, for using dev. It works with every single case, and there was an impl bug in pbr that fixes this bug. | 22:33 |
| clarkb | lifeless: it does not work for sorting this is specifically why we used post | 22:33 |
| clarkb | we had to or pip sorts wrong | 22:33 |
| lifeless | clarkb: I'm sorry, but you're wrong. | 22:33 |
| clarkb | how? can you show me the sort order? | 22:34 |
| lifeless | yes | 22:34 |
| clarkb | is the order above wrong? | 22:34 |
| lifeless | https://mail.python.org/pipermail/distutils-sig/2015-April/026134.html | 22:34 |
| lifeless | ignore post | 22:35 |
| *** jamesmcarthur has quit IRC | 22:35 | |
| lifeless | I know pbr 0.10 uses it. But just ignore it, ok? | 22:35 |
| *** erikmwilson is now known as Guest61780 | 22:35 | |
| *** erikmwilson has joined #openstack-infra | 22:35 | |
| lifeless | We can come back to *why* to ignore it later. | 22:35 |
| *** whoops has quit IRC | 22:35 | |
| clarkb | ou seem to agree that dev sorts first based on that post | 22:35 |
| clarkb | lifeless: I can't ignore it | 22:36 |
| lifeless | why can't you? | 22:36 |
| clarkb | that was a very important part of the puzzle when we unbroke everything 4 months ago | 22:36 |
| clarkb | because of sort order | 22:36 |
| lifeless | so the break that occured 4 months ago was because we didn't honour rc tags at all in master | 22:36 |
| clarkb | and we broke post versioning | 22:36 |
| lifeless | which was a genuine problem, and in deed this patch is fixing another instance of that. | 22:37 |
| clarkb | and about half a dozen other things | 22:37 |
| *** Guest61780 has quit IRC | 22:37 | |
| mordred | dev was how we fixed 0.10, but it's not applicable in master | 22:37 |
| lifeless | mordred: s/dev/post/ - | 22:37 |
| mordred | yeah. sorry - it's late here | 22:37 |
| *** ZZelle_ has joined #openstack-infra | 22:37 | |
| lifeless | clarkb: I am aware of that issue, and if you dig it up you'll see we only ever emitted dev versions, never rcX.devN versions | 22:37 |
| clarkb | can someone explain why using sort order? | 22:37 |
| lifeless | clarkb: and thats *why* this fix is fine. | 22:37 |
| clarkb | lifeless: that is for preversioning | 22:38 |
| lifeless | clarkb: no, everything. | 22:38 |
| clarkb | where we have a target that we are moving towards | 22:38 |
| clarkb | in the case where we have tagged a release and use post versioning the next commit must sort higher than the tag itself | 22:38 |
| lifeless | clarkb: again, all harmonised in master a year ago, except for a shallow bug that was fixed recently. | 22:38 |
| clarkb | using dev does not do that | 22:38 |
| lifeless | clarkb: I think it might help if you read the pbr master docs on this, because thats exactly what master does. | 22:38 |
| lifeless | clarkb: it bumps the version number automatically. | 22:39 |
| fungi | part of the new world order for pbr is insisting on preversioning everywhere and inventing a new goal you're developing toward if you haven't specified one in some metadata for your packaging | 22:39 |
| lifeless | clarkb: and there are a raft of tests. | 22:39 |
| lifeless | clarkb: and docs about this | 22:39 |
| fungi | i can't say i agree with it, but it seems to be what some people want | 22:39 |
| clarkb | lifeless: ok thats the missing piece and all you had to say "pbr emits the next version" | 22:39 |
| lifeless | clarkb: I feel like I'm arguing with you about what pbr 0.9 did | 22:39 |
| lifeless | clarkb: which prompted *all* this wwork in the first place | 22:39 |
| clarkb | so on a number line we have 1.1.dev1 < 1.1 < 1.2 | 22:39 |
| clarkb | ^ is all I needed | 22:39 |
| mordred | yes | 22:39 |
| lifeless | fungi: actually pbr is saying 'you don't need preversioning ever' | 22:40 |
| fungi | uh | 22:40 |
| lifeless | fungi: it doesn't make any assumptions about what you're working towards. It makes the smallest version bump it can. | 22:40 |
| lifeless | fungi: based on your commits | 22:40 |
| fungi | pbr is guessing at what your next tag will be | 22:40 |
| lifeless | fungi: you can choose to have a bigger gap | 22:40 |
| clarkb | sorry, 1.1.dev1 < 1.1 < 1.2.dev1 | 22:40 |
| fungi | and preversioning based on that | 22:40 |
| lifeless | fungi: no, its using semver to know what your next tag *must be no lower than* | 22:41 |
| mordred | fungi: pbr is working towards what the next logical tag would be not caring if you later decide to make a larger jump | 22:41 |
| lifeless | fungi: this is really quite a big conceptual difference. | 22:41 |
| lifeless | fungi: preversioning - some arbitrarily huge jump. | 22:41 |
| *** melwitt has quit IRC | 22:41 | |
| fungi | my point was, i _personally_ like things normal humans sort without having to know that dev means you sort before (a la ~ in dpkg) | 22:41 |
| lifeless | fungi: pbr - you must increment by *no less than Y* | 22:41 |
| mordred | so, if you have 1.1, it'll make a 1.2.dev1 - and then if you decide the next release is going to be 2.0 at some point, it still sorts fine | 22:41 |
| *** dangers is now known as dangers_away | 22:41 | |
| *** melwitt has joined #openstack-infra | 22:41 | |
| fungi | whereas post-versioning means i just get x commits since my last tag, appended to the version of the last tag. done. it's easy to remember conceptually, easy to sort on a number line | 22:42 |
| mordred | fungi: totally understand | 22:42 |
| lifeless | fungi: ok that I get. | 22:42 |
| lifeless | clarkb: yes, 1.1.dev1 < 1.1 < 1.2.dev1 | 22:43 |
| lifeless | clarkb: the issues 4 months back were with | 22:43 |
| fungi | i'm not protesting the default behavior in pbr because it seems to be the versioning scheme that openstack has enshrined for some reason i'm unable to fathom | 22:43 |
| mordred | fungi: you and me both | 22:43 |
| lifeless | 1.1.dev1 < 1.1.rc1 < 1.1.rc1.dev1 < 1.1 | 22:43 |
| lifeless | clarkb: and pbr wasn't generating 1.1.rc1.dev1 ever, but instead going back to 1.1.dev1 at the first commit after 1.1.rc1 | 22:44 |
| lifeless | clarkb: that was a deliberate choice we made, which broke when pep-440 shifted under us. | 22:44 |
| clarkb | lifeless: yes which is why I initially asked if maybe using tag + distance would be better here | 22:44 |
| clarkb | lifeless: because tag + distance is more readable and more closely follows the 1.1.rc1.dev1 | 22:44 |
| lifeless | clarkb: the fix a month or so back to pbr master was to undo that deliberate choice and use the pep-440 blessed numbers | 22:44 |
| clarkb | but I understand now there is a third factor which is distance' | 22:45 |
| *** imcsk8 has joined #openstack-infra | 22:45 | |
| lifeless | clarkb: given a tag of 1.1.rc1 and a distance of 1, what version are you proposing that pbr would create? | 22:45 |
| clarkb | the modficaton to tag to accomodate distance | 22:45 |
| clarkb | lifeless: 1.1.rc2.dev1 | 22:45 |
| lifeless | ok, so my number above were bad :) . yes, that the number that pbr will create with my patch | 22:45 |
| lifeless | it creates it without my patch for postversioning | 22:46 |
| lifeless | preversioning is a special case and had a bug. | 22:46 |
| lifeless | the intent was it create 1.1.rc2.dev1 | 22:46 |
| clarkb | yes I think your patch is sound, I am saying I think there is a better way to represent the modifcations here so that the code is readable and we don't have bugs like this | 22:46 |
| *** bknudson has quit IRC | 22:46 | |
| *** gordc has quit IRC | 22:46 | |
| clarkb | tag + distance' + distance | 22:47 |
| lifeless | whats distance' ? | 22:47 |
| clarkb | lifeless: the bump to accomodate distance | 22:47 |
| clarkb | lifeless: since we are never using post | 22:47 |
| lifeless | thats new_version, up about 10 lines out of the diff context | 22:47 |
| clarkb | in the above its 1.1.rc1 + distance' + dev$distance | 22:47 |
| *** Swami has joined #openstack-infra | 22:47 | |
| fungi | i'm interpreting +distance' there as a d() | 22:48 |
| lifeless | clarkb: and you're missing the 'scan git for sem-ver: pseudo headers' in your rephrasing of things | 22:48 |
| fungi | d(1.1.rc1) = 1.1.rc2 | 22:48 |
| clarkb | lifeless: no distance' can deal with the sem-ver | 22:48 |
| clarkb | fungi: ya its basically the derivative of raw distance to determine offset from tag | 22:49 |
| lifeless | clarkb: I'd like to land the minimal patch. What you're describing makes no sense to my head atm - I don't understand it | 22:49 |
| fungi | oh, as i "i've declared a an api break, bump me harder" | 22:49 |
| lifeless | clarkb: so if we want to refactor the code to make more sense to more people, thats fine, but its certainly not a minor tweak to fix an rc bug | 22:49 |
| lifeless | (and I support making it make more sense to more people) | 22:50 |
| mordred | more sense == ++ | 22:50 |
| clarkb | lifeless: basically distance determines two things. It determines the N in devN and it determines what tag should be incremented to | 22:50 |
| fungi | making sense to people is something it seems to not be doing currently ;) | 22:50 |
| lifeless | clarkb: no it doesn't | 22:50 |
| clarkb | lifeless: yes it does | 22:50 |
| lifeless | clarkb: thats not how the code uses distance today, at all all. It uses it for just one thing - the N in devN. | 22:50 |
| lifeless | clarkb: no, it doesn't. | 22:51 |
| clarkb | lifeless: yes I know the current code doesn't hence my suggestion that maybe this is easier without the crazy special casing | 22:51 |
| *** dboik has quit IRC | 22:51 | |
| clarkb | lifeless: and instead we calculate 3 values, and combine them always | 22:51 |
| lifeless | clarkb: what | 22:51 |
| lifeless | ? | 22:51 |
| clarkb | you have your tagged version, the amount to increment that version by and devN | 22:51 |
| clarkb | this is true for post and pre versioning | 22:51 |
| *** dboik has joined #openstack-infra | 22:51 | |
| clarkb | so rather than special case aroudn it, take the tuple and emit a new version | 22:52 |
| fungi | the amount to increment that version by is always 1 yeah? (excepting the declared api break case?) | 22:52 |
| lifeless | clarkb: so thats last-semver, increment, and distance | 22:52 |
| clarkb | fungi: I think its 0 for preversioning and >=1 for post versioning | 22:52 |
| lifeless | clarkb: so new_dev = new_version.to_dev(distance) is exactly what you're asking for | 22:52 |
| lifeless | clarkb: its not special cased | 22:52 |
| lifeless | 10:52 < clarkb> fungi: I think its 0 for preversioning and >=1 for post versioning | 22:53 |
| lifeless | ^ I don't understand what that can be in reference to | 22:53 |
| *** ZZelle_ has quit IRC | 22:53 | |
| lifeless | clarkb: if the 'its' there refers to distance, then no. distance is only and always the number of commits back to the tag. | 22:53 |
| clarkb | lifeless: 1.1.rc1 + one commit in post versioning is 1.1.rc2.dev1, in preversioning its 1.1.rc1.dev1 | 22:53 |
| lifeless | no | 22:54 |
| lifeless | 1.1.rc1 + 1 commit | 22:54 |
| lifeless | post versioning -> 1.1.rc2.dev1 | 22:54 |
| lifeless | pre_versioning if the target version is 1.1 -> 1.1.rc2.dev1 | 22:54 |
| lifeless | pre_versioning if the target version is (say) 1.2 -> 1.2.dev1 | 22:54 |
| lifeless | 1.1.rc1.dev1 would be going backwards | 22:55 |
| lifeless | and is the bug this patch fixes | 22:55 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Story Search Results now show loading spinner. https://review.openstack.org/178867 | 22:55 |
| clarkb | huh why distinguish then? is it purely because we want a target? | 22:55 |
| fungi | hrm... my understanding of post versioning was yet something else entirely. i thought those were both pre-versioning | 22:55 |
| clarkb | (I am noting that your two versions are identical) | 22:55 |
| lifeless | target_version is not None <- this is what determines pre_versioning | 22:55 |
| lifeless | clarkb: they are yes. | 22:56 |
| lifeless | clarkb: why distinguish what? | 22:56 |
| clarkb | lifeless: pre vs post if the version comes out the same either way | 22:56 |
| fungi | those seem like both pre-versioning, one with a target version and one without | 22:56 |
| lifeless | clarkb: because preversioning when the preversion is *higher* lifts teh version. | 22:56 |
| clarkb | lifeless: ya but you can do that easier with a tag | 22:57 |
| lifeless | clarkb: yes which is why I say that preversioning can be deprecated | 22:57 |
| lifeless | clarkb: there's no point and it can lead to bugs because its harder to reason about. | 22:57 |
| jhesketh | Morning | 22:57 |
| lifeless | clarkb: but until we get 0.11 out the door, we don't have the generation of versions needed to allow us to deprecate it | 22:57 |
| clarkb | fungi: basicall new world order is preversion everything against the calculate target version | 22:58 |
| lifeless | because 0.10 doesn't bump the version ever | 22:58 |
| clarkb | fungi: calculated target may come from a hard coded version or from a tag + 1 | 22:58 |
| lifeless | target_version == None <- postversioning | 22:58 |
| *** rm_work is now known as rm_work|away | 22:58 | |
| lifeless | target_version == something <- preversion | 22:58 |
| lifeless | clarkb: there is no calculated target for postversioning | 22:58 |
| fungi | clearly there needs to be yet another name for the sort of versioning i like, which none of this is. where is the "i released 1.1.0 and have 3 commits since then so call my local version 1.1.0.something3?" | 22:58 |
| clarkb | lifeless: it does bump the version but only by using post | 22:59 |
| fungi | that's what i thought we used to refer to as postversioning | 22:59 |
| clarkb | lifeless: it doesn't bump in bigger steps | 22:59 |
| lifeless | clarkb: it does the minimum bump yes. | 22:59 |
| clarkb | fungi: thats what 0.10 does | 22:59 |
| clarkb | fungi: so 0.10 does proper postversioning, master does preversion everything | 22:59 |
| lifeless | fungi: postversioning in the docs was 'calculates a new version for you', preversioning was 'counts down to your desired version' | 22:59 |
| fungi | i think i'll call it "versioning" to distinguish if from these other types | 22:59 |
| lifeless | until the pep-440-gate, 0.10's postversioning was entirely broken. the change to use .post made it sort-order compatible | 23:00 |
| lifeless | but not semantics compatible | 23:00 |
| clarkb | lifeless: I am still not sure I follow on that but ok | 23:00 |
| clarkb | lifeless: the pep clearly says both dev and post are public version | 23:00 |
| lifeless | fungi: we could add a mode to use local versions for what you want | 23:00 |
| clarkb | s | 23:00 |
| lifeless | clarkb: they are | 23:00 |
| lifeless | clarkb: they defined meaning for postN is 'minor errors in a final release' | 23:01 |
| fungi | i don't understand why "3rd commit since 1.1.0" isn't semantically compatible | 23:01 |
| lifeless | http://legacy.python.org/dev/peps/pep-0440/#post-releases | 23:01 |
| lifeless | "The use of post-releases to publish maintenance releases containing actual bug fixes is strongly discouraged. In general, it is better to use a longer release number and increment the final component for each maintenance release." | 23:01 |
| fungi | i guess for projects with multiple branches which have the 1.1.0 tag in their history, 1.1.0.post3 is vague. was that the main concern with them? | 23:02 |
| clarkb | we never use post releases to publish anything | 23:02 |
| *** camunoz_gone is now known as camunoz | 23:02 | |
| lifeless | fungi: thats a big part of it | 23:02 |
| lifeless | one of the use cases we were solving was generating public versions for folk doing CD | 23:03 |
| lifeless | including building into rpms etc | 23:03 |
| *** patrickeast has quit IRC | 23:03 | |
| lifeless | postN versions that are defined as not affecting the distributed software aren't suitable for that | 23:03 |
| *** tsg has joined #openstack-infra | 23:03 | |
| *** mase_x200 has joined #openstack-infra | 23:03 | |
| lifeless | oh yay | 23:03 |
| lifeless | 2015-04-29 21:59:00.241 | E: Unable to locate package libxslt-dev | 23:03 |
| lifeless | 2015-04-29 21:59:00.242 | E: Package 'pkg-config' has no installation candidate | 23:03 |
| lifeless | 2015-04-29 21:59:00.257 | E: Package 'libsqlite3-dev' has no installation candidate | 23:04 |
| lifeless | http://logs.openstack.org/52/178852/1/check/check-pbr-installation-dsvm/f5f6704//console.html | 23:04 |
| clarkb | lifeless: thats what devstack was doing for you | 23:04 |
| clarkb | so, basically pep440 has broken us in another way | 23:04 |
| lifeless | gnargle | 23:04 |
| clarkb | and this is pbr finally dealing with it | 23:04 |
| clarkb | I can deal with that, its just annoying | 23:04 |
| mordred | clarkb: yah | 23:04 |
| jogo | huh: http://logs.openstack.org/33/165233/12/check/check-grenade-dsvm-partial-ncpu/c3524f1/logs/grenade.sh.txt.gz#_2015-04-29_18_39_40_830 | 23:04 |
| bswartz | could someone do me a favor and take another look at this patch tonight? https://review.openstack.org/#/c/175063/ | 23:04 |
| lifeless | clarkb: ok, so we need to put the devstack call back ? | 23:04 |
| bswartz | we have some work that depends on the new repo getting created | 23:04 |
| fungi | i mean, with pep440-semver pbr and no target_version the versioning on two separate branches each with the same most recent tag is equally vague/conflicted, so it doesn't necessarily help, but gives you some additional avenues you can leverage to make it less vague by declaring target_version on your branches | 23:04 |
| clarkb | lifeless: or otherwise install the C libs that you need to install ppypi package | 23:04 |
| lifeless | clarkb: huh? thats apt failing | 23:05 |
| *** signed8bit has quit IRC | 23:05 | |
| lifeless | clarkb: that I'm pointing at | 23:05 |
| clarkb | lifeless: oh for some reason I read it as pipfailure | 23:05 |
| lifeless | clarkb: nah-uh | 23:05 |
| clarkb | lifeless: I do not know then, maybe mirror exploded | 23:05 |
| jogo | pypi mirror issues? | 23:05 |
| clarkb | they do that | 23:05 |
| clarkb | jogo: no apt mirror | 23:05 |
| lifeless | so I think I can make this code a teeny bit cleaner | 23:05 |
| lifeless | now we've gotten past the angst | 23:06 |
| jogo | clarkb: see the link I shaerd | 23:06 |
| clarkb | it occurs to me that pep440 doesn't seem to have addressed at all what you should do for commits beyond a release | 23:06 |
| clarkb | pbr is probably the only thing that will exist in the python world that addresses it, and it does so by making you use git and follow commmit rules | 23:06 |
| clarkb | which isn't good for the other half of python using hg | 23:06 |
| lifeless | clarkb: would you prefer http://paste.ubuntu.com/10946194/ ? | 23:07 |
| *** tqtran_ has joined #openstack-infra | 23:07 | |
| *** tqtran_ is now known as tqtran | 23:07 | |
| lifeless | clarkb: so there are other vcs-inferring setuptools plugins out there | 23:07 |
| *** dalgaaf has quit IRC | 23:07 | |
| clarkb | jogo: thats an hpcloud networking error | 23:08 |
| clarkb | jogo: no route to host | 23:08 |
| fungi | lifeless: on that failure log, did we maybe previously rely on devstack to apt-get update? | 23:08 |
| jogo | clarkb: ahh thanks | 23:08 |
| *** dannywilson has quit IRC | 23:08 | |
| *** ildikov has quit IRC | 23:08 | |
| clarkb | lifeless: ya I think thats a bit clearer | 23:08 |
| fungi | i don't see any apt-get update before the apt-get install (including in the setup logs) | 23:08 |
| lifeless | clarkb: pep-440's basic answer is a) use dev versions, and/or b) use local versions | 23:08 |
| lifeless | ok, so should we put the devstack call back for now ? | 23:09 |
| *** patrickeast has joined #openstack-infra | 23:09 | |
| clarkb | lifeless: but dev versions always sort lower than the version they append | 23:09 |
| lifeless | or should I include an apt-get update call in my patch ? | 23:09 |
| clarkb | lifeless: so you basically need machinery to do what pbr does and prediect what it will be | 23:09 |
| lifeless | clarkb: yes | 23:09 |
| clarkb | lifeless: yes, I think you may need an apt-get update because things may have gotten unsycned | 23:09 |
| lifeless | clarkb: for your chosen version scheme. Since pep-440 supports three or so schemes. | 23:09 |
| *** bhunter71 has joined #openstack-infra | 23:09 | |
| fungi | lifeless: include apt-get update in the integration test script | 23:10 |
| fungi | in pbr | 23:10 |
| fungi | should be self-testing at this point | 23:11 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling https://review.openstack.org/178852 | 23:11 |
| *** dannywilson has joined #openstack-infra | 23:11 | |
| lifeless | clarkb: there is c) add a nother point to your versions and use that. x.y.z.N | 23:11 |
| lifeless | clarkb: we can review a few of those if we wish for future pbr enhancements | 23:12 |
| lifeless | right now though I just want to get a years worth of stuff sitting pending released and in use | 23:12 |
| clarkb | lifeless: but c) isn't semver :) | 23:13 |
| lifeless | right | 23:13 |
| * mordred punches semver in the mouth | 23:13 | |
| lifeless | so its not pep-440 purely giving us pause | 23:13 |
| lifeless | its the interaction of a few things | 23:13 |
| lifeless | multiple folk's branches | 23:13 |
| lifeless | public versions | 23:13 |
| lifeless | semver | 23:13 |
| lifeless | (blame mordred) | 23:13 |
| lifeless | the nasty preversion or not-preversion stuff which is now a lot cleaner | 23:14 |
| *** Sukhdev has quit IRC | 23:14 | |
| lifeless | the's now one try:except and that if target_version is None if-block around it | 23:14 |
| *** kmartin_ has quit IRC | 23:17 | |
| *** viglesias has quit IRC | 23:17 | |
| *** kmartin has joined #openstack-infra | 23:17 | |
| *** ivar-lazzaro has quit IRC | 23:18 | |
| *** ivar-lazzaro has joined #openstack-infra | 23:19 | |
| *** ivar-lazzaro has quit IRC | 23:20 | |
| *** baoli has quit IRC | 23:20 | |
| *** viglesias has joined #openstack-infra | 23:20 | |
| *** ivar-lazzaro has joined #openstack-infra | 23:20 | |
| *** Rockyg has joined #openstack-infra | 23:20 | |
| *** alexsyip has joined #openstack-infra | 23:20 | |
| *** ildikov has joined #openstack-infra | 23:22 | |
| cinerama | pleia2: oh hey i bet you want an update on how things are going | 23:23 |
| *** sarob_ has joined #openstack-infra | 23:24 | |
| pleia2 | cinerama: whenever you have one :) | 23:24 |
| cinerama | pleia2: so the code to generate the zanata.xml is done. it's a little dodgy :) | 23:24 |
| pleia2 | woo | 23:24 |
| pleia2 | and that's ok, that's why we have good code review tooling :) | 23:24 |
| cinerama | pleia2: the best way to noninteractively get a zanata.xml with a full list of enabled locales was to just use the server REST API | 23:25 |
| pleia2 | ok | 23:25 |
| *** packet has quit IRC | 23:25 | |
| cinerama | pleia2: which is also how we're going to get our percentage completion stats to use when we go to pull files | 23:26 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Issue #1450210: Preversioning rc tag handling https://review.openstack.org/178852 | 23:26 |
| lifeless | clarkb: sadly the pastebin doesn't work | 23:26 |
| pleia2 | cinerama: so are these both things that the client polls for? | 23:26 |
| lifeless | clarkb: because devs are complex, so I've reverted to the thing that worked | 23:26 |
| cinerama | pleia2: so i'm working on the pull now. then there will be modified proposal scripts that use a mix of the existing bash functions we source in together with our new stuff | 23:26 |
| pleia2 | cinerama: gotcha | 23:27 |
| fungi | subclassing optparse.OptionParser is turning out to be gnarly. not only does it want to call sys.exit() but it also wants to write directly to sys.stdout. i think i just need to punt on an end-to-end test and simply test that the get_version() function isn't broken | 23:27 |
| cinerama | pleia2: so if we use the client itself we can get the statistics in a somewhat fiddlier format. we can't easily get a base zanata.xml out of it as the "init" subcommand is interactive only and i didn't really feel like doing the expect thing | 23:28 |
| pleia2 | cinerama: ah, right | 23:28 |
| clarkb | lifeless: ok | 23:29 |
| pleia2 | cinerama: I'm thinking lots of inline comments for these scripts? :) | 23:29 |
| cinerama | pleia2: so we need the xml to actually do anything, that requirement's not going away but i do wish they supported the percentage thing as we have in the transifex client. but we can work around it as i said | 23:30 |
| lifeless | clarkb: (because expressing the interactions inline in that function would retard comprehension even more) | 23:30 |
| * pleia2 nods | 23:30 | |
| *** jtriley has quit IRC | 23:30 | |
| cinerama | pleia2: that's probably a good idea. comments are usually an artifact of my development process but also there are things here which are obvious to me that are not going to be obvious to someone who hasn't been kicking this client around | 23:31 |
| lifeless | fungi: can you pastebin your patch ? | 23:31 |
| pleia2 | cinerama: yeah, and they will help us to identify sections of the code we can rework if the zanata devs can make the changes we need (like for percentage pain) | 23:31 |
| lifeless | fungi: since pbr has a helper for this anyhow.. | 23:31 |
| fungi | lifeless: sure, just a sec | 23:31 |
| lifeless | https://jenkins06.openstack.org/job/check-pbr-installation-dsvm/17/console is looking happier | 23:32 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add a --version option https://review.openstack.org/178874 | 23:32 |
| fungi | lifeless: that's ^ where i'm at so far with trying to subclass it | 23:32 |
| cinerama | pleia2: yep. bear in mind they have a variety of clients to pick from so hopefully that rest is going to be pretty stable | 23:33 |
| *** melwitt has quit IRC | 23:33 | |
| pleia2 | cinerama: yes, one hopes :) | 23:34 |
| fungi | lifeless: i'm thinking i also need to subclass optparse.OptionParser.print_version() to not write to stdout, but am having trouble figuring out how to feed that back into the logger | 23:34 |
| fungi | er, not subclass but shadow | 23:34 |
| lifeless | fungi: no, gotta decide what to test. | 23:35 |
| lifeless | fungi: my main concern is that you have logic about versions in there at all | 23:35 |
| fungi | lifeless: is the current wisdom that we should import pbr and make it a run-time dependency? | 23:36 |
| lifeless | fungi: its what we've been doing in many places | 23:36 |
| fungi | lifeless: i can switch to that easily enough, but it doesn't solve my immediate dilemma about whether and how to test optparse's magic --version handler | 23:37 |
| lifeless | fungi: there's some panic about performance | 23:37 |
| lifeless | fungi: don't test it, valueless. | 23:37 |
| fungi | we've definitely had instances in the past where --version didn't work in utilities, because it was untested | 23:38 |
| lifeless | we could even push the "%prog " thing back down | 23:38 |
| lifeless | fungi: sure, but it will be evaluated | 23:38 |
| lifeless | fungi: any test of the CLI will invoke the version code with this approach | 23:38 |
| fungi | lifeless: fair enough. i guess we'll generate the version on every invocation whether we display it or not in this case | 23:39 |
| fungi | so it's a safe assumption to make | 23:39 |
| lifeless | exactly | 23:39 |
| lifeless | it might not be a good version | 23:39 |
| lifeless | but it will be one | 23:39 |
| *** marun has quit IRC | 23:39 | |
| fungi | okay, i'll consider it an afternoon of sisyphean effort put to good use familiarizing myself with testing conventions and take the pragmatic route ;) | 23:40 |
| *** wenlock_ has quit IRC | 23:40 | |
| *** emagana has quit IRC | 23:40 | |
| lifeless | mordred: https://bugs.launchpad.net/pbr/+bug/1447666 <- any thoughts? | 23:41 |
| openstack | Launchpad bug 1447666 in PBR "PBR doesn't support builder option for build_sphinx command" [Undecided,New] | 23:41 |
| mordred | lifeless: I mean, "thoughts is a strong word | 23:41 |
| *** mase_x200 has quit IRC | 23:41 | |
| lifeless | mordred: I just haven't looked at that stuff at a ll | 23:41 |
| mordred | lifeless: oh - bother - I mean, yeah - we should probably not override command line options if one is given | 23:42 |
| mordred | lifeless: we should provide an option if one _isn't_ given | 23:42 |
| mordred | so I agree it's a bug | 23:42 |
| mordred | I'm not sure how much I care this instant | 23:42 |
| *** wenlock has quit IRC | 23:42 | |
| *** tnovacik has quit IRC | 23:44 | |
| lifeless | mordred: if you could say that - 'yes its a bug, right fix would be to honour the supplied command line' thats enough to move it to traiged yada yada yada | 23:45 |
| lifeless | mordred: or I can | 23:45 |
| openstackgerrit | Alexander Yip proposed openstack-infra/project-config: Add congress.yaml file https://review.openstack.org/178862 | 23:45 |
| mordred | lifeless: got it | 23:46 |
| alexsyip | Hi, I’m a newbie trying to enable some tempest tests for congress in jenkins. | 23:46 |
| alexsyip | I just pushed a change to gerrit https://review.openstack.org/178862 | 23:47 |
| mordred | lifeless: I cannot set triaged or importance | 23:47 |
| alexsyip | I’m wondering if that change looks right for turning on our tempest tests | 23:47 |
| *** dims has quit IRC | 23:50 | |
| *** chlong has joined #openstack-infra | 23:50 | |
| *** ddieterly has joined #openstack-infra | 23:51 | |
| *** dannywil_ has joined #openstack-infra | 23:52 | |
| *** dannywilson has quit IRC | 23:55 | |
| alexsyip | Also, is there anyway to test this congress.yaml without merging it? | 23:55 |
| lifeless | mordred: wat | 23:56 |
| lifeless | mordred: LOL | 23:56 |
| lifeless | mordred: I have anyhow | 23:57 |
| *** dannywil_ has quit IRC | 23:57 | |
| *** sputnik13 has quit IRC | 23:57 | |
| *** yamada-h has joined #openstack-infra | 23:58 | |
| *** HeOS has quit IRC | 23:58 | |
| openstackgerrit | Monty Taylor proposed openstack-dev/pbr: Unbreak the build_sphinx builder option https://review.openstack.org/178878 | 23:59 |
| mordred | lifeless: there ^^ | 23:59 |
| mordred | lifeless: turns out the fix was easy | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!